2023-03-09T14:26:03.332 INFO:root:teuthology version: 0.0.1.dev51+ga2dfb05 2023-03-09T14:26:03.333 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_pdonnell@teuthology --description fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/postgres}} --name pdonnell-2023-03-08_20:36:11-fs-wip-pdonnell-testing-20230308.150904-distro-default-smithi --verbose --archive /home/teuthworker/archive/pdonnell-2023-03-08_20:36:11-fs-wip-pdonnell-testing-20230308.150904-distro-default-smithi/7198014 -- /home/teuthworker/archive/pdonnell-2023-03-08_20:36:11-fs-wip-pdonnell-testing-20230308.150904-distro-default-smithi/7198014/orig.config.yaml 2023-03-09T14:26:03.396 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2023-03-09T14:26:03.489 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2023-03-08_20:36:11-fs-wip-pdonnell-testing-20230308.150904-distro-default-smithi/7198014 branch: wip-pdonnell-testing-20230308.150904 description: fs/workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/postgres}} email: pdonnell@redhat.com first_in_suite: false job_id: '7198014' kernel: client: sha1: distro kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: pdonnell-2023-03-08_20:36:11-fs-wip-pdonnell-testing-20230308.150904-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: wip-pdonnell-testing-20230308.150904 ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - slow metadata IO - SLOW_OPS - slow request sha1: e0ab30a2acead6db62b144ba6b6aeb23635adae7 subvols: create: 2 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mntopts: - context=system_u:object_r:postgresql_db_t:s0 mount_subvol_num: 0 mountpoint: /tmp/cephfs check-counter: counters: mds: - mds.exported - mds.imported - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount install: ceph: flavor: default sha1: e0ab30a2acead6db62b144ba6b6aeb23635adae7 kclient: client.0: mntopts: - context=system_u:object_r:postgresql_db_t:s0 mount_subvol_num: 1 mountpoint: /tmp/cephfs mntopts: - ms_mode=legacy - wsync syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-pdonnell-testing-20230308.150904 sha1: e0ab30a2acead6db62b144ba6b6aeb23635adae7 owner: scheduled_pdonnell@teuthology priority: 85 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 sha1: e0ab30a2acead6db62b144ba6b6aeb23635adae7 sleep_before_teardown: 0 suite: fs suite_branch: wip-pdonnell-testing-20230308.150904 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_e0ab30a2acead6db62b144ba6b6aeb23635adae7/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: e0ab30a2acead6db62b144ba6b6aeb23635adae7 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi038.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac= smithi122.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs= smithi181.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools:rhel8 --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - kclient: null - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1M - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6M3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - exec: client.0: - sudo ls -l /tmp/cephfs/ && sudo df -h /tmp/cephfs/ - sudo mkdir -m 755 --context=system_u:system_r:postgresql_t:s0 /tmp/cephfs/postgres && sudo chown postgres:postgres /tmp/cephfs/postgres - sudo -u postgres -- mkdir -m 700 --context=system_u:system_r:postgresql_t:s0 /tmp/cephfs/postgres/data - sudo semanage fcontext -a -t postgresql_db_t "/tmp/cephfs/postgres(/.*)?" - sudo grep -i postgresql /etc/selinux/targeted/contexts/files/file_contexts.local - sudo restorecon -R -v /tmp/cephfs/postgres - sudo ls -lZaR /tmp/cephfs/postgres/ - sudo mkdir -p /etc/systemd/system/postgresql.service.d/ && printf '[Service]\nEnvironment=PGDATA=/tmp/cephfs/postgres/data\nEnvironment=PGLOG=/tmp/cephfs/postgres/pgstartup.log\n' | sudo tee /etc/systemd/system/postgresql.service.d/env.conf - sudo -u postgres -- postgresql-setup --initdb - sudo ls -lZaR /tmp/cephfs/postgres/ - sudo systemctl start postgresql - sudo -u postgres -- pgbench -s 500 -i - sudo -u postgres -- pgbench -c 100 -j 4 --progress=5 --time=900 - sudo systemctl stop postgresql - sudo ls -lZaR /tmp/cephfs/postgres/ teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: a2dfb0599b24ec4a1585108a2792a4567ce0a378 timestamp: 2023-03-08_20:36:11 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.19450 2023-03-09T14:26:03.490 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_e0ab30a2acead6db62b144ba6b6aeb23635adae7/qa; will attempt to use it 2023-03-09T14:26:03.491 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_e0ab30a2acead6db62b144ba6b6aeb23635adae7/qa/tasks 2023-03-09T14:26:03.491 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-03-09T14:26:03.493 INFO:teuthology.task.internal:Checking packages... 2023-03-09T14:26:03.523 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash 'e0ab30a2acead6db62b144ba6b6aeb23635adae7' 2023-03-09T14:26:03.524 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-03-09T14:26:03.524 INFO:teuthology.packaging:ref: None 2023-03-09T14:26:03.524 INFO:teuthology.packaging:tag: None 2023-03-09T14:26:03.525 INFO:teuthology.packaging:branch: wip-pdonnell-testing-20230308.150904 2023-03-09T14:26:03.525 INFO:teuthology.packaging:sha1: e0ab30a2acead6db62b144ba6b6aeb23635adae7 2023-03-09T14:26:03.525 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-pdonnell-testing-20230308.150904 2023-03-09T14:26:03.676 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-2872.ge0ab30a2 2023-03-09T14:26:03.677 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-03-09T14:26:03.686 INFO:teuthology.task.internal:no buildpackages task found 2023-03-09T14:26:03.686 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-03-09T14:26:03.844 INFO:teuthology.task.internal:Saving configuration 2023-03-09T14:26:03.867 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-03-09T14:26:03.917 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-03-09T14:26:03.940 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi038.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-03-08_20:36:11-fs-wip-pdonnell-testing-20230308.150904-distro-default-smithi/7198014', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-03-09 14:20:49.952125', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-03-09T14:26:03.963 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi122.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-03-08_20:36:11-fs-wip-pdonnell-testing-20230308.150904-distro-default-smithi/7198014', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-03-09 14:20:49.953750', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-03-09T14:26:03.987 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi181.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-03-08_20:36:11-fs-wip-pdonnell-testing-20230308.150904-distro-default-smithi/7198014', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-03-09 14:20:49.955196', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-03-09T14:26:03.987 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-03-09T14:26:04.048 INFO:teuthology.task.internal:roles: ubuntu@smithi038.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2023-03-09T14:26:04.048 INFO:teuthology.task.internal:roles: ubuntu@smithi122.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2023-03-09T14:26:04.048 INFO:teuthology.task.internal:roles: ubuntu@smithi181.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2023-03-09T14:26:04.049 INFO:teuthology.run_tasks:Running task console_log... 2023-03-09T14:26:04.227 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f66301ca790>, signals=[15]) 2023-03-09T14:26:04.229 INFO:teuthology.run_tasks:Running task internal.connect... 2023-03-09T14:26:04.271 INFO:teuthology.task.internal:Opening connections... 2023-03-09T14:26:04.271 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi038.front.sepia.ceph.com 2023-03-09T14:26:04.274 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-03-09T14:26:04.344 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi122.front.sepia.ceph.com 2023-03-09T14:26:04.346 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-03-09T14:26:04.411 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi181.front.sepia.ceph.com 2023-03-09T14:26:04.412 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-03-09T14:26:04.480 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-03-09T14:26:04.489 DEBUG:teuthology.orchestra.run.smithi038:> uname -m 2023-03-09T14:26:04.511 INFO:teuthology.orchestra.run.smithi038.stdout:x86_64 2023-03-09T14:26:04.511 DEBUG:teuthology.orchestra.run.smithi038:> cat /etc/os-release 2023-03-09T14:26:04.571 INFO:teuthology.orchestra.run.smithi038.stdout:NAME="Red Hat Enterprise Linux" 2023-03-09T14:26:04.572 INFO:teuthology.orchestra.run.smithi038.stdout:VERSION="8.6 (Ootpa)" 2023-03-09T14:26:04.572 INFO:teuthology.orchestra.run.smithi038.stdout:ID="rhel" 2023-03-09T14:26:04.572 INFO:teuthology.orchestra.run.smithi038.stdout:ID_LIKE="fedora" 2023-03-09T14:26:04.572 INFO:teuthology.orchestra.run.smithi038.stdout:VERSION_ID="8.6" 2023-03-09T14:26:04.572 INFO:teuthology.orchestra.run.smithi038.stdout:PLATFORM_ID="platform:el8" 2023-03-09T14:26:04.572 INFO:teuthology.orchestra.run.smithi038.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-03-09T14:26:04.572 INFO:teuthology.orchestra.run.smithi038.stdout:ANSI_COLOR="0;31" 2023-03-09T14:26:04.573 INFO:teuthology.orchestra.run.smithi038.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-03-09T14:26:04.573 INFO:teuthology.orchestra.run.smithi038.stdout:HOME_URL="https://www.redhat.com/" 2023-03-09T14:26:04.573 INFO:teuthology.orchestra.run.smithi038.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-03-09T14:26:04.573 INFO:teuthology.orchestra.run.smithi038.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-03-09T14:26:04.573 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:26:04.573 INFO:teuthology.orchestra.run.smithi038.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-03-09T14:26:04.573 INFO:teuthology.orchestra.run.smithi038.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-03-09T14:26:04.574 INFO:teuthology.orchestra.run.smithi038.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-03-09T14:26:04.574 INFO:teuthology.orchestra.run.smithi038.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-03-09T14:26:04.574 INFO:teuthology.lock.ops:Updating smithi038.front.sepia.ceph.com on lock server 2023-03-09T14:26:04.599 DEBUG:teuthology.orchestra.run.smithi122:> uname -m 2023-03-09T14:26:04.619 INFO:teuthology.orchestra.run.smithi122.stdout:x86_64 2023-03-09T14:26:04.620 DEBUG:teuthology.orchestra.run.smithi122:> cat /etc/os-release 2023-03-09T14:26:04.679 INFO:teuthology.orchestra.run.smithi122.stdout:NAME="Red Hat Enterprise Linux" 2023-03-09T14:26:04.679 INFO:teuthology.orchestra.run.smithi122.stdout:VERSION="8.6 (Ootpa)" 2023-03-09T14:26:04.679 INFO:teuthology.orchestra.run.smithi122.stdout:ID="rhel" 2023-03-09T14:26:04.680 INFO:teuthology.orchestra.run.smithi122.stdout:ID_LIKE="fedora" 2023-03-09T14:26:04.680 INFO:teuthology.orchestra.run.smithi122.stdout:VERSION_ID="8.6" 2023-03-09T14:26:04.680 INFO:teuthology.orchestra.run.smithi122.stdout:PLATFORM_ID="platform:el8" 2023-03-09T14:26:04.680 INFO:teuthology.orchestra.run.smithi122.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-03-09T14:26:04.680 INFO:teuthology.orchestra.run.smithi122.stdout:ANSI_COLOR="0;31" 2023-03-09T14:26:04.680 INFO:teuthology.orchestra.run.smithi122.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-03-09T14:26:04.680 INFO:teuthology.orchestra.run.smithi122.stdout:HOME_URL="https://www.redhat.com/" 2023-03-09T14:26:04.680 INFO:teuthology.orchestra.run.smithi122.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-03-09T14:26:04.681 INFO:teuthology.orchestra.run.smithi122.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-03-09T14:26:04.681 INFO:teuthology.orchestra.run.smithi122.stdout: 2023-03-09T14:26:04.681 INFO:teuthology.orchestra.run.smithi122.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-03-09T14:26:04.681 INFO:teuthology.orchestra.run.smithi122.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-03-09T14:26:04.681 INFO:teuthology.orchestra.run.smithi122.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-03-09T14:26:04.681 INFO:teuthology.orchestra.run.smithi122.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-03-09T14:26:04.682 INFO:teuthology.lock.ops:Updating smithi122.front.sepia.ceph.com on lock server 2023-03-09T14:26:04.706 DEBUG:teuthology.orchestra.run.smithi181:> uname -m 2023-03-09T14:26:04.727 INFO:teuthology.orchestra.run.smithi181.stdout:x86_64 2023-03-09T14:26:04.734 DEBUG:teuthology.orchestra.run.smithi181:> cat /etc/os-release 2023-03-09T14:26:04.793 INFO:teuthology.orchestra.run.smithi181.stdout:NAME="Red Hat Enterprise Linux" 2023-03-09T14:26:04.794 INFO:teuthology.orchestra.run.smithi181.stdout:VERSION="8.6 (Ootpa)" 2023-03-09T14:26:04.794 INFO:teuthology.orchestra.run.smithi181.stdout:ID="rhel" 2023-03-09T14:26:04.794 INFO:teuthology.orchestra.run.smithi181.stdout:ID_LIKE="fedora" 2023-03-09T14:26:04.794 INFO:teuthology.orchestra.run.smithi181.stdout:VERSION_ID="8.6" 2023-03-09T14:26:04.794 INFO:teuthology.orchestra.run.smithi181.stdout:PLATFORM_ID="platform:el8" 2023-03-09T14:26:04.794 INFO:teuthology.orchestra.run.smithi181.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-03-09T14:26:04.794 INFO:teuthology.orchestra.run.smithi181.stdout:ANSI_COLOR="0;31" 2023-03-09T14:26:04.795 INFO:teuthology.orchestra.run.smithi181.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-03-09T14:26:04.795 INFO:teuthology.orchestra.run.smithi181.stdout:HOME_URL="https://www.redhat.com/" 2023-03-09T14:26:04.795 INFO:teuthology.orchestra.run.smithi181.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-03-09T14:26:04.795 INFO:teuthology.orchestra.run.smithi181.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-03-09T14:26:04.795 INFO:teuthology.orchestra.run.smithi181.stdout: 2023-03-09T14:26:04.795 INFO:teuthology.orchestra.run.smithi181.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-03-09T14:26:04.795 INFO:teuthology.orchestra.run.smithi181.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-03-09T14:26:04.796 INFO:teuthology.orchestra.run.smithi181.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-03-09T14:26:04.796 INFO:teuthology.orchestra.run.smithi181.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-03-09T14:26:04.796 INFO:teuthology.lock.ops:Updating smithi181.front.sepia.ceph.com on lock server 2023-03-09T14:26:04.815 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-03-09T14:26:04.831 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-03-09T14:26:04.958 INFO:teuthology.task.internal:Checking for old test directory... 2023-03-09T14:26:04.959 DEBUG:teuthology.orchestra.run.smithi038:> test '!' -e /home/ubuntu/cephtest 2023-03-09T14:26:04.962 DEBUG:teuthology.orchestra.run.smithi122:> test '!' -e /home/ubuntu/cephtest 2023-03-09T14:26:04.964 DEBUG:teuthology.orchestra.run.smithi181:> test '!' -e /home/ubuntu/cephtest 2023-03-09T14:26:04.982 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-03-09T14:26:05.033 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-03-09T14:26:05.034 DEBUG:teuthology.orchestra.run.smithi038:> test -z $(ls -A /var/lib/ceph) 2023-03-09T14:26:05.036 DEBUG:teuthology.orchestra.run.smithi122:> test -z $(ls -A /var/lib/ceph) 2023-03-09T14:26:05.039 DEBUG:teuthology.orchestra.run.smithi181:> test -z $(ls -A /var/lib/ceph) 2023-03-09T14:26:05.111 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-03-09T14:26:05.204 INFO:teuthology.run_tasks:Running task kernel... 2023-03-09T14:26:05.224 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'} 2023-03-09T14:26:05.225 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2023-03-09T14:26:05.226 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2023-03-09T14:26:05.226 DEBUG:teuthology.orchestra.run.smithi038:> test -f /run/.containerenv -o -f /.dockerenv 2023-03-09T14:26:05.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-03-09T14:26:05.246 DEBUG:teuthology.orchestra.run.smithi038:> uname -r 2023-03-09T14:26:05.306 INFO:teuthology.orchestra.run.smithi038.stdout:4.18.0-372.9.1.el8.x86_64 2023-03-09T14:26:05.307 INFO:teuthology.task.kernel:Running kernel on smithi038: 4.18.0-372.9.1.el8.x86_64 2023-03-09T14:26:05.307 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum install -y kernel 2023-03-09T14:26:06.572 INFO:teuthology.orchestra.run.smithi038.stdout:Updating Subscription Management repositories. 2023-03-09T14:26:06.573 INFO:teuthology.orchestra.run.smithi038.stdout:Unable to read consumer identity 2023-03-09T14:26:08.006 INFO:teuthology.orchestra.run.smithi038.stdout:Copr repo for python3-asyncssh owned by ceph 20 kB/s | 3.5 kB 00:00 2023-03-09T14:26:09.149 INFO:teuthology.orchestra.run.smithi038.stdout:Extra Packages for Enterprise Linux 12 MB/s | 14 MB 00:01 2023-03-09T14:26:12.415 INFO:teuthology.orchestra.run.smithi038.stdout:lab-extras 401 kB/s | 24 kB 00:00 2023-03-09T14:26:13.078 INFO:teuthology.orchestra.run.smithi038.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2023-03-09T14:26:13.143 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2023-03-09T14:26:13.145 INFO:teuthology.orchestra.run.smithi038.stdout:Nothing to do. 2023-03-09T14:26:13.145 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2023-03-09T14:26:13.265 DEBUG:teuthology.orchestra.run.smithi038:> echo no | sudo yum reinstall kernel || true 2023-03-09T14:26:13.615 INFO:teuthology.orchestra.run.smithi038.stdout:Updating Subscription Management repositories. 2023-03-09T14:26:13.615 INFO:teuthology.orchestra.run.smithi038.stdout:Unable to read consumer identity 2023-03-09T14:26:13.891 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:00:01 ago on Thu 09 Mar 2023 02:26:12 PM UTC. 2023-03-09T14:26:13.934 INFO:teuthology.orchestra.run.smithi038.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-03-09T14:26:13.976 INFO:teuthology.orchestra.run.smithi038.stderr:Error: No packages marked for reinstall. 2023-03-09T14:26:14.024 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum reinstall -y kernel || true 2023-03-09T14:26:14.372 INFO:teuthology.orchestra.run.smithi038.stdout:Updating Subscription Management repositories. 2023-03-09T14:26:14.373 INFO:teuthology.orchestra.run.smithi038.stdout:Unable to read consumer identity 2023-03-09T14:26:14.638 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:00:02 ago on Thu 09 Mar 2023 02:26:12 PM UTC. 2023-03-09T14:26:14.682 INFO:teuthology.orchestra.run.smithi038.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-03-09T14:26:14.717 INFO:teuthology.orchestra.run.smithi038.stderr:Error: No packages marked for reinstall. 2023-03-09T14:26:14.765 DEBUG:teuthology.orchestra.run.smithi038:> rpm -q kernel --last | head -n 1 2023-03-09T14:26:14.847 INFO:teuthology.orchestra.run.smithi038.stdout:kernel-4.18.0-372.9.1.el8.x86_64 Thu 14 Jul 2022 06:23:21 PM UTC 2023-03-09T14:26:14.848 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-372.9.1.el8.x86_64 2023-03-09T14:26:14.849 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2023-03-09T14:26:14.849 DEBUG:teuthology.parallel:result is None 2023-03-09T14:26:14.849 INFO:teuthology.run_tasks:Running task internal.base... 2023-03-09T14:26:14.858 INFO:teuthology.task.internal:Creating test directory... 2023-03-09T14:26:14.859 DEBUG:teuthology.orchestra.run.smithi038:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-03-09T14:26:14.861 DEBUG:teuthology.orchestra.run.smithi122:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-03-09T14:26:14.864 DEBUG:teuthology.orchestra.run.smithi181:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-03-09T14:26:14.884 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-03-09T14:26:14.956 INFO:teuthology.run_tasks:Running task internal.archive... 2023-03-09T14:26:15.052 INFO:teuthology.task.internal:Creating archive directory... 2023-03-09T14:26:15.052 DEBUG:teuthology.orchestra.run.smithi038:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-03-09T14:26:15.055 DEBUG:teuthology.orchestra.run.smithi122:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-03-09T14:26:15.057 DEBUG:teuthology.orchestra.run.smithi181:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-03-09T14:26:15.097 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-03-09T14:26:15.212 INFO:teuthology.task.internal:Enabling coredump saving... 2023-03-09T14:26:15.213 DEBUG:teuthology.orchestra.run.smithi122:> test -f /run/.containerenv -o -f /.dockerenv 2023-03-09T14:26:15.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-03-09T14:26:15.233 DEBUG:teuthology.orchestra.run.smithi181:> test -f /run/.containerenv -o -f /.dockerenv 2023-03-09T14:26:15.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-03-09T14:26:15.252 DEBUG:teuthology.orchestra.run.smithi038:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-03-09T14:26:15.254 DEBUG:teuthology.orchestra.run.smithi122:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-03-09T14:26:15.275 DEBUG:teuthology.orchestra.run.smithi181:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-03-09T14:26:15.288 INFO:teuthology.orchestra.run.smithi038.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-03-09T14:26:15.300 INFO:teuthology.orchestra.run.smithi038.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-03-09T14:26:15.320 INFO:teuthology.orchestra.run.smithi122.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-03-09T14:26:15.328 INFO:teuthology.orchestra.run.smithi181.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-03-09T14:26:15.336 INFO:teuthology.orchestra.run.smithi122.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-03-09T14:26:15.348 INFO:teuthology.orchestra.run.smithi181.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-03-09T14:26:15.349 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-03-09T14:26:15.357 INFO:teuthology.task.internal:Configuring sudo... 2023-03-09T14:26:15.358 DEBUG:teuthology.orchestra.run.smithi038:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-03-09T14:26:15.361 DEBUG:teuthology.orchestra.run.smithi122:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-03-09T14:26:15.363 DEBUG:teuthology.orchestra.run.smithi181:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-03-09T14:26:15.419 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-03-09T14:26:15.496 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-03-09T14:26:15.496 DEBUG:teuthology.orchestra.run.smithi038:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-03-09T14:26:15.499 DEBUG:teuthology.orchestra.run.smithi122:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-03-09T14:26:15.501 DEBUG:teuthology.orchestra.run.smithi181:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-03-09T14:26:15.518 DEBUG:teuthology.orchestra.run.smithi038:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-03-09T14:26:15.603 DEBUG:teuthology.orchestra.run.smithi038:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-03-09T14:26:15.651 DEBUG:teuthology.orchestra.run.smithi038:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-03-09T14:26:15.738 DEBUG:teuthology.orchestra.run.smithi038:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-03-09T14:26:15.778 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2023-03-09T14:26:15.779 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-03-09T14:26:15.846 DEBUG:teuthology.orchestra.run.smithi122:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-03-09T14:26:15.897 DEBUG:teuthology.orchestra.run.smithi122:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-03-09T14:26:15.945 DEBUG:teuthology.orchestra.run.smithi122:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-03-09T14:26:16.037 DEBUG:teuthology.orchestra.run.smithi122:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-03-09T14:26:16.082 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2023-03-09T14:26:16.083 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-03-09T14:26:16.155 DEBUG:teuthology.orchestra.run.smithi181:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-03-09T14:26:16.206 DEBUG:teuthology.orchestra.run.smithi181:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-03-09T14:26:16.264 DEBUG:teuthology.orchestra.run.smithi181:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-03-09T14:26:16.310 DEBUG:teuthology.orchestra.run.smithi181:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-03-09T14:26:16.353 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2023-03-09T14:26:16.353 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-03-09T14:26:16.422 DEBUG:teuthology.orchestra.run.smithi038:> sudo service rsyslog restart 2023-03-09T14:26:16.425 DEBUG:teuthology.orchestra.run.smithi122:> sudo service rsyslog restart 2023-03-09T14:26:16.428 DEBUG:teuthology.orchestra.run.smithi181:> sudo service rsyslog restart 2023-03-09T14:26:16.485 INFO:teuthology.orchestra.run.smithi038.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-03-09T14:26:16.495 INFO:teuthology.orchestra.run.smithi122.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-03-09T14:26:16.579 INFO:teuthology.orchestra.run.smithi181.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-03-09T14:26:16.763 INFO:teuthology.run_tasks:Running task internal.timer... 2023-03-09T14:26:16.771 INFO:teuthology.task.internal:Starting timer... 2023-03-09T14:26:16.771 INFO:teuthology.run_tasks:Running task pcp... 2023-03-09T14:26:16.885 INFO:teuthology.run_tasks:Running task selinux... 2023-03-09T14:26:16.958 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:logrotate_t:s0']} 2023-03-09T14:26:16.959 DEBUG:teuthology.orchestra.run.smithi038:> sudo service auditd rotate 2023-03-09T14:26:17.023 INFO:teuthology.orchestra.run.smithi038.stdout:Rotating logs: 2023-03-09T14:26:17.024 DEBUG:teuthology.orchestra.run.smithi122:> sudo service auditd rotate 2023-03-09T14:26:17.084 INFO:teuthology.orchestra.run.smithi122.stdout:Rotating logs: 2023-03-09T14:26:17.086 DEBUG:teuthology.orchestra.run.smithi181:> sudo service auditd rotate 2023-03-09T14:26:17.141 INFO:teuthology.orchestra.run.smithi181.stdout:Rotating logs: 2023-03-09T14:26:17.143 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-03-09T14:26:17.143 DEBUG:teuthology.orchestra.run.smithi038:> /usr/sbin/getenforce 2023-03-09T14:26:17.172 INFO:teuthology.orchestra.run.smithi038.stdout:Permissive 2023-03-09T14:26:17.173 DEBUG:teuthology.orchestra.run.smithi122:> /usr/sbin/getenforce 2023-03-09T14:26:17.197 INFO:teuthology.orchestra.run.smithi122.stdout:Permissive 2023-03-09T14:26:17.198 DEBUG:teuthology.orchestra.run.smithi181:> /usr/sbin/getenforce 2023-03-09T14:26:17.228 INFO:teuthology.orchestra.run.smithi181.stdout:Permissive 2023-03-09T14:26:17.228 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi038.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi122.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi181.front.sepia.ceph.com': 'permissive'} 2023-03-09T14:26:17.229 DEBUG:teuthology.orchestra.run.smithi038:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-03-09T14:26:17.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-03-09T14:26:17.259 DEBUG:teuthology.orchestra.run.smithi122:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-03-09T14:26:17.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-03-09T14:26:17.288 DEBUG:teuthology.orchestra.run.smithi181:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-03-09T14:26:17.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-03-09T14:26:17.315 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-03-09T14:26:17.316 DEBUG:teuthology.orchestra.run.smithi038:> sudo /usr/sbin/setenforce permissive 2023-03-09T14:26:17.343 DEBUG:teuthology.orchestra.run.smithi122:> sudo /usr/sbin/setenforce permissive 2023-03-09T14:26:17.372 DEBUG:teuthology.orchestra.run.smithi181:> sudo /usr/sbin/setenforce permissive 2023-03-09T14:26:17.398 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-03-09T14:26:17.409 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-03-09T14:26:17.410 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-03-09T14:26:17.429 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-03-09T14:26:17.431 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi038.front.sepia.ceph.com,smithi122.front.sepia.ceph.com,smithi181.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-03-09T14:33:45.203 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi038.front.sepia.ceph.com'), Remote(name='ubuntu@smithi122.front.sepia.ceph.com'), Remote(name='ubuntu@smithi181.front.sepia.ceph.com')] 2023-03-09T14:33:45.205 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi038.front.sepia.ceph.com' 2023-03-09T14:33:45.207 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-03-09T14:33:45.277 DEBUG:teuthology.orchestra.run.smithi038:> true 2023-03-09T14:33:45.348 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2023-03-09T14:33:45.349 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-03-09T14:33:45.416 DEBUG:teuthology.orchestra.run.smithi122:> true 2023-03-09T14:33:45.484 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi181.front.sepia.ceph.com' 2023-03-09T14:33:45.485 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi181.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-03-09T14:33:45.554 DEBUG:teuthology.orchestra.run.smithi181:> true 2023-03-09T14:33:45.624 INFO:teuthology.run_tasks:Running task clock... 2023-03-09T14:33:45.640 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-03-09T14:33:45.640 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-03-09T14:33:45.641 DEBUG:teuthology.orchestra.run.smithi038:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-03-09T14:33:45.644 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-03-09T14:33:45.644 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-03-09T14:33:45.646 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-03-09T14:33:45.646 DEBUG:teuthology.orchestra.run.smithi181:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-03-09T14:33:45.676 INFO:teuthology.orchestra.run.smithi122.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-03-09T14:33:45.680 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-03-09T14:33:45.691 INFO:teuthology.orchestra.run.smithi122.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-03-09T14:33:45.696 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-03-09T14:33:45.697 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-03-09T14:33:45.713 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-03-09T14:33:45.762 INFO:teuthology.orchestra.run.smithi122.stderr:sudo: ntpd: command not found 2023-03-09T14:33:45.771 INFO:teuthology.orchestra.run.smithi038.stderr:sudo: ntpd: command not found 2023-03-09T14:33:45.774 INFO:teuthology.orchestra.run.smithi122.stdout:506 Cannot talk to daemon 2023-03-09T14:33:45.785 INFO:teuthology.orchestra.run.smithi038.stdout:506 Cannot talk to daemon 2023-03-09T14:33:45.785 INFO:teuthology.orchestra.run.smithi181.stderr:sudo: ntpd: command not found 2023-03-09T14:33:45.789 INFO:teuthology.orchestra.run.smithi122.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-03-09T14:33:45.797 INFO:teuthology.orchestra.run.smithi181.stdout:506 Cannot talk to daemon 2023-03-09T14:33:45.801 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-03-09T14:33:45.802 INFO:teuthology.orchestra.run.smithi122.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-03-09T14:33:45.811 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-03-09T14:33:45.816 INFO:teuthology.orchestra.run.smithi038.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-03-09T14:33:45.825 INFO:teuthology.orchestra.run.smithi181.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-03-09T14:33:45.877 INFO:teuthology.orchestra.run.smithi122.stderr:bash: ntpq: command not found 2023-03-09T14:33:45.880 INFO:teuthology.orchestra.run.smithi122.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-03-09T14:33:45.880 INFO:teuthology.orchestra.run.smithi122.stdout:=============================================================================== 2023-03-09T14:33:45.880 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-03-09T14:33:45.880 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-03-09T14:33:45.881 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-03-09T14:33:45.881 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-03-09T14:33:45.894 INFO:teuthology.orchestra.run.smithi181.stderr:bash: ntpq: command not found 2023-03-09T14:33:45.897 INFO:teuthology.orchestra.run.smithi181.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-03-09T14:33:45.897 INFO:teuthology.orchestra.run.smithi181.stdout:=============================================================================== 2023-03-09T14:33:45.897 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-03-09T14:33:45.897 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-03-09T14:33:45.897 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-03-09T14:33:45.898 INFO:teuthology.orchestra.run.smithi181.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-03-09T14:33:45.900 INFO:teuthology.orchestra.run.smithi038.stderr:bash: ntpq: command not found 2023-03-09T14:33:45.903 INFO:teuthology.orchestra.run.smithi038.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-03-09T14:33:45.904 INFO:teuthology.orchestra.run.smithi038.stdout:=============================================================================== 2023-03-09T14:33:45.904 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-03-09T14:33:45.904 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-03-09T14:33:45.904 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-03-09T14:33:45.904 INFO:teuthology.orchestra.run.smithi038.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-03-09T14:33:45.905 INFO:teuthology.run_tasks:Running task pexec... 2023-03-09T14:33:45.918 INFO:teuthology.task.pexec:Executing custom commands... 2023-03-09T14:33:45.919 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi038.front.sepia.ceph.com 2023-03-09T14:33:45.919 DEBUG:teuthology.orchestra.run.smithi038:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-03-09T14:33:45.920 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi122.front.sepia.ceph.com 2023-03-09T14:33:45.920 DEBUG:teuthology.orchestra.run.smithi122:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-03-09T14:33:45.920 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi181.front.sepia.ceph.com 2023-03-09T14:33:45.921 DEBUG:teuthology.orchestra.run.smithi181:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-03-09T14:33:46.293 INFO:teuthology.orchestra.run.smithi122.stdout:Updating Subscription Management repositories. 2023-03-09T14:33:46.311 INFO:teuthology.orchestra.run.smithi181.stdout:Updating Subscription Management repositories. 2023-03-09T14:33:46.336 INFO:teuthology.orchestra.run.smithi038.stdout:Updating Subscription Management repositories. 2023-03-09T14:33:47.189 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:31 ago on Thu 09 Mar 2023 02:32:15 PM UTC. 2023-03-09T14:33:47.200 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:32 ago on Thu 09 Mar 2023 02:32:14 PM UTC. 2023-03-09T14:33:47.260 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:27 ago on Thu 09 Mar 2023 02:32:19 PM UTC. 2023-03-09T14:33:48.906 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2023-03-09T14:33:48.907 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2023-03-09T14:33:48.907 INFO:teuthology.orchestra.run.smithi122.stdout: Package Architecture Version Repository Size 2023-03-09T14:33:48.907 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2023-03-09T14:33:48.907 INFO:teuthology.orchestra.run.smithi122.stdout:Resetting modules: 2023-03-09T14:33:48.907 INFO:teuthology.orchestra.run.smithi122.stdout: container-tools 2023-03-09T14:33:48.908 INFO:teuthology.orchestra.run.smithi122.stdout: 2023-03-09T14:33:48.908 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2023-03-09T14:33:48.908 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2023-03-09T14:33:48.908 INFO:teuthology.orchestra.run.smithi122.stdout: 2023-03-09T14:33:48.953 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2023-03-09T14:33:48.954 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2023-03-09T14:33:48.954 INFO:teuthology.orchestra.run.smithi181.stdout: Package Architecture Version Repository Size 2023-03-09T14:33:48.954 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2023-03-09T14:33:48.955 INFO:teuthology.orchestra.run.smithi181.stdout:Resetting modules: 2023-03-09T14:33:48.955 INFO:teuthology.orchestra.run.smithi181.stdout: container-tools 2023-03-09T14:33:48.955 INFO:teuthology.orchestra.run.smithi181.stdout: 2023-03-09T14:33:48.955 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2023-03-09T14:33:48.955 INFO:teuthology.orchestra.run.smithi181.stdout:================================================================================ 2023-03-09T14:33:48.955 INFO:teuthology.orchestra.run.smithi181.stdout: 2023-03-09T14:33:49.232 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2023-03-09T14:33:49.233 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-03-09T14:33:49.233 INFO:teuthology.orchestra.run.smithi038.stdout: Package Architecture Version Repository Size 2023-03-09T14:33:49.233 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-03-09T14:33:49.233 INFO:teuthology.orchestra.run.smithi038.stdout:Resetting modules: 2023-03-09T14:33:49.233 INFO:teuthology.orchestra.run.smithi038.stdout: container-tools 2023-03-09T14:33:49.234 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:33:49.234 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2023-03-09T14:33:49.234 INFO:teuthology.orchestra.run.smithi038.stdout:================================================================================ 2023-03-09T14:33:49.234 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:33:49.871 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:34 ago on Thu 09 Mar 2023 02:32:15 PM UTC. 2023-03-09T14:33:49.917 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:35 ago on Thu 09 Mar 2023 02:32:14 PM UTC. 2023-03-09T14:33:50.213 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:30 ago on Thu 09 Mar 2023 02:32:19 PM UTC. 2023-03-09T14:33:53.062 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2023-03-09T14:33:53.162 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2023-03-09T14:33:53.331 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2023-03-09T14:33:53.533 INFO:teuthology.orchestra.run.smithi122.stdout:Updating Subscription Management repositories. 2023-03-09T14:33:53.812 INFO:teuthology.orchestra.run.smithi038.stdout:Updating Subscription Management repositories. 2023-03-09T14:33:53.898 INFO:teuthology.orchestra.run.smithi181.stdout:Updating Subscription Management repositories. 2023-03-09T14:33:54.346 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:38 ago on Thu 09 Mar 2023 02:32:15 PM UTC. 2023-03-09T14:33:54.668 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:35 ago on Thu 09 Mar 2023 02:32:19 PM UTC. 2023-03-09T14:33:54.813 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:40 ago on Thu 09 Mar 2023 02:32:14 PM UTC. 2023-03-09T14:33:56.206 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2023-03-09T14:33:56.300 INFO:teuthology.orchestra.run.smithi122.stdout:============================================================================================================ 2023-03-09T14:33:56.300 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2023-03-09T14:33:56.300 INFO:teuthology.orchestra.run.smithi122.stdout:============================================================================================================ 2023-03-09T14:33:56.300 INFO:teuthology.orchestra.run.smithi122.stdout:Installing group/module packages: 2023-03-09T14:33:56.300 INFO:teuthology.orchestra.run.smithi122.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-03-09T14:33:56.301 INFO:teuthology.orchestra.run.smithi122.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-03-09T14:33:56.301 INFO:teuthology.orchestra.run.smithi122.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-03-09T14:33:56.301 INFO:teuthology.orchestra.run.smithi122.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-03-09T14:33:56.301 INFO:teuthology.orchestra.run.smithi122.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-03-09T14:33:56.301 INFO:teuthology.orchestra.run.smithi122.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-03-09T14:33:56.301 INFO:teuthology.orchestra.run.smithi122.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-03-09T14:33:56.302 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2023-03-09T14:33:56.302 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-03-09T14:33:56.302 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-03-09T14:33:56.302 INFO:teuthology.orchestra.run.smithi122.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-03-09T14:33:56.302 INFO:teuthology.orchestra.run.smithi122.stdout:Installing module profiles: 2023-03-09T14:33:56.302 INFO:teuthology.orchestra.run.smithi122.stdout: container-tools/common 2023-03-09T14:33:56.303 INFO:teuthology.orchestra.run.smithi122.stdout: 2023-03-09T14:33:56.303 INFO:teuthology.orchestra.run.smithi122.stdout:Enabling module streams: 2023-03-09T14:33:56.303 INFO:teuthology.orchestra.run.smithi122.stdout: container-tools rhel8 2023-03-09T14:33:56.303 INFO:teuthology.orchestra.run.smithi122.stdout: 2023-03-09T14:33:56.303 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2023-03-09T14:33:56.303 INFO:teuthology.orchestra.run.smithi122.stdout:============================================================================================================ 2023-03-09T14:33:56.303 INFO:teuthology.orchestra.run.smithi122.stdout:Install 10 Packages 2023-03-09T14:33:56.303 INFO:teuthology.orchestra.run.smithi122.stdout: 2023-03-09T14:33:56.304 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 18 M 2023-03-09T14:33:56.304 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 65 M 2023-03-09T14:33:56.304 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2023-03-09T14:33:56.378 INFO:teuthology.orchestra.run.smithi122.stdout:(1/10): python3-pyxdg-0.25-16.el8.noarch.rpm 1.2 MB/s | 94 kB 00:00 2023-03-09T14:33:56.403 INFO:teuthology.orchestra.run.smithi122.stdout:(2/10): python3-pytoml-0.1.14-5.git7dea353.el8. 247 kB/s | 25 kB 00:00 2023-03-09T14:33:56.428 INFO:teuthology.orchestra.run.smithi122.stdout:(3/10): yajl-2.1.0-10.el8.x86_64.rpm 324 kB/s | 41 kB 00:00 2023-03-09T14:33:56.453 INFO:teuthology.orchestra.run.smithi122.stdout:(4/10): udica-0.2.6-4.module+el8.6.0+16771+28df 654 kB/s | 49 kB 00:00 2023-03-09T14:33:56.487 INFO:teuthology.orchestra.run.smithi122.stdout:(5/10): cockpit-podman-49.1-1.module+el8.6.0+16 6.1 MB/s | 524 kB 00:00 2023-03-09T14:33:56.512 INFO:teuthology.orchestra.run.smithi122.stdout:(6/10): python3-podman-4.0.1-1.module+el8.6.0+1 2.5 MB/s | 149 kB 00:00 2023-03-09T14:33:56.538 INFO:teuthology.orchestra.run.smithi122.stdout:(7/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 4.1 MB/s | 213 kB 00:00 2023-03-09T14:33:56.630 INFO:teuthology.orchestra.run.smithi122.stdout:(8/10): toolbox-0.0.99.3-0.6.module+el8.6.0+167 24 MB/s | 2.2 MB 00:00 2023-03-09T14:33:56.640 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2023-03-09T14:33:56.682 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2023-03-09T14:33:56.741 INFO:teuthology.orchestra.run.smithi038.stdout:============================================================================================================ 2023-03-09T14:33:56.742 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2023-03-09T14:33:56.742 INFO:teuthology.orchestra.run.smithi038.stdout:============================================================================================================ 2023-03-09T14:33:56.742 INFO:teuthology.orchestra.run.smithi038.stdout:Installing group/module packages: 2023-03-09T14:33:56.742 INFO:teuthology.orchestra.run.smithi038.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-03-09T14:33:56.743 INFO:teuthology.orchestra.run.smithi038.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-03-09T14:33:56.743 INFO:teuthology.orchestra.run.smithi038.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-03-09T14:33:56.743 INFO:teuthology.orchestra.run.smithi038.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-03-09T14:33:56.743 INFO:teuthology.orchestra.run.smithi038.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-03-09T14:33:56.743 INFO:teuthology.orchestra.run.smithi038.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-03-09T14:33:56.743 INFO:teuthology.orchestra.run.smithi038.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-03-09T14:33:56.743 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2023-03-09T14:33:56.743 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-03-09T14:33:56.744 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-03-09T14:33:56.744 INFO:teuthology.orchestra.run.smithi038.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-03-09T14:33:56.744 INFO:teuthology.orchestra.run.smithi038.stdout:Installing module profiles: 2023-03-09T14:33:56.744 INFO:teuthology.orchestra.run.smithi038.stdout: container-tools/common 2023-03-09T14:33:56.744 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:33:56.744 INFO:teuthology.orchestra.run.smithi038.stdout:Enabling module streams: 2023-03-09T14:33:56.744 INFO:teuthology.orchestra.run.smithi038.stdout: container-tools rhel8 2023-03-09T14:33:56.745 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:33:56.745 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2023-03-09T14:33:56.745 INFO:teuthology.orchestra.run.smithi038.stdout:============================================================================================================ 2023-03-09T14:33:56.745 INFO:teuthology.orchestra.run.smithi038.stdout:Install 10 Packages 2023-03-09T14:33:56.745 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:33:56.746 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 18 M 2023-03-09T14:33:56.746 INFO:teuthology.orchestra.run.smithi038.stdout:Installed size: 65 M 2023-03-09T14:33:56.746 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2023-03-09T14:33:56.755 INFO:teuthology.orchestra.run.smithi122.stdout:(9/10): skopeo-1.9.1-1.module+el8.6.0+16771+28d 29 MB/s | 7.0 MB 00:00 2023-03-09T14:33:56.779 INFO:teuthology.orchestra.run.smithi181.stdout:============================================================================================================ 2023-03-09T14:33:56.779 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2023-03-09T14:33:56.780 INFO:teuthology.orchestra.run.smithi181.stdout:============================================================================================================ 2023-03-09T14:33:56.780 INFO:teuthology.orchestra.run.smithi181.stdout:Installing group/module packages: 2023-03-09T14:33:56.780 INFO:teuthology.orchestra.run.smithi181.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-03-09T14:33:56.780 INFO:teuthology.orchestra.run.smithi181.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-03-09T14:33:56.780 INFO:teuthology.orchestra.run.smithi181.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-03-09T14:33:56.780 INFO:teuthology.orchestra.run.smithi181.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-03-09T14:33:56.780 INFO:teuthology.orchestra.run.smithi181.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-03-09T14:33:56.781 INFO:teuthology.orchestra.run.smithi181.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-03-09T14:33:56.781 INFO:teuthology.orchestra.run.smithi181.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-03-09T14:33:56.781 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2023-03-09T14:33:56.781 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-03-09T14:33:56.781 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-03-09T14:33:56.781 INFO:teuthology.orchestra.run.smithi181.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-03-09T14:33:56.781 INFO:teuthology.orchestra.run.smithi181.stdout:Installing module profiles: 2023-03-09T14:33:56.781 INFO:teuthology.orchestra.run.smithi181.stdout: container-tools/common 2023-03-09T14:33:56.782 INFO:teuthology.orchestra.run.smithi181.stdout: 2023-03-09T14:33:56.782 INFO:teuthology.orchestra.run.smithi181.stdout:Enabling module streams: 2023-03-09T14:33:56.782 INFO:teuthology.orchestra.run.smithi181.stdout: container-tools rhel8 2023-03-09T14:33:56.782 INFO:teuthology.orchestra.run.smithi181.stdout: 2023-03-09T14:33:56.782 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2023-03-09T14:33:56.782 INFO:teuthology.orchestra.run.smithi181.stdout:============================================================================================================ 2023-03-09T14:33:56.782 INFO:teuthology.orchestra.run.smithi181.stdout:Install 10 Packages 2023-03-09T14:33:56.783 INFO:teuthology.orchestra.run.smithi181.stdout: 2023-03-09T14:33:56.783 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 18 M 2023-03-09T14:33:56.783 INFO:teuthology.orchestra.run.smithi181.stdout:Installed size: 65 M 2023-03-09T14:33:56.783 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2023-03-09T14:33:56.807 INFO:teuthology.orchestra.run.smithi038.stdout:(1/10): yajl-2.1.0-10.el8.x86_64.rpm 647 kB/s | 41 kB 00:00 2023-03-09T14:33:56.824 INFO:teuthology.orchestra.run.smithi038.stdout:(2/10): python3-pytoml-0.1.14-5.git7dea353.el8. 310 kB/s | 25 kB 00:00 2023-03-09T14:33:56.841 INFO:teuthology.orchestra.run.smithi038.stdout:(3/10): udica-0.2.6-4.module+el8.6.0+16771+28df 1.4 MB/s | 49 kB 00:00 2023-03-09T14:33:56.867 INFO:teuthology.orchestra.run.smithi038.stdout:(4/10): python3-pyxdg-0.25-16.el8.noarch.rpm 767 kB/s | 94 kB 00:00 2023-03-09T14:33:56.872 INFO:teuthology.orchestra.run.smithi122.stdout:(10/10): buildah-1.26.2-1.module+el8.6.0+16771+ 18 MB/s | 8.0 MB 00:00 2023-03-09T14:33:56.872 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2023-03-09T14:33:56.873 INFO:teuthology.orchestra.run.smithi122.stdout:Total 32 MB/s | 18 MB 00:00 2023-03-09T14:33:56.900 INFO:teuthology.orchestra.run.smithi038.stdout:(5/10): cockpit-podman-49.1-1.module+el8.6.0+16 6.8 MB/s | 524 kB 00:00 2023-03-09T14:33:56.923 INFO:teuthology.orchestra.run.smithi181.stdout:(1/10): python3-pyxdg-0.25-16.el8.noarch.rpm 664 kB/s | 94 kB 00:00 2023-03-09T14:33:56.925 INFO:teuthology.orchestra.run.smithi038.stdout:(6/10): python3-podman-4.0.1-1.module+el8.6.0+1 2.5 MB/s | 149 kB 00:00 2023-03-09T14:33:56.940 INFO:teuthology.orchestra.run.smithi181.stdout:(2/10): python3-pytoml-0.1.14-5.git7dea353.el8. 157 kB/s | 25 kB 00:00 2023-03-09T14:33:56.957 INFO:teuthology.orchestra.run.smithi181.stdout:(3/10): yajl-2.1.0-10.el8.x86_64.rpm 233 kB/s | 41 kB 00:00 2023-03-09T14:33:56.974 INFO:teuthology.orchestra.run.smithi181.stdout:(4/10): udica-0.2.6-4.module+el8.6.0+16771+28df 977 kB/s | 49 kB 00:00 2023-03-09T14:33:57.008 INFO:teuthology.orchestra.run.smithi181.stdout:(5/10): cockpit-podman-49.1-1.module+el8.6.0+16 7.6 MB/s | 524 kB 00:00 2023-03-09T14:33:57.033 INFO:teuthology.orchestra.run.smithi181.stdout:(6/10): python3-podman-4.0.1-1.module+el8.6.0+1 2.5 MB/s | 149 kB 00:00 2023-03-09T14:33:57.058 INFO:teuthology.orchestra.run.smithi181.stdout:(7/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 4.1 MB/s | 213 kB 00:00 2023-03-09T14:33:57.068 INFO:teuthology.orchestra.run.smithi038.stdout:(7/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 1.2 MB/s | 213 kB 00:00 2023-03-09T14:33:57.076 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2023-03-09T14:33:57.095 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2023-03-09T14:33:57.095 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2023-03-09T14:33:57.168 INFO:teuthology.orchestra.run.smithi038.stdout:(8/10): toolbox-0.0.99.3-0.6.module+el8.6.0+167 22 MB/s | 2.2 MB 00:00 2023-03-09T14:33:57.175 INFO:teuthology.orchestra.run.smithi181.stdout:(8/10): toolbox-0.0.99.3-0.6.module+el8.6.0+167 19 MB/s | 2.2 MB 00:00 2023-03-09T14:33:57.276 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2023-03-09T14:33:57.280 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2023-03-09T14:33:57.292 INFO:teuthology.orchestra.run.smithi181.stdout:(9/10): skopeo-1.9.1-1.module+el8.6.0+16771+28d 27 MB/s | 7.0 MB 00:00 2023-03-09T14:33:57.302 INFO:teuthology.orchestra.run.smithi038.stdout:(9/10): skopeo-1.9.1-1.module+el8.6.0+16771+28d 19 MB/s | 7.0 MB 00:00 2023-03-09T14:33:57.376 INFO:teuthology.orchestra.run.smithi181.stdout:(10/10): buildah-1.26.2-1.module+el8.6.0+16771+ 19 MB/s | 8.0 MB 00:00 2023-03-09T14:33:57.376 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2023-03-09T14:33:57.377 INFO:teuthology.orchestra.run.smithi181.stdout:Total 31 MB/s | 18 MB 00:00 2023-03-09T14:33:57.419 INFO:teuthology.orchestra.run.smithi038.stdout:(10/10): buildah-1.26.2-1.module+el8.6.0+16771+ 14 MB/s | 8.0 MB 00:00 2023-03-09T14:33:57.419 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2023-03-09T14:33:57.419 INFO:teuthology.orchestra.run.smithi038.stdout:Total 27 MB/s | 18 MB 00:00 2023-03-09T14:33:57.578 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2023-03-09T14:33:57.597 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2023-03-09T14:33:57.597 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2023-03-09T14:33:57.629 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2023-03-09T14:33:57.649 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2023-03-09T14:33:57.649 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2023-03-09T14:33:57.779 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2023-03-09T14:33:57.783 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2023-03-09T14:33:57.804 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2023-03-09T14:33:57.843 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2023-03-09T14:33:57.847 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2023-03-09T14:33:58.009 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-03-09T14:33:58.207 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-03-09T14:33:58.288 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2023-03-09T14:33:58.382 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2023-03-09T14:33:58.393 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-03-09T14:33:58.483 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-03-09T14:33:58.571 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-03-09T14:33:58.604 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-03-09T14:33:58.647 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-03-09T14:33:58.733 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-03-09T14:33:58.883 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-03-09T14:33:58.965 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-03-09T14:33:58.986 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-03-09T14:33:59.086 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-03-09T14:33:59.168 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-03-09T14:33:59.425 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-03-09T14:33:59.540 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-03-09T14:33:59.763 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-03-09T14:34:00.185 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-03-09T14:34:00.336 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-03-09T14:34:00.631 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-03-09T14:34:00.892 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-03-09T14:34:01.044 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-03-09T14:34:01.065 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-03-09T14:34:01.241 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-03-09T14:34:01.276 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-03-09T14:34:01.292 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-03-09T14:34:01.358 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-03-09T14:34:01.358 INFO:teuthology.orchestra.run.smithi122.stdout:[/usr/lib/tmpfiles.d/journal-nocow.conf:26] Failed to resolve specifier: uninitialized /etc detected, skipping 2023-03-09T14:34:01.358 INFO:teuthology.orchestra.run.smithi122.stdout:All rules containing unresolvable specifiers will be skipped. 2023-03-09T14:34:01.359 INFO:teuthology.orchestra.run.smithi122.stdout: 2023-03-09T14:34:01.454 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-03-09T14:34:01.508 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-03-09T14:34:01.509 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-03-09T14:34:01.509 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-03-09T14:34:01.510 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-03-09T14:34:01.510 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-03-09T14:34:01.510 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-03-09T14:34:01.510 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-03-09T14:34:01.511 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-03-09T14:34:01.511 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-03-09T14:34:01.538 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-03-09T14:34:01.631 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-03-09T14:34:01.715 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-03-09T14:34:01.747 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-03-09T14:34:01.748 INFO:teuthology.orchestra.run.smithi181.stdout:[/usr/lib/tmpfiles.d/journal-nocow.conf:26] Failed to resolve specifier: uninitialized /etc detected, skipping 2023-03-09T14:34:01.748 INFO:teuthology.orchestra.run.smithi181.stdout:All rules containing unresolvable specifiers will be skipped. 2023-03-09T14:34:01.748 INFO:teuthology.orchestra.run.smithi181.stdout: 2023-03-09T14:34:01.874 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-03-09T14:34:01.901 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-03-09T14:34:01.902 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-03-09T14:34:01.902 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-03-09T14:34:01.902 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-03-09T14:34:01.902 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-03-09T14:34:01.902 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-03-09T14:34:01.902 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-03-09T14:34:01.903 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-03-09T14:34:01.903 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-03-09T14:34:01.992 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-03-09T14:34:01.992 INFO:teuthology.orchestra.run.smithi038.stdout:[/usr/lib/tmpfiles.d/journal-nocow.conf:26] Failed to resolve specifier: uninitialized /etc detected, skipping 2023-03-09T14:34:01.992 INFO:teuthology.orchestra.run.smithi038.stdout:All rules containing unresolvable specifiers will be skipped. 2023-03-09T14:34:01.992 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:34:02.152 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-03-09T14:34:02.153 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-03-09T14:34:02.153 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-03-09T14:34:02.153 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-03-09T14:34:02.153 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-03-09T14:34:02.154 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-03-09T14:34:02.154 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-03-09T14:34:02.154 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-03-09T14:34:02.154 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-03-09T14:34:02.607 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-03-09T14:34:02.607 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:47 ago on Thu 09 Mar 2023 02:32:15 PM UTC. 2023-03-09T14:34:03.038 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-03-09T14:34:03.038 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:48 ago on Thu 09 Mar 2023 02:32:14 PM UTC. 2023-03-09T14:34:03.354 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-03-09T14:34:03.355 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:43 ago on Thu 09 Mar 2023 02:32:19 PM UTC. 2023-03-09T14:34:05.386 INFO:teuthology.orchestra.run.smithi122.stdout:Installed products updated. 2023-03-09T14:34:05.584 INFO:teuthology.orchestra.run.smithi122.stdout: 2023-03-09T14:34:05.584 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2023-03-09T14:34:05.584 INFO:teuthology.orchestra.run.smithi122.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-03-09T14:34:05.585 INFO:teuthology.orchestra.run.smithi122.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-03-09T14:34:05.585 INFO:teuthology.orchestra.run.smithi122.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-03-09T14:34:05.585 INFO:teuthology.orchestra.run.smithi122.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-03-09T14:34:05.585 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-03-09T14:34:05.585 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-03-09T14:34:05.585 INFO:teuthology.orchestra.run.smithi122.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-03-09T14:34:05.585 INFO:teuthology.orchestra.run.smithi122.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-03-09T14:34:05.586 INFO:teuthology.orchestra.run.smithi122.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-03-09T14:34:05.586 INFO:teuthology.orchestra.run.smithi122.stdout: yajl-2.1.0-10.el8.x86_64 2023-03-09T14:34:05.586 INFO:teuthology.orchestra.run.smithi122.stdout: 2023-03-09T14:34:05.586 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2023-03-09T14:34:05.712 DEBUG:teuthology.parallel:result is None 2023-03-09T14:34:06.113 INFO:teuthology.orchestra.run.smithi181.stdout:Installed products updated. 2023-03-09T14:34:06.312 INFO:teuthology.orchestra.run.smithi181.stdout: 2023-03-09T14:34:06.313 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2023-03-09T14:34:06.313 INFO:teuthology.orchestra.run.smithi181.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-03-09T14:34:06.313 INFO:teuthology.orchestra.run.smithi181.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-03-09T14:34:06.313 INFO:teuthology.orchestra.run.smithi181.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-03-09T14:34:06.313 INFO:teuthology.orchestra.run.smithi181.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-03-09T14:34:06.313 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-03-09T14:34:06.313 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-03-09T14:34:06.314 INFO:teuthology.orchestra.run.smithi181.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-03-09T14:34:06.314 INFO:teuthology.orchestra.run.smithi181.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-03-09T14:34:06.314 INFO:teuthology.orchestra.run.smithi181.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-03-09T14:34:06.314 INFO:teuthology.orchestra.run.smithi181.stdout: yajl-2.1.0-10.el8.x86_64 2023-03-09T14:34:06.314 INFO:teuthology.orchestra.run.smithi181.stdout: 2023-03-09T14:34:06.314 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2023-03-09T14:34:06.316 INFO:teuthology.orchestra.run.smithi038.stdout:Installed products updated. 2023-03-09T14:34:06.449 DEBUG:teuthology.parallel:result is None 2023-03-09T14:34:06.538 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:34:06.538 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2023-03-09T14:34:06.538 INFO:teuthology.orchestra.run.smithi038.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-03-09T14:34:06.538 INFO:teuthology.orchestra.run.smithi038.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-03-09T14:34:06.539 INFO:teuthology.orchestra.run.smithi038.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-03-09T14:34:06.539 INFO:teuthology.orchestra.run.smithi038.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-03-09T14:34:06.539 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-03-09T14:34:06.539 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-03-09T14:34:06.539 INFO:teuthology.orchestra.run.smithi038.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-03-09T14:34:06.539 INFO:teuthology.orchestra.run.smithi038.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-03-09T14:34:06.539 INFO:teuthology.orchestra.run.smithi038.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-03-09T14:34:06.540 INFO:teuthology.orchestra.run.smithi038.stdout: yajl-2.1.0-10.el8.x86_64 2023-03-09T14:34:06.540 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:34:06.540 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2023-03-09T14:34:06.697 DEBUG:teuthology.parallel:result is None 2023-03-09T14:34:06.698 INFO:teuthology.run_tasks:Running task install... 2023-03-09T14:34:06.707 DEBUG:teuthology.task.install:project ceph 2023-03-09T14:34:06.708 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'e0ab30a2acead6db62b144ba6b6aeb23635adae7'}} 2023-03-09T14:34:06.708 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'e0ab30a2acead6db62b144ba6b6aeb23635adae7'} 2023-03-09T14:34:06.708 INFO:teuthology.task.install:Using flavor: default 2023-03-09T14:34:06.714 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-03-09T14:34:06.714 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-03-09T14:34:06.715 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'e0ab30a2acead6db62b144ba6b6aeb23635adae7', 'tag': None, 'wait_for_package': False} 2023-03-09T14:34:06.716 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=e0ab30a2acead6db62b144ba6b6aeb23635adae7 2023-03-09T14:34:06.718 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'e0ab30a2acead6db62b144ba6b6aeb23635adae7', 'tag': None, 'wait_for_package': False} 2023-03-09T14:34:06.718 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=e0ab30a2acead6db62b144ba6b6aeb23635adae7 2023-03-09T14:34:06.720 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'e0ab30a2acead6db62b144ba6b6aeb23635adae7', 'tag': None, 'wait_for_package': False} 2023-03-09T14:34:06.720 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=e0ab30a2acead6db62b144ba6b6aeb23635adae7 2023-03-09T14:34:06.870 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230308.150904/e0ab30a2acead6db62b144ba6b6aeb23635adae7/centos/8/flavors/default/ 2023-03-09T14:34:06.871 INFO:teuthology.task.install.rpm:Package version is 18.0.0-2872.ge0ab30a2 2023-03-09T14:34:06.876 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230308.150904/e0ab30a2acead6db62b144ba6b6aeb23635adae7/centos/8/flavors/default/ 2023-03-09T14:34:06.877 INFO:teuthology.task.install.rpm:Package version is 18.0.0-2872.ge0ab30a2 2023-03-09T14:34:06.881 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230308.150904/e0ab30a2acead6db62b144ba6b6aeb23635adae7/centos/8/flavors/default/ 2023-03-09T14:34:06.881 INFO:teuthology.task.install.rpm:Package version is 18.0.0-2872.ge0ab30a2 2023-03-09T14:34:06.979 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230308.150904/e0ab30a2acead6db62b144ba6b6aeb23635adae7/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230308.150904/e0ab30a2acead6db62b144ba6b6aeb23635adae7/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230308.150904/e0ab30a2acead6db62b144ba6b6aeb23635adae7/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-03-09T14:34:06.979 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2023-03-09T14:34:06.980 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-03-09T14:34:06.988 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230308.150904/e0ab30a2acead6db62b144ba6b6aeb23635adae7/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230308.150904/e0ab30a2acead6db62b144ba6b6aeb23635adae7/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230308.150904/e0ab30a2acead6db62b144ba6b6aeb23635adae7/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-03-09T14:34:06.988 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2023-03-09T14:34:06.988 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-03-09T14:34:06.992 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230308.150904/e0ab30a2acead6db62b144ba6b6aeb23635adae7/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230308.150904/e0ab30a2acead6db62b144ba6b6aeb23635adae7/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230308.150904/e0ab30a2acead6db62b144ba6b6aeb23635adae7/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-03-09T14:34:06.992 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2023-03-09T14:34:06.992 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-03-09T14:34:07.014 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-03-09T14:34:07.015 DEBUG:teuthology.orchestra.run.smithi038:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/e0ab30a2acead6db62b144ba6b6aeb23635adae7/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-03-09T14:34:07.020 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-03-09T14:34:07.021 DEBUG:teuthology.orchestra.run.smithi181:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/e0ab30a2acead6db62b144ba6b6aeb23635adae7/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-03-09T14:34:07.025 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-03-09T14:34:07.025 DEBUG:teuthology.orchestra.run.smithi122:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/e0ab30a2acead6db62b144ba6b6aeb23635adae7/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-03-09T14:34:07.083 DEBUG:teuthology.orchestra.run.smithi038:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-03-09T14:34:07.088 DEBUG:teuthology.orchestra.run.smithi181:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-03-09T14:34:07.099 DEBUG:teuthology.orchestra.run.smithi122:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-03-09T14:34:07.161 DEBUG:teuthology.orchestra.run.smithi038:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-03-09T14:34:07.165 DEBUG:teuthology.orchestra.run.smithi181:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-03-09T14:34:07.181 DEBUG:teuthology.orchestra.run.smithi122:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-03-09T14:34:07.227 INFO:teuthology.orchestra.run.smithi038.stdout:check_obsoletes = 1 2023-03-09T14:34:07.228 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum clean all 2023-03-09T14:34:07.231 INFO:teuthology.orchestra.run.smithi181.stdout:check_obsoletes = 1 2023-03-09T14:34:07.233 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum clean all 2023-03-09T14:34:07.252 INFO:teuthology.orchestra.run.smithi122.stdout:check_obsoletes = 1 2023-03-09T14:34:07.254 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum clean all 2023-03-09T14:34:07.603 INFO:teuthology.orchestra.run.smithi181.stdout:Updating Subscription Management repositories. 2023-03-09T14:34:07.621 INFO:teuthology.orchestra.run.smithi038.stdout:Updating Subscription Management repositories. 2023-03-09T14:34:07.628 INFO:teuthology.orchestra.run.smithi122.stdout:Updating Subscription Management repositories. 2023-03-09T14:34:07.903 INFO:teuthology.orchestra.run.smithi181.stdout:44 files removed 2023-03-09T14:34:07.948 DEBUG:teuthology.orchestra.run.smithi181:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-03-09T14:34:07.968 INFO:teuthology.orchestra.run.smithi122.stdout:44 files removed 2023-03-09T14:34:07.979 INFO:teuthology.orchestra.run.smithi038.stdout:44 files removed 2023-03-09T14:34:08.014 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-03-09T14:34:08.026 DEBUG:teuthology.orchestra.run.smithi038:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-03-09T14:34:08.282 INFO:teuthology.orchestra.run.smithi181.stdout:Updating Subscription Management repositories. 2023-03-09T14:34:08.344 INFO:teuthology.orchestra.run.smithi122.stdout:Updating Subscription Management repositories. 2023-03-09T14:34:08.375 INFO:teuthology.orchestra.run.smithi038.stdout:Updating Subscription Management repositories. 2023-03-09T14:34:08.888 INFO:teuthology.orchestra.run.smithi181.stdout:ceph packages for x86_64 310 kB/s | 80 kB 00:00 2023-03-09T14:34:08.962 INFO:teuthology.orchestra.run.smithi122.stdout:ceph packages for x86_64 300 kB/s | 80 kB 00:00 2023-03-09T14:34:09.014 INFO:teuthology.orchestra.run.smithi038.stdout:ceph packages for x86_64 321 kB/s | 80 kB 00:00 2023-03-09T14:34:09.138 INFO:teuthology.orchestra.run.smithi181.stdout:ceph noarch packages 43 kB/s | 9.4 kB 00:00 2023-03-09T14:34:09.213 INFO:teuthology.orchestra.run.smithi122.stdout:ceph noarch packages 43 kB/s | 9.4 kB 00:00 2023-03-09T14:34:09.282 INFO:teuthology.orchestra.run.smithi038.stdout:ceph noarch packages 40 kB/s | 9.4 kB 00:00 2023-03-09T14:34:09.372 INFO:teuthology.orchestra.run.smithi181.stdout:ceph source packages 7.0 kB/s | 1.5 kB 00:00 2023-03-09T14:34:09.447 INFO:teuthology.orchestra.run.smithi122.stdout:ceph source packages 6.9 kB/s | 1.5 kB 00:00 2023-03-09T14:34:09.541 INFO:teuthology.orchestra.run.smithi038.stdout:ceph source packages 6.3 kB/s | 1.5 kB 00:00 2023-03-09T14:34:09.548 INFO:teuthology.orchestra.run.smithi181.stdout:Copr repo for python3-asyncssh owned by ceph 22 kB/s | 3.5 kB 00:00 2023-03-09T14:34:09.590 INFO:teuthology.orchestra.run.smithi122.stdout:Copr repo for python3-asyncssh owned by ceph 28 kB/s | 3.5 kB 00:00 2023-03-09T14:34:09.700 INFO:teuthology.orchestra.run.smithi038.stdout:Copr repo for python3-asyncssh owned by ceph 25 kB/s | 3.5 kB 00:00 2023-03-09T14:34:11.581 INFO:teuthology.orchestra.run.smithi181.stdout:Extra Packages for Enterprise Linux 6.8 MB/s | 14 MB 00:02 2023-03-09T14:34:11.657 INFO:teuthology.orchestra.run.smithi122.stdout:Extra Packages for Enterprise Linux 6.7 MB/s | 14 MB 00:02 2023-03-09T14:34:11.784 INFO:teuthology.orchestra.run.smithi038.stdout:Extra Packages for Enterprise Linux 6.7 MB/s | 14 MB 00:02 2023-03-09T14:34:14.649 INFO:teuthology.orchestra.run.smithi181.stdout:lab-extras 254 kB/s | 24 kB 00:00 2023-03-09T14:34:14.782 INFO:teuthology.orchestra.run.smithi122.stdout:lab-extras 447 kB/s | 24 kB 00:00 2023-03-09T14:34:15.076 INFO:teuthology.orchestra.run.smithi038.stdout:lab-extras 438 kB/s | 24 kB 00:00 2023-03-09T14:34:15.659 INFO:teuthology.orchestra.run.smithi181.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 50 MB/s | 47 MB 00:00 2023-03-09T14:34:15.676 INFO:teuthology.orchestra.run.smithi122.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 57 MB/s | 47 MB 00:00 2023-03-09T14:34:16.345 INFO:teuthology.orchestra.run.smithi038.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 40 MB/s | 47 MB 00:01 2023-03-09T14:34:23.711 INFO:teuthology.orchestra.run.smithi122.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 59 MB/s | 53 MB 00:00 2023-03-09T14:34:23.768 INFO:teuthology.orchestra.run.smithi181.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 48 MB/s | 53 MB 00:01 2023-03-09T14:34:24.862 INFO:teuthology.orchestra.run.smithi038.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 56 MB/s | 53 MB 00:00 2023-03-09T14:34:35.038 INFO:teuthology.orchestra.run.smithi122.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-03-09T14:34:35.039 INFO:teuthology.orchestra.run.smithi122.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-03-09T14:34:35.053 INFO:teuthology.orchestra.run.smithi122.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-03-09T14:34:35.054 INFO:teuthology.orchestra.run.smithi122.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-03-09T14:34:35.056 INFO:teuthology.orchestra.run.smithi122.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-03-09T14:34:35.057 INFO:teuthology.orchestra.run.smithi122.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-03-09T14:34:35.059 INFO:teuthology.orchestra.run.smithi122.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-03-09T14:34:35.062 INFO:teuthology.orchestra.run.smithi122.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-03-09T14:34:35.063 INFO:teuthology.orchestra.run.smithi122.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-03-09T14:34:35.113 INFO:teuthology.orchestra.run.smithi181.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-03-09T14:34:35.114 INFO:teuthology.orchestra.run.smithi181.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-03-09T14:34:35.129 INFO:teuthology.orchestra.run.smithi181.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-03-09T14:34:35.130 INFO:teuthology.orchestra.run.smithi181.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-03-09T14:34:35.131 INFO:teuthology.orchestra.run.smithi181.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-03-09T14:34:35.132 INFO:teuthology.orchestra.run.smithi181.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-03-09T14:34:35.135 INFO:teuthology.orchestra.run.smithi181.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-03-09T14:34:35.138 INFO:teuthology.orchestra.run.smithi181.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-03-09T14:34:35.139 INFO:teuthology.orchestra.run.smithi181.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-03-09T14:34:35.196 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2023-03-09T14:34:35.209 INFO:teuthology.orchestra.run.smithi122.stdout:=========================================================================================================================== 2023-03-09T14:34:35.210 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2023-03-09T14:34:35.210 INFO:teuthology.orchestra.run.smithi122.stdout:=========================================================================================================================== 2023-03-09T14:34:35.210 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2023-03-09T14:34:35.210 INFO:teuthology.orchestra.run.smithi122.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-03-09T14:34:35.211 INFO:teuthology.orchestra.run.smithi122.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-03-09T14:34:35.211 INFO:teuthology.orchestra.run.smithi122.stdout: ceph x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 6.4 k 2023-03-09T14:34:35.211 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-base x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 5.4 M 2023-03-09T14:34:35.211 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-fuse x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 845 k 2023-03-09T14:34:35.211 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 166 k 2023-03-09T14:34:35.211 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 1.5 M 2023-03-09T14:34:35.211 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-cephadm noarch 2:18.0.0-2872.ge0ab30a2.el8 ceph-noarch 123 k 2023-03-09T14:34:35.212 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-dashboard noarch 2:18.0.0-2872.ge0ab30a2.el8 ceph-noarch 1.7 M 2023-03-09T14:34:35.212 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-2872.ge0ab30a2.el8 ceph-noarch 7.4 M 2023-03-09T14:34:35.212 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-rook noarch 2:18.0.0-2872.ge0ab30a2.el8 ceph-noarch 47 k 2023-03-09T14:34:35.212 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-radosgw x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 13 M 2023-03-09T14:34:35.212 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-test x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 47 M 2023-03-09T14:34:35.212 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-volume noarch 2:18.0.0-2872.ge0ab30a2.el8 ceph-noarch 263 k 2023-03-09T14:34:35.212 INFO:teuthology.orchestra.run.smithi122.stdout: cephadm noarch 2:18.0.0-2872.ge0ab30a2.el8 ceph-noarch 83 k 2023-03-09T14:34:35.212 INFO:teuthology.orchestra.run.smithi122.stdout: cephfs-mirror x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 257 k 2023-03-09T14:34:35.213 INFO:teuthology.orchestra.run.smithi122.stdout: cephfs-top noarch 2:18.0.0-2872.ge0ab30a2.el8 ceph-noarch 18 k 2023-03-09T14:34:35.213 INFO:teuthology.orchestra.run.smithi122.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-03-09T14:34:35.213 INFO:teuthology.orchestra.run.smithi122.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-03-09T14:34:35.213 INFO:teuthology.orchestra.run.smithi122.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-03-09T14:34:35.213 INFO:teuthology.orchestra.run.smithi122.stdout: libcephfs-devel x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 30 k 2023-03-09T14:34:35.213 INFO:teuthology.orchestra.run.smithi122.stdout: libcephfs2 x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 768 k 2023-03-09T14:34:35.214 INFO:teuthology.orchestra.run.smithi122.stdout: librados-devel x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 139 k 2023-03-09T14:34:35.214 INFO:teuthology.orchestra.run.smithi122.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-03-09T14:34:35.214 INFO:teuthology.orchestra.run.smithi122.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-03-09T14:34:35.214 INFO:teuthology.orchestra.run.smithi122.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-03-09T14:34:35.214 INFO:teuthology.orchestra.run.smithi122.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-03-09T14:34:35.214 INFO:teuthology.orchestra.run.smithi122.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-03-09T14:34:35.214 INFO:teuthology.orchestra.run.smithi122.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-03-09T14:34:35.215 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cephfs x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 190 k 2023-03-09T14:34:35.215 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rados x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 371 k 2023-03-09T14:34:35.215 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rbd x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 364 k 2023-03-09T14:34:35.215 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rgw x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 106 k 2023-03-09T14:34:35.215 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-fuse x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 94 k 2023-03-09T14:34:35.215 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-mirror x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 3.4 M 2023-03-09T14:34:35.215 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-nbd x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 180 k 2023-03-09T14:34:35.215 INFO:teuthology.orchestra.run.smithi122.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-03-09T14:34:35.216 INFO:teuthology.orchestra.run.smithi122.stdout:Upgrading: 2023-03-09T14:34:35.216 INFO:teuthology.orchestra.run.smithi122.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-03-09T14:34:35.216 INFO:teuthology.orchestra.run.smithi122.stdout: librados2 x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 3.7 M 2023-03-09T14:34:35.216 INFO:teuthology.orchestra.run.smithi122.stdout: librbd1 x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 3.6 M 2023-03-09T14:34:35.216 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2023-03-09T14:34:35.216 INFO:teuthology.orchestra.run.smithi122.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-03-09T14:34:35.216 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-common x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 22 M 2023-03-09T14:34:35.217 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-grafana-dashboards noarch 2:18.0.0-2872.ge0ab30a2.el8 ceph-noarch 23 k 2023-03-09T14:34:35.217 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mds x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 2.3 M 2023-03-09T14:34:35.217 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-modules-core noarch 2:18.0.0-2872.ge0ab30a2.el8 ceph-noarch 239 k 2023-03-09T14:34:35.217 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mon x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 4.6 M 2023-03-09T14:34:35.217 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-osd x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 18 M 2023-03-09T14:34:35.217 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-prometheus-alerts noarch 2:18.0.0-2872.ge0ab30a2.el8 ceph-noarch 15 k 2023-03-09T14:34:35.217 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-selinux x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 24 k 2023-03-09T14:34:35.218 INFO:teuthology.orchestra.run.smithi122.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-03-09T14:34:35.218 INFO:teuthology.orchestra.run.smithi122.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-03-09T14:34:35.218 INFO:teuthology.orchestra.run.smithi122.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-03-09T14:34:35.218 INFO:teuthology.orchestra.run.smithi122.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-03-09T14:34:35.218 INFO:teuthology.orchestra.run.smithi122.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-03-09T14:34:35.218 INFO:teuthology.orchestra.run.smithi122.stdout: libcephsqlite x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 137 k 2023-03-09T14:34:35.218 INFO:teuthology.orchestra.run.smithi122.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-03-09T14:34:35.219 INFO:teuthology.orchestra.run.smithi122.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-03-09T14:34:35.219 INFO:teuthology.orchestra.run.smithi122.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-03-09T14:34:35.219 INFO:teuthology.orchestra.run.smithi122.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-03-09T14:34:35.219 INFO:teuthology.orchestra.run.smithi122.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-03-09T14:34:35.219 INFO:teuthology.orchestra.run.smithi122.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-03-09T14:34:35.219 INFO:teuthology.orchestra.run.smithi122.stdout: libradosstriper1 x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 474 k 2023-03-09T14:34:35.219 INFO:teuthology.orchestra.run.smithi122.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-03-09T14:34:35.220 INFO:teuthology.orchestra.run.smithi122.stdout: librgw2 x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 6.5 M 2023-03-09T14:34:35.220 INFO:teuthology.orchestra.run.smithi122.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-03-09T14:34:35.220 INFO:teuthology.orchestra.run.smithi122.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-03-09T14:34:35.220 INFO:teuthology.orchestra.run.smithi122.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-03-09T14:34:35.220 INFO:teuthology.orchestra.run.smithi122.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-03-09T14:34:35.220 INFO:teuthology.orchestra.run.smithi122.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-03-09T14:34:35.220 INFO:teuthology.orchestra.run.smithi122.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-03-09T14:34:35.220 INFO:teuthology.orchestra.run.smithi122.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-03-09T14:34:35.221 INFO:teuthology.orchestra.run.smithi122.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-03-09T14:34:35.221 INFO:teuthology.orchestra.run.smithi122.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-03-09T14:34:35.221 INFO:teuthology.orchestra.run.smithi122.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-03-09T14:34:35.221 INFO:teuthology.orchestra.run.smithi122.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-03-09T14:34:35.221 INFO:teuthology.orchestra.run.smithi122.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-03-09T14:34:35.221 INFO:teuthology.orchestra.run.smithi122.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-03-09T14:34:35.221 INFO:teuthology.orchestra.run.smithi122.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-03-09T14:34:35.222 INFO:teuthology.orchestra.run.smithi122.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-03-09T14:34:35.222 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-03-09T14:34:35.222 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ceph-argparse x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 45 k 2023-03-09T14:34:35.222 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ceph-common x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 117 k 2023-03-09T14:34:35.222 INFO:teuthology.orchestra.run.smithi122.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-03-09T14:34:35.222 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-03-09T14:34:35.223 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-03-09T14:34:35.223 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-03-09T14:34:35.223 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-03-09T14:34:35.223 INFO:teuthology.orchestra.run.smithi122.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-03-09T14:34:35.223 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-03-09T14:34:35.223 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-03-09T14:34:35.223 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-03-09T14:34:35.224 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-03-09T14:34:35.224 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-03-09T14:34:35.224 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-03-09T14:34:35.224 INFO:teuthology.orchestra.run.smithi122.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-03-09T14:34:35.224 INFO:teuthology.orchestra.run.smithi122.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-03-09T14:34:35.224 INFO:teuthology.orchestra.run.smithi122.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-03-09T14:34:35.224 INFO:teuthology.orchestra.run.smithi122.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-03-09T14:34:35.224 INFO:teuthology.orchestra.run.smithi122.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-03-09T14:34:35.225 INFO:teuthology.orchestra.run.smithi122.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-03-09T14:34:35.225 INFO:teuthology.orchestra.run.smithi122.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-03-09T14:34:35.225 INFO:teuthology.orchestra.run.smithi122.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-03-09T14:34:35.225 INFO:teuthology.orchestra.run.smithi122.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-03-09T14:34:35.225 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-03-09T14:34:35.225 INFO:teuthology.orchestra.run.smithi122.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-03-09T14:34:35.226 INFO:teuthology.orchestra.run.smithi122.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-03-09T14:34:35.226 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-03-09T14:34:35.226 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-03-09T14:34:35.226 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-03-09T14:34:35.226 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-03-09T14:34:35.226 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-03-09T14:34:35.226 INFO:teuthology.orchestra.run.smithi122.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-03-09T14:34:35.226 INFO:teuthology.orchestra.run.smithi122.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-03-09T14:34:35.227 INFO:teuthology.orchestra.run.smithi122.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-03-09T14:34:35.227 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-03-09T14:34:35.227 INFO:teuthology.orchestra.run.smithi122.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-03-09T14:34:35.227 INFO:teuthology.orchestra.run.smithi122.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-03-09T14:34:35.227 INFO:teuthology.orchestra.run.smithi122.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-03-09T14:34:35.227 INFO:teuthology.orchestra.run.smithi122.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-03-09T14:34:35.227 INFO:teuthology.orchestra.run.smithi122.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-03-09T14:34:35.228 INFO:teuthology.orchestra.run.smithi122.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2023-03-09T14:34:35.228 INFO:teuthology.orchestra.run.smithi122.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-03-09T14:34:35.228 INFO:teuthology.orchestra.run.smithi122.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-03-09T14:34:35.228 INFO:teuthology.orchestra.run.smithi122.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-03-09T14:34:35.228 INFO:teuthology.orchestra.run.smithi122.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-03-09T14:34:35.228 INFO:teuthology.orchestra.run.smithi122.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-03-09T14:34:35.228 INFO:teuthology.orchestra.run.smithi122.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-03-09T14:34:35.228 INFO:teuthology.orchestra.run.smithi122.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-03-09T14:34:35.229 INFO:teuthology.orchestra.run.smithi122.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-03-09T14:34:35.229 INFO:teuthology.orchestra.run.smithi122.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-03-09T14:34:35.229 INFO:teuthology.orchestra.run.smithi122.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-03-09T14:34:35.229 INFO:teuthology.orchestra.run.smithi122.stdout:Enabling module streams: 2023-03-09T14:34:35.229 INFO:teuthology.orchestra.run.smithi122.stdout: postgresql 10 2023-03-09T14:34:35.229 INFO:teuthology.orchestra.run.smithi122.stdout: 2023-03-09T14:34:35.229 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2023-03-09T14:34:35.230 INFO:teuthology.orchestra.run.smithi122.stdout:=========================================================================================================================== 2023-03-09T14:34:35.230 INFO:teuthology.orchestra.run.smithi122.stdout:Install 128 Packages 2023-03-09T14:34:35.230 INFO:teuthology.orchestra.run.smithi122.stdout:Upgrade 3 Packages 2023-03-09T14:34:35.230 INFO:teuthology.orchestra.run.smithi122.stdout: 2023-03-09T14:34:35.231 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 203 M 2023-03-09T14:34:35.231 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2023-03-09T14:34:35.270 INFO:teuthology.orchestra.run.smithi181.stdout:Dependencies resolved. 2023-03-09T14:34:35.282 INFO:teuthology.orchestra.run.smithi181.stdout:=========================================================================================================================== 2023-03-09T14:34:35.282 INFO:teuthology.orchestra.run.smithi181.stdout: Package Arch Version Repository Size 2023-03-09T14:34:35.283 INFO:teuthology.orchestra.run.smithi181.stdout:=========================================================================================================================== 2023-03-09T14:34:35.283 INFO:teuthology.orchestra.run.smithi181.stdout:Installing: 2023-03-09T14:34:35.283 INFO:teuthology.orchestra.run.smithi181.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-03-09T14:34:35.283 INFO:teuthology.orchestra.run.smithi181.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-03-09T14:34:35.283 INFO:teuthology.orchestra.run.smithi181.stdout: ceph x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 6.4 k 2023-03-09T14:34:35.283 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-base x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 5.4 M 2023-03-09T14:34:35.283 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-fuse x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 845 k 2023-03-09T14:34:35.284 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 166 k 2023-03-09T14:34:35.284 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 1.5 M 2023-03-09T14:34:35.284 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-cephadm noarch 2:18.0.0-2872.ge0ab30a2.el8 ceph-noarch 123 k 2023-03-09T14:34:35.284 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-dashboard noarch 2:18.0.0-2872.ge0ab30a2.el8 ceph-noarch 1.7 M 2023-03-09T14:34:35.284 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-2872.ge0ab30a2.el8 ceph-noarch 7.4 M 2023-03-09T14:34:35.284 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-rook noarch 2:18.0.0-2872.ge0ab30a2.el8 ceph-noarch 47 k 2023-03-09T14:34:35.284 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-radosgw x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 13 M 2023-03-09T14:34:35.285 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-test x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 47 M 2023-03-09T14:34:35.285 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-volume noarch 2:18.0.0-2872.ge0ab30a2.el8 ceph-noarch 263 k 2023-03-09T14:34:35.285 INFO:teuthology.orchestra.run.smithi181.stdout: cephadm noarch 2:18.0.0-2872.ge0ab30a2.el8 ceph-noarch 83 k 2023-03-09T14:34:35.285 INFO:teuthology.orchestra.run.smithi181.stdout: cephfs-mirror x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 257 k 2023-03-09T14:34:35.285 INFO:teuthology.orchestra.run.smithi181.stdout: cephfs-top noarch 2:18.0.0-2872.ge0ab30a2.el8 ceph-noarch 18 k 2023-03-09T14:34:35.285 INFO:teuthology.orchestra.run.smithi181.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-03-09T14:34:35.285 INFO:teuthology.orchestra.run.smithi181.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-03-09T14:34:35.286 INFO:teuthology.orchestra.run.smithi181.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-03-09T14:34:35.286 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs-devel x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 30 k 2023-03-09T14:34:35.286 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs2 x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 768 k 2023-03-09T14:34:35.286 INFO:teuthology.orchestra.run.smithi181.stdout: librados-devel x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 139 k 2023-03-09T14:34:35.286 INFO:teuthology.orchestra.run.smithi181.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-03-09T14:34:35.286 INFO:teuthology.orchestra.run.smithi181.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-03-09T14:34:35.286 INFO:teuthology.orchestra.run.smithi181.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-03-09T14:34:35.287 INFO:teuthology.orchestra.run.smithi181.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-03-09T14:34:35.287 INFO:teuthology.orchestra.run.smithi181.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-03-09T14:34:35.287 INFO:teuthology.orchestra.run.smithi181.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-03-09T14:34:35.287 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cephfs x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 190 k 2023-03-09T14:34:35.287 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rados x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 371 k 2023-03-09T14:34:35.287 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rbd x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 364 k 2023-03-09T14:34:35.287 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rgw x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 106 k 2023-03-09T14:34:35.288 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-fuse x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 94 k 2023-03-09T14:34:35.288 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-mirror x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 3.4 M 2023-03-09T14:34:35.288 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-nbd x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 180 k 2023-03-09T14:34:35.288 INFO:teuthology.orchestra.run.smithi181.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-03-09T14:34:35.288 INFO:teuthology.orchestra.run.smithi181.stdout:Upgrading: 2023-03-09T14:34:35.288 INFO:teuthology.orchestra.run.smithi181.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-03-09T14:34:35.288 INFO:teuthology.orchestra.run.smithi181.stdout: librados2 x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 3.7 M 2023-03-09T14:34:35.289 INFO:teuthology.orchestra.run.smithi181.stdout: librbd1 x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 3.6 M 2023-03-09T14:34:35.289 INFO:teuthology.orchestra.run.smithi181.stdout:Installing dependencies: 2023-03-09T14:34:35.289 INFO:teuthology.orchestra.run.smithi181.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-03-09T14:34:35.289 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-common x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 22 M 2023-03-09T14:34:35.289 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-grafana-dashboards noarch 2:18.0.0-2872.ge0ab30a2.el8 ceph-noarch 23 k 2023-03-09T14:34:35.290 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mds x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 2.3 M 2023-03-09T14:34:35.290 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-modules-core noarch 2:18.0.0-2872.ge0ab30a2.el8 ceph-noarch 239 k 2023-03-09T14:34:35.290 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mon x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 4.6 M 2023-03-09T14:34:35.290 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-osd x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 18 M 2023-03-09T14:34:35.290 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-prometheus-alerts noarch 2:18.0.0-2872.ge0ab30a2.el8 ceph-noarch 15 k 2023-03-09T14:34:35.290 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-selinux x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 24 k 2023-03-09T14:34:35.290 INFO:teuthology.orchestra.run.smithi181.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-03-09T14:34:35.291 INFO:teuthology.orchestra.run.smithi181.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-03-09T14:34:35.291 INFO:teuthology.orchestra.run.smithi181.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-03-09T14:34:35.291 INFO:teuthology.orchestra.run.smithi181.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-03-09T14:34:35.291 INFO:teuthology.orchestra.run.smithi181.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-03-09T14:34:35.291 INFO:teuthology.orchestra.run.smithi181.stdout: libcephsqlite x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 137 k 2023-03-09T14:34:35.291 INFO:teuthology.orchestra.run.smithi181.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-03-09T14:34:35.291 INFO:teuthology.orchestra.run.smithi181.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-03-09T14:34:35.292 INFO:teuthology.orchestra.run.smithi181.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-03-09T14:34:35.292 INFO:teuthology.orchestra.run.smithi181.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-03-09T14:34:35.292 INFO:teuthology.orchestra.run.smithi181.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-03-09T14:34:35.292 INFO:teuthology.orchestra.run.smithi181.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-03-09T14:34:35.292 INFO:teuthology.orchestra.run.smithi181.stdout: libradosstriper1 x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 474 k 2023-03-09T14:34:35.292 INFO:teuthology.orchestra.run.smithi181.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-03-09T14:34:35.292 INFO:teuthology.orchestra.run.smithi181.stdout: librgw2 x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 6.5 M 2023-03-09T14:34:35.293 INFO:teuthology.orchestra.run.smithi181.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-03-09T14:34:35.293 INFO:teuthology.orchestra.run.smithi181.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-03-09T14:34:35.293 INFO:teuthology.orchestra.run.smithi181.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-03-09T14:34:35.293 INFO:teuthology.orchestra.run.smithi181.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-03-09T14:34:35.293 INFO:teuthology.orchestra.run.smithi181.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-03-09T14:34:35.293 INFO:teuthology.orchestra.run.smithi181.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-03-09T14:34:35.293 INFO:teuthology.orchestra.run.smithi181.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-03-09T14:34:35.294 INFO:teuthology.orchestra.run.smithi181.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-03-09T14:34:35.294 INFO:teuthology.orchestra.run.smithi181.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-03-09T14:34:35.294 INFO:teuthology.orchestra.run.smithi181.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-03-09T14:34:35.294 INFO:teuthology.orchestra.run.smithi181.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-03-09T14:34:35.294 INFO:teuthology.orchestra.run.smithi181.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-03-09T14:34:35.294 INFO:teuthology.orchestra.run.smithi181.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-03-09T14:34:35.294 INFO:teuthology.orchestra.run.smithi181.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-03-09T14:34:35.294 INFO:teuthology.orchestra.run.smithi181.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-03-09T14:34:35.295 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-03-09T14:34:35.295 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-argparse x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 45 k 2023-03-09T14:34:35.295 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-common x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 117 k 2023-03-09T14:34:35.295 INFO:teuthology.orchestra.run.smithi181.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-03-09T14:34:35.295 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-03-09T14:34:35.295 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-03-09T14:34:35.295 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-03-09T14:34:35.296 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-03-09T14:34:35.296 INFO:teuthology.orchestra.run.smithi181.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-03-09T14:34:35.296 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-03-09T14:34:35.296 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-03-09T14:34:35.296 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-03-09T14:34:35.296 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-03-09T14:34:35.296 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-03-09T14:34:35.297 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-03-09T14:34:35.297 INFO:teuthology.orchestra.run.smithi181.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-03-09T14:34:35.297 INFO:teuthology.orchestra.run.smithi181.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-03-09T14:34:35.297 INFO:teuthology.orchestra.run.smithi181.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-03-09T14:34:35.297 INFO:teuthology.orchestra.run.smithi181.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-03-09T14:34:35.297 INFO:teuthology.orchestra.run.smithi181.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-03-09T14:34:35.297 INFO:teuthology.orchestra.run.smithi181.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-03-09T14:34:35.297 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-03-09T14:34:35.298 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-03-09T14:34:35.298 INFO:teuthology.orchestra.run.smithi181.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-03-09T14:34:35.298 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-03-09T14:34:35.298 INFO:teuthology.orchestra.run.smithi181.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-03-09T14:34:35.298 INFO:teuthology.orchestra.run.smithi181.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-03-09T14:34:35.298 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-03-09T14:34:35.299 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-03-09T14:34:35.299 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-03-09T14:34:35.299 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-03-09T14:34:35.299 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-03-09T14:34:35.299 INFO:teuthology.orchestra.run.smithi181.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-03-09T14:34:35.299 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-03-09T14:34:35.300 INFO:teuthology.orchestra.run.smithi181.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-03-09T14:34:35.300 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-03-09T14:34:35.300 INFO:teuthology.orchestra.run.smithi181.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-03-09T14:34:35.300 INFO:teuthology.orchestra.run.smithi181.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-03-09T14:34:35.300 INFO:teuthology.orchestra.run.smithi181.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-03-09T14:34:35.300 INFO:teuthology.orchestra.run.smithi181.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-03-09T14:34:35.300 INFO:teuthology.orchestra.run.smithi181.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-03-09T14:34:35.300 INFO:teuthology.orchestra.run.smithi181.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2023-03-09T14:34:35.301 INFO:teuthology.orchestra.run.smithi181.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-03-09T14:34:35.301 INFO:teuthology.orchestra.run.smithi181.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-03-09T14:34:35.301 INFO:teuthology.orchestra.run.smithi181.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-03-09T14:34:35.301 INFO:teuthology.orchestra.run.smithi181.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-03-09T14:34:35.301 INFO:teuthology.orchestra.run.smithi181.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-03-09T14:34:35.301 INFO:teuthology.orchestra.run.smithi181.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-03-09T14:34:35.301 INFO:teuthology.orchestra.run.smithi181.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-03-09T14:34:35.302 INFO:teuthology.orchestra.run.smithi181.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-03-09T14:34:35.302 INFO:teuthology.orchestra.run.smithi181.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-03-09T14:34:35.302 INFO:teuthology.orchestra.run.smithi181.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-03-09T14:34:35.302 INFO:teuthology.orchestra.run.smithi181.stdout:Enabling module streams: 2023-03-09T14:34:35.302 INFO:teuthology.orchestra.run.smithi181.stdout: postgresql 10 2023-03-09T14:34:35.302 INFO:teuthology.orchestra.run.smithi181.stdout: 2023-03-09T14:34:35.302 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction Summary 2023-03-09T14:34:35.303 INFO:teuthology.orchestra.run.smithi181.stdout:=========================================================================================================================== 2023-03-09T14:34:35.303 INFO:teuthology.orchestra.run.smithi181.stdout:Install 128 Packages 2023-03-09T14:34:35.303 INFO:teuthology.orchestra.run.smithi181.stdout:Upgrade 3 Packages 2023-03-09T14:34:35.303 INFO:teuthology.orchestra.run.smithi181.stdout: 2023-03-09T14:34:35.304 INFO:teuthology.orchestra.run.smithi181.stdout:Total download size: 203 M 2023-03-09T14:34:35.304 INFO:teuthology.orchestra.run.smithi181.stdout:Downloading Packages: 2023-03-09T14:34:35.346 INFO:teuthology.orchestra.run.smithi122.stdout:(1/131): ceph-18.0.0-2872.ge0ab30a2.el8.x86_64. 53 kB/s | 6.4 kB 00:00 2023-03-09T14:34:35.426 INFO:teuthology.orchestra.run.smithi181.stdout:(1/131): ceph-18.0.0-2872.ge0ab30a2.el8.x86_64. 50 kB/s | 6.4 kB 00:00 2023-03-09T14:34:35.588 INFO:teuthology.orchestra.run.smithi122.stdout:(2/131): ceph-fuse-18.0.0-2872.ge0ab30a2.el8.x8 3.4 MB/s | 845 kB 00:00 2023-03-09T14:34:35.646 INFO:teuthology.orchestra.run.smithi122.stdout:(3/131): ceph-immutable-object-cache-18.0.0-287 2.8 MB/s | 166 kB 00:00 2023-03-09T14:34:35.885 INFO:teuthology.orchestra.run.smithi181.stdout:(2/131): ceph-fuse-18.0.0-2872.ge0ab30a2.el8.x8 1.8 MB/s | 845 kB 00:00 2023-03-09T14:34:36.035 INFO:teuthology.orchestra.run.smithi181.stdout:(3/131): ceph-immutable-object-cache-18.0.0-287 1.1 MB/s | 166 kB 00:00 2023-03-09T14:34:36.155 INFO:teuthology.orchestra.run.smithi122.stdout:(4/131): ceph-mds-18.0.0-2872.ge0ab30a2.el8.x86 4.5 MB/s | 2.3 MB 00:00 2023-03-09T14:34:36.230 INFO:teuthology.orchestra.run.smithi122.stdout:(5/131): ceph-base-18.0.0-2872.ge0ab30a2.el8.x8 5.3 MB/s | 5.4 MB 00:01 2023-03-09T14:34:36.469 INFO:teuthology.orchestra.run.smithi181.stdout:(4/131): ceph-base-18.0.0-2872.ge0ab30a2.el8.x8 4.6 MB/s | 5.4 MB 00:01 2023-03-09T14:34:36.556 INFO:teuthology.orchestra.run.smithi122.stdout:(6/131): ceph-mgr-18.0.0-2872.ge0ab30a2.el8.x86 3.9 MB/s | 1.5 MB 00:00 2023-03-09T14:34:36.783 INFO:teuthology.orchestra.run.smithi038.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-03-09T14:34:36.784 INFO:teuthology.orchestra.run.smithi038.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-03-09T14:34:36.799 INFO:teuthology.orchestra.run.smithi038.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-03-09T14:34:36.801 INFO:teuthology.orchestra.run.smithi038.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-03-09T14:34:36.801 INFO:teuthology.orchestra.run.smithi038.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-03-09T14:34:36.802 INFO:teuthology.orchestra.run.smithi038.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-03-09T14:34:36.805 INFO:teuthology.orchestra.run.smithi038.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-03-09T14:34:36.808 INFO:teuthology.orchestra.run.smithi038.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-03-09T14:34:36.809 INFO:teuthology.orchestra.run.smithi038.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-03-09T14:34:36.844 INFO:teuthology.orchestra.run.smithi181.stdout:(5/131): ceph-mgr-18.0.0-2872.ge0ab30a2.el8.x86 4.1 MB/s | 1.5 MB 00:00 2023-03-09T14:34:36.951 INFO:teuthology.orchestra.run.smithi038.stdout:Dependencies resolved. 2023-03-09T14:34:36.953 INFO:teuthology.orchestra.run.smithi181.stdout:(6/131): ceph-mds-18.0.0-2872.ge0ab30a2.el8.x86 2.5 MB/s | 2.3 MB 00:00 2023-03-09T14:34:36.964 INFO:teuthology.orchestra.run.smithi038.stdout:=========================================================================================================================== 2023-03-09T14:34:36.964 INFO:teuthology.orchestra.run.smithi038.stdout: Package Arch Version Repository Size 2023-03-09T14:34:36.964 INFO:teuthology.orchestra.run.smithi038.stdout:=========================================================================================================================== 2023-03-09T14:34:36.964 INFO:teuthology.orchestra.run.smithi038.stdout:Installing: 2023-03-09T14:34:36.965 INFO:teuthology.orchestra.run.smithi038.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-03-09T14:34:36.965 INFO:teuthology.orchestra.run.smithi038.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-03-09T14:34:36.965 INFO:teuthology.orchestra.run.smithi038.stdout: ceph x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 6.4 k 2023-03-09T14:34:36.965 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-base x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 5.4 M 2023-03-09T14:34:36.965 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-fuse x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 845 k 2023-03-09T14:34:36.965 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 166 k 2023-03-09T14:34:36.965 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 1.5 M 2023-03-09T14:34:36.966 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-cephadm noarch 2:18.0.0-2872.ge0ab30a2.el8 ceph-noarch 123 k 2023-03-09T14:34:36.966 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-dashboard noarch 2:18.0.0-2872.ge0ab30a2.el8 ceph-noarch 1.7 M 2023-03-09T14:34:36.966 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-2872.ge0ab30a2.el8 ceph-noarch 7.4 M 2023-03-09T14:34:36.966 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-rook noarch 2:18.0.0-2872.ge0ab30a2.el8 ceph-noarch 47 k 2023-03-09T14:34:36.967 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-radosgw x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 13 M 2023-03-09T14:34:36.967 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-test x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 47 M 2023-03-09T14:34:36.967 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-volume noarch 2:18.0.0-2872.ge0ab30a2.el8 ceph-noarch 263 k 2023-03-09T14:34:36.967 INFO:teuthology.orchestra.run.smithi038.stdout: cephadm noarch 2:18.0.0-2872.ge0ab30a2.el8 ceph-noarch 83 k 2023-03-09T14:34:36.967 INFO:teuthology.orchestra.run.smithi038.stdout: cephfs-mirror x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 257 k 2023-03-09T14:34:36.968 INFO:teuthology.orchestra.run.smithi038.stdout: cephfs-top noarch 2:18.0.0-2872.ge0ab30a2.el8 ceph-noarch 18 k 2023-03-09T14:34:36.968 INFO:teuthology.orchestra.run.smithi038.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-03-09T14:34:36.969 INFO:teuthology.orchestra.run.smithi038.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-03-09T14:34:36.969 INFO:teuthology.orchestra.run.smithi038.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-03-09T14:34:36.969 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs-devel x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 30 k 2023-03-09T14:34:36.969 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs2 x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 768 k 2023-03-09T14:34:36.969 INFO:teuthology.orchestra.run.smithi038.stdout: librados-devel x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 139 k 2023-03-09T14:34:36.969 INFO:teuthology.orchestra.run.smithi038.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-03-09T14:34:36.969 INFO:teuthology.orchestra.run.smithi038.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-03-09T14:34:36.970 INFO:teuthology.orchestra.run.smithi038.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-03-09T14:34:36.970 INFO:teuthology.orchestra.run.smithi038.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-03-09T14:34:36.970 INFO:teuthology.orchestra.run.smithi038.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-03-09T14:34:36.970 INFO:teuthology.orchestra.run.smithi038.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-03-09T14:34:36.970 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cephfs x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 190 k 2023-03-09T14:34:36.970 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rados x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 371 k 2023-03-09T14:34:36.970 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rbd x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 364 k 2023-03-09T14:34:36.971 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rgw x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 106 k 2023-03-09T14:34:36.971 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-fuse x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 94 k 2023-03-09T14:34:36.971 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-mirror x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 3.4 M 2023-03-09T14:34:36.971 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-nbd x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 180 k 2023-03-09T14:34:36.971 INFO:teuthology.orchestra.run.smithi038.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-03-09T14:34:36.971 INFO:teuthology.orchestra.run.smithi038.stdout:Upgrading: 2023-03-09T14:34:36.971 INFO:teuthology.orchestra.run.smithi038.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-03-09T14:34:36.972 INFO:teuthology.orchestra.run.smithi038.stdout: librados2 x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 3.7 M 2023-03-09T14:34:36.972 INFO:teuthology.orchestra.run.smithi038.stdout: librbd1 x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 3.6 M 2023-03-09T14:34:36.972 INFO:teuthology.orchestra.run.smithi038.stdout:Installing dependencies: 2023-03-09T14:34:36.972 INFO:teuthology.orchestra.run.smithi038.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-03-09T14:34:36.972 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-common x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 22 M 2023-03-09T14:34:36.972 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-grafana-dashboards noarch 2:18.0.0-2872.ge0ab30a2.el8 ceph-noarch 23 k 2023-03-09T14:34:36.972 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mds x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 2.3 M 2023-03-09T14:34:36.972 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-modules-core noarch 2:18.0.0-2872.ge0ab30a2.el8 ceph-noarch 239 k 2023-03-09T14:34:36.973 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mon x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 4.6 M 2023-03-09T14:34:36.973 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-osd x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 18 M 2023-03-09T14:34:36.973 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-prometheus-alerts noarch 2:18.0.0-2872.ge0ab30a2.el8 ceph-noarch 15 k 2023-03-09T14:34:36.973 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-selinux x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 24 k 2023-03-09T14:34:36.973 INFO:teuthology.orchestra.run.smithi038.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-03-09T14:34:36.973 INFO:teuthology.orchestra.run.smithi038.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-03-09T14:34:36.973 INFO:teuthology.orchestra.run.smithi038.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-03-09T14:34:36.973 INFO:teuthology.orchestra.run.smithi038.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-03-09T14:34:36.974 INFO:teuthology.orchestra.run.smithi038.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-03-09T14:34:36.974 INFO:teuthology.orchestra.run.smithi038.stdout: libcephsqlite x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 137 k 2023-03-09T14:34:36.974 INFO:teuthology.orchestra.run.smithi038.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-03-09T14:34:36.974 INFO:teuthology.orchestra.run.smithi038.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-03-09T14:34:36.974 INFO:teuthology.orchestra.run.smithi038.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-03-09T14:34:36.974 INFO:teuthology.orchestra.run.smithi038.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-03-09T14:34:36.974 INFO:teuthology.orchestra.run.smithi038.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-03-09T14:34:36.975 INFO:teuthology.orchestra.run.smithi038.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-03-09T14:34:36.975 INFO:teuthology.orchestra.run.smithi038.stdout: libradosstriper1 x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 474 k 2023-03-09T14:34:36.975 INFO:teuthology.orchestra.run.smithi038.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-03-09T14:34:36.975 INFO:teuthology.orchestra.run.smithi038.stdout: librgw2 x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 6.5 M 2023-03-09T14:34:36.975 INFO:teuthology.orchestra.run.smithi038.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-03-09T14:34:36.975 INFO:teuthology.orchestra.run.smithi038.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-03-09T14:34:36.975 INFO:teuthology.orchestra.run.smithi038.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-03-09T14:34:36.976 INFO:teuthology.orchestra.run.smithi038.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-03-09T14:34:36.976 INFO:teuthology.orchestra.run.smithi038.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-03-09T14:34:36.976 INFO:teuthology.orchestra.run.smithi038.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-03-09T14:34:36.976 INFO:teuthology.orchestra.run.smithi038.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-03-09T14:34:36.976 INFO:teuthology.orchestra.run.smithi038.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-03-09T14:34:36.976 INFO:teuthology.orchestra.run.smithi038.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-03-09T14:34:36.976 INFO:teuthology.orchestra.run.smithi038.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-03-09T14:34:36.976 INFO:teuthology.orchestra.run.smithi038.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-03-09T14:34:36.977 INFO:teuthology.orchestra.run.smithi038.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-03-09T14:34:36.977 INFO:teuthology.orchestra.run.smithi038.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-03-09T14:34:36.977 INFO:teuthology.orchestra.run.smithi038.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-03-09T14:34:36.977 INFO:teuthology.orchestra.run.smithi038.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-03-09T14:34:36.977 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-03-09T14:34:36.977 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-argparse x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 45 k 2023-03-09T14:34:36.977 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-common x86_64 2:18.0.0-2872.ge0ab30a2.el8 ceph 117 k 2023-03-09T14:34:36.978 INFO:teuthology.orchestra.run.smithi038.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-03-09T14:34:36.978 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-03-09T14:34:36.978 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-03-09T14:34:36.978 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-03-09T14:34:36.978 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-03-09T14:34:36.978 INFO:teuthology.orchestra.run.smithi038.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-03-09T14:34:36.979 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-03-09T14:34:36.979 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-03-09T14:34:36.979 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-03-09T14:34:36.979 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-03-09T14:34:36.979 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-03-09T14:34:36.979 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-03-09T14:34:36.979 INFO:teuthology.orchestra.run.smithi038.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-03-09T14:34:36.980 INFO:teuthology.orchestra.run.smithi038.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-03-09T14:34:36.980 INFO:teuthology.orchestra.run.smithi038.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-03-09T14:34:36.980 INFO:teuthology.orchestra.run.smithi038.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-03-09T14:34:36.980 INFO:teuthology.orchestra.run.smithi038.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-03-09T14:34:36.980 INFO:teuthology.orchestra.run.smithi038.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-03-09T14:34:36.980 INFO:teuthology.orchestra.run.smithi038.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-03-09T14:34:36.980 INFO:teuthology.orchestra.run.smithi038.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-03-09T14:34:36.981 INFO:teuthology.orchestra.run.smithi038.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-03-09T14:34:36.981 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-03-09T14:34:36.981 INFO:teuthology.orchestra.run.smithi038.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-03-09T14:34:36.981 INFO:teuthology.orchestra.run.smithi038.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-03-09T14:34:36.981 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-03-09T14:34:36.981 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-03-09T14:34:36.981 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-03-09T14:34:36.981 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-03-09T14:34:36.982 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-03-09T14:34:36.982 INFO:teuthology.orchestra.run.smithi038.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-03-09T14:34:36.982 INFO:teuthology.orchestra.run.smithi038.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-03-09T14:34:36.982 INFO:teuthology.orchestra.run.smithi038.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-03-09T14:34:36.982 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-03-09T14:34:36.982 INFO:teuthology.orchestra.run.smithi038.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-03-09T14:34:36.982 INFO:teuthology.orchestra.run.smithi038.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-03-09T14:34:36.983 INFO:teuthology.orchestra.run.smithi038.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-03-09T14:34:36.983 INFO:teuthology.orchestra.run.smithi038.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-03-09T14:34:36.983 INFO:teuthology.orchestra.run.smithi038.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-03-09T14:34:36.983 INFO:teuthology.orchestra.run.smithi038.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2023-03-09T14:34:36.983 INFO:teuthology.orchestra.run.smithi038.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-03-09T14:34:36.983 INFO:teuthology.orchestra.run.smithi038.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-03-09T14:34:36.983 INFO:teuthology.orchestra.run.smithi038.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-03-09T14:34:36.983 INFO:teuthology.orchestra.run.smithi038.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-03-09T14:34:36.984 INFO:teuthology.orchestra.run.smithi038.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-03-09T14:34:36.984 INFO:teuthology.orchestra.run.smithi038.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-03-09T14:34:36.984 INFO:teuthology.orchestra.run.smithi038.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-03-09T14:34:36.984 INFO:teuthology.orchestra.run.smithi038.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-03-09T14:34:36.984 INFO:teuthology.orchestra.run.smithi038.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-03-09T14:34:36.984 INFO:teuthology.orchestra.run.smithi038.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-03-09T14:34:36.984 INFO:teuthology.orchestra.run.smithi038.stdout:Enabling module streams: 2023-03-09T14:34:36.985 INFO:teuthology.orchestra.run.smithi038.stdout: postgresql 10 2023-03-09T14:34:36.985 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:34:36.985 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction Summary 2023-03-09T14:34:36.985 INFO:teuthology.orchestra.run.smithi038.stdout:=========================================================================================================================== 2023-03-09T14:34:36.985 INFO:teuthology.orchestra.run.smithi038.stdout:Install 128 Packages 2023-03-09T14:34:36.985 INFO:teuthology.orchestra.run.smithi038.stdout:Upgrade 3 Packages 2023-03-09T14:34:36.985 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:34:36.986 INFO:teuthology.orchestra.run.smithi038.stdout:Total download size: 203 M 2023-03-09T14:34:36.986 INFO:teuthology.orchestra.run.smithi038.stdout:Downloading Packages: 2023-03-09T14:34:37.023 INFO:teuthology.orchestra.run.smithi122.stdout:(7/131): ceph-mon-18.0.0-2872.ge0ab30a2.el8.x86 5.8 MB/s | 4.6 MB 00:00 2023-03-09T14:34:37.246 INFO:teuthology.orchestra.run.smithi038.stdout:(1/131): ceph-18.0.0-2872.ge0ab30a2.el8.x86_64. 24 kB/s | 6.4 kB 00:00 2023-03-09T14:34:37.903 INFO:teuthology.orchestra.run.smithi181.stdout:(7/131): ceph-mon-18.0.0-2872.ge0ab30a2.el8.x86 4.3 MB/s | 4.6 MB 00:01 2023-03-09T14:34:38.080 INFO:teuthology.orchestra.run.smithi038.stdout:(2/131): ceph-fuse-18.0.0-2872.ge0ab30a2.el8.x8 1.0 MB/s | 845 kB 00:00 2023-03-09T14:34:38.255 INFO:teuthology.orchestra.run.smithi038.stdout:(3/131): ceph-immutable-object-cache-18.0.0-287 946 kB/s | 166 kB 00:00 2023-03-09T14:34:39.323 INFO:teuthology.orchestra.run.smithi122.stdout:(8/131): ceph-common-18.0.0-2872.ge0ab30a2.el8. 5.3 MB/s | 22 MB 00:04 2023-03-09T14:34:39.465 INFO:teuthology.orchestra.run.smithi122.stdout:(9/131): ceph-radosgw-18.0.0-2872.ge0ab30a2.el8 5.4 MB/s | 13 MB 00:02 2023-03-09T14:34:39.490 INFO:teuthology.orchestra.run.smithi122.stdout:(10/131): ceph-selinux-18.0.0-2872.ge0ab30a2.el 145 kB/s | 24 kB 00:00 2023-03-09T14:34:39.648 INFO:teuthology.orchestra.run.smithi122.stdout:(11/131): cephfs-mirror-18.0.0-2872.ge0ab30a2.e 1.6 MB/s | 257 kB 00:00 2023-03-09T14:34:39.741 INFO:teuthology.orchestra.run.smithi122.stdout:(12/131): libcephfs-devel-18.0.0-2872.ge0ab30a2 332 kB/s | 30 kB 00:00 2023-03-09T14:34:39.756 INFO:teuthology.orchestra.run.smithi038.stdout:(4/131): ceph-mds-18.0.0-2872.ge0ab30a2.el8.x86 1.5 MB/s | 2.3 MB 00:01 2023-03-09T14:34:39.991 INFO:teuthology.orchestra.run.smithi122.stdout:(13/131): libcephfs2-18.0.0-2872.ge0ab30a2.el8. 3.0 MB/s | 768 kB 00:00 2023-03-09T14:34:40.049 INFO:teuthology.orchestra.run.smithi122.stdout:(14/131): libcephsqlite-18.0.0-2872.ge0ab30a2.e 2.3 MB/s | 137 kB 00:00 2023-03-09T14:34:40.133 INFO:teuthology.orchestra.run.smithi122.stdout:(15/131): librados-devel-18.0.0-2872.ge0ab30a2. 1.6 MB/s | 139 kB 00:00 2023-03-09T14:34:40.139 INFO:teuthology.orchestra.run.smithi038.stdout:(5/131): ceph-base-18.0.0-2872.ge0ab30a2.el8.x8 1.7 MB/s | 5.4 MB 00:03 2023-03-09T14:34:40.161 INFO:teuthology.orchestra.run.smithi181.stdout:(8/131): ceph-common-18.0.0-2872.ge0ab30a2.el8. 4.4 MB/s | 22 MB 00:04 2023-03-09T14:34:40.283 INFO:teuthology.orchestra.run.smithi122.stdout:(16/131): libradosstriper1-18.0.0-2872.ge0ab30a 3.1 MB/s | 474 kB 00:00 2023-03-09T14:34:40.377 INFO:teuthology.orchestra.run.smithi181.stdout:(9/131): ceph-radosgw-18.0.0-2872.ge0ab30a2.el8 5.3 MB/s | 13 MB 00:02 2023-03-09T14:34:40.395 INFO:teuthology.orchestra.run.smithi181.stdout:(10/131): ceph-selinux-18.0.0-2872.ge0ab30a2.el 104 kB/s | 24 kB 00:00 2023-03-09T14:34:40.503 INFO:teuthology.orchestra.run.smithi181.stdout:(11/131): cephfs-mirror-18.0.0-2872.ge0ab30a2.e 2.3 MB/s | 257 kB 00:00 2023-03-09T14:34:40.595 INFO:teuthology.orchestra.run.smithi181.stdout:(12/131): libcephfs-devel-18.0.0-2872.ge0ab30a2 333 kB/s | 30 kB 00:00 2023-03-09T14:34:40.673 INFO:teuthology.orchestra.run.smithi038.stdout:(6/131): ceph-mgr-18.0.0-2872.ge0ab30a2.el8.x86 1.7 MB/s | 1.5 MB 00:00 2023-03-09T14:34:40.862 INFO:teuthology.orchestra.run.smithi181.stdout:(13/131): libcephfs2-18.0.0-2872.ge0ab30a2.el8. 2.8 MB/s | 768 kB 00:00 2023-03-09T14:34:40.954 INFO:teuthology.orchestra.run.smithi181.stdout:(14/131): libcephsqlite-18.0.0-2872.ge0ab30a2.e 1.5 MB/s | 137 kB 00:00 2023-03-09T14:34:41.063 INFO:teuthology.orchestra.run.smithi181.stdout:(15/131): librados-devel-18.0.0-2872.ge0ab30a2. 1.3 MB/s | 139 kB 00:00 2023-03-09T14:34:41.296 INFO:teuthology.orchestra.run.smithi181.stdout:(16/131): libradosstriper1-18.0.0-2872.ge0ab30a 2.0 MB/s | 474 kB 00:00 2023-03-09T14:34:42.009 INFO:teuthology.orchestra.run.smithi122.stdout:(17/131): ceph-osd-18.0.0-2872.ge0ab30a2.el8.x8 3.2 MB/s | 18 MB 00:05 2023-03-09T14:34:42.092 INFO:teuthology.orchestra.run.smithi122.stdout:(18/131): librgw2-18.0.0-2872.ge0ab30a2.el8.x86 3.6 MB/s | 6.5 MB 00:01 2023-03-09T14:34:42.109 INFO:teuthology.orchestra.run.smithi122.stdout:(19/131): python3-ceph-argparse-18.0.0-2872.ge0 445 kB/s | 45 kB 00:00 2023-03-09T14:34:42.176 INFO:teuthology.orchestra.run.smithi122.stdout:(20/131): python3-ceph-common-18.0.0-2872.ge0ab 1.4 MB/s | 117 kB 00:00 2023-03-09T14:34:42.181 INFO:teuthology.orchestra.run.smithi038.stdout:(7/131): ceph-mon-18.0.0-2872.ge0ab30a2.el8.x86 2.3 MB/s | 4.6 MB 00:02 2023-03-09T14:34:42.243 INFO:teuthology.orchestra.run.smithi122.stdout:(21/131): python3-cephfs-18.0.0-2872.ge0ab30a2. 1.4 MB/s | 190 kB 00:00 2023-03-09T14:34:42.327 INFO:teuthology.orchestra.run.smithi122.stdout:(22/131): python3-rados-18.0.0-2872.ge0ab30a2.e 2.4 MB/s | 371 kB 00:00 2023-03-09T14:34:42.411 INFO:teuthology.orchestra.run.smithi122.stdout:(23/131): python3-rbd-18.0.0-2872.ge0ab30a2.el8 2.1 MB/s | 364 kB 00:00 2023-03-09T14:34:42.427 INFO:teuthology.orchestra.run.smithi122.stdout:(24/131): python3-rgw-18.0.0-2872.ge0ab30a2.el8 1.0 MB/s | 106 kB 00:00 2023-03-09T14:34:42.494 INFO:teuthology.orchestra.run.smithi122.stdout:(25/131): rbd-fuse-18.0.0-2872.ge0ab30a2.el8.x8 1.1 MB/s | 94 kB 00:00 2023-03-09T14:34:42.595 INFO:teuthology.orchestra.run.smithi122.stdout:(26/131): rbd-nbd-18.0.0-2872.ge0ab30a2.el8.x86 1.8 MB/s | 180 kB 00:00 2023-03-09T14:34:42.695 INFO:teuthology.orchestra.run.smithi122.stdout:(27/131): ceph-grafana-dashboards-18.0.0-2872.g 231 kB/s | 23 kB 00:00 2023-03-09T14:34:42.787 INFO:teuthology.orchestra.run.smithi122.stdout:(28/131): ceph-mgr-cephadm-18.0.0-2872.ge0ab30a 1.3 MB/s | 123 kB 00:00 2023-03-09T14:34:43.263 INFO:teuthology.orchestra.run.smithi181.stdout:(17/131): librgw2-18.0.0-2872.ge0ab30a2.el8.x86 3.3 MB/s | 6.5 MB 00:01 2023-03-09T14:34:43.356 INFO:teuthology.orchestra.run.smithi181.stdout:(18/131): python3-ceph-argparse-18.0.0-2872.ge0 488 kB/s | 45 kB 00:00 2023-03-09T14:34:43.404 INFO:teuthology.orchestra.run.smithi122.stdout:(29/131): rbd-mirror-18.0.0-2872.ge0ab30a2.el8. 3.5 MB/s | 3.4 MB 00:00 2023-03-09T14:34:43.431 INFO:teuthology.orchestra.run.smithi181.stdout:(19/131): python3-ceph-common-18.0.0-2872.ge0ab 1.5 MB/s | 117 kB 00:00 2023-03-09T14:34:43.480 INFO:teuthology.orchestra.run.smithi122.stdout:(30/131): ceph-mgr-dashboard-18.0.0-2872.ge0ab3 2.5 MB/s | 1.7 MB 00:00 2023-03-09T14:34:43.522 INFO:teuthology.orchestra.run.smithi181.stdout:(20/131): python3-cephfs-18.0.0-2872.ge0ab30a2. 2.0 MB/s | 190 kB 00:00 2023-03-09T14:34:43.647 INFO:teuthology.orchestra.run.smithi122.stdout:(31/131): ceph-mgr-modules-core-18.0.0-2872.ge0 1.4 MB/s | 239 kB 00:00 2023-03-09T14:34:43.689 INFO:teuthology.orchestra.run.smithi181.stdout:(21/131): python3-rados-18.0.0-2872.ge0ab30a2.e 2.2 MB/s | 371 kB 00:00 2023-03-09T14:34:43.755 INFO:teuthology.orchestra.run.smithi122.stdout:(32/131): ceph-mgr-rook-18.0.0-2872.ge0ab30a2.e 437 kB/s | 47 kB 00:00 2023-03-09T14:34:43.856 INFO:teuthology.orchestra.run.smithi181.stdout:(22/131): python3-rbd-18.0.0-2872.ge0ab30a2.el8 2.1 MB/s | 364 kB 00:00 2023-03-09T14:34:43.864 INFO:teuthology.orchestra.run.smithi122.stdout:(33/131): ceph-prometheus-alerts-18.0.0-2872.ge 134 kB/s | 15 kB 00:00 2023-03-09T14:34:43.948 INFO:teuthology.orchestra.run.smithi181.stdout:(23/131): python3-rgw-18.0.0-2872.ge0ab30a2.el8 1.1 MB/s | 106 kB 00:00 2023-03-09T14:34:44.031 INFO:teuthology.orchestra.run.smithi122.stdout:(34/131): ceph-volume-18.0.0-2872.ge0ab30a2.el8 1.5 MB/s | 263 kB 00:00 2023-03-09T14:34:44.057 INFO:teuthology.orchestra.run.smithi181.stdout:(24/131): rbd-fuse-18.0.0-2872.ge0ab30a2.el8.x8 869 kB/s | 94 kB 00:00 2023-03-09T14:34:44.123 INFO:teuthology.orchestra.run.smithi122.stdout:(35/131): cephadm-18.0.0-2872.ge0ab30a2.el8.noa 904 kB/s | 83 kB 00:00 2023-03-09T14:34:44.224 INFO:teuthology.orchestra.run.smithi122.stdout:(36/131): cephfs-top-18.0.0-2872.ge0ab30a2.el8. 181 kB/s | 18 kB 00:00 2023-03-09T14:34:44.532 INFO:teuthology.orchestra.run.smithi122.stdout:(37/131): gperftools-libs-2.7-9.el8.x86_64.rpm 993 kB/s | 306 kB 00:00 2023-03-09T14:34:44.591 INFO:teuthology.orchestra.run.smithi181.stdout:(25/131): ceph-osd-18.0.0-2872.ge0ab30a2.el8.x8 2.3 MB/s | 18 MB 00:07 2023-03-09T14:34:44.599 INFO:teuthology.orchestra.run.smithi122.stdout:(38/131): liboath-2.6.2-3.el8.x86_64.rpm 884 kB/s | 59 kB 00:00 2023-03-09T14:34:44.691 INFO:teuthology.orchestra.run.smithi181.stdout:(26/131): rbd-nbd-18.0.0-2872.ge0ab30a2.el8.x86 1.8 MB/s | 180 kB 00:00 2023-03-09T14:34:44.692 INFO:teuthology.orchestra.run.smithi122.stdout:(39/131): libunwind-1.3.1-3.el8.x86_64.rpm 818 kB/s | 75 kB 00:00 2023-03-09T14:34:44.791 INFO:teuthology.orchestra.run.smithi181.stdout:(27/131): ceph-grafana-dashboards-18.0.0-2872.g 231 kB/s | 23 kB 00:00 2023-03-09T14:34:44.908 INFO:teuthology.orchestra.run.smithi181.stdout:(28/131): ceph-mgr-cephadm-18.0.0-2872.ge0ab30a 1.0 MB/s | 123 kB 00:00 2023-03-09T14:34:44.916 INFO:teuthology.orchestra.run.smithi122.stdout:(40/131): python3-asyncssh-2.7.0-2.el8.noarch.r 2.1 MB/s | 476 kB 00:00 2023-03-09T14:34:44.967 INFO:teuthology.orchestra.run.smithi122.stdout:(41/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 875 kB/s | 44 kB 00:00 2023-03-09T14:34:45.067 INFO:teuthology.orchestra.run.smithi122.stdout:(42/131): python3-beautifulsoup4-4.6.3-2.el8.1. 1.8 MB/s | 185 kB 00:00 2023-03-09T14:34:45.125 INFO:teuthology.orchestra.run.smithi181.stdout:(29/131): rbd-mirror-18.0.0-2872.ge0ab30a2.el8. 3.2 MB/s | 3.4 MB 00:01 2023-03-09T14:34:45.134 INFO:teuthology.orchestra.run.smithi122.stdout:(43/131): python3-cachetools-3.1.1-4.el8.noarch 494 kB/s | 33 kB 00:00 2023-03-09T14:34:45.185 INFO:teuthology.orchestra.run.smithi122.stdout:(44/131): python3-certifi-2018.10.15-7.el8.noar 326 kB/s | 16 kB 00:00 2023-03-09T14:34:45.318 INFO:teuthology.orchestra.run.smithi122.stdout:(45/131): ceph-mgr-diskprediction-local-18.0.0- 3.9 MB/s | 7.4 MB 00:01 2023-03-09T14:34:45.360 INFO:teuthology.orchestra.run.smithi122.stdout:(46/131): python3-cheroot-8.5.2-1.el8.noarch.rp 983 kB/s | 173 kB 00:00 2023-03-09T14:34:45.427 INFO:teuthology.orchestra.run.smithi122.stdout:(47/131): python3-google-auth-1.1.1-10.el8.noar 1.2 MB/s | 81 kB 00:00 2023-03-09T14:34:45.469 INFO:teuthology.orchestra.run.smithi122.stdout:(48/131): python3-jaraco-6.2-6.el8.noarch.rpm 267 kB/s | 11 kB 00:00 2023-03-09T14:34:45.511 INFO:teuthology.orchestra.run.smithi122.stdout:(49/131): python3-jaraco-functools-2.0-4.el8.no 420 kB/s | 18 kB 00:00 2023-03-09T14:34:45.538 INFO:teuthology.orchestra.run.smithi181.stdout:(30/131): ceph-mgr-dashboard-18.0.0-2872.ge0ab3 2.8 MB/s | 1.7 MB 00:00 2023-03-09T14:34:45.539 INFO:teuthology.orchestra.run.smithi122.stdout:(50/131): python3-cherrypy-18.4.0-1.el8.noarch. 1.7 MB/s | 384 kB 00:00 2023-03-09T14:34:45.587 INFO:teuthology.orchestra.run.smithi122.stdout:(51/131): python3-logutils-0.3.5-11.el8.noarch. 975 kB/s | 49 kB 00:00 2023-03-09T14:34:45.637 INFO:teuthology.orchestra.run.smithi122.stdout:(52/131): python3-more-itertools-7.2.0-3.el8.no 1.1 MB/s | 59 kB 00:00 2023-03-09T14:34:45.684 INFO:teuthology.orchestra.run.smithi181.stdout:(31/131): ceph-mgr-modules-core-18.0.0-2872.ge0 1.6 MB/s | 239 kB 00:00 2023-03-09T14:34:45.751 INFO:teuthology.orchestra.run.smithi181.stdout:(32/131): ceph-mgr-rook-18.0.0-2872.ge0ab30a2.e 711 kB/s | 47 kB 00:00 2023-03-09T14:34:45.826 INFO:teuthology.orchestra.run.smithi181.stdout:(33/131): ceph-prometheus-alerts-18.0.0-2872.ge 194 kB/s | 15 kB 00:00 2023-03-09T14:34:45.943 INFO:teuthology.orchestra.run.smithi181.stdout:(34/131): ceph-volume-18.0.0-2872.ge0ab30a2.el8 2.2 MB/s | 263 kB 00:00 2023-03-09T14:34:46.010 INFO:teuthology.orchestra.run.smithi181.stdout:(35/131): cephadm-18.0.0-2872.ge0ab30a2.el8.noa 1.2 MB/s | 83 kB 00:00 2023-03-09T14:34:46.069 INFO:teuthology.orchestra.run.smithi181.stdout:(36/131): cephfs-top-18.0.0-2872.ge0ab30a2.el8. 309 kB/s | 18 kB 00:00 2023-03-09T14:34:46.096 INFO:teuthology.orchestra.run.smithi122.stdout:(53/131): ceph-test-18.0.0-2872.ge0ab30a2.el8.x 7.0 MB/s | 47 MB 00:06 2023-03-09T14:34:46.122 INFO:teuthology.orchestra.run.smithi122.stdout:(54/131): python3-natsort-7.1.1-2.el8.noarch.rp 124 kB/s | 60 kB 00:00 2023-03-09T14:34:46.172 INFO:teuthology.orchestra.run.smithi122.stdout:(55/131): python3-portend-2.6-1.el8.noarch.rpm 312 kB/s | 16 kB 00:00 2023-03-09T14:34:46.222 INFO:teuthology.orchestra.run.smithi122.stdout:(56/131): python3-kubernetes-11.0.0-6.el8.noarc 1.6 MB/s | 1.2 MB 00:00 2023-03-09T14:34:46.247 INFO:teuthology.orchestra.run.smithi122.stdout:(57/131): python3-repoze-lru-0.7-6.el8.noarch.r 447 kB/s | 34 kB 00:00 2023-03-09T14:34:46.261 INFO:teuthology.orchestra.run.smithi181.stdout:(37/131): gperftools-libs-2.7-9.el8.x86_64.rpm 1.6 MB/s | 306 kB 00:00 2023-03-09T14:34:46.298 INFO:teuthology.orchestra.run.smithi122.stdout:(58/131): python3-pecan-1.3.2-9.el8.noarch.rpm 1.4 MB/s | 283 kB 00:00 2023-03-09T14:34:46.303 INFO:teuthology.orchestra.run.smithi181.stdout:(38/131): liboath-2.6.2-3.el8.x86_64.rpm 1.4 MB/s | 59 kB 00:00 2023-03-09T14:34:46.323 INFO:teuthology.orchestra.run.smithi122.stdout:(59/131): python3-routes-2.4.1-12.el8.noarch.rp 1.9 MB/s | 196 kB 00:00 2023-03-09T14:34:46.348 INFO:teuthology.orchestra.run.smithi122.stdout:(60/131): python3-rsa-4.8-1.el8.noarch.rpm 612 kB/s | 61 kB 00:00 2023-03-09T14:34:46.353 INFO:teuthology.orchestra.run.smithi181.stdout:(39/131): libunwind-1.3.1-3.el8.x86_64.rpm 1.5 MB/s | 75 kB 00:00 2023-03-09T14:34:46.374 INFO:teuthology.orchestra.run.smithi122.stdout:(61/131): python3-simplegeneric-0.8.1-17.el8.no 248 kB/s | 19 kB 00:00 2023-03-09T14:34:46.399 INFO:teuthology.orchestra.run.smithi122.stdout:(62/131): python3-singledispatch-3.4.0.3-18.el8 478 kB/s | 24 kB 00:00 2023-03-09T14:34:46.424 INFO:teuthology.orchestra.run.smithi122.stdout:(63/131): python3-tempora-1.14.1-5.el8.noarch.r 388 kB/s | 29 kB 00:00 2023-03-09T14:34:46.450 INFO:teuthology.orchestra.run.smithi122.stdout:(64/131): python3-trustme-0.6.0-4.el8.noarch.rp 363 kB/s | 27 kB 00:00 2023-03-09T14:34:46.500 INFO:teuthology.orchestra.run.smithi122.stdout:(65/131): python3-waitress-1.2.1-2.el8.1.noarch 2.3 MB/s | 241 kB 00:00 2023-03-09T14:34:46.525 INFO:teuthology.orchestra.run.smithi122.stdout:(66/131): python3-websocket-client-0.56.0-5.el8 814 kB/s | 61 kB 00:00 2023-03-09T14:34:46.551 INFO:teuthology.orchestra.run.smithi122.stdout:(67/131): python3-webob-1.8.5-1.el8.1.noarch.rp 1.9 MB/s | 251 kB 00:00 2023-03-09T14:34:46.576 INFO:teuthology.orchestra.run.smithi122.stdout:(68/131): python3-webtest-2.0.33-1.el8.noarch.r 1.1 MB/s | 85 kB 00:00 2023-03-09T14:34:46.601 INFO:teuthology.orchestra.run.smithi122.stdout:(69/131): python3-zc-lockfile-2.0-2.el8.noarch. 304 kB/s | 23 kB 00:00 2023-03-09T14:34:46.606 INFO:teuthology.orchestra.run.smithi038.stdout:(8/131): ceph-radosgw-18.0.0-2872.ge0ab30a2.el8 3.0 MB/s | 13 MB 00:04 2023-03-09T14:34:46.626 INFO:teuthology.orchestra.run.smithi122.stdout:(70/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.4 MB/s | 70 kB 00:00 2023-03-09T14:34:46.646 INFO:teuthology.orchestra.run.smithi181.stdout:(40/131): python3-asyncssh-2.7.0-2.el8.noarch.r 1.6 MB/s | 476 kB 00:00 2023-03-09T14:34:46.656 INFO:teuthology.orchestra.run.smithi038.stdout:(9/131): ceph-selinux-18.0.0-2872.ge0ab30a2.el8 487 kB/s | 24 kB 00:00 2023-03-09T14:34:46.710 INFO:teuthology.orchestra.run.smithi122.stdout:(71/131): python3-pyasn1-modules-0.3.7-6.el8.no 1.4 MB/s | 110 kB 00:00 2023-03-09T14:34:46.737 INFO:teuthology.orchestra.run.smithi181.stdout:(41/131): ceph-mgr-diskprediction-local-18.0.0- 4.6 MB/s | 7.4 MB 00:01 2023-03-09T14:34:46.744 INFO:teuthology.orchestra.run.smithi122.stdout:(72/131): python3-pytz-2017.2-9.el8.noarch.rpm 1.6 MB/s | 54 kB 00:00 2023-03-09T14:34:46.762 INFO:teuthology.orchestra.run.smithi181.stdout:(42/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 376 kB/s | 44 kB 00:00 2023-03-09T14:34:46.786 INFO:teuthology.orchestra.run.smithi122.stdout:(73/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 6.1 MB/s | 259 kB 00:00 2023-03-09T14:34:46.821 INFO:teuthology.orchestra.run.smithi181.stdout:(43/131): python3-cachetools-3.1.1-4.el8.noarch 567 kB/s | 33 kB 00:00 2023-03-09T14:34:46.828 INFO:teuthology.orchestra.run.smithi122.stdout:(74/131): socat-1.7.4.1-1.el8.x86_64.rpm 7.6 MB/s | 323 kB 00:00 2023-03-09T14:34:46.861 INFO:teuthology.orchestra.run.smithi122.stdout:(75/131): python3-markupsafe-0.23-19.el8.x86_64 1.1 MB/s | 39 kB 00:00 2023-03-09T14:34:46.863 INFO:teuthology.orchestra.run.smithi181.stdout:(44/131): python3-certifi-2018.10.15-7.el8.noar 393 kB/s | 16 kB 00:00 2023-03-09T14:34:46.912 INFO:teuthology.orchestra.run.smithi122.stdout:(76/131): bison-3.0.4-10.el8.x86_64.rpm 13 MB/s | 688 kB 00:00 2023-03-09T14:34:46.913 INFO:teuthology.orchestra.run.smithi181.stdout:(45/131): python3-beautifulsoup4-4.6.3-2.el8.1. 1.0 MB/s | 185 kB 00:00 2023-03-09T14:34:46.915 INFO:teuthology.orchestra.run.smithi038.stdout:(10/131): ceph-common-18.0.0-2872.ge0ab30a2.el8 2.2 MB/s | 22 MB 00:09 2023-03-09T14:34:46.947 INFO:teuthology.orchestra.run.smithi181.stdout:(46/131): python3-cheroot-8.5.2-1.el8.noarch.rp 2.0 MB/s | 173 kB 00:00 2023-03-09T14:34:46.962 INFO:teuthology.orchestra.run.smithi122.stdout:(77/131): python3-jinja2-2.10.1-3.el8.noarch.rp 11 MB/s | 538 kB 00:00 2023-03-09T14:34:46.997 INFO:teuthology.orchestra.run.smithi181.stdout:(47/131): python3-cherrypy-18.4.0-1.el8.noarch. 4.5 MB/s | 384 kB 00:00 2023-03-09T14:34:47.004 INFO:teuthology.orchestra.run.smithi122.stdout:(78/131): thrift-0.13.0-2.el8.x86_64.rpm 3.8 MB/s | 1.7 MB 00:00 2023-03-09T14:34:47.022 INFO:teuthology.orchestra.run.smithi181.stdout:(48/131): python3-google-auth-1.1.1-10.el8.noar 1.1 MB/s | 81 kB 00:00 2023-03-09T14:34:47.023 INFO:teuthology.orchestra.run.smithi038.stdout:(11/131): cephfs-mirror-18.0.0-2872.ge0ab30a2.e 2.3 MB/s | 257 kB 00:00 2023-03-09T14:34:47.030 INFO:teuthology.orchestra.run.smithi122.stdout:(79/131): python3-mako-1.0.6-13.el8.noarch.rpm 2.3 MB/s | 157 kB 00:00 2023-03-09T14:34:47.048 INFO:teuthology.orchestra.run.smithi181.stdout:(49/131): python3-jaraco-6.2-6.el8.noarch.rpm 221 kB/s | 11 kB 00:00 2023-03-09T14:34:47.073 INFO:teuthology.orchestra.run.smithi181.stdout:(50/131): python3-jaraco-functools-2.0-4.el8.no 350 kB/s | 18 kB 00:00 2023-03-09T14:34:47.080 INFO:teuthology.orchestra.run.smithi122.stdout:(80/131): oniguruma-6.8.2-2.el8.x86_64.rpm 2.4 MB/s | 187 kB 00:00 2023-03-09T14:34:47.082 INFO:teuthology.orchestra.run.smithi038.stdout:(12/131): libcephfs-devel-18.0.0-2872.ge0ab30a2 523 kB/s | 30 kB 00:00 2023-03-09T14:34:47.123 INFO:teuthology.orchestra.run.smithi181.stdout:(51/131): python3-logutils-0.3.5-11.el8.noarch. 976 kB/s | 49 kB 00:00 2023-03-09T14:34:47.174 INFO:teuthology.orchestra.run.smithi181.stdout:(52/131): python3-more-itertools-7.2.0-3.el8.no 1.1 MB/s | 59 kB 00:00 2023-03-09T14:34:47.222 INFO:teuthology.orchestra.run.smithi122.stdout:(81/131): python3-scipy-1.0.0-21.module+el8.5.0 22 MB/s | 14 MB 00:00 2023-03-09T14:34:47.232 INFO:teuthology.orchestra.run.smithi181.stdout:(53/131): python3-kubernetes-11.0.0-6.el8.noarc 6.3 MB/s | 1.2 MB 00:00 2023-03-09T14:34:47.255 INFO:teuthology.orchestra.run.smithi122.stdout:(82/131): automake-1.16.1-7.el8.noarch.rpm 3.2 MB/s | 713 kB 00:00 2023-03-09T14:34:47.257 INFO:teuthology.orchestra.run.smithi181.stdout:(54/131): python3-natsort-7.1.1-2.el8.noarch.rp 722 kB/s | 60 kB 00:00 2023-03-09T14:34:47.281 INFO:teuthology.orchestra.run.smithi122.stdout:(83/131): python3-jsonpatch-1.21-2.el8.noarch.r 454 kB/s | 27 kB 00:00 2023-03-09T14:34:47.309 INFO:teuthology.orchestra.run.smithi181.stdout:(55/131): python3-portend-2.6-1.el8.noarch.rpm 312 kB/s | 16 kB 00:00 2023-03-09T14:34:47.333 INFO:teuthology.orchestra.run.smithi181.stdout:(56/131): python3-pecan-1.3.2-9.el8.noarch.rpm 2.8 MB/s | 283 kB 00:00 2023-03-09T14:34:47.335 INFO:teuthology.orchestra.run.smithi122.stdout:(84/131): flex-2.6.1-9.el8.x86_64.rpm 3.9 MB/s | 320 kB 00:00 2023-03-09T14:34:47.358 INFO:teuthology.orchestra.run.smithi181.stdout:(57/131): python3-repoze-lru-0.7-6.el8.noarch.r 672 kB/s | 34 kB 00:00 2023-03-09T14:34:47.365 INFO:teuthology.orchestra.run.smithi122.stdout:(85/131): python3-prettytable-0.7.2-14.el8.noar 524 kB/s | 44 kB 00:00 2023-03-09T14:34:47.384 INFO:teuthology.orchestra.run.smithi181.stdout:(58/131): python3-routes-2.4.1-12.el8.noarch.rp 3.8 MB/s | 196 kB 00:00 2023-03-09T14:34:47.409 INFO:teuthology.orchestra.run.smithi181.stdout:(59/131): python3-rsa-4.8-1.el8.noarch.rpm 1.2 MB/s | 61 kB 00:00 2023-03-09T14:34:47.415 INFO:teuthology.orchestra.run.smithi122.stdout:(86/131): python3-werkzeug-0.12.2-4.el8.noarch. 9.0 MB/s | 457 kB 00:00 2023-03-09T14:34:47.434 INFO:teuthology.orchestra.run.smithi181.stdout:(60/131): python3-simplegeneric-0.8.1-17.el8.no 372 kB/s | 19 kB 00:00 2023-03-09T14:34:47.457 INFO:teuthology.orchestra.run.smithi122.stdout:(87/131): python3-numpy-f2py-1.14.3-10.el8.x86_ 5.3 MB/s | 225 kB 00:00 2023-03-09T14:34:47.460 INFO:teuthology.orchestra.run.smithi181.stdout:(61/131): python3-singledispatch-3.4.0.3-18.el8 477 kB/s | 24 kB 00:00 2023-03-09T14:34:47.485 INFO:teuthology.orchestra.run.smithi181.stdout:(62/131): python3-tempora-1.14.1-5.el8.noarch.r 584 kB/s | 29 kB 00:00 2023-03-09T14:34:47.510 INFO:teuthology.orchestra.run.smithi181.stdout:(63/131): python3-trustme-0.6.0-4.el8.noarch.rp 548 kB/s | 27 kB 00:00 2023-03-09T14:34:47.525 INFO:teuthology.orchestra.run.smithi122.stdout:(88/131): python3-numpy-1.14.3-10.el8.x86_64.rp 20 MB/s | 3.7 MB 00:00 2023-03-09T14:34:47.535 INFO:teuthology.orchestra.run.smithi181.stdout:(64/131): python3-waitress-1.2.1-2.el8.1.noarch 4.7 MB/s | 241 kB 00:00 2023-03-09T14:34:47.549 INFO:teuthology.orchestra.run.smithi122.stdout:(89/131): uuid-1.6.2-43.el8.x86_64.rpm 695 kB/s | 64 kB 00:00 2023-03-09T14:34:47.575 INFO:teuthology.orchestra.run.smithi122.stdout:(90/131): python3-pyasn1-0.3.7-6.el8.noarch.rpm 2.5 MB/s | 126 kB 00:00 2023-03-09T14:34:47.586 INFO:teuthology.orchestra.run.smithi181.stdout:(65/131): python3-websocket-client-0.56.0-5.el8 1.2 MB/s | 61 kB 00:00 2023-03-09T14:34:47.600 INFO:teuthology.orchestra.run.smithi122.stdout:(91/131): python3-jsonpointer-1.10-11.el8.noarc 397 kB/s | 20 kB 00:00 2023-03-09T14:34:47.608 INFO:teuthology.orchestra.run.smithi038.stdout:(13/131): ceph-osd-18.0.0-2872.ge0ab30a2.el8.x8 2.5 MB/s | 18 MB 00:06 2023-03-09T14:34:47.611 INFO:teuthology.orchestra.run.smithi181.stdout:(66/131): python3-webob-1.8.5-1.el8.1.noarch.rp 2.4 MB/s | 251 kB 00:00 2023-03-09T14:34:47.636 INFO:teuthology.orchestra.run.smithi181.stdout:(67/131): python3-webtest-2.0.33-1.el8.noarch.r 1.7 MB/s | 85 kB 00:00 2023-03-09T14:34:47.650 INFO:teuthology.orchestra.run.smithi038.stdout:(14/131): libcephfs2-18.0.0-2872.ge0ab30a2.el8. 1.3 MB/s | 768 kB 00:00 2023-03-09T14:34:47.661 INFO:teuthology.orchestra.run.smithi181.stdout:(68/131): python3-zc-lockfile-2.0-2.el8.noarch. 458 kB/s | 23 kB 00:00 2023-03-09T14:34:47.675 INFO:teuthology.orchestra.run.smithi038.stdout:(15/131): libcephsqlite-18.0.0-2872.ge0ab30a2.e 2.0 MB/s | 137 kB 00:00 2023-03-09T14:34:47.675 INFO:teuthology.orchestra.run.smithi122.stdout:(92/131): python3-babel-2.5.1-7.el8.noarch.rpm 8.0 MB/s | 4.8 MB 00:00 2023-03-09T14:34:47.701 INFO:teuthology.orchestra.run.smithi122.stdout:(93/131): python3-pyOpenSSL-19.0.0-1.el8.noarch 823 kB/s | 103 kB 00:00 2023-03-09T14:34:47.717 INFO:teuthology.orchestra.run.smithi038.stdout:(16/131): librados-devel-18.0.0-2872.ge0ab30a2. 2.0 MB/s | 139 kB 00:00 2023-03-09T14:34:47.743 INFO:teuthology.orchestra.run.smithi122.stdout:(94/131): libtool-2.4.6-25.el8.x86_64.rpm 4.9 MB/s | 709 kB 00:00 2023-03-09T14:34:47.768 INFO:teuthology.orchestra.run.smithi122.stdout:(95/131): libpq-13.5-1.el8.x86_64.rpm 2.9 MB/s | 198 kB 00:00 2023-03-09T14:34:47.792 INFO:teuthology.orchestra.run.smithi038.stdout:(17/131): libradosstriper1-18.0.0-2872.ge0ab30a 4.0 MB/s | 474 kB 00:00 2023-03-09T14:34:47.810 INFO:teuthology.orchestra.run.smithi122.stdout:(96/131): jq-1.6-3.el8.x86_64.rpm 4.8 MB/s | 202 kB 00:00 2023-03-09T14:34:47.843 INFO:teuthology.orchestra.run.smithi038.stdout:(18/131): python3-ceph-argparse-18.0.0-2872.ge0 893 kB/s | 45 kB 00:00 2023-03-09T14:34:47.852 INFO:teuthology.orchestra.run.smithi122.stdout:(97/131): librdkafka-0.11.4-3.el8.x86_64.rpm 8.3 MB/s | 354 kB 00:00 2023-03-09T14:34:47.893 INFO:teuthology.orchestra.run.smithi038.stdout:(19/131): python3-ceph-common-18.0.0-2872.ge0ab 2.3 MB/s | 117 kB 00:00 2023-03-09T14:34:47.928 INFO:teuthology.orchestra.run.smithi122.stdout:(98/131): openblas-threads-0.3.15-3.el8.x86_64. 19 MB/s | 4.9 MB 00:00 2023-03-09T14:34:47.951 INFO:teuthology.orchestra.run.smithi038.stdout:(20/131): python3-cephfs-18.0.0-2872.ge0ab30a2. 3.2 MB/s | 190 kB 00:00 2023-03-09T14:34:48.027 INFO:teuthology.orchestra.run.smithi038.stdout:(21/131): python3-rados-18.0.0-2872.ge0ab30a2.e 4.8 MB/s | 371 kB 00:00 2023-03-09T14:34:48.028 INFO:teuthology.orchestra.run.smithi122.stdout:(99/131): openblas-0.3.15-3.el8.x86_64.rpm 17 MB/s | 4.7 MB 00:00 2023-03-09T14:34:48.061 INFO:teuthology.orchestra.run.smithi122.stdout:(100/131): postgresql-contrib-10.21-2.module+el 5.9 MB/s | 808 kB 00:00 2023-03-09T14:34:48.095 INFO:teuthology.orchestra.run.smithi122.stdout:(101/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 1.4 MB/s | 47 kB 00:00 2023-03-09T14:34:48.102 INFO:teuthology.orchestra.run.smithi038.stdout:(22/131): python3-rbd-18.0.0-2872.ge0ab30a2.el8 4.8 MB/s | 364 kB 00:00 2023-03-09T14:34:48.112 INFO:teuthology.orchestra.run.smithi181.stdout:(69/131): ceph-test-18.0.0-2872.ge0ab30a2.el8.x 6.0 MB/s | 47 MB 00:07 2023-03-09T14:34:48.137 INFO:teuthology.orchestra.run.smithi122.stdout:(102/131): postgresql-10.21-2.module+el8.6.0+15 14 MB/s | 1.5 MB 00:00 2023-03-09T14:34:48.138 INFO:teuthology.orchestra.run.smithi181.stdout:(70/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 146 kB/s | 70 kB 00:00 2023-03-09T14:34:48.152 INFO:teuthology.orchestra.run.smithi038.stdout:(23/131): python3-rgw-18.0.0-2872.ge0ab30a2.el8 2.1 MB/s | 106 kB 00:00 2023-03-09T14:34:48.162 INFO:teuthology.orchestra.run.smithi122.stdout:(103/131): libselinux-devel-2.9-5.el8.x86_64.rp 2.9 MB/s | 200 kB 00:00 2023-03-09T14:34:48.179 INFO:teuthology.orchestra.run.smithi122.stdout:(104/131): python3-jwt-1.6.1-2.el8.noarch.rpm 1.0 MB/s | 43 kB 00:00 2023-03-09T14:34:48.180 INFO:teuthology.orchestra.run.smithi181.stdout:(71/131): thrift-0.13.0-2.el8.x86_64.rpm 3.2 MB/s | 1.7 MB 00:00 2023-03-09T14:34:48.203 INFO:teuthology.orchestra.run.smithi038.stdout:(24/131): rbd-fuse-18.0.0-2872.ge0ab30a2.el8.x8 1.9 MB/s | 94 kB 00:00 2023-03-09T14:34:48.221 INFO:teuthology.orchestra.run.smithi181.stdout:(72/131): python3-pyasn1-modules-0.3.7-6.el8.no 1.4 MB/s | 110 kB 00:00 2023-03-09T14:34:48.246 INFO:teuthology.orchestra.run.smithi181.stdout:(73/131): python3-pytz-2017.2-9.el8.noarch.rpm 884 kB/s | 54 kB 00:00 2023-03-09T14:34:48.272 INFO:teuthology.orchestra.run.smithi181.stdout:(74/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 5.1 MB/s | 259 kB 00:00 2023-03-09T14:34:48.272 INFO:teuthology.orchestra.run.smithi122.stdout:(105/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 2.0 MB/s | 220 kB 00:00 2023-03-09T14:34:48.305 INFO:teuthology.orchestra.run.smithi181.stdout:(75/131): socat-1.7.4.1-1.el8.x86_64.rpm 5.4 MB/s | 323 kB 00:00 2023-03-09T14:34:48.330 INFO:teuthology.orchestra.run.smithi181.stdout:(76/131): python3-markupsafe-0.23-19.el8.x86_64 662 kB/s | 39 kB 00:00 2023-03-09T14:34:48.364 INFO:teuthology.orchestra.run.smithi181.stdout:(77/131): bison-3.0.4-10.el8.x86_64.rpm 11 MB/s | 688 kB 00:00 2023-03-09T14:34:48.398 INFO:teuthology.orchestra.run.smithi181.stdout:(78/131): python3-jinja2-2.10.1-3.el8.noarch.rp 7.9 MB/s | 538 kB 00:00 2023-03-09T14:34:48.423 INFO:teuthology.orchestra.run.smithi181.stdout:(79/131): python3-mako-1.0.6-13.el8.noarch.rpm 2.6 MB/s | 157 kB 00:00 2023-03-09T14:34:48.449 INFO:teuthology.orchestra.run.smithi181.stdout:(80/131): oniguruma-6.8.2-2.el8.x86_64.rpm 3.6 MB/s | 187 kB 00:00 2023-03-09T14:34:48.480 INFO:teuthology.orchestra.run.smithi122.stdout:(106/131): keyutils-libs-devel-1.5.10-9.el8.x86 160 kB/s | 48 kB 00:00 2023-03-09T14:34:48.482 INFO:teuthology.orchestra.run.smithi181.stdout:(81/131): automake-1.16.1-7.el8.noarch.rpm 12 MB/s | 713 kB 00:00 2023-03-09T14:34:48.506 INFO:teuthology.orchestra.run.smithi122.stdout:(107/131): libsepol-devel-2.9-3.el8.x86_64.rpm 372 kB/s | 87 kB 00:00 2023-03-09T14:34:48.524 INFO:teuthology.orchestra.run.smithi181.stdout:(82/131): python3-jsonpatch-1.21-2.el8.noarch.r 637 kB/s | 27 kB 00:00 2023-03-09T14:34:48.531 INFO:teuthology.orchestra.run.smithi122.stdout:(108/131): python3-requests-oauthlib-1.0.0-1.el 850 kB/s | 43 kB 00:00 2023-03-09T14:34:48.557 INFO:teuthology.orchestra.run.smithi122.stdout:(109/131): python3-pycparser-2.14-14.el8.noarch 2.1 MB/s | 109 kB 00:00 2023-03-09T14:34:48.566 INFO:teuthology.orchestra.run.smithi181.stdout:(83/131): flex-2.6.1-9.el8.x86_64.rpm 7.5 MB/s | 320 kB 00:00 2023-03-09T14:34:48.582 INFO:teuthology.orchestra.run.smithi122.stdout:(110/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 4.4 MB/s | 229 kB 00:00 2023-03-09T14:34:48.608 INFO:teuthology.orchestra.run.smithi181.stdout:(84/131): python3-prettytable-0.7.2-14.el8.noar 1.0 MB/s | 44 kB 00:00 2023-03-09T14:34:48.641 INFO:teuthology.orchestra.run.smithi122.stdout:(111/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 6.5 MB/s | 560 kB 00:00 2023-03-09T14:34:48.653 INFO:teuthology.orchestra.run.smithi038.stdout:(25/131): librgw2-18.0.0-2872.ge0ab30a2.el8.x86 7.0 MB/s | 6.5 MB 00:00 2023-03-09T14:34:48.666 INFO:teuthology.orchestra.run.smithi122.stdout:(112/131): libaio-devel-0.3.112-1.el8.x86_64.rp 223 kB/s | 19 kB 00:00 2023-03-09T14:34:48.691 INFO:teuthology.orchestra.run.smithi122.stdout:(113/131): python3-oauthlib-2.1.0-1.el8.noarch. 3.0 MB/s | 155 kB 00:00 2023-03-09T14:34:48.692 INFO:teuthology.orchestra.run.smithi181.stdout:(85/131): python3-babel-2.5.1-7.el8.noarch.rpm 20 MB/s | 4.8 MB 00:00 2023-03-09T14:34:48.720 INFO:teuthology.orchestra.run.smithi038.stdout:(26/131): rbd-mirror-18.0.0-2872.ge0ab30a2.el8. 6.7 MB/s | 3.4 MB 00:00 2023-03-09T14:34:48.733 INFO:teuthology.orchestra.run.smithi181.stdout:(86/131): python3-werkzeug-0.12.2-4.el8.noarch. 11 MB/s | 457 kB 00:00 2023-03-09T14:34:48.745 INFO:teuthology.orchestra.run.smithi038.stdout:(27/131): rbd-nbd-18.0.0-2872.ge0ab30a2.el8.x86 1.9 MB/s | 180 kB 00:00 2023-03-09T14:34:48.758 INFO:teuthology.orchestra.run.smithi122.stdout:(114/131): postgresql-server-10.21-2.module+el8 5.6 MB/s | 5.1 MB 00:00 2023-03-09T14:34:48.770 INFO:teuthology.orchestra.run.smithi038.stdout:(28/131): ceph-grafana-dashboards-18.0.0-2872.g 459 kB/s | 23 kB 00:00 2023-03-09T14:34:48.775 INFO:teuthology.orchestra.run.smithi181.stdout:(87/131): python3-numpy-f2py-1.14.3-10.el8.x86_ 5.3 MB/s | 225 kB 00:00 2023-03-09T14:34:48.783 INFO:teuthology.orchestra.run.smithi122.stdout:(115/131): libverto-devel-0.3.0-5.el8.x86_64.rp 195 kB/s | 18 kB 00:00 2023-03-09T14:34:48.788 INFO:teuthology.orchestra.run.smithi038.stdout:(29/131): ceph-mgr-cephadm-18.0.0-2872.ge0ab30a 2.9 MB/s | 123 kB 00:00 2023-03-09T14:34:48.817 INFO:teuthology.orchestra.run.smithi122.stdout:(116/131): python3-cryptography-3.2.1-5.el8.x86 3.6 MB/s | 559 kB 00:00 2023-03-09T14:34:48.843 INFO:teuthology.orchestra.run.smithi122.stdout:(117/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 4.0 MB/s | 238 kB 00:00 2023-03-09T14:34:48.851 INFO:teuthology.orchestra.run.smithi181.stdout:(88/131): python3-numpy-1.14.3-10.el8.x86_64.rp 15 MB/s | 3.7 MB 00:00 2023-03-09T14:34:48.867 INFO:teuthology.orchestra.run.smithi181.stdout:(89/131): uuid-1.6.2-43.el8.x86_64.rpm 696 kB/s | 64 kB 00:00 2023-03-09T14:34:48.876 INFO:teuthology.orchestra.run.smithi122.stdout:(118/131): pcre2-devel-10.32-2.el8.x86_64.rpm 5.0 MB/s | 605 kB 00:00 2023-03-09T14:34:48.893 INFO:teuthology.orchestra.run.smithi181.stdout:(90/131): python3-pyasn1-0.3.7-6.el8.noarch.rpm 3.0 MB/s | 126 kB 00:00 2023-03-09T14:34:48.901 INFO:teuthology.orchestra.run.smithi122.stdout:(119/131): xfsprogs-devel-5.0.0-10.el8.x86_64.r 1.2 MB/s | 72 kB 00:00 2023-03-09T14:34:48.909 INFO:teuthology.orchestra.run.smithi181.stdout:(91/131): python3-jsonpointer-1.10-11.el8.noarc 476 kB/s | 20 kB 00:00 2023-03-09T14:34:48.927 INFO:teuthology.orchestra.run.smithi122.stdout:(120/131): fuse-2.9.7-15.el8.x86_64.rpm 1.6 MB/s | 83 kB 00:00 2023-03-09T14:34:48.935 INFO:teuthology.orchestra.run.smithi181.stdout:(92/131): python3-pyOpenSSL-19.0.0-1.el8.noarch 2.4 MB/s | 103 kB 00:00 2023-03-09T14:34:48.952 INFO:teuthology.orchestra.run.smithi122.stdout:(121/131): elfutils-libelf-devel-0.186-1.el8.x8 1.2 MB/s | 60 kB 00:00 2023-03-09T14:34:48.977 INFO:teuthology.orchestra.run.smithi122.stdout:(122/131): libuuid-devel-2.32.1-35.el8.x86_64.r 1.9 MB/s | 98 kB 00:00 2023-03-09T14:34:49.003 INFO:teuthology.orchestra.run.smithi122.stdout:(123/131): zlib-devel-1.2.11-18.el8_5.x86_64.rp 1.1 MB/s | 58 kB 00:00 2023-03-09T14:34:49.013 INFO:teuthology.orchestra.run.smithi038.stdout:(30/131): ceph-mgr-dashboard-18.0.0-2872.ge0ab3 7.1 MB/s | 1.7 MB 00:00 2023-03-09T14:34:49.028 INFO:teuthology.orchestra.run.smithi122.stdout:(124/131): libcom_err-devel-1.45.6-4.el8.x86_64 769 kB/s | 39 kB 00:00 2023-03-09T14:34:49.062 INFO:teuthology.orchestra.run.smithi122.stdout:(125/131): libgfortran-8.5.0-10.1.el8_6.x86_64. 11 MB/s | 644 kB 00:00 2023-03-09T14:34:49.077 INFO:teuthology.orchestra.run.smithi181.stdout:(93/131): python3-scipy-1.0.0-21.module+el8.5.0 14 MB/s | 14 MB 00:00 2023-03-09T14:34:49.087 INFO:teuthology.orchestra.run.smithi122.stdout:(126/131): libquadmath-8.5.0-10.1.el8_6.x86_64. 2.9 MB/s | 171 kB 00:00 2023-03-09T14:34:49.097 INFO:teuthology.orchestra.run.smithi038.stdout:(31/131): ceph-mgr-modules-core-18.0.0-2872.ge0 2.8 MB/s | 239 kB 00:00 2023-03-09T14:34:49.110 INFO:teuthology.orchestra.run.smithi181.stdout:(94/131): libtool-2.4.6-25.el8.x86_64.rpm 3.5 MB/s | 709 kB 00:00 2023-03-09T14:34:49.112 INFO:teuthology.orchestra.run.smithi122.stdout:(127/131): NetworkManager-initscripts-updown-1. 2.7 MB/s | 138 kB 00:00 2023-03-09T14:34:49.138 INFO:teuthology.orchestra.run.smithi038.stdout:(32/131): ceph-mgr-rook-18.0.0-2872.ge0ab30a2.e 1.1 MB/s | 47 kB 00:00 2023-03-09T14:34:49.163 INFO:teuthology.orchestra.run.smithi122.stdout:(128/131): openssl-devel-1.1.1k-7.el8_6.x86_64. 6.7 MB/s | 2.3 MB 00:00 2023-03-09T14:34:49.172 INFO:teuthology.orchestra.run.smithi038.stdout:(33/131): ceph-prometheus-alerts-18.0.0-2872.ge 436 kB/s | 15 kB 00:00 2023-03-09T14:34:49.177 INFO:teuthology.orchestra.run.smithi181.stdout:(95/131): libpq-13.5-1.el8.x86_64.rpm 1.9 MB/s | 198 kB 00:00 2023-03-09T14:34:49.213 INFO:teuthology.orchestra.run.smithi122.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 16 MB/s | 798 kB 00:00 2023-03-09T14:34:49.219 INFO:teuthology.orchestra.run.smithi181.stdout:(96/131): jq-1.6-3.el8.x86_64.rpm 4.8 MB/s | 202 kB 00:00 2023-03-09T14:34:49.239 INFO:teuthology.orchestra.run.smithi038.stdout:(34/131): ceph-volume-18.0.0-2872.ge0ab30a2.el8 3.9 MB/s | 263 kB 00:00 2023-03-09T14:34:49.261 INFO:teuthology.orchestra.run.smithi181.stdout:(97/131): librdkafka-0.11.4-3.el8.x86_64.rpm 8.3 MB/s | 354 kB 00:00 2023-03-09T14:34:49.306 INFO:teuthology.orchestra.run.smithi038.stdout:(35/131): cephadm-18.0.0-2872.ge0ab30a2.el8.noa 1.2 MB/s | 83 kB 00:00 2023-03-09T14:34:49.353 INFO:teuthology.orchestra.run.smithi181.stdout:(98/131): openblas-threads-0.3.15-3.el8.x86_64. 12 MB/s | 4.9 MB 00:00 2023-03-09T14:34:49.365 INFO:teuthology.orchestra.run.smithi038.stdout:(36/131): cephfs-top-18.0.0-2872.ge0ab30a2.el8. 309 kB/s | 18 kB 00:00 2023-03-09T14:34:49.420 INFO:teuthology.orchestra.run.smithi181.stdout:(99/131): openblas-0.3.15-3.el8.x86_64.rpm 15 MB/s | 4.7 MB 00:00 2023-03-09T14:34:49.470 INFO:teuthology.orchestra.run.smithi181.stdout:(100/131): postgresql-contrib-10.21-2.module+el 6.8 MB/s | 808 kB 00:00 2023-03-09T14:34:49.504 INFO:teuthology.orchestra.run.smithi181.stdout:(101/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 1.4 MB/s | 47 kB 00:00 2023-03-09T14:34:49.546 INFO:teuthology.orchestra.run.smithi181.stdout:(102/131): postgresql-10.21-2.module+el8.6.0+15 12 MB/s | 1.5 MB 00:00 2023-03-09T14:34:49.557 INFO:teuthology.orchestra.run.smithi038.stdout:(37/131): gperftools-libs-2.7-9.el8.x86_64.rpm 1.6 MB/s | 306 kB 00:00 2023-03-09T14:34:49.571 INFO:teuthology.orchestra.run.smithi181.stdout:(103/131): libselinux-devel-2.9-5.el8.x86_64.rp 2.9 MB/s | 200 kB 00:00 2023-03-09T14:34:49.572 INFO:teuthology.orchestra.run.smithi122.stdout:(130/131): librbd1-18.0.0-2872.ge0ab30a2.el8.x8 7.8 MB/s | 3.6 MB 00:00 2023-03-09T14:34:49.596 INFO:teuthology.orchestra.run.smithi181.stdout:(104/131): python3-jwt-1.6.1-2.el8.noarch.rpm 847 kB/s | 43 kB 00:00 2023-03-09T14:34:49.607 INFO:teuthology.orchestra.run.smithi038.stdout:(38/131): liboath-2.6.2-3.el8.x86_64.rpm 1.2 MB/s | 59 kB 00:00 2023-03-09T14:34:49.622 INFO:teuthology.orchestra.run.smithi181.stdout:(105/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 4.3 MB/s | 220 kB 00:00 2023-03-09T14:34:49.647 INFO:teuthology.orchestra.run.smithi181.stdout:(106/131): keyutils-libs-devel-1.5.10-9.el8.x86 956 kB/s | 48 kB 00:00 2023-03-09T14:34:49.657 INFO:teuthology.orchestra.run.smithi038.stdout:(39/131): libunwind-1.3.1-3.el8.x86_64.rpm 1.5 MB/s | 75 kB 00:00 2023-03-09T14:34:49.672 INFO:teuthology.orchestra.run.smithi181.stdout:(107/131): libsepol-devel-2.9-3.el8.x86_64.rpm 1.7 MB/s | 87 kB 00:00 2023-03-09T14:34:49.697 INFO:teuthology.orchestra.run.smithi181.stdout:(108/131): python3-requests-oauthlib-1.0.0-1.el 848 kB/s | 43 kB 00:00 2023-03-09T14:34:49.723 INFO:teuthology.orchestra.run.smithi181.stdout:(109/131): python3-pycparser-2.14-14.el8.noarch 2.1 MB/s | 109 kB 00:00 2023-03-09T14:34:49.748 INFO:teuthology.orchestra.run.smithi181.stdout:(110/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 4.4 MB/s | 229 kB 00:00 2023-03-09T14:34:49.766 INFO:teuthology.orchestra.run.smithi038.stdout:(40/131): python3-asyncssh-2.7.0-2.el8.noarch.r 4.3 MB/s | 476 kB 00:00 2023-03-09T14:34:49.784 INFO:teuthology.orchestra.run.smithi181.stdout:(111/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 9.3 MB/s | 560 kB 00:00 2023-03-09T14:34:49.788 INFO:teuthology.orchestra.run.smithi122.stdout:(131/131): librados2-18.0.0-2872.ge0ab30a2.el8. 5.2 MB/s | 3.7 MB 00:00 2023-03-09T14:34:49.791 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2023-03-09T14:34:49.792 INFO:teuthology.orchestra.run.smithi122.stdout:Total 14 MB/s | 203 MB 00:14 2023-03-09T14:34:49.799 INFO:teuthology.orchestra.run.smithi181.stdout:(112/131): libaio-devel-0.3.112-1.el8.x86_64.rp 373 kB/s | 19 kB 00:00 2023-03-09T14:34:49.866 INFO:teuthology.orchestra.run.smithi181.stdout:(113/131): postgresql-server-10.21-2.module+el8 8.4 MB/s | 5.1 MB 00:00 2023-03-09T14:34:49.891 INFO:teuthology.orchestra.run.smithi181.stdout:(114/131): python3-oauthlib-2.1.0-1.el8.noarch. 1.4 MB/s | 155 kB 00:00 2023-03-09T14:34:49.892 INFO:teuthology.orchestra.run.smithi038.stdout:(41/131): ceph-mgr-diskprediction-local-18.0.0- 6.7 MB/s | 7.4 MB 00:01 2023-03-09T14:34:49.916 INFO:teuthology.orchestra.run.smithi181.stdout:(115/131): libverto-devel-0.3.0-5.el8.x86_64.rp 357 kB/s | 18 kB 00:00 2023-03-09T14:34:49.917 INFO:teuthology.orchestra.run.smithi038.stdout:(42/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 290 kB/s | 44 kB 00:00 2023-03-09T14:34:49.959 INFO:teuthology.orchestra.run.smithi181.stdout:(116/131): python3-cryptography-3.2.1-5.el8.x86 3.4 MB/s | 559 kB 00:00 2023-03-09T14:34:49.967 INFO:teuthology.orchestra.run.smithi038.stdout:(43/131): python3-cachetools-3.1.1-4.el8.noarch 660 kB/s | 33 kB 00:00 2023-03-09T14:34:49.984 INFO:teuthology.orchestra.run.smithi181.stdout:(117/131): pcre2-devel-10.32-2.el8.x86_64.rpm 6.4 MB/s | 605 kB 00:00 2023-03-09T14:34:50.009 INFO:teuthology.orchestra.run.smithi181.stdout:(118/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 2.5 MB/s | 238 kB 00:00 2023-03-09T14:34:50.017 INFO:teuthology.orchestra.run.smithi038.stdout:(44/131): python3-certifi-2018.10.15-7.el8.noar 327 kB/s | 16 kB 00:00 2023-03-09T14:34:50.034 INFO:teuthology.orchestra.run.smithi181.stdout:(119/131): xfsprogs-devel-5.0.0-10.el8.x86_64.r 1.4 MB/s | 72 kB 00:00 2023-03-09T14:34:50.043 INFO:teuthology.orchestra.run.smithi038.stdout:(45/131): python3-beautifulsoup4-4.6.3-2.el8.1. 1.2 MB/s | 185 kB 00:00 2023-03-09T14:34:50.060 INFO:teuthology.orchestra.run.smithi181.stdout:(120/131): fuse-2.9.7-15.el8.x86_64.rpm 1.6 MB/s | 83 kB 00:00 2023-03-09T14:34:50.061 INFO:teuthology.orchestra.run.smithi038.stdout:(46/131): python3-cheroot-8.5.2-1.el8.noarch.rp 4.0 MB/s | 173 kB 00:00 2023-03-09T14:34:50.085 INFO:teuthology.orchestra.run.smithi181.stdout:(121/131): elfutils-libelf-devel-0.186-1.el8.x8 1.2 MB/s | 60 kB 00:00 2023-03-09T14:34:50.110 INFO:teuthology.orchestra.run.smithi038.stdout:(47/131): python3-google-auth-1.1.1-10.el8.noar 1.6 MB/s | 81 kB 00:00 2023-03-09T14:34:50.111 INFO:teuthology.orchestra.run.smithi181.stdout:(122/131): libuuid-devel-2.32.1-35.el8.x86_64.r 1.9 MB/s | 98 kB 00:00 2023-03-09T14:34:50.135 INFO:teuthology.orchestra.run.smithi038.stdout:(48/131): python3-cherrypy-18.4.0-1.el8.noarch. 4.1 MB/s | 384 kB 00:00 2023-03-09T14:34:50.136 INFO:teuthology.orchestra.run.smithi181.stdout:(123/131): zlib-devel-1.2.11-18.el8_5.x86_64.rp 1.1 MB/s | 58 kB 00:00 2023-03-09T14:34:50.161 INFO:teuthology.orchestra.run.smithi181.stdout:(124/131): libcom_err-devel-1.45.6-4.el8.x86_64 769 kB/s | 39 kB 00:00 2023-03-09T14:34:50.161 INFO:teuthology.orchestra.run.smithi038.stdout:(49/131): python3-jaraco-6.2-6.el8.noarch.rpm 221 kB/s | 11 kB 00:00 2023-03-09T14:34:50.186 INFO:teuthology.orchestra.run.smithi038.stdout:(50/131): python3-jaraco-functools-2.0-4.el8.no 350 kB/s | 18 kB 00:00 2023-03-09T14:34:50.203 INFO:teuthology.orchestra.run.smithi181.stdout:(125/131): libgfortran-8.5.0-10.1.el8_6.x86_64. 9.4 MB/s | 644 kB 00:00 2023-03-09T14:34:50.236 INFO:teuthology.orchestra.run.smithi038.stdout:(51/131): python3-logutils-0.3.5-11.el8.noarch. 975 kB/s | 49 kB 00:00 2023-03-09T14:34:50.253 INFO:teuthology.orchestra.run.smithi181.stdout:(126/131): libquadmath-8.5.0-10.1.el8_6.x86_64. 1.8 MB/s | 171 kB 00:00 2023-03-09T14:34:50.278 INFO:teuthology.orchestra.run.smithi181.stdout:(127/131): NetworkManager-initscripts-updown-1. 1.8 MB/s | 138 kB 00:00 2023-03-09T14:34:50.286 INFO:teuthology.orchestra.run.smithi038.stdout:(52/131): python3-more-itertools-7.2.0-3.el8.no 1.1 MB/s | 59 kB 00:00 2023-03-09T14:34:50.328 INFO:teuthology.orchestra.run.smithi181.stdout:(128/131): openssl-devel-1.1.1k-7.el8_6.x86_64. 6.3 MB/s | 2.3 MB 00:00 2023-03-09T14:34:50.337 INFO:teuthology.orchestra.run.smithi038.stdout:(53/131): python3-natsort-7.1.1-2.el8.noarch.rp 1.2 MB/s | 60 kB 00:00 2023-03-09T14:34:50.370 INFO:teuthology.orchestra.run.smithi038.stdout:(54/131): python3-kubernetes-11.0.0-6.el8.noarc 5.6 MB/s | 1.2 MB 00:00 2023-03-09T14:34:50.379 INFO:teuthology.orchestra.run.smithi181.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 16 MB/s | 798 kB 00:00 2023-03-09T14:34:50.421 INFO:teuthology.orchestra.run.smithi038.stdout:(55/131): python3-pecan-1.3.2-9.el8.noarch.rpm 3.3 MB/s | 283 kB 00:00 2023-03-09T14:34:50.455 INFO:teuthology.orchestra.run.smithi038.stdout:(56/131): python3-portend-2.6-1.el8.noarch.rpm 186 kB/s | 16 kB 00:00 2023-03-09T14:34:50.480 INFO:teuthology.orchestra.run.smithi038.stdout:(57/131): python3-repoze-lru-0.7-6.el8.noarch.r 572 kB/s | 34 kB 00:00 2023-03-09T14:34:50.505 INFO:teuthology.orchestra.run.smithi038.stdout:(58/131): python3-routes-2.4.1-12.el8.noarch.rp 3.8 MB/s | 196 kB 00:00 2023-03-09T14:34:50.535 INFO:teuthology.orchestra.run.smithi038.stdout:(59/131): python3-rsa-4.8-1.el8.noarch.rpm 1.2 MB/s | 61 kB 00:00 2023-03-09T14:34:50.556 INFO:teuthology.orchestra.run.smithi038.stdout:(60/131): python3-simplegeneric-0.8.1-17.el8.no 372 kB/s | 19 kB 00:00 2023-03-09T14:34:50.605 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2023-03-09T14:34:50.612 INFO:teuthology.orchestra.run.smithi038.stdout:(61/131): python3-singledispatch-3.4.0.3-18.el8 294 kB/s | 24 kB 00:00 2023-03-09T14:34:50.640 INFO:teuthology.orchestra.run.smithi038.stdout:(62/131): python3-tempora-1.14.1-5.el8.noarch.r 349 kB/s | 29 kB 00:00 2023-03-09T14:34:50.665 INFO:teuthology.orchestra.run.smithi038.stdout:(63/131): python3-trustme-0.6.0-4.el8.noarch.rp 525 kB/s | 27 kB 00:00 2023-03-09T14:34:50.695 INFO:teuthology.orchestra.run.smithi181.stdout:(130/131): librbd1-18.0.0-2872.ge0ab30a2.el8.x8 8.5 MB/s | 3.6 MB 00:00 2023-03-09T14:34:50.724 INFO:teuthology.orchestra.run.smithi038.stdout:(64/131): python3-webob-1.8.5-1.el8.1.noarch.rp 4.2 MB/s | 251 kB 00:00 2023-03-09T14:34:50.726 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2023-03-09T14:34:50.727 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2023-03-09T14:34:50.749 INFO:teuthology.orchestra.run.smithi038.stdout:(65/131): python3-waitress-1.2.1-2.el8.1.noarch 2.2 MB/s | 241 kB 00:00 2023-03-09T14:34:50.774 INFO:teuthology.orchestra.run.smithi038.stdout:(66/131): python3-websocket-client-0.56.0-5.el8 1.2 MB/s | 61 kB 00:00 2023-03-09T14:34:50.800 INFO:teuthology.orchestra.run.smithi038.stdout:(67/131): python3-webtest-2.0.33-1.el8.noarch.r 1.7 MB/s | 85 kB 00:00 2023-03-09T14:34:50.825 INFO:teuthology.orchestra.run.smithi038.stdout:(68/131): python3-zc-lockfile-2.0-2.el8.noarch. 456 kB/s | 23 kB 00:00 2023-03-09T14:34:50.850 INFO:teuthology.orchestra.run.smithi038.stdout:(69/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.7 MB/s | 70 kB 00:00 2023-03-09T14:34:50.879 INFO:teuthology.orchestra.run.smithi181.stdout:(131/131): librados2-18.0.0-2872.ge0ab30a2.el8. 5.8 MB/s | 3.7 MB 00:00 2023-03-09T14:34:50.882 INFO:teuthology.orchestra.run.smithi181.stdout:-------------------------------------------------------------------------------- 2023-03-09T14:34:50.882 INFO:teuthology.orchestra.run.smithi181.stdout:Total 13 MB/s | 203 MB 00:15 2023-03-09T14:34:51.192 INFO:teuthology.orchestra.run.smithi038.stdout:(70/131): python3-scipy-1.0.0-21.module+el8.5.0 40 MB/s | 14 MB 00:00 2023-03-09T14:34:51.226 INFO:teuthology.orchestra.run.smithi038.stdout:(71/131): thrift-0.13.0-2.el8.x86_64.rpm 4.1 MB/s | 1.7 MB 00:00 2023-03-09T14:34:51.251 INFO:teuthology.orchestra.run.smithi038.stdout:(72/131): python3-pyasn1-modules-0.3.7-6.el8.no 1.8 MB/s | 110 kB 00:00 2023-03-09T14:34:51.293 INFO:teuthology.orchestra.run.smithi038.stdout:(73/131): python3-pytz-2017.2-9.el8.noarch.rpm 810 kB/s | 54 kB 00:00 2023-03-09T14:34:51.318 INFO:teuthology.orchestra.run.smithi038.stdout:(74/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 4.3 MB/s | 259 kB 00:00 2023-03-09T14:34:51.352 INFO:teuthology.orchestra.run.smithi038.stdout:(75/131): socat-1.7.4.1-1.el8.x86_64.rpm 5.4 MB/s | 323 kB 00:00 2023-03-09T14:34:51.377 INFO:teuthology.orchestra.run.smithi038.stdout:(76/131): python3-markupsafe-0.23-19.el8.x86_64 662 kB/s | 39 kB 00:00 2023-03-09T14:34:51.432 INFO:teuthology.orchestra.run.smithi038.stdout:(77/131): bison-3.0.4-10.el8.x86_64.rpm 8.5 MB/s | 688 kB 00:00 2023-03-09T14:34:51.470 INFO:teuthology.orchestra.run.smithi038.stdout:(78/131): python3-jinja2-2.10.1-3.el8.noarch.rp 5.7 MB/s | 538 kB 00:00 2023-03-09T14:34:51.495 INFO:teuthology.orchestra.run.smithi038.stdout:(79/131): python3-mako-1.0.6-13.el8.noarch.rpm 2.4 MB/s | 157 kB 00:00 2023-03-09T14:34:51.520 INFO:teuthology.orchestra.run.smithi038.stdout:(80/131): oniguruma-6.8.2-2.el8.x86_64.rpm 3.6 MB/s | 187 kB 00:00 2023-03-09T14:34:51.554 INFO:teuthology.orchestra.run.smithi038.stdout:(81/131): automake-1.16.1-7.el8.noarch.rpm 12 MB/s | 713 kB 00:00 2023-03-09T14:34:51.588 INFO:teuthology.orchestra.run.smithi038.stdout:(82/131): python3-jsonpatch-1.21-2.el8.noarch.r 788 kB/s | 27 kB 00:00 2023-03-09T14:34:51.630 INFO:teuthology.orchestra.run.smithi038.stdout:(83/131): flex-2.6.1-9.el8.x86_64.rpm 7.5 MB/s | 320 kB 00:00 2023-03-09T14:34:51.664 INFO:teuthology.orchestra.run.smithi038.stdout:(84/131): python3-prettytable-0.7.2-14.el8.noar 1.3 MB/s | 44 kB 00:00 2023-03-09T14:34:51.703 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction check 2023-03-09T14:34:51.814 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2023-03-09T14:34:51.818 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2023-03-09T14:34:51.823 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction check succeeded. 2023-03-09T14:34:51.823 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction test 2023-03-09T14:34:51.864 INFO:teuthology.orchestra.run.smithi038.stdout:(85/131): python3-babel-2.5.1-7.el8.noarch.rpm 14 MB/s | 4.8 MB 00:00 2023-03-09T14:34:51.922 INFO:teuthology.orchestra.run.smithi038.stdout:(86/131): python3-werkzeug-0.12.2-4.el8.noarch. 7.7 MB/s | 457 kB 00:00 2023-03-09T14:34:51.964 INFO:teuthology.orchestra.run.smithi038.stdout:(87/131): python3-numpy-f2py-1.14.3-10.el8.x86_ 5.3 MB/s | 225 kB 00:00 2023-03-09T14:34:52.040 INFO:teuthology.orchestra.run.smithi038.stdout:(88/131): python3-numpy-1.14.3-10.el8.x86_64.rp 9.9 MB/s | 3.7 MB 00:00 2023-03-09T14:34:52.065 INFO:teuthology.orchestra.run.smithi038.stdout:(89/131): uuid-1.6.2-43.el8.x86_64.rpm 637 kB/s | 64 kB 00:00 2023-03-09T14:34:52.090 INFO:teuthology.orchestra.run.smithi038.stdout:(90/131): python3-pyasn1-0.3.7-6.el8.noarch.rpm 2.5 MB/s | 126 kB 00:00 2023-03-09T14:34:52.116 INFO:teuthology.orchestra.run.smithi038.stdout:(91/131): python3-jsonpointer-1.10-11.el8.noarc 398 kB/s | 20 kB 00:00 2023-03-09T14:34:52.141 INFO:teuthology.orchestra.run.smithi038.stdout:(92/131): python3-pyOpenSSL-19.0.0-1.el8.noarch 2.0 MB/s | 103 kB 00:00 2023-03-09T14:34:52.575 INFO:teuthology.orchestra.run.smithi038.stdout:(93/131): ceph-test-18.0.0-2872.ge0ab30a2.el8.x 7.9 MB/s | 47 MB 00:05 2023-03-09T14:34:52.625 INFO:teuthology.orchestra.run.smithi038.stdout:(94/131): libtool-2.4.6-25.el8.x86_64.rpm 1.4 MB/s | 709 kB 00:00 2023-03-09T14:34:52.650 INFO:teuthology.orchestra.run.smithi038.stdout:(95/131): libpq-13.5-1.el8.x86_64.rpm 2.6 MB/s | 198 kB 00:00 2023-03-09T14:34:52.692 INFO:teuthology.orchestra.run.smithi038.stdout:(96/131): jq-1.6-3.el8.x86_64.rpm 4.8 MB/s | 202 kB 00:00 2023-03-09T14:34:52.742 INFO:teuthology.orchestra.run.smithi038.stdout:(97/131): librdkafka-0.11.4-3.el8.x86_64.rpm 6.9 MB/s | 354 kB 00:00 2023-03-09T14:34:52.931 INFO:teuthology.orchestra.run.smithi181.stdout:Transaction test succeeded. 2023-03-09T14:34:52.935 INFO:teuthology.orchestra.run.smithi181.stdout:Running transaction 2023-03-09T14:34:52.993 INFO:teuthology.orchestra.run.smithi038.stdout:(98/131): openblas-threads-0.3.15-3.el8.x86_64. 5.7 MB/s | 4.9 MB 00:00 2023-03-09T14:34:53.226 INFO:teuthology.orchestra.run.smithi038.stdout:(99/131): openblas-0.3.15-3.el8.x86_64.rpm 7.9 MB/s | 4.7 MB 00:00 2023-03-09T14:34:53.268 INFO:teuthology.orchestra.run.smithi038.stdout:(100/131): postgresql-contrib-10.21-2.module+el 2.9 MB/s | 808 kB 00:00 2023-03-09T14:34:53.294 INFO:teuthology.orchestra.run.smithi038.stdout:(101/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 1.8 MB/s | 47 kB 00:00 2023-03-09T14:34:53.344 INFO:teuthology.orchestra.run.smithi038.stdout:(102/131): postgresql-10.21-2.module+el8.6.0+15 13 MB/s | 1.5 MB 00:00 2023-03-09T14:34:53.361 INFO:teuthology.orchestra.run.smithi038.stdout:(103/131): libselinux-devel-2.9-5.el8.x86_64.rp 2.9 MB/s | 200 kB 00:00 2023-03-09T14:34:53.386 INFO:teuthology.orchestra.run.smithi038.stdout:(104/131): python3-jwt-1.6.1-2.el8.noarch.rpm 1.0 MB/s | 43 kB 00:00 2023-03-09T14:34:53.411 INFO:teuthology.orchestra.run.smithi038.stdout:(105/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 4.3 MB/s | 220 kB 00:00 2023-03-09T14:34:53.437 INFO:teuthology.orchestra.run.smithi038.stdout:(106/131): keyutils-libs-devel-1.5.10-9.el8.x86 957 kB/s | 48 kB 00:00 2023-03-09T14:34:53.462 INFO:teuthology.orchestra.run.smithi038.stdout:(107/131): libsepol-devel-2.9-3.el8.x86_64.rpm 1.7 MB/s | 87 kB 00:00 2023-03-09T14:34:53.487 INFO:teuthology.orchestra.run.smithi038.stdout:(108/131): python3-requests-oauthlib-1.0.0-1.el 849 kB/s | 43 kB 00:00 2023-03-09T14:34:53.513 INFO:teuthology.orchestra.run.smithi038.stdout:(109/131): python3-pycparser-2.14-14.el8.noarch 2.1 MB/s | 109 kB 00:00 2023-03-09T14:34:53.588 INFO:teuthology.orchestra.run.smithi038.stdout:(110/131): postgresql-server-10.21-2.module+el8 6.0 MB/s | 5.1 MB 00:00 2023-03-09T14:34:53.613 INFO:teuthology.orchestra.run.smithi038.stdout:(111/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 1.8 MB/s | 229 kB 00:00 2023-03-09T14:34:53.639 INFO:teuthology.orchestra.run.smithi038.stdout:(112/131): libaio-devel-0.3.112-1.el8.x86_64.rp 373 kB/s | 19 kB 00:00 2023-03-09T14:34:53.664 INFO:teuthology.orchestra.run.smithi038.stdout:(113/131): python3-oauthlib-2.1.0-1.el8.noarch. 3.0 MB/s | 155 kB 00:00 2023-03-09T14:34:53.698 INFO:teuthology.orchestra.run.smithi038.stdout:(114/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 3.0 MB/s | 560 kB 00:00 2023-03-09T14:34:53.723 INFO:teuthology.orchestra.run.smithi038.stdout:(115/131): libverto-devel-0.3.0-5.el8.x86_64.rp 306 kB/s | 18 kB 00:00 2023-03-09T14:34:53.757 INFO:teuthology.orchestra.run.smithi038.stdout:(116/131): python3-cryptography-3.2.1-5.el8.x86 4.7 MB/s | 559 kB 00:00 2023-03-09T14:34:53.790 INFO:teuthology.orchestra.run.smithi038.stdout:(117/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 3.5 MB/s | 238 kB 00:00 2023-03-09T14:34:53.824 INFO:teuthology.orchestra.run.smithi038.stdout:(118/131): pcre2-devel-10.32-2.el8.x86_64.rpm 4.7 MB/s | 605 kB 00:00 2023-03-09T14:34:53.849 INFO:teuthology.orchestra.run.smithi038.stdout:(119/131): xfsprogs-devel-5.0.0-10.el8.x86_64.r 1.2 MB/s | 72 kB 00:00 2023-03-09T14:34:53.883 INFO:teuthology.orchestra.run.smithi038.stdout:(120/131): fuse-2.9.7-15.el8.x86_64.rpm 1.4 MB/s | 83 kB 00:00 2023-03-09T14:34:53.929 INFO:teuthology.orchestra.run.smithi038.stdout:(121/131): elfutils-libelf-devel-0.186-1.el8.x8 754 kB/s | 60 kB 00:00 2023-03-09T14:34:53.959 INFO:teuthology.orchestra.run.smithi038.stdout:(122/131): libuuid-devel-2.32.1-35.el8.x86_64.r 1.3 MB/s | 98 kB 00:00 2023-03-09T14:34:53.984 INFO:teuthology.orchestra.run.smithi038.stdout:(123/131): zlib-devel-1.2.11-18.el8_5.x86_64.rp 1.0 MB/s | 58 kB 00:00 2023-03-09T14:34:54.009 INFO:teuthology.orchestra.run.smithi038.stdout:(124/131): libcom_err-devel-1.45.6-4.el8.x86_64 769 kB/s | 39 kB 00:00 2023-03-09T14:34:54.051 INFO:teuthology.orchestra.run.smithi038.stdout:(125/131): libgfortran-8.5.0-10.1.el8_6.x86_64. 9.4 MB/s | 644 kB 00:00 2023-03-09T14:34:54.076 INFO:teuthology.orchestra.run.smithi038.stdout:(126/131): libquadmath-8.5.0-10.1.el8_6.x86_64. 2.5 MB/s | 171 kB 00:00 2023-03-09T14:34:54.102 INFO:teuthology.orchestra.run.smithi038.stdout:(127/131): NetworkManager-initscripts-updown-1. 2.7 MB/s | 138 kB 00:00 2023-03-09T14:34:54.160 INFO:teuthology.orchestra.run.smithi038.stdout:(128/131): openssl-devel-1.1.1k-7.el8_6.x86_64. 5.8 MB/s | 2.3 MB 00:00 2023-03-09T14:34:54.219 INFO:teuthology.orchestra.run.smithi038.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 13 MB/s | 798 kB 00:00 2023-03-09T14:34:54.477 INFO:teuthology.orchestra.run.smithi038.stdout:(130/131): librbd1-18.0.0-2872.ge0ab30a2.el8.x8 9.5 MB/s | 3.6 MB 00:00 2023-03-09T14:34:54.636 INFO:teuthology.orchestra.run.smithi038.stdout:(131/131): librados2-18.0.0-2872.ge0ab30a2.el8. 6.5 MB/s | 3.7 MB 00:00 2023-03-09T14:34:54.649 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2023-03-09T14:34:54.649 INFO:teuthology.orchestra.run.smithi038.stdout:Total 11 MB/s | 203 MB 00:17 2023-03-09T14:34:55.462 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction check 2023-03-09T14:34:55.595 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction check succeeded. 2023-03-09T14:34:55.595 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction test 2023-03-09T14:34:56.764 INFO:teuthology.orchestra.run.smithi038.stdout:Transaction test succeeded. 2023-03-09T14:34:56.768 INFO:teuthology.orchestra.run.smithi038.stdout:Running transaction 2023-03-09T14:34:56.863 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2023-03-09T14:34:57.085 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-03-09T14:34:57.244 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-03-09T14:34:57.469 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 3/134 2023-03-09T14:34:57.654 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 4/134 2023-03-09T14:34:57.758 INFO:teuthology.orchestra.run.smithi181.stdout: Preparing : 1/1 2023-03-09T14:34:57.791 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 4/134 2023-03-09T14:34:57.956 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-03-09T14:34:58.029 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-03-09T14:34:58.109 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-03-09T14:34:58.315 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 3/134 2023-03-09T14:34:58.419 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-03-09T14:34:58.476 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 4/134 2023-03-09T14:34:58.573 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 4/134 2023-03-09T14:34:58.604 INFO:teuthology.orchestra.run.smithi122.stdout: Upgrading : librados2-2:18.0.0-2872.ge0ab30a2.el8.x86_64 6/134 2023-03-09T14:34:58.817 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-03-09T14:34:58.998 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librados2-2:18.0.0-2872.ge0ab30a2.el8.x86_64 6/134 2023-03-09T14:34:59.195 INFO:teuthology.orchestra.run.smithi122.stdout: Upgrading : librbd1-2:18.0.0-2872.ge0ab30a2.el8.x86_64 7/134 2023-03-09T14:34:59.218 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-03-09T14:34:59.324 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librbd1-2:18.0.0-2872.ge0ab30a2.el8.x86_64 7/134 2023-03-09T14:34:59.401 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : librados2-2:18.0.0-2872.ge0ab30a2.el8.x86_64 6/134 2023-03-09T14:34:59.779 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libcephfs2-2:18.0.0-2872.ge0ab30a2.el8.x86_64 8/134 2023-03-09T14:34:59.795 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librados2-2:18.0.0-2872.ge0ab30a2.el8.x86_64 6/134 2023-03-09T14:34:59.879 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: libcephfs2-2:18.0.0-2872.ge0ab30a2.el8.x86_64 8/134 2023-03-09T14:35:00.020 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : librbd1-2:18.0.0-2872.ge0ab30a2.el8.x86_64 7/134 2023-03-09T14:35:00.080 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-rados-2:18.0.0-2872.ge0ab30a2.el8.x86_64 9/134 2023-03-09T14:35:00.145 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librbd1-2:18.0.0-2872.ge0ab30a2.el8.x86_64 7/134 2023-03-09T14:35:00.313 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-03-09T14:35:00.325 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libcephfs2-2:18.0.0-2872.ge0ab30a2.el8.x86_64 8/134 2023-03-09T14:35:00.408 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libcephfs2-2:18.0.0-2872.ge0ab30a2.el8.x86_64 8/134 2023-03-09T14:35:00.540 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-03-09T14:35:00.599 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rados-2:18.0.0-2872.ge0ab30a2.el8.x86_64 9/134 2023-03-09T14:35:00.837 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-03-09T14:35:00.947 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-03-09T14:35:01.028 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-03-09T14:35:01.198 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-03-09T14:35:01.336 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libcephsqlite-2:18.0.0-2872.ge0ab30a2.el8.x86_64 14/134 2023-03-09T14:35:01.444 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: libcephsqlite-2:18.0.0-2872.ge0ab30a2.el8.x86_64 14/134 2023-03-09T14:35:01.450 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-03-09T14:35:01.636 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libradosstriper1-2:18.0.0-2872.ge0ab30a2.el8.x86 15/134 2023-03-09T14:35:01.670 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-03-09T14:35:01.720 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: libradosstriper1-2:18.0.0-2872.ge0ab30a2.el8.x86 15/134 2023-03-09T14:35:01.743 INFO:teuthology.orchestra.run.smithi038.stdout: Preparing : 1/1 2023-03-09T14:35:01.808 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libcephsqlite-2:18.0.0-2872.ge0ab30a2.el8.x86_64 14/134 2023-03-09T14:35:01.861 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-03-09T14:35:01.914 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libcephsqlite-2:18.0.0-2872.ge0ab30a2.el8.x86_64 14/134 2023-03-09T14:35:01.916 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-03-09T14:35:02.000 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-03-09T14:35:02.058 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libradosstriper1-2:18.0.0-2872.ge0ab30a2.el8.x86 15/134 2023-03-09T14:35:02.087 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-03-09T14:35:02.137 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libradosstriper1-2:18.0.0-2872.ge0ab30a2.el8.x86 15/134 2023-03-09T14:35:02.161 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-03-09T14:35:02.286 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 3/134 2023-03-09T14:35:02.299 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-03-09T14:35:02.440 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-03-09T14:35:02.452 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 4/134 2023-03-09T14:35:02.577 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 4/134 2023-03-09T14:35:02.582 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-03-09T14:35:02.775 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-03-09T14:35:02.844 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-03-09T14:35:02.978 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-03-09T14:35:03.188 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-03-09T14:35:03.269 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-03-09T14:35:03.391 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-03-09T14:35:03.567 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-03-09T14:35:03.761 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : openblas-0.3.15-3.el8.x86_64 19/134 2023-03-09T14:35:03.785 INFO:teuthology.orchestra.run.smithi038.stdout: Upgrading : librados2-2:18.0.0-2872.ge0ab30a2.el8.x86_64 6/134 2023-03-09T14:35:03.996 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-03-09T14:35:04.174 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : openblas-0.3.15-3.el8.x86_64 19/134 2023-03-09T14:35:04.214 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librados2-2:18.0.0-2872.ge0ab30a2.el8.x86_64 6/134 2023-03-09T14:35:04.427 INFO:teuthology.orchestra.run.smithi038.stdout: Upgrading : librbd1-2:18.0.0-2872.ge0ab30a2.el8.x86_64 7/134 2023-03-09T14:35:04.463 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 19/134 2023-03-09T14:35:04.557 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librbd1-2:18.0.0-2872.ge0ab30a2.el8.x86_64 7/134 2023-03-09T14:35:04.717 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 20/134 2023-03-09T14:35:04.718 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libcephfs2-2:18.0.0-2872.ge0ab30a2.el8.x86_64 8/134 2023-03-09T14:35:04.818 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libcephfs2-2:18.0.0-2872.ge0ab30a2.el8.x86_64 8/134 2023-03-09T14:35:04.879 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 19/134 2023-03-09T14:35:04.945 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 21/134 2023-03-09T14:35:05.003 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rados-2:18.0.0-2872.ge0ab30a2.el8.x86_64 9/134 2023-03-09T14:35:05.139 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 20/134 2023-03-09T14:35:05.242 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-03-09T14:35:05.366 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 21/134 2023-03-09T14:35:05.400 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 22/134 2023-03-09T14:35:05.463 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-03-09T14:35:05.603 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 22/134 2023-03-09T14:35:05.619 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 23/134 2023-03-09T14:35:05.780 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 23/134 2023-03-09T14:35:05.839 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 24/134 2023-03-09T14:35:05.857 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-03-09T14:35:05.982 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 24/134 2023-03-09T14:35:06.082 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-03-09T14:35:06.106 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-03-09T14:35:06.218 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-03-09T14:35:06.243 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libcephsqlite-2:18.0.0-2872.ge0ab30a2.el8.x86_64 14/134 2023-03-09T14:35:06.293 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-ceph-common-2:18.0.0-2872.ge0ab30a2.el8. 26/134 2023-03-09T14:35:06.334 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libcephsqlite-2:18.0.0-2872.ge0ab30a2.el8.x86_64 14/134 2023-03-09T14:35:06.380 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-ceph-common-2:18.0.0-2872.ge0ab30a2.el8. 26/134 2023-03-09T14:35:06.439 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-ceph-argparse-2:18.0.0-2872.ge0ab30a2.el 27/134 2023-03-09T14:35:06.476 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libradosstriper1-2:18.0.0-2872.ge0ab30a2.el8.x86 15/134 2023-03-09T14:35:06.493 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-ceph-argparse-2:18.0.0-2872.ge0ab30a2.el 27/134 2023-03-09T14:35:06.557 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libradosstriper1-2:18.0.0-2872.ge0ab30a2.el8.x86 15/134 2023-03-09T14:35:06.676 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-03-09T14:35:06.820 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-03-09T14:35:07.009 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-03-09T14:35:07.345 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cephfs-2:18.0.0-2872.ge0ab30a2.el8.x86_6 28/134 2023-03-09T14:35:07.407 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cephfs-2:18.0.0-2872.ge0ab30a2.el8.x86_6 28/134 2023-03-09T14:35:07.666 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-03-09T14:35:07.692 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 29/134 2023-03-09T14:35:07.705 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 29/134 2023-03-09T14:35:07.843 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-03-09T14:35:07.951 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 30/134 2023-03-09T14:35:07.954 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 30/134 2023-03-09T14:35:08.193 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 31/134 2023-03-09T14:35:08.211 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 31/134 2023-03-09T14:35:08.365 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-03-09T14:35:08.444 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-03-09T14:35:08.472 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-03-09T14:35:08.660 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : openblas-0.3.15-3.el8.x86_64 19/134 2023-03-09T14:35:09.407 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 19/134 2023-03-09T14:35:09.674 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 20/134 2023-03-09T14:35:09.899 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 21/134 2023-03-09T14:35:10.133 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-03-09T14:35:10.139 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 22/134 2023-03-09T14:35:10.201 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-03-09T14:35:10.325 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 23/134 2023-03-09T14:35:10.441 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 33/134 2023-03-09T14:35:10.503 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 33/134 2023-03-09T14:35:10.537 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 24/134 2023-03-09T14:35:10.662 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rsa-4.8-1.el8.noarch 34/134 2023-03-09T14:35:10.747 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-rsa-4.8-1.el8.noarch 34/134 2023-03-09T14:35:10.755 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-03-09T14:35:10.816 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-03-09T14:35:10.941 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-ceph-common-2:18.0.0-2872.ge0ab30a2.el8. 26/134 2023-03-09T14:35:10.945 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-03-09T14:35:11.012 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rbd-2:18.0.0-2872.ge0ab30a2.el8.x86_64 36/134 2023-03-09T14:35:11.071 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-ceph-argparse-2:18.0.0-2872.ge0ab30a2.el 27/134 2023-03-09T14:35:11.133 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-rbd-2:18.0.0-2872.ge0ab30a2.el8.x86_64 36/134 2023-03-09T14:35:11.193 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librados-devel-2:18.0.0-2872.ge0ab30a2.el8.x86_6 37/134 2023-03-09T14:35:11.289 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : librados-devel-2:18.0.0-2872.ge0ab30a2.el8.x86_6 37/134 2023-03-09T14:35:11.347 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 38/134 2023-03-09T14:35:11.493 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 38/134 2023-03-09T14:35:11.508 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 39/134 2023-03-09T14:35:11.671 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 39/134 2023-03-09T14:35:11.709 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : fuse-2.9.7-15.el8.x86_64 40/134 2023-03-09T14:35:11.851 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 41/134 2023-03-09T14:35:11.855 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : fuse-2.9.7-15.el8.x86_64 40/134 2023-03-09T14:35:12.022 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 41/134 2023-03-09T14:35:12.031 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cephfs-2:18.0.0-2872.ge0ab30a2.el8.x86_6 28/134 2023-03-09T14:35:12.104 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 42/134 2023-03-09T14:35:12.259 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 42/134 2023-03-09T14:35:12.302 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 43/134 2023-03-09T14:35:12.348 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 29/134 2023-03-09T14:35:12.466 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 43/134 2023-03-09T14:35:12.656 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 30/134 2023-03-09T14:35:12.667 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 44/134 2023-03-09T14:35:12.866 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 44/134 2023-03-09T14:35:12.925 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 31/134 2023-03-09T14:35:12.926 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 45/134 2023-03-09T14:35:13.100 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-03-09T14:35:13.133 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 46/134 2023-03-09T14:35:13.138 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 45/134 2023-03-09T14:35:13.326 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 47/134 2023-03-09T14:35:13.353 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 46/134 2023-03-09T14:35:13.587 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 47/134 2023-03-09T14:35:13.594 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 48/134 2023-03-09T14:35:13.865 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 48/134 2023-03-09T14:35:13.897 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 49/134 2023-03-09T14:35:14.130 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 50/134 2023-03-09T14:35:14.208 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 49/134 2023-03-09T14:35:14.349 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 51/134 2023-03-09T14:35:14.426 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 50/134 2023-03-09T14:35:14.577 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 52/134 2023-03-09T14:35:14.617 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 51/134 2023-03-09T14:35:14.743 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 53/134 2023-03-09T14:35:14.856 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 52/134 2023-03-09T14:35:14.900 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 54/134 2023-03-09T14:35:14.955 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-03-09T14:35:15.022 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 53/134 2023-03-09T14:35:15.186 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 55/134 2023-03-09T14:35:15.196 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 54/134 2023-03-09T14:35:15.271 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 33/134 2023-03-09T14:35:15.481 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rsa-4.8-1.el8.noarch 34/134 2023-03-09T14:35:15.487 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 56/134 2023-03-09T14:35:15.596 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 55/134 2023-03-09T14:35:15.688 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-03-09T14:35:15.719 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 57/134 2023-03-09T14:35:15.848 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 56/134 2023-03-09T14:35:15.890 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rbd-2:18.0.0-2872.ge0ab30a2.el8.x86_64 36/134 2023-03-09T14:35:15.937 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 58/134 2023-03-09T14:35:16.063 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librados-devel-2:18.0.0-2872.ge0ab30a2.el8.x86_6 37/134 2023-03-09T14:35:16.121 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 59/134 2023-03-09T14:35:16.156 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 57/134 2023-03-09T14:35:16.259 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 38/134 2023-03-09T14:35:16.273 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 60/134 2023-03-09T14:35:16.383 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 58/134 2023-03-09T14:35:16.447 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 39/134 2023-03-09T14:35:16.455 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : uuid-1.6.2-43.el8.x86_64 61/134 2023-03-09T14:35:16.527 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 61/134 2023-03-09T14:35:16.558 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 59/134 2023-03-09T14:35:16.644 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : fuse-2.9.7-15.el8.x86_64 40/134 2023-03-09T14:35:16.752 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 60/134 2023-03-09T14:35:16.781 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 62/134 2023-03-09T14:35:16.797 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 41/134 2023-03-09T14:35:16.935 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : uuid-1.6.2-43.el8.x86_64 61/134 2023-03-09T14:35:16.984 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : automake-1.16.1-7.el8.noarch 63/134 2023-03-09T14:35:17.007 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 61/134 2023-03-09T14:35:17.010 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 42/134 2023-03-09T14:35:17.122 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 64/134 2023-03-09T14:35:17.205 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 64/134 2023-03-09T14:35:17.209 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 43/134 2023-03-09T14:35:17.268 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 62/134 2023-03-09T14:35:17.427 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : jq-1.6-3.el8.x86_64 65/134 2023-03-09T14:35:17.538 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : automake-1.16.1-7.el8.noarch 63/134 2023-03-09T14:35:17.606 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 66/134 2023-03-09T14:35:17.609 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 44/134 2023-03-09T14:35:17.693 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 64/134 2023-03-09T14:35:17.770 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 64/134 2023-03-09T14:35:17.771 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-03-09T14:35:17.862 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 45/134 2023-03-09T14:35:17.967 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-03-09T14:35:17.983 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : jq-1.6-3.el8.x86_64 65/134 2023-03-09T14:35:18.078 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 46/134 2023-03-09T14:35:18.145 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 66/134 2023-03-09T14:35:18.212 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-03-09T14:35:18.302 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-03-09T14:35:18.333 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 47/134 2023-03-09T14:35:18.420 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 70/134 2023-03-09T14:35:18.489 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-03-09T14:35:18.584 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-03-09T14:35:18.607 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 48/134 2023-03-09T14:35:18.812 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-03-09T14:35:18.951 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 49/134 2023-03-09T14:35:19.067 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 70/134 2023-03-09T14:35:19.091 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-03-09T14:35:19.158 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 50/134 2023-03-09T14:35:19.231 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-03-09T14:35:19.273 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-03-09T14:35:19.353 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 51/134 2023-03-09T14:35:19.421 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-03-09T14:35:19.441 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-03-09T14:35:19.627 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-03-09T14:35:19.649 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-03-09T14:35:19.808 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-03-09T14:35:19.839 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-03-09T14:35:19.891 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 52/134 2023-03-09T14:35:19.945 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-03-09T14:35:20.054 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-03-09T14:35:20.063 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 53/134 2023-03-09T14:35:20.171 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-03-09T14:35:20.212 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-03-09T14:35:20.229 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 54/134 2023-03-09T14:35:20.357 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-03-09T14:35:20.376 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-03-09T14:35:20.545 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 55/134 2023-03-09T14:35:20.553 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-03-09T14:35:20.602 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-03-09T14:35:20.811 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 56/134 2023-03-09T14:35:20.813 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-03-09T14:35:20.871 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-03-09T14:35:21.004 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-03-09T14:35:21.049 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 57/134 2023-03-09T14:35:21.123 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-03-09T14:35:21.171 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-03-09T14:35:21.298 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 58/134 2023-03-09T14:35:21.373 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-03-09T14:35:21.398 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-03-09T14:35:21.474 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 59/134 2023-03-09T14:35:21.650 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-03-09T14:35:21.660 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 60/134 2023-03-09T14:35:21.824 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : uuid-1.6.2-43.el8.x86_64 61/134 2023-03-09T14:35:21.858 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-03-09T14:35:21.900 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 61/134 2023-03-09T14:35:22.126 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-03-09T14:35:22.160 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 62/134 2023-03-09T14:35:22.346 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : automake-1.16.1-7.el8.noarch 63/134 2023-03-09T14:35:22.480 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-03-09T14:35:22.491 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 64/134 2023-03-09T14:35:22.553 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 64/134 2023-03-09T14:35:22.750 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-03-09T14:35:22.756 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : jq-1.6-3.el8.x86_64 65/134 2023-03-09T14:35:23.005 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-03-09T14:35:23.076 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 66/134 2023-03-09T14:35:23.211 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-03-09T14:35:23.234 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-03-09T14:35:23.325 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-03-09T14:35:23.439 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-03-09T14:35:23.529 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-03-09T14:35:23.578 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-03-09T14:35:23.715 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-03-09T14:35:23.770 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-03-09T14:35:23.774 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-03-09T14:35:24.024 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 70/134 2023-03-09T14:35:24.117 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-03-09T14:35:24.188 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-03-09T14:35:24.345 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-03-09T14:35:24.406 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-03-09T14:35:24.537 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-03-09T14:35:24.546 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-03-09T14:35:24.615 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-03-09T14:35:24.712 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : librgw2-2:18.0.0-2872.ge0ab30a2.el8.x86_64 92/134 2023-03-09T14:35:24.773 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-03-09T14:35:24.782 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librgw2-2:18.0.0-2872.ge0ab30a2.el8.x86_64 92/134 2023-03-09T14:35:24.946 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-03-09T14:35:24.947 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : python3-rgw-2:18.0.0-2872.ge0ab30a2.el8.x86_64 93/134 2023-03-09T14:35:25.152 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-03-09T14:35:25.289 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-03-09T14:35:25.407 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-03-09T14:35:25.531 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-03-09T14:35:25.575 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : librgw2-2:18.0.0-2872.ge0ab30a2.el8.x86_64 92/134 2023-03-09T14:35:25.643 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librgw2-2:18.0.0-2872.ge0ab30a2.el8.x86_64 92/134 2023-03-09T14:35:25.763 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-03-09T14:35:25.818 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-rgw-2:18.0.0-2872.ge0ab30a2.el8.x86_64 93/134 2023-03-09T14:35:25.963 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-03-09T14:35:26.220 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-03-09T14:35:26.474 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-03-09T14:35:26.649 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-03-09T14:35:26.877 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-03-09T14:35:28.028 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-03-09T14:35:28.380 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-03-09T14:35:28.454 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-common-2:18.0.0-2872.ge0ab30a2.el8.x86_64 94/134 2023-03-09T14:35:28.659 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-03-09T14:35:28.679 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-common-2:18.0.0-2872.ge0ab30a2.el8.x86_64 94/134 2023-03-09T14:35:29.004 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-03-09T14:35:29.264 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-common-2:18.0.0-2872.ge0ab30a2.el8.x86_64 94/134 2023-03-09T14:35:29.298 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-03-09T14:35:29.397 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-common-2:18.0.0-2872.ge0ab30a2.el8.x86_64 94/134 2023-03-09T14:35:29.529 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-03-09T14:35:29.617 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-common-2:18.0.0-2872.ge0ab30a2.el8.x86_64 94/134 2023-03-09T14:35:29.911 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-base-2:18.0.0-2872.ge0ab30a2.el8.x86_64 95/134 2023-03-09T14:35:29.988 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-base-2:18.0.0-2872.ge0ab30a2.el8.x86_64 95/134 2023-03-09T14:35:30.095 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-selinux-2:18.0.0-2872.ge0ab30a2.el8.x86_64 96/134 2023-03-09T14:35:30.209 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-common-2:18.0.0-2872.ge0ab30a2.el8.x86_64 94/134 2023-03-09T14:35:30.331 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-03-09T14:35:30.455 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-base-2:18.0.0-2872.ge0ab30a2.el8.x86_64 95/134 2023-03-09T14:35:30.498 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : librgw2-2:18.0.0-2872.ge0ab30a2.el8.x86_64 92/134 2023-03-09T14:35:30.538 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-base-2:18.0.0-2872.ge0ab30a2.el8.x86_64 95/134 2023-03-09T14:35:30.573 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librgw2-2:18.0.0-2872.ge0ab30a2.el8.x86_64 92/134 2023-03-09T14:35:30.708 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-selinux-2:18.0.0-2872.ge0ab30a2.el8.x86_64 96/134 2023-03-09T14:35:30.749 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : python3-rgw-2:18.0.0-2872.ge0ab30a2.el8.x86_64 93/134 2023-03-09T14:35:34.523 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-common-2:18.0.0-2872.ge0ab30a2.el8.x86_64 94/134 2023-03-09T14:35:34.748 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-common-2:18.0.0-2872.ge0ab30a2.el8.x86_64 94/134 2023-03-09T14:35:35.376 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-common-2:18.0.0-2872.ge0ab30a2.el8.x86_64 94/134 2023-03-09T14:35:36.097 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-base-2:18.0.0-2872.ge0ab30a2.el8.x86_64 95/134 2023-03-09T14:35:36.175 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-base-2:18.0.0-2872.ge0ab30a2.el8.x86_64 95/134 2023-03-09T14:35:36.281 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-selinux-2:18.0.0-2872.ge0ab30a2.el8.x86_64 96/134 2023-03-09T14:35:49.101 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-selinux-2:18.0.0-2872.ge0ab30a2.el8.x86_64 96/134 2023-03-09T14:35:49.101 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /sys 2023-03-09T14:35:49.101 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /proc 2023-03-09T14:35:49.102 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /dev 2023-03-09T14:35:49.102 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /run 2023-03-09T14:35:49.102 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /mnt 2023-03-09T14:35:49.102 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /var/tmp 2023-03-09T14:35:49.102 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /home 2023-03-09T14:35:49.102 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /tmp 2023-03-09T14:35:49.102 INFO:teuthology.orchestra.run.smithi181.stdout:skipping the directory /dev 2023-03-09T14:35:49.103 INFO:teuthology.orchestra.run.smithi181.stdout: 2023-03-09T14:35:49.459 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-volume-2:18.0.0-2872.ge0ab30a2.el8.noarch 97/134 2023-03-09T14:35:49.996 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-selinux-2:18.0.0-2872.ge0ab30a2.el8.x86_64 96/134 2023-03-09T14:35:49.997 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /sys 2023-03-09T14:35:49.997 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /proc 2023-03-09T14:35:49.997 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /dev 2023-03-09T14:35:49.997 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /run 2023-03-09T14:35:49.997 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /mnt 2023-03-09T14:35:49.997 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /var/tmp 2023-03-09T14:35:49.997 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /home 2023-03-09T14:35:49.998 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /tmp 2023-03-09T14:35:49.998 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /dev 2023-03-09T14:35:49.998 INFO:teuthology.orchestra.run.smithi122.stdout: 2023-03-09T14:35:50.389 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-volume-2:18.0.0-2872.ge0ab30a2.el8.noarch 97/134 2023-03-09T14:35:51.022 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-volume-2:18.0.0-2872.ge0ab30a2.el8.noarch 97/134 2023-03-09T14:35:51.215 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-osd-2:18.0.0-2872.ge0ab30a2.el8.x86_64 98/134 2023-03-09T14:35:51.441 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-osd-2:18.0.0-2872.ge0ab30a2.el8.x86_64 98/134 2023-03-09T14:35:51.617 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mds-2:18.0.0-2872.ge0ab30a2.el8.x86_64 99/134 2023-03-09T14:35:51.952 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-volume-2:18.0.0-2872.ge0ab30a2.el8.noarch 97/134 2023-03-09T14:35:52.071 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mds-2:18.0.0-2872.ge0ab30a2.el8.x86_64 99/134 2023-03-09T14:35:52.136 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-osd-2:18.0.0-2872.ge0ab30a2.el8.x86_64 98/134 2023-03-09T14:35:52.245 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mon-2:18.0.0-2872.ge0ab30a2.el8.x86_64 100/134 2023-03-09T14:35:52.262 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mon-2:18.0.0-2872.ge0ab30a2.el8.x86_64 100/134 2023-03-09T14:35:52.362 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-osd-2:18.0.0-2872.ge0ab30a2.el8.x86_64 98/134 2023-03-09T14:35:52.539 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mds-2:18.0.0-2872.ge0ab30a2.el8.x86_64 99/134 2023-03-09T14:35:53.000 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mds-2:18.0.0-2872.ge0ab30a2.el8.x86_64 99/134 2023-03-09T14:35:53.139 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mon-2:18.0.0-2872.ge0ab30a2.el8.x86_64 100/134 2023-03-09T14:35:53.155 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mon-2:18.0.0-2872.ge0ab30a2.el8.x86_64 100/134 2023-03-09T14:35:53.896 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: cephadm-2:18.0.0-2872.ge0ab30a2.el8.noarch 101/134 2023-03-09T14:35:54.084 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : cephadm-2:18.0.0-2872.ge0ab30a2.el8.noarch 101/134 2023-03-09T14:35:54.206 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-2872.ge0ab30a2.e 102/134 2023-03-09T14:35:54.463 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-2872.ge0ab30a2. 103/134 2023-03-09T14:35:54.641 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-2872.ge0ab30a2.el8.noa 104/134 2023-03-09T14:35:54.806 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: cephadm-2:18.0.0-2872.ge0ab30a2.el8.noarch 101/134 2023-03-09T14:35:54.854 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-2872.ge0ab30a2.el8.noa 104/134 2023-03-09T14:35:55.013 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : cephadm-2:18.0.0-2872.ge0ab30a2.el8.noarch 101/134 2023-03-09T14:35:55.098 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-2872.ge0ab30a2.el8.n 105/134 2023-03-09T14:35:55.136 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-2872.ge0ab30a2.e 102/134 2023-03-09T14:35:55.327 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-2872.ge0ab30a2. 103/134 2023-03-09T14:35:55.529 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-2872.ge0ab30a2.el8.noa 104/134 2023-03-09T14:35:55.747 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-2872.ge0ab30a2.el8.noa 104/134 2023-03-09T14:35:56.051 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-2872.ge0ab30a2.el8.n 105/134 2023-03-09T14:35:56.092 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-2872.ge0ab30a2.el8.n 105/134 2023-03-09T14:35:56.241 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-2872.ge0a 106/134 2023-03-09T14:35:56.270 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-selinux-2:18.0.0-2872.ge0ab30a2.el8.x86_64 96/134 2023-03-09T14:35:56.271 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /sys 2023-03-09T14:35:56.271 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /proc 2023-03-09T14:35:56.271 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /dev 2023-03-09T14:35:56.271 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /run 2023-03-09T14:35:56.271 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /mnt 2023-03-09T14:35:56.271 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /var/tmp 2023-03-09T14:35:56.272 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /home 2023-03-09T14:35:56.272 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /tmp 2023-03-09T14:35:56.272 INFO:teuthology.orchestra.run.smithi038.stdout:skipping the directory /dev 2023-03-09T14:35:56.272 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:35:56.332 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-2872.ge0a 106/134 2023-03-09T14:35:56.637 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-volume-2:18.0.0-2872.ge0ab30a2.el8.noarch 97/134 2023-03-09T14:35:56.700 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-2872.ge0ab30a2.el 107/134 2023-03-09T14:35:56.894 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-2:18.0.0-2872.ge0ab30a2.el8.x86_64 108/134 2023-03-09T14:35:56.929 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-2:18.0.0-2872.ge0ab30a2.el8.x86_64 108/134 2023-03-09T14:35:57.008 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-2872.ge0ab30a2.el8.n 105/134 2023-03-09T14:35:57.066 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-mgr-rook-2:18.0.0-2872.ge0ab30a2.el8.noarch 109/134 2023-03-09T14:35:57.077 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-2872.ge0ab30a2.el8.noarch 109/134 2023-03-09T14:35:57.179 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-2872.ge0a 106/134 2023-03-09T14:35:57.270 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-2872.ge0a 106/134 2023-03-09T14:35:57.636 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-2872.ge0ab30a2.el 107/134 2023-03-09T14:35:57.850 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-2:18.0.0-2872.ge0ab30a2.el8.x86_64 108/134 2023-03-09T14:35:57.884 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mgr-2:18.0.0-2872.ge0ab30a2.el8.x86_64 108/134 2023-03-09T14:35:58.054 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-rook-2:18.0.0-2872.ge0ab30a2.el8.noarch 109/134 2023-03-09T14:35:58.065 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-2872.ge0ab30a2.el8.noarch 109/134 2023-03-09T14:35:58.301 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-volume-2:18.0.0-2872.ge0ab30a2.el8.noarch 97/134 2023-03-09T14:35:58.325 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-2:18.0.0-2872.ge0ab30a2.el8.x86_64 110/134 2023-03-09T14:35:58.460 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-osd-2:18.0.0-2872.ge0ab30a2.el8.x86_64 98/134 2023-03-09T14:35:58.491 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-radosgw-2:18.0.0-2872.ge0ab30a2.el8.x86_64 111/134 2023-03-09T14:35:58.530 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-2872.ge0ab30a2.el8.x86_64 111/134 2023-03-09T14:35:58.697 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-osd-2:18.0.0-2872.ge0ab30a2.el8.x86_64 98/134 2023-03-09T14:35:58.708 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-2872.ge0ab3 112/134 2023-03-09T14:35:58.750 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-2872.ge0ab3 112/134 2023-03-09T14:35:58.908 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : cephfs-mirror-2:18.0.0-2872.ge0ab30a2.el8.x86_64 113/134 2023-03-09T14:35:58.970 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mds-2:18.0.0-2872.ge0ab30a2.el8.x86_64 99/134 2023-03-09T14:35:59.253 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-2872.ge0ab30a2.el8.x86_64 113/134 2023-03-09T14:35:59.337 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-2:18.0.0-2872.ge0ab30a2.el8.x86_64 110/134 2023-03-09T14:35:59.416 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : rbd-mirror-2:18.0.0-2872.ge0ab30a2.el8.x86_64 114/134 2023-03-09T14:35:59.468 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mds-2:18.0.0-2872.ge0ab30a2.el8.x86_64 99/134 2023-03-09T14:35:59.561 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-radosgw-2:18.0.0-2872.ge0ab30a2.el8.x86_64 111/134 2023-03-09T14:35:59.600 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-2872.ge0ab30a2.el8.x86_64 111/134 2023-03-09T14:35:59.611 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mon-2:18.0.0-2872.ge0ab30a2.el8.x86_64 100/134 2023-03-09T14:35:59.629 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mon-2:18.0.0-2872.ge0ab30a2.el8.x86_64 100/134 2023-03-09T14:35:59.796 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-2872.ge0ab3 112/134 2023-03-09T14:35:59.837 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-2872.ge0ab3 112/134 2023-03-09T14:35:59.996 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : cephfs-mirror-2:18.0.0-2872.ge0ab30a2.el8.x86_64 113/134 2023-03-09T14:36:00.352 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-2872.ge0ab30a2.el8.x86_64 113/134 2023-03-09T14:36:00.504 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : rbd-mirror-2:18.0.0-2872.ge0ab30a2.el8.x86_64 114/134 2023-03-09T14:36:01.338 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: cephadm-2:18.0.0-2872.ge0ab30a2.el8.noarch 101/134 2023-03-09T14:36:01.504 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : cephadm-2:18.0.0-2872.ge0ab30a2.el8.noarch 101/134 2023-03-09T14:36:01.617 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-2872.ge0ab30a2.e 102/134 2023-03-09T14:36:01.785 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-2872.ge0ab30a2. 103/134 2023-03-09T14:36:02.010 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-2872.ge0ab30a2.el8.noa 104/134 2023-03-09T14:36:02.233 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-2872.ge0ab30a2.el8.noa 104/134 2023-03-09T14:36:02.485 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-2872.ge0ab30a2.el8.n 105/134 2023-03-09T14:36:03.505 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-2872.ge0ab30a2.el8.n 105/134 2023-03-09T14:36:03.652 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-2872.ge0a 106/134 2023-03-09T14:36:03.746 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-2872.ge0a 106/134 2023-03-09T14:36:03.842 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: rbd-mirror-2:18.0.0-2872.ge0ab30a2.el8.x86_64 114/134 2023-03-09T14:36:04.104 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-2872.ge0ab30a2.el 107/134 2023-03-09T14:36:04.140 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-test-2:18.0.0-2872.ge0ab30a2.el8.x86_64 115/134 2023-03-09T14:36:04.306 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libtool-2.4.6-25.el8.x86_64 116/134 2023-03-09T14:36:04.349 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-2:18.0.0-2872.ge0ab30a2.el8.x86_64 108/134 2023-03-09T14:36:04.386 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-2:18.0.0-2872.ge0ab30a2.el8.x86_64 108/134 2023-03-09T14:36:04.499 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 116/134 2023-03-09T14:36:04.543 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-mgr-rook-2:18.0.0-2872.ge0ab30a2.el8.noarch 109/134 2023-03-09T14:36:04.555 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-2872.ge0ab30a2.el8.noarch 109/134 2023-03-09T14:36:04.922 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: rbd-mirror-2:18.0.0-2872.ge0ab30a2.el8.x86_64 114/134 2023-03-09T14:36:05.209 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-test-2:18.0.0-2872.ge0ab30a2.el8.x86_64 115/134 2023-03-09T14:36:05.367 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 117/134 2023-03-09T14:36:05.403 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libtool-2.4.6-25.el8.x86_64 116/134 2023-03-09T14:36:05.592 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 116/134 2023-03-09T14:36:05.687 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 118/134 2023-03-09T14:36:05.867 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : ceph-fuse-2:18.0.0-2872.ge0ab30a2.el8.x86_64 119/134 2023-03-09T14:36:05.896 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-2:18.0.0-2872.ge0ab30a2.el8.x86_64 110/134 2023-03-09T14:36:06.052 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-03-09T14:36:06.064 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-radosgw-2:18.0.0-2872.ge0ab30a2.el8.x86_64 111/134 2023-03-09T14:36:06.109 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-2872.ge0ab30a2.el8.x86_64 111/134 2023-03-09T14:36:06.193 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libcephfs-devel-2:18.0.0-2872.ge0ab30a2.el8.x86_ 121/134 2023-03-09T14:36:06.285 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-2872.ge0ab3 112/134 2023-03-09T14:36:06.324 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-03-09T14:36:06.328 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-2872.ge0ab3 112/134 2023-03-09T14:36:06.486 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : cephfs-mirror-2:18.0.0-2872.ge0ab30a2.el8.x86_64 113/134 2023-03-09T14:36:06.496 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 117/134 2023-03-09T14:36:06.834 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 118/134 2023-03-09T14:36:06.852 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-2872.ge0ab30a2.el8.x86_64 113/134 2023-03-09T14:36:07.010 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : rbd-mirror-2:18.0.0-2872.ge0ab30a2.el8.x86_64 114/134 2023-03-09T14:36:07.014 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-fuse-2:18.0.0-2872.ge0ab30a2.el8.x86_64 119/134 2023-03-09T14:36:07.199 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-03-09T14:36:07.425 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libcephfs-devel-2:18.0.0-2872.ge0ab30a2.el8.x86_ 121/134 2023-03-09T14:36:07.596 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-03-09T14:36:08.416 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-03-09T14:36:08.673 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-03-09T14:36:08.691 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-03-09T14:36:08.865 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : cephfs-top-2:18.0.0-2872.ge0ab30a2.el8.noarch 124/134 2023-03-09T14:36:09.097 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : rbd-fuse-2:18.0.0-2872.ge0ab30a2.el8.x86_64 125/134 2023-03-09T14:36:09.407 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : rbd-nbd-2:18.0.0-2872.ge0ab30a2.el8.x86_64 126/134 2023-03-09T14:36:09.724 INFO:teuthology.orchestra.run.smithi181.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-03-09T14:36:09.750 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-03-09T14:36:09.847 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-03-09T14:36:09.856 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-03-09T14:36:10.028 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-03-09T14:36:10.036 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 129/134 2023-03-09T14:36:10.045 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-03-09T14:36:10.214 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : flex-2.6.1-9.el8.x86_64 130/134 2023-03-09T14:36:10.275 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : cephfs-top-2:18.0.0-2872.ge0ab30a2.el8.noarch 124/134 2023-03-09T14:36:10.296 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 130/134 2023-03-09T14:36:10.472 INFO:teuthology.orchestra.run.smithi181.stdout: Installing : bison-3.0.4-10.el8.x86_64 131/134 2023-03-09T14:36:10.485 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 131/134 2023-03-09T14:36:10.486 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-03-09T14:36:10.503 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : rbd-fuse-2:18.0.0-2872.ge0ab30a2.el8.x86_64 125/134 2023-03-09T14:36:10.675 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-03-09T14:36:10.676 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-03-09T14:36:10.812 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : rbd-nbd-2:18.0.0-2872.ge0ab30a2.el8.x86_64 126/134 2023-03-09T14:36:10.928 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-03-09T14:36:11.129 INFO:teuthology.orchestra.run.smithi181.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-03-09T14:36:11.146 INFO:teuthology.orchestra.run.smithi122.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-03-09T14:36:11.311 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-03-09T14:36:11.320 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-03-09T14:36:11.417 INFO:teuthology.orchestra.run.smithi181.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-03-09T14:36:11.417 INFO:teuthology.orchestra.run.smithi181.stdout:[/usr/lib/tmpfiles.d/journal-nocow.conf:26] Failed to resolve specifier: uninitialized /etc detected, skipping 2023-03-09T14:36:11.417 INFO:teuthology.orchestra.run.smithi181.stdout:All rules containing unresolvable specifiers will be skipped. 2023-03-09T14:36:11.417 INFO:teuthology.orchestra.run.smithi181.stdout: 2023-03-09T14:36:11.525 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 129/134 2023-03-09T14:36:11.663 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-2:18.0.0-2872.ge0ab30a2.el8.x86_64 1/134 2023-03-09T14:36:11.663 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-base-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2/134 2023-03-09T14:36:11.664 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-common-2:18.0.0-2872.ge0ab30a2.el8.x86_64 3/134 2023-03-09T14:36:11.664 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-fuse-2:18.0.0-2872.ge0ab30a2.el8.x86_64 4/134 2023-03-09T14:36:11.664 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-2872.ge0ab3 5/134 2023-03-09T14:36:11.664 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mds-2:18.0.0-2872.ge0ab30a2.el8.x86_64 6/134 2023-03-09T14:36:11.664 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-2:18.0.0-2872.ge0ab30a2.el8.x86_64 7/134 2023-03-09T14:36:11.664 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mon-2:18.0.0-2872.ge0ab30a2.el8.x86_64 8/134 2023-03-09T14:36:11.664 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-osd-2:18.0.0-2872.ge0ab30a2.el8.x86_64 9/134 2023-03-09T14:36:11.665 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-radosgw-2:18.0.0-2872.ge0ab30a2.el8.x86_64 10/134 2023-03-09T14:36:11.665 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-selinux-2:18.0.0-2872.ge0ab30a2.el8.x86_64 11/134 2023-03-09T14:36:11.665 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-test-2:18.0.0-2872.ge0ab30a2.el8.x86_64 12/134 2023-03-09T14:36:11.665 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : cephfs-mirror-2:18.0.0-2872.ge0ab30a2.el8.x86_64 13/134 2023-03-09T14:36:11.665 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libcephfs-devel-2:18.0.0-2872.ge0ab30a2.el8.x86_ 14/134 2023-03-09T14:36:11.665 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libcephfs2-2:18.0.0-2872.ge0ab30a2.el8.x86_64 15/134 2023-03-09T14:36:11.665 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libcephsqlite-2:18.0.0-2872.ge0ab30a2.el8.x86_64 16/134 2023-03-09T14:36:11.666 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librados-devel-2:18.0.0-2872.ge0ab30a2.el8.x86_6 17/134 2023-03-09T14:36:11.666 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libradosstriper1-2:18.0.0-2872.ge0ab30a2.el8.x86 18/134 2023-03-09T14:36:11.666 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librgw2-2:18.0.0-2872.ge0ab30a2.el8.x86_64 19/134 2023-03-09T14:36:11.666 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-ceph-argparse-2:18.0.0-2872.ge0ab30a2.el 20/134 2023-03-09T14:36:11.666 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-ceph-common-2:18.0.0-2872.ge0ab30a2.el8. 21/134 2023-03-09T14:36:11.666 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cephfs-2:18.0.0-2872.ge0ab30a2.el8.x86_6 22/134 2023-03-09T14:36:11.666 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rados-2:18.0.0-2872.ge0ab30a2.el8.x86_64 23/134 2023-03-09T14:36:11.667 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rbd-2:18.0.0-2872.ge0ab30a2.el8.x86_64 24/134 2023-03-09T14:36:11.667 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rgw-2:18.0.0-2872.ge0ab30a2.el8.x86_64 25/134 2023-03-09T14:36:11.667 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : rbd-fuse-2:18.0.0-2872.ge0ab30a2.el8.x86_64 26/134 2023-03-09T14:36:11.667 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : rbd-mirror-2:18.0.0-2872.ge0ab30a2.el8.x86_64 27/134 2023-03-09T14:36:11.667 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : rbd-nbd-2:18.0.0-2872.ge0ab30a2.el8.x86_64 28/134 2023-03-09T14:36:11.667 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-2872.ge0ab30a2. 29/134 2023-03-09T14:36:11.667 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-2872.ge0ab30a2.el8.noa 30/134 2023-03-09T14:36:11.667 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-2872.ge0ab30a2.el8.n 31/134 2023-03-09T14:36:11.668 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-2872.ge0a 32/134 2023-03-09T14:36:11.668 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-2872.ge0ab30a2.el 33/134 2023-03-09T14:36:11.668 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-mgr-rook-2:18.0.0-2872.ge0ab30a2.el8.noarch 34/134 2023-03-09T14:36:11.668 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-2872.ge0ab30a2.e 35/134 2023-03-09T14:36:11.668 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : ceph-volume-2:18.0.0-2872.ge0ab30a2.el8.noarch 36/134 2023-03-09T14:36:11.668 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : cephadm-2:18.0.0-2872.ge0ab30a2.el8.noarch 37/134 2023-03-09T14:36:11.668 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : cephfs-top-2:18.0.0-2872.ge0ab30a2.el8.noarch 38/134 2023-03-09T14:36:11.668 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-03-09T14:36:11.669 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-03-09T14:36:11.669 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-03-09T14:36:11.669 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-03-09T14:36:11.669 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-03-09T14:36:11.669 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-03-09T14:36:11.669 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-03-09T14:36:11.669 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-03-09T14:36:11.670 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-03-09T14:36:11.670 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-03-09T14:36:11.670 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-03-09T14:36:11.670 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-03-09T14:36:11.670 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-03-09T14:36:11.670 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-03-09T14:36:11.670 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-03-09T14:36:11.670 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-03-09T14:36:11.671 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-03-09T14:36:11.671 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-03-09T14:36:11.671 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-03-09T14:36:11.671 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-03-09T14:36:11.671 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-03-09T14:36:11.671 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/134 2023-03-09T14:36:11.671 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-03-09T14:36:11.671 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-03-09T14:36:11.672 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-03-09T14:36:11.672 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-03-09T14:36:11.672 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 65/134 2023-03-09T14:36:11.672 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-03-09T14:36:11.672 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-03-09T14:36:11.672 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-03-09T14:36:11.672 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-03-09T14:36:11.672 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-03-09T14:36:11.673 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-03-09T14:36:11.673 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 72/134 2023-03-09T14:36:11.673 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 73/134 2023-03-09T14:36:11.673 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 74/134 2023-03-09T14:36:11.673 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 75/134 2023-03-09T14:36:11.673 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 76/134 2023-03-09T14:36:11.673 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 77/134 2023-03-09T14:36:11.674 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : bison-3.0.4-10.el8.x86_64 78/134 2023-03-09T14:36:11.674 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 79/134 2023-03-09T14:36:11.674 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 80/134 2023-03-09T14:36:11.674 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 81/134 2023-03-09T14:36:11.674 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : automake-1.16.1-7.el8.noarch 82/134 2023-03-09T14:36:11.674 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 83/134 2023-03-09T14:36:11.674 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 84/134 2023-03-09T14:36:11.674 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : flex-2.6.1-9.el8.x86_64 85/134 2023-03-09T14:36:11.675 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 86/134 2023-03-09T14:36:11.675 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 87/134 2023-03-09T14:36:11.675 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 88/134 2023-03-09T14:36:11.675 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 89/134 2023-03-09T14:36:11.675 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 90/134 2023-03-09T14:36:11.675 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 91/134 2023-03-09T14:36:11.675 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 92/134 2023-03-09T14:36:11.676 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 93/134 2023-03-09T14:36:11.676 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 94/134 2023-03-09T14:36:11.676 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 95/134 2023-03-09T14:36:11.676 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libpq-13.5-1.el8.x86_64 96/134 2023-03-09T14:36:11.676 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 97/134 2023-03-09T14:36:11.676 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : jq-1.6-3.el8.x86_64 98/134 2023-03-09T14:36:11.676 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 99/134 2023-03-09T14:36:11.677 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 100/134 2023-03-09T14:36:11.677 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 101/134 2023-03-09T14:36:11.677 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 102/134 2023-03-09T14:36:11.677 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 103/134 2023-03-09T14:36:11.677 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 104/134 2023-03-09T14:36:11.677 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 105/134 2023-03-09T14:36:11.677 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 106/134 2023-03-09T14:36:11.677 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 107/134 2023-03-09T14:36:11.678 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 108/134 2023-03-09T14:36:11.678 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 109/134 2023-03-09T14:36:11.678 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 110/134 2023-03-09T14:36:11.678 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 111/134 2023-03-09T14:36:11.678 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 112/134 2023-03-09T14:36:11.678 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 113/134 2023-03-09T14:36:11.678 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 114/134 2023-03-09T14:36:11.679 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 115/134 2023-03-09T14:36:11.679 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 116/134 2023-03-09T14:36:11.679 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 117/134 2023-03-09T14:36:11.679 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 118/134 2023-03-09T14:36:11.679 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 119/134 2023-03-09T14:36:11.679 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-03-09T14:36:11.679 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 121/134 2023-03-09T14:36:11.679 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-03-09T14:36:11.680 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 123/134 2023-03-09T14:36:11.680 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 124/134 2023-03-09T14:36:11.680 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 125/134 2023-03-09T14:36:11.680 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 126/134 2023-03-09T14:36:11.680 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 127/134 2023-03-09T14:36:11.680 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-03-09T14:36:11.680 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librados2-2:18.0.0-2872.ge0ab30a2.el8.x86_64 129/134 2023-03-09T14:36:11.680 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-03-09T14:36:11.681 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librbd1-2:18.0.0-2872.ge0ab30a2.el8.x86_64 131/134 2023-03-09T14:36:11.681 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-03-09T14:36:11.681 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-03-09T14:36:11.731 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: rbd-mirror-2:18.0.0-2872.ge0ab30a2.el8.x86_64 114/134 2023-03-09T14:36:11.736 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : flex-2.6.1-9.el8.x86_64 130/134 2023-03-09T14:36:11.822 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 130/134 2023-03-09T14:36:12.003 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : bison-3.0.4-10.el8.x86_64 131/134 2023-03-09T14:36:12.007 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-test-2:18.0.0-2872.ge0ab30a2.el8.x86_64 115/134 2023-03-09T14:36:12.015 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 131/134 2023-03-09T14:36:12.015 INFO:teuthology.orchestra.run.smithi122.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-03-09T14:36:12.184 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libtool-2.4.6-25.el8.x86_64 116/134 2023-03-09T14:36:12.247 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-03-09T14:36:12.248 INFO:teuthology.orchestra.run.smithi122.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-03-09T14:36:12.390 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 116/134 2023-03-09T14:36:12.492 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-03-09T14:36:12.693 INFO:teuthology.orchestra.run.smithi122.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-03-09T14:36:13.014 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-03-09T14:36:13.014 INFO:teuthology.orchestra.run.smithi122.stdout:[/usr/lib/tmpfiles.d/journal-nocow.conf:26] Failed to resolve specifier: uninitialized /etc detected, skipping 2023-03-09T14:36:13.014 INFO:teuthology.orchestra.run.smithi122.stdout:All rules containing unresolvable specifiers will be skipped. 2023-03-09T14:36:13.014 INFO:teuthology.orchestra.run.smithi122.stdout: 2023-03-09T14:36:13.232 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-2:18.0.0-2872.ge0ab30a2.el8.x86_64 1/134 2023-03-09T14:36:13.233 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-base-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2/134 2023-03-09T14:36:13.233 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-common-2:18.0.0-2872.ge0ab30a2.el8.x86_64 3/134 2023-03-09T14:36:13.233 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-fuse-2:18.0.0-2872.ge0ab30a2.el8.x86_64 4/134 2023-03-09T14:36:13.233 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-2872.ge0ab3 5/134 2023-03-09T14:36:13.233 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mds-2:18.0.0-2872.ge0ab30a2.el8.x86_64 6/134 2023-03-09T14:36:13.233 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-2:18.0.0-2872.ge0ab30a2.el8.x86_64 7/134 2023-03-09T14:36:13.234 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mon-2:18.0.0-2872.ge0ab30a2.el8.x86_64 8/134 2023-03-09T14:36:13.234 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-osd-2:18.0.0-2872.ge0ab30a2.el8.x86_64 9/134 2023-03-09T14:36:13.234 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-radosgw-2:18.0.0-2872.ge0ab30a2.el8.x86_64 10/134 2023-03-09T14:36:13.234 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-selinux-2:18.0.0-2872.ge0ab30a2.el8.x86_64 11/134 2023-03-09T14:36:13.234 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-test-2:18.0.0-2872.ge0ab30a2.el8.x86_64 12/134 2023-03-09T14:36:13.234 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : cephfs-mirror-2:18.0.0-2872.ge0ab30a2.el8.x86_64 13/134 2023-03-09T14:36:13.234 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libcephfs-devel-2:18.0.0-2872.ge0ab30a2.el8.x86_ 14/134 2023-03-09T14:36:13.234 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libcephfs2-2:18.0.0-2872.ge0ab30a2.el8.x86_64 15/134 2023-03-09T14:36:13.235 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libcephsqlite-2:18.0.0-2872.ge0ab30a2.el8.x86_64 16/134 2023-03-09T14:36:13.235 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librados-devel-2:18.0.0-2872.ge0ab30a2.el8.x86_6 17/134 2023-03-09T14:36:13.235 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libradosstriper1-2:18.0.0-2872.ge0ab30a2.el8.x86 18/134 2023-03-09T14:36:13.235 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librgw2-2:18.0.0-2872.ge0ab30a2.el8.x86_64 19/134 2023-03-09T14:36:13.235 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-ceph-argparse-2:18.0.0-2872.ge0ab30a2.el 20/134 2023-03-09T14:36:13.235 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-ceph-common-2:18.0.0-2872.ge0ab30a2.el8. 21/134 2023-03-09T14:36:13.235 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cephfs-2:18.0.0-2872.ge0ab30a2.el8.x86_6 22/134 2023-03-09T14:36:13.236 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-rados-2:18.0.0-2872.ge0ab30a2.el8.x86_64 23/134 2023-03-09T14:36:13.236 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-rbd-2:18.0.0-2872.ge0ab30a2.el8.x86_64 24/134 2023-03-09T14:36:13.242 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-rgw-2:18.0.0-2872.ge0ab30a2.el8.x86_64 25/134 2023-03-09T14:36:13.243 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : rbd-fuse-2:18.0.0-2872.ge0ab30a2.el8.x86_64 26/134 2023-03-09T14:36:13.243 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : rbd-mirror-2:18.0.0-2872.ge0ab30a2.el8.x86_64 27/134 2023-03-09T14:36:13.243 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : rbd-nbd-2:18.0.0-2872.ge0ab30a2.el8.x86_64 28/134 2023-03-09T14:36:13.243 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-2872.ge0ab30a2. 29/134 2023-03-09T14:36:13.243 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-2872.ge0ab30a2.el8.noa 30/134 2023-03-09T14:36:13.243 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-2872.ge0ab30a2.el8.n 31/134 2023-03-09T14:36:13.244 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-2872.ge0a 32/134 2023-03-09T14:36:13.244 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-2872.ge0ab30a2.el 33/134 2023-03-09T14:36:13.244 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-rook-2:18.0.0-2872.ge0ab30a2.el8.noarch 34/134 2023-03-09T14:36:13.244 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-2872.ge0ab30a2.e 35/134 2023-03-09T14:36:13.244 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-volume-2:18.0.0-2872.ge0ab30a2.el8.noarch 36/134 2023-03-09T14:36:13.244 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : cephadm-2:18.0.0-2872.ge0ab30a2.el8.noarch 37/134 2023-03-09T14:36:13.244 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : cephfs-top-2:18.0.0-2872.ge0ab30a2.el8.noarch 38/134 2023-03-09T14:36:13.244 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-03-09T14:36:13.245 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-03-09T14:36:13.245 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-03-09T14:36:13.245 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-03-09T14:36:13.245 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-03-09T14:36:13.245 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-03-09T14:36:13.245 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-03-09T14:36:13.245 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-03-09T14:36:13.246 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-03-09T14:36:13.246 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-03-09T14:36:13.246 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-03-09T14:36:13.246 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-03-09T14:36:13.246 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-03-09T14:36:13.246 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-03-09T14:36:13.246 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-03-09T14:36:13.246 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-03-09T14:36:13.247 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-03-09T14:36:13.247 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-03-09T14:36:13.247 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-03-09T14:36:13.247 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-03-09T14:36:13.247 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-03-09T14:36:13.247 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/134 2023-03-09T14:36:13.247 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-03-09T14:36:13.247 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-03-09T14:36:13.248 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-03-09T14:36:13.248 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-03-09T14:36:13.248 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 65/134 2023-03-09T14:36:13.248 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-03-09T14:36:13.248 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-03-09T14:36:13.248 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-03-09T14:36:13.248 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-03-09T14:36:13.248 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-03-09T14:36:13.249 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-03-09T14:36:13.249 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 72/134 2023-03-09T14:36:13.249 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 73/134 2023-03-09T14:36:13.249 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 74/134 2023-03-09T14:36:13.249 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 75/134 2023-03-09T14:36:13.249 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 76/134 2023-03-09T14:36:13.249 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 77/134 2023-03-09T14:36:13.250 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : bison-3.0.4-10.el8.x86_64 78/134 2023-03-09T14:36:13.250 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 79/134 2023-03-09T14:36:13.250 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 80/134 2023-03-09T14:36:13.250 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 81/134 2023-03-09T14:36:13.250 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : automake-1.16.1-7.el8.noarch 82/134 2023-03-09T14:36:13.250 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 83/134 2023-03-09T14:36:13.250 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 84/134 2023-03-09T14:36:13.250 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : flex-2.6.1-9.el8.x86_64 85/134 2023-03-09T14:36:13.251 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 86/134 2023-03-09T14:36:13.251 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 87/134 2023-03-09T14:36:13.251 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 88/134 2023-03-09T14:36:13.251 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 89/134 2023-03-09T14:36:13.251 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 90/134 2023-03-09T14:36:13.251 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 91/134 2023-03-09T14:36:13.251 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 92/134 2023-03-09T14:36:13.252 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 93/134 2023-03-09T14:36:13.252 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 94/134 2023-03-09T14:36:13.252 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 95/134 2023-03-09T14:36:13.252 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libpq-13.5-1.el8.x86_64 96/134 2023-03-09T14:36:13.252 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 97/134 2023-03-09T14:36:13.252 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : jq-1.6-3.el8.x86_64 98/134 2023-03-09T14:36:13.252 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 99/134 2023-03-09T14:36:13.252 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 100/134 2023-03-09T14:36:13.253 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 101/134 2023-03-09T14:36:13.253 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 102/134 2023-03-09T14:36:13.253 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 103/134 2023-03-09T14:36:13.253 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 104/134 2023-03-09T14:36:13.253 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 105/134 2023-03-09T14:36:13.253 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 106/134 2023-03-09T14:36:13.253 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 107/134 2023-03-09T14:36:13.253 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 108/134 2023-03-09T14:36:13.254 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 109/134 2023-03-09T14:36:13.254 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 110/134 2023-03-09T14:36:13.254 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 111/134 2023-03-09T14:36:13.254 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 112/134 2023-03-09T14:36:13.254 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 113/134 2023-03-09T14:36:13.254 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 114/134 2023-03-09T14:36:13.254 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 115/134 2023-03-09T14:36:13.255 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 116/134 2023-03-09T14:36:13.255 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 117/134 2023-03-09T14:36:13.255 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 118/134 2023-03-09T14:36:13.255 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 119/134 2023-03-09T14:36:13.255 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-03-09T14:36:13.255 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 121/134 2023-03-09T14:36:13.255 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-03-09T14:36:13.255 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 123/134 2023-03-09T14:36:13.256 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 124/134 2023-03-09T14:36:13.256 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 125/134 2023-03-09T14:36:13.256 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 126/134 2023-03-09T14:36:13.256 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 127/134 2023-03-09T14:36:13.256 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-03-09T14:36:13.256 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librados2-2:18.0.0-2872.ge0ab30a2.el8.x86_64 129/134 2023-03-09T14:36:13.256 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-03-09T14:36:13.256 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librbd1-2:18.0.0-2872.ge0ab30a2.el8.x86_64 131/134 2023-03-09T14:36:13.257 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-03-09T14:36:13.257 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-03-09T14:36:13.394 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 117/134 2023-03-09T14:36:13.745 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 118/134 2023-03-09T14:36:13.929 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : ceph-fuse-2:18.0.0-2872.ge0ab30a2.el8.x86_64 119/134 2023-03-09T14:36:14.105 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-03-09T14:36:14.126 INFO:teuthology.orchestra.run.smithi181.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-03-09T14:36:14.127 INFO:teuthology.orchestra.run.smithi181.stdout:Last metadata expiration check: 0:01:51 ago on Thu 09 Mar 2023 02:34:23 PM UTC. 2023-03-09T14:36:14.272 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libcephfs-devel-2:18.0.0-2872.ge0ab30a2.el8.x86_ 121/134 2023-03-09T14:36:14.452 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-03-09T14:36:15.734 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-03-09T14:36:15.734 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:52 ago on Thu 09 Mar 2023 02:34:23 PM UTC. 2023-03-09T14:36:16.710 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-03-09T14:36:17.034 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-03-09T14:36:17.052 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-03-09T14:36:17.284 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : cephfs-top-2:18.0.0-2872.ge0ab30a2.el8.noarch 124/134 2023-03-09T14:36:17.384 INFO:teuthology.orchestra.run.smithi181.stdout:Installed products updated. 2023-03-09T14:36:17.485 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : rbd-fuse-2:18.0.0-2872.ge0ab30a2.el8.x86_64 125/134 2023-03-09T14:36:17.597 INFO:teuthology.orchestra.run.smithi181.stdout: 2023-03-09T14:36:17.597 INFO:teuthology.orchestra.run.smithi181.stdout:Upgraded: 2023-03-09T14:36:17.597 INFO:teuthology.orchestra.run.smithi181.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-03-09T14:36:17.598 INFO:teuthology.orchestra.run.smithi181.stdout: librados2-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:17.598 INFO:teuthology.orchestra.run.smithi181.stdout: librbd1-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:17.598 INFO:teuthology.orchestra.run.smithi181.stdout:Installed: 2023-03-09T14:36:17.598 INFO:teuthology.orchestra.run.smithi181.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-03-09T14:36:17.598 INFO:teuthology.orchestra.run.smithi181.stdout: automake-1.16.1-7.el8.noarch 2023-03-09T14:36:17.598 INFO:teuthology.orchestra.run.smithi181.stdout: bison-3.0.4-10.el8.x86_64 2023-03-09T14:36:17.599 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:17.599 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-base-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:17.599 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-common-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:17.599 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-fuse-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:17.599 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-grafana-dashboards-2:18.0.0-2872.ge0ab30a2.el8.noarch 2023-03-09T14:36:17.599 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-immutable-object-cache-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:17.599 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mds-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:17.600 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:17.600 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-cephadm-2:18.0.0-2872.ge0ab30a2.el8.noarch 2023-03-09T14:36:17.600 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-dashboard-2:18.0.0-2872.ge0ab30a2.el8.noarch 2023-03-09T14:36:17.600 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-diskprediction-local-2:18.0.0-2872.ge0ab30a2.el8.noarch 2023-03-09T14:36:17.600 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-modules-core-2:18.0.0-2872.ge0ab30a2.el8.noarch 2023-03-09T14:36:17.600 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mgr-rook-2:18.0.0-2872.ge0ab30a2.el8.noarch 2023-03-09T14:36:17.600 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-mon-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:17.601 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-osd-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:17.601 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-prometheus-alerts-2:18.0.0-2872.ge0ab30a2.el8.noarch 2023-03-09T14:36:17.601 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-radosgw-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:17.601 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-selinux-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:17.601 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-test-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:17.601 INFO:teuthology.orchestra.run.smithi181.stdout: ceph-volume-2:18.0.0-2872.ge0ab30a2.el8.noarch 2023-03-09T14:36:17.601 INFO:teuthology.orchestra.run.smithi181.stdout: cephadm-2:18.0.0-2872.ge0ab30a2.el8.noarch 2023-03-09T14:36:17.602 INFO:teuthology.orchestra.run.smithi181.stdout: cephfs-mirror-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:17.602 INFO:teuthology.orchestra.run.smithi181.stdout: cephfs-top-2:18.0.0-2872.ge0ab30a2.el8.noarch 2023-03-09T14:36:17.602 INFO:teuthology.orchestra.run.smithi181.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-03-09T14:36:17.602 INFO:teuthology.orchestra.run.smithi181.stdout: flex-2.6.1-9.el8.x86_64 2023-03-09T14:36:17.602 INFO:teuthology.orchestra.run.smithi181.stdout: fuse-2.9.7-15.el8.x86_64 2023-03-09T14:36:17.602 INFO:teuthology.orchestra.run.smithi181.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-03-09T14:36:17.602 INFO:teuthology.orchestra.run.smithi181.stdout: jq-1.6-3.el8.x86_64 2023-03-09T14:36:17.603 INFO:teuthology.orchestra.run.smithi181.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-03-09T14:36:17.603 INFO:teuthology.orchestra.run.smithi181.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-03-09T14:36:17.603 INFO:teuthology.orchestra.run.smithi181.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-03-09T14:36:17.603 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs-devel-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:17.603 INFO:teuthology.orchestra.run.smithi181.stdout: libcephfs2-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:17.603 INFO:teuthology.orchestra.run.smithi181.stdout: libcephsqlite-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:17.603 INFO:teuthology.orchestra.run.smithi181.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-03-09T14:36:17.604 INFO:teuthology.orchestra.run.smithi181.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-03-09T14:36:17.604 INFO:teuthology.orchestra.run.smithi181.stdout: liboath-2.6.2-3.el8.x86_64 2023-03-09T14:36:17.604 INFO:teuthology.orchestra.run.smithi181.stdout: libpq-13.5-1.el8.x86_64 2023-03-09T14:36:17.604 INFO:teuthology.orchestra.run.smithi181.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-03-09T14:36:17.604 INFO:teuthology.orchestra.run.smithi181.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-03-09T14:36:17.604 INFO:teuthology.orchestra.run.smithi181.stdout: librados-devel-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:17.604 INFO:teuthology.orchestra.run.smithi181.stdout: libradosstriper1-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:17.605 INFO:teuthology.orchestra.run.smithi181.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-03-09T14:36:17.605 INFO:teuthology.orchestra.run.smithi181.stdout: librgw2-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:17.605 INFO:teuthology.orchestra.run.smithi181.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-03-09T14:36:17.605 INFO:teuthology.orchestra.run.smithi181.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-03-09T14:36:17.605 INFO:teuthology.orchestra.run.smithi181.stdout: libtool-2.4.6-25.el8.x86_64 2023-03-09T14:36:17.605 INFO:teuthology.orchestra.run.smithi181.stdout: libunwind-1.3.1-3.el8.x86_64 2023-03-09T14:36:17.605 INFO:teuthology.orchestra.run.smithi181.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-03-09T14:36:17.606 INFO:teuthology.orchestra.run.smithi181.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-03-09T14:36:17.606 INFO:teuthology.orchestra.run.smithi181.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-03-09T14:36:17.606 INFO:teuthology.orchestra.run.smithi181.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-03-09T14:36:17.606 INFO:teuthology.orchestra.run.smithi181.stdout: openblas-0.3.15-3.el8.x86_64 2023-03-09T14:36:17.606 INFO:teuthology.orchestra.run.smithi181.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-03-09T14:36:17.606 INFO:teuthology.orchestra.run.smithi181.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-03-09T14:36:17.606 INFO:teuthology.orchestra.run.smithi181.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-03-09T14:36:17.607 INFO:teuthology.orchestra.run.smithi181.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-03-09T14:36:17.607 INFO:teuthology.orchestra.run.smithi181.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-03-09T14:36:17.607 INFO:teuthology.orchestra.run.smithi181.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-03-09T14:36:17.607 INFO:teuthology.orchestra.run.smithi181.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-03-09T14:36:17.607 INFO:teuthology.orchestra.run.smithi181.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-03-09T14:36:17.607 INFO:teuthology.orchestra.run.smithi181.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-03-09T14:36:17.607 INFO:teuthology.orchestra.run.smithi181.stdout: python3-babel-2.5.1-7.el8.noarch 2023-03-09T14:36:17.608 INFO:teuthology.orchestra.run.smithi181.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-03-09T14:36:17.608 INFO:teuthology.orchestra.run.smithi181.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-03-09T14:36:17.608 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-03-09T14:36:17.608 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-argparse-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:17.608 INFO:teuthology.orchestra.run.smithi181.stdout: python3-ceph-common-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:17.608 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cephfs-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:17.608 INFO:teuthology.orchestra.run.smithi181.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-03-09T14:36:17.609 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-03-09T14:36:17.609 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-03-09T14:36:17.609 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-03-09T14:36:17.609 INFO:teuthology.orchestra.run.smithi181.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-03-09T14:36:17.609 INFO:teuthology.orchestra.run.smithi181.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-03-09T14:36:17.609 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-6.2-6.el8.noarch 2023-03-09T14:36:17.610 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-03-09T14:36:17.610 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-03-09T14:36:17.610 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-03-09T14:36:17.610 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-03-09T14:36:17.610 INFO:teuthology.orchestra.run.smithi181.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-03-09T14:36:17.610 INFO:teuthology.orchestra.run.smithi181.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-03-09T14:36:17.611 INFO:teuthology.orchestra.run.smithi181.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-03-09T14:36:17.611 INFO:teuthology.orchestra.run.smithi181.stdout: python3-mako-1.0.6-13.el8.noarch 2023-03-09T14:36:17.611 INFO:teuthology.orchestra.run.smithi181.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-03-09T14:36:17.611 INFO:teuthology.orchestra.run.smithi181.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-03-09T14:36:17.611 INFO:teuthology.orchestra.run.smithi181.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-03-09T14:36:17.611 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-03-09T14:36:17.611 INFO:teuthology.orchestra.run.smithi181.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-03-09T14:36:17.611 INFO:teuthology.orchestra.run.smithi181.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-03-09T14:36:17.612 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-03-09T14:36:17.612 INFO:teuthology.orchestra.run.smithi181.stdout: python3-portend-2.6-1.el8.noarch 2023-03-09T14:36:17.612 INFO:teuthology.orchestra.run.smithi181.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-03-09T14:36:17.612 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-03-09T14:36:17.612 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-03-09T14:36:17.612 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-03-09T14:36:17.612 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pycparser-2.14-14.el8.noarch 2023-03-09T14:36:17.613 INFO:teuthology.orchestra.run.smithi181.stdout: python3-pytz-2017.2-9.el8.noarch 2023-03-09T14:36:17.613 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rados-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:17.613 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rbd-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:17.613 INFO:teuthology.orchestra.run.smithi181.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-03-09T14:36:17.613 INFO:teuthology.orchestra.run.smithi181.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-03-09T14:36:17.613 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rgw-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:17.613 INFO:teuthology.orchestra.run.smithi181.stdout: python3-routes-2.4.1-12.el8.noarch 2023-03-09T14:36:17.614 INFO:teuthology.orchestra.run.smithi181.stdout: python3-rsa-4.8-1.el8.noarch 2023-03-09T14:36:17.614 INFO:teuthology.orchestra.run.smithi181.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-03-09T14:36:17.614 INFO:teuthology.orchestra.run.smithi181.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-03-09T14:36:17.614 INFO:teuthology.orchestra.run.smithi181.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-03-09T14:36:17.614 INFO:teuthology.orchestra.run.smithi181.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-03-09T14:36:17.614 INFO:teuthology.orchestra.run.smithi181.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-03-09T14:36:17.614 INFO:teuthology.orchestra.run.smithi181.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2023-03-09T14:36:17.615 INFO:teuthology.orchestra.run.smithi181.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-03-09T14:36:17.615 INFO:teuthology.orchestra.run.smithi181.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-03-09T14:36:17.615 INFO:teuthology.orchestra.run.smithi181.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-03-09T14:36:17.615 INFO:teuthology.orchestra.run.smithi181.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-03-09T14:36:17.615 INFO:teuthology.orchestra.run.smithi181.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-03-09T14:36:17.615 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-fuse-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:17.615 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-mirror-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:17.616 INFO:teuthology.orchestra.run.smithi181.stdout: rbd-nbd-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:17.616 INFO:teuthology.orchestra.run.smithi181.stdout: socat-1.7.4.1-1.el8.x86_64 2023-03-09T14:36:17.616 INFO:teuthology.orchestra.run.smithi181.stdout: thrift-0.13.0-2.el8.x86_64 2023-03-09T14:36:17.616 INFO:teuthology.orchestra.run.smithi181.stdout: uuid-1.6.2-43.el8.x86_64 2023-03-09T14:36:17.616 INFO:teuthology.orchestra.run.smithi181.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-03-09T14:36:17.616 INFO:teuthology.orchestra.run.smithi181.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-03-09T14:36:17.617 INFO:teuthology.orchestra.run.smithi181.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-03-09T14:36:17.617 INFO:teuthology.orchestra.run.smithi181.stdout: 2023-03-09T14:36:17.617 INFO:teuthology.orchestra.run.smithi181.stdout:Complete! 2023-03-09T14:36:17.770 DEBUG:teuthology.parallel:result is None 2023-03-09T14:36:17.832 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : rbd-nbd-2:18.0.0-2872.ge0ab30a2.el8.x86_64 126/134 2023-03-09T14:36:18.119 INFO:teuthology.orchestra.run.smithi038.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-03-09T14:36:18.258 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-03-09T14:36:18.267 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-03-09T14:36:18.454 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 129/134 2023-03-09T14:36:18.658 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : flex-2.6.1-9.el8.x86_64 130/134 2023-03-09T14:36:18.744 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 130/134 2023-03-09T14:36:18.867 INFO:teuthology.orchestra.run.smithi122.stdout:Installed products updated. 2023-03-09T14:36:18.942 INFO:teuthology.orchestra.run.smithi038.stdout: Installing : bison-3.0.4-10.el8.x86_64 131/134 2023-03-09T14:36:18.956 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 131/134 2023-03-09T14:36:18.957 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-03-09T14:36:19.080 INFO:teuthology.orchestra.run.smithi122.stdout: 2023-03-09T14:36:19.080 INFO:teuthology.orchestra.run.smithi122.stdout:Upgraded: 2023-03-09T14:36:19.080 INFO:teuthology.orchestra.run.smithi122.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-03-09T14:36:19.080 INFO:teuthology.orchestra.run.smithi122.stdout: librados2-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:19.081 INFO:teuthology.orchestra.run.smithi122.stdout: librbd1-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:19.081 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2023-03-09T14:36:19.081 INFO:teuthology.orchestra.run.smithi122.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-03-09T14:36:19.081 INFO:teuthology.orchestra.run.smithi122.stdout: automake-1.16.1-7.el8.noarch 2023-03-09T14:36:19.081 INFO:teuthology.orchestra.run.smithi122.stdout: bison-3.0.4-10.el8.x86_64 2023-03-09T14:36:19.081 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:19.082 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-base-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:19.082 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-common-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:19.082 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-fuse-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:19.082 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-grafana-dashboards-2:18.0.0-2872.ge0ab30a2.el8.noarch 2023-03-09T14:36:19.082 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-immutable-object-cache-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:19.082 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mds-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:19.082 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:19.083 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-cephadm-2:18.0.0-2872.ge0ab30a2.el8.noarch 2023-03-09T14:36:19.083 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-dashboard-2:18.0.0-2872.ge0ab30a2.el8.noarch 2023-03-09T14:36:19.083 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-diskprediction-local-2:18.0.0-2872.ge0ab30a2.el8.noarch 2023-03-09T14:36:19.083 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-modules-core-2:18.0.0-2872.ge0ab30a2.el8.noarch 2023-03-09T14:36:19.083 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-rook-2:18.0.0-2872.ge0ab30a2.el8.noarch 2023-03-09T14:36:19.083 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mon-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:19.083 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-osd-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:19.084 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-prometheus-alerts-2:18.0.0-2872.ge0ab30a2.el8.noarch 2023-03-09T14:36:19.084 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-radosgw-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:19.084 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-selinux-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:19.084 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-test-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:19.084 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-volume-2:18.0.0-2872.ge0ab30a2.el8.noarch 2023-03-09T14:36:19.084 INFO:teuthology.orchestra.run.smithi122.stdout: cephadm-2:18.0.0-2872.ge0ab30a2.el8.noarch 2023-03-09T14:36:19.084 INFO:teuthology.orchestra.run.smithi122.stdout: cephfs-mirror-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:19.085 INFO:teuthology.orchestra.run.smithi122.stdout: cephfs-top-2:18.0.0-2872.ge0ab30a2.el8.noarch 2023-03-09T14:36:19.085 INFO:teuthology.orchestra.run.smithi122.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-03-09T14:36:19.085 INFO:teuthology.orchestra.run.smithi122.stdout: flex-2.6.1-9.el8.x86_64 2023-03-09T14:36:19.085 INFO:teuthology.orchestra.run.smithi122.stdout: fuse-2.9.7-15.el8.x86_64 2023-03-09T14:36:19.085 INFO:teuthology.orchestra.run.smithi122.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-03-09T14:36:19.085 INFO:teuthology.orchestra.run.smithi122.stdout: jq-1.6-3.el8.x86_64 2023-03-09T14:36:19.086 INFO:teuthology.orchestra.run.smithi122.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-03-09T14:36:19.086 INFO:teuthology.orchestra.run.smithi122.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-03-09T14:36:19.086 INFO:teuthology.orchestra.run.smithi122.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-03-09T14:36:19.086 INFO:teuthology.orchestra.run.smithi122.stdout: libcephfs-devel-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:19.086 INFO:teuthology.orchestra.run.smithi122.stdout: libcephfs2-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:19.086 INFO:teuthology.orchestra.run.smithi122.stdout: libcephsqlite-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:19.086 INFO:teuthology.orchestra.run.smithi122.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-03-09T14:36:19.087 INFO:teuthology.orchestra.run.smithi122.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-03-09T14:36:19.087 INFO:teuthology.orchestra.run.smithi122.stdout: liboath-2.6.2-3.el8.x86_64 2023-03-09T14:36:19.087 INFO:teuthology.orchestra.run.smithi122.stdout: libpq-13.5-1.el8.x86_64 2023-03-09T14:36:19.087 INFO:teuthology.orchestra.run.smithi122.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-03-09T14:36:19.087 INFO:teuthology.orchestra.run.smithi122.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-03-09T14:36:19.087 INFO:teuthology.orchestra.run.smithi122.stdout: librados-devel-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:19.087 INFO:teuthology.orchestra.run.smithi122.stdout: libradosstriper1-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:19.088 INFO:teuthology.orchestra.run.smithi122.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-03-09T14:36:19.088 INFO:teuthology.orchestra.run.smithi122.stdout: librgw2-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:19.088 INFO:teuthology.orchestra.run.smithi122.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-03-09T14:36:19.088 INFO:teuthology.orchestra.run.smithi122.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-03-09T14:36:19.088 INFO:teuthology.orchestra.run.smithi122.stdout: libtool-2.4.6-25.el8.x86_64 2023-03-09T14:36:19.088 INFO:teuthology.orchestra.run.smithi122.stdout: libunwind-1.3.1-3.el8.x86_64 2023-03-09T14:36:19.088 INFO:teuthology.orchestra.run.smithi122.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-03-09T14:36:19.089 INFO:teuthology.orchestra.run.smithi122.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-03-09T14:36:19.089 INFO:teuthology.orchestra.run.smithi122.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-03-09T14:36:19.089 INFO:teuthology.orchestra.run.smithi122.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-03-09T14:36:19.089 INFO:teuthology.orchestra.run.smithi122.stdout: openblas-0.3.15-3.el8.x86_64 2023-03-09T14:36:19.089 INFO:teuthology.orchestra.run.smithi122.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-03-09T14:36:19.090 INFO:teuthology.orchestra.run.smithi122.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-03-09T14:36:19.090 INFO:teuthology.orchestra.run.smithi122.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-03-09T14:36:19.090 INFO:teuthology.orchestra.run.smithi122.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-03-09T14:36:19.090 INFO:teuthology.orchestra.run.smithi122.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-03-09T14:36:19.090 INFO:teuthology.orchestra.run.smithi122.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-03-09T14:36:19.090 INFO:teuthology.orchestra.run.smithi122.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-03-09T14:36:19.091 INFO:teuthology.orchestra.run.smithi122.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-03-09T14:36:19.091 INFO:teuthology.orchestra.run.smithi122.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-03-09T14:36:19.091 INFO:teuthology.orchestra.run.smithi122.stdout: python3-babel-2.5.1-7.el8.noarch 2023-03-09T14:36:19.091 INFO:teuthology.orchestra.run.smithi122.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-03-09T14:36:19.091 INFO:teuthology.orchestra.run.smithi122.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-03-09T14:36:19.092 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-03-09T14:36:19.092 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ceph-argparse-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:19.092 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ceph-common-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:19.092 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cephfs-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:19.092 INFO:teuthology.orchestra.run.smithi122.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-03-09T14:36:19.092 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-03-09T14:36:19.092 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-03-09T14:36:19.093 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-03-09T14:36:19.093 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-03-09T14:36:19.093 INFO:teuthology.orchestra.run.smithi122.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-03-09T14:36:19.093 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-6.2-6.el8.noarch 2023-03-09T14:36:19.093 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-03-09T14:36:19.093 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-03-09T14:36:19.094 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-03-09T14:36:19.094 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-03-09T14:36:19.094 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-03-09T14:36:19.094 INFO:teuthology.orchestra.run.smithi122.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-03-09T14:36:19.094 INFO:teuthology.orchestra.run.smithi122.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-03-09T14:36:19.094 INFO:teuthology.orchestra.run.smithi122.stdout: python3-mako-1.0.6-13.el8.noarch 2023-03-09T14:36:19.094 INFO:teuthology.orchestra.run.smithi122.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-03-09T14:36:19.095 INFO:teuthology.orchestra.run.smithi122.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-03-09T14:36:19.095 INFO:teuthology.orchestra.run.smithi122.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-03-09T14:36:19.095 INFO:teuthology.orchestra.run.smithi122.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-03-09T14:36:19.095 INFO:teuthology.orchestra.run.smithi122.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-03-09T14:36:19.095 INFO:teuthology.orchestra.run.smithi122.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-03-09T14:36:19.095 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-03-09T14:36:19.095 INFO:teuthology.orchestra.run.smithi122.stdout: python3-portend-2.6-1.el8.noarch 2023-03-09T14:36:19.096 INFO:teuthology.orchestra.run.smithi122.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-03-09T14:36:19.096 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-03-09T14:36:19.096 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-03-09T14:36:19.096 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-03-09T14:36:19.096 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pycparser-2.14-14.el8.noarch 2023-03-09T14:36:19.096 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pytz-2017.2-9.el8.noarch 2023-03-09T14:36:19.096 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rados-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:19.097 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rbd-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:19.097 INFO:teuthology.orchestra.run.smithi122.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-03-09T14:36:19.097 INFO:teuthology.orchestra.run.smithi122.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-03-09T14:36:19.097 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rgw-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:19.097 INFO:teuthology.orchestra.run.smithi122.stdout: python3-routes-2.4.1-12.el8.noarch 2023-03-09T14:36:19.097 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rsa-4.8-1.el8.noarch 2023-03-09T14:36:19.097 INFO:teuthology.orchestra.run.smithi122.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-03-09T14:36:19.098 INFO:teuthology.orchestra.run.smithi122.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-03-09T14:36:19.098 INFO:teuthology.orchestra.run.smithi122.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-03-09T14:36:19.098 INFO:teuthology.orchestra.run.smithi122.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-03-09T14:36:19.098 INFO:teuthology.orchestra.run.smithi122.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-03-09T14:36:19.098 INFO:teuthology.orchestra.run.smithi122.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2023-03-09T14:36:19.098 INFO:teuthology.orchestra.run.smithi122.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-03-09T14:36:19.098 INFO:teuthology.orchestra.run.smithi122.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-03-09T14:36:19.099 INFO:teuthology.orchestra.run.smithi122.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-03-09T14:36:19.099 INFO:teuthology.orchestra.run.smithi122.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-03-09T14:36:19.099 INFO:teuthology.orchestra.run.smithi122.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-03-09T14:36:19.099 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-fuse-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:19.099 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-mirror-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:19.099 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-nbd-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:19.099 INFO:teuthology.orchestra.run.smithi122.stdout: socat-1.7.4.1-1.el8.x86_64 2023-03-09T14:36:19.100 INFO:teuthology.orchestra.run.smithi122.stdout: thrift-0.13.0-2.el8.x86_64 2023-03-09T14:36:19.100 INFO:teuthology.orchestra.run.smithi122.stdout: uuid-1.6.2-43.el8.x86_64 2023-03-09T14:36:19.100 INFO:teuthology.orchestra.run.smithi122.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-03-09T14:36:19.100 INFO:teuthology.orchestra.run.smithi122.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-03-09T14:36:19.101 INFO:teuthology.orchestra.run.smithi122.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-03-09T14:36:19.101 INFO:teuthology.orchestra.run.smithi122.stdout: 2023-03-09T14:36:19.101 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2023-03-09T14:36:19.145 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-03-09T14:36:19.146 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-03-09T14:36:19.256 DEBUG:teuthology.parallel:result is None 2023-03-09T14:36:19.391 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-03-09T14:36:19.591 INFO:teuthology.orchestra.run.smithi038.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-03-09T14:36:19.902 INFO:teuthology.orchestra.run.smithi038.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-03-09T14:36:19.903 INFO:teuthology.orchestra.run.smithi038.stdout:[/usr/lib/tmpfiles.d/journal-nocow.conf:26] Failed to resolve specifier: uninitialized /etc detected, skipping 2023-03-09T14:36:19.903 INFO:teuthology.orchestra.run.smithi038.stdout:All rules containing unresolvable specifiers will be skipped. 2023-03-09T14:36:19.903 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:36:20.156 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-2:18.0.0-2872.ge0ab30a2.el8.x86_64 1/134 2023-03-09T14:36:20.157 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-base-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2/134 2023-03-09T14:36:20.157 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-common-2:18.0.0-2872.ge0ab30a2.el8.x86_64 3/134 2023-03-09T14:36:20.157 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-fuse-2:18.0.0-2872.ge0ab30a2.el8.x86_64 4/134 2023-03-09T14:36:20.157 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-2872.ge0ab3 5/134 2023-03-09T14:36:20.158 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mds-2:18.0.0-2872.ge0ab30a2.el8.x86_64 6/134 2023-03-09T14:36:20.158 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-2:18.0.0-2872.ge0ab30a2.el8.x86_64 7/134 2023-03-09T14:36:20.158 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mon-2:18.0.0-2872.ge0ab30a2.el8.x86_64 8/134 2023-03-09T14:36:20.158 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-osd-2:18.0.0-2872.ge0ab30a2.el8.x86_64 9/134 2023-03-09T14:36:20.158 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-radosgw-2:18.0.0-2872.ge0ab30a2.el8.x86_64 10/134 2023-03-09T14:36:20.158 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-selinux-2:18.0.0-2872.ge0ab30a2.el8.x86_64 11/134 2023-03-09T14:36:20.159 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-test-2:18.0.0-2872.ge0ab30a2.el8.x86_64 12/134 2023-03-09T14:36:20.159 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : cephfs-mirror-2:18.0.0-2872.ge0ab30a2.el8.x86_64 13/134 2023-03-09T14:36:20.159 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libcephfs-devel-2:18.0.0-2872.ge0ab30a2.el8.x86_ 14/134 2023-03-09T14:36:20.159 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libcephfs2-2:18.0.0-2872.ge0ab30a2.el8.x86_64 15/134 2023-03-09T14:36:20.159 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libcephsqlite-2:18.0.0-2872.ge0ab30a2.el8.x86_64 16/134 2023-03-09T14:36:20.159 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librados-devel-2:18.0.0-2872.ge0ab30a2.el8.x86_6 17/134 2023-03-09T14:36:20.159 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libradosstriper1-2:18.0.0-2872.ge0ab30a2.el8.x86 18/134 2023-03-09T14:36:20.160 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librgw2-2:18.0.0-2872.ge0ab30a2.el8.x86_64 19/134 2023-03-09T14:36:20.160 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-ceph-argparse-2:18.0.0-2872.ge0ab30a2.el 20/134 2023-03-09T14:36:20.160 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-ceph-common-2:18.0.0-2872.ge0ab30a2.el8. 21/134 2023-03-09T14:36:20.160 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cephfs-2:18.0.0-2872.ge0ab30a2.el8.x86_6 22/134 2023-03-09T14:36:20.160 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rados-2:18.0.0-2872.ge0ab30a2.el8.x86_64 23/134 2023-03-09T14:36:20.160 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rbd-2:18.0.0-2872.ge0ab30a2.el8.x86_64 24/134 2023-03-09T14:36:20.160 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rgw-2:18.0.0-2872.ge0ab30a2.el8.x86_64 25/134 2023-03-09T14:36:20.168 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : rbd-fuse-2:18.0.0-2872.ge0ab30a2.el8.x86_64 26/134 2023-03-09T14:36:20.168 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : rbd-mirror-2:18.0.0-2872.ge0ab30a2.el8.x86_64 27/134 2023-03-09T14:36:20.168 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : rbd-nbd-2:18.0.0-2872.ge0ab30a2.el8.x86_64 28/134 2023-03-09T14:36:20.168 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-2872.ge0ab30a2. 29/134 2023-03-09T14:36:20.169 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-2872.ge0ab30a2.el8.noa 30/134 2023-03-09T14:36:20.169 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-2872.ge0ab30a2.el8.n 31/134 2023-03-09T14:36:20.169 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-2872.ge0a 32/134 2023-03-09T14:36:20.169 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-2872.ge0ab30a2.el 33/134 2023-03-09T14:36:20.169 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-mgr-rook-2:18.0.0-2872.ge0ab30a2.el8.noarch 34/134 2023-03-09T14:36:20.169 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-2872.ge0ab30a2.e 35/134 2023-03-09T14:36:20.169 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : ceph-volume-2:18.0.0-2872.ge0ab30a2.el8.noarch 36/134 2023-03-09T14:36:20.170 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : cephadm-2:18.0.0-2872.ge0ab30a2.el8.noarch 37/134 2023-03-09T14:36:20.170 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : cephfs-top-2:18.0.0-2872.ge0ab30a2.el8.noarch 38/134 2023-03-09T14:36:20.170 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-03-09T14:36:20.170 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-03-09T14:36:20.170 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-03-09T14:36:20.170 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-03-09T14:36:20.170 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-03-09T14:36:20.170 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-03-09T14:36:20.171 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-03-09T14:36:20.171 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-03-09T14:36:20.171 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-03-09T14:36:20.171 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-03-09T14:36:20.171 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-03-09T14:36:20.171 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-03-09T14:36:20.171 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-03-09T14:36:20.172 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-03-09T14:36:20.172 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-03-09T14:36:20.172 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-03-09T14:36:20.172 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-03-09T14:36:20.172 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-03-09T14:36:20.172 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-03-09T14:36:20.172 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-03-09T14:36:20.172 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-03-09T14:36:20.173 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/134 2023-03-09T14:36:20.173 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-03-09T14:36:20.173 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-03-09T14:36:20.173 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-03-09T14:36:20.173 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-03-09T14:36:20.176 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 65/134 2023-03-09T14:36:20.176 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-03-09T14:36:20.176 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-03-09T14:36:20.176 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-03-09T14:36:20.176 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-03-09T14:36:20.176 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-03-09T14:36:20.176 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-03-09T14:36:20.177 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 72/134 2023-03-09T14:36:20.177 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 73/134 2023-03-09T14:36:20.177 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 74/134 2023-03-09T14:36:20.177 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 75/134 2023-03-09T14:36:20.177 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 76/134 2023-03-09T14:36:20.177 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 77/134 2023-03-09T14:36:20.177 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : bison-3.0.4-10.el8.x86_64 78/134 2023-03-09T14:36:20.178 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 79/134 2023-03-09T14:36:20.178 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 80/134 2023-03-09T14:36:20.178 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 81/134 2023-03-09T14:36:20.178 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : automake-1.16.1-7.el8.noarch 82/134 2023-03-09T14:36:20.178 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 83/134 2023-03-09T14:36:20.178 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 84/134 2023-03-09T14:36:20.178 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : flex-2.6.1-9.el8.x86_64 85/134 2023-03-09T14:36:20.179 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 86/134 2023-03-09T14:36:20.179 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 87/134 2023-03-09T14:36:20.179 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 88/134 2023-03-09T14:36:20.179 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 89/134 2023-03-09T14:36:20.179 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 90/134 2023-03-09T14:36:20.179 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 91/134 2023-03-09T14:36:20.179 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 92/134 2023-03-09T14:36:20.179 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 93/134 2023-03-09T14:36:20.180 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 94/134 2023-03-09T14:36:20.180 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 95/134 2023-03-09T14:36:20.180 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libpq-13.5-1.el8.x86_64 96/134 2023-03-09T14:36:20.180 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 97/134 2023-03-09T14:36:20.180 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : jq-1.6-3.el8.x86_64 98/134 2023-03-09T14:36:20.180 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 99/134 2023-03-09T14:36:20.180 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 100/134 2023-03-09T14:36:20.181 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 101/134 2023-03-09T14:36:20.181 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 102/134 2023-03-09T14:36:20.181 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 103/134 2023-03-09T14:36:20.181 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 104/134 2023-03-09T14:36:20.181 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 105/134 2023-03-09T14:36:20.181 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 106/134 2023-03-09T14:36:20.181 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 107/134 2023-03-09T14:36:20.181 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 108/134 2023-03-09T14:36:20.182 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 109/134 2023-03-09T14:36:20.182 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 110/134 2023-03-09T14:36:20.182 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 111/134 2023-03-09T14:36:20.182 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 112/134 2023-03-09T14:36:20.182 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 113/134 2023-03-09T14:36:20.182 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 114/134 2023-03-09T14:36:20.182 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 115/134 2023-03-09T14:36:20.183 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 116/134 2023-03-09T14:36:20.183 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 117/134 2023-03-09T14:36:20.183 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 118/134 2023-03-09T14:36:20.183 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 119/134 2023-03-09T14:36:20.183 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-03-09T14:36:20.183 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 121/134 2023-03-09T14:36:20.183 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-03-09T14:36:20.183 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 123/134 2023-03-09T14:36:20.184 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 124/134 2023-03-09T14:36:20.184 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 125/134 2023-03-09T14:36:20.184 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 126/134 2023-03-09T14:36:20.184 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 127/134 2023-03-09T14:36:20.184 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-03-09T14:36:20.184 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librados2-2:18.0.0-2872.ge0ab30a2.el8.x86_64 129/134 2023-03-09T14:36:20.184 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-03-09T14:36:20.185 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librbd1-2:18.0.0-2872.ge0ab30a2.el8.x86_64 131/134 2023-03-09T14:36:20.185 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-03-09T14:36:20.185 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-03-09T14:36:22.593 INFO:teuthology.orchestra.run.smithi038.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-03-09T14:36:22.593 INFO:teuthology.orchestra.run.smithi038.stdout:Last metadata expiration check: 0:01:58 ago on Thu 09 Mar 2023 02:34:24 PM UTC. 2023-03-09T14:36:25.793 INFO:teuthology.orchestra.run.smithi038.stdout:Installed products updated. 2023-03-09T14:36:26.030 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:36:26.030 INFO:teuthology.orchestra.run.smithi038.stdout:Upgraded: 2023-03-09T14:36:26.030 INFO:teuthology.orchestra.run.smithi038.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-03-09T14:36:26.030 INFO:teuthology.orchestra.run.smithi038.stdout: librados2-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:26.030 INFO:teuthology.orchestra.run.smithi038.stdout: librbd1-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:26.031 INFO:teuthology.orchestra.run.smithi038.stdout:Installed: 2023-03-09T14:36:26.031 INFO:teuthology.orchestra.run.smithi038.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-03-09T14:36:26.031 INFO:teuthology.orchestra.run.smithi038.stdout: automake-1.16.1-7.el8.noarch 2023-03-09T14:36:26.031 INFO:teuthology.orchestra.run.smithi038.stdout: bison-3.0.4-10.el8.x86_64 2023-03-09T14:36:26.031 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:26.031 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-base-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:26.031 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-common-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:26.031 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-fuse-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:26.032 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-grafana-dashboards-2:18.0.0-2872.ge0ab30a2.el8.noarch 2023-03-09T14:36:26.032 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-immutable-object-cache-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:26.032 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mds-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:26.032 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:26.032 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-cephadm-2:18.0.0-2872.ge0ab30a2.el8.noarch 2023-03-09T14:36:26.032 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-dashboard-2:18.0.0-2872.ge0ab30a2.el8.noarch 2023-03-09T14:36:26.032 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-diskprediction-local-2:18.0.0-2872.ge0ab30a2.el8.noarch 2023-03-09T14:36:26.033 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-modules-core-2:18.0.0-2872.ge0ab30a2.el8.noarch 2023-03-09T14:36:26.033 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mgr-rook-2:18.0.0-2872.ge0ab30a2.el8.noarch 2023-03-09T14:36:26.033 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-mon-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:26.033 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-osd-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:26.033 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-prometheus-alerts-2:18.0.0-2872.ge0ab30a2.el8.noarch 2023-03-09T14:36:26.033 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-radosgw-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:26.033 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-selinux-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:26.033 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-test-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:26.034 INFO:teuthology.orchestra.run.smithi038.stdout: ceph-volume-2:18.0.0-2872.ge0ab30a2.el8.noarch 2023-03-09T14:36:26.034 INFO:teuthology.orchestra.run.smithi038.stdout: cephadm-2:18.0.0-2872.ge0ab30a2.el8.noarch 2023-03-09T14:36:26.034 INFO:teuthology.orchestra.run.smithi038.stdout: cephfs-mirror-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:26.034 INFO:teuthology.orchestra.run.smithi038.stdout: cephfs-top-2:18.0.0-2872.ge0ab30a2.el8.noarch 2023-03-09T14:36:26.034 INFO:teuthology.orchestra.run.smithi038.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-03-09T14:36:26.034 INFO:teuthology.orchestra.run.smithi038.stdout: flex-2.6.1-9.el8.x86_64 2023-03-09T14:36:26.034 INFO:teuthology.orchestra.run.smithi038.stdout: fuse-2.9.7-15.el8.x86_64 2023-03-09T14:36:26.035 INFO:teuthology.orchestra.run.smithi038.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-03-09T14:36:26.035 INFO:teuthology.orchestra.run.smithi038.stdout: jq-1.6-3.el8.x86_64 2023-03-09T14:36:26.035 INFO:teuthology.orchestra.run.smithi038.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-03-09T14:36:26.035 INFO:teuthology.orchestra.run.smithi038.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-03-09T14:36:26.035 INFO:teuthology.orchestra.run.smithi038.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-03-09T14:36:26.035 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs-devel-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:26.035 INFO:teuthology.orchestra.run.smithi038.stdout: libcephfs2-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:26.035 INFO:teuthology.orchestra.run.smithi038.stdout: libcephsqlite-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:26.036 INFO:teuthology.orchestra.run.smithi038.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-03-09T14:36:26.036 INFO:teuthology.orchestra.run.smithi038.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-03-09T14:36:26.036 INFO:teuthology.orchestra.run.smithi038.stdout: liboath-2.6.2-3.el8.x86_64 2023-03-09T14:36:26.036 INFO:teuthology.orchestra.run.smithi038.stdout: libpq-13.5-1.el8.x86_64 2023-03-09T14:36:26.036 INFO:teuthology.orchestra.run.smithi038.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-03-09T14:36:26.036 INFO:teuthology.orchestra.run.smithi038.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-03-09T14:36:26.036 INFO:teuthology.orchestra.run.smithi038.stdout: librados-devel-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:26.037 INFO:teuthology.orchestra.run.smithi038.stdout: libradosstriper1-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:26.037 INFO:teuthology.orchestra.run.smithi038.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-03-09T14:36:26.037 INFO:teuthology.orchestra.run.smithi038.stdout: librgw2-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:26.037 INFO:teuthology.orchestra.run.smithi038.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-03-09T14:36:26.037 INFO:teuthology.orchestra.run.smithi038.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-03-09T14:36:26.037 INFO:teuthology.orchestra.run.smithi038.stdout: libtool-2.4.6-25.el8.x86_64 2023-03-09T14:36:26.037 INFO:teuthology.orchestra.run.smithi038.stdout: libunwind-1.3.1-3.el8.x86_64 2023-03-09T14:36:26.038 INFO:teuthology.orchestra.run.smithi038.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-03-09T14:36:26.038 INFO:teuthology.orchestra.run.smithi038.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-03-09T14:36:26.038 INFO:teuthology.orchestra.run.smithi038.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-03-09T14:36:26.038 INFO:teuthology.orchestra.run.smithi038.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-03-09T14:36:26.038 INFO:teuthology.orchestra.run.smithi038.stdout: openblas-0.3.15-3.el8.x86_64 2023-03-09T14:36:26.038 INFO:teuthology.orchestra.run.smithi038.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-03-09T14:36:26.038 INFO:teuthology.orchestra.run.smithi038.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-03-09T14:36:26.038 INFO:teuthology.orchestra.run.smithi038.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-03-09T14:36:26.039 INFO:teuthology.orchestra.run.smithi038.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-03-09T14:36:26.039 INFO:teuthology.orchestra.run.smithi038.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-03-09T14:36:26.039 INFO:teuthology.orchestra.run.smithi038.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-03-09T14:36:26.039 INFO:teuthology.orchestra.run.smithi038.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-03-09T14:36:26.039 INFO:teuthology.orchestra.run.smithi038.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-03-09T14:36:26.039 INFO:teuthology.orchestra.run.smithi038.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-03-09T14:36:26.039 INFO:teuthology.orchestra.run.smithi038.stdout: python3-babel-2.5.1-7.el8.noarch 2023-03-09T14:36:26.039 INFO:teuthology.orchestra.run.smithi038.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-03-09T14:36:26.040 INFO:teuthology.orchestra.run.smithi038.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-03-09T14:36:26.040 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-03-09T14:36:26.040 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-argparse-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:26.040 INFO:teuthology.orchestra.run.smithi038.stdout: python3-ceph-common-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:26.040 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cephfs-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:26.040 INFO:teuthology.orchestra.run.smithi038.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-03-09T14:36:26.040 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-03-09T14:36:26.040 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-03-09T14:36:26.041 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-03-09T14:36:26.041 INFO:teuthology.orchestra.run.smithi038.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-03-09T14:36:26.041 INFO:teuthology.orchestra.run.smithi038.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-03-09T14:36:26.041 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-6.2-6.el8.noarch 2023-03-09T14:36:26.041 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-03-09T14:36:26.041 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-03-09T14:36:26.041 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-03-09T14:36:26.042 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-03-09T14:36:26.042 INFO:teuthology.orchestra.run.smithi038.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-03-09T14:36:26.042 INFO:teuthology.orchestra.run.smithi038.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-03-09T14:36:26.042 INFO:teuthology.orchestra.run.smithi038.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-03-09T14:36:26.042 INFO:teuthology.orchestra.run.smithi038.stdout: python3-mako-1.0.6-13.el8.noarch 2023-03-09T14:36:26.042 INFO:teuthology.orchestra.run.smithi038.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-03-09T14:36:26.042 INFO:teuthology.orchestra.run.smithi038.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-03-09T14:36:26.042 INFO:teuthology.orchestra.run.smithi038.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-03-09T14:36:26.043 INFO:teuthology.orchestra.run.smithi038.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-03-09T14:36:26.043 INFO:teuthology.orchestra.run.smithi038.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-03-09T14:36:26.043 INFO:teuthology.orchestra.run.smithi038.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-03-09T14:36:26.043 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-03-09T14:36:26.043 INFO:teuthology.orchestra.run.smithi038.stdout: python3-portend-2.6-1.el8.noarch 2023-03-09T14:36:26.043 INFO:teuthology.orchestra.run.smithi038.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-03-09T14:36:26.043 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-03-09T14:36:26.043 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-03-09T14:36:26.044 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-03-09T14:36:26.044 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pycparser-2.14-14.el8.noarch 2023-03-09T14:36:26.044 INFO:teuthology.orchestra.run.smithi038.stdout: python3-pytz-2017.2-9.el8.noarch 2023-03-09T14:36:26.044 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rados-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:26.044 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rbd-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:26.044 INFO:teuthology.orchestra.run.smithi038.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-03-09T14:36:26.044 INFO:teuthology.orchestra.run.smithi038.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-03-09T14:36:26.045 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rgw-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:26.045 INFO:teuthology.orchestra.run.smithi038.stdout: python3-routes-2.4.1-12.el8.noarch 2023-03-09T14:36:26.045 INFO:teuthology.orchestra.run.smithi038.stdout: python3-rsa-4.8-1.el8.noarch 2023-03-09T14:36:26.045 INFO:teuthology.orchestra.run.smithi038.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-03-09T14:36:26.045 INFO:teuthology.orchestra.run.smithi038.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-03-09T14:36:26.045 INFO:teuthology.orchestra.run.smithi038.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-03-09T14:36:26.045 INFO:teuthology.orchestra.run.smithi038.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-03-09T14:36:26.045 INFO:teuthology.orchestra.run.smithi038.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-03-09T14:36:26.046 INFO:teuthology.orchestra.run.smithi038.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2023-03-09T14:36:26.046 INFO:teuthology.orchestra.run.smithi038.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-03-09T14:36:26.046 INFO:teuthology.orchestra.run.smithi038.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-03-09T14:36:26.046 INFO:teuthology.orchestra.run.smithi038.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-03-09T14:36:26.046 INFO:teuthology.orchestra.run.smithi038.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-03-09T14:36:26.046 INFO:teuthology.orchestra.run.smithi038.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-03-09T14:36:26.046 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-fuse-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:26.046 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-mirror-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:26.047 INFO:teuthology.orchestra.run.smithi038.stdout: rbd-nbd-2:18.0.0-2872.ge0ab30a2.el8.x86_64 2023-03-09T14:36:26.047 INFO:teuthology.orchestra.run.smithi038.stdout: socat-1.7.4.1-1.el8.x86_64 2023-03-09T14:36:26.047 INFO:teuthology.orchestra.run.smithi038.stdout: thrift-0.13.0-2.el8.x86_64 2023-03-09T14:36:26.047 INFO:teuthology.orchestra.run.smithi038.stdout: uuid-1.6.2-43.el8.x86_64 2023-03-09T14:36:26.047 INFO:teuthology.orchestra.run.smithi038.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-03-09T14:36:26.047 INFO:teuthology.orchestra.run.smithi038.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-03-09T14:36:26.048 INFO:teuthology.orchestra.run.smithi038.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-03-09T14:36:26.048 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:36:26.048 INFO:teuthology.orchestra.run.smithi038.stdout:Complete! 2023-03-09T14:36:26.412 DEBUG:teuthology.parallel:result is None 2023-03-09T14:36:26.413 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=e0ab30a2acead6db62b144ba6b6aeb23635adae7 2023-03-09T14:36:26.551 DEBUG:teuthology.orchestra.run.smithi038:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-03-09T14:36:26.575 INFO:teuthology.orchestra.run.smithi038.stdout:18.0.0-2872.ge0ab30a2.el8 2023-03-09T14:36:26.576 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-2872.ge0ab30a2.el8 2023-03-09T14:36:26.576 INFO:teuthology.task.install:The correct ceph version 18.0.0-2872.ge0ab30a2 is installed. 2023-03-09T14:36:26.577 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=e0ab30a2acead6db62b144ba6b6aeb23635adae7 2023-03-09T14:36:26.716 DEBUG:teuthology.orchestra.run.smithi122:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-03-09T14:36:26.745 INFO:teuthology.orchestra.run.smithi122.stdout:18.0.0-2872.ge0ab30a2.el8 2023-03-09T14:36:26.745 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-2872.ge0ab30a2.el8 2023-03-09T14:36:26.745 INFO:teuthology.task.install:The correct ceph version 18.0.0-2872.ge0ab30a2 is installed. 2023-03-09T14:36:26.746 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=e0ab30a2acead6db62b144ba6b6aeb23635adae7 2023-03-09T14:36:26.885 DEBUG:teuthology.orchestra.run.smithi181:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-03-09T14:36:26.914 INFO:teuthology.orchestra.run.smithi181.stdout:18.0.0-2872.ge0ab30a2.el8 2023-03-09T14:36:26.914 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-2872.ge0ab30a2.el8 2023-03-09T14:36:26.914 INFO:teuthology.task.install:The correct ceph version 18.0.0-2872.ge0ab30a2 is installed. 2023-03-09T14:36:26.915 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-03-09T14:36:26.915 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2023-03-09T14:36:26.916 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-03-09T14:36:26.959 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2023-03-09T14:36:26.960 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-03-09T14:36:27.008 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2023-03-09T14:36:27.008 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-03-09T14:36:27.053 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-03-09T14:36:27.053 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2023-03-09T14:36:27.054 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/usr/bin/daemon-helper 2023-03-09T14:36:27.082 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-03-09T14:36:27.152 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2023-03-09T14:36:27.152 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/usr/bin/daemon-helper 2023-03-09T14:36:27.185 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-03-09T14:36:27.254 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2023-03-09T14:36:27.255 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/usr/bin/daemon-helper 2023-03-09T14:36:27.286 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-03-09T14:36:27.356 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-03-09T14:36:27.357 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2023-03-09T14:36:27.357 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/usr/bin/adjust-ulimits 2023-03-09T14:36:27.386 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-03-09T14:36:27.456 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2023-03-09T14:36:27.456 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/usr/bin/adjust-ulimits 2023-03-09T14:36:27.486 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-03-09T14:36:27.556 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2023-03-09T14:36:27.557 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/usr/bin/adjust-ulimits 2023-03-09T14:36:27.586 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-03-09T14:36:27.656 INFO:teuthology.run_tasks:Running task cephadm... 2023-03-09T14:36:27.751 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': 'e0ab30a2acead6db62b144ba6b6aeb23635adae7', 'subvols': {'create': 2}} 2023-03-09T14:36:27.752 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 2023-03-09T14:36:27.765 INFO:tasks.cephadm:Cluster fsid is c6733fe2-be87-11ed-9af2-001a4aab830c 2023-03-09T14:36:27.766 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2023-03-09T14:36:27.767 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.38', 'mon.b': '172.21.15.122', 'mon.c': '172.21.15.181'} 2023-03-09T14:36:27.767 INFO:tasks.cephadm:First mon is mon.a on smithi038 2023-03-09T14:36:27.767 INFO:tasks.cephadm:First mgr is x 2023-03-09T14:36:27.768 INFO:tasks.cephadm:Normalizing hostnames... 2023-03-09T14:36:27.768 DEBUG:teuthology.orchestra.run.smithi038:> sudo hostname $(hostname -s) 2023-03-09T14:36:27.802 DEBUG:teuthology.orchestra.run.smithi122:> sudo hostname $(hostname -s) 2023-03-09T14:36:27.834 DEBUG:teuthology.orchestra.run.smithi181:> sudo hostname $(hostname -s) 2023-03-09T14:36:27.865 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2023-03-09T14:36:27.866 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=e0ab30a2acead6db62b144ba6b6aeb23635adae7 2023-03-09T14:36:28.006 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'e0ab30a2acead6db62b144ba6b6aeb23635adae7', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic/68489/', 'root_build_cause': 'SCMTRIGGER', 'version': '18.0.0-2872-ge0ab30a2', 'node_name': '172.21.4.66+confusa04', 'job_name': 'ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic', 'package_manager_version': '18.0.0-2872.ge0ab30a2'}, 'url': 'https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230308.150904/e0ab30a2acead6db62b144ba6b6aeb23635adae7/centos/8/flavors/default/', 'distro_codename': None, 'modified': '2023-03-08 19:55:47.443026', 'distro_version': '8', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-pdonnell-testing-20230308.150904', 'chacra_url': 'https://2.chacra.ceph.com/repos/ceph/wip-pdonnell-testing-20230308.150904/e0ab30a2acead6db62b144ba6b6aeb23635adae7/centos/8/flavors/default/', 'archs': ['arm64', 'source', 'x86_64'], 'distro': 'centos'}] 2023-03-09T14:36:28.155 INFO:tasks.util.chacra:got chacra host 2.chacra.ceph.com, ref wip-pdonnell-testing-20230308.150904, sha1 e0ab30a2acead6db62b144ba6b6aeb23635adae7 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F8%2Fx86_64&flavor=default&sha1=e0ab30a2acead6db62b144ba6b6aeb23635adae7 2023-03-09T14:36:28.156 INFO:tasks.cephadm:Discovered cachra url: https://2.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20230308.150904/e0ab30a2acead6db62b144ba6b6aeb23635adae7/centos/8/x86_64/flavors/default/cephadm 2023-03-09T14:36:28.156 DEBUG:teuthology.orchestra.run.smithi038:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20230308.150904/e0ab30a2acead6db62b144ba6b6aeb23635adae7/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-03-09T14:36:28.955 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-rw-r--. 1 ubuntu ubuntu 381328 Mar 9 14:36 /home/ubuntu/cephtest/cephadm 2023-03-09T14:36:28.956 DEBUG:teuthology.orchestra.run.smithi122:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20230308.150904/e0ab30a2acead6db62b144ba6b6aeb23635adae7/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-03-09T14:36:29.912 INFO:teuthology.orchestra.run.smithi122.stdout:-rw-rw-r--. 1 ubuntu ubuntu 381328 Mar 9 14:36 /home/ubuntu/cephtest/cephadm 2023-03-09T14:36:29.912 DEBUG:teuthology.orchestra.run.smithi181:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20230308.150904/e0ab30a2acead6db62b144ba6b6aeb23635adae7/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-03-09T14:36:30.861 INFO:teuthology.orchestra.run.smithi181.stdout:-rw-rw-r--. 1 ubuntu ubuntu 381328 Mar 9 14:36 /home/ubuntu/cephtest/cephadm 2023-03-09T14:36:30.861 DEBUG:teuthology.orchestra.run.smithi038:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-03-09T14:36:30.884 DEBUG:teuthology.orchestra.run.smithi122:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-03-09T14:36:30.906 DEBUG:teuthology.orchestra.run.smithi181:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-03-09T14:36:30.943 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 on all hosts... 2023-03-09T14:36:30.944 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 pull 2023-03-09T14:36:30.946 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 pull 2023-03-09T14:36:30.948 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 pull 2023-03-09T14:36:35.210 INFO:teuthology.orchestra.run.smithi181.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7... 2023-03-09T14:36:35.778 INFO:teuthology.orchestra.run.smithi122.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7... 2023-03-09T14:36:36.869 INFO:teuthology.orchestra.run.smithi038.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7... 2023-03-09T14:37:02.357 INFO:teuthology.orchestra.run.smithi122.stdout:{ 2023-03-09T14:37:02.358 INFO:teuthology.orchestra.run.smithi122.stdout: "ceph_version": "ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev)", 2023-03-09T14:37:02.358 INFO:teuthology.orchestra.run.smithi122.stdout: "image_id": "d8c5b11cdf0668b87241f2e067b7af57c2bf892eb384b752b41170a6044e3574", 2023-03-09T14:37:02.358 INFO:teuthology.orchestra.run.smithi122.stdout: "repo_digests": [ 2023-03-09T14:37:02.358 INFO:teuthology.orchestra.run.smithi122.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:db34b9742404bb6527a0bef7f79ad3f77639becbbf91cd5a2a15e19553c9a89f" 2023-03-09T14:37:02.358 INFO:teuthology.orchestra.run.smithi122.stdout: ] 2023-03-09T14:37:02.359 INFO:teuthology.orchestra.run.smithi122.stdout:} 2023-03-09T14:37:03.960 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2023-03-09T14:37:03.961 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_version": "ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev)", 2023-03-09T14:37:03.961 INFO:teuthology.orchestra.run.smithi038.stdout: "image_id": "d8c5b11cdf0668b87241f2e067b7af57c2bf892eb384b752b41170a6044e3574", 2023-03-09T14:37:03.961 INFO:teuthology.orchestra.run.smithi038.stdout: "repo_digests": [ 2023-03-09T14:37:03.961 INFO:teuthology.orchestra.run.smithi038.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:db34b9742404bb6527a0bef7f79ad3f77639becbbf91cd5a2a15e19553c9a89f" 2023-03-09T14:37:03.961 INFO:teuthology.orchestra.run.smithi038.stdout: ] 2023-03-09T14:37:03.961 INFO:teuthology.orchestra.run.smithi038.stdout:} 2023-03-09T14:37:06.896 INFO:teuthology.orchestra.run.smithi181.stdout:{ 2023-03-09T14:37:06.897 INFO:teuthology.orchestra.run.smithi181.stdout: "ceph_version": "ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev)", 2023-03-09T14:37:06.897 INFO:teuthology.orchestra.run.smithi181.stdout: "image_id": "d8c5b11cdf0668b87241f2e067b7af57c2bf892eb384b752b41170a6044e3574", 2023-03-09T14:37:06.897 INFO:teuthology.orchestra.run.smithi181.stdout: "repo_digests": [ 2023-03-09T14:37:06.897 INFO:teuthology.orchestra.run.smithi181.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:db34b9742404bb6527a0bef7f79ad3f77639becbbf91cd5a2a15e19553c9a89f" 2023-03-09T14:37:06.897 INFO:teuthology.orchestra.run.smithi181.stdout: ] 2023-03-09T14:37:06.898 INFO:teuthology.orchestra.run.smithi181.stdout:} 2023-03-09T14:37:06.918 DEBUG:teuthology.orchestra.run.smithi038:> sudo mkdir -p /etc/ceph 2023-03-09T14:37:06.955 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /etc/ceph 2023-03-09T14:37:06.993 DEBUG:teuthology.orchestra.run.smithi181:> sudo mkdir -p /etc/ceph 2023-03-09T14:37:07.023 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 777 /etc/ceph 2023-03-09T14:37:07.058 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 777 /etc/ceph 2023-03-09T14:37:07.089 DEBUG:teuthology.orchestra.run.smithi181:> sudo chmod 777 /etc/ceph 2023-03-09T14:37:07.115 INFO:tasks.cephadm:Writing seed config... 2023-03-09T14:37:07.117 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2023-03-09T14:37:07.117 INFO:tasks.cephadm: override: [client] debug client = 20 2023-03-09T14:37:07.117 INFO:tasks.cephadm: override: [client] debug ms = 1 2023-03-09T14:37:07.118 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2023-03-09T14:37:07.118 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2023-03-09T14:37:07.118 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2023-03-09T14:37:07.118 INFO:tasks.cephadm: override: [mds] debug mds = 20 2023-03-09T14:37:07.118 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2023-03-09T14:37:07.119 INFO:tasks.cephadm: override: [mds] debug ms = 1 2023-03-09T14:37:07.119 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2023-03-09T14:37:07.119 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2023-03-09T14:37:07.119 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2023-03-09T14:37:07.119 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2023-03-09T14:37:07.120 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2023-03-09T14:37:07.120 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2023-03-09T14:37:07.120 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2023-03-09T14:37:07.120 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2023-03-09T14:37:07.120 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2023-03-09T14:37:07.121 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2023-03-09T14:37:07.121 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2023-03-09T14:37:07.121 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2023-03-09T14:37:07.121 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2023-03-09T14:37:07.121 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2023-03-09T14:37:07.122 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2023-03-09T14:37:07.122 INFO:tasks.cephadm: override: [mon] debug mon = 20 2023-03-09T14:37:07.122 INFO:tasks.cephadm: override: [mon] debug ms = 1 2023-03-09T14:37:07.122 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2023-03-09T14:37:07.122 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2023-03-09T14:37:07.123 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2023-03-09T14:37:07.123 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2023-03-09T14:37:07.123 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2023-03-09T14:37:07.123 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2023-03-09T14:37:07.123 INFO:tasks.cephadm: override: [osd] debug ms = 1 2023-03-09T14:37:07.123 INFO:tasks.cephadm: override: [osd] debug osd = 20 2023-03-09T14:37:07.124 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2023-03-09T14:37:07.124 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2023-03-09T14:37:07.124 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2023-03-09T14:37:07.124 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2023-03-09T14:37:07.124 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2023-03-09T14:37:07.125 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2023-03-09T14:37:07.125 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2023-03-09T14:37:07.125 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2023-03-09T14:37:07.125 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2023-03-09T14:37:07.126 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2023-03-09T14:37:07.126 DEBUG:teuthology.orchestra.run.smithi038:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2023-03-09T14:37:07.147 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = c6733fe2-be87-11ed-9af2-001a4aab830c ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2023-03-09T14:37:07.148 DEBUG:teuthology.orchestra.run.smithi038:mon.a> sudo journalctl -f -n 0 -u ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@mon.a.service 2023-03-09T14:37:07.189 DEBUG:teuthology.orchestra.run.smithi038:mgr.x> sudo journalctl -f -n 0 -u ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@mgr.x.service 2023-03-09T14:37:07.220 INFO:journalctl@ceph.mon.a.smithi038.stdout:-- Logs begin at Thu 2023-03-09 14:24:59 UTC. -- 2023-03-09T14:37:07.220 INFO:tasks.cephadm:Bootstrapping... 2023-03-09T14:37:07.221 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 -v bootstrap --fsid c6733fe2-be87-11ed-9af2-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.38 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2023-03-09T14:37:07.250 INFO:journalctl@ceph.mgr.x.smithi038.stdout:-- Logs begin at Thu 2023-03-09 14:24:59 UTC. -- 2023-03-09T14:37:07.485 INFO:teuthology.orchestra.run.smithi038.stdout:-------------------------------------------------------------------------------- 2023-03-09T14:37:07.486 INFO:teuthology.orchestra.run.smithi038.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7', '-v', 'bootstrap', '--fsid', 'c6733fe2-be87-11ed-9af2-001a4aab830c', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.38', '--skip-admin-label'] 2023-03-09T14:37:07.634 INFO:teuthology.orchestra.run.smithi038.stdout:/bin/podman: stdout 4.1.1 2023-03-09T14:37:07.635 INFO:teuthology.orchestra.run.smithi038.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2023-03-09T14:37:07.635 INFO:teuthology.orchestra.run.smithi038.stdout:Verifying podman|docker is present... 2023-03-09T14:37:07.781 INFO:teuthology.orchestra.run.smithi038.stdout:/bin/podman: stdout 4.1.1 2023-03-09T14:37:07.781 INFO:teuthology.orchestra.run.smithi038.stdout:Verifying lvm2 is present... 2023-03-09T14:37:07.781 INFO:teuthology.orchestra.run.smithi038.stdout:Verifying time synchronization is in place... 2023-03-09T14:37:07.789 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2023-03-09T14:37:07.789 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2023-03-09T14:37:07.794 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2023-03-09T14:37:07.795 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout inactive 2023-03-09T14:37:07.800 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout enabled 2023-03-09T14:37:07.806 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout active 2023-03-09T14:37:07.807 INFO:teuthology.orchestra.run.smithi038.stdout:Unit chronyd.service is enabled and running 2023-03-09T14:37:07.807 INFO:teuthology.orchestra.run.smithi038.stdout:Repeating the final host check... 2023-03-09T14:37:07.949 INFO:teuthology.orchestra.run.smithi038.stdout:/bin/podman: stdout 4.1.1 2023-03-09T14:37:07.950 INFO:teuthology.orchestra.run.smithi038.stdout:podman (/bin/podman) version 4.1.1 is present 2023-03-09T14:37:07.950 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl is present 2023-03-09T14:37:07.950 INFO:teuthology.orchestra.run.smithi038.stdout:lvcreate is present 2023-03-09T14:37:07.956 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2023-03-09T14:37:07.956 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2023-03-09T14:37:07.962 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2023-03-09T14:37:07.962 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout inactive 2023-03-09T14:37:07.968 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout enabled 2023-03-09T14:37:07.974 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout active 2023-03-09T14:37:07.974 INFO:teuthology.orchestra.run.smithi038.stdout:Unit chronyd.service is enabled and running 2023-03-09T14:37:07.974 INFO:teuthology.orchestra.run.smithi038.stdout:Host looks OK 2023-03-09T14:37:07.974 INFO:teuthology.orchestra.run.smithi038.stdout:Cluster fsid: c6733fe2-be87-11ed-9af2-001a4aab830c 2023-03-09T14:37:07.975 INFO:teuthology.orchestra.run.smithi038.stdout:Acquiring lock 140712493787624 on /run/cephadm/c6733fe2-be87-11ed-9af2-001a4aab830c.lock 2023-03-09T14:37:07.975 INFO:teuthology.orchestra.run.smithi038.stdout:Lock 140712493787624 acquired on /run/cephadm/c6733fe2-be87-11ed-9af2-001a4aab830c.lock 2023-03-09T14:37:07.975 INFO:teuthology.orchestra.run.smithi038.stdout:Verifying IP 172.21.15.38 port 3300 ... 2023-03-09T14:37:07.976 INFO:teuthology.orchestra.run.smithi038.stdout:Verifying IP 172.21.15.38 port 6789 ... 2023-03-09T14:37:07.976 INFO:teuthology.orchestra.run.smithi038.stdout:Base mon IP(s) is [172.21.15.38:3300, 172.21.15.38:6789], mon addrv is [v2:172.21.15.38:3300,v1:172.21.15.38:6789] 2023-03-09T14:37:07.979 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp metric 100 2023-03-09T14:37:07.979 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.38 metric 100 2023-03-09T14:37:07.982 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2023-03-09T14:37:07.982 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2023-03-09T14:37:07.982 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1755sec hoplimit 64 pref medium 2023-03-09T14:37:07.985 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2023-03-09T14:37:07.985 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2023-03-09T14:37:07.985 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2023-03-09T14:37:07.985 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout 3: ens1f0: mtu 1500 state UP qlen 1000 2023-03-09T14:37:07.985 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:15fc/64 scope link 2023-03-09T14:37:07.985 INFO:teuthology.orchestra.run.smithi038.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2023-03-09T14:37:07.986 INFO:teuthology.orchestra.run.smithi038.stdout:Mon IP `172.21.15.38` is in CIDR network `172.21.0.0/20` 2023-03-09T14:37:07.986 INFO:teuthology.orchestra.run.smithi038.stdout:Mon IP `172.21.15.38` is in CIDR network `172.21.0.0/20` 2023-03-09T14:37:07.986 INFO:teuthology.orchestra.run.smithi038.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2023-03-09T14:37:07.987 INFO:teuthology.orchestra.run.smithi038.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2023-03-09T14:37:07.987 INFO:teuthology.orchestra.run.smithi038.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7... 2023-03-09T14:37:08.431 INFO:teuthology.orchestra.run.smithi038.stdout:/bin/podman: stdout d8c5b11cdf0668b87241f2e067b7af57c2bf892eb384b752b41170a6044e3574 2023-03-09T14:37:08.431 INFO:teuthology.orchestra.run.smithi038.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7... 2023-03-09T14:37:08.431 INFO:teuthology.orchestra.run.smithi038.stdout:/bin/podman: stderr Getting image source signatures 2023-03-09T14:37:08.432 INFO:teuthology.orchestra.run.smithi038.stdout:/bin/podman: stderr Copying blob sha256:b1eecc17c5b67bfd7cb2b95e6d58ffc9b721b003f79362935d82520417abb815 2023-03-09T14:37:08.432 INFO:teuthology.orchestra.run.smithi038.stdout:/bin/podman: stderr Copying blob sha256:4ef69b820011cb216760536bd683e53e03e0de703a28853f710a3c81acd0e100 2023-03-09T14:37:08.432 INFO:teuthology.orchestra.run.smithi038.stdout:/bin/podman: stderr Copying blob sha256:c9a93b47ad083f6acb64b4e977bea670e87fb6dd3607ac0db42af43a5aaac227 2023-03-09T14:37:08.432 INFO:teuthology.orchestra.run.smithi038.stdout:/bin/podman: stderr Copying blob sha256:ea0a20a2c44861ba5e4cd80a32868b6f8265cd3ee81310d4d493a1b50d1c0d8c 2023-03-09T14:37:08.432 INFO:teuthology.orchestra.run.smithi038.stdout:/bin/podman: stderr Copying config sha256:d8c5b11cdf0668b87241f2e067b7af57c2bf892eb384b752b41170a6044e3574 2023-03-09T14:37:08.432 INFO:teuthology.orchestra.run.smithi038.stdout:/bin/podman: stderr Writing manifest to image destination 2023-03-09T14:37:08.432 INFO:teuthology.orchestra.run.smithi038.stdout:/bin/podman: stderr Storing signatures 2023-03-09T14:37:09.385 INFO:teuthology.orchestra.run.smithi038.stdout:ceph: stdout ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:37:09.386 INFO:teuthology.orchestra.run.smithi038.stdout:Ceph version: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:37:09.386 INFO:teuthology.orchestra.run.smithi038.stdout:Extracting ceph user uid/gid from container image... 2023-03-09T14:37:10.137 INFO:teuthology.orchestra.run.smithi038.stdout:stat: stdout 167 167 2023-03-09T14:37:10.138 INFO:teuthology.orchestra.run.smithi038.stdout:Creating initial keys... 2023-03-09T14:37:10.886 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph-authtool: stdout AQAW7wlkX+HrHBAA9EApq0h7gHqg0zffjbtp1Q== 2023-03-09T14:37:11.636 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph-authtool: stdout AQAX7wlk9e09EhAAxUOaSFea4eusbkMgj7nYgQ== 2023-03-09T14:37:12.387 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph-authtool: stdout AQAY7wlkMoZIABAASxqHl+Y5NXNt1/ARlRSt4Q== 2023-03-09T14:37:12.388 INFO:teuthology.orchestra.run.smithi038.stdout:Creating initial monmap... 2023-03-09T14:37:13.135 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2023-03-09T14:37:13.136 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = pacific 2023-03-09T14:37:13.136 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to c6733fe2-be87-11ed-9af2-001a4aab830c 2023-03-09T14:37:13.136 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-03-09T14:37:13.136 INFO:teuthology.orchestra.run.smithi038.stdout:monmaptool for a [v2:172.21.15.38:3300,v1:172.21.15.38:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2023-03-09T14:37:13.136 INFO:teuthology.orchestra.run.smithi038.stdout:setting min_mon_release = pacific 2023-03-09T14:37:13.136 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/monmaptool: set fsid to c6733fe2-be87-11ed-9af2-001a4aab830c 2023-03-09T14:37:13.137 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-03-09T14:37:13.137 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:37:13.137 INFO:teuthology.orchestra.run.smithi038.stdout:Creating mon... 2023-03-09T14:37:13.881 INFO:teuthology.orchestra.run.smithi038.stdout:create mon.a on 2023-03-09T14:37:13.983 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stderr Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2023-03-09T14:37:14.075 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2023-03-09T14:37:14.175 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-c6733fe2-be87-11ed-9af2-001a4aab830c.target → /etc/systemd/system/ceph-c6733fe2-be87-11ed-9af2-001a4aab830c.target. 2023-03-09T14:37:14.176 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-c6733fe2-be87-11ed-9af2-001a4aab830c.target → /etc/systemd/system/ceph-c6733fe2-be87-11ed-9af2-001a4aab830c.target. 2023-03-09T14:37:14.307 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@mon.a 2023-03-09T14:37:14.307 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stderr Failed to reset failed state of unit ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@mon.a.service: Unit ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@mon.a.service not loaded. 2023-03-09T14:37:14.402 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-c6733fe2-be87-11ed-9af2-001a4aab830c.target.wants/ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@mon.a.service → /etc/systemd/system/ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@.service. 2023-03-09T14:37:14.708 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:14 smithi038 systemd[1]: Starting Ceph mon.a for c6733fe2-be87-11ed-9af2-001a4aab830c... 2023-03-09T14:37:14.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:14 smithi038 podman[91885]: 2023-03-09T14:37:15.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:14 smithi038 bash[91885]: b44a56625438bd0d6b201bc6af8da84a517601aeb086a61a906eec0559ea8b85 2023-03-09T14:37:15.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:14 smithi038 ceph-mon[91925]: mkfs c6733fe2-be87-11ed-9af2-001a4aab830c 2023-03-09T14:37:15.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:14 smithi038 ceph-mon[91925]: mon.a is new leader, mons a in quorum (ranks 0) 2023-03-09T14:37:15.011 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-03-09T14:37:15.012 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout disabled 2023-03-09T14:37:15.019 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-03-09T14:37:15.019 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout inactive 2023-03-09T14:37:15.019 INFO:teuthology.orchestra.run.smithi038.stdout:firewalld.service is not enabled 2023-03-09T14:37:15.019 INFO:teuthology.orchestra.run.smithi038.stdout:Not possible to enable service . firewalld.service is not available 2023-03-09T14:37:15.020 INFO:teuthology.orchestra.run.smithi038.stdout:Waiting for mon to start... 2023-03-09T14:37:15.020 INFO:teuthology.orchestra.run.smithi038.stdout:Waiting for mon... 2023-03-09T14:37:15.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:14 smithi038 systemd[1]: Started Ceph mon.a for c6733fe2-be87-11ed-9af2-001a4aab830c. 2023-03-09T14:37:15.869 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout cluster: 2023-03-09T14:37:15.869 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout id: c6733fe2-be87-11ed-9af2-001a4aab830c 2023-03-09T14:37:15.870 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2023-03-09T14:37:15.870 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2023-03-09T14:37:15.871 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout services: 2023-03-09T14:37:15.871 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.644449s) 2023-03-09T14:37:15.871 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2023-03-09T14:37:15.871 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2023-03-09T14:37:15.871 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2023-03-09T14:37:15.871 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout data: 2023-03-09T14:37:15.871 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2023-03-09T14:37:15.871 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2023-03-09T14:37:15.872 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2023-03-09T14:37:15.872 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout pgs: 2023-03-09T14:37:15.872 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2023-03-09T14:37:15.872 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:15.450+0000 7fea3db17700 1 Processor -- start 2023-03-09T14:37:15.872 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:15.450+0000 7fea3db17700 1 -- start start 2023-03-09T14:37:15.872 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:15.450+0000 7fea3db17700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea381060f0 0x7fea381084d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:37:15.872 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:15.450+0000 7fea3db17700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea38079d50 con 0x7fea381060f0 2023-03-09T14:37:15.873 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:15.451+0000 7fea377fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea381060f0 0x7fea381084d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:37:15.873 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:15.451+0000 7fea377fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea381060f0 0x7fea381084d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35330/0 (socket says 172.21.15.38:35330) 2023-03-09T14:37:15.873 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:15.451+0000 7fea377fe700 1 -- 172.21.15.38:0/3165919905 learned_addr learned my addr 172.21.15.38:0/3165919905 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:37:15.873 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:15.451+0000 7fea377fe700 1 -- 172.21.15.38:0/3165919905 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea38079e90 con 0x7fea381060f0 2023-03-09T14:37:15.873 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:15.451+0000 7fea377fe700 1 --2- 172.21.15.38:0/3165919905 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea381060f0 0x7fea381084d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fea280098a0 tx=0x7fea28009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97bea4356b253f29 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:37:15.873 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:15.451+0000 7fea367fc700 1 -- 172.21.15.38:0/3165919905 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fea28004160 con 0x7fea381060f0 2023-03-09T14:37:15.873 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:15.452+0000 7fea367fc700 1 -- 172.21.15.38:0/3165919905 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea280042c0 con 0x7fea381060f0 2023-03-09T14:37:15.874 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:15.452+0000 7fea367fc700 1 -- 172.21.15.38:0/3165919905 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fea280045a0 con 0x7fea381060f0 2023-03-09T14:37:15.874 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:15.452+0000 7fea3db17700 1 -- 172.21.15.38:0/3165919905 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea381060f0 msgr2=0x7fea381084d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:37:15.874 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:15.452+0000 7fea3db17700 1 --2- 172.21.15.38:0/3165919905 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea381060f0 0x7fea381084d0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fea280098a0 tx=0x7fea28009bb0 comp rx=0 tx=0).stop 2023-03-09T14:37:15.874 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:15.452+0000 7fea3db17700 1 -- 172.21.15.38:0/3165919905 shutdown_connections 2023-03-09T14:37:15.874 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:15.452+0000 7fea3db17700 1 --2- 172.21.15.38:0/3165919905 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea381060f0 0x7fea381084d0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:37:15.874 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:15.452+0000 7fea3db17700 1 -- 172.21.15.38:0/3165919905 >> 172.21.15.38:0/3165919905 conn(0x7fea380ffb10 msgr2=0x7fea38101f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:37:15.874 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:15.452+0000 7fea3db17700 1 -- 172.21.15.38:0/3165919905 shutdown_connections 2023-03-09T14:37:15.875 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:15.452+0000 7fea3db17700 1 -- 172.21.15.38:0/3165919905 wait complete. 2023-03-09T14:37:15.875 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:15.453+0000 7fea3db17700 1 Processor -- start 2023-03-09T14:37:15.875 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:15.453+0000 7fea3db17700 1 -- start start 2023-03-09T14:37:15.875 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:15.453+0000 7fea3db17700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea381060f0 0x7fea38198a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:37:15.875 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:15.453+0000 7fea3db17700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea38198fc0 con 0x7fea381060f0 2023-03-09T14:37:15.875 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:15.453+0000 7fea377fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea381060f0 0x7fea38198a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:37:15.875 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:15.453+0000 7fea377fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea381060f0 0x7fea38198a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35332/0 (socket says 172.21.15.38:35332) 2023-03-09T14:37:15.876 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:15.453+0000 7fea377fe700 1 -- 172.21.15.38:0/1180118381 learned_addr learned my addr 172.21.15.38:0/1180118381 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:37:15.876 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:15.454+0000 7fea377fe700 1 -- 172.21.15.38:0/1180118381 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea28009550 con 0x7fea381060f0 2023-03-09T14:37:15.876 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:15.454+0000 7fea377fe700 1 --2- 172.21.15.38:0/1180118381 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea381060f0 0x7fea38198a80 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fea28004000 tx=0x7fea28004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:37:15.876 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:15.454+0000 7fea34ff9700 1 -- 172.21.15.38:0/1180118381 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fea28011630 con 0x7fea381060f0 2023-03-09T14:37:15.876 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:15.454+0000 7fea34ff9700 1 -- 172.21.15.38:0/1180118381 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea28011790 con 0x7fea381060f0 2023-03-09T14:37:15.876 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:15.454+0000 7fea3db17700 1 -- 172.21.15.38:0/1180118381 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea38108cc0 con 0x7fea381060f0 2023-03-09T14:37:15.876 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:15.454+0000 7fea34ff9700 1 -- 172.21.15.38:0/1180118381 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fea28011a30 con 0x7fea381060f0 2023-03-09T14:37:15.876 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:15.454+0000 7fea3db17700 1 -- 172.21.15.38:0/1180118381 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea38108b30 con 0x7fea381060f0 2023-03-09T14:37:15.877 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:15.455+0000 7fea34ff9700 1 -- 172.21.15.38:0/1180118381 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7fea28017020 con 0x7fea381060f0 2023-03-09T14:37:15.877 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:15.455+0000 7fea34ff9700 1 -- 172.21.15.38:0/1180118381 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fea2801c450 con 0x7fea381060f0 2023-03-09T14:37:15.877 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:15.455+0000 7fea3db17700 1 -- 172.21.15.38:0/1180118381 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea381928a0 con 0x7fea381060f0 2023-03-09T14:37:15.877 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:15.458+0000 7fea34ff9700 1 -- 172.21.15.38:0/1180118381 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72437 (secure 0 0 0) 0x7fea2801b450 con 0x7fea381060f0 2023-03-09T14:37:15.877 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:15.590+0000 7fea3db17700 1 -- 172.21.15.38:0/1180118381 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fea38061db0 con 0x7fea381060f0 2023-03-09T14:37:15.877 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:15.590+0000 7fea34ff9700 1 -- 172.21.15.38:0/1180118381 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+317 (secure 0 0 0) 0x7fea38061db0 con 0x7fea381060f0 2023-03-09T14:37:15.877 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:15.591+0000 7fea3db17700 1 -- 172.21.15.38:0/1180118381 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea381060f0 msgr2=0x7fea38198a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:37:15.878 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:15.591+0000 7fea3db17700 1 --2- 172.21.15.38:0/1180118381 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea381060f0 0x7fea38198a80 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fea28004000 tx=0x7fea28004750 comp rx=0 tx=0).stop 2023-03-09T14:37:15.878 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:15.591+0000 7fea3db17700 1 -- 172.21.15.38:0/1180118381 shutdown_connections 2023-03-09T14:37:15.878 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:15.591+0000 7fea3db17700 1 --2- 172.21.15.38:0/1180118381 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea381060f0 0x7fea38198a80 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:37:15.878 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:15.591+0000 7fea3db17700 1 -- 172.21.15.38:0/1180118381 >> 172.21.15.38:0/1180118381 conn(0x7fea380ffb10 msgr2=0x7fea381013c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:37:15.878 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:15.591+0000 7fea3db17700 1 -- 172.21.15.38:0/1180118381 shutdown_connections 2023-03-09T14:37:15.878 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:15.591+0000 7fea3db17700 1 -- 172.21.15.38:0/1180118381 wait complete. 2023-03-09T14:37:15.878 INFO:teuthology.orchestra.run.smithi038.stdout:mon is available 2023-03-09T14:37:15.878 INFO:teuthology.orchestra.run.smithi038.stdout:Assimilating anything we can from ceph.conf... 2023-03-09T14:37:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:15 smithi038 ceph-mon[91925]: mon.a is new leader, mons a in quorum (ranks 0) 2023-03-09T14:37:16.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:15 smithi038 ceph-mon[91925]: monmap e1: 1 mons at {a=[v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0]} removed_ranks: {} 2023-03-09T14:37:16.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:15 smithi038 ceph-mon[91925]: fsmap 2023-03-09T14:37:16.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:15 smithi038 ceph-mon[91925]: osdmap e1: 0 total, 0 up, 0 in 2023-03-09T14:37:16.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:15 smithi038 ceph-mon[91925]: mgrmap e1: no daemons active 2023-03-09T14:37:16.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:15 smithi038 ceph-mon[91925]: from='client.? 172.21.15.38:0/1180118381' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2023-03-09T14:37:16.915 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2023-03-09T14:37:16.915 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout [global] 2023-03-09T14:37:16.915 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout fsid = c6733fe2-be87-11ed-9af2-001a4aab830c 2023-03-09T14:37:16.915 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.38:3300,v1:172.21.15.38:6789] 2023-03-09T14:37:16.915 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2023-03-09T14:37:16.916 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2023-03-09T14:37:16.916 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2023-03-09T14:37:16.916 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2023-03-09T14:37:16.916 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2023-03-09T14:37:16.916 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout [mgr] 2023-03-09T14:37:16.916 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2023-03-09T14:37:16.916 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2023-03-09T14:37:16.916 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout [osd] 2023-03-09T14:37:16.917 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2023-03-09T14:37:16.917 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2023-03-09T14:37:16.917 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2023-03-09T14:37:16.917 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2023-03-09T14:37:16.917 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2023-03-09T14:37:16.917 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:16.361+0000 7f4bd168d700 1 Processor -- start 2023-03-09T14:37:16.917 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:16.361+0000 7f4bd168d700 1 -- start start 2023-03-09T14:37:16.918 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:16.361+0000 7f4bd168d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4bcc079cd0 0x7f4bcc108180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:37:16.918 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:16.361+0000 7f4bd168d700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bcc1086c0 con 0x7f4bcc079cd0 2023-03-09T14:37:16.918 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:16.361+0000 7f4bcaffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4bcc079cd0 0x7f4bcc108180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:37:16.918 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:16.361+0000 7f4bcaffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4bcc079cd0 0x7f4bcc108180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35334/0 (socket says 172.21.15.38:35334) 2023-03-09T14:37:16.918 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:16.361+0000 7f4bcaffd700 1 -- 172.21.15.38:0/38292900 learned_addr learned my addr 172.21.15.38:0/38292900 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:37:16.918 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:16.362+0000 7f4bcaffd700 1 -- 172.21.15.38:0/38292900 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bcc108800 con 0x7f4bcc079cd0 2023-03-09T14:37:16.918 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:16.362+0000 7f4bcaffd700 1 --2- 172.21.15.38:0/38292900 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4bcc079cd0 0x7f4bcc108180 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc0098a0 tx=0x7f4bbc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19ff849344b42975 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:37:16.919 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:16.362+0000 7f4bc9ffb700 1 -- 172.21.15.38:0/38292900 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4bbc004160 con 0x7f4bcc079cd0 2023-03-09T14:37:16.919 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:16.362+0000 7f4bc9ffb700 1 -- 172.21.15.38:0/38292900 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bbc0042c0 con 0x7f4bcc079cd0 2023-03-09T14:37:16.919 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:16.362+0000 7f4bc9ffb700 1 -- 172.21.15.38:0/38292900 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4bbc0045a0 con 0x7f4bcc079cd0 2023-03-09T14:37:16.919 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:16.363+0000 7f4bd168d700 1 -- 172.21.15.38:0/38292900 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4bcc079cd0 msgr2=0x7f4bcc108180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:37:16.919 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:16.363+0000 7f4bd168d700 1 --2- 172.21.15.38:0/38292900 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4bcc079cd0 0x7f4bcc108180 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc0098a0 tx=0x7f4bbc009bb0 comp rx=0 tx=0).stop 2023-03-09T14:37:16.919 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:16.363+0000 7f4bd168d700 1 -- 172.21.15.38:0/38292900 shutdown_connections 2023-03-09T14:37:16.919 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:16.363+0000 7f4bd168d700 1 --2- 172.21.15.38:0/38292900 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4bcc079cd0 0x7f4bcc108180 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:37:16.919 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:16.363+0000 7f4bd168d700 1 -- 172.21.15.38:0/38292900 >> 172.21.15.38:0/38292900 conn(0x7f4bcc0ffb10 msgr2=0x7f4bcc101f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:37:16.920 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:16.363+0000 7f4bd168d700 1 -- 172.21.15.38:0/38292900 shutdown_connections 2023-03-09T14:37:16.920 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:16.363+0000 7f4bd168d700 1 -- 172.21.15.38:0/38292900 wait complete. 2023-03-09T14:37:16.920 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:16.363+0000 7f4bd168d700 1 Processor -- start 2023-03-09T14:37:16.920 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:16.364+0000 7f4bd168d700 1 -- start start 2023-03-09T14:37:16.920 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:16.364+0000 7f4bd168d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4bcc079cd0 0x7f4bcc198a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:37:16.920 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:16.364+0000 7f4bd168d700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bcc198f80 con 0x7f4bcc079cd0 2023-03-09T14:37:16.920 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:16.364+0000 7f4bcaffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4bcc079cd0 0x7f4bcc198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:37:16.921 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:16.364+0000 7f4bcaffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4bcc079cd0 0x7f4bcc198a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35336/0 (socket says 172.21.15.38:35336) 2023-03-09T14:37:16.921 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:16.364+0000 7f4bcaffd700 1 -- 172.21.15.38:0/1044168762 learned_addr learned my addr 172.21.15.38:0/1044168762 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:37:16.921 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:16.364+0000 7f4bcaffd700 1 -- 172.21.15.38:0/1044168762 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bbc009550 con 0x7f4bcc079cd0 2023-03-09T14:37:16.921 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:16.364+0000 7f4bcaffd700 1 --2- 172.21.15.38:0/1044168762 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4bcc079cd0 0x7f4bcc198a40 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc004000 tx=0x7f4bbc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:37:16.921 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:16.364+0000 7f4bb3fff700 1 -- 172.21.15.38:0/1044168762 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4bbc011630 con 0x7f4bcc079cd0 2023-03-09T14:37:16.921 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:16.365+0000 7f4bb3fff700 1 -- 172.21.15.38:0/1044168762 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bbc011790 con 0x7f4bcc079cd0 2023-03-09T14:37:16.921 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:16.365+0000 7f4bb3fff700 1 -- 172.21.15.38:0/1044168762 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4bbc011a30 con 0x7f4bcc079cd0 2023-03-09T14:37:16.921 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:16.365+0000 7f4bd168d700 1 -- 172.21.15.38:0/1044168762 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bcc108ca0 con 0x7f4bcc079cd0 2023-03-09T14:37:16.922 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:16.365+0000 7f4bd168d700 1 -- 172.21.15.38:0/1044168762 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bcc108b10 con 0x7f4bcc079cd0 2023-03-09T14:37:16.922 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:16.365+0000 7f4bb3fff700 1 -- 172.21.15.38:0/1044168762 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7f4bbc017020 con 0x7f4bcc079cd0 2023-03-09T14:37:16.922 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:16.365+0000 7f4bb3fff700 1 -- 172.21.15.38:0/1044168762 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f4bbc01c450 con 0x7f4bcc079cd0 2023-03-09T14:37:16.922 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:16.366+0000 7f4bd168d700 1 -- 172.21.15.38:0/1044168762 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bcc108e30 con 0x7f4bcc079cd0 2023-03-09T14:37:16.922 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:16.368+0000 7f4bb3fff700 1 -- 172.21.15.38:0/1044168762 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72437 (secure 0 0 0) 0x7f4bbc01b450 con 0x7f4bcc079cd0 2023-03-09T14:37:16.922 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:16.495+0000 7f4bd168d700 1 -- 172.21.15.38:0/1044168762 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f4bcc1928a0 con 0x7f4bcc079cd0 2023-03-09T14:37:16.922 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:16.498+0000 7f4bb3fff700 1 -- 172.21.15.38:0/1044168762 <== mon.0 v2:172.21.15.38:3300/0 7 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4bbc01b5f0 con 0x7f4bcc079cd0 2023-03-09T14:37:16.923 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:16.499+0000 7f4bb3fff700 1 -- 172.21.15.38:0/1044168762 <== mon.0 v2:172.21.15.38:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+429 (secure 0 0 0) 0x7f4bbc024630 con 0x7f4bcc079cd0 2023-03-09T14:37:16.923 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:16.499+0000 7f4bd168d700 1 -- 172.21.15.38:0/1044168762 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4bcc079cd0 msgr2=0x7f4bcc198a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:37:16.923 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:16.499+0000 7f4bd168d700 1 --2- 172.21.15.38:0/1044168762 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4bcc079cd0 0x7f4bcc198a40 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc004000 tx=0x7f4bbc004750 comp rx=0 tx=0).stop 2023-03-09T14:37:16.923 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:16.500+0000 7f4bd168d700 1 -- 172.21.15.38:0/1044168762 shutdown_connections 2023-03-09T14:37:16.923 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:16.500+0000 7f4bd168d700 1 --2- 172.21.15.38:0/1044168762 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4bcc079cd0 0x7f4bcc198a40 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:37:16.923 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:16.500+0000 7f4bd168d700 1 -- 172.21.15.38:0/1044168762 >> 172.21.15.38:0/1044168762 conn(0x7f4bcc0ffb10 msgr2=0x7f4bcc101350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:37:16.923 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:16.500+0000 7f4bd168d700 1 -- 172.21.15.38:0/1044168762 shutdown_connections 2023-03-09T14:37:16.923 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:16.500+0000 7f4bd168d700 1 -- 172.21.15.38:0/1044168762 wait complete. 2023-03-09T14:37:16.924 INFO:teuthology.orchestra.run.smithi038.stdout:Generating new minimal ceph.conf... 2023-03-09T14:37:17.137 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:16 smithi038 ceph-mon[91925]: from='client.? 172.21.15.38:0/1044168762' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-03-09T14:37:17.137 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:16 smithi038 ceph-mon[91925]: from='client.? 172.21.15.38:0/1044168762' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-03-09T14:37:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:17 smithi038 ceph-mon[91925]: from='client.? 172.21.15.38:0/2801482101' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:37:18.865 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:17.405+0000 7f961e34d700 1 Processor -- start 2023-03-09T14:37:18.866 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:17.406+0000 7f961e34d700 1 -- start start 2023-03-09T14:37:18.866 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:17.406+0000 7f961e34d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9618104490 0x7f96181048a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:37:18.866 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:17.406+0000 7f961e34d700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9618104de0 con 0x7f9618104490 2023-03-09T14:37:18.866 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:17.406+0000 7f9617fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9618104490 0x7f96181048a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:37:18.866 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:17.406+0000 7f9617fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9618104490 0x7f96181048a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35338/0 (socket says 172.21.15.38:35338) 2023-03-09T14:37:18.866 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:17.406+0000 7f9617fff700 1 -- 172.21.15.38:0/1368090108 learned_addr learned my addr 172.21.15.38:0/1368090108 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:37:18.867 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:17.406+0000 7f9617fff700 1 -- 172.21.15.38:0/1368090108 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9618104f20 con 0x7f9618104490 2023-03-09T14:37:18.867 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:17.406+0000 7f9617fff700 1 --2- 172.21.15.38:0/1368090108 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9618104490 0x7f96181048a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f960800bb40 tx=0x7f960800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eaf03b02d7a75afe server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:37:18.867 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:17.407+0000 7f9616ffd700 1 -- 172.21.15.38:0/1368090108 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f960800dbe0 con 0x7f9618104490 2023-03-09T14:37:18.867 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:17.407+0000 7f9616ffd700 1 -- 172.21.15.38:0/1368090108 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f96080044d0 con 0x7f9618104490 2023-03-09T14:37:18.867 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:17.407+0000 7f9616ffd700 1 -- 172.21.15.38:0/1368090108 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9608010470 con 0x7f9618104490 2023-03-09T14:37:18.867 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:17.407+0000 7f961e34d700 1 -- 172.21.15.38:0/1368090108 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9618104490 msgr2=0x7f96181048a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:37:18.867 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:17.407+0000 7f961e34d700 1 --2- 172.21.15.38:0/1368090108 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9618104490 0x7f96181048a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f960800bb40 tx=0x7f960800be50 comp rx=0 tx=0).stop 2023-03-09T14:37:18.868 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:17.407+0000 7f961e34d700 1 -- 172.21.15.38:0/1368090108 shutdown_connections 2023-03-09T14:37:18.868 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:17.407+0000 7f961e34d700 1 --2- 172.21.15.38:0/1368090108 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9618104490 0x7f96181048a0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:37:18.868 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:17.407+0000 7f961e34d700 1 -- 172.21.15.38:0/1368090108 >> 172.21.15.38:0/1368090108 conn(0x7f96180ffb10 msgr2=0x7f9618101f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:37:18.868 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:17.407+0000 7f961e34d700 1 -- 172.21.15.38:0/1368090108 shutdown_connections 2023-03-09T14:37:18.868 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:17.407+0000 7f961e34d700 1 -- 172.21.15.38:0/1368090108 wait complete. 2023-03-09T14:37:18.868 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:17.408+0000 7f961e34d700 1 Processor -- start 2023-03-09T14:37:18.868 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:17.408+0000 7f961e34d700 1 -- start start 2023-03-09T14:37:18.869 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:17.408+0000 7f961e34d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9618104490 0x7f961819b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:37:18.869 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:17.408+0000 7f961e34d700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f961819b990 con 0x7f9618104490 2023-03-09T14:37:18.869 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:17.408+0000 7f9617fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9618104490 0x7f961819b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:37:18.869 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:17.408+0000 7f9617fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9618104490 0x7f961819b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35340/0 (socket says 172.21.15.38:35340) 2023-03-09T14:37:18.869 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:17.408+0000 7f9617fff700 1 -- 172.21.15.38:0/2801482101 learned_addr learned my addr 172.21.15.38:0/2801482101 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:37:18.870 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:17.409+0000 7f9617fff700 1 -- 172.21.15.38:0/2801482101 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f960800b7f0 con 0x7f9618104490 2023-03-09T14:37:18.870 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:17.409+0000 7f9617fff700 1 --2- 172.21.15.38:0/2801482101 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9618104490 0x7f961819b450 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9608003a80 tx=0x7f9608003df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:37:18.870 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:17.409+0000 7f96157fa700 1 -- 172.21.15.38:0/2801482101 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f96080106e0 con 0x7f9618104490 2023-03-09T14:37:18.870 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:17.409+0000 7f96157fa700 1 -- 172.21.15.38:0/2801482101 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9608010ce0 con 0x7f9618104490 2023-03-09T14:37:18.870 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:17.409+0000 7f961e34d700 1 -- 172.21.15.38:0/2801482101 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9618106520 con 0x7f9618104490 2023-03-09T14:37:18.870 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:17.409+0000 7f96157fa700 1 -- 172.21.15.38:0/2801482101 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9608021c60 con 0x7f9618104490 2023-03-09T14:37:18.870 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:17.409+0000 7f961e34d700 1 -- 172.21.15.38:0/2801482101 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9618106390 con 0x7f9618104490 2023-03-09T14:37:18.871 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:17.409+0000 7f96157fa700 1 -- 172.21.15.38:0/2801482101 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7f960802a3f0 con 0x7f9618104490 2023-03-09T14:37:18.871 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:17.409+0000 7f96157fa700 1 -- 172.21.15.38:0/2801482101 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f9608019430 con 0x7f9618104490 2023-03-09T14:37:18.871 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:17.410+0000 7f961e34d700 1 -- 172.21.15.38:0/2801482101 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9618078ff0 con 0x7f9618104490 2023-03-09T14:37:18.871 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:17.413+0000 7f96157fa700 1 -- 172.21.15.38:0/2801482101 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72437 (secure 0 0 0) 0x7f960802f060 con 0x7f9618104490 2023-03-09T14:37:18.871 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:17.538+0000 7f961e34d700 1 -- 172.21.15.38:0/2801482101 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f961802cf40 con 0x7f9618104490 2023-03-09T14:37:18.871 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:17.539+0000 7f96157fa700 1 -- 172.21.15.38:0/2801482101 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f9608021410 con 0x7f9618104490 2023-03-09T14:37:18.871 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:17.540+0000 7f961e34d700 1 -- 172.21.15.38:0/2801482101 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9618104490 msgr2=0x7f961819b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:37:18.872 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:17.540+0000 7f961e34d700 1 --2- 172.21.15.38:0/2801482101 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9618104490 0x7f961819b450 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9608003a80 tx=0x7f9608003df0 comp rx=0 tx=0).stop 2023-03-09T14:37:18.872 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:17.540+0000 7f961e34d700 1 -- 172.21.15.38:0/2801482101 shutdown_connections 2023-03-09T14:37:18.872 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:17.540+0000 7f961e34d700 1 --2- 172.21.15.38:0/2801482101 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9618104490 0x7f961819b450 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:37:18.872 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:17.540+0000 7f961e34d700 1 -- 172.21.15.38:0/2801482101 >> 172.21.15.38:0/2801482101 conn(0x7f96180ffb10 msgr2=0x7f9618192300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:37:18.872 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:17.540+0000 7f961e34d700 1 -- 172.21.15.38:0/2801482101 shutdown_connections 2023-03-09T14:37:18.872 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:17.540+0000 7f961e34d700 1 -- 172.21.15.38:0/2801482101 wait complete. 2023-03-09T14:37:18.872 INFO:teuthology.orchestra.run.smithi038.stdout:Restarting the monitor... 2023-03-09T14:37:19.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:18 smithi038 systemd[1]: Stopping Ceph mon.a for c6733fe2-be87-11ed-9af2-001a4aab830c... 2023-03-09T14:37:20.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:19 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mon-a[91901]: 2023-03-09T14:37:19.771+0000 7fce7f1e4700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-03-09T14:37:20.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:19 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mon-a[91901]: 2023-03-09T14:37:19.771+0000 7fce7f1e4700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-03-09T14:37:21.275 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:21 smithi038 bash[92258]: ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mon-a 2023-03-09T14:37:21.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:21 smithi038 systemd[1]: ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@mon.a.service: Succeeded. 2023-03-09T14:37:21.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:21 smithi038 systemd[1]: Stopped Ceph mon.a for c6733fe2-be87-11ed-9af2-001a4aab830c. 2023-03-09T14:37:21.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:21 smithi038 systemd[1]: Starting Ceph mon.a for c6733fe2-be87-11ed-9af2-001a4aab830c... 2023-03-09T14:37:22.062 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:21 smithi038 podman[92381]: 2023-03-09T14:37:22.062 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 bash[92381]: 46f003fc444fdfb4dd280ad9312a9ee7a6a601fc1de93f2985975b0799f4775d 2023-03-09T14:37:22.063 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: set uid:gid to 167:167 (ceph:ceph) 2023-03-09T14:37:22.063 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev), process ceph-mon, pid 7 2023-03-09T14:37:22.063 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: pidfile_write: ignore empty --pid-file 2023-03-09T14:37:22.063 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: load: jerasure load: lrc 2023-03-09T14:37:22.063 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: RocksDB version: 7.9.2 2023-03-09T14:37:22.063 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Git sha 0 2023-03-09T14:37:22.064 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Compile date 2023-03-08 17:02:48 2023-03-09T14:37:22.064 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: DB SUMMARY 2023-03-09T14:37:22.064 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: DB Session ID: 3L3H50R62FOF4P0ZGIYS 2023-03-09T14:37:22.064 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: CURRENT file: CURRENT 2023-03-09T14:37:22.064 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: IDENTITY file: IDENTITY 2023-03-09T14:37:22.064 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2023-03-09T14:37:22.064 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2023-03-09T14:37:22.064 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 97554 ; 2023-03-09T14:37:22.065 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.error_if_exists: 0 2023-03-09T14:37:22.065 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.create_if_missing: 0 2023-03-09T14:37:22.065 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.paranoid_checks: 1 2023-03-09T14:37:22.065 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.flush_verify_memtable_count: 1 2023-03-09T14:37:22.065 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-03-09T14:37:22.065 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-03-09T14:37:22.065 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.env: 0x55ba53f24720 2023-03-09T14:37:22.066 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.fs: PosixFileSystem 2023-03-09T14:37:22.066 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.info_log: 0x55ba55505540 2023-03-09T14:37:22.066 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.max_file_opening_threads: 16 2023-03-09T14:37:22.066 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.statistics: (nil) 2023-03-09T14:37:22.066 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.use_fsync: 0 2023-03-09T14:37:22.066 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.max_log_file_size: 0 2023-03-09T14:37:22.066 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-03-09T14:37:22.066 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.log_file_time_to_roll: 0 2023-03-09T14:37:22.067 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.keep_log_file_num: 1000 2023-03-09T14:37:22.067 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.recycle_log_file_num: 0 2023-03-09T14:37:22.068 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.allow_fallocate: 1 2023-03-09T14:37:22.068 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.allow_mmap_reads: 0 2023-03-09T14:37:22.068 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.allow_mmap_writes: 0 2023-03-09T14:37:22.068 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.use_direct_reads: 0 2023-03-09T14:37:22.068 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-03-09T14:37:22.068 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.create_missing_column_families: 0 2023-03-09T14:37:22.069 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.db_log_dir: 2023-03-09T14:37:22.069 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.wal_dir: 2023-03-09T14:37:22.069 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.table_cache_numshardbits: 6 2023-03-09T14:37:22.069 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.WAL_ttl_seconds: 0 2023-03-09T14:37:22.070 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.WAL_size_limit_MB: 0 2023-03-09T14:37:22.070 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-03-09T14:37:22.070 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-03-09T14:37:22.070 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.is_fd_close_on_exec: 1 2023-03-09T14:37:22.070 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.advise_random_on_open: 1 2023-03-09T14:37:22.070 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.db_write_buffer_size: 0 2023-03-09T14:37:22.070 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.write_buffer_manager: 0x55ba554dc5a0 2023-03-09T14:37:22.071 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-03-09T14:37:22.071 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-03-09T14:37:22.071 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.use_adaptive_mutex: 0 2023-03-09T14:37:22.071 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.rate_limiter: (nil) 2023-03-09T14:37:22.071 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-03-09T14:37:22.072 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.wal_recovery_mode: 2 2023-03-09T14:37:22.072 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.enable_thread_tracking: 0 2023-03-09T14:37:22.072 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.enable_pipelined_write: 0 2023-03-09T14:37:22.072 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.unordered_write: 0 2023-03-09T14:37:22.072 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-03-09T14:37:22.073 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-03-09T14:37:22.073 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-03-09T14:37:22.073 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-03-09T14:37:22.073 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.row_cache: None 2023-03-09T14:37:22.073 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.wal_filter: None 2023-03-09T14:37:22.073 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-03-09T14:37:22.073 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.allow_ingest_behind: 0 2023-03-09T14:37:22.074 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.two_write_queues: 0 2023-03-09T14:37:22.074 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.manual_wal_flush: 0 2023-03-09T14:37:22.074 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.wal_compression: 0 2023-03-09T14:37:22.074 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.atomic_flush: 0 2023-03-09T14:37:22.074 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-03-09T14:37:22.074 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.persist_stats_to_disk: 0 2023-03-09T14:37:22.074 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.write_dbid_to_manifest: 0 2023-03-09T14:37:22.074 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.log_readahead_size: 0 2023-03-09T14:37:22.075 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-03-09T14:37:22.075 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.best_efforts_recovery: 0 2023-03-09T14:37:22.075 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-03-09T14:37:22.075 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-03-09T14:37:22.075 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.allow_data_in_errors: 0 2023-03-09T14:37:22.076 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.db_host_id: __hostname__ 2023-03-09T14:37:22.076 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.enforce_single_del_contracts: true 2023-03-09T14:37:22.076 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.max_background_jobs: 2 2023-03-09T14:37:22.077 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.max_background_compactions: -1 2023-03-09T14:37:22.077 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.max_subcompactions: 1 2023-03-09T14:37:22.077 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-03-09T14:37:22.077 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-03-09T14:37:22.077 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.delayed_write_rate : 16777216 2023-03-09T14:37:22.077 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.max_total_wal_size: 0 2023-03-09T14:37:22.078 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-03-09T14:37:22.078 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.stats_dump_period_sec: 600 2023-03-09T14:37:22.078 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.stats_persist_period_sec: 600 2023-03-09T14:37:22.078 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-03-09T14:37:22.078 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.max_open_files: -1 2023-03-09T14:37:22.078 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.bytes_per_sync: 0 2023-03-09T14:37:22.078 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.wal_bytes_per_sync: 0 2023-03-09T14:37:22.078 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.strict_bytes_per_sync: 0 2023-03-09T14:37:22.079 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.compaction_readahead_size: 0 2023-03-09T14:37:22.079 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.max_background_flushes: -1 2023-03-09T14:37:22.079 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Compression algorithms supported: 2023-03-09T14:37:22.079 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-03-09T14:37:22.079 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: kZSTD supported: 0 2023-03-09T14:37:22.079 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: kXpressCompression supported: 0 2023-03-09T14:37:22.079 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: kLZ4HCCompression supported: 1 2023-03-09T14:37:22.079 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: kZlibCompression supported: 1 2023-03-09T14:37:22.080 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: kSnappyCompression supported: 1 2023-03-09T14:37:22.080 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: kLZ4Compression supported: 1 2023-03-09T14:37:22.080 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: kBZip2Compression supported: 0 2023-03-09T14:37:22.080 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Fast CRC32 supported: Supported on x86 2023-03-09T14:37:22.080 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: DMutex implementation: pthread_mutex_t 2023-03-09T14:37:22.080 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2023-03-09T14:37:22.080 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-03-09T14:37:22.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-03-09T14:37:22.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.merge_operator: 2023-03-09T14:37:22.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.compaction_filter: None 2023-03-09T14:37:22.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.compaction_filter_factory: None 2023-03-09T14:37:22.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.sst_partitioner_factory: None 2023-03-09T14:37:22.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.memtable_factory: SkipListFactory 2023-03-09T14:37:22.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.table_factory: BlockBasedTable 2023-03-09T14:37:22.081 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55ba55505640) 2023-03-09T14:37:22.082 INFO:journalctl@ceph.mon.a.smithi038.stdout: cache_index_and_filter_blocks: 1 2023-03-09T14:37:22.082 INFO:journalctl@ceph.mon.a.smithi038.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-03-09T14:37:22.082 INFO:journalctl@ceph.mon.a.smithi038.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-03-09T14:37:22.082 INFO:journalctl@ceph.mon.a.smithi038.stdout: pin_top_level_index_and_filter: 1 2023-03-09T14:37:22.082 INFO:journalctl@ceph.mon.a.smithi038.stdout: index_type: 0 2023-03-09T14:37:22.082 INFO:journalctl@ceph.mon.a.smithi038.stdout: data_block_index_type: 0 2023-03-09T14:37:22.082 INFO:journalctl@ceph.mon.a.smithi038.stdout: index_shortening: 1 2023-03-09T14:37:22.083 INFO:journalctl@ceph.mon.a.smithi038.stdout: data_block_hash_table_util_ratio: 0.750000 2023-03-09T14:37:22.083 INFO:journalctl@ceph.mon.a.smithi038.stdout: checksum: 4 2023-03-09T14:37:22.083 INFO:journalctl@ceph.mon.a.smithi038.stdout: no_block_cache: 0 2023-03-09T14:37:22.083 INFO:journalctl@ceph.mon.a.smithi038.stdout: block_cache: 0x55ba5555d1f0 2023-03-09T14:37:22.083 INFO:journalctl@ceph.mon.a.smithi038.stdout: block_cache_name: BinnedLRUCache 2023-03-09T14:37:22.083 INFO:journalctl@ceph.mon.a.smithi038.stdout: block_cache_options: 2023-03-09T14:37:22.083 INFO:journalctl@ceph.mon.a.smithi038.stdout: capacity : 536870912 2023-03-09T14:37:22.084 INFO:journalctl@ceph.mon.a.smithi038.stdout: num_shard_bits : 4 2023-03-09T14:37:22.084 INFO:journalctl@ceph.mon.a.smithi038.stdout: strict_capacity_limit : 0 2023-03-09T14:37:22.084 INFO:journalctl@ceph.mon.a.smithi038.stdout: high_pri_pool_ratio: 0.000 2023-03-09T14:37:22.084 INFO:journalctl@ceph.mon.a.smithi038.stdout: block_cache_compressed: (nil) 2023-03-09T14:37:22.084 INFO:journalctl@ceph.mon.a.smithi038.stdout: persistent_cache: (nil) 2023-03-09T14:37:22.084 INFO:journalctl@ceph.mon.a.smithi038.stdout: block_size: 4096 2023-03-09T14:37:22.084 INFO:journalctl@ceph.mon.a.smithi038.stdout: block_size_deviation: 10 2023-03-09T14:37:22.084 INFO:journalctl@ceph.mon.a.smithi038.stdout: block_restart_interval: 16 2023-03-09T14:37:22.085 INFO:journalctl@ceph.mon.a.smithi038.stdout: index_block_restart_interval: 1 2023-03-09T14:37:22.085 INFO:journalctl@ceph.mon.a.smithi038.stdout: metadata_block_size: 4096 2023-03-09T14:37:22.085 INFO:journalctl@ceph.mon.a.smithi038.stdout: partition_filters: 0 2023-03-09T14:37:22.085 INFO:journalctl@ceph.mon.a.smithi038.stdout: use_delta_encoding: 1 2023-03-09T14:37:22.085 INFO:journalctl@ceph.mon.a.smithi038.stdout: filter_policy: bloomfilter 2023-03-09T14:37:22.085 INFO:journalctl@ceph.mon.a.smithi038.stdout: whole_key_filtering: 1 2023-03-09T14:37:22.085 INFO:journalctl@ceph.mon.a.smithi038.stdout: verify_compression: 0 2023-03-09T14:37:22.085 INFO:journalctl@ceph.mon.a.smithi038.stdout: read_amp_bytes_per_bit: 0 2023-03-09T14:37:22.086 INFO:journalctl@ceph.mon.a.smithi038.stdout: format_version: 5 2023-03-09T14:37:22.086 INFO:journalctl@ceph.mon.a.smithi038.stdout: enable_index_compression: 1 2023-03-09T14:37:22.086 INFO:journalctl@ceph.mon.a.smithi038.stdout: block_align: 0 2023-03-09T14:37:22.086 INFO:journalctl@ceph.mon.a.smithi038.stdout: max_auto_readahead_size: 262144 2023-03-09T14:37:22.086 INFO:journalctl@ceph.mon.a.smithi038.stdout: prepopulate_block_cache: 0 2023-03-09T14:37:22.086 INFO:journalctl@ceph.mon.a.smithi038.stdout: initial_auto_readahead_size: 8192 2023-03-09T14:37:22.086 INFO:journalctl@ceph.mon.a.smithi038.stdout: num_file_reads_for_auto_readahead: 2 2023-03-09T14:37:22.086 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.write_buffer_size: 33554432 2023-03-09T14:37:22.087 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.max_write_buffer_number: 2 2023-03-09T14:37:22.087 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.compression: NoCompression 2023-03-09T14:37:22.087 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.bottommost_compression: Disabled 2023-03-09T14:37:22.087 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.prefix_extractor: nullptr 2023-03-09T14:37:22.087 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-03-09T14:37:22.087 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.num_levels: 7 2023-03-09T14:37:22.087 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-03-09T14:37:22.087 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-03-09T14:37:22.088 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-03-09T14:37:22.088 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-03-09T14:37:22.088 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-03-09T14:37:22.088 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-03-09T14:37:22.088 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-03-09T14:37:22.088 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-03-09T14:37:22.088 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-03-09T14:37:22.089 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-03-09T14:37:22.089 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-03-09T14:37:22.089 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-03-09T14:37:22.089 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.compression_opts.window_bits: -14 2023-03-09T14:37:22.089 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.compression_opts.level: 32767 2023-03-09T14:37:22.089 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.compression_opts.strategy: 0 2023-03-09T14:37:22.089 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-03-09T14:37:22.089 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-03-09T14:37:22.090 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-03-09T14:37:22.090 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-03-09T14:37:22.090 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.compression_opts.enabled: false 2023-03-09T14:37:22.090 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-03-09T14:37:22.090 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-03-09T14:37:22.090 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-03-09T14:37:22.090 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-03-09T14:37:22.090 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.target_file_size_base: 67108864 2023-03-09T14:37:22.091 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.target_file_size_multiplier: 1 2023-03-09T14:37:22.091 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-03-09T14:37:22.091 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-03-09T14:37:22.091 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-03-09T14:37:22.091 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-03-09T14:37:22.091 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-03-09T14:37:22.091 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-03-09T14:37:22.092 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-03-09T14:37:22.092 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-03-09T14:37:22.092 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-03-09T14:37:22.092 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-03-09T14:37:22.092 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-03-09T14:37:22.093 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-03-09T14:37:22.093 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-03-09T14:37:22.093 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.arena_block_size: 1048576 2023-03-09T14:37:22.093 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-03-09T14:37:22.093 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-03-09T14:37:22.094 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.disable_auto_compactions: 0 2023-03-09T14:37:22.094 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-03-09T14:37:22.094 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-03-09T14:37:22.094 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-03-09T14:37:22.094 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-03-09T14:37:22.094 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-03-09T14:37:22.094 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-03-09T14:37:22.094 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-03-09T14:37:22.095 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-03-09T14:37:22.095 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-03-09T14:37:22.095 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-03-09T14:37:22.095 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.table_properties_collectors: 2023-03-09T14:37:22.095 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.inplace_update_support: 0 2023-03-09T14:37:22.095 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.inplace_update_num_locks: 10000 2023-03-09T14:37:22.095 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-03-09T14:37:22.096 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-03-09T14:37:22.096 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.memtable_huge_page_size: 0 2023-03-09T14:37:22.096 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.bloom_locality: 0 2023-03-09T14:37:22.096 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.max_successive_merges: 0 2023-03-09T14:37:22.096 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.optimize_filters_for_hits: 0 2023-03-09T14:37:22.096 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.paranoid_file_checks: 0 2023-03-09T14:37:22.096 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.force_consistency_checks: 1 2023-03-09T14:37:22.096 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.report_bg_io_stats: 0 2023-03-09T14:37:22.097 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.ttl: 2592000 2023-03-09T14:37:22.097 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.periodic_compaction_seconds: 0 2023-03-09T14:37:22.097 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-03-09T14:37:22.097 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-03-09T14:37:22.097 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.enable_blob_files: false 2023-03-09T14:37:22.097 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.min_blob_size: 0 2023-03-09T14:37:22.097 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.blob_file_size: 268435456 2023-03-09T14:37:22.098 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.blob_compression_type: NoCompression 2023-03-09T14:37:22.098 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.enable_blob_garbage_collection: false 2023-03-09T14:37:22.098 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-03-09T14:37:22.098 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-03-09T14:37:22.098 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-03-09T14:37:22.098 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.blob_file_starting_level: 0 2023-03-09T14:37:22.098 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-03-09T14:37:22.098 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2023-03-09T14:37:22.099 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2023-03-09T14:37:22.099 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 0d9597d4-0073-489b-a420-e51085ea04a7 2023-03-09T14:37:22.099 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: EVENT_LOG_v1 {"time_micros": 1678372642013133, "job": 1, "event": "recovery_started", "wal_files": [9]} 2023-03-09T14:37:22.099 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2023-03-09T14:37:22.099 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: EVENT_LOG_v1 {"time_micros": 1678372642014752, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 92872, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 310, "table_properties": {"data_size": 90894, "index_size": 260, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13394, "raw_average_key_size": 49, "raw_value_size": 83598, "raw_average_value_size": 310, "num_data_blocks": 11, "num_entries": 269, "num_filter_entries": 269, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1678372642, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "0d9597d4-0073-489b-a420-e51085ea04a7", "db_session_id": "3L3H50R62FOF4P0ZGIYS", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2023-03-09T14:37:22.099 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: EVENT_LOG_v1 {"time_micros": 1678372642014859, "job": 1, "event": "recovery_finished"} 2023-03-09T14:37:22.099 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2023-03-09T14:37:22.100 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-03-09T14:37:22.100 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55ba555fc000 2023-03-09T14:37:22.100 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: DB pointer 0x55ba555ea000 2023-03-09T14:37:22.100 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-03-09T14:37:22.100 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-03-09T14:37:22.100 INFO:journalctl@ceph.mon.a.smithi038.stdout: ** DB Stats ** 2023-03-09T14:37:22.100 INFO:journalctl@ceph.mon.a.smithi038.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-03-09T14:37:22.101 INFO:journalctl@ceph.mon.a.smithi038.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-03-09T14:37:22.101 INFO:journalctl@ceph.mon.a.smithi038.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-03-09T14:37:22.101 INFO:journalctl@ceph.mon.a.smithi038.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-03-09T14:37:22.101 INFO:journalctl@ceph.mon.a.smithi038.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-03-09T14:37:22.101 INFO:journalctl@ceph.mon.a.smithi038.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-03-09T14:37:22.101 INFO:journalctl@ceph.mon.a.smithi038.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-03-09T14:37:22.101 INFO:journalctl@ceph.mon.a.smithi038.stdout: 2023-03-09T14:37:22.102 INFO:journalctl@ceph.mon.a.smithi038.stdout: ** Compaction Stats [default] ** 2023-03-09T14:37:22.102 INFO:journalctl@ceph.mon.a.smithi038.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-03-09T14:37:22.102 INFO:journalctl@ceph.mon.a.smithi038.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-03-09T14:37:22.102 INFO:journalctl@ceph.mon.a.smithi038.stdout: L0 2/0 92.53 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 92.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-03-09T14:37:22.102 INFO:journalctl@ceph.mon.a.smithi038.stdout: Sum 2/0 92.53 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 92.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-03-09T14:37:22.102 INFO:journalctl@ceph.mon.a.smithi038.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 92.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-03-09T14:37:22.102 INFO:journalctl@ceph.mon.a.smithi038.stdout: 2023-03-09T14:37:22.102 INFO:journalctl@ceph.mon.a.smithi038.stdout: ** Compaction Stats [default] ** 2023-03-09T14:37:22.103 INFO:journalctl@ceph.mon.a.smithi038.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-03-09T14:37:22.103 INFO:journalctl@ceph.mon.a.smithi038.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-03-09T14:37:22.103 INFO:journalctl@ceph.mon.a.smithi038.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 92.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-03-09T14:37:22.103 INFO:journalctl@ceph.mon.a.smithi038.stdout: 2023-03-09T14:37:22.103 INFO:journalctl@ceph.mon.a.smithi038.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-03-09T14:37:22.103 INFO:journalctl@ceph.mon.a.smithi038.stdout: 2023-03-09T14:37:22.103 INFO:journalctl@ceph.mon.a.smithi038.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-03-09T14:37:22.103 INFO:journalctl@ceph.mon.a.smithi038.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-03-09T14:37:22.104 INFO:journalctl@ceph.mon.a.smithi038.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-03-09T14:37:22.104 INFO:journalctl@ceph.mon.a.smithi038.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-03-09T14:37:22.104 INFO:journalctl@ceph.mon.a.smithi038.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-03-09T14:37:22.104 INFO:journalctl@ceph.mon.a.smithi038.stdout: AddFile(Keys): cumulative 0, interval 0 2023-03-09T14:37:22.104 INFO:journalctl@ceph.mon.a.smithi038.stdout: Cumulative compaction: 0.00 GB write, 11.56 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-03-09T14:37:22.104 INFO:journalctl@ceph.mon.a.smithi038.stdout: Interval compaction: 0.00 GB write, 11.56 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-03-09T14:37:22.104 INFO:journalctl@ceph.mon.a.smithi038.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-03-09T14:37:22.105 INFO:journalctl@ceph.mon.a.smithi038.stdout: Block cache BinnedLRUCache@0x55ba5555d1f0#7 capacity: 512.00 MB usage: 1.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2023-03-09T14:37:22.105 INFO:journalctl@ceph.mon.a.smithi038.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2023-03-09T14:37:22.105 INFO:journalctl@ceph.mon.a.smithi038.stdout: 2023-03-09T14:37:22.105 INFO:journalctl@ceph.mon.a.smithi038.stdout: ** File Read Latency Histogram By Level [default] ** 2023-03-09T14:37:22.105 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: starting mon.a rank 0 at public addrs [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] at bind addrs [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid c6733fe2-be87-11ed-9af2-001a4aab830c 2023-03-09T14:37:22.105 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: mon.a@-1(???) e1 preinit fsid c6733fe2-be87-11ed-9af2-001a4aab830c 2023-03-09T14:37:22.105 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: mon.a@-1(???).mds e1 new map 2023-03-09T14:37:22.105 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: mon.a@-1(???).mds e1 print_map 2023-03-09T14:37:22.106 INFO:journalctl@ceph.mon.a.smithi038.stdout: e1 2023-03-09T14:37:22.106 INFO:teuthology.orchestra.run.smithi038.stdout:Setting mon public_network to 172.21.0.0/20 2023-03-09T14:37:22.106 INFO:journalctl@ceph.mon.a.smithi038.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-03-09T14:37:22.107 INFO:journalctl@ceph.mon.a.smithi038.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-03-09T14:37:22.107 INFO:journalctl@ceph.mon.a.smithi038.stdout: legacy client fscid: -1 2023-03-09T14:37:22.107 INFO:journalctl@ceph.mon.a.smithi038.stdout: 2023-03-09T14:37:22.107 INFO:journalctl@ceph.mon.a.smithi038.stdout: No filesystems configured 2023-03-09T14:37:22.107 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2023-03-09T14:37:22.107 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-03-09T14:37:22.107 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-03-09T14:37:22.107 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-03-09T14:37:22.108 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2023-03-09T14:37:22.108 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: expand_channel_meta expand map: {default=false} 2023-03-09T14:37:22.108 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: expand_channel_meta from 'false' to 'false' 2023-03-09T14:37:22.108 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: expand_channel_meta expanded map: {default=false} 2023-03-09T14:37:22.108 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: expand_channel_meta expand map: {default=info} 2023-03-09T14:37:22.108 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: expand_channel_meta from 'info' to 'info' 2023-03-09T14:37:22.108 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: expand_channel_meta expanded map: {default=info} 2023-03-09T14:37:22.108 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: expand_channel_meta expand map: {default=daemon} 2023-03-09T14:37:22.109 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: expand_channel_meta from 'daemon' to 'daemon' 2023-03-09T14:37:22.109 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: expand_channel_meta expanded map: {default=daemon} 2023-03-09T14:37:22.109 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: expand_channel_meta expand map: {default=debug} 2023-03-09T14:37:22.109 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: expand_channel_meta from 'debug' to 'debug' 2023-03-09T14:37:22.109 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: expand_channel_meta expanded map: {default=debug} 2023-03-09T14:37:22.109 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 systemd[1]: Started Ceph mon.a for c6733fe2-be87-11ed-9af2-001a4aab830c. 2023-03-09T14:37:22.322 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: mon.a is new leader, mons a in quorum (ranks 0) 2023-03-09T14:37:22.322 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: monmap e1: 1 mons at {a=[v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0]} removed_ranks: {} 2023-03-09T14:37:22.322 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: fsmap 2023-03-09T14:37:22.322 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: osdmap e1: 0 total, 0 up, 0 in 2023-03-09T14:37:22.323 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:22 smithi038 ceph-mon[92422]: mgrmap e1: no daemons active 2023-03-09T14:37:23.114 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:22.577+0000 7face752d700 1 Processor -- start 2023-03-09T14:37:23.117 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:22.577+0000 7face752d700 1 -- start start 2023-03-09T14:37:23.118 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:22.577+0000 7face752d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7face0104490 0x7face01048a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:37:23.118 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:22.577+0000 7face752d700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face0104de0 con 0x7face0104490 2023-03-09T14:37:23.118 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:22.578+0000 7face52c9700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7face0104490 0x7face01048a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:37:23.118 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:22.578+0000 7face52c9700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7face0104490 0x7face01048a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35342/0 (socket says 172.21.15.38:35342) 2023-03-09T14:37:23.118 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:22.578+0000 7face52c9700 1 -- 172.21.15.38:0/1427173369 learned_addr learned my addr 172.21.15.38:0/1427173369 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:37:23.118 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:22.578+0000 7face52c9700 1 -- 172.21.15.38:0/1427173369 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7face0104f20 con 0x7face0104490 2023-03-09T14:37:23.118 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:22.578+0000 7face52c9700 1 --2- 172.21.15.38:0/1427173369 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7face0104490 0x7face01048a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7facd00098a0 tx=0x7facd0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0535a16bf34e3c5 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:37:23.119 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:22.579+0000 7facd7fff700 1 -- 172.21.15.38:0/1427173369 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7facd0004160 con 0x7face0104490 2023-03-09T14:37:23.119 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:22.579+0000 7facd7fff700 1 -- 172.21.15.38:0/1427173369 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7facd00042c0 con 0x7face0104490 2023-03-09T14:37:23.119 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:22.579+0000 7facd7fff700 1 -- 172.21.15.38:0/1427173369 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7facd0011a00 con 0x7face0104490 2023-03-09T14:37:23.119 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:22.579+0000 7face752d700 1 -- 172.21.15.38:0/1427173369 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7face0104490 msgr2=0x7face01048a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:37:23.119 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:22.579+0000 7face752d700 1 --2- 172.21.15.38:0/1427173369 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7face0104490 0x7face01048a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7facd00098a0 tx=0x7facd0009bb0 comp rx=0 tx=0).stop 2023-03-09T14:37:23.119 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:22.579+0000 7face752d700 1 -- 172.21.15.38:0/1427173369 shutdown_connections 2023-03-09T14:37:23.119 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:22.579+0000 7face752d700 1 --2- 172.21.15.38:0/1427173369 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7face0104490 0x7face01048a0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:37:23.120 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:22.579+0000 7face752d700 1 -- 172.21.15.38:0/1427173369 >> 172.21.15.38:0/1427173369 conn(0x7face00ffb10 msgr2=0x7face0101f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:37:23.120 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:22.579+0000 7face752d700 1 -- 172.21.15.38:0/1427173369 shutdown_connections 2023-03-09T14:37:23.120 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:22.579+0000 7face752d700 1 -- 172.21.15.38:0/1427173369 wait complete. 2023-03-09T14:37:23.120 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:22.580+0000 7face752d700 1 Processor -- start 2023-03-09T14:37:23.120 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:22.580+0000 7face752d700 1 -- start start 2023-03-09T14:37:23.120 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:22.580+0000 7face752d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7face0104490 0x7face018f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:37:23.120 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:22.580+0000 7face752d700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face018fbd0 con 0x7face0104490 2023-03-09T14:37:23.120 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:22.580+0000 7face52c9700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7face0104490 0x7face018f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:37:23.121 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:22.580+0000 7face52c9700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7face0104490 0x7face018f690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35344/0 (socket says 172.21.15.38:35344) 2023-03-09T14:37:23.121 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:22.580+0000 7face52c9700 1 -- 172.21.15.38:0/3769612974 learned_addr learned my addr 172.21.15.38:0/3769612974 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:37:23.121 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:22.581+0000 7face52c9700 1 -- 172.21.15.38:0/3769612974 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facd0009550 con 0x7face0104490 2023-03-09T14:37:23.121 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:22.581+0000 7face52c9700 1 --2- 172.21.15.38:0/3769612974 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7face0104490 0x7face018f690 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7facd0003eb0 tx=0x7facd0011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:37:23.121 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:22.581+0000 7facd67fc700 1 -- 172.21.15.38:0/3769612974 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7facd00038b0 con 0x7face0104490 2023-03-09T14:37:23.121 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:22.581+0000 7facd67fc700 1 -- 172.21.15.38:0/3769612974 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7facd0003a10 con 0x7face0104490 2023-03-09T14:37:23.121 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:22.581+0000 7facd67fc700 1 -- 172.21.15.38:0/3769612974 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7facd0019550 con 0x7face0104490 2023-03-09T14:37:23.122 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:22.581+0000 7face752d700 1 -- 172.21.15.38:0/3769612974 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7face0106520 con 0x7face0104490 2023-03-09T14:37:23.122 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:22.581+0000 7face752d700 1 -- 172.21.15.38:0/3769612974 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7face0106390 con 0x7face0104490 2023-03-09T14:37:23.122 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:22.581+0000 7facd67fc700 1 -- 172.21.15.38:0/3769612974 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7facd002a020 con 0x7face0104490 2023-03-09T14:37:23.122 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:22.581+0000 7facd67fc700 1 -- 172.21.15.38:0/3769612974 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7facd002db50 con 0x7face0104490 2023-03-09T14:37:23.122 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:22.582+0000 7face752d700 1 -- 172.21.15.38:0/3769612974 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7face0061db0 con 0x7face0104490 2023-03-09T14:37:23.122 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:22.585+0000 7facd67fc700 1 -- 172.21.15.38:0/3769612974 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72437 (secure 0 0 0) 0x7facd0003b80 con 0x7face0104490 2023-03-09T14:37:23.123 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:22.709+0000 7face752d700 1 -- 172.21.15.38:0/3769612974 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7face018e620 con 0x7face0104490 2023-03-09T14:37:23.123 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:22.712+0000 7facd67fc700 1 -- 172.21.15.38:0/3769612974 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7facd0019c90 con 0x7face0104490 2023-03-09T14:37:23.123 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:22.713+0000 7face752d700 1 -- 172.21.15.38:0/3769612974 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7face0104490 msgr2=0x7face018f690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:37:23.123 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:22.713+0000 7face752d700 1 --2- 172.21.15.38:0/3769612974 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7face0104490 0x7face018f690 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7facd0003eb0 tx=0x7facd0011ad0 comp rx=0 tx=0).stop 2023-03-09T14:37:23.123 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:22.713+0000 7face752d700 1 -- 172.21.15.38:0/3769612974 shutdown_connections 2023-03-09T14:37:23.123 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:22.713+0000 7face752d700 1 --2- 172.21.15.38:0/3769612974 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7face0104490 0x7face018f690 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:37:23.123 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:22.713+0000 7face752d700 1 -- 172.21.15.38:0/3769612974 >> 172.21.15.38:0/3769612974 conn(0x7face00ffb10 msgr2=0x7face0079130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:37:23.124 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:22.713+0000 7face752d700 1 -- 172.21.15.38:0/3769612974 shutdown_connections 2023-03-09T14:37:23.124 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:22.713+0000 7face752d700 1 -- 172.21.15.38:0/3769612974 wait complete. 2023-03-09T14:37:23.124 INFO:teuthology.orchestra.run.smithi038.stdout:Wrote config to /etc/ceph/ceph.conf 2023-03-09T14:37:23.124 INFO:teuthology.orchestra.run.smithi038.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2023-03-09T14:37:23.124 INFO:teuthology.orchestra.run.smithi038.stdout:Creating mgr... 2023-03-09T14:37:23.124 INFO:teuthology.orchestra.run.smithi038.stdout:Verifying port 9283 ... 2023-03-09T14:37:23.125 INFO:teuthology.orchestra.run.smithi038.stdout:Verifying port 8765 ... 2023-03-09T14:37:23.255 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@mgr.x 2023-03-09T14:37:23.255 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stderr Failed to reset failed state of unit ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@mgr.x.service: Unit ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@mgr.x.service not loaded. 2023-03-09T14:37:23.352 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-c6733fe2-be87-11ed-9af2-001a4aab830c.target.wants/ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@mgr.x.service → /etc/systemd/system/ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@.service. 2023-03-09T14:37:23.500 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:23 smithi038 systemd[1]: Starting Ceph mgr.x for c6733fe2-be87-11ed-9af2-001a4aab830c... 2023-03-09T14:37:23.929 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:23 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/3769612974' entity='client.admin' 2023-03-09T14:37:23.929 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:23 smithi038 podman[92652]: 2023-03-09T14:37:23.963 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-03-09T14:37:23.963 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout disabled 2023-03-09T14:37:23.971 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-03-09T14:37:23.971 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout inactive 2023-03-09T14:37:23.971 INFO:teuthology.orchestra.run.smithi038.stdout:firewalld.service is not enabled 2023-03-09T14:37:23.971 INFO:teuthology.orchestra.run.smithi038.stdout:Not possible to enable service . firewalld.service is not available 2023-03-09T14:37:23.979 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-03-09T14:37:23.980 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout disabled 2023-03-09T14:37:23.987 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-03-09T14:37:23.988 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout inactive 2023-03-09T14:37:23.988 INFO:teuthology.orchestra.run.smithi038.stdout:firewalld.service is not enabled 2023-03-09T14:37:23.988 INFO:teuthology.orchestra.run.smithi038.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2023-03-09T14:37:23.988 INFO:teuthology.orchestra.run.smithi038.stdout:Waiting for mgr to start... 2023-03-09T14:37:23.988 INFO:teuthology.orchestra.run.smithi038.stdout:Waiting for mgr... 2023-03-09T14:37:24.197 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:23 smithi038 bash[92652]: 6564c06a60865ad7f5c4460c0b83e0b7cedeb1ebd539296b7c33afc89b223007 2023-03-09T14:37:24.197 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:23 smithi038 systemd[1]: Started Ceph mgr.x for c6733fe2-be87-11ed-9af2-001a4aab830c. 2023-03-09T14:37:24.198 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:24 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:37:24.075+0000 7fb19b078200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-03-09T14:37:24.198 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:24 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:37:24.173+0000 7fb19b078200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-03-09T14:37:24.501 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:24 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:37:24.288+0000 7fb19b078200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-03-09T14:37:24.501 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:24 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:37:24.401+0000 7fb19b078200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-03-09T14:37:25.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:24 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/683095388' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-03-09T14:37:25.001 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:24 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:37:24.618+0000 7fb19b078200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-03-09T14:37:25.038 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2023-03-09T14:37:25.038 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout { 2023-03-09T14:37:25.038 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "fsid": "c6733fe2-be87-11ed-9af2-001a4aab830c", 2023-03-09T14:37:25.038 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "health": { 2023-03-09T14:37:25.039 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-03-09T14:37:25.039 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-03-09T14:37:25.039 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-03-09T14:37:25.039 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2023-03-09T14:37:25.039 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-03-09T14:37:25.039 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-03-09T14:37:25.039 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 0 2023-03-09T14:37:25.040 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2023-03-09T14:37:25.040 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-03-09T14:37:25.040 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "a" 2023-03-09T14:37:25.040 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2023-03-09T14:37:25.040 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2023-03-09T14:37:25.040 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "monmap": { 2023-03-09T14:37:25.040 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-03-09T14:37:25.040 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-03-09T14:37:25.041 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-03-09T14:37:25.041 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2023-03-09T14:37:25.041 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-03-09T14:37:25.041 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-03-09T14:37:25.041 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-03-09T14:37:25.041 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-03-09T14:37:25.041 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-03-09T14:37:25.048 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-03-09T14:37:25.048 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-03-09T14:37:25.049 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-03-09T14:37:25.049 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2023-03-09T14:37:25.049 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-03-09T14:37:25.050 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-03-09T14:37:25.050 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-03-09T14:37:25.050 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-03-09T14:37:25.050 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-03-09T14:37:25.050 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-03-09T14:37:25.050 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-03-09T14:37:25.050 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-03-09T14:37:25.050 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-03-09T14:37:25.051 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2023-03-09T14:37:25.051 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-03-09T14:37:25.051 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-03-09T14:37:25.051 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-03-09T14:37:25.051 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-03-09T14:37:25.051 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2023-03-09T14:37:25.051 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-03-09T14:37:25.051 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "available": false, 2023-03-09T14:37:25.052 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-03-09T14:37:25.052 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "modules": [ 2023-03-09T14:37:25.052 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "iostat", 2023-03-09T14:37:25.052 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "nfs", 2023-03-09T14:37:25.052 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "restful" 2023-03-09T14:37:25.052 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2023-03-09T14:37:25.052 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "services": {} 2023-03-09T14:37:25.053 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2023-03-09T14:37:25.053 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-03-09T14:37:25.053 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-03-09T14:37:25.053 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "modified": "2023-03-09T14:37:14.948462+0000", 2023-03-09T14:37:25.053 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "services": {} 2023-03-09T14:37:25.054 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2023-03-09T14:37:25.054 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-03-09T14:37:25.054 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout } 2023-03-09T14:37:25.054 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:24.500+0000 7f9a07900700 1 Processor -- start 2023-03-09T14:37:25.055 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:24.500+0000 7f9a07900700 1 -- start start 2023-03-09T14:37:25.055 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:24.500+0000 7f9a07900700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a001060f0 0x7f9a001084d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:37:25.055 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:24.500+0000 7f9a07900700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a00079d50 con 0x7f9a001060f0 2023-03-09T14:37:25.055 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:24.501+0000 7f9a0569c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a001060f0 0x7f9a001084d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:37:25.055 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:24.501+0000 7f9a0569c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a001060f0 0x7f9a001084d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35350/0 (socket says 172.21.15.38:35350) 2023-03-09T14:37:25.055 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:24.501+0000 7f9a0569c700 1 -- 172.21.15.38:0/2365541069 learned_addr learned my addr 172.21.15.38:0/2365541069 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:37:25.055 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:24.501+0000 7f9a0569c700 1 -- 172.21.15.38:0/2365541069 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a00079e90 con 0x7f9a001060f0 2023-03-09T14:37:25.056 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:24.501+0000 7f9a0569c700 1 --2- 172.21.15.38:0/2365541069 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a001060f0 0x7f9a001084d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f99fc0098a0 tx=0x7f99fc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=448b0732f4e07aa6 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:37:25.056 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:24.501+0000 7f99f7fff700 1 -- 172.21.15.38:0/2365541069 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f99fc004160 con 0x7f9a001060f0 2023-03-09T14:37:25.056 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:24.501+0000 7f99f7fff700 1 -- 172.21.15.38:0/2365541069 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f99fc0042c0 con 0x7f9a001060f0 2023-03-09T14:37:25.056 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:24.501+0000 7f99f7fff700 1 -- 172.21.15.38:0/2365541069 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f99fc011a00 con 0x7f9a001060f0 2023-03-09T14:37:25.056 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:24.502+0000 7f9a07900700 1 -- 172.21.15.38:0/2365541069 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a001060f0 msgr2=0x7f9a001084d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:37:25.056 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:24.502+0000 7f9a07900700 1 --2- 172.21.15.38:0/2365541069 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a001060f0 0x7f9a001084d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f99fc0098a0 tx=0x7f99fc009bb0 comp rx=0 tx=0).stop 2023-03-09T14:37:25.056 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:24.502+0000 7f9a07900700 1 -- 172.21.15.38:0/2365541069 shutdown_connections 2023-03-09T14:37:25.057 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:24.502+0000 7f9a07900700 1 --2- 172.21.15.38:0/2365541069 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a001060f0 0x7f9a001084d0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:37:25.057 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:24.502+0000 7f9a07900700 1 -- 172.21.15.38:0/2365541069 >> 172.21.15.38:0/2365541069 conn(0x7f9a000ffb50 msgr2=0x7f9a00101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:37:25.057 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:24.502+0000 7f9a07900700 1 -- 172.21.15.38:0/2365541069 shutdown_connections 2023-03-09T14:37:25.057 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:24.502+0000 7f9a07900700 1 -- 172.21.15.38:0/2365541069 wait complete. 2023-03-09T14:37:25.057 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:24.503+0000 7f9a07900700 1 Processor -- start 2023-03-09T14:37:25.057 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:24.503+0000 7f9a07900700 1 -- start start 2023-03-09T14:37:25.057 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:24.503+0000 7f9a07900700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a001060f0 0x7f9a00105a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:37:25.058 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:24.503+0000 7f9a07900700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a00105fa0 con 0x7f9a001060f0 2023-03-09T14:37:25.058 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:24.504+0000 7f9a0569c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a001060f0 0x7f9a00105a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:37:25.058 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:24.504+0000 7f9a0569c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a001060f0 0x7f9a00105a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35352/0 (socket says 172.21.15.38:35352) 2023-03-09T14:37:25.058 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:24.504+0000 7f9a0569c700 1 -- 172.21.15.38:0/683095388 learned_addr learned my addr 172.21.15.38:0/683095388 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:37:25.058 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:24.504+0000 7f9a0569c700 1 -- 172.21.15.38:0/683095388 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99fc009550 con 0x7f9a001060f0 2023-03-09T14:37:25.058 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:24.504+0000 7f9a0569c700 1 --2- 172.21.15.38:0/683095388 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a001060f0 0x7f9a00105a60 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f99fc009fb0 tx=0x7f99fc011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:37:25.058 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:24.504+0000 7f99f67fc700 1 -- 172.21.15.38:0/683095388 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f99fc003930 con 0x7f9a001060f0 2023-03-09T14:37:25.058 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:24.504+0000 7f99f67fc700 1 -- 172.21.15.38:0/683095388 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f99fc003a90 con 0x7f9a001060f0 2023-03-09T14:37:25.059 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:24.504+0000 7f9a07900700 1 -- 172.21.15.38:0/683095388 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a00109810 con 0x7f9a001060f0 2023-03-09T14:37:25.059 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:24.504+0000 7f99f67fc700 1 -- 172.21.15.38:0/683095388 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f99fc0195e0 con 0x7f9a001060f0 2023-03-09T14:37:25.059 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:24.504+0000 7f9a07900700 1 -- 172.21.15.38:0/683095388 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a00109680 con 0x7f9a001060f0 2023-03-09T14:37:25.059 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:24.505+0000 7f99f67fc700 1 -- 172.21.15.38:0/683095388 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7f99fc02a020 con 0x7f9a001060f0 2023-03-09T14:37:25.059 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:24.505+0000 7f99f67fc700 1 -- 172.21.15.38:0/683095388 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f99fc02dc00 con 0x7f9a001060f0 2023-03-09T14:37:25.059 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:24.505+0000 7f9a07900700 1 -- 172.21.15.38:0/683095388 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a00061db0 con 0x7f9a001060f0 2023-03-09T14:37:25.059 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:24.508+0000 7f99f67fc700 1 -- 172.21.15.38:0/683095388 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72437 (secure 0 0 0) 0x7f99fc003c00 con 0x7f9a001060f0 2023-03-09T14:37:25.060 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:24.641+0000 7f9a07900700 1 -- 172.21.15.38:0/683095388 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f9a00104f00 con 0x7f9a001060f0 2023-03-09T14:37:25.060 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:24.641+0000 7f99f67fc700 1 -- 172.21.15.38:0/683095388 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7f99fc02de60 con 0x7f9a001060f0 2023-03-09T14:37:25.060 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:24.642+0000 7f9a07900700 1 -- 172.21.15.38:0/683095388 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a001060f0 msgr2=0x7f9a00105a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:37:25.060 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:24.642+0000 7f9a07900700 1 --2- 172.21.15.38:0/683095388 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a001060f0 0x7f9a00105a60 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f99fc009fb0 tx=0x7f99fc011ad0 comp rx=0 tx=0).stop 2023-03-09T14:37:25.060 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:24.642+0000 7f9a07900700 1 -- 172.21.15.38:0/683095388 shutdown_connections 2023-03-09T14:37:25.060 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:24.642+0000 7f9a07900700 1 --2- 172.21.15.38:0/683095388 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a001060f0 0x7f9a00105a60 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:37:25.060 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:24.642+0000 7f9a07900700 1 -- 172.21.15.38:0/683095388 >> 172.21.15.38:0/683095388 conn(0x7f9a000ffb50 msgr2=0x7f9a001008d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:37:25.060 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:24.642+0000 7f9a07900700 1 -- 172.21.15.38:0/683095388 shutdown_connections 2023-03-09T14:37:25.061 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:24.642+0000 7f9a07900700 1 -- 172.21.15.38:0/683095388 wait complete. 2023-03-09T14:37:25.061 INFO:teuthology.orchestra.run.smithi038.stdout:mgr not available, waiting (1/15)... 2023-03-09T14:37:25.750 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:25 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:37:25.395+0000 7fb19b078200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-03-09T14:37:25.750 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:25 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:37:25.504+0000 7fb19b078200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-03-09T14:37:25.751 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:25 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:37:25.612+0000 7fb19b078200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-03-09T14:37:26.250 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:25 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:37:25.787+0000 7fb19b078200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-03-09T14:37:26.750 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:26 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:37:26.312+0000 7fb19b078200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-03-09T14:37:27.500 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:27 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:37:27.027+0000 7fb19b078200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-03-09T14:37:27.500 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:27 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:37:27.130+0000 7fb19b078200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-03-09T14:37:28.413 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:28 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:37:28.085+0000 7fb19b078200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-03-09T14:37:28.414 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:28 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:37:28.197+0000 7fb19b078200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-03-09T14:37:28.728 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:28 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:37:28.414+0000 7fb19b078200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-03-09T14:37:29.000 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:28 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:37:28.912+0000 7fb19b078200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-03-09T14:37:29.333 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:29 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:37:29.018+0000 7fb19b078200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-03-09T14:37:29.334 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:29 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:37:29.205+0000 7fb19b078200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-03-09T14:37:29.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:29 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/3606703392' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-03-09T14:37:29.751 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:29 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:37:29.536+0000 7fb19b078200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-03-09T14:37:30.750 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:30 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:37:30.362+0000 7fb19b078200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-03-09T14:37:30.750 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:30 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:37:30.520+0000 7fb19b078200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-03-09T14:37:31.167 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:30 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:37:30.760+0000 7fb19b078200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-03-09T14:37:31.167 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:30 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:37:30.896+0000 7fb19b078200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-03-09T14:37:31.294 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2023-03-09T14:37:31.297 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout { 2023-03-09T14:37:31.297 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "fsid": "c6733fe2-be87-11ed-9af2-001a4aab830c", 2023-03-09T14:37:31.297 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "health": { 2023-03-09T14:37:31.297 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-03-09T14:37:31.297 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-03-09T14:37:31.297 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-03-09T14:37:31.298 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2023-03-09T14:37:31.298 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-03-09T14:37:31.298 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-03-09T14:37:31.298 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 0 2023-03-09T14:37:31.298 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2023-03-09T14:37:31.298 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-03-09T14:37:31.298 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "a" 2023-03-09T14:37:31.298 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2023-03-09T14:37:31.299 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum_age": 7, 2023-03-09T14:37:31.299 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "monmap": { 2023-03-09T14:37:31.299 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-03-09T14:37:31.299 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-03-09T14:37:31.299 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-03-09T14:37:31.299 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2023-03-09T14:37:31.299 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-03-09T14:37:31.299 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-03-09T14:37:31.300 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-03-09T14:37:31.300 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-03-09T14:37:31.300 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-03-09T14:37:31.305 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-03-09T14:37:31.306 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-03-09T14:37:31.306 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-03-09T14:37:31.306 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2023-03-09T14:37:31.306 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-03-09T14:37:31.306 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-03-09T14:37:31.306 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-03-09T14:37:31.307 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-03-09T14:37:31.307 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-03-09T14:37:31.307 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-03-09T14:37:31.307 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-03-09T14:37:31.307 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-03-09T14:37:31.307 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-03-09T14:37:31.307 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2023-03-09T14:37:31.307 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-03-09T14:37:31.308 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-03-09T14:37:31.308 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-03-09T14:37:31.308 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-03-09T14:37:31.308 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2023-03-09T14:37:31.308 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-03-09T14:37:31.308 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "available": false, 2023-03-09T14:37:31.308 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-03-09T14:37:31.308 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "modules": [ 2023-03-09T14:37:31.309 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "iostat", 2023-03-09T14:37:31.309 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "nfs", 2023-03-09T14:37:31.309 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "restful" 2023-03-09T14:37:31.309 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2023-03-09T14:37:31.309 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "services": {} 2023-03-09T14:37:31.309 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2023-03-09T14:37:31.309 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-03-09T14:37:31.309 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-03-09T14:37:31.310 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "modified": "2023-03-09T14:37:14.948462+0000", 2023-03-09T14:37:31.310 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "services": {} 2023-03-09T14:37:31.310 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2023-03-09T14:37:31.310 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-03-09T14:37:31.310 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout } 2023-03-09T14:37:31.310 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:29.165+0000 7fc4cf73e700 1 Processor -- start 2023-03-09T14:37:31.310 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:29.165+0000 7fc4cf73e700 1 -- start start 2023-03-09T14:37:31.311 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:29.166+0000 7fc4cf73e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4c8106090 0x7fc4c8108470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:37:31.311 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:29.166+0000 7fc4cf73e700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c8079db0 con 0x7fc4c8106090 2023-03-09T14:37:31.311 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:29.166+0000 7fc4cd4da700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4c8106090 0x7fc4c8108470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:37:31.311 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:29.166+0000 7fc4cd4da700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4c8106090 0x7fc4c8108470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35354/0 (socket says 172.21.15.38:35354) 2023-03-09T14:37:31.311 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:29.166+0000 7fc4cd4da700 1 -- 172.21.15.38:0/2147101237 learned_addr learned my addr 172.21.15.38:0/2147101237 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:37:31.311 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:29.166+0000 7fc4cd4da700 1 -- 172.21.15.38:0/2147101237 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4c8079ef0 con 0x7fc4c8106090 2023-03-09T14:37:31.311 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:29.166+0000 7fc4cd4da700 1 --2- 172.21.15.38:0/2147101237 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4c8106090 0x7fc4c8108470 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc4c40098a0 tx=0x7fc4c4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78333c0076855d2f server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:37:31.311 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:29.166+0000 7fc4bffff700 1 -- 172.21.15.38:0/2147101237 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc4c4004160 con 0x7fc4c8106090 2023-03-09T14:37:31.312 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:29.166+0000 7fc4bffff700 1 -- 172.21.15.38:0/2147101237 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc4c40042c0 con 0x7fc4c8106090 2023-03-09T14:37:31.312 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:29.166+0000 7fc4bffff700 1 -- 172.21.15.38:0/2147101237 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc4c4011a00 con 0x7fc4c8106090 2023-03-09T14:37:31.312 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:29.167+0000 7fc4cf73e700 1 -- 172.21.15.38:0/2147101237 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4c8106090 msgr2=0x7fc4c8108470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:37:31.312 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:29.167+0000 7fc4cf73e700 1 --2- 172.21.15.38:0/2147101237 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4c8106090 0x7fc4c8108470 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc4c40098a0 tx=0x7fc4c4009bb0 comp rx=0 tx=0).stop 2023-03-09T14:37:31.312 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:29.167+0000 7fc4cf73e700 1 -- 172.21.15.38:0/2147101237 shutdown_connections 2023-03-09T14:37:31.312 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:29.167+0000 7fc4cf73e700 1 --2- 172.21.15.38:0/2147101237 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4c8106090 0x7fc4c8108470 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:37:31.312 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:29.167+0000 7fc4cf73e700 1 -- 172.21.15.38:0/2147101237 >> 172.21.15.38:0/2147101237 conn(0x7fc4c80ffb10 msgr2=0x7fc4c8101f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:37:31.312 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:29.167+0000 7fc4cf73e700 1 -- 172.21.15.38:0/2147101237 shutdown_connections 2023-03-09T14:37:31.313 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:29.167+0000 7fc4cf73e700 1 -- 172.21.15.38:0/2147101237 wait complete. 2023-03-09T14:37:31.313 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:29.168+0000 7fc4cf73e700 1 Processor -- start 2023-03-09T14:37:31.313 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:29.168+0000 7fc4cf73e700 1 -- start start 2023-03-09T14:37:31.313 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:29.168+0000 7fc4cf73e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4c8106090 0x7fc4c819b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:37:31.313 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:29.168+0000 7fc4cf73e700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4c819b9a0 con 0x7fc4c8106090 2023-03-09T14:37:31.313 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:29.168+0000 7fc4cd4da700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4c8106090 0x7fc4c819b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:37:31.313 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:29.168+0000 7fc4cd4da700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4c8106090 0x7fc4c819b460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35356/0 (socket says 172.21.15.38:35356) 2023-03-09T14:37:31.314 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:29.168+0000 7fc4cd4da700 1 -- 172.21.15.38:0/3606703392 learned_addr learned my addr 172.21.15.38:0/3606703392 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:37:31.314 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:29.168+0000 7fc4cd4da700 1 -- 172.21.15.38:0/3606703392 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4c4009550 con 0x7fc4c8106090 2023-03-09T14:37:31.314 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:29.169+0000 7fc4cd4da700 1 --2- 172.21.15.38:0/3606703392 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4c8106090 0x7fc4c819b460 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc4c4009fb0 tx=0x7fc4c4011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:37:31.314 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:29.169+0000 7fc4be7fc700 1 -- 172.21.15.38:0/3606703392 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc4c4003930 con 0x7fc4c8106090 2023-03-09T14:37:31.314 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:29.169+0000 7fc4be7fc700 1 -- 172.21.15.38:0/3606703392 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc4c4003a90 con 0x7fc4c8106090 2023-03-09T14:37:31.314 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:29.169+0000 7fc4cf73e700 1 -- 172.21.15.38:0/3606703392 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4c8109810 con 0x7fc4c8106090 2023-03-09T14:37:31.314 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:29.169+0000 7fc4be7fc700 1 -- 172.21.15.38:0/3606703392 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc4c40195e0 con 0x7fc4c8106090 2023-03-09T14:37:31.315 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:29.169+0000 7fc4cf73e700 1 -- 172.21.15.38:0/3606703392 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4c8109680 con 0x7fc4c8106090 2023-03-09T14:37:31.315 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:29.169+0000 7fc4be7fc700 1 -- 172.21.15.38:0/3606703392 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 1) v1 ==== 656+0+0 (secure 0 0 0) 0x7fc4c402a020 con 0x7fc4c8106090 2023-03-09T14:37:31.315 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:29.169+0000 7fc4be7fc700 1 -- 172.21.15.38:0/3606703392 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fc4c402dc00 con 0x7fc4c8106090 2023-03-09T14:37:31.315 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:29.170+0000 7fc4cf73e700 1 -- 172.21.15.38:0/3606703392 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4c8061db0 con 0x7fc4c8106090 2023-03-09T14:37:31.315 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:29.174+0000 7fc4be7fc700 1 -- 172.21.15.38:0/3606703392 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72437 (secure 0 0 0) 0x7fc4c4003c00 con 0x7fc4c8106090 2023-03-09T14:37:31.315 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:29.308+0000 7fc4cf73e700 1 -- 172.21.15.38:0/3606703392 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fc4c819c0f0 con 0x7fc4c8106090 2023-03-09T14:37:31.315 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:29.309+0000 7fc4be7fc700 1 -- 172.21.15.38:0/3606703392 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7fc4c4019d40 con 0x7fc4c8106090 2023-03-09T14:37:31.315 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:29.310+0000 7fc4cf73e700 1 -- 172.21.15.38:0/3606703392 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4c8106090 msgr2=0x7fc4c819b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:37:31.316 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:29.310+0000 7fc4cf73e700 1 --2- 172.21.15.38:0/3606703392 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4c8106090 0x7fc4c819b460 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc4c4009fb0 tx=0x7fc4c4011ad0 comp rx=0 tx=0).stop 2023-03-09T14:37:31.316 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:29.310+0000 7fc4cf73e700 1 -- 172.21.15.38:0/3606703392 shutdown_connections 2023-03-09T14:37:31.316 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:29.310+0000 7fc4cf73e700 1 --2- 172.21.15.38:0/3606703392 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc4c8106090 0x7fc4c819b460 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:37:31.316 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:29.310+0000 7fc4cf73e700 1 -- 172.21.15.38:0/3606703392 >> 172.21.15.38:0/3606703392 conn(0x7fc4c80ffb10 msgr2=0x7fc4c8100870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:37:31.316 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:29.310+0000 7fc4cf73e700 1 -- 172.21.15.38:0/3606703392 shutdown_connections 2023-03-09T14:37:31.316 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:29.310+0000 7fc4cf73e700 1 -- 172.21.15.38:0/3606703392 wait complete. 2023-03-09T14:37:31.316 INFO:teuthology.orchestra.run.smithi038.stdout:mgr not available, waiting (2/15)... 2023-03-09T14:37:31.500 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:31 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:37:31.164+0000 7fb19b078200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-03-09T14:37:32.500 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:32 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:37:32.200+0000 7fb19b078200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-03-09T14:37:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:33 smithi038 ceph-mon[92422]: Activating manager daemon x 2023-03-09T14:37:33.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:33 smithi038 ceph-mon[92422]: mgrmap e2: x(active, starting, since 0.00548297s) 2023-03-09T14:37:33.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:33 smithi038 ceph-mon[92422]: from='mgr.14100 172.21.15.38:0/779328319' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-03-09T14:37:33.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:33 smithi038 ceph-mon[92422]: from='mgr.14100 172.21.15.38:0/779328319' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-03-09T14:37:33.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:33 smithi038 ceph-mon[92422]: from='mgr.14100 172.21.15.38:0/779328319' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-03-09T14:37:33.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:33 smithi038 ceph-mon[92422]: from='mgr.14100 172.21.15.38:0/779328319' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-03-09T14:37:33.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:33 smithi038 ceph-mon[92422]: from='mgr.14100 172.21.15.38:0/779328319' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-03-09T14:37:33.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:33 smithi038 ceph-mon[92422]: Manager daemon x is now available 2023-03-09T14:37:33.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:33 smithi038 ceph-mon[92422]: from='mgr.14100 172.21.15.38:0/779328319' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-03-09T14:37:33.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:33 smithi038 ceph-mon[92422]: from='mgr.14100 172.21.15.38:0/779328319' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-03-09T14:37:33.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:33 smithi038 ceph-mon[92422]: from='mgr.14100 172.21.15.38:0/779328319' entity='mgr.x' 2023-03-09T14:37:33.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:33 smithi038 ceph-mon[92422]: from='mgr.14100 172.21.15.38:0/779328319' entity='mgr.x' 2023-03-09T14:37:33.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:33 smithi038 ceph-mon[92422]: from='mgr.14100 172.21.15.38:0/779328319' entity='mgr.x' 2023-03-09T14:37:34.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:34 smithi038 ceph-mon[92422]: mgrmap e3: x(active, since 1.01058s) 2023-03-09T14:37:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:36 smithi038 ceph-mon[92422]: mgrmap e4: x(active, since 3s) 2023-03-09T14:37:36.660 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2023-03-09T14:37:36.661 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout { 2023-03-09T14:37:36.661 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "fsid": "c6733fe2-be87-11ed-9af2-001a4aab830c", 2023-03-09T14:37:36.661 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "health": { 2023-03-09T14:37:36.661 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-03-09T14:37:36.661 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-03-09T14:37:36.661 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-03-09T14:37:36.661 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2023-03-09T14:37:36.662 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-03-09T14:37:36.662 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-03-09T14:37:36.662 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 0 2023-03-09T14:37:36.662 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2023-03-09T14:37:36.662 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-03-09T14:37:36.662 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "a" 2023-03-09T14:37:36.662 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2023-03-09T14:37:36.663 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "quorum_age": 14, 2023-03-09T14:37:36.663 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "monmap": { 2023-03-09T14:37:36.663 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-03-09T14:37:36.663 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-03-09T14:37:36.663 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-03-09T14:37:36.663 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2023-03-09T14:37:36.663 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-03-09T14:37:36.663 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-03-09T14:37:36.664 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-03-09T14:37:36.664 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-03-09T14:37:36.670 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-03-09T14:37:36.671 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-03-09T14:37:36.671 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-03-09T14:37:36.671 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-03-09T14:37:36.671 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2023-03-09T14:37:36.671 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-03-09T14:37:36.671 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-03-09T14:37:36.672 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-03-09T14:37:36.672 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-03-09T14:37:36.672 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-03-09T14:37:36.672 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-03-09T14:37:36.672 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-03-09T14:37:36.672 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-03-09T14:37:36.672 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-03-09T14:37:36.672 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2023-03-09T14:37:36.673 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-03-09T14:37:36.673 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-03-09T14:37:36.673 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-03-09T14:37:36.673 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-03-09T14:37:36.673 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2023-03-09T14:37:36.673 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-03-09T14:37:36.673 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "available": true, 2023-03-09T14:37:36.674 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-03-09T14:37:36.674 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "modules": [ 2023-03-09T14:37:36.674 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "iostat", 2023-03-09T14:37:36.674 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "nfs", 2023-03-09T14:37:36.674 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "restful" 2023-03-09T14:37:36.674 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ], 2023-03-09T14:37:36.674 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "services": {} 2023-03-09T14:37:36.674 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2023-03-09T14:37:36.675 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-03-09T14:37:36.675 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-03-09T14:37:36.675 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "modified": "2023-03-09T14:37:14.948462+0000", 2023-03-09T14:37:36.675 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "services": {} 2023-03-09T14:37:36.675 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout }, 2023-03-09T14:37:36.675 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-03-09T14:37:36.675 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout } 2023-03-09T14:37:36.675 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:35.788+0000 7f9b00afb700 1 Processor -- start 2023-03-09T14:37:36.676 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:35.788+0000 7f9b00afb700 1 -- start start 2023-03-09T14:37:36.676 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:35.788+0000 7f9b00afb700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9afc103ee0 0x7f9afc1042f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:37:36.676 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:35.788+0000 7f9b00afb700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9afc079d50 con 0x7f9afc103ee0 2023-03-09T14:37:36.676 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:35.789+0000 7f9afa59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9afc103ee0 0x7f9afc1042f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:37:36.676 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:35.789+0000 7f9afa59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9afc103ee0 0x7f9afc1042f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35382/0 (socket says 172.21.15.38:35382) 2023-03-09T14:37:36.676 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:35.789+0000 7f9afa59c700 1 -- 172.21.15.38:0/3486842050 learned_addr learned my addr 172.21.15.38:0/3486842050 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:37:36.676 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:35.789+0000 7f9afa59c700 1 -- 172.21.15.38:0/3486842050 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9afc079e90 con 0x7f9afc103ee0 2023-03-09T14:37:36.677 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:35.789+0000 7f9afa59c700 1 --2- 172.21.15.38:0/3486842050 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9afc103ee0 0x7f9afc1042f0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f9af00098a0 tx=0x7f9af0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=919a9553b11b4588 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:37:36.677 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:35.789+0000 7f9af959a700 1 -- 172.21.15.38:0/3486842050 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9af0004160 con 0x7f9afc103ee0 2023-03-09T14:37:36.677 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:35.789+0000 7f9af959a700 1 -- 172.21.15.38:0/3486842050 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9af00042c0 con 0x7f9afc103ee0 2023-03-09T14:37:36.677 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:35.789+0000 7f9af959a700 1 -- 172.21.15.38:0/3486842050 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9af0011a00 con 0x7f9afc103ee0 2023-03-09T14:37:36.677 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:35.790+0000 7f9b00afb700 1 -- 172.21.15.38:0/3486842050 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9afc103ee0 msgr2=0x7f9afc1042f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:37:36.677 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:35.790+0000 7f9b00afb700 1 --2- 172.21.15.38:0/3486842050 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9afc103ee0 0x7f9afc1042f0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f9af00098a0 tx=0x7f9af0009bb0 comp rx=0 tx=0).stop 2023-03-09T14:37:36.677 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:35.790+0000 7f9b00afb700 1 -- 172.21.15.38:0/3486842050 shutdown_connections 2023-03-09T14:37:36.677 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:35.790+0000 7f9b00afb700 1 --2- 172.21.15.38:0/3486842050 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9afc103ee0 0x7f9afc1042f0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:37:36.678 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:35.790+0000 7f9b00afb700 1 -- 172.21.15.38:0/3486842050 >> 172.21.15.38:0/3486842050 conn(0x7f9afc0ffb10 msgr2=0x7f9afc101f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:37:36.678 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:35.790+0000 7f9b00afb700 1 -- 172.21.15.38:0/3486842050 shutdown_connections 2023-03-09T14:37:36.678 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:35.790+0000 7f9b00afb700 1 -- 172.21.15.38:0/3486842050 wait complete. 2023-03-09T14:37:36.678 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:35.791+0000 7f9b00afb700 1 Processor -- start 2023-03-09T14:37:36.678 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:35.791+0000 7f9b00afb700 1 -- start start 2023-03-09T14:37:36.678 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:35.791+0000 7f9b00afb700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9afc103ee0 0x7f9afc19b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:37:36.678 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:35.791+0000 7f9b00afb700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9afc19b9c0 con 0x7f9afc103ee0 2023-03-09T14:37:36.679 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:35.791+0000 7f9afa59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9afc103ee0 0x7f9afc19b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:37:36.679 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:35.791+0000 7f9afa59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9afc103ee0 0x7f9afc19b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35384/0 (socket says 172.21.15.38:35384) 2023-03-09T14:37:36.679 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:35.791+0000 7f9afa59c700 1 -- 172.21.15.38:0/2886653754 learned_addr learned my addr 172.21.15.38:0/2886653754 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:37:36.679 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:35.792+0000 7f9afa59c700 1 -- 172.21.15.38:0/2886653754 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9af0009550 con 0x7f9afc103ee0 2023-03-09T14:37:36.679 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:35.792+0000 7f9afa59c700 1 --2- 172.21.15.38:0/2886653754 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9afc103ee0 0x7f9afc19b480 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f9af0006af0 tx=0x7f9af0011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:37:36.679 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:35.792+0000 7f9aeb7fe700 1 -- 172.21.15.38:0/2886653754 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9af0003a40 con 0x7f9afc103ee0 2023-03-09T14:37:36.679 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:35.792+0000 7f9aeb7fe700 1 -- 172.21.15.38:0/2886653754 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9af0003ba0 con 0x7f9afc103ee0 2023-03-09T14:37:36.679 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:35.792+0000 7f9b00afb700 1 -- 172.21.15.38:0/2886653754 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9afc105630 con 0x7f9afc103ee0 2023-03-09T14:37:36.680 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:35.792+0000 7f9aeb7fe700 1 -- 172.21.15.38:0/2886653754 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9af001a700 con 0x7f9afc103ee0 2023-03-09T14:37:36.680 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:35.792+0000 7f9b00afb700 1 -- 172.21.15.38:0/2886653754 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9afc1054a0 con 0x7f9afc103ee0 2023-03-09T14:37:36.680 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:35.793+0000 7f9aeb7fe700 1 -- 172.21.15.38:0/2886653754 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 4) v1 ==== 44607+0+0 (secure 0 0 0) 0x7f9af0017020 con 0x7f9afc103ee0 2023-03-09T14:37:36.680 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:35.793+0000 7f9aeb7fe700 1 --2- 172.21.15.38:0/2886653754 >> [v2:172.21.15.38:6800/3060474342,v1:172.21.15.38:6801/3060474342] conn(0x7f9ae4037d30 0x7f9ae403a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:37:36.680 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:35.793+0000 7f9aeb7fe700 1 -- 172.21.15.38:0/2886653754 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f9af004b530 con 0x7f9afc103ee0 2023-03-09T14:37:36.680 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:35.793+0000 7f9b00afb700 1 -- 172.21.15.38:0/2886653754 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9afc194c50 con 0x7f9afc103ee0 2023-03-09T14:37:36.680 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:35.793+0000 7f9af9d9b700 1 --2- 172.21.15.38:0/2886653754 >> [v2:172.21.15.38:6800/3060474342,v1:172.21.15.38:6801/3060474342] conn(0x7f9ae4037d30 0x7f9ae403a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:37:36.681 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:35.794+0000 7f9af9d9b700 1 --2- 172.21.15.38:0/2886653754 >> [v2:172.21.15.38:6800/3060474342,v1:172.21.15.38:6801/3060474342] conn(0x7f9ae4037d30 0x7f9ae403a1e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9aec006fd0 tx=0x7f9aec006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:37:36.681 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:35.799+0000 7f9aeb7fe700 1 -- 172.21.15.38:0/2886653754 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f9af002d3f0 con 0x7f9afc103ee0 2023-03-09T14:37:36.681 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:36.315+0000 7f9b00afb700 1 -- 172.21.15.38:0/2886653754 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f9afc19c100 con 0x7f9afc103ee0 2023-03-09T14:37:36.681 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:36.316+0000 7f9aeb7fe700 1 -- 172.21.15.38:0/2886653754 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7f9af0049020 con 0x7f9afc103ee0 2023-03-09T14:37:36.681 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:36.318+0000 7f9b00afb700 1 -- 172.21.15.38:0/2886653754 >> [v2:172.21.15.38:6800/3060474342,v1:172.21.15.38:6801/3060474342] conn(0x7f9ae4037d30 msgr2=0x7f9ae403a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:37:36.681 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:36.318+0000 7f9b00afb700 1 --2- 172.21.15.38:0/2886653754 >> [v2:172.21.15.38:6800/3060474342,v1:172.21.15.38:6801/3060474342] conn(0x7f9ae4037d30 0x7f9ae403a1e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9aec006fd0 tx=0x7f9aec006e40 comp rx=0 tx=0).stop 2023-03-09T14:37:36.681 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:36.318+0000 7f9b00afb700 1 -- 172.21.15.38:0/2886653754 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9afc103ee0 msgr2=0x7f9afc19b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:37:36.681 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:36.318+0000 7f9b00afb700 1 --2- 172.21.15.38:0/2886653754 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9afc103ee0 0x7f9afc19b480 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f9af0006af0 tx=0x7f9af0011ad0 comp rx=0 tx=0).stop 2023-03-09T14:37:36.682 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:36.318+0000 7f9b00afb700 1 -- 172.21.15.38:0/2886653754 shutdown_connections 2023-03-09T14:37:36.682 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:36.318+0000 7f9b00afb700 1 --2- 172.21.15.38:0/2886653754 >> [v2:172.21.15.38:6800/3060474342,v1:172.21.15.38:6801/3060474342] conn(0x7f9ae4037d30 0x7f9ae403a1e0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:37:36.682 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:36.318+0000 7f9b00afb700 1 --2- 172.21.15.38:0/2886653754 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9afc103ee0 0x7f9afc19b480 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:37:36.682 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:36.318+0000 7f9b00afb700 1 -- 172.21.15.38:0/2886653754 >> 172.21.15.38:0/2886653754 conn(0x7f9afc0ffb10 msgr2=0x7f9afc18faa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:37:36.682 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:36.318+0000 7f9b00afb700 1 -- 172.21.15.38:0/2886653754 shutdown_connections 2023-03-09T14:37:36.682 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:36.318+0000 7f9b00afb700 1 -- 172.21.15.38:0/2886653754 wait complete. 2023-03-09T14:37:36.682 INFO:teuthology.orchestra.run.smithi038.stdout:mgr is available 2023-03-09T14:37:37.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:37 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/2886653754' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-03-09T14:37:37.910 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2023-03-09T14:37:37.910 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout [global] 2023-03-09T14:37:37.910 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout fsid = c6733fe2-be87-11ed-9af2-001a4aab830c 2023-03-09T14:37:37.910 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2023-03-09T14:37:37.911 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2023-03-09T14:37:37.911 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2023-03-09T14:37:37.911 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2023-03-09T14:37:37.911 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2023-03-09T14:37:37.911 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout [mgr] 2023-03-09T14:37:37.911 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2023-03-09T14:37:37.911 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 2023-03-09T14:37:37.912 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout [osd] 2023-03-09T14:37:37.912 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2023-03-09T14:37:37.912 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2023-03-09T14:37:37.912 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2023-03-09T14:37:37.912 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2023-03-09T14:37:37.912 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2023-03-09T14:37:37.912 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:37.168+0000 7f95109ce700 1 Processor -- start 2023-03-09T14:37:37.912 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:37.168+0000 7f95109ce700 1 -- start start 2023-03-09T14:37:37.913 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:37.169+0000 7f95109ce700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f950c106bc0 0x7f950c106fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:37:37.913 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:37.169+0000 7f95109ce700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f950c107510 con 0x7f950c106bc0 2023-03-09T14:37:37.913 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:37.169+0000 7f950a59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f950c106bc0 0x7f950c106fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:37:37.913 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:37.169+0000 7f950a59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f950c106bc0 0x7f950c106fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35388/0 (socket says 172.21.15.38:35388) 2023-03-09T14:37:37.913 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:37.169+0000 7f950a59c700 1 -- 172.21.15.38:0/1093682010 learned_addr learned my addr 172.21.15.38:0/1093682010 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:37:37.913 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:37.169+0000 7f950a59c700 1 -- 172.21.15.38:0/1093682010 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f950c107650 con 0x7f950c106bc0 2023-03-09T14:37:37.913 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:37.169+0000 7f950a59c700 1 --2- 172.21.15.38:0/1093682010 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f950c106bc0 0x7f950c106fd0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f94fc0098a0 tx=0x7f94fc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9cb5f48c68fb877 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:37:37.914 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:37.169+0000 7f950959a700 1 -- 172.21.15.38:0/1093682010 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f94fc004160 con 0x7f950c106bc0 2023-03-09T14:37:37.914 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:37.169+0000 7f950959a700 1 -- 172.21.15.38:0/1093682010 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f94fc0042c0 con 0x7f950c106bc0 2023-03-09T14:37:37.914 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:37.169+0000 7f950959a700 1 -- 172.21.15.38:0/1093682010 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f94fc011a00 con 0x7f950c106bc0 2023-03-09T14:37:37.914 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:37.170+0000 7f95109ce700 1 -- 172.21.15.38:0/1093682010 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f950c106bc0 msgr2=0x7f950c106fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:37:37.914 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:37.170+0000 7f95109ce700 1 --2- 172.21.15.38:0/1093682010 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f950c106bc0 0x7f950c106fd0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f94fc0098a0 tx=0x7f94fc009bb0 comp rx=0 tx=0).stop 2023-03-09T14:37:37.914 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:37.170+0000 7f95109ce700 1 -- 172.21.15.38:0/1093682010 shutdown_connections 2023-03-09T14:37:37.914 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:37.170+0000 7f95109ce700 1 --2- 172.21.15.38:0/1093682010 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f950c106bc0 0x7f950c106fd0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:37:37.914 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:37.170+0000 7f95109ce700 1 -- 172.21.15.38:0/1093682010 >> 172.21.15.38:0/1093682010 conn(0x7f950c0ff260 msgr2=0x7f950c101670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:37:37.915 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:37.170+0000 7f95109ce700 1 -- 172.21.15.38:0/1093682010 shutdown_connections 2023-03-09T14:37:37.915 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:37.170+0000 7f95109ce700 1 -- 172.21.15.38:0/1093682010 wait complete. 2023-03-09T14:37:37.915 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:37.171+0000 7f95109ce700 1 Processor -- start 2023-03-09T14:37:37.915 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:37.171+0000 7f95109ce700 1 -- start start 2023-03-09T14:37:37.915 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:37.171+0000 7f95109ce700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f950c106bc0 0x7f950c199260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:37:37.915 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:37.171+0000 7f95109ce700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f950c1997a0 con 0x7f950c106bc0 2023-03-09T14:37:37.915 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:37.172+0000 7f950a59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f950c106bc0 0x7f950c199260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:37:37.916 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:37.172+0000 7f950a59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f950c106bc0 0x7f950c199260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35390/0 (socket says 172.21.15.38:35390) 2023-03-09T14:37:37.916 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:37.172+0000 7f950a59c700 1 -- 172.21.15.38:0/742423258 learned_addr learned my addr 172.21.15.38:0/742423258 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:37:37.916 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:37.172+0000 7f950a59c700 1 -- 172.21.15.38:0/742423258 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94fc009550 con 0x7f950c106bc0 2023-03-09T14:37:37.916 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:37.172+0000 7f950a59c700 1 --2- 172.21.15.38:0/742423258 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f950c106bc0 0x7f950c199260 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f94fc006b20 tx=0x7f94fc011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:37:37.916 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:37.172+0000 7f95037fe700 1 -- 172.21.15.38:0/742423258 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f94fc003b70 con 0x7f950c106bc0 2023-03-09T14:37:37.917 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:37.172+0000 7f95037fe700 1 -- 172.21.15.38:0/742423258 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f94fc003cd0 con 0x7f950c106bc0 2023-03-09T14:37:37.917 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:37.172+0000 7f95109ce700 1 -- 172.21.15.38:0/742423258 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f950c1042c0 con 0x7f950c106bc0 2023-03-09T14:37:37.917 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:37.172+0000 7f95037fe700 1 -- 172.21.15.38:0/742423258 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f94fc024760 con 0x7f950c106bc0 2023-03-09T14:37:37.917 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:37.172+0000 7f95109ce700 1 -- 172.21.15.38:0/742423258 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f950c104130 con 0x7f950c106bc0 2023-03-09T14:37:37.917 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:37.173+0000 7f95037fe700 1 -- 172.21.15.38:0/742423258 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 4) v1 ==== 44607+0+0 (secure 0 0 0) 0x7f94fc017020 con 0x7f950c106bc0 2023-03-09T14:37:37.917 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:37.174+0000 7f95037fe700 1 --2- 172.21.15.38:0/742423258 >> [v2:172.21.15.38:6800/3060474342,v1:172.21.15.38:6801/3060474342] conn(0x7f94f8037d80 0x7f94f803a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:37:37.917 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:37.174+0000 7f95037fe700 1 -- 172.21.15.38:0/742423258 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f94fc04b520 con 0x7f950c106bc0 2023-03-09T14:37:37.917 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:37.174+0000 7f9509d9b700 1 --2- 172.21.15.38:0/742423258 >> [v2:172.21.15.38:6800/3060474342,v1:172.21.15.38:6801/3060474342] conn(0x7f94f8037d80 0x7f94f803a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:37:37.918 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:37.174+0000 7f95109ce700 1 -- 172.21.15.38:0/742423258 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f950c104450 con 0x7f950c106bc0 2023-03-09T14:37:37.918 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:37.174+0000 7f9509d9b700 1 --2- 172.21.15.38:0/742423258 >> [v2:172.21.15.38:6800/3060474342,v1:172.21.15.38:6801/3060474342] conn(0x7f94f8037d80 0x7f94f803a230 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f94f4006fd0 tx=0x7f94f4006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:37:37.918 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:37.179+0000 7f95037fe700 1 -- 172.21.15.38:0/742423258 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f94fc0249d0 con 0x7f950c106bc0 2023-03-09T14:37:37.918 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:37.533+0000 7f95109ce700 1 -- 172.21.15.38:0/742423258 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f950c199cf0 con 0x7f950c106bc0 2023-03-09T14:37:37.918 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:37.534+0000 7f95037fe700 1 -- 172.21.15.38:0/742423258 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7f94fc018350 con 0x7f950c106bc0 2023-03-09T14:37:37.918 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:37.536+0000 7f95109ce700 1 -- 172.21.15.38:0/742423258 >> [v2:172.21.15.38:6800/3060474342,v1:172.21.15.38:6801/3060474342] conn(0x7f94f8037d80 msgr2=0x7f94f803a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:37:37.918 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:37.536+0000 7f95109ce700 1 --2- 172.21.15.38:0/742423258 >> [v2:172.21.15.38:6800/3060474342,v1:172.21.15.38:6801/3060474342] conn(0x7f94f8037d80 0x7f94f803a230 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f94f4006fd0 tx=0x7f94f4006e40 comp rx=0 tx=0).stop 2023-03-09T14:37:37.919 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:37.536+0000 7f95109ce700 1 -- 172.21.15.38:0/742423258 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f950c106bc0 msgr2=0x7f950c199260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:37:37.919 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:37.536+0000 7f95109ce700 1 --2- 172.21.15.38:0/742423258 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f950c106bc0 0x7f950c199260 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f94fc006b20 tx=0x7f94fc011ad0 comp rx=0 tx=0).stop 2023-03-09T14:37:37.919 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:37.536+0000 7f95109ce700 1 -- 172.21.15.38:0/742423258 shutdown_connections 2023-03-09T14:37:37.919 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:37.536+0000 7f95109ce700 1 --2- 172.21.15.38:0/742423258 >> [v2:172.21.15.38:6800/3060474342,v1:172.21.15.38:6801/3060474342] conn(0x7f94f8037d80 0x7f94f803a230 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:37:37.919 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:37.537+0000 7f95109ce700 1 --2- 172.21.15.38:0/742423258 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f950c106bc0 0x7f950c199260 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:37:37.919 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:37.537+0000 7f95109ce700 1 -- 172.21.15.38:0/742423258 >> 172.21.15.38:0/742423258 conn(0x7f950c0ff260 msgr2=0x7f950c0fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:37:37.919 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:37.537+0000 7f95109ce700 1 -- 172.21.15.38:0/742423258 shutdown_connections 2023-03-09T14:37:37.919 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:37.537+0000 7f95109ce700 1 -- 172.21.15.38:0/742423258 wait complete. 2023-03-09T14:37:37.920 INFO:teuthology.orchestra.run.smithi038.stdout:Enabling cephadm module... 2023-03-09T14:37:38.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:38 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/742423258' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-03-09T14:37:39.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:39 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/3511929667' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2023-03-09T14:37:39.501 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: ignoring --setuser ceph since I am not root 2023-03-09T14:37:39.501 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: ignoring --setgroup ceph since I am not root 2023-03-09T14:37:39.501 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:37:39.433+0000 7f58d6fd5200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-03-09T14:37:39.653 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:38.362+0000 7f9837473700 1 Processor -- start 2023-03-09T14:37:39.657 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:38.362+0000 7f9837473700 1 -- start start 2023-03-09T14:37:39.657 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:38.363+0000 7f9837473700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9830103890 0x7f9830103ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:37:39.657 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:38.363+0000 7f9837473700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98301041e0 con 0x7f9830103890 2023-03-09T14:37:39.657 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:38.363+0000 7f983520f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9830103890 0x7f9830103ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:37:39.657 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:38.363+0000 7f983520f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9830103890 0x7f9830103ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35394/0 (socket says 172.21.15.38:35394) 2023-03-09T14:37:39.658 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:38.363+0000 7f983520f700 1 -- 172.21.15.38:0/807787279 learned_addr learned my addr 172.21.15.38:0/807787279 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:37:39.658 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:38.363+0000 7f983520f700 1 -- 172.21.15.38:0/807787279 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9830104320 con 0x7f9830103890 2023-03-09T14:37:39.658 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:38.363+0000 7f983520f700 1 --2- 172.21.15.38:0/807787279 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9830103890 0x7f9830103ca0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f982c0098a0 tx=0x7f982c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c884c5fde7b59398 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:37:39.658 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:38.363+0000 7f9823fff700 1 -- 172.21.15.38:0/807787279 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f982c004160 con 0x7f9830103890 2023-03-09T14:37:39.658 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:38.363+0000 7f9823fff700 1 -- 172.21.15.38:0/807787279 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f982c0042c0 con 0x7f9830103890 2023-03-09T14:37:39.658 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:38.363+0000 7f9823fff700 1 -- 172.21.15.38:0/807787279 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f982c011a00 con 0x7f9830103890 2023-03-09T14:37:39.658 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:38.364+0000 7f9837473700 1 -- 172.21.15.38:0/807787279 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9830103890 msgr2=0x7f9830103ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:37:39.658 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:38.364+0000 7f9837473700 1 --2- 172.21.15.38:0/807787279 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9830103890 0x7f9830103ca0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f982c0098a0 tx=0x7f982c009bb0 comp rx=0 tx=0).stop 2023-03-09T14:37:39.659 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:38.364+0000 7f9837473700 1 -- 172.21.15.38:0/807787279 shutdown_connections 2023-03-09T14:37:39.659 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:38.364+0000 7f9837473700 1 --2- 172.21.15.38:0/807787279 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9830103890 0x7f9830103ca0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:37:39.659 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:38.364+0000 7f9837473700 1 -- 172.21.15.38:0/807787279 >> 172.21.15.38:0/807787279 conn(0x7f98300ff260 msgr2=0x7f9830101670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:37:39.659 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:38.364+0000 7f9837473700 1 -- 172.21.15.38:0/807787279 shutdown_connections 2023-03-09T14:37:39.659 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:38.364+0000 7f9837473700 1 -- 172.21.15.38:0/807787279 wait complete. 2023-03-09T14:37:39.659 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:38.365+0000 7f9837473700 1 Processor -- start 2023-03-09T14:37:39.659 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:38.365+0000 7f9837473700 1 -- start start 2023-03-09T14:37:39.660 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:38.365+0000 7f9837473700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9830103890 0x7f98301970d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:37:39.660 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:38.365+0000 7f9837473700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9830197610 con 0x7f9830103890 2023-03-09T14:37:39.660 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:38.365+0000 7f983520f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9830103890 0x7f98301970d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:37:39.660 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:38.365+0000 7f983520f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9830103890 0x7f98301970d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35396/0 (socket says 172.21.15.38:35396) 2023-03-09T14:37:39.660 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:38.365+0000 7f983520f700 1 -- 172.21.15.38:0/3511929667 learned_addr learned my addr 172.21.15.38:0/3511929667 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:37:39.660 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:38.365+0000 7f983520f700 1 -- 172.21.15.38:0/3511929667 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f982c009550 con 0x7f9830103890 2023-03-09T14:37:39.660 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:38.366+0000 7f983520f700 1 --2- 172.21.15.38:0/3511929667 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9830103890 0x7f98301970d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f982c006b20 tx=0x7f982c011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:37:39.660 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:38.366+0000 7f98227fc700 1 -- 172.21.15.38:0/3511929667 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f982c003b70 con 0x7f9830103890 2023-03-09T14:37:39.661 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:38.366+0000 7f98227fc700 1 -- 172.21.15.38:0/3511929667 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f982c003cd0 con 0x7f9830103890 2023-03-09T14:37:39.661 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:38.366+0000 7f9837473700 1 -- 172.21.15.38:0/3511929667 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9830107650 con 0x7f9830103890 2023-03-09T14:37:39.661 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:38.366+0000 7f98227fc700 1 -- 172.21.15.38:0/3511929667 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f982c0247c0 con 0x7f9830103890 2023-03-09T14:37:39.661 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:38.366+0000 7f9837473700 1 -- 172.21.15.38:0/3511929667 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98301074c0 con 0x7f9830103890 2023-03-09T14:37:39.661 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:38.367+0000 7f98227fc700 1 -- 172.21.15.38:0/3511929667 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 4) v1 ==== 44607+0+0 (secure 0 0 0) 0x7f982c017020 con 0x7f9830103890 2023-03-09T14:37:39.661 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:38.367+0000 7f98227fc700 1 --2- 172.21.15.38:0/3511929667 >> [v2:172.21.15.38:6800/3060474342,v1:172.21.15.38:6801/3060474342] conn(0x7f981c03c1d0 0x7f981c03e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:37:39.661 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:38.367+0000 7f98227fc700 1 -- 172.21.15.38:0/3511929667 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f982c04b360 con 0x7f9830103890 2023-03-09T14:37:39.662 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:38.367+0000 7f9837473700 1 -- 172.21.15.38:0/3511929667 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9830190fa0 con 0x7f9830103890 2023-03-09T14:37:39.662 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:38.367+0000 7f9834a0e700 1 --2- 172.21.15.38:0/3511929667 >> [v2:172.21.15.38:6800/3060474342,v1:172.21.15.38:6801/3060474342] conn(0x7f981c03c1d0 0x7f981c03e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:37:39.662 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:38.367+0000 7f9834a0e700 1 --2- 172.21.15.38:0/3511929667 >> [v2:172.21.15.38:6800/3060474342,v1:172.21.15.38:6801/3060474342] conn(0x7f981c03c1d0 0x7f981c03e680 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9824006fd0 tx=0x7f9824006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:37:39.662 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:38.372+0000 7f98227fc700 1 -- 172.21.15.38:0/3511929667 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f982c02d3f0 con 0x7f9830103890 2023-03-09T14:37:39.662 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:38.839+0000 7f9837473700 1 -- 172.21.15.38:0/3511929667 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f983002cef0 con 0x7f9830103890 2023-03-09T14:37:39.662 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:39.240+0000 7f98227fc700 1 -- 172.21.15.38:0/3511929667 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mgrmap(e 5) v1 ==== 44618+0+0 (secure 0 0 0) 0x7f982c00f050 con 0x7f9830103890 2023-03-09T14:37:39.662 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:39.242+0000 7f98227fc700 1 -- 172.21.15.38:0/3511929667 <== mon.0 v2:172.21.15.38:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f982c00bc90 con 0x7f9830103890 2023-03-09T14:37:39.662 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:39.244+0000 7f9837473700 1 -- 172.21.15.38:0/3511929667 >> [v2:172.21.15.38:6800/3060474342,v1:172.21.15.38:6801/3060474342] conn(0x7f981c03c1d0 msgr2=0x7f981c03e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:37:39.663 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:39.244+0000 7f9837473700 1 --2- 172.21.15.38:0/3511929667 >> [v2:172.21.15.38:6800/3060474342,v1:172.21.15.38:6801/3060474342] conn(0x7f981c03c1d0 0x7f981c03e680 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9824006fd0 tx=0x7f9824006e40 comp rx=0 tx=0).stop 2023-03-09T14:37:39.663 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:39.245+0000 7f9837473700 1 -- 172.21.15.38:0/3511929667 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9830103890 msgr2=0x7f98301970d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:37:39.663 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:39.245+0000 7f9837473700 1 --2- 172.21.15.38:0/3511929667 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9830103890 0x7f98301970d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f982c006b20 tx=0x7f982c011ad0 comp rx=0 tx=0).stop 2023-03-09T14:37:39.663 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:39.245+0000 7f9837473700 1 -- 172.21.15.38:0/3511929667 shutdown_connections 2023-03-09T14:37:39.663 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:39.245+0000 7f9837473700 1 --2- 172.21.15.38:0/3511929667 >> [v2:172.21.15.38:6800/3060474342,v1:172.21.15.38:6801/3060474342] conn(0x7f981c03c1d0 0x7f981c03e680 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:37:39.663 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:39.245+0000 7f9837473700 1 --2- 172.21.15.38:0/3511929667 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9830103890 0x7f98301970d0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:37:39.663 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:39.245+0000 7f9837473700 1 -- 172.21.15.38:0/3511929667 >> 172.21.15.38:0/3511929667 conn(0x7f98300ff260 msgr2=0x7f9830101660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:37:39.664 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:39.245+0000 7f9837473700 1 -- 172.21.15.38:0/3511929667 shutdown_connections 2023-03-09T14:37:39.664 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:39.245+0000 7f9837473700 1 -- 172.21.15.38:0/3511929667 wait complete. 2023-03-09T14:37:39.894 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:37:39.528+0000 7f58d6fd5200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-03-09T14:37:39.894 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:37:39.637+0000 7f58d6fd5200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-03-09T14:37:39.894 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:37:39.756+0000 7f58d6fd5200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-03-09T14:37:40.242 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:37:39.969+0000 7f58d6fd5200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-03-09T14:37:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:40 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/3511929667' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2023-03-09T14:37:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:40 smithi038 ceph-mon[92422]: mgrmap e5: x(active, since 7s) 2023-03-09T14:37:40.960 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:40 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:37:40.729+0000 7f58d6fd5200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-03-09T14:37:40.961 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:40 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:37:40.850+0000 7f58d6fd5200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-03-09T14:37:41.006 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout { 2023-03-09T14:37:41.006 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 5, 2023-03-09T14:37:41.006 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "available": true, 2023-03-09T14:37:41.006 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "active_name": "x", 2023-03-09T14:37:41.007 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_standby": 0 2023-03-09T14:37:41.007 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout } 2023-03-09T14:37:41.007 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:40.172+0000 7f180fad3700 1 Processor -- start 2023-03-09T14:37:41.007 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:40.173+0000 7f180fad3700 1 -- start start 2023-03-09T14:37:41.008 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:40.173+0000 7f180fad3700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f18081060d0 0x7f18081084b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:37:41.008 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:40.173+0000 7f180fad3700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1808079d70 con 0x7f18081060d0 2023-03-09T14:37:41.008 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:40.173+0000 7f180d86f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f18081060d0 0x7f18081084b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:37:41.008 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:40.173+0000 7f180d86f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f18081060d0 0x7f18081084b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35410/0 (socket says 172.21.15.38:35410) 2023-03-09T14:37:41.008 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:40.173+0000 7f180d86f700 1 -- 172.21.15.38:0/4214396753 learned_addr learned my addr 172.21.15.38:0/4214396753 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:37:41.008 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:40.173+0000 7f180d86f700 1 -- 172.21.15.38:0/4214396753 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1808079eb0 con 0x7f18081060d0 2023-03-09T14:37:41.008 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:40.173+0000 7f180d86f700 1 --2- 172.21.15.38:0/4214396753 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f18081060d0 0x7f18081084b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f18040098a0 tx=0x7f1804009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15fc3157e826a3a4 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:37:41.008 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:40.174+0000 7f180c86d700 1 -- 172.21.15.38:0/4214396753 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1804004160 con 0x7f18081060d0 2023-03-09T14:37:41.009 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:40.174+0000 7f180c86d700 1 -- 172.21.15.38:0/4214396753 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f18040042c0 con 0x7f18081060d0 2023-03-09T14:37:41.009 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:40.174+0000 7f180c86d700 1 -- 172.21.15.38:0/4214396753 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1804011a00 con 0x7f18081060d0 2023-03-09T14:37:41.009 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:40.174+0000 7f180fad3700 1 -- 172.21.15.38:0/4214396753 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f18081060d0 msgr2=0x7f18081084b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:37:41.009 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:40.174+0000 7f180fad3700 1 --2- 172.21.15.38:0/4214396753 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f18081060d0 0x7f18081084b0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f18040098a0 tx=0x7f1804009bb0 comp rx=0 tx=0).stop 2023-03-09T14:37:41.009 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:40.174+0000 7f180fad3700 1 -- 172.21.15.38:0/4214396753 shutdown_connections 2023-03-09T14:37:41.009 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:40.174+0000 7f180fad3700 1 --2- 172.21.15.38:0/4214396753 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f18081060d0 0x7f18081084b0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:37:41.009 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:40.174+0000 7f180fad3700 1 -- 172.21.15.38:0/4214396753 >> 172.21.15.38:0/4214396753 conn(0x7f18080ffb10 msgr2=0x7f1808101f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:37:41.010 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:40.174+0000 7f180fad3700 1 -- 172.21.15.38:0/4214396753 shutdown_connections 2023-03-09T14:37:41.010 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:40.174+0000 7f180fad3700 1 -- 172.21.15.38:0/4214396753 wait complete. 2023-03-09T14:37:41.010 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:40.175+0000 7f180fad3700 1 Processor -- start 2023-03-09T14:37:41.010 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:40.175+0000 7f180fad3700 1 -- start start 2023-03-09T14:37:41.010 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:40.175+0000 7f180fad3700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f18081060d0 0x7f1808199260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:37:41.010 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:40.175+0000 7f180fad3700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18081997a0 con 0x7f18081060d0 2023-03-09T14:37:41.010 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:40.175+0000 7f180d86f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f18081060d0 0x7f1808199260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:37:41.011 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:40.175+0000 7f180d86f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f18081060d0 0x7f1808199260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35412/0 (socket says 172.21.15.38:35412) 2023-03-09T14:37:41.011 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:40.175+0000 7f180d86f700 1 -- 172.21.15.38:0/255614143 learned_addr learned my addr 172.21.15.38:0/255614143 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:37:41.011 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:40.176+0000 7f180d86f700 1 -- 172.21.15.38:0/255614143 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1804009550 con 0x7f18081060d0 2023-03-09T14:37:41.011 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:40.176+0000 7f180d86f700 1 --2- 172.21.15.38:0/255614143 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f18081060d0 0x7f1808199260 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1804009fb0 tx=0x7f1804011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:37:41.011 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:40.176+0000 7f17feffd700 1 -- 172.21.15.38:0/255614143 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1804003930 con 0x7f18081060d0 2023-03-09T14:37:41.011 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:40.176+0000 7f17feffd700 1 -- 172.21.15.38:0/255614143 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1804003a90 con 0x7f18081060d0 2023-03-09T14:37:41.011 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:40.176+0000 7f17feffd700 1 -- 172.21.15.38:0/255614143 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f18040195e0 con 0x7f18081060d0 2023-03-09T14:37:41.011 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:40.176+0000 7f180fad3700 1 -- 172.21.15.38:0/255614143 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18081097f0 con 0x7f18081060d0 2023-03-09T14:37:41.012 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:40.176+0000 7f180fad3700 1 -- 172.21.15.38:0/255614143 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1808109660 con 0x7f18081060d0 2023-03-09T14:37:41.012 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:40.177+0000 7f17feffd700 1 -- 172.21.15.38:0/255614143 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 5) v1 ==== 44618+0+0 (secure 0 0 0) 0x7f180402a020 con 0x7f18081060d0 2023-03-09T14:37:41.012 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:40.177+0000 7f17feffd700 1 --2- 172.21.15.38:0/255614143 >> [v2:172.21.15.38:6800/3060474342,v1:172.21.15.38:6801/3060474342] conn(0x7f17f4037d30 0x7f17f403a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:37:41.012 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:40.177+0000 7f17feffd700 1 -- 172.21.15.38:0/255614143 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f180404b2f0 con 0x7f18081060d0 2023-03-09T14:37:41.012 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:40.177+0000 7f180fad3700 1 -- 172.21.15.38:0/255614143 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1808061db0 con 0x7f18081060d0 2023-03-09T14:37:41.012 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:40.177+0000 7f180d06e700 1 -- 172.21.15.38:0/255614143 >> [v2:172.21.15.38:6800/3060474342,v1:172.21.15.38:6801/3060474342] conn(0x7f17f4037d30 msgr2=0x7f17f403a1e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/3060474342 2023-03-09T14:37:41.012 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:40.177+0000 7f180d06e700 1 --2- 172.21.15.38:0/255614143 >> [v2:172.21.15.38:6800/3060474342,v1:172.21.15.38:6801/3060474342] conn(0x7f17f4037d30 0x7f17f403a1e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-03-09T14:37:41.013 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:40.182+0000 7f17feffd700 1 -- 172.21.15.38:0/255614143 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f1804023b80 con 0x7f18081060d0 2023-03-09T14:37:41.013 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:40.378+0000 7f180d06e700 1 -- 172.21.15.38:0/255614143 >> [v2:172.21.15.38:6800/3060474342,v1:172.21.15.38:6801/3060474342] conn(0x7f17f4037d30 msgr2=0x7f17f403a1e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/3060474342 2023-03-09T14:37:41.013 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:40.378+0000 7f180d06e700 1 --2- 172.21.15.38:0/255614143 >> [v2:172.21.15.38:6800/3060474342,v1:172.21.15.38:6801/3060474342] conn(0x7f17f4037d30 0x7f17f403a1e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-03-09T14:37:41.013 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:40.652+0000 7f180fad3700 1 -- 172.21.15.38:0/255614143 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f180819cc10 con 0x7f18081060d0 2023-03-09T14:37:41.013 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:40.652+0000 7f17feffd700 1 -- 172.21.15.38:0/255614143 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7f1804023b80 con 0x7f18081060d0 2023-03-09T14:37:41.013 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:40.654+0000 7f180fad3700 1 -- 172.21.15.38:0/255614143 >> [v2:172.21.15.38:6800/3060474342,v1:172.21.15.38:6801/3060474342] conn(0x7f17f4037d30 msgr2=0x7f17f403a1e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-03-09T14:37:41.013 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:40.654+0000 7f180fad3700 1 --2- 172.21.15.38:0/255614143 >> [v2:172.21.15.38:6800/3060474342,v1:172.21.15.38:6801/3060474342] conn(0x7f17f4037d30 0x7f17f403a1e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:37:41.013 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:40.654+0000 7f180fad3700 1 -- 172.21.15.38:0/255614143 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f18081060d0 msgr2=0x7f1808199260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:37:41.014 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:40.654+0000 7f180fad3700 1 --2- 172.21.15.38:0/255614143 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f18081060d0 0x7f1808199260 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1804009fb0 tx=0x7f1804011ad0 comp rx=0 tx=0).stop 2023-03-09T14:37:41.014 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:40.655+0000 7f180fad3700 1 -- 172.21.15.38:0/255614143 shutdown_connections 2023-03-09T14:37:41.014 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:40.655+0000 7f180fad3700 1 --2- 172.21.15.38:0/255614143 >> [v2:172.21.15.38:6800/3060474342,v1:172.21.15.38:6801/3060474342] conn(0x7f17f4037d30 0x7f17f403a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:37:41.014 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:40.655+0000 7f180fad3700 1 --2- 172.21.15.38:0/255614143 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f18081060d0 0x7f1808199260 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:37:41.014 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:40.655+0000 7f180fad3700 1 -- 172.21.15.38:0/255614143 >> 172.21.15.38:0/255614143 conn(0x7f18080ffb10 msgr2=0x7f18081008b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:37:41.014 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:40.655+0000 7f180fad3700 1 -- 172.21.15.38:0/255614143 shutdown_connections 2023-03-09T14:37:41.014 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:40.655+0000 7f180fad3700 1 -- 172.21.15.38:0/255614143 wait complete. 2023-03-09T14:37:41.015 INFO:teuthology.orchestra.run.smithi038.stdout:Waiting for the mgr to restart... 2023-03-09T14:37:41.015 INFO:teuthology.orchestra.run.smithi038.stdout:Waiting for mgr epoch 5... 2023-03-09T14:37:41.235 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:40 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:37:40.958+0000 7f58d6fd5200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-03-09T14:37:41.236 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:41 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:37:41.139+0000 7f58d6fd5200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-03-09T14:37:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:41 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/255614143' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-03-09T14:37:42.000 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:41 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:37:41.672+0000 7f58d6fd5200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-03-09T14:37:42.750 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:42 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:37:42.373+0000 7f58d6fd5200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-03-09T14:37:42.751 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:42 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:37:42.471+0000 7f58d6fd5200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-03-09T14:37:43.750 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:43 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:37:43.423+0000 7f58d6fd5200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-03-09T14:37:43.751 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:43 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:37:43.533+0000 7f58d6fd5200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-03-09T14:37:44.243 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:43 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:37:43.748+0000 7f58d6fd5200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-03-09T14:37:44.500 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:44 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:37:44.241+0000 7f58d6fd5200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-03-09T14:37:44.500 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:44 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:37:44.346+0000 7f58d6fd5200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-03-09T14:37:44.852 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:44 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:37:44.527+0000 7f58d6fd5200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-03-09T14:37:45.250 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:44 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:37:44.850+0000 7f58d6fd5200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-03-09T14:37:46.000 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:45 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:37:45.672+0000 7f58d6fd5200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-03-09T14:37:46.000 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:45 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:37:45.829+0000 7f58d6fd5200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-03-09T14:37:46.475 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:46 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:37:46.069+0000 7f58d6fd5200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-03-09T14:37:46.475 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:46 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:37:46.204+0000 7f58d6fd5200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-03-09T14:37:48.000 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:46 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:37:46.473+0000 7f58d6fd5200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-03-09T14:37:48.001 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:47 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:37:47.504+0000 7f58d6fd5200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-03-09T14:37:49.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:48 smithi038 ceph-mon[92422]: Active manager daemon x restarted 2023-03-09T14:37:49.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:48 smithi038 ceph-mon[92422]: Activating manager daemon x 2023-03-09T14:37:49.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:48 smithi038 ceph-mon[92422]: osdmap e2: 0 total, 0 up, 0 in 2023-03-09T14:37:49.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:48 smithi038 ceph-mon[92422]: mgrmap e6: x(active, starting, since 0.523841s) 2023-03-09T14:37:49.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:48 smithi038 ceph-mon[92422]: from='mgr.14118 172.21.15.38:0/2766508037' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-03-09T14:37:49.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:48 smithi038 ceph-mon[92422]: from='mgr.14118 172.21.15.38:0/2766508037' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-03-09T14:37:49.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:48 smithi038 ceph-mon[92422]: from='mgr.14118 172.21.15.38:0/2766508037' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-03-09T14:37:49.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:48 smithi038 ceph-mon[92422]: from='mgr.14118 172.21.15.38:0/2766508037' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-03-09T14:37:49.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:48 smithi038 ceph-mon[92422]: from='mgr.14118 172.21.15.38:0/2766508037' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-03-09T14:37:49.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:48 smithi038 ceph-mon[92422]: Manager daemon x is now available 2023-03-09T14:37:49.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:48 smithi038 ceph-mon[92422]: Found migration_current of "None". Setting to last migration. 2023-03-09T14:37:49.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:48 smithi038 ceph-mon[92422]: from='mgr.14118 172.21.15.38:0/2766508037' entity='mgr.x' 2023-03-09T14:37:49.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:48 smithi038 ceph-mon[92422]: from='mgr.14118 172.21.15.38:0/2766508037' entity='mgr.x' 2023-03-09T14:37:49.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:48 smithi038 ceph-mon[92422]: from='mgr.14118 172.21.15.38:0/2766508037' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:37:49.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:48 smithi038 ceph-mon[92422]: from='mgr.14118 172.21.15.38:0/2766508037' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:37:49.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:48 smithi038 ceph-mon[92422]: from='mgr.14118 172.21.15.38:0/2766508037' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:37:49.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:48 smithi038 ceph-mon[92422]: from='mgr.14118 172.21.15.38:0/2766508037' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-03-09T14:37:49.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:48 smithi038 ceph-mon[92422]: from='mgr.14118 172.21.15.38:0/2766508037' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-03-09T14:37:49.471 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout { 2023-03-09T14:37:49.471 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2023-03-09T14:37:49.471 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "initialized": true 2023-03-09T14:37:49.472 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout } 2023-03-09T14:37:49.472 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:41.492+0000 7f00ca9be700 1 Processor -- start 2023-03-09T14:37:49.472 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:41.492+0000 7f00ca9be700 1 -- start start 2023-03-09T14:37:49.472 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:41.493+0000 7f00ca9be700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00c4106090 0x7f00c4108470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:37:49.472 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:41.493+0000 7f00ca9be700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00c4079db0 con 0x7f00c4106090 2023-03-09T14:37:49.472 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:41.493+0000 7f00c3fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00c4106090 0x7f00c4108470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:37:49.472 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:41.493+0000 7f00c3fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00c4106090 0x7f00c4108470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35420/0 (socket says 172.21.15.38:35420) 2023-03-09T14:37:49.472 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:41.493+0000 7f00c3fff700 1 -- 172.21.15.38:0/1503349216 learned_addr learned my addr 172.21.15.38:0/1503349216 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:37:49.473 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:41.493+0000 7f00c3fff700 1 -- 172.21.15.38:0/1503349216 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00c4079ef0 con 0x7f00c4106090 2023-03-09T14:37:49.473 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:41.493+0000 7f00c3fff700 1 --2- 172.21.15.38:0/1503349216 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00c4106090 0x7f00c4108470 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f00b40098a0 tx=0x7f00b4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=567ac9795498c8ed server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:37:49.473 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:41.494+0000 7f00c2ffd700 1 -- 172.21.15.38:0/1503349216 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f00b4004160 con 0x7f00c4106090 2023-03-09T14:37:49.473 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:41.494+0000 7f00c2ffd700 1 -- 172.21.15.38:0/1503349216 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f00b40042c0 con 0x7f00c4106090 2023-03-09T14:37:49.473 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:41.494+0000 7f00c2ffd700 1 -- 172.21.15.38:0/1503349216 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f00b4011a00 con 0x7f00c4106090 2023-03-09T14:37:49.473 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:41.494+0000 7f00ca9be700 1 -- 172.21.15.38:0/1503349216 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00c4106090 msgr2=0x7f00c4108470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:37:49.473 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:41.494+0000 7f00ca9be700 1 --2- 172.21.15.38:0/1503349216 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00c4106090 0x7f00c4108470 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f00b40098a0 tx=0x7f00b4009bb0 comp rx=0 tx=0).stop 2023-03-09T14:37:49.474 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:41.494+0000 7f00ca9be700 1 -- 172.21.15.38:0/1503349216 shutdown_connections 2023-03-09T14:37:49.474 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:41.494+0000 7f00ca9be700 1 --2- 172.21.15.38:0/1503349216 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00c4106090 0x7f00c4108470 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:37:49.474 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:41.494+0000 7f00ca9be700 1 -- 172.21.15.38:0/1503349216 >> 172.21.15.38:0/1503349216 conn(0x7f00c40ffb10 msgr2=0x7f00c4101f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:37:49.474 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:41.494+0000 7f00ca9be700 1 -- 172.21.15.38:0/1503349216 shutdown_connections 2023-03-09T14:37:49.474 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:41.494+0000 7f00ca9be700 1 -- 172.21.15.38:0/1503349216 wait complete. 2023-03-09T14:37:49.474 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:41.495+0000 7f00ca9be700 1 Processor -- start 2023-03-09T14:37:49.474 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:41.495+0000 7f00ca9be700 1 -- start start 2023-03-09T14:37:49.475 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:41.495+0000 7f00ca9be700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00c4106090 0x7f00c4199260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:37:49.475 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:41.495+0000 7f00ca9be700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00c41997a0 con 0x7f00c4106090 2023-03-09T14:37:49.475 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:41.496+0000 7f00c3fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00c4106090 0x7f00c4199260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:37:49.475 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:41.496+0000 7f00c3fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00c4106090 0x7f00c4199260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35422/0 (socket says 172.21.15.38:35422) 2023-03-09T14:37:49.475 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:41.496+0000 7f00c3fff700 1 -- 172.21.15.38:0/3480643460 learned_addr learned my addr 172.21.15.38:0/3480643460 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:37:49.475 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:41.496+0000 7f00c3fff700 1 -- 172.21.15.38:0/3480643460 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00b4009550 con 0x7f00c4106090 2023-03-09T14:37:49.475 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:41.496+0000 7f00c3fff700 1 --2- 172.21.15.38:0/3480643460 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00c4106090 0x7f00c4199260 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f00b4006af0 tx=0x7f00b4011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:37:49.475 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:41.496+0000 7f00c17fa700 1 -- 172.21.15.38:0/3480643460 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f00b4003a40 con 0x7f00c4106090 2023-03-09T14:37:49.476 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:41.496+0000 7f00c17fa700 1 -- 172.21.15.38:0/3480643460 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f00b4003ba0 con 0x7f00c4106090 2023-03-09T14:37:49.476 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:41.496+0000 7f00ca9be700 1 -- 172.21.15.38:0/3480643460 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00c4109810 con 0x7f00c4106090 2023-03-09T14:37:49.476 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:41.496+0000 7f00c17fa700 1 -- 172.21.15.38:0/3480643460 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f00b401a700 con 0x7f00c4106090 2023-03-09T14:37:49.476 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:41.496+0000 7f00ca9be700 1 -- 172.21.15.38:0/3480643460 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00c4109680 con 0x7f00c4106090 2023-03-09T14:37:49.476 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:41.497+0000 7f00c17fa700 1 -- 172.21.15.38:0/3480643460 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 5) v1 ==== 44618+0+0 (secure 0 0 0) 0x7f00b4017020 con 0x7f00c4106090 2023-03-09T14:37:49.476 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:41.498+0000 7f00c17fa700 1 --2- 172.21.15.38:0/3480643460 >> [v2:172.21.15.38:6800/3060474342,v1:172.21.15.38:6801/3060474342] conn(0x7f00ac037d80 0x7f00ac03a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:37:49.476 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:41.498+0000 7f00c17fa700 1 -- 172.21.15.38:0/3480643460 --> [v2:172.21.15.38:6800/3060474342,v1:172.21.15.38:6801/3060474342] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f00ac03a940 con 0x7f00ac037d80 2023-03-09T14:37:49.477 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:41.498+0000 7f00c17fa700 1 -- 172.21.15.38:0/3480643460 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f00b404b580 con 0x7f00c4106090 2023-03-09T14:37:49.477 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:41.498+0000 7f00c37fe700 1 -- 172.21.15.38:0/3480643460 >> [v2:172.21.15.38:6800/3060474342,v1:172.21.15.38:6801/3060474342] conn(0x7f00ac037d80 msgr2=0x7f00ac03a230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/3060474342 2023-03-09T14:37:49.477 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:41.498+0000 7f00c37fe700 1 --2- 172.21.15.38:0/3480643460 >> [v2:172.21.15.38:6800/3060474342,v1:172.21.15.38:6801/3060474342] conn(0x7f00ac037d80 0x7f00ac03a230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-03-09T14:37:49.477 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:41.699+0000 7f00c37fe700 1 -- 172.21.15.38:0/3480643460 >> [v2:172.21.15.38:6800/3060474342,v1:172.21.15.38:6801/3060474342] conn(0x7f00ac037d80 msgr2=0x7f00ac03a230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/3060474342 2023-03-09T14:37:49.477 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:41.699+0000 7f00c37fe700 1 --2- 172.21.15.38:0/3480643460 >> [v2:172.21.15.38:6800/3060474342,v1:172.21.15.38:6801/3060474342] conn(0x7f00ac037d80 0x7f00ac03a230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-03-09T14:37:49.477 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:42.100+0000 7f00c37fe700 1 -- 172.21.15.38:0/3480643460 >> [v2:172.21.15.38:6800/3060474342,v1:172.21.15.38:6801/3060474342] conn(0x7f00ac037d80 msgr2=0x7f00ac03a230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/3060474342 2023-03-09T14:37:49.477 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:42.100+0000 7f00c37fe700 1 --2- 172.21.15.38:0/3480643460 >> [v2:172.21.15.38:6800/3060474342,v1:172.21.15.38:6801/3060474342] conn(0x7f00ac037d80 0x7f00ac03a230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-03-09T14:37:49.478 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:42.901+0000 7f00c37fe700 1 -- 172.21.15.38:0/3480643460 >> [v2:172.21.15.38:6800/3060474342,v1:172.21.15.38:6801/3060474342] conn(0x7f00ac037d80 msgr2=0x7f00ac03a230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/3060474342 2023-03-09T14:37:49.478 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:42.901+0000 7f00c37fe700 1 --2- 172.21.15.38:0/3480643460 >> [v2:172.21.15.38:6800/3060474342,v1:172.21.15.38:6801/3060474342] conn(0x7f00ac037d80 0x7f00ac03a230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-03-09T14:37:49.478 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:44.502+0000 7f00c37fe700 1 -- 172.21.15.38:0/3480643460 >> [v2:172.21.15.38:6800/3060474342,v1:172.21.15.38:6801/3060474342] conn(0x7f00ac037d80 msgr2=0x7f00ac03a230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/3060474342 2023-03-09T14:37:49.478 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:44.502+0000 7f00c37fe700 1 --2- 172.21.15.38:0/3480643460 >> [v2:172.21.15.38:6800/3060474342,v1:172.21.15.38:6801/3060474342] conn(0x7f00ac037d80 0x7f00ac03a230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-03-09T14:37:49.478 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:47.705+0000 7f00c37fe700 1 -- 172.21.15.38:0/3480643460 >> [v2:172.21.15.38:6800/3060474342,v1:172.21.15.38:6801/3060474342] conn(0x7f00ac037d80 msgr2=0x7f00ac03a230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/3060474342 2023-03-09T14:37:49.478 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:47.705+0000 7f00c37fe700 1 --2- 172.21.15.38:0/3480643460 >> [v2:172.21.15.38:6800/3060474342,v1:172.21.15.38:6801/3060474342] conn(0x7f00ac037d80 0x7f00ac03a230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2023-03-09T14:37:49.478 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:48.040+0000 7f00c17fa700 1 -- 172.21.15.38:0/3480643460 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mgrmap(e 6) v1 ==== 44428+0+0 (secure 0 0 0) 0x7f00b400f050 con 0x7f00c4106090 2023-03-09T14:37:49.478 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:48.040+0000 7f00c17fa700 1 -- 172.21.15.38:0/3480643460 >> [v2:172.21.15.38:6800/3060474342,v1:172.21.15.38:6801/3060474342] conn(0x7f00ac037d80 msgr2=0x7f00ac03a230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-03-09T14:37:49.479 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:48.040+0000 7f00c17fa700 1 --2- 172.21.15.38:0/3480643460 >> [v2:172.21.15.38:6800/3060474342,v1:172.21.15.38:6801/3060474342] conn(0x7f00ac037d80 0x7f00ac03a230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:37:49.479 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:49.041+0000 7f00c17fa700 1 -- 172.21.15.38:0/3480643460 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mgrmap(e 7) v1 ==== 44538+0+0 (secure 0 0 0) 0x7f00b404c9f0 con 0x7f00c4106090 2023-03-09T14:37:49.479 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:49.041+0000 7f00c17fa700 1 --2- 172.21.15.38:0/3480643460 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7f00ac03b6c0 0x7f00ac03daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:37:49.479 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:49.041+0000 7f00c17fa700 1 -- 172.21.15.38:0/3480643460 --> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f00ac03a940 con 0x7f00ac03b6c0 2023-03-09T14:37:49.479 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:49.041+0000 7f00c37fe700 1 --2- 172.21.15.38:0/3480643460 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7f00ac03b6c0 0x7f00ac03daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:37:49.479 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:49.042+0000 7f00c37fe700 1 --2- 172.21.15.38:0/3480643460 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7f00ac03b6c0 0x7f00ac03daa0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f00b8003a10 tx=0x7f00b8009290 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:37:49.479 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:49.043+0000 7f00c17fa700 1 -- 172.21.15.38:0/3480643460 <== mgr.14118 v2:172.21.15.38:6800/3381625669 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7f00ac03a940 con 0x7f00ac03b6c0 2023-03-09T14:37:49.480 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:49.054+0000 7f00ca9be700 1 -- 172.21.15.38:0/3480643460 --> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f00c419a590 con 0x7f00ac03b6c0 2023-03-09T14:37:49.480 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:49.054+0000 7f00c17fa700 1 -- 172.21.15.38:0/3480643460 <== mgr.14118 v2:172.21.15.38:6800/3381625669 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f00c419a590 con 0x7f00ac03b6c0 2023-03-09T14:37:49.480 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:49.055+0000 7f00ca9be700 1 -- 172.21.15.38:0/3480643460 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7f00ac03b6c0 msgr2=0x7f00ac03daa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:37:49.480 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:49.055+0000 7f00ca9be700 1 --2- 172.21.15.38:0/3480643460 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7f00ac03b6c0 0x7f00ac03daa0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f00b8003a10 tx=0x7f00b8009290 comp rx=0 tx=0).stop 2023-03-09T14:37:49.480 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:49.055+0000 7f00ca9be700 1 -- 172.21.15.38:0/3480643460 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00c4106090 msgr2=0x7f00c4199260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:37:49.480 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:49.055+0000 7f00ca9be700 1 --2- 172.21.15.38:0/3480643460 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00c4106090 0x7f00c4199260 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f00b4006af0 tx=0x7f00b4011ad0 comp rx=0 tx=0).stop 2023-03-09T14:37:49.480 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:49.055+0000 7f00ca9be700 1 -- 172.21.15.38:0/3480643460 shutdown_connections 2023-03-09T14:37:49.480 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:49.055+0000 7f00ca9be700 1 --2- 172.21.15.38:0/3480643460 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7f00ac03b6c0 0x7f00ac03daa0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:37:49.481 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:49.055+0000 7f00ca9be700 1 --2- 172.21.15.38:0/3480643460 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00c4106090 0x7f00c4199260 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:37:49.481 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:49.055+0000 7f00ca9be700 1 --2- 172.21.15.38:0/3480643460 >> [v2:172.21.15.38:6800/3060474342,v1:172.21.15.38:6801/3060474342] conn(0x7f00ac037d80 0x7f00ac03a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:37:49.481 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:49.055+0000 7f00ca9be700 1 -- 172.21.15.38:0/3480643460 >> 172.21.15.38:0/3480643460 conn(0x7f00c40ffb10 msgr2=0x7f00c41008b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:37:49.481 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:49.055+0000 7f00ca9be700 1 -- 172.21.15.38:0/3480643460 shutdown_connections 2023-03-09T14:37:49.481 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:49.055+0000 7f00ca9be700 1 -- 172.21.15.38:0/3480643460 wait complete. 2023-03-09T14:37:49.481 INFO:teuthology.orchestra.run.smithi038.stdout:mgr epoch 5 is available 2023-03-09T14:37:49.481 INFO:teuthology.orchestra.run.smithi038.stdout:Setting orchestrator backend to cephadm... 2023-03-09T14:37:49.703 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:49 smithi038 ceph-mon[92422]: from='mgr.14118 172.21.15.38:0/2766508037' entity='mgr.x' 2023-03-09T14:37:49.703 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:49 smithi038 ceph-mon[92422]: from='mgr.14118 172.21.15.38:0/2766508037' entity='mgr.x' 2023-03-09T14:37:49.703 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:49 smithi038 ceph-mon[92422]: [09/Mar/2023:14:37:49] ENGINE Bus STARTING 2023-03-09T14:37:49.704 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:49 smithi038 ceph-mon[92422]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-03-09T14:37:49.704 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:49 smithi038 ceph-mon[92422]: mgrmap e7: x(active, since 1.52618s) 2023-03-09T14:37:49.704 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:49 smithi038 ceph-mon[92422]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-03-09T14:37:49.704 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:49 smithi038 ceph-mon[92422]: [09/Mar/2023:14:37:49] ENGINE Serving on https://172.21.15.38:7150 2023-03-09T14:37:49.704 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:49 smithi038 ceph-mon[92422]: [09/Mar/2023:14:37:49] ENGINE Serving on http://172.21.15.38:8765 2023-03-09T14:37:49.704 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:49 smithi038 ceph-mon[92422]: [09/Mar/2023:14:37:49] ENGINE Bus STARTED 2023-03-09T14:37:49.704 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:49 smithi038 ceph-mon[92422]: from='mgr.14118 172.21.15.38:0/2766508037' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:37:50.716 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:49.912+0000 7f72ba37d700 1 Processor -- start 2023-03-09T14:37:50.717 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:49.912+0000 7f72ba37d700 1 -- start start 2023-03-09T14:37:50.717 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:49.912+0000 7f72ba37d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72b41060d0 0x7f72b41084b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:37:50.717 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:49.912+0000 7f72ba37d700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72b4079d70 con 0x7f72b41060d0 2023-03-09T14:37:50.717 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:49.912+0000 7f72b3fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72b41060d0 0x7f72b41084b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:37:50.718 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:49.912+0000 7f72b3fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72b41060d0 0x7f72b41084b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35478/0 (socket says 172.21.15.38:35478) 2023-03-09T14:37:50.718 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:49.912+0000 7f72b3fff700 1 -- 172.21.15.38:0/3165851295 learned_addr learned my addr 172.21.15.38:0/3165851295 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:37:50.718 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:49.912+0000 7f72b3fff700 1 -- 172.21.15.38:0/3165851295 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72b4079eb0 con 0x7f72b41060d0 2023-03-09T14:37:50.718 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:49.913+0000 7f72b3fff700 1 --2- 172.21.15.38:0/3165851295 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72b41060d0 0x7f72b41084b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f72a40098a0 tx=0x7f72a4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce5556e75026f98e server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:37:50.718 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:49.913+0000 7f72b2ffd700 1 -- 172.21.15.38:0/3165851295 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f72a4004160 con 0x7f72b41060d0 2023-03-09T14:37:50.718 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:49.913+0000 7f72b2ffd700 1 -- 172.21.15.38:0/3165851295 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f72a40042c0 con 0x7f72b41060d0 2023-03-09T14:37:50.718 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:49.913+0000 7f72b2ffd700 1 -- 172.21.15.38:0/3165851295 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f72a4011a00 con 0x7f72b41060d0 2023-03-09T14:37:50.718 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:49.913+0000 7f72ba37d700 1 -- 172.21.15.38:0/3165851295 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72b41060d0 msgr2=0x7f72b41084b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:37:50.719 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:49.913+0000 7f72ba37d700 1 --2- 172.21.15.38:0/3165851295 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72b41060d0 0x7f72b41084b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f72a40098a0 tx=0x7f72a4009bb0 comp rx=0 tx=0).stop 2023-03-09T14:37:50.719 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:49.913+0000 7f72ba37d700 1 -- 172.21.15.38:0/3165851295 shutdown_connections 2023-03-09T14:37:50.719 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:49.913+0000 7f72ba37d700 1 --2- 172.21.15.38:0/3165851295 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72b41060d0 0x7f72b41084b0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:37:50.719 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:49.913+0000 7f72ba37d700 1 -- 172.21.15.38:0/3165851295 >> 172.21.15.38:0/3165851295 conn(0x7f72b40ffb10 msgr2=0x7f72b4101f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:37:50.719 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:49.913+0000 7f72ba37d700 1 -- 172.21.15.38:0/3165851295 shutdown_connections 2023-03-09T14:37:50.719 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:49.914+0000 7f72ba37d700 1 -- 172.21.15.38:0/3165851295 wait complete. 2023-03-09T14:37:50.719 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:49.914+0000 7f72ba37d700 1 Processor -- start 2023-03-09T14:37:50.720 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:49.914+0000 7f72ba37d700 1 -- start start 2023-03-09T14:37:50.720 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:49.915+0000 7f72ba37d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72b41060d0 0x7f72b4199220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:37:50.720 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:49.915+0000 7f72ba37d700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72b4199760 con 0x7f72b41060d0 2023-03-09T14:37:50.720 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:49.915+0000 7f72b3fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72b41060d0 0x7f72b4199220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:37:50.720 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:49.915+0000 7f72b3fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72b41060d0 0x7f72b4199220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35480/0 (socket says 172.21.15.38:35480) 2023-03-09T14:37:50.720 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:49.915+0000 7f72b3fff700 1 -- 172.21.15.38:0/957688282 learned_addr learned my addr 172.21.15.38:0/957688282 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:37:50.720 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:49.915+0000 7f72b3fff700 1 -- 172.21.15.38:0/957688282 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72a4009550 con 0x7f72b41060d0 2023-03-09T14:37:50.720 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:49.915+0000 7f72b3fff700 1 --2- 172.21.15.38:0/957688282 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72b41060d0 0x7f72b4199220 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f72a4009fb0 tx=0x7f72a4011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:37:50.721 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:49.915+0000 7f72b17fa700 1 -- 172.21.15.38:0/957688282 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f72a4003930 con 0x7f72b41060d0 2023-03-09T14:37:50.721 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:49.916+0000 7f72ba37d700 1 -- 172.21.15.38:0/957688282 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72b41097f0 con 0x7f72b41060d0 2023-03-09T14:37:50.721 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:49.916+0000 7f72b17fa700 1 -- 172.21.15.38:0/957688282 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f72a4003a90 con 0x7f72b41060d0 2023-03-09T14:37:50.721 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:49.916+0000 7f72b17fa700 1 -- 172.21.15.38:0/957688282 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f72a40195e0 con 0x7f72b41060d0 2023-03-09T14:37:50.721 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:49.916+0000 7f72ba37d700 1 -- 172.21.15.38:0/957688282 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72b4109660 con 0x7f72b41060d0 2023-03-09T14:37:50.721 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:49.916+0000 7f72b17fa700 1 -- 172.21.15.38:0/957688282 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 7) v1 ==== 44538+0+0 (secure 0 0 0) 0x7f72a402a020 con 0x7f72b41060d0 2023-03-09T14:37:50.721 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:49.916+0000 7f72b17fa700 1 --2- 172.21.15.38:0/957688282 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7f72a0037c90 0x7f72a003a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:37:50.722 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:49.917+0000 7f72b17fa700 1 -- 172.21.15.38:0/957688282 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f72a404b680 con 0x7f72b41060d0 2023-03-09T14:37:50.722 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:49.917+0000 7f72b37fe700 1 --2- 172.21.15.38:0/957688282 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7f72a0037c90 0x7f72a003a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:37:50.722 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:49.917+0000 7f72ba37d700 1 -- 172.21.15.38:0/957688282 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72b4109980 con 0x7f72b41060d0 2023-03-09T14:37:50.722 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:49.917+0000 7f72b37fe700 1 --2- 172.21.15.38:0/957688282 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7f72a0037c90 0x7f72a003a140 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f729c006fd0 tx=0x7f729c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:37:50.722 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:49.922+0000 7f72b17fa700 1 -- 172.21.15.38:0/957688282 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f72a402d440 con 0x7f72b41060d0 2023-03-09T14:37:50.722 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:50.047+0000 7f72b17fa700 1 -- 172.21.15.38:0/957688282 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mgrmap(e 8) v1 ==== 44618+0+0 (secure 0 0 0) 0x7f72a4017020 con 0x7f72b41060d0 2023-03-09T14:37:50.722 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:50.310+0000 7f72ba37d700 1 -- 172.21.15.38:0/957688282 --> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f72b4100f30 con 0x7f72a0037c90 2023-03-09T14:37:50.722 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:50.317+0000 7f72b17fa700 1 -- 172.21.15.38:0/957688282 <== mgr.14118 v2:172.21.15.38:6800/3381625669 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f72b4100f30 con 0x7f72a0037c90 2023-03-09T14:37:50.723 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:50.319+0000 7f72ba37d700 1 -- 172.21.15.38:0/957688282 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7f72a0037c90 msgr2=0x7f72a003a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:37:50.723 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:50.319+0000 7f72ba37d700 1 --2- 172.21.15.38:0/957688282 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7f72a0037c90 0x7f72a003a140 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f729c006fd0 tx=0x7f729c006e40 comp rx=0 tx=0).stop 2023-03-09T14:37:50.723 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:50.319+0000 7f72ba37d700 1 -- 172.21.15.38:0/957688282 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72b41060d0 msgr2=0x7f72b4199220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:37:50.723 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:50.319+0000 7f72ba37d700 1 --2- 172.21.15.38:0/957688282 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72b41060d0 0x7f72b4199220 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f72a4009fb0 tx=0x7f72a4011ad0 comp rx=0 tx=0).stop 2023-03-09T14:37:50.723 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:50.320+0000 7f72ba37d700 1 -- 172.21.15.38:0/957688282 shutdown_connections 2023-03-09T14:37:50.723 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:50.320+0000 7f72ba37d700 1 --2- 172.21.15.38:0/957688282 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7f72a0037c90 0x7f72a003a140 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:37:50.723 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:50.320+0000 7f72ba37d700 1 --2- 172.21.15.38:0/957688282 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72b41060d0 0x7f72b4199220 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:37:50.724 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:50.320+0000 7f72ba37d700 1 -- 172.21.15.38:0/957688282 >> 172.21.15.38:0/957688282 conn(0x7f72b40ffb10 msgr2=0x7f72b4100820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:37:50.724 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:50.320+0000 7f72ba37d700 1 -- 172.21.15.38:0/957688282 shutdown_connections 2023-03-09T14:37:50.724 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:50.320+0000 7f72ba37d700 1 -- 172.21.15.38:0/957688282 wait complete. 2023-03-09T14:37:51.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:51 smithi038 ceph-mon[92422]: mgrmap e8: x(active, since 2s) 2023-03-09T14:37:51.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:51 smithi038 ceph-mon[92422]: from='mgr.14118 172.21.15.38:0/2766508037' entity='mgr.x' 2023-03-09T14:37:51.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:51 smithi038 ceph-mon[92422]: from='mgr.14118 172.21.15.38:0/2766508037' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:37:51.960 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout value unchanged 2023-03-09T14:37:51.960 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:51.189+0000 7f37ed29f700 1 Processor -- start 2023-03-09T14:37:51.961 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:51.189+0000 7f37ed29f700 1 -- start start 2023-03-09T14:37:51.961 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:51.189+0000 7f37ed29f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f37e8104490 0x7f37e81048a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:37:51.961 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:51.189+0000 7f37ed29f700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37e8104de0 con 0x7f37e8104490 2023-03-09T14:37:51.961 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:51.189+0000 7f37e6ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f37e8104490 0x7f37e81048a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:37:51.961 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:51.189+0000 7f37e6ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f37e8104490 0x7f37e81048a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35484/0 (socket says 172.21.15.38:35484) 2023-03-09T14:37:51.961 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:51.189+0000 7f37e6ffd700 1 -- 172.21.15.38:0/3773192455 learned_addr learned my addr 172.21.15.38:0/3773192455 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:37:51.962 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:51.190+0000 7f37e6ffd700 1 -- 172.21.15.38:0/3773192455 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37e8104f20 con 0x7f37e8104490 2023-03-09T14:37:51.962 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:51.190+0000 7f37e6ffd700 1 --2- 172.21.15.38:0/3773192455 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f37e8104490 0x7f37e81048a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f37dc00bb40 tx=0x7f37dc00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dac3e70b7be3fc1 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:37:51.962 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:51.190+0000 7f37e5ffb700 1 -- 172.21.15.38:0/3773192455 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f37dc00dbe0 con 0x7f37e8104490 2023-03-09T14:37:51.962 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:51.190+0000 7f37e5ffb700 1 -- 172.21.15.38:0/3773192455 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f37dc0044d0 con 0x7f37e8104490 2023-03-09T14:37:51.962 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:51.190+0000 7f37e5ffb700 1 -- 172.21.15.38:0/3773192455 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f37dc010470 con 0x7f37e8104490 2023-03-09T14:37:51.962 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:51.191+0000 7f37ed29f700 1 -- 172.21.15.38:0/3773192455 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f37e8104490 msgr2=0x7f37e81048a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:37:51.962 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:51.191+0000 7f37ed29f700 1 --2- 172.21.15.38:0/3773192455 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f37e8104490 0x7f37e81048a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f37dc00bb40 tx=0x7f37dc00be50 comp rx=0 tx=0).stop 2023-03-09T14:37:51.962 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:51.191+0000 7f37ed29f700 1 -- 172.21.15.38:0/3773192455 shutdown_connections 2023-03-09T14:37:51.963 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:51.191+0000 7f37ed29f700 1 --2- 172.21.15.38:0/3773192455 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f37e8104490 0x7f37e81048a0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:37:51.963 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:51.191+0000 7f37ed29f700 1 -- 172.21.15.38:0/3773192455 >> 172.21.15.38:0/3773192455 conn(0x7f37e80ffb10 msgr2=0x7f37e8101f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:37:51.963 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:51.191+0000 7f37ed29f700 1 -- 172.21.15.38:0/3773192455 shutdown_connections 2023-03-09T14:37:51.963 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:51.191+0000 7f37ed29f700 1 -- 172.21.15.38:0/3773192455 wait complete. 2023-03-09T14:37:51.963 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:51.191+0000 7f37ed29f700 1 Processor -- start 2023-03-09T14:37:51.963 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:51.192+0000 7f37ed29f700 1 -- start start 2023-03-09T14:37:51.963 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:51.192+0000 7f37ed29f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f37e8104490 0x7f37e819b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:37:51.964 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:51.192+0000 7f37ed29f700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37e819b990 con 0x7f37e8104490 2023-03-09T14:37:51.964 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:51.192+0000 7f37e6ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f37e8104490 0x7f37e819b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:37:51.964 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:51.192+0000 7f37e6ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f37e8104490 0x7f37e819b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35486/0 (socket says 172.21.15.38:35486) 2023-03-09T14:37:51.964 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:51.192+0000 7f37e6ffd700 1 -- 172.21.15.38:0/3381123365 learned_addr learned my addr 172.21.15.38:0/3381123365 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:37:51.964 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:51.192+0000 7f37e6ffd700 1 -- 172.21.15.38:0/3381123365 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37dc00b7f0 con 0x7f37e8104490 2023-03-09T14:37:51.964 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:51.192+0000 7f37e6ffd700 1 --2- 172.21.15.38:0/3381123365 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f37e8104490 0x7f37e819b450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f37dc00b340 tx=0x7f37dc003df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:37:51.964 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:51.192+0000 7f37cffff700 1 -- 172.21.15.38:0/3381123365 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f37dc004340 con 0x7f37e8104490 2023-03-09T14:37:51.964 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:51.192+0000 7f37cffff700 1 -- 172.21.15.38:0/3381123365 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f37dc010bd0 con 0x7f37e8104490 2023-03-09T14:37:51.965 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:51.192+0000 7f37cffff700 1 -- 172.21.15.38:0/3381123365 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f37dc021c60 con 0x7f37e8104490 2023-03-09T14:37:51.965 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:51.193+0000 7f37ed29f700 1 -- 172.21.15.38:0/3381123365 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37e8106520 con 0x7f37e8104490 2023-03-09T14:37:51.965 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:51.193+0000 7f37ed29f700 1 -- 172.21.15.38:0/3381123365 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37e8106390 con 0x7f37e8104490 2023-03-09T14:37:51.965 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:51.193+0000 7f37cffff700 1 -- 172.21.15.38:0/3381123365 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) v1 ==== 44618+0+0 (secure 0 0 0) 0x7f37dc021410 con 0x7f37e8104490 2023-03-09T14:37:51.965 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:51.194+0000 7f37cffff700 1 --2- 172.21.15.38:0/3381123365 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7f37d0037d80 0x7f37d003a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:37:51.965 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:51.194+0000 7f37cffff700 1 -- 172.21.15.38:0/3381123365 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f37dc04bda0 con 0x7f37e8104490 2023-03-09T14:37:51.965 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:51.194+0000 7f37e67fc700 1 --2- 172.21.15.38:0/3381123365 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7f37d0037d80 0x7f37d003a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:37:51.966 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:51.194+0000 7f37ed29f700 1 -- 172.21.15.38:0/3381123365 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37e8078ff0 con 0x7f37e8104490 2023-03-09T14:37:51.966 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:51.194+0000 7f37e67fc700 1 --2- 172.21.15.38:0/3381123365 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7f37d0037d80 0x7f37d003a230 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f37d8006fd0 tx=0x7f37d8006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:37:51.966 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:51.199+0000 7f37cffff700 1 -- 172.21.15.38:0/3381123365 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f37dc00f5b0 con 0x7f37e8104490 2023-03-09T14:37:51.966 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:51.559+0000 7f37ed29f700 1 -- 172.21.15.38:0/3381123365 --> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f37e819c0d0 con 0x7f37d0037d80 2023-03-09T14:37:51.966 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:51.560+0000 7f37cffff700 1 -- 172.21.15.38:0/3381123365 <== mgr.14118 v2:172.21.15.38:6800/3381625669 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f37e819c0d0 con 0x7f37d0037d80 2023-03-09T14:37:51.966 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:51.562+0000 7f37ed29f700 1 -- 172.21.15.38:0/3381123365 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7f37d0037d80 msgr2=0x7f37d003a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:37:51.966 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:51.562+0000 7f37ed29f700 1 --2- 172.21.15.38:0/3381123365 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7f37d0037d80 0x7f37d003a230 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f37d8006fd0 tx=0x7f37d8006e40 comp rx=0 tx=0).stop 2023-03-09T14:37:51.967 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:51.562+0000 7f37ed29f700 1 -- 172.21.15.38:0/3381123365 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f37e8104490 msgr2=0x7f37e819b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:37:51.967 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:51.562+0000 7f37ed29f700 1 --2- 172.21.15.38:0/3381123365 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f37e8104490 0x7f37e819b450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f37dc00b340 tx=0x7f37dc003df0 comp rx=0 tx=0).stop 2023-03-09T14:37:51.967 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:51.562+0000 7f37ed29f700 1 -- 172.21.15.38:0/3381123365 shutdown_connections 2023-03-09T14:37:51.967 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:51.562+0000 7f37ed29f700 1 --2- 172.21.15.38:0/3381123365 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7f37d0037d80 0x7f37d003a230 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:37:51.967 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:51.562+0000 7f37ed29f700 1 --2- 172.21.15.38:0/3381123365 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f37e8104490 0x7f37e819b450 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:37:51.967 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:51.562+0000 7f37ed29f700 1 -- 172.21.15.38:0/3381123365 >> 172.21.15.38:0/3381123365 conn(0x7f37e80ffb10 msgr2=0x7f37e8192300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:37:51.967 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:51.562+0000 7f37ed29f700 1 -- 172.21.15.38:0/3381123365 shutdown_connections 2023-03-09T14:37:51.967 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:51.562+0000 7f37ed29f700 1 -- 172.21.15.38:0/3381123365 wait complete. 2023-03-09T14:37:51.968 INFO:teuthology.orchestra.run.smithi038.stdout:Generating ssh key... 2023-03-09T14:37:52.189 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:52 smithi038 ceph-mon[92422]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:37:53.351 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:53 smithi038 ceph-mon[92422]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:37:53.352 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:53 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: Generating public/private rsa key pair. 2023-03-09T14:37:53.352 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:53 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: Your identification has been saved in /tmp/tmpojxp2pdc/key. 2023-03-09T14:37:53.352 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:53 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: Your public key has been saved in /tmp/tmpojxp2pdc/key.pub. 2023-03-09T14:37:53.352 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:53 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: The key fingerprint is: 2023-03-09T14:37:53.352 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:53 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: SHA256:/wkrxsss4Pzk6bTXU7cn6NQy4kJV5lw1fwBrXuUfjIY ceph-c6733fe2-be87-11ed-9af2-001a4aab830c 2023-03-09T14:37:53.352 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:53 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: The key's randomart image is: 2023-03-09T14:37:53.353 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:53 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: +---[RSA 3072]----+ 2023-03-09T14:37:53.353 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:53 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: | ...oo| 2023-03-09T14:37:53.353 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:53 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: | +.++o| 2023-03-09T14:37:53.353 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:53 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: | Eo+.o+| 2023-03-09T14:37:53.353 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:53 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: | .o+. +| 2023-03-09T14:37:53.353 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:53 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: | S. . .| 2023-03-09T14:37:53.353 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:53 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: | . .. ... | 2023-03-09T14:37:53.353 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:53 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: | o .oo .+.+o.. | 2023-03-09T14:37:53.354 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:53 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: | o+.=*.o*.+o .| 2023-03-09T14:37:53.354 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:53 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: | o*+++oo+ o | 2023-03-09T14:37:53.354 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:37:53 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: +----[SHA256]-----+ 2023-03-09T14:37:53.707 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:52.445+0000 7f1f72f40700 1 Processor -- start 2023-03-09T14:37:53.707 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:52.446+0000 7f1f72f40700 1 -- start start 2023-03-09T14:37:53.707 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:52.446+0000 7f1f72f40700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1f6c079d10 0x7f1f6c10a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:37:53.707 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:52.446+0000 7f1f72f40700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f6c105f30 con 0x7f1f6c079d10 2023-03-09T14:37:53.708 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:52.446+0000 7f1f70cdc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1f6c079d10 0x7f1f6c10a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:37:53.708 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:52.446+0000 7f1f70cdc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1f6c079d10 0x7f1f6c10a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35490/0 (socket says 172.21.15.38:35490) 2023-03-09T14:37:53.708 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:52.446+0000 7f1f70cdc700 1 -- 172.21.15.38:0/1160803886 learned_addr learned my addr 172.21.15.38:0/1160803886 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:37:53.708 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:52.446+0000 7f1f70cdc700 1 -- 172.21.15.38:0/1160803886 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f6c10aa80 con 0x7f1f6c079d10 2023-03-09T14:37:53.708 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:52.446+0000 7f1f70cdc700 1 --2- 172.21.15.38:0/1160803886 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1f6c079d10 0x7f1f6c10a540 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1f600098a0 tx=0x7f1f60009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73a638437cd88ce2 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:37:53.708 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:52.447+0000 7f1f6b7fe700 1 -- 172.21.15.38:0/1160803886 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1f60004160 con 0x7f1f6c079d10 2023-03-09T14:37:53.708 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:52.447+0000 7f1f6b7fe700 1 -- 172.21.15.38:0/1160803886 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1f600042c0 con 0x7f1f6c079d10 2023-03-09T14:37:53.709 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:52.447+0000 7f1f6b7fe700 1 -- 172.21.15.38:0/1160803886 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1f60011a00 con 0x7f1f6c079d10 2023-03-09T14:37:53.709 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:52.447+0000 7f1f72f40700 1 -- 172.21.15.38:0/1160803886 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1f6c079d10 msgr2=0x7f1f6c10a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:37:53.709 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:52.447+0000 7f1f72f40700 1 --2- 172.21.15.38:0/1160803886 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1f6c079d10 0x7f1f6c10a540 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1f600098a0 tx=0x7f1f60009bb0 comp rx=0 tx=0).stop 2023-03-09T14:37:53.709 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:52.447+0000 7f1f72f40700 1 -- 172.21.15.38:0/1160803886 shutdown_connections 2023-03-09T14:37:53.709 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:52.447+0000 7f1f72f40700 1 --2- 172.21.15.38:0/1160803886 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1f6c079d10 0x7f1f6c10a540 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:37:53.709 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:52.447+0000 7f1f72f40700 1 -- 172.21.15.38:0/1160803886 >> 172.21.15.38:0/1160803886 conn(0x7f1f6c0ffb10 msgr2=0x7f1f6c101f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:37:53.709 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:52.447+0000 7f1f72f40700 1 -- 172.21.15.38:0/1160803886 shutdown_connections 2023-03-09T14:37:53.710 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:52.447+0000 7f1f72f40700 1 -- 172.21.15.38:0/1160803886 wait complete. 2023-03-09T14:37:53.710 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:52.448+0000 7f1f72f40700 1 Processor -- start 2023-03-09T14:37:53.710 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:52.448+0000 7f1f72f40700 1 -- start start 2023-03-09T14:37:53.710 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:52.448+0000 7f1f72f40700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1f6c079d10 0x7f1f6c19b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:37:53.710 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:52.448+0000 7f1f72f40700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f6c19b9a0 con 0x7f1f6c079d10 2023-03-09T14:37:53.710 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:52.448+0000 7f1f70cdc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1f6c079d10 0x7f1f6c19b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:37:53.710 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:52.449+0000 7f1f70cdc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1f6c079d10 0x7f1f6c19b460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35492/0 (socket says 172.21.15.38:35492) 2023-03-09T14:37:53.711 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:52.449+0000 7f1f70cdc700 1 -- 172.21.15.38:0/3287091596 learned_addr learned my addr 172.21.15.38:0/3287091596 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:37:53.711 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:52.449+0000 7f1f70cdc700 1 -- 172.21.15.38:0/3287091596 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f60009550 con 0x7f1f6c079d10 2023-03-09T14:37:53.711 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:52.449+0000 7f1f70cdc700 1 --2- 172.21.15.38:0/3287091596 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1f6c079d10 0x7f1f6c19b460 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1f60006af0 tx=0x7f1f60011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:37:53.711 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:52.449+0000 7f1f69ffb700 1 -- 172.21.15.38:0/3287091596 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1f60003a40 con 0x7f1f6c079d10 2023-03-09T14:37:53.711 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:52.449+0000 7f1f69ffb700 1 -- 172.21.15.38:0/3287091596 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1f60003ba0 con 0x7f1f6c079d10 2023-03-09T14:37:53.711 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:52.449+0000 7f1f72f40700 1 -- 172.21.15.38:0/3287091596 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f6c10ba10 con 0x7f1f6c079d10 2023-03-09T14:37:53.711 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:52.449+0000 7f1f69ffb700 1 -- 172.21.15.38:0/3287091596 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1f6001a700 con 0x7f1f6c079d10 2023-03-09T14:37:53.711 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:52.449+0000 7f1f72f40700 1 -- 172.21.15.38:0/3287091596 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f6c10b880 con 0x7f1f6c079d10 2023-03-09T14:37:53.712 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:52.450+0000 7f1f69ffb700 1 -- 172.21.15.38:0/3287091596 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) v1 ==== 44618+0+0 (secure 0 0 0) 0x7f1f60017020 con 0x7f1f6c079d10 2023-03-09T14:37:53.712 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:52.450+0000 7f1f69ffb700 1 --2- 172.21.15.38:0/3287091596 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7f1f5c037dd0 0x7f1f5c03a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:37:53.712 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:52.450+0000 7f1f69ffb700 1 -- 172.21.15.38:0/3287091596 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f1f6004b620 con 0x7f1f6c079d10 2023-03-09T14:37:53.712 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:52.451+0000 7f1f6bfff700 1 --2- 172.21.15.38:0/3287091596 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7f1f5c037dd0 0x7f1f5c03a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:37:53.712 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:52.451+0000 7f1f72f40700 1 -- 172.21.15.38:0/3287091596 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f6c061db0 con 0x7f1f6c079d10 2023-03-09T14:37:53.712 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:52.451+0000 7f1f6bfff700 1 --2- 172.21.15.38:0/3287091596 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7f1f5c037dd0 0x7f1f5c03a280 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1f58006fd0 tx=0x7f1f58006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:37:53.712 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:52.456+0000 7f1f69ffb700 1 -- 172.21.15.38:0/3287091596 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f1f6002d3f0 con 0x7f1f6c079d10 2023-03-09T14:37:53.713 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:52.815+0000 7f1f72f40700 1 -- 172.21.15.38:0/3287091596 --> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f1f6c100f60 con 0x7f1f5c037dd0 2023-03-09T14:37:53.713 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:53.312+0000 7f1f69ffb700 1 -- 172.21.15.38:0/3287091596 <== mgr.14118 v2:172.21.15.38:6800/3381625669 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f1f6c100f60 con 0x7f1f5c037dd0 2023-03-09T14:37:53.713 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:53.314+0000 7f1f72f40700 1 -- 172.21.15.38:0/3287091596 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7f1f5c037dd0 msgr2=0x7f1f5c03a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:37:53.713 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:53.314+0000 7f1f72f40700 1 --2- 172.21.15.38:0/3287091596 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7f1f5c037dd0 0x7f1f5c03a280 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1f58006fd0 tx=0x7f1f58006e40 comp rx=0 tx=0).stop 2023-03-09T14:37:53.713 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:53.314+0000 7f1f72f40700 1 -- 172.21.15.38:0/3287091596 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1f6c079d10 msgr2=0x7f1f6c19b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:37:53.713 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:53.314+0000 7f1f72f40700 1 --2- 172.21.15.38:0/3287091596 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1f6c079d10 0x7f1f6c19b460 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f1f60006af0 tx=0x7f1f60011ad0 comp rx=0 tx=0).stop 2023-03-09T14:37:53.713 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:53.314+0000 7f1f72f40700 1 -- 172.21.15.38:0/3287091596 shutdown_connections 2023-03-09T14:37:53.713 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:53.314+0000 7f1f72f40700 1 --2- 172.21.15.38:0/3287091596 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7f1f5c037dd0 0x7f1f5c03a280 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:37:53.714 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:53.314+0000 7f1f72f40700 1 --2- 172.21.15.38:0/3287091596 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1f6c079d10 0x7f1f6c19b460 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:37:53.714 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:53.314+0000 7f1f72f40700 1 -- 172.21.15.38:0/3287091596 >> 172.21.15.38:0/3287091596 conn(0x7f1f6c0ffb10 msgr2=0x7f1f6c100850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:37:53.714 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:53.315+0000 7f1f72f40700 1 -- 172.21.15.38:0/3287091596 shutdown_connections 2023-03-09T14:37:53.714 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:53.315+0000 7f1f72f40700 1 -- 172.21.15.38:0/3287091596 wait complete. 2023-03-09T14:37:54.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:54 smithi038 ceph-mon[92422]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:37:54.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:54 smithi038 ceph-mon[92422]: Generating ssh key... 2023-03-09T14:37:54.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:54 smithi038 ceph-mon[92422]: from='mgr.14118 172.21.15.38:0/2766508037' entity='mgr.x' 2023-03-09T14:37:54.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:54 smithi038 ceph-mon[92422]: from='mgr.14118 172.21.15.38:0/2766508037' entity='mgr.x' 2023-03-09T14:37:54.950 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDhsbGemUhVpwqlPHk2x7Z0y9Uw1fxRhFUHkRkV82fljmtiu5YJpaUJei76FM2/wSE1Vanlqr9PFER836Go7lQ7iYo6RWyRCrQbZhM1FxoCEmCMFXhannKj19p+bCozso8GjpU4rZ9O3+usHVpl2jLgy8ZvqeV3J6J/SSztgvekyW7Vzzu1prKGvavOHnn6jAeWk3FRG9Q7ouKki+8O+olx0hXGoXcaCM42EavDZPYxXdKECgDJgqg89Uhp40vJOZttoliDajkUXZ7nS5rLNJPco/Gic/HKz4y6uC2uhtmbcUpznPUtuEGqQSpLiisf8hYwrSotoR3aN7Wl9JKIpFg/wq/7B46het2zCvT6sb1IYR86so2J5haBhDDnDrmEWwbj6KDlf2j1Hu7GOvyvB8Tb2aG7T2Vgnn7vgltDPX7WnxWkAREuEf1virBG6mZuInQWXsWmGvr68SjsWFuJRQZwYyQPeLOVTRRORe0Z0FXr8ThBD82D2p5qxCbKLJhx4Cs= ceph-c6733fe2-be87-11ed-9af2-001a4aab830c 2023-03-09T14:37:54.950 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:54.211+0000 7f7567169700 1 Processor -- start 2023-03-09T14:37:54.951 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:54.211+0000 7f7567169700 1 -- start start 2023-03-09T14:37:54.951 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:54.211+0000 7f7567169700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7560106090 0x7f7560108470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:37:54.951 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:54.211+0000 7f7567169700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7560079db0 con 0x7f7560106090 2023-03-09T14:37:54.951 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:54.211+0000 7f7564f05700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7560106090 0x7f7560108470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:37:54.951 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:54.211+0000 7f7564f05700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7560106090 0x7f7560108470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35496/0 (socket says 172.21.15.38:35496) 2023-03-09T14:37:54.951 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:54.211+0000 7f7564f05700 1 -- 172.21.15.38:0/4209887116 learned_addr learned my addr 172.21.15.38:0/4209887116 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:37:54.951 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:54.211+0000 7f7564f05700 1 -- 172.21.15.38:0/4209887116 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7560079ef0 con 0x7f7560106090 2023-03-09T14:37:54.952 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:54.212+0000 7f7564f05700 1 --2- 172.21.15.38:0/4209887116 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7560106090 0x7f7560108470 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f75540098a0 tx=0x7f7554009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8fd6f0769165dbc server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:37:54.952 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:54.212+0000 7f755f7fe700 1 -- 172.21.15.38:0/4209887116 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7554004160 con 0x7f7560106090 2023-03-09T14:37:54.952 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:54.212+0000 7f755f7fe700 1 -- 172.21.15.38:0/4209887116 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f75540042c0 con 0x7f7560106090 2023-03-09T14:37:54.952 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:54.212+0000 7f755f7fe700 1 -- 172.21.15.38:0/4209887116 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7554011a00 con 0x7f7560106090 2023-03-09T14:37:54.952 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:54.212+0000 7f7567169700 1 -- 172.21.15.38:0/4209887116 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7560106090 msgr2=0x7f7560108470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:37:54.952 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:54.212+0000 7f7567169700 1 --2- 172.21.15.38:0/4209887116 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7560106090 0x7f7560108470 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f75540098a0 tx=0x7f7554009bb0 comp rx=0 tx=0).stop 2023-03-09T14:37:54.952 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:54.212+0000 7f7567169700 1 -- 172.21.15.38:0/4209887116 shutdown_connections 2023-03-09T14:37:54.952 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:54.212+0000 7f7567169700 1 --2- 172.21.15.38:0/4209887116 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7560106090 0x7f7560108470 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:37:54.953 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:54.212+0000 7f7567169700 1 -- 172.21.15.38:0/4209887116 >> 172.21.15.38:0/4209887116 conn(0x7f75600ffb10 msgr2=0x7f7560101f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:37:54.953 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:54.213+0000 7f7567169700 1 -- 172.21.15.38:0/4209887116 shutdown_connections 2023-03-09T14:37:54.953 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:54.213+0000 7f7567169700 1 -- 172.21.15.38:0/4209887116 wait complete. 2023-03-09T14:37:54.953 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:54.213+0000 7f7567169700 1 Processor -- start 2023-03-09T14:37:54.953 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:54.213+0000 7f7567169700 1 -- start start 2023-03-09T14:37:54.954 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:54.214+0000 7f7567169700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7560106090 0x7f7560199260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:37:54.954 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:54.214+0000 7f7567169700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75601997a0 con 0x7f7560106090 2023-03-09T14:37:54.954 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:54.214+0000 7f7564f05700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7560106090 0x7f7560199260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:37:54.954 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:54.214+0000 7f7564f05700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7560106090 0x7f7560199260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35498/0 (socket says 172.21.15.38:35498) 2023-03-09T14:37:54.954 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:54.214+0000 7f7564f05700 1 -- 172.21.15.38:0/1385248611 learned_addr learned my addr 172.21.15.38:0/1385248611 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:37:54.954 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:54.214+0000 7f7564f05700 1 -- 172.21.15.38:0/1385248611 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7554009550 con 0x7f7560106090 2023-03-09T14:37:54.954 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:54.214+0000 7f7564f05700 1 --2- 172.21.15.38:0/1385248611 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7560106090 0x7f7560199260 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7554009fb0 tx=0x7f7554011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:37:54.954 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:54.214+0000 7f755dffb700 1 -- 172.21.15.38:0/1385248611 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7554003930 con 0x7f7560106090 2023-03-09T14:37:54.955 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:54.214+0000 7f755dffb700 1 -- 172.21.15.38:0/1385248611 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7554003a90 con 0x7f7560106090 2023-03-09T14:37:54.955 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:54.214+0000 7f755dffb700 1 -- 172.21.15.38:0/1385248611 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f75540195e0 con 0x7f7560106090 2023-03-09T14:37:54.955 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:54.214+0000 7f7567169700 1 -- 172.21.15.38:0/1385248611 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7560109810 con 0x7f7560106090 2023-03-09T14:37:54.955 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:54.215+0000 7f7567169700 1 -- 172.21.15.38:0/1385248611 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7560109680 con 0x7f7560106090 2023-03-09T14:37:54.955 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:54.215+0000 7f755dffb700 1 -- 172.21.15.38:0/1385248611 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) v1 ==== 44618+0+0 (secure 0 0 0) 0x7f755402a020 con 0x7f7560106090 2023-03-09T14:37:54.955 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:54.216+0000 7f755dffb700 1 --2- 172.21.15.38:0/1385248611 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7f7550037d80 0x7f755003a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:37:54.955 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:54.216+0000 7f755dffb700 1 -- 172.21.15.38:0/1385248611 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f755404b640 con 0x7f7560106090 2023-03-09T14:37:54.956 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:54.216+0000 7f7567169700 1 -- 172.21.15.38:0/1385248611 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7560061db0 con 0x7f7560106090 2023-03-09T14:37:54.956 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:54.216+0000 7f755ffff700 1 --2- 172.21.15.38:0/1385248611 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7f7550037d80 0x7f755003a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:37:54.956 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:54.216+0000 7f755ffff700 1 --2- 172.21.15.38:0/1385248611 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7f7550037d80 0x7f755003a230 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f754c006fd0 tx=0x7f754c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:37:54.956 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:54.221+0000 7f755dffb700 1 -- 172.21.15.38:0/1385248611 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f7554003c00 con 0x7f7560106090 2023-03-09T14:37:54.956 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:54.582+0000 7f7567169700 1 -- 172.21.15.38:0/1385248611 --> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f7560100fc0 con 0x7f7550037d80 2023-03-09T14:37:54.956 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:54.583+0000 7f755dffb700 1 -- 172.21.15.38:0/1385248611 <== mgr.14118 v2:172.21.15.38:6800/3381625669 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f7560100fc0 con 0x7f7550037d80 2023-03-09T14:37:54.956 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:54.585+0000 7f7567169700 1 -- 172.21.15.38:0/1385248611 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7f7550037d80 msgr2=0x7f755003a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:37:54.956 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:54.585+0000 7f7567169700 1 --2- 172.21.15.38:0/1385248611 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7f7550037d80 0x7f755003a230 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f754c006fd0 tx=0x7f754c006e40 comp rx=0 tx=0).stop 2023-03-09T14:37:54.957 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:54.585+0000 7f7567169700 1 -- 172.21.15.38:0/1385248611 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7560106090 msgr2=0x7f7560199260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:37:54.957 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:54.585+0000 7f7567169700 1 --2- 172.21.15.38:0/1385248611 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7560106090 0x7f7560199260 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7554009fb0 tx=0x7f7554011ad0 comp rx=0 tx=0).stop 2023-03-09T14:37:54.957 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:54.585+0000 7f7567169700 1 -- 172.21.15.38:0/1385248611 shutdown_connections 2023-03-09T14:37:54.957 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:54.586+0000 7f7567169700 1 --2- 172.21.15.38:0/1385248611 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7f7550037d80 0x7f755003a230 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:37:54.957 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:54.586+0000 7f7567169700 1 --2- 172.21.15.38:0/1385248611 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7560106090 0x7f7560199260 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:37:54.957 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:54.586+0000 7f7567169700 1 -- 172.21.15.38:0/1385248611 >> 172.21.15.38:0/1385248611 conn(0x7f75600ffb10 msgr2=0x7f75601008b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:37:54.957 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:54.586+0000 7f7567169700 1 -- 172.21.15.38:0/1385248611 shutdown_connections 2023-03-09T14:37:54.958 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:54.586+0000 7f7567169700 1 -- 172.21.15.38:0/1385248611 wait complete. 2023-03-09T14:37:54.958 INFO:teuthology.orchestra.run.smithi038.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2023-03-09T14:37:54.958 INFO:teuthology.orchestra.run.smithi038.stdout:Adding key to root@localhost authorized_keys... 2023-03-09T14:37:54.959 INFO:teuthology.orchestra.run.smithi038.stdout:Adding host smithi038... 2023-03-09T14:37:55.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:55 smithi038 ceph-mon[92422]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:37:56.378 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:56 smithi038 ceph-mon[92422]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi038", "addr": "172.21.15.38", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:37:58.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:58 smithi038 ceph-mon[92422]: Deploying cephadm binary to smithi038 2023-03-09T14:37:58.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:58 smithi038 ceph-mon[92422]: from='mgr.14118 172.21.15.38:0/2766508037' entity='mgr.x' 2023-03-09T14:37:58.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:58 smithi038 ceph-mon[92422]: from='mgr.14118 172.21.15.38:0/2766508037' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:37:58.307 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout Added host 'smithi038' with addr '172.21.15.38' 2023-03-09T14:37:58.307 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:55.459+0000 7f346eafa700 1 Processor -- start 2023-03-09T14:37:58.307 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:55.459+0000 7f346eafa700 1 -- start start 2023-03-09T14:37:58.307 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:55.459+0000 7f346eafa700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3468103ee0 0x7f34681042f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:37:58.307 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:55.459+0000 7f346eafa700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3468079d50 con 0x7f3468103ee0 2023-03-09T14:37:58.308 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:55.459+0000 7f346c896700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3468103ee0 0x7f34681042f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:37:58.308 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:55.459+0000 7f346c896700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3468103ee0 0x7f34681042f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35502/0 (socket says 172.21.15.38:35502) 2023-03-09T14:37:58.308 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:55.459+0000 7f346c896700 1 -- 172.21.15.38:0/1821843056 learned_addr learned my addr 172.21.15.38:0/1821843056 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:37:58.308 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:55.460+0000 7f346c896700 1 -- 172.21.15.38:0/1821843056 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3468079e90 con 0x7f3468103ee0 2023-03-09T14:37:58.308 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:55.460+0000 7f346c896700 1 --2- 172.21.15.38:0/1821843056 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3468103ee0 0x7f34681042f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f345c0098a0 tx=0x7f345c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45970505653a506c server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:37:58.308 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:55.460+0000 7f34677fe700 1 -- 172.21.15.38:0/1821843056 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f345c004160 con 0x7f3468103ee0 2023-03-09T14:37:58.309 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:55.460+0000 7f34677fe700 1 -- 172.21.15.38:0/1821843056 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f345c0042c0 con 0x7f3468103ee0 2023-03-09T14:37:58.309 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:55.460+0000 7f34677fe700 1 -- 172.21.15.38:0/1821843056 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f345c011a00 con 0x7f3468103ee0 2023-03-09T14:37:58.309 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:55.460+0000 7f346eafa700 1 -- 172.21.15.38:0/1821843056 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3468103ee0 msgr2=0x7f34681042f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:37:58.309 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:55.460+0000 7f346eafa700 1 --2- 172.21.15.38:0/1821843056 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3468103ee0 0x7f34681042f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f345c0098a0 tx=0x7f345c009bb0 comp rx=0 tx=0).stop 2023-03-09T14:37:58.309 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:55.461+0000 7f346eafa700 1 -- 172.21.15.38:0/1821843056 shutdown_connections 2023-03-09T14:37:58.309 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:55.461+0000 7f346eafa700 1 --2- 172.21.15.38:0/1821843056 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3468103ee0 0x7f34681042f0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:37:58.309 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:55.461+0000 7f346eafa700 1 -- 172.21.15.38:0/1821843056 >> 172.21.15.38:0/1821843056 conn(0x7f34680ffb10 msgr2=0x7f3468101f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:37:58.310 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:55.461+0000 7f346eafa700 1 -- 172.21.15.38:0/1821843056 shutdown_connections 2023-03-09T14:37:58.310 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:55.461+0000 7f346eafa700 1 -- 172.21.15.38:0/1821843056 wait complete. 2023-03-09T14:37:58.310 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:55.461+0000 7f346eafa700 1 Processor -- start 2023-03-09T14:37:58.310 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:55.461+0000 7f346eafa700 1 -- start start 2023-03-09T14:37:58.310 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:55.461+0000 7f346eafa700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3468103ee0 0x7f346819b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:37:58.310 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:55.461+0000 7f346eafa700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f346819b990 con 0x7f3468103ee0 2023-03-09T14:37:58.311 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:55.461+0000 7f346c896700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3468103ee0 0x7f346819b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:37:58.311 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:55.462+0000 7f346c896700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3468103ee0 0x7f346819b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35504/0 (socket says 172.21.15.38:35504) 2023-03-09T14:37:58.311 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:55.462+0000 7f346c896700 1 -- 172.21.15.38:0/2160380188 learned_addr learned my addr 172.21.15.38:0/2160380188 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:37:58.311 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:55.462+0000 7f346c896700 1 -- 172.21.15.38:0/2160380188 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f345c009550 con 0x7f3468103ee0 2023-03-09T14:37:58.311 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:55.462+0000 7f346c896700 1 --2- 172.21.15.38:0/2160380188 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3468103ee0 0x7f346819b450 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f345c009fb0 tx=0x7f345c011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:37:58.311 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:55.462+0000 7f3465ffb700 1 -- 172.21.15.38:0/2160380188 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f345c003930 con 0x7f3468103ee0 2023-03-09T14:37:58.312 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:55.462+0000 7f3465ffb700 1 -- 172.21.15.38:0/2160380188 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f345c003a90 con 0x7f3468103ee0 2023-03-09T14:37:58.312 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:55.462+0000 7f346eafa700 1 -- 172.21.15.38:0/2160380188 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3468105630 con 0x7f3468103ee0 2023-03-09T14:37:58.312 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:55.462+0000 7f3465ffb700 1 -- 172.21.15.38:0/2160380188 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f345c0195e0 con 0x7f3468103ee0 2023-03-09T14:37:58.312 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:55.462+0000 7f346eafa700 1 -- 172.21.15.38:0/2160380188 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34681054a0 con 0x7f3468103ee0 2023-03-09T14:37:58.312 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:55.463+0000 7f3465ffb700 1 -- 172.21.15.38:0/2160380188 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) v1 ==== 44618+0+0 (secure 0 0 0) 0x7f345c02a020 con 0x7f3468103ee0 2023-03-09T14:37:58.312 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:55.463+0000 7f3465ffb700 1 --2- 172.21.15.38:0/2160380188 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7f3458037d80 0x7f345803a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:37:58.312 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:55.463+0000 7f3465ffb700 1 -- 172.21.15.38:0/2160380188 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f345c04b860 con 0x7f3468103ee0 2023-03-09T14:37:58.312 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:55.463+0000 7f346eafa700 1 -- 172.21.15.38:0/2160380188 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34681057c0 con 0x7f3468103ee0 2023-03-09T14:37:58.313 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:55.464+0000 7f3467fff700 1 --2- 172.21.15.38:0/2160380188 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7f3458037d80 0x7f345803a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:37:58.313 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:55.464+0000 7f3467fff700 1 --2- 172.21.15.38:0/2160380188 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7f3458037d80 0x7f345803a230 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f3454006fd0 tx=0x7f3454006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:37:58.313 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:55.468+0000 7f3465ffb700 1 -- 172.21.15.38:0/2160380188 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f345c02d440 con 0x7f3468103ee0 2023-03-09T14:37:58.313 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:55.835+0000 7f346eafa700 1 -- 172.21.15.38:0/2160380188 --> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi038", "addr": "172.21.15.38", "target": ["mon-mgr", ""]}) v1 -- 0x7f3468060bc0 con 0x7f3458037d80 2023-03-09T14:37:58.313 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:57.905+0000 7f3465ffb700 1 -- 172.21.15.38:0/2160380188 <== mgr.14118 v2:172.21.15.38:6800/3381625669 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f3468060bc0 con 0x7f3458037d80 2023-03-09T14:37:58.313 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:57.907+0000 7f346eafa700 1 -- 172.21.15.38:0/2160380188 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7f3458037d80 msgr2=0x7f345803a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:37:58.314 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:57.907+0000 7f346eafa700 1 --2- 172.21.15.38:0/2160380188 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7f3458037d80 0x7f345803a230 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f3454006fd0 tx=0x7f3454006e40 comp rx=0 tx=0).stop 2023-03-09T14:37:58.314 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:57.908+0000 7f346eafa700 1 -- 172.21.15.38:0/2160380188 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3468103ee0 msgr2=0x7f346819b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:37:58.314 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:57.908+0000 7f346eafa700 1 --2- 172.21.15.38:0/2160380188 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3468103ee0 0x7f346819b450 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f345c009fb0 tx=0x7f345c011ad0 comp rx=0 tx=0).stop 2023-03-09T14:37:58.314 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:57.908+0000 7f346eafa700 1 -- 172.21.15.38:0/2160380188 shutdown_connections 2023-03-09T14:37:58.314 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:57.908+0000 7f346eafa700 1 --2- 172.21.15.38:0/2160380188 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7f3458037d80 0x7f345803a230 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:37:58.314 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:57.908+0000 7f346eafa700 1 --2- 172.21.15.38:0/2160380188 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3468103ee0 0x7f346819b450 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:37:58.314 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:57.908+0000 7f346eafa700 1 -- 172.21.15.38:0/2160380188 >> 172.21.15.38:0/2160380188 conn(0x7f34680ffb10 msgr2=0x7f346818faa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:37:58.314 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:57.908+0000 7f346eafa700 1 -- 172.21.15.38:0/2160380188 shutdown_connections 2023-03-09T14:37:58.315 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:57.908+0000 7f346eafa700 1 -- 172.21.15.38:0/2160380188 wait complete. 2023-03-09T14:37:58.315 INFO:teuthology.orchestra.run.smithi038.stdout:Deploying unmanaged mon service... 2023-03-09T14:37:59.298 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:37:59 smithi038 ceph-mon[92422]: Added host smithi038 2023-03-09T14:38:00.051 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2023-03-09T14:38:00.051 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:58.841+0000 7f70623ed700 1 Processor -- start 2023-03-09T14:38:00.051 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:58.841+0000 7f70623ed700 1 -- start start 2023-03-09T14:38:00.051 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:58.842+0000 7f70623ed700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f705c1060d0 0x7f705c1084b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:38:00.052 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:58.842+0000 7f70623ed700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f705c079d70 con 0x7f705c1060d0 2023-03-09T14:38:00.052 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:58.842+0000 7f705bfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f705c1060d0 0x7f705c1084b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:38:00.052 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:58.842+0000 7f705bfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f705c1060d0 0x7f705c1084b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35538/0 (socket says 172.21.15.38:35538) 2023-03-09T14:38:00.052 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:58.842+0000 7f705bfff700 1 -- 172.21.15.38:0/2771459449 learned_addr learned my addr 172.21.15.38:0/2771459449 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:38:00.052 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:58.842+0000 7f705bfff700 1 -- 172.21.15.38:0/2771459449 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f705c079eb0 con 0x7f705c1060d0 2023-03-09T14:38:00.052 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:58.842+0000 7f705bfff700 1 --2- 172.21.15.38:0/2771459449 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f705c1060d0 0x7f705c1084b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f704c0098a0 tx=0x7f704c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55cb6c72654d4f33 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:38:00.052 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:58.843+0000 7f705b7fe700 1 -- 172.21.15.38:0/2771459449 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f704c004160 con 0x7f705c1060d0 2023-03-09T14:38:00.052 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:58.843+0000 7f705b7fe700 1 -- 172.21.15.38:0/2771459449 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f704c0042c0 con 0x7f705c1060d0 2023-03-09T14:38:00.053 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:58.843+0000 7f705b7fe700 1 -- 172.21.15.38:0/2771459449 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f704c011a00 con 0x7f705c1060d0 2023-03-09T14:38:00.053 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:58.843+0000 7f70623ed700 1 -- 172.21.15.38:0/2771459449 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f705c1060d0 msgr2=0x7f705c1084b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:38:00.053 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:58.843+0000 7f70623ed700 1 --2- 172.21.15.38:0/2771459449 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f705c1060d0 0x7f705c1084b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f704c0098a0 tx=0x7f704c009bb0 comp rx=0 tx=0).stop 2023-03-09T14:38:00.053 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:58.843+0000 7f70623ed700 1 -- 172.21.15.38:0/2771459449 shutdown_connections 2023-03-09T14:38:00.053 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:58.843+0000 7f70623ed700 1 --2- 172.21.15.38:0/2771459449 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f705c1060d0 0x7f705c1084b0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:38:00.053 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:58.843+0000 7f70623ed700 1 -- 172.21.15.38:0/2771459449 >> 172.21.15.38:0/2771459449 conn(0x7f705c0ffb50 msgr2=0x7f705c101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:38:00.053 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:58.843+0000 7f70623ed700 1 -- 172.21.15.38:0/2771459449 shutdown_connections 2023-03-09T14:38:00.054 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:58.843+0000 7f70623ed700 1 -- 172.21.15.38:0/2771459449 wait complete. 2023-03-09T14:38:00.054 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:58.844+0000 7f70623ed700 1 Processor -- start 2023-03-09T14:38:00.054 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:58.844+0000 7f70623ed700 1 -- start start 2023-03-09T14:38:00.054 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:58.844+0000 7f70623ed700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f705c1060d0 0x7f705c105a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:38:00.054 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:58.844+0000 7f70623ed700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f705c105f80 con 0x7f705c1060d0 2023-03-09T14:38:00.054 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:58.844+0000 7f705bfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f705c1060d0 0x7f705c105a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:38:00.054 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:58.844+0000 7f705bfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f705c1060d0 0x7f705c105a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35540/0 (socket says 172.21.15.38:35540) 2023-03-09T14:38:00.055 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:58.844+0000 7f705bfff700 1 -- 172.21.15.38:0/1385767539 learned_addr learned my addr 172.21.15.38:0/1385767539 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:38:00.055 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:58.845+0000 7f705bfff700 1 -- 172.21.15.38:0/1385767539 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f704c009550 con 0x7f705c1060d0 2023-03-09T14:38:00.055 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:58.845+0000 7f705bfff700 1 --2- 172.21.15.38:0/1385767539 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f705c1060d0 0x7f705c105a40 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f704c009fb0 tx=0x7f704c011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:38:00.055 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:58.845+0000 7f7059ffb700 1 -- 172.21.15.38:0/1385767539 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f704c003930 con 0x7f705c1060d0 2023-03-09T14:38:00.055 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:58.845+0000 7f7059ffb700 1 -- 172.21.15.38:0/1385767539 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f704c003a90 con 0x7f705c1060d0 2023-03-09T14:38:00.055 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:58.845+0000 7f70623ed700 1 -- 172.21.15.38:0/1385767539 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f705c1097f0 con 0x7f705c1060d0 2023-03-09T14:38:00.055 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:58.845+0000 7f7059ffb700 1 -- 172.21.15.38:0/1385767539 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f704c0195e0 con 0x7f705c1060d0 2023-03-09T14:38:00.055 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:58.845+0000 7f70623ed700 1 -- 172.21.15.38:0/1385767539 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f705c109660 con 0x7f705c1060d0 2023-03-09T14:38:00.056 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:58.846+0000 7f7059ffb700 1 -- 172.21.15.38:0/1385767539 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) v1 ==== 44618+0+0 (secure 0 0 0) 0x7f704c02a020 con 0x7f705c1060d0 2023-03-09T14:38:00.056 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:58.846+0000 7f7059ffb700 1 --2- 172.21.15.38:0/1385767539 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7f7048037d30 0x7f704803a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:38:00.056 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:58.846+0000 7f7059ffb700 1 -- 172.21.15.38:0/1385767539 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f704c04b8e0 con 0x7f705c1060d0 2023-03-09T14:38:00.056 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:58.846+0000 7f70623ed700 1 -- 172.21.15.38:0/1385767539 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f705c061db0 con 0x7f705c1060d0 2023-03-09T14:38:00.056 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:58.846+0000 7f7053fff700 1 --2- 172.21.15.38:0/1385767539 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7f7048037d30 0x7f704803a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:38:00.056 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:58.847+0000 7f7053fff700 1 --2- 172.21.15.38:0/1385767539 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7f7048037d30 0x7f704803a1e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7044006fd0 tx=0x7f7044006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:38:00.056 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:58.851+0000 7f7059ffb700 1 -- 172.21.15.38:0/1385767539 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f704c02d440 con 0x7f705c1060d0 2023-03-09T14:38:00.056 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:59.250+0000 7f70623ed700 1 -- 172.21.15.38:0/1385767539 --> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f705c104f10 con 0x7f7048037d30 2023-03-09T14:38:00.057 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:59.260+0000 7f7059ffb700 1 -- 172.21.15.38:0/1385767539 <== mgr.14118 v2:172.21.15.38:6800/3381625669 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f705c104f10 con 0x7f7048037d30 2023-03-09T14:38:00.057 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:59.262+0000 7f70623ed700 1 -- 172.21.15.38:0/1385767539 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7f7048037d30 msgr2=0x7f704803a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:38:00.057 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:59.262+0000 7f70623ed700 1 --2- 172.21.15.38:0/1385767539 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7f7048037d30 0x7f704803a1e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7044006fd0 tx=0x7f7044006e40 comp rx=0 tx=0).stop 2023-03-09T14:38:00.057 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:59.262+0000 7f70623ed700 1 -- 172.21.15.38:0/1385767539 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f705c1060d0 msgr2=0x7f705c105a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:38:00.057 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:59.262+0000 7f70623ed700 1 --2- 172.21.15.38:0/1385767539 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f705c1060d0 0x7f705c105a40 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f704c009fb0 tx=0x7f704c011ad0 comp rx=0 tx=0).stop 2023-03-09T14:38:00.057 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:59.262+0000 7f70623ed700 1 -- 172.21.15.38:0/1385767539 shutdown_connections 2023-03-09T14:38:00.057 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:59.262+0000 7f70623ed700 1 --2- 172.21.15.38:0/1385767539 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7f7048037d30 0x7f704803a1e0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:38:00.058 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:59.262+0000 7f70623ed700 1 --2- 172.21.15.38:0/1385767539 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f705c1060d0 0x7f705c105a40 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:38:00.058 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:59.262+0000 7f70623ed700 1 -- 172.21.15.38:0/1385767539 >> 172.21.15.38:0/1385767539 conn(0x7f705c0ffb50 msgr2=0x7f705c100840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:38:00.058 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:59.262+0000 7f70623ed700 1 -- 172.21.15.38:0/1385767539 shutdown_connections 2023-03-09T14:38:00.058 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:37:59.262+0000 7f70623ed700 1 -- 172.21.15.38:0/1385767539 wait complete. 2023-03-09T14:38:00.058 INFO:teuthology.orchestra.run.smithi038.stdout:Deploying unmanaged mgr service... 2023-03-09T14:38:00.290 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:00 smithi038 ceph-mon[92422]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:38:00.291 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:00 smithi038 ceph-mon[92422]: Saving service mon spec with placement count:5 2023-03-09T14:38:00.291 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:00 smithi038 ceph-mon[92422]: from='mgr.14118 172.21.15.38:0/2766508037' entity='mgr.x' 2023-03-09T14:38:00.291 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:00 smithi038 ceph-mon[92422]: from='mgr.14118 172.21.15.38:0/2766508037' entity='mgr.x' 2023-03-09T14:38:01.601 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2023-03-09T14:38:01.601 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:00.571+0000 7f27345d8700 1 Processor -- start 2023-03-09T14:38:01.601 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:00.571+0000 7f27345d8700 1 -- start start 2023-03-09T14:38:01.601 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:00.571+0000 7f27345d8700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f272c1060d0 0x7f272c1084b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:38:01.601 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:00.571+0000 7f27345d8700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f272c079d70 con 0x7f272c1060d0 2023-03-09T14:38:01.602 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:00.571+0000 7f2732374700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f272c1060d0 0x7f272c1084b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:38:01.602 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:00.571+0000 7f2732374700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f272c1060d0 0x7f272c1084b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35544/0 (socket says 172.21.15.38:35544) 2023-03-09T14:38:01.602 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:00.571+0000 7f2732374700 1 -- 172.21.15.38:0/2268927612 learned_addr learned my addr 172.21.15.38:0/2268927612 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:38:01.602 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:00.572+0000 7f2732374700 1 -- 172.21.15.38:0/2268927612 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f272c079eb0 con 0x7f272c1060d0 2023-03-09T14:38:01.602 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:00.572+0000 7f2732374700 1 --2- 172.21.15.38:0/2268927612 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f272c1060d0 0x7f272c1084b0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f27280098a0 tx=0x7f2728009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d794bbc5ed1806db server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:38:01.602 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:00.572+0000 7f2731372700 1 -- 172.21.15.38:0/2268927612 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2728004160 con 0x7f272c1060d0 2023-03-09T14:38:01.602 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:00.572+0000 7f2731372700 1 -- 172.21.15.38:0/2268927612 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f27280042c0 con 0x7f272c1060d0 2023-03-09T14:38:01.603 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:00.572+0000 7f2731372700 1 -- 172.21.15.38:0/2268927612 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f27280119b0 con 0x7f272c1060d0 2023-03-09T14:38:01.603 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:00.573+0000 7f27345d8700 1 -- 172.21.15.38:0/2268927612 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f272c1060d0 msgr2=0x7f272c1084b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:38:01.603 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:00.573+0000 7f27345d8700 1 --2- 172.21.15.38:0/2268927612 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f272c1060d0 0x7f272c1084b0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f27280098a0 tx=0x7f2728009bb0 comp rx=0 tx=0).stop 2023-03-09T14:38:01.603 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:00.573+0000 7f27345d8700 1 -- 172.21.15.38:0/2268927612 shutdown_connections 2023-03-09T14:38:01.603 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:00.573+0000 7f27345d8700 1 --2- 172.21.15.38:0/2268927612 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f272c1060d0 0x7f272c1084b0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:38:01.603 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:00.573+0000 7f27345d8700 1 -- 172.21.15.38:0/2268927612 >> 172.21.15.38:0/2268927612 conn(0x7f272c0ffb10 msgr2=0x7f272c101f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:38:01.603 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:00.573+0000 7f27345d8700 1 -- 172.21.15.38:0/2268927612 shutdown_connections 2023-03-09T14:38:01.603 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:00.573+0000 7f27345d8700 1 -- 172.21.15.38:0/2268927612 wait complete. 2023-03-09T14:38:01.604 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:00.574+0000 7f27345d8700 1 Processor -- start 2023-03-09T14:38:01.604 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:00.574+0000 7f27345d8700 1 -- start start 2023-03-09T14:38:01.604 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:00.574+0000 7f27345d8700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f272c1060d0 0x7f272c105a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:38:01.604 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:00.574+0000 7f27345d8700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f272c105f80 con 0x7f272c1060d0 2023-03-09T14:38:01.604 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:00.574+0000 7f2732374700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f272c1060d0 0x7f272c105a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:38:01.604 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:00.574+0000 7f2732374700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f272c1060d0 0x7f272c105a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35546/0 (socket says 172.21.15.38:35546) 2023-03-09T14:38:01.604 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:00.574+0000 7f2732374700 1 -- 172.21.15.38:0/1554762761 learned_addr learned my addr 172.21.15.38:0/1554762761 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:38:01.605 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:00.575+0000 7f2732374700 1 -- 172.21.15.38:0/1554762761 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2728009550 con 0x7f272c1060d0 2023-03-09T14:38:01.605 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:00.575+0000 7f2732374700 1 --2- 172.21.15.38:0/1554762761 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f272c1060d0 0x7f272c105a40 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2728000c00 tx=0x7f2728011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:38:01.605 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:00.575+0000 7f27237fe700 1 -- 172.21.15.38:0/1554762761 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f27280039f0 con 0x7f272c1060d0 2023-03-09T14:38:01.605 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:00.575+0000 7f27237fe700 1 -- 172.21.15.38:0/1554762761 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2728003b50 con 0x7f272c1060d0 2023-03-09T14:38:01.605 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:00.575+0000 7f27345d8700 1 -- 172.21.15.38:0/1554762761 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f272c109830 con 0x7f272c1060d0 2023-03-09T14:38:01.605 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:00.575+0000 7f27237fe700 1 -- 172.21.15.38:0/1554762761 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f272801a780 con 0x7f272c1060d0 2023-03-09T14:38:01.605 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:00.575+0000 7f27345d8700 1 -- 172.21.15.38:0/1554762761 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f272c1096a0 con 0x7f272c1060d0 2023-03-09T14:38:01.606 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:00.576+0000 7f27237fe700 1 -- 172.21.15.38:0/1554762761 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) v1 ==== 44618+0+0 (secure 0 0 0) 0x7f2728017020 con 0x7f272c1060d0 2023-03-09T14:38:01.606 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:00.576+0000 7f27237fe700 1 --2- 172.21.15.38:0/1554762761 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7f2718037dd0 0x7f271803a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:38:01.606 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:00.576+0000 7f27237fe700 1 -- 172.21.15.38:0/1554762761 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f2728024a50 con 0x7f272c1060d0 2023-03-09T14:38:01.606 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:00.576+0000 7f27345d8700 1 -- 172.21.15.38:0/1554762761 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f272c061db0 con 0x7f272c1060d0 2023-03-09T14:38:01.606 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:00.576+0000 7f2731b73700 1 --2- 172.21.15.38:0/1554762761 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7f2718037dd0 0x7f271803a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:38:01.606 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:00.577+0000 7f2731b73700 1 --2- 172.21.15.38:0/1554762761 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7f2718037dd0 0x7f271803a280 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f271c006fd0 tx=0x7f271c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:38:01.606 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:00.581+0000 7f27237fe700 1 -- 172.21.15.38:0/1554762761 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f272801a9f0 con 0x7f272c1060d0 2023-03-09T14:38:01.607 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:00.964+0000 7f27345d8700 1 -- 172.21.15.38:0/1554762761 --> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f272c104f40 con 0x7f2718037dd0 2023-03-09T14:38:01.607 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:00.969+0000 7f27237fe700 1 -- 172.21.15.38:0/1554762761 <== mgr.14118 v2:172.21.15.38:6800/3381625669 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f272c104f40 con 0x7f2718037dd0 2023-03-09T14:38:01.607 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:00.972+0000 7f27345d8700 1 -- 172.21.15.38:0/1554762761 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7f2718037dd0 msgr2=0x7f271803a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:38:01.607 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:00.972+0000 7f27345d8700 1 --2- 172.21.15.38:0/1554762761 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7f2718037dd0 0x7f271803a280 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f271c006fd0 tx=0x7f271c006e40 comp rx=0 tx=0).stop 2023-03-09T14:38:01.607 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:00.972+0000 7f27345d8700 1 -- 172.21.15.38:0/1554762761 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f272c1060d0 msgr2=0x7f272c105a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:38:01.607 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:00.972+0000 7f27345d8700 1 --2- 172.21.15.38:0/1554762761 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f272c1060d0 0x7f272c105a40 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2728000c00 tx=0x7f2728011a30 comp rx=0 tx=0).stop 2023-03-09T14:38:01.607 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:00.972+0000 7f27345d8700 1 -- 172.21.15.38:0/1554762761 shutdown_connections 2023-03-09T14:38:01.607 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:00.972+0000 7f27345d8700 1 --2- 172.21.15.38:0/1554762761 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7f2718037dd0 0x7f271803a280 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:38:01.608 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:00.972+0000 7f27345d8700 1 --2- 172.21.15.38:0/1554762761 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f272c1060d0 0x7f272c105a40 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:38:01.608 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:00.972+0000 7f27345d8700 1 -- 172.21.15.38:0/1554762761 >> 172.21.15.38:0/1554762761 conn(0x7f272c0ffb10 msgr2=0x7f272c100820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:38:01.608 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:00.972+0000 7f27345d8700 1 -- 172.21.15.38:0/1554762761 shutdown_connections 2023-03-09T14:38:01.608 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:00.972+0000 7f27345d8700 1 -- 172.21.15.38:0/1554762761 wait complete. 2023-03-09T14:38:02.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:01 smithi038 ceph-mon[92422]: from='mgr.14118 172.21.15.38:0/2766508037' entity='mgr.x' 2023-03-09T14:38:02.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:01 smithi038 ceph-mon[92422]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:38:02.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:01 smithi038 ceph-mon[92422]: Saving service mgr spec with placement count:2 2023-03-09T14:38:02.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:01 smithi038 ceph-mon[92422]: from='mgr.14118 172.21.15.38:0/2766508037' entity='mgr.x' 2023-03-09T14:38:03.351 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:02.635+0000 7f8b1cb2e700 1 Processor -- start 2023-03-09T14:38:03.351 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:02.635+0000 7f8b1cb2e700 1 -- start start 2023-03-09T14:38:03.351 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:02.636+0000 7f8b1cb2e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8b18106090 0x7f8b18108470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:38:03.351 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:02.636+0000 7f8b1cb2e700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b18079db0 con 0x7f8b18106090 2023-03-09T14:38:03.352 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:02.636+0000 7f8b1659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8b18106090 0x7f8b18108470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:38:03.352 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:02.636+0000 7f8b1659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8b18106090 0x7f8b18108470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35550/0 (socket says 172.21.15.38:35550) 2023-03-09T14:38:03.352 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:02.636+0000 7f8b1659c700 1 -- 172.21.15.38:0/4240227592 learned_addr learned my addr 172.21.15.38:0/4240227592 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:38:03.352 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:02.636+0000 7f8b1659c700 1 -- 172.21.15.38:0/4240227592 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b18079ef0 con 0x7f8b18106090 2023-03-09T14:38:03.352 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:02.636+0000 7f8b1659c700 1 --2- 172.21.15.38:0/4240227592 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8b18106090 0x7f8b18108470 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8b0c0098a0 tx=0x7f8b0c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=864ce64b2c1547fb server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:38:03.352 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:02.636+0000 7f8b1559a700 1 -- 172.21.15.38:0/4240227592 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8b0c004160 con 0x7f8b18106090 2023-03-09T14:38:03.352 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:02.637+0000 7f8b1559a700 1 -- 172.21.15.38:0/4240227592 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8b0c0042c0 con 0x7f8b18106090 2023-03-09T14:38:03.353 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:02.637+0000 7f8b1559a700 1 -- 172.21.15.38:0/4240227592 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8b0c0119b0 con 0x7f8b18106090 2023-03-09T14:38:03.353 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:02.637+0000 7f8b1cb2e700 1 -- 172.21.15.38:0/4240227592 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8b18106090 msgr2=0x7f8b18108470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:38:03.353 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:02.637+0000 7f8b1cb2e700 1 --2- 172.21.15.38:0/4240227592 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8b18106090 0x7f8b18108470 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8b0c0098a0 tx=0x7f8b0c009bb0 comp rx=0 tx=0).stop 2023-03-09T14:38:03.353 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:02.637+0000 7f8b1cb2e700 1 -- 172.21.15.38:0/4240227592 shutdown_connections 2023-03-09T14:38:03.353 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:02.637+0000 7f8b1cb2e700 1 --2- 172.21.15.38:0/4240227592 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8b18106090 0x7f8b18108470 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:38:03.353 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:02.637+0000 7f8b1cb2e700 1 -- 172.21.15.38:0/4240227592 >> 172.21.15.38:0/4240227592 conn(0x7f8b180ffb10 msgr2=0x7f8b18101f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:38:03.354 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:02.637+0000 7f8b1cb2e700 1 -- 172.21.15.38:0/4240227592 shutdown_connections 2023-03-09T14:38:03.354 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:02.637+0000 7f8b1cb2e700 1 -- 172.21.15.38:0/4240227592 wait complete. 2023-03-09T14:38:03.354 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:02.638+0000 7f8b1cb2e700 1 Processor -- start 2023-03-09T14:38:03.354 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:02.638+0000 7f8b1cb2e700 1 -- start start 2023-03-09T14:38:03.354 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:02.638+0000 7f8b1cb2e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8b18106090 0x7f8b18105a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:38:03.354 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:02.638+0000 7f8b1cb2e700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b18105f40 con 0x7f8b18106090 2023-03-09T14:38:03.354 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:02.638+0000 7f8b1659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8b18106090 0x7f8b18105a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:38:03.354 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:02.638+0000 7f8b1659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8b18106090 0x7f8b18105a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35552/0 (socket says 172.21.15.38:35552) 2023-03-09T14:38:03.355 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:02.638+0000 7f8b1659c700 1 -- 172.21.15.38:0/3744663318 learned_addr learned my addr 172.21.15.38:0/3744663318 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:38:03.355 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:02.639+0000 7f8b1659c700 1 -- 172.21.15.38:0/3744663318 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b0c009550 con 0x7f8b18106090 2023-03-09T14:38:03.355 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:02.639+0000 7f8b1659c700 1 --2- 172.21.15.38:0/3744663318 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8b18106090 0x7f8b18105a00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f8b0c003e60 tx=0x7f8b0c011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:38:03.355 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:02.639+0000 7f8b077fe700 1 -- 172.21.15.38:0/3744663318 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8b0c003970 con 0x7f8b18106090 2023-03-09T14:38:03.355 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:02.639+0000 7f8b077fe700 1 -- 172.21.15.38:0/3744663318 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8b0c003ad0 con 0x7f8b18106090 2023-03-09T14:38:03.355 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:02.639+0000 7f8b1cb2e700 1 -- 172.21.15.38:0/3744663318 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b18109850 con 0x7f8b18106090 2023-03-09T14:38:03.355 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:02.639+0000 7f8b077fe700 1 -- 172.21.15.38:0/3744663318 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8b0c0195e0 con 0x7f8b18106090 2023-03-09T14:38:03.356 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:02.639+0000 7f8b1cb2e700 1 -- 172.21.15.38:0/3744663318 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b181096c0 con 0x7f8b18106090 2023-03-09T14:38:03.356 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:02.640+0000 7f8b077fe700 1 -- 172.21.15.38:0/3744663318 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) v1 ==== 44618+0+0 (secure 0 0 0) 0x7f8b0c02a020 con 0x7f8b18106090 2023-03-09T14:38:03.356 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:02.640+0000 7f8b077fe700 1 --2- 172.21.15.38:0/3744663318 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7f8b00037d80 0x7f8b0003a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:38:03.356 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:02.640+0000 7f8b1cb2e700 1 -- 172.21.15.38:0/3744663318 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b18061db0 con 0x7f8b18106090 2023-03-09T14:38:03.356 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:02.640+0000 7f8b077fe700 1 -- 172.21.15.38:0/3744663318 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f8b0c04b730 con 0x7f8b18106090 2023-03-09T14:38:03.356 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:02.641+0000 7f8b15d9b700 1 --2- 172.21.15.38:0/3744663318 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7f8b00037d80 0x7f8b0003a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:38:03.356 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:02.641+0000 7f8b15d9b700 1 --2- 172.21.15.38:0/3744663318 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7f8b00037d80 0x7f8b0003a230 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8b08006fd0 tx=0x7f8b08006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:38:03.356 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:02.645+0000 7f8b077fe700 1 -- 172.21.15.38:0/3744663318 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f8b0c02d440 con 0x7f8b18106090 2023-03-09T14:38:03.357 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:03.010+0000 7f8b1cb2e700 1 -- 172.21.15.38:0/3744663318 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f8b18104ee0 con 0x7f8b18106090 2023-03-09T14:38:03.357 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:03.019+0000 7f8b077fe700 1 -- 172.21.15.38:0/3744663318 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f8b0c018b40 con 0x7f8b18106090 2023-03-09T14:38:03.357 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:03.021+0000 7f8b1cb2e700 1 -- 172.21.15.38:0/3744663318 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7f8b00037d80 msgr2=0x7f8b0003a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:38:03.357 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:03.021+0000 7f8b1cb2e700 1 --2- 172.21.15.38:0/3744663318 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7f8b00037d80 0x7f8b0003a230 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8b08006fd0 tx=0x7f8b08006e40 comp rx=0 tx=0).stop 2023-03-09T14:38:03.357 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:03.021+0000 7f8b1cb2e700 1 -- 172.21.15.38:0/3744663318 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8b18106090 msgr2=0x7f8b18105a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:38:03.357 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:03.021+0000 7f8b1cb2e700 1 --2- 172.21.15.38:0/3744663318 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8b18106090 0x7f8b18105a00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f8b0c003e60 tx=0x7f8b0c011aa0 comp rx=0 tx=0).stop 2023-03-09T14:38:03.357 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:03.022+0000 7f8b1cb2e700 1 -- 172.21.15.38:0/3744663318 shutdown_connections 2023-03-09T14:38:03.358 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:03.022+0000 7f8b1cb2e700 1 --2- 172.21.15.38:0/3744663318 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7f8b00037d80 0x7f8b0003a230 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:38:03.358 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:03.022+0000 7f8b1cb2e700 1 --2- 172.21.15.38:0/3744663318 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8b18106090 0x7f8b18105a00 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:38:03.358 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:03.022+0000 7f8b1cb2e700 1 -- 172.21.15.38:0/3744663318 >> 172.21.15.38:0/3744663318 conn(0x7f8b180ffb10 msgr2=0x7f8b18100800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:38:03.358 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:03.022+0000 7f8b1cb2e700 1 -- 172.21.15.38:0/3744663318 shutdown_connections 2023-03-09T14:38:03.358 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:03.024+0000 7f8b1cb2e700 1 -- 172.21.15.38:0/3744663318 wait complete. 2023-03-09T14:38:04.349 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:04 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/3744663318' entity='client.admin' 2023-03-09T14:38:04.350 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:04 smithi038 ceph-mon[92422]: from='mgr.14118 172.21.15.38:0/2766508037' entity='mgr.x' 2023-03-09T14:38:04.606 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:03.841+0000 7ff699d88700 1 Processor -- start 2023-03-09T14:38:04.607 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:03.841+0000 7ff699d88700 1 -- start start 2023-03-09T14:38:04.607 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:03.841+0000 7ff699d88700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff6941060d0 0x7ff6941084b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:38:04.607 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:03.841+0000 7ff699d88700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff694079d70 con 0x7ff6941060d0 2023-03-09T14:38:04.607 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:03.842+0000 7ff6937fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff6941060d0 0x7ff6941084b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:38:04.607 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:03.842+0000 7ff6937fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff6941060d0 0x7ff6941084b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35556/0 (socket says 172.21.15.38:35556) 2023-03-09T14:38:04.607 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:03.842+0000 7ff6937fe700 1 -- 172.21.15.38:0/1973528280 learned_addr learned my addr 172.21.15.38:0/1973528280 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:38:04.607 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:03.842+0000 7ff6937fe700 1 -- 172.21.15.38:0/1973528280 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff694079eb0 con 0x7ff6941060d0 2023-03-09T14:38:04.608 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:03.842+0000 7ff6937fe700 1 --2- 172.21.15.38:0/1973528280 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff6941060d0 0x7ff6941084b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff6840098a0 tx=0x7ff684009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38cc64a102c46e server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:38:04.608 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:03.842+0000 7ff6927fc700 1 -- 172.21.15.38:0/1973528280 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff684004160 con 0x7ff6941060d0 2023-03-09T14:38:04.608 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:03.842+0000 7ff6927fc700 1 -- 172.21.15.38:0/1973528280 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff6840042c0 con 0x7ff6941060d0 2023-03-09T14:38:04.608 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:03.842+0000 7ff6927fc700 1 -- 172.21.15.38:0/1973528280 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff6840119b0 con 0x7ff6941060d0 2023-03-09T14:38:04.608 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:03.843+0000 7ff699d88700 1 -- 172.21.15.38:0/1973528280 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff6941060d0 msgr2=0x7ff6941084b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:38:04.608 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:03.843+0000 7ff699d88700 1 --2- 172.21.15.38:0/1973528280 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff6941060d0 0x7ff6941084b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff6840098a0 tx=0x7ff684009bb0 comp rx=0 tx=0).stop 2023-03-09T14:38:04.608 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:03.843+0000 7ff699d88700 1 -- 172.21.15.38:0/1973528280 shutdown_connections 2023-03-09T14:38:04.609 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:03.843+0000 7ff699d88700 1 --2- 172.21.15.38:0/1973528280 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff6941060d0 0x7ff6941084b0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:38:04.609 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:03.843+0000 7ff699d88700 1 -- 172.21.15.38:0/1973528280 >> 172.21.15.38:0/1973528280 conn(0x7ff6940ffb50 msgr2=0x7ff694101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:38:04.609 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:03.843+0000 7ff699d88700 1 -- 172.21.15.38:0/1973528280 shutdown_connections 2023-03-09T14:38:04.609 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:03.843+0000 7ff699d88700 1 -- 172.21.15.38:0/1973528280 wait complete. 2023-03-09T14:38:04.609 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:03.844+0000 7ff699d88700 1 Processor -- start 2023-03-09T14:38:04.610 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:03.844+0000 7ff699d88700 1 -- start start 2023-03-09T14:38:04.610 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:03.844+0000 7ff699d88700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff6941060d0 0x7ff694199280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:38:04.610 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:03.844+0000 7ff699d88700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6941997c0 con 0x7ff6941060d0 2023-03-09T14:38:04.610 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:03.844+0000 7ff6937fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff6941060d0 0x7ff694199280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:38:04.611 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:03.844+0000 7ff6937fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff6941060d0 0x7ff694199280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35558/0 (socket says 172.21.15.38:35558) 2023-03-09T14:38:04.611 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:03.844+0000 7ff6937fe700 1 -- 172.21.15.38:0/1248058438 learned_addr learned my addr 172.21.15.38:0/1248058438 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:38:04.611 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:03.845+0000 7ff6937fe700 1 -- 172.21.15.38:0/1248058438 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff684009550 con 0x7ff6941060d0 2023-03-09T14:38:04.612 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:03.845+0000 7ff6937fe700 1 --2- 172.21.15.38:0/1248058438 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff6941060d0 0x7ff694199280 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff684003e60 tx=0x7ff684011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:38:04.612 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:03.845+0000 7ff690ff9700 1 -- 172.21.15.38:0/1248058438 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff684003970 con 0x7ff6941060d0 2023-03-09T14:38:04.612 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:03.845+0000 7ff690ff9700 1 -- 172.21.15.38:0/1248058438 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff684003ad0 con 0x7ff6941060d0 2023-03-09T14:38:04.613 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:03.845+0000 7ff699d88700 1 -- 172.21.15.38:0/1248058438 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff694109830 con 0x7ff6941060d0 2023-03-09T14:38:04.613 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:03.845+0000 7ff690ff9700 1 -- 172.21.15.38:0/1248058438 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff6840195e0 con 0x7ff6941060d0 2023-03-09T14:38:04.613 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:03.845+0000 7ff699d88700 1 -- 172.21.15.38:0/1248058438 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6941096a0 con 0x7ff6941060d0 2023-03-09T14:38:04.613 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:03.846+0000 7ff690ff9700 1 -- 172.21.15.38:0/1248058438 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) v1 ==== 44618+0+0 (secure 0 0 0) 0x7ff68402a020 con 0x7ff6941060d0 2023-03-09T14:38:04.613 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:03.846+0000 7ff699d88700 1 -- 172.21.15.38:0/1248058438 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6941099c0 con 0x7ff6941060d0 2023-03-09T14:38:04.614 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:03.847+0000 7ff690ff9700 1 --2- 172.21.15.38:0/1248058438 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7ff67c037d30 0x7ff67c03a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:38:04.614 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:03.847+0000 7ff690ff9700 1 -- 172.21.15.38:0/1248058438 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7ff68404b6f0 con 0x7ff6941060d0 2023-03-09T14:38:04.614 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:03.847+0000 7ff692ffd700 1 --2- 172.21.15.38:0/1248058438 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7ff67c037d30 0x7ff67c03a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:38:04.614 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:03.847+0000 7ff692ffd700 1 --2- 172.21.15.38:0/1248058438 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7ff67c037d30 0x7ff67c03a1e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff688006fd0 tx=0x7ff688006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:38:04.614 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:03.851+0000 7ff690ff9700 1 -- 172.21.15.38:0/1248058438 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7ff68402b340 con 0x7ff6941060d0 2023-03-09T14:38:04.615 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:04.214+0000 7ff699d88700 1 -- 172.21.15.38:0/1248058438 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7ff69419cf10 con 0x7ff6941060d0 2023-03-09T14:38:04.615 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:04.219+0000 7ff690ff9700 1 -- 172.21.15.38:0/1248058438 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7ff684018b40 con 0x7ff6941060d0 2023-03-09T14:38:04.615 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:04.222+0000 7ff699d88700 1 -- 172.21.15.38:0/1248058438 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7ff67c037d30 msgr2=0x7ff67c03a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:38:04.615 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:04.222+0000 7ff699d88700 1 --2- 172.21.15.38:0/1248058438 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7ff67c037d30 0x7ff67c03a1e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff688006fd0 tx=0x7ff688006e40 comp rx=0 tx=0).stop 2023-03-09T14:38:04.615 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:04.222+0000 7ff699d88700 1 -- 172.21.15.38:0/1248058438 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff6941060d0 msgr2=0x7ff694199280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:38:04.615 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:04.222+0000 7ff699d88700 1 --2- 172.21.15.38:0/1248058438 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff6941060d0 0x7ff694199280 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff684003e60 tx=0x7ff684011aa0 comp rx=0 tx=0).stop 2023-03-09T14:38:04.615 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:04.222+0000 7ff699d88700 1 -- 172.21.15.38:0/1248058438 shutdown_connections 2023-03-09T14:38:04.616 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:04.222+0000 7ff699d88700 1 --2- 172.21.15.38:0/1248058438 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7ff67c037d30 0x7ff67c03a1e0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:38:04.616 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:04.222+0000 7ff699d88700 1 --2- 172.21.15.38:0/1248058438 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff6941060d0 0x7ff694199280 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:38:04.616 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:04.222+0000 7ff699d88700 1 -- 172.21.15.38:0/1248058438 >> 172.21.15.38:0/1248058438 conn(0x7ff6940ffb50 msgr2=0x7ff694100840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:38:04.616 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:04.222+0000 7ff699d88700 1 -- 172.21.15.38:0/1248058438 shutdown_connections 2023-03-09T14:38:04.616 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:04.222+0000 7ff699d88700 1 -- 172.21.15.38:0/1248058438 wait complete. 2023-03-09T14:38:04.616 INFO:teuthology.orchestra.run.smithi038.stdout:Enabling the dashboard module... 2023-03-09T14:38:05.485 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:05 smithi038 ceph-mon[92422]: from='mgr.14118 172.21.15.38:0/2766508037' entity='mgr.x' 2023-03-09T14:38:05.485 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:05 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/1248058438' entity='client.admin' 2023-03-09T14:38:06.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:06 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/1177655990' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2023-03-09T14:38:06.501 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:38:06 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: ignoring --setuser ceph since I am not root 2023-03-09T14:38:06.501 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:38:06 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: ignoring --setgroup ceph since I am not root 2023-03-09T14:38:06.501 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:38:06 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:38:06.426+0000 7f3bab2a9200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-03-09T14:38:06.654 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:05.114+0000 7fdf171f8700 1 Processor -- start 2023-03-09T14:38:06.655 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:05.114+0000 7fdf171f8700 1 -- start start 2023-03-09T14:38:06.655 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:05.114+0000 7fdf171f8700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf1007f5c0 0x7fdf1007f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:38:06.655 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:05.115+0000 7fdf171f8700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf1007ff10 con 0x7fdf1007f5c0 2023-03-09T14:38:06.655 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:05.115+0000 7fdf14f94700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf1007f5c0 0x7fdf1007f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:38:06.655 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:05.115+0000 7fdf14f94700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf1007f5c0 0x7fdf1007f9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35562/0 (socket says 172.21.15.38:35562) 2023-03-09T14:38:06.655 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:05.115+0000 7fdf14f94700 1 -- 172.21.15.38:0/3193390766 learned_addr learned my addr 172.21.15.38:0/3193390766 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:38:06.656 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:05.115+0000 7fdf14f94700 1 -- 172.21.15.38:0/3193390766 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf10080050 con 0x7fdf1007f5c0 2023-03-09T14:38:06.656 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:05.115+0000 7fdf14f94700 1 --2- 172.21.15.38:0/3193390766 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf1007f5c0 0x7fdf1007f9d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fdf0c0098a0 tx=0x7fdf0c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83e1227253b69dc5 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:38:06.656 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:05.115+0000 7fdf077fe700 1 -- 172.21.15.38:0/3193390766 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdf0c004160 con 0x7fdf1007f5c0 2023-03-09T14:38:06.656 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:05.115+0000 7fdf077fe700 1 -- 172.21.15.38:0/3193390766 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdf0c0042c0 con 0x7fdf1007f5c0 2023-03-09T14:38:06.656 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:05.115+0000 7fdf077fe700 1 -- 172.21.15.38:0/3193390766 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdf0c0119b0 con 0x7fdf1007f5c0 2023-03-09T14:38:06.656 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:05.116+0000 7fdf171f8700 1 -- 172.21.15.38:0/3193390766 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf1007f5c0 msgr2=0x7fdf1007f9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:38:06.656 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:05.116+0000 7fdf171f8700 1 --2- 172.21.15.38:0/3193390766 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf1007f5c0 0x7fdf1007f9d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fdf0c0098a0 tx=0x7fdf0c009bb0 comp rx=0 tx=0).stop 2023-03-09T14:38:06.657 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:05.116+0000 7fdf171f8700 1 -- 172.21.15.38:0/3193390766 shutdown_connections 2023-03-09T14:38:06.657 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:05.116+0000 7fdf171f8700 1 --2- 172.21.15.38:0/3193390766 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf1007f5c0 0x7fdf1007f9d0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:38:06.657 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:05.116+0000 7fdf171f8700 1 -- 172.21.15.38:0/3193390766 >> 172.21.15.38:0/3193390766 conn(0x7fdf1007ac40 msgr2=0x7fdf1007d050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:38:06.657 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:05.116+0000 7fdf171f8700 1 -- 172.21.15.38:0/3193390766 shutdown_connections 2023-03-09T14:38:06.657 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:05.116+0000 7fdf171f8700 1 -- 172.21.15.38:0/3193390766 wait complete. 2023-03-09T14:38:06.657 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:05.117+0000 7fdf171f8700 1 Processor -- start 2023-03-09T14:38:06.658 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:05.117+0000 7fdf171f8700 1 -- start start 2023-03-09T14:38:06.658 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:05.117+0000 7fdf171f8700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf1007f5c0 0x7fdf1019f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:38:06.658 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:05.117+0000 7fdf171f8700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf1019fdc0 con 0x7fdf1007f5c0 2023-03-09T14:38:06.658 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:05.117+0000 7fdf14f94700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf1007f5c0 0x7fdf1019f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:38:06.658 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:05.117+0000 7fdf14f94700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf1007f5c0 0x7fdf1019f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35564/0 (socket says 172.21.15.38:35564) 2023-03-09T14:38:06.658 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:05.117+0000 7fdf14f94700 1 -- 172.21.15.38:0/1177655990 learned_addr learned my addr 172.21.15.38:0/1177655990 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:38:06.659 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:05.118+0000 7fdf14f94700 1 -- 172.21.15.38:0/1177655990 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf0c009550 con 0x7fdf1007f5c0 2023-03-09T14:38:06.659 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:05.118+0000 7fdf14f94700 1 --2- 172.21.15.38:0/1177655990 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf1007f5c0 0x7fdf1019f880 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fdf0c003e60 tx=0x7fdf0c011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:38:06.659 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:05.118+0000 7fdf05ffb700 1 -- 172.21.15.38:0/1177655990 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdf0c0038b0 con 0x7fdf1007f5c0 2023-03-09T14:38:06.659 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:05.118+0000 7fdf05ffb700 1 -- 172.21.15.38:0/1177655990 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdf0c003a10 con 0x7fdf1007f5c0 2023-03-09T14:38:06.659 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:05.118+0000 7fdf171f8700 1 -- 172.21.15.38:0/1177655990 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf1010dc30 con 0x7fdf1007f5c0 2023-03-09T14:38:06.659 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:05.118+0000 7fdf05ffb700 1 -- 172.21.15.38:0/1177655990 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdf0c019550 con 0x7fdf1007f5c0 2023-03-09T14:38:06.660 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:05.118+0000 7fdf171f8700 1 -- 172.21.15.38:0/1177655990 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf1010daa0 con 0x7fdf1007f5c0 2023-03-09T14:38:06.660 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:05.119+0000 7fdf05ffb700 1 -- 172.21.15.38:0/1177655990 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 8) v1 ==== 44618+0+0 (secure 0 0 0) 0x7fdf0c02a020 con 0x7fdf1007f5c0 2023-03-09T14:38:06.660 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:05.120+0000 7fdf05ffb700 1 --2- 172.21.15.38:0/1177655990 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7fdef8037d30 0x7fdef803a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:38:06.660 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:05.120+0000 7fdf05ffb700 1 -- 172.21.15.38:0/1177655990 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fdf0c04b720 con 0x7fdf1007f5c0 2023-03-09T14:38:06.660 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:05.120+0000 7fdf171f8700 1 -- 172.21.15.38:0/1177655990 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf10110020 con 0x7fdf1007f5c0 2023-03-09T14:38:06.660 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:05.120+0000 7fdf07fff700 1 --2- 172.21.15.38:0/1177655990 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7fdef8037d30 0x7fdef803a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:38:06.661 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:05.120+0000 7fdf07fff700 1 --2- 172.21.15.38:0/1177655990 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7fdef8037d30 0x7fdef803a1e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fdf00006fd0 tx=0x7fdf00006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:38:06.661 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:05.125+0000 7fdf05ffb700 1 -- 172.21.15.38:0/1177655990 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7fdf0c02d440 con 0x7fdf1007f5c0 2023-03-09T14:38:06.661 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:05.600+0000 7fdf171f8700 1 -- 172.21.15.38:0/1177655990 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7fdf101a0310 con 0x7fdf1007f5c0 2023-03-09T14:38:06.661 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:06.220+0000 7fdf05ffb700 1 -- 172.21.15.38:0/1177655990 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mgrmap(e 9) v1 ==== 44631+0+0 (secure 0 0 0) 0x7fdf0c017020 con 0x7fdf1007f5c0 2023-03-09T14:38:06.662 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:06.222+0000 7fdf05ffb700 1 -- 172.21.15.38:0/1177655990 <== mon.0 v2:172.21.15.38:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7fdf0c00b5c0 con 0x7fdf1007f5c0 2023-03-09T14:38:06.662 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:06.225+0000 7fdf171f8700 1 -- 172.21.15.38:0/1177655990 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7fdef8037d30 msgr2=0x7fdef803a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:38:06.662 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:06.225+0000 7fdf171f8700 1 --2- 172.21.15.38:0/1177655990 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7fdef8037d30 0x7fdef803a1e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fdf00006fd0 tx=0x7fdf00006e40 comp rx=0 tx=0).stop 2023-03-09T14:38:06.662 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:06.225+0000 7fdf171f8700 1 -- 172.21.15.38:0/1177655990 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf1007f5c0 msgr2=0x7fdf1019f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:38:06.663 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:06.225+0000 7fdf171f8700 1 --2- 172.21.15.38:0/1177655990 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf1007f5c0 0x7fdf1019f880 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fdf0c003e60 tx=0x7fdf0c011a30 comp rx=0 tx=0).stop 2023-03-09T14:38:06.663 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:06.225+0000 7fdf171f8700 1 -- 172.21.15.38:0/1177655990 shutdown_connections 2023-03-09T14:38:06.663 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:06.225+0000 7fdf171f8700 1 --2- 172.21.15.38:0/1177655990 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7fdef8037d30 0x7fdef803a1e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:38:06.663 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:06.225+0000 7fdf171f8700 1 --2- 172.21.15.38:0/1177655990 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf1007f5c0 0x7fdf1019f880 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:38:06.664 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:06.225+0000 7fdf171f8700 1 -- 172.21.15.38:0/1177655990 >> 172.21.15.38:0/1177655990 conn(0x7fdf1007ac40 msgr2=0x7fdf1007b930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:38:06.664 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:06.225+0000 7fdf171f8700 1 -- 172.21.15.38:0/1177655990 shutdown_connections 2023-03-09T14:38:06.664 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:06.226+0000 7fdf171f8700 1 -- 172.21.15.38:0/1177655990 wait complete. 2023-03-09T14:38:06.856 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:38:06 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:38:06.522+0000 7f3bab2a9200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-03-09T14:38:06.856 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:38:06 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:38:06.631+0000 7f3bab2a9200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-03-09T14:38:06.857 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:38:06 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:38:06.757+0000 7f3bab2a9200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-03-09T14:38:07.224 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:38:06 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:38:06.971+0000 7f3bab2a9200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-03-09T14:38:07.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:07 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/1177655990' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2023-03-09T14:38:07.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:07 smithi038 ceph-mon[92422]: mgrmap e9: x(active, since 18s) 2023-03-09T14:38:07.975 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:38:07 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:38:07.753+0000 7f3bab2a9200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-03-09T14:38:07.975 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:38:07 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:38:07.864+0000 7f3bab2a9200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-03-09T14:38:08.006 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout { 2023-03-09T14:38:08.006 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "epoch": 9, 2023-03-09T14:38:08.007 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "available": true, 2023-03-09T14:38:08.007 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "active_name": "x", 2023-03-09T14:38:08.007 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "num_standby": 0 2023-03-09T14:38:08.010 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout } 2023-03-09T14:38:08.010 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:07.163+0000 7fa675660700 1 Processor -- start 2023-03-09T14:38:08.011 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:07.163+0000 7fa675660700 1 -- start start 2023-03-09T14:38:08.011 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:07.164+0000 7fa675660700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa6701060f0 0x7fa6701084d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:38:08.011 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:07.164+0000 7fa675660700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa670079d50 con 0x7fa6701060f0 2023-03-09T14:38:08.011 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:07.164+0000 7fa66effd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa6701060f0 0x7fa6701084d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:38:08.011 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:07.164+0000 7fa66effd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa6701060f0 0x7fa6701084d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35578/0 (socket says 172.21.15.38:35578) 2023-03-09T14:38:08.011 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:07.164+0000 7fa66effd700 1 -- 172.21.15.38:0/102248623 learned_addr learned my addr 172.21.15.38:0/102248623 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:38:08.011 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:07.164+0000 7fa66effd700 1 -- 172.21.15.38:0/102248623 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa670079e90 con 0x7fa6701060f0 2023-03-09T14:38:08.012 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:07.164+0000 7fa66effd700 1 --2- 172.21.15.38:0/102248623 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa6701060f0 0x7fa6701084d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa6600098a0 tx=0x7fa660009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22e74b26f25113ba server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:38:08.012 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:07.165+0000 7fa66dffb700 1 -- 172.21.15.38:0/102248623 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa660004160 con 0x7fa6701060f0 2023-03-09T14:38:08.012 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:07.165+0000 7fa66dffb700 1 -- 172.21.15.38:0/102248623 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa6600042c0 con 0x7fa6701060f0 2023-03-09T14:38:08.012 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:07.165+0000 7fa66dffb700 1 -- 172.21.15.38:0/102248623 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa6600119b0 con 0x7fa6701060f0 2023-03-09T14:38:08.012 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:07.165+0000 7fa675660700 1 -- 172.21.15.38:0/102248623 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa6701060f0 msgr2=0x7fa6701084d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:38:08.012 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:07.165+0000 7fa675660700 1 --2- 172.21.15.38:0/102248623 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa6701060f0 0x7fa6701084d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa6600098a0 tx=0x7fa660009bb0 comp rx=0 tx=0).stop 2023-03-09T14:38:08.012 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:07.165+0000 7fa675660700 1 -- 172.21.15.38:0/102248623 shutdown_connections 2023-03-09T14:38:08.013 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:07.165+0000 7fa675660700 1 --2- 172.21.15.38:0/102248623 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa6701060f0 0x7fa6701084d0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:38:08.013 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:07.165+0000 7fa675660700 1 -- 172.21.15.38:0/102248623 >> 172.21.15.38:0/102248623 conn(0x7fa6700ffb10 msgr2=0x7fa670101f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:38:08.013 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:07.165+0000 7fa675660700 1 -- 172.21.15.38:0/102248623 shutdown_connections 2023-03-09T14:38:08.013 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:07.166+0000 7fa675660700 1 -- 172.21.15.38:0/102248623 wait complete. 2023-03-09T14:38:08.013 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:07.166+0000 7fa675660700 1 Processor -- start 2023-03-09T14:38:08.013 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:07.166+0000 7fa675660700 1 -- start start 2023-03-09T14:38:08.013 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:07.167+0000 7fa675660700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa6701060f0 0x7fa670199240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:38:08.013 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:07.167+0000 7fa675660700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa670199780 con 0x7fa6701060f0 2023-03-09T14:38:08.014 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:07.167+0000 7fa66effd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa6701060f0 0x7fa670199240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:38:08.014 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:07.167+0000 7fa66effd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa6701060f0 0x7fa670199240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35580/0 (socket says 172.21.15.38:35580) 2023-03-09T14:38:08.014 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:07.167+0000 7fa66effd700 1 -- 172.21.15.38:0/3175987235 learned_addr learned my addr 172.21.15.38:0/3175987235 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:38:08.014 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:07.167+0000 7fa66effd700 1 -- 172.21.15.38:0/3175987235 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa660009550 con 0x7fa6701060f0 2023-03-09T14:38:08.014 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:07.167+0000 7fa66effd700 1 --2- 172.21.15.38:0/3175987235 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa6701060f0 0x7fa670199240 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa660000c00 tx=0x7fa660011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:38:08.014 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:07.167+0000 7fa657fff700 1 -- 172.21.15.38:0/3175987235 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa6600039f0 con 0x7fa6701060f0 2023-03-09T14:38:08.014 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:07.167+0000 7fa657fff700 1 -- 172.21.15.38:0/3175987235 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa660003b50 con 0x7fa6701060f0 2023-03-09T14:38:08.015 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:07.167+0000 7fa657fff700 1 -- 172.21.15.38:0/3175987235 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa66001a780 con 0x7fa6701060f0 2023-03-09T14:38:08.015 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:07.167+0000 7fa675660700 1 -- 172.21.15.38:0/3175987235 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa670109850 con 0x7fa6701060f0 2023-03-09T14:38:08.015 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:07.167+0000 7fa675660700 1 -- 172.21.15.38:0/3175987235 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6701096c0 con 0x7fa6701060f0 2023-03-09T14:38:08.015 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:07.169+0000 7fa675660700 1 -- 172.21.15.38:0/3175987235 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6701099e0 con 0x7fa6701060f0 2023-03-09T14:38:08.015 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:07.169+0000 7fa657fff700 1 -- 172.21.15.38:0/3175987235 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 9) v1 ==== 44631+0+0 (secure 0 0 0) 0x7fa660017020 con 0x7fa6701060f0 2023-03-09T14:38:08.015 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:07.169+0000 7fa657fff700 1 --2- 172.21.15.38:0/3175987235 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7fa658037e20 0x7fa65803a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:38:08.015 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:07.169+0000 7fa657fff700 1 -- 172.21.15.38:0/3175987235 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fa66004bab0 con 0x7fa6701060f0 2023-03-09T14:38:08.015 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:07.169+0000 7fa66e7fc700 1 -- 172.21.15.38:0/3175987235 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7fa658037e20 msgr2=0x7fa65803a2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/3381625669 2023-03-09T14:38:08.016 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:07.169+0000 7fa66e7fc700 1 --2- 172.21.15.38:0/3175987235 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7fa658037e20 0x7fa65803a2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-03-09T14:38:08.016 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:07.174+0000 7fa657fff700 1 -- 172.21.15.38:0/3175987235 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7fa66001a9f0 con 0x7fa6701060f0 2023-03-09T14:38:08.016 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:07.370+0000 7fa66e7fc700 1 -- 172.21.15.38:0/3175987235 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7fa658037e20 msgr2=0x7fa65803a2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/3381625669 2023-03-09T14:38:08.016 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:07.370+0000 7fa66e7fc700 1 --2- 172.21.15.38:0/3175987235 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7fa658037e20 0x7fa65803a2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-03-09T14:38:08.017 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:07.638+0000 7fa675660700 1 -- 172.21.15.38:0/3175987235 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fa67019ce60 con 0x7fa6701060f0 2023-03-09T14:38:08.017 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:07.639+0000 7fa657fff700 1 -- 172.21.15.38:0/3175987235 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7fa66002c350 con 0x7fa6701060f0 2023-03-09T14:38:08.018 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:07.641+0000 7fa675660700 1 -- 172.21.15.38:0/3175987235 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7fa658037e20 msgr2=0x7fa65803a2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-03-09T14:38:08.018 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:07.641+0000 7fa675660700 1 --2- 172.21.15.38:0/3175987235 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7fa658037e20 0x7fa65803a2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:38:08.018 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:07.641+0000 7fa675660700 1 -- 172.21.15.38:0/3175987235 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa6701060f0 msgr2=0x7fa670199240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:38:08.018 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:07.641+0000 7fa675660700 1 --2- 172.21.15.38:0/3175987235 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa6701060f0 0x7fa670199240 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa660000c00 tx=0x7fa660011a30 comp rx=0 tx=0).stop 2023-03-09T14:38:08.018 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:07.641+0000 7fa675660700 1 -- 172.21.15.38:0/3175987235 shutdown_connections 2023-03-09T14:38:08.018 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:07.641+0000 7fa675660700 1 --2- 172.21.15.38:0/3175987235 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7fa658037e20 0x7fa65803a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:38:08.018 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:07.641+0000 7fa675660700 1 --2- 172.21.15.38:0/3175987235 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa6701060f0 0x7fa670199240 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:38:08.019 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:07.641+0000 7fa675660700 1 -- 172.21.15.38:0/3175987235 >> 172.21.15.38:0/3175987235 conn(0x7fa6700ffb10 msgr2=0x7fa6701007e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:38:08.019 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:07.641+0000 7fa675660700 1 -- 172.21.15.38:0/3175987235 shutdown_connections 2023-03-09T14:38:08.019 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:07.641+0000 7fa675660700 1 -- 172.21.15.38:0/3175987235 wait complete. 2023-03-09T14:38:08.019 INFO:teuthology.orchestra.run.smithi038.stdout:Waiting for the mgr to restart... 2023-03-09T14:38:08.019 INFO:teuthology.orchestra.run.smithi038.stdout:Waiting for mgr epoch 9... 2023-03-09T14:38:08.226 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:38:07 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:38:07.972+0000 7f3bab2a9200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-03-09T14:38:08.226 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:38:08 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:38:08.162+0000 7f3bab2a9200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-03-09T14:38:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:08 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/3175987235' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-03-09T14:38:09.000 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:38:08 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:38:08.705+0000 7f3bab2a9200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-03-09T14:38:09.750 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:38:09 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:38:09.412+0000 7f3bab2a9200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-03-09T14:38:09.750 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:38:09 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:38:09.512+0000 7f3bab2a9200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-03-09T14:38:10.750 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:38:10 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:38:10.476+0000 7f3bab2a9200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-03-09T14:38:10.750 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:38:10 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:38:10.587+0000 7f3bab2a9200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-03-09T14:38:11.250 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:38:10 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:38:10.805+0000 7f3bab2a9200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-03-09T14:38:11.592 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:38:11 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:38:11.301+0000 7f3bab2a9200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-03-09T14:38:11.593 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:38:11 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:38:11.408+0000 7f3bab2a9200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-03-09T14:38:11.921 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:38:11 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:38:11.590+0000 7f3bab2a9200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-03-09T14:38:12.250 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:38:11 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:38:11.918+0000 7f3bab2a9200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-03-09T14:38:13.153 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:38:12 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:38:12.748+0000 7f3bab2a9200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-03-09T14:38:13.153 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:38:12 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:38:12.907+0000 7f3bab2a9200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-03-09T14:38:13.500 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:38:13 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:38:13.149+0000 7f3bab2a9200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-03-09T14:38:13.500 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:38:13 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:38:13.287+0000 7f3bab2a9200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-03-09T14:38:14.000 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:38:13 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:38:13.557+0000 7f3bab2a9200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-03-09T14:38:15.000 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:38:14 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:38:14.597+0000 7f3bab2a9200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-03-09T14:38:15.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:14 smithi038 ceph-mon[92422]: Active manager daemon x restarted 2023-03-09T14:38:15.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:14 smithi038 ceph-mon[92422]: Activating manager daemon x 2023-03-09T14:38:15.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:14 smithi038 ceph-mon[92422]: osdmap e3: 0 total, 0 up, 0 in 2023-03-09T14:38:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:15 smithi038 ceph-mon[92422]: mgrmap e10: x(active, starting, since 0.053233s) 2023-03-09T14:38:16.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:15 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-03-09T14:38:16.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:15 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-03-09T14:38:16.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:15 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-03-09T14:38:16.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:15 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-03-09T14:38:16.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:15 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-03-09T14:38:16.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:15 smithi038 ceph-mon[92422]: Manager daemon x is now available 2023-03-09T14:38:16.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:15 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:38:16.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:15 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:38:16.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:15 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-03-09T14:38:16.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:15 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-03-09T14:38:16.084 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout { 2023-03-09T14:38:16.085 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2023-03-09T14:38:16.085 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout "initialized": true 2023-03-09T14:38:16.085 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout } 2023-03-09T14:38:16.085 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:08.538+0000 7f7f03861700 1 Processor -- start 2023-03-09T14:38:16.085 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:08.538+0000 7f7f03861700 1 -- start start 2023-03-09T14:38:16.086 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:08.538+0000 7f7f03861700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7efc103890 0x7f7efc103ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:38:16.086 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:08.538+0000 7f7f03861700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7efc1041e0 con 0x7f7efc103890 2023-03-09T14:38:16.086 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:08.539+0000 7f7f015fd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7efc103890 0x7f7efc103ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:38:16.086 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:08.539+0000 7f7f015fd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7efc103890 0x7f7efc103ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35588/0 (socket says 172.21.15.38:35588) 2023-03-09T14:38:16.086 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:08.539+0000 7f7f015fd700 1 -- 172.21.15.38:0/685586506 learned_addr learned my addr 172.21.15.38:0/685586506 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:38:16.086 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:08.539+0000 7f7f015fd700 1 -- 172.21.15.38:0/685586506 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7efc104320 con 0x7f7efc103890 2023-03-09T14:38:16.086 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:08.539+0000 7f7f015fd700 1 --2- 172.21.15.38:0/685586506 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7efc103890 0x7f7efc103ca0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7ef80098a0 tx=0x7f7ef8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b9e178761bf8ed6 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:38:16.086 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:08.539+0000 7f7ef3fff700 1 -- 172.21.15.38:0/685586506 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7ef8004160 con 0x7f7efc103890 2023-03-09T14:38:16.087 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:08.539+0000 7f7ef3fff700 1 -- 172.21.15.38:0/685586506 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7ef80042c0 con 0x7f7efc103890 2023-03-09T14:38:16.087 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:08.539+0000 7f7ef3fff700 1 -- 172.21.15.38:0/685586506 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7ef80119b0 con 0x7f7efc103890 2023-03-09T14:38:16.087 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:08.540+0000 7f7f03861700 1 -- 172.21.15.38:0/685586506 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7efc103890 msgr2=0x7f7efc103ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:38:16.087 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:08.540+0000 7f7f03861700 1 --2- 172.21.15.38:0/685586506 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7efc103890 0x7f7efc103ca0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7ef80098a0 tx=0x7f7ef8009bb0 comp rx=0 tx=0).stop 2023-03-09T14:38:16.087 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:08.540+0000 7f7f03861700 1 -- 172.21.15.38:0/685586506 shutdown_connections 2023-03-09T14:38:16.087 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:08.540+0000 7f7f03861700 1 --2- 172.21.15.38:0/685586506 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7efc103890 0x7f7efc103ca0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:38:16.087 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:08.540+0000 7f7f03861700 1 -- 172.21.15.38:0/685586506 >> 172.21.15.38:0/685586506 conn(0x7f7efc0ff260 msgr2=0x7f7efc101670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:38:16.088 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:08.540+0000 7f7f03861700 1 -- 172.21.15.38:0/685586506 shutdown_connections 2023-03-09T14:38:16.088 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:08.540+0000 7f7f03861700 1 -- 172.21.15.38:0/685586506 wait complete. 2023-03-09T14:38:16.088 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:08.541+0000 7f7f03861700 1 Processor -- start 2023-03-09T14:38:16.088 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:08.541+0000 7f7f03861700 1 -- start start 2023-03-09T14:38:16.088 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:08.541+0000 7f7f03861700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7efc103890 0x7f7efc19b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:38:16.088 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:08.541+0000 7f7f03861700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7efc19ba20 con 0x7f7efc103890 2023-03-09T14:38:16.088 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:08.541+0000 7f7f015fd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7efc103890 0x7f7efc19b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:38:16.088 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:08.541+0000 7f7f015fd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7efc103890 0x7f7efc19b4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35590/0 (socket says 172.21.15.38:35590) 2023-03-09T14:38:16.089 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:08.541+0000 7f7f015fd700 1 -- 172.21.15.38:0/2630807383 learned_addr learned my addr 172.21.15.38:0/2630807383 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:38:16.089 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:08.541+0000 7f7f015fd700 1 -- 172.21.15.38:0/2630807383 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ef8009550 con 0x7f7efc103890 2023-03-09T14:38:16.089 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:08.542+0000 7f7f015fd700 1 --2- 172.21.15.38:0/2630807383 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7efc103890 0x7f7efc19b4e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7ef8003e60 tx=0x7f7ef8011980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:38:16.089 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:08.542+0000 7f7ef27fc700 1 -- 172.21.15.38:0/2630807383 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7ef8003af0 con 0x7f7efc103890 2023-03-09T14:38:16.089 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:08.542+0000 7f7ef27fc700 1 -- 172.21.15.38:0/2630807383 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7ef8003c50 con 0x7f7efc103890 2023-03-09T14:38:16.089 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:08.542+0000 7f7f03861700 1 -- 172.21.15.38:0/2630807383 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7efc107690 con 0x7f7efc103890 2023-03-09T14:38:16.089 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:08.542+0000 7f7ef27fc700 1 -- 172.21.15.38:0/2630807383 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7ef8024800 con 0x7f7efc103890 2023-03-09T14:38:16.090 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:08.542+0000 7f7f03861700 1 -- 172.21.15.38:0/2630807383 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7efc107500 con 0x7f7efc103890 2023-03-09T14:38:16.090 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:08.543+0000 7f7ef27fc700 1 -- 172.21.15.38:0/2630807383 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 9) v1 ==== 44631+0+0 (secure 0 0 0) 0x7f7ef8017020 con 0x7f7efc103890 2023-03-09T14:38:16.090 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:08.543+0000 7f7ef27fc700 1 --2- 172.21.15.38:0/2630807383 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7f7ee8037d80 0x7f7ee803a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:38:16.090 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:08.543+0000 7f7ef27fc700 1 -- 172.21.15.38:0/2630807383 --> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7ee803a940 con 0x7f7ee8037d80 2023-03-09T14:38:16.090 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:08.543+0000 7f7ef27fc700 1 -- 172.21.15.38:0/2630807383 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f7ef804b770 con 0x7f7efc103890 2023-03-09T14:38:16.090 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:08.543+0000 7f7f00dfc700 1 -- 172.21.15.38:0/2630807383 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7f7ee8037d80 msgr2=0x7f7ee803a230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/3381625669 2023-03-09T14:38:16.090 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:08.544+0000 7f7f00dfc700 1 --2- 172.21.15.38:0/2630807383 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7f7ee8037d80 0x7f7ee803a230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-03-09T14:38:16.091 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:08.745+0000 7f7f00dfc700 1 -- 172.21.15.38:0/2630807383 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7f7ee8037d80 msgr2=0x7f7ee803a230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/3381625669 2023-03-09T14:38:16.091 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:08.745+0000 7f7f00dfc700 1 --2- 172.21.15.38:0/2630807383 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7f7ee8037d80 0x7f7ee803a230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-03-09T14:38:16.091 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:09.146+0000 7f7f00dfc700 1 -- 172.21.15.38:0/2630807383 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7f7ee8037d80 msgr2=0x7f7ee803a230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/3381625669 2023-03-09T14:38:16.091 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:09.146+0000 7f7f00dfc700 1 --2- 172.21.15.38:0/2630807383 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7f7ee8037d80 0x7f7ee803a230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-03-09T14:38:16.091 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:09.947+0000 7f7f00dfc700 1 -- 172.21.15.38:0/2630807383 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7f7ee8037d80 msgr2=0x7f7ee803a230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/3381625669 2023-03-09T14:38:16.091 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:09.947+0000 7f7f00dfc700 1 --2- 172.21.15.38:0/2630807383 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7f7ee8037d80 0x7f7ee803a230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-03-09T14:38:16.091 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:11.548+0000 7f7f00dfc700 1 -- 172.21.15.38:0/2630807383 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7f7ee8037d80 msgr2=0x7f7ee803a230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/3381625669 2023-03-09T14:38:16.091 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:11.548+0000 7f7f00dfc700 1 --2- 172.21.15.38:0/2630807383 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7f7ee8037d80 0x7f7ee803a230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-03-09T14:38:16.092 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:14.664+0000 7f7ef27fc700 1 -- 172.21.15.38:0/2630807383 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mgrmap(e 10) v1 ==== 44441+0+0 (secure 0 0 0) 0x7f7ef800f050 con 0x7f7efc103890 2023-03-09T14:38:16.092 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:14.664+0000 7f7ef27fc700 1 -- 172.21.15.38:0/2630807383 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7f7ee8037d80 msgr2=0x7f7ee803a230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-03-09T14:38:16.092 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:14.664+0000 7f7ef27fc700 1 --2- 172.21.15.38:0/2630807383 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7f7ee8037d80 0x7f7ee803a230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:38:16.092 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:15.669+0000 7f7ef27fc700 1 -- 172.21.15.38:0/2630807383 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mgrmap(e 11) v1 ==== 44551+0+0 (secure 0 0 0) 0x7f7ef8023d90 con 0x7f7efc103890 2023-03-09T14:38:16.092 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:15.669+0000 7f7ef27fc700 1 --2- 172.21.15.38:0/2630807383 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f7ee803b800 0x7f7ee803dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:38:16.092 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:15.669+0000 7f7ef27fc700 1 -- 172.21.15.38:0/2630807383 --> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7ee803a940 con 0x7f7ee803b800 2023-03-09T14:38:16.092 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:15.670+0000 7f7f00dfc700 1 --2- 172.21.15.38:0/2630807383 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f7ee803b800 0x7f7ee803dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:38:16.092 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:15.671+0000 7f7f00dfc700 1 --2- 172.21.15.38:0/2630807383 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f7ee803b800 0x7f7ee803dbe0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f7eec003a10 tx=0x7f7eec0092b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:38:16.093 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:15.672+0000 7f7ef27fc700 1 -- 172.21.15.38:0/2630807383 <== mgr.14150 v2:172.21.15.38:6800/3328887328 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7f7ee803a940 con 0x7f7ee803b800 2023-03-09T14:38:16.093 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:15.686+0000 7f7ee7fff700 1 -- 172.21.15.38:0/2630807383 --> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f7edc002600 con 0x7f7ee803b800 2023-03-09T14:38:16.093 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:15.686+0000 7f7ef27fc700 1 -- 172.21.15.38:0/2630807383 <== mgr.14150 v2:172.21.15.38:6800/3328887328 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f7edc002600 con 0x7f7ee803b800 2023-03-09T14:38:16.093 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:15.687+0000 7f7ee7fff700 1 -- 172.21.15.38:0/2630807383 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f7ee803b800 msgr2=0x7f7ee803dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:38:16.093 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:15.687+0000 7f7ee7fff700 1 --2- 172.21.15.38:0/2630807383 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f7ee803b800 0x7f7ee803dbe0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f7eec003a10 tx=0x7f7eec0092b0 comp rx=0 tx=0).stop 2023-03-09T14:38:16.093 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:15.687+0000 7f7ee7fff700 1 -- 172.21.15.38:0/2630807383 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7efc103890 msgr2=0x7f7efc19b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:38:16.093 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:15.687+0000 7f7ee7fff700 1 --2- 172.21.15.38:0/2630807383 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7efc103890 0x7f7efc19b4e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7ef8003e60 tx=0x7f7ef8011980 comp rx=0 tx=0).stop 2023-03-09T14:38:16.094 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:15.687+0000 7f7ee7fff700 1 -- 172.21.15.38:0/2630807383 shutdown_connections 2023-03-09T14:38:16.094 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:15.687+0000 7f7ee7fff700 1 --2- 172.21.15.38:0/2630807383 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f7ee803b800 0x7f7ee803dbe0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:38:16.094 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:15.687+0000 7f7ee7fff700 1 --2- 172.21.15.38:0/2630807383 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7efc103890 0x7f7efc19b4e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:38:16.094 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:15.687+0000 7f7ee7fff700 1 --2- 172.21.15.38:0/2630807383 >> [v2:172.21.15.38:6800/3381625669,v1:172.21.15.38:6801/3381625669] conn(0x7f7ee8037d80 0x7f7ee803a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:38:16.094 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:15.687+0000 7f7ee7fff700 1 -- 172.21.15.38:0/2630807383 >> 172.21.15.38:0/2630807383 conn(0x7f7efc0ff260 msgr2=0x7f7efc0fff00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:38:16.094 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:15.687+0000 7f7ee7fff700 1 -- 172.21.15.38:0/2630807383 shutdown_connections 2023-03-09T14:38:16.094 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:15.687+0000 7f7ee7fff700 1 -- 172.21.15.38:0/2630807383 wait complete. 2023-03-09T14:38:16.095 INFO:teuthology.orchestra.run.smithi038.stdout:mgr epoch 9 is available 2023-03-09T14:38:16.095 INFO:teuthology.orchestra.run.smithi038.stdout:Generating a dashboard self-signed certificate... 2023-03-09T14:38:17.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:16 smithi038 ceph-mon[92422]: mgrmap e11: x(active, since 1.06091s) 2023-03-09T14:38:17.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:16 smithi038 ceph-mon[92422]: [09/Mar/2023:14:38:16] ENGINE Bus STARTING 2023-03-09T14:38:17.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:16 smithi038 ceph-mon[92422]: [09/Mar/2023:14:38:16] ENGINE Serving on http://172.21.15.38:8765 2023-03-09T14:38:17.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:16 smithi038 ceph-mon[92422]: [09/Mar/2023:14:38:16] ENGINE Serving on https://172.21.15.38:7150 2023-03-09T14:38:17.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:16 smithi038 ceph-mon[92422]: [09/Mar/2023:14:38:16] ENGINE Bus STARTED 2023-03-09T14:38:17.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:16 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:38:17.530 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2023-03-09T14:38:17.530 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:16.585+0000 7f812aa30700 1 Processor -- start 2023-03-09T14:38:17.531 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:16.585+0000 7f812aa30700 1 -- start start 2023-03-09T14:38:17.531 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:16.585+0000 7f812aa30700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8124105a50 0x7f8124105e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:38:17.531 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:16.585+0000 7f812aa30700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81241063a0 con 0x7f8124105a50 2023-03-09T14:38:17.531 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:16.585+0000 7f8123fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8124105a50 0x7f8124105e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:38:17.531 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:16.585+0000 7f8123fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8124105a50 0x7f8124105e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35644/0 (socket says 172.21.15.38:35644) 2023-03-09T14:38:17.531 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:16.585+0000 7f8123fff700 1 -- 172.21.15.38:0/1772308912 learned_addr learned my addr 172.21.15.38:0/1772308912 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:38:17.531 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:16.586+0000 7f8123fff700 1 -- 172.21.15.38:0/1772308912 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81241064e0 con 0x7f8124105a50 2023-03-09T14:38:17.531 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:16.586+0000 7f8123fff700 1 --2- 172.21.15.38:0/1772308912 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8124105a50 0x7f8124105e60 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f81140098a0 tx=0x7f8114009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a5a487b513082e2 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:38:17.532 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:16.586+0000 7f8122ffd700 1 -- 172.21.15.38:0/1772308912 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8114004160 con 0x7f8124105a50 2023-03-09T14:38:17.532 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:16.586+0000 7f8122ffd700 1 -- 172.21.15.38:0/1772308912 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f81140042c0 con 0x7f8124105a50 2023-03-09T14:38:17.532 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:16.586+0000 7f8122ffd700 1 -- 172.21.15.38:0/1772308912 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f81140119b0 con 0x7f8124105a50 2023-03-09T14:38:17.532 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:16.587+0000 7f812aa30700 1 -- 172.21.15.38:0/1772308912 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8124105a50 msgr2=0x7f8124105e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:38:17.532 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:16.587+0000 7f812aa30700 1 --2- 172.21.15.38:0/1772308912 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8124105a50 0x7f8124105e60 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f81140098a0 tx=0x7f8114009bb0 comp rx=0 tx=0).stop 2023-03-09T14:38:17.532 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:16.587+0000 7f812aa30700 1 -- 172.21.15.38:0/1772308912 shutdown_connections 2023-03-09T14:38:17.532 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:16.587+0000 7f812aa30700 1 --2- 172.21.15.38:0/1772308912 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8124105a50 0x7f8124105e60 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:38:17.533 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:16.587+0000 7f812aa30700 1 -- 172.21.15.38:0/1772308912 >> 172.21.15.38:0/1772308912 conn(0x7f81240ff260 msgr2=0x7f8124101670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:38:17.533 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:16.587+0000 7f812aa30700 1 -- 172.21.15.38:0/1772308912 shutdown_connections 2023-03-09T14:38:17.533 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:16.587+0000 7f812aa30700 1 -- 172.21.15.38:0/1772308912 wait complete. 2023-03-09T14:38:17.533 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:16.587+0000 7f812aa30700 1 Processor -- start 2023-03-09T14:38:17.533 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:16.588+0000 7f812aa30700 1 -- start start 2023-03-09T14:38:17.533 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:16.588+0000 7f812aa30700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8124105a50 0x7f81241981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:38:17.533 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:16.588+0000 7f812aa30700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81241986f0 con 0x7f8124105a50 2023-03-09T14:38:17.533 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:16.588+0000 7f8123fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8124105a50 0x7f81241981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:38:17.534 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:16.588+0000 7f8123fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8124105a50 0x7f81241981b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35646/0 (socket says 172.21.15.38:35646) 2023-03-09T14:38:17.534 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:16.588+0000 7f8123fff700 1 -- 172.21.15.38:0/4085403567 learned_addr learned my addr 172.21.15.38:0/4085403567 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:38:17.534 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:16.588+0000 7f8123fff700 1 -- 172.21.15.38:0/4085403567 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8114009550 con 0x7f8124105a50 2023-03-09T14:38:17.534 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:16.589+0000 7f8123fff700 1 --2- 172.21.15.38:0/4085403567 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8124105a50 0x7f81241981b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8114000c00 tx=0x7f8114011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:38:17.534 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:16.589+0000 7f81217fa700 1 -- 172.21.15.38:0/4085403567 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f81140039f0 con 0x7f8124105a50 2023-03-09T14:38:17.534 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:16.589+0000 7f812aa30700 1 -- 172.21.15.38:0/4085403567 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8124105580 con 0x7f8124105a50 2023-03-09T14:38:17.534 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:16.589+0000 7f81217fa700 1 -- 172.21.15.38:0/4085403567 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8114003b50 con 0x7f8124105a50 2023-03-09T14:38:17.535 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:16.589+0000 7f81217fa700 1 -- 172.21.15.38:0/4085403567 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f811401a780 con 0x7f8124105a50 2023-03-09T14:38:17.535 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:16.589+0000 7f812aa30700 1 -- 172.21.15.38:0/4085403567 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81241053f0 con 0x7f8124105a50 2023-03-09T14:38:17.535 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:16.590+0000 7f81217fa700 1 -- 172.21.15.38:0/4085403567 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 11) v1 ==== 44551+0+0 (secure 0 0 0) 0x7f8114017020 con 0x7f8124105a50 2023-03-09T14:38:17.535 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:16.590+0000 7f81217fa700 1 --2- 172.21.15.38:0/4085403567 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f810c037d40 0x7f810c03a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:38:17.535 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:16.590+0000 7f81217fa700 1 -- 172.21.15.38:0/4085403567 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f811404b710 con 0x7f8124105a50 2023-03-09T14:38:17.535 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:16.590+0000 7f81237fe700 1 --2- 172.21.15.38:0/4085403567 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f810c037d40 0x7f810c03a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:38:17.535 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:16.590+0000 7f812aa30700 1 -- 172.21.15.38:0/4085403567 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8124108980 con 0x7f8124105a50 2023-03-09T14:38:17.536 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:16.590+0000 7f81237fe700 1 --2- 172.21.15.38:0/4085403567 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f810c037d40 0x7f810c03a1f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8118006fd0 tx=0x7f8118006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:38:17.536 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:16.595+0000 7f81217fa700 1 -- 172.21.15.38:0/4085403567 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f811401aa40 con 0x7f8124105a50 2023-03-09T14:38:17.536 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:16.680+0000 7f81217fa700 1 -- 172.21.15.38:0/4085403567 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mgrmap(e 12) v1 ==== 44631+0+0 (secure 0 0 0) 0x7f811400f050 con 0x7f8124105a50 2023-03-09T14:38:17.536 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:16.971+0000 7f812aa30700 1 -- 172.21.15.38:0/4085403567 --> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f8124100680 con 0x7f810c037d40 2023-03-09T14:38:17.536 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:17.146+0000 7f81217fa700 1 -- 172.21.15.38:0/4085403567 <== mgr.14150 v2:172.21.15.38:6800/3328887328 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f8124100680 con 0x7f810c037d40 2023-03-09T14:38:17.536 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:17.149+0000 7f812aa30700 1 -- 172.21.15.38:0/4085403567 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f810c037d40 msgr2=0x7f810c03a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:38:17.536 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:17.149+0000 7f812aa30700 1 --2- 172.21.15.38:0/4085403567 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f810c037d40 0x7f810c03a1f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8118006fd0 tx=0x7f8118006e40 comp rx=0 tx=0).stop 2023-03-09T14:38:17.536 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:17.149+0000 7f812aa30700 1 -- 172.21.15.38:0/4085403567 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8124105a50 msgr2=0x7f81241981b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:38:17.537 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:17.149+0000 7f812aa30700 1 --2- 172.21.15.38:0/4085403567 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8124105a50 0x7f81241981b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8114000c00 tx=0x7f8114011a30 comp rx=0 tx=0).stop 2023-03-09T14:38:17.537 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:17.149+0000 7f812aa30700 1 -- 172.21.15.38:0/4085403567 shutdown_connections 2023-03-09T14:38:17.537 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:17.149+0000 7f812aa30700 1 --2- 172.21.15.38:0/4085403567 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f810c037d40 0x7f810c03a1f0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:38:17.537 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:17.149+0000 7f812aa30700 1 --2- 172.21.15.38:0/4085403567 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8124105a50 0x7f81241981b0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:38:17.537 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:17.149+0000 7f812aa30700 1 -- 172.21.15.38:0/4085403567 >> 172.21.15.38:0/4085403567 conn(0x7f81240ff260 msgr2=0x7f81240fff70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:38:17.537 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:17.150+0000 7f812aa30700 1 -- 172.21.15.38:0/4085403567 shutdown_connections 2023-03-09T14:38:17.537 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:17.150+0000 7f812aa30700 1 -- 172.21.15.38:0/4085403567 wait complete. 2023-03-09T14:38:17.537 INFO:teuthology.orchestra.run.smithi038.stdout:Creating initial admin user... 2023-03-09T14:38:17.812 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:17 smithi038 ceph-mon[92422]: mgrmap e12: x(active, since 2s) 2023-03-09T14:38:17.813 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:17 smithi038 ceph-mon[92422]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:38:17.813 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:17 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:38:17.813 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:17 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:38:18.939 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:18 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:38:19.383 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$6PffKvWVIELf21gaBAaq9.Sj7Ywobm.ezWqH5LpZFODn7uqVP80kC", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1678372698, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2023-03-09T14:38:19.383 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:18.080+0000 7f6752a6c700 1 Processor -- start 2023-03-09T14:38:19.383 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:18.080+0000 7f6752a6c700 1 -- start start 2023-03-09T14:38:19.383 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:18.080+0000 7f6752a6c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f674c105630 0x7f674c105a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:38:19.384 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:18.080+0000 7f6752a6c700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f674c105f80 con 0x7f674c105630 2023-03-09T14:38:19.384 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:18.080+0000 7f6750808700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f674c105630 0x7f674c105a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:38:19.384 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:18.080+0000 7f6750808700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f674c105630 0x7f674c105a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35650/0 (socket says 172.21.15.38:35650) 2023-03-09T14:38:19.384 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:18.080+0000 7f6750808700 1 -- 172.21.15.38:0/1206942779 learned_addr learned my addr 172.21.15.38:0/1206942779 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:38:19.384 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:18.081+0000 7f6750808700 1 -- 172.21.15.38:0/1206942779 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f674c1060c0 con 0x7f674c105630 2023-03-09T14:38:19.384 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:18.081+0000 7f6750808700 1 --2- 172.21.15.38:0/1206942779 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f674c105630 0x7f674c105a40 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f67400098a0 tx=0x7f6740009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=473b792a0632d3ce server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:38:19.384 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:18.081+0000 7f674b7fe700 1 -- 172.21.15.38:0/1206942779 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6740004160 con 0x7f674c105630 2023-03-09T14:38:19.385 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:18.081+0000 7f674b7fe700 1 -- 172.21.15.38:0/1206942779 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f67400042c0 con 0x7f674c105630 2023-03-09T14:38:19.385 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:18.081+0000 7f674b7fe700 1 -- 172.21.15.38:0/1206942779 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f67400119b0 con 0x7f674c105630 2023-03-09T14:38:19.385 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:18.081+0000 7f6752a6c700 1 -- 172.21.15.38:0/1206942779 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f674c105630 msgr2=0x7f674c105a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:38:19.385 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:18.081+0000 7f6752a6c700 1 --2- 172.21.15.38:0/1206942779 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f674c105630 0x7f674c105a40 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f67400098a0 tx=0x7f6740009bb0 comp rx=0 tx=0).stop 2023-03-09T14:38:19.385 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:18.082+0000 7f6752a6c700 1 -- 172.21.15.38:0/1206942779 shutdown_connections 2023-03-09T14:38:19.385 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:18.082+0000 7f6752a6c700 1 --2- 172.21.15.38:0/1206942779 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f674c105630 0x7f674c105a40 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:38:19.385 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:18.082+0000 7f6752a6c700 1 -- 172.21.15.38:0/1206942779 >> 172.21.15.38:0/1206942779 conn(0x7f674c100c70 msgr2=0x7f674c103080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:38:19.386 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:18.082+0000 7f6752a6c700 1 -- 172.21.15.38:0/1206942779 shutdown_connections 2023-03-09T14:38:19.386 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:18.082+0000 7f6752a6c700 1 -- 172.21.15.38:0/1206942779 wait complete. 2023-03-09T14:38:19.386 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:18.082+0000 7f6752a6c700 1 Processor -- start 2023-03-09T14:38:19.386 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:18.082+0000 7f6752a6c700 1 -- start start 2023-03-09T14:38:19.386 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:18.082+0000 7f6752a6c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f674c105630 0x7f674c07bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:38:19.386 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:18.082+0000 7f6752a6c700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f674c07c1c0 con 0x7f674c105630 2023-03-09T14:38:19.386 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:18.082+0000 7f6750808700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f674c105630 0x7f674c07bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:38:19.386 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:18.083+0000 7f6750808700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f674c105630 0x7f674c07bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35652/0 (socket says 172.21.15.38:35652) 2023-03-09T14:38:19.387 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:18.083+0000 7f6750808700 1 -- 172.21.15.38:0/1377951062 learned_addr learned my addr 172.21.15.38:0/1377951062 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:38:19.387 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:18.083+0000 7f6750808700 1 -- 172.21.15.38:0/1377951062 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6740009550 con 0x7f674c105630 2023-03-09T14:38:19.387 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:18.083+0000 7f6750808700 1 --2- 172.21.15.38:0/1377951062 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f674c105630 0x7f674c07bc80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6740003f00 tx=0x7f6740011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:38:19.387 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:18.083+0000 7f6749ffb700 1 -- 172.21.15.38:0/1377951062 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6740003b20 con 0x7f674c105630 2023-03-09T14:38:19.387 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:18.083+0000 7f6752a6c700 1 -- 172.21.15.38:0/1377951062 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f674c107700 con 0x7f674c105630 2023-03-09T14:38:19.387 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:18.083+0000 7f6749ffb700 1 -- 172.21.15.38:0/1377951062 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6740003c80 con 0x7f674c105630 2023-03-09T14:38:19.387 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:18.083+0000 7f6749ffb700 1 -- 172.21.15.38:0/1377951062 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6740019770 con 0x7f674c105630 2023-03-09T14:38:19.388 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:18.083+0000 7f6752a6c700 1 -- 172.21.15.38:0/1377951062 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f674c107570 con 0x7f674c105630 2023-03-09T14:38:19.388 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:18.084+0000 7f6749ffb700 1 -- 172.21.15.38:0/1377951062 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 12) v1 ==== 44631+0+0 (secure 0 0 0) 0x7f674002c070 con 0x7f674c105630 2023-03-09T14:38:19.388 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:18.084+0000 7f6749ffb700 1 --2- 172.21.15.38:0/1377951062 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f673c037dd0 0x7f673c03a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:38:19.388 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:18.084+0000 7f6749ffb700 1 -- 172.21.15.38:0/1377951062 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f674004ba30 con 0x7f674c105630 2023-03-09T14:38:19.388 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:18.084+0000 7f674bfff700 1 --2- 172.21.15.38:0/1377951062 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f673c037dd0 0x7f673c03a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:38:19.388 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:18.085+0000 7f6752a6c700 1 -- 172.21.15.38:0/1377951062 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f674c061db0 con 0x7f674c105630 2023-03-09T14:38:19.388 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:18.085+0000 7f674bfff700 1 --2- 172.21.15.38:0/1377951062 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f673c037dd0 0x7f673c03a280 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6738006fd0 tx=0x7f6738006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:38:19.388 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:18.090+0000 7f6749ffb700 1 -- 172.21.15.38:0/1377951062 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f674000b3e0 con 0x7f674c105630 2023-03-09T14:38:19.389 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:18.494+0000 7f6752a6c700 1 -- 172.21.15.38:0/1377951062 --> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f674c061bd0 con 0x7f673c037dd0 2023-03-09T14:38:19.389 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:18.742+0000 7f6749ffb700 1 -- 172.21.15.38:0/1377951062 <== mgr.14150 v2:172.21.15.38:6800/3328887328 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f674c061bd0 con 0x7f673c037dd0 2023-03-09T14:38:19.389 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:18.744+0000 7f6752a6c700 1 -- 172.21.15.38:0/1377951062 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f673c037dd0 msgr2=0x7f673c03a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:38:19.389 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:18.744+0000 7f6752a6c700 1 --2- 172.21.15.38:0/1377951062 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f673c037dd0 0x7f673c03a280 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6738006fd0 tx=0x7f6738006e40 comp rx=0 tx=0).stop 2023-03-09T14:38:19.389 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:18.744+0000 7f6752a6c700 1 -- 172.21.15.38:0/1377951062 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f674c105630 msgr2=0x7f674c07bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:38:19.389 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:18.744+0000 7f6752a6c700 1 --2- 172.21.15.38:0/1377951062 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f674c105630 0x7f674c07bc80 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6740003f00 tx=0x7f6740011a30 comp rx=0 tx=0).stop 2023-03-09T14:38:19.389 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:18.744+0000 7f6752a6c700 1 -- 172.21.15.38:0/1377951062 shutdown_connections 2023-03-09T14:38:19.390 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:18.744+0000 7f6752a6c700 1 --2- 172.21.15.38:0/1377951062 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f673c037dd0 0x7f673c03a280 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:38:19.390 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:18.744+0000 7f6752a6c700 1 --2- 172.21.15.38:0/1377951062 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f674c105630 0x7f674c07bc80 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:38:19.390 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:18.744+0000 7f6752a6c700 1 -- 172.21.15.38:0/1377951062 >> 172.21.15.38:0/1377951062 conn(0x7f674c100c70 msgr2=0x7f674c1019e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:38:19.390 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:18.744+0000 7f6752a6c700 1 -- 172.21.15.38:0/1377951062 shutdown_connections 2023-03-09T14:38:19.390 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:18.744+0000 7f6752a6c700 1 -- 172.21.15.38:0/1377951062 wait complete. 2023-03-09T14:38:19.390 INFO:teuthology.orchestra.run.smithi038.stdout:Fetching dashboard port number... 2023-03-09T14:38:20.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:19 smithi038 ceph-mon[92422]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:38:20.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:19 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:38:20.528 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stdout 8443 2023-03-09T14:38:20.528 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:19.827+0000 7f5c01250700 1 Processor -- start 2023-03-09T14:38:20.528 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:19.827+0000 7f5c01250700 1 -- start start 2023-03-09T14:38:20.528 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:19.827+0000 7f5c01250700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5bfc1060d0 0x7f5bfc1084b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:38:20.528 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:19.827+0000 7f5c01250700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bfc079d70 con 0x7f5bfc1060d0 2023-03-09T14:38:20.529 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:19.828+0000 7f5bfad9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5bfc1060d0 0x7f5bfc1084b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:38:20.529 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:19.828+0000 7f5bfad9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5bfc1060d0 0x7f5bfc1084b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35656/0 (socket says 172.21.15.38:35656) 2023-03-09T14:38:20.529 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:19.828+0000 7f5bfad9d700 1 -- 172.21.15.38:0/970604031 learned_addr learned my addr 172.21.15.38:0/970604031 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:38:20.529 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:19.828+0000 7f5bfad9d700 1 -- 172.21.15.38:0/970604031 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bfc079eb0 con 0x7f5bfc1060d0 2023-03-09T14:38:20.529 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:19.828+0000 7f5bfad9d700 1 --2- 172.21.15.38:0/970604031 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5bfc1060d0 0x7f5bfc1084b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5bec0098a0 tx=0x7f5bec009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5cc2715e5070a90 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:38:20.529 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:19.828+0000 7f5bf9d9b700 1 -- 172.21.15.38:0/970604031 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5bec004160 con 0x7f5bfc1060d0 2023-03-09T14:38:20.530 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:19.828+0000 7f5bf9d9b700 1 -- 172.21.15.38:0/970604031 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5bec0042c0 con 0x7f5bfc1060d0 2023-03-09T14:38:20.530 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:19.828+0000 7f5bf9d9b700 1 -- 172.21.15.38:0/970604031 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5bec0119b0 con 0x7f5bfc1060d0 2023-03-09T14:38:20.530 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:19.829+0000 7f5c01250700 1 -- 172.21.15.38:0/970604031 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5bfc1060d0 msgr2=0x7f5bfc1084b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:38:20.530 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:19.829+0000 7f5c01250700 1 --2- 172.21.15.38:0/970604031 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5bfc1060d0 0x7f5bfc1084b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5bec0098a0 tx=0x7f5bec009bb0 comp rx=0 tx=0).stop 2023-03-09T14:38:20.530 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:19.829+0000 7f5c01250700 1 -- 172.21.15.38:0/970604031 shutdown_connections 2023-03-09T14:38:20.530 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:19.829+0000 7f5c01250700 1 --2- 172.21.15.38:0/970604031 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5bfc1060d0 0x7f5bfc1084b0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:38:20.530 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:19.829+0000 7f5c01250700 1 -- 172.21.15.38:0/970604031 >> 172.21.15.38:0/970604031 conn(0x7f5bfc0ffb10 msgr2=0x7f5bfc101f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:38:20.531 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:19.829+0000 7f5c01250700 1 -- 172.21.15.38:0/970604031 shutdown_connections 2023-03-09T14:38:20.531 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:19.829+0000 7f5c01250700 1 -- 172.21.15.38:0/970604031 wait complete. 2023-03-09T14:38:20.531 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:19.830+0000 7f5c01250700 1 Processor -- start 2023-03-09T14:38:20.531 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:19.830+0000 7f5c01250700 1 -- start start 2023-03-09T14:38:20.531 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:19.830+0000 7f5c01250700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5bfc1060d0 0x7f5bfc199240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:38:20.531 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:19.830+0000 7f5c01250700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bfc199780 con 0x7f5bfc1060d0 2023-03-09T14:38:20.531 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:19.830+0000 7f5bfad9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5bfc1060d0 0x7f5bfc199240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:38:20.532 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:19.830+0000 7f5bfad9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5bfc1060d0 0x7f5bfc199240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35658/0 (socket says 172.21.15.38:35658) 2023-03-09T14:38:20.532 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:19.830+0000 7f5bfad9d700 1 -- 172.21.15.38:0/1599196350 learned_addr learned my addr 172.21.15.38:0/1599196350 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:38:20.532 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:19.831+0000 7f5bfad9d700 1 -- 172.21.15.38:0/1599196350 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bec009550 con 0x7f5bfc1060d0 2023-03-09T14:38:20.532 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:19.831+0000 7f5bfad9d700 1 --2- 172.21.15.38:0/1599196350 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5bfc1060d0 0x7f5bfc199240 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5bec003e60 tx=0x7f5bec011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:38:20.532 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:19.831+0000 7f5be3fff700 1 -- 172.21.15.38:0/1599196350 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5bec003970 con 0x7f5bfc1060d0 2023-03-09T14:38:20.532 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:19.831+0000 7f5be3fff700 1 -- 172.21.15.38:0/1599196350 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5bec003ad0 con 0x7f5bfc1060d0 2023-03-09T14:38:20.532 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:19.831+0000 7f5c01250700 1 -- 172.21.15.38:0/1599196350 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bfc109830 con 0x7f5bfc1060d0 2023-03-09T14:38:20.532 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:19.831+0000 7f5be3fff700 1 -- 172.21.15.38:0/1599196350 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5bec0195e0 con 0x7f5bfc1060d0 2023-03-09T14:38:20.533 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:19.831+0000 7f5c01250700 1 -- 172.21.15.38:0/1599196350 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bfc1096a0 con 0x7f5bfc1060d0 2023-03-09T14:38:20.533 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:19.832+0000 7f5c01250700 1 -- 172.21.15.38:0/1599196350 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bfc1099c0 con 0x7f5bfc1060d0 2023-03-09T14:38:20.533 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:19.833+0000 7f5be3fff700 1 -- 172.21.15.38:0/1599196350 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 12) v1 ==== 44631+0+0 (secure 0 0 0) 0x7f5bec02a020 con 0x7f5bfc1060d0 2023-03-09T14:38:20.533 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:19.833+0000 7f5be3fff700 1 --2- 172.21.15.38:0/1599196350 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f5be4037dc0 0x7f5be403a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:38:20.533 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:19.833+0000 7f5be3fff700 1 -- 172.21.15.38:0/1599196350 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f5bec04bb50 con 0x7f5bfc1060d0 2023-03-09T14:38:20.533 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:19.833+0000 7f5bfa59c700 1 --2- 172.21.15.38:0/1599196350 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f5be4037dc0 0x7f5be403a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:38:20.533 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:19.834+0000 7f5bfa59c700 1 --2- 172.21.15.38:0/1599196350 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f5be4037dc0 0x7f5be403a270 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5bf0006fd0 tx=0x7f5bf0006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:38:20.534 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:19.837+0000 7f5be3fff700 1 -- 172.21.15.38:0/1599196350 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f5bec003c40 con 0x7f5bfc1060d0 2023-03-09T14:38:20.534 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:20.194+0000 7f5c01250700 1 -- 172.21.15.38:0/1599196350 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f5bfc199dc0 con 0x7f5bfc1060d0 2023-03-09T14:38:20.534 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:20.195+0000 7f5be3fff700 1 -- 172.21.15.38:0/1599196350 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f5bec04a0b0 con 0x7f5bfc1060d0 2023-03-09T14:38:20.534 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:20.197+0000 7f5c01250700 1 -- 172.21.15.38:0/1599196350 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f5be4037dc0 msgr2=0x7f5be403a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:38:20.534 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:20.197+0000 7f5c01250700 1 --2- 172.21.15.38:0/1599196350 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f5be4037dc0 0x7f5be403a270 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5bf0006fd0 tx=0x7f5bf0006e40 comp rx=0 tx=0).stop 2023-03-09T14:38:20.534 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:20.197+0000 7f5c01250700 1 -- 172.21.15.38:0/1599196350 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5bfc1060d0 msgr2=0x7f5bfc199240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:38:20.534 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:20.197+0000 7f5c01250700 1 --2- 172.21.15.38:0/1599196350 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5bfc1060d0 0x7f5bfc199240 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5bec003e60 tx=0x7f5bec011aa0 comp rx=0 tx=0).stop 2023-03-09T14:38:20.534 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:20.197+0000 7f5c01250700 1 -- 172.21.15.38:0/1599196350 shutdown_connections 2023-03-09T14:38:20.535 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:20.197+0000 7f5c01250700 1 --2- 172.21.15.38:0/1599196350 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f5be4037dc0 0x7f5be403a270 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:38:20.535 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:20.197+0000 7f5c01250700 1 --2- 172.21.15.38:0/1599196350 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5bfc1060d0 0x7f5bfc199240 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:38:20.535 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:20.197+0000 7f5c01250700 1 -- 172.21.15.38:0/1599196350 >> 172.21.15.38:0/1599196350 conn(0x7f5bfc0ffb10 msgr2=0x7f5bfc100820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:38:20.535 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:20.198+0000 7f5c01250700 1 -- 172.21.15.38:0/1599196350 shutdown_connections 2023-03-09T14:38:20.535 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:20.198+0000 7f5c01250700 1 -- 172.21.15.38:0/1599196350 wait complete. 2023-03-09T14:38:20.536 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-03-09T14:38:20.537 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout disabled 2023-03-09T14:38:20.544 INFO:teuthology.orchestra.run.smithi038.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-03-09T14:38:20.544 INFO:teuthology.orchestra.run.smithi038.stdout:systemctl: stdout inactive 2023-03-09T14:38:20.544 INFO:teuthology.orchestra.run.smithi038.stdout:firewalld.service is not enabled 2023-03-09T14:38:20.544 INFO:teuthology.orchestra.run.smithi038.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2023-03-09T14:38:20.546 INFO:teuthology.orchestra.run.smithi038.stdout:Ceph Dashboard is now available at: 2023-03-09T14:38:20.546 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:38:20.546 INFO:teuthology.orchestra.run.smithi038.stdout: URL: https://smithi038.front.sepia.ceph.com:8443/ 2023-03-09T14:38:20.546 INFO:teuthology.orchestra.run.smithi038.stdout: User: admin 2023-03-09T14:38:20.546 INFO:teuthology.orchestra.run.smithi038.stdout: Password: 9n9ksei1wc 2023-03-09T14:38:20.546 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:38:20.547 INFO:teuthology.orchestra.run.smithi038.stdout:Saving cluster configuration to /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/config directory 2023-03-09T14:38:20.547 INFO:teuthology.orchestra.run.smithi038.stdout:Enabling autotune for osd_memory_target 2023-03-09T14:38:20.903 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:20 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/1599196350' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2023-03-09T14:38:21.895 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:21.170+0000 7f3135f6f700 1 Processor -- start 2023-03-09T14:38:21.895 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:21.170+0000 7f3135f6f700 1 -- start start 2023-03-09T14:38:21.896 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:21.170+0000 7f3135f6f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3130106ba0 0x7f3130106fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:38:21.896 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:21.170+0000 7f3135f6f700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31301074f0 con 0x7f3130106ba0 2023-03-09T14:38:21.896 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:21.171+0000 7f312f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3130106ba0 0x7f3130106fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:38:21.896 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:21.171+0000 7f312f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3130106ba0 0x7f3130106fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35666/0 (socket says 172.21.15.38:35666) 2023-03-09T14:38:21.896 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:21.171+0000 7f312f7fe700 1 -- 172.21.15.38:0/2307099251 learned_addr learned my addr 172.21.15.38:0/2307099251 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:38:21.896 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:21.171+0000 7f312f7fe700 1 -- 172.21.15.38:0/2307099251 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3130107630 con 0x7f3130106ba0 2023-03-09T14:38:21.896 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:21.171+0000 7f312f7fe700 1 --2- 172.21.15.38:0/2307099251 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3130106ba0 0x7f3130106fb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f31240098a0 tx=0x7f3124009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68e902690f982135 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:38:21.897 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:21.171+0000 7f312e7fc700 1 -- 172.21.15.38:0/2307099251 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3124004160 con 0x7f3130106ba0 2023-03-09T14:38:21.897 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:21.171+0000 7f312e7fc700 1 -- 172.21.15.38:0/2307099251 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f31240042c0 con 0x7f3130106ba0 2023-03-09T14:38:21.897 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:21.171+0000 7f312e7fc700 1 -- 172.21.15.38:0/2307099251 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f31240119b0 con 0x7f3130106ba0 2023-03-09T14:38:21.897 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:21.172+0000 7f3135f6f700 1 -- 172.21.15.38:0/2307099251 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3130106ba0 msgr2=0x7f3130106fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:38:21.897 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:21.172+0000 7f3135f6f700 1 --2- 172.21.15.38:0/2307099251 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3130106ba0 0x7f3130106fb0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f31240098a0 tx=0x7f3124009bb0 comp rx=0 tx=0).stop 2023-03-09T14:38:21.897 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:21.172+0000 7f3135f6f700 1 -- 172.21.15.38:0/2307099251 shutdown_connections 2023-03-09T14:38:21.898 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:21.172+0000 7f3135f6f700 1 --2- 172.21.15.38:0/2307099251 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3130106ba0 0x7f3130106fb0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:38:21.898 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:21.172+0000 7f3135f6f700 1 -- 172.21.15.38:0/2307099251 >> 172.21.15.38:0/2307099251 conn(0x7f31300ff260 msgr2=0x7f3130101670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:38:21.898 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:21.172+0000 7f3135f6f700 1 -- 172.21.15.38:0/2307099251 shutdown_connections 2023-03-09T14:38:21.898 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:21.172+0000 7f3135f6f700 1 -- 172.21.15.38:0/2307099251 wait complete. 2023-03-09T14:38:21.898 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:21.173+0000 7f3135f6f700 1 Processor -- start 2023-03-09T14:38:21.898 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:21.173+0000 7f3135f6f700 1 -- start start 2023-03-09T14:38:21.898 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:21.173+0000 7f3135f6f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3130106ba0 0x7f31301062a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:38:21.898 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:21.173+0000 7f3135f6f700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31301067e0 con 0x7f3130106ba0 2023-03-09T14:38:21.899 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:21.173+0000 7f312f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3130106ba0 0x7f31301062a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:38:21.899 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:21.173+0000 7f312f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3130106ba0 0x7f31301062a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35668/0 (socket says 172.21.15.38:35668) 2023-03-09T14:38:21.899 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:21.173+0000 7f312f7fe700 1 -- 172.21.15.38:0/2846318453 learned_addr learned my addr 172.21.15.38:0/2846318453 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:38:21.899 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:21.173+0000 7f312f7fe700 1 -- 172.21.15.38:0/2846318453 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3124009550 con 0x7f3130106ba0 2023-03-09T14:38:21.899 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:21.174+0000 7f312f7fe700 1 --2- 172.21.15.38:0/2846318453 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3130106ba0 0x7f31301062a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3124000c00 tx=0x7f3124011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:38:21.899 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:21.174+0000 7f312cff9700 1 -- 172.21.15.38:0/2846318453 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f31240039f0 con 0x7f3130106ba0 2023-03-09T14:38:21.899 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:21.174+0000 7f312cff9700 1 -- 172.21.15.38:0/2846318453 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3124003b50 con 0x7f3130106ba0 2023-03-09T14:38:21.900 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:21.174+0000 7f3135f6f700 1 -- 172.21.15.38:0/2846318453 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3130104330 con 0x7f3130106ba0 2023-03-09T14:38:21.900 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:21.174+0000 7f312cff9700 1 -- 172.21.15.38:0/2846318453 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f312401a780 con 0x7f3130106ba0 2023-03-09T14:38:21.900 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:21.174+0000 7f3135f6f700 1 -- 172.21.15.38:0/2846318453 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31301041a0 con 0x7f3130106ba0 2023-03-09T14:38:21.900 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:21.175+0000 7f312cff9700 1 -- 172.21.15.38:0/2846318453 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 44674+0+0 (secure 0 0 0) 0x7f3124017020 con 0x7f3130106ba0 2023-03-09T14:38:21.900 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:21.175+0000 7f312cff9700 1 --2- 172.21.15.38:0/2846318453 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f3118037ec0 0x7f311803a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:38:21.900 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:21.175+0000 7f312cff9700 1 -- 172.21.15.38:0/2846318453 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f312404b7d0 con 0x7f3130106ba0 2023-03-09T14:38:21.900 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:21.175+0000 7f312effd700 1 --2- 172.21.15.38:0/2846318453 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f3118037ec0 0x7f311803a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:38:21.900 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:21.175+0000 7f3135f6f700 1 -- 172.21.15.38:0/2846318453 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3130061db0 con 0x7f3130106ba0 2023-03-09T14:38:21.901 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:21.176+0000 7f312effd700 1 --2- 172.21.15.38:0/2846318453 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f3118037ec0 0x7f311803a370 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3120006fd0 tx=0x7f3120006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:38:21.901 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:21.180+0000 7f312cff9700 1 -- 172.21.15.38:0/2846318453 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f312401a9f0 con 0x7f3130106ba0 2023-03-09T14:38:21.901 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:21.534+0000 7f3135f6f700 1 -- 172.21.15.38:0/2846318453 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f3130105760 con 0x7f3130106ba0 2023-03-09T14:38:21.901 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:21.535+0000 7f312cff9700 1 -- 172.21.15.38:0/2846318453 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f3124018350 con 0x7f3130106ba0 2023-03-09T14:38:21.901 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:21.537+0000 7f3135f6f700 1 -- 172.21.15.38:0/2846318453 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f3118037ec0 msgr2=0x7f311803a370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:38:21.901 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:21.537+0000 7f3135f6f700 1 --2- 172.21.15.38:0/2846318453 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f3118037ec0 0x7f311803a370 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3120006fd0 tx=0x7f3120006e40 comp rx=0 tx=0).stop 2023-03-09T14:38:21.902 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:21.537+0000 7f3135f6f700 1 -- 172.21.15.38:0/2846318453 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3130106ba0 msgr2=0x7f31301062a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:38:21.902 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:21.537+0000 7f3135f6f700 1 --2- 172.21.15.38:0/2846318453 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3130106ba0 0x7f31301062a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3124000c00 tx=0x7f3124011a30 comp rx=0 tx=0).stop 2023-03-09T14:38:21.902 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:21.537+0000 7f3135f6f700 1 -- 172.21.15.38:0/2846318453 shutdown_connections 2023-03-09T14:38:21.902 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:21.537+0000 7f3135f6f700 1 --2- 172.21.15.38:0/2846318453 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f3118037ec0 0x7f311803a370 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:38:21.902 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:21.537+0000 7f3135f6f700 1 --2- 172.21.15.38:0/2846318453 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3130106ba0 0x7f31301062a0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:38:21.902 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:21.537+0000 7f3135f6f700 1 -- 172.21.15.38:0/2846318453 >> 172.21.15.38:0/2846318453 conn(0x7f31300ff260 msgr2=0x7f31300fff90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:38:21.902 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:21.537+0000 7f3135f6f700 1 -- 172.21.15.38:0/2846318453 shutdown_connections 2023-03-09T14:38:21.902 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:21.537+0000 7f3135f6f700 1 -- 172.21.15.38:0/2846318453 wait complete. 2023-03-09T14:38:21.903 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:21 smithi038 ceph-mon[92422]: mgrmap e13: x(active, since 6s) 2023-03-09T14:38:23.340 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:22.340+0000 7fa0926b7700 1 Processor -- start 2023-03-09T14:38:23.344 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:22.340+0000 7fa0926b7700 1 -- start start 2023-03-09T14:38:23.345 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:22.340+0000 7fa0926b7700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa08c07be30 0x7fa08c07a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:38:23.345 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:22.340+0000 7fa0926b7700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa08c07a860 con 0x7fa08c07be30 2023-03-09T14:38:23.345 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:22.341+0000 7fa08bfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa08c07be30 0x7fa08c07a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:38:23.345 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:22.341+0000 7fa08bfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa08c07be30 0x7fa08c07a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35672/0 (socket says 172.21.15.38:35672) 2023-03-09T14:38:23.345 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:22.341+0000 7fa08bfff700 1 -- 172.21.15.38:0/3375158188 learned_addr learned my addr 172.21.15.38:0/3375158188 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:38:23.345 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:22.341+0000 7fa08bfff700 1 -- 172.21.15.38:0/3375158188 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa08c07a9a0 con 0x7fa08c07be30 2023-03-09T14:38:23.345 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:22.341+0000 7fa08bfff700 1 --2- 172.21.15.38:0/3375158188 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa08c07be30 0x7fa08c07a320 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa07c0098a0 tx=0x7fa07c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58bb2d29ee38edc1 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:38:23.346 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:22.341+0000 7fa08b7fe700 1 -- 172.21.15.38:0/3375158188 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa07c004160 con 0x7fa08c07be30 2023-03-09T14:38:23.346 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:22.341+0000 7fa08b7fe700 1 -- 172.21.15.38:0/3375158188 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa07c0042c0 con 0x7fa08c07be30 2023-03-09T14:38:23.346 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:22.341+0000 7fa08b7fe700 1 -- 172.21.15.38:0/3375158188 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa07c0119b0 con 0x7fa08c07be30 2023-03-09T14:38:23.346 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:22.342+0000 7fa0926b7700 1 -- 172.21.15.38:0/3375158188 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa08c07be30 msgr2=0x7fa08c07a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:38:23.346 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:22.342+0000 7fa0926b7700 1 --2- 172.21.15.38:0/3375158188 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa08c07be30 0x7fa08c07a320 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa07c0098a0 tx=0x7fa07c009bb0 comp rx=0 tx=0).stop 2023-03-09T14:38:23.346 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:22.342+0000 7fa0926b7700 1 -- 172.21.15.38:0/3375158188 shutdown_connections 2023-03-09T14:38:23.346 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:22.342+0000 7fa0926b7700 1 --2- 172.21.15.38:0/3375158188 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa08c07be30 0x7fa08c07a320 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:38:23.347 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:22.342+0000 7fa0926b7700 1 -- 172.21.15.38:0/3375158188 >> 172.21.15.38:0/3375158188 conn(0x7fa08c100c70 msgr2=0x7fa08c103080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:38:23.347 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:22.342+0000 7fa0926b7700 1 -- 172.21.15.38:0/3375158188 shutdown_connections 2023-03-09T14:38:23.347 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:22.342+0000 7fa0926b7700 1 -- 172.21.15.38:0/3375158188 wait complete. 2023-03-09T14:38:23.347 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:22.343+0000 7fa0926b7700 1 Processor -- start 2023-03-09T14:38:23.347 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:22.343+0000 7fa0926b7700 1 -- start start 2023-03-09T14:38:23.347 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:22.343+0000 7fa0926b7700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa08c07be30 0x7fa08c19b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:38:23.347 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:22.343+0000 7fa0926b7700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa08c19ba80 con 0x7fa08c07be30 2023-03-09T14:38:23.347 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:22.343+0000 7fa08bfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa08c07be30 0x7fa08c19b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:38:23.348 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:22.343+0000 7fa08bfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa08c07be30 0x7fa08c19b540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35674/0 (socket says 172.21.15.38:35674) 2023-03-09T14:38:23.348 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:22.343+0000 7fa08bfff700 1 -- 172.21.15.38:0/691812937 learned_addr learned my addr 172.21.15.38:0/691812937 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:38:23.348 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:22.343+0000 7fa08bfff700 1 -- 172.21.15.38:0/691812937 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa07c009550 con 0x7fa08c07be30 2023-03-09T14:38:23.348 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:22.343+0000 7fa08bfff700 1 --2- 172.21.15.38:0/691812937 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa08c07be30 0x7fa08c19b540 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa07c006af0 tx=0x7fa07c011a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:38:23.348 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:22.344+0000 7fa089ffb700 1 -- 172.21.15.38:0/691812937 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa07c003a40 con 0x7fa08c07be30 2023-03-09T14:38:23.348 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:22.344+0000 7fa089ffb700 1 -- 172.21.15.38:0/691812937 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa07c003ba0 con 0x7fa08c07be30 2023-03-09T14:38:23.348 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:22.344+0000 7fa089ffb700 1 -- 172.21.15.38:0/691812937 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa07c019690 con 0x7fa08c07be30 2023-03-09T14:38:23.349 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:22.344+0000 7fa0926b7700 1 -- 172.21.15.38:0/691812937 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa08c10bb40 con 0x7fa08c07be30 2023-03-09T14:38:23.349 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:22.344+0000 7fa0926b7700 1 -- 172.21.15.38:0/691812937 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa08c10b9b0 con 0x7fa08c07be30 2023-03-09T14:38:23.349 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:22.345+0000 7fa089ffb700 1 -- 172.21.15.38:0/691812937 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 44674+0+0 (secure 0 0 0) 0x7fa07c02a020 con 0x7fa08c07be30 2023-03-09T14:38:23.349 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:22.345+0000 7fa089ffb700 1 --2- 172.21.15.38:0/691812937 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fa078037e60 0x7fa07803a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:38:23.349 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:22.345+0000 7fa089ffb700 1 -- 172.21.15.38:0/691812937 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fa07c04ba70 con 0x7fa08c07be30 2023-03-09T14:38:23.349 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:22.345+0000 7fa0926b7700 1 -- 172.21.15.38:0/691812937 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa08c10bcd0 con 0x7fa08c07be30 2023-03-09T14:38:23.349 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:22.345+0000 7fa083fff700 1 --2- 172.21.15.38:0/691812937 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fa078037e60 0x7fa07803a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:38:23.350 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:22.345+0000 7fa083fff700 1 --2- 172.21.15.38:0/691812937 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fa078037e60 0x7fa07803a310 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa074006fd0 tx=0x7fa074006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:38:23.350 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:22.350+0000 7fa089ffb700 1 -- 172.21.15.38:0/691812937 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7fa07c01ac40 con 0x7fa08c07be30 2023-03-09T14:38:23.350 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:22.919+0000 7fa0926b7700 1 -- 172.21.15.38:0/691812937 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7fa08c061db0 con 0x7fa08c07be30 2023-03-09T14:38:23.350 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:22.922+0000 7fa089ffb700 1 -- 172.21.15.38:0/691812937 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) v1 ==== 153+0+0 (secure 0 0 0) 0x7fa07c049020 con 0x7fa08c07be30 2023-03-09T14:38:23.350 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:22.925+0000 7fa0926b7700 1 -- 172.21.15.38:0/691812937 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fa078037e60 msgr2=0x7fa07803a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:38:23.350 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:22.925+0000 7fa0926b7700 1 --2- 172.21.15.38:0/691812937 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fa078037e60 0x7fa07803a310 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa074006fd0 tx=0x7fa074006e40 comp rx=0 tx=0).stop 2023-03-09T14:38:23.350 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:22.925+0000 7fa0926b7700 1 -- 172.21.15.38:0/691812937 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa08c07be30 msgr2=0x7fa08c19b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:38:23.350 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:22.925+0000 7fa0926b7700 1 --2- 172.21.15.38:0/691812937 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa08c07be30 0x7fa08c19b540 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa07c006af0 tx=0x7fa07c011a10 comp rx=0 tx=0).stop 2023-03-09T14:38:23.351 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:22.925+0000 7fa0926b7700 1 -- 172.21.15.38:0/691812937 shutdown_connections 2023-03-09T14:38:23.351 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:22.925+0000 7fa0926b7700 1 --2- 172.21.15.38:0/691812937 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fa078037e60 0x7fa07803a310 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:38:23.351 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:22.925+0000 7fa0926b7700 1 --2- 172.21.15.38:0/691812937 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa08c07be30 0x7fa08c19b540 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:38:23.351 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:22.925+0000 7fa0926b7700 1 -- 172.21.15.38:0/691812937 >> 172.21.15.38:0/691812937 conn(0x7fa08c100c70 msgr2=0x7fa08c102310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:38:23.351 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:22.925+0000 7fa0926b7700 1 -- 172.21.15.38:0/691812937 shutdown_connections 2023-03-09T14:38:23.351 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr 2023-03-09T14:38:22.925+0000 7fa0926b7700 1 -- 172.21.15.38:0/691812937 wait complete. 2023-03-09T14:38:23.351 INFO:teuthology.orchestra.run.smithi038.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2023-03-09T14:38:23.352 INFO:teuthology.orchestra.run.smithi038.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2023-03-09T14:38:23.352 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:38:23.352 INFO:teuthology.orchestra.run.smithi038.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2023-03-09T14:38:23.352 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:38:23.352 INFO:teuthology.orchestra.run.smithi038.stdout:Or, if you are only running a single cluster on this host: 2023-03-09T14:38:23.352 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:38:23.352 INFO:teuthology.orchestra.run.smithi038.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2023-03-09T14:38:23.352 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:38:23.353 INFO:teuthology.orchestra.run.smithi038.stdout:Please consider enabling telemetry to help improve Ceph: 2023-03-09T14:38:23.353 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:38:23.353 INFO:teuthology.orchestra.run.smithi038.stdout: ceph telemetry on 2023-03-09T14:38:23.353 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:38:23.353 INFO:teuthology.orchestra.run.smithi038.stdout:For more information see: 2023-03-09T14:38:23.353 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:38:23.353 INFO:teuthology.orchestra.run.smithi038.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2023-03-09T14:38:23.354 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:38:23.354 INFO:teuthology.orchestra.run.smithi038.stdout:Bootstrap complete. 2023-03-09T14:38:23.381 INFO:tasks.cephadm:Fetching config... 2023-03-09T14:38:23.381 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2023-03-09T14:38:23.382 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2023-03-09T14:38:23.399 INFO:tasks.cephadm:Fetching client.admin keyring... 2023-03-09T14:38:23.399 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2023-03-09T14:38:23.400 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2023-03-09T14:38:23.458 INFO:tasks.cephadm:Fetching mon keyring... 2023-03-09T14:38:23.459 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2023-03-09T14:38:23.459 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.a/keyring of=/dev/stdout 2023-03-09T14:38:23.526 INFO:tasks.cephadm:Fetching pub ssh key... 2023-03-09T14:38:23.526 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2023-03-09T14:38:23.527 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2023-03-09T14:38:23.584 INFO:tasks.cephadm:Installing pub ssh key for root users... 2023-03-09T14:38:23.585 DEBUG:teuthology.orchestra.run.smithi038:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDhsbGemUhVpwqlPHk2x7Z0y9Uw1fxRhFUHkRkV82fljmtiu5YJpaUJei76FM2/wSE1Vanlqr9PFER836Go7lQ7iYo6RWyRCrQbZhM1FxoCEmCMFXhannKj19p+bCozso8GjpU4rZ9O3+usHVpl2jLgy8ZvqeV3J6J/SSztgvekyW7Vzzu1prKGvavOHnn6jAeWk3FRG9Q7ouKki+8O+olx0hXGoXcaCM42EavDZPYxXdKECgDJgqg89Uhp40vJOZttoliDajkUXZ7nS5rLNJPco/Gic/HKz4y6uC2uhtmbcUpznPUtuEGqQSpLiisf8hYwrSotoR3aN7Wl9JKIpFg/wq/7B46het2zCvT6sb1IYR86so2J5haBhDDnDrmEWwbj6KDlf2j1Hu7GOvyvB8Tb2aG7T2Vgnn7vgltDPX7WnxWkAREuEf1virBG6mZuInQWXsWmGvr68SjsWFuJRQZwYyQPeLOVTRRORe0Z0FXr8ThBD82D2p5qxCbKLJhx4Cs= ceph-c6733fe2-be87-11ed-9af2-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-03-09T14:38:23.667 INFO:teuthology.orchestra.run.smithi038.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDhsbGemUhVpwqlPHk2x7Z0y9Uw1fxRhFUHkRkV82fljmtiu5YJpaUJei76FM2/wSE1Vanlqr9PFER836Go7lQ7iYo6RWyRCrQbZhM1FxoCEmCMFXhannKj19p+bCozso8GjpU4rZ9O3+usHVpl2jLgy8ZvqeV3J6J/SSztgvekyW7Vzzu1prKGvavOHnn6jAeWk3FRG9Q7ouKki+8O+olx0hXGoXcaCM42EavDZPYxXdKECgDJgqg89Uhp40vJOZttoliDajkUXZ7nS5rLNJPco/Gic/HKz4y6uC2uhtmbcUpznPUtuEGqQSpLiisf8hYwrSotoR3aN7Wl9JKIpFg/wq/7B46het2zCvT6sb1IYR86so2J5haBhDDnDrmEWwbj6KDlf2j1Hu7GOvyvB8Tb2aG7T2Vgnn7vgltDPX7WnxWkAREuEf1virBG6mZuInQWXsWmGvr68SjsWFuJRQZwYyQPeLOVTRRORe0Z0FXr8ThBD82D2p5qxCbKLJhx4Cs= ceph-c6733fe2-be87-11ed-9af2-001a4aab830c 2023-03-09T14:38:23.681 DEBUG:teuthology.orchestra.run.smithi122:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDhsbGemUhVpwqlPHk2x7Z0y9Uw1fxRhFUHkRkV82fljmtiu5YJpaUJei76FM2/wSE1Vanlqr9PFER836Go7lQ7iYo6RWyRCrQbZhM1FxoCEmCMFXhannKj19p+bCozso8GjpU4rZ9O3+usHVpl2jLgy8ZvqeV3J6J/SSztgvekyW7Vzzu1prKGvavOHnn6jAeWk3FRG9Q7ouKki+8O+olx0hXGoXcaCM42EavDZPYxXdKECgDJgqg89Uhp40vJOZttoliDajkUXZ7nS5rLNJPco/Gic/HKz4y6uC2uhtmbcUpznPUtuEGqQSpLiisf8hYwrSotoR3aN7Wl9JKIpFg/wq/7B46het2zCvT6sb1IYR86so2J5haBhDDnDrmEWwbj6KDlf2j1Hu7GOvyvB8Tb2aG7T2Vgnn7vgltDPX7WnxWkAREuEf1virBG6mZuInQWXsWmGvr68SjsWFuJRQZwYyQPeLOVTRRORe0Z0FXr8ThBD82D2p5qxCbKLJhx4Cs= ceph-c6733fe2-be87-11ed-9af2-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-03-09T14:38:23.727 INFO:teuthology.orchestra.run.smithi122.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDhsbGemUhVpwqlPHk2x7Z0y9Uw1fxRhFUHkRkV82fljmtiu5YJpaUJei76FM2/wSE1Vanlqr9PFER836Go7lQ7iYo6RWyRCrQbZhM1FxoCEmCMFXhannKj19p+bCozso8GjpU4rZ9O3+usHVpl2jLgy8ZvqeV3J6J/SSztgvekyW7Vzzu1prKGvavOHnn6jAeWk3FRG9Q7ouKki+8O+olx0hXGoXcaCM42EavDZPYxXdKECgDJgqg89Uhp40vJOZttoliDajkUXZ7nS5rLNJPco/Gic/HKz4y6uC2uhtmbcUpznPUtuEGqQSpLiisf8hYwrSotoR3aN7Wl9JKIpFg/wq/7B46het2zCvT6sb1IYR86so2J5haBhDDnDrmEWwbj6KDlf2j1Hu7GOvyvB8Tb2aG7T2Vgnn7vgltDPX7WnxWkAREuEf1virBG6mZuInQWXsWmGvr68SjsWFuJRQZwYyQPeLOVTRRORe0Z0FXr8ThBD82D2p5qxCbKLJhx4Cs= ceph-c6733fe2-be87-11ed-9af2-001a4aab830c 2023-03-09T14:38:23.741 DEBUG:teuthology.orchestra.run.smithi181:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDhsbGemUhVpwqlPHk2x7Z0y9Uw1fxRhFUHkRkV82fljmtiu5YJpaUJei76FM2/wSE1Vanlqr9PFER836Go7lQ7iYo6RWyRCrQbZhM1FxoCEmCMFXhannKj19p+bCozso8GjpU4rZ9O3+usHVpl2jLgy8ZvqeV3J6J/SSztgvekyW7Vzzu1prKGvavOHnn6jAeWk3FRG9Q7ouKki+8O+olx0hXGoXcaCM42EavDZPYxXdKECgDJgqg89Uhp40vJOZttoliDajkUXZ7nS5rLNJPco/Gic/HKz4y6uC2uhtmbcUpznPUtuEGqQSpLiisf8hYwrSotoR3aN7Wl9JKIpFg/wq/7B46het2zCvT6sb1IYR86so2J5haBhDDnDrmEWwbj6KDlf2j1Hu7GOvyvB8Tb2aG7T2Vgnn7vgltDPX7WnxWkAREuEf1virBG6mZuInQWXsWmGvr68SjsWFuJRQZwYyQPeLOVTRRORe0Z0FXr8ThBD82D2p5qxCbKLJhx4Cs= ceph-c6733fe2-be87-11ed-9af2-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-03-09T14:38:23.786 INFO:teuthology.orchestra.run.smithi181.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDhsbGemUhVpwqlPHk2x7Z0y9Uw1fxRhFUHkRkV82fljmtiu5YJpaUJei76FM2/wSE1Vanlqr9PFER836Go7lQ7iYo6RWyRCrQbZhM1FxoCEmCMFXhannKj19p+bCozso8GjpU4rZ9O3+usHVpl2jLgy8ZvqeV3J6J/SSztgvekyW7Vzzu1prKGvavOHnn6jAeWk3FRG9Q7ouKki+8O+olx0hXGoXcaCM42EavDZPYxXdKECgDJgqg89Uhp40vJOZttoliDajkUXZ7nS5rLNJPco/Gic/HKz4y6uC2uhtmbcUpznPUtuEGqQSpLiisf8hYwrSotoR3aN7Wl9JKIpFg/wq/7B46het2zCvT6sb1IYR86so2J5haBhDDnDrmEWwbj6KDlf2j1Hu7GOvyvB8Tb2aG7T2Vgnn7vgltDPX7WnxWkAREuEf1virBG6mZuInQWXsWmGvr68SjsWFuJRQZwYyQPeLOVTRRORe0Z0FXr8ThBD82D2p5qxCbKLJhx4Cs= ceph-c6733fe2-be87-11ed-9af2-001a4aab830c 2023-03-09T14:38:23.799 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph config set mgr mgr/cephadm/allow_ptrace true 2023-03-09T14:38:24.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:23 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/691812937' entity='client.admin' 2023-03-09T14:38:24.660 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.a/config 2023-03-09T14:38:26.331 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:26.328+0000 7f00e21fc700 1 -- 172.21.15.38:0/29974608 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00dc0fe8d0 msgr2=0x7f00dc0fece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:38:26.332 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:26.328+0000 7f00e21fc700 1 --2- 172.21.15.38:0/29974608 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00dc0fe8d0 0x7f00dc0fece0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f00d0009910 tx=0x7f00d0009c20 comp rx=0 tx=0).stop 2023-03-09T14:38:26.332 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:26.328+0000 7f00e21fc700 1 -- 172.21.15.38:0/29974608 shutdown_connections 2023-03-09T14:38:26.332 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:26.328+0000 7f00e21fc700 1 --2- 172.21.15.38:0/29974608 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00dc0fe8d0 0x7f00dc0fece0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:38:26.332 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:26.328+0000 7f00e21fc700 1 -- 172.21.15.38:0/29974608 >> 172.21.15.38:0/29974608 conn(0x7f00dc0f9f10 msgr2=0x7f00dc0fc320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:38:26.332 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:26.328+0000 7f00e21fc700 1 -- 172.21.15.38:0/29974608 shutdown_connections 2023-03-09T14:38:26.333 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:26.328+0000 7f00e21fc700 1 -- 172.21.15.38:0/29974608 wait complete. 2023-03-09T14:38:26.334 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:26.329+0000 7f00e21fc700 1 Processor -- start 2023-03-09T14:38:26.334 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:26.329+0000 7f00e21fc700 1 -- start start 2023-03-09T14:38:26.334 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:26.329+0000 7f00e21fc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00dc0fe8d0 0x7f00dc1939f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:38:26.335 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:26.329+0000 7f00e21fc700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00dc193f30 con 0x7f00dc0fe8d0 2023-03-09T14:38:26.335 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:26.329+0000 7f00db7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00dc0fe8d0 0x7f00dc1939f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:38:26.335 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:26.329+0000 7f00db7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00dc0fe8d0 0x7f00dc1939f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35680/0 (socket says 172.21.15.38:35680) 2023-03-09T14:38:26.335 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:26.329+0000 7f00db7fe700 1 -- 172.21.15.38:0/3869503301 learned_addr learned my addr 172.21.15.38:0/3869503301 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:38:26.335 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:26.329+0000 7f00db7fe700 1 -- 172.21.15.38:0/3869503301 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00d00095f0 con 0x7f00dc0fe8d0 2023-03-09T14:38:26.335 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:26.329+0000 7f00db7fe700 1 --2- 172.21.15.38:0/3869503301 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00dc0fe8d0 0x7f00dc1939f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f00d0004f40 tx=0x7f00d00057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:38:26.336 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:26.330+0000 7f00d8ff9700 1 -- 172.21.15.38:0/3869503301 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f00d001d070 con 0x7f00dc0fe8d0 2023-03-09T14:38:26.336 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:26.330+0000 7f00e21fc700 1 -- 172.21.15.38:0/3869503301 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00dc104720 con 0x7f00dc0fe8d0 2023-03-09T14:38:26.336 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:26.330+0000 7f00d8ff9700 1 -- 172.21.15.38:0/3869503301 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f00d00044e0 con 0x7f00dc0fe8d0 2023-03-09T14:38:26.336 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:26.330+0000 7f00d8ff9700 1 -- 172.21.15.38:0/3869503301 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f00d0003a20 con 0x7f00dc0fe8d0 2023-03-09T14:38:26.336 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:26.330+0000 7f00e21fc700 1 -- 172.21.15.38:0/3869503301 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00dc104590 con 0x7f00dc0fe8d0 2023-03-09T14:38:26.336 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:26.331+0000 7f00d8ff9700 1 -- 172.21.15.38:0/3869503301 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 44674+0+0 (secure 0 0 0) 0x7f00d000c040 con 0x7f00dc0fe8d0 2023-03-09T14:38:26.336 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:26.331+0000 7f00e21fc700 1 -- 172.21.15.38:0/3869503301 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00dc1048b0 con 0x7f00dc0fe8d0 2023-03-09T14:38:26.336 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:26.331+0000 7f00d8ff9700 1 --2- 172.21.15.38:0/3869503301 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f00c4037e20 0x7f00c403a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:38:26.337 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:26.331+0000 7f00d8ff9700 1 -- 172.21.15.38:0/3869503301 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f00d004b960 con 0x7f00dc0fe8d0 2023-03-09T14:38:26.337 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:26.331+0000 7f00daffd700 1 --2- 172.21.15.38:0/3869503301 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f00c4037e20 0x7f00c403a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:38:26.337 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:26.331+0000 7f00daffd700 1 --2- 172.21.15.38:0/3869503301 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f00c4037e20 0x7f00c403a2d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f00cc006fd0 tx=0x7f00cc006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:38:26.339 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:26.336+0000 7f00d8ff9700 1 -- 172.21.15.38:0/3869503301 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f00d002a770 con 0x7f00dc0fe8d0 2023-03-09T14:38:26.537 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:26.534+0000 7f00e21fc700 1 -- 172.21.15.38:0/3869503301 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f00dc194530 con 0x7f00dc0fe8d0 2023-03-09T14:38:26.541 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:26.538+0000 7f00d8ff9700 1 -- 172.21.15.38:0/3869503301 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f00d000e3e0 con 0x7f00dc0fe8d0 2023-03-09T14:38:26.543 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:26.540+0000 7f00e21fc700 1 -- 172.21.15.38:0/3869503301 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f00c4037e20 msgr2=0x7f00c403a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:38:26.544 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:26.540+0000 7f00e21fc700 1 --2- 172.21.15.38:0/3869503301 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f00c4037e20 0x7f00c403a2d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f00cc006fd0 tx=0x7f00cc006e40 comp rx=0 tx=0).stop 2023-03-09T14:38:26.544 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:26.540+0000 7f00e21fc700 1 -- 172.21.15.38:0/3869503301 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00dc0fe8d0 msgr2=0x7f00dc1939f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:38:26.544 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:26.540+0000 7f00e21fc700 1 --2- 172.21.15.38:0/3869503301 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00dc0fe8d0 0x7f00dc1939f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f00d0004f40 tx=0x7f00d00057c0 comp rx=0 tx=0).stop 2023-03-09T14:38:26.544 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:26.540+0000 7f00e21fc700 1 -- 172.21.15.38:0/3869503301 shutdown_connections 2023-03-09T14:38:26.544 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:26.540+0000 7f00e21fc700 1 --2- 172.21.15.38:0/3869503301 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f00c4037e20 0x7f00c403a2d0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:38:26.544 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:26.540+0000 7f00e21fc700 1 --2- 172.21.15.38:0/3869503301 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00dc0fe8d0 0x7f00dc1939f0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:38:26.544 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:26.540+0000 7f00e21fc700 1 -- 172.21.15.38:0/3869503301 >> 172.21.15.38:0/3869503301 conn(0x7f00dc0f9f10 msgr2=0x7f00dc0fab90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:38:26.545 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:26.541+0000 7f00e21fc700 1 -- 172.21.15.38:0/3869503301 shutdown_connections 2023-03-09T14:38:26.545 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:26.541+0000 7f00e21fc700 1 -- 172.21.15.38:0/3869503301 wait complete. 2023-03-09T14:38:26.965 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2023-03-09T14:38:26.965 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph orch client-keyring set client.admin '*' --mode 0755 2023-03-09T14:38:27.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:27 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:38:27.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:27 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:38:27.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:27 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:38:27.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:27 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:38:27.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:27 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:38:27.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:27 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:38:27.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:27 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/3869503301' entity='client.admin' 2023-03-09T14:38:27.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:27 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:38:27.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:27 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:38:27.622 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.a/config 2023-03-09T14:38:28.870 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:28.867+0000 7f85805b6700 1 -- 172.21.15.38:0/1649714977 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85780fe8d0 msgr2=0x7f85780fece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:38:28.870 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:28.867+0000 7f85805b6700 1 --2- 172.21.15.38:0/1649714977 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85780fe8d0 0x7f85780fece0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f856c009910 tx=0x7f856c009c20 comp rx=0 tx=0).stop 2023-03-09T14:38:28.870 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:28.867+0000 7f85805b6700 1 -- 172.21.15.38:0/1649714977 shutdown_connections 2023-03-09T14:38:28.871 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:28.867+0000 7f85805b6700 1 --2- 172.21.15.38:0/1649714977 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85780fe8d0 0x7f85780fece0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:38:28.871 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:28.867+0000 7f85805b6700 1 -- 172.21.15.38:0/1649714977 >> 172.21.15.38:0/1649714977 conn(0x7f85780f9e50 msgr2=0x7f85780fc280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:38:28.871 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:28.867+0000 7f85805b6700 1 -- 172.21.15.38:0/1649714977 shutdown_connections 2023-03-09T14:38:28.871 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:28.867+0000 7f85805b6700 1 -- 172.21.15.38:0/1649714977 wait complete. 2023-03-09T14:38:28.871 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:28.868+0000 7f85805b6700 1 Processor -- start 2023-03-09T14:38:28.871 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:28.868+0000 7f85805b6700 1 -- start start 2023-03-09T14:38:28.871 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:28.868+0000 7f85805b6700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85780fe8d0 0x7f8578194ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:38:28.872 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:28.868+0000 7f85805b6700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85781950e0 con 0x7f85780fe8d0 2023-03-09T14:38:28.872 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:28.868+0000 7f857e352700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85780fe8d0 0x7f8578194ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:38:28.872 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:28.868+0000 7f857e352700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85780fe8d0 0x7f8578194ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35686/0 (socket says 172.21.15.38:35686) 2023-03-09T14:38:28.872 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:28.868+0000 7f857e352700 1 -- 172.21.15.38:0/841764205 learned_addr learned my addr 172.21.15.38:0/841764205 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:38:28.873 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:28.868+0000 7f857e352700 1 -- 172.21.15.38:0/841764205 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f856c0095f0 con 0x7f85780fe8d0 2023-03-09T14:38:28.873 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:28.868+0000 7f857e352700 1 --2- 172.21.15.38:0/841764205 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85780fe8d0 0x7f8578194ba0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f856c006010 tx=0x7f856c0057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:38:28.874 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:28.869+0000 7f856b7fe700 1 -- 172.21.15.38:0/841764205 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f856c01d070 con 0x7f85780fe8d0 2023-03-09T14:38:28.874 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:28.869+0000 7f85805b6700 1 -- 172.21.15.38:0/841764205 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85781047a0 con 0x7f85780fe8d0 2023-03-09T14:38:28.874 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:28.869+0000 7f856b7fe700 1 -- 172.21.15.38:0/841764205 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f856c0044e0 con 0x7f85780fe8d0 2023-03-09T14:38:28.874 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:28.869+0000 7f856b7fe700 1 -- 172.21.15.38:0/841764205 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f856c0039b0 con 0x7f85780fe8d0 2023-03-09T14:38:28.874 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:28.869+0000 7f85805b6700 1 -- 172.21.15.38:0/841764205 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8578104610 con 0x7f85780fe8d0 2023-03-09T14:38:28.874 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:28.869+0000 7f856b7fe700 1 -- 172.21.15.38:0/841764205 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 44674+0+0 (secure 0 0 0) 0x7f856c00c040 con 0x7f85780fe8d0 2023-03-09T14:38:28.874 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:28.870+0000 7f85805b6700 1 -- 172.21.15.38:0/841764205 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f857818e470 con 0x7f85780fe8d0 2023-03-09T14:38:28.875 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:28.870+0000 7f856b7fe700 1 --2- 172.21.15.38:0/841764205 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f8564037e20 0x7f856403a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:38:28.875 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:28.870+0000 7f856b7fe700 1 -- 172.21.15.38:0/841764205 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f856c04b960 con 0x7f85780fe8d0 2023-03-09T14:38:28.875 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:28.870+0000 7f857db51700 1 --2- 172.21.15.38:0/841764205 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f8564037e20 0x7f856403a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:38:28.875 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:28.870+0000 7f857db51700 1 --2- 172.21.15.38:0/841764205 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f8564037e20 0x7f856403a2d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f8574006fd0 tx=0x7f8574006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:38:28.877 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:28.875+0000 7f856b7fe700 1 -- 172.21.15.38:0/841764205 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f856c02a770 con 0x7f85780fe8d0 2023-03-09T14:38:29.078 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:29.076+0000 7f85805b6700 1 -- 172.21.15.38:0/841764205 --> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f8578060bc0 con 0x7f8564037e20 2023-03-09T14:38:29.086 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:29.084+0000 7f856b7fe700 1 -- 172.21.15.38:0/841764205 <== mgr.14150 v2:172.21.15.38:6800/3328887328 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f8578060bc0 con 0x7f8564037e20 2023-03-09T14:38:29.089 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:29.087+0000 7f85805b6700 1 -- 172.21.15.38:0/841764205 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f8564037e20 msgr2=0x7f856403a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:38:29.089 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:29.087+0000 7f85805b6700 1 --2- 172.21.15.38:0/841764205 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f8564037e20 0x7f856403a2d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f8574006fd0 tx=0x7f8574006e40 comp rx=0 tx=0).stop 2023-03-09T14:38:29.089 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:29.087+0000 7f85805b6700 1 -- 172.21.15.38:0/841764205 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85780fe8d0 msgr2=0x7f8578194ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:38:29.090 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:29.087+0000 7f85805b6700 1 --2- 172.21.15.38:0/841764205 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85780fe8d0 0x7f8578194ba0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f856c006010 tx=0x7f856c0057c0 comp rx=0 tx=0).stop 2023-03-09T14:38:29.090 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:29.087+0000 7f85805b6700 1 -- 172.21.15.38:0/841764205 shutdown_connections 2023-03-09T14:38:29.090 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:29.087+0000 7f85805b6700 1 --2- 172.21.15.38:0/841764205 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f8564037e20 0x7f856403a2d0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:38:29.090 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:29.087+0000 7f85805b6700 1 --2- 172.21.15.38:0/841764205 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85780fe8d0 0x7f8578194ba0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:38:29.090 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:29.087+0000 7f85805b6700 1 -- 172.21.15.38:0/841764205 >> 172.21.15.38:0/841764205 conn(0x7f85780f9e50 msgr2=0x7f8578068a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:38:29.090 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:29.087+0000 7f85805b6700 1 -- 172.21.15.38:0/841764205 shutdown_connections 2023-03-09T14:38:29.091 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:29.087+0000 7f85805b6700 1 -- 172.21.15.38:0/841764205 wait complete. 2023-03-09T14:38:29.580 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi122 2023-03-09T14:38:29.581 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2023-03-09T14:38:29.581 DEBUG:teuthology.orchestra.run.smithi122:> dd of=/etc/ceph/ceph.conf 2023-03-09T14:38:29.601 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2023-03-09T14:38:29.602 DEBUG:teuthology.orchestra.run.smithi122:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-03-09T14:38:29.662 INFO:tasks.cephadm:Adding host smithi122 to orchestrator... 2023-03-09T14:38:29.662 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph orch host add smithi122 2023-03-09T14:38:30.364 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.a/config 2023-03-09T14:38:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:30 smithi038 ceph-mon[92422]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:38:30.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:30 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:38:30.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:30 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:38:30.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:30 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:38:30.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:30 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:38:30.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:30 smithi038 ceph-mon[92422]: Updating smithi038:/etc/ceph/ceph.conf 2023-03-09T14:38:30.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:30 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:38:30.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:30 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:38:30.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:30 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:38:31.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:31 smithi038 ceph-mon[92422]: Updating smithi038:/var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/config/ceph.conf 2023-03-09T14:38:31.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:31 smithi038 ceph-mon[92422]: Updating smithi038:/etc/ceph/ceph.client.admin.keyring 2023-03-09T14:38:31.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:31 smithi038 ceph-mon[92422]: Updating smithi038:/var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/config/ceph.client.admin.keyring 2023-03-09T14:38:31.538 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:31.536+0000 7f55f2469700 1 -- 172.21.15.38:0/828145662 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55ec104910 msgr2=0x7f55ec104ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:38:31.539 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:31.536+0000 7f55f2469700 1 --2- 172.21.15.38:0/828145662 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55ec104910 0x7f55ec104ce0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f55dc009910 tx=0x7f55dc009c20 comp rx=0 tx=0).stop 2023-03-09T14:38:31.539 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:31.536+0000 7f55f2469700 1 -- 172.21.15.38:0/828145662 shutdown_connections 2023-03-09T14:38:31.539 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:31.536+0000 7f55f2469700 1 --2- 172.21.15.38:0/828145662 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55ec104910 0x7f55ec104ce0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:38:31.539 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:31.536+0000 7f55f2469700 1 -- 172.21.15.38:0/828145662 >> 172.21.15.38:0/828145662 conn(0x7f55ec0745e0 msgr2=0x7f55ec0749e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:38:31.539 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:31.536+0000 7f55f2469700 1 -- 172.21.15.38:0/828145662 shutdown_connections 2023-03-09T14:38:31.540 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:31.536+0000 7f55f2469700 1 -- 172.21.15.38:0/828145662 wait complete. 2023-03-09T14:38:31.540 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:31.537+0000 7f55f2469700 1 Processor -- start 2023-03-09T14:38:31.540 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:31.537+0000 7f55f2469700 1 -- start start 2023-03-09T14:38:31.541 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:31.537+0000 7f55f2469700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55ec104910 0x7f55ec195c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:38:31.541 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:31.537+0000 7f55f2469700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55ec196160 con 0x7f55ec104910 2023-03-09T14:38:31.541 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:31.537+0000 7f55ebfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55ec104910 0x7f55ec195c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:38:31.542 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:31.537+0000 7f55ebfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55ec104910 0x7f55ec195c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35692/0 (socket says 172.21.15.38:35692) 2023-03-09T14:38:31.542 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:31.537+0000 7f55ebfff700 1 -- 172.21.15.38:0/437820089 learned_addr learned my addr 172.21.15.38:0/437820089 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:38:31.542 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:31.537+0000 7f55ebfff700 1 -- 172.21.15.38:0/437820089 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55dc0095f0 con 0x7f55ec104910 2023-03-09T14:38:31.542 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:31.538+0000 7f55ebfff700 1 --2- 172.21.15.38:0/437820089 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55ec104910 0x7f55ec195c20 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f55dc0092e0 tx=0x7f55dc004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:38:31.542 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:31.538+0000 7f55e97fa700 1 -- 172.21.15.38:0/437820089 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f55dc01d070 con 0x7f55ec104910 2023-03-09T14:38:31.542 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:31.538+0000 7f55f2469700 1 -- 172.21.15.38:0/437820089 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55ec108d00 con 0x7f55ec104910 2023-03-09T14:38:31.542 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:31.538+0000 7f55e97fa700 1 -- 172.21.15.38:0/437820089 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f55dc00bbb0 con 0x7f55ec104910 2023-03-09T14:38:31.542 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:31.538+0000 7f55e97fa700 1 -- 172.21.15.38:0/437820089 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f55dc003930 con 0x7f55ec104910 2023-03-09T14:38:31.543 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:31.538+0000 7f55f2469700 1 -- 172.21.15.38:0/437820089 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55ec108b70 con 0x7f55ec104910 2023-03-09T14:38:31.543 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:31.539+0000 7f55e97fa700 1 -- 172.21.15.38:0/437820089 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 44674+0+0 (secure 0 0 0) 0x7f55dc028030 con 0x7f55ec104910 2023-03-09T14:38:31.543 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:31.539+0000 7f55f2469700 1 -- 172.21.15.38:0/437820089 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55ec108e90 con 0x7f55ec104910 2023-03-09T14:38:31.543 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:31.539+0000 7f55e97fa700 1 --2- 172.21.15.38:0/437820089 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f55d8037e00 0x7f55d803a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:38:31.543 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:31.539+0000 7f55e97fa700 1 -- 172.21.15.38:0/437820089 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f55dc04b3c0 con 0x7f55ec104910 2023-03-09T14:38:31.543 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:31.539+0000 7f55eb7fe700 1 --2- 172.21.15.38:0/437820089 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f55d8037e00 0x7f55d803a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:38:31.543 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:31.539+0000 7f55eb7fe700 1 --2- 172.21.15.38:0/437820089 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f55d8037e00 0x7f55d803a2b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f55d4006fd0 tx=0x7f55d4006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:38:31.545 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:31.544+0000 7f55e97fa700 1 -- 172.21.15.38:0/437820089 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f55dc029360 con 0x7f55ec104910 2023-03-09T14:38:31.750 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:31.748+0000 7f55f2469700 1 -- 172.21.15.38:0/437820089 --> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi122", "target": ["mon-mgr", ""]}) v1 -- 0x7f55ec0680c0 con 0x7f55d8037e00 2023-03-09T14:38:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:33 smithi038 ceph-mon[92422]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi122", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:38:33.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:33 smithi038 ceph-mon[92422]: Deploying cephadm binary to smithi122 2023-03-09T14:38:33.818 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:33.815+0000 7f55e97fa700 1 -- 172.21.15.38:0/437820089 <== mgr.14150 v2:172.21.15.38:6800/3328887328 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f55ec0680c0 con 0x7f55d8037e00 2023-03-09T14:38:33.818 INFO:teuthology.orchestra.run.smithi038.stdout:Added host 'smithi122' with addr '172.21.15.122' 2023-03-09T14:38:33.821 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:33.818+0000 7f55f2469700 1 -- 172.21.15.38:0/437820089 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f55d8037e00 msgr2=0x7f55d803a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:38:33.821 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:33.818+0000 7f55f2469700 1 --2- 172.21.15.38:0/437820089 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f55d8037e00 0x7f55d803a2b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f55d4006fd0 tx=0x7f55d4006e40 comp rx=0 tx=0).stop 2023-03-09T14:38:33.821 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:33.818+0000 7f55f2469700 1 -- 172.21.15.38:0/437820089 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55ec104910 msgr2=0x7f55ec195c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:38:33.821 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:33.818+0000 7f55f2469700 1 --2- 172.21.15.38:0/437820089 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55ec104910 0x7f55ec195c20 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f55dc0092e0 tx=0x7f55dc004ff0 comp rx=0 tx=0).stop 2023-03-09T14:38:33.821 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:33.819+0000 7f55f2469700 1 -- 172.21.15.38:0/437820089 shutdown_connections 2023-03-09T14:38:33.821 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:33.819+0000 7f55f2469700 1 --2- 172.21.15.38:0/437820089 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f55d8037e00 0x7f55d803a2b0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:38:33.822 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:33.819+0000 7f55f2469700 1 --2- 172.21.15.38:0/437820089 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55ec104910 0x7f55ec195c20 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:38:33.822 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:33.819+0000 7f55f2469700 1 -- 172.21.15.38:0/437820089 >> 172.21.15.38:0/437820089 conn(0x7f55ec0745e0 msgr2=0x7f55ec105f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:38:33.822 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:33.819+0000 7f55f2469700 1 -- 172.21.15.38:0/437820089 shutdown_connections 2023-03-09T14:38:33.822 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:33.819+0000 7f55f2469700 1 -- 172.21.15.38:0/437820089 wait complete. 2023-03-09T14:38:34.219 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph orch host ls --format=json 2023-03-09T14:38:34.879 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.a/config 2023-03-09T14:38:35.085 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:34 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:38:35.085 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:34 smithi038 ceph-mon[92422]: Added host smithi122 2023-03-09T14:38:35.085 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:34 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:38:35.086 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:34 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:38:36.048 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:36.044+0000 7f29f9c63700 1 -- 172.21.15.38:0/1701220870 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f29f4105a00 msgr2=0x7f29f4105dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:38:36.049 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:36.044+0000 7f29f9c63700 1 --2- 172.21.15.38:0/1701220870 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f29f4105a00 0x7f29f4105dd0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f29e4009910 tx=0x7f29e4009c20 comp rx=0 tx=0).stop 2023-03-09T14:38:36.049 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:36.044+0000 7f29f9c63700 1 -- 172.21.15.38:0/1701220870 shutdown_connections 2023-03-09T14:38:36.049 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:36.044+0000 7f29f9c63700 1 --2- 172.21.15.38:0/1701220870 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f29f4105a00 0x7f29f4105dd0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:38:36.049 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:36.044+0000 7f29f9c63700 1 -- 172.21.15.38:0/1701220870 >> 172.21.15.38:0/1701220870 conn(0x7f29f40fb510 msgr2=0x7f29f40fd920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:38:36.049 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:36.044+0000 7f29f9c63700 1 -- 172.21.15.38:0/1701220870 shutdown_connections 2023-03-09T14:38:36.049 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:36.044+0000 7f29f9c63700 1 -- 172.21.15.38:0/1701220870 wait complete. 2023-03-09T14:38:36.049 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:36.045+0000 7f29f9c63700 1 Processor -- start 2023-03-09T14:38:36.050 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:36.045+0000 7f29f9c63700 1 -- start start 2023-03-09T14:38:36.050 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:36.045+0000 7f29f9c63700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f29f4105a00 0x7f29f4072a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:38:36.050 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:36.045+0000 7f29f9c63700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29f4074a10 con 0x7f29f4105a00 2023-03-09T14:38:36.050 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:36.046+0000 7f29f37fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f29f4105a00 0x7f29f4072a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:38:36.050 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:36.046+0000 7f29f37fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f29f4105a00 0x7f29f4072a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35718/0 (socket says 172.21.15.38:35718) 2023-03-09T14:38:36.050 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:36.046+0000 7f29f37fe700 1 -- 172.21.15.38:0/621924575 learned_addr learned my addr 172.21.15.38:0/621924575 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:38:36.050 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:36.046+0000 7f29f37fe700 1 -- 172.21.15.38:0/621924575 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29e40095f0 con 0x7f29f4105a00 2023-03-09T14:38:36.050 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:36.046+0000 7f29f37fe700 1 --2- 172.21.15.38:0/621924575 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f29f4105a00 0x7f29f4072a80 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f29e4004f40 tx=0x7f29e4005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:38:36.051 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:36.046+0000 7f29f0ff9700 1 -- 172.21.15.38:0/621924575 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f29e401d070 con 0x7f29f4105a00 2023-03-09T14:38:36.051 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:36.046+0000 7f29f0ff9700 1 -- 172.21.15.38:0/621924575 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f29e400be20 con 0x7f29f4105a00 2023-03-09T14:38:36.051 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:36.046+0000 7f29f0ff9700 1 -- 172.21.15.38:0/621924575 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f29e4022d00 con 0x7f29f4105a00 2023-03-09T14:38:36.051 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:36.046+0000 7f29f9c63700 1 -- 172.21.15.38:0/621924575 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29f4109df0 con 0x7f29f4105a00 2023-03-09T14:38:36.052 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:36.046+0000 7f29f9c63700 1 -- 172.21.15.38:0/621924575 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29f4109c60 con 0x7f29f4105a00 2023-03-09T14:38:36.052 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:36.047+0000 7f29f0ff9700 1 -- 172.21.15.38:0/621924575 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 44674+0+0 (secure 0 0 0) 0x7f29e4029030 con 0x7f29f4105a00 2023-03-09T14:38:36.052 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:36.047+0000 7f29f0ff9700 1 --2- 172.21.15.38:0/621924575 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f29e0037e50 0x7f29e003a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:38:36.052 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:36.047+0000 7f29f0ff9700 1 -- 172.21.15.38:0/621924575 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f29e404c6f0 con 0x7f29f4105a00 2023-03-09T14:38:36.052 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:36.047+0000 7f29f2ffd700 1 --2- 172.21.15.38:0/621924575 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f29e0037e50 0x7f29e003a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:38:36.053 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:36.047+0000 7f29f9c63700 1 -- 172.21.15.38:0/621924575 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29f4109f80 con 0x7f29f4105a00 2023-03-09T14:38:36.053 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:36.048+0000 7f29f2ffd700 1 --2- 172.21.15.38:0/621924575 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f29e0037e50 0x7f29e003a300 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f29dc006fd0 tx=0x7f29dc006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:38:36.054 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:36.052+0000 7f29f0ff9700 1 -- 172.21.15.38:0/621924575 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f29e4027030 con 0x7f29f4105a00 2023-03-09T14:38:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:36 smithi038 ceph-mon[92422]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:38:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:36 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:38:36.258 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:36.256+0000 7f29f9c63700 1 -- 172.21.15.38:0/621924575 --> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f29f4061e20 con 0x7f29e0037e50 2023-03-09T14:38:36.259 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:36.257+0000 7f29f0ff9700 1 -- 172.21.15.38:0/621924575 <== mgr.14150 v2:172.21.15.38:6800/3328887328 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+160 (secure 0 0 0) 0x7f29f4061e20 con 0x7f29e0037e50 2023-03-09T14:38:36.259 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:38:36.260 INFO:teuthology.orchestra.run.smithi038.stdout:[{"addr": "172.21.15.38", "hostname": "smithi038", "labels": [], "status": ""}, {"addr": "172.21.15.122", "hostname": "smithi122", "labels": [], "status": ""}] 2023-03-09T14:38:36.261 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:36.259+0000 7f29f9c63700 1 -- 172.21.15.38:0/621924575 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f29e0037e50 msgr2=0x7f29e003a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:38:36.261 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:36.259+0000 7f29f9c63700 1 --2- 172.21.15.38:0/621924575 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f29e0037e50 0x7f29e003a300 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f29dc006fd0 tx=0x7f29dc006e40 comp rx=0 tx=0).stop 2023-03-09T14:38:36.262 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:36.259+0000 7f29f9c63700 1 -- 172.21.15.38:0/621924575 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f29f4105a00 msgr2=0x7f29f4072a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:38:36.262 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:36.259+0000 7f29f9c63700 1 --2- 172.21.15.38:0/621924575 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f29f4105a00 0x7f29f4072a80 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f29e4004f40 tx=0x7f29e4005020 comp rx=0 tx=0).stop 2023-03-09T14:38:36.262 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:36.259+0000 7f29f9c63700 1 -- 172.21.15.38:0/621924575 shutdown_connections 2023-03-09T14:38:36.262 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:36.259+0000 7f29f9c63700 1 --2- 172.21.15.38:0/621924575 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f29e0037e50 0x7f29e003a300 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:38:36.262 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:36.259+0000 7f29f9c63700 1 --2- 172.21.15.38:0/621924575 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f29f4105a00 0x7f29f4072a80 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:38:36.262 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:36.259+0000 7f29f9c63700 1 -- 172.21.15.38:0/621924575 >> 172.21.15.38:0/621924575 conn(0x7f29f40fb510 msgr2=0x7f29f40fc170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:38:36.263 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:36.259+0000 7f29f9c63700 1 -- 172.21.15.38:0/621924575 shutdown_connections 2023-03-09T14:38:36.263 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:36.259+0000 7f29f9c63700 1 -- 172.21.15.38:0/621924575 wait complete. 2023-03-09T14:38:36.639 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi181 2023-03-09T14:38:36.640 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2023-03-09T14:38:36.640 DEBUG:teuthology.orchestra.run.smithi181:> dd of=/etc/ceph/ceph.conf 2023-03-09T14:38:36.660 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2023-03-09T14:38:36.661 DEBUG:teuthology.orchestra.run.smithi181:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-03-09T14:38:36.719 INFO:tasks.cephadm:Adding host smithi181 to orchestrator... 2023-03-09T14:38:36.720 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph orch host add smithi181 2023-03-09T14:38:37.374 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.a/config 2023-03-09T14:38:37.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:37 smithi038 ceph-mon[92422]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-03-09T14:38:37.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:37 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:38:38.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:38 smithi038 ceph-mon[92422]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:38:38.530 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:38.527+0000 7f6b3b0a2700 1 -- 172.21.15.38:0/3890845280 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6b34104910 msgr2=0x7f6b34104ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:38:38.530 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:38.527+0000 7f6b3b0a2700 1 --2- 172.21.15.38:0/3890845280 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6b34104910 0x7f6b34104ce0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6b28009910 tx=0x7f6b28009c20 comp rx=0 tx=0).stop 2023-03-09T14:38:38.530 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:38.527+0000 7f6b3b0a2700 1 -- 172.21.15.38:0/3890845280 shutdown_connections 2023-03-09T14:38:38.530 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:38.527+0000 7f6b3b0a2700 1 --2- 172.21.15.38:0/3890845280 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6b34104910 0x7f6b34104ce0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:38:38.531 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:38.527+0000 7f6b3b0a2700 1 -- 172.21.15.38:0/3890845280 >> 172.21.15.38:0/3890845280 conn(0x7f6b340745e0 msgr2=0x7f6b340749e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:38:38.531 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:38.527+0000 7f6b3b0a2700 1 -- 172.21.15.38:0/3890845280 shutdown_connections 2023-03-09T14:38:38.531 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:38.527+0000 7f6b3b0a2700 1 -- 172.21.15.38:0/3890845280 wait complete. 2023-03-09T14:38:38.531 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:38.527+0000 7f6b3b0a2700 1 Processor -- start 2023-03-09T14:38:38.531 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:38.528+0000 7f6b3b0a2700 1 -- start start 2023-03-09T14:38:38.531 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:38.528+0000 7f6b3b0a2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6b34104910 0x7f6b34194e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:38:38.531 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:38.528+0000 7f6b3b0a2700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b34195370 con 0x7f6b34104910 2023-03-09T14:38:38.532 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:38.528+0000 7f6b38e3e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6b34104910 0x7f6b34194e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:38:38.532 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:38.528+0000 7f6b38e3e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6b34104910 0x7f6b34194e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35724/0 (socket says 172.21.15.38:35724) 2023-03-09T14:38:38.532 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:38.528+0000 7f6b38e3e700 1 -- 172.21.15.38:0/1917078151 learned_addr learned my addr 172.21.15.38:0/1917078151 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:38:38.534 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:38.528+0000 7f6b38e3e700 1 -- 172.21.15.38:0/1917078151 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b280095f0 con 0x7f6b34104910 2023-03-09T14:38:38.534 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:38.528+0000 7f6b38e3e700 1 --2- 172.21.15.38:0/1917078151 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6b34104910 0x7f6b34194e30 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6b280092e0 tx=0x7f6b28004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:38:38.535 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:38.528+0000 7f6b31ffb700 1 -- 172.21.15.38:0/1917078151 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6b2801d070 con 0x7f6b34104910 2023-03-09T14:38:38.535 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:38.528+0000 7f6b31ffb700 1 -- 172.21.15.38:0/1917078151 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6b2800bbb0 con 0x7f6b34104910 2023-03-09T14:38:38.535 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:38.528+0000 7f6b3b0a2700 1 -- 172.21.15.38:0/1917078151 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b34108d00 con 0x7f6b34104910 2023-03-09T14:38:38.535 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:38.528+0000 7f6b31ffb700 1 -- 172.21.15.38:0/1917078151 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6b28003930 con 0x7f6b34104910 2023-03-09T14:38:38.536 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:38.529+0000 7f6b3b0a2700 1 -- 172.21.15.38:0/1917078151 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b34108b70 con 0x7f6b34104910 2023-03-09T14:38:38.536 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:38.529+0000 7f6b31ffb700 1 -- 172.21.15.38:0/1917078151 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 44674+0+0 (secure 0 0 0) 0x7f6b28028030 con 0x7f6b34104910 2023-03-09T14:38:38.536 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:38.530+0000 7f6b3b0a2700 1 -- 172.21.15.38:0/1917078151 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b34108e90 con 0x7f6b34104910 2023-03-09T14:38:38.536 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:38.530+0000 7f6b31ffb700 1 --2- 172.21.15.38:0/1917078151 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f6b24037e00 0x7f6b2403a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:38:38.537 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:38.530+0000 7f6b31ffb700 1 -- 172.21.15.38:0/1917078151 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f6b2804b340 con 0x7f6b34104910 2023-03-09T14:38:38.537 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:38.530+0000 7f6b33fff700 1 --2- 172.21.15.38:0/1917078151 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f6b24037e00 0x7f6b2403a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:38:38.537 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:38.530+0000 7f6b33fff700 1 --2- 172.21.15.38:0/1917078151 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f6b24037e00 0x7f6b2403a2b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f6b20006fd0 tx=0x7f6b20006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:38:38.538 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:38.535+0000 7f6b31ffb700 1 -- 172.21.15.38:0/1917078151 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f6b28029360 con 0x7f6b34104910 2023-03-09T14:38:38.739 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:38.737+0000 7f6b3b0a2700 1 -- 172.21.15.38:0/1917078151 --> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi181", "target": ["mon-mgr", ""]}) v1 -- 0x7f6b34196290 con 0x7f6b24037e00 2023-03-09T14:38:40.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:39 smithi038 ceph-mon[92422]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:38:40.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:39 smithi038 ceph-mon[92422]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi181", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:38:40.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:39 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:38:40.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:39 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:38:40.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:39 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:38:40.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:39 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:38:40.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:39 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi122", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:38:40.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:39 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:38:40.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:39 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:38:40.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:39 smithi038 ceph-mon[92422]: Updating smithi122:/etc/ceph/ceph.conf 2023-03-09T14:38:40.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:39 smithi038 ceph-mon[92422]: Updating smithi122:/var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/config/ceph.conf 2023-03-09T14:38:40.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:39 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:38:40.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:39 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:38:40.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:39 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:38:40.870 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:40.868+0000 7f6b31ffb700 1 -- 172.21.15.38:0/1917078151 <== mgr.14150 v2:172.21.15.38:6800/3328887328 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f6b34196290 con 0x7f6b24037e00 2023-03-09T14:38:40.871 INFO:teuthology.orchestra.run.smithi038.stdout:Added host 'smithi181' with addr '172.21.15.181' 2023-03-09T14:38:40.873 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:40.871+0000 7f6b3b0a2700 1 -- 172.21.15.38:0/1917078151 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f6b24037e00 msgr2=0x7f6b2403a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:38:40.873 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:40.871+0000 7f6b3b0a2700 1 --2- 172.21.15.38:0/1917078151 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f6b24037e00 0x7f6b2403a2b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f6b20006fd0 tx=0x7f6b20006e40 comp rx=0 tx=0).stop 2023-03-09T14:38:40.873 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:40.871+0000 7f6b3b0a2700 1 -- 172.21.15.38:0/1917078151 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6b34104910 msgr2=0x7f6b34194e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:38:40.874 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:40.871+0000 7f6b3b0a2700 1 --2- 172.21.15.38:0/1917078151 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6b34104910 0x7f6b34194e30 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6b280092e0 tx=0x7f6b28004ff0 comp rx=0 tx=0).stop 2023-03-09T14:38:40.874 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:40.871+0000 7f6b3b0a2700 1 -- 172.21.15.38:0/1917078151 shutdown_connections 2023-03-09T14:38:40.874 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:40.871+0000 7f6b3b0a2700 1 --2- 172.21.15.38:0/1917078151 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f6b24037e00 0x7f6b2403a2b0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:38:40.874 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:40.871+0000 7f6b3b0a2700 1 --2- 172.21.15.38:0/1917078151 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6b34104910 0x7f6b34194e30 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:38:40.874 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:40.871+0000 7f6b3b0a2700 1 -- 172.21.15.38:0/1917078151 >> 172.21.15.38:0/1917078151 conn(0x7f6b340745e0 msgr2=0x7f6b34067960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:38:40.874 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:40.871+0000 7f6b3b0a2700 1 -- 172.21.15.38:0/1917078151 shutdown_connections 2023-03-09T14:38:40.874 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:40.871+0000 7f6b3b0a2700 1 -- 172.21.15.38:0/1917078151 wait complete. 2023-03-09T14:38:41.227 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:40 smithi038 ceph-mon[92422]: Deploying cephadm binary to smithi181 2023-03-09T14:38:41.227 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:40 smithi038 ceph-mon[92422]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2023-03-09T14:38:41.228 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:40 smithi038 ceph-mon[92422]: Updating smithi122:/var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/config/ceph.client.admin.keyring 2023-03-09T14:38:41.228 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:40 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:38:41.240 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph orch host ls --format=json 2023-03-09T14:38:41.897 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.a/config 2023-03-09T14:38:42.147 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:41 smithi038 ceph-mon[92422]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:38:42.148 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:41 smithi038 ceph-mon[92422]: Added host smithi181 2023-03-09T14:38:42.148 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:41 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:38:42.148 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:41 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:38:43.138 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:43.135+0000 7fa28c4e6700 1 -- 172.21.15.38:0/2058187899 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa284105a00 msgr2=0x7fa284105dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:38:43.138 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:43.135+0000 7fa28c4e6700 1 --2- 172.21.15.38:0/2058187899 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa284105a00 0x7fa284105dd0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa280009910 tx=0x7fa280009c20 comp rx=0 tx=0).stop 2023-03-09T14:38:43.138 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:43.135+0000 7fa28c4e6700 1 -- 172.21.15.38:0/2058187899 shutdown_connections 2023-03-09T14:38:43.139 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:43.135+0000 7fa28c4e6700 1 --2- 172.21.15.38:0/2058187899 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa284105a00 0x7fa284105dd0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:38:43.139 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:43.135+0000 7fa28c4e6700 1 -- 172.21.15.38:0/2058187899 >> 172.21.15.38:0/2058187899 conn(0x7fa2840fb510 msgr2=0x7fa2840fd920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:38:43.139 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:43.135+0000 7fa28c4e6700 1 -- 172.21.15.38:0/2058187899 shutdown_connections 2023-03-09T14:38:43.139 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:43.135+0000 7fa28c4e6700 1 -- 172.21.15.38:0/2058187899 wait complete. 2023-03-09T14:38:43.139 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:43.135+0000 7fa28c4e6700 1 Processor -- start 2023-03-09T14:38:43.139 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:43.135+0000 7fa28c4e6700 1 -- start start 2023-03-09T14:38:43.139 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:43.136+0000 7fa28c4e6700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa284105a00 0x7fa284194e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:38:43.140 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:43.136+0000 7fa28c4e6700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa284195350 con 0x7fa284105a00 2023-03-09T14:38:43.140 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:43.136+0000 7fa28a282700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa284105a00 0x7fa284194e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:38:43.140 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:43.136+0000 7fa28a282700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa284105a00 0x7fa284194e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35750/0 (socket says 172.21.15.38:35750) 2023-03-09T14:38:43.140 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:43.136+0000 7fa28a282700 1 -- 172.21.15.38:0/4259188062 learned_addr learned my addr 172.21.15.38:0/4259188062 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:38:43.140 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:43.136+0000 7fa28a282700 1 -- 172.21.15.38:0/4259188062 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2800095f0 con 0x7fa284105a00 2023-03-09T14:38:43.140 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:43.136+0000 7fa28a282700 1 --2- 172.21.15.38:0/4259188062 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa284105a00 0x7fa284194e10 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa280004f40 tx=0x7fa280005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:38:43.141 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:43.136+0000 7fa27b7fe700 1 -- 172.21.15.38:0/4259188062 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa28001d070 con 0x7fa284105a00 2023-03-09T14:38:43.141 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:43.136+0000 7fa27b7fe700 1 -- 172.21.15.38:0/4259188062 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa28000be20 con 0x7fa284105a00 2023-03-09T14:38:43.142 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:43.136+0000 7fa28c4e6700 1 -- 172.21.15.38:0/4259188062 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa284109df0 con 0x7fa284105a00 2023-03-09T14:38:43.142 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:43.136+0000 7fa27b7fe700 1 -- 172.21.15.38:0/4259188062 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa280022d00 con 0x7fa284105a00 2023-03-09T14:38:43.142 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:43.136+0000 7fa28c4e6700 1 -- 172.21.15.38:0/4259188062 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa284109c60 con 0x7fa284105a00 2023-03-09T14:38:43.142 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:43.137+0000 7fa28c4e6700 1 -- 172.21.15.38:0/4259188062 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa284109f80 con 0x7fa284105a00 2023-03-09T14:38:43.142 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:43.138+0000 7fa27b7fe700 1 -- 172.21.15.38:0/4259188062 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 44674+0+0 (secure 0 0 0) 0x7fa280029030 con 0x7fa284105a00 2023-03-09T14:38:43.142 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:43.138+0000 7fa27b7fe700 1 --2- 172.21.15.38:0/4259188062 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fa270037e00 0x7fa27003a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:38:43.142 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:43.138+0000 7fa27b7fe700 1 -- 172.21.15.38:0/4259188062 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fa28004bb40 con 0x7fa284105a00 2023-03-09T14:38:43.143 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:43.138+0000 7fa289a81700 1 --2- 172.21.15.38:0/4259188062 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fa270037e00 0x7fa27003a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:38:43.143 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:43.138+0000 7fa289a81700 1 --2- 172.21.15.38:0/4259188062 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fa270037e00 0x7fa27003a2b0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fa274006fd0 tx=0x7fa274006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:38:43.144 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:43.142+0000 7fa27b7fe700 1 -- 172.21.15.38:0/4259188062 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7fa28000ed30 con 0x7fa284105a00 2023-03-09T14:38:43.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:43 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:38:43.348 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:43.346+0000 7fa28c4e6700 1 -- 172.21.15.38:0/4259188062 --> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa284196270 con 0x7fa270037e00 2023-03-09T14:38:43.349 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:43.347+0000 7fa27b7fe700 1 -- 172.21.15.38:0/4259188062 <== mgr.14150 v2:172.21.15.38:6800/3328887328 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+240 (secure 0 0 0) 0x7fa284196270 con 0x7fa270037e00 2023-03-09T14:38:43.350 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:38:43.350 INFO:teuthology.orchestra.run.smithi038.stdout:[{"addr": "172.21.15.38", "hostname": "smithi038", "labels": [], "status": ""}, {"addr": "172.21.15.122", "hostname": "smithi122", "labels": [], "status": ""}, {"addr": "172.21.15.181", "hostname": "smithi181", "labels": [], "status": ""}] 2023-03-09T14:38:43.351 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:43.349+0000 7fa28c4e6700 1 -- 172.21.15.38:0/4259188062 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fa270037e00 msgr2=0x7fa27003a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:38:43.352 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:43.349+0000 7fa28c4e6700 1 --2- 172.21.15.38:0/4259188062 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fa270037e00 0x7fa27003a2b0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fa274006fd0 tx=0x7fa274006e40 comp rx=0 tx=0).stop 2023-03-09T14:38:43.352 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:43.349+0000 7fa28c4e6700 1 -- 172.21.15.38:0/4259188062 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa284105a00 msgr2=0x7fa284194e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:38:43.352 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:43.349+0000 7fa28c4e6700 1 --2- 172.21.15.38:0/4259188062 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa284105a00 0x7fa284194e10 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa280004f40 tx=0x7fa280005020 comp rx=0 tx=0).stop 2023-03-09T14:38:43.352 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:43.349+0000 7fa28c4e6700 1 -- 172.21.15.38:0/4259188062 shutdown_connections 2023-03-09T14:38:43.352 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:43.349+0000 7fa28c4e6700 1 --2- 172.21.15.38:0/4259188062 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fa270037e00 0x7fa27003a2b0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:38:43.353 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:43.349+0000 7fa28c4e6700 1 --2- 172.21.15.38:0/4259188062 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa284105a00 0x7fa284194e10 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:38:43.353 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:43.349+0000 7fa28c4e6700 1 -- 172.21.15.38:0/4259188062 >> 172.21.15.38:0/4259188062 conn(0x7fa2840fb510 msgr2=0x7fa2840fc170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:38:43.353 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:43.349+0000 7fa28c4e6700 1 -- 172.21.15.38:0/4259188062 shutdown_connections 2023-03-09T14:38:43.353 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:43.349+0000 7fa28c4e6700 1 -- 172.21.15.38:0/4259188062 wait complete. 2023-03-09T14:38:43.755 INFO:tasks.cephadm:Setting crush tunables to default 2023-03-09T14:38:43.756 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph osd crush tunables default 2023-03-09T14:38:44.407 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.a/config 2023-03-09T14:38:44.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:44 smithi038 ceph-mon[92422]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:38:44.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:44 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:38:45.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:45 smithi038 ceph-mon[92422]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-03-09T14:38:45.637 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:45.634+0000 7f493d5eb700 1 -- 172.21.15.38:0/2542613345 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4938104900 msgr2=0x7f4938104cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:38:45.637 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:45.634+0000 7f493d5eb700 1 --2- 172.21.15.38:0/2542613345 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4938104900 0x7f4938104cd0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4928009910 tx=0x7f4928009c20 comp rx=0 tx=0).stop 2023-03-09T14:38:45.637 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:45.634+0000 7f493d5eb700 1 -- 172.21.15.38:0/2542613345 shutdown_connections 2023-03-09T14:38:45.637 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:45.634+0000 7f493d5eb700 1 --2- 172.21.15.38:0/2542613345 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4938104900 0x7f4938104cd0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:38:45.638 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:45.634+0000 7f493d5eb700 1 -- 172.21.15.38:0/2542613345 >> 172.21.15.38:0/2542613345 conn(0x7f49380fa3f0 msgr2=0x7f49380fc800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:38:45.638 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:45.634+0000 7f493d5eb700 1 -- 172.21.15.38:0/2542613345 shutdown_connections 2023-03-09T14:38:45.638 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:45.634+0000 7f493d5eb700 1 -- 172.21.15.38:0/2542613345 wait complete. 2023-03-09T14:38:45.639 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:45.635+0000 7f493d5eb700 1 Processor -- start 2023-03-09T14:38:45.640 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:45.635+0000 7f493d5eb700 1 -- start start 2023-03-09T14:38:45.640 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:45.636+0000 7f493d5eb700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4938104900 0x7f4938194eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:38:45.640 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:45.636+0000 7f493d5eb700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49381953f0 con 0x7f4938104900 2023-03-09T14:38:45.640 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:45.636+0000 7f4936ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4938104900 0x7f4938194eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:38:45.640 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:45.636+0000 7f4936ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4938104900 0x7f4938194eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35756/0 (socket says 172.21.15.38:35756) 2023-03-09T14:38:45.641 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:45.636+0000 7f4936ffd700 1 -- 172.21.15.38:0/4066462290 learned_addr learned my addr 172.21.15.38:0/4066462290 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:38:45.641 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:45.636+0000 7f4936ffd700 1 -- 172.21.15.38:0/4066462290 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49280095f0 con 0x7f4938104900 2023-03-09T14:38:45.641 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:45.636+0000 7f4936ffd700 1 --2- 172.21.15.38:0/4066462290 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4938104900 0x7f4938194eb0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f49280092e0 tx=0x7f4928004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:38:45.641 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:45.636+0000 7f492ffff700 1 -- 172.21.15.38:0/4066462290 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f492801d070 con 0x7f4938104900 2023-03-09T14:38:45.641 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:45.636+0000 7f492ffff700 1 -- 172.21.15.38:0/4066462290 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f492800bbb0 con 0x7f4938104900 2023-03-09T14:38:45.641 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:45.636+0000 7f493d5eb700 1 -- 172.21.15.38:0/4066462290 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4938108cf0 con 0x7f4938104900 2023-03-09T14:38:45.641 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:45.636+0000 7f492ffff700 1 -- 172.21.15.38:0/4066462290 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4928003930 con 0x7f4938104900 2023-03-09T14:38:45.642 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:45.636+0000 7f493d5eb700 1 -- 172.21.15.38:0/4066462290 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4938108b60 con 0x7f4938104900 2023-03-09T14:38:45.642 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:45.637+0000 7f492ffff700 1 -- 172.21.15.38:0/4066462290 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 44674+0+0 (secure 0 0 0) 0x7f4928028030 con 0x7f4938104900 2023-03-09T14:38:45.642 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:45.637+0000 7f493d5eb700 1 -- 172.21.15.38:0/4066462290 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4938108e80 con 0x7f4938104900 2023-03-09T14:38:45.642 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:45.637+0000 7f492ffff700 1 --2- 172.21.15.38:0/4066462290 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f4924037e50 0x7f492403a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:38:45.642 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:45.637+0000 7f492ffff700 1 -- 172.21.15.38:0/4066462290 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f492804b2f0 con 0x7f4938104900 2023-03-09T14:38:45.642 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:45.638+0000 7f49367fc700 1 --2- 172.21.15.38:0/4066462290 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f4924037e50 0x7f492403a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:38:45.643 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:45.638+0000 7f49367fc700 1 --2- 172.21.15.38:0/4066462290 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f4924037e50 0x7f492403a300 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f4920006fd0 tx=0x7f4920006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:38:45.644 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:45.642+0000 7f492ffff700 1 -- 172.21.15.38:0/4066462290 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f4928029360 con 0x7f4938104900 2023-03-09T14:38:45.848 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:45.845+0000 7f493d5eb700 1 -- 172.21.15.38:0/4066462290 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f4938196200 con 0x7f4938104900 2023-03-09T14:38:46.099 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:46.097+0000 7f492ffff700 1 -- 172.21.15.38:0/4066462290 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f4928026070 con 0x7f4938104900 2023-03-09T14:38:46.101 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:46.099+0000 7f493d5eb700 1 -- 172.21.15.38:0/4066462290 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f4924037e50 msgr2=0x7f492403a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:38:46.101 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:46.099+0000 7f493d5eb700 1 --2- 172.21.15.38:0/4066462290 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f4924037e50 0x7f492403a300 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f4920006fd0 tx=0x7f4920006e40 comp rx=0 tx=0).stop 2023-03-09T14:38:46.101 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:46.099+0000 7f493d5eb700 1 -- 172.21.15.38:0/4066462290 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4938104900 msgr2=0x7f4938194eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:38:46.102 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:46.099+0000 7f493d5eb700 1 --2- 172.21.15.38:0/4066462290 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4938104900 0x7f4938194eb0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f49280092e0 tx=0x7f4928004ff0 comp rx=0 tx=0).stop 2023-03-09T14:38:46.102 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:46.099+0000 7f493d5eb700 1 -- 172.21.15.38:0/4066462290 shutdown_connections 2023-03-09T14:38:46.102 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:46.099+0000 7f493d5eb700 1 --2- 172.21.15.38:0/4066462290 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f4924037e50 0x7f492403a300 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:38:46.102 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:46.099+0000 7f493d5eb700 1 --2- 172.21.15.38:0/4066462290 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4938104900 0x7f4938194eb0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:38:46.102 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:46.099+0000 7f493d5eb700 1 -- 172.21.15.38:0/4066462290 >> 172.21.15.38:0/4066462290 conn(0x7f49380fa3f0 msgr2=0x7f49380fc300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:38:46.102 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:46.099+0000 7f493d5eb700 1 -- 172.21.15.38:0/4066462290 shutdown_connections 2023-03-09T14:38:46.102 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:38:46.099+0000 7f493d5eb700 1 -- 172.21.15.38:0/4066462290 wait complete. 2023-03-09T14:38:46.103 INFO:teuthology.orchestra.run.smithi038.stderr:adjusted tunables profile to default 2023-03-09T14:38:46.455 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:46 smithi038 ceph-mon[92422]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:38:46.455 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:46 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/4066462290' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-03-09T14:38:46.456 INFO:tasks.cephadm:Adding mon.a on smithi038 2023-03-09T14:38:46.456 INFO:tasks.cephadm:Adding mon.b on smithi122 2023-03-09T14:38:46.456 INFO:tasks.cephadm:Adding mon.c on smithi181 2023-03-09T14:38:46.457 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph orch apply mon '3;smithi038:172.21.15.38=a;smithi122:172.21.15.122=b;smithi181:172.21.15.181=c' 2023-03-09T14:38:47.145 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/config/ceph.conf 2023-03-09T14:38:47.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:47 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/4066462290' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-03-09T14:38:47.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:47 smithi038 ceph-mon[92422]: osdmap e4: 0 total, 0 up, 0 in 2023-03-09T14:38:47.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:47 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:38:47.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:47 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:38:47.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:47 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:38:47.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:47 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:38:47.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:47 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi181", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:38:47.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:47 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:38:47.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:47 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:38:48.337 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:48.335+0000 7f73ff2a0700 1 -- 172.21.15.181:0/861137087 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f73f8104910 msgr2=0x7f73f8104ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:38:48.338 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:48.335+0000 7f73ff2a0700 1 --2- 172.21.15.181:0/861137087 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f73f8104910 0x7f73f8104ce0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f73f4009910 tx=0x7f73f4009c20 comp rx=0 tx=0).stop 2023-03-09T14:38:48.338 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:48.335+0000 7f73ff2a0700 1 -- 172.21.15.181:0/861137087 shutdown_connections 2023-03-09T14:38:48.338 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:48.335+0000 7f73ff2a0700 1 --2- 172.21.15.181:0/861137087 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f73f8104910 0x7f73f8104ce0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:38:48.338 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:48.335+0000 7f73ff2a0700 1 -- 172.21.15.181:0/861137087 >> 172.21.15.181:0/861137087 conn(0x7f73f80fa400 msgr2=0x7f73f80fc810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:38:48.338 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:48.335+0000 7f73ff2a0700 1 -- 172.21.15.181:0/861137087 shutdown_connections 2023-03-09T14:38:48.338 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:48.335+0000 7f73ff2a0700 1 -- 172.21.15.181:0/861137087 wait complete. 2023-03-09T14:38:48.338 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:48.336+0000 7f73ff2a0700 1 Processor -- start 2023-03-09T14:38:48.339 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:48.336+0000 7f73ff2a0700 1 -- start start 2023-03-09T14:38:48.339 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:48.336+0000 7f73ff2a0700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f73f8104910 0x7f73f8195c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:38:48.339 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:48.336+0000 7f73ff2a0700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73f8196170 con 0x7f73f8104910 2023-03-09T14:38:48.340 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:48.336+0000 7f73fd03c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f73f8104910 0x7f73f8195c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:38:48.341 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:48.336+0000 7f73fd03c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f73f8104910 0x7f73f8195c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.181:37498/0 (socket says 172.21.15.181:37498) 2023-03-09T14:38:48.341 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:48.336+0000 7f73fd03c700 1 -- 172.21.15.181:0/689604333 learned_addr learned my addr 172.21.15.181:0/689604333 (peer_addr_for_me v2:172.21.15.181:0/0) 2023-03-09T14:38:48.341 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:48.336+0000 7f73fd03c700 1 -- 172.21.15.181:0/689604333 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73f40095f0 con 0x7f73f8104910 2023-03-09T14:38:48.341 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:48.337+0000 7f73fd03c700 1 --2- 172.21.15.181:0/689604333 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f73f8104910 0x7f73f8195c30 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f73f40092e0 tx=0x7f73f4004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:38:48.341 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:48.337+0000 7f73ee7fc700 1 -- 172.21.15.181:0/689604333 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f73f401d070 con 0x7f73f8104910 2023-03-09T14:38:48.342 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:48.337+0000 7f73ff2a0700 1 -- 172.21.15.181:0/689604333 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73f8108d00 con 0x7f73f8104910 2023-03-09T14:38:48.342 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:48.337+0000 7f73ee7fc700 1 -- 172.21.15.181:0/689604333 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f73f400bbb0 con 0x7f73f8104910 2023-03-09T14:38:48.342 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:48.337+0000 7f73ee7fc700 1 -- 172.21.15.181:0/689604333 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f73f4003930 con 0x7f73f8104910 2023-03-09T14:38:48.342 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:48.337+0000 7f73ff2a0700 1 -- 172.21.15.181:0/689604333 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73f8108b70 con 0x7f73f8104910 2023-03-09T14:38:48.342 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:48.338+0000 7f73ff2a0700 1 -- 172.21.15.181:0/689604333 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73f8108e90 con 0x7f73f8104910 2023-03-09T14:38:48.343 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:48.338+0000 7f73ee7fc700 1 -- 172.21.15.181:0/689604333 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 44674+0+0 (secure 0 0 0) 0x7f73f4028030 con 0x7f73f8104910 2023-03-09T14:38:48.343 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:48.338+0000 7f73ee7fc700 1 --2- 172.21.15.181:0/689604333 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f73e4037e00 0x7f73e403a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:38:48.343 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:48.338+0000 7f73ee7fc700 1 -- 172.21.15.181:0/689604333 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f73f404b370 con 0x7f73f8104910 2023-03-09T14:38:48.343 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:48.338+0000 7f73fc83b700 1 --2- 172.21.15.181:0/689604333 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f73e4037e00 0x7f73e403a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:38:48.343 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:48.339+0000 7f73fc83b700 1 --2- 172.21.15.181:0/689604333 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f73e4037e00 0x7f73e403a2b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f73e8006fd0 tx=0x7f73e8006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:38:48.345 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:48.343+0000 7f73ee7fc700 1 -- 172.21.15.181:0/689604333 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f73f4029360 con 0x7f73f8104910 2023-03-09T14:38:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:48 smithi038 ceph-mon[92422]: Updating smithi181:/etc/ceph/ceph.conf 2023-03-09T14:38:48.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:48 smithi038 ceph-mon[92422]: Updating smithi181:/var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/config/ceph.conf 2023-03-09T14:38:48.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:48 smithi038 ceph-mon[92422]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:38:48.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:48 smithi038 ceph-mon[92422]: Updating smithi181:/etc/ceph/ceph.client.admin.keyring 2023-03-09T14:38:48.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:48 smithi038 ceph-mon[92422]: Updating smithi181:/var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/config/ceph.client.admin.keyring 2023-03-09T14:38:48.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:48 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:38:48.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:48 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:38:48.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:48 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:38:48.538 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:48.536+0000 7f73ff2a0700 1 -- 172.21.15.181:0/689604333 --> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi038:172.21.15.38=a;smithi122:172.21.15.122=b;smithi181:172.21.15.181=c", "target": ["mon-mgr", ""]}) v1 -- 0x7f73f8060bc0 con 0x7f73e4037e00 2023-03-09T14:38:48.551 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:48.550+0000 7f73ee7fc700 1 -- 172.21.15.181:0/689604333 <== mgr.14150 v2:172.21.15.38:6800/3328887328 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f73f8060bc0 con 0x7f73e4037e00 2023-03-09T14:38:48.552 INFO:teuthology.orchestra.run.smithi181.stdout:Scheduled mon update... 2023-03-09T14:38:48.553 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:48.552+0000 7f73ff2a0700 1 -- 172.21.15.181:0/689604333 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f73e4037e00 msgr2=0x7f73e403a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:38:48.554 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:48.552+0000 7f73ff2a0700 1 --2- 172.21.15.181:0/689604333 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f73e4037e00 0x7f73e403a2b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f73e8006fd0 tx=0x7f73e8006e40 comp rx=0 tx=0).stop 2023-03-09T14:38:48.554 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:48.552+0000 7f73ff2a0700 1 -- 172.21.15.181:0/689604333 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f73f8104910 msgr2=0x7f73f8195c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:38:48.554 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:48.552+0000 7f73ff2a0700 1 --2- 172.21.15.181:0/689604333 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f73f8104910 0x7f73f8195c30 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f73f40092e0 tx=0x7f73f4004ff0 comp rx=0 tx=0).stop 2023-03-09T14:38:48.554 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:48.552+0000 7f73ff2a0700 1 -- 172.21.15.181:0/689604333 shutdown_connections 2023-03-09T14:38:48.554 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:48.552+0000 7f73ff2a0700 1 --2- 172.21.15.181:0/689604333 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f73e4037e00 0x7f73e403a2b0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:38:48.554 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:48.552+0000 7f73ff2a0700 1 --2- 172.21.15.181:0/689604333 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f73f8104910 0x7f73f8195c30 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:38:48.555 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:48.552+0000 7f73ff2a0700 1 -- 172.21.15.181:0/689604333 >> 172.21.15.181:0/689604333 conn(0x7f73f80fa400 msgr2=0x7f73f80fb060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:38:48.555 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:48.552+0000 7f73ff2a0700 1 -- 172.21.15.181:0/689604333 shutdown_connections 2023-03-09T14:38:48.555 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:48.552+0000 7f73ff2a0700 1 -- 172.21.15.181:0/689604333 wait complete. 2023-03-09T14:38:48.912 DEBUG:teuthology.orchestra.run.smithi122:mon.b> sudo journalctl -f -n 0 -u ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@mon.b.service 2023-03-09T14:38:48.915 DEBUG:teuthology.orchestra.run.smithi181:mon.c> sudo journalctl -f -n 0 -u ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@mon.c.service 2023-03-09T14:38:48.918 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-03-09T14:38:48.918 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph mon dump -f json 2023-03-09T14:38:48.946 INFO:journalctl@ceph.mon.c.smithi181.stdout:-- Logs begin at Thu 2023-03-09 14:25:01 UTC. -- 2023-03-09T14:38:48.952 INFO:journalctl@ceph.mon.b.smithi122.stdout:-- Logs begin at Thu 2023-03-09 14:25:09 UTC. -- 2023-03-09T14:38:49.826 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/config/ceph.conf 2023-03-09T14:38:50.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:49 smithi038 ceph-mon[92422]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi038:172.21.15.38=a;smithi122:172.21.15.122=b;smithi181:172.21.15.181=c", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:38:50.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:49 smithi038 ceph-mon[92422]: Saving service mon spec with placement smithi038:172.21.15.38=a;smithi122:172.21.15.122=b;smithi181:172.21.15.181=c;count:3 2023-03-09T14:38:50.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:49 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:38:50.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:49 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:38:50.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:49 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:38:50.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:49 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:38:50.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:49 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:38:50.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:49 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-03-09T14:38:50.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:49 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:38:50.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:49 smithi038 ceph-mon[92422]: Deploying daemon mon.c on smithi181 2023-03-09T14:38:50.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:49 smithi038 ceph-mon[92422]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:38:51.126 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:51.124+0000 7f89465c9700 1 -- 172.21.15.181:0/724105734 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8940104910 msgr2=0x7f8940104ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:38:51.126 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:51.124+0000 7f89465c9700 1 --2- 172.21.15.181:0/724105734 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8940104910 0x7f8940104ce0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8930009910 tx=0x7f8930009c20 comp rx=0 tx=0).stop 2023-03-09T14:38:51.127 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:51.124+0000 7f89465c9700 1 -- 172.21.15.181:0/724105734 shutdown_connections 2023-03-09T14:38:51.127 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:51.124+0000 7f89465c9700 1 --2- 172.21.15.181:0/724105734 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8940104910 0x7f8940104ce0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:38:51.127 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:51.124+0000 7f89465c9700 1 -- 172.21.15.181:0/724105734 >> 172.21.15.181:0/724105734 conn(0x7f89400745e0 msgr2=0x7f89400749e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:38:51.127 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:51.124+0000 7f89465c9700 1 -- 172.21.15.181:0/724105734 shutdown_connections 2023-03-09T14:38:51.127 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:51.124+0000 7f89465c9700 1 -- 172.21.15.181:0/724105734 wait complete. 2023-03-09T14:38:51.127 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:51.124+0000 7f89465c9700 1 Processor -- start 2023-03-09T14:38:51.127 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:51.124+0000 7f89465c9700 1 -- start start 2023-03-09T14:38:51.129 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:51.125+0000 7f89465c9700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8940104910 0x7f8940194ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:38:51.129 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:51.125+0000 7f89465c9700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8940195410 con 0x7f8940104910 2023-03-09T14:38:51.129 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:51.125+0000 7f893ffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8940104910 0x7f8940194ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:38:51.129 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:51.125+0000 7f893ffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8940104910 0x7f8940194ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.181:37504/0 (socket says 172.21.15.181:37504) 2023-03-09T14:38:51.129 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:51.125+0000 7f893ffff700 1 -- 172.21.15.181:0/280638502 learned_addr learned my addr 172.21.15.181:0/280638502 (peer_addr_for_me v2:172.21.15.181:0/0) 2023-03-09T14:38:51.130 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:51.125+0000 7f893ffff700 1 -- 172.21.15.181:0/280638502 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89300095f0 con 0x7f8940104910 2023-03-09T14:38:51.130 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:51.125+0000 7f893ffff700 1 --2- 172.21.15.181:0/280638502 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8940104910 0x7f8940194ed0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f89300092e0 tx=0x7f8930004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:38:51.130 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:51.126+0000 7f893d7fa700 1 -- 172.21.15.181:0/280638502 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f893001d070 con 0x7f8940104910 2023-03-09T14:38:51.130 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:51.126+0000 7f893d7fa700 1 -- 172.21.15.181:0/280638502 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f893000bbb0 con 0x7f8940104910 2023-03-09T14:38:51.130 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:51.126+0000 7f89465c9700 1 -- 172.21.15.181:0/280638502 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8940108d00 con 0x7f8940104910 2023-03-09T14:38:51.130 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:51.126+0000 7f893d7fa700 1 -- 172.21.15.181:0/280638502 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8930003930 con 0x7f8940104910 2023-03-09T14:38:51.130 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:51.126+0000 7f89465c9700 1 -- 172.21.15.181:0/280638502 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8940108b70 con 0x7f8940104910 2023-03-09T14:38:51.131 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:51.127+0000 7f893d7fa700 1 -- 172.21.15.181:0/280638502 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 44674+0+0 (secure 0 0 0) 0x7f8930028030 con 0x7f8940104910 2023-03-09T14:38:51.131 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:51.127+0000 7f89465c9700 1 -- 172.21.15.181:0/280638502 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8940108e90 con 0x7f8940104910 2023-03-09T14:38:51.131 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:51.127+0000 7f893d7fa700 1 --2- 172.21.15.181:0/280638502 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f8928037e00 0x7f892803a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:38:51.131 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:51.127+0000 7f893d7fa700 1 -- 172.21.15.181:0/280638502 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f893004b370 con 0x7f8940104910 2023-03-09T14:38:51.131 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:51.127+0000 7f893f7fe700 1 --2- 172.21.15.181:0/280638502 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f8928037e00 0x7f892803a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:38:51.131 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:51.128+0000 7f893f7fe700 1 --2- 172.21.15.181:0/280638502 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f8928037e00 0x7f892803a2b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8934006fd0 tx=0x7f8934006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:38:51.134 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:51.132+0000 7f893d7fa700 1 -- 172.21.15.181:0/280638502 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f8930029360 con 0x7f8940104910 2023-03-09T14:38:51.446 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:51.443+0000 7f89465c9700 1 -- 172.21.15.181:0/280638502 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8940196220 con 0x7f8940104910 2023-03-09T14:38:51.447 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:51.443+0000 7f893d7fa700 1 -- 172.21.15.181:0/280638502 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+736 (secure 0 0 0) 0x7f8930026070 con 0x7f8940104910 2023-03-09T14:38:51.447 INFO:teuthology.orchestra.run.smithi181.stdout: 2023-03-09T14:38:51.447 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":1,"fsid":"c6733fe2-be87-11ed-9af2-001a4aab830c","modified":"2023-03-09T14:37:12.753639Z","created":"2023-03-09T14:37:12.753639Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-03-09T14:38:51.448 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:51.446+0000 7f89465c9700 1 -- 172.21.15.181:0/280638502 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f8928037e00 msgr2=0x7f892803a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:38:51.448 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:51.446+0000 7f89465c9700 1 --2- 172.21.15.181:0/280638502 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f8928037e00 0x7f892803a2b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f8934006fd0 tx=0x7f8934006e40 comp rx=0 tx=0).stop 2023-03-09T14:38:51.448 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:51.446+0000 7f89465c9700 1 -- 172.21.15.181:0/280638502 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8940104910 msgr2=0x7f8940194ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:38:51.449 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:51.446+0000 7f89465c9700 1 --2- 172.21.15.181:0/280638502 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8940104910 0x7f8940194ed0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f89300092e0 tx=0x7f8930004ff0 comp rx=0 tx=0).stop 2023-03-09T14:38:51.449 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:51.446+0000 7f89465c9700 1 -- 172.21.15.181:0/280638502 shutdown_connections 2023-03-09T14:38:51.449 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:51.446+0000 7f89465c9700 1 --2- 172.21.15.181:0/280638502 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f8928037e00 0x7f892803a2b0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:38:51.449 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:51.446+0000 7f89465c9700 1 --2- 172.21.15.181:0/280638502 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8940104910 0x7f8940194ed0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:38:51.449 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:51.446+0000 7f89465c9700 1 -- 172.21.15.181:0/280638502 >> 172.21.15.181:0/280638502 conn(0x7f89400745e0 msgr2=0x7f89400681f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:38:51.449 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:51.446+0000 7f89465c9700 1 -- 172.21.15.181:0/280638502 shutdown_connections 2023-03-09T14:38:51.449 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:51.446+0000 7f89465c9700 1 -- 172.21.15.181:0/280638502 wait complete. 2023-03-09T14:38:51.449 INFO:teuthology.orchestra.run.smithi181.stderr:dumped monmap epoch 1 2023-03-09T14:38:51.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:51 smithi181 systemd[1]: Starting Ceph mon.c for c6733fe2-be87-11ed-9af2-001a4aab830c... 2023-03-09T14:38:52.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:51 smithi038 ceph-mon[92422]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:38:52.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:51 smithi038 ceph-mon[92422]: from='client.? 172.21.15.181:0/280638502' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-03-09T14:38:52.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 podman[94149]: 2023-03-09T14:38:52.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 bash[94149]: a3624488422eee2890f2d56525aac86d34e4e5739ddbf6a12dab284bbea9132d 2023-03-09T14:38:52.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: set uid:gid to 167:167 (ceph:ceph) 2023-03-09T14:38:52.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev), process ceph-mon, pid 7 2023-03-09T14:38:52.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: pidfile_write: ignore empty --pid-file 2023-03-09T14:38:52.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: load: jerasure load: lrc 2023-03-09T14:38:52.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: RocksDB version: 7.9.2 2023-03-09T14:38:52.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Git sha 0 2023-03-09T14:38:52.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Compile date 2023-03-08 17:02:48 2023-03-09T14:38:52.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: DB SUMMARY 2023-03-09T14:38:52.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: DB Session ID: IPBWD6V8ET23EA8VI8W3 2023-03-09T14:38:52.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: CURRENT file: CURRENT 2023-03-09T14:38:52.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: IDENTITY file: IDENTITY 2023-03-09T14:38:52.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2023-03-09T14:38:52.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2023-03-09T14:38:52.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2023-03-09T14:38:52.485 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.error_if_exists: 0 2023-03-09T14:38:52.485 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.create_if_missing: 0 2023-03-09T14:38:52.485 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.paranoid_checks: 1 2023-03-09T14:38:52.485 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.flush_verify_memtable_count: 1 2023-03-09T14:38:52.485 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-03-09T14:38:52.485 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-03-09T14:38:52.486 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.env: 0x5647fa427720 2023-03-09T14:38:52.486 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.fs: PosixFileSystem 2023-03-09T14:38:52.486 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.info_log: 0x5647fae9f540 2023-03-09T14:38:52.486 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.max_file_opening_threads: 16 2023-03-09T14:38:52.486 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.statistics: (nil) 2023-03-09T14:38:52.486 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.use_fsync: 0 2023-03-09T14:38:52.486 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.max_log_file_size: 0 2023-03-09T14:38:52.487 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-03-09T14:38:52.487 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.log_file_time_to_roll: 0 2023-03-09T14:38:52.487 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.keep_log_file_num: 1000 2023-03-09T14:38:52.487 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.recycle_log_file_num: 0 2023-03-09T14:38:52.487 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.allow_fallocate: 1 2023-03-09T14:38:52.487 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.allow_mmap_reads: 0 2023-03-09T14:38:52.487 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.allow_mmap_writes: 0 2023-03-09T14:38:52.488 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.use_direct_reads: 0 2023-03-09T14:38:52.488 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-03-09T14:38:52.488 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.create_missing_column_families: 0 2023-03-09T14:38:52.488 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.db_log_dir: 2023-03-09T14:38:52.488 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.wal_dir: 2023-03-09T14:38:52.488 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.table_cache_numshardbits: 6 2023-03-09T14:38:52.488 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.WAL_ttl_seconds: 0 2023-03-09T14:38:52.489 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.WAL_size_limit_MB: 0 2023-03-09T14:38:52.489 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-03-09T14:38:52.489 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-03-09T14:38:52.489 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.is_fd_close_on_exec: 1 2023-03-09T14:38:52.489 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.advise_random_on_open: 1 2023-03-09T14:38:52.489 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.db_write_buffer_size: 0 2023-03-09T14:38:52.490 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.write_buffer_manager: 0x5647fae765a0 2023-03-09T14:38:52.490 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-03-09T14:38:52.490 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-03-09T14:38:52.490 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.use_adaptive_mutex: 0 2023-03-09T14:38:52.490 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.rate_limiter: (nil) 2023-03-09T14:38:52.490 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-03-09T14:38:52.490 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.wal_recovery_mode: 2 2023-03-09T14:38:52.491 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.enable_thread_tracking: 0 2023-03-09T14:38:52.491 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.enable_pipelined_write: 0 2023-03-09T14:38:52.491 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.unordered_write: 0 2023-03-09T14:38:52.491 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-03-09T14:38:52.491 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-03-09T14:38:52.491 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-03-09T14:38:52.491 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-03-09T14:38:52.491 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.row_cache: None 2023-03-09T14:38:52.492 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.wal_filter: None 2023-03-09T14:38:52.492 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-03-09T14:38:52.492 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.allow_ingest_behind: 0 2023-03-09T14:38:52.492 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.two_write_queues: 0 2023-03-09T14:38:52.492 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.manual_wal_flush: 0 2023-03-09T14:38:52.492 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.wal_compression: 0 2023-03-09T14:38:52.492 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.atomic_flush: 0 2023-03-09T14:38:52.493 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-03-09T14:38:52.493 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.persist_stats_to_disk: 0 2023-03-09T14:38:52.493 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.write_dbid_to_manifest: 0 2023-03-09T14:38:52.493 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.log_readahead_size: 0 2023-03-09T14:38:52.493 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-03-09T14:38:52.493 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.best_efforts_recovery: 0 2023-03-09T14:38:52.493 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-03-09T14:38:52.494 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-03-09T14:38:52.494 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.allow_data_in_errors: 0 2023-03-09T14:38:52.495 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.db_host_id: __hostname__ 2023-03-09T14:38:52.495 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.enforce_single_del_contracts: true 2023-03-09T14:38:52.496 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.max_background_jobs: 2 2023-03-09T14:38:52.496 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.max_background_compactions: -1 2023-03-09T14:38:52.496 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.max_subcompactions: 1 2023-03-09T14:38:52.496 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-03-09T14:38:52.496 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-03-09T14:38:52.496 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.delayed_write_rate : 16777216 2023-03-09T14:38:52.496 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.max_total_wal_size: 0 2023-03-09T14:38:52.497 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-03-09T14:38:52.497 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.stats_dump_period_sec: 600 2023-03-09T14:38:52.497 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.stats_persist_period_sec: 600 2023-03-09T14:38:52.497 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-03-09T14:38:52.497 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.max_open_files: -1 2023-03-09T14:38:52.497 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.bytes_per_sync: 0 2023-03-09T14:38:52.497 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.wal_bytes_per_sync: 0 2023-03-09T14:38:52.498 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.strict_bytes_per_sync: 0 2023-03-09T14:38:52.498 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.compaction_readahead_size: 0 2023-03-09T14:38:52.498 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.max_background_flushes: -1 2023-03-09T14:38:52.498 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Compression algorithms supported: 2023-03-09T14:38:52.498 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-03-09T14:38:52.498 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: kZSTD supported: 0 2023-03-09T14:38:52.498 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: kXpressCompression supported: 0 2023-03-09T14:38:52.499 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: kLZ4HCCompression supported: 1 2023-03-09T14:38:52.499 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: kZlibCompression supported: 1 2023-03-09T14:38:52.499 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: kSnappyCompression supported: 1 2023-03-09T14:38:52.499 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: kLZ4Compression supported: 1 2023-03-09T14:38:52.499 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: kBZip2Compression supported: 0 2023-03-09T14:38:52.499 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Fast CRC32 supported: Supported on x86 2023-03-09T14:38:52.499 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: DMutex implementation: pthread_mutex_t 2023-03-09T14:38:52.500 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2023-03-09T14:38:52.500 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-03-09T14:38:52.500 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-03-09T14:38:52.500 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.merge_operator: 2023-03-09T14:38:52.500 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.compaction_filter: None 2023-03-09T14:38:52.500 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.compaction_filter_factory: None 2023-03-09T14:38:52.500 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.sst_partitioner_factory: None 2023-03-09T14:38:52.500 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.memtable_factory: SkipListFactory 2023-03-09T14:38:52.501 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.table_factory: BlockBasedTable 2023-03-09T14:38:52.501 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5647fae9f640) 2023-03-09T14:38:52.501 INFO:journalctl@ceph.mon.c.smithi181.stdout: cache_index_and_filter_blocks: 1 2023-03-09T14:38:52.501 INFO:journalctl@ceph.mon.c.smithi181.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-03-09T14:38:52.501 INFO:journalctl@ceph.mon.c.smithi181.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-03-09T14:38:52.501 INFO:journalctl@ceph.mon.c.smithi181.stdout: pin_top_level_index_and_filter: 1 2023-03-09T14:38:52.501 INFO:journalctl@ceph.mon.c.smithi181.stdout: index_type: 0 2023-03-09T14:38:52.502 INFO:journalctl@ceph.mon.c.smithi181.stdout: data_block_index_type: 0 2023-03-09T14:38:52.502 INFO:journalctl@ceph.mon.c.smithi181.stdout: index_shortening: 1 2023-03-09T14:38:52.502 INFO:journalctl@ceph.mon.c.smithi181.stdout: data_block_hash_table_util_ratio: 0.750000 2023-03-09T14:38:52.502 INFO:journalctl@ceph.mon.c.smithi181.stdout: checksum: 4 2023-03-09T14:38:52.502 INFO:journalctl@ceph.mon.c.smithi181.stdout: no_block_cache: 0 2023-03-09T14:38:52.502 INFO:journalctl@ceph.mon.c.smithi181.stdout: block_cache: 0x5647faef71f0 2023-03-09T14:38:52.502 INFO:journalctl@ceph.mon.c.smithi181.stdout: block_cache_name: BinnedLRUCache 2023-03-09T14:38:52.503 INFO:journalctl@ceph.mon.c.smithi181.stdout: block_cache_options: 2023-03-09T14:38:52.503 INFO:journalctl@ceph.mon.c.smithi181.stdout: capacity : 536870912 2023-03-09T14:38:52.503 INFO:journalctl@ceph.mon.c.smithi181.stdout: num_shard_bits : 4 2023-03-09T14:38:52.503 INFO:journalctl@ceph.mon.c.smithi181.stdout: strict_capacity_limit : 0 2023-03-09T14:38:52.503 INFO:journalctl@ceph.mon.c.smithi181.stdout: high_pri_pool_ratio: 0.000 2023-03-09T14:38:52.503 INFO:journalctl@ceph.mon.c.smithi181.stdout: block_cache_compressed: (nil) 2023-03-09T14:38:52.504 INFO:journalctl@ceph.mon.c.smithi181.stdout: persistent_cache: (nil) 2023-03-09T14:38:52.504 INFO:journalctl@ceph.mon.c.smithi181.stdout: block_size: 4096 2023-03-09T14:38:52.504 INFO:journalctl@ceph.mon.c.smithi181.stdout: block_size_deviation: 10 2023-03-09T14:38:52.504 INFO:journalctl@ceph.mon.c.smithi181.stdout: block_restart_interval: 16 2023-03-09T14:38:52.504 INFO:journalctl@ceph.mon.c.smithi181.stdout: index_block_restart_interval: 1 2023-03-09T14:38:52.504 INFO:journalctl@ceph.mon.c.smithi181.stdout: metadata_block_size: 4096 2023-03-09T14:38:52.504 INFO:journalctl@ceph.mon.c.smithi181.stdout: partition_filters: 0 2023-03-09T14:38:52.504 INFO:journalctl@ceph.mon.c.smithi181.stdout: use_delta_encoding: 1 2023-03-09T14:38:52.505 INFO:journalctl@ceph.mon.c.smithi181.stdout: filter_policy: bloomfilter 2023-03-09T14:38:52.505 INFO:journalctl@ceph.mon.c.smithi181.stdout: whole_key_filtering: 1 2023-03-09T14:38:52.505 INFO:journalctl@ceph.mon.c.smithi181.stdout: verify_compression: 0 2023-03-09T14:38:52.505 INFO:journalctl@ceph.mon.c.smithi181.stdout: read_amp_bytes_per_bit: 0 2023-03-09T14:38:52.505 INFO:journalctl@ceph.mon.c.smithi181.stdout: format_version: 5 2023-03-09T14:38:52.505 INFO:journalctl@ceph.mon.c.smithi181.stdout: enable_index_compression: 1 2023-03-09T14:38:52.505 INFO:journalctl@ceph.mon.c.smithi181.stdout: block_align: 0 2023-03-09T14:38:52.506 INFO:journalctl@ceph.mon.c.smithi181.stdout: max_auto_readahead_size: 262144 2023-03-09T14:38:52.506 INFO:journalctl@ceph.mon.c.smithi181.stdout: prepopulate_block_cache: 0 2023-03-09T14:38:52.506 INFO:journalctl@ceph.mon.c.smithi181.stdout: initial_auto_readahead_size: 8192 2023-03-09T14:38:52.506 INFO:journalctl@ceph.mon.c.smithi181.stdout: num_file_reads_for_auto_readahead: 2 2023-03-09T14:38:52.506 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.write_buffer_size: 33554432 2023-03-09T14:38:52.506 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.max_write_buffer_number: 2 2023-03-09T14:38:52.506 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.compression: NoCompression 2023-03-09T14:38:52.507 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.bottommost_compression: Disabled 2023-03-09T14:38:52.507 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.prefix_extractor: nullptr 2023-03-09T14:38:52.507 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-03-09T14:38:52.507 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.num_levels: 7 2023-03-09T14:38:52.507 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-03-09T14:38:52.507 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-03-09T14:38:52.507 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-03-09T14:38:52.508 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-03-09T14:38:52.508 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-03-09T14:38:52.508 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-03-09T14:38:52.508 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-03-09T14:38:52.508 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-03-09T14:38:52.508 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-03-09T14:38:52.508 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-03-09T14:38:52.509 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-03-09T14:38:52.509 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-03-09T14:38:52.509 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.compression_opts.window_bits: -14 2023-03-09T14:38:52.509 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.compression_opts.level: 32767 2023-03-09T14:38:52.509 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.compression_opts.strategy: 0 2023-03-09T14:38:52.509 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-03-09T14:38:52.509 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-03-09T14:38:52.510 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-03-09T14:38:52.510 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-03-09T14:38:52.510 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.compression_opts.enabled: false 2023-03-09T14:38:52.510 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-03-09T14:38:52.510 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-03-09T14:38:52.510 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-03-09T14:38:52.510 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-03-09T14:38:52.510 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.target_file_size_base: 67108864 2023-03-09T14:38:52.511 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.target_file_size_multiplier: 1 2023-03-09T14:38:52.511 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-03-09T14:38:52.511 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-03-09T14:38:52.511 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-03-09T14:38:52.511 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-03-09T14:38:52.511 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-03-09T14:38:52.511 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-03-09T14:38:52.512 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-03-09T14:38:52.512 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-03-09T14:38:52.512 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-03-09T14:38:52.512 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-03-09T14:38:52.512 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-03-09T14:38:52.512 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-03-09T14:38:52.512 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-03-09T14:38:52.513 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.arena_block_size: 1048576 2023-03-09T14:38:52.513 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-03-09T14:38:52.513 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-03-09T14:38:52.513 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.disable_auto_compactions: 0 2023-03-09T14:38:52.513 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-03-09T14:38:52.513 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-03-09T14:38:52.513 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-03-09T14:38:52.514 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-03-09T14:38:52.514 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-03-09T14:38:52.514 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-03-09T14:38:52.514 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-03-09T14:38:52.514 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-03-09T14:38:52.514 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-03-09T14:38:52.514 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-03-09T14:38:52.515 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.table_properties_collectors: 2023-03-09T14:38:52.515 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.inplace_update_support: 0 2023-03-09T14:38:52.515 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.inplace_update_num_locks: 10000 2023-03-09T14:38:52.515 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-03-09T14:38:52.515 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-03-09T14:38:52.515 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.memtable_huge_page_size: 0 2023-03-09T14:38:52.515 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.bloom_locality: 0 2023-03-09T14:38:52.515 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.max_successive_merges: 0 2023-03-09T14:38:52.516 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.optimize_filters_for_hits: 0 2023-03-09T14:38:52.516 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.paranoid_file_checks: 0 2023-03-09T14:38:52.516 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.force_consistency_checks: 1 2023-03-09T14:38:52.516 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.report_bg_io_stats: 0 2023-03-09T14:38:52.516 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.ttl: 2592000 2023-03-09T14:38:52.516 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.periodic_compaction_seconds: 0 2023-03-09T14:38:52.516 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-03-09T14:38:52.517 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-03-09T14:38:52.517 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.enable_blob_files: false 2023-03-09T14:38:52.517 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.min_blob_size: 0 2023-03-09T14:38:52.517 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.blob_file_size: 268435456 2023-03-09T14:38:52.517 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.blob_compression_type: NoCompression 2023-03-09T14:38:52.517 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.enable_blob_garbage_collection: false 2023-03-09T14:38:52.517 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-03-09T14:38:52.518 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-03-09T14:38:52.518 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-03-09T14:38:52.518 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.blob_file_starting_level: 0 2023-03-09T14:38:52.518 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-03-09T14:38:52.518 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-03-09T14:38:52.518 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2023-03-09T14:38:52.518 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 934d1777-994b-4e83-8ca6-db45e39b9da7 2023-03-09T14:38:52.519 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: EVENT_LOG_v1 {"time_micros": 1678372732229527, "job": 1, "event": "recovery_started", "wal_files": [4]} 2023-03-09T14:38:52.519 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2023-03-09T14:38:52.519 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: EVENT_LOG_v1 {"time_micros": 1678372732229960, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1678372732, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "934d1777-994b-4e83-8ca6-db45e39b9da7", "db_session_id": "IPBWD6V8ET23EA8VI8W3", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2023-03-09T14:38:52.519 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: EVENT_LOG_v1 {"time_micros": 1678372732230027, "job": 1, "event": "recovery_finished"} 2023-03-09T14:38:52.519 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2023-03-09T14:38:52.519 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-03-09T14:38:52.519 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5647faf96000 2023-03-09T14:38:52.520 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: DB pointer 0x5647faf84000 2023-03-09T14:38:52.520 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-03-09T14:38:52.520 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-03-09T14:38:52.520 INFO:journalctl@ceph.mon.c.smithi181.stdout: ** DB Stats ** 2023-03-09T14:38:52.520 INFO:journalctl@ceph.mon.c.smithi181.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-03-09T14:38:52.520 INFO:journalctl@ceph.mon.c.smithi181.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-03-09T14:38:52.521 INFO:journalctl@ceph.mon.c.smithi181.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-03-09T14:38:52.521 INFO:journalctl@ceph.mon.c.smithi181.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-03-09T14:38:52.521 INFO:journalctl@ceph.mon.c.smithi181.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-03-09T14:38:52.521 INFO:journalctl@ceph.mon.c.smithi181.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-03-09T14:38:52.521 INFO:journalctl@ceph.mon.c.smithi181.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-03-09T14:38:52.521 INFO:journalctl@ceph.mon.c.smithi181.stdout: 2023-03-09T14:38:52.521 INFO:journalctl@ceph.mon.c.smithi181.stdout: ** Compaction Stats [default] ** 2023-03-09T14:38:52.522 INFO:journalctl@ceph.mon.c.smithi181.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-03-09T14:38:52.522 INFO:journalctl@ceph.mon.c.smithi181.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-03-09T14:38:52.522 INFO:journalctl@ceph.mon.c.smithi181.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-03-09T14:38:52.522 INFO:journalctl@ceph.mon.c.smithi181.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-03-09T14:38:52.522 INFO:journalctl@ceph.mon.c.smithi181.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-03-09T14:38:52.522 INFO:journalctl@ceph.mon.c.smithi181.stdout: 2023-03-09T14:38:52.523 INFO:journalctl@ceph.mon.c.smithi181.stdout: ** Compaction Stats [default] ** 2023-03-09T14:38:52.523 INFO:journalctl@ceph.mon.c.smithi181.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-03-09T14:38:52.523 INFO:journalctl@ceph.mon.c.smithi181.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-03-09T14:38:52.523 INFO:journalctl@ceph.mon.c.smithi181.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-03-09T14:38:52.523 INFO:journalctl@ceph.mon.c.smithi181.stdout: 2023-03-09T14:38:52.523 INFO:journalctl@ceph.mon.c.smithi181.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-03-09T14:38:52.523 INFO:journalctl@ceph.mon.c.smithi181.stdout: 2023-03-09T14:38:52.524 INFO:journalctl@ceph.mon.c.smithi181.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-03-09T14:38:52.524 INFO:journalctl@ceph.mon.c.smithi181.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-03-09T14:38:52.524 INFO:journalctl@ceph.mon.c.smithi181.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-03-09T14:38:52.524 INFO:journalctl@ceph.mon.c.smithi181.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-03-09T14:38:52.524 INFO:journalctl@ceph.mon.c.smithi181.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-03-09T14:38:52.524 INFO:journalctl@ceph.mon.c.smithi181.stdout: AddFile(Keys): cumulative 0, interval 0 2023-03-09T14:38:52.525 INFO:journalctl@ceph.mon.c.smithi181.stdout: Cumulative compaction: 0.00 GB write, 0.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-03-09T14:38:52.525 INFO:journalctl@ceph.mon.c.smithi181.stdout: Interval compaction: 0.00 GB write, 0.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-03-09T14:38:52.525 INFO:journalctl@ceph.mon.c.smithi181.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-03-09T14:38:52.525 INFO:journalctl@ceph.mon.c.smithi181.stdout: Block cache BinnedLRUCache@0x5647faef71f0#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2023-03-09T14:38:52.525 INFO:journalctl@ceph.mon.c.smithi181.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2023-03-09T14:38:52.525 INFO:journalctl@ceph.mon.c.smithi181.stdout: 2023-03-09T14:38:52.525 INFO:journalctl@ceph.mon.c.smithi181.stdout: ** File Read Latency Histogram By Level [default] ** 2023-03-09T14:38:52.526 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: mon.c does not exist in monmap, will attempt to join an existing cluster 2023-03-09T14:38:52.526 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: using public_addr v2:172.21.15.181:0/0 -> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2023-03-09T14:38:52.526 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: starting mon.c rank -1 at public addrs [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] at bind addrs [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid c6733fe2-be87-11ed-9af2-001a4aab830c 2023-03-09T14:38:52.526 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: mon.c@-1(???) e0 preinit fsid c6733fe2-be87-11ed-9af2-001a4aab830c 2023-03-09T14:38:52.526 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 systemd[1]: Started Ceph mon.c for c6733fe2-be87-11ed-9af2-001a4aab830c. 2023-03-09T14:38:52.526 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: mon.c@-1(synchronizing).mds e1 new map 2023-03-09T14:38:52.526 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: mon.c@-1(synchronizing).mds e1 print_map 2023-03-09T14:38:52.527 INFO:journalctl@ceph.mon.c.smithi181.stdout: e1 2023-03-09T14:38:52.527 INFO:journalctl@ceph.mon.c.smithi181.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-03-09T14:38:52.527 INFO:journalctl@ceph.mon.c.smithi181.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-03-09T14:38:52.527 INFO:journalctl@ceph.mon.c.smithi181.stdout: legacy client fscid: -1 2023-03-09T14:38:52.527 INFO:journalctl@ceph.mon.c.smithi181.stdout: 2023-03-09T14:38:52.527 INFO:journalctl@ceph.mon.c.smithi181.stdout: No filesystems configured 2023-03-09T14:38:52.528 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2023-03-09T14:38:52.528 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2023-03-09T14:38:52.528 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2023-03-09T14:38:52.528 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2023-03-09T14:38:52.528 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2023-03-09T14:38:52.528 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2023-03-09T14:38:52.528 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2023-03-09T14:38:52.529 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-03-09T14:38:52.529 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-03-09T14:38:52.529 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-03-09T14:38:52.529 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:38:52.529 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi181", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:38:52.529 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:38:52.529 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:38:52.530 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:38:52.530 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:38:52.530 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi122", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:38:52.530 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:38:52.530 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:38:52.530 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: Updating smithi122:/etc/ceph/ceph.conf 2023-03-09T14:38:52.530 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: Updating smithi122:/var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/config/ceph.conf 2023-03-09T14:38:52.531 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:38:52.531 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:38:52.531 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:38:52.531 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: Deploying cephadm binary to smithi181 2023-03-09T14:38:52.531 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2023-03-09T14:38:52.531 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: Updating smithi122:/var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/config/ceph.client.admin.keyring 2023-03-09T14:38:52.531 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:38:52.532 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:38:52.532 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: Added host smithi181 2023-03-09T14:38:52.532 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:38:52.532 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:38:52.532 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:38:52.532 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:38:52.532 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:38:52.533 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-03-09T14:38:52.533 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:38:52.533 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/4066462290' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-03-09T14:38:52.533 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/4066462290' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-03-09T14:38:52.533 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: osdmap e4: 0 total, 0 up, 0 in 2023-03-09T14:38:52.533 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:38:52.533 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:38:52.534 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:38:52.534 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:38:52.534 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi181", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:38:52.534 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:38:52.534 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:38:52.534 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: Updating smithi181:/etc/ceph/ceph.conf 2023-03-09T14:38:52.534 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: Updating smithi181:/var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/config/ceph.conf 2023-03-09T14:38:52.535 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:38:52.535 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: Updating smithi181:/etc/ceph/ceph.client.admin.keyring 2023-03-09T14:38:52.535 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: Updating smithi181:/var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/config/ceph.client.admin.keyring 2023-03-09T14:38:52.535 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:38:52.535 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:38:52.535 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:38:52.535 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi038:172.21.15.38=a;smithi122:172.21.15.122=b;smithi181:172.21.15.181=c", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:38:52.536 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: Saving service mon spec with placement smithi038:172.21.15.38=a;smithi122:172.21.15.122=b;smithi181:172.21.15.181=c;count:3 2023-03-09T14:38:52.536 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:38:52.536 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:38:52.536 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:38:52.536 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:38:52.536 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:38:52.536 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-03-09T14:38:52.537 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:38:52.537 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: Deploying daemon mon.c on smithi181 2023-03-09T14:38:52.537 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:38:52.537 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:38:52.537 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: from='client.? 172.21.15.181:0/280638502' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-03-09T14:38:52.537 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2023-03-09T14:38:52.537 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: expand_channel_meta expand map: {default=false} 2023-03-09T14:38:52.538 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: expand_channel_meta from 'false' to 'false' 2023-03-09T14:38:52.538 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: expand_channel_meta expanded map: {default=false} 2023-03-09T14:38:52.538 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: expand_channel_meta expand map: {default=info} 2023-03-09T14:38:52.538 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: expand_channel_meta from 'info' to 'info' 2023-03-09T14:38:52.538 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: expand_channel_meta expanded map: {default=info} 2023-03-09T14:38:52.538 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: expand_channel_meta expand map: {default=daemon} 2023-03-09T14:38:52.538 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: expand_channel_meta from 'daemon' to 'daemon' 2023-03-09T14:38:52.539 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: expand_channel_meta expanded map: {default=daemon} 2023-03-09T14:38:52.539 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: expand_channel_meta expand map: {default=debug} 2023-03-09T14:38:52.539 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: expand_channel_meta from 'debug' to 'debug' 2023-03-09T14:38:52.539 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: expand_channel_meta expanded map: {default=debug} 2023-03-09T14:38:52.539 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:52 smithi181 ceph-mon[94189]: mon.c@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete 2023-03-09T14:38:52.810 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-03-09T14:38:52.810 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph mon dump -f json 2023-03-09T14:38:53.450 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.c/config 2023-03-09T14:38:57.648 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:57.645+0000 7f95dae9d700 1 -- 172.21.15.181:0/813570306 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f95c40035c0 msgr2=0x7f95c4005a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:38:57.649 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:57.645+0000 7f95dae9d700 1 --2- 172.21.15.181:0/813570306 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f95c40035c0 0x7f95c4005a40 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f95d40ffeb0 tx=0x7f95c800fe60 comp rx=0 tx=0).stop 2023-03-09T14:38:57.649 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:57.645+0000 7f95dae9d700 1 -- 172.21.15.181:0/813570306 shutdown_connections 2023-03-09T14:38:57.649 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:57.645+0000 7f95dae9d700 1 --2- 172.21.15.181:0/813570306 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f95c40035c0 0x7f95c4005a40 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:38:57.649 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:57.645+0000 7f95dae9d700 1 -- 172.21.15.181:0/813570306 >> 172.21.15.181:0/813570306 conn(0x7f95d40fa520 msgr2=0x7f95d40fc930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:38:57.649 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:57.645+0000 7f95dae9d700 1 -- 172.21.15.181:0/813570306 shutdown_connections 2023-03-09T14:38:57.649 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:57.645+0000 7f95dae9d700 1 -- 172.21.15.181:0/813570306 wait complete. 2023-03-09T14:38:57.651 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:57.646+0000 7f95dae9d700 1 Processor -- start 2023-03-09T14:38:57.651 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:57.646+0000 7f95dae9d700 1 -- start start 2023-03-09T14:38:57.651 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:57.646+0000 7f95dae9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f95c40035c0 0x7f95d4197140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:38:57.651 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:57.646+0000 7f95dae9d700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f95d4197680 0x7f95d419bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:38:57.651 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:57.646+0000 7f95dae9d700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95d4197c00 con 0x7f95c40035c0 2023-03-09T14:38:57.652 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:57.646+0000 7f95dae9d700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95d4197d70 con 0x7f95d4197680 2023-03-09T14:38:57.652 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:57.647+0000 7f95d3fff700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f95d4197680 0x7f95d419bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:38:57.652 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:57.647+0000 7f95d3fff700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f95d4197680 0x7f95d419bae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:50438/0 (socket says 172.21.15.181:50438) 2023-03-09T14:38:57.652 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:57.647+0000 7f95d3fff700 1 -- 172.21.15.181:0/3468115796 learned_addr learned my addr 172.21.15.181:0/3468115796 (peer_addr_for_me v2:172.21.15.181:0/0) 2023-03-09T14:38:57.652 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:57.647+0000 7f95d3fff700 1 -- 172.21.15.181:0/3468115796 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f95d4197680 msgr2=0x7f95d419bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2023-03-09T14:38:57.652 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:57.647+0000 7f95d3fff700 1 -- 172.21.15.181:0/3468115796 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f95d4197680 msgr2=0x7f95d419bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-03-09T14:38:57.652 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:57.647+0000 7f95d3fff700 1 --2- 172.21.15.181:0/3468115796 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f95d4197680 0x7f95d419bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-03-09T14:38:57.653 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:57.647+0000 7f95d8c39700 1 --2- 172.21.15.181:0/3468115796 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f95c40035c0 0x7f95d4197140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:38:57.653 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:57.647+0000 7f95d3fff700 1 --2- 172.21.15.181:0/3468115796 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f95d4197680 0x7f95d419bae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-03-09T14:38:57.653 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:57.647+0000 7f95d8c39700 1 -- 172.21.15.181:0/3468115796 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f95d4197680 msgr2=0x7f95d419bae0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-03-09T14:38:57.653 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:57.647+0000 7f95d8c39700 1 --2- 172.21.15.181:0/3468115796 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f95d4197680 0x7f95d419bae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:38:57.653 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:57.647+0000 7f95d8c39700 1 -- 172.21.15.181:0/3468115796 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95c800f8b0 con 0x7f95c40035c0 2023-03-09T14:38:57.653 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:57.648+0000 7f95d8c39700 1 --2- 172.21.15.181:0/3468115796 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f95c40035c0 0x7f95d4197140 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f95d40ffeb0 tx=0x7f95c8013c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:38:57.654 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:57.648+0000 7f95d1ffb700 1 -- 172.21.15.181:0/3468115796 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f95c8023070 con 0x7f95c40035c0 2023-03-09T14:38:57.654 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:57.648+0000 7f95d1ffb700 1 -- 172.21.15.181:0/3468115796 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f95c8004bd0 con 0x7f95c40035c0 2023-03-09T14:38:57.654 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:57.648+0000 7f95dae9d700 1 -- 172.21.15.181:0/3468115796 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95d41068b0 con 0x7f95c40035c0 2023-03-09T14:38:57.654 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:57.648+0000 7f95d1ffb700 1 -- 172.21.15.181:0/3468115796 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f95c8003da0 con 0x7f95c40035c0 2023-03-09T14:38:57.654 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:57.648+0000 7f95dae9d700 1 -- 172.21.15.181:0/3468115796 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95d4106720 con 0x7f95c40035c0 2023-03-09T14:38:57.655 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:57.649+0000 7f95d1ffb700 1 -- 172.21.15.181:0/3468115796 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 44674+0+0 (secure 0 0 0) 0x7f95c802e020 con 0x7f95c40035c0 2023-03-09T14:38:57.655 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:57.649+0000 7f95dae9d700 1 -- 172.21.15.181:0/3468115796 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95d4106a40 con 0x7f95c40035c0 2023-03-09T14:38:57.655 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:57.649+0000 7f95d1ffb700 1 --2- 172.21.15.181:0/3468115796 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f95b8037cb0 0x7f95b803a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:38:57.655 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:57.650+0000 7f95d1ffb700 1 -- 172.21.15.181:0/3468115796 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f95c804c540 con 0x7f95c40035c0 2023-03-09T14:38:57.655 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:57.650+0000 7f95d3fff700 1 --2- 172.21.15.181:0/3468115796 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f95b8037cb0 0x7f95b803a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:38:57.655 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:57.650+0000 7f95d3fff700 1 --2- 172.21.15.181:0/3468115796 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f95b8037cb0 0x7f95b803a160 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f95c0009180 tx=0x7f95c0008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:38:57.657 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:57.655+0000 7f95d1ffb700 1 -- 172.21.15.181:0/3468115796 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f95c801a790 con 0x7f95c40035c0 2023-03-09T14:38:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:57 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-03-09T14:38:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:57 smithi181 ceph-mon[94189]: mon.a calling monitor election 2023-03-09T14:38:57.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:57 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-03-09T14:38:57.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:57 smithi181 ceph-mon[94189]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:38:57.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:57 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-03-09T14:38:57.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:57 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-03-09T14:38:57.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:57 smithi181 ceph-mon[94189]: mon.c calling monitor election 2023-03-09T14:38:57.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:57 smithi181 ceph-mon[94189]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:38:57.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:57 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-03-09T14:38:57.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:57 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-03-09T14:38:57.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:57 smithi181 ceph-mon[94189]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:38:57.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:57 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-03-09T14:38:57.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:57 smithi181 ceph-mon[94189]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-03-09T14:38:57.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:57 smithi181 ceph-mon[94189]: monmap e2: 2 mons at {a=[v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0],c=[v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0]} removed_ranks: {} 2023-03-09T14:38:57.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:57 smithi181 ceph-mon[94189]: fsmap 2023-03-09T14:38:57.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:57 smithi181 ceph-mon[94189]: osdmap e4: 0 total, 0 up, 0 in 2023-03-09T14:38:57.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:57 smithi181 ceph-mon[94189]: mgrmap e13: x(active, since 42s) 2023-03-09T14:38:57.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:57 smithi181 ceph-mon[94189]: overall HEALTH_OK 2023-03-09T14:38:57.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:57 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:38:57.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:57 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:38:57.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:57 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-03-09T14:38:57.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:57 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:38:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:57 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-03-09T14:38:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:57 smithi038 ceph-mon[92422]: mon.a calling monitor election 2023-03-09T14:38:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:57 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-03-09T14:38:57.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:57 smithi038 ceph-mon[92422]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:38:57.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:57 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-03-09T14:38:57.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:57 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-03-09T14:38:57.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:57 smithi038 ceph-mon[92422]: mon.c calling monitor election 2023-03-09T14:38:57.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:57 smithi038 ceph-mon[92422]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:38:57.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:57 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-03-09T14:38:57.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:57 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-03-09T14:38:57.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:57 smithi038 ceph-mon[92422]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:38:57.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:57 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-03-09T14:38:57.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:57 smithi038 ceph-mon[92422]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-03-09T14:38:57.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:57 smithi038 ceph-mon[92422]: monmap e2: 2 mons at {a=[v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0],c=[v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0]} removed_ranks: {} 2023-03-09T14:38:57.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:57 smithi038 ceph-mon[92422]: fsmap 2023-03-09T14:38:57.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:57 smithi038 ceph-mon[92422]: osdmap e4: 0 total, 0 up, 0 in 2023-03-09T14:38:57.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:57 smithi038 ceph-mon[92422]: mgrmap e13: x(active, since 42s) 2023-03-09T14:38:57.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:57 smithi038 ceph-mon[92422]: overall HEALTH_OK 2023-03-09T14:38:57.753 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:57 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:38:57.753 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:57 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:38:57.753 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:57 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-03-09T14:38:57.753 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:57 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:38:57.922 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:57.920+0000 7f95dae9d700 1 -- 172.21.15.181:0/3468115796 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f95d4061db0 con 0x7f95c40035c0 2023-03-09T14:38:57.923 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:57.920+0000 7f95d1ffb700 1 -- 172.21.15.181:0/3468115796 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1006 (secure 0 0 0) 0x7f95c80273c0 con 0x7f95c40035c0 2023-03-09T14:38:57.923 INFO:teuthology.orchestra.run.smithi181.stdout: 2023-03-09T14:38:57.923 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":2,"fsid":"c6733fe2-be87-11ed-9af2-001a4aab830c","modified":"2023-03-09T14:38:52.253980Z","created":"2023-03-09T14:37:12.753639Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2023-03-09T14:38:57.925 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:57.922+0000 7f95dae9d700 1 -- 172.21.15.181:0/3468115796 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f95b8037cb0 msgr2=0x7f95b803a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:38:57.925 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:57.922+0000 7f95dae9d700 1 --2- 172.21.15.181:0/3468115796 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f95b8037cb0 0x7f95b803a160 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f95c0009180 tx=0x7f95c0008040 comp rx=0 tx=0).stop 2023-03-09T14:38:57.926 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:57.922+0000 7f95dae9d700 1 -- 172.21.15.181:0/3468115796 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f95c40035c0 msgr2=0x7f95d4197140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:38:57.926 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:57.922+0000 7f95dae9d700 1 --2- 172.21.15.181:0/3468115796 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f95c40035c0 0x7f95d4197140 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f95d40ffeb0 tx=0x7f95c8013c90 comp rx=0 tx=0).stop 2023-03-09T14:38:57.926 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:57.922+0000 7f95dae9d700 1 -- 172.21.15.181:0/3468115796 shutdown_connections 2023-03-09T14:38:57.926 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:57.922+0000 7f95dae9d700 1 --2- 172.21.15.181:0/3468115796 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f95c40035c0 0x7f95d4197140 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:38:57.926 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:57.922+0000 7f95dae9d700 1 --2- 172.21.15.181:0/3468115796 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f95b8037cb0 0x7f95b803a160 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:38:57.926 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:57.922+0000 7f95dae9d700 1 --2- 172.21.15.181:0/3468115796 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f95d4197680 0x7f95d419bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:38:57.926 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:57.922+0000 7f95dae9d700 1 -- 172.21.15.181:0/3468115796 >> 172.21.15.181:0/3468115796 conn(0x7f95d40fa520 msgr2=0x7f95d4102570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:38:57.927 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:57.922+0000 7f95dae9d700 1 -- 172.21.15.181:0/3468115796 shutdown_connections 2023-03-09T14:38:57.927 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:38:57.922+0000 7f95dae9d700 1 -- 172.21.15.181:0/3468115796 wait complete. 2023-03-09T14:38:57.927 INFO:teuthology.orchestra.run.smithi181.stderr:dumped monmap epoch 2 2023-03-09T14:38:58.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:58 smithi181 ceph-mon[94189]: Deploying daemon mon.b on smithi122 2023-03-09T14:38:58.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:58 smithi181 ceph-mon[94189]: from='client.? 172.21.15.181:0/3468115796' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-03-09T14:38:58.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:58 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-03-09T14:38:58.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:58 smithi038 ceph-mon[92422]: Deploying daemon mon.b on smithi122 2023-03-09T14:38:58.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:58 smithi038 ceph-mon[92422]: from='client.? 172.21.15.181:0/3468115796' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-03-09T14:38:58.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:58 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-03-09T14:38:59.315 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-03-09T14:38:59.315 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph mon dump -f json 2023-03-09T14:38:59.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:38:59 smithi181 ceph-mon[94189]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:38:59.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:38:59 smithi038 ceph-mon[92422]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:38:59.961 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.c/config 2023-03-09T14:39:00.169 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:38:59 smithi122 systemd[1]: Starting Ceph mon.b for c6733fe2-be87-11ed-9af2-001a4aab830c... 2023-03-09T14:39:00.420 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 podman[93829]: 2023-03-09T14:39:00.420 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 bash[93829]: c4b19952a646bb066a6e4401fa74a10b5eba6651e8309f2841691c361b019109 2023-03-09T14:39:00.421 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: set uid:gid to 167:167 (ceph:ceph) 2023-03-09T14:39:00.421 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev), process ceph-mon, pid 7 2023-03-09T14:39:00.421 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: pidfile_write: ignore empty --pid-file 2023-03-09T14:39:00.421 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: load: jerasure load: lrc 2023-03-09T14:39:00.421 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: RocksDB version: 7.9.2 2023-03-09T14:39:00.421 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Git sha 0 2023-03-09T14:39:00.421 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Compile date 2023-03-08 17:02:48 2023-03-09T14:39:00.422 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: DB SUMMARY 2023-03-09T14:39:00.422 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: DB Session ID: WQ5XI6M5TJC0573JTY1Q 2023-03-09T14:39:00.422 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: CURRENT file: CURRENT 2023-03-09T14:39:00.422 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: IDENTITY file: IDENTITY 2023-03-09T14:39:00.422 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2023-03-09T14:39:00.422 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2023-03-09T14:39:00.422 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2023-03-09T14:39:00.423 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.error_if_exists: 0 2023-03-09T14:39:00.423 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.create_if_missing: 0 2023-03-09T14:39:00.423 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.paranoid_checks: 1 2023-03-09T14:39:00.423 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.flush_verify_memtable_count: 1 2023-03-09T14:39:00.423 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-03-09T14:39:00.423 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-03-09T14:39:00.423 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.env: 0x55b4b3446720 2023-03-09T14:39:00.423 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.fs: PosixFileSystem 2023-03-09T14:39:00.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.info_log: 0x55b4b5939540 2023-03-09T14:39:00.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.max_file_opening_threads: 16 2023-03-09T14:39:00.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.statistics: (nil) 2023-03-09T14:39:00.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.use_fsync: 0 2023-03-09T14:39:00.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.max_log_file_size: 0 2023-03-09T14:39:00.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-03-09T14:39:00.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.log_file_time_to_roll: 0 2023-03-09T14:39:00.424 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.keep_log_file_num: 1000 2023-03-09T14:39:00.425 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.recycle_log_file_num: 0 2023-03-09T14:39:00.425 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.allow_fallocate: 1 2023-03-09T14:39:00.425 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.allow_mmap_reads: 0 2023-03-09T14:39:00.425 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.allow_mmap_writes: 0 2023-03-09T14:39:00.425 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.use_direct_reads: 0 2023-03-09T14:39:00.425 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-03-09T14:39:00.425 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.create_missing_column_families: 0 2023-03-09T14:39:00.426 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.db_log_dir: 2023-03-09T14:39:00.426 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.wal_dir: 2023-03-09T14:39:00.426 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.table_cache_numshardbits: 6 2023-03-09T14:39:00.426 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.WAL_ttl_seconds: 0 2023-03-09T14:39:00.426 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.WAL_size_limit_MB: 0 2023-03-09T14:39:00.426 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-03-09T14:39:00.426 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-03-09T14:39:00.426 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.is_fd_close_on_exec: 1 2023-03-09T14:39:00.427 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.advise_random_on_open: 1 2023-03-09T14:39:00.427 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.db_write_buffer_size: 0 2023-03-09T14:39:00.427 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.write_buffer_manager: 0x55b4b59105a0 2023-03-09T14:39:00.427 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-03-09T14:39:00.427 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-03-09T14:39:00.427 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.use_adaptive_mutex: 0 2023-03-09T14:39:00.427 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.rate_limiter: (nil) 2023-03-09T14:39:00.427 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-03-09T14:39:00.428 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.wal_recovery_mode: 2 2023-03-09T14:39:00.428 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.enable_thread_tracking: 0 2023-03-09T14:39:00.428 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.enable_pipelined_write: 0 2023-03-09T14:39:00.428 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.unordered_write: 0 2023-03-09T14:39:00.428 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-03-09T14:39:00.428 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-03-09T14:39:00.428 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-03-09T14:39:00.428 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-03-09T14:39:00.429 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.row_cache: None 2023-03-09T14:39:00.429 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.wal_filter: None 2023-03-09T14:39:00.429 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-03-09T14:39:00.429 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.allow_ingest_behind: 0 2023-03-09T14:39:00.429 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.two_write_queues: 0 2023-03-09T14:39:00.429 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.manual_wal_flush: 0 2023-03-09T14:39:00.429 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.wal_compression: 0 2023-03-09T14:39:00.430 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.atomic_flush: 0 2023-03-09T14:39:00.430 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-03-09T14:39:00.430 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.persist_stats_to_disk: 0 2023-03-09T14:39:00.430 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.write_dbid_to_manifest: 0 2023-03-09T14:39:00.430 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.log_readahead_size: 0 2023-03-09T14:39:00.430 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-03-09T14:39:00.430 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.best_efforts_recovery: 0 2023-03-09T14:39:00.430 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-03-09T14:39:00.431 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-03-09T14:39:00.431 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.allow_data_in_errors: 0 2023-03-09T14:39:00.432 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.db_host_id: __hostname__ 2023-03-09T14:39:00.432 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.enforce_single_del_contracts: true 2023-03-09T14:39:00.433 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.max_background_jobs: 2 2023-03-09T14:39:00.433 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.max_background_compactions: -1 2023-03-09T14:39:00.433 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.max_subcompactions: 1 2023-03-09T14:39:00.433 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-03-09T14:39:00.433 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-03-09T14:39:00.433 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.delayed_write_rate : 16777216 2023-03-09T14:39:00.433 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.max_total_wal_size: 0 2023-03-09T14:39:00.433 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-03-09T14:39:00.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.stats_dump_period_sec: 600 2023-03-09T14:39:00.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.stats_persist_period_sec: 600 2023-03-09T14:39:00.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-03-09T14:39:00.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.max_open_files: -1 2023-03-09T14:39:00.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.bytes_per_sync: 0 2023-03-09T14:39:00.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.wal_bytes_per_sync: 0 2023-03-09T14:39:00.434 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.strict_bytes_per_sync: 0 2023-03-09T14:39:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.compaction_readahead_size: 0 2023-03-09T14:39:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.max_background_flushes: -1 2023-03-09T14:39:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Compression algorithms supported: 2023-03-09T14:39:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-03-09T14:39:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: kZSTD supported: 0 2023-03-09T14:39:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: kXpressCompression supported: 0 2023-03-09T14:39:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: kLZ4HCCompression supported: 1 2023-03-09T14:39:00.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: kZlibCompression supported: 1 2023-03-09T14:39:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: kSnappyCompression supported: 1 2023-03-09T14:39:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: kLZ4Compression supported: 1 2023-03-09T14:39:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: kBZip2Compression supported: 0 2023-03-09T14:39:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Fast CRC32 supported: Supported on x86 2023-03-09T14:39:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: DMutex implementation: pthread_mutex_t 2023-03-09T14:39:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2023-03-09T14:39:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-03-09T14:39:00.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-03-09T14:39:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.merge_operator: 2023-03-09T14:39:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.compaction_filter: None 2023-03-09T14:39:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.compaction_filter_factory: None 2023-03-09T14:39:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.sst_partitioner_factory: None 2023-03-09T14:39:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.memtable_factory: SkipListFactory 2023-03-09T14:39:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.table_factory: BlockBasedTable 2023-03-09T14:39:00.437 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55b4b5939640) 2023-03-09T14:39:00.438 INFO:journalctl@ceph.mon.b.smithi122.stdout: cache_index_and_filter_blocks: 1 2023-03-09T14:39:00.438 INFO:journalctl@ceph.mon.b.smithi122.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-03-09T14:39:00.438 INFO:journalctl@ceph.mon.b.smithi122.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-03-09T14:39:00.438 INFO:journalctl@ceph.mon.b.smithi122.stdout: pin_top_level_index_and_filter: 1 2023-03-09T14:39:00.438 INFO:journalctl@ceph.mon.b.smithi122.stdout: index_type: 0 2023-03-09T14:39:00.438 INFO:journalctl@ceph.mon.b.smithi122.stdout: data_block_index_type: 0 2023-03-09T14:39:00.438 INFO:journalctl@ceph.mon.b.smithi122.stdout: index_shortening: 1 2023-03-09T14:39:00.438 INFO:journalctl@ceph.mon.b.smithi122.stdout: data_block_hash_table_util_ratio: 0.750000 2023-03-09T14:39:00.439 INFO:journalctl@ceph.mon.b.smithi122.stdout: checksum: 4 2023-03-09T14:39:00.439 INFO:journalctl@ceph.mon.b.smithi122.stdout: no_block_cache: 0 2023-03-09T14:39:00.439 INFO:journalctl@ceph.mon.b.smithi122.stdout: block_cache: 0x55b4b59911f0 2023-03-09T14:39:00.439 INFO:journalctl@ceph.mon.b.smithi122.stdout: block_cache_name: BinnedLRUCache 2023-03-09T14:39:00.439 INFO:journalctl@ceph.mon.b.smithi122.stdout: block_cache_options: 2023-03-09T14:39:00.439 INFO:journalctl@ceph.mon.b.smithi122.stdout: capacity : 536870912 2023-03-09T14:39:00.439 INFO:journalctl@ceph.mon.b.smithi122.stdout: num_shard_bits : 4 2023-03-09T14:39:00.439 INFO:journalctl@ceph.mon.b.smithi122.stdout: strict_capacity_limit : 0 2023-03-09T14:39:00.440 INFO:journalctl@ceph.mon.b.smithi122.stdout: high_pri_pool_ratio: 0.000 2023-03-09T14:39:00.440 INFO:journalctl@ceph.mon.b.smithi122.stdout: block_cache_compressed: (nil) 2023-03-09T14:39:00.440 INFO:journalctl@ceph.mon.b.smithi122.stdout: persistent_cache: (nil) 2023-03-09T14:39:00.440 INFO:journalctl@ceph.mon.b.smithi122.stdout: block_size: 4096 2023-03-09T14:39:00.440 INFO:journalctl@ceph.mon.b.smithi122.stdout: block_size_deviation: 10 2023-03-09T14:39:00.440 INFO:journalctl@ceph.mon.b.smithi122.stdout: block_restart_interval: 16 2023-03-09T14:39:00.440 INFO:journalctl@ceph.mon.b.smithi122.stdout: index_block_restart_interval: 1 2023-03-09T14:39:00.440 INFO:journalctl@ceph.mon.b.smithi122.stdout: metadata_block_size: 4096 2023-03-09T14:39:00.441 INFO:journalctl@ceph.mon.b.smithi122.stdout: partition_filters: 0 2023-03-09T14:39:00.441 INFO:journalctl@ceph.mon.b.smithi122.stdout: use_delta_encoding: 1 2023-03-09T14:39:00.441 INFO:journalctl@ceph.mon.b.smithi122.stdout: filter_policy: bloomfilter 2023-03-09T14:39:00.441 INFO:journalctl@ceph.mon.b.smithi122.stdout: whole_key_filtering: 1 2023-03-09T14:39:00.441 INFO:journalctl@ceph.mon.b.smithi122.stdout: verify_compression: 0 2023-03-09T14:39:00.441 INFO:journalctl@ceph.mon.b.smithi122.stdout: read_amp_bytes_per_bit: 0 2023-03-09T14:39:00.441 INFO:journalctl@ceph.mon.b.smithi122.stdout: format_version: 5 2023-03-09T14:39:00.441 INFO:journalctl@ceph.mon.b.smithi122.stdout: enable_index_compression: 1 2023-03-09T14:39:00.442 INFO:journalctl@ceph.mon.b.smithi122.stdout: block_align: 0 2023-03-09T14:39:00.442 INFO:journalctl@ceph.mon.b.smithi122.stdout: max_auto_readahead_size: 262144 2023-03-09T14:39:00.442 INFO:journalctl@ceph.mon.b.smithi122.stdout: prepopulate_block_cache: 0 2023-03-09T14:39:00.442 INFO:journalctl@ceph.mon.b.smithi122.stdout: initial_auto_readahead_size: 8192 2023-03-09T14:39:00.442 INFO:journalctl@ceph.mon.b.smithi122.stdout: num_file_reads_for_auto_readahead: 2 2023-03-09T14:39:00.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.write_buffer_size: 33554432 2023-03-09T14:39:00.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.max_write_buffer_number: 2 2023-03-09T14:39:00.442 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.compression: NoCompression 2023-03-09T14:39:00.443 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.bottommost_compression: Disabled 2023-03-09T14:39:00.443 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.prefix_extractor: nullptr 2023-03-09T14:39:00.443 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-03-09T14:39:00.443 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.num_levels: 7 2023-03-09T14:39:00.443 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-03-09T14:39:00.443 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-03-09T14:39:00.443 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-03-09T14:39:00.443 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-03-09T14:39:00.444 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-03-09T14:39:00.444 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-03-09T14:39:00.444 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-03-09T14:39:00.444 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-03-09T14:39:00.444 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-03-09T14:39:00.444 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-03-09T14:39:00.444 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-03-09T14:39:00.445 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-03-09T14:39:00.445 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.compression_opts.window_bits: -14 2023-03-09T14:39:00.445 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.compression_opts.level: 32767 2023-03-09T14:39:00.445 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.compression_opts.strategy: 0 2023-03-09T14:39:00.445 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-03-09T14:39:00.445 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-03-09T14:39:00.445 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-03-09T14:39:00.445 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-03-09T14:39:00.446 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.compression_opts.enabled: false 2023-03-09T14:39:00.446 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-03-09T14:39:00.446 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-03-09T14:39:00.446 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-03-09T14:39:00.446 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-03-09T14:39:00.446 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.target_file_size_base: 67108864 2023-03-09T14:39:00.446 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.target_file_size_multiplier: 1 2023-03-09T14:39:00.446 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-03-09T14:39:00.447 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-03-09T14:39:00.447 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-03-09T14:39:00.447 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-03-09T14:39:00.447 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-03-09T14:39:00.447 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-03-09T14:39:00.447 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-03-09T14:39:00.447 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-03-09T14:39:00.447 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-03-09T14:39:00.448 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-03-09T14:39:00.448 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-03-09T14:39:00.448 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-03-09T14:39:00.448 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-03-09T14:39:00.448 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.arena_block_size: 1048576 2023-03-09T14:39:00.448 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-03-09T14:39:00.448 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-03-09T14:39:00.448 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.disable_auto_compactions: 0 2023-03-09T14:39:00.449 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-03-09T14:39:00.449 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-03-09T14:39:00.449 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-03-09T14:39:00.449 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-03-09T14:39:00.449 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-03-09T14:39:00.449 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-03-09T14:39:00.449 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-03-09T14:39:00.450 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-03-09T14:39:00.450 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-03-09T14:39:00.450 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-03-09T14:39:00.450 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.table_properties_collectors: 2023-03-09T14:39:00.450 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.inplace_update_support: 0 2023-03-09T14:39:00.450 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.inplace_update_num_locks: 10000 2023-03-09T14:39:00.450 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-03-09T14:39:00.450 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-03-09T14:39:00.451 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.memtable_huge_page_size: 0 2023-03-09T14:39:00.451 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.bloom_locality: 0 2023-03-09T14:39:00.451 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.max_successive_merges: 0 2023-03-09T14:39:00.451 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.optimize_filters_for_hits: 0 2023-03-09T14:39:00.451 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.paranoid_file_checks: 0 2023-03-09T14:39:00.451 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.force_consistency_checks: 1 2023-03-09T14:39:00.451 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.report_bg_io_stats: 0 2023-03-09T14:39:00.451 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.ttl: 2592000 2023-03-09T14:39:00.452 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.periodic_compaction_seconds: 0 2023-03-09T14:39:00.452 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-03-09T14:39:00.452 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-03-09T14:39:00.452 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.enable_blob_files: false 2023-03-09T14:39:00.452 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.min_blob_size: 0 2023-03-09T14:39:00.452 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.blob_file_size: 268435456 2023-03-09T14:39:00.452 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.blob_compression_type: NoCompression 2023-03-09T14:39:00.452 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.enable_blob_garbage_collection: false 2023-03-09T14:39:00.453 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-03-09T14:39:00.453 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-03-09T14:39:00.453 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-03-09T14:39:00.453 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.blob_file_starting_level: 0 2023-03-09T14:39:00.453 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-03-09T14:39:00.453 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-03-09T14:39:00.453 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2023-03-09T14:39:00.454 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 253fef85-11b8-48e9-b20f-a136b7aecb91 2023-03-09T14:39:00.454 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: EVENT_LOG_v1 {"time_micros": 1678372740398469, "job": 1, "event": "recovery_started", "wal_files": [4]} 2023-03-09T14:39:00.454 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2023-03-09T14:39:00.454 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: EVENT_LOG_v1 {"time_micros": 1678372740398892, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1678372740, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "253fef85-11b8-48e9-b20f-a136b7aecb91", "db_session_id": "WQ5XI6M5TJC0573JTY1Q", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2023-03-09T14:39:00.454 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: EVENT_LOG_v1 {"time_micros": 1678372740398968, "job": 1, "event": "recovery_finished"} 2023-03-09T14:39:00.454 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2023-03-09T14:39:00.454 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-03-09T14:39:00.454 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55b4b5a30000 2023-03-09T14:39:00.455 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: DB pointer 0x55b4b5a1e000 2023-03-09T14:39:00.455 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-03-09T14:39:00.455 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-03-09T14:39:00.455 INFO:journalctl@ceph.mon.b.smithi122.stdout: ** DB Stats ** 2023-03-09T14:39:00.455 INFO:journalctl@ceph.mon.b.smithi122.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-03-09T14:39:00.455 INFO:journalctl@ceph.mon.b.smithi122.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-03-09T14:39:00.455 INFO:journalctl@ceph.mon.b.smithi122.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-03-09T14:39:00.455 INFO:journalctl@ceph.mon.b.smithi122.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-03-09T14:39:00.456 INFO:journalctl@ceph.mon.b.smithi122.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-03-09T14:39:00.456 INFO:journalctl@ceph.mon.b.smithi122.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-03-09T14:39:00.456 INFO:journalctl@ceph.mon.b.smithi122.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-03-09T14:39:00.456 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2023-03-09T14:39:00.456 INFO:journalctl@ceph.mon.b.smithi122.stdout: ** Compaction Stats [default] ** 2023-03-09T14:39:00.456 INFO:journalctl@ceph.mon.b.smithi122.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-03-09T14:39:00.456 INFO:journalctl@ceph.mon.b.smithi122.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-03-09T14:39:00.457 INFO:journalctl@ceph.mon.b.smithi122.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-03-09T14:39:00.457 INFO:journalctl@ceph.mon.b.smithi122.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-03-09T14:39:00.457 INFO:journalctl@ceph.mon.b.smithi122.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-03-09T14:39:00.457 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2023-03-09T14:39:00.457 INFO:journalctl@ceph.mon.b.smithi122.stdout: ** Compaction Stats [default] ** 2023-03-09T14:39:00.457 INFO:journalctl@ceph.mon.b.smithi122.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-03-09T14:39:00.457 INFO:journalctl@ceph.mon.b.smithi122.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-03-09T14:39:00.457 INFO:journalctl@ceph.mon.b.smithi122.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.0 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-03-09T14:39:00.458 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2023-03-09T14:39:00.458 INFO:journalctl@ceph.mon.b.smithi122.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-03-09T14:39:00.458 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2023-03-09T14:39:00.458 INFO:journalctl@ceph.mon.b.smithi122.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-03-09T14:39:00.458 INFO:journalctl@ceph.mon.b.smithi122.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-03-09T14:39:00.458 INFO:journalctl@ceph.mon.b.smithi122.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-03-09T14:39:00.458 INFO:journalctl@ceph.mon.b.smithi122.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-03-09T14:39:00.458 INFO:journalctl@ceph.mon.b.smithi122.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-03-09T14:39:00.459 INFO:journalctl@ceph.mon.b.smithi122.stdout: AddFile(Keys): cumulative 0, interval 0 2023-03-09T14:39:00.459 INFO:journalctl@ceph.mon.b.smithi122.stdout: Cumulative compaction: 0.00 GB write, 0.71 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-03-09T14:39:00.459 INFO:journalctl@ceph.mon.b.smithi122.stdout: Interval compaction: 0.00 GB write, 0.71 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-03-09T14:39:00.459 INFO:journalctl@ceph.mon.b.smithi122.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-03-09T14:39:00.459 INFO:journalctl@ceph.mon.b.smithi122.stdout: Block cache BinnedLRUCache@0x55b4b59911f0#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2023-03-09T14:39:00.459 INFO:journalctl@ceph.mon.b.smithi122.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2023-03-09T14:39:00.459 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2023-03-09T14:39:00.459 INFO:journalctl@ceph.mon.b.smithi122.stdout: ** File Read Latency Histogram By Level [default] ** 2023-03-09T14:39:00.460 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: mon.b does not exist in monmap, will attempt to join an existing cluster 2023-03-09T14:39:00.460 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: using public_addr v2:172.21.15.122:0/0 -> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] 2023-03-09T14:39:00.460 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: starting mon.b rank -1 at public addrs [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] at bind addrs [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid c6733fe2-be87-11ed-9af2-001a4aab830c 2023-03-09T14:39:00.460 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: mon.b@-1(???) e0 preinit fsid c6733fe2-be87-11ed-9af2-001a4aab830c 2023-03-09T14:39:00.460 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: mon.b@-1(synchronizing).mds e1 new map 2023-03-09T14:39:00.460 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: mon.b@-1(synchronizing).mds e1 print_map 2023-03-09T14:39:00.460 INFO:journalctl@ceph.mon.b.smithi122.stdout: e1 2023-03-09T14:39:00.461 INFO:journalctl@ceph.mon.b.smithi122.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-03-09T14:39:00.461 INFO:journalctl@ceph.mon.b.smithi122.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-03-09T14:39:00.461 INFO:journalctl@ceph.mon.b.smithi122.stdout: legacy client fscid: -1 2023-03-09T14:39:00.461 INFO:journalctl@ceph.mon.b.smithi122.stdout: 2023-03-09T14:39:00.461 INFO:journalctl@ceph.mon.b.smithi122.stdout: No filesystems configured 2023-03-09T14:39:00.461 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2023-03-09T14:39:00.461 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2023-03-09T14:39:00.462 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2023-03-09T14:39:00.462 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2023-03-09T14:39:00.462 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2023-03-09T14:39:00.462 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2023-03-09T14:39:00.462 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2023-03-09T14:39:00.462 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-03-09T14:39:00.462 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-03-09T14:39:00.462 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-03-09T14:39:00.463 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:39:00.463 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi181", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:39:00.463 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:00.463 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:00.463 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:00.463 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:00.463 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi122", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:39:00.463 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:39:00.464 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:39:00.464 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: Updating smithi122:/etc/ceph/ceph.conf 2023-03-09T14:39:00.464 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: Updating smithi122:/var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/config/ceph.conf 2023-03-09T14:39:00.464 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:00.464 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:00.464 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:00.464 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: Deploying cephadm binary to smithi181 2023-03-09T14:39:00.464 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2023-03-09T14:39:00.465 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: Updating smithi122:/var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/config/ceph.client.admin.keyring 2023-03-09T14:39:00.465 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:00.465 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:39:00.465 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: Added host smithi181 2023-03-09T14:39:00.465 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:39:00.465 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:00.465 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:00.466 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:39:00.466 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:00.466 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-03-09T14:39:00.466 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:39:00.466 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/4066462290' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-03-09T14:39:00.466 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/4066462290' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-03-09T14:39:00.466 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: osdmap e4: 0 total, 0 up, 0 in 2023-03-09T14:39:00.466 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:00.467 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:00.467 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:00.467 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:00.467 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi181", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:39:00.467 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:39:00.467 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:39:00.467 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: Updating smithi181:/etc/ceph/ceph.conf 2023-03-09T14:39:00.467 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: Updating smithi181:/var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/config/ceph.conf 2023-03-09T14:39:00.468 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:39:00.468 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: Updating smithi181:/etc/ceph/ceph.client.admin.keyring 2023-03-09T14:39:00.468 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: Updating smithi181:/var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/config/ceph.client.admin.keyring 2023-03-09T14:39:00.468 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:00.468 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:00.468 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:00.468 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi038:172.21.15.38=a;smithi122:172.21.15.122=b;smithi181:172.21.15.181=c", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:39:00.469 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: Saving service mon spec with placement smithi038:172.21.15.38=a;smithi122:172.21.15.122=b;smithi181:172.21.15.181=c;count:3 2023-03-09T14:39:00.469 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:00.469 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:39:00.469 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:39:00.469 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:39:00.469 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:00.469 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-03-09T14:39:00.469 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:39:00.470 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: Deploying daemon mon.c on smithi181 2023-03-09T14:39:00.470 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:39:00.470 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:39:00.470 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: from='client.? 172.21.15.181:0/280638502' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-03-09T14:39:00.470 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-03-09T14:39:00.470 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: mon.a calling monitor election 2023-03-09T14:39:00.470 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-03-09T14:39:00.470 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:39:00.471 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-03-09T14:39:00.471 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-03-09T14:39:00.471 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: mon.c calling monitor election 2023-03-09T14:39:00.471 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:39:00.471 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-03-09T14:39:00.471 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-03-09T14:39:00.471 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:39:00.472 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-03-09T14:39:00.472 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-03-09T14:39:00.472 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: monmap e2: 2 mons at {a=[v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0],c=[v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0]} removed_ranks: {} 2023-03-09T14:39:00.472 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: fsmap 2023-03-09T14:39:00.472 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: osdmap e4: 0 total, 0 up, 0 in 2023-03-09T14:39:00.472 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: mgrmap e13: x(active, since 42s) 2023-03-09T14:39:00.472 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: overall HEALTH_OK 2023-03-09T14:39:00.472 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:00.473 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:00.473 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-03-09T14:39:00.473 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:39:00.473 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: Deploying daemon mon.b on smithi122 2023-03-09T14:39:00.473 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: from='client.? 172.21.15.181:0/3468115796' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-03-09T14:39:00.473 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-03-09T14:39:00.473 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:39:00.473 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2023-03-09T14:39:00.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: expand_channel_meta expand map: {default=false} 2023-03-09T14:39:00.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: expand_channel_meta from 'false' to 'false' 2023-03-09T14:39:00.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: expand_channel_meta expanded map: {default=false} 2023-03-09T14:39:00.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: expand_channel_meta expand map: {default=info} 2023-03-09T14:39:00.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: expand_channel_meta from 'info' to 'info' 2023-03-09T14:39:00.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: expand_channel_meta expanded map: {default=info} 2023-03-09T14:39:00.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: expand_channel_meta expand map: {default=daemon} 2023-03-09T14:39:00.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: expand_channel_meta from 'daemon' to 'daemon' 2023-03-09T14:39:00.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: expand_channel_meta expanded map: {default=daemon} 2023-03-09T14:39:00.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: expand_channel_meta expand map: {default=debug} 2023-03-09T14:39:00.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: expand_channel_meta from 'debug' to 'debug' 2023-03-09T14:39:00.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: expand_channel_meta expanded map: {default=debug} 2023-03-09T14:39:00.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 ceph-mon[93869]: mon.b@-1(synchronizing) e2 handle_conf_change mon_allow_pool_delete 2023-03-09T14:39:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:00 smithi122 systemd[1]: Started Ceph mon.b for c6733fe2-be87-11ed-9af2-001a4aab830c. 2023-03-09T14:39:05.554 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:05.551+0000 7fe83c86c700 1 -- 172.21.15.181:0/3296599966 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe8240035c0 msgr2=0x7fe824005a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:39:05.555 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:05.551+0000 7fe83c86c700 1 --2- 172.21.15.181:0/3296599966 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe8240035c0 0x7fe824005a40 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fe8380ffa60 tx=0x7fe82000fe60 comp rx=0 tx=0).stop 2023-03-09T14:39:05.555 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:05.551+0000 7fe83c86c700 1 -- 172.21.15.181:0/3296599966 shutdown_connections 2023-03-09T14:39:05.555 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:05.551+0000 7fe83c86c700 1 --2- 172.21.15.181:0/3296599966 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe8240035c0 0x7fe824005a40 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:39:05.555 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:05.551+0000 7fe83c86c700 1 -- 172.21.15.181:0/3296599966 >> 172.21.15.181:0/3296599966 conn(0x7fe8380fb650 msgr2=0x7fe8380fda60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:39:05.555 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:05.551+0000 7fe83c86c700 1 -- 172.21.15.181:0/3296599966 shutdown_connections 2023-03-09T14:39:05.555 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:05.551+0000 7fe83c86c700 1 -- 172.21.15.181:0/3296599966 wait complete. 2023-03-09T14:39:05.557 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:05.552+0000 7fe83c86c700 1 Processor -- start 2023-03-09T14:39:05.557 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:05.552+0000 7fe83c86c700 1 -- start start 2023-03-09T14:39:05.558 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:05.553+0000 7fe83c86c700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe8240035c0 0x7fe838195040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:39:05.558 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:05.553+0000 7fe83c86c700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe838195580 0x7fe8381999e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:39:05.558 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:05.553+0000 7fe83c86c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe838199f20 0x7fe83819a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:39:05.558 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:05.553+0000 7fe83c86c700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe838195c50 con 0x7fe838199f20 2023-03-09T14:39:05.558 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:05.553+0000 7fe83c86c700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8381a1c90 con 0x7fe838195580 2023-03-09T14:39:05.559 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:05.553+0000 7fe83c86c700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8381a1dd0 con 0x7fe8240035c0 2023-03-09T14:39:05.559 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:05.553+0000 7fe83659c700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe8240035c0 0x7fe838195040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:39:05.559 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:05.553+0000 7fe83659c700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe8240035c0 0x7fe838195040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:50464/0 (socket says 172.21.15.181:50464) 2023-03-09T14:39:05.559 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:05.553+0000 7fe83659c700 1 -- 172.21.15.181:0/361482335 learned_addr learned my addr 172.21.15.181:0/361482335 (peer_addr_for_me v2:172.21.15.181:0/0) 2023-03-09T14:39:05.559 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:05.553+0000 7fe835d9b700 1 --2- 172.21.15.181:0/361482335 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe838195580 0x7fe8381999e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:39:05.559 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:05.553+0000 7fe83659c700 1 -- 172.21.15.181:0/361482335 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe8240035c0 msgr2=0x7fe838195040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2023-03-09T14:39:05.560 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:05.553+0000 7fe83659c700 1 -- 172.21.15.181:0/361482335 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe8240035c0 msgr2=0x7fe838195040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-03-09T14:39:05.560 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:05.553+0000 7fe83659c700 1 --2- 172.21.15.181:0/361482335 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe8240035c0 0x7fe838195040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-03-09T14:39:05.560 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:05.553+0000 7fe836d9d700 1 --2- 172.21.15.181:0/361482335 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe838199f20 0x7fe83819a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:39:05.560 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:05.553+0000 7fe83659c700 1 --2- 172.21.15.181:0/361482335 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe8240035c0 0x7fe838195040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-03-09T14:39:05.560 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:05.553+0000 7fe835d9b700 1 -- 172.21.15.181:0/361482335 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe838195580 msgr2=0x7fe8381999e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2023-03-09T14:39:05.560 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:05.553+0000 7fe835d9b700 1 -- 172.21.15.181:0/361482335 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe838195580 msgr2=0x7fe8381999e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-03-09T14:39:05.560 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:05.553+0000 7fe835d9b700 1 --2- 172.21.15.181:0/361482335 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe838195580 0x7fe8381999e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-03-09T14:39:05.561 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:05.553+0000 7fe835d9b700 1 --2- 172.21.15.181:0/361482335 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe838195580 0x7fe8381999e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-03-09T14:39:05.561 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:05.553+0000 7fe836d9d700 1 -- 172.21.15.181:0/361482335 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe8240035c0 msgr2=0x7fe838195040 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-03-09T14:39:05.561 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:05.553+0000 7fe836d9d700 1 --2- 172.21.15.181:0/361482335 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe8240035c0 0x7fe838195040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:39:05.561 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:05.553+0000 7fe836d9d700 1 -- 172.21.15.181:0/361482335 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe838195580 msgr2=0x7fe8381999e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-03-09T14:39:05.561 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:05.553+0000 7fe836d9d700 1 --2- 172.21.15.181:0/361482335 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe838195580 0x7fe8381999e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:39:05.561 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:05.554+0000 7fe836d9d700 1 -- 172.21.15.181:0/361482335 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe82000f8b0 con 0x7fe838199f20 2023-03-09T14:39:05.562 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:05.554+0000 7fe836d9d700 1 --2- 172.21.15.181:0/361482335 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe838199f20 0x7fe83819a3c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fe83000bd50 tx=0x7fe83000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:39:05.562 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:05.554+0000 7fe82f7fe700 1 -- 172.21.15.181:0/361482335 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe83000cd10 con 0x7fe838199f20 2023-03-09T14:39:05.562 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:05.554+0000 7fe82f7fe700 1 -- 172.21.15.181:0/361482335 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe83000ce70 con 0x7fe838199f20 2023-03-09T14:39:05.562 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:05.554+0000 7fe82f7fe700 1 -- 172.21.15.181:0/361482335 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe830012840 con 0x7fe838199f20 2023-03-09T14:39:05.562 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:05.554+0000 7fe83c86c700 1 -- 172.21.15.181:0/361482335 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe838107960 con 0x7fe838199f20 2023-03-09T14:39:05.562 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:05.554+0000 7fe83c86c700 1 -- 172.21.15.181:0/361482335 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8381077d0 con 0x7fe838199f20 2023-03-09T14:39:05.564 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:05.555+0000 7fe82f7fe700 1 -- 172.21.15.181:0/361482335 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 44674+0+0 (secure 0 0 0) 0x7fe83001b030 con 0x7fe838199f20 2023-03-09T14:39:05.564 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:05.555+0000 7fe82f7fe700 1 --2- 172.21.15.181:0/361482335 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fe818038300 0x7fe81803a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:39:05.564 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:05.555+0000 7fe83c86c700 1 -- 172.21.15.181:0/361482335 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe838107af0 con 0x7fe838199f20 2023-03-09T14:39:05.564 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:05.555+0000 7fe82f7fe700 1 -- 172.21.15.181:0/361482335 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fe830049c60 con 0x7fe838199f20 2023-03-09T14:39:05.564 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:05.556+0000 7fe83659c700 1 --2- 172.21.15.181:0/361482335 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fe818038300 0x7fe81803a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:39:05.565 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:05.556+0000 7fe83659c700 1 --2- 172.21.15.181:0/361482335 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fe818038300 0x7fe81803a7b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe820005d10 tx=0x7fe820005c80 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:39:05.565 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:05.561+0000 7fe82f7fe700 1 -- 172.21.15.181:0/361482335 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7fe830011890 con 0x7fe838199f20 2023-03-09T14:39:05.845 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:05.843+0000 7fe83c86c700 1 -- 172.21.15.181:0/361482335 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe8380732f0 con 0x7fe838199f20 2023-03-09T14:39:05.847 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:05.843+0000 7fe82f7fe700 1 -- 172.21.15.181:0/361482335 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1274 (secure 0 0 0) 0x7fe8300112a0 con 0x7fe838199f20 2023-03-09T14:39:05.847 INFO:teuthology.orchestra.run.smithi181.stdout: 2023-03-09T14:39:05.848 INFO:teuthology.orchestra.run.smithi181.stdout:{"epoch":3,"fsid":"c6733fe2-be87-11ed-9af2-001a4aab830c","modified":"2023-03-09T14:39:00.423085Z","created":"2023-03-09T14:37:12.753639Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:3300","nonce":0},{"type":"v1","addr":"172.21.15.38:6789","nonce":0}]},"addr":"172.21.15.38:6789/0","public_addr":"172.21.15.38:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:3300","nonce":0},{"type":"v1","addr":"172.21.15.181:6789","nonce":0}]},"addr":"172.21.15.181:6789/0","public_addr":"172.21.15.181:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:3300","nonce":0},{"type":"v1","addr":"172.21.15.122:6789","nonce":0}]},"addr":"172.21.15.122:6789/0","public_addr":"172.21.15.122:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2023-03-09T14:39:05.850 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:05.846+0000 7fe83c86c700 1 -- 172.21.15.181:0/361482335 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fe818038300 msgr2=0x7fe81803a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:39:05.850 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:05.846+0000 7fe83c86c700 1 --2- 172.21.15.181:0/361482335 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fe818038300 0x7fe81803a7b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe820005d10 tx=0x7fe820005c80 comp rx=0 tx=0).stop 2023-03-09T14:39:05.850 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:05.846+0000 7fe83c86c700 1 -- 172.21.15.181:0/361482335 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe838199f20 msgr2=0x7fe83819a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:39:05.850 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:05.846+0000 7fe83c86c700 1 --2- 172.21.15.181:0/361482335 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe838199f20 0x7fe83819a3c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fe83000bd50 tx=0x7fe83000be30 comp rx=0 tx=0).stop 2023-03-09T14:39:05.850 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:05.846+0000 7fe83c86c700 1 -- 172.21.15.181:0/361482335 shutdown_connections 2023-03-09T14:39:05.850 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:05.846+0000 7fe83c86c700 1 --2- 172.21.15.181:0/361482335 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe838199f20 0x7fe83819a3c0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:39:05.851 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:05.846+0000 7fe83c86c700 1 --2- 172.21.15.181:0/361482335 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fe818038300 0x7fe81803a7b0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:39:05.851 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:05.846+0000 7fe83c86c700 1 --2- 172.21.15.181:0/361482335 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe8240035c0 0x7fe838195040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:39:05.851 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:05.846+0000 7fe83c86c700 1 --2- 172.21.15.181:0/361482335 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe838195580 0x7fe8381999e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:39:05.851 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:05.846+0000 7fe83c86c700 1 -- 172.21.15.181:0/361482335 >> 172.21.15.181:0/361482335 conn(0x7fe8380fb650 msgr2=0x7fe83818d930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:39:05.851 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:05.846+0000 7fe83c86c700 1 -- 172.21.15.181:0/361482335 shutdown_connections 2023-03-09T14:39:05.851 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:05.846+0000 7fe83c86c700 1 -- 172.21.15.181:0/361482335 wait complete. 2023-03-09T14:39:05.851 INFO:teuthology.orchestra.run.smithi181.stderr:dumped monmap epoch 3 2023-03-09T14:39:05.875 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:05 smithi181 ceph-mon[94189]: mon.c calling monitor election 2023-03-09T14:39:05.875 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:05 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-03-09T14:39:05.876 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:05 smithi181 ceph-mon[94189]: mon.a calling monitor election 2023-03-09T14:39:05.876 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:05 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-03-09T14:39:05.876 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:05 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-03-09T14:39:05.876 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:05 smithi181 ceph-mon[94189]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:39:05.876 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:05 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-03-09T14:39:05.876 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:05 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-03-09T14:39:05.876 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:05 smithi181 ceph-mon[94189]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:39:05.877 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:05 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-03-09T14:39:05.877 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:05 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-03-09T14:39:05.877 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:05 smithi181 ceph-mon[94189]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:39:05.877 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:05 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-03-09T14:39:05.877 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:05 smithi181 ceph-mon[94189]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-03-09T14:39:05.877 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:05 smithi181 ceph-mon[94189]: monmap e3: 3 mons at {a=[v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0],b=[v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0],c=[v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0]} removed_ranks: {} 2023-03-09T14:39:05.877 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:05 smithi181 ceph-mon[94189]: fsmap 2023-03-09T14:39:05.878 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:05 smithi181 ceph-mon[94189]: osdmap e4: 0 total, 0 up, 0 in 2023-03-09T14:39:05.878 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:05 smithi181 ceph-mon[94189]: mgrmap e13: x(active, since 50s) 2023-03-09T14:39:05.878 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:05 smithi181 ceph-mon[94189]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2023-03-09T14:39:05.878 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:05 smithi181 ceph-mon[94189]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2023-03-09T14:39:05.878 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:05 smithi181 ceph-mon[94189]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2023-03-09T14:39:05.878 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:05 smithi181 ceph-mon[94189]: mon.b (rank 2) addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] is down (out of quorum) 2023-03-09T14:39:05.878 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:05 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:05.879 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:05 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:05 smithi038 ceph-mon[92422]: mon.c calling monitor election 2023-03-09T14:39:06.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:05 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-03-09T14:39:06.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:05 smithi038 ceph-mon[92422]: mon.a calling monitor election 2023-03-09T14:39:06.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:05 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-03-09T14:39:06.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:05 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-03-09T14:39:06.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:05 smithi038 ceph-mon[92422]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:39:06.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:05 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-03-09T14:39:06.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:05 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-03-09T14:39:06.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:05 smithi038 ceph-mon[92422]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:39:06.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:05 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-03-09T14:39:06.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:05 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-03-09T14:39:06.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:05 smithi038 ceph-mon[92422]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:39:06.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:05 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-03-09T14:39:06.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:05 smithi038 ceph-mon[92422]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-03-09T14:39:06.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:05 smithi038 ceph-mon[92422]: monmap e3: 3 mons at {a=[v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0],b=[v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0],c=[v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0]} removed_ranks: {} 2023-03-09T14:39:06.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:05 smithi038 ceph-mon[92422]: fsmap 2023-03-09T14:39:06.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:05 smithi038 ceph-mon[92422]: osdmap e4: 0 total, 0 up, 0 in 2023-03-09T14:39:06.004 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:05 smithi038 ceph-mon[92422]: mgrmap e13: x(active, since 50s) 2023-03-09T14:39:06.004 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:05 smithi038 ceph-mon[92422]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2023-03-09T14:39:06.004 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:05 smithi038 ceph-mon[92422]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2023-03-09T14:39:06.004 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:05 smithi038 ceph-mon[92422]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2023-03-09T14:39:06.004 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:05 smithi038 ceph-mon[92422]: mon.b (rank 2) addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] is down (out of quorum) 2023-03-09T14:39:06.004 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:05 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:06.005 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:05 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:06.225 INFO:tasks.cephadm:Generating final ceph.conf file... 2023-03-09T14:39:06.226 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph config generate-minimal-conf 2023-03-09T14:39:06.892 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.a/config 2023-03-09T14:39:06.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:06 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:39:06.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:06 smithi181 ceph-mon[94189]: from='client.? 172.21.15.181:0/361482335' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-03-09T14:39:06.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:06 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-03-09T14:39:07.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:06 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:39:07.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:06 smithi038 ceph-mon[92422]: from='client.? 172.21.15.181:0/361482335' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-03-09T14:39:07.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:06 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-03-09T14:39:07.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:07 smithi122 ceph-mon[93869]: mon.c calling monitor election 2023-03-09T14:39:07.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:07 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-03-09T14:39:07.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:07 smithi122 ceph-mon[93869]: mon.a calling monitor election 2023-03-09T14:39:07.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:07 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-03-09T14:39:07.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:07 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-03-09T14:39:07.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:07 smithi122 ceph-mon[93869]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:39:07.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:07 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-03-09T14:39:07.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:07 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-03-09T14:39:07.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:07 smithi122 ceph-mon[93869]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:39:07.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:07 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-03-09T14:39:07.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:07 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-03-09T14:39:07.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:07 smithi122 ceph-mon[93869]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:39:07.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:07 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-03-09T14:39:07.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:07 smithi122 ceph-mon[93869]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-03-09T14:39:07.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:07 smithi122 ceph-mon[93869]: monmap e3: 3 mons at {a=[v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0],b=[v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0],c=[v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0]} removed_ranks: {} 2023-03-09T14:39:07.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:07 smithi122 ceph-mon[93869]: fsmap 2023-03-09T14:39:07.731 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:07 smithi122 ceph-mon[93869]: osdmap e4: 0 total, 0 up, 0 in 2023-03-09T14:39:07.731 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:07 smithi122 ceph-mon[93869]: mgrmap e13: x(active, since 50s) 2023-03-09T14:39:07.731 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:07 smithi122 ceph-mon[93869]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2023-03-09T14:39:07.731 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:07 smithi122 ceph-mon[93869]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2023-03-09T14:39:07.731 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:07 smithi122 ceph-mon[93869]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2023-03-09T14:39:07.731 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:07 smithi122 ceph-mon[93869]: mon.b (rank 2) addr [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] is down (out of quorum) 2023-03-09T14:39:07.731 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:07 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:07.732 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:07 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:07.732 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:07 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:39:07.732 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:07 smithi122 ceph-mon[93869]: from='client.? 172.21.15.181:0/361482335' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-03-09T14:39:07.732 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:07 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-03-09T14:39:08.200 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:08.197+0000 7f080a1b2700 1 -- 172.21.15.38:0/127323883 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0804104910 msgr2=0x7f0804104ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:39:08.201 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:08.197+0000 7f080a1b2700 1 --2- 172.21.15.38:0/127323883 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0804104910 0x7f0804104ce0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f07f4009910 tx=0x7f07f4009c20 comp rx=0 tx=0).stop 2023-03-09T14:39:08.201 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:08.197+0000 7f080a1b2700 1 -- 172.21.15.38:0/127323883 shutdown_connections 2023-03-09T14:39:08.201 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:08.197+0000 7f080a1b2700 1 --2- 172.21.15.38:0/127323883 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0804104910 0x7f0804104ce0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:39:08.201 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:08.197+0000 7f080a1b2700 1 -- 172.21.15.38:0/127323883 >> 172.21.15.38:0/127323883 conn(0x7f08040745e0 msgr2=0x7f08040749e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:39:08.201 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:08.197+0000 7f080a1b2700 1 -- 172.21.15.38:0/127323883 shutdown_connections 2023-03-09T14:39:08.201 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:08.197+0000 7f080a1b2700 1 -- 172.21.15.38:0/127323883 wait complete. 2023-03-09T14:39:08.203 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:08.198+0000 7f080a1b2700 1 Processor -- start 2023-03-09T14:39:08.204 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:08.198+0000 7f080a1b2700 1 -- start start 2023-03-09T14:39:08.204 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:08.198+0000 7f080a1b2700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0804104910 0x7f0804195040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:39:08.204 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:08.198+0000 7f080a1b2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0804195580 0x7f08041999e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:39:08.204 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:08.198+0000 7f080a1b2700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0804199f20 0x7f080419a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:39:08.204 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:08.198+0000 7f080a1b2700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0804195c50 con 0x7f0804195580 2023-03-09T14:39:08.204 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:08.198+0000 7f080a1b2700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08041a1c90 con 0x7f0804199f20 2023-03-09T14:39:08.205 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:08.198+0000 7f080a1b2700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08041a1dd0 con 0x7f0804104910 2023-03-09T14:39:08.205 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:08.198+0000 7f0802ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0804195580 0x7f08041999e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:39:08.205 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:08.198+0000 7f0802ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0804195580 0x7f08041999e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35764/0 (socket says 172.21.15.38:35764) 2023-03-09T14:39:08.205 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:08.198+0000 7f0802ffd700 1 -- 172.21.15.38:0/2273891420 learned_addr learned my addr 172.21.15.38:0/2273891420 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:39:08.205 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:08.199+0000 7f08037fe700 1 --2- 172.21.15.38:0/2273891420 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0804104910 0x7f0804195040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:39:08.205 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:08.199+0000 7f0802ffd700 1 -- 172.21.15.38:0/2273891420 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0804104910 msgr2=0x7f0804195040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:39:08.206 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:08.199+0000 7f0802ffd700 1 --2- 172.21.15.38:0/2273891420 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0804104910 0x7f0804195040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:39:08.206 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:08.199+0000 7f0802ffd700 1 -- 172.21.15.38:0/2273891420 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0804199f20 msgr2=0x7f080419a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:39:08.206 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:08.199+0000 7f0803fff700 1 --2- 172.21.15.38:0/2273891420 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0804199f20 0x7f080419a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:39:08.206 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:08.199+0000 7f0802ffd700 1 --2- 172.21.15.38:0/2273891420 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0804199f20 0x7f080419a3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:39:08.206 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:08.199+0000 7f0802ffd700 1 -- 172.21.15.38:0/2273891420 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07f40095f0 con 0x7f0804195580 2023-03-09T14:39:08.206 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:08.199+0000 7f0802ffd700 1 --2- 172.21.15.38:0/2273891420 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0804195580 0x7f08041999e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f07f800b780 tx=0x7f07f800bb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:39:08.206 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:08.199+0000 7f0800ff9700 1 -- 172.21.15.38:0/2273891420 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07f8004230 con 0x7f0804195580 2023-03-09T14:39:08.207 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:08.199+0000 7f0800ff9700 1 -- 172.21.15.38:0/2273891420 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f07f8004390 con 0x7f0804195580 2023-03-09T14:39:08.207 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:08.199+0000 7f0800ff9700 1 -- 172.21.15.38:0/2273891420 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07f8011b30 con 0x7f0804195580 2023-03-09T14:39:08.207 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:08.199+0000 7f080a1b2700 1 -- 172.21.15.38:0/2273891420 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0804108d00 con 0x7f0804195580 2023-03-09T14:39:08.207 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:08.199+0000 7f080a1b2700 1 -- 172.21.15.38:0/2273891420 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0804108b70 con 0x7f0804195580 2023-03-09T14:39:08.207 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:08.200+0000 7f0800ff9700 1 -- 172.21.15.38:0/2273891420 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 13) v1 ==== 44674+0+0 (secure 0 0 0) 0x7f07f8017020 con 0x7f0804195580 2023-03-09T14:39:08.207 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:08.200+0000 7f080a1b2700 1 -- 172.21.15.38:0/2273891420 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0804108e90 con 0x7f0804195580 2023-03-09T14:39:08.207 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:08.200+0000 7f0800ff9700 1 --2- 172.21.15.38:0/2273891420 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f07ec0382e0 0x7f07ec03a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:39:08.207 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:08.200+0000 7f0800ff9700 1 -- 172.21.15.38:0/2273891420 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f07f8049bb0 con 0x7f0804195580 2023-03-09T14:39:08.208 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:08.200+0000 7f08037fe700 1 --2- 172.21.15.38:0/2273891420 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f07ec0382e0 0x7f07ec03a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:39:08.208 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:08.201+0000 7f08037fe700 1 --2- 172.21.15.38:0/2273891420 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f07ec0382e0 0x7f07ec03a790 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f07f4000c00 tx=0x7f07f4005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:39:08.208 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:08.205+0000 7f0800ff9700 1 -- 172.21.15.38:0/2273891420 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f07f8004500 con 0x7f0804195580 2023-03-09T14:39:08.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:08 smithi038 ceph-mon[92422]: mon.b calling monitor election 2023-03-09T14:39:08.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:08 smithi038 ceph-mon[92422]: mon.b calling monitor election 2023-03-09T14:39:08.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:08 smithi038 ceph-mon[92422]: mon.a calling monitor election 2023-03-09T14:39:08.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:08 smithi038 ceph-mon[92422]: mon.c calling monitor election 2023-03-09T14:39:08.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:08 smithi038 ceph-mon[92422]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-03-09T14:39:08.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:08 smithi038 ceph-mon[92422]: monmap e3: 3 mons at {a=[v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0],b=[v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0],c=[v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0]} removed_ranks: {} 2023-03-09T14:39:08.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:08 smithi038 ceph-mon[92422]: fsmap 2023-03-09T14:39:08.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:08 smithi038 ceph-mon[92422]: osdmap e4: 0 total, 0 up, 0 in 2023-03-09T14:39:08.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:08 smithi038 ceph-mon[92422]: mgrmap e13: x(active, since 52s) 2023-03-09T14:39:08.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:08 smithi038 ceph-mon[92422]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2023-03-09T14:39:08.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:08 smithi038 ceph-mon[92422]: Cluster is now healthy 2023-03-09T14:39:08.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:08 smithi038 ceph-mon[92422]: overall HEALTH_OK 2023-03-09T14:39:08.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:08 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:08.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:08 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:08.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:08 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:08.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:08 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:08.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:08 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:08.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:08 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:08.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:08 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:08.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:08 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-03-09T14:39:08.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:08 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-03-09T14:39:08.254 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:08 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:39:08.413 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:08.410+0000 7f080a1b2700 1 -- 172.21.15.38:0/2273891420 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f0804196390 con 0x7f0804195580 2023-03-09T14:39:08.414 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:08.410+0000 7f0800ff9700 1 -- 172.21.15.38:0/2273891420 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+275 (secure 0 0 0) 0x7f07f80212f0 con 0x7f0804195580 2023-03-09T14:39:08.414 INFO:teuthology.orchestra.run.smithi038.stdout:# minimal ceph.conf for c6733fe2-be87-11ed-9af2-001a4aab830c 2023-03-09T14:39:08.414 INFO:teuthology.orchestra.run.smithi038.stdout:[global] 2023-03-09T14:39:08.414 INFO:teuthology.orchestra.run.smithi038.stdout: fsid = c6733fe2-be87-11ed-9af2-001a4aab830c 2023-03-09T14:39:08.414 INFO:teuthology.orchestra.run.smithi038.stdout: mon_host = [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] 2023-03-09T14:39:08.415 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:08.412+0000 7f080a1b2700 1 -- 172.21.15.38:0/2273891420 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f07ec0382e0 msgr2=0x7f07ec03a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:39:08.416 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:08.412+0000 7f080a1b2700 1 --2- 172.21.15.38:0/2273891420 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f07ec0382e0 0x7f07ec03a790 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f07f4000c00 tx=0x7f07f4005fb0 comp rx=0 tx=0).stop 2023-03-09T14:39:08.416 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:08.412+0000 7f080a1b2700 1 -- 172.21.15.38:0/2273891420 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0804195580 msgr2=0x7f08041999e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:39:08.416 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:08.412+0000 7f080a1b2700 1 --2- 172.21.15.38:0/2273891420 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0804195580 0x7f08041999e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f07f800b780 tx=0x7f07f800bb40 comp rx=0 tx=0).stop 2023-03-09T14:39:08.416 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:08.412+0000 7f080a1b2700 1 -- 172.21.15.38:0/2273891420 shutdown_connections 2023-03-09T14:39:08.416 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:08.412+0000 7f080a1b2700 1 --2- 172.21.15.38:0/2273891420 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f07ec0382e0 0x7f07ec03a790 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:39:08.416 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:08.412+0000 7f080a1b2700 1 --2- 172.21.15.38:0/2273891420 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0804199f20 0x7f080419a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:39:08.416 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:08.412+0000 7f080a1b2700 1 --2- 172.21.15.38:0/2273891420 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0804104910 0x7f0804195040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:39:08.417 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:08.412+0000 7f080a1b2700 1 --2- 172.21.15.38:0/2273891420 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0804195580 0x7f08041999e0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:39:08.417 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:08.412+0000 7f080a1b2700 1 -- 172.21.15.38:0/2273891420 >> 172.21.15.38:0/2273891420 conn(0x7f08040745e0 msgr2=0x7f0804102650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:39:08.417 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:08.413+0000 7f080a1b2700 1 -- 172.21.15.38:0/2273891420 shutdown_connections 2023-03-09T14:39:08.417 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:08.413+0000 7f080a1b2700 1 -- 172.21.15.38:0/2273891420 wait complete. 2023-03-09T14:39:08.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:08 smithi122 ceph-mon[93869]: mon.b calling monitor election 2023-03-09T14:39:08.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:08 smithi122 ceph-mon[93869]: mon.b calling monitor election 2023-03-09T14:39:08.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:08 smithi122 ceph-mon[93869]: mon.a calling monitor election 2023-03-09T14:39:08.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:08 smithi122 ceph-mon[93869]: mon.c calling monitor election 2023-03-09T14:39:08.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:08 smithi122 ceph-mon[93869]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-03-09T14:39:08.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:08 smithi122 ceph-mon[93869]: monmap e3: 3 mons at {a=[v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0],b=[v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0],c=[v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0]} removed_ranks: {} 2023-03-09T14:39:08.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:08 smithi122 ceph-mon[93869]: fsmap 2023-03-09T14:39:08.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:08 smithi122 ceph-mon[93869]: osdmap e4: 0 total, 0 up, 0 in 2023-03-09T14:39:08.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:08 smithi122 ceph-mon[93869]: mgrmap e13: x(active, since 52s) 2023-03-09T14:39:08.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:08 smithi122 ceph-mon[93869]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2023-03-09T14:39:08.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:08 smithi122 ceph-mon[93869]: Cluster is now healthy 2023-03-09T14:39:08.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:08 smithi122 ceph-mon[93869]: overall HEALTH_OK 2023-03-09T14:39:08.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:08 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:08.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:08 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:08.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:08 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:08.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:08 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:08.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:08 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:08.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:08 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:08.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:08 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:08.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:08 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-03-09T14:39:08.731 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:08 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-03-09T14:39:08.731 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:08 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:39:08.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:08 smithi181 ceph-mon[94189]: mon.b calling monitor election 2023-03-09T14:39:08.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:08 smithi181 ceph-mon[94189]: mon.b calling monitor election 2023-03-09T14:39:08.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:08 smithi181 ceph-mon[94189]: mon.a calling monitor election 2023-03-09T14:39:08.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:08 smithi181 ceph-mon[94189]: mon.c calling monitor election 2023-03-09T14:39:08.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:08 smithi181 ceph-mon[94189]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-03-09T14:39:08.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:08 smithi181 ceph-mon[94189]: monmap e3: 3 mons at {a=[v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0],b=[v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0],c=[v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0]} removed_ranks: {} 2023-03-09T14:39:08.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:08 smithi181 ceph-mon[94189]: fsmap 2023-03-09T14:39:08.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:08 smithi181 ceph-mon[94189]: osdmap e4: 0 total, 0 up, 0 in 2023-03-09T14:39:08.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:08 smithi181 ceph-mon[94189]: mgrmap e13: x(active, since 52s) 2023-03-09T14:39:08.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:08 smithi181 ceph-mon[94189]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2023-03-09T14:39:08.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:08 smithi181 ceph-mon[94189]: Cluster is now healthy 2023-03-09T14:39:08.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:08 smithi181 ceph-mon[94189]: overall HEALTH_OK 2023-03-09T14:39:08.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:08 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:08.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:08 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:08.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:08 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:08.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:08 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:08.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:08 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:08.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:08 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:08.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:08 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:08.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:08 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-03-09T14:39:08.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:08 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-03-09T14:39:08.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:08 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:39:08.940 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2023-03-09T14:39:08.940 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2023-03-09T14:39:08.940 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/ceph/ceph.conf 2023-03-09T14:39:08.971 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2023-03-09T14:39:08.972 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-03-09T14:39:09.039 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2023-03-09T14:39:09.040 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/ceph/ceph.conf 2023-03-09T14:39:09.076 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2023-03-09T14:39:09.077 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-03-09T14:39:09.147 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2023-03-09T14:39:09.147 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/ceph/ceph.conf 2023-03-09T14:39:09.183 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2023-03-09T14:39:09.184 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-03-09T14:39:09.253 INFO:tasks.cephadm:Adding mgr.x on smithi038 2023-03-09T14:39:09.254 INFO:tasks.cephadm:Adding mgr.y on smithi122 2023-03-09T14:39:09.254 INFO:tasks.cephadm:Adding mgr.z on smithi181 2023-03-09T14:39:09.254 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph orch apply mgr '3;smithi038=x;smithi122=y;smithi181=z' 2023-03-09T14:39:09.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:09 smithi122 ceph-mon[93869]: Updating smithi122:/var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/config/ceph.conf 2023-03-09T14:39:09.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:09 smithi122 ceph-mon[93869]: Updating smithi181:/var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/config/ceph.conf 2023-03-09T14:39:09.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:09 smithi122 ceph-mon[93869]: Updating smithi038:/var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/config/ceph.conf 2023-03-09T14:39:09.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:09 smithi122 ceph-mon[93869]: Reconfiguring mon.a (unknown last config time)... 2023-03-09T14:39:09.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:09 smithi122 ceph-mon[93869]: Reconfiguring daemon mon.a on smithi038 2023-03-09T14:39:09.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:09 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/2273891420' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:39:09.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:09 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-03-09T14:39:09.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:09 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:09.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:09 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:09.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:09 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-03-09T14:39:09.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:09 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-03-09T14:39:09.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:09 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:39:09.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:09 smithi181 ceph-mon[94189]: Updating smithi122:/var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/config/ceph.conf 2023-03-09T14:39:09.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:09 smithi181 ceph-mon[94189]: Updating smithi181:/var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/config/ceph.conf 2023-03-09T14:39:09.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:09 smithi181 ceph-mon[94189]: Updating smithi038:/var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/config/ceph.conf 2023-03-09T14:39:09.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:09 smithi181 ceph-mon[94189]: Reconfiguring mon.a (unknown last config time)... 2023-03-09T14:39:09.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:09 smithi181 ceph-mon[94189]: Reconfiguring daemon mon.a on smithi038 2023-03-09T14:39:09.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:09 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/2273891420' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:39:09.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:09 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-03-09T14:39:09.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:09 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:09.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:09 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:09.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:09 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-03-09T14:39:09.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:09 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-03-09T14:39:09.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:09 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:39:09.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:09 smithi038 ceph-mon[92422]: Updating smithi122:/var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/config/ceph.conf 2023-03-09T14:39:09.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:09 smithi038 ceph-mon[92422]: Updating smithi181:/var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/config/ceph.conf 2023-03-09T14:39:09.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:09 smithi038 ceph-mon[92422]: Updating smithi038:/var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/config/ceph.conf 2023-03-09T14:39:09.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:09 smithi038 ceph-mon[92422]: Reconfiguring mon.a (unknown last config time)... 2023-03-09T14:39:09.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:09 smithi038 ceph-mon[92422]: Reconfiguring daemon mon.a on smithi038 2023-03-09T14:39:09.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:09 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/2273891420' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:39:09.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:09 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-03-09T14:39:09.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:09 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:09.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:09 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:09.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:09 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-03-09T14:39:09.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:09 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-03-09T14:39:09.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:09 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:39:09.932 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.c/config 2023-03-09T14:39:10.433 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:10 smithi181 ceph-mon[94189]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:39:10.434 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:10 smithi181 ceph-mon[94189]: Reconfiguring mon.b (monmap changed)... 2023-03-09T14:39:10.434 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:10 smithi181 ceph-mon[94189]: Reconfiguring daemon mon.b on smithi122 2023-03-09T14:39:10.434 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:10 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:10 smithi122 ceph-mon[93869]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:39:10.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:10 smithi122 ceph-mon[93869]: Reconfiguring mon.b (monmap changed)... 2023-03-09T14:39:10.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:10 smithi122 ceph-mon[93869]: Reconfiguring daemon mon.b on smithi122 2023-03-09T14:39:10.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:10 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:10 smithi038 ceph-mon[92422]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:39:10.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:10 smithi038 ceph-mon[92422]: Reconfiguring mon.b (monmap changed)... 2023-03-09T14:39:10.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:10 smithi038 ceph-mon[92422]: Reconfiguring daemon mon.b on smithi122 2023-03-09T14:39:10.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:10 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:11.176 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:11.172+0000 7f2fa3760700 1 -- 172.21.15.181:0/3687184278 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2f9c0fea20 msgr2=0x7f2f9c0fee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:39:11.177 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:11.172+0000 7f2fa3760700 1 --2- 172.21.15.181:0/3687184278 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2f9c0fea20 0x7f2f9c0fee30 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f2f98009910 tx=0x7f2f98009c20 comp rx=0 tx=0).stop 2023-03-09T14:39:11.177 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:11.172+0000 7f2fa3760700 1 -- 172.21.15.181:0/3687184278 shutdown_connections 2023-03-09T14:39:11.177 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:11.172+0000 7f2fa3760700 1 --2- 172.21.15.181:0/3687184278 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2f9c0fea20 0x7f2f9c0fee30 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:39:11.177 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:11.172+0000 7f2fa3760700 1 -- 172.21.15.181:0/3687184278 >> 172.21.15.181:0/3687184278 conn(0x7f2f9c0f9fc0 msgr2=0x7f2f9c0fc3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:39:11.177 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:11.172+0000 7f2fa3760700 1 -- 172.21.15.181:0/3687184278 shutdown_connections 2023-03-09T14:39:11.177 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:11.172+0000 7f2fa3760700 1 -- 172.21.15.181:0/3687184278 wait complete. 2023-03-09T14:39:11.178 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:11.173+0000 7f2fa3760700 1 Processor -- start 2023-03-09T14:39:11.178 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:11.173+0000 7f2fa3760700 1 -- start start 2023-03-09T14:39:11.178 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:11.173+0000 7f2fa3760700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2f9c0fea20 0x7f2f9c0692c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:39:11.178 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:11.173+0000 7f2fa3760700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2f9c199ef0 0x7f2f9c19c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:39:11.178 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:11.173+0000 7f2fa3760700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2f9c19c810 0x7f2f9c19ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:39:11.178 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:11.173+0000 7f2fa3760700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f9c19f2a0 con 0x7f2f9c199ef0 2023-03-09T14:39:11.178 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:11.173+0000 7f2fa3760700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f9c19f410 con 0x7f2f9c0fea20 2023-03-09T14:39:11.179 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:11.173+0000 7f2fa3760700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f9c19f5d0 con 0x7f2f9c19c810 2023-03-09T14:39:11.179 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:11.173+0000 7f2fa1cfd700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2f9c19c810 0x7f2f9c19ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:39:11.179 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:11.173+0000 7f2fa1cfd700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2f9c19c810 0x7f2f9c19ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:50478/0 (socket says 172.21.15.181:50478) 2023-03-09T14:39:11.179 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:11.173+0000 7f2fa1cfd700 1 -- 172.21.15.181:0/2370467622 learned_addr learned my addr 172.21.15.181:0/2370467622 (peer_addr_for_me v2:172.21.15.181:0/0) 2023-03-09T14:39:11.179 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:11.173+0000 7f2fa0cfb700 1 --2- 172.21.15.181:0/2370467622 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2f9c199ef0 0x7f2f9c19c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:39:11.179 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:11.173+0000 7f2fa14fc700 1 --2- 172.21.15.181:0/2370467622 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2f9c0fea20 0x7f2f9c0692c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:39:11.181 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:11.174+0000 7f2fa1cfd700 1 -- 172.21.15.181:0/2370467622 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2f9c0fea20 msgr2=0x7f2f9c0692c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:39:11.181 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:11.174+0000 7f2fa1cfd700 1 --2- 172.21.15.181:0/2370467622 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2f9c0fea20 0x7f2f9c0692c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:39:11.181 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:11.174+0000 7f2fa1cfd700 1 -- 172.21.15.181:0/2370467622 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2f9c199ef0 msgr2=0x7f2f9c19c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:39:11.181 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:11.174+0000 7f2fa1cfd700 1 --2- 172.21.15.181:0/2370467622 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2f9c199ef0 0x7f2f9c19c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:39:11.181 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:11.174+0000 7f2fa1cfd700 1 -- 172.21.15.181:0/2370467622 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f980095f0 con 0x7f2f9c19c810 2023-03-09T14:39:11.181 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:11.174+0000 7f2fa1cfd700 1 --2- 172.21.15.181:0/2370467622 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2f9c19c810 0x7f2f9c19ebf0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2f9400ba50 tx=0x7f2f9400bd60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:39:11.182 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:11.174+0000 7f2f927fc700 1 -- 172.21.15.181:0/2370467622 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f9400c930 con 0x7f2f9c19c810 2023-03-09T14:39:11.182 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:11.174+0000 7f2f927fc700 1 -- 172.21.15.181:0/2370467622 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2f94014440 con 0x7f2f9c19c810 2023-03-09T14:39:11.182 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:11.174+0000 7f2fa3760700 1 -- 172.21.15.181:0/2370467622 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f9c1048a0 con 0x7f2f9c19c810 2023-03-09T14:39:11.182 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:11.174+0000 7f2f927fc700 1 -- 172.21.15.181:0/2370467622 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f940075b0 con 0x7f2f9c19c810 2023-03-09T14:39:11.182 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:11.174+0000 7f2fa3760700 1 -- 172.21.15.181:0/2370467622 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f9c104710 con 0x7f2f9c19c810 2023-03-09T14:39:11.182 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:11.175+0000 7f2f927fc700 1 -- 172.21.15.181:0/2370467622 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 13) v1 ==== 44674+0+0 (secure 0 0 0) 0x7f2f9401b030 con 0x7f2f9c19c810 2023-03-09T14:39:11.182 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:11.175+0000 7f2f927fc700 1 --2- 172.21.15.181:0/2370467622 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f2f880382b0 0x7f2f8803a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:39:11.183 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:11.175+0000 7f2f927fc700 1 -- 172.21.15.181:0/2370467622 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f2f94049950 con 0x7f2f9c19c810 2023-03-09T14:39:11.183 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:11.175+0000 7f2fa3760700 1 -- 172.21.15.181:0/2370467622 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f9c18d5d0 con 0x7f2f9c19c810 2023-03-09T14:39:11.183 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:11.175+0000 7f2fa14fc700 1 --2- 172.21.15.181:0/2370467622 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f2f880382b0 0x7f2f8803a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:39:11.183 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:11.176+0000 7f2fa14fc700 1 --2- 172.21.15.181:0/2370467622 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f2f880382b0 0x7f2f8803a760 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f2f98005130 tx=0x7f2f98005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:39:11.184 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:11.180+0000 7f2f927fc700 1 -- 172.21.15.181:0/2370467622 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f2f94019070 con 0x7f2f9c19c810 2023-03-09T14:39:11.394 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:11.391+0000 7f2fa3760700 1 -- 172.21.15.181:0/2370467622 --> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi038=x;smithi122=y;smithi181=z", "target": ["mon-mgr", ""]}) v1 -- 0x7f2f9c061900 con 0x7f2f880382b0 2023-03-09T14:39:11.403 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:11.400+0000 7f2f927fc700 1 -- 172.21.15.181:0/2370467622 <== mgr.14150 v2:172.21.15.38:6800/3328887328 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f2f9c061900 con 0x7f2f880382b0 2023-03-09T14:39:11.403 INFO:teuthology.orchestra.run.smithi181.stdout:Scheduled mgr update... 2023-03-09T14:39:11.405 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:11.402+0000 7f2fa3760700 1 -- 172.21.15.181:0/2370467622 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f2f880382b0 msgr2=0x7f2f8803a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:39:11.405 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:11.402+0000 7f2fa3760700 1 --2- 172.21.15.181:0/2370467622 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f2f880382b0 0x7f2f8803a760 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f2f98005130 tx=0x7f2f98005fd0 comp rx=0 tx=0).stop 2023-03-09T14:39:11.406 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:11.402+0000 7f2fa3760700 1 -- 172.21.15.181:0/2370467622 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2f9c19c810 msgr2=0x7f2f9c19ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:39:11.406 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:11.402+0000 7f2fa3760700 1 --2- 172.21.15.181:0/2370467622 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2f9c19c810 0x7f2f9c19ebf0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2f9400ba50 tx=0x7f2f9400bd60 comp rx=0 tx=0).stop 2023-03-09T14:39:11.406 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:11.403+0000 7f2fa3760700 1 -- 172.21.15.181:0/2370467622 shutdown_connections 2023-03-09T14:39:11.406 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:11.403+0000 7f2fa3760700 1 --2- 172.21.15.181:0/2370467622 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f2f880382b0 0x7f2f8803a760 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:39:11.406 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:11.403+0000 7f2fa3760700 1 --2- 172.21.15.181:0/2370467622 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2f9c19c810 0x7f2f9c19ebf0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:39:11.406 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:11.403+0000 7f2fa3760700 1 --2- 172.21.15.181:0/2370467622 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2f9c0fea20 0x7f2f9c0692c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:39:11.407 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:11.403+0000 7f2fa3760700 1 --2- 172.21.15.181:0/2370467622 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2f9c199ef0 0x7f2f9c19c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:39:11.407 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:11.403+0000 7f2fa3760700 1 -- 172.21.15.181:0/2370467622 >> 172.21.15.181:0/2370467622 conn(0x7f2f9c0f9fc0 msgr2=0x7f2f9c0fac20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:39:11.407 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:11.403+0000 7f2fa3760700 1 -- 172.21.15.181:0/2370467622 shutdown_connections 2023-03-09T14:39:11.407 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:39:11.403+0000 7f2fa3760700 1 -- 172.21.15.181:0/2370467622 wait complete. 2023-03-09T14:39:11.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:11 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:11.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:11 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:11.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:11 smithi181 ceph-mon[94189]: Reconfiguring mon.c (monmap changed)... 2023-03-09T14:39:11.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:11 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-03-09T14:39:11.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:11 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-03-09T14:39:11.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:11 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:39:11.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:11 smithi181 ceph-mon[94189]: Reconfiguring daemon mon.c on smithi181 2023-03-09T14:39:11.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:11 smithi181 ceph-mon[94189]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:39:11.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:11 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:11.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:11 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:11.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:11 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:11.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:11 smithi122 ceph-mon[93869]: Reconfiguring mon.c (monmap changed)... 2023-03-09T14:39:11.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:11 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-03-09T14:39:11.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:11 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-03-09T14:39:11.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:11 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:39:11.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:11 smithi122 ceph-mon[93869]: Reconfiguring daemon mon.c on smithi181 2023-03-09T14:39:11.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:11 smithi122 ceph-mon[93869]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:39:11.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:11 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:11.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:11 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:11.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:11 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:11.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:11 smithi038 ceph-mon[92422]: Reconfiguring mon.c (monmap changed)... 2023-03-09T14:39:11.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:11 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-03-09T14:39:11.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:11 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-03-09T14:39:11.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:11 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:39:11.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:11 smithi038 ceph-mon[92422]: Reconfiguring daemon mon.c on smithi181 2023-03-09T14:39:11.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:11 smithi038 ceph-mon[92422]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:39:11.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:11 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:11.888 DEBUG:teuthology.orchestra.run.smithi122:mgr.y> sudo journalctl -f -n 0 -u ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@mgr.y.service 2023-03-09T14:39:11.891 DEBUG:teuthology.orchestra.run.smithi181:mgr.z> sudo journalctl -f -n 0 -u ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@mgr.z.service 2023-03-09T14:39:11.893 INFO:tasks.cephadm:Deploying OSDs... 2023-03-09T14:39:11.893 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2023-03-09T14:39:11.894 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/scratch_devs of=/dev/stdout 2023-03-09T14:39:11.915 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-03-09T14:39:11.915 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_1 2023-03-09T14:39:11.922 INFO:journalctl@ceph.mgr.z.smithi181.stdout:-- Logs begin at Thu 2023-03-09 14:25:01 UTC. -- 2023-03-09T14:39:11.924 INFO:journalctl@ceph.mgr.y.smithi122.stdout:-- Logs begin at Thu 2023-03-09 14:25:09 UTC. -- 2023-03-09T14:39:11.975 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-03-09T14:39:11.976 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-03-09T14:39:11.976 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 6h/6d Inode: 355751 Links: 1 2023-03-09T14:39:11.976 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-03-09T14:39:11.976 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:device_t:s0 2023-03-09T14:39:11.976 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2023-03-09 14:38:25.517747521 +0000 2023-03-09T14:39:11.976 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2023-03-09 14:33:14.835707933 +0000 2023-03-09T14:39:11.977 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2023-03-09 14:33:14.835707933 +0000 2023-03-09T14:39:11.977 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: - 2023-03-09T14:39:11.977 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-03-09T14:39:12.048 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2023-03-09T14:39:12.049 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2023-03-09T14:39:12.049 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000303957 s, 1.7 MB/s 2023-03-09T14:39:12.050 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-03-09T14:39:12.111 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_2 2023-03-09T14:39:12.170 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-03-09T14:39:12.170 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-03-09T14:39:12.170 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 6h/6d Inode: 358281 Links: 1 2023-03-09T14:39:12.171 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-03-09T14:39:12.171 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:device_t:s0 2023-03-09T14:39:12.171 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2023-03-09 14:38:25.517747521 +0000 2023-03-09T14:39:12.171 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2023-03-09 14:33:15.374697642 +0000 2023-03-09T14:39:12.171 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2023-03-09 14:33:15.374697642 +0000 2023-03-09T14:39:12.171 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: - 2023-03-09T14:39:12.172 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-03-09T14:39:12.238 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2023-03-09T14:39:12.239 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2023-03-09T14:39:12.239 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000316132 s, 1.6 MB/s 2023-03-09T14:39:12.240 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-03-09T14:39:12.299 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_3 2023-03-09T14:39:12.358 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-03-09T14:39:12.359 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-03-09T14:39:12.359 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 6h/6d Inode: 355982 Links: 1 2023-03-09T14:39:12.359 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-03-09T14:39:12.359 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:device_t:s0 2023-03-09T14:39:12.359 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2023-03-09 14:38:25.517747521 +0000 2023-03-09T14:39:12.360 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2023-03-09 14:33:15.898687638 +0000 2023-03-09T14:39:12.360 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2023-03-09 14:33:15.898687638 +0000 2023-03-09T14:39:12.360 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: - 2023-03-09T14:39:12.360 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-03-09T14:39:12.426 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2023-03-09T14:39:12.427 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2023-03-09T14:39:12.427 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000307818 s, 1.7 MB/s 2023-03-09T14:39:12.428 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-03-09T14:39:12.487 DEBUG:teuthology.orchestra.run.smithi038:> stat /dev/vg_nvme/lv_4 2023-03-09T14:39:12.544 INFO:teuthology.orchestra.run.smithi038.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-03-09T14:39:12.545 INFO:teuthology.orchestra.run.smithi038.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-03-09T14:39:12.545 INFO:teuthology.orchestra.run.smithi038.stdout:Device: 6h/6d Inode: 365858 Links: 1 2023-03-09T14:39:12.545 INFO:teuthology.orchestra.run.smithi038.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-03-09T14:39:12.545 INFO:teuthology.orchestra.run.smithi038.stdout:Context: system_u:object_r:device_t:s0 2023-03-09T14:39:12.545 INFO:teuthology.orchestra.run.smithi038.stdout:Access: 2023-03-09 14:38:25.517747521 +0000 2023-03-09T14:39:12.546 INFO:teuthology.orchestra.run.smithi038.stdout:Modify: 2023-03-09 14:33:16.443677232 +0000 2023-03-09T14:39:12.546 INFO:teuthology.orchestra.run.smithi038.stdout:Change: 2023-03-09 14:33:16.443677232 +0000 2023-03-09T14:39:12.546 INFO:teuthology.orchestra.run.smithi038.stdout: Birth: - 2023-03-09T14:39:12.546 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-03-09T14:39:12.611 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records in 2023-03-09T14:39:12.612 INFO:teuthology.orchestra.run.smithi038.stderr:1+0 records out 2023-03-09T14:39:12.612 INFO:teuthology.orchestra.run.smithi038.stderr:512 bytes copied, 0.000376134 s, 1.4 MB/s 2023-03-09T14:39:12.613 DEBUG:teuthology.orchestra.run.smithi038:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-03-09T14:39:12.671 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2023-03-09T14:39:12.672 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/scratch_devs of=/dev/stdout 2023-03-09T14:39:12.693 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-03-09T14:39:12.693 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_1 2023-03-09T14:39:12.754 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-03-09T14:39:12.755 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-03-09T14:39:12.755 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 6h/6d Inode: 362801 Links: 1 2023-03-09T14:39:12.755 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-03-09T14:39:12.755 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:device_t:s0 2023-03-09T14:39:12.756 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2023-03-09 14:38:59.830169828 +0000 2023-03-09T14:39:12.756 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2023-03-09 14:33:10.430866952 +0000 2023-03-09T14:39:12.756 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2023-03-09 14:33:10.430866952 +0000 2023-03-09T14:39:12.756 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: - 2023-03-09T14:39:12.757 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-03-09T14:39:12.826 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2023-03-09T14:39:12.827 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2023-03-09T14:39:12.827 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000313252 s, 1.6 MB/s 2023-03-09T14:39:12.828 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-03-09T14:39:12.890 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_2 2023-03-09T14:39:12.951 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-03-09T14:39:12.951 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-03-09T14:39:12.951 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 6h/6d Inode: 361229 Links: 1 2023-03-09T14:39:12.951 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-03-09T14:39:12.951 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:device_t:s0 2023-03-09T14:39:12.952 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2023-03-09 14:38:59.830169828 +0000 2023-03-09T14:39:12.952 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2023-03-09 14:33:10.961854437 +0000 2023-03-09T14:39:12.952 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2023-03-09 14:33:10.961854437 +0000 2023-03-09T14:39:12.952 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: - 2023-03-09T14:39:12.952 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-03-09T14:39:13.023 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2023-03-09T14:39:13.023 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2023-03-09T14:39:13.023 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000223285 s, 2.3 MB/s 2023-03-09T14:39:13.024 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-03-09T14:39:13.086 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_3 2023-03-09T14:39:13.147 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-03-09T14:39:13.148 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-03-09T14:39:13.148 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 6h/6d Inode: 361335 Links: 1 2023-03-09T14:39:13.148 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-03-09T14:39:13.148 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:device_t:s0 2023-03-09T14:39:13.148 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2023-03-09 14:38:59.831169804 +0000 2023-03-09T14:39:13.149 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2023-03-09 14:33:11.476842299 +0000 2023-03-09T14:39:13.149 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2023-03-09 14:33:11.476842299 +0000 2023-03-09T14:39:13.149 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: - 2023-03-09T14:39:13.149 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-03-09T14:39:13.219 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2023-03-09T14:39:13.219 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2023-03-09T14:39:13.220 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000286422 s, 1.8 MB/s 2023-03-09T14:39:13.221 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-03-09T14:39:13.282 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_4 2023-03-09T14:39:13.344 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-03-09T14:39:13.344 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-03-09T14:39:13.344 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 6h/6d Inode: 364732 Links: 1 2023-03-09T14:39:13.344 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-03-09T14:39:13.345 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:device_t:s0 2023-03-09T14:39:13.345 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2023-03-09 14:38:59.831169804 +0000 2023-03-09T14:39:13.345 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2023-03-09 14:33:12.111827332 +0000 2023-03-09T14:39:13.345 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2023-03-09 14:33:12.111827332 +0000 2023-03-09T14:39:13.345 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: - 2023-03-09T14:39:13.345 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-03-09T14:39:13.374 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:13 smithi181 ceph-mon[94189]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi038=x;smithi122=y;smithi181=z", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:39:13.375 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:13 smithi181 ceph-mon[94189]: Saving service mgr spec with placement smithi038=x;smithi122=y;smithi181=z;count:3 2023-03-09T14:39:13.375 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:13 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:13.375 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:13 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:13.375 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:13 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:39:13.375 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:13 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:39:13.375 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:13 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:39:13.376 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:13 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:13.376 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:13 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-03-09T14:39:13.376 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:13 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-03-09T14:39:13.376 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:13 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-03-09T14:39:13.376 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:13 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:39:13.376 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:13 smithi181 ceph-mon[94189]: Deploying daemon mgr.z on smithi181 2023-03-09T14:39:13.412 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:13 smithi122 ceph-mon[93869]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi038=x;smithi122=y;smithi181=z", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:39:13.412 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:13 smithi122 ceph-mon[93869]: Saving service mgr spec with placement smithi038=x;smithi122=y;smithi181=z;count:3 2023-03-09T14:39:13.412 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:13 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:13.413 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:13 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:13.413 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:13 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:39:13.413 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:13 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:39:13.413 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:13 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:39:13.413 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:13 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:13.413 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:13 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-03-09T14:39:13.413 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:13 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-03-09T14:39:13.414 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:13 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-03-09T14:39:13.414 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:13 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:39:13.414 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:13 smithi122 ceph-mon[93869]: Deploying daemon mgr.z on smithi181 2023-03-09T14:39:13.415 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2023-03-09T14:39:13.415 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2023-03-09T14:39:13.415 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000282581 s, 1.8 MB/s 2023-03-09T14:39:13.416 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-03-09T14:39:13.477 DEBUG:teuthology.orchestra.run.smithi181:> set -ex 2023-03-09T14:39:13.478 DEBUG:teuthology.orchestra.run.smithi181:> dd if=/scratch_devs of=/dev/stdout 2023-03-09T14:39:13.496 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-03-09T14:39:13.496 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_1 2023-03-09T14:39:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:13 smithi038 ceph-mon[92422]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi038=x;smithi122=y;smithi181=z", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:39:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:13 smithi038 ceph-mon[92422]: Saving service mgr spec with placement smithi038=x;smithi122=y;smithi181=z;count:3 2023-03-09T14:39:13.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:13 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:13.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:13 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:13.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:13 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:39:13.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:13 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:39:13.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:13 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:39:13.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:13 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:13.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:13 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-03-09T14:39:13.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:13 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-03-09T14:39:13.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:13 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-03-09T14:39:13.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:13 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:39:13.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:13 smithi038 ceph-mon[92422]: Deploying daemon mgr.z on smithi181 2023-03-09T14:39:13.553 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-03-09T14:39:13.553 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-03-09T14:39:13.553 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 6h/6d Inode: 354855 Links: 1 2023-03-09T14:39:13.553 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-03-09T14:39:13.554 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2023-03-09T14:39:13.554 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2023-03-09 14:39:13.541628588 +0000 2023-03-09T14:39:13.554 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2023-03-09 14:33:10.072719874 +0000 2023-03-09T14:39:13.554 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2023-03-09 14:33:10.072719874 +0000 2023-03-09T14:39:13.554 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: - 2023-03-09T14:39:13.554 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-03-09T14:39:13.625 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2023-03-09T14:39:13.625 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2023-03-09T14:39:13.626 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000261522 s, 2.0 MB/s 2023-03-09T14:39:13.626 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 09 14:39:13 smithi181 systemd[1]: Starting Ceph mgr.z for c6733fe2-be87-11ed-9af2-001a4aab830c... 2023-03-09T14:39:13.627 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-03-09T14:39:13.686 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_2 2023-03-09T14:39:13.744 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-03-09T14:39:13.745 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-03-09T14:39:13.745 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 6h/6d Inode: 356127 Links: 1 2023-03-09T14:39:13.746 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-03-09T14:39:13.746 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2023-03-09T14:39:13.746 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2023-03-09 14:39:13.542628569 +0000 2023-03-09T14:39:13.747 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2023-03-09 14:33:10.592709647 +0000 2023-03-09T14:39:13.747 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2023-03-09 14:33:10.592709647 +0000 2023-03-09T14:39:13.747 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: - 2023-03-09T14:39:13.748 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-03-09T14:39:13.810 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2023-03-09T14:39:13.810 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2023-03-09T14:39:13.811 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000218741 s, 2.3 MB/s 2023-03-09T14:39:13.811 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-03-09T14:39:13.870 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_3 2023-03-09T14:39:13.928 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-03-09T14:39:13.928 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-03-09T14:39:13.928 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 6h/6d Inode: 364645 Links: 1 2023-03-09T14:39:13.928 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-03-09T14:39:13.929 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2023-03-09T14:39:13.929 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2023-03-09 14:39:13.542628569 +0000 2023-03-09T14:39:13.929 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2023-03-09 14:33:11.128699104 +0000 2023-03-09T14:39:13.929 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2023-03-09 14:33:11.128699104 +0000 2023-03-09T14:39:13.929 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: - 2023-03-09T14:39:13.929 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-03-09T14:39:13.981 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 09 14:39:13 smithi181 podman[96137]: 2023-03-09T14:39:13.993 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2023-03-09T14:39:13.994 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2023-03-09T14:39:13.994 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000223779 s, 2.3 MB/s 2023-03-09T14:39:13.995 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-03-09T14:39:14.051 DEBUG:teuthology.orchestra.run.smithi181:> stat /dev/vg_nvme/lv_4 2023-03-09T14:39:14.109 INFO:teuthology.orchestra.run.smithi181.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-03-09T14:39:14.109 INFO:teuthology.orchestra.run.smithi181.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-03-09T14:39:14.109 INFO:teuthology.orchestra.run.smithi181.stdout:Device: 6h/6d Inode: 366692 Links: 1 2023-03-09T14:39:14.109 INFO:teuthology.orchestra.run.smithi181.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-03-09T14:39:14.110 INFO:teuthology.orchestra.run.smithi181.stdout:Context: system_u:object_r:device_t:s0 2023-03-09T14:39:14.110 INFO:teuthology.orchestra.run.smithi181.stdout:Access: 2023-03-09 14:39:13.542628569 +0000 2023-03-09T14:39:14.110 INFO:teuthology.orchestra.run.smithi181.stdout:Modify: 2023-03-09 14:33:11.657688700 +0000 2023-03-09T14:39:14.110 INFO:teuthology.orchestra.run.smithi181.stdout:Change: 2023-03-09 14:33:11.657688700 +0000 2023-03-09T14:39:14.110 INFO:teuthology.orchestra.run.smithi181.stdout: Birth: - 2023-03-09T14:39:14.110 DEBUG:teuthology.orchestra.run.smithi181:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-03-09T14:39:14.174 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records in 2023-03-09T14:39:14.175 INFO:teuthology.orchestra.run.smithi181.stderr:1+0 records out 2023-03-09T14:39:14.175 INFO:teuthology.orchestra.run.smithi181.stderr:512 bytes copied, 0.000181076 s, 2.8 MB/s 2023-03-09T14:39:14.176 DEBUG:teuthology.orchestra.run.smithi181:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-03-09T14:39:14.235 INFO:tasks.cephadm:Deploying osd.0 on smithi038 with /dev/vg_nvme/lv_4... 2023-03-09T14:39:14.235 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-03-09T14:39:14.279 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:14 smithi181 ceph-mon[94189]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:39:14.280 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 09 14:39:14 smithi181 bash[96137]: 5cd1ddc89e0794a5b3f238d96af47486fb4328bd942d998146bcfd4d2a98dbee 2023-03-09T14:39:14.280 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 09 14:39:14 smithi181 ceph-mgr[96301]: -- 172.21.15.181:0/1864831172 <== mon.1 v2:172.21.15.181:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5589f1cbeb60 con 0x5589f1c92800 2023-03-09T14:39:14.280 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 09 14:39:14 smithi181 systemd[1]: Started Ceph mgr.z for c6733fe2-be87-11ed-9af2-001a4aab830c. 2023-03-09T14:39:14.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:14 smithi122 ceph-mon[93869]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:39:14.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:14 smithi038 ceph-mon[92422]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:39:14.538 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 09 14:39:14 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-z[96215]: 2023-03-09T14:39:14.276+0000 7fad3697d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-03-09T14:39:14.539 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 09 14:39:14 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-z[96215]: 2023-03-09T14:39:14.355+0000 7fad3697d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-03-09T14:39:14.539 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 09 14:39:14 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-z[96215]: 2023-03-09T14:39:14.443+0000 7fad3697d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-03-09T14:39:14.899 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.a/config 2023-03-09T14:39:14.981 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 09 14:39:14 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-z[96215]: 2023-03-09T14:39:14.536+0000 7fad3697d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-03-09T14:39:14.982 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 09 14:39:14 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-z[96215]: 2023-03-09T14:39:14.703+0000 7fad3697d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-03-09T14:39:15.449 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:15 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:15.449 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:15 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:15.449 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:15 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-03-09T14:39:15.449 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:15 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-03-09T14:39:15.450 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:15 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-03-09T14:39:15.450 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:15 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:39:15.450 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:15 smithi038 ceph-mon[92422]: Deploying daemon mgr.y on smithi122 2023-03-09T14:39:15.535 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:15 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:15.536 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:15 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:15.536 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:15 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-03-09T14:39:15.536 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:15 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-03-09T14:39:15.536 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:15 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-03-09T14:39:15.536 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:15 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:39:15.536 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:15 smithi122 ceph-mon[93869]: Deploying daemon mgr.y on smithi122 2023-03-09T14:39:15.632 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 09 14:39:15 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-z[96215]: 2023-03-09T14:39:15.305+0000 7fad3697d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-03-09T14:39:15.632 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 09 14:39:15 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-z[96215]: 2023-03-09T14:39:15.393+0000 7fad3697d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-03-09T14:39:15.633 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 09 14:39:15 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-z[96215]: 2023-03-09T14:39:15.480+0000 7fad3697d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-03-09T14:39:15.633 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 09 14:39:15 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-z[96215]: 2023-03-09T14:39:15.629+0000 7fad3697d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-03-09T14:39:15.633 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:15 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:15.633 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:15 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:15.633 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:15 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-03-09T14:39:15.634 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:15 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-03-09T14:39:15.634 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:15 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-03-09T14:39:15.634 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:15 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:39:15.634 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:15 smithi181 ceph-mon[94189]: Deploying daemon mgr.y on smithi122 2023-03-09T14:39:15.978 INFO:journalctl@ceph.mgr.y.smithi122.stdout:Mar 09 14:39:15 smithi122 systemd[1]: Starting Ceph mgr.y for c6733fe2-be87-11ed-9af2-001a4aab830c... 2023-03-09T14:39:16.338 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:16 smithi122 ceph-mon[93869]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:39:16.338 INFO:journalctl@ceph.mgr.y.smithi122.stdout:Mar 09 14:39:16 smithi122 podman[95302]: 2023-03-09T14:39:16.338 INFO:journalctl@ceph.mgr.y.smithi122.stdout:Mar 09 14:39:16 smithi122 bash[95302]: d96adcd25e38fb30ad7f5f36057f4b12823bd9bc4fc2ad6e746c5d2bbea3be34 2023-03-09T14:39:16.338 INFO:journalctl@ceph.mgr.y.smithi122.stdout:Mar 09 14:39:16 smithi122 ceph-mgr[95350]: -- 172.21.15.122:0/182796023 <== mon.2 v2:172.21.15.122:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a455ff0b60 con 0x55a455fc4c00 2023-03-09T14:39:16.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:16 smithi181 ceph-mon[94189]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:39:16.481 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 09 14:39:16 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-z[96215]: 2023-03-09T14:39:16.059+0000 7fad3697d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-03-09T14:39:16.505 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-03-09T14:39:16.506 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-03-09T14:39:16.506 INFO:teuthology.orchestra.run.smithi038.stderr: stderr: 10+0 records in 2023-03-09T14:39:16.506 INFO:teuthology.orchestra.run.smithi038.stderr:10+0 records out 2023-03-09T14:39:16.506 INFO:teuthology.orchestra.run.smithi038.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0160846 s, 652 MB/s 2023-03-09T14:39:16.506 INFO:teuthology.orchestra.run.smithi038.stderr: stderr: 2023-03-09T14:39:16.506 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping successful for: 2023-03-09T14:39:16.550 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:16 smithi038 ceph-mon[92422]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:39:16.590 INFO:journalctl@ceph.mgr.y.smithi122.stdout:Mar 09 14:39:16 smithi122 systemd[1]: Started Ceph mgr.y for c6733fe2-be87-11ed-9af2-001a4aab830c. 2023-03-09T14:39:16.590 INFO:journalctl@ceph.mgr.y.smithi122.stdout:Mar 09 14:39:16 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-y[95318]: 2023-03-09T14:39:16.411+0000 7fc0e9f71200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-03-09T14:39:16.591 INFO:journalctl@ceph.mgr.y.smithi122.stdout:Mar 09 14:39:16 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-y[95318]: 2023-03-09T14:39:16.493+0000 7fc0e9f71200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-03-09T14:39:16.867 INFO:journalctl@ceph.mgr.y.smithi122.stdout:Mar 09 14:39:16 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-y[95318]: 2023-03-09T14:39:16.587+0000 7fc0e9f71200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-03-09T14:39:16.867 INFO:journalctl@ceph.mgr.y.smithi122.stdout:Mar 09 14:39:16 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-y[95318]: 2023-03-09T14:39:16.686+0000 7fc0e9f71200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-03-09T14:39:16.948 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph orch daemon add osd smithi038:vg_nvme/lv_4 2023-03-09T14:39:16.981 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 09 14:39:16 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-z[96215]: 2023-03-09T14:39:16.661+0000 7fad3697d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-03-09T14:39:16.982 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 09 14:39:16 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-z[96215]: 2023-03-09T14:39:16.747+0000 7fad3697d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-03-09T14:39:17.227 INFO:journalctl@ceph.mgr.y.smithi122.stdout:Mar 09 14:39:16 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-y[95318]: 2023-03-09T14:39:16.863+0000 7fc0e9f71200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-03-09T14:39:17.603 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.a/config 2023-03-09T14:39:17.697 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:17 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:17.698 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:17 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:17.698 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:17 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:39:17.698 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:17 smithi122 ceph-mon[93869]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:39:17.698 INFO:journalctl@ceph.mgr.y.smithi122.stdout:Mar 09 14:39:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-y[95318]: 2023-03-09T14:39:17.499+0000 7fc0e9f71200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-03-09T14:39:17.699 INFO:journalctl@ceph.mgr.y.smithi122.stdout:Mar 09 14:39:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-y[95318]: 2023-03-09T14:39:17.596+0000 7fc0e9f71200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-03-09T14:39:17.699 INFO:journalctl@ceph.mgr.y.smithi122.stdout:Mar 09 14:39:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-y[95318]: 2023-03-09T14:39:17.693+0000 7fc0e9f71200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-03-09T14:39:17.731 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 09 14:39:17 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-z[96215]: 2023-03-09T14:39:17.540+0000 7fad3697d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-03-09T14:39:17.731 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 09 14:39:17 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-z[96215]: 2023-03-09T14:39:17.639+0000 7fad3697d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-03-09T14:39:17.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:17 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:17.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:17 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:17.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:17 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:39:17.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:17 smithi181 ceph-mon[94189]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:39:17.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:17 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:17.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:17 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:17.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:17 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:39:17.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:17 smithi038 ceph-mon[92422]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:39:17.977 INFO:journalctl@ceph.mgr.y.smithi122.stdout:Mar 09 14:39:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-y[95318]: 2023-03-09T14:39:17.861+0000 7fc0e9f71200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-03-09T14:39:18.231 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 09 14:39:17 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-z[96215]: 2023-03-09T14:39:17.821+0000 7fad3697d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-03-09T14:39:18.727 INFO:journalctl@ceph.mgr.y.smithi122.stdout:Mar 09 14:39:18 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-y[95318]: 2023-03-09T14:39:18.308+0000 7fc0e9f71200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-03-09T14:39:18.731 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 09 14:39:18 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-z[96215]: 2023-03-09T14:39:18.252+0000 7fad3697d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-03-09T14:39:18.731 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 09 14:39:18 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-z[96215]: 2023-03-09T14:39:18.339+0000 7fad3697d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-03-09T14:39:18.731 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 09 14:39:18 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-z[96215]: 2023-03-09T14:39:18.484+0000 7fad3697d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-03-09T14:39:18.811 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:18.808+0000 7fa7bd1bf700 1 -- 172.21.15.38:0/229944957 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa7b80ff1c0 msgr2=0x7fa7b80ff610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:39:18.811 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:18.808+0000 7fa7bd1bf700 1 --2- 172.21.15.38:0/229944957 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa7b80ff1c0 0x7fa7b80ff610 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa7a0009990 tx=0x7fa7a0009ca0 comp rx=0 tx=0).stop 2023-03-09T14:39:18.812 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:18.808+0000 7fa7bd1bf700 1 -- 172.21.15.38:0/229944957 shutdown_connections 2023-03-09T14:39:18.812 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:18.808+0000 7fa7bd1bf700 1 --2- 172.21.15.38:0/229944957 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa7b80ffb50 0x7fa7b8107420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:39:18.812 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:18.808+0000 7fa7bd1bf700 1 --2- 172.21.15.38:0/229944957 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa7b8100490 0x7fa7b81008a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:39:18.812 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:18.808+0000 7fa7bd1bf700 1 --2- 172.21.15.38:0/229944957 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa7b80ff1c0 0x7fa7b80ff610 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:39:18.812 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:18.808+0000 7fa7bd1bf700 1 -- 172.21.15.38:0/229944957 >> 172.21.15.38:0/229944957 conn(0x7fa7b80f9c80 msgr2=0x7fa7b80fc0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:39:18.812 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:18.808+0000 7fa7bd1bf700 1 -- 172.21.15.38:0/229944957 shutdown_connections 2023-03-09T14:39:18.812 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:18.808+0000 7fa7bd1bf700 1 -- 172.21.15.38:0/229944957 wait complete. 2023-03-09T14:39:18.814 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:18.809+0000 7fa7bd1bf700 1 Processor -- start 2023-03-09T14:39:18.814 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:18.809+0000 7fa7bd1bf700 1 -- start start 2023-03-09T14:39:18.815 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:18.809+0000 7fa7bd1bf700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa7b80ff1c0 0x7fa7b8195d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:39:18.815 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:18.809+0000 7fa7bd1bf700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa7b80ffb50 0x7fa7b81962a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:39:18.815 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:18.809+0000 7fa7bd1bf700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa7b8100490 0x7fa7b819d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:39:18.815 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:18.809+0000 7fa7bd1bf700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b819d8e0 con 0x7fa7b80ff1c0 2023-03-09T14:39:18.815 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:18.809+0000 7fa7bd1bf700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b819da20 con 0x7fa7b8100490 2023-03-09T14:39:18.815 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:18.809+0000 7fa7bd1bf700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b819dbb0 con 0x7fa7b80ffb50 2023-03-09T14:39:18.815 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:18.809+0000 7fa7b759e700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa7b8100490 0x7fa7b819d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:39:18.815 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:18.809+0000 7fa7b659c700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa7b80ffb50 0x7fa7b81962a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:39:18.816 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:18.809+0000 7fa7b659c700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa7b80ffb50 0x7fa7b81962a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.38:43476/0 (socket says 172.21.15.38:43476) 2023-03-09T14:39:18.816 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:18.809+0000 7fa7b759e700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa7b8100490 0x7fa7b819d310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.38:44970/0 (socket says 172.21.15.38:44970) 2023-03-09T14:39:18.816 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:18.809+0000 7fa7b659c700 1 -- 172.21.15.38:0/1594793420 learned_addr learned my addr 172.21.15.38:0/1594793420 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:39:18.816 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:18.809+0000 7fa7b6d9d700 1 --2- 172.21.15.38:0/1594793420 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa7b80ff1c0 0x7fa7b8195d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:39:18.816 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:18.809+0000 7fa7b659c700 1 -- 172.21.15.38:0/1594793420 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa7b8100490 msgr2=0x7fa7b819d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:39:18.816 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:18.809+0000 7fa7b659c700 1 --2- 172.21.15.38:0/1594793420 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa7b8100490 0x7fa7b819d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:39:18.816 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:18.809+0000 7fa7b659c700 1 -- 172.21.15.38:0/1594793420 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa7b80ff1c0 msgr2=0x7fa7b8195d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:39:18.817 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:18.809+0000 7fa7b659c700 1 --2- 172.21.15.38:0/1594793420 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa7b80ff1c0 0x7fa7b8195d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:39:18.817 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:18.809+0000 7fa7b659c700 1 -- 172.21.15.38:0/1594793420 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7a00095f0 con 0x7fa7b80ffb50 2023-03-09T14:39:18.817 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:18.809+0000 7fa7b759e700 1 --2- 172.21.15.38:0/1594793420 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa7b8100490 0x7fa7b819d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:39:18.817 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:18.810+0000 7fa7b659c700 1 --2- 172.21.15.38:0/1594793420 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa7b80ffb50 0x7fa7b81962a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa7a000b680 tx=0x7fa7a00057b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:39:18.817 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:18.810+0000 7fa7affff700 1 -- 172.21.15.38:0/1594793420 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7a001d070 con 0x7fa7b80ffb50 2023-03-09T14:39:18.817 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:18.810+0000 7fa7bd1bf700 1 -- 172.21.15.38:0/1594793420 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7b810a8a0 con 0x7fa7b80ffb50 2023-03-09T14:39:18.817 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:18.810+0000 7fa7affff700 1 -- 172.21.15.38:0/1594793420 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa7a000fbc0 con 0x7fa7b80ffb50 2023-03-09T14:39:18.818 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:18.810+0000 7fa7affff700 1 -- 172.21.15.38:0/1594793420 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7a0022ea0 con 0x7fa7b80ffb50 2023-03-09T14:39:18.818 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:18.810+0000 7fa7bd1bf700 1 -- 172.21.15.38:0/1594793420 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7b810a710 con 0x7fa7b80ffb50 2023-03-09T14:39:18.818 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:18.811+0000 7fa7affff700 1 -- 172.21.15.38:0/1594793420 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 13) v1 ==== 44674+0+0 (secure 0 0 0) 0x7fa7a0029030 con 0x7fa7b80ffb50 2023-03-09T14:39:18.818 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:18.811+0000 7fa7bd1bf700 1 -- 172.21.15.38:0/1594793420 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7b819e120 con 0x7fa7b80ffb50 2023-03-09T14:39:18.818 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:18.811+0000 7fa7affff700 1 --2- 172.21.15.38:0/1594793420 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fa7a4037f90 0x7fa7a403a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:39:18.818 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:18.811+0000 7fa7affff700 1 -- 172.21.15.38:0/1594793420 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fa7a004b6b0 con 0x7fa7b80ffb50 2023-03-09T14:39:18.818 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:18.811+0000 7fa7b6d9d700 1 --2- 172.21.15.38:0/1594793420 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fa7a4037f90 0x7fa7a403a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:39:18.819 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:18.812+0000 7fa7b6d9d700 1 --2- 172.21.15.38:0/1594793420 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fa7a4037f90 0x7fa7a403a440 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa7b8068af0 tx=0x7fa7a800b040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:39:18.819 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:18.816+0000 7fa7affff700 1 -- 172.21.15.38:0/1594793420 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7fa7a002a360 con 0x7fa7b80ffb50 2023-03-09T14:39:19.036 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:19.032+0000 7fa7bd1bf700 1 -- 172.21.15.38:0/1594793420 --> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fa7b80fe800 con 0x7fa7a4037f90 2023-03-09T14:39:19.104 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 09 14:39:18 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-z[96215]: 2023-03-09T14:39:18.742+0000 7fad3697d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-03-09T14:39:19.156 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:19 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:19.157 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:19 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:19.157 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:19 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:19.157 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:19 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:19.157 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:19 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:39:19.157 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:19 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:39:19.157 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:19 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:19.158 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:19 smithi038 ceph-mon[92422]: Reconfiguring mgr.x (unknown last config time)... 2023-03-09T14:39:19.158 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:19 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-03-09T14:39:19.158 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:19 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-03-09T14:39:19.158 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:19 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:39:19.158 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:19 smithi038 ceph-mon[92422]: Reconfiguring daemon mgr.x on smithi038 2023-03-09T14:39:19.158 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:19 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-03-09T14:39:19.158 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:19 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-03-09T14:39:19.158 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:19 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:39:19.227 INFO:journalctl@ceph.mgr.y.smithi122.stdout:Mar 09 14:39:18 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-y[95318]: 2023-03-09T14:39:18.895+0000 7fc0e9f71200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-03-09T14:39:19.228 INFO:journalctl@ceph.mgr.y.smithi122.stdout:Mar 09 14:39:18 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-y[95318]: 2023-03-09T14:39:18.977+0000 7fc0e9f71200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-03-09T14:39:19.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:19 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:19.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:19 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:19.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:19 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:19.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:19 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:19.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:19 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:39:19.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:19 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:39:19.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:19 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:19.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:19 smithi122 ceph-mon[93869]: Reconfiguring mgr.x (unknown last config time)... 2023-03-09T14:39:19.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:19 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-03-09T14:39:19.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:19 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-03-09T14:39:19.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:19 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:39:19.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:19 smithi122 ceph-mon[93869]: Reconfiguring daemon mgr.x on smithi038 2023-03-09T14:39:19.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:19 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-03-09T14:39:19.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:19 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-03-09T14:39:19.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:19 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:39:19.442 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:19 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:19.442 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:19 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:19.443 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:19 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:19.443 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:19 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:19.443 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:19 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:39:19.443 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:19 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:39:19.443 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:19 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:19.443 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:19 smithi181 ceph-mon[94189]: Reconfiguring mgr.x (unknown last config time)... 2023-03-09T14:39:19.444 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:19 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-03-09T14:39:19.444 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:19 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-03-09T14:39:19.444 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:19 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:39:19.444 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:19 smithi181 ceph-mon[94189]: Reconfiguring daemon mgr.x on smithi038 2023-03-09T14:39:19.444 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:19 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-03-09T14:39:19.444 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:19 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-03-09T14:39:19.444 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:19 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:39:19.731 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 09 14:39:19 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-z[96215]: 2023-03-09T14:39:19.439+0000 7fad3697d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-03-09T14:39:19.731 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 09 14:39:19 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-z[96215]: 2023-03-09T14:39:19.569+0000 7fad3697d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-03-09T14:39:19.991 INFO:journalctl@ceph.mgr.y.smithi122.stdout:Mar 09 14:39:19 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-y[95318]: 2023-03-09T14:39:19.729+0000 7fc0e9f71200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-03-09T14:39:19.992 INFO:journalctl@ceph.mgr.y.smithi122.stdout:Mar 09 14:39:19 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-y[95318]: 2023-03-09T14:39:19.820+0000 7fc0e9f71200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-03-09T14:39:20.086 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 09 14:39:19 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-z[96215]: 2023-03-09T14:39:19.757+0000 7fad3697d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-03-09T14:39:20.087 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 09 14:39:19 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-z[96215]: 2023-03-09T14:39:19.870+0000 7fad3697d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-03-09T14:39:20.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:20 smithi038 ceph-mon[92422]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:39:20.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:20 smithi038 ceph-mon[92422]: from='client.24118 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:39:20.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:20 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:20.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:20 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:20.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:20 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:39:20.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:20 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:20.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:20 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:39:20.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:20 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:39:20.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:20 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:20.404 INFO:journalctl@ceph.mgr.y.smithi122.stdout:Mar 09 14:39:19 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-y[95318]: 2023-03-09T14:39:19.988+0000 7fc0e9f71200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-03-09T14:39:20.404 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:20 smithi122 ceph-mon[93869]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:39:20.404 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:20 smithi122 ceph-mon[93869]: from='client.24118 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:39:20.404 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:20 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:20.405 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:20 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:20.405 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:20 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:39:20.405 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:20 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:20.405 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:20 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:39:20.405 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:20 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:39:20.405 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:20 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:20.481 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 09 14:39:20 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-z[96215]: 2023-03-09T14:39:20.083+0000 7fad3697d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-03-09T14:39:20.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:20 smithi181 ceph-mon[94189]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:39:20.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:20 smithi181 ceph-mon[94189]: from='client.24118 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:39:20.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:20 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:20.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:20 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:20.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:20 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:39:20.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:20 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:20.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:20 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:39:20.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:20 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:39:20.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:20 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:20.727 INFO:journalctl@ceph.mgr.y.smithi122.stdout:Mar 09 14:39:20 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-y[95318]: 2023-03-09T14:39:20.400+0000 7fc0e9f71200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-03-09T14:39:20.728 INFO:journalctl@ceph.mgr.y.smithi122.stdout:Mar 09 14:39:20 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-y[95318]: 2023-03-09T14:39:20.488+0000 7fc0e9f71200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-03-09T14:39:20.728 INFO:journalctl@ceph.mgr.y.smithi122.stdout:Mar 09 14:39:20 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-y[95318]: 2023-03-09T14:39:20.634+0000 7fc0e9f71200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-03-09T14:39:21.145 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:21.142+0000 7fa7affff700 1 -- 172.21.15.38:0/1594793420 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mgrmap(e 14) v1 ==== 88916+0+0 (secure 0 0 0) 0x7fa7a000c040 con 0x7fa7b80ffb50 2023-03-09T14:39:21.227 INFO:journalctl@ceph.mgr.y.smithi122.stdout:Mar 09 14:39:20 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-y[95318]: 2023-03-09T14:39:20.892+0000 7fc0e9f71200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-03-09T14:39:21.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:21 smithi122 ceph-mon[93869]: Standby manager daemon z started 2023-03-09T14:39:21.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:21 smithi122 ceph-mon[93869]: from='mgr.? 172.21.15.181:0/2248825088' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-03-09T14:39:21.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:21 smithi122 ceph-mon[93869]: from='mgr.? 172.21.15.181:0/2248825088' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-03-09T14:39:21.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:21 smithi122 ceph-mon[93869]: from='mgr.? 172.21.15.181:0/2248825088' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-03-09T14:39:21.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:21 smithi122 ceph-mon[93869]: from='mgr.? 172.21.15.181:0/2248825088' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-03-09T14:39:21.231 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 09 14:39:20 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-z[96215]: 2023-03-09T14:39:20.933+0000 7fad3697d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-03-09T14:39:21.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:21 smithi181 ceph-mon[94189]: Standby manager daemon z started 2023-03-09T14:39:21.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:21 smithi181 ceph-mon[94189]: from='mgr.? 172.21.15.181:0/2248825088' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-03-09T14:39:21.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:21 smithi181 ceph-mon[94189]: from='mgr.? 172.21.15.181:0/2248825088' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-03-09T14:39:21.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:21 smithi181 ceph-mon[94189]: from='mgr.? 172.21.15.181:0/2248825088' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-03-09T14:39:21.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:21 smithi181 ceph-mon[94189]: from='mgr.? 172.21.15.181:0/2248825088' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-03-09T14:39:21.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:21 smithi038 ceph-mon[92422]: Standby manager daemon z started 2023-03-09T14:39:21.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:21 smithi038 ceph-mon[92422]: from='mgr.? 172.21.15.181:0/2248825088' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-03-09T14:39:21.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:21 smithi038 ceph-mon[92422]: from='mgr.? 172.21.15.181:0/2248825088' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-03-09T14:39:21.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:21 smithi038 ceph-mon[92422]: from='mgr.? 172.21.15.181:0/2248825088' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-03-09T14:39:21.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:21 smithi038 ceph-mon[92422]: from='mgr.? 172.21.15.181:0/2248825088' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-03-09T14:39:21.905 INFO:journalctl@ceph.mgr.y.smithi122.stdout:Mar 09 14:39:21 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-y[95318]: 2023-03-09T14:39:21.585+0000 7fc0e9f71200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-03-09T14:39:21.905 INFO:journalctl@ceph.mgr.y.smithi122.stdout:Mar 09 14:39:21 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-y[95318]: 2023-03-09T14:39:21.715+0000 7fc0e9f71200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-03-09T14:39:21.906 INFO:journalctl@ceph.mgr.y.smithi122.stdout:Mar 09 14:39:21 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-y[95318]: 2023-03-09T14:39:21.902+0000 7fc0e9f71200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-03-09T14:39:22.227 INFO:journalctl@ceph.mgr.y.smithi122.stdout:Mar 09 14:39:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-y[95318]: 2023-03-09T14:39:22.014+0000 7fc0e9f71200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-03-09T14:39:22.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:22 smithi122 ceph-mon[93869]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:39:22.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:22 smithi122 ceph-mon[93869]: mgrmap e14: x(active, since 66s), standbys: z 2023-03-09T14:39:22.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:22 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-03-09T14:39:22.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:22 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/1270384565' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "51477039-15ee-4707-a34d-4ecfa18bebde"}]: dispatch 2023-03-09T14:39:22.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:22 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/1270384565' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "51477039-15ee-4707-a34d-4ecfa18bebde"}]': finished 2023-03-09T14:39:22.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:22 smithi122 ceph-mon[93869]: osdmap e5: 1 total, 0 up, 1 in 2023-03-09T14:39:22.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:22 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-03-09T14:39:22.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:22 smithi181 ceph-mon[94189]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:39:22.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:22 smithi181 ceph-mon[94189]: mgrmap e14: x(active, since 66s), standbys: z 2023-03-09T14:39:22.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:22 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-03-09T14:39:22.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:22 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/1270384565' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "51477039-15ee-4707-a34d-4ecfa18bebde"}]: dispatch 2023-03-09T14:39:22.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:22 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/1270384565' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "51477039-15ee-4707-a34d-4ecfa18bebde"}]': finished 2023-03-09T14:39:22.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:22 smithi181 ceph-mon[94189]: osdmap e5: 1 total, 0 up, 1 in 2023-03-09T14:39:22.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:22 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-03-09T14:39:22.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:22 smithi038 ceph-mon[92422]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:39:22.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:22 smithi038 ceph-mon[92422]: mgrmap e14: x(active, since 66s), standbys: z 2023-03-09T14:39:22.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:22 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-03-09T14:39:22.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:22 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/1270384565' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "51477039-15ee-4707-a34d-4ecfa18bebde"}]: dispatch 2023-03-09T14:39:22.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:22 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/1270384565' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "51477039-15ee-4707-a34d-4ecfa18bebde"}]': finished 2023-03-09T14:39:22.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:22 smithi038 ceph-mon[92422]: osdmap e5: 1 total, 0 up, 1 in 2023-03-09T14:39:22.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:22 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-03-09T14:39:22.727 INFO:journalctl@ceph.mgr.y.smithi122.stdout:Mar 09 14:39:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-y[95318]: 2023-03-09T14:39:22.227+0000 7fc0e9f71200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-03-09T14:39:23.158 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:23.154+0000 7fa7affff700 1 -- 172.21.15.38:0/1594793420 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mgrmap(e 15) v1 ==== 133158+0+0 (secure 0 0 0) 0x7fa7a0017bc0 con 0x7fa7b80ffb50 2023-03-09T14:39:23.478 INFO:journalctl@ceph.mgr.y.smithi122.stdout:Mar 09 14:39:23 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-y[95318]: 2023-03-09T14:39:23.069+0000 7fc0e9f71200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-03-09T14:39:23.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:23 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/739852805' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-03-09T14:39:23.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:23 smithi122 ceph-mon[93869]: Standby manager daemon y started 2023-03-09T14:39:23.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:23 smithi122 ceph-mon[93869]: from='mgr.? 172.21.15.122:0/2885520228' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-03-09T14:39:23.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:23 smithi122 ceph-mon[93869]: from='mgr.? 172.21.15.122:0/2885520228' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-03-09T14:39:23.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:23 smithi122 ceph-mon[93869]: from='mgr.? 172.21.15.122:0/2885520228' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-03-09T14:39:23.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:23 smithi122 ceph-mon[93869]: from='mgr.? 172.21.15.122:0/2885520228' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-03-09T14:39:23.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:23 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/739852805' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-03-09T14:39:23.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:23 smithi181 ceph-mon[94189]: Standby manager daemon y started 2023-03-09T14:39:23.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:23 smithi181 ceph-mon[94189]: from='mgr.? 172.21.15.122:0/2885520228' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-03-09T14:39:23.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:23 smithi181 ceph-mon[94189]: from='mgr.? 172.21.15.122:0/2885520228' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-03-09T14:39:23.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:23 smithi181 ceph-mon[94189]: from='mgr.? 172.21.15.122:0/2885520228' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-03-09T14:39:23.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:23 smithi181 ceph-mon[94189]: from='mgr.? 172.21.15.122:0/2885520228' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-03-09T14:39:23.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:23 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/739852805' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-03-09T14:39:23.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:23 smithi038 ceph-mon[92422]: Standby manager daemon y started 2023-03-09T14:39:23.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:23 smithi038 ceph-mon[92422]: from='mgr.? 172.21.15.122:0/2885520228' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-03-09T14:39:23.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:23 smithi038 ceph-mon[92422]: from='mgr.? 172.21.15.122:0/2885520228' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-03-09T14:39:23.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:23 smithi038 ceph-mon[92422]: from='mgr.? 172.21.15.122:0/2885520228' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-03-09T14:39:23.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:23 smithi038 ceph-mon[92422]: from='mgr.? 172.21.15.122:0/2885520228' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-03-09T14:39:24.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:24 smithi122 ceph-mon[93869]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:39:24.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:24 smithi122 ceph-mon[93869]: mgrmap e15: x(active, since 68s), standbys: z, y 2023-03-09T14:39:24.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:24 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-03-09T14:39:24.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:24 smithi181 ceph-mon[94189]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:39:24.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:24 smithi181 ceph-mon[94189]: mgrmap e15: x(active, since 68s), standbys: z, y 2023-03-09T14:39:24.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:24 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-03-09T14:39:24.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:24 smithi038 ceph-mon[92422]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:39:24.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:24 smithi038 ceph-mon[92422]: mgrmap e15: x(active, since 68s), standbys: z, y 2023-03-09T14:39:24.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:24 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-03-09T14:39:26.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:26 smithi122 ceph-mon[93869]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:39:26.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:26 smithi181 ceph-mon[94189]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:39:26.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:26 smithi038 ceph-mon[92422]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:39:28.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:28 smithi122 ceph-mon[93869]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:39:28.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:28 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-03-09T14:39:28.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:28 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:39:28.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:28 smithi181 ceph-mon[94189]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:39:28.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:28 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-03-09T14:39:28.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:28 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:39:28.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:28 smithi038 ceph-mon[92422]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:39:28.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:28 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-03-09T14:39:28.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:28 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:39:29.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:29 smithi122 ceph-mon[93869]: Deploying daemon osd.0 on smithi038 2023-03-09T14:39:29.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:29 smithi181 ceph-mon[94189]: Deploying daemon osd.0 on smithi038 2023-03-09T14:39:29.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:29 smithi038 ceph-mon[92422]: Deploying daemon osd.0 on smithi038 2023-03-09T14:39:30.374 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:30 smithi038 ceph-mon[92422]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:39:30.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:30 smithi122 ceph-mon[93869]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:39:30.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:30 smithi181 ceph-mon[94189]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:39:32.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:32 smithi122 ceph-mon[93869]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:39:32.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:32 smithi181 ceph-mon[94189]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:39:32.495 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:32 smithi038 ceph-mon[92422]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:39:33.932 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:33 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:33.932 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:33 smithi038 ceph-mon[92422]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:39:33.932 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:33 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:33.932 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:33 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:39:33.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:33 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:33.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:33 smithi122 ceph-mon[93869]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:39:33.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:33 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:33.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:33 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:39:33.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:33 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:33.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:33 smithi181 ceph-mon[94189]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:39:33.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:33 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:33.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:33 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:39:35.479 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:35.477+0000 7fa7affff700 1 -- 172.21.15.38:0/1594793420 <== mgr.14150 v2:172.21.15.38:6800/3328887328 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fa7b80fe800 con 0x7fa7a4037f90 2023-03-09T14:39:35.480 INFO:teuthology.orchestra.run.smithi038.stdout:Created osd(s) 0 on host 'smithi038' 2023-03-09T14:39:35.482 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:35.479+0000 7fa7bd1bf700 1 -- 172.21.15.38:0/1594793420 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fa7a4037f90 msgr2=0x7fa7a403a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:39:35.482 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:35.479+0000 7fa7bd1bf700 1 --2- 172.21.15.38:0/1594793420 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fa7a4037f90 0x7fa7a403a440 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa7b8068af0 tx=0x7fa7a800b040 comp rx=0 tx=0).stop 2023-03-09T14:39:35.482 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:35.479+0000 7fa7bd1bf700 1 -- 172.21.15.38:0/1594793420 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa7b80ffb50 msgr2=0x7fa7b81962a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:39:35.482 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:35.479+0000 7fa7bd1bf700 1 --2- 172.21.15.38:0/1594793420 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa7b80ffb50 0x7fa7b81962a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fa7a000b680 tx=0x7fa7a00057b0 comp rx=0 tx=0).stop 2023-03-09T14:39:35.482 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:35.480+0000 7fa7bd1bf700 1 -- 172.21.15.38:0/1594793420 shutdown_connections 2023-03-09T14:39:35.482 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:35.480+0000 7fa7bd1bf700 1 --2- 172.21.15.38:0/1594793420 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa7b80ff1c0 0x7fa7b8195d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:39:35.483 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:35.480+0000 7fa7bd1bf700 1 --2- 172.21.15.38:0/1594793420 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fa7a4037f90 0x7fa7a403a440 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:39:35.483 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:35.480+0000 7fa7bd1bf700 1 --2- 172.21.15.38:0/1594793420 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa7b8100490 0x7fa7b819d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:39:35.483 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:35.480+0000 7fa7bd1bf700 1 --2- 172.21.15.38:0/1594793420 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa7b80ffb50 0x7fa7b81962a0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:39:35.483 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:35.480+0000 7fa7bd1bf700 1 -- 172.21.15.38:0/1594793420 >> 172.21.15.38:0/1594793420 conn(0x7fa7b80f9c80 msgr2=0x7fa7b80fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:39:35.483 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:35.480+0000 7fa7bd1bf700 1 -- 172.21.15.38:0/1594793420 shutdown_connections 2023-03-09T14:39:35.483 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:39:35.480+0000 7fa7bd1bf700 1 -- 172.21.15.38:0/1594793420 wait complete. 2023-03-09T14:39:35.911 DEBUG:teuthology.orchestra.run.smithi038:osd.0> sudo journalctl -f -n 0 -u ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.0.service 2023-03-09T14:39:35.914 INFO:tasks.cephadm:Deploying osd.1 on smithi122 with /dev/vg_nvme/lv_4... 2023-03-09T14:39:35.914 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-03-09T14:39:35.948 INFO:journalctl@ceph.osd.0.smithi038.stdout:-- Logs begin at Thu 2023-03-09 14:24:59 UTC. -- 2023-03-09T14:39:36.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:35 smithi122 ceph-mon[93869]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:39:36.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:35 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:36.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:35 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:36.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:35 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:39:36.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:35 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:39:36.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:35 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:36.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:35 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:36.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:35 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:36.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:35 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:39:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:35 smithi181 ceph-mon[94189]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:39:36.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:35 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:36.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:35 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:36.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:35 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:39:36.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:35 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:39:36.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:35 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:36.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:35 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:36.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:35 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:35 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:39:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:35 smithi038 ceph-mon[92422]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:39:36.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:35 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:36.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:35 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:36.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:35 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:39:36.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:35 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:39:36.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:35 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:36.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:35 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:36.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:35 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:36.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:35 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:39:36.566 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.b/config 2023-03-09T14:39:36.737 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:39:36 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 2023-03-09T14:39:36.610+0000 7f26846b7640 -1 osd.0 0 log_to_monitors true 2023-03-09T14:39:37.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:36 smithi038 ceph-mon[92422]: from='osd.0 [v2:172.21.15.38:6802/1508072290,v1:172.21.15.38:6803/1508072290]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-03-09T14:39:37.047 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:36 smithi122 ceph-mon[93869]: from='osd.0 [v2:172.21.15.38:6802/1508072290,v1:172.21.15.38:6803/1508072290]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-03-09T14:39:37.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:36 smithi181 ceph-mon[94189]: from='osd.0 [v2:172.21.15.38:6802/1508072290,v1:172.21.15.38:6803/1508072290]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-03-09T14:39:38.084 INFO:teuthology.orchestra.run.smithi122.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-03-09T14:39:38.084 INFO:teuthology.orchestra.run.smithi122.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-03-09T14:39:38.084 INFO:teuthology.orchestra.run.smithi122.stderr: stderr: 10+0 records in 2023-03-09T14:39:38.084 INFO:teuthology.orchestra.run.smithi122.stderr:10+0 records out 2023-03-09T14:39:38.085 INFO:teuthology.orchestra.run.smithi122.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0106496 s, 985 MB/s 2023-03-09T14:39:38.085 INFO:teuthology.orchestra.run.smithi122.stderr:--> Zapping successful for: 2023-03-09T14:39:38.170 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:39:37 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 2023-03-09T14:39:37.910+0000 7f267863c700 -1 osd.0 0 waiting for initial osdmap 2023-03-09T14:39:38.171 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:39:37 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 2023-03-09T14:39:37.915+0000 7f267242d700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-09T14:39:38.171 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:37 smithi038 ceph-mon[92422]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:39:38.171 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:37 smithi038 ceph-mon[92422]: from='osd.0 [v2:172.21.15.38:6802/1508072290,v1:172.21.15.38:6803/1508072290]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-03-09T14:39:38.171 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:37 smithi038 ceph-mon[92422]: osdmap e6: 1 total, 0 up, 1 in 2023-03-09T14:39:38.171 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:37 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-03-09T14:39:38.172 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:37 smithi038 ceph-mon[92422]: from='osd.0 [v2:172.21.15.38:6802/1508072290,v1:172.21.15.38:6803/1508072290]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2023-03-09T14:39:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:37 smithi122 ceph-mon[93869]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:39:38.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:37 smithi122 ceph-mon[93869]: from='osd.0 [v2:172.21.15.38:6802/1508072290,v1:172.21.15.38:6803/1508072290]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-03-09T14:39:38.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:37 smithi122 ceph-mon[93869]: osdmap e6: 1 total, 0 up, 1 in 2023-03-09T14:39:38.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:37 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-03-09T14:39:38.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:37 smithi122 ceph-mon[93869]: from='osd.0 [v2:172.21.15.38:6802/1508072290,v1:172.21.15.38:6803/1508072290]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2023-03-09T14:39:38.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:37 smithi181 ceph-mon[94189]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:39:38.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:37 smithi181 ceph-mon[94189]: from='osd.0 [v2:172.21.15.38:6802/1508072290,v1:172.21.15.38:6803/1508072290]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-03-09T14:39:38.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:37 smithi181 ceph-mon[94189]: osdmap e6: 1 total, 0 up, 1 in 2023-03-09T14:39:38.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:37 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-03-09T14:39:38.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:37 smithi181 ceph-mon[94189]: from='osd.0 [v2:172.21.15.38:6802/1508072290,v1:172.21.15.38:6803/1508072290]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2023-03-09T14:39:38.518 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph orch daemon add osd smithi122:vg_nvme/lv_4 2023-03-09T14:39:39.154 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.b/config 2023-03-09T14:39:39.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:38 smithi122 ceph-mon[93869]: from='osd.0 [v2:172.21.15.38:6802/1508072290,v1:172.21.15.38:6803/1508072290]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2023-03-09T14:39:39.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:38 smithi122 ceph-mon[93869]: osdmap e7: 1 total, 0 up, 1 in 2023-03-09T14:39:39.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:38 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-03-09T14:39:39.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:38 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-03-09T14:39:39.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:38 smithi122 ceph-mon[93869]: Detected new or changed devices on smithi038 2023-03-09T14:39:39.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:38 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:39.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:38 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:39.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:38 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:39:39.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:38 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi038 to 17173M 2023-03-09T14:39:39.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:38 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:39.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:38 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:39:39.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:38 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:39:39.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:38 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:39.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:38 smithi181 ceph-mon[94189]: from='osd.0 [v2:172.21.15.38:6802/1508072290,v1:172.21.15.38:6803/1508072290]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2023-03-09T14:39:39.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:38 smithi181 ceph-mon[94189]: osdmap e7: 1 total, 0 up, 1 in 2023-03-09T14:39:39.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:38 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-03-09T14:39:39.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:38 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-03-09T14:39:39.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:38 smithi181 ceph-mon[94189]: Detected new or changed devices on smithi038 2023-03-09T14:39:39.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:38 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:39.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:38 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:39.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:38 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:39:39.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:38 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi038 to 17173M 2023-03-09T14:39:39.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:38 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:39.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:38 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:39:39.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:38 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:39:39.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:38 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:39.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:38 smithi038 ceph-mon[92422]: from='osd.0 [v2:172.21.15.38:6802/1508072290,v1:172.21.15.38:6803/1508072290]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2023-03-09T14:39:39.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:38 smithi038 ceph-mon[92422]: osdmap e7: 1 total, 0 up, 1 in 2023-03-09T14:39:39.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:38 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-03-09T14:39:39.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:38 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-03-09T14:39:39.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:38 smithi038 ceph-mon[92422]: Detected new or changed devices on smithi038 2023-03-09T14:39:39.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:38 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:39.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:38 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:39.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:38 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:39:39.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:38 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi038 to 17173M 2023-03-09T14:39:39.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:38 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:39.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:38 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:39:39.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:38 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:39:39.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:38 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:40.218 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:39 smithi122 ceph-mon[93869]: purged_snaps scrub starts 2023-03-09T14:39:40.219 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:39 smithi122 ceph-mon[93869]: purged_snaps scrub ok 2023-03-09T14:39:40.219 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:39 smithi122 ceph-mon[93869]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:39:40.219 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:39 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-03-09T14:39:40.219 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:39 smithi122 ceph-mon[93869]: osd.0 [v2:172.21.15.38:6802/1508072290,v1:172.21.15.38:6803/1508072290] boot 2023-03-09T14:39:40.219 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:39 smithi122 ceph-mon[93869]: osdmap e8: 1 total, 1 up, 1 in 2023-03-09T14:39:40.220 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:39 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-03-09T14:39:40.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:39 smithi181 ceph-mon[94189]: purged_snaps scrub starts 2023-03-09T14:39:40.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:39 smithi181 ceph-mon[94189]: purged_snaps scrub ok 2023-03-09T14:39:40.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:39 smithi181 ceph-mon[94189]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:39:40.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:39 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-03-09T14:39:40.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:39 smithi181 ceph-mon[94189]: osd.0 [v2:172.21.15.38:6802/1508072290,v1:172.21.15.38:6803/1508072290] boot 2023-03-09T14:39:40.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:39 smithi181 ceph-mon[94189]: osdmap e8: 1 total, 1 up, 1 in 2023-03-09T14:39:40.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:39 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-03-09T14:39:40.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:39 smithi038 ceph-mon[92422]: purged_snaps scrub starts 2023-03-09T14:39:40.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:39 smithi038 ceph-mon[92422]: purged_snaps scrub ok 2023-03-09T14:39:40.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:39 smithi038 ceph-mon[92422]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-03-09T14:39:40.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:39 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-03-09T14:39:40.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:39 smithi038 ceph-mon[92422]: osd.0 [v2:172.21.15.38:6802/1508072290,v1:172.21.15.38:6803/1508072290] boot 2023-03-09T14:39:40.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:39 smithi038 ceph-mon[92422]: osdmap e8: 1 total, 1 up, 1 in 2023-03-09T14:39:40.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:39 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-03-09T14:39:40.360 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:39:40.358+0000 7f7895b42700 1 -- 172.21.15.122:0/4262011752 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7890105c60 msgr2=0x7f78901080c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:39:40.360 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:39:40.358+0000 7f7895b42700 1 --2- 172.21.15.122:0/4262011752 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7890105c60 0x7f78901080c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f788800b0a0 tx=0x7f788800b3b0 comp rx=0 tx=0).stop 2023-03-09T14:39:40.361 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:39:40.358+0000 7f7895b42700 1 -- 172.21.15.122:0/4262011752 shutdown_connections 2023-03-09T14:39:40.361 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:39:40.358+0000 7f7895b42700 1 --2- 172.21.15.122:0/4262011752 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7890100a20 0x7f7890102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:39:40.361 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:39:40.358+0000 7f7895b42700 1 --2- 172.21.15.122:0/4262011752 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7890105c60 0x7f78901080c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:39:40.361 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:39:40.358+0000 7f7895b42700 1 --2- 172.21.15.122:0/4262011752 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7890103340 0x7f7890105720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:39:40.361 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:39:40.358+0000 7f7895b42700 1 -- 172.21.15.122:0/4262011752 >> 172.21.15.122:0/4262011752 conn(0x7f78900fa480 msgr2=0x7f78900fc8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:39:40.361 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:39:40.358+0000 7f7895b42700 1 -- 172.21.15.122:0/4262011752 shutdown_connections 2023-03-09T14:39:40.361 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:39:40.358+0000 7f7895b42700 1 -- 172.21.15.122:0/4262011752 wait complete. 2023-03-09T14:39:40.363 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:39:40.359+0000 7f7895b42700 1 Processor -- start 2023-03-09T14:39:40.363 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:39:40.359+0000 7f7895b42700 1 -- start start 2023-03-09T14:39:40.363 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:39:40.359+0000 7f7895b42700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7890100a20 0x7f7890195dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:39:40.363 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:39:40.359+0000 7f7895b42700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7890103340 0x7f7890196300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:39:40.364 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:39:40.359+0000 7f7895b42700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7890105c60 0x7f789019d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:39:40.364 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:39:40.359+0000 7f7895b42700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f789019d940 con 0x7f7890100a20 2023-03-09T14:39:40.364 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:39:40.359+0000 7f7895b42700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f789019da80 con 0x7f7890103340 2023-03-09T14:39:40.364 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:39:40.359+0000 7f7895b42700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f789019dc10 con 0x7f7890105c60 2023-03-09T14:39:40.364 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:39:40.359+0000 7f788effd700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7890103340 0x7f7890196300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:39:40.364 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:39:40.359+0000 7f788effd700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7890103340 0x7f7890196300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:49176/0 (socket says 172.21.15.122:49176) 2023-03-09T14:39:40.364 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:39:40.359+0000 7f788effd700 1 -- 172.21.15.122:0/3681196 learned_addr learned my addr 172.21.15.122:0/3681196 (peer_addr_for_me v2:172.21.15.122:0/0) 2023-03-09T14:39:40.364 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:39:40.359+0000 7f788f7fe700 1 --2- 172.21.15.122:0/3681196 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7890100a20 0x7f7890195dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:39:40.365 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:39:40.359+0000 7f788ffff700 1 --2- 172.21.15.122:0/3681196 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7890105c60 0x7f789019d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:39:40.365 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:39:40.359+0000 7f788effd700 1 -- 172.21.15.122:0/3681196 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7890105c60 msgr2=0x7f789019d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:39:40.365 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:39:40.359+0000 7f788effd700 1 --2- 172.21.15.122:0/3681196 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7890105c60 0x7f789019d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:39:40.365 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:39:40.359+0000 7f788effd700 1 -- 172.21.15.122:0/3681196 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7890100a20 msgr2=0x7f7890195dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:39:40.365 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:39:40.359+0000 7f788effd700 1 --2- 172.21.15.122:0/3681196 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7890100a20 0x7f7890195dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:39:40.365 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:39:40.359+0000 7f788effd700 1 -- 172.21.15.122:0/3681196 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7888009d00 con 0x7f7890103340 2023-03-09T14:39:40.365 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:39:40.360+0000 7f788ffff700 1 --2- 172.21.15.122:0/3681196 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7890105c60 0x7f789019d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:39:40.366 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:39:40.360+0000 7f788effd700 1 --2- 172.21.15.122:0/3681196 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7890103340 0x7f7890196300 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f788000d950 tx=0x7f788000dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:39:40.366 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:39:40.360+0000 7f788cff9700 1 -- 172.21.15.122:0/3681196 <== mon.2 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7880009940 con 0x7f7890103340 2023-03-09T14:39:40.366 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:39:40.360+0000 7f788cff9700 1 -- 172.21.15.122:0/3681196 <== mon.2 v2:172.21.15.122:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7880010460 con 0x7f7890103340 2023-03-09T14:39:40.366 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:39:40.360+0000 7f7895b42700 1 -- 172.21.15.122:0/3681196 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f789010b8e0 con 0x7f7890103340 2023-03-09T14:39:40.366 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:39:40.360+0000 7f788cff9700 1 -- 172.21.15.122:0/3681196 <== mon.2 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f788000f620 con 0x7f7890103340 2023-03-09T14:39:40.366 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:39:40.360+0000 7f7895b42700 1 -- 172.21.15.122:0/3681196 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f789010b750 con 0x7f7890103340 2023-03-09T14:39:40.366 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:39:40.361+0000 7f7895b42700 1 -- 172.21.15.122:0/3681196 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78900fbff0 con 0x7f7890103340 2023-03-09T14:39:40.367 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:39:40.362+0000 7f788cff9700 1 -- 172.21.15.122:0/3681196 <== mon.2 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 15) v1 ==== 133158+0+0 (secure 0 0 0) 0x7f7880022070 con 0x7f7890103340 2023-03-09T14:39:40.367 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:39:40.363+0000 7f788cff9700 1 --2- 172.21.15.122:0/3681196 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f787809f3f0 0x7f78780a18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:39:40.367 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:39:40.363+0000 7f788cff9700 1 -- 172.21.15.122:0/3681196 <== mon.2 v2:172.21.15.122:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1767+0+0 (secure 0 0 0) 0x7f78800c8040 con 0x7f7890103340 2023-03-09T14:39:40.368 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:39:40.363+0000 7f788f7fe700 1 --2- 172.21.15.122:0/3681196 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f787809f3f0 0x7f78780a18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:39:40.368 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:39:40.363+0000 7f788f7fe700 1 --2- 172.21.15.122:0/3681196 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f787809f3f0 0x7f78780a18a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7884005950 tx=0x7f788400b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:39:40.369 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:39:40.367+0000 7f788cff9700 1 -- 172.21.15.122:0/3681196 <== mon.2 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f7880093e30 con 0x7f7890103340 2023-03-09T14:39:40.561 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:39:40.559+0000 7f7895b42700 1 -- 172.21.15.122:0/3681196 --> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f78900618a0 con 0x7f787809f3f0 2023-03-09T14:39:41.451 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:41 smithi122 ceph-mon[93869]: osdmap e9: 1 total, 1 up, 1 in 2023-03-09T14:39:41.451 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:41 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-03-09T14:39:41.451 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:41 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-03-09T14:39:41.451 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:41 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:39:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:41 smithi181 ceph-mon[94189]: osdmap e9: 1 total, 1 up, 1 in 2023-03-09T14:39:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:41 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-03-09T14:39:41.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:41 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-03-09T14:39:41.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:41 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:39:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:41 smithi038 ceph-mon[92422]: osdmap e9: 1 total, 1 up, 1 in 2023-03-09T14:39:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:41 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-03-09T14:39:41.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:41 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-03-09T14:39:41.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:41 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:39:42.253 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:42 smithi122 ceph-mon[93869]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:39:42.254 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:42 smithi122 ceph-mon[93869]: pgmap v42: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-03-09T14:39:42.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:42 smithi181 ceph-mon[94189]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:39:42.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:42 smithi181 ceph-mon[94189]: pgmap v42: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-03-09T14:39:42.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:42 smithi038 ceph-mon[92422]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:39:42.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:42 smithi038 ceph-mon[92422]: pgmap v42: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-03-09T14:39:43.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:43 smithi122 ceph-mon[93869]: from='client.? 172.21.15.122:0/140556481' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "71f8a718-b0a9-4f39-bc54-267aa42a2578"}]: dispatch 2023-03-09T14:39:43.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:43 smithi122 ceph-mon[93869]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "71f8a718-b0a9-4f39-bc54-267aa42a2578"}]: dispatch 2023-03-09T14:39:43.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:43 smithi122 ceph-mon[93869]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "71f8a718-b0a9-4f39-bc54-267aa42a2578"}]': finished 2023-03-09T14:39:43.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:43 smithi122 ceph-mon[93869]: osdmap e10: 2 total, 1 up, 2 in 2023-03-09T14:39:43.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:43 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-03-09T14:39:43.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:43 smithi181 ceph-mon[94189]: from='client.? 172.21.15.122:0/140556481' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "71f8a718-b0a9-4f39-bc54-267aa42a2578"}]: dispatch 2023-03-09T14:39:43.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:43 smithi181 ceph-mon[94189]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "71f8a718-b0a9-4f39-bc54-267aa42a2578"}]: dispatch 2023-03-09T14:39:43.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:43 smithi181 ceph-mon[94189]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "71f8a718-b0a9-4f39-bc54-267aa42a2578"}]': finished 2023-03-09T14:39:43.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:43 smithi181 ceph-mon[94189]: osdmap e10: 2 total, 1 up, 2 in 2023-03-09T14:39:43.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:43 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-03-09T14:39:43.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:43 smithi038 ceph-mon[92422]: from='client.? 172.21.15.122:0/140556481' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "71f8a718-b0a9-4f39-bc54-267aa42a2578"}]: dispatch 2023-03-09T14:39:43.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:43 smithi038 ceph-mon[92422]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "71f8a718-b0a9-4f39-bc54-267aa42a2578"}]: dispatch 2023-03-09T14:39:43.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:43 smithi038 ceph-mon[92422]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "71f8a718-b0a9-4f39-bc54-267aa42a2578"}]': finished 2023-03-09T14:39:43.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:43 smithi038 ceph-mon[92422]: osdmap e10: 2 total, 1 up, 2 in 2023-03-09T14:39:43.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:43 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-03-09T14:39:44.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:44 smithi122 ceph-mon[93869]: pgmap v43: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-03-09T14:39:44.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:44 smithi122 ceph-mon[93869]: from='client.? 172.21.15.122:0/20895377' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-03-09T14:39:44.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:44 smithi181 ceph-mon[94189]: pgmap v43: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-03-09T14:39:44.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:44 smithi181 ceph-mon[94189]: from='client.? 172.21.15.122:0/20895377' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-03-09T14:39:44.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:44 smithi038 ceph-mon[92422]: pgmap v43: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-03-09T14:39:44.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:44 smithi038 ceph-mon[92422]: from='client.? 172.21.15.122:0/20895377' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-03-09T14:39:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:46 smithi122 ceph-mon[93869]: pgmap v45: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-03-09T14:39:46.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:46 smithi181 ceph-mon[94189]: pgmap v45: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-03-09T14:39:46.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:46 smithi038 ceph-mon[92422]: pgmap v45: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-03-09T14:39:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:48 smithi122 ceph-mon[93869]: pgmap v46: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-03-09T14:39:48.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:48 smithi181 ceph-mon[94189]: pgmap v46: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-03-09T14:39:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:48 smithi038 ceph-mon[92422]: pgmap v46: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-03-09T14:39:50.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:50 smithi122 ceph-mon[93869]: pgmap v47: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-03-09T14:39:50.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:50 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-03-09T14:39:50.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:50 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:39:50.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:50 smithi181 ceph-mon[94189]: pgmap v47: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-03-09T14:39:50.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:50 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-03-09T14:39:50.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:50 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:39:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:50 smithi038 ceph-mon[92422]: pgmap v47: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-03-09T14:39:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:50 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-03-09T14:39:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:50 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:39:51.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:51 smithi122 ceph-mon[93869]: Deploying daemon osd.1 on smithi122 2023-03-09T14:39:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:51 smithi038 ceph-mon[92422]: Deploying daemon osd.1 on smithi122 2023-03-09T14:39:51.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:51 smithi181 ceph-mon[94189]: Deploying daemon osd.1 on smithi122 2023-03-09T14:39:52.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:52 smithi122 ceph-mon[93869]: pgmap v48: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-03-09T14:39:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:52 smithi038 ceph-mon[92422]: pgmap v48: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-03-09T14:39:52.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:52 smithi181 ceph-mon[94189]: pgmap v48: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-03-09T14:39:54.463 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:54 smithi122 ceph-mon[93869]: pgmap v49: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-03-09T14:39:54.463 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:54 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:54.463 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:54 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:54.463 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:54 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:39:54.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:54 smithi038 ceph-mon[92422]: pgmap v49: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-03-09T14:39:54.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:54 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:54.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:54 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:54.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:54 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:39:54.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:54 smithi181 ceph-mon[94189]: pgmap v49: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-03-09T14:39:54.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:54 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:54.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:54 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:54.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:54 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:39:56.574 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:56 smithi122 ceph-mon[93869]: pgmap v50: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-03-09T14:39:56.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:56 smithi181 ceph-mon[94189]: pgmap v50: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-03-09T14:39:56.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:56 smithi038 ceph-mon[92422]: pgmap v50: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-03-09T14:39:56.963 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:39:56.959+0000 7f788cff9700 1 -- 172.21.15.122:0/3681196 <== mgr.14150 v2:172.21.15.38:6800/3328887328 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f78900618a0 con 0x7f787809f3f0 2023-03-09T14:39:56.964 INFO:teuthology.orchestra.run.smithi122.stdout:Created osd(s) 1 on host 'smithi122' 2023-03-09T14:39:56.964 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:39:56.961+0000 7f7895b42700 1 -- 172.21.15.122:0/3681196 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f787809f3f0 msgr2=0x7f78780a18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:39:56.965 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:39:56.961+0000 7f7895b42700 1 --2- 172.21.15.122:0/3681196 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f787809f3f0 0x7f78780a18a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7884005950 tx=0x7f788400b410 comp rx=0 tx=0).stop 2023-03-09T14:39:56.965 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:39:56.961+0000 7f7895b42700 1 -- 172.21.15.122:0/3681196 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7890103340 msgr2=0x7f7890196300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:39:56.965 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:39:56.961+0000 7f7895b42700 1 --2- 172.21.15.122:0/3681196 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7890103340 0x7f7890196300 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f788000d950 tx=0x7f788000dc60 comp rx=0 tx=0).stop 2023-03-09T14:39:56.965 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:39:56.961+0000 7f7895b42700 1 -- 172.21.15.122:0/3681196 shutdown_connections 2023-03-09T14:39:56.965 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:39:56.961+0000 7f7895b42700 1 --2- 172.21.15.122:0/3681196 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7890100a20 0x7f7890195dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:39:56.965 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:39:56.961+0000 7f7895b42700 1 --2- 172.21.15.122:0/3681196 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f787809f3f0 0x7f78780a18a0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:39:56.966 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:39:56.961+0000 7f7895b42700 1 --2- 172.21.15.122:0/3681196 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7890105c60 0x7f789019d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:39:56.966 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:39:56.961+0000 7f7895b42700 1 --2- 172.21.15.122:0/3681196 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7890103340 0x7f7890196300 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:39:56.966 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:39:56.961+0000 7f7895b42700 1 -- 172.21.15.122:0/3681196 >> 172.21.15.122:0/3681196 conn(0x7f78900fa480 msgr2=0x7f7890101680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:39:56.966 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:39:56.962+0000 7f7895b42700 1 -- 172.21.15.122:0/3681196 shutdown_connections 2023-03-09T14:39:56.966 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:39:56.962+0000 7f7895b42700 1 -- 172.21.15.122:0/3681196 wait complete. 2023-03-09T14:39:57.346 DEBUG:teuthology.orchestra.run.smithi122:osd.1> sudo journalctl -f -n 0 -u ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.1.service 2023-03-09T14:39:57.349 INFO:tasks.cephadm:Deploying osd.2 on smithi181 with /dev/vg_nvme/lv_4... 2023-03-09T14:39:57.350 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-03-09T14:39:57.380 INFO:journalctl@ceph.osd.1.smithi122.stdout:-- Logs begin at Thu 2023-03-09 14:25:09 UTC. -- 2023-03-09T14:39:57.606 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:57 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:57.607 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:57 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:57.607 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:57 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:39:57.607 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:57 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:39:57.607 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:57 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:57.607 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:57 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:57.607 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:57 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:57.607 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:57 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:39:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:57 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:57.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:57 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:57.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:57 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:39:57.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:57 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:39:57.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:57 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:57.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:57 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:57.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:57 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:57.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:57 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:39:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:57 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:57.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:57 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:57.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:57 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:39:57.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:57 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:39:57.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:57 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:57.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:57 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:57.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:57 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:39:57.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:57 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:39:58.003 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.c/config 2023-03-09T14:39:58.227 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:39:58 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 2023-03-09T14:39:58.006+0000 7f8bd57bd640 -1 osd.1 0 log_to_monitors true 2023-03-09T14:39:58.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:58 smithi181 ceph-mon[94189]: pgmap v51: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-03-09T14:39:58.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:58 smithi181 ceph-mon[94189]: from='osd.1 [v2:172.21.15.122:6800/3929268773,v1:172.21.15.122:6801/3929268773]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-03-09T14:39:58.543 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:58 smithi181 ceph-mon[94189]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-03-09T14:39:58.614 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:58 smithi122 ceph-mon[93869]: pgmap v51: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-03-09T14:39:58.614 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:58 smithi122 ceph-mon[93869]: from='osd.1 [v2:172.21.15.122:6800/3929268773,v1:172.21.15.122:6801/3929268773]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-03-09T14:39:58.614 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:58 smithi122 ceph-mon[93869]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-03-09T14:39:58.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:58 smithi038 ceph-mon[92422]: pgmap v51: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-03-09T14:39:58.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:58 smithi038 ceph-mon[92422]: from='osd.1 [v2:172.21.15.122:6800/3929268773,v1:172.21.15.122:6801/3929268773]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-03-09T14:39:58.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:58 smithi038 ceph-mon[92422]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-03-09T14:39:59.540 INFO:teuthology.orchestra.run.smithi181.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-03-09T14:39:59.540 INFO:teuthology.orchestra.run.smithi181.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-03-09T14:39:59.540 INFO:teuthology.orchestra.run.smithi181.stderr: stderr: 10+0 records in 2023-03-09T14:39:59.540 INFO:teuthology.orchestra.run.smithi181.stderr:10+0 records out 2023-03-09T14:39:59.540 INFO:teuthology.orchestra.run.smithi181.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0145256 s, 722 MB/s 2023-03-09T14:39:59.541 INFO:teuthology.orchestra.run.smithi181.stderr: stderr: 2023-03-09T14:39:59.541 INFO:teuthology.orchestra.run.smithi181.stderr:--> Zapping successful for: 2023-03-09T14:39:59.645 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:39:59 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 2023-03-09T14:39:59.335+0000 7f8bc9f43700 -1 osd.1 0 waiting for initial osdmap 2023-03-09T14:39:59.645 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:39:59 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 2023-03-09T14:39:59.340+0000 7f8bc3533700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-09T14:39:59.645 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:59 smithi122 ceph-mon[93869]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-03-09T14:39:59.645 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:59 smithi122 ceph-mon[93869]: osdmap e11: 2 total, 1 up, 2 in 2023-03-09T14:39:59.646 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:59 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-03-09T14:39:59.646 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:59 smithi122 ceph-mon[93869]: from='osd.1 [v2:172.21.15.122:6800/3929268773,v1:172.21.15.122:6801/3929268773]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-03-09T14:39:59.646 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:59 smithi122 ceph-mon[93869]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-03-09T14:39:59.646 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:39:59 smithi122 ceph-mon[93869]: pgmap v53: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-03-09T14:39:59.676 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:59 smithi181 ceph-mon[94189]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-03-09T14:39:59.676 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:59 smithi181 ceph-mon[94189]: osdmap e11: 2 total, 1 up, 2 in 2023-03-09T14:39:59.676 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:59 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-03-09T14:39:59.676 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:59 smithi181 ceph-mon[94189]: from='osd.1 [v2:172.21.15.122:6800/3929268773,v1:172.21.15.122:6801/3929268773]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-03-09T14:39:59.676 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:59 smithi181 ceph-mon[94189]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-03-09T14:39:59.676 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:39:59 smithi181 ceph-mon[94189]: pgmap v53: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-03-09T14:39:59.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:59 smithi038 ceph-mon[92422]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-03-09T14:39:59.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:59 smithi038 ceph-mon[92422]: osdmap e11: 2 total, 1 up, 2 in 2023-03-09T14:39:59.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:59 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-03-09T14:39:59.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:59 smithi038 ceph-mon[92422]: from='osd.1 [v2:172.21.15.122:6800/3929268773,v1:172.21.15.122:6801/3929268773]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-03-09T14:39:59.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:59 smithi038 ceph-mon[92422]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-03-09T14:39:59.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:39:59 smithi038 ceph-mon[92422]: pgmap v53: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2023-03-09T14:39:59.954 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph orch daemon add osd smithi181:vg_nvme/lv_4 2023-03-09T14:40:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:00 smithi181 ceph-mon[94189]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2023-03-09T14:40:00.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:00 smithi181 ceph-mon[94189]: osdmap e12: 2 total, 1 up, 2 in 2023-03-09T14:40:00.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:00 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-03-09T14:40:00.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:00 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-03-09T14:40:00.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:00 smithi181 ceph-mon[94189]: Detected new or changed devices on smithi122 2023-03-09T14:40:00.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:00 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:00.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:00 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:00.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:00 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:40:00.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:00 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 17168M 2023-03-09T14:40:00.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:00 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:00.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:00 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:40:00.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:00 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:40:00.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:00 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:00.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:00 smithi181 ceph-mon[94189]: overall HEALTH_OK 2023-03-09T14:40:00.597 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.c/config 2023-03-09T14:40:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:00 smithi122 ceph-mon[93869]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2023-03-09T14:40:00.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:00 smithi122 ceph-mon[93869]: osdmap e12: 2 total, 1 up, 2 in 2023-03-09T14:40:00.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:00 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-03-09T14:40:00.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:00 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-03-09T14:40:00.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:00 smithi122 ceph-mon[93869]: Detected new or changed devices on smithi122 2023-03-09T14:40:00.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:00 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:00.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:00 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:00.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:00 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:40:00.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:00 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 17168M 2023-03-09T14:40:00.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:00 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:00.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:00 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:40:00.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:00 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:40:00.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:00 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:00.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:00 smithi122 ceph-mon[93869]: overall HEALTH_OK 2023-03-09T14:40:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:00 smithi038 ceph-mon[92422]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2023-03-09T14:40:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:00 smithi038 ceph-mon[92422]: osdmap e12: 2 total, 1 up, 2 in 2023-03-09T14:40:00.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:00 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-03-09T14:40:00.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:00 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-03-09T14:40:00.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:00 smithi038 ceph-mon[92422]: Detected new or changed devices on smithi122 2023-03-09T14:40:00.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:00 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:00.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:00 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:00.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:00 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:40:00.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:00 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 17168M 2023-03-09T14:40:00.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:00 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:00.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:00 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:40:00.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:00 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:40:00.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:00 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:00.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:00 smithi038 ceph-mon[92422]: overall HEALTH_OK 2023-03-09T14:40:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:01 smithi181 ceph-mon[94189]: purged_snaps scrub starts 2023-03-09T14:40:01.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:01 smithi181 ceph-mon[94189]: purged_snaps scrub ok 2023-03-09T14:40:01.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:01 smithi181 ceph-mon[94189]: osd.1 [v2:172.21.15.122:6800/3929268773,v1:172.21.15.122:6801/3929268773] boot 2023-03-09T14:40:01.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:01 smithi181 ceph-mon[94189]: osdmap e13: 2 total, 2 up, 2 in 2023-03-09T14:40:01.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:01 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-03-09T14:40:01.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:01 smithi181 ceph-mon[94189]: pgmap v56: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-03-09T14:40:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:01 smithi122 ceph-mon[93869]: purged_snaps scrub starts 2023-03-09T14:40:01.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:01 smithi122 ceph-mon[93869]: purged_snaps scrub ok 2023-03-09T14:40:01.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:01 smithi122 ceph-mon[93869]: osd.1 [v2:172.21.15.122:6800/3929268773,v1:172.21.15.122:6801/3929268773] boot 2023-03-09T14:40:01.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:01 smithi122 ceph-mon[93869]: osdmap e13: 2 total, 2 up, 2 in 2023-03-09T14:40:01.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:01 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-03-09T14:40:01.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:01 smithi122 ceph-mon[93869]: pgmap v56: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-03-09T14:40:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:01 smithi038 ceph-mon[92422]: purged_snaps scrub starts 2023-03-09T14:40:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:01 smithi038 ceph-mon[92422]: purged_snaps scrub ok 2023-03-09T14:40:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:01 smithi038 ceph-mon[92422]: osd.1 [v2:172.21.15.122:6800/3929268773,v1:172.21.15.122:6801/3929268773] boot 2023-03-09T14:40:01.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:01 smithi038 ceph-mon[92422]: osdmap e13: 2 total, 2 up, 2 in 2023-03-09T14:40:01.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:01 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-03-09T14:40:01.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:01 smithi038 ceph-mon[92422]: pgmap v56: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-03-09T14:40:01.772 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:40:01.769+0000 7fbc6b5f9700 1 -- 172.21.15.181:0/2456108792 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc641009a0 msgr2=0x7fbc641072d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:40:01.772 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:40:01.769+0000 7fbc6b5f9700 1 --2- 172.21.15.181:0/2456108792 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc641009a0 0x7fbc641072d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fbc5c00b0a0 tx=0x7fbc5c00b3b0 comp rx=0 tx=0).stop 2023-03-09T14:40:01.773 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:40:01.769+0000 7fbc6b5f9700 1 -- 172.21.15.181:0/2456108792 shutdown_connections 2023-03-09T14:40:01.773 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:40:01.769+0000 7fbc6b5f9700 1 --2- 172.21.15.181:0/2456108792 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc641009a0 0x7fbc641072d0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:40:01.773 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:40:01.769+0000 7fbc6b5f9700 1 --2- 172.21.15.181:0/2456108792 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbc640feea0 0x7fbc640ff2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:40:01.773 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:40:01.769+0000 7fbc6b5f9700 1 --2- 172.21.15.181:0/2456108792 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbc640ffff0 0x7fbc64100460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:40:01.773 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:40:01.769+0000 7fbc6b5f9700 1 -- 172.21.15.181:0/2456108792 >> 172.21.15.181:0/2456108792 conn(0x7fbc640fa480 msgr2=0x7fbc640fc8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:40:01.773 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:40:01.769+0000 7fbc6b5f9700 1 -- 172.21.15.181:0/2456108792 shutdown_connections 2023-03-09T14:40:01.773 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:40:01.770+0000 7fbc6b5f9700 1 -- 172.21.15.181:0/2456108792 wait complete. 2023-03-09T14:40:01.775 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:40:01.770+0000 7fbc6b5f9700 1 Processor -- start 2023-03-09T14:40:01.775 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:40:01.770+0000 7fbc6b5f9700 1 -- start start 2023-03-09T14:40:01.775 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:40:01.770+0000 7fbc6b5f9700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbc640feea0 0x7fbc64194d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:40:01.775 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:40:01.770+0000 7fbc6b5f9700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbc640ffff0 0x7fbc64195260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:40:01.775 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:40:01.770+0000 7fbc6b5f9700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc641009a0 0x7fbc6419c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:40:01.776 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:40:01.770+0000 7fbc6b5f9700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc6419c860 con 0x7fbc640ffff0 2023-03-09T14:40:01.776 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:40:01.770+0000 7fbc6b5f9700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc6419c9d0 con 0x7fbc640feea0 2023-03-09T14:40:01.776 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:40:01.770+0000 7fbc6b5f9700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc6419cb90 con 0x7fbc641009a0 2023-03-09T14:40:01.776 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:40:01.770+0000 7fbc69b96700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc641009a0 0x7fbc6419c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:40:01.776 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:40:01.770+0000 7fbc69b96700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc641009a0 0x7fbc6419c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:50508/0 (socket says 172.21.15.181:50508) 2023-03-09T14:40:01.776 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:40:01.770+0000 7fbc69b96700 1 -- 172.21.15.181:0/2790179207 learned_addr learned my addr 172.21.15.181:0/2790179207 (peer_addr_for_me v2:172.21.15.181:0/0) 2023-03-09T14:40:01.776 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:40:01.771+0000 7fbc69395700 1 --2- 172.21.15.181:0/2790179207 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbc640feea0 0x7fbc64194d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:40:01.777 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:40:01.771+0000 7fbc68b94700 1 --2- 172.21.15.181:0/2790179207 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbc640ffff0 0x7fbc64195260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:40:01.777 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:40:01.771+0000 7fbc69b96700 1 -- 172.21.15.181:0/2790179207 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbc640feea0 msgr2=0x7fbc64194d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:40:01.777 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:40:01.771+0000 7fbc69b96700 1 --2- 172.21.15.181:0/2790179207 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbc640feea0 0x7fbc64194d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:40:01.777 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:40:01.771+0000 7fbc69b96700 1 -- 172.21.15.181:0/2790179207 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbc640ffff0 msgr2=0x7fbc64195260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:40:01.777 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:40:01.771+0000 7fbc69b96700 1 --2- 172.21.15.181:0/2790179207 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbc640ffff0 0x7fbc64195260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:40:01.777 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:40:01.771+0000 7fbc69b96700 1 -- 172.21.15.181:0/2790179207 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc5c009d00 con 0x7fbc641009a0 2023-03-09T14:40:01.777 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:40:01.771+0000 7fbc69b96700 1 --2- 172.21.15.181:0/2790179207 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc641009a0 0x7fbc6419c290 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fbc5c009fd0 tx=0x7fbc5c004b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:40:01.778 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:40:01.771+0000 7fbc5a7fc700 1 -- 172.21.15.181:0/2790179207 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc5c00e040 con 0x7fbc641009a0 2023-03-09T14:40:01.778 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:40:01.771+0000 7fbc5a7fc700 1 -- 172.21.15.181:0/2790179207 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbc5c004e00 con 0x7fbc641009a0 2023-03-09T14:40:01.778 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:40:01.771+0000 7fbc6b5f9700 1 -- 172.21.15.181:0/2790179207 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc6410a850 con 0x7fbc641009a0 2023-03-09T14:40:01.778 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:40:01.771+0000 7fbc5a7fc700 1 -- 172.21.15.181:0/2790179207 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc5c004280 con 0x7fbc641009a0 2023-03-09T14:40:01.778 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:40:01.771+0000 7fbc6b5f9700 1 -- 172.21.15.181:0/2790179207 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc6410a6c0 con 0x7fbc641009a0 2023-03-09T14:40:01.778 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:40:01.772+0000 7fbc6b5f9700 1 -- 172.21.15.181:0/2790179207 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc64066b10 con 0x7fbc641009a0 2023-03-09T14:40:01.779 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:40:01.773+0000 7fbc5a7fc700 1 -- 172.21.15.181:0/2790179207 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 15) v1 ==== 133158+0+0 (secure 0 0 0) 0x7fbc5c00bd70 con 0x7fbc641009a0 2023-03-09T14:40:01.779 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:40:01.773+0000 7fbc5a7fc700 1 --2- 172.21.15.181:0/2790179207 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fbc5009f370 0x7fbc500a1820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:40:01.779 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:40:01.774+0000 7fbc5a7fc700 1 -- 172.21.15.181:0/2790179207 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2309+0+0 (secure 0 0 0) 0x7fbc5c0c9cd0 con 0x7fbc641009a0 2023-03-09T14:40:01.779 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:40:01.774+0000 7fbc69395700 1 --2- 172.21.15.181:0/2790179207 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fbc5009f370 0x7fbc500a1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:40:01.779 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:40:01.774+0000 7fbc69395700 1 --2- 172.21.15.181:0/2790179207 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fbc5009f370 0x7fbc500a1820 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbc54005950 tx=0x7fbc54005d10 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:40:01.781 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:40:01.778+0000 7fbc5a7fc700 1 -- 172.21.15.181:0/2790179207 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7fbc5c095930 con 0x7fbc641009a0 2023-03-09T14:40:01.973 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:40:01.970+0000 7fbc6b5f9700 1 -- 172.21.15.181:0/2790179207 --> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fbc64060bc0 con 0x7fbc5009f370 2023-03-09T14:40:02.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:02 smithi122 ceph-mon[93869]: osdmap e14: 2 total, 2 up, 2 in 2023-03-09T14:40:02.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:02 smithi122 ceph-mon[93869]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:40:02.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:02 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-03-09T14:40:02.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:02 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-03-09T14:40:02.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:02 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:40:02.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:02 smithi181 ceph-mon[94189]: osdmap e14: 2 total, 2 up, 2 in 2023-03-09T14:40:02.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:02 smithi181 ceph-mon[94189]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:40:02.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:02 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-03-09T14:40:02.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:02 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-03-09T14:40:02.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:02 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:40:02.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:02 smithi038 ceph-mon[92422]: osdmap e14: 2 total, 2 up, 2 in 2023-03-09T14:40:02.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:02 smithi038 ceph-mon[92422]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:40:02.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:02 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-03-09T14:40:02.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:02 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-03-09T14:40:02.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:02 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:40:03.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:03 smithi181 ceph-mon[94189]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-03-09T14:40:03.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:03 smithi122 ceph-mon[93869]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-03-09T14:40:03.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:03 smithi038 ceph-mon[92422]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-03-09T14:40:04.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:04 smithi122 ceph-mon[93869]: from='client.? 172.21.15.181:0/672025422' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "961a2cbe-140e-437f-b131-1fcac573b19c"}]: dispatch 2023-03-09T14:40:04.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:04 smithi122 ceph-mon[93869]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "961a2cbe-140e-437f-b131-1fcac573b19c"}]: dispatch 2023-03-09T14:40:04.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:04 smithi122 ceph-mon[93869]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "961a2cbe-140e-437f-b131-1fcac573b19c"}]': finished 2023-03-09T14:40:04.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:04 smithi122 ceph-mon[93869]: osdmap e15: 3 total, 2 up, 3 in 2023-03-09T14:40:04.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:04 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-03-09T14:40:04.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:04 smithi181 ceph-mon[94189]: from='client.? 172.21.15.181:0/672025422' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "961a2cbe-140e-437f-b131-1fcac573b19c"}]: dispatch 2023-03-09T14:40:04.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:04 smithi181 ceph-mon[94189]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "961a2cbe-140e-437f-b131-1fcac573b19c"}]: dispatch 2023-03-09T14:40:04.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:04 smithi181 ceph-mon[94189]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "961a2cbe-140e-437f-b131-1fcac573b19c"}]': finished 2023-03-09T14:40:04.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:04 smithi181 ceph-mon[94189]: osdmap e15: 3 total, 2 up, 3 in 2023-03-09T14:40:04.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:04 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-03-09T14:40:04.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:04 smithi038 ceph-mon[92422]: from='client.? 172.21.15.181:0/672025422' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "961a2cbe-140e-437f-b131-1fcac573b19c"}]: dispatch 2023-03-09T14:40:04.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:04 smithi038 ceph-mon[92422]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "961a2cbe-140e-437f-b131-1fcac573b19c"}]: dispatch 2023-03-09T14:40:04.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:04 smithi038 ceph-mon[92422]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "961a2cbe-140e-437f-b131-1fcac573b19c"}]': finished 2023-03-09T14:40:04.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:04 smithi038 ceph-mon[92422]: osdmap e15: 3 total, 2 up, 3 in 2023-03-09T14:40:04.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:04 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-03-09T14:40:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:05 smithi122 ceph-mon[93869]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-03-09T14:40:05.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:05 smithi122 ceph-mon[93869]: from='client.? 172.21.15.181:0/849450238' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-03-09T14:40:05.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:05 smithi181 ceph-mon[94189]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-03-09T14:40:05.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:05 smithi181 ceph-mon[94189]: from='client.? 172.21.15.181:0/849450238' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-03-09T14:40:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:05 smithi038 ceph-mon[92422]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2023-03-09T14:40:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:05 smithi038 ceph-mon[92422]: from='client.? 172.21.15.181:0/849450238' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-03-09T14:40:08.443 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:08 smithi181 ceph-mon[94189]: pgmap v61: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-03-09T14:40:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:08 smithi122 ceph-mon[93869]: pgmap v61: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-03-09T14:40:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:08 smithi038 ceph-mon[92422]: pgmap v61: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-03-09T14:40:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:10 smithi122 ceph-mon[93869]: pgmap v62: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-03-09T14:40:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:10 smithi181 ceph-mon[94189]: pgmap v62: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-03-09T14:40:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:10 smithi038 ceph-mon[92422]: pgmap v62: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-03-09T14:40:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:11 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-03-09T14:40:11.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:11 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:40:11.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:11 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-03-09T14:40:11.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:11 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:40:11.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:11 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-03-09T14:40:11.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:11 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:40:12.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:12 smithi181 ceph-mon[94189]: pgmap v63: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-03-09T14:40:12.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:12 smithi181 ceph-mon[94189]: Deploying daemon osd.2 on smithi181 2023-03-09T14:40:12.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:12 smithi122 ceph-mon[93869]: pgmap v63: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-03-09T14:40:12.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:12 smithi122 ceph-mon[93869]: Deploying daemon osd.2 on smithi181 2023-03-09T14:40:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:12 smithi038 ceph-mon[92422]: pgmap v63: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-03-09T14:40:12.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:12 smithi038 ceph-mon[92422]: Deploying daemon osd.2 on smithi181 2023-03-09T14:40:14.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:14 smithi181 ceph-mon[94189]: pgmap v64: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-03-09T14:40:14.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:14 smithi122 ceph-mon[93869]: pgmap v64: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-03-09T14:40:14.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:14 smithi038 ceph-mon[92422]: pgmap v64: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-03-09T14:40:16.355 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:16 smithi181 ceph-mon[94189]: pgmap v65: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-03-09T14:40:16.355 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:16 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:16.355 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:16 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:16.355 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:16 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:40:16.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:16 smithi122 ceph-mon[93869]: pgmap v65: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-03-09T14:40:16.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:16 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:16.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:16 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:16.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:16 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:40:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:16 smithi038 ceph-mon[92422]: pgmap v65: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-03-09T14:40:16.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:16 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:16.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:16 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:16.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:16 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:40:18.143 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:40:18.140+0000 7fbc5a7fc700 1 -- 172.21.15.181:0/2790179207 <== mgr.14150 v2:172.21.15.38:6800/3328887328 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fbc64060bc0 con 0x7fbc5009f370 2023-03-09T14:40:18.143 INFO:teuthology.orchestra.run.smithi181.stdout:Created osd(s) 2 on host 'smithi181' 2023-03-09T14:40:18.145 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:40:18.142+0000 7fbc6b5f9700 1 -- 172.21.15.181:0/2790179207 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fbc5009f370 msgr2=0x7fbc500a1820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:40:18.145 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:40:18.142+0000 7fbc6b5f9700 1 --2- 172.21.15.181:0/2790179207 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fbc5009f370 0x7fbc500a1820 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fbc54005950 tx=0x7fbc54005d10 comp rx=0 tx=0).stop 2023-03-09T14:40:18.146 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:40:18.142+0000 7fbc6b5f9700 1 -- 172.21.15.181:0/2790179207 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc641009a0 msgr2=0x7fbc6419c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:40:18.146 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:40:18.142+0000 7fbc6b5f9700 1 --2- 172.21.15.181:0/2790179207 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc641009a0 0x7fbc6419c290 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fbc5c009fd0 tx=0x7fbc5c004b20 comp rx=0 tx=0).stop 2023-03-09T14:40:18.146 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:40:18.142+0000 7fbc6b5f9700 1 -- 172.21.15.181:0/2790179207 shutdown_connections 2023-03-09T14:40:18.146 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:40:18.142+0000 7fbc6b5f9700 1 --2- 172.21.15.181:0/2790179207 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fbc5009f370 0x7fbc500a1820 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:40:18.146 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:40:18.142+0000 7fbc6b5f9700 1 --2- 172.21.15.181:0/2790179207 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbc641009a0 0x7fbc6419c290 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:40:18.147 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:40:18.142+0000 7fbc6b5f9700 1 --2- 172.21.15.181:0/2790179207 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbc640feea0 0x7fbc64194d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:40:18.147 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:40:18.142+0000 7fbc6b5f9700 1 --2- 172.21.15.181:0/2790179207 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbc640ffff0 0x7fbc64195260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:40:18.147 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:40:18.142+0000 7fbc6b5f9700 1 -- 172.21.15.181:0/2790179207 >> 172.21.15.181:0/2790179207 conn(0x7fbc640fa480 msgr2=0x7fbc640fc6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:40:18.147 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:40:18.142+0000 7fbc6b5f9700 1 -- 172.21.15.181:0/2790179207 shutdown_connections 2023-03-09T14:40:18.147 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:40:18.142+0000 7fbc6b5f9700 1 -- 172.21.15.181:0/2790179207 wait complete. 2023-03-09T14:40:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:18 smithi122 ceph-mon[93869]: pgmap v66: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-03-09T14:40:18.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:18 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:18.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:18 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:18.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:18 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:40:18.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:18 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:40:18.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:18 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:18 smithi181 ceph-mon[94189]: pgmap v66: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-03-09T14:40:18.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:18 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:18.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:18 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:18.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:18 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:40:18.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:18 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:40:18.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:18 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:18 smithi038 ceph-mon[92422]: pgmap v66: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-03-09T14:40:18.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:18 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:18.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:18 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:18.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:18 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:40:18.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:18 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:40:18.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:18 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:18.579 DEBUG:teuthology.orchestra.run.smithi181:osd.2> sudo journalctl -f -n 0 -u ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.2.service 2023-03-09T14:40:18.581 INFO:tasks.cephadm:Deploying osd.3 on smithi038 with /dev/vg_nvme/lv_3... 2023-03-09T14:40:18.582 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-03-09T14:40:18.615 INFO:journalctl@ceph.osd.2.smithi181.stdout:-- Logs begin at Thu 2023-03-09 14:25:01 UTC. -- 2023-03-09T14:40:19.252 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.a/config 2023-03-09T14:40:19.381 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:19 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:19.382 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:19 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:19.382 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:19 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:40:19.382 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:40:19 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 2023-03-09T14:40:19.339+0000 7fdac5087640 -1 osd.2 0 log_to_monitors true 2023-03-09T14:40:19.455 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:19 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:19.455 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:19 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:19.456 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:19 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:40:19.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:19 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:19.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:19 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:19.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:19 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:40:20.309 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:20 smithi038 ceph-mon[92422]: pgmap v67: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-03-09T14:40:20.310 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:20 smithi038 ceph-mon[92422]: from='osd.2 [v2:172.21.15.181:6800/1310630803,v1:172.21.15.181:6801/1310630803]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-03-09T14:40:20.310 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:20 smithi038 ceph-mon[92422]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-03-09T14:40:20.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:20 smithi122 ceph-mon[93869]: pgmap v67: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-03-09T14:40:20.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:20 smithi122 ceph-mon[93869]: from='osd.2 [v2:172.21.15.181:6800/1310630803,v1:172.21.15.181:6801/1310630803]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-03-09T14:40:20.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:20 smithi122 ceph-mon[93869]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-03-09T14:40:20.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:20 smithi181 ceph-mon[94189]: pgmap v67: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-03-09T14:40:20.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:20 smithi181 ceph-mon[94189]: from='osd.2 [v2:172.21.15.181:6800/1310630803,v1:172.21.15.181:6801/1310630803]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-03-09T14:40:20.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:20 smithi181 ceph-mon[94189]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-03-09T14:40:20.792 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-03-09T14:40:20.792 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-03-09T14:40:20.792 INFO:teuthology.orchestra.run.smithi038.stderr: stderr: 10+0 records in 2023-03-09T14:40:20.793 INFO:teuthology.orchestra.run.smithi038.stderr:10+0 records out 2023-03-09T14:40:20.793 INFO:teuthology.orchestra.run.smithi038.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0136683 s, 767 MB/s 2023-03-09T14:40:20.793 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping successful for: 2023-03-09T14:40:21.200 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph orch daemon add osd smithi038:vg_nvme/lv_3 2023-03-09T14:40:21.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:21 smithi122 ceph-mon[93869]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-03-09T14:40:21.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:21 smithi122 ceph-mon[93869]: osdmap e16: 3 total, 2 up, 3 in 2023-03-09T14:40:21.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:21 smithi122 ceph-mon[93869]: from='osd.2 [v2:172.21.15.181:6800/1310630803,v1:172.21.15.181:6801/1310630803]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2023-03-09T14:40:21.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:21 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-03-09T14:40:21.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:21 smithi122 ceph-mon[93869]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2023-03-09T14:40:21.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:21 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:21.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:21 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:21.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:21 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:40:21.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:21 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:21.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:21 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:40:21.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:21 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:40:21.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:21 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:21.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:21 smithi122 ceph-mon[93869]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2023-03-09T14:40:21.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:21 smithi122 ceph-mon[93869]: osdmap e17: 3 total, 2 up, 3 in 2023-03-09T14:40:21.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:21 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-03-09T14:40:21.481 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:40:21 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 2023-03-09T14:40:21.165+0000 7fdab900c700 -1 osd.2 0 waiting for initial osdmap 2023-03-09T14:40:21.481 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:40:21 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 2023-03-09T14:40:21.174+0000 7fdab15fa700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-09T14:40:21.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:21 smithi181 ceph-mon[94189]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-03-09T14:40:21.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:21 smithi181 ceph-mon[94189]: osdmap e16: 3 total, 2 up, 3 in 2023-03-09T14:40:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:21 smithi181 ceph-mon[94189]: from='osd.2 [v2:172.21.15.181:6800/1310630803,v1:172.21.15.181:6801/1310630803]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2023-03-09T14:40:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:21 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-03-09T14:40:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:21 smithi181 ceph-mon[94189]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2023-03-09T14:40:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:21 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:21 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:21 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:40:21.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:21 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:21.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:21 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:40:21.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:21 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:40:21.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:21 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:21.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:21 smithi181 ceph-mon[94189]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2023-03-09T14:40:21.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:21 smithi181 ceph-mon[94189]: osdmap e17: 3 total, 2 up, 3 in 2023-03-09T14:40:21.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:21 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-03-09T14:40:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:21 smithi038 ceph-mon[92422]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-03-09T14:40:21.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:21 smithi038 ceph-mon[92422]: osdmap e16: 3 total, 2 up, 3 in 2023-03-09T14:40:21.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:21 smithi038 ceph-mon[92422]: from='osd.2 [v2:172.21.15.181:6800/1310630803,v1:172.21.15.181:6801/1310630803]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2023-03-09T14:40:21.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:21 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-03-09T14:40:21.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:21 smithi038 ceph-mon[92422]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2023-03-09T14:40:21.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:21 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:21.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:21 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:21.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:21 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:40:21.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:21 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:21.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:21 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:40:21.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:21 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:40:21.503 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:21 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:21.503 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:21 smithi038 ceph-mon[92422]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2023-03-09T14:40:21.503 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:21 smithi038 ceph-mon[92422]: osdmap e17: 3 total, 2 up, 3 in 2023-03-09T14:40:21.503 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:21 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-03-09T14:40:21.853 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.a/config 2023-03-09T14:40:22.406 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:22 smithi038 ceph-mon[92422]: pgmap v69: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-03-09T14:40:22.407 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:22 smithi038 ceph-mon[92422]: Detected new or changed devices on smithi181 2023-03-09T14:40:22.407 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:22 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 17168M 2023-03-09T14:40:22.407 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:22 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-03-09T14:40:22.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:22 smithi122 ceph-mon[93869]: pgmap v69: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-03-09T14:40:22.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:22 smithi122 ceph-mon[93869]: Detected new or changed devices on smithi181 2023-03-09T14:40:22.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:22 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 17168M 2023-03-09T14:40:22.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:22 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-03-09T14:40:22.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:22 smithi181 ceph-mon[94189]: pgmap v69: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-03-09T14:40:22.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:22 smithi181 ceph-mon[94189]: Detected new or changed devices on smithi181 2023-03-09T14:40:22.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:22 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 17168M 2023-03-09T14:40:22.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:22 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-03-09T14:40:23.111 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:40:23.109+0000 7f5b5214b700 1 -- 172.21.15.38:0/2572184048 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5b4c103340 msgr2=0x7f5b4c105720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:40:23.112 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:40:23.109+0000 7f5b5214b700 1 --2- 172.21.15.38:0/2572184048 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5b4c103340 0x7f5b4c105720 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f5b40009990 tx=0x7f5b40009ca0 comp rx=0 tx=0).stop 2023-03-09T14:40:23.112 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:40:23.109+0000 7f5b5214b700 1 -- 172.21.15.38:0/2572184048 shutdown_connections 2023-03-09T14:40:23.112 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:40:23.109+0000 7f5b5214b700 1 --2- 172.21.15.38:0/2572184048 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5b4c100a20 0x7f5b4c102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:40:23.112 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:40:23.109+0000 7f5b5214b700 1 --2- 172.21.15.38:0/2572184048 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5b4c105c60 0x7f5b4c1080c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:40:23.112 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:40:23.109+0000 7f5b5214b700 1 --2- 172.21.15.38:0/2572184048 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5b4c103340 0x7f5b4c105720 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:40:23.112 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:40:23.109+0000 7f5b5214b700 1 -- 172.21.15.38:0/2572184048 >> 172.21.15.38:0/2572184048 conn(0x7f5b4c0fa480 msgr2=0x7f5b4c0fc8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:40:23.113 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:40:23.109+0000 7f5b5214b700 1 -- 172.21.15.38:0/2572184048 shutdown_connections 2023-03-09T14:40:23.113 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:40:23.109+0000 7f5b5214b700 1 -- 172.21.15.38:0/2572184048 wait complete. 2023-03-09T14:40:23.113 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:40:23.109+0000 7f5b5214b700 1 Processor -- start 2023-03-09T14:40:23.113 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:40:23.110+0000 7f5b5214b700 1 -- start start 2023-03-09T14:40:23.114 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:40:23.110+0000 7f5b5214b700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5b4c100a20 0x7f5b4c195d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:40:23.114 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:40:23.110+0000 7f5b5214b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5b4c103340 0x7f5b4c196270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:40:23.114 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:40:23.110+0000 7f5b5214b700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5b4c105c60 0x7f5b4c19d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:40:23.115 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:40:23.110+0000 7f5b5214b700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b4c19d870 con 0x7f5b4c103340 2023-03-09T14:40:23.115 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:40:23.110+0000 7f5b5214b700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b4c19d9e0 con 0x7f5b4c100a20 2023-03-09T14:40:23.115 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:40:23.110+0000 7f5b5214b700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b4c19dba0 con 0x7f5b4c105c60 2023-03-09T14:40:23.115 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:40:23.110+0000 7f5b4affd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5b4c103340 0x7f5b4c196270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:40:23.115 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:40:23.110+0000 7f5b4affd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5b4c103340 0x7f5b4c196270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35848/0 (socket says 172.21.15.38:35848) 2023-03-09T14:40:23.115 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:40:23.110+0000 7f5b4affd700 1 -- 172.21.15.38:0/2291684705 learned_addr learned my addr 172.21.15.38:0/2291684705 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:40:23.115 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:40:23.110+0000 7f5b4b7fe700 1 --2- 172.21.15.38:0/2291684705 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5b4c100a20 0x7f5b4c195d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:40:23.115 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:40:23.110+0000 7f5b4bfff700 1 --2- 172.21.15.38:0/2291684705 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5b4c105c60 0x7f5b4c19d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:40:23.116 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:40:23.110+0000 7f5b4affd700 1 -- 172.21.15.38:0/2291684705 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5b4c105c60 msgr2=0x7f5b4c19d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:40:23.116 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:40:23.110+0000 7f5b4affd700 1 --2- 172.21.15.38:0/2291684705 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5b4c105c60 0x7f5b4c19d2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:40:23.116 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:40:23.110+0000 7f5b4affd700 1 -- 172.21.15.38:0/2291684705 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5b4c100a20 msgr2=0x7f5b4c195d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:40:23.116 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:40:23.110+0000 7f5b4affd700 1 --2- 172.21.15.38:0/2291684705 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5b4c100a20 0x7f5b4c195d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:40:23.116 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:40:23.110+0000 7f5b4affd700 1 -- 172.21.15.38:0/2291684705 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b400095f0 con 0x7f5b4c103340 2023-03-09T14:40:23.116 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:40:23.111+0000 7f5b4affd700 1 --2- 172.21.15.38:0/2291684705 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5b4c103340 0x7f5b4c196270 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5b400059d0 tx=0x7f5b400047a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:40:23.199 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:40:23.196+0000 7f5b48ff9700 1 -- 172.21.15.38:0/2291684705 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b4001d070 con 0x7f5b4c103340 2023-03-09T14:40:23.199 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:40:23.196+0000 7f5b5214b700 1 -- 172.21.15.38:0/2291684705 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b4c10b8f0 con 0x7f5b4c103340 2023-03-09T14:40:23.199 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:40:23.196+0000 7f5b48ff9700 1 -- 172.21.15.38:0/2291684705 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5b4000fb90 con 0x7f5b4c103340 2023-03-09T14:40:23.199 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:40:23.196+0000 7f5b48ff9700 1 -- 172.21.15.38:0/2291684705 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b40017650 con 0x7f5b4c103340 2023-03-09T14:40:23.200 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:40:23.196+0000 7f5b5214b700 1 -- 172.21.15.38:0/2291684705 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b4c10b760 con 0x7f5b4c103340 2023-03-09T14:40:23.200 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:40:23.197+0000 7f5b5214b700 1 -- 172.21.15.38:0/2291684705 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b4c0fbff0 con 0x7f5b4c103340 2023-03-09T14:40:23.201 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:40:23.199+0000 7f5b48ff9700 1 -- 172.21.15.38:0/2291684705 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 15) v1 ==== 133158+0+0 (secure 0 0 0) 0x7f5b40029030 con 0x7f5b4c103340 2023-03-09T14:40:23.201 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:40:23.200+0000 7f5b48ff9700 1 --2- 172.21.15.38:0/2291684705 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f5b3409f370 0x7f5b340a1820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:40:23.202 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:40:23.200+0000 7f5b4b7fe700 1 --2- 172.21.15.38:0/2291684705 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f5b3409f370 0x7f5b340a1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:40:23.202 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:40:23.200+0000 7f5b4b7fe700 1 --2- 172.21.15.38:0/2291684705 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f5b3409f370 0x7f5b340a1820 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5b4c197200 tx=0x7f5b3c00b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:40:23.207 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:40:23.205+0000 7f5b48ff9700 1 -- 172.21.15.38:0/2291684705 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 2851+0+0 (secure 0 0 0) 0x7f5b400998c0 con 0x7f5b4c103340 2023-03-09T14:40:23.212 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:40:23.210+0000 7f5b48ff9700 1 -- 172.21.15.38:0/2291684705 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f5b40095c00 con 0x7f5b4c103340 2023-03-09T14:40:23.423 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:40:23.421+0000 7f5b5214b700 1 -- 172.21.15.38:0/2291684705 --> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f5b4c0618d0 con 0x7f5b3409f370 2023-03-09T14:40:23.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:23 smithi122 ceph-mon[93869]: purged_snaps scrub starts 2023-03-09T14:40:23.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:23 smithi122 ceph-mon[93869]: purged_snaps scrub ok 2023-03-09T14:40:23.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:23 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-03-09T14:40:23.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:23 smithi122 ceph-mon[93869]: osd.2 [v2:172.21.15.181:6800/1310630803,v1:172.21.15.181:6801/1310630803] boot 2023-03-09T14:40:23.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:23 smithi122 ceph-mon[93869]: osdmap e18: 3 total, 3 up, 3 in 2023-03-09T14:40:23.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:23 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-03-09T14:40:23.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:23 smithi181 ceph-mon[94189]: purged_snaps scrub starts 2023-03-09T14:40:23.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:23 smithi181 ceph-mon[94189]: purged_snaps scrub ok 2023-03-09T14:40:23.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:23 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-03-09T14:40:23.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:23 smithi181 ceph-mon[94189]: osd.2 [v2:172.21.15.181:6800/1310630803,v1:172.21.15.181:6801/1310630803] boot 2023-03-09T14:40:23.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:23 smithi181 ceph-mon[94189]: osdmap e18: 3 total, 3 up, 3 in 2023-03-09T14:40:23.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:23 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-03-09T14:40:23.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:23 smithi038 ceph-mon[92422]: purged_snaps scrub starts 2023-03-09T14:40:23.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:23 smithi038 ceph-mon[92422]: purged_snaps scrub ok 2023-03-09T14:40:23.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:23 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-03-09T14:40:23.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:23 smithi038 ceph-mon[92422]: osd.2 [v2:172.21.15.181:6800/1310630803,v1:172.21.15.181:6801/1310630803] boot 2023-03-09T14:40:23.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:23 smithi038 ceph-mon[92422]: osdmap e18: 3 total, 3 up, 3 in 2023-03-09T14:40:23.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:23 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-03-09T14:40:24.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:24 smithi122 ceph-mon[93869]: pgmap v72: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-03-09T14:40:24.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:24 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-03-09T14:40:24.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:24 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-03-09T14:40:24.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:24 smithi122 ceph-mon[93869]: osdmap e19: 3 total, 3 up, 3 in 2023-03-09T14:40:24.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:24 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-03-09T14:40:24.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:24 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-03-09T14:40:24.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:24 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-03-09T14:40:24.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:24 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:40:24.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:24 smithi181 ceph-mon[94189]: pgmap v72: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-03-09T14:40:24.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:24 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-03-09T14:40:24.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:24 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-03-09T14:40:24.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:24 smithi181 ceph-mon[94189]: osdmap e19: 3 total, 3 up, 3 in 2023-03-09T14:40:24.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:24 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-03-09T14:40:24.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:24 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-03-09T14:40:24.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:24 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-03-09T14:40:24.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:24 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:40:24.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:24 smithi038 ceph-mon[92422]: pgmap v72: 0 pgs: ; 0 B data, 10 MiB used, 179 GiB / 179 GiB avail 2023-03-09T14:40:24.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:24 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-03-09T14:40:24.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:24 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-03-09T14:40:24.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:24 smithi038 ceph-mon[92422]: osdmap e19: 3 total, 3 up, 3 in 2023-03-09T14:40:24.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:24 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-03-09T14:40:24.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:24 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-03-09T14:40:24.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:24 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-03-09T14:40:24.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:24 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:40:24.959 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:40:24 smithi038 sudo[108313]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-03-09T14:40:24.959 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:40:24 smithi038 sudo[108313]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-03-09T14:40:24.960 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:40:24 smithi038 sudo[108313]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-03-09T14:40:24.960 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:40:24 smithi038 sudo[108313]: pam_unix(sudo:session): session closed for user root 2023-03-09T14:40:25.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:25 smithi122 ceph-mon[93869]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:40:25.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:25 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-03-09T14:40:25.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:25 smithi122 ceph-mon[93869]: osdmap e20: 3 total, 3 up, 3 in 2023-03-09T14:40:25.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:25 smithi181 ceph-mon[94189]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:40:25.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:25 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-03-09T14:40:25.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:25 smithi181 ceph-mon[94189]: osdmap e20: 3 total, 3 up, 3 in 2023-03-09T14:40:25.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:25 smithi038 ceph-mon[92422]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:40:25.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:25 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-03-09T14:40:25.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:25 smithi038 ceph-mon[92422]: osdmap e20: 3 total, 3 up, 3 in 2023-03-09T14:40:25.501 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:40:25 smithi038 sudo[108340]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-03-09T14:40:25.501 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:40:25 smithi038 sudo[108340]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-03-09T14:40:25.501 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:40:25 smithi038 sudo[108340]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-03-09T14:40:25.501 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:40:25 smithi038 sudo[108340]: pam_unix(sudo:session): session closed for user root 2023-03-09T14:40:25.758 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:40:25 smithi122 sudo[101237]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-03-09T14:40:25.758 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:40:25 smithi122 sudo[101237]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-03-09T14:40:25.758 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:40:25 smithi122 sudo[101237]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-03-09T14:40:25.758 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:40:25 smithi122 sudo[101237]: pam_unix(sudo:session): session closed for user root 2023-03-09T14:40:26.227 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:40:25 smithi122 sudo[101240]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-03-09T14:40:26.227 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:40:25 smithi122 sudo[101240]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-03-09T14:40:26.227 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:40:25 smithi122 sudo[101240]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-03-09T14:40:26.227 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:40:25 smithi122 sudo[101240]: pam_unix(sudo:session): session closed for user root 2023-03-09T14:40:26.271 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:40:26.269+0000 7f5b48ff9700 1 -- 172.21.15.38:0/2291684705 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mgrmap(e 16) v1 ==== 133198+0+0 (secure 0 0 0) 0x7f5b4000c040 con 0x7f5b4c103340 2023-03-09T14:40:26.279 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:40:26 smithi181 sudo[102213]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-03-09T14:40:26.280 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:40:26 smithi181 sudo[102213]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-03-09T14:40:26.280 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:40:26 smithi181 sudo[102213]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-03-09T14:40:26.280 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:40:26 smithi181 sudo[102213]: pam_unix(sudo:session): session closed for user root 2023-03-09T14:40:26.280 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:26 smithi181 ceph-mon[94189]: pgmap v75: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-03-09T14:40:26.280 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:26 smithi181 ceph-mon[94189]: osdmap e21: 3 total, 3 up, 3 in 2023-03-09T14:40:26.280 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:26 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:26.281 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:26 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:26.281 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:26 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/1930896425' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "969df8fa-2d19-42c7-9f2a-5c5fac15d4a5"}]: dispatch 2023-03-09T14:40:26.281 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:26 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/1930896425' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "969df8fa-2d19-42c7-9f2a-5c5fac15d4a5"}]': finished 2023-03-09T14:40:26.281 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:26 smithi181 ceph-mon[94189]: osdmap e22: 4 total, 3 up, 4 in 2023-03-09T14:40:26.281 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:26 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-03-09T14:40:26.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:26 smithi122 ceph-mon[93869]: pgmap v75: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-03-09T14:40:26.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:26 smithi122 ceph-mon[93869]: osdmap e21: 3 total, 3 up, 3 in 2023-03-09T14:40:26.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:26 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:26.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:26 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:26.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:26 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/1930896425' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "969df8fa-2d19-42c7-9f2a-5c5fac15d4a5"}]: dispatch 2023-03-09T14:40:26.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:26 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/1930896425' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "969df8fa-2d19-42c7-9f2a-5c5fac15d4a5"}]': finished 2023-03-09T14:40:26.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:26 smithi122 ceph-mon[93869]: osdmap e22: 4 total, 3 up, 4 in 2023-03-09T14:40:26.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:26 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-03-09T14:40:26.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:26 smithi038 ceph-mon[92422]: pgmap v75: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2023-03-09T14:40:26.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:26 smithi038 ceph-mon[92422]: osdmap e21: 3 total, 3 up, 3 in 2023-03-09T14:40:26.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:26 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:26.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:26 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:26.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:26 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/1930896425' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "969df8fa-2d19-42c7-9f2a-5c5fac15d4a5"}]: dispatch 2023-03-09T14:40:26.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:26 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/1930896425' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "969df8fa-2d19-42c7-9f2a-5c5fac15d4a5"}]': finished 2023-03-09T14:40:26.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:26 smithi038 ceph-mon[92422]: osdmap e22: 4 total, 3 up, 4 in 2023-03-09T14:40:26.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:26 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-03-09T14:40:26.731 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:40:26 smithi181 sudo[102216]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-03-09T14:40:26.731 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:40:26 smithi181 sudo[102216]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-03-09T14:40:26.732 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:40:26 smithi181 sudo[102216]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-03-09T14:40:26.732 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:40:26 smithi181 sudo[102216]: pam_unix(sudo:session): session closed for user root 2023-03-09T14:40:27.205 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:26 smithi038 sudo[108444]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-03-09T14:40:27.206 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:26 smithi038 sudo[108444]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-03-09T14:40:27.206 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:26 smithi038 sudo[108444]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-03-09T14:40:27.206 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:26 smithi038 sudo[108444]: pam_unix(sudo:session): session closed for user root 2023-03-09T14:40:27.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:27 smithi038 sudo[108494]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-03-09T14:40:27.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:27 smithi038 sudo[108494]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-03-09T14:40:27.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:27 smithi038 sudo[108494]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-03-09T14:40:27.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:27 smithi038 sudo[108494]: pam_unix(sudo:session): session closed for user root 2023-03-09T14:40:27.702 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:27 smithi181 sudo[102219]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-03-09T14:40:27.703 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:27 smithi181 sudo[102219]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-03-09T14:40:27.703 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:27 smithi181 sudo[102219]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-03-09T14:40:27.703 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:27 smithi181 sudo[102219]: pam_unix(sudo:session): session closed for user root 2023-03-09T14:40:27.959 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:27 smithi122 ceph-mon[93869]: mgrmap e16: x(active, since 2m), standbys: z, y 2023-03-09T14:40:27.960 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:27 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:27.960 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:27 smithi122 ceph-mon[93869]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-03-09T14:40:27.960 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:27 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/2646259227' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-03-09T14:40:27.960 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:27 smithi122 ceph-mon[93869]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-03-09T14:40:27.961 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:27 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-03-09T14:40:27.961 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:27 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-03-09T14:40:27.961 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:27 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-03-09T14:40:27.961 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:27 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-03-09T14:40:27.962 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:27 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-03-09T14:40:27.962 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:27 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-03-09T14:40:27.962 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:27 smithi122 sudo[101243]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-03-09T14:40:27.962 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:27 smithi122 sudo[101243]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-03-09T14:40:27.962 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:27 smithi122 sudo[101243]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-03-09T14:40:27.962 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:27 smithi122 sudo[101243]: pam_unix(sudo:session): session closed for user root 2023-03-09T14:40:27.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:27 smithi181 sudo[102222]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-03-09T14:40:27.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:27 smithi181 sudo[102222]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-03-09T14:40:27.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:27 smithi181 sudo[102222]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-03-09T14:40:27.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:27 smithi181 sudo[102222]: pam_unix(sudo:session): session closed for user root 2023-03-09T14:40:27.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:27 smithi181 ceph-mon[94189]: mgrmap e16: x(active, since 2m), standbys: z, y 2023-03-09T14:40:27.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:27 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:27.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:27 smithi181 ceph-mon[94189]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-03-09T14:40:27.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:27 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/2646259227' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-03-09T14:40:27.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:27 smithi181 ceph-mon[94189]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-03-09T14:40:27.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:27 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-03-09T14:40:27.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:27 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-03-09T14:40:27.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:27 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-03-09T14:40:27.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:27 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-03-09T14:40:27.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:27 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-03-09T14:40:27.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:27 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-03-09T14:40:28.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:27 smithi038 ceph-mon[92422]: mgrmap e16: x(active, since 2m), standbys: z, y 2023-03-09T14:40:28.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:27 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:28.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:27 smithi038 ceph-mon[92422]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-03-09T14:40:28.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:27 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/2646259227' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-03-09T14:40:28.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:27 smithi038 ceph-mon[92422]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-03-09T14:40:28.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:27 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-03-09T14:40:28.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:27 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-03-09T14:40:28.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:27 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-03-09T14:40:28.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:27 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-03-09T14:40:28.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:27 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-03-09T14:40:28.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:27 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-03-09T14:40:28.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:28 smithi122 sudo[101246]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-03-09T14:40:28.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:28 smithi122 sudo[101246]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-03-09T14:40:28.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:28 smithi122 sudo[101246]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-03-09T14:40:28.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:28 smithi122 sudo[101246]: pam_unix(sudo:session): session closed for user root 2023-03-09T14:40:28.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:28 smithi122 ceph-mon[93869]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2023-03-09T14:40:28.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:28 smithi122 ceph-mon[93869]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-03-09T14:40:28.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:28 smithi122 ceph-mon[93869]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-03-09T14:40:28.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:28 smithi122 ceph-mon[93869]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-03-09T14:40:28.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:28 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-03-09T14:40:28.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:28 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-03-09T14:40:28.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:28 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-03-09T14:40:28.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:28 smithi122 ceph-mon[93869]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-03-09T14:40:28.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:28 smithi181 ceph-mon[94189]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2023-03-09T14:40:28.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:28 smithi181 ceph-mon[94189]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-03-09T14:40:28.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:28 smithi181 ceph-mon[94189]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-03-09T14:40:28.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:28 smithi181 ceph-mon[94189]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-03-09T14:40:28.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:28 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-03-09T14:40:28.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:28 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-03-09T14:40:28.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:28 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-03-09T14:40:28.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:28 smithi181 ceph-mon[94189]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-03-09T14:40:29.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:28 smithi038 ceph-mon[92422]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2023-03-09T14:40:29.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:28 smithi038 ceph-mon[92422]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-03-09T14:40:29.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:28 smithi038 ceph-mon[92422]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-03-09T14:40:29.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:28 smithi038 ceph-mon[92422]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-03-09T14:40:29.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:28 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-03-09T14:40:29.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:28 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-03-09T14:40:29.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:28 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-03-09T14:40:29.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:28 smithi038 ceph-mon[92422]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-03-09T14:40:30.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:30 smithi122 ceph-mon[93869]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2023-03-09T14:40:30.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:30 smithi181 ceph-mon[94189]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2023-03-09T14:40:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:30 smithi038 ceph-mon[92422]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2023-03-09T14:40:32.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:32 smithi122 ceph-mon[93869]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 6.4 KiB/s rd, 279 KiB/s wr, 19 op/s 2023-03-09T14:40:32.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:32 smithi181 ceph-mon[94189]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 6.4 KiB/s rd, 279 KiB/s wr, 19 op/s 2023-03-09T14:40:32.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:32 smithi038 ceph-mon[92422]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 6.4 KiB/s rd, 279 KiB/s wr, 19 op/s 2023-03-09T14:40:33.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:33 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-03-09T14:40:33.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:33 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:40:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:33 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-03-09T14:40:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:33 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:40:33.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:33 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-03-09T14:40:33.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:33 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:40:34.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:34 smithi038 ceph-mon[92422]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 195 KiB/s wr, 13 op/s 2023-03-09T14:40:34.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:34 smithi038 ceph-mon[92422]: Deploying daemon osd.3 on smithi038 2023-03-09T14:40:34.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:34 smithi122 ceph-mon[93869]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 195 KiB/s wr, 13 op/s 2023-03-09T14:40:34.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:34 smithi122 ceph-mon[93869]: Deploying daemon osd.3 on smithi038 2023-03-09T14:40:34.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:34 smithi181 ceph-mon[94189]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 195 KiB/s wr, 13 op/s 2023-03-09T14:40:34.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:34 smithi181 ceph-mon[94189]: Deploying daemon osd.3 on smithi038 2023-03-09T14:40:36.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:36 smithi038 ceph-mon[92422]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 150 KiB/s wr, 10 op/s 2023-03-09T14:40:36.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:36 smithi122 ceph-mon[93869]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 150 KiB/s wr, 10 op/s 2023-03-09T14:40:36.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:36 smithi181 ceph-mon[94189]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 150 KiB/s wr, 10 op/s 2023-03-09T14:40:38.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:38 smithi038 ceph-mon[92422]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 2.8 KiB/s rd, 122 KiB/s wr, 8 op/s 2023-03-09T14:40:38.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:38 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:38.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:38 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:38.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:38 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:40:38.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:38 smithi122 ceph-mon[93869]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 2.8 KiB/s rd, 122 KiB/s wr, 8 op/s 2023-03-09T14:40:38.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:38 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:38.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:38 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:38.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:38 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:40:38.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:38 smithi181 ceph-mon[94189]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 2.8 KiB/s rd, 122 KiB/s wr, 8 op/s 2023-03-09T14:40:38.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:38 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:38.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:38 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:38.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:38 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:40:40.335 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:40:40.332+0000 7f5b48ff9700 1 -- 172.21.15.38:0/2291684705 <== mgr.14150 v2:172.21.15.38:6800/3328887328 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f5b4c0618d0 con 0x7f5b3409f370 2023-03-09T14:40:40.335 INFO:teuthology.orchestra.run.smithi038.stdout:Created osd(s) 3 on host 'smithi038' 2023-03-09T14:40:40.337 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:40:40.335+0000 7f5b5214b700 1 -- 172.21.15.38:0/2291684705 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f5b3409f370 msgr2=0x7f5b340a1820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:40:40.337 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:40:40.335+0000 7f5b5214b700 1 --2- 172.21.15.38:0/2291684705 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f5b3409f370 0x7f5b340a1820 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5b4c197200 tx=0x7f5b3c00b410 comp rx=0 tx=0).stop 2023-03-09T14:40:40.338 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:40:40.335+0000 7f5b5214b700 1 -- 172.21.15.38:0/2291684705 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5b4c103340 msgr2=0x7f5b4c196270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:40:40.338 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:40:40.335+0000 7f5b5214b700 1 --2- 172.21.15.38:0/2291684705 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5b4c103340 0x7f5b4c196270 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5b400059d0 tx=0x7f5b400047a0 comp rx=0 tx=0).stop 2023-03-09T14:40:40.338 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:40:40.335+0000 7f5b5214b700 1 -- 172.21.15.38:0/2291684705 shutdown_connections 2023-03-09T14:40:40.338 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:40:40.335+0000 7f5b5214b700 1 --2- 172.21.15.38:0/2291684705 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f5b3409f370 0x7f5b340a1820 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:40:40.338 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:40:40.335+0000 7f5b5214b700 1 --2- 172.21.15.38:0/2291684705 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5b4c105c60 0x7f5b4c19d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:40:40.338 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:40:40.335+0000 7f5b5214b700 1 --2- 172.21.15.38:0/2291684705 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5b4c100a20 0x7f5b4c195d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:40:40.338 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:40:40.335+0000 7f5b5214b700 1 --2- 172.21.15.38:0/2291684705 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5b4c103340 0x7f5b4c196270 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:40:40.338 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:40:40.335+0000 7f5b5214b700 1 -- 172.21.15.38:0/2291684705 >> 172.21.15.38:0/2291684705 conn(0x7f5b4c0fa480 msgr2=0x7f5b4c101680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:40:40.339 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:40:40.335+0000 7f5b5214b700 1 -- 172.21.15.38:0/2291684705 shutdown_connections 2023-03-09T14:40:40.339 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:40:40.335+0000 7f5b5214b700 1 -- 172.21.15.38:0/2291684705 wait complete. 2023-03-09T14:40:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:40 smithi038 ceph-mon[92422]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 109 KiB/s wr, 7 op/s 2023-03-09T14:40:40.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:40 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:40.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:40 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:40 smithi122 ceph-mon[93869]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 109 KiB/s wr, 7 op/s 2023-03-09T14:40:40.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:40 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:40.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:40 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:40.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:40 smithi181 ceph-mon[94189]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 109 KiB/s wr, 7 op/s 2023-03-09T14:40:40.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:40 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:40.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:40 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:40.738 DEBUG:teuthology.orchestra.run.smithi038:osd.3> sudo journalctl -f -n 0 -u ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.3.service 2023-03-09T14:40:40.741 INFO:tasks.cephadm:Deploying osd.4 on smithi122 with /dev/vg_nvme/lv_3... 2023-03-09T14:40:40.741 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-03-09T14:40:40.772 INFO:journalctl@ceph.osd.3.smithi038.stdout:-- Logs begin at Thu 2023-03-09 14:24:59 UTC. -- 2023-03-09T14:40:41.395 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.b/config 2023-03-09T14:40:41.500 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:40:41 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 2023-03-09T14:40:41.281+0000 7fcbac5e5640 -1 osd.3 0 log_to_monitors true 2023-03-09T14:40:41.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:41 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:40:41.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:41 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:40:41.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:41 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:41.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:41 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:41.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:41 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:41.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:41 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:40:41.571 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:41 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:40:41.571 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:41 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:40:41.571 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:41 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:41.572 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:41 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:41.572 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:41 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:41.572 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:41 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:40:41.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:41 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:40:41.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:41 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:40:41.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:41 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:41.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:41 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:41.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:41 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:41.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:41 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:40:42.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:42 smithi038 ceph-mon[92422]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 109 KiB/s wr, 7 op/s 2023-03-09T14:40:42.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:42 smithi038 ceph-mon[92422]: from='osd.3 [v2:172.21.15.38:6810/2719814100,v1:172.21.15.38:6811/2719814100]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-03-09T14:40:42.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:42 smithi122 ceph-mon[93869]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 109 KiB/s wr, 7 op/s 2023-03-09T14:40:42.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:42 smithi122 ceph-mon[93869]: from='osd.3 [v2:172.21.15.38:6810/2719814100,v1:172.21.15.38:6811/2719814100]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-03-09T14:40:42.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:42 smithi181 ceph-mon[94189]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 2.5 KiB/s rd, 109 KiB/s wr, 7 op/s 2023-03-09T14:40:42.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:42 smithi181 ceph-mon[94189]: from='osd.3 [v2:172.21.15.38:6810/2719814100,v1:172.21.15.38:6811/2719814100]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-03-09T14:40:42.908 INFO:teuthology.orchestra.run.smithi122.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-03-09T14:40:42.909 INFO:teuthology.orchestra.run.smithi122.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-03-09T14:40:42.909 INFO:teuthology.orchestra.run.smithi122.stderr: stderr: 10+0 records in 2023-03-09T14:40:42.909 INFO:teuthology.orchestra.run.smithi122.stderr:10+0 records out 2023-03-09T14:40:42.909 INFO:teuthology.orchestra.run.smithi122.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0107412 s, 976 MB/s 2023-03-09T14:40:42.909 INFO:teuthology.orchestra.run.smithi122.stderr:--> Zapping successful for: 2023-03-09T14:40:43.348 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph orch daemon add osd smithi122:vg_nvme/lv_3 2023-03-09T14:40:43.500 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:40:43 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 2023-03-09T14:40:43.260+0000 7fcba056a700 -1 osd.3 0 waiting for initial osdmap 2023-03-09T14:40:43.501 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:40:43 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 2023-03-09T14:40:43.265+0000 7fcb9a35b700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-09T14:40:43.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:43 smithi038 ceph-mon[92422]: from='osd.3 [v2:172.21.15.38:6810/2719814100,v1:172.21.15.38:6811/2719814100]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-03-09T14:40:43.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:43 smithi038 ceph-mon[92422]: osdmap e23: 4 total, 3 up, 4 in 2023-03-09T14:40:43.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:43 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-03-09T14:40:43.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:43 smithi038 ceph-mon[92422]: from='osd.3 [v2:172.21.15.38:6810/2719814100,v1:172.21.15.38:6811/2719814100]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2023-03-09T14:40:43.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:43 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:43.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:43 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:43.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:43 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:40:43.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:43 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:40:43.503 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:43 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:43.503 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:43 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:40:43.503 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:43 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:40:43.503 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:43 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:43.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:43 smithi122 ceph-mon[93869]: from='osd.3 [v2:172.21.15.38:6810/2719814100,v1:172.21.15.38:6811/2719814100]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-03-09T14:40:43.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:43 smithi122 ceph-mon[93869]: osdmap e23: 4 total, 3 up, 4 in 2023-03-09T14:40:43.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:43 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-03-09T14:40:43.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:43 smithi122 ceph-mon[93869]: from='osd.3 [v2:172.21.15.38:6810/2719814100,v1:172.21.15.38:6811/2719814100]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2023-03-09T14:40:43.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:43 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:43.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:43 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:43.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:43 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:40:43.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:43 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:40:43.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:43 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:43.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:43 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:40:43.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:43 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:40:43.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:43 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:43.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:43 smithi181 ceph-mon[94189]: from='osd.3 [v2:172.21.15.38:6810/2719814100,v1:172.21.15.38:6811/2719814100]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-03-09T14:40:43.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:43 smithi181 ceph-mon[94189]: osdmap e23: 4 total, 3 up, 4 in 2023-03-09T14:40:43.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:43 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-03-09T14:40:43.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:43 smithi181 ceph-mon[94189]: from='osd.3 [v2:172.21.15.38:6810/2719814100,v1:172.21.15.38:6811/2719814100]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2023-03-09T14:40:43.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:43 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:43.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:43 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:43.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:43 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:40:43.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:43 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:40:43.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:43 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:43.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:43 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:40:43.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:43 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:40:43.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:43 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:40:43.986 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.b/config 2023-03-09T14:40:44.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:44 smithi122 ceph-mon[93869]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2023-03-09T14:40:44.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:44 smithi122 ceph-mon[93869]: Detected new or changed devices on smithi038 2023-03-09T14:40:44.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:44 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi038 to 8586M 2023-03-09T14:40:44.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:44 smithi122 ceph-mon[93869]: from='osd.3 [v2:172.21.15.38:6810/2719814100,v1:172.21.15.38:6811/2719814100]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2023-03-09T14:40:44.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:44 smithi122 ceph-mon[93869]: osdmap e24: 4 total, 3 up, 4 in 2023-03-09T14:40:44.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:44 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-03-09T14:40:44.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:44 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-03-09T14:40:44.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:44 smithi122 ceph-mon[93869]: osd.3 [v2:172.21.15.38:6810/2719814100,v1:172.21.15.38:6811/2719814100] boot 2023-03-09T14:40:44.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:44 smithi122 ceph-mon[93869]: osdmap e25: 4 total, 4 up, 4 in 2023-03-09T14:40:44.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:44 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-03-09T14:40:44.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:44 smithi181 ceph-mon[94189]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2023-03-09T14:40:44.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:44 smithi181 ceph-mon[94189]: Detected new or changed devices on smithi038 2023-03-09T14:40:44.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:44 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi038 to 8586M 2023-03-09T14:40:44.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:44 smithi181 ceph-mon[94189]: from='osd.3 [v2:172.21.15.38:6810/2719814100,v1:172.21.15.38:6811/2719814100]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2023-03-09T14:40:44.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:44 smithi181 ceph-mon[94189]: osdmap e24: 4 total, 3 up, 4 in 2023-03-09T14:40:44.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:44 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-03-09T14:40:44.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:44 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-03-09T14:40:44.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:44 smithi181 ceph-mon[94189]: osd.3 [v2:172.21.15.38:6810/2719814100,v1:172.21.15.38:6811/2719814100] boot 2023-03-09T14:40:44.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:44 smithi181 ceph-mon[94189]: osdmap e25: 4 total, 4 up, 4 in 2023-03-09T14:40:44.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:44 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-03-09T14:40:44.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:44 smithi038 ceph-mon[92422]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2023-03-09T14:40:44.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:44 smithi038 ceph-mon[92422]: Detected new or changed devices on smithi038 2023-03-09T14:40:44.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:44 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi038 to 8586M 2023-03-09T14:40:44.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:44 smithi038 ceph-mon[92422]: from='osd.3 [v2:172.21.15.38:6810/2719814100,v1:172.21.15.38:6811/2719814100]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2023-03-09T14:40:44.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:44 smithi038 ceph-mon[92422]: osdmap e24: 4 total, 3 up, 4 in 2023-03-09T14:40:44.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:44 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-03-09T14:40:44.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:44 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-03-09T14:40:44.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:44 smithi038 ceph-mon[92422]: osd.3 [v2:172.21.15.38:6810/2719814100,v1:172.21.15.38:6811/2719814100] boot 2023-03-09T14:40:44.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:44 smithi038 ceph-mon[92422]: osdmap e25: 4 total, 4 up, 4 in 2023-03-09T14:40:44.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:44 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-03-09T14:40:45.192 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:40:45.187+0000 7f23a3c35700 1 -- 172.21.15.122:0/31822081 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f239c0feea0 msgr2=0x7f239c0ff2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:40:45.192 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:40:45.187+0000 7f23a3c35700 1 --2- 172.21.15.122:0/31822081 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f239c0feea0 0x7f239c0ff2b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f238c009990 tx=0x7f238c009ca0 comp rx=0 tx=0).stop 2023-03-09T14:40:45.192 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:40:45.187+0000 7f23a3c35700 1 -- 172.21.15.122:0/31822081 shutdown_connections 2023-03-09T14:40:45.193 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:40:45.187+0000 7f23a3c35700 1 --2- 172.21.15.122:0/31822081 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f239c1009a0 0x7f239c1072d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:40:45.193 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:40:45.187+0000 7f23a3c35700 1 --2- 172.21.15.122:0/31822081 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f239c0feea0 0x7f239c0ff2b0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:40:45.193 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:40:45.187+0000 7f23a3c35700 1 --2- 172.21.15.122:0/31822081 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f239c0ffff0 0x7f239c100460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:40:45.193 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:40:45.187+0000 7f23a3c35700 1 -- 172.21.15.122:0/31822081 >> 172.21.15.122:0/31822081 conn(0x7f239c0fa480 msgr2=0x7f239c0fc8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:40:45.193 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:40:45.187+0000 7f23a3c35700 1 -- 172.21.15.122:0/31822081 shutdown_connections 2023-03-09T14:40:45.193 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:40:45.187+0000 7f23a3c35700 1 -- 172.21.15.122:0/31822081 wait complete. 2023-03-09T14:40:45.194 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:40:45.187+0000 7f23a3c35700 1 Processor -- start 2023-03-09T14:40:45.194 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:40:45.188+0000 7f23a3c35700 1 -- start start 2023-03-09T14:40:45.194 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:40:45.188+0000 7f23a3c35700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f239c0feea0 0x7f239c195d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:40:45.194 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:40:45.188+0000 7f23a3c35700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f239c0ffff0 0x7f239c196290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:40:45.194 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:40:45.188+0000 7f23a3c35700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f239c1009a0 0x7f239c19d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:40:45.194 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:40:45.188+0000 7f23a3c35700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f239c19d890 con 0x7f239c1009a0 2023-03-09T14:40:45.194 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:40:45.188+0000 7f23a3c35700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f239c19da00 con 0x7f239c0ffff0 2023-03-09T14:40:45.195 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:40:45.188+0000 7f23a3c35700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f239c19dbc0 con 0x7f239c0feea0 2023-03-09T14:40:45.195 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:40:45.188+0000 7f23a11d0700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f239c0ffff0 0x7f239c196290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:40:45.195 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:40:45.188+0000 7f23a11d0700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f239c0ffff0 0x7f239c196290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:49256/0 (socket says 172.21.15.122:49256) 2023-03-09T14:40:45.195 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:40:45.188+0000 7f23a11d0700 1 -- 172.21.15.122:0/2171784838 learned_addr learned my addr 172.21.15.122:0/2171784838 (peer_addr_for_me v2:172.21.15.122:0/0) 2023-03-09T14:40:45.195 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:40:45.188+0000 7f23a21d2700 1 --2- 172.21.15.122:0/2171784838 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f239c1009a0 0x7f239c19d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:40:45.195 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:40:45.188+0000 7f23a19d1700 1 --2- 172.21.15.122:0/2171784838 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f239c0feea0 0x7f239c195d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:40:45.196 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:40:45.188+0000 7f23a11d0700 1 -- 172.21.15.122:0/2171784838 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f239c0feea0 msgr2=0x7f239c195d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:40:45.196 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:40:45.188+0000 7f23a11d0700 1 --2- 172.21.15.122:0/2171784838 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f239c0feea0 0x7f239c195d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:40:45.196 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:40:45.188+0000 7f23a11d0700 1 -- 172.21.15.122:0/2171784838 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f239c1009a0 msgr2=0x7f239c19d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:40:45.196 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:40:45.188+0000 7f23a11d0700 1 --2- 172.21.15.122:0/2171784838 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f239c1009a0 0x7f239c19d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:40:45.196 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:40:45.188+0000 7f23a11d0700 1 -- 172.21.15.122:0/2171784838 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f238c0095f0 con 0x7f239c0ffff0 2023-03-09T14:40:45.196 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:40:45.188+0000 7f23a11d0700 1 --2- 172.21.15.122:0/2171784838 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f239c0ffff0 0x7f239c196290 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f239800d950 tx=0x7f239800dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:40:45.197 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:40:45.188+0000 7f2392ffd700 1 -- 172.21.15.122:0/2171784838 <== mon.2 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2398009940 con 0x7f239c0ffff0 2023-03-09T14:40:45.197 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:40:45.188+0000 7f2392ffd700 1 -- 172.21.15.122:0/2171784838 <== mon.2 v2:172.21.15.122:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2398010460 con 0x7f239c0ffff0 2023-03-09T14:40:45.197 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:40:45.188+0000 7f23a3c35700 1 -- 172.21.15.122:0/2171784838 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f239c10a850 con 0x7f239c0ffff0 2023-03-09T14:40:45.197 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:40:45.188+0000 7f2392ffd700 1 -- 172.21.15.122:0/2171784838 <== mon.2 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f239800f620 con 0x7f239c0ffff0 2023-03-09T14:40:45.197 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:40:45.189+0000 7f23a3c35700 1 -- 172.21.15.122:0/2171784838 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f239c10a6c0 con 0x7f239c0ffff0 2023-03-09T14:40:45.198 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:40:45.189+0000 7f23a3c35700 1 -- 172.21.15.122:0/2171784838 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f239c106ec0 con 0x7f239c0ffff0 2023-03-09T14:40:45.198 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:40:45.191+0000 7f2392ffd700 1 -- 172.21.15.122:0/2171784838 <== mon.2 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 16) v1 ==== 133198+0+0 (secure 0 0 0) 0x7f2398022070 con 0x7f239c0ffff0 2023-03-09T14:40:45.198 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:40:45.191+0000 7f2392ffd700 1 --2- 172.21.15.122:0/2171784838 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f238809f3c0 0x7f23880a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:40:45.199 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:40:45.192+0000 7f2392ffd700 1 -- 172.21.15.122:0/2171784838 <== mon.2 v2:172.21.15.122:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 3702+0+0 (secure 0 0 0) 0x7f23980c88d0 con 0x7f239c0ffff0 2023-03-09T14:40:45.199 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:40:45.192+0000 7f23a19d1700 1 --2- 172.21.15.122:0/2171784838 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f238809f3c0 0x7f23880a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:40:45.199 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:40:45.192+0000 7f23a19d1700 1 --2- 172.21.15.122:0/2171784838 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f238809f3c0 0x7f23880a1870 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f238c00b5c0 tx=0x7f238c005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:40:45.199 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:40:45.195+0000 7f2392ffd700 1 -- 172.21.15.122:0/2171784838 <== mon.2 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f2398093ee0 con 0x7f239c0ffff0 2023-03-09T14:40:45.393 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:40:45.390+0000 7f23a3c35700 1 -- 172.21.15.122:0/2171784838 --> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f239c0618c0 con 0x7f238809f3c0 2023-03-09T14:40:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:45 smithi122 ceph-mon[93869]: purged_snaps scrub starts 2023-03-09T14:40:45.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:45 smithi122 ceph-mon[93869]: purged_snaps scrub ok 2023-03-09T14:40:45.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:45 smithi122 ceph-mon[93869]: osdmap e26: 4 total, 4 up, 4 in 2023-03-09T14:40:45.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:45 smithi181 ceph-mon[94189]: purged_snaps scrub starts 2023-03-09T14:40:45.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:45 smithi181 ceph-mon[94189]: purged_snaps scrub ok 2023-03-09T14:40:45.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:45 smithi181 ceph-mon[94189]: osdmap e26: 4 total, 4 up, 4 in 2023-03-09T14:40:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:45 smithi038 ceph-mon[92422]: purged_snaps scrub starts 2023-03-09T14:40:45.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:45 smithi038 ceph-mon[92422]: purged_snaps scrub ok 2023-03-09T14:40:45.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:45 smithi038 ceph-mon[92422]: osdmap e26: 4 total, 4 up, 4 in 2023-03-09T14:40:46.614 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:46 smithi122 ceph-mon[93869]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2023-03-09T14:40:46.614 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:46 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-03-09T14:40:46.615 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:46 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-03-09T14:40:46.615 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:46 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:40:46.615 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:46 smithi122 ceph-mon[93869]: osdmap e27: 4 total, 4 up, 4 in 2023-03-09T14:40:46.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:46 smithi181 ceph-mon[94189]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2023-03-09T14:40:46.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:46 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-03-09T14:40:46.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:46 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-03-09T14:40:46.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:46 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:40:46.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:46 smithi181 ceph-mon[94189]: osdmap e27: 4 total, 4 up, 4 in 2023-03-09T14:40:46.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:46 smithi038 ceph-mon[92422]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2023-03-09T14:40:46.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:46 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-03-09T14:40:46.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:46 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-03-09T14:40:46.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:46 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:40:46.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:46 smithi038 ceph-mon[92422]: osdmap e27: 4 total, 4 up, 4 in 2023-03-09T14:40:47.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:47 smithi122 ceph-mon[93869]: from='client.24212 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:40:47.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:47 smithi122 ceph-mon[93869]: osdmap e28: 4 total, 4 up, 4 in 2023-03-09T14:40:47.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:47 smithi181 ceph-mon[94189]: from='client.24212 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:40:47.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:47 smithi181 ceph-mon[94189]: osdmap e28: 4 total, 4 up, 4 in 2023-03-09T14:40:47.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:47 smithi038 ceph-mon[92422]: from='client.24212 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:40:47.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:47 smithi038 ceph-mon[92422]: osdmap e28: 4 total, 4 up, 4 in 2023-03-09T14:40:48.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:48 smithi122 ceph-mon[93869]: pgmap v93: 1 pgs: 1 remapped+peering; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2023-03-09T14:40:48.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:48 smithi122 ceph-mon[93869]: from='client.? 172.21.15.122:0/3823453515' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44a6dce3-e06e-4cbd-9404-62a86d01ab6b"}]: dispatch 2023-03-09T14:40:48.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:48 smithi122 ceph-mon[93869]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44a6dce3-e06e-4cbd-9404-62a86d01ab6b"}]: dispatch 2023-03-09T14:40:48.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:48 smithi122 ceph-mon[93869]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "44a6dce3-e06e-4cbd-9404-62a86d01ab6b"}]': finished 2023-03-09T14:40:48.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:48 smithi122 ceph-mon[93869]: osdmap e29: 5 total, 4 up, 5 in 2023-03-09T14:40:48.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:48 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-03-09T14:40:48.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:48 smithi181 ceph-mon[94189]: pgmap v93: 1 pgs: 1 remapped+peering; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2023-03-09T14:40:48.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:48 smithi181 ceph-mon[94189]: from='client.? 172.21.15.122:0/3823453515' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44a6dce3-e06e-4cbd-9404-62a86d01ab6b"}]: dispatch 2023-03-09T14:40:48.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:48 smithi181 ceph-mon[94189]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44a6dce3-e06e-4cbd-9404-62a86d01ab6b"}]: dispatch 2023-03-09T14:40:48.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:48 smithi181 ceph-mon[94189]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "44a6dce3-e06e-4cbd-9404-62a86d01ab6b"}]': finished 2023-03-09T14:40:48.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:48 smithi181 ceph-mon[94189]: osdmap e29: 5 total, 4 up, 5 in 2023-03-09T14:40:48.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:48 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-03-09T14:40:48.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:48 smithi038 ceph-mon[92422]: pgmap v93: 1 pgs: 1 remapped+peering; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2023-03-09T14:40:48.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:48 smithi038 ceph-mon[92422]: from='client.? 172.21.15.122:0/3823453515' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44a6dce3-e06e-4cbd-9404-62a86d01ab6b"}]: dispatch 2023-03-09T14:40:48.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:48 smithi038 ceph-mon[92422]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44a6dce3-e06e-4cbd-9404-62a86d01ab6b"}]: dispatch 2023-03-09T14:40:48.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:48 smithi038 ceph-mon[92422]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "44a6dce3-e06e-4cbd-9404-62a86d01ab6b"}]': finished 2023-03-09T14:40:48.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:48 smithi038 ceph-mon[92422]: osdmap e29: 5 total, 4 up, 5 in 2023-03-09T14:40:48.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:48 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-03-09T14:40:49.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:49 smithi122 ceph-mon[93869]: from='client.? 172.21.15.122:0/1945076084' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-03-09T14:40:49.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:49 smithi181 ceph-mon[94189]: from='client.? 172.21.15.122:0/1945076084' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-03-09T14:40:49.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:49 smithi038 ceph-mon[92422]: from='client.? 172.21.15.122:0/1945076084' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-03-09T14:40:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:50 smithi122 ceph-mon[93869]: pgmap v96: 1 pgs: 1 remapped+peering; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2023-03-09T14:40:50.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:50 smithi181 ceph-mon[94189]: pgmap v96: 1 pgs: 1 remapped+peering; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2023-03-09T14:40:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:50 smithi038 ceph-mon[92422]: pgmap v96: 1 pgs: 1 remapped+peering; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2023-03-09T14:40:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:52 smithi122 ceph-mon[93869]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 106 KiB/s, 0 objects/s recovering 2023-03-09T14:40:52.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:52 smithi181 ceph-mon[94189]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 106 KiB/s, 0 objects/s recovering 2023-03-09T14:40:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:52 smithi038 ceph-mon[92422]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 106 KiB/s, 0 objects/s recovering 2023-03-09T14:40:54.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:54 smithi122 ceph-mon[93869]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 90 KiB/s, 0 objects/s recovering 2023-03-09T14:40:54.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:54 smithi181 ceph-mon[94189]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 90 KiB/s, 0 objects/s recovering 2023-03-09T14:40:54.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:54 smithi038 ceph-mon[92422]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 90 KiB/s, 0 objects/s recovering 2023-03-09T14:40:55.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:55 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-03-09T14:40:55.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:55 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:40:55.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:55 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-03-09T14:40:55.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:55 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:40:55.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:55 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-03-09T14:40:55.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:55 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:40:56.665 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:56 smithi122 ceph-mon[93869]: Deploying daemon osd.4 on smithi122 2023-03-09T14:40:56.665 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:56 smithi122 ceph-mon[93869]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-03-09T14:40:56.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:56 smithi181 ceph-mon[94189]: Deploying daemon osd.4 on smithi122 2023-03-09T14:40:56.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:56 smithi181 ceph-mon[94189]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-03-09T14:40:56.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:56 smithi038 ceph-mon[92422]: Deploying daemon osd.4 on smithi122 2023-03-09T14:40:56.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:56 smithi038 ceph-mon[92422]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-03-09T14:40:58.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:40:58 smithi038 ceph-mon[92422]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 61 KiB/s, 0 objects/s recovering 2023-03-09T14:40:58.682 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:40:58 smithi122 ceph-mon[93869]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 61 KiB/s, 0 objects/s recovering 2023-03-09T14:40:58.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:40:58 smithi181 ceph-mon[94189]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 61 KiB/s, 0 objects/s recovering 2023-03-09T14:41:00.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:00 smithi122 ceph-mon[93869]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 53 KiB/s, 0 objects/s recovering 2023-03-09T14:41:00.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:00 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:00.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:00 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:00.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:00 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:41:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:00 smithi181 ceph-mon[94189]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 53 KiB/s, 0 objects/s recovering 2023-03-09T14:41:00.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:00 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:00.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:00 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:00.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:00 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:41:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:00 smithi038 ceph-mon[92422]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 53 KiB/s, 0 objects/s recovering 2023-03-09T14:41:00.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:00 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:00.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:00 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:00.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:00 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:41:01.917 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:41:01.914+0000 7f2392ffd700 1 -- 172.21.15.122:0/2171784838 <== mgr.14150 v2:172.21.15.38:6800/3328887328 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f239c0618c0 con 0x7f238809f3c0 2023-03-09T14:41:01.917 INFO:teuthology.orchestra.run.smithi122.stdout:Created osd(s) 4 on host 'smithi122' 2023-03-09T14:41:01.919 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:41:01.917+0000 7f23a3c35700 1 -- 172.21.15.122:0/2171784838 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f238809f3c0 msgr2=0x7f23880a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:41:01.919 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:41:01.917+0000 7f23a3c35700 1 --2- 172.21.15.122:0/2171784838 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f238809f3c0 0x7f23880a1870 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f238c00b5c0 tx=0x7f238c005fb0 comp rx=0 tx=0).stop 2023-03-09T14:41:01.919 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:41:01.917+0000 7f23a3c35700 1 -- 172.21.15.122:0/2171784838 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f239c0ffff0 msgr2=0x7f239c196290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:41:01.919 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:41:01.917+0000 7f23a3c35700 1 --2- 172.21.15.122:0/2171784838 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f239c0ffff0 0x7f239c196290 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f239800d950 tx=0x7f239800dc60 comp rx=0 tx=0).stop 2023-03-09T14:41:01.920 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:41:01.917+0000 7f23a3c35700 1 -- 172.21.15.122:0/2171784838 shutdown_connections 2023-03-09T14:41:01.920 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:41:01.917+0000 7f23a3c35700 1 --2- 172.21.15.122:0/2171784838 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f239c1009a0 0x7f239c19d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:41:01.920 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:41:01.917+0000 7f23a3c35700 1 --2- 172.21.15.122:0/2171784838 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f238809f3c0 0x7f23880a1870 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:41:01.920 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:41:01.917+0000 7f23a3c35700 1 --2- 172.21.15.122:0/2171784838 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f239c0feea0 0x7f239c195d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:41:01.920 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:41:01.917+0000 7f23a3c35700 1 --2- 172.21.15.122:0/2171784838 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f239c0ffff0 0x7f239c196290 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:41:01.920 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:41:01.917+0000 7f23a3c35700 1 -- 172.21.15.122:0/2171784838 >> 172.21.15.122:0/2171784838 conn(0x7f239c0fa480 msgr2=0x7f239c0fc690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:41:01.920 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:41:01.917+0000 7f23a3c35700 1 -- 172.21.15.122:0/2171784838 shutdown_connections 2023-03-09T14:41:01.920 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:41:01.917+0000 7f23a3c35700 1 -- 172.21.15.122:0/2171784838 wait complete. 2023-03-09T14:41:02.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:02 smithi122 ceph-mon[93869]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-03-09T14:41:02.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:02 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:02.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:02 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:02.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:02 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:02.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:02 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:02.296 DEBUG:teuthology.orchestra.run.smithi122:osd.4> sudo journalctl -f -n 0 -u ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.4.service 2023-03-09T14:41:02.298 INFO:tasks.cephadm:Deploying osd.5 on smithi181 with /dev/vg_nvme/lv_3... 2023-03-09T14:41:02.298 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-03-09T14:41:02.327 INFO:journalctl@ceph.osd.4.smithi122.stdout:-- Logs begin at Thu 2023-03-09 14:25:09 UTC. -- 2023-03-09T14:41:02.330 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:02 smithi181 ceph-mon[94189]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-03-09T14:41:02.331 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:02 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:02.331 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:02 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:02.331 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:02 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:02.331 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:02 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:02.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:02 smithi038 ceph-mon[92422]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-03-09T14:41:02.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:02 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:02.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:02 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:02.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:02 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:02.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:02 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:02.945 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.c/config 2023-03-09T14:41:03.228 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:41:02 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 2023-03-09T14:41:02.945+0000 7f83ce6b8640 -1 osd.4 0 log_to_monitors true 2023-03-09T14:41:03.474 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:03 smithi181 ceph-mon[94189]: from='osd.4 [v2:172.21.15.122:6808/2480764994,v1:172.21.15.122:6809/2480764994]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-03-09T14:41:03.474 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:03 smithi181 ceph-mon[94189]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-03-09T14:41:03.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:03 smithi038 ceph-mon[92422]: from='osd.4 [v2:172.21.15.122:6808/2480764994,v1:172.21.15.122:6809/2480764994]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-03-09T14:41:03.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:03 smithi038 ceph-mon[92422]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-03-09T14:41:03.578 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:03 smithi122 ceph-mon[93869]: from='osd.4 [v2:172.21.15.122:6808/2480764994,v1:172.21.15.122:6809/2480764994]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-03-09T14:41:03.578 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:03 smithi122 ceph-mon[93869]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-03-09T14:41:04.466 INFO:teuthology.orchestra.run.smithi181.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-03-09T14:41:04.467 INFO:teuthology.orchestra.run.smithi181.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-03-09T14:41:04.467 INFO:teuthology.orchestra.run.smithi181.stderr: stderr: 10+0 records in 2023-03-09T14:41:04.467 INFO:teuthology.orchestra.run.smithi181.stderr:10+0 records out 2023-03-09T14:41:04.467 INFO:teuthology.orchestra.run.smithi181.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111432 s, 941 MB/s 2023-03-09T14:41:04.468 INFO:teuthology.orchestra.run.smithi181.stderr:--> Zapping successful for: 2023-03-09T14:41:04.537 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:04 smithi122 ceph-mon[93869]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2023-03-09T14:41:04.537 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:04 smithi122 ceph-mon[93869]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-03-09T14:41:04.538 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:04 smithi122 ceph-mon[93869]: osdmap e30: 5 total, 4 up, 5 in 2023-03-09T14:41:04.538 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:04 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-03-09T14:41:04.538 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:04 smithi122 ceph-mon[93869]: from='osd.4 [v2:172.21.15.122:6808/2480764994,v1:172.21.15.122:6809/2480764994]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-03-09T14:41:04.538 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:04 smithi122 ceph-mon[93869]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-03-09T14:41:04.538 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:41:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 2023-03-09T14:41:04.255+0000 7f83c263d700 -1 osd.4 0 waiting for initial osdmap 2023-03-09T14:41:04.539 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:41:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 2023-03-09T14:41:04.263+0000 7f83bc42e700 -1 osd.4 31 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-09T14:41:04.599 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:04 smithi181 ceph-mon[94189]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2023-03-09T14:41:04.600 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:04 smithi181 ceph-mon[94189]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-03-09T14:41:04.600 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:04 smithi181 ceph-mon[94189]: osdmap e30: 5 total, 4 up, 5 in 2023-03-09T14:41:04.600 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:04 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-03-09T14:41:04.600 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:04 smithi181 ceph-mon[94189]: from='osd.4 [v2:172.21.15.122:6808/2480764994,v1:172.21.15.122:6809/2480764994]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-03-09T14:41:04.600 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:04 smithi181 ceph-mon[94189]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-03-09T14:41:04.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:04 smithi038 ceph-mon[92422]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2023-03-09T14:41:04.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:04 smithi038 ceph-mon[92422]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-03-09T14:41:04.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:04 smithi038 ceph-mon[92422]: osdmap e30: 5 total, 4 up, 5 in 2023-03-09T14:41:04.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:04 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-03-09T14:41:04.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:04 smithi038 ceph-mon[92422]: from='osd.4 [v2:172.21.15.122:6808/2480764994,v1:172.21.15.122:6809/2480764994]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-03-09T14:41:04.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:04 smithi038 ceph-mon[92422]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-03-09T14:41:04.799 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph orch daemon add osd smithi181:vg_nvme/lv_3 2023-03-09T14:41:05.485 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.c/config 2023-03-09T14:41:05.688 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:05 smithi181 ceph-mon[94189]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2023-03-09T14:41:05.689 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:05 smithi181 ceph-mon[94189]: osdmap e31: 5 total, 4 up, 5 in 2023-03-09T14:41:05.689 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:05 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-03-09T14:41:05.689 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:05 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-03-09T14:41:05.689 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:05 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:05.689 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:05 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:05.689 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:05 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:41:05.689 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:05 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:41:05.690 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:05 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:05.690 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:05 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:41:05.690 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:05 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:41:05.690 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:05 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:05.690 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:05 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:41:05.690 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:05 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:41:05.690 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:05 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:41:05.690 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:05 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:05 smithi122 ceph-mon[93869]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2023-03-09T14:41:05.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:05 smithi122 ceph-mon[93869]: osdmap e31: 5 total, 4 up, 5 in 2023-03-09T14:41:05.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:05 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-03-09T14:41:05.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:05 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-03-09T14:41:05.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:05 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:05.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:05 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:05.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:05 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:41:05.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:05 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:41:05.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:05 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:05.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:05 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:41:05.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:05 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:41:05.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:05 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:05.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:05 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:41:05.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:05 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:41:05.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:05 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:41:05.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:05 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:05 smithi038 ceph-mon[92422]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2023-03-09T14:41:05.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:05 smithi038 ceph-mon[92422]: osdmap e31: 5 total, 4 up, 5 in 2023-03-09T14:41:05.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:05 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-03-09T14:41:05.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:05 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-03-09T14:41:05.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:05 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:05.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:05 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:05.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:05 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:41:05.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:05 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:41:05.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:05 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:05.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:05 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:41:05.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:05 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:41:05.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:05 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:05.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:05 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:41:05.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:05 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:41:05.753 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:05 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:41:05.753 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:05 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:06.660 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:41:06.655+0000 7fba10cba700 1 -- 172.21.15.181:0/84478740 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fba0c1009a0 msgr2=0x7fba0c1072d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:41:06.661 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:41:06.655+0000 7fba10cba700 1 --2- 172.21.15.181:0/84478740 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fba0c1009a0 0x7fba0c1072d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fba0400b0a0 tx=0x7fba0400b3b0 comp rx=0 tx=0).stop 2023-03-09T14:41:06.661 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:41:06.655+0000 7fba10cba700 1 -- 172.21.15.181:0/84478740 shutdown_connections 2023-03-09T14:41:06.661 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:41:06.655+0000 7fba10cba700 1 --2- 172.21.15.181:0/84478740 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fba0c0feea0 0x7fba0c0ff2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:41:06.661 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:41:06.655+0000 7fba10cba700 1 --2- 172.21.15.181:0/84478740 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fba0c1009a0 0x7fba0c1072d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:41:06.661 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:41:06.655+0000 7fba10cba700 1 --2- 172.21.15.181:0/84478740 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba0c0ffff0 0x7fba0c100460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:41:06.661 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:41:06.655+0000 7fba10cba700 1 -- 172.21.15.181:0/84478740 >> 172.21.15.181:0/84478740 conn(0x7fba0c0fa480 msgr2=0x7fba0c0fc8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:41:06.662 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:41:06.656+0000 7fba10cba700 1 -- 172.21.15.181:0/84478740 shutdown_connections 2023-03-09T14:41:06.662 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:41:06.656+0000 7fba10cba700 1 -- 172.21.15.181:0/84478740 wait complete. 2023-03-09T14:41:06.662 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:41:06.656+0000 7fba10cba700 1 Processor -- start 2023-03-09T14:41:06.662 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:41:06.656+0000 7fba10cba700 1 -- start start 2023-03-09T14:41:06.662 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:41:06.656+0000 7fba10cba700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba0c0feea0 0x7fba0c068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:41:06.662 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:41:06.656+0000 7fba10cba700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fba0c0ffff0 0x7fba0c19aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:41:06.662 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:41:06.656+0000 7fba10cba700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fba0c1009a0 0x7fba0c19d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:41:06.663 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:41:06.656+0000 7fba10cba700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba0c19d910 con 0x7fba0c0ffff0 2023-03-09T14:41:06.663 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:41:06.656+0000 7fba10cba700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba0c19da80 con 0x7fba0c0feea0 2023-03-09T14:41:06.663 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:41:06.656+0000 7fba10cba700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba0c19dc40 con 0x7fba0c1009a0 2023-03-09T14:41:06.663 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:41:06.657+0000 7fba0ad9d700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fba0c1009a0 0x7fba0c19d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:41:06.663 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:41:06.657+0000 7fba0ad9d700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fba0c1009a0 0x7fba0c19d140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:50592/0 (socket says 172.21.15.181:50592) 2023-03-09T14:41:06.663 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:41:06.657+0000 7fba0ad9d700 1 -- 172.21.15.181:0/1719544106 learned_addr learned my addr 172.21.15.181:0/1719544106 (peer_addr_for_me v2:172.21.15.181:0/0) 2023-03-09T14:41:06.663 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:41:06.657+0000 7fba0a59c700 1 --2- 172.21.15.181:0/1719544106 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba0c0feea0 0x7fba0c068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:41:06.663 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:41:06.657+0000 7fba09d9b700 1 --2- 172.21.15.181:0/1719544106 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fba0c0ffff0 0x7fba0c19aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:41:06.664 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:41:06.657+0000 7fba0ad9d700 1 -- 172.21.15.181:0/1719544106 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba0c0feea0 msgr2=0x7fba0c068db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:41:06.664 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:41:06.657+0000 7fba0ad9d700 1 --2- 172.21.15.181:0/1719544106 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba0c0feea0 0x7fba0c068db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:41:06.664 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:41:06.657+0000 7fba0ad9d700 1 -- 172.21.15.181:0/1719544106 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fba0c0ffff0 msgr2=0x7fba0c19aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:41:06.664 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:41:06.657+0000 7fba0ad9d700 1 --2- 172.21.15.181:0/1719544106 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fba0c0ffff0 0x7fba0c19aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:41:06.664 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:41:06.657+0000 7fba0ad9d700 1 -- 172.21.15.181:0/1719544106 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba04009d00 con 0x7fba0c1009a0 2023-03-09T14:41:06.664 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:41:06.657+0000 7fba09d9b700 1 --2- 172.21.15.181:0/1719544106 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fba0c0ffff0 0x7fba0c19aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-09T14:41:06.664 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:41:06.657+0000 7fba0ad9d700 1 --2- 172.21.15.181:0/1719544106 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fba0c1009a0 0x7fba0c19d140 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fba040062a0 tx=0x7fba04004d00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:41:06.665 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:41:06.657+0000 7fb9fb7fe700 1 -- 172.21.15.181:0/1719544106 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba0400e040 con 0x7fba0c1009a0 2023-03-09T14:41:06.665 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:41:06.657+0000 7fba10cba700 1 -- 172.21.15.181:0/1719544106 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba0c10a800 con 0x7fba0c1009a0 2023-03-09T14:41:06.665 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:41:06.657+0000 7fb9fb7fe700 1 -- 172.21.15.181:0/1719544106 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fba0400cec0 con 0x7fba0c1009a0 2023-03-09T14:41:06.665 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:41:06.657+0000 7fb9fb7fe700 1 -- 172.21.15.181:0/1719544106 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba04004020 con 0x7fba0c1009a0 2023-03-09T14:41:06.665 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:41:06.657+0000 7fba10cba700 1 -- 172.21.15.181:0/1719544106 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba0c10a670 con 0x7fba0c1009a0 2023-03-09T14:41:06.666 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:41:06.658+0000 7fba10cba700 1 -- 172.21.15.181:0/1719544106 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba0c066b10 con 0x7fba0c1009a0 2023-03-09T14:41:06.666 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:41:06.659+0000 7fb9fb7fe700 1 -- 172.21.15.181:0/1719544106 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 16) v1 ==== 133198+0+0 (secure 0 0 0) 0x7fba0401d690 con 0x7fba0c1009a0 2023-03-09T14:41:06.666 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:41:06.660+0000 7fb9fb7fe700 1 --2- 172.21.15.181:0/1719544106 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fb9f409f3c0 0x7fb9f40a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:41:06.666 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:41:06.660+0000 7fb9fb7fe700 1 -- 172.21.15.181:0/1719544106 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 4101+0+0 (secure 0 0 0) 0x7fba040ca9f0 con 0x7fba0c1009a0 2023-03-09T14:41:06.667 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:41:06.660+0000 7fba0a59c700 1 --2- 172.21.15.181:0/1719544106 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fb9f409f3c0 0x7fb9f40a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:41:06.667 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:41:06.661+0000 7fba0a59c700 1 --2- 172.21.15.181:0/1719544106 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fb9f409f3c0 0x7fb9f40a1870 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fba00007c60 tx=0x7fba000073d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:41:06.667 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:41:06.664+0000 7fb9fb7fe700 1 -- 172.21.15.181:0/1719544106 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7fba040cce60 con 0x7fba0c1009a0 2023-03-09T14:41:06.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:06 smithi122 ceph-mon[93869]: purged_snaps scrub starts 2023-03-09T14:41:06.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:06 smithi122 ceph-mon[93869]: purged_snaps scrub ok 2023-03-09T14:41:06.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:06 smithi122 ceph-mon[93869]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2023-03-09T14:41:06.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:06 smithi122 ceph-mon[93869]: Detected new or changed devices on smithi122 2023-03-09T14:41:06.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:06 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 8584M 2023-03-09T14:41:06.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:06 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-03-09T14:41:06.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:06 smithi122 ceph-mon[93869]: osd.4 [v2:172.21.15.122:6808/2480764994,v1:172.21.15.122:6809/2480764994] boot 2023-03-09T14:41:06.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:06 smithi122 ceph-mon[93869]: osdmap e32: 5 total, 5 up, 5 in 2023-03-09T14:41:06.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:06 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-03-09T14:41:06.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:06 smithi181 ceph-mon[94189]: purged_snaps scrub starts 2023-03-09T14:41:06.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:06 smithi181 ceph-mon[94189]: purged_snaps scrub ok 2023-03-09T14:41:06.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:06 smithi181 ceph-mon[94189]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2023-03-09T14:41:06.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:06 smithi181 ceph-mon[94189]: Detected new or changed devices on smithi122 2023-03-09T14:41:06.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:06 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 8584M 2023-03-09T14:41:06.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:06 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-03-09T14:41:06.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:06 smithi181 ceph-mon[94189]: osd.4 [v2:172.21.15.122:6808/2480764994,v1:172.21.15.122:6809/2480764994] boot 2023-03-09T14:41:06.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:06 smithi181 ceph-mon[94189]: osdmap e32: 5 total, 5 up, 5 in 2023-03-09T14:41:06.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:06 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-03-09T14:41:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:06 smithi038 ceph-mon[92422]: purged_snaps scrub starts 2023-03-09T14:41:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:06 smithi038 ceph-mon[92422]: purged_snaps scrub ok 2023-03-09T14:41:06.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:06 smithi038 ceph-mon[92422]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 24 MiB used, 358 GiB / 358 GiB avail 2023-03-09T14:41:06.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:06 smithi038 ceph-mon[92422]: Detected new or changed devices on smithi122 2023-03-09T14:41:06.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:06 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 8584M 2023-03-09T14:41:06.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:06 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-03-09T14:41:06.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:06 smithi038 ceph-mon[92422]: osd.4 [v2:172.21.15.122:6808/2480764994,v1:172.21.15.122:6809/2480764994] boot 2023-03-09T14:41:06.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:06 smithi038 ceph-mon[92422]: osdmap e32: 5 total, 5 up, 5 in 2023-03-09T14:41:06.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:06 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-03-09T14:41:06.860 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:41:06.857+0000 7fba10cba700 1 -- 172.21.15.181:0/1719544106 --> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fba0c060bc0 con 0x7fb9f409f3c0 2023-03-09T14:41:07.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:07 smithi122 ceph-mon[93869]: osdmap e33: 5 total, 5 up, 5 in 2023-03-09T14:41:07.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:07 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-03-09T14:41:07.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:07 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-03-09T14:41:07.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:07 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:41:07.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:07 smithi181 ceph-mon[94189]: osdmap e33: 5 total, 5 up, 5 in 2023-03-09T14:41:07.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:07 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-03-09T14:41:07.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:07 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-03-09T14:41:07.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:07 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:41:07.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:07 smithi038 ceph-mon[92422]: osdmap e33: 5 total, 5 up, 5 in 2023-03-09T14:41:07.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:07 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-03-09T14:41:07.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:07 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-03-09T14:41:07.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:07 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:41:08.571 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:08 smithi181 ceph-mon[94189]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2023-03-09T14:41:08.572 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:08 smithi181 ceph-mon[94189]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:41:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:08 smithi122 ceph-mon[93869]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2023-03-09T14:41:08.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:08 smithi122 ceph-mon[93869]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:41:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:08 smithi038 ceph-mon[92422]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2023-03-09T14:41:08.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:08 smithi038 ceph-mon[92422]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:41:09.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:09 smithi122 ceph-mon[93869]: from='client.? 172.21.15.181:0/822308945' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "49749574-c00c-4e3a-80e6-e3686df58d5b"}]: dispatch 2023-03-09T14:41:09.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:09 smithi122 ceph-mon[93869]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "49749574-c00c-4e3a-80e6-e3686df58d5b"}]: dispatch 2023-03-09T14:41:09.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:09 smithi181 ceph-mon[94189]: from='client.? 172.21.15.181:0/822308945' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "49749574-c00c-4e3a-80e6-e3686df58d5b"}]: dispatch 2023-03-09T14:41:09.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:09 smithi181 ceph-mon[94189]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "49749574-c00c-4e3a-80e6-e3686df58d5b"}]: dispatch 2023-03-09T14:41:09.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:09 smithi038 ceph-mon[92422]: from='client.? 172.21.15.181:0/822308945' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "49749574-c00c-4e3a-80e6-e3686df58d5b"}]: dispatch 2023-03-09T14:41:09.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:09 smithi038 ceph-mon[92422]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "49749574-c00c-4e3a-80e6-e3686df58d5b"}]: dispatch 2023-03-09T14:41:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:10 smithi122 ceph-mon[93869]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2023-03-09T14:41:10.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:10 smithi122 ceph-mon[93869]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "49749574-c00c-4e3a-80e6-e3686df58d5b"}]': finished 2023-03-09T14:41:10.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:10 smithi122 ceph-mon[93869]: osdmap e34: 6 total, 5 up, 6 in 2023-03-09T14:41:10.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:10 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-03-09T14:41:10.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:10 smithi122 ceph-mon[93869]: from='client.? 172.21.15.181:0/435148492' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-03-09T14:41:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:10 smithi181 ceph-mon[94189]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2023-03-09T14:41:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:10 smithi181 ceph-mon[94189]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "49749574-c00c-4e3a-80e6-e3686df58d5b"}]': finished 2023-03-09T14:41:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:10 smithi181 ceph-mon[94189]: osdmap e34: 6 total, 5 up, 6 in 2023-03-09T14:41:10.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:10 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-03-09T14:41:10.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:10 smithi181 ceph-mon[94189]: from='client.? 172.21.15.181:0/435148492' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-03-09T14:41:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:10 smithi038 ceph-mon[92422]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2023-03-09T14:41:10.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:10 smithi038 ceph-mon[92422]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "49749574-c00c-4e3a-80e6-e3686df58d5b"}]': finished 2023-03-09T14:41:10.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:10 smithi038 ceph-mon[92422]: osdmap e34: 6 total, 5 up, 6 in 2023-03-09T14:41:10.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:10 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-03-09T14:41:10.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:10 smithi038 ceph-mon[92422]: from='client.? 172.21.15.181:0/435148492' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-03-09T14:41:11.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:11 smithi122 ceph-mon[93869]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-03-09T14:41:11.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:11 smithi181 ceph-mon[94189]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-03-09T14:41:11.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:11 smithi038 ceph-mon[92422]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-03-09T14:41:14.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:14 smithi038 ceph-mon[92422]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-03-09T14:41:14.510 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:14 smithi181 ceph-mon[94189]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-03-09T14:41:14.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:14 smithi122 ceph-mon[93869]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-03-09T14:41:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:16 smithi038 ceph-mon[92422]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-03-09T14:41:16.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:16 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-03-09T14:41:16.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:16 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:41:16.561 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:16 smithi181 ceph-mon[94189]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-03-09T14:41:16.562 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:16 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-03-09T14:41:16.562 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:16 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:41:16.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:16 smithi122 ceph-mon[93869]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-03-09T14:41:16.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:16 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-03-09T14:41:16.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:16 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:41:17.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:17 smithi038 ceph-mon[92422]: Deploying daemon osd.5 on smithi181 2023-03-09T14:41:17.714 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:17 smithi181 ceph-mon[94189]: Deploying daemon osd.5 on smithi181 2023-03-09T14:41:17.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:17 smithi122 ceph-mon[93869]: Deploying daemon osd.5 on smithi181 2023-03-09T14:41:18.325 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:18 smithi181 ceph-mon[94189]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-03-09T14:41:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:18 smithi038 ceph-mon[92422]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-03-09T14:41:18.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:18 smithi122 ceph-mon[93869]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-03-09T14:41:20.559 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:20 smithi181 ceph-mon[94189]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-03-09T14:41:20.560 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:20 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:20.560 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:20 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:20.560 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:20 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:41:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:20 smithi122 ceph-mon[93869]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-03-09T14:41:20.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:20 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:20.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:20 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:20.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:20 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:41:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:20 smithi038 ceph-mon[92422]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-03-09T14:41:20.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:20 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:20.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:20 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:20.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:20 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:41:22.588 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:22 smithi181 ceph-mon[94189]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-03-09T14:41:22.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:22 smithi122 ceph-mon[93869]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-03-09T14:41:22.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:22 smithi038 ceph-mon[92422]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-03-09T14:41:22.800 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:41:22.798+0000 7fb9fb7fe700 1 -- 172.21.15.181:0/1719544106 <== mgr.14150 v2:172.21.15.38:6800/3328887328 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fba0c060bc0 con 0x7fb9f409f3c0 2023-03-09T14:41:22.801 INFO:teuthology.orchestra.run.smithi181.stdout:Created osd(s) 5 on host 'smithi181' 2023-03-09T14:41:22.802 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:41:22.801+0000 7fba10cba700 1 -- 172.21.15.181:0/1719544106 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fb9f409f3c0 msgr2=0x7fb9f40a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:41:22.803 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:41:22.801+0000 7fba10cba700 1 --2- 172.21.15.181:0/1719544106 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fb9f409f3c0 0x7fb9f40a1870 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fba00007c60 tx=0x7fba000073d0 comp rx=0 tx=0).stop 2023-03-09T14:41:22.803 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:41:22.801+0000 7fba10cba700 1 -- 172.21.15.181:0/1719544106 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fba0c1009a0 msgr2=0x7fba0c19d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:41:22.803 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:41:22.801+0000 7fba10cba700 1 --2- 172.21.15.181:0/1719544106 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fba0c1009a0 0x7fba0c19d140 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fba040062a0 tx=0x7fba04004d00 comp rx=0 tx=0).stop 2023-03-09T14:41:22.803 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:41:22.801+0000 7fba10cba700 1 -- 172.21.15.181:0/1719544106 shutdown_connections 2023-03-09T14:41:22.803 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:41:22.801+0000 7fba10cba700 1 --2- 172.21.15.181:0/1719544106 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fb9f409f3c0 0x7fb9f40a1870 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:41:22.803 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:41:22.801+0000 7fba10cba700 1 --2- 172.21.15.181:0/1719544106 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fba0c1009a0 0x7fba0c19d140 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:41:22.803 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:41:22.801+0000 7fba10cba700 1 --2- 172.21.15.181:0/1719544106 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba0c0feea0 0x7fba0c068db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:41:22.804 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:41:22.801+0000 7fba10cba700 1 --2- 172.21.15.181:0/1719544106 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fba0c0ffff0 0x7fba0c19aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:41:22.804 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:41:22.801+0000 7fba10cba700 1 -- 172.21.15.181:0/1719544106 >> 172.21.15.181:0/1719544106 conn(0x7fba0c0fa480 msgr2=0x7fba0c0fc670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:41:22.804 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:41:22.801+0000 7fba10cba700 1 -- 172.21.15.181:0/1719544106 shutdown_connections 2023-03-09T14:41:22.804 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:41:22.801+0000 7fba10cba700 1 -- 172.21.15.181:0/1719544106 wait complete. 2023-03-09T14:41:23.261 DEBUG:teuthology.orchestra.run.smithi181:osd.5> sudo journalctl -f -n 0 -u ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.5.service 2023-03-09T14:41:23.263 INFO:tasks.cephadm:Deploying osd.6 on smithi038 with /dev/vg_nvme/lv_2... 2023-03-09T14:41:23.264 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-03-09T14:41:23.293 INFO:journalctl@ceph.osd.5.smithi181.stdout:-- Logs begin at Thu 2023-03-09 14:25:01 UTC. -- 2023-03-09T14:41:23.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:23 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:23.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:23 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:23.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:23 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:41:23.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:23 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:41:23.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:23 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:23.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:23 smithi122 ceph-mon[93869]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-03-09T14:41:23.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:23 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:23.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:23 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:23.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:23 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:41:23.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:23 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:23.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:23 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:23.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:23 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:41:23.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:23 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:41:23.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:23 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:23.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:23 smithi181 ceph-mon[94189]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-03-09T14:41:23.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:23 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:23.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:23 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:23.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:23 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:41:23.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:23 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:23.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:23 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:23.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:23 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:41:23.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:23 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:41:23.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:23 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:23.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:23 smithi038 ceph-mon[92422]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-03-09T14:41:23.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:23 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:23.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:23 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:23.753 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:23 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:41:23.932 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.a/config 2023-03-09T14:41:24.405 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 09 14:41:24 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-5[105518]: 2023-03-09T14:41:24.082+0000 7f5c1c7b3640 -1 osd.5 0 log_to_monitors true 2023-03-09T14:41:24.473 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:24 smithi038 ceph-mon[92422]: from='osd.5 [v2:172.21.15.181:6808/3141346246,v1:172.21.15.181:6809/3141346246]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-03-09T14:41:24.473 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:24 smithi038 ceph-mon[92422]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-03-09T14:41:24.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:24 smithi122 ceph-mon[93869]: from='osd.5 [v2:172.21.15.181:6808/3141346246,v1:172.21.15.181:6809/3141346246]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-03-09T14:41:24.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:24 smithi122 ceph-mon[93869]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-03-09T14:41:24.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:24 smithi181 ceph-mon[94189]: from='osd.5 [v2:172.21.15.181:6808/3141346246,v1:172.21.15.181:6809/3141346246]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-03-09T14:41:24.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:24 smithi181 ceph-mon[94189]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-03-09T14:41:25.640 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-03-09T14:41:25.640 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-03-09T14:41:25.640 INFO:teuthology.orchestra.run.smithi038.stderr: stderr: 10+0 records in 2023-03-09T14:41:25.640 INFO:teuthology.orchestra.run.smithi038.stderr:10+0 records out 2023-03-09T14:41:25.641 INFO:teuthology.orchestra.run.smithi038.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0154656 s, 678 MB/s 2023-03-09T14:41:25.641 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping successful for: 2023-03-09T14:41:25.674 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:25 smithi038 ceph-mon[92422]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-03-09T14:41:25.675 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:25 smithi038 ceph-mon[92422]: osdmap e35: 6 total, 5 up, 6 in 2023-03-09T14:41:25.675 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:25 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-03-09T14:41:25.675 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:25 smithi038 ceph-mon[92422]: from='osd.5 [v2:172.21.15.181:6808/3141346246,v1:172.21.15.181:6809/3141346246]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2023-03-09T14:41:25.675 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:25 smithi038 ceph-mon[92422]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2023-03-09T14:41:25.675 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:25 smithi038 ceph-mon[92422]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-03-09T14:41:25.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:25 smithi122 ceph-mon[93869]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-03-09T14:41:25.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:25 smithi122 ceph-mon[93869]: osdmap e35: 6 total, 5 up, 6 in 2023-03-09T14:41:25.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:25 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-03-09T14:41:25.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:25 smithi122 ceph-mon[93869]: from='osd.5 [v2:172.21.15.181:6808/3141346246,v1:172.21.15.181:6809/3141346246]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2023-03-09T14:41:25.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:25 smithi122 ceph-mon[93869]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2023-03-09T14:41:25.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:25 smithi122 ceph-mon[93869]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-03-09T14:41:25.731 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 09 14:41:25 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-5[105518]: 2023-03-09T14:41:25.427+0000 7f5c10738700 -1 osd.5 0 waiting for initial osdmap 2023-03-09T14:41:25.731 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 09 14:41:25 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-5[105518]: 2023-03-09T14:41:25.433+0000 7f5c0c52d700 -1 osd.5 36 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-09T14:41:25.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:25 smithi181 ceph-mon[94189]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-03-09T14:41:25.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:25 smithi181 ceph-mon[94189]: osdmap e35: 6 total, 5 up, 6 in 2023-03-09T14:41:25.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:25 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-03-09T14:41:25.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:25 smithi181 ceph-mon[94189]: from='osd.5 [v2:172.21.15.181:6808/3141346246,v1:172.21.15.181:6809/3141346246]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2023-03-09T14:41:25.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:25 smithi181 ceph-mon[94189]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2023-03-09T14:41:25.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:25 smithi181 ceph-mon[94189]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-03-09T14:41:26.083 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph orch daemon add osd smithi038:vg_nvme/lv_2 2023-03-09T14:41:26.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:26 smithi122 ceph-mon[93869]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2023-03-09T14:41:26.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:26 smithi122 ceph-mon[93869]: osdmap e36: 6 total, 5 up, 6 in 2023-03-09T14:41:26.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:26 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-03-09T14:41:26.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:26 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-03-09T14:41:26.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:26 smithi122 ceph-mon[93869]: Detected new or changed devices on smithi181 2023-03-09T14:41:26.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:26 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:26.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:26 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:26.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:26 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:41:26.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:26 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:41:26.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:26 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 8584M 2023-03-09T14:41:26.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:26 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:26.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:26 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:41:26.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:26 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:41:26.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:26 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:26.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:26 smithi181 ceph-mon[94189]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2023-03-09T14:41:26.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:26 smithi181 ceph-mon[94189]: osdmap e36: 6 total, 5 up, 6 in 2023-03-09T14:41:26.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:26 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-03-09T14:41:26.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:26 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-03-09T14:41:26.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:26 smithi181 ceph-mon[94189]: Detected new or changed devices on smithi181 2023-03-09T14:41:26.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:26 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:26.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:26 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:26.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:26 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:41:26.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:26 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:41:26.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:26 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 8584M 2023-03-09T14:41:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:26 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:26 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:41:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:26 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:41:26.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:26 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:26.736 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.a/config 2023-03-09T14:41:26.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:26 smithi038 ceph-mon[92422]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2023-03-09T14:41:26.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:26 smithi038 ceph-mon[92422]: osdmap e36: 6 total, 5 up, 6 in 2023-03-09T14:41:26.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:26 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-03-09T14:41:26.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:26 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-03-09T14:41:26.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:26 smithi038 ceph-mon[92422]: Detected new or changed devices on smithi181 2023-03-09T14:41:26.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:26 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:26.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:26 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:26.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:26 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:41:26.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:26 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:41:26.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:26 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 8584M 2023-03-09T14:41:26.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:26 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:26.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:26 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:41:26.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:26 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:41:26.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:26 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:27.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:27 smithi038 ceph-mon[92422]: purged_snaps scrub starts 2023-03-09T14:41:27.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:27 smithi038 ceph-mon[92422]: purged_snaps scrub ok 2023-03-09T14:41:27.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:27 smithi038 ceph-mon[92422]: osd.5 [v2:172.21.15.181:6808/3141346246,v1:172.21.15.181:6809/3141346246] boot 2023-03-09T14:41:27.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:27 smithi038 ceph-mon[92422]: osdmap e37: 6 total, 6 up, 6 in 2023-03-09T14:41:27.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:27 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-03-09T14:41:27.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:27 smithi038 ceph-mon[92422]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-03-09T14:41:27.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:27 smithi038 ceph-mon[92422]: osdmap e38: 6 total, 6 up, 6 in 2023-03-09T14:41:27.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:27 smithi122 ceph-mon[93869]: purged_snaps scrub starts 2023-03-09T14:41:27.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:27 smithi122 ceph-mon[93869]: purged_snaps scrub ok 2023-03-09T14:41:27.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:27 smithi122 ceph-mon[93869]: osd.5 [v2:172.21.15.181:6808/3141346246,v1:172.21.15.181:6809/3141346246] boot 2023-03-09T14:41:27.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:27 smithi122 ceph-mon[93869]: osdmap e37: 6 total, 6 up, 6 in 2023-03-09T14:41:27.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:27 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-03-09T14:41:27.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:27 smithi122 ceph-mon[93869]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-03-09T14:41:27.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:27 smithi122 ceph-mon[93869]: osdmap e38: 6 total, 6 up, 6 in 2023-03-09T14:41:27.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:27 smithi181 ceph-mon[94189]: purged_snaps scrub starts 2023-03-09T14:41:27.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:27 smithi181 ceph-mon[94189]: purged_snaps scrub ok 2023-03-09T14:41:27.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:27 smithi181 ceph-mon[94189]: osd.5 [v2:172.21.15.181:6808/3141346246,v1:172.21.15.181:6809/3141346246] boot 2023-03-09T14:41:27.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:27 smithi181 ceph-mon[94189]: osdmap e37: 6 total, 6 up, 6 in 2023-03-09T14:41:27.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:27 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-03-09T14:41:27.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:27 smithi181 ceph-mon[94189]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 30 MiB used, 447 GiB / 447 GiB avail 2023-03-09T14:41:27.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:27 smithi181 ceph-mon[94189]: osdmap e38: 6 total, 6 up, 6 in 2023-03-09T14:41:28.091 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:41:28.089+0000 7fe1ddb2d700 1 -- 172.21.15.38:0/1385673669 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe1d80feea0 msgr2=0x7fe1d80ff2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:41:28.092 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:41:28.089+0000 7fe1ddb2d700 1 --2- 172.21.15.38:0/1385673669 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe1d80feea0 0x7fe1d80ff2b0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe1c0009990 tx=0x7fe1c0009ca0 comp rx=0 tx=0).stop 2023-03-09T14:41:28.092 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:41:28.089+0000 7fe1ddb2d700 1 -- 172.21.15.38:0/1385673669 shutdown_connections 2023-03-09T14:41:28.092 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:41:28.089+0000 7fe1ddb2d700 1 --2- 172.21.15.38:0/1385673669 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe1d80feea0 0x7fe1d80ff2b0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:41:28.092 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:41:28.089+0000 7fe1ddb2d700 1 --2- 172.21.15.38:0/1385673669 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe1d81009a0 0x7fe1d81072d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:41:28.092 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:41:28.089+0000 7fe1ddb2d700 1 --2- 172.21.15.38:0/1385673669 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe1d80ffff0 0x7fe1d8100460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:41:28.093 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:41:28.089+0000 7fe1ddb2d700 1 -- 172.21.15.38:0/1385673669 >> 172.21.15.38:0/1385673669 conn(0x7fe1d80fa480 msgr2=0x7fe1d80fc8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:41:28.093 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:41:28.089+0000 7fe1ddb2d700 1 -- 172.21.15.38:0/1385673669 shutdown_connections 2023-03-09T14:41:28.093 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:41:28.089+0000 7fe1ddb2d700 1 -- 172.21.15.38:0/1385673669 wait complete. 2023-03-09T14:41:28.095 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:41:28.090+0000 7fe1ddb2d700 1 Processor -- start 2023-03-09T14:41:28.095 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:41:28.090+0000 7fe1ddb2d700 1 -- start start 2023-03-09T14:41:28.095 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:41:28.090+0000 7fe1ddb2d700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe1d80feea0 0x7fe1d8195cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:41:28.096 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:41:28.090+0000 7fe1ddb2d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe1d80ffff0 0x7fe1d8196200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:41:28.096 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:41:28.090+0000 7fe1ddb2d700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe1d81009a0 0x7fe1d819d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:41:28.096 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:41:28.090+0000 7fe1ddb2d700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d819d830 con 0x7fe1d80ffff0 2023-03-09T14:41:28.096 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:41:28.090+0000 7fe1ddb2d700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d819d9a0 con 0x7fe1d80feea0 2023-03-09T14:41:28.097 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:41:28.090+0000 7fe1ddb2d700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d819db60 con 0x7fe1d81009a0 2023-03-09T14:41:28.097 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:41:28.090+0000 7fe1d6ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe1d80ffff0 0x7fe1d8196200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:41:28.097 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:41:28.090+0000 7fe1d6ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe1d80ffff0 0x7fe1d8196200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:35972/0 (socket says 172.21.15.38:35972) 2023-03-09T14:41:28.097 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:41:28.090+0000 7fe1d6ffd700 1 -- 172.21.15.38:0/3314521433 learned_addr learned my addr 172.21.15.38:0/3314521433 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:41:28.097 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:41:28.090+0000 7fe1d77fe700 1 --2- 172.21.15.38:0/3314521433 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe1d80feea0 0x7fe1d8195cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:41:28.098 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:41:28.090+0000 7fe1d7fff700 1 --2- 172.21.15.38:0/3314521433 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe1d81009a0 0x7fe1d819d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:41:28.098 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:41:28.090+0000 7fe1d6ffd700 1 -- 172.21.15.38:0/3314521433 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe1d81009a0 msgr2=0x7fe1d819d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:41:28.098 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:41:28.090+0000 7fe1d6ffd700 1 --2- 172.21.15.38:0/3314521433 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe1d81009a0 0x7fe1d819d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:41:28.098 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:41:28.090+0000 7fe1d6ffd700 1 -- 172.21.15.38:0/3314521433 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe1d80feea0 msgr2=0x7fe1d8195cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:41:28.098 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:41:28.090+0000 7fe1d6ffd700 1 --2- 172.21.15.38:0/3314521433 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe1d80feea0 0x7fe1d8195cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:41:28.098 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:41:28.091+0000 7fe1d6ffd700 1 -- 172.21.15.38:0/3314521433 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1c00095f0 con 0x7fe1d80ffff0 2023-03-09T14:41:28.098 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:41:28.091+0000 7fe1d7fff700 1 --2- 172.21.15.38:0/3314521433 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe1d81009a0 0x7fe1d819d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-09T14:41:28.099 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:41:28.091+0000 7fe1d77fe700 1 --2- 172.21.15.38:0/3314521433 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe1d80feea0 0x7fe1d8195cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-09T14:41:28.099 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:41:28.091+0000 7fe1d6ffd700 1 --2- 172.21.15.38:0/3314521433 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe1d80ffff0 0x7fe1d8196200 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe1c800c9b0 tx=0x7fe1c800cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:41:28.099 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:41:28.091+0000 7fe1d4ff9700 1 -- 172.21.15.38:0/3314521433 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1c8007b00 con 0x7fe1d80ffff0 2023-03-09T14:41:28.099 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:41:28.091+0000 7fe1ddb2d700 1 -- 172.21.15.38:0/3314521433 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1d810a800 con 0x7fe1d80ffff0 2023-03-09T14:41:28.099 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:41:28.091+0000 7fe1d4ff9700 1 -- 172.21.15.38:0/3314521433 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe1c8007c60 con 0x7fe1d80ffff0 2023-03-09T14:41:28.099 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:41:28.091+0000 7fe1d4ff9700 1 -- 172.21.15.38:0/3314521433 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1c8018660 con 0x7fe1d80ffff0 2023-03-09T14:41:28.099 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:41:28.091+0000 7fe1ddb2d700 1 -- 172.21.15.38:0/3314521433 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1d810a670 con 0x7fe1d80ffff0 2023-03-09T14:41:28.100 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:41:28.092+0000 7fe1ddb2d700 1 -- 172.21.15.38:0/3314521433 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1d819e220 con 0x7fe1d80ffff0 2023-03-09T14:41:28.100 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:41:28.093+0000 7fe1d4ff9700 1 -- 172.21.15.38:0/3314521433 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 133198+0+0 (secure 0 0 0) 0x7fe1c8018880 con 0x7fe1d80ffff0 2023-03-09T14:41:28.101 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:41:28.093+0000 7fe1d4ff9700 1 --2- 172.21.15.38:0/3314521433 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fe1c409f3c0 0x7fe1c40a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:41:28.101 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:41:28.094+0000 7fe1d77fe700 1 --2- 172.21.15.38:0/3314521433 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fe1c409f3c0 0x7fe1c40a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:41:28.101 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:41:28.094+0000 7fe1d4ff9700 1 -- 172.21.15.38:0/3314521433 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 4562+0+0 (secure 0 0 0) 0x7fe1c80c9320 con 0x7fe1d80ffff0 2023-03-09T14:41:28.101 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:41:28.094+0000 7fe1d77fe700 1 --2- 172.21.15.38:0/3314521433 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fe1c409f3c0 0x7fe1c40a1870 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe1c0000c00 tx=0x7fe1c0005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:41:28.101 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:41:28.097+0000 7fe1d4ff9700 1 -- 172.21.15.38:0/3314521433 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7fe1c8094660 con 0x7fe1d80ffff0 2023-03-09T14:41:28.310 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:41:28.308+0000 7fe1ddb2d700 1 -- 172.21.15.38:0/3314521433 --> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fe1d819e7d0 con 0x7fe1c409f3c0 2023-03-09T14:41:29.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:29 smithi038 ceph-mon[92422]: osdmap e39: 6 total, 6 up, 6 in 2023-03-09T14:41:29.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:29 smithi038 ceph-mon[92422]: from='client.14355 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:41:29.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:29 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-03-09T14:41:29.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:29 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-03-09T14:41:29.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:29 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:41:29.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:29 smithi122 ceph-mon[93869]: osdmap e39: 6 total, 6 up, 6 in 2023-03-09T14:41:29.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:29 smithi122 ceph-mon[93869]: from='client.14355 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:41:29.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:29 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-03-09T14:41:29.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:29 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-03-09T14:41:29.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:29 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:41:29.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:29 smithi181 ceph-mon[94189]: osdmap e39: 6 total, 6 up, 6 in 2023-03-09T14:41:29.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:29 smithi181 ceph-mon[94189]: from='client.14355 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:41:29.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:29 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-03-09T14:41:29.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:29 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-03-09T14:41:29.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:29 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:41:30.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:30 smithi122 ceph-mon[93869]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2023-03-09T14:41:30.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:30 smithi122 ceph-mon[93869]: osdmap e40: 6 total, 6 up, 6 in 2023-03-09T14:41:30.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:30 smithi181 ceph-mon[94189]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2023-03-09T14:41:30.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:30 smithi181 ceph-mon[94189]: osdmap e40: 6 total, 6 up, 6 in 2023-03-09T14:41:30.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:30 smithi038 ceph-mon[92422]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2023-03-09T14:41:30.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:30 smithi038 ceph-mon[92422]: osdmap e40: 6 total, 6 up, 6 in 2023-03-09T14:41:31.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:31 smithi122 ceph-mon[93869]: osdmap e41: 6 total, 6 up, 6 in 2023-03-09T14:41:31.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:31 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/997863438' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "61c0437e-3f51-45a9-8a61-e98eea0093f8"}]: dispatch 2023-03-09T14:41:31.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:31 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/997863438' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "61c0437e-3f51-45a9-8a61-e98eea0093f8"}]': finished 2023-03-09T14:41:31.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:31 smithi122 ceph-mon[93869]: osdmap e42: 7 total, 6 up, 7 in 2023-03-09T14:41:31.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:31 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-03-09T14:41:31.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:31 smithi181 ceph-mon[94189]: osdmap e41: 6 total, 6 up, 6 in 2023-03-09T14:41:31.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:31 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/997863438' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "61c0437e-3f51-45a9-8a61-e98eea0093f8"}]: dispatch 2023-03-09T14:41:31.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:31 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/997863438' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "61c0437e-3f51-45a9-8a61-e98eea0093f8"}]': finished 2023-03-09T14:41:31.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:31 smithi181 ceph-mon[94189]: osdmap e42: 7 total, 6 up, 7 in 2023-03-09T14:41:31.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:31 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-03-09T14:41:31.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:31 smithi038 ceph-mon[92422]: osdmap e41: 6 total, 6 up, 6 in 2023-03-09T14:41:31.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:31 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/997863438' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "61c0437e-3f51-45a9-8a61-e98eea0093f8"}]: dispatch 2023-03-09T14:41:31.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:31 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/997863438' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "61c0437e-3f51-45a9-8a61-e98eea0093f8"}]': finished 2023-03-09T14:41:31.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:31 smithi038 ceph-mon[92422]: osdmap e42: 7 total, 6 up, 7 in 2023-03-09T14:41:31.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:31 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-03-09T14:41:32.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:32 smithi122 ceph-mon[93869]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-03-09T14:41:32.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:32 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/4017660363' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-03-09T14:41:32.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:32 smithi181 ceph-mon[94189]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-03-09T14:41:32.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:32 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/4017660363' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-03-09T14:41:32.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:32 smithi038 ceph-mon[92422]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-03-09T14:41:32.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:32 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/4017660363' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-03-09T14:41:34.588 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:34 smithi038 ceph-mon[92422]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail; 129 KiB/s, 0 objects/s recovering 2023-03-09T14:41:34.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:34 smithi122 ceph-mon[93869]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail; 129 KiB/s, 0 objects/s recovering 2023-03-09T14:41:34.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:34 smithi181 ceph-mon[94189]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail; 129 KiB/s, 0 objects/s recovering 2023-03-09T14:41:36.340 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:36 smithi038 ceph-mon[92422]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-03-09T14:41:36.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:36 smithi122 ceph-mon[93869]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-03-09T14:41:36.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:36 smithi181 ceph-mon[94189]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-03-09T14:41:38.438 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:38 smithi038 ceph-mon[92422]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail; 77 KiB/s, 0 objects/s recovering 2023-03-09T14:41:38.438 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:38 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-03-09T14:41:38.438 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:38 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:41:38.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:38 smithi122 ceph-mon[93869]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail; 77 KiB/s, 0 objects/s recovering 2023-03-09T14:41:38.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:38 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-03-09T14:41:38.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:38 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:41:38.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:38 smithi181 ceph-mon[94189]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail; 77 KiB/s, 0 objects/s recovering 2023-03-09T14:41:38.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:38 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-03-09T14:41:38.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:38 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:41:39.654 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:39 smithi038 ceph-mon[92422]: Deploying daemon osd.6 on smithi038 2023-03-09T14:41:39.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:39 smithi122 ceph-mon[93869]: Deploying daemon osd.6 on smithi038 2023-03-09T14:41:39.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:39 smithi181 ceph-mon[94189]: Deploying daemon osd.6 on smithi038 2023-03-09T14:41:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:40 smithi122 ceph-mon[93869]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail; 68 KiB/s, 0 objects/s recovering 2023-03-09T14:41:40.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:40 smithi181 ceph-mon[94189]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail; 68 KiB/s, 0 objects/s recovering 2023-03-09T14:41:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:40 smithi038 ceph-mon[92422]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail; 68 KiB/s, 0 objects/s recovering 2023-03-09T14:41:42.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:42 smithi122 ceph-mon[93869]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2023-03-09T14:41:42.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:42 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:42.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:42 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:42.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:42 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:41:42.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:42 smithi181 ceph-mon[94189]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2023-03-09T14:41:42.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:42 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:42.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:42 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:42.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:42 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:41:42.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:42 smithi038 ceph-mon[92422]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2023-03-09T14:41:42.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:42 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:42.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:42 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:42.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:42 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:41:44.596 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:44 smithi038 ceph-mon[92422]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2023-03-09T14:41:44.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:44 smithi122 ceph-mon[93869]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2023-03-09T14:41:44.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:44 smithi181 ceph-mon[94189]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2023-03-09T14:41:44.857 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:41:44.854+0000 7fe1d4ff9700 1 -- 172.21.15.38:0/3314521433 <== mgr.14150 v2:172.21.15.38:6800/3328887328 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fe1d819e7d0 con 0x7fe1c409f3c0 2023-03-09T14:41:44.857 INFO:teuthology.orchestra.run.smithi038.stdout:Created osd(s) 6 on host 'smithi038' 2023-03-09T14:41:44.858 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:41:44.856+0000 7fe1ddb2d700 1 -- 172.21.15.38:0/3314521433 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fe1c409f3c0 msgr2=0x7fe1c40a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:41:44.859 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:41:44.856+0000 7fe1ddb2d700 1 --2- 172.21.15.38:0/3314521433 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fe1c409f3c0 0x7fe1c40a1870 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe1c0000c00 tx=0x7fe1c0005fb0 comp rx=0 tx=0).stop 2023-03-09T14:41:44.859 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:41:44.856+0000 7fe1ddb2d700 1 -- 172.21.15.38:0/3314521433 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe1d80ffff0 msgr2=0x7fe1d8196200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:41:44.859 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:41:44.856+0000 7fe1ddb2d700 1 --2- 172.21.15.38:0/3314521433 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe1d80ffff0 0x7fe1d8196200 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe1c800c9b0 tx=0x7fe1c800cd70 comp rx=0 tx=0).stop 2023-03-09T14:41:44.859 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:41:44.856+0000 7fe1ddb2d700 1 -- 172.21.15.38:0/3314521433 shutdown_connections 2023-03-09T14:41:44.859 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:41:44.856+0000 7fe1ddb2d700 1 --2- 172.21.15.38:0/3314521433 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fe1c409f3c0 0x7fe1c40a1870 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:41:44.859 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:41:44.856+0000 7fe1ddb2d700 1 --2- 172.21.15.38:0/3314521433 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe1d81009a0 0x7fe1d819d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:41:44.860 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:41:44.856+0000 7fe1ddb2d700 1 --2- 172.21.15.38:0/3314521433 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe1d80feea0 0x7fe1d8195cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:41:44.860 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:41:44.856+0000 7fe1ddb2d700 1 --2- 172.21.15.38:0/3314521433 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe1d80ffff0 0x7fe1d8196200 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:41:44.860 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:41:44.856+0000 7fe1ddb2d700 1 -- 172.21.15.38:0/3314521433 >> 172.21.15.38:0/3314521433 conn(0x7fe1d80fa480 msgr2=0x7fe1d8068220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:41:44.860 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:41:44.856+0000 7fe1ddb2d700 1 -- 172.21.15.38:0/3314521433 shutdown_connections 2023-03-09T14:41:44.860 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:41:44.856+0000 7fe1ddb2d700 1 -- 172.21.15.38:0/3314521433 wait complete. 2023-03-09T14:41:45.222 DEBUG:teuthology.orchestra.run.smithi038:osd.6> sudo journalctl -f -n 0 -u ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.6.service 2023-03-09T14:41:45.224 INFO:tasks.cephadm:Deploying osd.7 on smithi122 with /dev/vg_nvme/lv_2... 2023-03-09T14:41:45.225 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-03-09T14:41:45.269 INFO:journalctl@ceph.osd.6.smithi038.stdout:-- Logs begin at Thu 2023-03-09 14:24:59 UTC. -- 2023-03-09T14:41:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:45 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:45.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:45 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:45.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:45 smithi038 ceph-mon[92422]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2023-03-09T14:41:45.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:45 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:45.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:45 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:45.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:45 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:45.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:45 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:45.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:45 smithi122 ceph-mon[93869]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2023-03-09T14:41:45.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:45 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:45.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:45 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:45.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:45 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:45.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:45 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:45.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:45 smithi181 ceph-mon[94189]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2023-03-09T14:41:45.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:45 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:45.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:45 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:45.889 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.b/config 2023-03-09T14:41:46.173 INFO:journalctl@ceph.osd.6.smithi038.stdout:Mar 09 14:41:45 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-6[116185]: 2023-03-09T14:41:45.834+0000 7f2a4aaa9640 -1 osd.6 0 log_to_monitors true 2023-03-09T14:41:46.414 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:46 smithi122 ceph-mon[93869]: from='osd.6 [v2:172.21.15.38:6818/2122788957,v1:172.21.15.38:6819/2122788957]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-03-09T14:41:46.493 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:46 smithi038 ceph-mon[92422]: from='osd.6 [v2:172.21.15.38:6818/2122788957,v1:172.21.15.38:6819/2122788957]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-03-09T14:41:46.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:46 smithi181 ceph-mon[94189]: from='osd.6 [v2:172.21.15.38:6818/2122788957,v1:172.21.15.38:6819/2122788957]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-03-09T14:41:47.413 INFO:teuthology.orchestra.run.smithi122.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-03-09T14:41:47.413 INFO:teuthology.orchestra.run.smithi122.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-03-09T14:41:47.413 INFO:teuthology.orchestra.run.smithi122.stderr: stderr: 10+0 records in 2023-03-09T14:41:47.414 INFO:teuthology.orchestra.run.smithi122.stderr:10+0 records out 2023-03-09T14:41:47.414 INFO:teuthology.orchestra.run.smithi122.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0110851 s, 946 MB/s 2023-03-09T14:41:47.414 INFO:teuthology.orchestra.run.smithi122.stderr:--> Zapping successful for: 2023-03-09T14:41:47.714 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:47 smithi038 ceph-mon[92422]: from='osd.6 [v2:172.21.15.38:6818/2122788957,v1:172.21.15.38:6819/2122788957]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-03-09T14:41:47.715 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:47 smithi038 ceph-mon[92422]: osdmap e43: 7 total, 6 up, 7 in 2023-03-09T14:41:47.715 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:47 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-03-09T14:41:47.715 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:47 smithi038 ceph-mon[92422]: from='osd.6 [v2:172.21.15.38:6818/2122788957,v1:172.21.15.38:6819/2122788957]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2023-03-09T14:41:47.715 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:47 smithi038 ceph-mon[92422]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2023-03-09T14:41:47.716 INFO:journalctl@ceph.osd.6.smithi038.stdout:Mar 09 14:41:47 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-6[116185]: 2023-03-09T14:41:47.355+0000 7f2a3ea2e700 -1 osd.6 0 waiting for initial osdmap 2023-03-09T14:41:47.716 INFO:journalctl@ceph.osd.6.smithi038.stdout:Mar 09 14:41:47 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-6[116185]: 2023-03-09T14:41:47.371+0000 7f2a3a022700 -1 osd.6 44 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-09T14:41:47.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:47 smithi122 ceph-mon[93869]: from='osd.6 [v2:172.21.15.38:6818/2122788957,v1:172.21.15.38:6819/2122788957]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-03-09T14:41:47.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:47 smithi122 ceph-mon[93869]: osdmap e43: 7 total, 6 up, 7 in 2023-03-09T14:41:47.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:47 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-03-09T14:41:47.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:47 smithi122 ceph-mon[93869]: from='osd.6 [v2:172.21.15.38:6818/2122788957,v1:172.21.15.38:6819/2122788957]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2023-03-09T14:41:47.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:47 smithi122 ceph-mon[93869]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2023-03-09T14:41:47.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:47 smithi181 ceph-mon[94189]: from='osd.6 [v2:172.21.15.38:6818/2122788957,v1:172.21.15.38:6819/2122788957]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-03-09T14:41:47.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:47 smithi181 ceph-mon[94189]: osdmap e43: 7 total, 6 up, 7 in 2023-03-09T14:41:47.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:47 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-03-09T14:41:47.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:47 smithi181 ceph-mon[94189]: from='osd.6 [v2:172.21.15.38:6818/2122788957,v1:172.21.15.38:6819/2122788957]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2023-03-09T14:41:47.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:47 smithi181 ceph-mon[94189]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2023-03-09T14:41:47.846 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph orch daemon add osd smithi122:vg_nvme/lv_2 2023-03-09T14:41:48.492 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.b/config 2023-03-09T14:41:48.704 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:48 smithi122 ceph-mon[93869]: from='osd.6 [v2:172.21.15.38:6818/2122788957,v1:172.21.15.38:6819/2122788957]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2023-03-09T14:41:48.704 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:48 smithi122 ceph-mon[93869]: osdmap e44: 7 total, 6 up, 7 in 2023-03-09T14:41:48.705 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:48 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-03-09T14:41:48.705 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:48 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-03-09T14:41:48.705 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:48 smithi122 ceph-mon[93869]: Detected new or changed devices on smithi038 2023-03-09T14:41:48.705 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:48 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:48.705 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:48 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:48.706 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:48 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:41:48.706 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:48 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:41:48.706 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:48 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:41:48.706 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:48 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi038 to 5724M 2023-03-09T14:41:48.706 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:48 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:48.706 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:48 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:41:48.706 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:48 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:41:48.706 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:48 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:48.707 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:48 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:41:48.707 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:48 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:41:48.707 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:48 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:41:48.707 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:48 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:48.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:48 smithi181 ceph-mon[94189]: from='osd.6 [v2:172.21.15.38:6818/2122788957,v1:172.21.15.38:6819/2122788957]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2023-03-09T14:41:48.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:48 smithi181 ceph-mon[94189]: osdmap e44: 7 total, 6 up, 7 in 2023-03-09T14:41:48.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:48 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-03-09T14:41:48.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:48 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-03-09T14:41:48.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:48 smithi181 ceph-mon[94189]: Detected new or changed devices on smithi038 2023-03-09T14:41:48.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:48 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:48.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:48 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:48.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:48 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:41:48.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:48 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:41:48.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:48 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:41:48.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:48 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi038 to 5724M 2023-03-09T14:41:48.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:48 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:48.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:48 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:41:48.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:48 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:41:48.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:48 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:48.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:48 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:41:48.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:48 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:41:48.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:48 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:41:48.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:48 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:48.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:48 smithi038 ceph-mon[92422]: from='osd.6 [v2:172.21.15.38:6818/2122788957,v1:172.21.15.38:6819/2122788957]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2023-03-09T14:41:48.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:48 smithi038 ceph-mon[92422]: osdmap e44: 7 total, 6 up, 7 in 2023-03-09T14:41:48.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:48 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-03-09T14:41:48.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:48 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-03-09T14:41:48.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:48 smithi038 ceph-mon[92422]: Detected new or changed devices on smithi038 2023-03-09T14:41:48.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:48 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:48.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:48 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:48.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:48 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:41:48.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:48 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:41:48.753 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:48 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:41:48.753 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:48 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi038 to 5724M 2023-03-09T14:41:48.753 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:48 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:48.753 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:48 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:41:48.753 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:48 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:41:48.753 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:48 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:48.753 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:48 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:41:48.754 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:48 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:41:48.754 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:48 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:41:48.754 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:48 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:41:49.690 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:41:49.686+0000 7f351c145700 1 -- 172.21.15.122:0/678264122 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35140fffd0 msgr2=0x7f35141003e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:41:49.691 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:41:49.686+0000 7f351c145700 1 --2- 172.21.15.122:0/678264122 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35140fffd0 0x7f35141003e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3504009990 tx=0x7f3504009ca0 comp rx=0 tx=0).stop 2023-03-09T14:41:49.691 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:41:49.686+0000 7f351c145700 1 -- 172.21.15.122:0/678264122 shutdown_connections 2023-03-09T14:41:49.691 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:41:49.686+0000 7f351c145700 1 --2- 172.21.15.122:0/678264122 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3514101b50 0x7f3514108410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:41:49.691 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:41:49.686+0000 7f351c145700 1 --2- 172.21.15.122:0/678264122 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35140fffd0 0x7f35141003e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:41:49.691 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:41:49.686+0000 7f351c145700 1 --2- 172.21.15.122:0/678264122 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35141011c0 0x7f3514101610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:41:49.691 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:41:49.686+0000 7f351c145700 1 -- 172.21.15.122:0/678264122 >> 172.21.15.122:0/678264122 conn(0x7f35140fb5f0 msgr2=0x7f35140fda20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:41:49.692 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:41:49.686+0000 7f351c145700 1 -- 172.21.15.122:0/678264122 shutdown_connections 2023-03-09T14:41:49.692 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:41:49.686+0000 7f351c145700 1 -- 172.21.15.122:0/678264122 wait complete. 2023-03-09T14:41:49.692 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:41:49.687+0000 7f351c145700 1 Processor -- start 2023-03-09T14:41:49.692 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:41:49.687+0000 7f351c145700 1 -- start start 2023-03-09T14:41:49.692 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:41:49.687+0000 7f351c145700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35140fffd0 0x7f3514195d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:41:49.692 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:41:49.687+0000 7f351c145700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35141011c0 0x7f3514196280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:41:49.692 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:41:49.687+0000 7f351c145700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3514101b50 0x7f351419d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:41:49.692 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:41:49.687+0000 7f351c145700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f351419d8c0 con 0x7f35140fffd0 2023-03-09T14:41:49.693 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:41:49.687+0000 7f351c145700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f351419da00 con 0x7f35141011c0 2023-03-09T14:41:49.693 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:41:49.687+0000 7f351c145700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f351419db90 con 0x7f3514101b50 2023-03-09T14:41:49.693 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:41:49.687+0000 7f35196e0700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35141011c0 0x7f3514196280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:41:49.693 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:41:49.687+0000 7f35196e0700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35141011c0 0x7f3514196280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:49360/0 (socket says 172.21.15.122:49360) 2023-03-09T14:41:49.693 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:41:49.687+0000 7f35196e0700 1 -- 172.21.15.122:0/3514772373 learned_addr learned my addr 172.21.15.122:0/3514772373 (peer_addr_for_me v2:172.21.15.122:0/0) 2023-03-09T14:41:49.693 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:41:49.687+0000 7f3519ee1700 1 --2- 172.21.15.122:0/3514772373 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35140fffd0 0x7f3514195d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:41:49.694 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:41:49.687+0000 7f35196e0700 1 -- 172.21.15.122:0/3514772373 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3514101b50 msgr2=0x7f351419d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:41:49.694 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:41:49.687+0000 7f351a6e2700 1 --2- 172.21.15.122:0/3514772373 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3514101b50 0x7f351419d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:41:49.694 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:41:49.688+0000 7f35196e0700 1 --2- 172.21.15.122:0/3514772373 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3514101b50 0x7f351419d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:41:49.694 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:41:49.688+0000 7f35196e0700 1 -- 172.21.15.122:0/3514772373 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35140fffd0 msgr2=0x7f3514195d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:41:49.694 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:41:49.688+0000 7f35196e0700 1 --2- 172.21.15.122:0/3514772373 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35140fffd0 0x7f3514195d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:41:49.694 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:41:49.688+0000 7f35196e0700 1 -- 172.21.15.122:0/3514772373 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35040095f0 con 0x7f35141011c0 2023-03-09T14:41:49.694 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:41:49.688+0000 7f3519ee1700 1 --2- 172.21.15.122:0/3514772373 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35140fffd0 0x7f3514195d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:41:49.694 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:41:49.688+0000 7f351a6e2700 1 --2- 172.21.15.122:0/3514772373 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3514101b50 0x7f351419d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-09T14:41:49.695 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:41:49.688+0000 7f35196e0700 1 --2- 172.21.15.122:0/3514772373 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35141011c0 0x7f3514196280 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f351000d950 tx=0x7f351000dd10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:41:49.695 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:41:49.688+0000 7f350affd700 1 -- 172.21.15.122:0/3514772373 <== mon.2 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3510009b40 con 0x7f35141011c0 2023-03-09T14:41:49.695 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:41:49.688+0000 7f351c145700 1 -- 172.21.15.122:0/3514772373 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f351410b930 con 0x7f35141011c0 2023-03-09T14:41:49.695 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:41:49.688+0000 7f350affd700 1 -- 172.21.15.122:0/3514772373 <== mon.2 v2:172.21.15.122:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3510009ca0 con 0x7f35141011c0 2023-03-09T14:41:49.695 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:41:49.688+0000 7f351c145700 1 -- 172.21.15.122:0/3514772373 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f351410b7a0 con 0x7f35141011c0 2023-03-09T14:41:49.696 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:41:49.688+0000 7f350affd700 1 -- 172.21.15.122:0/3514772373 <== mon.2 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f351000f620 con 0x7f35141011c0 2023-03-09T14:41:49.697 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:41:49.689+0000 7f351c145700 1 -- 172.21.15.122:0/3514772373 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f351419e290 con 0x7f35141011c0 2023-03-09T14:41:49.698 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:41:49.690+0000 7f350affd700 1 -- 172.21.15.122:0/3514772373 <== mon.2 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 16) v1 ==== 133198+0+0 (secure 0 0 0) 0x7f3510018030 con 0x7f35141011c0 2023-03-09T14:41:49.698 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:41:49.690+0000 7f350affd700 1 --2- 172.21.15.122:0/3514772373 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f350009f440 0x7f35000a18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:41:49.699 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:41:49.691+0000 7f350affd700 1 -- 172.21.15.122:0/3514772373 <== mon.2 v2:172.21.15.122:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 4965+0+0 (secure 0 0 0) 0x7f35100c92d0 con 0x7f35141011c0 2023-03-09T14:41:49.699 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:41:49.691+0000 7f3519ee1700 1 --2- 172.21.15.122:0/3514772373 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f350009f440 0x7f35000a18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:41:49.699 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:41:49.691+0000 7f3519ee1700 1 --2- 172.21.15.122:0/3514772373 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f350009f440 0x7f35000a18f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3504006010 tx=0x7f350400b540 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:41:49.699 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:41:49.695+0000 7f350affd700 1 -- 172.21.15.122:0/3514772373 <== mon.2 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f35100943f0 con 0x7f35141011c0 2023-03-09T14:41:49.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:49 smithi122 ceph-mon[93869]: purged_snaps scrub starts 2023-03-09T14:41:49.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:49 smithi122 ceph-mon[93869]: purged_snaps scrub ok 2023-03-09T14:41:49.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:49 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-03-09T14:41:49.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:49 smithi122 ceph-mon[93869]: osd.6 [v2:172.21.15.38:6818/2122788957,v1:172.21.15.38:6819/2122788957] boot 2023-03-09T14:41:49.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:49 smithi122 ceph-mon[93869]: osdmap e45: 7 total, 7 up, 7 in 2023-03-09T14:41:49.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:49 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-03-09T14:41:49.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:49 smithi122 ceph-mon[93869]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2023-03-09T14:41:49.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:49 smithi181 ceph-mon[94189]: purged_snaps scrub starts 2023-03-09T14:41:49.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:49 smithi181 ceph-mon[94189]: purged_snaps scrub ok 2023-03-09T14:41:49.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:49 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-03-09T14:41:49.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:49 smithi181 ceph-mon[94189]: osd.6 [v2:172.21.15.38:6818/2122788957,v1:172.21.15.38:6819/2122788957] boot 2023-03-09T14:41:49.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:49 smithi181 ceph-mon[94189]: osdmap e45: 7 total, 7 up, 7 in 2023-03-09T14:41:49.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:49 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-03-09T14:41:49.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:49 smithi181 ceph-mon[94189]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2023-03-09T14:41:49.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:49 smithi038 ceph-mon[92422]: purged_snaps scrub starts 2023-03-09T14:41:49.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:49 smithi038 ceph-mon[92422]: purged_snaps scrub ok 2023-03-09T14:41:49.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:49 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-03-09T14:41:49.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:49 smithi038 ceph-mon[92422]: osd.6 [v2:172.21.15.38:6818/2122788957,v1:172.21.15.38:6819/2122788957] boot 2023-03-09T14:41:49.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:49 smithi038 ceph-mon[92422]: osdmap e45: 7 total, 7 up, 7 in 2023-03-09T14:41:49.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:49 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-03-09T14:41:49.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:49 smithi038 ceph-mon[92422]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 36 MiB used, 536 GiB / 536 GiB avail 2023-03-09T14:41:49.893 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:41:49.890+0000 7f351c145700 1 -- 172.21.15.122:0/3514772373 --> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f3514060bc0 con 0x7f350009f440 2023-03-09T14:41:50.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:50 smithi122 ceph-mon[93869]: osdmap e46: 7 total, 7 up, 7 in 2023-03-09T14:41:50.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:50 smithi122 ceph-mon[93869]: from='client.24275 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:41:50.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:50 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-03-09T14:41:50.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:50 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-03-09T14:41:50.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:50 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:41:50.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:50 smithi181 ceph-mon[94189]: osdmap e46: 7 total, 7 up, 7 in 2023-03-09T14:41:50.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:50 smithi181 ceph-mon[94189]: from='client.24275 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:41:50.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:50 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-03-09T14:41:50.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:50 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-03-09T14:41:50.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:50 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:41:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:50 smithi038 ceph-mon[92422]: osdmap e46: 7 total, 7 up, 7 in 2023-03-09T14:41:50.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:50 smithi038 ceph-mon[92422]: from='client.24275 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:41:50.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:50 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-03-09T14:41:50.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:50 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-03-09T14:41:50.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:50 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:41:51.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:51 smithi122 ceph-mon[93869]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-03-09T14:41:51.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:51 smithi181 ceph-mon[94189]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-03-09T14:41:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:51 smithi038 ceph-mon[92422]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-03-09T14:41:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:53 smithi038 ceph-mon[92422]: from='client.? 172.21.15.122:0/4280135360' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "231fecdd-939f-4c7b-83a1-c4a1cffd7139"}]: dispatch 2023-03-09T14:41:53.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:53 smithi038 ceph-mon[92422]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "231fecdd-939f-4c7b-83a1-c4a1cffd7139"}]: dispatch 2023-03-09T14:41:53.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:53 smithi038 ceph-mon[92422]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "231fecdd-939f-4c7b-83a1-c4a1cffd7139"}]': finished 2023-03-09T14:41:53.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:53 smithi038 ceph-mon[92422]: osdmap e47: 8 total, 7 up, 8 in 2023-03-09T14:41:53.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:53 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-03-09T14:41:53.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:53 smithi038 ceph-mon[92422]: from='client.? 172.21.15.122:0/1105990076' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-03-09T14:41:53.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:53 smithi122 ceph-mon[93869]: from='client.? 172.21.15.122:0/4280135360' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "231fecdd-939f-4c7b-83a1-c4a1cffd7139"}]: dispatch 2023-03-09T14:41:53.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:53 smithi122 ceph-mon[93869]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "231fecdd-939f-4c7b-83a1-c4a1cffd7139"}]: dispatch 2023-03-09T14:41:53.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:53 smithi122 ceph-mon[93869]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "231fecdd-939f-4c7b-83a1-c4a1cffd7139"}]': finished 2023-03-09T14:41:53.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:53 smithi122 ceph-mon[93869]: osdmap e47: 8 total, 7 up, 8 in 2023-03-09T14:41:53.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:53 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-03-09T14:41:53.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:53 smithi122 ceph-mon[93869]: from='client.? 172.21.15.122:0/1105990076' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-03-09T14:41:53.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:53 smithi181 ceph-mon[94189]: from='client.? 172.21.15.122:0/4280135360' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "231fecdd-939f-4c7b-83a1-c4a1cffd7139"}]: dispatch 2023-03-09T14:41:53.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:53 smithi181 ceph-mon[94189]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "231fecdd-939f-4c7b-83a1-c4a1cffd7139"}]: dispatch 2023-03-09T14:41:53.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:53 smithi181 ceph-mon[94189]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "231fecdd-939f-4c7b-83a1-c4a1cffd7139"}]': finished 2023-03-09T14:41:53.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:53 smithi181 ceph-mon[94189]: osdmap e47: 8 total, 7 up, 8 in 2023-03-09T14:41:53.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:53 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-03-09T14:41:53.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:53 smithi181 ceph-mon[94189]: from='client.? 172.21.15.122:0/1105990076' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-03-09T14:41:54.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:54 smithi122 ceph-mon[93869]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-03-09T14:41:54.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:54 smithi181 ceph-mon[94189]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-03-09T14:41:54.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:54 smithi038 ceph-mon[92422]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-03-09T14:41:56.413 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:56 smithi122 ceph-mon[93869]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-03-09T14:41:56.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:56 smithi181 ceph-mon[94189]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-03-09T14:41:56.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:56 smithi038 ceph-mon[92422]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-03-09T14:41:58.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:58 smithi122 ceph-mon[93869]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-03-09T14:41:58.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:58 smithi181 ceph-mon[94189]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-03-09T14:41:58.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:58 smithi038 ceph-mon[92422]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2023-03-09T14:41:59.644 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:59 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-03-09T14:41:59.644 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:41:59 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:41:59.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:59 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-03-09T14:41:59.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:41:59 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:41:59.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:59 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-03-09T14:41:59.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:41:59 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:42:00.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:00 smithi122 ceph-mon[93869]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2023-03-09T14:42:00.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:00 smithi122 ceph-mon[93869]: Deploying daemon osd.7 on smithi122 2023-03-09T14:42:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:00 smithi181 ceph-mon[94189]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2023-03-09T14:42:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:00 smithi181 ceph-mon[94189]: Deploying daemon osd.7 on smithi122 2023-03-09T14:42:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:00 smithi038 ceph-mon[92422]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2023-03-09T14:42:00.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:00 smithi038 ceph-mon[92422]: Deploying daemon osd.7 on smithi122 2023-03-09T14:42:02.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:02 smithi122 ceph-mon[93869]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2023-03-09T14:42:02.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:02 smithi181 ceph-mon[94189]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2023-03-09T14:42:02.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:02 smithi038 ceph-mon[92422]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2023-03-09T14:42:04.580 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:04 smithi122 ceph-mon[93869]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2023-03-09T14:42:04.580 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:04 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:04.580 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:04 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:04.580 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:04 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:42:04.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:04 smithi181 ceph-mon[94189]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2023-03-09T14:42:04.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:04 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:04.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:04 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:04.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:04 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:42:04.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:04 smithi038 ceph-mon[92422]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2023-03-09T14:42:04.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:04 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:04.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:04 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:04.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:04 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:42:06.449 INFO:teuthology.orchestra.run.smithi122.stdout:Created osd(s) 7 on host 'smithi122' 2023-03-09T14:42:06.450 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:42:06.445+0000 7f350affd700 1 -- 172.21.15.122:0/3514772373 <== mgr.14150 v2:172.21.15.38:6800/3328887328 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f3514060bc0 con 0x7f350009f440 2023-03-09T14:42:06.451 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:42:06.447+0000 7f351c145700 1 -- 172.21.15.122:0/3514772373 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f350009f440 msgr2=0x7f35000a18f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:42:06.451 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:42:06.448+0000 7f351c145700 1 --2- 172.21.15.122:0/3514772373 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f350009f440 0x7f35000a18f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3504006010 tx=0x7f350400b540 comp rx=0 tx=0).stop 2023-03-09T14:42:06.451 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:42:06.448+0000 7f351c145700 1 -- 172.21.15.122:0/3514772373 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35141011c0 msgr2=0x7f3514196280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:42:06.451 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:42:06.448+0000 7f351c145700 1 --2- 172.21.15.122:0/3514772373 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35141011c0 0x7f3514196280 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f351000d950 tx=0x7f351000dd10 comp rx=0 tx=0).stop 2023-03-09T14:42:06.451 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:42:06.448+0000 7f351c145700 1 -- 172.21.15.122:0/3514772373 shutdown_connections 2023-03-09T14:42:06.452 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:42:06.448+0000 7f351c145700 1 --2- 172.21.15.122:0/3514772373 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f35140fffd0 0x7f3514195d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:42:06.452 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:42:06.448+0000 7f351c145700 1 --2- 172.21.15.122:0/3514772373 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f350009f440 0x7f35000a18f0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:42:06.452 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:42:06.448+0000 7f351c145700 1 --2- 172.21.15.122:0/3514772373 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3514101b50 0x7f351419d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:42:06.452 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:42:06.448+0000 7f351c145700 1 --2- 172.21.15.122:0/3514772373 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f35141011c0 0x7f3514196280 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:42:06.452 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:42:06.448+0000 7f351c145700 1 -- 172.21.15.122:0/3514772373 >> 172.21.15.122:0/3514772373 conn(0x7f35140fb5f0 msgr2=0x7f35140fd840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:42:06.452 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:42:06.448+0000 7f351c145700 1 -- 172.21.15.122:0/3514772373 shutdown_connections 2023-03-09T14:42:06.452 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:42:06.448+0000 7f351c145700 1 -- 172.21.15.122:0/3514772373 wait complete. 2023-03-09T14:42:06.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:06 smithi122 ceph-mon[93869]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2023-03-09T14:42:06.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:06 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:06.596 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:06 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:06.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:06 smithi181 ceph-mon[94189]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2023-03-09T14:42:06.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:06 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:06.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:06 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:06 smithi038 ceph-mon[92422]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2023-03-09T14:42:06.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:06 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:06.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:06 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:06.965 DEBUG:teuthology.orchestra.run.smithi122:osd.7> sudo journalctl -f -n 0 -u ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.7.service 2023-03-09T14:42:06.968 INFO:tasks.cephadm:Deploying osd.8 on smithi181 with /dev/vg_nvme/lv_2... 2023-03-09T14:42:06.968 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-03-09T14:42:06.996 INFO:journalctl@ceph.osd.7.smithi122.stdout:-- Logs begin at Thu 2023-03-09 14:25:09 UTC. -- 2023-03-09T14:42:07.627 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.c/config 2023-03-09T14:42:07.728 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:42:07 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 2023-03-09T14:42:07.396+0000 7f4f9296c640 -1 osd.7 0 log_to_monitors true 2023-03-09T14:42:07.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:07 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:07.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:07 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:07.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:07 smithi122 ceph-mon[93869]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2023-03-09T14:42:07.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:07 smithi122 ceph-mon[93869]: from='osd.7 [v2:172.21.15.122:6816/656874139,v1:172.21.15.122:6817/656874139]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-03-09T14:42:07.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:07 smithi122 ceph-mon[93869]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-03-09T14:42:07.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:07 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:07.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:07 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:07.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:07 smithi181 ceph-mon[94189]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2023-03-09T14:42:07.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:07 smithi181 ceph-mon[94189]: from='osd.7 [v2:172.21.15.122:6816/656874139,v1:172.21.15.122:6817/656874139]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-03-09T14:42:07.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:07 smithi181 ceph-mon[94189]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-03-09T14:42:07.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:07 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:07.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:07 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:07.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:07 smithi038 ceph-mon[92422]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2023-03-09T14:42:07.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:07 smithi038 ceph-mon[92422]: from='osd.7 [v2:172.21.15.122:6816/656874139,v1:172.21.15.122:6817/656874139]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-03-09T14:42:07.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:07 smithi038 ceph-mon[92422]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-03-09T14:42:09.160 INFO:teuthology.orchestra.run.smithi181.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-03-09T14:42:09.161 INFO:teuthology.orchestra.run.smithi181.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-03-09T14:42:09.161 INFO:teuthology.orchestra.run.smithi181.stderr: stderr: 10+0 records in 2023-03-09T14:42:09.161 INFO:teuthology.orchestra.run.smithi181.stderr:10+0 records out 2023-03-09T14:42:09.161 INFO:teuthology.orchestra.run.smithi181.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0113269 s, 926 MB/s 2023-03-09T14:42:09.161 INFO:teuthology.orchestra.run.smithi181.stderr: stderr: 2023-03-09T14:42:09.161 INFO:teuthology.orchestra.run.smithi181.stderr:--> Zapping successful for: 2023-03-09T14:42:09.463 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:09 smithi181 ceph-mon[94189]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-03-09T14:42:09.463 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:09 smithi181 ceph-mon[94189]: osdmap e48: 8 total, 7 up, 8 in 2023-03-09T14:42:09.463 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:09 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-03-09T14:42:09.463 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:09 smithi181 ceph-mon[94189]: from='osd.7 [v2:172.21.15.122:6816/656874139,v1:172.21.15.122:6817/656874139]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-03-09T14:42:09.463 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:09 smithi181 ceph-mon[94189]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-03-09T14:42:09.477 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:42:09 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 2023-03-09T14:42:09.260+0000 7f4f868f1700 -1 osd.7 0 waiting for initial osdmap 2023-03-09T14:42:09.478 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:42:09 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 2023-03-09T14:42:09.273+0000 7f4f7fee1700 -1 osd.7 49 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-09T14:42:09.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:09 smithi122 ceph-mon[93869]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-03-09T14:42:09.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:09 smithi122 ceph-mon[93869]: osdmap e48: 8 total, 7 up, 8 in 2023-03-09T14:42:09.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:09 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-03-09T14:42:09.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:09 smithi122 ceph-mon[93869]: from='osd.7 [v2:172.21.15.122:6816/656874139,v1:172.21.15.122:6817/656874139]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-03-09T14:42:09.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:09 smithi122 ceph-mon[93869]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-03-09T14:42:09.567 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph orch daemon add osd smithi181:vg_nvme/lv_2 2023-03-09T14:42:09.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:09 smithi038 ceph-mon[92422]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-03-09T14:42:09.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:09 smithi038 ceph-mon[92422]: osdmap e48: 8 total, 7 up, 8 in 2023-03-09T14:42:09.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:09 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-03-09T14:42:09.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:09 smithi038 ceph-mon[92422]: from='osd.7 [v2:172.21.15.122:6816/656874139,v1:172.21.15.122:6817/656874139]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-03-09T14:42:09.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:09 smithi038 ceph-mon[92422]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-03-09T14:42:10.209 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.c/config 2023-03-09T14:42:10.595 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:10 smithi181 ceph-mon[94189]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2023-03-09T14:42:10.596 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:10 smithi181 ceph-mon[94189]: Detected new or changed devices on smithi122 2023-03-09T14:42:10.596 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:10 smithi181 ceph-mon[94189]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2023-03-09T14:42:10.596 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:10 smithi181 ceph-mon[94189]: osdmap e49: 8 total, 7 up, 8 in 2023-03-09T14:42:10.596 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:10 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-03-09T14:42:10.596 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:10 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-03-09T14:42:10.596 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:10 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:10.596 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:10 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:10.597 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:10 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:42:10.597 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:10 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:42:10.597 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:10 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:42:10.597 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:10 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 5722M 2023-03-09T14:42:10.597 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:10 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:10.597 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:10 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:42:10.597 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:10 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:42:10.598 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:10 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:10.598 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:10 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:42:10.598 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:10 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:42:10.598 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:10 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:42:10.598 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:10 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:10.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:10 smithi122 ceph-mon[93869]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2023-03-09T14:42:10.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:10 smithi122 ceph-mon[93869]: Detected new or changed devices on smithi122 2023-03-09T14:42:10.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:10 smithi122 ceph-mon[93869]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2023-03-09T14:42:10.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:10 smithi122 ceph-mon[93869]: osdmap e49: 8 total, 7 up, 8 in 2023-03-09T14:42:10.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:10 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-03-09T14:42:10.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:10 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-03-09T14:42:10.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:10 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:10.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:10 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:10.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:10 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:42:10.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:10 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:42:10.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:10 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:42:10.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:10 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 5722M 2023-03-09T14:42:10.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:10 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:10.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:10 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:42:10.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:10 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:42:10.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:10 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:10.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:10 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:42:10.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:10 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:42:10.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:10 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:42:10.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:10 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:10 smithi038 ceph-mon[92422]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2023-03-09T14:42:10.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:10 smithi038 ceph-mon[92422]: Detected new or changed devices on smithi122 2023-03-09T14:42:10.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:10 smithi038 ceph-mon[92422]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2023-03-09T14:42:10.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:10 smithi038 ceph-mon[92422]: osdmap e49: 8 total, 7 up, 8 in 2023-03-09T14:42:10.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:10 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-03-09T14:42:10.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:10 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-03-09T14:42:10.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:10 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:10.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:10 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:10.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:10 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:42:10.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:10 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:42:10.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:10 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:42:10.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:10 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 5722M 2023-03-09T14:42:10.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:10 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:10.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:10 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:42:10.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:10 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:42:10.753 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:10 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:10.753 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:10 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:42:10.753 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:10 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:42:10.753 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:10 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:42:10.753 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:10 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:11.399 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:42:11.396+0000 7f8d2a3d5700 1 -- 172.21.15.181:0/2075832139 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8d240fe620 msgr2=0x7f8d240fea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:42:11.399 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:42:11.396+0000 7f8d2a3d5700 1 --2- 172.21.15.181:0/2075832139 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8d240fe620 0x7f8d240fea30 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f8d18009990 tx=0x7f8d18009ca0 comp rx=0 tx=0).stop 2023-03-09T14:42:11.399 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:42:11.396+0000 7f8d2a3d5700 1 -- 172.21.15.181:0/2075832139 shutdown_connections 2023-03-09T14:42:11.399 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:42:11.396+0000 7f8d2a3d5700 1 --2- 172.21.15.181:0/2075832139 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8d241001a0 0x7f8d24106a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:42:11.400 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:42:11.396+0000 7f8d2a3d5700 1 --2- 172.21.15.181:0/2075832139 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8d240fe620 0x7f8d240fea30 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:42:11.400 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:42:11.396+0000 7f8d2a3d5700 1 --2- 172.21.15.181:0/2075832139 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8d240ff810 0x7f8d240ffc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:42:11.400 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:42:11.396+0000 7f8d2a3d5700 1 -- 172.21.15.181:0/2075832139 >> 172.21.15.181:0/2075832139 conn(0x7f8d240f9c00 msgr2=0x7f8d240fc070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:42:11.400 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:42:11.396+0000 7f8d2a3d5700 1 -- 172.21.15.181:0/2075832139 shutdown_connections 2023-03-09T14:42:11.400 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:42:11.396+0000 7f8d2a3d5700 1 -- 172.21.15.181:0/2075832139 wait complete. 2023-03-09T14:42:11.400 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:42:11.397+0000 7f8d2a3d5700 1 Processor -- start 2023-03-09T14:42:11.400 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:42:11.397+0000 7f8d2a3d5700 1 -- start start 2023-03-09T14:42:11.402 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:42:11.397+0000 7f8d2a3d5700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8d240fe620 0x7f8d24107f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:42:11.403 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:42:11.397+0000 7f8d2a3d5700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8d240ff810 0x7f8d2419aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:42:11.403 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:42:11.397+0000 7f8d2a3d5700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8d241001a0 0x7f8d2419d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:42:11.403 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:42:11.397+0000 7f8d2a3d5700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2419d8c0 con 0x7f8d240ff810 2023-03-09T14:42:11.403 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:42:11.397+0000 7f8d2a3d5700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2419da30 con 0x7f8d241001a0 2023-03-09T14:42:11.403 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:42:11.397+0000 7f8d2a3d5700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2419dbf0 con 0x7f8d240fe620 2023-03-09T14:42:11.403 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:42:11.397+0000 7f8d23fff700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8d240fe620 0x7f8d24107f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:42:11.404 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:42:11.398+0000 7f8d23fff700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8d240fe620 0x7f8d24107f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:50696/0 (socket says 172.21.15.181:50696) 2023-03-09T14:42:11.404 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:42:11.398+0000 7f8d23fff700 1 -- 172.21.15.181:0/2291468787 learned_addr learned my addr 172.21.15.181:0/2291468787 (peer_addr_for_me v2:172.21.15.181:0/0) 2023-03-09T14:42:11.404 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:42:11.398+0000 7f8d28972700 1 --2- 172.21.15.181:0/2291468787 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8d241001a0 0x7f8d2419d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:42:11.404 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:42:11.398+0000 7f8d237fe700 1 --2- 172.21.15.181:0/2291468787 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8d240ff810 0x7f8d2419aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:42:11.404 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:42:11.398+0000 7f8d23fff700 1 -- 172.21.15.181:0/2291468787 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8d241001a0 msgr2=0x7f8d2419d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:42:11.404 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:42:11.398+0000 7f8d23fff700 1 --2- 172.21.15.181:0/2291468787 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8d241001a0 0x7f8d2419d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:42:11.404 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:42:11.398+0000 7f8d23fff700 1 -- 172.21.15.181:0/2291468787 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8d240ff810 msgr2=0x7f8d2419aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:42:11.405 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:42:11.398+0000 7f8d23fff700 1 --2- 172.21.15.181:0/2291468787 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8d240ff810 0x7f8d2419aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:42:11.405 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:42:11.398+0000 7f8d23fff700 1 -- 172.21.15.181:0/2291468787 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d180095f0 con 0x7f8d240fe620 2023-03-09T14:42:11.405 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:42:11.398+0000 7f8d237fe700 1 --2- 172.21.15.181:0/2291468787 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8d240ff810 0x7f8d2419aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-09T14:42:11.405 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:42:11.398+0000 7f8d23fff700 1 --2- 172.21.15.181:0/2291468787 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8d240fe620 0x7f8d24107f20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8d18000c00 tx=0x7f8d1800f700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:42:11.405 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:42:11.398+0000 7f8d217fa700 1 -- 172.21.15.181:0/2291468787 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d1801d070 con 0x7f8d240fe620 2023-03-09T14:42:11.405 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:42:11.398+0000 7f8d217fa700 1 -- 172.21.15.181:0/2291468787 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8d1800fca0 con 0x7f8d240fe620 2023-03-09T14:42:11.405 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:42:11.398+0000 7f8d2a3d5700 1 -- 172.21.15.181:0/2291468787 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d2410a800 con 0x7f8d240fe620 2023-03-09T14:42:11.405 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:42:11.398+0000 7f8d217fa700 1 -- 172.21.15.181:0/2291468787 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d180176e0 con 0x7f8d240fe620 2023-03-09T14:42:11.406 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:42:11.398+0000 7f8d2a3d5700 1 -- 172.21.15.181:0/2291468787 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d2410a670 con 0x7f8d240fe620 2023-03-09T14:42:11.406 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:42:11.399+0000 7f8d2a3d5700 1 -- 172.21.15.181:0/2291468787 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d24066b10 con 0x7f8d240fe620 2023-03-09T14:42:11.406 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:42:11.400+0000 7f8d217fa700 1 -- 172.21.15.181:0/2291468787 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 16) v1 ==== 133198+0+0 (secure 0 0 0) 0x7f8d18029030 con 0x7f8d240fe620 2023-03-09T14:42:11.406 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:42:11.401+0000 7f8d217fa700 1 --2- 172.21.15.181:0/2291468787 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f8d0c09f3c0 0x7f8d0c0a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:42:11.406 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:42:11.401+0000 7f8d217fa700 1 -- 172.21.15.181:0/2291468787 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5426+0+0 (secure 0 0 0) 0x7f8d180ca5e0 con 0x7f8d240fe620 2023-03-09T14:42:11.406 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:42:11.401+0000 7f8d237fe700 1 --2- 172.21.15.181:0/2291468787 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f8d0c09f3c0 0x7f8d0c0a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:42:11.407 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:42:11.402+0000 7f8d237fe700 1 --2- 172.21.15.181:0/2291468787 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f8d0c09f3c0 0x7f8d0c0a1870 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8d14005fd0 tx=0x7f8d14005dc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:42:11.407 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:42:11.405+0000 7f8d217fa700 1 -- 172.21.15.181:0/2291468787 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f8d1805c5b0 con 0x7f8d240fe620 2023-03-09T14:42:11.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:11 smithi181 ceph-mon[94189]: purged_snaps scrub starts 2023-03-09T14:42:11.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:11 smithi181 ceph-mon[94189]: purged_snaps scrub ok 2023-03-09T14:42:11.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:11 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-03-09T14:42:11.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:11 smithi181 ceph-mon[94189]: osd.7 [v2:172.21.15.122:6816/656874139,v1:172.21.15.122:6817/656874139] boot 2023-03-09T14:42:11.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:11 smithi181 ceph-mon[94189]: osdmap e50: 8 total, 8 up, 8 in 2023-03-09T14:42:11.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:11 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-03-09T14:42:11.601 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:42:11.599+0000 7f8d2a3d5700 1 -- 172.21.15.181:0/2291468787 --> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f8d24060bc0 con 0x7f8d0c09f3c0 2023-03-09T14:42:11.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:11 smithi122 ceph-mon[93869]: purged_snaps scrub starts 2023-03-09T14:42:11.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:11 smithi122 ceph-mon[93869]: purged_snaps scrub ok 2023-03-09T14:42:11.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:11 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-03-09T14:42:11.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:11 smithi122 ceph-mon[93869]: osd.7 [v2:172.21.15.122:6816/656874139,v1:172.21.15.122:6817/656874139] boot 2023-03-09T14:42:11.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:11 smithi122 ceph-mon[93869]: osdmap e50: 8 total, 8 up, 8 in 2023-03-09T14:42:11.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:11 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-03-09T14:42:11.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:11 smithi038 ceph-mon[92422]: purged_snaps scrub starts 2023-03-09T14:42:11.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:11 smithi038 ceph-mon[92422]: purged_snaps scrub ok 2023-03-09T14:42:11.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:11 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-03-09T14:42:11.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:11 smithi038 ceph-mon[92422]: osd.7 [v2:172.21.15.122:6816/656874139,v1:172.21.15.122:6817/656874139] boot 2023-03-09T14:42:11.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:11 smithi038 ceph-mon[92422]: osdmap e50: 8 total, 8 up, 8 in 2023-03-09T14:42:11.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:11 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-03-09T14:42:12.553 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:12 smithi181 ceph-mon[94189]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2023-03-09T14:42:12.553 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:12 smithi181 ceph-mon[94189]: osdmap e51: 8 total, 8 up, 8 in 2023-03-09T14:42:12.553 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:12 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-03-09T14:42:12.553 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:12 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-03-09T14:42:12.554 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:12 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:42:12.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:12 smithi122 ceph-mon[93869]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2023-03-09T14:42:12.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:12 smithi122 ceph-mon[93869]: osdmap e51: 8 total, 8 up, 8 in 2023-03-09T14:42:12.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:12 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-03-09T14:42:12.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:12 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-03-09T14:42:12.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:12 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:42:12.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:12 smithi038 ceph-mon[92422]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 42 MiB used, 626 GiB / 626 GiB avail 2023-03-09T14:42:12.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:12 smithi038 ceph-mon[92422]: osdmap e51: 8 total, 8 up, 8 in 2023-03-09T14:42:12.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:12 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-03-09T14:42:12.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:12 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-03-09T14:42:12.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:12 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:42:13.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:13 smithi122 ceph-mon[93869]: from='client.24289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:42:13.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:13 smithi122 ceph-mon[93869]: osdmap e52: 8 total, 8 up, 8 in 2023-03-09T14:42:13.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:13 smithi181 ceph-mon[94189]: from='client.24289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:42:13.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:13 smithi181 ceph-mon[94189]: osdmap e52: 8 total, 8 up, 8 in 2023-03-09T14:42:13.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:13 smithi038 ceph-mon[92422]: from='client.24289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:42:13.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:13 smithi038 ceph-mon[92422]: osdmap e52: 8 total, 8 up, 8 in 2023-03-09T14:42:14.701 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:14 smithi038 ceph-mon[92422]: pgmap v161: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-03-09T14:42:14.701 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:14 smithi038 ceph-mon[92422]: osdmap e53: 8 total, 8 up, 8 in 2023-03-09T14:42:14.702 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:14 smithi038 ceph-mon[92422]: from='client.? 172.21.15.181:0/2025723951' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "297e6d6d-85fd-43ce-9028-5e9dbe55c36e"}]: dispatch 2023-03-09T14:42:14.702 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:14 smithi038 ceph-mon[92422]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "297e6d6d-85fd-43ce-9028-5e9dbe55c36e"}]: dispatch 2023-03-09T14:42:14.702 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:14 smithi038 ceph-mon[92422]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "297e6d6d-85fd-43ce-9028-5e9dbe55c36e"}]': finished 2023-03-09T14:42:14.702 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:14 smithi038 ceph-mon[92422]: osdmap e54: 9 total, 8 up, 9 in 2023-03-09T14:42:14.702 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:14 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-03-09T14:42:14.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:14 smithi122 ceph-mon[93869]: pgmap v161: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-03-09T14:42:14.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:14 smithi122 ceph-mon[93869]: osdmap e53: 8 total, 8 up, 8 in 2023-03-09T14:42:14.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:14 smithi122 ceph-mon[93869]: from='client.? 172.21.15.181:0/2025723951' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "297e6d6d-85fd-43ce-9028-5e9dbe55c36e"}]: dispatch 2023-03-09T14:42:14.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:14 smithi122 ceph-mon[93869]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "297e6d6d-85fd-43ce-9028-5e9dbe55c36e"}]: dispatch 2023-03-09T14:42:14.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:14 smithi122 ceph-mon[93869]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "297e6d6d-85fd-43ce-9028-5e9dbe55c36e"}]': finished 2023-03-09T14:42:14.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:14 smithi122 ceph-mon[93869]: osdmap e54: 9 total, 8 up, 9 in 2023-03-09T14:42:14.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:14 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-03-09T14:42:14.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:14 smithi181 ceph-mon[94189]: pgmap v161: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-03-09T14:42:14.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:14 smithi181 ceph-mon[94189]: osdmap e53: 8 total, 8 up, 8 in 2023-03-09T14:42:14.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:14 smithi181 ceph-mon[94189]: from='client.? 172.21.15.181:0/2025723951' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "297e6d6d-85fd-43ce-9028-5e9dbe55c36e"}]: dispatch 2023-03-09T14:42:14.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:14 smithi181 ceph-mon[94189]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "297e6d6d-85fd-43ce-9028-5e9dbe55c36e"}]: dispatch 2023-03-09T14:42:14.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:14 smithi181 ceph-mon[94189]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "297e6d6d-85fd-43ce-9028-5e9dbe55c36e"}]': finished 2023-03-09T14:42:14.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:14 smithi181 ceph-mon[94189]: osdmap e54: 9 total, 8 up, 9 in 2023-03-09T14:42:14.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:14 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-03-09T14:42:15.001 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:42:14 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:42:14.698+0000 7f3b39495700 -1 calc_pg_upmaps abort due to max <= 0 2023-03-09T14:42:15.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:15 smithi122 ceph-mon[93869]: from='client.? 172.21.15.181:0/3827406201' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-03-09T14:42:15.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:15 smithi181 ceph-mon[94189]: from='client.? 172.21.15.181:0/3827406201' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-03-09T14:42:15.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:15 smithi038 ceph-mon[92422]: from='client.? 172.21.15.181:0/3827406201' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-03-09T14:42:16.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:16 smithi122 ceph-mon[93869]: pgmap v164: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-03-09T14:42:16.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:16 smithi181 ceph-mon[94189]: pgmap v164: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-03-09T14:42:16.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:16 smithi038 ceph-mon[92422]: pgmap v164: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2023-03-09T14:42:18.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:18 smithi122 ceph-mon[93869]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-03-09T14:42:18.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:18 smithi181 ceph-mon[94189]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-03-09T14:42:18.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:18 smithi038 ceph-mon[92422]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-03-09T14:42:19.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:19 smithi181 ceph-mon[94189]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-03-09T14:42:19.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:19 smithi122 ceph-mon[93869]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-03-09T14:42:19.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:19 smithi038 ceph-mon[92422]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-03-09T14:42:20.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:20 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-03-09T14:42:20.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:20 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:42:20.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:20 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-03-09T14:42:20.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:20 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:42:21.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:20 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-03-09T14:42:21.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:20 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:42:21.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:21 smithi181 ceph-mon[94189]: Deploying daemon osd.8 on smithi181 2023-03-09T14:42:21.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:21 smithi181 ceph-mon[94189]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-03-09T14:42:21.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:21 smithi122 ceph-mon[93869]: Deploying daemon osd.8 on smithi181 2023-03-09T14:42:21.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:21 smithi122 ceph-mon[93869]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-03-09T14:42:22.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:21 smithi038 ceph-mon[92422]: Deploying daemon osd.8 on smithi181 2023-03-09T14:42:22.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:21 smithi038 ceph-mon[92422]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-03-09T14:42:24.612 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:24 smithi181 ceph-mon[94189]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-03-09T14:42:24.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:24 smithi122 ceph-mon[93869]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-03-09T14:42:24.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:24 smithi038 ceph-mon[92422]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-03-09T14:42:26.224 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:25 smithi181 ceph-mon[94189]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-03-09T14:42:26.224 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:25 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:26.224 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:25 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:26.224 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:25 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:42:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:25 smithi122 ceph-mon[93869]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-03-09T14:42:26.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:25 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:26.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:25 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:26.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:25 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:42:26.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:25 smithi038 ceph-mon[92422]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-03-09T14:42:26.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:25 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:26.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:25 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:26.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:25 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:42:27.979 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:42:27.976+0000 7f8d217fa700 1 -- 172.21.15.181:0/2291468787 <== mgr.14150 v2:172.21.15.38:6800/3328887328 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f8d24060bc0 con 0x7f8d0c09f3c0 2023-03-09T14:42:27.980 INFO:teuthology.orchestra.run.smithi181.stdout:Created osd(s) 8 on host 'smithi181' 2023-03-09T14:42:27.982 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:42:27.979+0000 7f8d2a3d5700 1 -- 172.21.15.181:0/2291468787 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f8d0c09f3c0 msgr2=0x7f8d0c0a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:42:27.982 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:42:27.979+0000 7f8d2a3d5700 1 --2- 172.21.15.181:0/2291468787 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f8d0c09f3c0 0x7f8d0c0a1870 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8d14005fd0 tx=0x7f8d14005dc0 comp rx=0 tx=0).stop 2023-03-09T14:42:27.982 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:42:27.979+0000 7f8d2a3d5700 1 -- 172.21.15.181:0/2291468787 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8d240fe620 msgr2=0x7f8d24107f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:42:27.982 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:42:27.979+0000 7f8d2a3d5700 1 --2- 172.21.15.181:0/2291468787 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8d240fe620 0x7f8d24107f20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8d18000c00 tx=0x7f8d1800f700 comp rx=0 tx=0).stop 2023-03-09T14:42:27.982 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:42:27.979+0000 7f8d2a3d5700 1 -- 172.21.15.181:0/2291468787 shutdown_connections 2023-03-09T14:42:27.982 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:42:27.979+0000 7f8d2a3d5700 1 --2- 172.21.15.181:0/2291468787 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f8d0c09f3c0 0x7f8d0c0a1870 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:42:27.983 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:42:27.979+0000 7f8d2a3d5700 1 --2- 172.21.15.181:0/2291468787 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8d241001a0 0x7f8d2419d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:42:27.983 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:42:27.979+0000 7f8d2a3d5700 1 --2- 172.21.15.181:0/2291468787 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8d240fe620 0x7f8d24107f20 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:42:27.983 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:42:27.979+0000 7f8d2a3d5700 1 --2- 172.21.15.181:0/2291468787 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8d240ff810 0x7f8d2419aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:42:27.983 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:42:27.979+0000 7f8d2a3d5700 1 -- 172.21.15.181:0/2291468787 >> 172.21.15.181:0/2291468787 conn(0x7f8d240f9c00 msgr2=0x7f8d240fbf40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:42:27.983 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:42:27.979+0000 7f8d2a3d5700 1 -- 172.21.15.181:0/2291468787 shutdown_connections 2023-03-09T14:42:27.983 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:42:27.979+0000 7f8d2a3d5700 1 -- 172.21.15.181:0/2291468787 wait complete. 2023-03-09T14:42:28.404 DEBUG:teuthology.orchestra.run.smithi181:osd.8> sudo journalctl -f -n 0 -u ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.8.service 2023-03-09T14:42:28.407 INFO:tasks.cephadm:Deploying osd.9 on smithi038 with /dev/vg_nvme/lv_1... 2023-03-09T14:42:28.407 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-03-09T14:42:28.435 INFO:journalctl@ceph.osd.8.smithi181.stdout:-- Logs begin at Thu 2023-03-09 14:25:01 UTC. -- 2023-03-09T14:42:28.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:28 smithi122 ceph-mon[93869]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-03-09T14:42:28.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:28 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:28.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:28 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:28.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:28 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:42:28.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:28 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:42:28.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:28 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:28.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:28 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:28.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:28 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:28.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:28 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:42:28.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:28 smithi181 ceph-mon[94189]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-03-09T14:42:28.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:28 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:28.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:28 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:28.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:28 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:42:28.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:28 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:42:28.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:28 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:28.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:28 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:28.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:28 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:28.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:28 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:42:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:28 smithi038 ceph-mon[92422]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-03-09T14:42:28.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:28 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:28.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:28 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:28.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:28 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:42:28.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:28 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:42:28.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:28 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:28.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:28 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:28.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:28 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:28.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:28 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:42:29.077 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.a/config 2023-03-09T14:42:29.119 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:42:28 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 2023-03-09T14:42:28.850+0000 7f7fdd5fc640 -1 osd.8 0 log_to_monitors true 2023-03-09T14:42:29.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:29 smithi181 ceph-mon[94189]: from='osd.8 [v2:172.21.15.181:6816/755000703,v1:172.21.15.181:6817/755000703]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-03-09T14:42:29.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:29 smithi181 ceph-mon[94189]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-03-09T14:42:29.573 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:29 smithi038 ceph-mon[92422]: from='osd.8 [v2:172.21.15.181:6816/755000703,v1:172.21.15.181:6817/755000703]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-03-09T14:42:29.573 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:29 smithi038 ceph-mon[92422]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-03-09T14:42:29.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:29 smithi122 ceph-mon[93869]: from='osd.8 [v2:172.21.15.181:6816/755000703,v1:172.21.15.181:6817/755000703]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-03-09T14:42:29.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:29 smithi122 ceph-mon[93869]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-03-09T14:42:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:30 smithi038 ceph-mon[92422]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-03-09T14:42:30.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:30 smithi038 ceph-mon[92422]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-03-09T14:42:30.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:30 smithi038 ceph-mon[92422]: osdmap e55: 9 total, 8 up, 9 in 2023-03-09T14:42:30.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:30 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-03-09T14:42:30.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:30 smithi038 ceph-mon[92422]: from='osd.8 [v2:172.21.15.181:6816/755000703,v1:172.21.15.181:6817/755000703]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2023-03-09T14:42:30.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:30 smithi038 ceph-mon[92422]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2023-03-09T14:42:30.646 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-03-09T14:42:30.647 INFO:teuthology.orchestra.run.smithi038.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-03-09T14:42:30.647 INFO:teuthology.orchestra.run.smithi038.stderr: stderr: 10+0 records in 2023-03-09T14:42:30.647 INFO:teuthology.orchestra.run.smithi038.stderr:10+0 records out 2023-03-09T14:42:30.647 INFO:teuthology.orchestra.run.smithi038.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0071643 s, 1.5 GB/s 2023-03-09T14:42:30.647 INFO:teuthology.orchestra.run.smithi038.stderr:--> Zapping successful for: 2023-03-09T14:42:30.654 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:42:30 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 2023-03-09T14:42:30.349+0000 7f7fd1581700 -1 osd.8 0 waiting for initial osdmap 2023-03-09T14:42:30.655 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:42:30 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 2023-03-09T14:42:30.360+0000 7f7fc9b6f700 -1 osd.8 56 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-09T14:42:30.655 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:30 smithi181 ceph-mon[94189]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-03-09T14:42:30.655 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:30 smithi181 ceph-mon[94189]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-03-09T14:42:30.655 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:30 smithi181 ceph-mon[94189]: osdmap e55: 9 total, 8 up, 9 in 2023-03-09T14:42:30.655 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:30 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-03-09T14:42:30.655 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:30 smithi181 ceph-mon[94189]: from='osd.8 [v2:172.21.15.181:6816/755000703,v1:172.21.15.181:6817/755000703]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2023-03-09T14:42:30.656 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:30 smithi181 ceph-mon[94189]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2023-03-09T14:42:30.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:30 smithi122 ceph-mon[93869]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-03-09T14:42:30.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:30 smithi122 ceph-mon[93869]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-03-09T14:42:30.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:30 smithi122 ceph-mon[93869]: osdmap e55: 9 total, 8 up, 9 in 2023-03-09T14:42:30.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:30 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-03-09T14:42:30.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:30 smithi122 ceph-mon[93869]: from='osd.8 [v2:172.21.15.181:6816/755000703,v1:172.21.15.181:6817/755000703]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2023-03-09T14:42:30.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:30 smithi122 ceph-mon[93869]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2023-03-09T14:42:31.035 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph orch daemon add osd smithi038:vg_nvme/lv_1 2023-03-09T14:42:31.687 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.a/config 2023-03-09T14:42:31.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:31 smithi122 ceph-mon[93869]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2023-03-09T14:42:31.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:31 smithi122 ceph-mon[93869]: osdmap e56: 9 total, 8 up, 9 in 2023-03-09T14:42:31.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:31 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-03-09T14:42:31.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:31 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-03-09T14:42:31.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:31 smithi122 ceph-mon[93869]: Detected new or changed devices on smithi181 2023-03-09T14:42:31.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:31 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:31.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:31 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:31.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:31 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:42:31.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:31 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:42:31.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:31 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:42:31.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:31 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 5722M 2023-03-09T14:42:31.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:31 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:31.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:31 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:42:31.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:31 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:42:31.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:31 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:31.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:31 smithi122 ceph-mon[93869]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-03-09T14:42:31.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:31 smithi181 ceph-mon[94189]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2023-03-09T14:42:31.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:31 smithi181 ceph-mon[94189]: osdmap e56: 9 total, 8 up, 9 in 2023-03-09T14:42:31.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:31 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-03-09T14:42:31.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:31 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-03-09T14:42:31.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:31 smithi181 ceph-mon[94189]: Detected new or changed devices on smithi181 2023-03-09T14:42:31.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:31 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:31.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:31 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:31.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:31 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:42:31.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:31 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:42:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:31 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:42:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:31 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 5722M 2023-03-09T14:42:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:31 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:31 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:42:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:31 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:42:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:31 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:31 smithi181 ceph-mon[94189]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-03-09T14:42:31.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:31 smithi038 ceph-mon[92422]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2023-03-09T14:42:31.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:31 smithi038 ceph-mon[92422]: osdmap e56: 9 total, 8 up, 9 in 2023-03-09T14:42:31.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:31 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-03-09T14:42:31.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:31 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-03-09T14:42:31.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:31 smithi038 ceph-mon[92422]: Detected new or changed devices on smithi181 2023-03-09T14:42:31.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:31 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:31.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:31 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:31.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:31 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:42:31.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:31 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:42:31.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:31 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:42:31.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:31 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 5722M 2023-03-09T14:42:31.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:31 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:31.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:31 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:42:31.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:31 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:42:31.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:31 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:31.753 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:31 smithi038 ceph-mon[92422]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2023-03-09T14:42:32.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:32 smithi038 ceph-mon[92422]: purged_snaps scrub starts 2023-03-09T14:42:32.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:32 smithi038 ceph-mon[92422]: purged_snaps scrub ok 2023-03-09T14:42:32.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:32 smithi038 ceph-mon[92422]: osd.8 [v2:172.21.15.181:6816/755000703,v1:172.21.15.181:6817/755000703] boot 2023-03-09T14:42:32.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:32 smithi038 ceph-mon[92422]: osdmap e57: 9 total, 9 up, 9 in 2023-03-09T14:42:32.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:32 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-03-09T14:42:32.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:32 smithi038 ceph-mon[92422]: osdmap e58: 9 total, 9 up, 9 in 2023-03-09T14:42:32.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:32 smithi122 ceph-mon[93869]: purged_snaps scrub starts 2023-03-09T14:42:32.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:32 smithi122 ceph-mon[93869]: purged_snaps scrub ok 2023-03-09T14:42:32.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:32 smithi122 ceph-mon[93869]: osd.8 [v2:172.21.15.181:6816/755000703,v1:172.21.15.181:6817/755000703] boot 2023-03-09T14:42:32.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:32 smithi122 ceph-mon[93869]: osdmap e57: 9 total, 9 up, 9 in 2023-03-09T14:42:32.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:32 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-03-09T14:42:32.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:32 smithi122 ceph-mon[93869]: osdmap e58: 9 total, 9 up, 9 in 2023-03-09T14:42:32.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:32 smithi181 ceph-mon[94189]: purged_snaps scrub starts 2023-03-09T14:42:32.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:32 smithi181 ceph-mon[94189]: purged_snaps scrub ok 2023-03-09T14:42:32.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:32 smithi181 ceph-mon[94189]: osd.8 [v2:172.21.15.181:6816/755000703,v1:172.21.15.181:6817/755000703] boot 2023-03-09T14:42:32.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:32 smithi181 ceph-mon[94189]: osdmap e57: 9 total, 9 up, 9 in 2023-03-09T14:42:32.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:32 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-03-09T14:42:32.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:32 smithi181 ceph-mon[94189]: osdmap e58: 9 total, 9 up, 9 in 2023-03-09T14:42:32.875 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:42:32.871+0000 7f1b582d5700 1 -- 172.21.15.38:0/4223748530 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b500feea0 msgr2=0x7f1b500ff2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:42:32.876 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:42:32.871+0000 7f1b582d5700 1 --2- 172.21.15.38:0/4223748530 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b500feea0 0x7f1b500ff2b0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c009990 tx=0x7f1b4c009ca0 comp rx=0 tx=0).stop 2023-03-09T14:42:32.876 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:42:32.871+0000 7f1b582d5700 1 -- 172.21.15.38:0/4223748530 shutdown_connections 2023-03-09T14:42:32.876 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:42:32.871+0000 7f1b582d5700 1 --2- 172.21.15.38:0/4223748530 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b500feea0 0x7f1b500ff2b0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:42:32.876 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:42:32.871+0000 7f1b582d5700 1 --2- 172.21.15.38:0/4223748530 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1b501009a0 0x7f1b501072d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:42:32.876 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:42:32.871+0000 7f1b582d5700 1 --2- 172.21.15.38:0/4223748530 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1b500ffff0 0x7f1b50100460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:42:32.877 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:42:32.871+0000 7f1b582d5700 1 -- 172.21.15.38:0/4223748530 >> 172.21.15.38:0/4223748530 conn(0x7f1b500fa480 msgr2=0x7f1b500fc8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:42:32.877 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:42:32.871+0000 7f1b582d5700 1 -- 172.21.15.38:0/4223748530 shutdown_connections 2023-03-09T14:42:32.877 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:42:32.871+0000 7f1b582d5700 1 -- 172.21.15.38:0/4223748530 wait complete. 2023-03-09T14:42:32.877 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:42:32.872+0000 7f1b582d5700 1 Processor -- start 2023-03-09T14:42:32.877 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:42:32.872+0000 7f1b582d5700 1 -- start start 2023-03-09T14:42:32.877 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:42:32.872+0000 7f1b582d5700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1b500feea0 0x7f1b50195d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:42:32.877 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:42:32.872+0000 7f1b582d5700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1b500ffff0 0x7f1b50196250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:42:32.878 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:42:32.872+0000 7f1b582d5700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b501009a0 0x7f1b5019d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:42:32.878 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:42:32.872+0000 7f1b582d5700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5019d850 con 0x7f1b501009a0 2023-03-09T14:42:32.878 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:42:32.872+0000 7f1b582d5700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5019d9c0 con 0x7f1b500feea0 2023-03-09T14:42:32.878 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:42:32.872+0000 7f1b582d5700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5019db80 con 0x7f1b500ffff0 2023-03-09T14:42:32.878 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:42:32.872+0000 7f1b56872700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b501009a0 0x7f1b5019d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:42:32.878 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:42:32.872+0000 7f1b56872700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b501009a0 0x7f1b5019d280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36128/0 (socket says 172.21.15.38:36128) 2023-03-09T14:42:32.878 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:42:32.872+0000 7f1b56872700 1 -- 172.21.15.38:0/3371097121 learned_addr learned my addr 172.21.15.38:0/3371097121 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:42:32.879 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:42:32.872+0000 7f1b55870700 1 --2- 172.21.15.38:0/3371097121 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1b500ffff0 0x7f1b50196250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:42:32.879 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:42:32.872+0000 7f1b56071700 1 --2- 172.21.15.38:0/3371097121 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1b500feea0 0x7f1b50195d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:42:32.879 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:42:32.872+0000 7f1b56872700 1 -- 172.21.15.38:0/3371097121 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1b500ffff0 msgr2=0x7f1b50196250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:42:32.879 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:42:32.872+0000 7f1b56872700 1 --2- 172.21.15.38:0/3371097121 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1b500ffff0 0x7f1b50196250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:42:32.879 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:42:32.872+0000 7f1b56872700 1 -- 172.21.15.38:0/3371097121 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1b500feea0 msgr2=0x7f1b50195d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:42:32.879 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:42:32.872+0000 7f1b56872700 1 --2- 172.21.15.38:0/3371097121 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1b500feea0 0x7f1b50195d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:42:32.879 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:42:32.872+0000 7f1b56872700 1 -- 172.21.15.38:0/3371097121 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b4c0095f0 con 0x7f1b501009a0 2023-03-09T14:42:32.879 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:42:32.873+0000 7f1b56872700 1 --2- 172.21.15.38:0/3371097121 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b501009a0 0x7f1b5019d280 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f1b48007ee0 tx=0x7f1b4800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:42:32.880 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:42:32.873+0000 7f1b477fe700 1 -- 172.21.15.38:0/3371097121 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b48017070 con 0x7f1b501009a0 2023-03-09T14:42:32.880 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:42:32.873+0000 7f1b582d5700 1 -- 172.21.15.38:0/3371097121 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b5010a800 con 0x7f1b501009a0 2023-03-09T14:42:32.880 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:42:32.873+0000 7f1b477fe700 1 -- 172.21.15.38:0/3371097121 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1b4800f040 con 0x7f1b501009a0 2023-03-09T14:42:32.880 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:42:32.873+0000 7f1b477fe700 1 -- 172.21.15.38:0/3371097121 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b48005740 con 0x7f1b501009a0 2023-03-09T14:42:32.880 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:42:32.873+0000 7f1b582d5700 1 -- 172.21.15.38:0/3371097121 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b5010a670 con 0x7f1b501009a0 2023-03-09T14:42:32.881 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:42:32.874+0000 7f1b582d5700 1 -- 172.21.15.38:0/3371097121 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b5019e190 con 0x7f1b501009a0 2023-03-09T14:42:32.881 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:42:32.875+0000 7f1b477fe700 1 -- 172.21.15.38:0/3371097121 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 133198+0+0 (secure 0 0 0) 0x7f1b4800dc60 con 0x7f1b501009a0 2023-03-09T14:42:32.881 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:42:32.875+0000 7f1b477fe700 1 --2- 172.21.15.38:0/3371097121 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f1b3c09f3c0 0x7f1b3c0a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:42:32.881 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:42:32.875+0000 7f1b56071700 1 --2- 172.21.15.38:0/3371097121 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f1b3c09f3c0 0x7f1b3c0a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:42:32.882 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:42:32.876+0000 7f1b477fe700 1 -- 172.21.15.38:0/3371097121 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5858+0+0 (secure 0 0 0) 0x7f1b480c8d50 con 0x7f1b501009a0 2023-03-09T14:42:32.882 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:42:32.876+0000 7f1b56071700 1 --2- 172.21.15.38:0/3371097121 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f1b3c09f3c0 0x7f1b3c0a1870 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c00b5c0 tx=0x7f1b4c005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:42:32.883 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:42:32.880+0000 7f1b477fe700 1 -- 172.21.15.38:0/3371097121 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f1b48093af0 con 0x7f1b501009a0 2023-03-09T14:42:33.089 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:42:33.087+0000 7f1b582d5700 1 -- 172.21.15.38:0/3371097121 --> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f1b500689b0 con 0x7f1b3c09f3c0 2023-03-09T14:42:33.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:33 smithi122 ceph-mon[93869]: pgmap v177: 1 pgs: 1 peering; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2023-03-09T14:42:33.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:33 smithi122 ceph-mon[93869]: from='client.14415 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:42:33.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:33 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-03-09T14:42:33.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:33 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-03-09T14:42:33.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:33 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:42:33.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:33 smithi122 ceph-mon[93869]: osdmap e59: 9 total, 9 up, 9 in 2023-03-09T14:42:33.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:33 smithi181 ceph-mon[94189]: pgmap v177: 1 pgs: 1 peering; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2023-03-09T14:42:33.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:33 smithi181 ceph-mon[94189]: from='client.14415 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:42:33.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:33 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-03-09T14:42:33.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:33 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-03-09T14:42:33.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:33 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:42:33.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:33 smithi181 ceph-mon[94189]: osdmap e59: 9 total, 9 up, 9 in 2023-03-09T14:42:33.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:33 smithi038 ceph-mon[92422]: pgmap v177: 1 pgs: 1 peering; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2023-03-09T14:42:33.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:33 smithi038 ceph-mon[92422]: from='client.14415 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi038:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:42:33.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:33 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-03-09T14:42:33.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:33 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-03-09T14:42:33.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:33 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:42:33.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:33 smithi038 ceph-mon[92422]: osdmap e59: 9 total, 9 up, 9 in 2023-03-09T14:42:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:35 smithi122 ceph-mon[93869]: osdmap e60: 9 total, 9 up, 9 in 2023-03-09T14:42:35.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:35 smithi122 ceph-mon[93869]: pgmap v180: 1 pgs: 1 peering; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2023-03-09T14:42:35.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:35 smithi181 ceph-mon[94189]: osdmap e60: 9 total, 9 up, 9 in 2023-03-09T14:42:35.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:35 smithi181 ceph-mon[94189]: pgmap v180: 1 pgs: 1 peering; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2023-03-09T14:42:35.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:35 smithi038 ceph-mon[92422]: osdmap e60: 9 total, 9 up, 9 in 2023-03-09T14:42:35.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:35 smithi038 ceph-mon[92422]: pgmap v180: 1 pgs: 1 peering; 577 KiB data, 54 MiB used, 805 GiB / 805 GiB avail 2023-03-09T14:42:36.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:36 smithi122 ceph-mon[93869]: osdmap e61: 9 total, 9 up, 9 in 2023-03-09T14:42:36.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:36 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/2496443249' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "94cc0355-5a1a-4cfc-bd0c-562b6b41127e"}]: dispatch 2023-03-09T14:42:36.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:36 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/2496443249' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "94cc0355-5a1a-4cfc-bd0c-562b6b41127e"}]': finished 2023-03-09T14:42:36.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:36 smithi122 ceph-mon[93869]: osdmap e62: 10 total, 9 up, 10 in 2023-03-09T14:42:36.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:36 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-03-09T14:42:36.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:36 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/4019818870' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-03-09T14:42:36.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:36 smithi181 ceph-mon[94189]: osdmap e61: 9 total, 9 up, 9 in 2023-03-09T14:42:36.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:36 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/2496443249' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "94cc0355-5a1a-4cfc-bd0c-562b6b41127e"}]: dispatch 2023-03-09T14:42:36.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:36 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/2496443249' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "94cc0355-5a1a-4cfc-bd0c-562b6b41127e"}]': finished 2023-03-09T14:42:36.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:36 smithi181 ceph-mon[94189]: osdmap e62: 10 total, 9 up, 10 in 2023-03-09T14:42:36.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:36 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-03-09T14:42:36.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:36 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/4019818870' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-03-09T14:42:36.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:36 smithi038 ceph-mon[92422]: osdmap e61: 9 total, 9 up, 9 in 2023-03-09T14:42:36.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:36 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/2496443249' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "94cc0355-5a1a-4cfc-bd0c-562b6b41127e"}]: dispatch 2023-03-09T14:42:36.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:36 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/2496443249' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "94cc0355-5a1a-4cfc-bd0c-562b6b41127e"}]': finished 2023-03-09T14:42:36.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:36 smithi038 ceph-mon[92422]: osdmap e62: 10 total, 9 up, 10 in 2023-03-09T14:42:36.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:36 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-03-09T14:42:36.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:36 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/4019818870' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-03-09T14:42:38.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:38 smithi038 ceph-mon[92422]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-03-09T14:42:38.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:38 smithi122 ceph-mon[93869]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-03-09T14:42:38.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:38 smithi181 ceph-mon[94189]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-03-09T14:42:40.569 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:40 smithi038 ceph-mon[92422]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail; 105 KiB/s, 0 objects/s recovering 2023-03-09T14:42:40.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:40 smithi122 ceph-mon[93869]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail; 105 KiB/s, 0 objects/s recovering 2023-03-09T14:42:40.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:40 smithi181 ceph-mon[94189]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail; 105 KiB/s, 0 objects/s recovering 2023-03-09T14:42:42.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:42 smithi038 ceph-mon[92422]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail; 91 KiB/s, 0 objects/s recovering 2023-03-09T14:42:42.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:42 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-03-09T14:42:42.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:42 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:42:42.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:42 smithi122 ceph-mon[93869]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail; 91 KiB/s, 0 objects/s recovering 2023-03-09T14:42:42.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:42 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-03-09T14:42:42.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:42 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:42:42.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:42 smithi181 ceph-mon[94189]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail; 91 KiB/s, 0 objects/s recovering 2023-03-09T14:42:42.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:42 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-03-09T14:42:42.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:42 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:42:43.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:43 smithi038 ceph-mon[92422]: Deploying daemon osd.9 on smithi038 2023-03-09T14:42:43.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:43 smithi122 ceph-mon[93869]: Deploying daemon osd.9 on smithi038 2023-03-09T14:42:43.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:43 smithi181 ceph-mon[94189]: Deploying daemon osd.9 on smithi038 2023-03-09T14:42:44.451 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:44 smithi038 ceph-mon[92422]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-03-09T14:42:44.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:44 smithi122 ceph-mon[93869]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-03-09T14:42:44.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:44 smithi181 ceph-mon[94189]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-03-09T14:42:46.625 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:46 smithi038 ceph-mon[92422]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail; 62 KiB/s, 0 objects/s recovering 2023-03-09T14:42:46.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:46 smithi122 ceph-mon[93869]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail; 62 KiB/s, 0 objects/s recovering 2023-03-09T14:42:46.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:46 smithi181 ceph-mon[94189]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail; 62 KiB/s, 0 objects/s recovering 2023-03-09T14:42:48.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:47 smithi122 ceph-mon[93869]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail; 52 KiB/s, 0 objects/s recovering 2023-03-09T14:42:48.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:47 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:48.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:47 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:48.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:47 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:42:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:47 smithi181 ceph-mon[94189]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail; 52 KiB/s, 0 objects/s recovering 2023-03-09T14:42:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:47 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:48.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:47 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:48.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:47 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:42:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:47 smithi038 ceph-mon[92422]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail; 52 KiB/s, 0 objects/s recovering 2023-03-09T14:42:48.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:47 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:48.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:47 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:48.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:47 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:42:50.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:49 smithi038 ceph-mon[92422]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail 2023-03-09T14:42:50.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:49 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:50.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:49 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:50.226 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:42:50.222+0000 7f1b477fe700 1 -- 172.21.15.38:0/3371097121 <== mgr.14150 v2:172.21.15.38:6800/3328887328 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f1b500689b0 con 0x7f1b3c09f3c0 2023-03-09T14:42:50.226 INFO:teuthology.orchestra.run.smithi038.stdout:Created osd(s) 9 on host 'smithi038' 2023-03-09T14:42:50.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:49 smithi122 ceph-mon[93869]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail 2023-03-09T14:42:50.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:49 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:50.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:49 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:50.228 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:42:50.225+0000 7f1b582d5700 1 -- 172.21.15.38:0/3371097121 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f1b3c09f3c0 msgr2=0x7f1b3c0a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:42:50.228 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:42:50.225+0000 7f1b582d5700 1 --2- 172.21.15.38:0/3371097121 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f1b3c09f3c0 0x7f1b3c0a1870 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c00b5c0 tx=0x7f1b4c005fd0 comp rx=0 tx=0).stop 2023-03-09T14:42:50.229 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:42:50.225+0000 7f1b582d5700 1 -- 172.21.15.38:0/3371097121 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b501009a0 msgr2=0x7f1b5019d280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:42:50.229 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:42:50.225+0000 7f1b582d5700 1 --2- 172.21.15.38:0/3371097121 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b501009a0 0x7f1b5019d280 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f1b48007ee0 tx=0x7f1b4800d3b0 comp rx=0 tx=0).stop 2023-03-09T14:42:50.229 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:42:50.225+0000 7f1b582d5700 1 -- 172.21.15.38:0/3371097121 shutdown_connections 2023-03-09T14:42:50.229 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:42:50.225+0000 7f1b582d5700 1 --2- 172.21.15.38:0/3371097121 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b501009a0 0x7f1b5019d280 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:42:50.229 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:42:50.225+0000 7f1b582d5700 1 --2- 172.21.15.38:0/3371097121 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f1b3c09f3c0 0x7f1b3c0a1870 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:42:50.229 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:42:50.225+0000 7f1b582d5700 1 --2- 172.21.15.38:0/3371097121 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1b500feea0 0x7f1b50195d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:42:50.230 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:42:50.225+0000 7f1b582d5700 1 --2- 172.21.15.38:0/3371097121 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1b500ffff0 0x7f1b50196250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:42:50.230 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:42:50.225+0000 7f1b582d5700 1 -- 172.21.15.38:0/3371097121 >> 172.21.15.38:0/3371097121 conn(0x7f1b500fa480 msgr2=0x7f1b500682a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:42:50.230 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:42:50.225+0000 7f1b582d5700 1 -- 172.21.15.38:0/3371097121 shutdown_connections 2023-03-09T14:42:50.230 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:42:50.225+0000 7f1b582d5700 1 -- 172.21.15.38:0/3371097121 wait complete. 2023-03-09T14:42:50.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:49 smithi181 ceph-mon[94189]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail 2023-03-09T14:42:50.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:49 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:50.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:49 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:50.674 DEBUG:teuthology.orchestra.run.smithi038:osd.9> sudo journalctl -f -n 0 -u ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.9.service 2023-03-09T14:42:50.678 INFO:tasks.cephadm:Deploying osd.10 on smithi122 with /dev/vg_nvme/lv_1... 2023-03-09T14:42:50.678 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-03-09T14:42:50.708 INFO:journalctl@ceph.osd.9.smithi038.stdout:-- Logs begin at Thu 2023-03-09 14:24:59 UTC. -- 2023-03-09T14:42:50.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:50 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:42:50.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:50 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:42:50.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:50 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:50.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:50 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:50.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:50 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:50.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:50 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:42:50.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:50 smithi122 ceph-mon[93869]: from='osd.9 [v2:172.21.15.38:6826/2624555969,v1:172.21.15.38:6827/2624555969]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-03-09T14:42:51.000 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:42:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 2023-03-09T14:42:50.740+0000 7fa0158c9640 -1 osd.9 0 log_to_monitors true 2023-03-09T14:42:51.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:50 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:42:51.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:50 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:42:51.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:50 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:51.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:50 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:51.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:50 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:51.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:50 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:42:51.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:50 smithi038 ceph-mon[92422]: from='osd.9 [v2:172.21.15.38:6826/2624555969,v1:172.21.15.38:6827/2624555969]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-03-09T14:42:51.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:50 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:42:51.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:50 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:42:51.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:50 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:51.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:50 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:51.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:50 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:51.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:50 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:42:51.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:50 smithi181 ceph-mon[94189]: from='osd.9 [v2:172.21.15.38:6826/2624555969,v1:172.21.15.38:6827/2624555969]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-03-09T14:42:51.339 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.b/config 2023-03-09T14:42:52.261 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:52 smithi038 ceph-mon[92422]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail 2023-03-09T14:42:52.261 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:52 smithi038 ceph-mon[92422]: from='osd.9 [v2:172.21.15.38:6826/2624555969,v1:172.21.15.38:6827/2624555969]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-03-09T14:42:52.261 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:52 smithi038 ceph-mon[92422]: osdmap e63: 10 total, 9 up, 10 in 2023-03-09T14:42:52.262 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:52 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-03-09T14:42:52.262 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:52 smithi038 ceph-mon[92422]: from='osd.9 [v2:172.21.15.38:6826/2624555969,v1:172.21.15.38:6827/2624555969]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2023-03-09T14:42:52.262 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:42:52 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 2023-03-09T14:42:52.240+0000 7fa00984e700 -1 osd.9 0 waiting for initial osdmap 2023-03-09T14:42:52.262 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:42:52 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 2023-03-09T14:42:52.257+0000 7fa00363f700 -1 osd.9 64 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-09T14:42:52.365 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:52 smithi122 ceph-mon[93869]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail 2023-03-09T14:42:52.365 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:52 smithi122 ceph-mon[93869]: from='osd.9 [v2:172.21.15.38:6826/2624555969,v1:172.21.15.38:6827/2624555969]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-03-09T14:42:52.365 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:52 smithi122 ceph-mon[93869]: osdmap e63: 10 total, 9 up, 10 in 2023-03-09T14:42:52.366 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:52 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-03-09T14:42:52.366 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:52 smithi122 ceph-mon[93869]: from='osd.9 [v2:172.21.15.38:6826/2624555969,v1:172.21.15.38:6827/2624555969]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2023-03-09T14:42:52.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:52 smithi181 ceph-mon[94189]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail 2023-03-09T14:42:52.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:52 smithi181 ceph-mon[94189]: from='osd.9 [v2:172.21.15.38:6826/2624555969,v1:172.21.15.38:6827/2624555969]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-03-09T14:42:52.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:52 smithi181 ceph-mon[94189]: osdmap e63: 10 total, 9 up, 10 in 2023-03-09T14:42:52.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:52 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-03-09T14:42:52.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:52 smithi181 ceph-mon[94189]: from='osd.9 [v2:172.21.15.38:6826/2624555969,v1:172.21.15.38:6827/2624555969]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2023-03-09T14:42:52.835 INFO:teuthology.orchestra.run.smithi122.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-03-09T14:42:52.835 INFO:teuthology.orchestra.run.smithi122.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-03-09T14:42:52.835 INFO:teuthology.orchestra.run.smithi122.stderr: stderr: 10+0 records in 2023-03-09T14:42:52.835 INFO:teuthology.orchestra.run.smithi122.stderr:10+0 records out 2023-03-09T14:42:52.835 INFO:teuthology.orchestra.run.smithi122.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0108776 s, 964 MB/s 2023-03-09T14:42:52.835 INFO:teuthology.orchestra.run.smithi122.stderr: stderr: 2023-03-09T14:42:52.836 INFO:teuthology.orchestra.run.smithi122.stderr:--> Zapping successful for: 2023-03-09T14:42:53.391 DEBUG:teuthology.orchestra.run.smithi122:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph orch daemon add osd smithi122:vg_nvme/lv_1 2023-03-09T14:42:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:53 smithi038 ceph-mon[92422]: from='osd.9 [v2:172.21.15.38:6826/2624555969,v1:172.21.15.38:6827/2624555969]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2023-03-09T14:42:53.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:53 smithi038 ceph-mon[92422]: osdmap e64: 10 total, 9 up, 10 in 2023-03-09T14:42:53.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:53 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-03-09T14:42:53.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:53 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-03-09T14:42:53.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:53 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:53.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:53 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:53.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:53 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:42:53.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:53 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:42:53.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:53 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:42:53.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:53 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:42:53.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:53 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:53.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:53 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:42:53.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:53 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:42:53.503 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:53 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:53.503 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:53 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-03-09T14:42:53.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:53 smithi122 ceph-mon[93869]: from='osd.9 [v2:172.21.15.38:6826/2624555969,v1:172.21.15.38:6827/2624555969]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2023-03-09T14:42:53.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:53 smithi122 ceph-mon[93869]: osdmap e64: 10 total, 9 up, 10 in 2023-03-09T14:42:53.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:53 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-03-09T14:42:53.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:53 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-03-09T14:42:53.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:53 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:53.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:53 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:53.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:53 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:42:53.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:53 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:42:53.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:53 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:42:53.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:53 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:42:53.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:53 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:53.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:53 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:42:53.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:53 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:42:53.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:53 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:53.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:53 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-03-09T14:42:53.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:53 smithi181 ceph-mon[94189]: from='osd.9 [v2:172.21.15.38:6826/2624555969,v1:172.21.15.38:6827/2624555969]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]': finished 2023-03-09T14:42:53.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:53 smithi181 ceph-mon[94189]: osdmap e64: 10 total, 9 up, 10 in 2023-03-09T14:42:53.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:53 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-03-09T14:42:53.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:53 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-03-09T14:42:53.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:53 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:53.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:53 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:53.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:53 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:42:53.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:53 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:42:53.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:53 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:42:53.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:53 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:42:53.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:53 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:53.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:53 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:42:53.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:53 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:42:53.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:53 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:42:53.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:53 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-03-09T14:42:54.028 INFO:teuthology.orchestra.run.smithi122.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.b/config 2023-03-09T14:42:54.531 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:54 smithi122 ceph-mon[93869]: purged_snaps scrub starts 2023-03-09T14:42:54.531 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:54 smithi122 ceph-mon[93869]: purged_snaps scrub ok 2023-03-09T14:42:54.532 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:54 smithi122 ceph-mon[93869]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail 2023-03-09T14:42:54.532 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:54 smithi122 ceph-mon[93869]: Detected new or changed devices on smithi038 2023-03-09T14:42:54.532 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:54 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi038 to 4293M 2023-03-09T14:42:54.532 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:54 smithi122 ceph-mon[93869]: osd.9 [v2:172.21.15.38:6826/2624555969,v1:172.21.15.38:6827/2624555969] boot 2023-03-09T14:42:54.532 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:54 smithi122 ceph-mon[93869]: osdmap e65: 10 total, 10 up, 10 in 2023-03-09T14:42:54.533 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:54 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-03-09T14:42:54.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:54 smithi181 ceph-mon[94189]: purged_snaps scrub starts 2023-03-09T14:42:54.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:54 smithi181 ceph-mon[94189]: purged_snaps scrub ok 2023-03-09T14:42:54.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:54 smithi181 ceph-mon[94189]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail 2023-03-09T14:42:54.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:54 smithi181 ceph-mon[94189]: Detected new or changed devices on smithi038 2023-03-09T14:42:54.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:54 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi038 to 4293M 2023-03-09T14:42:54.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:54 smithi181 ceph-mon[94189]: osd.9 [v2:172.21.15.38:6826/2624555969,v1:172.21.15.38:6827/2624555969] boot 2023-03-09T14:42:54.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:54 smithi181 ceph-mon[94189]: osdmap e65: 10 total, 10 up, 10 in 2023-03-09T14:42:54.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:54 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-03-09T14:42:54.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:54 smithi038 ceph-mon[92422]: purged_snaps scrub starts 2023-03-09T14:42:54.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:54 smithi038 ceph-mon[92422]: purged_snaps scrub ok 2023-03-09T14:42:54.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:54 smithi038 ceph-mon[92422]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 55 MiB used, 805 GiB / 805 GiB avail 2023-03-09T14:42:54.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:54 smithi038 ceph-mon[92422]: Detected new or changed devices on smithi038 2023-03-09T14:42:54.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:54 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi038 to 4293M 2023-03-09T14:42:54.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:54 smithi038 ceph-mon[92422]: osd.9 [v2:172.21.15.38:6826/2624555969,v1:172.21.15.38:6827/2624555969] boot 2023-03-09T14:42:54.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:54 smithi038 ceph-mon[92422]: osdmap e65: 10 total, 10 up, 10 in 2023-03-09T14:42:54.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:54 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-03-09T14:42:55.221 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:42:55.217+0000 7f0fa19a2700 1 -- 172.21.15.122:0/812040022 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0f9c101910 msgr2=0x7f0f9c108240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:42:55.222 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:42:55.217+0000 7f0fa19a2700 1 --2- 172.21.15.122:0/812040022 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0f9c101910 0x7f0f9c108240 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0f9400b0a0 tx=0x7f0f9400b3b0 comp rx=0 tx=0).stop 2023-03-09T14:42:55.222 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:42:55.217+0000 7f0fa19a2700 1 -- 172.21.15.122:0/812040022 shutdown_connections 2023-03-09T14:42:55.222 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:42:55.217+0000 7f0fa19a2700 1 --2- 172.21.15.122:0/812040022 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0f9c101910 0x7f0f9c108240 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:42:55.222 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:42:55.217+0000 7f0fa19a2700 1 --2- 172.21.15.122:0/812040022 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0f9c0ffe10 0x7f0f9c100220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:42:55.222 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:42:55.217+0000 7f0fa19a2700 1 --2- 172.21.15.122:0/812040022 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0f9c100f60 0x7f0f9c1013d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:42:55.223 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:42:55.217+0000 7f0fa19a2700 1 -- 172.21.15.122:0/812040022 >> 172.21.15.122:0/812040022 conn(0x7f0f9c0fb3f0 msgr2=0x7f0f9c0fd860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:42:55.223 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:42:55.217+0000 7f0fa19a2700 1 -- 172.21.15.122:0/812040022 shutdown_connections 2023-03-09T14:42:55.223 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:42:55.217+0000 7f0fa19a2700 1 -- 172.21.15.122:0/812040022 wait complete. 2023-03-09T14:42:55.223 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:42:55.218+0000 7f0fa19a2700 1 Processor -- start 2023-03-09T14:42:55.223 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:42:55.218+0000 7f0fa19a2700 1 -- start start 2023-03-09T14:42:55.223 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:42:55.218+0000 7f0fa19a2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0f9c0ffe10 0x7f0f9c195c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:42:55.223 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:42:55.218+0000 7f0fa19a2700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0f9c100f60 0x7f0f9c1961b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:42:55.224 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:42:55.218+0000 7f0fa19a2700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0f9c101910 0x7f0f9c19d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:42:55.224 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:42:55.218+0000 7f0fa19a2700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f9c19d7b0 con 0x7f0f9c0ffe10 2023-03-09T14:42:55.224 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:42:55.218+0000 7f0fa19a2700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f9c19d920 con 0x7f0f9c100f60 2023-03-09T14:42:55.224 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:42:55.218+0000 7f0fa19a2700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f9c19dae0 con 0x7f0f9c101910 2023-03-09T14:42:55.224 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:42:55.218+0000 7f0f9bfff700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0f9c100f60 0x7f0f9c1961b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:42:55.224 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:42:55.218+0000 7f0f9bfff700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0f9c100f60 0x7f0f9c1961b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.122:49512/0 (socket says 172.21.15.122:49512) 2023-03-09T14:42:55.224 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:42:55.218+0000 7f0f9bfff700 1 -- 172.21.15.122:0/1938144045 learned_addr learned my addr 172.21.15.122:0/1938144045 (peer_addr_for_me v2:172.21.15.122:0/0) 2023-03-09T14:42:55.225 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:42:55.218+0000 7f0fa09a0700 1 --2- 172.21.15.122:0/1938144045 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0f9c0ffe10 0x7f0f9c195c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:42:55.225 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:42:55.218+0000 7f0fa11a1700 1 --2- 172.21.15.122:0/1938144045 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0f9c101910 0x7f0f9c19d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:42:55.225 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:42:55.218+0000 7f0f9bfff700 1 -- 172.21.15.122:0/1938144045 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0f9c101910 msgr2=0x7f0f9c19d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:42:55.225 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:42:55.218+0000 7f0f9bfff700 1 --2- 172.21.15.122:0/1938144045 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0f9c101910 0x7f0f9c19d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:42:55.225 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:42:55.218+0000 7f0f9bfff700 1 -- 172.21.15.122:0/1938144045 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0f9c0ffe10 msgr2=0x7f0f9c195c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:42:55.225 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:42:55.218+0000 7f0f9bfff700 1 --2- 172.21.15.122:0/1938144045 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0f9c0ffe10 0x7f0f9c195c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:42:55.225 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:42:55.218+0000 7f0f9bfff700 1 -- 172.21.15.122:0/1938144045 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f94009d00 con 0x7f0f9c100f60 2023-03-09T14:42:55.225 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:42:55.218+0000 7f0f9bfff700 1 --2- 172.21.15.122:0/1938144045 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0f9c100f60 0x7f0f9c1961b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0f8800c9b0 tx=0x7f0f8800ccc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:42:55.226 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:42:55.218+0000 7f0f99ffb700 1 -- 172.21.15.122:0/1938144045 <== mon.2 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f88007b00 con 0x7f0f9c100f60 2023-03-09T14:42:55.226 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:42:55.219+0000 7f0fa19a2700 1 -- 172.21.15.122:0/1938144045 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f9c10b7f0 con 0x7f0f9c100f60 2023-03-09T14:42:55.226 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:42:55.219+0000 7f0f99ffb700 1 -- 172.21.15.122:0/1938144045 <== mon.2 v2:172.21.15.122:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0f88007c60 con 0x7f0f9c100f60 2023-03-09T14:42:55.226 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:42:55.219+0000 7f0f99ffb700 1 -- 172.21.15.122:0/1938144045 <== mon.2 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f88018660 con 0x7f0f9c100f60 2023-03-09T14:42:55.227 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:42:55.219+0000 7f0fa19a2700 1 -- 172.21.15.122:0/1938144045 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f9c10b660 con 0x7f0f9c100f60 2023-03-09T14:42:55.227 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:42:55.220+0000 7f0fa19a2700 1 -- 172.21.15.122:0/1938144045 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f9c19e200 con 0x7f0f9c100f60 2023-03-09T14:42:55.227 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:42:55.220+0000 7f0f99ffb700 1 -- 172.21.15.122:0/1938144045 <== mon.2 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 16) v1 ==== 133198+0+0 (secure 0 0 0) 0x7f0f8800f450 con 0x7f0f9c100f60 2023-03-09T14:42:55.227 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:42:55.221+0000 7f0f99ffb700 1 --2- 172.21.15.122:0/1938144045 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f0f8c09f3c0 0x7f0f8c0a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:42:55.228 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:42:55.221+0000 7f0f99ffb700 1 -- 172.21.15.122:0/1938144045 <== mon.2 v2:172.21.15.122:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6261+0+0 (secure 0 0 0) 0x7f0f880c96b0 con 0x7f0f9c100f60 2023-03-09T14:42:55.228 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:42:55.221+0000 7f0fa09a0700 1 --2- 172.21.15.122:0/1938144045 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f0f8c09f3c0 0x7f0f8c0a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:42:55.228 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:42:55.222+0000 7f0fa09a0700 1 --2- 172.21.15.122:0/1938144045 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f0f8c09f3c0 0x7f0f8c0a1870 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0f9c197140 tx=0x7f0f90009190 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:42:55.228 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:42:55.225+0000 7f0f99ffb700 1 -- 172.21.15.122:0/1938144045 <== mon.2 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f0f880942c0 con 0x7f0f9c100f60 2023-03-09T14:42:55.424 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:42:55.420+0000 7f0fa19a2700 1 -- 172.21.15.122:0/1938144045 --> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f0f9c19e7b0 con 0x7f0f8c09f3c0 2023-03-09T14:42:55.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:55 smithi122 ceph-mon[93869]: osdmap e66: 10 total, 10 up, 10 in 2023-03-09T14:42:55.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:55 smithi181 ceph-mon[94189]: osdmap e66: 10 total, 10 up, 10 in 2023-03-09T14:42:55.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:55 smithi038 ceph-mon[92422]: osdmap e66: 10 total, 10 up, 10 in 2023-03-09T14:42:56.672 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:56 smithi122 ceph-mon[93869]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-03-09T14:42:56.673 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:56 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-03-09T14:42:56.673 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:56 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-03-09T14:42:56.673 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:56 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:42:56.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:56 smithi181 ceph-mon[94189]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-03-09T14:42:56.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:56 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-03-09T14:42:56.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:56 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-03-09T14:42:56.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:56 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:42:56.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:56 smithi038 ceph-mon[92422]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 60 MiB used, 894 GiB / 894 GiB avail 2023-03-09T14:42:56.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:56 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-03-09T14:42:56.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:56 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-03-09T14:42:56.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:56 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:42:57.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:57 smithi122 ceph-mon[93869]: from='client.24344 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:42:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:57 smithi181 ceph-mon[94189]: from='client.24344 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:42:57.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:57 smithi038 ceph-mon[92422]: from='client.24344 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi122:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:42:58.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:58 smithi122 ceph-mon[93869]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-03-09T14:42:58.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:58 smithi122 ceph-mon[93869]: from='client.? 172.21.15.122:0/1368229034' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f4f2e042-8d92-46bf-93b8-8b0f04795c2b"}]: dispatch 2023-03-09T14:42:58.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:58 smithi122 ceph-mon[93869]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f4f2e042-8d92-46bf-93b8-8b0f04795c2b"}]: dispatch 2023-03-09T14:42:58.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:58 smithi122 ceph-mon[93869]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f4f2e042-8d92-46bf-93b8-8b0f04795c2b"}]': finished 2023-03-09T14:42:58.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:58 smithi122 ceph-mon[93869]: osdmap e67: 11 total, 10 up, 11 in 2023-03-09T14:42:58.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:58 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-03-09T14:42:58.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:58 smithi181 ceph-mon[94189]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-03-09T14:42:58.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:58 smithi181 ceph-mon[94189]: from='client.? 172.21.15.122:0/1368229034' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f4f2e042-8d92-46bf-93b8-8b0f04795c2b"}]: dispatch 2023-03-09T14:42:58.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:58 smithi181 ceph-mon[94189]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f4f2e042-8d92-46bf-93b8-8b0f04795c2b"}]: dispatch 2023-03-09T14:42:58.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:58 smithi181 ceph-mon[94189]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f4f2e042-8d92-46bf-93b8-8b0f04795c2b"}]': finished 2023-03-09T14:42:58.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:58 smithi181 ceph-mon[94189]: osdmap e67: 11 total, 10 up, 11 in 2023-03-09T14:42:58.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:58 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-03-09T14:42:58.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:58 smithi038 ceph-mon[92422]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-03-09T14:42:58.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:58 smithi038 ceph-mon[92422]: from='client.? 172.21.15.122:0/1368229034' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f4f2e042-8d92-46bf-93b8-8b0f04795c2b"}]: dispatch 2023-03-09T14:42:58.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:58 smithi038 ceph-mon[92422]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f4f2e042-8d92-46bf-93b8-8b0f04795c2b"}]: dispatch 2023-03-09T14:42:58.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:58 smithi038 ceph-mon[92422]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f4f2e042-8d92-46bf-93b8-8b0f04795c2b"}]': finished 2023-03-09T14:42:58.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:58 smithi038 ceph-mon[92422]: osdmap e67: 11 total, 10 up, 11 in 2023-03-09T14:42:58.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:58 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-03-09T14:42:59.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:42:59 smithi122 ceph-mon[93869]: from='client.? 172.21.15.122:0/987485648' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-03-09T14:42:59.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:42:59 smithi181 ceph-mon[94189]: from='client.? 172.21.15.122:0/987485648' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-03-09T14:42:59.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:42:59 smithi038 ceph-mon[92422]: from='client.? 172.21.15.122:0/987485648' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-03-09T14:43:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:00 smithi122 ceph-mon[93869]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-03-09T14:43:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:00 smithi181 ceph-mon[94189]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-03-09T14:43:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:00 smithi038 ceph-mon[92422]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-03-09T14:43:02.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:02 smithi122 ceph-mon[93869]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-03-09T14:43:02.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:02 smithi181 ceph-mon[94189]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-03-09T14:43:02.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:02 smithi038 ceph-mon[92422]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-03-09T14:43:04.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:04 smithi122 ceph-mon[93869]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-03-09T14:43:04.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:04 smithi181 ceph-mon[94189]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-03-09T14:43:04.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:04 smithi038 ceph-mon[92422]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-03-09T14:43:05.614 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:05 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-03-09T14:43:05.614 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:05 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:43:05.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:05 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-03-09T14:43:05.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:05 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:43:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:05 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-03-09T14:43:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:05 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:43:06.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:06 smithi122 ceph-mon[93869]: Deploying daemon osd.10 on smithi122 2023-03-09T14:43:06.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:06 smithi122 ceph-mon[93869]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-03-09T14:43:06.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:06 smithi181 ceph-mon[94189]: Deploying daemon osd.10 on smithi122 2023-03-09T14:43:06.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:06 smithi181 ceph-mon[94189]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-03-09T14:43:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:06 smithi038 ceph-mon[92422]: Deploying daemon osd.10 on smithi122 2023-03-09T14:43:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:06 smithi038 ceph-mon[92422]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-03-09T14:43:08.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:08 smithi122 ceph-mon[93869]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-03-09T14:43:08.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:08 smithi181 ceph-mon[94189]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-03-09T14:43:08.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:08 smithi038 ceph-mon[92422]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-03-09T14:43:10.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:09 smithi122 ceph-mon[93869]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-03-09T14:43:10.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:09 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:10.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:09 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:10.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:09 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:43:10.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:09 smithi181 ceph-mon[94189]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-03-09T14:43:10.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:09 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:10.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:09 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:10.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:09 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:43:10.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:09 smithi038 ceph-mon[92422]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-03-09T14:43:10.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:09 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:10.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:09 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:10.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:09 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:43:11.950 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:43:11.947+0000 7f0f99ffb700 1 -- 172.21.15.122:0/1938144045 <== mgr.14150 v2:172.21.15.38:6800/3328887328 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f0f9c19e7b0 con 0x7f0f8c09f3c0 2023-03-09T14:43:11.951 INFO:teuthology.orchestra.run.smithi122.stdout:Created osd(s) 10 on host 'smithi122' 2023-03-09T14:43:11.952 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:43:11.949+0000 7f0fa19a2700 1 -- 172.21.15.122:0/1938144045 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f0f8c09f3c0 msgr2=0x7f0f8c0a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:43:11.952 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:43:11.949+0000 7f0fa19a2700 1 --2- 172.21.15.122:0/1938144045 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f0f8c09f3c0 0x7f0f8c0a1870 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0f9c197140 tx=0x7f0f90009190 comp rx=0 tx=0).stop 2023-03-09T14:43:11.953 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:43:11.949+0000 7f0fa19a2700 1 -- 172.21.15.122:0/1938144045 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0f9c100f60 msgr2=0x7f0f9c1961b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:43:11.953 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:43:11.949+0000 7f0fa19a2700 1 --2- 172.21.15.122:0/1938144045 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0f9c100f60 0x7f0f9c1961b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0f8800c9b0 tx=0x7f0f8800ccc0 comp rx=0 tx=0).stop 2023-03-09T14:43:11.953 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:43:11.950+0000 7f0fa19a2700 1 -- 172.21.15.122:0/1938144045 shutdown_connections 2023-03-09T14:43:11.953 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:43:11.950+0000 7f0fa19a2700 1 --2- 172.21.15.122:0/1938144045 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0f9c0ffe10 0x7f0f9c195c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:11.953 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:43:11.950+0000 7f0fa19a2700 1 --2- 172.21.15.122:0/1938144045 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f0f8c09f3c0 0x7f0f8c0a1870 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:11.953 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:43:11.950+0000 7f0fa19a2700 1 --2- 172.21.15.122:0/1938144045 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0f9c101910 0x7f0f9c19d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:11.953 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:43:11.950+0000 7f0fa19a2700 1 --2- 172.21.15.122:0/1938144045 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0f9c100f60 0x7f0f9c1961b0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:11.954 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:43:11.950+0000 7f0fa19a2700 1 -- 172.21.15.122:0/1938144045 >> 172.21.15.122:0/1938144045 conn(0x7f0f9c0fb3f0 msgr2=0x7f0f9c0fd5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:43:11.954 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:43:11.950+0000 7f0fa19a2700 1 -- 172.21.15.122:0/1938144045 shutdown_connections 2023-03-09T14:43:11.954 INFO:teuthology.orchestra.run.smithi122.stderr:2023-03-09T14:43:11.950+0000 7f0fa19a2700 1 -- 172.21.15.122:0/1938144045 wait complete. 2023-03-09T14:43:12.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:11 smithi122 ceph-mon[93869]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-03-09T14:43:12.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:11 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:12.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:11 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:12.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:11 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:12.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:11 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:12.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:11 smithi181 ceph-mon[94189]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-03-09T14:43:12.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:11 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:12.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:11 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:12.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:11 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:12.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:11 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:12.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:11 smithi038 ceph-mon[92422]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-03-09T14:43:12.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:11 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:12.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:11 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:12.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:11 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:12.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:11 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:12.423 DEBUG:teuthology.orchestra.run.smithi122:osd.10> sudo journalctl -f -n 0 -u ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.10.service 2023-03-09T14:43:12.425 INFO:tasks.cephadm:Deploying osd.11 on smithi181 with /dev/vg_nvme/lv_1... 2023-03-09T14:43:12.425 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-03-09T14:43:12.458 INFO:journalctl@ceph.osd.10.smithi122.stdout:-- Logs begin at Thu 2023-03-09 14:25:09 UTC. -- 2023-03-09T14:43:13.077 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.c/config 2023-03-09T14:43:13.228 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 09 14:43:12 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-10[115379]: 2023-03-09T14:43:12.955+0000 7f35bba91640 -1 osd.10 0 log_to_monitors true 2023-03-09T14:43:13.587 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:13 smithi181 ceph-mon[94189]: from='osd.10 [v2:172.21.15.122:6824/1427785280,v1:172.21.15.122:6825/1427785280]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-03-09T14:43:13.587 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:13 smithi181 ceph-mon[94189]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-03-09T14:43:13.591 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:13 smithi122 ceph-mon[93869]: from='osd.10 [v2:172.21.15.122:6824/1427785280,v1:172.21.15.122:6825/1427785280]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-03-09T14:43:13.591 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:13 smithi122 ceph-mon[93869]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-03-09T14:43:13.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:13 smithi038 ceph-mon[92422]: from='osd.10 [v2:172.21.15.122:6824/1427785280,v1:172.21.15.122:6825/1427785280]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-03-09T14:43:13.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:13 smithi038 ceph-mon[92422]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-03-09T14:43:14.609 INFO:teuthology.orchestra.run.smithi181.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-03-09T14:43:14.609 INFO:teuthology.orchestra.run.smithi181.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-03-09T14:43:14.609 INFO:teuthology.orchestra.run.smithi181.stderr: stderr: 10+0 records in 2023-03-09T14:43:14.609 INFO:teuthology.orchestra.run.smithi181.stderr:10+0 records out 2023-03-09T14:43:14.610 INFO:teuthology.orchestra.run.smithi181.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0145622 s, 720 MB/s 2023-03-09T14:43:14.610 INFO:teuthology.orchestra.run.smithi181.stderr: stderr: 2023-03-09T14:43:14.610 INFO:teuthology.orchestra.run.smithi181.stderr:--> Zapping successful for: 2023-03-09T14:43:14.645 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:14 smithi181 ceph-mon[94189]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-03-09T14:43:14.646 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:14 smithi181 ceph-mon[94189]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-03-09T14:43:14.646 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:14 smithi181 ceph-mon[94189]: osdmap e68: 11 total, 10 up, 11 in 2023-03-09T14:43:14.646 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:14 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-03-09T14:43:14.646 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:14 smithi181 ceph-mon[94189]: from='osd.10 [v2:172.21.15.122:6824/1427785280,v1:172.21.15.122:6825/1427785280]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-03-09T14:43:14.646 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:14 smithi181 ceph-mon[94189]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-03-09T14:43:14.697 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 09 14:43:14 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-10[115379]: 2023-03-09T14:43:14.288+0000 7f35afa16700 -1 osd.10 0 waiting for initial osdmap 2023-03-09T14:43:14.697 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 09 14:43:14 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-10[115379]: 2023-03-09T14:43:14.305+0000 7f35a9807700 -1 osd.10 69 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-09T14:43:14.697 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:14 smithi122 ceph-mon[93869]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-03-09T14:43:14.698 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:14 smithi122 ceph-mon[93869]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-03-09T14:43:14.698 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:14 smithi122 ceph-mon[93869]: osdmap e68: 11 total, 10 up, 11 in 2023-03-09T14:43:14.698 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:14 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-03-09T14:43:14.698 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:14 smithi122 ceph-mon[93869]: from='osd.10 [v2:172.21.15.122:6824/1427785280,v1:172.21.15.122:6825/1427785280]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-03-09T14:43:14.698 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:14 smithi122 ceph-mon[93869]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-03-09T14:43:14.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:14 smithi038 ceph-mon[92422]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-03-09T14:43:14.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:14 smithi038 ceph-mon[92422]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-03-09T14:43:14.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:14 smithi038 ceph-mon[92422]: osdmap e68: 11 total, 10 up, 11 in 2023-03-09T14:43:14.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:14 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-03-09T14:43:14.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:14 smithi038 ceph-mon[92422]: from='osd.10 [v2:172.21.15.122:6824/1427785280,v1:172.21.15.122:6825/1427785280]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-03-09T14:43:14.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:14 smithi038 ceph-mon[92422]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-03-09T14:43:15.017 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph orch daemon add osd smithi181:vg_nvme/lv_1 2023-03-09T14:43:15.656 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.c/config 2023-03-09T14:43:15.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:15 smithi122 ceph-mon[93869]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2023-03-09T14:43:15.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:15 smithi122 ceph-mon[93869]: osdmap e69: 11 total, 10 up, 11 in 2023-03-09T14:43:15.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:15 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-03-09T14:43:15.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:15 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-03-09T14:43:15.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:15 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:15.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:15 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:15.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:15 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:43:15.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:15 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:43:15.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:15 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:43:15.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:15 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:43:15.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:15 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:15.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:15 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:43:15.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:15 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:43:15.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:15 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:15.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:15 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:43:15.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:15 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:43:15.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:15 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:43:15.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:15 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:15.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:15 smithi122 ceph-mon[93869]: osd.10 [v2:172.21.15.122:6824/1427785280,v1:172.21.15.122:6825/1427785280] boot 2023-03-09T14:43:15.731 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:15 smithi122 ceph-mon[93869]: osdmap e70: 11 total, 11 up, 11 in 2023-03-09T14:43:15.731 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:15 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-03-09T14:43:15.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:15 smithi181 ceph-mon[94189]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2023-03-09T14:43:15.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:15 smithi181 ceph-mon[94189]: osdmap e69: 11 total, 10 up, 11 in 2023-03-09T14:43:15.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:15 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-03-09T14:43:15.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:15 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-03-09T14:43:15.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:15 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:15.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:15 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:15.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:15 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:43:15.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:15 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:43:15.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:15 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:43:15.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:15 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:43:15.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:15 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:15.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:15 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:43:15.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:15 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:43:15.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:15 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:15.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:15 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:43:15.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:15 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:43:15.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:15 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:43:15.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:15 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:15.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:15 smithi181 ceph-mon[94189]: osd.10 [v2:172.21.15.122:6824/1427785280,v1:172.21.15.122:6825/1427785280] boot 2023-03-09T14:43:15.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:15 smithi181 ceph-mon[94189]: osdmap e70: 11 total, 11 up, 11 in 2023-03-09T14:43:15.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:15 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-03-09T14:43:15.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:15 smithi038 ceph-mon[92422]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]': finished 2023-03-09T14:43:15.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:15 smithi038 ceph-mon[92422]: osdmap e69: 11 total, 10 up, 11 in 2023-03-09T14:43:15.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:15 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-03-09T14:43:15.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:15 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-03-09T14:43:15.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:15 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:15.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:15 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:15.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:15 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:43:15.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:15 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:43:15.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:15 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:43:15.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:15 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:43:15.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:15 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:15.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:15 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:43:15.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:15 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:43:15.753 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:15 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:15.753 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:15 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:43:15.753 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:15 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:43:15.753 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:15 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:43:15.753 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:15 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:15.753 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:15 smithi038 ceph-mon[92422]: osd.10 [v2:172.21.15.122:6824/1427785280,v1:172.21.15.122:6825/1427785280] boot 2023-03-09T14:43:15.753 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:15 smithi038 ceph-mon[92422]: osdmap e70: 11 total, 11 up, 11 in 2023-03-09T14:43:15.753 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:15 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-03-09T14:43:16.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:16 smithi181 ceph-mon[94189]: purged_snaps scrub starts 2023-03-09T14:43:16.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:16 smithi181 ceph-mon[94189]: purged_snaps scrub ok 2023-03-09T14:43:16.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:16 smithi181 ceph-mon[94189]: Detected new or changed devices on smithi122 2023-03-09T14:43:16.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:16 smithi181 ceph-mon[94189]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-03-09T14:43:16.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:16 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 4292M 2023-03-09T14:43:16.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:16 smithi181 ceph-mon[94189]: osdmap e71: 11 total, 11 up, 11 in 2023-03-09T14:43:16.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:16 smithi122 ceph-mon[93869]: purged_snaps scrub starts 2023-03-09T14:43:16.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:16 smithi122 ceph-mon[93869]: purged_snaps scrub ok 2023-03-09T14:43:16.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:16 smithi122 ceph-mon[93869]: Detected new or changed devices on smithi122 2023-03-09T14:43:16.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:16 smithi122 ceph-mon[93869]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-03-09T14:43:16.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:16 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 4292M 2023-03-09T14:43:16.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:16 smithi122 ceph-mon[93869]: osdmap e71: 11 total, 11 up, 11 in 2023-03-09T14:43:16.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:16 smithi038 ceph-mon[92422]: purged_snaps scrub starts 2023-03-09T14:43:16.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:16 smithi038 ceph-mon[92422]: purged_snaps scrub ok 2023-03-09T14:43:16.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:16 smithi038 ceph-mon[92422]: Detected new or changed devices on smithi122 2023-03-09T14:43:16.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:16 smithi038 ceph-mon[92422]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 61 MiB used, 894 GiB / 894 GiB avail 2023-03-09T14:43:16.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:16 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 4292M 2023-03-09T14:43:16.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:16 smithi038 ceph-mon[92422]: osdmap e71: 11 total, 11 up, 11 in 2023-03-09T14:43:16.842 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:16.839+0000 7f437fd50700 1 -- 172.21.15.181:0/2912555595 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f43780ffb70 msgr2=0x7f4378107440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:43:16.842 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:16.839+0000 7f437fd50700 1 --2- 172.21.15.181:0/2912555595 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f43780ffb70 0x7f4378107440 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f437000b0a0 tx=0x7f437000b3b0 comp rx=0 tx=0).stop 2023-03-09T14:43:16.843 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:16.839+0000 7f437fd50700 1 -- 172.21.15.181:0/2912555595 shutdown_connections 2023-03-09T14:43:16.843 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:16.839+0000 7f437fd50700 1 --2- 172.21.15.181:0/2912555595 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f43781004b0 0x7f43781008c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:16.843 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:16.839+0000 7f437fd50700 1 --2- 172.21.15.181:0/2912555595 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f43780ffb70 0x7f4378107440 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:16.843 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:16.839+0000 7f437fd50700 1 --2- 172.21.15.181:0/2912555595 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f43780ff1e0 0x7f43780ff630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:16.843 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:16.839+0000 7f437fd50700 1 -- 172.21.15.181:0/2912555595 >> 172.21.15.181:0/2912555595 conn(0x7f43780f9cc0 msgr2=0x7f43780fc0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:43:16.843 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:16.839+0000 7f437fd50700 1 -- 172.21.15.181:0/2912555595 shutdown_connections 2023-03-09T14:43:16.843 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:16.839+0000 7f437fd50700 1 -- 172.21.15.181:0/2912555595 wait complete. 2023-03-09T14:43:16.844 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:16.840+0000 7f437fd50700 1 Processor -- start 2023-03-09T14:43:16.844 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:16.840+0000 7f437fd50700 1 -- start start 2023-03-09T14:43:16.845 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:16.840+0000 7f437fd50700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f43780ff1e0 0x7f4378194d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:43:16.846 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:16.840+0000 7f437fd50700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f43780ffb70 0x7f4378195240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:43:16.846 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:16.840+0000 7f437fd50700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f43781004b0 0x7f437819c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:43:16.846 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:16.840+0000 7f437fd50700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f437819c840 con 0x7f43781004b0 2023-03-09T14:43:16.846 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:16.840+0000 7f437fd50700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f437819c9b0 con 0x7f43780ff1e0 2023-03-09T14:43:16.846 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:16.840+0000 7f437fd50700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f437819cb70 con 0x7f43780ffb70 2023-03-09T14:43:16.846 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:16.840+0000 7f437d2eb700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f43780ffb70 0x7f4378195240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:43:16.847 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:16.840+0000 7f437d2eb700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f43780ffb70 0x7f4378195240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:50832/0 (socket says 172.21.15.181:50832) 2023-03-09T14:43:16.847 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:16.840+0000 7f437d2eb700 1 -- 172.21.15.181:0/2705208913 learned_addr learned my addr 172.21.15.181:0/2705208913 (peer_addr_for_me v2:172.21.15.181:0/0) 2023-03-09T14:43:16.847 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:16.840+0000 7f437daec700 1 --2- 172.21.15.181:0/2705208913 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f43780ff1e0 0x7f4378194d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:43:16.847 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:16.840+0000 7f437e2ed700 1 --2- 172.21.15.181:0/2705208913 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f43781004b0 0x7f437819c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:43:16.847 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:16.840+0000 7f437d2eb700 1 -- 172.21.15.181:0/2705208913 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f43780ff1e0 msgr2=0x7f4378194d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:43:16.847 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:16.840+0000 7f437d2eb700 1 --2- 172.21.15.181:0/2705208913 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f43780ff1e0 0x7f4378194d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:16.847 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:16.840+0000 7f437d2eb700 1 -- 172.21.15.181:0/2705208913 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f43781004b0 msgr2=0x7f437819c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:43:16.847 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:16.840+0000 7f437d2eb700 1 --2- 172.21.15.181:0/2705208913 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f43781004b0 0x7f437819c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:16.848 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:16.840+0000 7f437d2eb700 1 -- 172.21.15.181:0/2705208913 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4370009d00 con 0x7f43780ffb70 2023-03-09T14:43:16.848 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:16.840+0000 7f437d2eb700 1 --2- 172.21.15.181:0/2705208913 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f43780ffb70 0x7f4378195240 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f437400d950 tx=0x7f437400dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:43:16.848 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:16.841+0000 7f436effd700 1 -- 172.21.15.181:0/2705208913 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4374009940 con 0x7f43780ffb70 2023-03-09T14:43:16.848 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:16.841+0000 7f436effd700 1 -- 172.21.15.181:0/2705208913 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4374010460 con 0x7f43780ffb70 2023-03-09T14:43:16.848 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:16.841+0000 7f436effd700 1 -- 172.21.15.181:0/2705208913 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f437400f620 con 0x7f43780ffb70 2023-03-09T14:43:16.848 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:16.841+0000 7f437fd50700 1 -- 172.21.15.181:0/2705208913 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f437810a880 con 0x7f43780ffb70 2023-03-09T14:43:16.848 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:16.841+0000 7f437fd50700 1 -- 172.21.15.181:0/2705208913 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f437810a6f0 con 0x7f43780ffb70 2023-03-09T14:43:16.849 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:16.842+0000 7f437fd50700 1 -- 172.21.15.181:0/2705208913 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43781064b0 con 0x7f43780ffb70 2023-03-09T14:43:16.849 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:16.843+0000 7f436effd700 1 -- 172.21.15.181:0/2705208913 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 16) v1 ==== 133198+0+0 (secure 0 0 0) 0x7f4374022070 con 0x7f43780ffb70 2023-03-09T14:43:16.849 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:16.843+0000 7f436effd700 1 --2- 172.21.15.181:0/2705208913 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f436409f440 0x7f43640a18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:43:16.850 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:16.843+0000 7f436effd700 1 -- 172.21.15.181:0/2705208913 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6723+0+0 (secure 0 0 0) 0x7f43740c9420 con 0x7f43780ffb70 2023-03-09T14:43:16.850 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:16.843+0000 7f437daec700 1 --2- 172.21.15.181:0/2705208913 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f436409f440 0x7f43640a18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:43:16.850 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:16.844+0000 7f437daec700 1 --2- 172.21.15.181:0/2705208913 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f436409f440 0x7f43640a18f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f43781961d0 tx=0x7f436800a3b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:43:16.850 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:16.846+0000 7f436effd700 1 -- 172.21.15.181:0/2705208913 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f4374093e60 con 0x7f43780ffb70 2023-03-09T14:43:17.042 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:17.039+0000 7f437fd50700 1 -- 172.21.15.181:0/2705208913 --> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f437802cef0 con 0x7f436409f440 2023-03-09T14:43:17.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:17 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-03-09T14:43:17.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:17 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-03-09T14:43:17.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:17 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:43:17.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:17 smithi122 ceph-mon[93869]: osdmap e72: 11 total, 11 up, 11 in 2023-03-09T14:43:17.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:17 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-03-09T14:43:17.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:17 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-03-09T14:43:17.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:17 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:43:17.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:17 smithi181 ceph-mon[94189]: osdmap e72: 11 total, 11 up, 11 in 2023-03-09T14:43:17.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:17 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-03-09T14:43:17.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:17 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-03-09T14:43:17.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:17 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:43:17.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:17 smithi038 ceph-mon[92422]: osdmap e72: 11 total, 11 up, 11 in 2023-03-09T14:43:18.591 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:18 smithi181 ceph-mon[94189]: pgmap v212: 1 pgs: 1 remapped+peering; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2023-03-09T14:43:18.592 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:18 smithi181 ceph-mon[94189]: from='client.24349 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:43:18.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:18 smithi122 ceph-mon[93869]: pgmap v212: 1 pgs: 1 remapped+peering; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2023-03-09T14:43:18.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:18 smithi122 ceph-mon[93869]: from='client.24349 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:43:18.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:18 smithi038 ceph-mon[92422]: pgmap v212: 1 pgs: 1 remapped+peering; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2023-03-09T14:43:18.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:18 smithi038 ceph-mon[92422]: from='client.24349 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi181:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:43:19.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:19 smithi122 ceph-mon[93869]: osdmap e73: 11 total, 11 up, 11 in 2023-03-09T14:43:19.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:19 smithi122 ceph-mon[93869]: osdmap e74: 11 total, 11 up, 11 in 2023-03-09T14:43:19.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:19 smithi181 ceph-mon[94189]: osdmap e73: 11 total, 11 up, 11 in 2023-03-09T14:43:19.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:19 smithi181 ceph-mon[94189]: osdmap e74: 11 total, 11 up, 11 in 2023-03-09T14:43:19.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:19 smithi038 ceph-mon[92422]: osdmap e73: 11 total, 11 up, 11 in 2023-03-09T14:43:19.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:19 smithi038 ceph-mon[92422]: osdmap e74: 11 total, 11 up, 11 in 2023-03-09T14:43:20.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:20 smithi122 ceph-mon[93869]: pgmap v215: 1 pgs: 1 remapped+peering; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2023-03-09T14:43:20.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:20 smithi122 ceph-mon[93869]: from='client.? 172.21.15.181:0/1542327293' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "07444936-eae5-4097-a799-60294cafe80b"}]: dispatch 2023-03-09T14:43:20.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:20 smithi122 ceph-mon[93869]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "07444936-eae5-4097-a799-60294cafe80b"}]: dispatch 2023-03-09T14:43:20.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:20 smithi122 ceph-mon[93869]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "07444936-eae5-4097-a799-60294cafe80b"}]': finished 2023-03-09T14:43:20.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:20 smithi122 ceph-mon[93869]: osdmap e75: 12 total, 11 up, 12 in 2023-03-09T14:43:20.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:20 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-03-09T14:43:20.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:20 smithi122 ceph-mon[93869]: from='client.? 172.21.15.181:0/2384697209' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-03-09T14:43:20.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:20 smithi181 ceph-mon[94189]: pgmap v215: 1 pgs: 1 remapped+peering; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2023-03-09T14:43:20.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:20 smithi181 ceph-mon[94189]: from='client.? 172.21.15.181:0/1542327293' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "07444936-eae5-4097-a799-60294cafe80b"}]: dispatch 2023-03-09T14:43:20.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:20 smithi181 ceph-mon[94189]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "07444936-eae5-4097-a799-60294cafe80b"}]: dispatch 2023-03-09T14:43:20.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:20 smithi181 ceph-mon[94189]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "07444936-eae5-4097-a799-60294cafe80b"}]': finished 2023-03-09T14:43:20.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:20 smithi181 ceph-mon[94189]: osdmap e75: 12 total, 11 up, 12 in 2023-03-09T14:43:20.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:20 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-03-09T14:43:20.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:20 smithi181 ceph-mon[94189]: from='client.? 172.21.15.181:0/2384697209' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-03-09T14:43:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:20 smithi038 ceph-mon[92422]: pgmap v215: 1 pgs: 1 remapped+peering; 577 KiB data, 67 MiB used, 983 GiB / 983 GiB avail 2023-03-09T14:43:20.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:20 smithi038 ceph-mon[92422]: from='client.? 172.21.15.181:0/1542327293' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "07444936-eae5-4097-a799-60294cafe80b"}]: dispatch 2023-03-09T14:43:20.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:20 smithi038 ceph-mon[92422]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "07444936-eae5-4097-a799-60294cafe80b"}]: dispatch 2023-03-09T14:43:20.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:20 smithi038 ceph-mon[92422]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "07444936-eae5-4097-a799-60294cafe80b"}]': finished 2023-03-09T14:43:20.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:20 smithi038 ceph-mon[92422]: osdmap e75: 12 total, 11 up, 12 in 2023-03-09T14:43:20.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:20 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-03-09T14:43:20.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:20 smithi038 ceph-mon[92422]: from='client.? 172.21.15.181:0/2384697209' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-03-09T14:43:22.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:22 smithi122 ceph-mon[93869]: pgmap v218: 1 pgs: 1 remapped+peering; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail 2023-03-09T14:43:22.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:22 smithi181 ceph-mon[94189]: pgmap v218: 1 pgs: 1 remapped+peering; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail 2023-03-09T14:43:22.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:22 smithi038 ceph-mon[92422]: pgmap v218: 1 pgs: 1 remapped+peering; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail 2023-03-09T14:43:24.652 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:24 smithi181 ceph-mon[94189]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail; 106 KiB/s, 0 objects/s recovering 2023-03-09T14:43:24.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:24 smithi122 ceph-mon[93869]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail; 106 KiB/s, 0 objects/s recovering 2023-03-09T14:43:24.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:24 smithi038 ceph-mon[92422]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail; 106 KiB/s, 0 objects/s recovering 2023-03-09T14:43:26.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:26 smithi122 ceph-mon[93869]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail; 90 KiB/s, 0 objects/s recovering 2023-03-09T14:43:26.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:26 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-03-09T14:43:26.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:26 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:43:26.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:26 smithi181 ceph-mon[94189]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail; 90 KiB/s, 0 objects/s recovering 2023-03-09T14:43:26.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:26 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-03-09T14:43:26.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:26 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:43:26.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:26 smithi038 ceph-mon[92422]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail; 90 KiB/s, 0 objects/s recovering 2023-03-09T14:43:26.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:26 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-03-09T14:43:26.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:26 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:43:27.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:27 smithi181 ceph-mon[94189]: Deploying daemon osd.11 on smithi181 2023-03-09T14:43:27.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:27 smithi122 ceph-mon[93869]: Deploying daemon osd.11 on smithi181 2023-03-09T14:43:27.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:27 smithi038 ceph-mon[92422]: Deploying daemon osd.11 on smithi181 2023-03-09T14:43:28.615 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:28 smithi181 ceph-mon[94189]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-03-09T14:43:28.733 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:28 smithi122 ceph-mon[93869]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-03-09T14:43:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:28 smithi038 ceph-mon[92422]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-03-09T14:43:29.618 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:29 smithi181 ceph-mon[94189]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail; 61 KiB/s, 0 objects/s recovering 2023-03-09T14:43:29.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:29 smithi122 ceph-mon[93869]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail; 61 KiB/s, 0 objects/s recovering 2023-03-09T14:43:29.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:29 smithi038 ceph-mon[92422]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail; 61 KiB/s, 0 objects/s recovering 2023-03-09T14:43:31.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:31 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:31.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:31 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:31.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:31 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:43:31.793 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:31 smithi181 ceph-mon[94189]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail; 52 KiB/s, 0 objects/s recovering 2023-03-09T14:43:31.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:31 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:31.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:31 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:31.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:31 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:43:31.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:31 smithi122 ceph-mon[93869]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail; 52 KiB/s, 0 objects/s recovering 2023-03-09T14:43:32.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:31 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:32.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:31 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:32.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:31 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:43:32.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:31 smithi038 ceph-mon[92422]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail; 52 KiB/s, 0 objects/s recovering 2023-03-09T14:43:33.459 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:33.456+0000 7f436effd700 1 -- 172.21.15.181:0/2705208913 <== mgr.14150 v2:172.21.15.38:6800/3328887328 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f437802cef0 con 0x7f436409f440 2023-03-09T14:43:33.459 INFO:teuthology.orchestra.run.smithi181.stdout:Created osd(s) 11 on host 'smithi181' 2023-03-09T14:43:33.461 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:33.458+0000 7f437fd50700 1 -- 172.21.15.181:0/2705208913 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f436409f440 msgr2=0x7f43640a18f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:43:33.461 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:33.458+0000 7f437fd50700 1 --2- 172.21.15.181:0/2705208913 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f436409f440 0x7f43640a18f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f43781961d0 tx=0x7f436800a3b0 comp rx=0 tx=0).stop 2023-03-09T14:43:33.461 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:33.458+0000 7f437fd50700 1 -- 172.21.15.181:0/2705208913 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f43780ffb70 msgr2=0x7f4378195240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:43:33.461 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:33.458+0000 7f437fd50700 1 --2- 172.21.15.181:0/2705208913 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f43780ffb70 0x7f4378195240 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f437400d950 tx=0x7f437400dc60 comp rx=0 tx=0).stop 2023-03-09T14:43:33.461 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:33.458+0000 7f437fd50700 1 -- 172.21.15.181:0/2705208913 shutdown_connections 2023-03-09T14:43:33.462 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:33.458+0000 7f437fd50700 1 --2- 172.21.15.181:0/2705208913 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f43781004b0 0x7f437819c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:33.462 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:33.458+0000 7f437fd50700 1 --2- 172.21.15.181:0/2705208913 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f436409f440 0x7f43640a18f0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:33.462 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:33.458+0000 7f437fd50700 1 --2- 172.21.15.181:0/2705208913 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f43780ff1e0 0x7f4378194d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:33.462 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:33.458+0000 7f437fd50700 1 --2- 172.21.15.181:0/2705208913 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f43780ffb70 0x7f4378195240 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:33.462 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:33.458+0000 7f437fd50700 1 -- 172.21.15.181:0/2705208913 >> 172.21.15.181:0/2705208913 conn(0x7f43780f9cc0 msgr2=0x7f43781035f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:43:33.462 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:33.458+0000 7f437fd50700 1 -- 172.21.15.181:0/2705208913 shutdown_connections 2023-03-09T14:43:33.462 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:33.458+0000 7f437fd50700 1 -- 172.21.15.181:0/2705208913 wait complete. 2023-03-09T14:43:33.948 DEBUG:teuthology.orchestra.run.smithi181:osd.11> sudo journalctl -f -n 0 -u ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.11.service 2023-03-09T14:43:33.950 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2023-03-09T14:43:33.951 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph osd stat -f json 2023-03-09T14:43:33.980 INFO:journalctl@ceph.osd.11.smithi181.stdout:-- Logs begin at Thu 2023-03-09 14:25:01 UTC. -- 2023-03-09T14:43:34.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:34 smithi038 ceph-mon[92422]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-03-09T14:43:34.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:34 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:34.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:34 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:34.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:34 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:43:34.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:34 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:43:34.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:34 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:34.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:34 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:34.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:34 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:43:34.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:34 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:34.431 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:34 smithi181 ceph-mon[94189]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-03-09T14:43:34.431 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:34 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:34.431 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:34 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:34.431 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:34 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:43:34.431 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:34 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:43:34.432 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:34 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:34.432 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:34 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:34.432 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:34 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:43:34.432 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:34 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:34.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:34 smithi122 ceph-mon[93869]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-03-09T14:43:34.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:34 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:34.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:34 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:34.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:34 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:43:34.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:34 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:43:34.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:34 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:34.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:34 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:34.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:34 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:43:34.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:34 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:34.619 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.a/config 2023-03-09T14:43:34.725 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 09 14:43:34 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-11[116500]: 2023-03-09T14:43:34.569+0000 7ff0a007e640 -1 osd.11 0 log_to_monitors true 2023-03-09T14:43:35.283 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:35 smithi181 ceph-mon[94189]: from='osd.11 [v2:172.21.15.181:6824/3183488640,v1:172.21.15.181:6825/3183488640]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-03-09T14:43:35.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:35 smithi122 ceph-mon[93869]: from='osd.11 [v2:172.21.15.181:6824/3183488640,v1:172.21.15.181:6825/3183488640]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-03-09T14:43:35.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:35 smithi038 ceph-mon[92422]: from='osd.11 [v2:172.21.15.181:6824/3183488640,v1:172.21.15.181:6825/3183488640]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-03-09T14:43:35.817 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:35.813+0000 7fe7bbc3a700 1 -- 172.21.15.38:0/3960094734 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7b40feea0 msgr2=0x7fe7b40ff2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:43:35.817 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:35.813+0000 7fe7bbc3a700 1 --2- 172.21.15.38:0/3960094734 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7b40feea0 0x7fe7b40ff2b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe7b0009990 tx=0x7fe7b0009ca0 comp rx=0 tx=0).stop 2023-03-09T14:43:35.818 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:35.813+0000 7fe7bbc3a700 1 -- 172.21.15.38:0/3960094734 shutdown_connections 2023-03-09T14:43:35.818 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:35.813+0000 7fe7bbc3a700 1 --2- 172.21.15.38:0/3960094734 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7b40feea0 0x7fe7b40ff2b0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:35.818 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:35.813+0000 7fe7bbc3a700 1 --2- 172.21.15.38:0/3960094734 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe7b41009a0 0x7fe7b41072d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:35.818 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:35.813+0000 7fe7bbc3a700 1 --2- 172.21.15.38:0/3960094734 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe7b40ffff0 0x7fe7b4100460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:35.818 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:35.813+0000 7fe7bbc3a700 1 -- 172.21.15.38:0/3960094734 >> 172.21.15.38:0/3960094734 conn(0x7fe7b40fa480 msgr2=0x7fe7b40fc8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:43:35.818 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:35.813+0000 7fe7bbc3a700 1 -- 172.21.15.38:0/3960094734 shutdown_connections 2023-03-09T14:43:35.818 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:35.813+0000 7fe7bbc3a700 1 -- 172.21.15.38:0/3960094734 wait complete. 2023-03-09T14:43:35.818 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:35.814+0000 7fe7bbc3a700 1 Processor -- start 2023-03-09T14:43:35.819 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:35.814+0000 7fe7bbc3a700 1 -- start start 2023-03-09T14:43:35.820 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:35.814+0000 7fe7bbc3a700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe7b40feea0 0x7fe7b4102e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:43:35.820 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:35.814+0000 7fe7bbc3a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7b40ffff0 0x7fe7b4101460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:43:35.821 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:35.814+0000 7fe7bbc3a700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe7b41009a0 0x7fe7b41019a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:43:35.821 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:35.814+0000 7fe7bbc3a700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b4102140 con 0x7fe7b40ffff0 2023-03-09T14:43:35.821 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:35.814+0000 7fe7bbc3a700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b41022b0 con 0x7fe7b41009a0 2023-03-09T14:43:35.821 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:35.814+0000 7fe7bbc3a700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b419fd50 con 0x7fe7b40feea0 2023-03-09T14:43:35.821 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:35.814+0000 7fe7b91d5700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7b40ffff0 0x7fe7b4101460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:43:35.821 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:35.814+0000 7fe7b91d5700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7b40ffff0 0x7fe7b4101460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36286/0 (socket says 172.21.15.38:36286) 2023-03-09T14:43:35.821 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:35.815+0000 7fe7b91d5700 1 -- 172.21.15.38:0/204581889 learned_addr learned my addr 172.21.15.38:0/204581889 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:43:35.822 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:35.815+0000 7fe7b99d6700 1 --2- 172.21.15.38:0/204581889 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe7b40feea0 0x7fe7b4102e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:43:35.822 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:35.815+0000 7fe7ba1d7700 1 --2- 172.21.15.38:0/204581889 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe7b41009a0 0x7fe7b41019a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:43:35.822 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:35.815+0000 7fe7b91d5700 1 -- 172.21.15.38:0/204581889 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe7b40feea0 msgr2=0x7fe7b4102e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:43:35.822 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:35.815+0000 7fe7b91d5700 1 --2- 172.21.15.38:0/204581889 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe7b40feea0 0x7fe7b4102e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:35.822 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:35.815+0000 7fe7b91d5700 1 -- 172.21.15.38:0/204581889 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe7b41009a0 msgr2=0x7fe7b41019a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:43:35.822 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:35.815+0000 7fe7b91d5700 1 --2- 172.21.15.38:0/204581889 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe7b41009a0 0x7fe7b41019a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:35.822 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:35.815+0000 7fe7b91d5700 1 -- 172.21.15.38:0/204581889 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7b00095f0 con 0x7fe7b40ffff0 2023-03-09T14:43:35.822 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:35.815+0000 7fe7ba1d7700 1 --2- 172.21.15.38:0/204581889 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe7b41009a0 0x7fe7b41019a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:43:35.823 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:35.815+0000 7fe7b91d5700 1 --2- 172.21.15.38:0/204581889 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7b40ffff0 0x7fe7b4101460 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe7a400e9a0 tx=0x7fe7a400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:43:35.823 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:35.815+0000 7fe7aaffd700 1 -- 172.21.15.38:0/204581889 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7a4009790 con 0x7fe7b40ffff0 2023-03-09T14:43:35.823 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:35.815+0000 7fe7aaffd700 1 -- 172.21.15.38:0/204581889 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe7a400cbd0 con 0x7fe7b40ffff0 2023-03-09T14:43:35.823 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:35.815+0000 7fe7bbc3a700 1 -- 172.21.15.38:0/204581889 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7b410a800 con 0x7fe7b40ffff0 2023-03-09T14:43:35.823 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:35.815+0000 7fe7aaffd700 1 -- 172.21.15.38:0/204581889 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7a4018680 con 0x7fe7b40ffff0 2023-03-09T14:43:35.823 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:35.815+0000 7fe7bbc3a700 1 -- 172.21.15.38:0/204581889 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7b410a670 con 0x7fe7b40ffff0 2023-03-09T14:43:35.823 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:35.817+0000 7fe7bbc3a700 1 -- 172.21.15.38:0/204581889 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7b4066b10 con 0x7fe7b40ffff0 2023-03-09T14:43:35.823 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:35.817+0000 7fe7aaffd700 1 -- 172.21.15.38:0/204581889 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 133198+0+0 (secure 0 0 0) 0x7fe7a400c040 con 0x7fe7b40ffff0 2023-03-09T14:43:35.824 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:35.818+0000 7fe7aaffd700 1 --2- 172.21.15.38:0/204581889 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fe7a009f440 0x7fe7a00a18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:43:35.824 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:35.818+0000 7fe7b99d6700 1 --2- 172.21.15.38:0/204581889 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fe7a009f440 0x7fe7a00a18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:43:35.824 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:35.818+0000 7fe7aaffd700 1 -- 172.21.15.38:0/204581889 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 6831+0+0 (secure 0 0 0) 0x7fe7a4014070 con 0x7fe7b40ffff0 2023-03-09T14:43:35.825 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:35.818+0000 7fe7b99d6700 1 --2- 172.21.15.38:0/204581889 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fe7a009f440 0x7fe7a00a18f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe7b000b2e0 tx=0x7fe7b0005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:43:35.825 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:35.822+0000 7fe7aaffd700 1 -- 172.21.15.38:0/204581889 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7fe7a40515b0 con 0x7fe7b40ffff0 2023-03-09T14:43:36.024 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:36.020+0000 7fe7bbc3a700 1 -- 172.21.15.38:0/204581889 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fe7b41a0820 con 0x7fe7b40ffff0 2023-03-09T14:43:36.024 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:36.021+0000 7fe7aaffd700 1 -- 172.21.15.38:0/204581889 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v76) v1 ==== 74+0+133 (secure 0 0 0) 0x7fe7a4094370 con 0x7fe7b40ffff0 2023-03-09T14:43:36.024 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:43:36.026 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:36.023+0000 7fe7bbc3a700 1 -- 172.21.15.38:0/204581889 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fe7a009f440 msgr2=0x7fe7a00a18f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:43:36.026 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:36.023+0000 7fe7bbc3a700 1 --2- 172.21.15.38:0/204581889 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fe7a009f440 0x7fe7a00a18f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe7b000b2e0 tx=0x7fe7b0005fb0 comp rx=0 tx=0).stop 2023-03-09T14:43:36.026 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:36.023+0000 7fe7bbc3a700 1 -- 172.21.15.38:0/204581889 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7b40ffff0 msgr2=0x7fe7b4101460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:43:36.026 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:36.023+0000 7fe7bbc3a700 1 --2- 172.21.15.38:0/204581889 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7b40ffff0 0x7fe7b4101460 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe7a400e9a0 tx=0x7fe7a400ecb0 comp rx=0 tx=0).stop 2023-03-09T14:43:36.027 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:36.023+0000 7fe7bbc3a700 1 -- 172.21.15.38:0/204581889 shutdown_connections 2023-03-09T14:43:36.027 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:36.023+0000 7fe7bbc3a700 1 --2- 172.21.15.38:0/204581889 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fe7a009f440 0x7fe7a00a18f0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:36.027 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:36.023+0000 7fe7bbc3a700 1 --2- 172.21.15.38:0/204581889 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe7b41009a0 0x7fe7b41019a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:36.027 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:36.023+0000 7fe7bbc3a700 1 --2- 172.21.15.38:0/204581889 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe7b40feea0 0x7fe7b4102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:36.027 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:36.023+0000 7fe7bbc3a700 1 --2- 172.21.15.38:0/204581889 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7b40ffff0 0x7fe7b4101460 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:36.027 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:36.023+0000 7fe7bbc3a700 1 -- 172.21.15.38:0/204581889 >> 172.21.15.38:0/204581889 conn(0x7fe7b40fa480 msgr2=0x7fe7b40fad10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:43:36.027 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:36.023+0000 7fe7bbc3a700 1 -- 172.21.15.38:0/204581889 shutdown_connections 2023-03-09T14:43:36.028 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:36.023+0000 7fe7bbc3a700 1 -- 172.21.15.38:0/204581889 wait complete. 2023-03-09T14:43:36.365 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:36 smithi038 ceph-mon[92422]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail 2023-03-09T14:43:36.366 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:36 smithi038 ceph-mon[92422]: from='osd.11 [v2:172.21.15.181:6824/3183488640,v1:172.21.15.181:6825/3183488640]' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-03-09T14:43:36.366 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:36 smithi038 ceph-mon[92422]: osdmap e76: 12 total, 11 up, 12 in 2023-03-09T14:43:36.366 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:36 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-03-09T14:43:36.366 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:36 smithi038 ceph-mon[92422]: from='osd.11 [v2:172.21.15.181:6824/3183488640,v1:172.21.15.181:6825/3183488640]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2023-03-09T14:43:36.366 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:36 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/204581889' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-03-09T14:43:36.366 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:36 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:36.367 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":76,"num_osds":12,"num_up_osds":11,"osd_up_since":1678372995,"num_in_osds":12,"osd_in_since":1678372999,"num_remapped_pgs":0} 2023-03-09T14:43:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:36 smithi122 ceph-mon[93869]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail 2023-03-09T14:43:36.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:36 smithi122 ceph-mon[93869]: from='osd.11 [v2:172.21.15.181:6824/3183488640,v1:172.21.15.181:6825/3183488640]' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-03-09T14:43:36.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:36 smithi122 ceph-mon[93869]: osdmap e76: 12 total, 11 up, 12 in 2023-03-09T14:43:36.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:36 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-03-09T14:43:36.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:36 smithi122 ceph-mon[93869]: from='osd.11 [v2:172.21.15.181:6824/3183488640,v1:172.21.15.181:6825/3183488640]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2023-03-09T14:43:36.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:36 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/204581889' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-03-09T14:43:36.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:36 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:36.483 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 09 14:43:36 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-11[116500]: 2023-03-09T14:43:36.176+0000 7ff094003700 -1 osd.11 0 waiting for initial osdmap 2023-03-09T14:43:36.483 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 09 14:43:36 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-11[116500]: 2023-03-09T14:43:36.193+0000 7ff08fdf8700 -1 osd.11 77 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-09T14:43:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:36 smithi181 ceph-mon[94189]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail 2023-03-09T14:43:36.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:36 smithi181 ceph-mon[94189]: from='osd.11 [v2:172.21.15.181:6824/3183488640,v1:172.21.15.181:6825/3183488640]' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-03-09T14:43:36.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:36 smithi181 ceph-mon[94189]: osdmap e76: 12 total, 11 up, 12 in 2023-03-09T14:43:36.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:36 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-03-09T14:43:36.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:36 smithi181 ceph-mon[94189]: from='osd.11 [v2:172.21.15.181:6824/3183488640,v1:172.21.15.181:6825/3183488640]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2023-03-09T14:43:36.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:36 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/204581889' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-03-09T14:43:36.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:36 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:37.369 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph osd stat -f json 2023-03-09T14:43:37.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:37 smithi122 ceph-mon[93869]: Detected new or changed devices on smithi181 2023-03-09T14:43:37.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:37 smithi122 ceph-mon[93869]: from='osd.11 [v2:172.21.15.181:6824/3183488640,v1:172.21.15.181:6825/3183488640]' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2023-03-09T14:43:37.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:37 smithi122 ceph-mon[93869]: osdmap e77: 12 total, 11 up, 12 in 2023-03-09T14:43:37.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:37 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-03-09T14:43:37.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:37 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-03-09T14:43:37.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:37 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:37.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:37 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:43:37.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:37 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:43:37.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:37 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:43:37.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:37 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:43:37.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:37 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 4292M 2023-03-09T14:43:37.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:37 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:37.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:37 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:43:37.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:37 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:43:37.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:37 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:37.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:37 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-03-09T14:43:37.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:37 smithi181 ceph-mon[94189]: Detected new or changed devices on smithi181 2023-03-09T14:43:37.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:37 smithi181 ceph-mon[94189]: from='osd.11 [v2:172.21.15.181:6824/3183488640,v1:172.21.15.181:6825/3183488640]' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2023-03-09T14:43:37.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:37 smithi181 ceph-mon[94189]: osdmap e77: 12 total, 11 up, 12 in 2023-03-09T14:43:37.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:37 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-03-09T14:43:37.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:37 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-03-09T14:43:37.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:37 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:37.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:37 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:43:37.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:37 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:43:37.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:37 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:43:37.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:37 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:43:37.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:37 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 4292M 2023-03-09T14:43:37.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:37 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:37.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:37 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:43:37.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:37 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:43:37.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:37 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:37.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:37 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-03-09T14:43:37.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:37 smithi038 ceph-mon[92422]: Detected new or changed devices on smithi181 2023-03-09T14:43:37.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:37 smithi038 ceph-mon[92422]: from='osd.11 [v2:172.21.15.181:6824/3183488640,v1:172.21.15.181:6825/3183488640]' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]': finished 2023-03-09T14:43:37.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:37 smithi038 ceph-mon[92422]: osdmap e77: 12 total, 11 up, 12 in 2023-03-09T14:43:37.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:37 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-03-09T14:43:37.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:37 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-03-09T14:43:37.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:37 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:37.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:37 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:43:37.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:37 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:43:37.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:37 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:43:37.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:37 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:43:37.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:37 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 4292M 2023-03-09T14:43:37.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:37 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:37.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:37 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:43:37.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:37 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:43:37.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:37 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:37.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:37 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-03-09T14:43:38.038 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.a/config 2023-03-09T14:43:38.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:38 smithi122 ceph-mon[93869]: purged_snaps scrub starts 2023-03-09T14:43:38.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:38 smithi122 ceph-mon[93869]: purged_snaps scrub ok 2023-03-09T14:43:38.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:38 smithi122 ceph-mon[93869]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail 2023-03-09T14:43:38.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:38 smithi122 ceph-mon[93869]: osd.11 [v2:172.21.15.181:6824/3183488640,v1:172.21.15.181:6825/3183488640] boot 2023-03-09T14:43:38.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:38 smithi122 ceph-mon[93869]: osdmap e78: 12 total, 12 up, 12 in 2023-03-09T14:43:38.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:38 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-03-09T14:43:38.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:38 smithi181 ceph-mon[94189]: purged_snaps scrub starts 2023-03-09T14:43:38.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:38 smithi181 ceph-mon[94189]: purged_snaps scrub ok 2023-03-09T14:43:38.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:38 smithi181 ceph-mon[94189]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail 2023-03-09T14:43:38.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:38 smithi181 ceph-mon[94189]: osd.11 [v2:172.21.15.181:6824/3183488640,v1:172.21.15.181:6825/3183488640] boot 2023-03-09T14:43:38.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:38 smithi181 ceph-mon[94189]: osdmap e78: 12 total, 12 up, 12 in 2023-03-09T14:43:38.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:38 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-03-09T14:43:38.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:38 smithi038 ceph-mon[92422]: purged_snaps scrub starts 2023-03-09T14:43:38.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:38 smithi038 ceph-mon[92422]: purged_snaps scrub ok 2023-03-09T14:43:38.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:38 smithi038 ceph-mon[92422]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 68 MiB used, 983 GiB / 983 GiB avail 2023-03-09T14:43:38.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:38 smithi038 ceph-mon[92422]: osd.11 [v2:172.21.15.181:6824/3183488640,v1:172.21.15.181:6825/3183488640] boot 2023-03-09T14:43:38.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:38 smithi038 ceph-mon[92422]: osdmap e78: 12 total, 12 up, 12 in 2023-03-09T14:43:38.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:38 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-03-09T14:43:39.262 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:39.258+0000 7f10ef974700 1 -- 172.21.15.38:0/536735210 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10e80feea0 msgr2=0x7f10e80ff2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:43:39.262 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:39.258+0000 7f10ef974700 1 --2- 172.21.15.38:0/536735210 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10e80feea0 0x7f10e80ff2b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f10d8009990 tx=0x7f10d8009ca0 comp rx=0 tx=0).stop 2023-03-09T14:43:39.262 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:39.258+0000 7f10ef974700 1 -- 172.21.15.38:0/536735210 shutdown_connections 2023-03-09T14:43:39.262 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:39.258+0000 7f10ef974700 1 --2- 172.21.15.38:0/536735210 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10e80feea0 0x7f10e80ff2b0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:39.262 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:39.258+0000 7f10ef974700 1 --2- 172.21.15.38:0/536735210 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f10e81009a0 0x7f10e81072d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:39.263 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:39.258+0000 7f10ef974700 1 --2- 172.21.15.38:0/536735210 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f10e80ffff0 0x7f10e8100460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:39.263 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:39.258+0000 7f10ef974700 1 -- 172.21.15.38:0/536735210 >> 172.21.15.38:0/536735210 conn(0x7f10e80fa480 msgr2=0x7f10e80fc8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:43:39.263 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:39.258+0000 7f10ef974700 1 -- 172.21.15.38:0/536735210 shutdown_connections 2023-03-09T14:43:39.263 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:39.258+0000 7f10ef974700 1 -- 172.21.15.38:0/536735210 wait complete. 2023-03-09T14:43:39.263 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:39.259+0000 7f10ef974700 1 Processor -- start 2023-03-09T14:43:39.263 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:39.259+0000 7f10ef974700 1 -- start start 2023-03-09T14:43:39.265 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:39.259+0000 7f10ef974700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10e80feea0 0x7f10e8069640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:43:39.265 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:39.259+0000 7f10ef974700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f10e80ffff0 0x7f10e819aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:43:39.265 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:39.259+0000 7f10ef974700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f10e81009a0 0x7f10e819d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:43:39.265 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:39.259+0000 7f10ef974700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10e819d8a0 con 0x7f10e80feea0 2023-03-09T14:43:39.266 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:39.259+0000 7f10ef974700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10e819da10 con 0x7f10e80ffff0 2023-03-09T14:43:39.266 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:39.259+0000 7f10ef974700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10e819dbd0 con 0x7f10e81009a0 2023-03-09T14:43:39.266 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:39.259+0000 7f10ed710700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10e80feea0 0x7f10e8069640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:43:39.266 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:39.259+0000 7f10ed710700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10e80feea0 0x7f10e8069640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36300/0 (socket says 172.21.15.38:36300) 2023-03-09T14:43:39.266 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:39.259+0000 7f10ed710700 1 -- 172.21.15.38:0/1137873211 learned_addr learned my addr 172.21.15.38:0/1137873211 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:43:39.266 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:39.259+0000 7f10ecf0f700 1 --2- 172.21.15.38:0/1137873211 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f10e80ffff0 0x7f10e819aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:43:39.266 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:39.259+0000 7f10ed710700 1 -- 172.21.15.38:0/1137873211 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f10e81009a0 msgr2=0x7f10e819d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:43:39.267 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:39.259+0000 7f10ed710700 1 --2- 172.21.15.38:0/1137873211 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f10e81009a0 0x7f10e819d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:39.267 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:39.259+0000 7f10ed710700 1 -- 172.21.15.38:0/1137873211 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f10e80ffff0 msgr2=0x7f10e819aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:43:39.267 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:39.259+0000 7f10ed710700 1 --2- 172.21.15.38:0/1137873211 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f10e80ffff0 0x7f10e819aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:39.267 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:39.259+0000 7f10ed710700 1 -- 172.21.15.38:0/1137873211 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10d80095f0 con 0x7f10e80feea0 2023-03-09T14:43:39.267 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:39.259+0000 7f10ecf0f700 1 --2- 172.21.15.38:0/1137873211 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f10e80ffff0 0x7f10e819aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:43:39.268 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:39.260+0000 7f10ed710700 1 --2- 172.21.15.38:0/1137873211 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10e80feea0 0x7f10e8069640 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f10d8000c00 tx=0x7f10d800f710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:43:39.268 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:39.260+0000 7f10de7fc700 1 -- 172.21.15.38:0/1137873211 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10d801d070 con 0x7f10e80feea0 2023-03-09T14:43:39.268 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:39.260+0000 7f10de7fc700 1 -- 172.21.15.38:0/1137873211 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f10d800fc00 con 0x7f10e80feea0 2023-03-09T14:43:39.268 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:39.260+0000 7f10ef974700 1 -- 172.21.15.38:0/1137873211 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10e810a880 con 0x7f10e80feea0 2023-03-09T14:43:39.268 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:39.260+0000 7f10de7fc700 1 -- 172.21.15.38:0/1137873211 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10d8017650 con 0x7f10e80feea0 2023-03-09T14:43:39.268 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:39.260+0000 7f10ef974700 1 -- 172.21.15.38:0/1137873211 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10e810a6f0 con 0x7f10e80feea0 2023-03-09T14:43:39.269 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:39.261+0000 7f10ef974700 1 -- 172.21.15.38:0/1137873211 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10e81063d0 con 0x7f10e80feea0 2023-03-09T14:43:39.269 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:39.263+0000 7f10de7fc700 1 -- 172.21.15.38:0/1137873211 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 133198+0+0 (secure 0 0 0) 0x7f10d8029030 con 0x7f10e80feea0 2023-03-09T14:43:39.269 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:39.263+0000 7f10de7fc700 1 --2- 172.21.15.38:0/1137873211 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f10d409f440 0x7f10d40a18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:43:39.270 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:39.263+0000 7f10de7fc700 1 -- 172.21.15.38:0/1137873211 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f10d80cad40 con 0x7f10e80feea0 2023-03-09T14:43:39.270 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:39.263+0000 7f10ecf0f700 1 --2- 172.21.15.38:0/1137873211 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f10d409f440 0x7f10d40a18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:43:39.270 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:39.264+0000 7f10ecf0f700 1 --2- 172.21.15.38:0/1137873211 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f10d409f440 0x7f10d40a18f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f10e80693b0 tx=0x7f10e4008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:43:39.270 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:39.267+0000 7f10de7fc700 1 -- 172.21.15.38:0/1137873211 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f10d8095680 con 0x7f10e80feea0 2023-03-09T14:43:39.467 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:39.464+0000 7f10ef974700 1 -- 172.21.15.38:0/1137873211 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f10e802cef0 con 0x7f10e80feea0 2023-03-09T14:43:39.467 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:39.464+0000 7f10de7fc700 1 -- 172.21.15.38:0/1137873211 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v79) v1 ==== 74+0+133 (secure 0 0 0) 0x7f10d8098ca0 con 0x7f10e80feea0 2023-03-09T14:43:39.468 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:43:39.469 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:39.466+0000 7f10ef974700 1 -- 172.21.15.38:0/1137873211 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f10d409f440 msgr2=0x7f10d40a18f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:43:39.469 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:39.466+0000 7f10ef974700 1 --2- 172.21.15.38:0/1137873211 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f10d409f440 0x7f10d40a18f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f10e80693b0 tx=0x7f10e4008040 comp rx=0 tx=0).stop 2023-03-09T14:43:39.469 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:39.466+0000 7f10ef974700 1 -- 172.21.15.38:0/1137873211 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10e80feea0 msgr2=0x7f10e8069640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:43:39.470 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:39.466+0000 7f10ef974700 1 --2- 172.21.15.38:0/1137873211 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10e80feea0 0x7f10e8069640 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f10d8000c00 tx=0x7f10d800f710 comp rx=0 tx=0).stop 2023-03-09T14:43:39.470 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:39.466+0000 7f10ef974700 1 -- 172.21.15.38:0/1137873211 shutdown_connections 2023-03-09T14:43:39.470 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:39.466+0000 7f10ef974700 1 --2- 172.21.15.38:0/1137873211 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10e80feea0 0x7f10e8069640 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:39.470 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:39.466+0000 7f10ef974700 1 --2- 172.21.15.38:0/1137873211 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f10d409f440 0x7f10d40a18f0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:39.470 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:39.466+0000 7f10ef974700 1 --2- 172.21.15.38:0/1137873211 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f10e81009a0 0x7f10e819d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:39.470 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:39.466+0000 7f10ef974700 1 --2- 172.21.15.38:0/1137873211 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f10e80ffff0 0x7f10e819aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:39.471 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:39.466+0000 7f10ef974700 1 -- 172.21.15.38:0/1137873211 >> 172.21.15.38:0/1137873211 conn(0x7f10e80fa480 msgr2=0x7f10e8103510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:43:39.471 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:39.466+0000 7f10ef974700 1 -- 172.21.15.38:0/1137873211 shutdown_connections 2023-03-09T14:43:39.471 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:39.466+0000 7f10ef974700 1 -- 172.21.15.38:0/1137873211 wait complete. 2023-03-09T14:43:39.490 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:39 smithi038 ceph-mon[92422]: osdmap e79: 12 total, 12 up, 12 in 2023-03-09T14:43:39.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:39 smithi122 ceph-mon[93869]: osdmap e79: 12 total, 12 up, 12 in 2023-03-09T14:43:39.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:39 smithi181 ceph-mon[94189]: osdmap e79: 12 total, 12 up, 12 in 2023-03-09T14:43:39.896 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":79,"num_osds":12,"num_up_osds":12,"osd_up_since":1678373017,"num_in_osds":12,"osd_in_since":1678372999,"num_remapped_pgs":0} 2023-03-09T14:43:39.897 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph osd dump --format=json 2023-03-09T14:43:40.555 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.a/config 2023-03-09T14:43:40.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:40 smithi122 ceph-mon[93869]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-03-09T14:43:40.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:40 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/1137873211' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-03-09T14:43:40.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:40 smithi181 ceph-mon[94189]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-03-09T14:43:40.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:40 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/1137873211' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-03-09T14:43:40.744 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:40 smithi038 ceph-mon[92422]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 74 MiB used, 1.0 TiB / 1.0 TiB avail 2023-03-09T14:43:40.745 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:40 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/1137873211' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-03-09T14:43:41.765 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:41.759+0000 7ff92370e700 1 -- 172.21.15.38:0/615760407 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff91c0ff120 msgr2=0x7ff91c0ff4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:43:41.766 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:41.759+0000 7ff92370e700 1 --2- 172.21.15.38:0/615760407 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff91c0ff120 0x7ff91c0ff4f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ff90c0099c0 tx=0x7ff90c009cd0 comp rx=0 tx=0).stop 2023-03-09T14:43:41.766 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:41.759+0000 7ff92370e700 1 -- 172.21.15.38:0/615760407 shutdown_connections 2023-03-09T14:43:41.766 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:41.759+0000 7ff92370e700 1 --2- 172.21.15.38:0/615760407 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff91c0ff120 0x7ff91c0ff4f0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:41.766 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:41.759+0000 7ff92370e700 1 --2- 172.21.15.38:0/615760407 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff91c104160 0x7ff91c10bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:41.767 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:41.759+0000 7ff92370e700 1 --2- 172.21.15.38:0/615760407 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff91c0ffac0 0x7ff91c103a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:41.767 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:41.759+0000 7ff92370e700 1 -- 172.21.15.38:0/615760407 >> 172.21.15.38:0/615760407 conn(0x7ff91c076d10 msgr2=0x7ff91c077110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:43:41.767 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:41.759+0000 7ff92370e700 1 -- 172.21.15.38:0/615760407 shutdown_connections 2023-03-09T14:43:41.768 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:41.760+0000 7ff92370e700 1 -- 172.21.15.38:0/615760407 wait complete. 2023-03-09T14:43:41.768 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:41.760+0000 7ff92370e700 1 Processor -- start 2023-03-09T14:43:41.768 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:41.760+0000 7ff92370e700 1 -- start start 2023-03-09T14:43:41.768 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:41.760+0000 7ff92370e700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff91c0ff120 0x7ff91c195020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:43:41.769 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:41.760+0000 7ff92370e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff91c0ffac0 0x7ff91c195560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:43:41.769 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:41.760+0000 7ff92370e700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff91c104160 0x7ff91c1999c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:43:41.769 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:41.760+0000 7ff92370e700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff91c19a000 con 0x7ff91c0ffac0 2023-03-09T14:43:41.769 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:41.760+0000 7ff92370e700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff91c19a170 con 0x7ff91c104160 2023-03-09T14:43:41.770 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:41.760+0000 7ff92370e700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff91c19a330 con 0x7ff91c0ff120 2023-03-09T14:43:41.770 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:41.760+0000 7ff920ca9700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff91c0ffac0 0x7ff91c195560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:43:41.770 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:41.760+0000 7ff921cab700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff91c104160 0x7ff91c1999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:43:41.770 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:41.760+0000 7ff920ca9700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff91c0ffac0 0x7ff91c195560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36326/0 (socket says 172.21.15.38:36326) 2023-03-09T14:43:41.770 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:41.761+0000 7ff9214aa700 1 --2- 172.21.15.38:0/2464678333 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff91c0ff120 0x7ff91c195020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:43:41.770 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:41.761+0000 7ff920ca9700 1 -- 172.21.15.38:0/2464678333 learned_addr learned my addr 172.21.15.38:0/2464678333 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:43:41.770 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:41.761+0000 7ff920ca9700 1 -- 172.21.15.38:0/2464678333 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff91c0ff120 msgr2=0x7ff91c195020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:43:41.770 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:41.761+0000 7ff920ca9700 1 --2- 172.21.15.38:0/2464678333 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff91c0ff120 0x7ff91c195020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:41.771 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:41.761+0000 7ff920ca9700 1 -- 172.21.15.38:0/2464678333 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff91c104160 msgr2=0x7ff91c1999c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:43:41.771 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:41.761+0000 7ff920ca9700 1 --2- 172.21.15.38:0/2464678333 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff91c104160 0x7ff91c1999c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:41.771 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:41.761+0000 7ff920ca9700 1 -- 172.21.15.38:0/2464678333 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff90c0095f0 con 0x7ff91c0ffac0 2023-03-09T14:43:41.771 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:41.761+0000 7ff920ca9700 1 --2- 172.21.15.38:0/2464678333 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff91c0ffac0 0x7ff91c195560 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ff91800d950 tx=0x7ff91800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:43:41.771 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:41.761+0000 7ff9127fc700 1 -- 172.21.15.38:0/2464678333 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff91800f890 con 0x7ff91c0ffac0 2023-03-09T14:43:41.771 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:41.761+0000 7ff9127fc700 1 -- 172.21.15.38:0/2464678333 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff918010460 con 0x7ff91c0ffac0 2023-03-09T14:43:41.771 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:41.761+0000 7ff92370e700 1 -- 172.21.15.38:0/2464678333 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff91c10ef20 con 0x7ff91c0ffac0 2023-03-09T14:43:41.772 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:41.761+0000 7ff9127fc700 1 -- 172.21.15.38:0/2464678333 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff91800e610 con 0x7ff91c0ffac0 2023-03-09T14:43:41.772 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:41.761+0000 7ff92370e700 1 -- 172.21.15.38:0/2464678333 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff91c10ed90 con 0x7ff91c0ffac0 2023-03-09T14:43:41.772 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:41.762+0000 7ff92370e700 1 -- 172.21.15.38:0/2464678333 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff91c10f0b0 con 0x7ff91c0ffac0 2023-03-09T14:43:41.773 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:41.764+0000 7ff9127fc700 1 -- 172.21.15.38:0/2464678333 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 133198+0+0 (secure 0 0 0) 0x7ff918023070 con 0x7ff91c0ffac0 2023-03-09T14:43:41.773 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:41.765+0000 7ff9127fc700 1 --2- 172.21.15.38:0/2464678333 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7ff90809f440 0x7ff9080a18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:43:41.773 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:41.765+0000 7ff9214aa700 1 --2- 172.21.15.38:0/2464678333 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7ff90809f440 0x7ff9080a18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:43:41.773 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:41.765+0000 7ff9127fc700 1 -- 172.21.15.38:0/2464678333 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ff9180c91c0 con 0x7ff91c0ffac0 2023-03-09T14:43:41.774 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:41.765+0000 7ff9214aa700 1 --2- 172.21.15.38:0/2464678333 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7ff90809f440 0x7ff9080a18f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff90c000c00 tx=0x7ff90c003680 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:43:41.774 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:41.768+0000 7ff9127fc700 1 -- 172.21.15.38:0/2464678333 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7ff918093af0 con 0x7ff91c0ffac0 2023-03-09T14:43:41.967 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:41.964+0000 7ff92370e700 1 -- 172.21.15.38:0/2464678333 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff91c066b10 con 0x7ff91c0ffac0 2023-03-09T14:43:41.968 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:41.965+0000 7ff9127fc700 1 -- 172.21.15.38:0/2464678333 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) v1 ==== 74+0+18435 (secure 0 0 0) 0x7ff918097110 con 0x7ff91c0ffac0 2023-03-09T14:43:41.968 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:43:41.969 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":79,"fsid":"c6733fe2-be87-11ed-9af2-001a4aab830c","created":"2023-03-09T14:37:14.947981+0000","modified":"2023-03-09T14:43:38.269957+0000","last_up_change":"2023-03-09T14:43:37.198407+0000","last_in_change":"2023-03-09T14:43:19.550566+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-09T14:40:23.196665+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51477039-15ee-4707-a34d-4ecfa18bebde","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6803","nonce":1508072290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6805","nonce":1508072290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6809","nonce":1508072290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6807","nonce":1508072290}]},"public_addr":"172.21.15.38:6803/1508072290","cluster_addr":"172.21.15.38:6805/1508072290","heartbeat_back_addr":"172.21.15.38:6809/1508072290","heartbeat_front_addr":"172.21.15.38:6807/1508072290","state":["exists","up"]},{"osd":1,"uuid":"71f8a718-b0a9-4f39-bc54-267aa42a2578","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6801","nonce":3929268773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6803","nonce":3929268773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6807","nonce":3929268773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6805","nonce":3929268773}]},"public_addr":"172.21.15.122:6801/3929268773","cluster_addr":"172.21.15.122:6803/3929268773","heartbeat_back_addr":"172.21.15.122:6807/3929268773","heartbeat_front_addr":"172.21.15.122:6805/3929268773","state":["exists","up"]},{"osd":2,"uuid":"961a2cbe-140e-437f-b131-1fcac573b19c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6801","nonce":1310630803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6803","nonce":1310630803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6807","nonce":1310630803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6805","nonce":1310630803}]},"public_addr":"172.21.15.181:6801/1310630803","cluster_addr":"172.21.15.181:6803/1310630803","heartbeat_back_addr":"172.21.15.181:6807/1310630803","heartbeat_front_addr":"172.21.15.181:6805/1310630803","state":["exists","up"]},{"osd":3,"uuid":"969df8fa-2d19-42c7-9f2a-5c5fac15d4a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6811","nonce":2719814100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6813","nonce":2719814100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6817","nonce":2719814100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6815","nonce":2719814100}]},"public_addr":"172.21.15.38:6811/2719814100","cluster_addr":"172.21.15.38:6813/2719814100","heartbeat_back_addr":"172.21.15.38:6817/2719814100","heartbeat_front_addr":"172.21.15.38:6815/2719814100","state":["exists","up"]},{"osd":4,"uuid":"44a6dce3-e06e-4cbd-9404-62a86d01ab6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6809","nonce":2480764994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6811","nonce":2480764994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6815","nonce":2480764994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6813","nonce":2480764994}]},"public_addr":"172.21.15.122:6809/2480764994","cluster_addr":"172.21.15.122:6811/2480764994","heartbeat_back_addr":"172.21.15.122:6815/2480764994","heartbeat_front_addr":"172.21.15.122:6813/2480764994","state":["exists","up"]},{"osd":5,"uuid":"49749574-c00c-4e3a-80e6-e3686df58d5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6809","nonce":3141346246}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6811","nonce":3141346246}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6815","nonce":3141346246}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6813","nonce":3141346246}]},"public_addr":"172.21.15.181:6809/3141346246","cluster_addr":"172.21.15.181:6811/3141346246","heartbeat_back_addr":"172.21.15.181:6815/3141346246","heartbeat_front_addr":"172.21.15.181:6813/3141346246","state":["exists","up"]},{"osd":6,"uuid":"61c0437e-3f51-45a9-8a61-e98eea0093f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6819","nonce":2122788957}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6821","nonce":2122788957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6825","nonce":2122788957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6823","nonce":2122788957}]},"public_addr":"172.21.15.38:6819/2122788957","cluster_addr":"172.21.15.38:6821/2122788957","heartbeat_back_addr":"172.21.15.38:6825/2122788957","heartbeat_front_addr":"172.21.15.38:6823/2122788957","state":["exists","up"]},{"osd":7,"uuid":"231fecdd-939f-4c7b-83a1-c4a1cffd7139","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6817","nonce":656874139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6819","nonce":656874139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6823","nonce":656874139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6821","nonce":656874139}]},"public_addr":"172.21.15.122:6817/656874139","cluster_addr":"172.21.15.122:6819/656874139","heartbeat_back_addr":"172.21.15.122:6823/656874139","heartbeat_front_addr":"172.21.15.122:6821/656874139","state":["exists","up"]},{"osd":8,"uuid":"297e6d6d-85fd-43ce-9028-5e9dbe55c36e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6817","nonce":755000703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6819","nonce":755000703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6823","nonce":755000703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6821","nonce":755000703}]},"public_addr":"172.21.15.181:6817/755000703","cluster_addr":"172.21.15.181:6819/755000703","heartbeat_back_addr":"172.21.15.181:6823/755000703","heartbeat_front_addr":"172.21.15.181:6821/755000703","state":["exists","up"]},{"osd":9,"uuid":"94cc0355-5a1a-4cfc-bd0c-562b6b41127e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6827","nonce":2624555969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6829","nonce":2624555969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6833","nonce":2624555969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6831","nonce":2624555969}]},"public_addr":"172.21.15.38:6827/2624555969","cluster_addr":"172.21.15.38:6829/2624555969","heartbeat_back_addr":"172.21.15.38:6833/2624555969","heartbeat_front_addr":"172.21.15.38:6831/2624555969","state":["exists","up"]},{"osd":10,"uuid":"f4f2e042-8d92-46bf-93b8-8b0f04795c2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6825","nonce":1427785280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6827","nonce":1427785280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6831","nonce":1427785280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6829","nonce":1427785280}]},"public_addr":"172.21.15.122:6825/1427785280","cluster_addr":"172.21.15.122:6827/1427785280","heartbeat_back_addr":"172.21.15.122:6831/1427785280","heartbeat_front_addr":"172.21.15.122:6829/1427785280","state":["exists","up"]},{"osd":11,"uuid":"07444936-eae5-4097-a799-60294cafe80b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6825","nonce":3183488640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6827","nonce":3183488640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6831","nonce":3183488640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6829","nonce":3183488640}]},"public_addr":"172.21.15.181:6825/3183488640","cluster_addr":"172.21.15.181:6827/3183488640","heartbeat_back_addr":"172.21.15.181:6831/3183488640","heartbeat_front_addr":"172.21.15.181:6829/3183488640","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:39:37.657556+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:39:58.960074+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:40:20.326952+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:40:42.244387+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:41:03.946973+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:41:25.097985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:41:46.877916+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:42:08.413827+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:42:29.822009+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:42:51.776684+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:43:13.937249+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:43:35.569199+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/4272288735":"2023-03-10T14:38:14.612767+0000","172.21.15.38:6801/3381625669":"2023-03-10T14:38:14.612767+0000","172.21.15.38:6800/3381625669":"2023-03-10T14:38:14.612767+0000","172.21.15.38:6800/3060474342":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/4253918615":"2023-03-10T14:38:14.612767+0000","172.21.15.38:0/3496083181":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/245602368":"2023-03-10T14:37:47.517775+0000","172.21.15.38:6801/3060474342":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/3622589182":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/1430302251":"2023-03-10T14:38:14.612767+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-09T14:43:41.970 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:41.967+0000 7ff92370e700 1 -- 172.21.15.38:0/2464678333 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7ff90809f440 msgr2=0x7ff9080a18f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:43:41.970 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:41.967+0000 7ff92370e700 1 --2- 172.21.15.38:0/2464678333 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7ff90809f440 0x7ff9080a18f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff90c000c00 tx=0x7ff90c003680 comp rx=0 tx=0).stop 2023-03-09T14:43:41.971 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:41.967+0000 7ff92370e700 1 -- 172.21.15.38:0/2464678333 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff91c0ffac0 msgr2=0x7ff91c195560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:43:41.971 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:41.967+0000 7ff92370e700 1 --2- 172.21.15.38:0/2464678333 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff91c0ffac0 0x7ff91c195560 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ff91800d950 tx=0x7ff91800dc60 comp rx=0 tx=0).stop 2023-03-09T14:43:41.971 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:41.967+0000 7ff92370e700 1 -- 172.21.15.38:0/2464678333 shutdown_connections 2023-03-09T14:43:41.971 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:41.967+0000 7ff92370e700 1 --2- 172.21.15.38:0/2464678333 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7ff90809f440 0x7ff9080a18f0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:41.971 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:41.967+0000 7ff92370e700 1 --2- 172.21.15.38:0/2464678333 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff91c104160 0x7ff91c1999c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:41.971 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:41.967+0000 7ff92370e700 1 --2- 172.21.15.38:0/2464678333 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff91c0ff120 0x7ff91c195020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:41.971 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:41.967+0000 7ff92370e700 1 --2- 172.21.15.38:0/2464678333 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff91c0ffac0 0x7ff91c195560 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:41.972 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:41.967+0000 7ff92370e700 1 -- 172.21.15.38:0/2464678333 >> 172.21.15.38:0/2464678333 conn(0x7ff91c076d10 msgr2=0x7ff91c0fc9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:43:41.972 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:41.967+0000 7ff92370e700 1 -- 172.21.15.38:0/2464678333 shutdown_connections 2023-03-09T14:43:41.972 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:41.967+0000 7ff92370e700 1 -- 172.21.15.38:0/2464678333 wait complete. 2023-03-09T14:43:42.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:42 smithi038 ceph-mon[92422]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-03-09T14:43:42.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:42 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/2464678333' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-03-09T14:43:42.506 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-09T14:40:23.196665+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-03-09T14:43:42.506 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph osd pool get .mgr pg_num 2023-03-09T14:43:42.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:42 smithi122 ceph-mon[93869]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-03-09T14:43:42.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:42 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/2464678333' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-03-09T14:43:42.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:42 smithi181 ceph-mon[94189]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-03-09T14:43:42.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:42 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/2464678333' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-03-09T14:43:43.163 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.a/config 2023-03-09T14:43:44.450 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:44.446+0000 7f3d788d5700 1 -- 172.21.15.38:0/1658081088 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d74104380 msgr2=0x7f3d7410bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:43:44.451 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:44.446+0000 7f3d788d5700 1 --2- 172.21.15.38:0/1658081088 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d74104380 0x7f3d7410bf50 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f3d6c00b0d0 tx=0x7f3d6c00b3e0 comp rx=0 tx=0).stop 2023-03-09T14:43:44.451 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:44.446+0000 7f3d788d5700 1 -- 172.21.15.38:0/1658081088 shutdown_connections 2023-03-09T14:43:44.451 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:44.446+0000 7f3d788d5700 1 --2- 172.21.15.38:0/1658081088 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d74104380 0x7f3d7410bf50 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:44.451 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:44.446+0000 7f3d788d5700 1 --2- 172.21.15.38:0/1658081088 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d740ff860 0x7f3d740ffc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:44.451 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:44.446+0000 7f3d788d5700 1 --2- 172.21.15.38:0/1658081088 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d74100200 0x7f3d74103ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:44.451 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:44.446+0000 7f3d788d5700 1 -- 172.21.15.38:0/1658081088 >> 172.21.15.38:0/1658081088 conn(0x7f3d74075580 msgr2=0x7f3d74075980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:43:44.451 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:44.446+0000 7f3d788d5700 1 -- 172.21.15.38:0/1658081088 shutdown_connections 2023-03-09T14:43:44.452 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:44.446+0000 7f3d788d5700 1 -- 172.21.15.38:0/1658081088 wait complete. 2023-03-09T14:43:44.452 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:44.447+0000 7f3d788d5700 1 Processor -- start 2023-03-09T14:43:44.452 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:44.447+0000 7f3d788d5700 1 -- start start 2023-03-09T14:43:44.453 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:44.447+0000 7f3d788d5700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d740ff860 0x7f3d74196480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:43:44.454 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:44.447+0000 7f3d788d5700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d74100200 0x7f3d741969c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:43:44.454 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:44.448+0000 7f3d788d5700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d74104380 0x7f3d7419ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:43:44.454 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:44.448+0000 7f3d788d5700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d7419b460 con 0x7f3d74104380 2023-03-09T14:43:44.454 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:44.448+0000 7f3d788d5700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d7419b5d0 con 0x7f3d740ff860 2023-03-09T14:43:44.454 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:44.448+0000 7f3d788d5700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d7419b790 con 0x7f3d74100200 2023-03-09T14:43:44.454 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:44.448+0000 7f3d72d9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d74104380 0x7f3d7419ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:43:44.454 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:44.448+0000 7f3d72d9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d74104380 0x7f3d7419ae20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36338/0 (socket says 172.21.15.38:36338) 2023-03-09T14:43:44.455 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:44.448+0000 7f3d72d9d700 1 -- 172.21.15.38:0/981218091 learned_addr learned my addr 172.21.15.38:0/981218091 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:43:44.455 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:44.448+0000 7f3d71d9b700 1 --2- 172.21.15.38:0/981218091 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d74100200 0x7f3d741969c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:43:44.455 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:44.448+0000 7f3d7259c700 1 --2- 172.21.15.38:0/981218091 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d740ff860 0x7f3d74196480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:43:44.455 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:44.448+0000 7f3d72d9d700 1 -- 172.21.15.38:0/981218091 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d74100200 msgr2=0x7f3d741969c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:43:44.455 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:44.448+0000 7f3d72d9d700 1 --2- 172.21.15.38:0/981218091 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d74100200 0x7f3d741969c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:44.456 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:44.448+0000 7f3d72d9d700 1 -- 172.21.15.38:0/981218091 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d740ff860 msgr2=0x7f3d74196480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:43:44.456 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:44.448+0000 7f3d72d9d700 1 --2- 172.21.15.38:0/981218091 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d740ff860 0x7f3d74196480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:44.456 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:44.448+0000 7f3d72d9d700 1 -- 172.21.15.38:0/981218091 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d6c009d00 con 0x7f3d74104380 2023-03-09T14:43:44.456 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:44.448+0000 7f3d72d9d700 1 --2- 172.21.15.38:0/981218091 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d74104380 0x7f3d7419ae20 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f3d6c007870 tx=0x7f3d6c0078a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:43:44.456 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:44.448+0000 7f3d637fe700 1 -- 172.21.15.38:0/981218091 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d6c00e040 con 0x7f3d74104380 2023-03-09T14:43:44.456 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:44.448+0000 7f3d788d5700 1 -- 172.21.15.38:0/981218091 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d7410f1c0 con 0x7f3d74104380 2023-03-09T14:43:44.456 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:44.448+0000 7f3d637fe700 1 -- 172.21.15.38:0/981218091 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3d6c003e10 con 0x7f3d74104380 2023-03-09T14:43:44.457 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:44.448+0000 7f3d637fe700 1 -- 172.21.15.38:0/981218091 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d6c01b980 con 0x7f3d74104380 2023-03-09T14:43:44.457 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:44.448+0000 7f3d788d5700 1 -- 172.21.15.38:0/981218091 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d7410f030 con 0x7f3d74104380 2023-03-09T14:43:44.457 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:44.449+0000 7f3d788d5700 1 -- 172.21.15.38:0/981218091 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d7410f350 con 0x7f3d74104380 2023-03-09T14:43:44.457 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:44.451+0000 7f3d637fe700 1 -- 172.21.15.38:0/981218091 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 133198+0+0 (secure 0 0 0) 0x7f3d6c02b070 con 0x7f3d74104380 2023-03-09T14:43:44.458 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:44.451+0000 7f3d637fe700 1 --2- 172.21.15.38:0/981218091 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f3d5c09f3c0 0x7f3d5c0a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:43:44.458 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:44.451+0000 7f3d637fe700 1 -- 172.21.15.38:0/981218091 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f3d6c0cb2c0 con 0x7f3d74104380 2023-03-09T14:43:44.458 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:44.451+0000 7f3d7259c700 1 --2- 172.21.15.38:0/981218091 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f3d5c09f3c0 0x7f3d5c0a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:43:44.458 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:44.452+0000 7f3d7259c700 1 --2- 172.21.15.38:0/981218091 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f3d5c09f3c0 0x7f3d5c0a1870 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3d68006fd0 tx=0x7f3d68008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:43:44.458 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:44.455+0000 7f3d637fe700 1 -- 172.21.15.38:0/981218091 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f3d6c095c00 con 0x7f3d74104380 2023-03-09T14:43:44.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:44 smithi038 ceph-mon[92422]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-03-09T14:43:44.655 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:44.652+0000 7f3d788d5700 1 -- 172.21.15.38:0/981218091 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3d741977d0 con 0x7f3d74104380 2023-03-09T14:43:44.655 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:44.652+0000 7f3d637fe700 1 -- 172.21.15.38:0/981218091 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v79) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3d6c099220 con 0x7f3d74104380 2023-03-09T14:43:44.656 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 1 2023-03-09T14:43:44.658 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:44.654+0000 7f3d788d5700 1 -- 172.21.15.38:0/981218091 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f3d5c09f3c0 msgr2=0x7f3d5c0a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:43:44.658 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:44.654+0000 7f3d788d5700 1 --2- 172.21.15.38:0/981218091 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f3d5c09f3c0 0x7f3d5c0a1870 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3d68006fd0 tx=0x7f3d68008040 comp rx=0 tx=0).stop 2023-03-09T14:43:44.658 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:44.654+0000 7f3d788d5700 1 -- 172.21.15.38:0/981218091 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d74104380 msgr2=0x7f3d7419ae20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:43:44.658 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:44.654+0000 7f3d788d5700 1 --2- 172.21.15.38:0/981218091 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d74104380 0x7f3d7419ae20 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f3d6c007870 tx=0x7f3d6c0078a0 comp rx=0 tx=0).stop 2023-03-09T14:43:44.658 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:44.654+0000 7f3d788d5700 1 -- 172.21.15.38:0/981218091 shutdown_connections 2023-03-09T14:43:44.658 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:44.654+0000 7f3d788d5700 1 --2- 172.21.15.38:0/981218091 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3d74104380 0x7f3d7419ae20 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:44.659 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:44.654+0000 7f3d788d5700 1 --2- 172.21.15.38:0/981218091 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f3d5c09f3c0 0x7f3d5c0a1870 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:44.659 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:44.654+0000 7f3d788d5700 1 --2- 172.21.15.38:0/981218091 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3d740ff860 0x7f3d74196480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:44.659 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:44.655+0000 7f3d788d5700 1 --2- 172.21.15.38:0/981218091 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3d74100200 0x7f3d741969c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:44.659 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:44.655+0000 7f3d788d5700 1 -- 172.21.15.38:0/981218091 >> 172.21.15.38:0/981218091 conn(0x7f3d74075580 msgr2=0x7f3d740fe580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:43:44.659 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:44.655+0000 7f3d788d5700 1 -- 172.21.15.38:0/981218091 shutdown_connections 2023-03-09T14:43:44.659 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:44.655+0000 7f3d788d5700 1 -- 172.21.15.38:0/981218091 wait complete. 2023-03-09T14:43:44.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:44 smithi122 ceph-mon[93869]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-03-09T14:43:44.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:44 smithi181 ceph-mon[94189]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-03-09T14:43:45.110 INFO:tasks.cephadm:Adding mds.a on smithi038 2023-03-09T14:43:45.110 INFO:tasks.cephadm:Adding mds.d on smithi038 2023-03-09T14:43:45.111 INFO:tasks.cephadm:Adding mds.g on smithi038 2023-03-09T14:43:45.111 INFO:tasks.cephadm:Adding mds.j on smithi038 2023-03-09T14:43:45.111 INFO:tasks.cephadm:Adding mds.b on smithi122 2023-03-09T14:43:45.111 INFO:tasks.cephadm:Adding mds.e on smithi122 2023-03-09T14:43:45.111 INFO:tasks.cephadm:Adding mds.h on smithi122 2023-03-09T14:43:45.112 INFO:tasks.cephadm:Adding mds.k on smithi122 2023-03-09T14:43:45.112 INFO:tasks.cephadm:Adding mds.c on smithi181 2023-03-09T14:43:45.112 INFO:tasks.cephadm:Adding mds.f on smithi181 2023-03-09T14:43:45.112 INFO:tasks.cephadm:Adding mds.i on smithi181 2023-03-09T14:43:45.112 INFO:tasks.cephadm:Adding mds.l on smithi181 2023-03-09T14:43:45.113 DEBUG:teuthology.orchestra.run.smithi181:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph orch apply mds all '12;smithi038=a;smithi038=d;smithi038=g;smithi038=j;smithi122=b;smithi122=e;smithi122=h;smithi122=k;smithi181=c;smithi181=f;smithi181=i;smithi181=l' 2023-03-09T14:43:45.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:45 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/981218091' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-03-09T14:43:45.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:45 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/981218091' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-03-09T14:43:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:45 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/981218091' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-03-09T14:43:45.770 INFO:teuthology.orchestra.run.smithi181.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.c/config 2023-03-09T14:43:46.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:46 smithi122 ceph-mon[93869]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-03-09T14:43:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:46 smithi181 ceph-mon[94189]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-03-09T14:43:46.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:46 smithi038 ceph-mon[92422]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-03-09T14:43:46.959 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:46.955+0000 7fc9c38fe700 1 -- 172.21.15.181:0/3678662232 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc9bc0feeb0 msgr2=0x7fc9bc0ff2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:43:46.959 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:46.955+0000 7fc9c38fe700 1 --2- 172.21.15.181:0/3678662232 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc9bc0feeb0 0x7fc9bc0ff2c0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc9b400b0a0 tx=0x7fc9b400b3b0 comp rx=0 tx=0).stop 2023-03-09T14:43:46.960 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:46.955+0000 7fc9c38fe700 1 -- 172.21.15.181:0/3678662232 shutdown_connections 2023-03-09T14:43:46.960 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:46.955+0000 7fc9c38fe700 1 --2- 172.21.15.181:0/3678662232 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc9bc100a40 0x7fc9bc107370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:46.960 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:46.955+0000 7fc9c38fe700 1 --2- 172.21.15.181:0/3678662232 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc9bc0feeb0 0x7fc9bc0ff2c0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:46.960 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:46.955+0000 7fc9c38fe700 1 --2- 172.21.15.181:0/3678662232 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc9bc100090 0x7fc9bc100500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:46.960 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:46.955+0000 7fc9c38fe700 1 -- 172.21.15.181:0/3678662232 >> 172.21.15.181:0/3678662232 conn(0x7fc9bc0fa500 msgr2=0x7fc9bc0fc950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:43:46.960 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:46.955+0000 7fc9c38fe700 1 -- 172.21.15.181:0/3678662232 shutdown_connections 2023-03-09T14:43:46.960 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:46.955+0000 7fc9c38fe700 1 -- 172.21.15.181:0/3678662232 wait complete. 2023-03-09T14:43:46.961 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:46.956+0000 7fc9c38fe700 1 Processor -- start 2023-03-09T14:43:46.961 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:46.956+0000 7fc9c38fe700 1 -- start start 2023-03-09T14:43:46.961 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:46.956+0000 7fc9c38fe700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc9bc0feeb0 0x7fc9bc069900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:43:46.961 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:46.956+0000 7fc9c38fe700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc9bc100090 0x7fc9bc19aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:43:46.961 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:46.956+0000 7fc9c38fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc9bc100a40 0x7fc9bc19d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:43:46.961 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:46.956+0000 7fc9c38fe700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9bc19d9d0 con 0x7fc9bc100a40 2023-03-09T14:43:46.961 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:46.956+0000 7fc9c38fe700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9bc19db40 con 0x7fc9bc100090 2023-03-09T14:43:46.962 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:46.956+0000 7fc9c38fe700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9bc19dd00 con 0x7fc9bc0feeb0 2023-03-09T14:43:46.962 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:46.956+0000 7fc9c169a700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc9bc0feeb0 0x7fc9bc069900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:43:46.962 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:46.956+0000 7fc9c169a700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc9bc0feeb0 0x7fc9bc069900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.181:50958/0 (socket says 172.21.15.181:50958) 2023-03-09T14:43:46.962 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:46.956+0000 7fc9c169a700 1 -- 172.21.15.181:0/2557327759 learned_addr learned my addr 172.21.15.181:0/2557327759 (peer_addr_for_me v2:172.21.15.181:0/0) 2023-03-09T14:43:46.962 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:46.956+0000 7fc9c0e99700 1 --2- 172.21.15.181:0/2557327759 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc9bc100090 0x7fc9bc19aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:43:46.962 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:46.956+0000 7fc9c1e9b700 1 --2- 172.21.15.181:0/2557327759 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc9bc100a40 0x7fc9bc19d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:43:46.962 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:46.956+0000 7fc9c169a700 1 -- 172.21.15.181:0/2557327759 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc9bc100090 msgr2=0x7fc9bc19aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:43:46.962 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:46.956+0000 7fc9c169a700 1 --2- 172.21.15.181:0/2557327759 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc9bc100090 0x7fc9bc19aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:46.963 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:46.956+0000 7fc9c169a700 1 -- 172.21.15.181:0/2557327759 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc9bc100a40 msgr2=0x7fc9bc19d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:43:46.963 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:46.956+0000 7fc9c169a700 1 --2- 172.21.15.181:0/2557327759 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc9bc100a40 0x7fc9bc19d230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:46.963 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:46.956+0000 7fc9c169a700 1 -- 172.21.15.181:0/2557327759 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9b4009d00 con 0x7fc9bc0feeb0 2023-03-09T14:43:46.963 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:46.956+0000 7fc9c169a700 1 --2- 172.21.15.181:0/2557327759 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc9bc0feeb0 0x7fc9bc069900 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc9b4004d00 tx=0x7fc9b4004d30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:43:46.963 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:46.956+0000 7fc9b27fc700 1 -- 172.21.15.181:0/2557327759 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9b400e040 con 0x7fc9bc0feeb0 2023-03-09T14:43:46.963 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:46.957+0000 7fc9c38fe700 1 -- 172.21.15.181:0/2557327759 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9bc10a870 con 0x7fc9bc0feeb0 2023-03-09T14:43:46.963 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:46.957+0000 7fc9b27fc700 1 -- 172.21.15.181:0/2557327759 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc9b400cec0 con 0x7fc9bc0feeb0 2023-03-09T14:43:46.964 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:46.957+0000 7fc9b27fc700 1 -- 172.21.15.181:0/2557327759 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9b4005630 con 0x7fc9bc0feeb0 2023-03-09T14:43:46.964 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:46.957+0000 7fc9c38fe700 1 -- 172.21.15.181:0/2557327759 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9bc10a6e0 con 0x7fc9bc0feeb0 2023-03-09T14:43:46.964 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:46.957+0000 7fc9c38fe700 1 -- 172.21.15.181:0/2557327759 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9bc106510 con 0x7fc9bc0feeb0 2023-03-09T14:43:46.965 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:46.958+0000 7fc9b27fc700 1 -- 172.21.15.181:0/2557327759 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 16) v1 ==== 133198+0+0 (secure 0 0 0) 0x7fc9b401bcb0 con 0x7fc9bc0feeb0 2023-03-09T14:43:46.965 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:46.959+0000 7fc9b27fc700 1 --2- 172.21.15.181:0/2557327759 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fc9a809f3c0 0x7fc9a80a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:43:46.965 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:46.959+0000 7fc9b27fc700 1 -- 172.21.15.181:0/2557327759 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fc9b40cba30 con 0x7fc9bc0feeb0 2023-03-09T14:43:46.965 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:46.959+0000 7fc9c0e99700 1 --2- 172.21.15.181:0/2557327759 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fc9a809f3c0 0x7fc9a80a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:43:46.965 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:46.960+0000 7fc9c0e99700 1 --2- 172.21.15.181:0/2557327759 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fc9a809f3c0 0x7fc9a80a1870 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc0febd0 tx=0x7fc9ac00b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:43:46.966 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:46.963+0000 7fc9b27fc700 1 -- 172.21.15.181:0/2557327759 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7fc9b4096370 con 0x7fc9bc0feeb0 2023-03-09T14:43:47.158 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:47.155+0000 7fc9c38fe700 1 -- 172.21.15.181:0/2557327759 --> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi038=a;smithi038=d;smithi038=g;smithi038=j;smithi122=b;smithi122=e;smithi122=h;smithi122=k;smithi181=c;smithi181=f;smithi181=i;smithi181=l", "target": ["mon-mgr", ""]}) v1 -- 0x7fc9bc0617b0 con 0x7fc9a809f3c0 2023-03-09T14:43:47.167 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:47.164+0000 7fc9b27fc700 1 -- 172.21.15.181:0/2557327759 <== mgr.14150 v2:172.21.15.38:6800/3328887328 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7fc9bc0617b0 con 0x7fc9a809f3c0 2023-03-09T14:43:47.167 INFO:teuthology.orchestra.run.smithi181.stdout:Scheduled mds.all update... 2023-03-09T14:43:47.169 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:47.166+0000 7fc9c38fe700 1 -- 172.21.15.181:0/2557327759 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fc9a809f3c0 msgr2=0x7fc9a80a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:43:47.170 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:47.166+0000 7fc9c38fe700 1 --2- 172.21.15.181:0/2557327759 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fc9a809f3c0 0x7fc9a80a1870 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc0febd0 tx=0x7fc9ac00b410 comp rx=0 tx=0).stop 2023-03-09T14:43:47.170 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:47.166+0000 7fc9c38fe700 1 -- 172.21.15.181:0/2557327759 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc9bc0feeb0 msgr2=0x7fc9bc069900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:43:47.170 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:47.166+0000 7fc9c38fe700 1 --2- 172.21.15.181:0/2557327759 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc9bc0feeb0 0x7fc9bc069900 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc9b4004d00 tx=0x7fc9b4004d30 comp rx=0 tx=0).stop 2023-03-09T14:43:47.170 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:47.166+0000 7fc9c38fe700 1 -- 172.21.15.181:0/2557327759 shutdown_connections 2023-03-09T14:43:47.170 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:47.166+0000 7fc9c38fe700 1 --2- 172.21.15.181:0/2557327759 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc9bc100a40 0x7fc9bc19d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:47.170 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:47.166+0000 7fc9c38fe700 1 --2- 172.21.15.181:0/2557327759 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fc9a809f3c0 0x7fc9a80a1870 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:47.171 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:47.166+0000 7fc9c38fe700 1 --2- 172.21.15.181:0/2557327759 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc9bc0feeb0 0x7fc9bc069900 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:47.171 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:47.166+0000 7fc9c38fe700 1 --2- 172.21.15.181:0/2557327759 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc9bc100090 0x7fc9bc19aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:47.171 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:47.166+0000 7fc9c38fe700 1 -- 172.21.15.181:0/2557327759 >> 172.21.15.181:0/2557327759 conn(0x7fc9bc0fa500 msgr2=0x7fc9bc103650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:43:47.171 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:47.167+0000 7fc9c38fe700 1 -- 172.21.15.181:0/2557327759 shutdown_connections 2023-03-09T14:43:47.171 INFO:teuthology.orchestra.run.smithi181.stderr:2023-03-09T14:43:47.167+0000 7fc9c38fe700 1 -- 172.21.15.181:0/2557327759 wait complete. 2023-03-09T14:43:47.524 DEBUG:teuthology.orchestra.run.smithi038:mds.a> sudo journalctl -f -n 0 -u ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@mds.a.service 2023-03-09T14:43:47.527 DEBUG:teuthology.orchestra.run.smithi038:mds.d> sudo journalctl -f -n 0 -u ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@mds.d.service 2023-03-09T14:43:47.566 INFO:journalctl@ceph.mds.a.smithi038.stdout:-- Logs begin at Thu 2023-03-09 14:24:59 UTC. -- 2023-03-09T14:43:47.567 DEBUG:teuthology.orchestra.run.smithi038:mds.g> sudo journalctl -f -n 0 -u ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@mds.g.service 2023-03-09T14:43:47.597 INFO:journalctl@ceph.mds.d.smithi038.stdout:-- Logs begin at Thu 2023-03-09 14:24:59 UTC. -- 2023-03-09T14:43:47.598 DEBUG:teuthology.orchestra.run.smithi038:mds.j> sudo journalctl -f -n 0 -u ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@mds.j.service 2023-03-09T14:43:47.626 INFO:journalctl@ceph.mds.g.smithi038.stdout:-- Logs begin at Thu 2023-03-09 14:24:59 UTC. -- 2023-03-09T14:43:47.627 DEBUG:teuthology.orchestra.run.smithi122:mds.b> sudo journalctl -f -n 0 -u ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@mds.b.service 2023-03-09T14:43:47.630 DEBUG:teuthology.orchestra.run.smithi122:mds.e> sudo journalctl -f -n 0 -u ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@mds.e.service 2023-03-09T14:43:47.655 INFO:journalctl@ceph.mds.j.smithi038.stdout:-- Logs begin at Thu 2023-03-09 14:24:59 UTC. -- 2023-03-09T14:43:47.670 INFO:journalctl@ceph.mds.b.smithi122.stdout:-- Logs begin at Thu 2023-03-09 14:25:09 UTC. -- 2023-03-09T14:43:47.671 DEBUG:teuthology.orchestra.run.smithi122:mds.h> sudo journalctl -f -n 0 -u ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@mds.h.service 2023-03-09T14:43:47.702 INFO:journalctl@ceph.mds.e.smithi122.stdout:-- Logs begin at Thu 2023-03-09 14:25:09 UTC. -- 2023-03-09T14:43:47.703 DEBUG:teuthology.orchestra.run.smithi122:mds.k> sudo journalctl -f -n 0 -u ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@mds.k.service 2023-03-09T14:43:47.731 INFO:journalctl@ceph.mds.h.smithi122.stdout:-- Logs begin at Thu 2023-03-09 14:25:09 UTC. -- 2023-03-09T14:43:47.732 DEBUG:teuthology.orchestra.run.smithi181:mds.c> sudo journalctl -f -n 0 -u ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@mds.c.service 2023-03-09T14:43:47.734 DEBUG:teuthology.orchestra.run.smithi181:mds.f> sudo journalctl -f -n 0 -u ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@mds.f.service 2023-03-09T14:43:47.759 INFO:journalctl@ceph.mds.k.smithi122.stdout:-- Logs begin at Thu 2023-03-09 14:25:09 UTC. -- 2023-03-09T14:43:47.766 INFO:journalctl@ceph.mds.c.smithi181.stdout:-- Logs begin at Thu 2023-03-09 14:25:01 UTC. -- 2023-03-09T14:43:47.767 DEBUG:teuthology.orchestra.run.smithi181:mds.i> sudo journalctl -f -n 0 -u ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@mds.i.service 2023-03-09T14:43:47.792 INFO:journalctl@ceph.mds.f.smithi181.stdout:-- Logs begin at Thu 2023-03-09 14:25:01 UTC. -- 2023-03-09T14:43:47.794 DEBUG:teuthology.orchestra.run.smithi181:mds.l> sudo journalctl -f -n 0 -u ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@mds.l.service 2023-03-09T14:43:47.818 INFO:journalctl@ceph.mds.i.smithi181.stdout:-- Logs begin at Thu 2023-03-09 14:25:01 UTC. -- 2023-03-09T14:43:47.821 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2023-03-09T14:43:47.822 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-09T14:43:47.844 INFO:journalctl@ceph.mds.l.smithi181.stdout:-- Logs begin at Thu 2023-03-09 14:25:01 UTC. -- 2023-03-09T14:43:47.970 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:47.968+0000 7f8274a0d700 1 -- 172.21.15.38:0/4019763169 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8270100910 msgr2=0x7f82701071d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:43:47.970 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:47.968+0000 7f8274a0d700 1 --2- 172.21.15.38:0/4019763169 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8270100910 0x7f82701071d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f826800b0a0 tx=0x7f826800b3b0 comp rx=0 tx=0).stop 2023-03-09T14:43:47.971 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:47.968+0000 7f8274a0d700 1 -- 172.21.15.38:0/4019763169 shutdown_connections 2023-03-09T14:43:47.971 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:47.968+0000 7f8274a0d700 1 --2- 172.21.15.38:0/4019763169 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8270100910 0x7f82701071d0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:47.971 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:47.968+0000 7f8274a0d700 1 --2- 172.21.15.38:0/4019763169 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f82700fed90 0x7f82700ff1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:47.971 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:47.968+0000 7f8274a0d700 1 --2- 172.21.15.38:0/4019763169 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f82700fff80 0x7f82701003d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:47.971 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:47.968+0000 7f8274a0d700 1 -- 172.21.15.38:0/4019763169 >> 172.21.15.38:0/4019763169 conn(0x7f82700fa370 msgr2=0x7f82700fc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:43:47.971 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:47.968+0000 7f8274a0d700 1 -- 172.21.15.38:0/4019763169 shutdown_connections 2023-03-09T14:43:47.971 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:47.968+0000 7f8274a0d700 1 -- 172.21.15.38:0/4019763169 wait complete. 2023-03-09T14:43:47.973 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:47.968+0000 7f8274a0d700 1 Processor -- start 2023-03-09T14:43:47.974 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:47.968+0000 7f8274a0d700 1 -- start start 2023-03-09T14:43:47.974 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:47.969+0000 7f8274a0d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f82700fed90 0x7f8270102d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:43:47.974 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:47.969+0000 7f8274a0d700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f82700fff80 0x7f8270101360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:43:47.974 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:47.969+0000 7f8274a0d700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8270100910 0x7f82701018a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:43:47.974 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:47.969+0000 7f8274a0d700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8270102040 con 0x7f82700fed90 2023-03-09T14:43:47.974 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:47.969+0000 7f8274a0d700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82701021b0 con 0x7f82700fff80 2023-03-09T14:43:47.974 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:47.969+0000 7f8274a0d700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f827019ffe0 con 0x7f8270100910 2023-03-09T14:43:47.975 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:47.969+0000 7f826e59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f82700fed90 0x7f8270102d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:43:47.975 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:47.969+0000 7f826e59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f82700fed90 0x7f8270102d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36346/0 (socket says 172.21.15.38:36346) 2023-03-09T14:43:47.975 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:47.969+0000 7f826e59c700 1 -- 172.21.15.38:0/270353100 learned_addr learned my addr 172.21.15.38:0/270353100 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:43:47.975 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:47.969+0000 7f826dd9b700 1 --2- 172.21.15.38:0/270353100 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f82700fff80 0x7f8270101360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:43:47.975 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:47.969+0000 7f826ed9d700 1 --2- 172.21.15.38:0/270353100 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8270100910 0x7f82701018a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:43:47.975 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:47.969+0000 7f826e59c700 1 -- 172.21.15.38:0/270353100 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8270100910 msgr2=0x7f82701018a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:43:47.976 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:47.969+0000 7f826e59c700 1 --2- 172.21.15.38:0/270353100 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8270100910 0x7f82701018a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:47.976 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:47.969+0000 7f826e59c700 1 -- 172.21.15.38:0/270353100 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f82700fff80 msgr2=0x7f8270101360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:43:47.976 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:47.969+0000 7f826e59c700 1 --2- 172.21.15.38:0/270353100 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f82700fff80 0x7f8270101360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:47.976 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:47.969+0000 7f826e59c700 1 -- 172.21.15.38:0/270353100 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8268009d00 con 0x7f82700fed90 2023-03-09T14:43:47.976 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:47.969+0000 7f826dd9b700 1 --2- 172.21.15.38:0/270353100 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f82700fff80 0x7f8270101360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-09T14:43:47.976 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:47.969+0000 7f826e59c700 1 --2- 172.21.15.38:0/270353100 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f82700fed90 0x7f8270102d20 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f826000d980 tx=0x7f826000dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:43:47.976 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:47.969+0000 7f82677fe700 1 -- 172.21.15.38:0/270353100 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f826000de60 con 0x7f82700fed90 2023-03-09T14:43:47.977 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:47.969+0000 7f82677fe700 1 -- 172.21.15.38:0/270353100 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f82600044e0 con 0x7f82700fed90 2023-03-09T14:43:47.977 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:47.969+0000 7f8274a0d700 1 -- 172.21.15.38:0/270353100 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f827010a770 con 0x7f82700fed90 2023-03-09T14:43:47.977 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:47.969+0000 7f82677fe700 1 -- 172.21.15.38:0/270353100 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8260003be0 con 0x7f82700fed90 2023-03-09T14:43:47.977 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:47.969+0000 7f8274a0d700 1 -- 172.21.15.38:0/270353100 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f827010a5e0 con 0x7f82700fed90 2023-03-09T14:43:47.977 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:47.970+0000 7f8274a0d700 1 -- 172.21.15.38:0/270353100 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82701a0730 con 0x7f82700fed90 2023-03-09T14:43:47.977 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:47.971+0000 7f82677fe700 1 -- 172.21.15.38:0/270353100 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 133198+0+0 (secure 0 0 0) 0x7f826000b030 con 0x7f82700fed90 2023-03-09T14:43:47.978 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:47.972+0000 7f82677fe700 1 --2- 172.21.15.38:0/270353100 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f825c09f3c0 0x7f825c0a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:43:47.978 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:47.972+0000 7f82677fe700 1 -- 172.21.15.38:0/270353100 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f82600c9550 con 0x7f82700fed90 2023-03-09T14:43:47.978 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:47.972+0000 7f826dd9b700 1 --2- 172.21.15.38:0/270353100 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f825c09f3c0 0x7f825c0a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:43:47.978 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:47.972+0000 7f826dd9b700 1 --2- 172.21.15.38:0/270353100 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f825c09f3c0 0x7f825c0a1870 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8258007900 tx=0x7f8258008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:43:47.979 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:47.976+0000 7f82677fe700 1 -- 172.21.15.38:0/270353100 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f8260093e00 con 0x7f82700fed90 2023-03-09T14:43:48.185 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.183+0000 7f8274a0d700 1 -- 172.21.15.38:0/270353100 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f82701a0ce0 con 0x7f82700fed90 2023-03-09T14:43:48.186 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.184+0000 7f82677fe700 1 -- 172.21.15.38:0/270353100 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) v1 ==== 74+0+18435 (secure 0 0 0) 0x7f8260009940 con 0x7f82700fed90 2023-03-09T14:43:48.186 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:43:48.187 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":79,"fsid":"c6733fe2-be87-11ed-9af2-001a4aab830c","created":"2023-03-09T14:37:14.947981+0000","modified":"2023-03-09T14:43:38.269957+0000","last_up_change":"2023-03-09T14:43:37.198407+0000","last_in_change":"2023-03-09T14:43:19.550566+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-09T14:40:23.196665+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51477039-15ee-4707-a34d-4ecfa18bebde","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6803","nonce":1508072290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6805","nonce":1508072290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6809","nonce":1508072290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6807","nonce":1508072290}]},"public_addr":"172.21.15.38:6803/1508072290","cluster_addr":"172.21.15.38:6805/1508072290","heartbeat_back_addr":"172.21.15.38:6809/1508072290","heartbeat_front_addr":"172.21.15.38:6807/1508072290","state":["exists","up"]},{"osd":1,"uuid":"71f8a718-b0a9-4f39-bc54-267aa42a2578","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6801","nonce":3929268773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6803","nonce":3929268773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6807","nonce":3929268773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6805","nonce":3929268773}]},"public_addr":"172.21.15.122:6801/3929268773","cluster_addr":"172.21.15.122:6803/3929268773","heartbeat_back_addr":"172.21.15.122:6807/3929268773","heartbeat_front_addr":"172.21.15.122:6805/3929268773","state":["exists","up"]},{"osd":2,"uuid":"961a2cbe-140e-437f-b131-1fcac573b19c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6801","nonce":1310630803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6803","nonce":1310630803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6807","nonce":1310630803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6805","nonce":1310630803}]},"public_addr":"172.21.15.181:6801/1310630803","cluster_addr":"172.21.15.181:6803/1310630803","heartbeat_back_addr":"172.21.15.181:6807/1310630803","heartbeat_front_addr":"172.21.15.181:6805/1310630803","state":["exists","up"]},{"osd":3,"uuid":"969df8fa-2d19-42c7-9f2a-5c5fac15d4a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6811","nonce":2719814100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6813","nonce":2719814100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6817","nonce":2719814100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6815","nonce":2719814100}]},"public_addr":"172.21.15.38:6811/2719814100","cluster_addr":"172.21.15.38:6813/2719814100","heartbeat_back_addr":"172.21.15.38:6817/2719814100","heartbeat_front_addr":"172.21.15.38:6815/2719814100","state":["exists","up"]},{"osd":4,"uuid":"44a6dce3-e06e-4cbd-9404-62a86d01ab6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6809","nonce":2480764994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6811","nonce":2480764994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6815","nonce":2480764994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6813","nonce":2480764994}]},"public_addr":"172.21.15.122:6809/2480764994","cluster_addr":"172.21.15.122:6811/2480764994","heartbeat_back_addr":"172.21.15.122:6815/2480764994","heartbeat_front_addr":"172.21.15.122:6813/2480764994","state":["exists","up"]},{"osd":5,"uuid":"49749574-c00c-4e3a-80e6-e3686df58d5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6809","nonce":3141346246}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6811","nonce":3141346246}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6815","nonce":3141346246}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6813","nonce":3141346246}]},"public_addr":"172.21.15.181:6809/3141346246","cluster_addr":"172.21.15.181:6811/3141346246","heartbeat_back_addr":"172.21.15.181:6815/3141346246","heartbeat_front_addr":"172.21.15.181:6813/3141346246","state":["exists","up"]},{"osd":6,"uuid":"61c0437e-3f51-45a9-8a61-e98eea0093f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6819","nonce":2122788957}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6821","nonce":2122788957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6825","nonce":2122788957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6823","nonce":2122788957}]},"public_addr":"172.21.15.38:6819/2122788957","cluster_addr":"172.21.15.38:6821/2122788957","heartbeat_back_addr":"172.21.15.38:6825/2122788957","heartbeat_front_addr":"172.21.15.38:6823/2122788957","state":["exists","up"]},{"osd":7,"uuid":"231fecdd-939f-4c7b-83a1-c4a1cffd7139","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6817","nonce":656874139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6819","nonce":656874139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6823","nonce":656874139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6821","nonce":656874139}]},"public_addr":"172.21.15.122:6817/656874139","cluster_addr":"172.21.15.122:6819/656874139","heartbeat_back_addr":"172.21.15.122:6823/656874139","heartbeat_front_addr":"172.21.15.122:6821/656874139","state":["exists","up"]},{"osd":8,"uuid":"297e6d6d-85fd-43ce-9028-5e9dbe55c36e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6817","nonce":755000703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6819","nonce":755000703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6823","nonce":755000703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6821","nonce":755000703}]},"public_addr":"172.21.15.181:6817/755000703","cluster_addr":"172.21.15.181:6819/755000703","heartbeat_back_addr":"172.21.15.181:6823/755000703","heartbeat_front_addr":"172.21.15.181:6821/755000703","state":["exists","up"]},{"osd":9,"uuid":"94cc0355-5a1a-4cfc-bd0c-562b6b41127e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6827","nonce":2624555969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6829","nonce":2624555969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6833","nonce":2624555969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6831","nonce":2624555969}]},"public_addr":"172.21.15.38:6827/2624555969","cluster_addr":"172.21.15.38:6829/2624555969","heartbeat_back_addr":"172.21.15.38:6833/2624555969","heartbeat_front_addr":"172.21.15.38:6831/2624555969","state":["exists","up"]},{"osd":10,"uuid":"f4f2e042-8d92-46bf-93b8-8b0f04795c2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6825","nonce":1427785280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6827","nonce":1427785280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6831","nonce":1427785280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6829","nonce":1427785280}]},"public_addr":"172.21.15.122:6825/1427785280","cluster_addr":"172.21.15.122:6827/1427785280","heartbeat_back_addr":"172.21.15.122:6831/1427785280","heartbeat_front_addr":"172.21.15.122:6829/1427785280","state":["exists","up"]},{"osd":11,"uuid":"07444936-eae5-4097-a799-60294cafe80b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6825","nonce":3183488640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6827","nonce":3183488640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6831","nonce":3183488640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6829","nonce":3183488640}]},"public_addr":"172.21.15.181:6825/3183488640","cluster_addr":"172.21.15.181:6827/3183488640","heartbeat_back_addr":"172.21.15.181:6831/3183488640","heartbeat_front_addr":"172.21.15.181:6829/3183488640","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:39:37.657556+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:39:58.960074+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:40:20.326952+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:40:42.244387+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:41:03.946973+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:41:25.097985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:41:46.877916+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:42:08.413827+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:42:29.822009+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:42:51.776684+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:43:13.937249+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:43:35.569199+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/4272288735":"2023-03-10T14:38:14.612767+0000","172.21.15.38:6801/3381625669":"2023-03-10T14:38:14.612767+0000","172.21.15.38:6800/3381625669":"2023-03-10T14:38:14.612767+0000","172.21.15.38:6800/3060474342":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/4253918615":"2023-03-10T14:38:14.612767+0000","172.21.15.38:0/3496083181":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/245602368":"2023-03-10T14:37:47.517775+0000","172.21.15.38:6801/3060474342":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/3622589182":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/1430302251":"2023-03-10T14:38:14.612767+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-09T14:43:48.189 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.186+0000 7f8274a0d700 1 -- 172.21.15.38:0/270353100 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f825c09f3c0 msgr2=0x7f825c0a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:43:48.190 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.186+0000 7f8274a0d700 1 --2- 172.21.15.38:0/270353100 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f825c09f3c0 0x7f825c0a1870 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8258007900 tx=0x7f8258008040 comp rx=0 tx=0).stop 2023-03-09T14:43:48.190 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.186+0000 7f8274a0d700 1 -- 172.21.15.38:0/270353100 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f82700fed90 msgr2=0x7f8270102d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:43:48.190 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.186+0000 7f8274a0d700 1 --2- 172.21.15.38:0/270353100 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f82700fed90 0x7f8270102d20 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f826000d980 tx=0x7f826000dd40 comp rx=0 tx=0).stop 2023-03-09T14:43:48.191 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.186+0000 7f8274a0d700 1 -- 172.21.15.38:0/270353100 shutdown_connections 2023-03-09T14:43:48.191 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.186+0000 7f8274a0d700 1 --2- 172.21.15.38:0/270353100 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f82700fed90 0x7f8270102d20 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:48.191 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.186+0000 7f8274a0d700 1 --2- 172.21.15.38:0/270353100 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f825c09f3c0 0x7f825c0a1870 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:48.192 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.186+0000 7f8274a0d700 1 --2- 172.21.15.38:0/270353100 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8270100910 0x7f82701018a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:48.192 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.186+0000 7f8274a0d700 1 --2- 172.21.15.38:0/270353100 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f82700fff80 0x7f8270101360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:48.192 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.186+0000 7f8274a0d700 1 -- 172.21.15.38:0/270353100 >> 172.21.15.38:0/270353100 conn(0x7f82700fa370 msgr2=0x7f8270068ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:43:48.192 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.186+0000 7f8274a0d700 1 -- 172.21.15.38:0/270353100 shutdown_connections 2023-03-09T14:43:48.192 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.186+0000 7f8274a0d700 1 -- 172.21.15.38:0/270353100 wait complete. 2023-03-09T14:43:48.201 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-09T14:40:23.196665+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-03-09T14:43:48.202 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-09T14:43:48.331 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.328+0000 7f0ef92fe700 1 -- 172.21.15.38:0/2033484898 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ef40fdf10 msgr2=0x7f0ef40fe360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:43:48.331 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.328+0000 7f0ef92fe700 1 --2- 172.21.15.38:0/2033484898 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ef40fdf10 0x7f0ef40fe360 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8009990 tx=0x7f0ee8009ca0 comp rx=0 tx=0).stop 2023-03-09T14:43:48.331 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.328+0000 7f0ef92fe700 1 -- 172.21.15.38:0/2033484898 shutdown_connections 2023-03-09T14:43:48.332 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.328+0000 7f0ef92fe700 1 --2- 172.21.15.38:0/2033484898 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0ef40fe8a0 0x7f0ef4108480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:48.332 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.328+0000 7f0ef92fe700 1 --2- 172.21.15.38:0/2033484898 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0ef41014a0 0x7f0ef41018b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:48.332 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.328+0000 7f0ef92fe700 1 --2- 172.21.15.38:0/2033484898 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ef40fdf10 0x7f0ef40fe360 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:48.332 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.328+0000 7f0ef92fe700 1 -- 172.21.15.38:0/2033484898 >> 172.21.15.38:0/2033484898 conn(0x7f0ef40f9b60 msgr2=0x7f0ef40fbf90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:43:48.332 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.328+0000 7f0ef92fe700 1 -- 172.21.15.38:0/2033484898 shutdown_connections 2023-03-09T14:43:48.332 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.328+0000 7f0ef92fe700 1 -- 172.21.15.38:0/2033484898 wait complete. 2023-03-09T14:43:48.332 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.329+0000 7f0ef92fe700 1 Processor -- start 2023-03-09T14:43:48.333 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.329+0000 7f0ef92fe700 1 -- start start 2023-03-09T14:43:48.334 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.329+0000 7f0ef92fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ef40fdf10 0x7f0ef4197dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:43:48.334 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.329+0000 7f0ef92fe700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0ef40fe8a0 0x7f0ef4198300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:43:48.335 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.329+0000 7f0ef92fe700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0ef41014a0 0x7f0ef419f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:43:48.335 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.329+0000 7f0ef92fe700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef419f900 con 0x7f0ef40fdf10 2023-03-09T14:43:48.335 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.329+0000 7f0ef92fe700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef419fa70 con 0x7f0ef40fe8a0 2023-03-09T14:43:48.335 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.329+0000 7f0ef92fe700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef419fc30 con 0x7f0ef41014a0 2023-03-09T14:43:48.335 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.329+0000 7f0ef2ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ef40fdf10 0x7f0ef4197dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:43:48.335 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.329+0000 7f0ef2ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ef40fdf10 0x7f0ef4197dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36360/0 (socket says 172.21.15.38:36360) 2023-03-09T14:43:48.335 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.329+0000 7f0ef2ffd700 1 -- 172.21.15.38:0/473231824 learned_addr learned my addr 172.21.15.38:0/473231824 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:43:48.335 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.329+0000 7f0ef37fe700 1 --2- 172.21.15.38:0/473231824 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0ef41014a0 0x7f0ef419f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:43:48.336 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.329+0000 7f0ef27fc700 1 --2- 172.21.15.38:0/473231824 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0ef40fe8a0 0x7f0ef4198300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:43:48.336 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.330+0000 7f0ef2ffd700 1 -- 172.21.15.38:0/473231824 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0ef41014a0 msgr2=0x7f0ef419f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:43:48.336 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.330+0000 7f0ef2ffd700 1 --2- 172.21.15.38:0/473231824 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0ef41014a0 0x7f0ef419f330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:48.336 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.330+0000 7f0ef2ffd700 1 -- 172.21.15.38:0/473231824 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0ef40fe8a0 msgr2=0x7f0ef4198300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:43:48.336 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.330+0000 7f0ef2ffd700 1 --2- 172.21.15.38:0/473231824 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0ef40fe8a0 0x7f0ef4198300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:48.336 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.330+0000 7f0ef2ffd700 1 -- 172.21.15.38:0/473231824 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ee80095f0 con 0x7f0ef40fdf10 2023-03-09T14:43:48.336 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.330+0000 7f0ef2ffd700 1 --2- 172.21.15.38:0/473231824 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ef40fdf10 0x7f0ef4197dc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f0ee400c990 tx=0x7f0ee400cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:43:48.337 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.330+0000 7f0edbfff700 1 -- 172.21.15.38:0/473231824 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee4007b00 con 0x7f0ef40fdf10 2023-03-09T14:43:48.337 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.330+0000 7f0edbfff700 1 -- 172.21.15.38:0/473231824 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0ee4007c60 con 0x7f0ef40fdf10 2023-03-09T14:43:48.337 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.330+0000 7f0ef92fe700 1 -- 172.21.15.38:0/473231824 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ef410b860 con 0x7f0ef40fdf10 2023-03-09T14:43:48.337 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.330+0000 7f0edbfff700 1 -- 172.21.15.38:0/473231824 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee4018660 con 0x7f0ef40fdf10 2023-03-09T14:43:48.337 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.330+0000 7f0ef92fe700 1 -- 172.21.15.38:0/473231824 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ef410b6d0 con 0x7f0ef40fdf10 2023-03-09T14:43:48.338 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.331+0000 7f0ef92fe700 1 -- 172.21.15.38:0/473231824 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ef41a0290 con 0x7f0ef40fdf10 2023-03-09T14:43:48.338 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.332+0000 7f0edbfff700 1 -- 172.21.15.38:0/473231824 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 133198+0+0 (secure 0 0 0) 0x7f0ee400f450 con 0x7f0ef40fdf10 2023-03-09T14:43:48.338 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.333+0000 7f0edbfff700 1 --2- 172.21.15.38:0/473231824 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f0edc09f3c0 0x7f0edc0a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:43:48.339 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.333+0000 7f0ef27fc700 1 --2- 172.21.15.38:0/473231824 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f0edc09f3c0 0x7f0edc0a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:43:48.339 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.333+0000 7f0edbfff700 1 -- 172.21.15.38:0/473231824 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f0ee40c9ba0 con 0x7f0ef40fdf10 2023-03-09T14:43:48.339 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.333+0000 7f0ef27fc700 1 --2- 172.21.15.38:0/473231824 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f0edc09f3c0 0x7f0edc0a1870 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f0ee800b5c0 tx=0x7f0ee8005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:43:48.339 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.336+0000 7f0edbfff700 1 -- 172.21.15.38:0/473231824 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f0ee40943f0 con 0x7f0ef40fdf10 2023-03-09T14:43:48.475 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:48 smithi181 ceph-mon[94189]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-03-09T14:43:48.475 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:48 smithi181 ceph-mon[94189]: from='client.24388 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi038=a;smithi038=d;smithi038=g;smithi038=j;smithi122=b;smithi122=e;smithi122=h;smithi122=k;smithi181=c;smithi181=f;smithi181=i;smithi181=l", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:43:48.476 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:48 smithi181 ceph-mon[94189]: Saving service mds.all spec with placement smithi038=a;smithi038=d;smithi038=g;smithi038=j;smithi122=b;smithi122=e;smithi122=h;smithi122=k;smithi181=c;smithi181=f;smithi181=i;smithi181=l;count:12 2023-03-09T14:43:48.476 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:48 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:48.476 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:48 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:43:48.476 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:48 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:43:48.476 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:48 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:43:48.476 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:48 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:48.477 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:48 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:48.477 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:48 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-03-09T14:43:48.477 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:48 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-03-09T14:43:48.477 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:48 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:43:48.477 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:48 smithi181 ceph-mon[94189]: Deploying daemon mds.l on smithi181 2023-03-09T14:43:48.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:48 smithi122 ceph-mon[93869]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-03-09T14:43:48.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:48 smithi122 ceph-mon[93869]: from='client.24388 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi038=a;smithi038=d;smithi038=g;smithi038=j;smithi122=b;smithi122=e;smithi122=h;smithi122=k;smithi181=c;smithi181=f;smithi181=i;smithi181=l", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:43:48.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:48 smithi122 ceph-mon[93869]: Saving service mds.all spec with placement smithi038=a;smithi038=d;smithi038=g;smithi038=j;smithi122=b;smithi122=e;smithi122=h;smithi122=k;smithi181=c;smithi181=f;smithi181=i;smithi181=l;count:12 2023-03-09T14:43:48.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:48 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:48.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:48 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:43:48.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:48 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:43:48.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:48 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:43:48.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:48 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:48.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:48 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:48.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:48 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-03-09T14:43:48.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:48 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-03-09T14:43:48.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:48 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:43:48.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:48 smithi122 ceph-mon[93869]: Deploying daemon mds.l on smithi181 2023-03-09T14:43:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:48 smithi038 ceph-mon[92422]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-03-09T14:43:48.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:48 smithi038 ceph-mon[92422]: from='client.24388 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi038=a;smithi038=d;smithi038=g;smithi038=j;smithi122=b;smithi122=e;smithi122=h;smithi122=k;smithi181=c;smithi181=f;smithi181=i;smithi181=l", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:43:48.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:48 smithi038 ceph-mon[92422]: Saving service mds.all spec with placement smithi038=a;smithi038=d;smithi038=g;smithi038=j;smithi122=b;smithi122=e;smithi122=h;smithi122=k;smithi181=c;smithi181=f;smithi181=i;smithi181=l;count:12 2023-03-09T14:43:48.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:48 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:48.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:48 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:43:48.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:48 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:43:48.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:48 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:43:48.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:48 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:48.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:48 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:48.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:48 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-03-09T14:43:48.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:48 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-03-09T14:43:48.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:48 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:43:48.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:48 smithi038 ceph-mon[92422]: Deploying daemon mds.l on smithi181 2023-03-09T14:43:48.542 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.540+0000 7f0ef92fe700 1 -- 172.21.15.38:0/473231824 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0ef41a0840 con 0x7f0ef40fdf10 2023-03-09T14:43:48.543 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.541+0000 7f0edbfff700 1 -- 172.21.15.38:0/473231824 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v79) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0ee4097a10 con 0x7f0ef40fdf10 2023-03-09T14:43:48.544 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 1 2023-03-09T14:43:48.546 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.543+0000 7f0ef92fe700 1 -- 172.21.15.38:0/473231824 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f0edc09f3c0 msgr2=0x7f0edc0a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:43:48.546 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.543+0000 7f0ef92fe700 1 --2- 172.21.15.38:0/473231824 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f0edc09f3c0 0x7f0edc0a1870 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f0ee800b5c0 tx=0x7f0ee8005fb0 comp rx=0 tx=0).stop 2023-03-09T14:43:48.546 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.543+0000 7f0ef92fe700 1 -- 172.21.15.38:0/473231824 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ef40fdf10 msgr2=0x7f0ef4197dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:43:48.546 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.543+0000 7f0ef92fe700 1 --2- 172.21.15.38:0/473231824 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ef40fdf10 0x7f0ef4197dc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f0ee400c990 tx=0x7f0ee400cca0 comp rx=0 tx=0).stop 2023-03-09T14:43:48.546 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.543+0000 7f0ef92fe700 1 -- 172.21.15.38:0/473231824 shutdown_connections 2023-03-09T14:43:48.547 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.543+0000 7f0ef92fe700 1 --2- 172.21.15.38:0/473231824 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ef40fdf10 0x7f0ef4197dc0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:48.547 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.543+0000 7f0ef92fe700 1 --2- 172.21.15.38:0/473231824 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f0edc09f3c0 0x7f0edc0a1870 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:48.547 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.543+0000 7f0ef92fe700 1 --2- 172.21.15.38:0/473231824 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0ef41014a0 0x7f0ef419f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:48.547 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.543+0000 7f0ef92fe700 1 --2- 172.21.15.38:0/473231824 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0ef40fe8a0 0x7f0ef4198300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:48.547 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.543+0000 7f0ef92fe700 1 -- 172.21.15.38:0/473231824 >> 172.21.15.38:0/473231824 conn(0x7f0ef40f9b60 msgr2=0x7f0ef40fbd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:43:48.548 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.543+0000 7f0ef92fe700 1 -- 172.21.15.38:0/473231824 shutdown_connections 2023-03-09T14:43:48.548 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.543+0000 7f0ef92fe700 1 -- 172.21.15.38:0/473231824 wait complete. 2023-03-09T14:43:48.558 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-09T14:43:48.684 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.680+0000 7f71bbd82700 1 -- 172.21.15.38:0/1385311210 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f71b40fff70 msgr2=0x7f71b41003c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:43:48.684 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.680+0000 7f71bbd82700 1 --2- 172.21.15.38:0/1385311210 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f71b40fff70 0x7f71b41003c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f71a4009990 tx=0x7f71a4009ca0 comp rx=0 tx=0).stop 2023-03-09T14:43:48.684 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.680+0000 7f71bbd82700 1 -- 172.21.15.38:0/1385311210 shutdown_connections 2023-03-09T14:43:48.684 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.680+0000 7f71bbd82700 1 --2- 172.21.15.38:0/1385311210 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f71b4100900 0x7f71b41071c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:48.684 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.680+0000 7f71bbd82700 1 --2- 172.21.15.38:0/1385311210 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71b40fed80 0x7f71b40ff190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:48.685 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.680+0000 7f71bbd82700 1 --2- 172.21.15.38:0/1385311210 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f71b40fff70 0x7f71b41003c0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:48.685 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.680+0000 7f71bbd82700 1 -- 172.21.15.38:0/1385311210 >> 172.21.15.38:0/1385311210 conn(0x7f71b40fa380 msgr2=0x7f71b40fc7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:43:48.685 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.680+0000 7f71bbd82700 1 -- 172.21.15.38:0/1385311210 shutdown_connections 2023-03-09T14:43:48.685 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.680+0000 7f71bbd82700 1 -- 172.21.15.38:0/1385311210 wait complete. 2023-03-09T14:43:48.685 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.681+0000 7f71bbd82700 1 Processor -- start 2023-03-09T14:43:48.685 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.681+0000 7f71bbd82700 1 -- start start 2023-03-09T14:43:48.686 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.681+0000 7f71bbd82700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f71b40fed80 0x7f71b4197e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:43:48.686 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.681+0000 7f71bbd82700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71b40fff70 0x7f71b4198370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:43:48.686 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.681+0000 7f71bbd82700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f71b4100900 0x7f71b419f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:43:48.686 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.681+0000 7f71bbd82700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71b419f9b0 con 0x7f71b4100900 2023-03-09T14:43:48.686 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.681+0000 7f71bbd82700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71b419faf0 con 0x7f71b40fed80 2023-03-09T14:43:48.686 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.681+0000 7f71bbd82700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71b419fc80 con 0x7f71b40fff70 2023-03-09T14:43:48.686 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.681+0000 7f71ba31f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f71b4100900 0x7f71b419f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:43:48.686 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.681+0000 7f71ba31f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f71b4100900 0x7f71b419f3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36378/0 (socket says 172.21.15.38:36378) 2023-03-09T14:43:48.687 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.681+0000 7f71ba31f700 1 -- 172.21.15.38:0/3519316409 learned_addr learned my addr 172.21.15.38:0/3519316409 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:43:48.687 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.681+0000 7f71b931d700 1 --2- 172.21.15.38:0/3519316409 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71b40fff70 0x7f71b4198370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:43:48.687 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.681+0000 7f71b9b1e700 1 --2- 172.21.15.38:0/3519316409 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f71b40fed80 0x7f71b4197e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:43:48.687 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.681+0000 7f71ba31f700 1 -- 172.21.15.38:0/3519316409 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71b40fff70 msgr2=0x7f71b4198370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:43:48.687 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.681+0000 7f71ba31f700 1 --2- 172.21.15.38:0/3519316409 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71b40fff70 0x7f71b4198370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:48.687 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.681+0000 7f71ba31f700 1 -- 172.21.15.38:0/3519316409 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f71b40fed80 msgr2=0x7f71b4197e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:43:48.687 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.681+0000 7f71ba31f700 1 --2- 172.21.15.38:0/3519316409 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f71b40fed80 0x7f71b4197e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:48.687 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.681+0000 7f71ba31f700 1 -- 172.21.15.38:0/3519316409 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71a40095f0 con 0x7f71b4100900 2023-03-09T14:43:48.688 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.681+0000 7f71ba31f700 1 --2- 172.21.15.38:0/3519316409 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f71b4100900 0x7f71b419f3e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f71ac007ee0 tx=0x7f71ac00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:43:48.688 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.682+0000 7f71aaffd700 1 -- 172.21.15.38:0/3519316409 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71ac017070 con 0x7f71b4100900 2023-03-09T14:43:48.688 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.682+0000 7f71aaffd700 1 -- 172.21.15.38:0/3519316409 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f71ac00f040 con 0x7f71b4100900 2023-03-09T14:43:48.688 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.682+0000 7f71bbd82700 1 -- 172.21.15.38:0/3519316409 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71b410a730 con 0x7f71b4100900 2023-03-09T14:43:48.688 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.682+0000 7f71aaffd700 1 -- 172.21.15.38:0/3519316409 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71ac005740 con 0x7f71b4100900 2023-03-09T14:43:48.688 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.682+0000 7f71bbd82700 1 -- 172.21.15.38:0/3519316409 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71b410a5a0 con 0x7f71b4100900 2023-03-09T14:43:48.690 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.683+0000 7f71bbd82700 1 -- 172.21.15.38:0/3519316409 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71b4067660 con 0x7f71b4100900 2023-03-09T14:43:48.690 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.684+0000 7f71aaffd700 1 -- 172.21.15.38:0/3519316409 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 133198+0+0 (secure 0 0 0) 0x7f71ac023070 con 0x7f71b4100900 2023-03-09T14:43:48.690 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.685+0000 7f71aaffd700 1 --2- 172.21.15.38:0/3519316409 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f71a009f3c0 0x7f71a00a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:43:48.690 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.685+0000 7f71b9b1e700 1 --2- 172.21.15.38:0/3519316409 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f71a009f3c0 0x7f71a00a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:43:48.690 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.685+0000 7f71aaffd700 1 -- 172.21.15.38:0/3519316409 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f71ac0c9410 con 0x7f71b4100900 2023-03-09T14:43:48.691 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.685+0000 7f71b9b1e700 1 --2- 172.21.15.38:0/3519316409 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f71a009f3c0 0x7f71a00a1870 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f71b0005950 tx=0x7f71b00058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:43:48.691 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.688+0000 7f71aaffd700 1 -- 172.21.15.38:0/3519316409 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f71ac093cc0 con 0x7f71b4100900 2023-03-09T14:43:48.732 INFO:journalctl@ceph.mds.l.smithi181.stdout:Mar 09 14:43:48 smithi181 systemd[1]: Starting Ceph mds.l for c6733fe2-be87-11ed-9af2-001a4aab830c... 2023-03-09T14:43:48.937 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.935+0000 7f71bbd82700 1 -- 172.21.15.38:0/3519316409 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f71b41a07c0 con 0x7f71b4100900 2023-03-09T14:43:48.938 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.935+0000 7f71aaffd700 1 -- 172.21.15.38:0/3519316409 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) v1 ==== 93+0+501 (secure 0 0 0) 0x7f71ac021020 con 0x7f71b4100900 2023-03-09T14:43:48.938 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:43:48.938 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2023-03-09T14:43:48.939 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.937+0000 7f71bbd82700 1 -- 172.21.15.38:0/3519316409 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f71a009f3c0 msgr2=0x7f71a00a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:43:48.940 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.937+0000 7f71bbd82700 1 --2- 172.21.15.38:0/3519316409 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f71a009f3c0 0x7f71a00a1870 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f71b0005950 tx=0x7f71b00058e0 comp rx=0 tx=0).stop 2023-03-09T14:43:48.940 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.937+0000 7f71bbd82700 1 -- 172.21.15.38:0/3519316409 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f71b4100900 msgr2=0x7f71b419f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:43:48.940 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.937+0000 7f71bbd82700 1 --2- 172.21.15.38:0/3519316409 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f71b4100900 0x7f71b419f3e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f71ac007ee0 tx=0x7f71ac00d3b0 comp rx=0 tx=0).stop 2023-03-09T14:43:48.940 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.937+0000 7f71bbd82700 1 -- 172.21.15.38:0/3519316409 shutdown_connections 2023-03-09T14:43:48.940 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.937+0000 7f71bbd82700 1 --2- 172.21.15.38:0/3519316409 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f71b4100900 0x7f71b419f3e0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:48.940 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.937+0000 7f71bbd82700 1 --2- 172.21.15.38:0/3519316409 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f71a009f3c0 0x7f71a00a1870 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:48.940 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.937+0000 7f71bbd82700 1 --2- 172.21.15.38:0/3519316409 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f71b40fed80 0x7f71b4197e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:48.941 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.937+0000 7f71bbd82700 1 --2- 172.21.15.38:0/3519316409 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f71b40fff70 0x7f71b4198370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:48.941 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.937+0000 7f71bbd82700 1 -- 172.21.15.38:0/3519316409 >> 172.21.15.38:0/3519316409 conn(0x7f71b40fa380 msgr2=0x7f71b4067a60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:43:48.941 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.937+0000 7f71bbd82700 1 -- 172.21.15.38:0/3519316409 shutdown_connections 2023-03-09T14:43:48.941 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:48.937+0000 7f71bbd82700 1 -- 172.21.15.38:0/3519316409 wait complete. 2023-03-09T14:43:48.941 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 1 2023-03-09T14:43:49.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:49 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/270353100' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-03-09T14:43:49.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:49 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/473231824' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-03-09T14:43:49.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:49 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/3519316409' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-03-09T14:43:49.263 INFO:journalctl@ceph.mds.l.smithi181.stdout:Mar 09 14:43:49 smithi181 podman[119307]: 2023-03-09T14:43:49.264 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:49 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/270353100' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-03-09T14:43:49.264 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:49 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/473231824' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-03-09T14:43:49.264 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:49 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/3519316409' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-03-09T14:43:49.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:49 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/270353100' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-03-09T14:43:49.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:49 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/473231824' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-03-09T14:43:49.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:49 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/3519316409' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-03-09T14:43:49.731 INFO:journalctl@ceph.mds.l.smithi181.stdout:Mar 09 14:43:49 smithi181 bash[119307]: 0e1798f50c1a9d5e09d8eb4438a0bc4aae58795fb0f9806307efd31427f7be81 2023-03-09T14:43:49.732 INFO:journalctl@ceph.mds.l.smithi181.stdout:Mar 09 14:43:49 smithi181 ceph-mds[119348]: -- 172.21.15.181:0/2459097006 <== mon.1 v2:172.21.15.181:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f2a656cb60 con 0x55f2a6540c00 2023-03-09T14:43:49.732 INFO:journalctl@ceph.mds.l.smithi181.stdout:Mar 09 14:43:49 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mds-l[119324]: starting mds.l at 2023-03-09T14:43:49.732 INFO:journalctl@ceph.mds.l.smithi181.stdout:Mar 09 14:43:49 smithi181 systemd[1]: Started Ceph mds.l for c6733fe2-be87-11ed-9af2-001a4aab830c. 2023-03-09T14:43:50.366 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:50 smithi122 ceph-mon[93869]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-03-09T14:43:50.366 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:50 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6832/415305571,v1:172.21.15.181:6833/415305571] up:boot 2023-03-09T14:43:50.366 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:50 smithi122 ceph-mon[93869]: fsmap 1 up:standby 2023-03-09T14:43:50.366 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:50 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-03-09T14:43:50.367 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:50 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:50.367 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:50 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:50.367 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:50 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-03-09T14:43:50.367 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:50 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-03-09T14:43:50.367 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:50 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:43:50.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:50 smithi181 ceph-mon[94189]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-03-09T14:43:50.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:50 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6832/415305571,v1:172.21.15.181:6833/415305571] up:boot 2023-03-09T14:43:50.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:50 smithi181 ceph-mon[94189]: fsmap 1 up:standby 2023-03-09T14:43:50.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:50 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-03-09T14:43:50.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:50 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:50.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:50 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:50.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:50 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-03-09T14:43:50.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:50 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-03-09T14:43:50.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:50 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:43:50.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:50 smithi038 ceph-mon[92422]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-03-09T14:43:50.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:50 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6832/415305571,v1:172.21.15.181:6833/415305571] up:boot 2023-03-09T14:43:50.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:50 smithi038 ceph-mon[92422]: fsmap 1 up:standby 2023-03-09T14:43:50.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:50 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-03-09T14:43:50.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:50 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:50.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:50 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:50.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:50 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-03-09T14:43:50.753 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:50 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-03-09T14:43:50.753 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:50 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:43:50.953 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-09T14:43:51.096 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:51.090+0000 7ffba494e700 1 -- 172.21.15.38:0/872892123 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffba01008d0 msgr2=0x7ffba0107190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:43:51.096 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:51.090+0000 7ffba494e700 1 --2- 172.21.15.38:0/872892123 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffba01008d0 0x7ffba0107190 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ffb9800b0a0 tx=0x7ffb9800b3b0 comp rx=0 tx=0).stop 2023-03-09T14:43:51.097 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:51.090+0000 7ffba494e700 1 -- 172.21.15.38:0/872892123 shutdown_connections 2023-03-09T14:43:51.097 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:51.090+0000 7ffba494e700 1 --2- 172.21.15.38:0/872892123 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffba01008d0 0x7ffba0107190 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:51.097 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:51.090+0000 7ffba494e700 1 --2- 172.21.15.38:0/872892123 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffba00fed50 0x7ffba00ff160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:51.098 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:51.090+0000 7ffba494e700 1 --2- 172.21.15.38:0/872892123 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffba00fff40 0x7ffba0100390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:51.098 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:51.090+0000 7ffba494e700 1 -- 172.21.15.38:0/872892123 >> 172.21.15.38:0/872892123 conn(0x7ffba00fa350 msgr2=0x7ffba00fc7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:43:51.098 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:51.090+0000 7ffba494e700 1 -- 172.21.15.38:0/872892123 shutdown_connections 2023-03-09T14:43:51.099 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:51.090+0000 7ffba494e700 1 -- 172.21.15.38:0/872892123 wait complete. 2023-03-09T14:43:51.099 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:51.091+0000 7ffba494e700 1 Processor -- start 2023-03-09T14:43:51.099 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:51.091+0000 7ffba494e700 1 -- start start 2023-03-09T14:43:51.099 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:51.092+0000 7ffba494e700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffba00fed50 0x7ffba0068610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:43:51.099 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:51.092+0000 7ffba494e700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffba00fff40 0x7ffba019a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:43:51.099 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:51.092+0000 7ffba494e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffba01008d0 0x7ffba019cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:43:51.099 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:51.092+0000 7ffba494e700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffba019d720 con 0x7ffba01008d0 2023-03-09T14:43:51.100 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:51.092+0000 7ffba494e700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffba019d890 con 0x7ffba00fed50 2023-03-09T14:43:51.100 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:51.092+0000 7ffba494e700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffba019da50 con 0x7ffba00fff40 2023-03-09T14:43:51.100 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:51.092+0000 7ffb9ed9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffba01008d0 0x7ffba019cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:43:51.100 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:51.092+0000 7ffb9ed9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffba01008d0 0x7ffba019cf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36392/0 (socket says 172.21.15.38:36392) 2023-03-09T14:43:51.100 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:51.092+0000 7ffb9ed9d700 1 -- 172.21.15.38:0/708048835 learned_addr learned my addr 172.21.15.38:0/708048835 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:43:51.101 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:51.092+0000 7ffb9e59c700 1 --2- 172.21.15.38:0/708048835 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffba00fed50 0x7ffba0068610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:43:51.101 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:51.092+0000 7ffb9dd9b700 1 --2- 172.21.15.38:0/708048835 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffba00fff40 0x7ffba019a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:43:51.101 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:51.092+0000 7ffb9ed9d700 1 -- 172.21.15.38:0/708048835 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffba00fff40 msgr2=0x7ffba019a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:43:51.101 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:51.092+0000 7ffb9ed9d700 1 --2- 172.21.15.38:0/708048835 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffba00fff40 0x7ffba019a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:51.101 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:51.092+0000 7ffb9ed9d700 1 -- 172.21.15.38:0/708048835 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffba00fed50 msgr2=0x7ffba0068610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:43:51.101 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:51.092+0000 7ffb9ed9d700 1 --2- 172.21.15.38:0/708048835 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffba00fed50 0x7ffba0068610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:51.101 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:51.092+0000 7ffb9ed9d700 1 -- 172.21.15.38:0/708048835 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb94009520 con 0x7ffba01008d0 2023-03-09T14:43:51.102 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:51.092+0000 7ffb9dd9b700 1 --2- 172.21.15.38:0/708048835 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffba00fff40 0x7ffba019a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:43:51.102 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:51.092+0000 7ffb9ed9d700 1 --2- 172.21.15.38:0/708048835 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffba01008d0 0x7ffba019cf80 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ffb9800b780 tx=0x7ffb9800bd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:43:51.102 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:51.092+0000 7ffb8f7fe700 1 -- 172.21.15.38:0/708048835 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb9800e040 con 0x7ffba01008d0 2023-03-09T14:43:51.102 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:51.093+0000 7ffb8f7fe700 1 -- 172.21.15.38:0/708048835 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffb98004e60 con 0x7ffba01008d0 2023-03-09T14:43:51.102 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:51.093+0000 7ffba494e700 1 -- 172.21.15.38:0/708048835 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb98009d00 con 0x7ffba01008d0 2023-03-09T14:43:51.102 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:51.093+0000 7ffb8f7fe700 1 -- 172.21.15.38:0/708048835 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb9801b740 con 0x7ffba01008d0 2023-03-09T14:43:51.102 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:51.093+0000 7ffba494e700 1 -- 172.21.15.38:0/708048835 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffba010a6c0 con 0x7ffba01008d0 2023-03-09T14:43:51.103 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:51.094+0000 7ffba494e700 1 -- 172.21.15.38:0/708048835 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffba010a850 con 0x7ffba01008d0 2023-03-09T14:43:51.103 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:51.095+0000 7ffb8f7fe700 1 -- 172.21.15.38:0/708048835 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 133198+0+0 (secure 0 0 0) 0x7ffb9801da40 con 0x7ffba01008d0 2023-03-09T14:43:51.103 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:51.095+0000 7ffb8f7fe700 1 --2- 172.21.15.38:0/708048835 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7ffb8809f3c0 0x7ffb880a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:43:51.104 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:51.095+0000 7ffb9e59c700 1 --2- 172.21.15.38:0/708048835 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7ffb8809f3c0 0x7ffb880a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:43:51.104 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:51.095+0000 7ffb8f7fe700 1 -- 172.21.15.38:0/708048835 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ffb980cb420 con 0x7ffba01008d0 2023-03-09T14:43:51.104 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:51.096+0000 7ffb9e59c700 1 --2- 172.21.15.38:0/708048835 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7ffb8809f3c0 0x7ffb880a1870 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ffba00ffda0 tx=0x7ffb9000a550 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:43:51.104 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:51.099+0000 7ffb8f7fe700 1 -- 172.21.15.38:0/708048835 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7ffb98095d60 con 0x7ffba01008d0 2023-03-09T14:43:51.228 INFO:journalctl@ceph.mds.k.smithi122.stdout:Mar 09 14:43:50 smithi122 systemd[1]: Starting Ceph mds.k for c6733fe2-be87-11ed-9af2-001a4aab830c... 2023-03-09T14:43:51.344 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:51.342+0000 7ffba494e700 1 -- 172.21.15.38:0/708048835 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffba01061f0 con 0x7ffba01008d0 2023-03-09T14:43:51.345 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:51.342+0000 7ffb8f7fe700 1 -- 172.21.15.38:0/708048835 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1257 (secure 0 0 0) 0x7ffb98099380 con 0x7ffba01008d0 2023-03-09T14:43:51.345 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:43:51.345 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6833/415305571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":415305571},{"type":"v1","addr":"172.21.15.181:6833","nonce":415305571}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2023-03-09T14:43:51.346 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:51.344+0000 7ffba494e700 1 -- 172.21.15.38:0/708048835 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7ffb8809f3c0 msgr2=0x7ffb880a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:43:51.346 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:51.344+0000 7ffba494e700 1 --2- 172.21.15.38:0/708048835 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7ffb8809f3c0 0x7ffb880a1870 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ffba00ffda0 tx=0x7ffb9000a550 comp rx=0 tx=0).stop 2023-03-09T14:43:51.347 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:51.344+0000 7ffba494e700 1 -- 172.21.15.38:0/708048835 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffba01008d0 msgr2=0x7ffba019cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:43:51.347 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:51.344+0000 7ffba494e700 1 --2- 172.21.15.38:0/708048835 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffba01008d0 0x7ffba019cf80 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ffb9800b780 tx=0x7ffb9800bd00 comp rx=0 tx=0).stop 2023-03-09T14:43:51.347 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:51.344+0000 7ffba494e700 1 -- 172.21.15.38:0/708048835 shutdown_connections 2023-03-09T14:43:51.347 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:51.344+0000 7ffba494e700 1 --2- 172.21.15.38:0/708048835 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffba01008d0 0x7ffba019cf80 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:51.347 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:51.344+0000 7ffba494e700 1 --2- 172.21.15.38:0/708048835 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7ffb8809f3c0 0x7ffb880a1870 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:51.347 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:51.344+0000 7ffba494e700 1 --2- 172.21.15.38:0/708048835 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffba00fed50 0x7ffba0068610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:51.348 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:51.344+0000 7ffba494e700 1 --2- 172.21.15.38:0/708048835 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffba00fff40 0x7ffba019a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:51.348 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:51.344+0000 7ffba494e700 1 -- 172.21.15.38:0/708048835 >> 172.21.15.38:0/708048835 conn(0x7ffba00fa350 msgr2=0x7ffba0103330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:43:51.348 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:51.344+0000 7ffba494e700 1 -- 172.21.15.38:0/708048835 shutdown_connections 2023-03-09T14:43:51.348 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:51.344+0000 7ffba494e700 1 -- 172.21.15.38:0/708048835 wait complete. 2023-03-09T14:43:51.348 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 2 2023-03-09T14:43:51.529 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:51 smithi122 ceph-mon[93869]: Deploying daemon mds.k on smithi122 2023-03-09T14:43:51.530 INFO:journalctl@ceph.mds.k.smithi122.stdout:Mar 09 14:43:51 smithi122 podman[118108]: 2023-03-09T14:43:51.530 INFO:journalctl@ceph.mds.k.smithi122.stdout:Mar 09 14:43:51 smithi122 bash[118108]: fea4dca72f4fb39ed890a797ccc459e15f43b37d9b9843cf148acc73b6828326 2023-03-09T14:43:51.530 INFO:journalctl@ceph.mds.k.smithi122.stdout:Mar 09 14:43:51 smithi122 ceph-mds[118148]: -- 172.21.15.122:0/2373067081 <== mon.2 v2:172.21.15.122:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5567a0408b60 con 0x5567a03dc800 2023-03-09T14:43:51.531 INFO:journalctl@ceph.mds.k.smithi122.stdout:Mar 09 14:43:51 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mds-k[118124]: starting mds.k at 2023-03-09T14:43:51.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:51 smithi181 ceph-mon[94189]: Deploying daemon mds.k on smithi122 2023-03-09T14:43:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:51 smithi038 ceph-mon[92422]: Deploying daemon mds.k on smithi122 2023-03-09T14:43:51.977 INFO:journalctl@ceph.mds.k.smithi122.stdout:Mar 09 14:43:51 smithi122 systemd[1]: Started Ceph mds.k for c6733fe2-be87-11ed-9af2-001a4aab830c. 2023-03-09T14:43:52.612 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:52 smithi038 ceph-mon[92422]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-03-09T14:43:52.613 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:52 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/708048835' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-03-09T14:43:52.613 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:52 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:52.613 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:52 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:52.613 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:52 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-03-09T14:43:52.613 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:52 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-03-09T14:43:52.613 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:52 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:43:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:52 smithi122 ceph-mon[93869]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-03-09T14:43:52.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:52 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/708048835' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-03-09T14:43:52.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:52 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:52.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:52 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:52.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:52 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-03-09T14:43:52.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:52 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-03-09T14:43:52.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:52 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:43:52.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:52 smithi181 ceph-mon[94189]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-03-09T14:43:52.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:52 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/708048835' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-03-09T14:43:52.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:52 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:52.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:52 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:52.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:52 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-03-09T14:43:52.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:52 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-03-09T14:43:52.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:52 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:43:53.251 INFO:journalctl@ceph.mds.j.smithi038.stdout:Mar 09 14:43:53 smithi038 systemd[1]: Starting Ceph mds.j for c6733fe2-be87-11ed-9af2-001a4aab830c... 2023-03-09T14:43:53.360 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-09T14:43:53.495 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:53.493+0000 7f5804291700 1 -- 172.21.15.38:0/2624733257 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f57fc0fff70 msgr2=0x7f57fc1003c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:43:53.496 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:53.493+0000 7f5804291700 1 --2- 172.21.15.38:0/2624733257 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f57fc0fff70 0x7f57fc1003c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f57ec009990 tx=0x7f57ec009ca0 comp rx=0 tx=0).stop 2023-03-09T14:43:53.496 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:53.493+0000 7f5804291700 1 -- 172.21.15.38:0/2624733257 shutdown_connections 2023-03-09T14:43:53.496 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:53.493+0000 7f5804291700 1 --2- 172.21.15.38:0/2624733257 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f57fc100900 0x7f57fc1071c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:53.497 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:53.493+0000 7f5804291700 1 --2- 172.21.15.38:0/2624733257 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f57fc0fed80 0x7f57fc0ff190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:53.497 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:53.493+0000 7f5804291700 1 --2- 172.21.15.38:0/2624733257 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f57fc0fff70 0x7f57fc1003c0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:53.497 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:53.493+0000 7f5804291700 1 -- 172.21.15.38:0/2624733257 >> 172.21.15.38:0/2624733257 conn(0x7f57fc0fa380 msgr2=0x7f57fc0fc7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:43:53.497 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:53.493+0000 7f5804291700 1 -- 172.21.15.38:0/2624733257 shutdown_connections 2023-03-09T14:43:53.497 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:53.493+0000 7f5804291700 1 -- 172.21.15.38:0/2624733257 wait complete. 2023-03-09T14:43:53.499 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:53.494+0000 7f5804291700 1 Processor -- start 2023-03-09T14:43:53.499 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:53.494+0000 7f5804291700 1 -- start start 2023-03-09T14:43:53.499 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:53.494+0000 7f5804291700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f57fc0fed80 0x7f57fc0685b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:43:53.499 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:53.494+0000 7f5804291700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f57fc0fff70 0x7f57fc19ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:43:53.500 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:53.494+0000 7f5804291700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f57fc100900 0x7f57fc19d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:43:53.500 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:53.494+0000 7f5804291700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57fc19d910 con 0x7f57fc0fed80 2023-03-09T14:43:53.500 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:53.494+0000 7f5804291700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57fc19da80 con 0x7f57fc100900 2023-03-09T14:43:53.500 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:53.494+0000 7f5804291700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57fc19dc40 con 0x7f57fc0fff70 2023-03-09T14:43:53.500 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:53.494+0000 7f580202d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f57fc0fed80 0x7f57fc0685b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:43:53.500 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:53.494+0000 7f580202d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f57fc0fed80 0x7f57fc0685b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36400/0 (socket says 172.21.15.38:36400) 2023-03-09T14:43:53.500 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:53.494+0000 7f580202d700 1 -- 172.21.15.38:0/90767048 learned_addr learned my addr 172.21.15.38:0/90767048 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:43:53.501 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:53.494+0000 7f580182c700 1 --2- 172.21.15.38:0/90767048 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f57fc0fff70 0x7f57fc19ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:43:53.501 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:53.494+0000 7f580282e700 1 --2- 172.21.15.38:0/90767048 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f57fc100900 0x7f57fc19d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:43:53.501 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:53.495+0000 7f580202d700 1 -- 172.21.15.38:0/90767048 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f57fc0fff70 msgr2=0x7f57fc19ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:43:53.501 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:53.495+0000 7f580202d700 1 --2- 172.21.15.38:0/90767048 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f57fc0fff70 0x7f57fc19ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:53.501 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:53.495+0000 7f580202d700 1 -- 172.21.15.38:0/90767048 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f57fc100900 msgr2=0x7f57fc19d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:43:53.501 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:53.495+0000 7f580202d700 1 --2- 172.21.15.38:0/90767048 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f57fc100900 0x7f57fc19d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:53.501 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:53.495+0000 7f580202d700 1 -- 172.21.15.38:0/90767048 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57ec0095f0 con 0x7f57fc0fed80 2023-03-09T14:43:53.502 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:53.495+0000 7f580282e700 1 --2- 172.21.15.38:0/90767048 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f57fc100900 0x7f57fc19d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:43:53.502 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:53.495+0000 7f580182c700 1 --2- 172.21.15.38:0/90767048 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f57fc0fff70 0x7f57fc19ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:43:53.502 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:53.495+0000 7f580202d700 1 --2- 172.21.15.38:0/90767048 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f57fc0fed80 0x7f57fc0685b0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f57f800b500 tx=0x7f57f800b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:43:53.502 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:53.495+0000 7f57f37fe700 1 -- 172.21.15.38:0/90767048 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57f8011830 con 0x7f57fc0fed80 2023-03-09T14:43:53.502 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:53.495+0000 7f5804291700 1 -- 172.21.15.38:0/90767048 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57fc10a850 con 0x7f57fc0fed80 2023-03-09T14:43:53.502 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:53.495+0000 7f5804291700 1 -- 172.21.15.38:0/90767048 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57fc10a6c0 con 0x7f57fc0fed80 2023-03-09T14:43:53.502 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:53.495+0000 7f57f37fe700 1 -- 172.21.15.38:0/90767048 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f57f8009330 con 0x7f57fc0fed80 2023-03-09T14:43:53.503 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:53.495+0000 7f57f37fe700 1 -- 172.21.15.38:0/90767048 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57f801ad40 con 0x7f57fc0fed80 2023-03-09T14:43:53.503 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:53.496+0000 7f5804291700 1 -- 172.21.15.38:0/90767048 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57e0005320 con 0x7f57fc0fed80 2023-03-09T14:43:53.503 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:53.498+0000 7f57f37fe700 1 -- 172.21.15.38:0/90767048 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 133198+0+0 (secure 0 0 0) 0x7f57f8021070 con 0x7f57fc0fed80 2023-03-09T14:43:53.504 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:53.498+0000 7f57f37fe700 1 --2- 172.21.15.38:0/90767048 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f57e809f3c0 0x7f57e80a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:43:53.504 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:53.498+0000 7f57f37fe700 1 -- 172.21.15.38:0/90767048 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f57f80c9600 con 0x7f57fc0fed80 2023-03-09T14:43:53.504 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:53.498+0000 7f580182c700 1 --2- 172.21.15.38:0/90767048 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f57e809f3c0 0x7f57e80a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:43:53.504 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:53.499+0000 7f580182c700 1 --2- 172.21.15.38:0/90767048 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f57e809f3c0 0x7f57e80a1870 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f57ec00b5c0 tx=0x7f57ec005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:43:53.505 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:53.503+0000 7f57f37fe700 1 -- 172.21.15.38:0/90767048 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f57f8093eb0 con 0x7f57fc0fed80 2023-03-09T14:43:53.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:53 smithi038 ceph-mon[92422]: Deploying daemon mds.j on smithi038 2023-03-09T14:43:53.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6832/659300520,v1:172.21.15.122:6833/659300520] up:boot 2023-03-09T14:43:53.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:53 smithi038 ceph-mon[92422]: fsmap 2 up:standby 2023-03-09T14:43:53.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:53 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-03-09T14:43:53.655 INFO:journalctl@ceph.mds.j.smithi038.stdout:Mar 09 14:43:53 smithi038 podman[125468]: 2023-03-09T14:43:53.656 INFO:journalctl@ceph.mds.j.smithi038.stdout:Mar 09 14:43:53 smithi038 ceph-mds[125551]: -- 172.21.15.38:0/50351944 <== mon.0 v2:172.21.15.38:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e6cc1a0b60 con 0x55e6cc175000 2023-03-09T14:43:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:53 smithi122 ceph-mon[93869]: Deploying daemon mds.j on smithi038 2023-03-09T14:43:53.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6832/659300520,v1:172.21.15.122:6833/659300520] up:boot 2023-03-09T14:43:53.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:53 smithi122 ceph-mon[93869]: fsmap 2 up:standby 2023-03-09T14:43:53.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:53 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-03-09T14:43:53.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:53 smithi181 ceph-mon[94189]: Deploying daemon mds.j on smithi038 2023-03-09T14:43:53.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6832/659300520,v1:172.21.15.122:6833/659300520] up:boot 2023-03-09T14:43:53.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:53 smithi181 ceph-mon[94189]: fsmap 2 up:standby 2023-03-09T14:43:53.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:53 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-03-09T14:43:53.759 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:53.757+0000 7f5804291700 1 -- 172.21.15.38:0/90767048 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f57e0005cc0 con 0x7f57fc0fed80 2023-03-09T14:43:53.759 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:53.757+0000 7f57f37fe700 1 -- 172.21.15.38:0/90767048 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2014 (secure 0 0 0) 0x7f57f80157f0 con 0x7f57fc0fed80 2023-03-09T14:43:53.760 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:43:53.760 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6833/415305571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":415305571},{"type":"v1","addr":"172.21.15.181:6833","nonce":415305571}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/659300520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":659300520},{"type":"v1","addr":"172.21.15.122:6833","nonce":659300520}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2023-03-09T14:43:53.762 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:53.759+0000 7f5804291700 1 -- 172.21.15.38:0/90767048 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f57e809f3c0 msgr2=0x7f57e80a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:43:53.762 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:53.759+0000 7f5804291700 1 --2- 172.21.15.38:0/90767048 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f57e809f3c0 0x7f57e80a1870 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f57ec00b5c0 tx=0x7f57ec005fb0 comp rx=0 tx=0).stop 2023-03-09T14:43:53.762 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:53.759+0000 7f5804291700 1 -- 172.21.15.38:0/90767048 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f57fc0fed80 msgr2=0x7f57fc0685b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:43:53.762 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:53.759+0000 7f5804291700 1 --2- 172.21.15.38:0/90767048 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f57fc0fed80 0x7f57fc0685b0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f57f800b500 tx=0x7f57f800b810 comp rx=0 tx=0).stop 2023-03-09T14:43:53.762 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:53.760+0000 7f5804291700 1 -- 172.21.15.38:0/90767048 shutdown_connections 2023-03-09T14:43:53.762 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:53.760+0000 7f5804291700 1 --2- 172.21.15.38:0/90767048 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f57fc0fed80 0x7f57fc0685b0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:53.763 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:53.760+0000 7f5804291700 1 --2- 172.21.15.38:0/90767048 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f57e809f3c0 0x7f57e80a1870 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:53.763 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:53.760+0000 7f5804291700 1 --2- 172.21.15.38:0/90767048 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f57fc100900 0x7f57fc19d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:53.763 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:53.760+0000 7f5804291700 1 --2- 172.21.15.38:0/90767048 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f57fc0fff70 0x7f57fc19ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:53.763 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:53.760+0000 7f5804291700 1 -- 172.21.15.38:0/90767048 >> 172.21.15.38:0/90767048 conn(0x7f57fc0fa380 msgr2=0x7f57fc0fc690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:43:53.763 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:53.760+0000 7f5804291700 1 -- 172.21.15.38:0/90767048 shutdown_connections 2023-03-09T14:43:53.763 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:53.760+0000 7f5804291700 1 -- 172.21.15.38:0/90767048 wait complete. 2023-03-09T14:43:53.764 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 3 2023-03-09T14:43:54.000 INFO:journalctl@ceph.mds.j.smithi038.stdout:Mar 09 14:43:53 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mds-j[125508]: starting mds.j at 2023-03-09T14:43:54.001 INFO:journalctl@ceph.mds.j.smithi038.stdout:Mar 09 14:43:53 smithi038 bash[125468]: 1fcb4c6e6566de8a391eedeb59984e723fa19129329f2c73a86ea18eaeb91a7d 2023-03-09T14:43:54.001 INFO:journalctl@ceph.mds.j.smithi038.stdout:Mar 09 14:43:53 smithi038 systemd[1]: Started Ceph mds.j for c6733fe2-be87-11ed-9af2-001a4aab830c. 2023-03-09T14:43:54.654 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:54 smithi038 ceph-mon[92422]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-03-09T14:43:54.654 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:54 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:54.654 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:54 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:54.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:54 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-03-09T14:43:54.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:54 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-03-09T14:43:54.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:54 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:43:54.655 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:54 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/90767048' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-03-09T14:43:54.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:54 smithi122 ceph-mon[93869]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-03-09T14:43:54.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:54 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:54.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:54 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:54.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:54 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-03-09T14:43:54.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:54 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-03-09T14:43:54.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:54 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:43:54.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:54 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/90767048' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-03-09T14:43:54.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:54 smithi181 ceph-mon[94189]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-03-09T14:43:54.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:54 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:54.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:54 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:54.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:54 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-03-09T14:43:54.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:54 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-03-09T14:43:54.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:54 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:43:54.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:54 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/90767048' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-03-09T14:43:55.326 INFO:journalctl@ceph.mds.d.smithi038.stdout:Mar 09 14:43:55 smithi038 systemd[1]: Starting Ceph mds.d for c6733fe2-be87-11ed-9af2-001a4aab830c... 2023-03-09T14:43:55.678 INFO:journalctl@ceph.mds.d.smithi038.stdout:Mar 09 14:43:55 smithi038 podman[125851]: 2023-03-09T14:43:55.679 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:55 smithi038 ceph-mon[92422]: Deploying daemon mds.d on smithi038 2023-03-09T14:43:55.679 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.38:6834/768918602,v1:172.21.15.38:6835/768918602] up:boot 2023-03-09T14:43:55.679 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:55 smithi038 ceph-mon[92422]: fsmap 3 up:standby 2023-03-09T14:43:55.679 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:55 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-03-09T14:43:55.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:55 smithi122 ceph-mon[93869]: Deploying daemon mds.d on smithi038 2023-03-09T14:43:55.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.38:6834/768918602,v1:172.21.15.38:6835/768918602] up:boot 2023-03-09T14:43:55.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:55 smithi122 ceph-mon[93869]: fsmap 3 up:standby 2023-03-09T14:43:55.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:55 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-03-09T14:43:55.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:55 smithi181 ceph-mon[94189]: Deploying daemon mds.d on smithi038 2023-03-09T14:43:55.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.38:6834/768918602,v1:172.21.15.38:6835/768918602] up:boot 2023-03-09T14:43:55.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:55 smithi181 ceph-mon[94189]: fsmap 3 up:standby 2023-03-09T14:43:55.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:55 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-03-09T14:43:55.776 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-09T14:43:55.928 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:55.925+0000 7fa35515e700 1 -- 172.21.15.38:0/1452251637 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa350103270 msgr2=0x7fa350105650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:43:55.928 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:55.925+0000 7fa35515e700 1 --2- 172.21.15.38:0/1452251637 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa350103270 0x7fa350105650 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa344009990 tx=0x7fa344009ca0 comp rx=0 tx=0).stop 2023-03-09T14:43:55.928 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:55.925+0000 7fa35515e700 1 -- 172.21.15.38:0/1452251637 shutdown_connections 2023-03-09T14:43:55.928 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:55.925+0000 7fa35515e700 1 --2- 172.21.15.38:0/1452251637 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa350105b90 0x7fa350107ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:55.928 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:55.925+0000 7fa35515e700 1 --2- 172.21.15.38:0/1452251637 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa350100950 0x7fa350102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:55.929 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:55.925+0000 7fa35515e700 1 --2- 172.21.15.38:0/1452251637 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa350103270 0x7fa350105650 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:55.929 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:55.925+0000 7fa35515e700 1 -- 172.21.15.38:0/1452251637 >> 172.21.15.38:0/1452251637 conn(0x7fa3500fa3b0 msgr2=0x7fa3500fc820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:43:55.929 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:55.925+0000 7fa35515e700 1 -- 172.21.15.38:0/1452251637 shutdown_connections 2023-03-09T14:43:55.929 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:55.925+0000 7fa35515e700 1 -- 172.21.15.38:0/1452251637 wait complete. 2023-03-09T14:43:55.931 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:55.926+0000 7fa35515e700 1 Processor -- start 2023-03-09T14:43:55.931 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:55.926+0000 7fa35515e700 1 -- start start 2023-03-09T14:43:55.931 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:55.926+0000 7fa35515e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa350100950 0x7fa3501002c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:43:55.931 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:55.926+0000 7fa35515e700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa350103270 0x7fa3500fe900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:43:55.931 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:55.926+0000 7fa35515e700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa350105b90 0x7fa3500fee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:43:55.932 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:55.926+0000 7fa35515e700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3500ff5e0 con 0x7fa350100950 2023-03-09T14:43:55.932 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:55.926+0000 7fa35515e700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3500ff750 con 0x7fa350105b90 2023-03-09T14:43:55.932 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:55.926+0000 7fa35515e700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa35019fc00 con 0x7fa350103270 2023-03-09T14:43:55.932 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:55.927+0000 7fa34ed9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa350100950 0x7fa3501002c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:43:55.932 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:55.927+0000 7fa34ed9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa350100950 0x7fa3501002c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36436/0 (socket says 172.21.15.38:36436) 2023-03-09T14:43:55.932 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:55.927+0000 7fa34ed9d700 1 -- 172.21.15.38:0/3901620183 learned_addr learned my addr 172.21.15.38:0/3901620183 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:43:55.932 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:55.927+0000 7fa34f59e700 1 --2- 172.21.15.38:0/3901620183 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa350105b90 0x7fa3500fee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:43:55.933 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:55.927+0000 7fa34e59c700 1 --2- 172.21.15.38:0/3901620183 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa350103270 0x7fa3500fe900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:43:55.933 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:55.927+0000 7fa34ed9d700 1 -- 172.21.15.38:0/3901620183 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa350103270 msgr2=0x7fa3500fe900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:43:55.933 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:55.927+0000 7fa34ed9d700 1 --2- 172.21.15.38:0/3901620183 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa350103270 0x7fa3500fe900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:55.933 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:55.927+0000 7fa34ed9d700 1 -- 172.21.15.38:0/3901620183 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa350105b90 msgr2=0x7fa3500fee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:43:55.933 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:55.927+0000 7fa34ed9d700 1 --2- 172.21.15.38:0/3901620183 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa350105b90 0x7fa3500fee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:55.933 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:55.927+0000 7fa34ed9d700 1 -- 172.21.15.38:0/3901620183 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3440095f0 con 0x7fa350100950 2023-03-09T14:43:55.933 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:55.927+0000 7fa34ed9d700 1 --2- 172.21.15.38:0/3901620183 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa350100950 0x7fa3501002c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa34000d950 tx=0x7fa34000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:43:55.933 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:55.927+0000 7fa337fff700 1 -- 172.21.15.38:0/3901620183 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa340009940 con 0x7fa350100950 2023-03-09T14:43:55.934 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:55.927+0000 7fa337fff700 1 -- 172.21.15.38:0/3901620183 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa340010460 con 0x7fa350100950 2023-03-09T14:43:55.934 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:55.927+0000 7fa35515e700 1 -- 172.21.15.38:0/3901620183 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa35010b820 con 0x7fa350100950 2023-03-09T14:43:55.934 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:55.927+0000 7fa337fff700 1 -- 172.21.15.38:0/3901620183 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa34000f620 con 0x7fa350100950 2023-03-09T14:43:55.934 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:55.927+0000 7fa35515e700 1 -- 172.21.15.38:0/3901620183 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa35010b690 con 0x7fa350100950 2023-03-09T14:43:55.934 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:55.928+0000 7fa35515e700 1 -- 172.21.15.38:0/3901620183 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3501a0210 con 0x7fa350100950 2023-03-09T14:43:55.935 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:55.930+0000 7fa337fff700 1 -- 172.21.15.38:0/3901620183 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 133198+0+0 (secure 0 0 0) 0x7fa340022070 con 0x7fa350100950 2023-03-09T14:43:55.935 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:55.930+0000 7fa337fff700 1 --2- 172.21.15.38:0/3901620183 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fa33809f3c0 0x7fa3380a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:43:55.935 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:55.930+0000 7fa337fff700 1 -- 172.21.15.38:0/3901620183 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fa3400c95e0 con 0x7fa350100950 2023-03-09T14:43:55.935 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:55.930+0000 7fa34e59c700 1 --2- 172.21.15.38:0/3901620183 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fa33809f3c0 0x7fa3380a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:43:55.935 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:55.931+0000 7fa34e59c700 1 --2- 172.21.15.38:0/3901620183 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fa33809f3c0 0x7fa3380a1870 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fa344006010 tx=0x7fa34400b540 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:43:55.936 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:55.934+0000 7fa337fff700 1 -- 172.21.15.38:0/3901620183 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7fa340093e90 con 0x7fa350100950 2023-03-09T14:43:56.201 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:56.198+0000 7fa35515e700 1 -- 172.21.15.38:0/3901620183 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa3501a08b0 con 0x7fa350100950 2023-03-09T14:43:56.201 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:56.199+0000 7fa337fff700 1 -- 172.21.15.38:0/3901620183 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2768 (secure 0 0 0) 0x7fa340020ae0 con 0x7fa350100950 2023-03-09T14:43:56.202 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:43:56.202 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/768918602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":768918602},{"type":"v1","addr":"172.21.15.38:6835","nonce":768918602}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6833/415305571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":415305571},{"type":"v1","addr":"172.21.15.181:6833","nonce":415305571}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/659300520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":659300520},{"type":"v1","addr":"172.21.15.122:6833","nonce":659300520}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2023-03-09T14:43:56.204 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:56.201+0000 7fa35515e700 1 -- 172.21.15.38:0/3901620183 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fa33809f3c0 msgr2=0x7fa3380a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:43:56.204 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:56.201+0000 7fa35515e700 1 --2- 172.21.15.38:0/3901620183 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fa33809f3c0 0x7fa3380a1870 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fa344006010 tx=0x7fa34400b540 comp rx=0 tx=0).stop 2023-03-09T14:43:56.204 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:56.201+0000 7fa35515e700 1 -- 172.21.15.38:0/3901620183 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa350100950 msgr2=0x7fa3501002c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:43:56.204 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:56.201+0000 7fa35515e700 1 --2- 172.21.15.38:0/3901620183 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa350100950 0x7fa3501002c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa34000d950 tx=0x7fa34000dc60 comp rx=0 tx=0).stop 2023-03-09T14:43:56.204 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:56.201+0000 7fa35515e700 1 -- 172.21.15.38:0/3901620183 shutdown_connections 2023-03-09T14:43:56.204 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:56.201+0000 7fa35515e700 1 --2- 172.21.15.38:0/3901620183 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa350100950 0x7fa3501002c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:56.204 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:56.201+0000 7fa35515e700 1 --2- 172.21.15.38:0/3901620183 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fa33809f3c0 0x7fa3380a1870 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:56.205 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:56.201+0000 7fa35515e700 1 --2- 172.21.15.38:0/3901620183 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa350105b90 0x7fa3500fee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:56.205 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:56.201+0000 7fa35515e700 1 --2- 172.21.15.38:0/3901620183 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa350103270 0x7fa3500fe900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:56.205 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:56.201+0000 7fa35515e700 1 -- 172.21.15.38:0/3901620183 >> 172.21.15.38:0/3901620183 conn(0x7fa3500fa3b0 msgr2=0x7fa350101dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:43:56.205 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:56.201+0000 7fa35515e700 1 -- 172.21.15.38:0/3901620183 shutdown_connections 2023-03-09T14:43:56.205 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:56.201+0000 7fa35515e700 1 -- 172.21.15.38:0/3901620183 wait complete. 2023-03-09T14:43:56.206 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 4 2023-03-09T14:43:56.215 INFO:journalctl@ceph.mds.d.smithi038.stdout:Mar 09 14:43:55 smithi038 bash[125851]: b320107d54956b352a57672e9c12c6d0f405b81c9f8342c91114f099bb3088e5 2023-03-09T14:43:56.215 INFO:journalctl@ceph.mds.d.smithi038.stdout:Mar 09 14:43:55 smithi038 ceph-mds[125910]: -- 172.21.15.38:0/3622984968 <== mon.0 v2:172.21.15.38:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559b8384eb60 con 0x559b83822c00 2023-03-09T14:43:56.215 INFO:journalctl@ceph.mds.d.smithi038.stdout:Mar 09 14:43:55 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mds-d[125867]: starting mds.d at 2023-03-09T14:43:56.215 INFO:journalctl@ceph.mds.d.smithi038.stdout:Mar 09 14:43:55 smithi038 systemd[1]: Started Ceph mds.d for c6733fe2-be87-11ed-9af2-001a4aab830c. 2023-03-09T14:43:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:56 smithi038 ceph-mon[92422]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-03-09T14:43:56.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:56 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:56.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:56 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:56.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:56 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-03-09T14:43:56.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:56 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-03-09T14:43:56.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:56 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:43:56.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:56 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/3901620183' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-03-09T14:43:56.605 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:56 smithi122 ceph-mon[93869]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-03-09T14:43:56.606 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:56 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:56.606 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:56 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:56.606 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:56 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-03-09T14:43:56.606 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:56 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-03-09T14:43:56.606 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:56 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:43:56.606 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:56 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/3901620183' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-03-09T14:43:56.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:56 smithi181 ceph-mon[94189]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-03-09T14:43:56.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:56 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:56.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:56 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:56.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:56 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-03-09T14:43:56.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:56 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-03-09T14:43:56.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:56 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:43:56.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:56 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/3901620183' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-03-09T14:43:57.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:57 smithi122 ceph-mon[93869]: Deploying daemon mds.e on smithi122 2023-03-09T14:43:57.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.38:6836/18381540,v1:172.21.15.38:6837/18381540] up:boot 2023-03-09T14:43:57.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:57 smithi122 ceph-mon[93869]: fsmap 4 up:standby 2023-03-09T14:43:57.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:57 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-03-09T14:43:57.479 INFO:journalctl@ceph.mds.e.smithi122.stdout:Mar 09 14:43:57 smithi122 systemd[1]: Starting Ceph mds.e for c6733fe2-be87-11ed-9af2-001a4aab830c... 2023-03-09T14:43:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:57 smithi181 ceph-mon[94189]: Deploying daemon mds.e on smithi122 2023-03-09T14:43:57.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.38:6836/18381540,v1:172.21.15.38:6837/18381540] up:boot 2023-03-09T14:43:57.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:57 smithi181 ceph-mon[94189]: fsmap 4 up:standby 2023-03-09T14:43:57.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:57 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-03-09T14:43:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:57 smithi038 ceph-mon[92422]: Deploying daemon mds.e on smithi122 2023-03-09T14:43:57.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.38:6836/18381540,v1:172.21.15.38:6837/18381540] up:boot 2023-03-09T14:43:57.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:57 smithi038 ceph-mon[92422]: fsmap 4 up:standby 2023-03-09T14:43:57.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:57 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-03-09T14:43:57.977 INFO:journalctl@ceph.mds.e.smithi122.stdout:Mar 09 14:43:57 smithi122 podman[118448]: 2023-03-09T14:43:57.978 INFO:journalctl@ceph.mds.e.smithi122.stdout:Mar 09 14:43:57 smithi122 bash[118448]: 19a2add889819069616f20440c8b36aa56a25207bcc9a0e3579b1b5222290767 2023-03-09T14:43:57.978 INFO:journalctl@ceph.mds.e.smithi122.stdout:Mar 09 14:43:57 smithi122 ceph-mds[118488]: -- 172.21.15.122:0/3203550184 <== mon.2 v2:172.21.15.122:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x561f7269ab60 con 0x561f7266ec00 2023-03-09T14:43:57.978 INFO:journalctl@ceph.mds.e.smithi122.stdout:Mar 09 14:43:57 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mds-e[118464]: starting mds.e at 2023-03-09T14:43:58.217 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-09T14:43:58.343 INFO:journalctl@ceph.mds.e.smithi122.stdout:Mar 09 14:43:57 smithi122 systemd[1]: Started Ceph mds.e for c6733fe2-be87-11ed-9af2-001a4aab830c. 2023-03-09T14:43:58.358 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:58.356+0000 7f1ba87ee700 1 -- 172.21.15.38:0/2592977204 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1ba01009a0 msgr2=0x7f1ba0102d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:43:58.359 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:58.356+0000 7f1ba87ee700 1 --2- 172.21.15.38:0/2592977204 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1ba01009a0 0x7f1ba0102d80 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f1b9c009990 tx=0x7f1b9c009ca0 comp rx=0 tx=0).stop 2023-03-09T14:43:58.359 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:58.356+0000 7f1ba87ee700 1 -- 172.21.15.38:0/2592977204 shutdown_connections 2023-03-09T14:43:58.359 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:58.356+0000 7f1ba87ee700 1 --2- 172.21.15.38:0/2592977204 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1ba01009a0 0x7f1ba0102d80 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:58.359 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:58.356+0000 7f1ba87ee700 1 --2- 172.21.15.38:0/2592977204 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ba0105be0 0x7f1ba0107fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:58.359 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:58.356+0000 7f1ba87ee700 1 --2- 172.21.15.38:0/2592977204 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1ba01032c0 0x7f1ba01056a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:58.360 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:58.356+0000 7f1ba87ee700 1 -- 172.21.15.38:0/2592977204 >> 172.21.15.38:0/2592977204 conn(0x7f1ba00fa380 msgr2=0x7f1ba00fc7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:43:58.360 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:58.356+0000 7f1ba87ee700 1 -- 172.21.15.38:0/2592977204 shutdown_connections 2023-03-09T14:43:58.360 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:58.356+0000 7f1ba87ee700 1 -- 172.21.15.38:0/2592977204 wait complete. 2023-03-09T14:43:58.362 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:58.357+0000 7f1ba87ee700 1 Processor -- start 2023-03-09T14:43:58.362 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:58.357+0000 7f1ba87ee700 1 -- start start 2023-03-09T14:43:58.362 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:58.357+0000 7f1ba87ee700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1ba01009a0 0x7f1ba0195c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:43:58.362 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:58.357+0000 7f1ba87ee700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1ba01032c0 0x7f1ba01961b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:43:58.362 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:58.357+0000 7f1ba87ee700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ba0105be0 0x7f1ba019d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:43:58.362 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:58.357+0000 7f1ba87ee700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba019d7b0 con 0x7f1ba01032c0 2023-03-09T14:43:58.362 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:58.357+0000 7f1ba87ee700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba019d920 con 0x7f1ba01009a0 2023-03-09T14:43:58.363 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:58.357+0000 7f1ba87ee700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba019dae0 con 0x7f1ba0105be0 2023-03-09T14:43:58.363 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:58.357+0000 7f1ba5d89700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1ba01032c0 0x7f1ba01961b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:43:58.363 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:58.357+0000 7f1ba5d89700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1ba01032c0 0x7f1ba01961b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36454/0 (socket says 172.21.15.38:36454) 2023-03-09T14:43:58.363 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:58.357+0000 7f1ba5d89700 1 -- 172.21.15.38:0/2735802794 learned_addr learned my addr 172.21.15.38:0/2735802794 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:43:58.363 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:58.357+0000 7f1ba6d8b700 1 --2- 172.21.15.38:0/2735802794 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ba0105be0 0x7f1ba019d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:43:58.363 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:58.357+0000 7f1ba658a700 1 --2- 172.21.15.38:0/2735802794 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1ba01009a0 0x7f1ba0195c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:43:58.363 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:58.357+0000 7f1ba5d89700 1 -- 172.21.15.38:0/2735802794 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ba0105be0 msgr2=0x7f1ba019d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:43:58.364 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:58.357+0000 7f1ba5d89700 1 --2- 172.21.15.38:0/2735802794 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ba0105be0 0x7f1ba019d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:58.364 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:58.357+0000 7f1ba5d89700 1 -- 172.21.15.38:0/2735802794 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1ba01009a0 msgr2=0x7f1ba0195c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:43:58.364 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:58.357+0000 7f1ba5d89700 1 --2- 172.21.15.38:0/2735802794 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1ba01009a0 0x7f1ba0195c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:58.364 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:58.357+0000 7f1ba5d89700 1 -- 172.21.15.38:0/2735802794 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b9c0095f0 con 0x7f1ba01032c0 2023-03-09T14:43:58.364 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:58.358+0000 7f1ba5d89700 1 --2- 172.21.15.38:0/2735802794 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1ba01032c0 0x7f1ba01961b0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1b9000d950 tx=0x7f1b9000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:43:58.364 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:58.358+0000 7f1b977fe700 1 -- 172.21.15.38:0/2735802794 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b9000f890 con 0x7f1ba01032c0 2023-03-09T14:43:58.364 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:58.358+0000 7f1b977fe700 1 -- 172.21.15.38:0/2735802794 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1b90010460 con 0x7f1ba01032c0 2023-03-09T14:43:58.364 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:58.358+0000 7f1ba87ee700 1 -- 172.21.15.38:0/2735802794 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ba010b810 con 0x7f1ba01032c0 2023-03-09T14:43:58.365 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:58.358+0000 7f1b977fe700 1 -- 172.21.15.38:0/2735802794 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b9000e610 con 0x7f1ba01032c0 2023-03-09T14:43:58.365 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:58.358+0000 7f1ba87ee700 1 -- 172.21.15.38:0/2735802794 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ba010b680 con 0x7f1ba01032c0 2023-03-09T14:43:58.365 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:58.359+0000 7f1ba87ee700 1 -- 172.21.15.38:0/2735802794 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ba019e190 con 0x7f1ba01032c0 2023-03-09T14:43:58.366 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:58.360+0000 7f1b977fe700 1 -- 172.21.15.38:0/2735802794 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 133198+0+0 (secure 0 0 0) 0x7f1b90022070 con 0x7f1ba01032c0 2023-03-09T14:43:58.366 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:58.360+0000 7f1b977fe700 1 --2- 172.21.15.38:0/2735802794 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f1b8c09f440 0x7f1b8c0a18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:43:58.366 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:58.361+0000 7f1ba658a700 1 --2- 172.21.15.38:0/2735802794 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f1b8c09f440 0x7f1b8c0a18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:43:58.366 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:58.361+0000 7f1b977fe700 1 -- 172.21.15.38:0/2735802794 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f1b900c9640 con 0x7f1ba01032c0 2023-03-09T14:43:58.366 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:58.361+0000 7f1ba658a700 1 --2- 172.21.15.38:0/2735802794 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f1b8c09f440 0x7f1b8c0a18f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1b9c00b2e0 tx=0x7f1b9c005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:43:58.367 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:58.364+0000 7f1b977fe700 1 -- 172.21.15.38:0/2735802794 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f1b90093e00 con 0x7f1ba01032c0 2023-03-09T14:43:58.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:58 smithi038 ceph-mon[92422]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-03-09T14:43:58.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:58 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:58.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:58 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:58.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:58 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-03-09T14:43:58.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:58 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-03-09T14:43:58.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:58 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:43:58.610 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:58.608+0000 7f1ba87ee700 1 -- 172.21.15.38:0/2735802794 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1ba019e740 con 0x7f1ba01032c0 2023-03-09T14:43:58.611 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:58.609+0000 7f1b977fe700 1 -- 172.21.15.38:0/2735802794 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4279 (secure 0 0 0) 0x7f1b90016090 con 0x7f1ba01032c0 2023-03-09T14:43:58.612 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:43:58.612 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/768918602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":768918602},{"type":"v1","addr":"172.21.15.38:6835","nonce":768918602}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/18381540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":18381540},{"type":"v1","addr":"172.21.15.38:6837","nonce":18381540}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6833/415305571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":415305571},{"type":"v1","addr":"172.21.15.181:6833","nonce":415305571}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/659300520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":659300520},{"type":"v1","addr":"172.21.15.122:6833","nonce":659300520}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/3675537941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":3675537941},{"type":"v1","addr":"172.21.15.122:6835","nonce":3675537941}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[]} 2023-03-09T14:43:58.613 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:58.611+0000 7f1ba87ee700 1 -- 172.21.15.38:0/2735802794 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f1b8c09f440 msgr2=0x7f1b8c0a18f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:43:58.613 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:58.611+0000 7f1ba87ee700 1 --2- 172.21.15.38:0/2735802794 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f1b8c09f440 0x7f1b8c0a18f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1b9c00b2e0 tx=0x7f1b9c005fb0 comp rx=0 tx=0).stop 2023-03-09T14:43:58.613 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:58.611+0000 7f1ba87ee700 1 -- 172.21.15.38:0/2735802794 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1ba01032c0 msgr2=0x7f1ba01961b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:43:58.614 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:58.611+0000 7f1ba87ee700 1 --2- 172.21.15.38:0/2735802794 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1ba01032c0 0x7f1ba01961b0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1b9000d950 tx=0x7f1b9000dc60 comp rx=0 tx=0).stop 2023-03-09T14:43:58.614 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:58.611+0000 7f1ba87ee700 1 -- 172.21.15.38:0/2735802794 shutdown_connections 2023-03-09T14:43:58.614 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:58.611+0000 7f1ba87ee700 1 --2- 172.21.15.38:0/2735802794 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f1b8c09f440 0x7f1b8c0a18f0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:58.614 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:58.611+0000 7f1ba87ee700 1 --2- 172.21.15.38:0/2735802794 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ba0105be0 0x7f1ba019d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:58.614 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:58.611+0000 7f1ba87ee700 1 --2- 172.21.15.38:0/2735802794 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1ba01009a0 0x7f1ba0195c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:58.614 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:58.611+0000 7f1ba87ee700 1 --2- 172.21.15.38:0/2735802794 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1ba01032c0 0x7f1ba01961b0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:43:58.614 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:58.611+0000 7f1ba87ee700 1 -- 172.21.15.38:0/2735802794 >> 172.21.15.38:0/2735802794 conn(0x7f1ba00fa380 msgr2=0x7f1ba0101e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:43:58.615 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:58.611+0000 7f1ba87ee700 1 -- 172.21.15.38:0/2735802794 shutdown_connections 2023-03-09T14:43:58.615 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:43:58.611+0000 7f1ba87ee700 1 -- 172.21.15.38:0/2735802794 wait complete. 2023-03-09T14:43:58.615 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 6 2023-03-09T14:43:58.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:58 smithi122 ceph-mon[93869]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-03-09T14:43:58.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:58 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:58.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:58 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:58.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:58 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-03-09T14:43:58.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:58 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-03-09T14:43:58.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:58 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:43:58.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:58 smithi181 ceph-mon[94189]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-03-09T14:43:58.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:58 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:58.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:58 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:43:58.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:58 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-03-09T14:43:58.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:58 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-03-09T14:43:58.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:58 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:43:59.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:59 smithi122 ceph-mon[93869]: Deploying daemon mds.f on smithi181 2023-03-09T14:43:59.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:boot 2023-03-09T14:43:59.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:59 smithi122 ceph-mon[93869]: fsmap 5 up:standby 2023-03-09T14:43:59.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:59 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-03-09T14:43:59.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:43:59 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/2735802794' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-03-09T14:43:59.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:59 smithi181 ceph-mon[94189]: Deploying daemon mds.f on smithi181 2023-03-09T14:43:59.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:boot 2023-03-09T14:43:59.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:59 smithi181 ceph-mon[94189]: fsmap 5 up:standby 2023-03-09T14:43:59.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:59 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-03-09T14:43:59.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:43:59 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/2735802794' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-03-09T14:43:59.732 INFO:journalctl@ceph.mds.f.smithi181.stdout:Mar 09 14:43:59 smithi181 systemd[1]: Starting Ceph mds.f for c6733fe2-be87-11ed-9af2-001a4aab830c... 2023-03-09T14:43:59.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:59 smithi038 ceph-mon[92422]: Deploying daemon mds.f on smithi181 2023-03-09T14:43:59.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:boot 2023-03-09T14:43:59.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:59 smithi038 ceph-mon[92422]: fsmap 5 up:standby 2023-03-09T14:43:59.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:59 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-03-09T14:43:59.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:43:59 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/2735802794' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-03-09T14:44:00.095 INFO:journalctl@ceph.mds.f.smithi181.stdout:Mar 09 14:43:59 smithi181 podman[119651]: 2023-03-09T14:44:00.350 INFO:journalctl@ceph.mds.f.smithi181.stdout:Mar 09 14:44:00 smithi181 bash[119651]: 6f38f2a2254468e88f28a7edb395778ac0fee4e2d93ce7964e86d6f82edd01df 2023-03-09T14:44:00.351 INFO:journalctl@ceph.mds.f.smithi181.stdout:Mar 09 14:44:00 smithi181 ceph-mds[119691]: -- 172.21.15.181:0/1014749837 <== mon.1 v2:172.21.15.181:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ef2ce12b60 con 0x55ef2cde6800 2023-03-09T14:44:00.351 INFO:journalctl@ceph.mds.f.smithi181.stdout:Mar 09 14:44:00 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mds-f[119667]: starting mds.f at 2023-03-09T14:44:00.351 INFO:journalctl@ceph.mds.f.smithi181.stdout:Mar 09 14:44:00 smithi181 systemd[1]: Started Ceph mds.f for c6733fe2-be87-11ed-9af2-001a4aab830c. 2023-03-09T14:44:00.626 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-09T14:44:00.656 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:00 smithi038 ceph-mon[92422]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-03-09T14:44:00.656 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:00 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:00.656 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:00 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:00.657 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:00 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-03-09T14:44:00.657 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:00 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-03-09T14:44:00.657 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:00 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:44:00.657 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:00 smithi038 ceph-mon[92422]: Deploying daemon mds.c on smithi181 2023-03-09T14:44:00.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:00 smithi122 ceph-mon[93869]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-03-09T14:44:00.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:00 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:00.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:00 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:00.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:00 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-03-09T14:44:00.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:00 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-03-09T14:44:00.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:00 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:44:00.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:00 smithi122 ceph-mon[93869]: Deploying daemon mds.c on smithi181 2023-03-09T14:44:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:00 smithi181 ceph-mon[94189]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-03-09T14:44:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:00 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:00 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:00 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-03-09T14:44:00.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:00 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-03-09T14:44:00.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:00 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:44:00.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:00 smithi181 ceph-mon[94189]: Deploying daemon mds.c on smithi181 2023-03-09T14:44:00.766 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:00.763+0000 7fc85948e700 1 -- 172.21.15.38:0/3806350226 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc854100450 msgr2=0x7fc854100860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:00.766 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:00.763+0000 7fc85948e700 1 --2- 172.21.15.38:0/3806350226 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc854100450 0x7fc854100860 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc844009990 tx=0x7fc844009ca0 comp rx=0 tx=0).stop 2023-03-09T14:44:00.766 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:00.763+0000 7fc85948e700 1 -- 172.21.15.38:0/3806350226 shutdown_connections 2023-03-09T14:44:00.766 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:00.763+0000 7fc85948e700 1 --2- 172.21.15.38:0/3806350226 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc854100450 0x7fc854100860 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:00.766 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:00.763+0000 7fc85948e700 1 --2- 172.21.15.38:0/3806350226 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8540ff9c0 0x7fc854107270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:00.767 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:00.763+0000 7fc85948e700 1 --2- 172.21.15.38:0/3806350226 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc8540ff010 0x7fc8540ff480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:00.767 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:00.763+0000 7fc85948e700 1 -- 172.21.15.38:0/3806350226 >> 172.21.15.38:0/3806350226 conn(0x7fc8540f9b30 msgr2=0x7fc8540fbfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:00.767 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:00.764+0000 7fc85948e700 1 -- 172.21.15.38:0/3806350226 shutdown_connections 2023-03-09T14:44:00.767 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:00.764+0000 7fc85948e700 1 -- 172.21.15.38:0/3806350226 wait complete. 2023-03-09T14:44:00.769 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:00.764+0000 7fc85948e700 1 Processor -- start 2023-03-09T14:44:00.769 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:00.765+0000 7fc85948e700 1 -- start start 2023-03-09T14:44:00.769 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:00.765+0000 7fc85948e700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8540ff010 0x7fc8540fea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:00.769 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:00.765+0000 7fc85948e700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc8540ff9c0 0x7fc85419ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:00.769 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:00.765+0000 7fc85948e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc854100450 0x7fc85419d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:00.769 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:00.765+0000 7fc85948e700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc85419da00 con 0x7fc854100450 2023-03-09T14:44:00.770 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:00.765+0000 7fc85948e700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc85419db70 con 0x7fc8540ff9c0 2023-03-09T14:44:00.770 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:00.765+0000 7fc85948e700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc85419dd30 con 0x7fc8540ff010 2023-03-09T14:44:00.770 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:00.765+0000 7fc852ffd700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8540ff010 0x7fc8540fea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:00.770 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:00.765+0000 7fc852ffd700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8540ff010 0x7fc8540fea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.38:44164/0 (socket says 172.21.15.38:44164) 2023-03-09T14:44:00.770 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:00.765+0000 7fc852ffd700 1 -- 172.21.15.38:0/1271759142 learned_addr learned my addr 172.21.15.38:0/1271759142 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:44:00.770 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:00.765+0000 7fc8537fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc854100450 0x7fc85419d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:00.770 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:00.765+0000 7fc8527fc700 1 --2- 172.21.15.38:0/1271759142 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc8540ff9c0 0x7fc85419ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:00.771 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:00.765+0000 7fc852ffd700 1 -- 172.21.15.38:0/1271759142 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc8540ff9c0 msgr2=0x7fc85419ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:00.771 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:00.765+0000 7fc852ffd700 1 --2- 172.21.15.38:0/1271759142 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc8540ff9c0 0x7fc85419ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:00.771 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:00.765+0000 7fc852ffd700 1 -- 172.21.15.38:0/1271759142 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc854100450 msgr2=0x7fc85419d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:00.771 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:00.765+0000 7fc852ffd700 1 --2- 172.21.15.38:0/1271759142 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc854100450 0x7fc85419d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:00.771 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:00.765+0000 7fc852ffd700 1 -- 172.21.15.38:0/1271759142 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8440095f0 con 0x7fc8540ff010 2023-03-09T14:44:00.771 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:00.765+0000 7fc8537fe700 1 --2- 172.21.15.38:0/1271759142 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc854100450 0x7fc85419d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:44:00.771 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:00.766+0000 7fc852ffd700 1 --2- 172.21.15.38:0/1271759142 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8540ff010 0x7fc8540fea70 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc844005910 tx=0x7fc84400f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:00.771 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:00.766+0000 7fc83bfff700 1 -- 172.21.15.38:0/1271759142 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc84401d070 con 0x7fc8540ff010 2023-03-09T14:44:00.772 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:00.766+0000 7fc83bfff700 1 -- 172.21.15.38:0/1271759142 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc844022470 con 0x7fc8540ff010 2023-03-09T14:44:00.772 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:00.766+0000 7fc85948e700 1 -- 172.21.15.38:0/1271759142 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc85410a780 con 0x7fc8540ff010 2023-03-09T14:44:00.772 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:00.766+0000 7fc83bfff700 1 -- 172.21.15.38:0/1271759142 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc844017650 con 0x7fc8540ff010 2023-03-09T14:44:00.772 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:00.766+0000 7fc85948e700 1 -- 172.21.15.38:0/1271759142 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc85410a5f0 con 0x7fc8540ff010 2023-03-09T14:44:00.773 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:00.767+0000 7fc85948e700 1 -- 172.21.15.38:0/1271759142 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc854106110 con 0x7fc8540ff010 2023-03-09T14:44:00.773 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:00.768+0000 7fc83bfff700 1 -- 172.21.15.38:0/1271759142 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 16) v1 ==== 133198+0+0 (secure 0 0 0) 0x7fc844029030 con 0x7fc8540ff010 2023-03-09T14:44:00.773 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:00.768+0000 7fc83bfff700 1 --2- 172.21.15.38:0/1271759142 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fc83c09f3c0 0x7fc83c0a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:00.773 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:00.769+0000 7fc8527fc700 1 --2- 172.21.15.38:0/1271759142 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fc83c09f3c0 0x7fc83c0a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:00.773 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:00.769+0000 7fc83bfff700 1 -- 172.21.15.38:0/1271759142 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fc8440caba0 con 0x7fc8540ff010 2023-03-09T14:44:00.773 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:00.769+0000 7fc8527fc700 1 --2- 172.21.15.38:0/1271759142 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fc83c09f3c0 0x7fc83c0a1870 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fc848005950 tx=0x7fc84800a300 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:00.774 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:00.772+0000 7fc83bfff700 1 -- 172.21.15.38:0/1271759142 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7fc8440954e0 con 0x7fc8540ff010 2023-03-09T14:44:01.023 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:01.020+0000 7fc85948e700 1 -- 172.21.15.38:0/1271759142 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc85402cef0 con 0x7fc8540ff010 2023-03-09T14:44:01.023 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:01.021+0000 7fc83bfff700 1 -- 172.21.15.38:0/1271759142 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+5039 (secure 0 0 0) 0x7fc844098b00 con 0x7fc8540ff010 2023-03-09T14:44:01.024 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:44:01.024 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/768918602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":768918602},{"type":"v1","addr":"172.21.15.38:6835","nonce":768918602}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/18381540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":18381540},{"type":"v1","addr":"172.21.15.38:6837","nonce":18381540}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6833/415305571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":415305571},{"type":"v1","addr":"172.21.15.181:6833","nonce":415305571}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/659300520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":659300520},{"type":"v1","addr":"172.21.15.122:6833","nonce":659300520}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24448,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/2576255861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":2576255861},{"type":"v1","addr":"172.21.15.181:6835","nonce":2576255861}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/3675537941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":3675537941},{"type":"v1","addr":"172.21.15.122:6835","nonce":3675537941}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[]} 2023-03-09T14:44:01.026 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:01.023+0000 7fc85948e700 1 -- 172.21.15.38:0/1271759142 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fc83c09f3c0 msgr2=0x7fc83c0a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:01.026 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:01.023+0000 7fc85948e700 1 --2- 172.21.15.38:0/1271759142 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fc83c09f3c0 0x7fc83c0a1870 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fc848005950 tx=0x7fc84800a300 comp rx=0 tx=0).stop 2023-03-09T14:44:01.026 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:01.023+0000 7fc85948e700 1 -- 172.21.15.38:0/1271759142 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8540ff010 msgr2=0x7fc8540fea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:01.026 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:01.023+0000 7fc85948e700 1 --2- 172.21.15.38:0/1271759142 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8540ff010 0x7fc8540fea70 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc844005910 tx=0x7fc84400f690 comp rx=0 tx=0).stop 2023-03-09T14:44:01.026 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:01.023+0000 7fc85948e700 1 -- 172.21.15.38:0/1271759142 shutdown_connections 2023-03-09T14:44:01.027 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:01.023+0000 7fc85948e700 1 --2- 172.21.15.38:0/1271759142 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc854100450 0x7fc85419d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:01.027 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:01.023+0000 7fc85948e700 1 --2- 172.21.15.38:0/1271759142 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fc83c09f3c0 0x7fc83c0a1870 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:01.027 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:01.023+0000 7fc85948e700 1 --2- 172.21.15.38:0/1271759142 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc8540ff010 0x7fc8540fea70 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:01.027 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:01.023+0000 7fc85948e700 1 --2- 172.21.15.38:0/1271759142 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc8540ff9c0 0x7fc85419ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:01.027 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:01.024+0000 7fc85948e700 1 -- 172.21.15.38:0/1271759142 >> 172.21.15.38:0/1271759142 conn(0x7fc8540f9b30 msgr2=0x7fc854103250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:01.027 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:01.024+0000 7fc85948e700 1 -- 172.21.15.38:0/1271759142 shutdown_connections 2023-03-09T14:44:01.027 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:01.024+0000 7fc85948e700 1 -- 172.21.15.38:0/1271759142 wait complete. 2023-03-09T14:44:01.028 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 7 2023-03-09T14:44:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6834/2576255861,v1:172.21.15.181:6835/2576255861] up:boot 2023-03-09T14:44:01.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:01 smithi181 ceph-mon[94189]: fsmap 6 up:standby 2023-03-09T14:44:01.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:01 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-03-09T14:44:01.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:01 smithi181 ceph-mon[94189]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-03-09T14:44:01.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:01 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/1271759142' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-03-09T14:44:01.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6834/2576255861,v1:172.21.15.181:6835/2576255861] up:boot 2023-03-09T14:44:01.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:01 smithi122 ceph-mon[93869]: fsmap 6 up:standby 2023-03-09T14:44:01.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:01 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-03-09T14:44:01.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:01 smithi122 ceph-mon[93869]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-03-09T14:44:01.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:01 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/1271759142' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-03-09T14:44:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6834/2576255861,v1:172.21.15.181:6835/2576255861] up:boot 2023-03-09T14:44:01.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:01 smithi038 ceph-mon[92422]: fsmap 6 up:standby 2023-03-09T14:44:01.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:01 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-03-09T14:44:01.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:01 smithi038 ceph-mon[92422]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-03-09T14:44:01.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:01 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/1271759142' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-03-09T14:44:01.982 INFO:journalctl@ceph.mds.c.smithi181.stdout:Mar 09 14:44:01 smithi181 systemd[1]: Starting Ceph mds.c for c6733fe2-be87-11ed-9af2-001a4aab830c... 2023-03-09T14:44:02.278 INFO:journalctl@ceph.mds.c.smithi181.stdout:Mar 09 14:44:02 smithi181 podman[119993]: 2023-03-09T14:44:02.731 INFO:journalctl@ceph.mds.c.smithi181.stdout:Mar 09 14:44:02 smithi181 bash[119993]: 7815be79ff545ced3c4e69b6784ce2c680c1bcfb937909a804b330ef098cdc64 2023-03-09T14:44:02.732 INFO:journalctl@ceph.mds.c.smithi181.stdout:Mar 09 14:44:02 smithi181 ceph-mds[120033]: -- 172.21.15.181:0/3745782206 <== mon.1 v2:172.21.15.181:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x563911bd6b60 con 0x563911bab000 2023-03-09T14:44:02.732 INFO:journalctl@ceph.mds.c.smithi181.stdout:Mar 09 14:44:02 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mds-c[120009]: starting mds.c at 2023-03-09T14:44:02.732 INFO:journalctl@ceph.mds.c.smithi181.stdout:Mar 09 14:44:02 smithi181 systemd[1]: Started Ceph mds.c for c6733fe2-be87-11ed-9af2-001a4aab830c. 2023-03-09T14:44:03.039 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-09T14:44:03.179 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:03.176+0000 7fc349fd6700 1 -- 172.21.15.38:0/1435630871 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3440fed80 msgr2=0x7fc3440ff190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:03.179 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:03.176+0000 7fc349fd6700 1 --2- 172.21.15.38:0/1435630871 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3440fed80 0x7fc3440ff190 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc334009990 tx=0x7fc334009ca0 comp rx=0 tx=0).stop 2023-03-09T14:44:03.179 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:03.176+0000 7fc349fd6700 1 -- 172.21.15.38:0/1435630871 shutdown_connections 2023-03-09T14:44:03.179 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:03.176+0000 7fc349fd6700 1 --2- 172.21.15.38:0/1435630871 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3440fed80 0x7fc3440ff190 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:03.180 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:03.176+0000 7fc349fd6700 1 --2- 172.21.15.38:0/1435630871 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc344100900 0x7fc3441071c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:03.180 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:03.176+0000 7fc349fd6700 1 --2- 172.21.15.38:0/1435630871 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc3440fff70 0x7fc3441003c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:03.180 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:03.176+0000 7fc349fd6700 1 -- 172.21.15.38:0/1435630871 >> 172.21.15.38:0/1435630871 conn(0x7fc3440fa380 msgr2=0x7fc3440fc7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:03.180 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:03.176+0000 7fc349fd6700 1 -- 172.21.15.38:0/1435630871 shutdown_connections 2023-03-09T14:44:03.180 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:03.176+0000 7fc349fd6700 1 -- 172.21.15.38:0/1435630871 wait complete. 2023-03-09T14:44:03.180 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:03.177+0000 7fc349fd6700 1 Processor -- start 2023-03-09T14:44:03.182 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:03.177+0000 7fc349fd6700 1 -- start start 2023-03-09T14:44:03.182 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:03.177+0000 7fc349fd6700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc3440fed80 0x7fc3440685b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:03.182 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:03.177+0000 7fc349fd6700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3440fff70 0x7fc34419a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:03.182 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:03.177+0000 7fc349fd6700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc344100900 0x7fc34419cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:03.183 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:03.177+0000 7fc349fd6700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc34419d790 con 0x7fc3440fff70 2023-03-09T14:44:03.183 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:03.177+0000 7fc349fd6700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc34419d900 con 0x7fc344100900 2023-03-09T14:44:03.183 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:03.177+0000 7fc349fd6700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc34419dac0 con 0x7fc3440fed80 2023-03-09T14:44:03.183 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:03.177+0000 7fc342ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3440fff70 0x7fc34419a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:03.183 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:03.177+0000 7fc342ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3440fff70 0x7fc34419a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36482/0 (socket says 172.21.15.38:36482) 2023-03-09T14:44:03.183 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:03.177+0000 7fc342ffd700 1 -- 172.21.15.38:0/2221879548 learned_addr learned my addr 172.21.15.38:0/2221879548 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:44:03.183 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:03.178+0000 7fc3437fe700 1 --2- 172.21.15.38:0/2221879548 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc3440fed80 0x7fc3440685b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:03.183 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:03.178+0000 7fc343fff700 1 --2- 172.21.15.38:0/2221879548 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc344100900 0x7fc34419cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:03.184 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:03.178+0000 7fc342ffd700 1 -- 172.21.15.38:0/2221879548 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc3440fed80 msgr2=0x7fc3440685b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:03.184 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:03.178+0000 7fc342ffd700 1 --2- 172.21.15.38:0/2221879548 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc3440fed80 0x7fc3440685b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:03.184 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:03.178+0000 7fc342ffd700 1 -- 172.21.15.38:0/2221879548 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc344100900 msgr2=0x7fc34419cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:03.184 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:03.178+0000 7fc342ffd700 1 --2- 172.21.15.38:0/2221879548 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc344100900 0x7fc34419cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:03.184 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:03.178+0000 7fc342ffd700 1 -- 172.21.15.38:0/2221879548 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3340095f0 con 0x7fc3440fff70 2023-03-09T14:44:03.184 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:03.178+0000 7fc343fff700 1 --2- 172.21.15.38:0/2221879548 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc344100900 0x7fc34419cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-09T14:44:03.184 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:03.178+0000 7fc342ffd700 1 --2- 172.21.15.38:0/2221879548 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3440fff70 0x7fc34419a9b0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc32c00e9a0 tx=0x7fc32c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:03.184 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:03.178+0000 7fc340ff9700 1 -- 172.21.15.38:0/2221879548 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc32c009790 con 0x7fc3440fff70 2023-03-09T14:44:03.185 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:03.178+0000 7fc340ff9700 1 -- 172.21.15.38:0/2221879548 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc32c00cbd0 con 0x7fc3440fff70 2023-03-09T14:44:03.185 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:03.178+0000 7fc340ff9700 1 -- 172.21.15.38:0/2221879548 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc32c018680 con 0x7fc3440fff70 2023-03-09T14:44:03.185 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:03.178+0000 7fc349fd6700 1 -- 172.21.15.38:0/2221879548 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc34410a730 con 0x7fc3440fff70 2023-03-09T14:44:03.185 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:03.178+0000 7fc349fd6700 1 -- 172.21.15.38:0/2221879548 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc34410a5a0 con 0x7fc3440fff70 2023-03-09T14:44:03.185 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:03.179+0000 7fc349fd6700 1 -- 172.21.15.38:0/2221879548 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc344105d00 con 0x7fc3440fff70 2023-03-09T14:44:03.186 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:03.181+0000 7fc340ff9700 1 -- 172.21.15.38:0/2221879548 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 133198+0+0 (secure 0 0 0) 0x7fc32c00c040 con 0x7fc3440fff70 2023-03-09T14:44:03.186 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:03.181+0000 7fc340ff9700 1 --2- 172.21.15.38:0/2221879548 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fc33009f3c0 0x7fc3300a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:03.186 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:03.182+0000 7fc3437fe700 1 --2- 172.21.15.38:0/2221879548 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fc33009f3c0 0x7fc3300a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:03.186 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:03.182+0000 7fc340ff9700 1 -- 172.21.15.38:0/2221879548 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fc32c014070 con 0x7fc3440fff70 2023-03-09T14:44:03.187 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:03.182+0000 7fc3437fe700 1 --2- 172.21.15.38:0/2221879548 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fc33009f3c0 0x7fc3300a1870 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc33400b2e0 tx=0x7fc334005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:03.187 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:03.185+0000 7fc340ff9700 1 -- 172.21.15.38:0/2221879548 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7fc32c094730 con 0x7fc3440fff70 2023-03-09T14:44:03.434 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:03.432+0000 7fc349fd6700 1 -- 172.21.15.38:0/2221879548 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc344061610 con 0x7fc3440fff70 2023-03-09T14:44:03.435 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:03.433+0000 7fc340ff9700 1 -- 172.21.15.38:0/2221879548 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+5799 (secure 0 0 0) 0x7fc32c097d50 con 0x7fc3440fff70 2023-03-09T14:44:03.435 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:44:03.436 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/768918602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":768918602},{"type":"v1","addr":"172.21.15.38:6835","nonce":768918602}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/18381540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":18381540},{"type":"v1","addr":"172.21.15.38:6837","nonce":18381540}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6833/415305571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":415305571},{"type":"v1","addr":"172.21.15.181:6833","nonce":415305571}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/659300520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":659300520},{"type":"v1","addr":"172.21.15.122:6833","nonce":659300520}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24448,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/2576255861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":2576255861},{"type":"v1","addr":"172.21.15.181:6835","nonce":2576255861}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24460,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2942560181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":2942560181},{"type":"v1","addr":"172.21.15.181:6837","nonce":2942560181}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/3675537941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":3675537941},{"type":"v1","addr":"172.21.15.122:6835","nonce":3675537941}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[]} 2023-03-09T14:44:03.437 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:03.435+0000 7fc349fd6700 1 -- 172.21.15.38:0/2221879548 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fc33009f3c0 msgr2=0x7fc3300a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:03.438 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:03.435+0000 7fc349fd6700 1 --2- 172.21.15.38:0/2221879548 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fc33009f3c0 0x7fc3300a1870 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc33400b2e0 tx=0x7fc334005fb0 comp rx=0 tx=0).stop 2023-03-09T14:44:03.438 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:03.435+0000 7fc349fd6700 1 -- 172.21.15.38:0/2221879548 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3440fff70 msgr2=0x7fc34419a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:03.438 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:03.435+0000 7fc349fd6700 1 --2- 172.21.15.38:0/2221879548 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3440fff70 0x7fc34419a9b0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fc32c00e9a0 tx=0x7fc32c00ecb0 comp rx=0 tx=0).stop 2023-03-09T14:44:03.438 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:03.435+0000 7fc349fd6700 1 -- 172.21.15.38:0/2221879548 shutdown_connections 2023-03-09T14:44:03.438 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:03.435+0000 7fc349fd6700 1 --2- 172.21.15.38:0/2221879548 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fc33009f3c0 0x7fc3300a1870 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:03.438 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:03.435+0000 7fc349fd6700 1 --2- 172.21.15.38:0/2221879548 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc344100900 0x7fc34419cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:03.438 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:03.435+0000 7fc349fd6700 1 --2- 172.21.15.38:0/2221879548 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc3440fed80 0x7fc3440685b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:03.439 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:03.435+0000 7fc349fd6700 1 --2- 172.21.15.38:0/2221879548 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc3440fff70 0x7fc34419a9b0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:03.439 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:03.435+0000 7fc349fd6700 1 -- 172.21.15.38:0/2221879548 >> 172.21.15.38:0/2221879548 conn(0x7fc3440fa380 msgr2=0x7fc3440fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:03.439 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:03.435+0000 7fc349fd6700 1 -- 172.21.15.38:0/2221879548 shutdown_connections 2023-03-09T14:44:03.439 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:03.435+0000 7fc349fd6700 1 -- 172.21.15.38:0/2221879548 wait complete. 2023-03-09T14:44:03.439 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 8 2023-03-09T14:44:03.603 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:03 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:03.603 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:03 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:03.603 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:03 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-03-09T14:44:03.603 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:03 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-03-09T14:44:03.603 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:03 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:44:03.603 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:03 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6836/2942560181,v1:172.21.15.181:6837/2942560181] up:boot 2023-03-09T14:44:03.604 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:03 smithi122 ceph-mon[93869]: fsmap 7 up:standby 2023-03-09T14:44:03.604 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:03 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-03-09T14:44:03.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:03 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:03.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:03 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:03.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:03 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-03-09T14:44:03.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:03 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-03-09T14:44:03.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:03 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:44:03.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:03 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6836/2942560181,v1:172.21.15.181:6837/2942560181] up:boot 2023-03-09T14:44:03.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:03 smithi181 ceph-mon[94189]: fsmap 7 up:standby 2023-03-09T14:44:03.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:03 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-03-09T14:44:03.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:03 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:03.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:03 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:03.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:03 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-03-09T14:44:03.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:03 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-03-09T14:44:03.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:03 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:44:03.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:03 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6836/2942560181,v1:172.21.15.181:6837/2942560181] up:boot 2023-03-09T14:44:03.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:03 smithi038 ceph-mon[92422]: fsmap 7 up:standby 2023-03-09T14:44:03.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:03 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-03-09T14:44:03.861 INFO:journalctl@ceph.mds.h.smithi122.stdout:Mar 09 14:44:03 smithi122 systemd[1]: Starting Ceph mds.h for c6733fe2-be87-11ed-9af2-001a4aab830c... 2023-03-09T14:44:04.404 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:04 smithi122 ceph-mon[93869]: Deploying daemon mds.h on smithi122 2023-03-09T14:44:04.405 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:04 smithi122 ceph-mon[93869]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-03-09T14:44:04.405 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:04 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/2221879548' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-03-09T14:44:04.405 INFO:journalctl@ceph.mds.h.smithi122.stdout:Mar 09 14:44:04 smithi122 podman[118790]: 2023-03-09T14:44:04.728 INFO:journalctl@ceph.mds.h.smithi122.stdout:Mar 09 14:44:04 smithi122 ceph-mds[118830]: -- 172.21.15.122:0/1298846131 <== mon.2 v2:172.21.15.122:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5568d0158b60 con 0x5568d012d000 2023-03-09T14:44:04.728 INFO:journalctl@ceph.mds.h.smithi122.stdout:Mar 09 14:44:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mds-h[118806]: starting mds.h at 2023-03-09T14:44:04.728 INFO:journalctl@ceph.mds.h.smithi122.stdout:Mar 09 14:44:04 smithi122 bash[118790]: 0fa6bd943db53318a37101d850f6d63fc745d6ac3f38ad2e19a72c9e2755d240 2023-03-09T14:44:04.729 INFO:journalctl@ceph.mds.h.smithi122.stdout:Mar 09 14:44:04 smithi122 systemd[1]: Started Ceph mds.h for c6733fe2-be87-11ed-9af2-001a4aab830c. 2023-03-09T14:44:04.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:04 smithi181 ceph-mon[94189]: Deploying daemon mds.h on smithi122 2023-03-09T14:44:04.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:04 smithi181 ceph-mon[94189]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-03-09T14:44:04.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:04 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/2221879548' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-03-09T14:44:04.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:04 smithi038 ceph-mon[92422]: Deploying daemon mds.h on smithi122 2023-03-09T14:44:04.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:04 smithi038 ceph-mon[92422]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-03-09T14:44:04.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:04 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/2221879548' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-03-09T14:44:05.450 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-09T14:44:05.591 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:05.588+0000 7f452011a700 1 -- 172.21.15.38:0/1585040109 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f45180fed80 msgr2=0x7f45180ff190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:05.591 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:05.588+0000 7f452011a700 1 --2- 172.21.15.38:0/1585040109 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f45180fed80 0x7f45180ff190 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f4514009990 tx=0x7f4514009ca0 comp rx=0 tx=0).stop 2023-03-09T14:44:05.591 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:05.588+0000 7f452011a700 1 -- 172.21.15.38:0/1585040109 shutdown_connections 2023-03-09T14:44:05.591 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:05.588+0000 7f452011a700 1 --2- 172.21.15.38:0/1585040109 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f45180fed80 0x7f45180ff190 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:05.591 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:05.588+0000 7f452011a700 1 --2- 172.21.15.38:0/1585040109 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4518100900 0x7f45181071c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:05.592 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:05.588+0000 7f452011a700 1 --2- 172.21.15.38:0/1585040109 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f45180fff70 0x7f45181003c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:05.592 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:05.588+0000 7f452011a700 1 -- 172.21.15.38:0/1585040109 >> 172.21.15.38:0/1585040109 conn(0x7f45180fa380 msgr2=0x7f45180fc7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:05.592 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:05.588+0000 7f452011a700 1 -- 172.21.15.38:0/1585040109 shutdown_connections 2023-03-09T14:44:05.592 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:05.588+0000 7f452011a700 1 -- 172.21.15.38:0/1585040109 wait complete. 2023-03-09T14:44:05.594 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:05.589+0000 7f452011a700 1 Processor -- start 2023-03-09T14:44:05.594 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:05.589+0000 7f452011a700 1 -- start start 2023-03-09T14:44:05.594 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:05.589+0000 7f452011a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f45180fed80 0x7f45180685b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:05.594 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:05.589+0000 7f452011a700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f45180fff70 0x7f451819a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:05.594 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:05.589+0000 7f452011a700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4518100900 0x7f451819cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:05.595 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:05.589+0000 7f452011a700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f451819d750 con 0x7f45180fed80 2023-03-09T14:44:05.595 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:05.589+0000 7f452011a700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f451819d8c0 con 0x7f4518100900 2023-03-09T14:44:05.595 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:05.589+0000 7f452011a700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f451819da80 con 0x7f45180fff70 2023-03-09T14:44:05.595 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:05.589+0000 7f451deb6700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f45180fed80 0x7f45180685b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:05.595 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:05.589+0000 7f451deb6700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f45180fed80 0x7f45180685b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36494/0 (socket says 172.21.15.38:36494) 2023-03-09T14:44:05.595 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:05.589+0000 7f451deb6700 1 -- 172.21.15.38:0/4217138904 learned_addr learned my addr 172.21.15.38:0/4217138904 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:44:05.595 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:05.589+0000 7f451d6b5700 1 --2- 172.21.15.38:0/4217138904 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f45180fff70 0x7f451819a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:05.595 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:05.589+0000 7f451e6b7700 1 --2- 172.21.15.38:0/4217138904 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4518100900 0x7f451819cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:05.596 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:05.589+0000 7f451deb6700 1 -- 172.21.15.38:0/4217138904 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f45180fff70 msgr2=0x7f451819a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:05.596 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:05.589+0000 7f451deb6700 1 --2- 172.21.15.38:0/4217138904 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f45180fff70 0x7f451819a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:05.596 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:05.590+0000 7f451deb6700 1 -- 172.21.15.38:0/4217138904 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4518100900 msgr2=0x7f451819cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:05.596 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:05.590+0000 7f451deb6700 1 --2- 172.21.15.38:0/4217138904 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4518100900 0x7f451819cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:05.596 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:05.590+0000 7f451deb6700 1 -- 172.21.15.38:0/4217138904 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45140095f0 con 0x7f45180fed80 2023-03-09T14:44:05.596 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:05.590+0000 7f451e6b7700 1 --2- 172.21.15.38:0/4217138904 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4518100900 0x7f451819cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:44:05.596 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:05.590+0000 7f451deb6700 1 --2- 172.21.15.38:0/4217138904 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f45180fed80 0x7f45180685b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f4514000c00 tx=0x7f45140047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:05.596 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:05.590+0000 7f450affd700 1 -- 172.21.15.38:0/4217138904 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f451401d070 con 0x7f45180fed80 2023-03-09T14:44:05.597 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:05.590+0000 7f452011a700 1 -- 172.21.15.38:0/4217138904 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f451810a6e0 con 0x7f45180fed80 2023-03-09T14:44:05.597 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:05.590+0000 7f450affd700 1 -- 172.21.15.38:0/4217138904 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4514004060 con 0x7f45180fed80 2023-03-09T14:44:05.597 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:05.590+0000 7f450affd700 1 -- 172.21.15.38:0/4217138904 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4514022d30 con 0x7f45180fed80 2023-03-09T14:44:05.597 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:05.590+0000 7f452011a700 1 -- 172.21.15.38:0/4217138904 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f451810a550 con 0x7f45180fed80 2023-03-09T14:44:05.597 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:05.591+0000 7f452011a700 1 -- 172.21.15.38:0/4217138904 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4518106360 con 0x7f45180fed80 2023-03-09T14:44:05.598 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:05.593+0000 7f450affd700 1 -- 172.21.15.38:0/4217138904 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 133198+0+0 (secure 0 0 0) 0x7f4514029030 con 0x7f45180fed80 2023-03-09T14:44:05.598 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:05.594+0000 7f450affd700 1 --2- 172.21.15.38:0/4217138904 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f450409f3c0 0x7f45040a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:05.598 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:05.594+0000 7f451d6b5700 1 --2- 172.21.15.38:0/4217138904 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f450409f3c0 0x7f45040a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:05.598 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:05.594+0000 7f450affd700 1 -- 172.21.15.38:0/4217138904 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f45140caeb0 con 0x7f45180fed80 2023-03-09T14:44:05.598 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:05.594+0000 7f451d6b5700 1 --2- 172.21.15.38:0/4217138904 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f450409f3c0 0x7f45040a1870 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f450c005950 tx=0x7f450c00b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:05.599 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:05.597+0000 7f450affd700 1 -- 172.21.15.38:0/4217138904 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f4514095700 con 0x7f45180fed80 2023-03-09T14:44:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:05 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:05.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:05 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:05.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:05 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-03-09T14:44:05.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:05 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-03-09T14:44:05.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:05 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:44:05.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:05 smithi038 ceph-mon[92422]: Deploying daemon mds.i on smithi181 2023-03-09T14:44:05.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:05 smithi038 ceph-mon[92422]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-03-09T14:44:05.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:05 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] up:boot 2023-03-09T14:44:05.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:05 smithi038 ceph-mon[92422]: fsmap 8 up:standby 2023-03-09T14:44:05.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:05 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-03-09T14:44:05.848 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:05.846+0000 7f452011a700 1 -- 172.21.15.38:0/4217138904 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f45180615c0 con 0x7f45180fed80 2023-03-09T14:44:05.848 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:05.846+0000 7f450affd700 1 -- 172.21.15.38:0/4217138904 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6559 (secure 0 0 0) 0x7f4514098d20 con 0x7f45180fed80 2023-03-09T14:44:05.849 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:44:05.849 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/768918602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":768918602},{"type":"v1","addr":"172.21.15.38:6835","nonce":768918602}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/18381540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":18381540},{"type":"v1","addr":"172.21.15.38:6837","nonce":18381540}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6833/415305571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":415305571},{"type":"v1","addr":"172.21.15.181:6833","nonce":415305571}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/659300520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":659300520},{"type":"v1","addr":"172.21.15.122:6833","nonce":659300520}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24448,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/2576255861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":2576255861},{"type":"v1","addr":"172.21.15.181:6835","nonce":2576255861}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24460,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2942560181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":2942560181},{"type":"v1","addr":"172.21.15.181:6837","nonce":2942560181}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/3675537941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":3675537941},{"type":"v1","addr":"172.21.15.122:6835","nonce":3675537941}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24494,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/2210214596","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":2210214596},{"type":"v1","addr":"172.21.15.122:6837","nonce":2210214596}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9}],"filesystems":[]} 2023-03-09T14:44:05.851 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:05.848+0000 7f452011a700 1 -- 172.21.15.38:0/4217138904 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f450409f3c0 msgr2=0x7f45040a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:05.851 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:05.848+0000 7f452011a700 1 --2- 172.21.15.38:0/4217138904 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f450409f3c0 0x7f45040a1870 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f450c005950 tx=0x7f450c00b410 comp rx=0 tx=0).stop 2023-03-09T14:44:05.851 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:05.848+0000 7f452011a700 1 -- 172.21.15.38:0/4217138904 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f45180fed80 msgr2=0x7f45180685b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:05.851 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:05.848+0000 7f452011a700 1 --2- 172.21.15.38:0/4217138904 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f45180fed80 0x7f45180685b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f4514000c00 tx=0x7f45140047b0 comp rx=0 tx=0).stop 2023-03-09T14:44:05.851 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:05.848+0000 7f452011a700 1 -- 172.21.15.38:0/4217138904 shutdown_connections 2023-03-09T14:44:05.851 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:05.848+0000 7f452011a700 1 --2- 172.21.15.38:0/4217138904 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f45180fed80 0x7f45180685b0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:05.851 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:05.848+0000 7f452011a700 1 --2- 172.21.15.38:0/4217138904 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f450409f3c0 0x7f45040a1870 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:05.852 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:05.848+0000 7f452011a700 1 --2- 172.21.15.38:0/4217138904 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4518100900 0x7f451819cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:05.852 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:05.848+0000 7f452011a700 1 --2- 172.21.15.38:0/4217138904 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f45180fff70 0x7f451819a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:05.852 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:05.848+0000 7f452011a700 1 -- 172.21.15.38:0/4217138904 >> 172.21.15.38:0/4217138904 conn(0x7f45180fa380 msgr2=0x7f45181034a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:05.852 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:05.849+0000 7f452011a700 1 -- 172.21.15.38:0/4217138904 shutdown_connections 2023-03-09T14:44:05.852 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:05.849+0000 7f452011a700 1 -- 172.21.15.38:0/4217138904 wait complete. 2023-03-09T14:44:05.852 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 9 2023-03-09T14:44:05.861 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:05 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:05.861 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:05 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:05.861 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:05 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-03-09T14:44:05.861 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:05 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-03-09T14:44:05.861 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:05 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:44:05.862 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:05 smithi181 ceph-mon[94189]: Deploying daemon mds.i on smithi181 2023-03-09T14:44:05.862 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:05 smithi181 ceph-mon[94189]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-03-09T14:44:05.862 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:05 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] up:boot 2023-03-09T14:44:05.862 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:05 smithi181 ceph-mon[94189]: fsmap 8 up:standby 2023-03-09T14:44:05.862 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:05 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-03-09T14:44:05.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:05 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:05.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:05 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:05.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:05 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-03-09T14:44:05.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:05 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-03-09T14:44:05.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:05 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:44:05.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:05 smithi122 ceph-mon[93869]: Deploying daemon mds.i on smithi181 2023-03-09T14:44:05.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:05 smithi122 ceph-mon[93869]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-03-09T14:44:05.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:05 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] up:boot 2023-03-09T14:44:05.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:05 smithi122 ceph-mon[93869]: fsmap 8 up:standby 2023-03-09T14:44:05.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:05 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-03-09T14:44:06.231 INFO:journalctl@ceph.mds.i.smithi181.stdout:Mar 09 14:44:06 smithi181 systemd[1]: Starting Ceph mds.i for c6733fe2-be87-11ed-9af2-001a4aab830c... 2023-03-09T14:44:06.653 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:06 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/4217138904' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-03-09T14:44:06.653 INFO:journalctl@ceph.mds.i.smithi181.stdout:Mar 09 14:44:06 smithi181 podman[120337]: 2023-03-09T14:44:06.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:06 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/4217138904' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-03-09T14:44:06.981 INFO:journalctl@ceph.mds.i.smithi181.stdout:Mar 09 14:44:06 smithi181 bash[120337]: fc04bc42d9d067144c1d36168276e0e7e89d713396a54fbca9c2493df1b7efb2 2023-03-09T14:44:06.981 INFO:journalctl@ceph.mds.i.smithi181.stdout:Mar 09 14:44:06 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mds-i[120353]: starting mds.i at 2023-03-09T14:44:06.981 INFO:journalctl@ceph.mds.i.smithi181.stdout:Mar 09 14:44:06 smithi181 systemd[1]: Started Ceph mds.i for c6733fe2-be87-11ed-9af2-001a4aab830c. 2023-03-09T14:44:07.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:06 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/4217138904' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-03-09T14:44:07.863 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-09T14:44:07.995 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:07.992+0000 7f0af959e700 1 -- 172.21.15.38:0/509596289 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0af40feb90 msgr2=0x7f0af40fefa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:07.995 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:07.992+0000 7f0af959e700 1 --2- 172.21.15.38:0/509596289 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0af40feb90 0x7f0af40fefa0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f0ae4009990 tx=0x7f0ae4009ca0 comp rx=0 tx=0).stop 2023-03-09T14:44:07.995 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:07.992+0000 7f0af959e700 1 -- 172.21.15.38:0/509596289 shutdown_connections 2023-03-09T14:44:07.996 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:07.993+0000 7f0af959e700 1 --2- 172.21.15.38:0/509596289 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0af40feb90 0x7f0af40fefa0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:07.996 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:07.993+0000 7f0af959e700 1 --2- 172.21.15.38:0/509596289 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0af41007d0 0x7f0af4107070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:07.996 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:07.993+0000 7f0af959e700 1 --2- 172.21.15.38:0/509596289 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0af40ffe20 0x7f0af4100290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:07.996 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:07.993+0000 7f0af959e700 1 -- 172.21.15.38:0/509596289 >> 172.21.15.38:0/509596289 conn(0x7f0af40fa1d0 msgr2=0x7f0af40fc5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:07.996 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:07.993+0000 7f0af959e700 1 -- 172.21.15.38:0/509596289 shutdown_connections 2023-03-09T14:44:07.996 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:07.993+0000 7f0af959e700 1 -- 172.21.15.38:0/509596289 wait complete. 2023-03-09T14:44:07.998 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:07.993+0000 7f0af959e700 1 Processor -- start 2023-03-09T14:44:07.998 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:07.993+0000 7f0af959e700 1 -- start start 2023-03-09T14:44:07.998 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:07.994+0000 7f0af959e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0af40feb90 0x7f0af40697e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:07.999 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:07.994+0000 7f0af959e700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0af40ffe20 0x7f0af419a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:07.999 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:07.994+0000 7f0af959e700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0af41007d0 0x7f0af419ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:07.999 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:07.994+0000 7f0af959e700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0af419d5a0 con 0x7f0af40feb90 2023-03-09T14:44:07.999 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:07.994+0000 7f0af959e700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0af419d710 con 0x7f0af40ffe20 2023-03-09T14:44:07.999 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:07.994+0000 7f0af959e700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0af419d8d0 con 0x7f0af41007d0 2023-03-09T14:44:07.999 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:07.994+0000 7f0af3fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0af40feb90 0x7f0af40697e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:07.999 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:07.994+0000 7f0af3fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0af40feb90 0x7f0af40697e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36508/0 (socket says 172.21.15.38:36508) 2023-03-09T14:44:08.000 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:07.994+0000 7f0af3fff700 1 -- 172.21.15.38:0/2923179786 learned_addr learned my addr 172.21.15.38:0/2923179786 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:44:08.000 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:07.994+0000 7f0af37fe700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0af40ffe20 0x7f0af419a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:08.000 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:07.994+0000 7f0af8d9d700 1 --2- 172.21.15.38:0/2923179786 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0af41007d0 0x7f0af419ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:08.000 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:07.994+0000 7f0af3fff700 1 -- 172.21.15.38:0/2923179786 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0af41007d0 msgr2=0x7f0af419ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:08.000 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:07.994+0000 7f0af3fff700 1 --2- 172.21.15.38:0/2923179786 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0af41007d0 0x7f0af419ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:08.000 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:07.994+0000 7f0af3fff700 1 -- 172.21.15.38:0/2923179786 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0af40ffe20 msgr2=0x7f0af419a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:08.000 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:07.994+0000 7f0af3fff700 1 --2- 172.21.15.38:0/2923179786 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0af40ffe20 0x7f0af419a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:08.001 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:07.994+0000 7f0af3fff700 1 -- 172.21.15.38:0/2923179786 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ae40095f0 con 0x7f0af40feb90 2023-03-09T14:44:08.001 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:07.994+0000 7f0af37fe700 1 --2- 172.21.15.38:0/2923179786 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0af40ffe20 0x7f0af419a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:44:08.001 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:07.994+0000 7f0af3fff700 1 --2- 172.21.15.38:0/2923179786 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0af40feb90 0x7f0af40697e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f0ae4004730 tx=0x7f0ae4004760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:08.001 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:07.994+0000 7f0af17fa700 1 -- 172.21.15.38:0/2923179786 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ae401d070 con 0x7f0af40feb90 2023-03-09T14:44:08.001 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:07.995+0000 7f0af17fa700 1 -- 172.21.15.38:0/2923179786 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0ae4004c90 con 0x7f0af40feb90 2023-03-09T14:44:08.001 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:07.995+0000 7f0af959e700 1 -- 172.21.15.38:0/2923179786 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0af410a540 con 0x7f0af40feb90 2023-03-09T14:44:08.001 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:07.995+0000 7f0af17fa700 1 -- 172.21.15.38:0/2923179786 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ae4005570 con 0x7f0af40feb90 2023-03-09T14:44:08.001 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:07.995+0000 7f0af959e700 1 -- 172.21.15.38:0/2923179786 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0af410a3b0 con 0x7f0af40feb90 2023-03-09T14:44:08.002 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:07.995+0000 7f0af959e700 1 -- 172.21.15.38:0/2923179786 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0af4105bb0 con 0x7f0af40feb90 2023-03-09T14:44:08.002 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:07.996+0000 7f0af17fa700 1 -- 172.21.15.38:0/2923179786 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 133198+0+0 (secure 0 0 0) 0x7f0ae400b050 con 0x7f0af40feb90 2023-03-09T14:44:08.003 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:07.997+0000 7f0af17fa700 1 --2- 172.21.15.38:0/2923179786 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f0adc09f3c0 0x7f0adc0a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:08.003 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:07.997+0000 7f0af37fe700 1 --2- 172.21.15.38:0/2923179786 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f0adc09f3c0 0x7f0adc0a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:08.003 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:07.998+0000 7f0af17fa700 1 -- 172.21.15.38:0/2923179786 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f0ae40caff0 con 0x7f0af40feb90 2023-03-09T14:44:08.003 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:07.998+0000 7f0af37fe700 1 --2- 172.21.15.38:0/2923179786 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f0adc09f3c0 0x7f0adc0a1870 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f0ae8011490 tx=0x7f0ae8009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:08.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:07 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:08.004 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:07 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:08.004 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:07 smithi038 ceph-mon[92422]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-03-09T14:44:08.004 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:07 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-03-09T14:44:08.004 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:07 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-03-09T14:44:08.004 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:07 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:44:08.004 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:07 smithi038 ceph-mon[92422]: Deploying daemon mds.g on smithi038 2023-03-09T14:44:08.004 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:boot 2023-03-09T14:44:08.005 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:07 smithi038 ceph-mon[92422]: fsmap 9 up:standby 2023-03-09T14:44:08.005 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:07 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-03-09T14:44:08.005 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:08.001+0000 7f0af17fa700 1 -- 172.21.15.38:0/2923179786 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f0ae4095930 con 0x7f0af40feb90 2023-03-09T14:44:08.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:07 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:08.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:07 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:08.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:07 smithi122 ceph-mon[93869]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-03-09T14:44:08.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:07 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-03-09T14:44:08.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:07 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-03-09T14:44:08.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:07 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:44:08.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:07 smithi122 ceph-mon[93869]: Deploying daemon mds.g on smithi038 2023-03-09T14:44:08.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:boot 2023-03-09T14:44:08.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:07 smithi122 ceph-mon[93869]: fsmap 9 up:standby 2023-03-09T14:44:08.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:07 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-03-09T14:44:08.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:07 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:08.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:07 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:08.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:07 smithi181 ceph-mon[94189]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-03-09T14:44:08.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:07 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-03-09T14:44:08.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:07 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-03-09T14:44:08.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:07 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:44:08.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:07 smithi181 ceph-mon[94189]: Deploying daemon mds.g on smithi038 2023-03-09T14:44:08.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:boot 2023-03-09T14:44:08.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:07 smithi181 ceph-mon[94189]: fsmap 9 up:standby 2023-03-09T14:44:08.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:07 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-03-09T14:44:08.274 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:08.272+0000 7f0af959e700 1 -- 172.21.15.38:0/2923179786 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0af402cef0 con 0x7f0af40feb90 2023-03-09T14:44:08.275 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:08.273+0000 7f0af17fa700 1 -- 172.21.15.38:0/2923179786 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+7321 (secure 0 0 0) 0x7f0ae4098f50 con 0x7f0af40feb90 2023-03-09T14:44:08.276 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:44:08.276 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/768918602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":768918602},{"type":"v1","addr":"172.21.15.38:6835","nonce":768918602}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/18381540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":18381540},{"type":"v1","addr":"172.21.15.38:6837","nonce":18381540}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6833/415305571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":415305571},{"type":"v1","addr":"172.21.15.181:6833","nonce":415305571}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/659300520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":659300520},{"type":"v1","addr":"172.21.15.122:6833","nonce":659300520}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24448,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/2576255861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":2576255861},{"type":"v1","addr":"172.21.15.181:6835","nonce":2576255861}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24460,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2942560181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":2942560181},{"type":"v1","addr":"172.21.15.181:6837","nonce":2942560181}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/3675537941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":3675537941},{"type":"v1","addr":"172.21.15.122:6835","nonce":3675537941}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24478,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6839/3902263660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":3902263660},{"type":"v1","addr":"172.21.15.181:6839","nonce":3902263660}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24494,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/2210214596","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":2210214596},{"type":"v1","addr":"172.21.15.122:6837","nonce":2210214596}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9}],"filesystems":[]} 2023-03-09T14:44:08.278 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:08.275+0000 7f0af959e700 1 -- 172.21.15.38:0/2923179786 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f0adc09f3c0 msgr2=0x7f0adc0a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:08.278 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:08.275+0000 7f0af959e700 1 --2- 172.21.15.38:0/2923179786 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f0adc09f3c0 0x7f0adc0a1870 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f0ae8011490 tx=0x7f0ae8009260 comp rx=0 tx=0).stop 2023-03-09T14:44:08.278 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:08.275+0000 7f0af959e700 1 -- 172.21.15.38:0/2923179786 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0af40feb90 msgr2=0x7f0af40697e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:08.278 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:08.275+0000 7f0af959e700 1 --2- 172.21.15.38:0/2923179786 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0af40feb90 0x7f0af40697e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f0ae4004730 tx=0x7f0ae4004760 comp rx=0 tx=0).stop 2023-03-09T14:44:08.278 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:08.276+0000 7f0af959e700 1 -- 172.21.15.38:0/2923179786 shutdown_connections 2023-03-09T14:44:08.279 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:08.276+0000 7f0af959e700 1 --2- 172.21.15.38:0/2923179786 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0af40feb90 0x7f0af40697e0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:08.279 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:08.276+0000 7f0af959e700 1 --2- 172.21.15.38:0/2923179786 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f0adc09f3c0 0x7f0adc0a1870 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:08.279 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:08.276+0000 7f0af959e700 1 --2- 172.21.15.38:0/2923179786 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0af41007d0 0x7f0af419ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:08.279 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:08.276+0000 7f0af959e700 1 --2- 172.21.15.38:0/2923179786 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0af40ffe20 0x7f0af419a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:08.279 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:08.276+0000 7f0af959e700 1 -- 172.21.15.38:0/2923179786 >> 172.21.15.38:0/2923179786 conn(0x7f0af40fa1d0 msgr2=0x7f0af40fc470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:08.279 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:08.276+0000 7f0af959e700 1 -- 172.21.15.38:0/2923179786 shutdown_connections 2023-03-09T14:44:08.279 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:08.276+0000 7f0af959e700 1 -- 172.21.15.38:0/2923179786 wait complete. 2023-03-09T14:44:08.280 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 10 2023-03-09T14:44:08.610 INFO:journalctl@ceph.mds.g.smithi038.stdout:Mar 09 14:44:08 smithi038 systemd[1]: Starting Ceph mds.g for c6733fe2-be87-11ed-9af2-001a4aab830c... 2023-03-09T14:44:08.878 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:08 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/2923179786' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-03-09T14:44:08.879 INFO:journalctl@ceph.mds.g.smithi038.stdout:Mar 09 14:44:08 smithi038 podman[126471]: 2023-03-09T14:44:09.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:08 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/2923179786' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-03-09T14:44:09.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:08 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/2923179786' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-03-09T14:44:09.250 INFO:journalctl@ceph.mds.g.smithi038.stdout:Mar 09 14:44:08 smithi038 ceph-mds[126511]: -- 172.21.15.38:0/2664740794 <== mon.0 v2:172.21.15.38:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55dd69302b60 con 0x55dd692d7000 2023-03-09T14:44:09.251 INFO:journalctl@ceph.mds.g.smithi038.stdout:Mar 09 14:44:08 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mds-g[126487]: starting mds.g at 2023-03-09T14:44:09.251 INFO:journalctl@ceph.mds.g.smithi038.stdout:Mar 09 14:44:08 smithi038 bash[126471]: 0d0e9b81d97b35d6b94c550db1d43175609c2cd5d0459e1c7ea25261b28b30d4 2023-03-09T14:44:09.251 INFO:journalctl@ceph.mds.g.smithi038.stdout:Mar 09 14:44:08 smithi038 systemd[1]: Started Ceph mds.g for c6733fe2-be87-11ed-9af2-001a4aab830c. 2023-03-09T14:44:10.294 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-09T14:44:10.323 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:10 smithi038 ceph-mon[92422]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-03-09T14:44:10.324 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:10 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:10.324 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:10 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:10.324 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:10 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-03-09T14:44:10.324 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:10 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-03-09T14:44:10.324 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:10 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:44:10.324 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:10 smithi038 ceph-mon[92422]: Deploying daemon mds.a on smithi038 2023-03-09T14:44:10.325 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:10 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.38:6838/4139020066,v1:172.21.15.38:6839/4139020066] up:boot 2023-03-09T14:44:10.325 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:10 smithi038 ceph-mon[92422]: fsmap 10 up:standby 2023-03-09T14:44:10.325 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:10 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-03-09T14:44:10.446 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:10.443+0000 7f466e2dd700 1 -- 172.21.15.38:0/3814629278 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f46680ffea0 msgr2=0x7f46681002b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:10.446 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:10.443+0000 7f466e2dd700 1 --2- 172.21.15.38:0/3814629278 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f46680ffea0 0x7f46681002b0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f4658009990 tx=0x7f4658009ca0 comp rx=0 tx=0).stop 2023-03-09T14:44:10.446 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:10.443+0000 7f466e2dd700 1 -- 172.21.15.38:0/3814629278 shutdown_connections 2023-03-09T14:44:10.446 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:10.443+0000 7f466e2dd700 1 --2- 172.21.15.38:0/3814629278 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f46680ffea0 0x7f46681002b0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:10.446 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:10.443+0000 7f466e2dd700 1 --2- 172.21.15.38:0/3814629278 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46681019a0 0x7f46681082d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:10.447 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:10.443+0000 7f466e2dd700 1 --2- 172.21.15.38:0/3814629278 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4668100ff0 0x7f4668101460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:10.447 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:10.443+0000 7f466e2dd700 1 -- 172.21.15.38:0/3814629278 >> 172.21.15.38:0/3814629278 conn(0x7f46680fb480 msgr2=0x7f46680fd8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:10.447 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:10.443+0000 7f466e2dd700 1 -- 172.21.15.38:0/3814629278 shutdown_connections 2023-03-09T14:44:10.447 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:10.443+0000 7f466e2dd700 1 -- 172.21.15.38:0/3814629278 wait complete. 2023-03-09T14:44:10.449 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:10.444+0000 7f466e2dd700 1 Processor -- start 2023-03-09T14:44:10.449 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:10.444+0000 7f466e2dd700 1 -- start start 2023-03-09T14:44:10.449 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:10.444+0000 7f466e2dd700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46680ffea0 0x7f4668195c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:10.449 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:10.444+0000 7f466e2dd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4668100ff0 0x7f46681961c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:10.449 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:10.444+0000 7f466e2dd700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f46681019a0 0x7f466819d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:10.450 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:10.444+0000 7f466e2dd700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f466819d7c0 con 0x7f4668100ff0 2023-03-09T14:44:10.450 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:10.444+0000 7f466e2dd700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f466819d930 con 0x7f46681019a0 2023-03-09T14:44:10.450 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:10.444+0000 7f466e2dd700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f466819daf0 con 0x7f46680ffea0 2023-03-09T14:44:10.450 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:10.444+0000 7f465ffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4668100ff0 0x7f46681961c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:10.450 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:10.444+0000 7f465ffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4668100ff0 0x7f46681961c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36536/0 (socket says 172.21.15.38:36536) 2023-03-09T14:44:10.450 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:10.444+0000 7f465ffff700 1 -- 172.21.15.38:0/1220010852 learned_addr learned my addr 172.21.15.38:0/1220010852 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:44:10.450 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:10.444+0000 7f4667fff700 1 --2- 172.21.15.38:0/1220010852 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46680ffea0 0x7f4668195c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:10.451 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:10.445+0000 7f466c87a700 1 --2- 172.21.15.38:0/1220010852 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f46681019a0 0x7f466819d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:10.451 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:10.445+0000 7f465ffff700 1 -- 172.21.15.38:0/1220010852 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46680ffea0 msgr2=0x7f4668195c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:10.451 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:10.445+0000 7f465ffff700 1 --2- 172.21.15.38:0/1220010852 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46680ffea0 0x7f4668195c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:10.451 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:10.445+0000 7f465ffff700 1 -- 172.21.15.38:0/1220010852 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f46681019a0 msgr2=0x7f466819d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:10.451 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:10.445+0000 7f465ffff700 1 --2- 172.21.15.38:0/1220010852 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f46681019a0 0x7f466819d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:10.451 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:10.445+0000 7f465ffff700 1 -- 172.21.15.38:0/1220010852 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46580095f0 con 0x7f4668100ff0 2023-03-09T14:44:10.451 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:10.445+0000 7f465ffff700 1 --2- 172.21.15.38:0/1220010852 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4668100ff0 0x7f46681961c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f465000c920 tx=0x7f465000cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:10.452 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:10.445+0000 7f4665ffb700 1 -- 172.21.15.38:0/1220010852 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4650007b00 con 0x7f4668100ff0 2023-03-09T14:44:10.452 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:10.445+0000 7f4665ffb700 1 -- 172.21.15.38:0/1220010852 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f465000ce70 con 0x7f4668100ff0 2023-03-09T14:44:10.452 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:10.445+0000 7f466e2dd700 1 -- 172.21.15.38:0/1220010852 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f466810b850 con 0x7f4668100ff0 2023-03-09T14:44:10.452 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:10.445+0000 7f4665ffb700 1 -- 172.21.15.38:0/1220010852 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46500054e0 con 0x7f4668100ff0 2023-03-09T14:44:10.452 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:10.445+0000 7f466e2dd700 1 -- 172.21.15.38:0/1220010852 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f466810b6c0 con 0x7f4668100ff0 2023-03-09T14:44:10.452 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:10.446+0000 7f466e2dd700 1 -- 172.21.15.38:0/1220010852 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f466819e150 con 0x7f4668100ff0 2023-03-09T14:44:10.453 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:10.447+0000 7f4665ffb700 1 -- 172.21.15.38:0/1220010852 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 133198+0+0 (secure 0 0 0) 0x7f465001f070 con 0x7f4668100ff0 2023-03-09T14:44:10.453 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:10.448+0000 7f4665ffb700 1 --2- 172.21.15.38:0/1220010852 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f465409f3c0 0x7f46540a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:10.453 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:10.448+0000 7f4667fff700 1 --2- 172.21.15.38:0/1220010852 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f465409f3c0 0x7f46540a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:10.453 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:10.448+0000 7f4665ffb700 1 -- 172.21.15.38:0/1220010852 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f46500c9770 con 0x7f4668100ff0 2023-03-09T14:44:10.454 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:10.448+0000 7f4667fff700 1 --2- 172.21.15.38:0/1220010852 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f465409f3c0 0x7f46540a1870 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f465800b5c0 tx=0x7f4658005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:10.457 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:10.455+0000 7f4665ffb700 1 -- 172.21.15.38:0/1220010852 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f4650094020 con 0x7f4668100ff0 2023-03-09T14:44:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:10 smithi122 ceph-mon[93869]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-03-09T14:44:10.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:10 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:10.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:10 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:10.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:10 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-03-09T14:44:10.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:10 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-03-09T14:44:10.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:10 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:44:10.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:10 smithi122 ceph-mon[93869]: Deploying daemon mds.a on smithi038 2023-03-09T14:44:10.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:10 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.38:6838/4139020066,v1:172.21.15.38:6839/4139020066] up:boot 2023-03-09T14:44:10.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:10 smithi122 ceph-mon[93869]: fsmap 10 up:standby 2023-03-09T14:44:10.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:10 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-03-09T14:44:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:10 smithi181 ceph-mon[94189]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-03-09T14:44:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:10 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:10.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:10 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:10.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:10 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-03-09T14:44:10.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:10 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-03-09T14:44:10.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:10 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:44:10.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:10 smithi181 ceph-mon[94189]: Deploying daemon mds.a on smithi038 2023-03-09T14:44:10.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:10 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.38:6838/4139020066,v1:172.21.15.38:6839/4139020066] up:boot 2023-03-09T14:44:10.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:10 smithi181 ceph-mon[94189]: fsmap 10 up:standby 2023-03-09T14:44:10.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:10 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-03-09T14:44:10.671 INFO:journalctl@ceph.mds.a.smithi038.stdout:Mar 09 14:44:10 smithi038 systemd[1]: Starting Ceph mds.a for c6733fe2-be87-11ed-9af2-001a4aab830c... 2023-03-09T14:44:10.717 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:10.715+0000 7f466e2dd700 1 -- 172.21.15.38:0/1220010852 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4668106dd0 con 0x7f4668100ff0 2023-03-09T14:44:10.718 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:10.716+0000 7f4665ffb700 1 -- 172.21.15.38:0/1220010852 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8079 (secure 0 0 0) 0x7f465001d090 con 0x7f4668100ff0 2023-03-09T14:44:10.718 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:44:10.718 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/768918602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":768918602},{"type":"v1","addr":"172.21.15.38:6835","nonce":768918602}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/18381540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":18381540},{"type":"v1","addr":"172.21.15.38:6837","nonce":18381540}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14610,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6839/4139020066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":4139020066},{"type":"v1","addr":"172.21.15.38:6839","nonce":4139020066}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6833/415305571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":415305571},{"type":"v1","addr":"172.21.15.181:6833","nonce":415305571}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/659300520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":659300520},{"type":"v1","addr":"172.21.15.122:6833","nonce":659300520}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24448,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/2576255861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":2576255861},{"type":"v1","addr":"172.21.15.181:6835","nonce":2576255861}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24460,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2942560181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":2942560181},{"type":"v1","addr":"172.21.15.181:6837","nonce":2942560181}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/3675537941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":3675537941},{"type":"v1","addr":"172.21.15.122:6835","nonce":3675537941}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24478,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6839/3902263660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":3902263660},{"type":"v1","addr":"172.21.15.181:6839","nonce":3902263660}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24494,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/2210214596","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":2210214596},{"type":"v1","addr":"172.21.15.122:6837","nonce":2210214596}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9}],"filesystems":[]} 2023-03-09T14:44:10.721 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:10.719+0000 7f466e2dd700 1 -- 172.21.15.38:0/1220010852 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f465409f3c0 msgr2=0x7f46540a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:10.721 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:10.719+0000 7f466e2dd700 1 --2- 172.21.15.38:0/1220010852 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f465409f3c0 0x7f46540a1870 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f465800b5c0 tx=0x7f4658005fb0 comp rx=0 tx=0).stop 2023-03-09T14:44:10.722 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:10.719+0000 7f466e2dd700 1 -- 172.21.15.38:0/1220010852 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4668100ff0 msgr2=0x7f46681961c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:10.722 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:10.719+0000 7f466e2dd700 1 --2- 172.21.15.38:0/1220010852 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4668100ff0 0x7f46681961c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f465000c920 tx=0x7f465000cc30 comp rx=0 tx=0).stop 2023-03-09T14:44:10.722 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:10.719+0000 7f466e2dd700 1 -- 172.21.15.38:0/1220010852 shutdown_connections 2023-03-09T14:44:10.722 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:10.719+0000 7f466e2dd700 1 --2- 172.21.15.38:0/1220010852 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f465409f3c0 0x7f46540a1870 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:10.722 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:10.719+0000 7f466e2dd700 1 --2- 172.21.15.38:0/1220010852 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f46681019a0 0x7f466819d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:10.722 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:10.719+0000 7f466e2dd700 1 --2- 172.21.15.38:0/1220010852 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f46680ffea0 0x7f4668195c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:10.722 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:10.719+0000 7f466e2dd700 1 --2- 172.21.15.38:0/1220010852 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4668100ff0 0x7f46681961c0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:10.723 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:10.719+0000 7f466e2dd700 1 -- 172.21.15.38:0/1220010852 >> 172.21.15.38:0/1220010852 conn(0x7f46680fb480 msgr2=0x7f46680fd770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:10.723 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:10.719+0000 7f466e2dd700 1 -- 172.21.15.38:0/1220010852 shutdown_connections 2023-03-09T14:44:10.723 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:10.719+0000 7f466e2dd700 1 -- 172.21.15.38:0/1220010852 wait complete. 2023-03-09T14:44:10.723 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 11 2023-03-09T14:44:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:11 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/1220010852' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-03-09T14:44:11.251 INFO:journalctl@ceph.mds.a.smithi038.stdout:Mar 09 14:44:10 smithi038 podman[126857]: 2023-03-09T14:44:11.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:11 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/1220010852' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-03-09T14:44:11.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:11 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/1220010852' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-03-09T14:44:11.750 INFO:journalctl@ceph.mds.a.smithi038.stdout:Mar 09 14:44:11 smithi038 ceph-mds[126897]: -- 172.21.15.38:0/1359713315 <== mon.0 v2:172.21.15.38:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d98776ab60 con 0x55d98773ec00 2023-03-09T14:44:11.751 INFO:journalctl@ceph.mds.a.smithi038.stdout:Mar 09 14:44:11 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mds-a[126873]: starting mds.a at 2023-03-09T14:44:11.751 INFO:journalctl@ceph.mds.a.smithi038.stdout:Mar 09 14:44:11 smithi038 bash[126857]: 42183dd3578010b3c607408494f21b5143c6eb65c464fa576b27f93cc63ae97e 2023-03-09T14:44:11.751 INFO:journalctl@ceph.mds.a.smithi038.stdout:Mar 09 14:44:11 smithi038 systemd[1]: Started Ceph mds.a for c6733fe2-be87-11ed-9af2-001a4aab830c. 2023-03-09T14:44:12.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:12 smithi122 ceph-mon[93869]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-03-09T14:44:12.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:12 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:12.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:12 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:12.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:12 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-03-09T14:44:12.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:12 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-03-09T14:44:12.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:12 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:44:12.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.38:6840/1397115322,v1:172.21.15.38:6841/1397115322] up:boot 2023-03-09T14:44:12.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:12 smithi122 ceph-mon[93869]: fsmap 11 up:standby 2023-03-09T14:44:12.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:12 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-03-09T14:44:12.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:12 smithi181 ceph-mon[94189]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-03-09T14:44:12.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:12 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:12.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:12 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:12.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:12 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-03-09T14:44:12.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:12 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-03-09T14:44:12.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:12 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:44:12.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.38:6840/1397115322,v1:172.21.15.38:6841/1397115322] up:boot 2023-03-09T14:44:12.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:12 smithi181 ceph-mon[94189]: fsmap 11 up:standby 2023-03-09T14:44:12.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:12 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-03-09T14:44:12.737 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-09T14:44:12.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:12 smithi038 ceph-mon[92422]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-03-09T14:44:12.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:12 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:12.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:12 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:12.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:12 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-03-09T14:44:12.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:12 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-03-09T14:44:12.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:12 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:44:12.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.38:6840/1397115322,v1:172.21.15.38:6841/1397115322] up:boot 2023-03-09T14:44:12.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:12 smithi038 ceph-mon[92422]: fsmap 11 up:standby 2023-03-09T14:44:12.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:12 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-03-09T14:44:12.881 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:12.878+0000 7f6f45ab0700 1 -- 172.21.15.38:0/1820690997 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f401009b0 msgr2=0x7f6f40102d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:12.881 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:12.878+0000 7f6f45ab0700 1 --2- 172.21.15.38:0/1820690997 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f401009b0 0x7f6f40102d90 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f6f3800b0a0 tx=0x7f6f3800b3b0 comp rx=0 tx=0).stop 2023-03-09T14:44:12.881 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:12.878+0000 7f6f45ab0700 1 -- 172.21.15.38:0/1820690997 shutdown_connections 2023-03-09T14:44:12.881 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:12.878+0000 7f6f45ab0700 1 --2- 172.21.15.38:0/1820690997 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f401009b0 0x7f6f40102d90 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:12.882 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:12.878+0000 7f6f45ab0700 1 --2- 172.21.15.38:0/1820690997 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6f40105bf0 0x7f6f40107ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:12.882 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:12.878+0000 7f6f45ab0700 1 --2- 172.21.15.38:0/1820690997 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6f401032d0 0x7f6f401056b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:12.882 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:12.878+0000 7f6f45ab0700 1 -- 172.21.15.38:0/1820690997 >> 172.21.15.38:0/1820690997 conn(0x7f6f400fa3b0 msgr2=0x7f6f400fc820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:12.882 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:12.878+0000 7f6f45ab0700 1 -- 172.21.15.38:0/1820690997 shutdown_connections 2023-03-09T14:44:12.882 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:12.878+0000 7f6f45ab0700 1 -- 172.21.15.38:0/1820690997 wait complete. 2023-03-09T14:44:12.882 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:12.879+0000 7f6f45ab0700 1 Processor -- start 2023-03-09T14:44:12.884 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:12.879+0000 7f6f45ab0700 1 -- start start 2023-03-09T14:44:12.884 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:12.879+0000 7f6f45ab0700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6f401009b0 0x7f6f40195d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:12.884 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:12.879+0000 7f6f45ab0700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6f401032d0 0x7f6f40196280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:12.885 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:12.879+0000 7f6f45ab0700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f40105bf0 0x7f6f4019d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:12.885 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:12.879+0000 7f6f45ab0700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f4019d880 con 0x7f6f40105bf0 2023-03-09T14:44:12.885 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:12.879+0000 7f6f45ab0700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f4019d9f0 con 0x7f6f401032d0 2023-03-09T14:44:12.885 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:12.879+0000 7f6f45ab0700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f4019dbb0 con 0x7f6f401009b0 2023-03-09T14:44:12.885 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:12.879+0000 7f6f3ffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f40105bf0 0x7f6f4019d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:12.885 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:12.879+0000 7f6f3f7fe700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6f401009b0 0x7f6f40195d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:12.885 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:12.879+0000 7f6f3ffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f40105bf0 0x7f6f4019d2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36566/0 (socket says 172.21.15.38:36566) 2023-03-09T14:44:12.886 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:12.879+0000 7f6f3ffff700 1 -- 172.21.15.38:0/2301461332 learned_addr learned my addr 172.21.15.38:0/2301461332 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:44:12.886 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:12.879+0000 7f6f3effd700 1 --2- 172.21.15.38:0/2301461332 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6f401032d0 0x7f6f40196280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:12.886 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:12.880+0000 7f6f3ffff700 1 -- 172.21.15.38:0/2301461332 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6f401009b0 msgr2=0x7f6f40195d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:12.886 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:12.880+0000 7f6f3ffff700 1 --2- 172.21.15.38:0/2301461332 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6f401009b0 0x7f6f40195d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:12.886 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:12.880+0000 7f6f3ffff700 1 -- 172.21.15.38:0/2301461332 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6f401032d0 msgr2=0x7f6f40196280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:12.886 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:12.880+0000 7f6f3ffff700 1 --2- 172.21.15.38:0/2301461332 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6f401032d0 0x7f6f40196280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:12.886 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:12.880+0000 7f6f3ffff700 1 -- 172.21.15.38:0/2301461332 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f38009d00 con 0x7f6f40105bf0 2023-03-09T14:44:12.886 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:12.880+0000 7f6f3ffff700 1 --2- 172.21.15.38:0/2301461332 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f40105bf0 0x7f6f4019d2b0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6f3000d980 tx=0x7f6f3000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:12.887 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:12.880+0000 7f6f3cff9700 1 -- 172.21.15.38:0/2301461332 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f30004a30 con 0x7f6f40105bf0 2023-03-09T14:44:12.887 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:12.880+0000 7f6f3cff9700 1 -- 172.21.15.38:0/2301461332 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6f300054e0 con 0x7f6f40105bf0 2023-03-09T14:44:12.887 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:12.880+0000 7f6f45ab0700 1 -- 172.21.15.38:0/2301461332 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f4010b810 con 0x7f6f40105bf0 2023-03-09T14:44:12.887 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:12.880+0000 7f6f3cff9700 1 -- 172.21.15.38:0/2301461332 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f3000fbc0 con 0x7f6f40105bf0 2023-03-09T14:44:12.887 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:12.880+0000 7f6f45ab0700 1 -- 172.21.15.38:0/2301461332 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f4010b680 con 0x7f6f40105bf0 2023-03-09T14:44:12.887 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:12.881+0000 7f6f45ab0700 1 -- 172.21.15.38:0/2301461332 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f400fbf20 con 0x7f6f40105bf0 2023-03-09T14:44:12.887 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:12.882+0000 7f6f3cff9700 1 -- 172.21.15.38:0/2301461332 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 133198+0+0 (secure 0 0 0) 0x7f6f30022070 con 0x7f6f40105bf0 2023-03-09T14:44:12.888 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:12.883+0000 7f6f3cff9700 1 --2- 172.21.15.38:0/2301461332 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f6f2809f3c0 0x7f6f280a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:12.888 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:12.883+0000 7f6f3f7fe700 1 --2- 172.21.15.38:0/2301461332 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f6f2809f3c0 0x7f6f280a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:12.888 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:12.883+0000 7f6f3cff9700 1 -- 172.21.15.38:0/2301461332 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f6f300c9930 con 0x7f6f40105bf0 2023-03-09T14:44:12.889 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:12.883+0000 7f6f3f7fe700 1 --2- 172.21.15.38:0/2301461332 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f6f2809f3c0 0x7f6f280a1870 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f6f38007ab0 tx=0x7f6f3800b7b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:12.889 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:12.886+0000 7f6f3cff9700 1 -- 172.21.15.38:0/2301461332 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f6f300941e0 con 0x7f6f40105bf0 2023-03-09T14:44:13.130 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:13.127+0000 7f6f45ab0700 1 -- 172.21.15.38:0/2301461332 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6f4002cef0 con 0x7f6f40105bf0 2023-03-09T14:44:13.130 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:13.128+0000 7f6f3cff9700 1 -- 172.21.15.38:0/2301461332 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+8837 (secure 0 0 0) 0x7f6f30017090 con 0x7f6f40105bf0 2023-03-09T14:44:13.131 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:44:13.131 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/768918602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":768918602},{"type":"v1","addr":"172.21.15.38:6835","nonce":768918602}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/18381540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":18381540},{"type":"v1","addr":"172.21.15.38:6837","nonce":18381540}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14610,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6839/4139020066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":4139020066},{"type":"v1","addr":"172.21.15.38:6839","nonce":4139020066}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14622,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6841/1397115322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":1397115322},{"type":"v1","addr":"172.21.15.38:6841","nonce":1397115322}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6833/415305571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":415305571},{"type":"v1","addr":"172.21.15.181:6833","nonce":415305571}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/659300520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":659300520},{"type":"v1","addr":"172.21.15.122:6833","nonce":659300520}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24448,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/2576255861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":2576255861},{"type":"v1","addr":"172.21.15.181:6835","nonce":2576255861}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24460,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2942560181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":2942560181},{"type":"v1","addr":"172.21.15.181:6837","nonce":2942560181}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/3675537941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":3675537941},{"type":"v1","addr":"172.21.15.122:6835","nonce":3675537941}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24478,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6839/3902263660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":3902263660},{"type":"v1","addr":"172.21.15.181:6839","nonce":3902263660}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24494,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/2210214596","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":2210214596},{"type":"v1","addr":"172.21.15.122:6837","nonce":2210214596}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9}],"filesystems":[]} 2023-03-09T14:44:13.132 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:13.130+0000 7f6f45ab0700 1 -- 172.21.15.38:0/2301461332 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f6f2809f3c0 msgr2=0x7f6f280a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:13.133 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:13.130+0000 7f6f45ab0700 1 --2- 172.21.15.38:0/2301461332 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f6f2809f3c0 0x7f6f280a1870 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f6f38007ab0 tx=0x7f6f3800b7b0 comp rx=0 tx=0).stop 2023-03-09T14:44:13.133 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:13.130+0000 7f6f45ab0700 1 -- 172.21.15.38:0/2301461332 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f40105bf0 msgr2=0x7f6f4019d2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:13.133 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:13.130+0000 7f6f45ab0700 1 --2- 172.21.15.38:0/2301461332 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f40105bf0 0x7f6f4019d2b0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6f3000d980 tx=0x7f6f3000dc90 comp rx=0 tx=0).stop 2023-03-09T14:44:13.133 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:13.130+0000 7f6f45ab0700 1 -- 172.21.15.38:0/2301461332 shutdown_connections 2023-03-09T14:44:13.133 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:13.130+0000 7f6f45ab0700 1 --2- 172.21.15.38:0/2301461332 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6f40105bf0 0x7f6f4019d2b0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:13.133 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:13.130+0000 7f6f45ab0700 1 --2- 172.21.15.38:0/2301461332 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f6f2809f3c0 0x7f6f280a1870 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:13.134 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:13.130+0000 7f6f45ab0700 1 --2- 172.21.15.38:0/2301461332 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6f401009b0 0x7f6f40195d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:13.134 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:13.130+0000 7f6f45ab0700 1 --2- 172.21.15.38:0/2301461332 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6f401032d0 0x7f6f40196280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:13.134 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:13.130+0000 7f6f45ab0700 1 -- 172.21.15.38:0/2301461332 >> 172.21.15.38:0/2301461332 conn(0x7f6f400fa3b0 msgr2=0x7f6f40101610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:13.134 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:13.130+0000 7f6f45ab0700 1 -- 172.21.15.38:0/2301461332 shutdown_connections 2023-03-09T14:44:13.134 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:13.130+0000 7f6f45ab0700 1 -- 172.21.15.38:0/2301461332 wait complete. 2023-03-09T14:44:13.135 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 12 2023-03-09T14:44:13.228 INFO:journalctl@ceph.mds.b.smithi122.stdout:Mar 09 14:44:13 smithi122 systemd[1]: Starting Ceph mds.b for c6733fe2-be87-11ed-9af2-001a4aab830c... 2023-03-09T14:44:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:13 smithi038 ceph-mon[92422]: Deploying daemon mds.b on smithi122 2023-03-09T14:44:13.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:13 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/2301461332' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-03-09T14:44:13.579 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:13 smithi122 ceph-mon[93869]: Deploying daemon mds.b on smithi122 2023-03-09T14:44:13.579 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:13 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/2301461332' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-03-09T14:44:13.580 INFO:journalctl@ceph.mds.b.smithi122.stdout:Mar 09 14:44:13 smithi122 podman[119129]: 2023-03-09T14:44:13.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:13 smithi181 ceph-mon[94189]: Deploying daemon mds.b on smithi122 2023-03-09T14:44:13.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:13 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/2301461332' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-03-09T14:44:13.978 INFO:journalctl@ceph.mds.b.smithi122.stdout:Mar 09 14:44:13 smithi122 ceph-mds[119169]: -- 172.21.15.122:0/1640519481 <== mon.2 v2:172.21.15.122:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5595e3c30b60 con 0x5595e3c04c00 2023-03-09T14:44:13.978 INFO:journalctl@ceph.mds.b.smithi122.stdout:Mar 09 14:44:13 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mds-b[119145]: starting mds.b at 2023-03-09T14:44:13.978 INFO:journalctl@ceph.mds.b.smithi122.stdout:Mar 09 14:44:13 smithi122 bash[119129]: f1052366a81a2364302c3e74a518252605b8278dbfea175905656786436576b4 2023-03-09T14:44:13.978 INFO:journalctl@ceph.mds.b.smithi122.stdout:Mar 09 14:44:13 smithi122 systemd[1]: Started Ceph mds.b for c6733fe2-be87-11ed-9af2-001a4aab830c. 2023-03-09T14:44:14.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:14 smithi122 ceph-mon[93869]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-03-09T14:44:14.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:14 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:14.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:14 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:14.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:14 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:44:14.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:14 smithi181 ceph-mon[94189]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-03-09T14:44:14.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:14 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:14.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:14 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:14.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:14 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:44:14.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:14 smithi038 ceph-mon[92422]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-03-09T14:44:14.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:14 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:14.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:14 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:14.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:14 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:44:15.146 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-09T14:44:15.282 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.279+0000 7ff12bc67700 1 -- 172.21.15.38:0/1770920516 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff124100090 msgr2=0x7ff124106950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:15.282 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.279+0000 7ff12bc67700 1 --2- 172.21.15.38:0/1770920516 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff124100090 0x7ff124106950 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff11c00c210 tx=0x7ff11c00c520 comp rx=0 tx=0).stop 2023-03-09T14:44:15.282 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.280+0000 7ff12bc67700 1 -- 172.21.15.38:0/1770920516 shutdown_connections 2023-03-09T14:44:15.283 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.280+0000 7ff12bc67700 1 --2- 172.21.15.38:0/1770920516 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff124100090 0x7ff124106950 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:15.283 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.280+0000 7ff12bc67700 1 --2- 172.21.15.38:0/1770920516 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff1240fe510 0x7ff1240fe920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:15.283 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.280+0000 7ff12bc67700 1 --2- 172.21.15.38:0/1770920516 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff1240ff700 0x7ff1240ffb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:15.283 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.280+0000 7ff12bc67700 1 -- 172.21.15.38:0/1770920516 >> 172.21.15.38:0/1770920516 conn(0x7ff1240f9b10 msgr2=0x7ff1240fbf60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:15.283 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.280+0000 7ff12bc67700 1 -- 172.21.15.38:0/1770920516 shutdown_connections 2023-03-09T14:44:15.283 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.280+0000 7ff12bc67700 1 -- 172.21.15.38:0/1770920516 wait complete. 2023-03-09T14:44:15.285 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.280+0000 7ff12bc67700 1 Processor -- start 2023-03-09T14:44:15.285 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.280+0000 7ff12bc67700 1 -- start start 2023-03-09T14:44:15.285 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.281+0000 7ff12bc67700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff1240fe510 0x7ff124193b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:15.285 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.281+0000 7ff12bc67700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff1240ff700 0x7ff124194080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:15.286 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.281+0000 7ff12bc67700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff124100090 0x7ff12419b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:15.286 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.281+0000 7ff12bc67700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff12419b6c0 con 0x7ff1240fe510 2023-03-09T14:44:15.286 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.281+0000 7ff12bc67700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff12419b800 con 0x7ff124100090 2023-03-09T14:44:15.286 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.281+0000 7ff12bc67700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff12419b990 con 0x7ff1240ff700 2023-03-09T14:44:15.286 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.281+0000 7ff129a03700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff1240fe510 0x7ff124193b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:15.286 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.281+0000 7ff129a03700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff1240fe510 0x7ff124193b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36574/0 (socket says 172.21.15.38:36574) 2023-03-09T14:44:15.286 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.281+0000 7ff129a03700 1 -- 172.21.15.38:0/1064078117 learned_addr learned my addr 172.21.15.38:0/1064078117 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:44:15.286 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.281+0000 7ff12a204700 1 --2- 172.21.15.38:0/1064078117 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff124100090 0x7ff12419b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:15.287 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.281+0000 7ff129202700 1 --2- 172.21.15.38:0/1064078117 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff1240ff700 0x7ff124194080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:15.287 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.281+0000 7ff129a03700 1 -- 172.21.15.38:0/1064078117 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff1240ff700 msgr2=0x7ff124194080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:15.287 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.281+0000 7ff129a03700 1 --2- 172.21.15.38:0/1064078117 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff1240ff700 0x7ff124194080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:15.287 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.281+0000 7ff129a03700 1 -- 172.21.15.38:0/1064078117 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff124100090 msgr2=0x7ff12419b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:15.287 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.281+0000 7ff129a03700 1 --2- 172.21.15.38:0/1064078117 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff124100090 0x7ff12419b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:15.287 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.281+0000 7ff129a03700 1 -- 172.21.15.38:0/1064078117 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff11c00ae10 con 0x7ff1240fe510 2023-03-09T14:44:15.287 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.281+0000 7ff129a03700 1 --2- 172.21.15.38:0/1064078117 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff1240fe510 0x7ff124193b40 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ff12000e970 tx=0x7ff12000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:15.287 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.281+0000 7ff116ffd700 1 -- 172.21.15.38:0/1064078117 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff120009790 con 0x7ff1240fe510 2023-03-09T14:44:15.288 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.281+0000 7ff116ffd700 1 -- 172.21.15.38:0/1064078117 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff12000c7e0 con 0x7ff1240fe510 2023-03-09T14:44:15.288 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.281+0000 7ff12bc67700 1 -- 172.21.15.38:0/1064078117 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1241096c0 con 0x7ff1240fe510 2023-03-09T14:44:15.288 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.281+0000 7ff116ffd700 1 -- 172.21.15.38:0/1064078117 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff120003c70 con 0x7ff1240fe510 2023-03-09T14:44:15.288 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.281+0000 7ff12bc67700 1 -- 172.21.15.38:0/1064078117 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff124109530 con 0x7ff1240fe510 2023-03-09T14:44:15.288 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.282+0000 7ff12bc67700 1 -- 172.21.15.38:0/1064078117 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1241053c0 con 0x7ff1240fe510 2023-03-09T14:44:15.289 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.283+0000 7ff116ffd700 1 -- 172.21.15.38:0/1064078117 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 133198+0+0 (secure 0 0 0) 0x7ff12001e020 con 0x7ff1240fe510 2023-03-09T14:44:15.289 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.284+0000 7ff116ffd700 1 --2- 172.21.15.38:0/1064078117 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7ff11009f3c0 0x7ff1100a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:15.289 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.284+0000 7ff129202700 1 --2- 172.21.15.38:0/1064078117 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7ff11009f3c0 0x7ff1100a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:15.290 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.284+0000 7ff116ffd700 1 -- 172.21.15.38:0/1064078117 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ff120014070 con 0x7ff1240fe510 2023-03-09T14:44:15.290 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.284+0000 7ff129202700 1 --2- 172.21.15.38:0/1064078117 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7ff11009f3c0 0x7ff1100a1870 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ff118005950 tx=0x7ff11800b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:15.290 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.288+0000 7ff116ffd700 1 -- 172.21.15.38:0/1064078117 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7ff120094380 con 0x7ff1240fe510 2023-03-09T14:44:15.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:15 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:boot 2023-03-09T14:44:15.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:15 smithi038 ceph-mon[92422]: fsmap 12 up:standby 2023-03-09T14:44:15.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:15 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-03-09T14:44:15.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:15 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:15.555 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.552+0000 7ff12bc67700 1 -- 172.21.15.38:0/1064078117 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff12402cef0 con 0x7ff1240fe510 2023-03-09T14:44:15.555 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.553+0000 7ff116ffd700 1 -- 172.21.15.38:0/1064078117 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+9595 (secure 0 0 0) 0x7ff1200979a0 con 0x7ff1240fe510 2023-03-09T14:44:15.556 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:44:15.556 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/768918602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":768918602},{"type":"v1","addr":"172.21.15.38:6835","nonce":768918602}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/18381540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":18381540},{"type":"v1","addr":"172.21.15.38:6837","nonce":18381540}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14610,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6839/4139020066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":4139020066},{"type":"v1","addr":"172.21.15.38:6839","nonce":4139020066}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14622,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6841/1397115322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":1397115322},{"type":"v1","addr":"172.21.15.38:6841","nonce":1397115322}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6833/415305571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":415305571},{"type":"v1","addr":"172.21.15.181:6833","nonce":415305571}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/659300520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":659300520},{"type":"v1","addr":"172.21.15.122:6833","nonce":659300520}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24448,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/2576255861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":2576255861},{"type":"v1","addr":"172.21.15.181:6835","nonce":2576255861}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24460,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2942560181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":2942560181},{"type":"v1","addr":"172.21.15.181:6837","nonce":2942560181}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/3675537941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":3675537941},{"type":"v1","addr":"172.21.15.122:6835","nonce":3675537941}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24478,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6839/3902263660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":3902263660},{"type":"v1","addr":"172.21.15.181:6839","nonce":3902263660}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24494,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/2210214596","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":2210214596},{"type":"v1","addr":"172.21.15.122:6837","nonce":2210214596}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24530,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6839/971050165","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6838","nonce":971050165},{"type":"v1","addr":"172.21.15.122:6839","nonce":971050165}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13}],"filesystems":[]} 2023-03-09T14:44:15.558 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.555+0000 7ff12bc67700 1 -- 172.21.15.38:0/1064078117 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7ff11009f3c0 msgr2=0x7ff1100a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:15.558 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.555+0000 7ff12bc67700 1 --2- 172.21.15.38:0/1064078117 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7ff11009f3c0 0x7ff1100a1870 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ff118005950 tx=0x7ff11800b410 comp rx=0 tx=0).stop 2023-03-09T14:44:15.558 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.555+0000 7ff12bc67700 1 -- 172.21.15.38:0/1064078117 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff1240fe510 msgr2=0x7ff124193b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:15.558 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.555+0000 7ff12bc67700 1 --2- 172.21.15.38:0/1064078117 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff1240fe510 0x7ff124193b40 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ff12000e970 tx=0x7ff12000ec80 comp rx=0 tx=0).stop 2023-03-09T14:44:15.558 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.556+0000 7ff12bc67700 1 -- 172.21.15.38:0/1064078117 shutdown_connections 2023-03-09T14:44:15.558 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.556+0000 7ff12bc67700 1 --2- 172.21.15.38:0/1064078117 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff1240fe510 0x7ff124193b40 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:15.559 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.556+0000 7ff12bc67700 1 --2- 172.21.15.38:0/1064078117 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7ff11009f3c0 0x7ff1100a1870 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:15.559 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.556+0000 7ff12bc67700 1 --2- 172.21.15.38:0/1064078117 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff124100090 0x7ff12419b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:15.559 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.556+0000 7ff12bc67700 1 --2- 172.21.15.38:0/1064078117 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff1240ff700 0x7ff124194080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:15.559 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.556+0000 7ff12bc67700 1 -- 172.21.15.38:0/1064078117 >> 172.21.15.38:0/1064078117 conn(0x7ff1240f9b10 msgr2=0x7ff1240fbd30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:15.559 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.556+0000 7ff12bc67700 1 -- 172.21.15.38:0/1064078117 shutdown_connections 2023-03-09T14:44:15.559 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.556+0000 7ff12bc67700 1 -- 172.21.15.38:0/1064078117 wait complete. 2023-03-09T14:44:15.560 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 13 2023-03-09T14:44:15.572 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-09T14:44:15.700 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.697+0000 7fb91059c700 1 -- 172.21.15.38:0/38851881 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb9080fffc0 msgr2=0x7fb908100410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:15.700 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.697+0000 7fb91059c700 1 --2- 172.21.15.38:0/38851881 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb9080fffc0 0x7fb908100410 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fb8f8009990 tx=0x7fb8f8009ca0 comp rx=0 tx=0).stop 2023-03-09T14:44:15.700 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.697+0000 7fb91059c700 1 -- 172.21.15.38:0/38851881 shutdown_connections 2023-03-09T14:44:15.700 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.697+0000 7fb91059c700 1 --2- 172.21.15.38:0/38851881 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb908100950 0x7fb908107210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:15.701 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.697+0000 7fb91059c700 1 --2- 172.21.15.38:0/38851881 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb9080fedd0 0x7fb9080ff1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:15.701 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.697+0000 7fb91059c700 1 --2- 172.21.15.38:0/38851881 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb9080fffc0 0x7fb908100410 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:15.701 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.697+0000 7fb91059c700 1 -- 172.21.15.38:0/38851881 >> 172.21.15.38:0/38851881 conn(0x7fb9080fa3b0 msgr2=0x7fb9080fc820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:15.701 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.697+0000 7fb91059c700 1 -- 172.21.15.38:0/38851881 shutdown_connections 2023-03-09T14:44:15.701 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.697+0000 7fb91059c700 1 -- 172.21.15.38:0/38851881 wait complete. 2023-03-09T14:44:15.701 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.698+0000 7fb91059c700 1 Processor -- start 2023-03-09T14:44:15.701 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.698+0000 7fb91059c700 1 -- start start 2023-03-09T14:44:15.703 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.698+0000 7fb91059c700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb9080fedd0 0x7fb908195bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:15.703 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.698+0000 7fb91059c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb9080fffc0 0x7fb9081960f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:15.703 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.698+0000 7fb91059c700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb908100950 0x7fb90819d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:15.703 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.698+0000 7fb91059c700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb90819d730 con 0x7fb9080fffc0 2023-03-09T14:44:15.704 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.698+0000 7fb91059c700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb90819d870 con 0x7fb908100950 2023-03-09T14:44:15.704 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.698+0000 7fb91059c700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb90819da00 con 0x7fb9080fedd0 2023-03-09T14:44:15.704 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.698+0000 7fb90db37700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb9080fffc0 0x7fb9081960f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:15.704 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.698+0000 7fb90db37700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb9080fffc0 0x7fb9081960f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36588/0 (socket says 172.21.15.38:36588) 2023-03-09T14:44:15.705 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.698+0000 7fb90e338700 1 --2- 172.21.15.38:0/1035975285 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb9080fedd0 0x7fb908195bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:15.705 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.698+0000 7fb90db37700 1 -- 172.21.15.38:0/1035975285 learned_addr learned my addr 172.21.15.38:0/1035975285 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:44:15.705 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.698+0000 7fb90eb39700 1 --2- 172.21.15.38:0/1035975285 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb908100950 0x7fb90819d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:15.705 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.699+0000 7fb90db37700 1 -- 172.21.15.38:0/1035975285 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb9080fedd0 msgr2=0x7fb908195bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:15.705 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.699+0000 7fb90db37700 1 --2- 172.21.15.38:0/1035975285 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb9080fedd0 0x7fb908195bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:15.705 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.699+0000 7fb90db37700 1 -- 172.21.15.38:0/1035975285 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb908100950 msgr2=0x7fb90819d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:15.705 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.699+0000 7fb90db37700 1 --2- 172.21.15.38:0/1035975285 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb908100950 0x7fb90819d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:15.705 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.699+0000 7fb90db37700 1 -- 172.21.15.38:0/1035975285 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8f80095f0 con 0x7fb9080fffc0 2023-03-09T14:44:15.706 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.699+0000 7fb90e338700 1 --2- 172.21.15.38:0/1035975285 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb9080fedd0 0x7fb908195bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:44:15.706 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.699+0000 7fb90db37700 1 --2- 172.21.15.38:0/1035975285 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb9080fffc0 0x7fb9081960f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fb8f8000c00 tx=0x7fb8f80047a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:15.706 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.699+0000 7fb8ff7fe700 1 -- 172.21.15.38:0/1035975285 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8f801d070 con 0x7fb9080fffc0 2023-03-09T14:44:15.706 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.699+0000 7fb8ff7fe700 1 -- 172.21.15.38:0/1035975285 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb8f8022470 con 0x7fb9080fffc0 2023-03-09T14:44:15.706 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.699+0000 7fb91059c700 1 -- 172.21.15.38:0/1035975285 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb90810b840 con 0x7fb9080fffc0 2023-03-09T14:44:15.706 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.699+0000 7fb8ff7fe700 1 -- 172.21.15.38:0/1035975285 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8f8017650 con 0x7fb9080fffc0 2023-03-09T14:44:15.706 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.699+0000 7fb91059c700 1 -- 172.21.15.38:0/1035975285 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb90810b6b0 con 0x7fb9080fffc0 2023-03-09T14:44:15.707 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.700+0000 7fb91059c700 1 -- 172.21.15.38:0/1035975285 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9081062d0 con 0x7fb9080fffc0 2023-03-09T14:44:15.707 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.701+0000 7fb8ff7fe700 1 -- 172.21.15.38:0/1035975285 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 133198+0+0 (secure 0 0 0) 0x7fb8f8029030 con 0x7fb9080fffc0 2023-03-09T14:44:15.707 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.702+0000 7fb8ff7fe700 1 --2- 172.21.15.38:0/1035975285 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fb8f409f3c0 0x7fb8f40a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:15.707 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.702+0000 7fb90e338700 1 --2- 172.21.15.38:0/1035975285 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fb8f409f3c0 0x7fb8f40a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:15.708 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.702+0000 7fb8ff7fe700 1 -- 172.21.15.38:0/1035975285 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fb8f80cad40 con 0x7fb9080fffc0 2023-03-09T14:44:15.708 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.703+0000 7fb90e338700 1 --2- 172.21.15.38:0/1035975285 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fb8f409f3c0 0x7fb8f40a1870 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fb9080ffe20 tx=0x7fb904009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:15.709 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.707+0000 7fb8ff7fe700 1 -- 172.21.15.38:0/1035975285 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7fb8f8095680 con 0x7fb9080fffc0 2023-03-09T14:44:15.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:15 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:boot 2023-03-09T14:44:15.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:15 smithi122 ceph-mon[93869]: fsmap 12 up:standby 2023-03-09T14:44:15.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:15 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-03-09T14:44:15.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:15 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:15.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:15 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:boot 2023-03-09T14:44:15.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:15 smithi181 ceph-mon[94189]: fsmap 12 up:standby 2023-03-09T14:44:15.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:15 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-03-09T14:44:15.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:15 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:15.925 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.923+0000 7fb91059c700 1 -- 172.21.15.38:0/1035975285 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb90802cef0 con 0x7fb9080fffc0 2023-03-09T14:44:15.926 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.924+0000 7fb8ff7fe700 1 -- 172.21.15.38:0/1035975285 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) v1 ==== 74+0+18435 (secure 0 0 0) 0x7fb8f8098ca0 con 0x7fb9080fffc0 2023-03-09T14:44:15.926 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:44:15.927 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":79,"fsid":"c6733fe2-be87-11ed-9af2-001a4aab830c","created":"2023-03-09T14:37:14.947981+0000","modified":"2023-03-09T14:43:38.269957+0000","last_up_change":"2023-03-09T14:43:37.198407+0000","last_in_change":"2023-03-09T14:43:19.550566+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-09T14:40:23.196665+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"51477039-15ee-4707-a34d-4ecfa18bebde","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6803","nonce":1508072290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6805","nonce":1508072290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6809","nonce":1508072290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6807","nonce":1508072290}]},"public_addr":"172.21.15.38:6803/1508072290","cluster_addr":"172.21.15.38:6805/1508072290","heartbeat_back_addr":"172.21.15.38:6809/1508072290","heartbeat_front_addr":"172.21.15.38:6807/1508072290","state":["exists","up"]},{"osd":1,"uuid":"71f8a718-b0a9-4f39-bc54-267aa42a2578","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6801","nonce":3929268773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6803","nonce":3929268773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6807","nonce":3929268773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6805","nonce":3929268773}]},"public_addr":"172.21.15.122:6801/3929268773","cluster_addr":"172.21.15.122:6803/3929268773","heartbeat_back_addr":"172.21.15.122:6807/3929268773","heartbeat_front_addr":"172.21.15.122:6805/3929268773","state":["exists","up"]},{"osd":2,"uuid":"961a2cbe-140e-437f-b131-1fcac573b19c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6801","nonce":1310630803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6803","nonce":1310630803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6807","nonce":1310630803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6805","nonce":1310630803}]},"public_addr":"172.21.15.181:6801/1310630803","cluster_addr":"172.21.15.181:6803/1310630803","heartbeat_back_addr":"172.21.15.181:6807/1310630803","heartbeat_front_addr":"172.21.15.181:6805/1310630803","state":["exists","up"]},{"osd":3,"uuid":"969df8fa-2d19-42c7-9f2a-5c5fac15d4a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6811","nonce":2719814100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6813","nonce":2719814100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6817","nonce":2719814100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6815","nonce":2719814100}]},"public_addr":"172.21.15.38:6811/2719814100","cluster_addr":"172.21.15.38:6813/2719814100","heartbeat_back_addr":"172.21.15.38:6817/2719814100","heartbeat_front_addr":"172.21.15.38:6815/2719814100","state":["exists","up"]},{"osd":4,"uuid":"44a6dce3-e06e-4cbd-9404-62a86d01ab6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6809","nonce":2480764994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6811","nonce":2480764994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6815","nonce":2480764994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6813","nonce":2480764994}]},"public_addr":"172.21.15.122:6809/2480764994","cluster_addr":"172.21.15.122:6811/2480764994","heartbeat_back_addr":"172.21.15.122:6815/2480764994","heartbeat_front_addr":"172.21.15.122:6813/2480764994","state":["exists","up"]},{"osd":5,"uuid":"49749574-c00c-4e3a-80e6-e3686df58d5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6809","nonce":3141346246}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6811","nonce":3141346246}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6815","nonce":3141346246}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6813","nonce":3141346246}]},"public_addr":"172.21.15.181:6809/3141346246","cluster_addr":"172.21.15.181:6811/3141346246","heartbeat_back_addr":"172.21.15.181:6815/3141346246","heartbeat_front_addr":"172.21.15.181:6813/3141346246","state":["exists","up"]},{"osd":6,"uuid":"61c0437e-3f51-45a9-8a61-e98eea0093f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6819","nonce":2122788957}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6821","nonce":2122788957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6825","nonce":2122788957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6823","nonce":2122788957}]},"public_addr":"172.21.15.38:6819/2122788957","cluster_addr":"172.21.15.38:6821/2122788957","heartbeat_back_addr":"172.21.15.38:6825/2122788957","heartbeat_front_addr":"172.21.15.38:6823/2122788957","state":["exists","up"]},{"osd":7,"uuid":"231fecdd-939f-4c7b-83a1-c4a1cffd7139","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":73,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6817","nonce":656874139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6819","nonce":656874139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6823","nonce":656874139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6821","nonce":656874139}]},"public_addr":"172.21.15.122:6817/656874139","cluster_addr":"172.21.15.122:6819/656874139","heartbeat_back_addr":"172.21.15.122:6823/656874139","heartbeat_front_addr":"172.21.15.122:6821/656874139","state":["exists","up"]},{"osd":8,"uuid":"297e6d6d-85fd-43ce-9028-5e9dbe55c36e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6817","nonce":755000703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6819","nonce":755000703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6823","nonce":755000703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6821","nonce":755000703}]},"public_addr":"172.21.15.181:6817/755000703","cluster_addr":"172.21.15.181:6819/755000703","heartbeat_back_addr":"172.21.15.181:6823/755000703","heartbeat_front_addr":"172.21.15.181:6821/755000703","state":["exists","up"]},{"osd":9,"uuid":"94cc0355-5a1a-4cfc-bd0c-562b6b41127e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6827","nonce":2624555969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6829","nonce":2624555969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6833","nonce":2624555969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6831","nonce":2624555969}]},"public_addr":"172.21.15.38:6827/2624555969","cluster_addr":"172.21.15.38:6829/2624555969","heartbeat_back_addr":"172.21.15.38:6833/2624555969","heartbeat_front_addr":"172.21.15.38:6831/2624555969","state":["exists","up"]},{"osd":10,"uuid":"f4f2e042-8d92-46bf-93b8-8b0f04795c2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6825","nonce":1427785280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6827","nonce":1427785280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6831","nonce":1427785280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6829","nonce":1427785280}]},"public_addr":"172.21.15.122:6825/1427785280","cluster_addr":"172.21.15.122:6827/1427785280","heartbeat_back_addr":"172.21.15.122:6831/1427785280","heartbeat_front_addr":"172.21.15.122:6829/1427785280","state":["exists","up"]},{"osd":11,"uuid":"07444936-eae5-4097-a799-60294cafe80b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6825","nonce":3183488640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6827","nonce":3183488640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6831","nonce":3183488640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6829","nonce":3183488640}]},"public_addr":"172.21.15.181:6825/3183488640","cluster_addr":"172.21.15.181:6827/3183488640","heartbeat_back_addr":"172.21.15.181:6831/3183488640","heartbeat_front_addr":"172.21.15.181:6829/3183488640","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:39:37.657556+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:39:58.960074+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:40:20.326952+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:40:42.244387+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:41:03.946973+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:41:25.097985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:41:46.877916+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:42:08.413827+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:42:29.822009+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:42:51.776684+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:43:13.937249+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:43:35.569199+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/4272288735":"2023-03-10T14:38:14.612767+0000","172.21.15.38:6801/3381625669":"2023-03-10T14:38:14.612767+0000","172.21.15.38:6800/3381625669":"2023-03-10T14:38:14.612767+0000","172.21.15.38:6800/3060474342":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/4253918615":"2023-03-10T14:38:14.612767+0000","172.21.15.38:0/3496083181":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/245602368":"2023-03-10T14:37:47.517775+0000","172.21.15.38:6801/3060474342":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/3622589182":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/1430302251":"2023-03-10T14:38:14.612767+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-09T14:44:15.928 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.926+0000 7fb91059c700 1 -- 172.21.15.38:0/1035975285 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fb8f409f3c0 msgr2=0x7fb8f40a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:15.928 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.926+0000 7fb91059c700 1 --2- 172.21.15.38:0/1035975285 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fb8f409f3c0 0x7fb8f40a1870 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fb9080ffe20 tx=0x7fb904009260 comp rx=0 tx=0).stop 2023-03-09T14:44:15.928 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.926+0000 7fb91059c700 1 -- 172.21.15.38:0/1035975285 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb9080fffc0 msgr2=0x7fb9081960f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:15.929 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.926+0000 7fb91059c700 1 --2- 172.21.15.38:0/1035975285 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb9080fffc0 0x7fb9081960f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fb8f8000c00 tx=0x7fb8f80047a0 comp rx=0 tx=0).stop 2023-03-09T14:44:15.929 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.926+0000 7fb91059c700 1 -- 172.21.15.38:0/1035975285 shutdown_connections 2023-03-09T14:44:15.929 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.926+0000 7fb91059c700 1 --2- 172.21.15.38:0/1035975285 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fb8f409f3c0 0x7fb8f40a1870 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:15.929 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.926+0000 7fb91059c700 1 --2- 172.21.15.38:0/1035975285 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb908100950 0x7fb90819d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:15.929 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.926+0000 7fb91059c700 1 --2- 172.21.15.38:0/1035975285 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb9080fedd0 0x7fb908195bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:15.929 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.926+0000 7fb91059c700 1 --2- 172.21.15.38:0/1035975285 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb9080fffc0 0x7fb9081960f0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:15.930 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.926+0000 7fb91059c700 1 -- 172.21.15.38:0/1035975285 >> 172.21.15.38:0/1035975285 conn(0x7fb9080fa3b0 msgr2=0x7fb908103410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:15.930 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.926+0000 7fb91059c700 1 -- 172.21.15.38:0/1035975285 shutdown_connections 2023-03-09T14:44:15.930 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:15.926+0000 7fb91059c700 1 -- 172.21.15.38:0/1035975285 wait complete. 2023-03-09T14:44:15.941 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-09T14:40:23.196665+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-03-09T14:44:15.941 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-09T14:44:16.076 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.074+0000 7fe7c0d65700 1 -- 172.21.15.38:0/1152728779 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7bc0fff90 msgr2=0x7fe7bc1003e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:16.077 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.074+0000 7fe7c0d65700 1 --2- 172.21.15.38:0/1152728779 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7bc0fff90 0x7fe7bc1003e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe7a4009990 tx=0x7fe7a4009ca0 comp rx=0 tx=0).stop 2023-03-09T14:44:16.077 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.074+0000 7fe7c0d65700 1 -- 172.21.15.38:0/1152728779 shutdown_connections 2023-03-09T14:44:16.077 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.074+0000 7fe7c0d65700 1 --2- 172.21.15.38:0/1152728779 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe7bc100920 0x7fe7bc1071e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:16.077 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.074+0000 7fe7c0d65700 1 --2- 172.21.15.38:0/1152728779 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe7bc0feda0 0x7fe7bc0ff1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:16.077 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.074+0000 7fe7c0d65700 1 --2- 172.21.15.38:0/1152728779 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7bc0fff90 0x7fe7bc1003e0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:16.077 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.074+0000 7fe7c0d65700 1 -- 172.21.15.38:0/1152728779 >> 172.21.15.38:0/1152728779 conn(0x7fe7bc0fa3a0 msgr2=0x7fe7bc0fc7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:16.078 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.074+0000 7fe7c0d65700 1 -- 172.21.15.38:0/1152728779 shutdown_connections 2023-03-09T14:44:16.078 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.074+0000 7fe7c0d65700 1 -- 172.21.15.38:0/1152728779 wait complete. 2023-03-09T14:44:16.079 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.075+0000 7fe7c0d65700 1 Processor -- start 2023-03-09T14:44:16.080 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.075+0000 7fe7c0d65700 1 -- start start 2023-03-09T14:44:16.080 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.075+0000 7fe7c0d65700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe7bc0feda0 0x7fe7bc069780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:16.080 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.075+0000 7fe7c0d65700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe7bc0fff90 0x7fe7bc19a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:16.080 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.075+0000 7fe7c0d65700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7bc100920 0x7fe7bc19cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:16.080 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.075+0000 7fe7c0d65700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7bc19d760 con 0x7fe7bc100920 2023-03-09T14:44:16.080 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.075+0000 7fe7c0d65700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7bc19d8d0 con 0x7fe7bc0fff90 2023-03-09T14:44:16.080 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.075+0000 7fe7c0d65700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7bc19da90 con 0x7fe7bc0feda0 2023-03-09T14:44:16.081 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.075+0000 7fe7bad9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7bc100920 0x7fe7bc19cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:16.081 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.075+0000 7fe7bad9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7bc100920 0x7fe7bc19cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36606/0 (socket says 172.21.15.38:36606) 2023-03-09T14:44:16.081 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.075+0000 7fe7bad9d700 1 -- 172.21.15.38:0/2406327627 learned_addr learned my addr 172.21.15.38:0/2406327627 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:44:16.081 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.075+0000 7fe7b1d9b700 1 --2- 172.21.15.38:0/2406327627 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe7bc0fff90 0x7fe7bc19a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:16.081 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.075+0000 7fe7ba59c700 1 --2- 172.21.15.38:0/2406327627 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe7bc0feda0 0x7fe7bc069780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:16.081 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.076+0000 7fe7bad9d700 1 -- 172.21.15.38:0/2406327627 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe7bc0feda0 msgr2=0x7fe7bc069780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:16.081 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.076+0000 7fe7bad9d700 1 --2- 172.21.15.38:0/2406327627 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe7bc0feda0 0x7fe7bc069780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:16.082 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.076+0000 7fe7bad9d700 1 -- 172.21.15.38:0/2406327627 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe7bc0fff90 msgr2=0x7fe7bc19a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:16.082 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.076+0000 7fe7bad9d700 1 --2- 172.21.15.38:0/2406327627 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe7bc0fff90 0x7fe7bc19a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:16.082 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.076+0000 7fe7bad9d700 1 -- 172.21.15.38:0/2406327627 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7a40095f0 con 0x7fe7bc100920 2023-03-09T14:44:16.082 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.076+0000 7fe7bad9d700 1 --2- 172.21.15.38:0/2406327627 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7bc100920 0x7fe7bc19cf90 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fe7b400ef30 tx=0x7fe7b400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:16.082 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.076+0000 7fe7b3fff700 1 -- 172.21.15.38:0/2406327627 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b4018070 con 0x7fe7bc100920 2023-03-09T14:44:16.082 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.076+0000 7fe7c0d65700 1 -- 172.21.15.38:0/2406327627 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7bc10a700 con 0x7fe7bc100920 2023-03-09T14:44:16.082 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.076+0000 7fe7b3fff700 1 -- 172.21.15.38:0/2406327627 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe7b400f040 con 0x7fe7bc100920 2023-03-09T14:44:16.082 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.076+0000 7fe7b3fff700 1 -- 172.21.15.38:0/2406327627 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b400cd00 con 0x7fe7bc100920 2023-03-09T14:44:16.083 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.076+0000 7fe7c0d65700 1 -- 172.21.15.38:0/2406327627 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7bc10a570 con 0x7fe7bc100920 2023-03-09T14:44:16.083 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.077+0000 7fe7c0d65700 1 -- 172.21.15.38:0/2406327627 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7bc105d20 con 0x7fe7bc100920 2023-03-09T14:44:16.083 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.078+0000 7fe7b3fff700 1 -- 172.21.15.38:0/2406327627 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 133198+0+0 (secure 0 0 0) 0x7fe7b4003a00 con 0x7fe7bc100920 2023-03-09T14:44:16.084 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.079+0000 7fe7b3fff700 1 --2- 172.21.15.38:0/2406327627 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fe7a809f3c0 0x7fe7a80a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:16.084 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.079+0000 7fe7ba59c700 1 --2- 172.21.15.38:0/2406327627 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fe7a809f3c0 0x7fe7a80a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:16.084 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.079+0000 7fe7b3fff700 1 -- 172.21.15.38:0/2406327627 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fe7b40c95c0 con 0x7fe7bc100920 2023-03-09T14:44:16.084 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.079+0000 7fe7ba59c700 1 --2- 172.21.15.38:0/2406327627 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fe7a809f3c0 0x7fe7a80a1870 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe7bc0ffdf0 tx=0x7fe7ac00b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:16.085 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.083+0000 7fe7b3fff700 1 -- 172.21.15.38:0/2406327627 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7fe7b4093f00 con 0x7fe7bc100920 2023-03-09T14:44:16.288 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.285+0000 7fe7c0d65700 1 -- 172.21.15.38:0/2406327627 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe7bc0618d0 con 0x7fe7bc100920 2023-03-09T14:44:16.288 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.286+0000 7fe7b3fff700 1 -- 172.21.15.38:0/2406327627 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v79) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe7b4097520 con 0x7fe7bc100920 2023-03-09T14:44:16.289 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 1 2023-03-09T14:44:16.290 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.288+0000 7fe7c0d65700 1 -- 172.21.15.38:0/2406327627 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fe7a809f3c0 msgr2=0x7fe7a80a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:16.290 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.288+0000 7fe7c0d65700 1 --2- 172.21.15.38:0/2406327627 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fe7a809f3c0 0x7fe7a80a1870 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe7bc0ffdf0 tx=0x7fe7ac00b410 comp rx=0 tx=0).stop 2023-03-09T14:44:16.291 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.288+0000 7fe7c0d65700 1 -- 172.21.15.38:0/2406327627 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7bc100920 msgr2=0x7fe7bc19cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:16.291 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.288+0000 7fe7c0d65700 1 --2- 172.21.15.38:0/2406327627 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7bc100920 0x7fe7bc19cf90 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fe7b400ef30 tx=0x7fe7b400c3a0 comp rx=0 tx=0).stop 2023-03-09T14:44:16.291 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.288+0000 7fe7c0d65700 1 -- 172.21.15.38:0/2406327627 shutdown_connections 2023-03-09T14:44:16.291 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.288+0000 7fe7c0d65700 1 --2- 172.21.15.38:0/2406327627 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7bc100920 0x7fe7bc19cf90 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:16.291 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.288+0000 7fe7c0d65700 1 --2- 172.21.15.38:0/2406327627 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fe7a809f3c0 0x7fe7a80a1870 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:16.291 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.288+0000 7fe7c0d65700 1 --2- 172.21.15.38:0/2406327627 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe7bc0feda0 0x7fe7bc069780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:16.291 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.288+0000 7fe7c0d65700 1 --2- 172.21.15.38:0/2406327627 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe7bc0fff90 0x7fe7bc19a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:16.291 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.288+0000 7fe7c0d65700 1 -- 172.21.15.38:0/2406327627 >> 172.21.15.38:0/2406327627 conn(0x7fe7bc0fa3a0 msgr2=0x7fe7bc0fc680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:16.292 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.288+0000 7fe7c0d65700 1 -- 172.21.15.38:0/2406327627 shutdown_connections 2023-03-09T14:44:16.292 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.288+0000 7fe7c0d65700 1 -- 172.21.15.38:0/2406327627 wait complete. 2023-03-09T14:44:16.304 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-03-09T14:44:16.433 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.430+0000 7fae168d9700 1 -- 172.21.15.38:0/2203408497 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae10100410 msgr2=0x7fae10100820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:16.433 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.430+0000 7fae168d9700 1 --2- 172.21.15.38:0/2203408497 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae10100410 0x7fae10100820 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fae00009990 tx=0x7fae00009ca0 comp rx=0 tx=0).stop 2023-03-09T14:44:16.433 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.430+0000 7fae168d9700 1 -- 172.21.15.38:0/2203408497 shutdown_connections 2023-03-09T14:44:16.434 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.430+0000 7fae168d9700 1 --2- 172.21.15.38:0/2203408497 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae10100410 0x7fae10100820 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:16.434 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.430+0000 7fae168d9700 1 --2- 172.21.15.38:0/2203408497 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae100ff980 0x7fae10107230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:16.434 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.430+0000 7fae168d9700 1 --2- 172.21.15.38:0/2203408497 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae100fefd0 0x7fae100ff440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:16.434 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.430+0000 7fae168d9700 1 -- 172.21.15.38:0/2203408497 >> 172.21.15.38:0/2203408497 conn(0x7fae100f9b10 msgr2=0x7fae100fbf60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:16.434 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.430+0000 7fae168d9700 1 -- 172.21.15.38:0/2203408497 shutdown_connections 2023-03-09T14:44:16.434 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.430+0000 7fae168d9700 1 -- 172.21.15.38:0/2203408497 wait complete. 2023-03-09T14:44:16.434 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.431+0000 7fae168d9700 1 Processor -- start 2023-03-09T14:44:16.435 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.431+0000 7fae168d9700 1 -- start start 2023-03-09T14:44:16.436 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.431+0000 7fae168d9700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae100fefd0 0x7fae100fea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:16.436 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.431+0000 7fae168d9700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae100ff980 0x7fae1019cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:16.436 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.431+0000 7fae168d9700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae10100410 0x7fae1019f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:16.437 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.431+0000 7fae168d9700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae1019f960 con 0x7fae100ff980 2023-03-09T14:44:16.437 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.431+0000 7fae168d9700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae1019fad0 con 0x7fae100fefd0 2023-03-09T14:44:16.437 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.431+0000 7fae168d9700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae1019fc90 con 0x7fae10100410 2023-03-09T14:44:16.437 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.431+0000 7fae0f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae100ff980 0x7fae1019cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:16.437 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.431+0000 7fae0ffff700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae100fefd0 0x7fae100fea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:16.437 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.432+0000 7fae0f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae100ff980 0x7fae1019cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36620/0 (socket says 172.21.15.38:36620) 2023-03-09T14:44:16.437 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.432+0000 7fae0ffff700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae100fefd0 0x7fae100fea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.38:45806/0 (socket says 172.21.15.38:45806) 2023-03-09T14:44:16.438 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.432+0000 7fae0f7fe700 1 -- 172.21.15.38:0/231540167 learned_addr learned my addr 172.21.15.38:0/231540167 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:44:16.438 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.432+0000 7fae14e76700 1 --2- 172.21.15.38:0/231540167 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae10100410 0x7fae1019f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:16.438 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.432+0000 7fae0f7fe700 1 -- 172.21.15.38:0/231540167 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae10100410 msgr2=0x7fae1019f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:16.438 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.432+0000 7fae0f7fe700 1 --2- 172.21.15.38:0/231540167 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae10100410 0x7fae1019f280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:16.438 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.432+0000 7fae0f7fe700 1 -- 172.21.15.38:0/231540167 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae100fefd0 msgr2=0x7fae100fea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:16.438 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.432+0000 7fae0f7fe700 1 --2- 172.21.15.38:0/231540167 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae100fefd0 0x7fae100fea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:16.438 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.432+0000 7fae0f7fe700 1 -- 172.21.15.38:0/231540167 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae000095f0 con 0x7fae100ff980 2023-03-09T14:44:16.438 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.432+0000 7fae0f7fe700 1 --2- 172.21.15.38:0/231540167 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae100ff980 0x7fae1019cd30 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fae0400d980 tx=0x7fae0400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:16.439 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.432+0000 7fae0d7fa700 1 -- 172.21.15.38:0/231540167 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae04004220 con 0x7fae100ff980 2023-03-09T14:44:16.439 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.432+0000 7fae168d9700 1 -- 172.21.15.38:0/231540167 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae1010a6f0 con 0x7fae100ff980 2023-03-09T14:44:16.439 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.432+0000 7fae0d7fa700 1 -- 172.21.15.38:0/231540167 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fae04007640 con 0x7fae100ff980 2023-03-09T14:44:16.439 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.432+0000 7fae0d7fa700 1 -- 172.21.15.38:0/231540167 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae04005690 con 0x7fae100ff980 2023-03-09T14:44:16.439 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.432+0000 7fae168d9700 1 -- 172.21.15.38:0/231540167 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae1010a560 con 0x7fae100ff980 2023-03-09T14:44:16.439 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.433+0000 7fae168d9700 1 -- 172.21.15.38:0/231540167 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae10105d70 con 0x7fae100ff980 2023-03-09T14:44:16.439 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.434+0000 7fae0d7fa700 1 -- 172.21.15.38:0/231540167 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 133198+0+0 (secure 0 0 0) 0x7fae04021070 con 0x7fae100ff980 2023-03-09T14:44:16.440 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.435+0000 7fae0d7fa700 1 --2- 172.21.15.38:0/231540167 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fadf809f3c0 0x7fadf80a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:16.440 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.435+0000 7fae0ffff700 1 --2- 172.21.15.38:0/231540167 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fadf809f3c0 0x7fadf80a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:16.440 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.435+0000 7fae0d7fa700 1 -- 172.21.15.38:0/231540167 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fae040c96e0 con 0x7fae100ff980 2023-03-09T14:44:16.440 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.435+0000 7fae0ffff700 1 --2- 172.21.15.38:0/231540167 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fadf809f3c0 0x7fadf80a1870 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fae0000b5c0 tx=0x7fae00005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:16.442 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.440+0000 7fae0d7fa700 1 -- 172.21.15.38:0/231540167 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7fae04093ea0 con 0x7fae100ff980 2023-03-09T14:44:16.675 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.672+0000 7fae168d9700 1 -- 172.21.15.38:0/231540167 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fae10061810 con 0x7fae100ff980 2023-03-09T14:44:16.675 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.673+0000 7fae0d7fa700 1 -- 172.21.15.38:0/231540167 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v79) v1 ==== 84+0+65 (secure 0 0 0) 0x7fae04016020 con 0x7fae100ff980 2023-03-09T14:44:16.676 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:44:16.676 INFO:teuthology.orchestra.run.smithi038.stdout:[ 2023-03-09T14:44:16.676 INFO:teuthology.orchestra.run.smithi038.stdout: { 2023-03-09T14:44:16.676 INFO:teuthology.orchestra.run.smithi038.stdout: "poolnum": 1, 2023-03-09T14:44:16.676 INFO:teuthology.orchestra.run.smithi038.stdout: "poolname": ".mgr" 2023-03-09T14:44:16.676 INFO:teuthology.orchestra.run.smithi038.stdout: } 2023-03-09T14:44:16.676 INFO:teuthology.orchestra.run.smithi038.stdout:] 2023-03-09T14:44:16.677 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.675+0000 7fae168d9700 1 -- 172.21.15.38:0/231540167 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fadf809f3c0 msgr2=0x7fadf80a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:16.678 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.675+0000 7fae168d9700 1 --2- 172.21.15.38:0/231540167 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fadf809f3c0 0x7fadf80a1870 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fae0000b5c0 tx=0x7fae00005960 comp rx=0 tx=0).stop 2023-03-09T14:44:16.678 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.675+0000 7fae168d9700 1 -- 172.21.15.38:0/231540167 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae100ff980 msgr2=0x7fae1019cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:16.678 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.675+0000 7fae168d9700 1 --2- 172.21.15.38:0/231540167 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae100ff980 0x7fae1019cd30 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fae0400d980 tx=0x7fae0400dc90 comp rx=0 tx=0).stop 2023-03-09T14:44:16.678 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.675+0000 7fae168d9700 1 -- 172.21.15.38:0/231540167 shutdown_connections 2023-03-09T14:44:16.678 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.675+0000 7fae168d9700 1 --2- 172.21.15.38:0/231540167 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fadf809f3c0 0x7fadf80a1870 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:16.678 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.675+0000 7fae168d9700 1 --2- 172.21.15.38:0/231540167 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fae10100410 0x7fae1019f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:16.678 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.675+0000 7fae168d9700 1 --2- 172.21.15.38:0/231540167 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fae100fefd0 0x7fae100fea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:16.679 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.675+0000 7fae168d9700 1 --2- 172.21.15.38:0/231540167 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fae100ff980 0x7fae1019cd30 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:16.679 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.675+0000 7fae168d9700 1 -- 172.21.15.38:0/231540167 >> 172.21.15.38:0/231540167 conn(0x7fae100f9b10 msgr2=0x7fae100fbeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:16.679 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.675+0000 7fae168d9700 1 -- 172.21.15.38:0/231540167 shutdown_connections 2023-03-09T14:44:16.679 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.675+0000 7fae168d9700 1 -- 172.21.15.38:0/231540167 wait complete. 2023-03-09T14:44:16.689 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:16 smithi038 ceph-mon[92422]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-03-09T14:44:16.689 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:16 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/1064078117' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-03-09T14:44:16.690 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:16 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/1035975285' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-03-09T14:44:16.690 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:16 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/2406327627' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-03-09T14:44:16.691 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-03-09T14:44:16.691 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-03-09T14:44:16.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:16 smithi122 ceph-mon[93869]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-03-09T14:44:16.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:16 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/1064078117' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-03-09T14:44:16.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:16 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/1035975285' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-03-09T14:44:16.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:16 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/2406327627' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-03-09T14:44:16.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:16 smithi181 ceph-mon[94189]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-03-09T14:44:16.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:16 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/1064078117' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-03-09T14:44:16.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:16 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/1035975285' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-03-09T14:44:16.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:16 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/2406327627' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-03-09T14:44:16.826 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.823+0000 7f748a419700 1 -- 172.21.15.38:0/3585908967 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7484100950 msgr2=0x7f7484107210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:16.826 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.823+0000 7f748a419700 1 --2- 172.21.15.38:0/3585908967 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7484100950 0x7f7484107210 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f747c00b0a0 tx=0x7f747c00b3b0 comp rx=0 tx=0).stop 2023-03-09T14:44:16.826 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.823+0000 7f748a419700 1 -- 172.21.15.38:0/3585908967 shutdown_connections 2023-03-09T14:44:16.826 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.823+0000 7f748a419700 1 --2- 172.21.15.38:0/3585908967 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7484100950 0x7f7484107210 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:16.826 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.823+0000 7f748a419700 1 --2- 172.21.15.38:0/3585908967 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f74840fedd0 0x7f74840ff1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:16.827 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.823+0000 7f748a419700 1 --2- 172.21.15.38:0/3585908967 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f74840fffc0 0x7f7484100410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:16.827 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.823+0000 7f748a419700 1 -- 172.21.15.38:0/3585908967 >> 172.21.15.38:0/3585908967 conn(0x7f74840fa3d0 msgr2=0x7f74840fc820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:16.827 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.823+0000 7f748a419700 1 -- 172.21.15.38:0/3585908967 shutdown_connections 2023-03-09T14:44:16.827 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.823+0000 7f748a419700 1 -- 172.21.15.38:0/3585908967 wait complete. 2023-03-09T14:44:16.827 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.824+0000 7f748a419700 1 Processor -- start 2023-03-09T14:44:16.827 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.824+0000 7f748a419700 1 -- start start 2023-03-09T14:44:16.829 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.824+0000 7f748a419700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f74840fedd0 0x7f7484197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:16.829 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.824+0000 7f748a419700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f74840fffc0 0x7f7484198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:16.829 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.824+0000 7f748a419700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7484100950 0x7f748419f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:16.829 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.824+0000 7f748a419700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f748419fa70 con 0x7f7484100950 2023-03-09T14:44:16.829 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.824+0000 7f748a419700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f748419fbb0 con 0x7f74840fffc0 2023-03-09T14:44:16.830 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.824+0000 7f748a419700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f748419fd40 con 0x7f74840fedd0 2023-03-09T14:44:16.830 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.824+0000 7f74889b6700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7484100950 0x7f748419f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:16.830 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.824+0000 7f7483fff700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f74840fedd0 0x7f7484197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:16.830 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.824+0000 7f74889b6700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7484100950 0x7f748419f4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36634/0 (socket says 172.21.15.38:36634) 2023-03-09T14:44:16.830 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.824+0000 7f7483fff700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f74840fedd0 0x7f7484197ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.38:44328/0 (socket says 172.21.15.38:44328) 2023-03-09T14:44:16.830 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.824+0000 7f74889b6700 1 -- 172.21.15.38:0/2802100888 learned_addr learned my addr 172.21.15.38:0/2802100888 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:44:16.830 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.824+0000 7f74837fe700 1 --2- 172.21.15.38:0/2802100888 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f74840fffc0 0x7f7484198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:16.831 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.824+0000 7f74889b6700 1 -- 172.21.15.38:0/2802100888 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f74840fedd0 msgr2=0x7f7484197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:16.831 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.824+0000 7f74889b6700 1 --2- 172.21.15.38:0/2802100888 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f74840fedd0 0x7f7484197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:16.831 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.824+0000 7f74889b6700 1 -- 172.21.15.38:0/2802100888 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f74840fffc0 msgr2=0x7f7484198430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:16.831 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.824+0000 7f74889b6700 1 --2- 172.21.15.38:0/2802100888 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f74840fffc0 0x7f7484198430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:16.831 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.824+0000 7f74889b6700 1 -- 172.21.15.38:0/2802100888 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f747c009d00 con 0x7f7484100950 2023-03-09T14:44:16.831 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.824+0000 7f74837fe700 1 --2- 172.21.15.38:0/2802100888 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f74840fffc0 0x7f7484198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:44:16.831 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.824+0000 7f7483fff700 1 --2- 172.21.15.38:0/2802100888 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f74840fedd0 0x7f7484197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:44:16.831 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.825+0000 7f74889b6700 1 --2- 172.21.15.38:0/2802100888 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7484100950 0x7f748419f4a0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f747c00b780 tx=0x7f747c00bd00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:16.832 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.825+0000 7f74817fa700 1 -- 172.21.15.38:0/2802100888 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f747c00e040 con 0x7f7484100950 2023-03-09T14:44:16.832 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.825+0000 7f74817fa700 1 -- 172.21.15.38:0/2802100888 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f747c004c60 con 0x7f7484100950 2023-03-09T14:44:16.832 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.825+0000 7f748a419700 1 -- 172.21.15.38:0/2802100888 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f748410a740 con 0x7f7484100950 2023-03-09T14:44:16.832 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.825+0000 7f74817fa700 1 -- 172.21.15.38:0/2802100888 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f747c01b6b0 con 0x7f7484100950 2023-03-09T14:44:16.832 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.825+0000 7f748a419700 1 -- 172.21.15.38:0/2802100888 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f748410a5b0 con 0x7f7484100950 2023-03-09T14:44:16.832 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.826+0000 7f748a419700 1 -- 172.21.15.38:0/2802100888 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74841a02b0 con 0x7f7484100950 2023-03-09T14:44:16.833 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.827+0000 7f74817fa700 1 -- 172.21.15.38:0/2802100888 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 133198+0+0 (secure 0 0 0) 0x7f747c01d480 con 0x7f7484100950 2023-03-09T14:44:16.833 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.828+0000 7f74817fa700 1 --2- 172.21.15.38:0/2802100888 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f746c09f3c0 0x7f746c0a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:16.833 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.828+0000 7f7483fff700 1 --2- 172.21.15.38:0/2802100888 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f746c09f3c0 0x7f746c0a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:16.833 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.828+0000 7f74817fa700 1 -- 172.21.15.38:0/2802100888 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f747c0cb430 con 0x7f7484100950 2023-03-09T14:44:16.834 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.828+0000 7f7483fff700 1 --2- 172.21.15.38:0/2802100888 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f746c09f3c0 0x7f746c0a1870 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f74840ffe20 tx=0x7f7474009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:16.834 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:16.832+0000 7f74817fa700 1 -- 172.21.15.38:0/2802100888 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f747c095c80 con 0x7f7484100950 2023-03-09T14:44:17.045 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:17.042+0000 7f748a419700 1 -- 172.21.15.38:0/2802100888 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f74841a09b0 con 0x7f7484100950 2023-03-09T14:44:17.651 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:17.648+0000 7f74817fa700 1 -- 172.21.15.38:0/2802100888 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v80) v1 ==== 138+0+0 (secure 0 0 0) 0x7f747c0992a0 con 0x7f7484100950 2023-03-09T14:44:17.655 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:17.653+0000 7f748a419700 1 -- 172.21.15.38:0/2802100888 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f746c09f3c0 msgr2=0x7f746c0a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:17.655 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:17.653+0000 7f748a419700 1 --2- 172.21.15.38:0/2802100888 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f746c09f3c0 0x7f746c0a1870 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f74840ffe20 tx=0x7f7474009260 comp rx=0 tx=0).stop 2023-03-09T14:44:17.655 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:17.653+0000 7f748a419700 1 -- 172.21.15.38:0/2802100888 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7484100950 msgr2=0x7f748419f4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:17.656 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:17.653+0000 7f748a419700 1 --2- 172.21.15.38:0/2802100888 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7484100950 0x7f748419f4a0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f747c00b780 tx=0x7f747c00bd00 comp rx=0 tx=0).stop 2023-03-09T14:44:17.656 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:17.654+0000 7f748a419700 1 -- 172.21.15.38:0/2802100888 shutdown_connections 2023-03-09T14:44:17.657 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:17.654+0000 7f748a419700 1 --2- 172.21.15.38:0/2802100888 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7484100950 0x7f748419f4a0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:17.657 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:17.654+0000 7f748a419700 1 --2- 172.21.15.38:0/2802100888 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f746c09f3c0 0x7f746c0a1870 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:17.657 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:17.654+0000 7f748a419700 1 --2- 172.21.15.38:0/2802100888 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f74840fedd0 0x7f7484197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:17.657 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:17.654+0000 7f748a419700 1 --2- 172.21.15.38:0/2802100888 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f74840fffc0 0x7f7484198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:17.657 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:17.654+0000 7f748a419700 1 -- 172.21.15.38:0/2802100888 >> 172.21.15.38:0/2802100888 conn(0x7f74840fa3d0 msgr2=0x7f7484068a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:17.657 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:17.654+0000 7f748a419700 1 -- 172.21.15.38:0/2802100888 shutdown_connections 2023-03-09T14:44:17.658 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:17.654+0000 7f748a419700 1 -- 172.21.15.38:0/2802100888 wait complete. 2023-03-09T14:44:17.658 INFO:teuthology.orchestra.run.smithi038.stderr:pool 'cephfs_metadata' created 2023-03-09T14:44:17.670 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2023-03-09T14:44:17.808 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:17.804+0000 7f044bb46700 1 -- 172.21.15.38:0/1091141851 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0444100910 msgr2=0x7f04441071d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:17.940 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:17.804+0000 7f044bb46700 1 --2- 172.21.15.38:0/1091141851 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0444100910 0x7f04441071d0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f043c00b0a0 tx=0x7f043c00b3b0 comp rx=0 tx=0).stop 2023-03-09T14:44:17.941 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:17.804+0000 7f044bb46700 1 -- 172.21.15.38:0/1091141851 shutdown_connections 2023-03-09T14:44:17.941 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:17.804+0000 7f044bb46700 1 --2- 172.21.15.38:0/1091141851 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0444100910 0x7f04441071d0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:17.941 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:17.804+0000 7f044bb46700 1 --2- 172.21.15.38:0/1091141851 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f04440fed90 0x7f04440ff1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:17.942 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:17.804+0000 7f044bb46700 1 --2- 172.21.15.38:0/1091141851 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f04440fff80 0x7f04441003d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:17.942 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:17.804+0000 7f044bb46700 1 -- 172.21.15.38:0/1091141851 >> 172.21.15.38:0/1091141851 conn(0x7f04440fa3b0 msgr2=0x7f04440fc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:17.942 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:17.804+0000 7f044bb46700 1 -- 172.21.15.38:0/1091141851 shutdown_connections 2023-03-09T14:44:17.942 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:17.804+0000 7f044bb46700 1 -- 172.21.15.38:0/1091141851 wait complete. 2023-03-09T14:44:17.943 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:17.805+0000 7f044bb46700 1 Processor -- start 2023-03-09T14:44:17.943 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:17.805+0000 7f044bb46700 1 -- start start 2023-03-09T14:44:17.943 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:17.805+0000 7f044bb46700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f04440fed90 0x7f0444102d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:17.943 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:17.805+0000 7f044bb46700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f04440fff80 0x7f0444101360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:17.943 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:17.805+0000 7f044bb46700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0444100910 0x7f04441018a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:17.944 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:17.805+0000 7f044bb46700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0444102040 con 0x7f0444100910 2023-03-09T14:44:17.944 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:17.805+0000 7f044bb46700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04441021b0 con 0x7f04440fff80 2023-03-09T14:44:17.944 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:17.805+0000 7f044bb46700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f044419ff50 con 0x7f04440fed90 2023-03-09T14:44:17.944 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:17.805+0000 7f044a0e3700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0444100910 0x7f04441018a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:17.945 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:17.805+0000 7f04498e2700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f04440fed90 0x7f0444102d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:17.945 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:17.805+0000 7f044a0e3700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0444100910 0x7f04441018a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36648/0 (socket says 172.21.15.38:36648) 2023-03-09T14:44:17.945 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:17.805+0000 7f044a0e3700 1 -- 172.21.15.38:0/755287196 learned_addr learned my addr 172.21.15.38:0/755287196 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:44:17.945 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:17.805+0000 7f04490e1700 1 --2- 172.21.15.38:0/755287196 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f04440fff80 0x7f0444101360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:17.945 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:17.805+0000 7f044a0e3700 1 -- 172.21.15.38:0/755287196 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f04440fed90 msgr2=0x7f0444102d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:17.945 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:17.805+0000 7f044a0e3700 1 --2- 172.21.15.38:0/755287196 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f04440fed90 0x7f0444102d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:17.945 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:17.805+0000 7f044a0e3700 1 -- 172.21.15.38:0/755287196 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f04440fff80 msgr2=0x7f0444101360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:17.946 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:17.805+0000 7f044a0e3700 1 --2- 172.21.15.38:0/755287196 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f04440fff80 0x7f0444101360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:17.946 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:17.805+0000 7f044a0e3700 1 -- 172.21.15.38:0/755287196 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f043c009d00 con 0x7f0444100910 2023-03-09T14:44:17.946 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:17.806+0000 7f044a0e3700 1 --2- 172.21.15.38:0/755287196 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0444100910 0x7f04441018a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f043c00d010 tx=0x7f043c00bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:17.946 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:17.806+0000 7f043affd700 1 -- 172.21.15.38:0/755287196 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f043c00e040 con 0x7f0444100910 2023-03-09T14:44:17.946 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:17.806+0000 7f043affd700 1 -- 172.21.15.38:0/755287196 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f043c00bdb0 con 0x7f0444100910 2023-03-09T14:44:17.946 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:17.806+0000 7f044bb46700 1 -- 172.21.15.38:0/755287196 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f044410a750 con 0x7f0444100910 2023-03-09T14:44:17.946 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:17.806+0000 7f043affd700 1 -- 172.21.15.38:0/755287196 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f043c005260 con 0x7f0444100910 2023-03-09T14:44:17.946 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:17.806+0000 7f044bb46700 1 -- 172.21.15.38:0/755287196 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f044410a5c0 con 0x7f0444100910 2023-03-09T14:44:17.948 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:17.807+0000 7f044bb46700 1 -- 172.21.15.38:0/755287196 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04441a04f0 con 0x7f0444100910 2023-03-09T14:44:17.948 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:17.809+0000 7f043affd700 1 -- 172.21.15.38:0/755287196 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 133198+0+0 (secure 0 0 0) 0x7f043c01d480 con 0x7f0444100910 2023-03-09T14:44:17.948 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:17.810+0000 7f043affd700 1 --2- 172.21.15.38:0/755287196 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f043009f3c0 0x7f04300a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:17.948 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:17.810+0000 7f04498e2700 1 --2- 172.21.15.38:0/755287196 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f043009f3c0 0x7f04300a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:17.949 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:17.810+0000 7f043affd700 1 -- 172.21.15.38:0/755287196 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7497+0+0 (secure 0 0 0) 0x7f043c0cb000 con 0x7f0444100910 2023-03-09T14:44:17.949 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:17.810+0000 7f04498e2700 1 --2- 172.21.15.38:0/755287196 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f043009f3c0 0x7f04300a1870 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f04440ffde0 tx=0x7f0440009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:17.949 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:17.813+0000 7f043affd700 1 -- 172.21.15.38:0/755287196 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f043c0957d0 con 0x7f0444100910 2023-03-09T14:44:17.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:17 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:17.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:17 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:17.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:17 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:17.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:17 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:17.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:17 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:17.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:17 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:17.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:17 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/231540167' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-03-09T14:44:17.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:17 smithi122 ceph-mon[93869]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-03-09T14:44:17.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:17 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/2802100888' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-03-09T14:44:17.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:17 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:44:17.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:17 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:44:17.980 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:17 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:17.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:17 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:17.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:17 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:17.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:17 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:17.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:17 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:17.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:17 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:17.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:17 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:17.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:17 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/231540167' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-03-09T14:44:17.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:17 smithi181 ceph-mon[94189]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-03-09T14:44:17.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:17 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/2802100888' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-03-09T14:44:17.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:17 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:44:17.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:17 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:44:17.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:17 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:17 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:18.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:17 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:18.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:17 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:18.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:17 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:18.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:17 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:18.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:17 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:18.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:17 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/231540167' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-03-09T14:44:18.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:17 smithi038 ceph-mon[92422]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-03-09T14:44:18.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:17 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/2802100888' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-03-09T14:44:18.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:17 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:44:18.004 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:17 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:44:18.004 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:17 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:18.020 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:18.018+0000 7f044bb46700 1 -- 172.21.15.38:0/755287196 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) v1 -- 0x7f04441a0d60 con 0x7f0444100910 2023-03-09T14:44:18.649 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:18.646+0000 7f043affd700 1 -- 172.21.15.38:0/755287196 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v81) v1 ==== 171+0+0 (secure 0 0 0) 0x7f043c011390 con 0x7f0444100910 2023-03-09T14:44:18.653 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:18.651+0000 7f044bb46700 1 -- 172.21.15.38:0/755287196 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f043009f3c0 msgr2=0x7f04300a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:18.653 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:18.651+0000 7f044bb46700 1 --2- 172.21.15.38:0/755287196 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f043009f3c0 0x7f04300a1870 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f04440ffde0 tx=0x7f0440009260 comp rx=0 tx=0).stop 2023-03-09T14:44:18.654 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:18.651+0000 7f044bb46700 1 -- 172.21.15.38:0/755287196 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0444100910 msgr2=0x7f04441018a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:18.654 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:18.651+0000 7f044bb46700 1 --2- 172.21.15.38:0/755287196 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0444100910 0x7f04441018a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f043c00d010 tx=0x7f043c00bc80 comp rx=0 tx=0).stop 2023-03-09T14:44:18.658 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:18.654+0000 7f044bb46700 1 -- 172.21.15.38:0/755287196 shutdown_connections 2023-03-09T14:44:18.659 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:18.654+0000 7f044bb46700 1 --2- 172.21.15.38:0/755287196 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0444100910 0x7f04441018a0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:18.659 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:18.654+0000 7f044bb46700 1 --2- 172.21.15.38:0/755287196 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f043009f3c0 0x7f04300a1870 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:18.659 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:18.654+0000 7f044bb46700 1 --2- 172.21.15.38:0/755287196 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f04440fed90 0x7f0444102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:18.660 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:18.654+0000 7f044bb46700 1 --2- 172.21.15.38:0/755287196 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f04440fff80 0x7f0444101360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:18.660 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:18.654+0000 7f044bb46700 1 -- 172.21.15.38:0/755287196 >> 172.21.15.38:0/755287196 conn(0x7f04440fa3b0 msgr2=0x7f0444067a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:18.666 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:18.664+0000 7f044bb46700 1 -- 172.21.15.38:0/755287196 shutdown_connections 2023-03-09T14:44:18.669 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:18.667+0000 7f044bb46700 1 -- 172.21.15.38:0/755287196 wait complete. 2023-03-09T14:44:18.670 INFO:teuthology.orchestra.run.smithi038.stderr:pool 'cephfs_data' created 2023-03-09T14:44:18.682 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-03-09T14:44:18.816 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:18.813+0000 7fba43435700 1 -- 172.21.15.38:0/4058162464 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fba3c0ff0b0 msgr2=0x7fba3c0ff520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:18.817 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:18.813+0000 7fba43435700 1 --2- 172.21.15.38:0/4058162464 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fba3c0ff0b0 0x7fba3c0ff520 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fba2c009990 tx=0x7fba2c009ca0 comp rx=0 tx=0).stop 2023-03-09T14:44:18.817 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:18.814+0000 7fba43435700 1 -- 172.21.15.38:0/4058162464 shutdown_connections 2023-03-09T14:44:18.817 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:18.814+0000 7fba43435700 1 --2- 172.21.15.38:0/4058162464 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba3c0ffa60 0x7fba3c0ffed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:18.817 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:18.814+0000 7fba43435700 1 --2- 172.21.15.38:0/4058162464 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fba3c0fe760 0x7fba3c0feb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:18.817 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:18.814+0000 7fba43435700 1 --2- 172.21.15.38:0/4058162464 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fba3c0ff0b0 0x7fba3c0ff520 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:18.817 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:18.814+0000 7fba43435700 1 -- 172.21.15.38:0/4058162464 >> 172.21.15.38:0/4058162464 conn(0x7fba3c0fa3b0 msgr2=0x7fba3c0fc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:18.818 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:18.814+0000 7fba43435700 1 -- 172.21.15.38:0/4058162464 shutdown_connections 2023-03-09T14:44:18.818 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:18.814+0000 7fba43435700 1 -- 172.21.15.38:0/4058162464 wait complete. 2023-03-09T14:44:18.819 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:18.815+0000 7fba43435700 1 Processor -- start 2023-03-09T14:44:18.820 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:18.815+0000 7fba43435700 1 -- start start 2023-03-09T14:44:18.820 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:18.815+0000 7fba43435700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fba3c0fe760 0x7fba3c197e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:18.820 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:18.815+0000 7fba43435700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fba3c0ff0b0 0x7fba3c198370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:18.820 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:18.815+0000 7fba43435700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba3c0ffa60 0x7fba3c19f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:18.820 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:18.815+0000 7fba43435700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba3c19f9b0 con 0x7fba3c0ff0b0 2023-03-09T14:44:18.820 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:18.815+0000 7fba43435700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba3c19faf0 con 0x7fba3c0ffa60 2023-03-09T14:44:18.820 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:18.815+0000 7fba43435700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba3c19fc80 con 0x7fba3c0fe760 2023-03-09T14:44:18.821 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:18.815+0000 7fba409d0700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fba3c0ff0b0 0x7fba3c198370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:18.821 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:18.815+0000 7fba409d0700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fba3c0ff0b0 0x7fba3c198370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36664/0 (socket says 172.21.15.38:36664) 2023-03-09T14:44:18.821 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:18.815+0000 7fba409d0700 1 -- 172.21.15.38:0/135819889 learned_addr learned my addr 172.21.15.38:0/135819889 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:44:18.821 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:18.815+0000 7fba411d1700 1 --2- 172.21.15.38:0/135819889 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fba3c0fe760 0x7fba3c197e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:18.822 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:18.815+0000 7fba419d2700 1 --2- 172.21.15.38:0/135819889 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba3c0ffa60 0x7fba3c19f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:18.822 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:18.815+0000 7fba409d0700 1 -- 172.21.15.38:0/135819889 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fba3c0fe760 msgr2=0x7fba3c197e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:18.822 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:18.815+0000 7fba409d0700 1 --2- 172.21.15.38:0/135819889 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fba3c0fe760 0x7fba3c197e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:18.822 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:18.815+0000 7fba409d0700 1 -- 172.21.15.38:0/135819889 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba3c0ffa60 msgr2=0x7fba3c19f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:18.822 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:18.815+0000 7fba409d0700 1 --2- 172.21.15.38:0/135819889 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba3c0ffa60 0x7fba3c19f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:18.822 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:18.815+0000 7fba409d0700 1 -- 172.21.15.38:0/135819889 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba2c0095f0 con 0x7fba3c0ff0b0 2023-03-09T14:44:18.822 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:18.815+0000 7fba409d0700 1 --2- 172.21.15.38:0/135819889 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fba3c0ff0b0 0x7fba3c198370 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fba2c0059d0 tx=0x7fba2c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:18.822 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:18.816+0000 7fba327fc700 1 -- 172.21.15.38:0/135819889 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba2c01d070 con 0x7fba3c0ff0b0 2023-03-09T14:44:18.823 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:18.816+0000 7fba43435700 1 -- 172.21.15.38:0/135819889 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba3c10d9d0 con 0x7fba3c0ff0b0 2023-03-09T14:44:18.823 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:18.816+0000 7fba327fc700 1 -- 172.21.15.38:0/135819889 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fba2c00fb90 con 0x7fba3c0ff0b0 2023-03-09T14:44:18.823 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:18.816+0000 7fba327fc700 1 -- 172.21.15.38:0/135819889 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba2c0173f0 con 0x7fba3c0ff0b0 2023-03-09T14:44:18.823 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:18.816+0000 7fba43435700 1 -- 172.21.15.38:0/135819889 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba3c10d840 con 0x7fba3c0ff0b0 2023-03-09T14:44:18.824 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:18.817+0000 7fba43435700 1 -- 172.21.15.38:0/135819889 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba3c109540 con 0x7fba3c0ff0b0 2023-03-09T14:44:18.825 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:18.818+0000 7fba327fc700 1 -- 172.21.15.38:0/135819889 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 133198+0+0 (secure 0 0 0) 0x7fba2c029030 con 0x7fba3c0ff0b0 2023-03-09T14:44:18.825 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:18.818+0000 7fba327fc700 1 --2- 172.21.15.38:0/135819889 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fba2809f3c0 0x7fba280a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:18.825 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:18.819+0000 7fba327fc700 1 -- 172.21.15.38:0/135819889 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7879+0+0 (secure 0 0 0) 0x7fba2c0caed0 con 0x7fba3c0ff0b0 2023-03-09T14:44:18.825 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:18.819+0000 7fba411d1700 1 --2- 172.21.15.38:0/135819889 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fba2809f3c0 0x7fba280a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:18.826 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:18.819+0000 7fba411d1700 1 --2- 172.21.15.38:0/135819889 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fba2809f3c0 0x7fba280a1870 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fba38009c60 tx=0x7fba38009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:18.826 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:18.823+0000 7fba327fc700 1 -- 172.21.15.38:0/135819889 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7fba2c05c5b0 con 0x7fba3c0ff0b0 2023-03-09T14:44:18.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:18 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:44:18.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:18 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/2802100888' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-03-09T14:44:18.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:18 smithi122 ceph-mon[93869]: osdmap e80: 12 total, 12 up, 12 in 2023-03-09T14:44:18.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:18 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/755287196' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2023-03-09T14:44:18.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:18 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:44:18.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:18 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/2802100888' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-03-09T14:44:18.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:18 smithi181 ceph-mon[94189]: osdmap e80: 12 total, 12 up, 12 in 2023-03-09T14:44:18.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:18 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/755287196' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2023-03-09T14:44:19.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:18 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:44:19.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:18 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/2802100888' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-03-09T14:44:19.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:18 smithi038 ceph-mon[92422]: osdmap e80: 12 total, 12 up, 12 in 2023-03-09T14:44:19.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:18 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/755287196' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2023-03-09T14:44:19.084 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:19.082+0000 7fba43435700 1 -- 172.21.15.38:0/135819889 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fba3c061640 con 0x7fba3c0ff0b0 2023-03-09T14:44:19.092 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:19.090+0000 7fba327fc700 1 -- 172.21.15.38:0/135819889 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7fba2c0270c0 con 0x7fba3c0ff0b0 2023-03-09T14:44:19.098 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:19.096+0000 7fba43435700 1 -- 172.21.15.38:0/135819889 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fba2809f3c0 msgr2=0x7fba280a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:19.098 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:19.096+0000 7fba43435700 1 --2- 172.21.15.38:0/135819889 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fba2809f3c0 0x7fba280a1870 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fba38009c60 tx=0x7fba38009260 comp rx=0 tx=0).stop 2023-03-09T14:44:19.099 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:19.096+0000 7fba43435700 1 -- 172.21.15.38:0/135819889 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fba3c0ff0b0 msgr2=0x7fba3c198370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:19.099 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:19.096+0000 7fba43435700 1 --2- 172.21.15.38:0/135819889 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fba3c0ff0b0 0x7fba3c198370 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fba2c0059d0 tx=0x7fba2c004730 comp rx=0 tx=0).stop 2023-03-09T14:44:19.099 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:19.096+0000 7fba43435700 1 -- 172.21.15.38:0/135819889 shutdown_connections 2023-03-09T14:44:19.099 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:19.096+0000 7fba43435700 1 --2- 172.21.15.38:0/135819889 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fba2809f3c0 0x7fba280a1870 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:19.099 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:19.096+0000 7fba43435700 1 --2- 172.21.15.38:0/135819889 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba3c0ffa60 0x7fba3c19f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:19.100 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:19.096+0000 7fba43435700 1 --2- 172.21.15.38:0/135819889 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fba3c0fe760 0x7fba3c197e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:19.100 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:19.096+0000 7fba43435700 1 --2- 172.21.15.38:0/135819889 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fba3c0ff0b0 0x7fba3c198370 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:19.100 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:19.096+0000 7fba43435700 1 -- 172.21.15.38:0/135819889 >> 172.21.15.38:0/135819889 conn(0x7fba3c0fa3b0 msgr2=0x7fba3c0fc0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:19.100 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:19.098+0000 7fba43435700 1 -- 172.21.15.38:0/135819889 shutdown_connections 2023-03-09T14:44:19.100 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:19.098+0000 7fba43435700 1 -- 172.21.15.38:0/135819889 wait complete. 2023-03-09T14:44:19.108 INFO:teuthology.orchestra.run.smithi038.stderr:new fs with metadata pool 2 and data pool 3 2023-03-09T14:44:19.126 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2023-03-09T14:44:19.126 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2023-03-09T14:44:19.263 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:19.260+0000 7f65004b3700 1 -- 172.21.15.38:0/4174251492 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f64f8103240 msgr2=0x7f64f8105620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:19.263 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:19.260+0000 7f65004b3700 1 --2- 172.21.15.38:0/4174251492 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f64f8103240 0x7f64f8105620 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f64e8009990 tx=0x7f64e8009ca0 comp rx=0 tx=0).stop 2023-03-09T14:44:19.263 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:19.260+0000 7f65004b3700 1 -- 172.21.15.38:0/4174251492 shutdown_connections 2023-03-09T14:44:19.264 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:19.260+0000 7f65004b3700 1 --2- 172.21.15.38:0/4174251492 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f64f8105b60 0x7f64f8107fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:19.264 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:19.260+0000 7f65004b3700 1 --2- 172.21.15.38:0/4174251492 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64f8100920 0x7f64f8102d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:19.264 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:19.260+0000 7f65004b3700 1 --2- 172.21.15.38:0/4174251492 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f64f8103240 0x7f64f8105620 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:19.264 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:19.260+0000 7f65004b3700 1 -- 172.21.15.38:0/4174251492 >> 172.21.15.38:0/4174251492 conn(0x7f64f80fa3a0 msgr2=0x7f64f80fc7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:19.264 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:19.260+0000 7f65004b3700 1 -- 172.21.15.38:0/4174251492 shutdown_connections 2023-03-09T14:44:19.264 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:19.261+0000 7f65004b3700 1 -- 172.21.15.38:0/4174251492 wait complete. 2023-03-09T14:44:19.264 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:19.261+0000 7f65004b3700 1 Processor -- start 2023-03-09T14:44:19.265 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:19.261+0000 7f65004b3700 1 -- start start 2023-03-09T14:44:19.266 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:19.261+0000 7f65004b3700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f64f8100920 0x7f64f8100290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:19.266 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:19.261+0000 7f65004b3700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f64f8103240 0x7f64f80fe8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:19.266 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:19.261+0000 7f65004b3700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64f8105b60 0x7f64f80fee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:19.267 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:19.261+0000 7f65004b3700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64f80ff5b0 con 0x7f64f8103240 2023-03-09T14:44:19.267 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:19.261+0000 7f65004b3700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64f80ff720 con 0x7f64f8105b60 2023-03-09T14:44:19.267 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:19.261+0000 7f65004b3700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64f819fbe0 con 0x7f64f8100920 2023-03-09T14:44:19.267 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:19.261+0000 7f64fda4e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f64f8103240 0x7f64f80fe8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:19.267 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:19.261+0000 7f64fda4e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f64f8103240 0x7f64f80fe8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36682/0 (socket says 172.21.15.38:36682) 2023-03-09T14:44:19.267 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:19.261+0000 7f64fda4e700 1 -- 172.21.15.38:0/3249866193 learned_addr learned my addr 172.21.15.38:0/3249866193 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:44:19.267 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:19.261+0000 7f64fe24f700 1 --2- 172.21.15.38:0/3249866193 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f64f8100920 0x7f64f8100290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:19.268 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:19.262+0000 7f64fea50700 1 --2- 172.21.15.38:0/3249866193 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64f8105b60 0x7f64f80fee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:19.268 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:19.262+0000 7f64fda4e700 1 -- 172.21.15.38:0/3249866193 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f64f8100920 msgr2=0x7f64f8100290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:19.268 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:19.262+0000 7f64fda4e700 1 --2- 172.21.15.38:0/3249866193 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f64f8100920 0x7f64f8100290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:19.268 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:19.262+0000 7f64fda4e700 1 -- 172.21.15.38:0/3249866193 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64f8105b60 msgr2=0x7f64f80fee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:19.268 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:19.262+0000 7f64fda4e700 1 --2- 172.21.15.38:0/3249866193 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64f8105b60 0x7f64f80fee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:19.268 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:19.262+0000 7f64fda4e700 1 -- 172.21.15.38:0/3249866193 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64e80095f0 con 0x7f64f8103240 2023-03-09T14:44:19.268 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:19.262+0000 7f64fda4e700 1 --2- 172.21.15.38:0/3249866193 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f64f8103240 0x7f64f80fe8d0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f64e80059d0 tx=0x7f64e80047a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:19.268 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:19.262+0000 7f64ef7fe700 1 -- 172.21.15.38:0/3249866193 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e801d070 con 0x7f64f8103240 2023-03-09T14:44:19.269 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:19.262+0000 7f64ef7fe700 1 -- 172.21.15.38:0/3249866193 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f64e800fc40 con 0x7f64f8103240 2023-03-09T14:44:19.269 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:19.262+0000 7f64ef7fe700 1 -- 172.21.15.38:0/3249866193 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e8017650 con 0x7f64f8103240 2023-03-09T14:44:19.269 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:19.262+0000 7f65004b3700 1 -- 172.21.15.38:0/3249866193 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64f810b7f0 con 0x7f64f8103240 2023-03-09T14:44:19.269 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:19.262+0000 7f65004b3700 1 -- 172.21.15.38:0/3249866193 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64f810b660 con 0x7f64f8103240 2023-03-09T14:44:19.269 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:19.263+0000 7f65004b3700 1 -- 172.21.15.38:0/3249866193 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64f80fbf10 con 0x7f64f8103240 2023-03-09T14:44:19.269 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:19.264+0000 7f64ef7fe700 1 -- 172.21.15.38:0/3249866193 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 133198+0+0 (secure 0 0 0) 0x7f64e8029030 con 0x7f64f8103240 2023-03-09T14:44:19.270 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:19.265+0000 7f64ef7fe700 1 --2- 172.21.15.38:0/3249866193 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f64e409f3c0 0x7f64e40a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:19.270 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:19.265+0000 7f64fe24f700 1 --2- 172.21.15.38:0/3249866193 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f64e409f3c0 0x7f64e40a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:19.270 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:19.265+0000 7f64ef7fe700 1 -- 172.21.15.38:0/3249866193 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f64e80cb850 con 0x7f64f8103240 2023-03-09T14:44:19.271 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:19.265+0000 7f64fe24f700 1 --2- 172.21.15.38:0/3249866193 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f64e409f3c0 0x7f64e40a1870 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f64f4005950 tx=0x7f64f400a300 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:19.271 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:19.268+0000 7f64ef7fe700 1 -- 172.21.15.38:0/3249866193 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f64e8095e40 con 0x7f64f8103240 2023-03-09T14:44:19.485 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:19.483+0000 7f65004b3700 1 -- 172.21.15.38:0/3249866193 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) v1 -- 0x7f64f8061390 con 0x7f64f8103240 2023-03-09T14:44:19.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:19 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/755287196' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2023-03-09T14:44:19.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:19 smithi122 ceph-mon[93869]: osdmap e81: 12 total, 12 up, 12 in 2023-03-09T14:44:19.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:19 smithi122 ceph-mon[93869]: pgmap v253: 97 pgs: 3 creating+peering, 1 active+clean, 93 unknown; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-03-09T14:44:19.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:19 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/135819889' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-03-09T14:44:19.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:19 smithi122 ceph-mon[93869]: daemon mds.b assigned to filesystem cephfs as rank 0 2023-03-09T14:44:19.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:19 smithi122 ceph-mon[93869]: osdmap e82: 12 total, 12 up, 12 in 2023-03-09T14:44:19.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:19 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/135819889' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-03-09T14:44:19.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:19 smithi122 ceph-mon[93869]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2023-03-09T14:44:19.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:19 smithi122 ceph-mon[93869]: daemon mds.b is now active in filesystem cephfs as rank 0 2023-03-09T14:44:19.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:19 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/3249866193' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2023-03-09T14:44:19.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:19 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/755287196' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2023-03-09T14:44:19.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:19 smithi181 ceph-mon[94189]: osdmap e81: 12 total, 12 up, 12 in 2023-03-09T14:44:19.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:19 smithi181 ceph-mon[94189]: pgmap v253: 97 pgs: 3 creating+peering, 1 active+clean, 93 unknown; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-03-09T14:44:19.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:19 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/135819889' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-03-09T14:44:19.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:19 smithi181 ceph-mon[94189]: daemon mds.b assigned to filesystem cephfs as rank 0 2023-03-09T14:44:19.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:19 smithi181 ceph-mon[94189]: osdmap e82: 12 total, 12 up, 12 in 2023-03-09T14:44:19.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:19 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/135819889' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-03-09T14:44:19.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:19 smithi181 ceph-mon[94189]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2023-03-09T14:44:19.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:19 smithi181 ceph-mon[94189]: daemon mds.b is now active in filesystem cephfs as rank 0 2023-03-09T14:44:19.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:19 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/3249866193' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2023-03-09T14:44:20.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:19 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/755287196' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2023-03-09T14:44:20.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:19 smithi038 ceph-mon[92422]: osdmap e81: 12 total, 12 up, 12 in 2023-03-09T14:44:20.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:19 smithi038 ceph-mon[92422]: pgmap v253: 97 pgs: 3 creating+peering, 1 active+clean, 93 unknown; 577 KiB data, 75 MiB used, 1.0 TiB / 1.0 TiB avail 2023-03-09T14:44:20.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:19 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/135819889' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-03-09T14:44:20.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:19 smithi038 ceph-mon[92422]: daemon mds.b assigned to filesystem cephfs as rank 0 2023-03-09T14:44:20.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:19 smithi038 ceph-mon[92422]: osdmap e82: 12 total, 12 up, 12 in 2023-03-09T14:44:20.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:19 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/135819889' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-03-09T14:44:20.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:19 smithi038 ceph-mon[92422]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2023-03-09T14:44:20.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:19 smithi038 ceph-mon[92422]: daemon mds.b is now active in filesystem cephfs as rank 0 2023-03-09T14:44:20.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:19 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/3249866193' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2023-03-09T14:44:20.098 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:20.095+0000 7f64ef7fe700 1 -- 172.21.15.38:0/3249866193 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v83) v1 ==== 157+0+0 (secure 0 0 0) 0x7f64e8099460 con 0x7f64f8103240 2023-03-09T14:44:20.100 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:20.097+0000 7f65004b3700 1 -- 172.21.15.38:0/3249866193 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f64e409f3c0 msgr2=0x7f64e40a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:20.101 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:20.097+0000 7f65004b3700 1 --2- 172.21.15.38:0/3249866193 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f64e409f3c0 0x7f64e40a1870 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f64f4005950 tx=0x7f64f400a300 comp rx=0 tx=0).stop 2023-03-09T14:44:20.101 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:20.097+0000 7f65004b3700 1 -- 172.21.15.38:0/3249866193 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f64f8103240 msgr2=0x7f64f80fe8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:20.101 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:20.097+0000 7f65004b3700 1 --2- 172.21.15.38:0/3249866193 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f64f8103240 0x7f64f80fe8d0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f64e80059d0 tx=0x7f64e80047a0 comp rx=0 tx=0).stop 2023-03-09T14:44:20.102 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:20.097+0000 7f65004b3700 1 -- 172.21.15.38:0/3249866193 shutdown_connections 2023-03-09T14:44:20.102 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:20.097+0000 7f65004b3700 1 --2- 172.21.15.38:0/3249866193 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f64e409f3c0 0x7f64e40a1870 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:20.102 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:20.097+0000 7f65004b3700 1 --2- 172.21.15.38:0/3249866193 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64f8105b60 0x7f64f80fee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:20.103 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:20.097+0000 7f65004b3700 1 --2- 172.21.15.38:0/3249866193 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f64f8100920 0x7f64f8100290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:20.103 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:20.097+0000 7f65004b3700 1 --2- 172.21.15.38:0/3249866193 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f64f8103240 0x7f64f80fe8d0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:20.103 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:20.097+0000 7f65004b3700 1 -- 172.21.15.38:0/3249866193 >> 172.21.15.38:0/3249866193 conn(0x7f64f80fa3a0 msgr2=0x7f64f8101580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:20.103 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:20.097+0000 7f65004b3700 1 -- 172.21.15.38:0/3249866193 shutdown_connections 2023-03-09T14:44:20.104 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:20.098+0000 7f65004b3700 1 -- 172.21.15.38:0/3249866193 wait complete. 2023-03-09T14:44:20.113 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2023-03-09T14:44:20.251 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:20.249+0000 7f7d06697700 1 -- 172.21.15.38:0/958989046 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7d001019f0 msgr2=0x7f7d00108320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:20.252 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:20.249+0000 7f7d06697700 1 --2- 172.21.15.38:0/958989046 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7d001019f0 0x7f7d00108320 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f7cf800b0a0 tx=0x7f7cf800b3b0 comp rx=0 tx=0).stop 2023-03-09T14:44:20.252 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:20.249+0000 7f7d06697700 1 -- 172.21.15.38:0/958989046 shutdown_connections 2023-03-09T14:44:20.252 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:20.249+0000 7f7d06697700 1 --2- 172.21.15.38:0/958989046 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7d001019f0 0x7f7d00108320 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:20.252 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:20.249+0000 7f7d06697700 1 --2- 172.21.15.38:0/958989046 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7d000ffe60 0x7f7d00100270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:20.252 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:20.249+0000 7f7d06697700 1 --2- 172.21.15.38:0/958989046 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7d00101040 0x7f7d001014b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:20.253 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:20.249+0000 7f7d06697700 1 -- 172.21.15.38:0/958989046 >> 172.21.15.38:0/958989046 conn(0x7f7d000fb4b0 msgr2=0x7f7d000fd900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:20.253 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:20.249+0000 7f7d06697700 1 -- 172.21.15.38:0/958989046 shutdown_connections 2023-03-09T14:44:20.253 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:20.249+0000 7f7d06697700 1 -- 172.21.15.38:0/958989046 wait complete. 2023-03-09T14:44:20.253 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:20.249+0000 7f7d06697700 1 Processor -- start 2023-03-09T14:44:20.253 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:20.250+0000 7f7d06697700 1 -- start start 2023-03-09T14:44:20.255 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:20.250+0000 7f7d06697700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7d000ffe60 0x7f7d00074530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:20.255 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:20.250+0000 7f7d06697700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7d00101040 0x7f7d00072b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:20.255 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:20.250+0000 7f7d06697700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7d001019f0 0x7f7d000730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:20.255 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:20.250+0000 7f7d06697700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d00073850 con 0x7f7d001019f0 2023-03-09T14:44:20.255 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:20.250+0000 7f7d06697700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d000739c0 con 0x7f7d00101040 2023-03-09T14:44:20.256 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:20.250+0000 7f7d06697700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d0019fea0 con 0x7f7d000ffe60 2023-03-09T14:44:20.256 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:20.250+0000 7f7d04c34700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7d001019f0 0x7f7d000730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:20.256 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:20.250+0000 7f7d04c34700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7d001019f0 0x7f7d000730b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36698/0 (socket says 172.21.15.38:36698) 2023-03-09T14:44:20.256 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:20.250+0000 7f7d04c34700 1 -- 172.21.15.38:0/609648367 learned_addr learned my addr 172.21.15.38:0/609648367 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:44:20.256 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:20.250+0000 7f7cff7fe700 1 --2- 172.21.15.38:0/609648367 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7d00101040 0x7f7d00072b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:20.256 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:20.250+0000 7f7cfffff700 1 --2- 172.21.15.38:0/609648367 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7d000ffe60 0x7f7d00074530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:20.256 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:20.250+0000 7f7d04c34700 1 -- 172.21.15.38:0/609648367 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7d000ffe60 msgr2=0x7f7d00074530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:20.256 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:20.250+0000 7f7d04c34700 1 --2- 172.21.15.38:0/609648367 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7d000ffe60 0x7f7d00074530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:20.257 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:20.250+0000 7f7d04c34700 1 -- 172.21.15.38:0/609648367 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7d00101040 msgr2=0x7f7d00072b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:20.257 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:20.250+0000 7f7d04c34700 1 --2- 172.21.15.38:0/609648367 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7d00101040 0x7f7d00072b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:20.257 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:20.250+0000 7f7d04c34700 1 -- 172.21.15.38:0/609648367 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cf8009d00 con 0x7f7d001019f0 2023-03-09T14:44:20.257 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:20.250+0000 7f7d04c34700 1 --2- 172.21.15.38:0/609648367 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7d001019f0 0x7f7d000730b0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f7cf80062a0 tx=0x7f7cf8004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:20.258 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:20.251+0000 7f7cfd7fa700 1 -- 172.21.15.38:0/609648367 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cf800e040 con 0x7f7d001019f0 2023-03-09T14:44:20.258 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:20.251+0000 7f7cfd7fa700 1 -- 172.21.15.38:0/609648367 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7cf800cec0 con 0x7f7d001019f0 2023-03-09T14:44:20.258 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:20.251+0000 7f7cfd7fa700 1 -- 172.21.15.38:0/609648367 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cf8004020 con 0x7f7d001019f0 2023-03-09T14:44:20.258 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:20.251+0000 7f7d06697700 1 -- 172.21.15.38:0/609648367 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d0010b8c0 con 0x7f7d001019f0 2023-03-09T14:44:20.258 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:20.251+0000 7f7d06697700 1 -- 172.21.15.38:0/609648367 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d0010b730 con 0x7f7d001019f0 2023-03-09T14:44:20.259 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:20.252+0000 7f7d06697700 1 -- 172.21.15.38:0/609648367 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d001074a0 con 0x7f7d001019f0 2023-03-09T14:44:20.259 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:20.253+0000 7f7cfd7fa700 1 -- 172.21.15.38:0/609648367 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 133198+0+0 (secure 0 0 0) 0x7f7cf801d690 con 0x7f7d001019f0 2023-03-09T14:44:20.259 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:20.253+0000 7f7cfd7fa700 1 --2- 172.21.15.38:0/609648367 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f7ce809f3c0 0x7f7ce80a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:20.260 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:20.254+0000 7f7cfffff700 1 --2- 172.21.15.38:0/609648367 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f7ce809f3c0 0x7f7ce80a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:20.260 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:20.254+0000 7f7cfd7fa700 1 -- 172.21.15.38:0/609648367 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8205+0+0 (secure 0 0 0) 0x7f7cf80cb730 con 0x7f7d001019f0 2023-03-09T14:44:20.260 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:20.254+0000 7f7cfffff700 1 --2- 172.21.15.38:0/609648367 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f7ce809f3c0 0x7f7ce80a1870 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f7cf0006fd0 tx=0x7f7cf0008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:20.261 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:20.258+0000 7f7cfd7fa700 1 -- 172.21.15.38:0/609648367 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f7cf8095ba0 con 0x7f7d001019f0 2023-03-09T14:44:20.472 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:20.470+0000 7f7d06697700 1 -- 172.21.15.38:0/609648367 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f7d00066dd0 con 0x7f7d001019f0 2023-03-09T14:44:21.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T14:44:21.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:21 smithi122 ceph-mon[93869]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-03-09T14:44:21.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:21 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/3249866193' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2023-03-09T14:44:21.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:21 smithi122 ceph-mon[93869]: osdmap e83: 12 total, 12 up, 12 in 2023-03-09T14:44:21.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:21 smithi122 ceph-mon[93869]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-03-09T14:44:21.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:21 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:21.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:21 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:21.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:21 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:21.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:21 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:21.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:21 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:21.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:21 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:21.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:21 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:44:21.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:21 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:44:21.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:21 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:21.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:21 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/609648367' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-03-09T14:44:21.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T14:44:21.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:21 smithi181 ceph-mon[94189]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-03-09T14:44:21.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:21 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/3249866193' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2023-03-09T14:44:21.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:21 smithi181 ceph-mon[94189]: osdmap e83: 12 total, 12 up, 12 in 2023-03-09T14:44:21.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:21 smithi181 ceph-mon[94189]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-03-09T14:44:21.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:21 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:21.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:21 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:21.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:21 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:21.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:21 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:21.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:21 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:21 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:21 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:44:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:21 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:44:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:21 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:21.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:21 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/609648367' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-03-09T14:44:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T14:44:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:21 smithi038 ceph-mon[92422]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-03-09T14:44:21.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:21 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/3249866193' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2023-03-09T14:44:21.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:21 smithi038 ceph-mon[92422]: osdmap e83: 12 total, 12 up, 12 in 2023-03-09T14:44:21.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:21 smithi038 ceph-mon[92422]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-03-09T14:44:21.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:21 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:21.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:21 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:21.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:21 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:21.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:21 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:21.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:21 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:21.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:21 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:21.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:21 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:44:21.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:21 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:44:21.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:21 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:44:21.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:21 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/609648367' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-03-09T14:44:22.463 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:22.461+0000 7f7cfd7fa700 1 -- 172.21.15.38:0/609648367 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v85) v1 ==== 228+0+0 (secure 0 0 0) 0x7f7cf80991c0 con 0x7f7d001019f0 2023-03-09T14:44:22.469 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:22.466+0000 7f7d06697700 1 -- 172.21.15.38:0/609648367 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f7ce809f3c0 msgr2=0x7f7ce80a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:22.469 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:22.466+0000 7f7d06697700 1 --2- 172.21.15.38:0/609648367 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f7ce809f3c0 0x7f7ce80a1870 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f7cf0006fd0 tx=0x7f7cf0008040 comp rx=0 tx=0).stop 2023-03-09T14:44:22.470 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:22.466+0000 7f7d06697700 1 -- 172.21.15.38:0/609648367 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7d001019f0 msgr2=0x7f7d000730b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:22.470 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:22.466+0000 7f7d06697700 1 --2- 172.21.15.38:0/609648367 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7d001019f0 0x7f7d000730b0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f7cf80062a0 tx=0x7f7cf8004d00 comp rx=0 tx=0).stop 2023-03-09T14:44:22.470 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:22.467+0000 7f7d06697700 1 -- 172.21.15.38:0/609648367 shutdown_connections 2023-03-09T14:44:22.470 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:22.467+0000 7f7d06697700 1 --2- 172.21.15.38:0/609648367 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7d001019f0 0x7f7d000730b0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:22.471 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:22.467+0000 7f7d06697700 1 --2- 172.21.15.38:0/609648367 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f7ce809f3c0 0x7f7ce80a1870 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:22.471 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:22.467+0000 7f7d06697700 1 --2- 172.21.15.38:0/609648367 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7d000ffe60 0x7f7d00074530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:22.471 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:22.467+0000 7f7d06697700 1 --2- 172.21.15.38:0/609648367 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7d00101040 0x7f7d00072b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:22.471 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:22.467+0000 7f7d06697700 1 -- 172.21.15.38:0/609648367 >> 172.21.15.38:0/609648367 conn(0x7f7d000fb4b0 msgr2=0x7f7d001045e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:22.471 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:22.467+0000 7f7d06697700 1 -- 172.21.15.38:0/609648367 shutdown_connections 2023-03-09T14:44:22.471 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:22.467+0000 7f7d06697700 1 -- 172.21.15.38:0/609648367 wait complete. 2023-03-09T14:44:22.472 INFO:teuthology.orchestra.run.smithi038.stderr:pool 'cephfs_data_ec' created 2023-03-09T14:44:22.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:22 smithi122 ceph-mon[93869]: pgmap v256: 97 pgs: 3 creating+peering, 47 active+clean, 47 unknown; 579 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 3 op/s 2023-03-09T14:44:22.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:22 smithi122 ceph-mon[93869]: osdmap e84: 12 total, 12 up, 12 in 2023-03-09T14:44:22.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:22 smithi181 ceph-mon[94189]: pgmap v256: 97 pgs: 3 creating+peering, 47 active+clean, 47 unknown; 579 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 3 op/s 2023-03-09T14:44:22.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:22 smithi181 ceph-mon[94189]: osdmap e84: 12 total, 12 up, 12 in 2023-03-09T14:44:22.482 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:22 smithi038 ceph-mon[92422]: pgmap v256: 97 pgs: 3 creating+peering, 47 active+clean, 47 unknown; 579 KiB data, 76 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 3 op/s 2023-03-09T14:44:22.483 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:22 smithi038 ceph-mon[92422]: osdmap e84: 12 total, 12 up, 12 in 2023-03-09T14:44:22.484 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2023-03-09T14:44:22.620 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:22.617+0000 7f063a52f700 1 -- 172.21.15.38:0/889377256 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0634100960 msgr2=0x7f0634107200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:22.620 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:22.617+0000 7f063a52f700 1 --2- 172.21.15.38:0/889377256 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0634100960 0x7f0634107200 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f062c00b0a0 tx=0x7f062c00b3b0 comp rx=0 tx=0).stop 2023-03-09T14:44:22.620 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:22.617+0000 7f063a52f700 1 -- 172.21.15.38:0/889377256 shutdown_connections 2023-03-09T14:44:22.621 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:22.617+0000 7f063a52f700 1 --2- 172.21.15.38:0/889377256 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0634100960 0x7f0634107200 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:22.621 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:22.617+0000 7f063a52f700 1 --2- 172.21.15.38:0/889377256 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f06340ff2a0 0x7f06340ff6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:22.621 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:22.617+0000 7f063a52f700 1 --2- 172.21.15.38:0/889377256 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f06340fffb0 0x7f0634100420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:22.621 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:22.617+0000 7f063a52f700 1 -- 172.21.15.38:0/889377256 >> 172.21.15.38:0/889377256 conn(0x7f06340f9b60 msgr2=0x7f06340fbf70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:22.621 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:22.617+0000 7f063a52f700 1 -- 172.21.15.38:0/889377256 shutdown_connections 2023-03-09T14:44:22.622 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:22.617+0000 7f063a52f700 1 -- 172.21.15.38:0/889377256 wait complete. 2023-03-09T14:44:22.622 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:22.617+0000 7f063a52f700 1 Processor -- start 2023-03-09T14:44:22.622 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:22.617+0000 7f063a52f700 1 -- start start 2023-03-09T14:44:22.622 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:22.618+0000 7f063a52f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f06340ff2a0 0x7f0634102d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:22.622 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:22.618+0000 7f063a52f700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f06340fffb0 0x7f0634101390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:22.622 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:22.618+0000 7f063a52f700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0634100960 0x7f06341018d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:22.623 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:22.618+0000 7f063a52f700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0634102070 con 0x7f06340ff2a0 2023-03-09T14:44:22.623 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:22.618+0000 7f063a52f700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06341021e0 con 0x7f06340fffb0 2023-03-09T14:44:22.623 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:22.618+0000 7f063a52f700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f063419fd60 con 0x7f0634100960 2023-03-09T14:44:22.623 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:22.618+0000 7f0633fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f06340ff2a0 0x7f0634102d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:22.623 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:22.618+0000 7f06337fe700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f06340fffb0 0x7f0634101390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:22.623 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:22.618+0000 7f0633fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f06340ff2a0 0x7f0634102d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36712/0 (socket says 172.21.15.38:36712) 2023-03-09T14:44:22.624 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:22.618+0000 7f0633fff700 1 -- 172.21.15.38:0/3220650337 learned_addr learned my addr 172.21.15.38:0/3220650337 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:44:22.624 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:22.618+0000 7f0638acc700 1 --2- 172.21.15.38:0/3220650337 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0634100960 0x7f06341018d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:22.626 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:22.618+0000 7f0633fff700 1 -- 172.21.15.38:0/3220650337 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0634100960 msgr2=0x7f06341018d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:22.626 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:22.618+0000 7f0633fff700 1 --2- 172.21.15.38:0/3220650337 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0634100960 0x7f06341018d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:22.626 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:22.618+0000 7f0633fff700 1 -- 172.21.15.38:0/3220650337 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f06340fffb0 msgr2=0x7f0634101390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:22.626 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:22.618+0000 7f0633fff700 1 --2- 172.21.15.38:0/3220650337 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f06340fffb0 0x7f0634101390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:22.626 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:22.618+0000 7f0633fff700 1 -- 172.21.15.38:0/3220650337 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f062c009d00 con 0x7f06340ff2a0 2023-03-09T14:44:22.626 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:22.618+0000 7f0633fff700 1 --2- 172.21.15.38:0/3220650337 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f06340ff2a0 0x7f0634102d50 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f062400c970 tx=0x7f062400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:22.627 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:22.618+0000 7f06317fa700 1 -- 172.21.15.38:0/3220650337 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0624004220 con 0x7f06340ff2a0 2023-03-09T14:44:22.627 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:22.618+0000 7f063a52f700 1 -- 172.21.15.38:0/3220650337 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f063410a6f0 con 0x7f06340ff2a0 2023-03-09T14:44:22.627 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:22.618+0000 7f06317fa700 1 -- 172.21.15.38:0/3220650337 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0624004380 con 0x7f06340ff2a0 2023-03-09T14:44:22.627 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:22.618+0000 7f06317fa700 1 -- 172.21.15.38:0/3220650337 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0624003b60 con 0x7f06340ff2a0 2023-03-09T14:44:22.627 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:22.619+0000 7f063a52f700 1 -- 172.21.15.38:0/3220650337 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f063410a560 con 0x7f06340ff2a0 2023-03-09T14:44:22.627 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:22.619+0000 7f063a52f700 1 -- 172.21.15.38:0/3220650337 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06341a03c0 con 0x7f06340ff2a0 2023-03-09T14:44:22.628 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:22.621+0000 7f06317fa700 1 -- 172.21.15.38:0/3220650337 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 133198+0+0 (secure 0 0 0) 0x7f0624003cc0 con 0x7f06340ff2a0 2023-03-09T14:44:22.628 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:22.622+0000 7f06317fa700 1 --2- 172.21.15.38:0/3220650337 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f061c09f3c0 0x7f061c0a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:22.628 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:22.622+0000 7f06317fa700 1 -- 172.21.15.38:0/3220650337 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8698+0+0 (secure 0 0 0) 0x7f06240c9ff0 con 0x7f06340ff2a0 2023-03-09T14:44:22.628 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:22.622+0000 7f06337fe700 1 --2- 172.21.15.38:0/3220650337 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f061c09f3c0 0x7f061c0a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:22.628 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:22.622+0000 7f06337fe700 1 --2- 172.21.15.38:0/3220650337 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f061c09f3c0 0x7f061c0a1870 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f06340fefc0 tx=0x7f062800b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:22.629 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:22.625+0000 7f06317fa700 1 -- 172.21.15.38:0/3220650337 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f0624094180 con 0x7f06340ff2a0 2023-03-09T14:44:22.830 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:22.828+0000 7f063a52f700 1 -- 172.21.15.38:0/3220650337 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) v1 -- 0x7f063419ff20 con 0x7f06340ff2a0 2023-03-09T14:44:23.473 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:23.471+0000 7f06317fa700 1 -- 172.21.15.38:0/3220650337 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v86) v1 ==== 169+0+0 (secure 0 0 0) 0x7f062401d020 con 0x7f06340ff2a0 2023-03-09T14:44:23.492 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:23.485+0000 7f063a52f700 1 -- 172.21.15.38:0/3220650337 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f061c09f3c0 msgr2=0x7f061c0a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:23.492 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:23.485+0000 7f063a52f700 1 --2- 172.21.15.38:0/3220650337 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f061c09f3c0 0x7f061c0a1870 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f06340fefc0 tx=0x7f062800b410 comp rx=0 tx=0).stop 2023-03-09T14:44:23.493 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:23.485+0000 7f063a52f700 1 -- 172.21.15.38:0/3220650337 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f06340ff2a0 msgr2=0x7f0634102d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:23.493 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:23.485+0000 7f063a52f700 1 --2- 172.21.15.38:0/3220650337 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f06340ff2a0 0x7f0634102d50 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f062400c970 tx=0x7f062400cc80 comp rx=0 tx=0).stop 2023-03-09T14:44:23.493 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:23.488+0000 7f063a52f700 1 -- 172.21.15.38:0/3220650337 shutdown_connections 2023-03-09T14:44:23.493 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:23.488+0000 7f063a52f700 1 --2- 172.21.15.38:0/3220650337 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f06340ff2a0 0x7f0634102d50 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:23.493 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:23.488+0000 7f063a52f700 1 --2- 172.21.15.38:0/3220650337 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f061c09f3c0 0x7f061c0a1870 secure :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f06340fefc0 tx=0x7f062800b410 comp rx=0 tx=0).stop 2023-03-09T14:44:23.493 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:23.488+0000 7f063a52f700 1 --2- 172.21.15.38:0/3220650337 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0634100960 0x7f06341018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:23.494 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:23.488+0000 7f063a52f700 1 --2- 172.21.15.38:0/3220650337 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f06340fffb0 0x7f0634101390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:23.494 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:23.488+0000 7f063a52f700 1 -- 172.21.15.38:0/3220650337 >> 172.21.15.38:0/3220650337 conn(0x7f06340f9b60 msgr2=0x7f06340fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:23.494 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:23.490+0000 7f063a52f700 1 -- 172.21.15.38:0/3220650337 shutdown_connections 2023-03-09T14:44:23.494 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:23.491+0000 7f063a52f700 1 -- 172.21.15.38:0/3220650337 wait complete. 2023-03-09T14:44:23.497 INFO:teuthology.orchestra.run.smithi038.stderr:set pool 4 allow_ec_overwrites to true 2023-03-09T14:44:23.511 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2023-03-09T14:44:23.643 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:23.641+0000 7f6cecce0700 1 -- 172.21.15.38:0/2058803007 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6ce8100930 msgr2=0x7f6ce8102d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:23.644 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:23.641+0000 7f6cecce0700 1 --2- 172.21.15.38:0/2058803007 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6ce8100930 0x7f6ce8102d10 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f6cd8009990 tx=0x7f6cd8009ca0 comp rx=0 tx=0).stop 2023-03-09T14:44:23.644 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:23.641+0000 7f6cecce0700 1 -- 172.21.15.38:0/2058803007 shutdown_connections 2023-03-09T14:44:23.644 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:23.641+0000 7f6cecce0700 1 --2- 172.21.15.38:0/2058803007 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6ce8100930 0x7f6ce8102d10 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:23.644 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:23.641+0000 7f6cecce0700 1 --2- 172.21.15.38:0/2058803007 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6ce8105b70 0x7f6ce8107fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:23.644 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:23.641+0000 7f6cecce0700 1 --2- 172.21.15.38:0/2058803007 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6ce8103250 0x7f6ce8105630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:23.645 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:23.641+0000 7f6cecce0700 1 -- 172.21.15.38:0/2058803007 >> 172.21.15.38:0/2058803007 conn(0x7f6ce80fa3b0 msgr2=0x7f6ce80fc800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:23.645 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:23.641+0000 7f6cecce0700 1 -- 172.21.15.38:0/2058803007 shutdown_connections 2023-03-09T14:44:23.645 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:23.641+0000 7f6cecce0700 1 -- 172.21.15.38:0/2058803007 wait complete. 2023-03-09T14:44:23.647 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:23.642+0000 7f6cecce0700 1 Processor -- start 2023-03-09T14:44:23.647 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:23.642+0000 7f6cecce0700 1 -- start start 2023-03-09T14:44:23.647 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:23.642+0000 7f6cecce0700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6ce8100930 0x7f6ce8195b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:23.647 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:23.642+0000 7f6cecce0700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6ce8103250 0x7f6ce81960d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:23.647 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:23.642+0000 7f6cecce0700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6ce8105b70 0x7f6ce819d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:23.647 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:23.642+0000 7f6cecce0700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce819d710 con 0x7f6ce8105b70 2023-03-09T14:44:23.648 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:23.642+0000 7f6cecce0700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce819d850 con 0x7f6ce8103250 2023-03-09T14:44:23.648 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:23.642+0000 7f6cecce0700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce819d9e0 con 0x7f6ce8100930 2023-03-09T14:44:23.648 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:23.642+0000 7f6ce6d9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6ce8105b70 0x7f6ce819d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:23.648 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:23.642+0000 7f6ce6d9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6ce8105b70 0x7f6ce819d140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36738/0 (socket says 172.21.15.38:36738) 2023-03-09T14:44:23.648 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:23.642+0000 7f6ce6d9d700 1 -- 172.21.15.38:0/2520119920 learned_addr learned my addr 172.21.15.38:0/2520119920 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:44:23.648 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:23.642+0000 7f6ce5d9b700 1 --2- 172.21.15.38:0/2520119920 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6ce8103250 0x7f6ce81960d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:23.648 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:23.642+0000 7f6ce659c700 1 --2- 172.21.15.38:0/2520119920 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6ce8100930 0x7f6ce8195b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:23.649 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:23.643+0000 7f6ce6d9d700 1 -- 172.21.15.38:0/2520119920 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6ce8100930 msgr2=0x7f6ce8195b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:23.649 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:23.643+0000 7f6ce6d9d700 1 --2- 172.21.15.38:0/2520119920 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6ce8100930 0x7f6ce8195b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:23.649 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:23.643+0000 7f6ce6d9d700 1 -- 172.21.15.38:0/2520119920 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6ce8103250 msgr2=0x7f6ce81960d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:23.649 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:23.643+0000 7f6ce6d9d700 1 --2- 172.21.15.38:0/2520119920 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6ce8103250 0x7f6ce81960d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:23.649 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:23.643+0000 7f6ce6d9d700 1 -- 172.21.15.38:0/2520119920 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cd80095f0 con 0x7f6ce8105b70 2023-03-09T14:44:23.649 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:23.643+0000 7f6ce6d9d700 1 --2- 172.21.15.38:0/2520119920 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6ce8105b70 0x7f6ce819d140 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f6ce0007d70 tx=0x7f6ce000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:23.649 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:23.643+0000 7f6cdf7fe700 1 -- 172.21.15.38:0/2520119920 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce000ad40 con 0x7f6ce8105b70 2023-03-09T14:44:23.650 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:23.643+0000 7f6cecce0700 1 -- 172.21.15.38:0/2520119920 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ce810b7f0 con 0x7f6ce8105b70 2023-03-09T14:44:23.650 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:23.643+0000 7f6cdf7fe700 1 -- 172.21.15.38:0/2520119920 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6ce00128e0 con 0x7f6ce8105b70 2023-03-09T14:44:23.650 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:23.643+0000 7f6cdf7fe700 1 -- 172.21.15.38:0/2520119920 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce00116b0 con 0x7f6ce8105b70 2023-03-09T14:44:23.650 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:23.643+0000 7f6cecce0700 1 -- 172.21.15.38:0/2520119920 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ce810b660 con 0x7f6ce8105b70 2023-03-09T14:44:23.650 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:23.644+0000 7f6cecce0700 1 -- 172.21.15.38:0/2520119920 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ce80fbf20 con 0x7f6ce8105b70 2023-03-09T14:44:23.651 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:23.645+0000 7f6cdf7fe700 1 -- 172.21.15.38:0/2520119920 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 133198+0+0 (secure 0 0 0) 0x7f6ce0022070 con 0x7f6ce8105b70 2023-03-09T14:44:23.651 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:23.646+0000 7f6cdf7fe700 1 --2- 172.21.15.38:0/2520119920 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f6cd409f3c0 0x7f6cd40a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:23.651 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:23.646+0000 7f6ce659c700 1 --2- 172.21.15.38:0/2520119920 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f6cd409f3c0 0x7f6cd40a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:23.651 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:23.646+0000 7f6cdf7fe700 1 -- 172.21.15.38:0/2520119920 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 8698+0+0 (secure 0 0 0) 0x7f6ce00ca4f0 con 0x7f6ce8105b70 2023-03-09T14:44:23.651 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:23.647+0000 7f6ce659c700 1 --2- 172.21.15.38:0/2520119920 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f6cd409f3c0 0x7f6cd40a1870 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6cd8006010 tx=0x7f6cd800b540 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:23.652 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:23.650+0000 7f6cdf7fe700 1 -- 172.21.15.38:0/2520119920 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f6ce00946e0 con 0x7f6ce8105b70 2023-03-09T14:44:23.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:23 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/609648367' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-03-09T14:44:23.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:23 smithi122 ceph-mon[93869]: osdmap e85: 12 total, 12 up, 12 in 2023-03-09T14:44:23.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:23 smithi122 ceph-mon[93869]: pgmap v259: 129 pgs: 3 creating+peering, 83 active+clean, 43 unknown; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 10 op/s 2023-03-09T14:44:23.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:23 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/3220650337' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2023-03-09T14:44:23.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:23 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/609648367' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-03-09T14:44:23.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:23 smithi181 ceph-mon[94189]: osdmap e85: 12 total, 12 up, 12 in 2023-03-09T14:44:23.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:23 smithi181 ceph-mon[94189]: pgmap v259: 129 pgs: 3 creating+peering, 83 active+clean, 43 unknown; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 10 op/s 2023-03-09T14:44:23.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:23 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/3220650337' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2023-03-09T14:44:23.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:23 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/609648367' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-03-09T14:44:23.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:23 smithi038 ceph-mon[92422]: osdmap e85: 12 total, 12 up, 12 in 2023-03-09T14:44:23.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:23 smithi038 ceph-mon[92422]: pgmap v259: 129 pgs: 3 creating+peering, 83 active+clean, 43 unknown; 579 KiB data, 77 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 10 op/s 2023-03-09T14:44:23.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:23 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/3220650337' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2023-03-09T14:44:23.901 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:23.899+0000 7f6cecce0700 1 -- 172.21.15.38:0/2520119920 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) v1 -- 0x7f6ce802cef0 con 0x7f6ce8105b70 2023-03-09T14:44:23.907 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:23.905+0000 7f6cdf7fe700 1 -- 172.21.15.38:0/2520119920 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) v1 ==== 137+0+0 (secure 0 0 0) 0x7f6ce00160b0 con 0x7f6ce8105b70 2023-03-09T14:44:23.909 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:23.907+0000 7f6cecce0700 1 -- 172.21.15.38:0/2520119920 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f6cd409f3c0 msgr2=0x7f6cd40a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:23.909 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:23.907+0000 7f6cecce0700 1 --2- 172.21.15.38:0/2520119920 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f6cd409f3c0 0x7f6cd40a1870 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6cd8006010 tx=0x7f6cd800b540 comp rx=0 tx=0).stop 2023-03-09T14:44:23.909 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:23.907+0000 7f6cecce0700 1 -- 172.21.15.38:0/2520119920 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6ce8105b70 msgr2=0x7f6ce819d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:23.910 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:23.907+0000 7f6cecce0700 1 --2- 172.21.15.38:0/2520119920 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6ce8105b70 0x7f6ce819d140 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f6ce0007d70 tx=0x7f6ce000a340 comp rx=0 tx=0).stop 2023-03-09T14:44:23.910 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:23.907+0000 7f6cecce0700 1 -- 172.21.15.38:0/2520119920 shutdown_connections 2023-03-09T14:44:23.910 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:23.907+0000 7f6cecce0700 1 --2- 172.21.15.38:0/2520119920 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6ce8105b70 0x7f6ce819d140 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:23.910 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:23.907+0000 7f6cecce0700 1 --2- 172.21.15.38:0/2520119920 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f6cd409f3c0 0x7f6cd40a1870 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:23.910 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:23.907+0000 7f6cecce0700 1 --2- 172.21.15.38:0/2520119920 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6ce8100930 0x7f6ce8195b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:23.910 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:23.907+0000 7f6cecce0700 1 --2- 172.21.15.38:0/2520119920 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6ce8103250 0x7f6ce81960d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:23.910 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:23.907+0000 7f6cecce0700 1 -- 172.21.15.38:0/2520119920 >> 172.21.15.38:0/2520119920 conn(0x7f6ce80fa3b0 msgr2=0x7f6ce8101590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:23.911 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:23.907+0000 7f6cecce0700 1 -- 172.21.15.38:0/2520119920 shutdown_connections 2023-03-09T14:44:23.911 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:23.907+0000 7f6cecce0700 1 -- 172.21.15.38:0/2520119920 wait complete. 2023-03-09T14:44:23.911 INFO:teuthology.orchestra.run.smithi038.stderr:added data pool 4 to fsmap 2023-03-09T14:44:23.922 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-09T14:44:24.047 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.044+0000 7f3ceb63a700 1 -- 172.21.15.38:0/1464333272 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3ce41032c0 msgr2=0x7f3ce41056a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:24.047 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.044+0000 7f3ceb63a700 1 --2- 172.21.15.38:0/1464333272 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3ce41032c0 0x7f3ce41056a0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f3cd4009990 tx=0x7f3cd4009ca0 comp rx=0 tx=0).stop 2023-03-09T14:44:24.047 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.044+0000 7f3ceb63a700 1 -- 172.21.15.38:0/1464333272 shutdown_connections 2023-03-09T14:44:24.047 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.044+0000 7f3ceb63a700 1 --2- 172.21.15.38:0/1464333272 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3ce4105be0 0x7f3ce4107fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:24.047 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.044+0000 7f3ceb63a700 1 --2- 172.21.15.38:0/1464333272 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3ce41009a0 0x7f3ce4102d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:24.048 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.044+0000 7f3ceb63a700 1 --2- 172.21.15.38:0/1464333272 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3ce41032c0 0x7f3ce41056a0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:24.048 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.044+0000 7f3ceb63a700 1 -- 172.21.15.38:0/1464333272 >> 172.21.15.38:0/1464333272 conn(0x7f3ce40fa380 msgr2=0x7f3ce40fc7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:24.048 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.044+0000 7f3ceb63a700 1 -- 172.21.15.38:0/1464333272 shutdown_connections 2023-03-09T14:44:24.048 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.044+0000 7f3ceb63a700 1 -- 172.21.15.38:0/1464333272 wait complete. 2023-03-09T14:44:24.048 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.045+0000 7f3ceb63a700 1 Processor -- start 2023-03-09T14:44:24.048 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.045+0000 7f3ceb63a700 1 -- start start 2023-03-09T14:44:24.050 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.045+0000 7f3ceb63a700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3ce41009a0 0x7f3ce4195d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:24.050 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.045+0000 7f3ceb63a700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3ce41032c0 0x7f3ce41962b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:24.050 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.045+0000 7f3ceb63a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3ce4105be0 0x7f3ce419d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:24.050 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.045+0000 7f3ceb63a700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ce419d8f0 con 0x7f3ce4105be0 2023-03-09T14:44:24.050 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.045+0000 7f3ceb63a700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ce419da30 con 0x7f3ce41009a0 2023-03-09T14:44:24.051 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.045+0000 7f3ceb63a700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ce419dbc0 con 0x7f3ce41032c0 2023-03-09T14:44:24.051 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.045+0000 7f3ce9bd7700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3ce4105be0 0x7f3ce419d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:24.051 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.045+0000 7f3ce9bd7700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3ce4105be0 0x7f3ce419d320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36752/0 (socket says 172.21.15.38:36752) 2023-03-09T14:44:24.051 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.045+0000 7f3ce9bd7700 1 -- 172.21.15.38:0/2650883906 learned_addr learned my addr 172.21.15.38:0/2650883906 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:44:24.051 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.045+0000 7f3ce93d6700 1 --2- 172.21.15.38:0/2650883906 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3ce41009a0 0x7f3ce4195d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:24.051 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.045+0000 7f3ce8bd5700 1 --2- 172.21.15.38:0/2650883906 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3ce41032c0 0x7f3ce41962b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:24.051 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.045+0000 7f3ce9bd7700 1 -- 172.21.15.38:0/2650883906 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3ce41032c0 msgr2=0x7f3ce41962b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:24.051 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.045+0000 7f3ce9bd7700 1 --2- 172.21.15.38:0/2650883906 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3ce41032c0 0x7f3ce41962b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:24.052 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.045+0000 7f3ce9bd7700 1 -- 172.21.15.38:0/2650883906 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3ce41009a0 msgr2=0x7f3ce4195d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:24.052 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.045+0000 7f3ce9bd7700 1 --2- 172.21.15.38:0/2650883906 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3ce41009a0 0x7f3ce4195d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:24.052 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.045+0000 7f3ce9bd7700 1 -- 172.21.15.38:0/2650883906 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cd40095f0 con 0x7f3ce4105be0 2023-03-09T14:44:24.052 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.045+0000 7f3ce9bd7700 1 --2- 172.21.15.38:0/2650883906 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3ce4105be0 0x7f3ce419d320 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f3cdc00c900 tx=0x7f3cdc00cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:24.052 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.046+0000 7f3cda7fc700 1 -- 172.21.15.38:0/2650883906 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cdc007b00 con 0x7f3ce4105be0 2023-03-09T14:44:24.052 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.046+0000 7f3cda7fc700 1 -- 172.21.15.38:0/2650883906 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3cdc00ce50 con 0x7f3ce4105be0 2023-03-09T14:44:24.052 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.046+0000 7f3cda7fc700 1 -- 172.21.15.38:0/2650883906 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cdc018660 con 0x7f3ce4105be0 2023-03-09T14:44:24.053 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.046+0000 7f3ceb63a700 1 -- 172.21.15.38:0/2650883906 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ce410b7c0 con 0x7f3ce4105be0 2023-03-09T14:44:24.053 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.046+0000 7f3ceb63a700 1 -- 172.21.15.38:0/2650883906 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ce410b630 con 0x7f3ce4105be0 2023-03-09T14:44:24.053 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.046+0000 7f3ceb63a700 1 -- 172.21.15.38:0/2650883906 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ce40fbef0 con 0x7f3ce4105be0 2023-03-09T14:44:24.053 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.048+0000 7f3cda7fc700 1 -- 172.21.15.38:0/2650883906 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 133198+0+0 (secure 0 0 0) 0x7f3cdc00f450 con 0x7f3ce4105be0 2023-03-09T14:44:24.054 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.049+0000 7f3cda7fc700 1 --2- 172.21.15.38:0/2650883906 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f3cd009f440 0x7f3cd00a18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:24.054 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.049+0000 7f3ce93d6700 1 --2- 172.21.15.38:0/2650883906 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f3cd009f440 0x7f3cd00a18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:24.054 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.049+0000 7f3cda7fc700 1 -- 172.21.15.38:0/2650883906 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f3cdc0ca060 con 0x7f3ce4105be0 2023-03-09T14:44:24.054 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.049+0000 7f3ce93d6700 1 --2- 172.21.15.38:0/2650883906 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f3cd009f440 0x7f3cd00a18f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3ce0009550 tx=0x7f3ce00091b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:24.054 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.052+0000 7f3cda7fc700 1 -- 172.21.15.38:0/2650883906 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f3cdc094250 con 0x7f3ce4105be0 2023-03-09T14:44:24.298 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.295+0000 7f3ceb63a700 1 -- 172.21.15.38:0/2650883906 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3ce402cef0 con 0x7f3ce4105be0 2023-03-09T14:44:24.298 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.296+0000 7f3cda7fc700 1 -- 172.21.15.38:0/2650883906 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+10741 (secure 0 0 0) 0x7f3cdc097870 con 0x7f3ce4105be0 2023-03-09T14:44:24.299 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:44:24.299 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/768918602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":768918602},{"type":"v1","addr":"172.21.15.38:6835","nonce":768918602}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/18381540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":18381540},{"type":"v1","addr":"172.21.15.38:6837","nonce":18381540}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14610,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6839/4139020066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":4139020066},{"type":"v1","addr":"172.21.15.38:6839","nonce":4139020066}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14622,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6841/1397115322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":1397115322},{"type":"v1","addr":"172.21.15.38:6841","nonce":1397115322}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6833/415305571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":415305571},{"type":"v1","addr":"172.21.15.181:6833","nonce":415305571}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/659300520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":659300520},{"type":"v1","addr":"172.21.15.122:6833","nonce":659300520}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24448,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/2576255861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":2576255861},{"type":"v1","addr":"172.21.15.181:6835","nonce":2576255861}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24460,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2942560181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":2942560181},{"type":"v1","addr":"172.21.15.181:6837","nonce":2942560181}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/3675537941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":3675537941},{"type":"v1","addr":"172.21.15.122:6835","nonce":3675537941}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24478,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6839/3902263660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":3902263660},{"type":"v1","addr":"172.21.15.181:6839","nonce":3902263660}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24494,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/2210214596","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":2210214596},{"type":"v1","addr":"172.21.15.122:6837","nonce":2210214596}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-09T14:44:19.083770+0000","modified":"2023-03-09T14:44:23.900464+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24530},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24530":{"gid":24530,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6839/971050165","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6838","nonce":971050165},{"type":"v1","addr":"172.21.15.122:6839","nonce":971050165}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2023-03-09T14:44:24.301 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.298+0000 7f3ceb63a700 1 -- 172.21.15.38:0/2650883906 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f3cd009f440 msgr2=0x7f3cd00a18f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:24.301 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.298+0000 7f3ceb63a700 1 --2- 172.21.15.38:0/2650883906 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f3cd009f440 0x7f3cd00a18f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3ce0009550 tx=0x7f3ce00091b0 comp rx=0 tx=0).stop 2023-03-09T14:44:24.301 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.298+0000 7f3ceb63a700 1 -- 172.21.15.38:0/2650883906 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3ce4105be0 msgr2=0x7f3ce419d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:24.301 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.298+0000 7f3ceb63a700 1 --2- 172.21.15.38:0/2650883906 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3ce4105be0 0x7f3ce419d320 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f3cdc00c900 tx=0x7f3cdc00cc10 comp rx=0 tx=0).stop 2023-03-09T14:44:24.301 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.298+0000 7f3ceb63a700 1 -- 172.21.15.38:0/2650883906 shutdown_connections 2023-03-09T14:44:24.301 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.298+0000 7f3ceb63a700 1 --2- 172.21.15.38:0/2650883906 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3ce4105be0 0x7f3ce419d320 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:24.302 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.298+0000 7f3ceb63a700 1 --2- 172.21.15.38:0/2650883906 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f3cd009f440 0x7f3cd00a18f0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:24.302 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.298+0000 7f3ceb63a700 1 --2- 172.21.15.38:0/2650883906 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3ce41009a0 0x7f3ce4195d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:24.302 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.298+0000 7f3ceb63a700 1 --2- 172.21.15.38:0/2650883906 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3ce41032c0 0x7f3ce41962b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:24.302 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.299+0000 7f3ceb63a700 1 -- 172.21.15.38:0/2650883906 >> 172.21.15.38:0/2650883906 conn(0x7f3ce40fa380 msgr2=0x7f3ce4101600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:24.302 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.299+0000 7f3ceb63a700 1 -- 172.21.15.38:0/2650883906 shutdown_connections 2023-03-09T14:44:24.302 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.299+0000 7f3ceb63a700 1 -- 172.21.15.38:0/2650883906 wait complete. 2023-03-09T14:44:24.303 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 17 2023-03-09T14:44:24.313 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-09T14:44:24.440 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.436+0000 7f5a35d3d700 1 -- 172.21.15.38:0/1525791469 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5a300fff80 msgr2=0x7f5a301003d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:24.441 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.436+0000 7f5a35d3d700 1 --2- 172.21.15.38:0/1525791469 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5a300fff80 0x7f5a301003d0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f5a20009990 tx=0x7f5a20009ca0 comp rx=0 tx=0).stop 2023-03-09T14:44:24.441 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.436+0000 7f5a35d3d700 1 -- 172.21.15.38:0/1525791469 shutdown_connections 2023-03-09T14:44:24.441 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.437+0000 7f5a35d3d700 1 --2- 172.21.15.38:0/1525791469 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5a30100910 0x7f5a301071d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:24.441 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.437+0000 7f5a35d3d700 1 --2- 172.21.15.38:0/1525791469 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5a300fed90 0x7f5a300ff1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:24.441 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.437+0000 7f5a35d3d700 1 --2- 172.21.15.38:0/1525791469 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5a300fff80 0x7f5a301003d0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:24.442 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.437+0000 7f5a35d3d700 1 -- 172.21.15.38:0/1525791469 >> 172.21.15.38:0/1525791469 conn(0x7f5a300fa370 msgr2=0x7f5a300fc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:24.442 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.437+0000 7f5a35d3d700 1 -- 172.21.15.38:0/1525791469 shutdown_connections 2023-03-09T14:44:24.442 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.437+0000 7f5a35d3d700 1 -- 172.21.15.38:0/1525791469 wait complete. 2023-03-09T14:44:24.442 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.437+0000 7f5a35d3d700 1 Processor -- start 2023-03-09T14:44:24.442 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.437+0000 7f5a35d3d700 1 -- start start 2023-03-09T14:44:24.442 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.437+0000 7f5a35d3d700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5a300fed90 0x7f5a30194d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:24.442 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.437+0000 7f5a35d3d700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5a300fff80 0x7f5a301952b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:24.442 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.438+0000 7f5a35d3d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5a30100910 0x7f5a3019c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:24.443 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.438+0000 7f5a35d3d700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a3019c8f0 con 0x7f5a30100910 2023-03-09T14:44:24.443 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.438+0000 7f5a35d3d700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a3019ca30 con 0x7f5a300fff80 2023-03-09T14:44:24.443 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.438+0000 7f5a35d3d700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a3019cbc0 con 0x7f5a300fed90 2023-03-09T14:44:24.443 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.438+0000 7f5a2ffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5a30100910 0x7f5a3019c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:24.443 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.438+0000 7f5a2ffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5a30100910 0x7f5a3019c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36766/0 (socket says 172.21.15.38:36766) 2023-03-09T14:44:24.443 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.438+0000 7f5a2ffff700 1 -- 172.21.15.38:0/530301329 learned_addr learned my addr 172.21.15.38:0/530301329 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:44:24.443 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.438+0000 7f5a2effd700 1 --2- 172.21.15.38:0/530301329 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5a300fff80 0x7f5a301952b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:24.444 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.438+0000 7f5a2f7fe700 1 --2- 172.21.15.38:0/530301329 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5a300fed90 0x7f5a30194d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:24.444 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.438+0000 7f5a2ffff700 1 -- 172.21.15.38:0/530301329 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5a300fed90 msgr2=0x7f5a30194d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:24.444 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.438+0000 7f5a2ffff700 1 --2- 172.21.15.38:0/530301329 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5a300fed90 0x7f5a30194d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:24.444 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.438+0000 7f5a2ffff700 1 -- 172.21.15.38:0/530301329 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5a300fff80 msgr2=0x7f5a301952b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:24.444 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.438+0000 7f5a2ffff700 1 --2- 172.21.15.38:0/530301329 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5a300fff80 0x7f5a301952b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:24.444 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.438+0000 7f5a2ffff700 1 -- 172.21.15.38:0/530301329 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a200095f0 con 0x7f5a30100910 2023-03-09T14:44:24.444 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.438+0000 7f5a2f7fe700 1 --2- 172.21.15.38:0/530301329 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5a300fed90 0x7f5a30194d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:44:24.444 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.438+0000 7f5a2ffff700 1 --2- 172.21.15.38:0/530301329 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5a30100910 0x7f5a3019c320 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f5a2800b970 tx=0x7f5a2800bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:24.445 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.438+0000 7f5a2cff9700 1 -- 172.21.15.38:0/530301329 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a2800c930 con 0x7f5a30100910 2023-03-09T14:44:24.445 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.438+0000 7f5a2cff9700 1 -- 172.21.15.38:0/530301329 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5a28014440 con 0x7f5a30100910 2023-03-09T14:44:24.445 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.438+0000 7f5a35d3d700 1 -- 172.21.15.38:0/530301329 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a3010a770 con 0x7f5a30100910 2023-03-09T14:44:24.445 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.438+0000 7f5a2cff9700 1 -- 172.21.15.38:0/530301329 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a280075d0 con 0x7f5a30100910 2023-03-09T14:44:24.445 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.438+0000 7f5a35d3d700 1 -- 172.21.15.38:0/530301329 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a3010a5e0 con 0x7f5a30100910 2023-03-09T14:44:24.446 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.439+0000 7f5a35d3d700 1 -- 172.21.15.38:0/530301329 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a30106070 con 0x7f5a30100910 2023-03-09T14:44:24.446 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.441+0000 7f5a2cff9700 1 -- 172.21.15.38:0/530301329 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 133198+0+0 (secure 0 0 0) 0x7f5a2801b030 con 0x7f5a30100910 2023-03-09T14:44:24.446 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.441+0000 7f5a2cff9700 1 --2- 172.21.15.38:0/530301329 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f5a1809f440 0x7f5a180a18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:24.446 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.442+0000 7f5a2f7fe700 1 --2- 172.21.15.38:0/530301329 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f5a1809f440 0x7f5a180a18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:24.447 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.442+0000 7f5a2cff9700 1 -- 172.21.15.38:0/530301329 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f5a28006e40 con 0x7f5a30100910 2023-03-09T14:44:24.447 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.442+0000 7f5a2f7fe700 1 --2- 172.21.15.38:0/530301329 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f5a1809f440 0x7f5a180a18f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f5a300ffde0 tx=0x7f5a24005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:24.447 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.445+0000 7f5a2cff9700 1 -- 172.21.15.38:0/530301329 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f5a28093b20 con 0x7f5a30100910 2023-03-09T14:44:24.648 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.645+0000 7f5a35d3d700 1 -- 172.21.15.38:0/530301329 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5a300617a0 con 0x7f5a30100910 2023-03-09T14:44:24.650 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.647+0000 7f5a2cff9700 1 -- 172.21.15.38:0/530301329 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+23973 (secure 0 0 0) 0x7f5a28019020 con 0x7f5a30100910 2023-03-09T14:44:24.650 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:44:24.651 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":87,"fsid":"c6733fe2-be87-11ed-9af2-001a4aab830c","created":"2023-03-09T14:37:14.947981+0000","modified":"2023-03-09T14:44:23.900228+0000","last_up_change":"2023-03-09T14:43:37.198407+0000","last_in_change":"2023-03-09T14:43:19.550566+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-09T14:40:23.196665+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-03-09T14:44:17.484374+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-03-09T14:44:18.457833+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-03-09T14:44:21.895589+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"51477039-15ee-4707-a34d-4ecfa18bebde","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6803","nonce":1508072290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6805","nonce":1508072290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6809","nonce":1508072290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6807","nonce":1508072290}]},"public_addr":"172.21.15.38:6803/1508072290","cluster_addr":"172.21.15.38:6805/1508072290","heartbeat_back_addr":"172.21.15.38:6809/1508072290","heartbeat_front_addr":"172.21.15.38:6807/1508072290","state":["exists","up"]},{"osd":1,"uuid":"71f8a718-b0a9-4f39-bc54-267aa42a2578","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6801","nonce":3929268773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6803","nonce":3929268773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6807","nonce":3929268773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6805","nonce":3929268773}]},"public_addr":"172.21.15.122:6801/3929268773","cluster_addr":"172.21.15.122:6803/3929268773","heartbeat_back_addr":"172.21.15.122:6807/3929268773","heartbeat_front_addr":"172.21.15.122:6805/3929268773","state":["exists","up"]},{"osd":2,"uuid":"961a2cbe-140e-437f-b131-1fcac573b19c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6801","nonce":1310630803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6803","nonce":1310630803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6807","nonce":1310630803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6805","nonce":1310630803}]},"public_addr":"172.21.15.181:6801/1310630803","cluster_addr":"172.21.15.181:6803/1310630803","heartbeat_back_addr":"172.21.15.181:6807/1310630803","heartbeat_front_addr":"172.21.15.181:6805/1310630803","state":["exists","up"]},{"osd":3,"uuid":"969df8fa-2d19-42c7-9f2a-5c5fac15d4a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6811","nonce":2719814100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6813","nonce":2719814100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6817","nonce":2719814100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6815","nonce":2719814100}]},"public_addr":"172.21.15.38:6811/2719814100","cluster_addr":"172.21.15.38:6813/2719814100","heartbeat_back_addr":"172.21.15.38:6817/2719814100","heartbeat_front_addr":"172.21.15.38:6815/2719814100","state":["exists","up"]},{"osd":4,"uuid":"44a6dce3-e06e-4cbd-9404-62a86d01ab6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6809","nonce":2480764994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6811","nonce":2480764994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6815","nonce":2480764994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6813","nonce":2480764994}]},"public_addr":"172.21.15.122:6809/2480764994","cluster_addr":"172.21.15.122:6811/2480764994","heartbeat_back_addr":"172.21.15.122:6815/2480764994","heartbeat_front_addr":"172.21.15.122:6813/2480764994","state":["exists","up"]},{"osd":5,"uuid":"49749574-c00c-4e3a-80e6-e3686df58d5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6809","nonce":3141346246}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6811","nonce":3141346246}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6815","nonce":3141346246}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6813","nonce":3141346246}]},"public_addr":"172.21.15.181:6809/3141346246","cluster_addr":"172.21.15.181:6811/3141346246","heartbeat_back_addr":"172.21.15.181:6815/3141346246","heartbeat_front_addr":"172.21.15.181:6813/3141346246","state":["exists","up"]},{"osd":6,"uuid":"61c0437e-3f51-45a9-8a61-e98eea0093f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6819","nonce":2122788957}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6821","nonce":2122788957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6825","nonce":2122788957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6823","nonce":2122788957}]},"public_addr":"172.21.15.38:6819/2122788957","cluster_addr":"172.21.15.38:6821/2122788957","heartbeat_back_addr":"172.21.15.38:6825/2122788957","heartbeat_front_addr":"172.21.15.38:6823/2122788957","state":["exists","up"]},{"osd":7,"uuid":"231fecdd-939f-4c7b-83a1-c4a1cffd7139","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6817","nonce":656874139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6819","nonce":656874139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6823","nonce":656874139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6821","nonce":656874139}]},"public_addr":"172.21.15.122:6817/656874139","cluster_addr":"172.21.15.122:6819/656874139","heartbeat_back_addr":"172.21.15.122:6823/656874139","heartbeat_front_addr":"172.21.15.122:6821/656874139","state":["exists","up"]},{"osd":8,"uuid":"297e6d6d-85fd-43ce-9028-5e9dbe55c36e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6817","nonce":755000703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6819","nonce":755000703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6823","nonce":755000703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6821","nonce":755000703}]},"public_addr":"172.21.15.181:6817/755000703","cluster_addr":"172.21.15.181:6819/755000703","heartbeat_back_addr":"172.21.15.181:6823/755000703","heartbeat_front_addr":"172.21.15.181:6821/755000703","state":["exists","up"]},{"osd":9,"uuid":"94cc0355-5a1a-4cfc-bd0c-562b6b41127e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6827","nonce":2624555969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6829","nonce":2624555969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6833","nonce":2624555969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6831","nonce":2624555969}]},"public_addr":"172.21.15.38:6827/2624555969","cluster_addr":"172.21.15.38:6829/2624555969","heartbeat_back_addr":"172.21.15.38:6833/2624555969","heartbeat_front_addr":"172.21.15.38:6831/2624555969","state":["exists","up"]},{"osd":10,"uuid":"f4f2e042-8d92-46bf-93b8-8b0f04795c2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6825","nonce":1427785280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6827","nonce":1427785280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6831","nonce":1427785280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6829","nonce":1427785280}]},"public_addr":"172.21.15.122:6825/1427785280","cluster_addr":"172.21.15.122:6827/1427785280","heartbeat_back_addr":"172.21.15.122:6831/1427785280","heartbeat_front_addr":"172.21.15.122:6829/1427785280","state":["exists","up"]},{"osd":11,"uuid":"07444936-eae5-4097-a799-60294cafe80b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6825","nonce":3183488640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6827","nonce":3183488640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6831","nonce":3183488640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6829","nonce":3183488640}]},"public_addr":"172.21.15.181:6825/3183488640","cluster_addr":"172.21.15.181:6827/3183488640","heartbeat_back_addr":"172.21.15.181:6831/3183488640","heartbeat_front_addr":"172.21.15.181:6829/3183488640","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:39:37.657556+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:39:58.960074+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:40:20.326952+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:40:42.244387+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:41:03.946973+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:41:25.097985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:41:46.877916+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:42:08.413827+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:42:29.822009+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:42:51.776684+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:43:13.937249+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:43:35.569199+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/4272288735":"2023-03-10T14:38:14.612767+0000","172.21.15.38:6801/3381625669":"2023-03-10T14:38:14.612767+0000","172.21.15.38:6800/3381625669":"2023-03-10T14:38:14.612767+0000","172.21.15.38:6800/3060474342":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/4253918615":"2023-03-10T14:38:14.612767+0000","172.21.15.38:0/3496083181":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/245602368":"2023-03-10T14:37:47.517775+0000","172.21.15.38:6801/3060474342":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/3622589182":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/1430302251":"2023-03-10T14:38:14.612767+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-09T14:44:24.652 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.649+0000 7f5a35d3d700 1 -- 172.21.15.38:0/530301329 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f5a1809f440 msgr2=0x7f5a180a18f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:24.652 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.649+0000 7f5a35d3d700 1 --2- 172.21.15.38:0/530301329 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f5a1809f440 0x7f5a180a18f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f5a300ffde0 tx=0x7f5a24005f90 comp rx=0 tx=0).stop 2023-03-09T14:44:24.653 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.649+0000 7f5a35d3d700 1 -- 172.21.15.38:0/530301329 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5a30100910 msgr2=0x7f5a3019c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:24.653 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.649+0000 7f5a35d3d700 1 --2- 172.21.15.38:0/530301329 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5a30100910 0x7f5a3019c320 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f5a2800b970 tx=0x7f5a2800bc80 comp rx=0 tx=0).stop 2023-03-09T14:44:24.653 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.649+0000 7f5a35d3d700 1 -- 172.21.15.38:0/530301329 shutdown_connections 2023-03-09T14:44:24.653 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.649+0000 7f5a35d3d700 1 --2- 172.21.15.38:0/530301329 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5a30100910 0x7f5a3019c320 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:24.653 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.649+0000 7f5a35d3d700 1 --2- 172.21.15.38:0/530301329 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f5a1809f440 0x7f5a180a18f0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:24.653 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.649+0000 7f5a35d3d700 1 --2- 172.21.15.38:0/530301329 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5a300fed90 0x7f5a30194d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:24.653 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.649+0000 7f5a35d3d700 1 --2- 172.21.15.38:0/530301329 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5a300fff80 0x7f5a301952b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:24.654 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.649+0000 7f5a35d3d700 1 -- 172.21.15.38:0/530301329 >> 172.21.15.38:0/530301329 conn(0x7f5a300fa370 msgr2=0x7f5a301031b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:24.654 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.649+0000 7f5a35d3d700 1 -- 172.21.15.38:0/530301329 shutdown_connections 2023-03-09T14:44:24.654 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.649+0000 7f5a35d3d700 1 -- 172.21.15.38:0/530301329 wait complete. 2023-03-09T14:44:24.664 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:24 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/3220650337' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2023-03-09T14:44:24.664 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:24 smithi038 ceph-mon[92422]: osdmap e86: 12 total, 12 up, 12 in 2023-03-09T14:44:24.664 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:24 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/2520119920' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2023-03-09T14:44:24.664 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:24 smithi038 ceph-mon[92422]: osdmap e87: 12 total, 12 up, 12 in 2023-03-09T14:44:24.665 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:24 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/2520119920' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2023-03-09T14:44:24.665 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:24 smithi038 ceph-mon[92422]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-03-09T14:44:24.665 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:24 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/2650883906' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-03-09T14:44:24.666 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-09T14:44:24.728 INFO:journalctl@ceph.mds.b.smithi122.stdout:Mar 09 14:44:24 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mds-b[119145]: 2023-03-09T14:44:24.304+0000 7fbfafa1a700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-03-09T14:44:24.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:24 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/3220650337' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2023-03-09T14:44:24.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:24 smithi122 ceph-mon[93869]: osdmap e86: 12 total, 12 up, 12 in 2023-03-09T14:44:24.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:24 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/2520119920' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2023-03-09T14:44:24.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:24 smithi122 ceph-mon[93869]: osdmap e87: 12 total, 12 up, 12 in 2023-03-09T14:44:24.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:24 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/2520119920' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2023-03-09T14:44:24.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:24 smithi122 ceph-mon[93869]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-03-09T14:44:24.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:24 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/2650883906' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-03-09T14:44:24.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:24 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/3220650337' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2023-03-09T14:44:24.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:24 smithi181 ceph-mon[94189]: osdmap e86: 12 total, 12 up, 12 in 2023-03-09T14:44:24.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:24 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/2520119920' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2023-03-09T14:44:24.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:24 smithi181 ceph-mon[94189]: osdmap e87: 12 total, 12 up, 12 in 2023-03-09T14:44:24.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:24 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/2520119920' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2023-03-09T14:44:24.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:24 smithi181 ceph-mon[94189]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-03-09T14:44:24.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:24 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/2650883906' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-03-09T14:44:24.792 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.789+0000 7f870f9a1700 1 -- 172.21.15.38:0/2925238859 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8708100900 msgr2=0x7f87081071c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:24.793 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.789+0000 7f870f9a1700 1 --2- 172.21.15.38:0/2925238859 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8708100900 0x7f87081071c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f870000b0a0 tx=0x7f870000b3b0 comp rx=0 tx=0).stop 2023-03-09T14:44:24.793 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.789+0000 7f870f9a1700 1 -- 172.21.15.38:0/2925238859 shutdown_connections 2023-03-09T14:44:24.793 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.789+0000 7f870f9a1700 1 --2- 172.21.15.38:0/2925238859 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8708100900 0x7f87081071c0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:24.793 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.789+0000 7f870f9a1700 1 --2- 172.21.15.38:0/2925238859 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f87080fed80 0x7f87080ff190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:24.793 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.789+0000 7f870f9a1700 1 --2- 172.21.15.38:0/2925238859 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87080fff70 0x7f87081003c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:24.794 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.789+0000 7f870f9a1700 1 -- 172.21.15.38:0/2925238859 >> 172.21.15.38:0/2925238859 conn(0x7f87080fa380 msgr2=0x7f87080fc7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:24.794 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.789+0000 7f870f9a1700 1 -- 172.21.15.38:0/2925238859 shutdown_connections 2023-03-09T14:44:24.794 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.789+0000 7f870f9a1700 1 -- 172.21.15.38:0/2925238859 wait complete. 2023-03-09T14:44:24.794 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.790+0000 7f870f9a1700 1 Processor -- start 2023-03-09T14:44:24.794 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.790+0000 7f870f9a1700 1 -- start start 2023-03-09T14:44:24.796 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.790+0000 7f870f9a1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f87080fed80 0x7f87080685b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:24.796 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.790+0000 7f870f9a1700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87080fff70 0x7f870819a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:24.796 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.790+0000 7f870f9a1700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8708100900 0x7f870819d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:24.796 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.790+0000 7f870f9a1700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f870819d7c0 con 0x7f87080fed80 2023-03-09T14:44:24.796 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.790+0000 7f870f9a1700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f870819d930 con 0x7f87080fff70 2023-03-09T14:44:24.796 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.790+0000 7f870f9a1700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f870819daf0 con 0x7f8708100900 2023-03-09T14:44:24.797 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.790+0000 7f870d73d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f87080fed80 0x7f87080685b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:24.797 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.790+0000 7f870d73d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f87080fed80 0x7f87080685b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36776/0 (socket says 172.21.15.38:36776) 2023-03-09T14:44:24.797 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.790+0000 7f870d73d700 1 -- 172.21.15.38:0/2139050813 learned_addr learned my addr 172.21.15.38:0/2139050813 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:44:24.797 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.790+0000 7f870df3e700 1 --2- 172.21.15.38:0/2139050813 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8708100900 0x7f870819d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:24.797 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.790+0000 7f870cf3c700 1 --2- 172.21.15.38:0/2139050813 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87080fff70 0x7f870819a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:24.797 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.790+0000 7f870d73d700 1 -- 172.21.15.38:0/2139050813 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8708100900 msgr2=0x7f870819d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:24.797 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.790+0000 7f870d73d700 1 --2- 172.21.15.38:0/2139050813 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8708100900 0x7f870819d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:24.798 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.790+0000 7f870d73d700 1 -- 172.21.15.38:0/2139050813 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87080fff70 msgr2=0x7f870819a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:24.798 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.790+0000 7f870d73d700 1 --2- 172.21.15.38:0/2139050813 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87080fff70 0x7f870819a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:24.798 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.790+0000 7f870d73d700 1 -- 172.21.15.38:0/2139050813 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8700009d00 con 0x7f87080fed80 2023-03-09T14:44:24.798 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.790+0000 7f870cf3c700 1 --2- 172.21.15.38:0/2139050813 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87080fff70 0x7f870819a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-09T14:44:24.798 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.790+0000 7f870d73d700 1 --2- 172.21.15.38:0/2139050813 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f87080fed80 0x7f87080685b0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f86f800c990 tx=0x7f86f800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:24.798 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.791+0000 7f86fe7fc700 1 -- 172.21.15.38:0/2139050813 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86f8007b00 con 0x7f87080fed80 2023-03-09T14:44:24.798 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.791+0000 7f870f9a1700 1 -- 172.21.15.38:0/2139050813 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f870810a740 con 0x7f87080fed80 2023-03-09T14:44:24.799 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.791+0000 7f86fe7fc700 1 -- 172.21.15.38:0/2139050813 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f86f8007c60 con 0x7f87080fed80 2023-03-09T14:44:24.799 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.791+0000 7f86fe7fc700 1 -- 172.21.15.38:0/2139050813 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86f8018660 con 0x7f87080fed80 2023-03-09T14:44:24.799 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.791+0000 7f870f9a1700 1 -- 172.21.15.38:0/2139050813 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f870810a5b0 con 0x7f87080fed80 2023-03-09T14:44:24.799 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.792+0000 7f870f9a1700 1 -- 172.21.15.38:0/2139050813 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8708105b70 con 0x7f87080fed80 2023-03-09T14:44:24.799 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.793+0000 7f86fe7fc700 1 -- 172.21.15.38:0/2139050813 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 133198+0+0 (secure 0 0 0) 0x7f86f8004020 con 0x7f87080fed80 2023-03-09T14:44:24.800 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.793+0000 7f86fe7fc700 1 --2- 172.21.15.38:0/2139050813 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f86f409f3c0 0x7f86f40a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:24.800 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.793+0000 7f870cf3c700 1 --2- 172.21.15.38:0/2139050813 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f86f409f3c0 0x7f86f40a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:24.800 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.793+0000 7f86fe7fc700 1 -- 172.21.15.38:0/2139050813 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f86f80ca2d0 con 0x7f87080fed80 2023-03-09T14:44:24.800 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.794+0000 7f870cf3c700 1 --2- 172.21.15.38:0/2139050813 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f86f409f3c0 0x7f86f40a1870 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8704005fd0 tx=0x7f8704005dc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:24.800 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.797+0000 7f86fe7fc700 1 -- 172.21.15.38:0/2139050813 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f86f80944d0 con 0x7f87080fed80 2023-03-09T14:44:25.000 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.997+0000 7f870f9a1700 1 -- 172.21.15.38:0/2139050813 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f870802cef0 con 0x7f87080fed80 2023-03-09T14:44:25.002 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:24.999+0000 7f86fe7fc700 1 -- 172.21.15.38:0/2139050813 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+23973 (secure 0 0 0) 0x7f86f8097af0 con 0x7f87080fed80 2023-03-09T14:44:25.002 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:44:25.003 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":87,"fsid":"c6733fe2-be87-11ed-9af2-001a4aab830c","created":"2023-03-09T14:37:14.947981+0000","modified":"2023-03-09T14:44:23.900228+0000","last_up_change":"2023-03-09T14:43:37.198407+0000","last_in_change":"2023-03-09T14:43:19.550566+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-09T14:40:23.196665+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-03-09T14:44:17.484374+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-03-09T14:44:18.457833+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-03-09T14:44:21.895589+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"51477039-15ee-4707-a34d-4ecfa18bebde","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6803","nonce":1508072290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6805","nonce":1508072290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6809","nonce":1508072290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6807","nonce":1508072290}]},"public_addr":"172.21.15.38:6803/1508072290","cluster_addr":"172.21.15.38:6805/1508072290","heartbeat_back_addr":"172.21.15.38:6809/1508072290","heartbeat_front_addr":"172.21.15.38:6807/1508072290","state":["exists","up"]},{"osd":1,"uuid":"71f8a718-b0a9-4f39-bc54-267aa42a2578","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6801","nonce":3929268773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6803","nonce":3929268773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6807","nonce":3929268773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6805","nonce":3929268773}]},"public_addr":"172.21.15.122:6801/3929268773","cluster_addr":"172.21.15.122:6803/3929268773","heartbeat_back_addr":"172.21.15.122:6807/3929268773","heartbeat_front_addr":"172.21.15.122:6805/3929268773","state":["exists","up"]},{"osd":2,"uuid":"961a2cbe-140e-437f-b131-1fcac573b19c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6801","nonce":1310630803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6803","nonce":1310630803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6807","nonce":1310630803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6805","nonce":1310630803}]},"public_addr":"172.21.15.181:6801/1310630803","cluster_addr":"172.21.15.181:6803/1310630803","heartbeat_back_addr":"172.21.15.181:6807/1310630803","heartbeat_front_addr":"172.21.15.181:6805/1310630803","state":["exists","up"]},{"osd":3,"uuid":"969df8fa-2d19-42c7-9f2a-5c5fac15d4a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6811","nonce":2719814100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6813","nonce":2719814100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6817","nonce":2719814100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6815","nonce":2719814100}]},"public_addr":"172.21.15.38:6811/2719814100","cluster_addr":"172.21.15.38:6813/2719814100","heartbeat_back_addr":"172.21.15.38:6817/2719814100","heartbeat_front_addr":"172.21.15.38:6815/2719814100","state":["exists","up"]},{"osd":4,"uuid":"44a6dce3-e06e-4cbd-9404-62a86d01ab6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6809","nonce":2480764994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6811","nonce":2480764994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6815","nonce":2480764994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6813","nonce":2480764994}]},"public_addr":"172.21.15.122:6809/2480764994","cluster_addr":"172.21.15.122:6811/2480764994","heartbeat_back_addr":"172.21.15.122:6815/2480764994","heartbeat_front_addr":"172.21.15.122:6813/2480764994","state":["exists","up"]},{"osd":5,"uuid":"49749574-c00c-4e3a-80e6-e3686df58d5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6809","nonce":3141346246}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6811","nonce":3141346246}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6815","nonce":3141346246}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6813","nonce":3141346246}]},"public_addr":"172.21.15.181:6809/3141346246","cluster_addr":"172.21.15.181:6811/3141346246","heartbeat_back_addr":"172.21.15.181:6815/3141346246","heartbeat_front_addr":"172.21.15.181:6813/3141346246","state":["exists","up"]},{"osd":6,"uuid":"61c0437e-3f51-45a9-8a61-e98eea0093f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6819","nonce":2122788957}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6821","nonce":2122788957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6825","nonce":2122788957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6823","nonce":2122788957}]},"public_addr":"172.21.15.38:6819/2122788957","cluster_addr":"172.21.15.38:6821/2122788957","heartbeat_back_addr":"172.21.15.38:6825/2122788957","heartbeat_front_addr":"172.21.15.38:6823/2122788957","state":["exists","up"]},{"osd":7,"uuid":"231fecdd-939f-4c7b-83a1-c4a1cffd7139","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6817","nonce":656874139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6819","nonce":656874139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6823","nonce":656874139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6821","nonce":656874139}]},"public_addr":"172.21.15.122:6817/656874139","cluster_addr":"172.21.15.122:6819/656874139","heartbeat_back_addr":"172.21.15.122:6823/656874139","heartbeat_front_addr":"172.21.15.122:6821/656874139","state":["exists","up"]},{"osd":8,"uuid":"297e6d6d-85fd-43ce-9028-5e9dbe55c36e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6817","nonce":755000703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6819","nonce":755000703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6823","nonce":755000703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6821","nonce":755000703}]},"public_addr":"172.21.15.181:6817/755000703","cluster_addr":"172.21.15.181:6819/755000703","heartbeat_back_addr":"172.21.15.181:6823/755000703","heartbeat_front_addr":"172.21.15.181:6821/755000703","state":["exists","up"]},{"osd":9,"uuid":"94cc0355-5a1a-4cfc-bd0c-562b6b41127e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6827","nonce":2624555969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6829","nonce":2624555969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6833","nonce":2624555969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6831","nonce":2624555969}]},"public_addr":"172.21.15.38:6827/2624555969","cluster_addr":"172.21.15.38:6829/2624555969","heartbeat_back_addr":"172.21.15.38:6833/2624555969","heartbeat_front_addr":"172.21.15.38:6831/2624555969","state":["exists","up"]},{"osd":10,"uuid":"f4f2e042-8d92-46bf-93b8-8b0f04795c2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6825","nonce":1427785280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6827","nonce":1427785280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6831","nonce":1427785280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6829","nonce":1427785280}]},"public_addr":"172.21.15.122:6825/1427785280","cluster_addr":"172.21.15.122:6827/1427785280","heartbeat_back_addr":"172.21.15.122:6831/1427785280","heartbeat_front_addr":"172.21.15.122:6829/1427785280","state":["exists","up"]},{"osd":11,"uuid":"07444936-eae5-4097-a799-60294cafe80b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6825","nonce":3183488640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6827","nonce":3183488640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6831","nonce":3183488640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6829","nonce":3183488640}]},"public_addr":"172.21.15.181:6825/3183488640","cluster_addr":"172.21.15.181:6827/3183488640","heartbeat_back_addr":"172.21.15.181:6831/3183488640","heartbeat_front_addr":"172.21.15.181:6829/3183488640","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:39:37.657556+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:39:58.960074+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:40:20.326952+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:40:42.244387+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:41:03.946973+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:41:25.097985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:41:46.877916+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:42:08.413827+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:42:29.822009+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:42:51.776684+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:43:13.937249+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:43:35.569199+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/4272288735":"2023-03-10T14:38:14.612767+0000","172.21.15.38:6801/3381625669":"2023-03-10T14:38:14.612767+0000","172.21.15.38:6800/3381625669":"2023-03-10T14:38:14.612767+0000","172.21.15.38:6800/3060474342":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/4253918615":"2023-03-10T14:38:14.612767+0000","172.21.15.38:0/3496083181":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/245602368":"2023-03-10T14:37:47.517775+0000","172.21.15.38:6801/3060474342":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/3622589182":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/1430302251":"2023-03-10T14:38:14.612767+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-09T14:44:25.004 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.001+0000 7f870f9a1700 1 -- 172.21.15.38:0/2139050813 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f86f409f3c0 msgr2=0x7f86f40a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:25.005 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.001+0000 7f870f9a1700 1 --2- 172.21.15.38:0/2139050813 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f86f409f3c0 0x7f86f40a1870 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f8704005fd0 tx=0x7f8704005dc0 comp rx=0 tx=0).stop 2023-03-09T14:44:25.005 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.001+0000 7f870f9a1700 1 -- 172.21.15.38:0/2139050813 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f87080fed80 msgr2=0x7f87080685b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:25.005 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.001+0000 7f870f9a1700 1 --2- 172.21.15.38:0/2139050813 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f87080fed80 0x7f87080685b0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f86f800c990 tx=0x7f86f800cd50 comp rx=0 tx=0).stop 2023-03-09T14:44:25.005 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.001+0000 7f870f9a1700 1 -- 172.21.15.38:0/2139050813 shutdown_connections 2023-03-09T14:44:25.005 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.001+0000 7f870f9a1700 1 --2- 172.21.15.38:0/2139050813 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f87080fed80 0x7f87080685b0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:25.005 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.001+0000 7f870f9a1700 1 --2- 172.21.15.38:0/2139050813 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f86f409f3c0 0x7f86f40a1870 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:25.006 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.001+0000 7f870f9a1700 1 --2- 172.21.15.38:0/2139050813 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8708100900 0x7f870819d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:25.006 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.001+0000 7f870f9a1700 1 --2- 172.21.15.38:0/2139050813 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f87080fff70 0x7f870819a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:25.006 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.001+0000 7f870f9a1700 1 -- 172.21.15.38:0/2139050813 >> 172.21.15.38:0/2139050813 conn(0x7f87080fa380 msgr2=0x7f87080fc520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:25.006 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.001+0000 7f870f9a1700 1 -- 172.21.15.38:0/2139050813 shutdown_connections 2023-03-09T14:44:25.006 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.001+0000 7f870f9a1700 1 -- 172.21.15.38:0/2139050813 wait complete. 2023-03-09T14:44:25.034 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-09T14:44:25.160 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.156+0000 7f149ee3d700 1 -- 172.21.15.38:0/965658650 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f14980fed80 msgr2=0x7f14980ff190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:25.160 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.156+0000 7f149ee3d700 1 --2- 172.21.15.38:0/965658650 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f14980fed80 0x7f14980ff190 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f148c009990 tx=0x7f148c009ca0 comp rx=0 tx=0).stop 2023-03-09T14:44:25.160 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.156+0000 7f149ee3d700 1 -- 172.21.15.38:0/965658650 shutdown_connections 2023-03-09T14:44:25.160 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.156+0000 7f149ee3d700 1 --2- 172.21.15.38:0/965658650 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f14980fed80 0x7f14980ff190 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:25.160 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.156+0000 7f149ee3d700 1 --2- 172.21.15.38:0/965658650 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1498100900 0x7f14981071c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:25.161 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.156+0000 7f149ee3d700 1 --2- 172.21.15.38:0/965658650 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f14980fff70 0x7f14981003c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:25.161 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.156+0000 7f149ee3d700 1 -- 172.21.15.38:0/965658650 >> 172.21.15.38:0/965658650 conn(0x7f14980fa380 msgr2=0x7f14980fc7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:25.161 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.156+0000 7f149ee3d700 1 -- 172.21.15.38:0/965658650 shutdown_connections 2023-03-09T14:44:25.161 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.156+0000 7f149ee3d700 1 -- 172.21.15.38:0/965658650 wait complete. 2023-03-09T14:44:25.162 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.157+0000 7f149ee3d700 1 Processor -- start 2023-03-09T14:44:25.163 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.157+0000 7f149ee3d700 1 -- start start 2023-03-09T14:44:25.163 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.157+0000 7f149ee3d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f14980fed80 0x7f14980685d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:25.163 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.157+0000 7f149ee3d700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f14980fff70 0x7f149819a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:25.163 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.157+0000 7f149ee3d700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1498100900 0x7f149819ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:25.163 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.157+0000 7f149ee3d700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f149819d660 con 0x7f14980fed80 2023-03-09T14:44:25.163 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.157+0000 7f149ee3d700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f149819d7d0 con 0x7f1498100900 2023-03-09T14:44:25.164 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.157+0000 7f149ee3d700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f149819d990 con 0x7f14980fff70 2023-03-09T14:44:25.164 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.157+0000 7f149cbd9700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f14980fed80 0x7f14980685d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:25.164 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.157+0000 7f149cbd9700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f14980fed80 0x7f14980685d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36790/0 (socket says 172.21.15.38:36790) 2023-03-09T14:44:25.164 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.157+0000 7f149cbd9700 1 -- 172.21.15.38:0/3978127944 learned_addr learned my addr 172.21.15.38:0/3978127944 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:44:25.164 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.157+0000 7f1497fff700 1 --2- 172.21.15.38:0/3978127944 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f14980fff70 0x7f149819a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:25.164 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.157+0000 7f149cbd9700 1 -- 172.21.15.38:0/3978127944 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f14980fff70 msgr2=0x7f149819a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:25.164 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.157+0000 7f149cbd9700 1 --2- 172.21.15.38:0/3978127944 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f14980fff70 0x7f149819a840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:25.165 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.157+0000 7f149cbd9700 1 -- 172.21.15.38:0/3978127944 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1498100900 msgr2=0x7f149819ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:25.165 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.157+0000 7f149cbd9700 1 --2- 172.21.15.38:0/3978127944 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1498100900 0x7f149819ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:25.165 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.157+0000 7f149cbd9700 1 -- 172.21.15.38:0/3978127944 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f148c0095f0 con 0x7f14980fed80 2023-03-09T14:44:25.165 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.158+0000 7f149cbd9700 1 --2- 172.21.15.38:0/3978127944 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f14980fed80 0x7f14980685d0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f148c000c00 tx=0x7f148c003940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:25.165 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.158+0000 7f1495ffb700 1 -- 172.21.15.38:0/3978127944 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f148c01d070 con 0x7f14980fed80 2023-03-09T14:44:25.165 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.158+0000 7f149ee3d700 1 -- 172.21.15.38:0/3978127944 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f149810a6e0 con 0x7f14980fed80 2023-03-09T14:44:25.165 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.158+0000 7f1495ffb700 1 -- 172.21.15.38:0/3978127944 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f148c022470 con 0x7f14980fed80 2023-03-09T14:44:25.165 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.158+0000 7f1495ffb700 1 -- 172.21.15.38:0/3978127944 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f148c00f6c0 con 0x7f14980fed80 2023-03-09T14:44:25.166 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.158+0000 7f149ee3d700 1 -- 172.21.15.38:0/3978127944 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f149810a550 con 0x7f14980fed80 2023-03-09T14:44:25.166 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.159+0000 7f149ee3d700 1 -- 172.21.15.38:0/3978127944 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1498105cb0 con 0x7f14980fed80 2023-03-09T14:44:25.166 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.160+0000 7f1495ffb700 1 -- 172.21.15.38:0/3978127944 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 133198+0+0 (secure 0 0 0) 0x7f148c029030 con 0x7f14980fed80 2023-03-09T14:44:25.167 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.161+0000 7f1495ffb700 1 --2- 172.21.15.38:0/3978127944 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f148809f3c0 0x7f14880a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:25.167 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.161+0000 7f1497fff700 1 --2- 172.21.15.38:0/3978127944 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f148809f3c0 0x7f14880a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:25.167 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.161+0000 7f1495ffb700 1 -- 172.21.15.38:0/3978127944 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f148c0cb400 con 0x7f14980fed80 2023-03-09T14:44:25.167 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.161+0000 7f1497fff700 1 --2- 172.21.15.38:0/3978127944 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f148809f3c0 0x7f14880a1870 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f1498068340 tx=0x7f1484005d50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:25.168 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.165+0000 7f1495ffb700 1 -- 172.21.15.38:0/3978127944 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f148c095600 con 0x7f14980fed80 2023-03-09T14:44:25.414 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.411+0000 7f149ee3d700 1 -- 172.21.15.38:0/3978127944 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f14980615c0 con 0x7f14980fed80 2023-03-09T14:44:25.415 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.412+0000 7f1495ffb700 1 -- 172.21.15.38:0/3978127944 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+10741 (secure 0 0 0) 0x7f148c027090 con 0x7f14980fed80 2023-03-09T14:44:25.415 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:44:25.415 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/768918602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":768918602},{"type":"v1","addr":"172.21.15.38:6835","nonce":768918602}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/18381540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":18381540},{"type":"v1","addr":"172.21.15.38:6837","nonce":18381540}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14610,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6839/4139020066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":4139020066},{"type":"v1","addr":"172.21.15.38:6839","nonce":4139020066}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14622,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6841/1397115322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":1397115322},{"type":"v1","addr":"172.21.15.38:6841","nonce":1397115322}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6833/415305571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":415305571},{"type":"v1","addr":"172.21.15.181:6833","nonce":415305571}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/659300520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":659300520},{"type":"v1","addr":"172.21.15.122:6833","nonce":659300520}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24448,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6835/2576255861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":2576255861},{"type":"v1","addr":"172.21.15.181:6835","nonce":2576255861}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24460,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6837/2942560181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":2942560181},{"type":"v1","addr":"172.21.15.181:6837","nonce":2942560181}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6835/3675537941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":3675537941},{"type":"v1","addr":"172.21.15.122:6835","nonce":3675537941}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24478,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6839/3902263660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":3902263660},{"type":"v1","addr":"172.21.15.181:6839","nonce":3902263660}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24494,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6837/2210214596","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":2210214596},{"type":"v1","addr":"172.21.15.122:6837","nonce":2210214596}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-03-09T14:44:19.083770+0000","modified":"2023-03-09T14:44:23.900464+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24530},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24530":{"gid":24530,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6839/971050165","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6838","nonce":971050165},{"type":"v1","addr":"172.21.15.122:6839","nonce":971050165}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2023-03-09T14:44:25.417 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.414+0000 7f149ee3d700 1 -- 172.21.15.38:0/3978127944 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f148809f3c0 msgr2=0x7f14880a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:25.417 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.414+0000 7f149ee3d700 1 --2- 172.21.15.38:0/3978127944 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f148809f3c0 0x7f14880a1870 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f1498068340 tx=0x7f1484005d50 comp rx=0 tx=0).stop 2023-03-09T14:44:25.418 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.414+0000 7f149ee3d700 1 -- 172.21.15.38:0/3978127944 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f14980fed80 msgr2=0x7f14980685d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:25.418 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.414+0000 7f149ee3d700 1 --2- 172.21.15.38:0/3978127944 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f14980fed80 0x7f14980685d0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f148c000c00 tx=0x7f148c003940 comp rx=0 tx=0).stop 2023-03-09T14:44:25.418 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.414+0000 7f149ee3d700 1 -- 172.21.15.38:0/3978127944 shutdown_connections 2023-03-09T14:44:25.418 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.414+0000 7f149ee3d700 1 --2- 172.21.15.38:0/3978127944 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f14980fed80 0x7f14980685d0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:25.418 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.414+0000 7f149ee3d700 1 --2- 172.21.15.38:0/3978127944 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f148809f3c0 0x7f14880a1870 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:25.418 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.414+0000 7f149ee3d700 1 --2- 172.21.15.38:0/3978127944 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1498100900 0x7f149819ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:25.419 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.414+0000 7f149ee3d700 1 --2- 172.21.15.38:0/3978127944 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f14980fff70 0x7f149819a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:25.419 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.414+0000 7f149ee3d700 1 -- 172.21.15.38:0/3978127944 >> 172.21.15.38:0/3978127944 conn(0x7f14980fa380 msgr2=0x7f14980fc610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:25.419 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.414+0000 7f149ee3d700 1 -- 172.21.15.38:0/3978127944 shutdown_connections 2023-03-09T14:44:25.419 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.414+0000 7f149ee3d700 1 -- 172.21.15.38:0/3978127944 wait complete. 2023-03-09T14:44:25.419 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 17 2023-03-09T14:44:25.430 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-03-09T14:44:19.083770+0000', 'modified': '2023-03-09T14:44:23.900464+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24530}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24530': {'gid': 24530, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6839/971050165', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6838', 'nonce': 971050165}, {'type': 'v1', 'addr': '172.21.15.122:6839', 'nonce': 971050165}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1} 2023-03-09T14:44:25.431 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-03-09T14:44:25.432 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-03-09T14:44:25.560 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.555+0000 7f8a9ab70700 1 -- 172.21.15.38:0/816950433 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8a94100920 msgr2=0x7f8a941071e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:25.561 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.555+0000 7f8a9ab70700 1 --2- 172.21.15.38:0/816950433 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8a94100920 0x7f8a941071e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c00b0a0 tx=0x7f8a8c00b3b0 comp rx=0 tx=0).stop 2023-03-09T14:44:25.561 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.555+0000 7f8a9ab70700 1 -- 172.21.15.38:0/816950433 shutdown_connections 2023-03-09T14:44:25.561 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.555+0000 7f8a9ab70700 1 --2- 172.21.15.38:0/816950433 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8a94100920 0x7f8a941071e0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:25.561 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.555+0000 7f8a9ab70700 1 --2- 172.21.15.38:0/816950433 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8a940feda0 0x7f8a940ff1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:25.561 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.555+0000 7f8a9ab70700 1 --2- 172.21.15.38:0/816950433 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a940fff90 0x7f8a941003e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:25.561 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.555+0000 7f8a9ab70700 1 -- 172.21.15.38:0/816950433 >> 172.21.15.38:0/816950433 conn(0x7f8a940fa3a0 msgr2=0x7f8a940fc7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:25.562 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.556+0000 7f8a9ab70700 1 -- 172.21.15.38:0/816950433 shutdown_connections 2023-03-09T14:44:25.562 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.556+0000 7f8a9ab70700 1 -- 172.21.15.38:0/816950433 wait complete. 2023-03-09T14:44:25.562 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.556+0000 7f8a9ab70700 1 Processor -- start 2023-03-09T14:44:25.562 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.556+0000 7f8a9ab70700 1 -- start start 2023-03-09T14:44:25.562 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.556+0000 7f8a9ab70700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8a940feda0 0x7f8a94195ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:25.562 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.556+0000 7f8a9ab70700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8a940fff90 0x7f8a94196400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:25.562 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.556+0000 7f8a9ab70700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a94100920 0x7f8a9419d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:25.562 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.556+0000 7f8a9ab70700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a9419da40 con 0x7f8a940feda0 2023-03-09T14:44:25.563 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.556+0000 7f8a9ab70700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a9419db80 con 0x7f8a940fff90 2023-03-09T14:44:25.563 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.556+0000 7f8a9ab70700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a9419dd10 con 0x7f8a94100920 2023-03-09T14:44:25.563 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.556+0000 7f8a9890c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8a940feda0 0x7f8a94195ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:25.563 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.556+0000 7f8a9890c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8a940feda0 0x7f8a94195ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36804/0 (socket says 172.21.15.38:36804) 2023-03-09T14:44:25.563 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.556+0000 7f8a9890c700 1 -- 172.21.15.38:0/653856306 learned_addr learned my addr 172.21.15.38:0/653856306 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:44:25.563 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.557+0000 7f8a93fff700 1 --2- 172.21.15.38:0/653856306 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8a940fff90 0x7f8a94196400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:25.563 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.557+0000 7f8a9910d700 1 --2- 172.21.15.38:0/653856306 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a94100920 0x7f8a9419d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:25.564 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.557+0000 7f8a9890c700 1 -- 172.21.15.38:0/653856306 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a94100920 msgr2=0x7f8a9419d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:25.564 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.557+0000 7f8a9890c700 1 --2- 172.21.15.38:0/653856306 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a94100920 0x7f8a9419d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:25.564 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.557+0000 7f8a9890c700 1 -- 172.21.15.38:0/653856306 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8a940fff90 msgr2=0x7f8a94196400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:25.564 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.557+0000 7f8a9890c700 1 --2- 172.21.15.38:0/653856306 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8a940fff90 0x7f8a94196400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:25.564 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.557+0000 7f8a9890c700 1 -- 172.21.15.38:0/653856306 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a8c009d00 con 0x7f8a940feda0 2023-03-09T14:44:25.564 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.557+0000 7f8a9910d700 1 --2- 172.21.15.38:0/653856306 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a94100920 0x7f8a9419d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-09T14:44:25.564 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.557+0000 7f8a9890c700 1 --2- 172.21.15.38:0/653856306 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8a940feda0 0x7f8a94195ec0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f8a8800d980 tx=0x7f8a8800dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:25.564 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.557+0000 7f8a91ffb700 1 -- 172.21.15.38:0/653856306 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a8800bb60 con 0x7f8a940feda0 2023-03-09T14:44:25.565 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.557+0000 7f8a91ffb700 1 -- 172.21.15.38:0/653856306 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8a88007620 con 0x7f8a940feda0 2023-03-09T14:44:25.565 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.557+0000 7f8a9ab70700 1 -- 172.21.15.38:0/653856306 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a9410a760 con 0x7f8a940feda0 2023-03-09T14:44:25.565 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.557+0000 7f8a91ffb700 1 -- 172.21.15.38:0/653856306 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a8800f620 con 0x7f8a940feda0 2023-03-09T14:44:25.565 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.557+0000 7f8a9ab70700 1 -- 172.21.15.38:0/653856306 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a9410a5d0 con 0x7f8a940feda0 2023-03-09T14:44:25.570 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.559+0000 7f8a91ffb700 1 -- 172.21.15.38:0/653856306 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 133198+0+0 (secure 0 0 0) 0x7f8a88018030 con 0x7f8a940feda0 2023-03-09T14:44:25.570 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.560+0000 7f8a91ffb700 1 --2- 172.21.15.38:0/653856306 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f8a8409f440 0x7f8a840a18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:25.570 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.560+0000 7f8a91ffb700 1 -- 172.21.15.38:0/653856306 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f8a880c9e50 con 0x7f8a940feda0 2023-03-09T14:44:25.570 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.560+0000 7f8a93fff700 1 --2- 172.21.15.38:0/653856306 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f8a8409f440 0x7f8a840a18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:25.571 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.560+0000 7f8a93fff700 1 --2- 172.21.15.38:0/653856306 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f8a8409f440 0x7f8a840a18f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f8a80005950 tx=0x7f8a800058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:25.571 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.561+0000 7f8aa6cf9b80 1 Processor -- start 2023-03-09T14:44:25.571 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.561+0000 7f8aa6cf9b80 1 -- start start 2023-03-09T14:44:25.571 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.561+0000 7f8aa6cf9b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5630fbe1f9f0 0x5630fbe1fe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:25.571 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.561+0000 7f8aa6cf9b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5630fbe20340 0x5630fbe20c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:25.571 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.561+0000 7f8aa6cf9b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5630fbe211f0 0x5630fbe21660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:25.571 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.561+0000 7f8aa6cf9b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630fbd09d50 con 0x5630fbe20340 2023-03-09T14:44:25.572 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.561+0000 7f8aa6cf9b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630fbd78730 con 0x5630fbe1f9f0 2023-03-09T14:44:25.572 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.561+0000 7f8aa6cf9b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630fbd71c60 con 0x5630fbe211f0 2023-03-09T14:44:25.572 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.561+0000 7f8a93fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5630fbe20340 0x5630fbe20c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:25.572 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.561+0000 7f8a93fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5630fbe20340 0x5630fbe20c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36814/0 (socket says 172.21.15.38:36814) 2023-03-09T14:44:25.572 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.561+0000 7f8a93fff700 1 -- 172.21.15.38:0/1582179926 learned_addr learned my addr 172.21.15.38:0/1582179926 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:44:25.572 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.561+0000 7f8a9890c700 1 --2- 172.21.15.38:0/1582179926 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5630fbe1f9f0 0x5630fbe1fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:25.572 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.561+0000 7f8a9910d700 1 --2- 172.21.15.38:0/1582179926 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5630fbe211f0 0x5630fbe21660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:25.572 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.561+0000 7f8a93fff700 1 -- 172.21.15.38:0/1582179926 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5630fbe211f0 msgr2=0x5630fbe21660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:44:25.573 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.561+0000 7f8a93fff700 1 --2- 172.21.15.38:0/1582179926 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5630fbe211f0 0x5630fbe21660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:25.573 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.561+0000 7f8a93fff700 1 -- 172.21.15.38:0/1582179926 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5630fbe1f9f0 msgr2=0x5630fbe1fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:44:25.573 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.561+0000 7f8a93fff700 1 --2- 172.21.15.38:0/1582179926 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5630fbe1f9f0 0x5630fbe1fe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:25.573 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.561+0000 7f8a93fff700 1 -- 172.21.15.38:0/1582179926 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5630fbd38140 con 0x5630fbe20340 2023-03-09T14:44:25.573 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.561+0000 7f8a9910d700 1 --2- 172.21.15.38:0/1582179926 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5630fbe211f0 0x5630fbe21660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-09T14:44:25.573 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.561+0000 7f8a9890c700 1 --2- 172.21.15.38:0/1582179926 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5630fbe1f9f0 0x5630fbe1fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:44:25.573 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.561+0000 7f8a93fff700 1 --2- 172.21.15.38:0/1582179926 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5630fbe20340 0x5630fbe20c60 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f8a8000fcd0 tx=0x7f8a80012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57b006c86c5bf000 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:25.574 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.562+0000 7f8a7effd700 1 -- 172.21.15.38:0/1582179926 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a80011c40 con 0x5630fbe20340 2023-03-09T14:44:25.574 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.562+0000 7f8a7effd700 1 -- 172.21.15.38:0/1582179926 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8a80011da0 con 0x5630fbe20340 2023-03-09T14:44:25.574 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.562+0000 7f8a7effd700 1 -- 172.21.15.38:0/1582179926 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a8001d6a0 con 0x5630fbe20340 2023-03-09T14:44:25.574 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.562+0000 7f8aa6cf9b80 1 -- 172.21.15.38:0/1582179926 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5630fbe20340 msgr2=0x5630fbe20c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:25.574 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.562+0000 7f8aa6cf9b80 1 --2- 172.21.15.38:0/1582179926 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5630fbe20340 0x5630fbe20c60 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f8a8000fcd0 tx=0x7f8a80012000 comp rx=0 tx=0).stop 2023-03-09T14:44:25.574 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.562+0000 7f8aa6cf9b80 1 -- 172.21.15.38:0/1582179926 shutdown_connections 2023-03-09T14:44:25.574 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.562+0000 7f8aa6cf9b80 1 --2- 172.21.15.38:0/1582179926 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5630fbe211f0 0x5630fbe21660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:25.574 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.562+0000 7f8aa6cf9b80 1 --2- 172.21.15.38:0/1582179926 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5630fbe1f9f0 0x5630fbe1fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:25.575 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.562+0000 7f8aa6cf9b80 1 --2- 172.21.15.38:0/1582179926 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5630fbe20340 0x5630fbe20c60 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:25.575 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.562+0000 7f8aa6cf9b80 1 -- 172.21.15.38:0/1582179926 >> 172.21.15.38:0/1582179926 conn(0x5630fbe1a500 msgr2=0x5630fbe1a8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:25.575 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.562+0000 7f8aa6cf9b80 1 -- 172.21.15.38:0/1582179926 shutdown_connections 2023-03-09T14:44:25.575 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.562+0000 7f8aa6cf9b80 1 -- 172.21.15.38:0/1582179926 wait complete. 2023-03-09T14:44:25.575 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.563+0000 7f8aa6cf9b80 1 -- start start 2023-03-09T14:44:25.575 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.563+0000 7f8aa6cf9b80 1 Processor -- start 2023-03-09T14:44:25.575 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.563+0000 7f8aa6cf9b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5630fbe1f9f0 0x5630fbe29790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:25.576 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.563+0000 7f8aa6cf9b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5630fbe20340 0x5630fbe29cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:25.576 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.563+0000 7f8aa6cf9b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5630fbe211f0 0x5630fbe30d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:25.576 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.563+0000 7f8aa6cf9b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630fbbf58d0 con 0x5630fbe211f0 2023-03-09T14:44:25.576 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.563+0000 7f8aa6cf9b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630fbe31250 con 0x5630fbe20340 2023-03-09T14:44:25.576 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.563+0000 7f8aa6cf9b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630fbe313e0 con 0x5630fbe1f9f0 2023-03-09T14:44:25.576 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.563+0000 7f8aa6cf9b80 10 client.0 _ll_register_callbacks cb 0x5630fbd174d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:44:25.576 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.563+0000 7f8a9910d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5630fbe211f0 0x5630fbe30d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:25.576 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.563+0000 7f8a9910d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5630fbe211f0 0x5630fbe30d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36822/0 (socket says 172.21.15.38:36822) 2023-03-09T14:44:25.577 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.563+0000 7f8a9910d700 1 -- 172.21.15.38:0/428832318 learned_addr learned my addr 172.21.15.38:0/428832318 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:44:25.577 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.563+0000 7f8a9890c700 1 --2- 172.21.15.38:0/428832318 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5630fbe1f9f0 0x5630fbe29790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:25.577 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.563+0000 7f8a93fff700 1 --2- 172.21.15.38:0/428832318 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5630fbe20340 0x5630fbe29cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:25.577 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.564+0000 7f8a9910d700 1 -- 172.21.15.38:0/428832318 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5630fbe1f9f0 msgr2=0x5630fbe29790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:44:25.577 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.564+0000 7f8a9910d700 1 --2- 172.21.15.38:0/428832318 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5630fbe1f9f0 0x5630fbe29790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:25.577 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.564+0000 7f8a9910d700 1 -- 172.21.15.38:0/428832318 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5630fbe20340 msgr2=0x5630fbe29cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:44:25.577 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.564+0000 7f8a9910d700 1 --2- 172.21.15.38:0/428832318 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5630fbe20340 0x5630fbe29cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:25.578 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.564+0000 7f8a9910d700 1 -- 172.21.15.38:0/428832318 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a8000f980 con 0x5630fbe211f0 2023-03-09T14:44:25.578 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.564+0000 7f8a93fff700 1 --2- 172.21.15.38:0/428832318 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5630fbe20340 0x5630fbe29cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:44:25.578 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.564+0000 7f8a9910d700 1 --2- 172.21.15.38:0/428832318 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5630fbe211f0 0x5630fbe30d10 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c00b070 tx=0x7f8a8c009300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afd5d725c0eb648a server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:25.578 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.564+0000 7f8a73fff700 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2023-03-09T14:44:25.578 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.564+0000 7f8a73fff700 1 -- 172.21.15.38:0/428832318 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a8c00e050 con 0x5630fbe211f0 2023-03-09T14:44:25.578 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.564+0000 7f8a73fff700 1 -- 172.21.15.38:0/428832318 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f8a8c005040 con 0x5630fbe211f0 2023-03-09T14:44:25.578 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.564+0000 7f8aa6cf9b80 1 -- 172.21.15.38:0/428832318 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5630fbd3f140 con 0x5630fbe211f0 2023-03-09T14:44:25.578 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.564+0000 7f8a73fff700 1 -- 172.21.15.38:0/428832318 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8a8c009b80 con 0x5630fbe211f0 2023-03-09T14:44:25.579 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.564+0000 7f8aa6cf9b80 10 client.14736 fetch_fsmap learned FSMap version 17 2023-03-09T14:44:25.579 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.564+0000 7f8aa6cf9b80 1 -- 172.21.15.38:0/428832318 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) v3 -- 0x5630fbb88c10 con 0x5630fbe211f0 2023-03-09T14:44:25.579 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.564+0000 7f8a73fff700 1 -- 172.21.15.38:0/428832318 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5630fbd3f140 con 0x5630fbe211f0 2023-03-09T14:44:25.579 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.564+0000 7f8a73fff700 1 -- 172.21.15.38:0/428832318 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 17) v1 ==== 7482+0+0 (secure 0 0 0) 0x7f8a8c00b0d0 con 0x5630fbe211f0 2023-03-09T14:44:25.579 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.564+0000 7f8aa6cf9b80 10 client.14736 fetch_fsmap finished waiting for FSMap version 17 2023-03-09T14:44:25.579 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.565+0000 7f8aa6cf9b80 10 client.14736 resolve_mds: resolved name 'b' to daemon mds.b 2023-03-09T14:44:25.579 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.565+0000 7f8aa6cf9b80 20 client.14736 populate_metadata read hostname 'smithi038' 2023-03-09T14:44:25.579 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.565+0000 7f8aa6cf9b80 1 --2- 172.21.15.38:0/428832318 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5630fbe34c20 0x5630fbe34000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:25.580 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.565+0000 7f8aa6cf9b80 4 client.14736 mds_command: new command op to 24530 tid=0[{"prefix": "get_command_descriptions"}] 2023-03-09T14:44:25.580 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.565+0000 7f8aa6cf9b80 1 -- 172.21.15.38:0/428832318 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5630fbbf5f10 con 0x5630fbe34c20 2023-03-09T14:44:25.580 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.565+0000 7f8a9890c700 1 --2- 172.21.15.38:0/428832318 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5630fbe34c20 0x5630fbe34000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:25.580 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.565+0000 7f8a73fff700 10 client.14736 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:44:25.580 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.565+0000 7f8a9890c700 1 --2- 172.21.15.38:0/428832318 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5630fbe34c20 0x5630fbe34000 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=62c1e55ec46d565c server_cookie=f0e1279dd85f16e4 in_seq=0 out_seq=0 2023-03-09T14:44:25.580 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.566+0000 7f8a73fff700 1 -- 172.21.15.38:0/428832318 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18278 (crc 0 0 0) 0x5630fbbf5f10 con 0x5630fbe34c20 2023-03-09T14:44:25.580 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.566+0000 7f8a73fff700 10 client.14736 handle_command_reply: tid=0 2023-03-09T14:44:25.581 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.566+0000 7f8aa6cf9b80 1 client.14736 shutdown 2023-03-09T14:44:25.581 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.566+0000 7f8aa6cf9b80 1 -- 172.21.15.38:0/428832318 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5630fbe211f0 msgr2=0x5630fbe30d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:25.581 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.566+0000 7f8aa6cf9b80 1 --2- 172.21.15.38:0/428832318 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5630fbe211f0 0x5630fbe30d10 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c00b070 tx=0x7f8a8c009300 comp rx=0 tx=0).stop 2023-03-09T14:44:25.581 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.566+0000 7f8aa6cf9b80 1 -- 172.21.15.38:0/428832318 shutdown_connections 2023-03-09T14:44:25.581 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.566+0000 7f8aa6cf9b80 1 --2- 172.21.15.38:0/428832318 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5630fbe34c20 0x5630fbe34000 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:25.581 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.566+0000 7f8aa6cf9b80 1 --2- 172.21.15.38:0/428832318 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5630fbe211f0 0x5630fbe30d10 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:25.581 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.566+0000 7f8aa6cf9b80 1 --2- 172.21.15.38:0/428832318 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5630fbe1f9f0 0x5630fbe29790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:25.581 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.566+0000 7f8a73fff700 0 client.14736 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:44:25.582 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.566+0000 7f8aa6cf9b80 1 --2- 172.21.15.38:0/428832318 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5630fbe20340 0x5630fbe29cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:25.582 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.566+0000 7f8aa6cf9b80 1 -- 172.21.15.38:0/428832318 >> 172.21.15.38:0/428832318 conn(0x5630fbe1a500 msgr2=0x5630fbe19bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:25.582 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.566+0000 7f8aa6cf9b80 1 -- 172.21.15.38:0/428832318 shutdown_connections 2023-03-09T14:44:25.582 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.566+0000 7f8aa6cf9b80 1 -- 172.21.15.38:0/428832318 wait complete. 2023-03-09T14:44:25.582 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.566+0000 7f8aa6cf9b80 20 client.14736 trim_cache size 0 max 16384 2023-03-09T14:44:25.588 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.584+0000 7f8aa6cf9b80 1 Processor -- start 2023-03-09T14:44:25.588 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.584+0000 7f8aa6cf9b80 1 -- start start 2023-03-09T14:44:25.588 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.584+0000 7f8aa6cf9b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5630fbe1f9f0 0x5630fbe34000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:25.589 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.584+0000 7f8aa6cf9b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5630fbe20340 0x5630fbe34540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:25.589 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.584+0000 7f8aa6cf9b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5630fbe211f0 0x5630fbe30c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:25.589 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.584+0000 7f8aa6cf9b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630fbd76a60 con 0x5630fbe211f0 2023-03-09T14:44:25.589 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.584+0000 7f8aa6cf9b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630fbd314b0 con 0x5630fbe1f9f0 2023-03-09T14:44:25.589 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.584+0000 7f8aa6cf9b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630fbc7ded0 con 0x5630fbe20340 2023-03-09T14:44:25.589 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.584+0000 7f8a9910d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5630fbe211f0 0x5630fbe30c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:25.589 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.584+0000 7f8a9910d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5630fbe211f0 0x5630fbe30c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36830/0 (socket says 172.21.15.38:36830) 2023-03-09T14:44:25.590 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.584+0000 7f8a9910d700 1 -- 172.21.15.38:0/68733084 learned_addr learned my addr 172.21.15.38:0/68733084 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:44:25.590 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.585+0000 7f8a93fff700 1 --2- 172.21.15.38:0/68733084 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5630fbe20340 0x5630fbe34540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:25.590 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.585+0000 7f8a9910d700 1 -- 172.21.15.38:0/68733084 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5630fbe20340 msgr2=0x5630fbe34540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:44:25.590 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.585+0000 7f8a9910d700 1 --2- 172.21.15.38:0/68733084 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5630fbe20340 0x5630fbe34540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:25.590 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.585+0000 7f8a9890c700 1 --2- 172.21.15.38:0/68733084 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5630fbe1f9f0 0x5630fbe34000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:25.590 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.585+0000 7f8a9910d700 1 -- 172.21.15.38:0/68733084 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5630fbe1f9f0 msgr2=0x5630fbe34000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:44:25.590 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.585+0000 7f8a9910d700 1 --2- 172.21.15.38:0/68733084 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5630fbe1f9f0 0x5630fbe34000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:25.591 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.585+0000 7f8a9910d700 1 -- 172.21.15.38:0/68733084 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5630fbb9fd90 con 0x5630fbe211f0 2023-03-09T14:44:25.591 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.585+0000 7f8a9910d700 1 --2- 172.21.15.38:0/68733084 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5630fbe211f0 0x5630fbe30c30 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c002230 tx=0x7f8a8c009b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81c60e6e5ca4c674 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:25.591 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.585+0000 7f8a7f7fe700 1 -- 172.21.15.38:0/68733084 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a8c00e070 con 0x5630fbe211f0 2023-03-09T14:44:25.591 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.585+0000 7f8a7f7fe700 1 -- 172.21.15.38:0/68733084 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8a8c00f040 con 0x5630fbe211f0 2023-03-09T14:44:25.591 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.585+0000 7f8a7f7fe700 1 -- 172.21.15.38:0/68733084 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a8c026b60 con 0x5630fbe211f0 2023-03-09T14:44:25.594 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.586+0000 7f8aa6cf9b80 1 -- 172.21.15.38:0/68733084 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5630fbe211f0 msgr2=0x5630fbe30c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:25.594 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.586+0000 7f8aa6cf9b80 1 --2- 172.21.15.38:0/68733084 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5630fbe211f0 0x5630fbe30c30 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c002230 tx=0x7f8a8c009b80 comp rx=0 tx=0).stop 2023-03-09T14:44:25.595 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.586+0000 7f8aa6cf9b80 1 -- 172.21.15.38:0/68733084 shutdown_connections 2023-03-09T14:44:25.595 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.586+0000 7f8aa6cf9b80 1 --2- 172.21.15.38:0/68733084 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5630fbe211f0 0x5630fbe30c30 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:25.595 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.586+0000 7f8aa6cf9b80 1 --2- 172.21.15.38:0/68733084 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5630fbe1f9f0 0x5630fbe34000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:25.595 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.586+0000 7f8aa6cf9b80 1 --2- 172.21.15.38:0/68733084 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5630fbe20340 0x5630fbe34540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:25.595 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.586+0000 7f8aa6cf9b80 1 -- 172.21.15.38:0/68733084 >> 172.21.15.38:0/68733084 conn(0x5630fbe1a500 msgr2=0x5630fbe27560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:25.595 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.586+0000 7f8aa6cf9b80 1 -- 172.21.15.38:0/68733084 shutdown_connections 2023-03-09T14:44:25.595 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.586+0000 7f8aa6cf9b80 1 -- 172.21.15.38:0/68733084 wait complete. 2023-03-09T14:44:25.596 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.586+0000 7f8aa6cf9b80 1 -- start start 2023-03-09T14:44:25.596 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.586+0000 7f8aa6cf9b80 1 Processor -- start 2023-03-09T14:44:25.596 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.586+0000 7f8aa6cf9b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5630fbe1f9f0 0x5630fbe2c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:25.596 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.586+0000 7f8aa6cf9b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5630fbe20340 0x5630fbe2c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:25.596 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.586+0000 7f8aa6cf9b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5630fbe211f0 0x5630fbe2f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:25.596 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.586+0000 7f8aa6cf9b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630fbe2fdc0 con 0x5630fbe1f9f0 2023-03-09T14:44:25.596 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.586+0000 7f8aa6cf9b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630fbe2ff30 con 0x5630fbe20340 2023-03-09T14:44:25.596 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.586+0000 7f8aa6cf9b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x5630fbe300f0 con 0x5630fbe211f0 2023-03-09T14:44:25.597 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.586+0000 7f8aa6cf9b80 10 client.0 _ll_register_callbacks cb 0x5630fbd174d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:44:25.597 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.586+0000 7f8a9890c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5630fbe1f9f0 0x5630fbe2c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:25.597 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.587+0000 7f8a9890c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5630fbe1f9f0 0x5630fbe2c480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36832/0 (socket says 172.21.15.38:36832) 2023-03-09T14:44:25.597 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.587+0000 7f8a9890c700 1 -- 172.21.15.38:0/2846861012 learned_addr learned my addr 172.21.15.38:0/2846861012 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:44:25.597 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.587+0000 7f8a93fff700 1 --2- 172.21.15.38:0/2846861012 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5630fbe20340 0x5630fbe2c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:25.597 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.587+0000 7f8a9910d700 1 --2- 172.21.15.38:0/2846861012 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5630fbe211f0 0x5630fbe2f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:25.597 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.587+0000 7f8a9890c700 1 -- 172.21.15.38:0/2846861012 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5630fbe211f0 msgr2=0x5630fbe2f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:44:25.598 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.587+0000 7f8a9890c700 1 --2- 172.21.15.38:0/2846861012 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5630fbe211f0 0x5630fbe2f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:25.598 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.587+0000 7f8a9890c700 1 -- 172.21.15.38:0/2846861012 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5630fbe20340 msgr2=0x5630fbe2c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:44:25.598 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.587+0000 7f8a9890c700 1 --2- 172.21.15.38:0/2846861012 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5630fbe20340 0x5630fbe2c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:25.598 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.587+0000 7f8a9890c700 1 -- 172.21.15.38:0/2846861012 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5630fbb88c10 con 0x5630fbe1f9f0 2023-03-09T14:44:25.598 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.587+0000 7f8a93fff700 1 --2- 172.21.15.38:0/2846861012 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5630fbe20340 0x5630fbe2c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-09T14:44:25.598 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.587+0000 7f8a9910d700 1 --2- 172.21.15.38:0/2846861012 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5630fbe211f0 0x5630fbe2f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-09T14:44:25.598 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.587+0000 7f8a9890c700 1 --2- 172.21.15.38:0/2846861012 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5630fbe1f9f0 0x5630fbe2c480 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f8a88097c70 tx=0x7f8a88097b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96b581d3f1e9696a server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:25.598 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.587+0000 7f8a73fff700 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2023-03-09T14:44:25.599 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.587+0000 7f8a73fff700 1 -- 172.21.15.38:0/2846861012 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a88098140 con 0x5630fbe1f9f0 2023-03-09T14:44:25.599 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.587+0000 7f8a73fff700 1 -- 172.21.15.38:0/2846861012 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f8a880971d0 con 0x5630fbe1f9f0 2023-03-09T14:44:25.599 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.587+0000 7f8aa6cf9b80 1 -- 172.21.15.38:0/2846861012 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5630fbd2f160 con 0x5630fbe1f9f0 2023-03-09T14:44:25.599 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.587+0000 7f8a73fff700 1 -- 172.21.15.38:0/2846861012 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8a88097390 con 0x5630fbe1f9f0 2023-03-09T14:44:25.599 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.587+0000 7f8aa6cf9b80 10 client.14742 fetch_fsmap learned FSMap version 17 2023-03-09T14:44:25.599 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.587+0000 7f8aa6cf9b80 1 -- 172.21.15.38:0/2846861012 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) v3 -- 0x5630fbb9e260 con 0x5630fbe1f9f0 2023-03-09T14:44:25.599 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.587+0000 7f8a73fff700 1 -- 172.21.15.38:0/2846861012 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5630fbd2f160 con 0x5630fbe1f9f0 2023-03-09T14:44:25.600 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.588+0000 7f8a73fff700 1 -- 172.21.15.38:0/2846861012 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 17) v1 ==== 7482+0+0 (secure 0 0 0) 0x7f8a880104f0 con 0x5630fbe1f9f0 2023-03-09T14:44:25.600 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.588+0000 7f8aa6cf9b80 10 client.14742 fetch_fsmap finished waiting for FSMap version 17 2023-03-09T14:44:25.600 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.588+0000 7f8aa6cf9b80 10 client.14742 resolve_mds: resolved name 'b' to daemon mds.b 2023-03-09T14:44:25.600 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.588+0000 7f8aa6cf9b80 20 client.14742 populate_metadata read hostname 'smithi038' 2023-03-09T14:44:25.600 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.588+0000 7f8aa6cf9b80 1 --2- 172.21.15.38:0/2846861012 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5630fbe8e700 0x5630fbe8b010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:25.600 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.588+0000 7f8aa6cf9b80 4 client.14742 mds_command: new command op to 24530 tid=0[{"prefix": "status"}] 2023-03-09T14:44:25.600 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.588+0000 7f8aa6cf9b80 1 -- 172.21.15.38:0/2846861012 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5630fbd19170 con 0x5630fbe8e700 2023-03-09T14:44:25.600 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.588+0000 7f8a93fff700 1 --2- 172.21.15.38:0/2846861012 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5630fbe8e700 0x5630fbe8b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:25.601 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.588+0000 7f8a73fff700 10 client.14742 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:44:25.601 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.588+0000 7f8a93fff700 1 --2- 172.21.15.38:0/2846861012 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5630fbe8e700 0x5630fbe8b010 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=65c928cedc0927d7 server_cookie=115f78806042ea1e in_seq=0 out_seq=0 2023-03-09T14:44:25.601 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.589+0000 7f8a73fff700 1 -- 172.21.15.38:0/2846861012 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+326 (crc 0 0 0) 0x5630fbd19170 con 0x5630fbe8e700 2023-03-09T14:44:25.601 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.589+0000 7f8a73fff700 10 client.14742 handle_command_reply: tid=0 2023-03-09T14:44:25.601 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.589+0000 7f8aa6cf9b80 1 client.14742 shutdown 2023-03-09T14:44:25.601 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.589+0000 7f8aa6cf9b80 1 -- 172.21.15.38:0/2846861012 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5630fbe1f9f0 msgr2=0x5630fbe2c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:25.601 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.589+0000 7f8aa6cf9b80 1 --2- 172.21.15.38:0/2846861012 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5630fbe1f9f0 0x5630fbe2c480 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f8a88097c70 tx=0x7f8a88097b60 comp rx=0 tx=0).stop 2023-03-09T14:44:25.602 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.589+0000 7f8aa6cf9b80 1 -- 172.21.15.38:0/2846861012 shutdown_connections 2023-03-09T14:44:25.602 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.589+0000 7f8aa6cf9b80 1 --2- 172.21.15.38:0/2846861012 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5630fbe8e700 0x5630fbe8b010 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:25.602 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.589+0000 7f8aa6cf9b80 1 --2- 172.21.15.38:0/2846861012 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5630fbe1f9f0 0x5630fbe2c480 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:25.602 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.589+0000 7f8a73fff700 0 client.14742 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:44:25.602 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.589+0000 7f8aa6cf9b80 1 --2- 172.21.15.38:0/2846861012 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5630fbe211f0 0x5630fbe2f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:25.602 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.589+0000 7f8aa6cf9b80 1 --2- 172.21.15.38:0/2846861012 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5630fbe20340 0x5630fbe2c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:25.602 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.589+0000 7f8aa6cf9b80 1 -- 172.21.15.38:0/2846861012 >> 172.21.15.38:0/2846861012 conn(0x5630fbe1a500 msgr2=0x5630fbe34000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:25.602 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.589+0000 7f8aa6cf9b80 1 -- 172.21.15.38:0/2846861012 shutdown_connections 2023-03-09T14:44:25.603 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.589+0000 7f8aa6cf9b80 1 -- 172.21.15.38:0/2846861012 wait complete. 2023-03-09T14:44:25.603 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.589+0000 7f8aa6cf9b80 20 client.14742 trim_cache size 0 max 16384 2023-03-09T14:44:25.603 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2023-03-09T14:44:25.603 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_fsid": "c6733fe2-be87-11ed-9af2-001a4aab830c", 2023-03-09T14:44:25.603 INFO:teuthology.orchestra.run.smithi038.stdout: "whoami": 0, 2023-03-09T14:44:25.604 INFO:teuthology.orchestra.run.smithi038.stdout: "id": 24530, 2023-03-09T14:44:25.604 INFO:teuthology.orchestra.run.smithi038.stdout: "want_state": "up:active", 2023-03-09T14:44:25.604 INFO:teuthology.orchestra.run.smithi038.stdout: "state": "up:active", 2023-03-09T14:44:25.604 INFO:teuthology.orchestra.run.smithi038.stdout: "fs_name": "cephfs", 2023-03-09T14:44:25.604 INFO:teuthology.orchestra.run.smithi038.stdout: "rank_uptime": 6.5024879870000003, 2023-03-09T14:44:25.604 INFO:teuthology.orchestra.run.smithi038.stdout: "mdsmap_epoch": 17, 2023-03-09T14:44:25.604 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_epoch": 87, 2023-03-09T14:44:25.604 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_epoch_barrier": 86, 2023-03-09T14:44:25.605 INFO:teuthology.orchestra.run.smithi038.stdout: "uptime": 12.010242965 2023-03-09T14:44:25.605 INFO:teuthology.orchestra.run.smithi038.stdout:} 2023-03-09T14:44:25.606 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.589+0000 7f8a9ab70700 1 -- 172.21.15.38:0/653856306 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f8a8409f440 msgr2=0x7f8a840a18f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:25.606 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.589+0000 7f8a9ab70700 1 --2- 172.21.15.38:0/653856306 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f8a8409f440 0x7f8a840a18f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f8a80005950 tx=0x7f8a800058e0 comp rx=0 tx=0).stop 2023-03-09T14:44:25.606 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.589+0000 7f8a9ab70700 1 -- 172.21.15.38:0/653856306 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8a940feda0 msgr2=0x7f8a94195ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:25.606 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.589+0000 7f8a9ab70700 1 --2- 172.21.15.38:0/653856306 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8a940feda0 0x7f8a94195ec0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f8a8800d980 tx=0x7f8a8800dd40 comp rx=0 tx=0).stop 2023-03-09T14:44:25.606 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.590+0000 7f8a9ab70700 1 -- 172.21.15.38:0/653856306 shutdown_connections 2023-03-09T14:44:25.607 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.590+0000 7f8a9ab70700 1 --2- 172.21.15.38:0/653856306 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8a940feda0 0x7f8a94195ec0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:25.607 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.590+0000 7f8a9ab70700 1 --2- 172.21.15.38:0/653856306 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f8a8409f440 0x7f8a840a18f0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:25.607 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.590+0000 7f8a9ab70700 1 --2- 172.21.15.38:0/653856306 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a94100920 0x7f8a9419d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:25.607 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.590+0000 7f8a9ab70700 1 --2- 172.21.15.38:0/653856306 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8a940fff90 0x7f8a94196400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:25.607 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.590+0000 7f8a9ab70700 1 -- 172.21.15.38:0/653856306 >> 172.21.15.38:0/653856306 conn(0x7f8a940fa3a0 msgr2=0x7f8a940fbeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:25.607 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.590+0000 7f8a9ab70700 1 -- 172.21.15.38:0/653856306 shutdown_connections 2023-03-09T14:44:25.607 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:25.590+0000 7f8a9ab70700 1 -- 172.21.15.38:0/653856306 wait complete. 2023-03-09T14:44:25.608 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-03-09T14:44:25.608 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2023-03-09T14:44:25.609 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph fs ls 2023-03-09T14:44:25.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:25 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/530301329' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-03-09T14:44:25.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:25 smithi181 ceph-mon[94189]: pgmap v262: 129 pgs: 7 creating+peering, 106 active+clean, 16 unknown; 579 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 7 op/s 2023-03-09T14:44:25.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:25 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/2139050813' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-03-09T14:44:25.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:25 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/3978127944' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-03-09T14:44:25.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:25 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/530301329' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-03-09T14:44:25.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:25 smithi038 ceph-mon[92422]: pgmap v262: 129 pgs: 7 creating+peering, 106 active+clean, 16 unknown; 579 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 7 op/s 2023-03-09T14:44:25.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:25 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/2139050813' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-03-09T14:44:25.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:25 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/3978127944' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-03-09T14:44:25.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:25 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/530301329' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-03-09T14:44:25.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:25 smithi122 ceph-mon[93869]: pgmap v262: 129 pgs: 7 creating+peering, 106 active+clean, 16 unknown; 579 KiB data, 78 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 7 op/s 2023-03-09T14:44:25.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:25 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/2139050813' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-03-09T14:44:25.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:25 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/3978127944' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-03-09T14:44:26.254 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.a/config 2023-03-09T14:44:27.907 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:27.902+0000 7fbb0d24f700 1 -- 172.21.15.38:0/1998846364 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbb080ffac0 msgr2=0x7fbb08103a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:27.907 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:27.902+0000 7fbb0d24f700 1 --2- 172.21.15.38:0/1998846364 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbb080ffac0 0x7fbb08103a80 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fbaf00099c0 tx=0x7fbaf0009cd0 comp rx=0 tx=0).stop 2023-03-09T14:44:27.908 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:27.902+0000 7fbb0d24f700 1 -- 172.21.15.38:0/1998846364 shutdown_connections 2023-03-09T14:44:27.908 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:27.902+0000 7fbb0d24f700 1 --2- 172.21.15.38:0/1998846364 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb08104160 0x7fbb0810bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:27.908 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:27.902+0000 7fbb0d24f700 1 --2- 172.21.15.38:0/1998846364 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb080ff120 0x7fbb080ff4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:27.908 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:27.902+0000 7fbb0d24f700 1 --2- 172.21.15.38:0/1998846364 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbb080ffac0 0x7fbb08103a80 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:27.908 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:27.902+0000 7fbb0d24f700 1 -- 172.21.15.38:0/1998846364 >> 172.21.15.38:0/1998846364 conn(0x7fbb08076d10 msgr2=0x7fbb08077110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:27.908 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:27.902+0000 7fbb0d24f700 1 -- 172.21.15.38:0/1998846364 shutdown_connections 2023-03-09T14:44:27.908 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:27.902+0000 7fbb0d24f700 1 -- 172.21.15.38:0/1998846364 wait complete. 2023-03-09T14:44:27.909 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:27.903+0000 7fbb0d24f700 1 Processor -- start 2023-03-09T14:44:27.909 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:27.903+0000 7fbb0d24f700 1 -- start start 2023-03-09T14:44:27.909 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:27.903+0000 7fbb0d24f700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb080ff120 0x7fbb08194fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:27.909 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:27.903+0000 7fbb0d24f700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb080ffac0 0x7fbb08195500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:27.909 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:27.903+0000 7fbb0d24f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbb08104160 0x7fbb08199960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:27.909 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:27.903+0000 7fbb0d24f700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb08199fa0 con 0x7fbb08104160 2023-03-09T14:44:27.909 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:27.903+0000 7fbb0d24f700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb0819a110 con 0x7fbb080ff120 2023-03-09T14:44:27.910 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:27.903+0000 7fbb0d24f700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb0819a2d0 con 0x7fbb080ffac0 2023-03-09T14:44:27.910 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:27.903+0000 7fbb0759e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbb08104160 0x7fbb08199960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:27.910 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:27.903+0000 7fbb0759e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbb08104160 0x7fbb08199960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36848/0 (socket says 172.21.15.38:36848) 2023-03-09T14:44:27.913 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:27.903+0000 7fbb0759e700 1 -- 172.21.15.38:0/2957901080 learned_addr learned my addr 172.21.15.38:0/2957901080 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:44:27.913 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:27.904+0000 7fbb0659c700 1 --2- 172.21.15.38:0/2957901080 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb080ffac0 0x7fbb08195500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:27.913 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:27.904+0000 7fbb06d9d700 1 --2- 172.21.15.38:0/2957901080 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb080ff120 0x7fbb08194fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:27.915 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:27.904+0000 7fbb0759e700 1 -- 172.21.15.38:0/2957901080 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb080ffac0 msgr2=0x7fbb08195500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:27.915 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:27.904+0000 7fbb0759e700 1 --2- 172.21.15.38:0/2957901080 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb080ffac0 0x7fbb08195500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:27.915 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:27.904+0000 7fbb0759e700 1 -- 172.21.15.38:0/2957901080 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb080ff120 msgr2=0x7fbb08194fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:27.915 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:27.904+0000 7fbb0759e700 1 --2- 172.21.15.38:0/2957901080 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb080ff120 0x7fbb08194fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:27.915 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:27.904+0000 7fbb0759e700 1 -- 172.21.15.38:0/2957901080 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaf00095f0 con 0x7fbb08104160 2023-03-09T14:44:27.915 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:27.904+0000 7fbb06d9d700 1 --2- 172.21.15.38:0/2957901080 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb080ff120 0x7fbb08194fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:44:27.916 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:27.904+0000 7fbb0759e700 1 --2- 172.21.15.38:0/2957901080 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbb08104160 0x7fbb08199960 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fbb00009ee0 tx=0x7fbb0000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:27.916 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:27.904+0000 7fbafffff700 1 -- 172.21.15.38:0/2957901080 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb0000ddd0 con 0x7fbb08104160 2023-03-09T14:44:27.916 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:27.904+0000 7fbb0d24f700 1 -- 172.21.15.38:0/2957901080 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb0810eee0 con 0x7fbb08104160 2023-03-09T14:44:27.916 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:27.904+0000 7fbafffff700 1 -- 172.21.15.38:0/2957901080 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbb00004020 con 0x7fbb08104160 2023-03-09T14:44:27.916 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:27.904+0000 7fbafffff700 1 -- 172.21.15.38:0/2957901080 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb000126c0 con 0x7fbb08104160 2023-03-09T14:44:27.917 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:27.904+0000 7fbb0d24f700 1 -- 172.21.15.38:0/2957901080 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb0810ed50 con 0x7fbb08104160 2023-03-09T14:44:27.917 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:27.905+0000 7fbb0d24f700 1 -- 172.21.15.38:0/2957901080 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb0810f070 con 0x7fbb08104160 2023-03-09T14:44:27.918 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:27.906+0000 7fbafffff700 1 -- 172.21.15.38:0/2957901080 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 133198+0+0 (secure 0 0 0) 0x7fbb00023070 con 0x7fbb08104160 2023-03-09T14:44:27.918 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:27.907+0000 7fbafffff700 1 --2- 172.21.15.38:0/2957901080 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fbaf409f3c0 0x7fbaf40a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:27.918 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:27.907+0000 7fbafffff700 1 -- 172.21.15.38:0/2957901080 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fbb000cbeb0 con 0x7fbb08104160 2023-03-09T14:44:27.918 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:27.907+0000 7fbb06d9d700 1 --2- 172.21.15.38:0/2957901080 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fbaf409f3c0 0x7fbaf40a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:27.918 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:27.907+0000 7fbb06d9d700 1 --2- 172.21.15.38:0/2957901080 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fbaf409f3c0 0x7fbaf40a1870 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fbaf8005950 tx=0x7fbaf80058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:27.918 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:27.910+0000 7fbafffff700 1 -- 172.21.15.38:0/2957901080 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7fbb00096020 con 0x7fbb08104160 2023-03-09T14:44:28.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:27 smithi038 ceph-mon[92422]: pgmap v263: 129 pgs: 7 creating+peering, 118 active+clean, 4 unknown; 579 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 1 op/s 2023-03-09T14:44:28.155 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:28.151+0000 7fbb0d24f700 1 -- 172.21.15.38:0/2957901080 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fbb08066b10 con 0x7fbb08104160 2023-03-09T14:44:28.155 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:28.151+0000 7fbafffff700 1 -- 172.21.15.38:0/2957901080 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) v1 ==== 53+0+88 (secure 0 0 0) 0x7fbb00018020 con 0x7fbb08104160 2023-03-09T14:44:28.155 INFO:teuthology.orchestra.run.smithi038.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-03-09T14:44:28.157 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:28.153+0000 7fbb0d24f700 1 -- 172.21.15.38:0/2957901080 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fbaf409f3c0 msgr2=0x7fbaf40a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:28.157 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:28.153+0000 7fbb0d24f700 1 --2- 172.21.15.38:0/2957901080 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fbaf409f3c0 0x7fbaf40a1870 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fbaf8005950 tx=0x7fbaf80058e0 comp rx=0 tx=0).stop 2023-03-09T14:44:28.157 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:28.153+0000 7fbb0d24f700 1 -- 172.21.15.38:0/2957901080 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbb08104160 msgr2=0x7fbb08199960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:28.157 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:28.153+0000 7fbb0d24f700 1 --2- 172.21.15.38:0/2957901080 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbb08104160 0x7fbb08199960 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fbb00009ee0 tx=0x7fbb0000d3b0 comp rx=0 tx=0).stop 2023-03-09T14:44:28.157 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:28.154+0000 7fbb0d24f700 1 -- 172.21.15.38:0/2957901080 shutdown_connections 2023-03-09T14:44:28.158 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:28.154+0000 7fbb0d24f700 1 --2- 172.21.15.38:0/2957901080 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbb08104160 0x7fbb08199960 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:28.158 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:28.154+0000 7fbb0d24f700 1 --2- 172.21.15.38:0/2957901080 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fbaf409f3c0 0x7fbaf40a1870 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:28.158 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:28.154+0000 7fbb0d24f700 1 --2- 172.21.15.38:0/2957901080 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbb080ff120 0x7fbb08194fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:28.158 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:28.154+0000 7fbb0d24f700 1 --2- 172.21.15.38:0/2957901080 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbb080ffac0 0x7fbb08195500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:28.158 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:28.154+0000 7fbb0d24f700 1 -- 172.21.15.38:0/2957901080 >> 172.21.15.38:0/2957901080 conn(0x7fbb08076d10 msgr2=0x7fbb080fc8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:28.158 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:28.154+0000 7fbb0d24f700 1 -- 172.21.15.38:0/2957901080 shutdown_connections 2023-03-09T14:44:28.158 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:28.154+0000 7fbb0d24f700 1 -- 172.21.15.38:0/2957901080 wait complete. 2023-03-09T14:44:28.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:27 smithi122 ceph-mon[93869]: pgmap v263: 129 pgs: 7 creating+peering, 118 active+clean, 4 unknown; 579 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 1 op/s 2023-03-09T14:44:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:27 smithi181 ceph-mon[94189]: pgmap v263: 129 pgs: 7 creating+peering, 118 active+clean, 4 unknown; 579 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 1 op/s 2023-03-09T14:44:28.607 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-03-09T14:44:28.608 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-03-09T14:44:28.608 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi038.front.sepia.ceph.com 2023-03-09T14:44:28.608 INFO:tasks.cephfs.mount:self.client.name = client.admin 2023-03-09T14:44:28.609 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2023-03-09T14:44:28.609 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-03-09T14:44:28.609 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-03-09T14:44:28.609 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-03-09T14:44:28.610 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2023-03-09T14:44:28.610 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-09T14:44:28.610 DEBUG:teuthology.orchestra.run.smithi038:> ip addr 2023-03-09T14:44:28.629 INFO:teuthology.orchestra.run.smithi038.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-03-09T14:44:28.630 INFO:teuthology.orchestra.run.smithi038.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-03-09T14:44:28.630 INFO:teuthology.orchestra.run.smithi038.stdout: inet 127.0.0.1/8 scope host lo 2023-03-09T14:44:28.630 INFO:teuthology.orchestra.run.smithi038.stdout: valid_lft forever preferred_lft forever 2023-03-09T14:44:28.630 INFO:teuthology.orchestra.run.smithi038.stdout: inet6 ::1/128 scope host 2023-03-09T14:44:28.630 INFO:teuthology.orchestra.run.smithi038.stdout: valid_lft forever preferred_lft forever 2023-03-09T14:44:28.630 INFO:teuthology.orchestra.run.smithi038.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-03-09T14:44:28.631 INFO:teuthology.orchestra.run.smithi038.stdout: link/ether 0c:c4:7a:6b:fd:a2 brd ff:ff:ff:ff:ff:ff 2023-03-09T14:44:28.631 INFO:teuthology.orchestra.run.smithi038.stdout:3: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-03-09T14:44:28.631 INFO:teuthology.orchestra.run.smithi038.stdout: link/ether 0c:c4:7a:bd:15:fc brd ff:ff:ff:ff:ff:ff 2023-03-09T14:44:28.631 INFO:teuthology.orchestra.run.smithi038.stdout: inet 172.21.15.38/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-03-09T14:44:28.631 INFO:teuthology.orchestra.run.smithi038.stdout: valid_lft 42046sec preferred_lft 42046sec 2023-03-09T14:44:28.631 INFO:teuthology.orchestra.run.smithi038.stdout: inet6 fe80::ec4:7aff:febd:15fc/64 scope link 2023-03-09T14:44:28.631 INFO:teuthology.orchestra.run.smithi038.stdout: valid_lft forever preferred_lft forever 2023-03-09T14:44:28.631 INFO:teuthology.orchestra.run.smithi038.stdout:4: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-03-09T14:44:28.632 INFO:teuthology.orchestra.run.smithi038.stdout: link/ether 0c:c4:7a:6b:fd:a3 brd ff:ff:ff:ff:ff:ff 2023-03-09T14:44:28.632 INFO:teuthology.orchestra.run.smithi038.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-03-09T14:44:28.632 INFO:teuthology.orchestra.run.smithi038.stdout: link/ether 0c:c4:7a:bd:15:fd brd ff:ff:ff:ff:ff:ff 2023-03-09T14:44:28.633 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-03-09T14:44:28.633 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-09T14:44:28.633 DEBUG:teuthology.orchestra.run.smithi038:> (cd / && exec bash -c ' 2023-03-09T14:44:28.634 DEBUG:teuthology.orchestra.run.smithi038:> set -e 2023-03-09T14:44:28.634 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip link add name ceph-brx type bridge 2023-03-09T14:44:28.634 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip addr flush dev ceph-brx 2023-03-09T14:44:28.634 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip link set ceph-brx up 2023-03-09T14:44:28.635 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-03-09T14:44:28.635 DEBUG:teuthology.orchestra.run.smithi038:> ') 2023-03-09T14:44:28.808 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-09T14:44:28.809 DEBUG:teuthology.orchestra.run.smithi038:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-03-09T14:44:28.838 INFO:teuthology.orchestra.run.smithi038.stdout:1 2023-03-09T14:44:28.839 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-09T14:44:28.840 DEBUG:teuthology.orchestra.run.smithi038:> route 2023-03-09T14:44:28.918 INFO:teuthology.orchestra.run.smithi038.stdout:Kernel IP routing table 2023-03-09T14:44:28.918 INFO:teuthology.orchestra.run.smithi038.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-03-09T14:44:28.918 INFO:teuthology.orchestra.run.smithi038.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-03-09T14:44:28.919 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-03-09T14:44:28.919 INFO:teuthology.orchestra.run.smithi038.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-03-09T14:44:28.919 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-09T14:44:28.920 DEBUG:teuthology.orchestra.run.smithi038:> (cd / && exec bash -c ' 2023-03-09T14:44:28.920 DEBUG:teuthology.orchestra.run.smithi038:> set -e 2023-03-09T14:44:28.920 DEBUG:teuthology.orchestra.run.smithi038:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-03-09T14:44:28.920 DEBUG:teuthology.orchestra.run.smithi038:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-03-09T14:44:28.921 DEBUG:teuthology.orchestra.run.smithi038:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-03-09T14:44:28.921 DEBUG:teuthology.orchestra.run.smithi038:> ') 2023-03-09T14:44:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:28 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/2957901080' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-03-09T14:44:29.171 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-09T14:44:29.172 DEBUG:teuthology.orchestra.run.smithi038:> ip netns list 2023-03-09T14:44:29.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:28 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/2957901080' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-03-09T14:44:29.230 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-09T14:44:29.230 DEBUG:teuthology.orchestra.run.smithi038:> ip netns list-id 2023-03-09T14:44:29.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:28 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/2957901080' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-03-09T14:44:29.288 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-09T14:44:29.288 DEBUG:teuthology.orchestra.run.smithi038:> (cd / && exec bash -c ' 2023-03-09T14:44:29.289 DEBUG:teuthology.orchestra.run.smithi038:> set -e 2023-03-09T14:44:29.289 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-03-09T14:44:29.289 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2023-03-09T14:44:29.289 DEBUG:teuthology.orchestra.run.smithi038:> ') 2023-03-09T14:44:29.386 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2023-03-09T14:44:29.387 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-09T14:44:29.387 DEBUG:teuthology.orchestra.run.smithi038:> (cd / && exec bash -c ' 2023-03-09T14:44:29.388 DEBUG:teuthology.orchestra.run.smithi038:> set -e 2023-03-09T14:44:29.388 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2023-03-09T14:44:29.389 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-03-09T14:44:29.390 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2023-03-09T14:44:29.390 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2023-03-09T14:44:29.391 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2023-03-09T14:44:29.391 DEBUG:teuthology.orchestra.run.smithi038:> ') 2023-03-09T14:44:29.608 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-09T14:44:29.608 DEBUG:teuthology.orchestra.run.smithi038:> (cd / && exec bash -c ' 2023-03-09T14:44:29.609 DEBUG:teuthology.orchestra.run.smithi038:> set -e 2023-03-09T14:44:29.609 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip link set brx.0 up 2023-03-09T14:44:29.609 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip link set dev brx.0 master ceph-brx 2023-03-09T14:44:29.609 DEBUG:teuthology.orchestra.run.smithi038:> ') 2023-03-09T14:44:29.709 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2023-03-09T14:44:29.710 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-03-09T14:44:29.711 DEBUG:teuthology.orchestra.run.smithi038:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2023-03-09T14:44:29.729 INFO:teuthology.orchestra.run.smithi038.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2023-03-09T14:44:29.729 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-03-09T14:44:29.729 DEBUG:teuthology.orchestra.run.smithi038:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2023-03-09T14:44:29.786 DEBUG:teuthology.orchestra.run.smithi038:> sudo modprobe fuse 2023-03-09T14:44:29.854 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-03-09T14:44:29.855 DEBUG:teuthology.orchestra.run.smithi038:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-03-09T14:44:29.918 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:29 smithi038 ceph-mon[92422]: pgmap v264: 129 pgs: 129 active+clean; 579 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 0 op/s 2023-03-09T14:44:29.921 INFO:teuthology.orchestra.run.smithi038.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-03-09T14:44:29.922 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-03-09T14:44:29.923 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-09T14:44:29.924 DEBUG:teuthology.orchestra.run.smithi038:> ls /sys/fs/fuse/connections 2023-03-09T14:44:29.981 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-03-09T14:44:29.982 DEBUG:teuthology.orchestra.run.smithi038:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2023-03-09T14:44:30.023 DEBUG:teuthology.orchestra.run.smithi038:> sudo modprobe fuse 2023-03-09T14:44:30.050 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-03-09T14:44:30.051 DEBUG:teuthology.orchestra.run.smithi038:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-03-09T14:44:30.120 INFO:teuthology.orchestra.run.smithi038.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-03-09T14:44:30.121 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-03-09T14:44:30.121 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-09T14:44:30.122 DEBUG:teuthology.orchestra.run.smithi038:> ls /sys/fs/fuse/connections 2023-03-09T14:44:30.141 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi038.stderr:2023-03-09T14:44:30.137+0000 7f9ac78515c0 -1 init, newargv = 0x555dd6d4b470 newargc=15 2023-03-09T14:44:30.141 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi038.stderr:ceph-fuse[129402]: starting ceph client 2023-03-09T14:44:30.147 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi038.stderr:ceph-fuse[129402]: starting fuse 2023-03-09T14:44:30.158 INFO:teuthology.orchestra.run.smithi038.stdout:124 2023-03-09T14:44:30.159 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [124] 2023-03-09T14:44:30.159 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-03-09T14:44:30.160 DEBUG:teuthology.orchestra.run.smithi038:> import glob 2023-03-09T14:44:30.160 DEBUG:teuthology.orchestra.run.smithi038:> import re 2023-03-09T14:44:30.160 DEBUG:teuthology.orchestra.run.smithi038:> import os 2023-03-09T14:44:30.161 DEBUG:teuthology.orchestra.run.smithi038:> import subprocess 2023-03-09T14:44:30.161 DEBUG:teuthology.orchestra.run.smithi038:> 2023-03-09T14:44:30.162 DEBUG:teuthology.orchestra.run.smithi038:> def _find_admin_socket(client_name): 2023-03-09T14:44:30.162 DEBUG:teuthology.orchestra.run.smithi038:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2023-03-09T14:44:30.162 DEBUG:teuthology.orchestra.run.smithi038:> files = glob.glob(asok_path) 2023-03-09T14:44:30.162 DEBUG:teuthology.orchestra.run.smithi038:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2023-03-09T14:44:30.163 DEBUG:teuthology.orchestra.run.smithi038:> 2023-03-09T14:44:30.163 DEBUG:teuthology.orchestra.run.smithi038:> # Given a non-glob path, it better be there 2023-03-09T14:44:30.163 DEBUG:teuthology.orchestra.run.smithi038:> if "*" not in asok_path: 2023-03-09T14:44:30.164 DEBUG:teuthology.orchestra.run.smithi038:> assert(len(files) == 1) 2023-03-09T14:44:30.164 DEBUG:teuthology.orchestra.run.smithi038:> return files[0] 2023-03-09T14:44:30.164 DEBUG:teuthology.orchestra.run.smithi038:> 2023-03-09T14:44:30.164 DEBUG:teuthology.orchestra.run.smithi038:> for f in files: 2023-03-09T14:44:30.165 DEBUG:teuthology.orchestra.run.smithi038:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-03-09T14:44:30.165 DEBUG:teuthology.orchestra.run.smithi038:> if os.path.exists("/proc/{0}".format(pid)): 2023-03-09T14:44:30.165 DEBUG:teuthology.orchestra.run.smithi038:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-03-09T14:44:30.166 DEBUG:teuthology.orchestra.run.smithi038:> contents = proc_f.read() 2023-03-09T14:44:30.166 DEBUG:teuthology.orchestra.run.smithi038:> if mountpoint in contents: 2023-03-09T14:44:30.168 DEBUG:teuthology.orchestra.run.smithi038:> return f 2023-03-09T14:44:30.168 DEBUG:teuthology.orchestra.run.smithi038:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-03-09T14:44:30.168 DEBUG:teuthology.orchestra.run.smithi038:> 2023-03-09T14:44:30.169 DEBUG:teuthology.orchestra.run.smithi038:> print(_find_admin_socket("client.admin")) 2023-03-09T14:44:30.169 DEBUG:teuthology.orchestra.run.smithi038:> ' 2023-03-09T14:44:30.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:29 smithi122 ceph-mon[93869]: pgmap v264: 129 pgs: 129 active+clean; 579 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 0 op/s 2023-03-09T14:44:30.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:29 smithi181 ceph-mon[94189]: pgmap v264: 129 pgs: 129 active+clean; 579 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s wr, 0 op/s 2023-03-09T14:44:30.284 INFO:teuthology.orchestra.run.smithi038.stdout:/var/run/ceph/ceph-client.admin.129402.asok 2023-03-09T14:44:30.466 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.129402.asok 2023-03-09T14:44:30.466 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-09T14:44:30.467 DEBUG:teuthology.orchestra.run.smithi038:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.129402.asok status 2023-03-09T14:44:30.627 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2023-03-09T14:44:30.628 INFO:teuthology.orchestra.run.smithi038.stdout: "metadata": { 2023-03-09T14:44:30.628 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_sha1": "e0ab30a2acead6db62b144ba6b6aeb23635adae7", 2023-03-09T14:44:30.628 INFO:teuthology.orchestra.run.smithi038.stdout: "ceph_version": "ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev)", 2023-03-09T14:44:30.628 INFO:teuthology.orchestra.run.smithi038.stdout: "entity_id": "admin", 2023-03-09T14:44:30.628 INFO:teuthology.orchestra.run.smithi038.stdout: "hostname": "smithi038", 2023-03-09T14:44:30.629 INFO:teuthology.orchestra.run.smithi038.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2023-03-09T14:44:30.629 INFO:teuthology.orchestra.run.smithi038.stdout: "pid": "129402", 2023-03-09T14:44:30.629 INFO:teuthology.orchestra.run.smithi038.stdout: "root": "/" 2023-03-09T14:44:30.629 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2023-03-09T14:44:30.629 INFO:teuthology.orchestra.run.smithi038.stdout: "dentry_count": 0, 2023-03-09T14:44:30.629 INFO:teuthology.orchestra.run.smithi038.stdout: "dentry_pinned_count": 0, 2023-03-09T14:44:30.629 INFO:teuthology.orchestra.run.smithi038.stdout: "id": 14754, 2023-03-09T14:44:30.630 INFO:teuthology.orchestra.run.smithi038.stdout: "inst": { 2023-03-09T14:44:30.630 INFO:teuthology.orchestra.run.smithi038.stdout: "name": { 2023-03-09T14:44:30.630 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "client", 2023-03-09T14:44:30.630 INFO:teuthology.orchestra.run.smithi038.stdout: "num": 14754 2023-03-09T14:44:30.630 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2023-03-09T14:44:30.630 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": { 2023-03-09T14:44:30.630 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2023-03-09T14:44:30.631 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "192.168.0.1:0", 2023-03-09T14:44:30.631 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3275265371 2023-03-09T14:44:30.631 INFO:teuthology.orchestra.run.smithi038.stdout: } 2023-03-09T14:44:30.631 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2023-03-09T14:44:30.631 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": { 2023-03-09T14:44:30.631 INFO:teuthology.orchestra.run.smithi038.stdout: "type": "v1", 2023-03-09T14:44:30.632 INFO:teuthology.orchestra.run.smithi038.stdout: "addr": "192.168.0.1:0", 2023-03-09T14:44:30.632 INFO:teuthology.orchestra.run.smithi038.stdout: "nonce": 3275265371 2023-03-09T14:44:30.632 INFO:teuthology.orchestra.run.smithi038.stdout: }, 2023-03-09T14:44:30.632 INFO:teuthology.orchestra.run.smithi038.stdout: "inst_str": "client.14754 192.168.0.1:0/3275265371", 2023-03-09T14:44:30.632 INFO:teuthology.orchestra.run.smithi038.stdout: "addr_str": "192.168.0.1:0/3275265371", 2023-03-09T14:44:30.632 INFO:teuthology.orchestra.run.smithi038.stdout: "inode_count": 1, 2023-03-09T14:44:30.632 INFO:teuthology.orchestra.run.smithi038.stdout: "mds_epoch": 17, 2023-03-09T14:44:30.633 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_epoch": 87, 2023-03-09T14:44:30.633 INFO:teuthology.orchestra.run.smithi038.stdout: "osd_epoch_barrier": 0, 2023-03-09T14:44:30.633 INFO:teuthology.orchestra.run.smithi038.stdout: "blocklisted": false, 2023-03-09T14:44:30.633 INFO:teuthology.orchestra.run.smithi038.stdout: "fs_name": "cephfs" 2023-03-09T14:44:30.633 INFO:teuthology.orchestra.run.smithi038.stdout:} 2023-03-09T14:44:30.637 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-09T14:44:30.638 DEBUG:teuthology.orchestra.run.smithi038:> stat --file-system '--printf=%T 2023-03-09T14:44:30.638 DEBUG:teuthology.orchestra.run.smithi038:> ' -- /home/ubuntu/cephtest/mnt.admin 2023-03-09T14:44:30.656 INFO:teuthology.orchestra.run.smithi038.stdout:fuseblk 2023-03-09T14:44:30.656 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2023-03-09T14:44:30.657 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-09T14:44:30.657 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2023-03-09T14:44:30.725 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-09T14:44:30.725 DEBUG:teuthology.orchestra.run.smithi038:> (cd /home/ubuntu/cephtest/mnt.admin && exec bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2023-03-09T14:44:30.823 INFO:teuthology.orchestra.run.smithi038.stdout:# file: . 2023-03-09T14:44:30.823 INFO:teuthology.orchestra.run.smithi038.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2023-03-09T14:44:30.823 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:44:30.824 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2023-03-09T14:44:30.824 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-09T14:44:30.842 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-09T14:44:31.008 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.004+0000 7f49ff132700 1 -- 172.21.15.38:0/4225017685 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f49f8103260 msgr2=0x7f49f8105640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:31.009 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.004+0000 7f49ff132700 1 --2- 172.21.15.38:0/4225017685 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f49f8103260 0x7f49f8105640 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f49e4009990 tx=0x7f49e4009ca0 comp rx=0 tx=0).stop 2023-03-09T14:44:31.009 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.004+0000 7f49ff132700 1 -- 172.21.15.38:0/4225017685 shutdown_connections 2023-03-09T14:44:31.009 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.004+0000 7f49ff132700 1 --2- 172.21.15.38:0/4225017685 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f49f8105b80 0x7f49f8107fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:31.009 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.004+0000 7f49ff132700 1 --2- 172.21.15.38:0/4225017685 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f49f8100940 0x7f49f8102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:31.009 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.004+0000 7f49ff132700 1 --2- 172.21.15.38:0/4225017685 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f49f8103260 0x7f49f8105640 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:31.009 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.004+0000 7f49ff132700 1 -- 172.21.15.38:0/4225017685 >> 172.21.15.38:0/4225017685 conn(0x7f49f80fa380 msgr2=0x7f49f80fc7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:31.010 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.004+0000 7f49ff132700 1 -- 172.21.15.38:0/4225017685 shutdown_connections 2023-03-09T14:44:31.010 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.004+0000 7f49ff132700 1 -- 172.21.15.38:0/4225017685 wait complete. 2023-03-09T14:44:31.010 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.004+0000 7f49ff132700 1 Processor -- start 2023-03-09T14:44:31.010 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.004+0000 7f49ff132700 1 -- start start 2023-03-09T14:44:31.010 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.004+0000 7f49ff132700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f49f8100940 0x7f49f8195c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:31.010 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.004+0000 7f49ff132700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f49f8103260 0x7f49f81961b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:31.010 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.004+0000 7f49ff132700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f49f8105b80 0x7f49f819d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:31.011 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.004+0000 7f49ff132700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49f819d7b0 con 0x7f49f8103260 2023-03-09T14:44:31.011 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.004+0000 7f49ff132700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49f819d920 con 0x7f49f8100940 2023-03-09T14:44:31.011 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.005+0000 7f49ff132700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49f819dae0 con 0x7f49f8105b80 2023-03-09T14:44:31.011 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.005+0000 7f49f7fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f49f8103260 0x7f49f81961b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:31.011 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.005+0000 7f49f7fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f49f8103260 0x7f49f81961b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36872/0 (socket says 172.21.15.38:36872) 2023-03-09T14:44:31.011 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.005+0000 7f49f7fff700 1 -- 172.21.15.38:0/1761062378 learned_addr learned my addr 172.21.15.38:0/1761062378 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:44:31.011 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.005+0000 7f49fcece700 1 --2- 172.21.15.38:0/1761062378 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f49f8100940 0x7f49f8195c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:31.011 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.005+0000 7f49fd6cf700 1 --2- 172.21.15.38:0/1761062378 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f49f8105b80 0x7f49f819d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:31.012 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.005+0000 7f49f7fff700 1 -- 172.21.15.38:0/1761062378 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f49f8105b80 msgr2=0x7f49f819d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:31.012 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.005+0000 7f49f7fff700 1 --2- 172.21.15.38:0/1761062378 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f49f8105b80 0x7f49f819d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:31.012 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.005+0000 7f49f7fff700 1 -- 172.21.15.38:0/1761062378 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f49f8100940 msgr2=0x7f49f8195c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:31.012 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.005+0000 7f49f7fff700 1 --2- 172.21.15.38:0/1761062378 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f49f8100940 0x7f49f8195c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:31.012 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.005+0000 7f49f7fff700 1 -- 172.21.15.38:0/1761062378 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49e40095f0 con 0x7f49f8103260 2023-03-09T14:44:31.012 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.005+0000 7f49fcece700 1 --2- 172.21.15.38:0/1761062378 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f49f8100940 0x7f49f8195c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-09T14:44:31.012 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.005+0000 7f49fd6cf700 1 --2- 172.21.15.38:0/1761062378 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f49f8105b80 0x7f49f819d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-09T14:44:31.013 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.005+0000 7f49f7fff700 1 --2- 172.21.15.38:0/1761062378 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f49f8103260 0x7f49f81961b0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f49e4006010 tx=0x7f49e400f7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:31.013 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.005+0000 7f49f5ffb700 1 -- 172.21.15.38:0/1761062378 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49e401d070 con 0x7f49f8103260 2023-03-09T14:44:31.013 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.005+0000 7f49f5ffb700 1 -- 172.21.15.38:0/1761062378 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f49e400fca0 con 0x7f49f8103260 2023-03-09T14:44:31.013 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.005+0000 7f49ff132700 1 -- 172.21.15.38:0/1761062378 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49f810b830 con 0x7f49f8103260 2023-03-09T14:44:31.013 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.005+0000 7f49f5ffb700 1 -- 172.21.15.38:0/1761062378 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49e40176e0 con 0x7f49f8103260 2023-03-09T14:44:31.014 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.006+0000 7f49ff132700 1 -- 172.21.15.38:0/1761062378 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49f810b6a0 con 0x7f49f8103260 2023-03-09T14:44:31.015 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.007+0000 7f49ff132700 1 -- 172.21.15.38:0/1761062378 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49f80fbef0 con 0x7f49f8103260 2023-03-09T14:44:31.015 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.008+0000 7f49f5ffb700 1 -- 172.21.15.38:0/1761062378 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 133198+0+0 (secure 0 0 0) 0x7f49e400c040 con 0x7f49f8103260 2023-03-09T14:44:31.015 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.009+0000 7f49f5ffb700 1 --2- 172.21.15.38:0/1761062378 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f49e809f3c0 0x7f49e80a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:31.015 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.009+0000 7f49fcece700 1 --2- 172.21.15.38:0/1761062378 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f49e809f3c0 0x7f49e80a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:31.015 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.009+0000 7f49f5ffb700 1 -- 172.21.15.38:0/1761062378 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f49e40cb500 con 0x7f49f8103260 2023-03-09T14:44:31.015 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.009+0000 7f49fcece700 1 --2- 172.21.15.38:0/1761062378 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f49e809f3c0 0x7f49e80a1870 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f49ec005fd0 tx=0x7f49ec005de0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:31.016 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.012+0000 7f49f5ffb700 1 -- 172.21.15.38:0/1761062378 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f49e4095760 con 0x7f49f8103260 2023-03-09T14:44:31.216 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.212+0000 7f49ff132700 1 -- 172.21.15.38:0/1761062378 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f49f8061390 con 0x7f49f8103260 2023-03-09T14:44:31.217 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.214+0000 7f49f5ffb700 1 -- 172.21.15.38:0/1761062378 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+23973 (secure 0 0 0) 0x7f49e4098d80 con 0x7f49f8103260 2023-03-09T14:44:31.218 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:44:31.218 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":87,"fsid":"c6733fe2-be87-11ed-9af2-001a4aab830c","created":"2023-03-09T14:37:14.947981+0000","modified":"2023-03-09T14:44:23.900228+0000","last_up_change":"2023-03-09T14:43:37.198407+0000","last_in_change":"2023-03-09T14:43:19.550566+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-09T14:40:23.196665+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-03-09T14:44:17.484374+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-03-09T14:44:18.457833+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-03-09T14:44:21.895589+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"51477039-15ee-4707-a34d-4ecfa18bebde","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6803","nonce":1508072290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6805","nonce":1508072290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6809","nonce":1508072290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6807","nonce":1508072290}]},"public_addr":"172.21.15.38:6803/1508072290","cluster_addr":"172.21.15.38:6805/1508072290","heartbeat_back_addr":"172.21.15.38:6809/1508072290","heartbeat_front_addr":"172.21.15.38:6807/1508072290","state":["exists","up"]},{"osd":1,"uuid":"71f8a718-b0a9-4f39-bc54-267aa42a2578","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6801","nonce":3929268773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6803","nonce":3929268773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6807","nonce":3929268773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6805","nonce":3929268773}]},"public_addr":"172.21.15.122:6801/3929268773","cluster_addr":"172.21.15.122:6803/3929268773","heartbeat_back_addr":"172.21.15.122:6807/3929268773","heartbeat_front_addr":"172.21.15.122:6805/3929268773","state":["exists","up"]},{"osd":2,"uuid":"961a2cbe-140e-437f-b131-1fcac573b19c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6801","nonce":1310630803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6803","nonce":1310630803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6807","nonce":1310630803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6805","nonce":1310630803}]},"public_addr":"172.21.15.181:6801/1310630803","cluster_addr":"172.21.15.181:6803/1310630803","heartbeat_back_addr":"172.21.15.181:6807/1310630803","heartbeat_front_addr":"172.21.15.181:6805/1310630803","state":["exists","up"]},{"osd":3,"uuid":"969df8fa-2d19-42c7-9f2a-5c5fac15d4a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6811","nonce":2719814100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6813","nonce":2719814100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6817","nonce":2719814100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6815","nonce":2719814100}]},"public_addr":"172.21.15.38:6811/2719814100","cluster_addr":"172.21.15.38:6813/2719814100","heartbeat_back_addr":"172.21.15.38:6817/2719814100","heartbeat_front_addr":"172.21.15.38:6815/2719814100","state":["exists","up"]},{"osd":4,"uuid":"44a6dce3-e06e-4cbd-9404-62a86d01ab6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6809","nonce":2480764994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6811","nonce":2480764994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6815","nonce":2480764994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6813","nonce":2480764994}]},"public_addr":"172.21.15.122:6809/2480764994","cluster_addr":"172.21.15.122:6811/2480764994","heartbeat_back_addr":"172.21.15.122:6815/2480764994","heartbeat_front_addr":"172.21.15.122:6813/2480764994","state":["exists","up"]},{"osd":5,"uuid":"49749574-c00c-4e3a-80e6-e3686df58d5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6809","nonce":3141346246}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6811","nonce":3141346246}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6815","nonce":3141346246}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6813","nonce":3141346246}]},"public_addr":"172.21.15.181:6809/3141346246","cluster_addr":"172.21.15.181:6811/3141346246","heartbeat_back_addr":"172.21.15.181:6815/3141346246","heartbeat_front_addr":"172.21.15.181:6813/3141346246","state":["exists","up"]},{"osd":6,"uuid":"61c0437e-3f51-45a9-8a61-e98eea0093f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6819","nonce":2122788957}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6821","nonce":2122788957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6825","nonce":2122788957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6823","nonce":2122788957}]},"public_addr":"172.21.15.38:6819/2122788957","cluster_addr":"172.21.15.38:6821/2122788957","heartbeat_back_addr":"172.21.15.38:6825/2122788957","heartbeat_front_addr":"172.21.15.38:6823/2122788957","state":["exists","up"]},{"osd":7,"uuid":"231fecdd-939f-4c7b-83a1-c4a1cffd7139","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6817","nonce":656874139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6819","nonce":656874139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6823","nonce":656874139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6821","nonce":656874139}]},"public_addr":"172.21.15.122:6817/656874139","cluster_addr":"172.21.15.122:6819/656874139","heartbeat_back_addr":"172.21.15.122:6823/656874139","heartbeat_front_addr":"172.21.15.122:6821/656874139","state":["exists","up"]},{"osd":8,"uuid":"297e6d6d-85fd-43ce-9028-5e9dbe55c36e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6817","nonce":755000703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6819","nonce":755000703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6823","nonce":755000703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6821","nonce":755000703}]},"public_addr":"172.21.15.181:6817/755000703","cluster_addr":"172.21.15.181:6819/755000703","heartbeat_back_addr":"172.21.15.181:6823/755000703","heartbeat_front_addr":"172.21.15.181:6821/755000703","state":["exists","up"]},{"osd":9,"uuid":"94cc0355-5a1a-4cfc-bd0c-562b6b41127e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6827","nonce":2624555969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6829","nonce":2624555969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6833","nonce":2624555969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6831","nonce":2624555969}]},"public_addr":"172.21.15.38:6827/2624555969","cluster_addr":"172.21.15.38:6829/2624555969","heartbeat_back_addr":"172.21.15.38:6833/2624555969","heartbeat_front_addr":"172.21.15.38:6831/2624555969","state":["exists","up"]},{"osd":10,"uuid":"f4f2e042-8d92-46bf-93b8-8b0f04795c2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6825","nonce":1427785280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6827","nonce":1427785280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6831","nonce":1427785280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6829","nonce":1427785280}]},"public_addr":"172.21.15.122:6825/1427785280","cluster_addr":"172.21.15.122:6827/1427785280","heartbeat_back_addr":"172.21.15.122:6831/1427785280","heartbeat_front_addr":"172.21.15.122:6829/1427785280","state":["exists","up"]},{"osd":11,"uuid":"07444936-eae5-4097-a799-60294cafe80b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6825","nonce":3183488640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6827","nonce":3183488640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6831","nonce":3183488640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6829","nonce":3183488640}]},"public_addr":"172.21.15.181:6825/3183488640","cluster_addr":"172.21.15.181:6827/3183488640","heartbeat_back_addr":"172.21.15.181:6831/3183488640","heartbeat_front_addr":"172.21.15.181:6829/3183488640","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:39:37.657556+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:39:58.960074+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:40:20.326952+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:40:42.244387+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:41:03.946973+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:41:25.097985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:41:46.877916+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:42:08.413827+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:42:29.822009+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:42:51.776684+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:43:13.937249+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:43:35.569199+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/4272288735":"2023-03-10T14:38:14.612767+0000","172.21.15.38:6801/3381625669":"2023-03-10T14:38:14.612767+0000","172.21.15.38:6800/3381625669":"2023-03-10T14:38:14.612767+0000","172.21.15.38:6800/3060474342":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/4253918615":"2023-03-10T14:38:14.612767+0000","172.21.15.38:0/3496083181":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/245602368":"2023-03-10T14:37:47.517775+0000","172.21.15.38:6801/3060474342":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/3622589182":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/1430302251":"2023-03-10T14:38:14.612767+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-09T14:44:31.219 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.216+0000 7f49ff132700 1 -- 172.21.15.38:0/1761062378 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f49e809f3c0 msgr2=0x7f49e80a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:31.219 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.216+0000 7f49ff132700 1 --2- 172.21.15.38:0/1761062378 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f49e809f3c0 0x7f49e80a1870 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f49ec005fd0 tx=0x7f49ec005de0 comp rx=0 tx=0).stop 2023-03-09T14:44:31.220 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.216+0000 7f49ff132700 1 -- 172.21.15.38:0/1761062378 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f49f8103260 msgr2=0x7f49f81961b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:31.220 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.216+0000 7f49ff132700 1 --2- 172.21.15.38:0/1761062378 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f49f8103260 0x7f49f81961b0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f49e4006010 tx=0x7f49e400f7b0 comp rx=0 tx=0).stop 2023-03-09T14:44:31.220 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.216+0000 7f49ff132700 1 -- 172.21.15.38:0/1761062378 shutdown_connections 2023-03-09T14:44:31.220 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.216+0000 7f49ff132700 1 --2- 172.21.15.38:0/1761062378 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f49e809f3c0 0x7f49e80a1870 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:31.220 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.216+0000 7f49ff132700 1 --2- 172.21.15.38:0/1761062378 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f49f8105b80 0x7f49f819d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:31.220 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.216+0000 7f49ff132700 1 --2- 172.21.15.38:0/1761062378 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f49f8100940 0x7f49f8195c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:31.220 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.216+0000 7f49ff132700 1 --2- 172.21.15.38:0/1761062378 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f49f8103260 0x7f49f81961b0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:31.221 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.216+0000 7f49ff132700 1 -- 172.21.15.38:0/1761062378 >> 172.21.15.38:0/1761062378 conn(0x7f49f80fa380 msgr2=0x7f49f81015a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:31.221 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.216+0000 7f49ff132700 1 -- 172.21.15.38:0/1761062378 shutdown_connections 2023-03-09T14:44:31.221 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.216+0000 7f49ff132700 1 -- 172.21.15.38:0/1761062378 wait complete. 2023-03-09T14:44:31.232 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-09T14:40:23.196665+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-03-09T14:44:17.484374+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-03-09T14:44:18.457833+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-03-09T14:44:21.895589+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-03-09T14:44:31.233 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-09T14:44:31.362 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.358+0000 7f7997847700 1 -- 172.21.15.38:0/1309794028 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7990100910 msgr2=0x7f79901071d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:31.362 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.358+0000 7f7997847700 1 --2- 172.21.15.38:0/1309794028 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7990100910 0x7f79901071d0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f798800b0a0 tx=0x7f798800b3b0 comp rx=0 tx=0).stop 2023-03-09T14:44:31.362 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.358+0000 7f7997847700 1 -- 172.21.15.38:0/1309794028 shutdown_connections 2023-03-09T14:44:31.363 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.358+0000 7f7997847700 1 --2- 172.21.15.38:0/1309794028 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7990100910 0x7f79901071d0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:31.363 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.358+0000 7f7997847700 1 --2- 172.21.15.38:0/1309794028 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f79900fed90 0x7f79900ff1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:31.363 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.358+0000 7f7997847700 1 --2- 172.21.15.38:0/1309794028 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f79900fff80 0x7f79901003d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:31.363 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.358+0000 7f7997847700 1 -- 172.21.15.38:0/1309794028 >> 172.21.15.38:0/1309794028 conn(0x7f79900fa3b0 msgr2=0x7f79900fc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:31.363 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.358+0000 7f7997847700 1 -- 172.21.15.38:0/1309794028 shutdown_connections 2023-03-09T14:44:31.363 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.358+0000 7f7997847700 1 -- 172.21.15.38:0/1309794028 wait complete. 2023-03-09T14:44:31.363 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.359+0000 7f7997847700 1 Processor -- start 2023-03-09T14:44:31.363 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.359+0000 7f7997847700 1 -- start start 2023-03-09T14:44:31.365 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.359+0000 7f7997847700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f79900fed90 0x7f7990197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:31.365 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.359+0000 7f7997847700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f79900fff80 0x7f7990198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:31.365 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.359+0000 7f7997847700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7990100910 0x7f799019f490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:31.366 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.359+0000 7f7997847700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f799019fa60 con 0x7f79900fff80 2023-03-09T14:44:31.366 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.359+0000 7f7997847700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f799019fba0 con 0x7f7990100910 2023-03-09T14:44:31.366 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.359+0000 7f7997847700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f799019fd30 con 0x7f79900fed90 2023-03-09T14:44:31.366 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.359+0000 7f7994de2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f79900fff80 0x7f7990198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:31.366 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.359+0000 7f7994de2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f79900fff80 0x7f7990198420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36886/0 (socket says 172.21.15.38:36886) 2023-03-09T14:44:31.366 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.359+0000 7f7994de2700 1 -- 172.21.15.38:0/1676035934 learned_addr learned my addr 172.21.15.38:0/1676035934 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:44:31.366 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.359+0000 7f79955e3700 1 --2- 172.21.15.38:0/1676035934 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f79900fed90 0x7f7990197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:31.366 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.359+0000 7f7995de4700 1 --2- 172.21.15.38:0/1676035934 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7990100910 0x7f799019f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:31.367 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.360+0000 7f7994de2700 1 -- 172.21.15.38:0/1676035934 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f79900fed90 msgr2=0x7f7990197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:31.367 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.360+0000 7f7994de2700 1 --2- 172.21.15.38:0/1676035934 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f79900fed90 0x7f7990197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:31.367 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.360+0000 7f7994de2700 1 -- 172.21.15.38:0/1676035934 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7990100910 msgr2=0x7f799019f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:31.367 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.360+0000 7f7994de2700 1 --2- 172.21.15.38:0/1676035934 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7990100910 0x7f799019f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:31.367 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.360+0000 7f7994de2700 1 -- 172.21.15.38:0/1676035934 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7988009d00 con 0x7f79900fff80 2023-03-09T14:44:31.367 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.360+0000 7f7994de2700 1 --2- 172.21.15.38:0/1676035934 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f79900fff80 0x7f7990198420 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f798400d950 tx=0x7f798400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:31.367 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.360+0000 7f79827fc700 1 -- 172.21.15.38:0/1676035934 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7984009940 con 0x7f79900fff80 2023-03-09T14:44:31.368 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.360+0000 7f79827fc700 1 -- 172.21.15.38:0/1676035934 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7984010460 con 0x7f79900fff80 2023-03-09T14:44:31.368 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.360+0000 7f79827fc700 1 -- 172.21.15.38:0/1676035934 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f798400f620 con 0x7f79900fff80 2023-03-09T14:44:31.368 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.360+0000 7f7997847700 1 -- 172.21.15.38:0/1676035934 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f799010da30 con 0x7f79900fff80 2023-03-09T14:44:31.368 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.360+0000 7f7997847700 1 -- 172.21.15.38:0/1676035934 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f799010d8a0 con 0x7f79900fff80 2023-03-09T14:44:31.368 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.361+0000 7f7997847700 1 -- 172.21.15.38:0/1676035934 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79901a0360 con 0x7f79900fff80 2023-03-09T14:44:31.369 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.363+0000 7f79827fc700 1 -- 172.21.15.38:0/1676035934 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 133198+0+0 (secure 0 0 0) 0x7f7984022070 con 0x7f79900fff80 2023-03-09T14:44:31.369 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.363+0000 7f79827fc700 1 --2- 172.21.15.38:0/1676035934 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f797c09f420 0x7f797c0a18d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:31.369 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.364+0000 7f79955e3700 1 --2- 172.21.15.38:0/1676035934 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f797c09f420 0x7f797c0a18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:31.369 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.364+0000 7f79827fc700 1 -- 172.21.15.38:0/1676035934 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f79840c9c10 con 0x7f79900fff80 2023-03-09T14:44:31.369 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.364+0000 7f79955e3700 1 --2- 172.21.15.38:0/1676035934 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f797c09f420 0x7f797c0a18d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f79900ffde0 tx=0x7f798c007400 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:31.370 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.367+0000 7f79827fc700 1 -- 172.21.15.38:0/1676035934 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f7984093d80 con 0x7f79900fff80 2023-03-09T14:44:31.574 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.571+0000 7f7997847700 1 -- 172.21.15.38:0/1676035934 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7990105c40 con 0x7f79900fff80 2023-03-09T14:44:31.574 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.571+0000 7f79827fc700 1 -- 172.21.15.38:0/1676035934 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7984020020 con 0x7f79900fff80 2023-03-09T14:44:31.575 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 1 2023-03-09T14:44:31.577 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.573+0000 7f7997847700 1 -- 172.21.15.38:0/1676035934 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f797c09f420 msgr2=0x7f797c0a18d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:31.577 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.573+0000 7f7997847700 1 --2- 172.21.15.38:0/1676035934 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f797c09f420 0x7f797c0a18d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f79900ffde0 tx=0x7f798c007400 comp rx=0 tx=0).stop 2023-03-09T14:44:31.577 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.573+0000 7f7997847700 1 -- 172.21.15.38:0/1676035934 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f79900fff80 msgr2=0x7f7990198420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:31.577 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.573+0000 7f7997847700 1 --2- 172.21.15.38:0/1676035934 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f79900fff80 0x7f7990198420 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f798400d950 tx=0x7f798400dc60 comp rx=0 tx=0).stop 2023-03-09T14:44:31.578 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.573+0000 7f7997847700 1 -- 172.21.15.38:0/1676035934 shutdown_connections 2023-03-09T14:44:31.578 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.573+0000 7f7997847700 1 --2- 172.21.15.38:0/1676035934 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f797c09f420 0x7f797c0a18d0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:31.578 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.573+0000 7f7997847700 1 --2- 172.21.15.38:0/1676035934 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7990100910 0x7f799019f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:31.578 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.573+0000 7f7997847700 1 --2- 172.21.15.38:0/1676035934 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f79900fed90 0x7f7990197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:31.578 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.573+0000 7f7997847700 1 --2- 172.21.15.38:0/1676035934 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f79900fff80 0x7f7990198420 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:31.578 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.573+0000 7f7997847700 1 -- 172.21.15.38:0/1676035934 >> 172.21.15.38:0/1676035934 conn(0x7f79900fa3b0 msgr2=0x7f79900fc5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:31.578 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.573+0000 7f7997847700 1 -- 172.21.15.38:0/1676035934 shutdown_connections 2023-03-09T14:44:31.578 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.573+0000 7f7997847700 1 -- 172.21.15.38:0/1676035934 wait complete. 2023-03-09T14:44:31.589 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-03-09T14:44:31.719 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.713+0000 7f72168b4700 1 -- 172.21.15.38:0/1973391673 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72100ffe80 msgr2=0x7f7210100290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:31.719 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.713+0000 7f72168b4700 1 --2- 172.21.15.38:0/1973391673 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72100ffe80 0x7f7210100290 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f7200009990 tx=0x7f7200009ca0 comp rx=0 tx=0).stop 2023-03-09T14:44:31.720 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.713+0000 7f72168b4700 1 -- 172.21.15.38:0/1973391673 shutdown_connections 2023-03-09T14:44:31.720 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.713+0000 7f72168b4700 1 --2- 172.21.15.38:0/1973391673 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f72100ffe80 0x7f7210100290 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:31.720 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.713+0000 7f72168b4700 1 --2- 172.21.15.38:0/1973391673 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7210101a10 0x7f7210108340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:31.720 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.713+0000 7f72168b4700 1 --2- 172.21.15.38:0/1973391673 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7210101060 0x7f72101014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:31.720 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.713+0000 7f72168b4700 1 -- 172.21.15.38:0/1973391673 >> 172.21.15.38:0/1973391673 conn(0x7f72100fb510 msgr2=0x7f72100fd920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:31.720 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.713+0000 7f72168b4700 1 -- 172.21.15.38:0/1973391673 shutdown_connections 2023-03-09T14:44:31.720 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.713+0000 7f72168b4700 1 -- 172.21.15.38:0/1973391673 wait complete. 2023-03-09T14:44:31.721 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.714+0000 7f72168b4700 1 Processor -- start 2023-03-09T14:44:31.721 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.714+0000 7f72168b4700 1 -- start start 2023-03-09T14:44:31.721 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.714+0000 7f72168b4700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f72100ffe80 0x7f7210074530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:31.721 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.714+0000 7f72168b4700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7210101060 0x7f7210072b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:31.721 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.714+0000 7f72168b4700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7210101a10 0x7f72100730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:31.721 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.714+0000 7f72168b4700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7210073850 con 0x7f7210101a10 2023-03-09T14:44:31.721 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.714+0000 7f72168b4700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72100739c0 con 0x7f7210101060 2023-03-09T14:44:31.722 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.714+0000 7f72168b4700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f721019fcc0 con 0x7f72100ffe80 2023-03-09T14:44:31.722 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.714+0000 7f7214e51700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7210101a10 0x7f72100730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:31.722 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.714+0000 7f7214e51700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7210101a10 0x7f72100730b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36912/0 (socket says 172.21.15.38:36912) 2023-03-09T14:44:31.722 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.714+0000 7f7214e51700 1 -- 172.21.15.38:0/2825414481 learned_addr learned my addr 172.21.15.38:0/2825414481 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:44:31.722 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.714+0000 7f720f7fe700 1 --2- 172.21.15.38:0/2825414481 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7210101060 0x7f7210072b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:31.722 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.715+0000 7f720ffff700 1 --2- 172.21.15.38:0/2825414481 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f72100ffe80 0x7f7210074530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:31.722 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.715+0000 7f7214e51700 1 -- 172.21.15.38:0/2825414481 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f72100ffe80 msgr2=0x7f7210074530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:31.723 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.715+0000 7f7214e51700 1 --2- 172.21.15.38:0/2825414481 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f72100ffe80 0x7f7210074530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:31.723 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.715+0000 7f7214e51700 1 -- 172.21.15.38:0/2825414481 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7210101060 msgr2=0x7f7210072b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:31.723 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.715+0000 7f7214e51700 1 --2- 172.21.15.38:0/2825414481 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7210101060 0x7f7210072b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:31.723 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.715+0000 7f7214e51700 1 -- 172.21.15.38:0/2825414481 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72000095f0 con 0x7f7210101a10 2023-03-09T14:44:31.723 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.715+0000 7f7214e51700 1 --2- 172.21.15.38:0/2825414481 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7210101a10 0x7f72100730b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f7208007ee0 tx=0x7f720800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:31.723 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.715+0000 7f720d7fa700 1 -- 172.21.15.38:0/2825414481 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7208017070 con 0x7f7210101a10 2023-03-09T14:44:31.723 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.715+0000 7f72168b4700 1 -- 172.21.15.38:0/2825414481 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f721010b860 con 0x7f7210101a10 2023-03-09T14:44:31.723 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.715+0000 7f720d7fa700 1 -- 172.21.15.38:0/2825414481 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f720800f040 con 0x7f7210101a10 2023-03-09T14:44:31.724 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.715+0000 7f720d7fa700 1 -- 172.21.15.38:0/2825414481 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7208005580 con 0x7f7210101a10 2023-03-09T14:44:31.724 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.715+0000 7f72168b4700 1 -- 172.21.15.38:0/2825414481 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f721010b6d0 con 0x7f7210101a10 2023-03-09T14:44:31.724 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.716+0000 7f72168b4700 1 -- 172.21.15.38:0/2825414481 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7210106e80 con 0x7f7210101a10 2023-03-09T14:44:31.725 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.717+0000 7f720d7fa700 1 -- 172.21.15.38:0/2825414481 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 133198+0+0 (secure 0 0 0) 0x7f7208004c40 con 0x7f7210101a10 2023-03-09T14:44:31.725 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.718+0000 7f720d7fa700 1 --2- 172.21.15.38:0/2825414481 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f71f809f3c0 0x7f71f80a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:31.725 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.718+0000 7f720d7fa700 1 -- 172.21.15.38:0/2825414481 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f72080c9e90 con 0x7f7210101a10 2023-03-09T14:44:31.725 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.718+0000 7f720ffff700 1 --2- 172.21.15.38:0/2825414481 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f71f809f3c0 0x7f71f80a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:31.725 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.718+0000 7f720ffff700 1 --2- 172.21.15.38:0/2825414481 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f71f809f3c0 0x7f71f80a1870 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f720000b2e0 tx=0x7f7200005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:31.725 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.721+0000 7f720d7fa700 1 -- 172.21.15.38:0/2825414481 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f7208094060 con 0x7f7210101a10 2023-03-09T14:44:31.925 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.922+0000 7f72168b4700 1 -- 172.21.15.38:0/2825414481 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7210066dd0 con 0x7f7210101a10 2023-03-09T14:44:31.926 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.922+0000 7f720d7fa700 1 -- 172.21.15.38:0/2825414481 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7208097680 con 0x7f7210101a10 2023-03-09T14:44:31.926 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 32 2023-03-09T14:44:31.928 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.925+0000 7f72168b4700 1 -- 172.21.15.38:0/2825414481 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f71f809f3c0 msgr2=0x7f71f80a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:31.928 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.925+0000 7f72168b4700 1 --2- 172.21.15.38:0/2825414481 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f71f809f3c0 0x7f71f80a1870 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f720000b2e0 tx=0x7f7200005fb0 comp rx=0 tx=0).stop 2023-03-09T14:44:31.928 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.925+0000 7f72168b4700 1 -- 172.21.15.38:0/2825414481 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7210101a10 msgr2=0x7f72100730b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:31.928 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.925+0000 7f72168b4700 1 --2- 172.21.15.38:0/2825414481 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7210101a10 0x7f72100730b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f7208007ee0 tx=0x7f720800d3b0 comp rx=0 tx=0).stop 2023-03-09T14:44:31.929 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.925+0000 7f72168b4700 1 -- 172.21.15.38:0/2825414481 shutdown_connections 2023-03-09T14:44:31.929 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.925+0000 7f72168b4700 1 --2- 172.21.15.38:0/2825414481 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7210101a10 0x7f72100730b0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:31.929 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.925+0000 7f72168b4700 1 --2- 172.21.15.38:0/2825414481 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f71f809f3c0 0x7f71f80a1870 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:31.929 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.925+0000 7f72168b4700 1 --2- 172.21.15.38:0/2825414481 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f72100ffe80 0x7f7210074530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:31.929 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.925+0000 7f72168b4700 1 --2- 172.21.15.38:0/2825414481 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7210101060 0x7f7210072b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:31.929 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.925+0000 7f72168b4700 1 -- 172.21.15.38:0/2825414481 >> 172.21.15.38:0/2825414481 conn(0x7f72100fb510 msgr2=0x7f72100fd7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:31.930 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.925+0000 7f72168b4700 1 -- 172.21.15.38:0/2825414481 shutdown_connections 2023-03-09T14:44:31.930 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:31.925+0000 7f72168b4700 1 -- 172.21.15.38:0/2825414481 wait complete. 2023-03-09T14:44:31.939 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:31 smithi038 ceph-mon[92422]: pgmap v265: 129 pgs: 129 active+clean; 579 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 0 op/s 2023-03-09T14:44:31.939 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:31 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/1761062378' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-03-09T14:44:31.939 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:31 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/1676035934' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-03-09T14:44:31.940 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-03-09T14:44:32.073 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.068+0000 7f2941456700 1 -- 172.21.15.38:0/3805036345 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f293c0fee00 msgr2=0x7f293c0ff210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:32.074 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.068+0000 7f2941456700 1 --2- 172.21.15.38:0/3805036345 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f293c0fee00 0x7f293c0ff210 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f292c009990 tx=0x7f292c009ca0 comp rx=0 tx=0).stop 2023-03-09T14:44:32.074 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.068+0000 7f2941456700 1 -- 172.21.15.38:0/3805036345 shutdown_connections 2023-03-09T14:44:32.074 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.068+0000 7f2941456700 1 --2- 172.21.15.38:0/3805036345 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f293c0fee00 0x7f293c0ff210 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:32.074 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.068+0000 7f2941456700 1 --2- 172.21.15.38:0/3805036345 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f293c100980 0x7f293c107240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:32.074 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.068+0000 7f2941456700 1 --2- 172.21.15.38:0/3805036345 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f293c0ffff0 0x7f293c100440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:32.074 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.068+0000 7f2941456700 1 -- 172.21.15.38:0/3805036345 >> 172.21.15.38:0/3805036345 conn(0x7f293c0fa420 msgr2=0x7f293c0fc850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:32.074 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.068+0000 7f2941456700 1 -- 172.21.15.38:0/3805036345 shutdown_connections 2023-03-09T14:44:32.075 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.068+0000 7f2941456700 1 -- 172.21.15.38:0/3805036345 wait complete. 2023-03-09T14:44:32.075 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.069+0000 7f2941456700 1 Processor -- start 2023-03-09T14:44:32.075 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.069+0000 7f2941456700 1 -- start start 2023-03-09T14:44:32.075 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.069+0000 7f2941456700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f293c0fee00 0x7f293c069820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:32.075 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.069+0000 7f2941456700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f293c0ffff0 0x7f293c19a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:32.075 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.069+0000 7f2941456700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f293c100980 0x7f293c19cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:32.075 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.069+0000 7f2941456700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f293c19d7c0 con 0x7f293c0fee00 2023-03-09T14:44:32.076 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.069+0000 7f2941456700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f293c19d930 con 0x7f293c100980 2023-03-09T14:44:32.076 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.069+0000 7f2941456700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f293c19daf0 con 0x7f293c0ffff0 2023-03-09T14:44:32.076 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.069+0000 7f293affd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f293c0fee00 0x7f293c069820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:32.076 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.070+0000 7f293affd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f293c0fee00 0x7f293c069820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36922/0 (socket says 172.21.15.38:36922) 2023-03-09T14:44:32.076 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.070+0000 7f293affd700 1 -- 172.21.15.38:0/200696901 learned_addr learned my addr 172.21.15.38:0/200696901 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:44:32.076 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.070+0000 7f293b7fe700 1 --2- 172.21.15.38:0/200696901 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f293c100980 0x7f293c19cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:32.076 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.070+0000 7f293a7fc700 1 --2- 172.21.15.38:0/200696901 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f293c0ffff0 0x7f293c19a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:32.077 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.070+0000 7f293affd700 1 -- 172.21.15.38:0/200696901 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f293c0ffff0 msgr2=0x7f293c19a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:32.077 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.070+0000 7f293affd700 1 --2- 172.21.15.38:0/200696901 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f293c0ffff0 0x7f293c19a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:32.077 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.070+0000 7f293affd700 1 -- 172.21.15.38:0/200696901 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f293c100980 msgr2=0x7f293c19cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:32.077 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.070+0000 7f293affd700 1 --2- 172.21.15.38:0/200696901 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f293c100980 0x7f293c19cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:32.077 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.070+0000 7f293affd700 1 -- 172.21.15.38:0/200696901 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f292c0095f0 con 0x7f293c0fee00 2023-03-09T14:44:32.077 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.070+0000 7f293a7fc700 1 --2- 172.21.15.38:0/200696901 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f293c0ffff0 0x7f293c19a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:44:32.078 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.070+0000 7f293affd700 1 --2- 172.21.15.38:0/200696901 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f293c0fee00 0x7f293c069820 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f292c004760 tx=0x7f292c009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:32.078 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.070+0000 7f2923fff700 1 -- 172.21.15.38:0/200696901 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f292c01d070 con 0x7f293c0fee00 2023-03-09T14:44:32.078 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.070+0000 7f2923fff700 1 -- 172.21.15.38:0/200696901 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f292c004cd0 con 0x7f293c0fee00 2023-03-09T14:44:32.078 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.070+0000 7f2923fff700 1 -- 172.21.15.38:0/200696901 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f292c005290 con 0x7f293c0fee00 2023-03-09T14:44:32.078 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.070+0000 7f2941456700 1 -- 172.21.15.38:0/200696901 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f293c10a740 con 0x7f293c0fee00 2023-03-09T14:44:32.078 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.070+0000 7f2941456700 1 -- 172.21.15.38:0/200696901 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f293c10a5b0 con 0x7f293c0fee00 2023-03-09T14:44:32.079 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.071+0000 7f2941456700 1 -- 172.21.15.38:0/200696901 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f293c105d80 con 0x7f293c0fee00 2023-03-09T14:44:32.079 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.072+0000 7f2923fff700 1 -- 172.21.15.38:0/200696901 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 133198+0+0 (secure 0 0 0) 0x7f292c028030 con 0x7f293c0fee00 2023-03-09T14:44:32.080 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.073+0000 7f2923fff700 1 --2- 172.21.15.38:0/200696901 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f292409f440 0x7f29240a18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:32.080 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.073+0000 7f2923fff700 1 -- 172.21.15.38:0/200696901 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f292c0cb210 con 0x7f293c0fee00 2023-03-09T14:44:32.080 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.073+0000 7f293a7fc700 1 --2- 172.21.15.38:0/200696901 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f292409f440 0x7f29240a18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:32.080 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.074+0000 7f293a7fc700 1 --2- 172.21.15.38:0/200696901 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f292409f440 0x7f29240a18f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f2930005950 tx=0x7f29300058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:32.080 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.076+0000 7f2923fff700 1 -- 172.21.15.38:0/200696901 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f292c095470 con 0x7f293c0fee00 2023-03-09T14:44:32.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:31 smithi122 ceph-mon[93869]: pgmap v265: 129 pgs: 129 active+clean; 579 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 0 op/s 2023-03-09T14:44:32.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:31 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/1761062378' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-03-09T14:44:32.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:31 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/1676035934' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-03-09T14:44:32.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:31 smithi181 ceph-mon[94189]: pgmap v265: 129 pgs: 129 active+clean; 579 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 0 op/s 2023-03-09T14:44:32.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:31 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/1761062378' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-03-09T14:44:32.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:31 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/1676035934' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-03-09T14:44:32.279 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.276+0000 7f2941456700 1 -- 172.21.15.38:0/200696901 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f293c061390 con 0x7f293c0fee00 2023-03-09T14:44:32.280 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.277+0000 7f2923fff700 1 -- 172.21.15.38:0/200696901 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) v1 ==== 100+0+11 (secure 0 0 0) 0x7f292c0260a0 con 0x7f293c0fee00 2023-03-09T14:44:32.280 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 64 2023-03-09T14:44:32.282 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.279+0000 7f2941456700 1 -- 172.21.15.38:0/200696901 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f292409f440 msgr2=0x7f29240a18f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:32.282 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.279+0000 7f2941456700 1 --2- 172.21.15.38:0/200696901 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f292409f440 0x7f29240a18f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f2930005950 tx=0x7f29300058e0 comp rx=0 tx=0).stop 2023-03-09T14:44:32.282 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.279+0000 7f2941456700 1 -- 172.21.15.38:0/200696901 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f293c0fee00 msgr2=0x7f293c069820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:32.283 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.279+0000 7f2941456700 1 --2- 172.21.15.38:0/200696901 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f293c0fee00 0x7f293c069820 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f292c004760 tx=0x7f292c009580 comp rx=0 tx=0).stop 2023-03-09T14:44:32.283 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.279+0000 7f2941456700 1 -- 172.21.15.38:0/200696901 shutdown_connections 2023-03-09T14:44:32.283 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.279+0000 7f2941456700 1 --2- 172.21.15.38:0/200696901 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f293c0fee00 0x7f293c069820 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:32.283 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.279+0000 7f2941456700 1 --2- 172.21.15.38:0/200696901 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f292409f440 0x7f29240a18f0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:32.283 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.279+0000 7f2941456700 1 --2- 172.21.15.38:0/200696901 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f293c100980 0x7f293c19cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:32.283 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.279+0000 7f2941456700 1 --2- 172.21.15.38:0/200696901 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f293c0ffff0 0x7f293c19a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:32.283 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.279+0000 7f2941456700 1 -- 172.21.15.38:0/200696901 >> 172.21.15.38:0/200696901 conn(0x7f293c0fa420 msgr2=0x7f293c0fc710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:32.284 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.279+0000 7f2941456700 1 -- 172.21.15.38:0/200696901 shutdown_connections 2023-03-09T14:44:32.284 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.279+0000 7f2941456700 1 -- 172.21.15.38:0/200696901 wait complete. 2023-03-09T14:44:32.294 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-03-09T14:44:32.424 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.419+0000 7fbd17d98700 1 -- 172.21.15.38:0/2932258019 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd10100910 msgr2=0x7fbd101071d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:32.424 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.419+0000 7fbd17d98700 1 --2- 172.21.15.38:0/2932258019 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd10100910 0x7fbd101071d0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fbd0800b0a0 tx=0x7fbd0800b3b0 comp rx=0 tx=0).stop 2023-03-09T14:44:32.424 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.419+0000 7fbd17d98700 1 -- 172.21.15.38:0/2932258019 shutdown_connections 2023-03-09T14:44:32.424 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.419+0000 7fbd17d98700 1 --2- 172.21.15.38:0/2932258019 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd10100910 0x7fbd101071d0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:32.425 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.419+0000 7fbd17d98700 1 --2- 172.21.15.38:0/2932258019 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbd100fed90 0x7fbd100ff1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:32.425 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.419+0000 7fbd17d98700 1 --2- 172.21.15.38:0/2932258019 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbd100fff80 0x7fbd101003d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:32.425 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.419+0000 7fbd17d98700 1 -- 172.21.15.38:0/2932258019 >> 172.21.15.38:0/2932258019 conn(0x7fbd100fa3b0 msgr2=0x7fbd100fc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:32.425 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.419+0000 7fbd17d98700 1 -- 172.21.15.38:0/2932258019 shutdown_connections 2023-03-09T14:44:32.425 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.419+0000 7fbd17d98700 1 -- 172.21.15.38:0/2932258019 wait complete. 2023-03-09T14:44:32.425 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.420+0000 7fbd17d98700 1 Processor -- start 2023-03-09T14:44:32.426 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.420+0000 7fbd17d98700 1 -- start start 2023-03-09T14:44:32.426 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.420+0000 7fbd17d98700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbd100fed90 0x7fbd10195c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:32.426 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.420+0000 7fbd17d98700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbd100fff80 0x7fbd101961b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:32.426 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.420+0000 7fbd17d98700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd10100910 0x7fbd1019d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:32.426 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.420+0000 7fbd17d98700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd1019d7e0 con 0x7fbd10100910 2023-03-09T14:44:32.426 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.420+0000 7fbd17d98700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd1019d950 con 0x7fbd100fff80 2023-03-09T14:44:32.426 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.420+0000 7fbd17d98700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd1019db10 con 0x7fbd100fed90 2023-03-09T14:44:32.427 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.420+0000 7fbd16335700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd10100910 0x7fbd1019d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:32.427 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.420+0000 7fbd16335700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd10100910 0x7fbd1019d210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36940/0 (socket says 172.21.15.38:36940) 2023-03-09T14:44:32.427 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.420+0000 7fbd16335700 1 -- 172.21.15.38:0/4018768759 learned_addr learned my addr 172.21.15.38:0/4018768759 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:44:32.427 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.420+0000 7fbd15333700 1 --2- 172.21.15.38:0/4018768759 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbd100fff80 0x7fbd101961b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:32.427 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.420+0000 7fbd15b34700 1 --2- 172.21.15.38:0/4018768759 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbd100fed90 0x7fbd10195c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:32.427 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.420+0000 7fbd16335700 1 -- 172.21.15.38:0/4018768759 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbd100fed90 msgr2=0x7fbd10195c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:32.427 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.420+0000 7fbd16335700 1 --2- 172.21.15.38:0/4018768759 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbd100fed90 0x7fbd10195c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:32.428 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.420+0000 7fbd16335700 1 -- 172.21.15.38:0/4018768759 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbd100fff80 msgr2=0x7fbd101961b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:32.428 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.420+0000 7fbd16335700 1 --2- 172.21.15.38:0/4018768759 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbd100fff80 0x7fbd101961b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:32.428 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.420+0000 7fbd16335700 1 -- 172.21.15.38:0/4018768759 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd08009d00 con 0x7fbd10100910 2023-03-09T14:44:32.428 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.420+0000 7fbd16335700 1 --2- 172.21.15.38:0/4018768759 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd10100910 0x7fbd1019d210 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fbd0800b830 tx=0x7fbd08004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:32.428 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.421+0000 7fbd06ffd700 1 -- 172.21.15.38:0/4018768759 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd0800e040 con 0x7fbd10100910 2023-03-09T14:44:32.428 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.421+0000 7fbd06ffd700 1 -- 172.21.15.38:0/4018768759 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fbd0801db50 con 0x7fbd10100910 2023-03-09T14:44:32.428 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.421+0000 7fbd17d98700 1 -- 172.21.15.38:0/4018768759 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd1010a700 con 0x7fbd10100910 2023-03-09T14:44:32.429 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.421+0000 7fbd06ffd700 1 -- 172.21.15.38:0/4018768759 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd0801bbd0 con 0x7fbd10100910 2023-03-09T14:44:32.429 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.421+0000 7fbd17d98700 1 -- 172.21.15.38:0/4018768759 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd1010a570 con 0x7fbd10100910 2023-03-09T14:44:32.430 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.422+0000 7fbd17d98700 1 -- 172.21.15.38:0/4018768759 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd10106d20 con 0x7fbd10100910 2023-03-09T14:44:32.430 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.423+0000 7fbd06ffd700 1 -- 172.21.15.38:0/4018768759 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 133198+0+0 (secure 0 0 0) 0x7fbd0801dcc0 con 0x7fbd10100910 2023-03-09T14:44:32.430 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.423+0000 7fbd06ffd700 1 --2- 172.21.15.38:0/4018768759 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fbcfc09f3c0 0x7fbcfc0a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:32.430 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.423+0000 7fbd15b34700 1 --2- 172.21.15.38:0/4018768759 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fbcfc09f3c0 0x7fbcfc0a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:32.430 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.423+0000 7fbd06ffd700 1 -- 172.21.15.38:0/4018768759 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fbd080cc350 con 0x7fbd10100910 2023-03-09T14:44:32.431 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.424+0000 7fbd15b34700 1 --2- 172.21.15.38:0/4018768759 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fbcfc09f3c0 0x7fbcfc0a1870 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fbd10196b40 tx=0x7fbd0000a5e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:32.431 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.427+0000 7fbd06ffd700 1 -- 172.21.15.38:0/4018768759 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7fbd080965b0 con 0x7fbd10100910 2023-03-09T14:44:32.630 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.626+0000 7fbd17d98700 1 -- 172.21.15.38:0/4018768759 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fbd10067660 con 0x7fbd10100910 2023-03-09T14:44:32.630 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.627+0000 7fbd06ffd700 1 -- 172.21.15.38:0/4018768759 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v87) v1 ==== 103+0+11 (secure 0 0 0) 0x7fbd08021020 con 0x7fbd10100910 2023-03-09T14:44:32.630 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 32 2023-03-09T14:44:32.632 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.629+0000 7fbd17d98700 1 -- 172.21.15.38:0/4018768759 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fbcfc09f3c0 msgr2=0x7fbcfc0a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:32.632 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.629+0000 7fbd17d98700 1 --2- 172.21.15.38:0/4018768759 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fbcfc09f3c0 0x7fbcfc0a1870 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fbd10196b40 tx=0x7fbd0000a5e0 comp rx=0 tx=0).stop 2023-03-09T14:44:32.632 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.629+0000 7fbd17d98700 1 -- 172.21.15.38:0/4018768759 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd10100910 msgr2=0x7fbd1019d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:32.632 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.629+0000 7fbd17d98700 1 --2- 172.21.15.38:0/4018768759 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd10100910 0x7fbd1019d210 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fbd0800b830 tx=0x7fbd08004c80 comp rx=0 tx=0).stop 2023-03-09T14:44:32.633 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.629+0000 7fbd17d98700 1 -- 172.21.15.38:0/4018768759 shutdown_connections 2023-03-09T14:44:32.633 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.629+0000 7fbd17d98700 1 --2- 172.21.15.38:0/4018768759 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fbd10100910 0x7fbd1019d210 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:32.633 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.629+0000 7fbd17d98700 1 --2- 172.21.15.38:0/4018768759 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fbcfc09f3c0 0x7fbcfc0a1870 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:32.633 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.629+0000 7fbd17d98700 1 --2- 172.21.15.38:0/4018768759 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fbd100fed90 0x7fbd10195c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:32.633 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.629+0000 7fbd17d98700 1 --2- 172.21.15.38:0/4018768759 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fbd100fff80 0x7fbd101961b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:32.633 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.629+0000 7fbd17d98700 1 -- 172.21.15.38:0/4018768759 >> 172.21.15.38:0/4018768759 conn(0x7fbd100fa3b0 msgr2=0x7fbd100fc4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:32.633 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.629+0000 7fbd17d98700 1 -- 172.21.15.38:0/4018768759 shutdown_connections 2023-03-09T14:44:32.634 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.629+0000 7fbd17d98700 1 -- 172.21.15.38:0/4018768759 wait complete. 2023-03-09T14:44:32.644 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-03-09T14:44:32.770 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.767+0000 7fcdb8e2b700 1 -- 172.21.15.38:0/476722004 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcdb4100910 msgr2=0x7fcdb41071d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:32.771 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.767+0000 7fcdb8e2b700 1 --2- 172.21.15.38:0/476722004 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcdb4100910 0x7fcdb41071d0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fcdac00b0a0 tx=0x7fcdac00b3b0 comp rx=0 tx=0).stop 2023-03-09T14:44:32.771 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.767+0000 7fcdb8e2b700 1 -- 172.21.15.38:0/476722004 shutdown_connections 2023-03-09T14:44:32.771 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.767+0000 7fcdb8e2b700 1 --2- 172.21.15.38:0/476722004 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcdb4100910 0x7fcdb41071d0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:32.771 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.767+0000 7fcdb8e2b700 1 --2- 172.21.15.38:0/476722004 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcdb40fed90 0x7fcdb40ff1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:32.771 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.767+0000 7fcdb8e2b700 1 --2- 172.21.15.38:0/476722004 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcdb40fff80 0x7fcdb41003d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:32.772 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.767+0000 7fcdb8e2b700 1 -- 172.21.15.38:0/476722004 >> 172.21.15.38:0/476722004 conn(0x7fcdb40fa3b0 msgr2=0x7fcdb40fc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:32.772 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.767+0000 7fcdb8e2b700 1 -- 172.21.15.38:0/476722004 shutdown_connections 2023-03-09T14:44:32.772 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.767+0000 7fcdb8e2b700 1 -- 172.21.15.38:0/476722004 wait complete. 2023-03-09T14:44:32.774 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.767+0000 7fcdb8e2b700 1 Processor -- start 2023-03-09T14:44:32.774 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.768+0000 7fcdb8e2b700 1 -- start start 2023-03-09T14:44:32.774 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.768+0000 7fcdb8e2b700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcdb40fed90 0x7fcdb4194d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:32.774 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.768+0000 7fcdb8e2b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcdb40fff80 0x7fcdb4195280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:32.775 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.768+0000 7fcdb8e2b700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcdb4100910 0x7fcdb419c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:32.775 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.768+0000 7fcdb8e2b700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb419c8c0 con 0x7fcdb40fff80 2023-03-09T14:44:32.775 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.768+0000 7fcdb8e2b700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb419ca00 con 0x7fcdb40fed90 2023-03-09T14:44:32.775 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.768+0000 7fcdb8e2b700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdb419cb90 con 0x7fcdb4100910 2023-03-09T14:44:32.775 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.768+0000 7fcdb1d9b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcdb40fff80 0x7fcdb4195280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:32.775 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.768+0000 7fcdb1d9b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcdb40fff80 0x7fcdb4195280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36952/0 (socket says 172.21.15.38:36952) 2023-03-09T14:44:32.775 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.768+0000 7fcdb1d9b700 1 -- 172.21.15.38:0/4120315201 learned_addr learned my addr 172.21.15.38:0/4120315201 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:44:32.776 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.768+0000 7fcdb259c700 1 --2- 172.21.15.38:0/4120315201 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcdb40fed90 0x7fcdb4194d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:32.776 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.768+0000 7fcdb2d9d700 1 --2- 172.21.15.38:0/4120315201 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcdb4100910 0x7fcdb419c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:32.776 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.768+0000 7fcdb1d9b700 1 -- 172.21.15.38:0/4120315201 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcdb4100910 msgr2=0x7fcdb419c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:32.776 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.768+0000 7fcdb1d9b700 1 --2- 172.21.15.38:0/4120315201 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcdb4100910 0x7fcdb419c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:32.776 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.768+0000 7fcdb1d9b700 1 -- 172.21.15.38:0/4120315201 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcdb40fed90 msgr2=0x7fcdb4194d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:32.776 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.768+0000 7fcdb1d9b700 1 --2- 172.21.15.38:0/4120315201 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcdb40fed90 0x7fcdb4194d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:32.776 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.768+0000 7fcdb1d9b700 1 -- 172.21.15.38:0/4120315201 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdac009d00 con 0x7fcdb40fff80 2023-03-09T14:44:32.777 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.768+0000 7fcdb2d9d700 1 --2- 172.21.15.38:0/4120315201 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcdb4100910 0x7fcdb419c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-09T14:44:32.777 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.768+0000 7fcdb1d9b700 1 --2- 172.21.15.38:0/4120315201 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcdb40fff80 0x7fcdb4195280 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fcda400c920 tx=0x7fcda400cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:32.777 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.769+0000 7fcda37fe700 1 -- 172.21.15.38:0/4120315201 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcda4007b00 con 0x7fcdb40fff80 2023-03-09T14:44:32.777 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.769+0000 7fcda37fe700 1 -- 172.21.15.38:0/4120315201 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcda400ce70 con 0x7fcdb40fff80 2023-03-09T14:44:32.777 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.769+0000 7fcdb8e2b700 1 -- 172.21.15.38:0/4120315201 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdb410a750 con 0x7fcdb40fff80 2023-03-09T14:44:32.777 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.769+0000 7fcda37fe700 1 -- 172.21.15.38:0/4120315201 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcda40054e0 con 0x7fcdb40fff80 2023-03-09T14:44:32.777 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.769+0000 7fcdb8e2b700 1 -- 172.21.15.38:0/4120315201 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdb410a5c0 con 0x7fcdb40fff80 2023-03-09T14:44:32.778 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.770+0000 7fcdb8e2b700 1 -- 172.21.15.38:0/4120315201 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdb4106070 con 0x7fcdb40fff80 2023-03-09T14:44:32.778 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.771+0000 7fcda37fe700 1 -- 172.21.15.38:0/4120315201 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 133198+0+0 (secure 0 0 0) 0x7fcda401f070 con 0x7fcdb40fff80 2023-03-09T14:44:32.778 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.771+0000 7fcda37fe700 1 --2- 172.21.15.38:0/4120315201 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fcd9c09f3c0 0x7fcd9c0a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:32.779 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.771+0000 7fcdb259c700 1 --2- 172.21.15.38:0/4120315201 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fcd9c09f3c0 0x7fcd9c0a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:32.779 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.771+0000 7fcda37fe700 1 -- 172.21.15.38:0/4120315201 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fcda40c9e20 con 0x7fcdb40fff80 2023-03-09T14:44:32.779 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.772+0000 7fcdb259c700 1 --2- 172.21.15.38:0/4120315201 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fcd9c09f3c0 0x7fcd9c0a1870 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fcdb40ffde0 tx=0x7fcda800a3b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:32.779 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.775+0000 7fcda37fe700 1 -- 172.21.15.38:0/4120315201 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7fcda4093ff0 con 0x7fcdb40fff80 2023-03-09T14:44:32.996 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.992+0000 7fcdb8e2b700 1 -- 172.21.15.38:0/4120315201 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fcdb402cef0 con 0x7fcdb40fff80 2023-03-09T14:44:32.996 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.993+0000 7fcda37fe700 1 -- 172.21.15.38:0/4120315201 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v87) v1 ==== 81+0+591 (secure 0 0 0) 0x7fcda401d070 con 0x7fcdb40fff80 2023-03-09T14:44:32.996 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:0/4272288735 2023-03-10T14:38:14.612767+0000 2023-03-09T14:44:32.997 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:6801/3381625669 2023-03-10T14:38:14.612767+0000 2023-03-09T14:44:32.997 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:6800/3381625669 2023-03-10T14:38:14.612767+0000 2023-03-09T14:44:32.997 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:6800/3060474342 2023-03-10T14:37:47.517775+0000 2023-03-09T14:44:32.997 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:0/4253918615 2023-03-10T14:38:14.612767+0000 2023-03-09T14:44:32.997 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:0/3496083181 2023-03-10T14:37:47.517775+0000 2023-03-09T14:44:32.997 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:0/245602368 2023-03-10T14:37:47.517775+0000 2023-03-09T14:44:32.998 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:6801/3060474342 2023-03-10T14:37:47.517775+0000 2023-03-09T14:44:32.998 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:0/3622589182 2023-03-10T14:37:47.517775+0000 2023-03-09T14:44:32.998 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:0/1430302251 2023-03-10T14:38:14.612767+0000 2023-03-09T14:44:32.999 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.995+0000 7fcdb8e2b700 1 -- 172.21.15.38:0/4120315201 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fcd9c09f3c0 msgr2=0x7fcd9c0a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:32.999 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.995+0000 7fcdb8e2b700 1 --2- 172.21.15.38:0/4120315201 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fcd9c09f3c0 0x7fcd9c0a1870 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fcdb40ffde0 tx=0x7fcda800a3b0 comp rx=0 tx=0).stop 2023-03-09T14:44:32.999 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.995+0000 7fcdb8e2b700 1 -- 172.21.15.38:0/4120315201 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcdb40fff80 msgr2=0x7fcdb4195280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:32.999 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.995+0000 7fcdb8e2b700 1 --2- 172.21.15.38:0/4120315201 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcdb40fff80 0x7fcdb4195280 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fcda400c920 tx=0x7fcda400cc30 comp rx=0 tx=0).stop 2023-03-09T14:44:32.999 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.995+0000 7fcdb8e2b700 1 -- 172.21.15.38:0/4120315201 shutdown_connections 2023-03-09T14:44:32.999 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.995+0000 7fcdb8e2b700 1 --2- 172.21.15.38:0/4120315201 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fcd9c09f3c0 0x7fcd9c0a1870 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:33.000 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.995+0000 7fcdb8e2b700 1 --2- 172.21.15.38:0/4120315201 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcdb4100910 0x7fcdb419c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:33.000 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.995+0000 7fcdb8e2b700 1 --2- 172.21.15.38:0/4120315201 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcdb40fed90 0x7fcdb4194d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:33.000 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.995+0000 7fcdb8e2b700 1 --2- 172.21.15.38:0/4120315201 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcdb40fff80 0x7fcdb4195280 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:33.000 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.995+0000 7fcdb8e2b700 1 -- 172.21.15.38:0/4120315201 >> 172.21.15.38:0/4120315201 conn(0x7fcdb40fa3b0 msgr2=0x7fcdb41031b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:33.000 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.995+0000 7fcdb8e2b700 1 -- 172.21.15.38:0/4120315201 shutdown_connections 2023-03-09T14:44:33.000 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:32.995+0000 7fcdb8e2b700 1 -- 172.21.15.38:0/4120315201 wait complete. 2023-03-09T14:44:33.001 INFO:teuthology.orchestra.run.smithi038.stderr:listed 10 entries 2023-03-09T14:44:33.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:32 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/2825414481' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-03-09T14:44:33.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:32 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/200696901' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-03-09T14:44:33.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:32 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/4018768759' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-03-09T14:44:33.010 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2023-03-09T14:44:33.011 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-09T14:44:33.028 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-09T14:44:33.193 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.190+0000 7f22f20fd700 1 -- 172.21.15.38:0/176381853 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f22ec100920 msgr2=0x7f22ec1071e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:33.256 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.190+0000 7f22f20fd700 1 --2- 172.21.15.38:0/176381853 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f22ec100920 0x7f22ec1071e0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f22e400b0a0 tx=0x7f22e400b3b0 comp rx=0 tx=0).stop 2023-03-09T14:44:33.256 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.190+0000 7f22f20fd700 1 -- 172.21.15.38:0/176381853 shutdown_connections 2023-03-09T14:44:33.256 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.190+0000 7f22f20fd700 1 --2- 172.21.15.38:0/176381853 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f22ec100920 0x7f22ec1071e0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:33.257 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.190+0000 7f22f20fd700 1 --2- 172.21.15.38:0/176381853 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f22ec0feda0 0x7f22ec0ff1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:33.257 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.190+0000 7f22f20fd700 1 --2- 172.21.15.38:0/176381853 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f22ec0fff90 0x7f22ec1003e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:33.257 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.190+0000 7f22f20fd700 1 -- 172.21.15.38:0/176381853 >> 172.21.15.38:0/176381853 conn(0x7f22ec0fa380 msgr2=0x7f22ec0fc7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:33.257 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.190+0000 7f22f20fd700 1 -- 172.21.15.38:0/176381853 shutdown_connections 2023-03-09T14:44:33.257 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.190+0000 7f22f20fd700 1 -- 172.21.15.38:0/176381853 wait complete. 2023-03-09T14:44:33.257 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.190+0000 7f22f20fd700 1 Processor -- start 2023-03-09T14:44:33.257 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.191+0000 7f22f20fd700 1 -- start start 2023-03-09T14:44:33.258 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:32 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/2825414481' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-03-09T14:44:33.259 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:32 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/200696901' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-03-09T14:44:33.259 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:32 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/4018768759' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-03-09T14:44:33.259 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:32 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/2825414481' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-03-09T14:44:33.259 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:32 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/200696901' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-03-09T14:44:33.259 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:32 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/4018768759' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-03-09T14:44:33.261 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.191+0000 7f22f20fd700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f22ec0feda0 0x7f22ec102d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:33.261 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.191+0000 7f22f20fd700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f22ec0fff90 0x7f22ec101370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:33.262 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.191+0000 7f22f20fd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f22ec100920 0x7f22ec1018b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:33.262 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.191+0000 7f22f20fd700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22ec102050 con 0x7f22ec100920 2023-03-09T14:44:33.262 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.191+0000 7f22f20fd700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22ec1021c0 con 0x7f22ec0feda0 2023-03-09T14:44:33.262 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.191+0000 7f22f20fd700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22ec19ffb0 con 0x7f22ec0fff90 2023-03-09T14:44:33.262 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.191+0000 7f22ebfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f22ec100920 0x7f22ec1018b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:33.262 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.191+0000 7f22ebfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f22ec100920 0x7f22ec1018b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36968/0 (socket says 172.21.15.38:36968) 2023-03-09T14:44:33.262 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.191+0000 7f22ebfff700 1 -- 172.21.15.38:0/235829509 learned_addr learned my addr 172.21.15.38:0/235829509 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:44:33.263 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.191+0000 7f22eb7fe700 1 --2- 172.21.15.38:0/235829509 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f22ec0feda0 0x7f22ec102d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:33.263 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.191+0000 7f22eaffd700 1 --2- 172.21.15.38:0/235829509 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f22ec0fff90 0x7f22ec101370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:33.263 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.191+0000 7f22ebfff700 1 -- 172.21.15.38:0/235829509 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f22ec0fff90 msgr2=0x7f22ec101370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:33.263 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.191+0000 7f22ebfff700 1 --2- 172.21.15.38:0/235829509 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f22ec0fff90 0x7f22ec101370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:33.263 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.191+0000 7f22ebfff700 1 -- 172.21.15.38:0/235829509 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f22ec0feda0 msgr2=0x7f22ec102d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:33.263 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.191+0000 7f22ebfff700 1 --2- 172.21.15.38:0/235829509 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f22ec0feda0 0x7f22ec102d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:33.264 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.191+0000 7f22ebfff700 1 -- 172.21.15.38:0/235829509 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22e4009d00 con 0x7f22ec100920 2023-03-09T14:44:33.264 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.191+0000 7f22eaffd700 1 --2- 172.21.15.38:0/235829509 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f22ec0fff90 0x7f22ec101370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-09T14:44:33.264 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.191+0000 7f22ebfff700 1 --2- 172.21.15.38:0/235829509 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f22ec100920 0x7f22ec1018b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f22e400b070 tx=0x7f22e4004cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:33.264 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.191+0000 7f22e8ff9700 1 -- 172.21.15.38:0/235829509 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22e400e040 con 0x7f22ec100920 2023-03-09T14:44:33.264 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.192+0000 7f22e8ff9700 1 -- 172.21.15.38:0/235829509 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f22e40092e0 con 0x7f22ec100920 2023-03-09T14:44:33.264 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.192+0000 7f22f20fd700 1 -- 172.21.15.38:0/235829509 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22ec10a760 con 0x7f22ec100920 2023-03-09T14:44:33.264 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.192+0000 7f22e8ff9700 1 -- 172.21.15.38:0/235829509 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22e40048b0 con 0x7f22ec100920 2023-03-09T14:44:33.265 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.192+0000 7f22f20fd700 1 -- 172.21.15.38:0/235829509 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22ec10a5d0 con 0x7f22ec100920 2023-03-09T14:44:33.265 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.193+0000 7f22f20fd700 1 -- 172.21.15.38:0/235829509 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22ec10a8f0 con 0x7f22ec100920 2023-03-09T14:44:33.265 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.194+0000 7f22e8ff9700 1 -- 172.21.15.38:0/235829509 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 133198+0+0 (secure 0 0 0) 0x7f22e4003a50 con 0x7f22ec100920 2023-03-09T14:44:33.266 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.194+0000 7f22e8ff9700 1 --2- 172.21.15.38:0/235829509 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f22d809f3c0 0x7f22d80a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:33.266 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.194+0000 7f22e8ff9700 1 -- 172.21.15.38:0/235829509 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f22e40cb710 con 0x7f22ec100920 2023-03-09T14:44:33.266 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.194+0000 7f22eb7fe700 1 --2- 172.21.15.38:0/235829509 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f22d809f3c0 0x7f22d80a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:33.266 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.195+0000 7f22eb7fe700 1 --2- 172.21.15.38:0/235829509 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f22d809f3c0 0x7f22d80a1870 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f22ec0ffdf0 tx=0x7f22d400a380 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:33.266 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.198+0000 7f22e8ff9700 1 -- 172.21.15.38:0/235829509 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f22e4095970 con 0x7f22ec100920 2023-03-09T14:44:33.404 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.401+0000 7f22f20fd700 1 -- 172.21.15.38:0/235829509 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f22ec067ab0 con 0x7f22ec100920 2023-03-09T14:44:33.405 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.402+0000 7f22e8ff9700 1 -- 172.21.15.38:0/235829509 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+23973 (secure 0 0 0) 0x7f22e4004000 con 0x7f22ec100920 2023-03-09T14:44:33.406 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:44:33.406 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":87,"fsid":"c6733fe2-be87-11ed-9af2-001a4aab830c","created":"2023-03-09T14:37:14.947981+0000","modified":"2023-03-09T14:44:23.900228+0000","last_up_change":"2023-03-09T14:43:37.198407+0000","last_in_change":"2023-03-09T14:43:19.550566+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-09T14:40:23.196665+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-03-09T14:44:17.484374+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-03-09T14:44:18.457833+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-03-09T14:44:21.895589+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"51477039-15ee-4707-a34d-4ecfa18bebde","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6803","nonce":1508072290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6805","nonce":1508072290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6809","nonce":1508072290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6807","nonce":1508072290}]},"public_addr":"172.21.15.38:6803/1508072290","cluster_addr":"172.21.15.38:6805/1508072290","heartbeat_back_addr":"172.21.15.38:6809/1508072290","heartbeat_front_addr":"172.21.15.38:6807/1508072290","state":["exists","up"]},{"osd":1,"uuid":"71f8a718-b0a9-4f39-bc54-267aa42a2578","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6801","nonce":3929268773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6803","nonce":3929268773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6807","nonce":3929268773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6805","nonce":3929268773}]},"public_addr":"172.21.15.122:6801/3929268773","cluster_addr":"172.21.15.122:6803/3929268773","heartbeat_back_addr":"172.21.15.122:6807/3929268773","heartbeat_front_addr":"172.21.15.122:6805/3929268773","state":["exists","up"]},{"osd":2,"uuid":"961a2cbe-140e-437f-b131-1fcac573b19c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6801","nonce":1310630803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6803","nonce":1310630803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6807","nonce":1310630803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6805","nonce":1310630803}]},"public_addr":"172.21.15.181:6801/1310630803","cluster_addr":"172.21.15.181:6803/1310630803","heartbeat_back_addr":"172.21.15.181:6807/1310630803","heartbeat_front_addr":"172.21.15.181:6805/1310630803","state":["exists","up"]},{"osd":3,"uuid":"969df8fa-2d19-42c7-9f2a-5c5fac15d4a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6811","nonce":2719814100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6813","nonce":2719814100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6817","nonce":2719814100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6815","nonce":2719814100}]},"public_addr":"172.21.15.38:6811/2719814100","cluster_addr":"172.21.15.38:6813/2719814100","heartbeat_back_addr":"172.21.15.38:6817/2719814100","heartbeat_front_addr":"172.21.15.38:6815/2719814100","state":["exists","up"]},{"osd":4,"uuid":"44a6dce3-e06e-4cbd-9404-62a86d01ab6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6809","nonce":2480764994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6811","nonce":2480764994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6815","nonce":2480764994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6813","nonce":2480764994}]},"public_addr":"172.21.15.122:6809/2480764994","cluster_addr":"172.21.15.122:6811/2480764994","heartbeat_back_addr":"172.21.15.122:6815/2480764994","heartbeat_front_addr":"172.21.15.122:6813/2480764994","state":["exists","up"]},{"osd":5,"uuid":"49749574-c00c-4e3a-80e6-e3686df58d5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6809","nonce":3141346246}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6811","nonce":3141346246}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6815","nonce":3141346246}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6813","nonce":3141346246}]},"public_addr":"172.21.15.181:6809/3141346246","cluster_addr":"172.21.15.181:6811/3141346246","heartbeat_back_addr":"172.21.15.181:6815/3141346246","heartbeat_front_addr":"172.21.15.181:6813/3141346246","state":["exists","up"]},{"osd":6,"uuid":"61c0437e-3f51-45a9-8a61-e98eea0093f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6819","nonce":2122788957}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6821","nonce":2122788957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6825","nonce":2122788957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6823","nonce":2122788957}]},"public_addr":"172.21.15.38:6819/2122788957","cluster_addr":"172.21.15.38:6821/2122788957","heartbeat_back_addr":"172.21.15.38:6825/2122788957","heartbeat_front_addr":"172.21.15.38:6823/2122788957","state":["exists","up"]},{"osd":7,"uuid":"231fecdd-939f-4c7b-83a1-c4a1cffd7139","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6817","nonce":656874139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6819","nonce":656874139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6823","nonce":656874139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6821","nonce":656874139}]},"public_addr":"172.21.15.122:6817/656874139","cluster_addr":"172.21.15.122:6819/656874139","heartbeat_back_addr":"172.21.15.122:6823/656874139","heartbeat_front_addr":"172.21.15.122:6821/656874139","state":["exists","up"]},{"osd":8,"uuid":"297e6d6d-85fd-43ce-9028-5e9dbe55c36e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6817","nonce":755000703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6819","nonce":755000703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6823","nonce":755000703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6821","nonce":755000703}]},"public_addr":"172.21.15.181:6817/755000703","cluster_addr":"172.21.15.181:6819/755000703","heartbeat_back_addr":"172.21.15.181:6823/755000703","heartbeat_front_addr":"172.21.15.181:6821/755000703","state":["exists","up"]},{"osd":9,"uuid":"94cc0355-5a1a-4cfc-bd0c-562b6b41127e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6827","nonce":2624555969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6829","nonce":2624555969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6833","nonce":2624555969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6831","nonce":2624555969}]},"public_addr":"172.21.15.38:6827/2624555969","cluster_addr":"172.21.15.38:6829/2624555969","heartbeat_back_addr":"172.21.15.38:6833/2624555969","heartbeat_front_addr":"172.21.15.38:6831/2624555969","state":["exists","up"]},{"osd":10,"uuid":"f4f2e042-8d92-46bf-93b8-8b0f04795c2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6825","nonce":1427785280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6827","nonce":1427785280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6831","nonce":1427785280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6829","nonce":1427785280}]},"public_addr":"172.21.15.122:6825/1427785280","cluster_addr":"172.21.15.122:6827/1427785280","heartbeat_back_addr":"172.21.15.122:6831/1427785280","heartbeat_front_addr":"172.21.15.122:6829/1427785280","state":["exists","up"]},{"osd":11,"uuid":"07444936-eae5-4097-a799-60294cafe80b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6825","nonce":3183488640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6827","nonce":3183488640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6831","nonce":3183488640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6829","nonce":3183488640}]},"public_addr":"172.21.15.181:6825/3183488640","cluster_addr":"172.21.15.181:6827/3183488640","heartbeat_back_addr":"172.21.15.181:6831/3183488640","heartbeat_front_addr":"172.21.15.181:6829/3183488640","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:39:37.657556+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:39:58.960074+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:40:20.326952+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:40:42.244387+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:41:03.946973+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:41:25.097985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:41:46.877916+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:42:08.413827+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:42:29.822009+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:42:51.776684+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:43:13.937249+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:43:35.569199+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/4272288735":"2023-03-10T14:38:14.612767+0000","172.21.15.38:6801/3381625669":"2023-03-10T14:38:14.612767+0000","172.21.15.38:6800/3381625669":"2023-03-10T14:38:14.612767+0000","172.21.15.38:6800/3060474342":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/4253918615":"2023-03-10T14:38:14.612767+0000","172.21.15.38:0/3496083181":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/245602368":"2023-03-10T14:37:47.517775+0000","172.21.15.38:6801/3060474342":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/3622589182":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/1430302251":"2023-03-10T14:38:14.612767+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-09T14:44:33.408 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.404+0000 7f22f20fd700 1 -- 172.21.15.38:0/235829509 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f22d809f3c0 msgr2=0x7f22d80a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:33.408 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.404+0000 7f22f20fd700 1 --2- 172.21.15.38:0/235829509 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f22d809f3c0 0x7f22d80a1870 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f22ec0ffdf0 tx=0x7f22d400a380 comp rx=0 tx=0).stop 2023-03-09T14:44:33.408 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.404+0000 7f22f20fd700 1 -- 172.21.15.38:0/235829509 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f22ec100920 msgr2=0x7f22ec1018b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:33.408 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.404+0000 7f22f20fd700 1 --2- 172.21.15.38:0/235829509 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f22ec100920 0x7f22ec1018b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f22e400b070 tx=0x7f22e4004cd0 comp rx=0 tx=0).stop 2023-03-09T14:44:33.409 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.405+0000 7f22f20fd700 1 -- 172.21.15.38:0/235829509 shutdown_connections 2023-03-09T14:44:33.409 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.405+0000 7f22f20fd700 1 --2- 172.21.15.38:0/235829509 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f22ec100920 0x7f22ec1018b0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:33.409 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.405+0000 7f22f20fd700 1 --2- 172.21.15.38:0/235829509 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f22d809f3c0 0x7f22d80a1870 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:33.409 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.405+0000 7f22f20fd700 1 --2- 172.21.15.38:0/235829509 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f22ec0feda0 0x7f22ec102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:33.409 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.405+0000 7f22f20fd700 1 --2- 172.21.15.38:0/235829509 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f22ec0fff90 0x7f22ec101370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:33.409 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.405+0000 7f22f20fd700 1 -- 172.21.15.38:0/235829509 >> 172.21.15.38:0/235829509 conn(0x7f22ec0fa380 msgr2=0x7f22ec068ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:33.409 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.405+0000 7f22f20fd700 1 -- 172.21.15.38:0/235829509 shutdown_connections 2023-03-09T14:44:33.410 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.405+0000 7f22f20fd700 1 -- 172.21.15.38:0/235829509 wait complete. 2023-03-09T14:44:33.420 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-09T14:40:23.196665+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-03-09T14:44:17.484374+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-03-09T14:44:18.457833+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-03-09T14:44:21.895589+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-03-09T14:44:33.421 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-09T14:44:33.549 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.546+0000 7f898e257700 1 -- 172.21.15.38:0/2553236209 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8988100fe0 msgr2=0x7f8988101450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:33.550 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.546+0000 7f898e257700 1 --2- 172.21.15.38:0/2553236209 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8988100fe0 0x7f8988101450 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f8978009990 tx=0x7f8978009ca0 comp rx=0 tx=0).stop 2023-03-09T14:44:33.550 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.546+0000 7f898e257700 1 -- 172.21.15.38:0/2553236209 shutdown_connections 2023-03-09T14:44:33.550 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.546+0000 7f898e257700 1 --2- 172.21.15.38:0/2553236209 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8988101990 0x7f89881082c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:33.550 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.546+0000 7f898e257700 1 --2- 172.21.15.38:0/2553236209 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f89880ffe90 0x7f89881002a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:33.550 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.546+0000 7f898e257700 1 --2- 172.21.15.38:0/2553236209 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8988100fe0 0x7f8988101450 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:33.550 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.546+0000 7f898e257700 1 -- 172.21.15.38:0/2553236209 >> 172.21.15.38:0/2553236209 conn(0x7f89880fb4d0 msgr2=0x7f89880fd8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:33.550 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.546+0000 7f898e257700 1 -- 172.21.15.38:0/2553236209 shutdown_connections 2023-03-09T14:44:33.551 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.546+0000 7f898e257700 1 -- 172.21.15.38:0/2553236209 wait complete. 2023-03-09T14:44:33.551 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.546+0000 7f898e257700 1 Processor -- start 2023-03-09T14:44:33.552 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.546+0000 7f898e257700 1 -- start start 2023-03-09T14:44:33.552 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.547+0000 7f898e257700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f89880ffe90 0x7f8988074530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:33.553 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.547+0000 7f898e257700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8988100fe0 0x7f8988072b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:33.553 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.547+0000 7f898e257700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8988101990 0x7f89880730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:33.553 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.547+0000 7f898e257700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8988073850 con 0x7f8988101990 2023-03-09T14:44:33.553 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.547+0000 7f898e257700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89880739c0 con 0x7f89880ffe90 2023-03-09T14:44:33.553 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.547+0000 7f898e257700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f898819fc70 con 0x7f8988100fe0 2023-03-09T14:44:33.553 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.547+0000 7f8987fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8988101990 0x7f89880730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:33.553 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.547+0000 7f8987fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8988101990 0x7f89880730b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36982/0 (socket says 172.21.15.38:36982) 2023-03-09T14:44:33.554 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.547+0000 7f8987fff700 1 -- 172.21.15.38:0/2440492903 learned_addr learned my addr 172.21.15.38:0/2440492903 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:44:33.554 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.547+0000 7f8986ffd700 1 --2- 172.21.15.38:0/2440492903 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8988100fe0 0x7f8988072b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:33.554 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.547+0000 7f89877fe700 1 --2- 172.21.15.38:0/2440492903 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f89880ffe90 0x7f8988074530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:33.554 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.547+0000 7f8987fff700 1 -- 172.21.15.38:0/2440492903 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8988100fe0 msgr2=0x7f8988072b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:33.554 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.547+0000 7f8987fff700 1 --2- 172.21.15.38:0/2440492903 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8988100fe0 0x7f8988072b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:33.554 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.547+0000 7f8987fff700 1 -- 172.21.15.38:0/2440492903 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f89880ffe90 msgr2=0x7f8988074530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:33.554 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.547+0000 7f8987fff700 1 --2- 172.21.15.38:0/2440492903 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f89880ffe90 0x7f8988074530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:33.555 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.547+0000 7f8987fff700 1 -- 172.21.15.38:0/2440492903 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89780095f0 con 0x7f8988101990 2023-03-09T14:44:33.555 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.547+0000 7f8987fff700 1 --2- 172.21.15.38:0/2440492903 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8988101990 0x7f89880730b0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f898000ada0 tx=0x7f898000c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:33.555 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.547+0000 7f8984ff9700 1 -- 172.21.15.38:0/2440492903 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f898000f070 con 0x7f8988101990 2023-03-09T14:44:33.555 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.547+0000 7f8984ff9700 1 -- 172.21.15.38:0/2440492903 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8980007b30 con 0x7f8988101990 2023-03-09T14:44:33.555 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.547+0000 7f898e257700 1 -- 172.21.15.38:0/2440492903 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f898810b7f0 con 0x7f8988101990 2023-03-09T14:44:33.555 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.547+0000 7f8984ff9700 1 -- 172.21.15.38:0/2440492903 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89800053e0 con 0x7f8988101990 2023-03-09T14:44:33.555 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.548+0000 7f898e257700 1 -- 172.21.15.38:0/2440492903 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f898810b660 con 0x7f8988101990 2023-03-09T14:44:33.555 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.549+0000 7f898e257700 1 -- 172.21.15.38:0/2440492903 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89881a02d0 con 0x7f8988101990 2023-03-09T14:44:33.556 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.550+0000 7f8984ff9700 1 -- 172.21.15.38:0/2440492903 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 133198+0+0 (secure 0 0 0) 0x7f8980019020 con 0x7f8988101990 2023-03-09T14:44:33.556 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.550+0000 7f8984ff9700 1 --2- 172.21.15.38:0/2440492903 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f897009f440 0x7f89700a18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:33.556 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.550+0000 7f89877fe700 1 --2- 172.21.15.38:0/2440492903 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f897009f440 0x7f89700a18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:33.557 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.550+0000 7f8984ff9700 1 -- 172.21.15.38:0/2440492903 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f89800c9a00 con 0x7f8988101990 2023-03-09T14:44:33.557 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.551+0000 7f89877fe700 1 --2- 172.21.15.38:0/2440492903 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f897009f440 0x7f89700a18f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f897c005fd0 tx=0x7f897c00a560 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:33.557 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.554+0000 7f8984ff9700 1 -- 172.21.15.38:0/2440492903 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f8980093c60 con 0x7f8988101990 2023-03-09T14:44:33.757 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.753+0000 7f898e257700 1 -- 172.21.15.38:0/2440492903 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8988106d80 con 0x7f8988101990 2023-03-09T14:44:33.757 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.754+0000 7f8984ff9700 1 -- 172.21.15.38:0/2440492903 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8980097280 con 0x7f8988101990 2023-03-09T14:44:33.758 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 1 2023-03-09T14:44:33.759 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.756+0000 7f898e257700 1 -- 172.21.15.38:0/2440492903 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f897009f440 msgr2=0x7f89700a18f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:33.759 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.756+0000 7f898e257700 1 --2- 172.21.15.38:0/2440492903 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f897009f440 0x7f89700a18f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f897c005fd0 tx=0x7f897c00a560 comp rx=0 tx=0).stop 2023-03-09T14:44:33.759 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.756+0000 7f898e257700 1 -- 172.21.15.38:0/2440492903 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8988101990 msgr2=0x7f89880730b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:33.760 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.756+0000 7f898e257700 1 --2- 172.21.15.38:0/2440492903 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8988101990 0x7f89880730b0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f898000ada0 tx=0x7f898000c330 comp rx=0 tx=0).stop 2023-03-09T14:44:33.760 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.756+0000 7f898e257700 1 -- 172.21.15.38:0/2440492903 shutdown_connections 2023-03-09T14:44:33.760 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.756+0000 7f898e257700 1 --2- 172.21.15.38:0/2440492903 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8988101990 0x7f89880730b0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:33.760 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.756+0000 7f898e257700 1 --2- 172.21.15.38:0/2440492903 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f897009f440 0x7f89700a18f0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:33.760 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.756+0000 7f898e257700 1 --2- 172.21.15.38:0/2440492903 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f89880ffe90 0x7f8988074530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:33.760 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.756+0000 7f898e257700 1 --2- 172.21.15.38:0/2440492903 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8988100fe0 0x7f8988072b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:33.760 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.756+0000 7f898e257700 1 -- 172.21.15.38:0/2440492903 >> 172.21.15.38:0/2440492903 conn(0x7f89880fb4d0 msgr2=0x7f89880fd750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:33.761 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.756+0000 7f898e257700 1 -- 172.21.15.38:0/2440492903 shutdown_connections 2023-03-09T14:44:33.761 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.756+0000 7f898e257700 1 -- 172.21.15.38:0/2440492903 wait complete. 2023-03-09T14:44:33.771 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-03-09T14:44:33.901 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.897+0000 7f3f7f7cd700 1 -- 172.21.15.38:0/1369630607 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3f780fff80 msgr2=0x7f3f781003d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:33.901 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.897+0000 7f3f7f7cd700 1 --2- 172.21.15.38:0/1369630607 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3f780fff80 0x7f3f781003d0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f3f68009990 tx=0x7f3f68009ca0 comp rx=0 tx=0).stop 2023-03-09T14:44:33.901 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.897+0000 7f3f7f7cd700 1 -- 172.21.15.38:0/1369630607 shutdown_connections 2023-03-09T14:44:33.901 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.897+0000 7f3f7f7cd700 1 --2- 172.21.15.38:0/1369630607 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3f78100910 0x7f3f781071d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:33.901 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.897+0000 7f3f7f7cd700 1 --2- 172.21.15.38:0/1369630607 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3f780fed90 0x7f3f780ff1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:33.902 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.897+0000 7f3f7f7cd700 1 --2- 172.21.15.38:0/1369630607 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3f780fff80 0x7f3f781003d0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:33.902 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.897+0000 7f3f7f7cd700 1 -- 172.21.15.38:0/1369630607 >> 172.21.15.38:0/1369630607 conn(0x7f3f780fa3b0 msgr2=0x7f3f780fc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:33.902 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.897+0000 7f3f7f7cd700 1 -- 172.21.15.38:0/1369630607 shutdown_connections 2023-03-09T14:44:33.902 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.897+0000 7f3f7f7cd700 1 -- 172.21.15.38:0/1369630607 wait complete. 2023-03-09T14:44:33.902 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.897+0000 7f3f7f7cd700 1 Processor -- start 2023-03-09T14:44:33.902 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.897+0000 7f3f7f7cd700 1 -- start start 2023-03-09T14:44:33.902 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.897+0000 7f3f7f7cd700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3f780fed90 0x7f3f780685d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:33.903 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.897+0000 7f3f7f7cd700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3f780fff80 0x7f3f7819a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:33.903 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.897+0000 7f3f7f7cd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3f78100910 0x7f3f7819d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:33.903 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.897+0000 7f3f7f7cd700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f7819d7d0 con 0x7f3f78100910 2023-03-09T14:44:33.903 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.897+0000 7f3f7f7cd700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f7819d940 con 0x7f3f780fed90 2023-03-09T14:44:33.903 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.897+0000 7f3f7f7cd700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f7819db00 con 0x7f3f780fff80 2023-03-09T14:44:33.903 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.898+0000 7f3f7dd6a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3f78100910 0x7f3f7819d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:33.903 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.898+0000 7f3f7dd6a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3f78100910 0x7f3f7819d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:36996/0 (socket says 172.21.15.38:36996) 2023-03-09T14:44:33.904 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.898+0000 7f3f7dd6a700 1 -- 172.21.15.38:0/1624215123 learned_addr learned my addr 172.21.15.38:0/1624215123 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:44:33.904 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.898+0000 7f3f7d569700 1 --2- 172.21.15.38:0/1624215123 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3f780fed90 0x7f3f780685d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:33.904 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.898+0000 7f3f7cd68700 1 --2- 172.21.15.38:0/1624215123 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3f780fff80 0x7f3f7819a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:33.905 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.898+0000 7f3f7dd6a700 1 -- 172.21.15.38:0/1624215123 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3f780fff80 msgr2=0x7f3f7819a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:33.905 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.898+0000 7f3f7dd6a700 1 --2- 172.21.15.38:0/1624215123 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3f780fff80 0x7f3f7819a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:33.905 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.898+0000 7f3f7dd6a700 1 -- 172.21.15.38:0/1624215123 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3f780fed90 msgr2=0x7f3f780685d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:33.906 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.898+0000 7f3f7dd6a700 1 --2- 172.21.15.38:0/1624215123 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3f780fed90 0x7f3f780685d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:33.906 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.898+0000 7f3f7dd6a700 1 -- 172.21.15.38:0/1624215123 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f680095f0 con 0x7f3f78100910 2023-03-09T14:44:33.906 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.898+0000 7f3f7dd6a700 1 --2- 172.21.15.38:0/1624215123 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3f78100910 0x7f3f7819d000 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f3f70007ee0 tx=0x7f3f7000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:33.906 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.898+0000 7f3f6e7fc700 1 -- 172.21.15.38:0/1624215123 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f70017070 con 0x7f3f78100910 2023-03-09T14:44:33.906 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.898+0000 7f3f7f7cd700 1 -- 172.21.15.38:0/1624215123 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f7810a740 con 0x7f3f78100910 2023-03-09T14:44:33.906 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.898+0000 7f3f6e7fc700 1 -- 172.21.15.38:0/1624215123 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3f7000f040 con 0x7f3f78100910 2023-03-09T14:44:33.906 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.898+0000 7f3f6e7fc700 1 -- 172.21.15.38:0/1624215123 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f70005580 con 0x7f3f78100910 2023-03-09T14:44:33.906 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.898+0000 7f3f7f7cd700 1 -- 172.21.15.38:0/1624215123 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f7810a5b0 con 0x7f3f78100910 2023-03-09T14:44:33.907 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.899+0000 7f3f7f7cd700 1 -- 172.21.15.38:0/1624215123 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f78106370 con 0x7f3f78100910 2023-03-09T14:44:33.907 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.901+0000 7f3f6e7fc700 1 -- 172.21.15.38:0/1624215123 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 133198+0+0 (secure 0 0 0) 0x7f3f70004c40 con 0x7f3f78100910 2023-03-09T14:44:33.907 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.901+0000 7f3f6e7fc700 1 --2- 172.21.15.38:0/1624215123 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f3f6409f3c0 0x7f3f640a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:33.907 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.901+0000 7f3f7d569700 1 --2- 172.21.15.38:0/1624215123 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f3f6409f3c0 0x7f3f640a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:33.907 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.902+0000 7f3f6e7fc700 1 -- 172.21.15.38:0/1624215123 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f3f700c9ef0 con 0x7f3f78100910 2023-03-09T14:44:33.907 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.902+0000 7f3f7d569700 1 --2- 172.21.15.38:0/1624215123 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f3f6409f3c0 0x7f3f640a1870 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3f780ffde0 tx=0x7f3f74009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:33.908 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:33.905+0000 7f3f6e7fc700 1 -- 172.21.15.38:0/1624215123 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f3f700940c0 con 0x7f3f78100910 2023-03-09T14:44:34.109 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.105+0000 7f3f7f7cd700 1 -- 172.21.15.38:0/1624215123 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3f78061390 con 0x7f3f78100910 2023-03-09T14:44:34.109 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.106+0000 7f3f6e7fc700 1 -- 172.21.15.38:0/1624215123 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3f700976e0 con 0x7f3f78100910 2023-03-09T14:44:34.109 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 32 2023-03-09T14:44:34.111 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.108+0000 7f3f7f7cd700 1 -- 172.21.15.38:0/1624215123 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f3f6409f3c0 msgr2=0x7f3f640a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:34.111 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.108+0000 7f3f7f7cd700 1 --2- 172.21.15.38:0/1624215123 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f3f6409f3c0 0x7f3f640a1870 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3f780ffde0 tx=0x7f3f74009260 comp rx=0 tx=0).stop 2023-03-09T14:44:34.111 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.108+0000 7f3f7f7cd700 1 -- 172.21.15.38:0/1624215123 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3f78100910 msgr2=0x7f3f7819d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:34.112 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.108+0000 7f3f7f7cd700 1 --2- 172.21.15.38:0/1624215123 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3f78100910 0x7f3f7819d000 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f3f70007ee0 tx=0x7f3f7000d3b0 comp rx=0 tx=0).stop 2023-03-09T14:44:34.112 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.108+0000 7f3f7f7cd700 1 -- 172.21.15.38:0/1624215123 shutdown_connections 2023-03-09T14:44:34.112 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.108+0000 7f3f7f7cd700 1 --2- 172.21.15.38:0/1624215123 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3f78100910 0x7f3f7819d000 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:34.112 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.108+0000 7f3f7f7cd700 1 --2- 172.21.15.38:0/1624215123 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f3f6409f3c0 0x7f3f640a1870 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:34.112 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.108+0000 7f3f7f7cd700 1 --2- 172.21.15.38:0/1624215123 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3f780fed90 0x7f3f780685d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:34.112 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.108+0000 7f3f7f7cd700 1 --2- 172.21.15.38:0/1624215123 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3f780fff80 0x7f3f7819a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:34.112 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.108+0000 7f3f7f7cd700 1 -- 172.21.15.38:0/1624215123 >> 172.21.15.38:0/1624215123 conn(0x7f3f780fa3b0 msgr2=0x7f3f781034b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:34.113 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.108+0000 7f3f7f7cd700 1 -- 172.21.15.38:0/1624215123 shutdown_connections 2023-03-09T14:44:34.113 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.108+0000 7f3f7f7cd700 1 -- 172.21.15.38:0/1624215123 wait complete. 2023-03-09T14:44:34.122 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:33 smithi038 ceph-mon[92422]: pgmap v266: 129 pgs: 129 active+clean; 579 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 110 B/s wr, 0 op/s 2023-03-09T14:44:34.122 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:33 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/4120315201' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-03-09T14:44:34.122 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:33 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/235829509' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-03-09T14:44:34.123 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:33 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/2440492903' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-03-09T14:44:34.124 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-03-09T14:44:34.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:33 smithi122 ceph-mon[93869]: pgmap v266: 129 pgs: 129 active+clean; 579 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 110 B/s wr, 0 op/s 2023-03-09T14:44:34.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:33 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/4120315201' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-03-09T14:44:34.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:33 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/235829509' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-03-09T14:44:34.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:33 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/2440492903' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-03-09T14:44:34.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:33 smithi181 ceph-mon[94189]: pgmap v266: 129 pgs: 129 active+clean; 579 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 110 B/s wr, 0 op/s 2023-03-09T14:44:34.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:33 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/4120315201' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-03-09T14:44:34.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:33 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/235829509' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-03-09T14:44:34.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:33 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/2440492903' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-03-09T14:44:34.251 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.248+0000 7f4ec88d8700 1 -- 172.21.15.38:0/1749791766 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ec40fffa0 msgr2=0x7f4ec41003f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:34.251 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.248+0000 7f4ec88d8700 1 --2- 172.21.15.38:0/1749791766 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ec40fffa0 0x7f4ec41003f0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f4eb4009990 tx=0x7f4eb4009ca0 comp rx=0 tx=0).stop 2023-03-09T14:44:34.252 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.248+0000 7f4ec88d8700 1 -- 172.21.15.38:0/1749791766 shutdown_connections 2023-03-09T14:44:34.252 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.248+0000 7f4ec88d8700 1 --2- 172.21.15.38:0/1749791766 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ec4100930 0x7f4ec41071f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:34.252 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.248+0000 7f4ec88d8700 1 --2- 172.21.15.38:0/1749791766 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ec40fedb0 0x7f4ec40ff1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:34.252 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.248+0000 7f4ec88d8700 1 --2- 172.21.15.38:0/1749791766 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ec40fffa0 0x7f4ec41003f0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:34.252 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.248+0000 7f4ec88d8700 1 -- 172.21.15.38:0/1749791766 >> 172.21.15.38:0/1749791766 conn(0x7f4ec40fa3b0 msgr2=0x7f4ec40fc800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:34.252 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.248+0000 7f4ec88d8700 1 -- 172.21.15.38:0/1749791766 shutdown_connections 2023-03-09T14:44:34.253 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.248+0000 7f4ec88d8700 1 -- 172.21.15.38:0/1749791766 wait complete. 2023-03-09T14:44:34.254 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.249+0000 7f4ec88d8700 1 Processor -- start 2023-03-09T14:44:34.254 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.249+0000 7f4ec88d8700 1 -- start start 2023-03-09T14:44:34.255 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.249+0000 7f4ec88d8700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ec40fedb0 0x7f4ec4102d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:34.255 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.249+0000 7f4ec88d8700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ec40fffa0 0x7f4ec4101380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:34.255 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.249+0000 7f4ec88d8700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ec4100930 0x7f4ec41018c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:34.255 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.249+0000 7f4ec88d8700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ec4102060 con 0x7f4ec40fedb0 2023-03-09T14:44:34.255 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.249+0000 7f4ec88d8700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ec41021d0 con 0x7f4ec40fffa0 2023-03-09T14:44:34.255 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.249+0000 7f4ec88d8700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ec419fbf0 con 0x7f4ec4100930 2023-03-09T14:44:34.256 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.249+0000 7f4ec259c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ec40fedb0 0x7f4ec4102d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:34.256 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.249+0000 7f4ec259c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ec40fedb0 0x7f4ec4102d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37006/0 (socket says 172.21.15.38:37006) 2023-03-09T14:44:34.256 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.249+0000 7f4ec259c700 1 -- 172.21.15.38:0/1216749134 learned_addr learned my addr 172.21.15.38:0/1216749134 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:44:34.256 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.249+0000 7f4ec2d9d700 1 --2- 172.21.15.38:0/1216749134 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ec4100930 0x7f4ec41018c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:34.256 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.249+0000 7f4ec1d9b700 1 --2- 172.21.15.38:0/1216749134 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ec40fffa0 0x7f4ec4101380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:34.256 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.250+0000 7f4ec259c700 1 -- 172.21.15.38:0/1216749134 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ec4100930 msgr2=0x7f4ec41018c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:34.256 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.250+0000 7f4ec259c700 1 --2- 172.21.15.38:0/1216749134 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ec4100930 0x7f4ec41018c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:34.256 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.250+0000 7f4ec259c700 1 -- 172.21.15.38:0/1216749134 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ec40fffa0 msgr2=0x7f4ec4101380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:34.257 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.250+0000 7f4ec259c700 1 --2- 172.21.15.38:0/1216749134 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ec40fffa0 0x7f4ec4101380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:34.257 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.250+0000 7f4ec259c700 1 -- 172.21.15.38:0/1216749134 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4eb40095f0 con 0x7f4ec40fedb0 2023-03-09T14:44:34.257 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.250+0000 7f4ec259c700 1 --2- 172.21.15.38:0/1216749134 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ec40fedb0 0x7f4ec4102d40 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f4eb800c990 tx=0x7f4eb800cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:34.257 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.250+0000 7f4eb37fe700 1 -- 172.21.15.38:0/1216749134 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eb8007940 con 0x7f4ec40fedb0 2023-03-09T14:44:34.257 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.250+0000 7f4ec88d8700 1 -- 172.21.15.38:0/1216749134 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ec410a710 con 0x7f4ec40fedb0 2023-03-09T14:44:34.257 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.250+0000 7f4eb37fe700 1 -- 172.21.15.38:0/1216749134 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4eb800f450 con 0x7f4ec40fedb0 2023-03-09T14:44:34.257 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.250+0000 7f4eb37fe700 1 -- 172.21.15.38:0/1216749134 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eb8018660 con 0x7f4ec40fedb0 2023-03-09T14:44:34.258 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.250+0000 7f4ec88d8700 1 -- 172.21.15.38:0/1216749134 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ec410a580 con 0x7f4ec40fedb0 2023-03-09T14:44:34.258 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.251+0000 7f4ec88d8700 1 -- 172.21.15.38:0/1216749134 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ec41a0290 con 0x7f4ec40fedb0 2023-03-09T14:44:34.258 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.252+0000 7f4eb37fe700 1 -- 172.21.15.38:0/1216749134 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 133198+0+0 (secure 0 0 0) 0x7f4eb800f8c0 con 0x7f4ec40fedb0 2023-03-09T14:44:34.259 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.253+0000 7f4eb37fe700 1 --2- 172.21.15.38:0/1216749134 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f4eac09f3c0 0x7f4eac0a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:34.259 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.253+0000 7f4ec1d9b700 1 --2- 172.21.15.38:0/1216749134 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f4eac09f3c0 0x7f4eac0a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:34.259 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.253+0000 7f4eb37fe700 1 -- 172.21.15.38:0/1216749134 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f4eb80ca080 con 0x7f4ec40fedb0 2023-03-09T14:44:34.259 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.253+0000 7f4ec1d9b700 1 --2- 172.21.15.38:0/1216749134 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f4eac09f3c0 0x7f4eac0a1870 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f4eb400b5c0 tx=0x7f4eb4005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:34.260 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.257+0000 7f4eb37fe700 1 -- 172.21.15.38:0/1216749134 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f4eb80941f0 con 0x7f4ec40fedb0 2023-03-09T14:44:34.460 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.457+0000 7f4ec88d8700 1 -- 172.21.15.38:0/1216749134 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f4ec41a0840 con 0x7f4ec40fedb0 2023-03-09T14:44:34.461 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.458+0000 7f4eb37fe700 1 -- 172.21.15.38:0/1216749134 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4eb8097810 con 0x7f4ec40fedb0 2023-03-09T14:44:34.461 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 64 2023-03-09T14:44:34.463 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.460+0000 7f4ec88d8700 1 -- 172.21.15.38:0/1216749134 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f4eac09f3c0 msgr2=0x7f4eac0a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:34.463 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.460+0000 7f4ec88d8700 1 --2- 172.21.15.38:0/1216749134 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f4eac09f3c0 0x7f4eac0a1870 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f4eb400b5c0 tx=0x7f4eb4005960 comp rx=0 tx=0).stop 2023-03-09T14:44:34.463 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.460+0000 7f4ec88d8700 1 -- 172.21.15.38:0/1216749134 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ec40fedb0 msgr2=0x7f4ec4102d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:34.463 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.460+0000 7f4ec88d8700 1 --2- 172.21.15.38:0/1216749134 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ec40fedb0 0x7f4ec4102d40 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f4eb800c990 tx=0x7f4eb800cca0 comp rx=0 tx=0).stop 2023-03-09T14:44:34.464 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.460+0000 7f4ec88d8700 1 -- 172.21.15.38:0/1216749134 shutdown_connections 2023-03-09T14:44:34.464 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.460+0000 7f4ec88d8700 1 --2- 172.21.15.38:0/1216749134 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4ec40fedb0 0x7f4ec4102d40 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:34.464 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.460+0000 7f4ec88d8700 1 --2- 172.21.15.38:0/1216749134 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f4eac09f3c0 0x7f4eac0a1870 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:34.464 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.460+0000 7f4ec88d8700 1 --2- 172.21.15.38:0/1216749134 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4ec4100930 0x7f4ec41018c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:34.464 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.460+0000 7f4ec88d8700 1 --2- 172.21.15.38:0/1216749134 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4ec40fffa0 0x7f4ec4101380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:34.464 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.460+0000 7f4ec88d8700 1 -- 172.21.15.38:0/1216749134 >> 172.21.15.38:0/1216749134 conn(0x7f4ec40fa3b0 msgr2=0x7f4ec4068a90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:34.464 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.460+0000 7f4ec88d8700 1 -- 172.21.15.38:0/1216749134 shutdown_connections 2023-03-09T14:44:34.464 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.460+0000 7f4ec88d8700 1 -- 172.21.15.38:0/1216749134 wait complete. 2023-03-09T14:44:34.475 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-03-09T14:44:34.604 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.600+0000 7f8f62762700 1 -- 172.21.15.38:0/2281246131 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f5c100910 msgr2=0x7f8f5c1071d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:34.604 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.600+0000 7f8f62762700 1 --2- 172.21.15.38:0/2281246131 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f5c100910 0x7f8f5c1071d0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f8f5400b0a0 tx=0x7f8f5400b3b0 comp rx=0 tx=0).stop 2023-03-09T14:44:34.604 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.600+0000 7f8f62762700 1 -- 172.21.15.38:0/2281246131 shutdown_connections 2023-03-09T14:44:34.605 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.600+0000 7f8f62762700 1 --2- 172.21.15.38:0/2281246131 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f5c100910 0x7f8f5c1071d0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:34.605 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.600+0000 7f8f62762700 1 --2- 172.21.15.38:0/2281246131 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f5c0fed90 0x7f8f5c0ff1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:34.605 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.600+0000 7f8f62762700 1 --2- 172.21.15.38:0/2281246131 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8f5c0fff80 0x7f8f5c1003d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:34.605 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.600+0000 7f8f62762700 1 -- 172.21.15.38:0/2281246131 >> 172.21.15.38:0/2281246131 conn(0x7f8f5c0fa3b0 msgr2=0x7f8f5c0fc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:34.605 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.600+0000 7f8f62762700 1 -- 172.21.15.38:0/2281246131 shutdown_connections 2023-03-09T14:44:34.605 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.600+0000 7f8f62762700 1 -- 172.21.15.38:0/2281246131 wait complete. 2023-03-09T14:44:34.605 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.601+0000 7f8f62762700 1 Processor -- start 2023-03-09T14:44:34.606 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.601+0000 7f8f62762700 1 -- start start 2023-03-09T14:44:34.607 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.601+0000 7f8f62762700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f5c0fed90 0x7f8f5c194c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:34.608 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.601+0000 7f8f62762700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f5c0fff80 0x7f8f5c195170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:34.608 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.601+0000 7f8f62762700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8f5c100910 0x7f8f5c19c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:34.608 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.601+0000 7f8f62762700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f5c19c770 con 0x7f8f5c0fff80 2023-03-09T14:44:34.608 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.601+0000 7f8f62762700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f5c19c8e0 con 0x7f8f5c100910 2023-03-09T14:44:34.608 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.601+0000 7f8f62762700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f5c19caa0 con 0x7f8f5c0fed90 2023-03-09T14:44:34.608 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.601+0000 7f8f5b7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f5c0fff80 0x7f8f5c195170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:34.608 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.601+0000 7f8f5b7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f5c0fff80 0x7f8f5c195170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37022/0 (socket says 172.21.15.38:37022) 2023-03-09T14:44:34.609 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.601+0000 7f8f5b7fe700 1 -- 172.21.15.38:0/2597660378 learned_addr learned my addr 172.21.15.38:0/2597660378 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:44:34.609 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.601+0000 7f8f60cff700 1 --2- 172.21.15.38:0/2597660378 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8f5c100910 0x7f8f5c19c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:34.609 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.601+0000 7f8f5bfff700 1 --2- 172.21.15.38:0/2597660378 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f5c0fed90 0x7f8f5c194c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:34.609 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.602+0000 7f8f5b7fe700 1 -- 172.21.15.38:0/2597660378 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f5c0fed90 msgr2=0x7f8f5c194c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:34.609 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.602+0000 7f8f5b7fe700 1 --2- 172.21.15.38:0/2597660378 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f5c0fed90 0x7f8f5c194c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:34.609 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.602+0000 7f8f5b7fe700 1 -- 172.21.15.38:0/2597660378 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8f5c100910 msgr2=0x7f8f5c19c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:34.610 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.602+0000 7f8f5b7fe700 1 --2- 172.21.15.38:0/2597660378 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8f5c100910 0x7f8f5c19c1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:34.610 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.602+0000 7f8f5b7fe700 1 -- 172.21.15.38:0/2597660378 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f54009d00 con 0x7f8f5c0fff80 2023-03-09T14:44:34.610 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.602+0000 7f8f5b7fe700 1 --2- 172.21.15.38:0/2597660378 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f5c0fff80 0x7f8f5c195170 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f8f5000e9a0 tx=0x7f8f5000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:34.610 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.602+0000 7f8f597fa700 1 -- 172.21.15.38:0/2597660378 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f50009790 con 0x7f8f5c0fff80 2023-03-09T14:44:34.610 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.602+0000 7f8f62762700 1 -- 172.21.15.38:0/2597660378 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f5c10a700 con 0x7f8f5c0fff80 2023-03-09T14:44:34.610 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.602+0000 7f8f597fa700 1 -- 172.21.15.38:0/2597660378 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8f50010430 con 0x7f8f5c0fff80 2023-03-09T14:44:34.610 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.602+0000 7f8f597fa700 1 -- 172.21.15.38:0/2597660378 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f50018680 con 0x7f8f5c0fff80 2023-03-09T14:44:34.610 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.602+0000 7f8f62762700 1 -- 172.21.15.38:0/2597660378 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f5c10a570 con 0x7f8f5c0fff80 2023-03-09T14:44:34.611 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.603+0000 7f8f62762700 1 -- 172.21.15.38:0/2597660378 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f5c1062e0 con 0x7f8f5c0fff80 2023-03-09T14:44:34.611 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.604+0000 7f8f597fa700 1 -- 172.21.15.38:0/2597660378 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 133198+0+0 (secure 0 0 0) 0x7f8f5000c040 con 0x7f8f5c0fff80 2023-03-09T14:44:34.611 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.605+0000 7f8f597fa700 1 --2- 172.21.15.38:0/2597660378 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f8f4409f3c0 0x7f8f440a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:34.612 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.605+0000 7f8f597fa700 1 -- 172.21.15.38:0/2597660378 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f8f50014070 con 0x7f8f5c0fff80 2023-03-09T14:44:34.612 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.605+0000 7f8f5bfff700 1 --2- 172.21.15.38:0/2597660378 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f8f4409f3c0 0x7f8f440a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:34.612 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.605+0000 7f8f5bfff700 1 --2- 172.21.15.38:0/2597660378 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f8f4409f3c0 0x7f8f440a1870 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c196100 tx=0x7f8f4c00b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:34.612 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.609+0000 7f8f597fa700 1 -- 172.21.15.38:0/2597660378 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f8f500515b0 con 0x7f8f5c0fff80 2023-03-09T14:44:34.815 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.812+0000 7f8f62762700 1 -- 172.21.15.38:0/2597660378 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f8f5c0618b0 con 0x7f8f5c0fff80 2023-03-09T14:44:34.815 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.812+0000 7f8f597fa700 1 -- 172.21.15.38:0/2597660378 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v87) v1 ==== 103+0+11 (secure 0 0 0) 0x7f8f50094350 con 0x7f8f5c0fff80 2023-03-09T14:44:34.816 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 32 2023-03-09T14:44:34.817 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.814+0000 7f8f62762700 1 -- 172.21.15.38:0/2597660378 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f8f4409f3c0 msgr2=0x7f8f440a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:34.818 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.814+0000 7f8f62762700 1 --2- 172.21.15.38:0/2597660378 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f8f4409f3c0 0x7f8f440a1870 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c196100 tx=0x7f8f4c00b410 comp rx=0 tx=0).stop 2023-03-09T14:44:34.818 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.814+0000 7f8f62762700 1 -- 172.21.15.38:0/2597660378 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f5c0fff80 msgr2=0x7f8f5c195170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:34.818 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.814+0000 7f8f62762700 1 --2- 172.21.15.38:0/2597660378 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f5c0fff80 0x7f8f5c195170 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f8f5000e9a0 tx=0x7f8f5000ecb0 comp rx=0 tx=0).stop 2023-03-09T14:44:34.818 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.814+0000 7f8f62762700 1 -- 172.21.15.38:0/2597660378 shutdown_connections 2023-03-09T14:44:34.818 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.814+0000 7f8f62762700 1 --2- 172.21.15.38:0/2597660378 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f8f4409f3c0 0x7f8f440a1870 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:34.818 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.814+0000 7f8f62762700 1 --2- 172.21.15.38:0/2597660378 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8f5c100910 0x7f8f5c19c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:34.819 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.814+0000 7f8f62762700 1 --2- 172.21.15.38:0/2597660378 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f5c0fed90 0x7f8f5c194c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:34.819 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.814+0000 7f8f62762700 1 --2- 172.21.15.38:0/2597660378 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f5c0fff80 0x7f8f5c195170 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:34.819 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.814+0000 7f8f62762700 1 -- 172.21.15.38:0/2597660378 >> 172.21.15.38:0/2597660378 conn(0x7f8f5c0fa3b0 msgr2=0x7f8f5c103420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:34.819 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.814+0000 7f8f62762700 1 -- 172.21.15.38:0/2597660378 shutdown_connections 2023-03-09T14:44:34.819 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.814+0000 7f8f62762700 1 -- 172.21.15.38:0/2597660378 wait complete. 2023-03-09T14:44:34.830 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-03-09T14:44:34.957 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.953+0000 7fada636e700 1 -- 172.21.15.38:0/4273658214 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fada0100fe0 msgr2=0x7fada0101450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:34.957 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.953+0000 7fada636e700 1 --2- 172.21.15.38:0/4273658214 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fada0100fe0 0x7fada0101450 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fad90009990 tx=0x7fad90009ca0 comp rx=0 tx=0).stop 2023-03-09T14:44:34.958 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.953+0000 7fada636e700 1 -- 172.21.15.38:0/4273658214 shutdown_connections 2023-03-09T14:44:34.958 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.953+0000 7fada636e700 1 --2- 172.21.15.38:0/4273658214 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fada0101990 0x7fada01082c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:34.958 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.953+0000 7fada636e700 1 --2- 172.21.15.38:0/4273658214 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fada00ffe90 0x7fada01002a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:34.958 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.953+0000 7fada636e700 1 --2- 172.21.15.38:0/4273658214 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fada0100fe0 0x7fada0101450 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:34.958 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.954+0000 7fada636e700 1 -- 172.21.15.38:0/4273658214 >> 172.21.15.38:0/4273658214 conn(0x7fada00fb4d0 msgr2=0x7fada00fd8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:34.958 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.954+0000 7fada636e700 1 -- 172.21.15.38:0/4273658214 shutdown_connections 2023-03-09T14:44:34.958 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.954+0000 7fada636e700 1 -- 172.21.15.38:0/4273658214 wait complete. 2023-03-09T14:44:34.960 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.954+0000 7fada636e700 1 Processor -- start 2023-03-09T14:44:34.961 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.954+0000 7fada636e700 1 -- start start 2023-03-09T14:44:34.961 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.955+0000 7fada636e700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fada00ffe90 0x7fada0195bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:34.961 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.955+0000 7fada636e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fada0100fe0 0x7fada01960f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:34.961 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.955+0000 7fada636e700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fada0101990 0x7fada019d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:34.961 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.955+0000 7fada636e700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fada019d730 con 0x7fada0100fe0 2023-03-09T14:44:34.962 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.955+0000 7fada636e700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fada019d870 con 0x7fada0101990 2023-03-09T14:44:34.962 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.955+0000 7fada636e700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fada019da00 con 0x7fada00ffe90 2023-03-09T14:44:34.962 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.955+0000 7fad9f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fada0100fe0 0x7fada01960f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:34.962 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.955+0000 7fad9f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fada0100fe0 0x7fada01960f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37038/0 (socket says 172.21.15.38:37038) 2023-03-09T14:44:34.962 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.955+0000 7fad9f7fe700 1 -- 172.21.15.38:0/3117465526 learned_addr learned my addr 172.21.15.38:0/3117465526 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:44:34.962 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.955+0000 7fad9ffff700 1 --2- 172.21.15.38:0/3117465526 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fada00ffe90 0x7fada0195bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:34.962 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.955+0000 7fada490b700 1 --2- 172.21.15.38:0/3117465526 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fada0101990 0x7fada019d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:34.963 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.955+0000 7fad9f7fe700 1 -- 172.21.15.38:0/3117465526 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fada00ffe90 msgr2=0x7fada0195bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:34.963 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.955+0000 7fad9f7fe700 1 --2- 172.21.15.38:0/3117465526 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fada00ffe90 0x7fada0195bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:34.963 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.955+0000 7fad9f7fe700 1 -- 172.21.15.38:0/3117465526 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fada0101990 msgr2=0x7fada019d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:34.963 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.955+0000 7fad9f7fe700 1 --2- 172.21.15.38:0/3117465526 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fada0101990 0x7fada019d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:34.963 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.955+0000 7fad9f7fe700 1 -- 172.21.15.38:0/3117465526 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad900095f0 con 0x7fada0100fe0 2023-03-09T14:44:34.963 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.955+0000 7fad9ffff700 1 --2- 172.21.15.38:0/3117465526 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fada00ffe90 0x7fada0195bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-09T14:44:34.963 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.955+0000 7fada490b700 1 --2- 172.21.15.38:0/3117465526 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fada0101990 0x7fada019d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-09T14:44:34.964 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.955+0000 7fad9f7fe700 1 --2- 172.21.15.38:0/3117465526 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fada0100fe0 0x7fada01960f0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fad90005910 tx=0x7fad9000f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:34.964 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.956+0000 7fad9d7fa700 1 -- 172.21.15.38:0/3117465526 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad9001d070 con 0x7fada0100fe0 2023-03-09T14:44:34.964 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.956+0000 7fad9d7fa700 1 -- 172.21.15.38:0/3117465526 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fad9000fc30 con 0x7fada0100fe0 2023-03-09T14:44:34.964 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.956+0000 7fad9d7fa700 1 -- 172.21.15.38:0/3117465526 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad90017650 con 0x7fada0100fe0 2023-03-09T14:44:34.964 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.956+0000 7fada636e700 1 -- 172.21.15.38:0/3117465526 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fada010b840 con 0x7fada0100fe0 2023-03-09T14:44:34.964 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.956+0000 7fada636e700 1 -- 172.21.15.38:0/3117465526 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fada010b6b0 con 0x7fada0100fe0 2023-03-09T14:44:34.964 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.957+0000 7fada636e700 1 -- 172.21.15.38:0/3117465526 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fada01073c0 con 0x7fada0100fe0 2023-03-09T14:44:34.965 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.958+0000 7fad9d7fa700 1 -- 172.21.15.38:0/3117465526 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 133198+0+0 (secure 0 0 0) 0x7fad90029030 con 0x7fada0100fe0 2023-03-09T14:44:34.965 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.958+0000 7fad9d7fa700 1 --2- 172.21.15.38:0/3117465526 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fad8809f3c0 0x7fad880a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:34.965 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.958+0000 7fad9ffff700 1 --2- 172.21.15.38:0/3117465526 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fad8809f3c0 0x7fad880a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:34.966 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.959+0000 7fad9d7fa700 1 -- 172.21.15.38:0/3117465526 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fad900cb450 con 0x7fada0100fe0 2023-03-09T14:44:34.966 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.959+0000 7fad9ffff700 1 --2- 172.21.15.38:0/3117465526 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fad8809f3c0 0x7fad880a1870 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fad94005fd0 tx=0x7fad94005e20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:34.966 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:34.963+0000 7fad9d7fa700 1 -- 172.21.15.38:0/3117465526 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7fad900956b0 con 0x7fada0100fe0 2023-03-09T14:44:35.186 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.183+0000 7fada636e700 1 -- 172.21.15.38:0/3117465526 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fada00615c0 con 0x7fada0100fe0 2023-03-09T14:44:35.187 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.183+0000 7fad9d7fa700 1 -- 172.21.15.38:0/3117465526 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v87) v1 ==== 81+0+591 (secure 0 0 0) 0x7fad90027260 con 0x7fada0100fe0 2023-03-09T14:44:35.187 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:0/4272288735 2023-03-10T14:38:14.612767+0000 2023-03-09T14:44:35.187 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:6801/3381625669 2023-03-10T14:38:14.612767+0000 2023-03-09T14:44:35.187 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:6800/3381625669 2023-03-10T14:38:14.612767+0000 2023-03-09T14:44:35.188 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:6800/3060474342 2023-03-10T14:37:47.517775+0000 2023-03-09T14:44:35.188 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:0/4253918615 2023-03-10T14:38:14.612767+0000 2023-03-09T14:44:35.188 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:0/3496083181 2023-03-10T14:37:47.517775+0000 2023-03-09T14:44:35.188 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:0/245602368 2023-03-10T14:37:47.517775+0000 2023-03-09T14:44:35.188 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:6801/3060474342 2023-03-10T14:37:47.517775+0000 2023-03-09T14:44:35.188 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:0/3622589182 2023-03-10T14:37:47.517775+0000 2023-03-09T14:44:35.188 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:0/1430302251 2023-03-10T14:38:14.612767+0000 2023-03-09T14:44:35.189 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.186+0000 7fada636e700 1 -- 172.21.15.38:0/3117465526 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fad8809f3c0 msgr2=0x7fad880a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:35.189 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.186+0000 7fada636e700 1 --2- 172.21.15.38:0/3117465526 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fad8809f3c0 0x7fad880a1870 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fad94005fd0 tx=0x7fad94005e20 comp rx=0 tx=0).stop 2023-03-09T14:44:35.189 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.186+0000 7fada636e700 1 -- 172.21.15.38:0/3117465526 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fada0100fe0 msgr2=0x7fada01960f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:35.190 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.186+0000 7fada636e700 1 --2- 172.21.15.38:0/3117465526 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fada0100fe0 0x7fada01960f0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fad90005910 tx=0x7fad9000f740 comp rx=0 tx=0).stop 2023-03-09T14:44:35.190 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.186+0000 7fada636e700 1 -- 172.21.15.38:0/3117465526 shutdown_connections 2023-03-09T14:44:35.190 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.186+0000 7fada636e700 1 --2- 172.21.15.38:0/3117465526 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fad8809f3c0 0x7fad880a1870 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:35.190 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.186+0000 7fada636e700 1 --2- 172.21.15.38:0/3117465526 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fada0101990 0x7fada019d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:35.190 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.186+0000 7fada636e700 1 --2- 172.21.15.38:0/3117465526 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fada00ffe90 0x7fada0195bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:35.190 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.186+0000 7fada636e700 1 --2- 172.21.15.38:0/3117465526 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fada0100fe0 0x7fada01960f0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:35.190 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.186+0000 7fada636e700 1 -- 172.21.15.38:0/3117465526 >> 172.21.15.38:0/3117465526 conn(0x7fada00fb4d0 msgr2=0x7fada0104500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:35.191 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.186+0000 7fada636e700 1 -- 172.21.15.38:0/3117465526 shutdown_connections 2023-03-09T14:44:35.191 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.186+0000 7fada636e700 1 -- 172.21.15.38:0/3117465526 wait complete. 2023-03-09T14:44:35.191 INFO:teuthology.orchestra.run.smithi038.stderr:listed 10 entries 2023-03-09T14:44:35.200 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:34 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/1624215123' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-03-09T14:44:35.200 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:34 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/1216749134' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-03-09T14:44:35.200 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:34 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/2597660378' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-03-09T14:44:35.201 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi038.front.sepia.ceph.com... 2023-03-09T14:44:35.201 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-09T14:44:35.202 DEBUG:teuthology.orchestra.run.smithi038:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2023-03-09T14:44:35.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:34 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/1624215123' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-03-09T14:44:35.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:34 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/1216749134' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-03-09T14:44:35.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:34 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/2597660378' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-03-09T14:44:35.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:34 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/1624215123' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-03-09T14:44:35.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:34 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/1216749134' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-03-09T14:44:35.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:34 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/2597660378' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-03-09T14:44:35.240 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi038.stderr:ceph-fuse[129402]: fuse finished with error 0 and tester_r 0 2023-03-09T14:44:35.242 INFO:teuthology.orchestra.run:waiting for 300 2023-03-09T14:44:35.331 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi038.front.sepia.ceph.com 2023-03-09T14:44:35.332 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-09T14:44:35.332 DEBUG:teuthology.orchestra.run.smithi038:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2023-03-09T14:44:35.364 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-09T14:44:35.531 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.527+0000 7f6c96bca700 1 -- 172.21.15.38:0/1682246180 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c90105b40 msgr2=0x7f6c90107fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:35.532 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.527+0000 7f6c96bca700 1 --2- 172.21.15.38:0/1682246180 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c90105b40 0x7f6c90107fa0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f6c8800b0a0 tx=0x7f6c8800b3b0 comp rx=0 tx=0).stop 2023-03-09T14:44:35.532 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.527+0000 7f6c96bca700 1 -- 172.21.15.38:0/1682246180 shutdown_connections 2023-03-09T14:44:35.532 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.527+0000 7f6c96bca700 1 --2- 172.21.15.38:0/1682246180 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c90105b40 0x7f6c90107fa0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:35.532 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.527+0000 7f6c96bca700 1 --2- 172.21.15.38:0/1682246180 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c90100900 0x7f6c90102ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:35.532 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.527+0000 7f6c96bca700 1 --2- 172.21.15.38:0/1682246180 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c90103220 0x7f6c90105600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:35.533 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.527+0000 7f6c96bca700 1 -- 172.21.15.38:0/1682246180 >> 172.21.15.38:0/1682246180 conn(0x7f6c900fa380 msgr2=0x7f6c900fc7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:35.533 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.527+0000 7f6c96bca700 1 -- 172.21.15.38:0/1682246180 shutdown_connections 2023-03-09T14:44:35.533 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.527+0000 7f6c96bca700 1 -- 172.21.15.38:0/1682246180 wait complete. 2023-03-09T14:44:35.533 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.527+0000 7f6c96bca700 1 Processor -- start 2023-03-09T14:44:35.533 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.527+0000 7f6c96bca700 1 -- start start 2023-03-09T14:44:35.533 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.527+0000 7f6c96bca700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c90100900 0x7f6c90195d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:35.533 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.527+0000 7f6c96bca700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c90103220 0x7f6c90196270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:35.534 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.527+0000 7f6c96bca700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c90105b40 0x7f6c9019d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:35.534 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.527+0000 7f6c96bca700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c9019d870 con 0x7f6c90103220 2023-03-09T14:44:35.534 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.527+0000 7f6c96bca700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c9019d9e0 con 0x7f6c90100900 2023-03-09T14:44:35.534 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.527+0000 7f6c96bca700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c9019dba0 con 0x7f6c90105b40 2023-03-09T14:44:35.534 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.528+0000 7f6c8ffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c90103220 0x7f6c90196270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:35.534 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.528+0000 7f6c8ffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c90103220 0x7f6c90196270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37052/0 (socket says 172.21.15.38:37052) 2023-03-09T14:44:35.534 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.528+0000 7f6c8ffff700 1 -- 172.21.15.38:0/583663196 learned_addr learned my addr 172.21.15.38:0/583663196 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:44:35.535 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.528+0000 7f6c95167700 1 --2- 172.21.15.38:0/583663196 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c90105b40 0x7f6c9019d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:35.535 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.528+0000 7f6c94966700 1 --2- 172.21.15.38:0/583663196 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c90100900 0x7f6c90195d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:35.535 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.528+0000 7f6c8ffff700 1 -- 172.21.15.38:0/583663196 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c90105b40 msgr2=0x7f6c9019d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:35.535 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.528+0000 7f6c8ffff700 1 --2- 172.21.15.38:0/583663196 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c90105b40 0x7f6c9019d2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:35.535 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.528+0000 7f6c8ffff700 1 -- 172.21.15.38:0/583663196 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c90100900 msgr2=0x7f6c90195d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:35.535 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.528+0000 7f6c8ffff700 1 --2- 172.21.15.38:0/583663196 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c90100900 0x7f6c90195d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:35.535 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.528+0000 7f6c8ffff700 1 -- 172.21.15.38:0/583663196 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c88009d00 con 0x7f6c90103220 2023-03-09T14:44:35.535 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.528+0000 7f6c94966700 1 --2- 172.21.15.38:0/583663196 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c90100900 0x7f6c90195d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:44:35.536 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.528+0000 7f6c95167700 1 --2- 172.21.15.38:0/583663196 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c90105b40 0x7f6c9019d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:44:35.536 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.528+0000 7f6c8ffff700 1 --2- 172.21.15.38:0/583663196 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c90103220 0x7f6c90196270 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c00d950 tx=0x7f6c7c00dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:35.536 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.528+0000 7f6c8dffb700 1 -- 172.21.15.38:0/583663196 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c7c009b40 con 0x7f6c90103220 2023-03-09T14:44:35.536 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.528+0000 7f6c8dffb700 1 -- 172.21.15.38:0/583663196 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6c7c009ca0 con 0x7f6c90103220 2023-03-09T14:44:35.536 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.528+0000 7f6c8dffb700 1 -- 172.21.15.38:0/583663196 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c7c00f620 con 0x7f6c90103220 2023-03-09T14:44:35.536 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.528+0000 7f6c96bca700 1 -- 172.21.15.38:0/583663196 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c9010b810 con 0x7f6c90103220 2023-03-09T14:44:35.536 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.528+0000 7f6c96bca700 1 -- 172.21.15.38:0/583663196 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c9010b680 con 0x7f6c90103220 2023-03-09T14:44:35.537 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.529+0000 7f6c96bca700 1 -- 172.21.15.38:0/583663196 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c900fbef0 con 0x7f6c90103220 2023-03-09T14:44:35.537 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.530+0000 7f6c8dffb700 1 -- 172.21.15.38:0/583663196 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 133198+0+0 (secure 0 0 0) 0x7f6c7c018030 con 0x7f6c90103220 2023-03-09T14:44:35.538 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.531+0000 7f6c8dffb700 1 --2- 172.21.15.38:0/583663196 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f6c8009f440 0x7f6c800a18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:35.538 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.531+0000 7f6c8dffb700 1 -- 172.21.15.38:0/583663196 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f6c7c0ca190 con 0x7f6c90103220 2023-03-09T14:44:35.538 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.531+0000 7f6c94966700 1 --2- 172.21.15.38:0/583663196 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f6c8009f440 0x7f6c800a18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:35.538 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.531+0000 7f6c94966700 1 --2- 172.21.15.38:0/583663196 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f6c8009f440 0x7f6c800a18f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f6c84005950 tx=0x7f6c84005f50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:35.538 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.535+0000 7f6c8dffb700 1 -- 172.21.15.38:0/583663196 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f6c7c094410 con 0x7f6c90103220 2023-03-09T14:44:35.736 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.733+0000 7f6c96bca700 1 -- 172.21.15.38:0/583663196 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6c9002cef0 con 0x7f6c90103220 2023-03-09T14:44:35.738 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.735+0000 7f6c8dffb700 1 -- 172.21.15.38:0/583663196 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+23973 (secure 0 0 0) 0x7f6c7c016020 con 0x7f6c90103220 2023-03-09T14:44:35.738 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:44:35.739 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":87,"fsid":"c6733fe2-be87-11ed-9af2-001a4aab830c","created":"2023-03-09T14:37:14.947981+0000","modified":"2023-03-09T14:44:23.900228+0000","last_up_change":"2023-03-09T14:43:37.198407+0000","last_in_change":"2023-03-09T14:43:19.550566+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-09T14:40:23.196665+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-03-09T14:44:17.484374+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-03-09T14:44:18.457833+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-03-09T14:44:21.895589+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"51477039-15ee-4707-a34d-4ecfa18bebde","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6803","nonce":1508072290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6805","nonce":1508072290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6809","nonce":1508072290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6807","nonce":1508072290}]},"public_addr":"172.21.15.38:6803/1508072290","cluster_addr":"172.21.15.38:6805/1508072290","heartbeat_back_addr":"172.21.15.38:6809/1508072290","heartbeat_front_addr":"172.21.15.38:6807/1508072290","state":["exists","up"]},{"osd":1,"uuid":"71f8a718-b0a9-4f39-bc54-267aa42a2578","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6801","nonce":3929268773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6803","nonce":3929268773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6807","nonce":3929268773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6805","nonce":3929268773}]},"public_addr":"172.21.15.122:6801/3929268773","cluster_addr":"172.21.15.122:6803/3929268773","heartbeat_back_addr":"172.21.15.122:6807/3929268773","heartbeat_front_addr":"172.21.15.122:6805/3929268773","state":["exists","up"]},{"osd":2,"uuid":"961a2cbe-140e-437f-b131-1fcac573b19c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6801","nonce":1310630803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6803","nonce":1310630803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6807","nonce":1310630803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6805","nonce":1310630803}]},"public_addr":"172.21.15.181:6801/1310630803","cluster_addr":"172.21.15.181:6803/1310630803","heartbeat_back_addr":"172.21.15.181:6807/1310630803","heartbeat_front_addr":"172.21.15.181:6805/1310630803","state":["exists","up"]},{"osd":3,"uuid":"969df8fa-2d19-42c7-9f2a-5c5fac15d4a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6811","nonce":2719814100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6813","nonce":2719814100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6817","nonce":2719814100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6815","nonce":2719814100}]},"public_addr":"172.21.15.38:6811/2719814100","cluster_addr":"172.21.15.38:6813/2719814100","heartbeat_back_addr":"172.21.15.38:6817/2719814100","heartbeat_front_addr":"172.21.15.38:6815/2719814100","state":["exists","up"]},{"osd":4,"uuid":"44a6dce3-e06e-4cbd-9404-62a86d01ab6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6809","nonce":2480764994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6811","nonce":2480764994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6815","nonce":2480764994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6813","nonce":2480764994}]},"public_addr":"172.21.15.122:6809/2480764994","cluster_addr":"172.21.15.122:6811/2480764994","heartbeat_back_addr":"172.21.15.122:6815/2480764994","heartbeat_front_addr":"172.21.15.122:6813/2480764994","state":["exists","up"]},{"osd":5,"uuid":"49749574-c00c-4e3a-80e6-e3686df58d5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6809","nonce":3141346246}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6811","nonce":3141346246}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6815","nonce":3141346246}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6813","nonce":3141346246}]},"public_addr":"172.21.15.181:6809/3141346246","cluster_addr":"172.21.15.181:6811/3141346246","heartbeat_back_addr":"172.21.15.181:6815/3141346246","heartbeat_front_addr":"172.21.15.181:6813/3141346246","state":["exists","up"]},{"osd":6,"uuid":"61c0437e-3f51-45a9-8a61-e98eea0093f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6819","nonce":2122788957}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6821","nonce":2122788957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6825","nonce":2122788957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6823","nonce":2122788957}]},"public_addr":"172.21.15.38:6819/2122788957","cluster_addr":"172.21.15.38:6821/2122788957","heartbeat_back_addr":"172.21.15.38:6825/2122788957","heartbeat_front_addr":"172.21.15.38:6823/2122788957","state":["exists","up"]},{"osd":7,"uuid":"231fecdd-939f-4c7b-83a1-c4a1cffd7139","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6817","nonce":656874139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6819","nonce":656874139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6823","nonce":656874139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6821","nonce":656874139}]},"public_addr":"172.21.15.122:6817/656874139","cluster_addr":"172.21.15.122:6819/656874139","heartbeat_back_addr":"172.21.15.122:6823/656874139","heartbeat_front_addr":"172.21.15.122:6821/656874139","state":["exists","up"]},{"osd":8,"uuid":"297e6d6d-85fd-43ce-9028-5e9dbe55c36e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6817","nonce":755000703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6819","nonce":755000703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6823","nonce":755000703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6821","nonce":755000703}]},"public_addr":"172.21.15.181:6817/755000703","cluster_addr":"172.21.15.181:6819/755000703","heartbeat_back_addr":"172.21.15.181:6823/755000703","heartbeat_front_addr":"172.21.15.181:6821/755000703","state":["exists","up"]},{"osd":9,"uuid":"94cc0355-5a1a-4cfc-bd0c-562b6b41127e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6827","nonce":2624555969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6829","nonce":2624555969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6833","nonce":2624555969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6831","nonce":2624555969}]},"public_addr":"172.21.15.38:6827/2624555969","cluster_addr":"172.21.15.38:6829/2624555969","heartbeat_back_addr":"172.21.15.38:6833/2624555969","heartbeat_front_addr":"172.21.15.38:6831/2624555969","state":["exists","up"]},{"osd":10,"uuid":"f4f2e042-8d92-46bf-93b8-8b0f04795c2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6825","nonce":1427785280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6827","nonce":1427785280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6831","nonce":1427785280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6829","nonce":1427785280}]},"public_addr":"172.21.15.122:6825/1427785280","cluster_addr":"172.21.15.122:6827/1427785280","heartbeat_back_addr":"172.21.15.122:6831/1427785280","heartbeat_front_addr":"172.21.15.122:6829/1427785280","state":["exists","up"]},{"osd":11,"uuid":"07444936-eae5-4097-a799-60294cafe80b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6825","nonce":3183488640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6827","nonce":3183488640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6831","nonce":3183488640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6829","nonce":3183488640}]},"public_addr":"172.21.15.181:6825/3183488640","cluster_addr":"172.21.15.181:6827/3183488640","heartbeat_back_addr":"172.21.15.181:6831/3183488640","heartbeat_front_addr":"172.21.15.181:6829/3183488640","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:39:37.657556+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:39:58.960074+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:40:20.326952+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:40:42.244387+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:41:03.946973+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:41:25.097985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:41:46.877916+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:42:08.413827+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:42:29.822009+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:42:51.776684+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:43:13.937249+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:43:35.569199+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/4272288735":"2023-03-10T14:38:14.612767+0000","172.21.15.38:6801/3381625669":"2023-03-10T14:38:14.612767+0000","172.21.15.38:6800/3381625669":"2023-03-10T14:38:14.612767+0000","172.21.15.38:6800/3060474342":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/4253918615":"2023-03-10T14:38:14.612767+0000","172.21.15.38:0/3496083181":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/245602368":"2023-03-10T14:37:47.517775+0000","172.21.15.38:6801/3060474342":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/3622589182":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/1430302251":"2023-03-10T14:38:14.612767+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-09T14:44:35.740 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.737+0000 7f6c96bca700 1 -- 172.21.15.38:0/583663196 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f6c8009f440 msgr2=0x7f6c800a18f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:35.740 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.737+0000 7f6c96bca700 1 --2- 172.21.15.38:0/583663196 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f6c8009f440 0x7f6c800a18f0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f6c84005950 tx=0x7f6c84005f50 comp rx=0 tx=0).stop 2023-03-09T14:44:35.740 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.737+0000 7f6c96bca700 1 -- 172.21.15.38:0/583663196 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c90103220 msgr2=0x7f6c90196270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:35.740 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.737+0000 7f6c96bca700 1 --2- 172.21.15.38:0/583663196 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c90103220 0x7f6c90196270 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c00d950 tx=0x7f6c7c00dd10 comp rx=0 tx=0).stop 2023-03-09T14:44:35.741 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.737+0000 7f6c96bca700 1 -- 172.21.15.38:0/583663196 shutdown_connections 2023-03-09T14:44:35.741 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.737+0000 7f6c96bca700 1 --2- 172.21.15.38:0/583663196 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f6c8009f440 0x7f6c800a18f0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:35.741 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.737+0000 7f6c96bca700 1 --2- 172.21.15.38:0/583663196 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c90105b40 0x7f6c9019d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:35.741 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.737+0000 7f6c96bca700 1 --2- 172.21.15.38:0/583663196 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c90100900 0x7f6c90195d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:35.741 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.737+0000 7f6c96bca700 1 --2- 172.21.15.38:0/583663196 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c90103220 0x7f6c90196270 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:35.741 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.737+0000 7f6c96bca700 1 -- 172.21.15.38:0/583663196 >> 172.21.15.38:0/583663196 conn(0x7f6c900fa380 msgr2=0x7f6c90101580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:35.741 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.737+0000 7f6c96bca700 1 -- 172.21.15.38:0/583663196 shutdown_connections 2023-03-09T14:44:35.742 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.737+0000 7f6c96bca700 1 -- 172.21.15.38:0/583663196 wait complete. 2023-03-09T14:44:35.753 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-09T14:44:35.879 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.875+0000 7f74eaa50700 1 -- 172.21.15.38:0/3744282196 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74e4100910 msgr2=0x7f74e41071d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:35.879 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.875+0000 7f74eaa50700 1 --2- 172.21.15.38:0/3744282196 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74e4100910 0x7f74e41071d0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f74dc00b0a0 tx=0x7f74dc00b3b0 comp rx=0 tx=0).stop 2023-03-09T14:44:35.880 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.875+0000 7f74eaa50700 1 -- 172.21.15.38:0/3744282196 shutdown_connections 2023-03-09T14:44:35.880 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.875+0000 7f74eaa50700 1 --2- 172.21.15.38:0/3744282196 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74e4100910 0x7f74e41071d0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:35.880 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.875+0000 7f74eaa50700 1 --2- 172.21.15.38:0/3744282196 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f74e40fed90 0x7f74e40ff1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:35.880 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.875+0000 7f74eaa50700 1 --2- 172.21.15.38:0/3744282196 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f74e40fff80 0x7f74e41003d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:35.880 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.875+0000 7f74eaa50700 1 -- 172.21.15.38:0/3744282196 >> 172.21.15.38:0/3744282196 conn(0x7f74e40fa370 msgr2=0x7f74e40fc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:35.880 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.876+0000 7f74eaa50700 1 -- 172.21.15.38:0/3744282196 shutdown_connections 2023-03-09T14:44:35.881 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.876+0000 7f74eaa50700 1 -- 172.21.15.38:0/3744282196 wait complete. 2023-03-09T14:44:35.881 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.876+0000 7f74eaa50700 1 Processor -- start 2023-03-09T14:44:35.881 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.876+0000 7f74eaa50700 1 -- start start 2023-03-09T14:44:35.883 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.876+0000 7f74eaa50700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f74e40fed90 0x7f74e4069650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:35.883 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.876+0000 7f74eaa50700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74e40fff80 0x7f74e419a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:35.883 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.877+0000 7f74eaa50700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f74e4100910 0x7f74e419cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:35.883 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.877+0000 7f74eaa50700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74e419d7c0 con 0x7f74e40fff80 2023-03-09T14:44:35.883 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.877+0000 7f74eaa50700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74e419d930 con 0x7f74e4100910 2023-03-09T14:44:35.884 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.877+0000 7f74eaa50700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74e419daf0 con 0x7f74e40fed90 2023-03-09T14:44:35.884 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.877+0000 7f74e37fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74e40fff80 0x7f74e419a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:35.884 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.877+0000 7f74e37fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74e40fff80 0x7f74e419a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37066/0 (socket says 172.21.15.38:37066) 2023-03-09T14:44:35.884 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.877+0000 7f74e37fe700 1 -- 172.21.15.38:0/113524522 learned_addr learned my addr 172.21.15.38:0/113524522 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:44:35.884 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.877+0000 7f74e8fed700 1 --2- 172.21.15.38:0/113524522 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f74e4100910 0x7f74e419cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:35.884 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.877+0000 7f74e3fff700 1 --2- 172.21.15.38:0/113524522 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f74e40fed90 0x7f74e4069650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:35.884 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.877+0000 7f74e37fe700 1 -- 172.21.15.38:0/113524522 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f74e40fed90 msgr2=0x7f74e4069650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:35.885 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.877+0000 7f74e37fe700 1 --2- 172.21.15.38:0/113524522 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f74e40fed90 0x7f74e4069650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:35.885 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.877+0000 7f74e37fe700 1 -- 172.21.15.38:0/113524522 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f74e4100910 msgr2=0x7f74e419cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:35.885 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.877+0000 7f74e37fe700 1 --2- 172.21.15.38:0/113524522 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f74e4100910 0x7f74e419cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:35.885 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.877+0000 7f74e37fe700 1 -- 172.21.15.38:0/113524522 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74dc009d00 con 0x7f74e40fff80 2023-03-09T14:44:35.885 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.877+0000 7f74e37fe700 1 --2- 172.21.15.38:0/113524522 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74e40fff80 0x7f74e419a9a0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f74d400c9b0 tx=0x7f74d400ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:35.885 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.877+0000 7f74e17fa700 1 -- 172.21.15.38:0/113524522 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74d4007b00 con 0x7f74e40fff80 2023-03-09T14:44:35.885 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.877+0000 7f74eaa50700 1 -- 172.21.15.38:0/113524522 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74e410a700 con 0x7f74e40fff80 2023-03-09T14:44:35.886 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.877+0000 7f74e17fa700 1 -- 172.21.15.38:0/113524522 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f74d4007c60 con 0x7f74e40fff80 2023-03-09T14:44:35.886 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.877+0000 7f74e17fa700 1 -- 172.21.15.38:0/113524522 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74d4018660 con 0x7f74e40fff80 2023-03-09T14:44:35.886 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.877+0000 7f74eaa50700 1 -- 172.21.15.38:0/113524522 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74e410a570 con 0x7f74e40fff80 2023-03-09T14:44:35.886 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.878+0000 7f74eaa50700 1 -- 172.21.15.38:0/113524522 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74e4106230 con 0x7f74e40fff80 2023-03-09T14:44:35.886 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.879+0000 7f74e17fa700 1 -- 172.21.15.38:0/113524522 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 133198+0+0 (secure 0 0 0) 0x7f74d400f450 con 0x7f74e40fff80 2023-03-09T14:44:35.886 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.880+0000 7f74e17fa700 1 --2- 172.21.15.38:0/113524522 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f74d009f3c0 0x7f74d00a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:35.887 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.880+0000 7f74e17fa700 1 -- 172.21.15.38:0/113524522 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f74d40ca0f0 con 0x7f74e40fff80 2023-03-09T14:44:35.887 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.880+0000 7f74e3fff700 1 --2- 172.21.15.38:0/113524522 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f74d009f3c0 0x7f74d00a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:35.887 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.880+0000 7f74e3fff700 1 --2- 172.21.15.38:0/113524522 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f74d009f3c0 0x7f74d00a1870 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f74cc005950 tx=0x7f74cc0058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:35.887 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:35.884+0000 7f74e17fa700 1 -- 172.21.15.38:0/113524522 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f74d40942c0 con 0x7f74e40fff80 2023-03-09T14:44:36.087 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:36.084+0000 7f74eaa50700 1 -- 172.21.15.38:0/113524522 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f74e402cef0 con 0x7f74e40fff80 2023-03-09T14:44:36.088 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:36.085+0000 7f74e17fa700 1 -- 172.21.15.38:0/113524522 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+23973 (secure 0 0 0) 0x7f74d40978e0 con 0x7f74e40fff80 2023-03-09T14:44:36.089 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:44:36.089 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":87,"fsid":"c6733fe2-be87-11ed-9af2-001a4aab830c","created":"2023-03-09T14:37:14.947981+0000","modified":"2023-03-09T14:44:23.900228+0000","last_up_change":"2023-03-09T14:43:37.198407+0000","last_in_change":"2023-03-09T14:43:19.550566+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-09T14:40:23.196665+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-03-09T14:44:17.484374+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-03-09T14:44:18.457833+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-03-09T14:44:21.895589+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"51477039-15ee-4707-a34d-4ecfa18bebde","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6803","nonce":1508072290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6805","nonce":1508072290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6809","nonce":1508072290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6807","nonce":1508072290}]},"public_addr":"172.21.15.38:6803/1508072290","cluster_addr":"172.21.15.38:6805/1508072290","heartbeat_back_addr":"172.21.15.38:6809/1508072290","heartbeat_front_addr":"172.21.15.38:6807/1508072290","state":["exists","up"]},{"osd":1,"uuid":"71f8a718-b0a9-4f39-bc54-267aa42a2578","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6801","nonce":3929268773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6803","nonce":3929268773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6807","nonce":3929268773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6805","nonce":3929268773}]},"public_addr":"172.21.15.122:6801/3929268773","cluster_addr":"172.21.15.122:6803/3929268773","heartbeat_back_addr":"172.21.15.122:6807/3929268773","heartbeat_front_addr":"172.21.15.122:6805/3929268773","state":["exists","up"]},{"osd":2,"uuid":"961a2cbe-140e-437f-b131-1fcac573b19c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6801","nonce":1310630803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6803","nonce":1310630803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6807","nonce":1310630803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6805","nonce":1310630803}]},"public_addr":"172.21.15.181:6801/1310630803","cluster_addr":"172.21.15.181:6803/1310630803","heartbeat_back_addr":"172.21.15.181:6807/1310630803","heartbeat_front_addr":"172.21.15.181:6805/1310630803","state":["exists","up"]},{"osd":3,"uuid":"969df8fa-2d19-42c7-9f2a-5c5fac15d4a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6811","nonce":2719814100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6813","nonce":2719814100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6817","nonce":2719814100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6815","nonce":2719814100}]},"public_addr":"172.21.15.38:6811/2719814100","cluster_addr":"172.21.15.38:6813/2719814100","heartbeat_back_addr":"172.21.15.38:6817/2719814100","heartbeat_front_addr":"172.21.15.38:6815/2719814100","state":["exists","up"]},{"osd":4,"uuid":"44a6dce3-e06e-4cbd-9404-62a86d01ab6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6809","nonce":2480764994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6811","nonce":2480764994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6815","nonce":2480764994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6813","nonce":2480764994}]},"public_addr":"172.21.15.122:6809/2480764994","cluster_addr":"172.21.15.122:6811/2480764994","heartbeat_back_addr":"172.21.15.122:6815/2480764994","heartbeat_front_addr":"172.21.15.122:6813/2480764994","state":["exists","up"]},{"osd":5,"uuid":"49749574-c00c-4e3a-80e6-e3686df58d5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6809","nonce":3141346246}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6811","nonce":3141346246}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6815","nonce":3141346246}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6813","nonce":3141346246}]},"public_addr":"172.21.15.181:6809/3141346246","cluster_addr":"172.21.15.181:6811/3141346246","heartbeat_back_addr":"172.21.15.181:6815/3141346246","heartbeat_front_addr":"172.21.15.181:6813/3141346246","state":["exists","up"]},{"osd":6,"uuid":"61c0437e-3f51-45a9-8a61-e98eea0093f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6819","nonce":2122788957}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6821","nonce":2122788957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6825","nonce":2122788957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6823","nonce":2122788957}]},"public_addr":"172.21.15.38:6819/2122788957","cluster_addr":"172.21.15.38:6821/2122788957","heartbeat_back_addr":"172.21.15.38:6825/2122788957","heartbeat_front_addr":"172.21.15.38:6823/2122788957","state":["exists","up"]},{"osd":7,"uuid":"231fecdd-939f-4c7b-83a1-c4a1cffd7139","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6817","nonce":656874139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6819","nonce":656874139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6823","nonce":656874139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6821","nonce":656874139}]},"public_addr":"172.21.15.122:6817/656874139","cluster_addr":"172.21.15.122:6819/656874139","heartbeat_back_addr":"172.21.15.122:6823/656874139","heartbeat_front_addr":"172.21.15.122:6821/656874139","state":["exists","up"]},{"osd":8,"uuid":"297e6d6d-85fd-43ce-9028-5e9dbe55c36e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6817","nonce":755000703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6819","nonce":755000703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6823","nonce":755000703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6821","nonce":755000703}]},"public_addr":"172.21.15.181:6817/755000703","cluster_addr":"172.21.15.181:6819/755000703","heartbeat_back_addr":"172.21.15.181:6823/755000703","heartbeat_front_addr":"172.21.15.181:6821/755000703","state":["exists","up"]},{"osd":9,"uuid":"94cc0355-5a1a-4cfc-bd0c-562b6b41127e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6827","nonce":2624555969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6829","nonce":2624555969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6833","nonce":2624555969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6831","nonce":2624555969}]},"public_addr":"172.21.15.38:6827/2624555969","cluster_addr":"172.21.15.38:6829/2624555969","heartbeat_back_addr":"172.21.15.38:6833/2624555969","heartbeat_front_addr":"172.21.15.38:6831/2624555969","state":["exists","up"]},{"osd":10,"uuid":"f4f2e042-8d92-46bf-93b8-8b0f04795c2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6825","nonce":1427785280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6827","nonce":1427785280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6831","nonce":1427785280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6829","nonce":1427785280}]},"public_addr":"172.21.15.122:6825/1427785280","cluster_addr":"172.21.15.122:6827/1427785280","heartbeat_back_addr":"172.21.15.122:6831/1427785280","heartbeat_front_addr":"172.21.15.122:6829/1427785280","state":["exists","up"]},{"osd":11,"uuid":"07444936-eae5-4097-a799-60294cafe80b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6825","nonce":3183488640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6827","nonce":3183488640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6831","nonce":3183488640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6829","nonce":3183488640}]},"public_addr":"172.21.15.181:6825/3183488640","cluster_addr":"172.21.15.181:6827/3183488640","heartbeat_back_addr":"172.21.15.181:6831/3183488640","heartbeat_front_addr":"172.21.15.181:6829/3183488640","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:39:37.657556+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:39:58.960074+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:40:20.326952+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:40:42.244387+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:41:03.946973+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:41:25.097985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:41:46.877916+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:42:08.413827+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:42:29.822009+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:42:51.776684+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:43:13.937249+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:43:35.569199+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/4272288735":"2023-03-10T14:38:14.612767+0000","172.21.15.38:6801/3381625669":"2023-03-10T14:38:14.612767+0000","172.21.15.38:6800/3381625669":"2023-03-10T14:38:14.612767+0000","172.21.15.38:6800/3060474342":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/4253918615":"2023-03-10T14:38:14.612767+0000","172.21.15.38:0/3496083181":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/245602368":"2023-03-10T14:37:47.517775+0000","172.21.15.38:6801/3060474342":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/3622589182":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/1430302251":"2023-03-10T14:38:14.612767+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-09T14:44:36.091 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:36.088+0000 7f74eaa50700 1 -- 172.21.15.38:0/113524522 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f74d009f3c0 msgr2=0x7f74d00a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:36.091 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:36.088+0000 7f74eaa50700 1 --2- 172.21.15.38:0/113524522 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f74d009f3c0 0x7f74d00a1870 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f74cc005950 tx=0x7f74cc0058e0 comp rx=0 tx=0).stop 2023-03-09T14:44:36.092 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:36.088+0000 7f74eaa50700 1 -- 172.21.15.38:0/113524522 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74e40fff80 msgr2=0x7f74e419a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:36.092 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:36.088+0000 7f74eaa50700 1 --2- 172.21.15.38:0/113524522 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74e40fff80 0x7f74e419a9a0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f74d400c9b0 tx=0x7f74d400ccc0 comp rx=0 tx=0).stop 2023-03-09T14:44:36.092 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:36.088+0000 7f74eaa50700 1 -- 172.21.15.38:0/113524522 shutdown_connections 2023-03-09T14:44:36.092 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:36.088+0000 7f74eaa50700 1 --2- 172.21.15.38:0/113524522 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f74d009f3c0 0x7f74d00a1870 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:36.092 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:36.088+0000 7f74eaa50700 1 --2- 172.21.15.38:0/113524522 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f74e4100910 0x7f74e419cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:36.093 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:36.088+0000 7f74eaa50700 1 --2- 172.21.15.38:0/113524522 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f74e40fed90 0x7f74e4069650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:36.093 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:36.088+0000 7f74eaa50700 1 --2- 172.21.15.38:0/113524522 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74e40fff80 0x7f74e419a9a0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:36.093 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:36.088+0000 7f74eaa50700 1 -- 172.21.15.38:0/113524522 >> 172.21.15.38:0/113524522 conn(0x7f74e40fa370 msgr2=0x7f74e4103370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:36.093 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:36.088+0000 7f74eaa50700 1 -- 172.21.15.38:0/113524522 shutdown_connections 2023-03-09T14:44:36.094 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:36.088+0000 7f74eaa50700 1 -- 172.21.15.38:0/113524522 wait complete. 2023-03-09T14:44:36.102 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:35 smithi038 ceph-mon[92422]: pgmap v267: 129 pgs: 129 active+clean; 579 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 94 B/s wr, 0 op/s 2023-03-09T14:44:36.103 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:35 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/3117465526' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-03-09T14:44:36.103 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:35 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/583663196' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-03-09T14:44:36.104 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-03-09T14:44:36.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:35 smithi122 ceph-mon[93869]: pgmap v267: 129 pgs: 129 active+clean; 579 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 94 B/s wr, 0 op/s 2023-03-09T14:44:36.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:35 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/3117465526' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-03-09T14:44:36.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:35 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/583663196' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-03-09T14:44:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:35 smithi181 ceph-mon[94189]: pgmap v267: 129 pgs: 129 active+clean; 579 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 94 B/s wr, 0 op/s 2023-03-09T14:44:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:35 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/3117465526' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-03-09T14:44:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:35 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/583663196' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-03-09T14:44:36.233 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:36.229+0000 7f3e4d8ef700 1 -- 172.21.15.38:0/2051230917 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3e48100910 msgr2=0x7f3e481071d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:36.233 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:36.229+0000 7f3e4d8ef700 1 --2- 172.21.15.38:0/2051230917 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3e48100910 0x7f3e481071d0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f3e4000b0a0 tx=0x7f3e4000b3b0 comp rx=0 tx=0).stop 2023-03-09T14:44:36.234 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:36.229+0000 7f3e4d8ef700 1 -- 172.21.15.38:0/2051230917 shutdown_connections 2023-03-09T14:44:36.234 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:36.229+0000 7f3e4d8ef700 1 --2- 172.21.15.38:0/2051230917 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3e48100910 0x7f3e481071d0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:36.234 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:36.229+0000 7f3e4d8ef700 1 --2- 172.21.15.38:0/2051230917 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3e480fed90 0x7f3e480ff1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:36.234 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:36.229+0000 7f3e4d8ef700 1 --2- 172.21.15.38:0/2051230917 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3e480fff80 0x7f3e481003d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:36.234 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:36.229+0000 7f3e4d8ef700 1 -- 172.21.15.38:0/2051230917 >> 172.21.15.38:0/2051230917 conn(0x7f3e480fa3b0 msgr2=0x7f3e480fc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:36.234 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:36.229+0000 7f3e4d8ef700 1 -- 172.21.15.38:0/2051230917 shutdown_connections 2023-03-09T14:44:36.234 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:36.229+0000 7f3e4d8ef700 1 -- 172.21.15.38:0/2051230917 wait complete. 2023-03-09T14:44:36.235 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:36.230+0000 7f3e4d8ef700 1 Processor -- start 2023-03-09T14:44:36.235 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:36.230+0000 7f3e4d8ef700 1 -- start start 2023-03-09T14:44:36.236 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:36.230+0000 7f3e4d8ef700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3e480fed90 0x7f3e48194bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:36.237 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:36.230+0000 7f3e4d8ef700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3e480fff80 0x7f3e48195130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:36.237 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:36.230+0000 7f3e4d8ef700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3e48100910 0x7f3e4819c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:36.237 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:36.230+0000 7f3e4d8ef700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4819c770 con 0x7f3e480fff80 2023-03-09T14:44:36.237 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:36.230+0000 7f3e4d8ef700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4819c8b0 con 0x7f3e480fed90 2023-03-09T14:44:36.237 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:36.230+0000 7f3e4d8ef700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4819ca40 con 0x7f3e48100910 2023-03-09T14:44:36.237 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:36.230+0000 7f3e467fc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3e480fff80 0x7f3e48195130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:36.238 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:36.230+0000 7f3e467fc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3e480fff80 0x7f3e48195130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37080/0 (socket says 172.21.15.38:37080) 2023-03-09T14:44:36.238 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:36.231+0000 7f3e467fc700 1 -- 172.21.15.38:0/3536822267 learned_addr learned my addr 172.21.15.38:0/3536822267 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:44:36.238 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:36.231+0000 7f3e477fe700 1 --2- 172.21.15.38:0/3536822267 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3e48100910 0x7f3e4819c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:36.238 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:36.231+0000 7f3e46ffd700 1 --2- 172.21.15.38:0/3536822267 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3e480fed90 0x7f3e48194bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:36.238 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:36.231+0000 7f3e467fc700 1 -- 172.21.15.38:0/3536822267 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3e48100910 msgr2=0x7f3e4819c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:36.238 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:36.231+0000 7f3e467fc700 1 --2- 172.21.15.38:0/3536822267 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3e48100910 0x7f3e4819c1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:36.238 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:36.231+0000 7f3e467fc700 1 -- 172.21.15.38:0/3536822267 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3e480fed90 msgr2=0x7f3e48194bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:36.239 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:36.231+0000 7f3e467fc700 1 --2- 172.21.15.38:0/3536822267 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3e480fed90 0x7f3e48194bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:36.239 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:36.231+0000 7f3e467fc700 1 -- 172.21.15.38:0/3536822267 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e40009d00 con 0x7f3e480fff80 2023-03-09T14:44:36.239 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:36.231+0000 7f3e46ffd700 1 --2- 172.21.15.38:0/3536822267 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3e480fed90 0x7f3e48194bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:44:36.239 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:36.231+0000 7f3e467fc700 1 --2- 172.21.15.38:0/3536822267 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3e480fff80 0x7f3e48195130 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f3e3000e9a0 tx=0x7f3e3000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:36.239 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:36.231+0000 7f3e4c8ed700 1 -- 172.21.15.38:0/3536822267 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e30009790 con 0x7f3e480fff80 2023-03-09T14:44:36.239 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:36.231+0000 7f3e4c8ed700 1 -- 172.21.15.38:0/3536822267 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3e3000cbd0 con 0x7f3e480fff80 2023-03-09T14:44:36.239 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:36.231+0000 7f3e4d8ef700 1 -- 172.21.15.38:0/3536822267 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e4810a700 con 0x7f3e480fff80 2023-03-09T14:44:36.239 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:36.231+0000 7f3e4c8ed700 1 -- 172.21.15.38:0/3536822267 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e30018680 con 0x7f3e480fff80 2023-03-09T14:44:36.240 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:36.231+0000 7f3e4d8ef700 1 -- 172.21.15.38:0/3536822267 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e4810a570 con 0x7f3e480fff80 2023-03-09T14:44:36.240 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:36.232+0000 7f3e4d8ef700 1 -- 172.21.15.38:0/3536822267 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e4819d0a0 con 0x7f3e480fff80 2023-03-09T14:44:36.240 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:36.233+0000 7f3e4c8ed700 1 -- 172.21.15.38:0/3536822267 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 133198+0+0 (secure 0 0 0) 0x7f3e3000c040 con 0x7f3e480fff80 2023-03-09T14:44:36.240 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:36.234+0000 7f3e4c8ed700 1 --2- 172.21.15.38:0/3536822267 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f3e3409f3c0 0x7f3e340a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:36.241 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:36.234+0000 7f3e4c8ed700 1 -- 172.21.15.38:0/3536822267 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f3e30014070 con 0x7f3e480fff80 2023-03-09T14:44:36.241 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:36.234+0000 7f3e46ffd700 1 --2- 172.21.15.38:0/3536822267 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f3e3409f3c0 0x7f3e340a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:36.241 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:36.234+0000 7f3e46ffd700 1 --2- 172.21.15.38:0/3536822267 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f3e3409f3c0 0x7f3e340a1870 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f3e480ffde0 tx=0x7f3e38009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:36.241 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:36.238+0000 7f3e4c8ed700 1 -- 172.21.15.38:0/3536822267 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f3e300515b0 con 0x7f3e480fff80 2023-03-09T14:44:36.487 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:36.484+0000 7f3e4d8ef700 1 -- 172.21.15.38:0/3536822267 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f3e48105c40 con 0x7f3e480fff80 2023-03-09T14:44:36.862 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:36.859+0000 7f3e4c8ed700 1 -- 172.21.15.38:0/3536822267 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) v1 ==== 118+0+0 (secure 0 0 0) 0x7f3e300943f0 con 0x7f3e480fff80 2023-03-09T14:44:36.865 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:36.861+0000 7f3e4d8ef700 1 -- 172.21.15.38:0/3536822267 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f3e3409f3c0 msgr2=0x7f3e340a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:36.865 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:36.861+0000 7f3e4d8ef700 1 --2- 172.21.15.38:0/3536822267 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f3e3409f3c0 0x7f3e340a1870 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f3e480ffde0 tx=0x7f3e38009260 comp rx=0 tx=0).stop 2023-03-09T14:44:36.865 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:36.861+0000 7f3e4d8ef700 1 -- 172.21.15.38:0/3536822267 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3e480fff80 msgr2=0x7f3e48195130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:36.865 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:36.861+0000 7f3e4d8ef700 1 --2- 172.21.15.38:0/3536822267 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3e480fff80 0x7f3e48195130 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f3e3000e9a0 tx=0x7f3e3000ecb0 comp rx=0 tx=0).stop 2023-03-09T14:44:36.865 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:36.861+0000 7f3e4d8ef700 1 -- 172.21.15.38:0/3536822267 shutdown_connections 2023-03-09T14:44:36.866 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:36.861+0000 7f3e4d8ef700 1 --2- 172.21.15.38:0/3536822267 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f3e3409f3c0 0x7f3e340a1870 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:36.866 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:36.861+0000 7f3e4d8ef700 1 --2- 172.21.15.38:0/3536822267 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3e48100910 0x7f3e4819c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:36.866 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:36.861+0000 7f3e4d8ef700 1 --2- 172.21.15.38:0/3536822267 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3e480fed90 0x7f3e48194bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:36.866 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:36.861+0000 7f3e4d8ef700 1 --2- 172.21.15.38:0/3536822267 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3e480fff80 0x7f3e48195130 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:36.866 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:36.861+0000 7f3e4d8ef700 1 -- 172.21.15.38:0/3536822267 >> 172.21.15.38:0/3536822267 conn(0x7f3e480fa3b0 msgr2=0x7f3e480fc5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:36.866 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:36.862+0000 7f3e4d8ef700 1 -- 172.21.15.38:0/3536822267 shutdown_connections 2023-03-09T14:44:36.866 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:36.862+0000 7f3e4d8ef700 1 -- 172.21.15.38:0/3536822267 wait complete. 2023-03-09T14:44:36.879 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2}} 2023-03-09T14:44:36.879 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2023-03-09T14:44:37.011 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.008+0000 7f0375168700 1 -- 172.21.15.38:0/222776289 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f03700fff80 msgr2=0x7f03701003d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:37.012 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.008+0000 7f0375168700 1 --2- 172.21.15.38:0/222776289 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f03700fff80 0x7f03701003d0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f0358009990 tx=0x7f0358009ca0 comp rx=0 tx=0).stop 2023-03-09T14:44:37.012 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.008+0000 7f0375168700 1 -- 172.21.15.38:0/222776289 shutdown_connections 2023-03-09T14:44:37.012 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.008+0000 7f0375168700 1 --2- 172.21.15.38:0/222776289 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0370100910 0x7f03701071d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:37.012 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.008+0000 7f0375168700 1 --2- 172.21.15.38:0/222776289 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f03700fed90 0x7f03700ff1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:37.013 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.008+0000 7f0375168700 1 --2- 172.21.15.38:0/222776289 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f03700fff80 0x7f03701003d0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:37.013 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.008+0000 7f0375168700 1 -- 172.21.15.38:0/222776289 >> 172.21.15.38:0/222776289 conn(0x7f03700fa3b0 msgr2=0x7f03700fc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:37.013 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.008+0000 7f0375168700 1 -- 172.21.15.38:0/222776289 shutdown_connections 2023-03-09T14:44:37.013 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.008+0000 7f0375168700 1 -- 172.21.15.38:0/222776289 wait complete. 2023-03-09T14:44:37.017 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.009+0000 7f0375168700 1 Processor -- start 2023-03-09T14:44:37.017 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.009+0000 7f0375168700 1 -- start start 2023-03-09T14:44:37.018 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.009+0000 7f0375168700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f03700fed90 0x7f03700685b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:37.018 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.009+0000 7f0375168700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f03700fff80 0x7f037019a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:37.018 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.009+0000 7f0375168700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0370100910 0x7f037019cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:37.018 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.009+0000 7f0375168700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f037019d740 con 0x7f03700fff80 2023-03-09T14:44:37.019 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.009+0000 7f0375168700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f037019d8b0 con 0x7f03700fed90 2023-03-09T14:44:37.019 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.009+0000 7f0375168700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f037019da70 con 0x7f0370100910 2023-03-09T14:44:37.019 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.009+0000 7f036e59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f03700fff80 0x7f037019a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:37.019 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.009+0000 7f036e59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f03700fff80 0x7f037019a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37094/0 (socket says 172.21.15.38:37094) 2023-03-09T14:44:37.020 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.009+0000 7f036e59c700 1 -- 172.21.15.38:0/1871861670 learned_addr learned my addr 172.21.15.38:0/1871861670 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:44:37.020 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.009+0000 7f036ed9d700 1 --2- 172.21.15.38:0/1871861670 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f03700fed90 0x7f03700685b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:37.020 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.009+0000 7f036f59e700 1 --2- 172.21.15.38:0/1871861670 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0370100910 0x7f037019cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:37.020 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.009+0000 7f036e59c700 1 -- 172.21.15.38:0/1871861670 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0370100910 msgr2=0x7f037019cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:37.020 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.009+0000 7f036e59c700 1 --2- 172.21.15.38:0/1871861670 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0370100910 0x7f037019cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:37.020 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.009+0000 7f036e59c700 1 -- 172.21.15.38:0/1871861670 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f03700fed90 msgr2=0x7f03700685b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:37.020 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.009+0000 7f036e59c700 1 --2- 172.21.15.38:0/1871861670 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f03700fed90 0x7f03700685b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:37.021 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.009+0000 7f036e59c700 1 -- 172.21.15.38:0/1871861670 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03580095f0 con 0x7f03700fff80 2023-03-09T14:44:37.021 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.010+0000 7f036e59c700 1 --2- 172.21.15.38:0/1871861670 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f03700fff80 0x7f037019a960 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f0358004760 tx=0x7f0358004790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:37.021 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.010+0000 7f0367fff700 1 -- 172.21.15.38:0/1871861670 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f035801d070 con 0x7f03700fff80 2023-03-09T14:44:37.021 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.010+0000 7f0375168700 1 -- 172.21.15.38:0/1871861670 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f037010a6f0 con 0x7f03700fff80 2023-03-09T14:44:37.021 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.010+0000 7f0367fff700 1 -- 172.21.15.38:0/1871861670 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0358004cf0 con 0x7f03700fff80 2023-03-09T14:44:37.021 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.010+0000 7f0367fff700 1 -- 172.21.15.38:0/1871861670 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0358003d00 con 0x7f03700fff80 2023-03-09T14:44:37.021 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.010+0000 7f0375168700 1 -- 172.21.15.38:0/1871861670 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f037010a560 con 0x7f03700fff80 2023-03-09T14:44:37.022 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.011+0000 7f0375168700 1 -- 172.21.15.38:0/1871861670 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0370105cc0 con 0x7f03700fff80 2023-03-09T14:44:37.022 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.012+0000 7f0367fff700 1 -- 172.21.15.38:0/1871861670 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 133198+0+0 (secure 0 0 0) 0x7f035800c040 con 0x7f03700fff80 2023-03-09T14:44:37.022 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.012+0000 7f0367fff700 1 --2- 172.21.15.38:0/1871861670 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f035c09f3c0 0x7f035c0a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:37.022 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.013+0000 7f0367fff700 1 -- 172.21.15.38:0/1871861670 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f03580caf70 con 0x7f03700fff80 2023-03-09T14:44:37.023 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.013+0000 7f036ed9d700 1 --2- 172.21.15.38:0/1871861670 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f035c09f3c0 0x7f035c0a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:37.023 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.013+0000 7f036ed9d700 1 --2- 172.21.15.38:0/1871861670 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f035c09f3c0 0x7f035c0a1870 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f03700ffde0 tx=0x7f036000b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:37.023 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.017+0000 7f0367fff700 1 -- 172.21.15.38:0/1871861670 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f035805b5b0 con 0x7f03700fff80 2023-03-09T14:44:37.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:36 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/113524522' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-03-09T14:44:37.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:36 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/3536822267' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-03-09T14:44:37.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:36 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/113524522' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-03-09T14:44:37.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:36 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/3536822267' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-03-09T14:44:37.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:36 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/113524522' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-03-09T14:44:37.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:36 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/3536822267' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-03-09T14:44:37.265 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.261+0000 7f0375168700 1 -- 172.21.15.38:0/1871861670 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) v1 -- 0x7f0370061830 con 0x7f03700fff80 2023-03-09T14:44:37.495 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.492+0000 7f0367fff700 1 -- 172.21.15.38:0/1871861670 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v19) v1 ==== 105+0+0 (secure 0 0 0) 0x7f035809ce60 con 0x7f03700fff80 2023-03-09T14:44:37.498 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.494+0000 7f0375168700 1 -- 172.21.15.38:0/1871861670 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f035c09f3c0 msgr2=0x7f035c0a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:37.498 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.494+0000 7f0375168700 1 --2- 172.21.15.38:0/1871861670 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f035c09f3c0 0x7f035c0a1870 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f03700ffde0 tx=0x7f036000b410 comp rx=0 tx=0).stop 2023-03-09T14:44:37.498 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.495+0000 7f0375168700 1 -- 172.21.15.38:0/1871861670 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f03700fff80 msgr2=0x7f037019a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:37.499 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.495+0000 7f0375168700 1 --2- 172.21.15.38:0/1871861670 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f03700fff80 0x7f037019a960 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f0358004760 tx=0x7f0358004790 comp rx=0 tx=0).stop 2023-03-09T14:44:37.499 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.495+0000 7f0375168700 1 -- 172.21.15.38:0/1871861670 shutdown_connections 2023-03-09T14:44:37.499 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.495+0000 7f0375168700 1 --2- 172.21.15.38:0/1871861670 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f035c09f3c0 0x7f035c0a1870 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:37.499 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.495+0000 7f0375168700 1 --2- 172.21.15.38:0/1871861670 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0370100910 0x7f037019cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:37.499 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.495+0000 7f0375168700 1 --2- 172.21.15.38:0/1871861670 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f03700fed90 0x7f03700685b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:37.499 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.495+0000 7f0375168700 1 --2- 172.21.15.38:0/1871861670 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f03700fff80 0x7f037019a960 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:37.499 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.495+0000 7f0375168700 1 -- 172.21.15.38:0/1871861670 >> 172.21.15.38:0/1871861670 conn(0x7f03700fa3b0 msgr2=0x7f03700fc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:37.500 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.495+0000 7f0375168700 1 -- 172.21.15.38:0/1871861670 shutdown_connections 2023-03-09T14:44:37.500 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.495+0000 7f0375168700 1 -- 172.21.15.38:0/1871861670 wait complete. 2023-03-09T14:44:37.514 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2023-03-09T14:44:37.647 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.641+0000 7f1ee2159700 1 -- 172.21.15.38:0/1376356506 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1edc0fed90 msgr2=0x7f1edc0ff1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:37.648 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.641+0000 7f1ee2159700 1 --2- 172.21.15.38:0/1376356506 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1edc0fed90 0x7f1edc0ff1a0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f1ed0009990 tx=0x7f1ed0009ca0 comp rx=0 tx=0).stop 2023-03-09T14:44:37.648 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.641+0000 7f1ee2159700 1 -- 172.21.15.38:0/1376356506 shutdown_connections 2023-03-09T14:44:37.648 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.641+0000 7f1ee2159700 1 --2- 172.21.15.38:0/1376356506 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1edc0fed90 0x7f1edc0ff1a0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:37.648 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.641+0000 7f1ee2159700 1 --2- 172.21.15.38:0/1376356506 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1edc100910 0x7f1edc1071d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:37.648 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.641+0000 7f1ee2159700 1 --2- 172.21.15.38:0/1376356506 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1edc0fff80 0x7f1edc1003d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:37.648 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.641+0000 7f1ee2159700 1 -- 172.21.15.38:0/1376356506 >> 172.21.15.38:0/1376356506 conn(0x7f1edc0fa3b0 msgr2=0x7f1edc0fc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:37.649 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.641+0000 7f1ee2159700 1 -- 172.21.15.38:0/1376356506 shutdown_connections 2023-03-09T14:44:37.649 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.641+0000 7f1ee2159700 1 -- 172.21.15.38:0/1376356506 wait complete. 2023-03-09T14:44:37.649 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.642+0000 7f1ee2159700 1 Processor -- start 2023-03-09T14:44:37.649 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.642+0000 7f1ee2159700 1 -- start start 2023-03-09T14:44:37.649 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.642+0000 7f1ee2159700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1edc0fed90 0x7f1edc195c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:37.650 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.642+0000 7f1ee2159700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1edc0fff80 0x7f1edc1961c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:37.650 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.642+0000 7f1ee2159700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1edc100910 0x7f1edc19d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:37.650 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.642+0000 7f1ee2159700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1edc19d7f0 con 0x7f1edc100910 2023-03-09T14:44:37.650 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.642+0000 7f1ee2159700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1edc19d960 con 0x7f1edc0fff80 2023-03-09T14:44:37.650 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.642+0000 7f1ee2159700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1edc19db20 con 0x7f1edc0fed90 2023-03-09T14:44:37.650 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.642+0000 7f1edbfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1edc100910 0x7f1edc19d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:37.650 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.642+0000 7f1edbfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1edc100910 0x7f1edc19d220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37110/0 (socket says 172.21.15.38:37110) 2023-03-09T14:44:37.650 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.642+0000 7f1edbfff700 1 -- 172.21.15.38:0/1712798596 learned_addr learned my addr 172.21.15.38:0/1712798596 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:44:37.651 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.642+0000 7f1edb7fe700 1 --2- 172.21.15.38:0/1712798596 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1edc0fed90 0x7f1edc195c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:37.651 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.642+0000 7f1edaffd700 1 --2- 172.21.15.38:0/1712798596 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1edc0fff80 0x7f1edc1961c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:37.651 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.643+0000 7f1edbfff700 1 -- 172.21.15.38:0/1712798596 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1edc0fed90 msgr2=0x7f1edc195c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:37.651 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.643+0000 7f1edbfff700 1 --2- 172.21.15.38:0/1712798596 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1edc0fed90 0x7f1edc195c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:37.651 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.643+0000 7f1edbfff700 1 -- 172.21.15.38:0/1712798596 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1edc0fff80 msgr2=0x7f1edc1961c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:37.651 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.643+0000 7f1edbfff700 1 --2- 172.21.15.38:0/1712798596 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1edc0fff80 0x7f1edc1961c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:37.651 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.643+0000 7f1edbfff700 1 -- 172.21.15.38:0/1712798596 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ed00095f0 con 0x7f1edc100910 2023-03-09T14:44:37.652 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.643+0000 7f1edbfff700 1 --2- 172.21.15.38:0/1712798596 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1edc100910 0x7f1edc19d220 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f1ed400ada0 tx=0x7f1ed400c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:37.652 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.643+0000 7f1ed8ff9700 1 -- 172.21.15.38:0/1712798596 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ed400f070 con 0x7f1edc100910 2023-03-09T14:44:37.652 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.643+0000 7f1ed8ff9700 1 -- 172.21.15.38:0/1712798596 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1ed4007b30 con 0x7f1edc100910 2023-03-09T14:44:37.652 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.643+0000 7f1ee2159700 1 -- 172.21.15.38:0/1712798596 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1edc10a6f0 con 0x7f1edc100910 2023-03-09T14:44:37.652 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.643+0000 7f1ed8ff9700 1 -- 172.21.15.38:0/1712798596 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ed4005280 con 0x7f1edc100910 2023-03-09T14:44:37.652 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.643+0000 7f1ee2159700 1 -- 172.21.15.38:0/1712798596 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1edc10a560 con 0x7f1edc100910 2023-03-09T14:44:37.652 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.644+0000 7f1ee2159700 1 -- 172.21.15.38:0/1712798596 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1edc19e100 con 0x7f1edc100910 2023-03-09T14:44:37.653 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.645+0000 7f1ed8ff9700 1 -- 172.21.15.38:0/1712798596 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 133198+0+0 (secure 0 0 0) 0x7f1ed4019020 con 0x7f1edc100910 2023-03-09T14:44:37.653 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.646+0000 7f1ed8ff9700 1 --2- 172.21.15.38:0/1712798596 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f1ec409f3c0 0x7f1ec40a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:37.653 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.646+0000 7f1ed8ff9700 1 -- 172.21.15.38:0/1712798596 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f1ed40c9a40 con 0x7f1edc100910 2023-03-09T14:44:37.654 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.646+0000 7f1edb7fe700 1 --2- 172.21.15.38:0/1712798596 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f1ec409f3c0 0x7f1ec40a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:37.654 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.646+0000 7f1edb7fe700 1 --2- 172.21.15.38:0/1712798596 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f1ec409f3c0 0x7f1ec40a1870 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1ed000b5c0 tx=0x7f1ed0005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:37.654 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.649+0000 7f1ed8ff9700 1 -- 172.21.15.38:0/1712798596 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f1ed4093c40 con 0x7f1edc100910 2023-03-09T14:44:37.900 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:37.897+0000 7f1ee2159700 1 -- 172.21.15.38:0/1712798596 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7f1edc19e7a0 con 0x7f1edc100910 2023-03-09T14:44:38.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:37 smithi122 ceph-mon[93869]: pgmap v268: 129 pgs: 129 active+clean; 581 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2023-03-09T14:44:38.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:37 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/3536822267' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2023-03-09T14:44:38.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:37 smithi122 ceph-mon[93869]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-03-09T14:44:38.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:37 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/1871861670' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2023-03-09T14:44:38.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:37 smithi122 ceph-mon[93869]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2023-03-09T14:44:38.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:37 smithi122 ceph-mon[93869]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-03-09T14:44:38.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:37 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/1871861670' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2023-03-09T14:44:38.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:37 smithi122 ceph-mon[93869]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2023-03-09T14:44:38.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:37 smithi122 ceph-mon[93869]: daemon mds.h is now active in filesystem cephfs as rank 1 2023-03-09T14:44:38.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:37 smithi181 ceph-mon[94189]: pgmap v268: 129 pgs: 129 active+clean; 581 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2023-03-09T14:44:38.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:37 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/3536822267' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2023-03-09T14:44:38.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:37 smithi181 ceph-mon[94189]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-03-09T14:44:38.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:37 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/1871861670' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2023-03-09T14:44:38.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:37 smithi181 ceph-mon[94189]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2023-03-09T14:44:38.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:37 smithi181 ceph-mon[94189]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-03-09T14:44:38.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:37 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/1871861670' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2023-03-09T14:44:38.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:37 smithi181 ceph-mon[94189]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2023-03-09T14:44:38.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:37 smithi181 ceph-mon[94189]: daemon mds.h is now active in filesystem cephfs as rank 1 2023-03-09T14:44:38.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:37 smithi038 ceph-mon[92422]: pgmap v268: 129 pgs: 129 active+clean; 581 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2023-03-09T14:44:38.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:37 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/3536822267' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2023-03-09T14:44:38.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:37 smithi038 ceph-mon[92422]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-03-09T14:44:38.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:37 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/1871861670' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2023-03-09T14:44:38.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:37 smithi038 ceph-mon[92422]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2023-03-09T14:44:38.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:37 smithi038 ceph-mon[92422]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-03-09T14:44:38.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:37 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/1871861670' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2023-03-09T14:44:38.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:37 smithi038 ceph-mon[92422]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2023-03-09T14:44:38.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:37 smithi038 ceph-mon[92422]: daemon mds.h is now active in filesystem cephfs as rank 1 2023-03-09T14:44:38.508 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:38.504+0000 7f1ed8ff9700 1 -- 172.21.15.38:0/1712798596 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v20) v1 ==== 121+0+0 (secure 0 0 0) 0x7f1ed4097260 con 0x7f1edc100910 2023-03-09T14:44:38.511 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:38.507+0000 7f1ee2159700 1 -- 172.21.15.38:0/1712798596 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f1ec409f3c0 msgr2=0x7f1ec40a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:38.511 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:38.507+0000 7f1ee2159700 1 --2- 172.21.15.38:0/1712798596 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f1ec409f3c0 0x7f1ec40a1870 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1ed000b5c0 tx=0x7f1ed0005960 comp rx=0 tx=0).stop 2023-03-09T14:44:38.511 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:38.507+0000 7f1ee2159700 1 -- 172.21.15.38:0/1712798596 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1edc100910 msgr2=0x7f1edc19d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:38.511 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:38.507+0000 7f1ee2159700 1 --2- 172.21.15.38:0/1712798596 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1edc100910 0x7f1edc19d220 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f1ed400ada0 tx=0x7f1ed400c330 comp rx=0 tx=0).stop 2023-03-09T14:44:38.511 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:38.507+0000 7f1ee2159700 1 -- 172.21.15.38:0/1712798596 shutdown_connections 2023-03-09T14:44:38.511 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:38.507+0000 7f1ee2159700 1 --2- 172.21.15.38:0/1712798596 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1edc100910 0x7f1edc19d220 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:38.511 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:38.507+0000 7f1ee2159700 1 --2- 172.21.15.38:0/1712798596 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f1ec409f3c0 0x7f1ec40a1870 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:38.512 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:38.507+0000 7f1ee2159700 1 --2- 172.21.15.38:0/1712798596 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1edc0fed90 0x7f1edc195c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:38.512 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:38.507+0000 7f1ee2159700 1 --2- 172.21.15.38:0/1712798596 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1edc0fff80 0x7f1edc1961c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:38.512 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:38.507+0000 7f1ee2159700 1 -- 172.21.15.38:0/1712798596 >> 172.21.15.38:0/1712798596 conn(0x7f1edc0fa3b0 msgr2=0x7f1edc067aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:38.512 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:38.508+0000 7f1ee2159700 1 -- 172.21.15.38:0/1712798596 shutdown_connections 2023-03-09T14:44:38.512 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:38.508+0000 7f1ee2159700 1 -- 172.21.15.38:0/1712798596 wait complete. 2023-03-09T14:44:38.530 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2023-03-09T14:44:38.665 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:38.661+0000 7f6c50b6f700 1 -- 172.21.15.38:0/4211704741 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c4c100ff0 msgr2=0x7f6c4c101460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:38.665 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:38.661+0000 7f6c50b6f700 1 --2- 172.21.15.38:0/4211704741 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c4c100ff0 0x7f6c4c101460 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f6c34009990 tx=0x7f6c34009ca0 comp rx=0 tx=0).stop 2023-03-09T14:44:38.665 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:38.661+0000 7f6c50b6f700 1 -- 172.21.15.38:0/4211704741 shutdown_connections 2023-03-09T14:44:38.665 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:38.661+0000 7f6c50b6f700 1 --2- 172.21.15.38:0/4211704741 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c4c1019a0 0x7f6c4c1082d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:38.666 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:38.661+0000 7f6c50b6f700 1 --2- 172.21.15.38:0/4211704741 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c4c0ffea0 0x7f6c4c1002b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:38.666 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:38.661+0000 7f6c50b6f700 1 --2- 172.21.15.38:0/4211704741 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c4c100ff0 0x7f6c4c101460 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:38.666 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:38.661+0000 7f6c50b6f700 1 -- 172.21.15.38:0/4211704741 >> 172.21.15.38:0/4211704741 conn(0x7f6c4c0fb4c0 msgr2=0x7f6c4c0fd8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:38.666 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:38.661+0000 7f6c50b6f700 1 -- 172.21.15.38:0/4211704741 shutdown_connections 2023-03-09T14:44:38.666 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:38.661+0000 7f6c50b6f700 1 -- 172.21.15.38:0/4211704741 wait complete. 2023-03-09T14:44:38.668 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:38.662+0000 7f6c50b6f700 1 Processor -- start 2023-03-09T14:44:38.668 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:38.662+0000 7f6c50b6f700 1 -- start start 2023-03-09T14:44:38.668 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:38.662+0000 7f6c50b6f700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c4c0ffea0 0x7f6c4c197e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:38.668 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:38.662+0000 7f6c50b6f700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c4c100ff0 0x7f6c4c198390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:38.668 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:38.662+0000 7f6c50b6f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c4c1019a0 0x7f6c4c19f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:38.669 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:38.662+0000 7f6c50b6f700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c4c19f9d0 con 0x7f6c4c1019a0 2023-03-09T14:44:38.669 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:38.662+0000 7f6c50b6f700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c4c19fb10 con 0x7f6c4c0ffea0 2023-03-09T14:44:38.669 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:38.662+0000 7f6c50b6f700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c4c19fca0 con 0x7f6c4c100ff0 2023-03-09T14:44:38.669 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:38.663+0000 7f6c4ad9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c4c1019a0 0x7f6c4c19f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:38.669 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:38.663+0000 7f6c4ad9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c4c1019a0 0x7f6c4c19f400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37124/0 (socket says 172.21.15.38:37124) 2023-03-09T14:44:38.669 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:38.663+0000 7f6c4ad9d700 1 -- 172.21.15.38:0/1602860555 learned_addr learned my addr 172.21.15.38:0/1602860555 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:44:38.669 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:38.663+0000 7f6c49d9b700 1 --2- 172.21.15.38:0/1602860555 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c4c100ff0 0x7f6c4c198390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:38.670 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:38.663+0000 7f6c4a59c700 1 --2- 172.21.15.38:0/1602860555 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c4c0ffea0 0x7f6c4c197e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:38.670 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:38.663+0000 7f6c4ad9d700 1 -- 172.21.15.38:0/1602860555 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c4c100ff0 msgr2=0x7f6c4c198390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:38.670 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:38.663+0000 7f6c4ad9d700 1 --2- 172.21.15.38:0/1602860555 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c4c100ff0 0x7f6c4c198390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:38.670 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:38.663+0000 7f6c4ad9d700 1 -- 172.21.15.38:0/1602860555 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c4c0ffea0 msgr2=0x7f6c4c197e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:38.671 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:38.663+0000 7f6c4ad9d700 1 --2- 172.21.15.38:0/1602860555 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c4c0ffea0 0x7f6c4c197e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:38.671 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:38.663+0000 7f6c4ad9d700 1 -- 172.21.15.38:0/1602860555 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c340095f0 con 0x7f6c4c1019a0 2023-03-09T14:44:38.671 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:38.663+0000 7f6c4ad9d700 1 --2- 172.21.15.38:0/1602860555 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c4c1019a0 0x7f6c4c19f400 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f6c4400ef30 tx=0x7f6c4400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:38.671 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:38.663+0000 7f6c437fe700 1 -- 172.21.15.38:0/1602860555 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c44018070 con 0x7f6c4c1019a0 2023-03-09T14:44:38.671 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:38.663+0000 7f6c437fe700 1 -- 172.21.15.38:0/1602860555 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6c4400f040 con 0x7f6c4c1019a0 2023-03-09T14:44:38.671 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:38.663+0000 7f6c50b6f700 1 -- 172.21.15.38:0/1602860555 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c4c10b800 con 0x7f6c4c1019a0 2023-03-09T14:44:38.671 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:38.663+0000 7f6c437fe700 1 -- 172.21.15.38:0/1602860555 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c4400cdb0 con 0x7f6c4c1019a0 2023-03-09T14:44:38.672 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:38.663+0000 7f6c50b6f700 1 -- 172.21.15.38:0/1602860555 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c4c10b670 con 0x7f6c4c1019a0 2023-03-09T14:44:38.672 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:38.664+0000 7f6c50b6f700 1 -- 172.21.15.38:0/1602860555 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c4c192560 con 0x7f6c4c1019a0 2023-03-09T14:44:38.673 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:38.665+0000 7f6c437fe700 1 -- 172.21.15.38:0/1602860555 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 133198+0+0 (secure 0 0 0) 0x7f6c44003a00 con 0x7f6c4c1019a0 2023-03-09T14:44:38.673 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:38.666+0000 7f6c437fe700 1 --2- 172.21.15.38:0/1602860555 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f6c3809f440 0x7f6c380a18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:38.673 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:38.666+0000 7f6c437fe700 1 -- 172.21.15.38:0/1602860555 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f6c440c9c80 con 0x7f6c4c1019a0 2023-03-09T14:44:38.673 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:38.666+0000 7f6c4a59c700 1 --2- 172.21.15.38:0/1602860555 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f6c3809f440 0x7f6c380a18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:38.673 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:38.666+0000 7f6c4a59c700 1 --2- 172.21.15.38:0/1602860555 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f6c3809f440 0x7f6c380a18f0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6c3c005950 tx=0x7f6c3c0058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:38.674 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:38.670+0000 7f6c437fe700 1 -- 172.21.15.38:0/1602860555 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f6c44093ee0 con 0x7f6c4c1019a0 2023-03-09T14:44:38.921 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:38.917+0000 7f6c50b6f700 1 -- 172.21.15.38:0/1602860555 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7f6c4c061590 con 0x7f6c4c1019a0 2023-03-09T14:44:39.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:38 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/1712798596' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-03-09T14:44:39.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:38 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] up:active 2023-03-09T14:44:39.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:38 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/1712798596' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-03-09T14:44:39.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:38 smithi122 ceph-mon[93869]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2023-03-09T14:44:39.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:38 smithi122 ceph-mon[93869]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-03-09T14:44:39.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:38 smithi122 ceph-mon[93869]: Cluster is now healthy 2023-03-09T14:44:39.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:38 smithi122 ceph-mon[93869]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2023-03-09T14:44:39.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:38 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:creating} 9 up:standby 2023-03-09T14:44:39.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:38 smithi122 ceph-mon[93869]: daemon mds.i is now active in filesystem cephfs as rank 2 2023-03-09T14:44:39.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:38 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/1712798596' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-03-09T14:44:39.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:38 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] up:active 2023-03-09T14:44:39.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:38 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/1712798596' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-03-09T14:44:39.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:38 smithi181 ceph-mon[94189]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2023-03-09T14:44:39.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:38 smithi181 ceph-mon[94189]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-03-09T14:44:39.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:38 smithi181 ceph-mon[94189]: Cluster is now healthy 2023-03-09T14:44:39.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:38 smithi181 ceph-mon[94189]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2023-03-09T14:44:39.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:38 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:creating} 9 up:standby 2023-03-09T14:44:39.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:38 smithi181 ceph-mon[94189]: daemon mds.i is now active in filesystem cephfs as rank 2 2023-03-09T14:44:39.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:38 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/1712798596' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-03-09T14:44:39.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:38 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] up:active 2023-03-09T14:44:39.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:38 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/1712798596' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-03-09T14:44:39.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:38 smithi038 ceph-mon[92422]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2023-03-09T14:44:39.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:38 smithi038 ceph-mon[92422]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-03-09T14:44:39.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:38 smithi038 ceph-mon[92422]: Cluster is now healthy 2023-03-09T14:44:39.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:38 smithi038 ceph-mon[92422]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2023-03-09T14:44:39.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:38 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:creating} 9 up:standby 2023-03-09T14:44:39.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:38 smithi038 ceph-mon[92422]: daemon mds.i is now active in filesystem cephfs as rank 2 2023-03-09T14:44:39.527 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:39.524+0000 7f6c437fe700 1 -- 172.21.15.38:0/1602860555 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v22) v1 ==== 115+0+0 (secure 0 0 0) 0x7f6c44097500 con 0x7f6c4c1019a0 2023-03-09T14:44:39.530 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:39.527+0000 7f6c50b6f700 1 -- 172.21.15.38:0/1602860555 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f6c3809f440 msgr2=0x7f6c380a18f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:39.531 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:39.527+0000 7f6c50b6f700 1 --2- 172.21.15.38:0/1602860555 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f6c3809f440 0x7f6c380a18f0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6c3c005950 tx=0x7f6c3c0058e0 comp rx=0 tx=0).stop 2023-03-09T14:44:39.531 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:39.527+0000 7f6c50b6f700 1 -- 172.21.15.38:0/1602860555 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c4c1019a0 msgr2=0x7f6c4c19f400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:39.531 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:39.527+0000 7f6c50b6f700 1 --2- 172.21.15.38:0/1602860555 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c4c1019a0 0x7f6c4c19f400 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f6c4400ef30 tx=0x7f6c4400c3a0 comp rx=0 tx=0).stop 2023-03-09T14:44:39.531 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:39.527+0000 7f6c50b6f700 1 -- 172.21.15.38:0/1602860555 shutdown_connections 2023-03-09T14:44:39.531 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:39.527+0000 7f6c50b6f700 1 --2- 172.21.15.38:0/1602860555 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c4c1019a0 0x7f6c4c19f400 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:39.531 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:39.527+0000 7f6c50b6f700 1 --2- 172.21.15.38:0/1602860555 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f6c3809f440 0x7f6c380a18f0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:39.531 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:39.527+0000 7f6c50b6f700 1 --2- 172.21.15.38:0/1602860555 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c4c0ffea0 0x7f6c4c197e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:39.532 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:39.527+0000 7f6c50b6f700 1 --2- 172.21.15.38:0/1602860555 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c4c100ff0 0x7f6c4c198390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:39.532 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:39.527+0000 7f6c50b6f700 1 -- 172.21.15.38:0/1602860555 >> 172.21.15.38:0/1602860555 conn(0x7f6c4c0fb4c0 msgr2=0x7f6c4c106970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:39.532 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:39.527+0000 7f6c50b6f700 1 -- 172.21.15.38:0/1602860555 shutdown_connections 2023-03-09T14:44:39.532 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:39.527+0000 7f6c50b6f700 1 -- 172.21.15.38:0/1602860555 wait complete. 2023-03-09T14:44:39.546 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2} subvols for filesystem 'cephfs' 2023-03-09T14:44:39.547 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 '' 2023-03-09T14:44:39.684 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:39.680+0000 7fb85700f700 1 -- 172.21.15.38:0/1603391162 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb8500fff80 msgr2=0x7fb8501003d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:39.684 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:39.680+0000 7fb85700f700 1 --2- 172.21.15.38:0/1603391162 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb8500fff80 0x7fb8501003d0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fb83c009990 tx=0x7fb83c009ca0 comp rx=0 tx=0).stop 2023-03-09T14:44:39.684 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:39.680+0000 7fb85700f700 1 -- 172.21.15.38:0/1603391162 shutdown_connections 2023-03-09T14:44:39.685 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:39.680+0000 7fb85700f700 1 --2- 172.21.15.38:0/1603391162 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb850100910 0x7fb8501071d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:39.685 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:39.680+0000 7fb85700f700 1 --2- 172.21.15.38:0/1603391162 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb8500fed90 0x7fb8500ff1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:39.685 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:39.680+0000 7fb85700f700 1 --2- 172.21.15.38:0/1603391162 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb8500fff80 0x7fb8501003d0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:39.685 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:39.680+0000 7fb85700f700 1 -- 172.21.15.38:0/1603391162 >> 172.21.15.38:0/1603391162 conn(0x7fb8500fa3b0 msgr2=0x7fb8500fc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:39.685 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:39.680+0000 7fb85700f700 1 -- 172.21.15.38:0/1603391162 shutdown_connections 2023-03-09T14:44:39.685 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:39.681+0000 7fb85700f700 1 -- 172.21.15.38:0/1603391162 wait complete. 2023-03-09T14:44:39.687 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:39.681+0000 7fb85700f700 1 Processor -- start 2023-03-09T14:44:39.688 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:39.681+0000 7fb85700f700 1 -- start start 2023-03-09T14:44:39.688 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:39.682+0000 7fb85700f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb8500fed90 0x7fb8500685d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:39.688 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:39.682+0000 7fb85700f700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb8500fff80 0x7fb85019a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:39.688 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:39.682+0000 7fb85700f700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb850100910 0x7fb85019cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:39.688 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:39.682+0000 7fb85700f700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb85019d750 con 0x7fb8500fed90 2023-03-09T14:44:39.688 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:39.682+0000 7fb85700f700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb85019d8c0 con 0x7fb8500fff80 2023-03-09T14:44:39.688 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:39.682+0000 7fb85700f700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb85019da80 con 0x7fb850100910 2023-03-09T14:44:39.689 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:39.682+0000 7fb854dab700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb8500fed90 0x7fb8500685d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:39.689 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:39.682+0000 7fb854dab700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb8500fed90 0x7fb8500685d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37134/0 (socket says 172.21.15.38:37134) 2023-03-09T14:44:39.689 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:39.682+0000 7fb854dab700 1 -- 172.21.15.38:0/1695764994 learned_addr learned my addr 172.21.15.38:0/1695764994 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:44:39.689 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:39.682+0000 7fb84ffff700 1 --2- 172.21.15.38:0/1695764994 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb8500fff80 0x7fb85019a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:39.689 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:39.682+0000 7fb854dab700 1 -- 172.21.15.38:0/1695764994 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb850100910 msgr2=0x7fb85019cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:39.689 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:39.682+0000 7fb854dab700 1 --2- 172.21.15.38:0/1695764994 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb850100910 0x7fb85019cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:39.689 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:39.682+0000 7fb854dab700 1 -- 172.21.15.38:0/1695764994 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb8500fff80 msgr2=0x7fb85019a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:39.690 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:39.682+0000 7fb854dab700 1 --2- 172.21.15.38:0/1695764994 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb8500fff80 0x7fb85019a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:39.690 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:39.682+0000 7fb854dab700 1 -- 172.21.15.38:0/1695764994 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb83c0095f0 con 0x7fb8500fed90 2023-03-09T14:44:39.690 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:39.682+0000 7fb84ffff700 1 --2- 172.21.15.38:0/1695764994 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb8500fff80 0x7fb85019a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-09T14:44:39.690 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:39.682+0000 7fb854dab700 1 --2- 172.21.15.38:0/1695764994 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb8500fed90 0x7fb8500685d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fb84400d900 tx=0x7fb84400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:39.690 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:39.682+0000 7fb84dffb700 1 -- 172.21.15.38:0/1695764994 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb84400b950 con 0x7fb8500fed90 2023-03-09T14:44:39.690 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:39.682+0000 7fb85700f700 1 -- 172.21.15.38:0/1695764994 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb85010a6f0 con 0x7fb8500fed90 2023-03-09T14:44:39.690 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:39.682+0000 7fb84dffb700 1 -- 172.21.15.38:0/1695764994 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb844007620 con 0x7fb8500fed90 2023-03-09T14:44:39.690 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:39.682+0000 7fb84dffb700 1 -- 172.21.15.38:0/1695764994 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb84400f3c0 con 0x7fb8500fed90 2023-03-09T14:44:39.691 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:39.683+0000 7fb85700f700 1 -- 172.21.15.38:0/1695764994 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb85010a560 con 0x7fb8500fed90 2023-03-09T14:44:39.691 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:39.683+0000 7fb85700f700 1 -- 172.21.15.38:0/1695764994 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb850105d10 con 0x7fb8500fed90 2023-03-09T14:44:39.691 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:39.684+0000 7fb84dffb700 1 -- 172.21.15.38:0/1695764994 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 133198+0+0 (secure 0 0 0) 0x7fb844022070 con 0x7fb8500fed90 2023-03-09T14:44:39.691 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:39.685+0000 7fb84dffb700 1 --2- 172.21.15.38:0/1695764994 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fb84009f3c0 0x7fb8400a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:39.692 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:39.685+0000 7fb84dffb700 1 -- 172.21.15.38:0/1695764994 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fb8440ca2d0 con 0x7fb8500fed90 2023-03-09T14:44:39.692 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:39.685+0000 7fb84ffff700 1 --2- 172.21.15.38:0/1695764994 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fb84009f3c0 0x7fb8400a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:39.692 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:39.686+0000 7fb84ffff700 1 --2- 172.21.15.38:0/1695764994 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fb84009f3c0 0x7fb8400a1870 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb83c00b2e0 tx=0x7fb83c005e80 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:39.692 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:39.689+0000 7fb84dffb700 1 -- 172.21.15.38:0/1695764994 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7fb8440944a0 con 0x7fb8500fed90 2023-03-09T14:44:39.728 INFO:journalctl@ceph.mds.b.smithi122.stdout:Mar 09 14:44:39 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mds-b[119145]: 2023-03-09T14:44:39.305+0000 7fbfafa1a700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-03-09T14:44:39.929 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:39.926+0000 7fb85700f700 1 -- 172.21.15.38:0/1695764994 --> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "", "target": ["mon-mgr", ""]}) v1 -- 0x7fb850061920 con 0x7fb84009f3c0 2023-03-09T14:44:39.973 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:39.970+0000 7fb84dffb700 1 -- 172.21.15.38:0/1695764994 <== mgr.14150 v2:172.21.15.38:6800/3328887328 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fb850061920 con 0x7fb84009f3c0 2023-03-09T14:44:39.975 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:39.972+0000 7fb85700f700 1 -- 172.21.15.38:0/1695764994 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fb84009f3c0 msgr2=0x7fb8400a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:39.975 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:39.972+0000 7fb85700f700 1 --2- 172.21.15.38:0/1695764994 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fb84009f3c0 0x7fb8400a1870 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fb83c00b2e0 tx=0x7fb83c005e80 comp rx=0 tx=0).stop 2023-03-09T14:44:39.975 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:39.972+0000 7fb85700f700 1 -- 172.21.15.38:0/1695764994 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb8500fed90 msgr2=0x7fb8500685d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:39.975 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:39.972+0000 7fb85700f700 1 --2- 172.21.15.38:0/1695764994 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb8500fed90 0x7fb8500685d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fb84400d900 tx=0x7fb84400dcc0 comp rx=0 tx=0).stop 2023-03-09T14:44:39.975 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:39.972+0000 7fb85700f700 1 -- 172.21.15.38:0/1695764994 shutdown_connections 2023-03-09T14:44:39.975 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:39.972+0000 7fb85700f700 1 --2- 172.21.15.38:0/1695764994 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb8500fed90 0x7fb8500685d0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:39.976 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:39.972+0000 7fb85700f700 1 --2- 172.21.15.38:0/1695764994 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fb84009f3c0 0x7fb8400a1870 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:39.976 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:39.972+0000 7fb85700f700 1 --2- 172.21.15.38:0/1695764994 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb850100910 0x7fb85019cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:39.976 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:39.972+0000 7fb85700f700 1 --2- 172.21.15.38:0/1695764994 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb8500fff80 0x7fb85019a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:39.976 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:39.972+0000 7fb85700f700 1 -- 172.21.15.38:0/1695764994 >> 172.21.15.38:0/1695764994 conn(0x7fb8500fa3b0 msgr2=0x7fb8500fc690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:39.976 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:39.972+0000 7fb85700f700 1 -- 172.21.15.38:0/1695764994 shutdown_connections 2023-03-09T14:44:39.976 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:39.972+0000 7fb85700f700 1 -- 172.21.15.38:0/1695764994 wait complete. 2023-03-09T14:44:39.988 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv_0 2023-03-09T14:44:40.117 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.112+0000 7f9a675ae700 1 -- 172.21.15.38:0/2451667403 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a60100930 msgr2=0x7f9a601071f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:40.118 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.112+0000 7f9a675ae700 1 --2- 172.21.15.38:0/2451667403 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a60100930 0x7f9a601071f0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f9a5800b0a0 tx=0x7f9a5800b3b0 comp rx=0 tx=0).stop 2023-03-09T14:44:40.118 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.113+0000 7f9a675ae700 1 -- 172.21.15.38:0/2451667403 shutdown_connections 2023-03-09T14:44:40.118 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.113+0000 7f9a675ae700 1 --2- 172.21.15.38:0/2451667403 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a60100930 0x7f9a601071f0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:40.118 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.113+0000 7f9a675ae700 1 --2- 172.21.15.38:0/2451667403 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9a600fedb0 0x7f9a600ff1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:40.119 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.113+0000 7f9a675ae700 1 --2- 172.21.15.38:0/2451667403 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9a600fffa0 0x7f9a601003f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:40.119 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.113+0000 7f9a675ae700 1 -- 172.21.15.38:0/2451667403 >> 172.21.15.38:0/2451667403 conn(0x7f9a600fa3b0 msgr2=0x7f9a600fc800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:40.119 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.113+0000 7f9a675ae700 1 -- 172.21.15.38:0/2451667403 shutdown_connections 2023-03-09T14:44:40.119 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.113+0000 7f9a675ae700 1 -- 172.21.15.38:0/2451667403 wait complete. 2023-03-09T14:44:40.119 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.113+0000 7f9a675ae700 1 Processor -- start 2023-03-09T14:44:40.119 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.113+0000 7f9a675ae700 1 -- start start 2023-03-09T14:44:40.119 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.113+0000 7f9a675ae700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9a600fedb0 0x7f9a60197e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:40.119 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.113+0000 7f9a675ae700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a600fffa0 0x7f9a601983a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:40.120 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.113+0000 7f9a675ae700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9a60100930 0x7f9a6019f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:40.120 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.113+0000 7f9a675ae700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a6019f9e0 con 0x7f9a600fffa0 2023-03-09T14:44:40.120 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.113+0000 7f9a675ae700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a6019fb20 con 0x7f9a60100930 2023-03-09T14:44:40.120 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.113+0000 7f9a675ae700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a6019fcb0 con 0x7f9a600fedb0 2023-03-09T14:44:40.120 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.114+0000 7f9a64b49700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a600fffa0 0x7f9a601983a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:40.120 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.114+0000 7f9a64b49700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a600fffa0 0x7f9a601983a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37168/0 (socket says 172.21.15.38:37168) 2023-03-09T14:44:40.120 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.114+0000 7f9a64b49700 1 -- 172.21.15.38:0/1314554929 learned_addr learned my addr 172.21.15.38:0/1314554929 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:44:40.121 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.114+0000 7f9a6534a700 1 --2- 172.21.15.38:0/1314554929 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9a600fedb0 0x7f9a60197e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:40.121 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.114+0000 7f9a65b4b700 1 --2- 172.21.15.38:0/1314554929 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9a60100930 0x7f9a6019f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:40.121 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.114+0000 7f9a64b49700 1 -- 172.21.15.38:0/1314554929 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9a600fedb0 msgr2=0x7f9a60197e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:40.121 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.114+0000 7f9a64b49700 1 --2- 172.21.15.38:0/1314554929 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9a600fedb0 0x7f9a60197e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:40.121 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.114+0000 7f9a64b49700 1 -- 172.21.15.38:0/1314554929 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9a60100930 msgr2=0x7f9a6019f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:40.121 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.114+0000 7f9a64b49700 1 --2- 172.21.15.38:0/1314554929 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9a60100930 0x7f9a6019f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:40.122 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.114+0000 7f9a64b49700 1 -- 172.21.15.38:0/1314554929 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a58009d00 con 0x7f9a600fffa0 2023-03-09T14:44:40.122 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.114+0000 7f9a65b4b700 1 --2- 172.21.15.38:0/1314554929 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9a60100930 0x7f9a6019f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-09T14:44:40.122 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.114+0000 7f9a64b49700 1 --2- 172.21.15.38:0/1314554929 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a600fffa0 0x7f9a601983a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f9a5000c9b0 tx=0x7f9a5000cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:40.123 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.114+0000 7f9a567fc700 1 -- 172.21.15.38:0/1314554929 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a50007b00 con 0x7f9a600fffa0 2023-03-09T14:44:40.123 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.114+0000 7f9a567fc700 1 -- 172.21.15.38:0/1314554929 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9a50007c60 con 0x7f9a600fffa0 2023-03-09T14:44:40.123 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.114+0000 7f9a567fc700 1 -- 172.21.15.38:0/1314554929 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a50018660 con 0x7f9a600fffa0 2023-03-09T14:44:40.123 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.114+0000 7f9a675ae700 1 -- 172.21.15.38:0/1314554929 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a6010a720 con 0x7f9a600fffa0 2023-03-09T14:44:40.123 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.114+0000 7f9a675ae700 1 -- 172.21.15.38:0/1314554929 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a6010a590 con 0x7f9a600fffa0 2023-03-09T14:44:40.124 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.115+0000 7f9a675ae700 1 -- 172.21.15.38:0/1314554929 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a601a0340 con 0x7f9a600fffa0 2023-03-09T14:44:40.125 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.116+0000 7f9a567fc700 1 -- 172.21.15.38:0/1314554929 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 133198+0+0 (secure 0 0 0) 0x7f9a50018880 con 0x7f9a600fffa0 2023-03-09T14:44:40.125 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.117+0000 7f9a567fc700 1 --2- 172.21.15.38:0/1314554929 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f9a4c09f440 0x7f9a4c0a18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:40.125 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.117+0000 7f9a6534a700 1 --2- 172.21.15.38:0/1314554929 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f9a4c09f440 0x7f9a4c0a18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:40.125 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.117+0000 7f9a567fc700 1 -- 172.21.15.38:0/1314554929 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f9a500ca390 con 0x7f9a600fffa0 2023-03-09T14:44:40.125 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.118+0000 7f9a6534a700 1 --2- 172.21.15.38:0/1314554929 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f9a4c09f440 0x7f9a4c0a18f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c005fd0 tx=0x7f9a5c005dd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:40.125 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.121+0000 7f9a567fc700 1 -- 172.21.15.38:0/1314554929 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f9a50094670 con 0x7f9a600fffa0 2023-03-09T14:44:40.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:39 smithi122 ceph-mon[93869]: pgmap v269: 129 pgs: 129 active+clean; 581 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2023-03-09T14:44:40.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:39 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/1602860555' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2023-03-09T14:44:40.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:39 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T14:44:40.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:39 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/1602860555' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2023-03-09T14:44:40.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:39 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 9 up:standby 2023-03-09T14:44:40.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:39 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T14:44:40.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:39 smithi181 ceph-mon[94189]: pgmap v269: 129 pgs: 129 active+clean; 581 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2023-03-09T14:44:40.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:39 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/1602860555' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2023-03-09T14:44:40.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:39 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T14:44:40.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:39 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/1602860555' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2023-03-09T14:44:40.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:39 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 9 up:standby 2023-03-09T14:44:40.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:39 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T14:44:40.250 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:44:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:44:39.931+0000 7f3b59d3d700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-09T14:44:40.251 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:44:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:44:39.931+0000 7f3b59d3d700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-09T14:44:40.251 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:44:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:44:39.931+0000 7f3b59d3d700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-09T14:44:40.251 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:44:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:44:39.931+0000 7f3b59d3d700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-09T14:44:40.251 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:44:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:44:39.931+0000 7f3b59d3d700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-09T14:44:40.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:39 smithi038 ceph-mon[92422]: pgmap v269: 129 pgs: 129 active+clean; 581 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2023-03-09T14:44:40.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:39 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/1602860555' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2023-03-09T14:44:40.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:39 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T14:44:40.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:39 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/1602860555' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2023-03-09T14:44:40.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:39 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 9 up:standby 2023-03-09T14:44:40.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:39 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T14:44:40.359 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.356+0000 7f9a675ae700 1 -- 172.21.15.38:0/1314554929 --> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) v1 -- 0x7f9a601a08f0 con 0x7f9a4c09f440 2023-03-09T14:44:40.364 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.360+0000 7f9a567fc700 1 -- 172.21.15.38:0/1314554929 <== mgr.14150 v2:172.21.15.38:6800/3328887328 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+60 (secure 0 0 0) 0x7f9a601a08f0 con 0x7f9a4c09f440 2023-03-09T14:44:40.364 INFO:teuthology.orchestra.run.smithi038.stdout:/volumes/_nogroup/sv_0/e97d26b2-b787-4721-b5ae-cd98725ab26a 2023-03-09T14:44:40.370 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.362+0000 7f9a675ae700 1 -- 172.21.15.38:0/1314554929 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f9a4c09f440 msgr2=0x7f9a4c0a18f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:40.371 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.362+0000 7f9a675ae700 1 --2- 172.21.15.38:0/1314554929 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f9a4c09f440 0x7f9a4c0a18f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c005fd0 tx=0x7f9a5c005dd0 comp rx=0 tx=0).stop 2023-03-09T14:44:40.371 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.362+0000 7f9a675ae700 1 -- 172.21.15.38:0/1314554929 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a600fffa0 msgr2=0x7f9a601983a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:40.371 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.362+0000 7f9a675ae700 1 --2- 172.21.15.38:0/1314554929 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a600fffa0 0x7f9a601983a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f9a5000c9b0 tx=0x7f9a5000cd70 comp rx=0 tx=0).stop 2023-03-09T14:44:40.371 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.362+0000 7f9a675ae700 1 -- 172.21.15.38:0/1314554929 shutdown_connections 2023-03-09T14:44:40.371 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.362+0000 7f9a675ae700 1 --2- 172.21.15.38:0/1314554929 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f9a4c09f440 0x7f9a4c0a18f0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:40.371 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.362+0000 7f9a675ae700 1 --2- 172.21.15.38:0/1314554929 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9a60100930 0x7f9a6019f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:40.371 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.362+0000 7f9a675ae700 1 --2- 172.21.15.38:0/1314554929 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9a600fedb0 0x7f9a60197e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:40.372 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.362+0000 7f9a675ae700 1 --2- 172.21.15.38:0/1314554929 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9a600fffa0 0x7f9a601983a0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:40.372 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.362+0000 7f9a675ae700 1 -- 172.21.15.38:0/1314554929 >> 172.21.15.38:0/1314554929 conn(0x7f9a600fa3b0 msgr2=0x7f9a60068ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:40.372 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.363+0000 7f9a675ae700 1 -- 172.21.15.38:0/1314554929 shutdown_connections 2023-03-09T14:44:40.372 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.363+0000 7f9a675ae700 1 -- 172.21.15.38:0/1314554929 wait complete. 2023-03-09T14:44:40.379 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 '' 2023-03-09T14:44:40.512 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.506+0000 7fe398427700 1 -- 172.21.15.38:0/6103133 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe390101110 msgr2=0x7fe390101560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:40.512 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.506+0000 7fe398427700 1 --2- 172.21.15.38:0/6103133 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe390101110 0x7fe390101560 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fe380009990 tx=0x7fe380009ca0 comp rx=0 tx=0).stop 2023-03-09T14:44:40.513 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.506+0000 7fe398427700 1 -- 172.21.15.38:0/6103133 shutdown_connections 2023-03-09T14:44:40.513 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.506+0000 7fe398427700 1 --2- 172.21.15.38:0/6103133 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe390101aa0 0x7fe390108360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:40.513 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.506+0000 7fe398427700 1 --2- 172.21.15.38:0/6103133 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe3900fe140 0x7fe3900fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:40.513 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.506+0000 7fe398427700 1 --2- 172.21.15.38:0/6103133 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe390101110 0x7fe390101560 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:40.514 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.506+0000 7fe398427700 1 -- 172.21.15.38:0/6103133 >> 172.21.15.38:0/6103133 conn(0x7fe3900f9b60 msgr2=0x7fe3900fbf70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:40.514 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.506+0000 7fe398427700 1 -- 172.21.15.38:0/6103133 shutdown_connections 2023-03-09T14:44:40.514 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.506+0000 7fe398427700 1 -- 172.21.15.38:0/6103133 wait complete. 2023-03-09T14:44:40.514 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.506+0000 7fe398427700 1 Processor -- start 2023-03-09T14:44:40.515 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.506+0000 7fe398427700 1 -- start start 2023-03-09T14:44:40.515 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.506+0000 7fe398427700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe3900fe140 0x7fe390195bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:40.515 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.506+0000 7fe398427700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe390101110 0x7fe390196130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:40.515 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.506+0000 7fe398427700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe390101aa0 0x7fe39019d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:40.516 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.506+0000 7fe398427700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe39019d770 con 0x7fe390101110 2023-03-09T14:44:40.516 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.506+0000 7fe398427700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe39019d8b0 con 0x7fe3900fe140 2023-03-09T14:44:40.516 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.506+0000 7fe398427700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe39019da40 con 0x7fe390101aa0 2023-03-09T14:44:40.516 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.507+0000 7fe3959c2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe390101110 0x7fe390196130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:40.517 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.507+0000 7fe3959c2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe390101110 0x7fe390196130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37184/0 (socket says 172.21.15.38:37184) 2023-03-09T14:44:40.517 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.507+0000 7fe3959c2700 1 -- 172.21.15.38:0/520624280 learned_addr learned my addr 172.21.15.38:0/520624280 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:44:40.517 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.507+0000 7fe3961c3700 1 --2- 172.21.15.38:0/520624280 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe3900fe140 0x7fe390195bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:40.518 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.507+0000 7fe3959c2700 1 -- 172.21.15.38:0/520624280 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe390101aa0 msgr2=0x7fe39019d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:40.518 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.507+0000 7fe3969c4700 1 --2- 172.21.15.38:0/520624280 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe390101aa0 0x7fe39019d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:40.518 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.507+0000 7fe3959c2700 1 --2- 172.21.15.38:0/520624280 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe390101aa0 0x7fe39019d1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:40.518 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.507+0000 7fe3959c2700 1 -- 172.21.15.38:0/520624280 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe3900fe140 msgr2=0x7fe390195bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:40.519 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.507+0000 7fe3959c2700 1 --2- 172.21.15.38:0/520624280 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe3900fe140 0x7fe390195bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:40.519 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.507+0000 7fe3959c2700 1 -- 172.21.15.38:0/520624280 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3800095f0 con 0x7fe390101110 2023-03-09T14:44:40.519 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.507+0000 7fe3959c2700 1 --2- 172.21.15.38:0/520624280 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe390101110 0x7fe390196130 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fe380000c00 tx=0x7fe38000f710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:40.519 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.507+0000 7fe3877fe700 1 -- 172.21.15.38:0/520624280 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe38001d070 con 0x7fe390101110 2023-03-09T14:44:40.520 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.507+0000 7fe3877fe700 1 -- 172.21.15.38:0/520624280 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe38000fb50 con 0x7fe390101110 2023-03-09T14:44:40.520 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.507+0000 7fe398427700 1 -- 172.21.15.38:0/520624280 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe39010b7e0 con 0x7fe390101110 2023-03-09T14:44:40.520 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.507+0000 7fe3877fe700 1 -- 172.21.15.38:0/520624280 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe380017650 con 0x7fe390101110 2023-03-09T14:44:40.520 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.507+0000 7fe398427700 1 -- 172.21.15.38:0/520624280 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe39010b650 con 0x7fe390101110 2023-03-09T14:44:40.520 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.508+0000 7fe398427700 1 -- 172.21.15.38:0/520624280 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe390106ea0 con 0x7fe390101110 2023-03-09T14:44:40.521 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.509+0000 7fe3877fe700 1 -- 172.21.15.38:0/520624280 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 16) v1 ==== 133198+0+0 (secure 0 0 0) 0x7fe380029030 con 0x7fe390101110 2023-03-09T14:44:40.522 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.510+0000 7fe3877fe700 1 --2- 172.21.15.38:0/520624280 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fe37c09f3c0 0x7fe37c0a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:40.522 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.510+0000 7fe3961c3700 1 --2- 172.21.15.38:0/520624280 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fe37c09f3c0 0x7fe37c0a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:40.522 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.510+0000 7fe3877fe700 1 -- 172.21.15.38:0/520624280 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fe3800cb210 con 0x7fe390101110 2023-03-09T14:44:40.522 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.511+0000 7fe3961c3700 1 --2- 172.21.15.38:0/520624280 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fe37c09f3c0 0x7fe37c0a1870 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fe390067840 tx=0x7fe38c00b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:40.522 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.515+0000 7fe3877fe700 1 -- 172.21.15.38:0/520624280 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7fe380095470 con 0x7fe390101110 2023-03-09T14:44:40.748 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.746+0000 7fe398427700 1 -- 172.21.15.38:0/520624280 --> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "", "target": ["mon-mgr", ""]}) v1 -- 0x7fe390061910 con 0x7fe37c09f3c0 2023-03-09T14:44:40.772 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.770+0000 7fe3877fe700 1 -- 172.21.15.38:0/520624280 <== mgr.14150 v2:172.21.15.38:6800/3328887328 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fe390061910 con 0x7fe37c09f3c0 2023-03-09T14:44:40.774 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.772+0000 7fe398427700 1 -- 172.21.15.38:0/520624280 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fe37c09f3c0 msgr2=0x7fe37c0a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:40.774 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.772+0000 7fe398427700 1 --2- 172.21.15.38:0/520624280 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fe37c09f3c0 0x7fe37c0a1870 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fe390067840 tx=0x7fe38c00b410 comp rx=0 tx=0).stop 2023-03-09T14:44:40.775 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.772+0000 7fe398427700 1 -- 172.21.15.38:0/520624280 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe390101110 msgr2=0x7fe390196130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:40.775 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.772+0000 7fe398427700 1 --2- 172.21.15.38:0/520624280 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe390101110 0x7fe390196130 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fe380000c00 tx=0x7fe38000f710 comp rx=0 tx=0).stop 2023-03-09T14:44:40.775 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.772+0000 7fe398427700 1 -- 172.21.15.38:0/520624280 shutdown_connections 2023-03-09T14:44:40.775 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.772+0000 7fe398427700 1 --2- 172.21.15.38:0/520624280 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fe37c09f3c0 0x7fe37c0a1870 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:40.775 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.772+0000 7fe398427700 1 --2- 172.21.15.38:0/520624280 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe390101aa0 0x7fe39019d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:40.775 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.772+0000 7fe398427700 1 --2- 172.21.15.38:0/520624280 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe3900fe140 0x7fe390195bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:40.775 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.772+0000 7fe398427700 1 --2- 172.21.15.38:0/520624280 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe390101110 0x7fe390196130 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:40.776 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.772+0000 7fe398427700 1 -- 172.21.15.38:0/520624280 >> 172.21.15.38:0/520624280 conn(0x7fe3900f9b60 msgr2=0x7fe3900fce60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:40.776 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.772+0000 7fe398427700 1 -- 172.21.15.38:0/520624280 shutdown_connections 2023-03-09T14:44:40.776 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.772+0000 7fe398427700 1 -- 172.21.15.38:0/520624280 wait complete. 2023-03-09T14:44:40.786 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv_1 2023-03-09T14:44:40.919 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.917+0000 7f3055926700 1 -- 172.21.15.38:0/1767152309 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3050100980 msgr2=0x7f3050107240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:40.920 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.917+0000 7f3055926700 1 --2- 172.21.15.38:0/1767152309 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3050100980 0x7f3050107240 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f304800b0a0 tx=0x7f304800b3b0 comp rx=0 tx=0).stop 2023-03-09T14:44:40.920 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.917+0000 7f3055926700 1 -- 172.21.15.38:0/1767152309 shutdown_connections 2023-03-09T14:44:40.920 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.917+0000 7f3055926700 1 --2- 172.21.15.38:0/1767152309 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3050100980 0x7f3050107240 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:40.920 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.917+0000 7f3055926700 1 --2- 172.21.15.38:0/1767152309 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f30500fee00 0x7f30500ff210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:40.920 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.917+0000 7f3055926700 1 --2- 172.21.15.38:0/1767152309 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f30500ffff0 0x7f3050100440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:40.920 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.917+0000 7f3055926700 1 -- 172.21.15.38:0/1767152309 >> 172.21.15.38:0/1767152309 conn(0x7f30500fa420 msgr2=0x7f30500fc850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:40.921 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.917+0000 7f3055926700 1 -- 172.21.15.38:0/1767152309 shutdown_connections 2023-03-09T14:44:40.921 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.917+0000 7f3055926700 1 -- 172.21.15.38:0/1767152309 wait complete. 2023-03-09T14:44:40.923 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.918+0000 7f3055926700 1 Processor -- start 2023-03-09T14:44:40.923 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.918+0000 7f3055926700 1 -- start start 2023-03-09T14:44:40.923 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.918+0000 7f3055926700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30500fee00 0x7f3050069860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:40.923 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.918+0000 7f3055926700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f30500ffff0 0x7f305019a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:40.923 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.918+0000 7f3055926700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3050100980 0x7f305019d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:40.923 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.918+0000 7f3055926700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f305019d7d0 con 0x7f30500fee00 2023-03-09T14:44:40.923 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.918+0000 7f3055926700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f305019d940 con 0x7f30500ffff0 2023-03-09T14:44:40.924 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.918+0000 7f3055926700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f305019db00 con 0x7f3050100980 2023-03-09T14:44:40.924 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.918+0000 7f304effd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30500fee00 0x7f3050069860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:40.924 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.918+0000 7f304effd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30500fee00 0x7f3050069860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37196/0 (socket says 172.21.15.38:37196) 2023-03-09T14:44:40.925 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.918+0000 7f304effd700 1 -- 172.21.15.38:0/3216894805 learned_addr learned my addr 172.21.15.38:0/3216894805 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:44:40.925 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.919+0000 7f304e7fc700 1 --2- 172.21.15.38:0/3216894805 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f30500ffff0 0x7f305019a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:40.925 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.919+0000 7f304f7fe700 1 --2- 172.21.15.38:0/3216894805 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3050100980 0x7f305019d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:40.925 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.919+0000 7f304effd700 1 -- 172.21.15.38:0/3216894805 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3050100980 msgr2=0x7f305019d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:40.925 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.919+0000 7f304effd700 1 --2- 172.21.15.38:0/3216894805 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3050100980 0x7f305019d030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:40.925 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.919+0000 7f304effd700 1 -- 172.21.15.38:0/3216894805 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f30500ffff0 msgr2=0x7f305019a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:40.925 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.919+0000 7f304effd700 1 --2- 172.21.15.38:0/3216894805 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f30500ffff0 0x7f305019a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:40.926 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.919+0000 7f304effd700 1 -- 172.21.15.38:0/3216894805 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3048009d00 con 0x7f30500fee00 2023-03-09T14:44:40.926 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.919+0000 7f304effd700 1 --2- 172.21.15.38:0/3216894805 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30500fee00 0x7f3050069860 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f304400d980 tx=0x7f304400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:40.926 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.919+0000 7f3054924700 1 -- 172.21.15.38:0/3216894805 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3044004020 con 0x7f30500fee00 2023-03-09T14:44:40.926 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.919+0000 7f3054924700 1 -- 172.21.15.38:0/3216894805 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3044009730 con 0x7f30500fee00 2023-03-09T14:44:40.926 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.919+0000 7f3054924700 1 -- 172.21.15.38:0/3216894805 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3044003a80 con 0x7f30500fee00 2023-03-09T14:44:40.926 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.919+0000 7f3055926700 1 -- 172.21.15.38:0/3216894805 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f305010a700 con 0x7f30500fee00 2023-03-09T14:44:40.926 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.919+0000 7f3055926700 1 -- 172.21.15.38:0/3216894805 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f305010a570 con 0x7f30500fee00 2023-03-09T14:44:40.927 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.920+0000 7f3055926700 1 -- 172.21.15.38:0/3216894805 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3050105c90 con 0x7f30500fee00 2023-03-09T14:44:40.927 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.921+0000 7f3054924700 1 -- 172.21.15.38:0/3216894805 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 133238+0+0 (secure 0 0 0) 0x7f304400b030 con 0x7f30500fee00 2023-03-09T14:44:40.927 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.922+0000 7f3054924700 1 --2- 172.21.15.38:0/3216894805 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f303809f480 0x7f30380a1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:40.928 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.922+0000 7f3054924700 1 -- 172.21.15.38:0/3216894805 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f30440c9bd0 con 0x7f30500fee00 2023-03-09T14:44:40.928 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.922+0000 7f304e7fc700 1 --2- 172.21.15.38:0/3216894805 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f303809f480 0x7f30380a1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:40.928 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.922+0000 7f304e7fc700 1 --2- 172.21.15.38:0/3216894805 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f303809f480 0x7f30380a1930 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f3040005950 tx=0x7f304000b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:40.928 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:40.926+0000 7f3054924700 1 -- 172.21.15.38:0/3216894805 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f3044093d00 con 0x7f30500fee00 2023-03-09T14:44:41.165 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.163+0000 7f3055926700 1 -- 172.21.15.38:0/3216894805 --> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f3050061920 con 0x7f303809f480 2023-03-09T14:44:41.169 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.167+0000 7f3054924700 1 -- 172.21.15.38:0/3216894805 <== mgr.14150 v2:172.21.15.38:6800/3328887328 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+60 (secure 0 0 0) 0x7f3050061920 con 0x7f303809f480 2023-03-09T14:44:41.169 INFO:teuthology.orchestra.run.smithi038.stdout:/volumes/_nogroup/sv_1/ba63a2c0-19ef-41bb-becc-0a0d0bb2d986 2023-03-09T14:44:41.171 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.169+0000 7f3055926700 1 -- 172.21.15.38:0/3216894805 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f303809f480 msgr2=0x7f30380a1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:41.172 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.169+0000 7f3055926700 1 --2- 172.21.15.38:0/3216894805 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f303809f480 0x7f30380a1930 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f3040005950 tx=0x7f304000b410 comp rx=0 tx=0).stop 2023-03-09T14:44:41.172 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.169+0000 7f3055926700 1 -- 172.21.15.38:0/3216894805 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30500fee00 msgr2=0x7f3050069860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:41.172 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.169+0000 7f3055926700 1 --2- 172.21.15.38:0/3216894805 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30500fee00 0x7f3050069860 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f304400d980 tx=0x7f304400dc90 comp rx=0 tx=0).stop 2023-03-09T14:44:41.172 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.169+0000 7f3055926700 1 -- 172.21.15.38:0/3216894805 shutdown_connections 2023-03-09T14:44:41.172 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.169+0000 7f3055926700 1 --2- 172.21.15.38:0/3216894805 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f30500fee00 0x7f3050069860 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:41.172 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.169+0000 7f3055926700 1 --2- 172.21.15.38:0/3216894805 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f303809f480 0x7f30380a1930 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:41.172 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.169+0000 7f3055926700 1 --2- 172.21.15.38:0/3216894805 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3050100980 0x7f305019d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:41.173 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.169+0000 7f3055926700 1 --2- 172.21.15.38:0/3216894805 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f30500ffff0 0x7f305019a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:41.173 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.169+0000 7f3055926700 1 -- 172.21.15.38:0/3216894805 >> 172.21.15.38:0/3216894805 conn(0x7f30500fa420 msgr2=0x7f30500fc5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:41.173 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.169+0000 7f3055926700 1 -- 172.21.15.38:0/3216894805 shutdown_connections 2023-03-09T14:44:41.173 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.169+0000 7f3055926700 1 -- 172.21.15.38:0/3216894805 wait complete. 2023-03-09T14:44:41.182 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:40 smithi038 ceph-mon[92422]: from='client.14868 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:44:41.184 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-09T14:44:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:40 smithi122 ceph-mon[93869]: from='client.14868 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:44:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:40 smithi181 ceph-mon[94189]: from='client.14868 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:44:41.309 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.307+0000 7f79314a9700 1 -- 172.21.15.38:0/3606452442 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f792c105bc0 msgr2=0x7f792c107fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:41.310 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.307+0000 7f79314a9700 1 --2- 172.21.15.38:0/3606452442 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f792c105bc0 0x7f792c107fa0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f792400b0a0 tx=0x7f792400b3b0 comp rx=0 tx=0).stop 2023-03-09T14:44:41.310 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.307+0000 7f79314a9700 1 -- 172.21.15.38:0/3606452442 shutdown_connections 2023-03-09T14:44:41.310 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.307+0000 7f79314a9700 1 --2- 172.21.15.38:0/3606452442 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f792c105bc0 0x7f792c107fa0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:41.310 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.307+0000 7f79314a9700 1 --2- 172.21.15.38:0/3606452442 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f792c100980 0x7f792c102d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:41.310 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.307+0000 7f79314a9700 1 --2- 172.21.15.38:0/3606452442 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f792c1032a0 0x7f792c105680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:41.311 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.307+0000 7f79314a9700 1 -- 172.21.15.38:0/3606452442 >> 172.21.15.38:0/3606452442 conn(0x7f792c0fa380 msgr2=0x7f792c0fc7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:41.311 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.307+0000 7f79314a9700 1 -- 172.21.15.38:0/3606452442 shutdown_connections 2023-03-09T14:44:41.311 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.307+0000 7f79314a9700 1 -- 172.21.15.38:0/3606452442 wait complete. 2023-03-09T14:44:41.311 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.307+0000 7f79314a9700 1 Processor -- start 2023-03-09T14:44:41.313 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.308+0000 7f79314a9700 1 -- start start 2023-03-09T14:44:41.313 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.308+0000 7f79314a9700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f792c100980 0x7f792c1002f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:41.313 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.308+0000 7f79314a9700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f792c1032a0 0x7f792c0fe930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:41.313 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.308+0000 7f79314a9700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f792c105bc0 0x7f792c0fee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:41.313 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.308+0000 7f79314a9700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f792c0ff610 con 0x7f792c1032a0 2023-03-09T14:44:41.314 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.308+0000 7f79314a9700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f792c0ff780 con 0x7f792c105bc0 2023-03-09T14:44:41.314 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.308+0000 7f79314a9700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f792c19fd70 con 0x7f792c100980 2023-03-09T14:44:41.314 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.308+0000 7f792a7fc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f792c1032a0 0x7f792c0fe930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:41.314 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.308+0000 7f792a7fc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f792c1032a0 0x7f792c0fe930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37212/0 (socket says 172.21.15.38:37212) 2023-03-09T14:44:41.314 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.308+0000 7f792a7fc700 1 -- 172.21.15.38:0/2949851846 learned_addr learned my addr 172.21.15.38:0/2949851846 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:44:41.314 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.308+0000 7f792b7fe700 1 --2- 172.21.15.38:0/2949851846 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f792c105bc0 0x7f792c0fee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:41.314 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.308+0000 7f792affd700 1 --2- 172.21.15.38:0/2949851846 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f792c100980 0x7f792c1002f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:41.314 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.308+0000 7f792a7fc700 1 -- 172.21.15.38:0/2949851846 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f792c100980 msgr2=0x7f792c1002f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:41.315 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.308+0000 7f792a7fc700 1 --2- 172.21.15.38:0/2949851846 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f792c100980 0x7f792c1002f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:41.315 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.308+0000 7f792a7fc700 1 -- 172.21.15.38:0/2949851846 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f792c105bc0 msgr2=0x7f792c0fee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:41.315 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.308+0000 7f792a7fc700 1 --2- 172.21.15.38:0/2949851846 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f792c105bc0 0x7f792c0fee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:41.315 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.308+0000 7f792a7fc700 1 -- 172.21.15.38:0/2949851846 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7924009d00 con 0x7f792c1032a0 2023-03-09T14:44:41.315 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.309+0000 7f792a7fc700 1 --2- 172.21.15.38:0/2949851846 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f792c1032a0 0x7f792c0fe930 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f791c00e9a0 tx=0x7f791c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:41.315 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.309+0000 7f7913fff700 1 -- 172.21.15.38:0/2949851846 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f791c009790 con 0x7f792c1032a0 2023-03-09T14:44:41.315 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.309+0000 7f7913fff700 1 -- 172.21.15.38:0/2949851846 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f791c00cbd0 con 0x7f792c1032a0 2023-03-09T14:44:41.316 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.309+0000 7f7913fff700 1 -- 172.21.15.38:0/2949851846 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f791c018680 con 0x7f792c1032a0 2023-03-09T14:44:41.316 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.309+0000 7f79314a9700 1 -- 172.21.15.38:0/2949851846 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f792c10b810 con 0x7f792c1032a0 2023-03-09T14:44:41.316 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.309+0000 7f79314a9700 1 -- 172.21.15.38:0/2949851846 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f792c10b680 con 0x7f792c1032a0 2023-03-09T14:44:41.316 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.310+0000 7f79314a9700 1 -- 172.21.15.38:0/2949851846 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f792c0fbef0 con 0x7f792c1032a0 2023-03-09T14:44:41.316 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.311+0000 7f7913fff700 1 -- 172.21.15.38:0/2949851846 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 133238+0+0 (secure 0 0 0) 0x7f791c00c040 con 0x7f792c1032a0 2023-03-09T14:44:41.317 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.311+0000 7f7913fff700 1 --2- 172.21.15.38:0/2949851846 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f791409f480 0x7f79140a1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:41.317 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.311+0000 7f7913fff700 1 -- 172.21.15.38:0/2949851846 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f791c014070 con 0x7f792c1032a0 2023-03-09T14:44:41.317 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.311+0000 7f792affd700 1 --2- 172.21.15.38:0/2949851846 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f791409f480 0x7f79140a1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:41.317 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.312+0000 7f792affd700 1 --2- 172.21.15.38:0/2949851846 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f791409f480 0x7f79140a1930 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f7920005950 tx=0x7f79200058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:41.317 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.315+0000 7f7913fff700 1 -- 172.21.15.38:0/2949851846 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f791c094680 con 0x7f792c1032a0 2023-03-09T14:44:41.566 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.564+0000 7f79314a9700 1 -- 172.21.15.38:0/2949851846 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f792c02cef0 con 0x7f792c1032a0 2023-03-09T14:44:41.566 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.564+0000 7f7913fff700 1 -- 172.21.15.38:0/2949851846 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+10801 (secure 0 0 0) 0x7f791c026090 con 0x7f792c1032a0 2023-03-09T14:44:41.567 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:44:41.567 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/768918602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":768918602},{"type":"v1","addr":"172.21.15.38:6835","nonce":768918602}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/18381540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":18381540},{"type":"v1","addr":"172.21.15.38:6837","nonce":18381540}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14610,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6839/4139020066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":4139020066},{"type":"v1","addr":"172.21.15.38:6839","nonce":4139020066}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14622,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6841/1397115322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":1397115322},{"type":"v1","addr":"172.21.15.38:6841","nonce":1397115322}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6833/415305571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":415305571},{"type":"v1","addr":"172.21.15.181:6833","nonce":415305571}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/659300520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":659300520},{"type":"v1","addr":"172.21.15.122:6833","nonce":659300520}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-03-09T14:44:19.083770+0000","modified":"2023-03-09T14:44:39.526150+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24530,"mds_1":24494,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24448":{"gid":24448,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6835/2576255861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":2576255861},{"type":"v1","addr":"172.21.15.181:6835","nonce":2576255861}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24460":{"gid":24460,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6837/2942560181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":2942560181},{"type":"v1","addr":"172.21.15.181:6837","nonce":2942560181}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24464":{"gid":24464,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.122:6835/3675537941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":3675537941},{"type":"v1","addr":"172.21.15.122:6835","nonce":3675537941}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24478":{"gid":24478,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":9,"addr":"172.21.15.181:6839/3902263660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":3902263660},{"type":"v1","addr":"172.21.15.181:6839","nonce":3902263660}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24494":{"gid":24494,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.122:6837/2210214596","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":2210214596},{"type":"v1","addr":"172.21.15.122:6837","nonce":2210214596}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24530":{"gid":24530,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6839/971050165","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6838","nonce":971050165},{"type":"v1","addr":"172.21.15.122:6839","nonce":971050165}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-09T14:44:41.569 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.566+0000 7f79314a9700 1 -- 172.21.15.38:0/2949851846 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f791409f480 msgr2=0x7f79140a1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:41.569 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.566+0000 7f79314a9700 1 --2- 172.21.15.38:0/2949851846 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f791409f480 0x7f79140a1930 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f7920005950 tx=0x7f79200058e0 comp rx=0 tx=0).stop 2023-03-09T14:44:41.569 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.566+0000 7f79314a9700 1 -- 172.21.15.38:0/2949851846 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f792c1032a0 msgr2=0x7f792c0fe930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:41.569 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.566+0000 7f79314a9700 1 --2- 172.21.15.38:0/2949851846 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f792c1032a0 0x7f792c0fe930 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f791c00e9a0 tx=0x7f791c00ecb0 comp rx=0 tx=0).stop 2023-03-09T14:44:41.569 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.567+0000 7f79314a9700 1 -- 172.21.15.38:0/2949851846 shutdown_connections 2023-03-09T14:44:41.569 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.567+0000 7f79314a9700 1 --2- 172.21.15.38:0/2949851846 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f791409f480 0x7f79140a1930 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:41.570 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.567+0000 7f79314a9700 1 --2- 172.21.15.38:0/2949851846 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f792c105bc0 0x7f792c0fee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:41.570 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.567+0000 7f79314a9700 1 --2- 172.21.15.38:0/2949851846 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f792c100980 0x7f792c1002f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:41.570 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.567+0000 7f79314a9700 1 --2- 172.21.15.38:0/2949851846 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f792c1032a0 0x7f792c0fe930 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:41.570 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.567+0000 7f79314a9700 1 -- 172.21.15.38:0/2949851846 >> 172.21.15.38:0/2949851846 conn(0x7f792c0fa380 msgr2=0x7f792c1015e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:41.570 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.567+0000 7f79314a9700 1 -- 172.21.15.38:0/2949851846 shutdown_connections 2023-03-09T14:44:41.570 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.567+0000 7f79314a9700 1 -- 172.21.15.38:0/2949851846 wait complete. 2023-03-09T14:44:41.571 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 23 2023-03-09T14:44:41.582 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-09T14:44:41.708 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.705+0000 7f0abcd7f700 1 -- 172.21.15.38:0/1500400878 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ab80fff70 msgr2=0x7f0ab81003c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:41.708 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.705+0000 7f0abcd7f700 1 --2- 172.21.15.38:0/1500400878 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ab80fff70 0x7f0ab81003c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f0aa0009990 tx=0x7f0aa0009ca0 comp rx=0 tx=0).stop 2023-03-09T14:44:41.708 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.705+0000 7f0abcd7f700 1 -- 172.21.15.38:0/1500400878 shutdown_connections 2023-03-09T14:44:41.709 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.705+0000 7f0abcd7f700 1 --2- 172.21.15.38:0/1500400878 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0ab8100900 0x7f0ab81071c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:41.709 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.705+0000 7f0abcd7f700 1 --2- 172.21.15.38:0/1500400878 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0ab80fed80 0x7f0ab80ff190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:41.709 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.705+0000 7f0abcd7f700 1 --2- 172.21.15.38:0/1500400878 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ab80fff70 0x7f0ab81003c0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:41.709 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.705+0000 7f0abcd7f700 1 -- 172.21.15.38:0/1500400878 >> 172.21.15.38:0/1500400878 conn(0x7f0ab80fa380 msgr2=0x7f0ab80fc7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:41.710 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.705+0000 7f0abcd7f700 1 -- 172.21.15.38:0/1500400878 shutdown_connections 2023-03-09T14:44:41.710 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.705+0000 7f0abcd7f700 1 -- 172.21.15.38:0/1500400878 wait complete. 2023-03-09T14:44:41.713 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.706+0000 7f0abcd7f700 1 Processor -- start 2023-03-09T14:44:41.713 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.706+0000 7f0abcd7f700 1 -- start start 2023-03-09T14:44:41.714 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.706+0000 7f0abcd7f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ab80fed80 0x7f0ab80685d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:41.714 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.706+0000 7f0abcd7f700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0ab80fff70 0x7f0ab819a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:41.714 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.706+0000 7f0abcd7f700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0ab8100900 0x7f0ab819cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:41.714 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.706+0000 7f0abcd7f700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab819d730 con 0x7f0ab80fed80 2023-03-09T14:44:41.715 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.706+0000 7f0abcd7f700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab819d8a0 con 0x7f0ab80fff70 2023-03-09T14:44:41.715 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.706+0000 7f0abcd7f700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab819da60 con 0x7f0ab8100900 2023-03-09T14:44:41.715 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.707+0000 7f0ab659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ab80fed80 0x7f0ab80685d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:41.715 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.707+0000 7f0ab659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ab80fed80 0x7f0ab80685d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37224/0 (socket says 172.21.15.38:37224) 2023-03-09T14:44:41.715 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.707+0000 7f0ab659c700 1 -- 172.21.15.38:0/43017438 learned_addr learned my addr 172.21.15.38:0/43017438 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:44:41.715 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.707+0000 7f0ab5d9b700 1 --2- 172.21.15.38:0/43017438 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0ab80fff70 0x7f0ab819a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:41.715 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.707+0000 7f0ab6d9d700 1 --2- 172.21.15.38:0/43017438 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0ab8100900 0x7f0ab819cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:41.715 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.707+0000 7f0ab659c700 1 -- 172.21.15.38:0/43017438 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0ab8100900 msgr2=0x7f0ab819cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:41.716 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.707+0000 7f0ab659c700 1 --2- 172.21.15.38:0/43017438 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0ab8100900 0x7f0ab819cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:41.716 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.707+0000 7f0ab659c700 1 -- 172.21.15.38:0/43017438 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0ab80fff70 msgr2=0x7f0ab819a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:41.716 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.707+0000 7f0ab659c700 1 --2- 172.21.15.38:0/43017438 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0ab80fff70 0x7f0ab819a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:41.716 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.707+0000 7f0ab659c700 1 -- 172.21.15.38:0/43017438 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aa00095f0 con 0x7f0ab80fed80 2023-03-09T14:44:41.716 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.707+0000 7f0ab659c700 1 --2- 172.21.15.38:0/43017438 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ab80fed80 0x7f0ab80685d0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f0aa800d980 tx=0x7f0aa800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:41.717 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.707+0000 7f0aaf7fe700 1 -- 172.21.15.38:0/43017438 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa8004020 con 0x7f0ab80fed80 2023-03-09T14:44:41.717 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.707+0000 7f0aaf7fe700 1 -- 172.21.15.38:0/43017438 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0aa8009730 con 0x7f0ab80fed80 2023-03-09T14:44:41.717 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.707+0000 7f0aaf7fe700 1 -- 172.21.15.38:0/43017438 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa8003a80 con 0x7f0ab80fed80 2023-03-09T14:44:41.717 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.707+0000 7f0abcd7f700 1 -- 172.21.15.38:0/43017438 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ab810a6e0 con 0x7f0ab80fed80 2023-03-09T14:44:41.717 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.707+0000 7f0abcd7f700 1 -- 172.21.15.38:0/43017438 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ab810a550 con 0x7f0ab80fed80 2023-03-09T14:44:41.717 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.708+0000 7f0abcd7f700 1 -- 172.21.15.38:0/43017438 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ab8105d00 con 0x7f0ab80fed80 2023-03-09T14:44:41.717 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.709+0000 7f0aaf7fe700 1 -- 172.21.15.38:0/43017438 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 133238+0+0 (secure 0 0 0) 0x7f0aa800b030 con 0x7f0ab80fed80 2023-03-09T14:44:41.718 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.710+0000 7f0aaf7fe700 1 --2- 172.21.15.38:0/43017438 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f0aa409f400 0x7f0aa40a18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:41.718 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.710+0000 7f0ab5d9b700 1 --2- 172.21.15.38:0/43017438 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f0aa409f400 0x7f0aa40a18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:41.718 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.710+0000 7f0aaf7fe700 1 -- 172.21.15.38:0/43017438 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f0aa80c9fa0 con 0x7f0ab80fed80 2023-03-09T14:44:41.718 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.710+0000 7f0ab5d9b700 1 --2- 172.21.15.38:0/43017438 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f0aa409f400 0x7f0aa40a18b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f0aa000b2e0 tx=0x7f0aa0005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:41.718 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.713+0000 7f0aaf7fe700 1 -- 172.21.15.38:0/43017438 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f0aa8094130 con 0x7f0ab80fed80 2023-03-09T14:44:41.918 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.915+0000 7f0abcd7f700 1 -- 172.21.15.38:0/43017438 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0ab8061390 con 0x7f0ab80fed80 2023-03-09T14:44:41.919 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.917+0000 7f0aaf7fe700 1 -- 172.21.15.38:0/43017438 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+23973 (secure 0 0 0) 0x7f0aa8097750 con 0x7f0ab80fed80 2023-03-09T14:44:41.920 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:44:41.920 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":87,"fsid":"c6733fe2-be87-11ed-9af2-001a4aab830c","created":"2023-03-09T14:37:14.947981+0000","modified":"2023-03-09T14:44:23.900228+0000","last_up_change":"2023-03-09T14:43:37.198407+0000","last_in_change":"2023-03-09T14:43:19.550566+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-09T14:40:23.196665+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-03-09T14:44:17.484374+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-03-09T14:44:18.457833+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-03-09T14:44:21.895589+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"51477039-15ee-4707-a34d-4ecfa18bebde","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6803","nonce":1508072290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6805","nonce":1508072290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6809","nonce":1508072290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6807","nonce":1508072290}]},"public_addr":"172.21.15.38:6803/1508072290","cluster_addr":"172.21.15.38:6805/1508072290","heartbeat_back_addr":"172.21.15.38:6809/1508072290","heartbeat_front_addr":"172.21.15.38:6807/1508072290","state":["exists","up"]},{"osd":1,"uuid":"71f8a718-b0a9-4f39-bc54-267aa42a2578","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6801","nonce":3929268773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6803","nonce":3929268773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6807","nonce":3929268773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6805","nonce":3929268773}]},"public_addr":"172.21.15.122:6801/3929268773","cluster_addr":"172.21.15.122:6803/3929268773","heartbeat_back_addr":"172.21.15.122:6807/3929268773","heartbeat_front_addr":"172.21.15.122:6805/3929268773","state":["exists","up"]},{"osd":2,"uuid":"961a2cbe-140e-437f-b131-1fcac573b19c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6801","nonce":1310630803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6803","nonce":1310630803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6807","nonce":1310630803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6805","nonce":1310630803}]},"public_addr":"172.21.15.181:6801/1310630803","cluster_addr":"172.21.15.181:6803/1310630803","heartbeat_back_addr":"172.21.15.181:6807/1310630803","heartbeat_front_addr":"172.21.15.181:6805/1310630803","state":["exists","up"]},{"osd":3,"uuid":"969df8fa-2d19-42c7-9f2a-5c5fac15d4a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6811","nonce":2719814100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6813","nonce":2719814100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6817","nonce":2719814100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6815","nonce":2719814100}]},"public_addr":"172.21.15.38:6811/2719814100","cluster_addr":"172.21.15.38:6813/2719814100","heartbeat_back_addr":"172.21.15.38:6817/2719814100","heartbeat_front_addr":"172.21.15.38:6815/2719814100","state":["exists","up"]},{"osd":4,"uuid":"44a6dce3-e06e-4cbd-9404-62a86d01ab6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6809","nonce":2480764994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6811","nonce":2480764994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6815","nonce":2480764994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6813","nonce":2480764994}]},"public_addr":"172.21.15.122:6809/2480764994","cluster_addr":"172.21.15.122:6811/2480764994","heartbeat_back_addr":"172.21.15.122:6815/2480764994","heartbeat_front_addr":"172.21.15.122:6813/2480764994","state":["exists","up"]},{"osd":5,"uuid":"49749574-c00c-4e3a-80e6-e3686df58d5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6809","nonce":3141346246}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6811","nonce":3141346246}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6815","nonce":3141346246}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6813","nonce":3141346246}]},"public_addr":"172.21.15.181:6809/3141346246","cluster_addr":"172.21.15.181:6811/3141346246","heartbeat_back_addr":"172.21.15.181:6815/3141346246","heartbeat_front_addr":"172.21.15.181:6813/3141346246","state":["exists","up"]},{"osd":6,"uuid":"61c0437e-3f51-45a9-8a61-e98eea0093f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6819","nonce":2122788957}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6821","nonce":2122788957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6825","nonce":2122788957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6823","nonce":2122788957}]},"public_addr":"172.21.15.38:6819/2122788957","cluster_addr":"172.21.15.38:6821/2122788957","heartbeat_back_addr":"172.21.15.38:6825/2122788957","heartbeat_front_addr":"172.21.15.38:6823/2122788957","state":["exists","up"]},{"osd":7,"uuid":"231fecdd-939f-4c7b-83a1-c4a1cffd7139","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6817","nonce":656874139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6819","nonce":656874139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6823","nonce":656874139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6821","nonce":656874139}]},"public_addr":"172.21.15.122:6817/656874139","cluster_addr":"172.21.15.122:6819/656874139","heartbeat_back_addr":"172.21.15.122:6823/656874139","heartbeat_front_addr":"172.21.15.122:6821/656874139","state":["exists","up"]},{"osd":8,"uuid":"297e6d6d-85fd-43ce-9028-5e9dbe55c36e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6817","nonce":755000703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6819","nonce":755000703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6823","nonce":755000703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6821","nonce":755000703}]},"public_addr":"172.21.15.181:6817/755000703","cluster_addr":"172.21.15.181:6819/755000703","heartbeat_back_addr":"172.21.15.181:6823/755000703","heartbeat_front_addr":"172.21.15.181:6821/755000703","state":["exists","up"]},{"osd":9,"uuid":"94cc0355-5a1a-4cfc-bd0c-562b6b41127e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6827","nonce":2624555969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6829","nonce":2624555969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6833","nonce":2624555969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6831","nonce":2624555969}]},"public_addr":"172.21.15.38:6827/2624555969","cluster_addr":"172.21.15.38:6829/2624555969","heartbeat_back_addr":"172.21.15.38:6833/2624555969","heartbeat_front_addr":"172.21.15.38:6831/2624555969","state":["exists","up"]},{"osd":10,"uuid":"f4f2e042-8d92-46bf-93b8-8b0f04795c2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6825","nonce":1427785280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6827","nonce":1427785280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6831","nonce":1427785280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6829","nonce":1427785280}]},"public_addr":"172.21.15.122:6825/1427785280","cluster_addr":"172.21.15.122:6827/1427785280","heartbeat_back_addr":"172.21.15.122:6831/1427785280","heartbeat_front_addr":"172.21.15.122:6829/1427785280","state":["exists","up"]},{"osd":11,"uuid":"07444936-eae5-4097-a799-60294cafe80b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6825","nonce":3183488640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6827","nonce":3183488640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6831","nonce":3183488640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6829","nonce":3183488640}]},"public_addr":"172.21.15.181:6825/3183488640","cluster_addr":"172.21.15.181:6827/3183488640","heartbeat_back_addr":"172.21.15.181:6831/3183488640","heartbeat_front_addr":"172.21.15.181:6829/3183488640","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:39:37.657556+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:39:58.960074+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:40:20.326952+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:40:42.244387+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:41:03.946973+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:41:25.097985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:41:46.877916+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:42:08.413827+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:42:29.822009+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:42:51.776684+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:43:13.937249+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:43:35.569199+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/4272288735":"2023-03-10T14:38:14.612767+0000","172.21.15.38:6801/3381625669":"2023-03-10T14:38:14.612767+0000","172.21.15.38:6800/3381625669":"2023-03-10T14:38:14.612767+0000","172.21.15.38:6800/3060474342":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/4253918615":"2023-03-10T14:38:14.612767+0000","172.21.15.38:0/3496083181":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/245602368":"2023-03-10T14:37:47.517775+0000","172.21.15.38:6801/3060474342":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/3622589182":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/1430302251":"2023-03-10T14:38:14.612767+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-09T14:44:41.921 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.919+0000 7f0abcd7f700 1 -- 172.21.15.38:0/43017438 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f0aa409f400 msgr2=0x7f0aa40a18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:41.922 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.919+0000 7f0abcd7f700 1 --2- 172.21.15.38:0/43017438 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f0aa409f400 0x7f0aa40a18b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f0aa000b2e0 tx=0x7f0aa0005960 comp rx=0 tx=0).stop 2023-03-09T14:44:41.922 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.919+0000 7f0abcd7f700 1 -- 172.21.15.38:0/43017438 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ab80fed80 msgr2=0x7f0ab80685d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:41.922 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.919+0000 7f0abcd7f700 1 --2- 172.21.15.38:0/43017438 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ab80fed80 0x7f0ab80685d0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f0aa800d980 tx=0x7f0aa800dc90 comp rx=0 tx=0).stop 2023-03-09T14:44:41.922 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.919+0000 7f0abcd7f700 1 -- 172.21.15.38:0/43017438 shutdown_connections 2023-03-09T14:44:41.922 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.919+0000 7f0abcd7f700 1 --2- 172.21.15.38:0/43017438 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ab80fed80 0x7f0ab80685d0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:41.922 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.919+0000 7f0abcd7f700 1 --2- 172.21.15.38:0/43017438 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f0aa409f400 0x7f0aa40a18b0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:41.923 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.919+0000 7f0abcd7f700 1 --2- 172.21.15.38:0/43017438 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0ab8100900 0x7f0ab819cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:41.923 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.919+0000 7f0abcd7f700 1 --2- 172.21.15.38:0/43017438 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0ab80fff70 0x7f0ab819a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:41.923 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.919+0000 7f0abcd7f700 1 -- 172.21.15.38:0/43017438 >> 172.21.15.38:0/43017438 conn(0x7f0ab80fa380 msgr2=0x7f0ab80fc660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:41.923 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.919+0000 7f0abcd7f700 1 -- 172.21.15.38:0/43017438 shutdown_connections 2023-03-09T14:44:41.923 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:41.919+0000 7f0abcd7f700 1 -- 172.21.15.38:0/43017438 wait complete. 2023-03-09T14:44:41.934 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-03-09T14:44:41.934 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-03-09T14:44:42.060 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.057+0000 7f190a503700 1 -- 172.21.15.38:0/2707876261 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f19040ff980 msgr2=0x7f1904107230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:42.060 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.057+0000 7f190a503700 1 --2- 172.21.15.38:0/2707876261 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f19040ff980 0x7f1904107230 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f18fc00b0a0 tx=0x7f18fc00b3b0 comp rx=0 tx=0).stop 2023-03-09T14:44:42.060 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.057+0000 7f190a503700 1 -- 172.21.15.38:0/2707876261 shutdown_connections 2023-03-09T14:44:42.060 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.057+0000 7f190a503700 1 --2- 172.21.15.38:0/2707876261 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f19040ff980 0x7f1904107230 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:42.061 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.057+0000 7f190a503700 1 --2- 172.21.15.38:0/2707876261 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1904100410 0x7f1904100820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:42.061 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.057+0000 7f190a503700 1 --2- 172.21.15.38:0/2707876261 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f19040fefd0 0x7f19040ff440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:42.061 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.057+0000 7f190a503700 1 -- 172.21.15.38:0/2707876261 >> 172.21.15.38:0/2707876261 conn(0x7f19040f9b10 msgr2=0x7f19040fbf60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:42.061 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.057+0000 7f190a503700 1 -- 172.21.15.38:0/2707876261 shutdown_connections 2023-03-09T14:44:42.061 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.057+0000 7f190a503700 1 -- 172.21.15.38:0/2707876261 wait complete. 2023-03-09T14:44:42.062 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.057+0000 7f190a503700 1 Processor -- start 2023-03-09T14:44:42.062 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.058+0000 7f190a503700 1 -- start start 2023-03-09T14:44:42.064 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.058+0000 7f190a503700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f19040fefd0 0x7f19040fea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:42.064 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.058+0000 7f190a503700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f19040ff980 0x7f190419cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:42.064 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.058+0000 7f190a503700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1904100410 0x7f190419f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:42.064 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.058+0000 7f190a503700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f190419f9b0 con 0x7f1904100410 2023-03-09T14:44:42.064 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.058+0000 7f190a503700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f190419fb20 con 0x7f19040ff980 2023-03-09T14:44:42.064 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.058+0000 7f190a503700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f190419fce0 con 0x7f19040fefd0 2023-03-09T14:44:42.065 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.058+0000 7f1908aa0700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1904100410 0x7f190419f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:42.065 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.058+0000 7f1908aa0700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1904100410 0x7f190419f300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37242/0 (socket says 172.21.15.38:37242) 2023-03-09T14:44:42.065 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.058+0000 7f1908aa0700 1 -- 172.21.15.38:0/3414333692 learned_addr learned my addr 172.21.15.38:0/3414333692 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:44:42.065 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.058+0000 7f19037fe700 1 --2- 172.21.15.38:0/3414333692 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f19040ff980 0x7f190419cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:42.065 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.058+0000 7f1903fff700 1 --2- 172.21.15.38:0/3414333692 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f19040fefd0 0x7f19040fea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:42.065 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.058+0000 7f1908aa0700 1 -- 172.21.15.38:0/3414333692 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f19040fefd0 msgr2=0x7f19040fea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:42.065 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.058+0000 7f1908aa0700 1 --2- 172.21.15.38:0/3414333692 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f19040fefd0 0x7f19040fea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:42.066 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.058+0000 7f1908aa0700 1 -- 172.21.15.38:0/3414333692 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f19040ff980 msgr2=0x7f190419cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:42.066 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.058+0000 7f1908aa0700 1 --2- 172.21.15.38:0/3414333692 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f19040ff980 0x7f190419cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:42.066 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.058+0000 7f1908aa0700 1 -- 172.21.15.38:0/3414333692 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18fc009d00 con 0x7f1904100410 2023-03-09T14:44:42.066 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.058+0000 7f1903fff700 1 --2- 172.21.15.38:0/3414333692 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f19040fefd0 0x7f19040fea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-09T14:44:42.066 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.058+0000 7f1908aa0700 1 --2- 172.21.15.38:0/3414333692 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1904100410 0x7f190419f300 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f18fc0062a0 tx=0x7f18fc004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:42.067 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.059+0000 7f19017fa700 1 -- 172.21.15.38:0/3414333692 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18fc00e040 con 0x7f1904100410 2023-03-09T14:44:42.067 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.059+0000 7f190a503700 1 -- 172.21.15.38:0/3414333692 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f190410a6f0 con 0x7f1904100410 2023-03-09T14:44:42.067 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.059+0000 7f19017fa700 1 -- 172.21.15.38:0/3414333692 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f18fc00cec0 con 0x7f1904100410 2023-03-09T14:44:42.067 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.059+0000 7f19017fa700 1 -- 172.21.15.38:0/3414333692 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18fc004020 con 0x7f1904100410 2023-03-09T14:44:42.067 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.059+0000 7f190a503700 1 -- 172.21.15.38:0/3414333692 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f190410a560 con 0x7f1904100410 2023-03-09T14:44:42.067 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.060+0000 7f190a503700 1 -- 172.21.15.38:0/3414333692 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1904105c20 con 0x7f1904100410 2023-03-09T14:44:42.068 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.061+0000 7f19017fa700 1 -- 172.21.15.38:0/3414333692 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 133238+0+0 (secure 0 0 0) 0x7f18fc01d690 con 0x7f1904100410 2023-03-09T14:44:42.068 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.061+0000 7f19017fa700 1 --2- 172.21.15.38:0/3414333692 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f18f009f400 0x7f18f00a18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:42.068 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.061+0000 7f19017fa700 1 -- 172.21.15.38:0/3414333692 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f18fc0cb9b0 con 0x7f1904100410 2023-03-09T14:44:42.068 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.061+0000 7f1903fff700 1 --2- 172.21.15.38:0/3414333692 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f18f009f400 0x7f18f00a18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:42.068 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.062+0000 7f1903fff700 1 --2- 172.21.15.38:0/3414333692 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f18f009f400 0x7f18f00a18b0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f18f4005fd0 tx=0x7f18f4005ee0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:42.069 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.065+0000 7f19017fa700 1 -- 172.21.15.38:0/3414333692 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f18fc095bd0 con 0x7f1904100410 2023-03-09T14:44:42.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:41 smithi122 ceph-mon[93869]: from='client.14880 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:44:42.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:41 smithi122 ceph-mon[93869]: from='client.14886 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:44:42.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:41 smithi122 ceph-mon[93869]: pgmap v270: 129 pgs: 129 active+clean; 598 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 597 B/s rd, 2.2 KiB/s wr, 4 op/s 2023-03-09T14:44:42.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:41 smithi122 ceph-mon[93869]: mgrmap e17: x(active, since 6m), standbys: z, y 2023-03-09T14:44:42.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:41 smithi122 ceph-mon[93869]: from='client.14892 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:44:42.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:41 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/2949851846' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-03-09T14:44:42.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:41 smithi181 ceph-mon[94189]: from='client.14880 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:44:42.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:41 smithi181 ceph-mon[94189]: from='client.14886 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:44:42.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:41 smithi181 ceph-mon[94189]: pgmap v270: 129 pgs: 129 active+clean; 598 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 597 B/s rd, 2.2 KiB/s wr, 4 op/s 2023-03-09T14:44:42.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:41 smithi181 ceph-mon[94189]: mgrmap e17: x(active, since 6m), standbys: z, y 2023-03-09T14:44:42.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:41 smithi181 ceph-mon[94189]: from='client.14892 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:44:42.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:41 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/2949851846' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-03-09T14:44:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:41 smithi038 ceph-mon[92422]: from='client.14880 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:44:42.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:41 smithi038 ceph-mon[92422]: from='client.14886 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:44:42.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:41 smithi038 ceph-mon[92422]: pgmap v270: 129 pgs: 129 active+clean; 598 KiB data, 79 MiB used, 1.0 TiB / 1.0 TiB avail; 597 B/s rd, 2.2 KiB/s wr, 4 op/s 2023-03-09T14:44:42.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:41 smithi038 ceph-mon[92422]: mgrmap e17: x(active, since 6m), standbys: z, y 2023-03-09T14:44:42.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:41 smithi038 ceph-mon[92422]: from='client.14892 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:44:42.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:41 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/2949851846' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-03-09T14:44:42.262 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.260+0000 7f190a503700 1 -- 172.21.15.38:0/3414333692 --> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1904061890 con 0x7f18f009f400 2023-03-09T14:44:42.278 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.275+0000 7f19017fa700 1 -- 172.21.15.38:0/3414333692 <== mgr.14150 v2:172.21.15.38:6800/3328887328 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+418536 (secure 0 0 0) 0x7f1904061890 con 0x7f18f009f400 2023-03-09T14:44:42.283 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:44:42.286 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.279+0000 7f190a503700 1 -- 172.21.15.38:0/3414333692 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f18f009f400 msgr2=0x7f18f00a18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:42.286 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.279+0000 7f190a503700 1 --2- 172.21.15.38:0/3414333692 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f18f009f400 0x7f18f00a18b0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f18f4005fd0 tx=0x7f18f4005ee0 comp rx=0 tx=0).stop 2023-03-09T14:44:42.286 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.279+0000 7f190a503700 1 -- 172.21.15.38:0/3414333692 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1904100410 msgr2=0x7f190419f300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:42.286 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.279+0000 7f190a503700 1 --2- 172.21.15.38:0/3414333692 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1904100410 0x7f190419f300 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f18fc0062a0 tx=0x7f18fc004d00 comp rx=0 tx=0).stop 2023-03-09T14:44:42.286 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.279+0000 7f190a503700 1 -- 172.21.15.38:0/3414333692 shutdown_connections 2023-03-09T14:44:42.287 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.279+0000 7f190a503700 1 --2- 172.21.15.38:0/3414333692 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1904100410 0x7f190419f300 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:42.287 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.279+0000 7f190a503700 1 --2- 172.21.15.38:0/3414333692 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f18f009f400 0x7f18f00a18b0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:42.287 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.279+0000 7f190a503700 1 --2- 172.21.15.38:0/3414333692 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f19040fefd0 0x7f19040fea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:42.287 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.279+0000 7f190a503700 1 --2- 172.21.15.38:0/3414333692 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f19040ff980 0x7f190419cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:42.287 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.279+0000 7f190a503700 1 -- 172.21.15.38:0/3414333692 >> 172.21.15.38:0/3414333692 conn(0x7f19040f9b10 msgr2=0x7f19040fbd30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:42.287 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.279+0000 7f190a503700 1 -- 172.21.15.38:0/3414333692 shutdown_connections 2023-03-09T14:44:42.287 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.279+0000 7f190a503700 1 -- 172.21.15.38:0/3414333692 wait complete. 2023-03-09T14:44:42.288 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2023-03-09T14:44:42.294 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":270,"stamp":"2023-03-09T14:44:40.750225+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":612515,"num_objects":38,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":141,"num_read_kb":126,"num_write":272,"num_write_kb":2978,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":25,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":214,"ondisk_log_size":214,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":81388,"kb_used_data":13132,"kb_used_omap":0,"kb_used_meta":68160,"kb_avail":1124860436,"statfs":{"total":1151940427776,"available":1151857086464,"internally_reserved":0,"allocated":13447168,"data_stored":7392765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":69795840},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19769,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":5,"num_read_kb":7,"num_write":46,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003359"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.473760+0000","last_change":"2023-03-09T14:44:19.116371+0000","last_active":"2023-03-09T14:44:23.473760+0000","last_peered":"2023-03-09T14:44:23.473760+0000","last_clean":"2023-03-09T14:44:23.473760+0000","last_became_active":"2023-03-09T14:44:19.116024+0000","last_became_peered":"2023-03-09T14:44:19.116024+0000","last_unstale":"2023-03-09T14:44:23.473760+0000","last_undegraded":"2023-03-09T14:44:23.473760+0000","last_fullsized":"2023-03-09T14:44:23.473760+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T16:43:33.917176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.475467+0000","last_change":"2023-03-09T14:44:19.117312+0000","last_active":"2023-03-09T14:44:23.475467+0000","last_peered":"2023-03-09T14:44:23.475467+0000","last_clean":"2023-03-09T14:44:23.475467+0000","last_became_active":"2023-03-09T14:44:19.116920+0000","last_became_peered":"2023-03-09T14:44:19.116920+0000","last_unstale":"2023-03-09T14:44:23.475467+0000","last_undegraded":"2023-03-09T14:44:23.475467+0000","last_fullsized":"2023-03-09T14:44:23.475467+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:33:15.046431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.472579+0000","last_change":"2023-03-09T14:44:19.113482+0000","last_active":"2023-03-09T14:44:23.472579+0000","last_peered":"2023-03-09T14:44:23.472579+0000","last_clean":"2023-03-09T14:44:23.472579+0000","last_became_active":"2023-03-09T14:44:19.113183+0000","last_became_peered":"2023-03-09T14:44:19.113183+0000","last_unstale":"2023-03-09T14:44:23.472579+0000","last_undegraded":"2023-03-09T14:44:23.472579+0000","last_fullsized":"2023-03-09T14:44:23.472579+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:17:31.382255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.471310+0000","last_change":"2023-03-09T14:44:19.105799+0000","last_active":"2023-03-09T14:44:23.471310+0000","last_peered":"2023-03-09T14:44:23.471310+0000","last_clean":"2023-03-09T14:44:23.471310+0000","last_became_active":"2023-03-09T14:44:19.105405+0000","last_became_peered":"2023-03-09T14:44:19.105405+0000","last_unstale":"2023-03-09T14:44:23.471310+0000","last_undegraded":"2023-03-09T14:44:23.471310+0000","last_fullsized":"2023-03-09T14:44:23.471310+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:38:23.049984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.472968+0000","last_change":"2023-03-09T14:44:19.100656+0000","last_active":"2023-03-09T14:44:23.472968+0000","last_peered":"2023-03-09T14:44:23.472968+0000","last_clean":"2023-03-09T14:44:23.472968+0000","last_became_active":"2023-03-09T14:44:19.100278+0000","last_became_peered":"2023-03-09T14:44:19.100278+0000","last_unstale":"2023-03-09T14:44:23.472968+0000","last_undegraded":"2023-03-09T14:44:23.472968+0000","last_fullsized":"2023-03-09T14:44:23.472968+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:25:39.240251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.469680+0000","last_change":"2023-03-09T14:44:19.101065+0000","last_active":"2023-03-09T14:44:23.469680+0000","last_peered":"2023-03-09T14:44:23.469680+0000","last_clean":"2023-03-09T14:44:23.469680+0000","last_became_active":"2023-03-09T14:44:19.100451+0000","last_became_peered":"2023-03-09T14:44:19.100451+0000","last_unstale":"2023-03-09T14:44:23.469680+0000","last_undegraded":"2023-03-09T14:44:23.469680+0000","last_fullsized":"2023-03-09T14:44:23.469680+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T01:25:15.158254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.473598+0000","last_change":"2023-03-09T14:44:19.107996+0000","last_active":"2023-03-09T14:44:23.473598+0000","last_peered":"2023-03-09T14:44:23.473598+0000","last_clean":"2023-03-09T14:44:23.473598+0000","last_became_active":"2023-03-09T14:44:19.107683+0000","last_became_peered":"2023-03-09T14:44:19.107683+0000","last_unstale":"2023-03-09T14:44:23.473598+0000","last_undegraded":"2023-03-09T14:44:23.473598+0000","last_fullsized":"2023-03-09T14:44:23.473598+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:16:18.605176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.469646+0000","last_change":"2023-03-09T14:44:19.103940+0000","last_active":"2023-03-09T14:44:23.469646+0000","last_peered":"2023-03-09T14:44:23.469646+0000","last_clean":"2023-03-09T14:44:23.469646+0000","last_became_active":"2023-03-09T14:44:19.103620+0000","last_became_peered":"2023-03-09T14:44:19.103620+0000","last_unstale":"2023-03-09T14:44:23.469646+0000","last_undegraded":"2023-03-09T14:44:23.469646+0000","last_fullsized":"2023-03-09T14:44:23.469646+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:06:22.749636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.477978+0000","last_change":"2023-03-09T14:44:19.102695+0000","last_active":"2023-03-09T14:44:23.477978+0000","last_peered":"2023-03-09T14:44:23.477978+0000","last_clean":"2023-03-09T14:44:23.477978+0000","last_became_active":"2023-03-09T14:44:19.101486+0000","last_became_peered":"2023-03-09T14:44:19.101486+0000","last_unstale":"2023-03-09T14:44:23.477978+0000","last_undegraded":"2023-03-09T14:44:23.477978+0000","last_fullsized":"2023-03-09T14:44:23.477978+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T16:34:51.836443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.469700+0000","last_change":"2023-03-09T14:44:19.103522+0000","last_active":"2023-03-09T14:44:23.469700+0000","last_peered":"2023-03-09T14:44:23.469700+0000","last_clean":"2023-03-09T14:44:23.469700+0000","last_became_active":"2023-03-09T14:44:19.103242+0000","last_became_peered":"2023-03-09T14:44:19.103242+0000","last_unstale":"2023-03-09T14:44:23.469700+0000","last_undegraded":"2023-03-09T14:44:23.469700+0000","last_fullsized":"2023-03-09T14:44:23.469700+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:41:03.137554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.474842+0000","last_change":"2023-03-09T14:44:19.114934+0000","last_active":"2023-03-09T14:44:23.474842+0000","last_peered":"2023-03-09T14:44:23.474842+0000","last_clean":"2023-03-09T14:44:23.474842+0000","last_became_active":"2023-03-09T14:44:19.114620+0000","last_became_peered":"2023-03-09T14:44:19.114620+0000","last_unstale":"2023-03-09T14:44:23.474842+0000","last_undegraded":"2023-03-09T14:44:23.474842+0000","last_fullsized":"2023-03-09T14:44:23.474842+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T01:48:40.082076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.471227+0000","last_change":"2023-03-09T14:44:19.107677+0000","last_active":"2023-03-09T14:44:23.471227+0000","last_peered":"2023-03-09T14:44:23.471227+0000","last_clean":"2023-03-09T14:44:23.471227+0000","last_became_active":"2023-03-09T14:44:19.107369+0000","last_became_peered":"2023-03-09T14:44:19.107369+0000","last_unstale":"2023-03-09T14:44:23.471227+0000","last_undegraded":"2023-03-09T14:44:23.471227+0000","last_fullsized":"2023-03-09T14:44:23.471227+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:47:40.117205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.477738+0000","last_change":"2023-03-09T14:44:19.108852+0000","last_active":"2023-03-09T14:44:23.477738+0000","last_peered":"2023-03-09T14:44:23.477738+0000","last_clean":"2023-03-09T14:44:23.477738+0000","last_became_active":"2023-03-09T14:44:19.108515+0000","last_became_peered":"2023-03-09T14:44:19.108515+0000","last_unstale":"2023-03-09T14:44:23.477738+0000","last_undegraded":"2023-03-09T14:44:23.477738+0000","last_fullsized":"2023-03-09T14:44:23.477738+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:00:36.243952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.473653+0000","last_change":"2023-03-09T14:44:19.118673+0000","last_active":"2023-03-09T14:44:23.473653+0000","last_peered":"2023-03-09T14:44:23.473653+0000","last_clean":"2023-03-09T14:44:23.473653+0000","last_became_active":"2023-03-09T14:44:19.117656+0000","last_became_peered":"2023-03-09T14:44:19.117656+0000","last_unstale":"2023-03-09T14:44:23.473653+0000","last_undegraded":"2023-03-09T14:44:23.473653+0000","last_fullsized":"2023-03-09T14:44:23.473653+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:02:50.401680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.473802+0000","last_change":"2023-03-09T14:44:19.112979+0000","last_active":"2023-03-09T14:44:23.473802+0000","last_peered":"2023-03-09T14:44:23.473802+0000","last_clean":"2023-03-09T14:44:23.473802+0000","last_became_active":"2023-03-09T14:44:19.112671+0000","last_became_peered":"2023-03-09T14:44:19.112671+0000","last_unstale":"2023-03-09T14:44:23.473802+0000","last_undegraded":"2023-03-09T14:44:23.473802+0000","last_fullsized":"2023-03-09T14:44:23.473802+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:00:43.465327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.474845+0000","last_change":"2023-03-09T14:44:19.104881+0000","last_active":"2023-03-09T14:44:23.474845+0000","last_peered":"2023-03-09T14:44:23.474845+0000","last_clean":"2023-03-09T14:44:23.474845+0000","last_became_active":"2023-03-09T14:44:19.104647+0000","last_became_peered":"2023-03-09T14:44:19.104647+0000","last_unstale":"2023-03-09T14:44:23.474845+0000","last_undegraded":"2023-03-09T14:44:23.474845+0000","last_fullsized":"2023-03-09T14:44:23.474845+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T20:16:24.625362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.473964+0000","last_change":"2023-03-09T14:44:19.117937+0000","last_active":"2023-03-09T14:44:23.473964+0000","last_peered":"2023-03-09T14:44:23.473964+0000","last_clean":"2023-03-09T14:44:23.473964+0000","last_became_active":"2023-03-09T14:44:19.117628+0000","last_became_peered":"2023-03-09T14:44:19.117628+0000","last_unstale":"2023-03-09T14:44:23.473964+0000","last_undegraded":"2023-03-09T14:44:23.473964+0000","last_fullsized":"2023-03-09T14:44:23.473964+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:22:18.619171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.479118+0000","last_change":"2023-03-09T14:44:19.105071+0000","last_active":"2023-03-09T14:44:23.479118+0000","last_peered":"2023-03-09T14:44:23.479118+0000","last_clean":"2023-03-09T14:44:23.479118+0000","last_became_active":"2023-03-09T14:44:19.103396+0000","last_became_peered":"2023-03-09T14:44:19.103396+0000","last_unstale":"2023-03-09T14:44:23.479118+0000","last_undegraded":"2023-03-09T14:44:23.479118+0000","last_fullsized":"2023-03-09T14:44:23.479118+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:14:51.924636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.475650+0000","last_change":"2023-03-09T14:44:19.104107+0000","last_active":"2023-03-09T14:44:23.475650+0000","last_peered":"2023-03-09T14:44:23.475650+0000","last_clean":"2023-03-09T14:44:23.475650+0000","last_became_active":"2023-03-09T14:44:19.103793+0000","last_became_peered":"2023-03-09T14:44:19.103793+0000","last_unstale":"2023-03-09T14:44:23.475650+0000","last_undegraded":"2023-03-09T14:44:23.475650+0000","last_fullsized":"2023-03-09T14:44:23.475650+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:57:29.169744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.490413+0000","last_change":"2023-03-09T14:44:23.489947+0000","last_active":"2023-03-09T14:44:23.490413+0000","last_peered":"2023-03-09T14:44:23.490413+0000","last_clean":"2023-03-09T14:44:23.490413+0000","last_became_active":"2023-03-09T14:44:23.489560+0000","last_became_peered":"2023-03-09T14:44:23.489560+0000","last_unstale":"2023-03-09T14:44:23.490413+0000","last_undegraded":"2023-03-09T14:44:23.490413+0000","last_fullsized":"2023-03-09T14:44:23.490413+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:06:10.202176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"87'9","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-03-09T14:44:38.524476+0000","last_change":"2023-03-09T14:44:18.653405+0000","last_active":"2023-03-09T14:44:38.524476+0000","last_peered":"2023-03-09T14:44:38.524476+0000","last_clean":"2023-03-09T14:44:38.524476+0000","last_became_active":"2023-03-09T14:44:18.653085+0000","last_became_peered":"2023-03-09T14:44:18.653085+0000","last_unstale":"2023-03-09T14:44:38.524476+0000","last_undegraded":"2023-03-09T14:44:38.524476+0000","last_fullsized":"2023-03-09T14:44:38.524476+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:57:26.782467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.471353+0000","last_change":"2023-03-09T14:44:19.105390+0000","last_active":"2023-03-09T14:44:23.471353+0000","last_peered":"2023-03-09T14:44:23.471353+0000","last_clean":"2023-03-09T14:44:23.471353+0000","last_became_active":"2023-03-09T14:44:19.104694+0000","last_became_peered":"2023-03-09T14:44:19.104694+0000","last_unstale":"2023-03-09T14:44:23.471353+0000","last_undegraded":"2023-03-09T14:44:23.471353+0000","last_fullsized":"2023-03-09T14:44:23.471353+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:19:08.101847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.904879+0000","last_change":"2023-03-09T14:44:23.484103+0000","last_active":"2023-03-09T14:44:23.904879+0000","last_peered":"2023-03-09T14:44:23.904879+0000","last_clean":"2023-03-09T14:44:23.904879+0000","last_became_active":"2023-03-09T14:44:23.483271+0000","last_became_peered":"2023-03-09T14:44:23.483271+0000","last_unstale":"2023-03-09T14:44:23.904879+0000","last_undegraded":"2023-03-09T14:44:23.904879+0000","last_fullsized":"2023-03-09T14:44:23.904879+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:24:06.212623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"87'2","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.904795+0000","last_change":"2023-03-09T14:44:18.667321+0000","last_active":"2023-03-09T14:44:23.904795+0000","last_peered":"2023-03-09T14:44:23.904795+0000","last_clean":"2023-03-09T14:44:23.904795+0000","last_became_active":"2023-03-09T14:44:18.666974+0000","last_became_peered":"2023-03-09T14:44:18.666974+0000","last_unstale":"2023-03-09T14:44:23.904795+0000","last_undegraded":"2023-03-09T14:44:23.904795+0000","last_fullsized":"2023-03-09T14:44:23.904795+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:31:07.327792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.473744+0000","last_change":"2023-03-09T14:44:19.116866+0000","last_active":"2023-03-09T14:44:23.473744+0000","last_peered":"2023-03-09T14:44:23.473744+0000","last_clean":"2023-03-09T14:44:23.473744+0000","last_became_active":"2023-03-09T14:44:19.116571+0000","last_became_peered":"2023-03-09T14:44:19.116571+0000","last_unstale":"2023-03-09T14:44:23.473744+0000","last_undegraded":"2023-03-09T14:44:23.473744+0000","last_fullsized":"2023-03-09T14:44:23.473744+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T16:22:12.723903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.905925+0000","last_change":"2023-03-09T14:44:23.488192+0000","last_active":"2023-03-09T14:44:23.905925+0000","last_peered":"2023-03-09T14:44:23.905925+0000","last_clean":"2023-03-09T14:44:23.905925+0000","last_became_active":"2023-03-09T14:44:23.487734+0000","last_became_peered":"2023-03-09T14:44:23.487734+0000","last_unstale":"2023-03-09T14:44:23.905925+0000","last_undegraded":"2023-03-09T14:44:23.905925+0000","last_fullsized":"2023-03-09T14:44:23.905925+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:35:50.319216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"82'2","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.475193+0000","last_change":"2023-03-09T14:44:18.663577+0000","last_active":"2023-03-09T14:44:23.475193+0000","last_peered":"2023-03-09T14:44:23.475193+0000","last_clean":"2023-03-09T14:44:23.475193+0000","last_became_active":"2023-03-09T14:44:18.663272+0000","last_became_peered":"2023-03-09T14:44:18.663272+0000","last_unstale":"2023-03-09T14:44:23.475193+0000","last_undegraded":"2023-03-09T14:44:23.475193+0000","last_fullsized":"2023-03-09T14:44:23.475193+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T20:55:19.661875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.475940+0000","last_change":"2023-03-09T14:44:19.106986+0000","last_active":"2023-03-09T14:44:23.475940+0000","last_peered":"2023-03-09T14:44:23.475940+0000","last_clean":"2023-03-09T14:44:23.475940+0000","last_became_active":"2023-03-09T14:44:19.106625+0000","last_became_peered":"2023-03-09T14:44:19.106625+0000","last_unstale":"2023-03-09T14:44:23.475940+0000","last_undegraded":"2023-03-09T14:44:23.475940+0000","last_fullsized":"2023-03-09T14:44:23.475940+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:03:54.521603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.905654+0000","last_change":"2023-03-09T14:44:23.483803+0000","last_active":"2023-03-09T14:44:23.905654+0000","last_peered":"2023-03-09T14:44:23.905654+0000","last_clean":"2023-03-09T14:44:23.905654+0000","last_became_active":"2023-03-09T14:44:23.483286+0000","last_became_peered":"2023-03-09T14:44:23.483286+0000","last_unstale":"2023-03-09T14:44:23.905654+0000","last_undegraded":"2023-03-09T14:44:23.905654+0000","last_fullsized":"2023-03-09T14:44:23.905654+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:32:35.975670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"82'2","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.475167+0000","last_change":"2023-03-09T14:44:18.657685+0000","last_active":"2023-03-09T14:44:23.475167+0000","last_peered":"2023-03-09T14:44:23.475167+0000","last_clean":"2023-03-09T14:44:23.475167+0000","last_became_active":"2023-03-09T14:44:18.657355+0000","last_became_peered":"2023-03-09T14:44:18.657355+0000","last_unstale":"2023-03-09T14:44:23.475167+0000","last_undegraded":"2023-03-09T14:44:23.475167+0000","last_fullsized":"2023-03-09T14:44:23.475167+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:33:59.259690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.480488+0000","last_change":"2023-03-09T14:44:19.111687+0000","last_active":"2023-03-09T14:44:23.480488+0000","last_peered":"2023-03-09T14:44:23.480488+0000","last_clean":"2023-03-09T14:44:23.480488+0000","last_became_active":"2023-03-09T14:44:19.111481+0000","last_became_peered":"2023-03-09T14:44:19.111481+0000","last_unstale":"2023-03-09T14:44:23.480488+0000","last_undegraded":"2023-03-09T14:44:23.480488+0000","last_fullsized":"2023-03-09T14:44:23.480488+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:06:14.452967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":12,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.487525+0000","last_change":"2023-03-09T14:44:23.485075+0000","last_active":"2023-03-09T14:44:23.487525+0000","last_peered":"2023-03-09T14:44:23.487525+0000","last_clean":"2023-03-09T14:44:23.487525+0000","last_became_active":"2023-03-09T14:44:23.484648+0000","last_became_peered":"2023-03-09T14:44:23.484648+0000","last_unstale":"2023-03-09T14:44:23.487525+0000","last_undegraded":"2023-03-09T14:44:23.487525+0000","last_fullsized":"2023-03-09T14:44:23.487525+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:49:40.332276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"82'1","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.479619+0000","last_change":"2023-03-09T14:44:18.665323+0000","last_active":"2023-03-09T14:44:23.479619+0000","last_peered":"2023-03-09T14:44:23.479619+0000","last_clean":"2023-03-09T14:44:23.479619+0000","last_became_active":"2023-03-09T14:44:18.665001+0000","last_became_peered":"2023-03-09T14:44:18.665001+0000","last_unstale":"2023-03-09T14:44:23.479619+0000","last_undegraded":"2023-03-09T14:44:23.479619+0000","last_fullsized":"2023-03-09T14:44:23.479619+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:22:20.325914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.477336+0000","last_change":"2023-03-09T14:44:19.107572+0000","last_active":"2023-03-09T14:44:23.477336+0000","last_peered":"2023-03-09T14:44:23.477336+0000","last_clean":"2023-03-09T14:44:23.477336+0000","last_became_active":"2023-03-09T14:44:19.106632+0000","last_became_peered":"2023-03-09T14:44:19.106632+0000","last_unstale":"2023-03-09T14:44:23.477336+0000","last_undegraded":"2023-03-09T14:44:23.477336+0000","last_fullsized":"2023-03-09T14:44:23.477336+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T19:04:01.379375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":12,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.494387+0000","last_change":"2023-03-09T14:44:23.494002+0000","last_active":"2023-03-09T14:44:23.494387+0000","last_peered":"2023-03-09T14:44:23.494387+0000","last_clean":"2023-03-09T14:44:23.494387+0000","last_became_active":"2023-03-09T14:44:23.493628+0000","last_became_peered":"2023-03-09T14:44:23.493628+0000","last_unstale":"2023-03-09T14:44:23.494387+0000","last_undegraded":"2023-03-09T14:44:23.494387+0000","last_fullsized":"2023-03-09T14:44:23.494387+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:30:47.970556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"87'3","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-03-09T14:44:38.524129+0000","last_change":"2023-03-09T14:44:18.661169+0000","last_active":"2023-03-09T14:44:38.524129+0000","last_peered":"2023-03-09T14:44:38.524129+0000","last_clean":"2023-03-09T14:44:38.524129+0000","last_became_active":"2023-03-09T14:44:18.660858+0000","last_became_peered":"2023-03-09T14:44:18.660858+0000","last_unstale":"2023-03-09T14:44:38.524129+0000","last_undegraded":"2023-03-09T14:44:38.524129+0000","last_fullsized":"2023-03-09T14:44:38.524129+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:18:45.551100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.479758+0000","last_change":"2023-03-09T14:44:19.111303+0000","last_active":"2023-03-09T14:44:23.479758+0000","last_peered":"2023-03-09T14:44:23.479758+0000","last_clean":"2023-03-09T14:44:23.479758+0000","last_became_active":"2023-03-09T14:44:19.111026+0000","last_became_peered":"2023-03-09T14:44:19.111026+0000","last_unstale":"2023-03-09T14:44:23.479758+0000","last_undegraded":"2023-03-09T14:44:23.479758+0000","last_fullsized":"2023-03-09T14:44:23.479758+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T02:26:32.496084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":12,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.493647+0000","last_change":"2023-03-09T14:44:23.493272+0000","last_active":"2023-03-09T14:44:23.493647+0000","last_peered":"2023-03-09T14:44:23.493647+0000","last_clean":"2023-03-09T14:44:23.493647+0000","last_became_active":"2023-03-09T14:44:23.492785+0000","last_became_peered":"2023-03-09T14:44:23.492785+0000","last_unstale":"2023-03-09T14:44:23.493647+0000","last_undegraded":"2023-03-09T14:44:23.493647+0000","last_fullsized":"2023-03-09T14:44:23.493647+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T20:18:39.758938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"87'2","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-03-09T14:44:38.523245+0000","last_change":"2023-03-09T14:44:18.658179+0000","last_active":"2023-03-09T14:44:38.523245+0000","last_peered":"2023-03-09T14:44:38.523245+0000","last_clean":"2023-03-09T14:44:38.523245+0000","last_became_active":"2023-03-09T14:44:18.657851+0000","last_became_peered":"2023-03-09T14:44:18.657851+0000","last_unstale":"2023-03-09T14:44:38.523245+0000","last_undegraded":"2023-03-09T14:44:38.523245+0000","last_fullsized":"2023-03-09T14:44:38.523245+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:52:49.291615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.469539+0000","last_change":"2023-03-09T14:44:19.104176+0000","last_active":"2023-03-09T14:44:23.469539+0000","last_peered":"2023-03-09T14:44:23.469539+0000","last_clean":"2023-03-09T14:44:23.469539+0000","last_became_active":"2023-03-09T14:44:19.103816+0000","last_became_peered":"2023-03-09T14:44:19.103816+0000","last_unstale":"2023-03-09T14:44:23.469539+0000","last_undegraded":"2023-03-09T14:44:23.469539+0000","last_fullsized":"2023-03-09T14:44:23.469539+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T02:06:41.876169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-03-09T14:44:24.313741+0000","last_change":"2023-03-09T14:44:23.490876+0000","last_active":"2023-03-09T14:44:24.313741+0000","last_peered":"2023-03-09T14:44:24.313741+0000","last_clean":"2023-03-09T14:44:24.313741+0000","last_became_active":"2023-03-09T14:44:23.490461+0000","last_became_peered":"2023-03-09T14:44:23.490461+0000","last_unstale":"2023-03-09T14:44:24.313741+0000","last_undegraded":"2023-03-09T14:44:24.313741+0000","last_fullsized":"2023-03-09T14:44:24.313741+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T16:28:40.456613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.474747+0000","last_change":"2023-03-09T14:44:18.662513+0000","last_active":"2023-03-09T14:44:23.474747+0000","last_peered":"2023-03-09T14:44:23.474747+0000","last_clean":"2023-03-09T14:44:23.474747+0000","last_became_active":"2023-03-09T14:44:18.661907+0000","last_became_peered":"2023-03-09T14:44:18.661907+0000","last_unstale":"2023-03-09T14:44:23.474747+0000","last_undegraded":"2023-03-09T14:44:23.474747+0000","last_fullsized":"2023-03-09T14:44:23.474747+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:45:17.373213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.476301+0000","last_change":"2023-03-09T14:44:19.115297+0000","last_active":"2023-03-09T14:44:23.476301+0000","last_peered":"2023-03-09T14:44:23.476301+0000","last_clean":"2023-03-09T14:44:23.476301+0000","last_became_active":"2023-03-09T14:44:19.114973+0000","last_became_peered":"2023-03-09T14:44:19.114973+0000","last_unstale":"2023-03-09T14:44:23.476301+0000","last_undegraded":"2023-03-09T14:44:23.476301+0000","last_fullsized":"2023-03-09T14:44:23.476301+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:11:22.226608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.904202+0000","last_change":"2023-03-09T14:44:23.482772+0000","last_active":"2023-03-09T14:44:23.904202+0000","last_peered":"2023-03-09T14:44:23.904202+0000","last_clean":"2023-03-09T14:44:23.904202+0000","last_became_active":"2023-03-09T14:44:23.481750+0000","last_became_peered":"2023-03-09T14:44:23.481750+0000","last_unstale":"2023-03-09T14:44:23.904202+0000","last_undegraded":"2023-03-09T14:44:23.904202+0000","last_fullsized":"2023-03-09T14:44:23.904202+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T19:26:03.737713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"87'3","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-03-09T14:44:37.502621+0000","last_change":"2023-03-09T14:44:18.665582+0000","last_active":"2023-03-09T14:44:37.502621+0000","last_peered":"2023-03-09T14:44:37.502621+0000","last_clean":"2023-03-09T14:44:37.502621+0000","last_became_active":"2023-03-09T14:44:18.665265+0000","last_became_peered":"2023-03-09T14:44:18.665265+0000","last_unstale":"2023-03-09T14:44:37.502621+0000","last_undegraded":"2023-03-09T14:44:37.502621+0000","last_fullsized":"2023-03-09T14:44:37.502621+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T16:15:54.119713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.477924+0000","last_change":"2023-03-09T14:44:19.100914+0000","last_active":"2023-03-09T14:44:23.477924+0000","last_peered":"2023-03-09T14:44:23.477924+0000","last_clean":"2023-03-09T14:44:23.477924+0000","last_became_active":"2023-03-09T14:44:19.100451+0000","last_became_peered":"2023-03-09T14:44:19.100451+0000","last_unstale":"2023-03-09T14:44:23.477924+0000","last_undegraded":"2023-03-09T14:44:23.477924+0000","last_fullsized":"2023-03-09T14:44:23.477924+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:18:41.852697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:24.249360+0000","last_change":"2023-03-09T14:44:23.490129+0000","last_active":"2023-03-09T14:44:24.249360+0000","last_peered":"2023-03-09T14:44:24.249360+0000","last_clean":"2023-03-09T14:44:24.249360+0000","last_became_active":"2023-03-09T14:44:23.489696+0000","last_became_peered":"2023-03-09T14:44:23.489696+0000","last_unstale":"2023-03-09T14:44:24.249360+0000","last_undegraded":"2023-03-09T14:44:24.249360+0000","last_fullsized":"2023-03-09T14:44:24.249360+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T16:47:41.943904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.474329+0000","last_change":"2023-03-09T14:44:18.661420+0000","last_active":"2023-03-09T14:44:23.474329+0000","last_peered":"2023-03-09T14:44:23.474329+0000","last_clean":"2023-03-09T14:44:23.474329+0000","last_became_active":"2023-03-09T14:44:18.661119+0000","last_became_peered":"2023-03-09T14:44:18.661119+0000","last_unstale":"2023-03-09T14:44:23.474329+0000","last_undegraded":"2023-03-09T14:44:23.474329+0000","last_fullsized":"2023-03-09T14:44:23.474329+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:27:26.403125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.473839+0000","last_change":"2023-03-09T14:44:19.119125+0000","last_active":"2023-03-09T14:44:23.473839+0000","last_peered":"2023-03-09T14:44:23.473839+0000","last_clean":"2023-03-09T14:44:23.473839+0000","last_became_active":"2023-03-09T14:44:19.118677+0000","last_became_peered":"2023-03-09T14:44:19.118677+0000","last_unstale":"2023-03-09T14:44:23.473839+0000","last_undegraded":"2023-03-09T14:44:23.473839+0000","last_fullsized":"2023-03-09T14:44:23.473839+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:06:28.740355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.906104+0000","last_change":"2023-03-09T14:44:23.484752+0000","last_active":"2023-03-09T14:44:23.906104+0000","last_peered":"2023-03-09T14:44:23.906104+0000","last_clean":"2023-03-09T14:44:23.906104+0000","last_became_active":"2023-03-09T14:44:23.484045+0000","last_became_peered":"2023-03-09T14:44:23.484045+0000","last_unstale":"2023-03-09T14:44:23.906104+0000","last_undegraded":"2023-03-09T14:44:23.906104+0000","last_fullsized":"2023-03-09T14:44:23.906104+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:58:39.704017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"82'3","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.480387+0000","last_change":"2023-03-09T14:44:18.665122+0000","last_active":"2023-03-09T14:44:23.480387+0000","last_peered":"2023-03-09T14:44:23.480387+0000","last_clean":"2023-03-09T14:44:23.480387+0000","last_became_active":"2023-03-09T14:44:18.664786+0000","last_became_peered":"2023-03-09T14:44:18.664786+0000","last_unstale":"2023-03-09T14:44:23.480387+0000","last_undegraded":"2023-03-09T14:44:23.480387+0000","last_fullsized":"2023-03-09T14:44:23.480387+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T20:29:20.919293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.476134+0000","last_change":"2023-03-09T14:44:19.108693+0000","last_active":"2023-03-09T14:44:23.476134+0000","last_peered":"2023-03-09T14:44:23.476134+0000","last_clean":"2023-03-09T14:44:23.476134+0000","last_became_active":"2023-03-09T14:44:19.108370+0000","last_became_peered":"2023-03-09T14:44:19.108370+0000","last_unstale":"2023-03-09T14:44:23.476134+0000","last_undegraded":"2023-03-09T14:44:23.476134+0000","last_fullsized":"2023-03-09T14:44:23.476134+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T16:30:58.726080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:24.249387+0000","last_change":"2023-03-09T14:44:23.488684+0000","last_active":"2023-03-09T14:44:24.249387+0000","last_peered":"2023-03-09T14:44:24.249387+0000","last_clean":"2023-03-09T14:44:24.249387+0000","last_became_active":"2023-03-09T14:44:23.488284+0000","last_became_peered":"2023-03-09T14:44:23.488284+0000","last_unstale":"2023-03-09T14:44:24.249387+0000","last_undegraded":"2023-03-09T14:44:24.249387+0000","last_fullsized":"2023-03-09T14:44:24.249387+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:56:41.323297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"87'18","reported_seq":38,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-03-09T14:44:39.963262+0000","last_change":"2023-03-09T14:44:18.661904+0000","last_active":"2023-03-09T14:44:39.963262+0000","last_peered":"2023-03-09T14:44:39.963262+0000","last_clean":"2023-03-09T14:44:39.963262+0000","last_became_active":"2023-03-09T14:44:18.661597+0000","last_became_peered":"2023-03-09T14:44:18.661597+0000","last_unstale":"2023-03-09T14:44:39.963262+0000","last_undegraded":"2023-03-09T14:44:39.963262+0000","last_fullsized":"2023-03-09T14:44:39.963262+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T14:57:04.978132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21095,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":6,"num_write":19,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.474881+0000","last_change":"2023-03-09T14:44:19.111671+0000","last_active":"2023-03-09T14:44:23.474881+0000","last_peered":"2023-03-09T14:44:23.474881+0000","last_clean":"2023-03-09T14:44:23.474881+0000","last_became_active":"2023-03-09T14:44:19.111172+0000","last_became_peered":"2023-03-09T14:44:19.111172+0000","last_unstale":"2023-03-09T14:44:23.474881+0000","last_undegraded":"2023-03-09T14:44:23.474881+0000","last_fullsized":"2023-03-09T14:44:23.474881+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T16:27:23.921853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-03-09T14:44:24.286526+0000","last_change":"2023-03-09T14:44:23.486338+0000","last_active":"2023-03-09T14:44:24.286526+0000","last_peered":"2023-03-09T14:44:24.286526+0000","last_clean":"2023-03-09T14:44:24.286526+0000","last_became_active":"2023-03-09T14:44:23.485729+0000","last_became_peered":"2023-03-09T14:44:23.485729+0000","last_unstale":"2023-03-09T14:44:24.286526+0000","last_undegraded":"2023-03-09T14:44:24.286526+0000","last_fullsized":"2023-03-09T14:44:24.286526+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:28:23.872633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"87'4","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-03-09T14:44:37.502627+0000","last_change":"2023-03-09T14:44:18.664942+0000","last_active":"2023-03-09T14:44:37.502627+0000","last_peered":"2023-03-09T14:44:37.502627+0000","last_clean":"2023-03-09T14:44:37.502627+0000","last_became_active":"2023-03-09T14:44:18.664007+0000","last_became_peered":"2023-03-09T14:44:18.664007+0000","last_unstale":"2023-03-09T14:44:37.502627+0000","last_undegraded":"2023-03-09T14:44:37.502627+0000","last_fullsized":"2023-03-09T14:44:37.502627+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:36:21.329388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.475377+0000","last_change":"2023-03-09T14:44:19.110140+0000","last_active":"2023-03-09T14:44:23.475377+0000","last_peered":"2023-03-09T14:44:23.475377+0000","last_clean":"2023-03-09T14:44:23.475377+0000","last_became_active":"2023-03-09T14:44:19.109811+0000","last_became_peered":"2023-03-09T14:44:19.109811+0000","last_unstale":"2023-03-09T14:44:23.475377+0000","last_undegraded":"2023-03-09T14:44:23.475377+0000","last_fullsized":"2023-03-09T14:44:23.475377+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T14:56:14.638662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":12,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.491669+0000","last_change":"2023-03-09T14:44:23.491318+0000","last_active":"2023-03-09T14:44:23.491669+0000","last_peered":"2023-03-09T14:44:23.491669+0000","last_clean":"2023-03-09T14:44:23.491669+0000","last_became_active":"2023-03-09T14:44:23.490915+0000","last_became_peered":"2023-03-09T14:44:23.490915+0000","last_unstale":"2023-03-09T14:44:23.491669+0000","last_undegraded":"2023-03-09T14:44:23.491669+0000","last_fullsized":"2023-03-09T14:44:23.491669+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:32:06.877573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.479568+0000","last_change":"2023-03-09T14:44:18.657323+0000","last_active":"2023-03-09T14:44:23.479568+0000","last_peered":"2023-03-09T14:44:23.479568+0000","last_clean":"2023-03-09T14:44:23.479568+0000","last_became_active":"2023-03-09T14:44:18.657017+0000","last_became_peered":"2023-03-09T14:44:18.657017+0000","last_unstale":"2023-03-09T14:44:23.479568+0000","last_undegraded":"2023-03-09T14:44:23.479568+0000","last_fullsized":"2023-03-09T14:44:23.479568+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:43:11.562723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.474017+0000","last_change":"2023-03-09T14:44:19.112194+0000","last_active":"2023-03-09T14:44:23.474017+0000","last_peered":"2023-03-09T14:44:23.474017+0000","last_clean":"2023-03-09T14:44:23.474017+0000","last_became_active":"2023-03-09T14:44:19.111924+0000","last_became_peered":"2023-03-09T14:44:19.111924+0000","last_unstale":"2023-03-09T14:44:23.474017+0000","last_undegraded":"2023-03-09T14:44:23.474017+0000","last_fullsized":"2023-03-09T14:44:23.474017+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T19:54:22.669848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:24.249416+0000","last_change":"2023-03-09T14:44:23.493993+0000","last_active":"2023-03-09T14:44:24.249416+0000","last_peered":"2023-03-09T14:44:24.249416+0000","last_clean":"2023-03-09T14:44:24.249416+0000","last_became_active":"2023-03-09T14:44:23.493535+0000","last_became_peered":"2023-03-09T14:44:23.493535+0000","last_unstale":"2023-03-09T14:44:24.249416+0000","last_undegraded":"2023-03-09T14:44:24.249416+0000","last_fullsized":"2023-03-09T14:44:24.249416+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T01:18:23.669790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.469908+0000","last_change":"2023-03-09T14:44:18.659914+0000","last_active":"2023-03-09T14:44:23.469908+0000","last_peered":"2023-03-09T14:44:23.469908+0000","last_clean":"2023-03-09T14:44:23.469908+0000","last_became_active":"2023-03-09T14:44:18.658835+0000","last_became_peered":"2023-03-09T14:44:18.658835+0000","last_unstale":"2023-03-09T14:44:23.469908+0000","last_undegraded":"2023-03-09T14:44:23.469908+0000","last_fullsized":"2023-03-09T14:44:23.469908+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T14:55:05.413260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.475687+0000","last_change":"2023-03-09T14:44:19.111058+0000","last_active":"2023-03-09T14:44:23.475687+0000","last_peered":"2023-03-09T14:44:23.475687+0000","last_clean":"2023-03-09T14:44:23.475687+0000","last_became_active":"2023-03-09T14:44:19.110760+0000","last_became_peered":"2023-03-09T14:44:19.110760+0000","last_unstale":"2023-03-09T14:44:23.475687+0000","last_undegraded":"2023-03-09T14:44:23.475687+0000","last_fullsized":"2023-03-09T14:44:23.475687+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:25:13.199416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-03-09T14:44:24.286025+0000","last_change":"2023-03-09T14:44:23.484710+0000","last_active":"2023-03-09T14:44:24.286025+0000","last_peered":"2023-03-09T14:44:24.286025+0000","last_clean":"2023-03-09T14:44:24.286025+0000","last_became_active":"2023-03-09T14:44:23.483928+0000","last_became_peered":"2023-03-09T14:44:23.483928+0000","last_unstale":"2023-03-09T14:44:24.286025+0000","last_undegraded":"2023-03-09T14:44:24.286025+0000","last_fullsized":"2023-03-09T14:44:24.286025+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T14:51:32.124111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"82'8","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.473728+0000","last_change":"2023-03-09T14:44:18.663692+0000","last_active":"2023-03-09T14:44:23.473728+0000","last_peered":"2023-03-09T14:44:23.473728+0000","last_clean":"2023-03-09T14:44:23.473728+0000","last_became_active":"2023-03-09T14:44:18.663400+0000","last_became_peered":"2023-03-09T14:44:18.663400+0000","last_unstale":"2023-03-09T14:44:23.473728+0000","last_undegraded":"2023-03-09T14:44:23.473728+0000","last_fullsized":"2023-03-09T14:44:23.473728+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:32:31.634315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.479630+0000","last_change":"2023-03-09T14:44:19.110819+0000","last_active":"2023-03-09T14:44:23.479630+0000","last_peered":"2023-03-09T14:44:23.479630+0000","last_clean":"2023-03-09T14:44:23.479630+0000","last_became_active":"2023-03-09T14:44:19.110542+0000","last_became_peered":"2023-03-09T14:44:19.110542+0000","last_unstale":"2023-03-09T14:44:23.479630+0000","last_undegraded":"2023-03-09T14:44:23.479630+0000","last_fullsized":"2023-03-09T14:44:23.479630+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:25:38.382685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-03-09T14:44:24.286231+0000","last_change":"2023-03-09T14:44:23.484614+0000","last_active":"2023-03-09T14:44:24.286231+0000","last_peered":"2023-03-09T14:44:24.286231+0000","last_clean":"2023-03-09T14:44:24.286231+0000","last_became_active":"2023-03-09T14:44:23.484061+0000","last_became_peered":"2023-03-09T14:44:23.484061+0000","last_unstale":"2023-03-09T14:44:24.286231+0000","last_undegraded":"2023-03-09T14:44:24.286231+0000","last_fullsized":"2023-03-09T14:44:24.286231+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T19:31:42.889122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"82'1","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.475017+0000","last_change":"2023-03-09T14:44:18.656056+0000","last_active":"2023-03-09T14:44:23.475017+0000","last_peered":"2023-03-09T14:44:23.475017+0000","last_clean":"2023-03-09T14:44:23.475017+0000","last_became_active":"2023-03-09T14:44:18.655709+0000","last_became_peered":"2023-03-09T14:44:18.655709+0000","last_unstale":"2023-03-09T14:44:23.475017+0000","last_undegraded":"2023-03-09T14:44:23.475017+0000","last_fullsized":"2023-03-09T14:44:23.475017+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T19:09:11.447249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.475132+0000","last_change":"2023-03-09T14:44:19.110292+0000","last_active":"2023-03-09T14:44:23.475132+0000","last_peered":"2023-03-09T14:44:23.475132+0000","last_clean":"2023-03-09T14:44:23.475132+0000","last_became_active":"2023-03-09T14:44:19.110006+0000","last_became_peered":"2023-03-09T14:44:19.110006+0000","last_unstale":"2023-03-09T14:44:23.475132+0000","last_undegraded":"2023-03-09T14:44:23.475132+0000","last_fullsized":"2023-03-09T14:44:23.475132+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:36:40.401486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-03-09T14:44:24.287375+0000","last_change":"2023-03-09T14:44:23.484004+0000","last_active":"2023-03-09T14:44:24.287375+0000","last_peered":"2023-03-09T14:44:24.287375+0000","last_clean":"2023-03-09T14:44:24.287375+0000","last_became_active":"2023-03-09T14:44:23.483581+0000","last_became_peered":"2023-03-09T14:44:23.483581+0000","last_unstale":"2023-03-09T14:44:24.287375+0000","last_undegraded":"2023-03-09T14:44:24.287375+0000","last_fullsized":"2023-03-09T14:44:24.287375+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:24:25.939838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.476558+0000","last_change":"2023-03-09T14:44:18.661856+0000","last_active":"2023-03-09T14:44:23.476558+0000","last_peered":"2023-03-09T14:44:23.476558+0000","last_clean":"2023-03-09T14:44:23.476558+0000","last_became_active":"2023-03-09T14:44:18.661500+0000","last_became_peered":"2023-03-09T14:44:18.661500+0000","last_unstale":"2023-03-09T14:44:23.476558+0000","last_undegraded":"2023-03-09T14:44:23.476558+0000","last_fullsized":"2023-03-09T14:44:23.476558+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:27:12.181159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.469525+0000","last_change":"2023-03-09T14:44:19.102058+0000","last_active":"2023-03-09T14:44:23.469525+0000","last_peered":"2023-03-09T14:44:23.469525+0000","last_clean":"2023-03-09T14:44:23.469525+0000","last_became_active":"2023-03-09T14:44:19.101762+0000","last_became_peered":"2023-03-09T14:44:19.101762+0000","last_unstale":"2023-03-09T14:44:23.469525+0000","last_undegraded":"2023-03-09T14:44:23.469525+0000","last_fullsized":"2023-03-09T14:44:23.469525+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:39:42.544086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-03-09T14:44:24.285308+0000","last_change":"2023-03-09T14:44:23.490237+0000","last_active":"2023-03-09T14:44:24.285308+0000","last_peered":"2023-03-09T14:44:24.285308+0000","last_clean":"2023-03-09T14:44:24.285308+0000","last_became_active":"2023-03-09T14:44:23.489844+0000","last_became_peered":"2023-03-09T14:44:23.489844+0000","last_unstale":"2023-03-09T14:44:24.285308+0000","last_undegraded":"2023-03-09T14:44:24.285308+0000","last_fullsized":"2023-03-09T14:44:24.285308+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T14:50:09.358698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.473792+0000","last_change":"2023-03-09T14:44:18.657684+0000","last_active":"2023-03-09T14:44:23.473792+0000","last_peered":"2023-03-09T14:44:23.473792+0000","last_clean":"2023-03-09T14:44:23.473792+0000","last_became_active":"2023-03-09T14:44:18.657295+0000","last_became_peered":"2023-03-09T14:44:18.657295+0000","last_unstale":"2023-03-09T14:44:23.473792+0000","last_undegraded":"2023-03-09T14:44:23.473792+0000","last_fullsized":"2023-03-09T14:44:23.473792+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:50:39.037653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.469107+0000","last_change":"2023-03-09T14:44:19.117584+0000","last_active":"2023-03-09T14:44:23.469107+0000","last_peered":"2023-03-09T14:44:23.469107+0000","last_clean":"2023-03-09T14:44:23.469107+0000","last_became_active":"2023-03-09T14:44:19.117262+0000","last_became_peered":"2023-03-09T14:44:19.117262+0000","last_unstale":"2023-03-09T14:44:23.469107+0000","last_undegraded":"2023-03-09T14:44:23.469107+0000","last_fullsized":"2023-03-09T14:44:23.469107+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T19:19:38.080048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-03-09T14:44:24.286054+0000","last_change":"2023-03-09T14:44:23.485769+0000","last_active":"2023-03-09T14:44:24.286054+0000","last_peered":"2023-03-09T14:44:24.286054+0000","last_clean":"2023-03-09T14:44:24.286054+0000","last_became_active":"2023-03-09T14:44:23.485317+0000","last_became_peered":"2023-03-09T14:44:23.485317+0000","last_unstale":"2023-03-09T14:44:24.286054+0000","last_undegraded":"2023-03-09T14:44:24.286054+0000","last_fullsized":"2023-03-09T14:44:24.286054+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:58:35.140512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.474918+0000","last_change":"2023-03-09T14:44:19.112255+0000","last_active":"2023-03-09T14:44:23.474918+0000","last_peered":"2023-03-09T14:44:23.474918+0000","last_clean":"2023-03-09T14:44:23.474918+0000","last_became_active":"2023-03-09T14:44:19.111988+0000","last_became_peered":"2023-03-09T14:44:19.111988+0000","last_unstale":"2023-03-09T14:44:23.474918+0000","last_undegraded":"2023-03-09T14:44:23.474918+0000","last_fullsized":"2023-03-09T14:44:23.474918+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:55:50.423701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"87'4","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-03-09T14:44:38.524897+0000","last_change":"2023-03-09T14:44:18.665291+0000","last_active":"2023-03-09T14:44:38.524897+0000","last_peered":"2023-03-09T14:44:38.524897+0000","last_clean":"2023-03-09T14:44:38.524897+0000","last_became_active":"2023-03-09T14:44:18.664950+0000","last_became_peered":"2023-03-09T14:44:18.664950+0000","last_unstale":"2023-03-09T14:44:38.524897+0000","last_undegraded":"2023-03-09T14:44:38.524897+0000","last_fullsized":"2023-03-09T14:44:38.524897+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:40:58.619717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-03-09T14:44:24.285763+0000","last_change":"2023-03-09T14:44:23.480872+0000","last_active":"2023-03-09T14:44:24.285763+0000","last_peered":"2023-03-09T14:44:24.285763+0000","last_clean":"2023-03-09T14:44:24.285763+0000","last_became_active":"2023-03-09T14:44:23.480495+0000","last_became_peered":"2023-03-09T14:44:23.480495+0000","last_unstale":"2023-03-09T14:44:24.285763+0000","last_undegraded":"2023-03-09T14:44:24.285763+0000","last_fullsized":"2023-03-09T14:44:24.285763+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:19:01.393034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.469682+0000","last_change":"2023-03-09T14:44:19.102833+0000","last_active":"2023-03-09T14:44:23.469682+0000","last_peered":"2023-03-09T14:44:23.469682+0000","last_clean":"2023-03-09T14:44:23.469682+0000","last_became_active":"2023-03-09T14:44:19.102484+0000","last_became_peered":"2023-03-09T14:44:19.102484+0000","last_unstale":"2023-03-09T14:44:23.469682+0000","last_undegraded":"2023-03-09T14:44:23.469682+0000","last_fullsized":"2023-03-09T14:44:23.469682+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:14:50.002907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.470135+0000","last_change":"2023-03-09T14:44:18.661603+0000","last_active":"2023-03-09T14:44:23.470135+0000","last_peered":"2023-03-09T14:44:23.470135+0000","last_clean":"2023-03-09T14:44:23.470135+0000","last_became_active":"2023-03-09T14:44:18.661275+0000","last_became_peered":"2023-03-09T14:44:18.661275+0000","last_unstale":"2023-03-09T14:44:23.470135+0000","last_undegraded":"2023-03-09T14:44:23.470135+0000","last_fullsized":"2023-03-09T14:44:23.470135+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:20:31.873778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":376,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.475580+0000","last_change":"2023-03-09T14:43:19.317554+0000","last_active":"2023-03-09T14:44:23.475580+0000","last_peered":"2023-03-09T14:44:23.475580+0000","last_clean":"2023-03-09T14:44:23.475580+0000","last_became_active":"2023-03-09T14:43:19.316912+0000","last_became_peered":"2023-03-09T14:43:19.316912+0000","last_unstale":"2023-03-09T14:44:23.475580+0000","last_undegraded":"2023-03-09T14:44:23.475580+0000","last_fullsized":"2023-03-09T14:44:23.475580+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:40:23.197173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:40:23.197173+0000","last_clean_scrub_stamp":"2023-03-09T14:40:23.197173+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T20:21:17.143331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-03-09T14:44:24.286143+0000","last_change":"2023-03-09T14:44:23.487738+0000","last_active":"2023-03-09T14:44:24.286143+0000","last_peered":"2023-03-09T14:44:24.286143+0000","last_clean":"2023-03-09T14:44:24.286143+0000","last_became_active":"2023-03-09T14:44:23.487193+0000","last_became_peered":"2023-03-09T14:44:23.487193+0000","last_unstale":"2023-03-09T14:44:24.286143+0000","last_undegraded":"2023-03-09T14:44:24.286143+0000","last_fullsized":"2023-03-09T14:44:24.286143+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:48:08.789071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.476422+0000","last_change":"2023-03-09T14:44:19.112576+0000","last_active":"2023-03-09T14:44:23.476422+0000","last_peered":"2023-03-09T14:44:23.476422+0000","last_clean":"2023-03-09T14:44:23.476422+0000","last_became_active":"2023-03-09T14:44:19.112231+0000","last_became_peered":"2023-03-09T14:44:19.112231+0000","last_unstale":"2023-03-09T14:44:23.476422+0000","last_undegraded":"2023-03-09T14:44:23.476422+0000","last_fullsized":"2023-03-09T14:44:23.476422+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:25:21.918088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.475646+0000","last_change":"2023-03-09T14:44:18.655478+0000","last_active":"2023-03-09T14:44:23.475646+0000","last_peered":"2023-03-09T14:44:23.475646+0000","last_clean":"2023-03-09T14:44:23.475646+0000","last_became_active":"2023-03-09T14:44:18.655180+0000","last_became_peered":"2023-03-09T14:44:18.655180+0000","last_unstale":"2023-03-09T14:44:23.475646+0000","last_undegraded":"2023-03-09T14:44:23.475646+0000","last_fullsized":"2023-03-09T14:44:23.475646+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:42:47.926376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.488348+0000","last_change":"2023-03-09T14:44:23.487052+0000","last_active":"2023-03-09T14:44:23.488348+0000","last_peered":"2023-03-09T14:44:23.488348+0000","last_clean":"2023-03-09T14:44:23.488348+0000","last_became_active":"2023-03-09T14:44:23.486603+0000","last_became_peered":"2023-03-09T14:44:23.486603+0000","last_unstale":"2023-03-09T14:44:23.488348+0000","last_undegraded":"2023-03-09T14:44:23.488348+0000","last_fullsized":"2023-03-09T14:44:23.488348+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:28:25.375052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.474962+0000","last_change":"2023-03-09T14:44:18.666400+0000","last_active":"2023-03-09T14:44:23.474962+0000","last_peered":"2023-03-09T14:44:23.474962+0000","last_clean":"2023-03-09T14:44:23.474962+0000","last_became_active":"2023-03-09T14:44:18.666045+0000","last_became_peered":"2023-03-09T14:44:18.666045+0000","last_unstale":"2023-03-09T14:44:23.474962+0000","last_undegraded":"2023-03-09T14:44:23.474962+0000","last_fullsized":"2023-03-09T14:44:23.474962+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:30:48.640124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.476223+0000","last_change":"2023-03-09T14:44:19.103657+0000","last_active":"2023-03-09T14:44:23.476223+0000","last_peered":"2023-03-09T14:44:23.476223+0000","last_clean":"2023-03-09T14:44:23.476223+0000","last_became_active":"2023-03-09T14:44:19.103364+0000","last_became_peered":"2023-03-09T14:44:19.103364+0000","last_unstale":"2023-03-09T14:44:23.476223+0000","last_undegraded":"2023-03-09T14:44:23.476223+0000","last_fullsized":"2023-03-09T14:44:23.476223+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:23:23.738554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.904079+0000","last_change":"2023-03-09T14:44:23.483487+0000","last_active":"2023-03-09T14:44:23.904079+0000","last_peered":"2023-03-09T14:44:23.904079+0000","last_clean":"2023-03-09T14:44:23.904079+0000","last_became_active":"2023-03-09T14:44:23.483050+0000","last_became_peered":"2023-03-09T14:44:23.483050+0000","last_unstale":"2023-03-09T14:44:23.904079+0000","last_undegraded":"2023-03-09T14:44:23.904079+0000","last_fullsized":"2023-03-09T14:44:23.904079+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:38:34.129355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"87'3","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-03-09T14:44:38.524026+0000","last_change":"2023-03-09T14:44:18.661944+0000","last_active":"2023-03-09T14:44:38.524026+0000","last_peered":"2023-03-09T14:44:38.524026+0000","last_clean":"2023-03-09T14:44:38.524026+0000","last_became_active":"2023-03-09T14:44:18.661575+0000","last_became_peered":"2023-03-09T14:44:18.661575+0000","last_unstale":"2023-03-09T14:44:38.524026+0000","last_undegraded":"2023-03-09T14:44:38.524026+0000","last_fullsized":"2023-03-09T14:44:38.524026+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T01:12:53.637866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.471841+0000","last_change":"2023-03-09T14:44:19.103807+0000","last_active":"2023-03-09T14:44:23.471841+0000","last_peered":"2023-03-09T14:44:23.471841+0000","last_clean":"2023-03-09T14:44:23.471841+0000","last_became_active":"2023-03-09T14:44:19.103480+0000","last_became_peered":"2023-03-09T14:44:19.103480+0000","last_unstale":"2023-03-09T14:44:23.471841+0000","last_undegraded":"2023-03-09T14:44:23.471841+0000","last_fullsized":"2023-03-09T14:44:23.471841+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:29:54.629265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.487495+0000","last_change":"2023-03-09T14:44:23.486789+0000","last_active":"2023-03-09T14:44:23.487495+0000","last_peered":"2023-03-09T14:44:23.487495+0000","last_clean":"2023-03-09T14:44:23.487495+0000","last_became_active":"2023-03-09T14:44:23.486367+0000","last_became_peered":"2023-03-09T14:44:23.486367+0000","last_unstale":"2023-03-09T14:44:23.487495+0000","last_undegraded":"2023-03-09T14:44:23.487495+0000","last_fullsized":"2023-03-09T14:44:23.487495+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:13:13.081388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"82'2","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.470707+0000","last_change":"2023-03-09T14:44:18.659492+0000","last_active":"2023-03-09T14:44:23.470707+0000","last_peered":"2023-03-09T14:44:23.470707+0000","last_clean":"2023-03-09T14:44:23.470707+0000","last_became_active":"2023-03-09T14:44:18.659187+0000","last_became_peered":"2023-03-09T14:44:18.659187+0000","last_unstale":"2023-03-09T14:44:23.470707+0000","last_undegraded":"2023-03-09T14:44:23.470707+0000","last_fullsized":"2023-03-09T14:44:23.470707+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:44:31.342785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.475564+0000","last_change":"2023-03-09T14:44:19.112841+0000","last_active":"2023-03-09T14:44:23.475564+0000","last_peered":"2023-03-09T14:44:23.475564+0000","last_clean":"2023-03-09T14:44:23.475564+0000","last_became_active":"2023-03-09T14:44:19.112515+0000","last_became_peered":"2023-03-09T14:44:19.112515+0000","last_unstale":"2023-03-09T14:44:23.475564+0000","last_undegraded":"2023-03-09T14:44:23.475564+0000","last_fullsized":"2023-03-09T14:44:23.475564+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T19:41:04.065529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.905391+0000","last_change":"2023-03-09T14:44:23.481992+0000","last_active":"2023-03-09T14:44:23.905391+0000","last_peered":"2023-03-09T14:44:23.905391+0000","last_clean":"2023-03-09T14:44:23.905391+0000","last_became_active":"2023-03-09T14:44:23.481508+0000","last_became_peered":"2023-03-09T14:44:23.481508+0000","last_unstale":"2023-03-09T14:44:23.905391+0000","last_undegraded":"2023-03-09T14:44:23.905391+0000","last_fullsized":"2023-03-09T14:44:23.905391+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:46:48.683402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"87'2","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-03-09T14:44:37.501019+0000","last_change":"2023-03-09T14:44:18.663133+0000","last_active":"2023-03-09T14:44:37.501019+0000","last_peered":"2023-03-09T14:44:37.501019+0000","last_clean":"2023-03-09T14:44:37.501019+0000","last_became_active":"2023-03-09T14:44:18.662809+0000","last_became_peered":"2023-03-09T14:44:18.662809+0000","last_unstale":"2023-03-09T14:44:37.501019+0000","last_undegraded":"2023-03-09T14:44:37.501019+0000","last_fullsized":"2023-03-09T14:44:37.501019+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:32:16.180642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.473842+0000","last_change":"2023-03-09T14:44:19.108566+0000","last_active":"2023-03-09T14:44:23.473842+0000","last_peered":"2023-03-09T14:44:23.473842+0000","last_clean":"2023-03-09T14:44:23.473842+0000","last_became_active":"2023-03-09T14:44:19.108217+0000","last_became_peered":"2023-03-09T14:44:19.108217+0000","last_unstale":"2023-03-09T14:44:23.473842+0000","last_undegraded":"2023-03-09T14:44:23.473842+0000","last_fullsized":"2023-03-09T14:44:23.473842+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:03:28.434395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:24.249427+0000","last_change":"2023-03-09T14:44:23.488640+0000","last_active":"2023-03-09T14:44:24.249427+0000","last_peered":"2023-03-09T14:44:24.249427+0000","last_clean":"2023-03-09T14:44:24.249427+0000","last_became_active":"2023-03-09T14:44:23.488178+0000","last_became_peered":"2023-03-09T14:44:23.488178+0000","last_unstale":"2023-03-09T14:44:24.249427+0000","last_undegraded":"2023-03-09T14:44:24.249427+0000","last_fullsized":"2023-03-09T14:44:24.249427+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:33:49.908974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"87'3","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-03-09T14:44:37.502373+0000","last_change":"2023-03-09T14:44:18.662925+0000","last_active":"2023-03-09T14:44:37.502373+0000","last_peered":"2023-03-09T14:44:37.502373+0000","last_clean":"2023-03-09T14:44:37.502373+0000","last_became_active":"2023-03-09T14:44:18.662354+0000","last_became_peered":"2023-03-09T14:44:18.662354+0000","last_unstale":"2023-03-09T14:44:37.502373+0000","last_undegraded":"2023-03-09T14:44:37.502373+0000","last_fullsized":"2023-03-09T14:44:37.502373+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T01:00:36.602938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.473168+0000","last_change":"2023-03-09T14:44:19.106465+0000","last_active":"2023-03-09T14:44:23.473168+0000","last_peered":"2023-03-09T14:44:23.473168+0000","last_clean":"2023-03-09T14:44:23.473168+0000","last_became_active":"2023-03-09T14:44:19.106001+0000","last_became_peered":"2023-03-09T14:44:19.106001+0000","last_unstale":"2023-03-09T14:44:23.473168+0000","last_undegraded":"2023-03-09T14:44:23.473168+0000","last_fullsized":"2023-03-09T14:44:23.473168+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:22:09.149291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.905541+0000","last_change":"2023-03-09T14:44:23.481512+0000","last_active":"2023-03-09T14:44:23.905541+0000","last_peered":"2023-03-09T14:44:23.905541+0000","last_clean":"2023-03-09T14:44:23.905541+0000","last_became_active":"2023-03-09T14:44:23.480370+0000","last_became_peered":"2023-03-09T14:44:23.480370+0000","last_unstale":"2023-03-09T14:44:23.905541+0000","last_undegraded":"2023-03-09T14:44:23.905541+0000","last_fullsized":"2023-03-09T14:44:23.905541+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:29:16.881539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.474195+0000","last_change":"2023-03-09T14:44:18.664744+0000","last_active":"2023-03-09T14:44:23.474195+0000","last_peered":"2023-03-09T14:44:23.474195+0000","last_clean":"2023-03-09T14:44:23.474195+0000","last_became_active":"2023-03-09T14:44:18.664443+0000","last_became_peered":"2023-03-09T14:44:18.664443+0000","last_unstale":"2023-03-09T14:44:23.474195+0000","last_undegraded":"2023-03-09T14:44:23.474195+0000","last_fullsized":"2023-03-09T14:44:23.474195+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:11:13.836270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.471063+0000","last_change":"2023-03-09T14:44:19.115102+0000","last_active":"2023-03-09T14:44:23.471063+0000","last_peered":"2023-03-09T14:44:23.471063+0000","last_clean":"2023-03-09T14:44:23.471063+0000","last_became_active":"2023-03-09T14:44:19.114565+0000","last_became_peered":"2023-03-09T14:44:19.114565+0000","last_unstale":"2023-03-09T14:44:23.471063+0000","last_undegraded":"2023-03-09T14:44:23.471063+0000","last_fullsized":"2023-03-09T14:44:23.471063+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:39:11.930231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.905161+0000","last_change":"2023-03-09T14:44:23.482500+0000","last_active":"2023-03-09T14:44:23.905161+0000","last_peered":"2023-03-09T14:44:23.905161+0000","last_clean":"2023-03-09T14:44:23.905161+0000","last_became_active":"2023-03-09T14:44:23.482217+0000","last_became_peered":"2023-03-09T14:44:23.482217+0000","last_unstale":"2023-03-09T14:44:23.905161+0000","last_undegraded":"2023-03-09T14:44:23.905161+0000","last_fullsized":"2023-03-09T14:44:23.905161+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T14:52:44.275037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"82'2","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.473997+0000","last_change":"2023-03-09T14:44:18.663824+0000","last_active":"2023-03-09T14:44:23.473997+0000","last_peered":"2023-03-09T14:44:23.473997+0000","last_clean":"2023-03-09T14:44:23.473997+0000","last_became_active":"2023-03-09T14:44:18.663379+0000","last_became_peered":"2023-03-09T14:44:18.663379+0000","last_unstale":"2023-03-09T14:44:23.473997+0000","last_undegraded":"2023-03-09T14:44:23.473997+0000","last_fullsized":"2023-03-09T14:44:23.473997+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:03:04.031061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.473902+0000","last_change":"2023-03-09T14:44:19.105765+0000","last_active":"2023-03-09T14:44:23.473902+0000","last_peered":"2023-03-09T14:44:23.473902+0000","last_clean":"2023-03-09T14:44:23.473902+0000","last_became_active":"2023-03-09T14:44:19.105430+0000","last_became_peered":"2023-03-09T14:44:19.105430+0000","last_unstale":"2023-03-09T14:44:23.473902+0000","last_undegraded":"2023-03-09T14:44:23.473902+0000","last_fullsized":"2023-03-09T14:44:23.473902+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T20:31:40.106798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-03-09T14:44:24.284816+0000","last_change":"2023-03-09T14:44:23.482079+0000","last_active":"2023-03-09T14:44:24.284816+0000","last_peered":"2023-03-09T14:44:24.284816+0000","last_clean":"2023-03-09T14:44:24.284816+0000","last_became_active":"2023-03-09T14:44:23.481620+0000","last_became_peered":"2023-03-09T14:44:23.481620+0000","last_unstale":"2023-03-09T14:44:24.284816+0000","last_undegraded":"2023-03-09T14:44:24.284816+0000","last_fullsized":"2023-03-09T14:44:24.284816+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:47:53.589111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.479406+0000","last_change":"2023-03-09T14:44:18.667558+0000","last_active":"2023-03-09T14:44:23.479406+0000","last_peered":"2023-03-09T14:44:23.479406+0000","last_clean":"2023-03-09T14:44:23.479406+0000","last_became_active":"2023-03-09T14:44:18.667260+0000","last_became_peered":"2023-03-09T14:44:18.667260+0000","last_unstale":"2023-03-09T14:44:23.479406+0000","last_undegraded":"2023-03-09T14:44:23.479406+0000","last_fullsized":"2023-03-09T14:44:23.479406+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T16:58:53.434611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.475024+0000","last_change":"2023-03-09T14:44:19.105947+0000","last_active":"2023-03-09T14:44:23.475024+0000","last_peered":"2023-03-09T14:44:23.475024+0000","last_clean":"2023-03-09T14:44:23.475024+0000","last_became_active":"2023-03-09T14:44:19.105669+0000","last_became_peered":"2023-03-09T14:44:19.105669+0000","last_unstale":"2023-03-09T14:44:23.475024+0000","last_undegraded":"2023-03-09T14:44:23.475024+0000","last_fullsized":"2023-03-09T14:44:23.475024+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:29:58.615481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-03-09T14:44:24.313456+0000","last_change":"2023-03-09T14:44:23.483237+0000","last_active":"2023-03-09T14:44:24.313456+0000","last_peered":"2023-03-09T14:44:24.313456+0000","last_clean":"2023-03-09T14:44:24.313456+0000","last_became_active":"2023-03-09T14:44:23.482831+0000","last_became_peered":"2023-03-09T14:44:23.482831+0000","last_unstale":"2023-03-09T14:44:24.313456+0000","last_undegraded":"2023-03-09T14:44:24.313456+0000","last_fullsized":"2023-03-09T14:44:24.313456+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:36:40.775581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"82'1","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.480598+0000","last_change":"2023-03-09T14:44:18.660954+0000","last_active":"2023-03-09T14:44:23.480598+0000","last_peered":"2023-03-09T14:44:23.480598+0000","last_clean":"2023-03-09T14:44:23.480598+0000","last_became_active":"2023-03-09T14:44:18.660673+0000","last_became_peered":"2023-03-09T14:44:18.660673+0000","last_unstale":"2023-03-09T14:44:23.480598+0000","last_undegraded":"2023-03-09T14:44:23.480598+0000","last_fullsized":"2023-03-09T14:44:23.480598+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T19:37:58.271196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.474234+0000","last_change":"2023-03-09T14:44:19.107146+0000","last_active":"2023-03-09T14:44:23.474234+0000","last_peered":"2023-03-09T14:44:23.474234+0000","last_clean":"2023-03-09T14:44:23.474234+0000","last_became_active":"2023-03-09T14:44:19.106834+0000","last_became_peered":"2023-03-09T14:44:19.106834+0000","last_unstale":"2023-03-09T14:44:23.474234+0000","last_undegraded":"2023-03-09T14:44:23.474234+0000","last_fullsized":"2023-03-09T14:44:23.474234+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T20:15:20.142449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-03-09T14:44:24.285655+0000","last_change":"2023-03-09T14:44:23.482811+0000","last_active":"2023-03-09T14:44:24.285655+0000","last_peered":"2023-03-09T14:44:24.285655+0000","last_clean":"2023-03-09T14:44:24.285655+0000","last_became_active":"2023-03-09T14:44:23.482371+0000","last_became_peered":"2023-03-09T14:44:23.482371+0000","last_unstale":"2023-03-09T14:44:24.285655+0000","last_undegraded":"2023-03-09T14:44:24.285655+0000","last_fullsized":"2023-03-09T14:44:24.285655+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T01:33:14.442810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"82'2","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.477425+0000","last_change":"2023-03-09T14:44:18.661569+0000","last_active":"2023-03-09T14:44:23.477425+0000","last_peered":"2023-03-09T14:44:23.477425+0000","last_clean":"2023-03-09T14:44:23.477425+0000","last_became_active":"2023-03-09T14:44:18.661266+0000","last_became_peered":"2023-03-09T14:44:18.661266+0000","last_unstale":"2023-03-09T14:44:23.477425+0000","last_undegraded":"2023-03-09T14:44:23.477425+0000","last_fullsized":"2023-03-09T14:44:23.477425+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:47:43.066180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.474622+0000","last_change":"2023-03-09T14:44:19.106611+0000","last_active":"2023-03-09T14:44:23.474622+0000","last_peered":"2023-03-09T14:44:23.474622+0000","last_clean":"2023-03-09T14:44:23.474622+0000","last_became_active":"2023-03-09T14:44:19.106324+0000","last_became_peered":"2023-03-09T14:44:19.106324+0000","last_unstale":"2023-03-09T14:44:23.474622+0000","last_undegraded":"2023-03-09T14:44:23.474622+0000","last_fullsized":"2023-03-09T14:44:23.474622+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:50:46.561502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.472996+0000","last_change":"2023-03-09T14:44:19.106292+0000","last_active":"2023-03-09T14:44:23.472996+0000","last_peered":"2023-03-09T14:44:23.472996+0000","last_clean":"2023-03-09T14:44:23.472996+0000","last_became_active":"2023-03-09T14:44:19.105607+0000","last_became_peered":"2023-03-09T14:44:19.105607+0000","last_unstale":"2023-03-09T14:44:23.472996+0000","last_undegraded":"2023-03-09T14:44:23.472996+0000","last_fullsized":"2023-03-09T14:44:23.472996+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T19:11:28.125433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.479822+0000","last_change":"2023-03-09T14:44:19.103423+0000","last_active":"2023-03-09T14:44:23.479822+0000","last_peered":"2023-03-09T14:44:23.479822+0000","last_clean":"2023-03-09T14:44:23.479822+0000","last_became_active":"2023-03-09T14:44:19.102918+0000","last_became_peered":"2023-03-09T14:44:19.102918+0000","last_unstale":"2023-03-09T14:44:23.479822+0000","last_undegraded":"2023-03-09T14:44:23.479822+0000","last_fullsized":"2023-03-09T14:44:23.479822+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:47:52.921642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.474107+0000","last_change":"2023-03-09T14:44:19.113511+0000","last_active":"2023-03-09T14:44:23.474107+0000","last_peered":"2023-03-09T14:44:23.474107+0000","last_clean":"2023-03-09T14:44:23.474107+0000","last_became_active":"2023-03-09T14:44:19.113224+0000","last_became_peered":"2023-03-09T14:44:19.113224+0000","last_unstale":"2023-03-09T14:44:23.474107+0000","last_undegraded":"2023-03-09T14:44:23.474107+0000","last_fullsized":"2023-03-09T14:44:23.474107+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T02:03:28.069616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.475109+0000","last_change":"2023-03-09T14:44:19.116174+0000","last_active":"2023-03-09T14:44:23.475109+0000","last_peered":"2023-03-09T14:44:23.475109+0000","last_clean":"2023-03-09T14:44:23.475109+0000","last_became_active":"2023-03-09T14:44:19.115813+0000","last_became_peered":"2023-03-09T14:44:19.115813+0000","last_unstale":"2023-03-09T14:44:23.475109+0000","last_undegraded":"2023-03-09T14:44:23.475109+0000","last_fullsized":"2023-03-09T14:44:23.475109+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:28:49.573771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.476310+0000","last_change":"2023-03-09T14:44:19.115287+0000","last_active":"2023-03-09T14:44:23.476310+0000","last_peered":"2023-03-09T14:44:23.476310+0000","last_clean":"2023-03-09T14:44:23.476310+0000","last_became_active":"2023-03-09T14:44:19.113458+0000","last_became_peered":"2023-03-09T14:44:19.113458+0000","last_unstale":"2023-03-09T14:44:23.476310+0000","last_undegraded":"2023-03-09T14:44:23.476310+0000","last_fullsized":"2023-03-09T14:44:23.476310+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:43:26.583237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.476398+0000","last_change":"2023-03-09T14:44:19.107308+0000","last_active":"2023-03-09T14:44:23.476398+0000","last_peered":"2023-03-09T14:44:23.476398+0000","last_clean":"2023-03-09T14:44:23.476398+0000","last_became_active":"2023-03-09T14:44:19.107005+0000","last_became_peered":"2023-03-09T14:44:19.107005+0000","last_unstale":"2023-03-09T14:44:23.476398+0000","last_undegraded":"2023-03-09T14:44:23.476398+0000","last_fullsized":"2023-03-09T14:44:23.476398+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:48:34.886664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.475838+0000","last_change":"2023-03-09T14:44:19.098454+0000","last_active":"2023-03-09T14:44:23.475838+0000","last_peered":"2023-03-09T14:44:23.475838+0000","last_clean":"2023-03-09T14:44:23.475838+0000","last_became_active":"2023-03-09T14:44:19.098146+0000","last_became_peered":"2023-03-09T14:44:19.098146+0000","last_unstale":"2023-03-09T14:44:23.475838+0000","last_undegraded":"2023-03-09T14:44:23.475838+0000","last_fullsized":"2023-03-09T14:44:23.475838+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:11:40.475674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.471936+0000","last_change":"2023-03-09T14:44:19.095947+0000","last_active":"2023-03-09T14:44:23.471936+0000","last_peered":"2023-03-09T14:44:23.471936+0000","last_clean":"2023-03-09T14:44:23.471936+0000","last_became_active":"2023-03-09T14:44:19.095640+0000","last_became_peered":"2023-03-09T14:44:19.095640+0000","last_unstale":"2023-03-09T14:44:23.471936+0000","last_undegraded":"2023-03-09T14:44:23.471936+0000","last_fullsized":"2023-03-09T14:44:23.471936+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T02:04:35.384016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.474791+0000","last_change":"2023-03-09T14:44:19.113927+0000","last_active":"2023-03-09T14:44:23.474791+0000","last_peered":"2023-03-09T14:44:23.474791+0000","last_clean":"2023-03-09T14:44:23.474791+0000","last_became_active":"2023-03-09T14:44:19.113163+0000","last_became_peered":"2023-03-09T14:44:19.113163+0000","last_unstale":"2023-03-09T14:44:23.474791+0000","last_undegraded":"2023-03-09T14:44:23.474791+0000","last_fullsized":"2023-03-09T14:44:23.474791+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T01:35:35.801317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.470072+0000","last_change":"2023-03-09T14:44:19.098415+0000","last_active":"2023-03-09T14:44:23.470072+0000","last_peered":"2023-03-09T14:44:23.470072+0000","last_clean":"2023-03-09T14:44:23.470072+0000","last_became_active":"2023-03-09T14:44:19.097600+0000","last_became_peered":"2023-03-09T14:44:19.097600+0000","last_unstale":"2023-03-09T14:44:23.470072+0000","last_undegraded":"2023-03-09T14:44:23.470072+0000","last_fullsized":"2023-03-09T14:44:23.470072+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:15:08.552402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.475985+0000","last_change":"2023-03-09T14:44:19.106386+0000","last_active":"2023-03-09T14:44:23.475985+0000","last_peered":"2023-03-09T14:44:23.475985+0000","last_clean":"2023-03-09T14:44:23.475985+0000","last_became_active":"2023-03-09T14:44:19.106112+0000","last_became_peered":"2023-03-09T14:44:19.106112+0000","last_unstale":"2023-03-09T14:44:23.475985+0000","last_undegraded":"2023-03-09T14:44:23.475985+0000","last_fullsized":"2023-03-09T14:44:23.475985+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T20:27:50.992168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.477821+0000","last_change":"2023-03-09T14:44:19.114534+0000","last_active":"2023-03-09T14:44:23.477821+0000","last_peered":"2023-03-09T14:44:23.477821+0000","last_clean":"2023-03-09T14:44:23.477821+0000","last_became_active":"2023-03-09T14:44:19.114226+0000","last_became_peered":"2023-03-09T14:44:19.114226+0000","last_unstale":"2023-03-09T14:44:23.477821+0000","last_undegraded":"2023-03-09T14:44:23.477821+0000","last_fullsized":"2023-03-09T14:44:23.477821+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:22:38.453116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.474647+0000","last_change":"2023-03-09T14:44:19.099195+0000","last_active":"2023-03-09T14:44:23.474647+0000","last_peered":"2023-03-09T14:44:23.474647+0000","last_clean":"2023-03-09T14:44:23.474647+0000","last_became_active":"2023-03-09T14:44:19.098892+0000","last_became_peered":"2023-03-09T14:44:19.098892+0000","last_unstale":"2023-03-09T14:44:23.474647+0000","last_undegraded":"2023-03-09T14:44:23.474647+0000","last_fullsized":"2023-03-09T14:44:23.474647+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:29:24.858200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":22147,"num_objects":36,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5,"num_read_kb":7,"num_write":91,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":25,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":212992,"data_stored":52572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77,"ondisk_log_size":77,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":78,"seq":335007449103,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6064,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739088,"statfs":{"total":95995035648,"available":95988826112,"internally_reserved":0,"allocated":958464,"data_stored":467097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.755}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40300000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46400000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69899999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48399999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45000000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79400000000000004}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72699999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78100000000000003}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68899999999999995}]}]},{"osd":10,"up_from":70,"seq":300647710738,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6940,"kb_used_data":1556,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738212,"statfs":{"total":95995035648,"available":95987929088,"internally_reserved":0,"allocated":1593344,"data_stored":1062490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60399999999999998}]},{"osd":1,"last update":"Thu Mar 9 14:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.34899999999999998}]},{"osd":2,"last update":"Thu Mar 9 14:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.51900000000000002}]},{"osd":3,"last update":"Thu Mar 9 14:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.2769999999999999,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.505}]},{"osd":4,"last update":"Thu Mar 9 14:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.1719999999999999,"5min":1.1719999999999999,"15min":1.1719999999999999},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.39800000000000002}]},{"osd":5,"last update":"Thu Mar 9 14:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.63},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.56100000000000005}]},{"osd":6,"last update":"Thu Mar 9 14:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.57399999999999995}]},{"osd":7,"last update":"Thu Mar 9 14:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.327,"5min":1.327,"15min":1.327},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.048,"5min":1.048,"15min":1.048},"last":0.53800000000000003}]},{"osd":8,"last update":"Thu Mar 9 14:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.246,"5min":1.246,"15min":1.246},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.67800000000000005}]},{"osd":9,"last update":"Thu Mar 9 14:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.298,"5min":1.298,"15min":1.298},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.4630000000000001,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":0.70399999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72699999999999998}]}]},{"osd":9,"up_from":65,"seq":279172874263,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6276,"kb_used_data":956,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93738876,"statfs":{"total":95995035648,"available":95988609024,"internally_reserved":0,"allocated":978944,"data_stored":467899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:43:55 2023","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.36199999999999999}]},{"osd":1,"last update":"Thu Mar 9 14:43:55 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.59499999999999997}]},{"osd":2,"last update":"Thu Mar 9 14:43:55 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.48999999999999999}]},{"osd":3,"last update":"Thu Mar 9 14:43:55 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":0.33700000000000002}]},{"osd":4,"last update":"Thu Mar 9 14:43:55 2023","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.47899999999999998}]},{"osd":5,"last update":"Thu Mar 9 14:43:55 2023","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":0.69399999999999995}]},{"osd":6,"last update":"Thu Mar 9 14:43:55 2023","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.623}]},{"osd":7,"last update":"Thu Mar 9 14:43:55 2023","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.627}]},{"osd":8,"last update":"Thu Mar 9 14:43:55 2023","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.133,"5min":1.133,"15min":1.133},"last":0.751},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.61699999999999999}]},{"osd":10,"last update":"Thu Mar 9 14:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.57399999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66700000000000004}]}]},{"osd":8,"up_from":57,"seq":244813135899,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6388,"kb_used_data":940,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738764,"statfs":{"total":95995035648,"available":95988494336,"internally_reserved":0,"allocated":962560,"data_stored":467354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.33300000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.69199999999999995,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54900000000000004,"15min":0.55100000000000005},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.83799999999999997,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.59399999999999997}]},{"osd":1,"last update":"Thu Mar 9 14:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.54700000000000004,"15min":0.53900000000000003},"min":{"1min":0.376,"5min":0.313,"15min":0.313},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.91400000000000003},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.56100000000000005,"15min":0.55900000000000005},"min":{"1min":0.34399999999999997,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.70499999999999996}]},{"osd":2,"last update":"Thu Mar 9 14:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.42199999999999999,"15min":0.41599999999999998},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.432,"15min":0.42699999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.66900000000000004}]},{"osd":3,"last update":"Thu Mar 9 14:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.55900000000000005,"15min":0.56100000000000005},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.81200000000000006,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.752},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.63100000000000001,"15min":0.61499999999999999},"min":{"1min":0.40600000000000003,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":3.9300000000000002,"5min":3.9300000000000002,"15min":3.9300000000000002},"last":0.86599999999999999}]},{"osd":4,"last update":"Thu Mar 9 14:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.56399999999999995,"15min":0.55600000000000005},"min":{"1min":0.36599999999999999,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.93799999999999994},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.59299999999999997,"15min":0.58499999999999996},"min":{"1min":0.45600000000000002,"5min":0.376,"15min":0.376},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":1.0009999999999999}]},{"osd":5,"last update":"Thu Mar 9 14:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.53400000000000003,"15min":0.53200000000000003},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.54200000000000004,"15min":0.53600000000000003},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.78000000000000003}]},{"osd":6,"last update":"Thu Mar 9 14:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.69599999999999995,"15min":0.67900000000000005},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":3.895,"5min":3.895,"15min":3.895},"last":0.95099999999999996},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.65000000000000002,"15min":0.64900000000000002},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.89000000000000001}]},{"osd":7,"last update":"Thu Mar 9 14:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.66400000000000003,"15min":0.66200000000000003},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.92600000000000005},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.66800000000000004,"15min":0.66300000000000003},"min":{"1min":0.49399999999999999,"5min":0.44,"15min":0.44},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":1.0609999999999999}]},{"osd":9,"last update":"Thu Mar 9 14:43:55 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":1.036},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":1.119}]},{"osd":10,"last update":"Thu Mar 9 14:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":1.099},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":1.1439999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98499999999999999}]}]},{"osd":7,"up_from":50,"seq":214748364832,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7032,"kb_used_data":1520,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738120,"statfs":{"total":95995035648,"available":95987834880,"internally_reserved":0,"allocated":1556480,"data_stored":1058031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:44:13 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.52100000000000002,"15min":0.51700000000000002},"min":{"1min":0.316,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.69999999999999996,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.375},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.51900000000000002,"15min":0.50900000000000001},"min":{"1min":0.32800000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.90100000000000002,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.33900000000000002}]},{"osd":1,"last update":"Thu Mar 9 14:44:13 2023","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.40300000000000002,"15min":0.39300000000000002},"min":{"1min":0.29799999999999999,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.39700000000000002,"15min":0.38800000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.45300000000000001}]},{"osd":2,"last update":"Thu Mar 9 14:44:13 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.53600000000000003,"15min":0.53000000000000003},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.80700000000000005,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.754},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.57399999999999995,"15min":0.56699999999999995},"min":{"1min":0.42199999999999999,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.66900000000000004}]},{"osd":3,"last update":"Thu Mar 9 14:44:13 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.54600000000000004,"15min":0.53400000000000003},"min":{"1min":0.39600000000000002,"5min":0.37,"15min":0.37},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.57799999999999996,"15min":0.56399999999999995},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.57999999999999996}]},{"osd":4,"last update":"Thu Mar 9 14:44:13 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.48299999999999998,"15min":0.47099999999999997},"min":{"1min":0.34899999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.47499999999999998,"15min":0.45900000000000002},"min":{"1min":0.36199999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.50700000000000001}]},{"osd":5,"last update":"Thu Mar 9 14:44:13 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.58099999999999996,"15min":0.57099999999999995},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.58399999999999996,"15min":0.57399999999999995},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.78500000000000003}]},{"osd":6,"last update":"Thu Mar 9 14:44:13 2023","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.627,"15min":0.61499999999999999},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.624,"15min":0.61299999999999999},"min":{"1min":0.49199999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.59499999999999997}]},{"osd":8,"last update":"Thu Mar 9 14:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.65400000000000003,"15min":0.65500000000000003},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.85199999999999998,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.64900000000000002,"15min":0.64700000000000002},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.88300000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.68200000000000005}]},{"osd":9,"last update":"Thu Mar 9 14:43:53 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.77100000000000002}]},{"osd":10,"last update":"Thu Mar 9 14:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.72699999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86699999999999999}]}]},{"osd":6,"up_from":45,"seq":193273528357,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6388,"kb_used_data":940,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738764,"statfs":{"total":95995035648,"available":95988494336,"internally_reserved":0,"allocated":962560,"data_stored":467187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:43:54 2023","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.41799999999999998,"15min":0.41399999999999998},"min":{"1min":0.27400000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.499,"15min":0.49299999999999999},"min":{"1min":0.34200000000000003,"5min":0.318,"15min":0.318},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.54300000000000004}]},{"osd":1,"last update":"Thu Mar 9 14:43:54 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.60099999999999998,"15min":0.60499999999999998},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.67900000000000005}]},{"osd":2,"last update":"Thu Mar 9 14:43:54 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.63,"15min":0.64400000000000002},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.76700000000000002,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.621,"5min":0.63500000000000001,"15min":0.63700000000000001},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.73199999999999998}]},{"osd":3,"last update":"Thu Mar 9 14:43:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.46600000000000003,"15min":0.46300000000000002},"min":{"1min":0.311,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.441},{"interface":"front","average":{"1min":0.625,"5min":0.56899999999999995,"15min":0.55900000000000005},"min":{"1min":0.42499999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.58699999999999997}]},{"osd":4,"last update":"Thu Mar 9 14:43:54 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.73299999999999998,"15min":0.753},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.80700000000000005,"5min":2.0840000000000001,"15min":2.0840000000000001},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.88200000000000001,"15min":0.91600000000000004},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.91600000000000004,"5min":4.1440000000000001,"15min":4.1440000000000001},"last":0.60299999999999998}]},{"osd":5,"last update":"Thu Mar 9 14:43:54 2023","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.70699999999999996,"15min":0.70299999999999996},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.627},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.67600000000000005,"15min":0.68700000000000006},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.79300000000000004,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.70099999999999996}]},{"osd":7,"last update":"Thu Mar 9 14:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.69999999999999996,"15min":0.70899999999999996},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.72999999999999998,"15min":0.73599999999999999},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.96499999999999997,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.71699999999999997}]},{"osd":8,"last update":"Thu Mar 9 14:44:35 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.77400000000000002,"15min":0.78800000000000003},"min":{"1min":0.54200000000000004,"5min":0.497,"15min":0.497},"max":{"1min":0.92500000000000004,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.70599999999999996,"15min":0.72199999999999998},"min":{"1min":0.502,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.81200000000000006,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.73999999999999999}]},{"osd":9,"last update":"Thu Mar 9 14:43:54 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.66700000000000004}]},{"osd":10,"last update":"Thu Mar 9 14:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.75800000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82199999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574905,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7292,"kb_used_data":948,"kb_used_omap":0,"kb_used_meta":6336,"kb_avail":93737860,"statfs":{"total":95995035648,"available":95987568640,"internally_reserved":0,"allocated":970752,"data_stored":470891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6488064},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:44:06 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.5,"15min":0.48999999999999999},"min":{"1min":0.34399999999999997,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.51800000000000002,"15min":0.48599999999999999},"min":{"1min":0.40999999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.90700000000000003,"5min":2.0670000000000002,"15min":2.0670000000000002},"last":0.69799999999999995}]},{"osd":2,"last update":"Thu Mar 9 14:44:31 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.54800000000000004,"15min":0.53000000000000003},"min":{"1min":0.38100000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.85399999999999998,"5min":1.3839999999999999,"15min":1.3839999999999999},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.52700000000000002,"15min":0.50700000000000001},"min":{"1min":0.376,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.77600000000000002,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.73599999999999999}]},{"osd":3,"last update":"Thu Mar 9 14:43:51 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.53600000000000003,"15min":0.504},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.81799999999999995,"5min":2.206,"15min":2.206},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.53300000000000003,"15min":0.51600000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.754}]},{"osd":4,"last update":"Thu Mar 9 14:44:15 2023","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.40100000000000002,"15min":0.38100000000000001},"min":{"1min":0.308,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.39800000000000002,"15min":0.378},"min":{"1min":0.27100000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.60899999999999999}]},{"osd":5,"last update":"Thu Mar 9 14:44:31 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.57299999999999995,"15min":0.54600000000000004},"min":{"1min":0.47999999999999998,"5min":0.373,"15min":0.373},"max":{"1min":0.92800000000000005,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.57599999999999996,"15min":0.54600000000000004},"min":{"1min":0.504,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.85799999999999998,"5min":1.363,"15min":1.363},"last":0.77100000000000002}]},{"osd":6,"last update":"Thu Mar 9 14:43:51 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.65100000000000002,"15min":0.65500000000000003},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.87,"5min":2.1930000000000001,"15min":2.1930000000000001},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.63,"5min":0.66000000000000003,"15min":0.66500000000000004},"min":{"1min":0.42999999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.97999999999999998,"5min":2.2519999999999998,"15min":2.2519999999999998},"last":0.71099999999999997}]},{"osd":7,"last update":"Thu Mar 9 14:44:10 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.501,"15min":0.48299999999999998},"min":{"1min":0.36399999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.51900000000000002,"15min":0.505},"min":{"1min":0.316,"5min":0.31,"15min":0.31},"max":{"1min":0.78400000000000003,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.78700000000000003}]},{"osd":8,"last update":"Thu Mar 9 14:44:33 2023","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.624,"15min":0.61199999999999999},"min":{"1min":0.503,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.88100000000000001,"5min":1.3009999999999999,"15min":1.3009999999999999},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.68100000000000005,"15min":0.67100000000000004},"min":{"1min":0.496,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.98299999999999998,"5min":1.423,"15min":1.423},"last":1.002}]},{"osd":9,"last update":"Thu Mar 9 14:43:57 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.94399999999999995},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.877}]},{"osd":10,"last update":"Thu Mar 9 14:44:19 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.95999999999999996},{"interface":"front","average":{"1min":0.752,"5min":0.752,"15min":0.752},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.98199999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0349999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738431,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7308,"kb_used_data":964,"kb_used_omap":0,"kb_used_meta":6336,"kb_avail":93737844,"statfs":{"total":95995035648,"available":95987552256,"internally_reserved":0,"allocated":987136,"data_stored":468543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6488064},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Mar 9 14:44:08 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.54000000000000004,"15min":0.51800000000000002},"min":{"1min":0.47099999999999997,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.54000000000000004,"15min":0.52600000000000002},"min":{"1min":0.36099999999999999,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.54500000000000004}]},{"osd":2,"last update":"Thu Mar 9 14:44:31 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.55800000000000005,"15min":0.55300000000000005},"min":{"1min":0.41599999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.71399999999999997,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.59999999999999998,"15min":0.57999999999999996},"min":{"1min":0.49199999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.76000000000000001,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.63500000000000001}]},{"osd":3,"last update":"Thu Mar 9 14:42:45 2023","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.35799999999999998,"15min":0.34999999999999998},"min":{"1min":0.27800000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.311},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.39200000000000002,"15min":0.38800000000000001},"min":{"1min":0.26400000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.59499999999999997,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.35699999999999998}]},{"osd":4,"last update":"Thu Mar 9 14:44:08 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.59699999999999998,"15min":0.57099999999999995},"min":{"1min":0.42099999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.139,"5min":1.139,"15min":1.139},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.55400000000000005,"15min":0.54400000000000004},"min":{"1min":0.44900000000000001,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81799999999999995,"5min":1.071,"15min":1.071},"last":0.60599999999999998}]},{"osd":5,"last update":"Thu Mar 9 14:44:37 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.58499999999999996,"15min":0.56299999999999994},"min":{"1min":0.48599999999999999,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.748,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.64500000000000002,"15min":0.626},"min":{"1min":0.46999999999999997,"5min":0.378,"15min":0.378},"max":{"1min":0.78600000000000003,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.71699999999999997}]},{"osd":6,"last update":"Thu Mar 9 14:43:51 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.52600000000000002,"15min":0.501},"min":{"1min":0.443,"5min":0.309,"15min":0.309},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.58699999999999997,"15min":0.55100000000000005},"min":{"1min":0.48499999999999999,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.159,"5min":1.159,"15min":1.159},"last":0.39400000000000002}]},{"osd":7,"last update":"Thu Mar 9 14:44:14 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.71299999999999997,"15min":0.71099999999999997},"min":{"1min":0.503,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.0760000000000001,"5min":1.2110000000000001,"15min":1.2110000000000001},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.64900000000000002,"15min":0.64500000000000002},"min":{"1min":0.52700000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.88500000000000001,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.69499999999999995}]},{"osd":8,"last update":"Thu Mar 9 14:44:37 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.72199999999999998,"15min":0.73299999999999998},"min":{"1min":0.57299999999999995,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.77900000000000003,"5min":1.038,"15min":1.038},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.80600000000000005,"15min":0.82299999999999995},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.81399999999999995,"5min":1.242,"15min":1.242},"last":0.67300000000000004}]},{"osd":9,"last update":"Thu Mar 9 14:43:55 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.105,"5min":1.105,"15min":1.105},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.87,"5min":0.87,"15min":0.87},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.27,"5min":1.27,"15min":1.27},"last":0.46700000000000003}]},{"osd":10,"last update":"Thu Mar 9 14:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.151,"5min":1.151,"15min":1.151},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.55300000000000005}]},{"osd":11,"last update":"Thu Mar 9 14:44:37 2023","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.64800000000000002}]}]},{"osd":2,"up_from":18,"seq":77309411382,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7368,"kb_used_data":960,"kb_used_omap":0,"kb_used_meta":6400,"kb_avail":93737784,"statfs":{"total":95995035648,"available":95987490816,"internally_reserved":0,"allocated":983040,"data_stored":471547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6553600},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:44:30 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.51500000000000001,"15min":0.495},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.47899999999999998,"15min":0.46899999999999997},"min":{"1min":0.30199999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.52600000000000002}]},{"osd":1,"last update":"Thu Mar 9 14:44:30 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.53500000000000003,"15min":0.499},"min":{"1min":0.30199999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.54600000000000004,"15min":0.505},"min":{"1min":0.41799999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":1.1890000000000001,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.85699999999999998}]},{"osd":3,"last update":"Thu Mar 9 14:43:48 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.55400000000000005,"15min":0.53400000000000003},"min":{"1min":0.39300000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.53800000000000003,"15min":0.53100000000000003},"min":{"1min":0.39800000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.77700000000000002,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.63500000000000001}]},{"osd":4,"last update":"Thu Mar 9 14:44:15 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.54900000000000004,"15min":0.52500000000000002},"min":{"1min":0.41599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":0.89300000000000002},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.56699999999999995,"15min":0.54500000000000004},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.234,"5min":1.234,"15min":1.234},"last":0.65900000000000003}]},{"osd":5,"last update":"Thu Mar 9 14:44:35 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.47199999999999998,"15min":0.46000000000000002},"min":{"1min":0.313,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.45200000000000001,"15min":0.44},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.747}]},{"osd":6,"last update":"Thu Mar 9 14:43:51 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.58699999999999997,"15min":0.57699999999999996},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.61799999999999999,"15min":0.61199999999999999},"min":{"1min":0.49199999999999999,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.76700000000000002}]},{"osd":7,"last update":"Thu Mar 9 14:44:15 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.2450000000000001,"5min":1.2450000000000001,"15min":1.2450000000000001},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.69699999999999995,"15min":0.69299999999999995},"min":{"1min":0.46100000000000002,"5min":0.375,"15min":0.375},"max":{"1min":1.2649999999999999,"5min":1.2649999999999999,"15min":1.2649999999999999},"last":0.93500000000000005}]},{"osd":8,"last update":"Thu Mar 9 14:44:35 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.65400000000000003,"15min":0.65100000000000002},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.94699999999999995,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.64600000000000002,"15min":0.65200000000000002},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.93999999999999995,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.83799999999999997}]},{"osd":9,"last update":"Thu Mar 9 14:43:53 2023","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.21,"5min":1.21,"15min":1.21},"last":0.878},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.72999999999999998}]},{"osd":10,"last update":"Thu Mar 9 14:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.2529999999999999,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.306,"5min":1.306,"15min":1.306},"last":0.91900000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96999999999999997}]}]},{"osd":3,"up_from":25,"seq":107374182449,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6652,"kb_used_data":948,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738500,"statfs":{"total":95995035648,"available":95988224000,"internally_reserved":0,"allocated":970752,"data_stored":467291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:43:48 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.34300000000000003,"15min":0.32100000000000001},"min":{"1min":0.28899999999999998,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.36399999999999999,"15min":0.33800000000000002},"min":{"1min":0.32800000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.374}]},{"osd":1,"last update":"Thu Mar 9 14:43:48 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.51300000000000001,"15min":0.499},"min":{"1min":0.378,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.79000000000000004,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.503},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.52200000000000002,"15min":0.48599999999999999},"min":{"1min":0.36099999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.60599999999999998}]},{"osd":2,"last update":"Thu Mar 9 14:43:48 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.53800000000000003,"15min":0.51000000000000001},"min":{"1min":0.40500000000000003,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.54900000000000004,"15min":0.52600000000000002},"min":{"1min":0.38500000000000001,"5min":0.312,"15min":0.312},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.58399999999999996}]},{"osd":4,"last update":"Thu Mar 9 14:44:10 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.56599999999999995,"15min":0.53700000000000003},"min":{"1min":0.52300000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.624},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.58099999999999996,"15min":0.54300000000000004},"min":{"1min":0.46800000000000003,"5min":0.373,"15min":0.373},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.68400000000000005}]},{"osd":5,"last update":"Thu Mar 9 14:44:32 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.56699999999999995,"15min":0.54300000000000004},"min":{"1min":0.433,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.60099999999999998,"15min":0.56499999999999995},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.64700000000000002}]},{"osd":6,"last update":"Thu Mar 9 14:43:49 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.55500000000000005,"15min":0.53300000000000003},"min":{"1min":0.35399999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.56699999999999995,"15min":0.54100000000000004},"min":{"1min":0.46999999999999997,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.441}]},{"osd":7,"last update":"Thu Mar 9 14:44:15 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.624,"15min":0.61799999999999999},"min":{"1min":0.46600000000000003,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.68000000000000005,"15min":0.66700000000000004},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.77200000000000002}]},{"osd":8,"last update":"Thu Mar 9 14:43:35 2023","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.749}]},{"osd":9,"last update":"Thu Mar 9 14:43:53 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.59399999999999997}]},{"osd":10,"last update":"Thu Mar 9 14:44:20 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.155,"5min":1.155,"15min":1.155},"last":0.79500000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67700000000000005}]}]},{"osd":4,"up_from":32,"seq":137438953517,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6516,"kb_used_data":940,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738636,"statfs":{"total":95995035648,"available":95988363264,"internally_reserved":0,"allocated":962560,"data_stored":467119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:44:09 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.53100000000000003,"15min":0.495},"min":{"1min":0.39000000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.84099999999999997,"5min":1.8320000000000001,"15min":1.8320000000000001},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.628,"5min":0.52900000000000003,"15min":0.495},"min":{"1min":0.42499999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":1.044,"5min":1.478,"15min":1.478},"last":0.52500000000000002}]},{"osd":1,"last update":"Thu Mar 9 14:44:09 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.36299999999999999,"15min":0.34300000000000003},"min":{"1min":0.27800000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.38500000000000001,"15min":0.36499999999999999},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.39900000000000002}]},{"osd":2,"last update":"Thu Mar 9 14:44:09 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.53300000000000003,"15min":0.52700000000000002},"min":{"1min":0.438,"5min":0.375,"15min":0.375},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.55400000000000005,"15min":0.53300000000000003},"min":{"1min":0.45300000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.72999999999999998}]},{"osd":3,"last update":"Thu Mar 9 14:44:09 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.56799999999999995,"15min":0.52800000000000002},"min":{"1min":0.53900000000000003,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.099,"5min":1.4510000000000001,"15min":1.4510000000000001},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.56399999999999995,"15min":0.53000000000000003},"min":{"1min":0.42799999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.91500000000000004,"5min":1.8640000000000001,"15min":1.8640000000000001},"last":0.504}]},{"osd":5,"last update":"Thu Mar 9 14:44:33 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.58099999999999996,"15min":0.55900000000000005},"min":{"1min":0.45800000000000002,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.59399999999999997,"15min":0.56599999999999995},"min":{"1min":0.47099999999999997,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.71799999999999997}]},{"osd":6,"last update":"Thu Mar 9 14:43:49 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.68600000000000005,"15min":0.69399999999999995},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.81100000000000005,"5min":1.889,"15min":1.889},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.67400000000000004,"15min":0.67500000000000004},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.03,"5min":1.8480000000000001,"15min":1.8480000000000001},"last":0.52100000000000002}]},{"osd":7,"last update":"Thu Mar 9 14:44:12 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.55500000000000005,"15min":0.52900000000000003},"min":{"1min":0.35299999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.56399999999999995,"15min":0.53400000000000003},"min":{"1min":0.37,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.53500000000000003}]},{"osd":8,"last update":"Thu Mar 9 14:43:36 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.79700000000000004}]},{"osd":9,"last update":"Thu Mar 9 14:43:58 2023","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.752,"15min":0.752},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.145,"5min":1.145,"15min":1.145},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.55200000000000005}]},{"osd":10,"last update":"Thu Mar 9 14:44:20 2023","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.228,"5min":1.228,"15min":1.228},"last":0.627}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75900000000000001}]}]},{"osd":5,"up_from":37,"seq":158913789993,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7164,"kb_used_data":1524,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93737988,"statfs":{"total":95995035648,"available":95987699712,"internally_reserved":0,"allocated":1560576,"data_stored":1057316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.51200000000000001,"15min":0.48599999999999999},"min":{"1min":0.35599999999999998,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.80000000000000004,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.51900000000000002,"15min":0.5},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.69899999999999995,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.29299999999999998}]},{"osd":1,"last update":"Thu Mar 9 14:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55500000000000005,"15min":0.54700000000000004},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.83599999999999997,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.54300000000000004,"15min":0.53100000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.52100000000000002}]},{"osd":2,"last update":"Thu Mar 9 14:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.375,"15min":0.35999999999999999},"min":{"1min":0.249,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.60099999999999998,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.38900000000000001,"15min":0.36899999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.39700000000000002}]},{"osd":3,"last update":"Thu Mar 9 14:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.54000000000000004,"15min":0.53500000000000003},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.71399999999999997,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56699999999999995,"15min":0.55100000000000005},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.752,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.313}]},{"osd":4,"last update":"Thu Mar 9 14:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.56999999999999995,"15min":0.55600000000000005},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.76200000000000001,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53000000000000003,"15min":0.52600000000000002},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.70399999999999996,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.55100000000000005}]},{"osd":6,"last update":"Thu Mar 9 14:43:49 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.498,"5min":0.315,"15min":0.315},"max":{"1min":0.72899999999999998,"5min":0.879,"15min":0.879},"last":0.503},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.63,"15min":0.624},"min":{"1min":0.39500000000000002,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.42899999999999999}]},{"osd":7,"last update":"Thu Mar 9 14:44:15 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.59099999999999997,"15min":0.58699999999999997},"min":{"1min":0.437,"5min":0.38,"15min":0.38},"max":{"1min":0.81000000000000005,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.59499999999999997,"15min":0.59199999999999997},"min":{"1min":0.41699999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.78700000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.54100000000000004}]},{"osd":8,"last update":"Thu Mar 9 14:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.59499999999999997,"15min":0.59899999999999998},"min":{"1min":0.35099999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.84099999999999997,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.54200000000000004,"15min":0.54400000000000004},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.45400000000000001}]},{"osd":9,"last update":"Thu Mar 9 14:43:56 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.38500000000000001}]},{"osd":10,"last update":"Thu Mar 9 14:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.63400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":21095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":23047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-03-09T14:44:42.306 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-03-09T14:44:42.459 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.456+0000 7f4215bde700 1 -- 172.21.15.38:0/2715131429 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f42100feda0 msgr2=0x7f42100ff1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:42.459 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.456+0000 7f4215bde700 1 --2- 172.21.15.38:0/2715131429 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f42100feda0 0x7f42100ff1b0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f4200009990 tx=0x7f4200009ca0 comp rx=0 tx=0).stop 2023-03-09T14:44:42.459 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.456+0000 7f4215bde700 1 -- 172.21.15.38:0/2715131429 shutdown_connections 2023-03-09T14:44:42.459 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.456+0000 7f4215bde700 1 --2- 172.21.15.38:0/2715131429 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f42100feda0 0x7f42100ff1b0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:42.460 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.456+0000 7f4215bde700 1 --2- 172.21.15.38:0/2715131429 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4210100920 0x7f42101071e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:42.460 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.456+0000 7f4215bde700 1 --2- 172.21.15.38:0/2715131429 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f42100fff90 0x7f42101003e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:42.460 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.456+0000 7f4215bde700 1 -- 172.21.15.38:0/2715131429 >> 172.21.15.38:0/2715131429 conn(0x7f42100fa380 msgr2=0x7f42100fc7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:42.460 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.456+0000 7f4215bde700 1 -- 172.21.15.38:0/2715131429 shutdown_connections 2023-03-09T14:44:42.460 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.456+0000 7f4215bde700 1 -- 172.21.15.38:0/2715131429 wait complete. 2023-03-09T14:44:42.460 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.457+0000 7f4215bde700 1 Processor -- start 2023-03-09T14:44:42.460 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.457+0000 7f4215bde700 1 -- start start 2023-03-09T14:44:42.462 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.457+0000 7f4215bde700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42100feda0 0x7f4210195dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:42.462 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.457+0000 7f4215bde700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f42100fff90 0x7f4210196310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:42.462 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.457+0000 7f4215bde700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4210100920 0x7f421019d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:42.463 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.457+0000 7f4215bde700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f421019d950 con 0x7f4210100920 2023-03-09T14:44:42.463 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.457+0000 7f4215bde700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f421019da90 con 0x7f42100feda0 2023-03-09T14:44:42.463 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.457+0000 7f4215bde700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f421019dc20 con 0x7f42100fff90 2023-03-09T14:44:42.463 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.457+0000 7f420ffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4210100920 0x7f421019d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:42.463 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.457+0000 7f420ffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4210100920 0x7f421019d380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37256/0 (socket says 172.21.15.38:37256) 2023-03-09T14:44:42.463 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.457+0000 7f420ffff700 1 -- 172.21.15.38:0/1455508699 learned_addr learned my addr 172.21.15.38:0/1455508699 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:44:42.463 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.457+0000 7f420f7fe700 1 --2- 172.21.15.38:0/1455508699 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42100feda0 0x7f4210195dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:42.463 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.457+0000 7f420effd700 1 --2- 172.21.15.38:0/1455508699 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f42100fff90 0x7f4210196310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:42.464 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.458+0000 7f420ffff700 1 -- 172.21.15.38:0/1455508699 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f42100fff90 msgr2=0x7f4210196310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:42.464 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.458+0000 7f420ffff700 1 --2- 172.21.15.38:0/1455508699 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f42100fff90 0x7f4210196310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:42.464 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.458+0000 7f420ffff700 1 -- 172.21.15.38:0/1455508699 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42100feda0 msgr2=0x7f4210195dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:42.464 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.458+0000 7f420ffff700 1 --2- 172.21.15.38:0/1455508699 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42100feda0 0x7f4210195dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:42.464 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.458+0000 7f420ffff700 1 -- 172.21.15.38:0/1455508699 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42000095f0 con 0x7f4210100920 2023-03-09T14:44:42.464 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.458+0000 7f420ffff700 1 --2- 172.21.15.38:0/1455508699 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4210100920 0x7f421019d380 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f420800e940 tx=0x7f420800ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:42.464 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.458+0000 7f420cff9700 1 -- 172.21.15.38:0/1455508699 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42080097b0 con 0x7f4210100920 2023-03-09T14:44:42.465 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.458+0000 7f420cff9700 1 -- 172.21.15.38:0/1455508699 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f420800ee90 con 0x7f4210100920 2023-03-09T14:44:42.465 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.458+0000 7f4215bde700 1 -- 172.21.15.38:0/1455508699 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f421010a750 con 0x7f4210100920 2023-03-09T14:44:42.465 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.458+0000 7f420cff9700 1 -- 172.21.15.38:0/1455508699 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4208005520 con 0x7f4210100920 2023-03-09T14:44:42.465 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.458+0000 7f4215bde700 1 -- 172.21.15.38:0/1455508699 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f421010a5c0 con 0x7f4210100920 2023-03-09T14:44:42.465 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.459+0000 7f4215bde700 1 -- 172.21.15.38:0/1455508699 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4210066dd0 con 0x7f4210100920 2023-03-09T14:44:42.465 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.460+0000 7f420cff9700 1 -- 172.21.15.38:0/1455508699 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 133238+0+0 (secure 0 0 0) 0x7f420801e020 con 0x7f4210100920 2023-03-09T14:44:42.466 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.460+0000 7f420cff9700 1 --2- 172.21.15.38:0/1455508699 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f41f809f400 0x7f41f80a18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:42.466 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.461+0000 7f420f7fe700 1 --2- 172.21.15.38:0/1455508699 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f41f809f400 0x7f41f80a18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:42.466 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.461+0000 7f420cff9700 1 -- 172.21.15.38:0/1455508699 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f4208014070 con 0x7f4210100920 2023-03-09T14:44:42.466 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.461+0000 7f420f7fe700 1 --2- 172.21.15.38:0/1455508699 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f41f809f400 0x7f41f80a18b0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f420000b2e0 tx=0x7f4200005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:42.467 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.465+0000 7f420cff9700 1 -- 172.21.15.38:0/1455508699 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f42080943e0 con 0x7f4210100920 2023-03-09T14:44:42.661 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.658+0000 7f4215bde700 1 -- 172.21.15.38:0/1455508699 --> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f421019e6b0 con 0x7f41f809f400 2023-03-09T14:44:42.676 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.674+0000 7f420cff9700 1 -- 172.21.15.38:0/1455508699 <== mgr.14150 v2:172.21.15.38:6800/3328887328 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+418536 (secure 0 0 0) 0x7f421019e6b0 con 0x7f41f809f400 2023-03-09T14:44:42.681 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:44:42.683 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.678+0000 7f4215bde700 1 -- 172.21.15.38:0/1455508699 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f41f809f400 msgr2=0x7f41f80a18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:42.683 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.678+0000 7f4215bde700 1 --2- 172.21.15.38:0/1455508699 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f41f809f400 0x7f41f80a18b0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f420000b2e0 tx=0x7f4200005fb0 comp rx=0 tx=0).stop 2023-03-09T14:44:42.683 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.678+0000 7f4215bde700 1 -- 172.21.15.38:0/1455508699 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4210100920 msgr2=0x7f421019d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:42.684 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.678+0000 7f4215bde700 1 --2- 172.21.15.38:0/1455508699 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4210100920 0x7f421019d380 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f420800e940 tx=0x7f420800ec50 comp rx=0 tx=0).stop 2023-03-09T14:44:42.684 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.678+0000 7f4215bde700 1 -- 172.21.15.38:0/1455508699 shutdown_connections 2023-03-09T14:44:42.684 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.678+0000 7f4215bde700 1 --2- 172.21.15.38:0/1455508699 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4210100920 0x7f421019d380 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:42.684 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.678+0000 7f4215bde700 1 --2- 172.21.15.38:0/1455508699 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f41f809f400 0x7f41f80a18b0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:42.684 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.678+0000 7f4215bde700 1 --2- 172.21.15.38:0/1455508699 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42100feda0 0x7f4210195dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:42.684 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.678+0000 7f4215bde700 1 --2- 172.21.15.38:0/1455508699 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f42100fff90 0x7f4210196310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:42.684 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.678+0000 7f4215bde700 1 -- 172.21.15.38:0/1455508699 >> 172.21.15.38:0/1455508699 conn(0x7f42100fa380 msgr2=0x7f42100689c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:42.685 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.678+0000 7f4215bde700 1 -- 172.21.15.38:0/1455508699 shutdown_connections 2023-03-09T14:44:42.685 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:42.678+0000 7f4215bde700 1 -- 172.21.15.38:0/1455508699 wait complete. 2023-03-09T14:44:42.685 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2023-03-09T14:44:42.693 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":270,"stamp":"2023-03-09T14:44:40.750225+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":612515,"num_objects":38,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":141,"num_read_kb":126,"num_write":272,"num_write_kb":2978,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":25,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":214,"ondisk_log_size":214,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":81388,"kb_used_data":13132,"kb_used_omap":0,"kb_used_meta":68160,"kb_avail":1124860436,"statfs":{"total":1151940427776,"available":1151857086464,"internally_reserved":0,"allocated":13447168,"data_stored":7392765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":69795840},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19769,"num_objects":14,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":5,"num_read_kb":7,"num_write":46,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":11,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003359"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.473760+0000","last_change":"2023-03-09T14:44:19.116371+0000","last_active":"2023-03-09T14:44:23.473760+0000","last_peered":"2023-03-09T14:44:23.473760+0000","last_clean":"2023-03-09T14:44:23.473760+0000","last_became_active":"2023-03-09T14:44:19.116024+0000","last_became_peered":"2023-03-09T14:44:19.116024+0000","last_unstale":"2023-03-09T14:44:23.473760+0000","last_undegraded":"2023-03-09T14:44:23.473760+0000","last_fullsized":"2023-03-09T14:44:23.473760+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T16:43:33.917176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.475467+0000","last_change":"2023-03-09T14:44:19.117312+0000","last_active":"2023-03-09T14:44:23.475467+0000","last_peered":"2023-03-09T14:44:23.475467+0000","last_clean":"2023-03-09T14:44:23.475467+0000","last_became_active":"2023-03-09T14:44:19.116920+0000","last_became_peered":"2023-03-09T14:44:19.116920+0000","last_unstale":"2023-03-09T14:44:23.475467+0000","last_undegraded":"2023-03-09T14:44:23.475467+0000","last_fullsized":"2023-03-09T14:44:23.475467+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:33:15.046431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.472579+0000","last_change":"2023-03-09T14:44:19.113482+0000","last_active":"2023-03-09T14:44:23.472579+0000","last_peered":"2023-03-09T14:44:23.472579+0000","last_clean":"2023-03-09T14:44:23.472579+0000","last_became_active":"2023-03-09T14:44:19.113183+0000","last_became_peered":"2023-03-09T14:44:19.113183+0000","last_unstale":"2023-03-09T14:44:23.472579+0000","last_undegraded":"2023-03-09T14:44:23.472579+0000","last_fullsized":"2023-03-09T14:44:23.472579+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:17:31.382255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.471310+0000","last_change":"2023-03-09T14:44:19.105799+0000","last_active":"2023-03-09T14:44:23.471310+0000","last_peered":"2023-03-09T14:44:23.471310+0000","last_clean":"2023-03-09T14:44:23.471310+0000","last_became_active":"2023-03-09T14:44:19.105405+0000","last_became_peered":"2023-03-09T14:44:19.105405+0000","last_unstale":"2023-03-09T14:44:23.471310+0000","last_undegraded":"2023-03-09T14:44:23.471310+0000","last_fullsized":"2023-03-09T14:44:23.471310+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:38:23.049984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.472968+0000","last_change":"2023-03-09T14:44:19.100656+0000","last_active":"2023-03-09T14:44:23.472968+0000","last_peered":"2023-03-09T14:44:23.472968+0000","last_clean":"2023-03-09T14:44:23.472968+0000","last_became_active":"2023-03-09T14:44:19.100278+0000","last_became_peered":"2023-03-09T14:44:19.100278+0000","last_unstale":"2023-03-09T14:44:23.472968+0000","last_undegraded":"2023-03-09T14:44:23.472968+0000","last_fullsized":"2023-03-09T14:44:23.472968+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:25:39.240251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.469680+0000","last_change":"2023-03-09T14:44:19.101065+0000","last_active":"2023-03-09T14:44:23.469680+0000","last_peered":"2023-03-09T14:44:23.469680+0000","last_clean":"2023-03-09T14:44:23.469680+0000","last_became_active":"2023-03-09T14:44:19.100451+0000","last_became_peered":"2023-03-09T14:44:19.100451+0000","last_unstale":"2023-03-09T14:44:23.469680+0000","last_undegraded":"2023-03-09T14:44:23.469680+0000","last_fullsized":"2023-03-09T14:44:23.469680+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T01:25:15.158254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.473598+0000","last_change":"2023-03-09T14:44:19.107996+0000","last_active":"2023-03-09T14:44:23.473598+0000","last_peered":"2023-03-09T14:44:23.473598+0000","last_clean":"2023-03-09T14:44:23.473598+0000","last_became_active":"2023-03-09T14:44:19.107683+0000","last_became_peered":"2023-03-09T14:44:19.107683+0000","last_unstale":"2023-03-09T14:44:23.473598+0000","last_undegraded":"2023-03-09T14:44:23.473598+0000","last_fullsized":"2023-03-09T14:44:23.473598+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:16:18.605176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.469646+0000","last_change":"2023-03-09T14:44:19.103940+0000","last_active":"2023-03-09T14:44:23.469646+0000","last_peered":"2023-03-09T14:44:23.469646+0000","last_clean":"2023-03-09T14:44:23.469646+0000","last_became_active":"2023-03-09T14:44:19.103620+0000","last_became_peered":"2023-03-09T14:44:19.103620+0000","last_unstale":"2023-03-09T14:44:23.469646+0000","last_undegraded":"2023-03-09T14:44:23.469646+0000","last_fullsized":"2023-03-09T14:44:23.469646+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:06:22.749636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.477978+0000","last_change":"2023-03-09T14:44:19.102695+0000","last_active":"2023-03-09T14:44:23.477978+0000","last_peered":"2023-03-09T14:44:23.477978+0000","last_clean":"2023-03-09T14:44:23.477978+0000","last_became_active":"2023-03-09T14:44:19.101486+0000","last_became_peered":"2023-03-09T14:44:19.101486+0000","last_unstale":"2023-03-09T14:44:23.477978+0000","last_undegraded":"2023-03-09T14:44:23.477978+0000","last_fullsized":"2023-03-09T14:44:23.477978+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T16:34:51.836443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.469700+0000","last_change":"2023-03-09T14:44:19.103522+0000","last_active":"2023-03-09T14:44:23.469700+0000","last_peered":"2023-03-09T14:44:23.469700+0000","last_clean":"2023-03-09T14:44:23.469700+0000","last_became_active":"2023-03-09T14:44:19.103242+0000","last_became_peered":"2023-03-09T14:44:19.103242+0000","last_unstale":"2023-03-09T14:44:23.469700+0000","last_undegraded":"2023-03-09T14:44:23.469700+0000","last_fullsized":"2023-03-09T14:44:23.469700+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:41:03.137554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.474842+0000","last_change":"2023-03-09T14:44:19.114934+0000","last_active":"2023-03-09T14:44:23.474842+0000","last_peered":"2023-03-09T14:44:23.474842+0000","last_clean":"2023-03-09T14:44:23.474842+0000","last_became_active":"2023-03-09T14:44:19.114620+0000","last_became_peered":"2023-03-09T14:44:19.114620+0000","last_unstale":"2023-03-09T14:44:23.474842+0000","last_undegraded":"2023-03-09T14:44:23.474842+0000","last_fullsized":"2023-03-09T14:44:23.474842+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T01:48:40.082076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.471227+0000","last_change":"2023-03-09T14:44:19.107677+0000","last_active":"2023-03-09T14:44:23.471227+0000","last_peered":"2023-03-09T14:44:23.471227+0000","last_clean":"2023-03-09T14:44:23.471227+0000","last_became_active":"2023-03-09T14:44:19.107369+0000","last_became_peered":"2023-03-09T14:44:19.107369+0000","last_unstale":"2023-03-09T14:44:23.471227+0000","last_undegraded":"2023-03-09T14:44:23.471227+0000","last_fullsized":"2023-03-09T14:44:23.471227+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:47:40.117205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.477738+0000","last_change":"2023-03-09T14:44:19.108852+0000","last_active":"2023-03-09T14:44:23.477738+0000","last_peered":"2023-03-09T14:44:23.477738+0000","last_clean":"2023-03-09T14:44:23.477738+0000","last_became_active":"2023-03-09T14:44:19.108515+0000","last_became_peered":"2023-03-09T14:44:19.108515+0000","last_unstale":"2023-03-09T14:44:23.477738+0000","last_undegraded":"2023-03-09T14:44:23.477738+0000","last_fullsized":"2023-03-09T14:44:23.477738+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:00:36.243952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.473653+0000","last_change":"2023-03-09T14:44:19.118673+0000","last_active":"2023-03-09T14:44:23.473653+0000","last_peered":"2023-03-09T14:44:23.473653+0000","last_clean":"2023-03-09T14:44:23.473653+0000","last_became_active":"2023-03-09T14:44:19.117656+0000","last_became_peered":"2023-03-09T14:44:19.117656+0000","last_unstale":"2023-03-09T14:44:23.473653+0000","last_undegraded":"2023-03-09T14:44:23.473653+0000","last_fullsized":"2023-03-09T14:44:23.473653+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:02:50.401680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.473802+0000","last_change":"2023-03-09T14:44:19.112979+0000","last_active":"2023-03-09T14:44:23.473802+0000","last_peered":"2023-03-09T14:44:23.473802+0000","last_clean":"2023-03-09T14:44:23.473802+0000","last_became_active":"2023-03-09T14:44:19.112671+0000","last_became_peered":"2023-03-09T14:44:19.112671+0000","last_unstale":"2023-03-09T14:44:23.473802+0000","last_undegraded":"2023-03-09T14:44:23.473802+0000","last_fullsized":"2023-03-09T14:44:23.473802+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:00:43.465327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.474845+0000","last_change":"2023-03-09T14:44:19.104881+0000","last_active":"2023-03-09T14:44:23.474845+0000","last_peered":"2023-03-09T14:44:23.474845+0000","last_clean":"2023-03-09T14:44:23.474845+0000","last_became_active":"2023-03-09T14:44:19.104647+0000","last_became_peered":"2023-03-09T14:44:19.104647+0000","last_unstale":"2023-03-09T14:44:23.474845+0000","last_undegraded":"2023-03-09T14:44:23.474845+0000","last_fullsized":"2023-03-09T14:44:23.474845+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T20:16:24.625362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.473964+0000","last_change":"2023-03-09T14:44:19.117937+0000","last_active":"2023-03-09T14:44:23.473964+0000","last_peered":"2023-03-09T14:44:23.473964+0000","last_clean":"2023-03-09T14:44:23.473964+0000","last_became_active":"2023-03-09T14:44:19.117628+0000","last_became_peered":"2023-03-09T14:44:19.117628+0000","last_unstale":"2023-03-09T14:44:23.473964+0000","last_undegraded":"2023-03-09T14:44:23.473964+0000","last_fullsized":"2023-03-09T14:44:23.473964+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:22:18.619171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.479118+0000","last_change":"2023-03-09T14:44:19.105071+0000","last_active":"2023-03-09T14:44:23.479118+0000","last_peered":"2023-03-09T14:44:23.479118+0000","last_clean":"2023-03-09T14:44:23.479118+0000","last_became_active":"2023-03-09T14:44:19.103396+0000","last_became_peered":"2023-03-09T14:44:19.103396+0000","last_unstale":"2023-03-09T14:44:23.479118+0000","last_undegraded":"2023-03-09T14:44:23.479118+0000","last_fullsized":"2023-03-09T14:44:23.479118+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:14:51.924636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.475650+0000","last_change":"2023-03-09T14:44:19.104107+0000","last_active":"2023-03-09T14:44:23.475650+0000","last_peered":"2023-03-09T14:44:23.475650+0000","last_clean":"2023-03-09T14:44:23.475650+0000","last_became_active":"2023-03-09T14:44:19.103793+0000","last_became_peered":"2023-03-09T14:44:19.103793+0000","last_unstale":"2023-03-09T14:44:23.475650+0000","last_undegraded":"2023-03-09T14:44:23.475650+0000","last_fullsized":"2023-03-09T14:44:23.475650+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:57:29.169744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.490413+0000","last_change":"2023-03-09T14:44:23.489947+0000","last_active":"2023-03-09T14:44:23.490413+0000","last_peered":"2023-03-09T14:44:23.490413+0000","last_clean":"2023-03-09T14:44:23.490413+0000","last_became_active":"2023-03-09T14:44:23.489560+0000","last_became_peered":"2023-03-09T14:44:23.489560+0000","last_unstale":"2023-03-09T14:44:23.490413+0000","last_undegraded":"2023-03-09T14:44:23.490413+0000","last_fullsized":"2023-03-09T14:44:23.490413+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:06:10.202176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"87'9","reported_seq":25,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-03-09T14:44:38.524476+0000","last_change":"2023-03-09T14:44:18.653405+0000","last_active":"2023-03-09T14:44:38.524476+0000","last_peered":"2023-03-09T14:44:38.524476+0000","last_clean":"2023-03-09T14:44:38.524476+0000","last_became_active":"2023-03-09T14:44:18.653085+0000","last_became_peered":"2023-03-09T14:44:18.653085+0000","last_unstale":"2023-03-09T14:44:38.524476+0000","last_undegraded":"2023-03-09T14:44:38.524476+0000","last_fullsized":"2023-03-09T14:44:38.524476+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:57:26.782467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.471353+0000","last_change":"2023-03-09T14:44:19.105390+0000","last_active":"2023-03-09T14:44:23.471353+0000","last_peered":"2023-03-09T14:44:23.471353+0000","last_clean":"2023-03-09T14:44:23.471353+0000","last_became_active":"2023-03-09T14:44:19.104694+0000","last_became_peered":"2023-03-09T14:44:19.104694+0000","last_unstale":"2023-03-09T14:44:23.471353+0000","last_undegraded":"2023-03-09T14:44:23.471353+0000","last_fullsized":"2023-03-09T14:44:23.471353+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:19:08.101847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.904879+0000","last_change":"2023-03-09T14:44:23.484103+0000","last_active":"2023-03-09T14:44:23.904879+0000","last_peered":"2023-03-09T14:44:23.904879+0000","last_clean":"2023-03-09T14:44:23.904879+0000","last_became_active":"2023-03-09T14:44:23.483271+0000","last_became_peered":"2023-03-09T14:44:23.483271+0000","last_unstale":"2023-03-09T14:44:23.904879+0000","last_undegraded":"2023-03-09T14:44:23.904879+0000","last_fullsized":"2023-03-09T14:44:23.904879+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:24:06.212623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"87'2","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.904795+0000","last_change":"2023-03-09T14:44:18.667321+0000","last_active":"2023-03-09T14:44:23.904795+0000","last_peered":"2023-03-09T14:44:23.904795+0000","last_clean":"2023-03-09T14:44:23.904795+0000","last_became_active":"2023-03-09T14:44:18.666974+0000","last_became_peered":"2023-03-09T14:44:18.666974+0000","last_unstale":"2023-03-09T14:44:23.904795+0000","last_undegraded":"2023-03-09T14:44:23.904795+0000","last_fullsized":"2023-03-09T14:44:23.904795+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:31:07.327792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.473744+0000","last_change":"2023-03-09T14:44:19.116866+0000","last_active":"2023-03-09T14:44:23.473744+0000","last_peered":"2023-03-09T14:44:23.473744+0000","last_clean":"2023-03-09T14:44:23.473744+0000","last_became_active":"2023-03-09T14:44:19.116571+0000","last_became_peered":"2023-03-09T14:44:19.116571+0000","last_unstale":"2023-03-09T14:44:23.473744+0000","last_undegraded":"2023-03-09T14:44:23.473744+0000","last_fullsized":"2023-03-09T14:44:23.473744+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T16:22:12.723903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.905925+0000","last_change":"2023-03-09T14:44:23.488192+0000","last_active":"2023-03-09T14:44:23.905925+0000","last_peered":"2023-03-09T14:44:23.905925+0000","last_clean":"2023-03-09T14:44:23.905925+0000","last_became_active":"2023-03-09T14:44:23.487734+0000","last_became_peered":"2023-03-09T14:44:23.487734+0000","last_unstale":"2023-03-09T14:44:23.905925+0000","last_undegraded":"2023-03-09T14:44:23.905925+0000","last_fullsized":"2023-03-09T14:44:23.905925+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:35:50.319216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"82'2","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.475193+0000","last_change":"2023-03-09T14:44:18.663577+0000","last_active":"2023-03-09T14:44:23.475193+0000","last_peered":"2023-03-09T14:44:23.475193+0000","last_clean":"2023-03-09T14:44:23.475193+0000","last_became_active":"2023-03-09T14:44:18.663272+0000","last_became_peered":"2023-03-09T14:44:18.663272+0000","last_unstale":"2023-03-09T14:44:23.475193+0000","last_undegraded":"2023-03-09T14:44:23.475193+0000","last_fullsized":"2023-03-09T14:44:23.475193+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T20:55:19.661875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.475940+0000","last_change":"2023-03-09T14:44:19.106986+0000","last_active":"2023-03-09T14:44:23.475940+0000","last_peered":"2023-03-09T14:44:23.475940+0000","last_clean":"2023-03-09T14:44:23.475940+0000","last_became_active":"2023-03-09T14:44:19.106625+0000","last_became_peered":"2023-03-09T14:44:19.106625+0000","last_unstale":"2023-03-09T14:44:23.475940+0000","last_undegraded":"2023-03-09T14:44:23.475940+0000","last_fullsized":"2023-03-09T14:44:23.475940+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:03:54.521603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.905654+0000","last_change":"2023-03-09T14:44:23.483803+0000","last_active":"2023-03-09T14:44:23.905654+0000","last_peered":"2023-03-09T14:44:23.905654+0000","last_clean":"2023-03-09T14:44:23.905654+0000","last_became_active":"2023-03-09T14:44:23.483286+0000","last_became_peered":"2023-03-09T14:44:23.483286+0000","last_unstale":"2023-03-09T14:44:23.905654+0000","last_undegraded":"2023-03-09T14:44:23.905654+0000","last_fullsized":"2023-03-09T14:44:23.905654+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:32:35.975670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"82'2","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.475167+0000","last_change":"2023-03-09T14:44:18.657685+0000","last_active":"2023-03-09T14:44:23.475167+0000","last_peered":"2023-03-09T14:44:23.475167+0000","last_clean":"2023-03-09T14:44:23.475167+0000","last_became_active":"2023-03-09T14:44:18.657355+0000","last_became_peered":"2023-03-09T14:44:18.657355+0000","last_unstale":"2023-03-09T14:44:23.475167+0000","last_undegraded":"2023-03-09T14:44:23.475167+0000","last_fullsized":"2023-03-09T14:44:23.475167+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:33:59.259690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.480488+0000","last_change":"2023-03-09T14:44:19.111687+0000","last_active":"2023-03-09T14:44:23.480488+0000","last_peered":"2023-03-09T14:44:23.480488+0000","last_clean":"2023-03-09T14:44:23.480488+0000","last_became_active":"2023-03-09T14:44:19.111481+0000","last_became_peered":"2023-03-09T14:44:19.111481+0000","last_unstale":"2023-03-09T14:44:23.480488+0000","last_undegraded":"2023-03-09T14:44:23.480488+0000","last_fullsized":"2023-03-09T14:44:23.480488+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:06:14.452967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":12,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.487525+0000","last_change":"2023-03-09T14:44:23.485075+0000","last_active":"2023-03-09T14:44:23.487525+0000","last_peered":"2023-03-09T14:44:23.487525+0000","last_clean":"2023-03-09T14:44:23.487525+0000","last_became_active":"2023-03-09T14:44:23.484648+0000","last_became_peered":"2023-03-09T14:44:23.484648+0000","last_unstale":"2023-03-09T14:44:23.487525+0000","last_undegraded":"2023-03-09T14:44:23.487525+0000","last_fullsized":"2023-03-09T14:44:23.487525+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:49:40.332276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"82'1","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.479619+0000","last_change":"2023-03-09T14:44:18.665323+0000","last_active":"2023-03-09T14:44:23.479619+0000","last_peered":"2023-03-09T14:44:23.479619+0000","last_clean":"2023-03-09T14:44:23.479619+0000","last_became_active":"2023-03-09T14:44:18.665001+0000","last_became_peered":"2023-03-09T14:44:18.665001+0000","last_unstale":"2023-03-09T14:44:23.479619+0000","last_undegraded":"2023-03-09T14:44:23.479619+0000","last_fullsized":"2023-03-09T14:44:23.479619+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:22:20.325914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.477336+0000","last_change":"2023-03-09T14:44:19.107572+0000","last_active":"2023-03-09T14:44:23.477336+0000","last_peered":"2023-03-09T14:44:23.477336+0000","last_clean":"2023-03-09T14:44:23.477336+0000","last_became_active":"2023-03-09T14:44:19.106632+0000","last_became_peered":"2023-03-09T14:44:19.106632+0000","last_unstale":"2023-03-09T14:44:23.477336+0000","last_undegraded":"2023-03-09T14:44:23.477336+0000","last_fullsized":"2023-03-09T14:44:23.477336+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T19:04:01.379375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":12,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.494387+0000","last_change":"2023-03-09T14:44:23.494002+0000","last_active":"2023-03-09T14:44:23.494387+0000","last_peered":"2023-03-09T14:44:23.494387+0000","last_clean":"2023-03-09T14:44:23.494387+0000","last_became_active":"2023-03-09T14:44:23.493628+0000","last_became_peered":"2023-03-09T14:44:23.493628+0000","last_unstale":"2023-03-09T14:44:23.494387+0000","last_undegraded":"2023-03-09T14:44:23.494387+0000","last_fullsized":"2023-03-09T14:44:23.494387+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:30:47.970556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"87'3","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-03-09T14:44:38.524129+0000","last_change":"2023-03-09T14:44:18.661169+0000","last_active":"2023-03-09T14:44:38.524129+0000","last_peered":"2023-03-09T14:44:38.524129+0000","last_clean":"2023-03-09T14:44:38.524129+0000","last_became_active":"2023-03-09T14:44:18.660858+0000","last_became_peered":"2023-03-09T14:44:18.660858+0000","last_unstale":"2023-03-09T14:44:38.524129+0000","last_undegraded":"2023-03-09T14:44:38.524129+0000","last_fullsized":"2023-03-09T14:44:38.524129+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:18:45.551100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.479758+0000","last_change":"2023-03-09T14:44:19.111303+0000","last_active":"2023-03-09T14:44:23.479758+0000","last_peered":"2023-03-09T14:44:23.479758+0000","last_clean":"2023-03-09T14:44:23.479758+0000","last_became_active":"2023-03-09T14:44:19.111026+0000","last_became_peered":"2023-03-09T14:44:19.111026+0000","last_unstale":"2023-03-09T14:44:23.479758+0000","last_undegraded":"2023-03-09T14:44:23.479758+0000","last_fullsized":"2023-03-09T14:44:23.479758+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T02:26:32.496084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":12,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.493647+0000","last_change":"2023-03-09T14:44:23.493272+0000","last_active":"2023-03-09T14:44:23.493647+0000","last_peered":"2023-03-09T14:44:23.493647+0000","last_clean":"2023-03-09T14:44:23.493647+0000","last_became_active":"2023-03-09T14:44:23.492785+0000","last_became_peered":"2023-03-09T14:44:23.492785+0000","last_unstale":"2023-03-09T14:44:23.493647+0000","last_undegraded":"2023-03-09T14:44:23.493647+0000","last_fullsized":"2023-03-09T14:44:23.493647+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T20:18:39.758938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"87'2","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-03-09T14:44:38.523245+0000","last_change":"2023-03-09T14:44:18.658179+0000","last_active":"2023-03-09T14:44:38.523245+0000","last_peered":"2023-03-09T14:44:38.523245+0000","last_clean":"2023-03-09T14:44:38.523245+0000","last_became_active":"2023-03-09T14:44:18.657851+0000","last_became_peered":"2023-03-09T14:44:18.657851+0000","last_unstale":"2023-03-09T14:44:38.523245+0000","last_undegraded":"2023-03-09T14:44:38.523245+0000","last_fullsized":"2023-03-09T14:44:38.523245+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:52:49.291615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.469539+0000","last_change":"2023-03-09T14:44:19.104176+0000","last_active":"2023-03-09T14:44:23.469539+0000","last_peered":"2023-03-09T14:44:23.469539+0000","last_clean":"2023-03-09T14:44:23.469539+0000","last_became_active":"2023-03-09T14:44:19.103816+0000","last_became_peered":"2023-03-09T14:44:19.103816+0000","last_unstale":"2023-03-09T14:44:23.469539+0000","last_undegraded":"2023-03-09T14:44:23.469539+0000","last_fullsized":"2023-03-09T14:44:23.469539+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T02:06:41.876169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-03-09T14:44:24.313741+0000","last_change":"2023-03-09T14:44:23.490876+0000","last_active":"2023-03-09T14:44:24.313741+0000","last_peered":"2023-03-09T14:44:24.313741+0000","last_clean":"2023-03-09T14:44:24.313741+0000","last_became_active":"2023-03-09T14:44:23.490461+0000","last_became_peered":"2023-03-09T14:44:23.490461+0000","last_unstale":"2023-03-09T14:44:24.313741+0000","last_undegraded":"2023-03-09T14:44:24.313741+0000","last_fullsized":"2023-03-09T14:44:24.313741+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T16:28:40.456613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.474747+0000","last_change":"2023-03-09T14:44:18.662513+0000","last_active":"2023-03-09T14:44:23.474747+0000","last_peered":"2023-03-09T14:44:23.474747+0000","last_clean":"2023-03-09T14:44:23.474747+0000","last_became_active":"2023-03-09T14:44:18.661907+0000","last_became_peered":"2023-03-09T14:44:18.661907+0000","last_unstale":"2023-03-09T14:44:23.474747+0000","last_undegraded":"2023-03-09T14:44:23.474747+0000","last_fullsized":"2023-03-09T14:44:23.474747+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:45:17.373213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.476301+0000","last_change":"2023-03-09T14:44:19.115297+0000","last_active":"2023-03-09T14:44:23.476301+0000","last_peered":"2023-03-09T14:44:23.476301+0000","last_clean":"2023-03-09T14:44:23.476301+0000","last_became_active":"2023-03-09T14:44:19.114973+0000","last_became_peered":"2023-03-09T14:44:19.114973+0000","last_unstale":"2023-03-09T14:44:23.476301+0000","last_undegraded":"2023-03-09T14:44:23.476301+0000","last_fullsized":"2023-03-09T14:44:23.476301+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:11:22.226608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.904202+0000","last_change":"2023-03-09T14:44:23.482772+0000","last_active":"2023-03-09T14:44:23.904202+0000","last_peered":"2023-03-09T14:44:23.904202+0000","last_clean":"2023-03-09T14:44:23.904202+0000","last_became_active":"2023-03-09T14:44:23.481750+0000","last_became_peered":"2023-03-09T14:44:23.481750+0000","last_unstale":"2023-03-09T14:44:23.904202+0000","last_undegraded":"2023-03-09T14:44:23.904202+0000","last_fullsized":"2023-03-09T14:44:23.904202+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T19:26:03.737713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"87'3","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-03-09T14:44:37.502621+0000","last_change":"2023-03-09T14:44:18.665582+0000","last_active":"2023-03-09T14:44:37.502621+0000","last_peered":"2023-03-09T14:44:37.502621+0000","last_clean":"2023-03-09T14:44:37.502621+0000","last_became_active":"2023-03-09T14:44:18.665265+0000","last_became_peered":"2023-03-09T14:44:18.665265+0000","last_unstale":"2023-03-09T14:44:37.502621+0000","last_undegraded":"2023-03-09T14:44:37.502621+0000","last_fullsized":"2023-03-09T14:44:37.502621+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T16:15:54.119713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.477924+0000","last_change":"2023-03-09T14:44:19.100914+0000","last_active":"2023-03-09T14:44:23.477924+0000","last_peered":"2023-03-09T14:44:23.477924+0000","last_clean":"2023-03-09T14:44:23.477924+0000","last_became_active":"2023-03-09T14:44:19.100451+0000","last_became_peered":"2023-03-09T14:44:19.100451+0000","last_unstale":"2023-03-09T14:44:23.477924+0000","last_undegraded":"2023-03-09T14:44:23.477924+0000","last_fullsized":"2023-03-09T14:44:23.477924+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:18:41.852697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:24.249360+0000","last_change":"2023-03-09T14:44:23.490129+0000","last_active":"2023-03-09T14:44:24.249360+0000","last_peered":"2023-03-09T14:44:24.249360+0000","last_clean":"2023-03-09T14:44:24.249360+0000","last_became_active":"2023-03-09T14:44:23.489696+0000","last_became_peered":"2023-03-09T14:44:23.489696+0000","last_unstale":"2023-03-09T14:44:24.249360+0000","last_undegraded":"2023-03-09T14:44:24.249360+0000","last_fullsized":"2023-03-09T14:44:24.249360+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T16:47:41.943904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.474329+0000","last_change":"2023-03-09T14:44:18.661420+0000","last_active":"2023-03-09T14:44:23.474329+0000","last_peered":"2023-03-09T14:44:23.474329+0000","last_clean":"2023-03-09T14:44:23.474329+0000","last_became_active":"2023-03-09T14:44:18.661119+0000","last_became_peered":"2023-03-09T14:44:18.661119+0000","last_unstale":"2023-03-09T14:44:23.474329+0000","last_undegraded":"2023-03-09T14:44:23.474329+0000","last_fullsized":"2023-03-09T14:44:23.474329+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:27:26.403125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.473839+0000","last_change":"2023-03-09T14:44:19.119125+0000","last_active":"2023-03-09T14:44:23.473839+0000","last_peered":"2023-03-09T14:44:23.473839+0000","last_clean":"2023-03-09T14:44:23.473839+0000","last_became_active":"2023-03-09T14:44:19.118677+0000","last_became_peered":"2023-03-09T14:44:19.118677+0000","last_unstale":"2023-03-09T14:44:23.473839+0000","last_undegraded":"2023-03-09T14:44:23.473839+0000","last_fullsized":"2023-03-09T14:44:23.473839+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:06:28.740355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.906104+0000","last_change":"2023-03-09T14:44:23.484752+0000","last_active":"2023-03-09T14:44:23.906104+0000","last_peered":"2023-03-09T14:44:23.906104+0000","last_clean":"2023-03-09T14:44:23.906104+0000","last_became_active":"2023-03-09T14:44:23.484045+0000","last_became_peered":"2023-03-09T14:44:23.484045+0000","last_unstale":"2023-03-09T14:44:23.906104+0000","last_undegraded":"2023-03-09T14:44:23.906104+0000","last_fullsized":"2023-03-09T14:44:23.906104+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:58:39.704017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"82'3","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.480387+0000","last_change":"2023-03-09T14:44:18.665122+0000","last_active":"2023-03-09T14:44:23.480387+0000","last_peered":"2023-03-09T14:44:23.480387+0000","last_clean":"2023-03-09T14:44:23.480387+0000","last_became_active":"2023-03-09T14:44:18.664786+0000","last_became_peered":"2023-03-09T14:44:18.664786+0000","last_unstale":"2023-03-09T14:44:23.480387+0000","last_undegraded":"2023-03-09T14:44:23.480387+0000","last_fullsized":"2023-03-09T14:44:23.480387+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T20:29:20.919293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.476134+0000","last_change":"2023-03-09T14:44:19.108693+0000","last_active":"2023-03-09T14:44:23.476134+0000","last_peered":"2023-03-09T14:44:23.476134+0000","last_clean":"2023-03-09T14:44:23.476134+0000","last_became_active":"2023-03-09T14:44:19.108370+0000","last_became_peered":"2023-03-09T14:44:19.108370+0000","last_unstale":"2023-03-09T14:44:23.476134+0000","last_undegraded":"2023-03-09T14:44:23.476134+0000","last_fullsized":"2023-03-09T14:44:23.476134+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T16:30:58.726080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:24.249387+0000","last_change":"2023-03-09T14:44:23.488684+0000","last_active":"2023-03-09T14:44:24.249387+0000","last_peered":"2023-03-09T14:44:24.249387+0000","last_clean":"2023-03-09T14:44:24.249387+0000","last_became_active":"2023-03-09T14:44:23.488284+0000","last_became_peered":"2023-03-09T14:44:23.488284+0000","last_unstale":"2023-03-09T14:44:24.249387+0000","last_undegraded":"2023-03-09T14:44:24.249387+0000","last_fullsized":"2023-03-09T14:44:24.249387+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:56:41.323297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"87'18","reported_seq":38,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-03-09T14:44:39.963262+0000","last_change":"2023-03-09T14:44:18.661904+0000","last_active":"2023-03-09T14:44:39.963262+0000","last_peered":"2023-03-09T14:44:39.963262+0000","last_clean":"2023-03-09T14:44:39.963262+0000","last_became_active":"2023-03-09T14:44:18.661597+0000","last_became_peered":"2023-03-09T14:44:18.661597+0000","last_unstale":"2023-03-09T14:44:39.963262+0000","last_undegraded":"2023-03-09T14:44:39.963262+0000","last_fullsized":"2023-03-09T14:44:39.963262+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T14:57:04.978132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":21095,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":6,"num_write":19,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.474881+0000","last_change":"2023-03-09T14:44:19.111671+0000","last_active":"2023-03-09T14:44:23.474881+0000","last_peered":"2023-03-09T14:44:23.474881+0000","last_clean":"2023-03-09T14:44:23.474881+0000","last_became_active":"2023-03-09T14:44:19.111172+0000","last_became_peered":"2023-03-09T14:44:19.111172+0000","last_unstale":"2023-03-09T14:44:23.474881+0000","last_undegraded":"2023-03-09T14:44:23.474881+0000","last_fullsized":"2023-03-09T14:44:23.474881+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T16:27:23.921853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-03-09T14:44:24.286526+0000","last_change":"2023-03-09T14:44:23.486338+0000","last_active":"2023-03-09T14:44:24.286526+0000","last_peered":"2023-03-09T14:44:24.286526+0000","last_clean":"2023-03-09T14:44:24.286526+0000","last_became_active":"2023-03-09T14:44:23.485729+0000","last_became_peered":"2023-03-09T14:44:23.485729+0000","last_unstale":"2023-03-09T14:44:24.286526+0000","last_undegraded":"2023-03-09T14:44:24.286526+0000","last_fullsized":"2023-03-09T14:44:24.286526+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:28:23.872633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"87'4","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-03-09T14:44:37.502627+0000","last_change":"2023-03-09T14:44:18.664942+0000","last_active":"2023-03-09T14:44:37.502627+0000","last_peered":"2023-03-09T14:44:37.502627+0000","last_clean":"2023-03-09T14:44:37.502627+0000","last_became_active":"2023-03-09T14:44:18.664007+0000","last_became_peered":"2023-03-09T14:44:18.664007+0000","last_unstale":"2023-03-09T14:44:37.502627+0000","last_undegraded":"2023-03-09T14:44:37.502627+0000","last_fullsized":"2023-03-09T14:44:37.502627+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:36:21.329388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.475377+0000","last_change":"2023-03-09T14:44:19.110140+0000","last_active":"2023-03-09T14:44:23.475377+0000","last_peered":"2023-03-09T14:44:23.475377+0000","last_clean":"2023-03-09T14:44:23.475377+0000","last_became_active":"2023-03-09T14:44:19.109811+0000","last_became_peered":"2023-03-09T14:44:19.109811+0000","last_unstale":"2023-03-09T14:44:23.475377+0000","last_undegraded":"2023-03-09T14:44:23.475377+0000","last_fullsized":"2023-03-09T14:44:23.475377+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T14:56:14.638662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":12,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.491669+0000","last_change":"2023-03-09T14:44:23.491318+0000","last_active":"2023-03-09T14:44:23.491669+0000","last_peered":"2023-03-09T14:44:23.491669+0000","last_clean":"2023-03-09T14:44:23.491669+0000","last_became_active":"2023-03-09T14:44:23.490915+0000","last_became_peered":"2023-03-09T14:44:23.490915+0000","last_unstale":"2023-03-09T14:44:23.491669+0000","last_undegraded":"2023-03-09T14:44:23.491669+0000","last_fullsized":"2023-03-09T14:44:23.491669+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:32:06.877573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.479568+0000","last_change":"2023-03-09T14:44:18.657323+0000","last_active":"2023-03-09T14:44:23.479568+0000","last_peered":"2023-03-09T14:44:23.479568+0000","last_clean":"2023-03-09T14:44:23.479568+0000","last_became_active":"2023-03-09T14:44:18.657017+0000","last_became_peered":"2023-03-09T14:44:18.657017+0000","last_unstale":"2023-03-09T14:44:23.479568+0000","last_undegraded":"2023-03-09T14:44:23.479568+0000","last_fullsized":"2023-03-09T14:44:23.479568+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:43:11.562723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.474017+0000","last_change":"2023-03-09T14:44:19.112194+0000","last_active":"2023-03-09T14:44:23.474017+0000","last_peered":"2023-03-09T14:44:23.474017+0000","last_clean":"2023-03-09T14:44:23.474017+0000","last_became_active":"2023-03-09T14:44:19.111924+0000","last_became_peered":"2023-03-09T14:44:19.111924+0000","last_unstale":"2023-03-09T14:44:23.474017+0000","last_undegraded":"2023-03-09T14:44:23.474017+0000","last_fullsized":"2023-03-09T14:44:23.474017+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T19:54:22.669848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:24.249416+0000","last_change":"2023-03-09T14:44:23.493993+0000","last_active":"2023-03-09T14:44:24.249416+0000","last_peered":"2023-03-09T14:44:24.249416+0000","last_clean":"2023-03-09T14:44:24.249416+0000","last_became_active":"2023-03-09T14:44:23.493535+0000","last_became_peered":"2023-03-09T14:44:23.493535+0000","last_unstale":"2023-03-09T14:44:24.249416+0000","last_undegraded":"2023-03-09T14:44:24.249416+0000","last_fullsized":"2023-03-09T14:44:24.249416+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T01:18:23.669790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.469908+0000","last_change":"2023-03-09T14:44:18.659914+0000","last_active":"2023-03-09T14:44:23.469908+0000","last_peered":"2023-03-09T14:44:23.469908+0000","last_clean":"2023-03-09T14:44:23.469908+0000","last_became_active":"2023-03-09T14:44:18.658835+0000","last_became_peered":"2023-03-09T14:44:18.658835+0000","last_unstale":"2023-03-09T14:44:23.469908+0000","last_undegraded":"2023-03-09T14:44:23.469908+0000","last_fullsized":"2023-03-09T14:44:23.469908+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T14:55:05.413260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.475687+0000","last_change":"2023-03-09T14:44:19.111058+0000","last_active":"2023-03-09T14:44:23.475687+0000","last_peered":"2023-03-09T14:44:23.475687+0000","last_clean":"2023-03-09T14:44:23.475687+0000","last_became_active":"2023-03-09T14:44:19.110760+0000","last_became_peered":"2023-03-09T14:44:19.110760+0000","last_unstale":"2023-03-09T14:44:23.475687+0000","last_undegraded":"2023-03-09T14:44:23.475687+0000","last_fullsized":"2023-03-09T14:44:23.475687+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:25:13.199416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-03-09T14:44:24.286025+0000","last_change":"2023-03-09T14:44:23.484710+0000","last_active":"2023-03-09T14:44:24.286025+0000","last_peered":"2023-03-09T14:44:24.286025+0000","last_clean":"2023-03-09T14:44:24.286025+0000","last_became_active":"2023-03-09T14:44:23.483928+0000","last_became_peered":"2023-03-09T14:44:23.483928+0000","last_unstale":"2023-03-09T14:44:24.286025+0000","last_undegraded":"2023-03-09T14:44:24.286025+0000","last_fullsized":"2023-03-09T14:44:24.286025+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T14:51:32.124111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"82'8","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.473728+0000","last_change":"2023-03-09T14:44:18.663692+0000","last_active":"2023-03-09T14:44:23.473728+0000","last_peered":"2023-03-09T14:44:23.473728+0000","last_clean":"2023-03-09T14:44:23.473728+0000","last_became_active":"2023-03-09T14:44:18.663400+0000","last_became_peered":"2023-03-09T14:44:18.663400+0000","last_unstale":"2023-03-09T14:44:23.473728+0000","last_undegraded":"2023-03-09T14:44:23.473728+0000","last_fullsized":"2023-03-09T14:44:23.473728+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:32:31.634315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.479630+0000","last_change":"2023-03-09T14:44:19.110819+0000","last_active":"2023-03-09T14:44:23.479630+0000","last_peered":"2023-03-09T14:44:23.479630+0000","last_clean":"2023-03-09T14:44:23.479630+0000","last_became_active":"2023-03-09T14:44:19.110542+0000","last_became_peered":"2023-03-09T14:44:19.110542+0000","last_unstale":"2023-03-09T14:44:23.479630+0000","last_undegraded":"2023-03-09T14:44:23.479630+0000","last_fullsized":"2023-03-09T14:44:23.479630+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:25:38.382685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-03-09T14:44:24.286231+0000","last_change":"2023-03-09T14:44:23.484614+0000","last_active":"2023-03-09T14:44:24.286231+0000","last_peered":"2023-03-09T14:44:24.286231+0000","last_clean":"2023-03-09T14:44:24.286231+0000","last_became_active":"2023-03-09T14:44:23.484061+0000","last_became_peered":"2023-03-09T14:44:23.484061+0000","last_unstale":"2023-03-09T14:44:24.286231+0000","last_undegraded":"2023-03-09T14:44:24.286231+0000","last_fullsized":"2023-03-09T14:44:24.286231+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T19:31:42.889122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"82'1","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.475017+0000","last_change":"2023-03-09T14:44:18.656056+0000","last_active":"2023-03-09T14:44:23.475017+0000","last_peered":"2023-03-09T14:44:23.475017+0000","last_clean":"2023-03-09T14:44:23.475017+0000","last_became_active":"2023-03-09T14:44:18.655709+0000","last_became_peered":"2023-03-09T14:44:18.655709+0000","last_unstale":"2023-03-09T14:44:23.475017+0000","last_undegraded":"2023-03-09T14:44:23.475017+0000","last_fullsized":"2023-03-09T14:44:23.475017+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T19:09:11.447249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.475132+0000","last_change":"2023-03-09T14:44:19.110292+0000","last_active":"2023-03-09T14:44:23.475132+0000","last_peered":"2023-03-09T14:44:23.475132+0000","last_clean":"2023-03-09T14:44:23.475132+0000","last_became_active":"2023-03-09T14:44:19.110006+0000","last_became_peered":"2023-03-09T14:44:19.110006+0000","last_unstale":"2023-03-09T14:44:23.475132+0000","last_undegraded":"2023-03-09T14:44:23.475132+0000","last_fullsized":"2023-03-09T14:44:23.475132+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:36:40.401486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-03-09T14:44:24.287375+0000","last_change":"2023-03-09T14:44:23.484004+0000","last_active":"2023-03-09T14:44:24.287375+0000","last_peered":"2023-03-09T14:44:24.287375+0000","last_clean":"2023-03-09T14:44:24.287375+0000","last_became_active":"2023-03-09T14:44:23.483581+0000","last_became_peered":"2023-03-09T14:44:23.483581+0000","last_unstale":"2023-03-09T14:44:24.287375+0000","last_undegraded":"2023-03-09T14:44:24.287375+0000","last_fullsized":"2023-03-09T14:44:24.287375+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:24:25.939838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.476558+0000","last_change":"2023-03-09T14:44:18.661856+0000","last_active":"2023-03-09T14:44:23.476558+0000","last_peered":"2023-03-09T14:44:23.476558+0000","last_clean":"2023-03-09T14:44:23.476558+0000","last_became_active":"2023-03-09T14:44:18.661500+0000","last_became_peered":"2023-03-09T14:44:18.661500+0000","last_unstale":"2023-03-09T14:44:23.476558+0000","last_undegraded":"2023-03-09T14:44:23.476558+0000","last_fullsized":"2023-03-09T14:44:23.476558+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:27:12.181159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.469525+0000","last_change":"2023-03-09T14:44:19.102058+0000","last_active":"2023-03-09T14:44:23.469525+0000","last_peered":"2023-03-09T14:44:23.469525+0000","last_clean":"2023-03-09T14:44:23.469525+0000","last_became_active":"2023-03-09T14:44:19.101762+0000","last_became_peered":"2023-03-09T14:44:19.101762+0000","last_unstale":"2023-03-09T14:44:23.469525+0000","last_undegraded":"2023-03-09T14:44:23.469525+0000","last_fullsized":"2023-03-09T14:44:23.469525+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:39:42.544086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-03-09T14:44:24.285308+0000","last_change":"2023-03-09T14:44:23.490237+0000","last_active":"2023-03-09T14:44:24.285308+0000","last_peered":"2023-03-09T14:44:24.285308+0000","last_clean":"2023-03-09T14:44:24.285308+0000","last_became_active":"2023-03-09T14:44:23.489844+0000","last_became_peered":"2023-03-09T14:44:23.489844+0000","last_unstale":"2023-03-09T14:44:24.285308+0000","last_undegraded":"2023-03-09T14:44:24.285308+0000","last_fullsized":"2023-03-09T14:44:24.285308+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T14:50:09.358698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.473792+0000","last_change":"2023-03-09T14:44:18.657684+0000","last_active":"2023-03-09T14:44:23.473792+0000","last_peered":"2023-03-09T14:44:23.473792+0000","last_clean":"2023-03-09T14:44:23.473792+0000","last_became_active":"2023-03-09T14:44:18.657295+0000","last_became_peered":"2023-03-09T14:44:18.657295+0000","last_unstale":"2023-03-09T14:44:23.473792+0000","last_undegraded":"2023-03-09T14:44:23.473792+0000","last_fullsized":"2023-03-09T14:44:23.473792+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:50:39.037653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.469107+0000","last_change":"2023-03-09T14:44:19.117584+0000","last_active":"2023-03-09T14:44:23.469107+0000","last_peered":"2023-03-09T14:44:23.469107+0000","last_clean":"2023-03-09T14:44:23.469107+0000","last_became_active":"2023-03-09T14:44:19.117262+0000","last_became_peered":"2023-03-09T14:44:19.117262+0000","last_unstale":"2023-03-09T14:44:23.469107+0000","last_undegraded":"2023-03-09T14:44:23.469107+0000","last_fullsized":"2023-03-09T14:44:23.469107+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T19:19:38.080048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-03-09T14:44:24.286054+0000","last_change":"2023-03-09T14:44:23.485769+0000","last_active":"2023-03-09T14:44:24.286054+0000","last_peered":"2023-03-09T14:44:24.286054+0000","last_clean":"2023-03-09T14:44:24.286054+0000","last_became_active":"2023-03-09T14:44:23.485317+0000","last_became_peered":"2023-03-09T14:44:23.485317+0000","last_unstale":"2023-03-09T14:44:24.286054+0000","last_undegraded":"2023-03-09T14:44:24.286054+0000","last_fullsized":"2023-03-09T14:44:24.286054+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:58:35.140512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.474918+0000","last_change":"2023-03-09T14:44:19.112255+0000","last_active":"2023-03-09T14:44:23.474918+0000","last_peered":"2023-03-09T14:44:23.474918+0000","last_clean":"2023-03-09T14:44:23.474918+0000","last_became_active":"2023-03-09T14:44:19.111988+0000","last_became_peered":"2023-03-09T14:44:19.111988+0000","last_unstale":"2023-03-09T14:44:23.474918+0000","last_undegraded":"2023-03-09T14:44:23.474918+0000","last_fullsized":"2023-03-09T14:44:23.474918+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:55:50.423701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"87'4","reported_seq":20,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-03-09T14:44:38.524897+0000","last_change":"2023-03-09T14:44:18.665291+0000","last_active":"2023-03-09T14:44:38.524897+0000","last_peered":"2023-03-09T14:44:38.524897+0000","last_clean":"2023-03-09T14:44:38.524897+0000","last_became_active":"2023-03-09T14:44:18.664950+0000","last_became_peered":"2023-03-09T14:44:18.664950+0000","last_unstale":"2023-03-09T14:44:38.524897+0000","last_undegraded":"2023-03-09T14:44:38.524897+0000","last_fullsized":"2023-03-09T14:44:38.524897+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:40:58.619717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-03-09T14:44:24.285763+0000","last_change":"2023-03-09T14:44:23.480872+0000","last_active":"2023-03-09T14:44:24.285763+0000","last_peered":"2023-03-09T14:44:24.285763+0000","last_clean":"2023-03-09T14:44:24.285763+0000","last_became_active":"2023-03-09T14:44:23.480495+0000","last_became_peered":"2023-03-09T14:44:23.480495+0000","last_unstale":"2023-03-09T14:44:24.285763+0000","last_undegraded":"2023-03-09T14:44:24.285763+0000","last_fullsized":"2023-03-09T14:44:24.285763+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:19:01.393034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.469682+0000","last_change":"2023-03-09T14:44:19.102833+0000","last_active":"2023-03-09T14:44:23.469682+0000","last_peered":"2023-03-09T14:44:23.469682+0000","last_clean":"2023-03-09T14:44:23.469682+0000","last_became_active":"2023-03-09T14:44:19.102484+0000","last_became_peered":"2023-03-09T14:44:19.102484+0000","last_unstale":"2023-03-09T14:44:23.469682+0000","last_undegraded":"2023-03-09T14:44:23.469682+0000","last_fullsized":"2023-03-09T14:44:23.469682+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:14:50.002907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.470135+0000","last_change":"2023-03-09T14:44:18.661603+0000","last_active":"2023-03-09T14:44:23.470135+0000","last_peered":"2023-03-09T14:44:23.470135+0000","last_clean":"2023-03-09T14:44:23.470135+0000","last_became_active":"2023-03-09T14:44:18.661275+0000","last_became_peered":"2023-03-09T14:44:18.661275+0000","last_unstale":"2023-03-09T14:44:23.470135+0000","last_undegraded":"2023-03-09T14:44:23.470135+0000","last_fullsized":"2023-03-09T14:44:23.470135+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:20:31.873778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":376,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.475580+0000","last_change":"2023-03-09T14:43:19.317554+0000","last_active":"2023-03-09T14:44:23.475580+0000","last_peered":"2023-03-09T14:44:23.475580+0000","last_clean":"2023-03-09T14:44:23.475580+0000","last_became_active":"2023-03-09T14:43:19.316912+0000","last_became_peered":"2023-03-09T14:43:19.316912+0000","last_unstale":"2023-03-09T14:44:23.475580+0000","last_undegraded":"2023-03-09T14:44:23.475580+0000","last_fullsized":"2023-03-09T14:44:23.475580+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:40:23.197173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:40:23.197173+0000","last_clean_scrub_stamp":"2023-03-09T14:40:23.197173+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T20:21:17.143331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-03-09T14:44:24.286143+0000","last_change":"2023-03-09T14:44:23.487738+0000","last_active":"2023-03-09T14:44:24.286143+0000","last_peered":"2023-03-09T14:44:24.286143+0000","last_clean":"2023-03-09T14:44:24.286143+0000","last_became_active":"2023-03-09T14:44:23.487193+0000","last_became_peered":"2023-03-09T14:44:23.487193+0000","last_unstale":"2023-03-09T14:44:24.286143+0000","last_undegraded":"2023-03-09T14:44:24.286143+0000","last_fullsized":"2023-03-09T14:44:24.286143+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:48:08.789071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.476422+0000","last_change":"2023-03-09T14:44:19.112576+0000","last_active":"2023-03-09T14:44:23.476422+0000","last_peered":"2023-03-09T14:44:23.476422+0000","last_clean":"2023-03-09T14:44:23.476422+0000","last_became_active":"2023-03-09T14:44:19.112231+0000","last_became_peered":"2023-03-09T14:44:19.112231+0000","last_unstale":"2023-03-09T14:44:23.476422+0000","last_undegraded":"2023-03-09T14:44:23.476422+0000","last_fullsized":"2023-03-09T14:44:23.476422+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:25:21.918088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.475646+0000","last_change":"2023-03-09T14:44:18.655478+0000","last_active":"2023-03-09T14:44:23.475646+0000","last_peered":"2023-03-09T14:44:23.475646+0000","last_clean":"2023-03-09T14:44:23.475646+0000","last_became_active":"2023-03-09T14:44:18.655180+0000","last_became_peered":"2023-03-09T14:44:18.655180+0000","last_unstale":"2023-03-09T14:44:23.475646+0000","last_undegraded":"2023-03-09T14:44:23.475646+0000","last_fullsized":"2023-03-09T14:44:23.475646+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:42:47.926376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.488348+0000","last_change":"2023-03-09T14:44:23.487052+0000","last_active":"2023-03-09T14:44:23.488348+0000","last_peered":"2023-03-09T14:44:23.488348+0000","last_clean":"2023-03-09T14:44:23.488348+0000","last_became_active":"2023-03-09T14:44:23.486603+0000","last_became_peered":"2023-03-09T14:44:23.486603+0000","last_unstale":"2023-03-09T14:44:23.488348+0000","last_undegraded":"2023-03-09T14:44:23.488348+0000","last_fullsized":"2023-03-09T14:44:23.488348+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:28:25.375052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.474962+0000","last_change":"2023-03-09T14:44:18.666400+0000","last_active":"2023-03-09T14:44:23.474962+0000","last_peered":"2023-03-09T14:44:23.474962+0000","last_clean":"2023-03-09T14:44:23.474962+0000","last_became_active":"2023-03-09T14:44:18.666045+0000","last_became_peered":"2023-03-09T14:44:18.666045+0000","last_unstale":"2023-03-09T14:44:23.474962+0000","last_undegraded":"2023-03-09T14:44:23.474962+0000","last_fullsized":"2023-03-09T14:44:23.474962+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:30:48.640124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.476223+0000","last_change":"2023-03-09T14:44:19.103657+0000","last_active":"2023-03-09T14:44:23.476223+0000","last_peered":"2023-03-09T14:44:23.476223+0000","last_clean":"2023-03-09T14:44:23.476223+0000","last_became_active":"2023-03-09T14:44:19.103364+0000","last_became_peered":"2023-03-09T14:44:19.103364+0000","last_unstale":"2023-03-09T14:44:23.476223+0000","last_undegraded":"2023-03-09T14:44:23.476223+0000","last_fullsized":"2023-03-09T14:44:23.476223+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:23:23.738554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.904079+0000","last_change":"2023-03-09T14:44:23.483487+0000","last_active":"2023-03-09T14:44:23.904079+0000","last_peered":"2023-03-09T14:44:23.904079+0000","last_clean":"2023-03-09T14:44:23.904079+0000","last_became_active":"2023-03-09T14:44:23.483050+0000","last_became_peered":"2023-03-09T14:44:23.483050+0000","last_unstale":"2023-03-09T14:44:23.904079+0000","last_undegraded":"2023-03-09T14:44:23.904079+0000","last_fullsized":"2023-03-09T14:44:23.904079+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:38:34.129355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"87'3","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-03-09T14:44:38.524026+0000","last_change":"2023-03-09T14:44:18.661944+0000","last_active":"2023-03-09T14:44:38.524026+0000","last_peered":"2023-03-09T14:44:38.524026+0000","last_clean":"2023-03-09T14:44:38.524026+0000","last_became_active":"2023-03-09T14:44:18.661575+0000","last_became_peered":"2023-03-09T14:44:18.661575+0000","last_unstale":"2023-03-09T14:44:38.524026+0000","last_undegraded":"2023-03-09T14:44:38.524026+0000","last_fullsized":"2023-03-09T14:44:38.524026+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T01:12:53.637866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.471841+0000","last_change":"2023-03-09T14:44:19.103807+0000","last_active":"2023-03-09T14:44:23.471841+0000","last_peered":"2023-03-09T14:44:23.471841+0000","last_clean":"2023-03-09T14:44:23.471841+0000","last_became_active":"2023-03-09T14:44:19.103480+0000","last_became_peered":"2023-03-09T14:44:19.103480+0000","last_unstale":"2023-03-09T14:44:23.471841+0000","last_undegraded":"2023-03-09T14:44:23.471841+0000","last_fullsized":"2023-03-09T14:44:23.471841+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:29:54.629265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.487495+0000","last_change":"2023-03-09T14:44:23.486789+0000","last_active":"2023-03-09T14:44:23.487495+0000","last_peered":"2023-03-09T14:44:23.487495+0000","last_clean":"2023-03-09T14:44:23.487495+0000","last_became_active":"2023-03-09T14:44:23.486367+0000","last_became_peered":"2023-03-09T14:44:23.486367+0000","last_unstale":"2023-03-09T14:44:23.487495+0000","last_undegraded":"2023-03-09T14:44:23.487495+0000","last_fullsized":"2023-03-09T14:44:23.487495+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:13:13.081388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"82'2","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.470707+0000","last_change":"2023-03-09T14:44:18.659492+0000","last_active":"2023-03-09T14:44:23.470707+0000","last_peered":"2023-03-09T14:44:23.470707+0000","last_clean":"2023-03-09T14:44:23.470707+0000","last_became_active":"2023-03-09T14:44:18.659187+0000","last_became_peered":"2023-03-09T14:44:18.659187+0000","last_unstale":"2023-03-09T14:44:23.470707+0000","last_undegraded":"2023-03-09T14:44:23.470707+0000","last_fullsized":"2023-03-09T14:44:23.470707+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:44:31.342785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.475564+0000","last_change":"2023-03-09T14:44:19.112841+0000","last_active":"2023-03-09T14:44:23.475564+0000","last_peered":"2023-03-09T14:44:23.475564+0000","last_clean":"2023-03-09T14:44:23.475564+0000","last_became_active":"2023-03-09T14:44:19.112515+0000","last_became_peered":"2023-03-09T14:44:19.112515+0000","last_unstale":"2023-03-09T14:44:23.475564+0000","last_undegraded":"2023-03-09T14:44:23.475564+0000","last_fullsized":"2023-03-09T14:44:23.475564+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T19:41:04.065529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.905391+0000","last_change":"2023-03-09T14:44:23.481992+0000","last_active":"2023-03-09T14:44:23.905391+0000","last_peered":"2023-03-09T14:44:23.905391+0000","last_clean":"2023-03-09T14:44:23.905391+0000","last_became_active":"2023-03-09T14:44:23.481508+0000","last_became_peered":"2023-03-09T14:44:23.481508+0000","last_unstale":"2023-03-09T14:44:23.905391+0000","last_undegraded":"2023-03-09T14:44:23.905391+0000","last_fullsized":"2023-03-09T14:44:23.905391+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:46:48.683402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"87'2","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-03-09T14:44:37.501019+0000","last_change":"2023-03-09T14:44:18.663133+0000","last_active":"2023-03-09T14:44:37.501019+0000","last_peered":"2023-03-09T14:44:37.501019+0000","last_clean":"2023-03-09T14:44:37.501019+0000","last_became_active":"2023-03-09T14:44:18.662809+0000","last_became_peered":"2023-03-09T14:44:18.662809+0000","last_unstale":"2023-03-09T14:44:37.501019+0000","last_undegraded":"2023-03-09T14:44:37.501019+0000","last_fullsized":"2023-03-09T14:44:37.501019+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:32:16.180642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.473842+0000","last_change":"2023-03-09T14:44:19.108566+0000","last_active":"2023-03-09T14:44:23.473842+0000","last_peered":"2023-03-09T14:44:23.473842+0000","last_clean":"2023-03-09T14:44:23.473842+0000","last_became_active":"2023-03-09T14:44:19.108217+0000","last_became_peered":"2023-03-09T14:44:19.108217+0000","last_unstale":"2023-03-09T14:44:23.473842+0000","last_undegraded":"2023-03-09T14:44:23.473842+0000","last_fullsized":"2023-03-09T14:44:23.473842+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:03:28.434395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:24.249427+0000","last_change":"2023-03-09T14:44:23.488640+0000","last_active":"2023-03-09T14:44:24.249427+0000","last_peered":"2023-03-09T14:44:24.249427+0000","last_clean":"2023-03-09T14:44:24.249427+0000","last_became_active":"2023-03-09T14:44:23.488178+0000","last_became_peered":"2023-03-09T14:44:23.488178+0000","last_unstale":"2023-03-09T14:44:24.249427+0000","last_undegraded":"2023-03-09T14:44:24.249427+0000","last_fullsized":"2023-03-09T14:44:24.249427+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:33:49.908974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"87'3","reported_seq":19,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-03-09T14:44:37.502373+0000","last_change":"2023-03-09T14:44:18.662925+0000","last_active":"2023-03-09T14:44:37.502373+0000","last_peered":"2023-03-09T14:44:37.502373+0000","last_clean":"2023-03-09T14:44:37.502373+0000","last_became_active":"2023-03-09T14:44:18.662354+0000","last_became_peered":"2023-03-09T14:44:18.662354+0000","last_unstale":"2023-03-09T14:44:37.502373+0000","last_undegraded":"2023-03-09T14:44:37.502373+0000","last_fullsized":"2023-03-09T14:44:37.502373+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T01:00:36.602938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.473168+0000","last_change":"2023-03-09T14:44:19.106465+0000","last_active":"2023-03-09T14:44:23.473168+0000","last_peered":"2023-03-09T14:44:23.473168+0000","last_clean":"2023-03-09T14:44:23.473168+0000","last_became_active":"2023-03-09T14:44:19.106001+0000","last_became_peered":"2023-03-09T14:44:19.106001+0000","last_unstale":"2023-03-09T14:44:23.473168+0000","last_undegraded":"2023-03-09T14:44:23.473168+0000","last_fullsized":"2023-03-09T14:44:23.473168+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:22:09.149291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.905541+0000","last_change":"2023-03-09T14:44:23.481512+0000","last_active":"2023-03-09T14:44:23.905541+0000","last_peered":"2023-03-09T14:44:23.905541+0000","last_clean":"2023-03-09T14:44:23.905541+0000","last_became_active":"2023-03-09T14:44:23.480370+0000","last_became_peered":"2023-03-09T14:44:23.480370+0000","last_unstale":"2023-03-09T14:44:23.905541+0000","last_undegraded":"2023-03-09T14:44:23.905541+0000","last_fullsized":"2023-03-09T14:44:23.905541+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:29:16.881539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.474195+0000","last_change":"2023-03-09T14:44:18.664744+0000","last_active":"2023-03-09T14:44:23.474195+0000","last_peered":"2023-03-09T14:44:23.474195+0000","last_clean":"2023-03-09T14:44:23.474195+0000","last_became_active":"2023-03-09T14:44:18.664443+0000","last_became_peered":"2023-03-09T14:44:18.664443+0000","last_unstale":"2023-03-09T14:44:23.474195+0000","last_undegraded":"2023-03-09T14:44:23.474195+0000","last_fullsized":"2023-03-09T14:44:23.474195+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:11:13.836270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.471063+0000","last_change":"2023-03-09T14:44:19.115102+0000","last_active":"2023-03-09T14:44:23.471063+0000","last_peered":"2023-03-09T14:44:23.471063+0000","last_clean":"2023-03-09T14:44:23.471063+0000","last_became_active":"2023-03-09T14:44:19.114565+0000","last_became_peered":"2023-03-09T14:44:19.114565+0000","last_unstale":"2023-03-09T14:44:23.471063+0000","last_undegraded":"2023-03-09T14:44:23.471063+0000","last_fullsized":"2023-03-09T14:44:23.471063+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:39:11.930231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.905161+0000","last_change":"2023-03-09T14:44:23.482500+0000","last_active":"2023-03-09T14:44:23.905161+0000","last_peered":"2023-03-09T14:44:23.905161+0000","last_clean":"2023-03-09T14:44:23.905161+0000","last_became_active":"2023-03-09T14:44:23.482217+0000","last_became_peered":"2023-03-09T14:44:23.482217+0000","last_unstale":"2023-03-09T14:44:23.905161+0000","last_undegraded":"2023-03-09T14:44:23.905161+0000","last_fullsized":"2023-03-09T14:44:23.905161+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T14:52:44.275037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"82'2","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.473997+0000","last_change":"2023-03-09T14:44:18.663824+0000","last_active":"2023-03-09T14:44:23.473997+0000","last_peered":"2023-03-09T14:44:23.473997+0000","last_clean":"2023-03-09T14:44:23.473997+0000","last_became_active":"2023-03-09T14:44:18.663379+0000","last_became_peered":"2023-03-09T14:44:18.663379+0000","last_unstale":"2023-03-09T14:44:23.473997+0000","last_undegraded":"2023-03-09T14:44:23.473997+0000","last_fullsized":"2023-03-09T14:44:23.473997+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:03:04.031061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.473902+0000","last_change":"2023-03-09T14:44:19.105765+0000","last_active":"2023-03-09T14:44:23.473902+0000","last_peered":"2023-03-09T14:44:23.473902+0000","last_clean":"2023-03-09T14:44:23.473902+0000","last_became_active":"2023-03-09T14:44:19.105430+0000","last_became_peered":"2023-03-09T14:44:19.105430+0000","last_unstale":"2023-03-09T14:44:23.473902+0000","last_undegraded":"2023-03-09T14:44:23.473902+0000","last_fullsized":"2023-03-09T14:44:23.473902+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T20:31:40.106798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-03-09T14:44:24.284816+0000","last_change":"2023-03-09T14:44:23.482079+0000","last_active":"2023-03-09T14:44:24.284816+0000","last_peered":"2023-03-09T14:44:24.284816+0000","last_clean":"2023-03-09T14:44:24.284816+0000","last_became_active":"2023-03-09T14:44:23.481620+0000","last_became_peered":"2023-03-09T14:44:23.481620+0000","last_unstale":"2023-03-09T14:44:24.284816+0000","last_undegraded":"2023-03-09T14:44:24.284816+0000","last_fullsized":"2023-03-09T14:44:24.284816+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:47:53.589111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.479406+0000","last_change":"2023-03-09T14:44:18.667558+0000","last_active":"2023-03-09T14:44:23.479406+0000","last_peered":"2023-03-09T14:44:23.479406+0000","last_clean":"2023-03-09T14:44:23.479406+0000","last_became_active":"2023-03-09T14:44:18.667260+0000","last_became_peered":"2023-03-09T14:44:18.667260+0000","last_unstale":"2023-03-09T14:44:23.479406+0000","last_undegraded":"2023-03-09T14:44:23.479406+0000","last_fullsized":"2023-03-09T14:44:23.479406+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T16:58:53.434611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.475024+0000","last_change":"2023-03-09T14:44:19.105947+0000","last_active":"2023-03-09T14:44:23.475024+0000","last_peered":"2023-03-09T14:44:23.475024+0000","last_clean":"2023-03-09T14:44:23.475024+0000","last_became_active":"2023-03-09T14:44:19.105669+0000","last_became_peered":"2023-03-09T14:44:19.105669+0000","last_unstale":"2023-03-09T14:44:23.475024+0000","last_undegraded":"2023-03-09T14:44:23.475024+0000","last_fullsized":"2023-03-09T14:44:23.475024+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:29:58.615481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-03-09T14:44:24.313456+0000","last_change":"2023-03-09T14:44:23.483237+0000","last_active":"2023-03-09T14:44:24.313456+0000","last_peered":"2023-03-09T14:44:24.313456+0000","last_clean":"2023-03-09T14:44:24.313456+0000","last_became_active":"2023-03-09T14:44:23.482831+0000","last_became_peered":"2023-03-09T14:44:23.482831+0000","last_unstale":"2023-03-09T14:44:24.313456+0000","last_undegraded":"2023-03-09T14:44:24.313456+0000","last_fullsized":"2023-03-09T14:44:24.313456+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:36:40.775581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"82'1","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.480598+0000","last_change":"2023-03-09T14:44:18.660954+0000","last_active":"2023-03-09T14:44:23.480598+0000","last_peered":"2023-03-09T14:44:23.480598+0000","last_clean":"2023-03-09T14:44:23.480598+0000","last_became_active":"2023-03-09T14:44:18.660673+0000","last_became_peered":"2023-03-09T14:44:18.660673+0000","last_unstale":"2023-03-09T14:44:23.480598+0000","last_undegraded":"2023-03-09T14:44:23.480598+0000","last_fullsized":"2023-03-09T14:44:23.480598+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T19:37:58.271196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.474234+0000","last_change":"2023-03-09T14:44:19.107146+0000","last_active":"2023-03-09T14:44:23.474234+0000","last_peered":"2023-03-09T14:44:23.474234+0000","last_clean":"2023-03-09T14:44:23.474234+0000","last_became_active":"2023-03-09T14:44:19.106834+0000","last_became_peered":"2023-03-09T14:44:19.106834+0000","last_unstale":"2023-03-09T14:44:23.474234+0000","last_undegraded":"2023-03-09T14:44:23.474234+0000","last_fullsized":"2023-03-09T14:44:23.474234+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T20:15:20.142449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-03-09T14:44:24.285655+0000","last_change":"2023-03-09T14:44:23.482811+0000","last_active":"2023-03-09T14:44:24.285655+0000","last_peered":"2023-03-09T14:44:24.285655+0000","last_clean":"2023-03-09T14:44:24.285655+0000","last_became_active":"2023-03-09T14:44:23.482371+0000","last_became_peered":"2023-03-09T14:44:23.482371+0000","last_unstale":"2023-03-09T14:44:24.285655+0000","last_undegraded":"2023-03-09T14:44:24.285655+0000","last_fullsized":"2023-03-09T14:44:24.285655+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T01:33:14.442810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"82'2","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.477425+0000","last_change":"2023-03-09T14:44:18.661569+0000","last_active":"2023-03-09T14:44:23.477425+0000","last_peered":"2023-03-09T14:44:23.477425+0000","last_clean":"2023-03-09T14:44:23.477425+0000","last_became_active":"2023-03-09T14:44:18.661266+0000","last_became_peered":"2023-03-09T14:44:18.661266+0000","last_unstale":"2023-03-09T14:44:23.477425+0000","last_undegraded":"2023-03-09T14:44:23.477425+0000","last_fullsized":"2023-03-09T14:44:23.477425+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:47:43.066180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.474622+0000","last_change":"2023-03-09T14:44:19.106611+0000","last_active":"2023-03-09T14:44:23.474622+0000","last_peered":"2023-03-09T14:44:23.474622+0000","last_clean":"2023-03-09T14:44:23.474622+0000","last_became_active":"2023-03-09T14:44:19.106324+0000","last_became_peered":"2023-03-09T14:44:19.106324+0000","last_unstale":"2023-03-09T14:44:23.474622+0000","last_undegraded":"2023-03-09T14:44:23.474622+0000","last_fullsized":"2023-03-09T14:44:23.474622+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:50:46.561502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.472996+0000","last_change":"2023-03-09T14:44:19.106292+0000","last_active":"2023-03-09T14:44:23.472996+0000","last_peered":"2023-03-09T14:44:23.472996+0000","last_clean":"2023-03-09T14:44:23.472996+0000","last_became_active":"2023-03-09T14:44:19.105607+0000","last_became_peered":"2023-03-09T14:44:19.105607+0000","last_unstale":"2023-03-09T14:44:23.472996+0000","last_undegraded":"2023-03-09T14:44:23.472996+0000","last_fullsized":"2023-03-09T14:44:23.472996+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T19:11:28.125433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.479822+0000","last_change":"2023-03-09T14:44:19.103423+0000","last_active":"2023-03-09T14:44:23.479822+0000","last_peered":"2023-03-09T14:44:23.479822+0000","last_clean":"2023-03-09T14:44:23.479822+0000","last_became_active":"2023-03-09T14:44:19.102918+0000","last_became_peered":"2023-03-09T14:44:19.102918+0000","last_unstale":"2023-03-09T14:44:23.479822+0000","last_undegraded":"2023-03-09T14:44:23.479822+0000","last_fullsized":"2023-03-09T14:44:23.479822+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:47:52.921642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.474107+0000","last_change":"2023-03-09T14:44:19.113511+0000","last_active":"2023-03-09T14:44:23.474107+0000","last_peered":"2023-03-09T14:44:23.474107+0000","last_clean":"2023-03-09T14:44:23.474107+0000","last_became_active":"2023-03-09T14:44:19.113224+0000","last_became_peered":"2023-03-09T14:44:19.113224+0000","last_unstale":"2023-03-09T14:44:23.474107+0000","last_undegraded":"2023-03-09T14:44:23.474107+0000","last_fullsized":"2023-03-09T14:44:23.474107+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T02:03:28.069616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.475109+0000","last_change":"2023-03-09T14:44:19.116174+0000","last_active":"2023-03-09T14:44:23.475109+0000","last_peered":"2023-03-09T14:44:23.475109+0000","last_clean":"2023-03-09T14:44:23.475109+0000","last_became_active":"2023-03-09T14:44:19.115813+0000","last_became_peered":"2023-03-09T14:44:19.115813+0000","last_unstale":"2023-03-09T14:44:23.475109+0000","last_undegraded":"2023-03-09T14:44:23.475109+0000","last_fullsized":"2023-03-09T14:44:23.475109+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:28:49.573771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.476310+0000","last_change":"2023-03-09T14:44:19.115287+0000","last_active":"2023-03-09T14:44:23.476310+0000","last_peered":"2023-03-09T14:44:23.476310+0000","last_clean":"2023-03-09T14:44:23.476310+0000","last_became_active":"2023-03-09T14:44:19.113458+0000","last_became_peered":"2023-03-09T14:44:19.113458+0000","last_unstale":"2023-03-09T14:44:23.476310+0000","last_undegraded":"2023-03-09T14:44:23.476310+0000","last_fullsized":"2023-03-09T14:44:23.476310+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:43:26.583237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.476398+0000","last_change":"2023-03-09T14:44:19.107308+0000","last_active":"2023-03-09T14:44:23.476398+0000","last_peered":"2023-03-09T14:44:23.476398+0000","last_clean":"2023-03-09T14:44:23.476398+0000","last_became_active":"2023-03-09T14:44:19.107005+0000","last_became_peered":"2023-03-09T14:44:19.107005+0000","last_unstale":"2023-03-09T14:44:23.476398+0000","last_undegraded":"2023-03-09T14:44:23.476398+0000","last_fullsized":"2023-03-09T14:44:23.476398+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:48:34.886664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.475838+0000","last_change":"2023-03-09T14:44:19.098454+0000","last_active":"2023-03-09T14:44:23.475838+0000","last_peered":"2023-03-09T14:44:23.475838+0000","last_clean":"2023-03-09T14:44:23.475838+0000","last_became_active":"2023-03-09T14:44:19.098146+0000","last_became_peered":"2023-03-09T14:44:19.098146+0000","last_unstale":"2023-03-09T14:44:23.475838+0000","last_undegraded":"2023-03-09T14:44:23.475838+0000","last_fullsized":"2023-03-09T14:44:23.475838+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:11:40.475674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.471936+0000","last_change":"2023-03-09T14:44:19.095947+0000","last_active":"2023-03-09T14:44:23.471936+0000","last_peered":"2023-03-09T14:44:23.471936+0000","last_clean":"2023-03-09T14:44:23.471936+0000","last_became_active":"2023-03-09T14:44:19.095640+0000","last_became_peered":"2023-03-09T14:44:19.095640+0000","last_unstale":"2023-03-09T14:44:23.471936+0000","last_undegraded":"2023-03-09T14:44:23.471936+0000","last_fullsized":"2023-03-09T14:44:23.471936+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T02:04:35.384016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.474791+0000","last_change":"2023-03-09T14:44:19.113927+0000","last_active":"2023-03-09T14:44:23.474791+0000","last_peered":"2023-03-09T14:44:23.474791+0000","last_clean":"2023-03-09T14:44:23.474791+0000","last_became_active":"2023-03-09T14:44:19.113163+0000","last_became_peered":"2023-03-09T14:44:19.113163+0000","last_unstale":"2023-03-09T14:44:23.474791+0000","last_undegraded":"2023-03-09T14:44:23.474791+0000","last_fullsized":"2023-03-09T14:44:23.474791+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T01:35:35.801317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.470072+0000","last_change":"2023-03-09T14:44:19.098415+0000","last_active":"2023-03-09T14:44:23.470072+0000","last_peered":"2023-03-09T14:44:23.470072+0000","last_clean":"2023-03-09T14:44:23.470072+0000","last_became_active":"2023-03-09T14:44:19.097600+0000","last_became_peered":"2023-03-09T14:44:19.097600+0000","last_unstale":"2023-03-09T14:44:23.470072+0000","last_undegraded":"2023-03-09T14:44:23.470072+0000","last_fullsized":"2023-03-09T14:44:23.470072+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:15:08.552402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.475985+0000","last_change":"2023-03-09T14:44:19.106386+0000","last_active":"2023-03-09T14:44:23.475985+0000","last_peered":"2023-03-09T14:44:23.475985+0000","last_clean":"2023-03-09T14:44:23.475985+0000","last_became_active":"2023-03-09T14:44:19.106112+0000","last_became_peered":"2023-03-09T14:44:19.106112+0000","last_unstale":"2023-03-09T14:44:23.475985+0000","last_undegraded":"2023-03-09T14:44:23.475985+0000","last_fullsized":"2023-03-09T14:44:23.475985+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T20:27:50.992168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.477821+0000","last_change":"2023-03-09T14:44:19.114534+0000","last_active":"2023-03-09T14:44:23.477821+0000","last_peered":"2023-03-09T14:44:23.477821+0000","last_clean":"2023-03-09T14:44:23.477821+0000","last_became_active":"2023-03-09T14:44:19.114226+0000","last_became_peered":"2023-03-09T14:44:19.114226+0000","last_unstale":"2023-03-09T14:44:23.477821+0000","last_undegraded":"2023-03-09T14:44:23.477821+0000","last_fullsized":"2023-03-09T14:44:23.477821+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:22:38.453116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-03-09T14:44:23.474647+0000","last_change":"2023-03-09T14:44:19.099195+0000","last_active":"2023-03-09T14:44:23.474647+0000","last_peered":"2023-03-09T14:44:23.474647+0000","last_clean":"2023-03-09T14:44:23.474647+0000","last_became_active":"2023-03-09T14:44:19.098892+0000","last_became_peered":"2023-03-09T14:44:19.098892+0000","last_unstale":"2023-03-09T14:44:23.474647+0000","last_undegraded":"2023-03-09T14:44:23.474647+0000","last_fullsized":"2023-03-09T14:44:23.474647+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:29:24.858200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":22147,"num_objects":36,"num_object_clones":0,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":0,"num_read":5,"num_read_kb":7,"num_write":91,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":25,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":212992,"data_stored":52572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77,"ondisk_log_size":77,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":78,"seq":335007449103,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6064,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739088,"statfs":{"total":95995035648,"available":95988826112,"internally_reserved":0,"allocated":958464,"data_stored":467097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.755}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40300000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46400000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69899999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48399999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45000000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79400000000000004}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72699999999999998}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78100000000000003}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68899999999999995}]}]},{"osd":10,"up_from":70,"seq":300647710738,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6940,"kb_used_data":1556,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738212,"statfs":{"total":95995035648,"available":95987929088,"internally_reserved":0,"allocated":1593344,"data_stored":1062490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60399999999999998}]},{"osd":1,"last update":"Thu Mar 9 14:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.34899999999999998}]},{"osd":2,"last update":"Thu Mar 9 14:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.51900000000000002}]},{"osd":3,"last update":"Thu Mar 9 14:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.2769999999999999,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.505}]},{"osd":4,"last update":"Thu Mar 9 14:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.1719999999999999,"5min":1.1719999999999999,"15min":1.1719999999999999},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.39800000000000002}]},{"osd":5,"last update":"Thu Mar 9 14:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.63},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.56100000000000005}]},{"osd":6,"last update":"Thu Mar 9 14:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.57399999999999995}]},{"osd":7,"last update":"Thu Mar 9 14:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.327,"5min":1.327,"15min":1.327},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.048,"5min":1.048,"15min":1.048},"last":0.53800000000000003}]},{"osd":8,"last update":"Thu Mar 9 14:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.246,"5min":1.246,"15min":1.246},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.67800000000000005}]},{"osd":9,"last update":"Thu Mar 9 14:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.298,"5min":1.298,"15min":1.298},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.4630000000000001,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":0.70399999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72699999999999998}]}]},{"osd":9,"up_from":65,"seq":279172874263,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6276,"kb_used_data":956,"kb_used_omap":0,"kb_used_meta":5312,"kb_avail":93738876,"statfs":{"total":95995035648,"available":95988609024,"internally_reserved":0,"allocated":978944,"data_stored":467899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5439488},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:43:55 2023","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.36199999999999999}]},{"osd":1,"last update":"Thu Mar 9 14:43:55 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.59499999999999997}]},{"osd":2,"last update":"Thu Mar 9 14:43:55 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.48999999999999999}]},{"osd":3,"last update":"Thu Mar 9 14:43:55 2023","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":0.33700000000000002}]},{"osd":4,"last update":"Thu Mar 9 14:43:55 2023","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.47899999999999998}]},{"osd":5,"last update":"Thu Mar 9 14:43:55 2023","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":0.69399999999999995}]},{"osd":6,"last update":"Thu Mar 9 14:43:55 2023","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.623}]},{"osd":7,"last update":"Thu Mar 9 14:43:55 2023","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.1180000000000001,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.627}]},{"osd":8,"last update":"Thu Mar 9 14:43:55 2023","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.133,"5min":1.133,"15min":1.133},"last":0.751},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.61699999999999999}]},{"osd":10,"last update":"Thu Mar 9 14:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.57399999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66700000000000004}]}]},{"osd":8,"up_from":57,"seq":244813135899,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6388,"kb_used_data":940,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738764,"statfs":{"total":95995035648,"available":95988494336,"internally_reserved":0,"allocated":962560,"data_stored":467354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.33300000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.69199999999999995,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54900000000000004,"15min":0.55100000000000005},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.83799999999999997,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.59399999999999997}]},{"osd":1,"last update":"Thu Mar 9 14:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.54700000000000004,"15min":0.53900000000000003},"min":{"1min":0.376,"5min":0.313,"15min":0.313},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.91400000000000003},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.56100000000000005,"15min":0.55900000000000005},"min":{"1min":0.34399999999999997,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.70499999999999996}]},{"osd":2,"last update":"Thu Mar 9 14:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.42199999999999999,"15min":0.41599999999999998},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.432,"15min":0.42699999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.66900000000000004}]},{"osd":3,"last update":"Thu Mar 9 14:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.55900000000000005,"15min":0.56100000000000005},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.81200000000000006,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.752},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.63100000000000001,"15min":0.61499999999999999},"min":{"1min":0.40600000000000003,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":3.9300000000000002,"5min":3.9300000000000002,"15min":3.9300000000000002},"last":0.86599999999999999}]},{"osd":4,"last update":"Thu Mar 9 14:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.56399999999999995,"15min":0.55600000000000005},"min":{"1min":0.36599999999999999,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.93799999999999994},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.59299999999999997,"15min":0.58499999999999996},"min":{"1min":0.45600000000000002,"5min":0.376,"15min":0.376},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":1.0009999999999999}]},{"osd":5,"last update":"Thu Mar 9 14:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.53400000000000003,"15min":0.53200000000000003},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.54200000000000004,"15min":0.53600000000000003},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.78000000000000003}]},{"osd":6,"last update":"Thu Mar 9 14:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.69599999999999995,"15min":0.67900000000000005},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":3.895,"5min":3.895,"15min":3.895},"last":0.95099999999999996},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.65000000000000002,"15min":0.64900000000000002},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.89000000000000001}]},{"osd":7,"last update":"Thu Mar 9 14:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.66400000000000003,"15min":0.66200000000000003},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.92600000000000005},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.66800000000000004,"15min":0.66300000000000003},"min":{"1min":0.49399999999999999,"5min":0.44,"15min":0.44},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":1.0609999999999999}]},{"osd":9,"last update":"Thu Mar 9 14:43:55 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":1.036},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":1.119}]},{"osd":10,"last update":"Thu Mar 9 14:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":1.099},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":1.1439999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98499999999999999}]}]},{"osd":7,"up_from":50,"seq":214748364832,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7032,"kb_used_data":1520,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738120,"statfs":{"total":95995035648,"available":95987834880,"internally_reserved":0,"allocated":1556480,"data_stored":1058031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:44:13 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.52100000000000002,"15min":0.51700000000000002},"min":{"1min":0.316,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.69999999999999996,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.375},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.51900000000000002,"15min":0.50900000000000001},"min":{"1min":0.32800000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.90100000000000002,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.33900000000000002}]},{"osd":1,"last update":"Thu Mar 9 14:44:13 2023","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.40300000000000002,"15min":0.39300000000000002},"min":{"1min":0.29799999999999999,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.39700000000000002,"15min":0.38800000000000001},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.45300000000000001}]},{"osd":2,"last update":"Thu Mar 9 14:44:13 2023","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.53600000000000003,"15min":0.53000000000000003},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.80700000000000005,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.754},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.57399999999999995,"15min":0.56699999999999995},"min":{"1min":0.42199999999999999,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.66900000000000004}]},{"osd":3,"last update":"Thu Mar 9 14:44:13 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.54600000000000004,"15min":0.53400000000000003},"min":{"1min":0.39600000000000002,"5min":0.37,"15min":0.37},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.57799999999999996,"15min":0.56399999999999995},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.57999999999999996}]},{"osd":4,"last update":"Thu Mar 9 14:44:13 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.48299999999999998,"15min":0.47099999999999997},"min":{"1min":0.34899999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.47499999999999998,"15min":0.45900000000000002},"min":{"1min":0.36199999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.50700000000000001}]},{"osd":5,"last update":"Thu Mar 9 14:44:13 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.58099999999999996,"15min":0.57099999999999995},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.58399999999999996,"15min":0.57399999999999995},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.78500000000000003}]},{"osd":6,"last update":"Thu Mar 9 14:44:13 2023","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.627,"15min":0.61499999999999999},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.624,"15min":0.61299999999999999},"min":{"1min":0.49199999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.59499999999999997}]},{"osd":8,"last update":"Thu Mar 9 14:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.65400000000000003,"15min":0.65500000000000003},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.85199999999999998,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.64900000000000002,"15min":0.64700000000000002},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.88300000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.68200000000000005}]},{"osd":9,"last update":"Thu Mar 9 14:43:53 2023","interfaces":[{"interface":"back","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.77100000000000002}]},{"osd":10,"last update":"Thu Mar 9 14:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.72699999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86699999999999999}]}]},{"osd":6,"up_from":45,"seq":193273528357,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6388,"kb_used_data":940,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":93738764,"statfs":{"total":95995035648,"available":95988494336,"internally_reserved":0,"allocated":962560,"data_stored":467187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:43:54 2023","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.41799999999999998,"15min":0.41399999999999998},"min":{"1min":0.27400000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.499,"15min":0.49299999999999999},"min":{"1min":0.34200000000000003,"5min":0.318,"15min":0.318},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.54300000000000004}]},{"osd":1,"last update":"Thu Mar 9 14:43:54 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.60099999999999998,"15min":0.60499999999999998},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.67900000000000005}]},{"osd":2,"last update":"Thu Mar 9 14:43:54 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.63,"15min":0.64400000000000002},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.76700000000000002,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.621,"5min":0.63500000000000001,"15min":0.63700000000000001},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.73199999999999998}]},{"osd":3,"last update":"Thu Mar 9 14:43:54 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.46600000000000003,"15min":0.46300000000000002},"min":{"1min":0.311,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.441},{"interface":"front","average":{"1min":0.625,"5min":0.56899999999999995,"15min":0.55900000000000005},"min":{"1min":0.42499999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.58699999999999997}]},{"osd":4,"last update":"Thu Mar 9 14:43:54 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.73299999999999998,"15min":0.753},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.80700000000000005,"5min":2.0840000000000001,"15min":2.0840000000000001},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.88200000000000001,"15min":0.91600000000000004},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.91600000000000004,"5min":4.1440000000000001,"15min":4.1440000000000001},"last":0.60299999999999998}]},{"osd":5,"last update":"Thu Mar 9 14:43:54 2023","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.70699999999999996,"15min":0.70299999999999996},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.627},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.67600000000000005,"15min":0.68700000000000006},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.79300000000000004,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.70099999999999996}]},{"osd":7,"last update":"Thu Mar 9 14:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.69999999999999996,"15min":0.70899999999999996},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.72999999999999998,"15min":0.73599999999999999},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.96499999999999997,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.71699999999999997}]},{"osd":8,"last update":"Thu Mar 9 14:44:35 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.77400000000000002,"15min":0.78800000000000003},"min":{"1min":0.54200000000000004,"5min":0.497,"15min":0.497},"max":{"1min":0.92500000000000004,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.70599999999999996,"15min":0.72199999999999998},"min":{"1min":0.502,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.81200000000000006,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.73999999999999999}]},{"osd":9,"last update":"Thu Mar 9 14:43:54 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.66700000000000004}]},{"osd":10,"last update":"Thu Mar 9 14:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.75800000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82199999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574905,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7292,"kb_used_data":948,"kb_used_omap":0,"kb_used_meta":6336,"kb_avail":93737860,"statfs":{"total":95995035648,"available":95987568640,"internally_reserved":0,"allocated":970752,"data_stored":470891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6488064},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:44:06 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.5,"15min":0.48999999999999999},"min":{"1min":0.34399999999999997,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.51800000000000002,"15min":0.48599999999999999},"min":{"1min":0.40999999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.90700000000000003,"5min":2.0670000000000002,"15min":2.0670000000000002},"last":0.69799999999999995}]},{"osd":2,"last update":"Thu Mar 9 14:44:31 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.54800000000000004,"15min":0.53000000000000003},"min":{"1min":0.38100000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.85399999999999998,"5min":1.3839999999999999,"15min":1.3839999999999999},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.52700000000000002,"15min":0.50700000000000001},"min":{"1min":0.376,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.77600000000000002,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.73599999999999999}]},{"osd":3,"last update":"Thu Mar 9 14:43:51 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.53600000000000003,"15min":0.504},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.81799999999999995,"5min":2.206,"15min":2.206},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.53300000000000003,"15min":0.51600000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.754}]},{"osd":4,"last update":"Thu Mar 9 14:44:15 2023","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.40100000000000002,"15min":0.38100000000000001},"min":{"1min":0.308,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.39800000000000002,"15min":0.378},"min":{"1min":0.27100000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.60899999999999999}]},{"osd":5,"last update":"Thu Mar 9 14:44:31 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.57299999999999995,"15min":0.54600000000000004},"min":{"1min":0.47999999999999998,"5min":0.373,"15min":0.373},"max":{"1min":0.92800000000000005,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.57599999999999996,"15min":0.54600000000000004},"min":{"1min":0.504,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.85799999999999998,"5min":1.363,"15min":1.363},"last":0.77100000000000002}]},{"osd":6,"last update":"Thu Mar 9 14:43:51 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.65100000000000002,"15min":0.65500000000000003},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.87,"5min":2.1930000000000001,"15min":2.1930000000000001},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.63,"5min":0.66000000000000003,"15min":0.66500000000000004},"min":{"1min":0.42999999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.97999999999999998,"5min":2.2519999999999998,"15min":2.2519999999999998},"last":0.71099999999999997}]},{"osd":7,"last update":"Thu Mar 9 14:44:10 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.501,"15min":0.48299999999999998},"min":{"1min":0.36399999999999999,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.51900000000000002,"15min":0.505},"min":{"1min":0.316,"5min":0.31,"15min":0.31},"max":{"1min":0.78400000000000003,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.78700000000000003}]},{"osd":8,"last update":"Thu Mar 9 14:44:33 2023","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.624,"15min":0.61199999999999999},"min":{"1min":0.503,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.88100000000000001,"5min":1.3009999999999999,"15min":1.3009999999999999},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.68100000000000005,"15min":0.67100000000000004},"min":{"1min":0.496,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.98299999999999998,"5min":1.423,"15min":1.423},"last":1.002}]},{"osd":9,"last update":"Thu Mar 9 14:43:57 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.94399999999999995},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.877}]},{"osd":10,"last update":"Thu Mar 9 14:44:19 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.95999999999999996},{"interface":"front","average":{"1min":0.752,"5min":0.752,"15min":0.752},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.98199999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0349999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738431,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7308,"kb_used_data":964,"kb_used_omap":0,"kb_used_meta":6336,"kb_avail":93737844,"statfs":{"total":95995035648,"available":95987552256,"internally_reserved":0,"allocated":987136,"data_stored":468543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6488064},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Mar 9 14:44:08 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.54000000000000004,"15min":0.51800000000000002},"min":{"1min":0.47099999999999997,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.54000000000000004,"15min":0.52600000000000002},"min":{"1min":0.36099999999999999,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.54500000000000004}]},{"osd":2,"last update":"Thu Mar 9 14:44:31 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.55800000000000005,"15min":0.55300000000000005},"min":{"1min":0.41599999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.71399999999999997,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.59999999999999998,"15min":0.57999999999999996},"min":{"1min":0.49199999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.76000000000000001,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.63500000000000001}]},{"osd":3,"last update":"Thu Mar 9 14:42:45 2023","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.35799999999999998,"15min":0.34999999999999998},"min":{"1min":0.27800000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.311},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.39200000000000002,"15min":0.38800000000000001},"min":{"1min":0.26400000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.59499999999999997,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.35699999999999998}]},{"osd":4,"last update":"Thu Mar 9 14:44:08 2023","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.59699999999999998,"15min":0.57099999999999995},"min":{"1min":0.42099999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.139,"5min":1.139,"15min":1.139},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.55400000000000005,"15min":0.54400000000000004},"min":{"1min":0.44900000000000001,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.81799999999999995,"5min":1.071,"15min":1.071},"last":0.60599999999999998}]},{"osd":5,"last update":"Thu Mar 9 14:44:37 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.58499999999999996,"15min":0.56299999999999994},"min":{"1min":0.48599999999999999,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.748,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.64500000000000002,"15min":0.626},"min":{"1min":0.46999999999999997,"5min":0.378,"15min":0.378},"max":{"1min":0.78600000000000003,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.71699999999999997}]},{"osd":6,"last update":"Thu Mar 9 14:43:51 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.52600000000000002,"15min":0.501},"min":{"1min":0.443,"5min":0.309,"15min":0.309},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.58699999999999997,"15min":0.55100000000000005},"min":{"1min":0.48499999999999999,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.159,"5min":1.159,"15min":1.159},"last":0.39400000000000002}]},{"osd":7,"last update":"Thu Mar 9 14:44:14 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.71299999999999997,"15min":0.71099999999999997},"min":{"1min":0.503,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.0760000000000001,"5min":1.2110000000000001,"15min":1.2110000000000001},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.64900000000000002,"15min":0.64500000000000002},"min":{"1min":0.52700000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.88500000000000001,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.69499999999999995}]},{"osd":8,"last update":"Thu Mar 9 14:44:37 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.72199999999999998,"15min":0.73299999999999998},"min":{"1min":0.57299999999999995,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.77900000000000003,"5min":1.038,"15min":1.038},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.80600000000000005,"15min":0.82299999999999995},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.81399999999999995,"5min":1.242,"15min":1.242},"last":0.67300000000000004}]},{"osd":9,"last update":"Thu Mar 9 14:43:55 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.105,"5min":1.105,"15min":1.105},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.87,"5min":0.87,"15min":0.87},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.27,"5min":1.27,"15min":1.27},"last":0.46700000000000003}]},{"osd":10,"last update":"Thu Mar 9 14:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.151,"5min":1.151,"15min":1.151},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.55300000000000005}]},{"osd":11,"last update":"Thu Mar 9 14:44:37 2023","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.64800000000000002}]}]},{"osd":2,"up_from":18,"seq":77309411382,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7368,"kb_used_data":960,"kb_used_omap":0,"kb_used_meta":6400,"kb_avail":93737784,"statfs":{"total":95995035648,"available":95987490816,"internally_reserved":0,"allocated":983040,"data_stored":471547,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6553600},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:44:30 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.51500000000000001,"15min":0.495},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.47899999999999998,"15min":0.46899999999999997},"min":{"1min":0.30199999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.52600000000000002}]},{"osd":1,"last update":"Thu Mar 9 14:44:30 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.53500000000000003,"15min":0.499},"min":{"1min":0.30199999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.54600000000000004,"15min":0.505},"min":{"1min":0.41799999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":1.1890000000000001,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.85699999999999998}]},{"osd":3,"last update":"Thu Mar 9 14:43:48 2023","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.55400000000000005,"15min":0.53400000000000003},"min":{"1min":0.39300000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.53800000000000003,"15min":0.53100000000000003},"min":{"1min":0.39800000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.77700000000000002,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.63500000000000001}]},{"osd":4,"last update":"Thu Mar 9 14:44:15 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.54900000000000004,"15min":0.52500000000000002},"min":{"1min":0.41599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":0.89300000000000002},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.56699999999999995,"15min":0.54500000000000004},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.234,"5min":1.234,"15min":1.234},"last":0.65900000000000003}]},{"osd":5,"last update":"Thu Mar 9 14:44:35 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.47199999999999998,"15min":0.46000000000000002},"min":{"1min":0.313,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.45200000000000001,"15min":0.44},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.747}]},{"osd":6,"last update":"Thu Mar 9 14:43:51 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.58699999999999997,"15min":0.57699999999999996},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.61799999999999999,"15min":0.61199999999999999},"min":{"1min":0.49199999999999999,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.76700000000000002}]},{"osd":7,"last update":"Thu Mar 9 14:44:15 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.2450000000000001,"5min":1.2450000000000001,"15min":1.2450000000000001},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.69699999999999995,"15min":0.69299999999999995},"min":{"1min":0.46100000000000002,"5min":0.375,"15min":0.375},"max":{"1min":1.2649999999999999,"5min":1.2649999999999999,"15min":1.2649999999999999},"last":0.93500000000000005}]},{"osd":8,"last update":"Thu Mar 9 14:44:35 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.65400000000000003,"15min":0.65100000000000002},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.94699999999999995,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.64600000000000002,"15min":0.65200000000000002},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.93999999999999995,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.83799999999999997}]},{"osd":9,"last update":"Thu Mar 9 14:43:53 2023","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.21,"5min":1.21,"15min":1.21},"last":0.878},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.72999999999999998}]},{"osd":10,"last update":"Thu Mar 9 14:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.2529999999999999,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.306,"5min":1.306,"15min":1.306},"last":0.91900000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96999999999999997}]}]},{"osd":3,"up_from":25,"seq":107374182449,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6652,"kb_used_data":948,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738500,"statfs":{"total":95995035648,"available":95988224000,"internally_reserved":0,"allocated":970752,"data_stored":467291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:43:48 2023","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.34300000000000003,"15min":0.32100000000000001},"min":{"1min":0.28899999999999998,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.36399999999999999,"15min":0.33800000000000002},"min":{"1min":0.32800000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.374}]},{"osd":1,"last update":"Thu Mar 9 14:43:48 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.51300000000000001,"15min":0.499},"min":{"1min":0.378,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.79000000000000004,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.503},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.52200000000000002,"15min":0.48599999999999999},"min":{"1min":0.36099999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.60599999999999998}]},{"osd":2,"last update":"Thu Mar 9 14:43:48 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.53800000000000003,"15min":0.51000000000000001},"min":{"1min":0.40500000000000003,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.54900000000000004,"15min":0.52600000000000002},"min":{"1min":0.38500000000000001,"5min":0.312,"15min":0.312},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.58399999999999996}]},{"osd":4,"last update":"Thu Mar 9 14:44:10 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.56599999999999995,"15min":0.53700000000000003},"min":{"1min":0.52300000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.624},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.58099999999999996,"15min":0.54300000000000004},"min":{"1min":0.46800000000000003,"5min":0.373,"15min":0.373},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.68400000000000005}]},{"osd":5,"last update":"Thu Mar 9 14:44:32 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.56699999999999995,"15min":0.54300000000000004},"min":{"1min":0.433,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.60099999999999998,"15min":0.56499999999999995},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.64700000000000002}]},{"osd":6,"last update":"Thu Mar 9 14:43:49 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.55500000000000005,"15min":0.53300000000000003},"min":{"1min":0.35399999999999998,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.56699999999999995,"15min":0.54100000000000004},"min":{"1min":0.46999999999999997,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.441}]},{"osd":7,"last update":"Thu Mar 9 14:44:15 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.624,"15min":0.61799999999999999},"min":{"1min":0.46600000000000003,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.68000000000000005,"15min":0.66700000000000004},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.77200000000000002}]},{"osd":8,"last update":"Thu Mar 9 14:43:35 2023","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.749}]},{"osd":9,"last update":"Thu Mar 9 14:43:53 2023","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.59399999999999997}]},{"osd":10,"last update":"Thu Mar 9 14:44:20 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.155,"5min":1.155,"15min":1.155},"last":0.79500000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67700000000000005}]}]},{"osd":4,"up_from":32,"seq":137438953517,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6516,"kb_used_data":940,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738636,"statfs":{"total":95995035648,"available":95988363264,"internally_reserved":0,"allocated":962560,"data_stored":467119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:44:09 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.53100000000000003,"15min":0.495},"min":{"1min":0.39000000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.84099999999999997,"5min":1.8320000000000001,"15min":1.8320000000000001},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.628,"5min":0.52900000000000003,"15min":0.495},"min":{"1min":0.42499999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":1.044,"5min":1.478,"15min":1.478},"last":0.52500000000000002}]},{"osd":1,"last update":"Thu Mar 9 14:44:09 2023","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.36299999999999999,"15min":0.34300000000000003},"min":{"1min":0.27800000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.38500000000000001,"15min":0.36499999999999999},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.39900000000000002}]},{"osd":2,"last update":"Thu Mar 9 14:44:09 2023","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.53300000000000003,"15min":0.52700000000000002},"min":{"1min":0.438,"5min":0.375,"15min":0.375},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.55400000000000005,"15min":0.53300000000000003},"min":{"1min":0.45300000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.72999999999999998}]},{"osd":3,"last update":"Thu Mar 9 14:44:09 2023","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.56799999999999995,"15min":0.52800000000000002},"min":{"1min":0.53900000000000003,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":1.099,"5min":1.4510000000000001,"15min":1.4510000000000001},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.56399999999999995,"15min":0.53000000000000003},"min":{"1min":0.42799999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.91500000000000004,"5min":1.8640000000000001,"15min":1.8640000000000001},"last":0.504}]},{"osd":5,"last update":"Thu Mar 9 14:44:33 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.58099999999999996,"15min":0.55900000000000005},"min":{"1min":0.45800000000000002,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.59399999999999997,"15min":0.56599999999999995},"min":{"1min":0.47099999999999997,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.71799999999999997}]},{"osd":6,"last update":"Thu Mar 9 14:43:49 2023","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.68600000000000005,"15min":0.69399999999999995},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.81100000000000005,"5min":1.889,"15min":1.889},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.67400000000000004,"15min":0.67500000000000004},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.03,"5min":1.8480000000000001,"15min":1.8480000000000001},"last":0.52100000000000002}]},{"osd":7,"last update":"Thu Mar 9 14:44:12 2023","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.55500000000000005,"15min":0.52900000000000003},"min":{"1min":0.35299999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.56399999999999995,"15min":0.53400000000000003},"min":{"1min":0.37,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.53500000000000003}]},{"osd":8,"last update":"Thu Mar 9 14:43:36 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.79700000000000004}]},{"osd":9,"last update":"Thu Mar 9 14:43:58 2023","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.752,"15min":0.752},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.145,"5min":1.145,"15min":1.145},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.55200000000000005}]},{"osd":10,"last update":"Thu Mar 9 14:44:20 2023","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.228,"5min":1.228,"15min":1.228},"last":0.627}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75900000000000001}]}]},{"osd":5,"up_from":37,"seq":158913789993,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7164,"kb_used_data":1524,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93737988,"statfs":{"total":95995035648,"available":95987699712,"internally_reserved":0,"allocated":1560576,"data_stored":1057316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.51200000000000001,"15min":0.48599999999999999},"min":{"1min":0.35599999999999998,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.80000000000000004,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.51900000000000002,"15min":0.5},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.69899999999999995,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.29299999999999998}]},{"osd":1,"last update":"Thu Mar 9 14:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55500000000000005,"15min":0.54700000000000004},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.83599999999999997,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.54300000000000004,"15min":0.53100000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.52100000000000002}]},{"osd":2,"last update":"Thu Mar 9 14:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.375,"15min":0.35999999999999999},"min":{"1min":0.249,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.60099999999999998,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.38900000000000001,"15min":0.36899999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.39700000000000002}]},{"osd":3,"last update":"Thu Mar 9 14:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.54000000000000004,"15min":0.53500000000000003},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.71399999999999997,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56699999999999995,"15min":0.55100000000000005},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.752,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.313}]},{"osd":4,"last update":"Thu Mar 9 14:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.56999999999999995,"15min":0.55600000000000005},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.76200000000000001,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53000000000000003,"15min":0.52600000000000002},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.70399999999999996,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.55100000000000005}]},{"osd":6,"last update":"Thu Mar 9 14:43:49 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"min":{"1min":0.498,"5min":0.315,"15min":0.315},"max":{"1min":0.72899999999999998,"5min":0.879,"15min":0.879},"last":0.503},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.63,"15min":0.624},"min":{"1min":0.39500000000000002,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.42899999999999999}]},{"osd":7,"last update":"Thu Mar 9 14:44:15 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.59099999999999997,"15min":0.58699999999999997},"min":{"1min":0.437,"5min":0.38,"15min":0.38},"max":{"1min":0.81000000000000005,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.59499999999999997,"15min":0.59199999999999997},"min":{"1min":0.41699999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.78700000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.54100000000000004}]},{"osd":8,"last update":"Thu Mar 9 14:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.59499999999999997,"15min":0.59899999999999998},"min":{"1min":0.35099999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.84099999999999997,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.54200000000000004,"15min":0.54400000000000004},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.45400000000000001}]},{"osd":9,"last update":"Thu Mar 9 14:43:56 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"min":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.38500000000000001}]},{"osd":10,"last update":"Thu Mar 9 14:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.63400000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":21095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":588,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":23047,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-03-09T14:44:42.704 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-03-09T14:44:42.704 INFO:tasks.cephadm:Setting up client nodes... 2023-03-09T14:44:42.705 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2023-03-09T14:44:43.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:42 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/43017438' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-03-09T14:44:43.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:42 smithi038 ceph-mon[92422]: from='client.14910 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-03-09T14:44:43.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:42 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/43017438' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-03-09T14:44:43.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:42 smithi122 ceph-mon[93869]: from='client.14910 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-03-09T14:44:43.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:42 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/43017438' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-03-09T14:44:43.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:42 smithi181 ceph-mon[94189]: from='client.14910 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-03-09T14:44:43.389 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.a/config 2023-03-09T14:44:43.946 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:43 smithi038 ceph-mon[92422]: from='client.14916 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-03-09T14:44:43.947 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:43 smithi038 ceph-mon[92422]: pgmap v271: 129 pgs: 129 active+clean; 601 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4.0 KiB/s wr, 9 op/s 2023-03-09T14:44:44.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:43 smithi122 ceph-mon[93869]: from='client.14916 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-03-09T14:44:44.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:43 smithi122 ceph-mon[93869]: pgmap v271: 129 pgs: 129 active+clean; 601 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4.0 KiB/s wr, 9 op/s 2023-03-09T14:44:44.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:43 smithi181 ceph-mon[94189]: from='client.14916 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-03-09T14:44:44.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:43 smithi181 ceph-mon[94189]: pgmap v271: 129 pgs: 129 active+clean; 601 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 4.0 KiB/s wr, 9 op/s 2023-03-09T14:44:44.694 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:44.690+0000 7f7a830bd700 1 -- 172.21.15.38:0/1235095663 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7a7c101b50 msgr2=0x7f7a7c108410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:44.694 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:44.690+0000 7f7a830bd700 1 --2- 172.21.15.38:0/1235095663 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7a7c101b50 0x7f7a7c108410 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f7a7400b0a0 tx=0x7f7a7400b3b0 comp rx=0 tx=0).stop 2023-03-09T14:44:44.694 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:44.690+0000 7f7a830bd700 1 -- 172.21.15.38:0/1235095663 shutdown_connections 2023-03-09T14:44:44.695 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:44.690+0000 7f7a830bd700 1 --2- 172.21.15.38:0/1235095663 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7a7c101b50 0x7f7a7c108410 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:44.695 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:44.690+0000 7f7a830bd700 1 --2- 172.21.15.38:0/1235095663 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7a7c0fffd0 0x7f7a7c1003e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:44.695 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:44.690+0000 7f7a830bd700 1 --2- 172.21.15.38:0/1235095663 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7a7c1011c0 0x7f7a7c101610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:44.695 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:44.690+0000 7f7a830bd700 1 -- 172.21.15.38:0/1235095663 >> 172.21.15.38:0/1235095663 conn(0x7f7a7c0fb5f0 msgr2=0x7f7a7c0fda20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:44.695 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:44.690+0000 7f7a830bd700 1 -- 172.21.15.38:0/1235095663 shutdown_connections 2023-03-09T14:44:44.695 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:44.690+0000 7f7a830bd700 1 -- 172.21.15.38:0/1235095663 wait complete. 2023-03-09T14:44:44.695 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:44.690+0000 7f7a830bd700 1 Processor -- start 2023-03-09T14:44:44.696 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:44.690+0000 7f7a830bd700 1 -- start start 2023-03-09T14:44:44.696 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:44.690+0000 7f7a830bd700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7a7c0fffd0 0x7f7a7c074530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:44.696 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:44.691+0000 7f7a830bd700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7a7c1011c0 0x7f7a7c072b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:44.696 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:44.691+0000 7f7a830bd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7a7c101b50 0x7f7a7c0730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:44.696 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:44.691+0000 7f7a830bd700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a7c073850 con 0x7f7a7c101b50 2023-03-09T14:44:44.696 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:44.691+0000 7f7a830bd700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a7c0739c0 con 0x7f7a7c0fffd0 2023-03-09T14:44:44.696 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:44.691+0000 7f7a830bd700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a7c19fee0 con 0x7f7a7c1011c0 2023-03-09T14:44:44.697 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:44.691+0000 7f7a8165a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7a7c101b50 0x7f7a7c0730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:44.697 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:44.691+0000 7f7a8165a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7a7c101b50 0x7f7a7c0730b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37270/0 (socket says 172.21.15.38:37270) 2023-03-09T14:44:44.697 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:44.691+0000 7f7a8165a700 1 -- 172.21.15.38:0/978870157 learned_addr learned my addr 172.21.15.38:0/978870157 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:44:44.697 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:44.691+0000 7f7a80e59700 1 --2- 172.21.15.38:0/978870157 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7a7c0fffd0 0x7f7a7c074530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:44.697 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:44.691+0000 7f7a7bfff700 1 --2- 172.21.15.38:0/978870157 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7a7c1011c0 0x7f7a7c072b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:44.698 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:44.691+0000 7f7a8165a700 1 -- 172.21.15.38:0/978870157 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7a7c1011c0 msgr2=0x7f7a7c072b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:44.698 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:44.691+0000 7f7a8165a700 1 --2- 172.21.15.38:0/978870157 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7a7c1011c0 0x7f7a7c072b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:44.698 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:44.691+0000 7f7a8165a700 1 -- 172.21.15.38:0/978870157 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7a7c0fffd0 msgr2=0x7f7a7c074530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:44.698 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:44.691+0000 7f7a8165a700 1 --2- 172.21.15.38:0/978870157 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7a7c0fffd0 0x7f7a7c074530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:44.698 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:44.691+0000 7f7a8165a700 1 -- 172.21.15.38:0/978870157 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a74009d00 con 0x7f7a7c101b50 2023-03-09T14:44:44.698 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:44.691+0000 7f7a8165a700 1 --2- 172.21.15.38:0/978870157 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7a7c101b50 0x7f7a7c0730b0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f7a7400d010 tx=0x7f7a74004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:44.699 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:44.691+0000 7f7a79ffb700 1 -- 172.21.15.38:0/978870157 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a7400e040 con 0x7f7a7c101b50 2023-03-09T14:44:44.699 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:44.691+0000 7f7a79ffb700 1 -- 172.21.15.38:0/978870157 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7a74009440 con 0x7f7a7c101b50 2023-03-09T14:44:44.699 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:44.691+0000 7f7a830bd700 1 -- 172.21.15.38:0/978870157 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a7c10b990 con 0x7f7a7c101b50 2023-03-09T14:44:44.699 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:44.691+0000 7f7a79ffb700 1 -- 172.21.15.38:0/978870157 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a74004020 con 0x7f7a7c101b50 2023-03-09T14:44:44.699 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:44.692+0000 7f7a830bd700 1 -- 172.21.15.38:0/978870157 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a7c10b800 con 0x7f7a7c101b50 2023-03-09T14:44:44.700 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:44.692+0000 7f7a830bd700 1 -- 172.21.15.38:0/978870157 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a7c10bb20 con 0x7f7a7c101b50 2023-03-09T14:44:44.700 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:44.694+0000 7f7a79ffb700 1 -- 172.21.15.38:0/978870157 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 133238+0+0 (secure 0 0 0) 0x7f7a7401d690 con 0x7f7a7c101b50 2023-03-09T14:44:44.700 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:44.694+0000 7f7a79ffb700 1 --2- 172.21.15.38:0/978870157 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f7a6c09f400 0x7f7a6c0a18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:44.700 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:44.694+0000 7f7a79ffb700 1 -- 172.21.15.38:0/978870157 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f7a740cc010 con 0x7f7a7c101b50 2023-03-09T14:44:44.701 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:44.694+0000 7f7a80e59700 1 --2- 172.21.15.38:0/978870157 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f7a6c09f400 0x7f7a6c0a18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:44.701 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:44.695+0000 7f7a80e59700 1 --2- 172.21.15.38:0/978870157 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f7a6c09f400 0x7f7a6c0a18b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c101020 tx=0x7f7a7000b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:44.701 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:44.698+0000 7f7a79ffb700 1 -- 172.21.15.38:0/978870157 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f7a74096230 con 0x7f7a7c101b50 2023-03-09T14:44:44.981 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:44.978+0000 7f7a830bd700 1 -- 172.21.15.38:0/978870157 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f7a7c066b10 con 0x7f7a7c101b50 2023-03-09T14:44:44.985 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:44.983+0000 7f7a79ffb700 1 -- 172.21.15.38:0/978870157 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v31) v1 ==== 170+0+59 (secure 0 0 0) 0x7f7a7c066b10 con 0x7f7a7c101b50 2023-03-09T14:44:44.985 INFO:teuthology.orchestra.run.smithi038.stdout:[client.0] 2023-03-09T14:44:44.985 INFO:teuthology.orchestra.run.smithi038.stdout: key = AQDc8Alku6FmOhAAgd6e1JmGbLa3rokcsNldXg== 2023-03-09T14:44:44.987 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:44.985+0000 7f7a830bd700 1 -- 172.21.15.38:0/978870157 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f7a6c09f400 msgr2=0x7f7a6c0a18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:44.987 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:44.985+0000 7f7a830bd700 1 --2- 172.21.15.38:0/978870157 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f7a6c09f400 0x7f7a6c0a18b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c101020 tx=0x7f7a7000b410 comp rx=0 tx=0).stop 2023-03-09T14:44:44.987 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:44.985+0000 7f7a830bd700 1 -- 172.21.15.38:0/978870157 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7a7c101b50 msgr2=0x7f7a7c0730b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:44.987 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:44.985+0000 7f7a830bd700 1 --2- 172.21.15.38:0/978870157 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7a7c101b50 0x7f7a7c0730b0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f7a7400d010 tx=0x7f7a74004c80 comp rx=0 tx=0).stop 2023-03-09T14:44:44.987 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:44.985+0000 7f7a830bd700 1 -- 172.21.15.38:0/978870157 shutdown_connections 2023-03-09T14:44:44.988 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:44.985+0000 7f7a830bd700 1 --2- 172.21.15.38:0/978870157 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7a7c101b50 0x7f7a7c0730b0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:44.988 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:44.985+0000 7f7a830bd700 1 --2- 172.21.15.38:0/978870157 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f7a6c09f400 0x7f7a6c0a18b0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:44.988 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:44.985+0000 7f7a830bd700 1 --2- 172.21.15.38:0/978870157 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7a7c0fffd0 0x7f7a7c074530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:44.988 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:44.985+0000 7f7a830bd700 1 --2- 172.21.15.38:0/978870157 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7a7c1011c0 0x7f7a7c072b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:44.988 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:44.985+0000 7f7a830bd700 1 -- 172.21.15.38:0/978870157 >> 172.21.15.38:0/978870157 conn(0x7f7a7c0fb5f0 msgr2=0x7f7a7c104630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:44.988 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:44.985+0000 7f7a830bd700 1 -- 172.21.15.38:0/978870157 shutdown_connections 2023-03-09T14:44:44.988 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:44.985+0000 7f7a830bd700 1 -- 172.21.15.38:0/978870157 wait complete. 2023-03-09T14:44:45.444 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2023-03-09T14:44:45.445 DEBUG:teuthology.orchestra.run.smithi038:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2023-03-09T14:44:45.445 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-03-09T14:44:45.488 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-03-09T14:44:45.489 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2023-03-09T14:44:45.489 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph mgr dump --format=json 2023-03-09T14:44:46.177 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.a/config 2023-03-09T14:44:46.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:45 smithi122 ceph-mon[93869]: pgmap v272: 129 pgs: 129 active+clean; 601 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4.2 KiB/s wr, 10 op/s 2023-03-09T14:44:46.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:45 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/978870157' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-03-09T14:44:46.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:45 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/978870157' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-03-09T14:44:46.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:45 smithi181 ceph-mon[94189]: pgmap v272: 129 pgs: 129 active+clean; 601 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4.2 KiB/s wr, 10 op/s 2023-03-09T14:44:46.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:45 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/978870157' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-03-09T14:44:46.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:45 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/978870157' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-03-09T14:44:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:45 smithi038 ceph-mon[92422]: pgmap v272: 129 pgs: 129 active+clean; 601 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4.2 KiB/s wr, 10 op/s 2023-03-09T14:44:46.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:45 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/978870157' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-03-09T14:44:46.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:45 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/978870157' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-03-09T14:44:47.378 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:47.376+0000 7f8a491c5700 1 -- 172.21.15.38:0/4179386472 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8a44100ce0 msgr2=0x7f8a44104bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:47.379 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:47.376+0000 7f8a491c5700 1 --2- 172.21.15.38:0/4179386472 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8a44100ce0 0x7f8a44104bc0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f8a380099c0 tx=0x7f8a38009cd0 comp rx=0 tx=0).stop 2023-03-09T14:44:47.379 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:47.376+0000 7f8a491c5700 1 -- 172.21.15.38:0/4179386472 shutdown_connections 2023-03-09T14:44:47.379 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:47.376+0000 7f8a491c5700 1 --2- 172.21.15.38:0/4179386472 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8a441052a0 0x7f8a4410ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:47.379 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:47.376+0000 7f8a491c5700 1 --2- 172.21.15.38:0/4179386472 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a44100340 0x7f8a44100710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:47.379 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:47.376+0000 7f8a491c5700 1 --2- 172.21.15.38:0/4179386472 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8a44100ce0 0x7f8a44104bc0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:47.379 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:47.376+0000 7f8a491c5700 1 -- 172.21.15.38:0/4179386472 >> 172.21.15.38:0/4179386472 conn(0x7f8a44077010 msgr2=0x7f8a44077410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:47.380 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:47.376+0000 7f8a491c5700 1 -- 172.21.15.38:0/4179386472 shutdown_connections 2023-03-09T14:44:47.380 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:47.376+0000 7f8a491c5700 1 -- 172.21.15.38:0/4179386472 wait complete. 2023-03-09T14:44:47.381 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:47.377+0000 7f8a491c5700 1 Processor -- start 2023-03-09T14:44:47.382 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:47.377+0000 7f8a491c5700 1 -- start start 2023-03-09T14:44:47.382 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:47.377+0000 7f8a491c5700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8a44100340 0x7f8a44072a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:47.382 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:47.377+0000 7f8a491c5700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a44100ce0 0x7f8a44072fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:47.382 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:47.377+0000 7f8a491c5700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8a441052a0 0x7f8a441a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:47.382 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:47.377+0000 7f8a491c5700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a440737f0 con 0x7f8a44100340 2023-03-09T14:44:47.382 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:47.377+0000 7f8a491c5700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a441a48f0 con 0x7f8a441052a0 2023-03-09T14:44:47.382 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:47.377+0000 7f8a491c5700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a441a4a30 con 0x7f8a44100ce0 2023-03-09T14:44:47.383 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:47.377+0000 7f8a42d9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8a44100340 0x7f8a44072a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:47.383 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:47.377+0000 7f8a42d9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8a44100340 0x7f8a44072a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37280/0 (socket says 172.21.15.38:37280) 2023-03-09T14:44:47.383 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:47.377+0000 7f8a42d9d700 1 -- 172.21.15.38:0/899888818 learned_addr learned my addr 172.21.15.38:0/899888818 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:44:47.383 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:47.377+0000 7f8a4259c700 1 --2- 172.21.15.38:0/899888818 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a44100ce0 0x7f8a44072fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:47.383 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:47.377+0000 7f8a4359e700 1 --2- 172.21.15.38:0/899888818 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8a441052a0 0x7f8a441a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:47.383 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:47.377+0000 7f8a42d9d700 1 -- 172.21.15.38:0/899888818 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a44100ce0 msgr2=0x7f8a44072fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:47.384 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:47.377+0000 7f8a42d9d700 1 --2- 172.21.15.38:0/899888818 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a44100ce0 0x7f8a44072fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:47.384 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:47.377+0000 7f8a42d9d700 1 -- 172.21.15.38:0/899888818 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8a441052a0 msgr2=0x7f8a441a43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:47.384 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:47.377+0000 7f8a42d9d700 1 --2- 172.21.15.38:0/899888818 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8a441052a0 0x7f8a441a43b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:47.384 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:47.377+0000 7f8a42d9d700 1 -- 172.21.15.38:0/899888818 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a380095f0 con 0x7f8a44100340 2023-03-09T14:44:47.384 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:47.378+0000 7f8a42d9d700 1 --2- 172.21.15.38:0/899888818 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8a44100340 0x7f8a44072a80 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f8a3400d950 tx=0x7f8a3400dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:47.384 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:47.378+0000 7f8a2bfff700 1 -- 172.21.15.38:0/899888818 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a34009b40 con 0x7f8a44100340 2023-03-09T14:44:47.385 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:47.378+0000 7f8a491c5700 1 -- 172.21.15.38:0/899888818 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a44110070 con 0x7f8a44100340 2023-03-09T14:44:47.385 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:47.378+0000 7f8a2bfff700 1 -- 172.21.15.38:0/899888818 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8a34009ca0 con 0x7f8a44100340 2023-03-09T14:44:47.385 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:47.378+0000 7f8a2bfff700 1 -- 172.21.15.38:0/899888818 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a3400f620 con 0x7f8a44100340 2023-03-09T14:44:47.385 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:47.378+0000 7f8a491c5700 1 -- 172.21.15.38:0/899888818 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a4410fee0 con 0x7f8a44100340 2023-03-09T14:44:47.385 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:47.379+0000 7f8a491c5700 1 -- 172.21.15.38:0/899888818 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a44110200 con 0x7f8a44100340 2023-03-09T14:44:47.386 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:47.380+0000 7f8a2bfff700 1 -- 172.21.15.38:0/899888818 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 133238+0+0 (secure 0 0 0) 0x7f8a34018030 con 0x7f8a44100340 2023-03-09T14:44:47.386 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:47.381+0000 7f8a2bfff700 1 --2- 172.21.15.38:0/899888818 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f8a2c09f400 0x7f8a2c0a18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:47.386 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:47.381+0000 7f8a2bfff700 1 -- 172.21.15.38:0/899888818 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f8a340ca2c0 con 0x7f8a44100340 2023-03-09T14:44:47.387 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:47.381+0000 7f8a4259c700 1 --2- 172.21.15.38:0/899888818 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f8a2c09f400 0x7f8a2c0a18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:47.387 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:47.381+0000 7f8a4259c700 1 --2- 172.21.15.38:0/899888818 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f8a2c09f400 0x7f8a2c0a18b0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f8a3800b580 tx=0x7f8a3801a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:47.387 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:47.384+0000 7f8a2bfff700 1 -- 172.21.15.38:0/899888818 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f8a34094450 con 0x7f8a44100340 2023-03-09T14:44:47.648 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:47.646+0000 7f8a491c5700 1 -- 172.21.15.38:0/899888818 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f8a441a5450 con 0x7f8a44100340 2023-03-09T14:44:47.653 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:47.651+0000 7f8a2bfff700 1 -- 172.21.15.38:0/899888818 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) v1 ==== 74+0+256252 (secure 0 0 0) 0x7f8a34016020 con 0x7f8a44100340 2023-03-09T14:44:47.656 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:44:47.657 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:47.654+0000 7f8a491c5700 1 -- 172.21.15.38:0/899888818 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f8a2c09f400 msgr2=0x7f8a2c0a18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:47.657 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:47.654+0000 7f8a491c5700 1 --2- 172.21.15.38:0/899888818 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f8a2c09f400 0x7f8a2c0a18b0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f8a3800b580 tx=0x7f8a3801a040 comp rx=0 tx=0).stop 2023-03-09T14:44:47.658 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:47.654+0000 7f8a491c5700 1 -- 172.21.15.38:0/899888818 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8a44100340 msgr2=0x7f8a44072a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:47.658 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:47.654+0000 7f8a491c5700 1 --2- 172.21.15.38:0/899888818 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8a44100340 0x7f8a44072a80 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f8a3400d950 tx=0x7f8a3400dd10 comp rx=0 tx=0).stop 2023-03-09T14:44:47.658 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:47.654+0000 7f8a491c5700 1 -- 172.21.15.38:0/899888818 shutdown_connections 2023-03-09T14:44:47.658 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:47.654+0000 7f8a491c5700 1 --2- 172.21.15.38:0/899888818 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8a44100340 0x7f8a44072a80 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:47.658 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:47.654+0000 7f8a491c5700 1 --2- 172.21.15.38:0/899888818 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f8a2c09f400 0x7f8a2c0a18b0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:47.658 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:47.654+0000 7f8a491c5700 1 --2- 172.21.15.38:0/899888818 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8a441052a0 0x7f8a441a43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:47.658 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:47.654+0000 7f8a491c5700 1 --2- 172.21.15.38:0/899888818 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a44100ce0 0x7f8a44072fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:47.659 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:47.654+0000 7f8a491c5700 1 -- 172.21.15.38:0/899888818 >> 172.21.15.38:0/899888818 conn(0x7f8a44077010 msgr2=0x7f8a440fdaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:47.659 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:47.654+0000 7f8a491c5700 1 -- 172.21.15.38:0/899888818 shutdown_connections 2023-03-09T14:44:47.659 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:47.654+0000 7f8a491c5700 1 -- 172.21.15.38:0/899888818 wait complete. 2023-03-09T14:44:48.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:47 smithi038 ceph-mon[92422]: pgmap v273: 129 pgs: 129 active+clean; 623 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 6.0 KiB/s wr, 12 op/s 2023-03-09T14:44:48.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:47 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/899888818' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-03-09T14:44:48.029 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":17,"active_gid":14150,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6800","nonce":3328887328},{"type":"v1","addr":"172.21.15.38:6801","nonce":3328887328}]},"active_addr":"172.21.15.38:6801/3328887328","active_change":"2023-03-09T14:38:14.613328+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":24106,"name":"z","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24110,"name":"y","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.38:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.38:0","nonce":3239315241}]},{"addrvec":[{"type":"v2","addr":"172.21.15.38:0","nonce":800282658}]},{"addrvec":[{"type":"v2","addr":"172.21.15.38:0","nonce":3529329557}]},{"addrvec":[{"type":"v2","addr":"172.21.15.38:0","nonce":36242693}]},{"addrvec":[{"type":"v2","addr":"172.21.15.38:0","nonce":2696946683}]}]} 2023-03-09T14:44:48.037 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2023-03-09T14:44:48.038 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2023-03-09T14:44:48.038 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph osd dump --format=json 2023-03-09T14:44:48.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:47 smithi122 ceph-mon[93869]: pgmap v273: 129 pgs: 129 active+clean; 623 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 6.0 KiB/s wr, 12 op/s 2023-03-09T14:44:48.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:47 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/899888818' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-03-09T14:44:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:47 smithi181 ceph-mon[94189]: pgmap v273: 129 pgs: 129 active+clean; 623 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 6.0 KiB/s wr, 12 op/s 2023-03-09T14:44:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:47 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/899888818' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-03-09T14:44:48.694 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.a/config 2023-03-09T14:44:49.923 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:49.920+0000 7fd32543c700 1 -- 172.21.15.38:0/2134772646 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3200ff120 msgr2=0x7fd3200ff4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:49.923 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:49.920+0000 7fd32543c700 1 --2- 172.21.15.38:0/2134772646 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3200ff120 0x7fd3200ff4f0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fd3100099c0 tx=0x7fd310009cd0 comp rx=0 tx=0).stop 2023-03-09T14:44:49.923 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:49.920+0000 7fd32543c700 1 -- 172.21.15.38:0/2134772646 shutdown_connections 2023-03-09T14:44:49.924 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:49.920+0000 7fd32543c700 1 --2- 172.21.15.38:0/2134772646 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd3200ff120 0x7fd3200ff4f0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:49.924 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:49.920+0000 7fd32543c700 1 --2- 172.21.15.38:0/2134772646 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd320104160 0x7fd32010bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:49.924 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:49.920+0000 7fd32543c700 1 --2- 172.21.15.38:0/2134772646 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd3200ffac0 0x7fd320103a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:49.924 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:49.920+0000 7fd32543c700 1 -- 172.21.15.38:0/2134772646 >> 172.21.15.38:0/2134772646 conn(0x7fd320076d10 msgr2=0x7fd320077110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:49.925 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:49.920+0000 7fd32543c700 1 -- 172.21.15.38:0/2134772646 shutdown_connections 2023-03-09T14:44:49.925 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:49.920+0000 7fd32543c700 1 -- 172.21.15.38:0/2134772646 wait complete. 2023-03-09T14:44:49.926 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:49.921+0000 7fd32543c700 1 Processor -- start 2023-03-09T14:44:49.927 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:49.921+0000 7fd32543c700 1 -- start start 2023-03-09T14:44:49.927 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:49.921+0000 7fd32543c700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd3200ff120 0x7fd320104cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:49.927 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:49.921+0000 7fd32543c700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd3200ffac0 0x7fd3201051f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:49.927 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:49.922+0000 7fd32543c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd320104160 0x7fd320105730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:49.927 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:49.922+0000 7fd32543c700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3201a8a40 con 0x7fd320104160 2023-03-09T14:44:49.927 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:49.922+0000 7fd32543c700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3201a8bb0 con 0x7fd3200ffac0 2023-03-09T14:44:49.928 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:49.922+0000 7fd32543c700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3201a8d70 con 0x7fd3200ff120 2023-03-09T14:44:49.928 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:49.922+0000 7fd31f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd320104160 0x7fd320105730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:49.928 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:49.922+0000 7fd31f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd320104160 0x7fd320105730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37296/0 (socket says 172.21.15.38:37296) 2023-03-09T14:44:49.928 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:49.922+0000 7fd31f7fe700 1 -- 172.21.15.38:0/940426878 learned_addr learned my addr 172.21.15.38:0/940426878 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:44:49.928 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:49.922+0000 7fd31effd700 1 --2- 172.21.15.38:0/940426878 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd3200ff120 0x7fd320104cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:49.928 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:49.922+0000 7fd31e7fc700 1 --2- 172.21.15.38:0/940426878 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd3200ffac0 0x7fd3201051f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:49.928 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:49.922+0000 7fd31f7fe700 1 -- 172.21.15.38:0/940426878 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd3200ff120 msgr2=0x7fd320104cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:49.928 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:49.922+0000 7fd31f7fe700 1 --2- 172.21.15.38:0/940426878 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd3200ff120 0x7fd320104cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:49.929 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:49.922+0000 7fd31f7fe700 1 -- 172.21.15.38:0/940426878 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd3200ffac0 msgr2=0x7fd3201051f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:49.929 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:49.922+0000 7fd31f7fe700 1 --2- 172.21.15.38:0/940426878 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd3200ffac0 0x7fd3201051f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:49.929 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:49.922+0000 7fd31f7fe700 1 -- 172.21.15.38:0/940426878 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3100095f0 con 0x7fd320104160 2023-03-09T14:44:49.929 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:49.922+0000 7fd31f7fe700 1 --2- 172.21.15.38:0/940426878 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd320104160 0x7fd320105730 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fd31800ada0 tx=0x7fd31800c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:49.929 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:49.922+0000 7fd317fff700 1 -- 172.21.15.38:0/940426878 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd31800f070 con 0x7fd320104160 2023-03-09T14:44:49.929 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:49.922+0000 7fd32543c700 1 -- 172.21.15.38:0/940426878 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd32010ef20 con 0x7fd320104160 2023-03-09T14:44:49.929 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:49.922+0000 7fd317fff700 1 -- 172.21.15.38:0/940426878 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd318007b30 con 0x7fd320104160 2023-03-09T14:44:49.930 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:49.922+0000 7fd32543c700 1 -- 172.21.15.38:0/940426878 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd32010ed90 con 0x7fd320104160 2023-03-09T14:44:49.930 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:49.922+0000 7fd317fff700 1 -- 172.21.15.38:0/940426878 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd318005260 con 0x7fd320104160 2023-03-09T14:44:49.930 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:49.924+0000 7fd32543c700 1 -- 172.21.15.38:0/940426878 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd32010f0b0 con 0x7fd320104160 2023-03-09T14:44:49.930 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:49.924+0000 7fd317fff700 1 -- 172.21.15.38:0/940426878 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 133238+0+0 (secure 0 0 0) 0x7fd318019020 con 0x7fd320104160 2023-03-09T14:44:49.931 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:49.925+0000 7fd317fff700 1 --2- 172.21.15.38:0/940426878 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fd30c09f400 0x7fd30c0a18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:49.931 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:49.925+0000 7fd31effd700 1 --2- 172.21.15.38:0/940426878 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fd30c09f400 0x7fd30c0a18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:49.931 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:49.925+0000 7fd317fff700 1 -- 172.21.15.38:0/940426878 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fd3180c93f0 con 0x7fd320104160 2023-03-09T14:44:49.931 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:49.926+0000 7fd31effd700 1 --2- 172.21.15.38:0/940426878 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fd30c09f400 0x7fd30c0a18b0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fd310000c00 tx=0x7fd310005c40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:49.932 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:49.930+0000 7fd317fff700 1 -- 172.21.15.38:0/940426878 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7fd318093610 con 0x7fd320104160 2023-03-09T14:44:50.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:49 smithi038 ceph-mon[92422]: pgmap v274: 129 pgs: 129 active+clean; 623 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 5.7 KiB/s wr, 13 op/s 2023-03-09T14:44:50.138 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:50.136+0000 7fd32543c700 1 -- 172.21.15.38:0/940426878 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd3201061e0 con 0x7fd320104160 2023-03-09T14:44:50.140 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:50.138+0000 7fd317fff700 1 -- 172.21.15.38:0/940426878 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+23973 (secure 0 0 0) 0x7fd318096c30 con 0x7fd320104160 2023-03-09T14:44:50.140 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:44:50.140 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":87,"fsid":"c6733fe2-be87-11ed-9af2-001a4aab830c","created":"2023-03-09T14:37:14.947981+0000","modified":"2023-03-09T14:44:23.900228+0000","last_up_change":"2023-03-09T14:43:37.198407+0000","last_in_change":"2023-03-09T14:43:19.550566+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-09T14:40:23.196665+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-03-09T14:44:17.484374+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-03-09T14:44:18.457833+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-03-09T14:44:21.895589+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"51477039-15ee-4707-a34d-4ecfa18bebde","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6803","nonce":1508072290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6805","nonce":1508072290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6809","nonce":1508072290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6807","nonce":1508072290}]},"public_addr":"172.21.15.38:6803/1508072290","cluster_addr":"172.21.15.38:6805/1508072290","heartbeat_back_addr":"172.21.15.38:6809/1508072290","heartbeat_front_addr":"172.21.15.38:6807/1508072290","state":["exists","up"]},{"osd":1,"uuid":"71f8a718-b0a9-4f39-bc54-267aa42a2578","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6801","nonce":3929268773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6803","nonce":3929268773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6807","nonce":3929268773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6805","nonce":3929268773}]},"public_addr":"172.21.15.122:6801/3929268773","cluster_addr":"172.21.15.122:6803/3929268773","heartbeat_back_addr":"172.21.15.122:6807/3929268773","heartbeat_front_addr":"172.21.15.122:6805/3929268773","state":["exists","up"]},{"osd":2,"uuid":"961a2cbe-140e-437f-b131-1fcac573b19c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6801","nonce":1310630803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6803","nonce":1310630803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6807","nonce":1310630803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6805","nonce":1310630803}]},"public_addr":"172.21.15.181:6801/1310630803","cluster_addr":"172.21.15.181:6803/1310630803","heartbeat_back_addr":"172.21.15.181:6807/1310630803","heartbeat_front_addr":"172.21.15.181:6805/1310630803","state":["exists","up"]},{"osd":3,"uuid":"969df8fa-2d19-42c7-9f2a-5c5fac15d4a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6811","nonce":2719814100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6813","nonce":2719814100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6817","nonce":2719814100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6815","nonce":2719814100}]},"public_addr":"172.21.15.38:6811/2719814100","cluster_addr":"172.21.15.38:6813/2719814100","heartbeat_back_addr":"172.21.15.38:6817/2719814100","heartbeat_front_addr":"172.21.15.38:6815/2719814100","state":["exists","up"]},{"osd":4,"uuid":"44a6dce3-e06e-4cbd-9404-62a86d01ab6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6809","nonce":2480764994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6811","nonce":2480764994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6815","nonce":2480764994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6813","nonce":2480764994}]},"public_addr":"172.21.15.122:6809/2480764994","cluster_addr":"172.21.15.122:6811/2480764994","heartbeat_back_addr":"172.21.15.122:6815/2480764994","heartbeat_front_addr":"172.21.15.122:6813/2480764994","state":["exists","up"]},{"osd":5,"uuid":"49749574-c00c-4e3a-80e6-e3686df58d5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6809","nonce":3141346246}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6811","nonce":3141346246}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6815","nonce":3141346246}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6813","nonce":3141346246}]},"public_addr":"172.21.15.181:6809/3141346246","cluster_addr":"172.21.15.181:6811/3141346246","heartbeat_back_addr":"172.21.15.181:6815/3141346246","heartbeat_front_addr":"172.21.15.181:6813/3141346246","state":["exists","up"]},{"osd":6,"uuid":"61c0437e-3f51-45a9-8a61-e98eea0093f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6819","nonce":2122788957}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6821","nonce":2122788957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6825","nonce":2122788957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6823","nonce":2122788957}]},"public_addr":"172.21.15.38:6819/2122788957","cluster_addr":"172.21.15.38:6821/2122788957","heartbeat_back_addr":"172.21.15.38:6825/2122788957","heartbeat_front_addr":"172.21.15.38:6823/2122788957","state":["exists","up"]},{"osd":7,"uuid":"231fecdd-939f-4c7b-83a1-c4a1cffd7139","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6817","nonce":656874139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6819","nonce":656874139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6823","nonce":656874139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6821","nonce":656874139}]},"public_addr":"172.21.15.122:6817/656874139","cluster_addr":"172.21.15.122:6819/656874139","heartbeat_back_addr":"172.21.15.122:6823/656874139","heartbeat_front_addr":"172.21.15.122:6821/656874139","state":["exists","up"]},{"osd":8,"uuid":"297e6d6d-85fd-43ce-9028-5e9dbe55c36e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6817","nonce":755000703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6819","nonce":755000703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6823","nonce":755000703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6821","nonce":755000703}]},"public_addr":"172.21.15.181:6817/755000703","cluster_addr":"172.21.15.181:6819/755000703","heartbeat_back_addr":"172.21.15.181:6823/755000703","heartbeat_front_addr":"172.21.15.181:6821/755000703","state":["exists","up"]},{"osd":9,"uuid":"94cc0355-5a1a-4cfc-bd0c-562b6b41127e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6827","nonce":2624555969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6829","nonce":2624555969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6833","nonce":2624555969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6831","nonce":2624555969}]},"public_addr":"172.21.15.38:6827/2624555969","cluster_addr":"172.21.15.38:6829/2624555969","heartbeat_back_addr":"172.21.15.38:6833/2624555969","heartbeat_front_addr":"172.21.15.38:6831/2624555969","state":["exists","up"]},{"osd":10,"uuid":"f4f2e042-8d92-46bf-93b8-8b0f04795c2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6825","nonce":1427785280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6827","nonce":1427785280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6831","nonce":1427785280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6829","nonce":1427785280}]},"public_addr":"172.21.15.122:6825/1427785280","cluster_addr":"172.21.15.122:6827/1427785280","heartbeat_back_addr":"172.21.15.122:6831/1427785280","heartbeat_front_addr":"172.21.15.122:6829/1427785280","state":["exists","up"]},{"osd":11,"uuid":"07444936-eae5-4097-a799-60294cafe80b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6825","nonce":3183488640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6827","nonce":3183488640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6831","nonce":3183488640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6829","nonce":3183488640}]},"public_addr":"172.21.15.181:6825/3183488640","cluster_addr":"172.21.15.181:6827/3183488640","heartbeat_back_addr":"172.21.15.181:6831/3183488640","heartbeat_front_addr":"172.21.15.181:6829/3183488640","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:39:37.657556+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:39:58.960074+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:40:20.326952+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:40:42.244387+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:41:03.946973+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:41:25.097985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:41:46.877916+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:42:08.413827+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:42:29.822009+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:42:51.776684+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:43:13.937249+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:43:35.569199+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/4272288735":"2023-03-10T14:38:14.612767+0000","172.21.15.38:6801/3381625669":"2023-03-10T14:38:14.612767+0000","172.21.15.38:6800/3381625669":"2023-03-10T14:38:14.612767+0000","172.21.15.38:6800/3060474342":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/4253918615":"2023-03-10T14:38:14.612767+0000","172.21.15.38:0/3496083181":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/245602368":"2023-03-10T14:37:47.517775+0000","172.21.15.38:6801/3060474342":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/3622589182":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/1430302251":"2023-03-10T14:38:14.612767+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-09T14:44:50.142 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:50.140+0000 7fd32543c700 1 -- 172.21.15.38:0/940426878 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fd30c09f400 msgr2=0x7fd30c0a18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:50.142 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:50.140+0000 7fd32543c700 1 --2- 172.21.15.38:0/940426878 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fd30c09f400 0x7fd30c0a18b0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fd310000c00 tx=0x7fd310005c40 comp rx=0 tx=0).stop 2023-03-09T14:44:50.143 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:50.140+0000 7fd32543c700 1 -- 172.21.15.38:0/940426878 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd320104160 msgr2=0x7fd320105730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:50.143 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:50.140+0000 7fd32543c700 1 --2- 172.21.15.38:0/940426878 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd320104160 0x7fd320105730 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fd31800ada0 tx=0x7fd31800c330 comp rx=0 tx=0).stop 2023-03-09T14:44:50.143 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:50.140+0000 7fd32543c700 1 -- 172.21.15.38:0/940426878 shutdown_connections 2023-03-09T14:44:50.143 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:50.140+0000 7fd32543c700 1 --2- 172.21.15.38:0/940426878 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd320104160 0x7fd320105730 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:50.143 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:50.140+0000 7fd32543c700 1 --2- 172.21.15.38:0/940426878 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fd30c09f400 0x7fd30c0a18b0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:50.144 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:50.140+0000 7fd32543c700 1 --2- 172.21.15.38:0/940426878 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd3200ff120 0x7fd320104cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:50.144 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:50.140+0000 7fd32543c700 1 --2- 172.21.15.38:0/940426878 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd3200ffac0 0x7fd3201051f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:50.144 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:50.140+0000 7fd32543c700 1 -- 172.21.15.38:0/940426878 >> 172.21.15.38:0/940426878 conn(0x7fd320076d10 msgr2=0x7fd320068460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:50.144 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:50.140+0000 7fd32543c700 1 -- 172.21.15.38:0/940426878 shutdown_connections 2023-03-09T14:44:50.144 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:50.140+0000 7fd32543c700 1 -- 172.21.15.38:0/940426878 wait complete. 2023-03-09T14:44:50.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:49 smithi122 ceph-mon[93869]: pgmap v274: 129 pgs: 129 active+clean; 623 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 5.7 KiB/s wr, 13 op/s 2023-03-09T14:44:50.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:49 smithi181 ceph-mon[94189]: pgmap v274: 129 pgs: 129 active+clean; 623 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 5.7 KiB/s wr, 13 op/s 2023-03-09T14:44:50.559 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2023-03-09T14:44:50.560 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph osd dump --format=json 2023-03-09T14:44:51.216 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.a/config 2023-03-09T14:44:51.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:50 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/940426878' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-03-09T14:44:51.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:50 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/940426878' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-03-09T14:44:51.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:50 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/940426878' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-03-09T14:44:52.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:51 smithi122 ceph-mon[93869]: pgmap v275: 129 pgs: 129 active+clean; 635 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 7.0 KiB/s wr, 15 op/s 2023-03-09T14:44:52.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:51 smithi181 ceph-mon[94189]: pgmap v275: 129 pgs: 129 active+clean; 635 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 7.0 KiB/s wr, 15 op/s 2023-03-09T14:44:52.244 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:51 smithi038 ceph-mon[92422]: pgmap v275: 129 pgs: 129 active+clean; 635 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 7.0 KiB/s wr, 15 op/s 2023-03-09T14:44:52.435 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:52.433+0000 7f8472250700 1 -- 172.21.15.38:0/3857988003 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f846c1052a0 msgr2=0x7f846c10ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:52.435 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:52.433+0000 7f8472250700 1 --2- 172.21.15.38:0/3857988003 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f846c1052a0 0x7f846c10ce70 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f846400b0d0 tx=0x7f846400b3e0 comp rx=0 tx=0).stop 2023-03-09T14:44:52.435 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:52.433+0000 7f8472250700 1 -- 172.21.15.38:0/3857988003 shutdown_connections 2023-03-09T14:44:52.436 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:52.433+0000 7f8472250700 1 --2- 172.21.15.38:0/3857988003 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f846c1052a0 0x7f846c10ce70 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:52.436 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:52.433+0000 7f8472250700 1 --2- 172.21.15.38:0/3857988003 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f846c100340 0x7f846c100710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:52.436 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:52.433+0000 7f8472250700 1 --2- 172.21.15.38:0/3857988003 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f846c100ce0 0x7f846c104bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:52.436 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:52.433+0000 7f8472250700 1 -- 172.21.15.38:0/3857988003 >> 172.21.15.38:0/3857988003 conn(0x7f846c077010 msgr2=0x7f846c077410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:52.436 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:52.433+0000 7f8472250700 1 -- 172.21.15.38:0/3857988003 shutdown_connections 2023-03-09T14:44:52.436 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:52.433+0000 7f8472250700 1 -- 172.21.15.38:0/3857988003 wait complete. 2023-03-09T14:44:52.438 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:52.433+0000 7f8472250700 1 Processor -- start 2023-03-09T14:44:52.438 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:52.434+0000 7f8472250700 1 -- start start 2023-03-09T14:44:52.438 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:52.434+0000 7f8472250700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f846c100340 0x7f846c198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:52.438 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:52.434+0000 7f8472250700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f846c100ce0 0x7f846c198960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:52.438 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:52.434+0000 7f8472250700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f846c1052a0 0x7f846c19cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:52.439 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:52.434+0000 7f8472250700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f846c19d400 con 0x7f846c100340 2023-03-09T14:44:52.439 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:52.434+0000 7f8472250700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f846c19d570 con 0x7f846c100ce0 2023-03-09T14:44:52.439 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:52.434+0000 7f8472250700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f846c19d730 con 0x7f846c1052a0 2023-03-09T14:44:52.439 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:52.434+0000 7f846b7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f846c100340 0x7f846c198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:52.439 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:52.434+0000 7f846b7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f846c100340 0x7f846c198420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37304/0 (socket says 172.21.15.38:37304) 2023-03-09T14:44:52.439 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:52.434+0000 7f846b7fe700 1 -- 172.21.15.38:0/3693090493 learned_addr learned my addr 172.21.15.38:0/3693090493 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:44:52.439 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:52.434+0000 7f846bfff700 1 --2- 172.21.15.38:0/3693090493 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f846c1052a0 0x7f846c19cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:52.439 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:52.434+0000 7f846affd700 1 --2- 172.21.15.38:0/3693090493 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f846c100ce0 0x7f846c198960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:52.440 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:52.434+0000 7f846b7fe700 1 -- 172.21.15.38:0/3693090493 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f846c1052a0 msgr2=0x7f846c19cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:52.440 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:52.434+0000 7f846b7fe700 1 --2- 172.21.15.38:0/3693090493 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f846c1052a0 0x7f846c19cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:52.440 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:52.434+0000 7f846b7fe700 1 -- 172.21.15.38:0/3693090493 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f846c100ce0 msgr2=0x7f846c198960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:52.440 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:52.434+0000 7f846b7fe700 1 --2- 172.21.15.38:0/3693090493 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f846c100ce0 0x7f846c198960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:52.440 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:52.434+0000 7f846b7fe700 1 -- 172.21.15.38:0/3693090493 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8464009d00 con 0x7f846c100340 2023-03-09T14:44:52.440 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:52.434+0000 7f846b7fe700 1 --2- 172.21.15.38:0/3693090493 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f846c100340 0x7f846c198420 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f845c00c920 tx=0x7f845c00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:52.440 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:52.435+0000 7f8468ff9700 1 -- 172.21.15.38:0/3693090493 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f845c007b00 con 0x7f846c100340 2023-03-09T14:44:52.441 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:52.435+0000 7f8468ff9700 1 -- 172.21.15.38:0/3693090493 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f845c00ce70 con 0x7f846c100340 2023-03-09T14:44:52.441 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:52.435+0000 7f8472250700 1 -- 172.21.15.38:0/3693090493 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f846c110070 con 0x7f846c100340 2023-03-09T14:44:52.441 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:52.435+0000 7f8468ff9700 1 -- 172.21.15.38:0/3693090493 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f845c0054e0 con 0x7f846c100340 2023-03-09T14:44:52.441 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:52.435+0000 7f8472250700 1 -- 172.21.15.38:0/3693090493 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f846c10fee0 con 0x7f846c100340 2023-03-09T14:44:52.441 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:52.436+0000 7f8472250700 1 -- 172.21.15.38:0/3693090493 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f846c110200 con 0x7f846c100340 2023-03-09T14:44:52.442 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:52.437+0000 7f8468ff9700 1 -- 172.21.15.38:0/3693090493 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 133238+0+0 (secure 0 0 0) 0x7f845c01f070 con 0x7f846c100340 2023-03-09T14:44:52.442 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:52.437+0000 7f8468ff9700 1 --2- 172.21.15.38:0/3693090493 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f845809f480 0x7f84580a1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:52.442 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:52.437+0000 7f8468ff9700 1 -- 172.21.15.38:0/3693090493 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f845c0c9e00 con 0x7f846c100340 2023-03-09T14:44:52.442 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:52.438+0000 7f846affd700 1 --2- 172.21.15.38:0/3693090493 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f845809f480 0x7f84580a1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:52.442 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:52.438+0000 7f846affd700 1 --2- 172.21.15.38:0/3693090493 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f845809f480 0x7f84580a1930 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f8454005fd0 tx=0x7f845400af20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:52.443 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:52.441+0000 7f8468ff9700 1 -- 172.21.15.38:0/3693090493 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f845c094010 con 0x7f846c100340 2023-03-09T14:44:52.642 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:52.639+0000 7f8472250700 1 -- 172.21.15.38:0/3693090493 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f846c1a5670 con 0x7f846c100340 2023-03-09T14:44:52.643 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:52.641+0000 7f8468ff9700 1 -- 172.21.15.38:0/3693090493 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+23973 (secure 0 0 0) 0x7f845c01d070 con 0x7f846c100340 2023-03-09T14:44:52.644 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:44:52.644 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":87,"fsid":"c6733fe2-be87-11ed-9af2-001a4aab830c","created":"2023-03-09T14:37:14.947981+0000","modified":"2023-03-09T14:44:23.900228+0000","last_up_change":"2023-03-09T14:43:37.198407+0000","last_in_change":"2023-03-09T14:43:19.550566+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-09T14:40:23.196665+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-03-09T14:44:17.484374+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-03-09T14:44:18.457833+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-03-09T14:44:21.895589+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"51477039-15ee-4707-a34d-4ecfa18bebde","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6803","nonce":1508072290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6805","nonce":1508072290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6809","nonce":1508072290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6807","nonce":1508072290}]},"public_addr":"172.21.15.38:6803/1508072290","cluster_addr":"172.21.15.38:6805/1508072290","heartbeat_back_addr":"172.21.15.38:6809/1508072290","heartbeat_front_addr":"172.21.15.38:6807/1508072290","state":["exists","up"]},{"osd":1,"uuid":"71f8a718-b0a9-4f39-bc54-267aa42a2578","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6801","nonce":3929268773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6803","nonce":3929268773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6807","nonce":3929268773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6805","nonce":3929268773}]},"public_addr":"172.21.15.122:6801/3929268773","cluster_addr":"172.21.15.122:6803/3929268773","heartbeat_back_addr":"172.21.15.122:6807/3929268773","heartbeat_front_addr":"172.21.15.122:6805/3929268773","state":["exists","up"]},{"osd":2,"uuid":"961a2cbe-140e-437f-b131-1fcac573b19c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6801","nonce":1310630803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6803","nonce":1310630803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6807","nonce":1310630803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6805","nonce":1310630803}]},"public_addr":"172.21.15.181:6801/1310630803","cluster_addr":"172.21.15.181:6803/1310630803","heartbeat_back_addr":"172.21.15.181:6807/1310630803","heartbeat_front_addr":"172.21.15.181:6805/1310630803","state":["exists","up"]},{"osd":3,"uuid":"969df8fa-2d19-42c7-9f2a-5c5fac15d4a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6811","nonce":2719814100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6813","nonce":2719814100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6817","nonce":2719814100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6815","nonce":2719814100}]},"public_addr":"172.21.15.38:6811/2719814100","cluster_addr":"172.21.15.38:6813/2719814100","heartbeat_back_addr":"172.21.15.38:6817/2719814100","heartbeat_front_addr":"172.21.15.38:6815/2719814100","state":["exists","up"]},{"osd":4,"uuid":"44a6dce3-e06e-4cbd-9404-62a86d01ab6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6809","nonce":2480764994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6811","nonce":2480764994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6815","nonce":2480764994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6813","nonce":2480764994}]},"public_addr":"172.21.15.122:6809/2480764994","cluster_addr":"172.21.15.122:6811/2480764994","heartbeat_back_addr":"172.21.15.122:6815/2480764994","heartbeat_front_addr":"172.21.15.122:6813/2480764994","state":["exists","up"]},{"osd":5,"uuid":"49749574-c00c-4e3a-80e6-e3686df58d5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6809","nonce":3141346246}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6811","nonce":3141346246}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6815","nonce":3141346246}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6813","nonce":3141346246}]},"public_addr":"172.21.15.181:6809/3141346246","cluster_addr":"172.21.15.181:6811/3141346246","heartbeat_back_addr":"172.21.15.181:6815/3141346246","heartbeat_front_addr":"172.21.15.181:6813/3141346246","state":["exists","up"]},{"osd":6,"uuid":"61c0437e-3f51-45a9-8a61-e98eea0093f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6819","nonce":2122788957}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6821","nonce":2122788957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6825","nonce":2122788957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6823","nonce":2122788957}]},"public_addr":"172.21.15.38:6819/2122788957","cluster_addr":"172.21.15.38:6821/2122788957","heartbeat_back_addr":"172.21.15.38:6825/2122788957","heartbeat_front_addr":"172.21.15.38:6823/2122788957","state":["exists","up"]},{"osd":7,"uuid":"231fecdd-939f-4c7b-83a1-c4a1cffd7139","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6817","nonce":656874139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6819","nonce":656874139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6823","nonce":656874139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6821","nonce":656874139}]},"public_addr":"172.21.15.122:6817/656874139","cluster_addr":"172.21.15.122:6819/656874139","heartbeat_back_addr":"172.21.15.122:6823/656874139","heartbeat_front_addr":"172.21.15.122:6821/656874139","state":["exists","up"]},{"osd":8,"uuid":"297e6d6d-85fd-43ce-9028-5e9dbe55c36e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6817","nonce":755000703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6819","nonce":755000703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6823","nonce":755000703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6821","nonce":755000703}]},"public_addr":"172.21.15.181:6817/755000703","cluster_addr":"172.21.15.181:6819/755000703","heartbeat_back_addr":"172.21.15.181:6823/755000703","heartbeat_front_addr":"172.21.15.181:6821/755000703","state":["exists","up"]},{"osd":9,"uuid":"94cc0355-5a1a-4cfc-bd0c-562b6b41127e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6827","nonce":2624555969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6829","nonce":2624555969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6833","nonce":2624555969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6831","nonce":2624555969}]},"public_addr":"172.21.15.38:6827/2624555969","cluster_addr":"172.21.15.38:6829/2624555969","heartbeat_back_addr":"172.21.15.38:6833/2624555969","heartbeat_front_addr":"172.21.15.38:6831/2624555969","state":["exists","up"]},{"osd":10,"uuid":"f4f2e042-8d92-46bf-93b8-8b0f04795c2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":81,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6825","nonce":1427785280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6827","nonce":1427785280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6831","nonce":1427785280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6829","nonce":1427785280}]},"public_addr":"172.21.15.122:6825/1427785280","cluster_addr":"172.21.15.122:6827/1427785280","heartbeat_back_addr":"172.21.15.122:6831/1427785280","heartbeat_front_addr":"172.21.15.122:6829/1427785280","state":["exists","up"]},{"osd":11,"uuid":"07444936-eae5-4097-a799-60294cafe80b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6825","nonce":3183488640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6827","nonce":3183488640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6831","nonce":3183488640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6829","nonce":3183488640}]},"public_addr":"172.21.15.181:6825/3183488640","cluster_addr":"172.21.15.181:6827/3183488640","heartbeat_back_addr":"172.21.15.181:6831/3183488640","heartbeat_front_addr":"172.21.15.181:6829/3183488640","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:39:37.657556+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:39:58.960074+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:40:20.326952+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:40:42.244387+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:41:03.946973+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:41:25.097985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:41:46.877916+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:42:08.413827+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:42:29.822009+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:42:51.776684+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:43:13.937249+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:43:35.569199+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/4272288735":"2023-03-10T14:38:14.612767+0000","172.21.15.38:6801/3381625669":"2023-03-10T14:38:14.612767+0000","172.21.15.38:6800/3381625669":"2023-03-10T14:38:14.612767+0000","172.21.15.38:6800/3060474342":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/4253918615":"2023-03-10T14:38:14.612767+0000","172.21.15.38:0/3496083181":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/245602368":"2023-03-10T14:37:47.517775+0000","172.21.15.38:6801/3060474342":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/3622589182":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/1430302251":"2023-03-10T14:38:14.612767+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-09T14:44:52.645 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:52.643+0000 7f8472250700 1 -- 172.21.15.38:0/3693090493 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f845809f480 msgr2=0x7f84580a1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:52.646 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:52.643+0000 7f8472250700 1 --2- 172.21.15.38:0/3693090493 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f845809f480 0x7f84580a1930 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f8454005fd0 tx=0x7f845400af20 comp rx=0 tx=0).stop 2023-03-09T14:44:52.646 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:52.643+0000 7f8472250700 1 -- 172.21.15.38:0/3693090493 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f846c100340 msgr2=0x7f846c198420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:52.646 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:52.643+0000 7f8472250700 1 --2- 172.21.15.38:0/3693090493 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f846c100340 0x7f846c198420 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f845c00c920 tx=0x7f845c00cc30 comp rx=0 tx=0).stop 2023-03-09T14:44:52.646 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:52.643+0000 7f8472250700 1 -- 172.21.15.38:0/3693090493 shutdown_connections 2023-03-09T14:44:52.646 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:52.643+0000 7f8472250700 1 --2- 172.21.15.38:0/3693090493 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f846c100340 0x7f846c198420 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:52.646 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:52.643+0000 7f8472250700 1 --2- 172.21.15.38:0/3693090493 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f845809f480 0x7f84580a1930 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:52.647 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:52.643+0000 7f8472250700 1 --2- 172.21.15.38:0/3693090493 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f846c1052a0 0x7f846c19cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:52.647 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:52.643+0000 7f8472250700 1 --2- 172.21.15.38:0/3693090493 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f846c100ce0 0x7f846c198960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:52.647 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:52.643+0000 7f8472250700 1 -- 172.21.15.38:0/3693090493 >> 172.21.15.38:0/3693090493 conn(0x7f846c077010 msgr2=0x7f846c0fdaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:52.647 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:52.643+0000 7f8472250700 1 -- 172.21.15.38:0/3693090493 shutdown_connections 2023-03-09T14:44:52.647 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:52.643+0000 7f8472250700 1 -- 172.21.15.38:0/3693090493 wait complete. 2023-03-09T14:44:53.082 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph tell osd.0 flush_pg_stats 2023-03-09T14:44:53.082 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph tell osd.1 flush_pg_stats 2023-03-09T14:44:53.083 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph tell osd.2 flush_pg_stats 2023-03-09T14:44:53.083 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph tell osd.3 flush_pg_stats 2023-03-09T14:44:53.084 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph tell osd.4 flush_pg_stats 2023-03-09T14:44:53.084 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph tell osd.5 flush_pg_stats 2023-03-09T14:44:53.085 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph tell osd.6 flush_pg_stats 2023-03-09T14:44:53.085 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph tell osd.7 flush_pg_stats 2023-03-09T14:44:53.086 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph tell osd.8 flush_pg_stats 2023-03-09T14:44:53.086 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph tell osd.9 flush_pg_stats 2023-03-09T14:44:53.087 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph tell osd.10 flush_pg_stats 2023-03-09T14:44:53.088 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph tell osd.11 flush_pg_stats 2023-03-09T14:44:53.547 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:53 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/3693090493' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-03-09T14:44:53.547 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:53 smithi038 ceph-mon[92422]: pgmap v276: 129 pgs: 129 active+clean; 635 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 5.2 KiB/s wr, 12 op/s 2023-03-09T14:44:53.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:53 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/3693090493' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-03-09T14:44:53.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:53 smithi122 ceph-mon[93869]: pgmap v276: 129 pgs: 129 active+clean; 635 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 5.2 KiB/s wr, 12 op/s 2023-03-09T14:44:53.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:53 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/3693090493' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-03-09T14:44:53.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:53 smithi181 ceph-mon[94189]: pgmap v276: 129 pgs: 129 active+clean; 635 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 5.2 KiB/s wr, 12 op/s 2023-03-09T14:44:54.346 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.a/config 2023-03-09T14:44:54.501 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.a/config 2023-03-09T14:44:54.699 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.a/config 2023-03-09T14:44:54.700 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.a/config 2023-03-09T14:44:54.996 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.a/config 2023-03-09T14:44:55.212 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.a/config 2023-03-09T14:44:55.440 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.a/config 2023-03-09T14:44:56.120 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:55 smithi038 ceph-mon[92422]: pgmap v277: 129 pgs: 129 active+clean; 635 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 3.3 KiB/s wr, 7 op/s 2023-03-09T14:44:56.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:55 smithi122 ceph-mon[93869]: pgmap v277: 129 pgs: 129 active+clean; 635 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 3.3 KiB/s wr, 7 op/s 2023-03-09T14:44:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:55 smithi181 ceph-mon[94189]: pgmap v277: 129 pgs: 129 active+clean; 635 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 3.3 KiB/s wr, 7 op/s 2023-03-09T14:44:56.378 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.a/config 2023-03-09T14:44:56.473 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.a/config 2023-03-09T14:44:56.712 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.a/config 2023-03-09T14:44:56.829 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.a/config 2023-03-09T14:44:56.830 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.a/config 2023-03-09T14:44:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:57 smithi038 ceph-mon[92422]: pgmap v278: 129 pgs: 129 active+clean; 635 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 3.2 KiB/s wr, 8 op/s 2023-03-09T14:44:58.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:57 smithi122 ceph-mon[93869]: pgmap v278: 129 pgs: 129 active+clean; 635 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 3.2 KiB/s wr, 8 op/s 2023-03-09T14:44:58.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:57 smithi181 ceph-mon[94189]: pgmap v278: 129 pgs: 129 active+clean; 635 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 3.2 KiB/s wr, 8 op/s 2023-03-09T14:44:59.326 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:58.833+0000 7f8305eaa700 1 -- 172.21.15.38:0/2096088021 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f83001010f0 msgr2=0x7f83001014c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:59.327 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:58.833+0000 7f8305eaa700 1 --2- 172.21.15.38:0/2096088021 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f83001010f0 0x7f83001014c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f82f0009990 tx=0x7f82f0009ca0 comp rx=0 tx=0).stop 2023-03-09T14:44:59.327 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:58.833+0000 7f8305eaa700 1 -- 172.21.15.38:0/2096088021 shutdown_connections 2023-03-09T14:44:59.327 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:58.833+0000 7f8305eaa700 1 --2- 172.21.15.38:0/2096088021 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f83001010f0 0x7f83001014c0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:59.327 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:58.833+0000 7f8305eaa700 1 --2- 172.21.15.38:0/2096088021 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f83000ff2f0 0x7f830010ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:59.327 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:58.833+0000 7f8305eaa700 1 --2- 172.21.15.38:0/2096088021 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f83000fe7a0 0x7f83000fec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:59.328 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:58.833+0000 7f8305eaa700 1 -- 172.21.15.38:0/2096088021 >> 172.21.15.38:0/2096088021 conn(0x7f83000745e0 msgr2=0x7f83000749e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:59.328 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:58.834+0000 7f8305eaa700 1 -- 172.21.15.38:0/2096088021 shutdown_connections 2023-03-09T14:44:59.328 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:58.834+0000 7f8305eaa700 1 -- 172.21.15.38:0/2096088021 wait complete. 2023-03-09T14:44:59.328 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:58.834+0000 7f8305eaa700 1 Processor -- start 2023-03-09T14:44:59.328 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:58.835+0000 7f8305eaa700 1 -- start start 2023-03-09T14:44:59.329 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:58.835+0000 7f8305eaa700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f83000fe7a0 0x7f8300198640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:59.330 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:58.835+0000 7f8305eaa700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f83000ff2f0 0x7f8300198b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:59.330 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:58.835+0000 7f8305eaa700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f830019cfe0 0x7f830019d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:59.330 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:58.835+0000 7f8305eaa700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f830019da50 con 0x7f83000fe7a0 2023-03-09T14:44:59.330 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:58.835+0000 7f8305eaa700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f830019dbc0 con 0x7f83000ff2f0 2023-03-09T14:44:59.330 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:58.835+0000 7f8305eaa700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f830019dd80 con 0x7f830019cfe0 2023-03-09T14:44:59.330 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:58.835+0000 7f82ff7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f83000fe7a0 0x7f8300198640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:59.330 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:58.835+0000 7f82ff7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f83000fe7a0 0x7f8300198640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37316/0 (socket says 172.21.15.38:37316) 2023-03-09T14:44:59.331 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:58.835+0000 7f82ff7fe700 1 -- 172.21.15.38:0/263656756 learned_addr learned my addr 172.21.15.38:0/263656756 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:44:59.331 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:58.835+0000 7f82fffff700 1 --2- 172.21.15.38:0/263656756 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f830019cfe0 0x7f830019d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:59.331 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:58.835+0000 7f82feffd700 1 --2- 172.21.15.38:0/263656756 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f83000ff2f0 0x7f8300198b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:59.331 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:58.835+0000 7f82ff7fe700 1 -- 172.21.15.38:0/263656756 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f830019cfe0 msgr2=0x7f830019d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:59.331 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:58.835+0000 7f82ff7fe700 1 --2- 172.21.15.38:0/263656756 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f830019cfe0 0x7f830019d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:59.331 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:58.835+0000 7f82ff7fe700 1 -- 172.21.15.38:0/263656756 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f83000ff2f0 msgr2=0x7f8300198b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:59.331 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:58.835+0000 7f82ff7fe700 1 --2- 172.21.15.38:0/263656756 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f83000ff2f0 0x7f8300198b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:59.331 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:58.835+0000 7f82ff7fe700 1 -- 172.21.15.38:0/263656756 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82f00095f0 con 0x7f83000fe7a0 2023-03-09T14:44:59.332 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:58.835+0000 7f82feffd700 1 --2- 172.21.15.38:0/263656756 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f83000ff2f0 0x7f8300198b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-09T14:44:59.332 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:58.835+0000 7f82ff7fe700 1 --2- 172.21.15.38:0/263656756 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f83000fe7a0 0x7f8300198640 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f82f0006010 tx=0x7f82f000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:59.332 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:58.836+0000 7f82fcff9700 1 -- 172.21.15.38:0/263656756 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82f001d070 con 0x7f83000fe7a0 2023-03-09T14:44:59.332 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:58.836+0000 7f82fcff9700 1 -- 172.21.15.38:0/263656756 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f82f0005030 con 0x7f83000fe7a0 2023-03-09T14:44:59.332 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:58.836+0000 7f8305eaa700 1 -- 172.21.15.38:0/263656756 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83001123c0 con 0x7f83000fe7a0 2023-03-09T14:44:59.332 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:58.836+0000 7f82fcff9700 1 -- 172.21.15.38:0/263656756 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82f0004a00 con 0x7f83000fe7a0 2023-03-09T14:44:59.332 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:58.836+0000 7f8305eaa700 1 -- 172.21.15.38:0/263656756 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8300112230 con 0x7f83000fe7a0 2023-03-09T14:44:59.333 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:58.837+0000 7f8305eaa700 1 -- 172.21.15.38:0/263656756 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f83000fcc10 con 0x7f83000fe7a0 2023-03-09T14:44:59.333 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:58.838+0000 7f82fcff9700 1 -- 172.21.15.38:0/263656756 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 133238+0+0 (secure 0 0 0) 0x7f82f0028020 con 0x7f83000fe7a0 2023-03-09T14:44:59.333 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:58.838+0000 7f82fcff9700 1 --2- 172.21.15.38:0/263656756 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f82e809f400 0x7f82e80a18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:59.333 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:58.839+0000 7f82feffd700 1 --2- 172.21.15.38:0/263656756 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f82e809f400 0x7f82e80a18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:59.333 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:58.839+0000 7f82fcff9700 1 -- 172.21.15.38:0/263656756 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f82f00cb1e0 con 0x7f83000fe7a0 2023-03-09T14:44:59.333 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:58.839+0000 7f82fcff9700 1 --2- 172.21.15.38:0/263656756 >> [v2:172.21.15.122:6800/3929268773,v1:172.21.15.122:6801/3929268773] conn(0x7f82e80a9ef0 0x7f82e80ac320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:59.333 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:58.839+0000 7f82fcff9700 1 -- 172.21.15.38:0/263656756 --> [v2:172.21.15.122:6800/3929268773,v1:172.21.15.122:6801/3929268773] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f82e80ac9f0 con 0x7f82e80a9ef0 2023-03-09T14:44:59.334 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:58.839+0000 7f82fcff9700 1 -- 172.21.15.38:0/263656756 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) v2 ==== 24+0+0 (secure 0 0 0) 0x7f82f00cb550 con 0x7f83000fe7a0 2023-03-09T14:44:59.334 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:58.839+0000 7f82feffd700 1 --2- 172.21.15.38:0/263656756 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f82e809f400 0x7f82e80a18b0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f82f4005d90 tx=0x7f82f4005d20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:59.334 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:58.839+0000 7f82fffff700 1 --2- 172.21.15.38:0/263656756 >> [v2:172.21.15.122:6800/3929268773,v1:172.21.15.122:6801/3929268773] conn(0x7f82e80a9ef0 0x7f82e80ac320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:59.334 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:58.840+0000 7f82fffff700 1 --2- 172.21.15.38:0/263656756 >> [v2:172.21.15.122:6800/3929268773,v1:172.21.15.122:6801/3929268773] conn(0x7f82e80a9ef0 0x7f82e80ac320 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:59.334 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:58.841+0000 7f82fcff9700 1 -- 172.21.15.38:0/263656756 <== osd.1 v2:172.21.15.122:6800/3929268773 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f82e80ac9f0 con 0x7f82e80a9ef0 2023-03-09T14:44:59.334 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:58.857+0000 7f8305eaa700 1 -- 172.21.15.38:0/263656756 --> [v2:172.21.15.122:6800/3929268773,v1:172.21.15.122:6801/3929268773] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8300112550 con 0x7f82e80a9ef0 2023-03-09T14:44:59.334 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:58.858+0000 7f82fcff9700 1 -- 172.21.15.38:0/263656756 <== osd.1 v2:172.21.15.122:6800/3929268773 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f8300112550 con 0x7f82e80a9ef0 2023-03-09T14:44:59.335 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:58.858+0000 7f8305eaa700 1 -- 172.21.15.38:0/263656756 >> [v2:172.21.15.122:6800/3929268773,v1:172.21.15.122:6801/3929268773] conn(0x7f82e80a9ef0 msgr2=0x7f82e80ac320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:59.335 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:58.858+0000 7f8305eaa700 1 --2- 172.21.15.38:0/263656756 >> [v2:172.21.15.122:6800/3929268773,v1:172.21.15.122:6801/3929268773] conn(0x7f82e80a9ef0 0x7f82e80ac320 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:59.335 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:58.858+0000 7f8305eaa700 1 -- 172.21.15.38:0/263656756 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f82e809f400 msgr2=0x7f82e80a18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:59.335 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:58.858+0000 7f8305eaa700 1 --2- 172.21.15.38:0/263656756 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f82e809f400 0x7f82e80a18b0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f82f4005d90 tx=0x7f82f4005d20 comp rx=0 tx=0).stop 2023-03-09T14:44:59.335 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:58.858+0000 7f8305eaa700 1 -- 172.21.15.38:0/263656756 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f83000fe7a0 msgr2=0x7f8300198640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:59.336 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:58.858+0000 7f8305eaa700 1 --2- 172.21.15.38:0/263656756 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f83000fe7a0 0x7f8300198640 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f82f0006010 tx=0x7f82f000f690 comp rx=0 tx=0).stop 2023-03-09T14:44:59.336 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:58.858+0000 7f82fffff700 1 -- 172.21.15.38:0/263656756 reap_dead start 2023-03-09T14:44:59.336 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:58.858+0000 7f8305eaa700 1 -- 172.21.15.38:0/263656756 shutdown_connections 2023-03-09T14:44:59.336 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:58.858+0000 7f8305eaa700 1 -- 172.21.15.38:0/263656756 >> 172.21.15.38:0/263656756 conn(0x7f83000745e0 msgr2=0x7f83000fc1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:59.336 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:58.858+0000 7f8305eaa700 1 -- 172.21.15.38:0/263656756 shutdown_connections 2023-03-09T14:44:59.336 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:58.858+0000 7f8305eaa700 1 -- 172.21.15.38:0/263656756 wait complete. 2023-03-09T14:44:59.488 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.485+0000 7f0ec79ae700 1 -- 172.21.15.38:0/2474507254 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ec00ffac0 msgr2=0x7f0ec0103a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:59.488 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.485+0000 7f0ec79ae700 1 --2- 172.21.15.38:0/2474507254 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ec00ffac0 0x7f0ec0103a80 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f0eb00099c0 tx=0x7f0eb0009cd0 comp rx=0 tx=0).stop 2023-03-09T14:44:59.488 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.485+0000 7f0ec79ae700 1 -- 172.21.15.38:0/2474507254 shutdown_connections 2023-03-09T14:44:59.489 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.485+0000 7f0ec79ae700 1 --2- 172.21.15.38:0/2474507254 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0ec0104160 0x7f0ec010bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:59.489 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.485+0000 7f0ec79ae700 1 --2- 172.21.15.38:0/2474507254 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0ec00ff120 0x7f0ec00ff4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:59.489 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.485+0000 7f0ec79ae700 1 --2- 172.21.15.38:0/2474507254 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ec00ffac0 0x7f0ec0103a80 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:59.489 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.485+0000 7f0ec79ae700 1 -- 172.21.15.38:0/2474507254 >> 172.21.15.38:0/2474507254 conn(0x7f0ec0076d10 msgr2=0x7f0ec0077110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:59.489 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.485+0000 7f0ec79ae700 1 -- 172.21.15.38:0/2474507254 shutdown_connections 2023-03-09T14:44:59.490 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.485+0000 7f0ec79ae700 1 -- 172.21.15.38:0/2474507254 wait complete. 2023-03-09T14:44:59.490 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.486+0000 7f0ec79ae700 1 Processor -- start 2023-03-09T14:44:59.490 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.486+0000 7f0ec79ae700 1 -- start start 2023-03-09T14:44:59.492 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.486+0000 7f0ec79ae700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0ec00ff120 0x7f0ec0100b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:59.492 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.486+0000 7f0ec79ae700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ec00ffac0 0x7f0ec0101060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:59.492 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.486+0000 7f0ec79ae700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0ec0104160 0x7f0ec01a4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:59.493 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.486+0000 7f0ec79ae700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec0101890 con 0x7f0ec00ffac0 2023-03-09T14:44:59.493 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.486+0000 7f0ec79ae700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec01a4840 con 0x7f0ec00ff120 2023-03-09T14:44:59.493 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.486+0000 7f0ec79ae700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ec01a4980 con 0x7f0ec0104160 2023-03-09T14:44:59.493 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.486+0000 7f0ec4f49700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ec00ffac0 0x7f0ec0101060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:59.493 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.486+0000 7f0ec4f49700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ec00ffac0 0x7f0ec0101060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37334/0 (socket says 172.21.15.38:37334) 2023-03-09T14:44:59.493 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.486+0000 7f0ec4f49700 1 -- 172.21.15.38:0/1285348620 learned_addr learned my addr 172.21.15.38:0/1285348620 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:44:59.493 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.486+0000 7f0ec574a700 1 --2- 172.21.15.38:0/1285348620 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0ec00ff120 0x7f0ec0100b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:59.494 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.486+0000 7f0ec5f4b700 1 --2- 172.21.15.38:0/1285348620 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0ec0104160 0x7f0ec01a4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:59.494 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.486+0000 7f0ec4f49700 1 -- 172.21.15.38:0/1285348620 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0ec0104160 msgr2=0x7f0ec01a4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:59.494 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.486+0000 7f0ec4f49700 1 --2- 172.21.15.38:0/1285348620 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0ec0104160 0x7f0ec01a4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:59.494 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.486+0000 7f0ec4f49700 1 -- 172.21.15.38:0/1285348620 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0ec00ff120 msgr2=0x7f0ec0100b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:59.494 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.486+0000 7f0ec4f49700 1 --2- 172.21.15.38:0/1285348620 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0ec00ff120 0x7f0ec0100b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:59.494 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.486+0000 7f0ec4f49700 1 -- 172.21.15.38:0/1285348620 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eb00095f0 con 0x7f0ec00ffac0 2023-03-09T14:44:59.494 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.486+0000 7f0ec5f4b700 1 --2- 172.21.15.38:0/1285348620 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0ec0104160 0x7f0ec01a4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:44:59.494 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.487+0000 7f0ec4f49700 1 --2- 172.21.15.38:0/1285348620 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ec00ffac0 0x7f0ec0101060 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f0eb0006010 tx=0x7f0eb0005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:59.495 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.487+0000 7f0eb67fc700 1 -- 172.21.15.38:0/1285348620 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eb001d070 con 0x7f0ec00ffac0 2023-03-09T14:44:59.495 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.487+0000 7f0eb67fc700 1 -- 172.21.15.38:0/1285348620 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0eb0004050 con 0x7f0ec00ffac0 2023-03-09T14:44:59.495 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.487+0000 7f0eb67fc700 1 -- 172.21.15.38:0/1285348620 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0eb0022e10 con 0x7f0ec00ffac0 2023-03-09T14:44:59.495 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.487+0000 7f0ec79ae700 1 -- 172.21.15.38:0/1285348620 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ec010eee0 con 0x7f0ec00ffac0 2023-03-09T14:44:59.495 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.487+0000 7f0ec79ae700 1 -- 172.21.15.38:0/1285348620 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ec010ed50 con 0x7f0ec00ffac0 2023-03-09T14:44:59.495 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.487+0000 7f0ec79ae700 1 -- 172.21.15.38:0/1285348620 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0ec0068cf0 con 0x7f0ec00ffac0 2023-03-09T14:44:59.495 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.489+0000 7f0eb67fc700 1 -- 172.21.15.38:0/1285348620 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 133238+0+0 (secure 0 0 0) 0x7f0eb0029030 con 0x7f0ec00ffac0 2023-03-09T14:44:59.496 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.489+0000 7f0eb67fc700 1 --2- 172.21.15.38:0/1285348620 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f0eac09f400 0x7f0eac0a18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:59.496 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.489+0000 7f0eb67fc700 1 -- 172.21.15.38:0/1285348620 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f0eb00cbc20 con 0x7f0ec00ffac0 2023-03-09T14:44:59.496 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.489+0000 7f0ec574a700 1 --2- 172.21.15.38:0/1285348620 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f0eac09f400 0x7f0eac0a18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:59.496 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.490+0000 7f0eb67fc700 1 --2- 172.21.15.38:0/1285348620 >> [v2:172.21.15.38:6818/2122788957,v1:172.21.15.38:6819/2122788957] conn(0x7f0eac0a9ef0 0x7f0eac0ac320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:59.496 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.490+0000 7f0eb67fc700 1 -- 172.21.15.38:0/1285348620 --> [v2:172.21.15.38:6818/2122788957,v1:172.21.15.38:6819/2122788957] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0eac0ac9f0 con 0x7f0eac0a9ef0 2023-03-09T14:44:59.496 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.490+0000 7f0eb67fc700 1 -- 172.21.15.38:0/1285348620 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0eb00cbff0 con 0x7f0ec00ffac0 2023-03-09T14:44:59.496 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.490+0000 7f0ec5f4b700 1 --2- 172.21.15.38:0/1285348620 >> [v2:172.21.15.38:6818/2122788957,v1:172.21.15.38:6819/2122788957] conn(0x7f0eac0a9ef0 0x7f0eac0ac320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:59.496 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.490+0000 7f0ec574a700 1 --2- 172.21.15.38:0/1285348620 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f0eac09f400 0x7f0eac0a18b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc005950 tx=0x7f0ebc00b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:59.497 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.490+0000 7f0ec5f4b700 1 --2- 172.21.15.38:0/1285348620 >> [v2:172.21.15.38:6818/2122788957,v1:172.21.15.38:6819/2122788957] conn(0x7f0eac0a9ef0 0x7f0eac0ac320 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:59.497 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.491+0000 7f0eb67fc700 1 -- 172.21.15.38:0/1285348620 <== osd.6 v2:172.21.15.38:6818/2122788957 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f0eac0ac9f0 con 0x7f0eac0a9ef0 2023-03-09T14:44:59.510 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.507+0000 7f0ec79ae700 1 -- 172.21.15.38:0/1285348620 --> [v2:172.21.15.38:6818/2122788957,v1:172.21.15.38:6819/2122788957] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0ec010f070 con 0x7f0eac0a9ef0 2023-03-09T14:44:59.511 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.508+0000 7f0eb67fc700 1 -- 172.21.15.38:0/1285348620 <== osd.6 v2:172.21.15.38:6818/2122788957 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f0ec010f070 con 0x7f0eac0a9ef0 2023-03-09T14:44:59.511 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.509+0000 7f0ec79ae700 1 -- 172.21.15.38:0/1285348620 >> [v2:172.21.15.38:6818/2122788957,v1:172.21.15.38:6819/2122788957] conn(0x7f0eac0a9ef0 msgr2=0x7f0eac0ac320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:59.511 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.509+0000 7f0ec79ae700 1 --2- 172.21.15.38:0/1285348620 >> [v2:172.21.15.38:6818/2122788957,v1:172.21.15.38:6819/2122788957] conn(0x7f0eac0a9ef0 0x7f0eac0ac320 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:59.512 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.509+0000 7f0ec79ae700 1 -- 172.21.15.38:0/1285348620 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f0eac09f400 msgr2=0x7f0eac0a18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:59.512 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.509+0000 7f0ec79ae700 1 --2- 172.21.15.38:0/1285348620 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f0eac09f400 0x7f0eac0a18b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc005950 tx=0x7f0ebc00b410 comp rx=0 tx=0).stop 2023-03-09T14:44:59.512 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.509+0000 7f0ec79ae700 1 -- 172.21.15.38:0/1285348620 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ec00ffac0 msgr2=0x7f0ec0101060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:59.512 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.509+0000 7f0ec79ae700 1 --2- 172.21.15.38:0/1285348620 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0ec00ffac0 0x7f0ec0101060 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f0eb0006010 tx=0x7f0eb0005740 comp rx=0 tx=0).stop 2023-03-09T14:44:59.512 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.509+0000 7f0ec5f4b700 1 -- 172.21.15.38:0/1285348620 reap_dead start 2023-03-09T14:44:59.512 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.509+0000 7f0ec79ae700 1 -- 172.21.15.38:0/1285348620 shutdown_connections 2023-03-09T14:44:59.512 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.509+0000 7f0ec79ae700 1 -- 172.21.15.38:0/1285348620 >> 172.21.15.38:0/1285348620 conn(0x7f0ec0076d10 msgr2=0x7f0ec0068320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:59.513 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.509+0000 7f0ec79ae700 1 -- 172.21.15.38:0/1285348620 shutdown_connections 2023-03-09T14:44:59.513 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.509+0000 7f0ec79ae700 1 -- 172.21.15.38:0/1285348620 wait complete. 2023-03-09T14:44:59.907 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.904+0000 7f4837498700 1 -- 172.21.15.38:0/1634672988 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4830100bd0 msgr2=0x7f4830104b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:59.907 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.904+0000 7f4837498700 1 --2- 172.21.15.38:0/1634672988 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4830100bd0 0x7f4830104b90 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f48200099c0 tx=0x7f4820009cd0 comp rx=0 tx=0).stop 2023-03-09T14:44:59.907 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.904+0000 7f4837498700 1 -- 172.21.15.38:0/1634672988 shutdown_connections 2023-03-09T14:44:59.908 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.904+0000 7f4837498700 1 --2- 172.21.15.38:0/1634672988 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4830105270 0x7f483010ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:59.908 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.904+0000 7f4837498700 1 --2- 172.21.15.38:0/1634672988 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4830100230 0x7f4830100600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:59.908 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.904+0000 7f4837498700 1 --2- 172.21.15.38:0/1634672988 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4830100bd0 0x7f4830104b90 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:59.908 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.904+0000 7f4837498700 1 -- 172.21.15.38:0/1634672988 >> 172.21.15.38:0/1634672988 conn(0x7f4830076d10 msgr2=0x7f4830077110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:59.908 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.905+0000 7f4837498700 1 -- 172.21.15.38:0/1634672988 shutdown_connections 2023-03-09T14:44:59.908 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.905+0000 7f4837498700 1 -- 172.21.15.38:0/1634672988 wait complete. 2023-03-09T14:44:59.910 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.905+0000 7f4837498700 1 Processor -- start 2023-03-09T14:44:59.910 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.905+0000 7f4837498700 1 -- start start 2023-03-09T14:44:59.910 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.905+0000 7f4837498700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4830100230 0x7f4830072a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:59.910 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.905+0000 7f4837498700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4830100bd0 0x7f4830072fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:59.910 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.906+0000 7f4837498700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4830105270 0x7f48301a4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:59.911 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.906+0000 7f4837498700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48300737f0 con 0x7f4830100230 2023-03-09T14:44:59.911 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.906+0000 7f4837498700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48301a4840 con 0x7f4830100bd0 2023-03-09T14:44:59.911 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.906+0000 7f4837498700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48301a4980 con 0x7f4830105270 2023-03-09T14:44:59.911 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.906+0000 7f4835234700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4830100230 0x7f4830072a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:59.911 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.906+0000 7f4835234700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4830100230 0x7f4830072a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37348/0 (socket says 172.21.15.38:37348) 2023-03-09T14:44:59.911 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.906+0000 7f4835234700 1 -- 172.21.15.38:0/1008456488 learned_addr learned my addr 172.21.15.38:0/1008456488 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:44:59.911 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.906+0000 7f4834a33700 1 --2- 172.21.15.38:0/1008456488 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4830100bd0 0x7f4830072fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:59.912 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.906+0000 7f4835a35700 1 --2- 172.21.15.38:0/1008456488 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4830105270 0x7f48301a4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:59.912 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.906+0000 7f4835234700 1 -- 172.21.15.38:0/1008456488 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4830105270 msgr2=0x7f48301a4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:59.912 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.906+0000 7f4835234700 1 --2- 172.21.15.38:0/1008456488 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4830105270 0x7f48301a4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:59.912 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.906+0000 7f4835234700 1 -- 172.21.15.38:0/1008456488 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4830100bd0 msgr2=0x7f4830072fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:59.912 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.906+0000 7f4835234700 1 --2- 172.21.15.38:0/1008456488 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4830100bd0 0x7f4830072fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:59.912 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.906+0000 7f4835234700 1 -- 172.21.15.38:0/1008456488 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48200095f0 con 0x7f4830100230 2023-03-09T14:44:59.912 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.906+0000 7f4835a35700 1 --2- 172.21.15.38:0/1008456488 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4830105270 0x7f48301a4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:44:59.913 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.906+0000 7f4835234700 1 --2- 172.21.15.38:0/1008456488 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4830100230 0x7f4830072a80 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f482c00d950 tx=0x7f482c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:59.913 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.906+0000 7f48267fc700 1 -- 172.21.15.38:0/1008456488 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f482c009940 con 0x7f4830100230 2023-03-09T14:44:59.913 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.906+0000 7f4837498700 1 -- 172.21.15.38:0/1008456488 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f483010fff0 con 0x7f4830100230 2023-03-09T14:44:59.913 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.906+0000 7f48267fc700 1 -- 172.21.15.38:0/1008456488 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f482c010460 con 0x7f4830100230 2023-03-09T14:44:59.913 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.906+0000 7f48267fc700 1 -- 172.21.15.38:0/1008456488 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f482c00f620 con 0x7f4830100230 2023-03-09T14:44:59.913 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.907+0000 7f4837498700 1 -- 172.21.15.38:0/1008456488 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f483010fe60 con 0x7f4830100230 2023-03-09T14:44:59.913 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.907+0000 7f4837498700 1 -- 172.21.15.38:0/1008456488 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f48301045c0 con 0x7f4830100230 2023-03-09T14:44:59.914 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.909+0000 7f48267fc700 1 -- 172.21.15.38:0/1008456488 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 133238+0+0 (secure 0 0 0) 0x7f482c022070 con 0x7f4830100230 2023-03-09T14:44:59.915 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.909+0000 7f48267fc700 1 --2- 172.21.15.38:0/1008456488 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f481c09f400 0x7f481c0a18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:59.915 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.909+0000 7f48267fc700 1 -- 172.21.15.38:0/1008456488 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f482c0ca350 con 0x7f4830100230 2023-03-09T14:44:59.915 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.909+0000 7f4834a33700 1 --2- 172.21.15.38:0/1008456488 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f481c09f400 0x7f481c0a18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:59.915 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.910+0000 7f48267fc700 1 --2- 172.21.15.38:0/1008456488 >> [v2:172.21.15.181:6816/755000703,v1:172.21.15.181:6817/755000703] conn(0x7f481c0a9ef0 0x7f481c0ac320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:44:59.915 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.910+0000 7f48267fc700 1 -- 172.21.15.38:0/1008456488 --> [v2:172.21.15.181:6816/755000703,v1:172.21.15.181:6817/755000703] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f481c0ac9f0 con 0x7f481c0a9ef0 2023-03-09T14:44:59.915 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.910+0000 7f48267fc700 1 -- 172.21.15.38:0/1008456488 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) v2 ==== 24+0+0 (secure 0 0 0) 0x7f482c0ca720 con 0x7f4830100230 2023-03-09T14:44:59.915 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.910+0000 7f4834a33700 1 --2- 172.21.15.38:0/1008456488 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f481c09f400 0x7f481c0a18b0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f4820009f50 tx=0x7f48200050e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:59.916 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.910+0000 7f4835a35700 1 --2- 172.21.15.38:0/1008456488 >> [v2:172.21.15.181:6816/755000703,v1:172.21.15.181:6817/755000703] conn(0x7f481c0a9ef0 0x7f481c0ac320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:44:59.916 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.910+0000 7f4835a35700 1 --2- 172.21.15.38:0/1008456488 >> [v2:172.21.15.181:6816/755000703,v1:172.21.15.181:6817/755000703] conn(0x7f481c0a9ef0 0x7f481c0ac320 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:44:59.916 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.911+0000 7f48267fc700 1 -- 172.21.15.38:0/1008456488 <== osd.8 v2:172.21.15.181:6816/755000703 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f481c0ac9f0 con 0x7f481c0a9ef0 2023-03-09T14:44:59.929 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.927+0000 7f4837498700 1 -- 172.21.15.38:0/1008456488 --> [v2:172.21.15.181:6816/755000703,v1:172.21.15.181:6817/755000703] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f4830110180 con 0x7f481c0a9ef0 2023-03-09T14:44:59.930 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.928+0000 7f48267fc700 1 -- 172.21.15.38:0/1008456488 <== osd.8 v2:172.21.15.181:6816/755000703 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f4830110180 con 0x7f481c0a9ef0 2023-03-09T14:44:59.931 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.928+0000 7f4837498700 1 -- 172.21.15.38:0/1008456488 >> [v2:172.21.15.181:6816/755000703,v1:172.21.15.181:6817/755000703] conn(0x7f481c0a9ef0 msgr2=0x7f481c0ac320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:59.931 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.928+0000 7f4837498700 1 --2- 172.21.15.38:0/1008456488 >> [v2:172.21.15.181:6816/755000703,v1:172.21.15.181:6817/755000703] conn(0x7f481c0a9ef0 0x7f481c0ac320 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:44:59.931 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.928+0000 7f4837498700 1 -- 172.21.15.38:0/1008456488 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f481c09f400 msgr2=0x7f481c0a18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:59.931 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.928+0000 7f4837498700 1 --2- 172.21.15.38:0/1008456488 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f481c09f400 0x7f481c0a18b0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f4820009f50 tx=0x7f48200050e0 comp rx=0 tx=0).stop 2023-03-09T14:44:59.931 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.928+0000 7f4837498700 1 -- 172.21.15.38:0/1008456488 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4830100230 msgr2=0x7f4830072a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:44:59.932 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.928+0000 7f4837498700 1 --2- 172.21.15.38:0/1008456488 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4830100230 0x7f4830072a80 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f482c00d950 tx=0x7f482c00dc60 comp rx=0 tx=0).stop 2023-03-09T14:44:59.932 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.928+0000 7f4835a35700 1 -- 172.21.15.38:0/1008456488 reap_dead start 2023-03-09T14:44:59.932 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.928+0000 7f4837498700 1 -- 172.21.15.38:0/1008456488 shutdown_connections 2023-03-09T14:44:59.932 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.928+0000 7f4837498700 1 -- 172.21.15.38:0/1008456488 >> 172.21.15.38:0/1008456488 conn(0x7f4830076d10 msgr2=0x7f4830103b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:44:59.932 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.928+0000 7f4837498700 1 -- 172.21.15.38:0/1008456488 shutdown_connections 2023-03-09T14:44:59.932 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.928+0000 7f4837498700 1 -- 172.21.15.38:0/1008456488 wait complete. 2023-03-09T14:45:00.027 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:44:59 smithi038 ceph-mon[92422]: pgmap v279: 129 pgs: 129 active+clean; 635 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2023-03-09T14:45:00.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:44:59 smithi122 ceph-mon[93869]: pgmap v279: 129 pgs: 129 active+clean; 635 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2023-03-09T14:45:00.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:44:59 smithi181 ceph-mon[94189]: pgmap v279: 129 pgs: 129 active+clean; 635 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2023-03-09T14:45:00.243 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.975+0000 7fd9f0384700 1 -- 172.21.15.38:0/342076472 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd9e80732c0 msgr2=0x7fd9e810c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:00.243 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.975+0000 7fd9f0384700 1 --2- 172.21.15.38:0/342076472 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd9e80732c0 0x7fd9e810c540 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd9d80098f0 tx=0x7fd9d8009c00 comp rx=0 tx=0).stop 2023-03-09T14:45:00.243 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.975+0000 7fd9f0384700 1 -- 172.21.15.38:0/342076472 shutdown_connections 2023-03-09T14:45:00.243 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.975+0000 7fd9f0384700 1 --2- 172.21.15.38:0/342076472 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd9e810ca80 0x7fd9e810ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:00.244 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.975+0000 7fd9f0384700 1 --2- 172.21.15.38:0/342076472 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd9e80729b0 0x7fd9e8072d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:00.244 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.975+0000 7fd9f0384700 1 --2- 172.21.15.38:0/342076472 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd9e80732c0 0x7fd9e810c540 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:00.244 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.975+0000 7fd9f0384700 1 -- 172.21.15.38:0/342076472 >> 172.21.15.38:0/342076472 conn(0x7fd9e8077010 msgr2=0x7fd9e8077410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:00.244 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.975+0000 7fd9f0384700 1 -- 172.21.15.38:0/342076472 shutdown_connections 2023-03-09T14:45:00.244 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.975+0000 7fd9f0384700 1 -- 172.21.15.38:0/342076472 wait complete. 2023-03-09T14:45:00.244 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.976+0000 7fd9f0384700 1 Processor -- start 2023-03-09T14:45:00.244 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.976+0000 7fd9f0384700 1 -- start start 2023-03-09T14:45:00.244 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.976+0000 7fd9f0384700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd9e80729b0 0x7fd9e81983c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:00.245 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.976+0000 7fd9f0384700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd9e80732c0 0x7fd9e8198900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:00.245 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.976+0000 7fd9f0384700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd9e810ca80 0x7fd9e819cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:00.245 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.976+0000 7fd9f0384700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e819d3a0 con 0x7fd9e80729b0 2023-03-09T14:45:00.245 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.976+0000 7fd9f0384700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e819d510 con 0x7fd9e80732c0 2023-03-09T14:45:00.245 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.976+0000 7fd9f0384700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e819d6d0 con 0x7fd9e810ca80 2023-03-09T14:45:00.245 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.976+0000 7fd9ee120700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd9e80729b0 0x7fd9e81983c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:00.245 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.976+0000 7fd9ee120700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd9e80729b0 0x7fd9e81983c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37364/0 (socket says 172.21.15.38:37364) 2023-03-09T14:45:00.246 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.976+0000 7fd9ee120700 1 -- 172.21.15.38:0/2855486721 learned_addr learned my addr 172.21.15.38:0/2855486721 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:00.246 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.977+0000 7fd9ed91f700 1 --2- 172.21.15.38:0/2855486721 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd9e80732c0 0x7fd9e8198900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:00.246 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.977+0000 7fd9ee921700 1 --2- 172.21.15.38:0/2855486721 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd9e810ca80 0x7fd9e819cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:00.246 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.977+0000 7fd9ee120700 1 -- 172.21.15.38:0/2855486721 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd9e810ca80 msgr2=0x7fd9e819cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:00.246 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.977+0000 7fd9ee120700 1 --2- 172.21.15.38:0/2855486721 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd9e810ca80 0x7fd9e819cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:00.246 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.977+0000 7fd9ee120700 1 -- 172.21.15.38:0/2855486721 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd9e80732c0 msgr2=0x7fd9e8198900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:00.246 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.977+0000 7fd9ee120700 1 --2- 172.21.15.38:0/2855486721 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd9e80732c0 0x7fd9e8198900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:00.246 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.977+0000 7fd9ee120700 1 -- 172.21.15.38:0/2855486721 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9d8009520 con 0x7fd9e80729b0 2023-03-09T14:45:00.247 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.977+0000 7fd9ee120700 1 --2- 172.21.15.38:0/2855486721 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd9e80729b0 0x7fd9e81983c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fd9e400e9c0 tx=0x7fd9e400ed80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:00.247 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.977+0000 7fd9df7fe700 1 -- 172.21.15.38:0/2855486721 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e40097b0 con 0x7fd9e80729b0 2023-03-09T14:45:00.247 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.977+0000 7fd9df7fe700 1 -- 172.21.15.38:0/2855486721 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd9e4004cf0 con 0x7fd9e80729b0 2023-03-09T14:45:00.247 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.977+0000 7fd9f0384700 1 -- 172.21.15.38:0/2855486721 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9e8112290 con 0x7fd9e80729b0 2023-03-09T14:45:00.247 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.977+0000 7fd9df7fe700 1 -- 172.21.15.38:0/2855486721 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e4005700 con 0x7fd9e80729b0 2023-03-09T14:45:00.247 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.977+0000 7fd9f0384700 1 -- 172.21.15.38:0/2855486721 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9e8112100 con 0x7fd9e80729b0 2023-03-09T14:45:00.247 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.978+0000 7fd9f0384700 1 -- 172.21.15.38:0/2855486721 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd9e80ff4e0 con 0x7fd9e80729b0 2023-03-09T14:45:00.248 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.979+0000 7fd9df7fe700 1 -- 172.21.15.38:0/2855486721 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 133238+0+0 (secure 0 0 0) 0x7fd9e401e020 con 0x7fd9e80729b0 2023-03-09T14:45:00.248 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.980+0000 7fd9df7fe700 1 --2- 172.21.15.38:0/2855486721 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fd9d409f480 0x7fd9d40a1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:00.248 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.980+0000 7fd9ed91f700 1 --2- 172.21.15.38:0/2855486721 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fd9d409f480 0x7fd9d40a1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:00.248 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.980+0000 7fd9df7fe700 1 -- 172.21.15.38:0/2855486721 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fd9e4014070 con 0x7fd9e80729b0 2023-03-09T14:45:00.248 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.980+0000 7fd9df7fe700 1 --2- 172.21.15.38:0/2855486721 >> [v2:172.21.15.181:6808/3141346246,v1:172.21.15.181:6809/3141346246] conn(0x7fd9d40a9f70 0x7fd9d40ac3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:00.248 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.980+0000 7fd9df7fe700 1 -- 172.21.15.38:0/2855486721 --> [v2:172.21.15.181:6808/3141346246,v1:172.21.15.181:6809/3141346246] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd9d40aca70 con 0x7fd9d40a9f70 2023-03-09T14:45:00.248 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.980+0000 7fd9df7fe700 1 -- 172.21.15.38:0/2855486721 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd9e40caa30 con 0x7fd9e80729b0 2023-03-09T14:45:00.249 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.980+0000 7fd9ed91f700 1 --2- 172.21.15.38:0/2855486721 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fd9d409f480 0x7fd9d40a1930 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd9d8009fd0 tx=0x7fd9d8019040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:00.249 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.980+0000 7fd9ee921700 1 --2- 172.21.15.38:0/2855486721 >> [v2:172.21.15.181:6808/3141346246,v1:172.21.15.181:6809/3141346246] conn(0x7fd9d40a9f70 0x7fd9d40ac3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:00.249 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.981+0000 7fd9ee921700 1 --2- 172.21.15.38:0/2855486721 >> [v2:172.21.15.181:6808/3141346246,v1:172.21.15.181:6809/3141346246] conn(0x7fd9d40a9f70 0x7fd9d40ac3a0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:00.249 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.981+0000 7fd9df7fe700 1 -- 172.21.15.38:0/2855486721 <== osd.5 v2:172.21.15.181:6808/3141346246 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fd9d40aca70 con 0x7fd9d40a9f70 2023-03-09T14:45:00.249 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.998+0000 7fd9f0384700 1 -- 172.21.15.38:0/2855486721 --> [v2:172.21.15.181:6808/3141346246,v1:172.21.15.181:6809/3141346246] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd9e8112420 con 0x7fd9d40a9f70 2023-03-09T14:45:00.249 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:44:59.999+0000 7fd9df7fe700 1 -- 172.21.15.38:0/2855486721 <== osd.5 v2:172.21.15.181:6808/3141346246 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fd9e8112420 con 0x7fd9d40a9f70 2023-03-09T14:45:00.250 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:00.000+0000 7fd9f0384700 1 -- 172.21.15.38:0/2855486721 >> [v2:172.21.15.181:6808/3141346246,v1:172.21.15.181:6809/3141346246] conn(0x7fd9d40a9f70 msgr2=0x7fd9d40ac3a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:00.250 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:00.000+0000 7fd9f0384700 1 --2- 172.21.15.38:0/2855486721 >> [v2:172.21.15.181:6808/3141346246,v1:172.21.15.181:6809/3141346246] conn(0x7fd9d40a9f70 0x7fd9d40ac3a0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:00.250 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:00.000+0000 7fd9f0384700 1 -- 172.21.15.38:0/2855486721 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fd9d409f480 msgr2=0x7fd9d40a1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:00.250 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:00.000+0000 7fd9f0384700 1 --2- 172.21.15.38:0/2855486721 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fd9d409f480 0x7fd9d40a1930 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fd9d8009fd0 tx=0x7fd9d8019040 comp rx=0 tx=0).stop 2023-03-09T14:45:00.250 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:00.000+0000 7fd9f0384700 1 -- 172.21.15.38:0/2855486721 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd9e80729b0 msgr2=0x7fd9e81983c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:00.250 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:00.000+0000 7fd9f0384700 1 --2- 172.21.15.38:0/2855486721 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd9e80729b0 0x7fd9e81983c0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fd9e400e9c0 tx=0x7fd9e400ed80 comp rx=0 tx=0).stop 2023-03-09T14:45:00.251 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:00.000+0000 7fd9ee921700 1 -- 172.21.15.38:0/2855486721 reap_dead start 2023-03-09T14:45:00.251 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:00.000+0000 7fd9f0384700 1 -- 172.21.15.38:0/2855486721 shutdown_connections 2023-03-09T14:45:00.251 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:00.000+0000 7fd9f0384700 1 -- 172.21.15.38:0/2855486721 >> 172.21.15.38:0/2855486721 conn(0x7fd9e8077010 msgr2=0x7fd9e80fead0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:00.251 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:00.000+0000 7fd9f0384700 1 -- 172.21.15.38:0/2855486721 shutdown_connections 2023-03-09T14:45:00.251 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:00.000+0000 7fd9f0384700 1 -- 172.21.15.38:0/2855486721 wait complete. 2023-03-09T14:45:01.584 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.392+0000 7f13571ac700 1 -- 172.21.15.38:0/1376713232 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1350104160 msgr2=0x7f135010bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:01.585 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.392+0000 7f13571ac700 1 --2- 172.21.15.38:0/1376713232 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1350104160 0x7f135010bd30 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f134800b0d0 tx=0x7f134800b3e0 comp rx=0 tx=0).stop 2023-03-09T14:45:01.585 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.392+0000 7f13571ac700 1 -- 172.21.15.38:0/1376713232 shutdown_connections 2023-03-09T14:45:01.585 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.392+0000 7f13571ac700 1 --2- 172.21.15.38:0/1376713232 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1350104160 0x7f135010bd30 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:01.585 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.392+0000 7f13571ac700 1 --2- 172.21.15.38:0/1376713232 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f13500ff120 0x7f13500ff4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:01.585 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.392+0000 7f13571ac700 1 --2- 172.21.15.38:0/1376713232 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f13500ffac0 0x7f1350103a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:01.585 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.392+0000 7f13571ac700 1 -- 172.21.15.38:0/1376713232 >> 172.21.15.38:0/1376713232 conn(0x7f1350076d10 msgr2=0x7f1350077110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:01.586 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.392+0000 7f13571ac700 1 -- 172.21.15.38:0/1376713232 shutdown_connections 2023-03-09T14:45:01.586 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.392+0000 7f13571ac700 1 -- 172.21.15.38:0/1376713232 wait complete. 2023-03-09T14:45:01.586 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.392+0000 7f13571ac700 1 Processor -- start 2023-03-09T14:45:01.586 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.392+0000 7f13571ac700 1 -- start start 2023-03-09T14:45:01.586 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.392+0000 7f13571ac700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f13500ff120 0x7f13501984c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:01.586 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.393+0000 7f13571ac700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f13500ffac0 0x7f1350198a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:01.586 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.393+0000 7f13571ac700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1350104160 0x7f135019ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:01.586 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.393+0000 7f13571ac700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f135019d4a0 con 0x7f1350104160 2023-03-09T14:45:01.587 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.393+0000 7f13571ac700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f135019d610 con 0x7f13500ff120 2023-03-09T14:45:01.587 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.393+0000 7f13571ac700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f135019d7d0 con 0x7f13500ffac0 2023-03-09T14:45:01.587 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.393+0000 7f1355749700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1350104160 0x7f135019ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:01.587 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.393+0000 7f1355749700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1350104160 0x7f135019ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37384/0 (socket says 172.21.15.38:37384) 2023-03-09T14:45:01.587 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.393+0000 7f1355749700 1 -- 172.21.15.38:0/982471447 learned_addr learned my addr 172.21.15.38:0/982471447 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:01.587 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.393+0000 7f134ffff700 1 --2- 172.21.15.38:0/982471447 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f13500ffac0 0x7f1350198a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:01.587 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.393+0000 7f1354f48700 1 --2- 172.21.15.38:0/982471447 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f13500ff120 0x7f13501984c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:01.588 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.393+0000 7f1355749700 1 -- 172.21.15.38:0/982471447 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f13500ffac0 msgr2=0x7f1350198a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:01.588 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.393+0000 7f1355749700 1 --2- 172.21.15.38:0/982471447 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f13500ffac0 0x7f1350198a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:01.588 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.393+0000 7f1355749700 1 -- 172.21.15.38:0/982471447 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f13500ff120 msgr2=0x7f13501984c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:01.588 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.393+0000 7f1355749700 1 --2- 172.21.15.38:0/982471447 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f13500ff120 0x7f13501984c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:01.588 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.393+0000 7f1355749700 1 -- 172.21.15.38:0/982471447 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1348009d00 con 0x7f1350104160 2023-03-09T14:45:01.588 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.393+0000 7f1355749700 1 --2- 172.21.15.38:0/982471447 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1350104160 0x7f135019ce60 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f1348007870 tx=0x7f13480078a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:01.588 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.393+0000 7f134dffb700 1 -- 172.21.15.38:0/982471447 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f134800e040 con 0x7f1350104160 2023-03-09T14:45:01.588 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.393+0000 7f13571ac700 1 -- 172.21.15.38:0/982471447 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f135010ef40 con 0x7f1350104160 2023-03-09T14:45:01.589 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.393+0000 7f134dffb700 1 -- 172.21.15.38:0/982471447 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1348003e10 con 0x7f1350104160 2023-03-09T14:45:01.589 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.393+0000 7f134dffb700 1 -- 172.21.15.38:0/982471447 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f134801b980 con 0x7f1350104160 2023-03-09T14:45:01.589 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.394+0000 7f13571ac700 1 -- 172.21.15.38:0/982471447 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f135010edb0 con 0x7f1350104160 2023-03-09T14:45:01.590 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.394+0000 7f13571ac700 1 -- 172.21.15.38:0/982471447 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f135010b960 con 0x7f1350104160 2023-03-09T14:45:01.590 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.396+0000 7f134dffb700 1 -- 172.21.15.38:0/982471447 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 133238+0+0 (secure 0 0 0) 0x7f13480055c0 con 0x7f1350104160 2023-03-09T14:45:01.590 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.396+0000 7f134dffb700 1 --2- 172.21.15.38:0/982471447 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f134009f400 0x7f13400a18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:01.590 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.396+0000 7f1354f48700 1 --2- 172.21.15.38:0/982471447 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f134009f400 0x7f13400a18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:01.590 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.396+0000 7f134dffb700 1 -- 172.21.15.38:0/982471447 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f13480cbb60 con 0x7f1350104160 2023-03-09T14:45:01.590 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.397+0000 7f134dffb700 1 --2- 172.21.15.38:0/982471447 >> [v2:172.21.15.181:6800/1310630803,v1:172.21.15.181:6801/1310630803] conn(0x7f13400a9ef0 0x7f13400ac320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:01.590 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.397+0000 7f134dffb700 1 -- 172.21.15.38:0/982471447 --> [v2:172.21.15.181:6800/1310630803,v1:172.21.15.181:6801/1310630803] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f13400ac9f0 con 0x7f13400a9ef0 2023-03-09T14:45:01.591 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.397+0000 7f134dffb700 1 -- 172.21.15.38:0/982471447 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) v2 ==== 24+0+0 (secure 0 0 0) 0x7f13480cbf30 con 0x7f1350104160 2023-03-09T14:45:01.591 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.397+0000 7f1354f48700 1 --2- 172.21.15.38:0/982471447 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f134009f400 0x7f13400a18b0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f1344005950 tx=0x7f13440058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:01.591 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.397+0000 7f134ffff700 1 --2- 172.21.15.38:0/982471447 >> [v2:172.21.15.181:6800/1310630803,v1:172.21.15.181:6801/1310630803] conn(0x7f13400a9ef0 0x7f13400ac320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:01.591 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.397+0000 7f134ffff700 1 --2- 172.21.15.38:0/982471447 >> [v2:172.21.15.181:6800/1310630803,v1:172.21.15.181:6801/1310630803] conn(0x7f13400a9ef0 0x7f13400ac320 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:01.591 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.398+0000 7f134dffb700 1 -- 172.21.15.38:0/982471447 <== osd.2 v2:172.21.15.181:6800/1310630803 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f13400ac9f0 con 0x7f13400a9ef0 2023-03-09T14:45:01.591 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.414+0000 7f13571ac700 1 -- 172.21.15.38:0/982471447 --> [v2:172.21.15.181:6800/1310630803,v1:172.21.15.181:6801/1310630803] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f135010f0d0 con 0x7f13400a9ef0 2023-03-09T14:45:01.591 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.415+0000 7f134dffb700 1 -- 172.21.15.38:0/982471447 <== osd.2 v2:172.21.15.181:6800/1310630803 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f135010f0d0 con 0x7f13400a9ef0 2023-03-09T14:45:01.592 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.415+0000 7f13571ac700 1 -- 172.21.15.38:0/982471447 >> [v2:172.21.15.181:6800/1310630803,v1:172.21.15.181:6801/1310630803] conn(0x7f13400a9ef0 msgr2=0x7f13400ac320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:01.592 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.415+0000 7f13571ac700 1 --2- 172.21.15.38:0/982471447 >> [v2:172.21.15.181:6800/1310630803,v1:172.21.15.181:6801/1310630803] conn(0x7f13400a9ef0 0x7f13400ac320 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:01.592 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.415+0000 7f13571ac700 1 -- 172.21.15.38:0/982471447 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f134009f400 msgr2=0x7f13400a18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:01.592 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.415+0000 7f13571ac700 1 --2- 172.21.15.38:0/982471447 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f134009f400 0x7f13400a18b0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f1344005950 tx=0x7f13440058e0 comp rx=0 tx=0).stop 2023-03-09T14:45:01.592 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.416+0000 7f13571ac700 1 -- 172.21.15.38:0/982471447 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1350104160 msgr2=0x7f135019ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:01.592 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.416+0000 7f13571ac700 1 --2- 172.21.15.38:0/982471447 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1350104160 0x7f135019ce60 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f1348007870 tx=0x7f13480078a0 comp rx=0 tx=0).stop 2023-03-09T14:45:01.592 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.416+0000 7f1355749700 1 -- 172.21.15.38:0/982471447 reap_dead start 2023-03-09T14:45:01.592 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.416+0000 7f13571ac700 1 -- 172.21.15.38:0/982471447 shutdown_connections 2023-03-09T14:45:01.593 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.416+0000 7f13571ac700 1 -- 172.21.15.38:0/982471447 >> 172.21.15.38:0/982471447 conn(0x7f1350076d10 msgr2=0x7f1350069380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:01.593 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.416+0000 7f13571ac700 1 -- 172.21.15.38:0/982471447 shutdown_connections 2023-03-09T14:45:01.593 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.416+0000 7f13571ac700 1 -- 172.21.15.38:0/982471447 wait complete. 2023-03-09T14:45:01.774 INFO:teuthology.orchestra.run.smithi038.stdout:55834574909 2023-03-09T14:45:01.774 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph osd last-stat-seq osd.1 2023-03-09T14:45:02.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:01 smithi038 ceph-mon[92422]: pgmap v280: 129 pgs: 129 active+clean; 636 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2023-03-09T14:45:02.175 INFO:teuthology.orchestra.run.smithi038.stdout:193273528362 2023-03-09T14:45:02.176 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph osd last-stat-seq osd.6 2023-03-09T14:45:02.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:01 smithi122 ceph-mon[93869]: pgmap v280: 129 pgs: 129 active+clean; 636 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2023-03-09T14:45:02.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:01 smithi181 ceph-mon[94189]: pgmap v280: 129 pgs: 129 active+clean; 636 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2023-03-09T14:45:02.334 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.720+0000 7f8dd063b700 1 -- 172.21.15.38:0/383126930 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8dc80ff120 msgr2=0x7f8dc80ff4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:02.334 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.720+0000 7f8dd063b700 1 --2- 172.21.15.38:0/383126930 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8dc80ff120 0x7f8dc80ff4f0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f8dc40099c0 tx=0x7f8dc4009cd0 comp rx=0 tx=0).stop 2023-03-09T14:45:02.334 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.720+0000 7f8dd063b700 1 -- 172.21.15.38:0/383126930 shutdown_connections 2023-03-09T14:45:02.334 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.720+0000 7f8dd063b700 1 --2- 172.21.15.38:0/383126930 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8dc80ff120 0x7f8dc80ff4f0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:02.334 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.720+0000 7f8dd063b700 1 --2- 172.21.15.38:0/383126930 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8dc8104160 0x7f8dc810bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:02.335 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.720+0000 7f8dd063b700 1 --2- 172.21.15.38:0/383126930 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8dc80ffac0 0x7f8dc8103a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:02.335 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.720+0000 7f8dd063b700 1 -- 172.21.15.38:0/383126930 >> 172.21.15.38:0/383126930 conn(0x7f8dc8076d10 msgr2=0x7f8dc8077110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:02.335 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.720+0000 7f8dd063b700 1 -- 172.21.15.38:0/383126930 shutdown_connections 2023-03-09T14:45:02.335 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.720+0000 7f8dd063b700 1 -- 172.21.15.38:0/383126930 wait complete. 2023-03-09T14:45:02.335 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.721+0000 7f8dd063b700 1 Processor -- start 2023-03-09T14:45:02.335 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.721+0000 7f8dd063b700 1 -- start start 2023-03-09T14:45:02.335 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.721+0000 7f8dd063b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8dc80ff120 0x7f8dc81960c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:02.335 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.721+0000 7f8dd063b700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8dc80ffac0 0x7f8dc8196600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:02.336 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.721+0000 7f8dd063b700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8dc8104160 0x7f8dc819aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:02.336 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.721+0000 7f8dd063b700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc819b0a0 con 0x7f8dc80ff120 2023-03-09T14:45:02.336 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.721+0000 7f8dd063b700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc819b210 con 0x7f8dc80ffac0 2023-03-09T14:45:02.336 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.721+0000 7f8dd063b700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc819b3d0 con 0x7f8dc8104160 2023-03-09T14:45:02.336 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.721+0000 7f8dce3d7700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8dc80ff120 0x7f8dc81960c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:02.336 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.721+0000 7f8dce3d7700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8dc80ff120 0x7f8dc81960c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37394/0 (socket says 172.21.15.38:37394) 2023-03-09T14:45:02.336 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.721+0000 7f8dce3d7700 1 -- 172.21.15.38:0/1730376683 learned_addr learned my addr 172.21.15.38:0/1730376683 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:02.337 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.721+0000 7f8dcdbd6700 1 --2- 172.21.15.38:0/1730376683 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8dc80ffac0 0x7f8dc8196600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:02.337 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.721+0000 7f8dcebd8700 1 --2- 172.21.15.38:0/1730376683 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8dc8104160 0x7f8dc819aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:02.337 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.721+0000 7f8dce3d7700 1 -- 172.21.15.38:0/1730376683 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8dc8104160 msgr2=0x7f8dc819aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:02.337 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.721+0000 7f8dce3d7700 1 --2- 172.21.15.38:0/1730376683 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8dc8104160 0x7f8dc819aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:02.337 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.721+0000 7f8dce3d7700 1 -- 172.21.15.38:0/1730376683 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8dc80ffac0 msgr2=0x7f8dc8196600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:02.337 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.721+0000 7f8dce3d7700 1 --2- 172.21.15.38:0/1730376683 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8dc80ffac0 0x7f8dc8196600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:02.337 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.721+0000 7f8dce3d7700 1 -- 172.21.15.38:0/1730376683 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dc40095f0 con 0x7f8dc80ff120 2023-03-09T14:45:02.338 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.721+0000 7f8dce3d7700 1 --2- 172.21.15.38:0/1730376683 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8dc80ff120 0x7f8dc81960c0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f8dc400b580 tx=0x7f8dc400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:02.338 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.722+0000 7f8dbf7fe700 1 -- 172.21.15.38:0/1730376683 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc401d070 con 0x7f8dc80ff120 2023-03-09T14:45:02.338 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.722+0000 7f8dbf7fe700 1 -- 172.21.15.38:0/1730376683 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8dc40054f0 con 0x7f8dc80ff120 2023-03-09T14:45:02.338 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.722+0000 7f8dd063b700 1 -- 172.21.15.38:0/1730376683 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dc810ef30 con 0x7f8dc80ff120 2023-03-09T14:45:02.338 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.722+0000 7f8dbf7fe700 1 -- 172.21.15.38:0/1730376683 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dc4004b30 con 0x7f8dc80ff120 2023-03-09T14:45:02.338 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.722+0000 7f8dd063b700 1 -- 172.21.15.38:0/1730376683 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dc810eda0 con 0x7f8dc80ff120 2023-03-09T14:45:02.338 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.722+0000 7f8dd063b700 1 -- 172.21.15.38:0/1730376683 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8dc8068560 con 0x7f8dc80ff120 2023-03-09T14:45:02.338 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.724+0000 7f8dbf7fe700 1 -- 172.21.15.38:0/1730376683 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 133238+0+0 (secure 0 0 0) 0x7f8dc4028030 con 0x7f8dc80ff120 2023-03-09T14:45:02.339 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.725+0000 7f8dbf7fe700 1 --2- 172.21.15.38:0/1730376683 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f8db409f480 0x7f8db40a1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:02.339 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.725+0000 7f8dcdbd6700 1 --2- 172.21.15.38:0/1730376683 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f8db409f480 0x7f8db40a1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:02.339 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.725+0000 7f8dbf7fe700 1 -- 172.21.15.38:0/1730376683 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f8dc40cb4e0 con 0x7f8dc80ff120 2023-03-09T14:45:02.339 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.726+0000 7f8dbf7fe700 1 --2- 172.21.15.38:0/1730376683 >> [v2:172.21.15.38:6826/2624555969,v1:172.21.15.38:6827/2624555969] conn(0x7f8db40a9f70 0x7f8db40ac3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:02.339 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.726+0000 7f8dbf7fe700 1 -- 172.21.15.38:0/1730376683 --> [v2:172.21.15.38:6826/2624555969,v1:172.21.15.38:6827/2624555969] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8db40aca70 con 0x7f8db40a9f70 2023-03-09T14:45:02.339 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.726+0000 7f8dbf7fe700 1 -- 172.21.15.38:0/1730376683 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8dc40cb890 con 0x7f8dc80ff120 2023-03-09T14:45:02.339 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.726+0000 7f8dcebd8700 1 --2- 172.21.15.38:0/1730376683 >> [v2:172.21.15.38:6826/2624555969,v1:172.21.15.38:6827/2624555969] conn(0x7f8db40a9f70 0x7f8db40ac3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:02.340 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.726+0000 7f8dcdbd6700 1 --2- 172.21.15.38:0/1730376683 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f8db409f480 0x7f8db40a1930 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f8dc81976d0 tx=0x7f8db800a380 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:02.340 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.726+0000 7f8dcebd8700 1 --2- 172.21.15.38:0/1730376683 >> [v2:172.21.15.38:6826/2624555969,v1:172.21.15.38:6827/2624555969] conn(0x7f8db40a9f70 0x7f8db40ac3a0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:02.340 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.727+0000 7f8dbf7fe700 1 -- 172.21.15.38:0/1730376683 <== osd.9 v2:172.21.15.38:6826/2624555969 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f8db40aca70 con 0x7f8db40a9f70 2023-03-09T14:45:02.340 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.742+0000 7f8dd063b700 1 -- 172.21.15.38:0/1730376683 --> [v2:172.21.15.38:6826/2624555969,v1:172.21.15.38:6827/2624555969] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8dc810f0c0 con 0x7f8db40a9f70 2023-03-09T14:45:02.341 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.743+0000 7f8dbf7fe700 1 -- 172.21.15.38:0/1730376683 <== osd.9 v2:172.21.15.38:6826/2624555969 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f8dc810f0c0 con 0x7f8db40a9f70 2023-03-09T14:45:02.341 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.743+0000 7f8dd063b700 1 -- 172.21.15.38:0/1730376683 >> [v2:172.21.15.38:6826/2624555969,v1:172.21.15.38:6827/2624555969] conn(0x7f8db40a9f70 msgr2=0x7f8db40ac3a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:02.341 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.743+0000 7f8dd063b700 1 --2- 172.21.15.38:0/1730376683 >> [v2:172.21.15.38:6826/2624555969,v1:172.21.15.38:6827/2624555969] conn(0x7f8db40a9f70 0x7f8db40ac3a0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:02.341 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.743+0000 7f8dd063b700 1 -- 172.21.15.38:0/1730376683 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f8db409f480 msgr2=0x7f8db40a1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:02.341 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.743+0000 7f8dd063b700 1 --2- 172.21.15.38:0/1730376683 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f8db409f480 0x7f8db40a1930 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f8dc81976d0 tx=0x7f8db800a380 comp rx=0 tx=0).stop 2023-03-09T14:45:02.341 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.743+0000 7f8dd063b700 1 -- 172.21.15.38:0/1730376683 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8dc80ff120 msgr2=0x7f8dc81960c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:02.341 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.743+0000 7f8dd063b700 1 --2- 172.21.15.38:0/1730376683 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8dc80ff120 0x7f8dc81960c0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f8dc400b580 tx=0x7f8dc400f690 comp rx=0 tx=0).stop 2023-03-09T14:45:02.342 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.743+0000 7f8dcebd8700 1 -- 172.21.15.38:0/1730376683 reap_dead start 2023-03-09T14:45:02.342 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.743+0000 7f8dd063b700 1 -- 172.21.15.38:0/1730376683 shutdown_connections 2023-03-09T14:45:02.342 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.743+0000 7f8dd063b700 1 -- 172.21.15.38:0/1730376683 >> 172.21.15.38:0/1730376683 conn(0x7f8dc8076d10 msgr2=0x7f8dc8102c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:02.342 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.743+0000 7f8dd063b700 1 -- 172.21.15.38:0/1730376683 shutdown_connections 2023-03-09T14:45:02.342 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:01.743+0000 7f8dd063b700 1 -- 172.21.15.38:0/1730376683 wait complete. 2023-03-09T14:45:02.415 INFO:teuthology.orchestra.run.smithi038.stdout:158913789998 2023-03-09T14:45:02.415 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph osd last-stat-seq osd.5 2023-03-09T14:45:02.661 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:02.658+0000 7fd5d9152700 1 -- 172.21.15.38:0/2059782688 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5d4101b10 msgr2=0x7fd5d410a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:02.662 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:02.658+0000 7fd5d9152700 1 --2- 172.21.15.38:0/2059782688 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5d4101b10 0x7fd5d410a000 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fd5c8009990 tx=0x7fd5c8009ca0 comp rx=0 tx=0).stop 2023-03-09T14:45:02.662 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:02.658+0000 7fd5d9152700 1 -- 172.21.15.38:0/2059782688 shutdown_connections 2023-03-09T14:45:02.662 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:02.658+0000 7fd5d9152700 1 --2- 172.21.15.38:0/2059782688 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd5d410a540 0x7fd5d410c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:02.662 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:02.658+0000 7fd5d9152700 1 --2- 172.21.15.38:0/2059782688 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd5d4101200 0x7fd5d41015d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:02.662 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:02.658+0000 7fd5d9152700 1 --2- 172.21.15.38:0/2059782688 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5d4101b10 0x7fd5d410a000 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:02.662 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:02.658+0000 7fd5d9152700 1 -- 172.21.15.38:0/2059782688 >> 172.21.15.38:0/2059782688 conn(0x7fd5d4076d10 msgr2=0x7fd5d4077110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:02.662 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:02.658+0000 7fd5d9152700 1 -- 172.21.15.38:0/2059782688 shutdown_connections 2023-03-09T14:45:02.663 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:02.658+0000 7fd5d9152700 1 -- 172.21.15.38:0/2059782688 wait complete. 2023-03-09T14:45:02.663 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:02.659+0000 7fd5d9152700 1 Processor -- start 2023-03-09T14:45:02.663 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:02.659+0000 7fd5d9152700 1 -- start start 2023-03-09T14:45:02.663 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:02.659+0000 7fd5d9152700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5d4101200 0x7fd5d41960e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:02.663 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:02.659+0000 7fd5d9152700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd5d4101b10 0x7fd5d4196620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:02.663 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:02.659+0000 7fd5d9152700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd5d410a540 0x7fd5d419aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:02.663 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:02.659+0000 7fd5d9152700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5d419b0c0 con 0x7fd5d4101200 2023-03-09T14:45:02.663 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:02.659+0000 7fd5d9152700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5d419b230 con 0x7fd5d4101b10 2023-03-09T14:45:02.664 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:02.659+0000 7fd5d9152700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5d419b3f0 con 0x7fd5d410a540 2023-03-09T14:45:02.665 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:02.659+0000 7fd5d2d9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5d4101200 0x7fd5d41960e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:02.666 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:02.659+0000 7fd5d2d9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5d4101200 0x7fd5d41960e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37408/0 (socket says 172.21.15.38:37408) 2023-03-09T14:45:02.666 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:02.659+0000 7fd5d2d9d700 1 -- 172.21.15.38:0/229310367 learned_addr learned my addr 172.21.15.38:0/229310367 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:02.666 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:02.660+0000 7fd5d359e700 1 --2- 172.21.15.38:0/229310367 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd5d410a540 0x7fd5d419aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:02.666 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:02.660+0000 7fd5d259c700 1 --2- 172.21.15.38:0/229310367 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd5d4101b10 0x7fd5d4196620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:02.666 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:02.660+0000 7fd5d2d9d700 1 -- 172.21.15.38:0/229310367 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd5d410a540 msgr2=0x7fd5d419aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:02.666 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:02.660+0000 7fd5d2d9d700 1 --2- 172.21.15.38:0/229310367 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd5d410a540 0x7fd5d419aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:02.667 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:02.660+0000 7fd5d2d9d700 1 -- 172.21.15.38:0/229310367 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd5d4101b10 msgr2=0x7fd5d4196620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:02.667 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:02.660+0000 7fd5d2d9d700 1 --2- 172.21.15.38:0/229310367 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd5d4101b10 0x7fd5d4196620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:02.667 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:02.660+0000 7fd5d2d9d700 1 -- 172.21.15.38:0/229310367 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5c80095f0 con 0x7fd5d4101200 2023-03-09T14:45:02.667 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:02.660+0000 7fd5d2d9d700 1 --2- 172.21.15.38:0/229310367 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5d4101200 0x7fd5d41960e0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fd5c400c920 tx=0x7fd5c400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:02.667 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:02.660+0000 7fd5bbfff700 1 -- 172.21.15.38:0/229310367 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5c4007b00 con 0x7fd5d4101200 2023-03-09T14:45:02.667 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:02.660+0000 7fd5bbfff700 1 -- 172.21.15.38:0/229310367 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd5c4004cf0 con 0x7fd5d4101200 2023-03-09T14:45:02.667 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:02.660+0000 7fd5d9152700 1 -- 172.21.15.38:0/229310367 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5d410ffe0 con 0x7fd5d4101200 2023-03-09T14:45:02.667 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:02.660+0000 7fd5bbfff700 1 -- 172.21.15.38:0/229310367 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5c4005610 con 0x7fd5d4101200 2023-03-09T14:45:02.668 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:02.660+0000 7fd5d9152700 1 -- 172.21.15.38:0/229310367 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5d410fe50 con 0x7fd5d4101200 2023-03-09T14:45:02.668 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:02.661+0000 7fd5d9152700 1 -- 172.21.15.38:0/229310367 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd5d40fd430 con 0x7fd5d4101200 2023-03-09T14:45:02.668 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:02.662+0000 7fd5bbfff700 1 -- 172.21.15.38:0/229310367 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 133238+0+0 (secure 0 0 0) 0x7fd5c4007c60 con 0x7fd5d4101200 2023-03-09T14:45:02.668 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:02.663+0000 7fd5bbfff700 1 --2- 172.21.15.38:0/229310367 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fd5bc09f400 0x7fd5bc0a18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:02.668 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:02.663+0000 7fd5d259c700 1 --2- 172.21.15.38:0/229310367 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fd5bc09f400 0x7fd5bc0a18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:02.668 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:02.663+0000 7fd5bbfff700 1 -- 172.21.15.38:0/229310367 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fd5c40ca350 con 0x7fd5d4101200 2023-03-09T14:45:02.668 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:02.663+0000 7fd5bbfff700 1 --2- 172.21.15.38:0/229310367 >> [v2:172.21.15.38:6802/1508072290,v1:172.21.15.38:6803/1508072290] conn(0x7fd5bc0a9ef0 0x7fd5bc0ac320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:02.669 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:02.663+0000 7fd5bbfff700 1 -- 172.21.15.38:0/229310367 --> [v2:172.21.15.38:6802/1508072290,v1:172.21.15.38:6803/1508072290] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd5bc0ac9f0 con 0x7fd5bc0a9ef0 2023-03-09T14:45:02.669 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:02.663+0000 7fd5bbfff700 1 -- 172.21.15.38:0/229310367 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd5c40ca6c0 con 0x7fd5d4101200 2023-03-09T14:45:02.669 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:02.663+0000 7fd5d359e700 1 --2- 172.21.15.38:0/229310367 >> [v2:172.21.15.38:6802/1508072290,v1:172.21.15.38:6803/1508072290] conn(0x7fd5bc0a9ef0 0x7fd5bc0ac320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:02.669 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:02.664+0000 7fd5d259c700 1 --2- 172.21.15.38:0/229310367 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fd5bc09f400 0x7fd5bc0a18b0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fd5c8006010 tx=0x7fd5c800b540 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:02.670 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:02.664+0000 7fd5d359e700 1 --2- 172.21.15.38:0/229310367 >> [v2:172.21.15.38:6802/1508072290,v1:172.21.15.38:6803/1508072290] conn(0x7fd5bc0a9ef0 0x7fd5bc0ac320 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:02.670 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:02.665+0000 7fd5bbfff700 1 -- 172.21.15.38:0/229310367 <== osd.0 v2:172.21.15.38:6802/1508072290 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fd5bc0ac9f0 con 0x7fd5bc0a9ef0 2023-03-09T14:45:02.683 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:02.681+0000 7fd5d9152700 1 -- 172.21.15.38:0/229310367 --> [v2:172.21.15.38:6802/1508072290,v1:172.21.15.38:6803/1508072290] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd5d4110170 con 0x7fd5bc0a9ef0 2023-03-09T14:45:02.684 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:02.682+0000 7fd5bbfff700 1 -- 172.21.15.38:0/229310367 <== osd.0 v2:172.21.15.38:6802/1508072290 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fd5d4110170 con 0x7fd5bc0a9ef0 2023-03-09T14:45:02.685 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:02.682+0000 7fd5d9152700 1 -- 172.21.15.38:0/229310367 >> [v2:172.21.15.38:6802/1508072290,v1:172.21.15.38:6803/1508072290] conn(0x7fd5bc0a9ef0 msgr2=0x7fd5bc0ac320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:02.685 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:02.682+0000 7fd5d9152700 1 --2- 172.21.15.38:0/229310367 >> [v2:172.21.15.38:6802/1508072290,v1:172.21.15.38:6803/1508072290] conn(0x7fd5bc0a9ef0 0x7fd5bc0ac320 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:02.685 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:02.682+0000 7fd5d9152700 1 -- 172.21.15.38:0/229310367 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fd5bc09f400 msgr2=0x7fd5bc0a18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:02.686 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:02.682+0000 7fd5d9152700 1 --2- 172.21.15.38:0/229310367 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fd5bc09f400 0x7fd5bc0a18b0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fd5c8006010 tx=0x7fd5c800b540 comp rx=0 tx=0).stop 2023-03-09T14:45:02.686 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:02.682+0000 7fd5d9152700 1 -- 172.21.15.38:0/229310367 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5d4101200 msgr2=0x7fd5d41960e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:02.686 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:02.682+0000 7fd5d9152700 1 --2- 172.21.15.38:0/229310367 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd5d4101200 0x7fd5d41960e0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fd5c400c920 tx=0x7fd5c400cce0 comp rx=0 tx=0).stop 2023-03-09T14:45:02.686 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:02.682+0000 7fd5d359e700 1 -- 172.21.15.38:0/229310367 reap_dead start 2023-03-09T14:45:02.686 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:02.682+0000 7fd5d9152700 1 -- 172.21.15.38:0/229310367 shutdown_connections 2023-03-09T14:45:02.686 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:02.682+0000 7fd5d9152700 1 -- 172.21.15.38:0/229310367 >> 172.21.15.38:0/229310367 conn(0x7fd5d4076d10 msgr2=0x7fd5d40fca20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:02.686 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:02.682+0000 7fd5d9152700 1 -- 172.21.15.38:0/229310367 shutdown_connections 2023-03-09T14:45:02.687 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:02.682+0000 7fd5d9152700 1 -- 172.21.15.38:0/229310367 wait complete. 2023-03-09T14:45:03.064 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:03.059+0000 7f6986b4e700 1 -- 172.21.15.38:0/1359265827 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6980100ce0 msgr2=0x7f6980104bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:03.065 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:03.059+0000 7f6986b4e700 1 --2- 172.21.15.38:0/1359265827 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6980100ce0 0x7f6980104bc0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f696c0099c0 tx=0x7f696c009cd0 comp rx=0 tx=0).stop 2023-03-09T14:45:03.065 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:03.059+0000 7f6986b4e700 1 -- 172.21.15.38:0/1359265827 shutdown_connections 2023-03-09T14:45:03.065 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:03.059+0000 7f6986b4e700 1 --2- 172.21.15.38:0/1359265827 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69801052a0 0x7f698010ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:03.065 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:03.059+0000 7f6986b4e700 1 --2- 172.21.15.38:0/1359265827 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6980100340 0x7f6980100710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:03.066 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:03.059+0000 7f6986b4e700 1 --2- 172.21.15.38:0/1359265827 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6980100ce0 0x7f6980104bc0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:03.066 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:03.060+0000 7f6986b4e700 1 -- 172.21.15.38:0/1359265827 >> 172.21.15.38:0/1359265827 conn(0x7f6980077010 msgr2=0x7f6980077410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:03.066 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:03.060+0000 7f6986b4e700 1 -- 172.21.15.38:0/1359265827 shutdown_connections 2023-03-09T14:45:03.066 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:03.060+0000 7f6986b4e700 1 -- 172.21.15.38:0/1359265827 wait complete. 2023-03-09T14:45:03.066 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:03.060+0000 7f6986b4e700 1 Processor -- start 2023-03-09T14:45:03.066 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:03.060+0000 7f6986b4e700 1 -- start start 2023-03-09T14:45:03.066 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:03.060+0000 7f6986b4e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6980100340 0x7f6980196130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:03.067 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:03.060+0000 7f6986b4e700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6980100ce0 0x7f6980196670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:03.067 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:03.060+0000 7f6986b4e700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69801052a0 0x7f698019aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:03.067 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:03.060+0000 7f6986b4e700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f698019b110 con 0x7f6980100340 2023-03-09T14:45:03.067 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:03.060+0000 7f6986b4e700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f698019b280 con 0x7f6980100ce0 2023-03-09T14:45:03.067 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:03.060+0000 7f6986b4e700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f698019b440 con 0x7f69801052a0 2023-03-09T14:45:03.067 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:03.061+0000 7f69848ea700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6980100340 0x7f6980196130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:03.067 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:03.061+0000 7f69848ea700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6980100340 0x7f6980196130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37424/0 (socket says 172.21.15.38:37424) 2023-03-09T14:45:03.068 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:03.061+0000 7f69848ea700 1 -- 172.21.15.38:0/2926921680 learned_addr learned my addr 172.21.15.38:0/2926921680 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:03.068 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:03.061+0000 7f69850eb700 1 --2- 172.21.15.38:0/2926921680 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69801052a0 0x7f698019aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:03.068 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:03.061+0000 7f697ffff700 1 --2- 172.21.15.38:0/2926921680 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6980100ce0 0x7f6980196670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:03.068 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:03.061+0000 7f69848ea700 1 -- 172.21.15.38:0/2926921680 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69801052a0 msgr2=0x7f698019aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:03.068 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:03.061+0000 7f69848ea700 1 --2- 172.21.15.38:0/2926921680 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69801052a0 0x7f698019aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:03.068 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:03.061+0000 7f69848ea700 1 -- 172.21.15.38:0/2926921680 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6980100ce0 msgr2=0x7f6980196670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:03.068 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:03.061+0000 7f69848ea700 1 --2- 172.21.15.38:0/2926921680 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6980100ce0 0x7f6980196670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:03.068 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:03.061+0000 7f69848ea700 1 -- 172.21.15.38:0/2926921680 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f696c0095f0 con 0x7f6980100340 2023-03-09T14:45:03.069 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:03.061+0000 7f697ffff700 1 --2- 172.21.15.38:0/2926921680 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6980100ce0 0x7f6980196670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:45:03.069 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:03.061+0000 7f69848ea700 1 --2- 172.21.15.38:0/2926921680 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6980100340 0x7f6980196130 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f697400c920 tx=0x7f697400cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:03.069 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:03.061+0000 7f697dffb700 1 -- 172.21.15.38:0/2926921680 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6974007b00 con 0x7f6980100340 2023-03-09T14:45:03.069 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:03.061+0000 7f697dffb700 1 -- 172.21.15.38:0/2926921680 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f697400ce70 con 0x7f6980100340 2023-03-09T14:45:03.069 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:03.061+0000 7f697dffb700 1 -- 172.21.15.38:0/2926921680 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69740054e0 con 0x7f6980100340 2023-03-09T14:45:03.070 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:03.061+0000 7f6986b4e700 1 -- 172.21.15.38:0/2926921680 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6980110020 con 0x7f6980100340 2023-03-09T14:45:03.070 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:03.062+0000 7f6986b4e700 1 -- 172.21.15.38:0/2926921680 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f698010fe90 con 0x7f6980100340 2023-03-09T14:45:03.071 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:03.062+0000 7f6986b4e700 1 -- 172.21.15.38:0/2926921680 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f69800fe540 con 0x7f6980100340 2023-03-09T14:45:03.071 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:03.063+0000 7f697dffb700 1 -- 172.21.15.38:0/2926921680 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 133238+0+0 (secure 0 0 0) 0x7f697401f070 con 0x7f6980100340 2023-03-09T14:45:03.071 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:03.064+0000 7f697dffb700 1 --2- 172.21.15.38:0/2926921680 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f697009f480 0x7f69700a1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:03.071 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:03.064+0000 7f697ffff700 1 --2- 172.21.15.38:0/2926921680 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f697009f480 0x7f69700a1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:03.071 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:03.064+0000 7f697dffb700 1 -- 172.21.15.38:0/2926921680 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f69740c9e40 con 0x7f6980100340 2023-03-09T14:45:03.071 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:03.064+0000 7f697dffb700 1 --2- 172.21.15.38:0/2926921680 >> [v2:172.21.15.181:6824/3183488640,v1:172.21.15.181:6825/3183488640] conn(0x7f69700a9f70 0x7f69700ac3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:03.072 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:03.064+0000 7f697dffb700 1 -- 172.21.15.38:0/2926921680 --> [v2:172.21.15.181:6824/3183488640,v1:172.21.15.181:6825/3183488640] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f69700aca70 con 0x7f69700a9f70 2023-03-09T14:45:03.072 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:03.064+0000 7f697dffb700 1 -- 172.21.15.38:0/2926921680 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) v2 ==== 24+0+0 (secure 0 0 0) 0x7f69740ca210 con 0x7f6980100340 2023-03-09T14:45:03.072 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:03.065+0000 7f697ffff700 1 --2- 172.21.15.38:0/2926921680 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f697009f480 0x7f69700a1930 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f696c0098e0 tx=0x7f696c01a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:03.072 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:03.065+0000 7f69850eb700 1 --2- 172.21.15.38:0/2926921680 >> [v2:172.21.15.181:6824/3183488640,v1:172.21.15.181:6825/3183488640] conn(0x7f69700a9f70 0x7f69700ac3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:03.072 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:03.065+0000 7f69850eb700 1 --2- 172.21.15.38:0/2926921680 >> [v2:172.21.15.181:6824/3183488640,v1:172.21.15.181:6825/3183488640] conn(0x7f69700a9f70 0x7f69700ac3a0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:03.072 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:03.066+0000 7f697dffb700 1 -- 172.21.15.38:0/2926921680 <== osd.11 v2:172.21.15.181:6824/3183488640 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f69700aca70 con 0x7f69700a9f70 2023-03-09T14:45:03.084 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:03.082+0000 7f6986b4e700 1 -- 172.21.15.38:0/2926921680 --> [v2:172.21.15.181:6824/3183488640,v1:172.21.15.181:6825/3183488640] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f69801101b0 con 0x7f69700a9f70 2023-03-09T14:45:03.084 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:03.082+0000 7f697dffb700 1 -- 172.21.15.38:0/2926921680 <== osd.11 v2:172.21.15.181:6824/3183488640 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f69801101b0 con 0x7f69700a9f70 2023-03-09T14:45:03.086 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:03.083+0000 7f6986b4e700 1 -- 172.21.15.38:0/2926921680 >> [v2:172.21.15.181:6824/3183488640,v1:172.21.15.181:6825/3183488640] conn(0x7f69700a9f70 msgr2=0x7f69700ac3a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:03.086 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:03.083+0000 7f6986b4e700 1 --2- 172.21.15.38:0/2926921680 >> [v2:172.21.15.181:6824/3183488640,v1:172.21.15.181:6825/3183488640] conn(0x7f69700a9f70 0x7f69700ac3a0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:03.087 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:03.083+0000 7f6986b4e700 1 -- 172.21.15.38:0/2926921680 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f697009f480 msgr2=0x7f69700a1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:03.087 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:03.083+0000 7f6986b4e700 1 --2- 172.21.15.38:0/2926921680 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f697009f480 0x7f69700a1930 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f696c0098e0 tx=0x7f696c01a040 comp rx=0 tx=0).stop 2023-03-09T14:45:03.087 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:03.083+0000 7f6986b4e700 1 -- 172.21.15.38:0/2926921680 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6980100340 msgr2=0x7f6980196130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:03.087 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:03.083+0000 7f6986b4e700 1 --2- 172.21.15.38:0/2926921680 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6980100340 0x7f6980196130 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f697400c920 tx=0x7f697400cc30 comp rx=0 tx=0).stop 2023-03-09T14:45:03.088 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:03.083+0000 7f69850eb700 1 -- 172.21.15.38:0/2926921680 reap_dead start 2023-03-09T14:45:03.088 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:03.083+0000 7f6986b4e700 1 -- 172.21.15.38:0/2926921680 shutdown_connections 2023-03-09T14:45:03.088 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:03.083+0000 7f6986b4e700 1 -- 172.21.15.38:0/2926921680 >> 172.21.15.38:0/2926921680 conn(0x7f6980077010 msgr2=0x7f69800fdaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:03.088 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:03.083+0000 7f6986b4e700 1 -- 172.21.15.38:0/2926921680 shutdown_connections 2023-03-09T14:45:03.089 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:03.083+0000 7f6986b4e700 1 -- 172.21.15.38:0/2926921680 wait complete. 2023-03-09T14:45:03.267 INFO:teuthology.orchestra.run.smithi038.stdout:244813135904 2023-03-09T14:45:03.268 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph osd last-stat-seq osd.8 2023-03-09T14:45:03.690 INFO:teuthology.orchestra.run.smithi038.stdout:77309411387 2023-03-09T14:45:03.690 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph osd last-stat-seq osd.2 2023-03-09T14:45:03.878 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:03 smithi038 ceph-mon[92422]: pgmap v281: 129 pgs: 129 active+clean; 636 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 6 op/s 2023-03-09T14:45:03.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:03 smithi122 ceph-mon[93869]: pgmap v281: 129 pgs: 129 active+clean; 636 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 6 op/s 2023-03-09T14:45:03.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:03 smithi181 ceph-mon[94189]: pgmap v281: 129 pgs: 129 active+clean; 636 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 6 op/s 2023-03-09T14:45:04.559 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.556+0000 7f9af4ba8700 1 -- 172.21.15.38:0/1455110210 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9af010a540 msgr2=0x7f9af010c9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:04.559 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.556+0000 7f9af4ba8700 1 --2- 172.21.15.38:0/1455110210 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9af010a540 0x7f9af010c9a0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f9ae800b0a0 tx=0x7f9ae800b3b0 comp rx=0 tx=0).stop 2023-03-09T14:45:04.559 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.556+0000 7f9af4ba8700 1 -- 172.21.15.38:0/1455110210 shutdown_connections 2023-03-09T14:45:04.559 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.556+0000 7f9af4ba8700 1 --2- 172.21.15.38:0/1455110210 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9af010a540 0x7f9af010c9a0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:04.560 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.556+0000 7f9af4ba8700 1 --2- 172.21.15.38:0/1455110210 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9af0101200 0x7f9af01015d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:04.560 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.556+0000 7f9af4ba8700 1 --2- 172.21.15.38:0/1455110210 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9af0101b10 0x7f9af010a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:04.560 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.556+0000 7f9af4ba8700 1 -- 172.21.15.38:0/1455110210 >> 172.21.15.38:0/1455110210 conn(0x7f9af0076d10 msgr2=0x7f9af0077110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:04.560 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.556+0000 7f9af4ba8700 1 -- 172.21.15.38:0/1455110210 shutdown_connections 2023-03-09T14:45:04.560 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.556+0000 7f9af4ba8700 1 -- 172.21.15.38:0/1455110210 wait complete. 2023-03-09T14:45:04.562 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.557+0000 7f9af4ba8700 1 Processor -- start 2023-03-09T14:45:04.562 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.557+0000 7f9af4ba8700 1 -- start start 2023-03-09T14:45:04.562 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.557+0000 7f9af4ba8700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9af0101200 0x7f9af00fee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:04.562 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.557+0000 7f9af4ba8700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9af0101b10 0x7f9af00ff390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:04.562 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.557+0000 7f9af4ba8700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9af010a540 0x7f9af01a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:04.562 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.557+0000 7f9af4ba8700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9af00ffbc0 con 0x7f9af010a540 2023-03-09T14:45:04.562 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.557+0000 7f9af4ba8700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9af01a48f0 con 0x7f9af0101200 2023-03-09T14:45:04.563 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.557+0000 7f9af4ba8700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9af01a4a30 con 0x7f9af0101b10 2023-03-09T14:45:04.563 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.557+0000 7f9aeed9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9af010a540 0x7f9af01a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:04.563 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.557+0000 7f9aeed9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9af010a540 0x7f9af01a43b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37444/0 (socket says 172.21.15.38:37444) 2023-03-09T14:45:04.563 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.557+0000 7f9aeed9d700 1 -- 172.21.15.38:0/3746658712 learned_addr learned my addr 172.21.15.38:0/3746658712 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:04.564 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.557+0000 7f9aedd9b700 1 --2- 172.21.15.38:0/3746658712 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9af0101b10 0x7f9af00ff390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:04.564 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.557+0000 7f9aee59c700 1 --2- 172.21.15.38:0/3746658712 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9af0101200 0x7f9af00fee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:04.564 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.558+0000 7f9aeed9d700 1 -- 172.21.15.38:0/3746658712 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9af0101b10 msgr2=0x7f9af00ff390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:04.564 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.558+0000 7f9aeed9d700 1 --2- 172.21.15.38:0/3746658712 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9af0101b10 0x7f9af00ff390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:04.564 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.558+0000 7f9aeed9d700 1 -- 172.21.15.38:0/3746658712 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9af0101200 msgr2=0x7f9af00fee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:04.564 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.558+0000 7f9aeed9d700 1 --2- 172.21.15.38:0/3746658712 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9af0101200 0x7f9af00fee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:04.564 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.558+0000 7f9aeed9d700 1 -- 172.21.15.38:0/3746658712 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ae8009d00 con 0x7f9af010a540 2023-03-09T14:45:04.564 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.558+0000 7f9aeed9d700 1 --2- 172.21.15.38:0/3746658712 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9af010a540 0x7f9af01a43b0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f9ae800b830 tx=0x7f9ae80079b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:04.565 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.558+0000 7f9adf7fe700 1 -- 172.21.15.38:0/3746658712 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ae800e040 con 0x7f9af010a540 2023-03-09T14:45:04.565 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.558+0000 7f9af4ba8700 1 -- 172.21.15.38:0/3746658712 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9af010ffd0 con 0x7f9af010a540 2023-03-09T14:45:04.565 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.558+0000 7f9adf7fe700 1 -- 172.21.15.38:0/3746658712 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9ae8004d60 con 0x7f9af010a540 2023-03-09T14:45:04.565 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.558+0000 7f9adf7fe700 1 -- 172.21.15.38:0/3746658712 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ae801d480 con 0x7f9af010a540 2023-03-09T14:45:04.565 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.558+0000 7f9af4ba8700 1 -- 172.21.15.38:0/3746658712 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9af010fe40 con 0x7f9af010a540 2023-03-09T14:45:04.565 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.559+0000 7f9af4ba8700 1 -- 172.21.15.38:0/3746658712 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9af0060e80 con 0x7f9af010a540 2023-03-09T14:45:04.566 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.560+0000 7f9adf7fe700 1 -- 172.21.15.38:0/3746658712 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 133238+0+0 (secure 0 0 0) 0x7f9ae80040b0 con 0x7f9af010a540 2023-03-09T14:45:04.566 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.561+0000 7f9adf7fe700 1 --2- 172.21.15.38:0/3746658712 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f9ad809f400 0x7f9ad80a18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:04.567 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.561+0000 7f9adf7fe700 1 -- 172.21.15.38:0/3746658712 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f9ae80cc390 con 0x7f9af010a540 2023-03-09T14:45:04.567 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.561+0000 7f9aee59c700 1 --2- 172.21.15.38:0/3746658712 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f9ad809f400 0x7f9ad80a18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:04.567 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.561+0000 7f9adf7fe700 1 --2- 172.21.15.38:0/3746658712 >> [v2:172.21.15.122:6816/656874139,v1:172.21.15.122:6817/656874139] conn(0x7f9ad80a9ef0 0x7f9ad80ac320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:04.567 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.561+0000 7f9adf7fe700 1 -- 172.21.15.38:0/3746658712 --> [v2:172.21.15.122:6816/656874139,v1:172.21.15.122:6817/656874139] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9ad80ac9f0 con 0x7f9ad80a9ef0 2023-03-09T14:45:04.567 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.561+0000 7f9adf7fe700 1 -- 172.21.15.38:0/3746658712 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9ae80cc760 con 0x7f9af010a540 2023-03-09T14:45:04.567 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.561+0000 7f9aee59c700 1 --2- 172.21.15.38:0/3746658712 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f9ad809f400 0x7f9ad80a18b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f9ae0005fd0 tx=0x7f9ae000a560 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:04.567 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.561+0000 7f9aedd9b700 1 --2- 172.21.15.38:0/3746658712 >> [v2:172.21.15.122:6816/656874139,v1:172.21.15.122:6817/656874139] conn(0x7f9ad80a9ef0 0x7f9ad80ac320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:04.568 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.562+0000 7f9aedd9b700 1 --2- 172.21.15.38:0/3746658712 >> [v2:172.21.15.122:6816/656874139,v1:172.21.15.122:6817/656874139] conn(0x7f9ad80a9ef0 0x7f9ad80ac320 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:04.568 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.563+0000 7f9adf7fe700 1 -- 172.21.15.38:0/3746658712 <== osd.7 v2:172.21.15.122:6816/656874139 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f9ad80ac9f0 con 0x7f9ad80a9ef0 2023-03-09T14:45:04.581 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.579+0000 7f9af4ba8700 1 -- 172.21.15.38:0/3746658712 --> [v2:172.21.15.122:6816/656874139,v1:172.21.15.122:6817/656874139] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9af0110160 con 0x7f9ad80a9ef0 2023-03-09T14:45:04.582 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.579+0000 7f9adf7fe700 1 -- 172.21.15.38:0/3746658712 <== osd.7 v2:172.21.15.122:6816/656874139 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f9af0110160 con 0x7f9ad80a9ef0 2023-03-09T14:45:04.582 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.580+0000 7f9af4ba8700 1 -- 172.21.15.38:0/3746658712 >> [v2:172.21.15.122:6816/656874139,v1:172.21.15.122:6817/656874139] conn(0x7f9ad80a9ef0 msgr2=0x7f9ad80ac320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:04.583 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.580+0000 7f9af4ba8700 1 --2- 172.21.15.38:0/3746658712 >> [v2:172.21.15.122:6816/656874139,v1:172.21.15.122:6817/656874139] conn(0x7f9ad80a9ef0 0x7f9ad80ac320 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:04.583 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.580+0000 7f9af4ba8700 1 -- 172.21.15.38:0/3746658712 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f9ad809f400 msgr2=0x7f9ad80a18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:04.583 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.580+0000 7f9af4ba8700 1 --2- 172.21.15.38:0/3746658712 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f9ad809f400 0x7f9ad80a18b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f9ae0005fd0 tx=0x7f9ae000a560 comp rx=0 tx=0).stop 2023-03-09T14:45:04.583 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.580+0000 7f9af4ba8700 1 -- 172.21.15.38:0/3746658712 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9af010a540 msgr2=0x7f9af01a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:04.583 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.580+0000 7f9af4ba8700 1 --2- 172.21.15.38:0/3746658712 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9af010a540 0x7f9af01a43b0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f9ae800b830 tx=0x7f9ae80079b0 comp rx=0 tx=0).stop 2023-03-09T14:45:04.584 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.580+0000 7f9aeed9d700 1 -- 172.21.15.38:0/3746658712 reap_dead start 2023-03-09T14:45:04.584 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.580+0000 7f9af4ba8700 1 -- 172.21.15.38:0/3746658712 shutdown_connections 2023-03-09T14:45:04.584 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.580+0000 7f9af4ba8700 1 -- 172.21.15.38:0/3746658712 >> 172.21.15.38:0/3746658712 conn(0x7f9af0076d10 msgr2=0x7f9af0101fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:04.584 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.580+0000 7f9af4ba8700 1 -- 172.21.15.38:0/3746658712 shutdown_connections 2023-03-09T14:45:04.584 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.580+0000 7f9af4ba8700 1 -- 172.21.15.38:0/3746658712 wait complete. 2023-03-09T14:45:04.959 INFO:teuthology.orchestra.run.smithi038.stdout:279172874268 2023-03-09T14:45:04.959 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph osd last-stat-seq osd.9 2023-03-09T14:45:05.018 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.015+0000 7f64c2b51700 1 -- 172.21.15.38:0/437639904 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f64bc101b10 msgr2=0x7f64bc10a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:05.019 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.015+0000 7f64c2b51700 1 --2- 172.21.15.38:0/437639904 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f64bc101b10 0x7f64bc10a000 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f64a8009990 tx=0x7f64a8009ca0 comp rx=0 tx=0).stop 2023-03-09T14:45:05.019 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.015+0000 7f64c2b51700 1 -- 172.21.15.38:0/437639904 shutdown_connections 2023-03-09T14:45:05.019 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.015+0000 7f64c2b51700 1 --2- 172.21.15.38:0/437639904 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f64bc10a540 0x7f64bc10c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:05.019 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.015+0000 7f64c2b51700 1 --2- 172.21.15.38:0/437639904 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64bc101200 0x7f64bc1015d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:05.019 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.015+0000 7f64c2b51700 1 --2- 172.21.15.38:0/437639904 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f64bc101b10 0x7f64bc10a000 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:05.020 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.015+0000 7f64c2b51700 1 -- 172.21.15.38:0/437639904 >> 172.21.15.38:0/437639904 conn(0x7f64bc076d10 msgr2=0x7f64bc077110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:05.020 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.015+0000 7f64c2b51700 1 -- 172.21.15.38:0/437639904 shutdown_connections 2023-03-09T14:45:05.020 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.015+0000 7f64c2b51700 1 -- 172.21.15.38:0/437639904 wait complete. 2023-03-09T14:45:05.020 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.016+0000 7f64c2b51700 1 Processor -- start 2023-03-09T14:45:05.020 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.016+0000 7f64c2b51700 1 -- start start 2023-03-09T14:45:05.020 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.016+0000 7f64c2b51700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64bc101200 0x7f64bc1960e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:05.020 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.016+0000 7f64c2b51700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f64bc101b10 0x7f64bc196620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:05.020 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.016+0000 7f64c2b51700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f64bc10a540 0x7f64bc19aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:05.021 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.016+0000 7f64c2b51700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64bc19b0c0 con 0x7f64bc101b10 2023-03-09T14:45:05.021 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.016+0000 7f64c2b51700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64bc19b230 con 0x7f64bc101200 2023-03-09T14:45:05.021 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.016+0000 7f64c2b51700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64bc19b3f0 con 0x7f64bc10a540 2023-03-09T14:45:05.021 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.016+0000 7f64bbfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f64bc101b10 0x7f64bc196620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:05.021 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.016+0000 7f64bbfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f64bc101b10 0x7f64bc196620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37474/0 (socket says 172.21.15.38:37474) 2023-03-09T14:45:05.021 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.016+0000 7f64bbfff700 1 -- 172.21.15.38:0/2338285217 learned_addr learned my addr 172.21.15.38:0/2338285217 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:05.022 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.016+0000 7f64c08ed700 1 --2- 172.21.15.38:0/2338285217 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64bc101200 0x7f64bc1960e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:05.022 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.016+0000 7f64c10ee700 1 --2- 172.21.15.38:0/2338285217 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f64bc10a540 0x7f64bc19aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:05.023 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.017+0000 7f64bbfff700 1 -- 172.21.15.38:0/2338285217 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f64bc10a540 msgr2=0x7f64bc19aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:05.023 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.017+0000 7f64bbfff700 1 --2- 172.21.15.38:0/2338285217 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f64bc10a540 0x7f64bc19aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:05.023 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.017+0000 7f64bbfff700 1 -- 172.21.15.38:0/2338285217 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64bc101200 msgr2=0x7f64bc1960e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:05.024 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.017+0000 7f64bbfff700 1 --2- 172.21.15.38:0/2338285217 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f64bc101200 0x7f64bc1960e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:05.024 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.017+0000 7f64bbfff700 1 -- 172.21.15.38:0/2338285217 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64a80095f0 con 0x7f64bc101b10 2023-03-09T14:45:05.024 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.017+0000 7f64bbfff700 1 --2- 172.21.15.38:0/2338285217 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f64bc101b10 0x7f64bc196620 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f64a80095c0 tx=0x7f64a800f710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:05.024 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.017+0000 7f64b9ffb700 1 -- 172.21.15.38:0/2338285217 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a801d070 con 0x7f64bc101b10 2023-03-09T14:45:05.024 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.017+0000 7f64b9ffb700 1 -- 172.21.15.38:0/2338285217 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f64a80054f0 con 0x7f64bc101b10 2023-03-09T14:45:05.024 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.017+0000 7f64c2b51700 1 -- 172.21.15.38:0/2338285217 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64bc10ffe0 con 0x7f64bc101b10 2023-03-09T14:45:05.024 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.017+0000 7f64b9ffb700 1 -- 172.21.15.38:0/2338285217 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64a8004b00 con 0x7f64bc101b10 2023-03-09T14:45:05.025 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.017+0000 7f64c2b51700 1 -- 172.21.15.38:0/2338285217 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64bc10fe50 con 0x7f64bc101b10 2023-03-09T14:45:05.025 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.018+0000 7f64c2b51700 1 -- 172.21.15.38:0/2338285217 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f64bc0fc3b0 con 0x7f64bc101b10 2023-03-09T14:45:05.025 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.019+0000 7f64b9ffb700 1 -- 172.21.15.38:0/2338285217 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 133238+0+0 (secure 0 0 0) 0x7f64a8028030 con 0x7f64bc101b10 2023-03-09T14:45:05.025 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.020+0000 7f64b9ffb700 1 --2- 172.21.15.38:0/2338285217 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f64ac09f400 0x7f64ac0a18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:05.025 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.020+0000 7f64b9ffb700 1 -- 172.21.15.38:0/2338285217 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f64a80cb440 con 0x7f64bc101b10 2023-03-09T14:45:05.025 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.020+0000 7f64c08ed700 1 --2- 172.21.15.38:0/2338285217 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f64ac09f400 0x7f64ac0a18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:05.025 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.020+0000 7f64b9ffb700 1 --2- 172.21.15.38:0/2338285217 >> [v2:172.21.15.122:6808/2480764994,v1:172.21.15.122:6809/2480764994] conn(0x7f64ac0a9ef0 0x7f64ac0ac320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:05.025 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.020+0000 7f64b9ffb700 1 -- 172.21.15.38:0/2338285217 --> [v2:172.21.15.122:6808/2480764994,v1:172.21.15.122:6809/2480764994] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f64ac0ac9f0 con 0x7f64ac0a9ef0 2023-03-09T14:45:05.026 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.020+0000 7f64b9ffb700 1 -- 172.21.15.38:0/2338285217 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) v2 ==== 24+0+0 (secure 0 0 0) 0x7f64a80cb7b0 con 0x7f64bc101b10 2023-03-09T14:45:05.026 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.020+0000 7f64c10ee700 1 --2- 172.21.15.38:0/2338285217 >> [v2:172.21.15.122:6808/2480764994,v1:172.21.15.122:6809/2480764994] conn(0x7f64ac0a9ef0 0x7f64ac0ac320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:05.026 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.020+0000 7f64c08ed700 1 --2- 172.21.15.38:0/2338285217 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f64ac09f400 0x7f64ac0a18b0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f64b0005950 tx=0x7f64b000f820 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:05.026 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.021+0000 7f64c10ee700 1 --2- 172.21.15.38:0/2338285217 >> [v2:172.21.15.122:6808/2480764994,v1:172.21.15.122:6809/2480764994] conn(0x7f64ac0a9ef0 0x7f64ac0ac320 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:05.026 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.022+0000 7f64b9ffb700 1 -- 172.21.15.38:0/2338285217 <== osd.4 v2:172.21.15.122:6808/2480764994 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f64ac0ac9f0 con 0x7f64ac0a9ef0 2023-03-09T14:45:05.044 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.038+0000 7f64c2b51700 1 -- 172.21.15.38:0/2338285217 --> [v2:172.21.15.122:6808/2480764994,v1:172.21.15.122:6809/2480764994] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f64bc110170 con 0x7f64ac0a9ef0 2023-03-09T14:45:05.044 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.039+0000 7f64b9ffb700 1 -- 172.21.15.38:0/2338285217 <== osd.4 v2:172.21.15.122:6808/2480764994 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f64bc110170 con 0x7f64ac0a9ef0 2023-03-09T14:45:05.044 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.039+0000 7f64c2b51700 1 -- 172.21.15.38:0/2338285217 >> [v2:172.21.15.122:6808/2480764994,v1:172.21.15.122:6809/2480764994] conn(0x7f64ac0a9ef0 msgr2=0x7f64ac0ac320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:05.044 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.039+0000 7f64c2b51700 1 --2- 172.21.15.38:0/2338285217 >> [v2:172.21.15.122:6808/2480764994,v1:172.21.15.122:6809/2480764994] conn(0x7f64ac0a9ef0 0x7f64ac0ac320 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:05.045 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.039+0000 7f64c2b51700 1 -- 172.21.15.38:0/2338285217 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f64ac09f400 msgr2=0x7f64ac0a18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:05.045 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.039+0000 7f64c2b51700 1 --2- 172.21.15.38:0/2338285217 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f64ac09f400 0x7f64ac0a18b0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f64b0005950 tx=0x7f64b000f820 comp rx=0 tx=0).stop 2023-03-09T14:45:05.045 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.039+0000 7f64c2b51700 1 -- 172.21.15.38:0/2338285217 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f64bc101b10 msgr2=0x7f64bc196620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:05.045 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.039+0000 7f64c2b51700 1 --2- 172.21.15.38:0/2338285217 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f64bc101b10 0x7f64bc196620 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f64a80095c0 tx=0x7f64a800f710 comp rx=0 tx=0).stop 2023-03-09T14:45:05.045 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.039+0000 7f64c10ee700 1 -- 172.21.15.38:0/2338285217 reap_dead start 2023-03-09T14:45:05.045 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.039+0000 7f64c2b51700 1 -- 172.21.15.38:0/2338285217 shutdown_connections 2023-03-09T14:45:05.045 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.039+0000 7f64c2b51700 1 -- 172.21.15.38:0/2338285217 >> 172.21.15.38:0/2338285217 conn(0x7f64bc076d10 msgr2=0x7f64bc0fb9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:05.046 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.039+0000 7f64c2b51700 1 -- 172.21.15.38:0/2338285217 shutdown_connections 2023-03-09T14:45:05.046 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.039+0000 7f64c2b51700 1 -- 172.21.15.38:0/2338285217 wait complete. 2023-03-09T14:45:05.206 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.698+0000 7f74aed69700 1 -- 172.21.15.38:0/1356549442 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74a80ffae0 msgr2=0x7f74a8103aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:05.206 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.698+0000 7f74aed69700 1 --2- 172.21.15.38:0/1356549442 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74a80ffae0 0x7f74a8103aa0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f7494009920 tx=0x7f7494009c30 comp rx=0 tx=0).stop 2023-03-09T14:45:05.206 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.698+0000 7f74aed69700 1 -- 172.21.15.38:0/1356549442 shutdown_connections 2023-03-09T14:45:05.207 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.698+0000 7f74aed69700 1 --2- 172.21.15.38:0/1356549442 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f74a8104180 0x7f74a810bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:05.207 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.698+0000 7f74aed69700 1 --2- 172.21.15.38:0/1356549442 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f74a80ff140 0x7f74a80ff510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:05.207 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.698+0000 7f74aed69700 1 --2- 172.21.15.38:0/1356549442 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74a80ffae0 0x7f74a8103aa0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:05.207 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.698+0000 7f74aed69700 1 -- 172.21.15.38:0/1356549442 >> 172.21.15.38:0/1356549442 conn(0x7f74a8076d40 msgr2=0x7f74a8077140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:05.207 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.698+0000 7f74aed69700 1 -- 172.21.15.38:0/1356549442 shutdown_connections 2023-03-09T14:45:05.207 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.698+0000 7f74aed69700 1 -- 172.21.15.38:0/1356549442 wait complete. 2023-03-09T14:45:05.207 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.699+0000 7f74aed69700 1 Processor -- start 2023-03-09T14:45:05.208 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.699+0000 7f74aed69700 1 -- start start 2023-03-09T14:45:05.208 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.699+0000 7f74aed69700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f74a80ff140 0x7f74a8198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:05.208 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.699+0000 7f74aed69700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f74a80ffae0 0x7f74a8198960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:05.208 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.699+0000 7f74aed69700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74a8104180 0x7f74a819cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:05.208 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.699+0000 7f74aed69700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74a819d400 con 0x7f74a8104180 2023-03-09T14:45:05.208 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.699+0000 7f74aed69700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74a819d570 con 0x7f74a80ffae0 2023-03-09T14:45:05.209 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.699+0000 7f74aed69700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74a819d730 con 0x7f74a80ff140 2023-03-09T14:45:05.209 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.699+0000 7f74ad306700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74a8104180 0x7f74a819cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:05.209 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.699+0000 7f74ad306700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74a8104180 0x7f74a819cdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37460/0 (socket says 172.21.15.38:37460) 2023-03-09T14:45:05.209 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.699+0000 7f74ad306700 1 -- 172.21.15.38:0/1676703864 learned_addr learned my addr 172.21.15.38:0/1676703864 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:05.209 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.699+0000 7f74acb05700 1 --2- 172.21.15.38:0/1676703864 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f74a80ff140 0x7f74a8198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:05.209 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.699+0000 7f74a7fff700 1 --2- 172.21.15.38:0/1676703864 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f74a80ffae0 0x7f74a8198960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:05.209 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.700+0000 7f74ad306700 1 -- 172.21.15.38:0/1676703864 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f74a80ff140 msgr2=0x7f74a8198420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:05.210 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.700+0000 7f74ad306700 1 --2- 172.21.15.38:0/1676703864 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f74a80ff140 0x7f74a8198420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:05.210 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.700+0000 7f74ad306700 1 -- 172.21.15.38:0/1676703864 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f74a80ffae0 msgr2=0x7f74a8198960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:05.210 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.700+0000 7f74ad306700 1 --2- 172.21.15.38:0/1676703864 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f74a80ffae0 0x7f74a8198960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:05.210 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.700+0000 7f74ad306700 1 -- 172.21.15.38:0/1676703864 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7494009520 con 0x7f74a8104180 2023-03-09T14:45:05.210 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.700+0000 7f74a7fff700 1 --2- 172.21.15.38:0/1676703864 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f74a80ffae0 0x7f74a8198960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:45:05.210 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.700+0000 7f74ad306700 1 --2- 172.21.15.38:0/1676703864 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74a8104180 0x7f74a819cdc0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f74a000c110 tx=0x7f74a000c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:05.210 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.700+0000 7f74a5ffb700 1 -- 172.21.15.38:0/1676703864 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a0017070 con 0x7f74a8104180 2023-03-09T14:45:05.210 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.700+0000 7f74a5ffb700 1 -- 172.21.15.38:0/1676703864 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f74a000e050 con 0x7f74a8104180 2023-03-09T14:45:05.211 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.700+0000 7f74aed69700 1 -- 172.21.15.38:0/1676703864 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74a810ef50 con 0x7f74a8104180 2023-03-09T14:45:05.211 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.700+0000 7f74a5ffb700 1 -- 172.21.15.38:0/1676703864 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a0007940 con 0x7f74a8104180 2023-03-09T14:45:05.211 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.700+0000 7f74aed69700 1 -- 172.21.15.38:0/1676703864 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74a810edc0 con 0x7f74a8104180 2023-03-09T14:45:05.211 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.701+0000 7f74aed69700 1 -- 172.21.15.38:0/1676703864 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f74a8060f00 con 0x7f74a8104180 2023-03-09T14:45:05.211 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.702+0000 7f74a5ffb700 1 -- 172.21.15.38:0/1676703864 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 133238+0+0 (secure 0 0 0) 0x7f74a0004ad0 con 0x7f74a8104180 2023-03-09T14:45:05.211 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.703+0000 7f74a5ffb700 1 --2- 172.21.15.38:0/1676703864 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f749809f400 0x7f74980a18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:05.211 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.703+0000 7f74a5ffb700 1 -- 172.21.15.38:0/1676703864 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f74a00ca6b0 con 0x7f74a8104180 2023-03-09T14:45:05.212 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.703+0000 7f74acb05700 1 --2- 172.21.15.38:0/1676703864 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f749809f400 0x7f74980a18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:05.212 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.703+0000 7f74a5ffb700 1 --2- 172.21.15.38:0/1676703864 >> [v2:172.21.15.38:6810/2719814100,v1:172.21.15.38:6811/2719814100] conn(0x7f74980a9ef0 0x7f74980ac320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:05.212 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.703+0000 7f74a5ffb700 1 -- 172.21.15.38:0/1676703864 --> [v2:172.21.15.38:6810/2719814100,v1:172.21.15.38:6811/2719814100] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f74980ac9f0 con 0x7f74980a9ef0 2023-03-09T14:45:05.212 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.703+0000 7f74a5ffb700 1 -- 172.21.15.38:0/1676703864 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) v2 ==== 24+0+0 (secure 0 0 0) 0x7f74a00caa80 con 0x7f74a8104180 2023-03-09T14:45:05.212 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.703+0000 7f74a7fff700 1 --2- 172.21.15.38:0/1676703864 >> [v2:172.21.15.38:6810/2719814100,v1:172.21.15.38:6811/2719814100] conn(0x7f74980a9ef0 0x7f74980ac320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:05.212 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.703+0000 7f74acb05700 1 --2- 172.21.15.38:0/1676703864 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f749809f400 0x7f74980a18b0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f749c009c60 tx=0x7f749c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:05.213 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.703+0000 7f74a7fff700 1 --2- 172.21.15.38:0/1676703864 >> [v2:172.21.15.38:6810/2719814100,v1:172.21.15.38:6811/2719814100] conn(0x7f74980a9ef0 0x7f74980ac320 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:05.213 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.704+0000 7f74a5ffb700 1 -- 172.21.15.38:0/1676703864 <== osd.3 v2:172.21.15.38:6810/2719814100 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f74980ac9f0 con 0x7f74980a9ef0 2023-03-09T14:45:05.213 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.723+0000 7f74aed69700 1 -- 172.21.15.38:0/1676703864 --> [v2:172.21.15.38:6810/2719814100,v1:172.21.15.38:6811/2719814100] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f74a810f0e0 con 0x7f74980a9ef0 2023-03-09T14:45:05.214 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.723+0000 7f74a5ffb700 1 -- 172.21.15.38:0/1676703864 <== osd.3 v2:172.21.15.38:6810/2719814100 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f74a810f0e0 con 0x7f74980a9ef0 2023-03-09T14:45:05.214 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.724+0000 7f74aed69700 1 -- 172.21.15.38:0/1676703864 >> [v2:172.21.15.38:6810/2719814100,v1:172.21.15.38:6811/2719814100] conn(0x7f74980a9ef0 msgr2=0x7f74980ac320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:05.214 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.724+0000 7f74aed69700 1 --2- 172.21.15.38:0/1676703864 >> [v2:172.21.15.38:6810/2719814100,v1:172.21.15.38:6811/2719814100] conn(0x7f74980a9ef0 0x7f74980ac320 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:05.214 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.724+0000 7f74aed69700 1 -- 172.21.15.38:0/1676703864 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f749809f400 msgr2=0x7f74980a18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:05.215 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.724+0000 7f74aed69700 1 --2- 172.21.15.38:0/1676703864 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f749809f400 0x7f74980a18b0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f749c009c60 tx=0x7f749c009260 comp rx=0 tx=0).stop 2023-03-09T14:45:05.215 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.724+0000 7f74aed69700 1 -- 172.21.15.38:0/1676703864 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74a8104180 msgr2=0x7f74a819cdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:05.215 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.724+0000 7f74aed69700 1 --2- 172.21.15.38:0/1676703864 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f74a8104180 0x7f74a819cdc0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f74a000c110 tx=0x7f74a000c420 comp rx=0 tx=0).stop 2023-03-09T14:45:05.215 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.724+0000 7f74ad306700 1 -- 172.21.15.38:0/1676703864 reap_dead start 2023-03-09T14:45:05.215 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.724+0000 7f74aed69700 1 -- 172.21.15.38:0/1676703864 shutdown_connections 2023-03-09T14:45:05.215 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.724+0000 7f74aed69700 1 -- 172.21.15.38:0/1676703864 >> 172.21.15.38:0/1676703864 conn(0x7f74a8076d40 msgr2=0x7f74a80fd1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:05.215 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.724+0000 7f74aed69700 1 -- 172.21.15.38:0/1676703864 shutdown_connections 2023-03-09T14:45:05.216 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:04.724+0000 7f74aed69700 1 -- 172.21.15.38:0/1676703864 wait complete. 2023-03-09T14:45:05.216 INFO:teuthology.orchestra.run.smithi038.stdout:34359738436 2023-03-09T14:45:05.217 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph osd last-stat-seq osd.0 2023-03-09T14:45:05.688 INFO:teuthology.orchestra.run.smithi038.stdout:214748364838 2023-03-09T14:45:05.689 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph osd last-stat-seq osd.7 2023-03-09T14:45:05.824 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.695+0000 7fa2937e3700 1 -- 172.21.15.38:0/3192564163 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa28c10a540 msgr2=0x7fa28c10c9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:05.824 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.695+0000 7fa2937e3700 1 --2- 172.21.15.38:0/3192564163 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa28c10a540 0x7fa28c10c9a0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fa28400b0a0 tx=0x7fa28400b3b0 comp rx=0 tx=0).stop 2023-03-09T14:45:05.824 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.695+0000 7fa2937e3700 1 -- 172.21.15.38:0/3192564163 shutdown_connections 2023-03-09T14:45:05.825 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.695+0000 7fa2937e3700 1 --2- 172.21.15.38:0/3192564163 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa28c10a540 0x7fa28c10c9a0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:05.825 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.695+0000 7fa2937e3700 1 --2- 172.21.15.38:0/3192564163 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa28c101200 0x7fa28c1015d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:05.825 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.695+0000 7fa2937e3700 1 --2- 172.21.15.38:0/3192564163 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa28c101b10 0x7fa28c10a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:05.825 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.695+0000 7fa2937e3700 1 -- 172.21.15.38:0/3192564163 >> 172.21.15.38:0/3192564163 conn(0x7fa28c076d10 msgr2=0x7fa28c077110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:05.825 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.695+0000 7fa2937e3700 1 -- 172.21.15.38:0/3192564163 shutdown_connections 2023-03-09T14:45:05.825 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.695+0000 7fa2937e3700 1 -- 172.21.15.38:0/3192564163 wait complete. 2023-03-09T14:45:05.826 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.696+0000 7fa2937e3700 1 Processor -- start 2023-03-09T14:45:05.826 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.696+0000 7fa2937e3700 1 -- start start 2023-03-09T14:45:05.826 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.696+0000 7fa2937e3700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa28c101200 0x7fa28c196190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:05.826 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.696+0000 7fa2937e3700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa28c101b10 0x7fa28c1966d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:05.826 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.696+0000 7fa2937e3700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa28c10a540 0x7fa28c19ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:05.826 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.696+0000 7fa2937e3700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa28c19b170 con 0x7fa28c101b10 2023-03-09T14:45:05.826 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.696+0000 7fa2937e3700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa28c19b2e0 con 0x7fa28c10a540 2023-03-09T14:45:05.827 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.696+0000 7fa2937e3700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa28c19b4a0 con 0x7fa28c101200 2023-03-09T14:45:05.827 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.696+0000 7fa290d7e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa28c101b10 0x7fa28c1966d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:05.827 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.696+0000 7fa290d7e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa28c101b10 0x7fa28c1966d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37490/0 (socket says 172.21.15.38:37490) 2023-03-09T14:45:05.827 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.696+0000 7fa290d7e700 1 -- 172.21.15.38:0/2076807824 learned_addr learned my addr 172.21.15.38:0/2076807824 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:05.827 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.696+0000 7fa29157f700 1 --2- 172.21.15.38:0/2076807824 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa28c101200 0x7fa28c196190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:05.827 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.697+0000 7fa290d7e700 1 -- 172.21.15.38:0/2076807824 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa28c101200 msgr2=0x7fa28c196190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:05.827 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.697+0000 7fa290d7e700 1 --2- 172.21.15.38:0/2076807824 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa28c101200 0x7fa28c196190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:05.827 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.697+0000 7fa291d80700 1 --2- 172.21.15.38:0/2076807824 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa28c10a540 0x7fa28c19ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:05.828 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.697+0000 7fa290d7e700 1 -- 172.21.15.38:0/2076807824 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa28c10a540 msgr2=0x7fa28c19ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:05.828 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.697+0000 7fa290d7e700 1 --2- 172.21.15.38:0/2076807824 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa28c10a540 0x7fa28c19ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:05.828 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.697+0000 7fa290d7e700 1 -- 172.21.15.38:0/2076807824 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa284009d00 con 0x7fa28c101b10 2023-03-09T14:45:05.828 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.697+0000 7fa29157f700 1 --2- 172.21.15.38:0/2076807824 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa28c101200 0x7fa28c196190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:45:05.828 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.697+0000 7fa291d80700 1 --2- 172.21.15.38:0/2076807824 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa28c10a540 0x7fa28c19ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-09T14:45:05.828 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.697+0000 7fa290d7e700 1 --2- 172.21.15.38:0/2076807824 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa28c101b10 0x7fa28c1966d0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fa27c00e9a0 tx=0x7fa27c00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:05.828 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.697+0000 7fa2827fc700 1 -- 172.21.15.38:0/2076807824 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa27c009790 con 0x7fa28c101b10 2023-03-09T14:45:05.829 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.697+0000 7fa2827fc700 1 -- 172.21.15.38:0/2076807824 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa27c00cbd0 con 0x7fa28c101b10 2023-03-09T14:45:05.829 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.697+0000 7fa2937e3700 1 -- 172.21.15.38:0/2076807824 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa28c10ffd0 con 0x7fa28c101b10 2023-03-09T14:45:05.829 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.697+0000 7fa2827fc700 1 -- 172.21.15.38:0/2076807824 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa27c018680 con 0x7fa28c101b10 2023-03-09T14:45:05.829 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.697+0000 7fa2937e3700 1 -- 172.21.15.38:0/2076807824 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa28c10fe40 con 0x7fa28c101b10 2023-03-09T14:45:05.829 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.698+0000 7fa2937e3700 1 -- 172.21.15.38:0/2076807824 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa28c060f00 con 0x7fa28c101b10 2023-03-09T14:45:05.829 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.700+0000 7fa2827fc700 1 -- 172.21.15.38:0/2076807824 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 133238+0+0 (secure 0 0 0) 0x7fa27c00c040 con 0x7fa28c101b10 2023-03-09T14:45:05.829 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.701+0000 7fa2827fc700 1 --2- 172.21.15.38:0/2076807824 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fa27809f400 0x7fa2780a18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:05.830 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.701+0000 7fa29157f700 1 --2- 172.21.15.38:0/2076807824 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fa27809f400 0x7fa2780a18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:05.830 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.701+0000 7fa2827fc700 1 -- 172.21.15.38:0/2076807824 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fa27c014070 con 0x7fa28c101b10 2023-03-09T14:45:05.830 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.701+0000 7fa2827fc700 1 --2- 172.21.15.38:0/2076807824 >> [v2:172.21.15.122:6824/1427785280,v1:172.21.15.122:6825/1427785280] conn(0x7fa2780a9ef0 0x7fa2780ac320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:05.830 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.701+0000 7fa2827fc700 1 -- 172.21.15.38:0/2076807824 --> [v2:172.21.15.122:6824/1427785280,v1:172.21.15.122:6825/1427785280] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa2780ac9f0 con 0x7fa2780a9ef0 2023-03-09T14:45:05.831 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.701+0000 7fa2827fc700 1 -- 172.21.15.38:0/2076807824 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_get_version_reply(handle=1 version=87) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa27c0cacd0 con 0x7fa28c101b10 2023-03-09T14:45:05.831 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.701+0000 7fa29157f700 1 --2- 172.21.15.38:0/2076807824 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fa27809f400 0x7fa2780a18b0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa288009cb0 tx=0x7fa288009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:05.831 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.702+0000 7fa291d80700 1 --2- 172.21.15.38:0/2076807824 >> [v2:172.21.15.122:6824/1427785280,v1:172.21.15.122:6825/1427785280] conn(0x7fa2780a9ef0 0x7fa2780ac320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:05.831 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.702+0000 7fa291d80700 1 --2- 172.21.15.38:0/2076807824 >> [v2:172.21.15.122:6824/1427785280,v1:172.21.15.122:6825/1427785280] conn(0x7fa2780a9ef0 0x7fa2780ac320 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:05.831 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.703+0000 7fa2827fc700 1 -- 172.21.15.38:0/2076807824 <== osd.10 v2:172.21.15.122:6824/1427785280 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fa2780ac9f0 con 0x7fa2780a9ef0 2023-03-09T14:45:05.831 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.721+0000 7fa2937e3700 1 -- 172.21.15.38:0/2076807824 --> [v2:172.21.15.122:6824/1427785280,v1:172.21.15.122:6825/1427785280] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa28c110160 con 0x7fa2780a9ef0 2023-03-09T14:45:05.831 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.722+0000 7fa2827fc700 1 -- 172.21.15.38:0/2076807824 <== osd.10 v2:172.21.15.122:6824/1427785280 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fa28c110160 con 0x7fa2780a9ef0 2023-03-09T14:45:05.832 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.722+0000 7fa2937e3700 1 -- 172.21.15.38:0/2076807824 >> [v2:172.21.15.122:6824/1427785280,v1:172.21.15.122:6825/1427785280] conn(0x7fa2780a9ef0 msgr2=0x7fa2780ac320 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:05.832 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.722+0000 7fa2937e3700 1 --2- 172.21.15.38:0/2076807824 >> [v2:172.21.15.122:6824/1427785280,v1:172.21.15.122:6825/1427785280] conn(0x7fa2780a9ef0 0x7fa2780ac320 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:05.832 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.722+0000 7fa2937e3700 1 -- 172.21.15.38:0/2076807824 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fa27809f400 msgr2=0x7fa2780a18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:05.832 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.722+0000 7fa2937e3700 1 --2- 172.21.15.38:0/2076807824 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fa27809f400 0x7fa2780a18b0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa288009cb0 tx=0x7fa288009260 comp rx=0 tx=0).stop 2023-03-09T14:45:05.832 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.722+0000 7fa2937e3700 1 -- 172.21.15.38:0/2076807824 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa28c101b10 msgr2=0x7fa28c1966d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:05.832 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.722+0000 7fa2937e3700 1 --2- 172.21.15.38:0/2076807824 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa28c101b10 0x7fa28c1966d0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fa27c00e9a0 tx=0x7fa27c00ed60 comp rx=0 tx=0).stop 2023-03-09T14:45:05.832 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.722+0000 7fa291d80700 1 -- 172.21.15.38:0/2076807824 reap_dead start 2023-03-09T14:45:05.832 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.723+0000 7fa2937e3700 1 -- 172.21.15.38:0/2076807824 shutdown_connections 2023-03-09T14:45:05.833 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.723+0000 7fa2937e3700 1 -- 172.21.15.38:0/2076807824 >> 172.21.15.38:0/2076807824 conn(0x7fa28c076d10 msgr2=0x7fa28c101fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:05.833 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.723+0000 7fa2937e3700 1 -- 172.21.15.38:0/2076807824 shutdown_connections 2023-03-09T14:45:05.833 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:05.723+0000 7fa2937e3700 1 -- 172.21.15.38:0/2076807824 wait complete. 2023-03-09T14:45:05.914 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:05 smithi038 ceph-mon[92422]: pgmap v282: 129 pgs: 129 active+clean; 636 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 5 op/s 2023-03-09T14:45:05.915 INFO:teuthology.orchestra.run.smithi038.stdout:335007449108 2023-03-09T14:45:05.915 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph osd last-stat-seq osd.11 2023-03-09T14:45:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:05 smithi122 ceph-mon[93869]: pgmap v282: 129 pgs: 129 active+clean; 636 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 5 op/s 2023-03-09T14:45:06.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:05 smithi181 ceph-mon[94189]: pgmap v282: 129 pgs: 129 active+clean; 636 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 5 op/s 2023-03-09T14:45:06.447 INFO:teuthology.orchestra.run.smithi038.stdout:300647710744 2023-03-09T14:45:06.447 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph osd last-stat-seq osd.10 2023-03-09T14:45:06.717 INFO:teuthology.orchestra.run.smithi038.stdout:107374182455 2023-03-09T14:45:06.717 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph osd last-stat-seq osd.3 2023-03-09T14:45:06.793 INFO:teuthology.orchestra.run.smithi038.stdout:137438953523 2023-03-09T14:45:06.794 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph osd last-stat-seq osd.4 2023-03-09T14:45:06.924 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.a/config 2023-03-09T14:45:06.996 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.a/config 2023-03-09T14:45:07.012 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.a/config 2023-03-09T14:45:07.135 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.a/config 2023-03-09T14:45:07.139 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.a/config 2023-03-09T14:45:07.251 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.a/config 2023-03-09T14:45:07.346 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.a/config 2023-03-09T14:45:07.522 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.a/config 2023-03-09T14:45:07.537 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.a/config 2023-03-09T14:45:07.539 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.a/config 2023-03-09T14:45:08.067 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:07 smithi038 ceph-mon[92422]: pgmap v283: 129 pgs: 129 active+clean; 636 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 7 op/s 2023-03-09T14:45:08.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:07 smithi122 ceph-mon[93869]: pgmap v283: 129 pgs: 129 active+clean; 636 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 7 op/s 2023-03-09T14:45:08.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:07 smithi181 ceph-mon[94189]: pgmap v283: 129 pgs: 129 active+clean; 636 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 7 op/s 2023-03-09T14:45:09.982 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:09 smithi038 ceph-mon[92422]: pgmap v284: 129 pgs: 129 active+clean; 636 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 255 B/s wr, 5 op/s 2023-03-09T14:45:10.173 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.a/config 2023-03-09T14:45:10.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:09 smithi122 ceph-mon[93869]: pgmap v284: 129 pgs: 129 active+clean; 636 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 255 B/s wr, 5 op/s 2023-03-09T14:45:10.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:09 smithi181 ceph-mon[94189]: pgmap v284: 129 pgs: 129 active+clean; 636 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 255 B/s wr, 5 op/s 2023-03-09T14:45:11.708 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.706+0000 7f27df55e700 1 -- 172.21.15.38:0/626228555 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27d8104150 msgr2=0x7f27d810bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:11.709 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.706+0000 7f27df55e700 1 --2- 172.21.15.38:0/626228555 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27d8104150 0x7f27d810bd20 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f27d000b0d0 tx=0x7f27d000b3e0 comp rx=0 tx=0).stop 2023-03-09T14:45:11.709 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.706+0000 7f27df55e700 1 -- 172.21.15.38:0/626228555 shutdown_connections 2023-03-09T14:45:11.709 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.706+0000 7f27df55e700 1 --2- 172.21.15.38:0/626228555 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27d8104150 0x7f27d810bd20 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:11.709 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.706+0000 7f27df55e700 1 --2- 172.21.15.38:0/626228555 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f27d80ff110 0x7f27d80ff4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:11.709 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.706+0000 7f27df55e700 1 --2- 172.21.15.38:0/626228555 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f27d80ffab0 0x7f27d8103a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:11.710 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.706+0000 7f27df55e700 1 -- 172.21.15.38:0/626228555 >> 172.21.15.38:0/626228555 conn(0x7f27d80fa980 msgr2=0x7f27d80fcd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:11.710 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.706+0000 7f27df55e700 1 -- 172.21.15.38:0/626228555 shutdown_connections 2023-03-09T14:45:11.710 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.706+0000 7f27df55e700 1 -- 172.21.15.38:0/626228555 wait complete. 2023-03-09T14:45:11.711 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.707+0000 7f27df55e700 1 Processor -- start 2023-03-09T14:45:11.711 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.707+0000 7f27df55e700 1 -- start start 2023-03-09T14:45:11.711 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.707+0000 7f27df55e700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f27d80ff110 0x7f27d8195080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:11.712 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.707+0000 7f27df55e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27d80ffab0 0x7f27d81955c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:11.712 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.707+0000 7f27df55e700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f27d8104150 0x7f27d8199a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:11.712 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.707+0000 7f27df55e700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27d819a060 con 0x7f27d80ffab0 2023-03-09T14:45:11.712 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.707+0000 7f27df55e700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27d819a1d0 con 0x7f27d80ff110 2023-03-09T14:45:11.712 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.707+0000 7f27df55e700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27d819a390 con 0x7f27d8104150 2023-03-09T14:45:11.712 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.707+0000 7f27dcaf9700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27d80ffab0 0x7f27d81955c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:11.712 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.707+0000 7f27dcaf9700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27d80ffab0 0x7f27d81955c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37506/0 (socket says 172.21.15.38:37506) 2023-03-09T14:45:11.712 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.707+0000 7f27dcaf9700 1 -- 172.21.15.38:0/2723132074 learned_addr learned my addr 172.21.15.38:0/2723132074 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:11.713 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.708+0000 7f27dd2fa700 1 --2- 172.21.15.38:0/2723132074 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f27d80ff110 0x7f27d8195080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:11.713 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.708+0000 7f27ddafb700 1 --2- 172.21.15.38:0/2723132074 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f27d8104150 0x7f27d8199a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:11.713 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.708+0000 7f27dcaf9700 1 -- 172.21.15.38:0/2723132074 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f27d8104150 msgr2=0x7f27d8199a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:11.713 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.708+0000 7f27dcaf9700 1 --2- 172.21.15.38:0/2723132074 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f27d8104150 0x7f27d8199a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:11.713 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.708+0000 7f27dcaf9700 1 -- 172.21.15.38:0/2723132074 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f27d80ff110 msgr2=0x7f27d8195080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:11.713 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.708+0000 7f27dcaf9700 1 --2- 172.21.15.38:0/2723132074 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f27d80ff110 0x7f27d8195080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:11.713 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.708+0000 7f27dcaf9700 1 -- 172.21.15.38:0/2723132074 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27d0009d00 con 0x7f27d80ffab0 2023-03-09T14:45:11.714 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.708+0000 7f27ddafb700 1 --2- 172.21.15.38:0/2723132074 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f27d8104150 0x7f27d8199a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:45:11.714 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.708+0000 7f27dcaf9700 1 --2- 172.21.15.38:0/2723132074 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27d80ffab0 0x7f27d81955c0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f27c80098c0 tx=0x7f27c8009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:11.714 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.708+0000 7f27ce7fc700 1 -- 172.21.15.38:0/2723132074 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27c8012070 con 0x7f27d80ffab0 2023-03-09T14:45:11.714 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.708+0000 7f27ce7fc700 1 -- 172.21.15.38:0/2723132074 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f27c8007980 con 0x7f27d80ffab0 2023-03-09T14:45:11.714 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.708+0000 7f27df55e700 1 -- 172.21.15.38:0/2723132074 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27d810ef30 con 0x7f27d80ffab0 2023-03-09T14:45:11.714 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.708+0000 7f27ce7fc700 1 -- 172.21.15.38:0/2723132074 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27c8018af0 con 0x7f27d80ffab0 2023-03-09T14:45:11.714 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.708+0000 7f27df55e700 1 -- 172.21.15.38:0/2723132074 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27d810eda0 con 0x7f27d80ffab0 2023-03-09T14:45:11.715 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.710+0000 7f27df55e700 1 -- 172.21.15.38:0/2723132074 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27d810f0c0 con 0x7f27d80ffab0 2023-03-09T14:45:11.715 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.710+0000 7f27ce7fc700 1 -- 172.21.15.38:0/2723132074 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 133238+0+0 (secure 0 0 0) 0x7f27c801f030 con 0x7f27d80ffab0 2023-03-09T14:45:11.715 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.711+0000 7f27ce7fc700 1 --2- 172.21.15.38:0/2723132074 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f27c409f480 0x7f27c40a1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:11.716 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.711+0000 7f27dd2fa700 1 --2- 172.21.15.38:0/2723132074 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f27c409f480 0x7f27c40a1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:11.716 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.711+0000 7f27ce7fc700 1 -- 172.21.15.38:0/2723132074 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f27c80cafc0 con 0x7f27d80ffab0 2023-03-09T14:45:11.716 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.712+0000 7f27dd2fa700 1 --2- 172.21.15.38:0/2723132074 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f27c409f480 0x7f27c40a1930 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f27d4005950 tx=0x7f27d400b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:11.719 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.717+0000 7f27ce7fc700 1 -- 172.21.15.38:0/2723132074 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f27c80525b0 con 0x7f27d80ffab0 2023-03-09T14:45:11.941 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.938+0000 7f6c487a6700 1 -- 172.21.15.38:0/2595565531 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c40105500 msgr2=0x7f6c4010bcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:11.941 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.938+0000 7f6c487a6700 1 --2- 172.21.15.38:0/2595565531 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c40105500 0x7f6c4010bcd0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f6c3800b0a0 tx=0x7f6c3800b3b0 comp rx=0 tx=0).stop 2023-03-09T14:45:11.941 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.938+0000 7f6c487a6700 1 -- 172.21.15.38:0/2595565531 shutdown_connections 2023-03-09T14:45:11.941 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.938+0000 7f6c487a6700 1 --2- 172.21.15.38:0/2595565531 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c40105500 0x7f6c4010bcd0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:11.942 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.938+0000 7f6c487a6700 1 --2- 172.21.15.38:0/2595565531 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c40103e80 0x7f6c40104250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:11.942 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.938+0000 7f6c487a6700 1 --2- 172.21.15.38:0/2595565531 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c40104b50 0x7f6c40104fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:11.942 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.938+0000 7f6c487a6700 1 -- 172.21.15.38:0/2595565531 >> 172.21.15.38:0/2595565531 conn(0x7f6c40076020 msgr2=0x7f6c40076420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:11.942 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.938+0000 7f6c487a6700 1 -- 172.21.15.38:0/2595565531 shutdown_connections 2023-03-09T14:45:11.943 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.939+0000 7f6c487a6700 1 -- 172.21.15.38:0/2595565531 wait complete. 2023-03-09T14:45:11.943 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.939+0000 7f6c487a6700 1 Processor -- start 2023-03-09T14:45:11.943 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.940+0000 7f6c487a6700 1 -- start start 2023-03-09T14:45:11.943 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.940+0000 7f6c487a6700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c40103e80 0x7f6c4010aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:11.943 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.940+0000 7f6c487a6700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c40104b50 0x7f6c4010b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:11.944 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.940+0000 7f6c487a6700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c40105500 0x7f6c401b0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:11.944 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.940+0000 7f6c487a6700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c4010b830 con 0x7f6c40105500 2023-03-09T14:45:11.944 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.940+0000 7f6c487a6700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c401b1390 con 0x7f6c40103e80 2023-03-09T14:45:11.944 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.940+0000 7f6c487a6700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c401b14d0 con 0x7f6c40104b50 2023-03-09T14:45:11.944 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.940+0000 7f6c46d43700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c40105500 0x7f6c401b0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:11.944 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.940+0000 7f6c46d43700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c40105500 0x7f6c401b0e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37536/0 (socket says 172.21.15.38:37536) 2023-03-09T14:45:11.944 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.940+0000 7f6c46d43700 1 -- 172.21.15.38:0/86630030 learned_addr learned my addr 172.21.15.38:0/86630030 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:11.945 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.940+0000 7f6c46542700 1 --2- 172.21.15.38:0/86630030 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c40103e80 0x7f6c4010aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:11.945 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.940+0000 7f6c45d41700 1 --2- 172.21.15.38:0/86630030 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c40104b50 0x7f6c4010b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:11.945 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.941+0000 7f6c46d43700 1 -- 172.21.15.38:0/86630030 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c40104b50 msgr2=0x7f6c4010b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:11.945 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.941+0000 7f6c46d43700 1 --2- 172.21.15.38:0/86630030 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c40104b50 0x7f6c4010b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:11.945 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.941+0000 7f6c46d43700 1 -- 172.21.15.38:0/86630030 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c40103e80 msgr2=0x7f6c4010aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:11.945 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.941+0000 7f6c46d43700 1 --2- 172.21.15.38:0/86630030 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c40103e80 0x7f6c4010aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:11.945 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.941+0000 7f6c46d43700 1 -- 172.21.15.38:0/86630030 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c38009d00 con 0x7f6c40105500 2023-03-09T14:45:11.946 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.941+0000 7f6c46542700 1 --2- 172.21.15.38:0/86630030 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c40103e80 0x7f6c4010aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:45:11.946 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.941+0000 7f27df55e700 1 -- 172.21.15.38:0/2723132074 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) v1 -- 0x7f27d8196470 con 0x7f27d80ffab0 2023-03-09T14:45:11.947 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.941+0000 7f6c46d43700 1 --2- 172.21.15.38:0/86630030 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c40105500 0x7f6c401b0e50 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f6c3800b780 tx=0x7f6c380094e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:11.947 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.941+0000 7f27ce7fc700 1 -- 172.21.15.38:0/2723132074 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f27c801d020 con 0x7f27d80ffab0 2023-03-09T14:45:11.947 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.941+0000 7f6c377fe700 1 -- 172.21.15.38:0/86630030 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c3800e040 con 0x7f6c40105500 2023-03-09T14:45:11.948 INFO:teuthology.orchestra.run.smithi038.stdout:244813135906 2023-03-09T14:45:11.948 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.941+0000 7f6c487a6700 1 -- 172.21.15.38:0/86630030 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c401100c0 con 0x7f6c40105500 2023-03-09T14:45:11.948 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.941+0000 7f6c487a6700 1 -- 172.21.15.38:0/86630030 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c4010ff30 con 0x7f6c40105500 2023-03-09T14:45:11.949 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.941+0000 7f6c377fe700 1 -- 172.21.15.38:0/86630030 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6c3801d470 con 0x7f6c40105500 2023-03-09T14:45:11.949 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.941+0000 7f6c377fe700 1 -- 172.21.15.38:0/86630030 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c3801b6c0 con 0x7f6c40105500 2023-03-09T14:45:11.949 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.943+0000 7f6c487a6700 1 -- 172.21.15.38:0/86630030 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c40110250 con 0x7f6c40105500 2023-03-09T14:45:11.949 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.944+0000 7f27df55e700 1 -- 172.21.15.38:0/2723132074 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f27c409f480 msgr2=0x7f27c40a1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:11.950 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.944+0000 7f27df55e700 1 --2- 172.21.15.38:0/2723132074 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f27c409f480 0x7f27c40a1930 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f27d4005950 tx=0x7f27d400b410 comp rx=0 tx=0).stop 2023-03-09T14:45:11.950 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.944+0000 7f27df55e700 1 -- 172.21.15.38:0/2723132074 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27d80ffab0 msgr2=0x7f27d81955c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:11.950 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.944+0000 7f27df55e700 1 --2- 172.21.15.38:0/2723132074 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27d80ffab0 0x7f27d81955c0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f27c80098c0 tx=0x7f27c8009bd0 comp rx=0 tx=0).stop 2023-03-09T14:45:11.950 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.944+0000 7f27df55e700 1 -- 172.21.15.38:0/2723132074 shutdown_connections 2023-03-09T14:45:11.950 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.944+0000 7f27df55e700 1 --2- 172.21.15.38:0/2723132074 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f27c409f480 0x7f27c40a1930 secure :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f27d4005950 tx=0x7f27d400b410 comp rx=0 tx=0).stop 2023-03-09T14:45:11.950 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.944+0000 7f27df55e700 1 --2- 172.21.15.38:0/2723132074 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f27d8104150 0x7f27d8199a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:11.950 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.944+0000 7f27df55e700 1 --2- 172.21.15.38:0/2723132074 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f27d80ff110 0x7f27d8195080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:11.951 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.944+0000 7f27df55e700 1 --2- 172.21.15.38:0/2723132074 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f27d80ffab0 0x7f27d81955c0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:11.951 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.944+0000 7f27df55e700 1 -- 172.21.15.38:0/2723132074 >> 172.21.15.38:0/2723132074 conn(0x7f27d80fa980 msgr2=0x7f27d80fb330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:11.951 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.945+0000 7f27df55e700 1 -- 172.21.15.38:0/2723132074 shutdown_connections 2023-03-09T14:45:11.951 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.945+0000 7f27df55e700 1 -- 172.21.15.38:0/2723132074 wait complete. 2023-03-09T14:45:11.951 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.944+0000 7f6c377fe700 1 -- 172.21.15.38:0/86630030 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 133238+0+0 (secure 0 0 0) 0x7f6c3801d5e0 con 0x7f6c40105500 2023-03-09T14:45:11.952 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.945+0000 7f6c377fe700 1 --2- 172.21.15.38:0/86630030 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f6c2c09f480 0x7f6c2c0a1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:11.952 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.945+0000 7f6c46542700 1 --2- 172.21.15.38:0/86630030 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f6c2c09f480 0x7f6c2c0a1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:11.952 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.945+0000 7f6c377fe700 1 -- 172.21.15.38:0/86630030 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f6c380cb660 con 0x7f6c40105500 2023-03-09T14:45:11.952 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.946+0000 7f6c46542700 1 --2- 172.21.15.38:0/86630030 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f6c2c09f480 0x7f6c2c0a1930 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f6c30011490 tx=0x7f6c30009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:11.952 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.950+0000 7f6c377fe700 1 -- 172.21.15.38:0/86630030 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f6c38095820 con 0x7f6c40105500 2023-03-09T14:45:11.973 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:11 smithi038 ceph-mon[92422]: pgmap v285: 129 pgs: 129 active+clean; 636 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 6 op/s 2023-03-09T14:45:12.107 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.868+0000 7f277f143700 1 -- 172.21.15.38:0/2596478424 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2778101b10 msgr2=0x7f277810a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:12.108 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.868+0000 7f277f143700 1 --2- 172.21.15.38:0/2596478424 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2778101b10 0x7f277810a000 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f2764009990 tx=0x7f2764009ca0 comp rx=0 tx=0).stop 2023-03-09T14:45:12.108 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.868+0000 7f277f143700 1 -- 172.21.15.38:0/2596478424 shutdown_connections 2023-03-09T14:45:12.108 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.868+0000 7f277f143700 1 --2- 172.21.15.38:0/2596478424 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f277810a540 0x7f277810c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:12.108 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.868+0000 7f277f143700 1 --2- 172.21.15.38:0/2596478424 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2778101200 0x7f27781015d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:12.108 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.868+0000 7f277f143700 1 --2- 172.21.15.38:0/2596478424 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2778101b10 0x7f277810a000 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:12.108 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.868+0000 7f277f143700 1 -- 172.21.15.38:0/2596478424 >> 172.21.15.38:0/2596478424 conn(0x7f2778076d10 msgr2=0x7f2778077110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:12.109 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.868+0000 7f277f143700 1 -- 172.21.15.38:0/2596478424 shutdown_connections 2023-03-09T14:45:12.109 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.868+0000 7f277f143700 1 -- 172.21.15.38:0/2596478424 wait complete. 2023-03-09T14:45:12.109 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.869+0000 7f277f143700 1 Processor -- start 2023-03-09T14:45:12.109 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.869+0000 7f277f143700 1 -- start start 2023-03-09T14:45:12.109 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.869+0000 7f277f143700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2778101200 0x7f27781982f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:12.109 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.869+0000 7f277f143700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2778101b10 0x7f2778198830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:12.109 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.869+0000 7f277f143700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f277810a540 0x7f277819cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:12.110 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.869+0000 7f277f143700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f277819d2d0 con 0x7f2778101b10 2023-03-09T14:45:12.110 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.869+0000 7f277f143700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f277819d440 con 0x7f277810a540 2023-03-09T14:45:12.110 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.869+0000 7f277f143700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f277819d600 con 0x7f2778101200 2023-03-09T14:45:12.110 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.869+0000 7f2777fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2778101b10 0x7f2778198830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:12.110 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.869+0000 7f2777fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2778101b10 0x7f2778198830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37520/0 (socket says 172.21.15.38:37520) 2023-03-09T14:45:12.110 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.869+0000 7f2777fff700 1 -- 172.21.15.38:0/1299958099 learned_addr learned my addr 172.21.15.38:0/1299958099 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:12.110 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.869+0000 7f277cedf700 1 --2- 172.21.15.38:0/1299958099 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2778101200 0x7f27781982f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:12.111 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.869+0000 7f277d6e0700 1 --2- 172.21.15.38:0/1299958099 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f277810a540 0x7f277819cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:12.111 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.869+0000 7f2777fff700 1 -- 172.21.15.38:0/1299958099 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2778101200 msgr2=0x7f27781982f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:12.111 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.869+0000 7f2777fff700 1 --2- 172.21.15.38:0/1299958099 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2778101200 0x7f27781982f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:12.111 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.869+0000 7f2777fff700 1 -- 172.21.15.38:0/1299958099 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f277810a540 msgr2=0x7f277819cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:12.111 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.869+0000 7f2777fff700 1 --2- 172.21.15.38:0/1299958099 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f277810a540 0x7f277819cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:12.111 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.869+0000 7f2777fff700 1 -- 172.21.15.38:0/1299958099 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27640095f0 con 0x7f2778101b10 2023-03-09T14:45:12.111 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.870+0000 7f2777fff700 1 --2- 172.21.15.38:0/1299958099 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2778101b10 0x7f2778198830 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f27640098e0 tx=0x7f276400f670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:12.111 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.870+0000 7f2775ffb700 1 -- 172.21.15.38:0/1299958099 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f276401d070 con 0x7f2778101b10 2023-03-09T14:45:12.112 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.870+0000 7f2775ffb700 1 -- 172.21.15.38:0/1299958099 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2764005510 con 0x7f2778101b10 2023-03-09T14:45:12.112 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.870+0000 7f277f143700 1 -- 172.21.15.38:0/1299958099 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f277810ffe0 con 0x7f2778101b10 2023-03-09T14:45:12.112 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.870+0000 7f2775ffb700 1 -- 172.21.15.38:0/1299958099 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2764004bd0 con 0x7f2778101b10 2023-03-09T14:45:12.112 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.870+0000 7f277f143700 1 -- 172.21.15.38:0/1299958099 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f277810fe50 con 0x7f2778101b10 2023-03-09T14:45:12.112 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.871+0000 7f277f143700 1 -- 172.21.15.38:0/1299958099 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2778110170 con 0x7f2778101b10 2023-03-09T14:45:12.112 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.872+0000 7f2775ffb700 1 -- 172.21.15.38:0/1299958099 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 133238+0+0 (secure 0 0 0) 0x7f2764028030 con 0x7f2778101b10 2023-03-09T14:45:12.113 INFO:teuthology.orchestra.run.smithi038.stdout:158913790000 2023-03-09T14:45:12.113 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.873+0000 7f2775ffb700 1 --2- 172.21.15.38:0/1299958099 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f276809f480 0x7f27680a1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:12.114 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.873+0000 7f2775ffb700 1 -- 172.21.15.38:0/1299958099 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f27640cac80 con 0x7f2778101b10 2023-03-09T14:45:12.114 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.873+0000 7f277cedf700 1 --2- 172.21.15.38:0/1299958099 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f276809f480 0x7f27680a1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:12.114 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.874+0000 7f277cedf700 1 --2- 172.21.15.38:0/1299958099 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f276809f480 0x7f27680a1930 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f276c007be0 tx=0x7f276c0073d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:12.114 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:11.877+0000 7f2775ffb700 1 -- 172.21.15.38:0/1299958099 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f276405b5b0 con 0x7f2778101b10 2023-03-09T14:45:12.114 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.103+0000 7f277f143700 1 -- 172.21.15.38:0/1299958099 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f27781a53c0 con 0x7f2778101b10 2023-03-09T14:45:12.114 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.103+0000 7f2775ffb700 1 -- 172.21.15.38:0/1299958099 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f2764094ea0 con 0x7f2778101b10 2023-03-09T14:45:12.114 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.106+0000 7f277f143700 1 -- 172.21.15.38:0/1299958099 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f276809f480 msgr2=0x7f27680a1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:12.115 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.106+0000 7f277f143700 1 --2- 172.21.15.38:0/1299958099 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f276809f480 0x7f27680a1930 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f276c007be0 tx=0x7f276c0073d0 comp rx=0 tx=0).stop 2023-03-09T14:45:12.115 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.106+0000 7f277f143700 1 -- 172.21.15.38:0/1299958099 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2778101b10 msgr2=0x7f2778198830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:12.115 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.106+0000 7f277f143700 1 --2- 172.21.15.38:0/1299958099 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2778101b10 0x7f2778198830 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f27640098e0 tx=0x7f276400f670 comp rx=0 tx=0).stop 2023-03-09T14:45:12.115 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.106+0000 7f277f143700 1 -- 172.21.15.38:0/1299958099 shutdown_connections 2023-03-09T14:45:12.115 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.106+0000 7f277f143700 1 --2- 172.21.15.38:0/1299958099 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f276809f480 0x7f27680a1930 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:12.115 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.106+0000 7f277f143700 1 --2- 172.21.15.38:0/1299958099 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f277810a540 0x7f277819cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:12.115 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.106+0000 7f277f143700 1 --2- 172.21.15.38:0/1299958099 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2778101200 0x7f27781982f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:12.115 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.106+0000 7f277f143700 1 --2- 172.21.15.38:0/1299958099 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2778101b10 0x7f2778198830 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:12.116 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.106+0000 7f277f143700 1 -- 172.21.15.38:0/1299958099 >> 172.21.15.38:0/1299958099 conn(0x7f2778076d10 msgr2=0x7f27780fd9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:12.116 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.106+0000 7f277f143700 1 -- 172.21.15.38:0/1299958099 shutdown_connections 2023-03-09T14:45:12.116 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.106+0000 7f277f143700 1 -- 172.21.15.38:0/1299958099 wait complete. 2023-03-09T14:45:12.170 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.168+0000 7f6c487a6700 1 -- 172.21.15.38:0/86630030 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f6c40102950 con 0x7f6c40105500 2023-03-09T14:45:12.171 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.169+0000 7f6c377fe700 1 -- 172.21.15.38:0/86630030 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f6c38098e40 con 0x7f6c40105500 2023-03-09T14:45:12.171 INFO:teuthology.orchestra.run.smithi038.stdout:193273528364 2023-03-09T14:45:12.173 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.171+0000 7f6c487a6700 1 -- 172.21.15.38:0/86630030 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f6c2c09f480 msgr2=0x7f6c2c0a1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:12.173 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.171+0000 7f6c487a6700 1 --2- 172.21.15.38:0/86630030 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f6c2c09f480 0x7f6c2c0a1930 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f6c30011490 tx=0x7f6c30009260 comp rx=0 tx=0).stop 2023-03-09T14:45:12.173 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.171+0000 7f6c487a6700 1 -- 172.21.15.38:0/86630030 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c40105500 msgr2=0x7f6c401b0e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:12.173 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.171+0000 7f6c487a6700 1 --2- 172.21.15.38:0/86630030 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c40105500 0x7f6c401b0e50 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f6c3800b780 tx=0x7f6c380094e0 comp rx=0 tx=0).stop 2023-03-09T14:45:12.174 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.171+0000 7f6c487a6700 1 -- 172.21.15.38:0/86630030 shutdown_connections 2023-03-09T14:45:12.174 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.171+0000 7f6c487a6700 1 --2- 172.21.15.38:0/86630030 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c40105500 0x7f6c401b0e50 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:12.174 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.171+0000 7f6c487a6700 1 --2- 172.21.15.38:0/86630030 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f6c2c09f480 0x7f6c2c0a1930 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:12.174 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.171+0000 7f6c487a6700 1 --2- 172.21.15.38:0/86630030 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c40103e80 0x7f6c4010aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:12.174 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.171+0000 7f6c487a6700 1 --2- 172.21.15.38:0/86630030 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c40104b50 0x7f6c4010b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:12.174 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.171+0000 7f6c487a6700 1 -- 172.21.15.38:0/86630030 >> 172.21.15.38:0/86630030 conn(0x7f6c40076020 msgr2=0x7f6c400ffa90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:12.174 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.171+0000 7f6c487a6700 1 -- 172.21.15.38:0/86630030 shutdown_connections 2023-03-09T14:45:12.175 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.171+0000 7f6c487a6700 1 -- 172.21.15.38:0/86630030 wait complete. 2023-03-09T14:45:12.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:11 smithi122 ceph-mon[93869]: pgmap v285: 129 pgs: 129 active+clean; 636 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 6 op/s 2023-03-09T14:45:12.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:11 smithi181 ceph-mon[94189]: pgmap v285: 129 pgs: 129 active+clean; 636 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 6 op/s 2023-03-09T14:45:13.040 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:13.036+0000 7f4de99f1700 1 -- 172.21.15.38:0/3227032094 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4de4101120 msgr2=0x7f4de41014f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:13.041 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:13.036+0000 7f4de99f1700 1 --2- 172.21.15.38:0/3227032094 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4de4101120 0x7f4de41014f0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f4dd4009990 tx=0x7f4dd4009ca0 comp rx=0 tx=0).stop 2023-03-09T14:45:13.041 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:13.036+0000 7f4de99f1700 1 -- 172.21.15.38:0/3227032094 shutdown_connections 2023-03-09T14:45:13.041 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:13.036+0000 7f4de99f1700 1 --2- 172.21.15.38:0/3227032094 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4de4101120 0x7f4de41014f0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:13.041 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:13.036+0000 7f4de99f1700 1 --2- 172.21.15.38:0/3227032094 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4de410a640 0x7f4de410ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:13.041 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:13.036+0000 7f4de99f1700 1 --2- 172.21.15.38:0/3227032094 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4de4101a30 0x7f4de410a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:13.041 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:13.036+0000 7f4de99f1700 1 -- 172.21.15.38:0/3227032094 >> 172.21.15.38:0/3227032094 conn(0x7f4de4074cb0 msgr2=0x7f4de40750b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:13.042 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:13.036+0000 7f4de99f1700 1 -- 172.21.15.38:0/3227032094 shutdown_connections 2023-03-09T14:45:13.042 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:13.036+0000 7f4de99f1700 1 -- 172.21.15.38:0/3227032094 wait complete. 2023-03-09T14:45:13.042 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:13.037+0000 7f4de99f1700 1 Processor -- start 2023-03-09T14:45:13.042 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:13.037+0000 7f4de99f1700 1 -- start start 2023-03-09T14:45:13.042 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:13.038+0000 7f4de99f1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4de4101120 0x7f4de4196130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:13.042 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:13.038+0000 7f4de99f1700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4de4101a30 0x7f4de4196670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:13.042 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:13.038+0000 7f4de99f1700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4de410a640 0x7f4de419aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:13.042 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:13.038+0000 7f4de99f1700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de419b110 con 0x7f4de4101120 2023-03-09T14:45:13.043 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:13.038+0000 7f4de99f1700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de419b280 con 0x7f4de4101a30 2023-03-09T14:45:13.043 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:13.038+0000 7f4de99f1700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de419b440 con 0x7f4de410a640 2023-03-09T14:45:13.043 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:13.038+0000 7f4de2ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4de4101120 0x7f4de4196130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:13.043 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:13.038+0000 7f4de2ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4de4101120 0x7f4de4196130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37558/0 (socket says 172.21.15.38:37558) 2023-03-09T14:45:13.043 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:13.038+0000 7f4de2ffd700 1 -- 172.21.15.38:0/3359801591 learned_addr learned my addr 172.21.15.38:0/3359801591 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:13.043 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:13.038+0000 7f4de37fe700 1 --2- 172.21.15.38:0/3359801591 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4de410a640 0x7f4de419aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:13.043 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:13.038+0000 7f4de27fc700 1 --2- 172.21.15.38:0/3359801591 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4de4101a30 0x7f4de4196670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:13.044 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:13.038+0000 7f4de2ffd700 1 -- 172.21.15.38:0/3359801591 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4de410a640 msgr2=0x7f4de419aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:13.044 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:13.038+0000 7f4de2ffd700 1 --2- 172.21.15.38:0/3359801591 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4de410a640 0x7f4de419aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:13.044 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:13.038+0000 7f4de2ffd700 1 -- 172.21.15.38:0/3359801591 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4de4101a30 msgr2=0x7f4de4196670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:13.044 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:13.038+0000 7f4de2ffd700 1 --2- 172.21.15.38:0/3359801591 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4de4101a30 0x7f4de4196670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:13.044 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:13.038+0000 7f4de2ffd700 1 -- 172.21.15.38:0/3359801591 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dd40095f0 con 0x7f4de4101120 2023-03-09T14:45:13.044 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:13.038+0000 7f4de27fc700 1 --2- 172.21.15.38:0/3359801591 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4de4101a30 0x7f4de4196670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:45:13.044 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:13.038+0000 7f4de2ffd700 1 --2- 172.21.15.38:0/3359801591 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4de4101120 0x7f4de4196130 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f4dd40095c0 tx=0x7f4dd400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:13.045 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:13.038+0000 7f4de89ef700 1 -- 172.21.15.38:0/3359801591 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dd401d070 con 0x7f4de4101120 2023-03-09T14:45:13.046 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:13.039+0000 7f4de99f1700 1 -- 172.21.15.38:0/3359801591 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4de4110100 con 0x7f4de4101120 2023-03-09T14:45:13.046 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:13.039+0000 7f4de89ef700 1 -- 172.21.15.38:0/3359801591 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4dd40054f0 con 0x7f4de4101120 2023-03-09T14:45:13.046 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:13.039+0000 7f4de89ef700 1 -- 172.21.15.38:0/3359801591 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dd4004b30 con 0x7f4de4101120 2023-03-09T14:45:13.046 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:13.039+0000 7f4de99f1700 1 -- 172.21.15.38:0/3359801591 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4de410ff70 con 0x7f4de4101120 2023-03-09T14:45:13.046 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:13.040+0000 7f4de99f1700 1 -- 172.21.15.38:0/3359801591 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4de4110290 con 0x7f4de4101120 2023-03-09T14:45:13.046 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:13.041+0000 7f4de89ef700 1 -- 172.21.15.38:0/3359801591 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 133238+0+0 (secure 0 0 0) 0x7f4dd4028030 con 0x7f4de4101120 2023-03-09T14:45:13.046 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:13.041+0000 7f4de89ef700 1 --2- 172.21.15.38:0/3359801591 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f4dcc09f400 0x7f4dcc0a18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:13.046 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:13.041+0000 7f4de27fc700 1 --2- 172.21.15.38:0/3359801591 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f4dcc09f400 0x7f4dcc0a18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:13.047 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:13.041+0000 7f4de89ef700 1 -- 172.21.15.38:0/3359801591 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f4dd40cade0 con 0x7f4de4101120 2023-03-09T14:45:13.047 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:13.042+0000 7f4de27fc700 1 --2- 172.21.15.38:0/3359801591 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f4dcc09f400 0x7f4dcc0a18b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f4dd8005950 tx=0x7f4dd80058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:13.047 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:13.045+0000 7f4de89ef700 1 -- 172.21.15.38:0/3359801591 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f4dd4095000 con 0x7f4de4101120 2023-03-09T14:45:13.059 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.762+0000 7f23f599f700 1 -- 172.21.15.38:0/339568155 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23f00fef50 msgr2=0x7f23f00ff320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:13.059 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.762+0000 7f23f599f700 1 --2- 172.21.15.38:0/339568155 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23f00fef50 0x7f23f00ff320 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f23e40099c0 tx=0x7f23e4009cd0 comp rx=0 tx=0).stop 2023-03-09T14:45:13.059 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.762+0000 7f23f599f700 1 -- 172.21.15.38:0/339568155 shutdown_connections 2023-03-09T14:45:13.059 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.762+0000 7f23f599f700 1 --2- 172.21.15.38:0/339568155 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23f00fef50 0x7f23f00ff320 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:13.059 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.762+0000 7f23f599f700 1 --2- 172.21.15.38:0/339568155 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23f0103f90 0x7f23f010bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:13.059 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.762+0000 7f23f599f700 1 --2- 172.21.15.38:0/339568155 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f23f00ff8f0 0x7f23f01038b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:13.060 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.762+0000 7f23f599f700 1 -- 172.21.15.38:0/339568155 >> 172.21.15.38:0/339568155 conn(0x7f23f0075e40 msgr2=0x7f23f0076240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:13.060 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.762+0000 7f23f599f700 1 -- 172.21.15.38:0/339568155 shutdown_connections 2023-03-09T14:45:13.060 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.762+0000 7f23f599f700 1 -- 172.21.15.38:0/339568155 wait complete. 2023-03-09T14:45:13.060 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.763+0000 7f23f599f700 1 Processor -- start 2023-03-09T14:45:13.060 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.763+0000 7f23f599f700 1 -- start start 2023-03-09T14:45:13.060 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.763+0000 7f23f599f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23f00fef50 0x7f23f0194e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:13.060 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.763+0000 7f23f599f700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f23f00ff8f0 0x7f23f0195350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:13.061 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.763+0000 7f23f599f700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23f0103f90 0x7f23f01997b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:13.061 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.763+0000 7f23f599f700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23f0199df0 con 0x7f23f00fef50 2023-03-09T14:45:13.061 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.763+0000 7f23f599f700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23f0199f60 con 0x7f23f0103f90 2023-03-09T14:45:13.061 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.763+0000 7f23f599f700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23f019a120 con 0x7f23f00ff8f0 2023-03-09T14:45:13.061 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.763+0000 7f23f499d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23f00fef50 0x7f23f0194e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:13.061 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.763+0000 7f23f499d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23f00fef50 0x7f23f0194e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37544/0 (socket says 172.21.15.38:37544) 2023-03-09T14:45:13.061 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.763+0000 7f23f499d700 1 -- 172.21.15.38:0/3556004980 learned_addr learned my addr 172.21.15.38:0/3556004980 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:13.061 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.763+0000 7f23effff700 1 --2- 172.21.15.38:0/3556004980 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f23f00ff8f0 0x7f23f0195350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:13.062 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.763+0000 7f23f519e700 1 --2- 172.21.15.38:0/3556004980 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23f0103f90 0x7f23f01997b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:13.062 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.763+0000 7f23f499d700 1 -- 172.21.15.38:0/3556004980 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f23f00ff8f0 msgr2=0x7f23f0195350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:13.062 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.763+0000 7f23f499d700 1 --2- 172.21.15.38:0/3556004980 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f23f00ff8f0 0x7f23f0195350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:13.062 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.763+0000 7f23f499d700 1 -- 172.21.15.38:0/3556004980 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23f0103f90 msgr2=0x7f23f01997b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:13.062 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.763+0000 7f23f499d700 1 --2- 172.21.15.38:0/3556004980 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23f0103f90 0x7f23f01997b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:13.062 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.763+0000 7f23f499d700 1 -- 172.21.15.38:0/3556004980 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23e40095f0 con 0x7f23f00fef50 2023-03-09T14:45:13.063 INFO:teuthology.orchestra.run.smithi038.stdout:77309411389 2023-03-09T14:45:13.063 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.764+0000 7f23f499d700 1 --2- 172.21.15.38:0/3556004980 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23f00fef50 0x7f23f0194e10 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f23e4009990 tx=0x7f23e4009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:13.063 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.764+0000 7f23edffb700 1 -- 172.21.15.38:0/3556004980 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e401d070 con 0x7f23f00fef50 2023-03-09T14:45:13.063 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.764+0000 7f23f599f700 1 -- 172.21.15.38:0/3556004980 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23f010ed60 con 0x7f23f00fef50 2023-03-09T14:45:13.064 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.764+0000 7f23edffb700 1 -- 172.21.15.38:0/3556004980 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f23e4003f90 con 0x7f23f00fef50 2023-03-09T14:45:13.064 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.764+0000 7f23edffb700 1 -- 172.21.15.38:0/3556004980 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23e4022ce0 con 0x7f23f00fef50 2023-03-09T14:45:13.064 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.764+0000 7f23f599f700 1 -- 172.21.15.38:0/3556004980 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23f010ebd0 con 0x7f23f00fef50 2023-03-09T14:45:13.064 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.765+0000 7f23f599f700 1 -- 172.21.15.38:0/3556004980 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23f010eef0 con 0x7f23f00fef50 2023-03-09T14:45:13.064 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.766+0000 7f23edffb700 1 -- 172.21.15.38:0/3556004980 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 133238+0+0 (secure 0 0 0) 0x7f23e4029030 con 0x7f23f00fef50 2023-03-09T14:45:13.064 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.767+0000 7f23edffb700 1 --2- 172.21.15.38:0/3556004980 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f23e009f400 0x7f23e00a18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:13.064 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.767+0000 7f23effff700 1 --2- 172.21.15.38:0/3556004980 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f23e009f400 0x7f23e00a18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:13.065 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.767+0000 7f23edffb700 1 -- 172.21.15.38:0/3556004980 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f23e40cb130 con 0x7f23f00fef50 2023-03-09T14:45:13.065 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.767+0000 7f23effff700 1 --2- 172.21.15.38:0/3556004980 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f23e009f400 0x7f23e00a18b0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f23f0196420 tx=0x7f23dc00a300 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:13.065 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.771+0000 7f23edffb700 1 -- 172.21.15.38:0/3556004980 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f23e4095350 con 0x7f23f00fef50 2023-03-09T14:45:13.065 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.997+0000 7f23f599f700 1 -- 172.21.15.38:0/3556004980 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f23f004e550 con 0x7f23f00fef50 2023-03-09T14:45:13.065 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.997+0000 7f23edffb700 1 -- 172.21.15.38:0/3556004980 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f23e4098970 con 0x7f23f00fef50 2023-03-09T14:45:13.065 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.999+0000 7f23f599f700 1 -- 172.21.15.38:0/3556004980 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f23e009f400 msgr2=0x7f23e00a18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:13.065 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.999+0000 7f23f599f700 1 --2- 172.21.15.38:0/3556004980 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f23e009f400 0x7f23e00a18b0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f23f0196420 tx=0x7f23dc00a300 comp rx=0 tx=0).stop 2023-03-09T14:45:13.065 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.999+0000 7f23f599f700 1 -- 172.21.15.38:0/3556004980 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23f00fef50 msgr2=0x7f23f0194e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:13.066 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.999+0000 7f23f599f700 1 --2- 172.21.15.38:0/3556004980 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23f00fef50 0x7f23f0194e10 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f23e4009990 tx=0x7f23e4009580 comp rx=0 tx=0).stop 2023-03-09T14:45:13.066 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.999+0000 7f23f599f700 1 -- 172.21.15.38:0/3556004980 shutdown_connections 2023-03-09T14:45:13.066 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.999+0000 7f23f599f700 1 --2- 172.21.15.38:0/3556004980 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f23f00fef50 0x7f23f0194e10 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:13.066 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.999+0000 7f23f599f700 1 --2- 172.21.15.38:0/3556004980 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f23e009f400 0x7f23e00a18b0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:13.066 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.999+0000 7f23f599f700 1 --2- 172.21.15.38:0/3556004980 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f23f0103f90 0x7f23f01997b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:13.066 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.999+0000 7f23f599f700 1 --2- 172.21.15.38:0/3556004980 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f23f00ff8f0 0x7f23f0195350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:13.066 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.999+0000 7f23f599f700 1 -- 172.21.15.38:0/3556004980 >> 172.21.15.38:0/3556004980 conn(0x7f23f0075e40 msgr2=0x7f23f0102890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:13.066 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:12.999+0000 7f23f599f700 1 -- 172.21.15.38:0/3556004980 shutdown_connections 2023-03-09T14:45:13.067 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:13.000+0000 7f23f599f700 1 -- 172.21.15.38:0/3556004980 wait complete. 2023-03-09T14:45:13.213 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:12 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/2723132074' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2023-03-09T14:45:13.213 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:12 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/1299958099' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-03-09T14:45:13.213 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:12 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/86630030' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2023-03-09T14:45:13.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:12 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/2723132074' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2023-03-09T14:45:13.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:12 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/1299958099' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-03-09T14:45:13.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:12 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/86630030' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2023-03-09T14:45:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:12 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/2723132074' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2023-03-09T14:45:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:12 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/1299958099' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-03-09T14:45:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:12 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/86630030' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2023-03-09T14:45:13.260 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:13.258+0000 7f4de99f1700 1 -- 172.21.15.38:0/3359801591 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f4de41a31a0 con 0x7f4de4101120 2023-03-09T14:45:13.261 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:13.258+0000 7f4de89ef700 1 -- 172.21.15.38:0/3359801591 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f4dd4026090 con 0x7f4de4101120 2023-03-09T14:45:13.261 INFO:teuthology.orchestra.run.smithi038.stdout:214748364839 2023-03-09T14:45:13.263 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:13.260+0000 7f4de99f1700 1 -- 172.21.15.38:0/3359801591 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f4dcc09f400 msgr2=0x7f4dcc0a18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:13.263 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:13.260+0000 7f4de99f1700 1 --2- 172.21.15.38:0/3359801591 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f4dcc09f400 0x7f4dcc0a18b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f4dd8005950 tx=0x7f4dd80058e0 comp rx=0 tx=0).stop 2023-03-09T14:45:13.263 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:13.261+0000 7f4de99f1700 1 -- 172.21.15.38:0/3359801591 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4de4101120 msgr2=0x7f4de4196130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:13.263 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:13.261+0000 7f4de99f1700 1 --2- 172.21.15.38:0/3359801591 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4de4101120 0x7f4de4196130 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f4dd40095c0 tx=0x7f4dd400f690 comp rx=0 tx=0).stop 2023-03-09T14:45:13.263 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:13.261+0000 7f4de99f1700 1 -- 172.21.15.38:0/3359801591 shutdown_connections 2023-03-09T14:45:13.264 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:13.261+0000 7f4de99f1700 1 --2- 172.21.15.38:0/3359801591 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4de4101120 0x7f4de4196130 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:13.264 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:13.261+0000 7f4de99f1700 1 --2- 172.21.15.38:0/3359801591 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f4dcc09f400 0x7f4dcc0a18b0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:13.264 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:13.261+0000 7f4de99f1700 1 --2- 172.21.15.38:0/3359801591 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4de410a640 0x7f4de419aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:13.264 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:13.261+0000 7f4de99f1700 1 --2- 172.21.15.38:0/3359801591 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4de4101a30 0x7f4de4196670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:13.264 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:13.261+0000 7f4de99f1700 1 -- 172.21.15.38:0/3359801591 >> 172.21.15.38:0/3359801591 conn(0x7f4de4074cb0 msgr2=0x7f4de40fda90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:13.264 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:13.261+0000 7f4de99f1700 1 -- 172.21.15.38:0/3359801591 shutdown_connections 2023-03-09T14:45:13.264 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:13.261+0000 7f4de99f1700 1 -- 172.21.15.38:0/3359801591 wait complete. 2023-03-09T14:45:14.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:13 smithi038 ceph-mon[92422]: pgmap v286: 129 pgs: 129 active+clean; 636 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T14:45:14.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:13 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/3556004980' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-03-09T14:45:14.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:13 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/3359801591' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2023-03-09T14:45:14.018 INFO:tasks.cephadm.ceph_manager.ceph:need seq 244813135904 got 244813135906 for osd.8 2023-03-09T14:45:14.018 DEBUG:teuthology.parallel:result is None 2023-03-09T14:45:14.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:13 smithi122 ceph-mon[93869]: pgmap v286: 129 pgs: 129 active+clean; 636 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T14:45:14.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:13 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/3556004980' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-03-09T14:45:14.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:13 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/3359801591' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2023-03-09T14:45:14.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:13 smithi181 ceph-mon[94189]: pgmap v286: 129 pgs: 129 active+clean; 636 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T14:45:14.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:13 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/3556004980' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-03-09T14:45:14.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:13 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/3359801591' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2023-03-09T14:45:14.440 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:14.257+0000 7eff0e0f9700 1 -- 172.21.15.38:0/4152045401 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff08104160 msgr2=0x7eff0810bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:14.441 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:14.257+0000 7eff0e0f9700 1 --2- 172.21.15.38:0/4152045401 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff08104160 0x7eff0810bd30 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7eff0000b0d0 tx=0x7eff0000b3e0 comp rx=0 tx=0).stop 2023-03-09T14:45:14.441 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:14.257+0000 7eff0e0f9700 1 -- 172.21.15.38:0/4152045401 shutdown_connections 2023-03-09T14:45:14.441 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:14.257+0000 7eff0e0f9700 1 --2- 172.21.15.38:0/4152045401 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff08104160 0x7eff0810bd30 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:14.441 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:14.257+0000 7eff0e0f9700 1 --2- 172.21.15.38:0/4152045401 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff080ff120 0x7eff080ff4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:14.442 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:14.257+0000 7eff0e0f9700 1 --2- 172.21.15.38:0/4152045401 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7eff080ffac0 0x7eff08103a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:14.442 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:14.257+0000 7eff0e0f9700 1 -- 172.21.15.38:0/4152045401 >> 172.21.15.38:0/4152045401 conn(0x7eff08076d10 msgr2=0x7eff08077110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:14.442 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:14.257+0000 7eff0e0f9700 1 -- 172.21.15.38:0/4152045401 shutdown_connections 2023-03-09T14:45:14.442 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:14.257+0000 7eff0e0f9700 1 -- 172.21.15.38:0/4152045401 wait complete. 2023-03-09T14:45:14.442 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:14.257+0000 7eff0e0f9700 1 Processor -- start 2023-03-09T14:45:14.442 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:14.258+0000 7eff0e0f9700 1 -- start start 2023-03-09T14:45:14.442 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:14.258+0000 7eff0e0f9700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff080ff120 0x7eff08197030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:14.442 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:14.258+0000 7eff0e0f9700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff080ffac0 0x7eff08197570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:14.443 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:14.258+0000 7eff0e0f9700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7eff08104160 0x7eff081a20b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:14.443 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:14.258+0000 7eff0e0f9700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff08197e80 con 0x7eff080ffac0 2023-03-09T14:45:14.443 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:14.258+0000 7eff0e0f9700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff081a25f0 con 0x7eff08104160 2023-03-09T14:45:14.443 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:14.258+0000 7eff0e0f9700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff081a2760 con 0x7eff080ff120 2023-03-09T14:45:14.443 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:14.258+0000 7eff06ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff080ffac0 0x7eff08197570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:14.443 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:14.258+0000 7eff06ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff080ffac0 0x7eff08197570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37574/0 (socket says 172.21.15.38:37574) 2023-03-09T14:45:14.444 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:14.258+0000 7eff06ffd700 1 -- 172.21.15.38:0/3518725949 learned_addr learned my addr 172.21.15.38:0/3518725949 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:14.444 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:14.258+0000 7eff077fe700 1 --2- 172.21.15.38:0/3518725949 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff080ff120 0x7eff08197030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:14.444 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:14.258+0000 7eff07fff700 1 --2- 172.21.15.38:0/3518725949 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7eff08104160 0x7eff081a20b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:14.444 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:14.258+0000 7eff06ffd700 1 -- 172.21.15.38:0/3518725949 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff080ff120 msgr2=0x7eff08197030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:14.444 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:14.258+0000 7eff06ffd700 1 --2- 172.21.15.38:0/3518725949 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff080ff120 0x7eff08197030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:14.444 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:14.258+0000 7eff06ffd700 1 -- 172.21.15.38:0/3518725949 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7eff08104160 msgr2=0x7eff081a20b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:14.444 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:14.258+0000 7eff06ffd700 1 --2- 172.21.15.38:0/3518725949 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7eff08104160 0x7eff081a20b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:14.444 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:14.258+0000 7eff06ffd700 1 -- 172.21.15.38:0/3518725949 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff00009d00 con 0x7eff080ffac0 2023-03-09T14:45:14.445 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:14.258+0000 7eff07fff700 1 --2- 172.21.15.38:0/3518725949 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7eff08104160 0x7eff081a20b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:45:14.445 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:14.259+0000 7eff06ffd700 1 --2- 172.21.15.38:0/3518725949 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff080ffac0 0x7eff08197570 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7efefc00d950 tx=0x7efefc00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:14.445 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:14.259+0000 7eff04ff9700 1 -- 172.21.15.38:0/3518725949 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efefc009930 con 0x7eff080ffac0 2023-03-09T14:45:14.445 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:14.259+0000 7eff0e0f9700 1 -- 172.21.15.38:0/3518725949 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff0810ef20 con 0x7eff080ffac0 2023-03-09T14:45:14.445 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:14.259+0000 7eff04ff9700 1 -- 172.21.15.38:0/3518725949 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efefc010460 con 0x7eff080ffac0 2023-03-09T14:45:14.445 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:14.259+0000 7eff04ff9700 1 -- 172.21.15.38:0/3518725949 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efefc00f620 con 0x7eff080ffac0 2023-03-09T14:45:14.445 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:14.259+0000 7eff0e0f9700 1 -- 172.21.15.38:0/3518725949 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff0810ed90 con 0x7eff080ffac0 2023-03-09T14:45:14.446 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:14.260+0000 7eff0e0f9700 1 -- 172.21.15.38:0/3518725949 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff0810f0b0 con 0x7eff080ffac0 2023-03-09T14:45:14.446 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:14.261+0000 7eff04ff9700 1 -- 172.21.15.38:0/3518725949 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 133238+0+0 (secure 0 0 0) 0x7efefc023070 con 0x7eff080ffac0 2023-03-09T14:45:14.446 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:14.261+0000 7eff04ff9700 1 --2- 172.21.15.38:0/3518725949 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7efef009f400 0x7efef00a18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:14.446 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:14.262+0000 7eff077fe700 1 --2- 172.21.15.38:0/3518725949 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7efef009f400 0x7efef00a18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:14.446 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:14.262+0000 7eff04ff9700 1 -- 172.21.15.38:0/3518725949 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8730+0+0 (secure 0 0 0) 0x7efefc0c9aa0 con 0x7eff080ffac0 2023-03-09T14:45:14.446 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:14.262+0000 7eff077fe700 1 --2- 172.21.15.38:0/3518725949 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7efef009f400 0x7efef00a18b0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7efef8005950 tx=0x7efef800b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:14.446 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:14.266+0000 7eff04ff9700 1 -- 172.21.15.38:0/3518725949 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7efefc093bd0 con 0x7eff080ffac0 2023-03-09T14:45:14.465 INFO:tasks.cephadm.ceph_manager.ceph:need seq 214748364838 got 214748364839 for osd.7 2023-03-09T14:45:14.466 DEBUG:teuthology.parallel:result is None 2023-03-09T14:45:14.473 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:14.471+0000 7eff0e0f9700 1 -- 172.21.15.38:0/3518725949 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) v1 -- 0x7eff08066b10 con 0x7eff080ffac0 2023-03-09T14:45:14.473 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:14.471+0000 7eff04ff9700 1 -- 172.21.15.38:0/3518725949 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7efefc021020 con 0x7eff080ffac0 2023-03-09T14:45:14.474 INFO:teuthology.orchestra.run.smithi038.stdout:300647710745 2023-03-09T14:45:14.475 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:14.473+0000 7eff0e0f9700 1 -- 172.21.15.38:0/3518725949 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7efef009f400 msgr2=0x7efef00a18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:14.475 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:14.473+0000 7eff0e0f9700 1 --2- 172.21.15.38:0/3518725949 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7efef009f400 0x7efef00a18b0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7efef8005950 tx=0x7efef800b410 comp rx=0 tx=0).stop 2023-03-09T14:45:14.476 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:14.473+0000 7eff0e0f9700 1 -- 172.21.15.38:0/3518725949 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff080ffac0 msgr2=0x7eff08197570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:14.476 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:14.473+0000 7eff0e0f9700 1 --2- 172.21.15.38:0/3518725949 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff080ffac0 0x7eff08197570 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7efefc00d950 tx=0x7efefc00dc60 comp rx=0 tx=0).stop 2023-03-09T14:45:14.476 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:14.473+0000 7eff0e0f9700 1 -- 172.21.15.38:0/3518725949 shutdown_connections 2023-03-09T14:45:14.476 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:14.473+0000 7eff0e0f9700 1 --2- 172.21.15.38:0/3518725949 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7efef009f400 0x7efef00a18b0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:14.476 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:14.473+0000 7eff0e0f9700 1 --2- 172.21.15.38:0/3518725949 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7eff08104160 0x7eff081a20b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:14.476 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:14.473+0000 7eff0e0f9700 1 --2- 172.21.15.38:0/3518725949 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7eff080ff120 0x7eff08197030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:14.476 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:14.473+0000 7eff0e0f9700 1 --2- 172.21.15.38:0/3518725949 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7eff080ffac0 0x7eff08197570 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:14.477 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:14.473+0000 7eff0e0f9700 1 -- 172.21.15.38:0/3518725949 >> 172.21.15.38:0/3518725949 conn(0x7eff08076d10 msgr2=0x7eff080fc870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:14.477 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:14.473+0000 7eff0e0f9700 1 -- 172.21.15.38:0/3518725949 shutdown_connections 2023-03-09T14:45:14.477 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:14.473+0000 7eff0e0f9700 1 -- 172.21.15.38:0/3518725949 wait complete. 2023-03-09T14:45:14.916 INFO:tasks.cephadm.ceph_manager.ceph:need seq 158913789998 got 158913790000 for osd.5 2023-03-09T14:45:14.916 DEBUG:teuthology.parallel:result is None 2023-03-09T14:45:15.003 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:15.000+0000 7ff4a3ff5700 1 -- 172.21.15.38:0/2445180523 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff49c101b10 msgr2=0x7ff49c10a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:15.003 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:15.000+0000 7ff4a3ff5700 1 --2- 172.21.15.38:0/2445180523 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff49c101b10 0x7ff49c10a000 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7ff48c009990 tx=0x7ff48c009ca0 comp rx=0 tx=0).stop 2023-03-09T14:45:15.003 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:15.000+0000 7ff4a3ff5700 1 -- 172.21.15.38:0/2445180523 shutdown_connections 2023-03-09T14:45:15.004 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:15.000+0000 7ff4a3ff5700 1 --2- 172.21.15.38:0/2445180523 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff49c10a540 0x7ff49c10c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:15.004 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:15.000+0000 7ff4a3ff5700 1 --2- 172.21.15.38:0/2445180523 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff49c101200 0x7ff49c1015d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:15.004 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:15.000+0000 7ff4a3ff5700 1 --2- 172.21.15.38:0/2445180523 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff49c101b10 0x7ff49c10a000 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:15.004 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:15.000+0000 7ff4a3ff5700 1 -- 172.21.15.38:0/2445180523 >> 172.21.15.38:0/2445180523 conn(0x7ff49c076d10 msgr2=0x7ff49c077110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:15.005 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:15.001+0000 7ff4a3ff5700 1 -- 172.21.15.38:0/2445180523 shutdown_connections 2023-03-09T14:45:15.005 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:15.001+0000 7ff4a3ff5700 1 -- 172.21.15.38:0/2445180523 wait complete. 2023-03-09T14:45:15.006 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:15.001+0000 7ff4a3ff5700 1 Processor -- start 2023-03-09T14:45:15.007 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:15.001+0000 7ff4a3ff5700 1 -- start start 2023-03-09T14:45:15.007 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:15.002+0000 7ff4a3ff5700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff49c101200 0x7ff49c1961d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:15.007 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:15.002+0000 7ff4a3ff5700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff49c101b10 0x7ff49c196710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:15.007 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:15.002+0000 7ff4a3ff5700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff49c10a540 0x7ff49c19ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:15.008 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:15.002+0000 7ff4a3ff5700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff49c19b1b0 con 0x7ff49c101b10 2023-03-09T14:45:15.008 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:15.002+0000 7ff4a3ff5700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff49c19b320 con 0x7ff49c101200 2023-03-09T14:45:15.008 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:15.002+0000 7ff4a3ff5700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff49c19b4e0 con 0x7ff49c10a540 2023-03-09T14:45:15.008 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:15.002+0000 7ff4a1590700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff49c101b10 0x7ff49c196710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:15.008 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:15.002+0000 7ff4a1590700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff49c101b10 0x7ff49c196710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37588/0 (socket says 172.21.15.38:37588) 2023-03-09T14:45:15.008 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:15.002+0000 7ff4a1590700 1 -- 172.21.15.38:0/2851773136 learned_addr learned my addr 172.21.15.38:0/2851773136 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:15.008 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:15.002+0000 7ff4a1d91700 1 --2- 172.21.15.38:0/2851773136 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff49c101200 0x7ff49c1961d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:15.009 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:15.002+0000 7ff4a2592700 1 --2- 172.21.15.38:0/2851773136 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff49c10a540 0x7ff49c19ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:15.009 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:15.002+0000 7ff4a1590700 1 -- 172.21.15.38:0/2851773136 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff49c10a540 msgr2=0x7ff49c19ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:15.009 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:15.002+0000 7ff4a1590700 1 --2- 172.21.15.38:0/2851773136 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff49c10a540 0x7ff49c19ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:15.009 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:15.002+0000 7ff4a1590700 1 -- 172.21.15.38:0/2851773136 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff49c101200 msgr2=0x7ff49c1961d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:15.009 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:15.002+0000 7ff4a1590700 1 --2- 172.21.15.38:0/2851773136 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff49c101200 0x7ff49c1961d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:15.009 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:15.002+0000 7ff4a1590700 1 -- 172.21.15.38:0/2851773136 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff48c0095f0 con 0x7ff49c101b10 2023-03-09T14:45:15.009 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:15.002+0000 7ff4a1590700 1 --2- 172.21.15.38:0/2851773136 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff49c101b10 0x7ff49c196710 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7ff48c000c00 tx=0x7ff48c00f6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:15.010 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:15.003+0000 7ff492ffd700 1 -- 172.21.15.38:0/2851773136 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff48c01d070 con 0x7ff49c101b10 2023-03-09T14:45:15.010 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:15.003+0000 7ff492ffd700 1 -- 172.21.15.38:0/2851773136 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff48c00fb80 con 0x7ff49c101b10 2023-03-09T14:45:15.010 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:15.003+0000 7ff4a3ff5700 1 -- 172.21.15.38:0/2851773136 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff49c1100d0 con 0x7ff49c101b10 2023-03-09T14:45:15.010 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:15.003+0000 7ff492ffd700 1 -- 172.21.15.38:0/2851773136 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff48c017650 con 0x7ff49c101b10 2023-03-09T14:45:15.010 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:15.003+0000 7ff4a3ff5700 1 -- 172.21.15.38:0/2851773136 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff49c10ff40 con 0x7ff49c101b10 2023-03-09T14:45:15.010 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:15.004+0000 7ff4a3ff5700 1 -- 172.21.15.38:0/2851773136 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff49c110260 con 0x7ff49c101b10 2023-03-09T14:45:15.011 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:15.005+0000 7ff492ffd700 1 -- 172.21.15.38:0/2851773136 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 133238+0+0 (secure 0 0 0) 0x7ff48c029030 con 0x7ff49c101b10 2023-03-09T14:45:15.011 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:15.006+0000 7ff492ffd700 1 --2- 172.21.15.38:0/2851773136 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7ff48809f480 0x7ff4880a1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:15.011 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:15.006+0000 7ff492ffd700 1 -- 172.21.15.38:0/2851773136 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ff48c0cb830 con 0x7ff49c101b10 2023-03-09T14:45:15.012 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:15.006+0000 7ff4a1d91700 1 --2- 172.21.15.38:0/2851773136 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7ff48809f480 0x7ff4880a1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:15.012 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:15.007+0000 7ff4a1d91700 1 --2- 172.21.15.38:0/2851773136 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7ff48809f480 0x7ff4880a1930 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff498009a10 tx=0x7ff498009190 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:15.012 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:15.010+0000 7ff492ffd700 1 -- 172.21.15.38:0/2851773136 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7ff48c095930 con 0x7ff49c101b10 2023-03-09T14:45:15.129 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:14 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/3518725949' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2023-03-09T14:45:15.129 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:14 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.2", "id": [7, 11]}]: dispatch 2023-03-09T14:45:15.129 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:14 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]: dispatch 2023-03-09T14:45:15.129 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:14 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.d", "id": [5, 11]}]: dispatch 2023-03-09T14:45:15.129 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:14 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]: dispatch 2023-03-09T14:45:15.130 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:14 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.17", "id": [5, 6]}]: dispatch 2023-03-09T14:45:15.130 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:14 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]: dispatch 2023-03-09T14:45:15.130 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:14 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]: dispatch 2023-03-09T14:45:15.130 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:14 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]: dispatch 2023-03-09T14:45:15.130 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:14 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]: dispatch 2023-03-09T14:45:15.130 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:14 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]: dispatch 2023-03-09T14:45:15.218 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:15.215+0000 7ff4a3ff5700 1 -- 172.21.15.38:0/2851773136 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7ff49c1a3240 con 0x7ff49c101b10 2023-03-09T14:45:15.218 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:15.216+0000 7ff492ffd700 1 -- 172.21.15.38:0/2851773136 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff48c031020 con 0x7ff49c101b10 2023-03-09T14:45:15.218 INFO:teuthology.orchestra.run.smithi038.stdout:55834574912 2023-03-09T14:45:15.220 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:15.218+0000 7ff4a3ff5700 1 -- 172.21.15.38:0/2851773136 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7ff48809f480 msgr2=0x7ff4880a1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:15.220 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:15.218+0000 7ff4a3ff5700 1 --2- 172.21.15.38:0/2851773136 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7ff48809f480 0x7ff4880a1930 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff498009a10 tx=0x7ff498009190 comp rx=0 tx=0).stop 2023-03-09T14:45:15.220 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:15.218+0000 7ff4a3ff5700 1 -- 172.21.15.38:0/2851773136 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff49c101b10 msgr2=0x7ff49c196710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:15.220 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:15.218+0000 7ff4a3ff5700 1 --2- 172.21.15.38:0/2851773136 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff49c101b10 0x7ff49c196710 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7ff48c000c00 tx=0x7ff48c00f6e0 comp rx=0 tx=0).stop 2023-03-09T14:45:15.221 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:15.218+0000 7ff4a3ff5700 1 -- 172.21.15.38:0/2851773136 shutdown_connections 2023-03-09T14:45:15.221 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:15.218+0000 7ff4a3ff5700 1 --2- 172.21.15.38:0/2851773136 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7ff48809f480 0x7ff4880a1930 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:15.221 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:15.218+0000 7ff4a3ff5700 1 --2- 172.21.15.38:0/2851773136 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff49c10a540 0x7ff49c19ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:15.221 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:15.218+0000 7ff4a3ff5700 1 --2- 172.21.15.38:0/2851773136 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff49c101200 0x7ff49c1961d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:15.221 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:15.218+0000 7ff4a3ff5700 1 --2- 172.21.15.38:0/2851773136 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff49c101b10 0x7ff49c196710 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:15.221 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:15.218+0000 7ff4a3ff5700 1 -- 172.21.15.38:0/2851773136 >> 172.21.15.38:0/2851773136 conn(0x7ff49c076d10 msgr2=0x7ff49c0fb9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:15.221 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:15.218+0000 7ff4a3ff5700 1 -- 172.21.15.38:0/2851773136 shutdown_connections 2023-03-09T14:45:15.222 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:15.218+0000 7ff4a3ff5700 1 -- 172.21.15.38:0/2851773136 wait complete. 2023-03-09T14:45:15.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:14 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/3518725949' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2023-03-09T14:45:15.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:14 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.2", "id": [7, 11]}]: dispatch 2023-03-09T14:45:15.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:14 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]: dispatch 2023-03-09T14:45:15.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:14 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.d", "id": [5, 11]}]: dispatch 2023-03-09T14:45:15.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:14 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]: dispatch 2023-03-09T14:45:15.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:14 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.17", "id": [5, 6]}]: dispatch 2023-03-09T14:45:15.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:14 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]: dispatch 2023-03-09T14:45:15.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:14 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]: dispatch 2023-03-09T14:45:15.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:14 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]: dispatch 2023-03-09T14:45:15.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:14 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]: dispatch 2023-03-09T14:45:15.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:14 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]: dispatch 2023-03-09T14:45:15.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:14 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/3518725949' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2023-03-09T14:45:15.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:14 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.2", "id": [7, 11]}]: dispatch 2023-03-09T14:45:15.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:14 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]: dispatch 2023-03-09T14:45:15.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:14 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.d", "id": [5, 11]}]: dispatch 2023-03-09T14:45:15.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:14 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]: dispatch 2023-03-09T14:45:15.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:14 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.17", "id": [5, 6]}]: dispatch 2023-03-09T14:45:15.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:14 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]: dispatch 2023-03-09T14:45:15.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:14 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]: dispatch 2023-03-09T14:45:15.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:14 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]: dispatch 2023-03-09T14:45:15.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:14 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]: dispatch 2023-03-09T14:45:15.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:14 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]: dispatch 2023-03-09T14:45:15.940 INFO:tasks.cephadm.ceph_manager.ceph:need seq 193273528362 got 193273528364 for osd.6 2023-03-09T14:45:15.940 DEBUG:teuthology.parallel:result is None 2023-03-09T14:45:16.113 INFO:tasks.cephadm.ceph_manager.ceph:need seq 300647710744 got 300647710745 for osd.10 2023-03-09T14:45:16.114 DEBUG:teuthology.parallel:result is None 2023-03-09T14:45:16.205 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:15 smithi038 ceph-mon[92422]: pgmap v287: 129 pgs: 129 active+clean; 636 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-03-09T14:45:16.205 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:15 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.2", "id": [7, 11]}]': finished 2023-03-09T14:45:16.205 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:15 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]': finished 2023-03-09T14:45:16.205 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:15 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.d", "id": [5, 11]}]': finished 2023-03-09T14:45:16.206 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:15 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2023-03-09T14:45:16.206 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:15 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.17", "id": [5, 6]}]': finished 2023-03-09T14:45:16.206 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:15 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]': finished 2023-03-09T14:45:16.206 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:15 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]': finished 2023-03-09T14:45:16.206 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:15 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]': finished 2023-03-09T14:45:16.206 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:15 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]': finished 2023-03-09T14:45:16.206 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:15 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]': finished 2023-03-09T14:45:16.206 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:15 smithi038 ceph-mon[92422]: osdmap e88: 12 total, 12 up, 12 in 2023-03-09T14:45:16.207 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:15 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/2851773136' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-03-09T14:45:16.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:15 smithi122 ceph-mon[93869]: pgmap v287: 129 pgs: 129 active+clean; 636 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-03-09T14:45:16.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:15 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.2", "id": [7, 11]}]': finished 2023-03-09T14:45:16.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:15 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]': finished 2023-03-09T14:45:16.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:15 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.d", "id": [5, 11]}]': finished 2023-03-09T14:45:16.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:15 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2023-03-09T14:45:16.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:15 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.17", "id": [5, 6]}]': finished 2023-03-09T14:45:16.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:15 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]': finished 2023-03-09T14:45:16.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:15 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]': finished 2023-03-09T14:45:16.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:15 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]': finished 2023-03-09T14:45:16.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:15 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]': finished 2023-03-09T14:45:16.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:15 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]': finished 2023-03-09T14:45:16.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:15 smithi122 ceph-mon[93869]: osdmap e88: 12 total, 12 up, 12 in 2023-03-09T14:45:16.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:15 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/2851773136' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-03-09T14:45:16.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:15 smithi181 ceph-mon[94189]: pgmap v287: 129 pgs: 129 active+clean; 636 KiB data, 80 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-03-09T14:45:16.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:15 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.2", "id": [7, 11]}]': finished 2023-03-09T14:45:16.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:15 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]': finished 2023-03-09T14:45:16.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:15 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.d", "id": [5, 11]}]': finished 2023-03-09T14:45:16.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:15 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2023-03-09T14:45:16.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:15 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.17", "id": [5, 6]}]': finished 2023-03-09T14:45:16.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:15 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]': finished 2023-03-09T14:45:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:15 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]': finished 2023-03-09T14:45:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:15 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]': finished 2023-03-09T14:45:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:15 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.c", "id": [9, 10]}]': finished 2023-03-09T14:45:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:15 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]': finished 2023-03-09T14:45:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:15 smithi181 ceph-mon[94189]: osdmap e88: 12 total, 12 up, 12 in 2023-03-09T14:45:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:15 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/2851773136' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-03-09T14:45:16.333 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.a/config 2023-03-09T14:45:16.358 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411387 got 77309411389 for osd.2 2023-03-09T14:45:16.359 DEBUG:teuthology.parallel:result is None 2023-03-09T14:45:16.448 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574909 got 55834574912 for osd.1 2023-03-09T14:45:16.449 DEBUG:teuthology.parallel:result is None 2023-03-09T14:45:16.691 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.637+0000 7ff720510700 1 -- 172.21.15.38:0/2369519272 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7180ff650 msgr2=0x7ff7180ffac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:16.691 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.637+0000 7ff720510700 1 --2- 172.21.15.38:0/2369519272 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7180ff650 0x7ff7180ffac0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7ff7080099c0 tx=0x7ff708009cd0 comp rx=0 tx=0).stop 2023-03-09T14:45:16.691 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.638+0000 7ff720510700 1 -- 172.21.15.38:0/2369519272 shutdown_connections 2023-03-09T14:45:16.691 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.638+0000 7ff720510700 1 --2- 172.21.15.38:0/2369519272 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff718100000 0x7ff71810bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:16.691 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.638+0000 7ff720510700 1 --2- 172.21.15.38:0/2369519272 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff718103e20 0x7ff7181041f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:16.692 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.638+0000 7ff720510700 1 --2- 172.21.15.38:0/2369519272 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff7180ff650 0x7ff7180ffac0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:16.692 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.638+0000 7ff720510700 1 -- 172.21.15.38:0/2369519272 >> 172.21.15.38:0/2369519272 conn(0x7ff718075580 msgr2=0x7ff718075980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:16.692 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.638+0000 7ff720510700 1 -- 172.21.15.38:0/2369519272 shutdown_connections 2023-03-09T14:45:16.692 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.638+0000 7ff720510700 1 -- 172.21.15.38:0/2369519272 wait complete. 2023-03-09T14:45:16.692 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.639+0000 7ff720510700 1 Processor -- start 2023-03-09T14:45:16.692 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.639+0000 7ff720510700 1 -- start start 2023-03-09T14:45:16.692 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.639+0000 7ff720510700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff7180ff650 0x7ff718196070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:16.693 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.639+0000 7ff720510700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff718100000 0x7ff7181965b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:16.693 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.639+0000 7ff720510700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff718103e20 0x7ff71819aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:16.693 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.639+0000 7ff720510700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff71819b050 con 0x7ff718100000 2023-03-09T14:45:16.693 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.639+0000 7ff720510700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff71819b1c0 con 0x7ff7180ff650 2023-03-09T14:45:16.693 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.639+0000 7ff720510700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff71819b380 con 0x7ff718103e20 2023-03-09T14:45:16.693 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.639+0000 7ff71daab700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff718100000 0x7ff7181965b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:16.693 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.639+0000 7ff71daab700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff718100000 0x7ff7181965b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37600/0 (socket says 172.21.15.38:37600) 2023-03-09T14:45:16.693 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.639+0000 7ff71daab700 1 -- 172.21.15.38:0/1358972204 learned_addr learned my addr 172.21.15.38:0/1358972204 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:16.694 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.639+0000 7ff71eaad700 1 --2- 172.21.15.38:0/1358972204 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff718103e20 0x7ff71819aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:16.694 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.639+0000 7ff71e2ac700 1 --2- 172.21.15.38:0/1358972204 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff7180ff650 0x7ff718196070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:16.694 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.639+0000 7ff71daab700 1 -- 172.21.15.38:0/1358972204 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff718103e20 msgr2=0x7ff71819aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:16.694 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.640+0000 7ff71daab700 1 --2- 172.21.15.38:0/1358972204 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff718103e20 0x7ff71819aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:16.694 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.640+0000 7ff71daab700 1 -- 172.21.15.38:0/1358972204 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff7180ff650 msgr2=0x7ff718196070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:16.694 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.640+0000 7ff71daab700 1 --2- 172.21.15.38:0/1358972204 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff7180ff650 0x7ff718196070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:16.694 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.640+0000 7ff71daab700 1 -- 172.21.15.38:0/1358972204 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7080095f0 con 0x7ff718100000 2023-03-09T14:45:16.695 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.640+0000 7ff71e2ac700 1 --2- 172.21.15.38:0/1358972204 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff7180ff650 0x7ff718196070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:45:16.695 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.640+0000 7ff71daab700 1 --2- 172.21.15.38:0/1358972204 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff718100000 0x7ff7181965b0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7ff708009990 tx=0x7ff708009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:16.695 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.640+0000 7ff70f7fe700 1 -- 172.21.15.38:0/1358972204 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff70801d070 con 0x7ff718100000 2023-03-09T14:45:16.695 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.640+0000 7ff70f7fe700 1 -- 172.21.15.38:0/1358972204 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff708003f90 con 0x7ff718100000 2023-03-09T14:45:16.695 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.640+0000 7ff720510700 1 -- 172.21.15.38:0/1358972204 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff71810efd0 con 0x7ff718100000 2023-03-09T14:45:16.695 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.640+0000 7ff70f7fe700 1 -- 172.21.15.38:0/1358972204 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff708022ce0 con 0x7ff718100000 2023-03-09T14:45:16.695 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.640+0000 7ff720510700 1 -- 172.21.15.38:0/1358972204 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff71810ee40 con 0x7ff718100000 2023-03-09T14:45:16.696 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.641+0000 7ff720510700 1 -- 172.21.15.38:0/1358972204 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff71810f160 con 0x7ff718100000 2023-03-09T14:45:16.696 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.642+0000 7ff70f7fe700 1 -- 172.21.15.38:0/1358972204 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 133238+0+0 (secure 0 0 0) 0x7ff708029030 con 0x7ff718100000 2023-03-09T14:45:16.696 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.643+0000 7ff70f7fe700 1 --2- 172.21.15.38:0/1358972204 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7ff70409f480 0x7ff7040a1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:16.696 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.643+0000 7ff71e2ac700 1 --2- 172.21.15.38:0/1358972204 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7ff70409f480 0x7ff7040a1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:16.696 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.643+0000 7ff70f7fe700 1 -- 172.21.15.38:0/1358972204 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ff7080cb1f0 con 0x7ff718100000 2023-03-09T14:45:16.696 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.643+0000 7ff71e2ac700 1 --2- 172.21.15.38:0/1358972204 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7ff70409f480 0x7ff7040a1930 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff714005950 tx=0x7ff7140058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:16.696 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.647+0000 7ff70f7fe700 1 -- 172.21.15.38:0/1358972204 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7ff7080952f0 con 0x7ff718100000 2023-03-09T14:45:16.748 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.746+0000 7f03557dc700 1 -- 172.21.15.38:0/1619849940 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0350100ba0 msgr2=0x7f0350104b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:16.749 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.746+0000 7f03557dc700 1 --2- 172.21.15.38:0/1619849940 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0350100ba0 0x7f0350104b60 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f03440099c0 tx=0x7f0344009cd0 comp rx=0 tx=0).stop 2023-03-09T14:45:16.751 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.746+0000 7f03557dc700 1 -- 172.21.15.38:0/1619849940 shutdown_connections 2023-03-09T14:45:16.751 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.746+0000 7f03557dc700 1 --2- 172.21.15.38:0/1619849940 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0350105240 0x7f035010ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:16.751 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.746+0000 7f03557dc700 1 --2- 172.21.15.38:0/1619849940 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0350100200 0x7f03501005d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:16.751 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.746+0000 7f03557dc700 1 --2- 172.21.15.38:0/1619849940 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0350100ba0 0x7f0350104b60 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:16.751 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.746+0000 7f03557dc700 1 -- 172.21.15.38:0/1619849940 >> 172.21.15.38:0/1619849940 conn(0x7f03500fbab0 msgr2=0x7f03500fdec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:16.752 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.746+0000 7f03557dc700 1 -- 172.21.15.38:0/1619849940 shutdown_connections 2023-03-09T14:45:16.752 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.746+0000 7f03557dc700 1 -- 172.21.15.38:0/1619849940 wait complete. 2023-03-09T14:45:16.752 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.747+0000 7f03557dc700 1 Processor -- start 2023-03-09T14:45:16.752 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.747+0000 7f03557dc700 1 -- start start 2023-03-09T14:45:16.752 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.747+0000 7f03557dc700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0350100200 0x7f0350196150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:16.752 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.747+0000 7f03557dc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0350100ba0 0x7f0350196690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:16.752 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.747+0000 7f03557dc700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0350105240 0x7f035019aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:16.753 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.747+0000 7f03557dc700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f035019b130 con 0x7f0350100ba0 2023-03-09T14:45:16.753 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.747+0000 7f03557dc700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f035019b2a0 con 0x7f0350105240 2023-03-09T14:45:16.753 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.747+0000 7f03557dc700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f035019b460 con 0x7f0350100200 2023-03-09T14:45:16.753 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.747+0000 7f034e7fc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0350100ba0 0x7f0350196690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:16.753 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.747+0000 7f034e7fc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0350100ba0 0x7f0350196690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37614/0 (socket says 172.21.15.38:37614) 2023-03-09T14:45:16.753 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.747+0000 7f034e7fc700 1 -- 172.21.15.38:0/3290090961 learned_addr learned my addr 172.21.15.38:0/3290090961 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:16.753 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.748+0000 7f034effd700 1 --2- 172.21.15.38:0/3290090961 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0350100200 0x7f0350196150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:16.754 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.748+0000 7f034f7fe700 1 --2- 172.21.15.38:0/3290090961 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0350105240 0x7f035019aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:16.754 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.748+0000 7f034e7fc700 1 -- 172.21.15.38:0/3290090961 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0350100200 msgr2=0x7f0350196150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:16.754 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.748+0000 7f034e7fc700 1 --2- 172.21.15.38:0/3290090961 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0350100200 0x7f0350196150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:16.754 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.748+0000 7f034e7fc700 1 -- 172.21.15.38:0/3290090961 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0350105240 msgr2=0x7f035019aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:16.754 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.748+0000 7f034e7fc700 1 --2- 172.21.15.38:0/3290090961 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0350105240 0x7f035019aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:16.754 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.748+0000 7f034e7fc700 1 -- 172.21.15.38:0/3290090961 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03440095f0 con 0x7f0350100ba0 2023-03-09T14:45:16.754 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.748+0000 7f034e7fc700 1 --2- 172.21.15.38:0/3290090961 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0350100ba0 0x7f0350196690 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f0344009fd0 tx=0x7f0344009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:16.755 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.748+0000 7f0337fff700 1 -- 172.21.15.38:0/3290090961 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f034401d070 con 0x7f0350100ba0 2023-03-09T14:45:16.755 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.748+0000 7f03557dc700 1 -- 172.21.15.38:0/3290090961 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f035010ffc0 con 0x7f0350100ba0 2023-03-09T14:45:16.755 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.748+0000 7f0337fff700 1 -- 172.21.15.38:0/3290090961 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0344005020 con 0x7f0350100ba0 2023-03-09T14:45:16.755 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.748+0000 7f0337fff700 1 -- 172.21.15.38:0/3290090961 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0344004aa0 con 0x7f0350100ba0 2023-03-09T14:45:16.755 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.748+0000 7f03557dc700 1 -- 172.21.15.38:0/3290090961 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f035010fe30 con 0x7f0350100ba0 2023-03-09T14:45:16.756 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.750+0000 7f03557dc700 1 -- 172.21.15.38:0/3290090961 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0350110150 con 0x7f0350100ba0 2023-03-09T14:45:16.756 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.751+0000 7f0337fff700 1 -- 172.21.15.38:0/3290090961 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 133238+0+0 (secure 0 0 0) 0x7f034400c040 con 0x7f0350100ba0 2023-03-09T14:45:16.756 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.752+0000 7f0337fff700 1 --2- 172.21.15.38:0/3290090961 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f03380a3810 0x7f03380a5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:16.756 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.752+0000 7f034effd700 1 --2- 172.21.15.38:0/3290090961 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f03380a3810 0x7f03380a5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:16.757 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.752+0000 7f0337fff700 1 -- 172.21.15.38:0/3290090961 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f03440cabc0 con 0x7f0350100ba0 2023-03-09T14:45:16.757 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.752+0000 7f034effd700 1 --2- 172.21.15.38:0/3290090961 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f03380a3810 0x7f03380a5cc0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f0340005950 tx=0x7f034000b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:16.758 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.756+0000 7f0337fff700 1 -- 172.21.15.38:0/3290090961 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f0344094c60 con 0x7f0350100ba0 2023-03-09T14:45:16.877 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.875+0000 7ff720510700 1 -- 172.21.15.38:0/1358972204 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) v1 -- 0x7ff7181973c0 con 0x7ff718100000 2023-03-09T14:45:16.881 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.879+0000 7ff70f7fe700 1 -- 172.21.15.38:0/1358972204 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7ff708098910 con 0x7ff718100000 2023-03-09T14:45:16.881 INFO:teuthology.orchestra.run.smithi038.stdout:335007449111 2023-03-09T14:45:16.883 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.881+0000 7ff720510700 1 -- 172.21.15.38:0/1358972204 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7ff70409f480 msgr2=0x7ff7040a1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:16.884 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.881+0000 7ff720510700 1 --2- 172.21.15.38:0/1358972204 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7ff70409f480 0x7ff7040a1930 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff714005950 tx=0x7ff7140058e0 comp rx=0 tx=0).stop 2023-03-09T14:45:16.884 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.881+0000 7ff720510700 1 -- 172.21.15.38:0/1358972204 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff718100000 msgr2=0x7ff7181965b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:16.884 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.881+0000 7ff720510700 1 --2- 172.21.15.38:0/1358972204 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff718100000 0x7ff7181965b0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7ff708009990 tx=0x7ff708009580 comp rx=0 tx=0).stop 2023-03-09T14:45:16.884 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.881+0000 7ff720510700 1 -- 172.21.15.38:0/1358972204 shutdown_connections 2023-03-09T14:45:16.884 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.881+0000 7ff720510700 1 --2- 172.21.15.38:0/1358972204 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7ff70409f480 0x7ff7040a1930 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:16.884 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.881+0000 7ff720510700 1 --2- 172.21.15.38:0/1358972204 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff718103e20 0x7ff71819aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:16.885 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.881+0000 7ff720510700 1 --2- 172.21.15.38:0/1358972204 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff7180ff650 0x7ff718196070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:16.885 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.881+0000 7ff720510700 1 --2- 172.21.15.38:0/1358972204 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff718100000 0x7ff7181965b0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:16.885 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.881+0000 7ff720510700 1 -- 172.21.15.38:0/1358972204 >> 172.21.15.38:0/1358972204 conn(0x7ff718075580 msgr2=0x7ff718102c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:16.885 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.881+0000 7ff720510700 1 -- 172.21.15.38:0/1358972204 shutdown_connections 2023-03-09T14:45:16.885 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.881+0000 7ff720510700 1 -- 172.21.15.38:0/1358972204 wait complete. 2023-03-09T14:45:16.981 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.979+0000 7f03557dc700 1 -- 172.21.15.38:0/3290090961 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f03501974a0 con 0x7f0350100ba0 2023-03-09T14:45:16.982 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.979+0000 7f0337fff700 1 -- 172.21.15.38:0/3290090961 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0344098280 con 0x7f0350100ba0 2023-03-09T14:45:16.982 INFO:teuthology.orchestra.run.smithi038.stdout:34359738439 2023-03-09T14:45:16.984 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.981+0000 7f03557dc700 1 -- 172.21.15.38:0/3290090961 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f03380a3810 msgr2=0x7f03380a5cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:16.984 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.981+0000 7f03557dc700 1 --2- 172.21.15.38:0/3290090961 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f03380a3810 0x7f03380a5cc0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f0340005950 tx=0x7f034000b410 comp rx=0 tx=0).stop 2023-03-09T14:45:16.984 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.982+0000 7f03557dc700 1 -- 172.21.15.38:0/3290090961 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0350100ba0 msgr2=0x7f0350196690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:16.984 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.982+0000 7f03557dc700 1 --2- 172.21.15.38:0/3290090961 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0350100ba0 0x7f0350196690 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f0344009fd0 tx=0x7f0344009580 comp rx=0 tx=0).stop 2023-03-09T14:45:16.984 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.982+0000 7f03557dc700 1 -- 172.21.15.38:0/3290090961 shutdown_connections 2023-03-09T14:45:16.985 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.982+0000 7f03557dc700 1 --2- 172.21.15.38:0/3290090961 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f03380a3810 0x7f03380a5cc0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:16.985 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.982+0000 7f03557dc700 1 --2- 172.21.15.38:0/3290090961 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0350105240 0x7f035019aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:16.985 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.982+0000 7f03557dc700 1 --2- 172.21.15.38:0/3290090961 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0350100200 0x7f0350196150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:16.985 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.982+0000 7f03557dc700 1 --2- 172.21.15.38:0/3290090961 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0350100ba0 0x7f0350196690 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:16.985 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.982+0000 7f03557dc700 1 -- 172.21.15.38:0/3290090961 >> 172.21.15.38:0/3290090961 conn(0x7f03500fbab0 msgr2=0x7f03500fc510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:16.985 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.982+0000 7f03557dc700 1 -- 172.21.15.38:0/3290090961 shutdown_connections 2023-03-09T14:45:16.985 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:16.982+0000 7f03557dc700 1 -- 172.21.15.38:0/3290090961 wait complete. 2023-03-09T14:45:17.059 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:16 smithi038 ceph-mon[92422]: osdmap e89: 12 total, 12 up, 12 in 2023-03-09T14:45:17.181 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.178+0000 7f771726f700 1 -- 172.21.15.38:0/1071982282 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7710100200 msgr2=0x7f77101005d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:17.181 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.178+0000 7f771726f700 1 --2- 172.21.15.38:0/1071982282 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7710100200 0x7f77101005d0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f770c0099c0 tx=0x7f770c009cd0 comp rx=0 tx=0).stop 2023-03-09T14:45:17.182 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.178+0000 7f771726f700 1 -- 172.21.15.38:0/1071982282 shutdown_connections 2023-03-09T14:45:17.182 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.178+0000 7f771726f700 1 --2- 172.21.15.38:0/1071982282 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7710100200 0x7f77101005d0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:17.182 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.178+0000 7f771726f700 1 --2- 172.21.15.38:0/1071982282 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7710105240 0x7f771010ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:17.182 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.178+0000 7f771726f700 1 --2- 172.21.15.38:0/1071982282 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7710100ba0 0x7f7710104b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:17.182 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.178+0000 7f771726f700 1 -- 172.21.15.38:0/1071982282 >> 172.21.15.38:0/1071982282 conn(0x7f77100fbab0 msgr2=0x7f77100fdec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:17.182 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.179+0000 7f771726f700 1 -- 172.21.15.38:0/1071982282 shutdown_connections 2023-03-09T14:45:17.183 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.179+0000 7f771726f700 1 -- 172.21.15.38:0/1071982282 wait complete. 2023-03-09T14:45:17.184 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.179+0000 7f771726f700 1 Processor -- start 2023-03-09T14:45:17.184 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.179+0000 7f771726f700 1 -- start start 2023-03-09T14:45:17.184 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.180+0000 7f771726f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7710100200 0x7f7710072a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:17.185 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.180+0000 7f771726f700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7710100ba0 0x7f7710072fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:17.185 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.180+0000 7f771726f700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7710105240 0x7f77101a4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:17.185 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.180+0000 7f771726f700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77100737f0 con 0x7f7710100200 2023-03-09T14:45:17.185 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.180+0000 7f771726f700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77101a4840 con 0x7f7710105240 2023-03-09T14:45:17.185 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.180+0000 7f771726f700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77101a4980 con 0x7f7710100ba0 2023-03-09T14:45:17.186 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.180+0000 7f771500b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7710100200 0x7f7710072a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:17.186 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.180+0000 7f771500b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7710100200 0x7f7710072a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37624/0 (socket says 172.21.15.38:37624) 2023-03-09T14:45:17.186 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.180+0000 7f771500b700 1 -- 172.21.15.38:0/3913513324 learned_addr learned my addr 172.21.15.38:0/3913513324 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:17.186 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.180+0000 7f771580c700 1 --2- 172.21.15.38:0/3913513324 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7710105240 0x7f77101a4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:17.186 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.180+0000 7f771480a700 1 --2- 172.21.15.38:0/3913513324 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7710100ba0 0x7f7710072fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:17.186 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.180+0000 7f771500b700 1 -- 172.21.15.38:0/3913513324 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7710100ba0 msgr2=0x7f7710072fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:17.187 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.180+0000 7f771500b700 1 --2- 172.21.15.38:0/3913513324 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7710100ba0 0x7f7710072fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:17.187 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.180+0000 7f771500b700 1 -- 172.21.15.38:0/3913513324 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7710105240 msgr2=0x7f77101a4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:17.187 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.180+0000 7f771500b700 1 --2- 172.21.15.38:0/3913513324 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7710105240 0x7f77101a4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:17.187 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.180+0000 7f771500b700 1 -- 172.21.15.38:0/3913513324 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f770c0095f0 con 0x7f7710100200 2023-03-09T14:45:17.187 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.180+0000 7f771480a700 1 --2- 172.21.15.38:0/3913513324 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7710100ba0 0x7f7710072fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:45:17.187 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.181+0000 7f771500b700 1 --2- 172.21.15.38:0/3913513324 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7710100200 0x7f7710072a80 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f770c009990 tx=0x7f770c009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:17.187 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.181+0000 7f77067fc700 1 -- 172.21.15.38:0/3913513324 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f770c01d070 con 0x7f7710100200 2023-03-09T14:45:17.187 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.181+0000 7f77067fc700 1 -- 172.21.15.38:0/3913513324 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f770c003f90 con 0x7f7710100200 2023-03-09T14:45:17.188 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.181+0000 7f77067fc700 1 -- 172.21.15.38:0/3913513324 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f770c022ce0 con 0x7f7710100200 2023-03-09T14:45:17.188 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.181+0000 7f771726f700 1 -- 172.21.15.38:0/3913513324 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7710110000 con 0x7f7710100200 2023-03-09T14:45:17.188 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.181+0000 7f771726f700 1 -- 172.21.15.38:0/3913513324 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f771010fe70 con 0x7f7710100200 2023-03-09T14:45:17.188 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.182+0000 7f771726f700 1 -- 172.21.15.38:0/3913513324 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7710110190 con 0x7f7710100200 2023-03-09T14:45:17.189 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.183+0000 7f77067fc700 1 -- 172.21.15.38:0/3913513324 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 133238+0+0 (secure 0 0 0) 0x7f770c029030 con 0x7f7710100200 2023-03-09T14:45:17.189 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.184+0000 7f77067fc700 1 --2- 172.21.15.38:0/3913513324 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f76fc09f480 0x7f76fc0a1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:17.189 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.184+0000 7f771480a700 1 --2- 172.21.15.38:0/3913513324 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f76fc09f480 0x7f76fc0a1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:17.189 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.184+0000 7f77067fc700 1 -- 172.21.15.38:0/3913513324 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f770c0cb4f0 con 0x7f7710100200 2023-03-09T14:45:17.189 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.185+0000 7f771480a700 1 --2- 172.21.15.38:0/3913513324 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f76fc09f480 0x7f76fc0a1930 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7700005950 tx=0x7f77000058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:17.190 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.188+0000 7f77067fc700 1 -- 172.21.15.38:0/3913513324 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f770c0955f0 con 0x7f7710100200 2023-03-09T14:45:17.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:16 smithi122 ceph-mon[93869]: osdmap e89: 12 total, 12 up, 12 in 2023-03-09T14:45:17.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:16 smithi181 ceph-mon[94189]: osdmap e89: 12 total, 12 up, 12 in 2023-03-09T14:45:17.270 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.268+0000 7f8227941700 1 -- 172.21.15.38:0/3392723982 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f82200ffac0 msgr2=0x7f8220103a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:17.271 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.268+0000 7f8227941700 1 --2- 172.21.15.38:0/3392723982 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f82200ffac0 0x7f8220103a80 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f82100099c0 tx=0x7f8210009cd0 comp rx=0 tx=0).stop 2023-03-09T14:45:17.271 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.268+0000 7f8227941700 1 -- 172.21.15.38:0/3392723982 shutdown_connections 2023-03-09T14:45:17.271 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.268+0000 7f8227941700 1 --2- 172.21.15.38:0/3392723982 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8220104160 0x7f822010bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:17.271 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.268+0000 7f8227941700 1 --2- 172.21.15.38:0/3392723982 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f82200ff120 0x7f82200ff4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:17.271 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.268+0000 7f8227941700 1 --2- 172.21.15.38:0/3392723982 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f82200ffac0 0x7f8220103a80 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:17.271 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.268+0000 7f8227941700 1 -- 172.21.15.38:0/3392723982 >> 172.21.15.38:0/3392723982 conn(0x7f8220076d10 msgr2=0x7f8220077110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:17.272 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.268+0000 7f8227941700 1 -- 172.21.15.38:0/3392723982 shutdown_connections 2023-03-09T14:45:17.272 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.268+0000 7f8227941700 1 -- 172.21.15.38:0/3392723982 wait complete. 2023-03-09T14:45:17.273 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.269+0000 7f8227941700 1 Processor -- start 2023-03-09T14:45:17.274 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.269+0000 7f8227941700 1 -- start start 2023-03-09T14:45:17.274 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.269+0000 7f8227941700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f82200ff120 0x7f8220198300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:17.274 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.269+0000 7f8227941700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f82200ffac0 0x7f8220198840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:17.274 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.269+0000 7f8227941700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8220104160 0x7f822019cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:17.274 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.269+0000 7f8227941700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f822019d2e0 con 0x7f82200ffac0 2023-03-09T14:45:17.274 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.269+0000 7f8227941700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f822019d450 con 0x7f8220104160 2023-03-09T14:45:17.275 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.269+0000 7f8227941700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f822019d610 con 0x7f82200ff120 2023-03-09T14:45:17.275 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.269+0000 7f8224edc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f82200ffac0 0x7f8220198840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:17.275 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.269+0000 7f8224edc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f82200ffac0 0x7f8220198840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37640/0 (socket says 172.21.15.38:37640) 2023-03-09T14:45:17.275 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.269+0000 7f8224edc700 1 -- 172.21.15.38:0/2585240206 learned_addr learned my addr 172.21.15.38:0/2585240206 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:17.275 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.269+0000 7f82256dd700 1 --2- 172.21.15.38:0/2585240206 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f82200ff120 0x7f8220198300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:17.276 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.269+0000 7f8224edc700 1 -- 172.21.15.38:0/2585240206 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f82200ff120 msgr2=0x7f8220198300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:17.276 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.269+0000 7f8224edc700 1 --2- 172.21.15.38:0/2585240206 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f82200ff120 0x7f8220198300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:17.276 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.269+0000 7f8224edc700 1 -- 172.21.15.38:0/2585240206 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8220104160 msgr2=0x7f822019cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:17.276 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.269+0000 7f8225ede700 1 --2- 172.21.15.38:0/2585240206 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8220104160 0x7f822019cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:17.276 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.269+0000 7f8224edc700 1 --2- 172.21.15.38:0/2585240206 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8220104160 0x7f822019cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:17.276 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.269+0000 7f8224edc700 1 -- 172.21.15.38:0/2585240206 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82100095f0 con 0x7f82200ffac0 2023-03-09T14:45:17.277 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.269+0000 7f82256dd700 1 --2- 172.21.15.38:0/2585240206 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f82200ff120 0x7f8220198300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-09T14:45:17.277 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.270+0000 7f8224edc700 1 --2- 172.21.15.38:0/2585240206 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f82200ffac0 0x7f8220198840 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f8210009990 tx=0x7f821000f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:17.277 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.270+0000 7f82167fc700 1 -- 172.21.15.38:0/2585240206 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f821001d070 con 0x7f82200ffac0 2023-03-09T14:45:17.277 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.270+0000 7f8227941700 1 -- 172.21.15.38:0/2585240206 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f822010ef30 con 0x7f82200ffac0 2023-03-09T14:45:17.277 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.270+0000 7f82167fc700 1 -- 172.21.15.38:0/2585240206 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f82100054e0 con 0x7f82200ffac0 2023-03-09T14:45:17.277 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.270+0000 7f82167fc700 1 -- 172.21.15.38:0/2585240206 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8210004b70 con 0x7f82200ffac0 2023-03-09T14:45:17.277 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.270+0000 7f8227941700 1 -- 172.21.15.38:0/2585240206 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f822010eda0 con 0x7f82200ffac0 2023-03-09T14:45:17.278 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.271+0000 7f8227941700 1 -- 172.21.15.38:0/2585240206 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f822010f0c0 con 0x7f82200ffac0 2023-03-09T14:45:17.278 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.272+0000 7f82167fc700 1 -- 172.21.15.38:0/2585240206 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 133238+0+0 (secure 0 0 0) 0x7f821000c040 con 0x7f82200ffac0 2023-03-09T14:45:17.278 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.273+0000 7f82167fc700 1 --2- 172.21.15.38:0/2585240206 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f820c09f400 0x7f820c0a18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:17.279 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.273+0000 7f82256dd700 1 --2- 172.21.15.38:0/2585240206 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f820c09f400 0x7f820c0a18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:17.279 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.273+0000 7f82167fc700 1 -- 172.21.15.38:0/2585240206 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f82100ca7e0 con 0x7f82200ffac0 2023-03-09T14:45:17.279 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.274+0000 7f82256dd700 1 --2- 172.21.15.38:0/2585240206 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f820c09f400 0x7f820c0a18b0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f821c009be0 tx=0x7f821c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:17.279 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.277+0000 7f82167fc700 1 -- 172.21.15.38:0/2585240206 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f82100948e0 con 0x7f82200ffac0 2023-03-09T14:45:17.393 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.391+0000 7f771726f700 1 -- 172.21.15.38:0/3913513324 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) v1 -- 0x7f7710066b10 con 0x7f7710100200 2023-03-09T14:45:17.394 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.391+0000 7f77067fc700 1 -- 172.21.15.38:0/3913513324 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f770c098c10 con 0x7f7710100200 2023-03-09T14:45:17.394 INFO:teuthology.orchestra.run.smithi038.stdout:279172874271 2023-03-09T14:45:17.396 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.393+0000 7f771726f700 1 -- 172.21.15.38:0/3913513324 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f76fc09f480 msgr2=0x7f76fc0a1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:17.396 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.393+0000 7f771726f700 1 --2- 172.21.15.38:0/3913513324 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f76fc09f480 0x7f76fc0a1930 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f7700005950 tx=0x7f77000058e0 comp rx=0 tx=0).stop 2023-03-09T14:45:17.396 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.393+0000 7f771726f700 1 -- 172.21.15.38:0/3913513324 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7710100200 msgr2=0x7f7710072a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:17.396 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.393+0000 7f771726f700 1 --2- 172.21.15.38:0/3913513324 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7710100200 0x7f7710072a80 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f770c009990 tx=0x7f770c009580 comp rx=0 tx=0).stop 2023-03-09T14:45:17.397 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.393+0000 7f771726f700 1 -- 172.21.15.38:0/3913513324 shutdown_connections 2023-03-09T14:45:17.397 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.393+0000 7f771726f700 1 --2- 172.21.15.38:0/3913513324 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f7710100200 0x7f7710072a80 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:17.397 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.393+0000 7f771726f700 1 --2- 172.21.15.38:0/3913513324 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f76fc09f480 0x7f76fc0a1930 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:17.397 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.393+0000 7f771726f700 1 --2- 172.21.15.38:0/3913513324 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f7710105240 0x7f77101a4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:17.398 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.393+0000 7f771726f700 1 --2- 172.21.15.38:0/3913513324 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f7710100ba0 0x7f7710072fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:17.398 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.393+0000 7f771726f700 1 -- 172.21.15.38:0/3913513324 >> 172.21.15.38:0/3913513324 conn(0x7f77100fbab0 msgr2=0x7f7710103b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:17.398 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.394+0000 7f771726f700 1 -- 172.21.15.38:0/3913513324 shutdown_connections 2023-03-09T14:45:17.398 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.394+0000 7f771726f700 1 -- 172.21.15.38:0/3913513324 wait complete. 2023-03-09T14:45:17.481 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.479+0000 7f8227941700 1 -- 172.21.15.38:0/2585240206 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f82201a5370 con 0x7f82200ffac0 2023-03-09T14:45:17.482 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.479+0000 7f82167fc700 1 -- 172.21.15.38:0/2585240206 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f8210097f00 con 0x7f82200ffac0 2023-03-09T14:45:17.482 INFO:teuthology.orchestra.run.smithi038.stdout:107374182457 2023-03-09T14:45:17.484 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.481+0000 7f8227941700 1 -- 172.21.15.38:0/2585240206 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f820c09f400 msgr2=0x7f820c0a18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:17.484 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.481+0000 7f8227941700 1 --2- 172.21.15.38:0/2585240206 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f820c09f400 0x7f820c0a18b0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f821c009be0 tx=0x7f821c009260 comp rx=0 tx=0).stop 2023-03-09T14:45:17.484 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.481+0000 7f8227941700 1 -- 172.21.15.38:0/2585240206 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f82200ffac0 msgr2=0x7f8220198840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:17.484 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.481+0000 7f8227941700 1 --2- 172.21.15.38:0/2585240206 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f82200ffac0 0x7f8220198840 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f8210009990 tx=0x7f821000f740 comp rx=0 tx=0).stop 2023-03-09T14:45:17.484 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.482+0000 7f8227941700 1 -- 172.21.15.38:0/2585240206 shutdown_connections 2023-03-09T14:45:17.485 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.482+0000 7f8227941700 1 --2- 172.21.15.38:0/2585240206 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f820c09f400 0x7f820c0a18b0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:17.485 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.482+0000 7f8227941700 1 --2- 172.21.15.38:0/2585240206 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8220104160 0x7f822019cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:17.485 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.482+0000 7f8227941700 1 --2- 172.21.15.38:0/2585240206 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f82200ff120 0x7f8220198300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:17.485 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.482+0000 7f8227941700 1 --2- 172.21.15.38:0/2585240206 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f82200ffac0 0x7f8220198840 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:17.485 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.482+0000 7f8227941700 1 -- 172.21.15.38:0/2585240206 >> 172.21.15.38:0/2585240206 conn(0x7f8220076d10 msgr2=0x7f8220068b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:17.485 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.482+0000 7f8227941700 1 -- 172.21.15.38:0/2585240206 shutdown_connections 2023-03-09T14:45:17.485 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:17.482+0000 7f8227941700 1 -- 172.21.15.38:0/2585240206 wait complete. 2023-03-09T14:45:17.591 INFO:tasks.cephadm.ceph_manager.ceph:need seq 335007449108 got 335007449111 for osd.11 2023-03-09T14:45:17.592 DEBUG:teuthology.parallel:result is None 2023-03-09T14:45:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:17 smithi038 ceph-mon[92422]: pgmap v290: 129 pgs: 3 peering, 126 active+clean; 636 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s; 86 B/s, 0 objects/s recovering 2023-03-09T14:45:18.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:17 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/1358972204' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2023-03-09T14:45:18.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:17 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/3290090961' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-03-09T14:45:18.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:17 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/3913513324' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2023-03-09T14:45:18.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:17 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/2585240206' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-03-09T14:45:18.184 INFO:tasks.cephadm.ceph_manager.ceph:need seq 279172874268 got 279172874271 for osd.9 2023-03-09T14:45:18.185 DEBUG:teuthology.parallel:result is None 2023-03-09T14:45:18.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:17 smithi122 ceph-mon[93869]: pgmap v290: 129 pgs: 3 peering, 126 active+clean; 636 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s; 86 B/s, 0 objects/s recovering 2023-03-09T14:45:18.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:17 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/1358972204' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2023-03-09T14:45:18.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:17 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/3290090961' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-03-09T14:45:18.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:17 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/3913513324' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2023-03-09T14:45:18.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:17 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/2585240206' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-03-09T14:45:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:17 smithi181 ceph-mon[94189]: pgmap v290: 129 pgs: 3 peering, 126 active+clean; 636 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s; 86 B/s, 0 objects/s recovering 2023-03-09T14:45:18.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:17 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/1358972204' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2023-03-09T14:45:18.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:17 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/3290090961' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-03-09T14:45:18.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:17 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/3913513324' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2023-03-09T14:45:18.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:17 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/2585240206' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-03-09T14:45:18.302 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738436 got 34359738439 for osd.0 2023-03-09T14:45:18.303 DEBUG:teuthology.parallel:result is None 2023-03-09T14:45:18.552 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182455 got 107374182457 for osd.3 2023-03-09T14:45:18.553 DEBUG:teuthology.parallel:result is None 2023-03-09T14:45:19.345 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:19.341+0000 7fe0fd037700 1 -- 172.21.15.38:0/1221577608 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe0f80ffac0 msgr2=0x7fe0f8103a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:19.345 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:19.341+0000 7fe0fd037700 1 --2- 172.21.15.38:0/1221577608 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe0f80ffac0 0x7fe0f8103a80 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fe0e00099c0 tx=0x7fe0e0009cd0 comp rx=0 tx=0).stop 2023-03-09T14:45:19.345 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:19.341+0000 7fe0fd037700 1 -- 172.21.15.38:0/1221577608 shutdown_connections 2023-03-09T14:45:19.345 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:19.341+0000 7fe0fd037700 1 --2- 172.21.15.38:0/1221577608 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe0f8104160 0x7fe0f810bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:19.346 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:19.341+0000 7fe0fd037700 1 --2- 172.21.15.38:0/1221577608 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe0f80ff120 0x7fe0f80ff4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:19.346 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:19.341+0000 7fe0fd037700 1 --2- 172.21.15.38:0/1221577608 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe0f80ffac0 0x7fe0f8103a80 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:19.346 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:19.341+0000 7fe0fd037700 1 -- 172.21.15.38:0/1221577608 >> 172.21.15.38:0/1221577608 conn(0x7fe0f8076d10 msgr2=0x7fe0f8077110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:19.346 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:19.341+0000 7fe0fd037700 1 -- 172.21.15.38:0/1221577608 shutdown_connections 2023-03-09T14:45:19.346 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:19.341+0000 7fe0fd037700 1 -- 172.21.15.38:0/1221577608 wait complete. 2023-03-09T14:45:19.348 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:19.342+0000 7fe0fd037700 1 Processor -- start 2023-03-09T14:45:19.348 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:19.342+0000 7fe0fd037700 1 -- start start 2023-03-09T14:45:19.348 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:19.342+0000 7fe0fd037700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe0f80ff120 0x7fe0f8196f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:19.348 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:19.343+0000 7fe0fd037700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe0f80ffac0 0x7fe0f8197470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:19.348 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:19.343+0000 7fe0fd037700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe0f8104160 0x7fe0f81a1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:19.349 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:19.343+0000 7fe0fd037700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f8197db0 con 0x7fe0f80ff120 2023-03-09T14:45:19.349 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:19.343+0000 7fe0fd037700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f81a24f0 con 0x7fe0f80ffac0 2023-03-09T14:45:19.349 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:19.343+0000 7fe0fd037700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f81a26b0 con 0x7fe0f8104160 2023-03-09T14:45:19.349 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:19.343+0000 7fe0f6d9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe0f80ff120 0x7fe0f8196f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:19.349 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:19.343+0000 7fe0f6d9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe0f80ff120 0x7fe0f8196f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37650/0 (socket says 172.21.15.38:37650) 2023-03-09T14:45:19.349 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:19.343+0000 7fe0f6d9d700 1 -- 172.21.15.38:0/646502947 learned_addr learned my addr 172.21.15.38:0/646502947 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:19.349 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:19.343+0000 7fe0f659c700 1 --2- 172.21.15.38:0/646502947 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe0f80ffac0 0x7fe0f8197470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:19.350 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:19.343+0000 7fe0f6d9d700 1 -- 172.21.15.38:0/646502947 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe0f8104160 msgr2=0x7fe0f81a1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:19.350 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:19.343+0000 7fe0f759e700 1 --2- 172.21.15.38:0/646502947 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe0f8104160 0x7fe0f81a1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:19.350 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:19.343+0000 7fe0f6d9d700 1 --2- 172.21.15.38:0/646502947 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe0f8104160 0x7fe0f81a1fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:19.350 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:19.343+0000 7fe0f6d9d700 1 -- 172.21.15.38:0/646502947 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe0f80ffac0 msgr2=0x7fe0f8197470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:19.351 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:19.343+0000 7fe0f6d9d700 1 --2- 172.21.15.38:0/646502947 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe0f80ffac0 0x7fe0f8197470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:19.351 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:19.343+0000 7fe0f6d9d700 1 -- 172.21.15.38:0/646502947 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0e00095f0 con 0x7fe0f80ff120 2023-03-09T14:45:19.351 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:19.343+0000 7fe0f6d9d700 1 --2- 172.21.15.38:0/646502947 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe0f80ff120 0x7fe0f8196f30 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fe0e800b530 tx=0x7fe0e800b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:19.351 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:19.343+0000 7fe0effff700 1 -- 172.21.15.38:0/646502947 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0e8011830 con 0x7fe0f80ff120 2023-03-09T14:45:19.351 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:19.343+0000 7fe0fd037700 1 -- 172.21.15.38:0/646502947 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0f810ef60 con 0x7fe0f80ff120 2023-03-09T14:45:19.351 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:19.343+0000 7fe0effff700 1 -- 172.21.15.38:0/646502947 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe0e8011e50 con 0x7fe0f80ff120 2023-03-09T14:45:19.351 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:19.343+0000 7fe0effff700 1 -- 172.21.15.38:0/646502947 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0e800f2e0 con 0x7fe0f80ff120 2023-03-09T14:45:19.352 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:19.343+0000 7fe0fd037700 1 -- 172.21.15.38:0/646502947 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0f810edd0 con 0x7fe0f80ff120 2023-03-09T14:45:19.352 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:19.344+0000 7fe0fd037700 1 -- 172.21.15.38:0/646502947 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0f810f0f0 con 0x7fe0f80ff120 2023-03-09T14:45:19.352 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:19.345+0000 7fe0effff700 1 -- 172.21.15.38:0/646502947 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 133238+0+0 (secure 0 0 0) 0x7fe0e8021070 con 0x7fe0f80ff120 2023-03-09T14:45:19.353 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:19.346+0000 7fe0effff700 1 --2- 172.21.15.38:0/646502947 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fe0e409f400 0x7fe0e40a18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:19.353 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:19.346+0000 7fe0f659c700 1 --2- 172.21.15.38:0/646502947 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fe0e409f400 0x7fe0e40a18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:19.353 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:19.347+0000 7fe0effff700 1 -- 172.21.15.38:0/646502947 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fe0e80c9bc0 con 0x7fe0f80ff120 2023-03-09T14:45:19.353 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:19.347+0000 7fe0f659c700 1 --2- 172.21.15.38:0/646502947 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fe0e409f400 0x7fe0e40a18b0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fe0e0009f50 tx=0x7fe0e0003700 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:19.354 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:19.351+0000 7fe0effff700 1 -- 172.21.15.38:0/646502947 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7fe0e8093c30 con 0x7fe0f80ff120 2023-03-09T14:45:19.558 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:19.554+0000 7fe0fd037700 1 -- 172.21.15.38:0/646502947 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fe0f81a3210 con 0x7fe0f80ff120 2023-03-09T14:45:19.558 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:19.555+0000 7fe0effff700 1 -- 172.21.15.38:0/646502947 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fe0e8015070 con 0x7fe0f80ff120 2023-03-09T14:45:19.558 INFO:teuthology.orchestra.run.smithi038.stdout:137438953526 2023-03-09T14:45:19.560 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:19.557+0000 7fe0fd037700 1 -- 172.21.15.38:0/646502947 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fe0e409f400 msgr2=0x7fe0e40a18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:19.560 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:19.557+0000 7fe0fd037700 1 --2- 172.21.15.38:0/646502947 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fe0e409f400 0x7fe0e40a18b0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fe0e0009f50 tx=0x7fe0e0003700 comp rx=0 tx=0).stop 2023-03-09T14:45:19.560 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:19.557+0000 7fe0fd037700 1 -- 172.21.15.38:0/646502947 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe0f80ff120 msgr2=0x7fe0f8196f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:19.561 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:19.557+0000 7fe0fd037700 1 --2- 172.21.15.38:0/646502947 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe0f80ff120 0x7fe0f8196f30 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fe0e800b530 tx=0x7fe0e800b840 comp rx=0 tx=0).stop 2023-03-09T14:45:19.561 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:19.557+0000 7fe0fd037700 1 -- 172.21.15.38:0/646502947 shutdown_connections 2023-03-09T14:45:19.561 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:19.557+0000 7fe0fd037700 1 --2- 172.21.15.38:0/646502947 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe0f80ff120 0x7fe0f8196f30 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:19.561 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:19.557+0000 7fe0fd037700 1 --2- 172.21.15.38:0/646502947 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fe0e409f400 0x7fe0e40a18b0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:19.561 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:19.557+0000 7fe0fd037700 1 --2- 172.21.15.38:0/646502947 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe0f8104160 0x7fe0f81a1fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:19.561 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:19.557+0000 7fe0fd037700 1 --2- 172.21.15.38:0/646502947 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe0f80ffac0 0x7fe0f8197470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:19.562 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:19.557+0000 7fe0fd037700 1 -- 172.21.15.38:0/646502947 >> 172.21.15.38:0/646502947 conn(0x7fe0f8076d10 msgr2=0x7fe0f8102a60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:19.562 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:19.557+0000 7fe0fd037700 1 -- 172.21.15.38:0/646502947 shutdown_connections 2023-03-09T14:45:19.562 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:19.557+0000 7fe0fd037700 1 -- 172.21.15.38:0/646502947 wait complete. 2023-03-09T14:45:19.962 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953523 got 137438953526 for osd.4 2023-03-09T14:45:19.963 DEBUG:teuthology.parallel:result is None 2023-03-09T14:45:19.963 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2023-03-09T14:45:19.963 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph pg dump --format=json 2023-03-09T14:45:20.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:19 smithi122 ceph-mon[93869]: pgmap v291: 129 pgs: 3 peering, 126 active+clean; 636 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 4 op/s; 172 B/s, 0 objects/s recovering 2023-03-09T14:45:20.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:19 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/646502947' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-03-09T14:45:20.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:19 smithi181 ceph-mon[94189]: pgmap v291: 129 pgs: 3 peering, 126 active+clean; 636 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 4 op/s; 172 B/s, 0 objects/s recovering 2023-03-09T14:45:20.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:19 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/646502947' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-03-09T14:45:20.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:19 smithi038 ceph-mon[92422]: pgmap v291: 129 pgs: 3 peering, 126 active+clean; 636 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 4 op/s; 172 B/s, 0 objects/s recovering 2023-03-09T14:45:20.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:19 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/646502947' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-03-09T14:45:20.619 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.a/config 2023-03-09T14:45:21.157 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:20 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:45:21.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:20 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:45:21.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:20 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:45:21.954 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:21.950+0000 7f4be6a60700 1 -- 172.21.15.38:0/1131661253 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4be0105240 msgr2=0x7f4be010ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:21.954 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:21.950+0000 7f4be6a60700 1 --2- 172.21.15.38:0/1131661253 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4be0105240 0x7f4be010ce10 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f4bd800b0d0 tx=0x7f4bd800b3e0 comp rx=0 tx=0).stop 2023-03-09T14:45:21.955 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:21.950+0000 7f4be6a60700 1 -- 172.21.15.38:0/1131661253 shutdown_connections 2023-03-09T14:45:21.955 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:21.950+0000 7f4be6a60700 1 --2- 172.21.15.38:0/1131661253 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4be0105240 0x7f4be010ce10 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:21.955 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:21.950+0000 7f4be6a60700 1 --2- 172.21.15.38:0/1131661253 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4be0100200 0x7f4be01005d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:21.955 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:21.950+0000 7f4be6a60700 1 --2- 172.21.15.38:0/1131661253 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4be0100ba0 0x7f4be0104b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:21.955 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:21.950+0000 7f4be6a60700 1 -- 172.21.15.38:0/1131661253 >> 172.21.15.38:0/1131661253 conn(0x7f4be00fbab0 msgr2=0x7f4be00fdec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:21.955 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:21.951+0000 7f4be6a60700 1 -- 172.21.15.38:0/1131661253 shutdown_connections 2023-03-09T14:45:21.956 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:21.951+0000 7f4be6a60700 1 -- 172.21.15.38:0/1131661253 wait complete. 2023-03-09T14:45:21.956 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:21.951+0000 7f4be6a60700 1 Processor -- start 2023-03-09T14:45:21.956 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:21.951+0000 7f4be6a60700 1 -- start start 2023-03-09T14:45:21.958 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:21.951+0000 7f4be6a60700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4be0100200 0x7f4be01961d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:21.958 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:21.952+0000 7f4be6a60700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4be0100ba0 0x7f4be0196710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:21.958 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:21.952+0000 7f4be6a60700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4be0105240 0x7f4be019ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:21.958 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:21.952+0000 7f4be6a60700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4be019b1b0 con 0x7f4be0105240 2023-03-09T14:45:21.958 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:21.952+0000 7f4be6a60700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4be019b320 con 0x7f4be0100200 2023-03-09T14:45:21.958 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:21.952+0000 7f4be6a60700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4be019b4e0 con 0x7f4be0100ba0 2023-03-09T14:45:21.959 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:21.952+0000 7f4be4ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4be0105240 0x7f4be019ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:21.959 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:21.952+0000 7f4bdffff700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4be0100200 0x7f4be01961d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:21.959 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:21.952+0000 7f4be4ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4be0105240 0x7f4be019ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37668/0 (socket says 172.21.15.38:37668) 2023-03-09T14:45:21.959 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:21.952+0000 7f4bdf7fe700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4be0100ba0 0x7f4be0196710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:21.959 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:21.952+0000 7f4be4ffd700 1 -- 172.21.15.38:0/333006173 learned_addr learned my addr 172.21.15.38:0/333006173 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:21.959 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:21.952+0000 7f4bdffff700 1 -- 172.21.15.38:0/333006173 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4be0100ba0 msgr2=0x7f4be0196710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:21.959 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:21.952+0000 7f4bdffff700 1 --2- 172.21.15.38:0/333006173 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4be0100ba0 0x7f4be0196710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:21.960 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:21.952+0000 7f4bdffff700 1 -- 172.21.15.38:0/333006173 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4be0105240 msgr2=0x7f4be019ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:21.960 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:21.952+0000 7f4bdffff700 1 --2- 172.21.15.38:0/333006173 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4be0105240 0x7f4be019ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:21.960 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:21.952+0000 7f4bdffff700 1 -- 172.21.15.38:0/333006173 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bd8009d00 con 0x7f4be0100200 2023-03-09T14:45:21.960 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:21.952+0000 7f4be4ffd700 1 --2- 172.21.15.38:0/333006173 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4be0105240 0x7f4be019ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:45:21.960 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:21.952+0000 7f4bdf7fe700 1 --2- 172.21.15.38:0/333006173 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4be0100ba0 0x7f4be0196710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:45:21.960 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:21.952+0000 7f4bdffff700 1 --2- 172.21.15.38:0/333006173 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4be0100200 0x7f4be01961d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4bd0009ad0 tx=0x7f4bd0009de0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:21.960 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:21.952+0000 7f4bdd7fa700 1 -- 172.21.15.38:0/333006173 <== mon.2 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd0014070 con 0x7f4be0100200 2023-03-09T14:45:21.960 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:21.952+0000 7f4be6a60700 1 -- 172.21.15.38:0/333006173 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4be0110060 con 0x7f4be0100200 2023-03-09T14:45:21.961 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:21.953+0000 7f4bdd7fa700 1 -- 172.21.15.38:0/333006173 <== mon.2 v2:172.21.15.122:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4bd0007af0 con 0x7f4be0100200 2023-03-09T14:45:21.961 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:21.953+0000 7f4bdd7fa700 1 -- 172.21.15.38:0/333006173 <== mon.2 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bd001ab70 con 0x7f4be0100200 2023-03-09T14:45:21.961 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:21.953+0000 7f4be6a60700 1 -- 172.21.15.38:0/333006173 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4be010fed0 con 0x7f4be0100200 2023-03-09T14:45:21.961 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:21.953+0000 7f4be6a60700 1 -- 172.21.15.38:0/333006173 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4be01101f0 con 0x7f4be0100200 2023-03-09T14:45:21.961 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:21.955+0000 7f4bdd7fa700 1 -- 172.21.15.38:0/333006173 <== mon.2 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 17) v1 ==== 133238+0+0 (secure 0 0 0) 0x7f4bd0021030 con 0x7f4be0100200 2023-03-09T14:45:21.962 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:21.955+0000 7f4bdd7fa700 1 --2- 172.21.15.38:0/333006173 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f4bc809f400 0x7f4bc80a18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:21.962 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:21.955+0000 7f4bdf7fe700 1 --2- 172.21.15.38:0/333006173 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f4bc809f400 0x7f4bc80a18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:21.962 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:21.955+0000 7f4bdd7fa700 1 -- 172.21.15.38:0/333006173 <== mon.2 v2:172.21.15.122:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f4bd00cd670 con 0x7f4be0100200 2023-03-09T14:45:21.962 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:21.956+0000 7f4bdf7fe700 1 --2- 172.21.15.38:0/333006173 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f4bc809f400 0x7f4bc80a18b0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4be01977e0 tx=0x7f4bd400a300 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:21.962 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:21.959+0000 7f4bdd7fa700 1 -- 172.21.15.38:0/333006173 <== mon.2 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f4bd00976e0 con 0x7f4be0100200 2023-03-09T14:45:22.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:21 smithi038 ceph-mon[92422]: pgmap v292: 129 pgs: 129 active+clean; 636 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 5 op/s; 254 B/s, 1 objects/s recovering 2023-03-09T14:45:22.159 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:22.155+0000 7f4be6a60700 1 -- 172.21.15.38:0/333006173 --> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4be00fcb80 con 0x7f4bc809f400 2023-03-09T14:45:22.174 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:22.170+0000 7f4bdd7fa700 1 -- 172.21.15.38:0/333006173 <== mgr.14150 v2:172.21.15.38:6800/3328887328 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+423841 (secure 0 0 0) 0x7f4be00fcb80 con 0x7f4bc809f400 2023-03-09T14:45:22.179 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:45:22.181 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:22.174+0000 7f4be6a60700 1 -- 172.21.15.38:0/333006173 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f4bc809f400 msgr2=0x7f4bc80a18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:22.182 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:22.174+0000 7f4be6a60700 1 --2- 172.21.15.38:0/333006173 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f4bc809f400 0x7f4bc80a18b0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4be01977e0 tx=0x7f4bd400a300 comp rx=0 tx=0).stop 2023-03-09T14:45:22.182 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:22.174+0000 7f4be6a60700 1 -- 172.21.15.38:0/333006173 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4be0100200 msgr2=0x7f4be01961d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:22.182 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:22.174+0000 7f4be6a60700 1 --2- 172.21.15.38:0/333006173 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4be0100200 0x7f4be01961d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4bd0009ad0 tx=0x7f4bd0009de0 comp rx=0 tx=0).stop 2023-03-09T14:45:22.182 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:22.174+0000 7f4be6a60700 1 -- 172.21.15.38:0/333006173 shutdown_connections 2023-03-09T14:45:22.182 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:22.174+0000 7f4be6a60700 1 --2- 172.21.15.38:0/333006173 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4be0105240 0x7f4be019ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:22.182 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:22.174+0000 7f4be6a60700 1 --2- 172.21.15.38:0/333006173 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f4bc809f400 0x7f4bc80a18b0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:22.183 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:22.174+0000 7f4be6a60700 1 --2- 172.21.15.38:0/333006173 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4be0100200 0x7f4be01961d0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:22.183 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:22.174+0000 7f4be6a60700 1 --2- 172.21.15.38:0/333006173 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4be0100ba0 0x7f4be0196710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:22.183 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:22.174+0000 7f4be6a60700 1 -- 172.21.15.38:0/333006173 >> 172.21.15.38:0/333006173 conn(0x7f4be00fbab0 msgr2=0x7f4be00fc470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:22.183 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:22.174+0000 7f4be6a60700 1 -- 172.21.15.38:0/333006173 shutdown_connections 2023-03-09T14:45:22.183 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:22.174+0000 7f4be6a60700 1 -- 172.21.15.38:0/333006173 wait complete. 2023-03-09T14:45:22.183 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2023-03-09T14:45:22.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:21 smithi122 ceph-mon[93869]: pgmap v292: 129 pgs: 129 active+clean; 636 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 5 op/s; 254 B/s, 1 objects/s recovering 2023-03-09T14:45:22.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:21 smithi181 ceph-mon[94189]: pgmap v292: 129 pgs: 129 active+clean; 636 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 5 op/s; 254 B/s, 1 objects/s recovering 2023-03-09T14:45:22.691 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":292,"stamp":"2023-03-09T14:45:20.761839+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":651232,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":398,"num_read_kb":317,"num_write":335,"num_write_kb":3044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1182773,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":273,"ondisk_log_size":273,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":423,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":82988,"kb_used_data":13772,"kb_used_omap":0,"kb_used_meta":69120,"kb_avail":1124858836,"statfs":{"total":1151940427776,"available":1151855448064,"internally_reserved":0,"allocated":14102528,"data_stored":7818713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":70778880},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":48,"num_read_kb":28,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":2037,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002289"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.859159+0000","last_change":"2023-03-09T14:44:19.116371+0000","last_active":"2023-03-09T14:45:14.859159+0000","last_peered":"2023-03-09T14:45:14.859159+0000","last_clean":"2023-03-09T14:45:14.859159+0000","last_became_active":"2023-03-09T14:44:19.116024+0000","last_became_peered":"2023-03-09T14:44:19.116024+0000","last_unstale":"2023-03-09T14:45:14.859159+0000","last_undegraded":"2023-03-09T14:45:14.859159+0000","last_fullsized":"2023-03-09T14:45:14.859159+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T16:43:33.917176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.864887+0000","last_change":"2023-03-09T14:44:19.117312+0000","last_active":"2023-03-09T14:45:14.864887+0000","last_peered":"2023-03-09T14:45:14.864887+0000","last_clean":"2023-03-09T14:45:14.864887+0000","last_became_active":"2023-03-09T14:44:19.116920+0000","last_became_peered":"2023-03-09T14:44:19.116920+0000","last_unstale":"2023-03-09T14:45:14.864887+0000","last_undegraded":"2023-03-09T14:45:14.864887+0000","last_fullsized":"2023-03-09T14:45:14.864887+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:33:15.046431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.864174+0000","last_change":"2023-03-09T14:44:19.113482+0000","last_active":"2023-03-09T14:45:14.864174+0000","last_peered":"2023-03-09T14:45:14.864174+0000","last_clean":"2023-03-09T14:45:14.864174+0000","last_became_active":"2023-03-09T14:44:19.113183+0000","last_became_peered":"2023-03-09T14:44:19.113183+0000","last_unstale":"2023-03-09T14:45:14.864174+0000","last_undegraded":"2023-03-09T14:45:14.864174+0000","last_fullsized":"2023-03-09T14:45:14.864174+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:17:31.382255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.860049+0000","last_change":"2023-03-09T14:44:19.105799+0000","last_active":"2023-03-09T14:45:14.860049+0000","last_peered":"2023-03-09T14:45:14.860049+0000","last_clean":"2023-03-09T14:45:14.860049+0000","last_became_active":"2023-03-09T14:44:19.105405+0000","last_became_peered":"2023-03-09T14:44:19.105405+0000","last_unstale":"2023-03-09T14:45:14.860049+0000","last_undegraded":"2023-03-09T14:45:14.860049+0000","last_fullsized":"2023-03-09T14:45:14.860049+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:38:23.049984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.860357+0000","last_change":"2023-03-09T14:44:19.100656+0000","last_active":"2023-03-09T14:45:14.860357+0000","last_peered":"2023-03-09T14:45:14.860357+0000","last_clean":"2023-03-09T14:45:14.860357+0000","last_became_active":"2023-03-09T14:44:19.100278+0000","last_became_peered":"2023-03-09T14:44:19.100278+0000","last_unstale":"2023-03-09T14:45:14.860357+0000","last_undegraded":"2023-03-09T14:45:14.860357+0000","last_fullsized":"2023-03-09T14:45:14.860357+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:25:39.240251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.857293+0000","last_change":"2023-03-09T14:44:19.101065+0000","last_active":"2023-03-09T14:45:14.857293+0000","last_peered":"2023-03-09T14:45:14.857293+0000","last_clean":"2023-03-09T14:45:14.857293+0000","last_became_active":"2023-03-09T14:44:19.100451+0000","last_became_peered":"2023-03-09T14:44:19.100451+0000","last_unstale":"2023-03-09T14:45:14.857293+0000","last_undegraded":"2023-03-09T14:45:14.857293+0000","last_fullsized":"2023-03-09T14:45:14.857293+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T01:25:15.158254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.852722+0000","last_change":"2023-03-09T14:44:19.107996+0000","last_active":"2023-03-09T14:45:14.852722+0000","last_peered":"2023-03-09T14:45:14.852722+0000","last_clean":"2023-03-09T14:45:14.852722+0000","last_became_active":"2023-03-09T14:44:19.107683+0000","last_became_peered":"2023-03-09T14:44:19.107683+0000","last_unstale":"2023-03-09T14:45:14.852722+0000","last_undegraded":"2023-03-09T14:45:14.852722+0000","last_fullsized":"2023-03-09T14:45:14.852722+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:16:18.605176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.864215+0000","last_change":"2023-03-09T14:44:19.103940+0000","last_active":"2023-03-09T14:45:14.864215+0000","last_peered":"2023-03-09T14:45:14.864215+0000","last_clean":"2023-03-09T14:45:14.864215+0000","last_became_active":"2023-03-09T14:44:19.103620+0000","last_became_peered":"2023-03-09T14:44:19.103620+0000","last_unstale":"2023-03-09T14:45:14.864215+0000","last_undegraded":"2023-03-09T14:45:14.864215+0000","last_fullsized":"2023-03-09T14:45:14.864215+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:06:22.749636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.858269+0000","last_change":"2023-03-09T14:44:19.102695+0000","last_active":"2023-03-09T14:45:14.858269+0000","last_peered":"2023-03-09T14:45:14.858269+0000","last_clean":"2023-03-09T14:45:14.858269+0000","last_became_active":"2023-03-09T14:44:19.101486+0000","last_became_peered":"2023-03-09T14:44:19.101486+0000","last_unstale":"2023-03-09T14:45:14.858269+0000","last_undegraded":"2023-03-09T14:45:14.858269+0000","last_fullsized":"2023-03-09T14:45:14.858269+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T16:34:51.836443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.858962+0000","last_change":"2023-03-09T14:44:19.103522+0000","last_active":"2023-03-09T14:45:14.858962+0000","last_peered":"2023-03-09T14:45:14.858962+0000","last_clean":"2023-03-09T14:45:14.858962+0000","last_became_active":"2023-03-09T14:44:19.103242+0000","last_became_peered":"2023-03-09T14:44:19.103242+0000","last_unstale":"2023-03-09T14:45:14.858962+0000","last_undegraded":"2023-03-09T14:45:14.858962+0000","last_fullsized":"2023-03-09T14:45:14.858962+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:41:03.137554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.853693+0000","last_change":"2023-03-09T14:44:19.114934+0000","last_active":"2023-03-09T14:45:14.853693+0000","last_peered":"2023-03-09T14:45:14.853693+0000","last_clean":"2023-03-09T14:45:14.853693+0000","last_became_active":"2023-03-09T14:44:19.114620+0000","last_became_peered":"2023-03-09T14:44:19.114620+0000","last_unstale":"2023-03-09T14:45:14.853693+0000","last_undegraded":"2023-03-09T14:45:14.853693+0000","last_fullsized":"2023-03-09T14:45:14.853693+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T01:48:40.082076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.860296+0000","last_change":"2023-03-09T14:44:19.107677+0000","last_active":"2023-03-09T14:45:14.860296+0000","last_peered":"2023-03-09T14:45:14.860296+0000","last_clean":"2023-03-09T14:45:14.860296+0000","last_became_active":"2023-03-09T14:44:19.107369+0000","last_became_peered":"2023-03-09T14:44:19.107369+0000","last_unstale":"2023-03-09T14:45:14.860296+0000","last_undegraded":"2023-03-09T14:45:14.860296+0000","last_fullsized":"2023-03-09T14:45:14.860296+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:47:40.117205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.861064+0000","last_change":"2023-03-09T14:44:19.108852+0000","last_active":"2023-03-09T14:45:14.861064+0000","last_peered":"2023-03-09T14:45:14.861064+0000","last_clean":"2023-03-09T14:45:14.861064+0000","last_became_active":"2023-03-09T14:44:19.108515+0000","last_became_peered":"2023-03-09T14:44:19.108515+0000","last_unstale":"2023-03-09T14:45:14.861064+0000","last_undegraded":"2023-03-09T14:45:14.861064+0000","last_fullsized":"2023-03-09T14:45:14.861064+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:00:36.243952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.863730+0000","last_change":"2023-03-09T14:44:19.118673+0000","last_active":"2023-03-09T14:45:14.863730+0000","last_peered":"2023-03-09T14:45:14.863730+0000","last_clean":"2023-03-09T14:45:14.863730+0000","last_became_active":"2023-03-09T14:44:19.117656+0000","last_became_peered":"2023-03-09T14:44:19.117656+0000","last_unstale":"2023-03-09T14:45:14.863730+0000","last_undegraded":"2023-03-09T14:45:14.863730+0000","last_fullsized":"2023-03-09T14:45:14.863730+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:02:50.401680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.863498+0000","last_change":"2023-03-09T14:44:19.112979+0000","last_active":"2023-03-09T14:45:14.863498+0000","last_peered":"2023-03-09T14:45:14.863498+0000","last_clean":"2023-03-09T14:45:14.863498+0000","last_became_active":"2023-03-09T14:44:19.112671+0000","last_became_peered":"2023-03-09T14:44:19.112671+0000","last_unstale":"2023-03-09T14:45:14.863498+0000","last_undegraded":"2023-03-09T14:45:14.863498+0000","last_fullsized":"2023-03-09T14:45:14.863498+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:00:43.465327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.856048+0000","last_change":"2023-03-09T14:44:19.104881+0000","last_active":"2023-03-09T14:45:14.856048+0000","last_peered":"2023-03-09T14:45:14.856048+0000","last_clean":"2023-03-09T14:45:14.856048+0000","last_became_active":"2023-03-09T14:44:19.104647+0000","last_became_peered":"2023-03-09T14:44:19.104647+0000","last_unstale":"2023-03-09T14:45:14.856048+0000","last_undegraded":"2023-03-09T14:45:14.856048+0000","last_fullsized":"2023-03-09T14:45:14.856048+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T20:16:24.625362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.859353+0000","last_change":"2023-03-09T14:44:19.117937+0000","last_active":"2023-03-09T14:45:14.859353+0000","last_peered":"2023-03-09T14:45:14.859353+0000","last_clean":"2023-03-09T14:45:14.859353+0000","last_became_active":"2023-03-09T14:44:19.117628+0000","last_became_peered":"2023-03-09T14:44:19.117628+0000","last_unstale":"2023-03-09T14:45:14.859353+0000","last_undegraded":"2023-03-09T14:45:14.859353+0000","last_fullsized":"2023-03-09T14:45:14.859353+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:22:18.619171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.860217+0000","last_change":"2023-03-09T14:44:19.105071+0000","last_active":"2023-03-09T14:45:14.860217+0000","last_peered":"2023-03-09T14:45:14.860217+0000","last_clean":"2023-03-09T14:45:14.860217+0000","last_became_active":"2023-03-09T14:44:19.103396+0000","last_became_peered":"2023-03-09T14:44:19.103396+0000","last_unstale":"2023-03-09T14:45:14.860217+0000","last_undegraded":"2023-03-09T14:45:14.860217+0000","last_fullsized":"2023-03-09T14:45:14.860217+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:14:51.924636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.857765+0000","last_change":"2023-03-09T14:44:19.104107+0000","last_active":"2023-03-09T14:45:14.857765+0000","last_peered":"2023-03-09T14:45:14.857765+0000","last_clean":"2023-03-09T14:45:14.857765+0000","last_became_active":"2023-03-09T14:44:19.103793+0000","last_became_peered":"2023-03-09T14:44:19.103793+0000","last_unstale":"2023-03-09T14:45:14.857765+0000","last_undegraded":"2023-03-09T14:45:14.857765+0000","last_fullsized":"2023-03-09T14:45:14.857765+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:57:29.169744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.856426+0000","last_change":"2023-03-09T14:44:23.489947+0000","last_active":"2023-03-09T14:45:14.856426+0000","last_peered":"2023-03-09T14:45:14.856426+0000","last_clean":"2023-03-09T14:45:14.856426+0000","last_became_active":"2023-03-09T14:44:23.489560+0000","last_became_peered":"2023-03-09T14:44:23.489560+0000","last_unstale":"2023-03-09T14:45:14.856426+0000","last_undegraded":"2023-03-09T14:45:14.856426+0000","last_fullsized":"2023-03-09T14:45:14.856426+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:06:10.202176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"87'9","reported_seq":49,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:15.961952+0000","last_change":"2023-03-09T14:45:15.961952+0000","last_active":"2023-03-09T14:45:15.961952+0000","last_peered":"2023-03-09T14:45:15.961952+0000","last_clean":"2023-03-09T14:45:15.961952+0000","last_became_active":"2023-03-09T14:45:15.865191+0000","last_became_peered":"2023-03-09T14:45:15.865191+0000","last_unstale":"2023-03-09T14:45:15.961952+0000","last_undegraded":"2023-03-09T14:45:15.961952+0000","last_fullsized":"2023-03-09T14:45:15.961952+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T01:21:05.925138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":566,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.863851+0000","last_change":"2023-03-09T14:44:19.105390+0000","last_active":"2023-03-09T14:45:14.863851+0000","last_peered":"2023-03-09T14:45:14.863851+0000","last_clean":"2023-03-09T14:45:14.863851+0000","last_became_active":"2023-03-09T14:44:19.104694+0000","last_became_peered":"2023-03-09T14:44:19.104694+0000","last_unstale":"2023-03-09T14:45:14.863851+0000","last_undegraded":"2023-03-09T14:45:14.863851+0000","last_fullsized":"2023-03-09T14:45:14.863851+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:19:08.101847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.856827+0000","last_change":"2023-03-09T14:44:23.484103+0000","last_active":"2023-03-09T14:45:14.856827+0000","last_peered":"2023-03-09T14:45:14.856827+0000","last_clean":"2023-03-09T14:45:14.856827+0000","last_became_active":"2023-03-09T14:44:23.483271+0000","last_became_peered":"2023-03-09T14:44:23.483271+0000","last_unstale":"2023-03-09T14:45:14.856827+0000","last_undegraded":"2023-03-09T14:45:14.856827+0000","last_fullsized":"2023-03-09T14:45:14.856827+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:24:06.212623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"87'2","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.864060+0000","last_change":"2023-03-09T14:44:18.667321+0000","last_active":"2023-03-09T14:45:14.864060+0000","last_peered":"2023-03-09T14:45:14.864060+0000","last_clean":"2023-03-09T14:45:14.864060+0000","last_became_active":"2023-03-09T14:44:18.666974+0000","last_became_peered":"2023-03-09T14:44:18.666974+0000","last_unstale":"2023-03-09T14:45:14.864060+0000","last_undegraded":"2023-03-09T14:45:14.864060+0000","last_fullsized":"2023-03-09T14:45:14.864060+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:31:07.327792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.864530+0000","last_change":"2023-03-09T14:44:19.116866+0000","last_active":"2023-03-09T14:45:14.864530+0000","last_peered":"2023-03-09T14:45:14.864530+0000","last_clean":"2023-03-09T14:45:14.864530+0000","last_became_active":"2023-03-09T14:44:19.116571+0000","last_became_peered":"2023-03-09T14:44:19.116571+0000","last_unstale":"2023-03-09T14:45:14.864530+0000","last_undegraded":"2023-03-09T14:45:14.864530+0000","last_fullsized":"2023-03-09T14:45:14.864530+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T16:22:12.723903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.859406+0000","last_change":"2023-03-09T14:44:23.488192+0000","last_active":"2023-03-09T14:45:14.859406+0000","last_peered":"2023-03-09T14:45:14.859406+0000","last_clean":"2023-03-09T14:45:14.859406+0000","last_became_active":"2023-03-09T14:44:23.487734+0000","last_became_peered":"2023-03-09T14:44:23.487734+0000","last_unstale":"2023-03-09T14:45:14.859406+0000","last_undegraded":"2023-03-09T14:45:14.859406+0000","last_fullsized":"2023-03-09T14:45:14.859406+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:35:50.319216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"87'4","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.861901+0000","last_change":"2023-03-09T14:44:18.663577+0000","last_active":"2023-03-09T14:45:14.861901+0000","last_peered":"2023-03-09T14:45:14.861901+0000","last_clean":"2023-03-09T14:45:14.861901+0000","last_became_active":"2023-03-09T14:44:18.663272+0000","last_became_peered":"2023-03-09T14:44:18.663272+0000","last_unstale":"2023-03-09T14:45:14.861901+0000","last_undegraded":"2023-03-09T14:45:14.861901+0000","last_fullsized":"2023-03-09T14:45:14.861901+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T20:55:19.661875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.857464+0000","last_change":"2023-03-09T14:44:19.106986+0000","last_active":"2023-03-09T14:45:14.857464+0000","last_peered":"2023-03-09T14:45:14.857464+0000","last_clean":"2023-03-09T14:45:14.857464+0000","last_became_active":"2023-03-09T14:44:19.106625+0000","last_became_peered":"2023-03-09T14:44:19.106625+0000","last_unstale":"2023-03-09T14:45:14.857464+0000","last_undegraded":"2023-03-09T14:45:14.857464+0000","last_fullsized":"2023-03-09T14:45:14.857464+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:03:54.521603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.858888+0000","last_change":"2023-03-09T14:44:23.483803+0000","last_active":"2023-03-09T14:45:14.858888+0000","last_peered":"2023-03-09T14:45:14.858888+0000","last_clean":"2023-03-09T14:45:14.858888+0000","last_became_active":"2023-03-09T14:44:23.483286+0000","last_became_peered":"2023-03-09T14:44:23.483286+0000","last_unstale":"2023-03-09T14:45:14.858888+0000","last_undegraded":"2023-03-09T14:45:14.858888+0000","last_fullsized":"2023-03-09T14:45:14.858888+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:32:35.975670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"82'2","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.861301+0000","last_change":"2023-03-09T14:44:18.657685+0000","last_active":"2023-03-09T14:45:14.861301+0000","last_peered":"2023-03-09T14:45:14.861301+0000","last_clean":"2023-03-09T14:45:14.861301+0000","last_became_active":"2023-03-09T14:44:18.657355+0000","last_became_peered":"2023-03-09T14:44:18.657355+0000","last_unstale":"2023-03-09T14:45:14.861301+0000","last_undegraded":"2023-03-09T14:45:14.861301+0000","last_fullsized":"2023-03-09T14:45:14.861301+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:33:59.259690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.853085+0000","last_change":"2023-03-09T14:44:19.111687+0000","last_active":"2023-03-09T14:45:14.853085+0000","last_peered":"2023-03-09T14:45:14.853085+0000","last_clean":"2023-03-09T14:45:14.853085+0000","last_became_active":"2023-03-09T14:44:19.111481+0000","last_became_peered":"2023-03-09T14:44:19.111481+0000","last_unstale":"2023-03-09T14:45:14.853085+0000","last_undegraded":"2023-03-09T14:45:14.853085+0000","last_fullsized":"2023-03-09T14:45:14.853085+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:06:14.452967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.857536+0000","last_change":"2023-03-09T14:44:23.485075+0000","last_active":"2023-03-09T14:45:14.857536+0000","last_peered":"2023-03-09T14:45:14.857536+0000","last_clean":"2023-03-09T14:45:14.857536+0000","last_became_active":"2023-03-09T14:44:23.484648+0000","last_became_peered":"2023-03-09T14:44:23.484648+0000","last_unstale":"2023-03-09T14:45:14.857536+0000","last_undegraded":"2023-03-09T14:45:14.857536+0000","last_fullsized":"2023-03-09T14:45:14.857536+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:49:40.332276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"87'5","reported_seq":80,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:15.869319+0000","last_change":"2023-03-09T14:45:15.868911+0000","last_active":"2023-03-09T14:45:15.869319+0000","last_peered":"2023-03-09T14:45:15.869319+0000","last_clean":"2023-03-09T14:45:15.869319+0000","last_became_active":"2023-03-09T14:45:15.861592+0000","last_became_peered":"2023-03-09T14:45:15.861592+0000","last_unstale":"2023-03-09T14:45:15.869319+0000","last_undegraded":"2023-03-09T14:45:15.869319+0000","last_fullsized":"2023-03-09T14:45:15.869319+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T20:17:55.938860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":40,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":622,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.861663+0000","last_change":"2023-03-09T14:44:19.107572+0000","last_active":"2023-03-09T14:45:14.861663+0000","last_peered":"2023-03-09T14:45:14.861663+0000","last_clean":"2023-03-09T14:45:14.861663+0000","last_became_active":"2023-03-09T14:44:19.106632+0000","last_became_peered":"2023-03-09T14:44:19.106632+0000","last_unstale":"2023-03-09T14:45:14.861663+0000","last_undegraded":"2023-03-09T14:45:14.861663+0000","last_fullsized":"2023-03-09T14:45:14.861663+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T19:04:01.379375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.857041+0000","last_change":"2023-03-09T14:44:23.494002+0000","last_active":"2023-03-09T14:45:14.857041+0000","last_peered":"2023-03-09T14:45:14.857041+0000","last_clean":"2023-03-09T14:45:14.857041+0000","last_became_active":"2023-03-09T14:44:23.493628+0000","last_became_peered":"2023-03-09T14:44:23.493628+0000","last_unstale":"2023-03-09T14:45:14.857041+0000","last_undegraded":"2023-03-09T14:45:14.857041+0000","last_fullsized":"2023-03-09T14:45:14.857041+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:30:47.970556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"87'3","reported_seq":36,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:15.964036+0000","last_change":"2023-03-09T14:45:15.964036+0000","last_active":"2023-03-09T14:45:15.964036+0000","last_peered":"2023-03-09T14:45:15.964036+0000","last_clean":"2023-03-09T14:45:15.964036+0000","last_became_active":"2023-03-09T14:45:15.865998+0000","last_became_peered":"2023-03-09T14:45:15.865998+0000","last_unstale":"2023-03-09T14:45:15.964036+0000","last_undegraded":"2023-03-09T14:45:15.964036+0000","last_fullsized":"2023-03-09T14:45:15.964036+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:16:58.213684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.852992+0000","last_change":"2023-03-09T14:44:19.111303+0000","last_active":"2023-03-09T14:45:14.852992+0000","last_peered":"2023-03-09T14:45:14.852992+0000","last_clean":"2023-03-09T14:45:14.852992+0000","last_became_active":"2023-03-09T14:44:19.111026+0000","last_became_peered":"2023-03-09T14:44:19.111026+0000","last_unstale":"2023-03-09T14:45:14.852992+0000","last_undegraded":"2023-03-09T14:45:14.852992+0000","last_fullsized":"2023-03-09T14:45:14.852992+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T02:26:32.496084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.859059+0000","last_change":"2023-03-09T14:44:23.493272+0000","last_active":"2023-03-09T14:45:14.859059+0000","last_peered":"2023-03-09T14:45:14.859059+0000","last_clean":"2023-03-09T14:45:14.859059+0000","last_became_active":"2023-03-09T14:44:23.492785+0000","last_became_peered":"2023-03-09T14:44:23.492785+0000","last_unstale":"2023-03-09T14:45:14.859059+0000","last_undegraded":"2023-03-09T14:45:14.859059+0000","last_fullsized":"2023-03-09T14:45:14.859059+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T20:18:39.758938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"87'2","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.861264+0000","last_change":"2023-03-09T14:44:18.658179+0000","last_active":"2023-03-09T14:45:14.861264+0000","last_peered":"2023-03-09T14:45:14.861264+0000","last_clean":"2023-03-09T14:45:14.861264+0000","last_became_active":"2023-03-09T14:44:18.657851+0000","last_became_peered":"2023-03-09T14:44:18.657851+0000","last_unstale":"2023-03-09T14:45:14.861264+0000","last_undegraded":"2023-03-09T14:45:14.861264+0000","last_fullsized":"2023-03-09T14:45:14.861264+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:52:49.291615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.864131+0000","last_change":"2023-03-09T14:44:19.104176+0000","last_active":"2023-03-09T14:45:14.864131+0000","last_peered":"2023-03-09T14:45:14.864131+0000","last_clean":"2023-03-09T14:45:14.864131+0000","last_became_active":"2023-03-09T14:44:19.103816+0000","last_became_peered":"2023-03-09T14:44:19.103816+0000","last_unstale":"2023-03-09T14:45:14.864131+0000","last_undegraded":"2023-03-09T14:45:14.864131+0000","last_fullsized":"2023-03-09T14:45:14.864131+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T02:06:41.876169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.864893+0000","last_change":"2023-03-09T14:44:23.490876+0000","last_active":"2023-03-09T14:45:14.864893+0000","last_peered":"2023-03-09T14:45:14.864893+0000","last_clean":"2023-03-09T14:45:14.864893+0000","last_became_active":"2023-03-09T14:44:23.490461+0000","last_became_peered":"2023-03-09T14:44:23.490461+0000","last_unstale":"2023-03-09T14:45:14.864893+0000","last_undegraded":"2023-03-09T14:45:14.864893+0000","last_fullsized":"2023-03-09T14:45:14.864893+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T16:28:40.456613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"87'1","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.852373+0000","last_change":"2023-03-09T14:44:18.662513+0000","last_active":"2023-03-09T14:45:14.852373+0000","last_peered":"2023-03-09T14:45:14.852373+0000","last_clean":"2023-03-09T14:45:14.852373+0000","last_became_active":"2023-03-09T14:44:18.661907+0000","last_became_peered":"2023-03-09T14:44:18.661907+0000","last_unstale":"2023-03-09T14:45:14.852373+0000","last_undegraded":"2023-03-09T14:45:14.852373+0000","last_fullsized":"2023-03-09T14:45:14.852373+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:45:17.373213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.860438+0000","last_change":"2023-03-09T14:44:19.115297+0000","last_active":"2023-03-09T14:45:14.860438+0000","last_peered":"2023-03-09T14:45:14.860438+0000","last_clean":"2023-03-09T14:45:14.860438+0000","last_became_active":"2023-03-09T14:44:19.114973+0000","last_became_peered":"2023-03-09T14:44:19.114973+0000","last_unstale":"2023-03-09T14:45:14.860438+0000","last_undegraded":"2023-03-09T14:45:14.860438+0000","last_fullsized":"2023-03-09T14:45:14.860438+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:11:22.226608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.864500+0000","last_change":"2023-03-09T14:44:23.482772+0000","last_active":"2023-03-09T14:45:14.864500+0000","last_peered":"2023-03-09T14:45:14.864500+0000","last_clean":"2023-03-09T14:45:14.864500+0000","last_became_active":"2023-03-09T14:44:23.481750+0000","last_became_peered":"2023-03-09T14:44:23.481750+0000","last_unstale":"2023-03-09T14:45:14.864500+0000","last_undegraded":"2023-03-09T14:45:14.864500+0000","last_fullsized":"2023-03-09T14:45:14.864500+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T19:26:03.737713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"87'3","reported_seq":79,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:18.872287+0000","last_change":"2023-03-09T14:45:15.870269+0000","last_active":"2023-03-09T14:45:18.872287+0000","last_peered":"2023-03-09T14:45:18.872287+0000","last_clean":"2023-03-09T14:45:18.872287+0000","last_became_active":"2023-03-09T14:45:15.862610+0000","last_became_peered":"2023-03-09T14:45:15.862610+0000","last_unstale":"2023-03-09T14:45:18.872287+0000","last_undegraded":"2023-03-09T14:45:18.872287+0000","last_fullsized":"2023-03-09T14:45:18.872287+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T02:37:23.157022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":41,"num_read_kb":41,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,9],"acting":[10,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.858136+0000","last_change":"2023-03-09T14:44:19.100914+0000","last_active":"2023-03-09T14:45:14.858136+0000","last_peered":"2023-03-09T14:45:14.858136+0000","last_clean":"2023-03-09T14:45:14.858136+0000","last_became_active":"2023-03-09T14:44:19.100451+0000","last_became_peered":"2023-03-09T14:44:19.100451+0000","last_unstale":"2023-03-09T14:45:14.858136+0000","last_undegraded":"2023-03-09T14:45:14.858136+0000","last_fullsized":"2023-03-09T14:45:14.858136+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:18:41.852697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.860006+0000","last_change":"2023-03-09T14:44:23.490129+0000","last_active":"2023-03-09T14:45:14.860006+0000","last_peered":"2023-03-09T14:45:14.860006+0000","last_clean":"2023-03-09T14:45:14.860006+0000","last_became_active":"2023-03-09T14:44:23.489696+0000","last_became_peered":"2023-03-09T14:44:23.489696+0000","last_unstale":"2023-03-09T14:45:14.860006+0000","last_undegraded":"2023-03-09T14:45:14.860006+0000","last_fullsized":"2023-03-09T14:45:14.860006+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T16:47:41.943904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"87'2","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:15.955780+0000","last_change":"2023-03-09T14:45:15.955780+0000","last_active":"2023-03-09T14:45:15.955780+0000","last_peered":"2023-03-09T14:45:15.955780+0000","last_clean":"2023-03-09T14:45:15.955780+0000","last_became_active":"2023-03-09T14:45:15.862567+0000","last_became_peered":"2023-03-09T14:45:15.862567+0000","last_unstale":"2023-03-09T14:45:15.955780+0000","last_undegraded":"2023-03-09T14:45:15.955780+0000","last_fullsized":"2023-03-09T14:45:15.955780+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:08:41.748638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.864609+0000","last_change":"2023-03-09T14:44:19.119125+0000","last_active":"2023-03-09T14:45:14.864609+0000","last_peered":"2023-03-09T14:45:14.864609+0000","last_clean":"2023-03-09T14:45:14.864609+0000","last_became_active":"2023-03-09T14:44:19.118677+0000","last_became_peered":"2023-03-09T14:44:19.118677+0000","last_unstale":"2023-03-09T14:45:14.864609+0000","last_undegraded":"2023-03-09T14:45:14.864609+0000","last_fullsized":"2023-03-09T14:45:14.864609+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:06:28.740355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.859670+0000","last_change":"2023-03-09T14:44:23.484752+0000","last_active":"2023-03-09T14:45:14.859670+0000","last_peered":"2023-03-09T14:45:14.859670+0000","last_clean":"2023-03-09T14:45:14.859670+0000","last_became_active":"2023-03-09T14:44:23.484045+0000","last_became_peered":"2023-03-09T14:44:23.484045+0000","last_unstale":"2023-03-09T14:45:14.859670+0000","last_undegraded":"2023-03-09T14:45:14.859670+0000","last_fullsized":"2023-03-09T14:45:14.859670+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:58:39.704017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"87'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.853129+0000","last_change":"2023-03-09T14:44:18.665122+0000","last_active":"2023-03-09T14:45:14.853129+0000","last_peered":"2023-03-09T14:45:14.853129+0000","last_clean":"2023-03-09T14:45:14.853129+0000","last_became_active":"2023-03-09T14:44:18.664786+0000","last_became_peered":"2023-03-09T14:44:18.664786+0000","last_unstale":"2023-03-09T14:45:14.853129+0000","last_undegraded":"2023-03-09T14:45:14.853129+0000","last_fullsized":"2023-03-09T14:45:14.853129+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T20:29:20.919293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":656,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.857673+0000","last_change":"2023-03-09T14:44:19.108693+0000","last_active":"2023-03-09T14:45:14.857673+0000","last_peered":"2023-03-09T14:45:14.857673+0000","last_clean":"2023-03-09T14:45:14.857673+0000","last_became_active":"2023-03-09T14:44:19.108370+0000","last_became_peered":"2023-03-09T14:44:19.108370+0000","last_unstale":"2023-03-09T14:45:14.857673+0000","last_undegraded":"2023-03-09T14:45:14.857673+0000","last_fullsized":"2023-03-09T14:45:14.857673+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T16:30:58.726080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:15.862206+0000","last_change":"2023-03-09T14:45:15.862206+0000","last_active":"2023-03-09T14:45:15.862206+0000","last_peered":"2023-03-09T14:45:15.862206+0000","last_clean":"2023-03-09T14:45:15.862206+0000","last_became_active":"2023-03-09T14:45:15.861775+0000","last_became_peered":"2023-03-09T14:45:15.861775+0000","last_unstale":"2023-03-09T14:45:15.862206+0000","last_undegraded":"2023-03-09T14:45:15.862206+0000","last_fullsized":"2023-03-09T14:45:15.862206+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:31:21.810745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.14","version":"87'30","reported_seq":137,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:19.867290+0000","last_change":"2023-03-09T14:44:18.661904+0000","last_active":"2023-03-09T14:45:19.867290+0000","last_peered":"2023-03-09T14:45:19.867290+0000","last_clean":"2023-03-09T14:45:19.867290+0000","last_became_active":"2023-03-09T14:44:18.661597+0000","last_became_peered":"2023-03-09T14:44:18.661597+0000","last_unstale":"2023-03-09T14:45:19.867290+0000","last_undegraded":"2023-03-09T14:45:19.867290+0000","last_fullsized":"2023-03-09T14:45:19.867290+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T14:57:04.978132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56666,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":89,"num_read_kb":100,"num_write":31,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.861600+0000","last_change":"2023-03-09T14:44:19.111671+0000","last_active":"2023-03-09T14:45:14.861600+0000","last_peered":"2023-03-09T14:45:14.861600+0000","last_clean":"2023-03-09T14:45:14.861600+0000","last_became_active":"2023-03-09T14:44:19.111172+0000","last_became_peered":"2023-03-09T14:44:19.111172+0000","last_unstale":"2023-03-09T14:45:14.861600+0000","last_undegraded":"2023-03-09T14:45:14.861600+0000","last_fullsized":"2023-03-09T14:45:14.861600+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T16:27:23.921853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.853428+0000","last_change":"2023-03-09T14:44:23.486338+0000","last_active":"2023-03-09T14:45:14.853428+0000","last_peered":"2023-03-09T14:45:14.853428+0000","last_clean":"2023-03-09T14:45:14.853428+0000","last_became_active":"2023-03-09T14:44:23.485729+0000","last_became_peered":"2023-03-09T14:44:23.485729+0000","last_unstale":"2023-03-09T14:45:14.853428+0000","last_undegraded":"2023-03-09T14:45:14.853428+0000","last_fullsized":"2023-03-09T14:45:14.853428+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:28:23.872633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.857067+0000","last_change":"2023-03-09T14:44:18.664942+0000","last_active":"2023-03-09T14:45:14.857067+0000","last_peered":"2023-03-09T14:45:14.857067+0000","last_clean":"2023-03-09T14:45:14.857067+0000","last_became_active":"2023-03-09T14:44:18.664007+0000","last_became_peered":"2023-03-09T14:44:18.664007+0000","last_unstale":"2023-03-09T14:45:14.857067+0000","last_undegraded":"2023-03-09T14:45:14.857067+0000","last_fullsized":"2023-03-09T14:45:14.857067+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:36:21.329388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.861918+0000","last_change":"2023-03-09T14:44:19.110140+0000","last_active":"2023-03-09T14:45:14.861918+0000","last_peered":"2023-03-09T14:45:14.861918+0000","last_clean":"2023-03-09T14:45:14.861918+0000","last_became_active":"2023-03-09T14:44:19.109811+0000","last_became_peered":"2023-03-09T14:44:19.109811+0000","last_unstale":"2023-03-09T14:45:14.861918+0000","last_undegraded":"2023-03-09T14:45:14.861918+0000","last_fullsized":"2023-03-09T14:45:14.861918+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T14:56:14.638662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.858304+0000","last_change":"2023-03-09T14:44:23.491318+0000","last_active":"2023-03-09T14:45:14.858304+0000","last_peered":"2023-03-09T14:45:14.858304+0000","last_clean":"2023-03-09T14:45:14.858304+0000","last_became_active":"2023-03-09T14:44:23.490915+0000","last_became_peered":"2023-03-09T14:44:23.490915+0000","last_unstale":"2023-03-09T14:45:14.858304+0000","last_undegraded":"2023-03-09T14:45:14.858304+0000","last_fullsized":"2023-03-09T14:45:14.858304+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:32:06.877573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"87'3","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.852806+0000","last_change":"2023-03-09T14:44:18.657323+0000","last_active":"2023-03-09T14:45:14.852806+0000","last_peered":"2023-03-09T14:45:14.852806+0000","last_clean":"2023-03-09T14:45:14.852806+0000","last_became_active":"2023-03-09T14:44:18.657017+0000","last_became_peered":"2023-03-09T14:44:18.657017+0000","last_unstale":"2023-03-09T14:45:14.852806+0000","last_undegraded":"2023-03-09T14:45:14.852806+0000","last_fullsized":"2023-03-09T14:45:14.852806+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:43:11.562723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.853198+0000","last_change":"2023-03-09T14:44:19.112194+0000","last_active":"2023-03-09T14:45:14.853198+0000","last_peered":"2023-03-09T14:45:14.853198+0000","last_clean":"2023-03-09T14:45:14.853198+0000","last_became_active":"2023-03-09T14:44:19.111924+0000","last_became_peered":"2023-03-09T14:44:19.111924+0000","last_unstale":"2023-03-09T14:45:14.853198+0000","last_undegraded":"2023-03-09T14:45:14.853198+0000","last_fullsized":"2023-03-09T14:45:14.853198+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T19:54:22.669848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.860733+0000","last_change":"2023-03-09T14:44:23.493993+0000","last_active":"2023-03-09T14:45:14.860733+0000","last_peered":"2023-03-09T14:45:14.860733+0000","last_clean":"2023-03-09T14:45:14.860733+0000","last_became_active":"2023-03-09T14:44:23.493535+0000","last_became_peered":"2023-03-09T14:44:23.493535+0000","last_unstale":"2023-03-09T14:45:14.860733+0000","last_undegraded":"2023-03-09T14:45:14.860733+0000","last_fullsized":"2023-03-09T14:45:14.860733+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T01:18:23.669790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"87'4","reported_seq":62,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:15.591163+0000","last_change":"2023-03-09T14:44:18.659914+0000","last_active":"2023-03-09T14:45:15.591163+0000","last_peered":"2023-03-09T14:45:15.591163+0000","last_clean":"2023-03-09T14:45:15.591163+0000","last_became_active":"2023-03-09T14:44:18.658835+0000","last_became_peered":"2023-03-09T14:44:18.658835+0000","last_unstale":"2023-03-09T14:45:15.591163+0000","last_undegraded":"2023-03-09T14:45:15.591163+0000","last_fullsized":"2023-03-09T14:45:15.591163+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T14:55:05.413260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":978,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":41,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.852607+0000","last_change":"2023-03-09T14:44:19.111058+0000","last_active":"2023-03-09T14:45:14.852607+0000","last_peered":"2023-03-09T14:45:14.852607+0000","last_clean":"2023-03-09T14:45:14.852607+0000","last_became_active":"2023-03-09T14:44:19.110760+0000","last_became_peered":"2023-03-09T14:44:19.110760+0000","last_unstale":"2023-03-09T14:45:14.852607+0000","last_undegraded":"2023-03-09T14:45:14.852607+0000","last_fullsized":"2023-03-09T14:45:14.852607+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:25:13.199416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.861493+0000","last_change":"2023-03-09T14:44:23.484710+0000","last_active":"2023-03-09T14:45:14.861493+0000","last_peered":"2023-03-09T14:45:14.861493+0000","last_clean":"2023-03-09T14:45:14.861493+0000","last_became_active":"2023-03-09T14:44:23.483928+0000","last_became_peered":"2023-03-09T14:44:23.483928+0000","last_unstale":"2023-03-09T14:45:14.861493+0000","last_undegraded":"2023-03-09T14:45:14.861493+0000","last_fullsized":"2023-03-09T14:45:14.861493+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T14:51:32.124111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"87'15","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.860284+0000","last_change":"2023-03-09T14:44:18.663692+0000","last_active":"2023-03-09T14:45:14.860284+0000","last_peered":"2023-03-09T14:45:14.860284+0000","last_clean":"2023-03-09T14:45:14.860284+0000","last_became_active":"2023-03-09T14:44:18.663400+0000","last_became_peered":"2023-03-09T14:44:18.663400+0000","last_unstale":"2023-03-09T14:45:14.860284+0000","last_undegraded":"2023-03-09T14:45:14.860284+0000","last_fullsized":"2023-03-09T14:45:14.860284+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:32:31.634315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.852446+0000","last_change":"2023-03-09T14:44:19.110819+0000","last_active":"2023-03-09T14:45:14.852446+0000","last_peered":"2023-03-09T14:45:14.852446+0000","last_clean":"2023-03-09T14:45:14.852446+0000","last_became_active":"2023-03-09T14:44:19.110542+0000","last_became_peered":"2023-03-09T14:44:19.110542+0000","last_unstale":"2023-03-09T14:45:14.852446+0000","last_undegraded":"2023-03-09T14:45:14.852446+0000","last_fullsized":"2023-03-09T14:45:14.852446+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:25:38.382685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.852360+0000","last_change":"2023-03-09T14:44:23.484614+0000","last_active":"2023-03-09T14:45:14.852360+0000","last_peered":"2023-03-09T14:45:14.852360+0000","last_clean":"2023-03-09T14:45:14.852360+0000","last_became_active":"2023-03-09T14:44:23.484061+0000","last_became_peered":"2023-03-09T14:44:23.484061+0000","last_unstale":"2023-03-09T14:45:14.852360+0000","last_undegraded":"2023-03-09T14:45:14.852360+0000","last_fullsized":"2023-03-09T14:45:14.852360+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T19:31:42.889122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"87'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.861838+0000","last_change":"2023-03-09T14:44:18.656056+0000","last_active":"2023-03-09T14:45:14.861838+0000","last_peered":"2023-03-09T14:45:14.861838+0000","last_clean":"2023-03-09T14:45:14.861838+0000","last_became_active":"2023-03-09T14:44:18.655709+0000","last_became_peered":"2023-03-09T14:44:18.655709+0000","last_unstale":"2023-03-09T14:45:14.861838+0000","last_undegraded":"2023-03-09T14:45:14.861838+0000","last_fullsized":"2023-03-09T14:45:14.861838+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T19:09:11.447249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.858951+0000","last_change":"2023-03-09T14:44:19.110292+0000","last_active":"2023-03-09T14:45:14.858951+0000","last_peered":"2023-03-09T14:45:14.858951+0000","last_clean":"2023-03-09T14:45:14.858951+0000","last_became_active":"2023-03-09T14:44:19.110006+0000","last_became_peered":"2023-03-09T14:44:19.110006+0000","last_unstale":"2023-03-09T14:45:14.858951+0000","last_undegraded":"2023-03-09T14:45:14.858951+0000","last_fullsized":"2023-03-09T14:45:14.858951+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:36:40.401486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.861551+0000","last_change":"2023-03-09T14:44:23.484004+0000","last_active":"2023-03-09T14:45:14.861551+0000","last_peered":"2023-03-09T14:45:14.861551+0000","last_clean":"2023-03-09T14:45:14.861551+0000","last_became_active":"2023-03-09T14:44:23.483581+0000","last_became_peered":"2023-03-09T14:44:23.483581+0000","last_unstale":"2023-03-09T14:45:14.861551+0000","last_undegraded":"2023-03-09T14:45:14.861551+0000","last_fullsized":"2023-03-09T14:45:14.861551+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:24:25.939838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.860756+0000","last_change":"2023-03-09T14:44:18.661856+0000","last_active":"2023-03-09T14:45:14.860756+0000","last_peered":"2023-03-09T14:45:14.860756+0000","last_clean":"2023-03-09T14:45:14.860756+0000","last_became_active":"2023-03-09T14:44:18.661500+0000","last_became_peered":"2023-03-09T14:44:18.661500+0000","last_unstale":"2023-03-09T14:45:14.860756+0000","last_undegraded":"2023-03-09T14:45:14.860756+0000","last_fullsized":"2023-03-09T14:45:14.860756+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:27:12.181159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.858358+0000","last_change":"2023-03-09T14:44:19.102058+0000","last_active":"2023-03-09T14:45:14.858358+0000","last_peered":"2023-03-09T14:45:14.858358+0000","last_clean":"2023-03-09T14:45:14.858358+0000","last_became_active":"2023-03-09T14:44:19.101762+0000","last_became_peered":"2023-03-09T14:44:19.101762+0000","last_unstale":"2023-03-09T14:45:14.858358+0000","last_undegraded":"2023-03-09T14:45:14.858358+0000","last_fullsized":"2023-03-09T14:45:14.858358+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:39:42.544086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.862303+0000","last_change":"2023-03-09T14:44:23.490237+0000","last_active":"2023-03-09T14:45:14.862303+0000","last_peered":"2023-03-09T14:45:14.862303+0000","last_clean":"2023-03-09T14:45:14.862303+0000","last_became_active":"2023-03-09T14:44:23.489844+0000","last_became_peered":"2023-03-09T14:44:23.489844+0000","last_unstale":"2023-03-09T14:45:14.862303+0000","last_undegraded":"2023-03-09T14:45:14.862303+0000","last_fullsized":"2023-03-09T14:45:14.862303+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T14:50:09.358698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"87'1","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:15.865433+0000","last_change":"2023-03-09T14:45:15.864913+0000","last_active":"2023-03-09T14:45:15.865433+0000","last_peered":"2023-03-09T14:45:15.865433+0000","last_clean":"2023-03-09T14:45:15.865433+0000","last_became_active":"2023-03-09T14:45:15.864527+0000","last_became_peered":"2023-03-09T14:45:15.864527+0000","last_unstale":"2023-03-09T14:45:15.865433+0000","last_undegraded":"2023-03-09T14:45:15.865433+0000","last_fullsized":"2023-03-09T14:45:15.865433+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:02:19.489659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6,1],"acting":[11,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.857166+0000","last_change":"2023-03-09T14:44:19.117584+0000","last_active":"2023-03-09T14:45:14.857166+0000","last_peered":"2023-03-09T14:45:14.857166+0000","last_clean":"2023-03-09T14:45:14.857166+0000","last_became_active":"2023-03-09T14:44:19.117262+0000","last_became_peered":"2023-03-09T14:44:19.117262+0000","last_unstale":"2023-03-09T14:45:14.857166+0000","last_undegraded":"2023-03-09T14:45:14.857166+0000","last_fullsized":"2023-03-09T14:45:14.857166+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T19:19:38.080048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.861827+0000","last_change":"2023-03-09T14:44:23.485769+0000","last_active":"2023-03-09T14:45:14.861827+0000","last_peered":"2023-03-09T14:45:14.861827+0000","last_clean":"2023-03-09T14:45:14.861827+0000","last_became_active":"2023-03-09T14:44:23.485317+0000","last_became_peered":"2023-03-09T14:44:23.485317+0000","last_unstale":"2023-03-09T14:45:14.861827+0000","last_undegraded":"2023-03-09T14:45:14.861827+0000","last_fullsized":"2023-03-09T14:45:14.861827+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:58:35.140512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.852564+0000","last_change":"2023-03-09T14:44:19.112255+0000","last_active":"2023-03-09T14:45:14.852564+0000","last_peered":"2023-03-09T14:45:14.852564+0000","last_clean":"2023-03-09T14:45:14.852564+0000","last_became_active":"2023-03-09T14:44:19.111988+0000","last_became_peered":"2023-03-09T14:44:19.111988+0000","last_unstale":"2023-03-09T14:45:14.852564+0000","last_undegraded":"2023-03-09T14:45:14.852564+0000","last_fullsized":"2023-03-09T14:45:14.852564+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:55:50.423701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"87'4","reported_seq":62,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:17.594145+0000","last_change":"2023-03-09T14:44:18.665291+0000","last_active":"2023-03-09T14:45:17.594145+0000","last_peered":"2023-03-09T14:45:17.594145+0000","last_clean":"2023-03-09T14:45:17.594145+0000","last_became_active":"2023-03-09T14:44:18.664950+0000","last_became_peered":"2023-03-09T14:44:18.664950+0000","last_unstale":"2023-03-09T14:45:17.594145+0000","last_undegraded":"2023-03-09T14:45:17.594145+0000","last_fullsized":"2023-03-09T14:45:17.594145+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:40:58.619717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":40,"num_read_kb":40,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"87'1","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.853567+0000","last_change":"2023-03-09T14:44:23.480872+0000","last_active":"2023-03-09T14:45:14.853567+0000","last_peered":"2023-03-09T14:45:14.853567+0000","last_clean":"2023-03-09T14:45:14.853567+0000","last_became_active":"2023-03-09T14:44:23.480495+0000","last_became_peered":"2023-03-09T14:44:23.480495+0000","last_unstale":"2023-03-09T14:45:14.853567+0000","last_undegraded":"2023-03-09T14:45:14.853567+0000","last_fullsized":"2023-03-09T14:45:14.853567+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:19:01.393034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.863487+0000","last_change":"2023-03-09T14:44:19.102833+0000","last_active":"2023-03-09T14:45:14.863487+0000","last_peered":"2023-03-09T14:45:14.863487+0000","last_clean":"2023-03-09T14:45:14.863487+0000","last_became_active":"2023-03-09T14:44:19.102484+0000","last_became_peered":"2023-03-09T14:44:19.102484+0000","last_unstale":"2023-03-09T14:45:14.863487+0000","last_undegraded":"2023-03-09T14:45:14.863487+0000","last_fullsized":"2023-03-09T14:45:14.863487+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:14:50.002907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"87'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.859537+0000","last_change":"2023-03-09T14:44:18.661603+0000","last_active":"2023-03-09T14:45:14.859537+0000","last_peered":"2023-03-09T14:45:14.859537+0000","last_clean":"2023-03-09T14:45:14.859537+0000","last_became_active":"2023-03-09T14:44:18.661275+0000","last_became_peered":"2023-03-09T14:44:18.661275+0000","last_unstale":"2023-03-09T14:45:14.859537+0000","last_undegraded":"2023-03-09T14:45:14.859537+0000","last_fullsized":"2023-03-09T14:45:14.859537+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:20:31.873778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":378,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.852517+0000","last_change":"2023-03-09T14:43:19.317554+0000","last_active":"2023-03-09T14:45:14.852517+0000","last_peered":"2023-03-09T14:45:14.852517+0000","last_clean":"2023-03-09T14:45:14.852517+0000","last_became_active":"2023-03-09T14:43:19.316912+0000","last_became_peered":"2023-03-09T14:43:19.316912+0000","last_unstale":"2023-03-09T14:45:14.852517+0000","last_undegraded":"2023-03-09T14:45:14.852517+0000","last_fullsized":"2023-03-09T14:45:14.852517+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:40:23.197173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:40:23.197173+0000","last_clean_scrub_stamp":"2023-03-09T14:40:23.197173+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T20:21:17.143331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.852927+0000","last_change":"2023-03-09T14:44:23.487738+0000","last_active":"2023-03-09T14:45:14.852927+0000","last_peered":"2023-03-09T14:45:14.852927+0000","last_clean":"2023-03-09T14:45:14.852927+0000","last_became_active":"2023-03-09T14:44:23.487193+0000","last_became_peered":"2023-03-09T14:44:23.487193+0000","last_unstale":"2023-03-09T14:45:14.852927+0000","last_undegraded":"2023-03-09T14:45:14.852927+0000","last_fullsized":"2023-03-09T14:45:14.852927+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:48:08.789071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.860260+0000","last_change":"2023-03-09T14:44:19.112576+0000","last_active":"2023-03-09T14:45:14.860260+0000","last_peered":"2023-03-09T14:45:14.860260+0000","last_clean":"2023-03-09T14:45:14.860260+0000","last_became_active":"2023-03-09T14:44:19.112231+0000","last_became_peered":"2023-03-09T14:44:19.112231+0000","last_unstale":"2023-03-09T14:45:14.860260+0000","last_undegraded":"2023-03-09T14:45:14.860260+0000","last_fullsized":"2023-03-09T14:45:14.860260+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:25:21.918088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"87'3","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.852608+0000","last_change":"2023-03-09T14:44:18.655478+0000","last_active":"2023-03-09T14:45:14.852608+0000","last_peered":"2023-03-09T14:45:14.852608+0000","last_clean":"2023-03-09T14:45:14.852608+0000","last_became_active":"2023-03-09T14:44:18.655180+0000","last_became_peered":"2023-03-09T14:44:18.655180+0000","last_unstale":"2023-03-09T14:45:14.852608+0000","last_undegraded":"2023-03-09T14:45:14.852608+0000","last_fullsized":"2023-03-09T14:45:14.852608+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:42:47.926376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.856627+0000","last_change":"2023-03-09T14:44:23.487052+0000","last_active":"2023-03-09T14:45:14.856627+0000","last_peered":"2023-03-09T14:45:14.856627+0000","last_clean":"2023-03-09T14:45:14.856627+0000","last_became_active":"2023-03-09T14:44:23.486603+0000","last_became_peered":"2023-03-09T14:44:23.486603+0000","last_unstale":"2023-03-09T14:45:14.856627+0000","last_undegraded":"2023-03-09T14:45:14.856627+0000","last_fullsized":"2023-03-09T14:45:14.856627+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:28:25.375052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"87'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.861722+0000","last_change":"2023-03-09T14:44:18.666400+0000","last_active":"2023-03-09T14:45:14.861722+0000","last_peered":"2023-03-09T14:45:14.861722+0000","last_clean":"2023-03-09T14:45:14.861722+0000","last_became_active":"2023-03-09T14:44:18.666045+0000","last_became_peered":"2023-03-09T14:44:18.666045+0000","last_unstale":"2023-03-09T14:45:14.861722+0000","last_undegraded":"2023-03-09T14:45:14.861722+0000","last_fullsized":"2023-03-09T14:45:14.861722+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:30:48.640124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.862443+0000","last_change":"2023-03-09T14:44:19.103657+0000","last_active":"2023-03-09T14:45:14.862443+0000","last_peered":"2023-03-09T14:45:14.862443+0000","last_clean":"2023-03-09T14:45:14.862443+0000","last_became_active":"2023-03-09T14:44:19.103364+0000","last_became_peered":"2023-03-09T14:44:19.103364+0000","last_unstale":"2023-03-09T14:45:14.862443+0000","last_undegraded":"2023-03-09T14:45:14.862443+0000","last_fullsized":"2023-03-09T14:45:14.862443+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:23:23.738554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.864043+0000","last_change":"2023-03-09T14:44:23.483487+0000","last_active":"2023-03-09T14:45:14.864043+0000","last_peered":"2023-03-09T14:45:14.864043+0000","last_clean":"2023-03-09T14:45:14.864043+0000","last_became_active":"2023-03-09T14:44:23.483050+0000","last_became_peered":"2023-03-09T14:44:23.483050+0000","last_unstale":"2023-03-09T14:45:14.864043+0000","last_undegraded":"2023-03-09T14:45:14.864043+0000","last_fullsized":"2023-03-09T14:45:14.864043+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:38:34.129355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"87'5","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.860131+0000","last_change":"2023-03-09T14:44:18.661944+0000","last_active":"2023-03-09T14:45:14.860131+0000","last_peered":"2023-03-09T14:45:14.860131+0000","last_clean":"2023-03-09T14:45:14.860131+0000","last_became_active":"2023-03-09T14:44:18.661575+0000","last_became_peered":"2023-03-09T14:44:18.661575+0000","last_unstale":"2023-03-09T14:45:14.860131+0000","last_undegraded":"2023-03-09T14:45:14.860131+0000","last_fullsized":"2023-03-09T14:45:14.860131+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T01:12:53.637866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.864223+0000","last_change":"2023-03-09T14:44:19.103807+0000","last_active":"2023-03-09T14:45:14.864223+0000","last_peered":"2023-03-09T14:45:14.864223+0000","last_clean":"2023-03-09T14:45:14.864223+0000","last_became_active":"2023-03-09T14:44:19.103480+0000","last_became_peered":"2023-03-09T14:44:19.103480+0000","last_unstale":"2023-03-09T14:45:14.864223+0000","last_undegraded":"2023-03-09T14:45:14.864223+0000","last_fullsized":"2023-03-09T14:45:14.864223+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:29:54.629265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.856235+0000","last_change":"2023-03-09T14:44:23.486789+0000","last_active":"2023-03-09T14:45:14.856235+0000","last_peered":"2023-03-09T14:45:14.856235+0000","last_clean":"2023-03-09T14:45:14.856235+0000","last_became_active":"2023-03-09T14:44:23.486367+0000","last_became_peered":"2023-03-09T14:44:23.486367+0000","last_unstale":"2023-03-09T14:45:14.856235+0000","last_undegraded":"2023-03-09T14:45:14.856235+0000","last_fullsized":"2023-03-09T14:45:14.856235+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:13:13.081388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"87'4","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.860080+0000","last_change":"2023-03-09T14:44:18.659492+0000","last_active":"2023-03-09T14:45:14.860080+0000","last_peered":"2023-03-09T14:45:14.860080+0000","last_clean":"2023-03-09T14:45:14.860080+0000","last_became_active":"2023-03-09T14:44:18.659187+0000","last_became_peered":"2023-03-09T14:44:18.659187+0000","last_unstale":"2023-03-09T14:45:14.860080+0000","last_undegraded":"2023-03-09T14:45:14.860080+0000","last_fullsized":"2023-03-09T14:45:14.860080+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:44:31.342785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.864968+0000","last_change":"2023-03-09T14:44:19.112841+0000","last_active":"2023-03-09T14:45:14.864968+0000","last_peered":"2023-03-09T14:45:14.864968+0000","last_clean":"2023-03-09T14:45:14.864968+0000","last_became_active":"2023-03-09T14:44:19.112515+0000","last_became_peered":"2023-03-09T14:44:19.112515+0000","last_unstale":"2023-03-09T14:45:14.864968+0000","last_undegraded":"2023-03-09T14:45:14.864968+0000","last_fullsized":"2023-03-09T14:45:14.864968+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T19:41:04.065529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.859850+0000","last_change":"2023-03-09T14:44:23.481992+0000","last_active":"2023-03-09T14:45:14.859850+0000","last_peered":"2023-03-09T14:45:14.859850+0000","last_clean":"2023-03-09T14:45:14.859850+0000","last_became_active":"2023-03-09T14:44:23.481508+0000","last_became_peered":"2023-03-09T14:44:23.481508+0000","last_unstale":"2023-03-09T14:45:14.859850+0000","last_undegraded":"2023-03-09T14:45:14.859850+0000","last_fullsized":"2023-03-09T14:45:14.859850+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:46:48.683402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"87'2","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.856199+0000","last_change":"2023-03-09T14:44:18.663133+0000","last_active":"2023-03-09T14:45:14.856199+0000","last_peered":"2023-03-09T14:45:14.856199+0000","last_clean":"2023-03-09T14:45:14.856199+0000","last_became_active":"2023-03-09T14:44:18.662809+0000","last_became_peered":"2023-03-09T14:44:18.662809+0000","last_unstale":"2023-03-09T14:45:14.856199+0000","last_undegraded":"2023-03-09T14:45:14.856199+0000","last_fullsized":"2023-03-09T14:45:14.856199+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:32:16.180642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.860368+0000","last_change":"2023-03-09T14:44:19.108566+0000","last_active":"2023-03-09T14:45:14.860368+0000","last_peered":"2023-03-09T14:45:14.860368+0000","last_clean":"2023-03-09T14:45:14.860368+0000","last_became_active":"2023-03-09T14:44:19.108217+0000","last_became_peered":"2023-03-09T14:44:19.108217+0000","last_unstale":"2023-03-09T14:45:14.860368+0000","last_undegraded":"2023-03-09T14:45:14.860368+0000","last_fullsized":"2023-03-09T14:45:14.860368+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:03:28.434395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.860862+0000","last_change":"2023-03-09T14:44:23.488640+0000","last_active":"2023-03-09T14:45:14.860862+0000","last_peered":"2023-03-09T14:45:14.860862+0000","last_clean":"2023-03-09T14:45:14.860862+0000","last_became_active":"2023-03-09T14:44:23.488178+0000","last_became_peered":"2023-03-09T14:44:23.488178+0000","last_unstale":"2023-03-09T14:45:14.860862+0000","last_undegraded":"2023-03-09T14:45:14.860862+0000","last_fullsized":"2023-03-09T14:45:14.860862+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:33:49.908974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"87'3","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.864673+0000","last_change":"2023-03-09T14:44:18.662925+0000","last_active":"2023-03-09T14:45:14.864673+0000","last_peered":"2023-03-09T14:45:14.864673+0000","last_clean":"2023-03-09T14:45:14.864673+0000","last_became_active":"2023-03-09T14:44:18.662354+0000","last_became_peered":"2023-03-09T14:44:18.662354+0000","last_unstale":"2023-03-09T14:45:14.864673+0000","last_undegraded":"2023-03-09T14:45:14.864673+0000","last_fullsized":"2023-03-09T14:45:14.864673+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T01:00:36.602938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.856803+0000","last_change":"2023-03-09T14:44:19.106465+0000","last_active":"2023-03-09T14:45:14.856803+0000","last_peered":"2023-03-09T14:45:14.856803+0000","last_clean":"2023-03-09T14:45:14.856803+0000","last_became_active":"2023-03-09T14:44:19.106001+0000","last_became_peered":"2023-03-09T14:44:19.106001+0000","last_unstale":"2023-03-09T14:45:14.856803+0000","last_undegraded":"2023-03-09T14:45:14.856803+0000","last_fullsized":"2023-03-09T14:45:14.856803+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:22:09.149291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"87'1","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:15.873389+0000","last_change":"2023-03-09T14:45:15.873389+0000","last_active":"2023-03-09T14:45:15.873389+0000","last_peered":"2023-03-09T14:45:15.873389+0000","last_clean":"2023-03-09T14:45:15.873389+0000","last_became_active":"2023-03-09T14:45:15.863408+0000","last_became_peered":"2023-03-09T14:45:15.863408+0000","last_unstale":"2023-03-09T14:45:15.873389+0000","last_undegraded":"2023-03-09T14:45:15.873389+0000","last_fullsized":"2023-03-09T14:45:15.873389+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T16:33:12.565135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":123,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,7,8],"acting":[4,10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"87'1","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.856858+0000","last_change":"2023-03-09T14:44:18.664744+0000","last_active":"2023-03-09T14:45:14.856858+0000","last_peered":"2023-03-09T14:45:14.856858+0000","last_clean":"2023-03-09T14:45:14.856858+0000","last_became_active":"2023-03-09T14:44:18.664443+0000","last_became_peered":"2023-03-09T14:44:18.664443+0000","last_unstale":"2023-03-09T14:45:14.856858+0000","last_undegraded":"2023-03-09T14:45:14.856858+0000","last_fullsized":"2023-03-09T14:45:14.856858+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:11:13.836270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.863353+0000","last_change":"2023-03-09T14:44:19.115102+0000","last_active":"2023-03-09T14:45:14.863353+0000","last_peered":"2023-03-09T14:45:14.863353+0000","last_clean":"2023-03-09T14:45:14.863353+0000","last_became_active":"2023-03-09T14:44:19.114565+0000","last_became_peered":"2023-03-09T14:44:19.114565+0000","last_unstale":"2023-03-09T14:45:14.863353+0000","last_undegraded":"2023-03-09T14:45:14.863353+0000","last_fullsized":"2023-03-09T14:45:14.863353+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:39:11.930231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.859836+0000","last_change":"2023-03-09T14:44:23.482500+0000","last_active":"2023-03-09T14:45:14.859836+0000","last_peered":"2023-03-09T14:45:14.859836+0000","last_clean":"2023-03-09T14:45:14.859836+0000","last_became_active":"2023-03-09T14:44:23.482217+0000","last_became_peered":"2023-03-09T14:44:23.482217+0000","last_unstale":"2023-03-09T14:45:14.859836+0000","last_undegraded":"2023-03-09T14:45:14.859836+0000","last_fullsized":"2023-03-09T14:45:14.859836+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T14:52:44.275037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"87'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.860725+0000","last_change":"2023-03-09T14:44:18.663824+0000","last_active":"2023-03-09T14:45:14.860725+0000","last_peered":"2023-03-09T14:45:14.860725+0000","last_clean":"2023-03-09T14:45:14.860725+0000","last_became_active":"2023-03-09T14:44:18.663379+0000","last_became_peered":"2023-03-09T14:44:18.663379+0000","last_unstale":"2023-03-09T14:45:14.860725+0000","last_undegraded":"2023-03-09T14:45:14.860725+0000","last_fullsized":"2023-03-09T14:45:14.860725+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:03:04.031061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.852902+0000","last_change":"2023-03-09T14:44:19.105765+0000","last_active":"2023-03-09T14:45:14.852902+0000","last_peered":"2023-03-09T14:45:14.852902+0000","last_clean":"2023-03-09T14:45:14.852902+0000","last_became_active":"2023-03-09T14:44:19.105430+0000","last_became_peered":"2023-03-09T14:44:19.105430+0000","last_unstale":"2023-03-09T14:45:14.852902+0000","last_undegraded":"2023-03-09T14:45:14.852902+0000","last_fullsized":"2023-03-09T14:45:14.852902+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T20:31:40.106798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.856901+0000","last_change":"2023-03-09T14:44:23.482079+0000","last_active":"2023-03-09T14:45:14.856901+0000","last_peered":"2023-03-09T14:45:14.856901+0000","last_clean":"2023-03-09T14:45:14.856901+0000","last_became_active":"2023-03-09T14:44:23.481620+0000","last_became_peered":"2023-03-09T14:44:23.481620+0000","last_unstale":"2023-03-09T14:45:14.856901+0000","last_undegraded":"2023-03-09T14:45:14.856901+0000","last_fullsized":"2023-03-09T14:45:14.856901+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:47:53.589111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"87'3","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:15.869980+0000","last_change":"2023-03-09T14:45:15.869485+0000","last_active":"2023-03-09T14:45:15.869980+0000","last_peered":"2023-03-09T14:45:15.869980+0000","last_clean":"2023-03-09T14:45:15.869980+0000","last_became_active":"2023-03-09T14:45:15.861428+0000","last_became_peered":"2023-03-09T14:45:15.861428+0000","last_unstale":"2023-03-09T14:45:15.869980+0000","last_undegraded":"2023-03-09T14:45:15.869980+0000","last_fullsized":"2023-03-09T14:45:15.869980+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:43:30.309431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":566,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.853743+0000","last_change":"2023-03-09T14:44:19.105947+0000","last_active":"2023-03-09T14:45:14.853743+0000","last_peered":"2023-03-09T14:45:14.853743+0000","last_clean":"2023-03-09T14:45:14.853743+0000","last_became_active":"2023-03-09T14:44:19.105669+0000","last_became_peered":"2023-03-09T14:44:19.105669+0000","last_unstale":"2023-03-09T14:45:14.853743+0000","last_undegraded":"2023-03-09T14:45:14.853743+0000","last_fullsized":"2023-03-09T14:45:14.853743+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:29:58.615481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.863833+0000","last_change":"2023-03-09T14:44:23.483237+0000","last_active":"2023-03-09T14:45:14.863833+0000","last_peered":"2023-03-09T14:45:14.863833+0000","last_clean":"2023-03-09T14:45:14.863833+0000","last_became_active":"2023-03-09T14:44:23.482831+0000","last_became_peered":"2023-03-09T14:44:23.482831+0000","last_unstale":"2023-03-09T14:45:14.863833+0000","last_undegraded":"2023-03-09T14:45:14.863833+0000","last_fullsized":"2023-03-09T14:45:14.863833+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:36:40.775581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"82'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:15.867430+0000","last_change":"2023-03-09T14:45:15.867049+0000","last_active":"2023-03-09T14:45:15.867430+0000","last_peered":"2023-03-09T14:45:15.867430+0000","last_clean":"2023-03-09T14:45:15.867430+0000","last_became_active":"2023-03-09T14:45:15.863218+0000","last_became_peered":"2023-03-09T14:45:15.863218+0000","last_unstale":"2023-03-09T14:45:15.867430+0000","last_undegraded":"2023-03-09T14:45:15.867430+0000","last_fullsized":"2023-03-09T14:45:15.867430+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:38:21.800533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.853014+0000","last_change":"2023-03-09T14:44:19.107146+0000","last_active":"2023-03-09T14:45:14.853014+0000","last_peered":"2023-03-09T14:45:14.853014+0000","last_clean":"2023-03-09T14:45:14.853014+0000","last_became_active":"2023-03-09T14:44:19.106834+0000","last_became_peered":"2023-03-09T14:44:19.106834+0000","last_unstale":"2023-03-09T14:45:14.853014+0000","last_undegraded":"2023-03-09T14:45:14.853014+0000","last_fullsized":"2023-03-09T14:45:14.853014+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T20:15:20.142449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.852185+0000","last_change":"2023-03-09T14:44:23.482811+0000","last_active":"2023-03-09T14:45:14.852185+0000","last_peered":"2023-03-09T14:45:14.852185+0000","last_clean":"2023-03-09T14:45:14.852185+0000","last_became_active":"2023-03-09T14:44:23.482371+0000","last_became_peered":"2023-03-09T14:44:23.482371+0000","last_unstale":"2023-03-09T14:45:14.852185+0000","last_undegraded":"2023-03-09T14:45:14.852185+0000","last_fullsized":"2023-03-09T14:45:14.852185+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T01:33:14.442810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"82'2","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.861583+0000","last_change":"2023-03-09T14:44:18.661569+0000","last_active":"2023-03-09T14:45:14.861583+0000","last_peered":"2023-03-09T14:45:14.861583+0000","last_clean":"2023-03-09T14:45:14.861583+0000","last_became_active":"2023-03-09T14:44:18.661266+0000","last_became_peered":"2023-03-09T14:44:18.661266+0000","last_unstale":"2023-03-09T14:45:14.861583+0000","last_undegraded":"2023-03-09T14:45:14.861583+0000","last_fullsized":"2023-03-09T14:45:14.861583+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:47:43.066180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.853803+0000","last_change":"2023-03-09T14:44:19.106611+0000","last_active":"2023-03-09T14:45:14.853803+0000","last_peered":"2023-03-09T14:45:14.853803+0000","last_clean":"2023-03-09T14:45:14.853803+0000","last_became_active":"2023-03-09T14:44:19.106324+0000","last_became_peered":"2023-03-09T14:44:19.106324+0000","last_unstale":"2023-03-09T14:45:14.853803+0000","last_undegraded":"2023-03-09T14:45:14.853803+0000","last_fullsized":"2023-03-09T14:45:14.853803+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:50:46.561502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.856648+0000","last_change":"2023-03-09T14:44:19.106292+0000","last_active":"2023-03-09T14:45:14.856648+0000","last_peered":"2023-03-09T14:45:14.856648+0000","last_clean":"2023-03-09T14:45:14.856648+0000","last_became_active":"2023-03-09T14:44:19.105607+0000","last_became_peered":"2023-03-09T14:44:19.105607+0000","last_unstale":"2023-03-09T14:45:14.856648+0000","last_undegraded":"2023-03-09T14:45:14.856648+0000","last_fullsized":"2023-03-09T14:45:14.856648+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T19:11:28.125433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.856646+0000","last_change":"2023-03-09T14:44:19.103423+0000","last_active":"2023-03-09T14:45:14.856646+0000","last_peered":"2023-03-09T14:45:14.856646+0000","last_clean":"2023-03-09T14:45:14.856646+0000","last_became_active":"2023-03-09T14:44:19.102918+0000","last_became_peered":"2023-03-09T14:44:19.102918+0000","last_unstale":"2023-03-09T14:45:14.856646+0000","last_undegraded":"2023-03-09T14:45:14.856646+0000","last_fullsized":"2023-03-09T14:45:14.856646+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:47:52.921642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.856208+0000","last_change":"2023-03-09T14:44:19.113511+0000","last_active":"2023-03-09T14:45:14.856208+0000","last_peered":"2023-03-09T14:45:14.856208+0000","last_clean":"2023-03-09T14:45:14.856208+0000","last_became_active":"2023-03-09T14:44:19.113224+0000","last_became_peered":"2023-03-09T14:44:19.113224+0000","last_unstale":"2023-03-09T14:45:14.856208+0000","last_undegraded":"2023-03-09T14:45:14.856208+0000","last_fullsized":"2023-03-09T14:45:14.856208+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T02:03:28.069616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.864074+0000","last_change":"2023-03-09T14:44:19.116174+0000","last_active":"2023-03-09T14:45:14.864074+0000","last_peered":"2023-03-09T14:45:14.864074+0000","last_clean":"2023-03-09T14:45:14.864074+0000","last_became_active":"2023-03-09T14:44:19.115813+0000","last_became_peered":"2023-03-09T14:44:19.115813+0000","last_unstale":"2023-03-09T14:45:14.864074+0000","last_undegraded":"2023-03-09T14:45:14.864074+0000","last_fullsized":"2023-03-09T14:45:14.864074+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:28:49.573771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.862549+0000","last_change":"2023-03-09T14:44:19.115287+0000","last_active":"2023-03-09T14:45:14.862549+0000","last_peered":"2023-03-09T14:45:14.862549+0000","last_clean":"2023-03-09T14:45:14.862549+0000","last_became_active":"2023-03-09T14:44:19.113458+0000","last_became_peered":"2023-03-09T14:44:19.113458+0000","last_unstale":"2023-03-09T14:45:14.862549+0000","last_undegraded":"2023-03-09T14:45:14.862549+0000","last_fullsized":"2023-03-09T14:45:14.862549+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:43:26.583237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.860272+0000","last_change":"2023-03-09T14:44:19.107308+0000","last_active":"2023-03-09T14:45:14.860272+0000","last_peered":"2023-03-09T14:45:14.860272+0000","last_clean":"2023-03-09T14:45:14.860272+0000","last_became_active":"2023-03-09T14:44:19.107005+0000","last_became_peered":"2023-03-09T14:44:19.107005+0000","last_unstale":"2023-03-09T14:45:14.860272+0000","last_undegraded":"2023-03-09T14:45:14.860272+0000","last_fullsized":"2023-03-09T14:45:14.860272+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:48:34.886664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.857474+0000","last_change":"2023-03-09T14:44:19.098454+0000","last_active":"2023-03-09T14:45:14.857474+0000","last_peered":"2023-03-09T14:45:14.857474+0000","last_clean":"2023-03-09T14:45:14.857474+0000","last_became_active":"2023-03-09T14:44:19.098146+0000","last_became_peered":"2023-03-09T14:44:19.098146+0000","last_unstale":"2023-03-09T14:45:14.857474+0000","last_undegraded":"2023-03-09T14:45:14.857474+0000","last_fullsized":"2023-03-09T14:45:14.857474+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:11:40.475674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.864884+0000","last_change":"2023-03-09T14:44:19.095947+0000","last_active":"2023-03-09T14:45:14.864884+0000","last_peered":"2023-03-09T14:45:14.864884+0000","last_clean":"2023-03-09T14:45:14.864884+0000","last_became_active":"2023-03-09T14:44:19.095640+0000","last_became_peered":"2023-03-09T14:44:19.095640+0000","last_unstale":"2023-03-09T14:45:14.864884+0000","last_undegraded":"2023-03-09T14:45:14.864884+0000","last_fullsized":"2023-03-09T14:45:14.864884+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T02:04:35.384016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.852390+0000","last_change":"2023-03-09T14:44:19.113927+0000","last_active":"2023-03-09T14:45:14.852390+0000","last_peered":"2023-03-09T14:45:14.852390+0000","last_clean":"2023-03-09T14:45:14.852390+0000","last_became_active":"2023-03-09T14:44:19.113163+0000","last_became_peered":"2023-03-09T14:44:19.113163+0000","last_unstale":"2023-03-09T14:45:14.852390+0000","last_undegraded":"2023-03-09T14:45:14.852390+0000","last_fullsized":"2023-03-09T14:45:14.852390+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T01:35:35.801317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.864582+0000","last_change":"2023-03-09T14:44:19.098415+0000","last_active":"2023-03-09T14:45:14.864582+0000","last_peered":"2023-03-09T14:45:14.864582+0000","last_clean":"2023-03-09T14:45:14.864582+0000","last_became_active":"2023-03-09T14:44:19.097600+0000","last_became_peered":"2023-03-09T14:44:19.097600+0000","last_unstale":"2023-03-09T14:45:14.864582+0000","last_undegraded":"2023-03-09T14:45:14.864582+0000","last_fullsized":"2023-03-09T14:45:14.864582+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:15:08.552402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.857512+0000","last_change":"2023-03-09T14:44:19.106386+0000","last_active":"2023-03-09T14:45:14.857512+0000","last_peered":"2023-03-09T14:45:14.857512+0000","last_clean":"2023-03-09T14:45:14.857512+0000","last_became_active":"2023-03-09T14:44:19.106112+0000","last_became_peered":"2023-03-09T14:44:19.106112+0000","last_unstale":"2023-03-09T14:45:14.857512+0000","last_undegraded":"2023-03-09T14:45:14.857512+0000","last_fullsized":"2023-03-09T14:45:14.857512+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T20:27:50.992168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.861146+0000","last_change":"2023-03-09T14:44:19.114534+0000","last_active":"2023-03-09T14:45:14.861146+0000","last_peered":"2023-03-09T14:45:14.861146+0000","last_clean":"2023-03-09T14:45:14.861146+0000","last_became_active":"2023-03-09T14:44:19.114226+0000","last_became_peered":"2023-03-09T14:44:19.114226+0000","last_unstale":"2023-03-09T14:45:14.861146+0000","last_undegraded":"2023-03-09T14:45:14.861146+0000","last_fullsized":"2023-03-09T14:45:14.861146+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:22:38.453116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.858390+0000","last_change":"2023-03-09T14:44:19.099195+0000","last_active":"2023-03-09T14:45:14.858390+0000","last_peered":"2023-03-09T14:45:14.858390+0000","last_clean":"2023-03-09T14:45:14.858390+0000","last_became_active":"2023-03-09T14:44:19.098892+0000","last_became_peered":"2023-03-09T14:44:19.098892+0000","last_unstale":"2023-03-09T14:45:14.858390+0000","last_undegraded":"2023-03-09T14:45:14.858390+0000","last_fullsized":"2023-03-09T14:45:14.858390+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:29:24.858200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":246,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":123,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":60618,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":262,"num_read_kb":198,"num_write":152,"num_write_kb":104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":1914,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":181854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134,"ondisk_log_size":134,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":78,"seq":335007449112,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6172,"kb_used_data":980,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93738980,"statfs":{"total":95995035648,"available":95988715520,"internally_reserved":0,"allocated":1003520,"data_stored":490897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:44:39 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.623},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.52100000000000002}]},{"osd":1,"last update":"Thu Mar 9 14:44:39 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.90000000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.64400000000000002}]},{"osd":2,"last update":"Thu Mar 9 14:44:39 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":1.2829999999999999,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.78700000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]},{"osd":4,"last update":"Thu Mar 9 14:44:39 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.113,"5min":1.113,"15min":1.113},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.67100000000000004}]},{"osd":5,"last update":"Thu Mar 9 14:44:39 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.70099999999999996}]},{"osd":6,"last update":"Thu Mar 9 14:44:39 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.67800000000000005}]},{"osd":7,"last update":"Thu Mar 9 14:44:39 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"last":0.94299999999999995},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.754}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73599999999999999}]},{"osd":9,"last update":"Thu Mar 9 14:44:39 2023","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.3240000000000001,"5min":1.3240000000000001,"15min":1.3240000000000001},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.96999999999999997}]},{"osd":10,"last update":"Thu Mar 9 14:44:39 2023","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.746,"15min":0.746},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.431,"5min":1.431,"15min":1.431},"last":0.79700000000000004}]}]},{"osd":10,"up_from":70,"seq":300647710747,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7156,"kb_used_data":1644,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93737996,"statfs":{"total":95995035648,"available":95987707904,"internally_reserved":0,"allocated":1683456,"data_stored":1138916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:45:19 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.36099999999999999}]},{"osd":1,"last update":"Thu Mar 9 14:45:19 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.46700000000000003,"15min":0.46999999999999997},"min":{"1min":0.30499999999999999,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.56699999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.47799999999999998,"15min":0.48199999999999998},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.61099999999999999,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.46800000000000003}]},{"osd":2,"last update":"Thu Mar 9 14:45:19 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.63600000000000001,"15min":0.63900000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.753,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.628,"5min":0.623,"15min":0.622},"min":{"1min":0.46400000000000002,"5min":0.441,"15min":0.441},"max":{"1min":0.82199999999999995,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.57599999999999996}]},{"osd":3,"last update":"Thu Mar 9 14:45:19 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.58399999999999996,"15min":0.58899999999999997},"min":{"1min":0.38300000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.74399999999999999,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.64200000000000002,"15min":0.64800000000000002},"min":{"1min":0.46600000000000003,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.76100000000000001,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.59999999999999998}]},{"osd":4,"last update":"Thu Mar 9 14:45:19 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.59099999999999997,"15min":0.59299999999999997},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.78200000000000003,"5min":1.1719999999999999,"15min":1.1719999999999999},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.54500000000000004,"15min":0.54900000000000004},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.77500000000000002,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.52500000000000002}]},{"osd":5,"last update":"Thu Mar 9 14:45:19 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.504,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.92700000000000005,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.62},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.90300000000000002,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.76900000000000002}]},{"osd":6,"last update":"Thu Mar 9 14:45:19 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.68300000000000005,"15min":0.68899999999999995},"min":{"1min":0.48799999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.86499999999999999,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.71099999999999997,"15min":0.71599999999999997},"min":{"1min":0.499,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.82699999999999996,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.73199999999999998}]},{"osd":7,"last update":"Thu Mar 9 14:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.327,"5min":1.327,"15min":1.327},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.048,"5min":1.048,"15min":1.048},"last":0.53800000000000003}]},{"osd":8,"last update":"Thu Mar 9 14:45:19 2023","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.76000000000000001,"15min":0.76200000000000001},"min":{"1min":0.54900000000000004,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":0.91000000000000003,"5min":1.246,"15min":1.246},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.77300000000000002,"15min":0.78200000000000003},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.95499999999999996,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.71199999999999997}]},{"osd":9,"last update":"Thu Mar 9 14:45:19 2023","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.77600000000000002,"15min":0.78400000000000003},"min":{"1min":0.54200000000000004,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.88100000000000001,"5min":1.298,"15min":1.298},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.77700000000000002,"15min":0.79200000000000004},"min":{"1min":0.47499999999999998,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.89200000000000002,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":0.67300000000000004}]},{"osd":11,"last update":"Thu Mar 9 14:44:41 2023","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.68500000000000005}]}]},{"osd":9,"up_from":65,"seq":279172874272,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6376,"kb_used_data":992,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738776,"statfs":{"total":95995035648,"available":95988506624,"internally_reserved":0,"allocated":1015808,"data_stored":491063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:45:00 2023","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.46200000000000002,"15min":0.47399999999999998},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.66300000000000003,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.49399999999999999,"15min":0.50700000000000001},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.63700000000000001,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.36699999999999999}]},{"osd":1,"last update":"Thu Mar 9 14:45:00 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.626,"15min":0.63600000000000001},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.008,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.63700000000000001,"15min":0.65000000000000002},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.79900000000000004,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.46000000000000002}]},{"osd":2,"last update":"Thu Mar 9 14:45:00 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.64600000000000002,"15min":0.65800000000000003},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.66100000000000003,"15min":0.67400000000000004},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.88300000000000001,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.54300000000000004}]},{"osd":3,"last update":"Thu Mar 9 14:45:00 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.56999999999999995,"15min":0.57999999999999996},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.59499999999999997,"15min":0.61499999999999999},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.85099999999999998,"5min":1.016,"15min":1.016},"last":0.34899999999999998}]},{"osd":4,"last update":"Thu Mar 9 14:45:00 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.67900000000000005,"15min":0.69699999999999995},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.80400000000000005,"5min":1.04,"15min":1.04},"last":0.505},{"interface":"front","average":{"1min":0.621,"5min":0.71699999999999997,"15min":0.73299999999999998},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.93899999999999995,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.48899999999999999}]},{"osd":5,"last update":"Thu Mar 9 14:45:00 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.73299999999999998,"15min":0.745},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.98599999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.74099999999999999,"15min":0.754},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.0369999999999999,"5min":1.069,"15min":1.069},"last":0.59299999999999997}]},{"osd":6,"last update":"Thu Mar 9 14:43:55 2023","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.623}]},{"osd":7,"last update":"Thu Mar 9 14:45:00 2023","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.79800000000000004,"15min":0.82599999999999996},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.0229999999999999,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.80900000000000005,"15min":0.83099999999999996},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.99299999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.497}]},{"osd":8,"last update":"Thu Mar 9 14:45:00 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.80200000000000005,"15min":0.81599999999999995},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.032,"5min":1.133,"15min":1.133},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.81999999999999995,"15min":0.83799999999999997},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.081,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.625}]},{"osd":10,"last update":"Thu Mar 9 14:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.57999999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65000000000000002}]}]},{"osd":8,"up_from":57,"seq":244813135908,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6496,"kb_used_data":984,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738656,"statfs":{"total":95995035648,"available":95988383744,"internally_reserved":0,"allocated":1007616,"data_stored":490086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.33300000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.69199999999999995,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54900000000000004,"15min":0.55100000000000005},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.83799999999999997,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.42799999999999999}]},{"osd":1,"last update":"Thu Mar 9 14:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.54700000000000004,"15min":0.53900000000000003},"min":{"1min":0.376,"5min":0.313,"15min":0.313},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.56100000000000005,"15min":0.55900000000000005},"min":{"1min":0.34399999999999997,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.65700000000000003}]},{"osd":2,"last update":"Thu Mar 9 14:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.42199999999999999,"15min":0.41599999999999998},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.432,"15min":0.42699999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.45100000000000001}]},{"osd":3,"last update":"Thu Mar 9 14:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.55900000000000005,"15min":0.56100000000000005},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.81200000000000006,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.63100000000000001,"15min":0.61499999999999999},"min":{"1min":0.40600000000000003,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":3.9300000000000002,"5min":3.9300000000000002,"15min":3.9300000000000002},"last":0.61899999999999999}]},{"osd":4,"last update":"Thu Mar 9 14:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.56399999999999995,"15min":0.55600000000000005},"min":{"1min":0.36599999999999999,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.59299999999999997,"15min":0.58499999999999996},"min":{"1min":0.45600000000000002,"5min":0.376,"15min":0.376},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.77000000000000002}]},{"osd":5,"last update":"Thu Mar 9 14:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.53400000000000003,"15min":0.53200000000000003},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.54200000000000004,"15min":0.53600000000000003},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.57899999999999996}]},{"osd":6,"last update":"Thu Mar 9 14:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.69599999999999995,"15min":0.67900000000000005},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":3.895,"5min":3.895,"15min":3.895},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.65000000000000002,"15min":0.64900000000000002},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.68899999999999995}]},{"osd":7,"last update":"Thu Mar 9 14:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.66400000000000003,"15min":0.66200000000000003},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.66800000000000004,"15min":0.66300000000000003},"min":{"1min":0.49399999999999999,"5min":0.44,"15min":0.44},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.747}]},{"osd":9,"last update":"Thu Mar 9 14:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.67600000000000005,"15min":0.66800000000000004},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.036,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.754,"5min":0.68600000000000005,"15min":0.67500000000000004},"min":{"1min":0.52900000000000003,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.119,"5min":1.119,"15min":1.119},"last":0.79200000000000004}]},{"osd":10,"last update":"Thu Mar 9 14:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.1319999999999999,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.84799999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80900000000000005}]}]},{"osd":7,"up_from":50,"seq":214748364841,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7148,"kb_used_data":1572,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738004,"statfs":{"total":95995035648,"available":95987716096,"internally_reserved":0,"allocated":1609728,"data_stored":1086891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:44:13 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.52100000000000002,"15min":0.51700000000000002},"min":{"1min":0.316,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.69999999999999996,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.375},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.51900000000000002,"15min":0.50900000000000001},"min":{"1min":0.32800000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.90100000000000002,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.33900000000000002}]},{"osd":1,"last update":"Thu Mar 9 14:45:14 2023","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.40600000000000003,"15min":0.39400000000000002},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.54500000000000004,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.40600000000000003,"15min":0.39100000000000001},"min":{"1min":0.29799999999999999,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.56999999999999995,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.38700000000000001}]},{"osd":2,"last update":"Thu Mar 9 14:45:14 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.54600000000000004,"15min":0.53400000000000003},"min":{"1min":0.33500000000000002,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.754,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.58399999999999996,"15min":0.56999999999999995},"min":{"1min":0.44600000000000001,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.80700000000000005,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.505}]},{"osd":3,"last update":"Thu Mar 9 14:45:14 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.55700000000000005,"15min":0.53800000000000003},"min":{"1min":0.40200000000000002,"5min":0.37,"15min":0.37},"max":{"1min":0.76000000000000001,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.57999999999999996,"15min":0.56399999999999995},"min":{"1min":0.44,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.76400000000000001,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.65200000000000002}]},{"osd":4,"last update":"Thu Mar 9 14:45:14 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48599999999999999,"15min":0.47199999999999998},"min":{"1min":0.34799999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.65100000000000002,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.47599999999999998,"15min":0.46000000000000002},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.68100000000000005,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.57599999999999996}]},{"osd":5,"last update":"Thu Mar 9 14:45:14 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.58799999999999997,"15min":0.57399999999999995},"min":{"1min":0.40300000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.76200000000000001,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.623,"5min":0.59499999999999997,"15min":0.57799999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.80200000000000005,"5min":0.874,"15min":0.874},"last":0.625}]},{"osd":6,"last update":"Thu Mar 9 14:45:14 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.63300000000000001,"15min":0.61699999999999999},"min":{"1min":0.50600000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.623,"15min":0.61299999999999999},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.81000000000000005,"5min":0.878,"15min":0.878},"last":0.66700000000000004}]},{"osd":8,"last update":"Thu Mar 9 14:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.65400000000000003,"15min":0.65500000000000003},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.85199999999999998,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.64900000000000002,"15min":0.64700000000000002},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.88300000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.69799999999999995}]},{"osd":9,"last update":"Thu Mar 9 14:44:55 2023","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.69299999999999995,"15min":0.69199999999999995},"min":{"1min":0.52600000000000002,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":0.91400000000000003,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.70199999999999996,"15min":0.70399999999999996},"min":{"1min":0.52000000000000002,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":0.88700000000000001,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.76800000000000002}]},{"osd":10,"last update":"Thu Mar 9 14:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.81200000000000006}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78000000000000003}]}]},{"osd":6,"up_from":45,"seq":193273528366,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6488,"kb_used_data":976,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738664,"statfs":{"total":95995035648,"available":95988391936,"internally_reserved":0,"allocated":999424,"data_stored":486345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:44:55 2023","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.41599999999999998,"15min":0.41399999999999998},"min":{"1min":0.28799999999999998,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.56000000000000005,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.498,"15min":0.49199999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.70599999999999996,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.53400000000000003}]},{"osd":1,"last update":"Thu Mar 9 14:44:55 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.58299999999999996,"15min":0.59899999999999998},"min":{"1min":0.36599999999999999,"5min":0.315,"15min":0.315},"max":{"1min":0.61399999999999999,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.55700000000000005}]},{"osd":2,"last update":"Thu Mar 9 14:44:55 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.60999999999999999,"15min":0.63700000000000001},"min":{"1min":0.37,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.72299999999999998,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.496},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.63,"15min":0.63600000000000001},"min":{"1min":0.41799999999999998,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.78300000000000003,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.59099999999999997}]},{"osd":3,"last update":"Thu Mar 9 14:44:55 2023","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46500000000000002,"15min":0.46300000000000002},"min":{"1min":0.30499999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.69699999999999995,"5min":0.745,"15min":0.745},"last":0.436},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.56399999999999995,"15min":0.55800000000000005},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.84199999999999997,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.57199999999999995}]},{"osd":4,"last update":"Thu Mar 9 14:44:55 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.68600000000000005,"15min":0.73699999999999999},"min":{"1min":0.40600000000000003,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.67100000000000004,"5min":2.0840000000000001,"15min":2.0840000000000001},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.81699999999999995,"15min":0.89500000000000002},"min":{"1min":0.47699999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.82599999999999996,"5min":4.1440000000000001,"15min":4.1440000000000001},"last":0.69299999999999995}]},{"osd":5,"last update":"Thu Mar 9 14:44:55 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.69699999999999995,"15min":0.69999999999999996},"min":{"1min":0.51400000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.83899999999999997,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.66000000000000003,"15min":0.68200000000000005},"min":{"1min":0.47799999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.74399999999999999,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.61499999999999999}]},{"osd":7,"last update":"Thu Mar 9 14:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.69999999999999996,"15min":0.70899999999999996},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.629},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.72999999999999998,"15min":0.73599999999999999},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.96499999999999997,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.76300000000000001}]},{"osd":8,"last update":"Thu Mar 9 14:44:35 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.77400000000000002,"15min":0.78800000000000003},"min":{"1min":0.54200000000000004,"5min":0.497,"15min":0.497},"max":{"1min":0.92500000000000004,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.70599999999999996,"15min":0.72199999999999998},"min":{"1min":0.502,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.81200000000000006,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.68100000000000005}]},{"osd":9,"last update":"Thu Mar 9 14:44:55 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.67000000000000004,"15min":0.68200000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.88300000000000001,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.73499999999999999,"15min":0.751},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.92700000000000005,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.60599999999999998}]},{"osd":10,"last update":"Thu Mar 9 14:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.79000000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.749}]}]},{"osd":1,"up_from":13,"seq":55834574914,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7572,"kb_used_data":1036,"kb_used_omap":0,"kb_used_meta":6528,"kb_avail":93737580,"statfs":{"total":95995035648,"available":95987281920,"internally_reserved":0,"allocated":1060864,"data_stored":543397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6684672},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:45:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.50900000000000001,"15min":0.49299999999999999},"min":{"1min":0.36599999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.63300000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.55100000000000005,"15min":0.497},"min":{"1min":0.42099999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.88700000000000001,"5min":2.0670000000000002,"15min":2.0670000000000002},"last":0.66100000000000003}]},{"osd":2,"last update":"Thu Mar 9 14:44:31 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.54800000000000004,"15min":0.53000000000000003},"min":{"1min":0.38100000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.85399999999999998,"5min":1.3839999999999999,"15min":1.3839999999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.52700000000000002,"15min":0.50700000000000001},"min":{"1min":0.376,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.77600000000000002,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.64100000000000001}]},{"osd":3,"last update":"Thu Mar 9 14:44:55 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.55100000000000005,"15min":0.50900000000000001},"min":{"1min":0.39000000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.73399999999999999,"5min":2.206,"15min":2.206},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.56200000000000006,"15min":0.52600000000000002},"min":{"1min":0.39700000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.84499999999999997,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.76600000000000001}]},{"osd":4,"last update":"Thu Mar 9 14:45:18 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.441,"15min":0.39500000000000002},"min":{"1min":0.39900000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.434,"15min":0.39000000000000001},"min":{"1min":0.34100000000000003,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.434}]},{"osd":5,"last update":"Thu Mar 9 14:44:31 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.57299999999999995,"15min":0.54600000000000004},"min":{"1min":0.47999999999999998,"5min":0.373,"15min":0.373},"max":{"1min":0.92800000000000005,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.57599999999999996,"15min":0.54600000000000004},"min":{"1min":0.504,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.85799999999999998,"5min":1.363,"15min":1.363},"last":0.755}]},{"osd":6,"last update":"Thu Mar 9 14:44:55 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.64800000000000002,"15min":0.65400000000000003},"min":{"1min":0.376,"5min":0.371,"15min":0.371},"max":{"1min":0.84699999999999998,"5min":2.1930000000000001,"15min":2.1930000000000001},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.65600000000000003,"15min":0.66400000000000003},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.81699999999999995,"5min":2.2519999999999998,"15min":2.2519999999999998},"last":0.82899999999999996}]},{"osd":7,"last update":"Thu Mar 9 14:45:11 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.55100000000000005,"15min":0.5},"min":{"1min":0.33800000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.55400000000000005,"15min":0.51600000000000001},"min":{"1min":0.35799999999999998,"5min":0.31,"15min":0.31},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.61399999999999999}]},{"osd":8,"last update":"Thu Mar 9 14:44:33 2023","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.624,"15min":0.61199999999999999},"min":{"1min":0.503,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.88100000000000001,"5min":1.3009999999999999,"15min":1.3009999999999999},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.68100000000000005,"15min":0.67100000000000004},"min":{"1min":0.496,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.98299999999999998,"5min":1.423,"15min":1.423},"last":0.871}]},{"osd":9,"last update":"Thu Mar 9 14:44:58 2023","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.68200000000000005,"15min":0.66400000000000003},"min":{"1min":0.49399999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.68799999999999994,"15min":0.67300000000000004},"min":{"1min":0.53300000000000003,"5min":0.376,"15min":0.376},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.79000000000000004}]},{"osd":10,"last update":"Thu Mar 9 14:45:19 2023","interfaces":[{"interface":"back","average":{"1min":0.86799999999999999,"5min":0.75600000000000001,"15min":0.73699999999999999},"min":{"1min":0.66800000000000004,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.77100000000000002,"15min":0.75800000000000001},"min":{"1min":0.68000000000000005,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.0109999999999999,"5min":1.022,"15min":1.022},"last":0.82099999999999995}]},{"osd":11,"last update":"Thu Mar 9 14:45:18 2023","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.1559999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.89400000000000002},{"interface":"front","average":{"1min":0.873,"5min":0.873,"15min":0.873},"min":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.85799999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738440,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7340,"kb_used_data":996,"kb_used_omap":0,"kb_used_meta":6336,"kb_avail":93737812,"statfs":{"total":95995035648,"available":95987519488,"internally_reserved":0,"allocated":1019904,"data_stored":487611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6488064},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Mar 9 14:45:11 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.55200000000000005,"15min":0.52200000000000002},"min":{"1min":0.375,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.67700000000000005,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.54300000000000004,"15min":0.52700000000000002},"min":{"1min":0.371,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.78900000000000003,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.66700000000000004}]},{"osd":2,"last update":"Thu Mar 9 14:44:31 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.55800000000000005,"15min":0.55300000000000005},"min":{"1min":0.41599999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.71399999999999997,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.59999999999999998,"15min":0.57999999999999996},"min":{"1min":0.49199999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.76000000000000001,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.54100000000000004}]},{"osd":3,"last update":"Thu Mar 9 14:42:45 2023","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.35799999999999998,"15min":0.34999999999999998},"min":{"1min":0.27800000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.39200000000000002,"15min":0.38800000000000001},"min":{"1min":0.26400000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.59499999999999997,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.38600000000000001}]},{"osd":4,"last update":"Thu Mar 9 14:45:11 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.59699999999999998,"15min":0.57099999999999995},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.78400000000000003,"5min":1.139,"15min":1.139},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.55800000000000005,"15min":0.54600000000000004},"min":{"1min":0.38700000000000001,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.88900000000000001,"5min":1.071,"15min":1.071},"last":0.54900000000000004}]},{"osd":5,"last update":"Thu Mar 9 14:44:37 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.58499999999999996,"15min":0.56299999999999994},"min":{"1min":0.48599999999999999,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.748,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.64500000000000002,"15min":0.626},"min":{"1min":0.46999999999999997,"5min":0.378,"15min":0.378},"max":{"1min":0.78600000000000003,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.58099999999999996}]},{"osd":6,"last update":"Thu Mar 9 14:44:52 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.53400000000000003,"15min":0.504},"min":{"1min":0.374,"5min":0.309,"15min":0.309},"max":{"1min":0.77900000000000003,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.44},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.57799999999999996,"15min":0.54800000000000004},"min":{"1min":0.38300000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.745,"5min":1.159,"15min":1.159},"last":0.496}]},{"osd":7,"last update":"Thu Mar 9 14:44:14 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.71299999999999997,"15min":0.71099999999999997},"min":{"1min":0.503,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.0760000000000001,"5min":1.2110000000000001,"15min":1.2110000000000001},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.64900000000000002,"15min":0.64500000000000002},"min":{"1min":0.52700000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.88500000000000001,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.70299999999999996}]},{"osd":8,"last update":"Thu Mar 9 14:44:37 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.72199999999999998,"15min":0.73299999999999998},"min":{"1min":0.57299999999999995,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.77900000000000003,"5min":1.038,"15min":1.038},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.80600000000000005,"15min":0.82299999999999995},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.81399999999999995,"5min":1.242,"15min":1.242},"last":0.56499999999999995}]},{"osd":9,"last update":"Thu Mar 9 14:44:59 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.74299999999999999,"15min":0.75900000000000001},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.90600000000000003,"5min":1.105,"15min":1.105},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.82199999999999995,"15min":0.85399999999999998},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.88800000000000001,"5min":1.27,"15min":1.27},"last":0.59599999999999997}]},{"osd":10,"last update":"Thu Mar 9 14:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.151,"5min":1.151,"15min":1.151},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.626}]},{"osd":11,"last update":"Thu Mar 9 14:44:37 2023","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.65700000000000003}]}]},{"osd":2,"up_from":18,"seq":77309411391,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7580,"kb_used_data":1044,"kb_used_omap":0,"kb_used_meta":6528,"kb_avail":93737572,"statfs":{"total":95995035648,"available":95987273728,"internally_reserved":0,"allocated":1069056,"data_stored":542977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6684672},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:44:30 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.51500000000000001,"15min":0.495},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.436},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.47899999999999998,"15min":0.46899999999999997},"min":{"1min":0.30199999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.32400000000000001}]},{"osd":1,"last update":"Thu Mar 9 14:44:30 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.53500000000000003,"15min":0.499},"min":{"1min":0.30199999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.54600000000000004,"15min":0.505},"min":{"1min":0.41799999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":1.1890000000000001,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.66500000000000004}]},{"osd":3,"last update":"Thu Mar 9 14:44:51 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.57299999999999995,"15min":0.54100000000000004},"min":{"1min":0.38800000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.54500000000000004,"15min":0.53300000000000003},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.48899999999999999}]},{"osd":4,"last update":"Thu Mar 9 14:44:15 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.54900000000000004,"15min":0.52500000000000002},"min":{"1min":0.41599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.56699999999999995,"15min":0.54500000000000004},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.234,"5min":1.234,"15min":1.234},"last":0.73399999999999999}]},{"osd":5,"last update":"Thu Mar 9 14:44:35 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.47199999999999998,"15min":0.46000000000000002},"min":{"1min":0.313,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.45200000000000001,"15min":0.44},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.39400000000000002}]},{"osd":6,"last update":"Thu Mar 9 14:44:52 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59199999999999997,"15min":0.57899999999999996},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.504},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.63,"15min":0.61599999999999999},"min":{"1min":0.41699999999999998,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.89200000000000002,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.63800000000000001}]},{"osd":7,"last update":"Thu Mar 9 14:44:15 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.2450000000000001,"5min":1.2450000000000001,"15min":1.2450000000000001},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.69699999999999995,"15min":0.69299999999999995},"min":{"1min":0.46100000000000002,"5min":0.375,"15min":0.375},"max":{"1min":1.2649999999999999,"5min":1.2649999999999999,"15min":1.2649999999999999},"last":0.76700000000000002}]},{"osd":8,"last update":"Thu Mar 9 14:44:35 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.65400000000000003,"15min":0.65100000000000002},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.94699999999999995,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.64600000000000002,"15min":0.65200000000000002},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.93999999999999995,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.621}]},{"osd":9,"last update":"Thu Mar 9 14:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.72399999999999998,"15min":0.72999999999999998},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.0109999999999999,"5min":1.21,"15min":1.21},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.67100000000000004,"15min":0.68000000000000005},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.93200000000000005,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.52700000000000002}]},{"osd":10,"last update":"Thu Mar 9 14:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.2529999999999999,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.306,"5min":1.306,"15min":1.306},"last":0.71199999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71999999999999997}]}]},{"osd":3,"up_from":25,"seq":107374182458,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6752,"kb_used_data":984,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738400,"statfs":{"total":95995035648,"available":95988121600,"internally_reserved":0,"allocated":1007616,"data_stored":486449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:44:48 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.36399999999999999,"15min":0.32800000000000001},"min":{"1min":0.308,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.66500000000000004,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.379,"15min":0.34300000000000003},"min":{"1min":0.32400000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.64000000000000001,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.39300000000000002}]},{"osd":1,"last update":"Thu Mar 9 14:44:48 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.53000000000000003,"15min":0.504},"min":{"1min":0.42799999999999999,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.74299999999999999,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.55100000000000005,"15min":0.496},"min":{"1min":0.40400000000000003,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.59099999999999997}]},{"osd":2,"last update":"Thu Mar 9 14:44:48 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.55800000000000005,"15min":0.51600000000000001},"min":{"1min":0.49099999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.82499999999999996,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.629},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.56799999999999995,"15min":0.53200000000000003},"min":{"1min":0.45300000000000001,"5min":0.312,"15min":0.312},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.57699999999999996}]},{"osd":4,"last update":"Thu Mar 9 14:45:10 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.58099999999999996,"15min":0.54200000000000004},"min":{"1min":0.45200000000000001,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.75900000000000001,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.622,"5min":0.60099999999999998,"15min":0.55000000000000004},"min":{"1min":0.497,"5min":0.373,"15min":0.373},"max":{"1min":0.84599999999999997,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.499}]},{"osd":5,"last update":"Thu Mar 9 14:44:32 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.56699999999999995,"15min":0.54300000000000004},"min":{"1min":0.433,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.60099999999999998,"15min":0.56499999999999995},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.621}]},{"osd":6,"last update":"Thu Mar 9 14:44:51 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.56000000000000005,"15min":0.53500000000000003},"min":{"1min":0.44,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.95799999999999996,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.57099999999999995,"15min":0.54300000000000004},"min":{"1min":0.438,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.97499999999999998,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.34499999999999997}]},{"osd":7,"last update":"Thu Mar 9 14:45:17 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.69399999999999995,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.68000000000000005,"15min":0.66700000000000004},"min":{"1min":0.46500000000000002,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.80400000000000005,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.54400000000000004}]},{"osd":8,"last update":"Thu Mar 9 14:43:35 2023","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.749}]},{"osd":9,"last update":"Thu Mar 9 14:44:54 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.76800000000000002,"15min":0.78800000000000003},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.1160000000000001,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.63,"5min":0.77600000000000002,"15min":0.80000000000000004},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.92300000000000004,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.42799999999999999}]},{"osd":10,"last update":"Thu Mar 9 14:44:20 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.155,"5min":1.155,"15min":1.155},"last":0.65700000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68200000000000005}]}]},{"osd":4,"up_from":32,"seq":137438953526,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6636,"kb_used_data":996,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738516,"statfs":{"total":95995035648,"available":95988240384,"internally_reserved":0,"allocated":1019904,"data_stored":492483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:44:09 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.53100000000000003,"15min":0.495},"min":{"1min":0.39000000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.84099999999999997,"5min":1.8320000000000001,"15min":1.8320000000000001},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.628,"5min":0.52900000000000003,"15min":0.495},"min":{"1min":0.42499999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":1.044,"5min":1.478,"15min":1.478},"last":0.52500000000000002}]},{"osd":1,"last update":"Thu Mar 9 14:45:14 2023","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.376,"15min":0.34799999999999998},"min":{"1min":0.311,"5min":0.253,"15min":0.253},"max":{"1min":0.53100000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.311},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.39600000000000002,"15min":0.36799999999999999},"min":{"1min":0.316,"5min":0.253,"15min":0.253},"max":{"1min":0.625,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.33200000000000002}]},{"osd":2,"last update":"Thu Mar 9 14:45:14 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.54100000000000004,"15min":0.53000000000000003},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.087,"5min":1.087,"15min":1.087},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.57199999999999995,"15min":0.53900000000000003},"min":{"1min":0.41499999999999998,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.56499999999999995}]},{"osd":3,"last update":"Thu Mar 9 14:45:14 2023","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":0.69899999999999995,"15min":0.57099999999999995},"min":{"1min":0.47699999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":9.9420000000000002,"5min":9.9420000000000002,"15min":9.9420000000000002},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.59699999999999998,"15min":0.54100000000000004},"min":{"1min":0.45200000000000001,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":2.6110000000000002,"5min":2.6110000000000002,"15min":2.6110000000000002},"last":0.45900000000000002}]},{"osd":5,"last update":"Thu Mar 9 14:44:33 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.58099999999999996,"15min":0.55900000000000005},"min":{"1min":0.45800000000000002,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.59399999999999997,"15min":0.56599999999999995},"min":{"1min":0.47099999999999997,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.495}]},{"osd":6,"last update":"Thu Mar 9 14:44:53 2023","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.69399999999999995,"15min":0.69699999999999995},"min":{"1min":0.45900000000000002,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":2.4609999999999999,"5min":2.4609999999999999,"15min":2.4609999999999999},"last":0.622},{"interface":"front","average":{"1min":0.92800000000000005,"5min":0.72499999999999998,"15min":0.69199999999999995},"min":{"1min":0.52100000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":4.6879999999999997,"5min":4.6879999999999997,"15min":4.6879999999999997},"last":0.54000000000000004}]},{"osd":7,"last update":"Thu Mar 9 14:45:14 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.56999999999999995,"15min":0.53400000000000003},"min":{"1min":0.42299999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.872,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.58199999999999996,"15min":0.54000000000000004},"min":{"1min":0.437,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.77400000000000002,"5min":0.998,"15min":0.998},"last":0.437}]},{"osd":8,"last update":"Thu Mar 9 14:44:36 2023","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.70799999999999996,"15min":0.68799999999999994},"min":{"1min":0.53000000000000003,"5min":0.377,"15min":0.377},"max":{"1min":1.7569999999999999,"5min":1.7569999999999999,"15min":1.7569999999999999},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.71499999999999997,"15min":0.69399999999999995},"min":{"1min":0.53800000000000003,"5min":0.498,"15min":0.498},"max":{"1min":1.746,"5min":1.746,"15min":1.746},"last":0.58999999999999997}]},{"osd":9,"last update":"Thu Mar 9 14:44:58 2023","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.78000000000000003,"15min":0.76100000000000001},"min":{"1min":0.59599999999999997,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":2.5419999999999998,"5min":2.5419999999999998,"15min":2.5419999999999998},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.72799999999999998,"15min":0.71399999999999997},"min":{"1min":0.55200000000000005,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":2.4969999999999999,"5min":2.4969999999999999,"15min":2.4969999999999999},"last":0.57999999999999996}]},{"osd":10,"last update":"Thu Mar 9 14:44:20 2023","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.228,"5min":1.228,"15min":1.228},"last":0.69799999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73299999999999998}]}]},{"osd":5,"up_from":37,"seq":158913790002,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7272,"kb_used_data":1568,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93737880,"statfs":{"total":95995035648,"available":95987589120,"internally_reserved":0,"allocated":1605632,"data_stored":1081598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.51200000000000001,"15min":0.48599999999999999},"min":{"1min":0.35599999999999998,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.80000000000000004,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.51900000000000002,"15min":0.5},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.69899999999999995,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.70299999999999996}]},{"osd":1,"last update":"Thu Mar 9 14:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55500000000000005,"15min":0.54700000000000004},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.83599999999999997,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.54300000000000004,"15min":0.53100000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.46200000000000002}]},{"osd":2,"last update":"Thu Mar 9 14:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.375,"15min":0.35999999999999999},"min":{"1min":0.249,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.60099999999999998,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.38900000000000001,"15min":0.36899999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.39700000000000002}]},{"osd":3,"last update":"Thu Mar 9 14:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.54000000000000004,"15min":0.53500000000000003},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.71399999999999997,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56699999999999995,"15min":0.55100000000000005},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.752,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.71199999999999997}]},{"osd":4,"last update":"Thu Mar 9 14:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.56999999999999995,"15min":0.55600000000000005},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.76200000000000001,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.629},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53000000000000003,"15min":0.52600000000000002},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.70399999999999996,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.42799999999999999}]},{"osd":6,"last update":"Thu Mar 9 14:44:52 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60099999999999998,"15min":0.59999999999999998},"min":{"1min":0.39200000000000002,"5min":0.315,"15min":0.315},"max":{"1min":0.79200000000000004,"5min":0.879,"15min":0.879},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.64000000000000001,"15min":0.627},"min":{"1min":0.42899999999999999,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.73399999999999999}]},{"osd":7,"last update":"Thu Mar 9 14:44:15 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.59099999999999997,"15min":0.58699999999999997},"min":{"1min":0.437,"5min":0.38,"15min":0.38},"max":{"1min":0.81000000000000005,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.59499999999999997,"15min":0.59199999999999997},"min":{"1min":0.41699999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.78700000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.60199999999999998}]},{"osd":8,"last update":"Thu Mar 9 14:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.59499999999999997,"15min":0.59899999999999998},"min":{"1min":0.35099999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.84099999999999997,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.54200000000000004,"15min":0.54400000000000004},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.50900000000000001}]},{"osd":9,"last update":"Thu Mar 9 14:44:58 2023","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.67800000000000005,"15min":0.64600000000000002},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":5.6749999999999998,"5min":5.6749999999999998,"15min":5.6749999999999998},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.71599999999999997,"15min":0.71499999999999997},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.468,"5min":1.468,"15min":1.468},"last":0.75600000000000001}]},{"osd":10,"last update":"Thu Mar 9 14:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.69199999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":57232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":56812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":58618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-03-09T14:45:22.702 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph pg dump --format=json 2023-03-09T14:45:23.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:22 smithi038 ceph-mon[92422]: from='client.24872 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-03-09T14:45:23.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:22 smithi122 ceph-mon[93869]: from='client.24872 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-03-09T14:45:23.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:22 smithi181 ceph-mon[94189]: from='client.24872 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-03-09T14:45:23.368 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.a/config 2023-03-09T14:45:24.373 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:24 smithi038 ceph-mon[92422]: pgmap v293: 129 pgs: 129 active+clean; 636 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s; 254 B/s, 1 objects/s recovering 2023-03-09T14:45:24.373 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:24 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:45:24.374 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:24 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:45:24.374 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:24 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:45:24.374 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:24 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:45:24.552 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:24.548+0000 7f92a1ec6700 1 -- 172.21.15.38:0/2265257547 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f929c101b10 msgr2=0x7f929c10a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:24.553 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:24.548+0000 7f92a1ec6700 1 --2- 172.21.15.38:0/2265257547 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f929c101b10 0x7f929c10a000 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f9290009990 tx=0x7f9290009ca0 comp rx=0 tx=0).stop 2023-03-09T14:45:24.553 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:24.548+0000 7f92a1ec6700 1 -- 172.21.15.38:0/2265257547 shutdown_connections 2023-03-09T14:45:24.553 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:24.548+0000 7f92a1ec6700 1 --2- 172.21.15.38:0/2265257547 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f929c10a540 0x7f929c10c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:24.553 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:24.548+0000 7f92a1ec6700 1 --2- 172.21.15.38:0/2265257547 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f929c101200 0x7f929c1015d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:24.553 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:24.548+0000 7f92a1ec6700 1 --2- 172.21.15.38:0/2265257547 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f929c101b10 0x7f929c10a000 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:24.553 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:24.548+0000 7f92a1ec6700 1 -- 172.21.15.38:0/2265257547 >> 172.21.15.38:0/2265257547 conn(0x7f929c076d10 msgr2=0x7f929c077110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:24.554 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:24.548+0000 7f92a1ec6700 1 -- 172.21.15.38:0/2265257547 shutdown_connections 2023-03-09T14:45:24.554 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:24.548+0000 7f92a1ec6700 1 -- 172.21.15.38:0/2265257547 wait complete. 2023-03-09T14:45:24.554 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:24.548+0000 7f92a1ec6700 1 Processor -- start 2023-03-09T14:45:24.554 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:24.548+0000 7f92a1ec6700 1 -- start start 2023-03-09T14:45:24.554 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:24.549+0000 7f92a1ec6700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f929c101200 0x7f929c0fee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:24.554 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:24.549+0000 7f92a1ec6700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f929c101b10 0x7f929c0ff390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:24.554 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:24.549+0000 7f92a1ec6700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f929c10a540 0x7f929c1a42b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:24.555 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:24.549+0000 7f92a1ec6700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f929c0ffbc0 con 0x7f929c101200 2023-03-09T14:45:24.555 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:24.549+0000 7f92a1ec6700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f929c1a47f0 con 0x7f929c10a540 2023-03-09T14:45:24.555 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:24.549+0000 7f92a1ec6700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f929c1a4930 con 0x7f929c101b10 2023-03-09T14:45:24.555 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:24.549+0000 7f929b7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f929c101200 0x7f929c0fee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:24.555 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:24.549+0000 7f929b7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f929c101200 0x7f929c0fee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37678/0 (socket says 172.21.15.38:37678) 2023-03-09T14:45:24.555 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:24.549+0000 7f929b7fe700 1 -- 172.21.15.38:0/213916005 learned_addr learned my addr 172.21.15.38:0/213916005 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:24.555 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:24.549+0000 7f929affd700 1 --2- 172.21.15.38:0/213916005 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f929c101b10 0x7f929c0ff390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:24.555 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:24.549+0000 7f929bfff700 1 --2- 172.21.15.38:0/213916005 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f929c10a540 0x7f929c1a42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:24.556 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:24.549+0000 7f929b7fe700 1 -- 172.21.15.38:0/213916005 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f929c101b10 msgr2=0x7f929c0ff390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:24.556 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:24.549+0000 7f929b7fe700 1 --2- 172.21.15.38:0/213916005 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f929c101b10 0x7f929c0ff390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:24.556 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:24.549+0000 7f929b7fe700 1 -- 172.21.15.38:0/213916005 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f929c10a540 msgr2=0x7f929c1a42b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:24.556 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:24.549+0000 7f929b7fe700 1 --2- 172.21.15.38:0/213916005 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f929c10a540 0x7f929c1a42b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:24.556 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:24.549+0000 7f929b7fe700 1 -- 172.21.15.38:0/213916005 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92900095f0 con 0x7f929c101200 2023-03-09T14:45:24.557 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:24.549+0000 7f929b7fe700 1 --2- 172.21.15.38:0/213916005 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f929c101200 0x7f929c0fee50 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f928c00c920 tx=0x7f928c00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:24.557 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:24.550+0000 7f9298ff9700 1 -- 172.21.15.38:0/213916005 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f928c007b00 con 0x7f929c101200 2023-03-09T14:45:24.558 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:24.550+0000 7f92a1ec6700 1 -- 172.21.15.38:0/213916005 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f929c10ffe0 con 0x7f929c101200 2023-03-09T14:45:24.558 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:24.550+0000 7f9298ff9700 1 -- 172.21.15.38:0/213916005 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f928c004cf0 con 0x7f929c101200 2023-03-09T14:45:24.558 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:24.550+0000 7f9298ff9700 1 -- 172.21.15.38:0/213916005 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f928c005520 con 0x7f929c101200 2023-03-09T14:45:24.558 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:24.550+0000 7f92a1ec6700 1 -- 172.21.15.38:0/213916005 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f929c10fe50 con 0x7f929c101200 2023-03-09T14:45:24.558 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:24.551+0000 7f92a1ec6700 1 -- 172.21.15.38:0/213916005 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f929c110170 con 0x7f929c101200 2023-03-09T14:45:24.558 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:24.552+0000 7f9298ff9700 1 -- 172.21.15.38:0/213916005 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 133238+0+0 (secure 0 0 0) 0x7f928c01f070 con 0x7f929c101200 2023-03-09T14:45:24.558 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:24.553+0000 7f9298ff9700 1 --2- 172.21.15.38:0/213916005 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f928409f400 0x7f92840a18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:24.558 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:24.553+0000 7f9298ff9700 1 -- 172.21.15.38:0/213916005 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f928c0c9cb0 con 0x7f929c101200 2023-03-09T14:45:24.559 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:24.553+0000 7f929affd700 1 --2- 172.21.15.38:0/213916005 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f928409f400 0x7f92840a18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:24.559 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:24.553+0000 7f929affd700 1 --2- 172.21.15.38:0/213916005 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f928409f400 0x7f92840a18b0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f9290006010 tx=0x7f929000b540 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:24.561 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:24.558+0000 7f9298ff9700 1 -- 172.21.15.38:0/213916005 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f928c093d20 con 0x7f929c101200 2023-03-09T14:45:24.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:24 smithi122 ceph-mon[93869]: pgmap v293: 129 pgs: 129 active+clean; 636 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s; 254 B/s, 1 objects/s recovering 2023-03-09T14:45:24.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:24 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:45:24.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:24 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:45:24.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:24 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:45:24.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:24 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:45:24.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:24 smithi181 ceph-mon[94189]: pgmap v293: 129 pgs: 129 active+clean; 636 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s; 254 B/s, 1 objects/s recovering 2023-03-09T14:45:24.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:24 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:45:24.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:24 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:45:24.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:24 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:45:24.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:24 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:45:24.755 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:24.752+0000 7f92a1ec6700 1 -- 172.21.15.38:0/213916005 --> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f929c0fc0b0 con 0x7f928409f400 2023-03-09T14:45:24.770 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:24.767+0000 7f9298ff9700 1 -- 172.21.15.38:0/213916005 <== mgr.14150 v2:172.21.15.38:6800/3328887328 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+424694 (secure 0 0 0) 0x7f929c0fc0b0 con 0x7f928409f400 2023-03-09T14:45:24.775 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:45:24.777 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:24.771+0000 7f92a1ec6700 1 -- 172.21.15.38:0/213916005 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f928409f400 msgr2=0x7f92840a18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:24.777 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:24.771+0000 7f92a1ec6700 1 --2- 172.21.15.38:0/213916005 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f928409f400 0x7f92840a18b0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f9290006010 tx=0x7f929000b540 comp rx=0 tx=0).stop 2023-03-09T14:45:24.777 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:24.771+0000 7f92a1ec6700 1 -- 172.21.15.38:0/213916005 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f929c101200 msgr2=0x7f929c0fee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:24.777 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:24.771+0000 7f92a1ec6700 1 --2- 172.21.15.38:0/213916005 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f929c101200 0x7f929c0fee50 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f928c00c920 tx=0x7f928c00cce0 comp rx=0 tx=0).stop 2023-03-09T14:45:24.777 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:24.771+0000 7f92a1ec6700 1 -- 172.21.15.38:0/213916005 shutdown_connections 2023-03-09T14:45:24.778 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:24.771+0000 7f92a1ec6700 1 --2- 172.21.15.38:0/213916005 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f929c101200 0x7f929c0fee50 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:24.778 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:24.771+0000 7f92a1ec6700 1 --2- 172.21.15.38:0/213916005 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f928409f400 0x7f92840a18b0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:24.778 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:24.771+0000 7f92a1ec6700 1 --2- 172.21.15.38:0/213916005 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f929c10a540 0x7f929c1a42b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:24.778 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:24.771+0000 7f92a1ec6700 1 --2- 172.21.15.38:0/213916005 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f929c101b10 0x7f929c0ff390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:24.778 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:24.771+0000 7f92a1ec6700 1 -- 172.21.15.38:0/213916005 >> 172.21.15.38:0/213916005 conn(0x7f929c076d10 msgr2=0x7f929c0fb9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:24.778 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:24.771+0000 7f92a1ec6700 1 -- 172.21.15.38:0/213916005 shutdown_connections 2023-03-09T14:45:24.778 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:24.771+0000 7f92a1ec6700 1 -- 172.21.15.38:0/213916005 wait complete. 2023-03-09T14:45:24.779 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2023-03-09T14:45:25.155 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":293,"stamp":"2023-03-09T14:45:22.762394+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":651232,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":408,"num_read_kb":317,"num_write":335,"num_write_kb":3044,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1182773,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":273,"ondisk_log_size":273,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":421,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":82988,"kb_used_data":13772,"kb_used_omap":0,"kb_used_meta":69120,"kb_avail":1124858836,"statfs":{"total":1151940427776,"available":1151855448064,"internally_reserved":0,"allocated":14102528,"data_stored":7818713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":70778880},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":53,"num_read_kb":23,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":2037,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002319"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.859159+0000","last_change":"2023-03-09T14:44:19.116371+0000","last_active":"2023-03-09T14:45:14.859159+0000","last_peered":"2023-03-09T14:45:14.859159+0000","last_clean":"2023-03-09T14:45:14.859159+0000","last_became_active":"2023-03-09T14:44:19.116024+0000","last_became_peered":"2023-03-09T14:44:19.116024+0000","last_unstale":"2023-03-09T14:45:14.859159+0000","last_undegraded":"2023-03-09T14:45:14.859159+0000","last_fullsized":"2023-03-09T14:45:14.859159+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T16:43:33.917176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.864887+0000","last_change":"2023-03-09T14:44:19.117312+0000","last_active":"2023-03-09T14:45:14.864887+0000","last_peered":"2023-03-09T14:45:14.864887+0000","last_clean":"2023-03-09T14:45:14.864887+0000","last_became_active":"2023-03-09T14:44:19.116920+0000","last_became_peered":"2023-03-09T14:44:19.116920+0000","last_unstale":"2023-03-09T14:45:14.864887+0000","last_undegraded":"2023-03-09T14:45:14.864887+0000","last_fullsized":"2023-03-09T14:45:14.864887+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:33:15.046431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.864174+0000","last_change":"2023-03-09T14:44:19.113482+0000","last_active":"2023-03-09T14:45:14.864174+0000","last_peered":"2023-03-09T14:45:14.864174+0000","last_clean":"2023-03-09T14:45:14.864174+0000","last_became_active":"2023-03-09T14:44:19.113183+0000","last_became_peered":"2023-03-09T14:44:19.113183+0000","last_unstale":"2023-03-09T14:45:14.864174+0000","last_undegraded":"2023-03-09T14:45:14.864174+0000","last_fullsized":"2023-03-09T14:45:14.864174+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:17:31.382255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.860049+0000","last_change":"2023-03-09T14:44:19.105799+0000","last_active":"2023-03-09T14:45:14.860049+0000","last_peered":"2023-03-09T14:45:14.860049+0000","last_clean":"2023-03-09T14:45:14.860049+0000","last_became_active":"2023-03-09T14:44:19.105405+0000","last_became_peered":"2023-03-09T14:44:19.105405+0000","last_unstale":"2023-03-09T14:45:14.860049+0000","last_undegraded":"2023-03-09T14:45:14.860049+0000","last_fullsized":"2023-03-09T14:45:14.860049+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:38:23.049984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.860357+0000","last_change":"2023-03-09T14:44:19.100656+0000","last_active":"2023-03-09T14:45:14.860357+0000","last_peered":"2023-03-09T14:45:14.860357+0000","last_clean":"2023-03-09T14:45:14.860357+0000","last_became_active":"2023-03-09T14:44:19.100278+0000","last_became_peered":"2023-03-09T14:44:19.100278+0000","last_unstale":"2023-03-09T14:45:14.860357+0000","last_undegraded":"2023-03-09T14:45:14.860357+0000","last_fullsized":"2023-03-09T14:45:14.860357+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:25:39.240251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.857293+0000","last_change":"2023-03-09T14:44:19.101065+0000","last_active":"2023-03-09T14:45:14.857293+0000","last_peered":"2023-03-09T14:45:14.857293+0000","last_clean":"2023-03-09T14:45:14.857293+0000","last_became_active":"2023-03-09T14:44:19.100451+0000","last_became_peered":"2023-03-09T14:44:19.100451+0000","last_unstale":"2023-03-09T14:45:14.857293+0000","last_undegraded":"2023-03-09T14:45:14.857293+0000","last_fullsized":"2023-03-09T14:45:14.857293+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T01:25:15.158254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.852722+0000","last_change":"2023-03-09T14:44:19.107996+0000","last_active":"2023-03-09T14:45:14.852722+0000","last_peered":"2023-03-09T14:45:14.852722+0000","last_clean":"2023-03-09T14:45:14.852722+0000","last_became_active":"2023-03-09T14:44:19.107683+0000","last_became_peered":"2023-03-09T14:44:19.107683+0000","last_unstale":"2023-03-09T14:45:14.852722+0000","last_undegraded":"2023-03-09T14:45:14.852722+0000","last_fullsized":"2023-03-09T14:45:14.852722+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:16:18.605176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.864215+0000","last_change":"2023-03-09T14:44:19.103940+0000","last_active":"2023-03-09T14:45:14.864215+0000","last_peered":"2023-03-09T14:45:14.864215+0000","last_clean":"2023-03-09T14:45:14.864215+0000","last_became_active":"2023-03-09T14:44:19.103620+0000","last_became_peered":"2023-03-09T14:44:19.103620+0000","last_unstale":"2023-03-09T14:45:14.864215+0000","last_undegraded":"2023-03-09T14:45:14.864215+0000","last_fullsized":"2023-03-09T14:45:14.864215+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:06:22.749636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.858269+0000","last_change":"2023-03-09T14:44:19.102695+0000","last_active":"2023-03-09T14:45:14.858269+0000","last_peered":"2023-03-09T14:45:14.858269+0000","last_clean":"2023-03-09T14:45:14.858269+0000","last_became_active":"2023-03-09T14:44:19.101486+0000","last_became_peered":"2023-03-09T14:44:19.101486+0000","last_unstale":"2023-03-09T14:45:14.858269+0000","last_undegraded":"2023-03-09T14:45:14.858269+0000","last_fullsized":"2023-03-09T14:45:14.858269+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T16:34:51.836443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.858962+0000","last_change":"2023-03-09T14:44:19.103522+0000","last_active":"2023-03-09T14:45:14.858962+0000","last_peered":"2023-03-09T14:45:14.858962+0000","last_clean":"2023-03-09T14:45:14.858962+0000","last_became_active":"2023-03-09T14:44:19.103242+0000","last_became_peered":"2023-03-09T14:44:19.103242+0000","last_unstale":"2023-03-09T14:45:14.858962+0000","last_undegraded":"2023-03-09T14:45:14.858962+0000","last_fullsized":"2023-03-09T14:45:14.858962+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:41:03.137554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.853693+0000","last_change":"2023-03-09T14:44:19.114934+0000","last_active":"2023-03-09T14:45:14.853693+0000","last_peered":"2023-03-09T14:45:14.853693+0000","last_clean":"2023-03-09T14:45:14.853693+0000","last_became_active":"2023-03-09T14:44:19.114620+0000","last_became_peered":"2023-03-09T14:44:19.114620+0000","last_unstale":"2023-03-09T14:45:14.853693+0000","last_undegraded":"2023-03-09T14:45:14.853693+0000","last_fullsized":"2023-03-09T14:45:14.853693+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T01:48:40.082076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.860296+0000","last_change":"2023-03-09T14:44:19.107677+0000","last_active":"2023-03-09T14:45:14.860296+0000","last_peered":"2023-03-09T14:45:14.860296+0000","last_clean":"2023-03-09T14:45:14.860296+0000","last_became_active":"2023-03-09T14:44:19.107369+0000","last_became_peered":"2023-03-09T14:44:19.107369+0000","last_unstale":"2023-03-09T14:45:14.860296+0000","last_undegraded":"2023-03-09T14:45:14.860296+0000","last_fullsized":"2023-03-09T14:45:14.860296+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:47:40.117205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.861064+0000","last_change":"2023-03-09T14:44:19.108852+0000","last_active":"2023-03-09T14:45:14.861064+0000","last_peered":"2023-03-09T14:45:14.861064+0000","last_clean":"2023-03-09T14:45:14.861064+0000","last_became_active":"2023-03-09T14:44:19.108515+0000","last_became_peered":"2023-03-09T14:44:19.108515+0000","last_unstale":"2023-03-09T14:45:14.861064+0000","last_undegraded":"2023-03-09T14:45:14.861064+0000","last_fullsized":"2023-03-09T14:45:14.861064+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:00:36.243952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.863730+0000","last_change":"2023-03-09T14:44:19.118673+0000","last_active":"2023-03-09T14:45:14.863730+0000","last_peered":"2023-03-09T14:45:14.863730+0000","last_clean":"2023-03-09T14:45:14.863730+0000","last_became_active":"2023-03-09T14:44:19.117656+0000","last_became_peered":"2023-03-09T14:44:19.117656+0000","last_unstale":"2023-03-09T14:45:14.863730+0000","last_undegraded":"2023-03-09T14:45:14.863730+0000","last_fullsized":"2023-03-09T14:45:14.863730+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:02:50.401680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.863498+0000","last_change":"2023-03-09T14:44:19.112979+0000","last_active":"2023-03-09T14:45:14.863498+0000","last_peered":"2023-03-09T14:45:14.863498+0000","last_clean":"2023-03-09T14:45:14.863498+0000","last_became_active":"2023-03-09T14:44:19.112671+0000","last_became_peered":"2023-03-09T14:44:19.112671+0000","last_unstale":"2023-03-09T14:45:14.863498+0000","last_undegraded":"2023-03-09T14:45:14.863498+0000","last_fullsized":"2023-03-09T14:45:14.863498+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:00:43.465327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.856048+0000","last_change":"2023-03-09T14:44:19.104881+0000","last_active":"2023-03-09T14:45:14.856048+0000","last_peered":"2023-03-09T14:45:14.856048+0000","last_clean":"2023-03-09T14:45:14.856048+0000","last_became_active":"2023-03-09T14:44:19.104647+0000","last_became_peered":"2023-03-09T14:44:19.104647+0000","last_unstale":"2023-03-09T14:45:14.856048+0000","last_undegraded":"2023-03-09T14:45:14.856048+0000","last_fullsized":"2023-03-09T14:45:14.856048+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T20:16:24.625362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.859353+0000","last_change":"2023-03-09T14:44:19.117937+0000","last_active":"2023-03-09T14:45:14.859353+0000","last_peered":"2023-03-09T14:45:14.859353+0000","last_clean":"2023-03-09T14:45:14.859353+0000","last_became_active":"2023-03-09T14:44:19.117628+0000","last_became_peered":"2023-03-09T14:44:19.117628+0000","last_unstale":"2023-03-09T14:45:14.859353+0000","last_undegraded":"2023-03-09T14:45:14.859353+0000","last_fullsized":"2023-03-09T14:45:14.859353+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:22:18.619171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.860217+0000","last_change":"2023-03-09T14:44:19.105071+0000","last_active":"2023-03-09T14:45:14.860217+0000","last_peered":"2023-03-09T14:45:14.860217+0000","last_clean":"2023-03-09T14:45:14.860217+0000","last_became_active":"2023-03-09T14:44:19.103396+0000","last_became_peered":"2023-03-09T14:44:19.103396+0000","last_unstale":"2023-03-09T14:45:14.860217+0000","last_undegraded":"2023-03-09T14:45:14.860217+0000","last_fullsized":"2023-03-09T14:45:14.860217+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:14:51.924636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.857765+0000","last_change":"2023-03-09T14:44:19.104107+0000","last_active":"2023-03-09T14:45:14.857765+0000","last_peered":"2023-03-09T14:45:14.857765+0000","last_clean":"2023-03-09T14:45:14.857765+0000","last_became_active":"2023-03-09T14:44:19.103793+0000","last_became_peered":"2023-03-09T14:44:19.103793+0000","last_unstale":"2023-03-09T14:45:14.857765+0000","last_undegraded":"2023-03-09T14:45:14.857765+0000","last_fullsized":"2023-03-09T14:45:14.857765+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:57:29.169744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.856426+0000","last_change":"2023-03-09T14:44:23.489947+0000","last_active":"2023-03-09T14:45:14.856426+0000","last_peered":"2023-03-09T14:45:14.856426+0000","last_clean":"2023-03-09T14:45:14.856426+0000","last_became_active":"2023-03-09T14:44:23.489560+0000","last_became_peered":"2023-03-09T14:44:23.489560+0000","last_unstale":"2023-03-09T14:45:14.856426+0000","last_undegraded":"2023-03-09T14:45:14.856426+0000","last_fullsized":"2023-03-09T14:45:14.856426+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:06:10.202176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"87'9","reported_seq":49,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:15.961952+0000","last_change":"2023-03-09T14:45:15.961952+0000","last_active":"2023-03-09T14:45:15.961952+0000","last_peered":"2023-03-09T14:45:15.961952+0000","last_clean":"2023-03-09T14:45:15.961952+0000","last_became_active":"2023-03-09T14:45:15.865191+0000","last_became_peered":"2023-03-09T14:45:15.865191+0000","last_unstale":"2023-03-09T14:45:15.961952+0000","last_undegraded":"2023-03-09T14:45:15.961952+0000","last_fullsized":"2023-03-09T14:45:15.961952+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T01:21:05.925138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":566,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.863851+0000","last_change":"2023-03-09T14:44:19.105390+0000","last_active":"2023-03-09T14:45:14.863851+0000","last_peered":"2023-03-09T14:45:14.863851+0000","last_clean":"2023-03-09T14:45:14.863851+0000","last_became_active":"2023-03-09T14:44:19.104694+0000","last_became_peered":"2023-03-09T14:44:19.104694+0000","last_unstale":"2023-03-09T14:45:14.863851+0000","last_undegraded":"2023-03-09T14:45:14.863851+0000","last_fullsized":"2023-03-09T14:45:14.863851+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:19:08.101847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.856827+0000","last_change":"2023-03-09T14:44:23.484103+0000","last_active":"2023-03-09T14:45:14.856827+0000","last_peered":"2023-03-09T14:45:14.856827+0000","last_clean":"2023-03-09T14:45:14.856827+0000","last_became_active":"2023-03-09T14:44:23.483271+0000","last_became_peered":"2023-03-09T14:44:23.483271+0000","last_unstale":"2023-03-09T14:45:14.856827+0000","last_undegraded":"2023-03-09T14:45:14.856827+0000","last_fullsized":"2023-03-09T14:45:14.856827+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:24:06.212623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"87'2","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.864060+0000","last_change":"2023-03-09T14:44:18.667321+0000","last_active":"2023-03-09T14:45:14.864060+0000","last_peered":"2023-03-09T14:45:14.864060+0000","last_clean":"2023-03-09T14:45:14.864060+0000","last_became_active":"2023-03-09T14:44:18.666974+0000","last_became_peered":"2023-03-09T14:44:18.666974+0000","last_unstale":"2023-03-09T14:45:14.864060+0000","last_undegraded":"2023-03-09T14:45:14.864060+0000","last_fullsized":"2023-03-09T14:45:14.864060+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:31:07.327792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.864530+0000","last_change":"2023-03-09T14:44:19.116866+0000","last_active":"2023-03-09T14:45:14.864530+0000","last_peered":"2023-03-09T14:45:14.864530+0000","last_clean":"2023-03-09T14:45:14.864530+0000","last_became_active":"2023-03-09T14:44:19.116571+0000","last_became_peered":"2023-03-09T14:44:19.116571+0000","last_unstale":"2023-03-09T14:45:14.864530+0000","last_undegraded":"2023-03-09T14:45:14.864530+0000","last_fullsized":"2023-03-09T14:45:14.864530+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T16:22:12.723903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.859406+0000","last_change":"2023-03-09T14:44:23.488192+0000","last_active":"2023-03-09T14:45:14.859406+0000","last_peered":"2023-03-09T14:45:14.859406+0000","last_clean":"2023-03-09T14:45:14.859406+0000","last_became_active":"2023-03-09T14:44:23.487734+0000","last_became_peered":"2023-03-09T14:44:23.487734+0000","last_unstale":"2023-03-09T14:45:14.859406+0000","last_undegraded":"2023-03-09T14:45:14.859406+0000","last_fullsized":"2023-03-09T14:45:14.859406+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:35:50.319216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"87'4","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.861901+0000","last_change":"2023-03-09T14:44:18.663577+0000","last_active":"2023-03-09T14:45:14.861901+0000","last_peered":"2023-03-09T14:45:14.861901+0000","last_clean":"2023-03-09T14:45:14.861901+0000","last_became_active":"2023-03-09T14:44:18.663272+0000","last_became_peered":"2023-03-09T14:44:18.663272+0000","last_unstale":"2023-03-09T14:45:14.861901+0000","last_undegraded":"2023-03-09T14:45:14.861901+0000","last_fullsized":"2023-03-09T14:45:14.861901+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T20:55:19.661875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.857464+0000","last_change":"2023-03-09T14:44:19.106986+0000","last_active":"2023-03-09T14:45:14.857464+0000","last_peered":"2023-03-09T14:45:14.857464+0000","last_clean":"2023-03-09T14:45:14.857464+0000","last_became_active":"2023-03-09T14:44:19.106625+0000","last_became_peered":"2023-03-09T14:44:19.106625+0000","last_unstale":"2023-03-09T14:45:14.857464+0000","last_undegraded":"2023-03-09T14:45:14.857464+0000","last_fullsized":"2023-03-09T14:45:14.857464+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:03:54.521603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.858888+0000","last_change":"2023-03-09T14:44:23.483803+0000","last_active":"2023-03-09T14:45:14.858888+0000","last_peered":"2023-03-09T14:45:14.858888+0000","last_clean":"2023-03-09T14:45:14.858888+0000","last_became_active":"2023-03-09T14:44:23.483286+0000","last_became_peered":"2023-03-09T14:44:23.483286+0000","last_unstale":"2023-03-09T14:45:14.858888+0000","last_undegraded":"2023-03-09T14:45:14.858888+0000","last_fullsized":"2023-03-09T14:45:14.858888+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:32:35.975670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"82'2","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.861301+0000","last_change":"2023-03-09T14:44:18.657685+0000","last_active":"2023-03-09T14:45:14.861301+0000","last_peered":"2023-03-09T14:45:14.861301+0000","last_clean":"2023-03-09T14:45:14.861301+0000","last_became_active":"2023-03-09T14:44:18.657355+0000","last_became_peered":"2023-03-09T14:44:18.657355+0000","last_unstale":"2023-03-09T14:45:14.861301+0000","last_undegraded":"2023-03-09T14:45:14.861301+0000","last_fullsized":"2023-03-09T14:45:14.861301+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:33:59.259690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.853085+0000","last_change":"2023-03-09T14:44:19.111687+0000","last_active":"2023-03-09T14:45:14.853085+0000","last_peered":"2023-03-09T14:45:14.853085+0000","last_clean":"2023-03-09T14:45:14.853085+0000","last_became_active":"2023-03-09T14:44:19.111481+0000","last_became_peered":"2023-03-09T14:44:19.111481+0000","last_unstale":"2023-03-09T14:45:14.853085+0000","last_undegraded":"2023-03-09T14:45:14.853085+0000","last_fullsized":"2023-03-09T14:45:14.853085+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:06:14.452967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.857536+0000","last_change":"2023-03-09T14:44:23.485075+0000","last_active":"2023-03-09T14:45:14.857536+0000","last_peered":"2023-03-09T14:45:14.857536+0000","last_clean":"2023-03-09T14:45:14.857536+0000","last_became_active":"2023-03-09T14:44:23.484648+0000","last_became_peered":"2023-03-09T14:44:23.484648+0000","last_unstale":"2023-03-09T14:45:14.857536+0000","last_undegraded":"2023-03-09T14:45:14.857536+0000","last_fullsized":"2023-03-09T14:45:14.857536+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:49:40.332276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"87'5","reported_seq":85,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:20.875689+0000","last_change":"2023-03-09T14:45:15.868911+0000","last_active":"2023-03-09T14:45:20.875689+0000","last_peered":"2023-03-09T14:45:20.875689+0000","last_clean":"2023-03-09T14:45:20.875689+0000","last_became_active":"2023-03-09T14:45:15.861592+0000","last_became_peered":"2023-03-09T14:45:15.861592+0000","last_unstale":"2023-03-09T14:45:20.875689+0000","last_undegraded":"2023-03-09T14:45:20.875689+0000","last_fullsized":"2023-03-09T14:45:20.875689+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T20:17:55.938860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":45,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":622,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.861663+0000","last_change":"2023-03-09T14:44:19.107572+0000","last_active":"2023-03-09T14:45:14.861663+0000","last_peered":"2023-03-09T14:45:14.861663+0000","last_clean":"2023-03-09T14:45:14.861663+0000","last_became_active":"2023-03-09T14:44:19.106632+0000","last_became_peered":"2023-03-09T14:44:19.106632+0000","last_unstale":"2023-03-09T14:45:14.861663+0000","last_undegraded":"2023-03-09T14:45:14.861663+0000","last_fullsized":"2023-03-09T14:45:14.861663+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T19:04:01.379375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.857041+0000","last_change":"2023-03-09T14:44:23.494002+0000","last_active":"2023-03-09T14:45:14.857041+0000","last_peered":"2023-03-09T14:45:14.857041+0000","last_clean":"2023-03-09T14:45:14.857041+0000","last_became_active":"2023-03-09T14:44:23.493628+0000","last_became_peered":"2023-03-09T14:44:23.493628+0000","last_unstale":"2023-03-09T14:45:14.857041+0000","last_undegraded":"2023-03-09T14:45:14.857041+0000","last_fullsized":"2023-03-09T14:45:14.857041+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:30:47.970556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"87'3","reported_seq":36,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:15.964036+0000","last_change":"2023-03-09T14:45:15.964036+0000","last_active":"2023-03-09T14:45:15.964036+0000","last_peered":"2023-03-09T14:45:15.964036+0000","last_clean":"2023-03-09T14:45:15.964036+0000","last_became_active":"2023-03-09T14:45:15.865998+0000","last_became_peered":"2023-03-09T14:45:15.865998+0000","last_unstale":"2023-03-09T14:45:15.964036+0000","last_undegraded":"2023-03-09T14:45:15.964036+0000","last_fullsized":"2023-03-09T14:45:15.964036+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:16:58.213684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.852992+0000","last_change":"2023-03-09T14:44:19.111303+0000","last_active":"2023-03-09T14:45:14.852992+0000","last_peered":"2023-03-09T14:45:14.852992+0000","last_clean":"2023-03-09T14:45:14.852992+0000","last_became_active":"2023-03-09T14:44:19.111026+0000","last_became_peered":"2023-03-09T14:44:19.111026+0000","last_unstale":"2023-03-09T14:45:14.852992+0000","last_undegraded":"2023-03-09T14:45:14.852992+0000","last_fullsized":"2023-03-09T14:45:14.852992+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T02:26:32.496084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.859059+0000","last_change":"2023-03-09T14:44:23.493272+0000","last_active":"2023-03-09T14:45:14.859059+0000","last_peered":"2023-03-09T14:45:14.859059+0000","last_clean":"2023-03-09T14:45:14.859059+0000","last_became_active":"2023-03-09T14:44:23.492785+0000","last_became_peered":"2023-03-09T14:44:23.492785+0000","last_unstale":"2023-03-09T14:45:14.859059+0000","last_undegraded":"2023-03-09T14:45:14.859059+0000","last_fullsized":"2023-03-09T14:45:14.859059+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T20:18:39.758938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"87'2","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.861264+0000","last_change":"2023-03-09T14:44:18.658179+0000","last_active":"2023-03-09T14:45:14.861264+0000","last_peered":"2023-03-09T14:45:14.861264+0000","last_clean":"2023-03-09T14:45:14.861264+0000","last_became_active":"2023-03-09T14:44:18.657851+0000","last_became_peered":"2023-03-09T14:44:18.657851+0000","last_unstale":"2023-03-09T14:45:14.861264+0000","last_undegraded":"2023-03-09T14:45:14.861264+0000","last_fullsized":"2023-03-09T14:45:14.861264+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:52:49.291615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.864131+0000","last_change":"2023-03-09T14:44:19.104176+0000","last_active":"2023-03-09T14:45:14.864131+0000","last_peered":"2023-03-09T14:45:14.864131+0000","last_clean":"2023-03-09T14:45:14.864131+0000","last_became_active":"2023-03-09T14:44:19.103816+0000","last_became_peered":"2023-03-09T14:44:19.103816+0000","last_unstale":"2023-03-09T14:45:14.864131+0000","last_undegraded":"2023-03-09T14:45:14.864131+0000","last_fullsized":"2023-03-09T14:45:14.864131+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T02:06:41.876169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.864893+0000","last_change":"2023-03-09T14:44:23.490876+0000","last_active":"2023-03-09T14:45:14.864893+0000","last_peered":"2023-03-09T14:45:14.864893+0000","last_clean":"2023-03-09T14:45:14.864893+0000","last_became_active":"2023-03-09T14:44:23.490461+0000","last_became_peered":"2023-03-09T14:44:23.490461+0000","last_unstale":"2023-03-09T14:45:14.864893+0000","last_undegraded":"2023-03-09T14:45:14.864893+0000","last_fullsized":"2023-03-09T14:45:14.864893+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T16:28:40.456613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"87'1","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.852373+0000","last_change":"2023-03-09T14:44:18.662513+0000","last_active":"2023-03-09T14:45:14.852373+0000","last_peered":"2023-03-09T14:45:14.852373+0000","last_clean":"2023-03-09T14:45:14.852373+0000","last_became_active":"2023-03-09T14:44:18.661907+0000","last_became_peered":"2023-03-09T14:44:18.661907+0000","last_unstale":"2023-03-09T14:45:14.852373+0000","last_undegraded":"2023-03-09T14:45:14.852373+0000","last_fullsized":"2023-03-09T14:45:14.852373+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:45:17.373213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.860438+0000","last_change":"2023-03-09T14:44:19.115297+0000","last_active":"2023-03-09T14:45:14.860438+0000","last_peered":"2023-03-09T14:45:14.860438+0000","last_clean":"2023-03-09T14:45:14.860438+0000","last_became_active":"2023-03-09T14:44:19.114973+0000","last_became_peered":"2023-03-09T14:44:19.114973+0000","last_unstale":"2023-03-09T14:45:14.860438+0000","last_undegraded":"2023-03-09T14:45:14.860438+0000","last_fullsized":"2023-03-09T14:45:14.860438+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:11:22.226608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.864500+0000","last_change":"2023-03-09T14:44:23.482772+0000","last_active":"2023-03-09T14:45:14.864500+0000","last_peered":"2023-03-09T14:45:14.864500+0000","last_clean":"2023-03-09T14:45:14.864500+0000","last_became_active":"2023-03-09T14:44:23.481750+0000","last_became_peered":"2023-03-09T14:44:23.481750+0000","last_unstale":"2023-03-09T14:45:14.864500+0000","last_undegraded":"2023-03-09T14:45:14.864500+0000","last_fullsized":"2023-03-09T14:45:14.864500+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T19:26:03.737713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"87'3","reported_seq":79,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:18.872287+0000","last_change":"2023-03-09T14:45:15.870269+0000","last_active":"2023-03-09T14:45:18.872287+0000","last_peered":"2023-03-09T14:45:18.872287+0000","last_clean":"2023-03-09T14:45:18.872287+0000","last_became_active":"2023-03-09T14:45:15.862610+0000","last_became_peered":"2023-03-09T14:45:15.862610+0000","last_unstale":"2023-03-09T14:45:18.872287+0000","last_undegraded":"2023-03-09T14:45:18.872287+0000","last_fullsized":"2023-03-09T14:45:18.872287+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T02:37:23.157022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":41,"num_read_kb":41,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,9],"acting":[10,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.858136+0000","last_change":"2023-03-09T14:44:19.100914+0000","last_active":"2023-03-09T14:45:14.858136+0000","last_peered":"2023-03-09T14:45:14.858136+0000","last_clean":"2023-03-09T14:45:14.858136+0000","last_became_active":"2023-03-09T14:44:19.100451+0000","last_became_peered":"2023-03-09T14:44:19.100451+0000","last_unstale":"2023-03-09T14:45:14.858136+0000","last_undegraded":"2023-03-09T14:45:14.858136+0000","last_fullsized":"2023-03-09T14:45:14.858136+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:18:41.852697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.860006+0000","last_change":"2023-03-09T14:44:23.490129+0000","last_active":"2023-03-09T14:45:14.860006+0000","last_peered":"2023-03-09T14:45:14.860006+0000","last_clean":"2023-03-09T14:45:14.860006+0000","last_became_active":"2023-03-09T14:44:23.489696+0000","last_became_peered":"2023-03-09T14:44:23.489696+0000","last_unstale":"2023-03-09T14:45:14.860006+0000","last_undegraded":"2023-03-09T14:45:14.860006+0000","last_fullsized":"2023-03-09T14:45:14.860006+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T16:47:41.943904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"87'2","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:15.955780+0000","last_change":"2023-03-09T14:45:15.955780+0000","last_active":"2023-03-09T14:45:15.955780+0000","last_peered":"2023-03-09T14:45:15.955780+0000","last_clean":"2023-03-09T14:45:15.955780+0000","last_became_active":"2023-03-09T14:45:15.862567+0000","last_became_peered":"2023-03-09T14:45:15.862567+0000","last_unstale":"2023-03-09T14:45:15.955780+0000","last_undegraded":"2023-03-09T14:45:15.955780+0000","last_fullsized":"2023-03-09T14:45:15.955780+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:08:41.748638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.864609+0000","last_change":"2023-03-09T14:44:19.119125+0000","last_active":"2023-03-09T14:45:14.864609+0000","last_peered":"2023-03-09T14:45:14.864609+0000","last_clean":"2023-03-09T14:45:14.864609+0000","last_became_active":"2023-03-09T14:44:19.118677+0000","last_became_peered":"2023-03-09T14:44:19.118677+0000","last_unstale":"2023-03-09T14:45:14.864609+0000","last_undegraded":"2023-03-09T14:45:14.864609+0000","last_fullsized":"2023-03-09T14:45:14.864609+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:06:28.740355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.859670+0000","last_change":"2023-03-09T14:44:23.484752+0000","last_active":"2023-03-09T14:45:14.859670+0000","last_peered":"2023-03-09T14:45:14.859670+0000","last_clean":"2023-03-09T14:45:14.859670+0000","last_became_active":"2023-03-09T14:44:23.484045+0000","last_became_peered":"2023-03-09T14:44:23.484045+0000","last_unstale":"2023-03-09T14:45:14.859670+0000","last_undegraded":"2023-03-09T14:45:14.859670+0000","last_fullsized":"2023-03-09T14:45:14.859670+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:58:39.704017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"87'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.853129+0000","last_change":"2023-03-09T14:44:18.665122+0000","last_active":"2023-03-09T14:45:14.853129+0000","last_peered":"2023-03-09T14:45:14.853129+0000","last_clean":"2023-03-09T14:45:14.853129+0000","last_became_active":"2023-03-09T14:44:18.664786+0000","last_became_peered":"2023-03-09T14:44:18.664786+0000","last_unstale":"2023-03-09T14:45:14.853129+0000","last_undegraded":"2023-03-09T14:45:14.853129+0000","last_fullsized":"2023-03-09T14:45:14.853129+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T20:29:20.919293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":656,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.857673+0000","last_change":"2023-03-09T14:44:19.108693+0000","last_active":"2023-03-09T14:45:14.857673+0000","last_peered":"2023-03-09T14:45:14.857673+0000","last_clean":"2023-03-09T14:45:14.857673+0000","last_became_active":"2023-03-09T14:44:19.108370+0000","last_became_peered":"2023-03-09T14:44:19.108370+0000","last_unstale":"2023-03-09T14:45:14.857673+0000","last_undegraded":"2023-03-09T14:45:14.857673+0000","last_fullsized":"2023-03-09T14:45:14.857673+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T16:30:58.726080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:15.862206+0000","last_change":"2023-03-09T14:45:15.862206+0000","last_active":"2023-03-09T14:45:15.862206+0000","last_peered":"2023-03-09T14:45:15.862206+0000","last_clean":"2023-03-09T14:45:15.862206+0000","last_became_active":"2023-03-09T14:45:15.861775+0000","last_became_peered":"2023-03-09T14:45:15.861775+0000","last_unstale":"2023-03-09T14:45:15.862206+0000","last_undegraded":"2023-03-09T14:45:15.862206+0000","last_fullsized":"2023-03-09T14:45:15.862206+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:31:21.810745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.14","version":"87'30","reported_seq":137,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:19.867290+0000","last_change":"2023-03-09T14:44:18.661904+0000","last_active":"2023-03-09T14:45:19.867290+0000","last_peered":"2023-03-09T14:45:19.867290+0000","last_clean":"2023-03-09T14:45:19.867290+0000","last_became_active":"2023-03-09T14:44:18.661597+0000","last_became_peered":"2023-03-09T14:44:18.661597+0000","last_unstale":"2023-03-09T14:45:19.867290+0000","last_undegraded":"2023-03-09T14:45:19.867290+0000","last_fullsized":"2023-03-09T14:45:19.867290+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T14:57:04.978132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56666,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":89,"num_read_kb":100,"num_write":31,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.861600+0000","last_change":"2023-03-09T14:44:19.111671+0000","last_active":"2023-03-09T14:45:14.861600+0000","last_peered":"2023-03-09T14:45:14.861600+0000","last_clean":"2023-03-09T14:45:14.861600+0000","last_became_active":"2023-03-09T14:44:19.111172+0000","last_became_peered":"2023-03-09T14:44:19.111172+0000","last_unstale":"2023-03-09T14:45:14.861600+0000","last_undegraded":"2023-03-09T14:45:14.861600+0000","last_fullsized":"2023-03-09T14:45:14.861600+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T16:27:23.921853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.853428+0000","last_change":"2023-03-09T14:44:23.486338+0000","last_active":"2023-03-09T14:45:14.853428+0000","last_peered":"2023-03-09T14:45:14.853428+0000","last_clean":"2023-03-09T14:45:14.853428+0000","last_became_active":"2023-03-09T14:44:23.485729+0000","last_became_peered":"2023-03-09T14:44:23.485729+0000","last_unstale":"2023-03-09T14:45:14.853428+0000","last_undegraded":"2023-03-09T14:45:14.853428+0000","last_fullsized":"2023-03-09T14:45:14.853428+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:28:23.872633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.857067+0000","last_change":"2023-03-09T14:44:18.664942+0000","last_active":"2023-03-09T14:45:14.857067+0000","last_peered":"2023-03-09T14:45:14.857067+0000","last_clean":"2023-03-09T14:45:14.857067+0000","last_became_active":"2023-03-09T14:44:18.664007+0000","last_became_peered":"2023-03-09T14:44:18.664007+0000","last_unstale":"2023-03-09T14:45:14.857067+0000","last_undegraded":"2023-03-09T14:45:14.857067+0000","last_fullsized":"2023-03-09T14:45:14.857067+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:36:21.329388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.861918+0000","last_change":"2023-03-09T14:44:19.110140+0000","last_active":"2023-03-09T14:45:14.861918+0000","last_peered":"2023-03-09T14:45:14.861918+0000","last_clean":"2023-03-09T14:45:14.861918+0000","last_became_active":"2023-03-09T14:44:19.109811+0000","last_became_peered":"2023-03-09T14:44:19.109811+0000","last_unstale":"2023-03-09T14:45:14.861918+0000","last_undegraded":"2023-03-09T14:45:14.861918+0000","last_fullsized":"2023-03-09T14:45:14.861918+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T14:56:14.638662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.858304+0000","last_change":"2023-03-09T14:44:23.491318+0000","last_active":"2023-03-09T14:45:14.858304+0000","last_peered":"2023-03-09T14:45:14.858304+0000","last_clean":"2023-03-09T14:45:14.858304+0000","last_became_active":"2023-03-09T14:44:23.490915+0000","last_became_peered":"2023-03-09T14:44:23.490915+0000","last_unstale":"2023-03-09T14:45:14.858304+0000","last_undegraded":"2023-03-09T14:45:14.858304+0000","last_fullsized":"2023-03-09T14:45:14.858304+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:32:06.877573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"87'3","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.852806+0000","last_change":"2023-03-09T14:44:18.657323+0000","last_active":"2023-03-09T14:45:14.852806+0000","last_peered":"2023-03-09T14:45:14.852806+0000","last_clean":"2023-03-09T14:45:14.852806+0000","last_became_active":"2023-03-09T14:44:18.657017+0000","last_became_peered":"2023-03-09T14:44:18.657017+0000","last_unstale":"2023-03-09T14:45:14.852806+0000","last_undegraded":"2023-03-09T14:45:14.852806+0000","last_fullsized":"2023-03-09T14:45:14.852806+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:43:11.562723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.853198+0000","last_change":"2023-03-09T14:44:19.112194+0000","last_active":"2023-03-09T14:45:14.853198+0000","last_peered":"2023-03-09T14:45:14.853198+0000","last_clean":"2023-03-09T14:45:14.853198+0000","last_became_active":"2023-03-09T14:44:19.111924+0000","last_became_peered":"2023-03-09T14:44:19.111924+0000","last_unstale":"2023-03-09T14:45:14.853198+0000","last_undegraded":"2023-03-09T14:45:14.853198+0000","last_fullsized":"2023-03-09T14:45:14.853198+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T19:54:22.669848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.860733+0000","last_change":"2023-03-09T14:44:23.493993+0000","last_active":"2023-03-09T14:45:14.860733+0000","last_peered":"2023-03-09T14:45:14.860733+0000","last_clean":"2023-03-09T14:45:14.860733+0000","last_became_active":"2023-03-09T14:44:23.493535+0000","last_became_peered":"2023-03-09T14:44:23.493535+0000","last_unstale":"2023-03-09T14:45:14.860733+0000","last_undegraded":"2023-03-09T14:45:14.860733+0000","last_fullsized":"2023-03-09T14:45:14.860733+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T01:18:23.669790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"87'4","reported_seq":67,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:19.599067+0000","last_change":"2023-03-09T14:44:18.659914+0000","last_active":"2023-03-09T14:45:19.599067+0000","last_peered":"2023-03-09T14:45:19.599067+0000","last_clean":"2023-03-09T14:45:19.599067+0000","last_became_active":"2023-03-09T14:44:18.658835+0000","last_became_peered":"2023-03-09T14:44:18.658835+0000","last_unstale":"2023-03-09T14:45:19.599067+0000","last_undegraded":"2023-03-09T14:45:19.599067+0000","last_fullsized":"2023-03-09T14:45:19.599067+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T14:55:05.413260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":978,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":46,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.852607+0000","last_change":"2023-03-09T14:44:19.111058+0000","last_active":"2023-03-09T14:45:14.852607+0000","last_peered":"2023-03-09T14:45:14.852607+0000","last_clean":"2023-03-09T14:45:14.852607+0000","last_became_active":"2023-03-09T14:44:19.110760+0000","last_became_peered":"2023-03-09T14:44:19.110760+0000","last_unstale":"2023-03-09T14:45:14.852607+0000","last_undegraded":"2023-03-09T14:45:14.852607+0000","last_fullsized":"2023-03-09T14:45:14.852607+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:25:13.199416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.861493+0000","last_change":"2023-03-09T14:44:23.484710+0000","last_active":"2023-03-09T14:45:14.861493+0000","last_peered":"2023-03-09T14:45:14.861493+0000","last_clean":"2023-03-09T14:45:14.861493+0000","last_became_active":"2023-03-09T14:44:23.483928+0000","last_became_peered":"2023-03-09T14:44:23.483928+0000","last_unstale":"2023-03-09T14:45:14.861493+0000","last_undegraded":"2023-03-09T14:45:14.861493+0000","last_fullsized":"2023-03-09T14:45:14.861493+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T14:51:32.124111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"87'15","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.860284+0000","last_change":"2023-03-09T14:44:18.663692+0000","last_active":"2023-03-09T14:45:14.860284+0000","last_peered":"2023-03-09T14:45:14.860284+0000","last_clean":"2023-03-09T14:45:14.860284+0000","last_became_active":"2023-03-09T14:44:18.663400+0000","last_became_peered":"2023-03-09T14:44:18.663400+0000","last_unstale":"2023-03-09T14:45:14.860284+0000","last_undegraded":"2023-03-09T14:45:14.860284+0000","last_fullsized":"2023-03-09T14:45:14.860284+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:32:31.634315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.852446+0000","last_change":"2023-03-09T14:44:19.110819+0000","last_active":"2023-03-09T14:45:14.852446+0000","last_peered":"2023-03-09T14:45:14.852446+0000","last_clean":"2023-03-09T14:45:14.852446+0000","last_became_active":"2023-03-09T14:44:19.110542+0000","last_became_peered":"2023-03-09T14:44:19.110542+0000","last_unstale":"2023-03-09T14:45:14.852446+0000","last_undegraded":"2023-03-09T14:45:14.852446+0000","last_fullsized":"2023-03-09T14:45:14.852446+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:25:38.382685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.852360+0000","last_change":"2023-03-09T14:44:23.484614+0000","last_active":"2023-03-09T14:45:14.852360+0000","last_peered":"2023-03-09T14:45:14.852360+0000","last_clean":"2023-03-09T14:45:14.852360+0000","last_became_active":"2023-03-09T14:44:23.484061+0000","last_became_peered":"2023-03-09T14:44:23.484061+0000","last_unstale":"2023-03-09T14:45:14.852360+0000","last_undegraded":"2023-03-09T14:45:14.852360+0000","last_fullsized":"2023-03-09T14:45:14.852360+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T19:31:42.889122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"87'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.861838+0000","last_change":"2023-03-09T14:44:18.656056+0000","last_active":"2023-03-09T14:45:14.861838+0000","last_peered":"2023-03-09T14:45:14.861838+0000","last_clean":"2023-03-09T14:45:14.861838+0000","last_became_active":"2023-03-09T14:44:18.655709+0000","last_became_peered":"2023-03-09T14:44:18.655709+0000","last_unstale":"2023-03-09T14:45:14.861838+0000","last_undegraded":"2023-03-09T14:45:14.861838+0000","last_fullsized":"2023-03-09T14:45:14.861838+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T19:09:11.447249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.858951+0000","last_change":"2023-03-09T14:44:19.110292+0000","last_active":"2023-03-09T14:45:14.858951+0000","last_peered":"2023-03-09T14:45:14.858951+0000","last_clean":"2023-03-09T14:45:14.858951+0000","last_became_active":"2023-03-09T14:44:19.110006+0000","last_became_peered":"2023-03-09T14:44:19.110006+0000","last_unstale":"2023-03-09T14:45:14.858951+0000","last_undegraded":"2023-03-09T14:45:14.858951+0000","last_fullsized":"2023-03-09T14:45:14.858951+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:36:40.401486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.861551+0000","last_change":"2023-03-09T14:44:23.484004+0000","last_active":"2023-03-09T14:45:14.861551+0000","last_peered":"2023-03-09T14:45:14.861551+0000","last_clean":"2023-03-09T14:45:14.861551+0000","last_became_active":"2023-03-09T14:44:23.483581+0000","last_became_peered":"2023-03-09T14:44:23.483581+0000","last_unstale":"2023-03-09T14:45:14.861551+0000","last_undegraded":"2023-03-09T14:45:14.861551+0000","last_fullsized":"2023-03-09T14:45:14.861551+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:24:25.939838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.860756+0000","last_change":"2023-03-09T14:44:18.661856+0000","last_active":"2023-03-09T14:45:14.860756+0000","last_peered":"2023-03-09T14:45:14.860756+0000","last_clean":"2023-03-09T14:45:14.860756+0000","last_became_active":"2023-03-09T14:44:18.661500+0000","last_became_peered":"2023-03-09T14:44:18.661500+0000","last_unstale":"2023-03-09T14:45:14.860756+0000","last_undegraded":"2023-03-09T14:45:14.860756+0000","last_fullsized":"2023-03-09T14:45:14.860756+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:27:12.181159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.858358+0000","last_change":"2023-03-09T14:44:19.102058+0000","last_active":"2023-03-09T14:45:14.858358+0000","last_peered":"2023-03-09T14:45:14.858358+0000","last_clean":"2023-03-09T14:45:14.858358+0000","last_became_active":"2023-03-09T14:44:19.101762+0000","last_became_peered":"2023-03-09T14:44:19.101762+0000","last_unstale":"2023-03-09T14:45:14.858358+0000","last_undegraded":"2023-03-09T14:45:14.858358+0000","last_fullsized":"2023-03-09T14:45:14.858358+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:39:42.544086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.862303+0000","last_change":"2023-03-09T14:44:23.490237+0000","last_active":"2023-03-09T14:45:14.862303+0000","last_peered":"2023-03-09T14:45:14.862303+0000","last_clean":"2023-03-09T14:45:14.862303+0000","last_became_active":"2023-03-09T14:44:23.489844+0000","last_became_peered":"2023-03-09T14:44:23.489844+0000","last_unstale":"2023-03-09T14:45:14.862303+0000","last_undegraded":"2023-03-09T14:45:14.862303+0000","last_fullsized":"2023-03-09T14:45:14.862303+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T14:50:09.358698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"87'1","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:15.865433+0000","last_change":"2023-03-09T14:45:15.864913+0000","last_active":"2023-03-09T14:45:15.865433+0000","last_peered":"2023-03-09T14:45:15.865433+0000","last_clean":"2023-03-09T14:45:15.865433+0000","last_became_active":"2023-03-09T14:45:15.864527+0000","last_became_peered":"2023-03-09T14:45:15.864527+0000","last_unstale":"2023-03-09T14:45:15.865433+0000","last_undegraded":"2023-03-09T14:45:15.865433+0000","last_fullsized":"2023-03-09T14:45:15.865433+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:02:19.489659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6,1],"acting":[11,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.857166+0000","last_change":"2023-03-09T14:44:19.117584+0000","last_active":"2023-03-09T14:45:14.857166+0000","last_peered":"2023-03-09T14:45:14.857166+0000","last_clean":"2023-03-09T14:45:14.857166+0000","last_became_active":"2023-03-09T14:44:19.117262+0000","last_became_peered":"2023-03-09T14:44:19.117262+0000","last_unstale":"2023-03-09T14:45:14.857166+0000","last_undegraded":"2023-03-09T14:45:14.857166+0000","last_fullsized":"2023-03-09T14:45:14.857166+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T19:19:38.080048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.861827+0000","last_change":"2023-03-09T14:44:23.485769+0000","last_active":"2023-03-09T14:45:14.861827+0000","last_peered":"2023-03-09T14:45:14.861827+0000","last_clean":"2023-03-09T14:45:14.861827+0000","last_became_active":"2023-03-09T14:44:23.485317+0000","last_became_peered":"2023-03-09T14:44:23.485317+0000","last_unstale":"2023-03-09T14:45:14.861827+0000","last_undegraded":"2023-03-09T14:45:14.861827+0000","last_fullsized":"2023-03-09T14:45:14.861827+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:58:35.140512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.852564+0000","last_change":"2023-03-09T14:44:19.112255+0000","last_active":"2023-03-09T14:45:14.852564+0000","last_peered":"2023-03-09T14:45:14.852564+0000","last_clean":"2023-03-09T14:45:14.852564+0000","last_became_active":"2023-03-09T14:44:19.111988+0000","last_became_peered":"2023-03-09T14:44:19.111988+0000","last_unstale":"2023-03-09T14:45:14.852564+0000","last_undegraded":"2023-03-09T14:45:14.852564+0000","last_fullsized":"2023-03-09T14:45:14.852564+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:55:50.423701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"87'4","reported_seq":62,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:17.594145+0000","last_change":"2023-03-09T14:44:18.665291+0000","last_active":"2023-03-09T14:45:17.594145+0000","last_peered":"2023-03-09T14:45:17.594145+0000","last_clean":"2023-03-09T14:45:17.594145+0000","last_became_active":"2023-03-09T14:44:18.664950+0000","last_became_peered":"2023-03-09T14:44:18.664950+0000","last_unstale":"2023-03-09T14:45:17.594145+0000","last_undegraded":"2023-03-09T14:45:17.594145+0000","last_fullsized":"2023-03-09T14:45:17.594145+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:40:58.619717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":40,"num_read_kb":40,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"87'1","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.853567+0000","last_change":"2023-03-09T14:44:23.480872+0000","last_active":"2023-03-09T14:45:14.853567+0000","last_peered":"2023-03-09T14:45:14.853567+0000","last_clean":"2023-03-09T14:45:14.853567+0000","last_became_active":"2023-03-09T14:44:23.480495+0000","last_became_peered":"2023-03-09T14:44:23.480495+0000","last_unstale":"2023-03-09T14:45:14.853567+0000","last_undegraded":"2023-03-09T14:45:14.853567+0000","last_fullsized":"2023-03-09T14:45:14.853567+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:19:01.393034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.863487+0000","last_change":"2023-03-09T14:44:19.102833+0000","last_active":"2023-03-09T14:45:14.863487+0000","last_peered":"2023-03-09T14:45:14.863487+0000","last_clean":"2023-03-09T14:45:14.863487+0000","last_became_active":"2023-03-09T14:44:19.102484+0000","last_became_peered":"2023-03-09T14:44:19.102484+0000","last_unstale":"2023-03-09T14:45:14.863487+0000","last_undegraded":"2023-03-09T14:45:14.863487+0000","last_fullsized":"2023-03-09T14:45:14.863487+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:14:50.002907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"87'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.859537+0000","last_change":"2023-03-09T14:44:18.661603+0000","last_active":"2023-03-09T14:45:14.859537+0000","last_peered":"2023-03-09T14:45:14.859537+0000","last_clean":"2023-03-09T14:45:14.859537+0000","last_became_active":"2023-03-09T14:44:18.661275+0000","last_became_peered":"2023-03-09T14:44:18.661275+0000","last_unstale":"2023-03-09T14:45:14.859537+0000","last_undegraded":"2023-03-09T14:45:14.859537+0000","last_fullsized":"2023-03-09T14:45:14.859537+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:20:31.873778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":378,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.852517+0000","last_change":"2023-03-09T14:43:19.317554+0000","last_active":"2023-03-09T14:45:14.852517+0000","last_peered":"2023-03-09T14:45:14.852517+0000","last_clean":"2023-03-09T14:45:14.852517+0000","last_became_active":"2023-03-09T14:43:19.316912+0000","last_became_peered":"2023-03-09T14:43:19.316912+0000","last_unstale":"2023-03-09T14:45:14.852517+0000","last_undegraded":"2023-03-09T14:45:14.852517+0000","last_fullsized":"2023-03-09T14:45:14.852517+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:40:23.197173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:40:23.197173+0000","last_clean_scrub_stamp":"2023-03-09T14:40:23.197173+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T20:21:17.143331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.852927+0000","last_change":"2023-03-09T14:44:23.487738+0000","last_active":"2023-03-09T14:45:14.852927+0000","last_peered":"2023-03-09T14:45:14.852927+0000","last_clean":"2023-03-09T14:45:14.852927+0000","last_became_active":"2023-03-09T14:44:23.487193+0000","last_became_peered":"2023-03-09T14:44:23.487193+0000","last_unstale":"2023-03-09T14:45:14.852927+0000","last_undegraded":"2023-03-09T14:45:14.852927+0000","last_fullsized":"2023-03-09T14:45:14.852927+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:48:08.789071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.860260+0000","last_change":"2023-03-09T14:44:19.112576+0000","last_active":"2023-03-09T14:45:14.860260+0000","last_peered":"2023-03-09T14:45:14.860260+0000","last_clean":"2023-03-09T14:45:14.860260+0000","last_became_active":"2023-03-09T14:44:19.112231+0000","last_became_peered":"2023-03-09T14:44:19.112231+0000","last_unstale":"2023-03-09T14:45:14.860260+0000","last_undegraded":"2023-03-09T14:45:14.860260+0000","last_fullsized":"2023-03-09T14:45:14.860260+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:25:21.918088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"87'3","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.852608+0000","last_change":"2023-03-09T14:44:18.655478+0000","last_active":"2023-03-09T14:45:14.852608+0000","last_peered":"2023-03-09T14:45:14.852608+0000","last_clean":"2023-03-09T14:45:14.852608+0000","last_became_active":"2023-03-09T14:44:18.655180+0000","last_became_peered":"2023-03-09T14:44:18.655180+0000","last_unstale":"2023-03-09T14:45:14.852608+0000","last_undegraded":"2023-03-09T14:45:14.852608+0000","last_fullsized":"2023-03-09T14:45:14.852608+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:42:47.926376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.856627+0000","last_change":"2023-03-09T14:44:23.487052+0000","last_active":"2023-03-09T14:45:14.856627+0000","last_peered":"2023-03-09T14:45:14.856627+0000","last_clean":"2023-03-09T14:45:14.856627+0000","last_became_active":"2023-03-09T14:44:23.486603+0000","last_became_peered":"2023-03-09T14:44:23.486603+0000","last_unstale":"2023-03-09T14:45:14.856627+0000","last_undegraded":"2023-03-09T14:45:14.856627+0000","last_fullsized":"2023-03-09T14:45:14.856627+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:28:25.375052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"87'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.861722+0000","last_change":"2023-03-09T14:44:18.666400+0000","last_active":"2023-03-09T14:45:14.861722+0000","last_peered":"2023-03-09T14:45:14.861722+0000","last_clean":"2023-03-09T14:45:14.861722+0000","last_became_active":"2023-03-09T14:44:18.666045+0000","last_became_peered":"2023-03-09T14:44:18.666045+0000","last_unstale":"2023-03-09T14:45:14.861722+0000","last_undegraded":"2023-03-09T14:45:14.861722+0000","last_fullsized":"2023-03-09T14:45:14.861722+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:30:48.640124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.862443+0000","last_change":"2023-03-09T14:44:19.103657+0000","last_active":"2023-03-09T14:45:14.862443+0000","last_peered":"2023-03-09T14:45:14.862443+0000","last_clean":"2023-03-09T14:45:14.862443+0000","last_became_active":"2023-03-09T14:44:19.103364+0000","last_became_peered":"2023-03-09T14:44:19.103364+0000","last_unstale":"2023-03-09T14:45:14.862443+0000","last_undegraded":"2023-03-09T14:45:14.862443+0000","last_fullsized":"2023-03-09T14:45:14.862443+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:23:23.738554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.864043+0000","last_change":"2023-03-09T14:44:23.483487+0000","last_active":"2023-03-09T14:45:14.864043+0000","last_peered":"2023-03-09T14:45:14.864043+0000","last_clean":"2023-03-09T14:45:14.864043+0000","last_became_active":"2023-03-09T14:44:23.483050+0000","last_became_peered":"2023-03-09T14:44:23.483050+0000","last_unstale":"2023-03-09T14:45:14.864043+0000","last_undegraded":"2023-03-09T14:45:14.864043+0000","last_fullsized":"2023-03-09T14:45:14.864043+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:38:34.129355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"87'5","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.860131+0000","last_change":"2023-03-09T14:44:18.661944+0000","last_active":"2023-03-09T14:45:14.860131+0000","last_peered":"2023-03-09T14:45:14.860131+0000","last_clean":"2023-03-09T14:45:14.860131+0000","last_became_active":"2023-03-09T14:44:18.661575+0000","last_became_peered":"2023-03-09T14:44:18.661575+0000","last_unstale":"2023-03-09T14:45:14.860131+0000","last_undegraded":"2023-03-09T14:45:14.860131+0000","last_fullsized":"2023-03-09T14:45:14.860131+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T01:12:53.637866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.864223+0000","last_change":"2023-03-09T14:44:19.103807+0000","last_active":"2023-03-09T14:45:14.864223+0000","last_peered":"2023-03-09T14:45:14.864223+0000","last_clean":"2023-03-09T14:45:14.864223+0000","last_became_active":"2023-03-09T14:44:19.103480+0000","last_became_peered":"2023-03-09T14:44:19.103480+0000","last_unstale":"2023-03-09T14:45:14.864223+0000","last_undegraded":"2023-03-09T14:45:14.864223+0000","last_fullsized":"2023-03-09T14:45:14.864223+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:29:54.629265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.856235+0000","last_change":"2023-03-09T14:44:23.486789+0000","last_active":"2023-03-09T14:45:14.856235+0000","last_peered":"2023-03-09T14:45:14.856235+0000","last_clean":"2023-03-09T14:45:14.856235+0000","last_became_active":"2023-03-09T14:44:23.486367+0000","last_became_peered":"2023-03-09T14:44:23.486367+0000","last_unstale":"2023-03-09T14:45:14.856235+0000","last_undegraded":"2023-03-09T14:45:14.856235+0000","last_fullsized":"2023-03-09T14:45:14.856235+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:13:13.081388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"87'4","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.860080+0000","last_change":"2023-03-09T14:44:18.659492+0000","last_active":"2023-03-09T14:45:14.860080+0000","last_peered":"2023-03-09T14:45:14.860080+0000","last_clean":"2023-03-09T14:45:14.860080+0000","last_became_active":"2023-03-09T14:44:18.659187+0000","last_became_peered":"2023-03-09T14:44:18.659187+0000","last_unstale":"2023-03-09T14:45:14.860080+0000","last_undegraded":"2023-03-09T14:45:14.860080+0000","last_fullsized":"2023-03-09T14:45:14.860080+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:44:31.342785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.864968+0000","last_change":"2023-03-09T14:44:19.112841+0000","last_active":"2023-03-09T14:45:14.864968+0000","last_peered":"2023-03-09T14:45:14.864968+0000","last_clean":"2023-03-09T14:45:14.864968+0000","last_became_active":"2023-03-09T14:44:19.112515+0000","last_became_peered":"2023-03-09T14:44:19.112515+0000","last_unstale":"2023-03-09T14:45:14.864968+0000","last_undegraded":"2023-03-09T14:45:14.864968+0000","last_fullsized":"2023-03-09T14:45:14.864968+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T19:41:04.065529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.859850+0000","last_change":"2023-03-09T14:44:23.481992+0000","last_active":"2023-03-09T14:45:14.859850+0000","last_peered":"2023-03-09T14:45:14.859850+0000","last_clean":"2023-03-09T14:45:14.859850+0000","last_became_active":"2023-03-09T14:44:23.481508+0000","last_became_peered":"2023-03-09T14:44:23.481508+0000","last_unstale":"2023-03-09T14:45:14.859850+0000","last_undegraded":"2023-03-09T14:45:14.859850+0000","last_fullsized":"2023-03-09T14:45:14.859850+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:46:48.683402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"87'2","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.856199+0000","last_change":"2023-03-09T14:44:18.663133+0000","last_active":"2023-03-09T14:45:14.856199+0000","last_peered":"2023-03-09T14:45:14.856199+0000","last_clean":"2023-03-09T14:45:14.856199+0000","last_became_active":"2023-03-09T14:44:18.662809+0000","last_became_peered":"2023-03-09T14:44:18.662809+0000","last_unstale":"2023-03-09T14:45:14.856199+0000","last_undegraded":"2023-03-09T14:45:14.856199+0000","last_fullsized":"2023-03-09T14:45:14.856199+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:32:16.180642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.860368+0000","last_change":"2023-03-09T14:44:19.108566+0000","last_active":"2023-03-09T14:45:14.860368+0000","last_peered":"2023-03-09T14:45:14.860368+0000","last_clean":"2023-03-09T14:45:14.860368+0000","last_became_active":"2023-03-09T14:44:19.108217+0000","last_became_peered":"2023-03-09T14:44:19.108217+0000","last_unstale":"2023-03-09T14:45:14.860368+0000","last_undegraded":"2023-03-09T14:45:14.860368+0000","last_fullsized":"2023-03-09T14:45:14.860368+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:03:28.434395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.860862+0000","last_change":"2023-03-09T14:44:23.488640+0000","last_active":"2023-03-09T14:45:14.860862+0000","last_peered":"2023-03-09T14:45:14.860862+0000","last_clean":"2023-03-09T14:45:14.860862+0000","last_became_active":"2023-03-09T14:44:23.488178+0000","last_became_peered":"2023-03-09T14:44:23.488178+0000","last_unstale":"2023-03-09T14:45:14.860862+0000","last_undegraded":"2023-03-09T14:45:14.860862+0000","last_fullsized":"2023-03-09T14:45:14.860862+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:33:49.908974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"87'3","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.864673+0000","last_change":"2023-03-09T14:44:18.662925+0000","last_active":"2023-03-09T14:45:14.864673+0000","last_peered":"2023-03-09T14:45:14.864673+0000","last_clean":"2023-03-09T14:45:14.864673+0000","last_became_active":"2023-03-09T14:44:18.662354+0000","last_became_peered":"2023-03-09T14:44:18.662354+0000","last_unstale":"2023-03-09T14:45:14.864673+0000","last_undegraded":"2023-03-09T14:45:14.864673+0000","last_fullsized":"2023-03-09T14:45:14.864673+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T01:00:36.602938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.856803+0000","last_change":"2023-03-09T14:44:19.106465+0000","last_active":"2023-03-09T14:45:14.856803+0000","last_peered":"2023-03-09T14:45:14.856803+0000","last_clean":"2023-03-09T14:45:14.856803+0000","last_became_active":"2023-03-09T14:44:19.106001+0000","last_became_peered":"2023-03-09T14:44:19.106001+0000","last_unstale":"2023-03-09T14:45:14.856803+0000","last_undegraded":"2023-03-09T14:45:14.856803+0000","last_fullsized":"2023-03-09T14:45:14.856803+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:22:09.149291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"87'1","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:15.873389+0000","last_change":"2023-03-09T14:45:15.873389+0000","last_active":"2023-03-09T14:45:15.873389+0000","last_peered":"2023-03-09T14:45:15.873389+0000","last_clean":"2023-03-09T14:45:15.873389+0000","last_became_active":"2023-03-09T14:45:15.863408+0000","last_became_peered":"2023-03-09T14:45:15.863408+0000","last_unstale":"2023-03-09T14:45:15.873389+0000","last_undegraded":"2023-03-09T14:45:15.873389+0000","last_fullsized":"2023-03-09T14:45:15.873389+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T16:33:12.565135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":123,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,7,8],"acting":[4,10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"87'1","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.856858+0000","last_change":"2023-03-09T14:44:18.664744+0000","last_active":"2023-03-09T14:45:14.856858+0000","last_peered":"2023-03-09T14:45:14.856858+0000","last_clean":"2023-03-09T14:45:14.856858+0000","last_became_active":"2023-03-09T14:44:18.664443+0000","last_became_peered":"2023-03-09T14:44:18.664443+0000","last_unstale":"2023-03-09T14:45:14.856858+0000","last_undegraded":"2023-03-09T14:45:14.856858+0000","last_fullsized":"2023-03-09T14:45:14.856858+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:11:13.836270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.863353+0000","last_change":"2023-03-09T14:44:19.115102+0000","last_active":"2023-03-09T14:45:14.863353+0000","last_peered":"2023-03-09T14:45:14.863353+0000","last_clean":"2023-03-09T14:45:14.863353+0000","last_became_active":"2023-03-09T14:44:19.114565+0000","last_became_peered":"2023-03-09T14:44:19.114565+0000","last_unstale":"2023-03-09T14:45:14.863353+0000","last_undegraded":"2023-03-09T14:45:14.863353+0000","last_fullsized":"2023-03-09T14:45:14.863353+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:39:11.930231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.859836+0000","last_change":"2023-03-09T14:44:23.482500+0000","last_active":"2023-03-09T14:45:14.859836+0000","last_peered":"2023-03-09T14:45:14.859836+0000","last_clean":"2023-03-09T14:45:14.859836+0000","last_became_active":"2023-03-09T14:44:23.482217+0000","last_became_peered":"2023-03-09T14:44:23.482217+0000","last_unstale":"2023-03-09T14:45:14.859836+0000","last_undegraded":"2023-03-09T14:45:14.859836+0000","last_fullsized":"2023-03-09T14:45:14.859836+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T14:52:44.275037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"87'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.860725+0000","last_change":"2023-03-09T14:44:18.663824+0000","last_active":"2023-03-09T14:45:14.860725+0000","last_peered":"2023-03-09T14:45:14.860725+0000","last_clean":"2023-03-09T14:45:14.860725+0000","last_became_active":"2023-03-09T14:44:18.663379+0000","last_became_peered":"2023-03-09T14:44:18.663379+0000","last_unstale":"2023-03-09T14:45:14.860725+0000","last_undegraded":"2023-03-09T14:45:14.860725+0000","last_fullsized":"2023-03-09T14:45:14.860725+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:03:04.031061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.852902+0000","last_change":"2023-03-09T14:44:19.105765+0000","last_active":"2023-03-09T14:45:14.852902+0000","last_peered":"2023-03-09T14:45:14.852902+0000","last_clean":"2023-03-09T14:45:14.852902+0000","last_became_active":"2023-03-09T14:44:19.105430+0000","last_became_peered":"2023-03-09T14:44:19.105430+0000","last_unstale":"2023-03-09T14:45:14.852902+0000","last_undegraded":"2023-03-09T14:45:14.852902+0000","last_fullsized":"2023-03-09T14:45:14.852902+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T20:31:40.106798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.856901+0000","last_change":"2023-03-09T14:44:23.482079+0000","last_active":"2023-03-09T14:45:14.856901+0000","last_peered":"2023-03-09T14:45:14.856901+0000","last_clean":"2023-03-09T14:45:14.856901+0000","last_became_active":"2023-03-09T14:44:23.481620+0000","last_became_peered":"2023-03-09T14:44:23.481620+0000","last_unstale":"2023-03-09T14:45:14.856901+0000","last_undegraded":"2023-03-09T14:45:14.856901+0000","last_fullsized":"2023-03-09T14:45:14.856901+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:47:53.589111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"87'3","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:15.869980+0000","last_change":"2023-03-09T14:45:15.869485+0000","last_active":"2023-03-09T14:45:15.869980+0000","last_peered":"2023-03-09T14:45:15.869980+0000","last_clean":"2023-03-09T14:45:15.869980+0000","last_became_active":"2023-03-09T14:45:15.861428+0000","last_became_peered":"2023-03-09T14:45:15.861428+0000","last_unstale":"2023-03-09T14:45:15.869980+0000","last_undegraded":"2023-03-09T14:45:15.869980+0000","last_fullsized":"2023-03-09T14:45:15.869980+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:43:30.309431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":566,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.853743+0000","last_change":"2023-03-09T14:44:19.105947+0000","last_active":"2023-03-09T14:45:14.853743+0000","last_peered":"2023-03-09T14:45:14.853743+0000","last_clean":"2023-03-09T14:45:14.853743+0000","last_became_active":"2023-03-09T14:44:19.105669+0000","last_became_peered":"2023-03-09T14:44:19.105669+0000","last_unstale":"2023-03-09T14:45:14.853743+0000","last_undegraded":"2023-03-09T14:45:14.853743+0000","last_fullsized":"2023-03-09T14:45:14.853743+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:29:58.615481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.863833+0000","last_change":"2023-03-09T14:44:23.483237+0000","last_active":"2023-03-09T14:45:14.863833+0000","last_peered":"2023-03-09T14:45:14.863833+0000","last_clean":"2023-03-09T14:45:14.863833+0000","last_became_active":"2023-03-09T14:44:23.482831+0000","last_became_peered":"2023-03-09T14:44:23.482831+0000","last_unstale":"2023-03-09T14:45:14.863833+0000","last_undegraded":"2023-03-09T14:45:14.863833+0000","last_fullsized":"2023-03-09T14:45:14.863833+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:36:40.775581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"82'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:15.867430+0000","last_change":"2023-03-09T14:45:15.867049+0000","last_active":"2023-03-09T14:45:15.867430+0000","last_peered":"2023-03-09T14:45:15.867430+0000","last_clean":"2023-03-09T14:45:15.867430+0000","last_became_active":"2023-03-09T14:45:15.863218+0000","last_became_peered":"2023-03-09T14:45:15.863218+0000","last_unstale":"2023-03-09T14:45:15.867430+0000","last_undegraded":"2023-03-09T14:45:15.867430+0000","last_fullsized":"2023-03-09T14:45:15.867430+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:38:21.800533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.853014+0000","last_change":"2023-03-09T14:44:19.107146+0000","last_active":"2023-03-09T14:45:14.853014+0000","last_peered":"2023-03-09T14:45:14.853014+0000","last_clean":"2023-03-09T14:45:14.853014+0000","last_became_active":"2023-03-09T14:44:19.106834+0000","last_became_peered":"2023-03-09T14:44:19.106834+0000","last_unstale":"2023-03-09T14:45:14.853014+0000","last_undegraded":"2023-03-09T14:45:14.853014+0000","last_fullsized":"2023-03-09T14:45:14.853014+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T20:15:20.142449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.852185+0000","last_change":"2023-03-09T14:44:23.482811+0000","last_active":"2023-03-09T14:45:14.852185+0000","last_peered":"2023-03-09T14:45:14.852185+0000","last_clean":"2023-03-09T14:45:14.852185+0000","last_became_active":"2023-03-09T14:44:23.482371+0000","last_became_peered":"2023-03-09T14:44:23.482371+0000","last_unstale":"2023-03-09T14:45:14.852185+0000","last_undegraded":"2023-03-09T14:45:14.852185+0000","last_fullsized":"2023-03-09T14:45:14.852185+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T01:33:14.442810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"82'2","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.861583+0000","last_change":"2023-03-09T14:44:18.661569+0000","last_active":"2023-03-09T14:45:14.861583+0000","last_peered":"2023-03-09T14:45:14.861583+0000","last_clean":"2023-03-09T14:45:14.861583+0000","last_became_active":"2023-03-09T14:44:18.661266+0000","last_became_peered":"2023-03-09T14:44:18.661266+0000","last_unstale":"2023-03-09T14:45:14.861583+0000","last_undegraded":"2023-03-09T14:45:14.861583+0000","last_fullsized":"2023-03-09T14:45:14.861583+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:47:43.066180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.853803+0000","last_change":"2023-03-09T14:44:19.106611+0000","last_active":"2023-03-09T14:45:14.853803+0000","last_peered":"2023-03-09T14:45:14.853803+0000","last_clean":"2023-03-09T14:45:14.853803+0000","last_became_active":"2023-03-09T14:44:19.106324+0000","last_became_peered":"2023-03-09T14:44:19.106324+0000","last_unstale":"2023-03-09T14:45:14.853803+0000","last_undegraded":"2023-03-09T14:45:14.853803+0000","last_fullsized":"2023-03-09T14:45:14.853803+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:50:46.561502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.856648+0000","last_change":"2023-03-09T14:44:19.106292+0000","last_active":"2023-03-09T14:45:14.856648+0000","last_peered":"2023-03-09T14:45:14.856648+0000","last_clean":"2023-03-09T14:45:14.856648+0000","last_became_active":"2023-03-09T14:44:19.105607+0000","last_became_peered":"2023-03-09T14:44:19.105607+0000","last_unstale":"2023-03-09T14:45:14.856648+0000","last_undegraded":"2023-03-09T14:45:14.856648+0000","last_fullsized":"2023-03-09T14:45:14.856648+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T19:11:28.125433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.856646+0000","last_change":"2023-03-09T14:44:19.103423+0000","last_active":"2023-03-09T14:45:14.856646+0000","last_peered":"2023-03-09T14:45:14.856646+0000","last_clean":"2023-03-09T14:45:14.856646+0000","last_became_active":"2023-03-09T14:44:19.102918+0000","last_became_peered":"2023-03-09T14:44:19.102918+0000","last_unstale":"2023-03-09T14:45:14.856646+0000","last_undegraded":"2023-03-09T14:45:14.856646+0000","last_fullsized":"2023-03-09T14:45:14.856646+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:47:52.921642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.856208+0000","last_change":"2023-03-09T14:44:19.113511+0000","last_active":"2023-03-09T14:45:14.856208+0000","last_peered":"2023-03-09T14:45:14.856208+0000","last_clean":"2023-03-09T14:45:14.856208+0000","last_became_active":"2023-03-09T14:44:19.113224+0000","last_became_peered":"2023-03-09T14:44:19.113224+0000","last_unstale":"2023-03-09T14:45:14.856208+0000","last_undegraded":"2023-03-09T14:45:14.856208+0000","last_fullsized":"2023-03-09T14:45:14.856208+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T02:03:28.069616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.864074+0000","last_change":"2023-03-09T14:44:19.116174+0000","last_active":"2023-03-09T14:45:14.864074+0000","last_peered":"2023-03-09T14:45:14.864074+0000","last_clean":"2023-03-09T14:45:14.864074+0000","last_became_active":"2023-03-09T14:44:19.115813+0000","last_became_peered":"2023-03-09T14:44:19.115813+0000","last_unstale":"2023-03-09T14:45:14.864074+0000","last_undegraded":"2023-03-09T14:45:14.864074+0000","last_fullsized":"2023-03-09T14:45:14.864074+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:28:49.573771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.862549+0000","last_change":"2023-03-09T14:44:19.115287+0000","last_active":"2023-03-09T14:45:14.862549+0000","last_peered":"2023-03-09T14:45:14.862549+0000","last_clean":"2023-03-09T14:45:14.862549+0000","last_became_active":"2023-03-09T14:44:19.113458+0000","last_became_peered":"2023-03-09T14:44:19.113458+0000","last_unstale":"2023-03-09T14:45:14.862549+0000","last_undegraded":"2023-03-09T14:45:14.862549+0000","last_fullsized":"2023-03-09T14:45:14.862549+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:43:26.583237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.860272+0000","last_change":"2023-03-09T14:44:19.107308+0000","last_active":"2023-03-09T14:45:14.860272+0000","last_peered":"2023-03-09T14:45:14.860272+0000","last_clean":"2023-03-09T14:45:14.860272+0000","last_became_active":"2023-03-09T14:44:19.107005+0000","last_became_peered":"2023-03-09T14:44:19.107005+0000","last_unstale":"2023-03-09T14:45:14.860272+0000","last_undegraded":"2023-03-09T14:45:14.860272+0000","last_fullsized":"2023-03-09T14:45:14.860272+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:48:34.886664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.857474+0000","last_change":"2023-03-09T14:44:19.098454+0000","last_active":"2023-03-09T14:45:14.857474+0000","last_peered":"2023-03-09T14:45:14.857474+0000","last_clean":"2023-03-09T14:45:14.857474+0000","last_became_active":"2023-03-09T14:44:19.098146+0000","last_became_peered":"2023-03-09T14:44:19.098146+0000","last_unstale":"2023-03-09T14:45:14.857474+0000","last_undegraded":"2023-03-09T14:45:14.857474+0000","last_fullsized":"2023-03-09T14:45:14.857474+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:11:40.475674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.864884+0000","last_change":"2023-03-09T14:44:19.095947+0000","last_active":"2023-03-09T14:45:14.864884+0000","last_peered":"2023-03-09T14:45:14.864884+0000","last_clean":"2023-03-09T14:45:14.864884+0000","last_became_active":"2023-03-09T14:44:19.095640+0000","last_became_peered":"2023-03-09T14:44:19.095640+0000","last_unstale":"2023-03-09T14:45:14.864884+0000","last_undegraded":"2023-03-09T14:45:14.864884+0000","last_fullsized":"2023-03-09T14:45:14.864884+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T02:04:35.384016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.852390+0000","last_change":"2023-03-09T14:44:19.113927+0000","last_active":"2023-03-09T14:45:14.852390+0000","last_peered":"2023-03-09T14:45:14.852390+0000","last_clean":"2023-03-09T14:45:14.852390+0000","last_became_active":"2023-03-09T14:44:19.113163+0000","last_became_peered":"2023-03-09T14:44:19.113163+0000","last_unstale":"2023-03-09T14:45:14.852390+0000","last_undegraded":"2023-03-09T14:45:14.852390+0000","last_fullsized":"2023-03-09T14:45:14.852390+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T01:35:35.801317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.864582+0000","last_change":"2023-03-09T14:44:19.098415+0000","last_active":"2023-03-09T14:45:14.864582+0000","last_peered":"2023-03-09T14:45:14.864582+0000","last_clean":"2023-03-09T14:45:14.864582+0000","last_became_active":"2023-03-09T14:44:19.097600+0000","last_became_peered":"2023-03-09T14:44:19.097600+0000","last_unstale":"2023-03-09T14:45:14.864582+0000","last_undegraded":"2023-03-09T14:45:14.864582+0000","last_fullsized":"2023-03-09T14:45:14.864582+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:15:08.552402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.857512+0000","last_change":"2023-03-09T14:44:19.106386+0000","last_active":"2023-03-09T14:45:14.857512+0000","last_peered":"2023-03-09T14:45:14.857512+0000","last_clean":"2023-03-09T14:45:14.857512+0000","last_became_active":"2023-03-09T14:44:19.106112+0000","last_became_peered":"2023-03-09T14:44:19.106112+0000","last_unstale":"2023-03-09T14:45:14.857512+0000","last_undegraded":"2023-03-09T14:45:14.857512+0000","last_fullsized":"2023-03-09T14:45:14.857512+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T20:27:50.992168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.861146+0000","last_change":"2023-03-09T14:44:19.114534+0000","last_active":"2023-03-09T14:45:14.861146+0000","last_peered":"2023-03-09T14:45:14.861146+0000","last_clean":"2023-03-09T14:45:14.861146+0000","last_became_active":"2023-03-09T14:44:19.114226+0000","last_became_peered":"2023-03-09T14:44:19.114226+0000","last_unstale":"2023-03-09T14:45:14.861146+0000","last_undegraded":"2023-03-09T14:45:14.861146+0000","last_fullsized":"2023-03-09T14:45:14.861146+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:22:38.453116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.858390+0000","last_change":"2023-03-09T14:44:19.099195+0000","last_active":"2023-03-09T14:45:14.858390+0000","last_peered":"2023-03-09T14:45:14.858390+0000","last_clean":"2023-03-09T14:45:14.858390+0000","last_became_active":"2023-03-09T14:44:19.098892+0000","last_became_peered":"2023-03-09T14:44:19.098892+0000","last_unstale":"2023-03-09T14:45:14.858390+0000","last_undegraded":"2023-03-09T14:45:14.858390+0000","last_fullsized":"2023-03-09T14:45:14.858390+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:29:24.858200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":246,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":123,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":60618,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":272,"num_read_kb":198,"num_write":152,"num_write_kb":104,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":1914,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":181854,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134,"ondisk_log_size":134,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":78,"seq":335007449112,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6172,"kb_used_data":980,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93738980,"statfs":{"total":95995035648,"available":95988715520,"internally_reserved":0,"allocated":1003520,"data_stored":490897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:44:39 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.623},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.52100000000000002}]},{"osd":1,"last update":"Thu Mar 9 14:44:39 2023","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.378,"5min":0.378,"15min":0.378},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.90000000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.64400000000000002}]},{"osd":2,"last update":"Thu Mar 9 14:44:39 2023","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":1.2829999999999999,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.78700000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]},{"osd":4,"last update":"Thu Mar 9 14:44:39 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.113,"5min":1.113,"15min":1.113},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.67100000000000004}]},{"osd":5,"last update":"Thu Mar 9 14:44:39 2023","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.70099999999999996}]},{"osd":6,"last update":"Thu Mar 9 14:44:39 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.67800000000000005}]},{"osd":7,"last update":"Thu Mar 9 14:44:39 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"last":0.94299999999999995},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.754}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73599999999999999}]},{"osd":9,"last update":"Thu Mar 9 14:44:39 2023","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.3240000000000001,"5min":1.3240000000000001,"15min":1.3240000000000001},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.96999999999999997}]},{"osd":10,"last update":"Thu Mar 9 14:44:39 2023","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.746,"15min":0.746},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.431,"5min":1.431,"15min":1.431},"last":0.79700000000000004}]}]},{"osd":10,"up_from":70,"seq":300647710747,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7156,"kb_used_data":1644,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93737996,"statfs":{"total":95995035648,"available":95987707904,"internally_reserved":0,"allocated":1683456,"data_stored":1138916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:45:19 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.36099999999999999}]},{"osd":1,"last update":"Thu Mar 9 14:45:19 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.46700000000000003,"15min":0.46999999999999997},"min":{"1min":0.30499999999999999,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.56699999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.47799999999999998,"15min":0.48199999999999998},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.61099999999999999,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.46800000000000003}]},{"osd":2,"last update":"Thu Mar 9 14:45:19 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.63600000000000001,"15min":0.63900000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.753,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.628,"5min":0.623,"15min":0.622},"min":{"1min":0.46400000000000002,"5min":0.441,"15min":0.441},"max":{"1min":0.82199999999999995,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.57599999999999996}]},{"osd":3,"last update":"Thu Mar 9 14:45:19 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.58399999999999996,"15min":0.58899999999999997},"min":{"1min":0.38300000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.74399999999999999,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.64200000000000002,"15min":0.64800000000000002},"min":{"1min":0.46600000000000003,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.76100000000000001,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.59999999999999998}]},{"osd":4,"last update":"Thu Mar 9 14:45:19 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.59099999999999997,"15min":0.59299999999999997},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.78200000000000003,"5min":1.1719999999999999,"15min":1.1719999999999999},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.54500000000000004,"15min":0.54900000000000004},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.77500000000000002,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.52500000000000002}]},{"osd":5,"last update":"Thu Mar 9 14:45:19 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.504,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.92700000000000005,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.62},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.90300000000000002,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.76900000000000002}]},{"osd":6,"last update":"Thu Mar 9 14:45:19 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.68300000000000005,"15min":0.68899999999999995},"min":{"1min":0.48799999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.86499999999999999,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.71099999999999997,"15min":0.71599999999999997},"min":{"1min":0.499,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.82699999999999996,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.73199999999999998}]},{"osd":7,"last update":"Thu Mar 9 14:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.327,"5min":1.327,"15min":1.327},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.048,"5min":1.048,"15min":1.048},"last":0.53800000000000003}]},{"osd":8,"last update":"Thu Mar 9 14:45:19 2023","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.76000000000000001,"15min":0.76200000000000001},"min":{"1min":0.54900000000000004,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":0.91000000000000003,"5min":1.246,"15min":1.246},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.77300000000000002,"15min":0.78200000000000003},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.95499999999999996,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.71199999999999997}]},{"osd":9,"last update":"Thu Mar 9 14:45:19 2023","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.77600000000000002,"15min":0.78400000000000003},"min":{"1min":0.54200000000000004,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.88100000000000001,"5min":1.298,"15min":1.298},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.77700000000000002,"15min":0.79200000000000004},"min":{"1min":0.47499999999999998,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.89200000000000002,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":0.67300000000000004}]},{"osd":11,"last update":"Thu Mar 9 14:44:41 2023","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.68500000000000005}]}]},{"osd":9,"up_from":65,"seq":279172874272,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6376,"kb_used_data":992,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738776,"statfs":{"total":95995035648,"available":95988506624,"internally_reserved":0,"allocated":1015808,"data_stored":491063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:45:00 2023","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.46200000000000002,"15min":0.47399999999999998},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.66300000000000003,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.49399999999999999,"15min":0.50700000000000001},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.63700000000000001,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.36699999999999999}]},{"osd":1,"last update":"Thu Mar 9 14:45:00 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.626,"15min":0.63600000000000001},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.008,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.63700000000000001,"15min":0.65000000000000002},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.79900000000000004,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.46000000000000002}]},{"osd":2,"last update":"Thu Mar 9 14:45:00 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.64600000000000002,"15min":0.65800000000000003},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.66100000000000003,"15min":0.67400000000000004},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.88300000000000001,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.54300000000000004}]},{"osd":3,"last update":"Thu Mar 9 14:45:00 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.56999999999999995,"15min":0.57999999999999996},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.59499999999999997,"15min":0.61499999999999999},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.85099999999999998,"5min":1.016,"15min":1.016},"last":0.34899999999999998}]},{"osd":4,"last update":"Thu Mar 9 14:45:00 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.67900000000000005,"15min":0.69699999999999995},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.80400000000000005,"5min":1.04,"15min":1.04},"last":0.505},{"interface":"front","average":{"1min":0.621,"5min":0.71699999999999997,"15min":0.73299999999999998},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.93899999999999995,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.48899999999999999}]},{"osd":5,"last update":"Thu Mar 9 14:45:00 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.73299999999999998,"15min":0.745},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.98599999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.74099999999999999,"15min":0.754},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.0369999999999999,"5min":1.069,"15min":1.069},"last":0.59299999999999997}]},{"osd":6,"last update":"Thu Mar 9 14:43:55 2023","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.623}]},{"osd":7,"last update":"Thu Mar 9 14:45:00 2023","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.79800000000000004,"15min":0.82599999999999996},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.0229999999999999,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.80900000000000005,"15min":0.83099999999999996},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.99299999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.497}]},{"osd":8,"last update":"Thu Mar 9 14:45:00 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.80200000000000005,"15min":0.81599999999999995},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.032,"5min":1.133,"15min":1.133},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.81999999999999995,"15min":0.83799999999999997},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.081,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.625}]},{"osd":10,"last update":"Thu Mar 9 14:44:18 2023","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.57999999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65000000000000002}]}]},{"osd":8,"up_from":57,"seq":244813135909,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6496,"kb_used_data":984,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738656,"statfs":{"total":95995035648,"available":95988383744,"internally_reserved":0,"allocated":1007616,"data_stored":490086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.33300000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.69199999999999995,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.54900000000000004,"15min":0.55100000000000005},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.83799999999999997,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.57899999999999996}]},{"osd":1,"last update":"Thu Mar 9 14:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.54700000000000004,"15min":0.53900000000000003},"min":{"1min":0.376,"5min":0.313,"15min":0.313},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.56100000000000005,"15min":0.55900000000000005},"min":{"1min":0.34399999999999997,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.73299999999999998}]},{"osd":2,"last update":"Thu Mar 9 14:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.42199999999999999,"15min":0.41599999999999998},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.432,"15min":0.42699999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.66900000000000004},"last":0.60499999999999998}]},{"osd":3,"last update":"Thu Mar 9 14:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.55900000000000005,"15min":0.56100000000000005},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.81200000000000006,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.498},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.63100000000000001,"15min":0.61499999999999999},"min":{"1min":0.40600000000000003,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":3.9300000000000002,"5min":3.9300000000000002,"15min":3.9300000000000002},"last":0.64400000000000002}]},{"osd":4,"last update":"Thu Mar 9 14:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.56399999999999995,"15min":0.55600000000000005},"min":{"1min":0.36599999999999999,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.627},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.59299999999999997,"15min":0.58499999999999996},"min":{"1min":0.45600000000000002,"5min":0.376,"15min":0.376},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.69799999999999995}]},{"osd":5,"last update":"Thu Mar 9 14:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.53400000000000003,"15min":0.53200000000000003},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.54200000000000004,"15min":0.53600000000000003},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.66400000000000003}]},{"osd":6,"last update":"Thu Mar 9 14:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.69599999999999995,"15min":0.67900000000000005},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":3.895,"5min":3.895,"15min":3.895},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.65000000000000002,"15min":0.64900000000000002},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.68400000000000005}]},{"osd":7,"last update":"Thu Mar 9 14:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.66400000000000003,"15min":0.66200000000000003},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.66800000000000004,"15min":0.66300000000000003},"min":{"1min":0.49399999999999999,"5min":0.44,"15min":0.44},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.77200000000000002}]},{"osd":9,"last update":"Thu Mar 9 14:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.67600000000000005,"15min":0.66800000000000004},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.036,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.754,"5min":0.68600000000000005,"15min":0.67500000000000004},"min":{"1min":0.52900000000000003,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.119,"5min":1.119,"15min":1.119},"last":0.80100000000000005}]},{"osd":10,"last update":"Thu Mar 9 14:45:18 2023","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.70099999999999996,"15min":0.68000000000000005},"min":{"1min":0.59499999999999997,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.099,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.75},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.70899999999999996,"15min":0.68700000000000006},"min":{"1min":0.60599999999999998,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.85499999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81399999999999995}]}]},{"osd":7,"up_from":50,"seq":214748364842,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7148,"kb_used_data":1572,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738004,"statfs":{"total":95995035648,"available":95987716096,"internally_reserved":0,"allocated":1609728,"data_stored":1086891,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:44:13 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.52100000000000002,"15min":0.51700000000000002},"min":{"1min":0.316,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.69999999999999996,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.375},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.51900000000000002,"15min":0.50900000000000001},"min":{"1min":0.32800000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.90100000000000002,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.33900000000000002}]},{"osd":1,"last update":"Thu Mar 9 14:45:14 2023","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.40600000000000003,"15min":0.39400000000000002},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.54500000000000004,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.40600000000000003,"15min":0.39100000000000001},"min":{"1min":0.29799999999999999,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.56999999999999995,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.36699999999999999}]},{"osd":2,"last update":"Thu Mar 9 14:45:14 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.54600000000000004,"15min":0.53400000000000003},"min":{"1min":0.33500000000000002,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.754,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.58399999999999996,"15min":0.56999999999999995},"min":{"1min":0.44600000000000001,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.80700000000000005,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.57499999999999996}]},{"osd":3,"last update":"Thu Mar 9 14:45:14 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.55700000000000005,"15min":0.53800000000000003},"min":{"1min":0.40200000000000002,"5min":0.37,"15min":0.37},"max":{"1min":0.76000000000000001,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.57999999999999996,"15min":0.56399999999999995},"min":{"1min":0.44,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.76400000000000001,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.54100000000000004}]},{"osd":4,"last update":"Thu Mar 9 14:45:14 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48599999999999999,"15min":0.47199999999999998},"min":{"1min":0.34799999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.65100000000000002,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.47599999999999998,"15min":0.46000000000000002},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.68100000000000005,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.42999999999999999}]},{"osd":5,"last update":"Thu Mar 9 14:45:14 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.58799999999999997,"15min":0.57399999999999995},"min":{"1min":0.40300000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.76200000000000001,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.622},{"interface":"front","average":{"1min":0.623,"5min":0.59499999999999997,"15min":0.57799999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.80200000000000005,"5min":0.874,"15min":0.874},"last":0.65100000000000002}]},{"osd":6,"last update":"Thu Mar 9 14:45:14 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.63300000000000001,"15min":0.61699999999999999},"min":{"1min":0.50600000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.623,"15min":0.61299999999999999},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.81000000000000005,"5min":0.878,"15min":0.878},"last":0.69899999999999995}]},{"osd":8,"last update":"Thu Mar 9 14:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.65400000000000003,"15min":0.65500000000000003},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.85199999999999998,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.65800000000000003,"5min":0.64900000000000002,"15min":0.64700000000000002},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.88300000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.59899999999999998}]},{"osd":9,"last update":"Thu Mar 9 14:44:55 2023","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.69299999999999995,"15min":0.69199999999999995},"min":{"1min":0.52600000000000002,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":0.91400000000000003,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.70199999999999996,"15min":0.70399999999999996},"min":{"1min":0.52000000000000002,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":0.88700000000000001,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.72699999999999998}]},{"osd":10,"last update":"Thu Mar 9 14:45:20 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.66300000000000003,"15min":0.66600000000000004},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.83799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.70299999999999996,"15min":0.70099999999999996},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.81799999999999995,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.77900000000000003}]},{"osd":11,"last update":"Thu Mar 9 14:45:20 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.751}]}]},{"osd":6,"up_from":45,"seq":193273528366,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6488,"kb_used_data":976,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738664,"statfs":{"total":95995035648,"available":95988391936,"internally_reserved":0,"allocated":999424,"data_stored":486345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:44:55 2023","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.41599999999999998,"15min":0.41399999999999998},"min":{"1min":0.28799999999999998,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.56000000000000005,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.498,"15min":0.49199999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.70599999999999996,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.53400000000000003}]},{"osd":1,"last update":"Thu Mar 9 14:44:55 2023","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.58299999999999996,"15min":0.59899999999999998},"min":{"1min":0.36599999999999999,"5min":0.315,"15min":0.315},"max":{"1min":0.61399999999999999,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.55700000000000005}]},{"osd":2,"last update":"Thu Mar 9 14:44:55 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.60999999999999999,"15min":0.63700000000000001},"min":{"1min":0.37,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.72299999999999998,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.496},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.63,"15min":0.63600000000000001},"min":{"1min":0.41799999999999998,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.78300000000000003,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.59099999999999997}]},{"osd":3,"last update":"Thu Mar 9 14:44:55 2023","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46500000000000002,"15min":0.46300000000000002},"min":{"1min":0.30499999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.69699999999999995,"5min":0.745,"15min":0.745},"last":0.436},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.56399999999999995,"15min":0.55800000000000005},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.84199999999999997,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.57199999999999995}]},{"osd":4,"last update":"Thu Mar 9 14:44:55 2023","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.68600000000000005,"15min":0.73699999999999999},"min":{"1min":0.40600000000000003,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.67100000000000004,"5min":2.0840000000000001,"15min":2.0840000000000001},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.81699999999999995,"15min":0.89500000000000002},"min":{"1min":0.47699999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.82599999999999996,"5min":4.1440000000000001,"15min":4.1440000000000001},"last":0.69299999999999995}]},{"osd":5,"last update":"Thu Mar 9 14:44:55 2023","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.69699999999999995,"15min":0.69999999999999996},"min":{"1min":0.51400000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.83899999999999997,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.66000000000000003,"15min":0.68200000000000005},"min":{"1min":0.47799999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.74399999999999999,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.61499999999999999}]},{"osd":7,"last update":"Thu Mar 9 14:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.69999999999999996,"15min":0.70899999999999996},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.629},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.72999999999999998,"15min":0.73599999999999999},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.96499999999999997,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.76300000000000001}]},{"osd":8,"last update":"Thu Mar 9 14:44:35 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.77400000000000002,"15min":0.78800000000000003},"min":{"1min":0.54200000000000004,"5min":0.497,"15min":0.497},"max":{"1min":0.92500000000000004,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.70599999999999996,"15min":0.72199999999999998},"min":{"1min":0.502,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.81200000000000006,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.68100000000000005}]},{"osd":9,"last update":"Thu Mar 9 14:44:55 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.67000000000000004,"15min":0.68200000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.88300000000000001,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.73499999999999999,"15min":0.751},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.92700000000000005,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.60599999999999998}]},{"osd":10,"last update":"Thu Mar 9 14:44:16 2023","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.79000000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.749}]}]},{"osd":1,"up_from":13,"seq":55834574914,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7572,"kb_used_data":1036,"kb_used_omap":0,"kb_used_meta":6528,"kb_avail":93737580,"statfs":{"total":95995035648,"available":95987281920,"internally_reserved":0,"allocated":1060864,"data_stored":543397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6684672},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:45:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.50900000000000001,"15min":0.49299999999999999},"min":{"1min":0.36599999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.63300000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.55100000000000005,"15min":0.497},"min":{"1min":0.42099999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.88700000000000001,"5min":2.0670000000000002,"15min":2.0670000000000002},"last":0.66100000000000003}]},{"osd":2,"last update":"Thu Mar 9 14:44:31 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.54800000000000004,"15min":0.53000000000000003},"min":{"1min":0.38100000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.85399999999999998,"5min":1.3839999999999999,"15min":1.3839999999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.52700000000000002,"15min":0.50700000000000001},"min":{"1min":0.376,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.77600000000000002,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.64100000000000001}]},{"osd":3,"last update":"Thu Mar 9 14:44:55 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.55100000000000005,"15min":0.50900000000000001},"min":{"1min":0.39000000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.73399999999999999,"5min":2.206,"15min":2.206},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.56200000000000006,"15min":0.52600000000000002},"min":{"1min":0.39700000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.84499999999999997,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.76600000000000001}]},{"osd":4,"last update":"Thu Mar 9 14:45:18 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.441,"15min":0.39500000000000002},"min":{"1min":0.39900000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.434,"15min":0.39000000000000001},"min":{"1min":0.34100000000000003,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.434}]},{"osd":5,"last update":"Thu Mar 9 14:44:31 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.57299999999999995,"15min":0.54600000000000004},"min":{"1min":0.47999999999999998,"5min":0.373,"15min":0.373},"max":{"1min":0.92800000000000005,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.57599999999999996,"15min":0.54600000000000004},"min":{"1min":0.504,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.85799999999999998,"5min":1.363,"15min":1.363},"last":0.755}]},{"osd":6,"last update":"Thu Mar 9 14:44:55 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.64800000000000002,"15min":0.65400000000000003},"min":{"1min":0.376,"5min":0.371,"15min":0.371},"max":{"1min":0.84699999999999998,"5min":2.1930000000000001,"15min":2.1930000000000001},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.65600000000000003,"15min":0.66400000000000003},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.81699999999999995,"5min":2.2519999999999998,"15min":2.2519999999999998},"last":0.82899999999999996}]},{"osd":7,"last update":"Thu Mar 9 14:45:11 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.55100000000000005,"15min":0.5},"min":{"1min":0.33800000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.55400000000000005,"15min":0.51600000000000001},"min":{"1min":0.35799999999999998,"5min":0.31,"15min":0.31},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.61399999999999999}]},{"osd":8,"last update":"Thu Mar 9 14:44:33 2023","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.624,"15min":0.61199999999999999},"min":{"1min":0.503,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.88100000000000001,"5min":1.3009999999999999,"15min":1.3009999999999999},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.68100000000000005,"15min":0.67100000000000004},"min":{"1min":0.496,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.98299999999999998,"5min":1.423,"15min":1.423},"last":0.871}]},{"osd":9,"last update":"Thu Mar 9 14:44:58 2023","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.68200000000000005,"15min":0.66400000000000003},"min":{"1min":0.49399999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.68799999999999994,"15min":0.67300000000000004},"min":{"1min":0.53300000000000003,"5min":0.376,"15min":0.376},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.79000000000000004}]},{"osd":10,"last update":"Thu Mar 9 14:45:19 2023","interfaces":[{"interface":"back","average":{"1min":0.86799999999999999,"5min":0.75600000000000001,"15min":0.73699999999999999},"min":{"1min":0.66800000000000004,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.77100000000000002,"15min":0.75800000000000001},"min":{"1min":0.68000000000000005,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.0109999999999999,"5min":1.022,"15min":1.022},"last":0.82099999999999995}]},{"osd":11,"last update":"Thu Mar 9 14:45:18 2023","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.1559999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.89400000000000002},{"interface":"front","average":{"1min":0.873,"5min":0.873,"15min":0.873},"min":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.85799999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738440,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7340,"kb_used_data":996,"kb_used_omap":0,"kb_used_meta":6336,"kb_avail":93737812,"statfs":{"total":95995035648,"available":95987519488,"internally_reserved":0,"allocated":1019904,"data_stored":487611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6488064},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Mar 9 14:45:11 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.55200000000000005,"15min":0.52200000000000002},"min":{"1min":0.375,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.67700000000000005,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.54300000000000004,"15min":0.52700000000000002},"min":{"1min":0.371,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.78900000000000003,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.66700000000000004}]},{"osd":2,"last update":"Thu Mar 9 14:44:31 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.55800000000000005,"15min":0.55300000000000005},"min":{"1min":0.41599999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.71399999999999997,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.59999999999999998,"15min":0.57999999999999996},"min":{"1min":0.49199999999999999,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.76000000000000001,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.54100000000000004}]},{"osd":3,"last update":"Thu Mar 9 14:42:45 2023","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.35799999999999998,"15min":0.34999999999999998},"min":{"1min":0.27800000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.39200000000000002,"15min":0.38800000000000001},"min":{"1min":0.26400000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.59499999999999997,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.38600000000000001}]},{"osd":4,"last update":"Thu Mar 9 14:45:11 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.59699999999999998,"15min":0.57099999999999995},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.78400000000000003,"5min":1.139,"15min":1.139},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.55800000000000005,"15min":0.54600000000000004},"min":{"1min":0.38700000000000001,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.88900000000000001,"5min":1.071,"15min":1.071},"last":0.54900000000000004}]},{"osd":5,"last update":"Thu Mar 9 14:44:37 2023","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.58499999999999996,"15min":0.56299999999999994},"min":{"1min":0.48599999999999999,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.748,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.64500000000000002,"15min":0.626},"min":{"1min":0.46999999999999997,"5min":0.378,"15min":0.378},"max":{"1min":0.78600000000000003,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.58099999999999996}]},{"osd":6,"last update":"Thu Mar 9 14:44:52 2023","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.53400000000000003,"15min":0.504},"min":{"1min":0.374,"5min":0.309,"15min":0.309},"max":{"1min":0.77900000000000003,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.44},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.57799999999999996,"15min":0.54800000000000004},"min":{"1min":0.38300000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.745,"5min":1.159,"15min":1.159},"last":0.496}]},{"osd":7,"last update":"Thu Mar 9 14:44:14 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.71299999999999997,"15min":0.71099999999999997},"min":{"1min":0.503,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.0760000000000001,"5min":1.2110000000000001,"15min":1.2110000000000001},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.64900000000000002,"15min":0.64500000000000002},"min":{"1min":0.52700000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.88500000000000001,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.70299999999999996}]},{"osd":8,"last update":"Thu Mar 9 14:44:37 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.72199999999999998,"15min":0.73299999999999998},"min":{"1min":0.57299999999999995,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.77900000000000003,"5min":1.038,"15min":1.038},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.80600000000000005,"15min":0.82299999999999995},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.81399999999999995,"5min":1.242,"15min":1.242},"last":0.56499999999999995}]},{"osd":9,"last update":"Thu Mar 9 14:44:59 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.74299999999999999,"15min":0.75900000000000001},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.90600000000000003,"5min":1.105,"15min":1.105},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.82199999999999995,"15min":0.85399999999999998},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.88800000000000001,"5min":1.27,"15min":1.27},"last":0.59599999999999997}]},{"osd":10,"last update":"Thu Mar 9 14:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.151,"5min":1.151,"15min":1.151},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.626}]},{"osd":11,"last update":"Thu Mar 9 14:44:37 2023","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"min":{"1min":0.622,"5min":0.622,"15min":0.622},"max":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.65700000000000003}]}]},{"osd":2,"up_from":18,"seq":77309411391,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7580,"kb_used_data":1044,"kb_used_omap":0,"kb_used_meta":6528,"kb_avail":93737572,"statfs":{"total":95995035648,"available":95987273728,"internally_reserved":0,"allocated":1069056,"data_stored":542977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6684672},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:44:30 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.51500000000000001,"15min":0.495},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.436},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.47899999999999998,"15min":0.46899999999999997},"min":{"1min":0.30199999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.32400000000000001}]},{"osd":1,"last update":"Thu Mar 9 14:44:30 2023","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.53500000000000003,"15min":0.499},"min":{"1min":0.30199999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.3340000000000001,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.54600000000000004,"15min":0.505},"min":{"1min":0.41799999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":1.1890000000000001,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.66500000000000004}]},{"osd":3,"last update":"Thu Mar 9 14:44:51 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.57299999999999995,"15min":0.54100000000000004},"min":{"1min":0.38800000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.54500000000000004,"15min":0.53300000000000003},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.48899999999999999}]},{"osd":4,"last update":"Thu Mar 9 14:44:15 2023","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.54900000000000004,"15min":0.52500000000000002},"min":{"1min":0.41599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.56699999999999995,"15min":0.54500000000000004},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.234,"5min":1.234,"15min":1.234},"last":0.73399999999999999}]},{"osd":5,"last update":"Thu Mar 9 14:44:35 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.47199999999999998,"15min":0.46000000000000002},"min":{"1min":0.313,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.45200000000000001,"15min":0.44},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.39400000000000002}]},{"osd":6,"last update":"Thu Mar 9 14:44:52 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59199999999999997,"15min":0.57899999999999996},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.504},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.63,"15min":0.61599999999999999},"min":{"1min":0.41699999999999998,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.89200000000000002,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.63800000000000001}]},{"osd":7,"last update":"Thu Mar 9 14:44:15 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.2450000000000001,"5min":1.2450000000000001,"15min":1.2450000000000001},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.69699999999999995,"15min":0.69299999999999995},"min":{"1min":0.46100000000000002,"5min":0.375,"15min":0.375},"max":{"1min":1.2649999999999999,"5min":1.2649999999999999,"15min":1.2649999999999999},"last":0.76700000000000002}]},{"osd":8,"last update":"Thu Mar 9 14:44:35 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.65400000000000003,"15min":0.65100000000000002},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.94699999999999995,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.64600000000000002,"15min":0.65200000000000002},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.93999999999999995,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.621}]},{"osd":9,"last update":"Thu Mar 9 14:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.72399999999999998,"15min":0.72999999999999998},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.0109999999999999,"5min":1.21,"15min":1.21},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.67100000000000004,"15min":0.68000000000000005},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.93200000000000005,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.52700000000000002}]},{"osd":10,"last update":"Thu Mar 9 14:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.2529999999999999,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.306,"5min":1.306,"15min":1.306},"last":0.71199999999999997}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71999999999999997}]}]},{"osd":3,"up_from":25,"seq":107374182458,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6752,"kb_used_data":984,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738400,"statfs":{"total":95995035648,"available":95988121600,"internally_reserved":0,"allocated":1007616,"data_stored":486449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:44:48 2023","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.36399999999999999,"15min":0.32800000000000001},"min":{"1min":0.308,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.66500000000000004,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.379,"15min":0.34300000000000003},"min":{"1min":0.32400000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.64000000000000001,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.39300000000000002}]},{"osd":1,"last update":"Thu Mar 9 14:44:48 2023","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.53000000000000003,"15min":0.504},"min":{"1min":0.42799999999999999,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.74299999999999999,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.55100000000000005,"15min":0.496},"min":{"1min":0.40400000000000003,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.59099999999999997}]},{"osd":2,"last update":"Thu Mar 9 14:44:48 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.55800000000000005,"15min":0.51600000000000001},"min":{"1min":0.49099999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.82499999999999996,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.629},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.56799999999999995,"15min":0.53200000000000003},"min":{"1min":0.45300000000000001,"5min":0.312,"15min":0.312},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.57699999999999996}]},{"osd":4,"last update":"Thu Mar 9 14:45:10 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.58099999999999996,"15min":0.54200000000000004},"min":{"1min":0.45200000000000001,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.75900000000000001,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.622,"5min":0.60099999999999998,"15min":0.55000000000000004},"min":{"1min":0.497,"5min":0.373,"15min":0.373},"max":{"1min":0.84599999999999997,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.499}]},{"osd":5,"last update":"Thu Mar 9 14:44:32 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.56699999999999995,"15min":0.54300000000000004},"min":{"1min":0.433,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.60099999999999998,"15min":0.56499999999999995},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.621}]},{"osd":6,"last update":"Thu Mar 9 14:44:51 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.56000000000000005,"15min":0.53500000000000003},"min":{"1min":0.44,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.95799999999999996,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.57099999999999995,"15min":0.54300000000000004},"min":{"1min":0.438,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.97499999999999998,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.34499999999999997}]},{"osd":7,"last update":"Thu Mar 9 14:45:17 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.69399999999999995,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.68000000000000005,"15min":0.66700000000000004},"min":{"1min":0.46500000000000002,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.80400000000000005,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.54400000000000004}]},{"osd":8,"last update":"Thu Mar 9 14:43:35 2023","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.749}]},{"osd":9,"last update":"Thu Mar 9 14:44:54 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.76800000000000002,"15min":0.78800000000000003},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.1160000000000001,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.63,"5min":0.77600000000000002,"15min":0.80000000000000004},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.92300000000000004,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.42799999999999999}]},{"osd":10,"last update":"Thu Mar 9 14:44:20 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.155,"5min":1.155,"15min":1.155},"last":0.65700000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68200000000000005}]}]},{"osd":4,"up_from":32,"seq":137438953527,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6636,"kb_used_data":996,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93738516,"statfs":{"total":95995035648,"available":95988240384,"internally_reserved":0,"allocated":1019904,"data_stored":492483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[0,1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:44:09 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.53100000000000003,"15min":0.495},"min":{"1min":0.39000000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.84099999999999997,"5min":1.8320000000000001,"15min":1.8320000000000001},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.628,"5min":0.52900000000000003,"15min":0.495},"min":{"1min":0.42499999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":1.044,"5min":1.478,"15min":1.478},"last":0.56599999999999995}]},{"osd":1,"last update":"Thu Mar 9 14:45:14 2023","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.376,"15min":0.34799999999999998},"min":{"1min":0.311,"5min":0.253,"15min":0.253},"max":{"1min":0.53100000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.39600000000000002,"15min":0.36799999999999999},"min":{"1min":0.316,"5min":0.253,"15min":0.253},"max":{"1min":0.625,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.73599999999999999}]},{"osd":2,"last update":"Thu Mar 9 14:45:14 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.54100000000000004,"15min":0.53000000000000003},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.087,"5min":1.087,"15min":1.087},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.57199999999999995,"15min":0.53900000000000003},"min":{"1min":0.41499999999999998,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.70499999999999996}]},{"osd":3,"last update":"Thu Mar 9 14:45:14 2023","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":0.69899999999999995,"15min":0.57099999999999995},"min":{"1min":0.47699999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":9.9420000000000002,"5min":9.9420000000000002,"15min":9.9420000000000002},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.59699999999999998,"15min":0.54100000000000004},"min":{"1min":0.45200000000000001,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":2.6110000000000002,"5min":2.6110000000000002,"15min":2.6110000000000002},"last":0.59199999999999997}]},{"osd":5,"last update":"Thu Mar 9 14:44:33 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.58099999999999996,"15min":0.55900000000000005},"min":{"1min":0.45800000000000002,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.755},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.59399999999999997,"15min":0.56599999999999995},"min":{"1min":0.47099999999999997,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.79200000000000004}]},{"osd":6,"last update":"Thu Mar 9 14:44:53 2023","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.69399999999999995,"15min":0.69699999999999995},"min":{"1min":0.45900000000000002,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":2.4609999999999999,"5min":2.4609999999999999,"15min":2.4609999999999999},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.92800000000000005,"5min":0.72499999999999998,"15min":0.69199999999999995},"min":{"1min":0.52100000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":4.6879999999999997,"5min":4.6879999999999997,"15min":4.6879999999999997},"last":0.82899999999999996}]},{"osd":7,"last update":"Thu Mar 9 14:45:14 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.56999999999999995,"15min":0.53400000000000003},"min":{"1min":0.42299999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.872,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.93200000000000005},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.58199999999999996,"15min":0.54000000000000004},"min":{"1min":0.437,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.77400000000000002,"5min":0.998,"15min":0.998},"last":0.97199999999999998}]},{"osd":8,"last update":"Thu Mar 9 14:44:36 2023","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.70799999999999996,"15min":0.68799999999999994},"min":{"1min":0.53000000000000003,"5min":0.377,"15min":0.377},"max":{"1min":1.7569999999999999,"5min":1.7569999999999999,"15min":1.7569999999999999},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.71499999999999997,"15min":0.69399999999999995},"min":{"1min":0.53800000000000003,"5min":0.498,"15min":0.498},"max":{"1min":1.746,"5min":1.746,"15min":1.746},"last":0.88}]},{"osd":9,"last update":"Thu Mar 9 14:44:58 2023","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.78000000000000003,"15min":0.76100000000000001},"min":{"1min":0.59599999999999997,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":2.5419999999999998,"5min":2.5419999999999998,"15min":2.5419999999999998},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.72799999999999998,"15min":0.71399999999999997},"min":{"1min":0.55200000000000005,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":2.4969999999999999,"5min":2.4969999999999999,"15min":2.4969999999999999},"last":0.85399999999999998}]},{"osd":10,"last update":"Thu Mar 9 14:45:20 2023","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.77100000000000002,"15min":0.77200000000000002},"min":{"1min":0.61099999999999999,"5min":0.432,"15min":0.432},"max":{"1min":1.0009999999999999,"5min":1.069,"15min":1.069},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.79000000000000004,"15min":0.79300000000000004},"min":{"1min":0.627,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.109,"5min":1.228,"15min":1.228},"last":0.91800000000000004}]},{"osd":11,"last update":"Thu Mar 9 14:45:18 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.1779999999999999,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.94599999999999995},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.89500000000000002}]}]},{"osd":5,"up_from":37,"seq":158913790003,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7272,"kb_used_data":1568,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93737880,"statfs":{"total":95995035648,"available":95987589120,"internally_reserved":0,"allocated":1605632,"data_stored":1081598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.51200000000000001,"15min":0.48599999999999999},"min":{"1min":0.35599999999999998,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.80000000000000004,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.51900000000000002,"15min":0.5},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.69899999999999995,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.45400000000000001}]},{"osd":1,"last update":"Thu Mar 9 14:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55500000000000005,"15min":0.54700000000000004},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.83599999999999997,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.54300000000000004,"15min":0.53100000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.56399999999999995}]},{"osd":2,"last update":"Thu Mar 9 14:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.375,"15min":0.35999999999999999},"min":{"1min":0.249,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.60099999999999998,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.38900000000000001,"15min":0.36899999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.41099999999999998}]},{"osd":3,"last update":"Thu Mar 9 14:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.54000000000000004,"15min":0.53500000000000003},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.71399999999999997,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56699999999999995,"15min":0.55100000000000005},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.752,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.39200000000000002}]},{"osd":4,"last update":"Thu Mar 9 14:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.56999999999999995,"15min":0.55600000000000005},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.76200000000000001,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53000000000000003,"15min":0.52600000000000002},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.70399999999999996,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.42599999999999999}]},{"osd":6,"last update":"Thu Mar 9 14:44:52 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.60099999999999998,"15min":0.59999999999999998},"min":{"1min":0.39200000000000002,"5min":0.315,"15min":0.315},"max":{"1min":0.79200000000000004,"5min":0.879,"15min":0.879},"last":0.438},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.64000000000000001,"15min":0.627},"min":{"1min":0.42899999999999999,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.53700000000000003}]},{"osd":7,"last update":"Thu Mar 9 14:45:18 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.61099999999999999,"15min":0.59399999999999997},"min":{"1min":0.42099999999999999,"5min":0.38,"15min":0.38},"max":{"1min":0.84799999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.61899999999999999,"15min":0.59999999999999998},"min":{"1min":0.46899999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.83299999999999996,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.55500000000000005}]},{"osd":8,"last update":"Thu Mar 9 14:44:34 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.59499999999999997,"15min":0.59899999999999998},"min":{"1min":0.35099999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.84099999999999997,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.54200000000000004,"15min":0.54400000000000004},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.46800000000000003}]},{"osd":9,"last update":"Thu Mar 9 14:44:58 2023","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.67800000000000005,"15min":0.64600000000000002},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":5.6749999999999998,"5min":5.6749999999999998,"15min":5.6749999999999998},"last":0.496},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.71599999999999997,"15min":0.71499999999999997},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.468,"5min":1.468,"15min":1.468},"last":0.58999999999999997}]},{"osd":10,"last update":"Thu Mar 9 14:45:18 2023","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.69799999999999995,"15min":0.68999999999999995},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.86499999999999999,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.70899999999999996,"15min":0.69999999999999996},"min":{"1min":0.54300000000000004,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":0.877,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.57199999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":57232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":56812,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":58618,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-03-09T14:45:25.166 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2023-03-09T14:45:25.166 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-03-09T14:45:25.166 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2023-03-09T14:45:25.167 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph health --format=json 2023-03-09T14:45:25.822 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.a/config 2023-03-09T14:45:25.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:25 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:45:25.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:25 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:45:25.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:25 smithi122 ceph-mon[93869]: from='client.15096 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-03-09T14:45:25.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:25 smithi122 ceph-mon[93869]: pgmap v294: 129 pgs: 129 active+clean; 636 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s; 205 B/s, 1 objects/s recovering 2023-03-09T14:45:25.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:25 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:45:25.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:25 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:45:25.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:25 smithi122 ceph-mon[93869]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:45:25.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:25 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:45:25.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:25 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:45:25.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:25 smithi181 ceph-mon[94189]: from='client.15096 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-03-09T14:45:25.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:25 smithi181 ceph-mon[94189]: pgmap v294: 129 pgs: 129 active+clean; 636 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s; 205 B/s, 1 objects/s recovering 2023-03-09T14:45:25.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:25 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:45:25.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:25 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:45:25.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:25 smithi181 ceph-mon[94189]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:45:26.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:25 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:45:26.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:25 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:45:26.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:25 smithi038 ceph-mon[92422]: from='client.15096 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-03-09T14:45:26.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:25 smithi038 ceph-mon[92422]: pgmap v294: 129 pgs: 129 active+clean; 636 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s; 205 B/s, 1 objects/s recovering 2023-03-09T14:45:26.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:25 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:45:26.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:25 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:45:26.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:25 smithi038 ceph-mon[92422]: from='mgr.14150 172.21.15.38:0/3602210893' entity='mgr.x' 2023-03-09T14:45:27.018 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.014+0000 7f2a6ea76700 1 -- 172.21.15.38:0/2295393551 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2a680fed80 msgr2=0x7f2a680ff150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:27.018 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.014+0000 7f2a6ea76700 1 --2- 172.21.15.38:0/2295393551 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2a680fed80 0x7f2a680ff150 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f2a5c009990 tx=0x7f2a5c009ca0 comp rx=0 tx=0).stop 2023-03-09T14:45:27.018 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.014+0000 7f2a6ea76700 1 -- 172.21.15.38:0/2295393551 shutdown_connections 2023-03-09T14:45:27.018 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.014+0000 7f2a6ea76700 1 --2- 172.21.15.38:0/2295393551 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2a680fed80 0x7f2a680ff150 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:27.019 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.014+0000 7f2a6ea76700 1 --2- 172.21.15.38:0/2295393551 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2a6810c930 0x7f2a6810ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:27.019 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.014+0000 7f2a6ea76700 1 --2- 172.21.15.38:0/2295393551 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a680ff690 0x7f2a6810c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:27.019 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.014+0000 7f2a6ea76700 1 -- 172.21.15.38:0/2295393551 >> 172.21.15.38:0/2295393551 conn(0x7f2a68076d10 msgr2=0x7f2a68077110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:27.019 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.014+0000 7f2a6ea76700 1 -- 172.21.15.38:0/2295393551 shutdown_connections 2023-03-09T14:45:27.019 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.014+0000 7f2a6ea76700 1 -- 172.21.15.38:0/2295393551 wait complete. 2023-03-09T14:45:27.019 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.015+0000 7f2a6ea76700 1 Processor -- start 2023-03-09T14:45:27.019 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.015+0000 7f2a6ea76700 1 -- start start 2023-03-09T14:45:27.021 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.015+0000 7f2a6ea76700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2a680fed80 0x7f2a68198380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:27.021 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.015+0000 7f2a6ea76700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a680ff690 0x7f2a681988c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:27.021 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.015+0000 7f2a6ea76700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2a6810c930 0x7f2a6819cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:27.022 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.015+0000 7f2a6ea76700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a6819d360 con 0x7f2a680fed80 2023-03-09T14:45:27.022 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.015+0000 7f2a6ea76700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a6819d4d0 con 0x7f2a680ff690 2023-03-09T14:45:27.022 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.015+0000 7f2a6ea76700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a6819d690 con 0x7f2a6810c930 2023-03-09T14:45:27.022 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.015+0000 7f2a6c812700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2a680fed80 0x7f2a68198380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:27.022 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.015+0000 7f2a6c812700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2a680fed80 0x7f2a68198380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37692/0 (socket says 172.21.15.38:37692) 2023-03-09T14:45:27.022 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.015+0000 7f2a6c812700 1 -- 172.21.15.38:0/2881230380 learned_addr learned my addr 172.21.15.38:0/2881230380 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:27.022 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.015+0000 7f2a6d013700 1 --2- 172.21.15.38:0/2881230380 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2a6810c930 0x7f2a6819cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:27.022 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.015+0000 7f2a67fff700 1 --2- 172.21.15.38:0/2881230380 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a680ff690 0x7f2a681988c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:27.023 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.015+0000 7f2a6c812700 1 -- 172.21.15.38:0/2881230380 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2a6810c930 msgr2=0x7f2a6819cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:27.023 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.015+0000 7f2a6c812700 1 --2- 172.21.15.38:0/2881230380 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2a6810c930 0x7f2a6819cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:27.023 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.015+0000 7f2a6c812700 1 -- 172.21.15.38:0/2881230380 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a680ff690 msgr2=0x7f2a681988c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:27.023 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.015+0000 7f2a6c812700 1 --2- 172.21.15.38:0/2881230380 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a680ff690 0x7f2a681988c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:27.023 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.015+0000 7f2a6c812700 1 -- 172.21.15.38:0/2881230380 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a5c0095f0 con 0x7f2a680fed80 2023-03-09T14:45:27.023 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.015+0000 7f2a67fff700 1 --2- 172.21.15.38:0/2881230380 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a680ff690 0x7f2a681988c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-09T14:45:27.023 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.016+0000 7f2a6c812700 1 --2- 172.21.15.38:0/2881230380 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2a680fed80 0x7f2a68198380 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f2a5c000c00 tx=0x7f2a5c004790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:27.024 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.016+0000 7f2a65ffb700 1 -- 172.21.15.38:0/2881230380 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a5c01d070 con 0x7f2a680fed80 2023-03-09T14:45:27.024 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.016+0000 7f2a6ea76700 1 -- 172.21.15.38:0/2881230380 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a68112220 con 0x7f2a680fed80 2023-03-09T14:45:27.024 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.016+0000 7f2a65ffb700 1 -- 172.21.15.38:0/2881230380 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2a5c021410 con 0x7f2a680fed80 2023-03-09T14:45:27.024 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.016+0000 7f2a65ffb700 1 -- 172.21.15.38:0/2881230380 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a5c017650 con 0x7f2a680fed80 2023-03-09T14:45:27.024 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.016+0000 7f2a6ea76700 1 -- 172.21.15.38:0/2881230380 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a68112090 con 0x7f2a680fed80 2023-03-09T14:45:27.024 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.017+0000 7f2a6ea76700 1 -- 172.21.15.38:0/2881230380 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a681123b0 con 0x7f2a680fed80 2023-03-09T14:45:27.024 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.018+0000 7f2a65ffb700 1 -- 172.21.15.38:0/2881230380 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 133238+0+0 (secure 0 0 0) 0x7f2a5c028030 con 0x7f2a680fed80 2023-03-09T14:45:27.025 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.019+0000 7f2a65ffb700 1 --2- 172.21.15.38:0/2881230380 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f2a5809f400 0x7f2a580a18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:27.025 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.019+0000 7f2a65ffb700 1 -- 172.21.15.38:0/2881230380 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f2a5c0cb2c0 con 0x7f2a680fed80 2023-03-09T14:45:27.025 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.019+0000 7f2a67fff700 1 --2- 172.21.15.38:0/2881230380 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f2a5809f400 0x7f2a580a18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:27.025 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.019+0000 7f2a67fff700 1 --2- 172.21.15.38:0/2881230380 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f2a5809f400 0x7f2a580a18b0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f2a68199990 tx=0x7f2a54009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:27.026 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.023+0000 7f2a65ffb700 1 -- 172.21.15.38:0/2881230380 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f2a5c095470 con 0x7f2a680fed80 2023-03-09T14:45:27.267 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.263+0000 7f2a6ea76700 1 -- 172.21.15.38:0/2881230380 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f2a681996d0 con 0x7f2a680fed80 2023-03-09T14:45:27.267 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.264+0000 7f2a65ffb700 1 -- 172.21.15.38:0/2881230380 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f2a5c026040 con 0x7f2a680fed80 2023-03-09T14:45:27.267 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:45:27.267 INFO:teuthology.orchestra.run.smithi038.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-03-09T14:45:27.269 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.266+0000 7f2a6ea76700 1 -- 172.21.15.38:0/2881230380 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f2a5809f400 msgr2=0x7f2a580a18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:27.269 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.266+0000 7f2a6ea76700 1 --2- 172.21.15.38:0/2881230380 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f2a5809f400 0x7f2a580a18b0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f2a68199990 tx=0x7f2a54009260 comp rx=0 tx=0).stop 2023-03-09T14:45:27.269 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.266+0000 7f2a6ea76700 1 -- 172.21.15.38:0/2881230380 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2a680fed80 msgr2=0x7f2a68198380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:27.269 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.266+0000 7f2a6ea76700 1 --2- 172.21.15.38:0/2881230380 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2a680fed80 0x7f2a68198380 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f2a5c000c00 tx=0x7f2a5c004790 comp rx=0 tx=0).stop 2023-03-09T14:45:27.270 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.266+0000 7f2a6ea76700 1 -- 172.21.15.38:0/2881230380 shutdown_connections 2023-03-09T14:45:27.270 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.266+0000 7f2a6ea76700 1 --2- 172.21.15.38:0/2881230380 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2a680fed80 0x7f2a68198380 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:27.270 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.266+0000 7f2a6ea76700 1 --2- 172.21.15.38:0/2881230380 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f2a5809f400 0x7f2a580a18b0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:27.270 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.266+0000 7f2a6ea76700 1 --2- 172.21.15.38:0/2881230380 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2a6810c930 0x7f2a6819cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:27.270 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.266+0000 7f2a6ea76700 1 --2- 172.21.15.38:0/2881230380 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a680ff690 0x7f2a681988c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:27.270 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.266+0000 7f2a6ea76700 1 -- 172.21.15.38:0/2881230380 >> 172.21.15.38:0/2881230380 conn(0x7f2a68076d10 msgr2=0x7f2a6810dcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:27.270 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.266+0000 7f2a6ea76700 1 -- 172.21.15.38:0/2881230380 shutdown_connections 2023-03-09T14:45:27.270 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.266+0000 7f2a6ea76700 1 -- 172.21.15.38:0/2881230380 wait complete. 2023-03-09T14:45:27.676 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2023-03-09T14:45:27.677 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-09T14:45:27.807 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.803+0000 7fef01f46700 1 -- 172.21.15.38:0/109706617 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feefc0fedf0 msgr2=0x7feefc0ff200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:27.807 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.803+0000 7fef01f46700 1 --2- 172.21.15.38:0/109706617 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feefc0fedf0 0x7feefc0ff200 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7feef0009990 tx=0x7feef0009ca0 comp rx=0 tx=0).stop 2023-03-09T14:45:27.807 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.803+0000 7fef01f46700 1 -- 172.21.15.38:0/109706617 shutdown_connections 2023-03-09T14:45:27.808 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.803+0000 7fef01f46700 1 --2- 172.21.15.38:0/109706617 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feefc0fedf0 0x7feefc0ff200 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:27.808 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.803+0000 7fef01f46700 1 --2- 172.21.15.38:0/109706617 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feefc100970 0x7feefc107230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:27.808 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.803+0000 7fef01f46700 1 --2- 172.21.15.38:0/109706617 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feefc0fffe0 0x7feefc100430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:27.808 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.803+0000 7fef01f46700 1 -- 172.21.15.38:0/109706617 >> 172.21.15.38:0/109706617 conn(0x7feefc0fa410 msgr2=0x7feefc0fc840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:27.808 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.803+0000 7fef01f46700 1 -- 172.21.15.38:0/109706617 shutdown_connections 2023-03-09T14:45:27.808 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.803+0000 7fef01f46700 1 -- 172.21.15.38:0/109706617 wait complete. 2023-03-09T14:45:27.809 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.804+0000 7fef01f46700 1 Processor -- start 2023-03-09T14:45:27.810 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.804+0000 7fef01f46700 1 -- start start 2023-03-09T14:45:27.811 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.804+0000 7fef01f46700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feefc0fedf0 0x7feefc195c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:27.811 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.804+0000 7fef01f46700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feefc0fffe0 0x7feefc1961b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:27.811 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.804+0000 7fef01f46700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feefc100970 0x7feefc19d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:27.811 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.804+0000 7fef01f46700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feefc19d7e0 con 0x7feefc0fedf0 2023-03-09T14:45:27.811 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.804+0000 7fef01f46700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feefc19d950 con 0x7feefc0fffe0 2023-03-09T14:45:27.811 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.804+0000 7fef01f46700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feefc19db10 con 0x7feefc100970 2023-03-09T14:45:27.812 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.804+0000 7feefb7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feefc0fedf0 0x7feefc195c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:27.812 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.804+0000 7feefb7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feefc0fedf0 0x7feefc195c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37706/0 (socket says 172.21.15.38:37706) 2023-03-09T14:45:27.812 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.804+0000 7feefb7fe700 1 -- 172.21.15.38:0/727891374 learned_addr learned my addr 172.21.15.38:0/727891374 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:27.812 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.804+0000 7feefbfff700 1 --2- 172.21.15.38:0/727891374 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feefc100970 0x7feefc19d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:27.812 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.804+0000 7feefaffd700 1 --2- 172.21.15.38:0/727891374 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feefc0fffe0 0x7feefc1961b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:27.812 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.805+0000 7feefb7fe700 1 -- 172.21.15.38:0/727891374 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feefc100970 msgr2=0x7feefc19d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:27.812 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.805+0000 7feefb7fe700 1 --2- 172.21.15.38:0/727891374 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feefc100970 0x7feefc19d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:27.813 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.805+0000 7feefb7fe700 1 -- 172.21.15.38:0/727891374 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feefc0fffe0 msgr2=0x7feefc1961b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:27.813 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.805+0000 7feefb7fe700 1 --2- 172.21.15.38:0/727891374 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feefc0fffe0 0x7feefc1961b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:27.813 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.805+0000 7feefb7fe700 1 -- 172.21.15.38:0/727891374 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feef00095f0 con 0x7feefc0fedf0 2023-03-09T14:45:27.813 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.805+0000 7feefb7fe700 1 --2- 172.21.15.38:0/727891374 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feefc0fedf0 0x7feefc195c70 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7feef0004760 tx=0x7feef0004790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:27.813 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.805+0000 7feef8ff9700 1 -- 172.21.15.38:0/727891374 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feef001d070 con 0x7feefc0fedf0 2023-03-09T14:45:27.813 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.805+0000 7feef8ff9700 1 -- 172.21.15.38:0/727891374 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feef0004cf0 con 0x7feefc0fedf0 2023-03-09T14:45:27.814 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.805+0000 7fef01f46700 1 -- 172.21.15.38:0/727891374 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feefc10a820 con 0x7feefc0fedf0 2023-03-09T14:45:27.814 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.805+0000 7feef8ff9700 1 -- 172.21.15.38:0/727891374 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feef0005280 con 0x7feefc0fedf0 2023-03-09T14:45:27.814 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.805+0000 7fef01f46700 1 -- 172.21.15.38:0/727891374 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feefc10a690 con 0x7feefc0fedf0 2023-03-09T14:45:27.814 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.806+0000 7fef01f46700 1 -- 172.21.15.38:0/727891374 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feefc19e020 con 0x7feefc0fedf0 2023-03-09T14:45:27.814 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.807+0000 7feef8ff9700 1 -- 172.21.15.38:0/727891374 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 133238+0+0 (secure 0 0 0) 0x7feef000c040 con 0x7feefc0fedf0 2023-03-09T14:45:27.815 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.808+0000 7feef8ff9700 1 --2- 172.21.15.38:0/727891374 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7feee409f400 0x7feee40a18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:27.816 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.808+0000 7feef8ff9700 1 -- 172.21.15.38:0/727891374 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7feef00cb3e0 con 0x7feefc0fedf0 2023-03-09T14:45:27.816 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.808+0000 7feefaffd700 1 --2- 172.21.15.38:0/727891374 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7feee409f400 0x7feee40a18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:27.816 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.808+0000 7feefaffd700 1 --2- 172.21.15.38:0/727891374 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7feee409f400 0x7feee40a18b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7feefc197170 tx=0x7feeec009190 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:27.817 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:27.812+0000 7feef8ff9700 1 -- 172.21.15.38:0/727891374 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7feef005b5b0 con 0x7feefc0fedf0 2023-03-09T14:45:28.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:27 smithi038 ceph-mon[92422]: pgmap v295: 129 pgs: 129 active+clean; 636 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s; 186 B/s, 1 objects/s recovering 2023-03-09T14:45:28.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:27 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/2881230380' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-03-09T14:45:28.016 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.013+0000 7fef01f46700 1 -- 172.21.15.38:0/727891374 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feefc19e5d0 con 0x7feefc0fedf0 2023-03-09T14:45:28.017 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.014+0000 7feef8ff9700 1 -- 172.21.15.38:0/727891374 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24445 (secure 0 0 0) 0x7feef009e030 con 0x7feefc0fedf0 2023-03-09T14:45:28.018 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:45:28.018 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":89,"fsid":"c6733fe2-be87-11ed-9af2-001a4aab830c","created":"2023-03-09T14:37:14.947981+0000","modified":"2023-03-09T14:45:15.848936+0000","last_up_change":"2023-03-09T14:43:37.198407+0000","last_in_change":"2023-03-09T14:43:19.550566+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-09T14:40:23.196665+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-03-09T14:44:17.484374+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-03-09T14:44:18.457833+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-03-09T14:44:21.895589+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"51477039-15ee-4707-a34d-4ecfa18bebde","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6803","nonce":1508072290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6805","nonce":1508072290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6809","nonce":1508072290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6807","nonce":1508072290}]},"public_addr":"172.21.15.38:6803/1508072290","cluster_addr":"172.21.15.38:6805/1508072290","heartbeat_back_addr":"172.21.15.38:6809/1508072290","heartbeat_front_addr":"172.21.15.38:6807/1508072290","state":["exists","up"]},{"osd":1,"uuid":"71f8a718-b0a9-4f39-bc54-267aa42a2578","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6801","nonce":3929268773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6803","nonce":3929268773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6807","nonce":3929268773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6805","nonce":3929268773}]},"public_addr":"172.21.15.122:6801/3929268773","cluster_addr":"172.21.15.122:6803/3929268773","heartbeat_back_addr":"172.21.15.122:6807/3929268773","heartbeat_front_addr":"172.21.15.122:6805/3929268773","state":["exists","up"]},{"osd":2,"uuid":"961a2cbe-140e-437f-b131-1fcac573b19c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6801","nonce":1310630803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6803","nonce":1310630803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6807","nonce":1310630803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6805","nonce":1310630803}]},"public_addr":"172.21.15.181:6801/1310630803","cluster_addr":"172.21.15.181:6803/1310630803","heartbeat_back_addr":"172.21.15.181:6807/1310630803","heartbeat_front_addr":"172.21.15.181:6805/1310630803","state":["exists","up"]},{"osd":3,"uuid":"969df8fa-2d19-42c7-9f2a-5c5fac15d4a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6811","nonce":2719814100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6813","nonce":2719814100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6817","nonce":2719814100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6815","nonce":2719814100}]},"public_addr":"172.21.15.38:6811/2719814100","cluster_addr":"172.21.15.38:6813/2719814100","heartbeat_back_addr":"172.21.15.38:6817/2719814100","heartbeat_front_addr":"172.21.15.38:6815/2719814100","state":["exists","up"]},{"osd":4,"uuid":"44a6dce3-e06e-4cbd-9404-62a86d01ab6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6809","nonce":2480764994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6811","nonce":2480764994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6815","nonce":2480764994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6813","nonce":2480764994}]},"public_addr":"172.21.15.122:6809/2480764994","cluster_addr":"172.21.15.122:6811/2480764994","heartbeat_back_addr":"172.21.15.122:6815/2480764994","heartbeat_front_addr":"172.21.15.122:6813/2480764994","state":["exists","up"]},{"osd":5,"uuid":"49749574-c00c-4e3a-80e6-e3686df58d5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6809","nonce":3141346246}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6811","nonce":3141346246}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6815","nonce":3141346246}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6813","nonce":3141346246}]},"public_addr":"172.21.15.181:6809/3141346246","cluster_addr":"172.21.15.181:6811/3141346246","heartbeat_back_addr":"172.21.15.181:6815/3141346246","heartbeat_front_addr":"172.21.15.181:6813/3141346246","state":["exists","up"]},{"osd":6,"uuid":"61c0437e-3f51-45a9-8a61-e98eea0093f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6819","nonce":2122788957}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6821","nonce":2122788957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6825","nonce":2122788957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6823","nonce":2122788957}]},"public_addr":"172.21.15.38:6819/2122788957","cluster_addr":"172.21.15.38:6821/2122788957","heartbeat_back_addr":"172.21.15.38:6825/2122788957","heartbeat_front_addr":"172.21.15.38:6823/2122788957","state":["exists","up"]},{"osd":7,"uuid":"231fecdd-939f-4c7b-83a1-c4a1cffd7139","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6817","nonce":656874139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6819","nonce":656874139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6823","nonce":656874139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6821","nonce":656874139}]},"public_addr":"172.21.15.122:6817/656874139","cluster_addr":"172.21.15.122:6819/656874139","heartbeat_back_addr":"172.21.15.122:6823/656874139","heartbeat_front_addr":"172.21.15.122:6821/656874139","state":["exists","up"]},{"osd":8,"uuid":"297e6d6d-85fd-43ce-9028-5e9dbe55c36e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6817","nonce":755000703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6819","nonce":755000703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6823","nonce":755000703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6821","nonce":755000703}]},"public_addr":"172.21.15.181:6817/755000703","cluster_addr":"172.21.15.181:6819/755000703","heartbeat_back_addr":"172.21.15.181:6823/755000703","heartbeat_front_addr":"172.21.15.181:6821/755000703","state":["exists","up"]},{"osd":9,"uuid":"94cc0355-5a1a-4cfc-bd0c-562b6b41127e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6827","nonce":2624555969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6829","nonce":2624555969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6833","nonce":2624555969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6831","nonce":2624555969}]},"public_addr":"172.21.15.38:6827/2624555969","cluster_addr":"172.21.15.38:6829/2624555969","heartbeat_back_addr":"172.21.15.38:6833/2624555969","heartbeat_front_addr":"172.21.15.38:6831/2624555969","state":["exists","up"]},{"osd":10,"uuid":"f4f2e042-8d92-46bf-93b8-8b0f04795c2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6825","nonce":1427785280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6827","nonce":1427785280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6831","nonce":1427785280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6829","nonce":1427785280}]},"public_addr":"172.21.15.122:6825/1427785280","cluster_addr":"172.21.15.122:6827/1427785280","heartbeat_back_addr":"172.21.15.122:6831/1427785280","heartbeat_front_addr":"172.21.15.122:6829/1427785280","state":["exists","up"]},{"osd":11,"uuid":"07444936-eae5-4097-a799-60294cafe80b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6825","nonce":3183488640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6827","nonce":3183488640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6831","nonce":3183488640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6829","nonce":3183488640}]},"public_addr":"172.21.15.181:6825/3183488640","cluster_addr":"172.21.15.181:6827/3183488640","heartbeat_back_addr":"172.21.15.181:6831/3183488640","heartbeat_front_addr":"172.21.15.181:6829/3183488640","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:39:37.657556+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:39:58.960074+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:40:20.326952+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:40:42.244387+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:41:03.946973+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:41:25.097985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:41:46.877916+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:42:08.413827+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:42:29.822009+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:42:51.776684+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:43:13.937249+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:43:35.569199+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.2","mappings":[{"from":7,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.d","mappings":[{"from":5,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.17","mappings":[{"from":5,"to":6}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/4272288735":"2023-03-10T14:38:14.612767+0000","172.21.15.38:6801/3381625669":"2023-03-10T14:38:14.612767+0000","172.21.15.38:6800/3381625669":"2023-03-10T14:38:14.612767+0000","172.21.15.38:6800/3060474342":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/4253918615":"2023-03-10T14:38:14.612767+0000","172.21.15.38:0/3496083181":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/245602368":"2023-03-10T14:37:47.517775+0000","172.21.15.38:6801/3060474342":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/3622589182":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/1430302251":"2023-03-10T14:38:14.612767+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-09T14:45:28.020 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.017+0000 7fef01f46700 1 -- 172.21.15.38:0/727891374 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7feee409f400 msgr2=0x7feee40a18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:28.020 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.017+0000 7fef01f46700 1 --2- 172.21.15.38:0/727891374 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7feee409f400 0x7feee40a18b0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7feefc197170 tx=0x7feeec009190 comp rx=0 tx=0).stop 2023-03-09T14:45:28.020 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.017+0000 7fef01f46700 1 -- 172.21.15.38:0/727891374 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feefc0fedf0 msgr2=0x7feefc195c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:28.020 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.017+0000 7fef01f46700 1 --2- 172.21.15.38:0/727891374 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feefc0fedf0 0x7feefc195c70 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7feef0004760 tx=0x7feef0004790 comp rx=0 tx=0).stop 2023-03-09T14:45:28.020 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.017+0000 7fef01f46700 1 -- 172.21.15.38:0/727891374 shutdown_connections 2023-03-09T14:45:28.021 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.017+0000 7fef01f46700 1 --2- 172.21.15.38:0/727891374 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feefc0fedf0 0x7feefc195c70 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:28.021 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.017+0000 7fef01f46700 1 --2- 172.21.15.38:0/727891374 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7feee409f400 0x7feee40a18b0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:28.021 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.017+0000 7fef01f46700 1 --2- 172.21.15.38:0/727891374 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feefc100970 0x7feefc19d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:28.021 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.017+0000 7fef01f46700 1 --2- 172.21.15.38:0/727891374 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feefc0fffe0 0x7feefc1961b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:28.021 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.017+0000 7fef01f46700 1 -- 172.21.15.38:0/727891374 >> 172.21.15.38:0/727891374 conn(0x7feefc0fa410 msgr2=0x7feefc068a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:28.021 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.017+0000 7fef01f46700 1 -- 172.21.15.38:0/727891374 shutdown_connections 2023-03-09T14:45:28.021 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.017+0000 7fef01f46700 1 -- 172.21.15.38:0/727891374 wait complete. 2023-03-09T14:45:28.034 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-09T14:40:23.196665+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-03-09T14:44:17.484374+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-03-09T14:44:18.457833+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-03-09T14:44:21.895589+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-03-09T14:45:28.036 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-09T14:45:28.165 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.162+0000 7f8554869700 1 -- 172.21.15.38:0/837742298 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85500ffe80 msgr2=0x7f8550100290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:28.166 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.162+0000 7f8554869700 1 --2- 172.21.15.38:0/837742298 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85500ffe80 0x7f8550100290 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f854000b0a0 tx=0x7f854000b3b0 comp rx=0 tx=0).stop 2023-03-09T14:45:28.166 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.162+0000 7f8554869700 1 -- 172.21.15.38:0/837742298 shutdown_connections 2023-03-09T14:45:28.166 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.162+0000 7f8554869700 1 --2- 172.21.15.38:0/837742298 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85500ffe80 0x7f8550100290 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:28.166 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.162+0000 7f8554869700 1 --2- 172.21.15.38:0/837742298 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8550101a10 0x7f8550108340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:28.166 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.162+0000 7f8554869700 1 --2- 172.21.15.38:0/837742298 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8550101060 0x7f85501014d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:28.166 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.162+0000 7f8554869700 1 -- 172.21.15.38:0/837742298 >> 172.21.15.38:0/837742298 conn(0x7f85500fb510 msgr2=0x7f85500fd920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:28.166 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.162+0000 7f8554869700 1 -- 172.21.15.38:0/837742298 shutdown_connections 2023-03-09T14:45:28.167 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.162+0000 7f8554869700 1 -- 172.21.15.38:0/837742298 wait complete. 2023-03-09T14:45:28.168 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.163+0000 7f8554869700 1 Processor -- start 2023-03-09T14:45:28.169 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.163+0000 7f8554869700 1 -- start start 2023-03-09T14:45:28.169 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.163+0000 7f8554869700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85500ffe80 0x7f8550195ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:28.169 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.163+0000 7f8554869700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8550101060 0x7f8550196420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:28.169 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.163+0000 7f8554869700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8550101a10 0x7f855019d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:28.169 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.163+0000 7f8554869700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f855019da60 con 0x7f85500ffe80 2023-03-09T14:45:28.169 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.163+0000 7f8554869700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f855019dba0 con 0x7f8550101060 2023-03-09T14:45:28.169 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.163+0000 7f8554869700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f855019dd30 con 0x7f8550101a10 2023-03-09T14:45:28.170 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.163+0000 7f854e59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85500ffe80 0x7f8550195ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:28.170 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.163+0000 7f854e59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85500ffe80 0x7f8550195ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37720/0 (socket says 172.21.15.38:37720) 2023-03-09T14:45:28.170 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.163+0000 7f854e59c700 1 -- 172.21.15.38:0/1042575184 learned_addr learned my addr 172.21.15.38:0/1042575184 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:28.170 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.163+0000 7f854dd9b700 1 --2- 172.21.15.38:0/1042575184 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8550101060 0x7f8550196420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:28.170 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.163+0000 7f854ed9d700 1 --2- 172.21.15.38:0/1042575184 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8550101a10 0x7f855019d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:28.170 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.163+0000 7f854e59c700 1 -- 172.21.15.38:0/1042575184 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8550101a10 msgr2=0x7f855019d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:28.170 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.163+0000 7f854e59c700 1 --2- 172.21.15.38:0/1042575184 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8550101a10 0x7f855019d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:28.171 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.163+0000 7f854e59c700 1 -- 172.21.15.38:0/1042575184 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8550101060 msgr2=0x7f8550196420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:28.171 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.163+0000 7f854e59c700 1 --2- 172.21.15.38:0/1042575184 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8550101060 0x7f8550196420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:28.171 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.163+0000 7f854e59c700 1 -- 172.21.15.38:0/1042575184 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8540009d00 con 0x7f85500ffe80 2023-03-09T14:45:28.171 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.164+0000 7f854e59c700 1 --2- 172.21.15.38:0/1042575184 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85500ffe80 0x7f8550195ee0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f85400062a0 tx=0x7f8540004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:28.171 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.164+0000 7f85477fe700 1 -- 172.21.15.38:0/1042575184 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f854000e040 con 0x7f85500ffe80 2023-03-09T14:45:28.171 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.164+0000 7f85477fe700 1 -- 172.21.15.38:0/1042575184 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f854000cec0 con 0x7f85500ffe80 2023-03-09T14:45:28.171 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.164+0000 7f85477fe700 1 -- 172.21.15.38:0/1042575184 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f854001b9f0 con 0x7f85500ffe80 2023-03-09T14:45:28.171 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.164+0000 7f8554869700 1 -- 172.21.15.38:0/1042575184 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f855010b930 con 0x7f85500ffe80 2023-03-09T14:45:28.172 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.164+0000 7f8554869700 1 -- 172.21.15.38:0/1042575184 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f855010b7a0 con 0x7f85500ffe80 2023-03-09T14:45:28.172 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.165+0000 7f8554869700 1 -- 172.21.15.38:0/1042575184 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f855019e270 con 0x7f85500ffe80 2023-03-09T14:45:28.172 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.166+0000 7f85477fe700 1 -- 172.21.15.38:0/1042575184 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 133238+0+0 (secure 0 0 0) 0x7f854001bb70 con 0x7f85500ffe80 2023-03-09T14:45:28.173 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.166+0000 7f85477fe700 1 --2- 172.21.15.38:0/1042575184 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f853c09f400 0x7f853c0a18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:28.173 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.167+0000 7f85477fe700 1 -- 172.21.15.38:0/1042575184 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f85400cc630 con 0x7f85500ffe80 2023-03-09T14:45:28.173 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.167+0000 7f854dd9b700 1 --2- 172.21.15.38:0/1042575184 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f853c09f400 0x7f853c0a18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:28.173 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.167+0000 7f854dd9b700 1 --2- 172.21.15.38:0/1042575184 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f853c09f400 0x7f853c0a18b0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f85500ffba0 tx=0x7f8538009190 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:28.173 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.170+0000 7f85477fe700 1 -- 172.21.15.38:0/1042575184 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f8540096730 con 0x7f85500ffe80 2023-03-09T14:45:28.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:27 smithi122 ceph-mon[93869]: pgmap v295: 129 pgs: 129 active+clean; 636 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s; 186 B/s, 1 objects/s recovering 2023-03-09T14:45:28.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:27 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/2881230380' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-03-09T14:45:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:27 smithi181 ceph-mon[94189]: pgmap v295: 129 pgs: 129 active+clean; 636 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s; 186 B/s, 1 objects/s recovering 2023-03-09T14:45:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:27 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/2881230380' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-03-09T14:45:28.370 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.367+0000 7f8554869700 1 -- 172.21.15.38:0/1042575184 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8550106e50 con 0x7f85500ffe80 2023-03-09T14:45:28.371 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.367+0000 7f85477fe700 1 -- 172.21.15.38:0/1042575184 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7f85400210a0 con 0x7f85500ffe80 2023-03-09T14:45:28.371 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 1 2023-03-09T14:45:28.373 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.370+0000 7f8554869700 1 -- 172.21.15.38:0/1042575184 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f853c09f400 msgr2=0x7f853c0a18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:28.373 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.370+0000 7f8554869700 1 --2- 172.21.15.38:0/1042575184 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f853c09f400 0x7f853c0a18b0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f85500ffba0 tx=0x7f8538009190 comp rx=0 tx=0).stop 2023-03-09T14:45:28.373 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.370+0000 7f8554869700 1 -- 172.21.15.38:0/1042575184 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85500ffe80 msgr2=0x7f8550195ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:28.373 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.370+0000 7f8554869700 1 --2- 172.21.15.38:0/1042575184 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85500ffe80 0x7f8550195ee0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f85400062a0 tx=0x7f8540004d00 comp rx=0 tx=0).stop 2023-03-09T14:45:28.374 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.370+0000 7f8554869700 1 -- 172.21.15.38:0/1042575184 shutdown_connections 2023-03-09T14:45:28.374 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.370+0000 7f8554869700 1 --2- 172.21.15.38:0/1042575184 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f85500ffe80 0x7f8550195ee0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:28.374 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.370+0000 7f8554869700 1 --2- 172.21.15.38:0/1042575184 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f853c09f400 0x7f853c0a18b0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:28.374 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.370+0000 7f8554869700 1 --2- 172.21.15.38:0/1042575184 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8550101a10 0x7f855019d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:28.374 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.370+0000 7f8554869700 1 --2- 172.21.15.38:0/1042575184 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8550101060 0x7f8550196420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:28.374 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.370+0000 7f8554869700 1 -- 172.21.15.38:0/1042575184 >> 172.21.15.38:0/1042575184 conn(0x7f85500fb510 msgr2=0x7f85500fd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:28.374 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.370+0000 7f8554869700 1 -- 172.21.15.38:0/1042575184 shutdown_connections 2023-03-09T14:45:28.375 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.370+0000 7f8554869700 1 -- 172.21.15.38:0/1042575184 wait complete. 2023-03-09T14:45:28.386 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-03-09T14:45:28.515 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.512+0000 7f5525713700 1 -- 172.21.15.38:0/2700492753 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55200fdfa0 msgr2=0x7f55200fe3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:28.516 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.512+0000 7f5525713700 1 --2- 172.21.15.38:0/2700492753 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55200fdfa0 0x7f55200fe3f0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f5510009990 tx=0x7f5510009ca0 comp rx=0 tx=0).stop 2023-03-09T14:45:28.516 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.512+0000 7f5525713700 1 -- 172.21.15.38:0/2700492753 shutdown_connections 2023-03-09T14:45:28.516 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.512+0000 7f5525713700 1 --2- 172.21.15.38:0/2700492753 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f55200fe930 0x7f5520108510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:28.516 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.512+0000 7f5525713700 1 --2- 172.21.15.38:0/2700492753 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5520101530 0x7f5520101940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:28.516 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.512+0000 7f5525713700 1 --2- 172.21.15.38:0/2700492753 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f55200fdfa0 0x7f55200fe3f0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:28.516 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.512+0000 7f5525713700 1 -- 172.21.15.38:0/2700492753 >> 172.21.15.38:0/2700492753 conn(0x7f55200f9bd0 msgr2=0x7f55200fc020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:28.519 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.512+0000 7f5525713700 1 -- 172.21.15.38:0/2700492753 shutdown_connections 2023-03-09T14:45:28.519 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.512+0000 7f5525713700 1 -- 172.21.15.38:0/2700492753 wait complete. 2023-03-09T14:45:28.519 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.512+0000 7f5525713700 1 Processor -- start 2023-03-09T14:45:28.519 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.513+0000 7f5525713700 1 -- start start 2023-03-09T14:45:28.519 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.513+0000 7f5525713700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f55200fdfa0 0x7f5520195d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:28.519 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.513+0000 7f5525713700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f55200fe930 0x7f5520196240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:28.520 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.513+0000 7f5525713700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5520101530 0x7f552019d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:28.520 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.513+0000 7f5525713700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f552019d840 con 0x7f5520101530 2023-03-09T14:45:28.520 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.513+0000 7f5525713700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f552019d9b0 con 0x7f55200fe930 2023-03-09T14:45:28.520 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.513+0000 7f5525713700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f552019db70 con 0x7f55200fdfa0 2023-03-09T14:45:28.520 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.513+0000 7f551f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5520101530 0x7f552019d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:28.520 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.513+0000 7f551f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5520101530 0x7f552019d270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37738/0 (socket says 172.21.15.38:37738) 2023-03-09T14:45:28.520 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.513+0000 7f551f7fe700 1 -- 172.21.15.38:0/964437746 learned_addr learned my addr 172.21.15.38:0/964437746 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:28.521 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.513+0000 7f551effd700 1 --2- 172.21.15.38:0/964437746 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f55200fdfa0 0x7f5520195d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:28.521 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.513+0000 7f551e7fc700 1 --2- 172.21.15.38:0/964437746 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f55200fe930 0x7f5520196240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:28.521 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.513+0000 7f551f7fe700 1 -- 172.21.15.38:0/964437746 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f55200fdfa0 msgr2=0x7f5520195d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:28.521 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.513+0000 7f551f7fe700 1 --2- 172.21.15.38:0/964437746 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f55200fdfa0 0x7f5520195d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:28.521 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.513+0000 7f551f7fe700 1 -- 172.21.15.38:0/964437746 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f55200fe930 msgr2=0x7f5520196240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:28.521 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.513+0000 7f551f7fe700 1 --2- 172.21.15.38:0/964437746 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f55200fe930 0x7f5520196240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:28.521 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.513+0000 7f551f7fe700 1 -- 172.21.15.38:0/964437746 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55100095f0 con 0x7f5520101530 2023-03-09T14:45:28.522 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.513+0000 7f551effd700 1 --2- 172.21.15.38:0/964437746 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f55200fdfa0 0x7f5520195d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:45:28.522 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.513+0000 7f551f7fe700 1 --2- 172.21.15.38:0/964437746 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5520101530 0x7f552019d270 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f551800ef30 tx=0x7f551800c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:28.522 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.514+0000 7f5507fff700 1 -- 172.21.15.38:0/964437746 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5518018070 con 0x7f5520101530 2023-03-09T14:45:28.522 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.514+0000 7f5507fff700 1 -- 172.21.15.38:0/964437746 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f551800f040 con 0x7f5520101530 2023-03-09T14:45:28.522 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.514+0000 7f5507fff700 1 -- 172.21.15.38:0/964437746 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5518003dc0 con 0x7f5520101530 2023-03-09T14:45:28.522 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.514+0000 7f5525713700 1 -- 172.21.15.38:0/964437746 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f552010b8f0 con 0x7f5520101530 2023-03-09T14:45:28.522 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.514+0000 7f5525713700 1 -- 172.21.15.38:0/964437746 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f552010b760 con 0x7f5520101530 2023-03-09T14:45:28.522 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.515+0000 7f5525713700 1 -- 172.21.15.38:0/964437746 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5520107660 con 0x7f5520101530 2023-03-09T14:45:28.523 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.516+0000 7f5507fff700 1 -- 172.21.15.38:0/964437746 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 133238+0+0 (secure 0 0 0) 0x7f551800caa0 con 0x7f5520101530 2023-03-09T14:45:28.523 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.516+0000 7f5507fff700 1 --2- 172.21.15.38:0/964437746 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f550809f400 0x7f55080a18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:28.523 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.516+0000 7f551effd700 1 --2- 172.21.15.38:0/964437746 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f550809f400 0x7f55080a18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:28.523 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.516+0000 7f5507fff700 1 -- 172.21.15.38:0/964437746 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f55180c9f60 con 0x7f5520101530 2023-03-09T14:45:28.524 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.517+0000 7f551effd700 1 --2- 172.21.15.38:0/964437746 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f550809f400 0x7f55080a18b0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f55201971d0 tx=0x7f5514009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:28.524 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.521+0000 7f5507fff700 1 -- 172.21.15.38:0/964437746 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f5518094060 con 0x7f5520101530 2023-03-09T14:45:28.723 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.720+0000 7f5525713700 1 -- 172.21.15.38:0/964437746 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5520061610 con 0x7f5520101530 2023-03-09T14:45:28.724 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.720+0000 7f5507fff700 1 -- 172.21.15.38:0/964437746 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5518097680 con 0x7f5520101530 2023-03-09T14:45:28.724 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 32 2023-03-09T14:45:28.725 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.722+0000 7f5525713700 1 -- 172.21.15.38:0/964437746 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f550809f400 msgr2=0x7f55080a18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:28.726 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.722+0000 7f5525713700 1 --2- 172.21.15.38:0/964437746 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f550809f400 0x7f55080a18b0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f55201971d0 tx=0x7f5514009260 comp rx=0 tx=0).stop 2023-03-09T14:45:28.726 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.722+0000 7f5525713700 1 -- 172.21.15.38:0/964437746 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5520101530 msgr2=0x7f552019d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:28.726 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.722+0000 7f5525713700 1 --2- 172.21.15.38:0/964437746 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5520101530 0x7f552019d270 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f551800ef30 tx=0x7f551800c450 comp rx=0 tx=0).stop 2023-03-09T14:45:28.726 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.722+0000 7f5525713700 1 -- 172.21.15.38:0/964437746 shutdown_connections 2023-03-09T14:45:28.726 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.722+0000 7f5525713700 1 --2- 172.21.15.38:0/964437746 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5520101530 0x7f552019d270 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:28.726 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.722+0000 7f5525713700 1 --2- 172.21.15.38:0/964437746 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f550809f400 0x7f55080a18b0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:28.727 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.722+0000 7f5525713700 1 --2- 172.21.15.38:0/964437746 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f55200fdfa0 0x7f5520195d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:28.727 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.722+0000 7f5525713700 1 --2- 172.21.15.38:0/964437746 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f55200fe930 0x7f5520196240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:28.727 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.722+0000 7f5525713700 1 -- 172.21.15.38:0/964437746 >> 172.21.15.38:0/964437746 conn(0x7f55200f9bd0 msgr2=0x7f55201047a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:28.727 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.722+0000 7f5525713700 1 -- 172.21.15.38:0/964437746 shutdown_connections 2023-03-09T14:45:28.727 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.723+0000 7f5525713700 1 -- 172.21.15.38:0/964437746 wait complete. 2023-03-09T14:45:28.738 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-03-09T14:45:28.872 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.866+0000 7fd524461700 1 -- 172.21.15.38:0/3712981902 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd51c0fff80 msgr2=0x7fd51c1003d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:28.872 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.866+0000 7fd524461700 1 --2- 172.21.15.38:0/3712981902 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd51c0fff80 0x7fd51c1003d0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fd50c009990 tx=0x7fd50c009ca0 comp rx=0 tx=0).stop 2023-03-09T14:45:28.872 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.867+0000 7fd524461700 1 -- 172.21.15.38:0/3712981902 shutdown_connections 2023-03-09T14:45:28.872 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.867+0000 7fd524461700 1 --2- 172.21.15.38:0/3712981902 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd51c100910 0x7fd51c1071d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:28.873 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.867+0000 7fd524461700 1 --2- 172.21.15.38:0/3712981902 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd51c0fed90 0x7fd51c0ff1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:28.873 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.867+0000 7fd524461700 1 --2- 172.21.15.38:0/3712981902 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd51c0fff80 0x7fd51c1003d0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:28.873 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.867+0000 7fd524461700 1 -- 172.21.15.38:0/3712981902 >> 172.21.15.38:0/3712981902 conn(0x7fd51c0fa3b0 msgr2=0x7fd51c0fc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:28.873 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.867+0000 7fd524461700 1 -- 172.21.15.38:0/3712981902 shutdown_connections 2023-03-09T14:45:28.873 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.867+0000 7fd524461700 1 -- 172.21.15.38:0/3712981902 wait complete. 2023-03-09T14:45:28.873 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.867+0000 7fd524461700 1 Processor -- start 2023-03-09T14:45:28.873 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.867+0000 7fd524461700 1 -- start start 2023-03-09T14:45:28.874 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.867+0000 7fd524461700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd51c0fed90 0x7fd51c194b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:28.874 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.867+0000 7fd524461700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd51c0fff80 0x7fd51c195080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:28.874 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.867+0000 7fd524461700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd51c100910 0x7fd51c19c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:28.874 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.867+0000 7fd524461700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd51c19c6c0 con 0x7fd51c0fff80 2023-03-09T14:45:28.874 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.867+0000 7fd524461700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd51c19c800 con 0x7fd51c100910 2023-03-09T14:45:28.875 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.867+0000 7fd524461700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd51c19c990 con 0x7fd51c0fed90 2023-03-09T14:45:28.875 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.868+0000 7fd5219fc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd51c0fff80 0x7fd51c195080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:28.875 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.868+0000 7fd5219fc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd51c0fff80 0x7fd51c195080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37750/0 (socket says 172.21.15.38:37750) 2023-03-09T14:45:28.875 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.868+0000 7fd5219fc700 1 -- 172.21.15.38:0/4062222853 learned_addr learned my addr 172.21.15.38:0/4062222853 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:28.875 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.868+0000 7fd5229fe700 1 --2- 172.21.15.38:0/4062222853 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd51c100910 0x7fd51c19c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:28.876 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.868+0000 7fd5221fd700 1 --2- 172.21.15.38:0/4062222853 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd51c0fed90 0x7fd51c194b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:28.876 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.868+0000 7fd5219fc700 1 -- 172.21.15.38:0/4062222853 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd51c0fed90 msgr2=0x7fd51c194b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:28.876 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.868+0000 7fd5219fc700 1 --2- 172.21.15.38:0/4062222853 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd51c0fed90 0x7fd51c194b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:28.876 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.868+0000 7fd5219fc700 1 -- 172.21.15.38:0/4062222853 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd51c100910 msgr2=0x7fd51c19c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:28.876 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.868+0000 7fd5219fc700 1 --2- 172.21.15.38:0/4062222853 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd51c100910 0x7fd51c19c0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:28.877 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.868+0000 7fd5219fc700 1 -- 172.21.15.38:0/4062222853 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd50c0095f0 con 0x7fd51c0fff80 2023-03-09T14:45:28.877 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.868+0000 7fd5219fc700 1 --2- 172.21.15.38:0/4062222853 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd51c0fff80 0x7fd51c195080 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fd50c00c010 tx=0x7fd50c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:28.878 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.868+0000 7fd5137fe700 1 -- 172.21.15.38:0/4062222853 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd50c01d070 con 0x7fd51c0fff80 2023-03-09T14:45:28.878 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.868+0000 7fd5137fe700 1 -- 172.21.15.38:0/4062222853 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd50c0054f0 con 0x7fd51c0fff80 2023-03-09T14:45:28.878 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.868+0000 7fd5137fe700 1 -- 172.21.15.38:0/4062222853 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd50c004ca0 con 0x7fd51c0fff80 2023-03-09T14:45:28.879 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.868+0000 7fd524461700 1 -- 172.21.15.38:0/4062222853 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd51c10a740 con 0x7fd51c0fff80 2023-03-09T14:45:28.879 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.868+0000 7fd524461700 1 -- 172.21.15.38:0/4062222853 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd51c10a5b0 con 0x7fd51c0fff80 2023-03-09T14:45:28.880 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.869+0000 7fd524461700 1 -- 172.21.15.38:0/4062222853 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd51c19cf30 con 0x7fd51c0fff80 2023-03-09T14:45:28.881 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.870+0000 7fd5137fe700 1 -- 172.21.15.38:0/4062222853 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 133238+0+0 (secure 0 0 0) 0x7fd50c00b050 con 0x7fd51c0fff80 2023-03-09T14:45:28.881 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.871+0000 7fd5137fe700 1 --2- 172.21.15.38:0/4062222853 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fd50809f480 0x7fd5080a1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:28.881 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.871+0000 7fd5221fd700 1 --2- 172.21.15.38:0/4062222853 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fd50809f480 0x7fd5080a1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:28.882 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.871+0000 7fd5137fe700 1 -- 172.21.15.38:0/4062222853 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fd50c0cb790 con 0x7fd51c0fff80 2023-03-09T14:45:28.882 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.872+0000 7fd5221fd700 1 --2- 172.21.15.38:0/4062222853 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fd50809f480 0x7fd5080a1930 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fd518005950 tx=0x7fd5180058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:28.882 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:28.875+0000 7fd5137fe700 1 -- 172.21.15.38:0/4062222853 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7fd50c095890 con 0x7fd51c0fff80 2023-03-09T14:45:29.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:28 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/727891374' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-03-09T14:45:29.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:28 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/1042575184' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-03-09T14:45:29.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:28 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/964437746' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-03-09T14:45:29.079 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.076+0000 7fd524461700 1 -- 172.21.15.38:0/4062222853 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd51c105cc0 con 0x7fd51c0fff80 2023-03-09T14:45:29.079 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.076+0000 7fd5137fe700 1 -- 172.21.15.38:0/4062222853 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd50c02f0a0 con 0x7fd51c0fff80 2023-03-09T14:45:29.080 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 64 2023-03-09T14:45:29.082 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.078+0000 7fd524461700 1 -- 172.21.15.38:0/4062222853 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fd50809f480 msgr2=0x7fd5080a1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:29.082 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.078+0000 7fd524461700 1 --2- 172.21.15.38:0/4062222853 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fd50809f480 0x7fd5080a1930 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fd518005950 tx=0x7fd5180058e0 comp rx=0 tx=0).stop 2023-03-09T14:45:29.082 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.078+0000 7fd524461700 1 -- 172.21.15.38:0/4062222853 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd51c0fff80 msgr2=0x7fd51c195080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:29.082 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.078+0000 7fd524461700 1 --2- 172.21.15.38:0/4062222853 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd51c0fff80 0x7fd51c195080 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fd50c00c010 tx=0x7fd50c004730 comp rx=0 tx=0).stop 2023-03-09T14:45:29.082 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.078+0000 7fd524461700 1 -- 172.21.15.38:0/4062222853 shutdown_connections 2023-03-09T14:45:29.082 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.078+0000 7fd524461700 1 --2- 172.21.15.38:0/4062222853 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fd50809f480 0x7fd5080a1930 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:29.082 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.078+0000 7fd524461700 1 --2- 172.21.15.38:0/4062222853 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd51c100910 0x7fd51c19c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:29.083 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.078+0000 7fd524461700 1 --2- 172.21.15.38:0/4062222853 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd51c0fed90 0x7fd51c194b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:29.083 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.078+0000 7fd524461700 1 --2- 172.21.15.38:0/4062222853 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd51c0fff80 0x7fd51c195080 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:29.083 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.078+0000 7fd524461700 1 -- 172.21.15.38:0/4062222853 >> 172.21.15.38:0/4062222853 conn(0x7fd51c0fa3b0 msgr2=0x7fd51c0fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:29.083 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.078+0000 7fd524461700 1 -- 172.21.15.38:0/4062222853 shutdown_connections 2023-03-09T14:45:29.083 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.078+0000 7fd524461700 1 -- 172.21.15.38:0/4062222853 wait complete. 2023-03-09T14:45:29.094 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-03-09T14:45:29.227 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.220+0000 7fd7d43f6700 1 -- 172.21.15.38:0/3979330756 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd7cc0ff850 msgr2=0x7fd7cc1000e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:29.227 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.220+0000 7fd7d43f6700 1 --2- 172.21.15.38:0/3979330756 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd7cc0ff850 0x7fd7cc1000e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fd7c400b0a0 tx=0x7fd7c400b3b0 comp rx=0 tx=0).stop 2023-03-09T14:45:29.227 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.220+0000 7fd7d43f6700 1 -- 172.21.15.38:0/3979330756 shutdown_connections 2023-03-09T14:45:29.228 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.220+0000 7fd7d43f6700 1 --2- 172.21.15.38:0/3979330756 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd7cc0ff850 0x7fd7cc1000e0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:29.228 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.220+0000 7fd7d43f6700 1 --2- 172.21.15.38:0/3979330756 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd7cc0fe140 0x7fd7cc0fe550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:29.228 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.220+0000 7fd7d43f6700 1 --2- 172.21.15.38:0/3979330756 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd7cc0feea0 0x7fd7cc0ff310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:29.228 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.220+0000 7fd7d43f6700 1 -- 172.21.15.38:0/3979330756 >> 172.21.15.38:0/3979330756 conn(0x7fd7cc0f9b60 msgr2=0x7fd7cc0fbf70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:29.228 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.220+0000 7fd7d43f6700 1 -- 172.21.15.38:0/3979330756 shutdown_connections 2023-03-09T14:45:29.228 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.220+0000 7fd7d43f6700 1 -- 172.21.15.38:0/3979330756 wait complete. 2023-03-09T14:45:29.228 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.221+0000 7fd7d43f6700 1 Processor -- start 2023-03-09T14:45:29.229 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.221+0000 7fd7d43f6700 1 -- start start 2023-03-09T14:45:29.229 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.221+0000 7fd7d43f6700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd7cc0fe140 0x7fd7cc197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:29.229 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.221+0000 7fd7d43f6700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd7cc0feea0 0x7fd7cc198440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:29.229 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.221+0000 7fd7d43f6700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd7cc0ff850 0x7fd7cc19f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:29.229 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.221+0000 7fd7d43f6700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7cc19fa80 con 0x7fd7cc0feea0 2023-03-09T14:45:29.229 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.221+0000 7fd7d43f6700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7cc19fbc0 con 0x7fd7cc0ff850 2023-03-09T14:45:29.229 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.221+0000 7fd7d43f6700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7cc19fd50 con 0x7fd7cc0fe140 2023-03-09T14:45:29.230 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.221+0000 7fd7d1991700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd7cc0feea0 0x7fd7cc198440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:29.230 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.221+0000 7fd7d1991700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd7cc0feea0 0x7fd7cc198440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37764/0 (socket says 172.21.15.38:37764) 2023-03-09T14:45:29.230 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.221+0000 7fd7d1991700 1 -- 172.21.15.38:0/3642833665 learned_addr learned my addr 172.21.15.38:0/3642833665 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:29.230 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.221+0000 7fd7d2192700 1 --2- 172.21.15.38:0/3642833665 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd7cc0fe140 0x7fd7cc197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:29.230 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.221+0000 7fd7d2993700 1 --2- 172.21.15.38:0/3642833665 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd7cc0ff850 0x7fd7cc19f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:29.230 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.221+0000 7fd7d1991700 1 -- 172.21.15.38:0/3642833665 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd7cc0fe140 msgr2=0x7fd7cc197f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:29.230 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.221+0000 7fd7d1991700 1 --2- 172.21.15.38:0/3642833665 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd7cc0fe140 0x7fd7cc197f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:29.230 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.221+0000 7fd7d1991700 1 -- 172.21.15.38:0/3642833665 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd7cc0ff850 msgr2=0x7fd7cc19f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:29.231 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.221+0000 7fd7d1991700 1 --2- 172.21.15.38:0/3642833665 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd7cc0ff850 0x7fd7cc19f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:29.231 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.221+0000 7fd7d1991700 1 -- 172.21.15.38:0/3642833665 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7c4009d00 con 0x7fd7cc0feea0 2023-03-09T14:45:29.231 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.221+0000 7fd7d2993700 1 --2- 172.21.15.38:0/3642833665 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd7cc0ff850 0x7fd7cc19f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:45:29.231 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.221+0000 7fd7d1991700 1 --2- 172.21.15.38:0/3642833665 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd7cc0feea0 0x7fd7cc198440 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fd7bc00d950 tx=0x7fd7bc00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:29.231 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.221+0000 7fd7c37fe700 1 -- 172.21.15.38:0/3642833665 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7bc009940 con 0x7fd7cc0feea0 2023-03-09T14:45:29.231 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.222+0000 7fd7c37fe700 1 -- 172.21.15.38:0/3642833665 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd7bc010460 con 0x7fd7cc0feea0 2023-03-09T14:45:29.231 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.222+0000 7fd7d43f6700 1 -- 172.21.15.38:0/3642833665 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7cc10da40 con 0x7fd7cc0feea0 2023-03-09T14:45:29.232 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.222+0000 7fd7c37fe700 1 -- 172.21.15.38:0/3642833665 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7bc00f620 con 0x7fd7cc0feea0 2023-03-09T14:45:29.232 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.222+0000 7fd7d43f6700 1 -- 172.21.15.38:0/3642833665 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7cc10d8b0 con 0x7fd7cc0feea0 2023-03-09T14:45:29.232 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.223+0000 7fd7d43f6700 1 -- 172.21.15.38:0/3642833665 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7cc109ce0 con 0x7fd7cc0feea0 2023-03-09T14:45:29.232 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.224+0000 7fd7c37fe700 1 -- 172.21.15.38:0/3642833665 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 133238+0+0 (secure 0 0 0) 0x7fd7bc022070 con 0x7fd7cc0feea0 2023-03-09T14:45:29.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:28 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/727891374' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-03-09T14:45:29.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:28 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/1042575184' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-03-09T14:45:29.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:28 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/964437746' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-03-09T14:45:29.233 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:28 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/727891374' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-03-09T14:45:29.233 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:28 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/1042575184' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-03-09T14:45:29.234 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:28 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/964437746' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-03-09T14:45:29.234 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.224+0000 7fd7c37fe700 1 --2- 172.21.15.38:0/3642833665 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fd7b809f400 0x7fd7b80a18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:29.234 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.224+0000 7fd7d2192700 1 --2- 172.21.15.38:0/3642833665 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fd7b809f400 0x7fd7b80a18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:29.234 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.224+0000 7fd7c37fe700 1 -- 172.21.15.38:0/3642833665 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fd7bc0c9e80 con 0x7fd7cc0feea0 2023-03-09T14:45:29.235 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.225+0000 7fd7d2192700 1 --2- 172.21.15.38:0/3642833665 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fd7b809f400 0x7fd7b80a18b0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd7c8005950 tx=0x7fd7c800b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:29.235 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.229+0000 7fd7c37fe700 1 -- 172.21.15.38:0/3642833665 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7fd7bc093ef0 con 0x7fd7cc0feea0 2023-03-09T14:45:29.432 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.429+0000 7fd7d43f6700 1 -- 172.21.15.38:0/3642833665 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fd7cc061390 con 0x7fd7cc0feea0 2023-03-09T14:45:29.433 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.429+0000 7fd7c37fe700 1 -- 172.21.15.38:0/3642833665 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) v1 ==== 103+0+11 (secure 0 0 0) 0x7fd7bc020020 con 0x7fd7cc0feea0 2023-03-09T14:45:29.433 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 32 2023-03-09T14:45:29.435 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.432+0000 7fd7d43f6700 1 -- 172.21.15.38:0/3642833665 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fd7b809f400 msgr2=0x7fd7b80a18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:29.435 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.432+0000 7fd7d43f6700 1 --2- 172.21.15.38:0/3642833665 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fd7b809f400 0x7fd7b80a18b0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd7c8005950 tx=0x7fd7c800b410 comp rx=0 tx=0).stop 2023-03-09T14:45:29.435 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.432+0000 7fd7d43f6700 1 -- 172.21.15.38:0/3642833665 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd7cc0feea0 msgr2=0x7fd7cc198440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:29.435 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.432+0000 7fd7d43f6700 1 --2- 172.21.15.38:0/3642833665 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd7cc0feea0 0x7fd7cc198440 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fd7bc00d950 tx=0x7fd7bc00dc60 comp rx=0 tx=0).stop 2023-03-09T14:45:29.436 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.432+0000 7fd7d43f6700 1 -- 172.21.15.38:0/3642833665 shutdown_connections 2023-03-09T14:45:29.436 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.432+0000 7fd7d43f6700 1 --2- 172.21.15.38:0/3642833665 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fd7b809f400 0x7fd7b80a18b0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:29.436 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.432+0000 7fd7d43f6700 1 --2- 172.21.15.38:0/3642833665 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd7cc0ff850 0x7fd7cc19f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:29.436 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.432+0000 7fd7d43f6700 1 --2- 172.21.15.38:0/3642833665 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd7cc0fe140 0x7fd7cc197f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:29.436 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.432+0000 7fd7d43f6700 1 --2- 172.21.15.38:0/3642833665 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd7cc0feea0 0x7fd7cc198440 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:29.436 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.432+0000 7fd7d43f6700 1 -- 172.21.15.38:0/3642833665 >> 172.21.15.38:0/3642833665 conn(0x7fd7cc0f9b60 msgr2=0x7fd7cc0fc8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:29.436 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.432+0000 7fd7d43f6700 1 -- 172.21.15.38:0/3642833665 shutdown_connections 2023-03-09T14:45:29.437 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.432+0000 7fd7d43f6700 1 -- 172.21.15.38:0/3642833665 wait complete. 2023-03-09T14:45:29.448 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-09T14:45:29.575 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.570+0000 7f2f22eab700 1 -- 172.21.15.38:0/1760260848 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2f1c100900 msgr2=0x7f2f1c1071c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:29.575 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.570+0000 7f2f22eab700 1 --2- 172.21.15.38:0/1760260848 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2f1c100900 0x7f2f1c1071c0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f2f1400b0a0 tx=0x7f2f1400b3b0 comp rx=0 tx=0).stop 2023-03-09T14:45:29.575 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.570+0000 7f2f22eab700 1 -- 172.21.15.38:0/1760260848 shutdown_connections 2023-03-09T14:45:29.575 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.570+0000 7f2f22eab700 1 --2- 172.21.15.38:0/1760260848 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2f1c100900 0x7f2f1c1071c0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:29.576 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.570+0000 7f2f22eab700 1 --2- 172.21.15.38:0/1760260848 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2f1c0fed80 0x7f2f1c0ff190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:29.576 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.570+0000 7f2f22eab700 1 --2- 172.21.15.38:0/1760260848 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2f1c0fff70 0x7f2f1c1003c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:29.576 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.570+0000 7f2f22eab700 1 -- 172.21.15.38:0/1760260848 >> 172.21.15.38:0/1760260848 conn(0x7f2f1c0fa380 msgr2=0x7f2f1c0fc7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:29.576 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.570+0000 7f2f22eab700 1 -- 172.21.15.38:0/1760260848 shutdown_connections 2023-03-09T14:45:29.576 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.570+0000 7f2f22eab700 1 -- 172.21.15.38:0/1760260848 wait complete. 2023-03-09T14:45:29.576 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.571+0000 7f2f22eab700 1 Processor -- start 2023-03-09T14:45:29.576 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.571+0000 7f2f22eab700 1 -- start start 2023-03-09T14:45:29.577 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.571+0000 7f2f22eab700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2f1c0fed80 0x7f2f1c068610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:29.577 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.571+0000 7f2f22eab700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2f1c0fff70 0x7f2f1c19a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:29.577 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.571+0000 7f2f22eab700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2f1c100900 0x7f2f1c19d050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:29.577 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.571+0000 7f2f22eab700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f1c19d7f0 con 0x7f2f1c0fff70 2023-03-09T14:45:29.577 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.571+0000 7f2f22eab700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f1c19d960 con 0x7f2f1c100900 2023-03-09T14:45:29.577 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.571+0000 7f2f22eab700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f1c19db20 con 0x7f2f1c0fed80 2023-03-09T14:45:29.577 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.571+0000 7f2f1bfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2f1c0fff70 0x7f2f1c19a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:29.578 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.571+0000 7f2f1bfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2f1c0fff70 0x7f2f1c19a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37778/0 (socket says 172.21.15.38:37778) 2023-03-09T14:45:29.578 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.571+0000 7f2f1bfff700 1 -- 172.21.15.38:0/1335451159 learned_addr learned my addr 172.21.15.38:0/1335451159 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:29.578 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.571+0000 7f2f20c47700 1 --2- 172.21.15.38:0/1335451159 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2f1c0fed80 0x7f2f1c068610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:29.578 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.571+0000 7f2f21448700 1 --2- 172.21.15.38:0/1335451159 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2f1c100900 0x7f2f1c19d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:29.578 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.571+0000 7f2f1bfff700 1 -- 172.21.15.38:0/1335451159 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2f1c0fed80 msgr2=0x7f2f1c068610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:29.578 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.571+0000 7f2f1bfff700 1 --2- 172.21.15.38:0/1335451159 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2f1c0fed80 0x7f2f1c068610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:29.579 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.571+0000 7f2f1bfff700 1 -- 172.21.15.38:0/1335451159 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2f1c100900 msgr2=0x7f2f1c19d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:29.579 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.571+0000 7f2f1bfff700 1 --2- 172.21.15.38:0/1335451159 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2f1c100900 0x7f2f1c19d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:29.579 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.571+0000 7f2f1bfff700 1 -- 172.21.15.38:0/1335451159 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f14009d00 con 0x7f2f1c0fff70 2023-03-09T14:45:29.579 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.571+0000 7f2f1bfff700 1 --2- 172.21.15.38:0/1335451159 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2f1c0fff70 0x7f2f1c19a9c0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f2f0800e9a0 tx=0x7f2f0800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:29.579 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.572+0000 7f2f19ffb700 1 -- 172.21.15.38:0/1335451159 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f08009790 con 0x7f2f1c0fff70 2023-03-09T14:45:29.579 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.572+0000 7f2f19ffb700 1 -- 172.21.15.38:0/1335451159 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2f0800cbd0 con 0x7f2f1c0fff70 2023-03-09T14:45:29.579 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.572+0000 7f2f19ffb700 1 -- 172.21.15.38:0/1335451159 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f08018680 con 0x7f2f1c0fff70 2023-03-09T14:45:29.579 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.572+0000 7f2f22eab700 1 -- 172.21.15.38:0/1335451159 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f1c10a6f0 con 0x7f2f1c0fff70 2023-03-09T14:45:29.580 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.572+0000 7f2f22eab700 1 -- 172.21.15.38:0/1335451159 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f1c10a560 con 0x7f2f1c0fff70 2023-03-09T14:45:29.580 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.573+0000 7f2f22eab700 1 -- 172.21.15.38:0/1335451159 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f1c105c30 con 0x7f2f1c0fff70 2023-03-09T14:45:29.581 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.574+0000 7f2f19ffb700 1 -- 172.21.15.38:0/1335451159 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 133238+0+0 (secure 0 0 0) 0x7f2f0800c040 con 0x7f2f1c0fff70 2023-03-09T14:45:29.581 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.574+0000 7f2f19ffb700 1 --2- 172.21.15.38:0/1335451159 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f2f0c09f480 0x7f2f0c0a1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:29.581 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.575+0000 7f2f19ffb700 1 -- 172.21.15.38:0/1335451159 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f2f08014070 con 0x7f2f1c0fff70 2023-03-09T14:45:29.581 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.575+0000 7f2f20c47700 1 --2- 172.21.15.38:0/1335451159 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f2f0c09f480 0x7f2f0c0a1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:29.581 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.575+0000 7f2f20c47700 1 --2- 172.21.15.38:0/1335451159 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f2f0c09f480 0x7f2f0c0a1930 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f2f1c0ffdd0 tx=0x7f2f10009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:29.581 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.579+0000 7f2f19ffb700 1 -- 172.21.15.38:0/1335451159 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f2f080515b0 con 0x7f2f1c0fff70 2023-03-09T14:45:29.829 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.826+0000 7f2f22eab700 1 -- 172.21.15.38:0/1335451159 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2f1c061640 con 0x7f2f1c0fff70 2023-03-09T14:45:29.830 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.826+0000 7f2f19ffb700 1 -- 172.21.15.38:0/1335451159 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+10801 (secure 0 0 0) 0x7f2f080943a0 con 0x7f2f1c0fff70 2023-03-09T14:45:29.830 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:45:29.830 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/768918602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":768918602},{"type":"v1","addr":"172.21.15.38:6835","nonce":768918602}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/18381540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":18381540},{"type":"v1","addr":"172.21.15.38:6837","nonce":18381540}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14610,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6839/4139020066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":4139020066},{"type":"v1","addr":"172.21.15.38:6839","nonce":4139020066}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14622,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6841/1397115322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":1397115322},{"type":"v1","addr":"172.21.15.38:6841","nonce":1397115322}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6833/415305571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":415305571},{"type":"v1","addr":"172.21.15.181:6833","nonce":415305571}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/659300520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":659300520},{"type":"v1","addr":"172.21.15.122:6833","nonce":659300520}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-03-09T14:44:19.083770+0000","modified":"2023-03-09T14:44:39.526150+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24530,"mds_1":24494,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24448":{"gid":24448,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6835/2576255861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":2576255861},{"type":"v1","addr":"172.21.15.181:6835","nonce":2576255861}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24460":{"gid":24460,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6837/2942560181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":2942560181},{"type":"v1","addr":"172.21.15.181:6837","nonce":2942560181}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24464":{"gid":24464,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.122:6835/3675537941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":3675537941},{"type":"v1","addr":"172.21.15.122:6835","nonce":3675537941}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24478":{"gid":24478,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":9,"addr":"172.21.15.181:6839/3902263660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":3902263660},{"type":"v1","addr":"172.21.15.181:6839","nonce":3902263660}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24494":{"gid":24494,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.122:6837/2210214596","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":2210214596},{"type":"v1","addr":"172.21.15.122:6837","nonce":2210214596}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24530":{"gid":24530,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6839/971050165","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6838","nonce":971050165},{"type":"v1","addr":"172.21.15.122:6839","nonce":971050165}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-09T14:45:29.832 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.829+0000 7f2f22eab700 1 -- 172.21.15.38:0/1335451159 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f2f0c09f480 msgr2=0x7f2f0c0a1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:29.832 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.829+0000 7f2f22eab700 1 --2- 172.21.15.38:0/1335451159 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f2f0c09f480 0x7f2f0c0a1930 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f2f1c0ffdd0 tx=0x7f2f10009260 comp rx=0 tx=0).stop 2023-03-09T14:45:29.832 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.829+0000 7f2f22eab700 1 -- 172.21.15.38:0/1335451159 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2f1c0fff70 msgr2=0x7f2f1c19a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:29.832 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.829+0000 7f2f22eab700 1 --2- 172.21.15.38:0/1335451159 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2f1c0fff70 0x7f2f1c19a9c0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f2f0800e9a0 tx=0x7f2f0800ecb0 comp rx=0 tx=0).stop 2023-03-09T14:45:29.833 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.829+0000 7f2f22eab700 1 -- 172.21.15.38:0/1335451159 shutdown_connections 2023-03-09T14:45:29.833 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.829+0000 7f2f22eab700 1 --2- 172.21.15.38:0/1335451159 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f2f0c09f480 0x7f2f0c0a1930 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:29.833 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.829+0000 7f2f22eab700 1 --2- 172.21.15.38:0/1335451159 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2f1c100900 0x7f2f1c19d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:29.833 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.829+0000 7f2f22eab700 1 --2- 172.21.15.38:0/1335451159 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2f1c0fed80 0x7f2f1c068610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:29.833 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.829+0000 7f2f22eab700 1 --2- 172.21.15.38:0/1335451159 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2f1c0fff70 0x7f2f1c19a9c0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:29.833 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.829+0000 7f2f22eab700 1 -- 172.21.15.38:0/1335451159 >> 172.21.15.38:0/1335451159 conn(0x7f2f1c0fa380 msgr2=0x7f2f1c0fc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:29.833 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.829+0000 7f2f22eab700 1 -- 172.21.15.38:0/1335451159 shutdown_connections 2023-03-09T14:45:29.834 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.829+0000 7f2f22eab700 1 -- 172.21.15.38:0/1335451159 wait complete. 2023-03-09T14:45:29.834 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 23 2023-03-09T14:45:29.846 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2023-03-09T14:44:19.083770+0000', 'modified': '2023-03-09T14:44:39.526150+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24530, 'mds_1': 24494, 'mds_2': 24478}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24448': {'gid': 24448, 'name': 'f', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.181:6835/2576255861', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6834', 'nonce': 2576255861}, {'type': 'v1', 'addr': '172.21.15.181:6835', 'nonce': 2576255861}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24460': {'gid': 24460, 'name': 'c', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.181:6837/2942560181', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6836', 'nonce': 2942560181}, {'type': 'v1', 'addr': '172.21.15.181:6837', 'nonce': 2942560181}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24464': {'gid': 24464, 'name': 'e', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.122:6835/3675537941', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 3675537941}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 3675537941}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24478': {'gid': 24478, 'name': 'i', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.181:6839/3902263660', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6838', 'nonce': 3902263660}, {'type': 'v1', 'addr': '172.21.15.181:6839', 'nonce': 3902263660}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24494': {'gid': 24494, 'name': 'h', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.122:6837/2210214596', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 2210214596}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 2210214596}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24530': {'gid': 24530, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6839/971050165', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6838', 'nonce': 971050165}, {'type': 'v1', 'addr': '172.21.15.122:6839', 'nonce': 971050165}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-03-09T14:45:29.846 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2023-03-09T14:45:29.847 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2023-03-09T14:45:29.974 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.971+0000 7ffb4ad92700 1 -- 172.21.15.38:0/1943541888 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffb44100990 msgr2=0x7ffb44102d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:29.975 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.971+0000 7ffb4ad92700 1 --2- 172.21.15.38:0/1943541888 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffb44100990 0x7ffb44102d70 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7ffb38009990 tx=0x7ffb38009ca0 comp rx=0 tx=0).stop 2023-03-09T14:45:29.975 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.971+0000 7ffb4ad92700 1 -- 172.21.15.38:0/1943541888 shutdown_connections 2023-03-09T14:45:29.975 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.971+0000 7ffb4ad92700 1 --2- 172.21.15.38:0/1943541888 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffb44100990 0x7ffb44102d70 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:29.975 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.971+0000 7ffb4ad92700 1 --2- 172.21.15.38:0/1943541888 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffb44105bd0 0x7ffb44107fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:29.975 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.971+0000 7ffb4ad92700 1 --2- 172.21.15.38:0/1943541888 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffb441032b0 0x7ffb44105690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:29.975 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.971+0000 7ffb4ad92700 1 -- 172.21.15.38:0/1943541888 >> 172.21.15.38:0/1943541888 conn(0x7ffb440fa3b0 msgr2=0x7ffb440fc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:29.976 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.971+0000 7ffb4ad92700 1 -- 172.21.15.38:0/1943541888 shutdown_connections 2023-03-09T14:45:29.976 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.971+0000 7ffb4ad92700 1 -- 172.21.15.38:0/1943541888 wait complete. 2023-03-09T14:45:29.976 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.971+0000 7ffb4ad92700 1 Processor -- start 2023-03-09T14:45:29.976 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.971+0000 7ffb4ad92700 1 -- start start 2023-03-09T14:45:29.977 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.972+0000 7ffb4ad92700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffb44100990 0x7ffb44195c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:29.978 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.972+0000 7ffb4ad92700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffb441032b0 0x7ffb441961b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:29.978 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.972+0000 7ffb4ad92700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffb44105bd0 0x7ffb4419d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:29.978 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.972+0000 7ffb4ad92700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4419d7b0 con 0x7ffb44105bd0 2023-03-09T14:45:29.978 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.972+0000 7ffb4ad92700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4419d920 con 0x7ffb441032b0 2023-03-09T14:45:29.978 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.972+0000 7ffb4ad92700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4419dae0 con 0x7ffb44100990 2023-03-09T14:45:29.978 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.972+0000 7ffb4932f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffb44105bd0 0x7ffb4419d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:29.979 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.972+0000 7ffb4932f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffb44105bd0 0x7ffb4419d1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37794/0 (socket says 172.21.15.38:37794) 2023-03-09T14:45:29.979 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.972+0000 7ffb4932f700 1 -- 172.21.15.38:0/2084288664 learned_addr learned my addr 172.21.15.38:0/2084288664 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:29.979 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.972+0000 7ffb43fff700 1 --2- 172.21.15.38:0/2084288664 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffb441032b0 0x7ffb441961b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:29.979 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.972+0000 7ffb48b2e700 1 --2- 172.21.15.38:0/2084288664 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffb44100990 0x7ffb44195c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:29.979 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.972+0000 7ffb4932f700 1 -- 172.21.15.38:0/2084288664 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffb44100990 msgr2=0x7ffb44195c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:29.979 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.972+0000 7ffb4932f700 1 --2- 172.21.15.38:0/2084288664 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffb44100990 0x7ffb44195c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:29.979 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.972+0000 7ffb4932f700 1 -- 172.21.15.38:0/2084288664 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffb441032b0 msgr2=0x7ffb441961b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:29.979 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.972+0000 7ffb4932f700 1 --2- 172.21.15.38:0/2084288664 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffb441032b0 0x7ffb441961b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:29.980 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.972+0000 7ffb4932f700 1 -- 172.21.15.38:0/2084288664 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb380095f0 con 0x7ffb44105bd0 2023-03-09T14:45:29.980 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.972+0000 7ffb43fff700 1 --2- 172.21.15.38:0/2084288664 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffb441032b0 0x7ffb441961b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:45:29.980 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.972+0000 7ffb4932f700 1 --2- 172.21.15.38:0/2084288664 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffb44105bd0 0x7ffb4419d1e0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7ffb3c007ee0 tx=0x7ffb3c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:29.980 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.972+0000 7ffb41ffb700 1 -- 172.21.15.38:0/2084288664 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb3c017070 con 0x7ffb44105bd0 2023-03-09T14:45:29.980 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.972+0000 7ffb41ffb700 1 -- 172.21.15.38:0/2084288664 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffb3c00f040 con 0x7ffb44105bd0 2023-03-09T14:45:29.980 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.972+0000 7ffb4ad92700 1 -- 172.21.15.38:0/2084288664 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb4410b820 con 0x7ffb44105bd0 2023-03-09T14:45:29.980 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.972+0000 7ffb41ffb700 1 -- 172.21.15.38:0/2084288664 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb3c005740 con 0x7ffb44105bd0 2023-03-09T14:45:29.980 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.973+0000 7ffb4ad92700 1 -- 172.21.15.38:0/2084288664 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb4410b690 con 0x7ffb44105bd0 2023-03-09T14:45:29.981 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.974+0000 7ffb41ffb700 1 -- 172.21.15.38:0/2084288664 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 133238+0+0 (secure 0 0 0) 0x7ffb3c0058a0 con 0x7ffb44105bd0 2023-03-09T14:45:29.985 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.975+0000 7ffb41ffb700 1 --2- 172.21.15.38:0/2084288664 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7ffb3409f400 0x7ffb340a18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:29.985 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.975+0000 7ffb41ffb700 1 -- 172.21.15.38:0/2084288664 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ffb3c0c9b40 con 0x7ffb44105bd0 2023-03-09T14:45:29.986 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.975+0000 7ffb48b2e700 1 --2- 172.21.15.38:0/2084288664 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7ffb3409f400 0x7ffb340a18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:29.986 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.976+0000 7ffb48b2e700 1 --2- 172.21.15.38:0/2084288664 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7ffb3409f400 0x7ffb340a18b0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ffb3800b5c0 tx=0x7ffb38005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:29.986 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.976+0000 7ffb56f1bb80 1 Processor -- start 2023-03-09T14:45:29.986 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.976+0000 7ffb56f1bb80 1 -- start start 2023-03-09T14:45:29.986 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.976+0000 7ffb56f1bb80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55682f8a49f0 0x55682f8a4e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:29.986 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.976+0000 7ffb56f1bb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55682f8a5340 0x55682f8a5c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:29.986 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.976+0000 7ffb56f1bb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55682f8a61f0 0x55682f8a6660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:29.986 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.976+0000 7ffb56f1bb80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55682f78ed50 con 0x55682f8a49f0 2023-03-09T14:45:29.987 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.976+0000 7ffb56f1bb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55682f7fd730 con 0x55682f8a61f0 2023-03-09T14:45:29.987 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.976+0000 7ffb56f1bb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55682f7f6c60 con 0x55682f8a5340 2023-03-09T14:45:29.987 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.976+0000 7ffb48b2e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55682f8a49f0 0x55682f8a4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:29.987 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.976+0000 7ffb48b2e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55682f8a49f0 0x55682f8a4e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37798/0 (socket says 172.21.15.38:37798) 2023-03-09T14:45:29.987 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.976+0000 7ffb48b2e700 1 -- 172.21.15.38:0/2712416995 learned_addr learned my addr 172.21.15.38:0/2712416995 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:29.987 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.977+0000 7ffb43fff700 1 --2- 172.21.15.38:0/2712416995 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55682f8a5340 0x55682f8a5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:29.987 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.977+0000 7ffb48b2e700 1 -- 172.21.15.38:0/2712416995 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55682f8a5340 msgr2=0x55682f8a5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:45:29.988 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.977+0000 7ffb4932f700 1 --2- 172.21.15.38:0/2712416995 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55682f8a61f0 0x55682f8a6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:29.988 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.977+0000 7ffb48b2e700 1 --2- 172.21.15.38:0/2712416995 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55682f8a5340 0x55682f8a5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:29.988 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.977+0000 7ffb48b2e700 1 -- 172.21.15.38:0/2712416995 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55682f8a61f0 msgr2=0x55682f8a6660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:45:29.988 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.977+0000 7ffb48b2e700 1 --2- 172.21.15.38:0/2712416995 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55682f8a61f0 0x55682f8a6660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:29.988 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.977+0000 7ffb48b2e700 1 -- 172.21.15.38:0/2712416995 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55682f7bd140 con 0x55682f8a49f0 2023-03-09T14:45:29.988 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.977+0000 7ffb43fff700 1 --2- 172.21.15.38:0/2712416995 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55682f8a5340 0x55682f8a5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-09T14:45:29.988 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.977+0000 7ffb4932f700 1 --2- 172.21.15.38:0/2712416995 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55682f8a61f0 0x55682f8a6660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-09T14:45:29.988 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.977+0000 7ffb48b2e700 1 --2- 172.21.15.38:0/2712416995 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55682f8a49f0 0x55682f8a4e00 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7ffb3800b7d0 tx=0x7ffb3801f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b052f212e9dc748c server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:29.989 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.977+0000 7ffb2effd700 1 -- 172.21.15.38:0/2712416995 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb3802b070 con 0x55682f8a49f0 2023-03-09T14:45:29.989 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.977+0000 7ffb2effd700 1 -- 172.21.15.38:0/2712416995 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffb3801e040 con 0x55682f8a49f0 2023-03-09T14:45:29.989 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.977+0000 7ffb2effd700 1 -- 172.21.15.38:0/2712416995 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb380248b0 con 0x55682f8a49f0 2023-03-09T14:45:29.989 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.978+0000 7ffb56f1bb80 1 -- 172.21.15.38:0/2712416995 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55682f8a49f0 msgr2=0x55682f8a4e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:29.989 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.978+0000 7ffb56f1bb80 1 --2- 172.21.15.38:0/2712416995 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55682f8a49f0 0x55682f8a4e00 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7ffb3800b7d0 tx=0x7ffb3801f000 comp rx=0 tx=0).stop 2023-03-09T14:45:29.989 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.978+0000 7ffb56f1bb80 1 -- 172.21.15.38:0/2712416995 shutdown_connections 2023-03-09T14:45:29.989 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.978+0000 7ffb56f1bb80 1 --2- 172.21.15.38:0/2712416995 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55682f8a49f0 0x55682f8a4e00 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:29.990 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.978+0000 7ffb56f1bb80 1 --2- 172.21.15.38:0/2712416995 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55682f8a61f0 0x55682f8a6660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:29.990 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.978+0000 7ffb56f1bb80 1 --2- 172.21.15.38:0/2712416995 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55682f8a5340 0x55682f8a5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:29.990 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.978+0000 7ffb56f1bb80 1 -- 172.21.15.38:0/2712416995 >> 172.21.15.38:0/2712416995 conn(0x55682f89f500 msgr2=0x55682f89f8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:29.990 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.978+0000 7ffb56f1bb80 1 -- 172.21.15.38:0/2712416995 shutdown_connections 2023-03-09T14:45:29.990 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.978+0000 7ffb56f1bb80 1 -- 172.21.15.38:0/2712416995 wait complete. 2023-03-09T14:45:29.990 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.978+0000 7ffb56f1bb80 1 -- start start 2023-03-09T14:45:29.990 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.978+0000 7ffb56f1bb80 1 Processor -- start 2023-03-09T14:45:29.991 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.979+0000 7ffb56f1bb80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55682f8a49f0 0x55682f8ae620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:29.991 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.979+0000 7ffb56f1bb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55682f8a5340 0x55682f8aeb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:29.991 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.979+0000 7ffb56f1bb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55682f8a61f0 0x55682f8b5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:29.991 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.979+0000 7ffb56f1bb80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55682f67a8d0 con 0x55682f8a49f0 2023-03-09T14:45:29.991 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.979+0000 7ffb56f1bb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55682f8b60e0 con 0x55682f8a61f0 2023-03-09T14:45:29.991 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.979+0000 7ffb56f1bb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55682f8b6270 con 0x55682f8a5340 2023-03-09T14:45:29.991 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.979+0000 7ffb56f1bb80 10 client.0 _ll_register_callbacks cb 0x55682f79c4d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:45:29.991 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.979+0000 7ffb48b2e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55682f8a49f0 0x55682f8ae620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:29.992 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.979+0000 7ffb48b2e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55682f8a49f0 0x55682f8ae620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37804/0 (socket says 172.21.15.38:37804) 2023-03-09T14:45:29.992 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.979+0000 7ffb48b2e700 1 -- 172.21.15.38:0/1313598180 learned_addr learned my addr 172.21.15.38:0/1313598180 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:29.992 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.979+0000 7ffb43fff700 1 --2- 172.21.15.38:0/1313598180 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55682f8a5340 0x55682f8aeb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:29.992 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.979+0000 7ffb4932f700 1 --2- 172.21.15.38:0/1313598180 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55682f8a61f0 0x55682f8b5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:29.992 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.979+0000 7ffb48b2e700 1 -- 172.21.15.38:0/1313598180 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55682f8a5340 msgr2=0x55682f8aeb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:45:29.992 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.979+0000 7ffb48b2e700 1 --2- 172.21.15.38:0/1313598180 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55682f8a5340 0x55682f8aeb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:29.992 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.979+0000 7ffb48b2e700 1 -- 172.21.15.38:0/1313598180 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55682f8a61f0 msgr2=0x55682f8b5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:45:29.993 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.979+0000 7ffb48b2e700 1 --2- 172.21.15.38:0/1313598180 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55682f8a61f0 0x55682f8b5ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:29.993 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.979+0000 7ffb48b2e700 1 -- 172.21.15.38:0/1313598180 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb44108ba0 con 0x55682f8a49f0 2023-03-09T14:45:29.993 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.979+0000 7ffb4932f700 1 --2- 172.21.15.38:0/1313598180 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55682f8a61f0 0x55682f8b5ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-09T14:45:29.993 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.979+0000 7ffb48b2e700 1 --2- 172.21.15.38:0/1313598180 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55682f8a49f0 0x55682f8ae620 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7ffb38009ce0 tx=0x7ffb38025df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba1a4e2bc19de8ce server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:29.993 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.979+0000 7ffb23fff700 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2023-03-09T14:45:29.993 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.979+0000 7ffb23fff700 1 -- 172.21.15.38:0/1313598180 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb3802b050 con 0x55682f8a49f0 2023-03-09T14:45:29.993 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.979+0000 7ffb23fff700 1 -- 172.21.15.38:0/1313598180 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ffb3800f830 con 0x55682f8a49f0 2023-03-09T14:45:29.993 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.979+0000 7ffb56f1bb80 1 -- 172.21.15.38:0/1313598180 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55682f7c4140 con 0x55682f8a49f0 2023-03-09T14:45:29.994 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.980+0000 7ffb23fff700 1 -- 172.21.15.38:0/1313598180 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffb3800f9f0 con 0x55682f8a49f0 2023-03-09T14:45:29.994 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.980+0000 7ffb56f1bb80 10 client.15150 fetch_fsmap learned FSMap version 23 2023-03-09T14:45:29.994 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.980+0000 7ffb56f1bb80 1 -- 172.21.15.38:0/1313598180 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55682f60dc10 con 0x55682f8a49f0 2023-03-09T14:45:29.994 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.980+0000 7ffb23fff700 1 -- 172.21.15.38:0/1313598180 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55682f7c4140 con 0x55682f8a49f0 2023-03-09T14:45:29.994 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.980+0000 7ffb23fff700 1 -- 172.21.15.38:0/1313598180 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7ffb3802f520 con 0x55682f8a49f0 2023-03-09T14:45:29.997 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.980+0000 7ffb56f1bb80 10 client.15150 fetch_fsmap finished waiting for FSMap version 23 2023-03-09T14:45:29.997 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.980+0000 7ffb56f1bb80 10 client.15150 resolve_mds: resolved name 'i' to daemon mds.i 2023-03-09T14:45:29.997 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.980+0000 7ffb56f1bb80 20 client.15150 populate_metadata read hostname 'smithi038' 2023-03-09T14:45:29.997 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.980+0000 7ffb56f1bb80 1 --2- 172.21.15.38:0/1313598180 >> [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] conn(0x55682f8b9bb0 0x55682f8b8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:29.997 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.980+0000 7ffb56f1bb80 4 client.15150 mds_command: new command op to 24478 tid=0[{"prefix": "get_command_descriptions"}] 2023-03-09T14:45:29.998 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.980+0000 7ffb56f1bb80 1 -- 172.21.15.38:0/1313598180 --> [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55682f67af10 con 0x55682f8b9bb0 2023-03-09T14:45:29.998 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.980+0000 7ffb43fff700 1 --2- 172.21.15.38:0/1313598180 >> [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] conn(0x55682f8b9bb0 0x55682f8b8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:29.998 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.981+0000 7ffb43fff700 1 --2- 172.21.15.38:0/1313598180 >> [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] conn(0x55682f8b9bb0 0x55682f8b8cc0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d939e13ae8ffbb5d server_cookie=9d92653bf6216484 in_seq=0 out_seq=0 2023-03-09T14:45:29.998 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.981+0000 7ffb23fff700 10 client.15150 ms_handle_connect on v2:172.21.15.181:6838/3902263660 2023-03-09T14:45:29.998 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.981+0000 7ffb23fff700 1 -- 172.21.15.38:0/1313598180 <== mds.2 v2:172.21.15.181:6838/3902263660 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18278 (crc 0 0 0) 0x55682f67af10 con 0x55682f8b9bb0 2023-03-09T14:45:29.999 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.981+0000 7ffb23fff700 10 client.15150 handle_command_reply: tid=0 2023-03-09T14:45:29.999 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.981+0000 7ffb56f1bb80 1 client.15150 shutdown 2023-03-09T14:45:29.999 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.982+0000 7ffb56f1bb80 1 -- 172.21.15.38:0/1313598180 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55682f8a49f0 msgr2=0x55682f8ae620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:29.999 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.982+0000 7ffb56f1bb80 1 --2- 172.21.15.38:0/1313598180 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55682f8a49f0 0x55682f8ae620 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7ffb38009ce0 tx=0x7ffb38025df0 comp rx=0 tx=0).stop 2023-03-09T14:45:29.999 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.982+0000 7ffb56f1bb80 1 -- 172.21.15.38:0/1313598180 shutdown_connections 2023-03-09T14:45:29.999 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.982+0000 7ffb56f1bb80 1 --2- 172.21.15.38:0/1313598180 >> [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] conn(0x55682f8b9bb0 0x55682f8b8cc0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:29.999 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.982+0000 7ffb56f1bb80 1 --2- 172.21.15.38:0/1313598180 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55682f8a49f0 0x55682f8ae620 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.000 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.982+0000 7ffb23fff700 0 client.15150 ms_handle_reset on v2:172.21.15.181:6838/3902263660 2023-03-09T14:45:30.000 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.982+0000 7ffb56f1bb80 1 --2- 172.21.15.38:0/1313598180 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55682f8a61f0 0x55682f8b5ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.000 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.982+0000 7ffb56f1bb80 1 --2- 172.21.15.38:0/1313598180 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55682f8a5340 0x55682f8aeb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.000 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.982+0000 7ffb56f1bb80 1 -- 172.21.15.38:0/1313598180 >> 172.21.15.38:0/1313598180 conn(0x55682f89f500 msgr2=0x55682f89ebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:30.000 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.982+0000 7ffb56f1bb80 1 -- 172.21.15.38:0/1313598180 shutdown_connections 2023-03-09T14:45:30.000 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.982+0000 7ffb56f1bb80 1 -- 172.21.15.38:0/1313598180 wait complete. 2023-03-09T14:45:30.000 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.982+0000 7ffb56f1bb80 20 client.15150 trim_cache size 0 max 16384 2023-03-09T14:45:30.009 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.999+0000 7ffb56f1bb80 1 Processor -- start 2023-03-09T14:45:30.010 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.999+0000 7ffb56f1bb80 1 -- start start 2023-03-09T14:45:30.010 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.999+0000 7ffb56f1bb80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55682f8a49f0 0x55682f8b8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:30.010 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:29.999+0000 7ffb56f1bb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55682f8a5340 0x55682f8b5b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:30.010 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.000+0000 7ffb56f1bb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55682f8a61f0 0x55682f89ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:30.011 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.000+0000 7ffb56f1bb80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55682f8a5a00 con 0x55682f8a49f0 2023-03-09T14:45:30.011 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.000+0000 7ffb56f1bb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55682f8ad200 con 0x55682f8a5340 2023-03-09T14:45:30.011 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.000+0000 7ffb56f1bb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55682f67af10 con 0x55682f8a61f0 2023-03-09T14:45:30.011 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.000+0000 7ffb48b2e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55682f8a49f0 0x55682f8b8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:30.012 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.000+0000 7ffb48b2e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55682f8a49f0 0x55682f8b8cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37812/0 (socket says 172.21.15.38:37812) 2023-03-09T14:45:30.012 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.000+0000 7ffb48b2e700 1 -- 172.21.15.38:0/1025758517 learned_addr learned my addr 172.21.15.38:0/1025758517 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:30.012 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.000+0000 7ffb43fff700 1 --2- 172.21.15.38:0/1025758517 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55682f8a5340 0x55682f8b5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:30.013 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.000+0000 7ffb48b2e700 1 -- 172.21.15.38:0/1025758517 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55682f8a61f0 msgr2=0x55682f89ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:45:30.013 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.000+0000 7ffb48b2e700 1 --2- 172.21.15.38:0/1025758517 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55682f8a61f0 0x55682f89ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.013 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.000+0000 7ffb48b2e700 1 -- 172.21.15.38:0/1025758517 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55682f8a5340 msgr2=0x55682f8b5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:45:30.014 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.000+0000 7ffb48b2e700 1 --2- 172.21.15.38:0/1025758517 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55682f8a5340 0x55682f8b5b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.014 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.000+0000 7ffb48b2e700 1 -- 172.21.15.38:0/1025758517 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55682f624d90 con 0x55682f8a49f0 2023-03-09T14:45:30.014 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.000+0000 7ffb43fff700 1 --2- 172.21.15.38:0/1025758517 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55682f8a5340 0x55682f8b5b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-09T14:45:30.015 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.000+0000 7ffb48b2e700 1 --2- 172.21.15.38:0/1025758517 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55682f8a49f0 0x55682f8b8cc0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7ffb3800c010 tx=0x7ffb38010f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c901ade7767d1a17 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:30.015 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.000+0000 7ffb2f7fe700 1 -- 172.21.15.38:0/1025758517 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb3802b040 con 0x55682f8a49f0 2023-03-09T14:45:30.015 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.000+0000 7ffb2f7fe700 1 -- 172.21.15.38:0/1025758517 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ffb3801e070 con 0x55682f8a49f0 2023-03-09T14:45:30.015 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.000+0000 7ffb2f7fe700 1 -- 172.21.15.38:0/1025758517 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb3802e610 con 0x55682f8a49f0 2023-03-09T14:45:30.016 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.001+0000 7ffb56f1bb80 1 -- 172.21.15.38:0/1025758517 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55682f8a49f0 msgr2=0x55682f8b8cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:30.016 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.001+0000 7ffb56f1bb80 1 --2- 172.21.15.38:0/1025758517 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55682f8a49f0 0x55682f8b8cc0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7ffb3800c010 tx=0x7ffb38010f90 comp rx=0 tx=0).stop 2023-03-09T14:45:30.016 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.001+0000 7ffb56f1bb80 1 -- 172.21.15.38:0/1025758517 shutdown_connections 2023-03-09T14:45:30.017 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.001+0000 7ffb56f1bb80 1 --2- 172.21.15.38:0/1025758517 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55682f8a49f0 0x55682f8b8cc0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.017 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.001+0000 7ffb56f1bb80 1 --2- 172.21.15.38:0/1025758517 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55682f8a61f0 0x55682f89ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.017 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.001+0000 7ffb56f1bb80 1 --2- 172.21.15.38:0/1025758517 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55682f8a5340 0x55682f8b5b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.017 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.001+0000 7ffb56f1bb80 1 -- 172.21.15.38:0/1025758517 >> 172.21.15.38:0/1025758517 conn(0x55682f89f500 msgr2=0x55682f8c2310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:30.018 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.001+0000 7ffb56f1bb80 1 -- 172.21.15.38:0/1025758517 shutdown_connections 2023-03-09T14:45:30.018 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.001+0000 7ffb56f1bb80 1 -- 172.21.15.38:0/1025758517 wait complete. 2023-03-09T14:45:30.018 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.001+0000 7ffb56f1bb80 1 -- start start 2023-03-09T14:45:30.018 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.001+0000 7ffb56f1bb80 1 Processor -- start 2023-03-09T14:45:30.019 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.002+0000 7ffb56f1bb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55682f8a49f0 0x55682f8b0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:30.019 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.002+0000 7ffb56f1bb80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55682f8a5340 0x55682f8b1310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:30.019 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.002+0000 7ffb56f1bb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55682f8a61f0 0x55682f90d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:30.019 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.002+0000 7ffb56f1bb80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55682f8b1a20 con 0x55682f8a5340 2023-03-09T14:45:30.020 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.002+0000 7ffb56f1bb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55682f90d960 con 0x55682f8a61f0 2023-03-09T14:45:30.020 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.002+0000 7ffb56f1bb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55682f90db20 con 0x55682f8a49f0 2023-03-09T14:45:30.020 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.002+0000 7ffb56f1bb80 10 client.0 _ll_register_callbacks cb 0x55682f79c4d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:45:30.020 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.002+0000 7ffb43fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55682f8a5340 0x55682f8b1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:30.021 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.002+0000 7ffb43fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55682f8a5340 0x55682f8b1310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37820/0 (socket says 172.21.15.38:37820) 2023-03-09T14:45:30.021 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.002+0000 7ffb43fff700 1 -- 172.21.15.38:0/999156504 learned_addr learned my addr 172.21.15.38:0/999156504 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:30.021 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.002+0000 7ffb4932f700 1 --2- 172.21.15.38:0/999156504 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55682f8a61f0 0x55682f90d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:30.021 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.002+0000 7ffb43fff700 1 -- 172.21.15.38:0/999156504 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55682f8a49f0 msgr2=0x55682f8b0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:45:30.021 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.002+0000 7ffb48b2e700 1 --2- 172.21.15.38:0/999156504 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55682f8a49f0 0x55682f8b0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:30.022 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.002+0000 7ffb43fff700 1 --2- 172.21.15.38:0/999156504 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55682f8a49f0 0x55682f8b0dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.022 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.002+0000 7ffb43fff700 1 -- 172.21.15.38:0/999156504 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55682f8a61f0 msgr2=0x55682f90d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:45:30.022 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.002+0000 7ffb43fff700 1 --2- 172.21.15.38:0/999156504 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55682f8a61f0 0x55682f90d420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.022 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.002+0000 7ffb43fff700 1 -- 172.21.15.38:0/999156504 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55682f60dc10 con 0x55682f8a5340 2023-03-09T14:45:30.022 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.002+0000 7ffb4932f700 1 --2- 172.21.15.38:0/999156504 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55682f8a61f0 0x55682f90d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:45:30.022 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.002+0000 7ffb43fff700 1 --2- 172.21.15.38:0/999156504 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55682f8a5340 0x55682f8b1310 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x55682f8adaa0 tx=0x7ffb30019b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3618c545501c5161 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:30.022 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.002+0000 7ffb23fff700 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2023-03-09T14:45:30.023 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.002+0000 7ffb23fff700 1 -- 172.21.15.38:0/999156504 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb30017050 con 0x55682f8a5340 2023-03-09T14:45:30.023 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.002+0000 7ffb23fff700 1 -- 172.21.15.38:0/999156504 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ffb30022470 con 0x55682f8a5340 2023-03-09T14:45:30.023 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.002+0000 7ffb56f1bb80 1 -- 172.21.15.38:0/999156504 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55682f7b4160 con 0x55682f8a5340 2023-03-09T14:45:30.023 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.002+0000 7ffb23fff700 1 -- 172.21.15.38:0/999156504 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffb30022630 con 0x55682f8a5340 2023-03-09T14:45:30.023 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.002+0000 7ffb56f1bb80 10 client.15156 fetch_fsmap learned FSMap version 23 2023-03-09T14:45:30.023 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.002+0000 7ffb56f1bb80 1 -- 172.21.15.38:0/999156504 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55682f623260 con 0x55682f8a5340 2023-03-09T14:45:30.023 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.003+0000 7ffb23fff700 1 -- 172.21.15.38:0/999156504 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55682f7b4160 con 0x55682f8a5340 2023-03-09T14:45:30.023 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.003+0000 7ffb23fff700 1 -- 172.21.15.38:0/999156504 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7ffb30011810 con 0x55682f8a5340 2023-03-09T14:45:30.024 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.003+0000 7ffb56f1bb80 10 client.15156 fetch_fsmap finished waiting for FSMap version 23 2023-03-09T14:45:30.024 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.003+0000 7ffb56f1bb80 10 client.15156 resolve_mds: resolved name 'i' to daemon mds.i 2023-03-09T14:45:30.024 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.003+0000 7ffb56f1bb80 20 client.15156 populate_metadata read hostname 'smithi038' 2023-03-09T14:45:30.024 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.003+0000 7ffb56f1bb80 1 --2- 172.21.15.38:0/999156504 >> [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] conn(0x55682f912880 0x55682f90f210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:30.024 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.003+0000 7ffb56f1bb80 4 client.15156 mds_command: new command op to 24478 tid=0[{"prefix": "status"}] 2023-03-09T14:45:30.024 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.003+0000 7ffb56f1bb80 1 -- 172.21.15.38:0/999156504 --> [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55682f79e170 con 0x55682f912880 2023-03-09T14:45:30.025 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.003+0000 7ffb48b2e700 1 --2- 172.21.15.38:0/999156504 >> [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] conn(0x55682f912880 0x55682f90f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:30.025 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.004+0000 7ffb23fff700 10 client.15156 ms_handle_connect on v2:172.21.15.181:6838/3902263660 2023-03-09T14:45:30.025 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.004+0000 7ffb48b2e700 1 --2- 172.21.15.38:0/999156504 >> [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] conn(0x55682f912880 0x55682f90f210 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9f3885bbf2c84b35 server_cookie=d492e4d21d91053d in_seq=0 out_seq=0 2023-03-09T14:45:30.025 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.004+0000 7ffb23fff700 1 -- 172.21.15.38:0/999156504 <== mds.2 v2:172.21.15.181:6838/3902263660 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x55682f79e170 con 0x55682f912880 2023-03-09T14:45:30.025 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.004+0000 7ffb23fff700 10 client.15156 handle_command_reply: tid=0 2023-03-09T14:45:30.025 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.004+0000 7ffb56f1bb80 1 client.15156 shutdown 2023-03-09T14:45:30.025 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.004+0000 7ffb56f1bb80 1 -- 172.21.15.38:0/999156504 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55682f8a5340 msgr2=0x55682f8b1310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:30.026 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.004+0000 7ffb56f1bb80 1 --2- 172.21.15.38:0/999156504 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55682f8a5340 0x55682f8b1310 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x55682f8adaa0 tx=0x7ffb30019b30 comp rx=0 tx=0).stop 2023-03-09T14:45:30.026 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.004+0000 7ffb56f1bb80 1 -- 172.21.15.38:0/999156504 shutdown_connections 2023-03-09T14:45:30.026 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.004+0000 7ffb56f1bb80 1 --2- 172.21.15.38:0/999156504 >> [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] conn(0x55682f912880 0x55682f90f210 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.026 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.004+0000 7ffb56f1bb80 1 --2- 172.21.15.38:0/999156504 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55682f8a61f0 0x55682f90d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.026 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.004+0000 7ffb23fff700 0 client.15156 ms_handle_reset on v2:172.21.15.181:6838/3902263660 2023-03-09T14:45:30.026 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.004+0000 7ffb56f1bb80 1 --2- 172.21.15.38:0/999156504 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55682f8a49f0 0x55682f8b0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.026 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.004+0000 7ffb56f1bb80 1 --2- 172.21.15.38:0/999156504 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55682f8a5340 0x55682f8b1310 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.027 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.004+0000 7ffb56f1bb80 1 -- 172.21.15.38:0/999156504 >> 172.21.15.38:0/999156504 conn(0x55682f89f500 msgr2=0x55682f8b8cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:30.027 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.004+0000 7ffb56f1bb80 1 -- 172.21.15.38:0/999156504 shutdown_connections 2023-03-09T14:45:30.027 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.004+0000 7ffb56f1bb80 1 -- 172.21.15.38:0/999156504 wait complete. 2023-03-09T14:45:30.027 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.004+0000 7ffb56f1bb80 20 client.15156 trim_cache size 0 max 16384 2023-03-09T14:45:30.027 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2023-03-09T14:45:30.028 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_fsid": "c6733fe2-be87-11ed-9af2-001a4aab830c", 2023-03-09T14:45:30.028 INFO:teuthology.orchestra.run.smithi038.stdout: "whoami": 2, 2023-03-09T14:45:30.028 INFO:teuthology.orchestra.run.smithi038.stdout: "id": 24478, 2023-03-09T14:45:30.028 INFO:teuthology.orchestra.run.smithi038.stdout: "want_state": "up:active", 2023-03-09T14:45:30.028 INFO:teuthology.orchestra.run.smithi038.stdout: "state": "up:active", 2023-03-09T14:45:30.028 INFO:teuthology.orchestra.run.smithi038.stdout: "fs_name": "cephfs", 2023-03-09T14:45:30.028 INFO:teuthology.orchestra.run.smithi038.stdout: "rank_uptime": 51.491591055000001, 2023-03-09T14:45:30.029 INFO:teuthology.orchestra.run.smithi038.stdout: "mdsmap_epoch": 23, 2023-03-09T14:45:30.029 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_epoch": 89, 2023-03-09T14:45:30.029 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_epoch_barrier": 87, 2023-03-09T14:45:30.029 INFO:teuthology.orchestra.run.smithi038.stdout: "uptime": 83.351706832000005 2023-03-09T14:45:30.029 INFO:teuthology.orchestra.run.smithi038.stdout:} 2023-03-09T14:45:30.030 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.005+0000 7ffb4ad92700 1 -- 172.21.15.38:0/2084288664 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7ffb3409f400 msgr2=0x7ffb340a18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:30.030 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.005+0000 7ffb4ad92700 1 --2- 172.21.15.38:0/2084288664 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7ffb3409f400 0x7ffb340a18b0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7ffb3800b5c0 tx=0x7ffb38005fb0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.031 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.005+0000 7ffb4ad92700 1 -- 172.21.15.38:0/2084288664 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffb44105bd0 msgr2=0x7ffb4419d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:30.031 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.005+0000 7ffb4ad92700 1 --2- 172.21.15.38:0/2084288664 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffb44105bd0 0x7ffb4419d1e0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7ffb3c007ee0 tx=0x7ffb3c00d3b0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.031 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.005+0000 7ffb4ad92700 1 -- 172.21.15.38:0/2084288664 shutdown_connections 2023-03-09T14:45:30.031 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.005+0000 7ffb4ad92700 1 --2- 172.21.15.38:0/2084288664 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ffb44105bd0 0x7ffb4419d1e0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.031 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.005+0000 7ffb4ad92700 1 --2- 172.21.15.38:0/2084288664 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7ffb3409f400 0x7ffb340a18b0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.031 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.005+0000 7ffb4ad92700 1 --2- 172.21.15.38:0/2084288664 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ffb44100990 0x7ffb44195c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.031 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.005+0000 7ffb4ad92700 1 --2- 172.21.15.38:0/2084288664 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ffb441032b0 0x7ffb441961b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.031 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.005+0000 7ffb4ad92700 1 -- 172.21.15.38:0/2084288664 >> 172.21.15.38:0/2084288664 conn(0x7ffb440fa3b0 msgr2=0x7ffb441015f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:30.032 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.005+0000 7ffb4ad92700 1 -- 172.21.15.38:0/2084288664 shutdown_connections 2023-03-09T14:45:30.032 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.005+0000 7ffb4ad92700 1 -- 172.21.15.38:0/2084288664 wait complete. 2023-03-09T14:45:30.032 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2023-03-09T14:45:30.159 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.155+0000 7f11427c7700 1 -- 172.21.15.38:0/17979582 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f113c0ff6f0 msgr2=0x7f113c0ffb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:30.159 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.155+0000 7f11427c7700 1 --2- 172.21.15.38:0/17979582 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f113c0ff6f0 0x7f113c0ffb40 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f112c009990 tx=0x7f112c009ca0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.159 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.155+0000 7f11427c7700 1 -- 172.21.15.38:0/17979582 shutdown_connections 2023-03-09T14:45:30.159 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.155+0000 7f11427c7700 1 --2- 172.21.15.38:0/17979582 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f113c100080 0x7f113c106940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.159 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.155+0000 7f11427c7700 1 --2- 172.21.15.38:0/17979582 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f113c0fe500 0x7f113c0fe910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.160 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.155+0000 7f11427c7700 1 --2- 172.21.15.38:0/17979582 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f113c0ff6f0 0x7f113c0ffb40 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.160 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.155+0000 7f11427c7700 1 -- 172.21.15.38:0/17979582 >> 172.21.15.38:0/17979582 conn(0x7f113c0f9b60 msgr2=0x7f113c0fbf70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:30.160 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.155+0000 7f11427c7700 1 -- 172.21.15.38:0/17979582 shutdown_connections 2023-03-09T14:45:30.160 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.155+0000 7f11427c7700 1 -- 172.21.15.38:0/17979582 wait complete. 2023-03-09T14:45:30.161 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.155+0000 7f11427c7700 1 Processor -- start 2023-03-09T14:45:30.161 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.156+0000 7f11427c7700 1 -- start start 2023-03-09T14:45:30.163 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.156+0000 7f11427c7700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f113c0fe500 0x7f113c197e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:30.163 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.156+0000 7f11427c7700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f113c0ff6f0 0x7f113c198350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:30.163 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.156+0000 7f11427c7700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f113c100080 0x7f113c19f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:30.163 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.156+0000 7f11427c7700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f113c19f990 con 0x7f113c0ff6f0 2023-03-09T14:45:30.164 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.156+0000 7f11427c7700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f113c19fad0 con 0x7f113c0fe500 2023-03-09T14:45:30.164 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.156+0000 7f11427c7700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f113c19fc60 con 0x7f113c100080 2023-03-09T14:45:30.164 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.156+0000 7f113b7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f113c0ff6f0 0x7f113c198350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:30.164 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.156+0000 7f113b7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f113c0ff6f0 0x7f113c198350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37834/0 (socket says 172.21.15.38:37834) 2023-03-09T14:45:30.164 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.156+0000 7f113b7fe700 1 -- 172.21.15.38:0/2778819786 learned_addr learned my addr 172.21.15.38:0/2778819786 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:30.164 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.156+0000 7f1140d64700 1 --2- 172.21.15.38:0/2778819786 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f113c100080 0x7f113c19f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:30.164 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.156+0000 7f113bfff700 1 --2- 172.21.15.38:0/2778819786 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f113c0fe500 0x7f113c197e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:30.165 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.156+0000 7f113b7fe700 1 -- 172.21.15.38:0/2778819786 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f113c100080 msgr2=0x7f113c19f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:30.165 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.156+0000 7f113b7fe700 1 --2- 172.21.15.38:0/2778819786 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f113c100080 0x7f113c19f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.165 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.156+0000 7f113b7fe700 1 -- 172.21.15.38:0/2778819786 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f113c0fe500 msgr2=0x7f113c197e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:30.165 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.156+0000 7f113b7fe700 1 --2- 172.21.15.38:0/2778819786 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f113c0fe500 0x7f113c197e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.165 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.156+0000 7f113b7fe700 1 -- 172.21.15.38:0/2778819786 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f112c0095f0 con 0x7f113c0ff6f0 2023-03-09T14:45:30.165 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.156+0000 7f1140d64700 1 --2- 172.21.15.38:0/2778819786 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f113c100080 0x7f113c19f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-09T14:45:30.165 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.156+0000 7f113b7fe700 1 --2- 172.21.15.38:0/2778819786 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f113c0ff6f0 0x7f113c198350 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f112c000c00 tx=0x7f112c00f510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:30.166 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.157+0000 7f11397fa700 1 -- 172.21.15.38:0/2778819786 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f112c01d070 con 0x7f113c0ff6f0 2023-03-09T14:45:30.166 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.157+0000 7f11397fa700 1 -- 172.21.15.38:0/2778819786 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f112c00fc60 con 0x7f113c0ff6f0 2023-03-09T14:45:30.166 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.157+0000 7f11427c7700 1 -- 172.21.15.38:0/2778819786 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f113c109690 con 0x7f113c0ff6f0 2023-03-09T14:45:30.166 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.157+0000 7f11397fa700 1 -- 172.21.15.38:0/2778819786 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f112c017650 con 0x7f113c0ff6f0 2023-03-09T14:45:30.166 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.157+0000 7f11427c7700 1 -- 172.21.15.38:0/2778819786 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f113c109500 con 0x7f113c0ff6f0 2023-03-09T14:45:30.166 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.159+0000 7f11397fa700 1 -- 172.21.15.38:0/2778819786 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 133238+0+0 (secure 0 0 0) 0x7f112c029030 con 0x7f113c0ff6f0 2023-03-09T14:45:30.166 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.159+0000 7f11397fa700 1 --2- 172.21.15.38:0/2778819786 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f112409f400 0x7f11240a18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:30.167 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.159+0000 7f11397fa700 1 -- 172.21.15.38:0/2778819786 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f112c0cb6d0 con 0x7f113c0ff6f0 2023-03-09T14:45:30.167 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.159+0000 7f113bfff700 1 --2- 172.21.15.38:0/2778819786 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f112409f400 0x7f11240a18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:30.167 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.160+0000 7f113bfff700 1 --2- 172.21.15.38:0/2778819786 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f112409f400 0x7f11240a18b0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f113c0ff550 tx=0x7f1130009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:30.172 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.160+0000 7f114e950b80 1 Processor -- start 2023-03-09T14:45:30.172 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.160+0000 7f114e950b80 1 -- start start 2023-03-09T14:45:30.172 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.160+0000 7f114e950b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bb46d859f0 0x55bb46d85e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:30.172 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.161+0000 7f114e950b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb46d86340 0x55bb46d86c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:30.172 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.161+0000 7f114e950b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bb46d871f0 0x55bb46d87660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:30.173 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.161+0000 7f114e950b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb46c6fd50 con 0x55bb46d86340 2023-03-09T14:45:30.173 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.161+0000 7f114e950b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb46cde730 con 0x55bb46d871f0 2023-03-09T14:45:30.173 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.161+0000 7f114e950b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb46cd7c60 con 0x55bb46d859f0 2023-03-09T14:45:30.173 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.161+0000 7f113b7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb46d86340 0x55bb46d86c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:30.173 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.161+0000 7f113b7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb46d86340 0x55bb46d86c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37842/0 (socket says 172.21.15.38:37842) 2023-03-09T14:45:30.173 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.161+0000 7f113b7fe700 1 -- 172.21.15.38:0/555414224 learned_addr learned my addr 172.21.15.38:0/555414224 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:30.174 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.161+0000 7f1140d64700 1 --2- 172.21.15.38:0/555414224 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bb46d871f0 0x55bb46d87660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:30.174 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.161+0000 7f113bfff700 1 --2- 172.21.15.38:0/555414224 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bb46d859f0 0x55bb46d85e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:30.174 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.161+0000 7f113b7fe700 1 -- 172.21.15.38:0/555414224 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bb46d859f0 msgr2=0x55bb46d85e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:45:30.174 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.161+0000 7f113b7fe700 1 --2- 172.21.15.38:0/555414224 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bb46d859f0 0x55bb46d85e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.174 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.161+0000 7f113b7fe700 1 -- 172.21.15.38:0/555414224 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bb46d871f0 msgr2=0x55bb46d87660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:45:30.174 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.161+0000 7f113b7fe700 1 --2- 172.21.15.38:0/555414224 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bb46d871f0 0x55bb46d87660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.174 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.161+0000 7f113b7fe700 1 -- 172.21.15.38:0/555414224 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bb46c9e140 con 0x55bb46d86340 2023-03-09T14:45:30.175 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.161+0000 7f113b7fe700 1 --2- 172.21.15.38:0/555414224 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb46d86340 0x55bb46d86c60 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f112c0c9160 tx=0x7f112c098ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27a0078c581f7e69 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:30.175 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.161+0000 7f11227fc700 1 -- 172.21.15.38:0/555414224 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f112c09f0d0 con 0x55bb46d86340 2023-03-09T14:45:30.175 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.161+0000 7f11227fc700 1 -- 172.21.15.38:0/555414224 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f112c0a15f0 con 0x55bb46d86340 2023-03-09T14:45:30.175 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.161+0000 7f11227fc700 1 -- 172.21.15.38:0/555414224 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f112c0a1160 con 0x55bb46d86340 2023-03-09T14:45:30.175 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.162+0000 7f114e950b80 1 -- 172.21.15.38:0/555414224 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb46d86340 msgr2=0x55bb46d86c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:30.175 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.162+0000 7f114e950b80 1 --2- 172.21.15.38:0/555414224 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb46d86340 0x55bb46d86c60 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f112c0c9160 tx=0x7f112c098ec0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.175 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.162+0000 7f114e950b80 1 -- 172.21.15.38:0/555414224 shutdown_connections 2023-03-09T14:45:30.176 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.162+0000 7f114e950b80 1 --2- 172.21.15.38:0/555414224 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bb46d871f0 0x55bb46d87660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.176 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.162+0000 7f114e950b80 1 --2- 172.21.15.38:0/555414224 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bb46d859f0 0x55bb46d85e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.176 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.162+0000 7f114e950b80 1 --2- 172.21.15.38:0/555414224 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb46d86340 0x55bb46d86c60 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.176 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.162+0000 7f114e950b80 1 -- 172.21.15.38:0/555414224 >> 172.21.15.38:0/555414224 conn(0x55bb46d80500 msgr2=0x55bb46d808f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:30.176 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.162+0000 7f114e950b80 1 -- 172.21.15.38:0/555414224 shutdown_connections 2023-03-09T14:45:30.176 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.162+0000 7f114e950b80 1 -- 172.21.15.38:0/555414224 wait complete. 2023-03-09T14:45:30.176 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.162+0000 7f114e950b80 1 -- start start 2023-03-09T14:45:30.177 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.163+0000 7f114e950b80 1 Processor -- start 2023-03-09T14:45:30.177 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.163+0000 7f114e950b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb46d859f0 0x55bb46d8f400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:30.177 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.163+0000 7f114e950b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bb46d86340 0x55bb46d8f940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:30.177 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.163+0000 7f114e950b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bb46d871f0 0x55bb46d96980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:30.177 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.163+0000 7f114e950b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb46b5b8d0 con 0x55bb46d859f0 2023-03-09T14:45:30.177 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.163+0000 7f114e950b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb46d96ec0 con 0x55bb46d871f0 2023-03-09T14:45:30.177 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.163+0000 7f114e950b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb46d97050 con 0x55bb46d86340 2023-03-09T14:45:30.178 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.163+0000 7f114e950b80 10 client.0 _ll_register_callbacks cb 0x55bb46c7d4d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:45:30.178 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.163+0000 7f113bfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb46d859f0 0x55bb46d8f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:30.178 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.163+0000 7f113bfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb46d859f0 0x55bb46d8f400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37846/0 (socket says 172.21.15.38:37846) 2023-03-09T14:45:30.178 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.163+0000 7f113bfff700 1 -- 172.21.15.38:0/1931012465 learned_addr learned my addr 172.21.15.38:0/1931012465 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:30.178 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.163+0000 7f113b7fe700 1 --2- 172.21.15.38:0/1931012465 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bb46d86340 0x55bb46d8f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:30.178 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.163+0000 7f1140d64700 1 --2- 172.21.15.38:0/1931012465 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bb46d871f0 0x55bb46d96980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:30.178 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.163+0000 7f113bfff700 1 -- 172.21.15.38:0/1931012465 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bb46d86340 msgr2=0x55bb46d8f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:45:30.179 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.163+0000 7f113bfff700 1 --2- 172.21.15.38:0/1931012465 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bb46d86340 0x55bb46d8f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.179 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.163+0000 7f113bfff700 1 -- 172.21.15.38:0/1931012465 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bb46d871f0 msgr2=0x55bb46d96980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:45:30.179 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.163+0000 7f113bfff700 1 --2- 172.21.15.38:0/1931012465 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bb46d871f0 0x55bb46d96980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.179 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.163+0000 7f113bfff700 1 -- 172.21.15.38:0/1931012465 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f113c109500 con 0x55bb46d859f0 2023-03-09T14:45:30.179 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.163+0000 7f1140d64700 1 --2- 172.21.15.38:0/1931012465 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bb46d871f0 0x55bb46d96980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-09T14:45:30.179 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.163+0000 7f113bfff700 1 --2- 172.21.15.38:0/1931012465 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb46d859f0 0x55bb46d8f400 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f1130010d10 tx=0x7f1130004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a9a7d1b5eadeb73 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:30.179 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.164+0000 7f111b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2023-03-09T14:45:30.180 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.164+0000 7f111b7fe700 1 -- 172.21.15.38:0/1931012465 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1130012e70 con 0x55bb46d859f0 2023-03-09T14:45:30.180 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.164+0000 7f111b7fe700 1 -- 172.21.15.38:0/1931012465 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f1130020430 con 0x55bb46d859f0 2023-03-09T14:45:30.180 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.164+0000 7f114e950b80 1 -- 172.21.15.38:0/1931012465 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55bb46ca5140 con 0x55bb46d859f0 2023-03-09T14:45:30.180 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.164+0000 7f111b7fe700 1 -- 172.21.15.38:0/1931012465 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1130020610 con 0x55bb46d859f0 2023-03-09T14:45:30.180 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.164+0000 7f114e950b80 10 client.15168 fetch_fsmap learned FSMap version 23 2023-03-09T14:45:30.180 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.164+0000 7f114e950b80 1 -- 172.21.15.38:0/1931012465 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55bb46aeec10 con 0x55bb46d859f0 2023-03-09T14:45:30.180 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.164+0000 7f111b7fe700 1 -- 172.21.15.38:0/1931012465 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55bb46ca5140 con 0x55bb46d859f0 2023-03-09T14:45:30.181 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.164+0000 7f111b7fe700 1 -- 172.21.15.38:0/1931012465 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f1130026020 con 0x55bb46d859f0 2023-03-09T14:45:30.181 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.164+0000 7f114e950b80 10 client.15168 fetch_fsmap finished waiting for FSMap version 23 2023-03-09T14:45:30.182 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.164+0000 7f114e950b80 10 client.15168 resolve_mds: resolved name 'h' to daemon mds.h 2023-03-09T14:45:30.182 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.164+0000 7f114e950b80 20 client.15168 populate_metadata read hostname 'smithi038' 2023-03-09T14:45:30.182 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.164+0000 7f114e950b80 1 --2- 172.21.15.38:0/1931012465 >> [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] conn(0x55bb46d9a980 0x55bb46d99db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:30.182 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.164+0000 7f114e950b80 4 client.15168 mds_command: new command op to 24494 tid=0[{"prefix": "get_command_descriptions"}] 2023-03-09T14:45:30.182 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.164+0000 7f114e950b80 1 -- 172.21.15.38:0/1931012465 --> [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bb46b5bf10 con 0x55bb46d9a980 2023-03-09T14:45:30.182 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.165+0000 7f113b7fe700 1 --2- 172.21.15.38:0/1931012465 >> [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] conn(0x55bb46d9a980 0x55bb46d99db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:30.183 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.165+0000 7f111b7fe700 10 client.15168 ms_handle_connect on v2:172.21.15.122:6836/2210214596 2023-03-09T14:45:30.183 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.165+0000 7f113b7fe700 1 --2- 172.21.15.38:0/1931012465 >> [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] conn(0x55bb46d9a980 0x55bb46d99db0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=bfc308a93bd20f63 server_cookie=a50a077356b8b6b8 in_seq=0 out_seq=0 2023-03-09T14:45:30.183 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.166+0000 7f111b7fe700 1 -- 172.21.15.38:0/1931012465 <== mds.1 v2:172.21.15.122:6836/2210214596 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18278 (crc 0 0 0) 0x55bb46b5bf10 con 0x55bb46d9a980 2023-03-09T14:45:30.183 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.166+0000 7f111b7fe700 10 client.15168 handle_command_reply: tid=0 2023-03-09T14:45:30.183 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.166+0000 7f114e950b80 1 client.15168 shutdown 2023-03-09T14:45:30.183 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.166+0000 7f114e950b80 1 -- 172.21.15.38:0/1931012465 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb46d859f0 msgr2=0x55bb46d8f400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:30.183 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.166+0000 7f114e950b80 1 --2- 172.21.15.38:0/1931012465 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb46d859f0 0x55bb46d8f400 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f1130010d10 tx=0x7f1130004750 comp rx=0 tx=0).stop 2023-03-09T14:45:30.184 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.166+0000 7f114e950b80 1 -- 172.21.15.38:0/1931012465 shutdown_connections 2023-03-09T14:45:30.184 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.166+0000 7f114e950b80 1 --2- 172.21.15.38:0/1931012465 >> [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] conn(0x55bb46d9a980 0x55bb46d99db0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.184 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.166+0000 7f114e950b80 1 --2- 172.21.15.38:0/1931012465 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb46d859f0 0x55bb46d8f400 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.184 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.166+0000 7f114e950b80 1 --2- 172.21.15.38:0/1931012465 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bb46d871f0 0x55bb46d96980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.184 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.166+0000 7f114e950b80 1 --2- 172.21.15.38:0/1931012465 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bb46d86340 0x55bb46d8f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.184 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.166+0000 7f111b7fe700 0 client.15168 ms_handle_reset on v2:172.21.15.122:6836/2210214596 2023-03-09T14:45:30.184 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.166+0000 7f114e950b80 1 -- 172.21.15.38:0/1931012465 >> 172.21.15.38:0/1931012465 conn(0x55bb46d80500 msgr2=0x55bb46d7fbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:30.185 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.166+0000 7f114e950b80 1 -- 172.21.15.38:0/1931012465 shutdown_connections 2023-03-09T14:45:30.185 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.166+0000 7f114e950b80 1 -- 172.21.15.38:0/1931012465 wait complete. 2023-03-09T14:45:30.185 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.166+0000 7f114e950b80 20 client.15168 trim_cache size 0 max 16384 2023-03-09T14:45:30.188 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.184+0000 7f114e950b80 1 Processor -- start 2023-03-09T14:45:30.188 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.184+0000 7f114e950b80 1 -- start start 2023-03-09T14:45:30.188 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.184+0000 7f114e950b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb46d859f0 0x55bb46d7fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:30.189 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.184+0000 7f114e950b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bb46d86340 0x55bb46d96910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:30.189 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.184+0000 7f114e950b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bb46d871f0 0x55bb46d99db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:30.189 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.184+0000 7f114e950b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb46d86af0 con 0x55bb46d859f0 2023-03-09T14:45:30.189 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.184+0000 7f114e950b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb46c77530 con 0x55bb46d86340 2023-03-09T14:45:30.189 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.184+0000 7f114e950b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb46c974b0 con 0x55bb46d871f0 2023-03-09T14:45:30.189 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.184+0000 7f113bfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb46d859f0 0x55bb46d7fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:30.189 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.184+0000 7f113bfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb46d859f0 0x55bb46d7fbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37854/0 (socket says 172.21.15.38:37854) 2023-03-09T14:45:30.190 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.184+0000 7f113bfff700 1 -- 172.21.15.38:0/1003501464 learned_addr learned my addr 172.21.15.38:0/1003501464 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:30.190 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.184+0000 7f1140d64700 1 --2- 172.21.15.38:0/1003501464 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bb46d871f0 0x55bb46d99db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:30.190 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.184+0000 7f113b7fe700 1 --2- 172.21.15.38:0/1003501464 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bb46d86340 0x55bb46d96910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:30.190 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.185+0000 7f113bfff700 1 -- 172.21.15.38:0/1003501464 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bb46d871f0 msgr2=0x55bb46d99db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:45:30.190 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.185+0000 7f113bfff700 1 --2- 172.21.15.38:0/1003501464 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bb46d871f0 0x55bb46d99db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.190 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.185+0000 7f113bfff700 1 -- 172.21.15.38:0/1003501464 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bb46d86340 msgr2=0x55bb46d96910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:45:30.190 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.185+0000 7f113bfff700 1 --2- 172.21.15.38:0/1003501464 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bb46d86340 0x55bb46d96910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.191 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.185+0000 7f113bfff700 1 -- 172.21.15.38:0/1003501464 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bb46b05d90 con 0x55bb46d859f0 2023-03-09T14:45:30.191 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.185+0000 7f113b7fe700 1 --2- 172.21.15.38:0/1003501464 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bb46d86340 0x55bb46d96910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-09T14:45:30.191 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.185+0000 7f113bfff700 1 --2- 172.21.15.38:0/1003501464 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb46d859f0 0x55bb46d7fbd0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f1130010780 tx=0x7f11300092d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=238d3ab60bdec260 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:30.191 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.185+0000 7f1122ffd700 1 -- 172.21.15.38:0/1003501464 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11300115e0 con 0x55bb46d859f0 2023-03-09T14:45:30.191 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.185+0000 7f1122ffd700 1 -- 172.21.15.38:0/1003501464 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1130011c20 con 0x55bb46d859f0 2023-03-09T14:45:30.192 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.185+0000 7f1122ffd700 1 -- 172.21.15.38:0/1003501464 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11300036a0 con 0x55bb46d859f0 2023-03-09T14:45:30.195 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.185+0000 7f114e950b80 1 -- 172.21.15.38:0/1003501464 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb46d859f0 msgr2=0x55bb46d7fbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:30.195 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.185+0000 7f114e950b80 1 --2- 172.21.15.38:0/1003501464 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb46d859f0 0x55bb46d7fbd0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f1130010780 tx=0x7f11300092d0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.195 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.186+0000 7f114e950b80 1 -- 172.21.15.38:0/1003501464 shutdown_connections 2023-03-09T14:45:30.196 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.186+0000 7f114e950b80 1 --2- 172.21.15.38:0/1003501464 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb46d859f0 0x55bb46d7fbd0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.196 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.186+0000 7f114e950b80 1 --2- 172.21.15.38:0/1003501464 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bb46d871f0 0x55bb46d99db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.196 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.186+0000 7f114e950b80 1 --2- 172.21.15.38:0/1003501464 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bb46d86340 0x55bb46d96910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.196 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.186+0000 7f114e950b80 1 -- 172.21.15.38:0/1003501464 >> 172.21.15.38:0/1003501464 conn(0x55bb46d80500 msgr2=0x55bb46d97430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:30.196 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.186+0000 7f114e950b80 1 -- 172.21.15.38:0/1003501464 shutdown_connections 2023-03-09T14:45:30.196 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.186+0000 7f114e950b80 1 -- 172.21.15.38:0/1003501464 wait complete. 2023-03-09T14:45:30.196 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.186+0000 7f114e950b80 1 -- start start 2023-03-09T14:45:30.197 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.186+0000 7f114e950b80 1 Processor -- start 2023-03-09T14:45:30.197 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.186+0000 7f114e950b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bb46d859f0 0x55bb46d91c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:30.197 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.186+0000 7f114e950b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb46d86340 0x55bb46d92190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:30.197 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.186+0000 7f114e950b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bb46d871f0 0x55bb46d951b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:30.197 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.186+0000 7f114e950b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb46d956f0 con 0x55bb46d86340 2023-03-09T14:45:30.197 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.186+0000 7f114e950b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb46d95860 con 0x55bb46d871f0 2023-03-09T14:45:30.197 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.186+0000 7f114e950b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bb46d95a20 con 0x55bb46d859f0 2023-03-09T14:45:30.198 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.186+0000 7f114e950b80 10 client.0 _ll_register_callbacks cb 0x55bb46c7d4d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:45:30.198 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.186+0000 7f113b7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb46d86340 0x55bb46d92190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:30.198 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.186+0000 7f113b7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb46d86340 0x55bb46d92190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37862/0 (socket says 172.21.15.38:37862) 2023-03-09T14:45:30.198 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.186+0000 7f113b7fe700 1 -- 172.21.15.38:0/1540280488 learned_addr learned my addr 172.21.15.38:0/1540280488 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:30.198 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.187+0000 7f1140d64700 1 --2- 172.21.15.38:0/1540280488 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bb46d871f0 0x55bb46d951b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:30.198 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.187+0000 7f113bfff700 1 --2- 172.21.15.38:0/1540280488 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bb46d859f0 0x55bb46d91c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:30.198 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.187+0000 7f113b7fe700 1 -- 172.21.15.38:0/1540280488 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bb46d859f0 msgr2=0x55bb46d91c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:45:30.199 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.187+0000 7f113b7fe700 1 --2- 172.21.15.38:0/1540280488 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bb46d859f0 0x55bb46d91c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.199 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.187+0000 7f113b7fe700 1 -- 172.21.15.38:0/1540280488 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bb46d871f0 msgr2=0x55bb46d951b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:45:30.199 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.187+0000 7f113b7fe700 1 --2- 172.21.15.38:0/1540280488 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bb46d871f0 0x55bb46d951b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.199 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.187+0000 7f113b7fe700 1 -- 172.21.15.38:0/1540280488 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bb46aeec10 con 0x55bb46d86340 2023-03-09T14:45:30.199 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.187+0000 7f113b7fe700 1 --2- 172.21.15.38:0/1540280488 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb46d86340 0x55bb46d92190 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f112c006010 tx=0x7f112c009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdd075528a4f397b server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:30.199 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.187+0000 7f111b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2023-03-09T14:45:30.199 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.187+0000 7f111b7fe700 1 -- 172.21.15.38:0/1540280488 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f112c09d040 con 0x55bb46d86340 2023-03-09T14:45:30.200 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.187+0000 7f111b7fe700 1 -- 172.21.15.38:0/1540280488 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f112c09d1a0 con 0x55bb46d86340 2023-03-09T14:45:30.200 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.187+0000 7f114e950b80 1 -- 172.21.15.38:0/1540280488 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55bb46c95160 con 0x55bb46d86340 2023-03-09T14:45:30.200 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.187+0000 7f111b7fe700 1 -- 172.21.15.38:0/1540280488 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f112c098980 con 0x55bb46d86340 2023-03-09T14:45:30.200 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.187+0000 7f114e950b80 10 client.15174 fetch_fsmap learned FSMap version 23 2023-03-09T14:45:30.200 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.187+0000 7f114e950b80 1 -- 172.21.15.38:0/1540280488 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55bb46b04260 con 0x55bb46d86340 2023-03-09T14:45:30.200 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.188+0000 7f111b7fe700 1 -- 172.21.15.38:0/1540280488 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55bb46c95160 con 0x55bb46d86340 2023-03-09T14:45:30.200 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.188+0000 7f111b7fe700 1 -- 172.21.15.38:0/1540280488 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f112c0b7020 con 0x55bb46d86340 2023-03-09T14:45:30.201 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.188+0000 7f114e950b80 10 client.15174 fetch_fsmap finished waiting for FSMap version 23 2023-03-09T14:45:30.201 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.188+0000 7f114e950b80 10 client.15174 resolve_mds: resolved name 'h' to daemon mds.h 2023-03-09T14:45:30.201 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.188+0000 7f114e950b80 20 client.15174 populate_metadata read hostname 'smithi038' 2023-03-09T14:45:30.201 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.188+0000 7f114e950b80 1 --2- 172.21.15.38:0/1540280488 >> [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] conn(0x55bb46df3a40 0x55bb46df0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:30.201 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.188+0000 7f114e950b80 4 client.15174 mds_command: new command op to 24494 tid=0[{"prefix": "status"}] 2023-03-09T14:45:30.201 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.188+0000 7f114e950b80 1 -- 172.21.15.38:0/1540280488 --> [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55bb46c7f170 con 0x55bb46df3a40 2023-03-09T14:45:30.201 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.188+0000 7f113bfff700 1 --2- 172.21.15.38:0/1540280488 >> [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] conn(0x55bb46df3a40 0x55bb46df0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:30.202 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.188+0000 7f111b7fe700 10 client.15174 ms_handle_connect on v2:172.21.15.122:6836/2210214596 2023-03-09T14:45:30.202 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.188+0000 7f113bfff700 1 --2- 172.21.15.38:0/1540280488 >> [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] conn(0x55bb46df3a40 0x55bb46df0830 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=39d7a91313dc571f server_cookie=d549a0a654810ee6 in_seq=0 out_seq=0 2023-03-09T14:45:30.202 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.189+0000 7f111b7fe700 1 -- 172.21.15.38:0/1540280488 <== mds.1 v2:172.21.15.122:6836/2210214596 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x55bb46c7f170 con 0x55bb46df3a40 2023-03-09T14:45:30.202 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.189+0000 7f111b7fe700 10 client.15174 handle_command_reply: tid=0 2023-03-09T14:45:30.202 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.189+0000 7f114e950b80 1 client.15174 shutdown 2023-03-09T14:45:30.202 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.189+0000 7f114e950b80 1 -- 172.21.15.38:0/1540280488 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb46d86340 msgr2=0x55bb46d92190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:30.202 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.189+0000 7f114e950b80 1 --2- 172.21.15.38:0/1540280488 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb46d86340 0x55bb46d92190 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f112c006010 tx=0x7f112c009310 comp rx=0 tx=0).stop 2023-03-09T14:45:30.203 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.189+0000 7f114e950b80 1 -- 172.21.15.38:0/1540280488 shutdown_connections 2023-03-09T14:45:30.203 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.189+0000 7f114e950b80 1 --2- 172.21.15.38:0/1540280488 >> [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] conn(0x55bb46df3a40 0x55bb46df0830 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.203 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.189+0000 7f114e950b80 1 --2- 172.21.15.38:0/1540280488 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bb46d871f0 0x55bb46d951b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.203 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.189+0000 7f114e950b80 1 --2- 172.21.15.38:0/1540280488 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bb46d859f0 0x55bb46d91c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.203 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.189+0000 7f114e950b80 1 --2- 172.21.15.38:0/1540280488 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bb46d86340 0x55bb46d92190 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.203 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.189+0000 7f111b7fe700 0 client.15174 ms_handle_reset on v2:172.21.15.122:6836/2210214596 2023-03-09T14:45:30.203 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.189+0000 7f114e950b80 1 -- 172.21.15.38:0/1540280488 >> 172.21.15.38:0/1540280488 conn(0x55bb46d80500 msgr2=0x55bb46d97430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:30.204 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.189+0000 7f114e950b80 1 -- 172.21.15.38:0/1540280488 shutdown_connections 2023-03-09T14:45:30.204 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.189+0000 7f114e950b80 1 -- 172.21.15.38:0/1540280488 wait complete. 2023-03-09T14:45:30.204 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.189+0000 7f114e950b80 20 client.15174 trim_cache size 0 max 16384 2023-03-09T14:45:30.204 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2023-03-09T14:45:30.205 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_fsid": "c6733fe2-be87-11ed-9af2-001a4aab830c", 2023-03-09T14:45:30.205 INFO:teuthology.orchestra.run.smithi038.stdout: "whoami": 1, 2023-03-09T14:45:30.205 INFO:teuthology.orchestra.run.smithi038.stdout: "id": 24494, 2023-03-09T14:45:30.205 INFO:teuthology.orchestra.run.smithi038.stdout: "want_state": "up:active", 2023-03-09T14:45:30.205 INFO:teuthology.orchestra.run.smithi038.stdout: "state": "up:active", 2023-03-09T14:45:30.205 INFO:teuthology.orchestra.run.smithi038.stdout: "fs_name": "cephfs", 2023-03-09T14:45:30.205 INFO:teuthology.orchestra.run.smithi038.stdout: "rank_uptime": 52.697851241999999, 2023-03-09T14:45:30.206 INFO:teuthology.orchestra.run.smithi038.stdout: "mdsmap_epoch": 23, 2023-03-09T14:45:30.206 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_epoch": 89, 2023-03-09T14:45:30.206 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_epoch_barrier": 87, 2023-03-09T14:45:30.206 INFO:teuthology.orchestra.run.smithi038.stdout: "uptime": 85.785091648000005 2023-03-09T14:45:30.206 INFO:teuthology.orchestra.run.smithi038.stdout:} 2023-03-09T14:45:30.207 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.189+0000 7f11427c7700 1 -- 172.21.15.38:0/2778819786 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f112409f400 msgr2=0x7f11240a18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:30.207 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.189+0000 7f11427c7700 1 --2- 172.21.15.38:0/2778819786 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f112409f400 0x7f11240a18b0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f113c0ff550 tx=0x7f1130009260 comp rx=0 tx=0).stop 2023-03-09T14:45:30.207 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.189+0000 7f11427c7700 1 -- 172.21.15.38:0/2778819786 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f113c0ff6f0 msgr2=0x7f113c198350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:30.207 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.189+0000 7f11427c7700 1 --2- 172.21.15.38:0/2778819786 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f113c0ff6f0 0x7f113c198350 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f112c000c00 tx=0x7f112c00f510 comp rx=0 tx=0).stop 2023-03-09T14:45:30.208 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.190+0000 7f11427c7700 1 -- 172.21.15.38:0/2778819786 shutdown_connections 2023-03-09T14:45:30.208 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.190+0000 7f11427c7700 1 --2- 172.21.15.38:0/2778819786 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f112409f400 0x7f11240a18b0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.208 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.190+0000 7f11427c7700 1 --2- 172.21.15.38:0/2778819786 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f113c100080 0x7f113c19f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.208 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.190+0000 7f11427c7700 1 --2- 172.21.15.38:0/2778819786 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f113c0fe500 0x7f113c197e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.208 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.190+0000 7f11427c7700 1 --2- 172.21.15.38:0/2778819786 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f113c0ff6f0 0x7f113c198350 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.208 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.190+0000 7f11427c7700 1 -- 172.21.15.38:0/2778819786 >> 172.21.15.38:0/2778819786 conn(0x7f113c0f9b60 msgr2=0x7f113c102bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:30.208 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.190+0000 7f11427c7700 1 -- 172.21.15.38:0/2778819786 shutdown_connections 2023-03-09T14:45:30.209 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.190+0000 7f11427c7700 1 -- 172.21.15.38:0/2778819786 wait complete. 2023-03-09T14:45:30.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:29 smithi038 ceph-mon[92422]: pgmap v296: 129 pgs: 129 active+clean; 636 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s; 112 B/s, 0 objects/s recovering 2023-03-09T14:45:30.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:29 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/4062222853' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-03-09T14:45:30.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:29 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/3642833665' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-03-09T14:45:30.209 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:29 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/1335451159' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-03-09T14:45:30.210 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-03-09T14:45:30.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:29 smithi122 ceph-mon[93869]: pgmap v296: 129 pgs: 129 active+clean; 636 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s; 112 B/s, 0 objects/s recovering 2023-03-09T14:45:30.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:29 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/4062222853' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-03-09T14:45:30.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:29 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/3642833665' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-03-09T14:45:30.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:29 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/1335451159' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-03-09T14:45:30.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:29 smithi181 ceph-mon[94189]: pgmap v296: 129 pgs: 129 active+clean; 636 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s; 112 B/s, 0 objects/s recovering 2023-03-09T14:45:30.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:29 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/4062222853' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-03-09T14:45:30.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:29 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/3642833665' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-03-09T14:45:30.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:29 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/1335451159' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-03-09T14:45:30.339 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.334+0000 7f107c319700 1 -- 172.21.15.38:0/3077424671 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10740fed90 msgr2=0x7f10740ff1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:30.340 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.334+0000 7f107c319700 1 --2- 172.21.15.38:0/3077424671 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10740fed90 0x7f10740ff1a0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f1070009990 tx=0x7f1070009ca0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.340 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.334+0000 7f107c319700 1 -- 172.21.15.38:0/3077424671 shutdown_connections 2023-03-09T14:45:30.340 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.334+0000 7f107c319700 1 --2- 172.21.15.38:0/3077424671 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10740fed90 0x7f10740ff1a0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.340 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.334+0000 7f107c319700 1 --2- 172.21.15.38:0/3077424671 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1074100910 0x7f10741071d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.341 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.334+0000 7f107c319700 1 --2- 172.21.15.38:0/3077424671 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f10740fff80 0x7f10741003d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.341 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.334+0000 7f107c319700 1 -- 172.21.15.38:0/3077424671 >> 172.21.15.38:0/3077424671 conn(0x7f10740fa3b0 msgr2=0x7f10740fc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:30.341 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.334+0000 7f107c319700 1 -- 172.21.15.38:0/3077424671 shutdown_connections 2023-03-09T14:45:30.341 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.335+0000 7f107c319700 1 -- 172.21.15.38:0/3077424671 wait complete. 2023-03-09T14:45:30.342 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.335+0000 7f107c319700 1 Processor -- start 2023-03-09T14:45:30.342 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.335+0000 7f107c319700 1 -- start start 2023-03-09T14:45:30.342 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.335+0000 7f107c319700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10740fed90 0x7f1074194b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:30.342 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.335+0000 7f107c319700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f10740fff80 0x7f10741950d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:30.343 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.335+0000 7f107c319700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1074100910 0x7f107419c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:30.343 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.335+0000 7f107c319700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f107419c710 con 0x7f10740fed90 2023-03-09T14:45:30.343 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.335+0000 7f107c319700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f107419c850 con 0x7f10740fff80 2023-03-09T14:45:30.343 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.335+0000 7f107c319700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f107419c9e0 con 0x7f1074100910 2023-03-09T14:45:30.344 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.335+0000 7f107a0b5700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10740fed90 0x7f1074194b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:30.344 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.335+0000 7f107a0b5700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10740fed90 0x7f1074194b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37874/0 (socket says 172.21.15.38:37874) 2023-03-09T14:45:30.344 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.335+0000 7f107a0b5700 1 -- 172.21.15.38:0/3758613033 learned_addr learned my addr 172.21.15.38:0/3758613033 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:30.344 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.336+0000 7f107a8b6700 1 --2- 172.21.15.38:0/3758613033 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1074100910 0x7f107419c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:30.345 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.336+0000 7f10798b4700 1 --2- 172.21.15.38:0/3758613033 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f10740fff80 0x7f10741950d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:30.345 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.336+0000 7f107a0b5700 1 -- 172.21.15.38:0/3758613033 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1074100910 msgr2=0x7f107419c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:30.345 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.336+0000 7f107a0b5700 1 --2- 172.21.15.38:0/3758613033 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1074100910 0x7f107419c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.345 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.336+0000 7f107a0b5700 1 -- 172.21.15.38:0/3758613033 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f10740fff80 msgr2=0x7f10741950d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:30.345 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.336+0000 7f107a0b5700 1 --2- 172.21.15.38:0/3758613033 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f10740fff80 0x7f10741950d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.346 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.336+0000 7f107a0b5700 1 -- 172.21.15.38:0/3758613033 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10700095f0 con 0x7f10740fed90 2023-03-09T14:45:30.346 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.336+0000 7f107a0b5700 1 --2- 172.21.15.38:0/3758613033 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10740fed90 0x7f1074194b90 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f1070000c00 tx=0x7f10700047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:30.346 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.336+0000 7f106b7fe700 1 -- 172.21.15.38:0/3758613033 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f107001d070 con 0x7f10740fed90 2023-03-09T14:45:30.346 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.336+0000 7f106b7fe700 1 -- 172.21.15.38:0/3758613033 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1070004cf0 con 0x7f10740fed90 2023-03-09T14:45:30.346 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.336+0000 7f107c319700 1 -- 172.21.15.38:0/3758613033 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f107410a6f0 con 0x7f10740fed90 2023-03-09T14:45:30.347 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.336+0000 7f106b7fe700 1 -- 172.21.15.38:0/3758613033 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1070005020 con 0x7f10740fed90 2023-03-09T14:45:30.347 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.336+0000 7f107c319700 1 -- 172.21.15.38:0/3758613033 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f107410a560 con 0x7f10740fed90 2023-03-09T14:45:30.352 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.339+0000 7f106b7fe700 1 -- 172.21.15.38:0/3758613033 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 133238+0+0 (secure 0 0 0) 0x7f107000c040 con 0x7f10740fed90 2023-03-09T14:45:30.352 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.339+0000 7f106b7fe700 1 --2- 172.21.15.38:0/3758613033 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f106009f400 0x7f10600a18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:30.353 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.340+0000 7f106b7fe700 1 -- 172.21.15.38:0/3758613033 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f10700cb0d0 con 0x7f10740fed90 2023-03-09T14:45:30.353 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.340+0000 7f10798b4700 1 --2- 172.21.15.38:0/3758613033 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f106009f400 0x7f10600a18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:30.353 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.340+0000 7f10798b4700 1 --2- 172.21.15.38:0/3758613033 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f106009f400 0x7f10600a18b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f1064005950 tx=0x7f106400b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:30.353 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.340+0000 7f10884a2b80 1 Processor -- start 2023-03-09T14:45:30.353 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.340+0000 7f10884a2b80 1 -- start start 2023-03-09T14:45:30.354 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.340+0000 7f10884a2b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55eb0c3029f0 0x55eb0c302e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:30.354 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.340+0000 7f10884a2b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eb0c303340 0x55eb0c303c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:30.354 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.340+0000 7f10884a2b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55eb0c3041f0 0x55eb0c304660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:30.354 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.340+0000 7f10884a2b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb0c1ecd50 con 0x55eb0c3029f0 2023-03-09T14:45:30.354 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.340+0000 7f10884a2b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb0c25b730 con 0x55eb0c3041f0 2023-03-09T14:45:30.354 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.340+0000 7f10884a2b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb0c254c60 con 0x55eb0c303340 2023-03-09T14:45:30.354 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.341+0000 7f107a0b5700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55eb0c3029f0 0x55eb0c302e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:30.355 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.341+0000 7f107a0b5700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55eb0c3029f0 0x55eb0c302e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37882/0 (socket says 172.21.15.38:37882) 2023-03-09T14:45:30.355 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.341+0000 7f107a0b5700 1 -- 172.21.15.38:0/995495235 learned_addr learned my addr 172.21.15.38:0/995495235 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:30.355 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.341+0000 7f107a0b5700 1 -- 172.21.15.38:0/995495235 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eb0c303340 msgr2=0x55eb0c303c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:45:30.355 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.341+0000 7f10798b4700 1 --2- 172.21.15.38:0/995495235 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eb0c303340 0x55eb0c303c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:30.355 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.341+0000 7f107a8b6700 1 --2- 172.21.15.38:0/995495235 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55eb0c3041f0 0x55eb0c304660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:30.355 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.341+0000 7f107a0b5700 1 --2- 172.21.15.38:0/995495235 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eb0c303340 0x55eb0c303c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.355 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.341+0000 7f107a0b5700 1 -- 172.21.15.38:0/995495235 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55eb0c3041f0 msgr2=0x55eb0c304660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:45:30.355 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.341+0000 7f107a0b5700 1 --2- 172.21.15.38:0/995495235 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55eb0c3041f0 0x55eb0c304660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.356 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.341+0000 7f107a0b5700 1 -- 172.21.15.38:0/995495235 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55eb0c21b140 con 0x55eb0c3029f0 2023-03-09T14:45:30.356 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.341+0000 7f10798b4700 1 --2- 172.21.15.38:0/995495235 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eb0c303340 0x55eb0c303c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-09T14:45:30.356 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.341+0000 7f107a0b5700 1 --2- 172.21.15.38:0/995495235 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55eb0c3029f0 0x55eb0c302e00 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f10700a0ac0 tx=0x7f1070005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3180982d1d0bba15 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:30.356 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.341+0000 7f1068daf700 1 -- 172.21.15.38:0/995495235 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1070098e90 con 0x55eb0c3029f0 2023-03-09T14:45:30.356 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.341+0000 7f1068daf700 1 -- 172.21.15.38:0/995495235 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f10700a0ff0 con 0x55eb0c3029f0 2023-03-09T14:45:30.357 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.341+0000 7f1068daf700 1 -- 172.21.15.38:0/995495235 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1070098580 con 0x55eb0c3029f0 2023-03-09T14:45:30.357 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.342+0000 7f10884a2b80 1 -- 172.21.15.38:0/995495235 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55eb0c3029f0 msgr2=0x55eb0c302e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:30.357 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.342+0000 7f10884a2b80 1 --2- 172.21.15.38:0/995495235 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55eb0c3029f0 0x55eb0c302e00 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f10700a0ac0 tx=0x7f1070005ef0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.357 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.342+0000 7f10884a2b80 1 -- 172.21.15.38:0/995495235 shutdown_connections 2023-03-09T14:45:30.357 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.342+0000 7f10884a2b80 1 --2- 172.21.15.38:0/995495235 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55eb0c3029f0 0x55eb0c302e00 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.357 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.342+0000 7f10884a2b80 1 --2- 172.21.15.38:0/995495235 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55eb0c3041f0 0x55eb0c304660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.357 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.342+0000 7f10884a2b80 1 --2- 172.21.15.38:0/995495235 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eb0c303340 0x55eb0c303c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.358 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.342+0000 7f10884a2b80 1 -- 172.21.15.38:0/995495235 >> 172.21.15.38:0/995495235 conn(0x55eb0c2fd500 msgr2=0x55eb0c2fd8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:30.358 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.342+0000 7f10884a2b80 1 -- 172.21.15.38:0/995495235 shutdown_connections 2023-03-09T14:45:30.358 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.342+0000 7f10884a2b80 1 -- 172.21.15.38:0/995495235 wait complete. 2023-03-09T14:45:30.358 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.342+0000 7f10884a2b80 1 -- start start 2023-03-09T14:45:30.358 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.343+0000 7f10884a2b80 1 Processor -- start 2023-03-09T14:45:30.358 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.343+0000 7f10884a2b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55eb0c3029f0 0x55eb0c30c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:30.358 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.343+0000 7f10884a2b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eb0c303340 0x55eb0c30c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:30.359 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.343+0000 7f10884a2b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55eb0c3041f0 0x55eb0c313930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:30.359 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.343+0000 7f10884a2b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb0c0d88d0 con 0x55eb0c3029f0 2023-03-09T14:45:30.359 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.343+0000 7f10884a2b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb0c313e70 con 0x55eb0c3041f0 2023-03-09T14:45:30.359 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.343+0000 7f10884a2b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb0c314000 con 0x55eb0c303340 2023-03-09T14:45:30.359 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.343+0000 7f10884a2b80 10 client.0 _ll_register_callbacks cb 0x55eb0c1fa4d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:45:30.359 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.343+0000 7f107a0b5700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55eb0c3029f0 0x55eb0c30c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:30.359 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.343+0000 7f107a0b5700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55eb0c3029f0 0x55eb0c30c3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37888/0 (socket says 172.21.15.38:37888) 2023-03-09T14:45:30.359 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.343+0000 7f107a0b5700 1 -- 172.21.15.38:0/64832524 learned_addr learned my addr 172.21.15.38:0/64832524 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:30.360 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.343+0000 7f107a8b6700 1 --2- 172.21.15.38:0/64832524 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55eb0c3041f0 0x55eb0c313930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:30.360 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.343+0000 7f10798b4700 1 --2- 172.21.15.38:0/64832524 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eb0c303340 0x55eb0c30c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:30.360 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.343+0000 7f107a0b5700 1 -- 172.21.15.38:0/64832524 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eb0c303340 msgr2=0x55eb0c30c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:45:30.360 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.343+0000 7f107a0b5700 1 --2- 172.21.15.38:0/64832524 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eb0c303340 0x55eb0c30c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.360 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.343+0000 7f107a0b5700 1 -- 172.21.15.38:0/64832524 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55eb0c3041f0 msgr2=0x55eb0c313930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:45:30.360 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.343+0000 7f107a0b5700 1 --2- 172.21.15.38:0/64832524 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55eb0c3041f0 0x55eb0c313930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.361 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.343+0000 7f107a0b5700 1 -- 172.21.15.38:0/64832524 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f107410a560 con 0x55eb0c3029f0 2023-03-09T14:45:30.361 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.343+0000 7f107a8b6700 1 --2- 172.21.15.38:0/64832524 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55eb0c3041f0 0x55eb0c313930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:45:30.361 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.343+0000 7f10798b4700 1 --2- 172.21.15.38:0/64832524 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eb0c303340 0x55eb0c30c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-09T14:45:30.361 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.343+0000 7f107a0b5700 1 --2- 172.21.15.38:0/64832524 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55eb0c3029f0 0x55eb0c30c3b0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f1070098e20 tx=0x7f107000b540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11f8d7dd0864d0f8 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:30.361 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.343+0000 7f105dffb700 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2023-03-09T14:45:30.361 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.344+0000 7f105dffb700 1 -- 172.21.15.38:0/64832524 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10700a0ff0 con 0x55eb0c3029f0 2023-03-09T14:45:30.361 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.344+0000 7f105dffb700 1 -- 172.21.15.38:0/64832524 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f1070098e90 con 0x55eb0c3029f0 2023-03-09T14:45:30.362 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.344+0000 7f10884a2b80 1 -- 172.21.15.38:0/64832524 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55eb0c222140 con 0x55eb0c3029f0 2023-03-09T14:45:30.362 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.344+0000 7f105dffb700 1 -- 172.21.15.38:0/64832524 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1070004a40 con 0x55eb0c3029f0 2023-03-09T14:45:30.362 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.344+0000 7f10884a2b80 10 client.15186 fetch_fsmap learned FSMap version 23 2023-03-09T14:45:30.362 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.344+0000 7f10884a2b80 1 -- 172.21.15.38:0/64832524 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55eb0c06bc10 con 0x55eb0c3029f0 2023-03-09T14:45:30.362 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.344+0000 7f105dffb700 1 -- 172.21.15.38:0/64832524 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55eb0c222140 con 0x55eb0c3029f0 2023-03-09T14:45:30.362 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.344+0000 7f105dffb700 1 -- 172.21.15.38:0/64832524 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f1070099c70 con 0x55eb0c3029f0 2023-03-09T14:45:30.362 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.344+0000 7f10884a2b80 10 client.15186 fetch_fsmap finished waiting for FSMap version 23 2023-03-09T14:45:30.363 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.344+0000 7f10884a2b80 10 client.15186 resolve_mds: resolved name 'b' to daemon mds.b 2023-03-09T14:45:30.363 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.344+0000 7f10884a2b80 20 client.15186 populate_metadata read hostname 'smithi038' 2023-03-09T14:45:30.363 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.344+0000 7f10884a2b80 1 --2- 172.21.15.38:0/64832524 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55eb0c317340 0x55eb0c316a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:30.363 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.344+0000 7f10884a2b80 4 client.15186 mds_command: new command op to 24530 tid=0[{"prefix": "get_command_descriptions"}] 2023-03-09T14:45:30.363 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.344+0000 7f10884a2b80 1 -- 172.21.15.38:0/64832524 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55eb0c0d8f10 con 0x55eb0c317340 2023-03-09T14:45:30.364 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.344+0000 7f10798b4700 1 --2- 172.21.15.38:0/64832524 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55eb0c317340 0x55eb0c316a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:30.364 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.345+0000 7f10798b4700 1 --2- 172.21.15.38:0/64832524 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55eb0c317340 0x55eb0c316a50 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9925cabe1bae7f34 server_cookie=175c5f62058311f1 in_seq=0 out_seq=0 2023-03-09T14:45:30.364 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.345+0000 7f105dffb700 10 client.15186 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:45:30.364 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.345+0000 7f105dffb700 1 -- 172.21.15.38:0/64832524 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18278 (crc 0 0 0) 0x55eb0c0d8f10 con 0x55eb0c317340 2023-03-09T14:45:30.364 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.345+0000 7f105dffb700 10 client.15186 handle_command_reply: tid=0 2023-03-09T14:45:30.364 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.345+0000 7f10884a2b80 1 client.15186 shutdown 2023-03-09T14:45:30.365 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.346+0000 7f10884a2b80 1 -- 172.21.15.38:0/64832524 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55eb0c3029f0 msgr2=0x55eb0c30c3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:30.365 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.346+0000 7f10884a2b80 1 --2- 172.21.15.38:0/64832524 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55eb0c3029f0 0x55eb0c30c3b0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f1070098e20 tx=0x7f107000b540 comp rx=0 tx=0).stop 2023-03-09T14:45:30.365 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.346+0000 7f10884a2b80 1 -- 172.21.15.38:0/64832524 shutdown_connections 2023-03-09T14:45:30.365 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.346+0000 7f10884a2b80 1 --2- 172.21.15.38:0/64832524 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55eb0c317340 0x55eb0c316a50 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.365 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.346+0000 7f10884a2b80 1 --2- 172.21.15.38:0/64832524 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55eb0c3029f0 0x55eb0c30c3b0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.365 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.346+0000 7f10884a2b80 1 --2- 172.21.15.38:0/64832524 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55eb0c3041f0 0x55eb0c313930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.365 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.346+0000 7f105dffb700 0 client.15186 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:45:30.366 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.346+0000 7f10884a2b80 1 --2- 172.21.15.38:0/64832524 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eb0c303340 0x55eb0c30c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.366 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.346+0000 7f10884a2b80 1 -- 172.21.15.38:0/64832524 >> 172.21.15.38:0/64832524 conn(0x55eb0c2fd500 msgr2=0x55eb0c2fcbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:30.366 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.346+0000 7f10884a2b80 1 -- 172.21.15.38:0/64832524 shutdown_connections 2023-03-09T14:45:30.366 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.346+0000 7f10884a2b80 1 -- 172.21.15.38:0/64832524 wait complete. 2023-03-09T14:45:30.366 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.346+0000 7f10884a2b80 20 client.15186 trim_cache size 0 max 16384 2023-03-09T14:45:30.371 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.363+0000 7f10884a2b80 1 Processor -- start 2023-03-09T14:45:30.371 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.364+0000 7f10884a2b80 1 -- start start 2023-03-09T14:45:30.371 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.364+0000 7f10884a2b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eb0c3029f0 0x55eb0c316a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:30.371 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.364+0000 7f10884a2b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55eb0c303340 0x55eb0c3138f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:30.371 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.364+0000 7f10884a2b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55eb0c3041f0 0x55eb0c2fcbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:30.371 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.364+0000 7f10884a2b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb0c30d2b0 con 0x55eb0c303340 2023-03-09T14:45:30.372 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.364+0000 7f10884a2b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb0c0d8f10 con 0x55eb0c3041f0 2023-03-09T14:45:30.372 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.364+0000 7f10884a2b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb0c1f4530 con 0x55eb0c3029f0 2023-03-09T14:45:30.372 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.364+0000 7f10798b4700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55eb0c303340 0x55eb0c3138f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:30.372 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.364+0000 7f10798b4700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55eb0c303340 0x55eb0c3138f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37898/0 (socket says 172.21.15.38:37898) 2023-03-09T14:45:30.372 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.364+0000 7f10798b4700 1 -- 172.21.15.38:0/2519365376 learned_addr learned my addr 172.21.15.38:0/2519365376 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:30.372 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.364+0000 7f107a8b6700 1 --2- 172.21.15.38:0/2519365376 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55eb0c3041f0 0x55eb0c2fcbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:30.372 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.364+0000 7f10798b4700 1 -- 172.21.15.38:0/2519365376 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eb0c3029f0 msgr2=0x55eb0c316a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:45:30.373 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.364+0000 7f107a0b5700 1 --2- 172.21.15.38:0/2519365376 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eb0c3029f0 0x55eb0c316a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:30.373 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.364+0000 7f10798b4700 1 --2- 172.21.15.38:0/2519365376 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eb0c3029f0 0x55eb0c316a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.373 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.364+0000 7f10798b4700 1 -- 172.21.15.38:0/2519365376 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55eb0c3041f0 msgr2=0x55eb0c2fcbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:45:30.373 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.364+0000 7f10798b4700 1 --2- 172.21.15.38:0/2519365376 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55eb0c3041f0 0x55eb0c2fcbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.373 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.364+0000 7f10798b4700 1 -- 172.21.15.38:0/2519365376 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55eb0c082d90 con 0x55eb0c303340 2023-03-09T14:45:30.373 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.364+0000 7f107a0b5700 1 --2- 172.21.15.38:0/2519365376 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eb0c3029f0 0x55eb0c316a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-09T14:45:30.373 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.364+0000 7f10798b4700 1 --2- 172.21.15.38:0/2519365376 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55eb0c303340 0x55eb0c3138f0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x55eb0c30b830 tx=0x7f1064013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee44136cb5a1e966 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:30.374 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.364+0000 7f10695b0700 1 -- 172.21.15.38:0/2519365376 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1064029070 con 0x55eb0c303340 2023-03-09T14:45:30.374 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.364+0000 7f10695b0700 1 -- 172.21.15.38:0/2519365376 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1064015070 con 0x55eb0c303340 2023-03-09T14:45:30.374 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.365+0000 7f10695b0700 1 -- 172.21.15.38:0/2519365376 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1064024070 con 0x55eb0c303340 2023-03-09T14:45:30.374 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.365+0000 7f10884a2b80 1 -- 172.21.15.38:0/2519365376 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55eb0c303340 msgr2=0x55eb0c3138f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:30.374 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.365+0000 7f10884a2b80 1 --2- 172.21.15.38:0/2519365376 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55eb0c303340 0x55eb0c3138f0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x55eb0c30b830 tx=0x7f1064013000 comp rx=0 tx=0).stop 2023-03-09T14:45:30.374 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.365+0000 7f10884a2b80 1 -- 172.21.15.38:0/2519365376 shutdown_connections 2023-03-09T14:45:30.375 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.365+0000 7f10884a2b80 1 --2- 172.21.15.38:0/2519365376 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55eb0c3041f0 0x55eb0c2fcbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.375 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.365+0000 7f10884a2b80 1 --2- 172.21.15.38:0/2519365376 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eb0c3029f0 0x55eb0c316a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.375 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.365+0000 7f10884a2b80 1 --2- 172.21.15.38:0/2519365376 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55eb0c303340 0x55eb0c3138f0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.375 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.365+0000 7f10884a2b80 1 -- 172.21.15.38:0/2519365376 >> 172.21.15.38:0/2519365376 conn(0x55eb0c2fd500 msgr2=0x55eb0c3200b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:30.375 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.365+0000 7f10884a2b80 1 -- 172.21.15.38:0/2519365376 shutdown_connections 2023-03-09T14:45:30.375 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.365+0000 7f10884a2b80 1 -- 172.21.15.38:0/2519365376 wait complete. 2023-03-09T14:45:30.375 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.365+0000 7f10884a2b80 1 -- start start 2023-03-09T14:45:30.375 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.365+0000 7f10884a2b80 1 Processor -- start 2023-03-09T14:45:30.376 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.366+0000 7f10884a2b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55eb0c3029f0 0x55eb0c30f390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:30.376 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.366+0000 7f10884a2b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eb0c303340 0x55eb0c30f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:30.376 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.366+0000 7f10884a2b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55eb0c3041f0 0x55eb0c312880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:30.376 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.366+0000 7f10884a2b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb0c312dc0 con 0x55eb0c3029f0 2023-03-09T14:45:30.376 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.366+0000 7f10884a2b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb0c312f30 con 0x55eb0c3041f0 2023-03-09T14:45:30.377 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.366+0000 7f10884a2b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eb0c3130f0 con 0x55eb0c303340 2023-03-09T14:45:30.377 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.366+0000 7f10884a2b80 10 client.0 _ll_register_callbacks cb 0x55eb0c1fa4d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:45:30.377 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.366+0000 7f107a0b5700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55eb0c3029f0 0x55eb0c30f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:30.377 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.366+0000 7f107a0b5700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55eb0c3029f0 0x55eb0c30f390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37902/0 (socket says 172.21.15.38:37902) 2023-03-09T14:45:30.377 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.366+0000 7f107a0b5700 1 -- 172.21.15.38:0/2942597584 learned_addr learned my addr 172.21.15.38:0/2942597584 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:30.377 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.366+0000 7f107a8b6700 1 --2- 172.21.15.38:0/2942597584 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55eb0c3041f0 0x55eb0c312880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:30.378 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.366+0000 7f10798b4700 1 --2- 172.21.15.38:0/2942597584 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eb0c303340 0x55eb0c30f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:30.378 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.366+0000 7f107a0b5700 1 -- 172.21.15.38:0/2942597584 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eb0c303340 msgr2=0x55eb0c30f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:45:30.378 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.366+0000 7f107a0b5700 1 --2- 172.21.15.38:0/2942597584 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eb0c303340 0x55eb0c30f8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.378 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.366+0000 7f107a0b5700 1 -- 172.21.15.38:0/2942597584 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55eb0c3041f0 msgr2=0x55eb0c312880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:45:30.378 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.366+0000 7f107a0b5700 1 --2- 172.21.15.38:0/2942597584 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55eb0c3041f0 0x55eb0c312880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.378 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.366+0000 7f107a0b5700 1 -- 172.21.15.38:0/2942597584 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55eb0c081260 con 0x55eb0c3029f0 2023-03-09T14:45:30.378 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.366+0000 7f107a0b5700 1 --2- 172.21.15.38:0/2942597584 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55eb0c3029f0 0x55eb0c30f390 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f1070098b90 tx=0x7f10700a0d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2d3d0e1f520bb8a server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:30.379 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.366+0000 7f105dffb700 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2023-03-09T14:45:30.379 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.367+0000 7f105dffb700 1 -- 172.21.15.38:0/2942597584 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1070097ff0 con 0x55eb0c3029f0 2023-03-09T14:45:30.379 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.367+0000 7f105dffb700 1 -- 172.21.15.38:0/2942597584 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f1070098e90 con 0x55eb0c3029f0 2023-03-09T14:45:30.379 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.367+0000 7f10884a2b80 1 -- 172.21.15.38:0/2942597584 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55eb0c212160 con 0x55eb0c3029f0 2023-03-09T14:45:30.379 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.367+0000 7f105dffb700 1 -- 172.21.15.38:0/2942597584 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1070098c60 con 0x55eb0c3029f0 2023-03-09T14:45:30.379 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.367+0000 7f10884a2b80 10 client.15192 fetch_fsmap learned FSMap version 23 2023-03-09T14:45:30.380 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.367+0000 7f10884a2b80 1 -- 172.21.15.38:0/2942597584 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55eb0c0a9200 con 0x55eb0c3029f0 2023-03-09T14:45:30.380 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.367+0000 7f105dffb700 1 -- 172.21.15.38:0/2942597584 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55eb0c212160 con 0x55eb0c3029f0 2023-03-09T14:45:30.380 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.367+0000 7f105dffb700 1 -- 172.21.15.38:0/2942597584 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f10700995e0 con 0x55eb0c3029f0 2023-03-09T14:45:30.380 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.367+0000 7f10884a2b80 10 client.15192 fetch_fsmap finished waiting for FSMap version 23 2023-03-09T14:45:30.380 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.367+0000 7f10884a2b80 10 client.15192 resolve_mds: resolved name 'b' to daemon mds.b 2023-03-09T14:45:30.380 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.367+0000 7f10884a2b80 20 client.15192 populate_metadata read hostname 'smithi038' 2023-03-09T14:45:30.380 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.367+0000 7f10884a2b80 1 --2- 172.21.15.38:0/2942597584 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55eb0c370960 0x55eb0c36d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:30.381 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.367+0000 7f10884a2b80 4 client.15192 mds_command: new command op to 24530 tid=0[{"prefix": "status"}] 2023-03-09T14:45:30.381 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.367+0000 7f10884a2b80 1 -- 172.21.15.38:0/2942597584 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55eb0c1fc170 con 0x55eb0c370960 2023-03-09T14:45:30.381 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.367+0000 7f10798b4700 1 --2- 172.21.15.38:0/2942597584 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55eb0c370960 0x55eb0c36d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:30.381 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.368+0000 7f105dffb700 10 client.15192 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:45:30.381 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.368+0000 7f10798b4700 1 --2- 172.21.15.38:0/2942597584 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55eb0c370960 0x55eb0c36d420 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=283b13df5c3f8806 server_cookie=4e7355a23229fde5 in_seq=0 out_seq=0 2023-03-09T14:45:30.382 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.368+0000 7f105dffb700 1 -- 172.21.15.38:0/2942597584 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x55eb0c1fc170 con 0x55eb0c370960 2023-03-09T14:45:30.383 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.368+0000 7f105dffb700 10 client.15192 handle_command_reply: tid=0 2023-03-09T14:45:30.383 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.368+0000 7f10884a2b80 1 client.15192 shutdown 2023-03-09T14:45:30.383 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.368+0000 7f10884a2b80 1 -- 172.21.15.38:0/2942597584 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55eb0c3029f0 msgr2=0x55eb0c30f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:30.383 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.368+0000 7f10884a2b80 1 --2- 172.21.15.38:0/2942597584 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55eb0c3029f0 0x55eb0c30f390 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f1070098b90 tx=0x7f10700a0d70 comp rx=0 tx=0).stop 2023-03-09T14:45:30.383 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.368+0000 7f10884a2b80 1 -- 172.21.15.38:0/2942597584 shutdown_connections 2023-03-09T14:45:30.383 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.368+0000 7f10884a2b80 1 --2- 172.21.15.38:0/2942597584 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55eb0c370960 0x55eb0c36d420 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.383 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.368+0000 7f10884a2b80 1 --2- 172.21.15.38:0/2942597584 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55eb0c3029f0 0x55eb0c30f390 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.383 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.368+0000 7f10884a2b80 1 --2- 172.21.15.38:0/2942597584 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55eb0c3041f0 0x55eb0c312880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.384 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.368+0000 7f10884a2b80 1 --2- 172.21.15.38:0/2942597584 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55eb0c303340 0x55eb0c30f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.384 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.368+0000 7f105dffb700 0 client.15192 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:45:30.384 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.368+0000 7f10884a2b80 1 -- 172.21.15.38:0/2942597584 >> 172.21.15.38:0/2942597584 conn(0x55eb0c2fd500 msgr2=0x55eb0c316a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:30.384 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.368+0000 7f10884a2b80 1 -- 172.21.15.38:0/2942597584 shutdown_connections 2023-03-09T14:45:30.384 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.368+0000 7f10884a2b80 1 -- 172.21.15.38:0/2942597584 wait complete. 2023-03-09T14:45:30.385 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.368+0000 7f10884a2b80 20 client.15192 trim_cache size 0 max 16384 2023-03-09T14:45:30.385 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2023-03-09T14:45:30.385 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_fsid": "c6733fe2-be87-11ed-9af2-001a4aab830c", 2023-03-09T14:45:30.385 INFO:teuthology.orchestra.run.smithi038.stdout: "whoami": 0, 2023-03-09T14:45:30.386 INFO:teuthology.orchestra.run.smithi038.stdout: "id": 24530, 2023-03-09T14:45:30.386 INFO:teuthology.orchestra.run.smithi038.stdout: "want_state": "up:active", 2023-03-09T14:45:30.386 INFO:teuthology.orchestra.run.smithi038.stdout: "state": "up:active", 2023-03-09T14:45:30.386 INFO:teuthology.orchestra.run.smithi038.stdout: "fs_name": "cephfs", 2023-03-09T14:45:30.386 INFO:teuthology.orchestra.run.smithi038.stdout: "rank_uptime": 71.281893256000004, 2023-03-09T14:45:30.386 INFO:teuthology.orchestra.run.smithi038.stdout: "mdsmap_epoch": 23, 2023-03-09T14:45:30.386 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_epoch": 89, 2023-03-09T14:45:30.387 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_epoch_barrier": 87, 2023-03-09T14:45:30.387 INFO:teuthology.orchestra.run.smithi038.stdout: "uptime": 76.789651328000005 2023-03-09T14:45:30.387 INFO:teuthology.orchestra.run.smithi038.stdout:} 2023-03-09T14:45:30.388 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.369+0000 7f107c319700 1 -- 172.21.15.38:0/3758613033 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f106009f400 msgr2=0x7f10600a18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:30.388 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.369+0000 7f107c319700 1 --2- 172.21.15.38:0/3758613033 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f106009f400 0x7f10600a18b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f1064005950 tx=0x7f106400b410 comp rx=0 tx=0).stop 2023-03-09T14:45:30.388 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.369+0000 7f107c319700 1 -- 172.21.15.38:0/3758613033 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10740fed90 msgr2=0x7f1074194b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:30.389 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.369+0000 7f107c319700 1 --2- 172.21.15.38:0/3758613033 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10740fed90 0x7f1074194b90 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f1070000c00 tx=0x7f10700047b0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.389 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.369+0000 7f107c319700 1 -- 172.21.15.38:0/3758613033 shutdown_connections 2023-03-09T14:45:30.389 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.369+0000 7f107c319700 1 --2- 172.21.15.38:0/3758613033 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10740fed90 0x7f1074194b90 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.389 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.369+0000 7f107c319700 1 --2- 172.21.15.38:0/3758613033 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f106009f400 0x7f10600a18b0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.389 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.369+0000 7f107c319700 1 --2- 172.21.15.38:0/3758613033 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1074100910 0x7f107419c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.389 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.369+0000 7f107c319700 1 --2- 172.21.15.38:0/3758613033 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f10740fff80 0x7f10741950d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:30.389 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.369+0000 7f107c319700 1 -- 172.21.15.38:0/3758613033 >> 172.21.15.38:0/3758613033 conn(0x7f10740fa3b0 msgr2=0x7f10740fc6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:30.390 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.369+0000 7f107c319700 1 -- 172.21.15.38:0/3758613033 shutdown_connections 2023-03-09T14:45:30.390 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:30.369+0000 7f107c319700 1 -- 172.21.15.38:0/3758613033 wait complete. 2023-03-09T14:45:30.390 INFO:tasks.cephadm:Setup complete, yielding 2023-03-09T14:45:30.390 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-03-09T14:45:30.405 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi038.front.sepia.ceph.com 2023-03-09T14:45:30.409 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- bash -c 'ceph orch status' 2023-03-09T14:45:31.064 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.a/config 2023-03-09T14:45:32.122 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:31 smithi038 ceph-mon[92422]: pgmap v297: 129 pgs: 129 active+clean; 636 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s; 54 B/s, 0 objects/s recovering 2023-03-09T14:45:32.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:31 smithi122 ceph-mon[93869]: pgmap v297: 129 pgs: 129 active+clean; 636 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s; 54 B/s, 0 objects/s recovering 2023-03-09T14:45:32.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:31 smithi181 ceph-mon[94189]: pgmap v297: 129 pgs: 129 active+clean; 636 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s; 54 B/s, 0 objects/s recovering 2023-03-09T14:45:32.310 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:32.306+0000 7f1b264d2700 1 -- 172.21.15.38:0/2050456615 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b201009a0 msgr2=0x7f1b201072d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:32.311 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:32.306+0000 7f1b264d2700 1 --2- 172.21.15.38:0/2050456615 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b201009a0 0x7f1b201072d0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f1b1800b0a0 tx=0x7f1b1800b3b0 comp rx=0 tx=0).stop 2023-03-09T14:45:32.311 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:32.306+0000 7f1b264d2700 1 -- 172.21.15.38:0/2050456615 shutdown_connections 2023-03-09T14:45:32.311 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:32.306+0000 7f1b264d2700 1 --2- 172.21.15.38:0/2050456615 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b201009a0 0x7f1b201072d0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:32.311 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:32.306+0000 7f1b264d2700 1 --2- 172.21.15.38:0/2050456615 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1b200feea0 0x7f1b200ff2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:32.311 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:32.306+0000 7f1b264d2700 1 --2- 172.21.15.38:0/2050456615 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1b200ffff0 0x7f1b20100460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:32.312 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:32.306+0000 7f1b264d2700 1 -- 172.21.15.38:0/2050456615 >> 172.21.15.38:0/2050456615 conn(0x7f1b200fa480 msgr2=0x7f1b200fc8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:32.312 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:32.306+0000 7f1b264d2700 1 -- 172.21.15.38:0/2050456615 shutdown_connections 2023-03-09T14:45:32.312 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:32.306+0000 7f1b264d2700 1 -- 172.21.15.38:0/2050456615 wait complete. 2023-03-09T14:45:32.312 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:32.307+0000 7f1b264d2700 1 Processor -- start 2023-03-09T14:45:32.312 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:32.307+0000 7f1b264d2700 1 -- start start 2023-03-09T14:45:32.312 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:32.307+0000 7f1b264d2700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1b200feea0 0x7f1b20068db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:32.312 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:32.307+0000 7f1b264d2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b200ffff0 0x7f1b2019ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:32.313 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:32.307+0000 7f1b264d2700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1b201009a0 0x7f1b2019d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:32.313 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:32.307+0000 7f1b264d2700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b2019d920 con 0x7f1b200ffff0 2023-03-09T14:45:32.313 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:32.307+0000 7f1b264d2700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b2019da90 con 0x7f1b201009a0 2023-03-09T14:45:32.313 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:32.307+0000 7f1b264d2700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b2019dc50 con 0x7f1b200feea0 2023-03-09T14:45:32.313 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:32.307+0000 7f1b1f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b200ffff0 0x7f1b2019ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:32.314 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:32.307+0000 7f1b1f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b200ffff0 0x7f1b2019ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37918/0 (socket says 172.21.15.38:37918) 2023-03-09T14:45:32.314 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:32.307+0000 7f1b1f7fe700 1 -- 172.21.15.38:0/4058598866 learned_addr learned my addr 172.21.15.38:0/4058598866 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:32.314 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:32.307+0000 7f1b1ffff700 1 --2- 172.21.15.38:0/4058598866 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1b200feea0 0x7f1b20068db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:32.315 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:32.307+0000 7f1b24a6f700 1 --2- 172.21.15.38:0/4058598866 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1b201009a0 0x7f1b2019d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:32.316 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:32.308+0000 7f1b1f7fe700 1 -- 172.21.15.38:0/4058598866 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1b200feea0 msgr2=0x7f1b20068db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:32.316 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:32.308+0000 7f1b1f7fe700 1 --2- 172.21.15.38:0/4058598866 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1b200feea0 0x7f1b20068db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:32.316 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:32.308+0000 7f1b1f7fe700 1 -- 172.21.15.38:0/4058598866 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1b201009a0 msgr2=0x7f1b2019d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:32.316 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:32.308+0000 7f1b1f7fe700 1 --2- 172.21.15.38:0/4058598866 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1b201009a0 0x7f1b2019d150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:32.316 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:32.308+0000 7f1b1f7fe700 1 -- 172.21.15.38:0/4058598866 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b18009d00 con 0x7f1b200ffff0 2023-03-09T14:45:32.317 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:32.308+0000 7f1b1f7fe700 1 --2- 172.21.15.38:0/4058598866 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b200ffff0 0x7f1b2019ab00 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f1b1000c9e0 tx=0x7f1b1000cda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:32.317 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:32.308+0000 7f1b1d7fa700 1 -- 172.21.15.38:0/4058598866 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b10011b60 con 0x7f1b200ffff0 2023-03-09T14:45:32.317 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:32.308+0000 7f1b1d7fa700 1 -- 172.21.15.38:0/4058598866 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1b10004cf0 con 0x7f1b200ffff0 2023-03-09T14:45:32.317 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:32.308+0000 7f1b264d2700 1 -- 172.21.15.38:0/4058598866 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b2010a800 con 0x7f1b200ffff0 2023-03-09T14:45:32.318 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:32.308+0000 7f1b1d7fa700 1 -- 172.21.15.38:0/4058598866 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b100056f0 con 0x7f1b200ffff0 2023-03-09T14:45:32.318 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:32.308+0000 7f1b264d2700 1 -- 172.21.15.38:0/4058598866 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b2010a670 con 0x7f1b200ffff0 2023-03-09T14:45:32.319 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:32.309+0000 7f1b264d2700 1 -- 172.21.15.38:0/4058598866 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b201063b0 con 0x7f1b200ffff0 2023-03-09T14:45:32.319 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:32.310+0000 7f1b1d7fa700 1 -- 172.21.15.38:0/4058598866 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 133238+0+0 (secure 0 0 0) 0x7f1b10004e60 con 0x7f1b200ffff0 2023-03-09T14:45:32.319 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:32.311+0000 7f1b1d7fa700 1 --2- 172.21.15.38:0/4058598866 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f1b0c09f400 0x7f1b0c0a18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:32.319 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:32.311+0000 7f1b1ffff700 1 --2- 172.21.15.38:0/4058598866 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f1b0c09f400 0x7f1b0c0a18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:32.320 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:32.311+0000 7f1b1d7fa700 1 -- 172.21.15.38:0/4058598866 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f1b100ce050 con 0x7f1b200ffff0 2023-03-09T14:45:32.320 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:32.311+0000 7f1b1ffff700 1 --2- 172.21.15.38:0/4058598866 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f1b0c09f400 0x7f1b0c0a18b0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f1b08005fd0 tx=0x7f1b0800a560 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:32.321 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:32.315+0000 7f1b1d7fa700 1 -- 172.21.15.38:0/4058598866 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f1b100525b0 con 0x7f1b200ffff0 2023-03-09T14:45:32.534 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:32.531+0000 7f1b264d2700 1 -- 172.21.15.38:0/4058598866 --> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1b20061820 con 0x7f1b0c09f400 2023-03-09T14:45:32.535 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:32.532+0000 7f1b1d7fa700 1 -- 172.21.15.38:0/4058598866 <== mgr.14150 v2:172.21.15.38:6800/3328887328 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f1b20061820 con 0x7f1b0c09f400 2023-03-09T14:45:32.535 INFO:teuthology.orchestra.run.smithi038.stdout:Backend: cephadm 2023-03-09T14:45:32.536 INFO:teuthology.orchestra.run.smithi038.stdout:Available: Yes 2023-03-09T14:45:32.536 INFO:teuthology.orchestra.run.smithi038.stdout:Paused: No 2023-03-09T14:45:32.537 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:32.534+0000 7f1b264d2700 1 -- 172.21.15.38:0/4058598866 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f1b0c09f400 msgr2=0x7f1b0c0a18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:32.538 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:32.534+0000 7f1b264d2700 1 --2- 172.21.15.38:0/4058598866 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f1b0c09f400 0x7f1b0c0a18b0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f1b08005fd0 tx=0x7f1b0800a560 comp rx=0 tx=0).stop 2023-03-09T14:45:32.538 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:32.534+0000 7f1b264d2700 1 -- 172.21.15.38:0/4058598866 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b200ffff0 msgr2=0x7f1b2019ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:32.538 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:32.534+0000 7f1b264d2700 1 --2- 172.21.15.38:0/4058598866 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b200ffff0 0x7f1b2019ab00 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f1b1000c9e0 tx=0x7f1b1000cda0 comp rx=0 tx=0).stop 2023-03-09T14:45:32.538 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:32.534+0000 7f1b264d2700 1 -- 172.21.15.38:0/4058598866 shutdown_connections 2023-03-09T14:45:32.538 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:32.534+0000 7f1b264d2700 1 --2- 172.21.15.38:0/4058598866 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f1b0c09f400 0x7f1b0c0a18b0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:32.538 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:32.534+0000 7f1b264d2700 1 --2- 172.21.15.38:0/4058598866 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1b201009a0 0x7f1b2019d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:32.539 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:32.534+0000 7f1b264d2700 1 --2- 172.21.15.38:0/4058598866 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1b200feea0 0x7f1b20068db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:32.539 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:32.534+0000 7f1b264d2700 1 --2- 172.21.15.38:0/4058598866 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1b200ffff0 0x7f1b2019ab00 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:32.539 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:32.534+0000 7f1b264d2700 1 -- 172.21.15.38:0/4058598866 >> 172.21.15.38:0/4058598866 conn(0x7f1b200fa480 msgr2=0x7f1b201034f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:32.539 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:32.534+0000 7f1b264d2700 1 -- 172.21.15.38:0/4058598866 shutdown_connections 2023-03-09T14:45:32.539 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:32.534+0000 7f1b264d2700 1 -- 172.21.15.38:0/4058598866 wait complete. 2023-03-09T14:45:32.920 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- bash -c 'ceph orch ps' 2023-03-09T14:45:33.572 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.a/config 2023-03-09T14:45:33.812 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:33 smithi038 ceph-mon[92422]: from='client.15198 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:45:33.812 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:33 smithi038 ceph-mon[92422]: pgmap v298: 129 pgs: 129 active+clean; 636 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T14:45:33.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:33 smithi122 ceph-mon[93869]: from='client.15198 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:45:33.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:33 smithi122 ceph-mon[93869]: pgmap v298: 129 pgs: 129 active+clean; 636 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T14:45:33.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:33 smithi181 ceph-mon[94189]: from='client.15198 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:45:33.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:33 smithi181 ceph-mon[94189]: pgmap v298: 129 pgs: 129 active+clean; 636 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T14:45:34.768 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:34.763+0000 7f48a7152700 1 -- 172.21.15.38:0/2412949481 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48a01011a0 msgr2=0x7f48a01015f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:34.768 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:34.763+0000 7f48a7152700 1 --2- 172.21.15.38:0/2412949481 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48a01011a0 0x7f48a01015f0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f488c009990 tx=0x7f488c009ca0 comp rx=0 tx=0).stop 2023-03-09T14:45:34.768 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:34.763+0000 7f48a7152700 1 -- 172.21.15.38:0/2412949481 shutdown_connections 2023-03-09T14:45:34.768 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:34.763+0000 7f48a7152700 1 --2- 172.21.15.38:0/2412949481 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f48a0101b30 0x7f48a01083f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:34.769 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:34.763+0000 7f48a7152700 1 --2- 172.21.15.38:0/2412949481 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f48a00fffb0 0x7f48a01003c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:34.769 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:34.763+0000 7f48a7152700 1 --2- 172.21.15.38:0/2412949481 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48a01011a0 0x7f48a01015f0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:34.769 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:34.763+0000 7f48a7152700 1 -- 172.21.15.38:0/2412949481 >> 172.21.15.38:0/2412949481 conn(0x7f48a00fb590 msgr2=0x7f48a00fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:34.769 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:34.763+0000 7f48a7152700 1 -- 172.21.15.38:0/2412949481 shutdown_connections 2023-03-09T14:45:34.769 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:34.763+0000 7f48a7152700 1 -- 172.21.15.38:0/2412949481 wait complete. 2023-03-09T14:45:34.770 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:34.763+0000 7f48a7152700 1 Processor -- start 2023-03-09T14:45:34.770 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:34.764+0000 7f48a7152700 1 -- start start 2023-03-09T14:45:34.770 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:34.764+0000 7f48a7152700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48a00fffb0 0x7f48a0195d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:34.770 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:34.764+0000 7f48a7152700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f48a01011a0 0x7f48a0196280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:34.770 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:34.764+0000 7f48a7152700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f48a0101b30 0x7f48a019d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:34.770 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:34.764+0000 7f48a7152700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48a019d8c0 con 0x7f48a00fffb0 2023-03-09T14:45:34.770 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:34.764+0000 7f48a7152700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48a019da00 con 0x7f48a0101b30 2023-03-09T14:45:34.771 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:34.764+0000 7f48a7152700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48a019db90 con 0x7f48a01011a0 2023-03-09T14:45:34.771 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:34.764+0000 7f48a4eee700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48a00fffb0 0x7f48a0195d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:34.771 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:34.764+0000 7f48a4eee700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48a00fffb0 0x7f48a0195d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37928/0 (socket says 172.21.15.38:37928) 2023-03-09T14:45:34.771 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:34.764+0000 7f48a4eee700 1 -- 172.21.15.38:0/3216310479 learned_addr learned my addr 172.21.15.38:0/3216310479 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:34.771 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:34.764+0000 7f489ffff700 1 --2- 172.21.15.38:0/3216310479 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f48a01011a0 0x7f48a0196280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:34.771 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:34.764+0000 7f48a56ef700 1 --2- 172.21.15.38:0/3216310479 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f48a0101b30 0x7f48a019d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:34.772 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:34.764+0000 7f48a4eee700 1 -- 172.21.15.38:0/3216310479 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f48a01011a0 msgr2=0x7f48a0196280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:34.772 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:34.764+0000 7f48a4eee700 1 --2- 172.21.15.38:0/3216310479 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f48a01011a0 0x7f48a0196280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:34.772 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:34.764+0000 7f48a4eee700 1 -- 172.21.15.38:0/3216310479 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f48a0101b30 msgr2=0x7f48a019d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:34.772 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:34.764+0000 7f48a4eee700 1 --2- 172.21.15.38:0/3216310479 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f48a0101b30 0x7f48a019d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:34.772 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:34.764+0000 7f48a4eee700 1 -- 172.21.15.38:0/3216310479 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f488c0095f0 con 0x7f48a00fffb0 2023-03-09T14:45:34.772 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:34.764+0000 7f48a56ef700 1 --2- 172.21.15.38:0/3216310479 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f48a0101b30 0x7f48a019d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:45:34.772 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:34.764+0000 7f48a4eee700 1 --2- 172.21.15.38:0/3216310479 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48a00fffb0 0x7f48a0195d40 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f489400d980 tx=0x7f489400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:34.773 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:34.765+0000 7f489dffb700 1 -- 172.21.15.38:0/3216310479 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f489400b950 con 0x7f48a00fffb0 2023-03-09T14:45:34.773 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:34.765+0000 7f489dffb700 1 -- 172.21.15.38:0/3216310479 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4894007620 con 0x7f48a00fffb0 2023-03-09T14:45:34.773 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:34.765+0000 7f48a7152700 1 -- 172.21.15.38:0/3216310479 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48a010b910 con 0x7f48a00fffb0 2023-03-09T14:45:34.773 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:34.765+0000 7f489dffb700 1 -- 172.21.15.38:0/3216310479 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f489400f3c0 con 0x7f48a00fffb0 2023-03-09T14:45:34.773 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:34.765+0000 7f48a7152700 1 -- 172.21.15.38:0/3216310479 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48a010b780 con 0x7f48a00fffb0 2023-03-09T14:45:34.774 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:34.766+0000 7f48a7152700 1 -- 172.21.15.38:0/3216310479 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48a019e1f0 con 0x7f48a00fffb0 2023-03-09T14:45:34.775 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:34.767+0000 7f489dffb700 1 -- 172.21.15.38:0/3216310479 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 133238+0+0 (secure 0 0 0) 0x7f4894018030 con 0x7f48a00fffb0 2023-03-09T14:45:34.775 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:34.768+0000 7f489dffb700 1 --2- 172.21.15.38:0/3216310479 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f489009f480 0x7f48900a1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:34.775 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:34.768+0000 7f489ffff700 1 --2- 172.21.15.38:0/3216310479 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f489009f480 0x7f48900a1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:34.775 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:34.768+0000 7f489dffb700 1 -- 172.21.15.38:0/3216310479 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f48940c9eb0 con 0x7f48a00fffb0 2023-03-09T14:45:34.775 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:34.768+0000 7f489ffff700 1 --2- 172.21.15.38:0/3216310479 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f489009f480 0x7f48900a1930 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f488c00b5c0 tx=0x7f488c005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:34.776 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:34.772+0000 7f489dffb700 1 -- 172.21.15.38:0/3216310479 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f4894093ec0 con 0x7f48a00fffb0 2023-03-09T14:45:34.981 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:34.979+0000 7f48a7152700 1 -- 172.21.15.38:0/3216310479 --> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f48a0060bc0 con 0x7f489009f480 2023-03-09T14:45:35.001 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:34.999+0000 7f489dffb700 1 -- 172.21.15.38:0/3216310479 <== mgr.14150 v2:172.21.15.38:6800/3328887328 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+4185 (secure 0 0 0) 0x7f48a0060bc0 con 0x7f489009f480 2023-03-09T14:45:35.002 INFO:teuthology.orchestra.run.smithi038.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-03-09T14:45:35.002 INFO:teuthology.orchestra.run.smithi038.stdout:mds.a smithi038 running (84s) 10s ago 83s 15.3M - 18.0.0-2872-ge0ab30a2 d8c5b11cdf06 42183dd35780 2023-03-09T14:45:35.002 INFO:teuthology.orchestra.run.smithi038.stdout:mds.b smithi122 running (81s) 11s ago 81s 31.1M - 18.0.0-2872-ge0ab30a2 d8c5b11cdf06 f1052366a81a 2023-03-09T14:45:35.002 INFO:teuthology.orchestra.run.smithi038.stdout:mds.c smithi181 running (93s) 11s ago 92s 19.9M - 18.0.0-2872-ge0ab30a2 d8c5b11cdf06 7815be79ff54 2023-03-09T14:45:35.003 INFO:teuthology.orchestra.run.smithi038.stdout:mds.d smithi038 running (99s) 10s ago 99s 15.1M - 18.0.0-2872-ge0ab30a2 d8c5b11cdf06 b320107d5495 2023-03-09T14:45:35.003 INFO:teuthology.orchestra.run.smithi038.stdout:mds.e smithi122 running (97s) 11s ago 96s 20.4M - 18.0.0-2872-ge0ab30a2 d8c5b11cdf06 19a2add88981 2023-03-09T14:45:35.003 INFO:teuthology.orchestra.run.smithi038.stdout:mds.f smithi181 running (95s) 11s ago 94s 20.1M - 18.0.0-2872-ge0ab30a2 d8c5b11cdf06 6f38f2a22544 2023-03-09T14:45:35.003 INFO:teuthology.orchestra.run.smithi038.stdout:mds.g smithi038 running (86s) 10s ago 86s 14.3M - 18.0.0-2872-ge0ab30a2 d8c5b11cdf06 0d0e9b81d97b 2023-03-09T14:45:35.003 INFO:teuthology.orchestra.run.smithi038.stdout:mds.h smithi122 running (90s) 11s ago 90s 18.9M - 18.0.0-2872-ge0ab30a2 d8c5b11cdf06 0fa6bd943db5 2023-03-09T14:45:35.003 INFO:teuthology.orchestra.run.smithi038.stdout:mds.i smithi181 running (88s) 11s ago 88s 18.9M - 18.0.0-2872-ge0ab30a2 d8c5b11cdf06 fc04bc42d9d0 2023-03-09T14:45:35.003 INFO:teuthology.orchestra.run.smithi038.stdout:mds.j smithi038 running (101s) 10s ago 101s 15.2M - 18.0.0-2872-ge0ab30a2 d8c5b11cdf06 1fcb4c6e6566 2023-03-09T14:45:35.004 INFO:teuthology.orchestra.run.smithi038.stdout:mds.k smithi122 running (103s) 11s ago 103s 17.7M - 18.0.0-2872-ge0ab30a2 d8c5b11cdf06 fea4dca72f4f 2023-03-09T14:45:35.004 INFO:teuthology.orchestra.run.smithi038.stdout:mds.l smithi181 running (106s) 11s ago 105s 15.2M - 18.0.0-2872-ge0ab30a2 d8c5b11cdf06 0e1798f50c1a 2023-03-09T14:45:35.004 INFO:teuthology.orchestra.run.smithi038.stdout:mgr.x smithi038 *:9283,8765 running (8m) 10s ago 8m 528M - 18.0.0-2872-ge0ab30a2 d8c5b11cdf06 6564c06a6086 2023-03-09T14:45:35.004 INFO:teuthology.orchestra.run.smithi038.stdout:mgr.y smithi122 *:8443,8765 running (6m) 11s ago 6m 464M - 18.0.0-2872-ge0ab30a2 d8c5b11cdf06 d96adcd25e38 2023-03-09T14:45:35.004 INFO:teuthology.orchestra.run.smithi038.stdout:mgr.z smithi181 *:8443,8765 running (6m) 11s ago 6m 462M - 18.0.0-2872-ge0ab30a2 d8c5b11cdf06 5cd1ddc89e07 2023-03-09T14:45:35.004 INFO:teuthology.orchestra.run.smithi038.stdout:mon.a smithi038 running (8m) 10s ago 8m 93.4M 2048M 18.0.0-2872-ge0ab30a2 d8c5b11cdf06 46f003fc444f 2023-03-09T14:45:35.004 INFO:teuthology.orchestra.run.smithi038.stdout:mon.b smithi122 running (6m) 11s ago 6m 60.5M 2048M 18.0.0-2872-ge0ab30a2 d8c5b11cdf06 c4b19952a646 2023-03-09T14:45:35.004 INFO:teuthology.orchestra.run.smithi038.stdout:mon.c smithi181 running (6m) 11s ago 6m 61.2M 2048M 18.0.0-2872-ge0ab30a2 d8c5b11cdf06 a3624488422e 2023-03-09T14:45:35.005 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0 smithi038 running (6m) 10s ago 6m 72.3M 4293M 18.0.0-2872-ge0ab30a2 d8c5b11cdf06 ddc1823e8ef4 2023-03-09T14:45:35.005 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1 smithi122 running (5m) 11s ago 5m 77.0M 4292M 18.0.0-2872-ge0ab30a2 d8c5b11cdf06 dd7272022893 2023-03-09T14:45:35.005 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2 smithi181 running (5m) 11s ago 5m 74.3M 4292M 18.0.0-2872-ge0ab30a2 d8c5b11cdf06 2932a8d483ec 2023-03-09T14:45:35.005 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3 smithi038 running (4m) 10s ago 4m 65.7M 4293M 18.0.0-2872-ge0ab30a2 d8c5b11cdf06 36be90ea5b84 2023-03-09T14:45:35.005 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4 smithi122 running (4m) 11s ago 4m 65.5M 4292M 18.0.0-2872-ge0ab30a2 d8c5b11cdf06 ab3b0a38dadf 2023-03-09T14:45:35.005 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5 smithi181 running (4m) 11s ago 4m 71.5M 4292M 18.0.0-2872-ge0ab30a2 d8c5b11cdf06 1874ae24c8e9 2023-03-09T14:45:35.005 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6 smithi038 running (3m) 10s ago 3m 63.3M 4293M 18.0.0-2872-ge0ab30a2 d8c5b11cdf06 eef9e0e0cb0f 2023-03-09T14:45:35.005 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7 smithi122 running (3m) 11s ago 3m 65.6M 4292M 18.0.0-2872-ge0ab30a2 d8c5b11cdf06 7c2c446757f4 2023-03-09T14:45:35.006 INFO:teuthology.orchestra.run.smithi038.stdout:osd.8 smithi181 running (3m) 11s ago 3m 62.0M 4292M 18.0.0-2872-ge0ab30a2 d8c5b11cdf06 07d76e547640 2023-03-09T14:45:35.006 INFO:teuthology.orchestra.run.smithi038.stdout:osd.9 smithi038 running (2m) 10s ago 2m 59.8M 4293M 18.0.0-2872-ge0ab30a2 d8c5b11cdf06 eba76e898a66 2023-03-09T14:45:35.006 INFO:teuthology.orchestra.run.smithi038.stdout:osd.10 smithi122 running (2m) 11s ago 2m 66.4M 4292M 18.0.0-2872-ge0ab30a2 d8c5b11cdf06 86bfeff88653 2023-03-09T14:45:35.006 INFO:teuthology.orchestra.run.smithi038.stdout:osd.11 smithi181 running (2m) 11s ago 2m 56.2M 4292M 18.0.0-2872-ge0ab30a2 d8c5b11cdf06 4e5905f69288 2023-03-09T14:45:35.007 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:35.002+0000 7f48a7152700 1 -- 172.21.15.38:0/3216310479 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f489009f480 msgr2=0x7f48900a1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:35.007 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:35.002+0000 7f48a7152700 1 --2- 172.21.15.38:0/3216310479 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f489009f480 0x7f48900a1930 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f488c00b5c0 tx=0x7f488c005960 comp rx=0 tx=0).stop 2023-03-09T14:45:35.007 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:35.002+0000 7f48a7152700 1 -- 172.21.15.38:0/3216310479 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48a00fffb0 msgr2=0x7f48a0195d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:35.007 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:35.002+0000 7f48a7152700 1 --2- 172.21.15.38:0/3216310479 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48a00fffb0 0x7f48a0195d40 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f489400d980 tx=0x7f489400dc90 comp rx=0 tx=0).stop 2023-03-09T14:45:35.007 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:35.002+0000 7f48a7152700 1 -- 172.21.15.38:0/3216310479 shutdown_connections 2023-03-09T14:45:35.007 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:35.002+0000 7f48a7152700 1 --2- 172.21.15.38:0/3216310479 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48a00fffb0 0x7f48a0195d40 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:35.008 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:35.002+0000 7f48a7152700 1 --2- 172.21.15.38:0/3216310479 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f489009f480 0x7f48900a1930 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:35.008 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:35.002+0000 7f48a7152700 1 --2- 172.21.15.38:0/3216310479 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f48a0101b30 0x7f48a019d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:35.008 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:35.002+0000 7f48a7152700 1 --2- 172.21.15.38:0/3216310479 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f48a01011a0 0x7f48a0196280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:35.008 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:35.002+0000 7f48a7152700 1 -- 172.21.15.38:0/3216310479 >> 172.21.15.38:0/3216310479 conn(0x7f48a00fb590 msgr2=0x7f48a00fd810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:35.008 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:35.002+0000 7f48a7152700 1 -- 172.21.15.38:0/3216310479 shutdown_connections 2023-03-09T14:45:35.008 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:35.002+0000 7f48a7152700 1 -- 172.21.15.38:0/3216310479 wait complete. 2023-03-09T14:45:35.434 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- bash -c 'ceph orch ls' 2023-03-09T14:45:36.089 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.a/config 2023-03-09T14:45:36.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:35 smithi122 ceph-mon[93869]: pgmap v299: 129 pgs: 129 active+clean; 636 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-03-09T14:45:36.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:35 smithi122 ceph-mon[93869]: from='client.15204 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:45:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:35 smithi181 ceph-mon[94189]: pgmap v299: 129 pgs: 129 active+clean; 636 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-03-09T14:45:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:35 smithi181 ceph-mon[94189]: from='client.15204 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:45:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:35 smithi038 ceph-mon[92422]: pgmap v299: 129 pgs: 129 active+clean; 636 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-03-09T14:45:36.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:35 smithi038 ceph-mon[92422]: from='client.15204 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:45:37.313 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:37.307+0000 7fce7eac1700 1 -- 172.21.15.38:0/2862674769 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fce781011c0 msgr2=0x7fce78101610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:37.314 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:37.307+0000 7fce7eac1700 1 --2- 172.21.15.38:0/2862674769 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fce781011c0 0x7fce78101610 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fce64009990 tx=0x7fce64009ca0 comp rx=0 tx=0).stop 2023-03-09T14:45:37.314 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:37.307+0000 7fce7eac1700 1 -- 172.21.15.38:0/2862674769 shutdown_connections 2023-03-09T14:45:37.314 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:37.307+0000 7fce7eac1700 1 --2- 172.21.15.38:0/2862674769 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fce78101b50 0x7fce78108410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:37.315 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:37.307+0000 7fce7eac1700 1 --2- 172.21.15.38:0/2862674769 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fce780fffd0 0x7fce781003e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:37.315 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:37.307+0000 7fce7eac1700 1 --2- 172.21.15.38:0/2862674769 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fce781011c0 0x7fce78101610 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:37.315 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:37.307+0000 7fce7eac1700 1 -- 172.21.15.38:0/2862674769 >> 172.21.15.38:0/2862674769 conn(0x7fce780fb5b0 msgr2=0x7fce780fda20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:37.315 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:37.307+0000 7fce7eac1700 1 -- 172.21.15.38:0/2862674769 shutdown_connections 2023-03-09T14:45:37.316 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:37.307+0000 7fce7eac1700 1 -- 172.21.15.38:0/2862674769 wait complete. 2023-03-09T14:45:37.316 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:37.308+0000 7fce7eac1700 1 Processor -- start 2023-03-09T14:45:37.316 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:37.308+0000 7fce7eac1700 1 -- start start 2023-03-09T14:45:37.317 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:37.308+0000 7fce7eac1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fce780fffd0 0x7fce78195d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:37.317 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:37.308+0000 7fce7eac1700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fce781011c0 0x7fce781962b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:37.317 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:37.308+0000 7fce7eac1700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fce78101b50 0x7fce7819d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:37.318 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:37.308+0000 7fce7eac1700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce7819d8f0 con 0x7fce780fffd0 2023-03-09T14:45:37.318 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:37.308+0000 7fce7eac1700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce7819da30 con 0x7fce78101b50 2023-03-09T14:45:37.318 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:37.308+0000 7fce7eac1700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce7819dbc0 con 0x7fce781011c0 2023-03-09T14:45:37.318 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:37.308+0000 7fce7c85d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fce780fffd0 0x7fce78195d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:37.318 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:37.309+0000 7fce7c85d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fce780fffd0 0x7fce78195d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37940/0 (socket says 172.21.15.38:37940) 2023-03-09T14:45:37.319 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:37.309+0000 7fce7c85d700 1 -- 172.21.15.38:0/1842924882 learned_addr learned my addr 172.21.15.38:0/1842924882 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:37.319 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:37.309+0000 7fce7d05e700 1 --2- 172.21.15.38:0/1842924882 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fce78101b50 0x7fce7819d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:37.319 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:37.309+0000 7fce77fff700 1 --2- 172.21.15.38:0/1842924882 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fce781011c0 0x7fce781962b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:37.319 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:37.309+0000 7fce7c85d700 1 -- 172.21.15.38:0/1842924882 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fce781011c0 msgr2=0x7fce781962b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:37.320 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:37.309+0000 7fce7c85d700 1 --2- 172.21.15.38:0/1842924882 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fce781011c0 0x7fce781962b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:37.320 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:37.309+0000 7fce7c85d700 1 -- 172.21.15.38:0/1842924882 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fce78101b50 msgr2=0x7fce7819d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:37.320 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:37.309+0000 7fce7c85d700 1 --2- 172.21.15.38:0/1842924882 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fce78101b50 0x7fce7819d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:37.320 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:37.309+0000 7fce7c85d700 1 -- 172.21.15.38:0/1842924882 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce640095f0 con 0x7fce780fffd0 2023-03-09T14:45:37.321 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:37.309+0000 7fce77fff700 1 --2- 172.21.15.38:0/1842924882 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fce781011c0 0x7fce781962b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-09T14:45:37.321 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:37.309+0000 7fce7d05e700 1 --2- 172.21.15.38:0/1842924882 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fce78101b50 0x7fce7819d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:45:37.321 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:37.309+0000 7fce7c85d700 1 --2- 172.21.15.38:0/1842924882 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fce780fffd0 0x7fce78195d70 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fce6c00d980 tx=0x7fce6c00dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:37.322 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:37.309+0000 7fce75ffb700 1 -- 172.21.15.38:0/1842924882 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce6c00de60 con 0x7fce780fffd0 2023-03-09T14:45:37.322 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:37.309+0000 7fce75ffb700 1 -- 172.21.15.38:0/1842924882 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fce6c0044e0 con 0x7fce780fffd0 2023-03-09T14:45:37.322 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:37.309+0000 7fce7eac1700 1 -- 172.21.15.38:0/1842924882 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce7810b930 con 0x7fce780fffd0 2023-03-09T14:45:37.323 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:37.309+0000 7fce75ffb700 1 -- 172.21.15.38:0/1842924882 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce6c003be0 con 0x7fce780fffd0 2023-03-09T14:45:37.323 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:37.309+0000 7fce7eac1700 1 -- 172.21.15.38:0/1842924882 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce7810b7a0 con 0x7fce780fffd0 2023-03-09T14:45:37.323 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:37.310+0000 7fce7eac1700 1 -- 172.21.15.38:0/1842924882 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce781075c0 con 0x7fce780fffd0 2023-03-09T14:45:37.324 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:37.311+0000 7fce75ffb700 1 -- 172.21.15.38:0/1842924882 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 133238+0+0 (secure 0 0 0) 0x7fce6c00b030 con 0x7fce780fffd0 2023-03-09T14:45:37.324 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:37.312+0000 7fce75ffb700 1 --2- 172.21.15.38:0/1842924882 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fce6809f400 0x7fce680a18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:37.324 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:37.312+0000 7fce75ffb700 1 -- 172.21.15.38:0/1842924882 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fce6c0c9dc0 con 0x7fce780fffd0 2023-03-09T14:45:37.324 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:37.312+0000 7fce77fff700 1 --2- 172.21.15.38:0/1842924882 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fce6809f400 0x7fce680a18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:37.325 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:37.312+0000 7fce77fff700 1 --2- 172.21.15.38:0/1842924882 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fce6809f400 0x7fce680a18b0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fce64000c00 tx=0x7fce64005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:37.325 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:37.316+0000 7fce75ffb700 1 -- 172.21.15.38:0/1842924882 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7fce6c093e30 con 0x7fce780fffd0 2023-03-09T14:45:37.522 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:37.520+0000 7fce7eac1700 1 -- 172.21.15.38:0/1842924882 --> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fce78061910 con 0x7fce6809f400 2023-03-09T14:45:37.530 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:37.527+0000 7fce75ffb700 1 -- 172.21.15.38:0/1842924882 <== mgr.14150 v2:172.21.15.38:6800/3328887328 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+985 (secure 0 0 0) 0x7fce78061910 con 0x7fce6809f400 2023-03-09T14:45:37.530 INFO:teuthology.orchestra.run.smithi038.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2023-03-09T14:45:37.530 INFO:teuthology.orchestra.run.smithi038.stdout:mds.all 12/12 14s ago 110s smithi038=a;smithi038=d;smithi038=g;smithi038=j;smithi122=b;smithi122=e;smithi122=h;smithi122=k;smithi181=c;smithi181=f;smithi181=i;smithi181=l;count:12 2023-03-09T14:45:37.530 INFO:teuthology.orchestra.run.smithi038.stdout:mgr 3/3 14s ago 6m smithi038=x;smithi122=y;smithi181=z;count:3 2023-03-09T14:45:37.530 INFO:teuthology.orchestra.run.smithi038.stdout:mon 3/3 14s ago 6m smithi038:172.21.15.38=a;smithi122:172.21.15.122=b;smithi181:172.21.15.181=c;count:3 2023-03-09T14:45:37.531 INFO:teuthology.orchestra.run.smithi038.stdout:osd 12 14s ago - 2023-03-09T14:45:37.532 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:37.529+0000 7fce7eac1700 1 -- 172.21.15.38:0/1842924882 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fce6809f400 msgr2=0x7fce680a18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:37.532 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:37.529+0000 7fce7eac1700 1 --2- 172.21.15.38:0/1842924882 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fce6809f400 0x7fce680a18b0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fce64000c00 tx=0x7fce64005fb0 comp rx=0 tx=0).stop 2023-03-09T14:45:37.532 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:37.529+0000 7fce7eac1700 1 -- 172.21.15.38:0/1842924882 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fce780fffd0 msgr2=0x7fce78195d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:37.532 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:37.529+0000 7fce7eac1700 1 --2- 172.21.15.38:0/1842924882 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fce780fffd0 0x7fce78195d70 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fce6c00d980 tx=0x7fce6c00dd40 comp rx=0 tx=0).stop 2023-03-09T14:45:37.532 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:37.529+0000 7fce7eac1700 1 -- 172.21.15.38:0/1842924882 shutdown_connections 2023-03-09T14:45:37.532 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:37.529+0000 7fce7eac1700 1 --2- 172.21.15.38:0/1842924882 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fce780fffd0 0x7fce78195d70 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:37.532 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:37.529+0000 7fce7eac1700 1 --2- 172.21.15.38:0/1842924882 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fce6809f400 0x7fce680a18b0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:37.533 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:37.529+0000 7fce7eac1700 1 --2- 172.21.15.38:0/1842924882 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fce78101b50 0x7fce7819d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:37.533 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:37.529+0000 7fce7eac1700 1 --2- 172.21.15.38:0/1842924882 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fce781011c0 0x7fce781962b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:37.533 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:37.529+0000 7fce7eac1700 1 -- 172.21.15.38:0/1842924882 >> 172.21.15.38:0/1842924882 conn(0x7fce780fb5b0 msgr2=0x7fce78104700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:37.533 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:37.529+0000 7fce7eac1700 1 -- 172.21.15.38:0/1842924882 shutdown_connections 2023-03-09T14:45:37.533 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:37.530+0000 7fce7eac1700 1 -- 172.21.15.38:0/1842924882 wait complete. 2023-03-09T14:45:37.956 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- bash -c 'ceph orch host ls' 2023-03-09T14:45:38.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:37 smithi122 ceph-mon[93869]: pgmap v300: 129 pgs: 129 active+clean; 636 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2023-03-09T14:45:38.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:37 smithi181 ceph-mon[94189]: pgmap v300: 129 pgs: 129 active+clean; 636 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2023-03-09T14:45:38.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:37 smithi038 ceph-mon[92422]: pgmap v300: 129 pgs: 129 active+clean; 636 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2023-03-09T14:45:38.609 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.a/config 2023-03-09T14:45:39.130 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:38 smithi038 ceph-mon[92422]: from='client.15210 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:45:39.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:38 smithi122 ceph-mon[93869]: from='client.15210 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:45:39.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:38 smithi181 ceph-mon[94189]: from='client.15210 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:45:39.812 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:39.807+0000 7feccb0ea700 1 -- 172.21.15.38:0/3406961102 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fecc41011a0 msgr2=0x7fecc41015f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:39.812 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:39.807+0000 7feccb0ea700 1 --2- 172.21.15.38:0/3406961102 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fecc41011a0 0x7fecc41015f0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fecb0009990 tx=0x7fecb0009ca0 comp rx=0 tx=0).stop 2023-03-09T14:45:39.813 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:39.807+0000 7feccb0ea700 1 -- 172.21.15.38:0/3406961102 shutdown_connections 2023-03-09T14:45:39.813 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:39.807+0000 7feccb0ea700 1 --2- 172.21.15.38:0/3406961102 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fecc4101b30 0x7fecc41083f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:39.813 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:39.807+0000 7feccb0ea700 1 --2- 172.21.15.38:0/3406961102 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fecc40fffb0 0x7fecc41003c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:39.813 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:39.807+0000 7feccb0ea700 1 --2- 172.21.15.38:0/3406961102 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fecc41011a0 0x7fecc41015f0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:39.813 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:39.807+0000 7feccb0ea700 1 -- 172.21.15.38:0/3406961102 >> 172.21.15.38:0/3406961102 conn(0x7fecc40fb590 msgr2=0x7fecc40fda00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:39.813 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:39.807+0000 7feccb0ea700 1 -- 172.21.15.38:0/3406961102 shutdown_connections 2023-03-09T14:45:39.813 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:39.807+0000 7feccb0ea700 1 -- 172.21.15.38:0/3406961102 wait complete. 2023-03-09T14:45:39.814 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:39.808+0000 7feccb0ea700 1 Processor -- start 2023-03-09T14:45:39.814 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:39.808+0000 7feccb0ea700 1 -- start start 2023-03-09T14:45:39.814 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:39.808+0000 7feccb0ea700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fecc40fffb0 0x7fecc4195d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:39.814 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:39.808+0000 7feccb0ea700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fecc41011a0 0x7fecc4196280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:39.814 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:39.808+0000 7feccb0ea700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fecc4101b30 0x7fecc419d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:39.814 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:39.808+0000 7feccb0ea700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecc419d880 con 0x7fecc4101b30 2023-03-09T14:45:39.814 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:39.808+0000 7feccb0ea700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecc419d9f0 con 0x7fecc41011a0 2023-03-09T14:45:39.815 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:39.808+0000 7feccb0ea700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecc419dbb0 con 0x7fecc40fffb0 2023-03-09T14:45:39.815 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:39.808+0000 7fecc9687700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fecc4101b30 0x7fecc419d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:39.815 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:39.808+0000 7fecc9687700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fecc4101b30 0x7fecc419d2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37958/0 (socket says 172.21.15.38:37958) 2023-03-09T14:45:39.815 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:39.808+0000 7fecc9687700 1 -- 172.21.15.38:0/2516765815 learned_addr learned my addr 172.21.15.38:0/2516765815 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:39.815 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:39.808+0000 7fecc8e86700 1 --2- 172.21.15.38:0/2516765815 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fecc40fffb0 0x7fecc4195d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:39.815 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:39.808+0000 7fecc3fff700 1 --2- 172.21.15.38:0/2516765815 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fecc41011a0 0x7fecc4196280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:39.815 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:39.808+0000 7fecc9687700 1 -- 172.21.15.38:0/2516765815 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fecc40fffb0 msgr2=0x7fecc4195d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:39.815 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:39.808+0000 7fecc9687700 1 --2- 172.21.15.38:0/2516765815 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fecc40fffb0 0x7fecc4195d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:39.816 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:39.809+0000 7fecc9687700 1 -- 172.21.15.38:0/2516765815 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fecc41011a0 msgr2=0x7fecc4196280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:39.816 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:39.809+0000 7fecc9687700 1 --2- 172.21.15.38:0/2516765815 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fecc41011a0 0x7fecc4196280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:39.816 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:39.809+0000 7fecc9687700 1 -- 172.21.15.38:0/2516765815 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecb00095f0 con 0x7fecc4101b30 2023-03-09T14:45:39.816 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:39.809+0000 7fecc9687700 1 --2- 172.21.15.38:0/2516765815 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fecc4101b30 0x7fecc419d2b0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fecbc00ef30 tx=0x7fecbc00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:39.816 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:39.809+0000 7fecc1ffb700 1 -- 172.21.15.38:0/2516765815 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecbc018070 con 0x7fecc4101b30 2023-03-09T14:45:39.817 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:39.809+0000 7fecc1ffb700 1 -- 172.21.15.38:0/2516765815 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fecbc00f040 con 0x7fecc4101b30 2023-03-09T14:45:39.817 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:39.809+0000 7feccb0ea700 1 -- 172.21.15.38:0/2516765815 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecc410b910 con 0x7fecc4101b30 2023-03-09T14:45:39.817 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:39.809+0000 7fecc1ffb700 1 -- 172.21.15.38:0/2516765815 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecbc00cdb0 con 0x7fecc4101b30 2023-03-09T14:45:39.817 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:39.809+0000 7feccb0ea700 1 -- 172.21.15.38:0/2516765815 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecc410b780 con 0x7fecc4101b30 2023-03-09T14:45:39.818 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:39.810+0000 7feccb0ea700 1 -- 172.21.15.38:0/2516765815 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecc41075a0 con 0x7fecc4101b30 2023-03-09T14:45:39.818 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:39.811+0000 7fecc1ffb700 1 -- 172.21.15.38:0/2516765815 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 133238+0+0 (secure 0 0 0) 0x7fecbc003a00 con 0x7fecc4101b30 2023-03-09T14:45:39.818 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:39.812+0000 7fecc1ffb700 1 --2- 172.21.15.38:0/2516765815 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fecb409f480 0x7fecb40a1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:39.818 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:39.812+0000 7fecc8e86700 1 --2- 172.21.15.38:0/2516765815 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fecb409f480 0x7fecb40a1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:39.819 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:39.812+0000 7fecc1ffb700 1 -- 172.21.15.38:0/2516765815 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fecbc0c9ed0 con 0x7fecc4101b30 2023-03-09T14:45:39.819 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:39.812+0000 7fecc8e86700 1 --2- 172.21.15.38:0/2516765815 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fecb409f480 0x7fecb40a1930 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fecc4197210 tx=0x7fecb800b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:39.819 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:39.816+0000 7fecc1ffb700 1 -- 172.21.15.38:0/2516765815 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7fecbc093f70 con 0x7fecc4101b30 2023-03-09T14:45:40.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:39 smithi038 ceph-mon[92422]: pgmap v301: 129 pgs: 129 active+clean; 636 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T14:45:40.024 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:40.022+0000 7feccb0ea700 1 -- 172.21.15.38:0/2516765815 --> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fecc4061920 con 0x7fecb409f480 2023-03-09T14:45:40.026 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:40.023+0000 7fecc1ffb700 1 -- 172.21.15.38:0/2516765815 <== mgr.14150 v2:172.21.15.38:6800/3328887328 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+191 (secure 0 0 0) 0x7fecc4061920 con 0x7fecb409f480 2023-03-09T14:45:40.026 INFO:teuthology.orchestra.run.smithi038.stdout:HOST ADDR LABELS STATUS 2023-03-09T14:45:40.026 INFO:teuthology.orchestra.run.smithi038.stdout:smithi038 172.21.15.38 2023-03-09T14:45:40.026 INFO:teuthology.orchestra.run.smithi038.stdout:smithi122 172.21.15.122 2023-03-09T14:45:40.026 INFO:teuthology.orchestra.run.smithi038.stdout:smithi181 172.21.15.181 2023-03-09T14:45:40.027 INFO:teuthology.orchestra.run.smithi038.stdout:3 hosts in cluster 2023-03-09T14:45:40.028 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:40.026+0000 7feccb0ea700 1 -- 172.21.15.38:0/2516765815 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fecb409f480 msgr2=0x7fecb40a1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:40.028 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:40.026+0000 7feccb0ea700 1 --2- 172.21.15.38:0/2516765815 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fecb409f480 0x7fecb40a1930 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fecc4197210 tx=0x7fecb800b410 comp rx=0 tx=0).stop 2023-03-09T14:45:40.028 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:40.026+0000 7feccb0ea700 1 -- 172.21.15.38:0/2516765815 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fecc4101b30 msgr2=0x7fecc419d2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:40.028 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:40.026+0000 7feccb0ea700 1 --2- 172.21.15.38:0/2516765815 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fecc4101b30 0x7fecc419d2b0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fecbc00ef30 tx=0x7fecbc00c3a0 comp rx=0 tx=0).stop 2023-03-09T14:45:40.029 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:40.026+0000 7feccb0ea700 1 -- 172.21.15.38:0/2516765815 shutdown_connections 2023-03-09T14:45:40.029 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:40.026+0000 7feccb0ea700 1 --2- 172.21.15.38:0/2516765815 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fecc4101b30 0x7fecc419d2b0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:40.029 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:40.026+0000 7feccb0ea700 1 --2- 172.21.15.38:0/2516765815 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fecb409f480 0x7fecb40a1930 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:40.029 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:40.026+0000 7feccb0ea700 1 --2- 172.21.15.38:0/2516765815 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fecc40fffb0 0x7fecc4195d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:40.029 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:40.026+0000 7feccb0ea700 1 --2- 172.21.15.38:0/2516765815 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fecc41011a0 0x7fecc4196280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:40.029 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:40.026+0000 7feccb0ea700 1 -- 172.21.15.38:0/2516765815 >> 172.21.15.38:0/2516765815 conn(0x7fecc40fb590 msgr2=0x7fecc41046e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:40.029 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:40.026+0000 7feccb0ea700 1 -- 172.21.15.38:0/2516765815 shutdown_connections 2023-03-09T14:45:40.030 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:40.026+0000 7feccb0ea700 1 -- 172.21.15.38:0/2516765815 wait complete. 2023-03-09T14:45:40.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:39 smithi122 ceph-mon[93869]: pgmap v301: 129 pgs: 129 active+clean; 636 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T14:45:40.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:39 smithi181 ceph-mon[94189]: pgmap v301: 129 pgs: 129 active+clean; 636 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T14:45:40.466 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- bash -c 'ceph orch device ls' 2023-03-09T14:45:41.120 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.a/config 2023-03-09T14:45:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:40 smithi122 ceph-mon[93869]: from='client.15216 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:45:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:40 smithi181 ceph-mon[94189]: from='client.15216 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:45:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:40 smithi038 ceph-mon[92422]: from='client.15216 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:45:41.889 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:41 smithi038 ceph-mon[92422]: pgmap v302: 129 pgs: 129 active+clean; 636 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-03-09T14:45:42.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:41 smithi122 ceph-mon[93869]: pgmap v302: 129 pgs: 129 active+clean; 636 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-03-09T14:45:42.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:41 smithi181 ceph-mon[94189]: pgmap v302: 129 pgs: 129 active+clean; 636 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-03-09T14:45:42.351 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:42.346+0000 7f7338464700 1 -- 172.21.15.38:0/3050733910 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f73300ffff0 msgr2=0x7f7330100460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:42.351 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:42.346+0000 7f7338464700 1 --2- 172.21.15.38:0/3050733910 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f73300ffff0 0x7f7330100460 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f7320009990 tx=0x7f7320009ca0 comp rx=0 tx=0).stop 2023-03-09T14:45:42.351 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:42.347+0000 7f7338464700 1 -- 172.21.15.38:0/3050733910 shutdown_connections 2023-03-09T14:45:42.352 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:42.347+0000 7f7338464700 1 --2- 172.21.15.38:0/3050733910 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f73301009a0 0x7f73301072d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:42.352 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:42.347+0000 7f7338464700 1 --2- 172.21.15.38:0/3050733910 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f73300feea0 0x7f73300ff2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:42.352 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:42.347+0000 7f7338464700 1 --2- 172.21.15.38:0/3050733910 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f73300ffff0 0x7f7330100460 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:42.352 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:42.347+0000 7f7338464700 1 -- 172.21.15.38:0/3050733910 >> 172.21.15.38:0/3050733910 conn(0x7f73300fa480 msgr2=0x7f73300fc8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:42.352 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:42.347+0000 7f7338464700 1 -- 172.21.15.38:0/3050733910 shutdown_connections 2023-03-09T14:45:42.352 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:42.347+0000 7f7338464700 1 -- 172.21.15.38:0/3050733910 wait complete. 2023-03-09T14:45:42.352 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:42.347+0000 7f7338464700 1 Processor -- start 2023-03-09T14:45:42.352 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:42.347+0000 7f7338464700 1 -- start start 2023-03-09T14:45:42.353 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:42.348+0000 7f7338464700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f73300feea0 0x7f7330102e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:42.353 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:42.348+0000 7f7338464700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f73300ffff0 0x7f7330101460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:42.353 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:42.348+0000 7f7338464700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f73301009a0 0x7f73301019a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:42.353 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:42.348+0000 7f7338464700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7330102140 con 0x7f73301009a0 2023-03-09T14:45:42.353 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:42.348+0000 7f7338464700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73301022b0 con 0x7f73300ffff0 2023-03-09T14:45:42.353 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:42.348+0000 7f7338464700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f733019fd50 con 0x7f73300feea0 2023-03-09T14:45:42.353 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:42.348+0000 7f7336a01700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f73301009a0 0x7f73301019a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:42.354 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:42.348+0000 7f7336a01700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f73301009a0 0x7f73301019a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37970/0 (socket says 172.21.15.38:37970) 2023-03-09T14:45:42.354 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:42.348+0000 7f7336a01700 1 -- 172.21.15.38:0/1519615789 learned_addr learned my addr 172.21.15.38:0/1519615789 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:42.354 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:42.348+0000 7f7336200700 1 --2- 172.21.15.38:0/1519615789 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f73300feea0 0x7f7330102e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:42.354 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:42.348+0000 7f73359ff700 1 --2- 172.21.15.38:0/1519615789 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f73300ffff0 0x7f7330101460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:42.354 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:42.348+0000 7f7336a01700 1 -- 172.21.15.38:0/1519615789 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f73300feea0 msgr2=0x7f7330102e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:42.354 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:42.348+0000 7f7336a01700 1 --2- 172.21.15.38:0/1519615789 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f73300feea0 0x7f7330102e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:42.354 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:42.348+0000 7f7336a01700 1 -- 172.21.15.38:0/1519615789 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f73300ffff0 msgr2=0x7f7330101460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:42.354 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:42.348+0000 7f7336a01700 1 --2- 172.21.15.38:0/1519615789 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f73300ffff0 0x7f7330101460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:42.355 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:42.348+0000 7f7336a01700 1 -- 172.21.15.38:0/1519615789 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73200095f0 con 0x7f73301009a0 2023-03-09T14:45:42.355 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:42.348+0000 7f7336a01700 1 --2- 172.21.15.38:0/1519615789 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f73301009a0 0x7f73301019a0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f7328007ee0 tx=0x7f732800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:42.355 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:42.348+0000 7f73277fe700 1 -- 172.21.15.38:0/1519615789 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7328017070 con 0x7f73301009a0 2023-03-09T14:45:42.355 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:42.349+0000 7f73277fe700 1 -- 172.21.15.38:0/1519615789 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f732800f040 con 0x7f73301009a0 2023-03-09T14:45:42.355 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:42.349+0000 7f7338464700 1 -- 172.21.15.38:0/1519615789 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f733010a830 con 0x7f73301009a0 2023-03-09T14:45:42.355 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:42.349+0000 7f73277fe700 1 -- 172.21.15.38:0/1519615789 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7328005580 con 0x7f73301009a0 2023-03-09T14:45:42.355 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:42.349+0000 7f7338464700 1 -- 172.21.15.38:0/1519615789 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f733010a6a0 con 0x7f73301009a0 2023-03-09T14:45:42.356 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:42.350+0000 7f7338464700 1 -- 172.21.15.38:0/1519615789 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73301a0380 con 0x7f73301009a0 2023-03-09T14:45:42.356 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:42.351+0000 7f73277fe700 1 -- 172.21.15.38:0/1519615789 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 133238+0+0 (secure 0 0 0) 0x7f7328004c40 con 0x7f73301009a0 2023-03-09T14:45:42.356 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:42.351+0000 7f73277fe700 1 --2- 172.21.15.38:0/1519615789 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f731c09f400 0x7f731c0a18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:42.357 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:42.351+0000 7f73277fe700 1 -- 172.21.15.38:0/1519615789 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f73280c99f0 con 0x7f73301009a0 2023-03-09T14:45:42.357 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:42.351+0000 7f7336200700 1 --2- 172.21.15.38:0/1519615789 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f731c09f400 0x7f731c0a18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:42.357 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:42.352+0000 7f7336200700 1 --2- 172.21.15.38:0/1519615789 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f731c09f400 0x7f731c0a18b0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f732c005950 tx=0x7f732c00b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:42.357 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:42.356+0000 7f73277fe700 1 -- 172.21.15.38:0/1519615789 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f7328093a60 con 0x7f73301009a0 2023-03-09T14:45:42.566 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:42.564+0000 7f7338464700 1 -- 172.21.15.38:0/1519615789 --> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f73300692c0 con 0x7f731c09f400 2023-03-09T14:45:42.568 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:42.566+0000 7f73277fe700 1 -- 172.21.15.38:0/1519615789 <== mgr.14150 v2:172.21.15.38:6800/3328887328 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+492 (secure 0 0 0) 0x7f73300692c0 con 0x7f731c09f400 2023-03-09T14:45:42.568 INFO:teuthology.orchestra.run.smithi038.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2023-03-09T14:45:42.569 INFO:teuthology.orchestra.run.smithi038.stdout:smithi038 /dev/nvme0n1 ssd INTEL_SSDPEDKE040T7_BTLE736103BB4P0KGN 4000G No 2m ago LVM detected, locked 2023-03-09T14:45:42.569 INFO:teuthology.orchestra.run.smithi038.stdout:smithi122 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT6204010U400BGN 400G No 2m ago LVM detected, locked 2023-03-09T14:45:42.569 INFO:teuthology.orchestra.run.smithi038.stdout:smithi181 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT6246000E400BGN 400G No 2m ago LVM detected, locked 2023-03-09T14:45:42.570 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:42.568+0000 7f7338464700 1 -- 172.21.15.38:0/1519615789 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f731c09f400 msgr2=0x7f731c0a18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:42.571 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:42.568+0000 7f7338464700 1 --2- 172.21.15.38:0/1519615789 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f731c09f400 0x7f731c0a18b0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f732c005950 tx=0x7f732c00b410 comp rx=0 tx=0).stop 2023-03-09T14:45:42.571 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:42.568+0000 7f7338464700 1 -- 172.21.15.38:0/1519615789 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f73301009a0 msgr2=0x7f73301019a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:42.571 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:42.568+0000 7f7338464700 1 --2- 172.21.15.38:0/1519615789 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f73301009a0 0x7f73301019a0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f7328007ee0 tx=0x7f732800d3b0 comp rx=0 tx=0).stop 2023-03-09T14:45:42.571 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:42.568+0000 7f7338464700 1 -- 172.21.15.38:0/1519615789 shutdown_connections 2023-03-09T14:45:42.571 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:42.568+0000 7f7338464700 1 --2- 172.21.15.38:0/1519615789 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f73301009a0 0x7f73301019a0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:42.571 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:42.568+0000 7f7338464700 1 --2- 172.21.15.38:0/1519615789 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f731c09f400 0x7f731c0a18b0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:42.571 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:42.568+0000 7f7338464700 1 --2- 172.21.15.38:0/1519615789 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f73300feea0 0x7f7330102e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:42.572 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:42.568+0000 7f7338464700 1 --2- 172.21.15.38:0/1519615789 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f73300ffff0 0x7f7330101460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:42.572 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:42.568+0000 7f7338464700 1 -- 172.21.15.38:0/1519615789 >> 172.21.15.38:0/1519615789 conn(0x7f73300fa480 msgr2=0x7f7330068bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:42.572 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:42.568+0000 7f7338464700 1 -- 172.21.15.38:0/1519615789 shutdown_connections 2023-03-09T14:45:42.572 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:42.568+0000 7f7338464700 1 -- 172.21.15.38:0/1519615789 wait complete. 2023-03-09T14:45:42.987 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-03-09T14:45:43.002 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi038.front.sepia.ceph.com 2023-03-09T14:45:43.003 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- bash -c 'ceph fs dump' 2023-03-09T14:45:43.657 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.a/config 2023-03-09T14:45:43.878 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:43 smithi038 ceph-mon[92422]: from='client.15222 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:45:43.878 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:43 smithi038 ceph-mon[92422]: pgmap v303: 129 pgs: 129 active+clean; 636 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T14:45:43.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:43 smithi122 ceph-mon[93869]: from='client.15222 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:45:43.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:43 smithi122 ceph-mon[93869]: pgmap v303: 129 pgs: 129 active+clean; 636 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T14:45:43.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:43 smithi181 ceph-mon[94189]: from='client.15222 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:45:43.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:43 smithi181 ceph-mon[94189]: pgmap v303: 129 pgs: 129 active+clean; 636 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T14:45:45.007 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:45.003+0000 7f4b7fe1e700 1 -- 172.21.15.38:0/3594335350 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b78100ae0 msgr2=0x7f4b78102ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:45.008 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:45.003+0000 7f4b7fe1e700 1 --2- 172.21.15.38:0/3594335350 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b78100ae0 0x7f4b78102ec0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f4b74009990 tx=0x7f4b74009ca0 comp rx=0 tx=0).stop 2023-03-09T14:45:45.008 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:45.003+0000 7f4b7fe1e700 1 -- 172.21.15.38:0/3594335350 shutdown_connections 2023-03-09T14:45:45.008 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:45.003+0000 7f4b7fe1e700 1 --2- 172.21.15.38:0/3594335350 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b78100ae0 0x7f4b78102ec0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:45.008 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:45.003+0000 7f4b7fe1e700 1 --2- 172.21.15.38:0/3594335350 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4b78105d20 0x7f4b78108100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:45.008 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:45.003+0000 7f4b7fe1e700 1 --2- 172.21.15.38:0/3594335350 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4b78103400 0x7f4b781057e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:45.009 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:45.003+0000 7f4b7fe1e700 1 -- 172.21.15.38:0/3594335350 >> 172.21.15.38:0/3594335350 conn(0x7f4b780fa480 msgr2=0x7f4b780fc8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:45.009 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:45.003+0000 7f4b7fe1e700 1 -- 172.21.15.38:0/3594335350 shutdown_connections 2023-03-09T14:45:45.009 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:45.003+0000 7f4b7fe1e700 1 -- 172.21.15.38:0/3594335350 wait complete. 2023-03-09T14:45:45.009 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:45.004+0000 7f4b7fe1e700 1 Processor -- start 2023-03-09T14:45:45.009 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:45.004+0000 7f4b7fe1e700 1 -- start start 2023-03-09T14:45:45.009 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:45.004+0000 7f4b7fe1e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b78100ae0 0x7f4b78100450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:45.009 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:45.004+0000 7f4b7fe1e700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4b78103400 0x7f4b780fea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:45.010 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:45.004+0000 7f4b7fe1e700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4b78105d20 0x7f4b780fefd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:45.010 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:45.004+0000 7f4b7fe1e700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b780ff770 con 0x7f4b78100ae0 2023-03-09T14:45:45.010 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:45.004+0000 7f4b7fe1e700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b780ff8e0 con 0x7f4b78103400 2023-03-09T14:45:45.010 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:45.004+0000 7f4b7fe1e700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b7819fd50 con 0x7f4b78105d20 2023-03-09T14:45:45.010 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:45.004+0000 7f4b7dbba700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b78100ae0 0x7f4b78100450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:45.010 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:45.004+0000 7f4b7dbba700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b78100ae0 0x7f4b78100450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37980/0 (socket says 172.21.15.38:37980) 2023-03-09T14:45:45.010 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:45.004+0000 7f4b7dbba700 1 -- 172.21.15.38:0/1334303327 learned_addr learned my addr 172.21.15.38:0/1334303327 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:45.011 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:45.004+0000 7f4b7e3bb700 1 --2- 172.21.15.38:0/1334303327 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4b78105d20 0x7f4b780fefd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:45.011 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:45.004+0000 7f4b7d3b9700 1 --2- 172.21.15.38:0/1334303327 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4b78103400 0x7f4b780fea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:45.011 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:45.005+0000 7f4b7dbba700 1 -- 172.21.15.38:0/1334303327 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4b78105d20 msgr2=0x7f4b780fefd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:45.011 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:45.005+0000 7f4b7dbba700 1 --2- 172.21.15.38:0/1334303327 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4b78105d20 0x7f4b780fefd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:45.011 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:45.005+0000 7f4b7dbba700 1 -- 172.21.15.38:0/1334303327 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4b78103400 msgr2=0x7f4b780fea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:45.011 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:45.005+0000 7f4b7dbba700 1 --2- 172.21.15.38:0/1334303327 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4b78103400 0x7f4b780fea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:45.011 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:45.005+0000 7f4b7dbba700 1 -- 172.21.15.38:0/1334303327 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b740095f0 con 0x7f4b78100ae0 2023-03-09T14:45:45.012 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:45.005+0000 7f4b7dbba700 1 --2- 172.21.15.38:0/1334303327 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b78100ae0 0x7f4b78100450 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f4b74000c00 tx=0x7f4b7400f710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:45.012 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:45.005+0000 7f4b6affd700 1 -- 172.21.15.38:0/1334303327 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b7401d070 con 0x7f4b78100ae0 2023-03-09T14:45:45.012 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:45.005+0000 7f4b6affd700 1 -- 172.21.15.38:0/1334303327 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4b7400fb50 con 0x7f4b78100ae0 2023-03-09T14:45:45.012 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:45.005+0000 7f4b7fe1e700 1 -- 172.21.15.38:0/1334303327 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b7810b8e0 con 0x7f4b78100ae0 2023-03-09T14:45:45.012 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:45.005+0000 7f4b6affd700 1 -- 172.21.15.38:0/1334303327 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b740173f0 con 0x7f4b78100ae0 2023-03-09T14:45:45.012 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:45.005+0000 7f4b7fe1e700 1 -- 172.21.15.38:0/1334303327 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b7810b750 con 0x7f4b78100ae0 2023-03-09T14:45:45.013 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:45.006+0000 7f4b7fe1e700 1 -- 172.21.15.38:0/1334303327 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b780fbff0 con 0x7f4b78100ae0 2023-03-09T14:45:45.013 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:45.007+0000 7f4b6affd700 1 -- 172.21.15.38:0/1334303327 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 17) v1 ==== 133238+0+0 (secure 0 0 0) 0x7f4b74029030 con 0x7f4b78100ae0 2023-03-09T14:45:45.014 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:45.008+0000 7f4b6affd700 1 --2- 172.21.15.38:0/1334303327 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f4b6409f400 0x7f4b640a18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:45.014 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:45.008+0000 7f4b7d3b9700 1 --2- 172.21.15.38:0/1334303327 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f4b6409f400 0x7f4b640a18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:45.014 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:45.008+0000 7f4b6affd700 1 -- 172.21.15.38:0/1334303327 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f4b740cb6f0 con 0x7f4b78100ae0 2023-03-09T14:45:45.014 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:45.008+0000 7f4b7d3b9700 1 --2- 172.21.15.38:0/1334303327 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f4b6409f400 0x7f4b640a18b0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c005950 tx=0x7f4b6c00b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:45.014 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:45.011+0000 7f4b6affd700 1 -- 172.21.15.38:0/1334303327 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f4b740957f0 con 0x7f4b78100ae0 2023-03-09T14:45:45.260 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:45.258+0000 7f4b7fe1e700 1 -- 172.21.15.38:0/1334303327 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f4b7802cef0 con 0x7f4b78100ae0 2023-03-09T14:45:45.261 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:45.258+0000 7f4b6affd700 1 -- 172.21.15.38:0/1334303327 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 23 v23) v1 ==== 76+0+2896 (secure 0 0 0) 0x7f4b74027b60 con 0x7f4b78100ae0 2023-03-09T14:45:45.261 INFO:teuthology.orchestra.run.smithi038.stdout:e23 2023-03-09T14:45:45.262 INFO:teuthology.orchestra.run.smithi038.stdout:enable_multiple, ever_enabled_multiple: 1,1 2023-03-09T14:45:45.262 INFO:teuthology.orchestra.run.smithi038.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-03-09T14:45:45.262 INFO:teuthology.orchestra.run.smithi038.stdout:legacy client fscid: 1 2023-03-09T14:45:45.262 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:45:45.262 INFO:teuthology.orchestra.run.smithi038.stdout:Filesystem 'cephfs' (1) 2023-03-09T14:45:45.262 INFO:teuthology.orchestra.run.smithi038.stdout:fs_name cephfs 2023-03-09T14:45:45.262 INFO:teuthology.orchestra.run.smithi038.stdout:epoch 23 2023-03-09T14:45:45.263 INFO:teuthology.orchestra.run.smithi038.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2023-03-09T14:45:45.263 INFO:teuthology.orchestra.run.smithi038.stdout:created 2023-03-09T14:44:19.083770+0000 2023-03-09T14:45:45.263 INFO:teuthology.orchestra.run.smithi038.stdout:modified 2023-03-09T14:44:39.526150+0000 2023-03-09T14:45:45.263 INFO:teuthology.orchestra.run.smithi038.stdout:tableserver 0 2023-03-09T14:45:45.263 INFO:teuthology.orchestra.run.smithi038.stdout:root 0 2023-03-09T14:45:45.263 INFO:teuthology.orchestra.run.smithi038.stdout:session_timeout 300 2023-03-09T14:45:45.263 INFO:teuthology.orchestra.run.smithi038.stdout:session_autoclose 300 2023-03-09T14:45:45.264 INFO:teuthology.orchestra.run.smithi038.stdout:max_file_size 1099511627776 2023-03-09T14:45:45.264 INFO:teuthology.orchestra.run.smithi038.stdout:max_xattr_size 65536 2023-03-09T14:45:45.264 INFO:teuthology.orchestra.run.smithi038.stdout:required_client_features {} 2023-03-09T14:45:45.264 INFO:teuthology.orchestra.run.smithi038.stdout:last_failure 0 2023-03-09T14:45:45.264 INFO:teuthology.orchestra.run.smithi038.stdout:last_failure_osd_epoch 0 2023-03-09T14:45:45.264 INFO:teuthology.orchestra.run.smithi038.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-03-09T14:45:45.264 INFO:teuthology.orchestra.run.smithi038.stdout:max_mds 3 2023-03-09T14:45:45.264 INFO:teuthology.orchestra.run.smithi038.stdout:in 0,1,2 2023-03-09T14:45:45.265 INFO:teuthology.orchestra.run.smithi038.stdout:up {0=24530,1=24494,2=24478} 2023-03-09T14:45:45.265 INFO:teuthology.orchestra.run.smithi038.stdout:failed 2023-03-09T14:45:45.265 INFO:teuthology.orchestra.run.smithi038.stdout:damaged 2023-03-09T14:45:45.265 INFO:teuthology.orchestra.run.smithi038.stdout:stopped 2023-03-09T14:45:45.265 INFO:teuthology.orchestra.run.smithi038.stdout:data_pools [3,4] 2023-03-09T14:45:45.265 INFO:teuthology.orchestra.run.smithi038.stdout:metadata_pool 2 2023-03-09T14:45:45.265 INFO:teuthology.orchestra.run.smithi038.stdout:inline_data disabled 2023-03-09T14:45:45.266 INFO:teuthology.orchestra.run.smithi038.stdout:balancer 2023-03-09T14:45:45.266 INFO:teuthology.orchestra.run.smithi038.stdout:bal_rank_mask -1 2023-03-09T14:45:45.266 INFO:teuthology.orchestra.run.smithi038.stdout:standby_count_wanted 0 2023-03-09T14:45:45.266 INFO:teuthology.orchestra.run.smithi038.stdout:[mds.b{0:24530} state up:active seq 3 addr [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] compat {c=[1],r=[1],i=[7ff]}] 2023-03-09T14:45:45.266 INFO:teuthology.orchestra.run.smithi038.stdout:[mds.e{0:24464} state up:standby-replay seq 1 addr [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] compat {c=[1],r=[1],i=[7ff]}] 2023-03-09T14:45:45.266 INFO:teuthology.orchestra.run.smithi038.stdout:[mds.h{1:24494} state up:active seq 10 addr [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] compat {c=[1],r=[1],i=[7ff]}] 2023-03-09T14:45:45.266 INFO:teuthology.orchestra.run.smithi038.stdout:[mds.c{1:24460} state up:standby-replay seq 1 addr [v2:172.21.15.181:6836/2942560181,v1:172.21.15.181:6837/2942560181] compat {c=[1],r=[1],i=[7ff]}] 2023-03-09T14:45:45.267 INFO:teuthology.orchestra.run.smithi038.stdout:[mds.i{2:24478} state up:active seq 9 addr [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] compat {c=[1],r=[1],i=[7ff]}] 2023-03-09T14:45:45.267 INFO:teuthology.orchestra.run.smithi038.stdout:[mds.f{2:24448} state up:standby-replay seq 1 addr [v2:172.21.15.181:6834/2576255861,v1:172.21.15.181:6835/2576255861] compat {c=[1],r=[1],i=[7ff]}] 2023-03-09T14:45:45.267 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:45:45.267 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:45:45.267 INFO:teuthology.orchestra.run.smithi038.stdout:Standby daemons: 2023-03-09T14:45:45.267 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:45:45.267 INFO:teuthology.orchestra.run.smithi038.stdout:[mds.j{-1:14544} state up:standby seq 1 addr [v2:172.21.15.38:6834/768918602,v1:172.21.15.38:6835/768918602] compat {c=[1],r=[1],i=[7ff]}] 2023-03-09T14:45:45.267 INFO:teuthology.orchestra.run.smithi038.stdout:[mds.d{-1:14550} state up:standby seq 1 addr [v2:172.21.15.38:6836/18381540,v1:172.21.15.38:6837/18381540] compat {c=[1],r=[1],i=[7ff]}] 2023-03-09T14:45:45.268 INFO:teuthology.orchestra.run.smithi038.stdout:[mds.g{-1:14610} state up:standby seq 1 addr [v2:172.21.15.38:6838/4139020066,v1:172.21.15.38:6839/4139020066] compat {c=[1],r=[1],i=[7ff]}] 2023-03-09T14:45:45.268 INFO:teuthology.orchestra.run.smithi038.stdout:[mds.a{-1:14622} state up:standby seq 1 addr [v2:172.21.15.38:6840/1397115322,v1:172.21.15.38:6841/1397115322] compat {c=[1],r=[1],i=[7ff]}] 2023-03-09T14:45:45.268 INFO:teuthology.orchestra.run.smithi038.stdout:[mds.l{-1:24406} state up:standby seq 1 addr [v2:172.21.15.181:6832/415305571,v1:172.21.15.181:6833/415305571] compat {c=[1],r=[1],i=[7ff]}] 2023-03-09T14:45:45.268 INFO:teuthology.orchestra.run.smithi038.stdout:[mds.k{-1:24437} state up:standby seq 1 addr [v2:172.21.15.122:6832/659300520,v1:172.21.15.122:6833/659300520] compat {c=[1],r=[1],i=[7ff]}] 2023-03-09T14:45:45.269 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:45.261+0000 7f4b7fe1e700 1 -- 172.21.15.38:0/1334303327 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f4b6409f400 msgr2=0x7f4b640a18b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:45.269 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:45.261+0000 7f4b7fe1e700 1 --2- 172.21.15.38:0/1334303327 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f4b6409f400 0x7f4b640a18b0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f4b6c005950 tx=0x7f4b6c00b410 comp rx=0 tx=0).stop 2023-03-09T14:45:45.269 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:45.261+0000 7f4b7fe1e700 1 -- 172.21.15.38:0/1334303327 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b78100ae0 msgr2=0x7f4b78100450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:45.269 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:45.261+0000 7f4b7fe1e700 1 --2- 172.21.15.38:0/1334303327 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b78100ae0 0x7f4b78100450 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f4b74000c00 tx=0x7f4b7400f710 comp rx=0 tx=0).stop 2023-03-09T14:45:45.269 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:45.261+0000 7f4b7fe1e700 1 -- 172.21.15.38:0/1334303327 shutdown_connections 2023-03-09T14:45:45.270 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:45.261+0000 7f4b7fe1e700 1 --2- 172.21.15.38:0/1334303327 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b78100ae0 0x7f4b78100450 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:45.270 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:45.261+0000 7f4b7fe1e700 1 --2- 172.21.15.38:0/1334303327 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f4b6409f400 0x7f4b640a18b0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:45.270 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:45.261+0000 7f4b7fe1e700 1 --2- 172.21.15.38:0/1334303327 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4b78105d20 0x7f4b780fefd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:45.270 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:45.261+0000 7f4b7fe1e700 1 --2- 172.21.15.38:0/1334303327 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4b78103400 0x7f4b780fea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:45.270 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:45.261+0000 7f4b7fe1e700 1 -- 172.21.15.38:0/1334303327 >> 172.21.15.38:0/1334303327 conn(0x7f4b780fa480 msgr2=0x7f4b78101800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:45.270 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:45.261+0000 7f4b7fe1e700 1 -- 172.21.15.38:0/1334303327 shutdown_connections 2023-03-09T14:45:45.270 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:45.261+0000 7f4b7fe1e700 1 -- 172.21.15.38:0/1334303327 wait complete. 2023-03-09T14:45:45.271 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 23 2023-03-09T14:45:45.620 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- bash -c 'ceph osd dump' 2023-03-09T14:45:46.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:45 smithi038 ceph-mon[92422]: pgmap v304: 129 pgs: 129 active+clean; 636 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-03-09T14:45:46.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:45 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/1334303327' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-03-09T14:45:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:45 smithi122 ceph-mon[93869]: pgmap v304: 129 pgs: 129 active+clean; 636 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-03-09T14:45:46.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:45 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/1334303327' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-03-09T14:45:46.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:45 smithi181 ceph-mon[94189]: pgmap v304: 129 pgs: 129 active+clean; 636 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-03-09T14:45:46.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:45 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/1334303327' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-03-09T14:45:46.281 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.a/config 2023-03-09T14:45:47.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:46 smithi122 ceph-mon[93869]: mgrmap e18: x(active, since 7m), standbys: z, y 2023-03-09T14:45:47.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:46 smithi181 ceph-mon[94189]: mgrmap e18: x(active, since 7m), standbys: z, y 2023-03-09T14:45:47.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:46 smithi038 ceph-mon[92422]: mgrmap e18: x(active, since 7m), standbys: z, y 2023-03-09T14:45:47.490 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:47.485+0000 7f1ccb216700 1 -- 172.21.15.38:0/728734604 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1cc40feea0 msgr2=0x7f1cc40ff2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:47.490 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:47.485+0000 7f1ccb216700 1 --2- 172.21.15.38:0/728734604 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1cc40feea0 0x7f1cc40ff2b0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f1cb8009990 tx=0x7f1cb8009ca0 comp rx=0 tx=0).stop 2023-03-09T14:45:47.490 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:47.486+0000 7f1ccb216700 1 -- 172.21.15.38:0/728734604 shutdown_connections 2023-03-09T14:45:47.490 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:47.486+0000 7f1ccb216700 1 --2- 172.21.15.38:0/728734604 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1cc40feea0 0x7f1cc40ff2b0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:47.490 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:47.486+0000 7f1ccb216700 1 --2- 172.21.15.38:0/728734604 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1cc41009a0 0x7f1cc41072d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:47.491 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:47.486+0000 7f1ccb216700 1 --2- 172.21.15.38:0/728734604 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1cc40ffff0 0x7f1cc4100460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:47.491 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:47.486+0000 7f1ccb216700 1 -- 172.21.15.38:0/728734604 >> 172.21.15.38:0/728734604 conn(0x7f1cc40fa480 msgr2=0x7f1cc40fc8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:47.491 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:47.486+0000 7f1ccb216700 1 -- 172.21.15.38:0/728734604 shutdown_connections 2023-03-09T14:45:47.491 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:47.486+0000 7f1ccb216700 1 -- 172.21.15.38:0/728734604 wait complete. 2023-03-09T14:45:47.491 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:47.486+0000 7f1ccb216700 1 Processor -- start 2023-03-09T14:45:47.491 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:47.486+0000 7f1ccb216700 1 -- start start 2023-03-09T14:45:47.491 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:47.486+0000 7f1ccb216700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1cc40feea0 0x7f1cc4195d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:47.492 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:47.486+0000 7f1ccb216700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1cc40ffff0 0x7f1cc41962b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:47.492 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:47.486+0000 7f1ccb216700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1cc41009a0 0x7f1cc419d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:47.492 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:47.486+0000 7f1ccb216700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc419d8f0 con 0x7f1cc40ffff0 2023-03-09T14:45:47.492 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:47.486+0000 7f1ccb216700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc419da30 con 0x7f1cc40feea0 2023-03-09T14:45:47.492 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:47.486+0000 7f1ccb216700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc419dbc0 con 0x7f1cc41009a0 2023-03-09T14:45:47.492 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:47.487+0000 7f1cc3fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1cc40ffff0 0x7f1cc41962b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:47.492 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:47.487+0000 7f1cc8fb2700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1cc40feea0 0x7f1cc4195d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:47.493 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:47.487+0000 7f1cc3fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1cc40ffff0 0x7f1cc41962b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:37994/0 (socket says 172.21.15.38:37994) 2023-03-09T14:45:47.493 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:47.487+0000 7f1cc3fff700 1 -- 172.21.15.38:0/1586731238 learned_addr learned my addr 172.21.15.38:0/1586731238 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:47.493 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:47.487+0000 7f1cc97b3700 1 --2- 172.21.15.38:0/1586731238 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1cc41009a0 0x7f1cc419d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:47.493 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:47.487+0000 7f1cc3fff700 1 -- 172.21.15.38:0/1586731238 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1cc41009a0 msgr2=0x7f1cc419d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:47.493 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:47.487+0000 7f1cc3fff700 1 --2- 172.21.15.38:0/1586731238 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1cc41009a0 0x7f1cc419d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:47.493 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:47.487+0000 7f1cc3fff700 1 -- 172.21.15.38:0/1586731238 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1cc40feea0 msgr2=0x7f1cc4195d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:47.493 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:47.487+0000 7f1cc3fff700 1 --2- 172.21.15.38:0/1586731238 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1cc40feea0 0x7f1cc4195d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:47.494 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:47.487+0000 7f1cc3fff700 1 -- 172.21.15.38:0/1586731238 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cb80095f0 con 0x7f1cc40ffff0 2023-03-09T14:45:47.494 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:47.487+0000 7f1cc3fff700 1 --2- 172.21.15.38:0/1586731238 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1cc40ffff0 0x7f1cc41962b0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f1cb000d950 tx=0x7f1cb000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:47.494 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:47.487+0000 7f1cc1ffb700 1 -- 172.21.15.38:0/1586731238 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb0009940 con 0x7f1cc40ffff0 2023-03-09T14:45:47.494 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:47.487+0000 7f1cc1ffb700 1 -- 172.21.15.38:0/1586731238 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1cb0010460 con 0x7f1cc40ffff0 2023-03-09T14:45:47.494 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:47.487+0000 7f1ccb216700 1 -- 172.21.15.38:0/1586731238 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cc410a880 con 0x7f1cc40ffff0 2023-03-09T14:45:47.494 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:47.487+0000 7f1cc1ffb700 1 -- 172.21.15.38:0/1586731238 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb000f620 con 0x7f1cc40ffff0 2023-03-09T14:45:47.494 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:47.488+0000 7f1ccb216700 1 -- 172.21.15.38:0/1586731238 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cc410a6f0 con 0x7f1cc40ffff0 2023-03-09T14:45:47.495 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:47.489+0000 7f1ccb216700 1 -- 172.21.15.38:0/1586731238 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cc419e230 con 0x7f1cc40ffff0 2023-03-09T14:45:47.496 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:47.490+0000 7f1cc1ffb700 1 -- 172.21.15.38:0/1586731238 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) v1 ==== 133198+0+0 (secure 0 0 0) 0x7f1cb0022070 con 0x7f1cc40ffff0 2023-03-09T14:45:47.496 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:47.490+0000 7f1cc1ffb700 1 --2- 172.21.15.38:0/1586731238 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f1cb409f3c0 0x7f1cb40a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:47.496 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:47.490+0000 7f1cc1ffb700 1 -- 172.21.15.38:0/1586731238 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f1cb00c9de0 con 0x7f1cc40ffff0 2023-03-09T14:45:47.496 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:47.490+0000 7f1cc8fb2700 1 --2- 172.21.15.38:0/1586731238 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f1cb409f3c0 0x7f1cb40a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:47.496 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:47.491+0000 7f1cc8fb2700 1 --2- 172.21.15.38:0/1586731238 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f1cb409f3c0 0x7f1cb40a1870 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f1cb800b5c0 tx=0x7f1cb8005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:47.497 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:47.494+0000 7f1cc1ffb700 1 -- 172.21.15.38:0/1586731238 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f1cb0093e90 con 0x7f1cc40ffff0 2023-03-09T14:45:47.694 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:47.692+0000 7f1ccb216700 1 -- 172.21.15.38:0/1586731238 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump"} v 0) v1 -- 0x7f1cc419e7e0 con 0x7f1cc40ffff0 2023-03-09T14:45:47.695 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:47.693+0000 7f1cc1ffb700 1 -- 172.21.15.38:0/1586731238 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v89) v1 ==== 56+0+5607 (secure 0 0 0) 0x7f1cb0020070 con 0x7f1cc40ffff0 2023-03-09T14:45:47.695 INFO:teuthology.orchestra.run.smithi038.stdout:epoch 89 2023-03-09T14:45:47.696 INFO:teuthology.orchestra.run.smithi038.stdout:fsid c6733fe2-be87-11ed-9af2-001a4aab830c 2023-03-09T14:45:47.696 INFO:teuthology.orchestra.run.smithi038.stdout:created 2023-03-09T14:37:14.947981+0000 2023-03-09T14:45:47.696 INFO:teuthology.orchestra.run.smithi038.stdout:modified 2023-03-09T14:45:15.848936+0000 2023-03-09T14:45:47.696 INFO:teuthology.orchestra.run.smithi038.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2023-03-09T14:45:47.696 INFO:teuthology.orchestra.run.smithi038.stdout:crush_version 27 2023-03-09T14:45:47.696 INFO:teuthology.orchestra.run.smithi038.stdout:full_ratio 0.95 2023-03-09T14:45:47.696 INFO:teuthology.orchestra.run.smithi038.stdout:backfillfull_ratio 0.9 2023-03-09T14:45:47.697 INFO:teuthology.orchestra.run.smithi038.stdout:nearfull_ratio 0.85 2023-03-09T14:45:47.697 INFO:teuthology.orchestra.run.smithi038.stdout:require_min_compat_client luminous 2023-03-09T14:45:47.697 INFO:teuthology.orchestra.run.smithi038.stdout:min_compat_client luminous 2023-03-09T14:45:47.697 INFO:teuthology.orchestra.run.smithi038.stdout:require_osd_release reef 2023-03-09T14:45:47.697 INFO:teuthology.orchestra.run.smithi038.stdout:stretch_mode_enabled false 2023-03-09T14:45:47.697 INFO:teuthology.orchestra.run.smithi038.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2023-03-09T14:45:47.697 INFO:teuthology.orchestra.run.smithi038.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 82 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 2.25 2023-03-09T14:45:47.698 INFO:teuthology.orchestra.run.smithi038.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 83 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2023-03-09T14:45:47.698 INFO:teuthology.orchestra.run.smithi038.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 87 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2023-03-09T14:45:47.698 INFO:teuthology.orchestra.run.smithi038.stdout:max_osd 12 2023-03-09T14:45:47.698 INFO:teuthology.orchestra.run.smithi038.stdout:osd.0 up in weight 1 up_from 8 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.38:6802/1508072290,v1:172.21.15.38:6803/1508072290] [v2:172.21.15.38:6804/1508072290,v1:172.21.15.38:6805/1508072290] exists,up 51477039-15ee-4707-a34d-4ecfa18bebde 2023-03-09T14:45:47.698 INFO:teuthology.orchestra.run.smithi038.stdout:osd.1 up in weight 1 up_from 13 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.122:6800/3929268773,v1:172.21.15.122:6801/3929268773] [v2:172.21.15.122:6802/3929268773,v1:172.21.15.122:6803/3929268773] exists,up 71f8a718-b0a9-4f39-bc54-267aa42a2578 2023-03-09T14:45:47.698 INFO:teuthology.orchestra.run.smithi038.stdout:osd.2 up in weight 1 up_from 18 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.181:6800/1310630803,v1:172.21.15.181:6801/1310630803] [v2:172.21.15.181:6802/1310630803,v1:172.21.15.181:6803/1310630803] exists,up 961a2cbe-140e-437f-b131-1fcac573b19c 2023-03-09T14:45:47.698 INFO:teuthology.orchestra.run.smithi038.stdout:osd.3 up in weight 1 up_from 25 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.38:6810/2719814100,v1:172.21.15.38:6811/2719814100] [v2:172.21.15.38:6812/2719814100,v1:172.21.15.38:6813/2719814100] exists,up 969df8fa-2d19-42c7-9f2a-5c5fac15d4a5 2023-03-09T14:45:47.698 INFO:teuthology.orchestra.run.smithi038.stdout:osd.4 up in weight 1 up_from 32 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.122:6808/2480764994,v1:172.21.15.122:6809/2480764994] [v2:172.21.15.122:6810/2480764994,v1:172.21.15.122:6811/2480764994] exists,up 44a6dce3-e06e-4cbd-9404-62a86d01ab6b 2023-03-09T14:45:47.699 INFO:teuthology.orchestra.run.smithi038.stdout:osd.5 up in weight 1 up_from 37 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.181:6808/3141346246,v1:172.21.15.181:6809/3141346246] [v2:172.21.15.181:6810/3141346246,v1:172.21.15.181:6811/3141346246] exists,up 49749574-c00c-4e3a-80e6-e3686df58d5b 2023-03-09T14:45:47.699 INFO:teuthology.orchestra.run.smithi038.stdout:osd.6 up in weight 1 up_from 45 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.38:6818/2122788957,v1:172.21.15.38:6819/2122788957] [v2:172.21.15.38:6820/2122788957,v1:172.21.15.38:6821/2122788957] exists,up 61c0437e-3f51-45a9-8a61-e98eea0093f8 2023-03-09T14:45:47.699 INFO:teuthology.orchestra.run.smithi038.stdout:osd.7 up in weight 1 up_from 50 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.122:6816/656874139,v1:172.21.15.122:6817/656874139] [v2:172.21.15.122:6818/656874139,v1:172.21.15.122:6819/656874139] exists,up 231fecdd-939f-4c7b-83a1-c4a1cffd7139 2023-03-09T14:45:47.699 INFO:teuthology.orchestra.run.smithi038.stdout:osd.8 up in weight 1 up_from 57 up_thru 85 down_at 0 last_clean_interval [0,0) [v2:172.21.15.181:6816/755000703,v1:172.21.15.181:6817/755000703] [v2:172.21.15.181:6818/755000703,v1:172.21.15.181:6819/755000703] exists,up 297e6d6d-85fd-43ce-9028-5e9dbe55c36e 2023-03-09T14:45:47.699 INFO:teuthology.orchestra.run.smithi038.stdout:osd.9 up in weight 1 up_from 65 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.38:6826/2624555969,v1:172.21.15.38:6827/2624555969] [v2:172.21.15.38:6828/2624555969,v1:172.21.15.38:6829/2624555969] exists,up 94cc0355-5a1a-4cfc-bd0c-562b6b41127e 2023-03-09T14:45:47.699 INFO:teuthology.orchestra.run.smithi038.stdout:osd.10 up in weight 1 up_from 70 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.122:6824/1427785280,v1:172.21.15.122:6825/1427785280] [v2:172.21.15.122:6826/1427785280,v1:172.21.15.122:6827/1427785280] exists,up f4f2e042-8d92-46bf-93b8-8b0f04795c2b 2023-03-09T14:45:47.699 INFO:teuthology.orchestra.run.smithi038.stdout:osd.11 up in weight 1 up_from 78 up_thru 88 down_at 0 last_clean_interval [0,0) [v2:172.21.15.181:6824/3183488640,v1:172.21.15.181:6825/3183488640] [v2:172.21.15.181:6826/3183488640,v1:172.21.15.181:6827/3183488640] exists,up 07444936-eae5-4097-a799-60294cafe80b 2023-03-09T14:45:47.700 INFO:teuthology.orchestra.run.smithi038.stdout:pg_upmap_items 2.2 [7,11] 2023-03-09T14:45:47.700 INFO:teuthology.orchestra.run.smithi038.stdout:pg_upmap_items 2.c [2,4] 2023-03-09T14:45:47.700 INFO:teuthology.orchestra.run.smithi038.stdout:pg_upmap_items 2.d [5,11] 2023-03-09T14:45:47.700 INFO:teuthology.orchestra.run.smithi038.stdout:pg_upmap_items 2.16 [2,4] 2023-03-09T14:45:47.700 INFO:teuthology.orchestra.run.smithi038.stdout:pg_upmap_items 2.17 [5,6] 2023-03-09T14:45:47.700 INFO:teuthology.orchestra.run.smithi038.stdout:pg_upmap_items 2.1a [8,1] 2023-03-09T14:45:47.700 INFO:teuthology.orchestra.run.smithi038.stdout:pg_upmap_items 2.1b [2,4] 2023-03-09T14:45:47.700 INFO:teuthology.orchestra.run.smithi038.stdout:pg_upmap_items 2.1f [8,1] 2023-03-09T14:45:47.701 INFO:teuthology.orchestra.run.smithi038.stdout:pg_upmap_items 4.c [9,10] 2023-03-09T14:45:47.701 INFO:teuthology.orchestra.run.smithi038.stdout:pg_upmap_items 4.12 [9,10] 2023-03-09T14:45:47.701 INFO:teuthology.orchestra.run.smithi038.stdout:blocklist 172.21.15.38:0/4272288735 expires 2023-03-10T14:38:14.612767+0000 2023-03-09T14:45:47.701 INFO:teuthology.orchestra.run.smithi038.stdout:blocklist 172.21.15.38:6801/3381625669 expires 2023-03-10T14:38:14.612767+0000 2023-03-09T14:45:47.701 INFO:teuthology.orchestra.run.smithi038.stdout:blocklist 172.21.15.38:6800/3381625669 expires 2023-03-10T14:38:14.612767+0000 2023-03-09T14:45:47.701 INFO:teuthology.orchestra.run.smithi038.stdout:blocklist 172.21.15.38:6800/3060474342 expires 2023-03-10T14:37:47.517775+0000 2023-03-09T14:45:47.701 INFO:teuthology.orchestra.run.smithi038.stdout:blocklist 172.21.15.38:0/4253918615 expires 2023-03-10T14:38:14.612767+0000 2023-03-09T14:45:47.701 INFO:teuthology.orchestra.run.smithi038.stdout:blocklist 172.21.15.38:0/3496083181 expires 2023-03-10T14:37:47.517775+0000 2023-03-09T14:45:47.702 INFO:teuthology.orchestra.run.smithi038.stdout:blocklist 172.21.15.38:0/245602368 expires 2023-03-10T14:37:47.517775+0000 2023-03-09T14:45:47.702 INFO:teuthology.orchestra.run.smithi038.stdout:blocklist 172.21.15.38:6801/3060474342 expires 2023-03-10T14:37:47.517775+0000 2023-03-09T14:45:47.702 INFO:teuthology.orchestra.run.smithi038.stdout:blocklist 172.21.15.38:0/3622589182 expires 2023-03-10T14:37:47.517775+0000 2023-03-09T14:45:47.702 INFO:teuthology.orchestra.run.smithi038.stdout:blocklist 172.21.15.38:0/1430302251 expires 2023-03-10T14:38:14.612767+0000 2023-03-09T14:45:47.703 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:47.695+0000 7f1ccb216700 1 -- 172.21.15.38:0/1586731238 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f1cb409f3c0 msgr2=0x7f1cb40a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:47.703 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:47.695+0000 7f1ccb216700 1 --2- 172.21.15.38:0/1586731238 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f1cb409f3c0 0x7f1cb40a1870 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f1cb800b5c0 tx=0x7f1cb8005960 comp rx=0 tx=0).stop 2023-03-09T14:45:47.703 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:47.695+0000 7f1ccb216700 1 -- 172.21.15.38:0/1586731238 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1cc40ffff0 msgr2=0x7f1cc41962b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:47.703 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:47.695+0000 7f1ccb216700 1 --2- 172.21.15.38:0/1586731238 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1cc40ffff0 0x7f1cc41962b0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f1cb000d950 tx=0x7f1cb000dc60 comp rx=0 tx=0).stop 2023-03-09T14:45:47.703 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:47.695+0000 7f1ccb216700 1 -- 172.21.15.38:0/1586731238 shutdown_connections 2023-03-09T14:45:47.703 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:47.695+0000 7f1ccb216700 1 --2- 172.21.15.38:0/1586731238 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f1cb409f3c0 0x7f1cb40a1870 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:47.704 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:47.695+0000 7f1ccb216700 1 --2- 172.21.15.38:0/1586731238 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1cc41009a0 0x7f1cc419d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:47.704 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:47.695+0000 7f1ccb216700 1 --2- 172.21.15.38:0/1586731238 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1cc40feea0 0x7f1cc4195d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:47.704 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:47.695+0000 7f1ccb216700 1 --2- 172.21.15.38:0/1586731238 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1cc40ffff0 0x7f1cc41962b0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:47.704 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:47.695+0000 7f1ccb216700 1 -- 172.21.15.38:0/1586731238 >> 172.21.15.38:0/1586731238 conn(0x7f1cc40fa480 msgr2=0x7f1cc40682e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:47.704 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:47.695+0000 7f1ccb216700 1 -- 172.21.15.38:0/1586731238 shutdown_connections 2023-03-09T14:45:47.704 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:47.695+0000 7f1ccb216700 1 -- 172.21.15.38:0/1586731238 wait complete. 2023-03-09T14:45:48.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:47 smithi038 ceph-mon[92422]: pgmap v305: 129 pgs: 129 active+clean; 636 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 341 B/s wr, 7 op/s 2023-03-09T14:45:48.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:47 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/1586731238' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2023-03-09T14:45:48.138 INFO:teuthology.run_tasks:Running task fs.ready... 2023-03-09T14:45:48.155 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-09T14:45:48.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:47 smithi122 ceph-mon[93869]: pgmap v305: 129 pgs: 129 active+clean; 636 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 341 B/s wr, 7 op/s 2023-03-09T14:45:48.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:47 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/1586731238' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2023-03-09T14:45:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:47 smithi181 ceph-mon[94189]: pgmap v305: 129 pgs: 129 active+clean; 636 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 341 B/s wr, 7 op/s 2023-03-09T14:45:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:47 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/1586731238' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2023-03-09T14:45:48.286 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.283+0000 7faa9c6f9700 1 -- 172.21.15.38:0/227538503 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa94105b40 msgr2=0x7faa94107fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:48.286 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.283+0000 7faa9c6f9700 1 --2- 172.21.15.38:0/227538503 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa94105b40 0x7faa94107fa0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7faa8c00b0a0 tx=0x7faa8c00b3b0 comp rx=0 tx=0).stop 2023-03-09T14:45:48.286 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.283+0000 7faa9c6f9700 1 -- 172.21.15.38:0/227538503 shutdown_connections 2023-03-09T14:45:48.286 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.283+0000 7faa9c6f9700 1 --2- 172.21.15.38:0/227538503 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa94105b40 0x7faa94107fa0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:48.287 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.283+0000 7faa9c6f9700 1 --2- 172.21.15.38:0/227538503 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faa94100900 0x7faa94102ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:48.287 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.283+0000 7faa9c6f9700 1 --2- 172.21.15.38:0/227538503 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faa94103220 0x7faa94105600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:48.287 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.283+0000 7faa9c6f9700 1 -- 172.21.15.38:0/227538503 >> 172.21.15.38:0/227538503 conn(0x7faa940fa380 msgr2=0x7faa940fc7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:48.287 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.283+0000 7faa9c6f9700 1 -- 172.21.15.38:0/227538503 shutdown_connections 2023-03-09T14:45:48.287 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.283+0000 7faa9c6f9700 1 -- 172.21.15.38:0/227538503 wait complete. 2023-03-09T14:45:48.289 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.284+0000 7faa9c6f9700 1 Processor -- start 2023-03-09T14:45:48.289 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.284+0000 7faa9c6f9700 1 -- start start 2023-03-09T14:45:48.289 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.285+0000 7faa9c6f9700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faa94100900 0x7faa94195ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:48.290 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.285+0000 7faa9c6f9700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faa94103220 0x7faa94196400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:48.290 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.285+0000 7faa9c6f9700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa94105b40 0x7faa9419d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:48.290 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.285+0000 7faa9c6f9700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa9419da40 con 0x7faa94105b40 2023-03-09T14:45:48.290 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.285+0000 7faa9c6f9700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa9419db80 con 0x7faa94103220 2023-03-09T14:45:48.290 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.285+0000 7faa9c6f9700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa9419dd10 con 0x7faa94100900 2023-03-09T14:45:48.290 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.285+0000 7faa9ac96700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa94105b40 0x7faa9419d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:48.291 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.285+0000 7faa9ac96700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa94105b40 0x7faa9419d470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38008/0 (socket says 172.21.15.38:38008) 2023-03-09T14:45:48.291 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.285+0000 7faa9ac96700 1 -- 172.21.15.38:0/628388141 learned_addr learned my addr 172.21.15.38:0/628388141 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:48.291 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.285+0000 7faa9a495700 1 --2- 172.21.15.38:0/628388141 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faa94100900 0x7faa94195ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:48.291 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.285+0000 7faa99c94700 1 --2- 172.21.15.38:0/628388141 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faa94103220 0x7faa94196400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:48.291 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.285+0000 7faa9ac96700 1 -- 172.21.15.38:0/628388141 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faa94100900 msgr2=0x7faa94195ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:48.291 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.285+0000 7faa9ac96700 1 --2- 172.21.15.38:0/628388141 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faa94100900 0x7faa94195ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:48.291 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.285+0000 7faa9ac96700 1 -- 172.21.15.38:0/628388141 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faa94103220 msgr2=0x7faa94196400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:48.292 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.285+0000 7faa9ac96700 1 --2- 172.21.15.38:0/628388141 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faa94103220 0x7faa94196400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:48.292 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.285+0000 7faa9ac96700 1 -- 172.21.15.38:0/628388141 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa8c009d00 con 0x7faa94105b40 2023-03-09T14:45:48.292 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.285+0000 7faa9ac96700 1 --2- 172.21.15.38:0/628388141 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa94105b40 0x7faa9419d470 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7faa8c0062a0 tx=0x7faa8c004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:48.292 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.285+0000 7faa877fe700 1 -- 172.21.15.38:0/628388141 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa8c00e040 con 0x7faa94105b40 2023-03-09T14:45:48.292 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.285+0000 7faa877fe700 1 -- 172.21.15.38:0/628388141 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7faa8c00cec0 con 0x7faa94105b40 2023-03-09T14:45:48.292 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.285+0000 7faa9c6f9700 1 -- 172.21.15.38:0/628388141 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa9410b7c0 con 0x7faa94105b40 2023-03-09T14:45:48.292 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.285+0000 7faa877fe700 1 -- 172.21.15.38:0/628388141 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa8c004020 con 0x7faa94105b40 2023-03-09T14:45:48.293 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.286+0000 7faa9c6f9700 1 -- 172.21.15.38:0/628388141 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa9410b630 con 0x7faa94105b40 2023-03-09T14:45:48.293 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.287+0000 7faa9c6f9700 1 -- 172.21.15.38:0/628388141 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa940fbef0 con 0x7faa94105b40 2023-03-09T14:45:48.293 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.287+0000 7faa877fe700 1 -- 172.21.15.38:0/628388141 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) v1 ==== 133198+0+0 (secure 0 0 0) 0x7faa8c01d690 con 0x7faa94105b40 2023-03-09T14:45:48.294 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.288+0000 7faa877fe700 1 --2- 172.21.15.38:0/628388141 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7faa8009f3c0 0x7faa800a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:48.294 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.288+0000 7faa9a495700 1 --2- 172.21.15.38:0/628388141 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7faa8009f3c0 0x7faa800a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:48.294 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.288+0000 7faa877fe700 1 -- 172.21.15.38:0/628388141 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7faa8c0cba60 con 0x7faa94105b40 2023-03-09T14:45:48.294 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.289+0000 7faa9a495700 1 --2- 172.21.15.38:0/628388141 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7faa8009f3c0 0x7faa800a1870 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7faa90005950 tx=0x7faa9000b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:48.295 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.293+0000 7faa877fe700 1 -- 172.21.15.38:0/628388141 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7faa8c095ba0 con 0x7faa94105b40 2023-03-09T14:45:48.494 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.492+0000 7faa9c6f9700 1 -- 172.21.15.38:0/628388141 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faa9402cef0 con 0x7faa94105b40 2023-03-09T14:45:48.496 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.494+0000 7faa877fe700 1 -- 172.21.15.38:0/628388141 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24445 (secure 0 0 0) 0x7faa8c0991c0 con 0x7faa94105b40 2023-03-09T14:45:48.497 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:45:48.497 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":89,"fsid":"c6733fe2-be87-11ed-9af2-001a4aab830c","created":"2023-03-09T14:37:14.947981+0000","modified":"2023-03-09T14:45:15.848936+0000","last_up_change":"2023-03-09T14:43:37.198407+0000","last_in_change":"2023-03-09T14:43:19.550566+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-09T14:40:23.196665+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-03-09T14:44:17.484374+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-03-09T14:44:18.457833+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-03-09T14:44:21.895589+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"51477039-15ee-4707-a34d-4ecfa18bebde","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6803","nonce":1508072290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6805","nonce":1508072290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6809","nonce":1508072290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6807","nonce":1508072290}]},"public_addr":"172.21.15.38:6803/1508072290","cluster_addr":"172.21.15.38:6805/1508072290","heartbeat_back_addr":"172.21.15.38:6809/1508072290","heartbeat_front_addr":"172.21.15.38:6807/1508072290","state":["exists","up"]},{"osd":1,"uuid":"71f8a718-b0a9-4f39-bc54-267aa42a2578","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6801","nonce":3929268773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6803","nonce":3929268773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6807","nonce":3929268773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6805","nonce":3929268773}]},"public_addr":"172.21.15.122:6801/3929268773","cluster_addr":"172.21.15.122:6803/3929268773","heartbeat_back_addr":"172.21.15.122:6807/3929268773","heartbeat_front_addr":"172.21.15.122:6805/3929268773","state":["exists","up"]},{"osd":2,"uuid":"961a2cbe-140e-437f-b131-1fcac573b19c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6801","nonce":1310630803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6803","nonce":1310630803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6807","nonce":1310630803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6805","nonce":1310630803}]},"public_addr":"172.21.15.181:6801/1310630803","cluster_addr":"172.21.15.181:6803/1310630803","heartbeat_back_addr":"172.21.15.181:6807/1310630803","heartbeat_front_addr":"172.21.15.181:6805/1310630803","state":["exists","up"]},{"osd":3,"uuid":"969df8fa-2d19-42c7-9f2a-5c5fac15d4a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6811","nonce":2719814100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6813","nonce":2719814100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6817","nonce":2719814100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6815","nonce":2719814100}]},"public_addr":"172.21.15.38:6811/2719814100","cluster_addr":"172.21.15.38:6813/2719814100","heartbeat_back_addr":"172.21.15.38:6817/2719814100","heartbeat_front_addr":"172.21.15.38:6815/2719814100","state":["exists","up"]},{"osd":4,"uuid":"44a6dce3-e06e-4cbd-9404-62a86d01ab6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6809","nonce":2480764994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6811","nonce":2480764994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6815","nonce":2480764994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6813","nonce":2480764994}]},"public_addr":"172.21.15.122:6809/2480764994","cluster_addr":"172.21.15.122:6811/2480764994","heartbeat_back_addr":"172.21.15.122:6815/2480764994","heartbeat_front_addr":"172.21.15.122:6813/2480764994","state":["exists","up"]},{"osd":5,"uuid":"49749574-c00c-4e3a-80e6-e3686df58d5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6809","nonce":3141346246}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6811","nonce":3141346246}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6815","nonce":3141346246}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6813","nonce":3141346246}]},"public_addr":"172.21.15.181:6809/3141346246","cluster_addr":"172.21.15.181:6811/3141346246","heartbeat_back_addr":"172.21.15.181:6815/3141346246","heartbeat_front_addr":"172.21.15.181:6813/3141346246","state":["exists","up"]},{"osd":6,"uuid":"61c0437e-3f51-45a9-8a61-e98eea0093f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6819","nonce":2122788957}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6821","nonce":2122788957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6825","nonce":2122788957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6823","nonce":2122788957}]},"public_addr":"172.21.15.38:6819/2122788957","cluster_addr":"172.21.15.38:6821/2122788957","heartbeat_back_addr":"172.21.15.38:6825/2122788957","heartbeat_front_addr":"172.21.15.38:6823/2122788957","state":["exists","up"]},{"osd":7,"uuid":"231fecdd-939f-4c7b-83a1-c4a1cffd7139","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6817","nonce":656874139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6819","nonce":656874139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6823","nonce":656874139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6821","nonce":656874139}]},"public_addr":"172.21.15.122:6817/656874139","cluster_addr":"172.21.15.122:6819/656874139","heartbeat_back_addr":"172.21.15.122:6823/656874139","heartbeat_front_addr":"172.21.15.122:6821/656874139","state":["exists","up"]},{"osd":8,"uuid":"297e6d6d-85fd-43ce-9028-5e9dbe55c36e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6817","nonce":755000703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6819","nonce":755000703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6823","nonce":755000703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6821","nonce":755000703}]},"public_addr":"172.21.15.181:6817/755000703","cluster_addr":"172.21.15.181:6819/755000703","heartbeat_back_addr":"172.21.15.181:6823/755000703","heartbeat_front_addr":"172.21.15.181:6821/755000703","state":["exists","up"]},{"osd":9,"uuid":"94cc0355-5a1a-4cfc-bd0c-562b6b41127e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6827","nonce":2624555969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6829","nonce":2624555969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6833","nonce":2624555969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6831","nonce":2624555969}]},"public_addr":"172.21.15.38:6827/2624555969","cluster_addr":"172.21.15.38:6829/2624555969","heartbeat_back_addr":"172.21.15.38:6833/2624555969","heartbeat_front_addr":"172.21.15.38:6831/2624555969","state":["exists","up"]},{"osd":10,"uuid":"f4f2e042-8d92-46bf-93b8-8b0f04795c2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6825","nonce":1427785280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6827","nonce":1427785280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6831","nonce":1427785280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6829","nonce":1427785280}]},"public_addr":"172.21.15.122:6825/1427785280","cluster_addr":"172.21.15.122:6827/1427785280","heartbeat_back_addr":"172.21.15.122:6831/1427785280","heartbeat_front_addr":"172.21.15.122:6829/1427785280","state":["exists","up"]},{"osd":11,"uuid":"07444936-eae5-4097-a799-60294cafe80b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6825","nonce":3183488640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6827","nonce":3183488640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6831","nonce":3183488640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6829","nonce":3183488640}]},"public_addr":"172.21.15.181:6825/3183488640","cluster_addr":"172.21.15.181:6827/3183488640","heartbeat_back_addr":"172.21.15.181:6831/3183488640","heartbeat_front_addr":"172.21.15.181:6829/3183488640","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:39:37.657556+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:39:58.960074+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:40:20.326952+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:40:42.244387+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:41:03.946973+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:41:25.097985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:41:46.877916+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:42:08.413827+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:42:29.822009+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:42:51.776684+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:43:13.937249+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:43:35.569199+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.2","mappings":[{"from":7,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.d","mappings":[{"from":5,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.17","mappings":[{"from":5,"to":6}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/4272288735":"2023-03-10T14:38:14.612767+0000","172.21.15.38:6801/3381625669":"2023-03-10T14:38:14.612767+0000","172.21.15.38:6800/3381625669":"2023-03-10T14:38:14.612767+0000","172.21.15.38:6800/3060474342":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/4253918615":"2023-03-10T14:38:14.612767+0000","172.21.15.38:0/3496083181":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/245602368":"2023-03-10T14:37:47.517775+0000","172.21.15.38:6801/3060474342":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/3622589182":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/1430302251":"2023-03-10T14:38:14.612767+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-09T14:45:48.498 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.496+0000 7faa9c6f9700 1 -- 172.21.15.38:0/628388141 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7faa8009f3c0 msgr2=0x7faa800a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:48.498 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.496+0000 7faa9c6f9700 1 --2- 172.21.15.38:0/628388141 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7faa8009f3c0 0x7faa800a1870 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7faa90005950 tx=0x7faa9000b410 comp rx=0 tx=0).stop 2023-03-09T14:45:48.499 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.496+0000 7faa9c6f9700 1 -- 172.21.15.38:0/628388141 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa94105b40 msgr2=0x7faa9419d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:48.499 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.496+0000 7faa9c6f9700 1 --2- 172.21.15.38:0/628388141 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa94105b40 0x7faa9419d470 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7faa8c0062a0 tx=0x7faa8c004d00 comp rx=0 tx=0).stop 2023-03-09T14:45:48.499 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.496+0000 7faa9c6f9700 1 -- 172.21.15.38:0/628388141 shutdown_connections 2023-03-09T14:45:48.499 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.496+0000 7faa9c6f9700 1 --2- 172.21.15.38:0/628388141 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7faa94105b40 0x7faa9419d470 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:48.499 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.496+0000 7faa9c6f9700 1 --2- 172.21.15.38:0/628388141 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7faa8009f3c0 0x7faa800a1870 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:48.499 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.496+0000 7faa9c6f9700 1 --2- 172.21.15.38:0/628388141 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7faa94100900 0x7faa94195ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:48.499 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.496+0000 7faa9c6f9700 1 --2- 172.21.15.38:0/628388141 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7faa94103220 0x7faa94196400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:48.500 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.496+0000 7faa9c6f9700 1 -- 172.21.15.38:0/628388141 >> 172.21.15.38:0/628388141 conn(0x7faa940fa380 msgr2=0x7faa94101560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:48.500 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.496+0000 7faa9c6f9700 1 -- 172.21.15.38:0/628388141 shutdown_connections 2023-03-09T14:45:48.500 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.496+0000 7faa9c6f9700 1 -- 172.21.15.38:0/628388141 wait complete. 2023-03-09T14:45:48.512 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-09T14:40:23.196665+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-03-09T14:44:17.484374+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-03-09T14:44:18.457833+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-03-09T14:44:21.895589+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-03-09T14:45:48.513 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-09T14:45:48.643 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.640+0000 7fcc71ad7700 1 -- 172.21.15.38:0/3396210312 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcc6c0ffe90 msgr2=0x7fcc6c1002a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:48.643 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.640+0000 7fcc71ad7700 1 --2- 172.21.15.38:0/3396210312 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcc6c0ffe90 0x7fcc6c1002a0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c009990 tx=0x7fcc5c009ca0 comp rx=0 tx=0).stop 2023-03-09T14:45:48.643 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.640+0000 7fcc71ad7700 1 -- 172.21.15.38:0/3396210312 shutdown_connections 2023-03-09T14:45:48.643 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.640+0000 7fcc71ad7700 1 --2- 172.21.15.38:0/3396210312 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcc6c0ffe90 0x7fcc6c1002a0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:48.643 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.640+0000 7fcc71ad7700 1 --2- 172.21.15.38:0/3396210312 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcc6c101990 0x7fcc6c1082c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:48.643 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.640+0000 7fcc71ad7700 1 --2- 172.21.15.38:0/3396210312 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc6c100fe0 0x7fcc6c101450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:48.644 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.640+0000 7fcc71ad7700 1 -- 172.21.15.38:0/3396210312 >> 172.21.15.38:0/3396210312 conn(0x7fcc6c0fb4d0 msgr2=0x7fcc6c0fd8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:48.644 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.640+0000 7fcc71ad7700 1 -- 172.21.15.38:0/3396210312 shutdown_connections 2023-03-09T14:45:48.644 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.640+0000 7fcc71ad7700 1 -- 172.21.15.38:0/3396210312 wait complete. 2023-03-09T14:45:48.644 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.641+0000 7fcc71ad7700 1 Processor -- start 2023-03-09T14:45:48.646 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.641+0000 7fcc71ad7700 1 -- start start 2023-03-09T14:45:48.646 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.641+0000 7fcc71ad7700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcc6c0ffe90 0x7fcc6c195c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:48.646 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.641+0000 7fcc71ad7700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc6c100fe0 0x7fcc6c196180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:48.646 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.641+0000 7fcc71ad7700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcc6c101990 0x7fcc6c19d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:48.647 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.641+0000 7fcc71ad7700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc6c19d780 con 0x7fcc6c0ffe90 2023-03-09T14:45:48.647 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.641+0000 7fcc71ad7700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc6c19d8f0 con 0x7fcc6c101990 2023-03-09T14:45:48.647 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.641+0000 7fcc71ad7700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc6c19dab0 con 0x7fcc6c100fe0 2023-03-09T14:45:48.647 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.641+0000 7fcc6b7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcc6c0ffe90 0x7fcc6c195c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:48.647 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.641+0000 7fcc6b7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcc6c0ffe90 0x7fcc6c195c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38018/0 (socket says 172.21.15.38:38018) 2023-03-09T14:45:48.647 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.641+0000 7fcc6b7fe700 1 -- 172.21.15.38:0/1089251790 learned_addr learned my addr 172.21.15.38:0/1089251790 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:48.647 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.641+0000 7fcc6bfff700 1 --2- 172.21.15.38:0/1089251790 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcc6c101990 0x7fcc6c19d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:48.648 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.641+0000 7fcc6affd700 1 --2- 172.21.15.38:0/1089251790 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc6c100fe0 0x7fcc6c196180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:48.648 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.641+0000 7fcc6b7fe700 1 -- 172.21.15.38:0/1089251790 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc6c100fe0 msgr2=0x7fcc6c196180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:48.648 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.641+0000 7fcc6b7fe700 1 --2- 172.21.15.38:0/1089251790 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc6c100fe0 0x7fcc6c196180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:48.648 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.641+0000 7fcc6b7fe700 1 -- 172.21.15.38:0/1089251790 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcc6c101990 msgr2=0x7fcc6c19d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:48.648 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.641+0000 7fcc6b7fe700 1 --2- 172.21.15.38:0/1089251790 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcc6c101990 0x7fcc6c19d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:48.648 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.641+0000 7fcc6b7fe700 1 -- 172.21.15.38:0/1089251790 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc5c0095f0 con 0x7fcc6c0ffe90 2023-03-09T14:45:48.648 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.642+0000 7fcc6b7fe700 1 --2- 172.21.15.38:0/1089251790 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcc6c0ffe90 0x7fcc6c195c40 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c000c00 tx=0x7fcc5c00f710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:48.648 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.642+0000 7fcc68ff9700 1 -- 172.21.15.38:0/1089251790 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc5c01d070 con 0x7fcc6c0ffe90 2023-03-09T14:45:48.649 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.642+0000 7fcc68ff9700 1 -- 172.21.15.38:0/1089251790 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcc5c00fc00 con 0x7fcc6c0ffe90 2023-03-09T14:45:48.649 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.642+0000 7fcc71ad7700 1 -- 172.21.15.38:0/1089251790 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc6c10b7f0 con 0x7fcc6c0ffe90 2023-03-09T14:45:48.649 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.642+0000 7fcc68ff9700 1 -- 172.21.15.38:0/1089251790 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc5c017650 con 0x7fcc6c0ffe90 2023-03-09T14:45:48.649 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.642+0000 7fcc71ad7700 1 -- 172.21.15.38:0/1089251790 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc6c10b660 con 0x7fcc6c0ffe90 2023-03-09T14:45:48.649 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.643+0000 7fcc71ad7700 1 -- 172.21.15.38:0/1089251790 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc6c106e00 con 0x7fcc6c0ffe90 2023-03-09T14:45:48.649 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.644+0000 7fcc68ff9700 1 -- 172.21.15.38:0/1089251790 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) v1 ==== 133198+0+0 (secure 0 0 0) 0x7fcc5c029030 con 0x7fcc6c0ffe90 2023-03-09T14:45:48.650 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.645+0000 7fcc68ff9700 1 --2- 172.21.15.38:0/1089251790 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fcc5809f3c0 0x7fcc580a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:48.650 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.645+0000 7fcc6affd700 1 --2- 172.21.15.38:0/1089251790 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fcc5809f3c0 0x7fcc580a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:48.650 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.645+0000 7fcc68ff9700 1 -- 172.21.15.38:0/1089251790 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fcc5c0cb540 con 0x7fcc6c0ffe90 2023-03-09T14:45:48.650 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.645+0000 7fcc6affd700 1 --2- 172.21.15.38:0/1089251790 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fcc5809f3c0 0x7fcc580a1870 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fcc54005950 tx=0x7fcc540058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:48.651 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.649+0000 7fcc68ff9700 1 -- 172.21.15.38:0/1089251790 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7fcc5c095680 con 0x7fcc6c0ffe90 2023-03-09T14:45:48.852 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.849+0000 7fcc71ad7700 1 -- 172.21.15.38:0/1089251790 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcc6c061390 con 0x7fcc6c0ffe90 2023-03-09T14:45:48.852 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.849+0000 7fcc68ff9700 1 -- 172.21.15.38:0/1089251790 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcc5c098ca0 con 0x7fcc6c0ffe90 2023-03-09T14:45:48.852 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 1 2023-03-09T14:45:48.854 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.851+0000 7fcc71ad7700 1 -- 172.21.15.38:0/1089251790 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fcc5809f3c0 msgr2=0x7fcc580a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:48.854 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.852+0000 7fcc71ad7700 1 --2- 172.21.15.38:0/1089251790 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fcc5809f3c0 0x7fcc580a1870 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fcc54005950 tx=0x7fcc540058e0 comp rx=0 tx=0).stop 2023-03-09T14:45:48.854 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.852+0000 7fcc71ad7700 1 -- 172.21.15.38:0/1089251790 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcc6c0ffe90 msgr2=0x7fcc6c195c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:48.854 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.852+0000 7fcc71ad7700 1 --2- 172.21.15.38:0/1089251790 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcc6c0ffe90 0x7fcc6c195c40 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c000c00 tx=0x7fcc5c00f710 comp rx=0 tx=0).stop 2023-03-09T14:45:48.855 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.852+0000 7fcc71ad7700 1 -- 172.21.15.38:0/1089251790 shutdown_connections 2023-03-09T14:45:48.855 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.852+0000 7fcc71ad7700 1 --2- 172.21.15.38:0/1089251790 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcc6c0ffe90 0x7fcc6c195c40 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:48.855 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.852+0000 7fcc71ad7700 1 --2- 172.21.15.38:0/1089251790 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fcc5809f3c0 0x7fcc580a1870 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:48.855 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.852+0000 7fcc71ad7700 1 --2- 172.21.15.38:0/1089251790 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcc6c101990 0x7fcc6c19d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:48.855 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.852+0000 7fcc71ad7700 1 --2- 172.21.15.38:0/1089251790 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcc6c100fe0 0x7fcc6c196180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:48.855 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.852+0000 7fcc71ad7700 1 -- 172.21.15.38:0/1089251790 >> 172.21.15.38:0/1089251790 conn(0x7fcc6c0fb4d0 msgr2=0x7fcc6c0fd780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:48.855 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.852+0000 7fcc71ad7700 1 -- 172.21.15.38:0/1089251790 shutdown_connections 2023-03-09T14:45:48.856 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.852+0000 7fcc71ad7700 1 -- 172.21.15.38:0/1089251790 wait complete. 2023-03-09T14:45:48.867 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-03-09T14:45:48.997 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.994+0000 7efefcfab700 1 -- 172.21.15.38:0/77259086 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efef80fed90 msgr2=0x7efef80ff1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:48.998 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.994+0000 7efefcfab700 1 --2- 172.21.15.38:0/77259086 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efef80fed90 0x7efef80ff1a0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7efee8009990 tx=0x7efee8009ca0 comp rx=0 tx=0).stop 2023-03-09T14:45:48.998 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.994+0000 7efefcfab700 1 -- 172.21.15.38:0/77259086 shutdown_connections 2023-03-09T14:45:48.998 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.994+0000 7efefcfab700 1 --2- 172.21.15.38:0/77259086 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efef80fed90 0x7efef80ff1a0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:48.998 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.994+0000 7efefcfab700 1 --2- 172.21.15.38:0/77259086 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efef8100910 0x7efef81071d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:48.998 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.994+0000 7efefcfab700 1 --2- 172.21.15.38:0/77259086 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efef80fff80 0x7efef81003d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:48.998 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.994+0000 7efefcfab700 1 -- 172.21.15.38:0/77259086 >> 172.21.15.38:0/77259086 conn(0x7efef80fa3b0 msgr2=0x7efef80fc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:48.998 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.995+0000 7efefcfab700 1 -- 172.21.15.38:0/77259086 shutdown_connections 2023-03-09T14:45:48.999 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.995+0000 7efefcfab700 1 -- 172.21.15.38:0/77259086 wait complete. 2023-03-09T14:45:48.999 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.995+0000 7efefcfab700 1 Processor -- start 2023-03-09T14:45:48.999 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.995+0000 7efefcfab700 1 -- start start 2023-03-09T14:45:49.001 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.995+0000 7efefcfab700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efef80fed90 0x7efef8194cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:49.001 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.996+0000 7efefcfab700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efef80fff80 0x7efef8195200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:49.001 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.996+0000 7efefcfab700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efef8100910 0x7efef819c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:49.001 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.996+0000 7efefcfab700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef819c830 con 0x7efef80fed90 2023-03-09T14:45:49.001 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.996+0000 7efefcfab700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef819c9a0 con 0x7efef8100910 2023-03-09T14:45:49.001 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.996+0000 7efefcfab700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef819cb60 con 0x7efef80fff80 2023-03-09T14:45:49.002 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.996+0000 7efef659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efef80fed90 0x7efef8194cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:49.002 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.996+0000 7efef659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efef80fed90 0x7efef8194cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38032/0 (socket says 172.21.15.38:38032) 2023-03-09T14:45:49.002 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.996+0000 7efef659c700 1 -- 172.21.15.38:0/2468259754 learned_addr learned my addr 172.21.15.38:0/2468259754 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:49.002 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.996+0000 7efef6d9d700 1 --2- 172.21.15.38:0/2468259754 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efef8100910 0x7efef819c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:49.002 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.996+0000 7efef659c700 1 -- 172.21.15.38:0/2468259754 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efef80fff80 msgr2=0x7efef8195200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:49.002 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.996+0000 7efef5d9b700 1 --2- 172.21.15.38:0/2468259754 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efef80fff80 0x7efef8195200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:49.002 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.996+0000 7efef659c700 1 --2- 172.21.15.38:0/2468259754 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efef80fff80 0x7efef8195200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:49.003 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.996+0000 7efef659c700 1 -- 172.21.15.38:0/2468259754 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efef8100910 msgr2=0x7efef819c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:49.003 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.996+0000 7efef659c700 1 --2- 172.21.15.38:0/2468259754 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efef8100910 0x7efef819c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:49.003 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.996+0000 7efef659c700 1 -- 172.21.15.38:0/2468259754 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efee80095f0 con 0x7efef80fed90 2023-03-09T14:45:49.003 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.996+0000 7efef659c700 1 --2- 172.21.15.38:0/2468259754 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efef80fed90 0x7efef8194cc0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7efee8000c00 tx=0x7efee8005f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:49.003 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.996+0000 7efee77fe700 1 -- 172.21.15.38:0/2468259754 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee801d070 con 0x7efef80fed90 2023-03-09T14:45:49.003 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.996+0000 7efee77fe700 1 -- 172.21.15.38:0/2468259754 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7efee8004060 con 0x7efef80fed90 2023-03-09T14:45:49.003 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.996+0000 7efefcfab700 1 -- 172.21.15.38:0/2468259754 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efef810a7c0 con 0x7efef80fed90 2023-03-09T14:45:49.003 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.996+0000 7efee77fe700 1 -- 172.21.15.38:0/2468259754 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efee8022d40 con 0x7efef80fed90 2023-03-09T14:45:49.004 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.996+0000 7efefcfab700 1 -- 172.21.15.38:0/2468259754 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efef810a630 con 0x7efef80fed90 2023-03-09T14:45:49.004 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.997+0000 7efefcfab700 1 -- 172.21.15.38:0/2468259754 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efef8106070 con 0x7efef80fed90 2023-03-09T14:45:49.004 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.999+0000 7efee77fe700 1 -- 172.21.15.38:0/2468259754 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) v1 ==== 133198+0+0 (secure 0 0 0) 0x7efee8029030 con 0x7efef80fed90 2023-03-09T14:45:49.004 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.999+0000 7efee77fe700 1 --2- 172.21.15.38:0/2468259754 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7efee009f3c0 0x7efee00a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:49.005 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.999+0000 7efef5d9b700 1 --2- 172.21.15.38:0/2468259754 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7efee009f3c0 0x7efee00a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:49.005 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:48.999+0000 7efee77fe700 1 -- 172.21.15.38:0/2468259754 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7efee80cb5d0 con 0x7efef80fed90 2023-03-09T14:45:49.005 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.000+0000 7efef5d9b700 1 --2- 172.21.15.38:0/2468259754 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7efee009f3c0 0x7efee00a1870 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7efef81961c0 tx=0x7efeec005d50 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:49.005 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.003+0000 7efee77fe700 1 -- 172.21.15.38:0/2468259754 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7efee8095710 con 0x7efef80fed90 2023-03-09T14:45:49.206 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.203+0000 7efefcfab700 1 -- 172.21.15.38:0/2468259754 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7efef8061540 con 0x7efef80fed90 2023-03-09T14:45:49.206 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.204+0000 7efee77fe700 1 -- 172.21.15.38:0/2468259754 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) v1 ==== 104+0+11 (secure 0 0 0) 0x7efee8098d30 con 0x7efef80fed90 2023-03-09T14:45:49.206 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 32 2023-03-09T14:45:49.208 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.206+0000 7efefcfab700 1 -- 172.21.15.38:0/2468259754 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7efee009f3c0 msgr2=0x7efee00a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:49.208 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.206+0000 7efefcfab700 1 --2- 172.21.15.38:0/2468259754 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7efee009f3c0 0x7efee00a1870 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7efef81961c0 tx=0x7efeec005d50 comp rx=0 tx=0).stop 2023-03-09T14:45:49.208 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.206+0000 7efefcfab700 1 -- 172.21.15.38:0/2468259754 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efef80fed90 msgr2=0x7efef8194cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:49.208 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.206+0000 7efefcfab700 1 --2- 172.21.15.38:0/2468259754 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efef80fed90 0x7efef8194cc0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7efee8000c00 tx=0x7efee8005f10 comp rx=0 tx=0).stop 2023-03-09T14:45:49.208 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.206+0000 7efefcfab700 1 -- 172.21.15.38:0/2468259754 shutdown_connections 2023-03-09T14:45:49.209 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.206+0000 7efefcfab700 1 --2- 172.21.15.38:0/2468259754 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7efef80fed90 0x7efef8194cc0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:49.209 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.206+0000 7efefcfab700 1 --2- 172.21.15.38:0/2468259754 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7efee009f3c0 0x7efee00a1870 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:49.209 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.206+0000 7efefcfab700 1 --2- 172.21.15.38:0/2468259754 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7efef8100910 0x7efef819c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:49.209 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.206+0000 7efefcfab700 1 --2- 172.21.15.38:0/2468259754 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7efef80fff80 0x7efef8195200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:49.209 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.206+0000 7efefcfab700 1 -- 172.21.15.38:0/2468259754 >> 172.21.15.38:0/2468259754 conn(0x7efef80fa3b0 msgr2=0x7efef81031b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:49.209 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.206+0000 7efefcfab700 1 -- 172.21.15.38:0/2468259754 shutdown_connections 2023-03-09T14:45:49.209 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.206+0000 7efefcfab700 1 -- 172.21.15.38:0/2468259754 wait complete. 2023-03-09T14:45:49.219 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:48 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/628388141' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-03-09T14:45:49.219 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:48 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/1089251790' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-03-09T14:45:49.220 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-03-09T14:45:49.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:48 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/628388141' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-03-09T14:45:49.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:48 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/1089251790' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-03-09T14:45:49.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:48 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/628388141' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-03-09T14:45:49.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:48 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/1089251790' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-03-09T14:45:49.350 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.347+0000 7f8974d0f700 1 -- 172.21.15.38:0/2966507347 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89700fff80 msgr2=0x7f89701003d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:49.350 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.347+0000 7f8974d0f700 1 --2- 172.21.15.38:0/2966507347 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89700fff80 0x7f89701003d0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f8964009990 tx=0x7f8964009ca0 comp rx=0 tx=0).stop 2023-03-09T14:45:49.350 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.347+0000 7f8974d0f700 1 -- 172.21.15.38:0/2966507347 shutdown_connections 2023-03-09T14:45:49.350 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.347+0000 7f8974d0f700 1 --2- 172.21.15.38:0/2966507347 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8970100910 0x7f89701071d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:49.350 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.347+0000 7f8974d0f700 1 --2- 172.21.15.38:0/2966507347 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f89700fed90 0x7f89700ff1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:49.351 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.347+0000 7f8974d0f700 1 --2- 172.21.15.38:0/2966507347 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89700fff80 0x7f89701003d0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:49.351 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.347+0000 7f8974d0f700 1 -- 172.21.15.38:0/2966507347 >> 172.21.15.38:0/2966507347 conn(0x7f89700fa3b0 msgr2=0x7f89700fc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:49.351 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.347+0000 7f8974d0f700 1 -- 172.21.15.38:0/2966507347 shutdown_connections 2023-03-09T14:45:49.351 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.347+0000 7f8974d0f700 1 -- 172.21.15.38:0/2966507347 wait complete. 2023-03-09T14:45:49.353 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.348+0000 7f8974d0f700 1 Processor -- start 2023-03-09T14:45:49.353 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.348+0000 7f8974d0f700 1 -- start start 2023-03-09T14:45:49.353 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.348+0000 7f8974d0f700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f89700fed90 0x7f8970069780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:49.353 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.348+0000 7f8974d0f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89700fff80 0x7f897019a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:49.353 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.348+0000 7f8974d0f700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8970100910 0x7f897019cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:49.353 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.348+0000 7f8974d0f700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f897019d750 con 0x7f89700fff80 2023-03-09T14:45:49.354 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.348+0000 7f8974d0f700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f897019d8c0 con 0x7f8970100910 2023-03-09T14:45:49.354 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.348+0000 7f8974d0f700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f897019da80 con 0x7f89700fed90 2023-03-09T14:45:49.354 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.348+0000 7f896dd9b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89700fff80 0x7f897019a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:49.354 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.348+0000 7f896dd9b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89700fff80 0x7f897019a930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38048/0 (socket says 172.21.15.38:38048) 2023-03-09T14:45:49.354 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.348+0000 7f896dd9b700 1 -- 172.21.15.38:0/885213581 learned_addr learned my addr 172.21.15.38:0/885213581 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:49.354 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.348+0000 7f896e59c700 1 --2- 172.21.15.38:0/885213581 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f89700fed90 0x7f8970069780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:49.354 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.349+0000 7f896ed9d700 1 --2- 172.21.15.38:0/885213581 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8970100910 0x7f897019cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:49.354 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.349+0000 7f896dd9b700 1 -- 172.21.15.38:0/885213581 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f89700fed90 msgr2=0x7f8970069780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:49.355 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.349+0000 7f896dd9b700 1 --2- 172.21.15.38:0/885213581 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f89700fed90 0x7f8970069780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:49.355 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.349+0000 7f896dd9b700 1 -- 172.21.15.38:0/885213581 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8970100910 msgr2=0x7f897019cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:49.355 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.349+0000 7f896dd9b700 1 --2- 172.21.15.38:0/885213581 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8970100910 0x7f897019cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:49.355 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.349+0000 7f896dd9b700 1 -- 172.21.15.38:0/885213581 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89640095f0 con 0x7f89700fff80 2023-03-09T14:45:49.355 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.349+0000 7f896ed9d700 1 --2- 172.21.15.38:0/885213581 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8970100910 0x7f897019cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:45:49.355 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.349+0000 7f896dd9b700 1 --2- 172.21.15.38:0/885213581 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89700fff80 0x7f897019a930 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f8964004760 tx=0x7f8964004790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:49.355 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.349+0000 7f895f7fe700 1 -- 172.21.15.38:0/885213581 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f896401d070 con 0x7f89700fff80 2023-03-09T14:45:49.355 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.349+0000 7f8974d0f700 1 -- 172.21.15.38:0/885213581 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f897010a6f0 con 0x7f89700fff80 2023-03-09T14:45:49.356 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.349+0000 7f895f7fe700 1 -- 172.21.15.38:0/885213581 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8964004cf0 con 0x7f89700fff80 2023-03-09T14:45:49.356 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.349+0000 7f895f7fe700 1 -- 172.21.15.38:0/885213581 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8964005280 con 0x7f89700fff80 2023-03-09T14:45:49.356 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.349+0000 7f8974d0f700 1 -- 172.21.15.38:0/885213581 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f897010a560 con 0x7f89700fff80 2023-03-09T14:45:49.356 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.350+0000 7f8974d0f700 1 -- 172.21.15.38:0/885213581 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8970105cc0 con 0x7f89700fff80 2023-03-09T14:45:49.357 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.351+0000 7f895f7fe700 1 -- 172.21.15.38:0/885213581 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) v1 ==== 133198+0+0 (secure 0 0 0) 0x7f896400c040 con 0x7f89700fff80 2023-03-09T14:45:49.357 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.352+0000 7f895f7fe700 1 --2- 172.21.15.38:0/885213581 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f895809f3c0 0x7f89580a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:49.357 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.352+0000 7f895f7fe700 1 -- 172.21.15.38:0/885213581 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f89640cb560 con 0x7f89700fff80 2023-03-09T14:45:49.357 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.352+0000 7f896e59c700 1 --2- 172.21.15.38:0/885213581 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f895809f3c0 0x7f89580a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:49.357 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.352+0000 7f896e59c700 1 --2- 172.21.15.38:0/885213581 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f895809f3c0 0x7f89580a1870 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f89700ffde0 tx=0x7f896000b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:49.358 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.355+0000 7f895f7fe700 1 -- 172.21.15.38:0/885213581 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f89640956a0 con 0x7f89700fff80 2023-03-09T14:45:49.560 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.557+0000 7f8974d0f700 1 -- 172.21.15.38:0/885213581 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8970061390 con 0x7f89700fff80 2023-03-09T14:45:49.560 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.558+0000 7f895f7fe700 1 -- 172.21.15.38:0/885213581 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8964030020 con 0x7f89700fff80 2023-03-09T14:45:49.560 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 64 2023-03-09T14:45:49.562 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.560+0000 7f8974d0f700 1 -- 172.21.15.38:0/885213581 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f895809f3c0 msgr2=0x7f89580a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:49.563 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.560+0000 7f8974d0f700 1 --2- 172.21.15.38:0/885213581 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f895809f3c0 0x7f89580a1870 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f89700ffde0 tx=0x7f896000b410 comp rx=0 tx=0).stop 2023-03-09T14:45:49.563 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.560+0000 7f8974d0f700 1 -- 172.21.15.38:0/885213581 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89700fff80 msgr2=0x7f897019a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:49.563 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.560+0000 7f8974d0f700 1 --2- 172.21.15.38:0/885213581 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89700fff80 0x7f897019a930 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f8964004760 tx=0x7f8964004790 comp rx=0 tx=0).stop 2023-03-09T14:45:49.563 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.560+0000 7f8974d0f700 1 -- 172.21.15.38:0/885213581 shutdown_connections 2023-03-09T14:45:49.563 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.560+0000 7f8974d0f700 1 --2- 172.21.15.38:0/885213581 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f895809f3c0 0x7f89580a1870 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:49.563 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.560+0000 7f8974d0f700 1 --2- 172.21.15.38:0/885213581 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8970100910 0x7f897019cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:49.564 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.560+0000 7f8974d0f700 1 --2- 172.21.15.38:0/885213581 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f89700fed90 0x7f8970069780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:49.564 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.560+0000 7f8974d0f700 1 --2- 172.21.15.38:0/885213581 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f89700fff80 0x7f897019a930 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:49.564 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.560+0000 7f8974d0f700 1 -- 172.21.15.38:0/885213581 >> 172.21.15.38:0/885213581 conn(0x7f89700fa3b0 msgr2=0x7f89700fc5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:49.564 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.560+0000 7f8974d0f700 1 -- 172.21.15.38:0/885213581 shutdown_connections 2023-03-09T14:45:49.564 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.560+0000 7f8974d0f700 1 -- 172.21.15.38:0/885213581 wait complete. 2023-03-09T14:45:49.575 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-03-09T14:45:49.704 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.701+0000 7fc7cf5e6700 1 -- 172.21.15.38:0/3307867424 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7c8100910 msgr2=0x7fc7c81071d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:49.704 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.701+0000 7fc7cf5e6700 1 --2- 172.21.15.38:0/3307867424 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7c8100910 0x7fc7c81071d0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fc7c000b0a0 tx=0x7fc7c000b3b0 comp rx=0 tx=0).stop 2023-03-09T14:45:49.704 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.701+0000 7fc7cf5e6700 1 -- 172.21.15.38:0/3307867424 shutdown_connections 2023-03-09T14:45:49.704 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.701+0000 7fc7cf5e6700 1 --2- 172.21.15.38:0/3307867424 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7c8100910 0x7fc7c81071d0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:49.704 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.701+0000 7fc7cf5e6700 1 --2- 172.21.15.38:0/3307867424 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7c80fed90 0x7fc7c80ff1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:49.704 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.701+0000 7fc7cf5e6700 1 --2- 172.21.15.38:0/3307867424 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc7c80fff80 0x7fc7c81003d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:49.705 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.701+0000 7fc7cf5e6700 1 -- 172.21.15.38:0/3307867424 >> 172.21.15.38:0/3307867424 conn(0x7fc7c80fa3b0 msgr2=0x7fc7c80fc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:49.705 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.701+0000 7fc7cf5e6700 1 -- 172.21.15.38:0/3307867424 shutdown_connections 2023-03-09T14:45:49.705 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.701+0000 7fc7cf5e6700 1 -- 172.21.15.38:0/3307867424 wait complete. 2023-03-09T14:45:49.707 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.702+0000 7fc7cf5e6700 1 Processor -- start 2023-03-09T14:45:49.707 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.702+0000 7fc7cf5e6700 1 -- start start 2023-03-09T14:45:49.707 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.702+0000 7fc7cf5e6700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc7c80fed90 0x7fc7c80685d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:49.707 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.702+0000 7fc7cf5e6700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7c80fff80 0x7fc7c819ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:49.707 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.702+0000 7fc7cf5e6700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7c8100910 0x7fc7c819d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:49.708 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.702+0000 7fc7cf5e6700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c819db00 con 0x7fc7c80fff80 2023-03-09T14:45:49.708 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.702+0000 7fc7cf5e6700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c819dc70 con 0x7fc7c8100910 2023-03-09T14:45:49.708 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.702+0000 7fc7cf5e6700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7c819de30 con 0x7fc7c80fed90 2023-03-09T14:45:49.708 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.702+0000 7fc7ccb81700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7c80fff80 0x7fc7c819ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:49.708 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.702+0000 7fc7ccb81700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7c80fff80 0x7fc7c819ace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38062/0 (socket says 172.21.15.38:38062) 2023-03-09T14:45:49.708 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.702+0000 7fc7ccb81700 1 -- 172.21.15.38:0/469611873 learned_addr learned my addr 172.21.15.38:0/469611873 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:49.708 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.702+0000 7fc7cd382700 1 --2- 172.21.15.38:0/469611873 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc7c80fed90 0x7fc7c80685d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:49.709 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.702+0000 7fc7cdb83700 1 --2- 172.21.15.38:0/469611873 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7c8100910 0x7fc7c819d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:49.709 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.702+0000 7fc7ccb81700 1 -- 172.21.15.38:0/469611873 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc7c80fed90 msgr2=0x7fc7c80685d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:49.709 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.702+0000 7fc7ccb81700 1 --2- 172.21.15.38:0/469611873 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc7c80fed90 0x7fc7c80685d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:49.709 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.703+0000 7fc7ccb81700 1 -- 172.21.15.38:0/469611873 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7c8100910 msgr2=0x7fc7c819d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:49.710 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.703+0000 7fc7ccb81700 1 --2- 172.21.15.38:0/469611873 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7c8100910 0x7fc7c819d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:49.710 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.703+0000 7fc7ccb81700 1 -- 172.21.15.38:0/469611873 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7c0009d00 con 0x7fc7c80fff80 2023-03-09T14:45:49.710 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.703+0000 7fc7ccb81700 1 --2- 172.21.15.38:0/469611873 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7c80fff80 0x7fc7c819ace0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fc7c400b530 tx=0x7fc7c400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:49.710 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.703+0000 7fc7be7fc700 1 -- 172.21.15.38:0/469611873 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7c4011830 con 0x7fc7c80fff80 2023-03-09T14:45:49.710 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.703+0000 7fc7cf5e6700 1 -- 172.21.15.38:0/469611873 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7c810a7c0 con 0x7fc7c80fff80 2023-03-09T14:45:49.710 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.703+0000 7fc7be7fc700 1 -- 172.21.15.38:0/469611873 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc7c4011e50 con 0x7fc7c80fff80 2023-03-09T14:45:49.710 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.703+0000 7fc7be7fc700 1 -- 172.21.15.38:0/469611873 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7c400f2e0 con 0x7fc7c80fff80 2023-03-09T14:45:49.711 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.703+0000 7fc7cf5e6700 1 -- 172.21.15.38:0/469611873 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7c810a630 con 0x7fc7c80fff80 2023-03-09T14:45:49.711 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.704+0000 7fc7cf5e6700 1 -- 172.21.15.38:0/469611873 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7c8106070 con 0x7fc7c80fff80 2023-03-09T14:45:49.711 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.705+0000 7fc7be7fc700 1 -- 172.21.15.38:0/469611873 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) v1 ==== 133198+0+0 (secure 0 0 0) 0x7fc7c4021070 con 0x7fc7c80fff80 2023-03-09T14:45:49.711 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.705+0000 7fc7be7fc700 1 --2- 172.21.15.38:0/469611873 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fc7b409f3c0 0x7fc7b40a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:49.712 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.706+0000 7fc7cd382700 1 --2- 172.21.15.38:0/469611873 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fc7b409f3c0 0x7fc7b40a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:49.712 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.706+0000 7fc7be7fc700 1 -- 172.21.15.38:0/469611873 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fc7c40c99a0 con 0x7fc7c80fff80 2023-03-09T14:45:49.712 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.706+0000 7fc7cd382700 1 --2- 172.21.15.38:0/469611873 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fc7b409f3c0 0x7fc7b40a1870 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fc7c80ffde0 tx=0x7fc7b800f820 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:49.712 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.710+0000 7fc7be7fc700 1 -- 172.21.15.38:0/469611873 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7fc7c4093a50 con 0x7fc7c80fff80 2023-03-09T14:45:49.914 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.912+0000 7fc7cf5e6700 1 -- 172.21.15.38:0/469611873 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fc7c8061850 con 0x7fc7c80fff80 2023-03-09T14:45:49.914 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.912+0000 7fc7be7fc700 1 -- 172.21.15.38:0/469611873 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) v1 ==== 103+0+11 (secure 0 0 0) 0x7fc7c4015070 con 0x7fc7c80fff80 2023-03-09T14:45:49.915 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 32 2023-03-09T14:45:49.917 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.914+0000 7fc7cf5e6700 1 -- 172.21.15.38:0/469611873 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fc7b409f3c0 msgr2=0x7fc7b40a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:49.917 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.914+0000 7fc7cf5e6700 1 --2- 172.21.15.38:0/469611873 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fc7b409f3c0 0x7fc7b40a1870 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fc7c80ffde0 tx=0x7fc7b800f820 comp rx=0 tx=0).stop 2023-03-09T14:45:49.917 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.914+0000 7fc7cf5e6700 1 -- 172.21.15.38:0/469611873 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7c80fff80 msgr2=0x7fc7c819ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:49.917 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.914+0000 7fc7cf5e6700 1 --2- 172.21.15.38:0/469611873 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7c80fff80 0x7fc7c819ace0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fc7c400b530 tx=0x7fc7c400b840 comp rx=0 tx=0).stop 2023-03-09T14:45:49.918 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.914+0000 7fc7cf5e6700 1 -- 172.21.15.38:0/469611873 shutdown_connections 2023-03-09T14:45:49.918 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.914+0000 7fc7cf5e6700 1 --2- 172.21.15.38:0/469611873 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fc7b409f3c0 0x7fc7b40a1870 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:49.918 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.914+0000 7fc7cf5e6700 1 --2- 172.21.15.38:0/469611873 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7c8100910 0x7fc7c819d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:49.918 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.914+0000 7fc7cf5e6700 1 --2- 172.21.15.38:0/469611873 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc7c80fed90 0x7fc7c80685d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:49.918 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.914+0000 7fc7cf5e6700 1 --2- 172.21.15.38:0/469611873 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7c80fff80 0x7fc7c819ace0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:49.918 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.914+0000 7fc7cf5e6700 1 -- 172.21.15.38:0/469611873 >> 172.21.15.38:0/469611873 conn(0x7fc7c80fa3b0 msgr2=0x7fc7c81031b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:49.918 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.914+0000 7fc7cf5e6700 1 -- 172.21.15.38:0/469611873 shutdown_connections 2023-03-09T14:45:49.919 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:49.914+0000 7fc7cf5e6700 1 -- 172.21.15.38:0/469611873 wait complete. 2023-03-09T14:45:49.929 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-09T14:45:50.055 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.053+0000 7fba66666700 1 -- 172.21.15.38:0/648325986 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fba600fff80 msgr2=0x7fba601003d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:50.056 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.053+0000 7fba66666700 1 --2- 172.21.15.38:0/648325986 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fba600fff80 0x7fba601003d0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fba48009990 tx=0x7fba48009ca0 comp rx=0 tx=0).stop 2023-03-09T14:45:50.056 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.053+0000 7fba66666700 1 -- 172.21.15.38:0/648325986 shutdown_connections 2023-03-09T14:45:50.056 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.053+0000 7fba66666700 1 --2- 172.21.15.38:0/648325986 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fba60100910 0x7fba601071d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:50.056 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.053+0000 7fba66666700 1 --2- 172.21.15.38:0/648325986 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba600fed90 0x7fba600ff1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:50.056 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.053+0000 7fba66666700 1 --2- 172.21.15.38:0/648325986 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fba600fff80 0x7fba601003d0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:50.057 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.053+0000 7fba66666700 1 -- 172.21.15.38:0/648325986 >> 172.21.15.38:0/648325986 conn(0x7fba600fa370 msgr2=0x7fba600fc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:50.057 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.053+0000 7fba66666700 1 -- 172.21.15.38:0/648325986 shutdown_connections 2023-03-09T14:45:50.057 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.053+0000 7fba66666700 1 -- 172.21.15.38:0/648325986 wait complete. 2023-03-09T14:45:50.057 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.054+0000 7fba66666700 1 Processor -- start 2023-03-09T14:45:50.057 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.054+0000 7fba66666700 1 -- start start 2023-03-09T14:45:50.059 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.054+0000 7fba66666700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fba600fed90 0x7fba60195be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:50.059 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.054+0000 7fba66666700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fba600fff80 0x7fba60196120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:50.059 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.054+0000 7fba66666700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba60100910 0x7fba6019d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:50.059 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.054+0000 7fba66666700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba6019d760 con 0x7fba600fff80 2023-03-09T14:45:50.059 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.054+0000 7fba66666700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba6019d8a0 con 0x7fba60100910 2023-03-09T14:45:50.060 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.054+0000 7fba66666700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba6019da30 con 0x7fba600fed90 2023-03-09T14:45:50.060 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.054+0000 7fba57fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fba600fff80 0x7fba60196120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:50.060 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.054+0000 7fba57fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fba600fff80 0x7fba60196120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38076/0 (socket says 172.21.15.38:38076) 2023-03-09T14:45:50.060 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.054+0000 7fba57fff700 1 -- 172.21.15.38:0/4232633290 learned_addr learned my addr 172.21.15.38:0/4232633290 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:50.060 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.054+0000 7fba5ffff700 1 --2- 172.21.15.38:0/4232633290 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fba600fed90 0x7fba60195be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:50.060 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.054+0000 7fba64c03700 1 --2- 172.21.15.38:0/4232633290 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba60100910 0x7fba6019d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:50.060 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.054+0000 7fba57fff700 1 -- 172.21.15.38:0/4232633290 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fba600fed90 msgr2=0x7fba60195be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:50.061 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.054+0000 7fba57fff700 1 --2- 172.21.15.38:0/4232633290 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fba600fed90 0x7fba60195be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:50.061 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.054+0000 7fba57fff700 1 -- 172.21.15.38:0/4232633290 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba60100910 msgr2=0x7fba6019d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:50.061 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.054+0000 7fba57fff700 1 --2- 172.21.15.38:0/4232633290 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba60100910 0x7fba6019d190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:50.061 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.054+0000 7fba57fff700 1 -- 172.21.15.38:0/4232633290 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba480095f0 con 0x7fba600fff80 2023-03-09T14:45:50.061 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.054+0000 7fba57fff700 1 --2- 172.21.15.38:0/4232633290 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fba600fff80 0x7fba60196120 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fba48000c00 tx=0x7fba4800f510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:50.061 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.054+0000 7fba5dffb700 1 -- 172.21.15.38:0/4232633290 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba4801d070 con 0x7fba600fff80 2023-03-09T14:45:50.061 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.055+0000 7fba66666700 1 -- 172.21.15.38:0/4232633290 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba6010b800 con 0x7fba600fff80 2023-03-09T14:45:50.062 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.055+0000 7fba5dffb700 1 -- 172.21.15.38:0/4232633290 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fba4800fc60 con 0x7fba600fff80 2023-03-09T14:45:50.062 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.055+0000 7fba5dffb700 1 -- 172.21.15.38:0/4232633290 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba48017650 con 0x7fba600fff80 2023-03-09T14:45:50.062 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.055+0000 7fba66666700 1 -- 172.21.15.38:0/4232633290 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba6010b670 con 0x7fba600fff80 2023-03-09T14:45:50.062 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.056+0000 7fba66666700 1 -- 172.21.15.38:0/4232633290 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba60105cc0 con 0x7fba600fff80 2023-03-09T14:45:50.062 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.057+0000 7fba5dffb700 1 -- 172.21.15.38:0/4232633290 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) v1 ==== 133198+0+0 (secure 0 0 0) 0x7fba48029030 con 0x7fba600fff80 2023-03-09T14:45:50.063 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.057+0000 7fba5dffb700 1 --2- 172.21.15.38:0/4232633290 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fba4c09f3c0 0x7fba4c0a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:50.063 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.058+0000 7fba5ffff700 1 --2- 172.21.15.38:0/4232633290 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fba4c09f3c0 0x7fba4c0a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:50.063 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.058+0000 7fba5dffb700 1 -- 172.21.15.38:0/4232633290 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fba480cb660 con 0x7fba600fff80 2023-03-09T14:45:50.063 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.058+0000 7fba5ffff700 1 --2- 172.21.15.38:0/4232633290 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fba4c09f3c0 0x7fba4c0a1870 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fba600ffde0 tx=0x7fba50009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:50.064 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.062+0000 7fba5dffb700 1 -- 172.21.15.38:0/4232633290 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7fba480957a0 con 0x7fba600fff80 2023-03-09T14:45:50.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:49 smithi122 ceph-mon[93869]: pgmap v306: 129 pgs: 129 active+clean; 636 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2023-03-09T14:45:50.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:49 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/2468259754' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-03-09T14:45:50.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:49 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/885213581' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-03-09T14:45:50.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:49 smithi181 ceph-mon[94189]: pgmap v306: 129 pgs: 129 active+clean; 636 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2023-03-09T14:45:50.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:49 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/2468259754' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-03-09T14:45:50.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:49 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/885213581' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-03-09T14:45:50.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:49 smithi038 ceph-mon[92422]: pgmap v306: 129 pgs: 129 active+clean; 636 KiB data, 81 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2023-03-09T14:45:50.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:49 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/2468259754' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-03-09T14:45:50.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:49 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/885213581' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-03-09T14:45:50.310 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.308+0000 7fba66666700 1 -- 172.21.15.38:0/4232633290 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fba60061830 con 0x7fba600fff80 2023-03-09T14:45:50.311 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.309+0000 7fba5dffb700 1 -- 172.21.15.38:0/4232633290 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+10801 (secure 0 0 0) 0x7fba48027090 con 0x7fba600fff80 2023-03-09T14:45:50.311 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:45:50.312 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/768918602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":768918602},{"type":"v1","addr":"172.21.15.38:6835","nonce":768918602}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/18381540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":18381540},{"type":"v1","addr":"172.21.15.38:6837","nonce":18381540}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14610,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6839/4139020066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":4139020066},{"type":"v1","addr":"172.21.15.38:6839","nonce":4139020066}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14622,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6841/1397115322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":1397115322},{"type":"v1","addr":"172.21.15.38:6841","nonce":1397115322}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6833/415305571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":415305571},{"type":"v1","addr":"172.21.15.181:6833","nonce":415305571}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/659300520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":659300520},{"type":"v1","addr":"172.21.15.122:6833","nonce":659300520}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-03-09T14:44:19.083770+0000","modified":"2023-03-09T14:44:39.526150+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24530,"mds_1":24494,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24448":{"gid":24448,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6835/2576255861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":2576255861},{"type":"v1","addr":"172.21.15.181:6835","nonce":2576255861}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24460":{"gid":24460,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6837/2942560181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":2942560181},{"type":"v1","addr":"172.21.15.181:6837","nonce":2942560181}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24464":{"gid":24464,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.122:6835/3675537941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":3675537941},{"type":"v1","addr":"172.21.15.122:6835","nonce":3675537941}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24478":{"gid":24478,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":9,"addr":"172.21.15.181:6839/3902263660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":3902263660},{"type":"v1","addr":"172.21.15.181:6839","nonce":3902263660}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24494":{"gid":24494,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.122:6837/2210214596","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":2210214596},{"type":"v1","addr":"172.21.15.122:6837","nonce":2210214596}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24530":{"gid":24530,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6839/971050165","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6838","nonce":971050165},{"type":"v1","addr":"172.21.15.122:6839","nonce":971050165}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-09T14:45:50.313 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.311+0000 7fba66666700 1 -- 172.21.15.38:0/4232633290 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fba4c09f3c0 msgr2=0x7fba4c0a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:50.313 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.311+0000 7fba66666700 1 --2- 172.21.15.38:0/4232633290 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fba4c09f3c0 0x7fba4c0a1870 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fba600ffde0 tx=0x7fba50009260 comp rx=0 tx=0).stop 2023-03-09T14:45:50.313 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.311+0000 7fba66666700 1 -- 172.21.15.38:0/4232633290 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fba600fff80 msgr2=0x7fba60196120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:50.314 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.311+0000 7fba66666700 1 --2- 172.21.15.38:0/4232633290 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fba600fff80 0x7fba60196120 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fba48000c00 tx=0x7fba4800f510 comp rx=0 tx=0).stop 2023-03-09T14:45:50.314 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.311+0000 7fba66666700 1 -- 172.21.15.38:0/4232633290 shutdown_connections 2023-03-09T14:45:50.314 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.311+0000 7fba66666700 1 --2- 172.21.15.38:0/4232633290 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fba4c09f3c0 0x7fba4c0a1870 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:50.314 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.311+0000 7fba66666700 1 --2- 172.21.15.38:0/4232633290 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fba60100910 0x7fba6019d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:50.314 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.311+0000 7fba66666700 1 --2- 172.21.15.38:0/4232633290 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fba600fed90 0x7fba60195be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:50.314 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.311+0000 7fba66666700 1 --2- 172.21.15.38:0/4232633290 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fba600fff80 0x7fba60196120 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:50.314 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.311+0000 7fba66666700 1 -- 172.21.15.38:0/4232633290 >> 172.21.15.38:0/4232633290 conn(0x7fba600fa370 msgr2=0x7fba600fc610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:50.315 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.311+0000 7fba66666700 1 -- 172.21.15.38:0/4232633290 shutdown_connections 2023-03-09T14:45:50.315 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.311+0000 7fba66666700 1 -- 172.21.15.38:0/4232633290 wait complete. 2023-03-09T14:45:50.315 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 23 2023-03-09T14:45:50.326 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-09T14:45:50.452 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.449+0000 7f087d467700 1 -- 172.21.15.38:0/2838105342 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08780fffc0 msgr2=0x7f0878100410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:50.452 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.449+0000 7f087d467700 1 --2- 172.21.15.38:0/2838105342 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08780fffc0 0x7f0878100410 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f0860009990 tx=0x7f0860009ca0 comp rx=0 tx=0).stop 2023-03-09T14:45:50.452 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.449+0000 7f087d467700 1 -- 172.21.15.38:0/2838105342 shutdown_connections 2023-03-09T14:45:50.452 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.449+0000 7f087d467700 1 --2- 172.21.15.38:0/2838105342 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0878100950 0x7f0878107210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:50.452 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.449+0000 7f087d467700 1 --2- 172.21.15.38:0/2838105342 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f08780fedd0 0x7f08780ff1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:50.453 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.449+0000 7f087d467700 1 --2- 172.21.15.38:0/2838105342 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08780fffc0 0x7f0878100410 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:50.453 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.449+0000 7f087d467700 1 -- 172.21.15.38:0/2838105342 >> 172.21.15.38:0/2838105342 conn(0x7f08780fa3d0 msgr2=0x7f08780fc820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:50.453 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.449+0000 7f087d467700 1 -- 172.21.15.38:0/2838105342 shutdown_connections 2023-03-09T14:45:50.453 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.449+0000 7f087d467700 1 -- 172.21.15.38:0/2838105342 wait complete. 2023-03-09T14:45:50.455 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.450+0000 7f087d467700 1 Processor -- start 2023-03-09T14:45:50.455 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.450+0000 7f087d467700 1 -- start start 2023-03-09T14:45:50.455 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.450+0000 7f087d467700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f08780fedd0 0x7f0878195c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:50.455 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.451+0000 7f087d467700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08780fffc0 0x7f0878196180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:50.455 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.451+0000 7f087d467700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0878100950 0x7f087819d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:50.456 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.451+0000 7f087d467700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f087819d780 con 0x7f08780fffc0 2023-03-09T14:45:50.456 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.451+0000 7f087d467700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f087819d8f0 con 0x7f0878100950 2023-03-09T14:45:50.456 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.451+0000 7f087d467700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f087819dab0 con 0x7f08780fedd0 2023-03-09T14:45:50.456 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.451+0000 7f08767fc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08780fffc0 0x7f0878196180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:50.456 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.451+0000 7f08767fc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08780fffc0 0x7f0878196180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38090/0 (socket says 172.21.15.38:38090) 2023-03-09T14:45:50.456 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.451+0000 7f08767fc700 1 -- 172.21.15.38:0/163461557 learned_addr learned my addr 172.21.15.38:0/163461557 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:50.456 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.451+0000 7f0876ffd700 1 --2- 172.21.15.38:0/163461557 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f08780fedd0 0x7f0878195c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:50.456 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.451+0000 7f08777fe700 1 --2- 172.21.15.38:0/163461557 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0878100950 0x7f087819d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:50.457 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.451+0000 7f08767fc700 1 -- 172.21.15.38:0/163461557 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f08780fedd0 msgr2=0x7f0878195c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:50.457 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.451+0000 7f08767fc700 1 --2- 172.21.15.38:0/163461557 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f08780fedd0 0x7f0878195c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:50.457 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.451+0000 7f08767fc700 1 -- 172.21.15.38:0/163461557 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0878100950 msgr2=0x7f087819d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:50.457 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.451+0000 7f08767fc700 1 --2- 172.21.15.38:0/163461557 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0878100950 0x7f087819d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:50.457 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.451+0000 7f08767fc700 1 -- 172.21.15.38:0/163461557 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08600095f0 con 0x7f08780fffc0 2023-03-09T14:45:50.458 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.451+0000 7f08777fe700 1 --2- 172.21.15.38:0/163461557 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0878100950 0x7f087819d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-09T14:45:50.458 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.451+0000 7f0876ffd700 1 --2- 172.21.15.38:0/163461557 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f08780fedd0 0x7f0878195c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-09T14:45:50.458 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.451+0000 7f08767fc700 1 --2- 172.21.15.38:0/163461557 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08780fffc0 0x7f0878196180 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f0860004760 tx=0x7f0860009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:50.458 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.451+0000 7f086ffff700 1 -- 172.21.15.38:0/163461557 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f086001d070 con 0x7f08780fffc0 2023-03-09T14:45:50.458 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.451+0000 7f086ffff700 1 -- 172.21.15.38:0/163461557 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0860004200 con 0x7f08780fffc0 2023-03-09T14:45:50.458 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.451+0000 7f087d467700 1 -- 172.21.15.38:0/163461557 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f087810b840 con 0x7f08780fffc0 2023-03-09T14:45:50.458 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.451+0000 7f086ffff700 1 -- 172.21.15.38:0/163461557 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08600049f0 con 0x7f08780fffc0 2023-03-09T14:45:50.458 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.451+0000 7f087d467700 1 -- 172.21.15.38:0/163461557 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f087810b6b0 con 0x7f08780fffc0 2023-03-09T14:45:50.459 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.452+0000 7f087d467700 1 -- 172.21.15.38:0/163461557 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0878106370 con 0x7f08780fffc0 2023-03-09T14:45:50.459 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.454+0000 7f086ffff700 1 -- 172.21.15.38:0/163461557 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) v1 ==== 133198+0+0 (secure 0 0 0) 0x7f0860028030 con 0x7f08780fffc0 2023-03-09T14:45:50.459 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.455+0000 7f086ffff700 1 --2- 172.21.15.38:0/163461557 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f086409f3c0 0x7f08640a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:50.460 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.455+0000 7f0876ffd700 1 --2- 172.21.15.38:0/163461557 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f086409f3c0 0x7f08640a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:50.460 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.455+0000 7f086ffff700 1 -- 172.21.15.38:0/163461557 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f08600cb3f0 con 0x7f08780fffc0 2023-03-09T14:45:50.460 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.455+0000 7f0876ffd700 1 --2- 172.21.15.38:0/163461557 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f086409f3c0 0x7f08640a1870 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f0868005fd0 tx=0x7f0868005f20 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:50.460 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.458+0000 7f086ffff700 1 -- 172.21.15.38:0/163461557 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f08600954d0 con 0x7f08780fffc0 2023-03-09T14:45:50.662 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.660+0000 7f087d467700 1 -- 172.21.15.38:0/163461557 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f087802cef0 con 0x7f08780fffc0 2023-03-09T14:45:50.664 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.662+0000 7f086ffff700 1 -- 172.21.15.38:0/163461557 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24445 (secure 0 0 0) 0x7f0860026090 con 0x7f08780fffc0 2023-03-09T14:45:50.665 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:45:50.665 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":89,"fsid":"c6733fe2-be87-11ed-9af2-001a4aab830c","created":"2023-03-09T14:37:14.947981+0000","modified":"2023-03-09T14:45:15.848936+0000","last_up_change":"2023-03-09T14:43:37.198407+0000","last_in_change":"2023-03-09T14:43:19.550566+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-09T14:40:23.196665+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-03-09T14:44:17.484374+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-03-09T14:44:18.457833+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-03-09T14:44:21.895589+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"51477039-15ee-4707-a34d-4ecfa18bebde","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6803","nonce":1508072290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6805","nonce":1508072290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6809","nonce":1508072290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6807","nonce":1508072290}]},"public_addr":"172.21.15.38:6803/1508072290","cluster_addr":"172.21.15.38:6805/1508072290","heartbeat_back_addr":"172.21.15.38:6809/1508072290","heartbeat_front_addr":"172.21.15.38:6807/1508072290","state":["exists","up"]},{"osd":1,"uuid":"71f8a718-b0a9-4f39-bc54-267aa42a2578","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6801","nonce":3929268773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6803","nonce":3929268773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6807","nonce":3929268773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6805","nonce":3929268773}]},"public_addr":"172.21.15.122:6801/3929268773","cluster_addr":"172.21.15.122:6803/3929268773","heartbeat_back_addr":"172.21.15.122:6807/3929268773","heartbeat_front_addr":"172.21.15.122:6805/3929268773","state":["exists","up"]},{"osd":2,"uuid":"961a2cbe-140e-437f-b131-1fcac573b19c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6801","nonce":1310630803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6803","nonce":1310630803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6807","nonce":1310630803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6805","nonce":1310630803}]},"public_addr":"172.21.15.181:6801/1310630803","cluster_addr":"172.21.15.181:6803/1310630803","heartbeat_back_addr":"172.21.15.181:6807/1310630803","heartbeat_front_addr":"172.21.15.181:6805/1310630803","state":["exists","up"]},{"osd":3,"uuid":"969df8fa-2d19-42c7-9f2a-5c5fac15d4a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6811","nonce":2719814100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6813","nonce":2719814100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6817","nonce":2719814100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6815","nonce":2719814100}]},"public_addr":"172.21.15.38:6811/2719814100","cluster_addr":"172.21.15.38:6813/2719814100","heartbeat_back_addr":"172.21.15.38:6817/2719814100","heartbeat_front_addr":"172.21.15.38:6815/2719814100","state":["exists","up"]},{"osd":4,"uuid":"44a6dce3-e06e-4cbd-9404-62a86d01ab6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6809","nonce":2480764994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6811","nonce":2480764994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6815","nonce":2480764994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6813","nonce":2480764994}]},"public_addr":"172.21.15.122:6809/2480764994","cluster_addr":"172.21.15.122:6811/2480764994","heartbeat_back_addr":"172.21.15.122:6815/2480764994","heartbeat_front_addr":"172.21.15.122:6813/2480764994","state":["exists","up"]},{"osd":5,"uuid":"49749574-c00c-4e3a-80e6-e3686df58d5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6809","nonce":3141346246}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6811","nonce":3141346246}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6815","nonce":3141346246}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6813","nonce":3141346246}]},"public_addr":"172.21.15.181:6809/3141346246","cluster_addr":"172.21.15.181:6811/3141346246","heartbeat_back_addr":"172.21.15.181:6815/3141346246","heartbeat_front_addr":"172.21.15.181:6813/3141346246","state":["exists","up"]},{"osd":6,"uuid":"61c0437e-3f51-45a9-8a61-e98eea0093f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6819","nonce":2122788957}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6821","nonce":2122788957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6825","nonce":2122788957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6823","nonce":2122788957}]},"public_addr":"172.21.15.38:6819/2122788957","cluster_addr":"172.21.15.38:6821/2122788957","heartbeat_back_addr":"172.21.15.38:6825/2122788957","heartbeat_front_addr":"172.21.15.38:6823/2122788957","state":["exists","up"]},{"osd":7,"uuid":"231fecdd-939f-4c7b-83a1-c4a1cffd7139","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6817","nonce":656874139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6819","nonce":656874139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6823","nonce":656874139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6821","nonce":656874139}]},"public_addr":"172.21.15.122:6817/656874139","cluster_addr":"172.21.15.122:6819/656874139","heartbeat_back_addr":"172.21.15.122:6823/656874139","heartbeat_front_addr":"172.21.15.122:6821/656874139","state":["exists","up"]},{"osd":8,"uuid":"297e6d6d-85fd-43ce-9028-5e9dbe55c36e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6817","nonce":755000703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6819","nonce":755000703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6823","nonce":755000703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6821","nonce":755000703}]},"public_addr":"172.21.15.181:6817/755000703","cluster_addr":"172.21.15.181:6819/755000703","heartbeat_back_addr":"172.21.15.181:6823/755000703","heartbeat_front_addr":"172.21.15.181:6821/755000703","state":["exists","up"]},{"osd":9,"uuid":"94cc0355-5a1a-4cfc-bd0c-562b6b41127e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6827","nonce":2624555969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6829","nonce":2624555969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6833","nonce":2624555969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6831","nonce":2624555969}]},"public_addr":"172.21.15.38:6827/2624555969","cluster_addr":"172.21.15.38:6829/2624555969","heartbeat_back_addr":"172.21.15.38:6833/2624555969","heartbeat_front_addr":"172.21.15.38:6831/2624555969","state":["exists","up"]},{"osd":10,"uuid":"f4f2e042-8d92-46bf-93b8-8b0f04795c2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6825","nonce":1427785280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6827","nonce":1427785280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6831","nonce":1427785280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6829","nonce":1427785280}]},"public_addr":"172.21.15.122:6825/1427785280","cluster_addr":"172.21.15.122:6827/1427785280","heartbeat_back_addr":"172.21.15.122:6831/1427785280","heartbeat_front_addr":"172.21.15.122:6829/1427785280","state":["exists","up"]},{"osd":11,"uuid":"07444936-eae5-4097-a799-60294cafe80b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6825","nonce":3183488640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6827","nonce":3183488640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6831","nonce":3183488640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6829","nonce":3183488640}]},"public_addr":"172.21.15.181:6825/3183488640","cluster_addr":"172.21.15.181:6827/3183488640","heartbeat_back_addr":"172.21.15.181:6831/3183488640","heartbeat_front_addr":"172.21.15.181:6829/3183488640","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:39:37.657556+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:39:58.960074+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:40:20.326952+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:40:42.244387+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:41:03.946973+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:41:25.097985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:41:46.877916+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:42:08.413827+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:42:29.822009+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:42:51.776684+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:43:13.937249+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:43:35.569199+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.2","mappings":[{"from":7,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.d","mappings":[{"from":5,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.17","mappings":[{"from":5,"to":6}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/4272288735":"2023-03-10T14:38:14.612767+0000","172.21.15.38:6801/3381625669":"2023-03-10T14:38:14.612767+0000","172.21.15.38:6800/3381625669":"2023-03-10T14:38:14.612767+0000","172.21.15.38:6800/3060474342":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/4253918615":"2023-03-10T14:38:14.612767+0000","172.21.15.38:0/3496083181":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/245602368":"2023-03-10T14:37:47.517775+0000","172.21.15.38:6801/3060474342":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/3622589182":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/1430302251":"2023-03-10T14:38:14.612767+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-09T14:45:50.666 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.664+0000 7f087d467700 1 -- 172.21.15.38:0/163461557 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f086409f3c0 msgr2=0x7f08640a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:50.666 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.664+0000 7f087d467700 1 --2- 172.21.15.38:0/163461557 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f086409f3c0 0x7f08640a1870 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f0868005fd0 tx=0x7f0868005f20 comp rx=0 tx=0).stop 2023-03-09T14:45:50.667 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.664+0000 7f087d467700 1 -- 172.21.15.38:0/163461557 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08780fffc0 msgr2=0x7f0878196180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:50.667 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.664+0000 7f087d467700 1 --2- 172.21.15.38:0/163461557 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08780fffc0 0x7f0878196180 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f0860004760 tx=0x7f0860009f90 comp rx=0 tx=0).stop 2023-03-09T14:45:50.667 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.664+0000 7f087d467700 1 -- 172.21.15.38:0/163461557 shutdown_connections 2023-03-09T14:45:50.667 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.664+0000 7f087d467700 1 --2- 172.21.15.38:0/163461557 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f086409f3c0 0x7f08640a1870 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:50.667 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.664+0000 7f087d467700 1 --2- 172.21.15.38:0/163461557 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0878100950 0x7f087819d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:50.667 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.664+0000 7f087d467700 1 --2- 172.21.15.38:0/163461557 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f08780fedd0 0x7f0878195c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:50.667 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.664+0000 7f087d467700 1 --2- 172.21.15.38:0/163461557 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f08780fffc0 0x7f0878196180 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:50.668 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.664+0000 7f087d467700 1 -- 172.21.15.38:0/163461557 >> 172.21.15.38:0/163461557 conn(0x7f08780fa3d0 msgr2=0x7f08781034b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:50.668 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.664+0000 7f087d467700 1 -- 172.21.15.38:0/163461557 shutdown_connections 2023-03-09T14:45:50.668 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.664+0000 7f087d467700 1 -- 172.21.15.38:0/163461557 wait complete. 2023-03-09T14:45:50.679 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-09T14:40:23.196665+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-03-09T14:44:17.484374+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-03-09T14:44:18.457833+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-03-09T14:44:21.895589+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-03-09T14:45:50.680 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-09T14:45:50.810 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.806+0000 7f10230ea700 1 -- 172.21.15.38:0/639166537 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f101c100910 msgr2=0x7f101c1071d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:50.810 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.806+0000 7f10230ea700 1 --2- 172.21.15.38:0/639166537 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f101c100910 0x7f101c1071d0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f101400b0a0 tx=0x7f101400b3b0 comp rx=0 tx=0).stop 2023-03-09T14:45:50.811 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.806+0000 7f10230ea700 1 -- 172.21.15.38:0/639166537 shutdown_connections 2023-03-09T14:45:50.811 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.806+0000 7f10230ea700 1 --2- 172.21.15.38:0/639166537 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f101c100910 0x7f101c1071d0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:50.811 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.806+0000 7f10230ea700 1 --2- 172.21.15.38:0/639166537 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f101c0fed90 0x7f101c0ff1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:50.811 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.806+0000 7f10230ea700 1 --2- 172.21.15.38:0/639166537 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f101c0fff80 0x7f101c1003d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:50.811 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.806+0000 7f10230ea700 1 -- 172.21.15.38:0/639166537 >> 172.21.15.38:0/639166537 conn(0x7f101c0fa3b0 msgr2=0x7f101c0fc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:50.811 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.806+0000 7f10230ea700 1 -- 172.21.15.38:0/639166537 shutdown_connections 2023-03-09T14:45:50.811 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.806+0000 7f10230ea700 1 -- 172.21.15.38:0/639166537 wait complete. 2023-03-09T14:45:50.812 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.807+0000 7f10230ea700 1 Processor -- start 2023-03-09T14:45:50.812 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.807+0000 7f10230ea700 1 -- start start 2023-03-09T14:45:50.812 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.807+0000 7f10230ea700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f101c0fed90 0x7f101c0685b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:50.812 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.807+0000 7f10230ea700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f101c0fff80 0x7f101c19ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:50.812 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.807+0000 7f10230ea700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f101c100910 0x7f101c19d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:50.812 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.807+0000 7f10230ea700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f101c19da10 con 0x7f101c0fed90 2023-03-09T14:45:50.812 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.807+0000 7f10230ea700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f101c19db80 con 0x7f101c100910 2023-03-09T14:45:50.813 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.807+0000 7f10230ea700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f101c19dd40 con 0x7f101c0fff80 2023-03-09T14:45:50.813 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.807+0000 7f1020e86700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f101c0fed90 0x7f101c0685b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:50.813 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.807+0000 7f1020e86700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f101c0fed90 0x7f101c0685b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38102/0 (socket says 172.21.15.38:38102) 2023-03-09T14:45:50.813 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.807+0000 7f1020e86700 1 -- 172.21.15.38:0/3323647816 learned_addr learned my addr 172.21.15.38:0/3323647816 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:50.813 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.808+0000 7f101bfff700 1 --2- 172.21.15.38:0/3323647816 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f101c0fff80 0x7f101c19ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:50.813 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.808+0000 7f1020e86700 1 -- 172.21.15.38:0/3323647816 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f101c0fff80 msgr2=0x7f101c19ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:50.813 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.808+0000 7f1020e86700 1 --2- 172.21.15.38:0/3323647816 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f101c0fff80 0x7f101c19ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:50.813 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.808+0000 7f1020e86700 1 -- 172.21.15.38:0/3323647816 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f101c100910 msgr2=0x7f101c19d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:50.814 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.808+0000 7f1020e86700 1 --2- 172.21.15.38:0/3323647816 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f101c100910 0x7f101c19d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:50.814 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.808+0000 7f1020e86700 1 -- 172.21.15.38:0/3323647816 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1014009d00 con 0x7f101c0fed90 2023-03-09T14:45:50.814 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.808+0000 7f101bfff700 1 --2- 172.21.15.38:0/3323647816 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f101c0fff80 0x7f101c19ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-09T14:45:50.814 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.808+0000 7f1020e86700 1 --2- 172.21.15.38:0/3323647816 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f101c0fed90 0x7f101c0685b0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f101000d980 tx=0x7f101000dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:50.814 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.808+0000 7f1019ffb700 1 -- 172.21.15.38:0/3323647816 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f101000de60 con 0x7f101c0fed90 2023-03-09T14:45:50.814 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.808+0000 7f1019ffb700 1 -- 172.21.15.38:0/3323647816 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f10100044e0 con 0x7f101c0fed90 2023-03-09T14:45:50.814 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.808+0000 7f1019ffb700 1 -- 172.21.15.38:0/3323647816 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1010003be0 con 0x7f101c0fed90 2023-03-09T14:45:50.815 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.808+0000 7f10230ea700 1 -- 172.21.15.38:0/3323647816 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f101c10a750 con 0x7f101c0fed90 2023-03-09T14:45:50.815 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.808+0000 7f10230ea700 1 -- 172.21.15.38:0/3323647816 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f101c10a5c0 con 0x7f101c0fed90 2023-03-09T14:45:50.815 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.809+0000 7f10230ea700 1 -- 172.21.15.38:0/3323647816 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f101c106070 con 0x7f101c0fed90 2023-03-09T14:45:50.816 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.810+0000 7f1019ffb700 1 -- 172.21.15.38:0/3323647816 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) v1 ==== 133198+0+0 (secure 0 0 0) 0x7f101000b030 con 0x7f101c0fed90 2023-03-09T14:45:50.816 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.811+0000 7f1019ffb700 1 --2- 172.21.15.38:0/3323647816 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f100c09f3c0 0x7f100c0a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:50.816 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.811+0000 7f1019ffb700 1 -- 172.21.15.38:0/3323647816 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f10100c9d50 con 0x7f101c0fed90 2023-03-09T14:45:50.816 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.811+0000 7f101bfff700 1 --2- 172.21.15.38:0/3323647816 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f100c09f3c0 0x7f100c0a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:50.816 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.811+0000 7f101bfff700 1 --2- 172.21.15.38:0/3323647816 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f100c09f3c0 0x7f100c0a1870 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1008009be0 tx=0x7f1008009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:50.817 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:50.815+0000 7f1019ffb700 1 -- 172.21.15.38:0/3323647816 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f1010093e00 con 0x7f101c0fed90 2023-03-09T14:45:51.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:50 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/469611873' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-03-09T14:45:51.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:50 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/4232633290' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-03-09T14:45:51.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:50 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/163461557' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-03-09T14:45:51.017 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.015+0000 7f10230ea700 1 -- 172.21.15.38:0/3323647816 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f101c061540 con 0x7f101c0fed90 2023-03-09T14:45:51.018 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.015+0000 7f1019ffb700 1 -- 172.21.15.38:0/3323647816 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1010009940 con 0x7f101c0fed90 2023-03-09T14:45:51.018 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 1 2023-03-09T14:45:51.020 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.017+0000 7f10230ea700 1 -- 172.21.15.38:0/3323647816 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f100c09f3c0 msgr2=0x7f100c0a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:51.020 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.017+0000 7f10230ea700 1 --2- 172.21.15.38:0/3323647816 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f100c09f3c0 0x7f100c0a1870 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f1008009be0 tx=0x7f1008009260 comp rx=0 tx=0).stop 2023-03-09T14:45:51.020 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.017+0000 7f10230ea700 1 -- 172.21.15.38:0/3323647816 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f101c0fed90 msgr2=0x7f101c0685b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:51.020 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.017+0000 7f10230ea700 1 --2- 172.21.15.38:0/3323647816 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f101c0fed90 0x7f101c0685b0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f101000d980 tx=0x7f101000dd40 comp rx=0 tx=0).stop 2023-03-09T14:45:51.020 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.017+0000 7f10230ea700 1 -- 172.21.15.38:0/3323647816 shutdown_connections 2023-03-09T14:45:51.020 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.017+0000 7f10230ea700 1 --2- 172.21.15.38:0/3323647816 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f101c0fed90 0x7f101c0685b0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:51.021 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.017+0000 7f10230ea700 1 --2- 172.21.15.38:0/3323647816 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f100c09f3c0 0x7f100c0a1870 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:51.021 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.017+0000 7f10230ea700 1 --2- 172.21.15.38:0/3323647816 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f101c100910 0x7f101c19d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:51.021 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.017+0000 7f10230ea700 1 --2- 172.21.15.38:0/3323647816 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f101c0fff80 0x7f101c19ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:51.021 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.017+0000 7f10230ea700 1 -- 172.21.15.38:0/3323647816 >> 172.21.15.38:0/3323647816 conn(0x7f101c0fa3b0 msgr2=0x7f101c1031b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:51.021 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.017+0000 7f10230ea700 1 -- 172.21.15.38:0/3323647816 shutdown_connections 2023-03-09T14:45:51.021 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.018+0000 7f10230ea700 1 -- 172.21.15.38:0/3323647816 wait complete. 2023-03-09T14:45:51.032 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-03-09T14:45:51.161 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.158+0000 7f50f5f9e700 1 -- 172.21.15.38:0/657521781 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f50f0101930 msgr2=0x7f50f01081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:51.161 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.158+0000 7f50f5f9e700 1 --2- 172.21.15.38:0/657521781 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f50f0101930 0x7f50f01081f0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f50e800b0a0 tx=0x7f50e800b3b0 comp rx=0 tx=0).stop 2023-03-09T14:45:51.161 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.158+0000 7f50f5f9e700 1 -- 172.21.15.38:0/657521781 shutdown_connections 2023-03-09T14:45:51.162 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.158+0000 7f50f5f9e700 1 --2- 172.21.15.38:0/657521781 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f50f0101930 0x7f50f01081f0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:51.162 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.158+0000 7f50f5f9e700 1 --2- 172.21.15.38:0/657521781 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f50f00ffdb0 0x7f50f01001c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:51.162 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.158+0000 7f50f5f9e700 1 --2- 172.21.15.38:0/657521781 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f50f0100fa0 0x7f50f01013f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:51.162 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.158+0000 7f50f5f9e700 1 -- 172.21.15.38:0/657521781 >> 172.21.15.38:0/657521781 conn(0x7f50f00fb3d0 msgr2=0x7f50f00fd800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:51.162 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.159+0000 7f50f5f9e700 1 -- 172.21.15.38:0/657521781 shutdown_connections 2023-03-09T14:45:51.162 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.159+0000 7f50f5f9e700 1 -- 172.21.15.38:0/657521781 wait complete. 2023-03-09T14:45:51.164 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.159+0000 7f50f5f9e700 1 Processor -- start 2023-03-09T14:45:51.164 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.159+0000 7f50f5f9e700 1 -- start start 2023-03-09T14:45:51.164 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.159+0000 7f50f5f9e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f50f00ffdb0 0x7f50f0074530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:51.164 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.159+0000 7f50f5f9e700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f50f0100fa0 0x7f50f0072b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:51.165 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.159+0000 7f50f5f9e700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f50f0101930 0x7f50f00730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:51.165 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.159+0000 7f50f5f9e700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50f0073850 con 0x7f50f00ffdb0 2023-03-09T14:45:51.165 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.159+0000 7f50f5f9e700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50f00739c0 con 0x7f50f0100fa0 2023-03-09T14:45:51.165 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.159+0000 7f50f5f9e700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50f019fc10 con 0x7f50f0101930 2023-03-09T14:45:51.165 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.160+0000 7f50f4f9c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f50f00ffdb0 0x7f50f0074530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:51.165 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.160+0000 7f50f4f9c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f50f00ffdb0 0x7f50f0074530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38116/0 (socket says 172.21.15.38:38116) 2023-03-09T14:45:51.165 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.160+0000 7f50f4f9c700 1 -- 172.21.15.38:0/917301423 learned_addr learned my addr 172.21.15.38:0/917301423 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:51.166 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.160+0000 7f50effff700 1 --2- 172.21.15.38:0/917301423 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f50f0100fa0 0x7f50f0072b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:51.166 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.160+0000 7f50f579d700 1 --2- 172.21.15.38:0/917301423 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f50f0101930 0x7f50f00730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:51.166 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.160+0000 7f50f4f9c700 1 -- 172.21.15.38:0/917301423 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f50f0101930 msgr2=0x7f50f00730b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:51.166 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.160+0000 7f50f4f9c700 1 --2- 172.21.15.38:0/917301423 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f50f0101930 0x7f50f00730b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:51.166 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.160+0000 7f50f4f9c700 1 -- 172.21.15.38:0/917301423 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f50f0100fa0 msgr2=0x7f50f0072b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:51.166 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.160+0000 7f50f4f9c700 1 --2- 172.21.15.38:0/917301423 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f50f0100fa0 0x7f50f0072b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:51.166 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.160+0000 7f50f4f9c700 1 -- 172.21.15.38:0/917301423 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50e8009d00 con 0x7f50f00ffdb0 2023-03-09T14:45:51.166 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.160+0000 7f50f4f9c700 1 --2- 172.21.15.38:0/917301423 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f50f00ffdb0 0x7f50f0074530 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f50e400c970 tx=0x7f50e400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:51.167 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.160+0000 7f50edffb700 1 -- 172.21.15.38:0/917301423 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50e4004220 con 0x7f50f00ffdb0 2023-03-09T14:45:51.167 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.160+0000 7f50edffb700 1 -- 172.21.15.38:0/917301423 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f50e4007d60 con 0x7f50f00ffdb0 2023-03-09T14:45:51.167 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.160+0000 7f50f5f9e700 1 -- 172.21.15.38:0/917301423 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50f010b720 con 0x7f50f00ffdb0 2023-03-09T14:45:51.167 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.160+0000 7f50edffb700 1 -- 172.21.15.38:0/917301423 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50e4003c90 con 0x7f50f00ffdb0 2023-03-09T14:45:51.167 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.161+0000 7f50f5f9e700 1 -- 172.21.15.38:0/917301423 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50f010b590 con 0x7f50f00ffdb0 2023-03-09T14:45:51.167 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.161+0000 7f50f5f9e700 1 -- 172.21.15.38:0/917301423 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50f01a02e0 con 0x7f50f00ffdb0 2023-03-09T14:45:51.168 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.162+0000 7f50edffb700 1 -- 172.21.15.38:0/917301423 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) v1 ==== 133198+0+0 (secure 0 0 0) 0x7f50e401f070 con 0x7f50f00ffdb0 2023-03-09T14:45:51.168 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.163+0000 7f50edffb700 1 --2- 172.21.15.38:0/917301423 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f50e009f3c0 0x7f50e00a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:51.168 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.163+0000 7f50edffb700 1 -- 172.21.15.38:0/917301423 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f50e40c9b00 con 0x7f50f00ffdb0 2023-03-09T14:45:51.169 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.163+0000 7f50effff700 1 --2- 172.21.15.38:0/917301423 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f50e009f3c0 0x7f50e00a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:51.169 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.164+0000 7f50effff700 1 --2- 172.21.15.38:0/917301423 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f50e009f3c0 0x7f50e00a1870 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f50dc005950 tx=0x7f50dc00b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:51.169 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.167+0000 7f50edffb700 1 -- 172.21.15.38:0/917301423 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f50e4093bb0 con 0x7f50f00ffdb0 2023-03-09T14:45:51.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:50 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/469611873' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-03-09T14:45:51.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:50 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/4232633290' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-03-09T14:45:51.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:50 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/163461557' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-03-09T14:45:51.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:50 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/469611873' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-03-09T14:45:51.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:50 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/4232633290' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-03-09T14:45:51.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:50 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/163461557' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-03-09T14:45:51.370 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.368+0000 7f50f5f9e700 1 -- 172.21.15.38:0/917301423 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f50f0106bf0 con 0x7f50f00ffdb0 2023-03-09T14:45:51.371 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.368+0000 7f50edffb700 1 -- 172.21.15.38:0/917301423 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) v1 ==== 104+0+11 (secure 0 0 0) 0x7f50e40c9e60 con 0x7f50f00ffdb0 2023-03-09T14:45:51.371 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 32 2023-03-09T14:45:51.373 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.370+0000 7f50f5f9e700 1 -- 172.21.15.38:0/917301423 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f50e009f3c0 msgr2=0x7f50e00a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:51.373 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.370+0000 7f50f5f9e700 1 --2- 172.21.15.38:0/917301423 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f50e009f3c0 0x7f50e00a1870 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f50dc005950 tx=0x7f50dc00b410 comp rx=0 tx=0).stop 2023-03-09T14:45:51.373 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.370+0000 7f50f5f9e700 1 -- 172.21.15.38:0/917301423 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f50f00ffdb0 msgr2=0x7f50f0074530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:51.373 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.370+0000 7f50f5f9e700 1 --2- 172.21.15.38:0/917301423 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f50f00ffdb0 0x7f50f0074530 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f50e400c970 tx=0x7f50e400cc80 comp rx=0 tx=0).stop 2023-03-09T14:45:51.373 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.370+0000 7f50f5f9e700 1 -- 172.21.15.38:0/917301423 shutdown_connections 2023-03-09T14:45:51.373 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.370+0000 7f50f5f9e700 1 --2- 172.21.15.38:0/917301423 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f50f00ffdb0 0x7f50f0074530 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:51.374 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.370+0000 7f50f5f9e700 1 --2- 172.21.15.38:0/917301423 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f50e009f3c0 0x7f50e00a1870 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:51.374 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.370+0000 7f50f5f9e700 1 --2- 172.21.15.38:0/917301423 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f50f0101930 0x7f50f00730b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:51.374 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.370+0000 7f50f5f9e700 1 --2- 172.21.15.38:0/917301423 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f50f0100fa0 0x7f50f0072b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:51.374 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.370+0000 7f50f5f9e700 1 -- 172.21.15.38:0/917301423 >> 172.21.15.38:0/917301423 conn(0x7f50f00fb3d0 msgr2=0x7f50f00fd560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:51.374 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.371+0000 7f50f5f9e700 1 -- 172.21.15.38:0/917301423 shutdown_connections 2023-03-09T14:45:51.374 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.371+0000 7f50f5f9e700 1 -- 172.21.15.38:0/917301423 wait complete. 2023-03-09T14:45:51.385 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-03-09T14:45:51.517 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.514+0000 7fcb3afe4700 1 -- 172.21.15.38:0/2247478460 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb34100910 msgr2=0x7fcb341071d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:51.517 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.514+0000 7fcb3afe4700 1 --2- 172.21.15.38:0/2247478460 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb34100910 0x7fcb341071d0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fcb2c00b0a0 tx=0x7fcb2c00b3b0 comp rx=0 tx=0).stop 2023-03-09T14:45:51.517 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.514+0000 7fcb3afe4700 1 -- 172.21.15.38:0/2247478460 shutdown_connections 2023-03-09T14:45:51.517 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.514+0000 7fcb3afe4700 1 --2- 172.21.15.38:0/2247478460 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb34100910 0x7fcb341071d0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:51.518 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.514+0000 7fcb3afe4700 1 --2- 172.21.15.38:0/2247478460 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcb340fed90 0x7fcb340ff1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:51.518 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.514+0000 7fcb3afe4700 1 --2- 172.21.15.38:0/2247478460 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb340fff80 0x7fcb341003d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:51.518 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.514+0000 7fcb3afe4700 1 -- 172.21.15.38:0/2247478460 >> 172.21.15.38:0/2247478460 conn(0x7fcb340fa3b0 msgr2=0x7fcb340fc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:51.518 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.514+0000 7fcb3afe4700 1 -- 172.21.15.38:0/2247478460 shutdown_connections 2023-03-09T14:45:51.518 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.514+0000 7fcb3afe4700 1 -- 172.21.15.38:0/2247478460 wait complete. 2023-03-09T14:45:51.518 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.515+0000 7fcb3afe4700 1 Processor -- start 2023-03-09T14:45:51.519 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.515+0000 7fcb3afe4700 1 -- start start 2023-03-09T14:45:51.520 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.515+0000 7fcb3afe4700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcb340fed90 0x7fcb340685b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:51.520 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.515+0000 7fcb3afe4700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb340fff80 0x7fcb3419aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:51.521 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.515+0000 7fcb3afe4700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb34100910 0x7fcb3419d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:51.521 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.515+0000 7fcb3afe4700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3419d870 con 0x7fcb34100910 2023-03-09T14:45:51.521 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.515+0000 7fcb3afe4700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3419d9e0 con 0x7fcb340fed90 2023-03-09T14:45:51.521 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.515+0000 7fcb3afe4700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3419dba0 con 0x7fcb340fff80 2023-03-09T14:45:51.521 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.515+0000 7fcb39581700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb34100910 0x7fcb3419d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:51.521 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.515+0000 7fcb39581700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb34100910 0x7fcb3419d0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38134/0 (socket says 172.21.15.38:38134) 2023-03-09T14:45:51.521 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.515+0000 7fcb39581700 1 -- 172.21.15.38:0/1203916422 learned_addr learned my addr 172.21.15.38:0/1203916422 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:51.522 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.515+0000 7fcb33fff700 1 --2- 172.21.15.38:0/1203916422 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb340fff80 0x7fcb3419aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:51.522 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.515+0000 7fcb38d80700 1 --2- 172.21.15.38:0/1203916422 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcb340fed90 0x7fcb340685b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:51.522 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.515+0000 7fcb39581700 1 -- 172.21.15.38:0/1203916422 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb340fff80 msgr2=0x7fcb3419aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:51.522 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.515+0000 7fcb39581700 1 --2- 172.21.15.38:0/1203916422 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb340fff80 0x7fcb3419aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:51.522 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.515+0000 7fcb39581700 1 -- 172.21.15.38:0/1203916422 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcb340fed90 msgr2=0x7fcb340685b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:51.522 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.515+0000 7fcb39581700 1 --2- 172.21.15.38:0/1203916422 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcb340fed90 0x7fcb340685b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:51.522 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.516+0000 7fcb39581700 1 -- 172.21.15.38:0/1203916422 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb2c009d00 con 0x7fcb34100910 2023-03-09T14:45:51.523 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.516+0000 7fcb39581700 1 --2- 172.21.15.38:0/1203916422 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb34100910 0x7fcb3419d0d0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fcb2c00b830 tx=0x7fcb2c004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:51.523 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.516+0000 7fcb31ffb700 1 -- 172.21.15.38:0/1203916422 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb2c00e040 con 0x7fcb34100910 2023-03-09T14:45:51.523 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.516+0000 7fcb31ffb700 1 -- 172.21.15.38:0/1203916422 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcb2c01db50 con 0x7fcb34100910 2023-03-09T14:45:51.523 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.516+0000 7fcb3afe4700 1 -- 172.21.15.38:0/1203916422 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb3410a720 con 0x7fcb34100910 2023-03-09T14:45:51.523 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.516+0000 7fcb31ffb700 1 -- 172.21.15.38:0/1203916422 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb2c01bb20 con 0x7fcb34100910 2023-03-09T14:45:51.523 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.516+0000 7fcb3afe4700 1 -- 172.21.15.38:0/1203916422 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb3410a590 con 0x7fcb34100910 2023-03-09T14:45:51.523 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.517+0000 7fcb3afe4700 1 -- 172.21.15.38:0/1203916422 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb341062d0 con 0x7fcb34100910 2023-03-09T14:45:51.524 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.518+0000 7fcb31ffb700 1 -- 172.21.15.38:0/1203916422 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) v1 ==== 133198+0+0 (secure 0 0 0) 0x7fcb2c01dcc0 con 0x7fcb34100910 2023-03-09T14:45:51.524 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.519+0000 7fcb31ffb700 1 --2- 172.21.15.38:0/1203916422 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fcb2409f440 0x7fcb240a18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:51.524 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.519+0000 7fcb38d80700 1 --2- 172.21.15.38:0/1203916422 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fcb2409f440 0x7fcb240a18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:51.524 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.519+0000 7fcb31ffb700 1 -- 172.21.15.38:0/1203916422 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fcb2c0cbe20 con 0x7fcb34100910 2023-03-09T14:45:51.525 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.520+0000 7fcb38d80700 1 --2- 172.21.15.38:0/1203916422 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fcb2409f440 0x7fcb240a18f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fcb340ffde0 tx=0x7fcb2800a380 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:51.525 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.523+0000 7fcb31ffb700 1 -- 172.21.15.38:0/1203916422 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7fcb2c095f60 con 0x7fcb34100910 2023-03-09T14:45:51.726 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.723+0000 7fcb3afe4700 1 -- 172.21.15.38:0/1203916422 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fcb34061390 con 0x7fcb34100910 2023-03-09T14:45:51.726 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.724+0000 7fcb31ffb700 1 -- 172.21.15.38:0/1203916422 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) v1 ==== 100+0+11 (secure 0 0 0) 0x7fcb2c099580 con 0x7fcb34100910 2023-03-09T14:45:51.727 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 64 2023-03-09T14:45:51.728 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.726+0000 7fcb3afe4700 1 -- 172.21.15.38:0/1203916422 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fcb2409f440 msgr2=0x7fcb240a18f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:51.729 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.726+0000 7fcb3afe4700 1 --2- 172.21.15.38:0/1203916422 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fcb2409f440 0x7fcb240a18f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fcb340ffde0 tx=0x7fcb2800a380 comp rx=0 tx=0).stop 2023-03-09T14:45:51.729 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.726+0000 7fcb3afe4700 1 -- 172.21.15.38:0/1203916422 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb34100910 msgr2=0x7fcb3419d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:51.729 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.726+0000 7fcb3afe4700 1 --2- 172.21.15.38:0/1203916422 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb34100910 0x7fcb3419d0d0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fcb2c00b830 tx=0x7fcb2c004c80 comp rx=0 tx=0).stop 2023-03-09T14:45:51.729 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.726+0000 7fcb3afe4700 1 -- 172.21.15.38:0/1203916422 shutdown_connections 2023-03-09T14:45:51.729 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.726+0000 7fcb3afe4700 1 --2- 172.21.15.38:0/1203916422 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcb34100910 0x7fcb3419d0d0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:51.729 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.726+0000 7fcb3afe4700 1 --2- 172.21.15.38:0/1203916422 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fcb2409f440 0x7fcb240a18f0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:51.730 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.726+0000 7fcb3afe4700 1 --2- 172.21.15.38:0/1203916422 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcb340fed90 0x7fcb340685b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:51.730 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.726+0000 7fcb3afe4700 1 --2- 172.21.15.38:0/1203916422 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcb340fff80 0x7fcb3419aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:51.730 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.726+0000 7fcb3afe4700 1 -- 172.21.15.38:0/1203916422 >> 172.21.15.38:0/1203916422 conn(0x7fcb340fa3b0 msgr2=0x7fcb34103410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:51.730 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.726+0000 7fcb3afe4700 1 -- 172.21.15.38:0/1203916422 shutdown_connections 2023-03-09T14:45:51.730 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.726+0000 7fcb3afe4700 1 -- 172.21.15.38:0/1203916422 wait complete. 2023-03-09T14:45:51.742 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-03-09T14:45:51.872 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.869+0000 7fe5ac20a700 1 -- 172.21.15.38:0/4267881173 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5a4105bb0 msgr2=0x7fe5a4107fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:51.873 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.869+0000 7fe5ac20a700 1 --2- 172.21.15.38:0/4267881173 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5a4105bb0 0x7fe5a4107fb0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fe59c00b0a0 tx=0x7fe59c00b3b0 comp rx=0 tx=0).stop 2023-03-09T14:45:51.873 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.869+0000 7fe5ac20a700 1 -- 172.21.15.38:0/4267881173 shutdown_connections 2023-03-09T14:45:51.873 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.869+0000 7fe5ac20a700 1 --2- 172.21.15.38:0/4267881173 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5a4105bb0 0x7fe5a4107fb0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:51.873 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.869+0000 7fe5ac20a700 1 --2- 172.21.15.38:0/4267881173 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe5a4100970 0x7fe5a4102d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:51.873 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.869+0000 7fe5ac20a700 1 --2- 172.21.15.38:0/4267881173 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe5a4103290 0x7fe5a4105670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:51.873 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.869+0000 7fe5ac20a700 1 -- 172.21.15.38:0/4267881173 >> 172.21.15.38:0/4267881173 conn(0x7fe5a40fa3b0 msgr2=0x7fe5a40fc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:51.874 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.869+0000 7fe5ac20a700 1 -- 172.21.15.38:0/4267881173 shutdown_connections 2023-03-09T14:45:51.874 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.869+0000 7fe5ac20a700 1 -- 172.21.15.38:0/4267881173 wait complete. 2023-03-09T14:45:51.874 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.870+0000 7fe5ac20a700 1 Processor -- start 2023-03-09T14:45:51.874 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.870+0000 7fe5ac20a700 1 -- start start 2023-03-09T14:45:51.874 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.870+0000 7fe5ac20a700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe5a4100970 0x7fe5a4197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:51.874 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.870+0000 7fe5ac20a700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe5a4103290 0x7fe5a4198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:51.875 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.870+0000 7fe5ac20a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5a4105bb0 0x7fe5a419f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:51.875 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.870+0000 7fe5ac20a700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a419fa70 con 0x7fe5a4105bb0 2023-03-09T14:45:51.875 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.870+0000 7fe5ac20a700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a419fbb0 con 0x7fe5a4100970 2023-03-09T14:45:51.875 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.870+0000 7fe5ac20a700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a419fd40 con 0x7fe5a4103290 2023-03-09T14:45:51.875 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.870+0000 7fe5aa7a7700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5a4105bb0 0x7fe5a419f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:51.875 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.870+0000 7fe5aa7a7700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5a4105bb0 0x7fe5a419f4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38146/0 (socket says 172.21.15.38:38146) 2023-03-09T14:45:51.876 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.870+0000 7fe5aa7a7700 1 -- 172.21.15.38:0/1128597342 learned_addr learned my addr 172.21.15.38:0/1128597342 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:51.876 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.870+0000 7fe5a9fa6700 1 --2- 172.21.15.38:0/1128597342 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe5a4100970 0x7fe5a4197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:51.876 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.870+0000 7fe5a97a5700 1 --2- 172.21.15.38:0/1128597342 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe5a4103290 0x7fe5a4198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:51.876 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.870+0000 7fe5aa7a7700 1 -- 172.21.15.38:0/1128597342 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe5a4103290 msgr2=0x7fe5a4198430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:51.876 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.870+0000 7fe5aa7a7700 1 --2- 172.21.15.38:0/1128597342 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe5a4103290 0x7fe5a4198430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:51.876 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.870+0000 7fe5aa7a7700 1 -- 172.21.15.38:0/1128597342 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe5a4100970 msgr2=0x7fe5a4197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:51.876 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.870+0000 7fe5aa7a7700 1 --2- 172.21.15.38:0/1128597342 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe5a4100970 0x7fe5a4197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:51.876 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.870+0000 7fe5aa7a7700 1 -- 172.21.15.38:0/1128597342 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe59c009d00 con 0x7fe5a4105bb0 2023-03-09T14:45:51.878 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.871+0000 7fe5aa7a7700 1 --2- 172.21.15.38:0/1128597342 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5a4105bb0 0x7fe5a419f4a0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fe59c00b780 tx=0x7fe59c00bd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:51.878 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.871+0000 7fe59affd700 1 -- 172.21.15.38:0/1128597342 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe59c00e040 con 0x7fe5a4105bb0 2023-03-09T14:45:51.878 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.871+0000 7fe59affd700 1 -- 172.21.15.38:0/1128597342 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe59c0094a0 con 0x7fe5a4105bb0 2023-03-09T14:45:51.878 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.871+0000 7fe5ac20a700 1 -- 172.21.15.38:0/1128597342 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5a410b7d0 con 0x7fe5a4105bb0 2023-03-09T14:45:51.878 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.871+0000 7fe59affd700 1 -- 172.21.15.38:0/1128597342 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe59c005390 con 0x7fe5a4105bb0 2023-03-09T14:45:51.878 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.871+0000 7fe5ac20a700 1 -- 172.21.15.38:0/1128597342 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5a410b640 con 0x7fe5a4105bb0 2023-03-09T14:45:51.879 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.872+0000 7fe5ac20a700 1 -- 172.21.15.38:0/1128597342 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5a4107720 con 0x7fe5a4105bb0 2023-03-09T14:45:51.879 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.873+0000 7fe59affd700 1 -- 172.21.15.38:0/1128597342 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) v1 ==== 133198+0+0 (secure 0 0 0) 0x7fe59c01d470 con 0x7fe5a4105bb0 2023-03-09T14:45:51.879 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.873+0000 7fe59affd700 1 --2- 172.21.15.38:0/1128597342 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fe59009f3c0 0x7fe5900a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:51.879 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.874+0000 7fe5a9fa6700 1 --2- 172.21.15.38:0/1128597342 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fe59009f3c0 0x7fe5900a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:51.879 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.874+0000 7fe59affd700 1 -- 172.21.15.38:0/1128597342 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fe59c0cb840 con 0x7fe5a4105bb0 2023-03-09T14:45:51.879 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.874+0000 7fe5a9fa6700 1 --2- 172.21.15.38:0/1128597342 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fe59009f3c0 0x7fe5900a1870 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fe594009c60 tx=0x7fe594009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:51.880 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:51.878+0000 7fe59affd700 1 -- 172.21.15.38:0/1128597342 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7fe59c095980 con 0x7fe5a4105bb0 2023-03-09T14:45:52.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:51 smithi038 ceph-mon[92422]: pgmap v307: 129 pgs: 129 active+clean; 636 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 7 op/s 2023-03-09T14:45:52.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:51 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/3323647816' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-03-09T14:45:52.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:51 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/917301423' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-03-09T14:45:52.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:51 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/1203916422' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-03-09T14:45:52.081 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.079+0000 7fe5ac20a700 1 -- 172.21.15.38:0/1128597342 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fe5a4061390 con 0x7fe5a4105bb0 2023-03-09T14:45:52.081 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.079+0000 7fe59affd700 1 -- 172.21.15.38:0/1128597342 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) v1 ==== 103+0+11 (secure 0 0 0) 0x7fe59c021020 con 0x7fe5a4105bb0 2023-03-09T14:45:52.082 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 32 2023-03-09T14:45:52.083 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.081+0000 7fe5ac20a700 1 -- 172.21.15.38:0/1128597342 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fe59009f3c0 msgr2=0x7fe5900a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:52.084 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.081+0000 7fe5ac20a700 1 --2- 172.21.15.38:0/1128597342 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fe59009f3c0 0x7fe5900a1870 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fe594009c60 tx=0x7fe594009260 comp rx=0 tx=0).stop 2023-03-09T14:45:52.084 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.081+0000 7fe5ac20a700 1 -- 172.21.15.38:0/1128597342 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5a4105bb0 msgr2=0x7fe5a419f4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:52.084 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.081+0000 7fe5ac20a700 1 --2- 172.21.15.38:0/1128597342 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5a4105bb0 0x7fe5a419f4a0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fe59c00b780 tx=0x7fe59c00bd60 comp rx=0 tx=0).stop 2023-03-09T14:45:52.084 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.081+0000 7fe5ac20a700 1 -- 172.21.15.38:0/1128597342 shutdown_connections 2023-03-09T14:45:52.084 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.081+0000 7fe5ac20a700 1 --2- 172.21.15.38:0/1128597342 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5a4105bb0 0x7fe5a419f4a0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:52.084 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.081+0000 7fe5ac20a700 1 --2- 172.21.15.38:0/1128597342 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fe59009f3c0 0x7fe5900a1870 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:52.084 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.081+0000 7fe5ac20a700 1 --2- 172.21.15.38:0/1128597342 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe5a4100970 0x7fe5a4197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:52.085 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.081+0000 7fe5ac20a700 1 --2- 172.21.15.38:0/1128597342 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe5a4103290 0x7fe5a4198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:52.085 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.081+0000 7fe5ac20a700 1 -- 172.21.15.38:0/1128597342 >> 172.21.15.38:0/1128597342 conn(0x7fe5a40fa3b0 msgr2=0x7fe5a40feff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:52.085 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.081+0000 7fe5ac20a700 1 -- 172.21.15.38:0/1128597342 shutdown_connections 2023-03-09T14:45:52.085 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.081+0000 7fe5ac20a700 1 -- 172.21.15.38:0/1128597342 wait complete. 2023-03-09T14:45:52.096 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-09T14:45:52.225 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.220+0000 7f48ebc15700 1 -- 172.21.15.38:0/606517054 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48e4103290 msgr2=0x7f48e4105670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:52.226 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.220+0000 7f48ebc15700 1 --2- 172.21.15.38:0/606517054 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48e4103290 0x7f48e4105670 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f48d4009990 tx=0x7f48d4009ca0 comp rx=0 tx=0).stop 2023-03-09T14:45:52.226 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.220+0000 7f48ebc15700 1 -- 172.21.15.38:0/606517054 shutdown_connections 2023-03-09T14:45:52.226 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.220+0000 7f48ebc15700 1 --2- 172.21.15.38:0/606517054 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f48e4105bb0 0x7f48e4107fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:52.226 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.220+0000 7f48ebc15700 1 --2- 172.21.15.38:0/606517054 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f48e4100970 0x7f48e4102d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:52.227 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.220+0000 7f48ebc15700 1 --2- 172.21.15.38:0/606517054 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48e4103290 0x7f48e4105670 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:52.227 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.220+0000 7f48ebc15700 1 -- 172.21.15.38:0/606517054 >> 172.21.15.38:0/606517054 conn(0x7f48e40fa370 msgr2=0x7f48e40fc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:52.227 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.220+0000 7f48ebc15700 1 -- 172.21.15.38:0/606517054 shutdown_connections 2023-03-09T14:45:52.228 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.220+0000 7f48ebc15700 1 -- 172.21.15.38:0/606517054 wait complete. 2023-03-09T14:45:52.228 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.220+0000 7f48ebc15700 1 Processor -- start 2023-03-09T14:45:52.228 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.220+0000 7f48ebc15700 1 -- start start 2023-03-09T14:45:52.228 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.220+0000 7f48ebc15700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f48e4100970 0x7f48e41002e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:52.229 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.220+0000 7f48ebc15700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48e4103290 0x7f48e40fe920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:52.229 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.220+0000 7f48ebc15700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f48e4105bb0 0x7f48e40fee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:52.229 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.221+0000 7f48ebc15700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48e40ff600 con 0x7f48e4103290 2023-03-09T14:45:52.229 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.221+0000 7f48ebc15700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48e40ff770 con 0x7f48e4105bb0 2023-03-09T14:45:52.230 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.221+0000 7f48ebc15700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48e419fba0 con 0x7f48e4100970 2023-03-09T14:45:52.230 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.221+0000 7f48e91b0700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48e4103290 0x7f48e40fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:52.230 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.221+0000 7f48e91b0700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48e4103290 0x7f48e40fe920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38158/0 (socket says 172.21.15.38:38158) 2023-03-09T14:45:52.230 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.221+0000 7f48e91b0700 1 -- 172.21.15.38:0/3206712195 learned_addr learned my addr 172.21.15.38:0/3206712195 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:52.231 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.221+0000 7f48ea1b2700 1 --2- 172.21.15.38:0/3206712195 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f48e4105bb0 0x7f48e40fee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:52.231 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.221+0000 7f48e99b1700 1 --2- 172.21.15.38:0/3206712195 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f48e4100970 0x7f48e41002e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:52.231 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.221+0000 7f48e91b0700 1 -- 172.21.15.38:0/3206712195 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f48e4100970 msgr2=0x7f48e41002e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:52.232 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.221+0000 7f48e91b0700 1 --2- 172.21.15.38:0/3206712195 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f48e4100970 0x7f48e41002e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:52.232 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.221+0000 7f48e91b0700 1 -- 172.21.15.38:0/3206712195 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f48e4105bb0 msgr2=0x7f48e40fee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:52.232 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.221+0000 7f48e91b0700 1 --2- 172.21.15.38:0/3206712195 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f48e4105bb0 0x7f48e40fee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:52.232 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.221+0000 7f48e91b0700 1 -- 172.21.15.38:0/3206712195 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48d40095f0 con 0x7f48e4103290 2023-03-09T14:45:52.232 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.221+0000 7f48e91b0700 1 --2- 172.21.15.38:0/3206712195 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48e4103290 0x7f48e40fe920 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f48d4000c00 tx=0x7f48d400f710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:52.232 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.221+0000 7f48daffd700 1 -- 172.21.15.38:0/3206712195 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d401d070 con 0x7f48e4103290 2023-03-09T14:45:52.233 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.221+0000 7f48daffd700 1 -- 172.21.15.38:0/3206712195 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f48d400fb50 con 0x7f48e4103290 2023-03-09T14:45:52.233 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.221+0000 7f48ebc15700 1 -- 172.21.15.38:0/3206712195 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48e410b7e0 con 0x7f48e4103290 2023-03-09T14:45:52.233 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.221+0000 7f48daffd700 1 -- 172.21.15.38:0/3206712195 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48d40173f0 con 0x7f48e4103290 2023-03-09T14:45:52.233 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.221+0000 7f48ebc15700 1 -- 172.21.15.38:0/3206712195 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48e410b650 con 0x7f48e4103290 2023-03-09T14:45:52.233 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.222+0000 7f48ebc15700 1 -- 172.21.15.38:0/3206712195 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48e41a0190 con 0x7f48e4103290 2023-03-09T14:45:52.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:51 smithi181 ceph-mon[94189]: pgmap v307: 129 pgs: 129 active+clean; 636 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 7 op/s 2023-03-09T14:45:52.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:51 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/3323647816' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-03-09T14:45:52.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:51 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/917301423' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-03-09T14:45:52.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:51 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/1203916422' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-03-09T14:45:52.235 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:51 smithi122 ceph-mon[93869]: pgmap v307: 129 pgs: 129 active+clean; 636 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 7 op/s 2023-03-09T14:45:52.235 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:51 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/3323647816' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-03-09T14:45:52.235 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:51 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/917301423' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-03-09T14:45:52.235 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:51 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/1203916422' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-03-09T14:45:52.236 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.224+0000 7f48daffd700 1 -- 172.21.15.38:0/3206712195 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) v1 ==== 133198+0+0 (secure 0 0 0) 0x7f48d4029030 con 0x7f48e4103290 2023-03-09T14:45:52.236 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.224+0000 7f48daffd700 1 --2- 172.21.15.38:0/3206712195 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f48d009f440 0x7f48d00a18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:52.236 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.225+0000 7f48e99b1700 1 --2- 172.21.15.38:0/3206712195 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f48d009f440 0x7f48d00a18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:52.236 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.225+0000 7f48daffd700 1 -- 172.21.15.38:0/3206712195 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f48d40cb580 con 0x7f48e4103290 2023-03-09T14:45:52.236 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.225+0000 7f48e99b1700 1 --2- 172.21.15.38:0/3206712195 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f48d009f440 0x7f48d00a18f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f48e000ac50 tx=0x7f48e000a380 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:52.236 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.228+0000 7f48daffd700 1 -- 172.21.15.38:0/3206712195 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f48d40956c0 con 0x7f48e4103290 2023-03-09T14:45:52.478 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.475+0000 7f48ebc15700 1 -- 172.21.15.38:0/3206712195 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f48e41a0740 con 0x7f48e4103290 2023-03-09T14:45:52.478 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.476+0000 7f48daffd700 1 -- 172.21.15.38:0/3206712195 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+10801 (secure 0 0 0) 0x7f48d4027090 con 0x7f48e4103290 2023-03-09T14:45:52.479 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:45:52.479 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/768918602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":768918602},{"type":"v1","addr":"172.21.15.38:6835","nonce":768918602}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/18381540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":18381540},{"type":"v1","addr":"172.21.15.38:6837","nonce":18381540}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14610,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6839/4139020066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":4139020066},{"type":"v1","addr":"172.21.15.38:6839","nonce":4139020066}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14622,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6841/1397115322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":1397115322},{"type":"v1","addr":"172.21.15.38:6841","nonce":1397115322}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6833/415305571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":415305571},{"type":"v1","addr":"172.21.15.181:6833","nonce":415305571}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/659300520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":659300520},{"type":"v1","addr":"172.21.15.122:6833","nonce":659300520}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-03-09T14:44:19.083770+0000","modified":"2023-03-09T14:44:39.526150+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24530,"mds_1":24494,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24448":{"gid":24448,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6835/2576255861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":2576255861},{"type":"v1","addr":"172.21.15.181:6835","nonce":2576255861}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24460":{"gid":24460,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6837/2942560181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":2942560181},{"type":"v1","addr":"172.21.15.181:6837","nonce":2942560181}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24464":{"gid":24464,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.122:6835/3675537941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":3675537941},{"type":"v1","addr":"172.21.15.122:6835","nonce":3675537941}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24478":{"gid":24478,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":9,"addr":"172.21.15.181:6839/3902263660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":3902263660},{"type":"v1","addr":"172.21.15.181:6839","nonce":3902263660}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24494":{"gid":24494,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.122:6837/2210214596","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":2210214596},{"type":"v1","addr":"172.21.15.122:6837","nonce":2210214596}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24530":{"gid":24530,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6839/971050165","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6838","nonce":971050165},{"type":"v1","addr":"172.21.15.122:6839","nonce":971050165}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-09T14:45:52.480 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.478+0000 7f48ebc15700 1 -- 172.21.15.38:0/3206712195 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f48d009f440 msgr2=0x7f48d00a18f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:52.481 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.478+0000 7f48ebc15700 1 --2- 172.21.15.38:0/3206712195 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f48d009f440 0x7f48d00a18f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f48e000ac50 tx=0x7f48e000a380 comp rx=0 tx=0).stop 2023-03-09T14:45:52.481 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.478+0000 7f48ebc15700 1 -- 172.21.15.38:0/3206712195 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48e4103290 msgr2=0x7f48e40fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:52.481 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.478+0000 7f48ebc15700 1 --2- 172.21.15.38:0/3206712195 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48e4103290 0x7f48e40fe920 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f48d4000c00 tx=0x7f48d400f710 comp rx=0 tx=0).stop 2023-03-09T14:45:52.481 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.478+0000 7f48ebc15700 1 -- 172.21.15.38:0/3206712195 shutdown_connections 2023-03-09T14:45:52.481 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.478+0000 7f48ebc15700 1 --2- 172.21.15.38:0/3206712195 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f48d009f440 0x7f48d00a18f0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:52.481 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.478+0000 7f48ebc15700 1 --2- 172.21.15.38:0/3206712195 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f48e4105bb0 0x7f48e40fee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:52.481 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.478+0000 7f48ebc15700 1 --2- 172.21.15.38:0/3206712195 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f48e4100970 0x7f48e41002e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:52.482 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.478+0000 7f48ebc15700 1 --2- 172.21.15.38:0/3206712195 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f48e4103290 0x7f48e40fe920 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:52.482 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.478+0000 7f48ebc15700 1 -- 172.21.15.38:0/3206712195 >> 172.21.15.38:0/3206712195 conn(0x7f48e40fa370 msgr2=0x7f48e4101db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:52.482 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.478+0000 7f48ebc15700 1 -- 172.21.15.38:0/3206712195 shutdown_connections 2023-03-09T14:45:52.482 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.478+0000 7f48ebc15700 1 -- 172.21.15.38:0/3206712195 wait complete. 2023-03-09T14:45:52.482 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 23 2023-03-09T14:45:52.493 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-09T14:45:52.620 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.617+0000 7fe54ecb7700 1 -- 172.21.15.38:0/4089771778 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe548100900 msgr2=0x7fe5481071c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:52.621 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.617+0000 7fe54ecb7700 1 --2- 172.21.15.38:0/4089771778 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe548100900 0x7fe5481071c0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fe54000b0a0 tx=0x7fe54000b3b0 comp rx=0 tx=0).stop 2023-03-09T14:45:52.621 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.617+0000 7fe54ecb7700 1 -- 172.21.15.38:0/4089771778 shutdown_connections 2023-03-09T14:45:52.621 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.617+0000 7fe54ecb7700 1 --2- 172.21.15.38:0/4089771778 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe548100900 0x7fe5481071c0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:52.621 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.617+0000 7fe54ecb7700 1 --2- 172.21.15.38:0/4089771778 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe5480fed80 0x7fe5480ff190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:52.621 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.617+0000 7fe54ecb7700 1 --2- 172.21.15.38:0/4089771778 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe5480fff70 0x7fe5481003c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:52.621 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.617+0000 7fe54ecb7700 1 -- 172.21.15.38:0/4089771778 >> 172.21.15.38:0/4089771778 conn(0x7fe5480fa380 msgr2=0x7fe5480fc7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:52.622 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.617+0000 7fe54ecb7700 1 -- 172.21.15.38:0/4089771778 shutdown_connections 2023-03-09T14:45:52.622 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.617+0000 7fe54ecb7700 1 -- 172.21.15.38:0/4089771778 wait complete. 2023-03-09T14:45:52.622 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.618+0000 7fe54ecb7700 1 Processor -- start 2023-03-09T14:45:52.622 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.618+0000 7fe54ecb7700 1 -- start start 2023-03-09T14:45:52.624 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.618+0000 7fe54ecb7700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5480fed80 0x7fe548194e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:52.624 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.618+0000 7fe54ecb7700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe5480fff70 0x7fe548195340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:52.624 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.618+0000 7fe54ecb7700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe548100900 0x7fe54819c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:52.625 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.618+0000 7fe54ecb7700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe54819c980 con 0x7fe5480fed80 2023-03-09T14:45:52.625 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.618+0000 7fe54ecb7700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe54819cac0 con 0x7fe5480fff70 2023-03-09T14:45:52.625 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.618+0000 7fe54ecb7700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe54819cc50 con 0x7fe548100900 2023-03-09T14:45:52.625 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.619+0000 7fe54d254700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe548100900 0x7fe54819c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:52.626 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.619+0000 7fe54d254700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe548100900 0x7fe54819c3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.38:45872/0 (socket says 172.21.15.38:45872) 2023-03-09T14:45:52.626 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.619+0000 7fe54d254700 1 -- 172.21.15.38:0/1617115187 learned_addr learned my addr 172.21.15.38:0/1617115187 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:52.626 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.619+0000 7fe547fff700 1 --2- 172.21.15.38:0/1617115187 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe5480fff70 0x7fe548195340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:52.626 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.619+0000 7fe54ca53700 1 --2- 172.21.15.38:0/1617115187 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5480fed80 0x7fe548194e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:52.627 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.619+0000 7fe547fff700 1 -- 172.21.15.38:0/1617115187 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe548100900 msgr2=0x7fe54819c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:52.627 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.619+0000 7fe547fff700 1 --2- 172.21.15.38:0/1617115187 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe548100900 0x7fe54819c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:52.627 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.619+0000 7fe547fff700 1 -- 172.21.15.38:0/1617115187 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5480fed80 msgr2=0x7fe548194e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:52.628 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.619+0000 7fe547fff700 1 --2- 172.21.15.38:0/1617115187 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5480fed80 0x7fe548194e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:52.628 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.619+0000 7fe547fff700 1 -- 172.21.15.38:0/1617115187 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe53c009520 con 0x7fe5480fff70 2023-03-09T14:45:52.628 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.619+0000 7fe54d254700 1 --2- 172.21.15.38:0/1617115187 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe548100900 0x7fe54819c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:45:52.628 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.619+0000 7fe54ca53700 1 --2- 172.21.15.38:0/1617115187 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5480fed80 0x7fe548194e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:45:52.628 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.619+0000 7fe547fff700 1 --2- 172.21.15.38:0/1617115187 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe5480fff70 0x7fe548195340 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe53400e9f0 tx=0x7fe53400ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:52.629 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.619+0000 7fe545ffb700 1 -- 172.21.15.38:0/1617115187 <== mon.2 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe53400eee0 con 0x7fe5480fff70 2023-03-09T14:45:52.629 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.620+0000 7fe54ecb7700 1 -- 172.21.15.38:0/1617115187 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe540009d00 con 0x7fe5480fff70 2023-03-09T14:45:52.629 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.620+0000 7fe545ffb700 1 -- 172.21.15.38:0/1617115187 <== mon.2 v2:172.21.15.122:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe5340044e0 con 0x7fe5480fff70 2023-03-09T14:45:52.629 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.620+0000 7fe545ffb700 1 -- 172.21.15.38:0/1617115187 <== mon.2 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe534005020 con 0x7fe5480fff70 2023-03-09T14:45:52.629 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.620+0000 7fe54ecb7700 1 -- 172.21.15.38:0/1617115187 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe54810a760 con 0x7fe5480fff70 2023-03-09T14:45:52.629 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.621+0000 7fe54ecb7700 1 -- 172.21.15.38:0/1617115187 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe54810a8f0 con 0x7fe5480fff70 2023-03-09T14:45:52.630 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.622+0000 7fe545ffb700 1 -- 172.21.15.38:0/1617115187 <== mon.2 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 18) v1 ==== 133198+0+0 (secure 0 0 0) 0x7fe53400c040 con 0x7fe5480fff70 2023-03-09T14:45:52.630 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.622+0000 7fe545ffb700 1 --2- 172.21.15.38:0/1617115187 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fe53809f3c0 0x7fe5380a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:52.630 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.622+0000 7fe54ca53700 1 --2- 172.21.15.38:0/1617115187 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fe53809f3c0 0x7fe5380a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:52.631 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.622+0000 7fe545ffb700 1 -- 172.21.15.38:0/1617115187 <== mon.2 v2:172.21.15.122:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fe534014070 con 0x7fe5480fff70 2023-03-09T14:45:52.631 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.623+0000 7fe54ca53700 1 --2- 172.21.15.38:0/1617115187 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fe53809f3c0 0x7fe5380a1870 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fe5480ffdd0 tx=0x7fe53c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:52.631 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.626+0000 7fe545ffb700 1 -- 172.21.15.38:0/1617115187 <== mon.2 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7fe534094120 con 0x7fe5480fff70 2023-03-09T14:45:52.829 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.827+0000 7fe54ecb7700 1 -- 172.21.15.38:0/1617115187 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe548106060 con 0x7fe5480fff70 2023-03-09T14:45:52.831 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.829+0000 7fe545ffb700 1 -- 172.21.15.38:0/1617115187 <== mon.2 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24445 (secure 0 0 0) 0x7fe53401c020 con 0x7fe5480fff70 2023-03-09T14:45:52.832 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:45:52.833 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":89,"fsid":"c6733fe2-be87-11ed-9af2-001a4aab830c","created":"2023-03-09T14:37:14.947981+0000","modified":"2023-03-09T14:45:15.848936+0000","last_up_change":"2023-03-09T14:43:37.198407+0000","last_in_change":"2023-03-09T14:43:19.550566+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-09T14:40:23.196665+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-03-09T14:44:17.484374+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-03-09T14:44:18.457833+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-03-09T14:44:21.895589+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"51477039-15ee-4707-a34d-4ecfa18bebde","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6803","nonce":1508072290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6805","nonce":1508072290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6809","nonce":1508072290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6807","nonce":1508072290}]},"public_addr":"172.21.15.38:6803/1508072290","cluster_addr":"172.21.15.38:6805/1508072290","heartbeat_back_addr":"172.21.15.38:6809/1508072290","heartbeat_front_addr":"172.21.15.38:6807/1508072290","state":["exists","up"]},{"osd":1,"uuid":"71f8a718-b0a9-4f39-bc54-267aa42a2578","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6801","nonce":3929268773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6803","nonce":3929268773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6807","nonce":3929268773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6805","nonce":3929268773}]},"public_addr":"172.21.15.122:6801/3929268773","cluster_addr":"172.21.15.122:6803/3929268773","heartbeat_back_addr":"172.21.15.122:6807/3929268773","heartbeat_front_addr":"172.21.15.122:6805/3929268773","state":["exists","up"]},{"osd":2,"uuid":"961a2cbe-140e-437f-b131-1fcac573b19c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6801","nonce":1310630803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6803","nonce":1310630803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6807","nonce":1310630803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6805","nonce":1310630803}]},"public_addr":"172.21.15.181:6801/1310630803","cluster_addr":"172.21.15.181:6803/1310630803","heartbeat_back_addr":"172.21.15.181:6807/1310630803","heartbeat_front_addr":"172.21.15.181:6805/1310630803","state":["exists","up"]},{"osd":3,"uuid":"969df8fa-2d19-42c7-9f2a-5c5fac15d4a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6811","nonce":2719814100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6813","nonce":2719814100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6817","nonce":2719814100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6815","nonce":2719814100}]},"public_addr":"172.21.15.38:6811/2719814100","cluster_addr":"172.21.15.38:6813/2719814100","heartbeat_back_addr":"172.21.15.38:6817/2719814100","heartbeat_front_addr":"172.21.15.38:6815/2719814100","state":["exists","up"]},{"osd":4,"uuid":"44a6dce3-e06e-4cbd-9404-62a86d01ab6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6809","nonce":2480764994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6811","nonce":2480764994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6815","nonce":2480764994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6813","nonce":2480764994}]},"public_addr":"172.21.15.122:6809/2480764994","cluster_addr":"172.21.15.122:6811/2480764994","heartbeat_back_addr":"172.21.15.122:6815/2480764994","heartbeat_front_addr":"172.21.15.122:6813/2480764994","state":["exists","up"]},{"osd":5,"uuid":"49749574-c00c-4e3a-80e6-e3686df58d5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6809","nonce":3141346246}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6811","nonce":3141346246}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6815","nonce":3141346246}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6813","nonce":3141346246}]},"public_addr":"172.21.15.181:6809/3141346246","cluster_addr":"172.21.15.181:6811/3141346246","heartbeat_back_addr":"172.21.15.181:6815/3141346246","heartbeat_front_addr":"172.21.15.181:6813/3141346246","state":["exists","up"]},{"osd":6,"uuid":"61c0437e-3f51-45a9-8a61-e98eea0093f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6819","nonce":2122788957}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6821","nonce":2122788957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6825","nonce":2122788957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6823","nonce":2122788957}]},"public_addr":"172.21.15.38:6819/2122788957","cluster_addr":"172.21.15.38:6821/2122788957","heartbeat_back_addr":"172.21.15.38:6825/2122788957","heartbeat_front_addr":"172.21.15.38:6823/2122788957","state":["exists","up"]},{"osd":7,"uuid":"231fecdd-939f-4c7b-83a1-c4a1cffd7139","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6817","nonce":656874139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6819","nonce":656874139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6823","nonce":656874139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6821","nonce":656874139}]},"public_addr":"172.21.15.122:6817/656874139","cluster_addr":"172.21.15.122:6819/656874139","heartbeat_back_addr":"172.21.15.122:6823/656874139","heartbeat_front_addr":"172.21.15.122:6821/656874139","state":["exists","up"]},{"osd":8,"uuid":"297e6d6d-85fd-43ce-9028-5e9dbe55c36e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6817","nonce":755000703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6819","nonce":755000703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6823","nonce":755000703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6821","nonce":755000703}]},"public_addr":"172.21.15.181:6817/755000703","cluster_addr":"172.21.15.181:6819/755000703","heartbeat_back_addr":"172.21.15.181:6823/755000703","heartbeat_front_addr":"172.21.15.181:6821/755000703","state":["exists","up"]},{"osd":9,"uuid":"94cc0355-5a1a-4cfc-bd0c-562b6b41127e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6827","nonce":2624555969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6829","nonce":2624555969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6833","nonce":2624555969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6831","nonce":2624555969}]},"public_addr":"172.21.15.38:6827/2624555969","cluster_addr":"172.21.15.38:6829/2624555969","heartbeat_back_addr":"172.21.15.38:6833/2624555969","heartbeat_front_addr":"172.21.15.38:6831/2624555969","state":["exists","up"]},{"osd":10,"uuid":"f4f2e042-8d92-46bf-93b8-8b0f04795c2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6825","nonce":1427785280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6827","nonce":1427785280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6831","nonce":1427785280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6829","nonce":1427785280}]},"public_addr":"172.21.15.122:6825/1427785280","cluster_addr":"172.21.15.122:6827/1427785280","heartbeat_back_addr":"172.21.15.122:6831/1427785280","heartbeat_front_addr":"172.21.15.122:6829/1427785280","state":["exists","up"]},{"osd":11,"uuid":"07444936-eae5-4097-a799-60294cafe80b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6825","nonce":3183488640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6827","nonce":3183488640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6831","nonce":3183488640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6829","nonce":3183488640}]},"public_addr":"172.21.15.181:6825/3183488640","cluster_addr":"172.21.15.181:6827/3183488640","heartbeat_back_addr":"172.21.15.181:6831/3183488640","heartbeat_front_addr":"172.21.15.181:6829/3183488640","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:39:37.657556+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:39:58.960074+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:40:20.326952+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:40:42.244387+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:41:03.946973+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:41:25.097985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:41:46.877916+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:42:08.413827+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:42:29.822009+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:42:51.776684+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:43:13.937249+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:43:35.569199+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.2","mappings":[{"from":7,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.d","mappings":[{"from":5,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.17","mappings":[{"from":5,"to":6}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/4272288735":"2023-03-10T14:38:14.612767+0000","172.21.15.38:6801/3381625669":"2023-03-10T14:38:14.612767+0000","172.21.15.38:6800/3381625669":"2023-03-10T14:38:14.612767+0000","172.21.15.38:6800/3060474342":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/4253918615":"2023-03-10T14:38:14.612767+0000","172.21.15.38:0/3496083181":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/245602368":"2023-03-10T14:37:47.517775+0000","172.21.15.38:6801/3060474342":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/3622589182":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/1430302251":"2023-03-10T14:38:14.612767+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-09T14:45:52.834 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.831+0000 7fe54ecb7700 1 -- 172.21.15.38:0/1617115187 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fe53809f3c0 msgr2=0x7fe5380a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:52.834 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.831+0000 7fe54ecb7700 1 --2- 172.21.15.38:0/1617115187 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fe53809f3c0 0x7fe5380a1870 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fe5480ffdd0 tx=0x7fe53c009260 comp rx=0 tx=0).stop 2023-03-09T14:45:52.834 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.831+0000 7fe54ecb7700 1 -- 172.21.15.38:0/1617115187 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe5480fff70 msgr2=0x7fe548195340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:52.834 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.831+0000 7fe54ecb7700 1 --2- 172.21.15.38:0/1617115187 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe5480fff70 0x7fe548195340 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe53400e9f0 tx=0x7fe53400ed00 comp rx=0 tx=0).stop 2023-03-09T14:45:52.835 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.831+0000 7fe54ecb7700 1 -- 172.21.15.38:0/1617115187 shutdown_connections 2023-03-09T14:45:52.835 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.831+0000 7fe54ecb7700 1 --2- 172.21.15.38:0/1617115187 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe5480fed80 0x7fe548194e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:52.835 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.831+0000 7fe54ecb7700 1 --2- 172.21.15.38:0/1617115187 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fe53809f3c0 0x7fe5380a1870 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:52.835 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.831+0000 7fe54ecb7700 1 --2- 172.21.15.38:0/1617115187 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe548100900 0x7fe54819c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:52.835 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.831+0000 7fe54ecb7700 1 --2- 172.21.15.38:0/1617115187 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe5480fff70 0x7fe548195340 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:52.836 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.831+0000 7fe54ecb7700 1 -- 172.21.15.38:0/1617115187 >> 172.21.15.38:0/1617115187 conn(0x7fe5480fa380 msgr2=0x7fe5481031a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:52.836 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.831+0000 7fe54ecb7700 1 -- 172.21.15.38:0/1617115187 shutdown_connections 2023-03-09T14:45:52.836 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.831+0000 7fe54ecb7700 1 -- 172.21.15.38:0/1617115187 wait complete. 2023-03-09T14:45:52.847 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2023-03-09T14:44:19.083770+0000', 'modified': '2023-03-09T14:44:39.526150+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24530, 'mds_1': 24494, 'mds_2': 24478}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24448': {'gid': 24448, 'name': 'f', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.181:6835/2576255861', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6834', 'nonce': 2576255861}, {'type': 'v1', 'addr': '172.21.15.181:6835', 'nonce': 2576255861}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24460': {'gid': 24460, 'name': 'c', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.181:6837/2942560181', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6836', 'nonce': 2942560181}, {'type': 'v1', 'addr': '172.21.15.181:6837', 'nonce': 2942560181}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24464': {'gid': 24464, 'name': 'e', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.122:6835/3675537941', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6834', 'nonce': 3675537941}, {'type': 'v1', 'addr': '172.21.15.122:6835', 'nonce': 3675537941}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24478': {'gid': 24478, 'name': 'i', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.181:6839/3902263660', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.181:6838', 'nonce': 3902263660}, {'type': 'v1', 'addr': '172.21.15.181:6839', 'nonce': 3902263660}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24494': {'gid': 24494, 'name': 'h', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.122:6837/2210214596', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6836', 'nonce': 2210214596}, {'type': 'v1', 'addr': '172.21.15.122:6837', 'nonce': 2210214596}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24530': {'gid': 24530, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.122:6839/971050165', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.122:6838', 'nonce': 971050165}, {'type': 'v1', 'addr': '172.21.15.122:6839', 'nonce': 971050165}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-03-09T14:45:52.847 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2023-03-09T14:45:52.848 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2023-03-09T14:45:52.976 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.974+0000 7f5fdcf76700 1 -- 172.21.15.38:0/2119553537 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5fd80fedb0 msgr2=0x7f5fd80ff1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:52.977 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.974+0000 7f5fdcf76700 1 --2- 172.21.15.38:0/2119553537 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5fd80fedb0 0x7f5fd80ff1c0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f5fc8009990 tx=0x7f5fc8009ca0 comp rx=0 tx=0).stop 2023-03-09T14:45:52.977 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.974+0000 7f5fdcf76700 1 -- 172.21.15.38:0/2119553537 shutdown_connections 2023-03-09T14:45:52.977 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.974+0000 7f5fdcf76700 1 --2- 172.21.15.38:0/2119553537 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5fd80fedb0 0x7f5fd80ff1c0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:52.977 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.974+0000 7f5fdcf76700 1 --2- 172.21.15.38:0/2119553537 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5fd8100930 0x7f5fd81071f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:52.977 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.974+0000 7f5fdcf76700 1 --2- 172.21.15.38:0/2119553537 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5fd80fffa0 0x7f5fd81003f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:52.978 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.974+0000 7f5fdcf76700 1 -- 172.21.15.38:0/2119553537 >> 172.21.15.38:0/2119553537 conn(0x7f5fd80fa3b0 msgr2=0x7f5fd80fc800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:52.978 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.974+0000 7f5fdcf76700 1 -- 172.21.15.38:0/2119553537 shutdown_connections 2023-03-09T14:45:52.978 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.974+0000 7f5fdcf76700 1 -- 172.21.15.38:0/2119553537 wait complete. 2023-03-09T14:45:52.978 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.974+0000 7f5fdcf76700 1 Processor -- start 2023-03-09T14:45:52.978 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.974+0000 7f5fdcf76700 1 -- start start 2023-03-09T14:45:52.979 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.975+0000 7f5fdcf76700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5fd80fedb0 0x7f5fd8069600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:52.980 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.975+0000 7f5fdcf76700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5fd80fffa0 0x7f5fd819aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:52.980 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.975+0000 7f5fdcf76700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5fd8100930 0x7f5fd819d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:52.980 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.975+0000 7f5fdcf76700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd819d870 con 0x7f5fd80fffa0 2023-03-09T14:45:52.980 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.975+0000 7f5fdcf76700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd819d9e0 con 0x7f5fd8100930 2023-03-09T14:45:52.980 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.975+0000 7f5fdcf76700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd819dba0 con 0x7f5fd80fedb0 2023-03-09T14:45:52.980 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.975+0000 7f5fd5d9b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5fd80fffa0 0x7f5fd819aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:52.981 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.975+0000 7f5fd5d9b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5fd80fffa0 0x7f5fd819aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38186/0 (socket says 172.21.15.38:38186) 2023-03-09T14:45:52.981 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.975+0000 7f5fd5d9b700 1 -- 172.21.15.38:0/3604452645 learned_addr learned my addr 172.21.15.38:0/3604452645 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:52.981 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.975+0000 7f5fd6d9d700 1 --2- 172.21.15.38:0/3604452645 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5fd8100930 0x7f5fd819d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:52.981 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.975+0000 7f5fd659c700 1 --2- 172.21.15.38:0/3604452645 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5fd80fedb0 0x7f5fd8069600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:52.981 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.975+0000 7f5fd5d9b700 1 -- 172.21.15.38:0/3604452645 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5fd80fedb0 msgr2=0x7f5fd8069600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:52.981 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.975+0000 7f5fd5d9b700 1 --2- 172.21.15.38:0/3604452645 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5fd80fedb0 0x7f5fd8069600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:52.981 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.975+0000 7f5fd5d9b700 1 -- 172.21.15.38:0/3604452645 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5fd8100930 msgr2=0x7f5fd819d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:52.982 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.975+0000 7f5fd5d9b700 1 --2- 172.21.15.38:0/3604452645 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5fd8100930 0x7f5fd819d0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:52.982 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.975+0000 7f5fd5d9b700 1 -- 172.21.15.38:0/3604452645 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fc80095f0 con 0x7f5fd80fffa0 2023-03-09T14:45:52.982 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.975+0000 7f5fd5d9b700 1 --2- 172.21.15.38:0/3604452645 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5fd80fffa0 0x7f5fd819aa50 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f5fcc00c9b0 tx=0x7f5fcc00ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:52.982 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.975+0000 7f5fc77fe700 1 -- 172.21.15.38:0/3604452645 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fcc007b00 con 0x7f5fd80fffa0 2023-03-09T14:45:52.982 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.975+0000 7f5fc77fe700 1 -- 172.21.15.38:0/3604452645 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5fcc007c60 con 0x7f5fd80fffa0 2023-03-09T14:45:52.982 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.975+0000 7f5fc77fe700 1 -- 172.21.15.38:0/3604452645 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fcc018660 con 0x7f5fd80fffa0 2023-03-09T14:45:52.982 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.975+0000 7f5fdcf76700 1 -- 172.21.15.38:0/3604452645 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fd810a760 con 0x7f5fd80fffa0 2023-03-09T14:45:52.983 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.976+0000 7f5fdcf76700 1 -- 172.21.15.38:0/3604452645 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fd810a5d0 con 0x7f5fd80fffa0 2023-03-09T14:45:52.983 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.978+0000 7f5fc77fe700 1 -- 172.21.15.38:0/3604452645 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) v1 ==== 133198+0+0 (secure 0 0 0) 0x7f5fcc00f450 con 0x7f5fd80fffa0 2023-03-09T14:45:52.987 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.978+0000 7f5fc77fe700 1 --2- 172.21.15.38:0/3604452645 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f5fc009f3c0 0x7f5fc00a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:52.987 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.978+0000 7f5fd659c700 1 --2- 172.21.15.38:0/3604452645 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f5fc009f3c0 0x7f5fc00a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:52.988 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.978+0000 7f5fc77fe700 1 -- 172.21.15.38:0/3604452645 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f5fcc0ca210 con 0x7f5fd80fffa0 2023-03-09T14:45:52.988 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.979+0000 7f5fd659c700 1 --2- 172.21.15.38:0/3604452645 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f5fc009f3c0 0x7f5fc00a1870 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f5fc800b5c0 tx=0x7f5fc8005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:52.988 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.979+0000 7f5fe90ffb80 1 Processor -- start 2023-03-09T14:45:52.988 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.979+0000 7f5fe90ffb80 1 -- start start 2023-03-09T14:45:52.989 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.979+0000 7f5fe90ffb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559e25787e00 0x559e25788210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:52.989 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.979+0000 7f5fe90ffb80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559e25788750 0x559e25789070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:52.989 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.979+0000 7f5fe90ffb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e25789600 0x559e25789a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:52.989 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.979+0000 7f5fe90ffb80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e2566dd50 con 0x559e25788750 2023-03-09T14:45:52.989 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.979+0000 7f5fe90ffb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e256dc730 con 0x559e25789600 2023-03-09T14:45:52.989 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.979+0000 7f5fe90ffb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e256d5c60 con 0x559e25787e00 2023-03-09T14:45:52.990 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.979+0000 7f5fd5d9b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559e25788750 0x559e25789070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:52.990 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.979+0000 7f5fd5d9b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559e25788750 0x559e25789070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38194/0 (socket says 172.21.15.38:38194) 2023-03-09T14:45:52.990 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.979+0000 7f5fd5d9b700 1 -- 172.21.15.38:0/2188162228 learned_addr learned my addr 172.21.15.38:0/2188162228 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:52.990 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.980+0000 7f5fd6d9d700 1 --2- 172.21.15.38:0/2188162228 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e25789600 0x559e25789a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:52.990 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.980+0000 7f5fd659c700 1 --2- 172.21.15.38:0/2188162228 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559e25787e00 0x559e25788210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:52.990 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.980+0000 7f5fd5d9b700 1 -- 172.21.15.38:0/2188162228 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559e25787e00 msgr2=0x559e25788210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:45:52.990 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.980+0000 7f5fd5d9b700 1 --2- 172.21.15.38:0/2188162228 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559e25787e00 0x559e25788210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:52.991 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.980+0000 7f5fd5d9b700 1 -- 172.21.15.38:0/2188162228 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e25789600 msgr2=0x559e25789a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:45:52.991 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.980+0000 7f5fd5d9b700 1 --2- 172.21.15.38:0/2188162228 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e25789600 0x559e25789a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:52.991 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.980+0000 7f5fd5d9b700 1 -- 172.21.15.38:0/2188162228 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559e2569c140 con 0x559e25788750 2023-03-09T14:45:52.991 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.980+0000 7f5fd6d9d700 1 --2- 172.21.15.38:0/2188162228 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e25789600 0x559e25789a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:45:52.991 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.980+0000 7f5fd5d9b700 1 --2- 172.21.15.38:0/2188162228 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559e25788750 0x559e25789070 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f5fcc00cfb0 tx=0x7f5fcc097960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83543f1112a2402b server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:52.991 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.980+0000 7f5fc4ff9700 1 -- 172.21.15.38:0/2188162228 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fcc0a01d0 con 0x559e25788750 2023-03-09T14:45:52.991 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.980+0000 7f5fc4ff9700 1 -- 172.21.15.38:0/2188162228 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5fcc0960c0 con 0x559e25788750 2023-03-09T14:45:52.991 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.980+0000 7f5fc4ff9700 1 -- 172.21.15.38:0/2188162228 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fcc098440 con 0x559e25788750 2023-03-09T14:45:52.992 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.981+0000 7f5fe90ffb80 1 -- 172.21.15.38:0/2188162228 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559e25788750 msgr2=0x559e25789070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:52.992 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.981+0000 7f5fe90ffb80 1 --2- 172.21.15.38:0/2188162228 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559e25788750 0x559e25789070 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f5fcc00cfb0 tx=0x7f5fcc097960 comp rx=0 tx=0).stop 2023-03-09T14:45:52.992 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.981+0000 7f5fe90ffb80 1 -- 172.21.15.38:0/2188162228 shutdown_connections 2023-03-09T14:45:52.992 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.981+0000 7f5fe90ffb80 1 --2- 172.21.15.38:0/2188162228 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e25789600 0x559e25789a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:52.992 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.981+0000 7f5fe90ffb80 1 --2- 172.21.15.38:0/2188162228 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559e25787e00 0x559e25788210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:52.992 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.981+0000 7f5fe90ffb80 1 --2- 172.21.15.38:0/2188162228 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559e25788750 0x559e25789070 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:52.992 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.981+0000 7f5fe90ffb80 1 -- 172.21.15.38:0/2188162228 >> 172.21.15.38:0/2188162228 conn(0x559e2577e500 msgr2=0x559e2577e8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:52.992 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.981+0000 7f5fe90ffb80 1 -- 172.21.15.38:0/2188162228 shutdown_connections 2023-03-09T14:45:52.993 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.981+0000 7f5fe90ffb80 1 -- 172.21.15.38:0/2188162228 wait complete. 2023-03-09T14:45:52.993 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.981+0000 7f5fe90ffb80 1 -- start start 2023-03-09T14:45:52.993 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.981+0000 7f5fe90ffb80 1 Processor -- start 2023-03-09T14:45:52.993 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.982+0000 7f5fe90ffb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559e25787e00 0x559e25791a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:52.993 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.982+0000 7f5fe90ffb80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559e25788750 0x559e25791f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:52.993 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.982+0000 7f5fe90ffb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e25789600 0x559e25798fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:52.993 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.982+0000 7f5fe90ffb80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e255598d0 con 0x559e25788750 2023-03-09T14:45:52.994 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.982+0000 7f5fe90ffb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e25799510 con 0x559e25789600 2023-03-09T14:45:52.994 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.982+0000 7f5fe90ffb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e257996a0 con 0x559e25787e00 2023-03-09T14:45:52.994 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.982+0000 7f5fe90ffb80 10 client.0 _ll_register_callbacks cb 0x559e2567b4d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:45:52.994 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.982+0000 7f5fd5d9b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559e25788750 0x559e25791f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:52.994 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.982+0000 7f5fd5d9b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559e25788750 0x559e25791f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38200/0 (socket says 172.21.15.38:38200) 2023-03-09T14:45:52.994 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.982+0000 7f5fd5d9b700 1 -- 172.21.15.38:0/77579553 learned_addr learned my addr 172.21.15.38:0/77579553 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:52.994 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.982+0000 7f5fd659c700 1 --2- 172.21.15.38:0/77579553 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559e25787e00 0x559e25791a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:52.995 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.982+0000 7f5fd6d9d700 1 --2- 172.21.15.38:0/77579553 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e25789600 0x559e25798fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:52.995 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.982+0000 7f5fd5d9b700 1 -- 172.21.15.38:0/77579553 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559e25787e00 msgr2=0x559e25791a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:45:52.995 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.982+0000 7f5fd5d9b700 1 --2- 172.21.15.38:0/77579553 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559e25787e00 0x559e25791a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:52.995 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.982+0000 7f5fd5d9b700 1 -- 172.21.15.38:0/77579553 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e25789600 msgr2=0x559e25798fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:45:52.995 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.982+0000 7f5fd5d9b700 1 --2- 172.21.15.38:0/77579553 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e25789600 0x559e25798fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:52.995 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.982+0000 7f5fd5d9b700 1 -- 172.21.15.38:0/77579553 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fd810a5d0 con 0x559e25788750 2023-03-09T14:45:52.995 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.982+0000 7f5fd5d9b700 1 --2- 172.21.15.38:0/77579553 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559e25788750 0x559e25791f90 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f5fcc09fc40 tx=0x7f5fcc0a02b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d4c6831b1682492 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:52.995 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.982+0000 7f5fbdffb700 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2023-03-09T14:45:52.996 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.982+0000 7f5fbdffb700 1 -- 172.21.15.38:0/77579553 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fcc0b4070 con 0x559e25788750 2023-03-09T14:45:52.996 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.982+0000 7f5fbdffb700 1 -- 172.21.15.38:0/77579553 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f5fcc00f900 con 0x559e25788750 2023-03-09T14:45:52.996 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.982+0000 7f5fe90ffb80 1 -- 172.21.15.38:0/77579553 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559e256a3140 con 0x559e25788750 2023-03-09T14:45:52.996 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.983+0000 7f5fbdffb700 1 -- 172.21.15.38:0/77579553 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5fcc096320 con 0x559e25788750 2023-03-09T14:45:52.996 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.983+0000 7f5fe90ffb80 10 client.15324 fetch_fsmap learned FSMap version 23 2023-03-09T14:45:52.996 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.983+0000 7f5fe90ffb80 1 -- 172.21.15.38:0/77579553 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x559e254ecc10 con 0x559e25788750 2023-03-09T14:45:52.996 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.983+0000 7f5fbdffb700 1 -- 172.21.15.38:0/77579553 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x559e256a3140 con 0x559e25788750 2023-03-09T14:45:52.997 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.983+0000 7f5fbdffb700 1 -- 172.21.15.38:0/77579553 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f5fcc09fce0 con 0x559e25788750 2023-03-09T14:45:52.997 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.983+0000 7f5fe90ffb80 10 client.15324 fetch_fsmap finished waiting for FSMap version 23 2023-03-09T14:45:52.997 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.983+0000 7f5fe90ffb80 10 client.15324 resolve_mds: resolved name 'i' to daemon mds.i 2023-03-09T14:45:52.997 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.983+0000 7f5fe90ffb80 20 client.15324 populate_metadata read hostname 'smithi038' 2023-03-09T14:45:52.997 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.983+0000 7f5fe90ffb80 1 --2- 172.21.15.38:0/77579553 >> [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] conn(0x559e2579cfd0 0x559e2579c180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:52.997 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.983+0000 7f5fe90ffb80 4 client.15324 mds_command: new command op to 24478 tid=0[{"prefix": "get_command_descriptions"}] 2023-03-09T14:45:52.998 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.983+0000 7f5fe90ffb80 1 -- 172.21.15.38:0/77579553 --> [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559e25559f10 con 0x559e2579cfd0 2023-03-09T14:45:52.998 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.983+0000 7f5fd659c700 1 --2- 172.21.15.38:0/77579553 >> [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] conn(0x559e2579cfd0 0x559e2579c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:52.998 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.984+0000 7f5fbdffb700 10 client.15324 ms_handle_connect on v2:172.21.15.181:6838/3902263660 2023-03-09T14:45:52.998 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.984+0000 7f5fd659c700 1 --2- 172.21.15.38:0/77579553 >> [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] conn(0x559e2579cfd0 0x559e2579c180 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=95be12a38c7df64 server_cookie=c2066fad61052684 in_seq=0 out_seq=0 2023-03-09T14:45:52.998 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.984+0000 7f5fbdffb700 1 -- 172.21.15.38:0/77579553 <== mds.2 v2:172.21.15.181:6838/3902263660 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18278 (crc 0 0 0) 0x559e25559f10 con 0x559e2579cfd0 2023-03-09T14:45:52.998 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.984+0000 7f5fbdffb700 10 client.15324 handle_command_reply: tid=0 2023-03-09T14:45:52.998 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.984+0000 7f5fe90ffb80 1 client.15324 shutdown 2023-03-09T14:45:52.998 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.984+0000 7f5fe90ffb80 1 -- 172.21.15.38:0/77579553 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559e25788750 msgr2=0x559e25791f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:52.999 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.984+0000 7f5fe90ffb80 1 --2- 172.21.15.38:0/77579553 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559e25788750 0x559e25791f90 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f5fcc09fc40 tx=0x7f5fcc0a02b0 comp rx=0 tx=0).stop 2023-03-09T14:45:52.999 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.984+0000 7f5fe90ffb80 1 -- 172.21.15.38:0/77579553 shutdown_connections 2023-03-09T14:45:52.999 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.984+0000 7f5fe90ffb80 1 --2- 172.21.15.38:0/77579553 >> [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] conn(0x559e2579cfd0 0x559e2579c180 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:52.999 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.984+0000 7f5fe90ffb80 1 --2- 172.21.15.38:0/77579553 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e25789600 0x559e25798fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:52.999 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.984+0000 7f5fe90ffb80 1 --2- 172.21.15.38:0/77579553 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559e25787e00 0x559e25791a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:52.999 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.984+0000 7f5fe90ffb80 1 --2- 172.21.15.38:0/77579553 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559e25788750 0x559e25791f90 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:52.999 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.984+0000 7f5fbdffb700 0 client.15324 ms_handle_reset on v2:172.21.15.181:6838/3902263660 2023-03-09T14:45:52.999 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.984+0000 7f5fe90ffb80 1 -- 172.21.15.38:0/77579553 >> 172.21.15.38:0/77579553 conn(0x559e2577e500 msgr2=0x559e2577dbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:53.000 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.984+0000 7f5fe90ffb80 1 -- 172.21.15.38:0/77579553 shutdown_connections 2023-03-09T14:45:53.000 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.985+0000 7f5fe90ffb80 1 -- 172.21.15.38:0/77579553 wait complete. 2023-03-09T14:45:53.000 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:52.985+0000 7f5fe90ffb80 20 client.15324 trim_cache size 0 max 16384 2023-03-09T14:45:53.008 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.002+0000 7f5fe90ffb80 1 Processor -- start 2023-03-09T14:45:53.008 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.002+0000 7f5fe90ffb80 1 -- start start 2023-03-09T14:45:53.009 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.003+0000 7f5fe90ffb80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559e25787e00 0x559e257a5730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:53.009 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.003+0000 7f5fe90ffb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559e25788750 0x559e25798f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:53.009 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.003+0000 7f5fe90ffb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e25789600 0x559e25799a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:53.009 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.003+0000 7f5fe90ffb80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e256daa60 con 0x559e25787e00 2023-03-09T14:45:53.009 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.003+0000 7f5fe90ffb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e256954b0 con 0x559e25789600 2023-03-09T14:45:53.009 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.003+0000 7f5fe90ffb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e255e1ed0 con 0x559e25788750 2023-03-09T14:45:53.009 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.003+0000 7f5fd659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559e25787e00 0x559e257a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:53.010 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.003+0000 7f5fd659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559e25787e00 0x559e257a5730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38206/0 (socket says 172.21.15.38:38206) 2023-03-09T14:45:53.010 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.003+0000 7f5fd659c700 1 -- 172.21.15.38:0/729851863 learned_addr learned my addr 172.21.15.38:0/729851863 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:53.010 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.003+0000 7f5fd6d9d700 1 --2- 172.21.15.38:0/729851863 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e25789600 0x559e25799a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:53.010 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.003+0000 7f5fd659c700 1 -- 172.21.15.38:0/729851863 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559e25788750 msgr2=0x559e25798f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:45:53.010 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.003+0000 7f5fd5d9b700 1 --2- 172.21.15.38:0/729851863 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559e25788750 0x559e25798f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:53.010 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.003+0000 7f5fd659c700 1 --2- 172.21.15.38:0/729851863 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559e25788750 0x559e25798f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.010 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.003+0000 7f5fd659c700 1 -- 172.21.15.38:0/729851863 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e25789600 msgr2=0x559e25799a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:45:53.011 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.003+0000 7f5fd659c700 1 --2- 172.21.15.38:0/729851863 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e25789600 0x559e25799a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.011 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.003+0000 7f5fd659c700 1 -- 172.21.15.38:0/729851863 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559e25503d90 con 0x559e25787e00 2023-03-09T14:45:53.011 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.003+0000 7f5fd6d9d700 1 --2- 172.21.15.38:0/729851863 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e25789600 0x559e25799a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-09T14:45:53.011 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.003+0000 7f5fd5d9b700 1 --2- 172.21.15.38:0/729851863 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559e25788750 0x559e25798f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-09T14:45:53.011 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.003+0000 7f5fd659c700 1 --2- 172.21.15.38:0/729851863 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559e25787e00 0x559e257a5730 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f5fd80ffe00 tx=0x7f5fc8022000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c91b3bf111abbe1 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:53.011 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.003+0000 7f5fc57fa700 1 -- 172.21.15.38:0/729851863 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fc8034d00 con 0x559e25787e00 2023-03-09T14:45:53.011 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.003+0000 7f5fc57fa700 1 -- 172.21.15.38:0/729851863 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5fc802c040 con 0x559e25787e00 2023-03-09T14:45:53.011 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.003+0000 7f5fc57fa700 1 -- 172.21.15.38:0/729851863 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fc8004500 con 0x559e25787e00 2023-03-09T14:45:53.012 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.004+0000 7f5fe90ffb80 1 -- 172.21.15.38:0/729851863 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559e25787e00 msgr2=0x559e257a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:53.012 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.004+0000 7f5fe90ffb80 1 --2- 172.21.15.38:0/729851863 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559e25787e00 0x559e257a5730 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f5fd80ffe00 tx=0x7f5fc8022000 comp rx=0 tx=0).stop 2023-03-09T14:45:53.012 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.004+0000 7f5fe90ffb80 1 -- 172.21.15.38:0/729851863 shutdown_connections 2023-03-09T14:45:53.012 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.004+0000 7f5fe90ffb80 1 --2- 172.21.15.38:0/729851863 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559e25787e00 0x559e257a5730 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.012 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.004+0000 7f5fe90ffb80 1 --2- 172.21.15.38:0/729851863 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e25789600 0x559e25799a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.012 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.004+0000 7f5fe90ffb80 1 --2- 172.21.15.38:0/729851863 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559e25788750 0x559e25798f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.012 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.004+0000 7f5fe90ffb80 1 -- 172.21.15.38:0/729851863 >> 172.21.15.38:0/729851863 conn(0x559e2577e500 msgr2=0x559e25790790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:53.013 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.004+0000 7f5fe90ffb80 1 -- 172.21.15.38:0/729851863 shutdown_connections 2023-03-09T14:45:53.013 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.004+0000 7f5fe90ffb80 1 -- 172.21.15.38:0/729851863 wait complete. 2023-03-09T14:45:53.013 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.004+0000 7f5fe90ffb80 1 -- start start 2023-03-09T14:45:53.013 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.004+0000 7f5fe90ffb80 1 Processor -- start 2023-03-09T14:45:53.013 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.005+0000 7f5fe90ffb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559e25787e00 0x559e25794340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:53.013 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.005+0000 7f5fe90ffb80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559e25788750 0x559e25794880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:53.014 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.005+0000 7f5fe90ffb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e25789600 0x559e257978a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:53.014 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.005+0000 7f5fe90ffb80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e25797de0 con 0x559e25788750 2023-03-09T14:45:53.014 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.005+0000 7f5fe90ffb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e25797f50 con 0x559e25789600 2023-03-09T14:45:53.014 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.005+0000 7f5fe90ffb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e25798110 con 0x559e25787e00 2023-03-09T14:45:53.014 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.005+0000 7f5fe90ffb80 10 client.0 _ll_register_callbacks cb 0x559e2567b4d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:45:53.014 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.005+0000 7f5fd5d9b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559e25788750 0x559e25794880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:53.014 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.005+0000 7f5fd5d9b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559e25788750 0x559e25794880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38214/0 (socket says 172.21.15.38:38214) 2023-03-09T14:45:53.014 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.005+0000 7f5fd5d9b700 1 -- 172.21.15.38:0/1782291787 learned_addr learned my addr 172.21.15.38:0/1782291787 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:53.015 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.005+0000 7f5fd659c700 1 --2- 172.21.15.38:0/1782291787 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559e25787e00 0x559e25794340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:53.015 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.005+0000 7f5fd6d9d700 1 --2- 172.21.15.38:0/1782291787 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e25789600 0x559e257978a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:53.015 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.005+0000 7f5fd5d9b700 1 -- 172.21.15.38:0/1782291787 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559e25787e00 msgr2=0x559e25794340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:45:53.015 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.005+0000 7f5fd5d9b700 1 --2- 172.21.15.38:0/1782291787 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559e25787e00 0x559e25794340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.015 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.005+0000 7f5fd5d9b700 1 -- 172.21.15.38:0/1782291787 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e25789600 msgr2=0x559e257978a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:45:53.015 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.005+0000 7f5fd5d9b700 1 --2- 172.21.15.38:0/1782291787 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e25789600 0x559e257978a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.016 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.005+0000 7f5fd5d9b700 1 -- 172.21.15.38:0/1782291787 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fd8107b20 con 0x559e25788750 2023-03-09T14:45:53.016 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.005+0000 7f5fd5d9b700 1 --2- 172.21.15.38:0/1782291787 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559e25788750 0x559e25794880 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f5fcc000c00 tx=0x7f5fcc096530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b76199c5f3fe127 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:53.016 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.005+0000 7f5fbdffb700 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2023-03-09T14:45:53.016 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.005+0000 7f5fbdffb700 1 -- 172.21.15.38:0/1782291787 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fcc0b4020 con 0x559e25788750 2023-03-09T14:45:53.016 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.005+0000 7f5fbdffb700 1 -- 172.21.15.38:0/1782291787 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f5fcc092d00 con 0x559e25788750 2023-03-09T14:45:53.016 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.005+0000 7f5fe90ffb80 1 -- 172.21.15.38:0/1782291787 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559e25693160 con 0x559e25788750 2023-03-09T14:45:53.016 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.006+0000 7f5fbdffb700 1 -- 172.21.15.38:0/1782291787 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5fcc0942c0 con 0x559e25788750 2023-03-09T14:45:53.017 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.006+0000 7f5fe90ffb80 10 client.15330 fetch_fsmap learned FSMap version 23 2023-03-09T14:45:53.017 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.006+0000 7f5fe90ffb80 1 -- 172.21.15.38:0/1782291787 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x559e25502260 con 0x559e25788750 2023-03-09T14:45:53.017 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.006+0000 7f5fbdffb700 1 -- 172.21.15.38:0/1782291787 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x559e25693160 con 0x559e25788750 2023-03-09T14:45:53.017 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.006+0000 7f5fbdffb700 1 -- 172.21.15.38:0/1782291787 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f5fcc098270 con 0x559e25788750 2023-03-09T14:45:53.017 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.006+0000 7f5fe90ffb80 10 client.15330 fetch_fsmap finished waiting for FSMap version 23 2023-03-09T14:45:53.017 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.006+0000 7f5fe90ffb80 10 client.15330 resolve_mds: resolved name 'i' to daemon mds.i 2023-03-09T14:45:53.018 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.006+0000 7f5fe90ffb80 20 client.15330 populate_metadata read hostname 'smithi038' 2023-03-09T14:45:53.018 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.006+0000 7f5fe90ffb80 1 --2- 172.21.15.38:0/1782291787 >> [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] conn(0x559e257f5820 0x559e25799270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:53.018 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.006+0000 7f5fe90ffb80 4 client.15330 mds_command: new command op to 24478 tid=0[{"prefix": "status"}] 2023-03-09T14:45:53.018 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.006+0000 7f5fe90ffb80 1 -- 172.21.15.38:0/1782291787 --> [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] -- command(tid 0: {"prefix": "status"}) v1 -- 0x559e2567d170 con 0x559e257f5820 2023-03-09T14:45:53.018 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.006+0000 7f5fd659c700 1 --2- 172.21.15.38:0/1782291787 >> [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] conn(0x559e257f5820 0x559e25799270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:53.020 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.007+0000 7f5fbdffb700 10 client.15330 ms_handle_connect on v2:172.21.15.181:6838/3902263660 2023-03-09T14:45:53.020 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.007+0000 7f5fd659c700 1 --2- 172.21.15.38:0/1782291787 >> [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] conn(0x559e257f5820 0x559e25799270 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4fcf7a7e3da93af server_cookie=318dbe5adb624d6f in_seq=0 out_seq=0 2023-03-09T14:45:53.020 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.007+0000 7f5fbdffb700 1 -- 172.21.15.38:0/1782291787 <== mds.2 v2:172.21.15.181:6838/3902263660 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x559e2567d170 con 0x559e257f5820 2023-03-09T14:45:53.020 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.007+0000 7f5fbdffb700 10 client.15330 handle_command_reply: tid=0 2023-03-09T14:45:53.021 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.007+0000 7f5fe90ffb80 1 client.15330 shutdown 2023-03-09T14:45:53.021 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.007+0000 7f5fe90ffb80 1 -- 172.21.15.38:0/1782291787 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559e25788750 msgr2=0x559e25794880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:53.021 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.007+0000 7f5fe90ffb80 1 --2- 172.21.15.38:0/1782291787 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559e25788750 0x559e25794880 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f5fcc000c00 tx=0x7f5fcc096530 comp rx=0 tx=0).stop 2023-03-09T14:45:53.021 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.007+0000 7f5fe90ffb80 1 -- 172.21.15.38:0/1782291787 shutdown_connections 2023-03-09T14:45:53.021 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.007+0000 7f5fe90ffb80 1 --2- 172.21.15.38:0/1782291787 >> [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] conn(0x559e257f5820 0x559e25799270 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.021 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.007+0000 7f5fe90ffb80 1 --2- 172.21.15.38:0/1782291787 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559e25789600 0x559e257978a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.021 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.007+0000 7f5fbdffb700 0 client.15330 ms_handle_reset on v2:172.21.15.181:6838/3902263660 2023-03-09T14:45:53.022 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.007+0000 7f5fe90ffb80 1 --2- 172.21.15.38:0/1782291787 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559e25787e00 0x559e25794340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.022 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.007+0000 7f5fe90ffb80 1 --2- 172.21.15.38:0/1782291787 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559e25788750 0x559e25794880 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.022 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.007+0000 7f5fe90ffb80 1 -- 172.21.15.38:0/1782291787 >> 172.21.15.38:0/1782291787 conn(0x559e2577e500 msgr2=0x559e25790790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:53.022 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.007+0000 7f5fe90ffb80 1 -- 172.21.15.38:0/1782291787 shutdown_connections 2023-03-09T14:45:53.022 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.007+0000 7f5fe90ffb80 1 -- 172.21.15.38:0/1782291787 wait complete. 2023-03-09T14:45:53.022 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.007+0000 7f5fe90ffb80 20 client.15330 trim_cache size 0 max 16384 2023-03-09T14:45:53.023 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2023-03-09T14:45:53.023 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_fsid": "c6733fe2-be87-11ed-9af2-001a4aab830c", 2023-03-09T14:45:53.023 INFO:teuthology.orchestra.run.smithi038.stdout: "whoami": 2, 2023-03-09T14:45:53.023 INFO:teuthology.orchestra.run.smithi038.stdout: "id": 24478, 2023-03-09T14:45:53.023 INFO:teuthology.orchestra.run.smithi038.stdout: "want_state": "up:active", 2023-03-09T14:45:53.024 INFO:teuthology.orchestra.run.smithi038.stdout: "state": "up:active", 2023-03-09T14:45:53.024 INFO:teuthology.orchestra.run.smithi038.stdout: "fs_name": "cephfs", 2023-03-09T14:45:53.024 INFO:teuthology.orchestra.run.smithi038.stdout: "rank_uptime": 74.493540867999997, 2023-03-09T14:45:53.024 INFO:teuthology.orchestra.run.smithi038.stdout: "mdsmap_epoch": 23, 2023-03-09T14:45:53.024 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_epoch": 89, 2023-03-09T14:45:53.024 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_epoch_barrier": 87, 2023-03-09T14:45:53.024 INFO:teuthology.orchestra.run.smithi038.stdout: "uptime": 106.35365928100001 2023-03-09T14:45:53.024 INFO:teuthology.orchestra.run.smithi038.stdout:} 2023-03-09T14:45:53.025 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.007+0000 7f5fdcf76700 1 -- 172.21.15.38:0/3604452645 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f5fc009f3c0 msgr2=0x7f5fc00a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:53.026 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.008+0000 7f5fdcf76700 1 --2- 172.21.15.38:0/3604452645 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f5fc009f3c0 0x7f5fc00a1870 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f5fc800b5c0 tx=0x7f5fc8005fb0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.026 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.008+0000 7f5fdcf76700 1 -- 172.21.15.38:0/3604452645 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5fd80fffa0 msgr2=0x7f5fd819aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:53.026 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.008+0000 7f5fdcf76700 1 --2- 172.21.15.38:0/3604452645 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5fd80fffa0 0x7f5fd819aa50 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f5fcc00c9b0 tx=0x7f5fcc00ccc0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.026 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.008+0000 7f5fdcf76700 1 -- 172.21.15.38:0/3604452645 shutdown_connections 2023-03-09T14:45:53.026 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.008+0000 7f5fdcf76700 1 --2- 172.21.15.38:0/3604452645 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f5fc009f3c0 0x7f5fc00a1870 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.026 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.008+0000 7f5fdcf76700 1 --2- 172.21.15.38:0/3604452645 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5fd8100930 0x7f5fd819d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.027 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.008+0000 7f5fdcf76700 1 --2- 172.21.15.38:0/3604452645 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5fd80fedb0 0x7f5fd8069600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.027 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.008+0000 7f5fdcf76700 1 --2- 172.21.15.38:0/3604452645 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5fd80fffa0 0x7f5fd819aa50 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.027 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.008+0000 7f5fdcf76700 1 -- 172.21.15.38:0/3604452645 >> 172.21.15.38:0/3604452645 conn(0x7f5fd80fa3b0 msgr2=0x7f5fd81031d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:53.027 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.008+0000 7f5fdcf76700 1 -- 172.21.15.38:0/3604452645 shutdown_connections 2023-03-09T14:45:53.027 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.008+0000 7f5fdcf76700 1 -- 172.21.15.38:0/3604452645 wait complete. 2023-03-09T14:45:53.028 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2023-03-09T14:45:53.156 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.152+0000 7f76327cd700 1 -- 172.21.15.38:0/1605103157 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f762c0fffc0 msgr2=0x7f762c100410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:53.156 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.152+0000 7f76327cd700 1 --2- 172.21.15.38:0/1605103157 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f762c0fffc0 0x7f762c100410 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f7614009990 tx=0x7f7614009ca0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.156 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.152+0000 7f76327cd700 1 -- 172.21.15.38:0/1605103157 shutdown_connections 2023-03-09T14:45:53.157 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.152+0000 7f76327cd700 1 --2- 172.21.15.38:0/1605103157 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f762c100950 0x7f762c107210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.157 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.152+0000 7f76327cd700 1 --2- 172.21.15.38:0/1605103157 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f762c0fedd0 0x7f762c0ff1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.157 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.152+0000 7f76327cd700 1 --2- 172.21.15.38:0/1605103157 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f762c0fffc0 0x7f762c100410 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.157 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.152+0000 7f76327cd700 1 -- 172.21.15.38:0/1605103157 >> 172.21.15.38:0/1605103157 conn(0x7f762c0fa3d0 msgr2=0x7f762c0fc820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:53.157 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.152+0000 7f76327cd700 1 -- 172.21.15.38:0/1605103157 shutdown_connections 2023-03-09T14:45:53.157 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.152+0000 7f76327cd700 1 -- 172.21.15.38:0/1605103157 wait complete. 2023-03-09T14:45:53.157 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.153+0000 7f76327cd700 1 Processor -- start 2023-03-09T14:45:53.158 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.153+0000 7f76327cd700 1 -- start start 2023-03-09T14:45:53.158 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.153+0000 7f76327cd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f762c0fedd0 0x7f762c0695c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:53.158 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.153+0000 7f76327cd700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f762c0fffc0 0x7f762c19a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:53.158 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.153+0000 7f76327cd700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f762c100950 0x7f762c19cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:53.158 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.153+0000 7f76327cd700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f762c19d6d0 con 0x7f762c0fedd0 2023-03-09T14:45:53.158 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.153+0000 7f76327cd700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f762c19d840 con 0x7f762c0fffc0 2023-03-09T14:45:53.158 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.153+0000 7f76327cd700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f762c19da00 con 0x7f762c100950 2023-03-09T14:45:53.159 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.153+0000 7f762bfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f762c0fedd0 0x7f762c0695c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:53.159 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.153+0000 7f762bfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f762c0fedd0 0x7f762c0695c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38226/0 (socket says 172.21.15.38:38226) 2023-03-09T14:45:53.159 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.153+0000 7f762bfff700 1 -- 172.21.15.38:0/3939417108 learned_addr learned my addr 172.21.15.38:0/3939417108 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:53.159 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.153+0000 7f7630d6a700 1 --2- 172.21.15.38:0/3939417108 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f762c100950 0x7f762c19cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:53.159 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.153+0000 7f762bfff700 1 -- 172.21.15.38:0/3939417108 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f762c100950 msgr2=0x7f762c19cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:53.159 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.153+0000 7f7623fff700 1 --2- 172.21.15.38:0/3939417108 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f762c0fffc0 0x7f762c19a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:53.159 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.153+0000 7f762bfff700 1 --2- 172.21.15.38:0/3939417108 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f762c100950 0x7f762c19cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.160 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.153+0000 7f762bfff700 1 -- 172.21.15.38:0/3939417108 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f762c0fffc0 msgr2=0x7f762c19a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:53.160 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.153+0000 7f762bfff700 1 --2- 172.21.15.38:0/3939417108 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f762c0fffc0 0x7f762c19a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.160 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.153+0000 7f762bfff700 1 -- 172.21.15.38:0/3939417108 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76140095f0 con 0x7f762c0fedd0 2023-03-09T14:45:53.160 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.153+0000 7f762bfff700 1 --2- 172.21.15.38:0/3939417108 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f762c0fedd0 0x7f762c0695c0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f761c00d980 tx=0x7f761c00dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:53.160 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.153+0000 7f7629ffb700 1 -- 172.21.15.38:0/3939417108 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f761c00bb60 con 0x7f762c0fedd0 2023-03-09T14:45:53.160 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.154+0000 7f7629ffb700 1 -- 172.21.15.38:0/3939417108 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f761c007620 con 0x7f762c0fedd0 2023-03-09T14:45:53.161 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.154+0000 7f76327cd700 1 -- 172.21.15.38:0/3939417108 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f762c10a730 con 0x7f762c0fedd0 2023-03-09T14:45:53.161 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.154+0000 7f7629ffb700 1 -- 172.21.15.38:0/3939417108 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f761c00f620 con 0x7f762c0fedd0 2023-03-09T14:45:53.161 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.154+0000 7f76327cd700 1 -- 172.21.15.38:0/3939417108 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f762c10a5a0 con 0x7f762c0fedd0 2023-03-09T14:45:53.165 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.156+0000 7f7629ffb700 1 -- 172.21.15.38:0/3939417108 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) v1 ==== 133198+0+0 (secure 0 0 0) 0x7f761c018030 con 0x7f762c0fedd0 2023-03-09T14:45:53.166 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.156+0000 7f7629ffb700 1 --2- 172.21.15.38:0/3939417108 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f761809f3c0 0x7f76180a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:53.166 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.156+0000 7f7623fff700 1 --2- 172.21.15.38:0/3939417108 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f761809f3c0 0x7f76180a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:53.166 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.156+0000 7f7629ffb700 1 -- 172.21.15.38:0/3939417108 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f761c0c9fc0 con 0x7f762c0fedd0 2023-03-09T14:45:53.166 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.157+0000 7f763e956b80 1 Processor -- start 2023-03-09T14:45:53.166 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.157+0000 7f7623fff700 1 --2- 172.21.15.38:0/3939417108 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f761809f3c0 0x7f76180a1870 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f761400b5c0 tx=0x7f7614005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:53.166 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.157+0000 7f763e956b80 1 -- start start 2023-03-09T14:45:53.167 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.157+0000 7f763e956b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ea3ab279f0 0x55ea3ab27e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:53.167 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.157+0000 7f763e956b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ea3ab28340 0x55ea3ab28c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:53.167 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.157+0000 7f763e956b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ea3ab291f0 0x55ea3ab29660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:53.167 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.157+0000 7f763e956b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea3aa11d50 con 0x55ea3ab291f0 2023-03-09T14:45:53.167 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.157+0000 7f763e956b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea3aa80730 con 0x55ea3ab28340 2023-03-09T14:45:53.167 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.157+0000 7f763e956b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea3aa79c60 con 0x55ea3ab279f0 2023-03-09T14:45:53.167 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.157+0000 7f7630d6a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ea3ab291f0 0x55ea3ab29660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:53.168 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.157+0000 7f7630d6a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ea3ab291f0 0x55ea3ab29660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38238/0 (socket says 172.21.15.38:38238) 2023-03-09T14:45:53.168 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.157+0000 7f7630d6a700 1 -- 172.21.15.38:0/1645472879 learned_addr learned my addr 172.21.15.38:0/1645472879 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:53.168 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.157+0000 7f762bfff700 1 --2- 172.21.15.38:0/1645472879 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ea3ab279f0 0x55ea3ab27e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:53.168 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.157+0000 7f7623fff700 1 --2- 172.21.15.38:0/1645472879 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ea3ab28340 0x55ea3ab28c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:53.168 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.158+0000 7f7630d6a700 1 -- 172.21.15.38:0/1645472879 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ea3ab279f0 msgr2=0x55ea3ab27e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:45:53.169 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.158+0000 7f7630d6a700 1 --2- 172.21.15.38:0/1645472879 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ea3ab279f0 0x55ea3ab27e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.169 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.158+0000 7f7630d6a700 1 -- 172.21.15.38:0/1645472879 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ea3ab28340 msgr2=0x55ea3ab28c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:45:53.169 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.158+0000 7f7630d6a700 1 --2- 172.21.15.38:0/1645472879 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ea3ab28340 0x55ea3ab28c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.169 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.158+0000 7f7630d6a700 1 -- 172.21.15.38:0/1645472879 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ea3aa40140 con 0x55ea3ab291f0 2023-03-09T14:45:53.169 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.158+0000 7f7630d6a700 1 --2- 172.21.15.38:0/1645472879 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ea3ab291f0 0x55ea3ab29660 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f7624007f70 tx=0x7f762400e440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42d9e6dad41fd5d4 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:53.169 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.158+0000 7f76227fc700 1 -- 172.21.15.38:0/1645472879 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7624018070 con 0x55ea3ab291f0 2023-03-09T14:45:53.170 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.158+0000 7f76227fc700 1 -- 172.21.15.38:0/1645472879 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7624007bb0 con 0x55ea3ab291f0 2023-03-09T14:45:53.170 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.158+0000 7f76227fc700 1 -- 172.21.15.38:0/1645472879 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7624014b20 con 0x55ea3ab291f0 2023-03-09T14:45:53.170 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.159+0000 7f763e956b80 1 -- 172.21.15.38:0/1645472879 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ea3ab291f0 msgr2=0x55ea3ab29660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:53.170 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.159+0000 7f763e956b80 1 --2- 172.21.15.38:0/1645472879 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ea3ab291f0 0x55ea3ab29660 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f7624007f70 tx=0x7f762400e440 comp rx=0 tx=0).stop 2023-03-09T14:45:53.170 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.159+0000 7f763e956b80 1 -- 172.21.15.38:0/1645472879 shutdown_connections 2023-03-09T14:45:53.170 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.159+0000 7f763e956b80 1 --2- 172.21.15.38:0/1645472879 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ea3ab291f0 0x55ea3ab29660 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.170 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.159+0000 7f763e956b80 1 --2- 172.21.15.38:0/1645472879 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ea3ab279f0 0x55ea3ab27e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.171 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.159+0000 7f763e956b80 1 --2- 172.21.15.38:0/1645472879 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ea3ab28340 0x55ea3ab28c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.171 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.159+0000 7f763e956b80 1 -- 172.21.15.38:0/1645472879 >> 172.21.15.38:0/1645472879 conn(0x55ea3ab22500 msgr2=0x55ea3ab228f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:53.171 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.159+0000 7f763e956b80 1 -- 172.21.15.38:0/1645472879 shutdown_connections 2023-03-09T14:45:53.171 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.159+0000 7f763e956b80 1 -- 172.21.15.38:0/1645472879 wait complete. 2023-03-09T14:45:53.171 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.159+0000 7f763e956b80 1 -- start start 2023-03-09T14:45:53.171 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.159+0000 7f763e956b80 1 Processor -- start 2023-03-09T14:45:53.171 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.160+0000 7f763e956b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ea3ab279f0 0x55ea3ab31730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:53.172 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.160+0000 7f763e956b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ea3ab28340 0x55ea3ab31c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:53.172 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.160+0000 7f763e956b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ea3ab291f0 0x55ea3ab38cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:53.172 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.160+0000 7f763e956b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea3a8fd8d0 con 0x55ea3ab291f0 2023-03-09T14:45:53.172 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.160+0000 7f763e956b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea3ab391f0 con 0x55ea3ab28340 2023-03-09T14:45:53.172 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.160+0000 7f763e956b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea3ab39380 con 0x55ea3ab279f0 2023-03-09T14:45:53.172 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.160+0000 7f763e956b80 10 client.0 _ll_register_callbacks cb 0x55ea3aa1f4d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:45:53.172 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.160+0000 7f762bfff700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ea3ab279f0 0x55ea3ab31730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:53.173 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.160+0000 7f762bfff700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ea3ab279f0 0x55ea3ab31730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.38:45938/0 (socket says 172.21.15.38:45938) 2023-03-09T14:45:53.173 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.160+0000 7f762bfff700 1 -- 172.21.15.38:0/2548147113 learned_addr learned my addr 172.21.15.38:0/2548147113 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:53.173 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.160+0000 7f7623fff700 1 --2- 172.21.15.38:0/2548147113 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ea3ab28340 0x55ea3ab31c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:53.173 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.160+0000 7f7630d6a700 1 --2- 172.21.15.38:0/2548147113 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ea3ab291f0 0x55ea3ab38cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:53.173 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.160+0000 7f762bfff700 1 -- 172.21.15.38:0/2548147113 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ea3ab28340 msgr2=0x55ea3ab31c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:45:53.173 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.160+0000 7f762bfff700 1 --2- 172.21.15.38:0/2548147113 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ea3ab28340 0x55ea3ab31c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.174 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.160+0000 7f762bfff700 1 -- 172.21.15.38:0/2548147113 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ea3ab291f0 msgr2=0x55ea3ab38cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:45:53.174 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.160+0000 7f762bfff700 1 --2- 172.21.15.38:0/2548147113 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ea3ab291f0 0x55ea3ab38cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.174 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.160+0000 7f762bfff700 1 -- 172.21.15.38:0/2548147113 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f762400f040 con 0x55ea3ab279f0 2023-03-09T14:45:53.174 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.160+0000 7f7630d6a700 1 --2- 172.21.15.38:0/2548147113 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ea3ab291f0 0x55ea3ab38cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:45:53.174 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.160+0000 7f7623fff700 1 --2- 172.21.15.38:0/2548147113 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ea3ab28340 0x55ea3ab31c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:45:53.174 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.160+0000 7f762bfff700 1 --2- 172.21.15.38:0/2548147113 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ea3ab279f0 0x55ea3ab31730 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f761c00b540 tx=0x7f761c010ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7f2e497ff2ff89b3 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:53.174 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.160+0000 7f760b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2023-03-09T14:45:53.174 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.161+0000 7f760b7fe700 1 -- 172.21.15.38:0/2548147113 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f761c09ff20 con 0x55ea3ab279f0 2023-03-09T14:45:53.175 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.161+0000 7f763e956b80 1 -- 172.21.15.38:0/2548147113 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ea3aa47140 con 0x55ea3ab279f0 2023-03-09T14:45:53.175 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.161+0000 7f760b7fe700 1 -- 172.21.15.38:0/2548147113 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f761c09a040 con 0x55ea3ab279f0 2023-03-09T14:45:53.175 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.161+0000 7f760b7fe700 1 -- 172.21.15.38:0/2548147113 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f761c097220 con 0x55ea3ab279f0 2023-03-09T14:45:53.175 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.161+0000 7f763e956b80 10 client.25018 fetch_fsmap learned FSMap version 23 2023-03-09T14:45:53.175 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.161+0000 7f763e956b80 1 -- 172.21.15.38:0/2548147113 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55ea3a890c10 con 0x55ea3ab279f0 2023-03-09T14:45:53.175 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.161+0000 7f760b7fe700 1 -- 172.21.15.38:0/2548147113 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55ea3aa47140 con 0x55ea3ab279f0 2023-03-09T14:45:53.175 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.161+0000 7f760b7fe700 1 -- 172.21.15.38:0/2548147113 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f761c0b1020 con 0x55ea3ab279f0 2023-03-09T14:45:53.176 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.161+0000 7f763e956b80 10 client.25018 fetch_fsmap finished waiting for FSMap version 23 2023-03-09T14:45:53.176 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.161+0000 7f763e956b80 10 client.25018 resolve_mds: resolved name 'h' to daemon mds.h 2023-03-09T14:45:53.176 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.161+0000 7f763e956b80 20 client.25018 populate_metadata read hostname 'smithi038' 2023-03-09T14:45:53.176 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.161+0000 7f763e956b80 1 --2- 172.21.15.38:0/2548147113 >> [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] conn(0x55ea3ab3cb40 0x55ea3ab3bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:53.176 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.161+0000 7f763e956b80 4 client.25018 mds_command: new command op to 24494 tid=0[{"prefix": "get_command_descriptions"}] 2023-03-09T14:45:53.176 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.161+0000 7f763e956b80 1 -- 172.21.15.38:0/2548147113 --> [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ea3a8fdf10 con 0x55ea3ab3cb40 2023-03-09T14:45:53.176 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.162+0000 7f7623fff700 1 --2- 172.21.15.38:0/2548147113 >> [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] conn(0x55ea3ab3cb40 0x55ea3ab3bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:53.177 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.162+0000 7f760b7fe700 10 client.25018 ms_handle_connect on v2:172.21.15.122:6836/2210214596 2023-03-09T14:45:53.177 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.162+0000 7f7623fff700 1 --2- 172.21.15.38:0/2548147113 >> [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] conn(0x55ea3ab3cb40 0x55ea3ab3bfd0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=dc6a80d7b6bb3905 server_cookie=8f1e3324d2e9160b in_seq=0 out_seq=0 2023-03-09T14:45:53.177 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.163+0000 7f760b7fe700 1 -- 172.21.15.38:0/2548147113 <== mds.1 v2:172.21.15.122:6836/2210214596 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18278 (crc 0 0 0) 0x55ea3a8fdf10 con 0x55ea3ab3cb40 2023-03-09T14:45:53.177 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.163+0000 7f760b7fe700 10 client.25018 handle_command_reply: tid=0 2023-03-09T14:45:53.177 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.163+0000 7f763e956b80 1 client.25018 shutdown 2023-03-09T14:45:53.178 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.163+0000 7f763e956b80 1 -- 172.21.15.38:0/2548147113 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ea3ab279f0 msgr2=0x55ea3ab31730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:53.178 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.163+0000 7f763e956b80 1 --2- 172.21.15.38:0/2548147113 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ea3ab279f0 0x55ea3ab31730 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f761c00b540 tx=0x7f761c010ef0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.178 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.163+0000 7f763e956b80 1 -- 172.21.15.38:0/2548147113 shutdown_connections 2023-03-09T14:45:53.178 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.163+0000 7f763e956b80 1 --2- 172.21.15.38:0/2548147113 >> [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] conn(0x55ea3ab3cb40 0x55ea3ab3bfd0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.178 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.163+0000 7f763e956b80 1 --2- 172.21.15.38:0/2548147113 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ea3ab291f0 0x55ea3ab38cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.178 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.163+0000 7f763e956b80 1 --2- 172.21.15.38:0/2548147113 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ea3ab279f0 0x55ea3ab31730 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.178 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.163+0000 7f763e956b80 1 --2- 172.21.15.38:0/2548147113 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ea3ab28340 0x55ea3ab31c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.179 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.163+0000 7f760b7fe700 0 client.25018 ms_handle_reset on v2:172.21.15.122:6836/2210214596 2023-03-09T14:45:53.179 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.163+0000 7f763e956b80 1 -- 172.21.15.38:0/2548147113 >> 172.21.15.38:0/2548147113 conn(0x55ea3ab22500 msgr2=0x55ea3ab21bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:53.179 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.163+0000 7f763e956b80 1 -- 172.21.15.38:0/2548147113 shutdown_connections 2023-03-09T14:45:53.179 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.163+0000 7f763e956b80 1 -- 172.21.15.38:0/2548147113 wait complete. 2023-03-09T14:45:53.179 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.163+0000 7f763e956b80 20 client.25018 trim_cache size 0 max 16384 2023-03-09T14:45:53.184 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.181+0000 7f763e956b80 1 Processor -- start 2023-03-09T14:45:53.184 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.181+0000 7f763e956b80 1 -- start start 2023-03-09T14:45:53.184 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.181+0000 7f763e956b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ea3ab279f0 0x55ea3ab3bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:53.185 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.181+0000 7f763e956b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ea3ab28340 0x55ea3ab38c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:53.185 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.181+0000 7f763e956b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ea3ab291f0 0x55ea3ab2f500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:53.185 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.181+0000 7f763e956b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea3ab32630 con 0x55ea3ab291f0 2023-03-09T14:45:53.185 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.181+0000 7f763e956b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea3a8fdf10 con 0x55ea3ab279f0 2023-03-09T14:45:53.185 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.181+0000 7f763e956b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea3aa47140 con 0x55ea3ab28340 2023-03-09T14:45:53.185 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.181+0000 7f7630d6a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ea3ab291f0 0x55ea3ab2f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:53.186 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.181+0000 7f7630d6a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ea3ab291f0 0x55ea3ab2f500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38252/0 (socket says 172.21.15.38:38252) 2023-03-09T14:45:53.186 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.181+0000 7f7630d6a700 1 -- 172.21.15.38:0/15140195 learned_addr learned my addr 172.21.15.38:0/15140195 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:53.186 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.181+0000 7f7623fff700 1 --2- 172.21.15.38:0/15140195 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ea3ab28340 0x55ea3ab38c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:53.186 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.181+0000 7f762bfff700 1 --2- 172.21.15.38:0/15140195 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ea3ab279f0 0x55ea3ab3bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:53.186 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.182+0000 7f7630d6a700 1 -- 172.21.15.38:0/15140195 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ea3ab28340 msgr2=0x55ea3ab38c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:45:53.186 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.182+0000 7f7630d6a700 1 --2- 172.21.15.38:0/15140195 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ea3ab28340 0x55ea3ab38c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.187 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.182+0000 7f7630d6a700 1 -- 172.21.15.38:0/15140195 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ea3ab279f0 msgr2=0x55ea3ab3bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:45:53.187 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.182+0000 7f7630d6a700 1 --2- 172.21.15.38:0/15140195 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ea3ab279f0 0x55ea3ab3bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.187 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.182+0000 7f7630d6a700 1 -- 172.21.15.38:0/15140195 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ea3a8a7d90 con 0x55ea3ab291f0 2023-03-09T14:45:53.187 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.182+0000 7f7630d6a700 1 --2- 172.21.15.38:0/15140195 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ea3ab291f0 0x55ea3ab2f500 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f7624000f60 tx=0x7f7624007c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ea26039f22fed07 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:53.187 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.182+0000 7f7622ffd700 1 -- 172.21.15.38:0/15140195 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7624018040 con 0x55ea3ab291f0 2023-03-09T14:45:53.187 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.182+0000 7f7622ffd700 1 -- 172.21.15.38:0/15140195 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f762400ec70 con 0x55ea3ab291f0 2023-03-09T14:45:53.187 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.182+0000 7f7622ffd700 1 -- 172.21.15.38:0/15140195 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7624014d20 con 0x55ea3ab291f0 2023-03-09T14:45:53.191 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.182+0000 7f763e956b80 1 -- 172.21.15.38:0/15140195 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ea3ab291f0 msgr2=0x55ea3ab2f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:53.191 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.182+0000 7f763e956b80 1 --2- 172.21.15.38:0/15140195 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ea3ab291f0 0x55ea3ab2f500 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f7624000f60 tx=0x7f7624007c70 comp rx=0 tx=0).stop 2023-03-09T14:45:53.191 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.182+0000 7f763e956b80 1 -- 172.21.15.38:0/15140195 shutdown_connections 2023-03-09T14:45:53.192 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.182+0000 7f763e956b80 1 --2- 172.21.15.38:0/15140195 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ea3ab291f0 0x55ea3ab2f500 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.192 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.183+0000 7f763e956b80 1 --2- 172.21.15.38:0/15140195 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ea3ab279f0 0x55ea3ab3bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.192 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.183+0000 7f763e956b80 1 --2- 172.21.15.38:0/15140195 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ea3ab28340 0x55ea3ab38c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.192 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.183+0000 7f763e956b80 1 -- 172.21.15.38:0/15140195 >> 172.21.15.38:0/15140195 conn(0x55ea3ab22500 msgr2=0x55ea3ab39760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:53.193 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.183+0000 7f763e956b80 1 -- 172.21.15.38:0/15140195 shutdown_connections 2023-03-09T14:45:53.193 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.183+0000 7f763e956b80 1 -- 172.21.15.38:0/15140195 wait complete. 2023-03-09T14:45:53.193 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.183+0000 7f763e956b80 1 -- start start 2023-03-09T14:45:53.194 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.183+0000 7f763e956b80 1 Processor -- start 2023-03-09T14:45:53.194 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.183+0000 7f763e956b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ea3ab279f0 0x55ea3ab344b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:53.194 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.183+0000 7f763e956b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ea3ab28340 0x55ea3ab349f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:53.194 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.183+0000 7f763e956b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ea3ab291f0 0x55ea3ab379c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:53.195 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.183+0000 7f763e956b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea3ab37f00 con 0x55ea3ab291f0 2023-03-09T14:45:53.195 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.183+0000 7f763e956b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea3ab38070 con 0x55ea3ab279f0 2023-03-09T14:45:53.195 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.183+0000 7f763e956b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea3ab38230 con 0x55ea3ab28340 2023-03-09T14:45:53.195 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.183+0000 7f763e956b80 10 client.0 _ll_register_callbacks cb 0x55ea3aa1f4d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:45:53.195 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.183+0000 7f7630d6a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ea3ab291f0 0x55ea3ab379c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:53.195 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.183+0000 7f7630d6a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ea3ab291f0 0x55ea3ab379c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38258/0 (socket says 172.21.15.38:38258) 2023-03-09T14:45:53.196 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.183+0000 7f7630d6a700 1 -- 172.21.15.38:0/210886005 learned_addr learned my addr 172.21.15.38:0/210886005 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:53.196 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.184+0000 7f762bfff700 1 --2- 172.21.15.38:0/210886005 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ea3ab279f0 0x55ea3ab344b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:53.196 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.184+0000 7f7623fff700 1 --2- 172.21.15.38:0/210886005 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ea3ab28340 0x55ea3ab349f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:53.196 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.184+0000 7f7630d6a700 1 -- 172.21.15.38:0/210886005 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ea3ab28340 msgr2=0x55ea3ab349f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:45:53.196 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.184+0000 7f7630d6a700 1 --2- 172.21.15.38:0/210886005 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ea3ab28340 0x55ea3ab349f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.197 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.184+0000 7f7630d6a700 1 -- 172.21.15.38:0/210886005 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ea3ab279f0 msgr2=0x55ea3ab344b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:45:53.197 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.184+0000 7f7630d6a700 1 --2- 172.21.15.38:0/210886005 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ea3ab279f0 0x55ea3ab344b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.197 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.184+0000 7f7630d6a700 1 -- 172.21.15.38:0/210886005 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ea3aa40140 con 0x55ea3ab291f0 2023-03-09T14:45:53.197 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.184+0000 7f7630d6a700 1 --2- 172.21.15.38:0/210886005 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ea3ab291f0 0x55ea3ab379c0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f7624006130 tx=0x7f7624003c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6321a10edd283ec server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:53.197 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.184+0000 7f760b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2023-03-09T14:45:53.197 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.184+0000 7f760b7fe700 1 -- 172.21.15.38:0/210886005 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76240145c0 con 0x55ea3ab291f0 2023-03-09T14:45:53.197 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.184+0000 7f763e956b80 1 -- 172.21.15.38:0/210886005 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ea3aa37160 con 0x55ea3ab291f0 2023-03-09T14:45:53.198 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.184+0000 7f760b7fe700 1 -- 172.21.15.38:0/210886005 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f7624014c30 con 0x55ea3ab291f0 2023-03-09T14:45:53.198 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.184+0000 7f760b7fe700 1 -- 172.21.15.38:0/210886005 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7624014df0 con 0x55ea3ab291f0 2023-03-09T14:45:53.198 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.184+0000 7f763e956b80 10 client.15348 fetch_fsmap learned FSMap version 23 2023-03-09T14:45:53.198 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.184+0000 7f763e956b80 1 -- 172.21.15.38:0/210886005 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55ea3a8a6260 con 0x55ea3ab291f0 2023-03-09T14:45:53.198 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.184+0000 7f760b7fe700 1 -- 172.21.15.38:0/210886005 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7624039080 con 0x55ea3ab291f0 2023-03-09T14:45:53.198 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.185+0000 7f760b7fe700 1 -- 172.21.15.38:0/210886005 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f762401e470 con 0x55ea3ab291f0 2023-03-09T14:45:53.198 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.185+0000 7f763e956b80 10 client.15348 fetch_fsmap finished waiting for FSMap version 23 2023-03-09T14:45:53.199 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.185+0000 7f763e956b80 10 client.15348 resolve_mds: resolved name 'h' to daemon mds.h 2023-03-09T14:45:53.199 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.185+0000 7f763e956b80 20 client.15348 populate_metadata read hostname 'smithi038' 2023-03-09T14:45:53.199 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.185+0000 7f763e956b80 1 --2- 172.21.15.38:0/210886005 >> [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] conn(0x55ea3ab962f0 0x55ea3ab92c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:53.199 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.185+0000 7f763e956b80 4 client.15348 mds_command: new command op to 24494 tid=0[{"prefix": "status"}] 2023-03-09T14:45:53.199 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.185+0000 7f763e956b80 1 -- 172.21.15.38:0/210886005 --> [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55ea3aa21170 con 0x55ea3ab962f0 2023-03-09T14:45:53.199 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.185+0000 7f762bfff700 1 --2- 172.21.15.38:0/210886005 >> [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] conn(0x55ea3ab962f0 0x55ea3ab92c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:53.200 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.185+0000 7f760b7fe700 10 client.15348 ms_handle_connect on v2:172.21.15.122:6836/2210214596 2023-03-09T14:45:53.200 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.185+0000 7f762bfff700 1 --2- 172.21.15.38:0/210886005 >> [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] conn(0x55ea3ab962f0 0x55ea3ab92c80 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e1aa7e8e7a0f3a4e server_cookie=c910181ddb016fc1 in_seq=0 out_seq=0 2023-03-09T14:45:53.200 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.186+0000 7f760b7fe700 1 -- 172.21.15.38:0/210886005 <== mds.1 v2:172.21.15.122:6836/2210214596 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x55ea3aa21170 con 0x55ea3ab962f0 2023-03-09T14:45:53.201 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.186+0000 7f760b7fe700 10 client.15348 handle_command_reply: tid=0 2023-03-09T14:45:53.201 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.186+0000 7f763e956b80 1 client.15348 shutdown 2023-03-09T14:45:53.201 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.186+0000 7f763e956b80 1 -- 172.21.15.38:0/210886005 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ea3ab291f0 msgr2=0x55ea3ab379c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:53.201 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.186+0000 7f763e956b80 1 --2- 172.21.15.38:0/210886005 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ea3ab291f0 0x55ea3ab379c0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f7624006130 tx=0x7f7624003c30 comp rx=0 tx=0).stop 2023-03-09T14:45:53.201 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.186+0000 7f763e956b80 1 -- 172.21.15.38:0/210886005 shutdown_connections 2023-03-09T14:45:53.201 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.186+0000 7f763e956b80 1 --2- 172.21.15.38:0/210886005 >> [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] conn(0x55ea3ab962f0 0x55ea3ab92c80 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.202 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.186+0000 7f763e956b80 1 --2- 172.21.15.38:0/210886005 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55ea3ab291f0 0x55ea3ab379c0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.202 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.186+0000 7f763e956b80 1 --2- 172.21.15.38:0/210886005 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55ea3ab279f0 0x55ea3ab344b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.202 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.186+0000 7f760b7fe700 0 client.15348 ms_handle_reset on v2:172.21.15.122:6836/2210214596 2023-03-09T14:45:53.202 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.186+0000 7f763e956b80 1 --2- 172.21.15.38:0/210886005 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55ea3ab28340 0x55ea3ab349f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.202 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.186+0000 7f763e956b80 1 -- 172.21.15.38:0/210886005 >> 172.21.15.38:0/210886005 conn(0x55ea3ab22500 msgr2=0x55ea3ab39760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:53.202 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.186+0000 7f763e956b80 1 -- 172.21.15.38:0/210886005 shutdown_connections 2023-03-09T14:45:53.202 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.186+0000 7f763e956b80 1 -- 172.21.15.38:0/210886005 wait complete. 2023-03-09T14:45:53.202 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.186+0000 7f763e956b80 20 client.15348 trim_cache size 0 max 16384 2023-03-09T14:45:53.203 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2023-03-09T14:45:53.203 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_fsid": "c6733fe2-be87-11ed-9af2-001a4aab830c", 2023-03-09T14:45:53.203 INFO:teuthology.orchestra.run.smithi038.stdout: "whoami": 1, 2023-03-09T14:45:53.203 INFO:teuthology.orchestra.run.smithi038.stdout: "id": 24494, 2023-03-09T14:45:53.203 INFO:teuthology.orchestra.run.smithi038.stdout: "want_state": "up:active", 2023-03-09T14:45:53.204 INFO:teuthology.orchestra.run.smithi038.stdout: "state": "up:active", 2023-03-09T14:45:53.204 INFO:teuthology.orchestra.run.smithi038.stdout: "fs_name": "cephfs", 2023-03-09T14:45:53.204 INFO:teuthology.orchestra.run.smithi038.stdout: "rank_uptime": 75.693757265000002, 2023-03-09T14:45:53.204 INFO:teuthology.orchestra.run.smithi038.stdout: "mdsmap_epoch": 23, 2023-03-09T14:45:53.204 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_epoch": 89, 2023-03-09T14:45:53.204 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_epoch_barrier": 87, 2023-03-09T14:45:53.204 INFO:teuthology.orchestra.run.smithi038.stdout: "uptime": 108.78098670999999 2023-03-09T14:45:53.205 INFO:teuthology.orchestra.run.smithi038.stdout:} 2023-03-09T14:45:53.205 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.186+0000 7f76327cd700 1 -- 172.21.15.38:0/3939417108 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f761809f3c0 msgr2=0x7f76180a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:53.205 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.186+0000 7f76327cd700 1 --2- 172.21.15.38:0/3939417108 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f761809f3c0 0x7f76180a1870 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f761400b5c0 tx=0x7f7614005fb0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.206 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.186+0000 7f76327cd700 1 -- 172.21.15.38:0/3939417108 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f762c0fedd0 msgr2=0x7f762c0695c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:53.206 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.186+0000 7f76327cd700 1 --2- 172.21.15.38:0/3939417108 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f762c0fedd0 0x7f762c0695c0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f761c00d980 tx=0x7f761c00dd40 comp rx=0 tx=0).stop 2023-03-09T14:45:53.206 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.186+0000 7f76327cd700 1 -- 172.21.15.38:0/3939417108 shutdown_connections 2023-03-09T14:45:53.206 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.186+0000 7f76327cd700 1 --2- 172.21.15.38:0/3939417108 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f762c0fedd0 0x7f762c0695c0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.206 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.186+0000 7f76327cd700 1 --2- 172.21.15.38:0/3939417108 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f761809f3c0 0x7f76180a1870 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.206 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.186+0000 7f76327cd700 1 --2- 172.21.15.38:0/3939417108 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f762c100950 0x7f762c19cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.206 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.186+0000 7f76327cd700 1 --2- 172.21.15.38:0/3939417108 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f762c0fffc0 0x7f762c19a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.207 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.186+0000 7f76327cd700 1 -- 172.21.15.38:0/3939417108 >> 172.21.15.38:0/3939417108 conn(0x7f762c0fa3d0 msgr2=0x7f762c0fc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:53.207 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.187+0000 7f76327cd700 1 -- 172.21.15.38:0/3939417108 shutdown_connections 2023-03-09T14:45:53.207 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.187+0000 7f76327cd700 1 -- 172.21.15.38:0/3939417108 wait complete. 2023-03-09T14:45:53.207 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:52 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/1128597342' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-03-09T14:45:53.207 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:52 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/3206712195' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-03-09T14:45:53.207 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:52 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/1617115187' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-03-09T14:45:53.208 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-03-09T14:45:53.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:52 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/1128597342' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-03-09T14:45:53.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:52 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/3206712195' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-03-09T14:45:53.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:52 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/1617115187' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-03-09T14:45:53.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:52 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/1128597342' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-03-09T14:45:53.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:52 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/3206712195' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-03-09T14:45:53.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:52 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/1617115187' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-03-09T14:45:53.335 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.332+0000 7fa442d96700 1 -- 172.21.15.38:0/4174255065 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa43c100950 msgr2=0x7fa43c107210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:53.335 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.332+0000 7fa442d96700 1 --2- 172.21.15.38:0/4174255065 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa43c100950 0x7fa43c107210 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fa43400b0a0 tx=0x7fa43400b3b0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.335 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.332+0000 7fa442d96700 1 -- 172.21.15.38:0/4174255065 shutdown_connections 2023-03-09T14:45:53.335 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.332+0000 7fa442d96700 1 --2- 172.21.15.38:0/4174255065 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa43c100950 0x7fa43c107210 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.336 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.332+0000 7fa442d96700 1 --2- 172.21.15.38:0/4174255065 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa43c0fedd0 0x7fa43c0ff1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.336 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.332+0000 7fa442d96700 1 --2- 172.21.15.38:0/4174255065 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa43c0fffc0 0x7fa43c100410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.336 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.332+0000 7fa442d96700 1 -- 172.21.15.38:0/4174255065 >> 172.21.15.38:0/4174255065 conn(0x7fa43c0fa3d0 msgr2=0x7fa43c0fc820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:53.336 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.332+0000 7fa442d96700 1 -- 172.21.15.38:0/4174255065 shutdown_connections 2023-03-09T14:45:53.336 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.332+0000 7fa442d96700 1 -- 172.21.15.38:0/4174255065 wait complete. 2023-03-09T14:45:53.338 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.333+0000 7fa442d96700 1 Processor -- start 2023-03-09T14:45:53.338 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.333+0000 7fa442d96700 1 -- start start 2023-03-09T14:45:53.338 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.333+0000 7fa442d96700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa43c0fedd0 0x7fa43c0695a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:53.338 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.333+0000 7fa442d96700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa43c0fffc0 0x7fa43c19abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:53.338 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.333+0000 7fa442d96700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa43c100950 0x7fa43c19d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:53.338 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.333+0000 7fa442d96700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa43c19da00 con 0x7fa43c0fffc0 2023-03-09T14:45:53.339 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.333+0000 7fa442d96700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa43c19db70 con 0x7fa43c0fedd0 2023-03-09T14:45:53.339 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.333+0000 7fa442d96700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa43c19dd30 con 0x7fa43c100950 2023-03-09T14:45:53.339 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.333+0000 7fa43bfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa43c0fffc0 0x7fa43c19abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:53.339 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.333+0000 7fa43bfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa43c0fffc0 0x7fa43c19abe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38270/0 (socket says 172.21.15.38:38270) 2023-03-09T14:45:53.339 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.333+0000 7fa43bfff700 1 -- 172.21.15.38:0/2151597858 learned_addr learned my addr 172.21.15.38:0/2151597858 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:53.339 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.333+0000 7fa440b32700 1 --2- 172.21.15.38:0/2151597858 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa43c0fedd0 0x7fa43c0695a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:53.339 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.334+0000 7fa441333700 1 --2- 172.21.15.38:0/2151597858 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa43c100950 0x7fa43c19d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:53.340 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.334+0000 7fa43bfff700 1 -- 172.21.15.38:0/2151597858 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa43c100950 msgr2=0x7fa43c19d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:53.340 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.334+0000 7fa43bfff700 1 --2- 172.21.15.38:0/2151597858 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa43c100950 0x7fa43c19d230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.340 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.334+0000 7fa43bfff700 1 -- 172.21.15.38:0/2151597858 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa43c0fedd0 msgr2=0x7fa43c0695a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:53.340 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.334+0000 7fa43bfff700 1 --2- 172.21.15.38:0/2151597858 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa43c0fedd0 0x7fa43c0695a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.340 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.334+0000 7fa43bfff700 1 -- 172.21.15.38:0/2151597858 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa434009d00 con 0x7fa43c0fffc0 2023-03-09T14:45:53.340 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.334+0000 7fa43bfff700 1 --2- 172.21.15.38:0/2151597858 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa43c0fffc0 0x7fa43c19abe0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fa42800d980 tx=0x7fa42800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:53.340 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.334+0000 7fa439ffb700 1 -- 172.21.15.38:0/2151597858 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa428004220 con 0x7fa43c0fffc0 2023-03-09T14:45:53.340 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.334+0000 7fa439ffb700 1 -- 172.21.15.38:0/2151597858 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa428007640 con 0x7fa43c0fffc0 2023-03-09T14:45:53.341 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.334+0000 7fa442d96700 1 -- 172.21.15.38:0/2151597858 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa43c10a790 con 0x7fa43c0fffc0 2023-03-09T14:45:53.341 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.334+0000 7fa439ffb700 1 -- 172.21.15.38:0/2151597858 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa428005610 con 0x7fa43c0fffc0 2023-03-09T14:45:53.341 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.334+0000 7fa442d96700 1 -- 172.21.15.38:0/2151597858 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa43c10a600 con 0x7fa43c0fffc0 2023-03-09T14:45:53.345 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.337+0000 7fa439ffb700 1 -- 172.21.15.38:0/2151597858 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) v1 ==== 133198+0+0 (secure 0 0 0) 0x7fa428021070 con 0x7fa43c0fffc0 2023-03-09T14:45:53.345 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.337+0000 7fa439ffb700 1 --2- 172.21.15.38:0/2151597858 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fa42c09f3c0 0x7fa42c0a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:53.345 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.338+0000 7fa439ffb700 1 -- 172.21.15.38:0/2151597858 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fa4280c9e80 con 0x7fa43c0fffc0 2023-03-09T14:45:53.345 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.338+0000 7fa440b32700 1 --2- 172.21.15.38:0/2151597858 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fa42c09f3c0 0x7fa42c0a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:53.345 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.338+0000 7fa440b32700 1 --2- 172.21.15.38:0/2151597858 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fa42c09f3c0 0x7fa42c0a1870 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fa43c0ffe20 tx=0x7fa430009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:53.345 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.338+0000 7fa44ef1fb80 1 Processor -- start 2023-03-09T14:45:53.346 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.338+0000 7fa44ef1fb80 1 -- start start 2023-03-09T14:45:53.346 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.339+0000 7fa44ef1fb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564413af69f0 0x564413af6e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:53.346 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.339+0000 7fa44ef1fb80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564413af7340 0x564413af7c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:53.346 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.339+0000 7fa44ef1fb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564413af81f0 0x564413af8660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:53.346 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.339+0000 7fa44ef1fb80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644139e0d50 con 0x564413af7340 2023-03-09T14:45:53.346 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.339+0000 7fa44ef1fb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x564413a4f730 con 0x564413af81f0 2023-03-09T14:45:53.346 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.339+0000 7fa44ef1fb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x564413a48c60 con 0x564413af69f0 2023-03-09T14:45:53.346 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.339+0000 7fa43bfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564413af7340 0x564413af7c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:53.347 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.339+0000 7fa43bfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564413af7340 0x564413af7c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38278/0 (socket says 172.21.15.38:38278) 2023-03-09T14:45:53.347 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.339+0000 7fa43bfff700 1 -- 172.21.15.38:0/1978526084 learned_addr learned my addr 172.21.15.38:0/1978526084 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:53.347 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.339+0000 7fa441333700 1 --2- 172.21.15.38:0/1978526084 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564413af81f0 0x564413af8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:53.347 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.339+0000 7fa440b32700 1 --2- 172.21.15.38:0/1978526084 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564413af69f0 0x564413af6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:53.347 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.339+0000 7fa43bfff700 1 -- 172.21.15.38:0/1978526084 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564413af69f0 msgr2=0x564413af6e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:45:53.347 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.339+0000 7fa43bfff700 1 --2- 172.21.15.38:0/1978526084 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564413af69f0 0x564413af6e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.347 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.339+0000 7fa43bfff700 1 -- 172.21.15.38:0/1978526084 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564413af81f0 msgr2=0x564413af8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:45:53.347 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.339+0000 7fa43bfff700 1 --2- 172.21.15.38:0/1978526084 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564413af81f0 0x564413af8660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.348 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.339+0000 7fa43bfff700 1 -- 172.21.15.38:0/1978526084 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564413a0f140 con 0x564413af7340 2023-03-09T14:45:53.348 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.339+0000 7fa43bfff700 1 --2- 172.21.15.38:0/1978526084 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564413af7340 0x564413af7c60 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fa4280058b0 tx=0x7fa42800f440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3eb69b5d927244b server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:53.348 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.339+0000 7fa426ffd700 1 -- 172.21.15.38:0/1978526084 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4280af070 con 0x564413af7340 2023-03-09T14:45:53.348 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.339+0000 7fa426ffd700 1 -- 172.21.15.38:0/1978526084 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa42809fd80 con 0x564413af7340 2023-03-09T14:45:53.348 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.339+0000 7fa426ffd700 1 -- 172.21.15.38:0/1978526084 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa42809fb10 con 0x564413af7340 2023-03-09T14:45:53.348 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.340+0000 7fa44ef1fb80 1 -- 172.21.15.38:0/1978526084 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564413af7340 msgr2=0x564413af7c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:53.348 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.340+0000 7fa44ef1fb80 1 --2- 172.21.15.38:0/1978526084 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564413af7340 0x564413af7c60 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fa4280058b0 tx=0x7fa42800f440 comp rx=0 tx=0).stop 2023-03-09T14:45:53.349 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.340+0000 7fa44ef1fb80 1 -- 172.21.15.38:0/1978526084 shutdown_connections 2023-03-09T14:45:53.349 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.340+0000 7fa44ef1fb80 1 --2- 172.21.15.38:0/1978526084 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564413af81f0 0x564413af8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.349 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.340+0000 7fa44ef1fb80 1 --2- 172.21.15.38:0/1978526084 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564413af69f0 0x564413af6e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.349 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.340+0000 7fa44ef1fb80 1 --2- 172.21.15.38:0/1978526084 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564413af7340 0x564413af7c60 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.349 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.340+0000 7fa44ef1fb80 1 -- 172.21.15.38:0/1978526084 >> 172.21.15.38:0/1978526084 conn(0x564413af1500 msgr2=0x564413af18f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:53.349 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.340+0000 7fa44ef1fb80 1 -- 172.21.15.38:0/1978526084 shutdown_connections 2023-03-09T14:45:53.349 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.340+0000 7fa44ef1fb80 1 -- 172.21.15.38:0/1978526084 wait complete. 2023-03-09T14:45:53.350 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.340+0000 7fa44ef1fb80 1 -- start start 2023-03-09T14:45:53.350 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.340+0000 7fa44ef1fb80 1 Processor -- start 2023-03-09T14:45:53.350 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.341+0000 7fa44ef1fb80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564413af69f0 0x564413b00700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:53.350 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.341+0000 7fa44ef1fb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564413af7340 0x564413b00c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:53.350 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.341+0000 7fa44ef1fb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564413af81f0 0x564413b07c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:53.350 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.341+0000 7fa44ef1fb80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644138cc8d0 con 0x564413af69f0 2023-03-09T14:45:53.350 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.341+0000 7fa44ef1fb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x564413b081c0 con 0x564413af81f0 2023-03-09T14:45:53.350 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.341+0000 7fa44ef1fb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x564413b08350 con 0x564413af7340 2023-03-09T14:45:53.351 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.341+0000 7fa44ef1fb80 10 client.0 _ll_register_callbacks cb 0x5644139ee4d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:45:53.351 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.341+0000 7fa440b32700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564413af69f0 0x564413b00700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:53.351 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.341+0000 7fa440b32700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564413af69f0 0x564413b00700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38282/0 (socket says 172.21.15.38:38282) 2023-03-09T14:45:53.351 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.341+0000 7fa440b32700 1 -- 172.21.15.38:0/3995261970 learned_addr learned my addr 172.21.15.38:0/3995261970 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:53.351 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.341+0000 7fa43bfff700 1 --2- 172.21.15.38:0/3995261970 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564413af7340 0x564413b00c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:53.351 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.341+0000 7fa441333700 1 --2- 172.21.15.38:0/3995261970 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564413af81f0 0x564413b07c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:53.351 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.341+0000 7fa440b32700 1 -- 172.21.15.38:0/3995261970 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564413af7340 msgr2=0x564413b00c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:45:53.352 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.341+0000 7fa440b32700 1 --2- 172.21.15.38:0/3995261970 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564413af7340 0x564413b00c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.352 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.341+0000 7fa440b32700 1 -- 172.21.15.38:0/3995261970 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564413af81f0 msgr2=0x564413b07c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:45:53.352 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.341+0000 7fa440b32700 1 --2- 172.21.15.38:0/3995261970 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564413af81f0 0x564413b07c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.352 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.341+0000 7fa440b32700 1 -- 172.21.15.38:0/3995261970 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa43c10a600 con 0x564413af69f0 2023-03-09T14:45:53.352 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.342+0000 7fa440b32700 1 --2- 172.21.15.38:0/3995261970 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564413af69f0 0x564413b00700 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fa43001d8c0 tx=0x7fa4300047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=464d9c208955119b server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:53.352 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.342+0000 7fa41bfff700 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2023-03-09T14:45:53.352 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.342+0000 7fa41bfff700 1 -- 172.21.15.38:0/3995261970 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa430022070 con 0x564413af69f0 2023-03-09T14:45:53.352 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.342+0000 7fa41bfff700 1 -- 172.21.15.38:0/3995261970 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fa43001c690 con 0x564413af69f0 2023-03-09T14:45:53.353 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.342+0000 7fa44ef1fb80 1 -- 172.21.15.38:0/3995261970 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564413a16140 con 0x564413af69f0 2023-03-09T14:45:53.353 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.342+0000 7fa41bfff700 1 -- 172.21.15.38:0/3995261970 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa430004020 con 0x564413af69f0 2023-03-09T14:45:53.353 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.342+0000 7fa44ef1fb80 10 client.15360 fetch_fsmap learned FSMap version 23 2023-03-09T14:45:53.353 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.342+0000 7fa44ef1fb80 1 -- 172.21.15.38:0/3995261970 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x56441385fc10 con 0x564413af69f0 2023-03-09T14:45:53.353 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.342+0000 7fa41bfff700 1 -- 172.21.15.38:0/3995261970 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x564413a16140 con 0x564413af69f0 2023-03-09T14:45:53.353 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.342+0000 7fa41bfff700 1 -- 172.21.15.38:0/3995261970 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fa43002e070 con 0x564413af69f0 2023-03-09T14:45:53.353 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.342+0000 7fa44ef1fb80 10 client.15360 fetch_fsmap finished waiting for FSMap version 23 2023-03-09T14:45:53.353 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.342+0000 7fa44ef1fb80 10 client.15360 resolve_mds: resolved name 'b' to daemon mds.b 2023-03-09T14:45:53.354 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.342+0000 7fa44ef1fb80 20 client.15360 populate_metadata read hostname 'smithi038' 2023-03-09T14:45:53.354 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.342+0000 7fa44ef1fb80 1 --2- 172.21.15.38:0/3995261970 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x564413b0bc80 0x564413b0b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:53.354 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.342+0000 7fa44ef1fb80 4 client.15360 mds_command: new command op to 24530 tid=0[{"prefix": "get_command_descriptions"}] 2023-03-09T14:45:53.354 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.342+0000 7fa44ef1fb80 1 -- 172.21.15.38:0/3995261970 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5644138ccf10 con 0x564413b0bc80 2023-03-09T14:45:53.354 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.343+0000 7fa43bfff700 1 --2- 172.21.15.38:0/3995261970 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x564413b0bc80 0x564413b0b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:53.355 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.343+0000 7fa43bfff700 1 --2- 172.21.15.38:0/3995261970 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x564413b0bc80 0x564413b0b0b0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f7f0106603ccbff1 server_cookie=c8edc17d3ec043b9 in_seq=0 out_seq=0 2023-03-09T14:45:53.355 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.343+0000 7fa41bfff700 10 client.15360 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:45:53.355 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.344+0000 7fa41bfff700 1 -- 172.21.15.38:0/3995261970 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18278 (crc 0 0 0) 0x5644138ccf10 con 0x564413b0bc80 2023-03-09T14:45:53.356 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.344+0000 7fa41bfff700 10 client.15360 handle_command_reply: tid=0 2023-03-09T14:45:53.356 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.344+0000 7fa44ef1fb80 1 client.15360 shutdown 2023-03-09T14:45:53.356 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.344+0000 7fa44ef1fb80 1 -- 172.21.15.38:0/3995261970 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564413af69f0 msgr2=0x564413b00700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:53.356 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.344+0000 7fa44ef1fb80 1 --2- 172.21.15.38:0/3995261970 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564413af69f0 0x564413b00700 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fa43001d8c0 tx=0x7fa4300047c0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.356 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.344+0000 7fa44ef1fb80 1 -- 172.21.15.38:0/3995261970 shutdown_connections 2023-03-09T14:45:53.356 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.344+0000 7fa44ef1fb80 1 --2- 172.21.15.38:0/3995261970 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x564413b0bc80 0x564413b0b0b0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.356 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.344+0000 7fa44ef1fb80 1 --2- 172.21.15.38:0/3995261970 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564413af69f0 0x564413b00700 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.357 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.344+0000 7fa41bfff700 0 client.15360 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:45:53.357 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.344+0000 7fa44ef1fb80 1 --2- 172.21.15.38:0/3995261970 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564413af81f0 0x564413b07c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.357 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.344+0000 7fa44ef1fb80 1 --2- 172.21.15.38:0/3995261970 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564413af7340 0x564413b00c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.357 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.344+0000 7fa44ef1fb80 1 -- 172.21.15.38:0/3995261970 >> 172.21.15.38:0/3995261970 conn(0x564413af1500 msgr2=0x564413af0bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:53.357 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.344+0000 7fa44ef1fb80 1 -- 172.21.15.38:0/3995261970 shutdown_connections 2023-03-09T14:45:53.357 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.344+0000 7fa44ef1fb80 1 -- 172.21.15.38:0/3995261970 wait complete. 2023-03-09T14:45:53.357 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.344+0000 7fa44ef1fb80 20 client.15360 trim_cache size 0 max 16384 2023-03-09T14:45:53.365 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.362+0000 7fa44ef1fb80 1 Processor -- start 2023-03-09T14:45:53.365 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.362+0000 7fa44ef1fb80 1 -- start start 2023-03-09T14:45:53.365 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.362+0000 7fa44ef1fb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564413af69f0 0x564413b08730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:53.366 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.362+0000 7fa44ef1fb80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564413af7340 0x564413b07c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:53.366 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.362+0000 7fa44ef1fb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564413af81f0 0x564413b0b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:53.366 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.362+0000 7fa44ef1fb80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x564413b002d0 con 0x564413af7340 2023-03-09T14:45:53.366 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.362+0000 7fa44ef1fb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x564413af6690 con 0x564413af69f0 2023-03-09T14:45:53.366 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.362+0000 7fa44ef1fb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x564413b018f0 con 0x564413af81f0 2023-03-09T14:45:53.366 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.362+0000 7fa43bfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564413af7340 0x564413b07c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:53.366 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.363+0000 7fa43bfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564413af7340 0x564413b07c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38292/0 (socket says 172.21.15.38:38292) 2023-03-09T14:45:53.367 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.363+0000 7fa43bfff700 1 -- 172.21.15.38:0/2304232532 learned_addr learned my addr 172.21.15.38:0/2304232532 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:53.367 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.363+0000 7fa440b32700 1 --2- 172.21.15.38:0/2304232532 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564413af69f0 0x564413b08730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:53.367 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.363+0000 7fa43bfff700 1 -- 172.21.15.38:0/2304232532 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564413af81f0 msgr2=0x564413b0b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:45:53.367 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.363+0000 7fa43bfff700 1 --2- 172.21.15.38:0/2304232532 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564413af81f0 0x564413b0b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.367 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.363+0000 7fa43bfff700 1 -- 172.21.15.38:0/2304232532 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564413af69f0 msgr2=0x564413b08730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:45:53.367 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.363+0000 7fa43bfff700 1 --2- 172.21.15.38:0/2304232532 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564413af69f0 0x564413b08730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.367 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.363+0000 7fa43bfff700 1 -- 172.21.15.38:0/2304232532 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564413876d90 con 0x564413af7340 2023-03-09T14:45:53.367 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.363+0000 7fa43bfff700 1 --2- 172.21.15.38:0/2304232532 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564413af7340 0x564413b07c10 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fa4280c7930 tx=0x7fa4280c9040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7893818155ea77dc server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:53.368 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.363+0000 7fa4277fe700 1 -- 172.21.15.38:0/2304232532 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4280af040 con 0x564413af7340 2023-03-09T14:45:53.368 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.363+0000 7fa4277fe700 1 -- 172.21.15.38:0/2304232532 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa4280c5070 con 0x564413af7340 2023-03-09T14:45:53.368 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.363+0000 7fa4277fe700 1 -- 172.21.15.38:0/2304232532 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa42809fb50 con 0x564413af7340 2023-03-09T14:45:53.371 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.364+0000 7fa44ef1fb80 1 -- 172.21.15.38:0/2304232532 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564413af7340 msgr2=0x564413b07c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:53.371 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.364+0000 7fa44ef1fb80 1 --2- 172.21.15.38:0/2304232532 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564413af7340 0x564413b07c10 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fa4280c7930 tx=0x7fa4280c9040 comp rx=0 tx=0).stop 2023-03-09T14:45:53.371 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.364+0000 7fa44ef1fb80 1 -- 172.21.15.38:0/2304232532 shutdown_connections 2023-03-09T14:45:53.371 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.364+0000 7fa44ef1fb80 1 --2- 172.21.15.38:0/2304232532 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564413af81f0 0x564413b0b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.371 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.364+0000 7fa44ef1fb80 1 --2- 172.21.15.38:0/2304232532 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564413af69f0 0x564413b08730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.372 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.364+0000 7fa44ef1fb80 1 --2- 172.21.15.38:0/2304232532 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564413af7340 0x564413b07c10 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.372 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.364+0000 7fa44ef1fb80 1 -- 172.21.15.38:0/2304232532 >> 172.21.15.38:0/2304232532 conn(0x564413af1500 msgr2=0x564413aff440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:53.372 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.364+0000 7fa44ef1fb80 1 -- 172.21.15.38:0/2304232532 shutdown_connections 2023-03-09T14:45:53.372 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.364+0000 7fa44ef1fb80 1 -- 172.21.15.38:0/2304232532 wait complete. 2023-03-09T14:45:53.372 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.364+0000 7fa44ef1fb80 1 -- start start 2023-03-09T14:45:53.372 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.364+0000 7fa44ef1fb80 1 Processor -- start 2023-03-09T14:45:53.372 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.364+0000 7fa44ef1fb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564413af69f0 0x564413b02e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:53.373 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.364+0000 7fa44ef1fb80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564413af7340 0x564413b033c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:53.373 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.364+0000 7fa44ef1fb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564413af81f0 0x564413b5fae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:53.373 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.364+0000 7fa44ef1fb80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x564413b07cf0 con 0x564413af7340 2023-03-09T14:45:53.373 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.364+0000 7fa44ef1fb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x564413b07e30 con 0x564413af81f0 2023-03-09T14:45:53.373 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.365+0000 7fa44ef1fb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x564413b07ff0 con 0x564413af69f0 2023-03-09T14:45:53.373 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.365+0000 7fa44ef1fb80 10 client.0 _ll_register_callbacks cb 0x5644139ee4d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:45:53.373 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.365+0000 7fa43bfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564413af7340 0x564413b033c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:53.374 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.365+0000 7fa43bfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564413af7340 0x564413b033c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38298/0 (socket says 172.21.15.38:38298) 2023-03-09T14:45:53.374 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.365+0000 7fa43bfff700 1 -- 172.21.15.38:0/521680844 learned_addr learned my addr 172.21.15.38:0/521680844 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:53.374 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.365+0000 7fa440b32700 1 --2- 172.21.15.38:0/521680844 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564413af69f0 0x564413b02e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:53.374 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.365+0000 7fa441333700 1 --2- 172.21.15.38:0/521680844 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564413af81f0 0x564413b5fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:53.374 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.365+0000 7fa43bfff700 1 -- 172.21.15.38:0/521680844 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564413af69f0 msgr2=0x564413b02e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:45:53.374 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.365+0000 7fa43bfff700 1 --2- 172.21.15.38:0/521680844 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564413af69f0 0x564413b02e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.374 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.365+0000 7fa43bfff700 1 -- 172.21.15.38:0/521680844 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564413af81f0 msgr2=0x564413b5fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:45:53.374 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.365+0000 7fa43bfff700 1 --2- 172.21.15.38:0/521680844 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564413af81f0 0x564413b5fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.375 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.365+0000 7fa43bfff700 1 -- 172.21.15.38:0/521680844 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564413a0f140 con 0x564413af7340 2023-03-09T14:45:53.375 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.365+0000 7fa441333700 1 --2- 172.21.15.38:0/521680844 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564413af81f0 0x564413b5fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-09T14:45:53.375 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.365+0000 7fa43bfff700 1 --2- 172.21.15.38:0/521680844 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564413af7340 0x564413b033c0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fa42800b540 tx=0x7fa4280c7860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23065075db0009b0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:53.375 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.365+0000 7fa41bfff700 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2023-03-09T14:45:53.375 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.365+0000 7fa41bfff700 1 -- 172.21.15.38:0/521680844 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4280af040 con 0x564413af7340 2023-03-09T14:45:53.375 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.365+0000 7fa41bfff700 1 -- 172.21.15.38:0/521680844 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fa42809f8f0 con 0x564413af7340 2023-03-09T14:45:53.375 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.365+0000 7fa44ef1fb80 1 -- 172.21.15.38:0/521680844 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564413a06160 con 0x564413af7340 2023-03-09T14:45:53.375 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.365+0000 7fa41bfff700 1 -- 172.21.15.38:0/521680844 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa428097280 con 0x564413af7340 2023-03-09T14:45:53.376 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.365+0000 7fa44ef1fb80 10 client.15366 fetch_fsmap learned FSMap version 23 2023-03-09T14:45:53.376 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.365+0000 7fa44ef1fb80 1 -- 172.21.15.38:0/521680844 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x564413875260 con 0x564413af7340 2023-03-09T14:45:53.376 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.366+0000 7fa41bfff700 1 -- 172.21.15.38:0/521680844 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x564413a06160 con 0x564413af7340 2023-03-09T14:45:53.376 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.366+0000 7fa41bfff700 1 -- 172.21.15.38:0/521680844 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fa428096af0 con 0x564413af7340 2023-03-09T14:45:53.376 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.366+0000 7fa44ef1fb80 10 client.15366 fetch_fsmap finished waiting for FSMap version 23 2023-03-09T14:45:53.376 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.366+0000 7fa44ef1fb80 10 client.15366 resolve_mds: resolved name 'b' to daemon mds.b 2023-03-09T14:45:53.376 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.366+0000 7fa44ef1fb80 20 client.15366 populate_metadata read hostname 'smithi038' 2023-03-09T14:45:53.377 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.366+0000 7fa44ef1fb80 1 --2- 172.21.15.38:0/521680844 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x564413b646b0 0x564413b610a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:53.377 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.366+0000 7fa44ef1fb80 4 client.15366 mds_command: new command op to 24530 tid=0[{"prefix": "status"}] 2023-03-09T14:45:53.377 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.366+0000 7fa44ef1fb80 1 -- 172.21.15.38:0/521680844 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5644139f0170 con 0x564413b646b0 2023-03-09T14:45:53.377 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.366+0000 7fa440b32700 1 --2- 172.21.15.38:0/521680844 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x564413b646b0 0x564413b610a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:53.377 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.366+0000 7fa41bfff700 10 client.15366 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:45:53.377 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.366+0000 7fa440b32700 1 --2- 172.21.15.38:0/521680844 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x564413b646b0 0x564413b610a0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2014fc76834241f6 server_cookie=3b4d03920d177e54 in_seq=0 out_seq=0 2023-03-09T14:45:53.377 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.366+0000 7fa41bfff700 1 -- 172.21.15.38:0/521680844 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x5644139f0170 con 0x564413b646b0 2023-03-09T14:45:53.377 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.367+0000 7fa41bfff700 10 client.15366 handle_command_reply: tid=0 2023-03-09T14:45:53.378 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.367+0000 7fa44ef1fb80 1 client.15366 shutdown 2023-03-09T14:45:53.378 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.367+0000 7fa44ef1fb80 1 -- 172.21.15.38:0/521680844 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564413af7340 msgr2=0x564413b033c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:53.378 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.367+0000 7fa44ef1fb80 1 --2- 172.21.15.38:0/521680844 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564413af7340 0x564413b033c0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fa42800b540 tx=0x7fa4280c7860 comp rx=0 tx=0).stop 2023-03-09T14:45:53.378 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.367+0000 7fa44ef1fb80 1 -- 172.21.15.38:0/521680844 shutdown_connections 2023-03-09T14:45:53.378 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.367+0000 7fa44ef1fb80 1 --2- 172.21.15.38:0/521680844 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x564413b646b0 0x564413b610a0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.378 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.367+0000 7fa44ef1fb80 1 --2- 172.21.15.38:0/521680844 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564413af81f0 0x564413b5fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.378 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.367+0000 7fa44ef1fb80 1 --2- 172.21.15.38:0/521680844 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564413af69f0 0x564413b02e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.378 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.367+0000 7fa41bfff700 0 client.15366 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:45:53.379 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.367+0000 7fa44ef1fb80 1 --2- 172.21.15.38:0/521680844 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564413af7340 0x564413b033c0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.379 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.367+0000 7fa44ef1fb80 1 -- 172.21.15.38:0/521680844 >> 172.21.15.38:0/521680844 conn(0x564413af1500 msgr2=0x564413b08730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:53.379 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.367+0000 7fa44ef1fb80 1 -- 172.21.15.38:0/521680844 shutdown_connections 2023-03-09T14:45:53.379 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.367+0000 7fa44ef1fb80 1 -- 172.21.15.38:0/521680844 wait complete. 2023-03-09T14:45:53.379 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.367+0000 7fa44ef1fb80 20 client.15366 trim_cache size 0 max 16384 2023-03-09T14:45:53.379 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2023-03-09T14:45:53.380 INFO:teuthology.orchestra.run.smithi038.stdout: "cluster_fsid": "c6733fe2-be87-11ed-9af2-001a4aab830c", 2023-03-09T14:45:53.380 INFO:teuthology.orchestra.run.smithi038.stdout: "whoami": 0, 2023-03-09T14:45:53.380 INFO:teuthology.orchestra.run.smithi038.stdout: "id": 24530, 2023-03-09T14:45:53.380 INFO:teuthology.orchestra.run.smithi038.stdout: "want_state": "up:active", 2023-03-09T14:45:53.380 INFO:teuthology.orchestra.run.smithi038.stdout: "state": "up:active", 2023-03-09T14:45:53.381 INFO:teuthology.orchestra.run.smithi038.stdout: "fs_name": "cephfs", 2023-03-09T14:45:53.381 INFO:teuthology.orchestra.run.smithi038.stdout: "rank_uptime": 94.279396937000001, 2023-03-09T14:45:53.381 INFO:teuthology.orchestra.run.smithi038.stdout: "mdsmap_epoch": 23, 2023-03-09T14:45:53.381 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_epoch": 89, 2023-03-09T14:45:53.381 INFO:teuthology.orchestra.run.smithi038.stdout: "osdmap_epoch_barrier": 87, 2023-03-09T14:45:53.381 INFO:teuthology.orchestra.run.smithi038.stdout: "uptime": 99.787152481999996 2023-03-09T14:45:53.381 INFO:teuthology.orchestra.run.smithi038.stdout:} 2023-03-09T14:45:53.382 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.367+0000 7fa442d96700 1 -- 172.21.15.38:0/2151597858 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fa42c09f3c0 msgr2=0x7fa42c0a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:53.383 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.367+0000 7fa442d96700 1 --2- 172.21.15.38:0/2151597858 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fa42c09f3c0 0x7fa42c0a1870 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fa43c0ffe20 tx=0x7fa430009260 comp rx=0 tx=0).stop 2023-03-09T14:45:53.383 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.367+0000 7fa442d96700 1 -- 172.21.15.38:0/2151597858 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa43c0fffc0 msgr2=0x7fa43c19abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:53.383 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.367+0000 7fa442d96700 1 --2- 172.21.15.38:0/2151597858 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa43c0fffc0 0x7fa43c19abe0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fa42800d980 tx=0x7fa42800dc90 comp rx=0 tx=0).stop 2023-03-09T14:45:53.383 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.367+0000 7fa442d96700 1 -- 172.21.15.38:0/2151597858 shutdown_connections 2023-03-09T14:45:53.383 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.367+0000 7fa442d96700 1 --2- 172.21.15.38:0/2151597858 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fa42c09f3c0 0x7fa42c0a1870 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.383 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.367+0000 7fa442d96700 1 --2- 172.21.15.38:0/2151597858 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa43c100950 0x7fa43c19d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.383 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.367+0000 7fa442d96700 1 --2- 172.21.15.38:0/2151597858 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa43c0fedd0 0x7fa43c0695a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.384 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.367+0000 7fa442d96700 1 --2- 172.21.15.38:0/2151597858 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa43c0fffc0 0x7fa43c19abe0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:53.384 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.367+0000 7fa442d96700 1 -- 172.21.15.38:0/2151597858 >> 172.21.15.38:0/2151597858 conn(0x7fa43c0fa3d0 msgr2=0x7fa43c1031f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:53.384 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.368+0000 7fa442d96700 1 -- 172.21.15.38:0/2151597858 shutdown_connections 2023-03-09T14:45:53.384 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:53.368+0000 7fa442d96700 1 -- 172.21.15.38:0/2151597858 wait complete. 2023-03-09T14:45:53.384 INFO:teuthology.run_tasks:Running task kclient... 2023-03-09T14:45:53.401 INFO:tasks.kclient:Mounting kernel clients... 2023-03-09T14:45:53.402 INFO:tasks.kclient:config is {'client.0': {}} 2023-03-09T14:45:53.402 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-09T14:45:53.402 DEBUG:teuthology.orchestra.run.smithi038:> ip netns list 2023-03-09T14:45:53.419 INFO:teuthology.orchestra.run.smithi038.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2023-03-09T14:45:53.420 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-09T14:45:53.420 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-03-09T14:45:53.500 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-09T14:45:53.501 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip link delete ceph-brx 2023-03-09T14:45:53.602 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=legacy', 'wsync', 'context=system_u:object_r:postgresql_db_t:s0'], 'syntax': 'v1', 'mount_subvol_num': 1, 'mountpoint': '/tmp/cephfs'} 2023-03-09T14:45:53.603 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-03-09T14:45:53.603 INFO:tasks.cephfs.mount:using client_config["hostfs_mntpt"] = /tmp/cephfs 2023-03-09T14:45:53.604 INFO:tasks.cephfs.mount:hostfs_mntpt = /tmp/cephfs 2023-03-09T14:45:53.604 DEBUG:teuthology.orchestra.run.smithi038:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:e0ab30a2acead6db62b144ba6b6aeb23635adae7 shell --fsid c6733fe2-be87-11ed-9af2-001a4aab830c -- ceph fs ls 2023-03-09T14:45:54.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:53 smithi122 ceph-mon[93869]: pgmap v308: 129 pgs: 129 active+clean; 636 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-09T14:45:54.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:53 smithi181 ceph-mon[94189]: pgmap v308: 129 pgs: 129 active+clean; 636 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-09T14:45:54.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:53 smithi038 ceph-mon[92422]: pgmap v308: 129 pgs: 129 active+clean; 636 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-09T14:45:54.258 INFO:teuthology.orchestra.run.smithi038.stderr:Inferring config /var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/mon.a/config 2023-03-09T14:45:55.471 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:55.467+0000 7f69907de700 1 -- 172.21.15.38:0/4076428876 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6988104160 msgr2=0x7f698810bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:55.471 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:55.467+0000 7f69907de700 1 --2- 172.21.15.38:0/4076428876 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6988104160 0x7f698810bd30 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f698000b0d0 tx=0x7f698000b3e0 comp rx=0 tx=0).stop 2023-03-09T14:45:55.471 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:55.467+0000 7f69907de700 1 -- 172.21.15.38:0/4076428876 shutdown_connections 2023-03-09T14:45:55.471 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:55.467+0000 7f69907de700 1 --2- 172.21.15.38:0/4076428876 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6988104160 0x7f698810bd30 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:55.471 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:55.467+0000 7f69907de700 1 --2- 172.21.15.38:0/4076428876 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69880ff120 0x7f69880ff4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:55.472 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:55.467+0000 7f69907de700 1 --2- 172.21.15.38:0/4076428876 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69880ffac0 0x7f6988103a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:55.472 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:55.467+0000 7f69907de700 1 -- 172.21.15.38:0/4076428876 >> 172.21.15.38:0/4076428876 conn(0x7f6988076d10 msgr2=0x7f6988077110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:55.472 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:55.467+0000 7f69907de700 1 -- 172.21.15.38:0/4076428876 shutdown_connections 2023-03-09T14:45:55.472 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:55.467+0000 7f69907de700 1 -- 172.21.15.38:0/4076428876 wait complete. 2023-03-09T14:45:55.472 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:55.468+0000 7f69907de700 1 Processor -- start 2023-03-09T14:45:55.472 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:55.468+0000 7f69907de700 1 -- start start 2023-03-09T14:45:55.472 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:55.468+0000 7f69907de700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69880ff120 0x7f6988196fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:55.473 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:55.468+0000 7f69907de700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69880ffac0 0x7f6988197510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:55.473 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:55.468+0000 7f69907de700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6988104160 0x7f69881a2050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:55.473 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:55.468+0000 7f69907de700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69881a2590 con 0x7f69880ffac0 2023-03-09T14:45:55.473 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:55.468+0000 7f69907de700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69881a26d0 con 0x7f6988104160 2023-03-09T14:45:55.473 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:55.468+0000 7f69907de700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69881a2810 con 0x7f69880ff120 2023-03-09T14:45:55.473 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:55.468+0000 7f698dd79700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69880ffac0 0x7f6988197510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:55.473 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:55.468+0000 7f698dd79700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69880ffac0 0x7f6988197510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38312/0 (socket says 172.21.15.38:38312) 2023-03-09T14:45:55.474 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:55.468+0000 7f698dd79700 1 -- 172.21.15.38:0/2599044859 learned_addr learned my addr 172.21.15.38:0/2599044859 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:55.474 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:55.468+0000 7f698e57a700 1 --2- 172.21.15.38:0/2599044859 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69880ff120 0x7f6988196fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:55.474 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:55.468+0000 7f698ed7b700 1 --2- 172.21.15.38:0/2599044859 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6988104160 0x7f69881a2050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:55.474 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:55.469+0000 7f698dd79700 1 -- 172.21.15.38:0/2599044859 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69880ff120 msgr2=0x7f6988196fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:55.474 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:55.469+0000 7f698dd79700 1 --2- 172.21.15.38:0/2599044859 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69880ff120 0x7f6988196fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:55.474 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:55.469+0000 7f698dd79700 1 -- 172.21.15.38:0/2599044859 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6988104160 msgr2=0x7f69881a2050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:55.474 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:55.469+0000 7f698dd79700 1 --2- 172.21.15.38:0/2599044859 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6988104160 0x7f69881a2050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:55.474 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:55.469+0000 7f698dd79700 1 -- 172.21.15.38:0/2599044859 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6980009d00 con 0x7f69880ffac0 2023-03-09T14:45:55.475 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:55.469+0000 7f698ed7b700 1 --2- 172.21.15.38:0/2599044859 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6988104160 0x7f69881a2050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:45:55.475 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:55.469+0000 7f698dd79700 1 --2- 172.21.15.38:0/2599044859 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69880ffac0 0x7f6988197510 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f698400d950 tx=0x7f698400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:55.476 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:55.469+0000 7f697f7fe700 1 -- 172.21.15.38:0/2599044859 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6984009930 con 0x7f69880ffac0 2023-03-09T14:45:55.476 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:55.469+0000 7f697f7fe700 1 -- 172.21.15.38:0/2599044859 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6984010460 con 0x7f69880ffac0 2023-03-09T14:45:55.476 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:55.469+0000 7f69907de700 1 -- 172.21.15.38:0/2599044859 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f698810ef20 con 0x7f69880ffac0 2023-03-09T14:45:55.476 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:55.469+0000 7f697f7fe700 1 -- 172.21.15.38:0/2599044859 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f698400f620 con 0x7f69880ffac0 2023-03-09T14:45:55.476 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:55.469+0000 7f69907de700 1 -- 172.21.15.38:0/2599044859 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f698810ed90 con 0x7f69880ffac0 2023-03-09T14:45:55.477 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:55.470+0000 7f69907de700 1 -- 172.21.15.38:0/2599044859 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f698810f0b0 con 0x7f69880ffac0 2023-03-09T14:45:55.477 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:55.471+0000 7f697f7fe700 1 -- 172.21.15.38:0/2599044859 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) v1 ==== 133198+0+0 (secure 0 0 0) 0x7f6984023070 con 0x7f69880ffac0 2023-03-09T14:45:55.477 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:55.471+0000 7f697f7fe700 1 --2- 172.21.15.38:0/2599044859 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f697409f3c0 0x7f69740a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:55.477 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:55.472+0000 7f698e57a700 1 --2- 172.21.15.38:0/2599044859 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f697409f3c0 0x7f69740a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:55.477 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:55.472+0000 7f697f7fe700 1 -- 172.21.15.38:0/2599044859 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f69840c9a90 con 0x7f69880ffac0 2023-03-09T14:45:55.478 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:55.472+0000 7f698e57a700 1 --2- 172.21.15.38:0/2599044859 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f697409f3c0 0x7f69740a1870 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f6978005950 tx=0x7f697800b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:55.478 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:55.475+0000 7f697f7fe700 1 -- 172.21.15.38:0/2599044859 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f6984093b40 con 0x7f69880ffac0 2023-03-09T14:45:55.721 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:55.719+0000 7f69907de700 1 -- 172.21.15.38:0/2599044859 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f6988066b10 con 0x7f69880ffac0 2023-03-09T14:45:55.721 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:55.719+0000 7f697f7fe700 1 -- 172.21.15.38:0/2599044859 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v23) v1 ==== 53+0+88 (secure 0 0 0) 0x7f6984021020 con 0x7f69880ffac0 2023-03-09T14:45:55.722 INFO:teuthology.orchestra.run.smithi038.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-03-09T14:45:55.723 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:55.721+0000 7f69907de700 1 -- 172.21.15.38:0/2599044859 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f697409f3c0 msgr2=0x7f69740a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:55.723 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:55.721+0000 7f69907de700 1 --2- 172.21.15.38:0/2599044859 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f697409f3c0 0x7f69740a1870 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f6978005950 tx=0x7f697800b410 comp rx=0 tx=0).stop 2023-03-09T14:45:55.724 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:55.721+0000 7f69907de700 1 -- 172.21.15.38:0/2599044859 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69880ffac0 msgr2=0x7f6988197510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:55.724 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:55.721+0000 7f69907de700 1 --2- 172.21.15.38:0/2599044859 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69880ffac0 0x7f6988197510 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f698400d950 tx=0x7f698400dc60 comp rx=0 tx=0).stop 2023-03-09T14:45:55.724 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:55.721+0000 7f69907de700 1 -- 172.21.15.38:0/2599044859 shutdown_connections 2023-03-09T14:45:55.724 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:55.721+0000 7f69907de700 1 --2- 172.21.15.38:0/2599044859 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f697409f3c0 0x7f69740a1870 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:55.724 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:55.721+0000 7f69907de700 1 --2- 172.21.15.38:0/2599044859 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6988104160 0x7f69881a2050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:55.724 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:55.721+0000 7f69907de700 1 --2- 172.21.15.38:0/2599044859 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69880ff120 0x7f6988196fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:55.724 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:55.721+0000 7f69907de700 1 --2- 172.21.15.38:0/2599044859 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69880ffac0 0x7f6988197510 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:55.725 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:55.721+0000 7f69907de700 1 -- 172.21.15.38:0/2599044859 >> 172.21.15.38:0/2599044859 conn(0x7f6988076d10 msgr2=0x7f69880fc870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:55.725 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:55.721+0000 7f69907de700 1 -- 172.21.15.38:0/2599044859 shutdown_connections 2023-03-09T14:45:55.725 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:55.721+0000 7f69907de700 1 -- 172.21.15.38:0/2599044859 wait complete. 2023-03-09T14:45:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:55 smithi038 ceph-mon[92422]: pgmap v309: 129 pgs: 129 active+clean; 636 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-09T14:45:56.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:55 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/2599044859' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-03-09T14:45:56.118 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-03-09T14:45:56.118 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-03-09T14:45:56.119 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi038.front.sepia.ceph.com 2023-03-09T14:45:56.119 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-03-09T14:45:56.119 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /tmp/cephfs 2023-03-09T14:45:56.119 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-03-09T14:45:56.119 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-03-09T14:45:56.120 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-03-09T14:45:56.120 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/tmp/cephfs' 2023-03-09T14:45:56.120 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-09T14:45:56.120 DEBUG:teuthology.orchestra.run.smithi038:> ip addr 2023-03-09T14:45:56.139 INFO:teuthology.orchestra.run.smithi038.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-03-09T14:45:56.140 INFO:teuthology.orchestra.run.smithi038.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-03-09T14:45:56.140 INFO:teuthology.orchestra.run.smithi038.stdout: inet 127.0.0.1/8 scope host lo 2023-03-09T14:45:56.140 INFO:teuthology.orchestra.run.smithi038.stdout: valid_lft forever preferred_lft forever 2023-03-09T14:45:56.140 INFO:teuthology.orchestra.run.smithi038.stdout: inet6 ::1/128 scope host 2023-03-09T14:45:56.140 INFO:teuthology.orchestra.run.smithi038.stdout: valid_lft forever preferred_lft forever 2023-03-09T14:45:56.140 INFO:teuthology.orchestra.run.smithi038.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-03-09T14:45:56.140 INFO:teuthology.orchestra.run.smithi038.stdout: link/ether 0c:c4:7a:6b:fd:a2 brd ff:ff:ff:ff:ff:ff 2023-03-09T14:45:56.141 INFO:teuthology.orchestra.run.smithi038.stdout:3: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-03-09T14:45:56.141 INFO:teuthology.orchestra.run.smithi038.stdout: link/ether 0c:c4:7a:bd:15:fc brd ff:ff:ff:ff:ff:ff 2023-03-09T14:45:56.141 INFO:teuthology.orchestra.run.smithi038.stdout: inet 172.21.15.38/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-03-09T14:45:56.141 INFO:teuthology.orchestra.run.smithi038.stdout: valid_lft 41958sec preferred_lft 41958sec 2023-03-09T14:45:56.141 INFO:teuthology.orchestra.run.smithi038.stdout: inet6 fe80::ec4:7aff:febd:15fc/64 scope link 2023-03-09T14:45:56.141 INFO:teuthology.orchestra.run.smithi038.stdout: valid_lft forever preferred_lft forever 2023-03-09T14:45:56.141 INFO:teuthology.orchestra.run.smithi038.stdout:4: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-03-09T14:45:56.142 INFO:teuthology.orchestra.run.smithi038.stdout: link/ether 0c:c4:7a:6b:fd:a3 brd ff:ff:ff:ff:ff:ff 2023-03-09T14:45:56.142 INFO:teuthology.orchestra.run.smithi038.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-03-09T14:45:56.142 INFO:teuthology.orchestra.run.smithi038.stdout: link/ether 0c:c4:7a:bd:15:fd brd ff:ff:ff:ff:ff:ff 2023-03-09T14:45:56.142 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-03-09T14:45:56.143 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-09T14:45:56.143 DEBUG:teuthology.orchestra.run.smithi038:> (cd / && exec bash -c ' 2023-03-09T14:45:56.143 DEBUG:teuthology.orchestra.run.smithi038:> set -e 2023-03-09T14:45:56.143 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip link add name ceph-brx type bridge 2023-03-09T14:45:56.144 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip addr flush dev ceph-brx 2023-03-09T14:45:56.144 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip link set ceph-brx up 2023-03-09T14:45:56.144 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-03-09T14:45:56.144 DEBUG:teuthology.orchestra.run.smithi038:> ') 2023-03-09T14:45:56.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:55 smithi122 ceph-mon[93869]: pgmap v309: 129 pgs: 129 active+clean; 636 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-09T14:45:56.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:55 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/2599044859' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-03-09T14:45:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:55 smithi181 ceph-mon[94189]: pgmap v309: 129 pgs: 129 active+clean; 636 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-09T14:45:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:55 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/2599044859' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-03-09T14:45:56.263 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-09T14:45:56.263 DEBUG:teuthology.orchestra.run.smithi038:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-03-09T14:45:56.291 INFO:teuthology.orchestra.run.smithi038.stdout:1 2023-03-09T14:45:56.293 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-09T14:45:56.293 DEBUG:teuthology.orchestra.run.smithi038:> route 2023-03-09T14:45:56.352 INFO:teuthology.orchestra.run.smithi038.stdout:Kernel IP routing table 2023-03-09T14:45:56.353 INFO:teuthology.orchestra.run.smithi038.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-03-09T14:45:56.353 INFO:teuthology.orchestra.run.smithi038.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-03-09T14:45:56.353 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-03-09T14:45:56.353 INFO:teuthology.orchestra.run.smithi038.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-03-09T14:45:56.353 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-09T14:45:56.354 DEBUG:teuthology.orchestra.run.smithi038:> (cd / && exec bash -c ' 2023-03-09T14:45:56.354 DEBUG:teuthology.orchestra.run.smithi038:> set -e 2023-03-09T14:45:56.354 DEBUG:teuthology.orchestra.run.smithi038:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-03-09T14:45:56.355 DEBUG:teuthology.orchestra.run.smithi038:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-03-09T14:45:56.355 DEBUG:teuthology.orchestra.run.smithi038:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-03-09T14:45:56.355 DEBUG:teuthology.orchestra.run.smithi038:> ') 2023-03-09T14:45:56.461 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-09T14:45:56.462 DEBUG:teuthology.orchestra.run.smithi038:> ip netns list 2023-03-09T14:45:56.479 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-09T14:45:56.479 DEBUG:teuthology.orchestra.run.smithi038:> ip netns list-id 2023-03-09T14:45:56.536 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-09T14:45:56.537 DEBUG:teuthology.orchestra.run.smithi038:> (cd / && exec bash -c ' 2023-03-09T14:45:56.537 DEBUG:teuthology.orchestra.run.smithi038:> set -e 2023-03-09T14:45:56.537 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip netns add ceph-ns--tmp-cephfs 2023-03-09T14:45:56.537 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip netns set ceph-ns--tmp-cephfs 0 2023-03-09T14:45:56.538 DEBUG:teuthology.orchestra.run.smithi038:> ') 2023-03-09T14:45:56.632 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--tmp-cephfs' with 192.168.0.1/16 2023-03-09T14:45:56.633 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-09T14:45:56.633 DEBUG:teuthology.orchestra.run.smithi038:> (cd / && exec bash -c ' 2023-03-09T14:45:56.633 DEBUG:teuthology.orchestra.run.smithi038:> set -e 2023-03-09T14:45:56.634 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip link add veth0 netns ceph-ns--tmp-cephfs type veth peer name brx.0 2023-03-09T14:45:56.634 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip netns exec ceph-ns--tmp-cephfs ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-03-09T14:45:56.634 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip netns exec ceph-ns--tmp-cephfs ip link set veth0 up 2023-03-09T14:45:56.634 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip netns exec ceph-ns--tmp-cephfs ip link set lo up 2023-03-09T14:45:56.635 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip netns exec ceph-ns--tmp-cephfs ip route add default via 192.168.255.254 2023-03-09T14:45:56.635 DEBUG:teuthology.orchestra.run.smithi038:> ') 2023-03-09T14:45:56.830 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-09T14:45:56.830 DEBUG:teuthology.orchestra.run.smithi038:> (cd / && exec bash -c ' 2023-03-09T14:45:56.830 DEBUG:teuthology.orchestra.run.smithi038:> set -e 2023-03-09T14:45:56.831 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip link set brx.0 up 2023-03-09T14:45:56.831 DEBUG:teuthology.orchestra.run.smithi038:> sudo ip link set dev brx.0 master ceph-brx 2023-03-09T14:45:56.831 DEBUG:teuthology.orchestra.run.smithi038:> ') 2023-03-09T14:45:56.952 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-03-09T14:45:56.952 DEBUG:teuthology.orchestra.run.smithi038:> mkdir -p -v /tmp/cephfs 2023-03-09T14:45:56.980 INFO:teuthology.orchestra.run.smithi038.stdout:mkdir: created directory '/tmp/cephfs' 2023-03-09T14:45:56.980 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-03-09T14:45:56.980 DEBUG:teuthology.orchestra.run.smithi038:> chmod 0000 /tmp/cephfs 2023-03-09T14:45:57.037 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2023-03-09T14:45:57.037 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2023-03-09T14:45:57.037 INFO:tasks.cephfs.kernel_mount:mounting using device: :/volumes/_nogroup/sv_1/ba63a2c0-19ef-41bb-becc-0a0d0bb2d986 2023-03-09T14:45:57.038 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-09T14:45:57.038 DEBUG:teuthology.orchestra.run.smithi038:> sudo nsenter --net=/var/run/netns/ceph-ns--tmp-cephfs adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/volumes/_nogroup/sv_1/ba63a2c0-19ef-41bb-becc-0a0d0bb2d986 /tmp/cephfs -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,wsync,context=system_u:object_r:postgresql_db_t:s0 2023-03-09T14:45:57.123 INFO:teuthology.orchestra.run.smithi038.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,wsync,context="system_u:object_r:postgresql_db_t:s0" 2023-03-09T14:45:57.124 INFO:teuthology.orchestra.run.smithi038.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,wsync,context="system_u:object_r:postgresql_db_t:s0"". 2023-03-09T14:45:57.124 INFO:teuthology.orchestra.run.smithi038.stdout:invalid new device string format 2023-03-09T14:45:57.267 INFO:teuthology.orchestra.run.smithi038.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,wsync,context="system_u:object_r:postgresql_db_t:s0" 2023-03-09T14:45:57.267 INFO:teuthology.orchestra.run.smithi038.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,wsync,context="system_u:object_r:postgresql_db_t:s0"". 2023-03-09T14:45:57.267 INFO:teuthology.orchestra.run.smithi038.stdout:invalid new device string format 2023-03-09T14:45:57.268 INFO:teuthology.orchestra.run.smithi038.stdout:mount.ceph: resolved to: "172.21.15.38:6789,172.21.15.122:6789,172.21.15.181:6789" 2023-03-09T14:45:57.268 INFO:teuthology.orchestra.run.smithi038.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.38:6789,172.21.15.122:6789,172.21.15.181:6789:/volumes/_nogroup/sv_1/ba63a2c0-19ef-41bb-becc-0a0d0bb2d986 2023-03-09T14:45:57.268 INFO:teuthology.orchestra.run.smithi038.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,wsync,context="system_u:object_r:postgresql_db_t:s0",key=0,fsid=c6733fe2-be87-11ed-9af2-001a4aab830c" will pass to kernel 2023-03-09T14:45:57.275 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-03-09T14:45:57.275 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-09T14:45:57.275 DEBUG:teuthology.orchestra.run.smithi038:> sudo chmod 1777 /tmp/cephfs 2023-03-09T14:45:57.335 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-09T14:45:57.335 DEBUG:teuthology.orchestra.run.smithi038:> (cd /tmp/cephfs && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-03-09T14:45:57.369 INFO:teuthology.orchestra.run.smithi038.stdout:client15375 2023-03-09T14:45:57.370 DEBUG:teuthology.orchestra.run.smithi038:> set -ex 2023-03-09T14:45:57.370 DEBUG:teuthology.orchestra.run.smithi038:> dd if=/proc/self/mounts of=/dev/stdout 2023-03-09T14:45:57.430 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-09T14:45:57.431 DEBUG:teuthology.orchestra.run.smithi038:> (cd /tmp/cephfs && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-03-09T14:45:57.503 INFO:teuthology.orchestra.run.smithi038.stdout:client15375 2023-03-09T14:45:57.504 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-09T14:45:57.504 DEBUG:teuthology.orchestra.run.smithi038:> (cd /tmp/cephfs && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-03-09T14:45:57.577 INFO:teuthology.orchestra.run.smithi038.stdout:client15375 2023-03-09T14:45:57.578 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-03-09T14:45:57.578 DEBUG:teuthology.orchestra.run.smithi038:> (cd /tmp/cephfs && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c.client15375/status') 2023-03-09T14:45:57.659 INFO:teuthology.orchestra.run.smithi038.stdout:instance: client.15375 (3)192.168.0.1:0/585902771 2023-03-09T14:45:57.660 INFO:teuthology.orchestra.run.smithi038.stdout:blocklisted: false 2023-03-09T14:45:57.660 INFO:teuthology.orchestra.run.smithi038.stderr:0+1 records in 2023-03-09T14:45:57.661 INFO:teuthology.orchestra.run.smithi038.stderr:0+1 records out 2023-03-09T14:45:57.661 INFO:teuthology.orchestra.run.smithi038.stderr:69 bytes copied, 2.5575e-05 s, 2.7 MB/s 2023-03-09T14:45:57.661 INFO:teuthology.run_tasks:Running task check-counter... 2023-03-09T14:45:57.675 INFO:teuthology.run_tasks:Running task fwd_scrub... 2023-03-09T14:45:57.718 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-09T14:45:57.844 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:57.842+0000 7fe3c9d07700 1 -- 172.21.15.38:0/4073242512 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe3c40fff80 msgr2=0x7fe3c41003d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:57.845 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:57.842+0000 7fe3c9d07700 1 --2- 172.21.15.38:0/4073242512 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe3c40fff80 0x7fe3c41003d0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fe3ac009990 tx=0x7fe3ac009ca0 comp rx=0 tx=0).stop 2023-03-09T14:45:57.845 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:57.842+0000 7fe3c9d07700 1 -- 172.21.15.38:0/4073242512 shutdown_connections 2023-03-09T14:45:57.845 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:57.842+0000 7fe3c9d07700 1 --2- 172.21.15.38:0/4073242512 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe3c4100910 0x7fe3c41071d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:57.845 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:57.842+0000 7fe3c9d07700 1 --2- 172.21.15.38:0/4073242512 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe3c40fed90 0x7fe3c40ff1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:57.846 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:57.842+0000 7fe3c9d07700 1 --2- 172.21.15.38:0/4073242512 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe3c40fff80 0x7fe3c41003d0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:57.846 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:57.842+0000 7fe3c9d07700 1 -- 172.21.15.38:0/4073242512 >> 172.21.15.38:0/4073242512 conn(0x7fe3c40fa370 msgr2=0x7fe3c40fc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:57.846 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:57.842+0000 7fe3c9d07700 1 -- 172.21.15.38:0/4073242512 shutdown_connections 2023-03-09T14:45:57.846 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:57.842+0000 7fe3c9d07700 1 -- 172.21.15.38:0/4073242512 wait complete. 2023-03-09T14:45:57.848 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:57.843+0000 7fe3c9d07700 1 Processor -- start 2023-03-09T14:45:57.848 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:57.843+0000 7fe3c9d07700 1 -- start start 2023-03-09T14:45:57.849 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:57.843+0000 7fe3c9d07700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe3c40fed90 0x7fe3c4069690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:57.849 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:57.843+0000 7fe3c9d07700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe3c40fff80 0x7fe3c419a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:57.849 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:57.843+0000 7fe3c9d07700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe3c4100910 0x7fe3c419cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:57.849 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:57.843+0000 7fe3c9d07700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c419d730 con 0x7fe3c40fff80 2023-03-09T14:45:57.849 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:57.843+0000 7fe3c9d07700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c419d8a0 con 0x7fe3c40fed90 2023-03-09T14:45:57.849 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:57.843+0000 7fe3c9d07700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c419da60 con 0x7fe3c4100910 2023-03-09T14:45:57.850 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:57.843+0000 7fe3c2ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe3c40fff80 0x7fe3c419a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:57.850 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:57.843+0000 7fe3c2ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe3c40fff80 0x7fe3c419a900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38332/0 (socket says 172.21.15.38:38332) 2023-03-09T14:45:57.850 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:57.843+0000 7fe3c2ffd700 1 -- 172.21.15.38:0/219501349 learned_addr learned my addr 172.21.15.38:0/219501349 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:57.850 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:57.844+0000 7fe3c3fff700 1 --2- 172.21.15.38:0/219501349 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe3c4100910 0x7fe3c419cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:57.850 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:57.844+0000 7fe3c2ffd700 1 -- 172.21.15.38:0/219501349 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe3c4100910 msgr2=0x7fe3c419cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:57.850 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:57.844+0000 7fe3c2ffd700 1 --2- 172.21.15.38:0/219501349 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe3c4100910 0x7fe3c419cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:57.851 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:57.844+0000 7fe3c37fe700 1 --2- 172.21.15.38:0/219501349 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe3c40fed90 0x7fe3c4069690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:57.851 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:57.844+0000 7fe3c2ffd700 1 -- 172.21.15.38:0/219501349 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe3c40fed90 msgr2=0x7fe3c4069690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:57.851 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:57.844+0000 7fe3c2ffd700 1 --2- 172.21.15.38:0/219501349 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe3c40fed90 0x7fe3c4069690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:57.851 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:57.844+0000 7fe3c2ffd700 1 -- 172.21.15.38:0/219501349 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3ac0095f0 con 0x7fe3c40fff80 2023-03-09T14:45:57.851 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:57.844+0000 7fe3c2ffd700 1 --2- 172.21.15.38:0/219501349 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe3c40fff80 0x7fe3c419a900 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fe3ac004760 tx=0x7fe3ac009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:57.851 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:57.844+0000 7fe3c0ff9700 1 -- 172.21.15.38:0/219501349 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3ac01d070 con 0x7fe3c40fff80 2023-03-09T14:45:57.852 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:57.844+0000 7fe3c0ff9700 1 -- 172.21.15.38:0/219501349 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe3ac004200 con 0x7fe3c40fff80 2023-03-09T14:45:57.852 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:57.844+0000 7fe3c0ff9700 1 -- 172.21.15.38:0/219501349 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3ac0049f0 con 0x7fe3c40fff80 2023-03-09T14:45:57.852 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:57.844+0000 7fe3c9d07700 1 -- 172.21.15.38:0/219501349 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3c410a6f0 con 0x7fe3c40fff80 2023-03-09T14:45:57.852 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:57.844+0000 7fe3c9d07700 1 -- 172.21.15.38:0/219501349 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3c410a560 con 0x7fe3c40fff80 2023-03-09T14:45:57.853 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:57.845+0000 7fe3c9d07700 1 -- 172.21.15.38:0/219501349 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3c4105cc0 con 0x7fe3c40fff80 2023-03-09T14:45:57.854 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:57.847+0000 7fe3c0ff9700 1 -- 172.21.15.38:0/219501349 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) v1 ==== 133198+0+0 (secure 0 0 0) 0x7fe3ac028030 con 0x7fe3c40fff80 2023-03-09T14:45:57.855 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:57.847+0000 7fe3c0ff9700 1 --2- 172.21.15.38:0/219501349 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fe3b009f3c0 0x7fe3b00a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:57.855 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:57.847+0000 7fe3c0ff9700 1 -- 172.21.15.38:0/219501349 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fe3ac0cb2e0 con 0x7fe3c40fff80 2023-03-09T14:45:57.855 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:57.847+0000 7fe3c37fe700 1 --2- 172.21.15.38:0/219501349 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fe3b009f3c0 0x7fe3b00a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:57.855 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:57.848+0000 7fe3c37fe700 1 --2- 172.21.15.38:0/219501349 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fe3b009f3c0 0x7fe3b00a1870 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fe3b4005950 tx=0x7fe3b40058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:57.855 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:57.850+0000 7fe3c0ff9700 1 -- 172.21.15.38:0/219501349 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7fe3ac0954d0 con 0x7fe3c40fff80 2023-03-09T14:45:58.058 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.056+0000 7fe3c9d07700 1 -- 172.21.15.38:0/219501349 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe3c402cef0 con 0x7fe3c40fff80 2023-03-09T14:45:58.060 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.058+0000 7fe3c0ff9700 1 -- 172.21.15.38:0/219501349 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24445 (secure 0 0 0) 0x7fe3ac026090 con 0x7fe3c40fff80 2023-03-09T14:45:58.060 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:45:58.061 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":89,"fsid":"c6733fe2-be87-11ed-9af2-001a4aab830c","created":"2023-03-09T14:37:14.947981+0000","modified":"2023-03-09T14:45:15.848936+0000","last_up_change":"2023-03-09T14:43:37.198407+0000","last_in_change":"2023-03-09T14:43:19.550566+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-09T14:40:23.196665+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-03-09T14:44:17.484374+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-03-09T14:44:18.457833+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-03-09T14:44:21.895589+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"51477039-15ee-4707-a34d-4ecfa18bebde","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6803","nonce":1508072290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6805","nonce":1508072290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6809","nonce":1508072290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6807","nonce":1508072290}]},"public_addr":"172.21.15.38:6803/1508072290","cluster_addr":"172.21.15.38:6805/1508072290","heartbeat_back_addr":"172.21.15.38:6809/1508072290","heartbeat_front_addr":"172.21.15.38:6807/1508072290","state":["exists","up"]},{"osd":1,"uuid":"71f8a718-b0a9-4f39-bc54-267aa42a2578","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6801","nonce":3929268773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6803","nonce":3929268773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6807","nonce":3929268773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6805","nonce":3929268773}]},"public_addr":"172.21.15.122:6801/3929268773","cluster_addr":"172.21.15.122:6803/3929268773","heartbeat_back_addr":"172.21.15.122:6807/3929268773","heartbeat_front_addr":"172.21.15.122:6805/3929268773","state":["exists","up"]},{"osd":2,"uuid":"961a2cbe-140e-437f-b131-1fcac573b19c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6801","nonce":1310630803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6803","nonce":1310630803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6807","nonce":1310630803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6805","nonce":1310630803}]},"public_addr":"172.21.15.181:6801/1310630803","cluster_addr":"172.21.15.181:6803/1310630803","heartbeat_back_addr":"172.21.15.181:6807/1310630803","heartbeat_front_addr":"172.21.15.181:6805/1310630803","state":["exists","up"]},{"osd":3,"uuid":"969df8fa-2d19-42c7-9f2a-5c5fac15d4a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6811","nonce":2719814100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6813","nonce":2719814100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6817","nonce":2719814100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6815","nonce":2719814100}]},"public_addr":"172.21.15.38:6811/2719814100","cluster_addr":"172.21.15.38:6813/2719814100","heartbeat_back_addr":"172.21.15.38:6817/2719814100","heartbeat_front_addr":"172.21.15.38:6815/2719814100","state":["exists","up"]},{"osd":4,"uuid":"44a6dce3-e06e-4cbd-9404-62a86d01ab6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6809","nonce":2480764994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6811","nonce":2480764994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6815","nonce":2480764994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6813","nonce":2480764994}]},"public_addr":"172.21.15.122:6809/2480764994","cluster_addr":"172.21.15.122:6811/2480764994","heartbeat_back_addr":"172.21.15.122:6815/2480764994","heartbeat_front_addr":"172.21.15.122:6813/2480764994","state":["exists","up"]},{"osd":5,"uuid":"49749574-c00c-4e3a-80e6-e3686df58d5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6809","nonce":3141346246}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6811","nonce":3141346246}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6815","nonce":3141346246}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6813","nonce":3141346246}]},"public_addr":"172.21.15.181:6809/3141346246","cluster_addr":"172.21.15.181:6811/3141346246","heartbeat_back_addr":"172.21.15.181:6815/3141346246","heartbeat_front_addr":"172.21.15.181:6813/3141346246","state":["exists","up"]},{"osd":6,"uuid":"61c0437e-3f51-45a9-8a61-e98eea0093f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6819","nonce":2122788957}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6821","nonce":2122788957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6825","nonce":2122788957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6823","nonce":2122788957}]},"public_addr":"172.21.15.38:6819/2122788957","cluster_addr":"172.21.15.38:6821/2122788957","heartbeat_back_addr":"172.21.15.38:6825/2122788957","heartbeat_front_addr":"172.21.15.38:6823/2122788957","state":["exists","up"]},{"osd":7,"uuid":"231fecdd-939f-4c7b-83a1-c4a1cffd7139","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6817","nonce":656874139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6819","nonce":656874139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6823","nonce":656874139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6821","nonce":656874139}]},"public_addr":"172.21.15.122:6817/656874139","cluster_addr":"172.21.15.122:6819/656874139","heartbeat_back_addr":"172.21.15.122:6823/656874139","heartbeat_front_addr":"172.21.15.122:6821/656874139","state":["exists","up"]},{"osd":8,"uuid":"297e6d6d-85fd-43ce-9028-5e9dbe55c36e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6817","nonce":755000703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6819","nonce":755000703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6823","nonce":755000703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6821","nonce":755000703}]},"public_addr":"172.21.15.181:6817/755000703","cluster_addr":"172.21.15.181:6819/755000703","heartbeat_back_addr":"172.21.15.181:6823/755000703","heartbeat_front_addr":"172.21.15.181:6821/755000703","state":["exists","up"]},{"osd":9,"uuid":"94cc0355-5a1a-4cfc-bd0c-562b6b41127e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6827","nonce":2624555969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6829","nonce":2624555969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6833","nonce":2624555969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6831","nonce":2624555969}]},"public_addr":"172.21.15.38:6827/2624555969","cluster_addr":"172.21.15.38:6829/2624555969","heartbeat_back_addr":"172.21.15.38:6833/2624555969","heartbeat_front_addr":"172.21.15.38:6831/2624555969","state":["exists","up"]},{"osd":10,"uuid":"f4f2e042-8d92-46bf-93b8-8b0f04795c2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6825","nonce":1427785280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6827","nonce":1427785280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6831","nonce":1427785280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6829","nonce":1427785280}]},"public_addr":"172.21.15.122:6825/1427785280","cluster_addr":"172.21.15.122:6827/1427785280","heartbeat_back_addr":"172.21.15.122:6831/1427785280","heartbeat_front_addr":"172.21.15.122:6829/1427785280","state":["exists","up"]},{"osd":11,"uuid":"07444936-eae5-4097-a799-60294cafe80b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6825","nonce":3183488640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6827","nonce":3183488640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6831","nonce":3183488640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6829","nonce":3183488640}]},"public_addr":"172.21.15.181:6825/3183488640","cluster_addr":"172.21.15.181:6827/3183488640","heartbeat_back_addr":"172.21.15.181:6831/3183488640","heartbeat_front_addr":"172.21.15.181:6829/3183488640","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:39:37.657556+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:39:58.960074+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:40:20.326952+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:40:42.244387+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:41:03.946973+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:41:25.097985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:41:46.877916+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:42:08.413827+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:42:29.822009+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:42:51.776684+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:43:13.937249+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:43:35.569199+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.2","mappings":[{"from":7,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.d","mappings":[{"from":5,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.17","mappings":[{"from":5,"to":6}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/4272288735":"2023-03-10T14:38:14.612767+0000","172.21.15.38:6801/3381625669":"2023-03-10T14:38:14.612767+0000","172.21.15.38:6800/3381625669":"2023-03-10T14:38:14.612767+0000","172.21.15.38:6800/3060474342":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/4253918615":"2023-03-10T14:38:14.612767+0000","172.21.15.38:0/3496083181":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/245602368":"2023-03-10T14:37:47.517775+0000","172.21.15.38:6801/3060474342":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/3622589182":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/1430302251":"2023-03-10T14:38:14.612767+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-09T14:45:58.062 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.060+0000 7fe3c9d07700 1 -- 172.21.15.38:0/219501349 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fe3b009f3c0 msgr2=0x7fe3b00a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:58.062 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.060+0000 7fe3c9d07700 1 --2- 172.21.15.38:0/219501349 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fe3b009f3c0 0x7fe3b00a1870 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fe3b4005950 tx=0x7fe3b40058e0 comp rx=0 tx=0).stop 2023-03-09T14:45:58.062 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.060+0000 7fe3c9d07700 1 -- 172.21.15.38:0/219501349 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe3c40fff80 msgr2=0x7fe3c419a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:58.063 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.060+0000 7fe3c9d07700 1 --2- 172.21.15.38:0/219501349 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe3c40fff80 0x7fe3c419a900 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fe3ac004760 tx=0x7fe3ac009f90 comp rx=0 tx=0).stop 2023-03-09T14:45:58.063 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.060+0000 7fe3c9d07700 1 -- 172.21.15.38:0/219501349 shutdown_connections 2023-03-09T14:45:58.063 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.060+0000 7fe3c9d07700 1 --2- 172.21.15.38:0/219501349 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fe3b009f3c0 0x7fe3b00a1870 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:58.063 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.060+0000 7fe3c9d07700 1 --2- 172.21.15.38:0/219501349 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe3c4100910 0x7fe3c419cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:58.063 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.060+0000 7fe3c9d07700 1 --2- 172.21.15.38:0/219501349 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe3c40fed90 0x7fe3c4069690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:58.063 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.060+0000 7fe3c9d07700 1 --2- 172.21.15.38:0/219501349 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe3c40fff80 0x7fe3c419a900 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:58.063 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.060+0000 7fe3c9d07700 1 -- 172.21.15.38:0/219501349 >> 172.21.15.38:0/219501349 conn(0x7fe3c40fa370 msgr2=0x7fe3c40fc610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:58.064 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.060+0000 7fe3c9d07700 1 -- 172.21.15.38:0/219501349 shutdown_connections 2023-03-09T14:45:58.064 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.060+0000 7fe3c9d07700 1 -- 172.21.15.38:0/219501349 wait complete. 2023-03-09T14:45:58.076 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-09T14:40:23.196665+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-03-09T14:44:17.484374+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-03-09T14:44:18.457833+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-03-09T14:44:21.895589+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-03-09T14:45:58.077 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-09T14:45:58.208 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.204+0000 7fd8cf4d6700 1 -- 172.21.15.38:0/3580841412 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8c8103270 msgr2=0x7fd8c8105650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:58.208 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.204+0000 7fd8cf4d6700 1 --2- 172.21.15.38:0/3580841412 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8c8103270 0x7fd8c8105650 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fd8c4009990 tx=0x7fd8c4009ca0 comp rx=0 tx=0).stop 2023-03-09T14:45:58.209 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.204+0000 7fd8cf4d6700 1 -- 172.21.15.38:0/3580841412 shutdown_connections 2023-03-09T14:45:58.209 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.204+0000 7fd8cf4d6700 1 --2- 172.21.15.38:0/3580841412 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd8c8105b90 0x7fd8c8107ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:58.209 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.204+0000 7fd8cf4d6700 1 --2- 172.21.15.38:0/3580841412 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd8c8100950 0x7fd8c8102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:58.209 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.204+0000 7fd8cf4d6700 1 --2- 172.21.15.38:0/3580841412 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8c8103270 0x7fd8c8105650 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:58.209 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.204+0000 7fd8cf4d6700 1 -- 172.21.15.38:0/3580841412 >> 172.21.15.38:0/3580841412 conn(0x7fd8c80fa3d0 msgr2=0x7fd8c80fc820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:58.209 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.204+0000 7fd8cf4d6700 1 -- 172.21.15.38:0/3580841412 shutdown_connections 2023-03-09T14:45:58.210 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.204+0000 7fd8cf4d6700 1 -- 172.21.15.38:0/3580841412 wait complete. 2023-03-09T14:45:58.210 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.205+0000 7fd8cf4d6700 1 Processor -- start 2023-03-09T14:45:58.210 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.205+0000 7fd8cf4d6700 1 -- start start 2023-03-09T14:45:58.210 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.205+0000 7fd8cf4d6700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8c8100950 0x7fd8c81002c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:58.210 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.205+0000 7fd8cf4d6700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd8c8103270 0x7fd8c80fe900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:58.210 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.205+0000 7fd8cf4d6700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd8c8105b90 0x7fd8c80fee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:58.210 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.205+0000 7fd8cf4d6700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c80ff5e0 con 0x7fd8c8100950 2023-03-09T14:45:58.211 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.205+0000 7fd8cf4d6700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c80ff750 con 0x7fd8c8105b90 2023-03-09T14:45:58.211 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.205+0000 7fd8cf4d6700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8c819fcb0 con 0x7fd8c8103270 2023-03-09T14:45:58.211 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.205+0000 7fd8cd272700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8c8100950 0x7fd8c81002c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:58.211 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.205+0000 7fd8cd272700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8c8100950 0x7fd8c81002c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38342/0 (socket says 172.21.15.38:38342) 2023-03-09T14:45:58.211 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.205+0000 7fd8cd272700 1 -- 172.21.15.38:0/3814604970 learned_addr learned my addr 172.21.15.38:0/3814604970 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:58.211 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.205+0000 7fd8cca71700 1 --2- 172.21.15.38:0/3814604970 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd8c8103270 0x7fd8c80fe900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:58.211 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.205+0000 7fd8cda73700 1 --2- 172.21.15.38:0/3814604970 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd8c8105b90 0x7fd8c80fee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:58.211 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.205+0000 7fd8cd272700 1 -- 172.21.15.38:0/3814604970 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd8c8103270 msgr2=0x7fd8c80fe900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:58.212 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.205+0000 7fd8cd272700 1 --2- 172.21.15.38:0/3814604970 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd8c8103270 0x7fd8c80fe900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:58.212 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.206+0000 7fd8cd272700 1 -- 172.21.15.38:0/3814604970 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd8c8105b90 msgr2=0x7fd8c80fee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:58.212 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.206+0000 7fd8cd272700 1 --2- 172.21.15.38:0/3814604970 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd8c8105b90 0x7fd8c80fee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:58.212 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.206+0000 7fd8cd272700 1 -- 172.21.15.38:0/3814604970 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8c40095f0 con 0x7fd8c8100950 2023-03-09T14:45:58.212 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.206+0000 7fd8cda73700 1 --2- 172.21.15.38:0/3814604970 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd8c8105b90 0x7fd8c80fee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-09T14:45:58.212 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.206+0000 7fd8cd272700 1 --2- 172.21.15.38:0/3814604970 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8c8100950 0x7fd8c81002c0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fd8b800c920 tx=0x7fd8b800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:58.212 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.206+0000 7fd8be7fc700 1 -- 172.21.15.38:0/3814604970 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b8007b00 con 0x7fd8c8100950 2023-03-09T14:45:58.213 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.206+0000 7fd8be7fc700 1 -- 172.21.15.38:0/3814604970 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd8b800ce70 con 0x7fd8c8100950 2023-03-09T14:45:58.213 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.206+0000 7fd8cf4d6700 1 -- 172.21.15.38:0/3814604970 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8c810b870 con 0x7fd8c8100950 2023-03-09T14:45:58.213 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.206+0000 7fd8be7fc700 1 -- 172.21.15.38:0/3814604970 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b80054e0 con 0x7fd8c8100950 2023-03-09T14:45:58.213 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.206+0000 7fd8cf4d6700 1 -- 172.21.15.38:0/3814604970 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8c810b6e0 con 0x7fd8c8100950 2023-03-09T14:45:58.214 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.207+0000 7fd8cf4d6700 1 -- 172.21.15.38:0/3814604970 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8c80fbf40 con 0x7fd8c8100950 2023-03-09T14:45:58.214 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.208+0000 7fd8be7fc700 1 -- 172.21.15.38:0/3814604970 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) v1 ==== 133198+0+0 (secure 0 0 0) 0x7fd8b801f070 con 0x7fd8c8100950 2023-03-09T14:45:58.214 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.208+0000 7fd8be7fc700 1 --2- 172.21.15.38:0/3814604970 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fd8b409f440 0x7fd8b40a18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:58.214 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.209+0000 7fd8cca71700 1 --2- 172.21.15.38:0/3814604970 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fd8b409f440 0x7fd8b40a18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:58.214 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.209+0000 7fd8be7fc700 1 -- 172.21.15.38:0/3814604970 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fd8b80c9b20 con 0x7fd8c8100950 2023-03-09T14:45:58.215 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.209+0000 7fd8cca71700 1 --2- 172.21.15.38:0/3814604970 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fd8b409f440 0x7fd8b40a18f0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fd8c400b5c0 tx=0x7fd8c4005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:58.215 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:57 smithi038 ceph-mon[92422]: pgmap v310: 129 pgs: 129 active+clean; 636 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 7 op/s 2023-03-09T14:45:58.215 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.213+0000 7fd8be7fc700 1 -- 172.21.15.38:0/3814604970 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7fd8b8093b70 con 0x7fd8c8100950 2023-03-09T14:45:58.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:57 smithi122 ceph-mon[93869]: pgmap v310: 129 pgs: 129 active+clean; 636 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 7 op/s 2023-03-09T14:45:58.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:57 smithi181 ceph-mon[94189]: pgmap v310: 129 pgs: 129 active+clean; 636 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 7 op/s 2023-03-09T14:45:58.416 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.414+0000 7fd8cf4d6700 1 -- 172.21.15.38:0/3814604970 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd8c8061640 con 0x7fd8c8100950 2023-03-09T14:45:58.417 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.415+0000 7fd8be7fc700 1 -- 172.21.15.38:0/3814604970 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd8b801d030 con 0x7fd8c8100950 2023-03-09T14:45:58.417 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 1 2023-03-09T14:45:58.419 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.416+0000 7fd8cf4d6700 1 -- 172.21.15.38:0/3814604970 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fd8b409f440 msgr2=0x7fd8b40a18f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:58.419 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.416+0000 7fd8cf4d6700 1 --2- 172.21.15.38:0/3814604970 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fd8b409f440 0x7fd8b40a18f0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fd8c400b5c0 tx=0x7fd8c4005fb0 comp rx=0 tx=0).stop 2023-03-09T14:45:58.419 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.416+0000 7fd8cf4d6700 1 -- 172.21.15.38:0/3814604970 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8c8100950 msgr2=0x7fd8c81002c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:58.419 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.416+0000 7fd8cf4d6700 1 --2- 172.21.15.38:0/3814604970 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8c8100950 0x7fd8c81002c0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fd8b800c920 tx=0x7fd8b800cc30 comp rx=0 tx=0).stop 2023-03-09T14:45:58.419 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.416+0000 7fd8cf4d6700 1 -- 172.21.15.38:0/3814604970 shutdown_connections 2023-03-09T14:45:58.420 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.417+0000 7fd8cf4d6700 1 --2- 172.21.15.38:0/3814604970 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd8c8100950 0x7fd8c81002c0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:58.420 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.417+0000 7fd8cf4d6700 1 --2- 172.21.15.38:0/3814604970 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fd8b409f440 0x7fd8b40a18f0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:58.420 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.417+0000 7fd8cf4d6700 1 --2- 172.21.15.38:0/3814604970 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd8c8105b90 0x7fd8c80fee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:58.420 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.417+0000 7fd8cf4d6700 1 --2- 172.21.15.38:0/3814604970 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd8c8103270 0x7fd8c80fe900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:58.420 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.417+0000 7fd8cf4d6700 1 -- 172.21.15.38:0/3814604970 >> 172.21.15.38:0/3814604970 conn(0x7fd8c80fa3d0 msgr2=0x7fd8c81015d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:58.420 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.417+0000 7fd8cf4d6700 1 -- 172.21.15.38:0/3814604970 shutdown_connections 2023-03-09T14:45:58.420 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.417+0000 7fd8cf4d6700 1 -- 172.21.15.38:0/3814604970 wait complete. 2023-03-09T14:45:58.431 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-03-09T14:45:58.561 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.558+0000 7f42e52f9700 1 -- 172.21.15.38:0/2971210305 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f42e00fffb0 msgr2=0x7f42e0100400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:58.561 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.558+0000 7f42e52f9700 1 --2- 172.21.15.38:0/2971210305 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f42e00fffb0 0x7f42e0100400 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f42d0009990 tx=0x7f42d0009ca0 comp rx=0 tx=0).stop 2023-03-09T14:45:58.561 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.558+0000 7f42e52f9700 1 -- 172.21.15.38:0/2971210305 shutdown_connections 2023-03-09T14:45:58.561 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.558+0000 7f42e52f9700 1 --2- 172.21.15.38:0/2971210305 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42e0100940 0x7f42e0107200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:58.561 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.558+0000 7f42e52f9700 1 --2- 172.21.15.38:0/2971210305 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f42e00fedc0 0x7f42e00ff1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:58.562 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.558+0000 7f42e52f9700 1 --2- 172.21.15.38:0/2971210305 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f42e00fffb0 0x7f42e0100400 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:58.562 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.558+0000 7f42e52f9700 1 -- 172.21.15.38:0/2971210305 >> 172.21.15.38:0/2971210305 conn(0x7f42e00fa3e0 msgr2=0x7f42e00fc810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:58.562 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.558+0000 7f42e52f9700 1 -- 172.21.15.38:0/2971210305 shutdown_connections 2023-03-09T14:45:58.562 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.558+0000 7f42e52f9700 1 -- 172.21.15.38:0/2971210305 wait complete. 2023-03-09T14:45:58.564 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.559+0000 7f42e52f9700 1 Processor -- start 2023-03-09T14:45:58.564 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.559+0000 7f42e52f9700 1 -- start start 2023-03-09T14:45:58.564 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.559+0000 7f42e52f9700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f42e00fedc0 0x7f42e0194db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:58.564 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.559+0000 7f42e52f9700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f42e00fffb0 0x7f42e01952f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:58.564 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.559+0000 7f42e52f9700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42e0100940 0x7f42e019c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:58.565 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.559+0000 7f42e52f9700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42e019c8f0 con 0x7f42e00fffb0 2023-03-09T14:45:58.565 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.559+0000 7f42e52f9700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42e019ca60 con 0x7f42e0100940 2023-03-09T14:45:58.565 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.559+0000 7f42e52f9700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42e019cc20 con 0x7f42e00fedc0 2023-03-09T14:45:58.565 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.560+0000 7f42de7fc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f42e00fffb0 0x7f42e01952f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:58.565 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.560+0000 7f42de7fc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f42e00fffb0 0x7f42e01952f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38358/0 (socket says 172.21.15.38:38358) 2023-03-09T14:45:58.565 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.560+0000 7f42de7fc700 1 -- 172.21.15.38:0/515940321 learned_addr learned my addr 172.21.15.38:0/515940321 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:58.565 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.560+0000 7f42deffd700 1 --2- 172.21.15.38:0/515940321 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f42e00fedc0 0x7f42e0194db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:58.566 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.560+0000 7f42df7fe700 1 --2- 172.21.15.38:0/515940321 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42e0100940 0x7f42e019c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:58.566 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.560+0000 7f42de7fc700 1 -- 172.21.15.38:0/515940321 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f42e00fedc0 msgr2=0x7f42e0194db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:58.566 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.560+0000 7f42de7fc700 1 --2- 172.21.15.38:0/515940321 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f42e00fedc0 0x7f42e0194db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:58.566 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.560+0000 7f42de7fc700 1 -- 172.21.15.38:0/515940321 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42e0100940 msgr2=0x7f42e019c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:58.566 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.560+0000 7f42de7fc700 1 --2- 172.21.15.38:0/515940321 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42e0100940 0x7f42e019c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:58.567 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.560+0000 7f42de7fc700 1 -- 172.21.15.38:0/515940321 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42d00095f0 con 0x7f42e00fffb0 2023-03-09T14:45:58.567 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.560+0000 7f42deffd700 1 --2- 172.21.15.38:0/515940321 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f42e00fedc0 0x7f42e0194db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:45:58.567 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.560+0000 7f42de7fc700 1 --2- 172.21.15.38:0/515940321 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f42e00fffb0 0x7f42e01952f0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f42d0005910 tx=0x7f42d0005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:58.567 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.560+0000 7f42c7fff700 1 -- 172.21.15.38:0/515940321 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42d001d070 con 0x7f42e00fffb0 2023-03-09T14:45:58.567 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.560+0000 7f42e52f9700 1 -- 172.21.15.38:0/515940321 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42e010a770 con 0x7f42e00fffb0 2023-03-09T14:45:58.567 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.560+0000 7f42c7fff700 1 -- 172.21.15.38:0/515940321 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f42d0004060 con 0x7f42e00fffb0 2023-03-09T14:45:58.567 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.560+0000 7f42c7fff700 1 -- 172.21.15.38:0/515940321 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42d0022e10 con 0x7f42e00fffb0 2023-03-09T14:45:58.567 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.560+0000 7f42e52f9700 1 -- 172.21.15.38:0/515940321 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42e010a5e0 con 0x7f42e00fffb0 2023-03-09T14:45:58.568 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.561+0000 7f42e52f9700 1 -- 172.21.15.38:0/515940321 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42e01060a0 con 0x7f42e00fffb0 2023-03-09T14:45:58.568 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.562+0000 7f42c7fff700 1 -- 172.21.15.38:0/515940321 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) v1 ==== 133198+0+0 (secure 0 0 0) 0x7f42d0029030 con 0x7f42e00fffb0 2023-03-09T14:45:58.569 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.563+0000 7f42c7fff700 1 --2- 172.21.15.38:0/515940321 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f42c809f3c0 0x7f42c80a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:58.569 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.563+0000 7f42deffd700 1 --2- 172.21.15.38:0/515940321 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f42c809f3c0 0x7f42c80a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:58.569 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.563+0000 7f42c7fff700 1 -- 172.21.15.38:0/515940321 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f42d00cb930 con 0x7f42e00fffb0 2023-03-09T14:45:58.569 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.563+0000 7f42deffd700 1 --2- 172.21.15.38:0/515940321 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f42c809f3c0 0x7f42c80a1870 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f42d4005950 tx=0x7f42d40058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:58.569 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.566+0000 7f42c7fff700 1 -- 172.21.15.38:0/515940321 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f42d0095a70 con 0x7f42e00fffb0 2023-03-09T14:45:58.769 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.767+0000 7f42e52f9700 1 -- 172.21.15.38:0/515940321 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f42e0061540 con 0x7f42e00fffb0 2023-03-09T14:45:58.770 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.767+0000 7f42c7fff700 1 -- 172.21.15.38:0/515940321 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) v1 ==== 104+0+11 (secure 0 0 0) 0x7f42d0099090 con 0x7f42e00fffb0 2023-03-09T14:45:58.770 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 32 2023-03-09T14:45:58.772 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.769+0000 7f42e52f9700 1 -- 172.21.15.38:0/515940321 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f42c809f3c0 msgr2=0x7f42c80a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:58.772 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.769+0000 7f42e52f9700 1 --2- 172.21.15.38:0/515940321 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f42c809f3c0 0x7f42c80a1870 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f42d4005950 tx=0x7f42d40058e0 comp rx=0 tx=0).stop 2023-03-09T14:45:58.772 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.769+0000 7f42e52f9700 1 -- 172.21.15.38:0/515940321 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f42e00fffb0 msgr2=0x7f42e01952f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:58.773 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.769+0000 7f42e52f9700 1 --2- 172.21.15.38:0/515940321 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f42e00fffb0 0x7f42e01952f0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f42d0005910 tx=0x7f42d0005e80 comp rx=0 tx=0).stop 2023-03-09T14:45:58.773 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.769+0000 7f42e52f9700 1 -- 172.21.15.38:0/515940321 shutdown_connections 2023-03-09T14:45:58.773 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.769+0000 7f42e52f9700 1 --2- 172.21.15.38:0/515940321 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f42c809f3c0 0x7f42c80a1870 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:58.773 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.769+0000 7f42e52f9700 1 --2- 172.21.15.38:0/515940321 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f42e0100940 0x7f42e019c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:58.773 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.769+0000 7f42e52f9700 1 --2- 172.21.15.38:0/515940321 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f42e00fedc0 0x7f42e0194db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:58.773 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.769+0000 7f42e52f9700 1 --2- 172.21.15.38:0/515940321 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f42e00fffb0 0x7f42e01952f0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:58.774 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.769+0000 7f42e52f9700 1 -- 172.21.15.38:0/515940321 >> 172.21.15.38:0/515940321 conn(0x7f42e00fa3e0 msgr2=0x7f42e01031e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:58.774 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.770+0000 7f42e52f9700 1 -- 172.21.15.38:0/515940321 shutdown_connections 2023-03-09T14:45:58.774 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.770+0000 7f42e52f9700 1 -- 172.21.15.38:0/515940321 wait complete. 2023-03-09T14:45:58.784 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-03-09T14:45:58.916 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.912+0000 7f65ce284700 1 -- 172.21.15.38:0/3331964332 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f65c8103260 msgr2=0x7f65c8105640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:58.917 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.912+0000 7f65ce284700 1 --2- 172.21.15.38:0/3331964332 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f65c8103260 0x7f65c8105640 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f65b0009990 tx=0x7f65b0009ca0 comp rx=0 tx=0).stop 2023-03-09T14:45:58.917 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.912+0000 7f65ce284700 1 -- 172.21.15.38:0/3331964332 shutdown_connections 2023-03-09T14:45:58.917 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.912+0000 7f65ce284700 1 --2- 172.21.15.38:0/3331964332 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f65c8105b80 0x7f65c8107fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:58.917 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.912+0000 7f65ce284700 1 --2- 172.21.15.38:0/3331964332 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f65c8100940 0x7f65c8102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:58.917 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.912+0000 7f65ce284700 1 --2- 172.21.15.38:0/3331964332 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f65c8103260 0x7f65c8105640 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:58.918 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.912+0000 7f65ce284700 1 -- 172.21.15.38:0/3331964332 >> 172.21.15.38:0/3331964332 conn(0x7f65c80fa3e0 msgr2=0x7f65c80fc810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:58.918 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.912+0000 7f65ce284700 1 -- 172.21.15.38:0/3331964332 shutdown_connections 2023-03-09T14:45:58.918 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.912+0000 7f65ce284700 1 -- 172.21.15.38:0/3331964332 wait complete. 2023-03-09T14:45:58.918 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.913+0000 7f65ce284700 1 Processor -- start 2023-03-09T14:45:58.918 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.913+0000 7f65ce284700 1 -- start start 2023-03-09T14:45:58.918 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.913+0000 7f65ce284700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f65c8100940 0x7f65c8195c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:58.919 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.913+0000 7f65ce284700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f65c8103260 0x7f65c8196190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:58.919 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.913+0000 7f65ce284700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f65c8105b80 0x7f65c819d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:58.919 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.913+0000 7f65ce284700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65c819d790 con 0x7f65c8100940 2023-03-09T14:45:58.919 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.913+0000 7f65ce284700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65c819d900 con 0x7f65c8105b80 2023-03-09T14:45:58.919 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.913+0000 7f65ce284700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65c819dac0 con 0x7f65c8103260 2023-03-09T14:45:58.919 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.913+0000 7f65c7fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f65c8100940 0x7f65c8195c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:58.920 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.913+0000 7f65c7fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f65c8100940 0x7f65c8195c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38370/0 (socket says 172.21.15.38:38370) 2023-03-09T14:45:58.920 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.913+0000 7f65c7fff700 1 -- 172.21.15.38:0/2339869682 learned_addr learned my addr 172.21.15.38:0/2339869682 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:58.920 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.914+0000 7f65bffff700 1 --2- 172.21.15.38:0/2339869682 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f65c8103260 0x7f65c8196190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:58.920 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.914+0000 7f65cc821700 1 --2- 172.21.15.38:0/2339869682 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f65c8105b80 0x7f65c819d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:58.920 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.914+0000 7f65c7fff700 1 -- 172.21.15.38:0/2339869682 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f65c8103260 msgr2=0x7f65c8196190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:58.920 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.914+0000 7f65c7fff700 1 --2- 172.21.15.38:0/2339869682 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f65c8103260 0x7f65c8196190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:58.920 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.914+0000 7f65c7fff700 1 -- 172.21.15.38:0/2339869682 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f65c8105b80 msgr2=0x7f65c819d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:58.920 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.914+0000 7f65c7fff700 1 --2- 172.21.15.38:0/2339869682 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f65c8105b80 0x7f65c819d1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:58.921 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.914+0000 7f65c7fff700 1 -- 172.21.15.38:0/2339869682 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65b00095f0 con 0x7f65c8100940 2023-03-09T14:45:58.921 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.914+0000 7f65cc821700 1 --2- 172.21.15.38:0/2339869682 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f65c8105b80 0x7f65c819d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-09T14:45:58.921 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.914+0000 7f65c7fff700 1 --2- 172.21.15.38:0/2339869682 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f65c8100940 0x7f65c8195c50 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f65b800d950 tx=0x7f65b800dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:58.921 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.914+0000 7f65c5ffb700 1 -- 172.21.15.38:0/2339869682 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b8009b40 con 0x7f65c8100940 2023-03-09T14:45:58.922 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.914+0000 7f65c5ffb700 1 -- 172.21.15.38:0/2339869682 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f65b8009ca0 con 0x7f65c8100940 2023-03-09T14:45:58.922 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.914+0000 7f65ce284700 1 -- 172.21.15.38:0/2339869682 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65c810b860 con 0x7f65c8100940 2023-03-09T14:45:58.922 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.914+0000 7f65c5ffb700 1 -- 172.21.15.38:0/2339869682 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b800f620 con 0x7f65c8100940 2023-03-09T14:45:58.923 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.914+0000 7f65ce284700 1 -- 172.21.15.38:0/2339869682 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65c810b6d0 con 0x7f65c8100940 2023-03-09T14:45:58.923 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.915+0000 7f65ce284700 1 -- 172.21.15.38:0/2339869682 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65c819e120 con 0x7f65c8100940 2023-03-09T14:45:58.924 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.916+0000 7f65c5ffb700 1 -- 172.21.15.38:0/2339869682 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) v1 ==== 133198+0+0 (secure 0 0 0) 0x7f65b8018030 con 0x7f65c8100940 2023-03-09T14:45:58.924 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.917+0000 7f65c5ffb700 1 --2- 172.21.15.38:0/2339869682 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f65b409f3c0 0x7f65b40a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:58.924 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.917+0000 7f65bffff700 1 --2- 172.21.15.38:0/2339869682 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f65b409f3c0 0x7f65b40a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:58.924 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.917+0000 7f65c5ffb700 1 -- 172.21.15.38:0/2339869682 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f65b80ca2e0 con 0x7f65c8100940 2023-03-09T14:45:58.924 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.917+0000 7f65bffff700 1 --2- 172.21.15.38:0/2339869682 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f65b409f3c0 0x7f65b40a1870 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f65b00036f0 tx=0x7f65b0003680 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:58.925 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:58.921+0000 7f65c5ffb700 1 -- 172.21.15.38:0/2339869682 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f65b8094440 con 0x7f65c8100940 2023-03-09T14:45:59.127 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.125+0000 7f65ce284700 1 -- 172.21.15.38:0/2339869682 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f65c819e6d0 con 0x7f65c8100940 2023-03-09T14:45:59.128 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.125+0000 7f65c5ffb700 1 -- 172.21.15.38:0/2339869682 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) v1 ==== 100+0+11 (secure 0 0 0) 0x7f65b8016020 con 0x7f65c8100940 2023-03-09T14:45:59.128 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 64 2023-03-09T14:45:59.130 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.127+0000 7f65ce284700 1 -- 172.21.15.38:0/2339869682 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f65b409f3c0 msgr2=0x7f65b40a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:59.130 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.127+0000 7f65ce284700 1 --2- 172.21.15.38:0/2339869682 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f65b409f3c0 0x7f65b40a1870 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f65b00036f0 tx=0x7f65b0003680 comp rx=0 tx=0).stop 2023-03-09T14:45:59.130 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.127+0000 7f65ce284700 1 -- 172.21.15.38:0/2339869682 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f65c8100940 msgr2=0x7f65c8195c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:59.131 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.127+0000 7f65ce284700 1 --2- 172.21.15.38:0/2339869682 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f65c8100940 0x7f65c8195c50 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f65b800d950 tx=0x7f65b800dd10 comp rx=0 tx=0).stop 2023-03-09T14:45:59.131 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.127+0000 7f65ce284700 1 -- 172.21.15.38:0/2339869682 shutdown_connections 2023-03-09T14:45:59.131 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.127+0000 7f65ce284700 1 --2- 172.21.15.38:0/2339869682 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f65c8100940 0x7f65c8195c50 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:59.132 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.127+0000 7f65ce284700 1 --2- 172.21.15.38:0/2339869682 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f65b409f3c0 0x7f65b40a1870 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:59.132 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.127+0000 7f65ce284700 1 --2- 172.21.15.38:0/2339869682 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f65c8105b80 0x7f65c819d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:59.132 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.127+0000 7f65ce284700 1 --2- 172.21.15.38:0/2339869682 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f65c8103260 0x7f65c8196190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:59.133 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.127+0000 7f65ce284700 1 -- 172.21.15.38:0/2339869682 >> 172.21.15.38:0/2339869682 conn(0x7f65c80fa3e0 msgr2=0x7f65c8101d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:59.133 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.127+0000 7f65ce284700 1 -- 172.21.15.38:0/2339869682 shutdown_connections 2023-03-09T14:45:59.133 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.127+0000 7f65ce284700 1 -- 172.21.15.38:0/2339869682 wait complete. 2023-03-09T14:45:59.142 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-03-09T14:45:59.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:58 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/219501349' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-03-09T14:45:59.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:58 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/3814604970' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-03-09T14:45:59.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:58 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/515940321' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-03-09T14:45:59.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:58 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/219501349' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-03-09T14:45:59.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:58 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/3814604970' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-03-09T14:45:59.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:58 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/515940321' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-03-09T14:45:59.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:58 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/219501349' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-03-09T14:45:59.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:58 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/3814604970' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-03-09T14:45:59.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:58 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/515940321' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-03-09T14:45:59.274 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.268+0000 7f5ea57fe700 1 -- 172.21.15.38:0/3430684678 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5ea00fee00 msgr2=0x7f5ea00ff210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:59.275 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.269+0000 7f5ea57fe700 1 --2- 172.21.15.38:0/3430684678 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5ea00fee00 0x7f5ea00ff210 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f5e94009990 tx=0x7f5e94009ca0 comp rx=0 tx=0).stop 2023-03-09T14:45:59.275 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.269+0000 7f5ea57fe700 1 -- 172.21.15.38:0/3430684678 shutdown_connections 2023-03-09T14:45:59.275 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.269+0000 7f5ea57fe700 1 --2- 172.21.15.38:0/3430684678 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5ea00fee00 0x7f5ea00ff210 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:59.276 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.269+0000 7f5ea57fe700 1 --2- 172.21.15.38:0/3430684678 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5ea0100980 0x7f5ea0107240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:59.276 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.269+0000 7f5ea57fe700 1 --2- 172.21.15.38:0/3430684678 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5ea00ffff0 0x7f5ea0100440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:59.276 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.269+0000 7f5ea57fe700 1 -- 172.21.15.38:0/3430684678 >> 172.21.15.38:0/3430684678 conn(0x7f5ea00fa420 msgr2=0x7f5ea00fc850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:59.277 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.269+0000 7f5ea57fe700 1 -- 172.21.15.38:0/3430684678 shutdown_connections 2023-03-09T14:45:59.277 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.269+0000 7f5ea57fe700 1 -- 172.21.15.38:0/3430684678 wait complete. 2023-03-09T14:45:59.277 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.269+0000 7f5ea57fe700 1 Processor -- start 2023-03-09T14:45:59.278 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.269+0000 7f5ea57fe700 1 -- start start 2023-03-09T14:45:59.278 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.270+0000 7f5ea57fe700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5ea00fee00 0x7f5ea0197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:59.278 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.270+0000 7f5ea57fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5ea00ffff0 0x7f5ea0198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:59.278 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.270+0000 7f5ea57fe700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5ea0100980 0x7f5ea019f440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:59.279 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.270+0000 7f5ea57fe700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea019fa10 con 0x7f5ea00ffff0 2023-03-09T14:45:59.279 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.270+0000 7f5ea57fe700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea019fb50 con 0x7f5ea0100980 2023-03-09T14:45:59.279 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.270+0000 7f5ea57fe700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea019fce0 con 0x7f5ea00fee00 2023-03-09T14:45:59.280 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.270+0000 7f5e9e7fc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5ea00ffff0 0x7f5ea0198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:59.280 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.270+0000 7f5e9e7fc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5ea00ffff0 0x7f5ea0198420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38386/0 (socket says 172.21.15.38:38386) 2023-03-09T14:45:59.280 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.270+0000 7f5e9e7fc700 1 -- 172.21.15.38:0/864340605 learned_addr learned my addr 172.21.15.38:0/864340605 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:59.281 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.270+0000 7f5e9f7fe700 1 --2- 172.21.15.38:0/864340605 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5ea0100980 0x7f5ea019f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:59.281 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.270+0000 7f5e9e7fc700 1 -- 172.21.15.38:0/864340605 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5ea00fee00 msgr2=0x7f5ea0197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:59.281 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.270+0000 7f5e9effd700 1 --2- 172.21.15.38:0/864340605 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5ea00fee00 0x7f5ea0197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:59.281 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.270+0000 7f5e9e7fc700 1 --2- 172.21.15.38:0/864340605 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5ea00fee00 0x7f5ea0197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:59.282 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.270+0000 7f5e9e7fc700 1 -- 172.21.15.38:0/864340605 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5ea0100980 msgr2=0x7f5ea019f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:59.282 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.270+0000 7f5e9e7fc700 1 --2- 172.21.15.38:0/864340605 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5ea0100980 0x7f5ea019f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:59.282 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.270+0000 7f5e9e7fc700 1 -- 172.21.15.38:0/864340605 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e940095f0 con 0x7f5ea00ffff0 2023-03-09T14:45:59.283 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.270+0000 7f5e9f7fe700 1 --2- 172.21.15.38:0/864340605 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5ea0100980 0x7f5ea019f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-09T14:45:59.283 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.270+0000 7f5e9effd700 1 --2- 172.21.15.38:0/864340605 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5ea00fee00 0x7f5ea0197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-09T14:45:59.283 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.270+0000 7f5e9e7fc700 1 --2- 172.21.15.38:0/864340605 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5ea00ffff0 0x7f5ea0198420 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f5e9000d950 tx=0x7f5e9000dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:59.284 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.270+0000 7f5e87fff700 1 -- 172.21.15.38:0/864340605 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e90009b40 con 0x7f5ea00ffff0 2023-03-09T14:45:59.284 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.270+0000 7f5e87fff700 1 -- 172.21.15.38:0/864340605 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f5e90009ca0 con 0x7f5ea00ffff0 2023-03-09T14:45:59.284 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.270+0000 7f5ea57fe700 1 -- 172.21.15.38:0/864340605 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ea010b890 con 0x7f5ea00ffff0 2023-03-09T14:45:59.284 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.270+0000 7f5e87fff700 1 -- 172.21.15.38:0/864340605 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e9000f620 con 0x7f5ea00ffff0 2023-03-09T14:45:59.285 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.271+0000 7f5ea57fe700 1 -- 172.21.15.38:0/864340605 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ea010b700 con 0x7f5ea00ffff0 2023-03-09T14:45:59.285 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.272+0000 7f5ea57fe700 1 -- 172.21.15.38:0/864340605 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ea01925f0 con 0x7f5ea00ffff0 2023-03-09T14:45:59.287 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.272+0000 7f5e87fff700 1 -- 172.21.15.38:0/864340605 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) v1 ==== 133198+0+0 (secure 0 0 0) 0x7f5e90018030 con 0x7f5ea00ffff0 2023-03-09T14:45:59.287 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.273+0000 7f5e87fff700 1 --2- 172.21.15.38:0/864340605 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f5e8809f510 0x7f5e880a19c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:59.287 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.273+0000 7f5e87fff700 1 -- 172.21.15.38:0/864340605 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f5e900ca290 con 0x7f5ea00ffff0 2023-03-09T14:45:59.288 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.273+0000 7f5e9effd700 1 --2- 172.21.15.38:0/864340605 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f5e8809f510 0x7f5e880a19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:59.288 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.274+0000 7f5e9effd700 1 --2- 172.21.15.38:0/864340605 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f5e8809f510 0x7f5e880a19c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5e94000c00 tx=0x7f5e94005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:59.288 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.277+0000 7f5e87fff700 1 -- 172.21.15.38:0/864340605 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f5e90094340 con 0x7f5ea00ffff0 2023-03-09T14:45:59.480 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.478+0000 7f5ea57fe700 1 -- 172.21.15.38:0/864340605 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f5ea00613c0 con 0x7f5ea00ffff0 2023-03-09T14:45:59.481 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.478+0000 7f5e87fff700 1 -- 172.21.15.38:0/864340605 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) v1 ==== 103+0+11 (secure 0 0 0) 0x7f5e90016020 con 0x7f5ea00ffff0 2023-03-09T14:45:59.481 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 32 2023-03-09T14:45:59.482 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.480+0000 7f5ea57fe700 1 -- 172.21.15.38:0/864340605 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f5e8809f510 msgr2=0x7f5e880a19c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:59.483 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.480+0000 7f5ea57fe700 1 --2- 172.21.15.38:0/864340605 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f5e8809f510 0x7f5e880a19c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5e94000c00 tx=0x7f5e94005fb0 comp rx=0 tx=0).stop 2023-03-09T14:45:59.483 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.480+0000 7f5ea57fe700 1 -- 172.21.15.38:0/864340605 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5ea00ffff0 msgr2=0x7f5ea0198420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:59.483 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.480+0000 7f5ea57fe700 1 --2- 172.21.15.38:0/864340605 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5ea00ffff0 0x7f5ea0198420 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f5e9000d950 tx=0x7f5e9000dd10 comp rx=0 tx=0).stop 2023-03-09T14:45:59.483 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.480+0000 7f5ea57fe700 1 -- 172.21.15.38:0/864340605 shutdown_connections 2023-03-09T14:45:59.483 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.480+0000 7f5ea57fe700 1 --2- 172.21.15.38:0/864340605 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f5e8809f510 0x7f5e880a19c0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:59.483 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.480+0000 7f5ea57fe700 1 --2- 172.21.15.38:0/864340605 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f5ea0100980 0x7f5ea019f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:59.484 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.480+0000 7f5ea57fe700 1 --2- 172.21.15.38:0/864340605 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f5ea00fee00 0x7f5ea0197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:59.484 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.480+0000 7f5ea57fe700 1 --2- 172.21.15.38:0/864340605 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f5ea00ffff0 0x7f5ea0198420 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:59.484 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.480+0000 7f5ea57fe700 1 -- 172.21.15.38:0/864340605 >> 172.21.15.38:0/864340605 conn(0x7f5ea00fa420 msgr2=0x7f5ea0105920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:59.484 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.480+0000 7f5ea57fe700 1 -- 172.21.15.38:0/864340605 shutdown_connections 2023-03-09T14:45:59.484 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.480+0000 7f5ea57fe700 1 -- 172.21.15.38:0/864340605 wait complete. 2023-03-09T14:45:59.496 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-09T14:45:59.625 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.619+0000 7ff4c1f52700 1 -- 172.21.15.38:0/3734551180 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4bc100900 msgr2=0x7ff4bc102ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:59.626 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.619+0000 7ff4c1f52700 1 --2- 172.21.15.38:0/3734551180 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4bc100900 0x7ff4bc102ce0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac009990 tx=0x7ff4ac009ca0 comp rx=0 tx=0).stop 2023-03-09T14:45:59.626 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.619+0000 7ff4c1f52700 1 -- 172.21.15.38:0/3734551180 shutdown_connections 2023-03-09T14:45:59.626 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.619+0000 7ff4c1f52700 1 --2- 172.21.15.38:0/3734551180 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4bc100900 0x7ff4bc102ce0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:59.626 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.619+0000 7ff4c1f52700 1 --2- 172.21.15.38:0/3734551180 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4bc105b40 0x7ff4bc107fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:59.626 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.619+0000 7ff4c1f52700 1 --2- 172.21.15.38:0/3734551180 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4bc103220 0x7ff4bc105600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:59.626 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.619+0000 7ff4c1f52700 1 -- 172.21.15.38:0/3734551180 >> 172.21.15.38:0/3734551180 conn(0x7ff4bc0fa380 msgr2=0x7ff4bc0fc7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:59.627 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.619+0000 7ff4c1f52700 1 -- 172.21.15.38:0/3734551180 shutdown_connections 2023-03-09T14:45:59.627 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.619+0000 7ff4c1f52700 1 -- 172.21.15.38:0/3734551180 wait complete. 2023-03-09T14:45:59.627 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.620+0000 7ff4c1f52700 1 Processor -- start 2023-03-09T14:45:59.627 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.620+0000 7ff4c1f52700 1 -- start start 2023-03-09T14:45:59.627 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.620+0000 7ff4c1f52700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4bc100900 0x7ff4bc195d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:59.628 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.620+0000 7ff4c1f52700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4bc103220 0x7ff4bc196260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:59.628 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.621+0000 7ff4c1f52700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4bc105b40 0x7ff4bc19d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:59.628 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.621+0000 7ff4c1f52700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4bc19d860 con 0x7ff4bc103220 2023-03-09T14:45:59.628 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.621+0000 7ff4c1f52700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4bc19d9d0 con 0x7ff4bc100900 2023-03-09T14:45:59.628 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.621+0000 7ff4c1f52700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4bc19db90 con 0x7ff4bc105b40 2023-03-09T14:45:59.628 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.621+0000 7ff4baffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4bc103220 0x7ff4bc196260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:59.629 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.621+0000 7ff4baffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4bc103220 0x7ff4bc196260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38400/0 (socket says 172.21.15.38:38400) 2023-03-09T14:45:59.629 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.621+0000 7ff4baffd700 1 -- 172.21.15.38:0/657308214 learned_addr learned my addr 172.21.15.38:0/657308214 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:59.629 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.621+0000 7ff4bb7fe700 1 --2- 172.21.15.38:0/657308214 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4bc100900 0x7ff4bc195d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:59.629 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.621+0000 7ff4bbfff700 1 --2- 172.21.15.38:0/657308214 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4bc105b40 0x7ff4bc19d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:59.629 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.621+0000 7ff4baffd700 1 -- 172.21.15.38:0/657308214 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4bc105b40 msgr2=0x7ff4bc19d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:59.629 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.621+0000 7ff4baffd700 1 --2- 172.21.15.38:0/657308214 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4bc105b40 0x7ff4bc19d290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:59.630 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.621+0000 7ff4baffd700 1 -- 172.21.15.38:0/657308214 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4bc100900 msgr2=0x7ff4bc195d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:59.630 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.621+0000 7ff4baffd700 1 --2- 172.21.15.38:0/657308214 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4bc100900 0x7ff4bc195d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:59.630 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.621+0000 7ff4baffd700 1 -- 172.21.15.38:0/657308214 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4ac0095f0 con 0x7ff4bc103220 2023-03-09T14:45:59.631 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.621+0000 7ff4bb7fe700 1 --2- 172.21.15.38:0/657308214 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4bc100900 0x7ff4bc195d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:45:59.631 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.621+0000 7ff4baffd700 1 --2- 172.21.15.38:0/657308214 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4bc103220 0x7ff4bc196260 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7ff4b000d950 tx=0x7ff4b000dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:59.631 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.621+0000 7ff4b8ff9700 1 -- 172.21.15.38:0/657308214 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b0009b40 con 0x7ff4bc103220 2023-03-09T14:45:59.631 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.621+0000 7ff4c1f52700 1 -- 172.21.15.38:0/657308214 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4bc10b7c0 con 0x7ff4bc103220 2023-03-09T14:45:59.632 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.621+0000 7ff4b8ff9700 1 -- 172.21.15.38:0/657308214 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff4b0009ca0 con 0x7ff4bc103220 2023-03-09T14:45:59.632 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.621+0000 7ff4b8ff9700 1 -- 172.21.15.38:0/657308214 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b000f620 con 0x7ff4bc103220 2023-03-09T14:45:59.632 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.621+0000 7ff4c1f52700 1 -- 172.21.15.38:0/657308214 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4bc10b630 con 0x7ff4bc103220 2023-03-09T14:45:59.633 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.622+0000 7ff4c1f52700 1 -- 172.21.15.38:0/657308214 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4bc0fbef0 con 0x7ff4bc103220 2023-03-09T14:45:59.634 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.623+0000 7ff4b8ff9700 1 -- 172.21.15.38:0/657308214 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) v1 ==== 133198+0+0 (secure 0 0 0) 0x7ff4b0018030 con 0x7ff4bc103220 2023-03-09T14:45:59.635 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.624+0000 7ff4b8ff9700 1 --2- 172.21.15.38:0/657308214 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7ff4a409f3c0 0x7ff4a40a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:59.635 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.624+0000 7ff4bb7fe700 1 --2- 172.21.15.38:0/657308214 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7ff4a409f3c0 0x7ff4a40a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:59.635 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.624+0000 7ff4b8ff9700 1 -- 172.21.15.38:0/657308214 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ff4b00ca3e0 con 0x7ff4bc103220 2023-03-09T14:45:59.636 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.625+0000 7ff4bb7fe700 1 --2- 172.21.15.38:0/657308214 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7ff4a409f3c0 0x7ff4a40a1870 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac006010 tx=0x7ff4ac00b540 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:59.636 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.627+0000 7ff4b8ff9700 1 -- 172.21.15.38:0/657308214 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7ff4b0094430 con 0x7ff4bc103220 2023-03-09T14:45:59.828 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.826+0000 7ff4c1f52700 1 -- 172.21.15.38:0/657308214 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff4bc02cef0 con 0x7ff4bc103220 2023-03-09T14:45:59.829 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.828+0000 7ff4b8ff9700 1 -- 172.21.15.38:0/657308214 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24445 (secure 0 0 0) 0x7ff4b0016020 con 0x7ff4bc103220 2023-03-09T14:45:59.830 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:45:59.830 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":89,"fsid":"c6733fe2-be87-11ed-9af2-001a4aab830c","created":"2023-03-09T14:37:14.947981+0000","modified":"2023-03-09T14:45:15.848936+0000","last_up_change":"2023-03-09T14:43:37.198407+0000","last_in_change":"2023-03-09T14:43:19.550566+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-09T14:40:23.196665+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-03-09T14:44:17.484374+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-03-09T14:44:18.457833+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-03-09T14:44:21.895589+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"51477039-15ee-4707-a34d-4ecfa18bebde","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6803","nonce":1508072290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6805","nonce":1508072290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6809","nonce":1508072290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6807","nonce":1508072290}]},"public_addr":"172.21.15.38:6803/1508072290","cluster_addr":"172.21.15.38:6805/1508072290","heartbeat_back_addr":"172.21.15.38:6809/1508072290","heartbeat_front_addr":"172.21.15.38:6807/1508072290","state":["exists","up"]},{"osd":1,"uuid":"71f8a718-b0a9-4f39-bc54-267aa42a2578","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6801","nonce":3929268773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6803","nonce":3929268773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6807","nonce":3929268773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6805","nonce":3929268773}]},"public_addr":"172.21.15.122:6801/3929268773","cluster_addr":"172.21.15.122:6803/3929268773","heartbeat_back_addr":"172.21.15.122:6807/3929268773","heartbeat_front_addr":"172.21.15.122:6805/3929268773","state":["exists","up"]},{"osd":2,"uuid":"961a2cbe-140e-437f-b131-1fcac573b19c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6801","nonce":1310630803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6803","nonce":1310630803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6807","nonce":1310630803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6805","nonce":1310630803}]},"public_addr":"172.21.15.181:6801/1310630803","cluster_addr":"172.21.15.181:6803/1310630803","heartbeat_back_addr":"172.21.15.181:6807/1310630803","heartbeat_front_addr":"172.21.15.181:6805/1310630803","state":["exists","up"]},{"osd":3,"uuid":"969df8fa-2d19-42c7-9f2a-5c5fac15d4a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6811","nonce":2719814100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6813","nonce":2719814100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6817","nonce":2719814100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6815","nonce":2719814100}]},"public_addr":"172.21.15.38:6811/2719814100","cluster_addr":"172.21.15.38:6813/2719814100","heartbeat_back_addr":"172.21.15.38:6817/2719814100","heartbeat_front_addr":"172.21.15.38:6815/2719814100","state":["exists","up"]},{"osd":4,"uuid":"44a6dce3-e06e-4cbd-9404-62a86d01ab6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6809","nonce":2480764994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6811","nonce":2480764994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6815","nonce":2480764994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6813","nonce":2480764994}]},"public_addr":"172.21.15.122:6809/2480764994","cluster_addr":"172.21.15.122:6811/2480764994","heartbeat_back_addr":"172.21.15.122:6815/2480764994","heartbeat_front_addr":"172.21.15.122:6813/2480764994","state":["exists","up"]},{"osd":5,"uuid":"49749574-c00c-4e3a-80e6-e3686df58d5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6809","nonce":3141346246}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6811","nonce":3141346246}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6815","nonce":3141346246}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6813","nonce":3141346246}]},"public_addr":"172.21.15.181:6809/3141346246","cluster_addr":"172.21.15.181:6811/3141346246","heartbeat_back_addr":"172.21.15.181:6815/3141346246","heartbeat_front_addr":"172.21.15.181:6813/3141346246","state":["exists","up"]},{"osd":6,"uuid":"61c0437e-3f51-45a9-8a61-e98eea0093f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6819","nonce":2122788957}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6821","nonce":2122788957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6825","nonce":2122788957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6823","nonce":2122788957}]},"public_addr":"172.21.15.38:6819/2122788957","cluster_addr":"172.21.15.38:6821/2122788957","heartbeat_back_addr":"172.21.15.38:6825/2122788957","heartbeat_front_addr":"172.21.15.38:6823/2122788957","state":["exists","up"]},{"osd":7,"uuid":"231fecdd-939f-4c7b-83a1-c4a1cffd7139","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6817","nonce":656874139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6819","nonce":656874139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6823","nonce":656874139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6821","nonce":656874139}]},"public_addr":"172.21.15.122:6817/656874139","cluster_addr":"172.21.15.122:6819/656874139","heartbeat_back_addr":"172.21.15.122:6823/656874139","heartbeat_front_addr":"172.21.15.122:6821/656874139","state":["exists","up"]},{"osd":8,"uuid":"297e6d6d-85fd-43ce-9028-5e9dbe55c36e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6817","nonce":755000703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6819","nonce":755000703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6823","nonce":755000703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6821","nonce":755000703}]},"public_addr":"172.21.15.181:6817/755000703","cluster_addr":"172.21.15.181:6819/755000703","heartbeat_back_addr":"172.21.15.181:6823/755000703","heartbeat_front_addr":"172.21.15.181:6821/755000703","state":["exists","up"]},{"osd":9,"uuid":"94cc0355-5a1a-4cfc-bd0c-562b6b41127e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6827","nonce":2624555969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6829","nonce":2624555969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6833","nonce":2624555969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6831","nonce":2624555969}]},"public_addr":"172.21.15.38:6827/2624555969","cluster_addr":"172.21.15.38:6829/2624555969","heartbeat_back_addr":"172.21.15.38:6833/2624555969","heartbeat_front_addr":"172.21.15.38:6831/2624555969","state":["exists","up"]},{"osd":10,"uuid":"f4f2e042-8d92-46bf-93b8-8b0f04795c2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6825","nonce":1427785280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6827","nonce":1427785280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6831","nonce":1427785280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6829","nonce":1427785280}]},"public_addr":"172.21.15.122:6825/1427785280","cluster_addr":"172.21.15.122:6827/1427785280","heartbeat_back_addr":"172.21.15.122:6831/1427785280","heartbeat_front_addr":"172.21.15.122:6829/1427785280","state":["exists","up"]},{"osd":11,"uuid":"07444936-eae5-4097-a799-60294cafe80b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6825","nonce":3183488640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6827","nonce":3183488640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6831","nonce":3183488640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6829","nonce":3183488640}]},"public_addr":"172.21.15.181:6825/3183488640","cluster_addr":"172.21.15.181:6827/3183488640","heartbeat_back_addr":"172.21.15.181:6831/3183488640","heartbeat_front_addr":"172.21.15.181:6829/3183488640","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:39:37.657556+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:39:58.960074+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:40:20.326952+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:40:42.244387+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:41:03.946973+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:41:25.097985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:41:46.877916+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:42:08.413827+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:42:29.822009+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:42:51.776684+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:43:13.937249+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:43:35.569199+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.2","mappings":[{"from":7,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.d","mappings":[{"from":5,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.17","mappings":[{"from":5,"to":6}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/4272288735":"2023-03-10T14:38:14.612767+0000","172.21.15.38:6801/3381625669":"2023-03-10T14:38:14.612767+0000","172.21.15.38:6800/3381625669":"2023-03-10T14:38:14.612767+0000","172.21.15.38:6800/3060474342":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/4253918615":"2023-03-10T14:38:14.612767+0000","172.21.15.38:0/3496083181":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/245602368":"2023-03-10T14:37:47.517775+0000","172.21.15.38:6801/3060474342":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/3622589182":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/1430302251":"2023-03-10T14:38:14.612767+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-09T14:45:59.832 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.830+0000 7ff4c1f52700 1 -- 172.21.15.38:0/657308214 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7ff4a409f3c0 msgr2=0x7ff4a40a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:59.833 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.830+0000 7ff4c1f52700 1 --2- 172.21.15.38:0/657308214 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7ff4a409f3c0 0x7ff4a40a1870 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac006010 tx=0x7ff4ac00b540 comp rx=0 tx=0).stop 2023-03-09T14:45:59.833 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.830+0000 7ff4c1f52700 1 -- 172.21.15.38:0/657308214 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4bc103220 msgr2=0x7ff4bc196260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:59.833 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.830+0000 7ff4c1f52700 1 --2- 172.21.15.38:0/657308214 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4bc103220 0x7ff4bc196260 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7ff4b000d950 tx=0x7ff4b000dd10 comp rx=0 tx=0).stop 2023-03-09T14:45:59.833 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.830+0000 7ff4c1f52700 1 -- 172.21.15.38:0/657308214 shutdown_connections 2023-03-09T14:45:59.833 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.830+0000 7ff4c1f52700 1 --2- 172.21.15.38:0/657308214 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7ff4a409f3c0 0x7ff4a40a1870 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:59.833 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.830+0000 7ff4c1f52700 1 --2- 172.21.15.38:0/657308214 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4bc105b40 0x7ff4bc19d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:59.833 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.830+0000 7ff4c1f52700 1 --2- 172.21.15.38:0/657308214 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4bc100900 0x7ff4bc195d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:59.834 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.830+0000 7ff4c1f52700 1 --2- 172.21.15.38:0/657308214 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4bc103220 0x7ff4bc196260 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:59.834 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.830+0000 7ff4c1f52700 1 -- 172.21.15.38:0/657308214 >> 172.21.15.38:0/657308214 conn(0x7ff4bc0fa380 msgr2=0x7ff4bc103e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:59.834 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.830+0000 7ff4c1f52700 1 -- 172.21.15.38:0/657308214 shutdown_connections 2023-03-09T14:45:59.834 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.830+0000 7ff4c1f52700 1 -- 172.21.15.38:0/657308214 wait complete. 2023-03-09T14:45:59.845 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-09T14:40:23.196665+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-03-09T14:44:17.484374+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-03-09T14:44:18.457833+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-03-09T14:44:21.895589+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-03-09T14:45:59.846 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-09T14:45:59.975 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.973+0000 7fca36b38700 1 -- 172.21.15.38:0/1124661410 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fca300fff80 msgr2=0x7fca301003d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:59.976 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.973+0000 7fca36b38700 1 --2- 172.21.15.38:0/1124661410 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fca300fff80 0x7fca301003d0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fca20009990 tx=0x7fca20009ca0 comp rx=0 tx=0).stop 2023-03-09T14:45:59.976 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.973+0000 7fca36b38700 1 -- 172.21.15.38:0/1124661410 shutdown_connections 2023-03-09T14:45:59.976 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.973+0000 7fca36b38700 1 --2- 172.21.15.38:0/1124661410 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fca30100910 0x7fca301071d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:59.976 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.973+0000 7fca36b38700 1 --2- 172.21.15.38:0/1124661410 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fca300fed90 0x7fca300ff1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:59.976 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.973+0000 7fca36b38700 1 --2- 172.21.15.38:0/1124661410 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fca300fff80 0x7fca301003d0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:59.977 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.973+0000 7fca36b38700 1 -- 172.21.15.38:0/1124661410 >> 172.21.15.38:0/1124661410 conn(0x7fca300fa3b0 msgr2=0x7fca300fc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:45:59.977 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.973+0000 7fca36b38700 1 -- 172.21.15.38:0/1124661410 shutdown_connections 2023-03-09T14:45:59.977 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.973+0000 7fca36b38700 1 -- 172.21.15.38:0/1124661410 wait complete. 2023-03-09T14:45:59.979 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.974+0000 7fca36b38700 1 Processor -- start 2023-03-09T14:45:59.979 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.974+0000 7fca36b38700 1 -- start start 2023-03-09T14:45:59.979 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.974+0000 7fca36b38700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fca300fed90 0x7fca30194b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:59.979 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.974+0000 7fca36b38700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fca300fff80 0x7fca30195080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:59.979 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.974+0000 7fca36b38700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fca30100910 0x7fca3019c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:59.979 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.974+0000 7fca36b38700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca3019c6c0 con 0x7fca300fff80 2023-03-09T14:45:59.979 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.974+0000 7fca36b38700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca3019c800 con 0x7fca30100910 2023-03-09T14:45:59.980 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.974+0000 7fca36b38700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca3019c990 con 0x7fca300fed90 2023-03-09T14:45:59.980 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.974+0000 7fca27fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fca300fff80 0x7fca30195080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:59.980 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.974+0000 7fca27fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fca300fff80 0x7fca30195080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38414/0 (socket says 172.21.15.38:38414) 2023-03-09T14:45:59.980 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.974+0000 7fca348d4700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fca300fed90 0x7fca30194b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:59.980 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.974+0000 7fca27fff700 1 -- 172.21.15.38:0/612771049 learned_addr learned my addr 172.21.15.38:0/612771049 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:45:59.980 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.974+0000 7fca350d5700 1 --2- 172.21.15.38:0/612771049 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fca30100910 0x7fca3019c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:59.980 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.974+0000 7fca27fff700 1 -- 172.21.15.38:0/612771049 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fca300fed90 msgr2=0x7fca30194b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:59.981 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.974+0000 7fca27fff700 1 --2- 172.21.15.38:0/612771049 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fca300fed90 0x7fca30194b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:59.981 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.974+0000 7fca27fff700 1 -- 172.21.15.38:0/612771049 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fca30100910 msgr2=0x7fca3019c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:45:59.981 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.974+0000 7fca27fff700 1 --2- 172.21.15.38:0/612771049 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fca30100910 0x7fca3019c0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:45:59.981 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.974+0000 7fca27fff700 1 -- 172.21.15.38:0/612771049 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca200095f0 con 0x7fca300fff80 2023-03-09T14:45:59.981 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.975+0000 7fca350d5700 1 --2- 172.21.15.38:0/612771049 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fca30100910 0x7fca3019c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-09T14:45:59.981 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.975+0000 7fca27fff700 1 --2- 172.21.15.38:0/612771049 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fca300fff80 0x7fca30195080 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fca2000c010 tx=0x7fca20004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:59.981 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.975+0000 7fca25ffb700 1 -- 172.21.15.38:0/612771049 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca2001f070 con 0x7fca300fff80 2023-03-09T14:45:59.981 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.975+0000 7fca25ffb700 1 -- 172.21.15.38:0/612771049 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fca2001ad70 con 0x7fca300fff80 2023-03-09T14:45:59.982 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.975+0000 7fca36b38700 1 -- 172.21.15.38:0/612771049 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca3010a740 con 0x7fca300fff80 2023-03-09T14:45:59.982 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.975+0000 7fca25ffb700 1 -- 172.21.15.38:0/612771049 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca2000f460 con 0x7fca300fff80 2023-03-09T14:45:59.982 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.975+0000 7fca36b38700 1 -- 172.21.15.38:0/612771049 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca3010a5b0 con 0x7fca300fff80 2023-03-09T14:45:59.982 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.976+0000 7fca36b38700 1 -- 172.21.15.38:0/612771049 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca30067dc0 con 0x7fca300fff80 2023-03-09T14:45:59.983 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.977+0000 7fca25ffb700 1 -- 172.21.15.38:0/612771049 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) v1 ==== 133198+0+0 (secure 0 0 0) 0x7fca2002a030 con 0x7fca300fff80 2023-03-09T14:45:59.983 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.978+0000 7fca25ffb700 1 --2- 172.21.15.38:0/612771049 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fca1809f440 0x7fca180a18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:45:59.983 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.978+0000 7fca25ffb700 1 -- 172.21.15.38:0/612771049 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fca200cd9d0 con 0x7fca300fff80 2023-03-09T14:45:59.983 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.978+0000 7fca348d4700 1 --2- 172.21.15.38:0/612771049 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fca1809f440 0x7fca180a18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:45:59.983 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.978+0000 7fca348d4700 1 --2- 172.21.15.38:0/612771049 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fca1809f440 0x7fca180a18f0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fca300ffde0 tx=0x7fca2c00b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:45:59.984 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:45:59.981+0000 7fca25ffb700 1 -- 172.21.15.38:0/612771049 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7fca20097ab0 con 0x7fca300fff80 2023-03-09T14:46:00.183 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.181+0000 7fca36b38700 1 -- 172.21.15.38:0/612771049 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fca30068810 con 0x7fca300fff80 2023-03-09T14:46:00.184 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.181+0000 7fca25ffb700 1 -- 172.21.15.38:0/612771049 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7fca200280a0 con 0x7fca300fff80 2023-03-09T14:46:00.184 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 1 2023-03-09T14:46:00.186 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.183+0000 7fca36b38700 1 -- 172.21.15.38:0/612771049 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fca1809f440 msgr2=0x7fca180a18f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:00.186 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.183+0000 7fca36b38700 1 --2- 172.21.15.38:0/612771049 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fca1809f440 0x7fca180a18f0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fca300ffde0 tx=0x7fca2c00b410 comp rx=0 tx=0).stop 2023-03-09T14:46:00.186 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.183+0000 7fca36b38700 1 -- 172.21.15.38:0/612771049 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fca300fff80 msgr2=0x7fca30195080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:00.186 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.183+0000 7fca36b38700 1 --2- 172.21.15.38:0/612771049 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fca300fff80 0x7fca30195080 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fca2000c010 tx=0x7fca20004730 comp rx=0 tx=0).stop 2023-03-09T14:46:00.187 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.183+0000 7fca36b38700 1 -- 172.21.15.38:0/612771049 shutdown_connections 2023-03-09T14:46:00.187 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.183+0000 7fca36b38700 1 --2- 172.21.15.38:0/612771049 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fca1809f440 0x7fca180a18f0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:00.187 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.183+0000 7fca36b38700 1 --2- 172.21.15.38:0/612771049 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fca30100910 0x7fca3019c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:00.187 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.183+0000 7fca36b38700 1 --2- 172.21.15.38:0/612771049 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fca300fed90 0x7fca30194b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:00.187 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.183+0000 7fca36b38700 1 --2- 172.21.15.38:0/612771049 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fca300fff80 0x7fca30195080 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:00.187 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.183+0000 7fca36b38700 1 -- 172.21.15.38:0/612771049 >> 172.21.15.38:0/612771049 conn(0x7fca300fa3b0 msgr2=0x7fca300fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:00.187 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.183+0000 7fca36b38700 1 -- 172.21.15.38:0/612771049 shutdown_connections 2023-03-09T14:46:00.188 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.183+0000 7fca36b38700 1 -- 172.21.15.38:0/612771049 wait complete. 2023-03-09T14:46:00.197 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:59 smithi038 ceph-mon[92422]: pgmap v311: 129 pgs: 129 active+clean; 636 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-09T14:46:00.197 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:59 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/2339869682' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-03-09T14:46:00.197 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:59 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/864340605' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-03-09T14:46:00.197 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:45:59 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/657308214' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-03-09T14:46:00.198 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-03-09T14:46:00.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:59 smithi122 ceph-mon[93869]: pgmap v311: 129 pgs: 129 active+clean; 636 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-09T14:46:00.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:59 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/2339869682' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-03-09T14:46:00.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:59 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/864340605' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-03-09T14:46:00.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:45:59 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/657308214' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-03-09T14:46:00.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:59 smithi181 ceph-mon[94189]: pgmap v311: 129 pgs: 129 active+clean; 636 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-09T14:46:00.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:59 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/2339869682' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-03-09T14:46:00.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:59 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/864340605' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-03-09T14:46:00.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:45:59 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/657308214' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-03-09T14:46:00.331 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.326+0000 7f2a08a81700 1 -- 172.21.15.38:0/1534886474 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2a04103230 msgr2=0x7f2a04105610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:00.332 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.326+0000 7f2a08a81700 1 --2- 172.21.15.38:0/1534886474 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2a04103230 0x7f2a04105610 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f29f8009990 tx=0x7f29f8009ca0 comp rx=0 tx=0).stop 2023-03-09T14:46:00.332 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.326+0000 7f2a08a81700 1 -- 172.21.15.38:0/1534886474 shutdown_connections 2023-03-09T14:46:00.332 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.326+0000 7f2a08a81700 1 --2- 172.21.15.38:0/1534886474 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2a04105b50 0x7f2a04107fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:00.333 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.326+0000 7f2a08a81700 1 --2- 172.21.15.38:0/1534886474 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a04100910 0x7f2a04102cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:00.333 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.326+0000 7f2a08a81700 1 --2- 172.21.15.38:0/1534886474 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2a04103230 0x7f2a04105610 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:00.333 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.326+0000 7f2a08a81700 1 -- 172.21.15.38:0/1534886474 >> 172.21.15.38:0/1534886474 conn(0x7f2a040fa3b0 msgr2=0x7f2a040fc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:00.334 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.326+0000 7f2a08a81700 1 -- 172.21.15.38:0/1534886474 shutdown_connections 2023-03-09T14:46:00.334 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.326+0000 7f2a08a81700 1 -- 172.21.15.38:0/1534886474 wait complete. 2023-03-09T14:46:00.334 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.327+0000 7f2a08a81700 1 Processor -- start 2023-03-09T14:46:00.335 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.327+0000 7f2a08a81700 1 -- start start 2023-03-09T14:46:00.335 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.327+0000 7f2a08a81700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2a04100910 0x7f2a04195c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:00.335 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.327+0000 7f2a08a81700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a04103230 0x7f2a041961a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:00.336 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.327+0000 7f2a08a81700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2a04105b50 0x7f2a0419d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:00.336 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.327+0000 7f2a08a81700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0419d7a0 con 0x7f2a04100910 2023-03-09T14:46:00.336 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.327+0000 7f2a08a81700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0419d910 con 0x7f2a04103230 2023-03-09T14:46:00.336 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.327+0000 7f2a08a81700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0419dad0 con 0x7f2a04105b50 2023-03-09T14:46:00.336 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.327+0000 7f2a0259c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2a04100910 0x7f2a04195c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:00.337 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.327+0000 7f2a0259c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2a04100910 0x7f2a04195c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38426/0 (socket says 172.21.15.38:38426) 2023-03-09T14:46:00.337 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.327+0000 7f2a0259c700 1 -- 172.21.15.38:0/302997020 learned_addr learned my addr 172.21.15.38:0/302997020 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:46:00.337 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.327+0000 7f2a01d9b700 1 --2- 172.21.15.38:0/302997020 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a04103230 0x7f2a041961a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:00.338 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.327+0000 7f2a02d9d700 1 --2- 172.21.15.38:0/302997020 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2a04105b50 0x7f2a0419d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:00.338 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.328+0000 7f2a0259c700 1 -- 172.21.15.38:0/302997020 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2a04105b50 msgr2=0x7f2a0419d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:00.338 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.328+0000 7f2a0259c700 1 --2- 172.21.15.38:0/302997020 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2a04105b50 0x7f2a0419d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:00.338 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.328+0000 7f2a0259c700 1 -- 172.21.15.38:0/302997020 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a04103230 msgr2=0x7f2a041961a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:00.338 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.328+0000 7f2a0259c700 1 --2- 172.21.15.38:0/302997020 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a04103230 0x7f2a041961a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:00.338 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.328+0000 7f2a0259c700 1 -- 172.21.15.38:0/302997020 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29f80095f0 con 0x7f2a04100910 2023-03-09T14:46:00.339 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.328+0000 7f2a0259c700 1 --2- 172.21.15.38:0/302997020 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2a04100910 0x7f2a04195c60 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f29f400d950 tx=0x7f29f400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:00.339 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.328+0000 7f29f37fe700 1 -- 172.21.15.38:0/302997020 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f4009940 con 0x7f2a04100910 2023-03-09T14:46:00.339 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.328+0000 7f29f37fe700 1 -- 172.21.15.38:0/302997020 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f29f4010460 con 0x7f2a04100910 2023-03-09T14:46:00.339 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.328+0000 7f2a08a81700 1 -- 172.21.15.38:0/302997020 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a0410b830 con 0x7f2a04100910 2023-03-09T14:46:00.339 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.328+0000 7f29f37fe700 1 -- 172.21.15.38:0/302997020 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f400f620 con 0x7f2a04100910 2023-03-09T14:46:00.339 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.328+0000 7f2a08a81700 1 -- 172.21.15.38:0/302997020 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a0410b6a0 con 0x7f2a04100910 2023-03-09T14:46:00.339 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.329+0000 7f2a08a81700 1 -- 172.21.15.38:0/302997020 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a0419e130 con 0x7f2a04100910 2023-03-09T14:46:00.340 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.330+0000 7f29f37fe700 1 -- 172.21.15.38:0/302997020 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) v1 ==== 133198+0+0 (secure 0 0 0) 0x7f29f4022070 con 0x7f2a04100910 2023-03-09T14:46:00.341 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.330+0000 7f29f37fe700 1 --2- 172.21.15.38:0/302997020 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f29ec09f3c0 0x7f29ec0a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:00.341 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.331+0000 7f29f37fe700 1 -- 172.21.15.38:0/302997020 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f29f40c9e10 con 0x7f2a04100910 2023-03-09T14:46:00.341 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.331+0000 7f2a01d9b700 1 --2- 172.21.15.38:0/302997020 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f29ec09f3c0 0x7f29ec0a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:00.341 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.331+0000 7f2a01d9b700 1 --2- 172.21.15.38:0/302997020 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f29ec09f3c0 0x7f29ec0a1870 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f29f800b5c0 tx=0x7f29f8005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:00.341 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.335+0000 7f29f37fe700 1 -- 172.21.15.38:0/302997020 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f29f4093ec0 con 0x7f2a04100910 2023-03-09T14:46:00.536 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.534+0000 7f2a08a81700 1 -- 172.21.15.38:0/302997020 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2a0419e6e0 con 0x7f2a04100910 2023-03-09T14:46:00.537 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.534+0000 7f29f37fe700 1 -- 172.21.15.38:0/302997020 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) v1 ==== 104+0+11 (secure 0 0 0) 0x7f29f40cb020 con 0x7f2a04100910 2023-03-09T14:46:00.537 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 32 2023-03-09T14:46:00.539 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.537+0000 7f2a08a81700 1 -- 172.21.15.38:0/302997020 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f29ec09f3c0 msgr2=0x7f29ec0a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:00.539 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.537+0000 7f2a08a81700 1 --2- 172.21.15.38:0/302997020 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f29ec09f3c0 0x7f29ec0a1870 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f29f800b5c0 tx=0x7f29f8005fb0 comp rx=0 tx=0).stop 2023-03-09T14:46:00.539 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.537+0000 7f2a08a81700 1 -- 172.21.15.38:0/302997020 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2a04100910 msgr2=0x7f2a04195c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:00.540 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.537+0000 7f2a08a81700 1 --2- 172.21.15.38:0/302997020 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2a04100910 0x7f2a04195c60 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f29f400d950 tx=0x7f29f400dc60 comp rx=0 tx=0).stop 2023-03-09T14:46:00.540 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.537+0000 7f2a08a81700 1 -- 172.21.15.38:0/302997020 shutdown_connections 2023-03-09T14:46:00.540 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.537+0000 7f2a08a81700 1 --2- 172.21.15.38:0/302997020 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2a04100910 0x7f2a04195c60 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:00.540 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.537+0000 7f2a08a81700 1 --2- 172.21.15.38:0/302997020 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f29ec09f3c0 0x7f29ec0a1870 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:00.540 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.537+0000 7f2a08a81700 1 --2- 172.21.15.38:0/302997020 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2a04105b50 0x7f2a0419d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:00.540 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.537+0000 7f2a08a81700 1 --2- 172.21.15.38:0/302997020 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2a04103230 0x7f2a041961a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:00.540 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.537+0000 7f2a08a81700 1 -- 172.21.15.38:0/302997020 >> 172.21.15.38:0/302997020 conn(0x7f2a040fa3b0 msgr2=0x7f2a04101dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:00.541 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.537+0000 7f2a08a81700 1 -- 172.21.15.38:0/302997020 shutdown_connections 2023-03-09T14:46:00.541 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.537+0000 7f2a08a81700 1 -- 172.21.15.38:0/302997020 wait complete. 2023-03-09T14:46:00.551 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-03-09T14:46:00.682 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.678+0000 7f9c3b8d8700 1 -- 172.21.15.38:0/348799197 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c34103290 msgr2=0x7f9c34105670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:00.682 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.678+0000 7f9c3b8d8700 1 --2- 172.21.15.38:0/348799197 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c34103290 0x7f9c34105670 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f9c24009990 tx=0x7f9c24009ca0 comp rx=0 tx=0).stop 2023-03-09T14:46:00.683 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.678+0000 7f9c3b8d8700 1 -- 172.21.15.38:0/348799197 shutdown_connections 2023-03-09T14:46:00.683 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.678+0000 7f9c3b8d8700 1 --2- 172.21.15.38:0/348799197 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9c34105bb0 0x7f9c34107fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:00.683 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.678+0000 7f9c3b8d8700 1 --2- 172.21.15.38:0/348799197 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9c34100970 0x7f9c34102d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:00.683 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.678+0000 7f9c3b8d8700 1 --2- 172.21.15.38:0/348799197 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c34103290 0x7f9c34105670 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:00.683 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.678+0000 7f9c3b8d8700 1 -- 172.21.15.38:0/348799197 >> 172.21.15.38:0/348799197 conn(0x7f9c340fa3b0 msgr2=0x7f9c340fc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:00.684 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.678+0000 7f9c3b8d8700 1 -- 172.21.15.38:0/348799197 shutdown_connections 2023-03-09T14:46:00.684 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.678+0000 7f9c3b8d8700 1 -- 172.21.15.38:0/348799197 wait complete. 2023-03-09T14:46:00.684 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.679+0000 7f9c3b8d8700 1 Processor -- start 2023-03-09T14:46:00.684 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.679+0000 7f9c3b8d8700 1 -- start start 2023-03-09T14:46:00.684 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.679+0000 7f9c3b8d8700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9c34100970 0x7f9c34195bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:00.684 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.679+0000 7f9c3b8d8700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c34103290 0x7f9c341960f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:00.685 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.679+0000 7f9c3b8d8700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9c34105bb0 0x7f9c3419d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:00.685 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.679+0000 7f9c3b8d8700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c3419d730 con 0x7f9c34103290 2023-03-09T14:46:00.685 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.679+0000 7f9c3b8d8700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c3419d870 con 0x7f9c34105bb0 2023-03-09T14:46:00.685 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.679+0000 7f9c3b8d8700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c3419da00 con 0x7f9c34100970 2023-03-09T14:46:00.685 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.679+0000 7f9c38e73700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c34103290 0x7f9c341960f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:00.685 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.679+0000 7f9c38e73700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c34103290 0x7f9c341960f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38442/0 (socket says 172.21.15.38:38442) 2023-03-09T14:46:00.686 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.679+0000 7f9c38e73700 1 -- 172.21.15.38:0/1782638996 learned_addr learned my addr 172.21.15.38:0/1782638996 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:46:00.686 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.679+0000 7f9c39e75700 1 --2- 172.21.15.38:0/1782638996 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9c34105bb0 0x7f9c3419d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:00.686 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.679+0000 7f9c39674700 1 --2- 172.21.15.38:0/1782638996 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9c34100970 0x7f9c34195bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:00.686 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.679+0000 7f9c38e73700 1 -- 172.21.15.38:0/1782638996 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9c34100970 msgr2=0x7f9c34195bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:00.686 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.679+0000 7f9c38e73700 1 --2- 172.21.15.38:0/1782638996 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9c34100970 0x7f9c34195bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:00.686 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.679+0000 7f9c38e73700 1 -- 172.21.15.38:0/1782638996 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9c34105bb0 msgr2=0x7f9c3419d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:00.686 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.679+0000 7f9c38e73700 1 --2- 172.21.15.38:0/1782638996 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9c34105bb0 0x7f9c3419d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:00.687 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.680+0000 7f9c38e73700 1 -- 172.21.15.38:0/1782638996 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c30009520 con 0x7f9c34103290 2023-03-09T14:46:00.687 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.680+0000 7f9c39e75700 1 --2- 172.21.15.38:0/1782638996 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9c34105bb0 0x7f9c3419d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:46:00.687 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.680+0000 7f9c38e73700 1 --2- 172.21.15.38:0/1782638996 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c34103290 0x7f9c341960f0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f9c240059d0 tx=0x7f9c24005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:00.687 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.680+0000 7f9c2a7fc700 1 -- 172.21.15.38:0/1782638996 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c2401d070 con 0x7f9c34103290 2023-03-09T14:46:00.687 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.680+0000 7f9c3b8d8700 1 -- 172.21.15.38:0/1782638996 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c240095f0 con 0x7f9c34103290 2023-03-09T14:46:00.687 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.680+0000 7f9c2a7fc700 1 -- 172.21.15.38:0/1782638996 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9c2400fc40 con 0x7f9c34103290 2023-03-09T14:46:00.687 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.680+0000 7f9c2a7fc700 1 -- 172.21.15.38:0/1782638996 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c24017650 con 0x7f9c34103290 2023-03-09T14:46:00.687 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.680+0000 7f9c3b8d8700 1 -- 172.21.15.38:0/1782638996 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c3410b800 con 0x7f9c34103290 2023-03-09T14:46:00.688 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.681+0000 7f9c3b8d8700 1 -- 172.21.15.38:0/1782638996 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c340fbf20 con 0x7f9c34103290 2023-03-09T14:46:00.689 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.682+0000 7f9c2a7fc700 1 -- 172.21.15.38:0/1782638996 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) v1 ==== 133198+0+0 (secure 0 0 0) 0x7f9c24029030 con 0x7f9c34103290 2023-03-09T14:46:00.689 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.682+0000 7f9c2a7fc700 1 --2- 172.21.15.38:0/1782638996 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f9c2009f3c0 0x7f9c200a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:00.689 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.683+0000 7f9c2a7fc700 1 -- 172.21.15.38:0/1782638996 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f9c240cbd10 con 0x7f9c34103290 2023-03-09T14:46:00.689 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.683+0000 7f9c39674700 1 --2- 172.21.15.38:0/1782638996 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f9c2009f3c0 0x7f9c200a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:00.689 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.683+0000 7f9c39674700 1 --2- 172.21.15.38:0/1782638996 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f9c2009f3c0 0x7f9c200a1870 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f9c30009d80 tx=0x7f9c30009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:00.689 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.686+0000 7f9c2a7fc700 1 -- 172.21.15.38:0/1782638996 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f9c24095e50 con 0x7f9c34103290 2023-03-09T14:46:00.889 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.886+0000 7f9c3b8d8700 1 -- 172.21.15.38:0/1782638996 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9c34067660 con 0x7f9c34103290 2023-03-09T14:46:00.889 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.887+0000 7f9c2a7fc700 1 -- 172.21.15.38:0/1782638996 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9c24099470 con 0x7f9c34103290 2023-03-09T14:46:00.889 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 64 2023-03-09T14:46:00.891 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.889+0000 7f9c3b8d8700 1 -- 172.21.15.38:0/1782638996 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f9c2009f3c0 msgr2=0x7f9c200a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:00.891 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.889+0000 7f9c3b8d8700 1 --2- 172.21.15.38:0/1782638996 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f9c2009f3c0 0x7f9c200a1870 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f9c30009d80 tx=0x7f9c30009260 comp rx=0 tx=0).stop 2023-03-09T14:46:00.892 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.889+0000 7f9c3b8d8700 1 -- 172.21.15.38:0/1782638996 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c34103290 msgr2=0x7f9c341960f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:00.892 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.889+0000 7f9c3b8d8700 1 --2- 172.21.15.38:0/1782638996 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c34103290 0x7f9c341960f0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f9c240059d0 tx=0x7f9c24005ef0 comp rx=0 tx=0).stop 2023-03-09T14:46:00.892 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.889+0000 7f9c3b8d8700 1 -- 172.21.15.38:0/1782638996 shutdown_connections 2023-03-09T14:46:00.892 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.889+0000 7f9c3b8d8700 1 --2- 172.21.15.38:0/1782638996 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f9c2009f3c0 0x7f9c200a1870 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:00.892 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.889+0000 7f9c3b8d8700 1 --2- 172.21.15.38:0/1782638996 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9c34105bb0 0x7f9c3419d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:00.892 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.889+0000 7f9c3b8d8700 1 --2- 172.21.15.38:0/1782638996 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9c34100970 0x7f9c34195bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:00.892 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.889+0000 7f9c3b8d8700 1 --2- 172.21.15.38:0/1782638996 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9c34103290 0x7f9c341960f0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:00.893 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.889+0000 7f9c3b8d8700 1 -- 172.21.15.38:0/1782638996 >> 172.21.15.38:0/1782638996 conn(0x7f9c340fa3b0 msgr2=0x7f9c341015d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:00.893 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.889+0000 7f9c3b8d8700 1 -- 172.21.15.38:0/1782638996 shutdown_connections 2023-03-09T14:46:00.893 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:00.889+0000 7f9c3b8d8700 1 -- 172.21.15.38:0/1782638996 wait complete. 2023-03-09T14:46:00.904 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-03-09T14:46:01.035 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.032+0000 7f4d8e6d0700 1 -- 172.21.15.38:0/3594470383 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d880ff2a0 msgr2=0x7f4d880ff6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:01.035 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.032+0000 7f4d8e6d0700 1 --2- 172.21.15.38:0/3594470383 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d880ff2a0 0x7f4d880ff6b0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f4d78009990 tx=0x7f4d78009ca0 comp rx=0 tx=0).stop 2023-03-09T14:46:01.036 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.032+0000 7f4d8e6d0700 1 -- 172.21.15.38:0/3594470383 shutdown_connections 2023-03-09T14:46:01.036 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.032+0000 7f4d8e6d0700 1 --2- 172.21.15.38:0/3594470383 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d880ff2a0 0x7f4d880ff6b0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:01.036 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.032+0000 7f4d8e6d0700 1 --2- 172.21.15.38:0/3594470383 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4d88100960 0x7f4d88107200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:01.036 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.032+0000 7f4d8e6d0700 1 --2- 172.21.15.38:0/3594470383 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4d880fffb0 0x7f4d88100420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:01.036 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.032+0000 7f4d8e6d0700 1 -- 172.21.15.38:0/3594470383 >> 172.21.15.38:0/3594470383 conn(0x7f4d880f9b60 msgr2=0x7f4d880fbf70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:01.036 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.032+0000 7f4d8e6d0700 1 -- 172.21.15.38:0/3594470383 shutdown_connections 2023-03-09T14:46:01.036 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.032+0000 7f4d8e6d0700 1 -- 172.21.15.38:0/3594470383 wait complete. 2023-03-09T14:46:01.036 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.033+0000 7f4d8e6d0700 1 Processor -- start 2023-03-09T14:46:01.037 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.033+0000 7f4d8e6d0700 1 -- start start 2023-03-09T14:46:01.038 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.033+0000 7f4d8e6d0700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4d880ff2a0 0x7f4d88194b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:01.039 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.033+0000 7f4d8e6d0700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4d880fffb0 0x7f4d88195050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:01.039 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.033+0000 7f4d8e6d0700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d88100960 0x7f4d8819c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:01.039 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.033+0000 7f4d8e6d0700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8819c690 con 0x7f4d88100960 2023-03-09T14:46:01.039 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.033+0000 7f4d8e6d0700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8819c7d0 con 0x7f4d880ff2a0 2023-03-09T14:46:01.039 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.033+0000 7f4d8e6d0700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8819c960 con 0x7f4d880fffb0 2023-03-09T14:46:01.039 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.034+0000 7f4d8cc6d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d88100960 0x7f4d8819c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:01.039 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.034+0000 7f4d8cc6d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d88100960 0x7f4d8819c0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38458/0 (socket says 172.21.15.38:38458) 2023-03-09T14:46:01.040 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.034+0000 7f4d8cc6d700 1 -- 172.21.15.38:0/1257975182 learned_addr learned my addr 172.21.15.38:0/1257975182 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:46:01.040 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.034+0000 7f4d87fff700 1 --2- 172.21.15.38:0/1257975182 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4d880ff2a0 0x7f4d88194b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:01.040 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.034+0000 7f4d877fe700 1 --2- 172.21.15.38:0/1257975182 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4d880fffb0 0x7f4d88195050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:01.040 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.034+0000 7f4d8cc6d700 1 -- 172.21.15.38:0/1257975182 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4d880fffb0 msgr2=0x7f4d88195050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:01.040 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.034+0000 7f4d8cc6d700 1 --2- 172.21.15.38:0/1257975182 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4d880fffb0 0x7f4d88195050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:01.040 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.034+0000 7f4d8cc6d700 1 -- 172.21.15.38:0/1257975182 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4d880ff2a0 msgr2=0x7f4d88194b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:01.040 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.034+0000 7f4d8cc6d700 1 --2- 172.21.15.38:0/1257975182 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4d880ff2a0 0x7f4d88194b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:01.040 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.034+0000 7f4d8cc6d700 1 -- 172.21.15.38:0/1257975182 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d780095f0 con 0x7f4d88100960 2023-03-09T14:46:01.041 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.034+0000 7f4d87fff700 1 --2- 172.21.15.38:0/1257975182 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4d880ff2a0 0x7f4d88194b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:46:01.041 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.034+0000 7f4d8cc6d700 1 --2- 172.21.15.38:0/1257975182 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d88100960 0x7f4d8819c0c0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f4d8000ef30 tx=0x7f4d8000c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:01.041 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.034+0000 7f4d857fa700 1 -- 172.21.15.38:0/1257975182 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d80018070 con 0x7f4d88100960 2023-03-09T14:46:01.041 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.034+0000 7f4d857fa700 1 -- 172.21.15.38:0/1257975182 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4d8000f040 con 0x7f4d88100960 2023-03-09T14:46:01.041 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.034+0000 7f4d8e6d0700 1 -- 172.21.15.38:0/1257975182 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d8810a6f0 con 0x7f4d88100960 2023-03-09T14:46:01.041 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.034+0000 7f4d857fa700 1 -- 172.21.15.38:0/1257975182 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d80003dc0 con 0x7f4d88100960 2023-03-09T14:46:01.041 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.034+0000 7f4d8e6d0700 1 -- 172.21.15.38:0/1257975182 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d8810a560 con 0x7f4d88100960 2023-03-09T14:46:01.042 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.035+0000 7f4d8e6d0700 1 -- 172.21.15.38:0/1257975182 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d88105d40 con 0x7f4d88100960 2023-03-09T14:46:01.042 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.036+0000 7f4d857fa700 1 -- 172.21.15.38:0/1257975182 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) v1 ==== 133198+0+0 (secure 0 0 0) 0x7f4d8000caa0 con 0x7f4d88100960 2023-03-09T14:46:01.042 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.037+0000 7f4d857fa700 1 --2- 172.21.15.38:0/1257975182 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f4d7009f3c0 0x7f4d700a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:01.042 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.037+0000 7f4d857fa700 1 -- 172.21.15.38:0/1257975182 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f4d800c9ef0 con 0x7f4d88100960 2023-03-09T14:46:01.043 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.037+0000 7f4d87fff700 1 --2- 172.21.15.38:0/1257975182 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f4d7009f3c0 0x7f4d700a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:01.043 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.038+0000 7f4d87fff700 1 --2- 172.21.15.38:0/1257975182 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f4d7009f3c0 0x7f4d700a1870 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f4d78006010 tx=0x7f4d7800b260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:01.043 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.041+0000 7f4d857fa700 1 -- 172.21.15.38:0/1257975182 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f4d80093fd0 con 0x7f4d88100960 2023-03-09T14:46:01.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:00 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/612771049' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-03-09T14:46:01.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:00 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/302997020' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-03-09T14:46:01.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:00 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/1782638996' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-03-09T14:46:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:00 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/612771049' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-03-09T14:46:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:00 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/302997020' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-03-09T14:46:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:00 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/1782638996' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-03-09T14:46:01.243 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.241+0000 7f4d8e6d0700 1 -- 172.21.15.38:0/1257975182 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f4d880613c0 con 0x7f4d88100960 2023-03-09T14:46:01.244 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.241+0000 7f4d857fa700 1 -- 172.21.15.38:0/1257975182 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) v1 ==== 103+0+11 (secure 0 0 0) 0x7f4d800975f0 con 0x7f4d88100960 2023-03-09T14:46:01.244 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 32 2023-03-09T14:46:01.246 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.243+0000 7f4d8e6d0700 1 -- 172.21.15.38:0/1257975182 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f4d7009f3c0 msgr2=0x7f4d700a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:01.246 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.243+0000 7f4d8e6d0700 1 --2- 172.21.15.38:0/1257975182 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f4d7009f3c0 0x7f4d700a1870 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f4d78006010 tx=0x7f4d7800b260 comp rx=0 tx=0).stop 2023-03-09T14:46:01.247 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.243+0000 7f4d8e6d0700 1 -- 172.21.15.38:0/1257975182 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d88100960 msgr2=0x7f4d8819c0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:01.247 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.243+0000 7f4d8e6d0700 1 --2- 172.21.15.38:0/1257975182 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d88100960 0x7f4d8819c0c0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f4d8000ef30 tx=0x7f4d8000c450 comp rx=0 tx=0).stop 2023-03-09T14:46:01.247 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.243+0000 7f4d8e6d0700 1 -- 172.21.15.38:0/1257975182 shutdown_connections 2023-03-09T14:46:01.248 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.243+0000 7f4d8e6d0700 1 --2- 172.21.15.38:0/1257975182 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d88100960 0x7f4d8819c0c0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:01.248 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.243+0000 7f4d8e6d0700 1 --2- 172.21.15.38:0/1257975182 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f4d7009f3c0 0x7f4d700a1870 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:01.248 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.243+0000 7f4d8e6d0700 1 --2- 172.21.15.38:0/1257975182 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4d880ff2a0 0x7f4d88194b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:01.248 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.243+0000 7f4d8e6d0700 1 --2- 172.21.15.38:0/1257975182 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4d880fffb0 0x7f4d88195050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:01.249 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.243+0000 7f4d8e6d0700 1 -- 172.21.15.38:0/1257975182 >> 172.21.15.38:0/1257975182 conn(0x7f4d880f9b60 msgr2=0x7f4d880fce70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:01.249 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.243+0000 7f4d8e6d0700 1 -- 172.21.15.38:0/1257975182 shutdown_connections 2023-03-09T14:46:01.249 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.243+0000 7f4d8e6d0700 1 -- 172.21.15.38:0/1257975182 wait complete. 2023-03-09T14:46:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:00 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/612771049' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-03-09T14:46:01.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:00 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/302997020' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-03-09T14:46:01.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:00 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/1782638996' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-03-09T14:46:01.258 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2023-03-09T14:46:01.258 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-09T14:46:01.385 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.382+0000 7f6902a46700 1 -- 172.21.15.38:0/2172042242 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f68fc100980 msgr2=0x7f68fc102d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:01.385 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.382+0000 7f6902a46700 1 --2- 172.21.15.38:0/2172042242 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f68fc100980 0x7f68fc102d60 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f68ec009990 tx=0x7f68ec009ca0 comp rx=0 tx=0).stop 2023-03-09T14:46:01.385 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.382+0000 7f6902a46700 1 -- 172.21.15.38:0/2172042242 shutdown_connections 2023-03-09T14:46:01.386 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.382+0000 7f6902a46700 1 --2- 172.21.15.38:0/2172042242 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f68fc100980 0x7f68fc102d60 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:01.386 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.382+0000 7f6902a46700 1 --2- 172.21.15.38:0/2172042242 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f68fc105bc0 0x7f68fc107fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:01.386 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.382+0000 7f6902a46700 1 --2- 172.21.15.38:0/2172042242 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f68fc1032a0 0x7f68fc105680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:01.386 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.382+0000 7f6902a46700 1 -- 172.21.15.38:0/2172042242 >> 172.21.15.38:0/2172042242 conn(0x7f68fc0fa380 msgr2=0x7f68fc0fc7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:01.386 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.383+0000 7f6902a46700 1 -- 172.21.15.38:0/2172042242 shutdown_connections 2023-03-09T14:46:01.386 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.383+0000 7f6902a46700 1 -- 172.21.15.38:0/2172042242 wait complete. 2023-03-09T14:46:01.388 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.383+0000 7f6902a46700 1 Processor -- start 2023-03-09T14:46:01.388 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.383+0000 7f6902a46700 1 -- start start 2023-03-09T14:46:01.388 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.383+0000 7f6902a46700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f68fc100980 0x7f68fc1002f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:01.388 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.383+0000 7f6902a46700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f68fc1032a0 0x7f68fc0fe930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:01.389 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.383+0000 7f6902a46700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f68fc105bc0 0x7f68fc0fee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:01.389 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.383+0000 7f6902a46700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68fc0ff610 con 0x7f68fc105bc0 2023-03-09T14:46:01.389 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.384+0000 7f6902a46700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68fc0ff780 con 0x7f68fc1032a0 2023-03-09T14:46:01.389 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.384+0000 7f6902a46700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68fc19fd30 con 0x7f68fc100980 2023-03-09T14:46:01.389 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.384+0000 7f6900fe3700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f68fc105bc0 0x7f68fc0fee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:01.389 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.384+0000 7f6900fe3700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f68fc105bc0 0x7f68fc0fee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38470/0 (socket says 172.21.15.38:38470) 2023-03-09T14:46:01.390 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.384+0000 7f6900fe3700 1 -- 172.21.15.38:0/3862024022 learned_addr learned my addr 172.21.15.38:0/3862024022 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:46:01.390 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.384+0000 7f68fb7fe700 1 --2- 172.21.15.38:0/3862024022 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f68fc1032a0 0x7f68fc0fe930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:01.390 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.384+0000 7f68fbfff700 1 --2- 172.21.15.38:0/3862024022 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f68fc100980 0x7f68fc1002f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:01.390 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.384+0000 7f6900fe3700 1 -- 172.21.15.38:0/3862024022 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f68fc100980 msgr2=0x7f68fc1002f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:01.390 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.384+0000 7f6900fe3700 1 --2- 172.21.15.38:0/3862024022 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f68fc100980 0x7f68fc1002f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:01.390 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.384+0000 7f6900fe3700 1 -- 172.21.15.38:0/3862024022 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f68fc1032a0 msgr2=0x7f68fc0fe930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:01.390 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.384+0000 7f6900fe3700 1 --2- 172.21.15.38:0/3862024022 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f68fc1032a0 0x7f68fc0fe930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:01.391 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.384+0000 7f6900fe3700 1 -- 172.21.15.38:0/3862024022 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68ec0095f0 con 0x7f68fc105bc0 2023-03-09T14:46:01.391 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.384+0000 7f68fb7fe700 1 --2- 172.21.15.38:0/3862024022 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f68fc1032a0 0x7f68fc0fe930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:46:01.391 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.384+0000 7f6900fe3700 1 --2- 172.21.15.38:0/3862024022 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f68fc105bc0 0x7f68fc0fee70 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f68f4007d70 tx=0x7f68f400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:01.391 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.384+0000 7f68f97fa700 1 -- 172.21.15.38:0/3862024022 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68f400ad40 con 0x7f68fc105bc0 2023-03-09T14:46:01.391 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.384+0000 7f68f97fa700 1 -- 172.21.15.38:0/3862024022 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f68f40128e0 con 0x7f68fc105bc0 2023-03-09T14:46:01.391 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.384+0000 7f6902a46700 1 -- 172.21.15.38:0/3862024022 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68fc10b7c0 con 0x7f68fc105bc0 2023-03-09T14:46:01.391 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.384+0000 7f68f97fa700 1 -- 172.21.15.38:0/3862024022 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68f4011780 con 0x7f68fc105bc0 2023-03-09T14:46:01.391 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.384+0000 7f6902a46700 1 -- 172.21.15.38:0/3862024022 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68fc10b630 con 0x7f68fc105bc0 2023-03-09T14:46:01.392 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.385+0000 7f6902a46700 1 -- 172.21.15.38:0/3862024022 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68fc0fbef0 con 0x7f68fc105bc0 2023-03-09T14:46:01.392 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.386+0000 7f68f97fa700 1 -- 172.21.15.38:0/3862024022 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) v1 ==== 133198+0+0 (secure 0 0 0) 0x7f68f4022070 con 0x7f68fc105bc0 2023-03-09T14:46:01.392 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.387+0000 7f68f97fa700 1 --2- 172.21.15.38:0/3862024022 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f68e409f3c0 0x7f68e40a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:01.393 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.387+0000 7f68fbfff700 1 --2- 172.21.15.38:0/3862024022 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f68e409f3c0 0x7f68e40a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:01.394 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.388+0000 7f68f97fa700 1 -- 172.21.15.38:0/3862024022 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f68f40ca330 con 0x7f68fc105bc0 2023-03-09T14:46:01.394 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.388+0000 7f68fbfff700 1 --2- 172.21.15.38:0/3862024022 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f68e409f3c0 0x7f68e40a1870 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f68ec006010 tx=0x7f68ec00b540 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:01.395 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.392+0000 7f68f97fa700 1 -- 172.21.15.38:0/3862024022 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f68f40943e0 con 0x7f68fc105bc0 2023-03-09T14:46:01.641 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.639+0000 7f6902a46700 1 -- 172.21.15.38:0/3862024022 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f68fc02cef0 con 0x7f68fc105bc0 2023-03-09T14:46:01.642 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.640+0000 7f68f97fa700 1 -- 172.21.15.38:0/3862024022 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+10801 (secure 0 0 0) 0x7f68f4016070 con 0x7f68fc105bc0 2023-03-09T14:46:01.642 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:46:01.642 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/768918602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":768918602},{"type":"v1","addr":"172.21.15.38:6835","nonce":768918602}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/18381540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":18381540},{"type":"v1","addr":"172.21.15.38:6837","nonce":18381540}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14610,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6839/4139020066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":4139020066},{"type":"v1","addr":"172.21.15.38:6839","nonce":4139020066}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14622,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6841/1397115322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":1397115322},{"type":"v1","addr":"172.21.15.38:6841","nonce":1397115322}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6833/415305571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":415305571},{"type":"v1","addr":"172.21.15.181:6833","nonce":415305571}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/659300520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":659300520},{"type":"v1","addr":"172.21.15.122:6833","nonce":659300520}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-03-09T14:44:19.083770+0000","modified":"2023-03-09T14:44:39.526150+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24530,"mds_1":24494,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24448":{"gid":24448,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6835/2576255861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":2576255861},{"type":"v1","addr":"172.21.15.181:6835","nonce":2576255861}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24460":{"gid":24460,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6837/2942560181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":2942560181},{"type":"v1","addr":"172.21.15.181:6837","nonce":2942560181}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24464":{"gid":24464,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.122:6835/3675537941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":3675537941},{"type":"v1","addr":"172.21.15.122:6835","nonce":3675537941}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24478":{"gid":24478,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":9,"addr":"172.21.15.181:6839/3902263660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":3902263660},{"type":"v1","addr":"172.21.15.181:6839","nonce":3902263660}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24494":{"gid":24494,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.122:6837/2210214596","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":2210214596},{"type":"v1","addr":"172.21.15.122:6837","nonce":2210214596}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24530":{"gid":24530,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6839/971050165","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6838","nonce":971050165},{"type":"v1","addr":"172.21.15.122:6839","nonce":971050165}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-09T14:46:01.644 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.642+0000 7f6902a46700 1 -- 172.21.15.38:0/3862024022 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f68e409f3c0 msgr2=0x7f68e40a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:01.644 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.642+0000 7f6902a46700 1 --2- 172.21.15.38:0/3862024022 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f68e409f3c0 0x7f68e40a1870 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f68ec006010 tx=0x7f68ec00b540 comp rx=0 tx=0).stop 2023-03-09T14:46:01.645 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.642+0000 7f6902a46700 1 -- 172.21.15.38:0/3862024022 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f68fc105bc0 msgr2=0x7f68fc0fee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:01.645 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.642+0000 7f6902a46700 1 --2- 172.21.15.38:0/3862024022 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f68fc105bc0 0x7f68fc0fee70 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f68f4007d70 tx=0x7f68f400a340 comp rx=0 tx=0).stop 2023-03-09T14:46:01.645 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.642+0000 7f6902a46700 1 -- 172.21.15.38:0/3862024022 shutdown_connections 2023-03-09T14:46:01.645 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.642+0000 7f6902a46700 1 --2- 172.21.15.38:0/3862024022 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f68fc105bc0 0x7f68fc0fee70 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:01.645 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.642+0000 7f6902a46700 1 --2- 172.21.15.38:0/3862024022 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f68e409f3c0 0x7f68e40a1870 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:01.645 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.642+0000 7f6902a46700 1 --2- 172.21.15.38:0/3862024022 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f68fc100980 0x7f68fc1002f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:01.645 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.642+0000 7f6902a46700 1 --2- 172.21.15.38:0/3862024022 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f68fc1032a0 0x7f68fc0fe930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:01.646 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.642+0000 7f6902a46700 1 -- 172.21.15.38:0/3862024022 >> 172.21.15.38:0/3862024022 conn(0x7f68fc0fa380 msgr2=0x7f68fc1015e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:01.646 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.642+0000 7f6902a46700 1 -- 172.21.15.38:0/3862024022 shutdown_connections 2023-03-09T14:46:01.646 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.642+0000 7f6902a46700 1 -- 172.21.15.38:0/3862024022 wait complete. 2023-03-09T14:46:01.646 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 23 2023-03-09T14:46:01.657 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2023-03-09T14:46:01.657 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2023-03-09T14:46:01.658 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-03-09T14:46:01.783 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.781+0000 7f0d9cec2700 1 -- 172.21.15.38:0/3788346275 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0d980fed80 msgr2=0x7f0d980ff190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:01.784 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.781+0000 7f0d9cec2700 1 --2- 172.21.15.38:0/3788346275 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0d980fed80 0x7f0d980ff190 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f0d88009990 tx=0x7f0d88009ca0 comp rx=0 tx=0).stop 2023-03-09T14:46:01.784 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.781+0000 7f0d9cec2700 1 -- 172.21.15.38:0/3788346275 shutdown_connections 2023-03-09T14:46:01.784 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.781+0000 7f0d9cec2700 1 --2- 172.21.15.38:0/3788346275 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0d980fed80 0x7f0d980ff190 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:01.784 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.781+0000 7f0d9cec2700 1 --2- 172.21.15.38:0/3788346275 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0d98100900 0x7f0d981071c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:01.784 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.781+0000 7f0d9cec2700 1 --2- 172.21.15.38:0/3788346275 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0d980fff70 0x7f0d981003c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:01.785 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.781+0000 7f0d9cec2700 1 -- 172.21.15.38:0/3788346275 >> 172.21.15.38:0/3788346275 conn(0x7f0d980fa380 msgr2=0x7f0d980fc7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:01.785 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.781+0000 7f0d9cec2700 1 -- 172.21.15.38:0/3788346275 shutdown_connections 2023-03-09T14:46:01.785 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.781+0000 7f0d9cec2700 1 -- 172.21.15.38:0/3788346275 wait complete. 2023-03-09T14:46:01.787 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.782+0000 7f0d9cec2700 1 Processor -- start 2023-03-09T14:46:01.787 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.782+0000 7f0d9cec2700 1 -- start start 2023-03-09T14:46:01.787 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.782+0000 7f0d9cec2700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0d980fed80 0x7f0d98195c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:01.787 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.782+0000 7f0d9cec2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0d980fff70 0x7f0d981961a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:01.787 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.782+0000 7f0d9cec2700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0d98100900 0x7f0d9819d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:01.787 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.782+0000 7f0d9cec2700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d9819d7a0 con 0x7f0d980fff70 2023-03-09T14:46:01.788 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.782+0000 7f0d9cec2700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d9819d910 con 0x7f0d98100900 2023-03-09T14:46:01.788 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.782+0000 7f0d9cec2700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d9819dad0 con 0x7f0d980fed80 2023-03-09T14:46:01.788 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.782+0000 7f0d95d9b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0d980fff70 0x7f0d981961a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:01.788 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.782+0000 7f0d95d9b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0d980fff70 0x7f0d981961a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38484/0 (socket says 172.21.15.38:38484) 2023-03-09T14:46:01.788 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.782+0000 7f0d95d9b700 1 -- 172.21.15.38:0/3571021306 learned_addr learned my addr 172.21.15.38:0/3571021306 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:46:01.788 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.782+0000 7f0d96d9d700 1 --2- 172.21.15.38:0/3571021306 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0d98100900 0x7f0d9819d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:01.788 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.782+0000 7f0d9659c700 1 --2- 172.21.15.38:0/3571021306 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0d980fed80 0x7f0d98195c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:01.788 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.782+0000 7f0d95d9b700 1 -- 172.21.15.38:0/3571021306 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0d980fed80 msgr2=0x7f0d98195c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:01.789 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.782+0000 7f0d95d9b700 1 --2- 172.21.15.38:0/3571021306 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0d980fed80 0x7f0d98195c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:01.789 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.782+0000 7f0d95d9b700 1 -- 172.21.15.38:0/3571021306 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0d98100900 msgr2=0x7f0d9819d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:01.789 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.783+0000 7f0d95d9b700 1 --2- 172.21.15.38:0/3571021306 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0d98100900 0x7f0d9819d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:01.789 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.783+0000 7f0d95d9b700 1 -- 172.21.15.38:0/3571021306 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d880095f0 con 0x7f0d980fff70 2023-03-09T14:46:01.789 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.783+0000 7f0d96d9d700 1 --2- 172.21.15.38:0/3571021306 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0d98100900 0x7f0d9819d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:46:01.789 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.783+0000 7f0d95d9b700 1 --2- 172.21.15.38:0/3571021306 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0d980fff70 0x7f0d981961a0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c00d980 tx=0x7f0d8c00dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:01.789 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.783+0000 7f0d877fe700 1 -- 172.21.15.38:0/3571021306 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d8c00bb60 con 0x7f0d980fff70 2023-03-09T14:46:01.790 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.783+0000 7f0d9cec2700 1 -- 172.21.15.38:0/3571021306 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d9810a6e0 con 0x7f0d980fff70 2023-03-09T14:46:01.790 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.783+0000 7f0d877fe700 1 -- 172.21.15.38:0/3571021306 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0d8c007640 con 0x7f0d980fff70 2023-03-09T14:46:01.790 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.783+0000 7f0d877fe700 1 -- 172.21.15.38:0/3571021306 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d8c004e90 con 0x7f0d980fff70 2023-03-09T14:46:01.790 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.783+0000 7f0d9cec2700 1 -- 172.21.15.38:0/3571021306 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d9810a550 con 0x7f0d980fff70 2023-03-09T14:46:01.790 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.784+0000 7f0d9cec2700 1 -- 172.21.15.38:0/3571021306 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d98106eb0 con 0x7f0d980fff70 2023-03-09T14:46:01.791 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.785+0000 7f0d877fe700 1 -- 172.21.15.38:0/3571021306 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) v1 ==== 133198+0+0 (secure 0 0 0) 0x7f0d8c021070 con 0x7f0d980fff70 2023-03-09T14:46:01.791 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.785+0000 7f0d877fe700 1 --2- 172.21.15.38:0/3571021306 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f0d8009f3c0 0x7f0d800a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:01.791 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.786+0000 7f0d877fe700 1 -- 172.21.15.38:0/3571021306 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f0d8c0c9f10 con 0x7f0d980fff70 2023-03-09T14:46:01.791 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.786+0000 7f0d9659c700 1 --2- 172.21.15.38:0/3571021306 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f0d8009f3c0 0x7f0d800a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:01.791 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.786+0000 7f0d9659c700 1 --2- 172.21.15.38:0/3571021306 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f0d8009f3c0 0x7f0d800a1870 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f0d88006010 tx=0x7f0d8800b540 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:01.792 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.790+0000 7f0d877fe700 1 -- 172.21.15.38:0/3571021306 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f0d8c093fc0 con 0x7f0d980fff70 2023-03-09T14:46:01.988 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:01.985+0000 7f0d9cec2700 1 -- 172.21.15.38:0/3571021306 --> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0d980618c0 con 0x7f0d8009f3c0 2023-03-09T14:46:02.003 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.001+0000 7f0d877fe700 1 -- 172.21.15.38:0/3571021306 <== mgr.14150 v2:172.21.15.38:6800/3328887328 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+427321 (secure 0 0 0) 0x7f0d980618c0 con 0x7f0d8009f3c0 2023-03-09T14:46:02.008 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:46:02.010 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.005+0000 7f0d9cec2700 1 -- 172.21.15.38:0/3571021306 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f0d8009f3c0 msgr2=0x7f0d800a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:02.011 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.005+0000 7f0d9cec2700 1 --2- 172.21.15.38:0/3571021306 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f0d8009f3c0 0x7f0d800a1870 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f0d88006010 tx=0x7f0d8800b540 comp rx=0 tx=0).stop 2023-03-09T14:46:02.011 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.005+0000 7f0d9cec2700 1 -- 172.21.15.38:0/3571021306 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0d980fff70 msgr2=0x7f0d981961a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:02.011 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.005+0000 7f0d9cec2700 1 --2- 172.21.15.38:0/3571021306 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0d980fff70 0x7f0d981961a0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c00d980 tx=0x7f0d8c00dd40 comp rx=0 tx=0).stop 2023-03-09T14:46:02.011 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.005+0000 7f0d9cec2700 1 -- 172.21.15.38:0/3571021306 shutdown_connections 2023-03-09T14:46:02.012 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.005+0000 7f0d9cec2700 1 --2- 172.21.15.38:0/3571021306 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f0d8009f3c0 0x7f0d800a1870 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:02.012 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.005+0000 7f0d9cec2700 1 --2- 172.21.15.38:0/3571021306 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0d98100900 0x7f0d9819d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:02.012 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.005+0000 7f0d9cec2700 1 --2- 172.21.15.38:0/3571021306 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0d980fed80 0x7f0d98195c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:02.012 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.005+0000 7f0d9cec2700 1 --2- 172.21.15.38:0/3571021306 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0d980fff70 0x7f0d981961a0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:02.012 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.005+0000 7f0d9cec2700 1 -- 172.21.15.38:0/3571021306 >> 172.21.15.38:0/3571021306 conn(0x7f0d980fa380 msgr2=0x7f0d980fc620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:02.013 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.005+0000 7f0d9cec2700 1 -- 172.21.15.38:0/3571021306 shutdown_connections 2023-03-09T14:46:02.013 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.005+0000 7f0d9cec2700 1 -- 172.21.15.38:0/3571021306 wait complete. 2023-03-09T14:46:02.013 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2023-03-09T14:46:02.020 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":312,"stamp":"2023-03-09T14:46:00.782608+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":656228,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":642,"num_read_kb":445,"num_write":346,"num_write_kb":3056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1182773,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":782,"ondisk_log_size":782,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":83828,"kb_used_data":13780,"kb_used_omap":0,"kb_used_meta":69952,"kb_avail":1124857996,"statfs":{"total":1151940427776,"available":1151854587904,"internally_reserved":0,"allocated":14110720,"data_stored":7829731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":71630848},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4606,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":82,"num_read_kb":51,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003330"},"pg_stats":[{"pgid":"3.3f","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.799653+0000","last_change":"2023-03-09T14:44:19.116371+0000","last_active":"2023-03-09T14:45:44.799653+0000","last_peered":"2023-03-09T14:45:44.799653+0000","last_clean":"2023-03-09T14:45:44.799653+0000","last_became_active":"2023-03-09T14:44:19.116024+0000","last_became_peered":"2023-03-09T14:44:19.116024+0000","last_unstale":"2023-03-09T14:45:44.799653+0000","last_undegraded":"2023-03-09T14:45:44.799653+0000","last_fullsized":"2023-03-09T14:45:44.799653+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T16:43:33.917176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"89'4","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.788538+0000","last_change":"2023-03-09T14:44:19.117312+0000","last_active":"2023-03-09T14:45:44.788538+0000","last_peered":"2023-03-09T14:45:44.788538+0000","last_clean":"2023-03-09T14:45:44.788538+0000","last_became_active":"2023-03-09T14:44:19.116920+0000","last_became_peered":"2023-03-09T14:44:19.116920+0000","last_unstale":"2023-03-09T14:45:44.788538+0000","last_undegraded":"2023-03-09T14:45:44.788538+0000","last_fullsized":"2023-03-09T14:45:44.788538+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:33:15.046431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.798813+0000","last_change":"2023-03-09T14:44:19.113482+0000","last_active":"2023-03-09T14:45:44.798813+0000","last_peered":"2023-03-09T14:45:44.798813+0000","last_clean":"2023-03-09T14:45:44.798813+0000","last_became_active":"2023-03-09T14:44:19.113183+0000","last_became_peered":"2023-03-09T14:44:19.113183+0000","last_unstale":"2023-03-09T14:45:44.798813+0000","last_undegraded":"2023-03-09T14:45:44.798813+0000","last_fullsized":"2023-03-09T14:45:44.798813+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:17:31.382255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.786039+0000","last_change":"2023-03-09T14:44:19.105799+0000","last_active":"2023-03-09T14:45:44.786039+0000","last_peered":"2023-03-09T14:45:44.786039+0000","last_clean":"2023-03-09T14:45:44.786039+0000","last_became_active":"2023-03-09T14:44:19.105405+0000","last_became_peered":"2023-03-09T14:44:19.105405+0000","last_unstale":"2023-03-09T14:45:44.786039+0000","last_undegraded":"2023-03-09T14:45:44.786039+0000","last_fullsized":"2023-03-09T14:45:44.786039+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:38:23.049984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"89'4","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.782474+0000","last_change":"2023-03-09T14:44:19.100656+0000","last_active":"2023-03-09T14:45:44.782474+0000","last_peered":"2023-03-09T14:45:44.782474+0000","last_clean":"2023-03-09T14:45:44.782474+0000","last_became_active":"2023-03-09T14:44:19.100278+0000","last_became_peered":"2023-03-09T14:44:19.100278+0000","last_unstale":"2023-03-09T14:45:44.782474+0000","last_undegraded":"2023-03-09T14:45:44.782474+0000","last_fullsized":"2023-03-09T14:45:44.782474+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:25:39.240251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.787745+0000","last_change":"2023-03-09T14:44:19.101065+0000","last_active":"2023-03-09T14:45:44.787745+0000","last_peered":"2023-03-09T14:45:44.787745+0000","last_clean":"2023-03-09T14:45:44.787745+0000","last_became_active":"2023-03-09T14:44:19.100451+0000","last_became_peered":"2023-03-09T14:44:19.100451+0000","last_unstale":"2023-03-09T14:45:44.787745+0000","last_undegraded":"2023-03-09T14:45:44.787745+0000","last_fullsized":"2023-03-09T14:45:44.787745+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T01:25:15.158254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"89'5","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.787475+0000","last_change":"2023-03-09T14:44:19.107996+0000","last_active":"2023-03-09T14:45:44.787475+0000","last_peered":"2023-03-09T14:45:44.787475+0000","last_clean":"2023-03-09T14:45:44.787475+0000","last_became_active":"2023-03-09T14:44:19.107683+0000","last_became_peered":"2023-03-09T14:44:19.107683+0000","last_unstale":"2023-03-09T14:45:44.787475+0000","last_undegraded":"2023-03-09T14:45:44.787475+0000","last_fullsized":"2023-03-09T14:45:44.787475+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:16:18.605176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.786169+0000","last_change":"2023-03-09T14:44:19.103940+0000","last_active":"2023-03-09T14:45:44.786169+0000","last_peered":"2023-03-09T14:45:44.786169+0000","last_clean":"2023-03-09T14:45:44.786169+0000","last_became_active":"2023-03-09T14:44:19.103620+0000","last_became_peered":"2023-03-09T14:44:19.103620+0000","last_unstale":"2023-03-09T14:45:44.786169+0000","last_undegraded":"2023-03-09T14:45:44.786169+0000","last_fullsized":"2023-03-09T14:45:44.786169+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:06:22.749636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"89'11","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.797150+0000","last_change":"2023-03-09T14:44:19.102695+0000","last_active":"2023-03-09T14:45:44.797150+0000","last_peered":"2023-03-09T14:45:44.797150+0000","last_clean":"2023-03-09T14:45:44.797150+0000","last_became_active":"2023-03-09T14:44:19.101486+0000","last_became_peered":"2023-03-09T14:44:19.101486+0000","last_unstale":"2023-03-09T14:45:44.797150+0000","last_undegraded":"2023-03-09T14:45:44.797150+0000","last_fullsized":"2023-03-09T14:45:44.797150+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T16:34:51.836443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"89'16","reported_seq":33,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.788716+0000","last_change":"2023-03-09T14:44:19.103522+0000","last_active":"2023-03-09T14:45:44.788716+0000","last_peered":"2023-03-09T14:45:44.788716+0000","last_clean":"2023-03-09T14:45:44.788716+0000","last_became_active":"2023-03-09T14:44:19.103242+0000","last_became_peered":"2023-03-09T14:44:19.103242+0000","last_unstale":"2023-03-09T14:45:44.788716+0000","last_undegraded":"2023-03-09T14:45:44.788716+0000","last_fullsized":"2023-03-09T14:45:44.788716+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:41:03.137554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.799291+0000","last_change":"2023-03-09T14:44:19.114934+0000","last_active":"2023-03-09T14:45:44.799291+0000","last_peered":"2023-03-09T14:45:44.799291+0000","last_clean":"2023-03-09T14:45:44.799291+0000","last_became_active":"2023-03-09T14:44:19.114620+0000","last_became_peered":"2023-03-09T14:44:19.114620+0000","last_unstale":"2023-03-09T14:45:44.799291+0000","last_undegraded":"2023-03-09T14:45:44.799291+0000","last_fullsized":"2023-03-09T14:45:44.799291+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T01:48:40.082076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.785702+0000","last_change":"2023-03-09T14:44:19.107677+0000","last_active":"2023-03-09T14:45:44.785702+0000","last_peered":"2023-03-09T14:45:44.785702+0000","last_clean":"2023-03-09T14:45:44.785702+0000","last_became_active":"2023-03-09T14:44:19.107369+0000","last_became_peered":"2023-03-09T14:44:19.107369+0000","last_unstale":"2023-03-09T14:45:44.785702+0000","last_undegraded":"2023-03-09T14:45:44.785702+0000","last_fullsized":"2023-03-09T14:45:44.785702+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:47:40.117205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"89'10","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.786986+0000","last_change":"2023-03-09T14:44:19.108852+0000","last_active":"2023-03-09T14:45:44.786986+0000","last_peered":"2023-03-09T14:45:44.786986+0000","last_clean":"2023-03-09T14:45:44.786986+0000","last_became_active":"2023-03-09T14:44:19.108515+0000","last_became_peered":"2023-03-09T14:44:19.108515+0000","last_unstale":"2023-03-09T14:45:44.786986+0000","last_undegraded":"2023-03-09T14:45:44.786986+0000","last_fullsized":"2023-03-09T14:45:44.786986+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:00:36.243952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.786886+0000","last_change":"2023-03-09T14:44:19.118673+0000","last_active":"2023-03-09T14:45:44.786886+0000","last_peered":"2023-03-09T14:45:44.786886+0000","last_clean":"2023-03-09T14:45:44.786886+0000","last_became_active":"2023-03-09T14:44:19.117656+0000","last_became_peered":"2023-03-09T14:44:19.117656+0000","last_unstale":"2023-03-09T14:45:44.786886+0000","last_undegraded":"2023-03-09T14:45:44.786886+0000","last_fullsized":"2023-03-09T14:45:44.786886+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:02:50.401680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.789818+0000","last_change":"2023-03-09T14:44:19.112979+0000","last_active":"2023-03-09T14:45:44.789818+0000","last_peered":"2023-03-09T14:45:44.789818+0000","last_clean":"2023-03-09T14:45:44.789818+0000","last_became_active":"2023-03-09T14:44:19.112671+0000","last_became_peered":"2023-03-09T14:44:19.112671+0000","last_unstale":"2023-03-09T14:45:44.789818+0000","last_undegraded":"2023-03-09T14:45:44.789818+0000","last_fullsized":"2023-03-09T14:45:44.789818+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:00:43.465327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.795403+0000","last_change":"2023-03-09T14:44:19.104881+0000","last_active":"2023-03-09T14:45:44.795403+0000","last_peered":"2023-03-09T14:45:44.795403+0000","last_clean":"2023-03-09T14:45:44.795403+0000","last_became_active":"2023-03-09T14:44:19.104647+0000","last_became_peered":"2023-03-09T14:44:19.104647+0000","last_unstale":"2023-03-09T14:45:44.795403+0000","last_undegraded":"2023-03-09T14:45:44.795403+0000","last_fullsized":"2023-03-09T14:45:44.795403+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T20:16:24.625362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.799646+0000","last_change":"2023-03-09T14:44:19.117937+0000","last_active":"2023-03-09T14:45:44.799646+0000","last_peered":"2023-03-09T14:45:44.799646+0000","last_clean":"2023-03-09T14:45:44.799646+0000","last_became_active":"2023-03-09T14:44:19.117628+0000","last_became_peered":"2023-03-09T14:44:19.117628+0000","last_unstale":"2023-03-09T14:45:44.799646+0000","last_undegraded":"2023-03-09T14:45:44.799646+0000","last_fullsized":"2023-03-09T14:45:44.799646+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:22:18.619171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"89'10","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.793895+0000","last_change":"2023-03-09T14:44:19.105071+0000","last_active":"2023-03-09T14:45:44.793895+0000","last_peered":"2023-03-09T14:45:44.793895+0000","last_clean":"2023-03-09T14:45:44.793895+0000","last_became_active":"2023-03-09T14:44:19.103396+0000","last_became_peered":"2023-03-09T14:44:19.103396+0000","last_unstale":"2023-03-09T14:45:44.793895+0000","last_undegraded":"2023-03-09T14:45:44.793895+0000","last_fullsized":"2023-03-09T14:45:44.793895+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:14:51.924636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.783812+0000","last_change":"2023-03-09T14:44:19.104107+0000","last_active":"2023-03-09T14:45:44.783812+0000","last_peered":"2023-03-09T14:45:44.783812+0000","last_clean":"2023-03-09T14:45:44.783812+0000","last_became_active":"2023-03-09T14:44:19.103793+0000","last_became_peered":"2023-03-09T14:44:19.103793+0000","last_unstale":"2023-03-09T14:45:44.783812+0000","last_undegraded":"2023-03-09T14:45:44.783812+0000","last_fullsized":"2023-03-09T14:45:44.783812+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:57:29.169744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.856426+0000","last_change":"2023-03-09T14:44:23.489947+0000","last_active":"2023-03-09T14:45:14.856426+0000","last_peered":"2023-03-09T14:45:14.856426+0000","last_clean":"2023-03-09T14:45:14.856426+0000","last_became_active":"2023-03-09T14:44:23.489560+0000","last_became_peered":"2023-03-09T14:44:23.489560+0000","last_unstale":"2023-03-09T14:45:14.856426+0000","last_undegraded":"2023-03-09T14:45:14.856426+0000","last_fullsized":"2023-03-09T14:45:14.856426+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:06:10.202176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"87'9","reported_seq":49,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:15.961952+0000","last_change":"2023-03-09T14:45:15.961952+0000","last_active":"2023-03-09T14:45:15.961952+0000","last_peered":"2023-03-09T14:45:15.961952+0000","last_clean":"2023-03-09T14:45:15.961952+0000","last_became_active":"2023-03-09T14:45:15.865191+0000","last_became_peered":"2023-03-09T14:45:15.865191+0000","last_unstale":"2023-03-09T14:45:15.961952+0000","last_undegraded":"2023-03-09T14:45:15.961952+0000","last_fullsized":"2023-03-09T14:45:15.961952+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T01:21:05.925138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":566,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"89'5","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.785218+0000","last_change":"2023-03-09T14:44:19.105390+0000","last_active":"2023-03-09T14:45:44.785218+0000","last_peered":"2023-03-09T14:45:44.785218+0000","last_clean":"2023-03-09T14:45:44.785218+0000","last_became_active":"2023-03-09T14:44:19.104694+0000","last_became_peered":"2023-03-09T14:44:19.104694+0000","last_unstale":"2023-03-09T14:45:44.785218+0000","last_undegraded":"2023-03-09T14:45:44.785218+0000","last_fullsized":"2023-03-09T14:45:44.785218+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:19:08.101847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.856827+0000","last_change":"2023-03-09T14:44:23.484103+0000","last_active":"2023-03-09T14:45:14.856827+0000","last_peered":"2023-03-09T14:45:14.856827+0000","last_clean":"2023-03-09T14:45:14.856827+0000","last_became_active":"2023-03-09T14:44:23.483271+0000","last_became_peered":"2023-03-09T14:44:23.483271+0000","last_unstale":"2023-03-09T14:45:14.856827+0000","last_undegraded":"2023-03-09T14:45:14.856827+0000","last_fullsized":"2023-03-09T14:45:14.856827+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:24:06.212623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"87'2","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.864060+0000","last_change":"2023-03-09T14:44:18.667321+0000","last_active":"2023-03-09T14:45:14.864060+0000","last_peered":"2023-03-09T14:45:14.864060+0000","last_clean":"2023-03-09T14:45:14.864060+0000","last_became_active":"2023-03-09T14:44:18.666974+0000","last_became_peered":"2023-03-09T14:44:18.666974+0000","last_unstale":"2023-03-09T14:45:14.864060+0000","last_undegraded":"2023-03-09T14:45:14.864060+0000","last_fullsized":"2023-03-09T14:45:14.864060+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:31:07.327792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.799185+0000","last_change":"2023-03-09T14:44:19.116866+0000","last_active":"2023-03-09T14:45:44.799185+0000","last_peered":"2023-03-09T14:45:44.799185+0000","last_clean":"2023-03-09T14:45:44.799185+0000","last_became_active":"2023-03-09T14:44:19.116571+0000","last_became_peered":"2023-03-09T14:44:19.116571+0000","last_unstale":"2023-03-09T14:45:44.799185+0000","last_undegraded":"2023-03-09T14:45:44.799185+0000","last_fullsized":"2023-03-09T14:45:44.799185+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T16:22:12.723903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.859406+0000","last_change":"2023-03-09T14:44:23.488192+0000","last_active":"2023-03-09T14:45:14.859406+0000","last_peered":"2023-03-09T14:45:14.859406+0000","last_clean":"2023-03-09T14:45:14.859406+0000","last_became_active":"2023-03-09T14:44:23.487734+0000","last_became_peered":"2023-03-09T14:44:23.487734+0000","last_unstale":"2023-03-09T14:45:14.859406+0000","last_undegraded":"2023-03-09T14:45:14.859406+0000","last_fullsized":"2023-03-09T14:45:14.859406+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:35:50.319216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"87'4","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.861901+0000","last_change":"2023-03-09T14:44:18.663577+0000","last_active":"2023-03-09T14:45:14.861901+0000","last_peered":"2023-03-09T14:45:14.861901+0000","last_clean":"2023-03-09T14:45:14.861901+0000","last_became_active":"2023-03-09T14:44:18.663272+0000","last_became_peered":"2023-03-09T14:44:18.663272+0000","last_unstale":"2023-03-09T14:45:14.861901+0000","last_undegraded":"2023-03-09T14:45:14.861901+0000","last_fullsized":"2023-03-09T14:45:14.861901+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T20:55:19.661875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.794071+0000","last_change":"2023-03-09T14:44:19.106986+0000","last_active":"2023-03-09T14:45:44.794071+0000","last_peered":"2023-03-09T14:45:44.794071+0000","last_clean":"2023-03-09T14:45:44.794071+0000","last_became_active":"2023-03-09T14:44:19.106625+0000","last_became_peered":"2023-03-09T14:44:19.106625+0000","last_unstale":"2023-03-09T14:45:44.794071+0000","last_undegraded":"2023-03-09T14:45:44.794071+0000","last_fullsized":"2023-03-09T14:45:44.794071+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:03:54.521603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.858888+0000","last_change":"2023-03-09T14:44:23.483803+0000","last_active":"2023-03-09T14:45:14.858888+0000","last_peered":"2023-03-09T14:45:14.858888+0000","last_clean":"2023-03-09T14:45:14.858888+0000","last_became_active":"2023-03-09T14:44:23.483286+0000","last_became_peered":"2023-03-09T14:44:23.483286+0000","last_unstale":"2023-03-09T14:45:14.858888+0000","last_undegraded":"2023-03-09T14:45:14.858888+0000","last_fullsized":"2023-03-09T14:45:14.858888+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:32:35.975670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"82'2","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.861301+0000","last_change":"2023-03-09T14:44:18.657685+0000","last_active":"2023-03-09T14:45:14.861301+0000","last_peered":"2023-03-09T14:45:14.861301+0000","last_clean":"2023-03-09T14:45:14.861301+0000","last_became_active":"2023-03-09T14:44:18.657355+0000","last_became_peered":"2023-03-09T14:44:18.657355+0000","last_unstale":"2023-03-09T14:45:14.861301+0000","last_undegraded":"2023-03-09T14:45:14.861301+0000","last_fullsized":"2023-03-09T14:45:14.861301+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:33:59.259690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"89'15","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.799331+0000","last_change":"2023-03-09T14:44:19.111687+0000","last_active":"2023-03-09T14:45:44.799331+0000","last_peered":"2023-03-09T14:45:44.799331+0000","last_clean":"2023-03-09T14:45:44.799331+0000","last_became_active":"2023-03-09T14:44:19.111481+0000","last_became_peered":"2023-03-09T14:44:19.111481+0000","last_unstale":"2023-03-09T14:45:44.799331+0000","last_undegraded":"2023-03-09T14:45:44.799331+0000","last_fullsized":"2023-03-09T14:45:44.799331+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:06:14.452967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.857536+0000","last_change":"2023-03-09T14:44:23.485075+0000","last_active":"2023-03-09T14:45:14.857536+0000","last_peered":"2023-03-09T14:45:14.857536+0000","last_clean":"2023-03-09T14:45:14.857536+0000","last_became_active":"2023-03-09T14:44:23.484648+0000","last_became_peered":"2023-03-09T14:44:23.484648+0000","last_unstale":"2023-03-09T14:45:14.857536+0000","last_undegraded":"2023-03-09T14:45:14.857536+0000","last_fullsized":"2023-03-09T14:45:14.857536+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:49:40.332276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"87'5","reported_seq":120,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:55.926611+0000","last_change":"2023-03-09T14:45:15.868911+0000","last_active":"2023-03-09T14:45:55.926611+0000","last_peered":"2023-03-09T14:45:55.926611+0000","last_clean":"2023-03-09T14:45:55.926611+0000","last_became_active":"2023-03-09T14:45:15.861592+0000","last_became_peered":"2023-03-09T14:45:15.861592+0000","last_unstale":"2023-03-09T14:45:55.926611+0000","last_undegraded":"2023-03-09T14:45:55.926611+0000","last_fullsized":"2023-03-09T14:45:55.926611+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T20:17:55.938860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":80,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":622,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.786994+0000","last_change":"2023-03-09T14:44:19.107572+0000","last_active":"2023-03-09T14:45:44.786994+0000","last_peered":"2023-03-09T14:45:44.786994+0000","last_clean":"2023-03-09T14:45:44.786994+0000","last_became_active":"2023-03-09T14:44:19.106632+0000","last_became_peered":"2023-03-09T14:44:19.106632+0000","last_unstale":"2023-03-09T14:45:44.786994+0000","last_undegraded":"2023-03-09T14:45:44.786994+0000","last_fullsized":"2023-03-09T14:45:44.786994+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T19:04:01.379375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.857041+0000","last_change":"2023-03-09T14:44:23.494002+0000","last_active":"2023-03-09T14:45:14.857041+0000","last_peered":"2023-03-09T14:45:14.857041+0000","last_clean":"2023-03-09T14:45:14.857041+0000","last_became_active":"2023-03-09T14:44:23.493628+0000","last_became_peered":"2023-03-09T14:44:23.493628+0000","last_unstale":"2023-03-09T14:45:14.857041+0000","last_undegraded":"2023-03-09T14:45:14.857041+0000","last_fullsized":"2023-03-09T14:45:14.857041+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:30:47.970556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"87'3","reported_seq":36,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:15.964036+0000","last_change":"2023-03-09T14:45:15.964036+0000","last_active":"2023-03-09T14:45:15.964036+0000","last_peered":"2023-03-09T14:45:15.964036+0000","last_clean":"2023-03-09T14:45:15.964036+0000","last_became_active":"2023-03-09T14:45:15.865998+0000","last_became_peered":"2023-03-09T14:45:15.865998+0000","last_unstale":"2023-03-09T14:45:15.964036+0000","last_undegraded":"2023-03-09T14:45:15.964036+0000","last_fullsized":"2023-03-09T14:45:15.964036+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:16:58.213684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.786013+0000","last_change":"2023-03-09T14:44:19.111303+0000","last_active":"2023-03-09T14:45:44.786013+0000","last_peered":"2023-03-09T14:45:44.786013+0000","last_clean":"2023-03-09T14:45:44.786013+0000","last_became_active":"2023-03-09T14:44:19.111026+0000","last_became_peered":"2023-03-09T14:44:19.111026+0000","last_unstale":"2023-03-09T14:45:44.786013+0000","last_undegraded":"2023-03-09T14:45:44.786013+0000","last_fullsized":"2023-03-09T14:45:44.786013+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T02:26:32.496084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.859059+0000","last_change":"2023-03-09T14:44:23.493272+0000","last_active":"2023-03-09T14:45:14.859059+0000","last_peered":"2023-03-09T14:45:14.859059+0000","last_clean":"2023-03-09T14:45:14.859059+0000","last_became_active":"2023-03-09T14:44:23.492785+0000","last_became_peered":"2023-03-09T14:44:23.492785+0000","last_unstale":"2023-03-09T14:45:14.859059+0000","last_undegraded":"2023-03-09T14:45:14.859059+0000","last_fullsized":"2023-03-09T14:45:14.859059+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T20:18:39.758938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"87'2","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.861264+0000","last_change":"2023-03-09T14:44:18.658179+0000","last_active":"2023-03-09T14:45:14.861264+0000","last_peered":"2023-03-09T14:45:14.861264+0000","last_clean":"2023-03-09T14:45:14.861264+0000","last_became_active":"2023-03-09T14:44:18.657851+0000","last_became_peered":"2023-03-09T14:44:18.657851+0000","last_unstale":"2023-03-09T14:45:14.861264+0000","last_undegraded":"2023-03-09T14:45:14.861264+0000","last_fullsized":"2023-03-09T14:45:14.861264+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:52:49.291615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.782372+0000","last_change":"2023-03-09T14:44:19.104176+0000","last_active":"2023-03-09T14:45:44.782372+0000","last_peered":"2023-03-09T14:45:44.782372+0000","last_clean":"2023-03-09T14:45:44.782372+0000","last_became_active":"2023-03-09T14:44:19.103816+0000","last_became_peered":"2023-03-09T14:44:19.103816+0000","last_unstale":"2023-03-09T14:45:44.782372+0000","last_undegraded":"2023-03-09T14:45:44.782372+0000","last_fullsized":"2023-03-09T14:45:44.782372+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T02:06:41.876169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.864893+0000","last_change":"2023-03-09T14:44:23.490876+0000","last_active":"2023-03-09T14:45:14.864893+0000","last_peered":"2023-03-09T14:45:14.864893+0000","last_clean":"2023-03-09T14:45:14.864893+0000","last_became_active":"2023-03-09T14:44:23.490461+0000","last_became_peered":"2023-03-09T14:44:23.490461+0000","last_unstale":"2023-03-09T14:45:14.864893+0000","last_undegraded":"2023-03-09T14:45:14.864893+0000","last_fullsized":"2023-03-09T14:45:14.864893+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T16:28:40.456613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"87'1","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.852373+0000","last_change":"2023-03-09T14:44:18.662513+0000","last_active":"2023-03-09T14:45:14.852373+0000","last_peered":"2023-03-09T14:45:14.852373+0000","last_clean":"2023-03-09T14:45:14.852373+0000","last_became_active":"2023-03-09T14:44:18.661907+0000","last_became_peered":"2023-03-09T14:44:18.661907+0000","last_unstale":"2023-03-09T14:45:14.852373+0000","last_undegraded":"2023-03-09T14:45:14.852373+0000","last_fullsized":"2023-03-09T14:45:14.852373+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:45:17.373213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"89'11","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.779797+0000","last_change":"2023-03-09T14:44:19.115297+0000","last_active":"2023-03-09T14:45:44.779797+0000","last_peered":"2023-03-09T14:45:44.779797+0000","last_clean":"2023-03-09T14:45:44.779797+0000","last_became_active":"2023-03-09T14:44:19.114973+0000","last_became_peered":"2023-03-09T14:44:19.114973+0000","last_unstale":"2023-03-09T14:45:44.779797+0000","last_undegraded":"2023-03-09T14:45:44.779797+0000","last_fullsized":"2023-03-09T14:45:44.779797+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:11:22.226608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.864500+0000","last_change":"2023-03-09T14:44:23.482772+0000","last_active":"2023-03-09T14:45:14.864500+0000","last_peered":"2023-03-09T14:45:14.864500+0000","last_clean":"2023-03-09T14:45:14.864500+0000","last_became_active":"2023-03-09T14:44:23.481750+0000","last_became_peered":"2023-03-09T14:44:23.481750+0000","last_unstale":"2023-03-09T14:45:14.864500+0000","last_undegraded":"2023-03-09T14:45:14.864500+0000","last_fullsized":"2023-03-09T14:45:14.864500+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T19:26:03.737713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"87'3","reported_seq":119,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:58.929566+0000","last_change":"2023-03-09T14:45:15.870269+0000","last_active":"2023-03-09T14:45:58.929566+0000","last_peered":"2023-03-09T14:45:58.929566+0000","last_clean":"2023-03-09T14:45:58.929566+0000","last_became_active":"2023-03-09T14:45:15.862610+0000","last_became_peered":"2023-03-09T14:45:15.862610+0000","last_unstale":"2023-03-09T14:45:58.929566+0000","last_undegraded":"2023-03-09T14:45:58.929566+0000","last_fullsized":"2023-03-09T14:45:58.929566+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T02:37:23.157022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":81,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,9],"acting":[10,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"89'10","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.799241+0000","last_change":"2023-03-09T14:44:19.100914+0000","last_active":"2023-03-09T14:45:44.799241+0000","last_peered":"2023-03-09T14:45:44.799241+0000","last_clean":"2023-03-09T14:45:44.799241+0000","last_became_active":"2023-03-09T14:44:19.100451+0000","last_became_peered":"2023-03-09T14:44:19.100451+0000","last_unstale":"2023-03-09T14:45:44.799241+0000","last_undegraded":"2023-03-09T14:45:44.799241+0000","last_fullsized":"2023-03-09T14:45:44.799241+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:18:41.852697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.860006+0000","last_change":"2023-03-09T14:44:23.490129+0000","last_active":"2023-03-09T14:45:14.860006+0000","last_peered":"2023-03-09T14:45:14.860006+0000","last_clean":"2023-03-09T14:45:14.860006+0000","last_became_active":"2023-03-09T14:44:23.489696+0000","last_became_peered":"2023-03-09T14:44:23.489696+0000","last_unstale":"2023-03-09T14:45:14.860006+0000","last_undegraded":"2023-03-09T14:45:14.860006+0000","last_fullsized":"2023-03-09T14:45:14.860006+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T16:47:41.943904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"87'2","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:15.955780+0000","last_change":"2023-03-09T14:45:15.955780+0000","last_active":"2023-03-09T14:45:15.955780+0000","last_peered":"2023-03-09T14:45:15.955780+0000","last_clean":"2023-03-09T14:45:15.955780+0000","last_became_active":"2023-03-09T14:45:15.862567+0000","last_became_peered":"2023-03-09T14:45:15.862567+0000","last_unstale":"2023-03-09T14:45:15.955780+0000","last_undegraded":"2023-03-09T14:45:15.955780+0000","last_fullsized":"2023-03-09T14:45:15.955780+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:08:41.748638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.786794+0000","last_change":"2023-03-09T14:44:19.119125+0000","last_active":"2023-03-09T14:45:44.786794+0000","last_peered":"2023-03-09T14:45:44.786794+0000","last_clean":"2023-03-09T14:45:44.786794+0000","last_became_active":"2023-03-09T14:44:19.118677+0000","last_became_peered":"2023-03-09T14:44:19.118677+0000","last_unstale":"2023-03-09T14:45:44.786794+0000","last_undegraded":"2023-03-09T14:45:44.786794+0000","last_fullsized":"2023-03-09T14:45:44.786794+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:06:28.740355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.859670+0000","last_change":"2023-03-09T14:44:23.484752+0000","last_active":"2023-03-09T14:45:14.859670+0000","last_peered":"2023-03-09T14:45:14.859670+0000","last_clean":"2023-03-09T14:45:14.859670+0000","last_became_active":"2023-03-09T14:44:23.484045+0000","last_became_peered":"2023-03-09T14:44:23.484045+0000","last_unstale":"2023-03-09T14:45:14.859670+0000","last_undegraded":"2023-03-09T14:45:14.859670+0000","last_fullsized":"2023-03-09T14:45:14.859670+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:58:39.704017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"87'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.853129+0000","last_change":"2023-03-09T14:44:18.665122+0000","last_active":"2023-03-09T14:45:14.853129+0000","last_peered":"2023-03-09T14:45:14.853129+0000","last_clean":"2023-03-09T14:45:14.853129+0000","last_became_active":"2023-03-09T14:44:18.664786+0000","last_became_peered":"2023-03-09T14:44:18.664786+0000","last_unstale":"2023-03-09T14:45:14.853129+0000","last_undegraded":"2023-03-09T14:45:14.853129+0000","last_fullsized":"2023-03-09T14:45:14.853129+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T20:29:20.919293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":656,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"89'11","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.797201+0000","last_change":"2023-03-09T14:44:19.108693+0000","last_active":"2023-03-09T14:45:44.797201+0000","last_peered":"2023-03-09T14:45:44.797201+0000","last_clean":"2023-03-09T14:45:44.797201+0000","last_became_active":"2023-03-09T14:44:19.108370+0000","last_became_peered":"2023-03-09T14:44:19.108370+0000","last_unstale":"2023-03-09T14:45:44.797201+0000","last_undegraded":"2023-03-09T14:45:44.797201+0000","last_fullsized":"2023-03-09T14:45:44.797201+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T16:30:58.726080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:15.862206+0000","last_change":"2023-03-09T14:45:15.862206+0000","last_active":"2023-03-09T14:45:15.862206+0000","last_peered":"2023-03-09T14:45:15.862206+0000","last_clean":"2023-03-09T14:45:15.862206+0000","last_became_active":"2023-03-09T14:45:15.861775+0000","last_became_peered":"2023-03-09T14:45:15.861775+0000","last_unstale":"2023-03-09T14:45:15.862206+0000","last_undegraded":"2023-03-09T14:45:15.862206+0000","last_fullsized":"2023-03-09T14:45:15.862206+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:31:21.810745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.14","version":"89'35","reported_seq":225,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:59.931365+0000","last_change":"2023-03-09T14:44:18.661904+0000","last_active":"2023-03-09T14:45:59.931365+0000","last_peered":"2023-03-09T14:45:59.931365+0000","last_clean":"2023-03-09T14:45:59.931365+0000","last_became_active":"2023-03-09T14:44:18.661597+0000","last_became_peered":"2023-03-09T14:44:18.661597+0000","last_unstale":"2023-03-09T14:45:59.931365+0000","last_undegraded":"2023-03-09T14:45:59.931365+0000","last_fullsized":"2023-03-09T14:45:59.931365+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T14:57:04.978132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61521,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":172,"num_read_kb":147,"num_write":36,"num_write_kb":76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.786319+0000","last_change":"2023-03-09T14:44:19.111671+0000","last_active":"2023-03-09T14:45:44.786319+0000","last_peered":"2023-03-09T14:45:44.786319+0000","last_clean":"2023-03-09T14:45:44.786319+0000","last_became_active":"2023-03-09T14:44:19.111172+0000","last_became_peered":"2023-03-09T14:44:19.111172+0000","last_unstale":"2023-03-09T14:45:44.786319+0000","last_undegraded":"2023-03-09T14:45:44.786319+0000","last_fullsized":"2023-03-09T14:45:44.786319+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T16:27:23.921853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.853428+0000","last_change":"2023-03-09T14:44:23.486338+0000","last_active":"2023-03-09T14:45:14.853428+0000","last_peered":"2023-03-09T14:45:14.853428+0000","last_clean":"2023-03-09T14:45:14.853428+0000","last_became_active":"2023-03-09T14:44:23.485729+0000","last_became_peered":"2023-03-09T14:44:23.485729+0000","last_unstale":"2023-03-09T14:45:14.853428+0000","last_undegraded":"2023-03-09T14:45:14.853428+0000","last_fullsized":"2023-03-09T14:45:14.853428+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:28:23.872633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.857067+0000","last_change":"2023-03-09T14:44:18.664942+0000","last_active":"2023-03-09T14:45:14.857067+0000","last_peered":"2023-03-09T14:45:14.857067+0000","last_clean":"2023-03-09T14:45:14.857067+0000","last_became_active":"2023-03-09T14:44:18.664007+0000","last_became_peered":"2023-03-09T14:44:18.664007+0000","last_unstale":"2023-03-09T14:45:14.857067+0000","last_undegraded":"2023-03-09T14:45:14.857067+0000","last_fullsized":"2023-03-09T14:45:14.857067+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:36:21.329388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.782333+0000","last_change":"2023-03-09T14:44:19.110140+0000","last_active":"2023-03-09T14:45:44.782333+0000","last_peered":"2023-03-09T14:45:44.782333+0000","last_clean":"2023-03-09T14:45:44.782333+0000","last_became_active":"2023-03-09T14:44:19.109811+0000","last_became_peered":"2023-03-09T14:44:19.109811+0000","last_unstale":"2023-03-09T14:45:44.782333+0000","last_undegraded":"2023-03-09T14:45:44.782333+0000","last_fullsized":"2023-03-09T14:45:44.782333+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T14:56:14.638662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.858304+0000","last_change":"2023-03-09T14:44:23.491318+0000","last_active":"2023-03-09T14:45:14.858304+0000","last_peered":"2023-03-09T14:45:14.858304+0000","last_clean":"2023-03-09T14:45:14.858304+0000","last_became_active":"2023-03-09T14:44:23.490915+0000","last_became_peered":"2023-03-09T14:44:23.490915+0000","last_unstale":"2023-03-09T14:45:14.858304+0000","last_undegraded":"2023-03-09T14:45:14.858304+0000","last_fullsized":"2023-03-09T14:45:14.858304+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:32:06.877573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"87'3","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.852806+0000","last_change":"2023-03-09T14:44:18.657323+0000","last_active":"2023-03-09T14:45:14.852806+0000","last_peered":"2023-03-09T14:45:14.852806+0000","last_clean":"2023-03-09T14:45:14.852806+0000","last_became_active":"2023-03-09T14:44:18.657017+0000","last_became_peered":"2023-03-09T14:44:18.657017+0000","last_unstale":"2023-03-09T14:45:14.852806+0000","last_undegraded":"2023-03-09T14:45:14.852806+0000","last_fullsized":"2023-03-09T14:45:14.852806+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:43:11.562723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"89'15","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.791050+0000","last_change":"2023-03-09T14:44:19.112194+0000","last_active":"2023-03-09T14:45:44.791050+0000","last_peered":"2023-03-09T14:45:44.791050+0000","last_clean":"2023-03-09T14:45:44.791050+0000","last_became_active":"2023-03-09T14:44:19.111924+0000","last_became_peered":"2023-03-09T14:44:19.111924+0000","last_unstale":"2023-03-09T14:45:44.791050+0000","last_undegraded":"2023-03-09T14:45:44.791050+0000","last_fullsized":"2023-03-09T14:45:44.791050+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T19:54:22.669848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.860733+0000","last_change":"2023-03-09T14:44:23.493993+0000","last_active":"2023-03-09T14:45:14.860733+0000","last_peered":"2023-03-09T14:45:14.860733+0000","last_clean":"2023-03-09T14:45:14.860733+0000","last_became_active":"2023-03-09T14:44:23.493535+0000","last_became_peered":"2023-03-09T14:44:23.493535+0000","last_unstale":"2023-03-09T14:45:14.860733+0000","last_undegraded":"2023-03-09T14:45:14.860733+0000","last_fullsized":"2023-03-09T14:45:14.860733+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T01:18:23.669790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'5","reported_seq":104,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:54.664806+0000","last_change":"2023-03-09T14:44:18.659914+0000","last_active":"2023-03-09T14:45:54.664806+0000","last_peered":"2023-03-09T14:45:54.664806+0000","last_clean":"2023-03-09T14:45:54.664806+0000","last_became_active":"2023-03-09T14:44:18.658835+0000","last_became_peered":"2023-03-09T14:44:18.658835+0000","last_unstale":"2023-03-09T14:45:54.664806+0000","last_undegraded":"2023-03-09T14:45:54.664806+0000","last_fullsized":"2023-03-09T14:45:54.664806+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T14:55:05.413260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1119,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":82,"num_read_kb":5,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.787772+0000","last_change":"2023-03-09T14:44:19.111058+0000","last_active":"2023-03-09T14:45:44.787772+0000","last_peered":"2023-03-09T14:45:44.787772+0000","last_clean":"2023-03-09T14:45:44.787772+0000","last_became_active":"2023-03-09T14:44:19.110760+0000","last_became_peered":"2023-03-09T14:44:19.110760+0000","last_unstale":"2023-03-09T14:45:44.787772+0000","last_undegraded":"2023-03-09T14:45:44.787772+0000","last_fullsized":"2023-03-09T14:45:44.787772+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:25:13.199416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.861493+0000","last_change":"2023-03-09T14:44:23.484710+0000","last_active":"2023-03-09T14:45:14.861493+0000","last_peered":"2023-03-09T14:45:14.861493+0000","last_clean":"2023-03-09T14:45:14.861493+0000","last_became_active":"2023-03-09T14:44:23.483928+0000","last_became_peered":"2023-03-09T14:44:23.483928+0000","last_unstale":"2023-03-09T14:45:14.861493+0000","last_undegraded":"2023-03-09T14:45:14.861493+0000","last_fullsized":"2023-03-09T14:45:14.861493+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T14:51:32.124111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"87'15","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.860284+0000","last_change":"2023-03-09T14:44:18.663692+0000","last_active":"2023-03-09T14:45:14.860284+0000","last_peered":"2023-03-09T14:45:14.860284+0000","last_clean":"2023-03-09T14:45:14.860284+0000","last_became_active":"2023-03-09T14:44:18.663400+0000","last_became_peered":"2023-03-09T14:44:18.663400+0000","last_unstale":"2023-03-09T14:45:14.860284+0000","last_undegraded":"2023-03-09T14:45:14.860284+0000","last_fullsized":"2023-03-09T14:45:14.860284+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:32:31.634315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.797691+0000","last_change":"2023-03-09T14:44:19.110819+0000","last_active":"2023-03-09T14:45:44.797691+0000","last_peered":"2023-03-09T14:45:44.797691+0000","last_clean":"2023-03-09T14:45:44.797691+0000","last_became_active":"2023-03-09T14:44:19.110542+0000","last_became_peered":"2023-03-09T14:44:19.110542+0000","last_unstale":"2023-03-09T14:45:44.797691+0000","last_undegraded":"2023-03-09T14:45:44.797691+0000","last_fullsized":"2023-03-09T14:45:44.797691+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:25:38.382685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.852360+0000","last_change":"2023-03-09T14:44:23.484614+0000","last_active":"2023-03-09T14:45:14.852360+0000","last_peered":"2023-03-09T14:45:14.852360+0000","last_clean":"2023-03-09T14:45:14.852360+0000","last_became_active":"2023-03-09T14:44:23.484061+0000","last_became_peered":"2023-03-09T14:44:23.484061+0000","last_unstale":"2023-03-09T14:45:14.852360+0000","last_undegraded":"2023-03-09T14:45:14.852360+0000","last_fullsized":"2023-03-09T14:45:14.852360+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T19:31:42.889122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"87'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.861838+0000","last_change":"2023-03-09T14:44:18.656056+0000","last_active":"2023-03-09T14:45:14.861838+0000","last_peered":"2023-03-09T14:45:14.861838+0000","last_clean":"2023-03-09T14:45:14.861838+0000","last_became_active":"2023-03-09T14:44:18.655709+0000","last_became_peered":"2023-03-09T14:44:18.655709+0000","last_unstale":"2023-03-09T14:45:14.861838+0000","last_undegraded":"2023-03-09T14:45:14.861838+0000","last_fullsized":"2023-03-09T14:45:14.861838+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T19:09:11.447249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"89'5","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.790663+0000","last_change":"2023-03-09T14:44:19.110292+0000","last_active":"2023-03-09T14:45:44.790663+0000","last_peered":"2023-03-09T14:45:44.790663+0000","last_clean":"2023-03-09T14:45:44.790663+0000","last_became_active":"2023-03-09T14:44:19.110006+0000","last_became_peered":"2023-03-09T14:44:19.110006+0000","last_unstale":"2023-03-09T14:45:44.790663+0000","last_undegraded":"2023-03-09T14:45:44.790663+0000","last_fullsized":"2023-03-09T14:45:44.790663+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:36:40.401486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.861551+0000","last_change":"2023-03-09T14:44:23.484004+0000","last_active":"2023-03-09T14:45:14.861551+0000","last_peered":"2023-03-09T14:45:14.861551+0000","last_clean":"2023-03-09T14:45:14.861551+0000","last_became_active":"2023-03-09T14:44:23.483581+0000","last_became_peered":"2023-03-09T14:44:23.483581+0000","last_unstale":"2023-03-09T14:45:14.861551+0000","last_undegraded":"2023-03-09T14:45:14.861551+0000","last_fullsized":"2023-03-09T14:45:14.861551+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:24:25.939838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.860756+0000","last_change":"2023-03-09T14:44:18.661856+0000","last_active":"2023-03-09T14:45:14.860756+0000","last_peered":"2023-03-09T14:45:14.860756+0000","last_clean":"2023-03-09T14:45:14.860756+0000","last_became_active":"2023-03-09T14:44:18.661500+0000","last_became_peered":"2023-03-09T14:44:18.661500+0000","last_unstale":"2023-03-09T14:45:14.860756+0000","last_undegraded":"2023-03-09T14:45:14.860756+0000","last_fullsized":"2023-03-09T14:45:14.860756+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:27:12.181159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"89'4","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.783682+0000","last_change":"2023-03-09T14:44:19.102058+0000","last_active":"2023-03-09T14:45:44.783682+0000","last_peered":"2023-03-09T14:45:44.783682+0000","last_clean":"2023-03-09T14:45:44.783682+0000","last_became_active":"2023-03-09T14:44:19.101762+0000","last_became_peered":"2023-03-09T14:44:19.101762+0000","last_unstale":"2023-03-09T14:45:44.783682+0000","last_undegraded":"2023-03-09T14:45:44.783682+0000","last_fullsized":"2023-03-09T14:45:44.783682+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:39:42.544086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.862303+0000","last_change":"2023-03-09T14:44:23.490237+0000","last_active":"2023-03-09T14:45:14.862303+0000","last_peered":"2023-03-09T14:45:14.862303+0000","last_clean":"2023-03-09T14:45:14.862303+0000","last_became_active":"2023-03-09T14:44:23.489844+0000","last_became_peered":"2023-03-09T14:44:23.489844+0000","last_unstale":"2023-03-09T14:45:14.862303+0000","last_undegraded":"2023-03-09T14:45:14.862303+0000","last_fullsized":"2023-03-09T14:45:14.862303+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T14:50:09.358698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"87'1","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:15.865433+0000","last_change":"2023-03-09T14:45:15.864913+0000","last_active":"2023-03-09T14:45:15.865433+0000","last_peered":"2023-03-09T14:45:15.865433+0000","last_clean":"2023-03-09T14:45:15.865433+0000","last_became_active":"2023-03-09T14:45:15.864527+0000","last_became_peered":"2023-03-09T14:45:15.864527+0000","last_unstale":"2023-03-09T14:45:15.865433+0000","last_undegraded":"2023-03-09T14:45:15.865433+0000","last_fullsized":"2023-03-09T14:45:15.865433+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:02:19.489659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6,1],"acting":[11,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.790944+0000","last_change":"2023-03-09T14:44:19.117584+0000","last_active":"2023-03-09T14:45:44.790944+0000","last_peered":"2023-03-09T14:45:44.790944+0000","last_clean":"2023-03-09T14:45:44.790944+0000","last_became_active":"2023-03-09T14:44:19.117262+0000","last_became_peered":"2023-03-09T14:44:19.117262+0000","last_unstale":"2023-03-09T14:45:44.790944+0000","last_undegraded":"2023-03-09T14:45:44.790944+0000","last_fullsized":"2023-03-09T14:45:44.790944+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T19:19:38.080048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.861827+0000","last_change":"2023-03-09T14:44:23.485769+0000","last_active":"2023-03-09T14:45:14.861827+0000","last_peered":"2023-03-09T14:45:14.861827+0000","last_clean":"2023-03-09T14:45:14.861827+0000","last_became_active":"2023-03-09T14:44:23.485317+0000","last_became_peered":"2023-03-09T14:44:23.485317+0000","last_unstale":"2023-03-09T14:45:14.861827+0000","last_undegraded":"2023-03-09T14:45:14.861827+0000","last_fullsized":"2023-03-09T14:45:14.861827+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:58:35.140512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.786487+0000","last_change":"2023-03-09T14:44:19.112255+0000","last_active":"2023-03-09T14:45:44.786487+0000","last_peered":"2023-03-09T14:45:44.786487+0000","last_clean":"2023-03-09T14:45:44.786487+0000","last_became_active":"2023-03-09T14:44:19.111988+0000","last_became_peered":"2023-03-09T14:44:19.111988+0000","last_unstale":"2023-03-09T14:45:44.786487+0000","last_undegraded":"2023-03-09T14:45:44.786487+0000","last_fullsized":"2023-03-09T14:45:44.786487+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:55:50.423701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'5","reported_seq":103,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:57.669850+0000","last_change":"2023-03-09T14:44:18.665291+0000","last_active":"2023-03-09T14:45:57.669850+0000","last_peered":"2023-03-09T14:45:57.669850+0000","last_clean":"2023-03-09T14:45:57.669850+0000","last_became_active":"2023-03-09T14:44:18.664950+0000","last_became_peered":"2023-03-09T14:44:18.664950+0000","last_unstale":"2023-03-09T14:45:57.669850+0000","last_undegraded":"2023-03-09T14:45:57.669850+0000","last_fullsized":"2023-03-09T14:45:57.669850+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:40:58.619717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":80,"num_read_kb":80,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"87'1","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.853567+0000","last_change":"2023-03-09T14:44:23.480872+0000","last_active":"2023-03-09T14:45:14.853567+0000","last_peered":"2023-03-09T14:45:14.853567+0000","last_clean":"2023-03-09T14:45:14.853567+0000","last_became_active":"2023-03-09T14:44:23.480495+0000","last_became_peered":"2023-03-09T14:44:23.480495+0000","last_unstale":"2023-03-09T14:45:14.853567+0000","last_undegraded":"2023-03-09T14:45:14.853567+0000","last_fullsized":"2023-03-09T14:45:14.853567+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:19:01.393034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.797742+0000","last_change":"2023-03-09T14:44:19.102833+0000","last_active":"2023-03-09T14:45:44.797742+0000","last_peered":"2023-03-09T14:45:44.797742+0000","last_clean":"2023-03-09T14:45:44.797742+0000","last_became_active":"2023-03-09T14:44:19.102484+0000","last_became_peered":"2023-03-09T14:44:19.102484+0000","last_unstale":"2023-03-09T14:45:44.797742+0000","last_undegraded":"2023-03-09T14:45:44.797742+0000","last_fullsized":"2023-03-09T14:45:44.797742+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:14:50.002907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"87'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.859537+0000","last_change":"2023-03-09T14:44:18.661603+0000","last_active":"2023-03-09T14:45:14.859537+0000","last_peered":"2023-03-09T14:45:14.859537+0000","last_clean":"2023-03-09T14:45:14.859537+0000","last_became_active":"2023-03-09T14:44:18.661275+0000","last_became_peered":"2023-03-09T14:44:18.661275+0000","last_unstale":"2023-03-09T14:45:14.859537+0000","last_undegraded":"2023-03-09T14:45:14.859537+0000","last_fullsized":"2023-03-09T14:45:14.859537+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:20:31.873778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":378,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.852517+0000","last_change":"2023-03-09T14:43:19.317554+0000","last_active":"2023-03-09T14:45:14.852517+0000","last_peered":"2023-03-09T14:45:14.852517+0000","last_clean":"2023-03-09T14:45:14.852517+0000","last_became_active":"2023-03-09T14:43:19.316912+0000","last_became_peered":"2023-03-09T14:43:19.316912+0000","last_unstale":"2023-03-09T14:45:14.852517+0000","last_undegraded":"2023-03-09T14:45:14.852517+0000","last_fullsized":"2023-03-09T14:45:14.852517+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:40:23.197173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:40:23.197173+0000","last_clean_scrub_stamp":"2023-03-09T14:40:23.197173+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T20:21:17.143331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.852927+0000","last_change":"2023-03-09T14:44:23.487738+0000","last_active":"2023-03-09T14:45:14.852927+0000","last_peered":"2023-03-09T14:45:14.852927+0000","last_clean":"2023-03-09T14:45:14.852927+0000","last_became_active":"2023-03-09T14:44:23.487193+0000","last_became_peered":"2023-03-09T14:44:23.487193+0000","last_unstale":"2023-03-09T14:45:14.852927+0000","last_undegraded":"2023-03-09T14:45:14.852927+0000","last_fullsized":"2023-03-09T14:45:14.852927+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:48:08.789071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"89'5","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.784152+0000","last_change":"2023-03-09T14:44:19.112576+0000","last_active":"2023-03-09T14:45:44.784152+0000","last_peered":"2023-03-09T14:45:44.784152+0000","last_clean":"2023-03-09T14:45:44.784152+0000","last_became_active":"2023-03-09T14:44:19.112231+0000","last_became_peered":"2023-03-09T14:44:19.112231+0000","last_unstale":"2023-03-09T14:45:44.784152+0000","last_undegraded":"2023-03-09T14:45:44.784152+0000","last_fullsized":"2023-03-09T14:45:44.784152+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:25:21.918088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"87'3","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.852608+0000","last_change":"2023-03-09T14:44:18.655478+0000","last_active":"2023-03-09T14:45:14.852608+0000","last_peered":"2023-03-09T14:45:14.852608+0000","last_clean":"2023-03-09T14:45:14.852608+0000","last_became_active":"2023-03-09T14:44:18.655180+0000","last_became_peered":"2023-03-09T14:44:18.655180+0000","last_unstale":"2023-03-09T14:45:14.852608+0000","last_undegraded":"2023-03-09T14:45:14.852608+0000","last_fullsized":"2023-03-09T14:45:14.852608+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:42:47.926376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.856627+0000","last_change":"2023-03-09T14:44:23.487052+0000","last_active":"2023-03-09T14:45:14.856627+0000","last_peered":"2023-03-09T14:45:14.856627+0000","last_clean":"2023-03-09T14:45:14.856627+0000","last_became_active":"2023-03-09T14:44:23.486603+0000","last_became_peered":"2023-03-09T14:44:23.486603+0000","last_unstale":"2023-03-09T14:45:14.856627+0000","last_undegraded":"2023-03-09T14:45:14.856627+0000","last_fullsized":"2023-03-09T14:45:14.856627+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:28:25.375052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"87'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.861722+0000","last_change":"2023-03-09T14:44:18.666400+0000","last_active":"2023-03-09T14:45:14.861722+0000","last_peered":"2023-03-09T14:45:14.861722+0000","last_clean":"2023-03-09T14:45:14.861722+0000","last_became_active":"2023-03-09T14:44:18.666045+0000","last_became_peered":"2023-03-09T14:44:18.666045+0000","last_unstale":"2023-03-09T14:45:14.861722+0000","last_undegraded":"2023-03-09T14:45:14.861722+0000","last_fullsized":"2023-03-09T14:45:14.861722+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:30:48.640124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.784135+0000","last_change":"2023-03-09T14:44:19.103657+0000","last_active":"2023-03-09T14:45:44.784135+0000","last_peered":"2023-03-09T14:45:44.784135+0000","last_clean":"2023-03-09T14:45:44.784135+0000","last_became_active":"2023-03-09T14:44:19.103364+0000","last_became_peered":"2023-03-09T14:44:19.103364+0000","last_unstale":"2023-03-09T14:45:44.784135+0000","last_undegraded":"2023-03-09T14:45:44.784135+0000","last_fullsized":"2023-03-09T14:45:44.784135+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:23:23.738554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.864043+0000","last_change":"2023-03-09T14:44:23.483487+0000","last_active":"2023-03-09T14:45:14.864043+0000","last_peered":"2023-03-09T14:45:14.864043+0000","last_clean":"2023-03-09T14:45:14.864043+0000","last_became_active":"2023-03-09T14:44:23.483050+0000","last_became_peered":"2023-03-09T14:44:23.483050+0000","last_unstale":"2023-03-09T14:45:14.864043+0000","last_undegraded":"2023-03-09T14:45:14.864043+0000","last_fullsized":"2023-03-09T14:45:14.864043+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:38:34.129355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'7","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:48.585960+0000","last_change":"2023-03-09T14:44:18.661944+0000","last_active":"2023-03-09T14:45:48.585960+0000","last_peered":"2023-03-09T14:45:48.585960+0000","last_clean":"2023-03-09T14:45:48.585960+0000","last_became_active":"2023-03-09T14:44:18.661575+0000","last_became_peered":"2023-03-09T14:44:18.661575+0000","last_unstale":"2023-03-09T14:45:48.585960+0000","last_undegraded":"2023-03-09T14:45:48.585960+0000","last_fullsized":"2023-03-09T14:45:48.585960+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T01:12:53.637866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"89'5","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.787402+0000","last_change":"2023-03-09T14:44:19.103807+0000","last_active":"2023-03-09T14:45:44.787402+0000","last_peered":"2023-03-09T14:45:44.787402+0000","last_clean":"2023-03-09T14:45:44.787402+0000","last_became_active":"2023-03-09T14:44:19.103480+0000","last_became_peered":"2023-03-09T14:44:19.103480+0000","last_unstale":"2023-03-09T14:45:44.787402+0000","last_undegraded":"2023-03-09T14:45:44.787402+0000","last_fullsized":"2023-03-09T14:45:44.787402+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:29:54.629265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.856235+0000","last_change":"2023-03-09T14:44:23.486789+0000","last_active":"2023-03-09T14:45:14.856235+0000","last_peered":"2023-03-09T14:45:14.856235+0000","last_clean":"2023-03-09T14:45:14.856235+0000","last_became_active":"2023-03-09T14:44:23.486367+0000","last_became_peered":"2023-03-09T14:44:23.486367+0000","last_unstale":"2023-03-09T14:45:14.856235+0000","last_undegraded":"2023-03-09T14:45:14.856235+0000","last_fullsized":"2023-03-09T14:45:14.856235+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:13:13.081388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"87'4","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.860080+0000","last_change":"2023-03-09T14:44:18.659492+0000","last_active":"2023-03-09T14:45:14.860080+0000","last_peered":"2023-03-09T14:45:14.860080+0000","last_clean":"2023-03-09T14:45:14.860080+0000","last_became_active":"2023-03-09T14:44:18.659187+0000","last_became_peered":"2023-03-09T14:44:18.659187+0000","last_unstale":"2023-03-09T14:45:14.860080+0000","last_undegraded":"2023-03-09T14:45:14.860080+0000","last_fullsized":"2023-03-09T14:45:14.860080+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:44:31.342785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.789313+0000","last_change":"2023-03-09T14:44:19.112841+0000","last_active":"2023-03-09T14:45:44.789313+0000","last_peered":"2023-03-09T14:45:44.789313+0000","last_clean":"2023-03-09T14:45:44.789313+0000","last_became_active":"2023-03-09T14:44:19.112515+0000","last_became_peered":"2023-03-09T14:44:19.112515+0000","last_unstale":"2023-03-09T14:45:44.789313+0000","last_undegraded":"2023-03-09T14:45:44.789313+0000","last_fullsized":"2023-03-09T14:45:44.789313+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T19:41:04.065529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.859850+0000","last_change":"2023-03-09T14:44:23.481992+0000","last_active":"2023-03-09T14:45:14.859850+0000","last_peered":"2023-03-09T14:45:14.859850+0000","last_clean":"2023-03-09T14:45:14.859850+0000","last_became_active":"2023-03-09T14:44:23.481508+0000","last_became_peered":"2023-03-09T14:44:23.481508+0000","last_unstale":"2023-03-09T14:45:14.859850+0000","last_undegraded":"2023-03-09T14:45:14.859850+0000","last_fullsized":"2023-03-09T14:45:14.859850+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:46:48.683402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"87'2","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.856199+0000","last_change":"2023-03-09T14:44:18.663133+0000","last_active":"2023-03-09T14:45:14.856199+0000","last_peered":"2023-03-09T14:45:14.856199+0000","last_clean":"2023-03-09T14:45:14.856199+0000","last_became_active":"2023-03-09T14:44:18.662809+0000","last_became_peered":"2023-03-09T14:44:18.662809+0000","last_unstale":"2023-03-09T14:45:14.856199+0000","last_undegraded":"2023-03-09T14:45:14.856199+0000","last_fullsized":"2023-03-09T14:45:14.856199+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:32:16.180642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"89'2","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.784561+0000","last_change":"2023-03-09T14:44:19.108566+0000","last_active":"2023-03-09T14:45:44.784561+0000","last_peered":"2023-03-09T14:45:44.784561+0000","last_clean":"2023-03-09T14:45:44.784561+0000","last_became_active":"2023-03-09T14:44:19.108217+0000","last_became_peered":"2023-03-09T14:44:19.108217+0000","last_unstale":"2023-03-09T14:45:44.784561+0000","last_undegraded":"2023-03-09T14:45:44.784561+0000","last_fullsized":"2023-03-09T14:45:44.784561+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:03:28.434395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.860862+0000","last_change":"2023-03-09T14:44:23.488640+0000","last_active":"2023-03-09T14:45:14.860862+0000","last_peered":"2023-03-09T14:45:14.860862+0000","last_clean":"2023-03-09T14:45:14.860862+0000","last_became_active":"2023-03-09T14:44:23.488178+0000","last_became_peered":"2023-03-09T14:44:23.488178+0000","last_unstale":"2023-03-09T14:45:14.860862+0000","last_undegraded":"2023-03-09T14:45:14.860862+0000","last_fullsized":"2023-03-09T14:45:14.860862+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:33:49.908974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"87'3","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.864673+0000","last_change":"2023-03-09T14:44:18.662925+0000","last_active":"2023-03-09T14:45:14.864673+0000","last_peered":"2023-03-09T14:45:14.864673+0000","last_clean":"2023-03-09T14:45:14.864673+0000","last_became_active":"2023-03-09T14:44:18.662354+0000","last_became_peered":"2023-03-09T14:44:18.662354+0000","last_unstale":"2023-03-09T14:45:14.864673+0000","last_undegraded":"2023-03-09T14:45:14.864673+0000","last_fullsized":"2023-03-09T14:45:14.864673+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T01:00:36.602938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.790399+0000","last_change":"2023-03-09T14:44:19.106465+0000","last_active":"2023-03-09T14:45:44.790399+0000","last_peered":"2023-03-09T14:45:44.790399+0000","last_clean":"2023-03-09T14:45:44.790399+0000","last_became_active":"2023-03-09T14:44:19.106001+0000","last_became_peered":"2023-03-09T14:44:19.106001+0000","last_unstale":"2023-03-09T14:45:44.790399+0000","last_undegraded":"2023-03-09T14:45:44.790399+0000","last_fullsized":"2023-03-09T14:45:44.790399+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:22:09.149291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"87'1","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:15.873389+0000","last_change":"2023-03-09T14:45:15.873389+0000","last_active":"2023-03-09T14:45:15.873389+0000","last_peered":"2023-03-09T14:45:15.873389+0000","last_clean":"2023-03-09T14:45:15.873389+0000","last_became_active":"2023-03-09T14:45:15.863408+0000","last_became_peered":"2023-03-09T14:45:15.863408+0000","last_unstale":"2023-03-09T14:45:15.873389+0000","last_undegraded":"2023-03-09T14:45:15.873389+0000","last_fullsized":"2023-03-09T14:45:15.873389+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T16:33:12.565135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":123,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,7,8],"acting":[4,10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"87'1","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.856858+0000","last_change":"2023-03-09T14:44:18.664744+0000","last_active":"2023-03-09T14:45:14.856858+0000","last_peered":"2023-03-09T14:45:14.856858+0000","last_clean":"2023-03-09T14:45:14.856858+0000","last_became_active":"2023-03-09T14:44:18.664443+0000","last_became_peered":"2023-03-09T14:44:18.664443+0000","last_unstale":"2023-03-09T14:45:14.856858+0000","last_undegraded":"2023-03-09T14:45:14.856858+0000","last_fullsized":"2023-03-09T14:45:14.856858+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:11:13.836270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.790126+0000","last_change":"2023-03-09T14:44:19.115102+0000","last_active":"2023-03-09T14:45:44.790126+0000","last_peered":"2023-03-09T14:45:44.790126+0000","last_clean":"2023-03-09T14:45:44.790126+0000","last_became_active":"2023-03-09T14:44:19.114565+0000","last_became_peered":"2023-03-09T14:44:19.114565+0000","last_unstale":"2023-03-09T14:45:44.790126+0000","last_undegraded":"2023-03-09T14:45:44.790126+0000","last_fullsized":"2023-03-09T14:45:44.790126+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:39:11.930231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.859836+0000","last_change":"2023-03-09T14:44:23.482500+0000","last_active":"2023-03-09T14:45:14.859836+0000","last_peered":"2023-03-09T14:45:14.859836+0000","last_clean":"2023-03-09T14:45:14.859836+0000","last_became_active":"2023-03-09T14:44:23.482217+0000","last_became_peered":"2023-03-09T14:44:23.482217+0000","last_unstale":"2023-03-09T14:45:14.859836+0000","last_undegraded":"2023-03-09T14:45:14.859836+0000","last_fullsized":"2023-03-09T14:45:14.859836+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T14:52:44.275037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"87'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.860725+0000","last_change":"2023-03-09T14:44:18.663824+0000","last_active":"2023-03-09T14:45:14.860725+0000","last_peered":"2023-03-09T14:45:14.860725+0000","last_clean":"2023-03-09T14:45:14.860725+0000","last_became_active":"2023-03-09T14:44:18.663379+0000","last_became_peered":"2023-03-09T14:44:18.663379+0000","last_unstale":"2023-03-09T14:45:14.860725+0000","last_undegraded":"2023-03-09T14:45:14.860725+0000","last_fullsized":"2023-03-09T14:45:14.860725+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:03:04.031061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"89'5","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.798601+0000","last_change":"2023-03-09T14:44:19.105765+0000","last_active":"2023-03-09T14:45:44.798601+0000","last_peered":"2023-03-09T14:45:44.798601+0000","last_clean":"2023-03-09T14:45:44.798601+0000","last_became_active":"2023-03-09T14:44:19.105430+0000","last_became_peered":"2023-03-09T14:44:19.105430+0000","last_unstale":"2023-03-09T14:45:44.798601+0000","last_undegraded":"2023-03-09T14:45:44.798601+0000","last_fullsized":"2023-03-09T14:45:44.798601+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T20:31:40.106798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.856901+0000","last_change":"2023-03-09T14:44:23.482079+0000","last_active":"2023-03-09T14:45:14.856901+0000","last_peered":"2023-03-09T14:45:14.856901+0000","last_clean":"2023-03-09T14:45:14.856901+0000","last_became_active":"2023-03-09T14:44:23.481620+0000","last_became_peered":"2023-03-09T14:44:23.481620+0000","last_unstale":"2023-03-09T14:45:14.856901+0000","last_undegraded":"2023-03-09T14:45:14.856901+0000","last_fullsized":"2023-03-09T14:45:14.856901+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:47:53.589111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"87'3","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:15.869980+0000","last_change":"2023-03-09T14:45:15.869485+0000","last_active":"2023-03-09T14:45:15.869980+0000","last_peered":"2023-03-09T14:45:15.869980+0000","last_clean":"2023-03-09T14:45:15.869980+0000","last_became_active":"2023-03-09T14:45:15.861428+0000","last_became_peered":"2023-03-09T14:45:15.861428+0000","last_unstale":"2023-03-09T14:45:15.869980+0000","last_undegraded":"2023-03-09T14:45:15.869980+0000","last_fullsized":"2023-03-09T14:45:15.869980+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:43:30.309431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":566,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.799201+0000","last_change":"2023-03-09T14:44:19.105947+0000","last_active":"2023-03-09T14:45:44.799201+0000","last_peered":"2023-03-09T14:45:44.799201+0000","last_clean":"2023-03-09T14:45:44.799201+0000","last_became_active":"2023-03-09T14:44:19.105669+0000","last_became_peered":"2023-03-09T14:44:19.105669+0000","last_unstale":"2023-03-09T14:45:44.799201+0000","last_undegraded":"2023-03-09T14:45:44.799201+0000","last_fullsized":"2023-03-09T14:45:44.799201+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:29:58.615481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.863833+0000","last_change":"2023-03-09T14:44:23.483237+0000","last_active":"2023-03-09T14:45:14.863833+0000","last_peered":"2023-03-09T14:45:14.863833+0000","last_clean":"2023-03-09T14:45:14.863833+0000","last_became_active":"2023-03-09T14:44:23.482831+0000","last_became_peered":"2023-03-09T14:44:23.482831+0000","last_unstale":"2023-03-09T14:45:14.863833+0000","last_undegraded":"2023-03-09T14:45:14.863833+0000","last_fullsized":"2023-03-09T14:45:14.863833+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:36:40.775581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"82'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:15.867430+0000","last_change":"2023-03-09T14:45:15.867049+0000","last_active":"2023-03-09T14:45:15.867430+0000","last_peered":"2023-03-09T14:45:15.867430+0000","last_clean":"2023-03-09T14:45:15.867430+0000","last_became_active":"2023-03-09T14:45:15.863218+0000","last_became_peered":"2023-03-09T14:45:15.863218+0000","last_unstale":"2023-03-09T14:45:15.867430+0000","last_undegraded":"2023-03-09T14:45:15.867430+0000","last_fullsized":"2023-03-09T14:45:15.867430+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:38:21.800533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.c","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.787235+0000","last_change":"2023-03-09T14:44:19.107146+0000","last_active":"2023-03-09T14:45:44.787235+0000","last_peered":"2023-03-09T14:45:44.787235+0000","last_clean":"2023-03-09T14:45:44.787235+0000","last_became_active":"2023-03-09T14:44:19.106834+0000","last_became_peered":"2023-03-09T14:44:19.106834+0000","last_unstale":"2023-03-09T14:45:44.787235+0000","last_undegraded":"2023-03-09T14:45:44.787235+0000","last_fullsized":"2023-03-09T14:45:44.787235+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T20:15:20.142449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.852185+0000","last_change":"2023-03-09T14:44:23.482811+0000","last_active":"2023-03-09T14:45:14.852185+0000","last_peered":"2023-03-09T14:45:14.852185+0000","last_clean":"2023-03-09T14:45:14.852185+0000","last_became_active":"2023-03-09T14:44:23.482371+0000","last_became_peered":"2023-03-09T14:44:23.482371+0000","last_unstale":"2023-03-09T14:45:14.852185+0000","last_undegraded":"2023-03-09T14:45:14.852185+0000","last_fullsized":"2023-03-09T14:45:14.852185+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T01:33:14.442810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"82'2","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.861583+0000","last_change":"2023-03-09T14:44:18.661569+0000","last_active":"2023-03-09T14:45:14.861583+0000","last_peered":"2023-03-09T14:45:14.861583+0000","last_clean":"2023-03-09T14:45:14.861583+0000","last_became_active":"2023-03-09T14:44:18.661266+0000","last_became_peered":"2023-03-09T14:44:18.661266+0000","last_unstale":"2023-03-09T14:45:14.861583+0000","last_undegraded":"2023-03-09T14:45:14.861583+0000","last_fullsized":"2023-03-09T14:45:14.861583+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:47:43.066180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.786924+0000","last_change":"2023-03-09T14:44:19.106611+0000","last_active":"2023-03-09T14:45:44.786924+0000","last_peered":"2023-03-09T14:45:44.786924+0000","last_clean":"2023-03-09T14:45:44.786924+0000","last_became_active":"2023-03-09T14:44:19.106324+0000","last_became_peered":"2023-03-09T14:44:19.106324+0000","last_unstale":"2023-03-09T14:45:44.786924+0000","last_undegraded":"2023-03-09T14:45:44.786924+0000","last_fullsized":"2023-03-09T14:45:44.786924+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:50:46.561502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.787174+0000","last_change":"2023-03-09T14:44:19.106292+0000","last_active":"2023-03-09T14:45:44.787174+0000","last_peered":"2023-03-09T14:45:44.787174+0000","last_clean":"2023-03-09T14:45:44.787174+0000","last_became_active":"2023-03-09T14:44:19.105607+0000","last_became_peered":"2023-03-09T14:44:19.105607+0000","last_unstale":"2023-03-09T14:45:44.787174+0000","last_undegraded":"2023-03-09T14:45:44.787174+0000","last_fullsized":"2023-03-09T14:45:44.787174+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T19:11:28.125433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"89'12","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.786929+0000","last_change":"2023-03-09T14:44:19.103423+0000","last_active":"2023-03-09T14:45:44.786929+0000","last_peered":"2023-03-09T14:45:44.786929+0000","last_clean":"2023-03-09T14:45:44.786929+0000","last_became_active":"2023-03-09T14:44:19.102918+0000","last_became_peered":"2023-03-09T14:44:19.102918+0000","last_unstale":"2023-03-09T14:45:44.786929+0000","last_undegraded":"2023-03-09T14:45:44.786929+0000","last_fullsized":"2023-03-09T14:45:44.786929+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:47:52.921642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.790566+0000","last_change":"2023-03-09T14:44:19.113511+0000","last_active":"2023-03-09T14:45:44.790566+0000","last_peered":"2023-03-09T14:45:44.790566+0000","last_clean":"2023-03-09T14:45:44.790566+0000","last_became_active":"2023-03-09T14:44:19.113224+0000","last_became_peered":"2023-03-09T14:44:19.113224+0000","last_unstale":"2023-03-09T14:45:44.790566+0000","last_undegraded":"2023-03-09T14:45:44.790566+0000","last_fullsized":"2023-03-09T14:45:44.790566+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T02:03:28.069616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.787373+0000","last_change":"2023-03-09T14:44:19.116174+0000","last_active":"2023-03-09T14:45:44.787373+0000","last_peered":"2023-03-09T14:45:44.787373+0000","last_clean":"2023-03-09T14:45:44.787373+0000","last_became_active":"2023-03-09T14:44:19.115813+0000","last_became_peered":"2023-03-09T14:44:19.115813+0000","last_unstale":"2023-03-09T14:45:44.787373+0000","last_undegraded":"2023-03-09T14:45:44.787373+0000","last_fullsized":"2023-03-09T14:45:44.787373+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:28:49.573771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"89'4","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.790565+0000","last_change":"2023-03-09T14:44:19.115287+0000","last_active":"2023-03-09T14:45:44.790565+0000","last_peered":"2023-03-09T14:45:44.790565+0000","last_clean":"2023-03-09T14:45:44.790565+0000","last_became_active":"2023-03-09T14:44:19.113458+0000","last_became_peered":"2023-03-09T14:44:19.113458+0000","last_unstale":"2023-03-09T14:45:44.790565+0000","last_undegraded":"2023-03-09T14:45:44.790565+0000","last_fullsized":"2023-03-09T14:45:44.790565+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:43:26.583237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.787054+0000","last_change":"2023-03-09T14:44:19.107308+0000","last_active":"2023-03-09T14:45:44.787054+0000","last_peered":"2023-03-09T14:45:44.787054+0000","last_clean":"2023-03-09T14:45:44.787054+0000","last_became_active":"2023-03-09T14:44:19.107005+0000","last_became_peered":"2023-03-09T14:44:19.107005+0000","last_unstale":"2023-03-09T14:45:44.787054+0000","last_undegraded":"2023-03-09T14:45:44.787054+0000","last_fullsized":"2023-03-09T14:45:44.787054+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:48:34.886664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"89'15","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.800402+0000","last_change":"2023-03-09T14:44:19.098454+0000","last_active":"2023-03-09T14:45:44.800402+0000","last_peered":"2023-03-09T14:45:44.800402+0000","last_clean":"2023-03-09T14:45:44.800402+0000","last_became_active":"2023-03-09T14:44:19.098146+0000","last_became_peered":"2023-03-09T14:44:19.098146+0000","last_unstale":"2023-03-09T14:45:44.800402+0000","last_undegraded":"2023-03-09T14:45:44.800402+0000","last_fullsized":"2023-03-09T14:45:44.800402+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:11:40.475674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"89'2","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.790577+0000","last_change":"2023-03-09T14:44:19.095947+0000","last_active":"2023-03-09T14:45:44.790577+0000","last_peered":"2023-03-09T14:45:44.790577+0000","last_clean":"2023-03-09T14:45:44.790577+0000","last_became_active":"2023-03-09T14:44:19.095640+0000","last_became_peered":"2023-03-09T14:44:19.095640+0000","last_unstale":"2023-03-09T14:45:44.790577+0000","last_undegraded":"2023-03-09T14:45:44.790577+0000","last_fullsized":"2023-03-09T14:45:44.790577+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T02:04:35.384016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.781900+0000","last_change":"2023-03-09T14:44:19.113927+0000","last_active":"2023-03-09T14:45:44.781900+0000","last_peered":"2023-03-09T14:45:44.781900+0000","last_clean":"2023-03-09T14:45:44.781900+0000","last_became_active":"2023-03-09T14:44:19.113163+0000","last_became_peered":"2023-03-09T14:44:19.113163+0000","last_unstale":"2023-03-09T14:45:44.781900+0000","last_undegraded":"2023-03-09T14:45:44.781900+0000","last_fullsized":"2023-03-09T14:45:44.781900+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T01:35:35.801317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.787768+0000","last_change":"2023-03-09T14:44:19.098415+0000","last_active":"2023-03-09T14:45:44.787768+0000","last_peered":"2023-03-09T14:45:44.787768+0000","last_clean":"2023-03-09T14:45:44.787768+0000","last_became_active":"2023-03-09T14:44:19.097600+0000","last_became_peered":"2023-03-09T14:44:19.097600+0000","last_unstale":"2023-03-09T14:45:44.787768+0000","last_undegraded":"2023-03-09T14:45:44.787768+0000","last_fullsized":"2023-03-09T14:45:44.787768+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:15:08.552402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"89'10","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.795312+0000","last_change":"2023-03-09T14:44:19.106386+0000","last_active":"2023-03-09T14:45:44.795312+0000","last_peered":"2023-03-09T14:45:44.795312+0000","last_clean":"2023-03-09T14:45:44.795312+0000","last_became_active":"2023-03-09T14:44:19.106112+0000","last_became_peered":"2023-03-09T14:44:19.106112+0000","last_unstale":"2023-03-09T14:45:44.795312+0000","last_undegraded":"2023-03-09T14:45:44.795312+0000","last_fullsized":"2023-03-09T14:45:44.795312+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T20:27:50.992168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"89'11","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.788971+0000","last_change":"2023-03-09T14:44:19.114534+0000","last_active":"2023-03-09T14:45:44.788971+0000","last_peered":"2023-03-09T14:45:44.788971+0000","last_clean":"2023-03-09T14:45:44.788971+0000","last_became_active":"2023-03-09T14:44:19.114226+0000","last_became_peered":"2023-03-09T14:44:19.114226+0000","last_unstale":"2023-03-09T14:45:44.788971+0000","last_undegraded":"2023-03-09T14:45:44.788971+0000","last_fullsized":"2023-03-09T14:45:44.788971+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:22:38.453116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.790943+0000","last_change":"2023-03-09T14:44:19.099195+0000","last_active":"2023-03-09T14:45:44.790943+0000","last_peered":"2023-03-09T14:45:44.790943+0000","last_clean":"2023-03-09T14:45:44.790943+0000","last_became_active":"2023-03-09T14:44:19.098892+0000","last_became_peered":"2023-03-09T14:44:19.098892+0000","last_unstale":"2023-03-09T14:45:44.790943+0000","last_undegraded":"2023-03-09T14:45:44.790943+0000","last_fullsized":"2023-03-09T14:45:44.790943+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:29:24.858200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":246,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":123,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":65614,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":506,"num_read_kb":326,"num_write":163,"num_write_kb":116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":1914,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":192872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143,"ondisk_log_size":143,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":78,"seq":335007449120,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6236,"kb_used_data":980,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":93738916,"statfs":{"total":95995035648,"available":95988649984,"internally_reserved":0,"allocated":1003520,"data_stored":490897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:45:43 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.58299999999999996,"15min":0.58499999999999996},"min":{"1min":0.433,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.57899999999999996,"15min":0.58399999999999996},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.76000000000000001,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.70499999999999996}]},{"osd":1,"last update":"Thu Mar 9 14:45:43 2023","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.59999999999999998,"15min":0.58799999999999997},"min":{"1min":0.47399999999999998,"5min":0.378,"15min":0.378},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.40200000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.79600000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.76800000000000002}]},{"osd":2,"last update":"Thu Mar 9 14:45:43 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.504,"15min":0.505},"min":{"1min":0.38,"5min":0.313,"15min":0.313},"max":{"1min":0.76000000000000001,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.51300000000000001,"15min":0.503},"min":{"1min":0.33500000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.79300000000000004,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.45300000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78200000000000003}]},{"osd":4,"last update":"Thu Mar 9 14:45:43 2023","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.66900000000000004,"15min":0.65700000000000003},"min":{"1min":0.50700000000000001,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.91200000000000003,"5min":1.113,"15min":1.113},"last":0.89400000000000002},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.64800000000000002,"15min":0.64400000000000002},"min":{"1min":0.56299999999999994,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.85099999999999998,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.79400000000000004}]},{"osd":5,"last update":"Thu Mar 9 14:45:43 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.55700000000000005,"15min":0.54700000000000004},"min":{"1min":0.42399999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.63,"15min":0.61399999999999999},"min":{"1min":0.41599999999999998,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.94099999999999995,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.60499999999999998}]},{"osd":6,"last update":"Thu Mar 9 14:45:43 2023","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.67700000000000005,"15min":0.66900000000000004},"min":{"1min":0.45500000000000002,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.7829999999999999,"5min":1.7829999999999999,"15min":1.7829999999999999},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.752,"5min":0.70299999999999996,"15min":0.69499999999999995},"min":{"1min":0.53200000000000003,"5min":0.497,"15min":0.497},"max":{"1min":1.423,"5min":1.423,"15min":1.423},"last":0.63200000000000001}]},{"osd":7,"last update":"Thu Mar 9 14:45:43 2023","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.73499999999999999,"15min":0.71999999999999997},"min":{"1min":0.56299999999999994,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.0189999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.73599999999999999,"15min":0.73099999999999998},"min":{"1min":0.61299999999999999,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":0.93999999999999995,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.81599999999999995}]},{"osd":8,"last update":"Thu Mar 9 14:45:21 2023","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.91800000000000004}]},{"osd":9,"last update":"Thu Mar 9 14:45:43 2023","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.79400000000000004,"15min":0.78300000000000003},"min":{"1min":0.58699999999999997,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.548,"5min":1.548,"15min":1.548},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.872,"5min":0.78100000000000003,"15min":0.76600000000000001},"min":{"1min":0.54600000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.996,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.93999999999999995}]},{"osd":10,"last update":"Thu Mar 9 14:45:43 2023","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.755,"15min":0.749},"min":{"1min":0.64300000000000002,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":0.97499999999999998,"5min":1.01,"15min":1.01},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.79800000000000004,"15min":0.79400000000000004},"min":{"1min":0.66600000000000004,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.012,"5min":1.431,"15min":1.431},"last":0.753}]}]},{"osd":10,"up_from":70,"seq":300647710755,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7224,"kb_used_data":1648,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93737928,"statfs":{"total":95995035648,"available":95987638272,"internally_reserved":0,"allocated":1687552,"data_stored":1144230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:45:19 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.502}]},{"osd":1,"last update":"Thu Mar 9 14:45:19 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.46700000000000003,"15min":0.46999999999999997},"min":{"1min":0.30499999999999999,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.56699999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.47799999999999998,"15min":0.48199999999999998},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.61099999999999999,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.47499999999999998}]},{"osd":2,"last update":"Thu Mar 9 14:45:19 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.63600000000000001,"15min":0.63900000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.753,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.751},{"interface":"front","average":{"1min":0.628,"5min":0.623,"15min":0.622},"min":{"1min":0.46400000000000002,"5min":0.441,"15min":0.441},"max":{"1min":0.82199999999999995,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.69599999999999995}]},{"osd":3,"last update":"Thu Mar 9 14:45:19 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.58399999999999996,"15min":0.58899999999999997},"min":{"1min":0.38300000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.74399999999999999,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.64200000000000002,"15min":0.64800000000000002},"min":{"1min":0.46600000000000003,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.76100000000000001,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.63500000000000001}]},{"osd":4,"last update":"Thu Mar 9 14:45:19 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.59099999999999997,"15min":0.59299999999999997},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.78200000000000003,"5min":1.1719999999999999,"15min":1.1719999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.54500000000000004,"15min":0.54900000000000004},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.77500000000000002,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.62}]},{"osd":5,"last update":"Thu Mar 9 14:45:19 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.504,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.92700000000000005,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.90300000000000002,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.73699999999999999}]},{"osd":6,"last update":"Thu Mar 9 14:45:19 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.68300000000000005,"15min":0.68899999999999995},"min":{"1min":0.48799999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.86499999999999999,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.71099999999999997,"15min":0.71599999999999997},"min":{"1min":0.499,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.82699999999999996,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.72299999999999998}]},{"osd":7,"last update":"Thu Mar 9 14:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.327,"5min":1.327,"15min":1.327},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.048,"5min":1.048,"15min":1.048},"last":0.53800000000000003}]},{"osd":8,"last update":"Thu Mar 9 14:45:19 2023","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.76000000000000001,"15min":0.76200000000000001},"min":{"1min":0.54900000000000004,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":0.91000000000000003,"5min":1.246,"15min":1.246},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.77300000000000002,"15min":0.78200000000000003},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.95499999999999996,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.871}]},{"osd":9,"last update":"Thu Mar 9 14:45:19 2023","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.77600000000000002,"15min":0.78400000000000003},"min":{"1min":0.54200000000000004,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.88100000000000001,"5min":1.298,"15min":1.298},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.77700000000000002,"15min":0.79200000000000004},"min":{"1min":0.47499999999999998,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.89200000000000002,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":0.71199999999999997}]},{"osd":11,"last update":"Thu Mar 9 14:45:42 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.755,"15min":0.753},"min":{"1min":0.68799999999999994,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.628,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.88300000000000001}]}]},{"osd":9,"up_from":65,"seq":279172874280,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6376,"kb_used_data":992,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738776,"statfs":{"total":95995035648,"available":95988506624,"internally_reserved":0,"allocated":1015808,"data_stored":491063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:45:00 2023","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.46200000000000002,"15min":0.47399999999999998},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.66300000000000003,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.49399999999999999,"15min":0.50700000000000001},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.63700000000000001,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.36699999999999999}]},{"osd":1,"last update":"Thu Mar 9 14:45:00 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.626,"15min":0.63600000000000001},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.008,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.63700000000000001,"15min":0.65000000000000002},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.79900000000000004,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.56999999999999995}]},{"osd":2,"last update":"Thu Mar 9 14:45:00 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.64600000000000002,"15min":0.65800000000000003},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.66100000000000003,"15min":0.67400000000000004},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.88300000000000001,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.497}]},{"osd":3,"last update":"Thu Mar 9 14:45:00 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.56999999999999995,"15min":0.57999999999999996},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.59499999999999997,"15min":0.61499999999999999},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.85099999999999998,"5min":1.016,"15min":1.016},"last":0.38}]},{"osd":4,"last update":"Thu Mar 9 14:45:00 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.67900000000000005,"15min":0.69699999999999995},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.80400000000000005,"5min":1.04,"15min":1.04},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.621,"5min":0.71699999999999997,"15min":0.73299999999999998},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.93899999999999995,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.58099999999999996}]},{"osd":5,"last update":"Thu Mar 9 14:45:00 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.73299999999999998,"15min":0.745},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.98599999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.74099999999999999,"15min":0.754},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.0369999999999999,"5min":1.069,"15min":1.069},"last":0.52300000000000002}]},{"osd":6,"last update":"Thu Mar 9 14:43:55 2023","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.623}]},{"osd":7,"last update":"Thu Mar 9 14:45:00 2023","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.79800000000000004,"15min":0.82599999999999996},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.0229999999999999,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.80900000000000005,"15min":0.83099999999999996},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.99299999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.71499999999999997}]},{"osd":8,"last update":"Thu Mar 9 14:45:00 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.80200000000000005,"15min":0.81599999999999995},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.032,"5min":1.133,"15min":1.133},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.81999999999999995,"15min":0.83799999999999997},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.081,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.48199999999999998}]},{"osd":10,"last update":"Thu Mar 9 14:45:21 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.77800000000000002,"15min":0.79200000000000004},"min":{"1min":0.49199999999999999,"5min":0.435,"15min":0.435},"max":{"1min":0.97299999999999998,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.77900000000000003,"15min":0.79200000000000004},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.056,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.72999999999999998}]},{"osd":11,"last update":"Thu Mar 9 14:45:23 2023","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":2.6659999999999999,"5min":2.6659999999999999,"15min":2.6659999999999999},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.60699999999999998}]}]},{"osd":8,"up_from":57,"seq":244813135916,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6560,"kb_used_data":984,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738592,"statfs":{"total":95995035648,"available":95988318208,"internally_reserved":0,"allocated":1007616,"data_stored":490086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.51900000000000002,"15min":0.51500000000000001},"min":{"1min":0.46600000000000003,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.73199999999999998,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.54000000000000004,"15min":0.54800000000000004},"min":{"1min":0.36299999999999999,"5min":0.31,"15min":0.31},"max":{"1min":0.65700000000000003,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.55100000000000005}]},{"osd":1,"last update":"Thu Mar 9 14:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.57399999999999995,"15min":0.54800000000000004},"min":{"1min":0.434,"5min":0.313,"15min":0.313},"max":{"1min":0.88400000000000001,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.57799999999999996,"15min":0.56399999999999995},"min":{"1min":0.41699999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.79200000000000004,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.71599999999999997}]},{"osd":2,"last update":"Thu Mar 9 14:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.438,"15min":0.42199999999999999},"min":{"1min":0.314,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.45000000000000001,"15min":0.433},"min":{"1min":0.39700000000000002,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.51700000000000002}]},{"osd":3,"last update":"Thu Mar 9 14:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.47599999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.69299999999999995,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.621},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.628,"15min":0.61399999999999999},"min":{"1min":0.51200000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.67700000000000005,"5min":3.9300000000000002,"15min":3.9300000000000002},"last":0.60299999999999998}]},{"osd":4,"last update":"Thu Mar 9 14:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.58399999999999996,"15min":0.56299999999999994},"min":{"1min":0.35999999999999999,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.78400000000000003,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.61599999999999999,"15min":0.59299999999999997},"min":{"1min":0.48899999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.85599999999999998,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.68200000000000005}]},{"osd":5,"last update":"Thu Mar 9 14:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.54900000000000004,"15min":0.53700000000000003},"min":{"1min":0.47799999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.70299999999999996,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.629,"5min":0.56100000000000005,"15min":0.54200000000000004},"min":{"1min":0.40500000000000003,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.84599999999999997}]},{"osd":6,"last update":"Thu Mar 9 14:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.70599999999999996,"15min":0.68200000000000005},"min":{"1min":0.53200000000000003,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.89500000000000002,"5min":3.895,"15min":3.895},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.66000000000000003,"15min":0.65200000000000002},"min":{"1min":0.51100000000000001,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.82199999999999995,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.81599999999999995}]},{"osd":7,"last update":"Thu Mar 9 14:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.67700000000000005,"15min":0.66700000000000004},"min":{"1min":0.51400000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.84899999999999998,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.68999999999999995,"15min":0.67000000000000004},"min":{"1min":0.501,"5min":0.44,"15min":0.44},"max":{"1min":0.90500000000000003,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.93799999999999994}]},{"osd":9,"last update":"Thu Mar 9 14:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.67600000000000005,"15min":0.66800000000000004},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.036,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.754,"5min":0.68600000000000005,"15min":0.67500000000000004},"min":{"1min":0.52900000000000003,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.119,"5min":1.119,"15min":1.119},"last":0.747}]},{"osd":10,"last update":"Thu Mar 9 14:45:18 2023","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.70099999999999996,"15min":0.68000000000000005},"min":{"1min":0.59499999999999997,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.099,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.95899999999999996},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.70899999999999996,"15min":0.68700000000000006},"min":{"1min":0.60599999999999998,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.79600000000000004}]},{"osd":11,"last update":"Thu Mar 9 14:45:23 2023","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.91400000000000003},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.72899999999999998}]}]},{"osd":7,"up_from":50,"seq":214748364849,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7212,"kb_used_data":1572,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93737940,"statfs":{"total":95995035648,"available":95987650560,"internally_reserved":0,"allocated":1609728,"data_stored":1087032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:44:13 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.52100000000000002,"15min":0.51700000000000002},"min":{"1min":0.316,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.69999999999999996,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.375},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.51900000000000002,"15min":0.50900000000000001},"min":{"1min":0.32800000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.90100000000000002,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.33900000000000002}]},{"osd":1,"last update":"Thu Mar 9 14:45:14 2023","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.40600000000000003,"15min":0.39400000000000002},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.54500000000000004,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.40600000000000003,"15min":0.39100000000000001},"min":{"1min":0.29799999999999999,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.56999999999999995,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.45600000000000002}]},{"osd":2,"last update":"Thu Mar 9 14:45:14 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.54600000000000004,"15min":0.53400000000000003},"min":{"1min":0.33500000000000002,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.754,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.58399999999999996,"15min":0.56999999999999995},"min":{"1min":0.44600000000000001,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.80700000000000005,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.71699999999999997}]},{"osd":3,"last update":"Thu Mar 9 14:45:14 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.55700000000000005,"15min":0.53800000000000003},"min":{"1min":0.40200000000000002,"5min":0.37,"15min":0.37},"max":{"1min":0.76000000000000001,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.57999999999999996,"15min":0.56399999999999995},"min":{"1min":0.44,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.76400000000000001,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.78000000000000003}]},{"osd":4,"last update":"Thu Mar 9 14:45:14 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48599999999999999,"15min":0.47199999999999998},"min":{"1min":0.34799999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.65100000000000002,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.47599999999999998,"15min":0.46000000000000002},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.68100000000000005,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.497}]},{"osd":5,"last update":"Thu Mar 9 14:45:14 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.58799999999999997,"15min":0.57399999999999995},"min":{"1min":0.40300000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.76200000000000001,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.623,"5min":0.59499999999999997,"15min":0.57799999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.80200000000000005,"5min":0.874,"15min":0.874},"last":0.63200000000000001}]},{"osd":6,"last update":"Thu Mar 9 14:45:14 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.63300000000000001,"15min":0.61699999999999999},"min":{"1min":0.50600000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.623,"15min":0.61299999999999999},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.81000000000000005,"5min":0.878,"15min":0.878},"last":0.60599999999999998}]},{"osd":8,"last update":"Thu Mar 9 14:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.66600000000000004,"15min":0.65900000000000003},"min":{"1min":0.57299999999999995,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.84799999999999998,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.65700000000000003,"15min":0.65000000000000002},"min":{"1min":0.53700000000000003,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.83699999999999997}]},{"osd":9,"last update":"Thu Mar 9 14:44:55 2023","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.69299999999999995,"15min":0.69199999999999995},"min":{"1min":0.52600000000000002,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":0.91400000000000003,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.70199999999999996,"15min":0.70399999999999996},"min":{"1min":0.52000000000000002,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":0.88700000000000001,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.87}]},{"osd":10,"last update":"Thu Mar 9 14:45:20 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.66300000000000003,"15min":0.66600000000000004},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.83799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.70299999999999996,"15min":0.70099999999999996},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.81799999999999995,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.88700000000000001}]},{"osd":11,"last update":"Thu Mar 9 14:45:20 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.91000000000000003},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.82599999999999996}]}]},{"osd":6,"up_from":45,"seq":193273528374,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6552,"kb_used_data":976,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738600,"statfs":{"total":95995035648,"available":95988326400,"internally_reserved":0,"allocated":999424,"data_stored":486345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:45:55 2023","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.40300000000000002,"15min":0.40899999999999997},"min":{"1min":0.28000000000000003,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.503,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.49299999999999999,"15min":0.49099999999999999},"min":{"1min":0.35699999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.66300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.45700000000000002}]},{"osd":1,"last update":"Thu Mar 9 14:45:55 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.56999999999999995,"15min":0.59499999999999997},"min":{"1min":0.38,"5min":0.315,"15min":0.315},"max":{"1min":0.71699999999999997,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.42099999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.75700000000000001,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.53000000000000003}]},{"osd":2,"last update":"Thu Mar 9 14:45:55 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.59599999999999997,"15min":0.63200000000000001},"min":{"1min":0.38900000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.72799999999999998,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.622,"15min":0.63300000000000001},"min":{"1min":0.42399999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.79900000000000004,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.39200000000000002}]},{"osd":3,"last update":"Thu Mar 9 14:45:55 2023","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.44700000000000001,"15min":0.45700000000000002},"min":{"1min":0.32300000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.48899999999999999,"5min":0.745,"15min":0.745},"last":0.308},{"interface":"front","average":{"1min":0.501,"5min":0.55300000000000005,"15min":0.55400000000000005},"min":{"1min":0.36699999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.72899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.505}]},{"osd":4,"last update":"Thu Mar 9 14:45:55 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.64500000000000002,"15min":0.72299999999999998},"min":{"1min":0.41999999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.73299999999999998,"5min":2.0840000000000001,"15min":2.0840000000000001},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.747,"15min":0.871},"min":{"1min":0.42999999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.71599999999999997,"5min":4.1440000000000001,"15min":4.1440000000000001},"last":0.64000000000000001}]},{"osd":5,"last update":"Thu Mar 9 14:45:55 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.68300000000000005,"15min":0.69499999999999995},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.86099999999999999,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.64200000000000002,"15min":0.67600000000000005},"min":{"1min":0.47799999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.746,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.48599999999999999}]},{"osd":7,"last update":"Thu Mar 9 14:45:20 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.68100000000000005,"15min":0.70299999999999996},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.78600000000000003,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.71499999999999997,"15min":0.73099999999999998},"min":{"1min":0.5,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.87,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.71399999999999997}]},{"osd":8,"last update":"Thu Mar 9 14:45:36 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.75,"15min":0.78000000000000003},"min":{"1min":0.54600000000000004,"5min":0.497,"15min":0.497},"max":{"1min":0.84699999999999998,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.68799999999999994,"15min":0.71599999999999997},"min":{"1min":0.502,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.77700000000000002,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.436}]},{"osd":9,"last update":"Thu Mar 9 14:45:55 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.64300000000000002,"15min":0.67300000000000004},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.72099999999999997,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.70599999999999996,"15min":0.74199999999999999},"min":{"1min":0.49099999999999999,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.77400000000000002,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.56599999999999995}]},{"osd":10,"last update":"Thu Mar 9 14:45:20 2023","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.70599999999999996,"15min":0.70799999999999996},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.82099999999999995,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.73999999999999999,"15min":0.746},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":0.89800000000000002,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.745}]},{"osd":11,"last update":"Thu Mar 9 14:45:28 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.65200000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574922,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7640,"kb_used_data":1040,"kb_used_omap":0,"kb_used_meta":6592,"kb_avail":93737512,"statfs":{"total":95995035648,"available":95987212288,"internally_reserved":0,"allocated":1064960,"data_stored":548252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6750208},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:45:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.50900000000000001,"15min":0.49299999999999999},"min":{"1min":0.36599999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.63300000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.55100000000000005,"15min":0.497},"min":{"1min":0.42099999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.88700000000000001,"5min":2.0670000000000002,"15min":2.0670000000000002},"last":0.495}]},{"osd":2,"last update":"Thu Mar 9 14:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.59099999999999997,"15min":0.54400000000000004},"min":{"1min":0.60899999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.05,"5min":1.3839999999999999,"15min":1.3839999999999999},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.56000000000000005,"15min":0.51800000000000002},"min":{"1min":0.45300000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.96399999999999997,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.57999999999999996}]},{"osd":3,"last update":"Thu Mar 9 14:45:56 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56699999999999995,"15min":0.51500000000000001},"min":{"1min":0.38300000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.83999999999999997,"5min":2.206,"15min":2.206},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.59099999999999997,"15min":0.53500000000000003},"min":{"1min":0.43099999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.81699999999999995,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.54800000000000004}]},{"osd":4,"last update":"Thu Mar 9 14:45:18 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.441,"15min":0.39500000000000002},"min":{"1min":0.39900000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.434,"15min":0.39000000000000001},"min":{"1min":0.34100000000000003,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.82099999999999995}]},{"osd":5,"last update":"Thu Mar 9 14:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.623,"15min":0.56299999999999994},"min":{"1min":0.53200000000000003,"5min":0.373,"15min":0.373},"max":{"1min":1.081,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.61299999999999999,"15min":0.55800000000000005},"min":{"1min":0.53400000000000003,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.92900000000000005,"5min":1.363,"15min":1.363},"last":0.61799999999999999}]},{"osd":6,"last update":"Thu Mar 9 14:45:56 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.65200000000000002,"15min":0.65600000000000003},"min":{"1min":0.47199999999999998,"5min":0.371,"15min":0.371},"max":{"1min":0.91600000000000004,"5min":2.1930000000000001,"15min":2.1930000000000001},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.65800000000000003,"15min":0.66400000000000003},"min":{"1min":0.45300000000000001,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.90200000000000002,"5min":2.2519999999999998,"15min":2.2519999999999998},"last":0.65700000000000003}]},{"osd":7,"last update":"Thu Mar 9 14:45:11 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.55100000000000005,"15min":0.5},"min":{"1min":0.33800000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.55400000000000005,"15min":0.51600000000000001},"min":{"1min":0.35799999999999998,"5min":0.31,"15min":0.31},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.69199999999999995}]},{"osd":8,"last update":"Thu Mar 9 14:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.65600000000000003,"15min":0.622},"min":{"1min":0.57199999999999995,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.92600000000000005,"5min":1.3009999999999999,"15min":1.3009999999999999},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.71999999999999997,"15min":0.68400000000000005},"min":{"1min":0.497,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.1339999999999999,"5min":1.423,"15min":1.423},"last":0.80700000000000005}]},{"osd":9,"last update":"Thu Mar 9 14:45:58 2023","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.70899999999999996,"15min":0.67300000000000004},"min":{"1min":0.60099999999999998,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.70899999999999996,"15min":0.68000000000000005},"min":{"1min":0.65600000000000003,"5min":0.376,"15min":0.376},"max":{"1min":0.96299999999999997,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.79600000000000004}]},{"osd":10,"last update":"Thu Mar 9 14:45:19 2023","interfaces":[{"interface":"back","average":{"1min":0.86799999999999999,"5min":0.75600000000000001,"15min":0.73699999999999999},"min":{"1min":0.66800000000000004,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.77100000000000002,"15min":0.75800000000000001},"min":{"1min":0.68000000000000005,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.0109999999999999,"5min":1.022,"15min":1.022},"last":0.84999999999999998}]},{"osd":11,"last update":"Thu Mar 9 14:45:18 2023","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.1559999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.873,"5min":0.873,"15min":0.873},"min":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.879}]}]},{"osd":0,"up_from":8,"seq":34359738448,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7468,"kb_used_data":996,"kb_used_omap":0,"kb_used_meta":6464,"kb_avail":93737684,"statfs":{"total":95995035648,"available":95987388416,"internally_reserved":0,"allocated":1019904,"data_stored":487611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6619136},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Mar 9 14:45:11 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.55200000000000005,"15min":0.52200000000000002},"min":{"1min":0.375,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.67700000000000005,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.54300000000000004,"15min":0.52700000000000002},"min":{"1min":0.371,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.78900000000000003,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.71799999999999997}]},{"osd":2,"last update":"Thu Mar 9 14:45:31 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.56599999999999995,"15min":0.55600000000000005},"min":{"1min":0.40699999999999997,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.80800000000000005,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60699999999999998,"15min":0.58199999999999996},"min":{"1min":0.45200000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.79400000000000004,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.68200000000000005}]},{"osd":3,"last update":"Thu Mar 9 14:45:20 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.379}]},{"osd":4,"last update":"Thu Mar 9 14:45:11 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.59699999999999998,"15min":0.57099999999999995},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.78400000000000003,"5min":1.139,"15min":1.139},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.55800000000000005,"15min":0.54600000000000004},"min":{"1min":0.38700000000000001,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.88900000000000001,"5min":1.071,"15min":1.071},"last":0.60499999999999998}]},{"osd":5,"last update":"Thu Mar 9 14:45:38 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.60699999999999998,"15min":0.57099999999999995},"min":{"1min":0.504,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.66100000000000003,"15min":0.63100000000000001},"min":{"1min":0.53300000000000003,"5min":0.378,"15min":0.378},"max":{"1min":0.85099999999999998,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.873}]},{"osd":6,"last update":"Thu Mar 9 14:45:57 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.54400000000000004,"15min":0.50700000000000001},"min":{"1min":0.35199999999999998,"5min":0.309,"15min":0.309},"max":{"1min":0.88200000000000001,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.58499999999999996,"15min":0.55100000000000005},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.79500000000000004,"5min":1.159,"15min":1.159},"last":0.78000000000000003}]},{"osd":7,"last update":"Thu Mar 9 14:45:25 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.83899999999999997}]},{"osd":8,"last update":"Thu Mar 9 14:45:38 2023","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.72199999999999998,"15min":0.73299999999999998},"min":{"1min":0.58899999999999997,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.97399999999999998,"5min":1.038,"15min":1.038},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.755,"5min":0.79000000000000004,"15min":0.81799999999999995},"min":{"1min":0.56499999999999995,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.93899999999999995,"5min":1.242,"15min":1.242},"last":0.90100000000000002}]},{"osd":9,"last update":"Thu Mar 9 14:44:59 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.74299999999999999,"15min":0.75900000000000001},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.90600000000000003,"5min":1.105,"15min":1.105},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.82199999999999995,"15min":0.85399999999999998},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.88800000000000001,"5min":1.27,"15min":1.27},"last":0.629}]},{"osd":10,"last update":"Thu Mar 9 14:45:20 2023","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.79000000000000004,"15min":0.79800000000000004},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.93500000000000005,"5min":1.151,"15min":1.151},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.70999999999999996,"15min":0.71199999999999997},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.65100000000000002}]},{"osd":11,"last update":"Thu Mar 9 14:45:38 2023","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.73899999999999999,"15min":0.73299999999999998},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.71899999999999997,"15min":0.71799999999999997},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.70399999999999996}]}]},{"osd":2,"up_from":18,"seq":77309411399,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7708,"kb_used_data":1044,"kb_used_omap":0,"kb_used_meta":6656,"kb_avail":93737444,"statfs":{"total":95995035648,"available":95987142656,"internally_reserved":0,"allocated":1069056,"data_stored":543544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6815744},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:45:32 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.52900000000000003,"15min":0.5},"min":{"1min":0.40500000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.48399999999999999,"15min":0.46999999999999997},"min":{"1min":0.32400000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.51200000000000001}]},{"osd":1,"last update":"Thu Mar 9 14:45:32 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.56899999999999995,"15min":0.51000000000000001},"min":{"1min":0.41299999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.83599999999999997,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.58399999999999996,"15min":0.51800000000000002},"min":{"1min":0.44600000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.85699999999999998,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.81200000000000006}]},{"osd":3,"last update":"Thu Mar 9 14:45:52 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.57899999999999996,"15min":0.54300000000000004},"min":{"1min":0.36599999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.81999999999999995,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.63},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.55200000000000005,"15min":0.53600000000000003},"min":{"1min":0.39800000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.70799999999999996,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.66200000000000003}]},{"osd":4,"last update":"Thu Mar 9 14:45:21 2023","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.83499999999999996}]},{"osd":5,"last update":"Thu Mar 9 14:45:38 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.48999999999999999,"15min":0.46600000000000003},"min":{"1min":0.32100000000000001,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.77300000000000002,"5min":0.871,"15min":0.871},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.46300000000000002,"15min":0.44400000000000001},"min":{"1min":0.33600000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.747,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.64600000000000002}]},{"osd":6,"last update":"Thu Mar 9 14:45:52 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.59199999999999997,"15min":0.57899999999999996},"min":{"1min":0.41099999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.79600000000000004,"5min":1.002,"15min":1.002},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.64700000000000002,"15min":0.621},"min":{"1min":0.53600000000000003,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.879,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.755}]},{"osd":7,"last update":"Thu Mar 9 14:45:22 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.87},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.88300000000000001}]},{"osd":8,"last update":"Thu Mar 9 14:45:38 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.65700000000000003,"15min":0.65200000000000002},"min":{"1min":0.42199999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.90400000000000003,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.92400000000000004},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64500000000000002,"15min":0.65100000000000002},"min":{"1min":0.45800000000000002,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.94099999999999995,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.73899999999999999}]},{"osd":9,"last update":"Thu Mar 9 14:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.72399999999999998,"15min":0.72999999999999998},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.0109999999999999,"5min":1.21,"15min":1.21},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.67100000000000004,"15min":0.68000000000000005},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.93200000000000005,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.71999999999999997}]},{"osd":10,"last update":"Thu Mar 9 14:45:18 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68200000000000005,"15min":0.68100000000000005},"min":{"1min":0.55500000000000005,"5min":0.376,"15min":0.376},"max":{"1min":0.90600000000000003,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.73999999999999999,"15min":0.72899999999999998},"min":{"1min":0.624,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.091,"5min":1.306,"15min":1.306},"last":0.90700000000000003}]},{"osd":11,"last update":"Thu Mar 9 14:45:18 2023","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.95899999999999996},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.149,"5min":1.149,"15min":1.149},"last":0.94399999999999995}]}]},{"osd":3,"up_from":25,"seq":107374182466,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6816,"kb_used_data":984,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738336,"statfs":{"total":95995035648,"available":95988056064,"internally_reserved":0,"allocated":1007616,"data_stored":486449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:45:48 2023","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.377,"15min":0.33200000000000002},"min":{"1min":0.27700000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.498,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":2.669},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.39300000000000002,"15min":0.34799999999999998},"min":{"1min":0.27900000000000003,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.57899999999999996,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":3.1110000000000002}]},{"osd":1,"last update":"Thu Mar 9 14:45:48 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.53500000000000003,"15min":0.50600000000000001},"min":{"1min":0.312,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.77700000000000002,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57199999999999995,"15min":0.503},"min":{"1min":0.41399999999999998,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.67400000000000004,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":6.2329999999999997}]},{"osd":2,"last update":"Thu Mar 9 14:45:48 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.57599999999999996,"15min":0.52200000000000002},"min":{"1min":0.40500000000000003,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.70699999999999996,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":1.361},{"interface":"front","average":{"1min":0.621,"5min":0.58899999999999997,"15min":0.53900000000000003},"min":{"1min":0.40899999999999997,"5min":0.312,"15min":0.312},"max":{"1min":1.776,"5min":1.776,"15min":1.776},"last":1.2350000000000001}]},{"osd":4,"last update":"Thu Mar 9 14:45:10 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.58099999999999996,"15min":0.54200000000000004},"min":{"1min":0.45200000000000001,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.75900000000000001,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":3.5019999999999998},{"interface":"front","average":{"1min":0.622,"5min":0.60099999999999998,"15min":0.55000000000000004},"min":{"1min":0.497,"5min":0.373,"15min":0.373},"max":{"1min":0.84599999999999997,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":2.4449999999999998}]},{"osd":5,"last update":"Thu Mar 9 14:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.58499999999999996,"15min":0.54900000000000004},"min":{"1min":0.438,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.861,"5min":1.861,"15min":1.861},"last":1.341},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.60699999999999998,"15min":0.56699999999999995},"min":{"1min":0.46200000000000002,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.68600000000000005,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":7.2220000000000004}]},{"osd":6,"last update":"Thu Mar 9 14:45:54 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.55600000000000005,"15min":0.53400000000000003},"min":{"1min":0.36299999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.72199999999999998,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":7.9909999999999997},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.57399999999999995,"15min":0.54400000000000004},"min":{"1min":0.34499999999999997,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.81000000000000005,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":3.8759999999999999}]},{"osd":7,"last update":"Thu Mar 9 14:45:17 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.69399999999999995,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":1.5580000000000001},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.68000000000000005,"15min":0.66700000000000004},"min":{"1min":0.46500000000000002,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.80400000000000005,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":9.3710000000000004}]},{"osd":8,"last update":"Thu Mar 9 14:43:35 2023","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.749}]},{"osd":9,"last update":"Thu Mar 9 14:45:54 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.73099999999999998,"15min":0.77600000000000002},"min":{"1min":0.434,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.75700000000000001,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":10.67},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.73999999999999999,"15min":0.78800000000000003},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.80700000000000005,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":4.4400000000000004}]},{"osd":10,"last update":"Thu Mar 9 14:45:25 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.69899999999999995,"15min":0.70799999999999996},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":0.76400000000000001,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":1.667},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.79200000000000004,"15min":0.80500000000000005},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.83199999999999996,"5min":1.155,"15min":1.155},"last":11.667999999999999}]},{"osd":11,"last update":"Thu Mar 9 14:45:19 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":12.622},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":4.1820000000000004}]}]},{"osd":4,"up_from":32,"seq":137438953534,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6700,"kb_used_data":996,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738452,"statfs":{"total":95995035648,"available":95988174848,"internally_reserved":0,"allocated":1019904,"data_stored":492624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:44:09 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.53100000000000003,"15min":0.495},"min":{"1min":0.39000000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.84099999999999997,"5min":1.8320000000000001,"15min":1.8320000000000001},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.628,"5min":0.52900000000000003,"15min":0.495},"min":{"1min":0.42499999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":1.044,"5min":1.478,"15min":1.478},"last":0.86899999999999999}]},{"osd":1,"last update":"Thu Mar 9 14:45:14 2023","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.376,"15min":0.34799999999999998},"min":{"1min":0.311,"5min":0.253,"15min":0.253},"max":{"1min":0.53100000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.39600000000000002,"15min":0.36799999999999999},"min":{"1min":0.316,"5min":0.253,"15min":0.253},"max":{"1min":0.625,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.48099999999999998}]},{"osd":2,"last update":"Thu Mar 9 14:45:14 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.54100000000000004,"15min":0.53000000000000003},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.087,"5min":1.087,"15min":1.087},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.57199999999999995,"15min":0.53900000000000003},"min":{"1min":0.41499999999999998,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.80100000000000005}]},{"osd":3,"last update":"Thu Mar 9 14:45:14 2023","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":0.69899999999999995,"15min":0.57099999999999995},"min":{"1min":0.47699999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":9.9420000000000002,"5min":9.9420000000000002,"15min":9.9420000000000002},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.59699999999999998,"15min":0.54100000000000004},"min":{"1min":0.45200000000000001,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":2.6110000000000002,"5min":2.6110000000000002,"15min":2.6110000000000002},"last":0.77800000000000002}]},{"osd":5,"last update":"Thu Mar 9 14:45:33 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.59399999999999997,"15min":0.56299999999999994},"min":{"1min":0.47099999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.755,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.629,"5min":0.60999999999999999,"15min":0.57099999999999995},"min":{"1min":0.495,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.85999999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.61699999999999999}]},{"osd":6,"last update":"Thu Mar 9 14:45:54 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.66400000000000003,"15min":0.68700000000000006},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.77000000000000002,"5min":2.4609999999999999,"15min":2.4609999999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.72599999999999998,"15min":0.69199999999999995},"min":{"1min":0.48899999999999999,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.91400000000000003,"5min":4.6879999999999997,"15min":4.6879999999999997},"last":0.91400000000000003}]},{"osd":7,"last update":"Thu Mar 9 14:45:14 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.56999999999999995,"15min":0.53400000000000003},"min":{"1min":0.42299999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.872,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.58199999999999996,"15min":0.54000000000000004},"min":{"1min":0.437,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.77400000000000002,"5min":0.998,"15min":0.998},"last":0.65000000000000002}]},{"osd":8,"last update":"Thu Mar 9 14:45:36 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.71599999999999997,"15min":0.69099999999999995},"min":{"1min":0.40200000000000002,"5min":0.377,"15min":0.377},"max":{"1min":0.90100000000000002,"5min":1.7569999999999999,"15min":1.7569999999999999},"last":0.88500000000000001},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.72399999999999998,"15min":0.69699999999999995},"min":{"1min":0.58999999999999997,"5min":0.498,"15min":0.498},"max":{"1min":0.89100000000000001,"5min":1.746,"15min":1.746},"last":0.76300000000000001}]},{"osd":9,"last update":"Thu Mar 9 14:44:58 2023","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.78000000000000003,"15min":0.76100000000000001},"min":{"1min":0.59599999999999997,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":2.5419999999999998,"5min":2.5419999999999998,"15min":2.5419999999999998},"last":0.89700000000000002},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.72799999999999998,"15min":0.71399999999999997},"min":{"1min":0.55200000000000005,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":2.4969999999999999,"5min":2.4969999999999999,"15min":2.4969999999999999},"last":0.82299999999999995}]},{"osd":10,"last update":"Thu Mar 9 14:45:20 2023","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.77100000000000002,"15min":0.77200000000000002},"min":{"1min":0.61099999999999999,"5min":0.432,"15min":0.432},"max":{"1min":1.0009999999999999,"5min":1.069,"15min":1.069},"last":0.93799999999999994},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.79000000000000004,"15min":0.79300000000000004},"min":{"1min":0.627,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.109,"5min":1.228,"15min":1.228},"last":0.81599999999999995}]},{"osd":11,"last update":"Thu Mar 9 14:45:18 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.1779999999999999,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.85799999999999998}]}]},{"osd":5,"up_from":37,"seq":158913790010,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7336,"kb_used_data":1568,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93737816,"statfs":{"total":95995035648,"available":95987523584,"internally_reserved":0,"allocated":1605632,"data_stored":1081598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:45:38 2023","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.59499999999999997,"15min":0.51300000000000001},"min":{"1min":0.32400000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":9.7189999999999994,"5min":9.7189999999999994,"15min":9.7189999999999994},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.56299999999999994,"15min":0.51500000000000001},"min":{"1min":0.307,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":5.4660000000000002,"5min":5.4660000000000002,"15min":5.4660000000000002},"last":0.45100000000000001}]},{"osd":1,"last update":"Thu Mar 9 14:45:38 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.57499999999999996,"15min":0.55400000000000005},"min":{"1min":0.45400000000000001,"5min":0.316,"15min":0.316},"max":{"1min":0.77500000000000002,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.56100000000000005,"15min":0.53700000000000003},"min":{"1min":0.46200000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.752,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.66300000000000003}]},{"osd":2,"last update":"Thu Mar 9 14:45:38 2023","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.39500000000000002,"15min":0.36699999999999999},"min":{"1min":0.313,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.61499999999999999,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.41299999999999998,"15min":0.377},"min":{"1min":0.33900000000000002,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.65500000000000003,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.57299999999999995}]},{"osd":3,"last update":"Thu Mar 9 14:45:38 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.54800000000000004,"15min":0.53800000000000003},"min":{"1min":0.34499999999999997,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":2.7269999999999999,"5min":2.7269999999999999,"15min":2.7269999999999999},"last":0.435},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.622,"15min":0.56899999999999995},"min":{"1min":0.39200000000000002,"5min":0.313,"15min":0.313},"max":{"1min":5.6130000000000004,"5min":5.6130000000000004,"15min":5.6130000000000004},"last":0.58399999999999996}]},{"osd":4,"last update":"Thu Mar 9 14:45:38 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.58999999999999997,"15min":0.56299999999999994},"min":{"1min":0.51400000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.78000000000000003,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.625},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.53700000000000003,"15min":0.52800000000000002},"min":{"1min":0.42599999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.65800000000000003,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.53100000000000003}]},{"osd":6,"last update":"Thu Mar 9 14:45:52 2023","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.626,"15min":0.60799999999999998},"min":{"1min":0.438,"5min":0.315,"15min":0.315},"max":{"1min":2.5369999999999999,"5min":2.5369999999999999,"15min":2.5369999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.66800000000000004,"15min":0.63600000000000001},"min":{"1min":0.504,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":2.5019999999999998,"5min":2.5019999999999998,"15min":2.5019999999999998},"last":0.61499999999999999}]},{"osd":7,"last update":"Thu Mar 9 14:45:18 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.61099999999999999,"15min":0.59399999999999997},"min":{"1min":0.42099999999999999,"5min":0.38,"15min":0.38},"max":{"1min":0.84799999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.61899999999999999,"15min":0.59999999999999998},"min":{"1min":0.46899999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.83299999999999996,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.63900000000000001}]},{"osd":8,"last update":"Thu Mar 9 14:45:38 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.60199999999999998,"15min":0.60099999999999998},"min":{"1min":0.439,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.83499999999999996,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.42999999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.70699999999999996,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.60199999999999998}]},{"osd":9,"last update":"Thu Mar 9 14:44:58 2023","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.67800000000000005,"15min":0.64600000000000002},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":5.6749999999999998,"5min":5.6749999999999998,"15min":5.6749999999999998},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.71599999999999997,"15min":0.71499999999999997},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.468,"5min":1.468,"15min":1.468},"last":0.65600000000000003}]},{"osd":10,"last update":"Thu Mar 9 14:45:18 2023","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.69799999999999995,"15min":0.68999999999999995},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.86499999999999999,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.70899999999999996,"15min":0.69999999999999996},"min":{"1min":0.54300000000000004,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":0.877,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.67400000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":62087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":57379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":63932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-03-09T14:46:02.031 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-03-09T14:46:02.185 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.183+0000 7ff87919b700 1 -- 172.21.15.38:0/3410997610 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff874100900 msgr2=0x7ff8741071c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:02.186 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.183+0000 7ff87919b700 1 --2- 172.21.15.38:0/3410997610 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff874100900 0x7ff8741071c0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7ff86c00b0a0 tx=0x7ff86c00b3b0 comp rx=0 tx=0).stop 2023-03-09T14:46:02.186 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.183+0000 7ff87919b700 1 -- 172.21.15.38:0/3410997610 shutdown_connections 2023-03-09T14:46:02.187 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.183+0000 7ff87919b700 1 --2- 172.21.15.38:0/3410997610 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff874100900 0x7ff8741071c0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:02.187 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.183+0000 7ff87919b700 1 --2- 172.21.15.38:0/3410997610 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff8740fed80 0x7ff8740ff190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:02.187 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.183+0000 7ff87919b700 1 --2- 172.21.15.38:0/3410997610 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff8740fff70 0x7ff8741003c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:02.187 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.183+0000 7ff87919b700 1 -- 172.21.15.38:0/3410997610 >> 172.21.15.38:0/3410997610 conn(0x7ff8740fa380 msgr2=0x7ff8740fc7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:02.187 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.183+0000 7ff87919b700 1 -- 172.21.15.38:0/3410997610 shutdown_connections 2023-03-09T14:46:02.187 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.183+0000 7ff87919b700 1 -- 172.21.15.38:0/3410997610 wait complete. 2023-03-09T14:46:02.187 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.183+0000 7ff87919b700 1 Processor -- start 2023-03-09T14:46:02.189 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.184+0000 7ff87919b700 1 -- start start 2023-03-09T14:46:02.189 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.184+0000 7ff87919b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff8740fed80 0x7ff874197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:02.190 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.184+0000 7ff87919b700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff8740fff70 0x7ff874198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:02.190 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.184+0000 7ff87919b700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff874100900 0x7ff87419f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:02.190 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.184+0000 7ff87919b700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff87419fad0 con 0x7ff8740fed80 2023-03-09T14:46:02.190 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.184+0000 7ff87919b700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff87419fc10 con 0x7ff874100900 2023-03-09T14:46:02.190 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.184+0000 7ff87919b700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff87419fda0 con 0x7ff8740fff70 2023-03-09T14:46:02.190 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.184+0000 7ff872d9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff8740fed80 0x7ff874197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:02.191 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.184+0000 7ff872d9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff8740fed80 0x7ff874197f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38496/0 (socket says 172.21.15.38:38496) 2023-03-09T14:46:02.191 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.184+0000 7ff872d9d700 1 -- 172.21.15.38:0/3434529976 learned_addr learned my addr 172.21.15.38:0/3434529976 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:46:02.191 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.184+0000 7ff87259c700 1 --2- 172.21.15.38:0/3434529976 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff8740fff70 0x7ff874198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:02.191 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.184+0000 7ff87359e700 1 --2- 172.21.15.38:0/3434529976 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff874100900 0x7ff87419f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:02.191 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.184+0000 7ff872d9d700 1 -- 172.21.15.38:0/3434529976 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff8740fff70 msgr2=0x7ff874198490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:02.191 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.184+0000 7ff872d9d700 1 --2- 172.21.15.38:0/3434529976 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff8740fff70 0x7ff874198490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:02.191 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.184+0000 7ff872d9d700 1 -- 172.21.15.38:0/3434529976 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff874100900 msgr2=0x7ff87419f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:02.192 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.184+0000 7ff872d9d700 1 --2- 172.21.15.38:0/3434529976 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff874100900 0x7ff87419f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:02.192 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.184+0000 7ff872d9d700 1 -- 172.21.15.38:0/3434529976 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff86c009d00 con 0x7ff8740fed80 2023-03-09T14:46:02.192 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.184+0000 7ff872d9d700 1 --2- 172.21.15.38:0/3434529976 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff8740fed80 0x7ff874197f50 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7ff86800c970 tx=0x7ff86800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:02.192 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.185+0000 7ff85bfff700 1 -- 172.21.15.38:0/3434529976 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff868004220 con 0x7ff8740fed80 2023-03-09T14:46:02.192 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.185+0000 7ff85bfff700 1 -- 172.21.15.38:0/3434529976 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff868007d60 con 0x7ff8740fed80 2023-03-09T14:46:02.193 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.185+0000 7ff87919b700 1 -- 172.21.15.38:0/3434529976 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff87410a6f0 con 0x7ff8740fed80 2023-03-09T14:46:02.193 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.185+0000 7ff85bfff700 1 -- 172.21.15.38:0/3434529976 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff868003c90 con 0x7ff8740fed80 2023-03-09T14:46:02.193 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.185+0000 7ff87919b700 1 -- 172.21.15.38:0/3434529976 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff87410a560 con 0x7ff8740fed80 2023-03-09T14:46:02.193 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.186+0000 7ff87919b700 1 -- 172.21.15.38:0/3434529976 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff874067660 con 0x7ff8740fed80 2023-03-09T14:46:02.194 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.187+0000 7ff85bfff700 1 -- 172.21.15.38:0/3434529976 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) v1 ==== 133198+0+0 (secure 0 0 0) 0x7ff86801f070 con 0x7ff8740fed80 2023-03-09T14:46:02.194 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.188+0000 7ff85bfff700 1 --2- 172.21.15.38:0/3434529976 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7ff85c09f3c0 0x7ff85c0a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:02.195 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.188+0000 7ff85bfff700 1 -- 172.21.15.38:0/3434529976 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ff8680c9f50 con 0x7ff8740fed80 2023-03-09T14:46:02.195 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.188+0000 7ff87259c700 1 --2- 172.21.15.38:0/3434529976 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7ff85c09f3c0 0x7ff85c0a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:02.195 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.188+0000 7ff87259c700 1 --2- 172.21.15.38:0/3434529976 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7ff85c09f3c0 0x7ff85c0a1870 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7ff864005950 tx=0x7ff86400b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:02.195 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.191+0000 7ff85bfff700 1 -- 172.21.15.38:0/3434529976 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7ff868094000 con 0x7ff8740fed80 2023-03-09T14:46:02.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:01 smithi122 ceph-mon[93869]: pgmap v312: 129 pgs: 129 active+clean; 641 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 7 op/s 2023-03-09T14:46:02.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:01 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/1257975182' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-03-09T14:46:02.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:01 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/3862024022' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-03-09T14:46:02.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:01 smithi181 ceph-mon[94189]: pgmap v312: 129 pgs: 129 active+clean; 641 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 7 op/s 2023-03-09T14:46:02.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:01 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/1257975182' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-03-09T14:46:02.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:01 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/3862024022' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-03-09T14:46:02.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:01 smithi038 ceph-mon[92422]: pgmap v312: 129 pgs: 129 active+clean; 641 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 7 op/s 2023-03-09T14:46:02.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:01 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/1257975182' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-03-09T14:46:02.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:01 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/3862024022' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-03-09T14:46:02.389 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.387+0000 7ff87919b700 1 -- 172.21.15.38:0/3434529976 --> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff8741a0910 con 0x7ff85c09f3c0 2023-03-09T14:46:02.404 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.402+0000 7ff85bfff700 1 -- 172.21.15.38:0/3434529976 <== mgr.14150 v2:172.21.15.38:6800/3328887328 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+427321 (secure 0 0 0) 0x7ff8741a0910 con 0x7ff85c09f3c0 2023-03-09T14:46:02.409 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:46:02.412 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.406+0000 7ff87919b700 1 -- 172.21.15.38:0/3434529976 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7ff85c09f3c0 msgr2=0x7ff85c0a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:02.412 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.406+0000 7ff87919b700 1 --2- 172.21.15.38:0/3434529976 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7ff85c09f3c0 0x7ff85c0a1870 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7ff864005950 tx=0x7ff86400b410 comp rx=0 tx=0).stop 2023-03-09T14:46:02.412 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.406+0000 7ff87919b700 1 -- 172.21.15.38:0/3434529976 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff8740fed80 msgr2=0x7ff874197f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:02.412 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.406+0000 7ff87919b700 1 --2- 172.21.15.38:0/3434529976 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff8740fed80 0x7ff874197f50 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7ff86800c970 tx=0x7ff86800cc80 comp rx=0 tx=0).stop 2023-03-09T14:46:02.413 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.406+0000 7ff87919b700 1 -- 172.21.15.38:0/3434529976 shutdown_connections 2023-03-09T14:46:02.413 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.406+0000 7ff87919b700 1 --2- 172.21.15.38:0/3434529976 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff8740fed80 0x7ff874197f50 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:02.413 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.406+0000 7ff87919b700 1 --2- 172.21.15.38:0/3434529976 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7ff85c09f3c0 0x7ff85c0a1870 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:02.413 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.406+0000 7ff87919b700 1 --2- 172.21.15.38:0/3434529976 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff874100900 0x7ff87419f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:02.413 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.406+0000 7ff87919b700 1 --2- 172.21.15.38:0/3434529976 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff8740fff70 0x7ff874198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:02.413 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.406+0000 7ff87919b700 1 -- 172.21.15.38:0/3434529976 >> 172.21.15.38:0/3434529976 conn(0x7ff8740fa380 msgr2=0x7ff874067a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:02.413 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.406+0000 7ff87919b700 1 -- 172.21.15.38:0/3434529976 shutdown_connections 2023-03-09T14:46:02.413 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.406+0000 7ff87919b700 1 -- 172.21.15.38:0/3434529976 wait complete. 2023-03-09T14:46:02.414 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2023-03-09T14:46:02.421 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":312,"stamp":"2023-03-09T14:46:00.782608+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":656228,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":642,"num_read_kb":445,"num_write":346,"num_write_kb":3056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1182773,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":782,"ondisk_log_size":782,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":83828,"kb_used_data":13780,"kb_used_omap":0,"kb_used_meta":69952,"kb_avail":1124857996,"statfs":{"total":1151940427776,"available":1151854587904,"internally_reserved":0,"allocated":14110720,"data_stored":7829731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":71630848},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4606,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":82,"num_read_kb":51,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003330"},"pg_stats":[{"pgid":"3.3f","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.799653+0000","last_change":"2023-03-09T14:44:19.116371+0000","last_active":"2023-03-09T14:45:44.799653+0000","last_peered":"2023-03-09T14:45:44.799653+0000","last_clean":"2023-03-09T14:45:44.799653+0000","last_became_active":"2023-03-09T14:44:19.116024+0000","last_became_peered":"2023-03-09T14:44:19.116024+0000","last_unstale":"2023-03-09T14:45:44.799653+0000","last_undegraded":"2023-03-09T14:45:44.799653+0000","last_fullsized":"2023-03-09T14:45:44.799653+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T16:43:33.917176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"89'4","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.788538+0000","last_change":"2023-03-09T14:44:19.117312+0000","last_active":"2023-03-09T14:45:44.788538+0000","last_peered":"2023-03-09T14:45:44.788538+0000","last_clean":"2023-03-09T14:45:44.788538+0000","last_became_active":"2023-03-09T14:44:19.116920+0000","last_became_peered":"2023-03-09T14:44:19.116920+0000","last_unstale":"2023-03-09T14:45:44.788538+0000","last_undegraded":"2023-03-09T14:45:44.788538+0000","last_fullsized":"2023-03-09T14:45:44.788538+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:33:15.046431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.798813+0000","last_change":"2023-03-09T14:44:19.113482+0000","last_active":"2023-03-09T14:45:44.798813+0000","last_peered":"2023-03-09T14:45:44.798813+0000","last_clean":"2023-03-09T14:45:44.798813+0000","last_became_active":"2023-03-09T14:44:19.113183+0000","last_became_peered":"2023-03-09T14:44:19.113183+0000","last_unstale":"2023-03-09T14:45:44.798813+0000","last_undegraded":"2023-03-09T14:45:44.798813+0000","last_fullsized":"2023-03-09T14:45:44.798813+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:17:31.382255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.786039+0000","last_change":"2023-03-09T14:44:19.105799+0000","last_active":"2023-03-09T14:45:44.786039+0000","last_peered":"2023-03-09T14:45:44.786039+0000","last_clean":"2023-03-09T14:45:44.786039+0000","last_became_active":"2023-03-09T14:44:19.105405+0000","last_became_peered":"2023-03-09T14:44:19.105405+0000","last_unstale":"2023-03-09T14:45:44.786039+0000","last_undegraded":"2023-03-09T14:45:44.786039+0000","last_fullsized":"2023-03-09T14:45:44.786039+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:38:23.049984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"89'4","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.782474+0000","last_change":"2023-03-09T14:44:19.100656+0000","last_active":"2023-03-09T14:45:44.782474+0000","last_peered":"2023-03-09T14:45:44.782474+0000","last_clean":"2023-03-09T14:45:44.782474+0000","last_became_active":"2023-03-09T14:44:19.100278+0000","last_became_peered":"2023-03-09T14:44:19.100278+0000","last_unstale":"2023-03-09T14:45:44.782474+0000","last_undegraded":"2023-03-09T14:45:44.782474+0000","last_fullsized":"2023-03-09T14:45:44.782474+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:25:39.240251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.787745+0000","last_change":"2023-03-09T14:44:19.101065+0000","last_active":"2023-03-09T14:45:44.787745+0000","last_peered":"2023-03-09T14:45:44.787745+0000","last_clean":"2023-03-09T14:45:44.787745+0000","last_became_active":"2023-03-09T14:44:19.100451+0000","last_became_peered":"2023-03-09T14:44:19.100451+0000","last_unstale":"2023-03-09T14:45:44.787745+0000","last_undegraded":"2023-03-09T14:45:44.787745+0000","last_fullsized":"2023-03-09T14:45:44.787745+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T01:25:15.158254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"89'5","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.787475+0000","last_change":"2023-03-09T14:44:19.107996+0000","last_active":"2023-03-09T14:45:44.787475+0000","last_peered":"2023-03-09T14:45:44.787475+0000","last_clean":"2023-03-09T14:45:44.787475+0000","last_became_active":"2023-03-09T14:44:19.107683+0000","last_became_peered":"2023-03-09T14:44:19.107683+0000","last_unstale":"2023-03-09T14:45:44.787475+0000","last_undegraded":"2023-03-09T14:45:44.787475+0000","last_fullsized":"2023-03-09T14:45:44.787475+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:16:18.605176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.786169+0000","last_change":"2023-03-09T14:44:19.103940+0000","last_active":"2023-03-09T14:45:44.786169+0000","last_peered":"2023-03-09T14:45:44.786169+0000","last_clean":"2023-03-09T14:45:44.786169+0000","last_became_active":"2023-03-09T14:44:19.103620+0000","last_became_peered":"2023-03-09T14:44:19.103620+0000","last_unstale":"2023-03-09T14:45:44.786169+0000","last_undegraded":"2023-03-09T14:45:44.786169+0000","last_fullsized":"2023-03-09T14:45:44.786169+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:06:22.749636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"89'11","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.797150+0000","last_change":"2023-03-09T14:44:19.102695+0000","last_active":"2023-03-09T14:45:44.797150+0000","last_peered":"2023-03-09T14:45:44.797150+0000","last_clean":"2023-03-09T14:45:44.797150+0000","last_became_active":"2023-03-09T14:44:19.101486+0000","last_became_peered":"2023-03-09T14:44:19.101486+0000","last_unstale":"2023-03-09T14:45:44.797150+0000","last_undegraded":"2023-03-09T14:45:44.797150+0000","last_fullsized":"2023-03-09T14:45:44.797150+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T16:34:51.836443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"89'16","reported_seq":33,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.788716+0000","last_change":"2023-03-09T14:44:19.103522+0000","last_active":"2023-03-09T14:45:44.788716+0000","last_peered":"2023-03-09T14:45:44.788716+0000","last_clean":"2023-03-09T14:45:44.788716+0000","last_became_active":"2023-03-09T14:44:19.103242+0000","last_became_peered":"2023-03-09T14:44:19.103242+0000","last_unstale":"2023-03-09T14:45:44.788716+0000","last_undegraded":"2023-03-09T14:45:44.788716+0000","last_fullsized":"2023-03-09T14:45:44.788716+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:41:03.137554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.799291+0000","last_change":"2023-03-09T14:44:19.114934+0000","last_active":"2023-03-09T14:45:44.799291+0000","last_peered":"2023-03-09T14:45:44.799291+0000","last_clean":"2023-03-09T14:45:44.799291+0000","last_became_active":"2023-03-09T14:44:19.114620+0000","last_became_peered":"2023-03-09T14:44:19.114620+0000","last_unstale":"2023-03-09T14:45:44.799291+0000","last_undegraded":"2023-03-09T14:45:44.799291+0000","last_fullsized":"2023-03-09T14:45:44.799291+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T01:48:40.082076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.785702+0000","last_change":"2023-03-09T14:44:19.107677+0000","last_active":"2023-03-09T14:45:44.785702+0000","last_peered":"2023-03-09T14:45:44.785702+0000","last_clean":"2023-03-09T14:45:44.785702+0000","last_became_active":"2023-03-09T14:44:19.107369+0000","last_became_peered":"2023-03-09T14:44:19.107369+0000","last_unstale":"2023-03-09T14:45:44.785702+0000","last_undegraded":"2023-03-09T14:45:44.785702+0000","last_fullsized":"2023-03-09T14:45:44.785702+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:47:40.117205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"89'10","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.786986+0000","last_change":"2023-03-09T14:44:19.108852+0000","last_active":"2023-03-09T14:45:44.786986+0000","last_peered":"2023-03-09T14:45:44.786986+0000","last_clean":"2023-03-09T14:45:44.786986+0000","last_became_active":"2023-03-09T14:44:19.108515+0000","last_became_peered":"2023-03-09T14:44:19.108515+0000","last_unstale":"2023-03-09T14:45:44.786986+0000","last_undegraded":"2023-03-09T14:45:44.786986+0000","last_fullsized":"2023-03-09T14:45:44.786986+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:00:36.243952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.786886+0000","last_change":"2023-03-09T14:44:19.118673+0000","last_active":"2023-03-09T14:45:44.786886+0000","last_peered":"2023-03-09T14:45:44.786886+0000","last_clean":"2023-03-09T14:45:44.786886+0000","last_became_active":"2023-03-09T14:44:19.117656+0000","last_became_peered":"2023-03-09T14:44:19.117656+0000","last_unstale":"2023-03-09T14:45:44.786886+0000","last_undegraded":"2023-03-09T14:45:44.786886+0000","last_fullsized":"2023-03-09T14:45:44.786886+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:02:50.401680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.789818+0000","last_change":"2023-03-09T14:44:19.112979+0000","last_active":"2023-03-09T14:45:44.789818+0000","last_peered":"2023-03-09T14:45:44.789818+0000","last_clean":"2023-03-09T14:45:44.789818+0000","last_became_active":"2023-03-09T14:44:19.112671+0000","last_became_peered":"2023-03-09T14:44:19.112671+0000","last_unstale":"2023-03-09T14:45:44.789818+0000","last_undegraded":"2023-03-09T14:45:44.789818+0000","last_fullsized":"2023-03-09T14:45:44.789818+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:00:43.465327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.795403+0000","last_change":"2023-03-09T14:44:19.104881+0000","last_active":"2023-03-09T14:45:44.795403+0000","last_peered":"2023-03-09T14:45:44.795403+0000","last_clean":"2023-03-09T14:45:44.795403+0000","last_became_active":"2023-03-09T14:44:19.104647+0000","last_became_peered":"2023-03-09T14:44:19.104647+0000","last_unstale":"2023-03-09T14:45:44.795403+0000","last_undegraded":"2023-03-09T14:45:44.795403+0000","last_fullsized":"2023-03-09T14:45:44.795403+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T20:16:24.625362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.799646+0000","last_change":"2023-03-09T14:44:19.117937+0000","last_active":"2023-03-09T14:45:44.799646+0000","last_peered":"2023-03-09T14:45:44.799646+0000","last_clean":"2023-03-09T14:45:44.799646+0000","last_became_active":"2023-03-09T14:44:19.117628+0000","last_became_peered":"2023-03-09T14:44:19.117628+0000","last_unstale":"2023-03-09T14:45:44.799646+0000","last_undegraded":"2023-03-09T14:45:44.799646+0000","last_fullsized":"2023-03-09T14:45:44.799646+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:22:18.619171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"89'10","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.793895+0000","last_change":"2023-03-09T14:44:19.105071+0000","last_active":"2023-03-09T14:45:44.793895+0000","last_peered":"2023-03-09T14:45:44.793895+0000","last_clean":"2023-03-09T14:45:44.793895+0000","last_became_active":"2023-03-09T14:44:19.103396+0000","last_became_peered":"2023-03-09T14:44:19.103396+0000","last_unstale":"2023-03-09T14:45:44.793895+0000","last_undegraded":"2023-03-09T14:45:44.793895+0000","last_fullsized":"2023-03-09T14:45:44.793895+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:14:51.924636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.783812+0000","last_change":"2023-03-09T14:44:19.104107+0000","last_active":"2023-03-09T14:45:44.783812+0000","last_peered":"2023-03-09T14:45:44.783812+0000","last_clean":"2023-03-09T14:45:44.783812+0000","last_became_active":"2023-03-09T14:44:19.103793+0000","last_became_peered":"2023-03-09T14:44:19.103793+0000","last_unstale":"2023-03-09T14:45:44.783812+0000","last_undegraded":"2023-03-09T14:45:44.783812+0000","last_fullsized":"2023-03-09T14:45:44.783812+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:57:29.169744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.856426+0000","last_change":"2023-03-09T14:44:23.489947+0000","last_active":"2023-03-09T14:45:14.856426+0000","last_peered":"2023-03-09T14:45:14.856426+0000","last_clean":"2023-03-09T14:45:14.856426+0000","last_became_active":"2023-03-09T14:44:23.489560+0000","last_became_peered":"2023-03-09T14:44:23.489560+0000","last_unstale":"2023-03-09T14:45:14.856426+0000","last_undegraded":"2023-03-09T14:45:14.856426+0000","last_fullsized":"2023-03-09T14:45:14.856426+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:06:10.202176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"87'9","reported_seq":49,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:15.961952+0000","last_change":"2023-03-09T14:45:15.961952+0000","last_active":"2023-03-09T14:45:15.961952+0000","last_peered":"2023-03-09T14:45:15.961952+0000","last_clean":"2023-03-09T14:45:15.961952+0000","last_became_active":"2023-03-09T14:45:15.865191+0000","last_became_peered":"2023-03-09T14:45:15.865191+0000","last_unstale":"2023-03-09T14:45:15.961952+0000","last_undegraded":"2023-03-09T14:45:15.961952+0000","last_fullsized":"2023-03-09T14:45:15.961952+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T01:21:05.925138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":566,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"89'5","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.785218+0000","last_change":"2023-03-09T14:44:19.105390+0000","last_active":"2023-03-09T14:45:44.785218+0000","last_peered":"2023-03-09T14:45:44.785218+0000","last_clean":"2023-03-09T14:45:44.785218+0000","last_became_active":"2023-03-09T14:44:19.104694+0000","last_became_peered":"2023-03-09T14:44:19.104694+0000","last_unstale":"2023-03-09T14:45:44.785218+0000","last_undegraded":"2023-03-09T14:45:44.785218+0000","last_fullsized":"2023-03-09T14:45:44.785218+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:19:08.101847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.856827+0000","last_change":"2023-03-09T14:44:23.484103+0000","last_active":"2023-03-09T14:45:14.856827+0000","last_peered":"2023-03-09T14:45:14.856827+0000","last_clean":"2023-03-09T14:45:14.856827+0000","last_became_active":"2023-03-09T14:44:23.483271+0000","last_became_peered":"2023-03-09T14:44:23.483271+0000","last_unstale":"2023-03-09T14:45:14.856827+0000","last_undegraded":"2023-03-09T14:45:14.856827+0000","last_fullsized":"2023-03-09T14:45:14.856827+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:24:06.212623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"87'2","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.864060+0000","last_change":"2023-03-09T14:44:18.667321+0000","last_active":"2023-03-09T14:45:14.864060+0000","last_peered":"2023-03-09T14:45:14.864060+0000","last_clean":"2023-03-09T14:45:14.864060+0000","last_became_active":"2023-03-09T14:44:18.666974+0000","last_became_peered":"2023-03-09T14:44:18.666974+0000","last_unstale":"2023-03-09T14:45:14.864060+0000","last_undegraded":"2023-03-09T14:45:14.864060+0000","last_fullsized":"2023-03-09T14:45:14.864060+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:31:07.327792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.799185+0000","last_change":"2023-03-09T14:44:19.116866+0000","last_active":"2023-03-09T14:45:44.799185+0000","last_peered":"2023-03-09T14:45:44.799185+0000","last_clean":"2023-03-09T14:45:44.799185+0000","last_became_active":"2023-03-09T14:44:19.116571+0000","last_became_peered":"2023-03-09T14:44:19.116571+0000","last_unstale":"2023-03-09T14:45:44.799185+0000","last_undegraded":"2023-03-09T14:45:44.799185+0000","last_fullsized":"2023-03-09T14:45:44.799185+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T16:22:12.723903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.859406+0000","last_change":"2023-03-09T14:44:23.488192+0000","last_active":"2023-03-09T14:45:14.859406+0000","last_peered":"2023-03-09T14:45:14.859406+0000","last_clean":"2023-03-09T14:45:14.859406+0000","last_became_active":"2023-03-09T14:44:23.487734+0000","last_became_peered":"2023-03-09T14:44:23.487734+0000","last_unstale":"2023-03-09T14:45:14.859406+0000","last_undegraded":"2023-03-09T14:45:14.859406+0000","last_fullsized":"2023-03-09T14:45:14.859406+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:35:50.319216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"87'4","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.861901+0000","last_change":"2023-03-09T14:44:18.663577+0000","last_active":"2023-03-09T14:45:14.861901+0000","last_peered":"2023-03-09T14:45:14.861901+0000","last_clean":"2023-03-09T14:45:14.861901+0000","last_became_active":"2023-03-09T14:44:18.663272+0000","last_became_peered":"2023-03-09T14:44:18.663272+0000","last_unstale":"2023-03-09T14:45:14.861901+0000","last_undegraded":"2023-03-09T14:45:14.861901+0000","last_fullsized":"2023-03-09T14:45:14.861901+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T20:55:19.661875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.794071+0000","last_change":"2023-03-09T14:44:19.106986+0000","last_active":"2023-03-09T14:45:44.794071+0000","last_peered":"2023-03-09T14:45:44.794071+0000","last_clean":"2023-03-09T14:45:44.794071+0000","last_became_active":"2023-03-09T14:44:19.106625+0000","last_became_peered":"2023-03-09T14:44:19.106625+0000","last_unstale":"2023-03-09T14:45:44.794071+0000","last_undegraded":"2023-03-09T14:45:44.794071+0000","last_fullsized":"2023-03-09T14:45:44.794071+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:03:54.521603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.858888+0000","last_change":"2023-03-09T14:44:23.483803+0000","last_active":"2023-03-09T14:45:14.858888+0000","last_peered":"2023-03-09T14:45:14.858888+0000","last_clean":"2023-03-09T14:45:14.858888+0000","last_became_active":"2023-03-09T14:44:23.483286+0000","last_became_peered":"2023-03-09T14:44:23.483286+0000","last_unstale":"2023-03-09T14:45:14.858888+0000","last_undegraded":"2023-03-09T14:45:14.858888+0000","last_fullsized":"2023-03-09T14:45:14.858888+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:32:35.975670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"82'2","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.861301+0000","last_change":"2023-03-09T14:44:18.657685+0000","last_active":"2023-03-09T14:45:14.861301+0000","last_peered":"2023-03-09T14:45:14.861301+0000","last_clean":"2023-03-09T14:45:14.861301+0000","last_became_active":"2023-03-09T14:44:18.657355+0000","last_became_peered":"2023-03-09T14:44:18.657355+0000","last_unstale":"2023-03-09T14:45:14.861301+0000","last_undegraded":"2023-03-09T14:45:14.861301+0000","last_fullsized":"2023-03-09T14:45:14.861301+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:33:59.259690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"89'15","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.799331+0000","last_change":"2023-03-09T14:44:19.111687+0000","last_active":"2023-03-09T14:45:44.799331+0000","last_peered":"2023-03-09T14:45:44.799331+0000","last_clean":"2023-03-09T14:45:44.799331+0000","last_became_active":"2023-03-09T14:44:19.111481+0000","last_became_peered":"2023-03-09T14:44:19.111481+0000","last_unstale":"2023-03-09T14:45:44.799331+0000","last_undegraded":"2023-03-09T14:45:44.799331+0000","last_fullsized":"2023-03-09T14:45:44.799331+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:06:14.452967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.857536+0000","last_change":"2023-03-09T14:44:23.485075+0000","last_active":"2023-03-09T14:45:14.857536+0000","last_peered":"2023-03-09T14:45:14.857536+0000","last_clean":"2023-03-09T14:45:14.857536+0000","last_became_active":"2023-03-09T14:44:23.484648+0000","last_became_peered":"2023-03-09T14:44:23.484648+0000","last_unstale":"2023-03-09T14:45:14.857536+0000","last_undegraded":"2023-03-09T14:45:14.857536+0000","last_fullsized":"2023-03-09T14:45:14.857536+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:49:40.332276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"87'5","reported_seq":120,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:55.926611+0000","last_change":"2023-03-09T14:45:15.868911+0000","last_active":"2023-03-09T14:45:55.926611+0000","last_peered":"2023-03-09T14:45:55.926611+0000","last_clean":"2023-03-09T14:45:55.926611+0000","last_became_active":"2023-03-09T14:45:15.861592+0000","last_became_peered":"2023-03-09T14:45:15.861592+0000","last_unstale":"2023-03-09T14:45:55.926611+0000","last_undegraded":"2023-03-09T14:45:55.926611+0000","last_fullsized":"2023-03-09T14:45:55.926611+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T20:17:55.938860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":80,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":622,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.786994+0000","last_change":"2023-03-09T14:44:19.107572+0000","last_active":"2023-03-09T14:45:44.786994+0000","last_peered":"2023-03-09T14:45:44.786994+0000","last_clean":"2023-03-09T14:45:44.786994+0000","last_became_active":"2023-03-09T14:44:19.106632+0000","last_became_peered":"2023-03-09T14:44:19.106632+0000","last_unstale":"2023-03-09T14:45:44.786994+0000","last_undegraded":"2023-03-09T14:45:44.786994+0000","last_fullsized":"2023-03-09T14:45:44.786994+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T19:04:01.379375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.857041+0000","last_change":"2023-03-09T14:44:23.494002+0000","last_active":"2023-03-09T14:45:14.857041+0000","last_peered":"2023-03-09T14:45:14.857041+0000","last_clean":"2023-03-09T14:45:14.857041+0000","last_became_active":"2023-03-09T14:44:23.493628+0000","last_became_peered":"2023-03-09T14:44:23.493628+0000","last_unstale":"2023-03-09T14:45:14.857041+0000","last_undegraded":"2023-03-09T14:45:14.857041+0000","last_fullsized":"2023-03-09T14:45:14.857041+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:30:47.970556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"87'3","reported_seq":36,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:15.964036+0000","last_change":"2023-03-09T14:45:15.964036+0000","last_active":"2023-03-09T14:45:15.964036+0000","last_peered":"2023-03-09T14:45:15.964036+0000","last_clean":"2023-03-09T14:45:15.964036+0000","last_became_active":"2023-03-09T14:45:15.865998+0000","last_became_peered":"2023-03-09T14:45:15.865998+0000","last_unstale":"2023-03-09T14:45:15.964036+0000","last_undegraded":"2023-03-09T14:45:15.964036+0000","last_fullsized":"2023-03-09T14:45:15.964036+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:16:58.213684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.786013+0000","last_change":"2023-03-09T14:44:19.111303+0000","last_active":"2023-03-09T14:45:44.786013+0000","last_peered":"2023-03-09T14:45:44.786013+0000","last_clean":"2023-03-09T14:45:44.786013+0000","last_became_active":"2023-03-09T14:44:19.111026+0000","last_became_peered":"2023-03-09T14:44:19.111026+0000","last_unstale":"2023-03-09T14:45:44.786013+0000","last_undegraded":"2023-03-09T14:45:44.786013+0000","last_fullsized":"2023-03-09T14:45:44.786013+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T02:26:32.496084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.859059+0000","last_change":"2023-03-09T14:44:23.493272+0000","last_active":"2023-03-09T14:45:14.859059+0000","last_peered":"2023-03-09T14:45:14.859059+0000","last_clean":"2023-03-09T14:45:14.859059+0000","last_became_active":"2023-03-09T14:44:23.492785+0000","last_became_peered":"2023-03-09T14:44:23.492785+0000","last_unstale":"2023-03-09T14:45:14.859059+0000","last_undegraded":"2023-03-09T14:45:14.859059+0000","last_fullsized":"2023-03-09T14:45:14.859059+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T20:18:39.758938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"87'2","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.861264+0000","last_change":"2023-03-09T14:44:18.658179+0000","last_active":"2023-03-09T14:45:14.861264+0000","last_peered":"2023-03-09T14:45:14.861264+0000","last_clean":"2023-03-09T14:45:14.861264+0000","last_became_active":"2023-03-09T14:44:18.657851+0000","last_became_peered":"2023-03-09T14:44:18.657851+0000","last_unstale":"2023-03-09T14:45:14.861264+0000","last_undegraded":"2023-03-09T14:45:14.861264+0000","last_fullsized":"2023-03-09T14:45:14.861264+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:52:49.291615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.782372+0000","last_change":"2023-03-09T14:44:19.104176+0000","last_active":"2023-03-09T14:45:44.782372+0000","last_peered":"2023-03-09T14:45:44.782372+0000","last_clean":"2023-03-09T14:45:44.782372+0000","last_became_active":"2023-03-09T14:44:19.103816+0000","last_became_peered":"2023-03-09T14:44:19.103816+0000","last_unstale":"2023-03-09T14:45:44.782372+0000","last_undegraded":"2023-03-09T14:45:44.782372+0000","last_fullsized":"2023-03-09T14:45:44.782372+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T02:06:41.876169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.864893+0000","last_change":"2023-03-09T14:44:23.490876+0000","last_active":"2023-03-09T14:45:14.864893+0000","last_peered":"2023-03-09T14:45:14.864893+0000","last_clean":"2023-03-09T14:45:14.864893+0000","last_became_active":"2023-03-09T14:44:23.490461+0000","last_became_peered":"2023-03-09T14:44:23.490461+0000","last_unstale":"2023-03-09T14:45:14.864893+0000","last_undegraded":"2023-03-09T14:45:14.864893+0000","last_fullsized":"2023-03-09T14:45:14.864893+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T16:28:40.456613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"87'1","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.852373+0000","last_change":"2023-03-09T14:44:18.662513+0000","last_active":"2023-03-09T14:45:14.852373+0000","last_peered":"2023-03-09T14:45:14.852373+0000","last_clean":"2023-03-09T14:45:14.852373+0000","last_became_active":"2023-03-09T14:44:18.661907+0000","last_became_peered":"2023-03-09T14:44:18.661907+0000","last_unstale":"2023-03-09T14:45:14.852373+0000","last_undegraded":"2023-03-09T14:45:14.852373+0000","last_fullsized":"2023-03-09T14:45:14.852373+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:45:17.373213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"89'11","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.779797+0000","last_change":"2023-03-09T14:44:19.115297+0000","last_active":"2023-03-09T14:45:44.779797+0000","last_peered":"2023-03-09T14:45:44.779797+0000","last_clean":"2023-03-09T14:45:44.779797+0000","last_became_active":"2023-03-09T14:44:19.114973+0000","last_became_peered":"2023-03-09T14:44:19.114973+0000","last_unstale":"2023-03-09T14:45:44.779797+0000","last_undegraded":"2023-03-09T14:45:44.779797+0000","last_fullsized":"2023-03-09T14:45:44.779797+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:11:22.226608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.864500+0000","last_change":"2023-03-09T14:44:23.482772+0000","last_active":"2023-03-09T14:45:14.864500+0000","last_peered":"2023-03-09T14:45:14.864500+0000","last_clean":"2023-03-09T14:45:14.864500+0000","last_became_active":"2023-03-09T14:44:23.481750+0000","last_became_peered":"2023-03-09T14:44:23.481750+0000","last_unstale":"2023-03-09T14:45:14.864500+0000","last_undegraded":"2023-03-09T14:45:14.864500+0000","last_fullsized":"2023-03-09T14:45:14.864500+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T19:26:03.737713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"87'3","reported_seq":119,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:58.929566+0000","last_change":"2023-03-09T14:45:15.870269+0000","last_active":"2023-03-09T14:45:58.929566+0000","last_peered":"2023-03-09T14:45:58.929566+0000","last_clean":"2023-03-09T14:45:58.929566+0000","last_became_active":"2023-03-09T14:45:15.862610+0000","last_became_peered":"2023-03-09T14:45:15.862610+0000","last_unstale":"2023-03-09T14:45:58.929566+0000","last_undegraded":"2023-03-09T14:45:58.929566+0000","last_fullsized":"2023-03-09T14:45:58.929566+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T02:37:23.157022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":81,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,9],"acting":[10,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"89'10","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.799241+0000","last_change":"2023-03-09T14:44:19.100914+0000","last_active":"2023-03-09T14:45:44.799241+0000","last_peered":"2023-03-09T14:45:44.799241+0000","last_clean":"2023-03-09T14:45:44.799241+0000","last_became_active":"2023-03-09T14:44:19.100451+0000","last_became_peered":"2023-03-09T14:44:19.100451+0000","last_unstale":"2023-03-09T14:45:44.799241+0000","last_undegraded":"2023-03-09T14:45:44.799241+0000","last_fullsized":"2023-03-09T14:45:44.799241+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:18:41.852697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.860006+0000","last_change":"2023-03-09T14:44:23.490129+0000","last_active":"2023-03-09T14:45:14.860006+0000","last_peered":"2023-03-09T14:45:14.860006+0000","last_clean":"2023-03-09T14:45:14.860006+0000","last_became_active":"2023-03-09T14:44:23.489696+0000","last_became_peered":"2023-03-09T14:44:23.489696+0000","last_unstale":"2023-03-09T14:45:14.860006+0000","last_undegraded":"2023-03-09T14:45:14.860006+0000","last_fullsized":"2023-03-09T14:45:14.860006+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T16:47:41.943904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"87'2","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:15.955780+0000","last_change":"2023-03-09T14:45:15.955780+0000","last_active":"2023-03-09T14:45:15.955780+0000","last_peered":"2023-03-09T14:45:15.955780+0000","last_clean":"2023-03-09T14:45:15.955780+0000","last_became_active":"2023-03-09T14:45:15.862567+0000","last_became_peered":"2023-03-09T14:45:15.862567+0000","last_unstale":"2023-03-09T14:45:15.955780+0000","last_undegraded":"2023-03-09T14:45:15.955780+0000","last_fullsized":"2023-03-09T14:45:15.955780+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:08:41.748638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.786794+0000","last_change":"2023-03-09T14:44:19.119125+0000","last_active":"2023-03-09T14:45:44.786794+0000","last_peered":"2023-03-09T14:45:44.786794+0000","last_clean":"2023-03-09T14:45:44.786794+0000","last_became_active":"2023-03-09T14:44:19.118677+0000","last_became_peered":"2023-03-09T14:44:19.118677+0000","last_unstale":"2023-03-09T14:45:44.786794+0000","last_undegraded":"2023-03-09T14:45:44.786794+0000","last_fullsized":"2023-03-09T14:45:44.786794+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:06:28.740355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.859670+0000","last_change":"2023-03-09T14:44:23.484752+0000","last_active":"2023-03-09T14:45:14.859670+0000","last_peered":"2023-03-09T14:45:14.859670+0000","last_clean":"2023-03-09T14:45:14.859670+0000","last_became_active":"2023-03-09T14:44:23.484045+0000","last_became_peered":"2023-03-09T14:44:23.484045+0000","last_unstale":"2023-03-09T14:45:14.859670+0000","last_undegraded":"2023-03-09T14:45:14.859670+0000","last_fullsized":"2023-03-09T14:45:14.859670+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:58:39.704017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"87'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.853129+0000","last_change":"2023-03-09T14:44:18.665122+0000","last_active":"2023-03-09T14:45:14.853129+0000","last_peered":"2023-03-09T14:45:14.853129+0000","last_clean":"2023-03-09T14:45:14.853129+0000","last_became_active":"2023-03-09T14:44:18.664786+0000","last_became_peered":"2023-03-09T14:44:18.664786+0000","last_unstale":"2023-03-09T14:45:14.853129+0000","last_undegraded":"2023-03-09T14:45:14.853129+0000","last_fullsized":"2023-03-09T14:45:14.853129+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T20:29:20.919293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":656,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"89'11","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.797201+0000","last_change":"2023-03-09T14:44:19.108693+0000","last_active":"2023-03-09T14:45:44.797201+0000","last_peered":"2023-03-09T14:45:44.797201+0000","last_clean":"2023-03-09T14:45:44.797201+0000","last_became_active":"2023-03-09T14:44:19.108370+0000","last_became_peered":"2023-03-09T14:44:19.108370+0000","last_unstale":"2023-03-09T14:45:44.797201+0000","last_undegraded":"2023-03-09T14:45:44.797201+0000","last_fullsized":"2023-03-09T14:45:44.797201+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T16:30:58.726080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:15.862206+0000","last_change":"2023-03-09T14:45:15.862206+0000","last_active":"2023-03-09T14:45:15.862206+0000","last_peered":"2023-03-09T14:45:15.862206+0000","last_clean":"2023-03-09T14:45:15.862206+0000","last_became_active":"2023-03-09T14:45:15.861775+0000","last_became_peered":"2023-03-09T14:45:15.861775+0000","last_unstale":"2023-03-09T14:45:15.862206+0000","last_undegraded":"2023-03-09T14:45:15.862206+0000","last_fullsized":"2023-03-09T14:45:15.862206+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:31:21.810745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.14","version":"89'35","reported_seq":225,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:59.931365+0000","last_change":"2023-03-09T14:44:18.661904+0000","last_active":"2023-03-09T14:45:59.931365+0000","last_peered":"2023-03-09T14:45:59.931365+0000","last_clean":"2023-03-09T14:45:59.931365+0000","last_became_active":"2023-03-09T14:44:18.661597+0000","last_became_peered":"2023-03-09T14:44:18.661597+0000","last_unstale":"2023-03-09T14:45:59.931365+0000","last_undegraded":"2023-03-09T14:45:59.931365+0000","last_fullsized":"2023-03-09T14:45:59.931365+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T14:57:04.978132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61521,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":172,"num_read_kb":147,"num_write":36,"num_write_kb":76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.786319+0000","last_change":"2023-03-09T14:44:19.111671+0000","last_active":"2023-03-09T14:45:44.786319+0000","last_peered":"2023-03-09T14:45:44.786319+0000","last_clean":"2023-03-09T14:45:44.786319+0000","last_became_active":"2023-03-09T14:44:19.111172+0000","last_became_peered":"2023-03-09T14:44:19.111172+0000","last_unstale":"2023-03-09T14:45:44.786319+0000","last_undegraded":"2023-03-09T14:45:44.786319+0000","last_fullsized":"2023-03-09T14:45:44.786319+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T16:27:23.921853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.853428+0000","last_change":"2023-03-09T14:44:23.486338+0000","last_active":"2023-03-09T14:45:14.853428+0000","last_peered":"2023-03-09T14:45:14.853428+0000","last_clean":"2023-03-09T14:45:14.853428+0000","last_became_active":"2023-03-09T14:44:23.485729+0000","last_became_peered":"2023-03-09T14:44:23.485729+0000","last_unstale":"2023-03-09T14:45:14.853428+0000","last_undegraded":"2023-03-09T14:45:14.853428+0000","last_fullsized":"2023-03-09T14:45:14.853428+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:28:23.872633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.857067+0000","last_change":"2023-03-09T14:44:18.664942+0000","last_active":"2023-03-09T14:45:14.857067+0000","last_peered":"2023-03-09T14:45:14.857067+0000","last_clean":"2023-03-09T14:45:14.857067+0000","last_became_active":"2023-03-09T14:44:18.664007+0000","last_became_peered":"2023-03-09T14:44:18.664007+0000","last_unstale":"2023-03-09T14:45:14.857067+0000","last_undegraded":"2023-03-09T14:45:14.857067+0000","last_fullsized":"2023-03-09T14:45:14.857067+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:36:21.329388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.782333+0000","last_change":"2023-03-09T14:44:19.110140+0000","last_active":"2023-03-09T14:45:44.782333+0000","last_peered":"2023-03-09T14:45:44.782333+0000","last_clean":"2023-03-09T14:45:44.782333+0000","last_became_active":"2023-03-09T14:44:19.109811+0000","last_became_peered":"2023-03-09T14:44:19.109811+0000","last_unstale":"2023-03-09T14:45:44.782333+0000","last_undegraded":"2023-03-09T14:45:44.782333+0000","last_fullsized":"2023-03-09T14:45:44.782333+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T14:56:14.638662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.858304+0000","last_change":"2023-03-09T14:44:23.491318+0000","last_active":"2023-03-09T14:45:14.858304+0000","last_peered":"2023-03-09T14:45:14.858304+0000","last_clean":"2023-03-09T14:45:14.858304+0000","last_became_active":"2023-03-09T14:44:23.490915+0000","last_became_peered":"2023-03-09T14:44:23.490915+0000","last_unstale":"2023-03-09T14:45:14.858304+0000","last_undegraded":"2023-03-09T14:45:14.858304+0000","last_fullsized":"2023-03-09T14:45:14.858304+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:32:06.877573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"87'3","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.852806+0000","last_change":"2023-03-09T14:44:18.657323+0000","last_active":"2023-03-09T14:45:14.852806+0000","last_peered":"2023-03-09T14:45:14.852806+0000","last_clean":"2023-03-09T14:45:14.852806+0000","last_became_active":"2023-03-09T14:44:18.657017+0000","last_became_peered":"2023-03-09T14:44:18.657017+0000","last_unstale":"2023-03-09T14:45:14.852806+0000","last_undegraded":"2023-03-09T14:45:14.852806+0000","last_fullsized":"2023-03-09T14:45:14.852806+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:43:11.562723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"89'15","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.791050+0000","last_change":"2023-03-09T14:44:19.112194+0000","last_active":"2023-03-09T14:45:44.791050+0000","last_peered":"2023-03-09T14:45:44.791050+0000","last_clean":"2023-03-09T14:45:44.791050+0000","last_became_active":"2023-03-09T14:44:19.111924+0000","last_became_peered":"2023-03-09T14:44:19.111924+0000","last_unstale":"2023-03-09T14:45:44.791050+0000","last_undegraded":"2023-03-09T14:45:44.791050+0000","last_fullsized":"2023-03-09T14:45:44.791050+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T19:54:22.669848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.860733+0000","last_change":"2023-03-09T14:44:23.493993+0000","last_active":"2023-03-09T14:45:14.860733+0000","last_peered":"2023-03-09T14:45:14.860733+0000","last_clean":"2023-03-09T14:45:14.860733+0000","last_became_active":"2023-03-09T14:44:23.493535+0000","last_became_peered":"2023-03-09T14:44:23.493535+0000","last_unstale":"2023-03-09T14:45:14.860733+0000","last_undegraded":"2023-03-09T14:45:14.860733+0000","last_fullsized":"2023-03-09T14:45:14.860733+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T01:18:23.669790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'5","reported_seq":104,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:54.664806+0000","last_change":"2023-03-09T14:44:18.659914+0000","last_active":"2023-03-09T14:45:54.664806+0000","last_peered":"2023-03-09T14:45:54.664806+0000","last_clean":"2023-03-09T14:45:54.664806+0000","last_became_active":"2023-03-09T14:44:18.658835+0000","last_became_peered":"2023-03-09T14:44:18.658835+0000","last_unstale":"2023-03-09T14:45:54.664806+0000","last_undegraded":"2023-03-09T14:45:54.664806+0000","last_fullsized":"2023-03-09T14:45:54.664806+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T14:55:05.413260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1119,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":82,"num_read_kb":5,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.787772+0000","last_change":"2023-03-09T14:44:19.111058+0000","last_active":"2023-03-09T14:45:44.787772+0000","last_peered":"2023-03-09T14:45:44.787772+0000","last_clean":"2023-03-09T14:45:44.787772+0000","last_became_active":"2023-03-09T14:44:19.110760+0000","last_became_peered":"2023-03-09T14:44:19.110760+0000","last_unstale":"2023-03-09T14:45:44.787772+0000","last_undegraded":"2023-03-09T14:45:44.787772+0000","last_fullsized":"2023-03-09T14:45:44.787772+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:25:13.199416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.861493+0000","last_change":"2023-03-09T14:44:23.484710+0000","last_active":"2023-03-09T14:45:14.861493+0000","last_peered":"2023-03-09T14:45:14.861493+0000","last_clean":"2023-03-09T14:45:14.861493+0000","last_became_active":"2023-03-09T14:44:23.483928+0000","last_became_peered":"2023-03-09T14:44:23.483928+0000","last_unstale":"2023-03-09T14:45:14.861493+0000","last_undegraded":"2023-03-09T14:45:14.861493+0000","last_fullsized":"2023-03-09T14:45:14.861493+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T14:51:32.124111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"87'15","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.860284+0000","last_change":"2023-03-09T14:44:18.663692+0000","last_active":"2023-03-09T14:45:14.860284+0000","last_peered":"2023-03-09T14:45:14.860284+0000","last_clean":"2023-03-09T14:45:14.860284+0000","last_became_active":"2023-03-09T14:44:18.663400+0000","last_became_peered":"2023-03-09T14:44:18.663400+0000","last_unstale":"2023-03-09T14:45:14.860284+0000","last_undegraded":"2023-03-09T14:45:14.860284+0000","last_fullsized":"2023-03-09T14:45:14.860284+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:32:31.634315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.797691+0000","last_change":"2023-03-09T14:44:19.110819+0000","last_active":"2023-03-09T14:45:44.797691+0000","last_peered":"2023-03-09T14:45:44.797691+0000","last_clean":"2023-03-09T14:45:44.797691+0000","last_became_active":"2023-03-09T14:44:19.110542+0000","last_became_peered":"2023-03-09T14:44:19.110542+0000","last_unstale":"2023-03-09T14:45:44.797691+0000","last_undegraded":"2023-03-09T14:45:44.797691+0000","last_fullsized":"2023-03-09T14:45:44.797691+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:25:38.382685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.852360+0000","last_change":"2023-03-09T14:44:23.484614+0000","last_active":"2023-03-09T14:45:14.852360+0000","last_peered":"2023-03-09T14:45:14.852360+0000","last_clean":"2023-03-09T14:45:14.852360+0000","last_became_active":"2023-03-09T14:44:23.484061+0000","last_became_peered":"2023-03-09T14:44:23.484061+0000","last_unstale":"2023-03-09T14:45:14.852360+0000","last_undegraded":"2023-03-09T14:45:14.852360+0000","last_fullsized":"2023-03-09T14:45:14.852360+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T19:31:42.889122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"87'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.861838+0000","last_change":"2023-03-09T14:44:18.656056+0000","last_active":"2023-03-09T14:45:14.861838+0000","last_peered":"2023-03-09T14:45:14.861838+0000","last_clean":"2023-03-09T14:45:14.861838+0000","last_became_active":"2023-03-09T14:44:18.655709+0000","last_became_peered":"2023-03-09T14:44:18.655709+0000","last_unstale":"2023-03-09T14:45:14.861838+0000","last_undegraded":"2023-03-09T14:45:14.861838+0000","last_fullsized":"2023-03-09T14:45:14.861838+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T19:09:11.447249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"89'5","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.790663+0000","last_change":"2023-03-09T14:44:19.110292+0000","last_active":"2023-03-09T14:45:44.790663+0000","last_peered":"2023-03-09T14:45:44.790663+0000","last_clean":"2023-03-09T14:45:44.790663+0000","last_became_active":"2023-03-09T14:44:19.110006+0000","last_became_peered":"2023-03-09T14:44:19.110006+0000","last_unstale":"2023-03-09T14:45:44.790663+0000","last_undegraded":"2023-03-09T14:45:44.790663+0000","last_fullsized":"2023-03-09T14:45:44.790663+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:36:40.401486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.861551+0000","last_change":"2023-03-09T14:44:23.484004+0000","last_active":"2023-03-09T14:45:14.861551+0000","last_peered":"2023-03-09T14:45:14.861551+0000","last_clean":"2023-03-09T14:45:14.861551+0000","last_became_active":"2023-03-09T14:44:23.483581+0000","last_became_peered":"2023-03-09T14:44:23.483581+0000","last_unstale":"2023-03-09T14:45:14.861551+0000","last_undegraded":"2023-03-09T14:45:14.861551+0000","last_fullsized":"2023-03-09T14:45:14.861551+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:24:25.939838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.860756+0000","last_change":"2023-03-09T14:44:18.661856+0000","last_active":"2023-03-09T14:45:14.860756+0000","last_peered":"2023-03-09T14:45:14.860756+0000","last_clean":"2023-03-09T14:45:14.860756+0000","last_became_active":"2023-03-09T14:44:18.661500+0000","last_became_peered":"2023-03-09T14:44:18.661500+0000","last_unstale":"2023-03-09T14:45:14.860756+0000","last_undegraded":"2023-03-09T14:45:14.860756+0000","last_fullsized":"2023-03-09T14:45:14.860756+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:27:12.181159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"89'4","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.783682+0000","last_change":"2023-03-09T14:44:19.102058+0000","last_active":"2023-03-09T14:45:44.783682+0000","last_peered":"2023-03-09T14:45:44.783682+0000","last_clean":"2023-03-09T14:45:44.783682+0000","last_became_active":"2023-03-09T14:44:19.101762+0000","last_became_peered":"2023-03-09T14:44:19.101762+0000","last_unstale":"2023-03-09T14:45:44.783682+0000","last_undegraded":"2023-03-09T14:45:44.783682+0000","last_fullsized":"2023-03-09T14:45:44.783682+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:39:42.544086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.862303+0000","last_change":"2023-03-09T14:44:23.490237+0000","last_active":"2023-03-09T14:45:14.862303+0000","last_peered":"2023-03-09T14:45:14.862303+0000","last_clean":"2023-03-09T14:45:14.862303+0000","last_became_active":"2023-03-09T14:44:23.489844+0000","last_became_peered":"2023-03-09T14:44:23.489844+0000","last_unstale":"2023-03-09T14:45:14.862303+0000","last_undegraded":"2023-03-09T14:45:14.862303+0000","last_fullsized":"2023-03-09T14:45:14.862303+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T14:50:09.358698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"87'1","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:15.865433+0000","last_change":"2023-03-09T14:45:15.864913+0000","last_active":"2023-03-09T14:45:15.865433+0000","last_peered":"2023-03-09T14:45:15.865433+0000","last_clean":"2023-03-09T14:45:15.865433+0000","last_became_active":"2023-03-09T14:45:15.864527+0000","last_became_peered":"2023-03-09T14:45:15.864527+0000","last_unstale":"2023-03-09T14:45:15.865433+0000","last_undegraded":"2023-03-09T14:45:15.865433+0000","last_fullsized":"2023-03-09T14:45:15.865433+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:02:19.489659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6,1],"acting":[11,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.790944+0000","last_change":"2023-03-09T14:44:19.117584+0000","last_active":"2023-03-09T14:45:44.790944+0000","last_peered":"2023-03-09T14:45:44.790944+0000","last_clean":"2023-03-09T14:45:44.790944+0000","last_became_active":"2023-03-09T14:44:19.117262+0000","last_became_peered":"2023-03-09T14:44:19.117262+0000","last_unstale":"2023-03-09T14:45:44.790944+0000","last_undegraded":"2023-03-09T14:45:44.790944+0000","last_fullsized":"2023-03-09T14:45:44.790944+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T19:19:38.080048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.861827+0000","last_change":"2023-03-09T14:44:23.485769+0000","last_active":"2023-03-09T14:45:14.861827+0000","last_peered":"2023-03-09T14:45:14.861827+0000","last_clean":"2023-03-09T14:45:14.861827+0000","last_became_active":"2023-03-09T14:44:23.485317+0000","last_became_peered":"2023-03-09T14:44:23.485317+0000","last_unstale":"2023-03-09T14:45:14.861827+0000","last_undegraded":"2023-03-09T14:45:14.861827+0000","last_fullsized":"2023-03-09T14:45:14.861827+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:58:35.140512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.786487+0000","last_change":"2023-03-09T14:44:19.112255+0000","last_active":"2023-03-09T14:45:44.786487+0000","last_peered":"2023-03-09T14:45:44.786487+0000","last_clean":"2023-03-09T14:45:44.786487+0000","last_became_active":"2023-03-09T14:44:19.111988+0000","last_became_peered":"2023-03-09T14:44:19.111988+0000","last_unstale":"2023-03-09T14:45:44.786487+0000","last_undegraded":"2023-03-09T14:45:44.786487+0000","last_fullsized":"2023-03-09T14:45:44.786487+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:55:50.423701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'5","reported_seq":103,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:57.669850+0000","last_change":"2023-03-09T14:44:18.665291+0000","last_active":"2023-03-09T14:45:57.669850+0000","last_peered":"2023-03-09T14:45:57.669850+0000","last_clean":"2023-03-09T14:45:57.669850+0000","last_became_active":"2023-03-09T14:44:18.664950+0000","last_became_peered":"2023-03-09T14:44:18.664950+0000","last_unstale":"2023-03-09T14:45:57.669850+0000","last_undegraded":"2023-03-09T14:45:57.669850+0000","last_fullsized":"2023-03-09T14:45:57.669850+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:40:58.619717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":80,"num_read_kb":80,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"87'1","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.853567+0000","last_change":"2023-03-09T14:44:23.480872+0000","last_active":"2023-03-09T14:45:14.853567+0000","last_peered":"2023-03-09T14:45:14.853567+0000","last_clean":"2023-03-09T14:45:14.853567+0000","last_became_active":"2023-03-09T14:44:23.480495+0000","last_became_peered":"2023-03-09T14:44:23.480495+0000","last_unstale":"2023-03-09T14:45:14.853567+0000","last_undegraded":"2023-03-09T14:45:14.853567+0000","last_fullsized":"2023-03-09T14:45:14.853567+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:19:01.393034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.797742+0000","last_change":"2023-03-09T14:44:19.102833+0000","last_active":"2023-03-09T14:45:44.797742+0000","last_peered":"2023-03-09T14:45:44.797742+0000","last_clean":"2023-03-09T14:45:44.797742+0000","last_became_active":"2023-03-09T14:44:19.102484+0000","last_became_peered":"2023-03-09T14:44:19.102484+0000","last_unstale":"2023-03-09T14:45:44.797742+0000","last_undegraded":"2023-03-09T14:45:44.797742+0000","last_fullsized":"2023-03-09T14:45:44.797742+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:14:50.002907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"87'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.859537+0000","last_change":"2023-03-09T14:44:18.661603+0000","last_active":"2023-03-09T14:45:14.859537+0000","last_peered":"2023-03-09T14:45:14.859537+0000","last_clean":"2023-03-09T14:45:14.859537+0000","last_became_active":"2023-03-09T14:44:18.661275+0000","last_became_peered":"2023-03-09T14:44:18.661275+0000","last_unstale":"2023-03-09T14:45:14.859537+0000","last_undegraded":"2023-03-09T14:45:14.859537+0000","last_fullsized":"2023-03-09T14:45:14.859537+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:20:31.873778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":378,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.852517+0000","last_change":"2023-03-09T14:43:19.317554+0000","last_active":"2023-03-09T14:45:14.852517+0000","last_peered":"2023-03-09T14:45:14.852517+0000","last_clean":"2023-03-09T14:45:14.852517+0000","last_became_active":"2023-03-09T14:43:19.316912+0000","last_became_peered":"2023-03-09T14:43:19.316912+0000","last_unstale":"2023-03-09T14:45:14.852517+0000","last_undegraded":"2023-03-09T14:45:14.852517+0000","last_fullsized":"2023-03-09T14:45:14.852517+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:40:23.197173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:40:23.197173+0000","last_clean_scrub_stamp":"2023-03-09T14:40:23.197173+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T20:21:17.143331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.852927+0000","last_change":"2023-03-09T14:44:23.487738+0000","last_active":"2023-03-09T14:45:14.852927+0000","last_peered":"2023-03-09T14:45:14.852927+0000","last_clean":"2023-03-09T14:45:14.852927+0000","last_became_active":"2023-03-09T14:44:23.487193+0000","last_became_peered":"2023-03-09T14:44:23.487193+0000","last_unstale":"2023-03-09T14:45:14.852927+0000","last_undegraded":"2023-03-09T14:45:14.852927+0000","last_fullsized":"2023-03-09T14:45:14.852927+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:48:08.789071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"89'5","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.784152+0000","last_change":"2023-03-09T14:44:19.112576+0000","last_active":"2023-03-09T14:45:44.784152+0000","last_peered":"2023-03-09T14:45:44.784152+0000","last_clean":"2023-03-09T14:45:44.784152+0000","last_became_active":"2023-03-09T14:44:19.112231+0000","last_became_peered":"2023-03-09T14:44:19.112231+0000","last_unstale":"2023-03-09T14:45:44.784152+0000","last_undegraded":"2023-03-09T14:45:44.784152+0000","last_fullsized":"2023-03-09T14:45:44.784152+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:25:21.918088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"87'3","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.852608+0000","last_change":"2023-03-09T14:44:18.655478+0000","last_active":"2023-03-09T14:45:14.852608+0000","last_peered":"2023-03-09T14:45:14.852608+0000","last_clean":"2023-03-09T14:45:14.852608+0000","last_became_active":"2023-03-09T14:44:18.655180+0000","last_became_peered":"2023-03-09T14:44:18.655180+0000","last_unstale":"2023-03-09T14:45:14.852608+0000","last_undegraded":"2023-03-09T14:45:14.852608+0000","last_fullsized":"2023-03-09T14:45:14.852608+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:42:47.926376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.856627+0000","last_change":"2023-03-09T14:44:23.487052+0000","last_active":"2023-03-09T14:45:14.856627+0000","last_peered":"2023-03-09T14:45:14.856627+0000","last_clean":"2023-03-09T14:45:14.856627+0000","last_became_active":"2023-03-09T14:44:23.486603+0000","last_became_peered":"2023-03-09T14:44:23.486603+0000","last_unstale":"2023-03-09T14:45:14.856627+0000","last_undegraded":"2023-03-09T14:45:14.856627+0000","last_fullsized":"2023-03-09T14:45:14.856627+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:28:25.375052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"87'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.861722+0000","last_change":"2023-03-09T14:44:18.666400+0000","last_active":"2023-03-09T14:45:14.861722+0000","last_peered":"2023-03-09T14:45:14.861722+0000","last_clean":"2023-03-09T14:45:14.861722+0000","last_became_active":"2023-03-09T14:44:18.666045+0000","last_became_peered":"2023-03-09T14:44:18.666045+0000","last_unstale":"2023-03-09T14:45:14.861722+0000","last_undegraded":"2023-03-09T14:45:14.861722+0000","last_fullsized":"2023-03-09T14:45:14.861722+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:30:48.640124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.784135+0000","last_change":"2023-03-09T14:44:19.103657+0000","last_active":"2023-03-09T14:45:44.784135+0000","last_peered":"2023-03-09T14:45:44.784135+0000","last_clean":"2023-03-09T14:45:44.784135+0000","last_became_active":"2023-03-09T14:44:19.103364+0000","last_became_peered":"2023-03-09T14:44:19.103364+0000","last_unstale":"2023-03-09T14:45:44.784135+0000","last_undegraded":"2023-03-09T14:45:44.784135+0000","last_fullsized":"2023-03-09T14:45:44.784135+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:23:23.738554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.864043+0000","last_change":"2023-03-09T14:44:23.483487+0000","last_active":"2023-03-09T14:45:14.864043+0000","last_peered":"2023-03-09T14:45:14.864043+0000","last_clean":"2023-03-09T14:45:14.864043+0000","last_became_active":"2023-03-09T14:44:23.483050+0000","last_became_peered":"2023-03-09T14:44:23.483050+0000","last_unstale":"2023-03-09T14:45:14.864043+0000","last_undegraded":"2023-03-09T14:45:14.864043+0000","last_fullsized":"2023-03-09T14:45:14.864043+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:38:34.129355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'7","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:48.585960+0000","last_change":"2023-03-09T14:44:18.661944+0000","last_active":"2023-03-09T14:45:48.585960+0000","last_peered":"2023-03-09T14:45:48.585960+0000","last_clean":"2023-03-09T14:45:48.585960+0000","last_became_active":"2023-03-09T14:44:18.661575+0000","last_became_peered":"2023-03-09T14:44:18.661575+0000","last_unstale":"2023-03-09T14:45:48.585960+0000","last_undegraded":"2023-03-09T14:45:48.585960+0000","last_fullsized":"2023-03-09T14:45:48.585960+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T01:12:53.637866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"89'5","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.787402+0000","last_change":"2023-03-09T14:44:19.103807+0000","last_active":"2023-03-09T14:45:44.787402+0000","last_peered":"2023-03-09T14:45:44.787402+0000","last_clean":"2023-03-09T14:45:44.787402+0000","last_became_active":"2023-03-09T14:44:19.103480+0000","last_became_peered":"2023-03-09T14:44:19.103480+0000","last_unstale":"2023-03-09T14:45:44.787402+0000","last_undegraded":"2023-03-09T14:45:44.787402+0000","last_fullsized":"2023-03-09T14:45:44.787402+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:29:54.629265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.856235+0000","last_change":"2023-03-09T14:44:23.486789+0000","last_active":"2023-03-09T14:45:14.856235+0000","last_peered":"2023-03-09T14:45:14.856235+0000","last_clean":"2023-03-09T14:45:14.856235+0000","last_became_active":"2023-03-09T14:44:23.486367+0000","last_became_peered":"2023-03-09T14:44:23.486367+0000","last_unstale":"2023-03-09T14:45:14.856235+0000","last_undegraded":"2023-03-09T14:45:14.856235+0000","last_fullsized":"2023-03-09T14:45:14.856235+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:13:13.081388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"87'4","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.860080+0000","last_change":"2023-03-09T14:44:18.659492+0000","last_active":"2023-03-09T14:45:14.860080+0000","last_peered":"2023-03-09T14:45:14.860080+0000","last_clean":"2023-03-09T14:45:14.860080+0000","last_became_active":"2023-03-09T14:44:18.659187+0000","last_became_peered":"2023-03-09T14:44:18.659187+0000","last_unstale":"2023-03-09T14:45:14.860080+0000","last_undegraded":"2023-03-09T14:45:14.860080+0000","last_fullsized":"2023-03-09T14:45:14.860080+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:44:31.342785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.789313+0000","last_change":"2023-03-09T14:44:19.112841+0000","last_active":"2023-03-09T14:45:44.789313+0000","last_peered":"2023-03-09T14:45:44.789313+0000","last_clean":"2023-03-09T14:45:44.789313+0000","last_became_active":"2023-03-09T14:44:19.112515+0000","last_became_peered":"2023-03-09T14:44:19.112515+0000","last_unstale":"2023-03-09T14:45:44.789313+0000","last_undegraded":"2023-03-09T14:45:44.789313+0000","last_fullsized":"2023-03-09T14:45:44.789313+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T19:41:04.065529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.859850+0000","last_change":"2023-03-09T14:44:23.481992+0000","last_active":"2023-03-09T14:45:14.859850+0000","last_peered":"2023-03-09T14:45:14.859850+0000","last_clean":"2023-03-09T14:45:14.859850+0000","last_became_active":"2023-03-09T14:44:23.481508+0000","last_became_peered":"2023-03-09T14:44:23.481508+0000","last_unstale":"2023-03-09T14:45:14.859850+0000","last_undegraded":"2023-03-09T14:45:14.859850+0000","last_fullsized":"2023-03-09T14:45:14.859850+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:46:48.683402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"87'2","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.856199+0000","last_change":"2023-03-09T14:44:18.663133+0000","last_active":"2023-03-09T14:45:14.856199+0000","last_peered":"2023-03-09T14:45:14.856199+0000","last_clean":"2023-03-09T14:45:14.856199+0000","last_became_active":"2023-03-09T14:44:18.662809+0000","last_became_peered":"2023-03-09T14:44:18.662809+0000","last_unstale":"2023-03-09T14:45:14.856199+0000","last_undegraded":"2023-03-09T14:45:14.856199+0000","last_fullsized":"2023-03-09T14:45:14.856199+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:32:16.180642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"89'2","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.784561+0000","last_change":"2023-03-09T14:44:19.108566+0000","last_active":"2023-03-09T14:45:44.784561+0000","last_peered":"2023-03-09T14:45:44.784561+0000","last_clean":"2023-03-09T14:45:44.784561+0000","last_became_active":"2023-03-09T14:44:19.108217+0000","last_became_peered":"2023-03-09T14:44:19.108217+0000","last_unstale":"2023-03-09T14:45:44.784561+0000","last_undegraded":"2023-03-09T14:45:44.784561+0000","last_fullsized":"2023-03-09T14:45:44.784561+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:03:28.434395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.860862+0000","last_change":"2023-03-09T14:44:23.488640+0000","last_active":"2023-03-09T14:45:14.860862+0000","last_peered":"2023-03-09T14:45:14.860862+0000","last_clean":"2023-03-09T14:45:14.860862+0000","last_became_active":"2023-03-09T14:44:23.488178+0000","last_became_peered":"2023-03-09T14:44:23.488178+0000","last_unstale":"2023-03-09T14:45:14.860862+0000","last_undegraded":"2023-03-09T14:45:14.860862+0000","last_fullsized":"2023-03-09T14:45:14.860862+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:33:49.908974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"87'3","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.864673+0000","last_change":"2023-03-09T14:44:18.662925+0000","last_active":"2023-03-09T14:45:14.864673+0000","last_peered":"2023-03-09T14:45:14.864673+0000","last_clean":"2023-03-09T14:45:14.864673+0000","last_became_active":"2023-03-09T14:44:18.662354+0000","last_became_peered":"2023-03-09T14:44:18.662354+0000","last_unstale":"2023-03-09T14:45:14.864673+0000","last_undegraded":"2023-03-09T14:45:14.864673+0000","last_fullsized":"2023-03-09T14:45:14.864673+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T01:00:36.602938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.790399+0000","last_change":"2023-03-09T14:44:19.106465+0000","last_active":"2023-03-09T14:45:44.790399+0000","last_peered":"2023-03-09T14:45:44.790399+0000","last_clean":"2023-03-09T14:45:44.790399+0000","last_became_active":"2023-03-09T14:44:19.106001+0000","last_became_peered":"2023-03-09T14:44:19.106001+0000","last_unstale":"2023-03-09T14:45:44.790399+0000","last_undegraded":"2023-03-09T14:45:44.790399+0000","last_fullsized":"2023-03-09T14:45:44.790399+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:22:09.149291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"87'1","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:15.873389+0000","last_change":"2023-03-09T14:45:15.873389+0000","last_active":"2023-03-09T14:45:15.873389+0000","last_peered":"2023-03-09T14:45:15.873389+0000","last_clean":"2023-03-09T14:45:15.873389+0000","last_became_active":"2023-03-09T14:45:15.863408+0000","last_became_peered":"2023-03-09T14:45:15.863408+0000","last_unstale":"2023-03-09T14:45:15.873389+0000","last_undegraded":"2023-03-09T14:45:15.873389+0000","last_fullsized":"2023-03-09T14:45:15.873389+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T16:33:12.565135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":123,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,7,8],"acting":[4,10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"87'1","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.856858+0000","last_change":"2023-03-09T14:44:18.664744+0000","last_active":"2023-03-09T14:45:14.856858+0000","last_peered":"2023-03-09T14:45:14.856858+0000","last_clean":"2023-03-09T14:45:14.856858+0000","last_became_active":"2023-03-09T14:44:18.664443+0000","last_became_peered":"2023-03-09T14:44:18.664443+0000","last_unstale":"2023-03-09T14:45:14.856858+0000","last_undegraded":"2023-03-09T14:45:14.856858+0000","last_fullsized":"2023-03-09T14:45:14.856858+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:11:13.836270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.790126+0000","last_change":"2023-03-09T14:44:19.115102+0000","last_active":"2023-03-09T14:45:44.790126+0000","last_peered":"2023-03-09T14:45:44.790126+0000","last_clean":"2023-03-09T14:45:44.790126+0000","last_became_active":"2023-03-09T14:44:19.114565+0000","last_became_peered":"2023-03-09T14:44:19.114565+0000","last_unstale":"2023-03-09T14:45:44.790126+0000","last_undegraded":"2023-03-09T14:45:44.790126+0000","last_fullsized":"2023-03-09T14:45:44.790126+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:39:11.930231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.859836+0000","last_change":"2023-03-09T14:44:23.482500+0000","last_active":"2023-03-09T14:45:14.859836+0000","last_peered":"2023-03-09T14:45:14.859836+0000","last_clean":"2023-03-09T14:45:14.859836+0000","last_became_active":"2023-03-09T14:44:23.482217+0000","last_became_peered":"2023-03-09T14:44:23.482217+0000","last_unstale":"2023-03-09T14:45:14.859836+0000","last_undegraded":"2023-03-09T14:45:14.859836+0000","last_fullsized":"2023-03-09T14:45:14.859836+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T14:52:44.275037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"87'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.860725+0000","last_change":"2023-03-09T14:44:18.663824+0000","last_active":"2023-03-09T14:45:14.860725+0000","last_peered":"2023-03-09T14:45:14.860725+0000","last_clean":"2023-03-09T14:45:14.860725+0000","last_became_active":"2023-03-09T14:44:18.663379+0000","last_became_peered":"2023-03-09T14:44:18.663379+0000","last_unstale":"2023-03-09T14:45:14.860725+0000","last_undegraded":"2023-03-09T14:45:14.860725+0000","last_fullsized":"2023-03-09T14:45:14.860725+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:03:04.031061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"89'5","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.798601+0000","last_change":"2023-03-09T14:44:19.105765+0000","last_active":"2023-03-09T14:45:44.798601+0000","last_peered":"2023-03-09T14:45:44.798601+0000","last_clean":"2023-03-09T14:45:44.798601+0000","last_became_active":"2023-03-09T14:44:19.105430+0000","last_became_peered":"2023-03-09T14:44:19.105430+0000","last_unstale":"2023-03-09T14:45:44.798601+0000","last_undegraded":"2023-03-09T14:45:44.798601+0000","last_fullsized":"2023-03-09T14:45:44.798601+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T20:31:40.106798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.856901+0000","last_change":"2023-03-09T14:44:23.482079+0000","last_active":"2023-03-09T14:45:14.856901+0000","last_peered":"2023-03-09T14:45:14.856901+0000","last_clean":"2023-03-09T14:45:14.856901+0000","last_became_active":"2023-03-09T14:44:23.481620+0000","last_became_peered":"2023-03-09T14:44:23.481620+0000","last_unstale":"2023-03-09T14:45:14.856901+0000","last_undegraded":"2023-03-09T14:45:14.856901+0000","last_fullsized":"2023-03-09T14:45:14.856901+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:47:53.589111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"87'3","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:15.869980+0000","last_change":"2023-03-09T14:45:15.869485+0000","last_active":"2023-03-09T14:45:15.869980+0000","last_peered":"2023-03-09T14:45:15.869980+0000","last_clean":"2023-03-09T14:45:15.869980+0000","last_became_active":"2023-03-09T14:45:15.861428+0000","last_became_peered":"2023-03-09T14:45:15.861428+0000","last_unstale":"2023-03-09T14:45:15.869980+0000","last_undegraded":"2023-03-09T14:45:15.869980+0000","last_fullsized":"2023-03-09T14:45:15.869980+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:43:30.309431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":566,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.799201+0000","last_change":"2023-03-09T14:44:19.105947+0000","last_active":"2023-03-09T14:45:44.799201+0000","last_peered":"2023-03-09T14:45:44.799201+0000","last_clean":"2023-03-09T14:45:44.799201+0000","last_became_active":"2023-03-09T14:44:19.105669+0000","last_became_peered":"2023-03-09T14:44:19.105669+0000","last_unstale":"2023-03-09T14:45:44.799201+0000","last_undegraded":"2023-03-09T14:45:44.799201+0000","last_fullsized":"2023-03-09T14:45:44.799201+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:29:58.615481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.863833+0000","last_change":"2023-03-09T14:44:23.483237+0000","last_active":"2023-03-09T14:45:14.863833+0000","last_peered":"2023-03-09T14:45:14.863833+0000","last_clean":"2023-03-09T14:45:14.863833+0000","last_became_active":"2023-03-09T14:44:23.482831+0000","last_became_peered":"2023-03-09T14:44:23.482831+0000","last_unstale":"2023-03-09T14:45:14.863833+0000","last_undegraded":"2023-03-09T14:45:14.863833+0000","last_fullsized":"2023-03-09T14:45:14.863833+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:36:40.775581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"82'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:15.867430+0000","last_change":"2023-03-09T14:45:15.867049+0000","last_active":"2023-03-09T14:45:15.867430+0000","last_peered":"2023-03-09T14:45:15.867430+0000","last_clean":"2023-03-09T14:45:15.867430+0000","last_became_active":"2023-03-09T14:45:15.863218+0000","last_became_peered":"2023-03-09T14:45:15.863218+0000","last_unstale":"2023-03-09T14:45:15.867430+0000","last_undegraded":"2023-03-09T14:45:15.867430+0000","last_fullsized":"2023-03-09T14:45:15.867430+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:38:21.800533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.c","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.787235+0000","last_change":"2023-03-09T14:44:19.107146+0000","last_active":"2023-03-09T14:45:44.787235+0000","last_peered":"2023-03-09T14:45:44.787235+0000","last_clean":"2023-03-09T14:45:44.787235+0000","last_became_active":"2023-03-09T14:44:19.106834+0000","last_became_peered":"2023-03-09T14:44:19.106834+0000","last_unstale":"2023-03-09T14:45:44.787235+0000","last_undegraded":"2023-03-09T14:45:44.787235+0000","last_fullsized":"2023-03-09T14:45:44.787235+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T20:15:20.142449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.852185+0000","last_change":"2023-03-09T14:44:23.482811+0000","last_active":"2023-03-09T14:45:14.852185+0000","last_peered":"2023-03-09T14:45:14.852185+0000","last_clean":"2023-03-09T14:45:14.852185+0000","last_became_active":"2023-03-09T14:44:23.482371+0000","last_became_peered":"2023-03-09T14:44:23.482371+0000","last_unstale":"2023-03-09T14:45:14.852185+0000","last_undegraded":"2023-03-09T14:45:14.852185+0000","last_fullsized":"2023-03-09T14:45:14.852185+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T01:33:14.442810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"82'2","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.861583+0000","last_change":"2023-03-09T14:44:18.661569+0000","last_active":"2023-03-09T14:45:14.861583+0000","last_peered":"2023-03-09T14:45:14.861583+0000","last_clean":"2023-03-09T14:45:14.861583+0000","last_became_active":"2023-03-09T14:44:18.661266+0000","last_became_peered":"2023-03-09T14:44:18.661266+0000","last_unstale":"2023-03-09T14:45:14.861583+0000","last_undegraded":"2023-03-09T14:45:14.861583+0000","last_fullsized":"2023-03-09T14:45:14.861583+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:47:43.066180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.786924+0000","last_change":"2023-03-09T14:44:19.106611+0000","last_active":"2023-03-09T14:45:44.786924+0000","last_peered":"2023-03-09T14:45:44.786924+0000","last_clean":"2023-03-09T14:45:44.786924+0000","last_became_active":"2023-03-09T14:44:19.106324+0000","last_became_peered":"2023-03-09T14:44:19.106324+0000","last_unstale":"2023-03-09T14:45:44.786924+0000","last_undegraded":"2023-03-09T14:45:44.786924+0000","last_fullsized":"2023-03-09T14:45:44.786924+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:50:46.561502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.787174+0000","last_change":"2023-03-09T14:44:19.106292+0000","last_active":"2023-03-09T14:45:44.787174+0000","last_peered":"2023-03-09T14:45:44.787174+0000","last_clean":"2023-03-09T14:45:44.787174+0000","last_became_active":"2023-03-09T14:44:19.105607+0000","last_became_peered":"2023-03-09T14:44:19.105607+0000","last_unstale":"2023-03-09T14:45:44.787174+0000","last_undegraded":"2023-03-09T14:45:44.787174+0000","last_fullsized":"2023-03-09T14:45:44.787174+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T19:11:28.125433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"89'12","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.786929+0000","last_change":"2023-03-09T14:44:19.103423+0000","last_active":"2023-03-09T14:45:44.786929+0000","last_peered":"2023-03-09T14:45:44.786929+0000","last_clean":"2023-03-09T14:45:44.786929+0000","last_became_active":"2023-03-09T14:44:19.102918+0000","last_became_peered":"2023-03-09T14:44:19.102918+0000","last_unstale":"2023-03-09T14:45:44.786929+0000","last_undegraded":"2023-03-09T14:45:44.786929+0000","last_fullsized":"2023-03-09T14:45:44.786929+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:47:52.921642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.790566+0000","last_change":"2023-03-09T14:44:19.113511+0000","last_active":"2023-03-09T14:45:44.790566+0000","last_peered":"2023-03-09T14:45:44.790566+0000","last_clean":"2023-03-09T14:45:44.790566+0000","last_became_active":"2023-03-09T14:44:19.113224+0000","last_became_peered":"2023-03-09T14:44:19.113224+0000","last_unstale":"2023-03-09T14:45:44.790566+0000","last_undegraded":"2023-03-09T14:45:44.790566+0000","last_fullsized":"2023-03-09T14:45:44.790566+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T02:03:28.069616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.787373+0000","last_change":"2023-03-09T14:44:19.116174+0000","last_active":"2023-03-09T14:45:44.787373+0000","last_peered":"2023-03-09T14:45:44.787373+0000","last_clean":"2023-03-09T14:45:44.787373+0000","last_became_active":"2023-03-09T14:44:19.115813+0000","last_became_peered":"2023-03-09T14:44:19.115813+0000","last_unstale":"2023-03-09T14:45:44.787373+0000","last_undegraded":"2023-03-09T14:45:44.787373+0000","last_fullsized":"2023-03-09T14:45:44.787373+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:28:49.573771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"89'4","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.790565+0000","last_change":"2023-03-09T14:44:19.115287+0000","last_active":"2023-03-09T14:45:44.790565+0000","last_peered":"2023-03-09T14:45:44.790565+0000","last_clean":"2023-03-09T14:45:44.790565+0000","last_became_active":"2023-03-09T14:44:19.113458+0000","last_became_peered":"2023-03-09T14:44:19.113458+0000","last_unstale":"2023-03-09T14:45:44.790565+0000","last_undegraded":"2023-03-09T14:45:44.790565+0000","last_fullsized":"2023-03-09T14:45:44.790565+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:43:26.583237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.787054+0000","last_change":"2023-03-09T14:44:19.107308+0000","last_active":"2023-03-09T14:45:44.787054+0000","last_peered":"2023-03-09T14:45:44.787054+0000","last_clean":"2023-03-09T14:45:44.787054+0000","last_became_active":"2023-03-09T14:44:19.107005+0000","last_became_peered":"2023-03-09T14:44:19.107005+0000","last_unstale":"2023-03-09T14:45:44.787054+0000","last_undegraded":"2023-03-09T14:45:44.787054+0000","last_fullsized":"2023-03-09T14:45:44.787054+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:48:34.886664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"89'15","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.800402+0000","last_change":"2023-03-09T14:44:19.098454+0000","last_active":"2023-03-09T14:45:44.800402+0000","last_peered":"2023-03-09T14:45:44.800402+0000","last_clean":"2023-03-09T14:45:44.800402+0000","last_became_active":"2023-03-09T14:44:19.098146+0000","last_became_peered":"2023-03-09T14:44:19.098146+0000","last_unstale":"2023-03-09T14:45:44.800402+0000","last_undegraded":"2023-03-09T14:45:44.800402+0000","last_fullsized":"2023-03-09T14:45:44.800402+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:11:40.475674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"89'2","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.790577+0000","last_change":"2023-03-09T14:44:19.095947+0000","last_active":"2023-03-09T14:45:44.790577+0000","last_peered":"2023-03-09T14:45:44.790577+0000","last_clean":"2023-03-09T14:45:44.790577+0000","last_became_active":"2023-03-09T14:44:19.095640+0000","last_became_peered":"2023-03-09T14:44:19.095640+0000","last_unstale":"2023-03-09T14:45:44.790577+0000","last_undegraded":"2023-03-09T14:45:44.790577+0000","last_fullsized":"2023-03-09T14:45:44.790577+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T02:04:35.384016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.781900+0000","last_change":"2023-03-09T14:44:19.113927+0000","last_active":"2023-03-09T14:45:44.781900+0000","last_peered":"2023-03-09T14:45:44.781900+0000","last_clean":"2023-03-09T14:45:44.781900+0000","last_became_active":"2023-03-09T14:44:19.113163+0000","last_became_peered":"2023-03-09T14:44:19.113163+0000","last_unstale":"2023-03-09T14:45:44.781900+0000","last_undegraded":"2023-03-09T14:45:44.781900+0000","last_fullsized":"2023-03-09T14:45:44.781900+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T01:35:35.801317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.787768+0000","last_change":"2023-03-09T14:44:19.098415+0000","last_active":"2023-03-09T14:45:44.787768+0000","last_peered":"2023-03-09T14:45:44.787768+0000","last_clean":"2023-03-09T14:45:44.787768+0000","last_became_active":"2023-03-09T14:44:19.097600+0000","last_became_peered":"2023-03-09T14:44:19.097600+0000","last_unstale":"2023-03-09T14:45:44.787768+0000","last_undegraded":"2023-03-09T14:45:44.787768+0000","last_fullsized":"2023-03-09T14:45:44.787768+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:15:08.552402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"89'10","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.795312+0000","last_change":"2023-03-09T14:44:19.106386+0000","last_active":"2023-03-09T14:45:44.795312+0000","last_peered":"2023-03-09T14:45:44.795312+0000","last_clean":"2023-03-09T14:45:44.795312+0000","last_became_active":"2023-03-09T14:44:19.106112+0000","last_became_peered":"2023-03-09T14:44:19.106112+0000","last_unstale":"2023-03-09T14:45:44.795312+0000","last_undegraded":"2023-03-09T14:45:44.795312+0000","last_fullsized":"2023-03-09T14:45:44.795312+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T20:27:50.992168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"89'11","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.788971+0000","last_change":"2023-03-09T14:44:19.114534+0000","last_active":"2023-03-09T14:45:44.788971+0000","last_peered":"2023-03-09T14:45:44.788971+0000","last_clean":"2023-03-09T14:45:44.788971+0000","last_became_active":"2023-03-09T14:44:19.114226+0000","last_became_peered":"2023-03-09T14:44:19.114226+0000","last_unstale":"2023-03-09T14:45:44.788971+0000","last_undegraded":"2023-03-09T14:45:44.788971+0000","last_fullsized":"2023-03-09T14:45:44.788971+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:22:38.453116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.790943+0000","last_change":"2023-03-09T14:44:19.099195+0000","last_active":"2023-03-09T14:45:44.790943+0000","last_peered":"2023-03-09T14:45:44.790943+0000","last_clean":"2023-03-09T14:45:44.790943+0000","last_became_active":"2023-03-09T14:44:19.098892+0000","last_became_peered":"2023-03-09T14:44:19.098892+0000","last_unstale":"2023-03-09T14:45:44.790943+0000","last_undegraded":"2023-03-09T14:45:44.790943+0000","last_fullsized":"2023-03-09T14:45:44.790943+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:29:24.858200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":246,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":123,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":65614,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":506,"num_read_kb":326,"num_write":163,"num_write_kb":116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":1914,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":192872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":143,"ondisk_log_size":143,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":78,"seq":335007449120,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6236,"kb_used_data":980,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":93738916,"statfs":{"total":95995035648,"available":95988649984,"internally_reserved":0,"allocated":1003520,"data_stored":490897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:45:43 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.58299999999999996,"15min":0.58499999999999996},"min":{"1min":0.433,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.57899999999999996,"15min":0.58399999999999996},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.76000000000000001,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.70499999999999996}]},{"osd":1,"last update":"Thu Mar 9 14:45:43 2023","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.59999999999999998,"15min":0.58799999999999997},"min":{"1min":0.47399999999999998,"5min":0.378,"15min":0.378},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.40200000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.79600000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.76800000000000002}]},{"osd":2,"last update":"Thu Mar 9 14:45:43 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.504,"15min":0.505},"min":{"1min":0.38,"5min":0.313,"15min":0.313},"max":{"1min":0.76000000000000001,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.51300000000000001,"15min":0.503},"min":{"1min":0.33500000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.79300000000000004,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.45300000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78200000000000003}]},{"osd":4,"last update":"Thu Mar 9 14:45:43 2023","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.66900000000000004,"15min":0.65700000000000003},"min":{"1min":0.50700000000000001,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.91200000000000003,"5min":1.113,"15min":1.113},"last":0.89400000000000002},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.64800000000000002,"15min":0.64400000000000002},"min":{"1min":0.56299999999999994,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.85099999999999998,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.79400000000000004}]},{"osd":5,"last update":"Thu Mar 9 14:45:43 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.55700000000000005,"15min":0.54700000000000004},"min":{"1min":0.42399999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.63,"15min":0.61399999999999999},"min":{"1min":0.41599999999999998,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.94099999999999995,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.60499999999999998}]},{"osd":6,"last update":"Thu Mar 9 14:45:43 2023","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.67700000000000005,"15min":0.66900000000000004},"min":{"1min":0.45500000000000002,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.7829999999999999,"5min":1.7829999999999999,"15min":1.7829999999999999},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.752,"5min":0.70299999999999996,"15min":0.69499999999999995},"min":{"1min":0.53200000000000003,"5min":0.497,"15min":0.497},"max":{"1min":1.423,"5min":1.423,"15min":1.423},"last":0.63200000000000001}]},{"osd":7,"last update":"Thu Mar 9 14:45:43 2023","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.73499999999999999,"15min":0.71999999999999997},"min":{"1min":0.56299999999999994,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.0189999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.73599999999999999,"15min":0.73099999999999998},"min":{"1min":0.61299999999999999,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":0.93999999999999995,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.81599999999999995}]},{"osd":8,"last update":"Thu Mar 9 14:45:21 2023","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.91800000000000004}]},{"osd":9,"last update":"Thu Mar 9 14:45:43 2023","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.79400000000000004,"15min":0.78300000000000003},"min":{"1min":0.58699999999999997,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.548,"5min":1.548,"15min":1.548},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.872,"5min":0.78100000000000003,"15min":0.76600000000000001},"min":{"1min":0.54600000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.996,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.93999999999999995}]},{"osd":10,"last update":"Thu Mar 9 14:45:43 2023","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.755,"15min":0.749},"min":{"1min":0.64300000000000002,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":0.97499999999999998,"5min":1.01,"15min":1.01},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.79800000000000004,"15min":0.79400000000000004},"min":{"1min":0.66600000000000004,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.012,"5min":1.431,"15min":1.431},"last":0.753}]}]},{"osd":10,"up_from":70,"seq":300647710755,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7224,"kb_used_data":1648,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93737928,"statfs":{"total":95995035648,"available":95987638272,"internally_reserved":0,"allocated":1687552,"data_stored":1144230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:45:19 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.502}]},{"osd":1,"last update":"Thu Mar 9 14:45:19 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.46700000000000003,"15min":0.46999999999999997},"min":{"1min":0.30499999999999999,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.56699999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.47799999999999998,"15min":0.48199999999999998},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.61099999999999999,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.47499999999999998}]},{"osd":2,"last update":"Thu Mar 9 14:45:19 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.63600000000000001,"15min":0.63900000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.753,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.751},{"interface":"front","average":{"1min":0.628,"5min":0.623,"15min":0.622},"min":{"1min":0.46400000000000002,"5min":0.441,"15min":0.441},"max":{"1min":0.82199999999999995,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.69599999999999995}]},{"osd":3,"last update":"Thu Mar 9 14:45:19 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.58399999999999996,"15min":0.58899999999999997},"min":{"1min":0.38300000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.74399999999999999,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.64200000000000002,"15min":0.64800000000000002},"min":{"1min":0.46600000000000003,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.76100000000000001,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.63500000000000001}]},{"osd":4,"last update":"Thu Mar 9 14:45:19 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.59099999999999997,"15min":0.59299999999999997},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.78200000000000003,"5min":1.1719999999999999,"15min":1.1719999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.54500000000000004,"15min":0.54900000000000004},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.77500000000000002,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.62}]},{"osd":5,"last update":"Thu Mar 9 14:45:19 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.504,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.92700000000000005,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.90300000000000002,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.73699999999999999}]},{"osd":6,"last update":"Thu Mar 9 14:45:19 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.68300000000000005,"15min":0.68899999999999995},"min":{"1min":0.48799999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.86499999999999999,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.71099999999999997,"15min":0.71599999999999997},"min":{"1min":0.499,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.82699999999999996,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.72299999999999998}]},{"osd":7,"last update":"Thu Mar 9 14:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.327,"5min":1.327,"15min":1.327},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.048,"5min":1.048,"15min":1.048},"last":0.53800000000000003}]},{"osd":8,"last update":"Thu Mar 9 14:45:19 2023","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.76000000000000001,"15min":0.76200000000000001},"min":{"1min":0.54900000000000004,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":0.91000000000000003,"5min":1.246,"15min":1.246},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.77300000000000002,"15min":0.78200000000000003},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.95499999999999996,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.871}]},{"osd":9,"last update":"Thu Mar 9 14:45:19 2023","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.77600000000000002,"15min":0.78400000000000003},"min":{"1min":0.54200000000000004,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.88100000000000001,"5min":1.298,"15min":1.298},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.77700000000000002,"15min":0.79200000000000004},"min":{"1min":0.47499999999999998,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.89200000000000002,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":0.71199999999999997}]},{"osd":11,"last update":"Thu Mar 9 14:45:42 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.755,"15min":0.753},"min":{"1min":0.68799999999999994,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.628,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.88300000000000001}]}]},{"osd":9,"up_from":65,"seq":279172874280,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6376,"kb_used_data":992,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738776,"statfs":{"total":95995035648,"available":95988506624,"internally_reserved":0,"allocated":1015808,"data_stored":491063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:45:00 2023","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.46200000000000002,"15min":0.47399999999999998},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.66300000000000003,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.49399999999999999,"15min":0.50700000000000001},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.63700000000000001,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.36699999999999999}]},{"osd":1,"last update":"Thu Mar 9 14:45:00 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.626,"15min":0.63600000000000001},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.008,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.63700000000000001,"15min":0.65000000000000002},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.79900000000000004,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.56999999999999995}]},{"osd":2,"last update":"Thu Mar 9 14:45:00 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.64600000000000002,"15min":0.65800000000000003},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.66100000000000003,"15min":0.67400000000000004},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.88300000000000001,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.497}]},{"osd":3,"last update":"Thu Mar 9 14:45:00 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.56999999999999995,"15min":0.57999999999999996},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.59499999999999997,"15min":0.61499999999999999},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.85099999999999998,"5min":1.016,"15min":1.016},"last":0.38}]},{"osd":4,"last update":"Thu Mar 9 14:45:00 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.67900000000000005,"15min":0.69699999999999995},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.80400000000000005,"5min":1.04,"15min":1.04},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.621,"5min":0.71699999999999997,"15min":0.73299999999999998},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.93899999999999995,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.58099999999999996}]},{"osd":5,"last update":"Thu Mar 9 14:45:00 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.73299999999999998,"15min":0.745},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.98599999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.74099999999999999,"15min":0.754},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.0369999999999999,"5min":1.069,"15min":1.069},"last":0.52300000000000002}]},{"osd":6,"last update":"Thu Mar 9 14:43:55 2023","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.623}]},{"osd":7,"last update":"Thu Mar 9 14:45:00 2023","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.79800000000000004,"15min":0.82599999999999996},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.0229999999999999,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.80900000000000005,"15min":0.83099999999999996},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.99299999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.71499999999999997}]},{"osd":8,"last update":"Thu Mar 9 14:45:00 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.80200000000000005,"15min":0.81599999999999995},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.032,"5min":1.133,"15min":1.133},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.81999999999999995,"15min":0.83799999999999997},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.081,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.48199999999999998}]},{"osd":10,"last update":"Thu Mar 9 14:45:21 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.77800000000000002,"15min":0.79200000000000004},"min":{"1min":0.49199999999999999,"5min":0.435,"15min":0.435},"max":{"1min":0.97299999999999998,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.77900000000000003,"15min":0.79200000000000004},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.056,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.72999999999999998}]},{"osd":11,"last update":"Thu Mar 9 14:45:23 2023","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":2.6659999999999999,"5min":2.6659999999999999,"15min":2.6659999999999999},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.60699999999999998}]}]},{"osd":8,"up_from":57,"seq":244813135916,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6560,"kb_used_data":984,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738592,"statfs":{"total":95995035648,"available":95988318208,"internally_reserved":0,"allocated":1007616,"data_stored":490086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.51900000000000002,"15min":0.51500000000000001},"min":{"1min":0.46600000000000003,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.73199999999999998,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.54000000000000004,"15min":0.54800000000000004},"min":{"1min":0.36299999999999999,"5min":0.31,"15min":0.31},"max":{"1min":0.65700000000000003,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.55100000000000005}]},{"osd":1,"last update":"Thu Mar 9 14:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.57399999999999995,"15min":0.54800000000000004},"min":{"1min":0.434,"5min":0.313,"15min":0.313},"max":{"1min":0.88400000000000001,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.57799999999999996,"15min":0.56399999999999995},"min":{"1min":0.41699999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.79200000000000004,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.71599999999999997}]},{"osd":2,"last update":"Thu Mar 9 14:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.438,"15min":0.42199999999999999},"min":{"1min":0.314,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.45000000000000001,"15min":0.433},"min":{"1min":0.39700000000000002,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.51700000000000002}]},{"osd":3,"last update":"Thu Mar 9 14:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.47599999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.69299999999999995,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.621},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.628,"15min":0.61399999999999999},"min":{"1min":0.51200000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.67700000000000005,"5min":3.9300000000000002,"15min":3.9300000000000002},"last":0.60299999999999998}]},{"osd":4,"last update":"Thu Mar 9 14:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.58399999999999996,"15min":0.56299999999999994},"min":{"1min":0.35999999999999999,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.78400000000000003,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.61599999999999999,"15min":0.59299999999999997},"min":{"1min":0.48899999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.85599999999999998,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.68200000000000005}]},{"osd":5,"last update":"Thu Mar 9 14:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.54900000000000004,"15min":0.53700000000000003},"min":{"1min":0.47799999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.70299999999999996,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.629,"5min":0.56100000000000005,"15min":0.54200000000000004},"min":{"1min":0.40500000000000003,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.84599999999999997}]},{"osd":6,"last update":"Thu Mar 9 14:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.70599999999999996,"15min":0.68200000000000005},"min":{"1min":0.53200000000000003,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.89500000000000002,"5min":3.895,"15min":3.895},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.66000000000000003,"15min":0.65200000000000002},"min":{"1min":0.51100000000000001,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.82199999999999995,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.81599999999999995}]},{"osd":7,"last update":"Thu Mar 9 14:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.67700000000000005,"15min":0.66700000000000004},"min":{"1min":0.51400000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.84899999999999998,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.68999999999999995,"15min":0.67000000000000004},"min":{"1min":0.501,"5min":0.44,"15min":0.44},"max":{"1min":0.90500000000000003,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.93799999999999994}]},{"osd":9,"last update":"Thu Mar 9 14:44:57 2023","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.67600000000000005,"15min":0.66800000000000004},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.036,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.754,"5min":0.68600000000000005,"15min":0.67500000000000004},"min":{"1min":0.52900000000000003,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.119,"5min":1.119,"15min":1.119},"last":0.747}]},{"osd":10,"last update":"Thu Mar 9 14:45:18 2023","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.70099999999999996,"15min":0.68000000000000005},"min":{"1min":0.59499999999999997,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.099,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.95899999999999996},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.70899999999999996,"15min":0.68700000000000006},"min":{"1min":0.60599999999999998,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.79600000000000004}]},{"osd":11,"last update":"Thu Mar 9 14:45:23 2023","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.91400000000000003},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.72899999999999998}]}]},{"osd":7,"up_from":50,"seq":214748364849,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7212,"kb_used_data":1572,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93737940,"statfs":{"total":95995035648,"available":95987650560,"internally_reserved":0,"allocated":1609728,"data_stored":1087032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:44:13 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.52100000000000002,"15min":0.51700000000000002},"min":{"1min":0.316,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.69999999999999996,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.375},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.51900000000000002,"15min":0.50900000000000001},"min":{"1min":0.32800000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.90100000000000002,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.33900000000000002}]},{"osd":1,"last update":"Thu Mar 9 14:45:14 2023","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.40600000000000003,"15min":0.39400000000000002},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.54500000000000004,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.40600000000000003,"15min":0.39100000000000001},"min":{"1min":0.29799999999999999,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.56999999999999995,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.45600000000000002}]},{"osd":2,"last update":"Thu Mar 9 14:45:14 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.54600000000000004,"15min":0.53400000000000003},"min":{"1min":0.33500000000000002,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.754,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.58399999999999996,"15min":0.56999999999999995},"min":{"1min":0.44600000000000001,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.80700000000000005,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.71699999999999997}]},{"osd":3,"last update":"Thu Mar 9 14:45:14 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.55700000000000005,"15min":0.53800000000000003},"min":{"1min":0.40200000000000002,"5min":0.37,"15min":0.37},"max":{"1min":0.76000000000000001,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.57999999999999996,"15min":0.56399999999999995},"min":{"1min":0.44,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.76400000000000001,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.78000000000000003}]},{"osd":4,"last update":"Thu Mar 9 14:45:14 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48599999999999999,"15min":0.47199999999999998},"min":{"1min":0.34799999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.65100000000000002,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.47599999999999998,"15min":0.46000000000000002},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.68100000000000005,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.497}]},{"osd":5,"last update":"Thu Mar 9 14:45:14 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.58799999999999997,"15min":0.57399999999999995},"min":{"1min":0.40300000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.76200000000000001,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.623,"5min":0.59499999999999997,"15min":0.57799999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.80200000000000005,"5min":0.874,"15min":0.874},"last":0.63200000000000001}]},{"osd":6,"last update":"Thu Mar 9 14:45:14 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.63300000000000001,"15min":0.61699999999999999},"min":{"1min":0.50600000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.623,"15min":0.61299999999999999},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.81000000000000005,"5min":0.878,"15min":0.878},"last":0.60599999999999998}]},{"osd":8,"last update":"Thu Mar 9 14:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.66600000000000004,"15min":0.65900000000000003},"min":{"1min":0.57299999999999995,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.84799999999999998,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.65700000000000003,"15min":0.65000000000000002},"min":{"1min":0.53700000000000003,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.83699999999999997}]},{"osd":9,"last update":"Thu Mar 9 14:44:55 2023","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.69299999999999995,"15min":0.69199999999999995},"min":{"1min":0.52600000000000002,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":0.91400000000000003,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.81399999999999995},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.70199999999999996,"15min":0.70399999999999996},"min":{"1min":0.52000000000000002,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":0.88700000000000001,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.87}]},{"osd":10,"last update":"Thu Mar 9 14:45:20 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.66300000000000003,"15min":0.66600000000000004},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.83799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.70299999999999996,"15min":0.70099999999999996},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.81799999999999995,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.88700000000000001}]},{"osd":11,"last update":"Thu Mar 9 14:45:20 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.91000000000000003},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.82599999999999996}]}]},{"osd":6,"up_from":45,"seq":193273528374,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6552,"kb_used_data":976,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738600,"statfs":{"total":95995035648,"available":95988326400,"internally_reserved":0,"allocated":999424,"data_stored":486345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:45:55 2023","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.40300000000000002,"15min":0.40899999999999997},"min":{"1min":0.28000000000000003,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.503,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.49299999999999999,"15min":0.49099999999999999},"min":{"1min":0.35699999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.66300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.45700000000000002}]},{"osd":1,"last update":"Thu Mar 9 14:45:55 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.56999999999999995,"15min":0.59499999999999997},"min":{"1min":0.38,"5min":0.315,"15min":0.315},"max":{"1min":0.71699999999999997,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.42099999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.75700000000000001,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.53000000000000003}]},{"osd":2,"last update":"Thu Mar 9 14:45:55 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.59599999999999997,"15min":0.63200000000000001},"min":{"1min":0.38900000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.72799999999999998,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.622,"15min":0.63300000000000001},"min":{"1min":0.42399999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.79900000000000004,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.39200000000000002}]},{"osd":3,"last update":"Thu Mar 9 14:45:55 2023","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.44700000000000001,"15min":0.45700000000000002},"min":{"1min":0.32300000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.48899999999999999,"5min":0.745,"15min":0.745},"last":0.308},{"interface":"front","average":{"1min":0.501,"5min":0.55300000000000005,"15min":0.55400000000000005},"min":{"1min":0.36699999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.72899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.505}]},{"osd":4,"last update":"Thu Mar 9 14:45:55 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.64500000000000002,"15min":0.72299999999999998},"min":{"1min":0.41999999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.73299999999999998,"5min":2.0840000000000001,"15min":2.0840000000000001},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.747,"15min":0.871},"min":{"1min":0.42999999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.71599999999999997,"5min":4.1440000000000001,"15min":4.1440000000000001},"last":0.64000000000000001}]},{"osd":5,"last update":"Thu Mar 9 14:45:55 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.68300000000000005,"15min":0.69499999999999995},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.86099999999999999,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.64200000000000002,"15min":0.67600000000000005},"min":{"1min":0.47799999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.746,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.48599999999999999}]},{"osd":7,"last update":"Thu Mar 9 14:45:20 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.68100000000000005,"15min":0.70299999999999996},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.78600000000000003,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.71499999999999997,"15min":0.73099999999999998},"min":{"1min":0.5,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.87,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.71399999999999997}]},{"osd":8,"last update":"Thu Mar 9 14:45:36 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.75,"15min":0.78000000000000003},"min":{"1min":0.54600000000000004,"5min":0.497,"15min":0.497},"max":{"1min":0.84699999999999998,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.68799999999999994,"15min":0.71599999999999997},"min":{"1min":0.502,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.77700000000000002,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.436}]},{"osd":9,"last update":"Thu Mar 9 14:45:55 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.64300000000000002,"15min":0.67300000000000004},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.72099999999999997,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.70599999999999996,"15min":0.74199999999999999},"min":{"1min":0.49099999999999999,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.77400000000000002,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.56599999999999995}]},{"osd":10,"last update":"Thu Mar 9 14:45:20 2023","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.70599999999999996,"15min":0.70799999999999996},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.82099999999999995,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.73999999999999999,"15min":0.746},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":0.89800000000000002,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.745}]},{"osd":11,"last update":"Thu Mar 9 14:45:28 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.65200000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574922,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7640,"kb_used_data":1040,"kb_used_omap":0,"kb_used_meta":6592,"kb_avail":93737512,"statfs":{"total":95995035648,"available":95987212288,"internally_reserved":0,"allocated":1064960,"data_stored":548252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6750208},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:45:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.50900000000000001,"15min":0.49299999999999999},"min":{"1min":0.36599999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.63300000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.55100000000000005,"15min":0.497},"min":{"1min":0.42099999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.88700000000000001,"5min":2.0670000000000002,"15min":2.0670000000000002},"last":0.495}]},{"osd":2,"last update":"Thu Mar 9 14:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.59099999999999997,"15min":0.54400000000000004},"min":{"1min":0.60899999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.05,"5min":1.3839999999999999,"15min":1.3839999999999999},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.56000000000000005,"15min":0.51800000000000002},"min":{"1min":0.45300000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.96399999999999997,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.57999999999999996}]},{"osd":3,"last update":"Thu Mar 9 14:45:56 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56699999999999995,"15min":0.51500000000000001},"min":{"1min":0.38300000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.83999999999999997,"5min":2.206,"15min":2.206},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.59099999999999997,"15min":0.53500000000000003},"min":{"1min":0.43099999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.81699999999999995,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.54800000000000004}]},{"osd":4,"last update":"Thu Mar 9 14:45:18 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.441,"15min":0.39500000000000002},"min":{"1min":0.39900000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.434,"15min":0.39000000000000001},"min":{"1min":0.34100000000000003,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.82099999999999995}]},{"osd":5,"last update":"Thu Mar 9 14:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.623,"15min":0.56299999999999994},"min":{"1min":0.53200000000000003,"5min":0.373,"15min":0.373},"max":{"1min":1.081,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.61299999999999999,"15min":0.55800000000000005},"min":{"1min":0.53400000000000003,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.92900000000000005,"5min":1.363,"15min":1.363},"last":0.61799999999999999}]},{"osd":6,"last update":"Thu Mar 9 14:45:56 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.65200000000000002,"15min":0.65600000000000003},"min":{"1min":0.47199999999999998,"5min":0.371,"15min":0.371},"max":{"1min":0.91600000000000004,"5min":2.1930000000000001,"15min":2.1930000000000001},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.65800000000000003,"15min":0.66400000000000003},"min":{"1min":0.45300000000000001,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.90200000000000002,"5min":2.2519999999999998,"15min":2.2519999999999998},"last":0.65700000000000003}]},{"osd":7,"last update":"Thu Mar 9 14:45:11 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.55100000000000005,"15min":0.5},"min":{"1min":0.33800000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.55400000000000005,"15min":0.51600000000000001},"min":{"1min":0.35799999999999998,"5min":0.31,"15min":0.31},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.69199999999999995}]},{"osd":8,"last update":"Thu Mar 9 14:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.65600000000000003,"15min":0.622},"min":{"1min":0.57199999999999995,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.92600000000000005,"5min":1.3009999999999999,"15min":1.3009999999999999},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.71999999999999997,"15min":0.68400000000000005},"min":{"1min":0.497,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.1339999999999999,"5min":1.423,"15min":1.423},"last":0.80700000000000005}]},{"osd":9,"last update":"Thu Mar 9 14:45:58 2023","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.70899999999999996,"15min":0.67300000000000004},"min":{"1min":0.60099999999999998,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.70899999999999996,"15min":0.68000000000000005},"min":{"1min":0.65600000000000003,"5min":0.376,"15min":0.376},"max":{"1min":0.96299999999999997,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.79600000000000004}]},{"osd":10,"last update":"Thu Mar 9 14:45:19 2023","interfaces":[{"interface":"back","average":{"1min":0.86799999999999999,"5min":0.75600000000000001,"15min":0.73699999999999999},"min":{"1min":0.66800000000000004,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.77100000000000002,"15min":0.75800000000000001},"min":{"1min":0.68000000000000005,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.0109999999999999,"5min":1.022,"15min":1.022},"last":0.84999999999999998}]},{"osd":11,"last update":"Thu Mar 9 14:45:18 2023","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.1559999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.873,"5min":0.873,"15min":0.873},"min":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.879}]}]},{"osd":0,"up_from":8,"seq":34359738448,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7468,"kb_used_data":996,"kb_used_omap":0,"kb_used_meta":6464,"kb_avail":93737684,"statfs":{"total":95995035648,"available":95987388416,"internally_reserved":0,"allocated":1019904,"data_stored":487611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6619136},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Mar 9 14:45:11 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.55200000000000005,"15min":0.52200000000000002},"min":{"1min":0.375,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.67700000000000005,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.54300000000000004,"15min":0.52700000000000002},"min":{"1min":0.371,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.78900000000000003,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.71799999999999997}]},{"osd":2,"last update":"Thu Mar 9 14:45:31 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.56599999999999995,"15min":0.55600000000000005},"min":{"1min":0.40699999999999997,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.80800000000000005,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60699999999999998,"15min":0.58199999999999996},"min":{"1min":0.45200000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.79400000000000004,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.68200000000000005}]},{"osd":3,"last update":"Thu Mar 9 14:45:20 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.379}]},{"osd":4,"last update":"Thu Mar 9 14:45:11 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.59699999999999998,"15min":0.57099999999999995},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.78400000000000003,"5min":1.139,"15min":1.139},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.55800000000000005,"15min":0.54600000000000004},"min":{"1min":0.38700000000000001,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.88900000000000001,"5min":1.071,"15min":1.071},"last":0.60499999999999998}]},{"osd":5,"last update":"Thu Mar 9 14:45:38 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.60699999999999998,"15min":0.57099999999999995},"min":{"1min":0.504,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.66100000000000003,"15min":0.63100000000000001},"min":{"1min":0.53300000000000003,"5min":0.378,"15min":0.378},"max":{"1min":0.85099999999999998,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.873}]},{"osd":6,"last update":"Thu Mar 9 14:45:57 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.54400000000000004,"15min":0.50700000000000001},"min":{"1min":0.35199999999999998,"5min":0.309,"15min":0.309},"max":{"1min":0.88200000000000001,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.58499999999999996,"15min":0.55100000000000005},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.79500000000000004,"5min":1.159,"15min":1.159},"last":0.78000000000000003}]},{"osd":7,"last update":"Thu Mar 9 14:45:25 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.83899999999999997}]},{"osd":8,"last update":"Thu Mar 9 14:45:38 2023","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.72199999999999998,"15min":0.73299999999999998},"min":{"1min":0.58899999999999997,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.97399999999999998,"5min":1.038,"15min":1.038},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.755,"5min":0.79000000000000004,"15min":0.81799999999999995},"min":{"1min":0.56499999999999995,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.93899999999999995,"5min":1.242,"15min":1.242},"last":0.90100000000000002}]},{"osd":9,"last update":"Thu Mar 9 14:44:59 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.74299999999999999,"15min":0.75900000000000001},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.90600000000000003,"5min":1.105,"15min":1.105},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.82199999999999995,"15min":0.85399999999999998},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.88800000000000001,"5min":1.27,"15min":1.27},"last":0.629}]},{"osd":10,"last update":"Thu Mar 9 14:45:20 2023","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.79000000000000004,"15min":0.79800000000000004},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.93500000000000005,"5min":1.151,"15min":1.151},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.70999999999999996,"15min":0.71199999999999997},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.65100000000000002}]},{"osd":11,"last update":"Thu Mar 9 14:45:38 2023","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.73899999999999999,"15min":0.73299999999999998},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.71899999999999997,"15min":0.71799999999999997},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.70399999999999996}]}]},{"osd":2,"up_from":18,"seq":77309411399,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7708,"kb_used_data":1044,"kb_used_omap":0,"kb_used_meta":6656,"kb_avail":93737444,"statfs":{"total":95995035648,"available":95987142656,"internally_reserved":0,"allocated":1069056,"data_stored":543544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6815744},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:45:32 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.52900000000000003,"15min":0.5},"min":{"1min":0.40500000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.48399999999999999,"15min":0.46999999999999997},"min":{"1min":0.32400000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.51200000000000001}]},{"osd":1,"last update":"Thu Mar 9 14:45:32 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.56899999999999995,"15min":0.51000000000000001},"min":{"1min":0.41299999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.83599999999999997,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.58399999999999996,"15min":0.51800000000000002},"min":{"1min":0.44600000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.85699999999999998,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.81200000000000006}]},{"osd":3,"last update":"Thu Mar 9 14:45:52 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.57899999999999996,"15min":0.54300000000000004},"min":{"1min":0.36599999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.81999999999999995,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.63},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.55200000000000005,"15min":0.53600000000000003},"min":{"1min":0.39800000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.70799999999999996,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.66200000000000003}]},{"osd":4,"last update":"Thu Mar 9 14:45:21 2023","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.83499999999999996}]},{"osd":5,"last update":"Thu Mar 9 14:45:38 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.48999999999999999,"15min":0.46600000000000003},"min":{"1min":0.32100000000000001,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.77300000000000002,"5min":0.871,"15min":0.871},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.46300000000000002,"15min":0.44400000000000001},"min":{"1min":0.33600000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.747,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.64600000000000002}]},{"osd":6,"last update":"Thu Mar 9 14:45:52 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.59199999999999997,"15min":0.57899999999999996},"min":{"1min":0.41099999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.79600000000000004,"5min":1.002,"15min":1.002},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.64700000000000002,"15min":0.621},"min":{"1min":0.53600000000000003,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.879,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.755}]},{"osd":7,"last update":"Thu Mar 9 14:45:22 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.87},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.88300000000000001}]},{"osd":8,"last update":"Thu Mar 9 14:45:38 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.65700000000000003,"15min":0.65200000000000002},"min":{"1min":0.42199999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.90400000000000003,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.92400000000000004},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64500000000000002,"15min":0.65100000000000002},"min":{"1min":0.45800000000000002,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.94099999999999995,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.73899999999999999}]},{"osd":9,"last update":"Thu Mar 9 14:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.72399999999999998,"15min":0.72999999999999998},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.0109999999999999,"5min":1.21,"15min":1.21},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.67100000000000004,"15min":0.68000000000000005},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.93200000000000005,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.71999999999999997}]},{"osd":10,"last update":"Thu Mar 9 14:45:18 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68200000000000005,"15min":0.68100000000000005},"min":{"1min":0.55500000000000005,"5min":0.376,"15min":0.376},"max":{"1min":0.90600000000000003,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.73999999999999999,"15min":0.72899999999999998},"min":{"1min":0.624,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.091,"5min":1.306,"15min":1.306},"last":0.90700000000000003}]},{"osd":11,"last update":"Thu Mar 9 14:45:18 2023","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.95899999999999996},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.149,"5min":1.149,"15min":1.149},"last":0.94399999999999995}]}]},{"osd":3,"up_from":25,"seq":107374182466,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6816,"kb_used_data":984,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738336,"statfs":{"total":95995035648,"available":95988056064,"internally_reserved":0,"allocated":1007616,"data_stored":486449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:45:48 2023","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.377,"15min":0.33200000000000002},"min":{"1min":0.27700000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.498,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":2.669},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.39300000000000002,"15min":0.34799999999999998},"min":{"1min":0.27900000000000003,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.57899999999999996,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":3.1110000000000002}]},{"osd":1,"last update":"Thu Mar 9 14:45:48 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.53500000000000003,"15min":0.50600000000000001},"min":{"1min":0.312,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.77700000000000002,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57199999999999995,"15min":0.503},"min":{"1min":0.41399999999999998,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.67400000000000004,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":6.2329999999999997}]},{"osd":2,"last update":"Thu Mar 9 14:45:48 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.57599999999999996,"15min":0.52200000000000002},"min":{"1min":0.40500000000000003,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.70699999999999996,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":1.361},{"interface":"front","average":{"1min":0.621,"5min":0.58899999999999997,"15min":0.53900000000000003},"min":{"1min":0.40899999999999997,"5min":0.312,"15min":0.312},"max":{"1min":1.776,"5min":1.776,"15min":1.776},"last":1.2350000000000001}]},{"osd":4,"last update":"Thu Mar 9 14:45:10 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.58099999999999996,"15min":0.54200000000000004},"min":{"1min":0.45200000000000001,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.75900000000000001,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":3.5019999999999998},{"interface":"front","average":{"1min":0.622,"5min":0.60099999999999998,"15min":0.55000000000000004},"min":{"1min":0.497,"5min":0.373,"15min":0.373},"max":{"1min":0.84599999999999997,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":2.4449999999999998}]},{"osd":5,"last update":"Thu Mar 9 14:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.58499999999999996,"15min":0.54900000000000004},"min":{"1min":0.438,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.861,"5min":1.861,"15min":1.861},"last":1.341},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.60699999999999998,"15min":0.56699999999999995},"min":{"1min":0.46200000000000002,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.68600000000000005,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":7.2220000000000004}]},{"osd":6,"last update":"Thu Mar 9 14:45:54 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.55600000000000005,"15min":0.53400000000000003},"min":{"1min":0.36299999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.72199999999999998,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":7.9909999999999997},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.57399999999999995,"15min":0.54400000000000004},"min":{"1min":0.34499999999999997,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.81000000000000005,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":3.8759999999999999}]},{"osd":7,"last update":"Thu Mar 9 14:45:17 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.69399999999999995,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":1.5580000000000001},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.68000000000000005,"15min":0.66700000000000004},"min":{"1min":0.46500000000000002,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.80400000000000005,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":9.3710000000000004}]},{"osd":8,"last update":"Thu Mar 9 14:43:35 2023","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.749}]},{"osd":9,"last update":"Thu Mar 9 14:45:54 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.73099999999999998,"15min":0.77600000000000002},"min":{"1min":0.434,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.75700000000000001,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":10.67},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.73999999999999999,"15min":0.78800000000000003},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.80700000000000005,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":4.4400000000000004}]},{"osd":10,"last update":"Thu Mar 9 14:45:25 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.69899999999999995,"15min":0.70799999999999996},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":0.76400000000000001,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":1.667},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.79200000000000004,"15min":0.80500000000000005},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.83199999999999996,"5min":1.155,"15min":1.155},"last":11.667999999999999}]},{"osd":11,"last update":"Thu Mar 9 14:45:19 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":12.622},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":4.1820000000000004}]}]},{"osd":4,"up_from":32,"seq":137438953534,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6700,"kb_used_data":996,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738452,"statfs":{"total":95995035648,"available":95988174848,"internally_reserved":0,"allocated":1019904,"data_stored":492624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:44:09 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.53100000000000003,"15min":0.495},"min":{"1min":0.39000000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.84099999999999997,"5min":1.8320000000000001,"15min":1.8320000000000001},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.628,"5min":0.52900000000000003,"15min":0.495},"min":{"1min":0.42499999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":1.044,"5min":1.478,"15min":1.478},"last":0.86899999999999999}]},{"osd":1,"last update":"Thu Mar 9 14:45:14 2023","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.376,"15min":0.34799999999999998},"min":{"1min":0.311,"5min":0.253,"15min":0.253},"max":{"1min":0.53100000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.39600000000000002,"15min":0.36799999999999999},"min":{"1min":0.316,"5min":0.253,"15min":0.253},"max":{"1min":0.625,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.48099999999999998}]},{"osd":2,"last update":"Thu Mar 9 14:45:14 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.54100000000000004,"15min":0.53000000000000003},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.087,"5min":1.087,"15min":1.087},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.57199999999999995,"15min":0.53900000000000003},"min":{"1min":0.41499999999999998,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.80100000000000005}]},{"osd":3,"last update":"Thu Mar 9 14:45:14 2023","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":0.69899999999999995,"15min":0.57099999999999995},"min":{"1min":0.47699999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":9.9420000000000002,"5min":9.9420000000000002,"15min":9.9420000000000002},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.59699999999999998,"15min":0.54100000000000004},"min":{"1min":0.45200000000000001,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":2.6110000000000002,"5min":2.6110000000000002,"15min":2.6110000000000002},"last":0.77800000000000002}]},{"osd":5,"last update":"Thu Mar 9 14:45:33 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.59399999999999997,"15min":0.56299999999999994},"min":{"1min":0.47099999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.755,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.629,"5min":0.60999999999999999,"15min":0.57099999999999995},"min":{"1min":0.495,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.85999999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.61699999999999999}]},{"osd":6,"last update":"Thu Mar 9 14:45:54 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.66400000000000003,"15min":0.68700000000000006},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.77000000000000002,"5min":2.4609999999999999,"15min":2.4609999999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.72599999999999998,"15min":0.69199999999999995},"min":{"1min":0.48899999999999999,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.91400000000000003,"5min":4.6879999999999997,"15min":4.6879999999999997},"last":0.91400000000000003}]},{"osd":7,"last update":"Thu Mar 9 14:45:14 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.56999999999999995,"15min":0.53400000000000003},"min":{"1min":0.42299999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.872,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.58199999999999996,"15min":0.54000000000000004},"min":{"1min":0.437,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.77400000000000002,"5min":0.998,"15min":0.998},"last":0.65000000000000002}]},{"osd":8,"last update":"Thu Mar 9 14:45:36 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.71599999999999997,"15min":0.69099999999999995},"min":{"1min":0.40200000000000002,"5min":0.377,"15min":0.377},"max":{"1min":0.90100000000000002,"5min":1.7569999999999999,"15min":1.7569999999999999},"last":0.88500000000000001},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.72399999999999998,"15min":0.69699999999999995},"min":{"1min":0.58999999999999997,"5min":0.498,"15min":0.498},"max":{"1min":0.89100000000000001,"5min":1.746,"15min":1.746},"last":0.76300000000000001}]},{"osd":9,"last update":"Thu Mar 9 14:44:58 2023","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.78000000000000003,"15min":0.76100000000000001},"min":{"1min":0.59599999999999997,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":2.5419999999999998,"5min":2.5419999999999998,"15min":2.5419999999999998},"last":0.89700000000000002},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.72799999999999998,"15min":0.71399999999999997},"min":{"1min":0.55200000000000005,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":2.4969999999999999,"5min":2.4969999999999999,"15min":2.4969999999999999},"last":0.82299999999999995}]},{"osd":10,"last update":"Thu Mar 9 14:45:20 2023","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.77100000000000002,"15min":0.77200000000000002},"min":{"1min":0.61099999999999999,"5min":0.432,"15min":0.432},"max":{"1min":1.0009999999999999,"5min":1.069,"15min":1.069},"last":0.93799999999999994},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.79000000000000004,"15min":0.79300000000000004},"min":{"1min":0.627,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.109,"5min":1.228,"15min":1.228},"last":0.81599999999999995}]},{"osd":11,"last update":"Thu Mar 9 14:45:18 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.1779999999999999,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.85799999999999998}]}]},{"osd":5,"up_from":37,"seq":158913790010,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7336,"kb_used_data":1568,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93737816,"statfs":{"total":95995035648,"available":95987523584,"internally_reserved":0,"allocated":1605632,"data_stored":1081598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:45:38 2023","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.59499999999999997,"15min":0.51300000000000001},"min":{"1min":0.32400000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":9.7189999999999994,"5min":9.7189999999999994,"15min":9.7189999999999994},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.56299999999999994,"15min":0.51500000000000001},"min":{"1min":0.307,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":5.4660000000000002,"5min":5.4660000000000002,"15min":5.4660000000000002},"last":0.45100000000000001}]},{"osd":1,"last update":"Thu Mar 9 14:45:38 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.57499999999999996,"15min":0.55400000000000005},"min":{"1min":0.45400000000000001,"5min":0.316,"15min":0.316},"max":{"1min":0.77500000000000002,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.56100000000000005,"15min":0.53700000000000003},"min":{"1min":0.46200000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.752,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.66300000000000003}]},{"osd":2,"last update":"Thu Mar 9 14:45:38 2023","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.39500000000000002,"15min":0.36699999999999999},"min":{"1min":0.313,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.61499999999999999,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.41299999999999998,"15min":0.377},"min":{"1min":0.33900000000000002,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.65500000000000003,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.57299999999999995}]},{"osd":3,"last update":"Thu Mar 9 14:45:38 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.54800000000000004,"15min":0.53800000000000003},"min":{"1min":0.34499999999999997,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":2.7269999999999999,"5min":2.7269999999999999,"15min":2.7269999999999999},"last":0.435},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.622,"15min":0.56899999999999995},"min":{"1min":0.39200000000000002,"5min":0.313,"15min":0.313},"max":{"1min":5.6130000000000004,"5min":5.6130000000000004,"15min":5.6130000000000004},"last":0.58399999999999996}]},{"osd":4,"last update":"Thu Mar 9 14:45:38 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.58999999999999997,"15min":0.56299999999999994},"min":{"1min":0.51400000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.78000000000000003,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.625},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.53700000000000003,"15min":0.52800000000000002},"min":{"1min":0.42599999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.65800000000000003,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.53100000000000003}]},{"osd":6,"last update":"Thu Mar 9 14:45:52 2023","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.626,"15min":0.60799999999999998},"min":{"1min":0.438,"5min":0.315,"15min":0.315},"max":{"1min":2.5369999999999999,"5min":2.5369999999999999,"15min":2.5369999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.66800000000000004,"15min":0.63600000000000001},"min":{"1min":0.504,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":2.5019999999999998,"5min":2.5019999999999998,"15min":2.5019999999999998},"last":0.61499999999999999}]},{"osd":7,"last update":"Thu Mar 9 14:45:18 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.61099999999999999,"15min":0.59399999999999997},"min":{"1min":0.42099999999999999,"5min":0.38,"15min":0.38},"max":{"1min":0.84799999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.61899999999999999,"15min":0.59999999999999998},"min":{"1min":0.46899999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.83299999999999996,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.63900000000000001}]},{"osd":8,"last update":"Thu Mar 9 14:45:38 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.60199999999999998,"15min":0.60099999999999998},"min":{"1min":0.439,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.83499999999999996,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.42999999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.70699999999999996,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.60199999999999998}]},{"osd":9,"last update":"Thu Mar 9 14:44:58 2023","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.67800000000000005,"15min":0.64600000000000002},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":5.6749999999999998,"5min":5.6749999999999998,"15min":5.6749999999999998},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.71599999999999997,"15min":0.71499999999999997},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.468,"5min":1.468,"15min":1.468},"last":0.65600000000000003}]},{"osd":10,"last update":"Thu Mar 9 14:45:18 2023","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.69799999999999995,"15min":0.68999999999999995},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.86499999999999999,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.70899999999999996,"15min":0.69999999999999996},"min":{"1min":0.54300000000000004,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":0.877,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.67400000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":62087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":57379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":63932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-03-09T14:46:02.431 INFO:tasks.fwd_scrub.ceph_manager:clean! 2023-03-09T14:46:02.432 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-03-09T14:46:02.590 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.585+0000 7f00c704c700 1 -- 172.21.15.38:0/2198220689 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00c01019a0 msgr2=0x7f00c01082d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:02.590 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.585+0000 7f00c704c700 1 --2- 172.21.15.38:0/2198220689 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00c01019a0 0x7f00c01082d0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f00b800b0a0 tx=0x7f00b800b3b0 comp rx=0 tx=0).stop 2023-03-09T14:46:02.590 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.585+0000 7f00c704c700 1 -- 172.21.15.38:0/2198220689 shutdown_connections 2023-03-09T14:46:02.591 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.585+0000 7f00c704c700 1 --2- 172.21.15.38:0/2198220689 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00c01019a0 0x7f00c01082d0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:02.591 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.585+0000 7f00c704c700 1 --2- 172.21.15.38:0/2198220689 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f00c00ffea0 0x7f00c01002b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:02.591 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.585+0000 7f00c704c700 1 --2- 172.21.15.38:0/2198220689 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00c0100ff0 0x7f00c0101460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:02.591 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.585+0000 7f00c704c700 1 -- 172.21.15.38:0/2198220689 >> 172.21.15.38:0/2198220689 conn(0x7f00c00fb480 msgr2=0x7f00c00fd8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:02.592 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.585+0000 7f00c704c700 1 -- 172.21.15.38:0/2198220689 shutdown_connections 2023-03-09T14:46:02.592 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.585+0000 7f00c704c700 1 -- 172.21.15.38:0/2198220689 wait complete. 2023-03-09T14:46:02.592 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.586+0000 7f00c704c700 1 Processor -- start 2023-03-09T14:46:02.592 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.586+0000 7f00c704c700 1 -- start start 2023-03-09T14:46:02.593 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.586+0000 7f00c704c700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f00c00ffea0 0x7f00c0195c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:02.593 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.586+0000 7f00c704c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00c0100ff0 0x7f00c01961c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:02.593 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.586+0000 7f00c704c700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00c01019a0 0x7f00c019d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:02.593 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.586+0000 7f00c704c700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00c019d7f0 con 0x7f00c0100ff0 2023-03-09T14:46:02.593 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.586+0000 7f00c704c700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00c019d960 con 0x7f00c00ffea0 2023-03-09T14:46:02.593 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.586+0000 7f00c704c700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00c019db20 con 0x7f00c01019a0 2023-03-09T14:46:02.593 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.586+0000 7f00bffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00c0100ff0 0x7f00c01961c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:02.594 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.586+0000 7f00bffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00c0100ff0 0x7f00c01961c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38510/0 (socket says 172.21.15.38:38510) 2023-03-09T14:46:02.594 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.586+0000 7f00bffff700 1 -- 172.21.15.38:0/2983195006 learned_addr learned my addr 172.21.15.38:0/2983195006 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:46:02.594 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.586+0000 7f00c55e9700 1 --2- 172.21.15.38:0/2983195006 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00c01019a0 0x7f00c019d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:02.594 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.586+0000 7f00c4de8700 1 --2- 172.21.15.38:0/2983195006 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f00c00ffea0 0x7f00c0195c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:02.594 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.586+0000 7f00bffff700 1 -- 172.21.15.38:0/2983195006 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00c01019a0 msgr2=0x7f00c019d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:02.594 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.586+0000 7f00bffff700 1 --2- 172.21.15.38:0/2983195006 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00c01019a0 0x7f00c019d220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:02.594 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.586+0000 7f00bffff700 1 -- 172.21.15.38:0/2983195006 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f00c00ffea0 msgr2=0x7f00c0195c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:02.595 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.586+0000 7f00bffff700 1 --2- 172.21.15.38:0/2983195006 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f00c00ffea0 0x7f00c0195c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:02.595 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.586+0000 7f00bffff700 1 -- 172.21.15.38:0/2983195006 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00b8009d00 con 0x7f00c0100ff0 2023-03-09T14:46:02.595 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.586+0000 7f00c55e9700 1 --2- 172.21.15.38:0/2983195006 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00c01019a0 0x7f00c019d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:46:02.595 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.586+0000 7f00c4de8700 1 --2- 172.21.15.38:0/2983195006 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f00c00ffea0 0x7f00c0195c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-09T14:46:02.595 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.587+0000 7f00bffff700 1 --2- 172.21.15.38:0/2983195006 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00c0100ff0 0x7f00c01961c0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f00b400d950 tx=0x7f00b400dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:02.595 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.587+0000 7f00bdffb700 1 -- 172.21.15.38:0/2983195006 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00b4009b40 con 0x7f00c0100ff0 2023-03-09T14:46:02.595 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.587+0000 7f00bdffb700 1 -- 172.21.15.38:0/2983195006 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f00b4009ca0 con 0x7f00c0100ff0 2023-03-09T14:46:02.596 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.587+0000 7f00c704c700 1 -- 172.21.15.38:0/2983195006 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00c010b800 con 0x7f00c0100ff0 2023-03-09T14:46:02.596 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.587+0000 7f00bdffb700 1 -- 172.21.15.38:0/2983195006 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00b400f620 con 0x7f00c0100ff0 2023-03-09T14:46:02.596 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.587+0000 7f00c704c700 1 -- 172.21.15.38:0/2983195006 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00c010b670 con 0x7f00c0100ff0 2023-03-09T14:46:02.596 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.588+0000 7f00c704c700 1 -- 172.21.15.38:0/2983195006 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00c0106d20 con 0x7f00c0100ff0 2023-03-09T14:46:02.597 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.589+0000 7f00bdffb700 1 -- 172.21.15.38:0/2983195006 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) v1 ==== 133198+0+0 (secure 0 0 0) 0x7f00b4018030 con 0x7f00c0100ff0 2023-03-09T14:46:02.597 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.589+0000 7f00bdffb700 1 --2- 172.21.15.38:0/2983195006 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f00a809f440 0x7f00a80a18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:02.597 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.589+0000 7f00bdffb700 1 -- 172.21.15.38:0/2983195006 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f00b40ca340 con 0x7f00c0100ff0 2023-03-09T14:46:02.598 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.589+0000 7f00c4de8700 1 --2- 172.21.15.38:0/2983195006 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f00a809f440 0x7f00a80a18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:02.598 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.590+0000 7f00c4de8700 1 --2- 172.21.15.38:0/2983195006 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f00a809f440 0x7f00a80a18f0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f00c0196b50 tx=0x7f00b0009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:02.598 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.594+0000 7f00bdffb700 1 -- 172.21.15.38:0/2983195006 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f00b40943f0 con 0x7f00c0100ff0 2023-03-09T14:46:02.793 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.791+0000 7f00c704c700 1 -- 172.21.15.38:0/2983195006 --> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f00c0061840 con 0x7f00a809f440 2023-03-09T14:46:02.808 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.806+0000 7f00bdffb700 1 -- 172.21.15.38:0/2983195006 <== mgr.14150 v2:172.21.15.38:6800/3328887328 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+427295 (secure 0 0 0) 0x7f00c0061840 con 0x7f00a809f440 2023-03-09T14:46:02.813 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:46:02.815 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.809+0000 7f00c704c700 1 -- 172.21.15.38:0/2983195006 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f00a809f440 msgr2=0x7f00a80a18f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:02.815 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.809+0000 7f00c704c700 1 --2- 172.21.15.38:0/2983195006 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f00a809f440 0x7f00a80a18f0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f00c0196b50 tx=0x7f00b0009260 comp rx=0 tx=0).stop 2023-03-09T14:46:02.816 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.810+0000 7f00c704c700 1 -- 172.21.15.38:0/2983195006 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00c0100ff0 msgr2=0x7f00c01961c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:02.816 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.810+0000 7f00c704c700 1 --2- 172.21.15.38:0/2983195006 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00c0100ff0 0x7f00c01961c0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f00b400d950 tx=0x7f00b400dd10 comp rx=0 tx=0).stop 2023-03-09T14:46:02.816 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.810+0000 7f00c704c700 1 -- 172.21.15.38:0/2983195006 shutdown_connections 2023-03-09T14:46:02.816 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.810+0000 7f00c704c700 1 --2- 172.21.15.38:0/2983195006 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f00a809f440 0x7f00a80a18f0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:02.816 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.810+0000 7f00c704c700 1 --2- 172.21.15.38:0/2983195006 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f00c01019a0 0x7f00c019d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:02.816 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.810+0000 7f00c704c700 1 --2- 172.21.15.38:0/2983195006 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f00c00ffea0 0x7f00c0195c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:02.816 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.810+0000 7f00c704c700 1 --2- 172.21.15.38:0/2983195006 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f00c0100ff0 0x7f00c01961c0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:02.817 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.810+0000 7f00c704c700 1 -- 172.21.15.38:0/2983195006 >> 172.21.15.38:0/2983195006 conn(0x7f00c00fb480 msgr2=0x7f00c00fd680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:02.817 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.810+0000 7f00c704c700 1 -- 172.21.15.38:0/2983195006 shutdown_connections 2023-03-09T14:46:02.817 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.810+0000 7f00c704c700 1 -- 172.21.15.38:0/2983195006 wait complete. 2023-03-09T14:46:02.817 INFO:teuthology.orchestra.run.smithi038.stderr:dumped all 2023-03-09T14:46:02.825 INFO:teuthology.orchestra.run.smithi038.stdout:{"pg_ready":true,"pg_map":{"version":313,"stamp":"2023-03-09T14:46:02.783146+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":656546,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":653,"num_read_kb":446,"num_write":347,"num_write_kb":3057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1182773,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":783,"ondisk_log_size":783,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":83828,"kb_used_data":13780,"kb_used_omap":0,"kb_used_meta":69952,"kb_avail":1124857996,"statfs":{"total":1151940427776,"available":1151854587904,"internally_reserved":0,"allocated":14110720,"data_stored":7830367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":71630848},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4924,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":73,"num_read_kb":37,"num_write":5,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003375"},"pg_stats":[{"pgid":"3.3f","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.799653+0000","last_change":"2023-03-09T14:44:19.116371+0000","last_active":"2023-03-09T14:45:44.799653+0000","last_peered":"2023-03-09T14:45:44.799653+0000","last_clean":"2023-03-09T14:45:44.799653+0000","last_became_active":"2023-03-09T14:44:19.116024+0000","last_became_peered":"2023-03-09T14:44:19.116024+0000","last_unstale":"2023-03-09T14:45:44.799653+0000","last_undegraded":"2023-03-09T14:45:44.799653+0000","last_fullsized":"2023-03-09T14:45:44.799653+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T16:43:33.917176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"89'4","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.788538+0000","last_change":"2023-03-09T14:44:19.117312+0000","last_active":"2023-03-09T14:45:44.788538+0000","last_peered":"2023-03-09T14:45:44.788538+0000","last_clean":"2023-03-09T14:45:44.788538+0000","last_became_active":"2023-03-09T14:44:19.116920+0000","last_became_peered":"2023-03-09T14:44:19.116920+0000","last_unstale":"2023-03-09T14:45:44.788538+0000","last_undegraded":"2023-03-09T14:45:44.788538+0000","last_fullsized":"2023-03-09T14:45:44.788538+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:33:15.046431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.798813+0000","last_change":"2023-03-09T14:44:19.113482+0000","last_active":"2023-03-09T14:45:44.798813+0000","last_peered":"2023-03-09T14:45:44.798813+0000","last_clean":"2023-03-09T14:45:44.798813+0000","last_became_active":"2023-03-09T14:44:19.113183+0000","last_became_peered":"2023-03-09T14:44:19.113183+0000","last_unstale":"2023-03-09T14:45:44.798813+0000","last_undegraded":"2023-03-09T14:45:44.798813+0000","last_fullsized":"2023-03-09T14:45:44.798813+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:17:31.382255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.786039+0000","last_change":"2023-03-09T14:44:19.105799+0000","last_active":"2023-03-09T14:45:44.786039+0000","last_peered":"2023-03-09T14:45:44.786039+0000","last_clean":"2023-03-09T14:45:44.786039+0000","last_became_active":"2023-03-09T14:44:19.105405+0000","last_became_peered":"2023-03-09T14:44:19.105405+0000","last_unstale":"2023-03-09T14:45:44.786039+0000","last_undegraded":"2023-03-09T14:45:44.786039+0000","last_fullsized":"2023-03-09T14:45:44.786039+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:38:23.049984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"89'4","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.782474+0000","last_change":"2023-03-09T14:44:19.100656+0000","last_active":"2023-03-09T14:45:44.782474+0000","last_peered":"2023-03-09T14:45:44.782474+0000","last_clean":"2023-03-09T14:45:44.782474+0000","last_became_active":"2023-03-09T14:44:19.100278+0000","last_became_peered":"2023-03-09T14:44:19.100278+0000","last_unstale":"2023-03-09T14:45:44.782474+0000","last_undegraded":"2023-03-09T14:45:44.782474+0000","last_fullsized":"2023-03-09T14:45:44.782474+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:25:39.240251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.787745+0000","last_change":"2023-03-09T14:44:19.101065+0000","last_active":"2023-03-09T14:45:44.787745+0000","last_peered":"2023-03-09T14:45:44.787745+0000","last_clean":"2023-03-09T14:45:44.787745+0000","last_became_active":"2023-03-09T14:44:19.100451+0000","last_became_peered":"2023-03-09T14:44:19.100451+0000","last_unstale":"2023-03-09T14:45:44.787745+0000","last_undegraded":"2023-03-09T14:45:44.787745+0000","last_fullsized":"2023-03-09T14:45:44.787745+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T01:25:15.158254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"89'5","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.787475+0000","last_change":"2023-03-09T14:44:19.107996+0000","last_active":"2023-03-09T14:45:44.787475+0000","last_peered":"2023-03-09T14:45:44.787475+0000","last_clean":"2023-03-09T14:45:44.787475+0000","last_became_active":"2023-03-09T14:44:19.107683+0000","last_became_peered":"2023-03-09T14:44:19.107683+0000","last_unstale":"2023-03-09T14:45:44.787475+0000","last_undegraded":"2023-03-09T14:45:44.787475+0000","last_fullsized":"2023-03-09T14:45:44.787475+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:16:18.605176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.786169+0000","last_change":"2023-03-09T14:44:19.103940+0000","last_active":"2023-03-09T14:45:44.786169+0000","last_peered":"2023-03-09T14:45:44.786169+0000","last_clean":"2023-03-09T14:45:44.786169+0000","last_became_active":"2023-03-09T14:44:19.103620+0000","last_became_peered":"2023-03-09T14:44:19.103620+0000","last_unstale":"2023-03-09T14:45:44.786169+0000","last_undegraded":"2023-03-09T14:45:44.786169+0000","last_fullsized":"2023-03-09T14:45:44.786169+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:06:22.749636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"89'11","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.797150+0000","last_change":"2023-03-09T14:44:19.102695+0000","last_active":"2023-03-09T14:45:44.797150+0000","last_peered":"2023-03-09T14:45:44.797150+0000","last_clean":"2023-03-09T14:45:44.797150+0000","last_became_active":"2023-03-09T14:44:19.101486+0000","last_became_peered":"2023-03-09T14:44:19.101486+0000","last_unstale":"2023-03-09T14:45:44.797150+0000","last_undegraded":"2023-03-09T14:45:44.797150+0000","last_fullsized":"2023-03-09T14:45:44.797150+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T16:34:51.836443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"89'16","reported_seq":33,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.788716+0000","last_change":"2023-03-09T14:44:19.103522+0000","last_active":"2023-03-09T14:45:44.788716+0000","last_peered":"2023-03-09T14:45:44.788716+0000","last_clean":"2023-03-09T14:45:44.788716+0000","last_became_active":"2023-03-09T14:44:19.103242+0000","last_became_peered":"2023-03-09T14:44:19.103242+0000","last_unstale":"2023-03-09T14:45:44.788716+0000","last_undegraded":"2023-03-09T14:45:44.788716+0000","last_fullsized":"2023-03-09T14:45:44.788716+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:41:03.137554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.799291+0000","last_change":"2023-03-09T14:44:19.114934+0000","last_active":"2023-03-09T14:45:44.799291+0000","last_peered":"2023-03-09T14:45:44.799291+0000","last_clean":"2023-03-09T14:45:44.799291+0000","last_became_active":"2023-03-09T14:44:19.114620+0000","last_became_peered":"2023-03-09T14:44:19.114620+0000","last_unstale":"2023-03-09T14:45:44.799291+0000","last_undegraded":"2023-03-09T14:45:44.799291+0000","last_fullsized":"2023-03-09T14:45:44.799291+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T01:48:40.082076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.785702+0000","last_change":"2023-03-09T14:44:19.107677+0000","last_active":"2023-03-09T14:45:44.785702+0000","last_peered":"2023-03-09T14:45:44.785702+0000","last_clean":"2023-03-09T14:45:44.785702+0000","last_became_active":"2023-03-09T14:44:19.107369+0000","last_became_peered":"2023-03-09T14:44:19.107369+0000","last_unstale":"2023-03-09T14:45:44.785702+0000","last_undegraded":"2023-03-09T14:45:44.785702+0000","last_fullsized":"2023-03-09T14:45:44.785702+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:47:40.117205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"89'10","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.786986+0000","last_change":"2023-03-09T14:44:19.108852+0000","last_active":"2023-03-09T14:45:44.786986+0000","last_peered":"2023-03-09T14:45:44.786986+0000","last_clean":"2023-03-09T14:45:44.786986+0000","last_became_active":"2023-03-09T14:44:19.108515+0000","last_became_peered":"2023-03-09T14:44:19.108515+0000","last_unstale":"2023-03-09T14:45:44.786986+0000","last_undegraded":"2023-03-09T14:45:44.786986+0000","last_fullsized":"2023-03-09T14:45:44.786986+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:00:36.243952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.786886+0000","last_change":"2023-03-09T14:44:19.118673+0000","last_active":"2023-03-09T14:45:44.786886+0000","last_peered":"2023-03-09T14:45:44.786886+0000","last_clean":"2023-03-09T14:45:44.786886+0000","last_became_active":"2023-03-09T14:44:19.117656+0000","last_became_peered":"2023-03-09T14:44:19.117656+0000","last_unstale":"2023-03-09T14:45:44.786886+0000","last_undegraded":"2023-03-09T14:45:44.786886+0000","last_fullsized":"2023-03-09T14:45:44.786886+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:02:50.401680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.789818+0000","last_change":"2023-03-09T14:44:19.112979+0000","last_active":"2023-03-09T14:45:44.789818+0000","last_peered":"2023-03-09T14:45:44.789818+0000","last_clean":"2023-03-09T14:45:44.789818+0000","last_became_active":"2023-03-09T14:44:19.112671+0000","last_became_peered":"2023-03-09T14:44:19.112671+0000","last_unstale":"2023-03-09T14:45:44.789818+0000","last_undegraded":"2023-03-09T14:45:44.789818+0000","last_fullsized":"2023-03-09T14:45:44.789818+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:00:43.465327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.795403+0000","last_change":"2023-03-09T14:44:19.104881+0000","last_active":"2023-03-09T14:45:44.795403+0000","last_peered":"2023-03-09T14:45:44.795403+0000","last_clean":"2023-03-09T14:45:44.795403+0000","last_became_active":"2023-03-09T14:44:19.104647+0000","last_became_peered":"2023-03-09T14:44:19.104647+0000","last_unstale":"2023-03-09T14:45:44.795403+0000","last_undegraded":"2023-03-09T14:45:44.795403+0000","last_fullsized":"2023-03-09T14:45:44.795403+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T20:16:24.625362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.799646+0000","last_change":"2023-03-09T14:44:19.117937+0000","last_active":"2023-03-09T14:45:44.799646+0000","last_peered":"2023-03-09T14:45:44.799646+0000","last_clean":"2023-03-09T14:45:44.799646+0000","last_became_active":"2023-03-09T14:44:19.117628+0000","last_became_peered":"2023-03-09T14:44:19.117628+0000","last_unstale":"2023-03-09T14:45:44.799646+0000","last_undegraded":"2023-03-09T14:45:44.799646+0000","last_fullsized":"2023-03-09T14:45:44.799646+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:22:18.619171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"89'10","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.793895+0000","last_change":"2023-03-09T14:44:19.105071+0000","last_active":"2023-03-09T14:45:44.793895+0000","last_peered":"2023-03-09T14:45:44.793895+0000","last_clean":"2023-03-09T14:45:44.793895+0000","last_became_active":"2023-03-09T14:44:19.103396+0000","last_became_peered":"2023-03-09T14:44:19.103396+0000","last_unstale":"2023-03-09T14:45:44.793895+0000","last_undegraded":"2023-03-09T14:45:44.793895+0000","last_fullsized":"2023-03-09T14:45:44.793895+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:14:51.924636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.783812+0000","last_change":"2023-03-09T14:44:19.104107+0000","last_active":"2023-03-09T14:45:44.783812+0000","last_peered":"2023-03-09T14:45:44.783812+0000","last_clean":"2023-03-09T14:45:44.783812+0000","last_became_active":"2023-03-09T14:44:19.103793+0000","last_became_peered":"2023-03-09T14:44:19.103793+0000","last_unstale":"2023-03-09T14:45:44.783812+0000","last_undegraded":"2023-03-09T14:45:44.783812+0000","last_fullsized":"2023-03-09T14:45:44.783812+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:57:29.169744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.856426+0000","last_change":"2023-03-09T14:44:23.489947+0000","last_active":"2023-03-09T14:45:14.856426+0000","last_peered":"2023-03-09T14:45:14.856426+0000","last_clean":"2023-03-09T14:45:14.856426+0000","last_became_active":"2023-03-09T14:44:23.489560+0000","last_became_peered":"2023-03-09T14:44:23.489560+0000","last_unstale":"2023-03-09T14:45:14.856426+0000","last_undegraded":"2023-03-09T14:45:14.856426+0000","last_fullsized":"2023-03-09T14:45:14.856426+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:06:10.202176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"87'9","reported_seq":49,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:15.961952+0000","last_change":"2023-03-09T14:45:15.961952+0000","last_active":"2023-03-09T14:45:15.961952+0000","last_peered":"2023-03-09T14:45:15.961952+0000","last_clean":"2023-03-09T14:45:15.961952+0000","last_became_active":"2023-03-09T14:45:15.865191+0000","last_became_peered":"2023-03-09T14:45:15.865191+0000","last_unstale":"2023-03-09T14:45:15.961952+0000","last_undegraded":"2023-03-09T14:45:15.961952+0000","last_fullsized":"2023-03-09T14:45:15.961952+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T01:21:05.925138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":566,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"89'5","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.785218+0000","last_change":"2023-03-09T14:44:19.105390+0000","last_active":"2023-03-09T14:45:44.785218+0000","last_peered":"2023-03-09T14:45:44.785218+0000","last_clean":"2023-03-09T14:45:44.785218+0000","last_became_active":"2023-03-09T14:44:19.104694+0000","last_became_peered":"2023-03-09T14:44:19.104694+0000","last_unstale":"2023-03-09T14:45:44.785218+0000","last_undegraded":"2023-03-09T14:45:44.785218+0000","last_fullsized":"2023-03-09T14:45:44.785218+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:19:08.101847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.856827+0000","last_change":"2023-03-09T14:44:23.484103+0000","last_active":"2023-03-09T14:45:14.856827+0000","last_peered":"2023-03-09T14:45:14.856827+0000","last_clean":"2023-03-09T14:45:14.856827+0000","last_became_active":"2023-03-09T14:44:23.483271+0000","last_became_peered":"2023-03-09T14:44:23.483271+0000","last_unstale":"2023-03-09T14:45:14.856827+0000","last_undegraded":"2023-03-09T14:45:14.856827+0000","last_fullsized":"2023-03-09T14:45:14.856827+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:24:06.212623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"87'2","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.864060+0000","last_change":"2023-03-09T14:44:18.667321+0000","last_active":"2023-03-09T14:45:14.864060+0000","last_peered":"2023-03-09T14:45:14.864060+0000","last_clean":"2023-03-09T14:45:14.864060+0000","last_became_active":"2023-03-09T14:44:18.666974+0000","last_became_peered":"2023-03-09T14:44:18.666974+0000","last_unstale":"2023-03-09T14:45:14.864060+0000","last_undegraded":"2023-03-09T14:45:14.864060+0000","last_fullsized":"2023-03-09T14:45:14.864060+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:31:07.327792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.799185+0000","last_change":"2023-03-09T14:44:19.116866+0000","last_active":"2023-03-09T14:45:44.799185+0000","last_peered":"2023-03-09T14:45:44.799185+0000","last_clean":"2023-03-09T14:45:44.799185+0000","last_became_active":"2023-03-09T14:44:19.116571+0000","last_became_peered":"2023-03-09T14:44:19.116571+0000","last_unstale":"2023-03-09T14:45:44.799185+0000","last_undegraded":"2023-03-09T14:45:44.799185+0000","last_fullsized":"2023-03-09T14:45:44.799185+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T16:22:12.723903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.859406+0000","last_change":"2023-03-09T14:44:23.488192+0000","last_active":"2023-03-09T14:45:14.859406+0000","last_peered":"2023-03-09T14:45:14.859406+0000","last_clean":"2023-03-09T14:45:14.859406+0000","last_became_active":"2023-03-09T14:44:23.487734+0000","last_became_peered":"2023-03-09T14:44:23.487734+0000","last_unstale":"2023-03-09T14:45:14.859406+0000","last_undegraded":"2023-03-09T14:45:14.859406+0000","last_fullsized":"2023-03-09T14:45:14.859406+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:35:50.319216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"87'4","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.861901+0000","last_change":"2023-03-09T14:44:18.663577+0000","last_active":"2023-03-09T14:45:14.861901+0000","last_peered":"2023-03-09T14:45:14.861901+0000","last_clean":"2023-03-09T14:45:14.861901+0000","last_became_active":"2023-03-09T14:44:18.663272+0000","last_became_peered":"2023-03-09T14:44:18.663272+0000","last_unstale":"2023-03-09T14:45:14.861901+0000","last_undegraded":"2023-03-09T14:45:14.861901+0000","last_fullsized":"2023-03-09T14:45:14.861901+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T20:55:19.661875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.794071+0000","last_change":"2023-03-09T14:44:19.106986+0000","last_active":"2023-03-09T14:45:44.794071+0000","last_peered":"2023-03-09T14:45:44.794071+0000","last_clean":"2023-03-09T14:45:44.794071+0000","last_became_active":"2023-03-09T14:44:19.106625+0000","last_became_peered":"2023-03-09T14:44:19.106625+0000","last_unstale":"2023-03-09T14:45:44.794071+0000","last_undegraded":"2023-03-09T14:45:44.794071+0000","last_fullsized":"2023-03-09T14:45:44.794071+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:03:54.521603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.858888+0000","last_change":"2023-03-09T14:44:23.483803+0000","last_active":"2023-03-09T14:45:14.858888+0000","last_peered":"2023-03-09T14:45:14.858888+0000","last_clean":"2023-03-09T14:45:14.858888+0000","last_became_active":"2023-03-09T14:44:23.483286+0000","last_became_peered":"2023-03-09T14:44:23.483286+0000","last_unstale":"2023-03-09T14:45:14.858888+0000","last_undegraded":"2023-03-09T14:45:14.858888+0000","last_fullsized":"2023-03-09T14:45:14.858888+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:32:35.975670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"82'2","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.861301+0000","last_change":"2023-03-09T14:44:18.657685+0000","last_active":"2023-03-09T14:45:14.861301+0000","last_peered":"2023-03-09T14:45:14.861301+0000","last_clean":"2023-03-09T14:45:14.861301+0000","last_became_active":"2023-03-09T14:44:18.657355+0000","last_became_peered":"2023-03-09T14:44:18.657355+0000","last_unstale":"2023-03-09T14:45:14.861301+0000","last_undegraded":"2023-03-09T14:45:14.861301+0000","last_fullsized":"2023-03-09T14:45:14.861301+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:33:59.259690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"89'15","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.799331+0000","last_change":"2023-03-09T14:44:19.111687+0000","last_active":"2023-03-09T14:45:44.799331+0000","last_peered":"2023-03-09T14:45:44.799331+0000","last_clean":"2023-03-09T14:45:44.799331+0000","last_became_active":"2023-03-09T14:44:19.111481+0000","last_became_peered":"2023-03-09T14:44:19.111481+0000","last_unstale":"2023-03-09T14:45:44.799331+0000","last_undegraded":"2023-03-09T14:45:44.799331+0000","last_fullsized":"2023-03-09T14:45:44.799331+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:06:14.452967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.857536+0000","last_change":"2023-03-09T14:44:23.485075+0000","last_active":"2023-03-09T14:45:14.857536+0000","last_peered":"2023-03-09T14:45:14.857536+0000","last_clean":"2023-03-09T14:45:14.857536+0000","last_became_active":"2023-03-09T14:44:23.484648+0000","last_became_peered":"2023-03-09T14:44:23.484648+0000","last_unstale":"2023-03-09T14:45:14.857536+0000","last_undegraded":"2023-03-09T14:45:14.857536+0000","last_fullsized":"2023-03-09T14:45:14.857536+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:49:40.332276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"87'5","reported_seq":125,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:46:00.933398+0000","last_change":"2023-03-09T14:45:15.868911+0000","last_active":"2023-03-09T14:46:00.933398+0000","last_peered":"2023-03-09T14:46:00.933398+0000","last_clean":"2023-03-09T14:46:00.933398+0000","last_became_active":"2023-03-09T14:45:15.861592+0000","last_became_peered":"2023-03-09T14:45:15.861592+0000","last_unstale":"2023-03-09T14:46:00.933398+0000","last_undegraded":"2023-03-09T14:46:00.933398+0000","last_fullsized":"2023-03-09T14:46:00.933398+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T20:17:55.938860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":85,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":622,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.786994+0000","last_change":"2023-03-09T14:44:19.107572+0000","last_active":"2023-03-09T14:45:44.786994+0000","last_peered":"2023-03-09T14:45:44.786994+0000","last_clean":"2023-03-09T14:45:44.786994+0000","last_became_active":"2023-03-09T14:44:19.106632+0000","last_became_peered":"2023-03-09T14:44:19.106632+0000","last_unstale":"2023-03-09T14:45:44.786994+0000","last_undegraded":"2023-03-09T14:45:44.786994+0000","last_fullsized":"2023-03-09T14:45:44.786994+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T19:04:01.379375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.857041+0000","last_change":"2023-03-09T14:44:23.494002+0000","last_active":"2023-03-09T14:45:14.857041+0000","last_peered":"2023-03-09T14:45:14.857041+0000","last_clean":"2023-03-09T14:45:14.857041+0000","last_became_active":"2023-03-09T14:44:23.493628+0000","last_became_peered":"2023-03-09T14:44:23.493628+0000","last_unstale":"2023-03-09T14:45:14.857041+0000","last_undegraded":"2023-03-09T14:45:14.857041+0000","last_fullsized":"2023-03-09T14:45:14.857041+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:30:47.970556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"87'3","reported_seq":36,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:15.964036+0000","last_change":"2023-03-09T14:45:15.964036+0000","last_active":"2023-03-09T14:45:15.964036+0000","last_peered":"2023-03-09T14:45:15.964036+0000","last_clean":"2023-03-09T14:45:15.964036+0000","last_became_active":"2023-03-09T14:45:15.865998+0000","last_became_peered":"2023-03-09T14:45:15.865998+0000","last_unstale":"2023-03-09T14:45:15.964036+0000","last_undegraded":"2023-03-09T14:45:15.964036+0000","last_fullsized":"2023-03-09T14:45:15.964036+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:16:58.213684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.786013+0000","last_change":"2023-03-09T14:44:19.111303+0000","last_active":"2023-03-09T14:45:44.786013+0000","last_peered":"2023-03-09T14:45:44.786013+0000","last_clean":"2023-03-09T14:45:44.786013+0000","last_became_active":"2023-03-09T14:44:19.111026+0000","last_became_peered":"2023-03-09T14:44:19.111026+0000","last_unstale":"2023-03-09T14:45:44.786013+0000","last_undegraded":"2023-03-09T14:45:44.786013+0000","last_fullsized":"2023-03-09T14:45:44.786013+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T02:26:32.496084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.859059+0000","last_change":"2023-03-09T14:44:23.493272+0000","last_active":"2023-03-09T14:45:14.859059+0000","last_peered":"2023-03-09T14:45:14.859059+0000","last_clean":"2023-03-09T14:45:14.859059+0000","last_became_active":"2023-03-09T14:44:23.492785+0000","last_became_peered":"2023-03-09T14:44:23.492785+0000","last_unstale":"2023-03-09T14:45:14.859059+0000","last_undegraded":"2023-03-09T14:45:14.859059+0000","last_fullsized":"2023-03-09T14:45:14.859059+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T20:18:39.758938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"87'2","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.861264+0000","last_change":"2023-03-09T14:44:18.658179+0000","last_active":"2023-03-09T14:45:14.861264+0000","last_peered":"2023-03-09T14:45:14.861264+0000","last_clean":"2023-03-09T14:45:14.861264+0000","last_became_active":"2023-03-09T14:44:18.657851+0000","last_became_peered":"2023-03-09T14:44:18.657851+0000","last_unstale":"2023-03-09T14:45:14.861264+0000","last_undegraded":"2023-03-09T14:45:14.861264+0000","last_fullsized":"2023-03-09T14:45:14.861264+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:52:49.291615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.782372+0000","last_change":"2023-03-09T14:44:19.104176+0000","last_active":"2023-03-09T14:45:44.782372+0000","last_peered":"2023-03-09T14:45:44.782372+0000","last_clean":"2023-03-09T14:45:44.782372+0000","last_became_active":"2023-03-09T14:44:19.103816+0000","last_became_peered":"2023-03-09T14:44:19.103816+0000","last_unstale":"2023-03-09T14:45:44.782372+0000","last_undegraded":"2023-03-09T14:45:44.782372+0000","last_fullsized":"2023-03-09T14:45:44.782372+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T02:06:41.876169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.864893+0000","last_change":"2023-03-09T14:44:23.490876+0000","last_active":"2023-03-09T14:45:14.864893+0000","last_peered":"2023-03-09T14:45:14.864893+0000","last_clean":"2023-03-09T14:45:14.864893+0000","last_became_active":"2023-03-09T14:44:23.490461+0000","last_became_peered":"2023-03-09T14:44:23.490461+0000","last_unstale":"2023-03-09T14:45:14.864893+0000","last_undegraded":"2023-03-09T14:45:14.864893+0000","last_fullsized":"2023-03-09T14:45:14.864893+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T16:28:40.456613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"87'1","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.852373+0000","last_change":"2023-03-09T14:44:18.662513+0000","last_active":"2023-03-09T14:45:14.852373+0000","last_peered":"2023-03-09T14:45:14.852373+0000","last_clean":"2023-03-09T14:45:14.852373+0000","last_became_active":"2023-03-09T14:44:18.661907+0000","last_became_peered":"2023-03-09T14:44:18.661907+0000","last_unstale":"2023-03-09T14:45:14.852373+0000","last_undegraded":"2023-03-09T14:45:14.852373+0000","last_fullsized":"2023-03-09T14:45:14.852373+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:45:17.373213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"89'11","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.779797+0000","last_change":"2023-03-09T14:44:19.115297+0000","last_active":"2023-03-09T14:45:44.779797+0000","last_peered":"2023-03-09T14:45:44.779797+0000","last_clean":"2023-03-09T14:45:44.779797+0000","last_became_active":"2023-03-09T14:44:19.114973+0000","last_became_peered":"2023-03-09T14:44:19.114973+0000","last_unstale":"2023-03-09T14:45:44.779797+0000","last_undegraded":"2023-03-09T14:45:44.779797+0000","last_fullsized":"2023-03-09T14:45:44.779797+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:11:22.226608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.864500+0000","last_change":"2023-03-09T14:44:23.482772+0000","last_active":"2023-03-09T14:45:14.864500+0000","last_peered":"2023-03-09T14:45:14.864500+0000","last_clean":"2023-03-09T14:45:14.864500+0000","last_became_active":"2023-03-09T14:44:23.481750+0000","last_became_peered":"2023-03-09T14:44:23.481750+0000","last_unstale":"2023-03-09T14:45:14.864500+0000","last_undegraded":"2023-03-09T14:45:14.864500+0000","last_fullsized":"2023-03-09T14:45:14.864500+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T19:26:03.737713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"87'3","reported_seq":119,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:58.929566+0000","last_change":"2023-03-09T14:45:15.870269+0000","last_active":"2023-03-09T14:45:58.929566+0000","last_peered":"2023-03-09T14:45:58.929566+0000","last_clean":"2023-03-09T14:45:58.929566+0000","last_became_active":"2023-03-09T14:45:15.862610+0000","last_became_peered":"2023-03-09T14:45:15.862610+0000","last_unstale":"2023-03-09T14:45:58.929566+0000","last_undegraded":"2023-03-09T14:45:58.929566+0000","last_fullsized":"2023-03-09T14:45:58.929566+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T02:37:23.157022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":81,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,6,9],"acting":[10,6,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"89'10","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.799241+0000","last_change":"2023-03-09T14:44:19.100914+0000","last_active":"2023-03-09T14:45:44.799241+0000","last_peered":"2023-03-09T14:45:44.799241+0000","last_clean":"2023-03-09T14:45:44.799241+0000","last_became_active":"2023-03-09T14:44:19.100451+0000","last_became_peered":"2023-03-09T14:44:19.100451+0000","last_unstale":"2023-03-09T14:45:44.799241+0000","last_undegraded":"2023-03-09T14:45:44.799241+0000","last_fullsized":"2023-03-09T14:45:44.799241+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:18:41.852697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.860006+0000","last_change":"2023-03-09T14:44:23.490129+0000","last_active":"2023-03-09T14:45:14.860006+0000","last_peered":"2023-03-09T14:45:14.860006+0000","last_clean":"2023-03-09T14:45:14.860006+0000","last_became_active":"2023-03-09T14:44:23.489696+0000","last_became_peered":"2023-03-09T14:44:23.489696+0000","last_unstale":"2023-03-09T14:45:14.860006+0000","last_undegraded":"2023-03-09T14:45:14.860006+0000","last_fullsized":"2023-03-09T14:45:14.860006+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T16:47:41.943904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"87'2","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:15.955780+0000","last_change":"2023-03-09T14:45:15.955780+0000","last_active":"2023-03-09T14:45:15.955780+0000","last_peered":"2023-03-09T14:45:15.955780+0000","last_clean":"2023-03-09T14:45:15.955780+0000","last_became_active":"2023-03-09T14:45:15.862567+0000","last_became_peered":"2023-03-09T14:45:15.862567+0000","last_unstale":"2023-03-09T14:45:15.955780+0000","last_undegraded":"2023-03-09T14:45:15.955780+0000","last_fullsized":"2023-03-09T14:45:15.955780+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:08:41.748638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.786794+0000","last_change":"2023-03-09T14:44:19.119125+0000","last_active":"2023-03-09T14:45:44.786794+0000","last_peered":"2023-03-09T14:45:44.786794+0000","last_clean":"2023-03-09T14:45:44.786794+0000","last_became_active":"2023-03-09T14:44:19.118677+0000","last_became_peered":"2023-03-09T14:44:19.118677+0000","last_unstale":"2023-03-09T14:45:44.786794+0000","last_undegraded":"2023-03-09T14:45:44.786794+0000","last_fullsized":"2023-03-09T14:45:44.786794+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:06:28.740355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.859670+0000","last_change":"2023-03-09T14:44:23.484752+0000","last_active":"2023-03-09T14:45:14.859670+0000","last_peered":"2023-03-09T14:45:14.859670+0000","last_clean":"2023-03-09T14:45:14.859670+0000","last_became_active":"2023-03-09T14:44:23.484045+0000","last_became_peered":"2023-03-09T14:44:23.484045+0000","last_unstale":"2023-03-09T14:45:14.859670+0000","last_undegraded":"2023-03-09T14:45:14.859670+0000","last_fullsized":"2023-03-09T14:45:14.859670+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:58:39.704017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"87'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.853129+0000","last_change":"2023-03-09T14:44:18.665122+0000","last_active":"2023-03-09T14:45:14.853129+0000","last_peered":"2023-03-09T14:45:14.853129+0000","last_clean":"2023-03-09T14:45:14.853129+0000","last_became_active":"2023-03-09T14:44:18.664786+0000","last_became_peered":"2023-03-09T14:44:18.664786+0000","last_unstale":"2023-03-09T14:45:14.853129+0000","last_undegraded":"2023-03-09T14:45:14.853129+0000","last_fullsized":"2023-03-09T14:45:14.853129+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T20:29:20.919293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":656,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"89'11","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.797201+0000","last_change":"2023-03-09T14:44:19.108693+0000","last_active":"2023-03-09T14:45:44.797201+0000","last_peered":"2023-03-09T14:45:44.797201+0000","last_clean":"2023-03-09T14:45:44.797201+0000","last_became_active":"2023-03-09T14:44:19.108370+0000","last_became_peered":"2023-03-09T14:44:19.108370+0000","last_unstale":"2023-03-09T14:45:44.797201+0000","last_undegraded":"2023-03-09T14:45:44.797201+0000","last_fullsized":"2023-03-09T14:45:44.797201+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T16:30:58.726080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:15.862206+0000","last_change":"2023-03-09T14:45:15.862206+0000","last_active":"2023-03-09T14:45:15.862206+0000","last_peered":"2023-03-09T14:45:15.862206+0000","last_clean":"2023-03-09T14:45:15.862206+0000","last_became_active":"2023-03-09T14:45:15.861775+0000","last_became_peered":"2023-03-09T14:45:15.861775+0000","last_unstale":"2023-03-09T14:45:15.862206+0000","last_undegraded":"2023-03-09T14:45:15.862206+0000","last_fullsized":"2023-03-09T14:45:15.862206+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:31:21.810745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.14","version":"89'35","reported_seq":225,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:59.931365+0000","last_change":"2023-03-09T14:44:18.661904+0000","last_active":"2023-03-09T14:45:59.931365+0000","last_peered":"2023-03-09T14:45:59.931365+0000","last_clean":"2023-03-09T14:45:59.931365+0000","last_became_active":"2023-03-09T14:44:18.661597+0000","last_became_peered":"2023-03-09T14:44:18.661597+0000","last_unstale":"2023-03-09T14:45:59.931365+0000","last_undegraded":"2023-03-09T14:45:59.931365+0000","last_fullsized":"2023-03-09T14:45:59.931365+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T14:57:04.978132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61521,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":172,"num_read_kb":147,"num_write":36,"num_write_kb":76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.786319+0000","last_change":"2023-03-09T14:44:19.111671+0000","last_active":"2023-03-09T14:45:44.786319+0000","last_peered":"2023-03-09T14:45:44.786319+0000","last_clean":"2023-03-09T14:45:44.786319+0000","last_became_active":"2023-03-09T14:44:19.111172+0000","last_became_peered":"2023-03-09T14:44:19.111172+0000","last_unstale":"2023-03-09T14:45:44.786319+0000","last_undegraded":"2023-03-09T14:45:44.786319+0000","last_fullsized":"2023-03-09T14:45:44.786319+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T16:27:23.921853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.853428+0000","last_change":"2023-03-09T14:44:23.486338+0000","last_active":"2023-03-09T14:45:14.853428+0000","last_peered":"2023-03-09T14:45:14.853428+0000","last_clean":"2023-03-09T14:45:14.853428+0000","last_became_active":"2023-03-09T14:44:23.485729+0000","last_became_peered":"2023-03-09T14:44:23.485729+0000","last_unstale":"2023-03-09T14:45:14.853428+0000","last_undegraded":"2023-03-09T14:45:14.853428+0000","last_fullsized":"2023-03-09T14:45:14.853428+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:28:23.872633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"87'6","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.857067+0000","last_change":"2023-03-09T14:44:18.664942+0000","last_active":"2023-03-09T14:45:14.857067+0000","last_peered":"2023-03-09T14:45:14.857067+0000","last_clean":"2023-03-09T14:45:14.857067+0000","last_became_active":"2023-03-09T14:44:18.664007+0000","last_became_peered":"2023-03-09T14:44:18.664007+0000","last_unstale":"2023-03-09T14:45:14.857067+0000","last_undegraded":"2023-03-09T14:45:14.857067+0000","last_fullsized":"2023-03-09T14:45:14.857067+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:36:21.329388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.782333+0000","last_change":"2023-03-09T14:44:19.110140+0000","last_active":"2023-03-09T14:45:44.782333+0000","last_peered":"2023-03-09T14:45:44.782333+0000","last_clean":"2023-03-09T14:45:44.782333+0000","last_became_active":"2023-03-09T14:44:19.109811+0000","last_became_peered":"2023-03-09T14:44:19.109811+0000","last_unstale":"2023-03-09T14:45:44.782333+0000","last_undegraded":"2023-03-09T14:45:44.782333+0000","last_fullsized":"2023-03-09T14:45:44.782333+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T14:56:14.638662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.858304+0000","last_change":"2023-03-09T14:44:23.491318+0000","last_active":"2023-03-09T14:45:14.858304+0000","last_peered":"2023-03-09T14:45:14.858304+0000","last_clean":"2023-03-09T14:45:14.858304+0000","last_became_active":"2023-03-09T14:44:23.490915+0000","last_became_peered":"2023-03-09T14:44:23.490915+0000","last_unstale":"2023-03-09T14:45:14.858304+0000","last_undegraded":"2023-03-09T14:45:14.858304+0000","last_fullsized":"2023-03-09T14:45:14.858304+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:32:06.877573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"87'3","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.852806+0000","last_change":"2023-03-09T14:44:18.657323+0000","last_active":"2023-03-09T14:45:14.852806+0000","last_peered":"2023-03-09T14:45:14.852806+0000","last_clean":"2023-03-09T14:45:14.852806+0000","last_became_active":"2023-03-09T14:44:18.657017+0000","last_became_peered":"2023-03-09T14:44:18.657017+0000","last_unstale":"2023-03-09T14:45:14.852806+0000","last_undegraded":"2023-03-09T14:45:14.852806+0000","last_fullsized":"2023-03-09T14:45:14.852806+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:43:11.562723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"89'15","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.791050+0000","last_change":"2023-03-09T14:44:19.112194+0000","last_active":"2023-03-09T14:45:44.791050+0000","last_peered":"2023-03-09T14:45:44.791050+0000","last_clean":"2023-03-09T14:45:44.791050+0000","last_became_active":"2023-03-09T14:44:19.111924+0000","last_became_peered":"2023-03-09T14:44:19.111924+0000","last_unstale":"2023-03-09T14:45:44.791050+0000","last_undegraded":"2023-03-09T14:45:44.791050+0000","last_fullsized":"2023-03-09T14:45:44.791050+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T19:54:22.669848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.860733+0000","last_change":"2023-03-09T14:44:23.493993+0000","last_active":"2023-03-09T14:45:14.860733+0000","last_peered":"2023-03-09T14:45:14.860733+0000","last_clean":"2023-03-09T14:45:14.860733+0000","last_became_active":"2023-03-09T14:44:23.493535+0000","last_became_peered":"2023-03-09T14:44:23.493535+0000","last_unstale":"2023-03-09T14:45:14.860733+0000","last_undegraded":"2023-03-09T14:45:14.860733+0000","last_fullsized":"2023-03-09T14:45:14.860733+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T01:18:23.669790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'6","reported_seq":111,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:59.674795+0000","last_change":"2023-03-09T14:44:18.659914+0000","last_active":"2023-03-09T14:45:59.674795+0000","last_peered":"2023-03-09T14:45:59.674795+0000","last_clean":"2023-03-09T14:45:59.674795+0000","last_became_active":"2023-03-09T14:44:18.658835+0000","last_became_peered":"2023-03-09T14:44:18.658835+0000","last_unstale":"2023-03-09T14:45:59.674795+0000","last_undegraded":"2023-03-09T14:45:59.674795+0000","last_fullsized":"2023-03-09T14:45:59.674795+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T14:55:05.413260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1437,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":88,"num_read_kb":6,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.787772+0000","last_change":"2023-03-09T14:44:19.111058+0000","last_active":"2023-03-09T14:45:44.787772+0000","last_peered":"2023-03-09T14:45:44.787772+0000","last_clean":"2023-03-09T14:45:44.787772+0000","last_became_active":"2023-03-09T14:44:19.110760+0000","last_became_peered":"2023-03-09T14:44:19.110760+0000","last_unstale":"2023-03-09T14:45:44.787772+0000","last_undegraded":"2023-03-09T14:45:44.787772+0000","last_fullsized":"2023-03-09T14:45:44.787772+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:25:13.199416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.861493+0000","last_change":"2023-03-09T14:44:23.484710+0000","last_active":"2023-03-09T14:45:14.861493+0000","last_peered":"2023-03-09T14:45:14.861493+0000","last_clean":"2023-03-09T14:45:14.861493+0000","last_became_active":"2023-03-09T14:44:23.483928+0000","last_became_peered":"2023-03-09T14:44:23.483928+0000","last_unstale":"2023-03-09T14:45:14.861493+0000","last_undegraded":"2023-03-09T14:45:14.861493+0000","last_fullsized":"2023-03-09T14:45:14.861493+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T14:51:32.124111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"87'15","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.860284+0000","last_change":"2023-03-09T14:44:18.663692+0000","last_active":"2023-03-09T14:45:14.860284+0000","last_peered":"2023-03-09T14:45:14.860284+0000","last_clean":"2023-03-09T14:45:14.860284+0000","last_became_active":"2023-03-09T14:44:18.663400+0000","last_became_peered":"2023-03-09T14:44:18.663400+0000","last_unstale":"2023-03-09T14:45:14.860284+0000","last_undegraded":"2023-03-09T14:45:14.860284+0000","last_fullsized":"2023-03-09T14:45:14.860284+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:32:31.634315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.797691+0000","last_change":"2023-03-09T14:44:19.110819+0000","last_active":"2023-03-09T14:45:44.797691+0000","last_peered":"2023-03-09T14:45:44.797691+0000","last_clean":"2023-03-09T14:45:44.797691+0000","last_became_active":"2023-03-09T14:44:19.110542+0000","last_became_peered":"2023-03-09T14:44:19.110542+0000","last_unstale":"2023-03-09T14:45:44.797691+0000","last_undegraded":"2023-03-09T14:45:44.797691+0000","last_fullsized":"2023-03-09T14:45:44.797691+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:25:38.382685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.852360+0000","last_change":"2023-03-09T14:44:23.484614+0000","last_active":"2023-03-09T14:45:14.852360+0000","last_peered":"2023-03-09T14:45:14.852360+0000","last_clean":"2023-03-09T14:45:14.852360+0000","last_became_active":"2023-03-09T14:44:23.484061+0000","last_became_peered":"2023-03-09T14:44:23.484061+0000","last_unstale":"2023-03-09T14:45:14.852360+0000","last_undegraded":"2023-03-09T14:45:14.852360+0000","last_fullsized":"2023-03-09T14:45:14.852360+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T19:31:42.889122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"87'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.861838+0000","last_change":"2023-03-09T14:44:18.656056+0000","last_active":"2023-03-09T14:45:14.861838+0000","last_peered":"2023-03-09T14:45:14.861838+0000","last_clean":"2023-03-09T14:45:14.861838+0000","last_became_active":"2023-03-09T14:44:18.655709+0000","last_became_peered":"2023-03-09T14:44:18.655709+0000","last_unstale":"2023-03-09T14:45:14.861838+0000","last_undegraded":"2023-03-09T14:45:14.861838+0000","last_fullsized":"2023-03-09T14:45:14.861838+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T19:09:11.447249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"89'5","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.790663+0000","last_change":"2023-03-09T14:44:19.110292+0000","last_active":"2023-03-09T14:45:44.790663+0000","last_peered":"2023-03-09T14:45:44.790663+0000","last_clean":"2023-03-09T14:45:44.790663+0000","last_became_active":"2023-03-09T14:44:19.110006+0000","last_became_peered":"2023-03-09T14:44:19.110006+0000","last_unstale":"2023-03-09T14:45:44.790663+0000","last_undegraded":"2023-03-09T14:45:44.790663+0000","last_fullsized":"2023-03-09T14:45:44.790663+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:36:40.401486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.861551+0000","last_change":"2023-03-09T14:44:23.484004+0000","last_active":"2023-03-09T14:45:14.861551+0000","last_peered":"2023-03-09T14:45:14.861551+0000","last_clean":"2023-03-09T14:45:14.861551+0000","last_became_active":"2023-03-09T14:44:23.483581+0000","last_became_peered":"2023-03-09T14:44:23.483581+0000","last_unstale":"2023-03-09T14:45:14.861551+0000","last_undegraded":"2023-03-09T14:45:14.861551+0000","last_fullsized":"2023-03-09T14:45:14.861551+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:24:25.939838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.860756+0000","last_change":"2023-03-09T14:44:18.661856+0000","last_active":"2023-03-09T14:45:14.860756+0000","last_peered":"2023-03-09T14:45:14.860756+0000","last_clean":"2023-03-09T14:45:14.860756+0000","last_became_active":"2023-03-09T14:44:18.661500+0000","last_became_peered":"2023-03-09T14:44:18.661500+0000","last_unstale":"2023-03-09T14:45:14.860756+0000","last_undegraded":"2023-03-09T14:45:14.860756+0000","last_fullsized":"2023-03-09T14:45:14.860756+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:27:12.181159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"89'4","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.783682+0000","last_change":"2023-03-09T14:44:19.102058+0000","last_active":"2023-03-09T14:45:44.783682+0000","last_peered":"2023-03-09T14:45:44.783682+0000","last_clean":"2023-03-09T14:45:44.783682+0000","last_became_active":"2023-03-09T14:44:19.101762+0000","last_became_peered":"2023-03-09T14:44:19.101762+0000","last_unstale":"2023-03-09T14:45:44.783682+0000","last_undegraded":"2023-03-09T14:45:44.783682+0000","last_fullsized":"2023-03-09T14:45:44.783682+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:39:42.544086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.862303+0000","last_change":"2023-03-09T14:44:23.490237+0000","last_active":"2023-03-09T14:45:14.862303+0000","last_peered":"2023-03-09T14:45:14.862303+0000","last_clean":"2023-03-09T14:45:14.862303+0000","last_became_active":"2023-03-09T14:44:23.489844+0000","last_became_peered":"2023-03-09T14:44:23.489844+0000","last_unstale":"2023-03-09T14:45:14.862303+0000","last_undegraded":"2023-03-09T14:45:14.862303+0000","last_fullsized":"2023-03-09T14:45:14.862303+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T14:50:09.358698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"87'1","reported_seq":11,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:15.865433+0000","last_change":"2023-03-09T14:45:15.864913+0000","last_active":"2023-03-09T14:45:15.865433+0000","last_peered":"2023-03-09T14:45:15.865433+0000","last_clean":"2023-03-09T14:45:15.865433+0000","last_became_active":"2023-03-09T14:45:15.864527+0000","last_became_peered":"2023-03-09T14:45:15.864527+0000","last_unstale":"2023-03-09T14:45:15.865433+0000","last_undegraded":"2023-03-09T14:45:15.865433+0000","last_fullsized":"2023-03-09T14:45:15.865433+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:02:19.489659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,6,1],"acting":[11,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.790944+0000","last_change":"2023-03-09T14:44:19.117584+0000","last_active":"2023-03-09T14:45:44.790944+0000","last_peered":"2023-03-09T14:45:44.790944+0000","last_clean":"2023-03-09T14:45:44.790944+0000","last_became_active":"2023-03-09T14:44:19.117262+0000","last_became_peered":"2023-03-09T14:44:19.117262+0000","last_unstale":"2023-03-09T14:45:44.790944+0000","last_undegraded":"2023-03-09T14:45:44.790944+0000","last_fullsized":"2023-03-09T14:45:44.790944+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T19:19:38.080048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.861827+0000","last_change":"2023-03-09T14:44:23.485769+0000","last_active":"2023-03-09T14:45:14.861827+0000","last_peered":"2023-03-09T14:45:14.861827+0000","last_clean":"2023-03-09T14:45:14.861827+0000","last_became_active":"2023-03-09T14:44:23.485317+0000","last_became_peered":"2023-03-09T14:44:23.485317+0000","last_unstale":"2023-03-09T14:45:14.861827+0000","last_undegraded":"2023-03-09T14:45:14.861827+0000","last_fullsized":"2023-03-09T14:45:14.861827+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:58:35.140512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.786487+0000","last_change":"2023-03-09T14:44:19.112255+0000","last_active":"2023-03-09T14:45:44.786487+0000","last_peered":"2023-03-09T14:45:44.786487+0000","last_clean":"2023-03-09T14:45:44.786487+0000","last_became_active":"2023-03-09T14:44:19.111988+0000","last_became_peered":"2023-03-09T14:44:19.111988+0000","last_unstale":"2023-03-09T14:45:44.786487+0000","last_undegraded":"2023-03-09T14:45:44.786487+0000","last_fullsized":"2023-03-09T14:45:44.786487+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:55:50.423701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'5","reported_seq":103,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:57.669850+0000","last_change":"2023-03-09T14:44:18.665291+0000","last_active":"2023-03-09T14:45:57.669850+0000","last_peered":"2023-03-09T14:45:57.669850+0000","last_clean":"2023-03-09T14:45:57.669850+0000","last_became_active":"2023-03-09T14:44:18.664950+0000","last_became_peered":"2023-03-09T14:44:18.664950+0000","last_unstale":"2023-03-09T14:45:57.669850+0000","last_undegraded":"2023-03-09T14:45:57.669850+0000","last_fullsized":"2023-03-09T14:45:57.669850+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:40:58.619717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":80,"num_read_kb":80,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"87'1","reported_seq":15,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.853567+0000","last_change":"2023-03-09T14:44:23.480872+0000","last_active":"2023-03-09T14:45:14.853567+0000","last_peered":"2023-03-09T14:45:14.853567+0000","last_clean":"2023-03-09T14:45:14.853567+0000","last_became_active":"2023-03-09T14:44:23.480495+0000","last_became_peered":"2023-03-09T14:44:23.480495+0000","last_unstale":"2023-03-09T14:45:14.853567+0000","last_undegraded":"2023-03-09T14:45:14.853567+0000","last_fullsized":"2023-03-09T14:45:14.853567+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:19:01.393034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.797742+0000","last_change":"2023-03-09T14:44:19.102833+0000","last_active":"2023-03-09T14:45:44.797742+0000","last_peered":"2023-03-09T14:45:44.797742+0000","last_clean":"2023-03-09T14:45:44.797742+0000","last_became_active":"2023-03-09T14:44:19.102484+0000","last_became_peered":"2023-03-09T14:44:19.102484+0000","last_unstale":"2023-03-09T14:45:44.797742+0000","last_undegraded":"2023-03-09T14:45:44.797742+0000","last_fullsized":"2023-03-09T14:45:44.797742+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:14:50.002907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"87'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.859537+0000","last_change":"2023-03-09T14:44:18.661603+0000","last_active":"2023-03-09T14:45:14.859537+0000","last_peered":"2023-03-09T14:45:14.859537+0000","last_clean":"2023-03-09T14:45:14.859537+0000","last_became_active":"2023-03-09T14:44:18.661275+0000","last_became_peered":"2023-03-09T14:44:18.661275+0000","last_unstale":"2023-03-09T14:45:14.859537+0000","last_undegraded":"2023-03-09T14:45:14.859537+0000","last_fullsized":"2023-03-09T14:45:14.859537+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:20:31.873778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":378,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.852517+0000","last_change":"2023-03-09T14:43:19.317554+0000","last_active":"2023-03-09T14:45:14.852517+0000","last_peered":"2023-03-09T14:45:14.852517+0000","last_clean":"2023-03-09T14:45:14.852517+0000","last_became_active":"2023-03-09T14:43:19.316912+0000","last_became_peered":"2023-03-09T14:43:19.316912+0000","last_unstale":"2023-03-09T14:45:14.852517+0000","last_undegraded":"2023-03-09T14:45:14.852517+0000","last_fullsized":"2023-03-09T14:45:14.852517+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:40:23.197173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:40:23.197173+0000","last_clean_scrub_stamp":"2023-03-09T14:40:23.197173+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T20:21:17.143331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.852927+0000","last_change":"2023-03-09T14:44:23.487738+0000","last_active":"2023-03-09T14:45:14.852927+0000","last_peered":"2023-03-09T14:45:14.852927+0000","last_clean":"2023-03-09T14:45:14.852927+0000","last_became_active":"2023-03-09T14:44:23.487193+0000","last_became_peered":"2023-03-09T14:44:23.487193+0000","last_unstale":"2023-03-09T14:45:14.852927+0000","last_undegraded":"2023-03-09T14:45:14.852927+0000","last_fullsized":"2023-03-09T14:45:14.852927+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:48:08.789071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"89'5","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.784152+0000","last_change":"2023-03-09T14:44:19.112576+0000","last_active":"2023-03-09T14:45:44.784152+0000","last_peered":"2023-03-09T14:45:44.784152+0000","last_clean":"2023-03-09T14:45:44.784152+0000","last_became_active":"2023-03-09T14:44:19.112231+0000","last_became_peered":"2023-03-09T14:44:19.112231+0000","last_unstale":"2023-03-09T14:45:44.784152+0000","last_undegraded":"2023-03-09T14:45:44.784152+0000","last_fullsized":"2023-03-09T14:45:44.784152+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:25:21.918088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"87'3","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.852608+0000","last_change":"2023-03-09T14:44:18.655478+0000","last_active":"2023-03-09T14:45:14.852608+0000","last_peered":"2023-03-09T14:45:14.852608+0000","last_clean":"2023-03-09T14:45:14.852608+0000","last_became_active":"2023-03-09T14:44:18.655180+0000","last_became_peered":"2023-03-09T14:44:18.655180+0000","last_unstale":"2023-03-09T14:45:14.852608+0000","last_undegraded":"2023-03-09T14:45:14.852608+0000","last_fullsized":"2023-03-09T14:45:14.852608+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:42:47.926376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.856627+0000","last_change":"2023-03-09T14:44:23.487052+0000","last_active":"2023-03-09T14:45:14.856627+0000","last_peered":"2023-03-09T14:45:14.856627+0000","last_clean":"2023-03-09T14:45:14.856627+0000","last_became_active":"2023-03-09T14:44:23.486603+0000","last_became_peered":"2023-03-09T14:44:23.486603+0000","last_unstale":"2023-03-09T14:45:14.856627+0000","last_undegraded":"2023-03-09T14:45:14.856627+0000","last_fullsized":"2023-03-09T14:45:14.856627+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:28:25.375052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"87'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.861722+0000","last_change":"2023-03-09T14:44:18.666400+0000","last_active":"2023-03-09T14:45:14.861722+0000","last_peered":"2023-03-09T14:45:14.861722+0000","last_clean":"2023-03-09T14:45:14.861722+0000","last_became_active":"2023-03-09T14:44:18.666045+0000","last_became_peered":"2023-03-09T14:44:18.666045+0000","last_unstale":"2023-03-09T14:45:14.861722+0000","last_undegraded":"2023-03-09T14:45:14.861722+0000","last_fullsized":"2023-03-09T14:45:14.861722+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:30:48.640124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.784135+0000","last_change":"2023-03-09T14:44:19.103657+0000","last_active":"2023-03-09T14:45:44.784135+0000","last_peered":"2023-03-09T14:45:44.784135+0000","last_clean":"2023-03-09T14:45:44.784135+0000","last_became_active":"2023-03-09T14:44:19.103364+0000","last_became_peered":"2023-03-09T14:44:19.103364+0000","last_unstale":"2023-03-09T14:45:44.784135+0000","last_undegraded":"2023-03-09T14:45:44.784135+0000","last_fullsized":"2023-03-09T14:45:44.784135+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:23:23.738554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.864043+0000","last_change":"2023-03-09T14:44:23.483487+0000","last_active":"2023-03-09T14:45:14.864043+0000","last_peered":"2023-03-09T14:45:14.864043+0000","last_clean":"2023-03-09T14:45:14.864043+0000","last_became_active":"2023-03-09T14:44:23.483050+0000","last_became_peered":"2023-03-09T14:44:23.483050+0000","last_unstale":"2023-03-09T14:45:14.864043+0000","last_undegraded":"2023-03-09T14:45:14.864043+0000","last_fullsized":"2023-03-09T14:45:14.864043+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:38:34.129355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'7","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:48.585960+0000","last_change":"2023-03-09T14:44:18.661944+0000","last_active":"2023-03-09T14:45:48.585960+0000","last_peered":"2023-03-09T14:45:48.585960+0000","last_clean":"2023-03-09T14:45:48.585960+0000","last_became_active":"2023-03-09T14:44:18.661575+0000","last_became_peered":"2023-03-09T14:44:18.661575+0000","last_unstale":"2023-03-09T14:45:48.585960+0000","last_undegraded":"2023-03-09T14:45:48.585960+0000","last_fullsized":"2023-03-09T14:45:48.585960+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T01:12:53.637866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"89'5","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.787402+0000","last_change":"2023-03-09T14:44:19.103807+0000","last_active":"2023-03-09T14:45:44.787402+0000","last_peered":"2023-03-09T14:45:44.787402+0000","last_clean":"2023-03-09T14:45:44.787402+0000","last_became_active":"2023-03-09T14:44:19.103480+0000","last_became_peered":"2023-03-09T14:44:19.103480+0000","last_unstale":"2023-03-09T14:45:44.787402+0000","last_undegraded":"2023-03-09T14:45:44.787402+0000","last_fullsized":"2023-03-09T14:45:44.787402+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:29:54.629265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.856235+0000","last_change":"2023-03-09T14:44:23.486789+0000","last_active":"2023-03-09T14:45:14.856235+0000","last_peered":"2023-03-09T14:45:14.856235+0000","last_clean":"2023-03-09T14:45:14.856235+0000","last_became_active":"2023-03-09T14:44:23.486367+0000","last_became_peered":"2023-03-09T14:44:23.486367+0000","last_unstale":"2023-03-09T14:45:14.856235+0000","last_undegraded":"2023-03-09T14:45:14.856235+0000","last_fullsized":"2023-03-09T14:45:14.856235+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:13:13.081388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"87'4","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.860080+0000","last_change":"2023-03-09T14:44:18.659492+0000","last_active":"2023-03-09T14:45:14.860080+0000","last_peered":"2023-03-09T14:45:14.860080+0000","last_clean":"2023-03-09T14:45:14.860080+0000","last_became_active":"2023-03-09T14:44:18.659187+0000","last_became_peered":"2023-03-09T14:44:18.659187+0000","last_unstale":"2023-03-09T14:45:14.860080+0000","last_undegraded":"2023-03-09T14:45:14.860080+0000","last_fullsized":"2023-03-09T14:45:14.860080+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:44:31.342785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.789313+0000","last_change":"2023-03-09T14:44:19.112841+0000","last_active":"2023-03-09T14:45:44.789313+0000","last_peered":"2023-03-09T14:45:44.789313+0000","last_clean":"2023-03-09T14:45:44.789313+0000","last_became_active":"2023-03-09T14:44:19.112515+0000","last_became_peered":"2023-03-09T14:44:19.112515+0000","last_unstale":"2023-03-09T14:45:44.789313+0000","last_undegraded":"2023-03-09T14:45:44.789313+0000","last_fullsized":"2023-03-09T14:45:44.789313+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T19:41:04.065529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.859850+0000","last_change":"2023-03-09T14:44:23.481992+0000","last_active":"2023-03-09T14:45:14.859850+0000","last_peered":"2023-03-09T14:45:14.859850+0000","last_clean":"2023-03-09T14:45:14.859850+0000","last_became_active":"2023-03-09T14:44:23.481508+0000","last_became_peered":"2023-03-09T14:44:23.481508+0000","last_unstale":"2023-03-09T14:45:14.859850+0000","last_undegraded":"2023-03-09T14:45:14.859850+0000","last_fullsized":"2023-03-09T14:45:14.859850+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:46:48.683402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"87'2","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.856199+0000","last_change":"2023-03-09T14:44:18.663133+0000","last_active":"2023-03-09T14:45:14.856199+0000","last_peered":"2023-03-09T14:45:14.856199+0000","last_clean":"2023-03-09T14:45:14.856199+0000","last_became_active":"2023-03-09T14:44:18.662809+0000","last_became_peered":"2023-03-09T14:44:18.662809+0000","last_unstale":"2023-03-09T14:45:14.856199+0000","last_undegraded":"2023-03-09T14:45:14.856199+0000","last_fullsized":"2023-03-09T14:45:14.856199+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:32:16.180642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"89'2","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.784561+0000","last_change":"2023-03-09T14:44:19.108566+0000","last_active":"2023-03-09T14:45:44.784561+0000","last_peered":"2023-03-09T14:45:44.784561+0000","last_clean":"2023-03-09T14:45:44.784561+0000","last_became_active":"2023-03-09T14:44:19.108217+0000","last_became_peered":"2023-03-09T14:44:19.108217+0000","last_unstale":"2023-03-09T14:45:44.784561+0000","last_undegraded":"2023-03-09T14:45:44.784561+0000","last_fullsized":"2023-03-09T14:45:44.784561+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:03:28.434395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.860862+0000","last_change":"2023-03-09T14:44:23.488640+0000","last_active":"2023-03-09T14:45:14.860862+0000","last_peered":"2023-03-09T14:45:14.860862+0000","last_clean":"2023-03-09T14:45:14.860862+0000","last_became_active":"2023-03-09T14:44:23.488178+0000","last_became_peered":"2023-03-09T14:44:23.488178+0000","last_unstale":"2023-03-09T14:45:14.860862+0000","last_undegraded":"2023-03-09T14:45:14.860862+0000","last_fullsized":"2023-03-09T14:45:14.860862+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:33:49.908974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"87'3","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.864673+0000","last_change":"2023-03-09T14:44:18.662925+0000","last_active":"2023-03-09T14:45:14.864673+0000","last_peered":"2023-03-09T14:45:14.864673+0000","last_clean":"2023-03-09T14:45:14.864673+0000","last_became_active":"2023-03-09T14:44:18.662354+0000","last_became_peered":"2023-03-09T14:44:18.662354+0000","last_unstale":"2023-03-09T14:45:14.864673+0000","last_undegraded":"2023-03-09T14:45:14.864673+0000","last_fullsized":"2023-03-09T14:45:14.864673+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T01:00:36.602938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.790399+0000","last_change":"2023-03-09T14:44:19.106465+0000","last_active":"2023-03-09T14:45:44.790399+0000","last_peered":"2023-03-09T14:45:44.790399+0000","last_clean":"2023-03-09T14:45:44.790399+0000","last_became_active":"2023-03-09T14:44:19.106001+0000","last_became_peered":"2023-03-09T14:44:19.106001+0000","last_unstale":"2023-03-09T14:45:44.790399+0000","last_undegraded":"2023-03-09T14:45:44.790399+0000","last_fullsized":"2023-03-09T14:45:44.790399+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:22:09.149291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"87'1","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:15.873389+0000","last_change":"2023-03-09T14:45:15.873389+0000","last_active":"2023-03-09T14:45:15.873389+0000","last_peered":"2023-03-09T14:45:15.873389+0000","last_clean":"2023-03-09T14:45:15.873389+0000","last_became_active":"2023-03-09T14:45:15.863408+0000","last_became_peered":"2023-03-09T14:45:15.863408+0000","last_unstale":"2023-03-09T14:45:15.873389+0000","last_undegraded":"2023-03-09T14:45:15.873389+0000","last_fullsized":"2023-03-09T14:45:15.873389+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T16:33:12.565135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":123,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,7,8],"acting":[4,10,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"87'1","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.856858+0000","last_change":"2023-03-09T14:44:18.664744+0000","last_active":"2023-03-09T14:45:14.856858+0000","last_peered":"2023-03-09T14:45:14.856858+0000","last_clean":"2023-03-09T14:45:14.856858+0000","last_became_active":"2023-03-09T14:44:18.664443+0000","last_became_peered":"2023-03-09T14:44:18.664443+0000","last_unstale":"2023-03-09T14:45:14.856858+0000","last_undegraded":"2023-03-09T14:45:14.856858+0000","last_fullsized":"2023-03-09T14:45:14.856858+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:11:13.836270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.790126+0000","last_change":"2023-03-09T14:44:19.115102+0000","last_active":"2023-03-09T14:45:44.790126+0000","last_peered":"2023-03-09T14:45:44.790126+0000","last_clean":"2023-03-09T14:45:44.790126+0000","last_became_active":"2023-03-09T14:44:19.114565+0000","last_became_peered":"2023-03-09T14:44:19.114565+0000","last_unstale":"2023-03-09T14:45:44.790126+0000","last_undegraded":"2023-03-09T14:45:44.790126+0000","last_fullsized":"2023-03-09T14:45:44.790126+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:39:11.930231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.859836+0000","last_change":"2023-03-09T14:44:23.482500+0000","last_active":"2023-03-09T14:45:14.859836+0000","last_peered":"2023-03-09T14:45:14.859836+0000","last_clean":"2023-03-09T14:45:14.859836+0000","last_became_active":"2023-03-09T14:44:23.482217+0000","last_became_peered":"2023-03-09T14:44:23.482217+0000","last_unstale":"2023-03-09T14:45:14.859836+0000","last_undegraded":"2023-03-09T14:45:14.859836+0000","last_fullsized":"2023-03-09T14:45:14.859836+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T14:52:44.275037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"87'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.860725+0000","last_change":"2023-03-09T14:44:18.663824+0000","last_active":"2023-03-09T14:45:14.860725+0000","last_peered":"2023-03-09T14:45:14.860725+0000","last_clean":"2023-03-09T14:45:14.860725+0000","last_became_active":"2023-03-09T14:44:18.663379+0000","last_became_peered":"2023-03-09T14:44:18.663379+0000","last_unstale":"2023-03-09T14:45:14.860725+0000","last_undegraded":"2023-03-09T14:45:14.860725+0000","last_fullsized":"2023-03-09T14:45:14.860725+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T22:03:04.031061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"89'5","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.798601+0000","last_change":"2023-03-09T14:44:19.105765+0000","last_active":"2023-03-09T14:45:44.798601+0000","last_peered":"2023-03-09T14:45:44.798601+0000","last_clean":"2023-03-09T14:45:44.798601+0000","last_became_active":"2023-03-09T14:44:19.105430+0000","last_became_peered":"2023-03-09T14:44:19.105430+0000","last_unstale":"2023-03-09T14:45:44.798601+0000","last_undegraded":"2023-03-09T14:45:44.798601+0000","last_fullsized":"2023-03-09T14:45:44.798601+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T20:31:40.106798+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.856901+0000","last_change":"2023-03-09T14:44:23.482079+0000","last_active":"2023-03-09T14:45:14.856901+0000","last_peered":"2023-03-09T14:45:14.856901+0000","last_clean":"2023-03-09T14:45:14.856901+0000","last_became_active":"2023-03-09T14:44:23.481620+0000","last_became_peered":"2023-03-09T14:44:23.481620+0000","last_unstale":"2023-03-09T14:45:14.856901+0000","last_undegraded":"2023-03-09T14:45:14.856901+0000","last_fullsized":"2023-03-09T14:45:14.856901+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:47:53.589111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"87'3","reported_seq":17,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:15.869980+0000","last_change":"2023-03-09T14:45:15.869485+0000","last_active":"2023-03-09T14:45:15.869980+0000","last_peered":"2023-03-09T14:45:15.869980+0000","last_clean":"2023-03-09T14:45:15.869980+0000","last_became_active":"2023-03-09T14:45:15.861428+0000","last_became_peered":"2023-03-09T14:45:15.861428+0000","last_unstale":"2023-03-09T14:45:15.869980+0000","last_undegraded":"2023-03-09T14:45:15.869980+0000","last_fullsized":"2023-03-09T14:45:15.869980+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:43:30.309431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":566,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":566,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.799201+0000","last_change":"2023-03-09T14:44:19.105947+0000","last_active":"2023-03-09T14:45:44.799201+0000","last_peered":"2023-03-09T14:45:44.799201+0000","last_clean":"2023-03-09T14:45:44.799201+0000","last_became_active":"2023-03-09T14:44:19.105669+0000","last_became_peered":"2023-03-09T14:44:19.105669+0000","last_unstale":"2023-03-09T14:45:44.799201+0000","last_undegraded":"2023-03-09T14:45:44.799201+0000","last_fullsized":"2023-03-09T14:45:44.799201+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:29:58.615481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.863833+0000","last_change":"2023-03-09T14:44:23.483237+0000","last_active":"2023-03-09T14:45:14.863833+0000","last_peered":"2023-03-09T14:45:14.863833+0000","last_clean":"2023-03-09T14:45:14.863833+0000","last_became_active":"2023-03-09T14:44:23.482831+0000","last_became_peered":"2023-03-09T14:44:23.482831+0000","last_unstale":"2023-03-09T14:45:14.863833+0000","last_undegraded":"2023-03-09T14:45:14.863833+0000","last_fullsized":"2023-03-09T14:45:14.863833+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:36:40.775581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"82'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:15.867430+0000","last_change":"2023-03-09T14:45:15.867049+0000","last_active":"2023-03-09T14:45:15.867430+0000","last_peered":"2023-03-09T14:45:15.867430+0000","last_clean":"2023-03-09T14:45:15.867430+0000","last_became_active":"2023-03-09T14:45:15.863218+0000","last_became_peered":"2023-03-09T14:45:15.863218+0000","last_unstale":"2023-03-09T14:45:15.867430+0000","last_undegraded":"2023-03-09T14:45:15.867430+0000","last_fullsized":"2023-03-09T14:45:15.867430+0000","mapping_epoch":88,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":89,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:38:21.800533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":70,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.c","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.787235+0000","last_change":"2023-03-09T14:44:19.107146+0000","last_active":"2023-03-09T14:45:44.787235+0000","last_peered":"2023-03-09T14:45:44.787235+0000","last_clean":"2023-03-09T14:45:44.787235+0000","last_became_active":"2023-03-09T14:44:19.106834+0000","last_became_peered":"2023-03-09T14:44:19.106834+0000","last_unstale":"2023-03-09T14:45:44.787235+0000","last_undegraded":"2023-03-09T14:45:44.787235+0000","last_fullsized":"2023-03-09T14:45:44.787235+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T20:15:20.142449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.852185+0000","last_change":"2023-03-09T14:44:23.482811+0000","last_active":"2023-03-09T14:45:14.852185+0000","last_peered":"2023-03-09T14:45:14.852185+0000","last_clean":"2023-03-09T14:45:14.852185+0000","last_became_active":"2023-03-09T14:44:23.482371+0000","last_became_peered":"2023-03-09T14:44:23.482371+0000","last_unstale":"2023-03-09T14:45:14.852185+0000","last_undegraded":"2023-03-09T14:45:14.852185+0000","last_fullsized":"2023-03-09T14:45:14.852185+0000","mapping_epoch":85,"log_start":"0'0","ondisk_log_start":"0'0","created":85,"last_epoch_clean":86,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:22.453277+0000","last_clean_scrub_stamp":"2023-03-09T14:44:22.453277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T01:33:14.442810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"82'2","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2023-03-09T14:45:14.861583+0000","last_change":"2023-03-09T14:44:18.661569+0000","last_active":"2023-03-09T14:45:14.861583+0000","last_peered":"2023-03-09T14:45:14.861583+0000","last_clean":"2023-03-09T14:45:14.861583+0000","last_became_active":"2023-03-09T14:44:18.661266+0000","last_became_peered":"2023-03-09T14:44:18.661266+0000","last_unstale":"2023-03-09T14:45:14.861583+0000","last_undegraded":"2023-03-09T14:45:14.861583+0000","last_fullsized":"2023-03-09T14:45:14.861583+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:17.633832+0000","last_clean_scrub_stamp":"2023-03-09T14:44:17.633832+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T15:47:43.066180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.786924+0000","last_change":"2023-03-09T14:44:19.106611+0000","last_active":"2023-03-09T14:45:44.786924+0000","last_peered":"2023-03-09T14:45:44.786924+0000","last_clean":"2023-03-09T14:45:44.786924+0000","last_became_active":"2023-03-09T14:44:19.106324+0000","last_became_peered":"2023-03-09T14:44:19.106324+0000","last_unstale":"2023-03-09T14:45:44.786924+0000","last_undegraded":"2023-03-09T14:45:44.786924+0000","last_fullsized":"2023-03-09T14:45:44.786924+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:50:46.561502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"89'9","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.787174+0000","last_change":"2023-03-09T14:44:19.106292+0000","last_active":"2023-03-09T14:45:44.787174+0000","last_peered":"2023-03-09T14:45:44.787174+0000","last_clean":"2023-03-09T14:45:44.787174+0000","last_became_active":"2023-03-09T14:44:19.105607+0000","last_became_peered":"2023-03-09T14:44:19.105607+0000","last_unstale":"2023-03-09T14:45:44.787174+0000","last_undegraded":"2023-03-09T14:45:44.787174+0000","last_fullsized":"2023-03-09T14:45:44.787174+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T19:11:28.125433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"89'12","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.786929+0000","last_change":"2023-03-09T14:44:19.103423+0000","last_active":"2023-03-09T14:45:44.786929+0000","last_peered":"2023-03-09T14:45:44.786929+0000","last_clean":"2023-03-09T14:45:44.786929+0000","last_became_active":"2023-03-09T14:44:19.102918+0000","last_became_peered":"2023-03-09T14:44:19.102918+0000","last_unstale":"2023-03-09T14:45:44.786929+0000","last_undegraded":"2023-03-09T14:45:44.786929+0000","last_fullsized":"2023-03-09T14:45:44.786929+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T18:47:52.921642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.790566+0000","last_change":"2023-03-09T14:44:19.113511+0000","last_active":"2023-03-09T14:45:44.790566+0000","last_peered":"2023-03-09T14:45:44.790566+0000","last_clean":"2023-03-09T14:45:44.790566+0000","last_became_active":"2023-03-09T14:44:19.113224+0000","last_became_peered":"2023-03-09T14:44:19.113224+0000","last_unstale":"2023-03-09T14:45:44.790566+0000","last_undegraded":"2023-03-09T14:45:44.790566+0000","last_fullsized":"2023-03-09T14:45:44.790566+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T02:03:28.069616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"89'7","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.787373+0000","last_change":"2023-03-09T14:44:19.116174+0000","last_active":"2023-03-09T14:45:44.787373+0000","last_peered":"2023-03-09T14:45:44.787373+0000","last_clean":"2023-03-09T14:45:44.787373+0000","last_became_active":"2023-03-09T14:44:19.115813+0000","last_became_peered":"2023-03-09T14:44:19.115813+0000","last_unstale":"2023-03-09T14:45:44.787373+0000","last_undegraded":"2023-03-09T14:45:44.787373+0000","last_fullsized":"2023-03-09T14:45:44.787373+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:28:49.573771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"89'4","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.790565+0000","last_change":"2023-03-09T14:44:19.115287+0000","last_active":"2023-03-09T14:45:44.790565+0000","last_peered":"2023-03-09T14:45:44.790565+0000","last_clean":"2023-03-09T14:45:44.790565+0000","last_became_active":"2023-03-09T14:44:19.113458+0000","last_became_peered":"2023-03-09T14:44:19.113458+0000","last_unstale":"2023-03-09T14:45:44.790565+0000","last_undegraded":"2023-03-09T14:45:44.790565+0000","last_fullsized":"2023-03-09T14:45:44.790565+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T23:43:26.583237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.787054+0000","last_change":"2023-03-09T14:44:19.107308+0000","last_active":"2023-03-09T14:45:44.787054+0000","last_peered":"2023-03-09T14:45:44.787054+0000","last_clean":"2023-03-09T14:45:44.787054+0000","last_became_active":"2023-03-09T14:44:19.107005+0000","last_became_peered":"2023-03-09T14:44:19.107005+0000","last_unstale":"2023-03-09T14:45:44.787054+0000","last_undegraded":"2023-03-09T14:45:44.787054+0000","last_fullsized":"2023-03-09T14:45:44.787054+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:48:34.886664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"89'15","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.800402+0000","last_change":"2023-03-09T14:44:19.098454+0000","last_active":"2023-03-09T14:45:44.800402+0000","last_peered":"2023-03-09T14:45:44.800402+0000","last_clean":"2023-03-09T14:45:44.800402+0000","last_became_active":"2023-03-09T14:44:19.098146+0000","last_became_peered":"2023-03-09T14:44:19.098146+0000","last_unstale":"2023-03-09T14:45:44.800402+0000","last_undegraded":"2023-03-09T14:45:44.800402+0000","last_fullsized":"2023-03-09T14:45:44.800402+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:11:40.475674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"89'2","reported_seq":19,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.790577+0000","last_change":"2023-03-09T14:44:19.095947+0000","last_active":"2023-03-09T14:45:44.790577+0000","last_peered":"2023-03-09T14:45:44.790577+0000","last_clean":"2023-03-09T14:45:44.790577+0000","last_became_active":"2023-03-09T14:44:19.095640+0000","last_became_peered":"2023-03-09T14:44:19.095640+0000","last_unstale":"2023-03-09T14:45:44.790577+0000","last_undegraded":"2023-03-09T14:45:44.790577+0000","last_fullsized":"2023-03-09T14:45:44.790577+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T02:04:35.384016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"89'6","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.781900+0000","last_change":"2023-03-09T14:44:19.113927+0000","last_active":"2023-03-09T14:45:44.781900+0000","last_peered":"2023-03-09T14:45:44.781900+0000","last_clean":"2023-03-09T14:45:44.781900+0000","last_became_active":"2023-03-09T14:44:19.113163+0000","last_became_peered":"2023-03-09T14:44:19.113163+0000","last_unstale":"2023-03-09T14:45:44.781900+0000","last_undegraded":"2023-03-09T14:45:44.781900+0000","last_fullsized":"2023-03-09T14:45:44.781900+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T01:35:35.801317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.787768+0000","last_change":"2023-03-09T14:44:19.098415+0000","last_active":"2023-03-09T14:45:44.787768+0000","last_peered":"2023-03-09T14:45:44.787768+0000","last_clean":"2023-03-09T14:45:44.787768+0000","last_became_active":"2023-03-09T14:44:19.097600+0000","last_became_peered":"2023-03-09T14:44:19.097600+0000","last_unstale":"2023-03-09T14:45:44.787768+0000","last_undegraded":"2023-03-09T14:45:44.787768+0000","last_fullsized":"2023-03-09T14:45:44.787768+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-11T00:15:08.552402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"89'10","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.795312+0000","last_change":"2023-03-09T14:44:19.106386+0000","last_active":"2023-03-09T14:45:44.795312+0000","last_peered":"2023-03-09T14:45:44.795312+0000","last_clean":"2023-03-09T14:45:44.795312+0000","last_became_active":"2023-03-09T14:44:19.106112+0000","last_became_peered":"2023-03-09T14:44:19.106112+0000","last_unstale":"2023-03-09T14:45:44.795312+0000","last_undegraded":"2023-03-09T14:45:44.795312+0000","last_fullsized":"2023-03-09T14:45:44.795312+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T20:27:50.992168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"89'11","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.788971+0000","last_change":"2023-03-09T14:44:19.114534+0000","last_active":"2023-03-09T14:45:44.788971+0000","last_peered":"2023-03-09T14:45:44.788971+0000","last_clean":"2023-03-09T14:45:44.788971+0000","last_became_active":"2023-03-09T14:44:19.114226+0000","last_became_peered":"2023-03-09T14:44:19.114226+0000","last_unstale":"2023-03-09T14:45:44.788971+0000","last_undegraded":"2023-03-09T14:45:44.788971+0000","last_fullsized":"2023-03-09T14:45:44.788971+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T21:22:38.453116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"89'8","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2023-03-09T14:45:44.790943+0000","last_change":"2023-03-09T14:44:19.099195+0000","last_active":"2023-03-09T14:45:44.790943+0000","last_peered":"2023-03-09T14:45:44.790943+0000","last_clean":"2023-03-09T14:45:44.790943+0000","last_became_active":"2023-03-09T14:44:19.098892+0000","last_became_peered":"2023-03-09T14:44:19.098892+0000","last_unstale":"2023-03-09T14:45:44.790943+0000","last_undegraded":"2023-03-09T14:45:44.790943+0000","last_fullsized":"2023-03-09T14:45:44.790943+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":81,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-03-09T14:44:18.641620+0000","last_clean_scrub_stamp":"2023-03-09T14:44:18.641620+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-03-10T17:29:24.858200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":246,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":123,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4751360,"data_stored":4722944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":65932,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":517,"num_read_kb":327,"num_write":164,"num_write_kb":117,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":1914,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":413696,"data_stored":193508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":144,"ondisk_log_size":144,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":78,"seq":335007449120,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6236,"kb_used_data":980,"kb_used_omap":0,"kb_used_meta":5248,"kb_avail":93738916,"statfs":{"total":95995035648,"available":95988649984,"internally_reserved":0,"allocated":1003520,"data_stored":490897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5373952},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:45:43 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.58299999999999996,"15min":0.58499999999999996},"min":{"1min":0.433,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.57899999999999996,"15min":0.58399999999999996},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.76000000000000001,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.70499999999999996}]},{"osd":1,"last update":"Thu Mar 9 14:45:43 2023","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.59999999999999998,"15min":0.58799999999999997},"min":{"1min":0.47399999999999998,"5min":0.378,"15min":0.378},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.40200000000000002,"5min":0.372,"15min":0.372},"max":{"1min":0.79600000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.76800000000000002}]},{"osd":2,"last update":"Thu Mar 9 14:45:43 2023","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.504,"15min":0.505},"min":{"1min":0.38,"5min":0.313,"15min":0.313},"max":{"1min":0.76000000000000001,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.51300000000000001,"15min":0.503},"min":{"1min":0.33500000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.79300000000000004,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.45300000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78200000000000003}]},{"osd":4,"last update":"Thu Mar 9 14:45:43 2023","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.66900000000000004,"15min":0.65700000000000003},"min":{"1min":0.50700000000000001,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.91200000000000003,"5min":1.113,"15min":1.113},"last":0.89400000000000002},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.64800000000000002,"15min":0.64400000000000002},"min":{"1min":0.56299999999999994,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.85099999999999998,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.79400000000000004}]},{"osd":5,"last update":"Thu Mar 9 14:45:43 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.55700000000000005,"15min":0.54700000000000004},"min":{"1min":0.42399999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.63,"15min":0.61399999999999999},"min":{"1min":0.41599999999999998,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.94099999999999995,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.60499999999999998}]},{"osd":6,"last update":"Thu Mar 9 14:45:43 2023","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.67700000000000005,"15min":0.66900000000000004},"min":{"1min":0.45500000000000002,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":1.7829999999999999,"5min":1.7829999999999999,"15min":1.7829999999999999},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.752,"5min":0.70299999999999996,"15min":0.69499999999999995},"min":{"1min":0.53200000000000003,"5min":0.497,"15min":0.497},"max":{"1min":1.423,"5min":1.423,"15min":1.423},"last":0.63200000000000001}]},{"osd":7,"last update":"Thu Mar 9 14:45:43 2023","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.73499999999999999,"15min":0.71999999999999997},"min":{"1min":0.56299999999999994,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.0189999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.73599999999999999,"15min":0.73099999999999998},"min":{"1min":0.61299999999999999,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":0.93999999999999995,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.81599999999999995}]},{"osd":8,"last update":"Thu Mar 9 14:45:21 2023","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.91800000000000004}]},{"osd":9,"last update":"Thu Mar 9 14:45:43 2023","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.79400000000000004,"15min":0.78300000000000003},"min":{"1min":0.58699999999999997,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.548,"5min":1.548,"15min":1.548},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.872,"5min":0.78100000000000003,"15min":0.76600000000000001},"min":{"1min":0.54600000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.996,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.93999999999999995}]},{"osd":10,"last update":"Thu Mar 9 14:45:43 2023","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.755,"15min":0.749},"min":{"1min":0.64300000000000002,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":0.97499999999999998,"5min":1.01,"15min":1.01},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.79800000000000004,"15min":0.79400000000000004},"min":{"1min":0.66600000000000004,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.012,"5min":1.431,"15min":1.431},"last":0.753}]}]},{"osd":10,"up_from":70,"seq":300647710755,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7224,"kb_used_data":1648,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93737928,"statfs":{"total":95995035648,"available":95987638272,"internally_reserved":0,"allocated":1687552,"data_stored":1144230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:45:19 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.502}]},{"osd":1,"last update":"Thu Mar 9 14:45:19 2023","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.46700000000000003,"15min":0.46999999999999997},"min":{"1min":0.30499999999999999,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.56699999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.47799999999999998,"15min":0.48199999999999998},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.61099999999999999,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.47499999999999998}]},{"osd":2,"last update":"Thu Mar 9 14:45:19 2023","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.63600000000000001,"15min":0.63900000000000001},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.753,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.751},{"interface":"front","average":{"1min":0.628,"5min":0.623,"15min":0.622},"min":{"1min":0.46400000000000002,"5min":0.441,"15min":0.441},"max":{"1min":0.82199999999999995,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.69599999999999995}]},{"osd":3,"last update":"Thu Mar 9 14:45:19 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.58399999999999996,"15min":0.58899999999999997},"min":{"1min":0.38300000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.74399999999999999,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.64200000000000002,"15min":0.64800000000000002},"min":{"1min":0.46600000000000003,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.76100000000000001,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.63500000000000001}]},{"osd":4,"last update":"Thu Mar 9 14:45:19 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.59099999999999997,"15min":0.59299999999999997},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.78200000000000003,"5min":1.1719999999999999,"15min":1.1719999999999999},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.54500000000000004,"15min":0.54900000000000004},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.77500000000000002,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.62}]},{"osd":5,"last update":"Thu Mar 9 14:45:19 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.504,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.92700000000000005,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.90300000000000002,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.73699999999999999}]},{"osd":6,"last update":"Thu Mar 9 14:45:19 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.68300000000000005,"15min":0.68899999999999995},"min":{"1min":0.48799999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.86499999999999999,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.71099999999999997,"15min":0.71599999999999997},"min":{"1min":0.499,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":0.82699999999999996,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.72299999999999998}]},{"osd":7,"last update":"Thu Mar 9 14:44:17 2023","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":1.327,"5min":1.327,"15min":1.327},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.048,"5min":1.048,"15min":1.048},"last":0.53800000000000003}]},{"osd":8,"last update":"Thu Mar 9 14:45:19 2023","interfaces":[{"interface":"back","average":{"1min":0.752,"5min":0.76000000000000001,"15min":0.76200000000000001},"min":{"1min":0.54900000000000004,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":0.91000000000000003,"5min":1.246,"15min":1.246},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.77300000000000002,"15min":0.78200000000000003},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.95499999999999996,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.871}]},{"osd":9,"last update":"Thu Mar 9 14:45:19 2023","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.77600000000000002,"15min":0.78400000000000003},"min":{"1min":0.54200000000000004,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.88100000000000001,"5min":1.298,"15min":1.298},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.77700000000000002,"15min":0.79200000000000004},"min":{"1min":0.47499999999999998,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.89200000000000002,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":0.71199999999999997}]},{"osd":11,"last update":"Thu Mar 9 14:45:42 2023","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.755,"15min":0.753},"min":{"1min":0.68799999999999994,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.74099999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"min":{"1min":0.628,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.88300000000000001}]}]},{"osd":9,"up_from":65,"seq":279172874280,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6376,"kb_used_data":992,"kb_used_omap":0,"kb_used_meta":5376,"kb_avail":93738776,"statfs":{"total":95995035648,"available":95988506624,"internally_reserved":0,"allocated":1015808,"data_stored":491063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5505024},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:45:00 2023","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.46200000000000002,"15min":0.47399999999999998},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.66300000000000003,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.49399999999999999,"15min":0.50700000000000001},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.63700000000000001,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.36699999999999999}]},{"osd":1,"last update":"Thu Mar 9 14:45:00 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.626,"15min":0.63600000000000001},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.008,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.63700000000000001,"15min":0.65000000000000002},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.79900000000000004,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.56999999999999995}]},{"osd":2,"last update":"Thu Mar 9 14:45:00 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.64600000000000002,"15min":0.65800000000000003},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.66100000000000003,"15min":0.67400000000000004},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.88300000000000001,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.497}]},{"osd":3,"last update":"Thu Mar 9 14:45:00 2023","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.56999999999999995,"15min":0.57999999999999996},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":1.3220000000000001,"5min":1.3220000000000001,"15min":1.3220000000000001},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.59499999999999997,"15min":0.61499999999999999},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.85099999999999998,"5min":1.016,"15min":1.016},"last":0.38}]},{"osd":4,"last update":"Thu Mar 9 14:45:00 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.67900000000000005,"15min":0.69699999999999995},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.80400000000000005,"5min":1.04,"15min":1.04},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.621,"5min":0.71699999999999997,"15min":0.73299999999999998},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.93899999999999995,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.58099999999999996}]},{"osd":5,"last update":"Thu Mar 9 14:45:00 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.73299999999999998,"15min":0.745},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.98599999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.74099999999999999,"15min":0.754},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.0369999999999999,"5min":1.069,"15min":1.069},"last":0.52300000000000002}]},{"osd":6,"last update":"Thu Mar 9 14:43:55 2023","interfaces":[{"interface":"back","average":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.115,"5min":1.115,"15min":1.115},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.749,"5min":0.749,"15min":0.749},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.085,"5min":1.085,"15min":1.085},"last":0.623}]},{"osd":7,"last update":"Thu Mar 9 14:45:00 2023","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.79800000000000004,"15min":0.82599999999999996},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.0229999999999999,"5min":1.1180000000000001,"15min":1.1180000000000001},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.80900000000000005,"15min":0.83099999999999996},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.99299999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.71499999999999997}]},{"osd":8,"last update":"Thu Mar 9 14:45:00 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.80200000000000005,"15min":0.81599999999999995},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.032,"5min":1.133,"15min":1.133},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.71099999999999997,"5min":0.81999999999999995,"15min":0.83799999999999997},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.081,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.48199999999999998}]},{"osd":10,"last update":"Thu Mar 9 14:45:21 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.77800000000000002,"15min":0.79200000000000004},"min":{"1min":0.49199999999999999,"5min":0.435,"15min":0.435},"max":{"1min":0.97299999999999998,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.70099999999999996,"5min":0.77900000000000003,"15min":0.79200000000000004},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.056,"5min":1.0620000000000001,"15min":1.0620000000000001},"last":0.72999999999999998}]},{"osd":11,"last update":"Thu Mar 9 14:45:23 2023","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":2.6659999999999999,"5min":2.6659999999999999,"15min":2.6659999999999999},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.60699999999999998}]}]},{"osd":8,"up_from":57,"seq":244813135917,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6560,"kb_used_data":984,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738592,"statfs":{"total":95995035648,"available":95988318208,"internally_reserved":0,"allocated":1007616,"data_stored":490086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.51900000000000002,"15min":0.51500000000000001},"min":{"1min":0.46600000000000003,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.73199999999999998,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.54000000000000004,"15min":0.54800000000000004},"min":{"1min":0.36299999999999999,"5min":0.31,"15min":0.31},"max":{"1min":0.65700000000000003,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.42499999999999999}]},{"osd":1,"last update":"Thu Mar 9 14:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.57399999999999995,"15min":0.54800000000000004},"min":{"1min":0.434,"5min":0.313,"15min":0.313},"max":{"1min":0.88400000000000001,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.57799999999999996,"15min":0.56399999999999995},"min":{"1min":0.41699999999999998,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.79200000000000004,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.65400000000000003}]},{"osd":2,"last update":"Thu Mar 9 14:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.438,"15min":0.42199999999999999},"min":{"1min":0.314,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.45000000000000001,"15min":0.433},"min":{"1min":0.39700000000000002,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.47699999999999998}]},{"osd":3,"last update":"Thu Mar 9 14:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.47599999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.69299999999999995,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.628,"15min":0.61399999999999999},"min":{"1min":0.51200000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.67700000000000005,"5min":3.9300000000000002,"15min":3.9300000000000002},"last":0.45100000000000001}]},{"osd":4,"last update":"Thu Mar 9 14:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.58399999999999996,"15min":0.56299999999999994},"min":{"1min":0.35999999999999999,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.78400000000000003,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.61599999999999999,"15min":0.59299999999999997},"min":{"1min":0.48899999999999999,"5min":0.376,"15min":0.376},"max":{"1min":0.85599999999999998,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.68400000000000005}]},{"osd":5,"last update":"Thu Mar 9 14:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.54900000000000004,"15min":0.53700000000000003},"min":{"1min":0.47799999999999998,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.70299999999999996,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.629,"5min":0.56100000000000005,"15min":0.54200000000000004},"min":{"1min":0.40500000000000003,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.70399999999999996}]},{"osd":6,"last update":"Thu Mar 9 14:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.70599999999999996,"15min":0.68200000000000005},"min":{"1min":0.53200000000000003,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.89500000000000002,"5min":3.895,"15min":3.895},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.66000000000000003,"15min":0.65200000000000002},"min":{"1min":0.51100000000000001,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.82199999999999995,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.755}]},{"osd":7,"last update":"Thu Mar 9 14:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.67700000000000005,"15min":0.66700000000000004},"min":{"1min":0.51400000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.84899999999999998,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.68999999999999995,"15min":0.67000000000000004},"min":{"1min":0.501,"5min":0.44,"15min":0.44},"max":{"1min":0.90500000000000003,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.79400000000000004}]},{"osd":9,"last update":"Thu Mar 9 14:46:00 2023","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.69399999999999995,"15min":0.67400000000000004},"min":{"1min":0.60699999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.95299999999999996,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.71199999999999997,"15min":0.68400000000000005},"min":{"1min":0.67300000000000004,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.0169999999999999,"5min":1.119,"15min":1.119},"last":0.77600000000000002}]},{"osd":10,"last update":"Thu Mar 9 14:45:18 2023","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.70099999999999996,"15min":0.68000000000000005},"min":{"1min":0.59499999999999997,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.099,"5min":1.1319999999999999,"15min":1.1319999999999999},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.70899999999999996,"15min":0.68700000000000006},"min":{"1min":0.60599999999999998,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.80600000000000005}]},{"osd":11,"last update":"Thu Mar 9 14:45:23 2023","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.82999999999999996}]}]},{"osd":7,"up_from":50,"seq":214748364850,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7212,"kb_used_data":1572,"kb_used_omap":0,"kb_used_meta":5632,"kb_avail":93737940,"statfs":{"total":95995035648,"available":95987650560,"internally_reserved":0,"allocated":1609728,"data_stored":1087350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5767168},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:44:13 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.52100000000000002,"15min":0.51700000000000002},"min":{"1min":0.316,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.69999999999999996,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.375},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.51900000000000002,"15min":0.50900000000000001},"min":{"1min":0.32800000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.90100000000000002,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.33900000000000002}]},{"osd":1,"last update":"Thu Mar 9 14:45:14 2023","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.40600000000000003,"15min":0.39400000000000002},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.54500000000000004,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.40600000000000003,"15min":0.39100000000000001},"min":{"1min":0.29799999999999999,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.56999999999999995,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.52900000000000003}]},{"osd":2,"last update":"Thu Mar 9 14:45:14 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.54600000000000004,"15min":0.53400000000000003},"min":{"1min":0.33500000000000002,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.754,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.58399999999999996,"15min":0.56999999999999995},"min":{"1min":0.44600000000000001,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.80700000000000005,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.64900000000000002}]},{"osd":3,"last update":"Thu Mar 9 14:45:14 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.55700000000000005,"15min":0.53800000000000003},"min":{"1min":0.40200000000000002,"5min":0.37,"15min":0.37},"max":{"1min":0.76000000000000001,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.57999999999999996,"15min":0.56399999999999995},"min":{"1min":0.44,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.76400000000000001,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.48899999999999999}]},{"osd":4,"last update":"Thu Mar 9 14:45:14 2023","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48599999999999999,"15min":0.47199999999999998},"min":{"1min":0.34799999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.65100000000000002,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.47599999999999998,"15min":0.46000000000000002},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.68100000000000005,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.42099999999999999}]},{"osd":5,"last update":"Thu Mar 9 14:45:14 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.58799999999999997,"15min":0.57399999999999995},"min":{"1min":0.40300000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.76200000000000001,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.623,"5min":0.59499999999999997,"15min":0.57799999999999996},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.80200000000000005,"5min":0.874,"15min":0.874},"last":0.61399999999999999}]},{"osd":6,"last update":"Thu Mar 9 14:45:14 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.63300000000000001,"15min":0.61699999999999999},"min":{"1min":0.50600000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.623,"15min":0.61299999999999999},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.81000000000000005,"5min":0.878,"15min":0.878},"last":0.68000000000000005}]},{"osd":8,"last update":"Thu Mar 9 14:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.66600000000000004,"15min":0.65900000000000003},"min":{"1min":0.57299999999999995,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.84799999999999998,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.68799999999999994,"5min":0.65700000000000003,"15min":0.65000000000000002},"min":{"1min":0.53700000000000003,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.66600000000000004}]},{"osd":9,"last update":"Thu Mar 9 14:45:58 2023","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.70099999999999996,"15min":0.69499999999999995},"min":{"1min":0.53300000000000003,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":0.89000000000000001,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.70599999999999996,"15min":0.70599999999999996},"min":{"1min":0.52800000000000002,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.628}]},{"osd":10,"last update":"Thu Mar 9 14:45:20 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.66300000000000003,"15min":0.66600000000000004},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.83799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.70299999999999996,"15min":0.70099999999999996},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.81799999999999995,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.747}]},{"osd":11,"last update":"Thu Mar 9 14:45:20 2023","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.76200000000000001}]}]},{"osd":6,"up_from":45,"seq":193273528374,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6552,"kb_used_data":976,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738600,"statfs":{"total":95995035648,"available":95988326400,"internally_reserved":0,"allocated":999424,"data_stored":486345,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:45:55 2023","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.40300000000000002,"15min":0.40899999999999997},"min":{"1min":0.28000000000000003,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.503,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.49299999999999999,"15min":0.49099999999999999},"min":{"1min":0.35699999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.66300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.45700000000000002}]},{"osd":1,"last update":"Thu Mar 9 14:45:55 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.56999999999999995,"15min":0.59499999999999997},"min":{"1min":0.38,"5min":0.315,"15min":0.315},"max":{"1min":0.71699999999999997,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.42099999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.75700000000000001,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.53000000000000003}]},{"osd":2,"last update":"Thu Mar 9 14:45:55 2023","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.59599999999999997,"15min":0.63200000000000001},"min":{"1min":0.38900000000000001,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.72799999999999998,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.622,"15min":0.63300000000000001},"min":{"1min":0.42399999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.79900000000000004,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.39200000000000002}]},{"osd":3,"last update":"Thu Mar 9 14:45:55 2023","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.44700000000000001,"15min":0.45700000000000002},"min":{"1min":0.32300000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.48899999999999999,"5min":0.745,"15min":0.745},"last":0.308},{"interface":"front","average":{"1min":0.501,"5min":0.55300000000000005,"15min":0.55400000000000005},"min":{"1min":0.36699999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.72899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.505}]},{"osd":4,"last update":"Thu Mar 9 14:45:55 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.64500000000000002,"15min":0.72299999999999998},"min":{"1min":0.41999999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.73299999999999998,"5min":2.0840000000000001,"15min":2.0840000000000001},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.747,"15min":0.871},"min":{"1min":0.42999999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.71599999999999997,"5min":4.1440000000000001,"15min":4.1440000000000001},"last":0.64000000000000001}]},{"osd":5,"last update":"Thu Mar 9 14:45:55 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.68300000000000005,"15min":0.69499999999999995},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.86099999999999999,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.64200000000000002,"15min":0.67600000000000005},"min":{"1min":0.47799999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.746,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.48599999999999999}]},{"osd":7,"last update":"Thu Mar 9 14:45:20 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.68100000000000005,"15min":0.70299999999999996},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.78600000000000003,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.71499999999999997,"15min":0.73099999999999998},"min":{"1min":0.5,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":0.87,"5min":0.99099999999999999,"15min":0.99099999999999999},"last":0.71399999999999997}]},{"osd":8,"last update":"Thu Mar 9 14:45:36 2023","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.75,"15min":0.78000000000000003},"min":{"1min":0.54600000000000004,"5min":0.497,"15min":0.497},"max":{"1min":0.84699999999999998,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.68799999999999994,"15min":0.71599999999999997},"min":{"1min":0.502,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.77700000000000002,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.436}]},{"osd":9,"last update":"Thu Mar 9 14:45:55 2023","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.64300000000000002,"15min":0.67300000000000004},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.72099999999999997,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.70599999999999996,"15min":0.74199999999999999},"min":{"1min":0.49099999999999999,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.77400000000000002,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.56599999999999995}]},{"osd":10,"last update":"Thu Mar 9 14:45:20 2023","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.70599999999999996,"15min":0.70799999999999996},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.82099999999999995,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.73999999999999999,"15min":0.746},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":0.89800000000000002,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.745}]},{"osd":11,"last update":"Thu Mar 9 14:45:28 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.65200000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574922,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7640,"kb_used_data":1040,"kb_used_omap":0,"kb_used_meta":6592,"kb_avail":93737512,"statfs":{"total":95995035648,"available":95987212288,"internally_reserved":0,"allocated":1064960,"data_stored":548252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6750208},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:45:09 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.50900000000000001,"15min":0.49299999999999999},"min":{"1min":0.36599999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.63300000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.55100000000000005,"15min":0.497},"min":{"1min":0.42099999999999999,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.88700000000000001,"5min":2.0670000000000002,"15min":2.0670000000000002},"last":0.495}]},{"osd":2,"last update":"Thu Mar 9 14:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.59099999999999997,"15min":0.54400000000000004},"min":{"1min":0.60899999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.05,"5min":1.3839999999999999,"15min":1.3839999999999999},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.56000000000000005,"15min":0.51800000000000002},"min":{"1min":0.45300000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.96399999999999997,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.57999999999999996}]},{"osd":3,"last update":"Thu Mar 9 14:45:56 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56699999999999995,"15min":0.51500000000000001},"min":{"1min":0.38300000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.83999999999999997,"5min":2.206,"15min":2.206},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.59099999999999997,"15min":0.53500000000000003},"min":{"1min":0.43099999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.81699999999999995,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.54800000000000004}]},{"osd":4,"last update":"Thu Mar 9 14:45:18 2023","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.441,"15min":0.39500000000000002},"min":{"1min":0.39900000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.434,"15min":0.39000000000000001},"min":{"1min":0.34100000000000003,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.82099999999999995}]},{"osd":5,"last update":"Thu Mar 9 14:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.623,"15min":0.56299999999999994},"min":{"1min":0.53200000000000003,"5min":0.373,"15min":0.373},"max":{"1min":1.081,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.61299999999999999,"15min":0.55800000000000005},"min":{"1min":0.53400000000000003,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.92900000000000005,"5min":1.363,"15min":1.363},"last":0.61799999999999999}]},{"osd":6,"last update":"Thu Mar 9 14:45:56 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.65200000000000002,"15min":0.65600000000000003},"min":{"1min":0.47199999999999998,"5min":0.371,"15min":0.371},"max":{"1min":0.91600000000000004,"5min":2.1930000000000001,"15min":2.1930000000000001},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.65800000000000003,"15min":0.66400000000000003},"min":{"1min":0.45300000000000001,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.90200000000000002,"5min":2.2519999999999998,"15min":2.2519999999999998},"last":0.65700000000000003}]},{"osd":7,"last update":"Thu Mar 9 14:45:11 2023","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.55100000000000005,"15min":0.5},"min":{"1min":0.33800000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.93400000000000005,"5min":0.93400000000000005,"15min":0.93400000000000005},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.55400000000000005,"15min":0.51600000000000001},"min":{"1min":0.35799999999999998,"5min":0.31,"15min":0.31},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.69199999999999995}]},{"osd":8,"last update":"Thu Mar 9 14:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.65600000000000003,"15min":0.622},"min":{"1min":0.57199999999999995,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":0.92600000000000005,"5min":1.3009999999999999,"15min":1.3009999999999999},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.71999999999999997,"15min":0.68400000000000005},"min":{"1min":0.497,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.1339999999999999,"5min":1.423,"15min":1.423},"last":0.80700000000000005}]},{"osd":9,"last update":"Thu Mar 9 14:45:58 2023","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.70899999999999996,"15min":0.67300000000000004},"min":{"1min":0.60099999999999998,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.70899999999999996,"15min":0.68000000000000005},"min":{"1min":0.65600000000000003,"5min":0.376,"15min":0.376},"max":{"1min":0.96299999999999997,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.79600000000000004}]},{"osd":10,"last update":"Thu Mar 9 14:45:19 2023","interfaces":[{"interface":"back","average":{"1min":0.86799999999999999,"5min":0.75600000000000001,"15min":0.73699999999999999},"min":{"1min":0.66800000000000004,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.77100000000000002,"15min":0.75800000000000001},"min":{"1min":0.68000000000000005,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.0109999999999999,"5min":1.022,"15min":1.022},"last":0.84999999999999998}]},{"osd":11,"last update":"Thu Mar 9 14:45:18 2023","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.1559999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.873,"5min":0.873,"15min":0.873},"min":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.0449999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.879}]}]},{"osd":0,"up_from":8,"seq":34359738448,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7468,"kb_used_data":996,"kb_used_omap":0,"kb_used_meta":6464,"kb_avail":93737684,"statfs":{"total":95995035648,"available":95987388416,"internally_reserved":0,"allocated":1019904,"data_stored":487611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6619136},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Mar 9 14:45:11 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.55200000000000005,"15min":0.52200000000000002},"min":{"1min":0.375,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.67700000000000005,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.54300000000000004,"15min":0.52700000000000002},"min":{"1min":0.371,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.78900000000000003,"5min":1.0569999999999999,"15min":1.0569999999999999},"last":0.71799999999999997}]},{"osd":2,"last update":"Thu Mar 9 14:45:31 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.56599999999999995,"15min":0.55600000000000005},"min":{"1min":0.40699999999999997,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.80800000000000005,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.60699999999999998,"15min":0.58199999999999996},"min":{"1min":0.45200000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.79400000000000004,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.68200000000000005}]},{"osd":3,"last update":"Thu Mar 9 14:45:20 2023","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.379}]},{"osd":4,"last update":"Thu Mar 9 14:45:11 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.59699999999999998,"15min":0.57099999999999995},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.78400000000000003,"5min":1.139,"15min":1.139},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.55800000000000005,"15min":0.54600000000000004},"min":{"1min":0.38700000000000001,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.88900000000000001,"5min":1.071,"15min":1.071},"last":0.60499999999999998}]},{"osd":5,"last update":"Thu Mar 9 14:45:38 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.60699999999999998,"15min":0.57099999999999995},"min":{"1min":0.504,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.66100000000000003,"15min":0.63100000000000001},"min":{"1min":0.53300000000000003,"5min":0.378,"15min":0.378},"max":{"1min":0.85099999999999998,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.873}]},{"osd":6,"last update":"Thu Mar 9 14:45:57 2023","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.54400000000000004,"15min":0.50700000000000001},"min":{"1min":0.35199999999999998,"5min":0.309,"15min":0.309},"max":{"1min":0.88200000000000001,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.58499999999999996,"15min":0.55100000000000005},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.79500000000000004,"5min":1.159,"15min":1.159},"last":0.78000000000000003}]},{"osd":7,"last update":"Thu Mar 9 14:45:25 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.83899999999999997}]},{"osd":8,"last update":"Thu Mar 9 14:45:38 2023","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.72199999999999998,"15min":0.73299999999999998},"min":{"1min":0.58899999999999997,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.97399999999999998,"5min":1.038,"15min":1.038},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.755,"5min":0.79000000000000004,"15min":0.81799999999999995},"min":{"1min":0.56499999999999995,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.93899999999999995,"5min":1.242,"15min":1.242},"last":0.90100000000000002}]},{"osd":9,"last update":"Thu Mar 9 14:44:59 2023","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.74299999999999999,"15min":0.75900000000000001},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.90600000000000003,"5min":1.105,"15min":1.105},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.82199999999999995,"15min":0.85399999999999998},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.88800000000000001,"5min":1.27,"15min":1.27},"last":0.629}]},{"osd":10,"last update":"Thu Mar 9 14:45:20 2023","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.79000000000000004,"15min":0.79800000000000004},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":0.93500000000000005,"5min":1.151,"15min":1.151},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.70999999999999996,"15min":0.71199999999999997},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.65100000000000002}]},{"osd":11,"last update":"Thu Mar 9 14:45:38 2023","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.73899999999999999,"15min":0.73299999999999998},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.71899999999999997,"15min":0.71799999999999997},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.70399999999999996}]}]},{"osd":2,"up_from":18,"seq":77309411399,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7708,"kb_used_data":1044,"kb_used_omap":0,"kb_used_meta":6656,"kb_avail":93737444,"statfs":{"total":95995035648,"available":95987142656,"internally_reserved":0,"allocated":1069056,"data_stored":543544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6815744},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:45:32 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.52900000000000003,"15min":0.5},"min":{"1min":0.40500000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.48399999999999999,"15min":0.46999999999999997},"min":{"1min":0.32400000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.51200000000000001}]},{"osd":1,"last update":"Thu Mar 9 14:45:32 2023","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.56899999999999995,"15min":0.51000000000000001},"min":{"1min":0.41299999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.83599999999999997,"5min":1.3340000000000001,"15min":1.3340000000000001},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.58399999999999996,"15min":0.51800000000000002},"min":{"1min":0.44600000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.85699999999999998,"5min":1.1890000000000001,"15min":1.1890000000000001},"last":0.81200000000000006}]},{"osd":3,"last update":"Thu Mar 9 14:45:52 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.57899999999999996,"15min":0.54300000000000004},"min":{"1min":0.36599999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.81999999999999995,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.63},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.55200000000000005,"15min":0.53600000000000003},"min":{"1min":0.39800000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.70799999999999996,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.66200000000000003}]},{"osd":4,"last update":"Thu Mar 9 14:45:21 2023","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.83499999999999996}]},{"osd":5,"last update":"Thu Mar 9 14:45:38 2023","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.48999999999999999,"15min":0.46600000000000003},"min":{"1min":0.32100000000000001,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.77300000000000002,"5min":0.871,"15min":0.871},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.46300000000000002,"15min":0.44400000000000001},"min":{"1min":0.33600000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.747,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.64600000000000002}]},{"osd":6,"last update":"Thu Mar 9 14:45:52 2023","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.59199999999999997,"15min":0.57899999999999996},"min":{"1min":0.41099999999999998,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.79600000000000004,"5min":1.002,"15min":1.002},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.69599999999999995,"5min":0.64700000000000002,"15min":0.621},"min":{"1min":0.53600000000000003,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.879,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.755}]},{"osd":7,"last update":"Thu Mar 9 14:45:22 2023","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"min":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.87},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.88300000000000001}]},{"osd":8,"last update":"Thu Mar 9 14:45:38 2023","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.65700000000000003,"15min":0.65200000000000002},"min":{"1min":0.42199999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.90400000000000003,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.92400000000000004},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.64500000000000002,"15min":0.65100000000000002},"min":{"1min":0.45800000000000002,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.94099999999999995,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.73899999999999999}]},{"osd":9,"last update":"Thu Mar 9 14:44:56 2023","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.72399999999999998,"15min":0.72999999999999998},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.0109999999999999,"5min":1.21,"15min":1.21},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.67100000000000004,"15min":0.68000000000000005},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.93200000000000005,"5min":0.94699999999999995,"15min":0.94699999999999995},"last":0.71999999999999997}]},{"osd":10,"last update":"Thu Mar 9 14:45:18 2023","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.68200000000000005,"15min":0.68100000000000005},"min":{"1min":0.55500000000000005,"5min":0.376,"15min":0.376},"max":{"1min":0.90600000000000003,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.73999999999999999,"15min":0.72899999999999998},"min":{"1min":0.624,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.091,"5min":1.306,"15min":1.306},"last":0.90700000000000003}]},{"osd":11,"last update":"Thu Mar 9 14:45:18 2023","interfaces":[{"interface":"back","average":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.95899999999999996},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.149,"5min":1.149,"15min":1.149},"last":0.94399999999999995}]}]},{"osd":3,"up_from":25,"seq":107374182466,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6816,"kb_used_data":984,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738336,"statfs":{"total":95995035648,"available":95988056064,"internally_reserved":0,"allocated":1007616,"data_stored":486449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:45:48 2023","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.377,"15min":0.33200000000000002},"min":{"1min":0.27700000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.498,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":2.669},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.39300000000000002,"15min":0.34799999999999998},"min":{"1min":0.27900000000000003,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.57899999999999996,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":3.1110000000000002}]},{"osd":1,"last update":"Thu Mar 9 14:45:48 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.53500000000000003,"15min":0.50600000000000001},"min":{"1min":0.312,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.77700000000000002,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.57199999999999995,"15min":0.503},"min":{"1min":0.41399999999999998,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.67400000000000004,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":6.2329999999999997}]},{"osd":2,"last update":"Thu Mar 9 14:45:48 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.57599999999999996,"15min":0.52200000000000002},"min":{"1min":0.40500000000000003,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.70699999999999996,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":1.361},{"interface":"front","average":{"1min":0.621,"5min":0.58899999999999997,"15min":0.53900000000000003},"min":{"1min":0.40899999999999997,"5min":0.312,"15min":0.312},"max":{"1min":1.776,"5min":1.776,"15min":1.776},"last":1.2350000000000001}]},{"osd":4,"last update":"Thu Mar 9 14:45:10 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.58099999999999996,"15min":0.54200000000000004},"min":{"1min":0.45200000000000001,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.75900000000000001,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":3.5019999999999998},{"interface":"front","average":{"1min":0.622,"5min":0.60099999999999998,"15min":0.55000000000000004},"min":{"1min":0.497,"5min":0.373,"15min":0.373},"max":{"1min":0.84599999999999997,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":2.4449999999999998}]},{"osd":5,"last update":"Thu Mar 9 14:45:35 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.58499999999999996,"15min":0.54900000000000004},"min":{"1min":0.438,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.861,"5min":1.861,"15min":1.861},"last":1.341},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.60699999999999998,"15min":0.56699999999999995},"min":{"1min":0.46200000000000002,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.68600000000000005,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":7.2220000000000004}]},{"osd":6,"last update":"Thu Mar 9 14:45:54 2023","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.55600000000000005,"15min":0.53400000000000003},"min":{"1min":0.36299999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.72199999999999998,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":7.9909999999999997},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.57399999999999995,"15min":0.54400000000000004},"min":{"1min":0.34499999999999997,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.81000000000000005,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":3.8759999999999999}]},{"osd":7,"last update":"Thu Mar 9 14:45:17 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.69399999999999995,"5min":0.98299999999999998,"15min":0.98299999999999998},"last":1.5580000000000001},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.68000000000000005,"15min":0.66700000000000004},"min":{"1min":0.46500000000000002,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.80400000000000005,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":9.3710000000000004}]},{"osd":8,"last update":"Thu Mar 9 14:43:35 2023","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.1299999999999999,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.749}]},{"osd":9,"last update":"Thu Mar 9 14:45:54 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.73099999999999998,"15min":0.77600000000000002},"min":{"1min":0.434,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.75700000000000001,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":10.67},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.73999999999999999,"15min":0.78800000000000003},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.80700000000000005,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":4.4400000000000004}]},{"osd":10,"last update":"Thu Mar 9 14:45:25 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.69899999999999995,"15min":0.70799999999999996},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":0.76400000000000001,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":1.667},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.79200000000000004,"15min":0.80500000000000005},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.83199999999999996,"5min":1.155,"15min":1.155},"last":11.667999999999999}]},{"osd":11,"last update":"Thu Mar 9 14:45:19 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":12.622},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":4.1820000000000004}]}]},{"osd":4,"up_from":32,"seq":137438953535,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6700,"kb_used_data":996,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738452,"statfs":{"total":95995035648,"available":95988174848,"internally_reserved":0,"allocated":1019904,"data_stored":492942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:44:09 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.53100000000000003,"15min":0.495},"min":{"1min":0.39000000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.84099999999999997,"5min":1.8320000000000001,"15min":1.8320000000000001},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.628,"5min":0.52900000000000003,"15min":0.495},"min":{"1min":0.42499999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":1.044,"5min":1.478,"15min":1.478},"last":0.53500000000000003}]},{"osd":1,"last update":"Thu Mar 9 14:45:14 2023","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.376,"15min":0.34799999999999998},"min":{"1min":0.311,"5min":0.253,"15min":0.253},"max":{"1min":0.53100000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.39600000000000002,"15min":0.36799999999999999},"min":{"1min":0.316,"5min":0.253,"15min":0.253},"max":{"1min":0.625,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.42999999999999999}]},{"osd":2,"last update":"Thu Mar 9 14:45:14 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.54100000000000004,"15min":0.53000000000000003},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.087,"5min":1.087,"15min":1.087},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.57199999999999995,"15min":0.53900000000000003},"min":{"1min":0.41499999999999998,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"last":0.71999999999999997}]},{"osd":3,"last update":"Thu Mar 9 14:45:14 2023","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":0.69899999999999995,"15min":0.57099999999999995},"min":{"1min":0.47699999999999998,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":9.9420000000000002,"5min":9.9420000000000002,"15min":9.9420000000000002},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.67700000000000005,"5min":0.59699999999999998,"15min":0.54100000000000004},"min":{"1min":0.45200000000000001,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":2.6110000000000002,"5min":2.6110000000000002,"15min":2.6110000000000002},"last":0.40899999999999997}]},{"osd":5,"last update":"Thu Mar 9 14:45:33 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.59399999999999997,"15min":0.56299999999999994},"min":{"1min":0.47099999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.755,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.62},{"interface":"front","average":{"1min":0.629,"5min":0.60999999999999999,"15min":0.57099999999999995},"min":{"1min":0.495,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.85999999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.60699999999999998}]},{"osd":6,"last update":"Thu Mar 9 14:45:54 2023","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.66400000000000003,"15min":0.68700000000000006},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.77000000000000002,"5min":2.4609999999999999,"15min":2.4609999999999999},"last":0.496},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.72599999999999998,"15min":0.69199999999999995},"min":{"1min":0.48899999999999999,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":0.91400000000000003,"5min":4.6879999999999997,"15min":4.6879999999999997},"last":0.68999999999999995}]},{"osd":7,"last update":"Thu Mar 9 14:45:14 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.56999999999999995,"15min":0.53400000000000003},"min":{"1min":0.42299999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.872,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.58199999999999996,"15min":0.54000000000000004},"min":{"1min":0.437,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.77400000000000002,"5min":0.998,"15min":0.998},"last":0.57299999999999995}]},{"osd":8,"last update":"Thu Mar 9 14:45:36 2023","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.71599999999999997,"15min":0.69099999999999995},"min":{"1min":0.40200000000000002,"5min":0.377,"15min":0.377},"max":{"1min":0.90100000000000002,"5min":1.7569999999999999,"15min":1.7569999999999999},"last":0.745},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.72399999999999998,"15min":0.69699999999999995},"min":{"1min":0.58999999999999997,"5min":0.498,"15min":0.498},"max":{"1min":0.89100000000000001,"5min":1.746,"15min":1.746},"last":0.70399999999999996}]},{"osd":9,"last update":"Thu Mar 9 14:45:58 2023","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.77600000000000002,"15min":0.76000000000000001},"min":{"1min":0.56799999999999995,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.89700000000000002,"5min":2.5419999999999998,"15min":2.5419999999999998},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.71799999999999997,"15min":0.71099999999999997},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.85399999999999998,"5min":2.4969999999999999,"15min":2.4969999999999999},"last":0.59499999999999997}]},{"osd":10,"last update":"Thu Mar 9 14:45:20 2023","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.77100000000000002,"15min":0.77200000000000002},"min":{"1min":0.61099999999999999,"5min":0.432,"15min":0.432},"max":{"1min":1.0009999999999999,"5min":1.069,"15min":1.069},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.79000000000000004,"15min":0.79300000000000004},"min":{"1min":0.627,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.109,"5min":1.228,"15min":1.228},"last":0.77900000000000003}]},{"osd":11,"last update":"Thu Mar 9 14:45:18 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.1779999999999999,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.753},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.67600000000000005}]}]},{"osd":5,"up_from":37,"seq":158913790011,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7336,"kb_used_data":1568,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93737816,"statfs":{"total":95995035648,"available":95987523584,"internally_reserved":0,"allocated":1605632,"data_stored":1081598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Mar 9 14:45:38 2023","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.59499999999999997,"15min":0.51300000000000001},"min":{"1min":0.32400000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":9.7189999999999994,"5min":9.7189999999999994,"15min":9.7189999999999994},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.56299999999999994,"15min":0.51500000000000001},"min":{"1min":0.307,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":5.4660000000000002,"5min":5.4660000000000002,"15min":5.4660000000000002},"last":0.437}]},{"osd":1,"last update":"Thu Mar 9 14:45:38 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.57499999999999996,"15min":0.55400000000000005},"min":{"1min":0.45400000000000001,"5min":0.316,"15min":0.316},"max":{"1min":0.77500000000000002,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.56100000000000005,"15min":0.53700000000000003},"min":{"1min":0.46200000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.752,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.60199999999999998}]},{"osd":2,"last update":"Thu Mar 9 14:45:38 2023","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.39500000000000002,"15min":0.36699999999999999},"min":{"1min":0.313,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.61499999999999999,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.626},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.41299999999999998,"15min":0.377},"min":{"1min":0.33900000000000002,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.65500000000000003,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.56399999999999995}]},{"osd":3,"last update":"Thu Mar 9 14:45:38 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.54800000000000004,"15min":0.53800000000000003},"min":{"1min":0.34499999999999997,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":2.7269999999999999,"5min":2.7269999999999999,"15min":2.7269999999999999},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.622,"15min":0.56899999999999995},"min":{"1min":0.39200000000000002,"5min":0.313,"15min":0.313},"max":{"1min":5.6130000000000004,"5min":5.6130000000000004,"15min":5.6130000000000004},"last":0.67600000000000005}]},{"osd":4,"last update":"Thu Mar 9 14:45:38 2023","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.58999999999999997,"15min":0.56299999999999994},"min":{"1min":0.51400000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.78000000000000003,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.53700000000000003,"15min":0.52800000000000002},"min":{"1min":0.42599999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.65800000000000003,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.57599999999999996}]},{"osd":6,"last update":"Thu Mar 9 14:45:52 2023","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.626,"15min":0.60799999999999998},"min":{"1min":0.438,"5min":0.315,"15min":0.315},"max":{"1min":2.5369999999999999,"5min":2.5369999999999999,"15min":2.5369999999999999},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.66800000000000004,"15min":0.63600000000000001},"min":{"1min":0.504,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":2.5019999999999998,"5min":2.5019999999999998,"15min":2.5019999999999998},"last":0.76100000000000001}]},{"osd":7,"last update":"Thu Mar 9 14:45:18 2023","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.61099999999999999,"15min":0.59399999999999997},"min":{"1min":0.42099999999999999,"5min":0.38,"15min":0.38},"max":{"1min":0.84799999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.61899999999999999,"15min":0.59999999999999998},"min":{"1min":0.46899999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.83299999999999996,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.80400000000000005}]},{"osd":8,"last update":"Thu Mar 9 14:45:38 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.60199999999999998,"15min":0.60099999999999998},"min":{"1min":0.439,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.83499999999999996,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.42999999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.70699999999999996,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.71699999999999997}]},{"osd":9,"last update":"Thu Mar 9 14:45:58 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.67600000000000005,"15min":0.64500000000000002},"min":{"1min":0.496,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.81699999999999995,"5min":5.6749999999999998,"15min":5.6749999999999998},"last":0.751},{"interface":"front","average":{"1min":0.745,"5min":0.72199999999999998,"15min":0.71699999999999997},"min":{"1min":0.58999999999999997,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.91600000000000004,"5min":1.468,"15min":1.468},"last":0.78400000000000003}]},{"osd":10,"last update":"Thu Mar 9 14:45:18 2023","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.69799999999999995,"15min":0.68999999999999995},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.86499999999999999,"5min":0.90500000000000003,"15min":0.90500000000000003},"last":0.81299999999999994},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.70899999999999996,"15min":0.69999999999999996},"min":{"1min":0.54300000000000004,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":0.877,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.82699999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1446,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":62087,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":57379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":63932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":636,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-03-09T14:46:02.835 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-09T14:46:02.993 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.990+0000 7f383cc01700 1 -- 172.21.15.38:0/2639321748 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f38380fff70 msgr2=0x7f38381003c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:02.993 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.990+0000 7f383cc01700 1 --2- 172.21.15.38:0/2639321748 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f38380fff70 0x7f38381003c0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f382c009990 tx=0x7f382c009ca0 comp rx=0 tx=0).stop 2023-03-09T14:46:02.993 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.990+0000 7f383cc01700 1 -- 172.21.15.38:0/2639321748 shutdown_connections 2023-03-09T14:46:02.993 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.990+0000 7f383cc01700 1 --2- 172.21.15.38:0/2639321748 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3838100900 0x7f38381071c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:02.994 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.990+0000 7f383cc01700 1 --2- 172.21.15.38:0/2639321748 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f38380fed80 0x7f38380ff190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:02.994 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.990+0000 7f383cc01700 1 --2- 172.21.15.38:0/2639321748 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f38380fff70 0x7f38381003c0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:02.994 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.990+0000 7f383cc01700 1 -- 172.21.15.38:0/2639321748 >> 172.21.15.38:0/2639321748 conn(0x7f38380fa380 msgr2=0x7f38380fc7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:02.994 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.990+0000 7f383cc01700 1 -- 172.21.15.38:0/2639321748 shutdown_connections 2023-03-09T14:46:02.994 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.990+0000 7f383cc01700 1 -- 172.21.15.38:0/2639321748 wait complete. 2023-03-09T14:46:02.997 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.991+0000 7f383cc01700 1 Processor -- start 2023-03-09T14:46:02.997 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.991+0000 7f383cc01700 1 -- start start 2023-03-09T14:46:02.997 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.991+0000 7f383cc01700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f38380fed80 0x7f38380685d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:02.998 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.991+0000 7f383cc01700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f38380fff70 0x7f383819a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:02.998 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.991+0000 7f383cc01700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3838100900 0x7f383819cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:02.998 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.991+0000 7f383cc01700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f383819d780 con 0x7f3838100900 2023-03-09T14:46:02.998 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.991+0000 7f383cc01700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f383819d8f0 con 0x7f38380fff70 2023-03-09T14:46:02.998 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.991+0000 7f383cc01700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f383819dab0 con 0x7f38380fed80 2023-03-09T14:46:02.998 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.991+0000 7f3836d9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3838100900 0x7f383819cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:02.998 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.991+0000 7f3836d9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3838100900 0x7f383819cfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38526/0 (socket says 172.21.15.38:38526) 2023-03-09T14:46:02.999 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.991+0000 7f3836d9d700 1 -- 172.21.15.38:0/3091986615 learned_addr learned my addr 172.21.15.38:0/3091986615 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:46:02.999 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.991+0000 7f3835d9b700 1 --2- 172.21.15.38:0/3091986615 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f38380fff70 0x7f383819a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:03.000 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.991+0000 7f383659c700 1 --2- 172.21.15.38:0/3091986615 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f38380fed80 0x7f38380685d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:03.000 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.992+0000 7f3836d9d700 1 -- 172.21.15.38:0/3091986615 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f38380fed80 msgr2=0x7f38380685d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:03.000 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.992+0000 7f3836d9d700 1 --2- 172.21.15.38:0/3091986615 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f38380fed80 0x7f38380685d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:03.000 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.992+0000 7f3836d9d700 1 -- 172.21.15.38:0/3091986615 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f38380fff70 msgr2=0x7f383819a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:03.000 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.992+0000 7f3836d9d700 1 --2- 172.21.15.38:0/3091986615 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f38380fff70 0x7f383819a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:03.000 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.992+0000 7f3836d9d700 1 -- 172.21.15.38:0/3091986615 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f382c0095f0 con 0x7f3838100900 2023-03-09T14:46:03.001 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.992+0000 7f3836d9d700 1 --2- 172.21.15.38:0/3091986615 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3838100900 0x7f383819cfb0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f3830007ee0 tx=0x7f383000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:03.001 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.992+0000 7f38277fe700 1 -- 172.21.15.38:0/3091986615 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3830017070 con 0x7f3838100900 2023-03-09T14:46:03.001 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.992+0000 7f38277fe700 1 -- 172.21.15.38:0/3091986615 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f383000f040 con 0x7f3838100900 2023-03-09T14:46:03.001 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.992+0000 7f383cc01700 1 -- 172.21.15.38:0/3091986615 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f383810a730 con 0x7f3838100900 2023-03-09T14:46:03.001 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.992+0000 7f38277fe700 1 -- 172.21.15.38:0/3091986615 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3830005580 con 0x7f3838100900 2023-03-09T14:46:03.001 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.992+0000 7f383cc01700 1 -- 172.21.15.38:0/3091986615 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f383810a5a0 con 0x7f3838100900 2023-03-09T14:46:03.001 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.993+0000 7f383cc01700 1 -- 172.21.15.38:0/3091986615 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f383819e160 con 0x7f3838100900 2023-03-09T14:46:03.002 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.994+0000 7f38277fe700 1 -- 172.21.15.38:0/3091986615 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) v1 ==== 133198+0+0 (secure 0 0 0) 0x7f3830004c40 con 0x7f3838100900 2023-03-09T14:46:03.002 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.994+0000 7f38277fe700 1 --2- 172.21.15.38:0/3091986615 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f382009f440 0x7f38200a18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:03.002 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.995+0000 7f383659c700 1 --2- 172.21.15.38:0/3091986615 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f382009f440 0x7f38200a18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:03.002 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.995+0000 7f38277fe700 1 -- 172.21.15.38:0/3091986615 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f38300c9900 con 0x7f3838100900 2023-03-09T14:46:03.002 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.995+0000 7f383659c700 1 --2- 172.21.15.38:0/3091986615 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f382009f440 0x7f38200a18f0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f38380ffdd0 tx=0x7f3828009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:03.003 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:02.999+0000 7f38277fe700 1 -- 172.21.15.38:0/3091986615 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f38300939b0 con 0x7f3838100900 2023-03-09T14:46:03.204 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.202+0000 7f383cc01700 1 -- 172.21.15.38:0/3091986615 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3838105cb0 con 0x7f3838100900 2023-03-09T14:46:03.205 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.203+0000 7f38277fe700 1 -- 172.21.15.38:0/3091986615 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24445 (secure 0 0 0) 0x7f3830096fd0 con 0x7f3838100900 2023-03-09T14:46:03.206 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:46:03.206 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":89,"fsid":"c6733fe2-be87-11ed-9af2-001a4aab830c","created":"2023-03-09T14:37:14.947981+0000","modified":"2023-03-09T14:45:15.848936+0000","last_up_change":"2023-03-09T14:43:37.198407+0000","last_in_change":"2023-03-09T14:43:19.550566+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-09T14:40:23.196665+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-03-09T14:44:17.484374+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-03-09T14:44:18.457833+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-03-09T14:44:21.895589+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"51477039-15ee-4707-a34d-4ecfa18bebde","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6803","nonce":1508072290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6805","nonce":1508072290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6809","nonce":1508072290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6807","nonce":1508072290}]},"public_addr":"172.21.15.38:6803/1508072290","cluster_addr":"172.21.15.38:6805/1508072290","heartbeat_back_addr":"172.21.15.38:6809/1508072290","heartbeat_front_addr":"172.21.15.38:6807/1508072290","state":["exists","up"]},{"osd":1,"uuid":"71f8a718-b0a9-4f39-bc54-267aa42a2578","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6801","nonce":3929268773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6803","nonce":3929268773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6807","nonce":3929268773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6805","nonce":3929268773}]},"public_addr":"172.21.15.122:6801/3929268773","cluster_addr":"172.21.15.122:6803/3929268773","heartbeat_back_addr":"172.21.15.122:6807/3929268773","heartbeat_front_addr":"172.21.15.122:6805/3929268773","state":["exists","up"]},{"osd":2,"uuid":"961a2cbe-140e-437f-b131-1fcac573b19c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6801","nonce":1310630803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6803","nonce":1310630803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6807","nonce":1310630803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6805","nonce":1310630803}]},"public_addr":"172.21.15.181:6801/1310630803","cluster_addr":"172.21.15.181:6803/1310630803","heartbeat_back_addr":"172.21.15.181:6807/1310630803","heartbeat_front_addr":"172.21.15.181:6805/1310630803","state":["exists","up"]},{"osd":3,"uuid":"969df8fa-2d19-42c7-9f2a-5c5fac15d4a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6811","nonce":2719814100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6813","nonce":2719814100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6817","nonce":2719814100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6815","nonce":2719814100}]},"public_addr":"172.21.15.38:6811/2719814100","cluster_addr":"172.21.15.38:6813/2719814100","heartbeat_back_addr":"172.21.15.38:6817/2719814100","heartbeat_front_addr":"172.21.15.38:6815/2719814100","state":["exists","up"]},{"osd":4,"uuid":"44a6dce3-e06e-4cbd-9404-62a86d01ab6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6809","nonce":2480764994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6811","nonce":2480764994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6815","nonce":2480764994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6813","nonce":2480764994}]},"public_addr":"172.21.15.122:6809/2480764994","cluster_addr":"172.21.15.122:6811/2480764994","heartbeat_back_addr":"172.21.15.122:6815/2480764994","heartbeat_front_addr":"172.21.15.122:6813/2480764994","state":["exists","up"]},{"osd":5,"uuid":"49749574-c00c-4e3a-80e6-e3686df58d5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6809","nonce":3141346246}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6811","nonce":3141346246}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6815","nonce":3141346246}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6813","nonce":3141346246}]},"public_addr":"172.21.15.181:6809/3141346246","cluster_addr":"172.21.15.181:6811/3141346246","heartbeat_back_addr":"172.21.15.181:6815/3141346246","heartbeat_front_addr":"172.21.15.181:6813/3141346246","state":["exists","up"]},{"osd":6,"uuid":"61c0437e-3f51-45a9-8a61-e98eea0093f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6819","nonce":2122788957}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6821","nonce":2122788957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6825","nonce":2122788957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6823","nonce":2122788957}]},"public_addr":"172.21.15.38:6819/2122788957","cluster_addr":"172.21.15.38:6821/2122788957","heartbeat_back_addr":"172.21.15.38:6825/2122788957","heartbeat_front_addr":"172.21.15.38:6823/2122788957","state":["exists","up"]},{"osd":7,"uuid":"231fecdd-939f-4c7b-83a1-c4a1cffd7139","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6817","nonce":656874139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6819","nonce":656874139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6823","nonce":656874139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6821","nonce":656874139}]},"public_addr":"172.21.15.122:6817/656874139","cluster_addr":"172.21.15.122:6819/656874139","heartbeat_back_addr":"172.21.15.122:6823/656874139","heartbeat_front_addr":"172.21.15.122:6821/656874139","state":["exists","up"]},{"osd":8,"uuid":"297e6d6d-85fd-43ce-9028-5e9dbe55c36e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6817","nonce":755000703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6819","nonce":755000703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6823","nonce":755000703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6821","nonce":755000703}]},"public_addr":"172.21.15.181:6817/755000703","cluster_addr":"172.21.15.181:6819/755000703","heartbeat_back_addr":"172.21.15.181:6823/755000703","heartbeat_front_addr":"172.21.15.181:6821/755000703","state":["exists","up"]},{"osd":9,"uuid":"94cc0355-5a1a-4cfc-bd0c-562b6b41127e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6827","nonce":2624555969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6829","nonce":2624555969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6833","nonce":2624555969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6831","nonce":2624555969}]},"public_addr":"172.21.15.38:6827/2624555969","cluster_addr":"172.21.15.38:6829/2624555969","heartbeat_back_addr":"172.21.15.38:6833/2624555969","heartbeat_front_addr":"172.21.15.38:6831/2624555969","state":["exists","up"]},{"osd":10,"uuid":"f4f2e042-8d92-46bf-93b8-8b0f04795c2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6825","nonce":1427785280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6827","nonce":1427785280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6831","nonce":1427785280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6829","nonce":1427785280}]},"public_addr":"172.21.15.122:6825/1427785280","cluster_addr":"172.21.15.122:6827/1427785280","heartbeat_back_addr":"172.21.15.122:6831/1427785280","heartbeat_front_addr":"172.21.15.122:6829/1427785280","state":["exists","up"]},{"osd":11,"uuid":"07444936-eae5-4097-a799-60294cafe80b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6825","nonce":3183488640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6827","nonce":3183488640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6831","nonce":3183488640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6829","nonce":3183488640}]},"public_addr":"172.21.15.181:6825/3183488640","cluster_addr":"172.21.15.181:6827/3183488640","heartbeat_back_addr":"172.21.15.181:6831/3183488640","heartbeat_front_addr":"172.21.15.181:6829/3183488640","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:39:37.657556+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:39:58.960074+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:40:20.326952+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:40:42.244387+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:41:03.946973+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:41:25.097985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:41:46.877916+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:42:08.413827+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:42:29.822009+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:42:51.776684+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:43:13.937249+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:43:35.569199+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.2","mappings":[{"from":7,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.d","mappings":[{"from":5,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.17","mappings":[{"from":5,"to":6}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/4272288735":"2023-03-10T14:38:14.612767+0000","172.21.15.38:6801/3381625669":"2023-03-10T14:38:14.612767+0000","172.21.15.38:6800/3381625669":"2023-03-10T14:38:14.612767+0000","172.21.15.38:6800/3060474342":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/4253918615":"2023-03-10T14:38:14.612767+0000","172.21.15.38:0/3496083181":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/245602368":"2023-03-10T14:37:47.517775+0000","172.21.15.38:6801/3060474342":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/3622589182":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/1430302251":"2023-03-10T14:38:14.612767+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-09T14:46:03.208 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.206+0000 7f383cc01700 1 -- 172.21.15.38:0/3091986615 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f382009f440 msgr2=0x7f38200a18f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:03.208 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.206+0000 7f383cc01700 1 --2- 172.21.15.38:0/3091986615 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f382009f440 0x7f38200a18f0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f38380ffdd0 tx=0x7f3828009260 comp rx=0 tx=0).stop 2023-03-09T14:46:03.208 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.206+0000 7f383cc01700 1 -- 172.21.15.38:0/3091986615 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3838100900 msgr2=0x7f383819cfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:03.209 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.206+0000 7f383cc01700 1 --2- 172.21.15.38:0/3091986615 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3838100900 0x7f383819cfb0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f3830007ee0 tx=0x7f383000d3b0 comp rx=0 tx=0).stop 2023-03-09T14:46:03.209 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.206+0000 7f383cc01700 1 -- 172.21.15.38:0/3091986615 shutdown_connections 2023-03-09T14:46:03.209 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.206+0000 7f383cc01700 1 --2- 172.21.15.38:0/3091986615 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3838100900 0x7f383819cfb0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:03.209 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.206+0000 7f383cc01700 1 --2- 172.21.15.38:0/3091986615 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f382009f440 0x7f38200a18f0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:03.209 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.206+0000 7f383cc01700 1 --2- 172.21.15.38:0/3091986615 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f38380fed80 0x7f38380685d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:03.209 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.206+0000 7f383cc01700 1 --2- 172.21.15.38:0/3091986615 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f38380fff70 0x7f383819a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:03.209 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.206+0000 7f383cc01700 1 -- 172.21.15.38:0/3091986615 >> 172.21.15.38:0/3091986615 conn(0x7f38380fa380 msgr2=0x7f38380fc650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:03.209 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.206+0000 7f383cc01700 1 -- 172.21.15.38:0/3091986615 shutdown_connections 2023-03-09T14:46:03.210 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.206+0000 7f383cc01700 1 -- 172.21.15.38:0/3091986615 wait complete. 2023-03-09T14:46:03.219 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:02 smithi038 ceph-mon[92422]: from='client.15447 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-03-09T14:46:03.221 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-09T14:40:23.196665+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-03-09T14:44:17.484374+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-03-09T14:44:18.457833+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '83', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-03-09T14:44:21.895589+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '87', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-03-09T14:46:03.221 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-09T14:46:03.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:02 smithi122 ceph-mon[93869]: from='client.15447 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-03-09T14:46:03.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:02 smithi181 ceph-mon[94189]: from='client.15447 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-03-09T14:46:03.350 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.348+0000 7fc0d932e700 1 -- 172.21.15.38:0/3925452603 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0d4103250 msgr2=0x7fc0d4105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:03.350 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.348+0000 7fc0d932e700 1 --2- 172.21.15.38:0/3925452603 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0d4103250 0x7fc0d4105630 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fc0c8009990 tx=0x7fc0c8009ca0 comp rx=0 tx=0).stop 2023-03-09T14:46:03.351 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.348+0000 7fc0d932e700 1 -- 172.21.15.38:0/3925452603 shutdown_connections 2023-03-09T14:46:03.351 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.348+0000 7fc0d932e700 1 --2- 172.21.15.38:0/3925452603 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc0d4105b70 0x7fc0d4107fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:03.351 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.348+0000 7fc0d932e700 1 --2- 172.21.15.38:0/3925452603 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc0d4100930 0x7fc0d4102d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:03.351 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.348+0000 7fc0d932e700 1 --2- 172.21.15.38:0/3925452603 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0d4103250 0x7fc0d4105630 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:03.351 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.348+0000 7fc0d932e700 1 -- 172.21.15.38:0/3925452603 >> 172.21.15.38:0/3925452603 conn(0x7fc0d40fa3d0 msgr2=0x7fc0d40fc800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:03.351 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.348+0000 7fc0d932e700 1 -- 172.21.15.38:0/3925452603 shutdown_connections 2023-03-09T14:46:03.351 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.348+0000 7fc0d932e700 1 -- 172.21.15.38:0/3925452603 wait complete. 2023-03-09T14:46:03.352 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.348+0000 7fc0d932e700 1 Processor -- start 2023-03-09T14:46:03.353 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.348+0000 7fc0d932e700 1 -- start start 2023-03-09T14:46:03.353 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.349+0000 7fc0d932e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0d4100930 0x7fc0d4195c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:03.353 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.349+0000 7fc0d932e700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc0d4103250 0x7fc0d4196160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:03.354 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.349+0000 7fc0d932e700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc0d4105b70 0x7fc0d419d190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:03.354 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.349+0000 7fc0d932e700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d419d760 con 0x7fc0d4100930 2023-03-09T14:46:03.354 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.349+0000 7fc0d932e700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d419d8d0 con 0x7fc0d4105b70 2023-03-09T14:46:03.354 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.349+0000 7fc0d932e700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d419da90 con 0x7fc0d4103250 2023-03-09T14:46:03.354 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.349+0000 7fc0d2ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0d4100930 0x7fc0d4195c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:03.354 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.349+0000 7fc0d2ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0d4100930 0x7fc0d4195c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38534/0 (socket says 172.21.15.38:38534) 2023-03-09T14:46:03.354 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.349+0000 7fc0d2ffd700 1 -- 172.21.15.38:0/1158542333 learned_addr learned my addr 172.21.15.38:0/1158542333 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:46:03.355 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.349+0000 7fc0d27fc700 1 --2- 172.21.15.38:0/1158542333 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc0d4103250 0x7fc0d4196160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:03.355 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.349+0000 7fc0d37fe700 1 --2- 172.21.15.38:0/1158542333 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc0d4105b70 0x7fc0d419d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:03.355 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.349+0000 7fc0d2ffd700 1 -- 172.21.15.38:0/1158542333 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc0d4103250 msgr2=0x7fc0d4196160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:03.355 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.349+0000 7fc0d2ffd700 1 --2- 172.21.15.38:0/1158542333 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc0d4103250 0x7fc0d4196160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:03.355 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.349+0000 7fc0d2ffd700 1 -- 172.21.15.38:0/1158542333 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc0d4105b70 msgr2=0x7fc0d419d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:03.355 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.349+0000 7fc0d2ffd700 1 --2- 172.21.15.38:0/1158542333 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc0d4105b70 0x7fc0d419d190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:03.355 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.349+0000 7fc0d2ffd700 1 -- 172.21.15.38:0/1158542333 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0c80095f0 con 0x7fc0d4100930 2023-03-09T14:46:03.355 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.349+0000 7fc0d37fe700 1 --2- 172.21.15.38:0/1158542333 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc0d4105b70 0x7fc0d419d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-09T14:46:03.356 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.349+0000 7fc0d27fc700 1 --2- 172.21.15.38:0/1158542333 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc0d4103250 0x7fc0d4196160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:46:03.356 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.349+0000 7fc0d2ffd700 1 --2- 172.21.15.38:0/1158542333 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0d4100930 0x7fc0d4195c20 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fc0c400d980 tx=0x7fc0c400dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:03.356 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.349+0000 7fc0bbfff700 1 -- 172.21.15.38:0/1158542333 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0c400bb60 con 0x7fc0d4100930 2023-03-09T14:46:03.356 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.349+0000 7fc0d932e700 1 -- 172.21.15.38:0/1158542333 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0d410b850 con 0x7fc0d4100930 2023-03-09T14:46:03.356 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.349+0000 7fc0bbfff700 1 -- 172.21.15.38:0/1158542333 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc0c4007640 con 0x7fc0d4100930 2023-03-09T14:46:03.356 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.349+0000 7fc0bbfff700 1 -- 172.21.15.38:0/1158542333 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0c4004e90 con 0x7fc0d4100930 2023-03-09T14:46:03.356 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.349+0000 7fc0d932e700 1 -- 172.21.15.38:0/1158542333 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0d410b6c0 con 0x7fc0d4100930 2023-03-09T14:46:03.357 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.350+0000 7fc0d932e700 1 -- 172.21.15.38:0/1158542333 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0d40fbf40 con 0x7fc0d4100930 2023-03-09T14:46:03.357 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.351+0000 7fc0bbfff700 1 -- 172.21.15.38:0/1158542333 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) v1 ==== 133198+0+0 (secure 0 0 0) 0x7fc0c4021070 con 0x7fc0d4100930 2023-03-09T14:46:03.357 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.352+0000 7fc0bbfff700 1 --2- 172.21.15.38:0/1158542333 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fc0bc09f3c0 0x7fc0bc0a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:03.357 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.352+0000 7fc0d27fc700 1 --2- 172.21.15.38:0/1158542333 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fc0bc09f3c0 0x7fc0bc0a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:03.358 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.352+0000 7fc0bbfff700 1 -- 172.21.15.38:0/1158542333 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fc0c40c9f40 con 0x7fc0d4100930 2023-03-09T14:46:03.358 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.353+0000 7fc0d27fc700 1 --2- 172.21.15.38:0/1158542333 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fc0bc09f3c0 0x7fc0bc0a1870 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fc0c800b5c0 tx=0x7fc0c8005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:03.358 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.356+0000 7fc0bbfff700 1 -- 172.21.15.38:0/1158542333 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7fc0c4093ff0 con 0x7fc0d4100930 2023-03-09T14:46:03.558 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.556+0000 7fc0d932e700 1 -- 172.21.15.38:0/1158542333 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc0d4061640 con 0x7fc0d4100930 2023-03-09T14:46:03.559 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.557+0000 7fc0bbfff700 1 -- 172.21.15.38:0/1158542333 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc0c4016030 con 0x7fc0d4100930 2023-03-09T14:46:03.559 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 1 2023-03-09T14:46:03.561 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.559+0000 7fc0d932e700 1 -- 172.21.15.38:0/1158542333 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fc0bc09f3c0 msgr2=0x7fc0bc0a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:03.561 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.559+0000 7fc0d932e700 1 --2- 172.21.15.38:0/1158542333 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fc0bc09f3c0 0x7fc0bc0a1870 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fc0c800b5c0 tx=0x7fc0c8005fb0 comp rx=0 tx=0).stop 2023-03-09T14:46:03.562 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.559+0000 7fc0d932e700 1 -- 172.21.15.38:0/1158542333 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0d4100930 msgr2=0x7fc0d4195c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:03.562 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.559+0000 7fc0d932e700 1 --2- 172.21.15.38:0/1158542333 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0d4100930 0x7fc0d4195c20 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fc0c400d980 tx=0x7fc0c400dd40 comp rx=0 tx=0).stop 2023-03-09T14:46:03.562 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.559+0000 7fc0d932e700 1 -- 172.21.15.38:0/1158542333 shutdown_connections 2023-03-09T14:46:03.562 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.559+0000 7fc0d932e700 1 --2- 172.21.15.38:0/1158542333 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc0d4100930 0x7fc0d4195c20 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:03.562 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.559+0000 7fc0d932e700 1 --2- 172.21.15.38:0/1158542333 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fc0bc09f3c0 0x7fc0bc0a1870 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:03.562 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.559+0000 7fc0d932e700 1 --2- 172.21.15.38:0/1158542333 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc0d4105b70 0x7fc0d419d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:03.562 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.559+0000 7fc0d932e700 1 --2- 172.21.15.38:0/1158542333 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc0d4103250 0x7fc0d4196160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:03.562 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.559+0000 7fc0d932e700 1 -- 172.21.15.38:0/1158542333 >> 172.21.15.38:0/1158542333 conn(0x7fc0d40fa3d0 msgr2=0x7fc0d4101590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:03.563 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.559+0000 7fc0d932e700 1 -- 172.21.15.38:0/1158542333 shutdown_connections 2023-03-09T14:46:03.563 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.559+0000 7fc0d932e700 1 -- 172.21.15.38:0/1158542333 wait complete. 2023-03-09T14:46:03.574 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-03-09T14:46:03.702 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.700+0000 7fa682791700 1 -- 172.21.15.38:0/1148579002 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa67c105bf0 msgr2=0x7fa67c107fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:03.703 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.700+0000 7fa682791700 1 --2- 172.21.15.38:0/1148579002 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa67c105bf0 0x7fa67c107fd0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fa67400b0a0 tx=0x7fa67400b3b0 comp rx=0 tx=0).stop 2023-03-09T14:46:03.703 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.700+0000 7fa682791700 1 -- 172.21.15.38:0/1148579002 shutdown_connections 2023-03-09T14:46:03.703 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.700+0000 7fa682791700 1 --2- 172.21.15.38:0/1148579002 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa67c105bf0 0x7fa67c107fd0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:03.703 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.700+0000 7fa682791700 1 --2- 172.21.15.38:0/1148579002 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa67c1009b0 0x7fa67c102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:03.703 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.700+0000 7fa682791700 1 --2- 172.21.15.38:0/1148579002 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa67c1032d0 0x7fa67c1056b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:03.704 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.700+0000 7fa682791700 1 -- 172.21.15.38:0/1148579002 >> 172.21.15.38:0/1148579002 conn(0x7fa67c0fa3b0 msgr2=0x7fa67c0fc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:03.704 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.700+0000 7fa682791700 1 -- 172.21.15.38:0/1148579002 shutdown_connections 2023-03-09T14:46:03.704 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.700+0000 7fa682791700 1 -- 172.21.15.38:0/1148579002 wait complete. 2023-03-09T14:46:03.704 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.700+0000 7fa682791700 1 Processor -- start 2023-03-09T14:46:03.704 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.701+0000 7fa682791700 1 -- start start 2023-03-09T14:46:03.706 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.701+0000 7fa682791700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa67c1009b0 0x7fa67c100320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:03.706 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.701+0000 7fa682791700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa67c1032d0 0x7fa67c0fe960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:03.706 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.701+0000 7fa682791700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa67c105bf0 0x7fa67c0feea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:03.706 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.701+0000 7fa682791700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa67c0ff640 con 0x7fa67c105bf0 2023-03-09T14:46:03.706 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.701+0000 7fa682791700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa67c0ff7b0 con 0x7fa67c1032d0 2023-03-09T14:46:03.706 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.701+0000 7fa682791700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa67c19fd90 con 0x7fa67c1009b0 2023-03-09T14:46:03.706 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.701+0000 7fa680d2e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa67c105bf0 0x7fa67c0feea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:03.707 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.701+0000 7fa680d2e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa67c105bf0 0x7fa67c0feea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38552/0 (socket says 172.21.15.38:38552) 2023-03-09T14:46:03.707 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.701+0000 7fa680d2e700 1 -- 172.21.15.38:0/203223053 learned_addr learned my addr 172.21.15.38:0/203223053 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:46:03.707 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.701+0000 7fa67bfff700 1 --2- 172.21.15.38:0/203223053 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa67c1009b0 0x7fa67c100320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:03.707 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.701+0000 7fa67b7fe700 1 --2- 172.21.15.38:0/203223053 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa67c1032d0 0x7fa67c0fe960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:03.707 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.701+0000 7fa680d2e700 1 -- 172.21.15.38:0/203223053 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa67c1009b0 msgr2=0x7fa67c100320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:03.707 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.701+0000 7fa680d2e700 1 --2- 172.21.15.38:0/203223053 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa67c1009b0 0x7fa67c100320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:03.707 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.701+0000 7fa680d2e700 1 -- 172.21.15.38:0/203223053 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa67c1032d0 msgr2=0x7fa67c0fe960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:03.708 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.701+0000 7fa680d2e700 1 --2- 172.21.15.38:0/203223053 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa67c1032d0 0x7fa67c0fe960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:03.708 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.701+0000 7fa680d2e700 1 -- 172.21.15.38:0/203223053 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa674009d00 con 0x7fa67c105bf0 2023-03-09T14:46:03.708 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.702+0000 7fa680d2e700 1 --2- 172.21.15.38:0/203223053 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa67c105bf0 0x7fa67c0feea0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fa67400b070 tx=0x7fa674004a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:03.708 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.702+0000 7fa6797fa700 1 -- 172.21.15.38:0/203223053 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa67400e040 con 0x7fa67c105bf0 2023-03-09T14:46:03.708 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.702+0000 7fa6797fa700 1 -- 172.21.15.38:0/203223053 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa674004dd0 con 0x7fa67c105bf0 2023-03-09T14:46:03.708 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.702+0000 7fa682791700 1 -- 172.21.15.38:0/203223053 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa67c10b820 con 0x7fa67c105bf0 2023-03-09T14:46:03.708 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.702+0000 7fa6797fa700 1 -- 172.21.15.38:0/203223053 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa67401b6b0 con 0x7fa67c105bf0 2023-03-09T14:46:03.708 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.702+0000 7fa682791700 1 -- 172.21.15.38:0/203223053 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa67c10b690 con 0x7fa67c105bf0 2023-03-09T14:46:03.709 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.703+0000 7fa682791700 1 -- 172.21.15.38:0/203223053 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa67c0fbf20 con 0x7fa67c105bf0 2023-03-09T14:46:03.709 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.704+0000 7fa6797fa700 1 -- 172.21.15.38:0/203223053 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) v1 ==== 133198+0+0 (secure 0 0 0) 0x7fa67400bd70 con 0x7fa67c105bf0 2023-03-09T14:46:03.709 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.705+0000 7fa6797fa700 1 --2- 172.21.15.38:0/203223053 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fa66409f3c0 0x7fa6640a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:03.710 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.705+0000 7fa67bfff700 1 --2- 172.21.15.38:0/203223053 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fa66409f3c0 0x7fa6640a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:03.710 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.705+0000 7fa6797fa700 1 -- 172.21.15.38:0/203223053 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fa6740cb500 con 0x7fa67c105bf0 2023-03-09T14:46:03.710 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.705+0000 7fa67bfff700 1 --2- 172.21.15.38:0/203223053 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fa66409f3c0 0x7fa6640a1870 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fa66c005950 tx=0x7fa66c00b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:03.711 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.709+0000 7fa6797fa700 1 -- 172.21.15.38:0/203223053 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7fa6740955e0 con 0x7fa67c105bf0 2023-03-09T14:46:03.910 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.908+0000 7fa682791700 1 -- 172.21.15.38:0/203223053 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa67c067660 con 0x7fa67c105bf0 2023-03-09T14:46:03.910 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.908+0000 7fa6797fa700 1 -- 172.21.15.38:0/203223053 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa674098c00 con 0x7fa67c105bf0 2023-03-09T14:46:03.911 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 32 2023-03-09T14:46:03.913 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.910+0000 7fa682791700 1 -- 172.21.15.38:0/203223053 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fa66409f3c0 msgr2=0x7fa6640a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:03.913 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.910+0000 7fa682791700 1 --2- 172.21.15.38:0/203223053 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fa66409f3c0 0x7fa6640a1870 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fa66c005950 tx=0x7fa66c00b410 comp rx=0 tx=0).stop 2023-03-09T14:46:03.913 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.910+0000 7fa682791700 1 -- 172.21.15.38:0/203223053 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa67c105bf0 msgr2=0x7fa67c0feea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:03.913 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.910+0000 7fa682791700 1 --2- 172.21.15.38:0/203223053 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa67c105bf0 0x7fa67c0feea0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fa67400b070 tx=0x7fa674004a50 comp rx=0 tx=0).stop 2023-03-09T14:46:03.913 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.910+0000 7fa682791700 1 -- 172.21.15.38:0/203223053 shutdown_connections 2023-03-09T14:46:03.913 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.910+0000 7fa682791700 1 --2- 172.21.15.38:0/203223053 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa67c105bf0 0x7fa67c0feea0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:03.914 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.910+0000 7fa682791700 1 --2- 172.21.15.38:0/203223053 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fa66409f3c0 0x7fa6640a1870 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:03.914 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.910+0000 7fa682791700 1 --2- 172.21.15.38:0/203223053 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa67c1009b0 0x7fa67c100320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:03.914 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.910+0000 7fa682791700 1 --2- 172.21.15.38:0/203223053 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa67c1032d0 0x7fa67c0fe960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:03.914 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.910+0000 7fa682791700 1 -- 172.21.15.38:0/203223053 >> 172.21.15.38:0/203223053 conn(0x7fa67c0fa3b0 msgr2=0x7fa67c101610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:03.914 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.910+0000 7fa682791700 1 -- 172.21.15.38:0/203223053 shutdown_connections 2023-03-09T14:46:03.914 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:03.910+0000 7fa682791700 1 -- 172.21.15.38:0/203223053 wait complete. 2023-03-09T14:46:03.925 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-03-09T14:46:04.054 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.051+0000 7fa288982700 1 -- 172.21.15.38:0/2492277645 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2840fff80 msgr2=0x7fa2841003d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:04.054 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.051+0000 7fa288982700 1 --2- 172.21.15.38:0/2492277645 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2840fff80 0x7fa2841003d0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fa278009990 tx=0x7fa278009ca0 comp rx=0 tx=0).stop 2023-03-09T14:46:04.054 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.052+0000 7fa288982700 1 -- 172.21.15.38:0/2492277645 shutdown_connections 2023-03-09T14:46:04.055 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.052+0000 7fa288982700 1 --2- 172.21.15.38:0/2492277645 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa284100910 0x7fa2841071d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:04.055 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.052+0000 7fa288982700 1 --2- 172.21.15.38:0/2492277645 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa2840fed90 0x7fa2840ff1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:04.055 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.052+0000 7fa288982700 1 --2- 172.21.15.38:0/2492277645 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2840fff80 0x7fa2841003d0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:04.055 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.052+0000 7fa288982700 1 -- 172.21.15.38:0/2492277645 >> 172.21.15.38:0/2492277645 conn(0x7fa2840fa3b0 msgr2=0x7fa2840fc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:04.055 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.052+0000 7fa288982700 1 -- 172.21.15.38:0/2492277645 shutdown_connections 2023-03-09T14:46:04.055 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.052+0000 7fa288982700 1 -- 172.21.15.38:0/2492277645 wait complete. 2023-03-09T14:46:04.057 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.052+0000 7fa288982700 1 Processor -- start 2023-03-09T14:46:04.057 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.052+0000 7fa288982700 1 -- start start 2023-03-09T14:46:04.057 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.053+0000 7fa288982700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa2840fed90 0x7fa2840685d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:04.057 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.053+0000 7fa288982700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2840fff80 0x7fa28419a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:04.058 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.053+0000 7fa288982700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa284100910 0x7fa28419cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:04.058 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.053+0000 7fa288982700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa28419d770 con 0x7fa2840fff80 2023-03-09T14:46:04.058 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.053+0000 7fa288982700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa28419d8e0 con 0x7fa2840fed90 2023-03-09T14:46:04.058 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.053+0000 7fa288982700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa28419daa0 con 0x7fa284100910 2023-03-09T14:46:04.058 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.053+0000 7fa281d9b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2840fff80 0x7fa28419a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:04.058 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.053+0000 7fa281d9b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2840fff80 0x7fa28419a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38564/0 (socket says 172.21.15.38:38564) 2023-03-09T14:46:04.058 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.053+0000 7fa281d9b700 1 -- 172.21.15.38:0/712998236 learned_addr learned my addr 172.21.15.38:0/712998236 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:46:04.058 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.053+0000 7fa28259c700 1 --2- 172.21.15.38:0/712998236 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa2840fed90 0x7fa2840685d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:04.059 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.053+0000 7fa282d9d700 1 --2- 172.21.15.38:0/712998236 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa284100910 0x7fa28419cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:04.059 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.053+0000 7fa281d9b700 1 -- 172.21.15.38:0/712998236 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa284100910 msgr2=0x7fa28419cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:04.059 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.053+0000 7fa281d9b700 1 --2- 172.21.15.38:0/712998236 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa284100910 0x7fa28419cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:04.059 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.053+0000 7fa281d9b700 1 -- 172.21.15.38:0/712998236 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa2840fed90 msgr2=0x7fa2840685d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:04.059 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.053+0000 7fa281d9b700 1 --2- 172.21.15.38:0/712998236 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa2840fed90 0x7fa2840685d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:04.059 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.053+0000 7fa281d9b700 1 -- 172.21.15.38:0/712998236 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2780095f0 con 0x7fa2840fff80 2023-03-09T14:46:04.059 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.053+0000 7fa282d9d700 1 --2- 172.21.15.38:0/712998236 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa284100910 0x7fa28419cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:46:04.060 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.053+0000 7fa281d9b700 1 --2- 172.21.15.38:0/712998236 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2840fff80 0x7fa28419a950 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fa278004760 tx=0x7fa278004790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:04.060 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.053+0000 7fa2737fe700 1 -- 172.21.15.38:0/712998236 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa27801d070 con 0x7fa2840fff80 2023-03-09T14:46:04.060 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.053+0000 7fa2737fe700 1 -- 172.21.15.38:0/712998236 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa278004cf0 con 0x7fa2840fff80 2023-03-09T14:46:04.060 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.053+0000 7fa288982700 1 -- 172.21.15.38:0/712998236 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa28410a6f0 con 0x7fa2840fff80 2023-03-09T14:46:04.060 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.053+0000 7fa2737fe700 1 -- 172.21.15.38:0/712998236 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa278005280 con 0x7fa2840fff80 2023-03-09T14:46:04.060 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.054+0000 7fa288982700 1 -- 172.21.15.38:0/712998236 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa28410a560 con 0x7fa2840fff80 2023-03-09T14:46:04.060 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.054+0000 7fa288982700 1 -- 172.21.15.38:0/712998236 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa284105d10 con 0x7fa2840fff80 2023-03-09T14:46:04.061 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.056+0000 7fa2737fe700 1 -- 172.21.15.38:0/712998236 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) v1 ==== 133198+0+0 (secure 0 0 0) 0x7fa27800c040 con 0x7fa2840fff80 2023-03-09T14:46:04.061 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.056+0000 7fa2737fe700 1 --2- 172.21.15.38:0/712998236 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fa26c09f3c0 0x7fa26c0a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:04.061 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.057+0000 7fa28259c700 1 --2- 172.21.15.38:0/712998236 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fa26c09f3c0 0x7fa26c0a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:04.062 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.057+0000 7fa2737fe700 1 -- 172.21.15.38:0/712998236 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fa2780cb560 con 0x7fa2840fff80 2023-03-09T14:46:04.062 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.057+0000 7fa28259c700 1 --2- 172.21.15.38:0/712998236 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fa26c09f3c0 0x7fa26c0a1870 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa2840ffde0 tx=0x7fa27400b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:04.062 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.060+0000 7fa2737fe700 1 -- 172.21.15.38:0/712998236 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7fa2780956a0 con 0x7fa2840fff80 2023-03-09T14:46:04.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:03 smithi122 ceph-mon[93869]: from='client.15453 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-03-09T14:46:04.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:03 smithi122 ceph-mon[93869]: pgmap v313: 129 pgs: 129 active+clean; 641 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 682 B/s wr, 6 op/s 2023-03-09T14:46:04.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:03 smithi122 ceph-mon[93869]: from='client.15459 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-03-09T14:46:04.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:03 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/3091986615' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-03-09T14:46:04.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:03 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/1158542333' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-03-09T14:46:04.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:03 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/203223053' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-03-09T14:46:04.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:03 smithi181 ceph-mon[94189]: from='client.15453 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-03-09T14:46:04.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:03 smithi181 ceph-mon[94189]: pgmap v313: 129 pgs: 129 active+clean; 641 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 682 B/s wr, 6 op/s 2023-03-09T14:46:04.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:03 smithi181 ceph-mon[94189]: from='client.15459 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-03-09T14:46:04.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:03 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/3091986615' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-03-09T14:46:04.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:03 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/1158542333' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-03-09T14:46:04.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:03 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/203223053' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-03-09T14:46:04.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:03 smithi038 ceph-mon[92422]: from='client.15453 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-03-09T14:46:04.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:03 smithi038 ceph-mon[92422]: pgmap v313: 129 pgs: 129 active+clean; 641 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 682 B/s wr, 6 op/s 2023-03-09T14:46:04.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:03 smithi038 ceph-mon[92422]: from='client.15459 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-03-09T14:46:04.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:03 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/3091986615' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-03-09T14:46:04.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:03 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/1158542333' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-03-09T14:46:04.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:03 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/203223053' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-03-09T14:46:04.263 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.260+0000 7fa288982700 1 -- 172.21.15.38:0/712998236 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa284061390 con 0x7fa2840fff80 2023-03-09T14:46:04.263 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.261+0000 7fa2737fe700 1 -- 172.21.15.38:0/712998236 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa278030020 con 0x7fa2840fff80 2023-03-09T14:46:04.264 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 64 2023-03-09T14:46:04.265 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.263+0000 7fa288982700 1 -- 172.21.15.38:0/712998236 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fa26c09f3c0 msgr2=0x7fa26c0a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:04.266 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.263+0000 7fa288982700 1 --2- 172.21.15.38:0/712998236 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fa26c09f3c0 0x7fa26c0a1870 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa2840ffde0 tx=0x7fa27400b410 comp rx=0 tx=0).stop 2023-03-09T14:46:04.266 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.263+0000 7fa288982700 1 -- 172.21.15.38:0/712998236 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2840fff80 msgr2=0x7fa28419a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:04.266 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.263+0000 7fa288982700 1 --2- 172.21.15.38:0/712998236 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2840fff80 0x7fa28419a950 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fa278004760 tx=0x7fa278004790 comp rx=0 tx=0).stop 2023-03-09T14:46:04.266 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.263+0000 7fa288982700 1 -- 172.21.15.38:0/712998236 shutdown_connections 2023-03-09T14:46:04.266 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.263+0000 7fa288982700 1 --2- 172.21.15.38:0/712998236 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fa26c09f3c0 0x7fa26c0a1870 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:04.266 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.263+0000 7fa288982700 1 --2- 172.21.15.38:0/712998236 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa284100910 0x7fa28419cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:04.267 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.263+0000 7fa288982700 1 --2- 172.21.15.38:0/712998236 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa2840fed90 0x7fa2840685d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:04.267 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.263+0000 7fa288982700 1 --2- 172.21.15.38:0/712998236 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa2840fff80 0x7fa28419a950 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:04.267 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.263+0000 7fa288982700 1 -- 172.21.15.38:0/712998236 >> 172.21.15.38:0/712998236 conn(0x7fa2840fa3b0 msgr2=0x7fa2840fc640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:04.267 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.263+0000 7fa288982700 1 -- 172.21.15.38:0/712998236 shutdown_connections 2023-03-09T14:46:04.267 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.263+0000 7fa288982700 1 -- 172.21.15.38:0/712998236 wait complete. 2023-03-09T14:46:04.278 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-03-09T14:46:04.407 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.405+0000 7f4600ccc700 1 -- 172.21.15.38:0/1483308614 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f45fc0ff100 msgr2=0x7f45fc0ff570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:04.408 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.405+0000 7f4600ccc700 1 --2- 172.21.15.38:0/1483308614 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f45fc0ff100 0x7f45fc0ff570 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f45ec009990 tx=0x7f45ec009ca0 comp rx=0 tx=0).stop 2023-03-09T14:46:04.408 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.405+0000 7f4600ccc700 1 -- 172.21.15.38:0/1483308614 shutdown_connections 2023-03-09T14:46:04.408 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.405+0000 7f4600ccc700 1 --2- 172.21.15.38:0/1483308614 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f45fc0ffab0 0x7f45fc0fff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:04.408 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.405+0000 7f4600ccc700 1 --2- 172.21.15.38:0/1483308614 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f45fc0fe7b0 0x7f45fc0febc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:04.408 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.405+0000 7f4600ccc700 1 --2- 172.21.15.38:0/1483308614 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f45fc0ff100 0x7f45fc0ff570 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:04.409 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.405+0000 7f4600ccc700 1 -- 172.21.15.38:0/1483308614 >> 172.21.15.38:0/1483308614 conn(0x7f45fc0fa3e0 msgr2=0x7f45fc0fc810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:04.409 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.405+0000 7f4600ccc700 1 -- 172.21.15.38:0/1483308614 shutdown_connections 2023-03-09T14:46:04.409 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.405+0000 7f4600ccc700 1 -- 172.21.15.38:0/1483308614 wait complete. 2023-03-09T14:46:04.410 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.406+0000 7f4600ccc700 1 Processor -- start 2023-03-09T14:46:04.411 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.406+0000 7f4600ccc700 1 -- start start 2023-03-09T14:46:04.411 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.406+0000 7f4600ccc700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f45fc0fe7b0 0x7f45fc197de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:04.411 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.406+0000 7f4600ccc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f45fc0ff100 0x7f45fc198320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:04.411 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.406+0000 7f4600ccc700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f45fc0ffab0 0x7f45fc19f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:04.411 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.406+0000 7f4600ccc700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45fc19f960 con 0x7f45fc0ff100 2023-03-09T14:46:04.411 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.406+0000 7f4600ccc700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45fc19faa0 con 0x7f45fc0ffab0 2023-03-09T14:46:04.411 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.406+0000 7f4600ccc700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45fc19fc30 con 0x7f45fc0fe7b0 2023-03-09T14:46:04.412 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.406+0000 7f45f9d9b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f45fc0ff100 0x7f45fc198320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:04.412 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.406+0000 7f45f9d9b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f45fc0ff100 0x7f45fc198320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38578/0 (socket says 172.21.15.38:38578) 2023-03-09T14:46:04.412 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.406+0000 7f45fa59c700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f45fc0fe7b0 0x7f45fc197de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:04.412 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.406+0000 7f45f9d9b700 1 -- 172.21.15.38:0/1470870671 learned_addr learned my addr 172.21.15.38:0/1470870671 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:46:04.412 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.407+0000 7f45fad9d700 1 --2- 172.21.15.38:0/1470870671 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f45fc0ffab0 0x7f45fc19f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:04.412 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.407+0000 7f45f9d9b700 1 -- 172.21.15.38:0/1470870671 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f45fc0fe7b0 msgr2=0x7f45fc197de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:04.412 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.407+0000 7f45f9d9b700 1 --2- 172.21.15.38:0/1470870671 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f45fc0fe7b0 0x7f45fc197de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:04.413 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.407+0000 7f45f9d9b700 1 -- 172.21.15.38:0/1470870671 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f45fc0ffab0 msgr2=0x7f45fc19f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:04.413 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.407+0000 7f45f9d9b700 1 --2- 172.21.15.38:0/1470870671 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f45fc0ffab0 0x7f45fc19f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:04.413 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.407+0000 7f45f9d9b700 1 -- 172.21.15.38:0/1470870671 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45ec0095f0 con 0x7f45fc0ff100 2023-03-09T14:46:04.413 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.407+0000 7f45f9d9b700 1 --2- 172.21.15.38:0/1470870671 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f45fc0ff100 0x7f45fc198320 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f45ec000c00 tx=0x7f45ec0047a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:04.413 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.407+0000 7f45eb7fe700 1 -- 172.21.15.38:0/1470870671 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45ec01d070 con 0x7f45fc0ff100 2023-03-09T14:46:04.413 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.407+0000 7f4600ccc700 1 -- 172.21.15.38:0/1470870671 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45fc10da20 con 0x7f45fc0ff100 2023-03-09T14:46:04.413 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.407+0000 7f45eb7fe700 1 -- 172.21.15.38:0/1470870671 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f45ec022470 con 0x7f45fc0ff100 2023-03-09T14:46:04.414 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.407+0000 7f45eb7fe700 1 -- 172.21.15.38:0/1470870671 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45ec017650 con 0x7f45fc0ff100 2023-03-09T14:46:04.414 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.407+0000 7f4600ccc700 1 -- 172.21.15.38:0/1470870671 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45fc10d890 con 0x7f45fc0ff100 2023-03-09T14:46:04.414 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.408+0000 7f4600ccc700 1 -- 172.21.15.38:0/1470870671 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45fc109590 con 0x7f45fc0ff100 2023-03-09T14:46:04.414 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.409+0000 7f45eb7fe700 1 -- 172.21.15.38:0/1470870671 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) v1 ==== 133198+0+0 (secure 0 0 0) 0x7f45ec029030 con 0x7f45fc0ff100 2023-03-09T14:46:04.415 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.410+0000 7f45eb7fe700 1 --2- 172.21.15.38:0/1470870671 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f45e409f3c0 0x7f45e40a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:04.415 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.410+0000 7f45fa59c700 1 --2- 172.21.15.38:0/1470870671 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f45e409f3c0 0x7f45e40a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:04.415 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.410+0000 7f45eb7fe700 1 -- 172.21.15.38:0/1470870671 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f45ec0cb640 con 0x7f45fc0ff100 2023-03-09T14:46:04.415 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.410+0000 7f45fa59c700 1 --2- 172.21.15.38:0/1470870671 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f45e409f3c0 0x7f45e40a1870 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f45f0009c40 tx=0x7f45f0009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:04.416 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.414+0000 7f45eb7fe700 1 -- 172.21.15.38:0/1470870671 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f45ec095720 con 0x7f45fc0ff100 2023-03-09T14:46:04.618 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.616+0000 7f4600ccc700 1 -- 172.21.15.38:0/1470870671 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f45fc0618b0 con 0x7f45fc0ff100 2023-03-09T14:46:04.619 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.617+0000 7f45eb7fe700 1 -- 172.21.15.38:0/1470870671 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) v1 ==== 103+0+11 (secure 0 0 0) 0x7f45ec0270a0 con 0x7f45fc0ff100 2023-03-09T14:46:04.619 INFO:teuthology.orchestra.run.smithi038.stdout:pg_num: 32 2023-03-09T14:46:04.621 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.619+0000 7f4600ccc700 1 -- 172.21.15.38:0/1470870671 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f45e409f3c0 msgr2=0x7f45e40a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:04.621 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.619+0000 7f4600ccc700 1 --2- 172.21.15.38:0/1470870671 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f45e409f3c0 0x7f45e40a1870 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f45f0009c40 tx=0x7f45f0009260 comp rx=0 tx=0).stop 2023-03-09T14:46:04.622 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.619+0000 7f4600ccc700 1 -- 172.21.15.38:0/1470870671 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f45fc0ff100 msgr2=0x7f45fc198320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:04.622 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.619+0000 7f4600ccc700 1 --2- 172.21.15.38:0/1470870671 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f45fc0ff100 0x7f45fc198320 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f45ec000c00 tx=0x7f45ec0047a0 comp rx=0 tx=0).stop 2023-03-09T14:46:04.622 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.619+0000 7f4600ccc700 1 -- 172.21.15.38:0/1470870671 shutdown_connections 2023-03-09T14:46:04.622 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.619+0000 7f4600ccc700 1 --2- 172.21.15.38:0/1470870671 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f45e409f3c0 0x7f45e40a1870 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:04.622 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.619+0000 7f4600ccc700 1 --2- 172.21.15.38:0/1470870671 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f45fc0ffab0 0x7f45fc19f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:04.622 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.619+0000 7f4600ccc700 1 --2- 172.21.15.38:0/1470870671 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f45fc0fe7b0 0x7f45fc197de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:04.622 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.619+0000 7f4600ccc700 1 --2- 172.21.15.38:0/1470870671 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f45fc0ff100 0x7f45fc198320 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:04.623 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.619+0000 7f4600ccc700 1 -- 172.21.15.38:0/1470870671 >> 172.21.15.38:0/1470870671 conn(0x7f45fc0fa3e0 msgr2=0x7f45fc0fc0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:04.623 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.619+0000 7f4600ccc700 1 -- 172.21.15.38:0/1470870671 shutdown_connections 2023-03-09T14:46:04.623 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.619+0000 7f4600ccc700 1 -- 172.21.15.38:0/1470870671 wait complete. 2023-03-09T14:46:04.634 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-09T14:46:04.762 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.757+0000 7fda78027700 1 -- 172.21.15.38:0/129563267 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fda700fff70 msgr2=0x7fda701003c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:04.762 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.757+0000 7fda78027700 1 --2- 172.21.15.38:0/129563267 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fda700fff70 0x7fda701003c0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fda64009990 tx=0x7fda64009ca0 comp rx=0 tx=0).stop 2023-03-09T14:46:04.763 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.757+0000 7fda78027700 1 -- 172.21.15.38:0/129563267 shutdown_connections 2023-03-09T14:46:04.763 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.757+0000 7fda78027700 1 --2- 172.21.15.38:0/129563267 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fda70100900 0x7fda701071c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:04.763 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.757+0000 7fda78027700 1 --2- 172.21.15.38:0/129563267 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fda700fed80 0x7fda700ff190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:04.763 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.757+0000 7fda78027700 1 --2- 172.21.15.38:0/129563267 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fda700fff70 0x7fda701003c0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:04.763 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.757+0000 7fda78027700 1 -- 172.21.15.38:0/129563267 >> 172.21.15.38:0/129563267 conn(0x7fda700fa380 msgr2=0x7fda700fc7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:04.764 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.757+0000 7fda78027700 1 -- 172.21.15.38:0/129563267 shutdown_connections 2023-03-09T14:46:04.764 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.757+0000 7fda78027700 1 -- 172.21.15.38:0/129563267 wait complete. 2023-03-09T14:46:04.764 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.758+0000 7fda78027700 1 Processor -- start 2023-03-09T14:46:04.764 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.758+0000 7fda78027700 1 -- start start 2023-03-09T14:46:04.764 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.758+0000 7fda78027700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fda700fed80 0x7fda70102d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:04.764 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.758+0000 7fda78027700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fda700fff70 0x7fda70101350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:04.765 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.758+0000 7fda78027700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fda70100900 0x7fda70101890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:04.765 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.758+0000 7fda78027700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda70102030 con 0x7fda700fff70 2023-03-09T14:46:04.765 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.758+0000 7fda78027700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda701021a0 con 0x7fda70100900 2023-03-09T14:46:04.765 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.758+0000 7fda78027700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda7019fc60 con 0x7fda700fed80 2023-03-09T14:46:04.765 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.758+0000 7fda755c2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fda700fff70 0x7fda70101350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:04.766 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.758+0000 7fda755c2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fda700fff70 0x7fda70101350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38592/0 (socket says 172.21.15.38:38592) 2023-03-09T14:46:04.766 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.758+0000 7fda755c2700 1 -- 172.21.15.38:0/656546350 learned_addr learned my addr 172.21.15.38:0/656546350 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:46:04.766 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.758+0000 7fda75dc3700 1 --2- 172.21.15.38:0/656546350 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fda700fed80 0x7fda70102d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:04.766 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.758+0000 7fda765c4700 1 --2- 172.21.15.38:0/656546350 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fda70100900 0x7fda70101890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:04.766 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.758+0000 7fda755c2700 1 -- 172.21.15.38:0/656546350 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fda700fed80 msgr2=0x7fda70102d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:04.767 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.758+0000 7fda755c2700 1 --2- 172.21.15.38:0/656546350 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fda700fed80 0x7fda70102d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:04.767 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.758+0000 7fda755c2700 1 -- 172.21.15.38:0/656546350 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fda70100900 msgr2=0x7fda70101890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:04.767 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.758+0000 7fda755c2700 1 --2- 172.21.15.38:0/656546350 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fda70100900 0x7fda70101890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:04.767 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.758+0000 7fda755c2700 1 -- 172.21.15.38:0/656546350 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda640095f0 con 0x7fda700fff70 2023-03-09T14:46:04.768 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.758+0000 7fda75dc3700 1 --2- 172.21.15.38:0/656546350 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fda700fed80 0x7fda70102d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:46:04.768 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.759+0000 7fda755c2700 1 --2- 172.21.15.38:0/656546350 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fda700fff70 0x7fda70101350 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fda64000c00 tx=0x7fda640047a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:04.768 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.759+0000 7fda62ffd700 1 -- 172.21.15.38:0/656546350 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda6401d070 con 0x7fda700fff70 2023-03-09T14:46:04.769 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.759+0000 7fda78027700 1 -- 172.21.15.38:0/656546350 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda7010a6e0 con 0x7fda700fff70 2023-03-09T14:46:04.769 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.759+0000 7fda62ffd700 1 -- 172.21.15.38:0/656546350 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fda64022470 con 0x7fda700fff70 2023-03-09T14:46:04.769 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.759+0000 7fda62ffd700 1 -- 172.21.15.38:0/656546350 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda64017650 con 0x7fda700fff70 2023-03-09T14:46:04.769 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.759+0000 7fda78027700 1 -- 172.21.15.38:0/656546350 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda7010a550 con 0x7fda700fff70 2023-03-09T14:46:04.769 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.760+0000 7fda78027700 1 -- 172.21.15.38:0/656546350 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda701a0170 con 0x7fda700fff70 2023-03-09T14:46:04.771 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.761+0000 7fda62ffd700 1 -- 172.21.15.38:0/656546350 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) v1 ==== 133198+0+0 (secure 0 0 0) 0x7fda64029030 con 0x7fda700fff70 2023-03-09T14:46:04.771 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.761+0000 7fda62ffd700 1 --2- 172.21.15.38:0/656546350 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fda5c09f3c0 0x7fda5c0a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:04.771 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.762+0000 7fda75dc3700 1 --2- 172.21.15.38:0/656546350 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fda5c09f3c0 0x7fda5c0a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:04.772 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.762+0000 7fda62ffd700 1 -- 172.21.15.38:0/656546350 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fda640cb970 con 0x7fda700fff70 2023-03-09T14:46:04.772 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.762+0000 7fda75dc3700 1 --2- 172.21.15.38:0/656546350 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fda5c09f3c0 0x7fda5c0a1870 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fda6c005950 tx=0x7fda6c0058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:04.772 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:04.765+0000 7fda62ffd700 1 -- 172.21.15.38:0/656546350 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7fda64095ab0 con 0x7fda700fff70 2023-03-09T14:46:05.017 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.015+0000 7fda78027700 1 -- 172.21.15.38:0/656546350 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fda701a0720 con 0x7fda700fff70 2023-03-09T14:46:05.018 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.015+0000 7fda62ffd700 1 -- 172.21.15.38:0/656546350 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+10801 (secure 0 0 0) 0x7fda640990d0 con 0x7fda700fff70 2023-03-09T14:46:05.019 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:46:05.019 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/768918602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":768918602},{"type":"v1","addr":"172.21.15.38:6835","nonce":768918602}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/18381540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":18381540},{"type":"v1","addr":"172.21.15.38:6837","nonce":18381540}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14610,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6839/4139020066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":4139020066},{"type":"v1","addr":"172.21.15.38:6839","nonce":4139020066}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14622,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6841/1397115322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":1397115322},{"type":"v1","addr":"172.21.15.38:6841","nonce":1397115322}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6833/415305571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":415305571},{"type":"v1","addr":"172.21.15.181:6833","nonce":415305571}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/659300520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":659300520},{"type":"v1","addr":"172.21.15.122:6833","nonce":659300520}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-03-09T14:44:19.083770+0000","modified":"2023-03-09T14:44:39.526150+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24530,"mds_1":24494,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24448":{"gid":24448,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6835/2576255861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":2576255861},{"type":"v1","addr":"172.21.15.181:6835","nonce":2576255861}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24460":{"gid":24460,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6837/2942560181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":2942560181},{"type":"v1","addr":"172.21.15.181:6837","nonce":2942560181}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24464":{"gid":24464,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.122:6835/3675537941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":3675537941},{"type":"v1","addr":"172.21.15.122:6835","nonce":3675537941}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24478":{"gid":24478,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":9,"addr":"172.21.15.181:6839/3902263660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":3902263660},{"type":"v1","addr":"172.21.15.181:6839","nonce":3902263660}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24494":{"gid":24494,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.122:6837/2210214596","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":2210214596},{"type":"v1","addr":"172.21.15.122:6837","nonce":2210214596}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24530":{"gid":24530,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6839/971050165","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6838","nonce":971050165},{"type":"v1","addr":"172.21.15.122:6839","nonce":971050165}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-09T14:46:05.020 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.018+0000 7fda78027700 1 -- 172.21.15.38:0/656546350 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fda5c09f3c0 msgr2=0x7fda5c0a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:05.020 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.018+0000 7fda78027700 1 --2- 172.21.15.38:0/656546350 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fda5c09f3c0 0x7fda5c0a1870 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fda6c005950 tx=0x7fda6c0058e0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.020 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.018+0000 7fda78027700 1 -- 172.21.15.38:0/656546350 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fda700fff70 msgr2=0x7fda70101350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:05.021 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.018+0000 7fda78027700 1 --2- 172.21.15.38:0/656546350 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fda700fff70 0x7fda70101350 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fda64000c00 tx=0x7fda640047a0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.021 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.018+0000 7fda78027700 1 -- 172.21.15.38:0/656546350 shutdown_connections 2023-03-09T14:46:05.021 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.018+0000 7fda78027700 1 --2- 172.21.15.38:0/656546350 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fda5c09f3c0 0x7fda5c0a1870 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.021 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.018+0000 7fda78027700 1 --2- 172.21.15.38:0/656546350 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fda70100900 0x7fda70101890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.021 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.018+0000 7fda78027700 1 --2- 172.21.15.38:0/656546350 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fda700fed80 0x7fda70102d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.022 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.018+0000 7fda78027700 1 --2- 172.21.15.38:0/656546350 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fda700fff70 0x7fda70101350 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.022 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.018+0000 7fda78027700 1 -- 172.21.15.38:0/656546350 >> 172.21.15.38:0/656546350 conn(0x7fda700fa380 msgr2=0x7fda70067a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:05.022 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.018+0000 7fda78027700 1 -- 172.21.15.38:0/656546350 shutdown_connections 2023-03-09T14:46:05.022 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.018+0000 7fda78027700 1 -- 172.21.15.38:0/656546350 wait complete. 2023-03-09T14:46:05.022 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 23 2023-03-09T14:46:05.033 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-09T14:46:05.158 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.156+0000 7ff414b76700 1 -- 172.21.15.38:0/3653748664 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff410100ff0 msgr2=0x7ff410101460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:05.159 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.156+0000 7ff414b76700 1 --2- 172.21.15.38:0/3653748664 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff410100ff0 0x7ff410101460 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7ff400009990 tx=0x7ff400009ca0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.159 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.156+0000 7ff414b76700 1 -- 172.21.15.38:0/3653748664 shutdown_connections 2023-03-09T14:46:05.159 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.156+0000 7ff414b76700 1 --2- 172.21.15.38:0/3653748664 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4101019a0 0x7ff4101082d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.159 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.156+0000 7ff414b76700 1 --2- 172.21.15.38:0/3653748664 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4100ffea0 0x7ff4101002b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.159 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.156+0000 7ff414b76700 1 --2- 172.21.15.38:0/3653748664 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff410100ff0 0x7ff410101460 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.159 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.156+0000 7ff414b76700 1 -- 172.21.15.38:0/3653748664 >> 172.21.15.38:0/3653748664 conn(0x7ff4100fb480 msgr2=0x7ff4100fd8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:05.160 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.156+0000 7ff414b76700 1 -- 172.21.15.38:0/3653748664 shutdown_connections 2023-03-09T14:46:05.160 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.156+0000 7ff414b76700 1 -- 172.21.15.38:0/3653748664 wait complete. 2023-03-09T14:46:05.161 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.157+0000 7ff414b76700 1 Processor -- start 2023-03-09T14:46:05.162 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.157+0000 7ff414b76700 1 -- start start 2023-03-09T14:46:05.162 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.157+0000 7ff414b76700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4100ffea0 0x7ff410195c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:05.162 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.157+0000 7ff414b76700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff410100ff0 0x7ff410196170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:05.162 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.157+0000 7ff414b76700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4101019a0 0x7ff41019d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:05.162 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.157+0000 7ff414b76700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff41019d770 con 0x7ff410100ff0 2023-03-09T14:46:05.162 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.157+0000 7ff414b76700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff41019d8e0 con 0x7ff4100ffea0 2023-03-09T14:46:05.163 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.157+0000 7ff414b76700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff41019daa0 con 0x7ff4101019a0 2023-03-09T14:46:05.163 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.157+0000 7ff40dd9b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff410100ff0 0x7ff410196170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:05.163 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.157+0000 7ff40dd9b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff410100ff0 0x7ff410196170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38606/0 (socket says 172.21.15.38:38606) 2023-03-09T14:46:05.163 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.157+0000 7ff40dd9b700 1 -- 172.21.15.38:0/460798957 learned_addr learned my addr 172.21.15.38:0/460798957 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:46:05.163 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.157+0000 7ff40ed9d700 1 --2- 172.21.15.38:0/460798957 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4101019a0 0x7ff41019d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:05.163 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.157+0000 7ff40e59c700 1 --2- 172.21.15.38:0/460798957 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4100ffea0 0x7ff410195c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:05.163 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.157+0000 7ff40dd9b700 1 -- 172.21.15.38:0/460798957 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4101019a0 msgr2=0x7ff41019d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:05.163 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.157+0000 7ff40dd9b700 1 --2- 172.21.15.38:0/460798957 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4101019a0 0x7ff41019d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.164 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.157+0000 7ff40dd9b700 1 -- 172.21.15.38:0/460798957 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4100ffea0 msgr2=0x7ff410195c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:05.164 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.157+0000 7ff40dd9b700 1 --2- 172.21.15.38:0/460798957 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4100ffea0 0x7ff410195c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.164 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.157+0000 7ff40dd9b700 1 -- 172.21.15.38:0/460798957 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4000095f0 con 0x7ff410100ff0 2023-03-09T14:46:05.164 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.158+0000 7ff40dd9b700 1 --2- 172.21.15.38:0/460798957 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff410100ff0 0x7ff410196170 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7ff400004730 tx=0x7ff400004760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:05.164 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.158+0000 7ff3ff7fe700 1 -- 172.21.15.38:0/460798957 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff40001d070 con 0x7ff410100ff0 2023-03-09T14:46:05.164 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.158+0000 7ff3ff7fe700 1 -- 172.21.15.38:0/460798957 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff400004c90 con 0x7ff410100ff0 2023-03-09T14:46:05.164 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.158+0000 7ff414b76700 1 -- 172.21.15.38:0/460798957 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff41010b850 con 0x7ff410100ff0 2023-03-09T14:46:05.165 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.158+0000 7ff3ff7fe700 1 -- 172.21.15.38:0/460798957 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4000054b0 con 0x7ff410100ff0 2023-03-09T14:46:05.165 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.158+0000 7ff414b76700 1 -- 172.21.15.38:0/460798957 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff41010b6c0 con 0x7ff410100ff0 2023-03-09T14:46:05.165 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.159+0000 7ff414b76700 1 -- 172.21.15.38:0/460798957 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff41019dfe0 con 0x7ff410100ff0 2023-03-09T14:46:05.166 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.160+0000 7ff3ff7fe700 1 -- 172.21.15.38:0/460798957 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) v1 ==== 133198+0+0 (secure 0 0 0) 0x7ff40000b050 con 0x7ff410100ff0 2023-03-09T14:46:05.166 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.160+0000 7ff3ff7fe700 1 --2- 172.21.15.38:0/460798957 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7ff3f809f3c0 0x7ff3f80a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:05.166 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.161+0000 7ff40e59c700 1 --2- 172.21.15.38:0/460798957 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7ff3f809f3c0 0x7ff3f80a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:05.166 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.161+0000 7ff3ff7fe700 1 -- 172.21.15.38:0/460798957 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ff4000cb790 con 0x7ff410100ff0 2023-03-09T14:46:05.166 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.161+0000 7ff40e59c700 1 --2- 172.21.15.38:0/460798957 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7ff3f809f3c0 0x7ff3f80a1870 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7ff410197100 tx=0x7ff404009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:05.167 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.164+0000 7ff3ff7fe700 1 -- 172.21.15.38:0/460798957 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7ff400095870 con 0x7ff410100ff0 2023-03-09T14:46:05.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:04 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/712998236' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-03-09T14:46:05.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:04 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/1470870671' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-03-09T14:46:05.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:04 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/712998236' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-03-09T14:46:05.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:04 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/1470870671' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-03-09T14:46:05.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:04 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/712998236' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-03-09T14:46:05.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:04 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/1470870671' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-03-09T14:46:05.375 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.373+0000 7ff414b76700 1 -- 172.21.15.38:0/460798957 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff410106d70 con 0x7ff410100ff0 2023-03-09T14:46:05.376 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.374+0000 7ff3ff7fe700 1 -- 172.21.15.38:0/460798957 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24445 (secure 0 0 0) 0x7ff40002f090 con 0x7ff410100ff0 2023-03-09T14:46:05.377 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:46:05.377 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":89,"fsid":"c6733fe2-be87-11ed-9af2-001a4aab830c","created":"2023-03-09T14:37:14.947981+0000","modified":"2023-03-09T14:45:15.848936+0000","last_up_change":"2023-03-09T14:43:37.198407+0000","last_in_change":"2023-03-09T14:43:19.550566+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-09T14:40:23.196665+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-03-09T14:44:17.484374+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-03-09T14:44:18.457833+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"83","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-03-09T14:44:21.895589+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"87","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"51477039-15ee-4707-a34d-4ecfa18bebde","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6802","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6803","nonce":1508072290}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6804","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6805","nonce":1508072290}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6808","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6809","nonce":1508072290}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6806","nonce":1508072290},{"type":"v1","addr":"172.21.15.38:6807","nonce":1508072290}]},"public_addr":"172.21.15.38:6803/1508072290","cluster_addr":"172.21.15.38:6805/1508072290","heartbeat_back_addr":"172.21.15.38:6809/1508072290","heartbeat_front_addr":"172.21.15.38:6807/1508072290","state":["exists","up"]},{"osd":1,"uuid":"71f8a718-b0a9-4f39-bc54-267aa42a2578","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6800","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6801","nonce":3929268773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6802","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6803","nonce":3929268773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6806","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6807","nonce":3929268773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6804","nonce":3929268773},{"type":"v1","addr":"172.21.15.122:6805","nonce":3929268773}]},"public_addr":"172.21.15.122:6801/3929268773","cluster_addr":"172.21.15.122:6803/3929268773","heartbeat_back_addr":"172.21.15.122:6807/3929268773","heartbeat_front_addr":"172.21.15.122:6805/3929268773","state":["exists","up"]},{"osd":2,"uuid":"961a2cbe-140e-437f-b131-1fcac573b19c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6800","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6801","nonce":1310630803}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6802","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6803","nonce":1310630803}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6806","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6807","nonce":1310630803}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6804","nonce":1310630803},{"type":"v1","addr":"172.21.15.181:6805","nonce":1310630803}]},"public_addr":"172.21.15.181:6801/1310630803","cluster_addr":"172.21.15.181:6803/1310630803","heartbeat_back_addr":"172.21.15.181:6807/1310630803","heartbeat_front_addr":"172.21.15.181:6805/1310630803","state":["exists","up"]},{"osd":3,"uuid":"969df8fa-2d19-42c7-9f2a-5c5fac15d4a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6810","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6811","nonce":2719814100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6812","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6813","nonce":2719814100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6816","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6817","nonce":2719814100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6814","nonce":2719814100},{"type":"v1","addr":"172.21.15.38:6815","nonce":2719814100}]},"public_addr":"172.21.15.38:6811/2719814100","cluster_addr":"172.21.15.38:6813/2719814100","heartbeat_back_addr":"172.21.15.38:6817/2719814100","heartbeat_front_addr":"172.21.15.38:6815/2719814100","state":["exists","up"]},{"osd":4,"uuid":"44a6dce3-e06e-4cbd-9404-62a86d01ab6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6808","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6809","nonce":2480764994}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6810","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6811","nonce":2480764994}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6814","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6815","nonce":2480764994}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6812","nonce":2480764994},{"type":"v1","addr":"172.21.15.122:6813","nonce":2480764994}]},"public_addr":"172.21.15.122:6809/2480764994","cluster_addr":"172.21.15.122:6811/2480764994","heartbeat_back_addr":"172.21.15.122:6815/2480764994","heartbeat_front_addr":"172.21.15.122:6813/2480764994","state":["exists","up"]},{"osd":5,"uuid":"49749574-c00c-4e3a-80e6-e3686df58d5b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6808","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6809","nonce":3141346246}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6810","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6811","nonce":3141346246}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6814","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6815","nonce":3141346246}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6812","nonce":3141346246},{"type":"v1","addr":"172.21.15.181:6813","nonce":3141346246}]},"public_addr":"172.21.15.181:6809/3141346246","cluster_addr":"172.21.15.181:6811/3141346246","heartbeat_back_addr":"172.21.15.181:6815/3141346246","heartbeat_front_addr":"172.21.15.181:6813/3141346246","state":["exists","up"]},{"osd":6,"uuid":"61c0437e-3f51-45a9-8a61-e98eea0093f8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6818","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6819","nonce":2122788957}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6820","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6821","nonce":2122788957}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6824","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6825","nonce":2122788957}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6822","nonce":2122788957},{"type":"v1","addr":"172.21.15.38:6823","nonce":2122788957}]},"public_addr":"172.21.15.38:6819/2122788957","cluster_addr":"172.21.15.38:6821/2122788957","heartbeat_back_addr":"172.21.15.38:6825/2122788957","heartbeat_front_addr":"172.21.15.38:6823/2122788957","state":["exists","up"]},{"osd":7,"uuid":"231fecdd-939f-4c7b-83a1-c4a1cffd7139","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6816","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6817","nonce":656874139}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6818","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6819","nonce":656874139}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6822","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6823","nonce":656874139}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6820","nonce":656874139},{"type":"v1","addr":"172.21.15.122:6821","nonce":656874139}]},"public_addr":"172.21.15.122:6817/656874139","cluster_addr":"172.21.15.122:6819/656874139","heartbeat_back_addr":"172.21.15.122:6823/656874139","heartbeat_front_addr":"172.21.15.122:6821/656874139","state":["exists","up"]},{"osd":8,"uuid":"297e6d6d-85fd-43ce-9028-5e9dbe55c36e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":85,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6816","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6817","nonce":755000703}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6818","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6819","nonce":755000703}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6822","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6823","nonce":755000703}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6820","nonce":755000703},{"type":"v1","addr":"172.21.15.181:6821","nonce":755000703}]},"public_addr":"172.21.15.181:6817/755000703","cluster_addr":"172.21.15.181:6819/755000703","heartbeat_back_addr":"172.21.15.181:6823/755000703","heartbeat_front_addr":"172.21.15.181:6821/755000703","state":["exists","up"]},{"osd":9,"uuid":"94cc0355-5a1a-4cfc-bd0c-562b6b41127e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6826","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6827","nonce":2624555969}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6828","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6829","nonce":2624555969}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6832","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6833","nonce":2624555969}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6830","nonce":2624555969},{"type":"v1","addr":"172.21.15.38:6831","nonce":2624555969}]},"public_addr":"172.21.15.38:6827/2624555969","cluster_addr":"172.21.15.38:6829/2624555969","heartbeat_back_addr":"172.21.15.38:6833/2624555969","heartbeat_front_addr":"172.21.15.38:6831/2624555969","state":["exists","up"]},{"osd":10,"uuid":"f4f2e042-8d92-46bf-93b8-8b0f04795c2b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":70,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6824","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6825","nonce":1427785280}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6826","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6827","nonce":1427785280}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6830","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6831","nonce":1427785280}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6828","nonce":1427785280},{"type":"v1","addr":"172.21.15.122:6829","nonce":1427785280}]},"public_addr":"172.21.15.122:6825/1427785280","cluster_addr":"172.21.15.122:6827/1427785280","heartbeat_back_addr":"172.21.15.122:6831/1427785280","heartbeat_front_addr":"172.21.15.122:6829/1427785280","state":["exists","up"]},{"osd":11,"uuid":"07444936-eae5-4097-a799-60294cafe80b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":78,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6824","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6825","nonce":3183488640}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6826","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6827","nonce":3183488640}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6830","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6831","nonce":3183488640}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6828","nonce":3183488640},{"type":"v1","addr":"172.21.15.181:6829","nonce":3183488640}]},"public_addr":"172.21.15.181:6825/3183488640","cluster_addr":"172.21.15.181:6827/3183488640","heartbeat_back_addr":"172.21.15.181:6831/3183488640","heartbeat_front_addr":"172.21.15.181:6829/3183488640","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:39:37.657556+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:39:58.960074+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:40:20.326952+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:40:42.244387+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:41:03.946973+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:41:25.097985+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:41:46.877916+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:42:08.413827+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:42:29.822009+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:42:51.776684+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:43:13.937249+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-09T14:43:35.569199+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.2","mappings":[{"from":7,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.d","mappings":[{"from":5,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.17","mappings":[{"from":5,"to":6}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"4.c","mappings":[{"from":9,"to":10}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.38:0/4272288735":"2023-03-10T14:38:14.612767+0000","172.21.15.38:6801/3381625669":"2023-03-10T14:38:14.612767+0000","172.21.15.38:6800/3381625669":"2023-03-10T14:38:14.612767+0000","172.21.15.38:6800/3060474342":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/4253918615":"2023-03-10T14:38:14.612767+0000","172.21.15.38:0/3496083181":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/245602368":"2023-03-10T14:37:47.517775+0000","172.21.15.38:6801/3060474342":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/3622589182":"2023-03-10T14:37:47.517775+0000","172.21.15.38:0/1430302251":"2023-03-10T14:38:14.612767+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-09T14:46:05.379 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.377+0000 7ff414b76700 1 -- 172.21.15.38:0/460798957 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7ff3f809f3c0 msgr2=0x7ff3f80a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:05.379 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.377+0000 7ff414b76700 1 --2- 172.21.15.38:0/460798957 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7ff3f809f3c0 0x7ff3f80a1870 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7ff410197100 tx=0x7ff404009260 comp rx=0 tx=0).stop 2023-03-09T14:46:05.379 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.377+0000 7ff414b76700 1 -- 172.21.15.38:0/460798957 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff410100ff0 msgr2=0x7ff410196170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:05.380 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.377+0000 7ff414b76700 1 --2- 172.21.15.38:0/460798957 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff410100ff0 0x7ff410196170 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7ff400004730 tx=0x7ff400004760 comp rx=0 tx=0).stop 2023-03-09T14:46:05.380 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.377+0000 7ff414b76700 1 -- 172.21.15.38:0/460798957 shutdown_connections 2023-03-09T14:46:05.380 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.377+0000 7ff414b76700 1 --2- 172.21.15.38:0/460798957 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7ff3f809f3c0 0x7ff3f80a1870 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.380 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.377+0000 7ff414b76700 1 --2- 172.21.15.38:0/460798957 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4101019a0 0x7ff41019d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.380 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.377+0000 7ff414b76700 1 --2- 172.21.15.38:0/460798957 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4100ffea0 0x7ff410195c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.380 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.377+0000 7ff414b76700 1 --2- 172.21.15.38:0/460798957 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff410100ff0 0x7ff410196170 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.380 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.377+0000 7ff414b76700 1 -- 172.21.15.38:0/460798957 >> 172.21.15.38:0/460798957 conn(0x7ff4100fb480 msgr2=0x7ff4100fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:05.381 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.377+0000 7ff414b76700 1 -- 172.21.15.38:0/460798957 shutdown_connections 2023-03-09T14:46:05.381 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.377+0000 7ff414b76700 1 -- 172.21.15.38:0/460798957 wait complete. 2023-03-09T14:46:05.394 DEBUG:tasks.fwd_scrub:Yielding 2023-03-09T14:46:05.394 INFO:teuthology.run_tasks:Running task exec... 2023-03-09T14:46:05.408 INFO:teuthology.task.exec:Executing custom commands... 2023-03-09T14:46:05.408 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi038.front.sepia.ceph.com 2023-03-09T14:46:05.408 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2023-03-09T14:46:05.409 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2023-03-09T14:46:05.409 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-03-09T14:46:05.409 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-03-09T14:46:05.540 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.535+0000 7f1babb43700 1 -- 172.21.15.38:0/2218271658 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1ba40fffa0 msgr2=0x7f1ba41003f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:05.540 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.535+0000 7f1babb43700 1 --2- 172.21.15.38:0/2218271658 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1ba40fffa0 0x7f1ba41003f0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f1b94009990 tx=0x7f1b94009ca0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.541 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.535+0000 7f1babb43700 1 -- 172.21.15.38:0/2218271658 shutdown_connections 2023-03-09T14:46:05.541 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.535+0000 7f1babb43700 1 --2- 172.21.15.38:0/2218271658 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1ba4100930 0x7f1ba41071f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.541 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.535+0000 7f1babb43700 1 --2- 172.21.15.38:0/2218271658 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ba40fedb0 0x7f1ba40ff1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.542 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.535+0000 7f1babb43700 1 --2- 172.21.15.38:0/2218271658 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1ba40fffa0 0x7f1ba41003f0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.542 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.535+0000 7f1babb43700 1 -- 172.21.15.38:0/2218271658 >> 172.21.15.38:0/2218271658 conn(0x7f1ba40fa3b0 msgr2=0x7f1ba40fc800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:05.542 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.536+0000 7f1babb43700 1 -- 172.21.15.38:0/2218271658 shutdown_connections 2023-03-09T14:46:05.542 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.536+0000 7f1babb43700 1 -- 172.21.15.38:0/2218271658 wait complete. 2023-03-09T14:46:05.542 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.536+0000 7f1babb43700 1 Processor -- start 2023-03-09T14:46:05.542 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.536+0000 7f1babb43700 1 -- start start 2023-03-09T14:46:05.543 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.536+0000 7f1babb43700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1ba40fedb0 0x7f1ba4197df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:05.543 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.536+0000 7f1babb43700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1ba40fffa0 0x7f1ba4198330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:05.543 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.536+0000 7f1babb43700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ba4100930 0x7f1ba419f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:05.544 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.536+0000 7f1babb43700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba419f970 con 0x7f1ba40fffa0 2023-03-09T14:46:05.544 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.536+0000 7f1babb43700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba419fab0 con 0x7f1ba40fedb0 2023-03-09T14:46:05.544 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.536+0000 7f1babb43700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba419fc40 con 0x7f1ba4100930 2023-03-09T14:46:05.544 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.536+0000 7f1ba90de700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1ba40fffa0 0x7f1ba4198330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:05.544 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.536+0000 7f1ba90de700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1ba40fffa0 0x7f1ba4198330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38620/0 (socket says 172.21.15.38:38620) 2023-03-09T14:46:05.544 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.537+0000 7f1ba90de700 1 -- 172.21.15.38:0/1223986875 learned_addr learned my addr 172.21.15.38:0/1223986875 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:46:05.544 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.537+0000 7f1ba98df700 1 --2- 172.21.15.38:0/1223986875 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1ba40fedb0 0x7f1ba4197df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:05.544 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.537+0000 7f1ba90de700 1 -- 172.21.15.38:0/1223986875 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ba4100930 msgr2=0x7f1ba419f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:05.545 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.537+0000 7f1ba90de700 1 --2- 172.21.15.38:0/1223986875 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ba4100930 0x7f1ba419f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.545 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.537+0000 7f1ba90de700 1 -- 172.21.15.38:0/1223986875 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1ba40fedb0 msgr2=0x7f1ba4197df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:05.545 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.537+0000 7f1ba90de700 1 --2- 172.21.15.38:0/1223986875 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1ba40fedb0 0x7f1ba4197df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.545 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.537+0000 7f1ba90de700 1 -- 172.21.15.38:0/1223986875 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b940095f0 con 0x7f1ba40fffa0 2023-03-09T14:46:05.545 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.537+0000 7f1ba90de700 1 --2- 172.21.15.38:0/1223986875 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1ba40fffa0 0x7f1ba4198330 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f1b940092e0 tx=0x7f1b94004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:05.545 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.537+0000 7f1b9affd700 1 -- 172.21.15.38:0/1223986875 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b9401d070 con 0x7f1ba40fffa0 2023-03-09T14:46:05.545 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.537+0000 7f1b9affd700 1 -- 172.21.15.38:0/1223986875 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1b94005030 con 0x7f1ba40fffa0 2023-03-09T14:46:05.546 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.537+0000 7f1babb43700 1 -- 172.21.15.38:0/1223986875 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ba410a710 con 0x7f1ba40fffa0 2023-03-09T14:46:05.546 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.537+0000 7f1b9affd700 1 -- 172.21.15.38:0/1223986875 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b94004c50 con 0x7f1ba40fffa0 2023-03-09T14:46:05.546 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.537+0000 7f1babb43700 1 -- 172.21.15.38:0/1223986875 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ba410a580 con 0x7f1ba40fffa0 2023-03-09T14:46:05.546 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.539+0000 7f1babb43700 1 -- 172.21.15.38:0/1223986875 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ba41a01b0 con 0x7f1ba40fffa0 2023-03-09T14:46:05.547 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.539+0000 7fdbc6e5d700 1 -- 172.21.15.38:0/3214516869 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdbc0101040 msgr2=0x7fdbc01014b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:05.547 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.539+0000 7fdbc6e5d700 1 --2- 172.21.15.38:0/3214516869 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdbc0101040 0x7fdbc01014b0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fdbac009990 tx=0x7fdbac009ca0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.547 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.539+0000 7fdbc6e5d700 1 -- 172.21.15.38:0/3214516869 shutdown_connections 2023-03-09T14:46:05.547 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.539+0000 7fdbc6e5d700 1 --2- 172.21.15.38:0/3214516869 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdbc01019f0 0x7fdbc0108320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.548 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.539+0000 7fdbc6e5d700 1 --2- 172.21.15.38:0/3214516869 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdbc00ffe60 0x7fdbc0100270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.548 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.539+0000 7fdbc6e5d700 1 --2- 172.21.15.38:0/3214516869 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdbc0101040 0x7fdbc01014b0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.548 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.539+0000 7fdbc6e5d700 1 -- 172.21.15.38:0/3214516869 >> 172.21.15.38:0/3214516869 conn(0x7fdbc00fb4d0 msgr2=0x7fdbc00fd900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:05.548 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.539+0000 7fdbc6e5d700 1 -- 172.21.15.38:0/3214516869 shutdown_connections 2023-03-09T14:46:05.548 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.539+0000 7fdbc6e5d700 1 -- 172.21.15.38:0/3214516869 wait complete. 2023-03-09T14:46:05.548 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.540+0000 7f1b9affd700 1 -- 172.21.15.38:0/1223986875 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) v1 ==== 133198+0+0 (secure 0 0 0) 0x7f1b9400b050 con 0x7f1ba40fffa0 2023-03-09T14:46:05.549 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.540+0000 7fdbc6e5d700 1 Processor -- start 2023-03-09T14:46:05.549 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.540+0000 7fdbc6e5d700 1 -- start start 2023-03-09T14:46:05.550 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.540+0000 7fdbc6e5d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdbc00ffe60 0x7fdbc0071d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:05.550 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.540+0000 7fdbc6e5d700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdbc0101040 0x7fdbc00722c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:05.550 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.540+0000 7fdbc6e5d700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdbc01019f0 0x7fdbc00703c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:05.550 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.540+0000 7fdbc6e5d700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc0070b60 con 0x7fdbc00ffe60 2023-03-09T14:46:05.550 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.540+0000 7fdbc6e5d700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc0070cd0 con 0x7fdbc0101040 2023-03-09T14:46:05.550 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.540+0000 7fdbc6e5d700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc0070e90 con 0x7fdbc01019f0 2023-03-09T14:46:05.551 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.540+0000 7fdbc4bf9700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdbc00ffe60 0x7fdbc0071d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:05.551 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.540+0000 7fdbc4bf9700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdbc00ffe60 0x7fdbc0071d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38630/0 (socket says 172.21.15.38:38630) 2023-03-09T14:46:05.551 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.540+0000 7fdbc4bf9700 1 -- 172.21.15.38:0/2652783232 learned_addr learned my addr 172.21.15.38:0/2652783232 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:46:05.551 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.540+0000 7f1b9affd700 1 --2- 172.21.15.38:0/1223986875 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f1b9009f440 0x7f1b900a18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:05.552 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.541+0000 7fdbbffff700 1 --2- 172.21.15.38:0/2652783232 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdbc0101040 0x7fdbc00722c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:05.552 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.541+0000 7f1b9affd700 1 -- 172.21.15.38:0/1223986875 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f1b940cb190 con 0x7f1ba40fffa0 2023-03-09T14:46:05.552 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.541+0000 7f1ba98df700 1 --2- 172.21.15.38:0/1223986875 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f1b9009f440 0x7f1b900a18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:05.553 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.541+0000 7fdbc53fa700 1 --2- 172.21.15.38:0/2652783232 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdbc01019f0 0x7fdbc00703c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:05.553 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.541+0000 7fdbc4bf9700 1 -- 172.21.15.38:0/2652783232 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdbc01019f0 msgr2=0x7fdbc00703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:05.553 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.541+0000 7fdbc4bf9700 1 --2- 172.21.15.38:0/2652783232 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdbc01019f0 0x7fdbc00703c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.553 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.541+0000 7fdbc4bf9700 1 -- 172.21.15.38:0/2652783232 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdbc0101040 msgr2=0x7fdbc00722c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:05.553 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.541+0000 7fdbc4bf9700 1 --2- 172.21.15.38:0/2652783232 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdbc0101040 0x7fdbc00722c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.554 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.541+0000 7fdbc4bf9700 1 -- 172.21.15.38:0/2652783232 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbac0095f0 con 0x7fdbc00ffe60 2023-03-09T14:46:05.554 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.541+0000 7fdbbffff700 1 --2- 172.21.15.38:0/2652783232 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdbc0101040 0x7fdbc00722c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:46:05.554 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.541+0000 7fdbc4bf9700 1 --2- 172.21.15.38:0/2652783232 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdbc00ffe60 0x7fdbc0071d80 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fdbb400d980 tx=0x7fdbb400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:05.554 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.541+0000 7f1ba98df700 1 --2- 172.21.15.38:0/1223986875 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f1b9009f440 0x7f1b900a18f0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f1ba40ffe00 tx=0x7f1ba0007400 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:05.555 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.545+0000 7fdbbdffb700 1 -- 172.21.15.38:0/2652783232 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb4004020 con 0x7fdbc00ffe60 2023-03-09T14:46:05.555 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.545+0000 7fdbbdffb700 1 -- 172.21.15.38:0/2652783232 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdbb4009730 con 0x7fdbc00ffe60 2023-03-09T14:46:05.555 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.545+0000 7fdbc6e5d700 1 -- 172.21.15.38:0/2652783232 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbc006af70 con 0x7fdbc00ffe60 2023-03-09T14:46:05.555 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.545+0000 7fdbbdffb700 1 -- 172.21.15.38:0/2652783232 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb4003a80 con 0x7fdbc00ffe60 2023-03-09T14:46:05.556 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.545+0000 7f1b9affd700 1 -- 172.21.15.38:0/1223986875 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f1b9405a5b0 con 0x7f1ba40fffa0 2023-03-09T14:46:05.560 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.545+0000 7fdbc6e5d700 1 -- 172.21.15.38:0/2652783232 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbc006ade0 con 0x7fdbc00ffe60 2023-03-09T14:46:05.560 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.547+0000 7fdbbdffb700 1 -- 172.21.15.38:0/2652783232 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) v1 ==== 133198+0+0 (secure 0 0 0) 0x7fdbb400b030 con 0x7fdbc00ffe60 2023-03-09T14:46:05.560 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.548+0000 7fdbbdffb700 1 --2- 172.21.15.38:0/2652783232 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fdbb009f440 0x7fdbb00a18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:05.561 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.548+0000 7fdbbffff700 1 --2- 172.21.15.38:0/2652783232 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fdbb009f440 0x7fdbb00a18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:05.561 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.548+0000 7fdbbdffb700 1 -- 172.21.15.38:0/2652783232 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fdbb40c9c20 con 0x7fdbc00ffe60 2023-03-09T14:46:05.561 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.549+0000 7fdbbffff700 1 --2- 172.21.15.38:0/2652783232 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fdbb009f440 0x7fdbb00a18f0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fdbac00b2e0 tx=0x7fdbac005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:05.561 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.549+0000 7fdbd2fe6b80 1 Processor -- start 2023-03-09T14:46:05.561 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.549+0000 7fdbd2fe6b80 1 -- start start 2023-03-09T14:46:05.561 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.549+0000 7fdbd2fe6b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5617c5815ac0 0x5617c5815ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:05.561 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.549+0000 7fdbd2fe6b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5617c5816410 0x5617c5816d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:05.562 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.549+0000 7fdbd2fe6b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5617c58172c0 0x5617c5817730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:05.562 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.549+0000 7fdbd2fe6b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617c56ffe20 con 0x5617c5816410 2023-03-09T14:46:05.562 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.549+0000 7fdbd2fe6b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617c576e800 con 0x5617c58172c0 2023-03-09T14:46:05.562 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.549+0000 7fdbd2fe6b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617c5767d30 con 0x5617c5815ac0 2023-03-09T14:46:05.562 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.549+0000 7fdbbffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5617c5816410 0x5617c5816d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:05.562 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.549+0000 7fdbbffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5617c5816410 0x5617c5816d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38642/0 (socket says 172.21.15.38:38642) 2023-03-09T14:46:05.562 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.549+0000 7fdbbffff700 1 -- 172.21.15.38:0/3439242856 learned_addr learned my addr 172.21.15.38:0/3439242856 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:46:05.562 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.550+0000 7fdbc4bf9700 1 --2- 172.21.15.38:0/3439242856 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5617c5815ac0 0x5617c5815ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:05.563 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.550+0000 7fdbc53fa700 1 --2- 172.21.15.38:0/3439242856 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5617c58172c0 0x5617c5817730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:05.563 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.550+0000 7fdbbffff700 1 -- 172.21.15.38:0/3439242856 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5617c5815ac0 msgr2=0x5617c5815ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:46:05.563 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.550+0000 7fdbbffff700 1 --2- 172.21.15.38:0/3439242856 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5617c5815ac0 0x5617c5815ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.563 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.550+0000 7fdbbffff700 1 -- 172.21.15.38:0/3439242856 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5617c58172c0 msgr2=0x5617c5817730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:46:05.563 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.550+0000 7fdbbffff700 1 --2- 172.21.15.38:0/3439242856 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5617c58172c0 0x5617c5817730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.563 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.550+0000 7fdbbffff700 1 -- 172.21.15.38:0/3439242856 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5617c572e210 con 0x5617c5816410 2023-03-09T14:46:05.563 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.550+0000 7fdbc4bf9700 1 --2- 172.21.15.38:0/3439242856 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5617c5815ac0 0x5617c5815ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:46:05.564 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.550+0000 7fdbbffff700 1 --2- 172.21.15.38:0/3439242856 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5617c5816410 0x5617c5816d30 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fdbac00bd50 tx=0x7fdbac020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1738883ea43c55e server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:05.564 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.550+0000 7fdbaaffd700 1 -- 172.21.15.38:0/3439242856 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbac02b070 con 0x5617c5816410 2023-03-09T14:46:05.564 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.550+0000 7fdbaaffd700 1 -- 172.21.15.38:0/3439242856 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdbac01e040 con 0x5617c5816410 2023-03-09T14:46:05.564 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.550+0000 7fdbaaffd700 1 -- 172.21.15.38:0/3439242856 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbac00fb80 con 0x5617c5816410 2023-03-09T14:46:05.565 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.551+0000 7fdbd2fe6b80 1 -- 172.21.15.38:0/3439242856 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5617c5816410 msgr2=0x5617c5816d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:05.565 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.551+0000 7fdbd2fe6b80 1 --2- 172.21.15.38:0/3439242856 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5617c5816410 0x5617c5816d30 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fdbac00bd50 tx=0x7fdbac020000 comp rx=0 tx=0).stop 2023-03-09T14:46:05.565 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.551+0000 7fdbd2fe6b80 1 -- 172.21.15.38:0/3439242856 shutdown_connections 2023-03-09T14:46:05.565 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.551+0000 7fdbd2fe6b80 1 --2- 172.21.15.38:0/3439242856 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5617c58172c0 0x5617c5817730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.565 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.551+0000 7fdbd2fe6b80 1 --2- 172.21.15.38:0/3439242856 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5617c5815ac0 0x5617c5815ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.566 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.551+0000 7fdbd2fe6b80 1 --2- 172.21.15.38:0/3439242856 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5617c5816410 0x5617c5816d30 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.566 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.551+0000 7fdbd2fe6b80 1 -- 172.21.15.38:0/3439242856 >> 172.21.15.38:0/3439242856 conn(0x5617c58105d0 msgr2=0x5617c58109c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:05.566 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.551+0000 7fdbd2fe6b80 1 -- 172.21.15.38:0/3439242856 shutdown_connections 2023-03-09T14:46:05.566 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.551+0000 7fdbd2fe6b80 1 -- 172.21.15.38:0/3439242856 wait complete. 2023-03-09T14:46:05.566 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.551+0000 7fdbd2fe6b80 1 -- start start 2023-03-09T14:46:05.566 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.551+0000 7fdbd2fe6b80 1 Processor -- start 2023-03-09T14:46:05.566 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.552+0000 7fdbd2fe6b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5617c5815ac0 0x5617c581f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:05.566 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.552+0000 7fdbd2fe6b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5617c5816410 0x5617c581fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:05.567 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.552+0000 7fdbd2fe6b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5617c58172c0 0x5617c5826d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:05.567 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.552+0000 7fdbd2fe6b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617c55eb9a0 con 0x5617c5815ac0 2023-03-09T14:46:05.567 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.552+0000 7fdbd2fe6b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617c5827250 con 0x5617c58172c0 2023-03-09T14:46:05.567 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.552+0000 7fdbd2fe6b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617c58273e0 con 0x5617c5816410 2023-03-09T14:46:05.567 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.552+0000 7fdbd2fe6b80 10 client.0 _ll_register_callbacks cb 0x5617c570d5a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:46:05.568 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.552+0000 7fdbc4bf9700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5617c5815ac0 0x5617c581f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:05.568 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.552+0000 7fdbc4bf9700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5617c5815ac0 0x5617c581f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38646/0 (socket says 172.21.15.38:38646) 2023-03-09T14:46:05.568 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.552+0000 7fdbc4bf9700 1 -- 172.21.15.38:0/2835149388 learned_addr learned my addr 172.21.15.38:0/2835149388 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:46:05.568 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.552+0000 7fdbc53fa700 1 --2- 172.21.15.38:0/2835149388 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5617c58172c0 0x5617c5826d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:05.568 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.552+0000 7fdbbffff700 1 --2- 172.21.15.38:0/2835149388 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5617c5816410 0x5617c581fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:05.568 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.552+0000 7fdbc4bf9700 1 -- 172.21.15.38:0/2835149388 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5617c5816410 msgr2=0x5617c581fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:46:05.568 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.552+0000 7fdbc4bf9700 1 --2- 172.21.15.38:0/2835149388 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5617c5816410 0x5617c581fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.568 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.552+0000 7fdbc4bf9700 1 -- 172.21.15.38:0/2835149388 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5617c58172c0 msgr2=0x5617c5826d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:46:05.569 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.552+0000 7fdbc4bf9700 1 --2- 172.21.15.38:0/2835149388 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5617c58172c0 0x5617c5826d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.569 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.552+0000 7fdbc4bf9700 1 -- 172.21.15.38:0/2835149388 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbc006ade0 con 0x5617c5815ac0 2023-03-09T14:46:05.569 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.552+0000 7fdbc53fa700 1 --2- 172.21.15.38:0/2835149388 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5617c58172c0 0x5617c5826d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:46:05.569 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.552+0000 7fdb9ffff700 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2023-03-09T14:46:05.569 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.552+0000 7fdbc4bf9700 1 --2- 172.21.15.38:0/2835149388 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5617c5815ac0 0x5617c581f790 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fdbb4000c00 tx=0x7fdbb4097a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27f19573ab4ffa28 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:05.569 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.553+0000 7fdb9ffff700 1 -- 172.21.15.38:0/2835149388 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb4009b80 con 0x5617c5815ac0 2023-03-09T14:46:05.569 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.553+0000 7fdb9ffff700 1 -- 172.21.15.38:0/2835149388 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fdbb4096e80 con 0x5617c5815ac0 2023-03-09T14:46:05.570 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.553+0000 7fdbd2fe6b80 1 -- 172.21.15.38:0/2835149388 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5617c5735210 con 0x5617c5815ac0 2023-03-09T14:46:05.570 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.553+0000 7fdb9ffff700 1 -- 172.21.15.38:0/2835149388 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdbb4009ce0 con 0x5617c5815ac0 2023-03-09T14:46:05.570 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.553+0000 7fdbd2fe6b80 10 client.15519 fetch_fsmap learned FSMap version 23 2023-03-09T14:46:05.570 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.553+0000 7fdbd2fe6b80 1 -- 172.21.15.38:0/2835149388 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x7fdbc0108c10 con 0x5617c5815ac0 2023-03-09T14:46:05.570 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.553+0000 7fdb9ffff700 1 -- 172.21.15.38:0/2835149388 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5617c5735210 con 0x5617c5815ac0 2023-03-09T14:46:05.570 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.553+0000 7fdb9ffff700 1 -- 172.21.15.38:0/2835149388 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fdbb4093780 con 0x5617c5815ac0 2023-03-09T14:46:05.571 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.553+0000 7fdbd2fe6b80 10 client.15519 fetch_fsmap finished waiting for FSMap version 23 2023-03-09T14:46:05.571 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.553+0000 7fdbd2fe6b80 10 client.15519 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T14:46:05.571 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.553+0000 7fdbd2fe6b80 20 client.15519 populate_metadata read hostname 'smithi038' 2023-03-09T14:46:05.571 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.553+0000 7fdbd2fe6b80 1 --2- 172.21.15.38:0/2835149388 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5617c582b7c0 0x5617c582dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:05.571 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.553+0000 7fdbd2fe6b80 4 client.15519 mds_command: new command op to 24530 tid=0[{"prefix": "get_command_descriptions"}] 2023-03-09T14:46:05.571 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.553+0000 7fdbd2fe6b80 1 -- 172.21.15.38:0/2835149388 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5617c55ebfe0 con 0x5617c582b7c0 2023-03-09T14:46:05.571 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.554+0000 7fdbbffff700 1 --2- 172.21.15.38:0/2835149388 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5617c582b7c0 0x5617c582dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:05.572 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.554+0000 7fdb9ffff700 10 client.15519 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:46:05.572 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.554+0000 7fdbbffff700 1 --2- 172.21.15.38:0/2835149388 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5617c582b7c0 0x5617c582dbd0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d3d6da342dc718d0 server_cookie=6e4e63a87fa9c199 in_seq=0 out_seq=0 2023-03-09T14:46:05.573 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.555+0000 7fdb9ffff700 1 -- 172.21.15.38:0/2835149388 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18278 (crc 0 0 0) 0x5617c55ebfe0 con 0x5617c582b7c0 2023-03-09T14:46:05.573 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.555+0000 7fdb9ffff700 10 client.15519 handle_command_reply: tid=0 2023-03-09T14:46:05.573 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.555+0000 7fdbd2fe6b80 1 client.15519 shutdown 2023-03-09T14:46:05.573 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.555+0000 7fdbd2fe6b80 1 -- 172.21.15.38:0/2835149388 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5617c5815ac0 msgr2=0x5617c581f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:05.573 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.555+0000 7fdbd2fe6b80 1 --2- 172.21.15.38:0/2835149388 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5617c5815ac0 0x5617c581f790 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fdbb4000c00 tx=0x7fdbb4097a40 comp rx=0 tx=0).stop 2023-03-09T14:46:05.573 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.555+0000 7fdbd2fe6b80 1 -- 172.21.15.38:0/2835149388 shutdown_connections 2023-03-09T14:46:05.573 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.555+0000 7fdbd2fe6b80 1 --2- 172.21.15.38:0/2835149388 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5617c582b7c0 0x5617c582dbd0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.574 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.555+0000 7fdbd2fe6b80 1 --2- 172.21.15.38:0/2835149388 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5617c5815ac0 0x5617c581f790 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.574 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.555+0000 7fdb9ffff700 0 client.15519 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:46:05.574 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.555+0000 7fdbd2fe6b80 1 --2- 172.21.15.38:0/2835149388 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5617c58172c0 0x5617c5826d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.574 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.555+0000 7fdbd2fe6b80 1 --2- 172.21.15.38:0/2835149388 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5617c5816410 0x5617c581fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.574 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.555+0000 7fdbd2fe6b80 1 -- 172.21.15.38:0/2835149388 >> 172.21.15.38:0/2835149388 conn(0x5617c58105d0 msgr2=0x5617c580fca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:05.574 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.555+0000 7fdbd2fe6b80 1 -- 172.21.15.38:0/2835149388 shutdown_connections 2023-03-09T14:46:05.574 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.555+0000 7fdbd2fe6b80 1 -- 172.21.15.38:0/2835149388 wait complete. 2023-03-09T14:46:05.575 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.555+0000 7fdbd2fe6b80 20 client.15519 trim_cache size 0 max 16384 2023-03-09T14:46:05.576 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.573+0000 7fdbd2fe6b80 1 Processor -- start 2023-03-09T14:46:05.576 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.573+0000 7fdbd2fe6b80 1 -- start start 2023-03-09T14:46:05.576 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.573+0000 7fdbd2fe6b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5617c58105d0 0x5617c581d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:05.577 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.573+0000 7fdbd2fe6b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5617c5815ac0 0x5617c581a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:05.577 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.573+0000 7fdbd2fe6b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5617c5816410 0x5617c581a700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:05.577 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.573+0000 7fdbd2fe6b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617c581ec10 con 0x5617c5815ac0 2023-03-09T14:46:05.577 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.573+0000 7fdbd2fe6b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617c582b5a0 con 0x5617c58105d0 2023-03-09T14:46:05.577 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.573+0000 7fdbd2fe6b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617c581d460 con 0x5617c5816410 2023-03-09T14:46:05.577 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.573+0000 7fdbbffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5617c5815ac0 0x5617c581a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:05.577 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.573+0000 7fdbbffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5617c5815ac0 0x5617c581a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38654/0 (socket says 172.21.15.38:38654) 2023-03-09T14:46:05.577 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.573+0000 7fdbbffff700 1 -- 172.21.15.38:0/2599680547 learned_addr learned my addr 172.21.15.38:0/2599680547 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:46:05.578 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.573+0000 7fdbc4bf9700 1 --2- 172.21.15.38:0/2599680547 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5617c58105d0 0x5617c581d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:05.578 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.573+0000 7fdbc53fa700 1 --2- 172.21.15.38:0/2599680547 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5617c5816410 0x5617c581a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:05.578 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.573+0000 7fdbbffff700 1 -- 172.21.15.38:0/2599680547 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5617c5816410 msgr2=0x5617c581a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:46:05.578 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.573+0000 7fdbbffff700 1 --2- 172.21.15.38:0/2599680547 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5617c5816410 0x5617c581a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.578 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.573+0000 7fdbbffff700 1 -- 172.21.15.38:0/2599680547 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5617c58105d0 msgr2=0x5617c581d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:46:05.578 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.573+0000 7fdbbffff700 1 --2- 172.21.15.38:0/2599680547 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5617c58105d0 0x5617c581d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.578 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.573+0000 7fdbbffff700 1 -- 172.21.15.38:0/2599680547 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5617c557ece0 con 0x5617c5815ac0 2023-03-09T14:46:05.579 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.574+0000 7fdbbffff700 1 --2- 172.21.15.38:0/2599680547 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5617c5815ac0 0x5617c581a1c0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fdbac031040 tx=0x7fdbac0053a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d675e6a25328c92 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:05.579 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.574+0000 7fdbab7fe700 1 -- 172.21.15.38:0/2599680547 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbac02b040 con 0x5617c5815ac0 2023-03-09T14:46:05.579 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.574+0000 7fdbab7fe700 1 -- 172.21.15.38:0/2599680547 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdbac01e040 con 0x5617c5815ac0 2023-03-09T14:46:05.579 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.574+0000 7fdbab7fe700 1 -- 172.21.15.38:0/2599680547 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbac004ac0 con 0x5617c5815ac0 2023-03-09T14:46:05.582 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.574+0000 7fdbd2fe6b80 1 -- 172.21.15.38:0/2599680547 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5617c5815ac0 msgr2=0x5617c581a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:05.582 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.574+0000 7fdbd2fe6b80 1 --2- 172.21.15.38:0/2599680547 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5617c5815ac0 0x5617c581a1c0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fdbac031040 tx=0x7fdbac0053a0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.582 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.574+0000 7fdbd2fe6b80 1 -- 172.21.15.38:0/2599680547 shutdown_connections 2023-03-09T14:46:05.582 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.575+0000 7fdbd2fe6b80 1 --2- 172.21.15.38:0/2599680547 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5617c5816410 0x5617c581a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.583 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.575+0000 7fdbd2fe6b80 1 --2- 172.21.15.38:0/2599680547 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5617c58105d0 0x5617c581d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.583 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.575+0000 7fdbd2fe6b80 1 --2- 172.21.15.38:0/2599680547 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5617c5815ac0 0x5617c581a1c0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.583 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.575+0000 7fdbd2fe6b80 1 -- 172.21.15.38:0/2599680547 >> 172.21.15.38:0/2599680547 conn(0x5617c5810d00 msgr2=0x5617c580fca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:05.583 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.575+0000 7fdbd2fe6b80 1 -- 172.21.15.38:0/2599680547 shutdown_connections 2023-03-09T14:46:05.583 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.575+0000 7fdbd2fe6b80 1 -- 172.21.15.38:0/2599680547 wait complete. 2023-03-09T14:46:05.583 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.575+0000 7fdbd2fe6b80 1 -- start start 2023-03-09T14:46:05.583 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.575+0000 7fdbd2fe6b80 1 Processor -- start 2023-03-09T14:46:05.584 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.575+0000 7fdbd2fe6b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5617c58105d0 0x5617c587c520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:05.584 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.575+0000 7fdbd2fe6b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5617c5815ac0 0x5617c587ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:05.584 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.575+0000 7fdbd2fe6b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5617c5816410 0x5617c5883aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:05.584 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.575+0000 7fdbd2fe6b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617c5883fe0 con 0x5617c5815ac0 2023-03-09T14:46:05.584 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.575+0000 7fdbd2fe6b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617c5884120 con 0x5617c5816410 2023-03-09T14:46:05.584 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.575+0000 7fdbd2fe6b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617c58842b0 con 0x5617c58105d0 2023-03-09T14:46:05.584 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.575+0000 7fdbd2fe6b80 10 client.0 _ll_register_callbacks cb 0x5617c570d5a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:46:05.584 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.575+0000 7fdbbffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5617c5815ac0 0x5617c587ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:05.585 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.576+0000 7fdbbffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5617c5815ac0 0x5617c587ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38662/0 (socket says 172.21.15.38:38662) 2023-03-09T14:46:05.585 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.576+0000 7fdbbffff700 1 -- 172.21.15.38:0/1737066403 learned_addr learned my addr 172.21.15.38:0/1737066403 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:46:05.585 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.576+0000 7fdbc4bf9700 1 --2- 172.21.15.38:0/1737066403 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5617c58105d0 0x5617c587c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:05.585 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.576+0000 7fdbc53fa700 1 --2- 172.21.15.38:0/1737066403 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5617c5816410 0x5617c5883aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:05.585 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.576+0000 7fdbbffff700 1 -- 172.21.15.38:0/1737066403 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5617c58105d0 msgr2=0x5617c587c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:46:05.585 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.576+0000 7fdbbffff700 1 --2- 172.21.15.38:0/1737066403 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5617c58105d0 0x5617c587c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.585 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.576+0000 7fdbbffff700 1 -- 172.21.15.38:0/1737066403 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5617c5816410 msgr2=0x5617c5883aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:46:05.586 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.576+0000 7fdbbffff700 1 --2- 172.21.15.38:0/1737066403 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5617c5816410 0x5617c5883aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.586 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.576+0000 7fdbbffff700 1 -- 172.21.15.38:0/1737066403 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5617c572e210 con 0x5617c5815ac0 2023-03-09T14:46:05.586 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.576+0000 7fdbc4bf9700 1 --2- 172.21.15.38:0/1737066403 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5617c58105d0 0x5617c587c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:46:05.586 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.576+0000 7fdbc53fa700 1 --2- 172.21.15.38:0/1737066403 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5617c5816410 0x5617c5883aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-09T14:46:05.586 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.576+0000 7fdbbffff700 1 --2- 172.21.15.38:0/1737066403 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5617c5815ac0 0x5617c587ca60 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fdbac00b680 tx=0x7fdbac039510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4520356d76a7c7a8 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:05.586 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.576+0000 7fdb9ffff700 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2023-03-09T14:46:05.586 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.576+0000 7fdb9ffff700 1 -- 172.21.15.38:0/1737066403 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbac02b050 con 0x5617c5815ac0 2023-03-09T14:46:05.586 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.576+0000 7fdb9ffff700 1 -- 172.21.15.38:0/1737066403 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fdbac039bd0 con 0x5617c5815ac0 2023-03-09T14:46:05.587 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.576+0000 7fdbd2fe6b80 1 -- 172.21.15.38:0/1737066403 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5617c5725230 con 0x5617c5815ac0 2023-03-09T14:46:05.587 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.576+0000 7fdb9ffff700 1 -- 172.21.15.38:0/1737066403 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdbac00f9e0 con 0x5617c5815ac0 2023-03-09T14:46:05.587 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.576+0000 7fdbd2fe6b80 10 client.15525 fetch_fsmap learned FSMap version 23 2023-03-09T14:46:05.587 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.576+0000 7fdbd2fe6b80 1 -- 172.21.15.38:0/1737066403 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5617c55bc2d0 con 0x5617c5815ac0 2023-03-09T14:46:05.587 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.577+0000 7fdb9ffff700 1 -- 172.21.15.38:0/1737066403 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5617c5725230 con 0x5617c5815ac0 2023-03-09T14:46:05.587 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.577+0000 7fdb9ffff700 1 -- 172.21.15.38:0/1737066403 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fdbac037040 con 0x5617c5815ac0 2023-03-09T14:46:05.587 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.577+0000 7fdbd2fe6b80 10 client.15525 fetch_fsmap finished waiting for FSMap version 23 2023-03-09T14:46:05.588 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.577+0000 7fdbd2fe6b80 10 client.15525 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T14:46:05.588 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.577+0000 7fdbd2fe6b80 20 client.15525 populate_metadata read hostname 'smithi038' 2023-03-09T14:46:05.588 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.577+0000 7fdbd2fe6b80 1 --2- 172.21.15.38:0/1737066403 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5617c58172c0 0x5617c5887940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:05.588 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.577+0000 7fdbd2fe6b80 4 client.15525 mds_command: new command op to 24530 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-03-09T14:46:05.588 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.577+0000 7fdbd2fe6b80 1 -- 172.21.15.38:0/1737066403 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5617c570f240 con 0x5617c58172c0 2023-03-09T14:46:05.588 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.577+0000 7fdbc4bf9700 1 --2- 172.21.15.38:0/1737066403 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5617c58172c0 0x5617c5887940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:05.588 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.577+0000 7fdb9ffff700 10 client.15525 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:46:05.588 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.577+0000 7fdbc4bf9700 1 --2- 172.21.15.38:0/1737066403 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5617c58172c0 0x5617c5887940 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b57bf448e3d8bf73 server_cookie=ddff73b3e410023c in_seq=0 out_seq=0 2023-03-09T14:46:05.589 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.579+0000 7fdb9ffff700 1 -- 172.21.15.38:0/1737066403 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5617c570f240 con 0x5617c58172c0 2023-03-09T14:46:05.589 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.579+0000 7fdb9ffff700 10 client.15525 handle_command_reply: tid=0 2023-03-09T14:46:05.589 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.579+0000 7fdbd2fe6b80 1 client.15525 shutdown 2023-03-09T14:46:05.589 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.579+0000 7fdbd2fe6b80 1 -- 172.21.15.38:0/1737066403 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5617c5815ac0 msgr2=0x5617c587ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:05.589 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.579+0000 7fdbd2fe6b80 1 --2- 172.21.15.38:0/1737066403 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5617c5815ac0 0x5617c587ca60 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fdbac00b680 tx=0x7fdbac039510 comp rx=0 tx=0).stop 2023-03-09T14:46:05.589 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.579+0000 7fdbd2fe6b80 1 -- 172.21.15.38:0/1737066403 shutdown_connections 2023-03-09T14:46:05.589 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.579+0000 7fdbd2fe6b80 1 --2- 172.21.15.38:0/1737066403 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5617c58172c0 0x5617c5887940 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.590 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.579+0000 7fdbd2fe6b80 1 --2- 172.21.15.38:0/1737066403 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5617c5816410 0x5617c5883aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.590 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.579+0000 7fdbd2fe6b80 1 --2- 172.21.15.38:0/1737066403 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5617c58105d0 0x5617c587c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.590 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.579+0000 7fdbd2fe6b80 1 --2- 172.21.15.38:0/1737066403 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5617c5815ac0 0x5617c587ca60 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.590 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.579+0000 7fdbd2fe6b80 1 -- 172.21.15.38:0/1737066403 >> 172.21.15.38:0/1737066403 conn(0x5617c5810d00 msgr2=0x5617c581d6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:05.590 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.579+0000 7fdb9ffff700 0 client.15525 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:46:05.590 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.579+0000 7fdbd2fe6b80 1 -- 172.21.15.38:0/1737066403 shutdown_connections 2023-03-09T14:46:05.590 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.579+0000 7fdbd2fe6b80 1 -- 172.21.15.38:0/1737066403 wait complete. 2023-03-09T14:46:05.590 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.579+0000 7fdbd2fe6b80 20 client.15525 trim_cache size 0 max 16384 2023-03-09T14:46:05.591 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2023-03-09T14:46:05.591 INFO:teuthology.orchestra.run.smithi038.stdout: "return_code": 0, 2023-03-09T14:46:05.591 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_tag": "74d53bfa-6121-4cb0-882f-6502e2585fd1", 2023-03-09T14:46:05.591 INFO:teuthology.orchestra.run.smithi038.stdout: "mode": "asynchronous" 2023-03-09T14:46:05.591 INFO:teuthology.orchestra.run.smithi038.stdout:} 2023-03-09T14:46:05.592 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.580+0000 7fdbc6e5d700 1 -- 172.21.15.38:0/2652783232 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fdbb009f440 msgr2=0x7fdbb00a18f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:05.593 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.580+0000 7fdbc6e5d700 1 --2- 172.21.15.38:0/2652783232 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fdbb009f440 0x7fdbb00a18f0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fdbac00b2e0 tx=0x7fdbac005fd0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.593 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.580+0000 7fdbc6e5d700 1 -- 172.21.15.38:0/2652783232 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdbc00ffe60 msgr2=0x7fdbc0071d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:05.593 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.580+0000 7fdbc6e5d700 1 --2- 172.21.15.38:0/2652783232 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdbc00ffe60 0x7fdbc0071d80 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fdbb400d980 tx=0x7fdbb400dc90 comp rx=0 tx=0).stop 2023-03-09T14:46:05.593 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.580+0000 7fdbc6e5d700 1 -- 172.21.15.38:0/2652783232 shutdown_connections 2023-03-09T14:46:05.593 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.580+0000 7fdbc6e5d700 1 --2- 172.21.15.38:0/2652783232 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdbc00ffe60 0x7fdbc0071d80 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.594 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.580+0000 7fdbc6e5d700 1 --2- 172.21.15.38:0/2652783232 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7fdbb009f440 0x7fdbb00a18f0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.594 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.580+0000 7fdbc6e5d700 1 --2- 172.21.15.38:0/2652783232 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdbc01019f0 0x7fdbc00703c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.594 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.580+0000 7fdbc6e5d700 1 --2- 172.21.15.38:0/2652783232 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdbc0101040 0x7fdbc00722c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.594 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.580+0000 7fdbc6e5d700 1 -- 172.21.15.38:0/2652783232 >> 172.21.15.38:0/2652783232 conn(0x7fdbc00fb4d0 msgr2=0x7fdbc00fd7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:05.594 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.580+0000 7fdbc6e5d700 1 -- 172.21.15.38:0/2652783232 shutdown_connections 2023-03-09T14:46:05.595 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.580+0000 7fdbc6e5d700 1 -- 172.21.15.38:0/2652783232 wait complete. 2023-03-09T14:46:05.596 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-03-09T14:46:05.724 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.722+0000 7f1fdf3b4700 1 -- 172.21.15.38:0/3307468550 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1fd8105bf0 msgr2=0x7f1fd8107fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:05.725 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.722+0000 7f1fdf3b4700 1 --2- 172.21.15.38:0/3307468550 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1fd8105bf0 0x7f1fd8107fd0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f1fd000b0a0 tx=0x7f1fd000b3b0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.725 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.722+0000 7f1fdf3b4700 1 -- 172.21.15.38:0/3307468550 shutdown_connections 2023-03-09T14:46:05.725 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.722+0000 7f1fdf3b4700 1 --2- 172.21.15.38:0/3307468550 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1fd8105bf0 0x7f1fd8107fd0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.725 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.722+0000 7f1fdf3b4700 1 --2- 172.21.15.38:0/3307468550 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1fd81009b0 0x7f1fd8102d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.725 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.722+0000 7f1fdf3b4700 1 --2- 172.21.15.38:0/3307468550 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1fd81032d0 0x7f1fd81056b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.725 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.722+0000 7f1fdf3b4700 1 -- 172.21.15.38:0/3307468550 >> 172.21.15.38:0/3307468550 conn(0x7f1fd80fa3b0 msgr2=0x7f1fd80fc7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:05.726 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.722+0000 7f1fdf3b4700 1 -- 172.21.15.38:0/3307468550 shutdown_connections 2023-03-09T14:46:05.726 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.722+0000 7f1fdf3b4700 1 -- 172.21.15.38:0/3307468550 wait complete. 2023-03-09T14:46:05.727 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.722+0000 7f1fdf3b4700 1 Processor -- start 2023-03-09T14:46:05.727 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.723+0000 7f1fdf3b4700 1 -- start start 2023-03-09T14:46:05.727 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.723+0000 7f1fdf3b4700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1fd81009b0 0x7f1fd8195d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:05.728 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.723+0000 7f1fdf3b4700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1fd81032d0 0x7f1fd8196290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:05.728 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.723+0000 7f1fdf3b4700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1fd8105bf0 0x7f1fd819d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:05.728 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.723+0000 7f1fdf3b4700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd819d890 con 0x7f1fd81032d0 2023-03-09T14:46:05.728 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.723+0000 7f1fdf3b4700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd819da00 con 0x7f1fd8105bf0 2023-03-09T14:46:05.728 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.723+0000 7f1fdf3b4700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd819dbc0 con 0x7f1fd81009b0 2023-03-09T14:46:05.728 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.723+0000 7f1fdc94f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1fd81032d0 0x7f1fd8196290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:05.728 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.723+0000 7f1fdc94f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1fd81032d0 0x7f1fd8196290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38674/0 (socket says 172.21.15.38:38674) 2023-03-09T14:46:05.729 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.723+0000 7f1fdc94f700 1 -- 172.21.15.38:0/753108508 learned_addr learned my addr 172.21.15.38:0/753108508 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:46:05.729 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.723+0000 7f1fdd150700 1 --2- 172.21.15.38:0/753108508 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1fd81009b0 0x7f1fd8195d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:05.729 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.723+0000 7f1fdd951700 1 --2- 172.21.15.38:0/753108508 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1fd8105bf0 0x7f1fd819d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:05.729 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.723+0000 7f1fdc94f700 1 -- 172.21.15.38:0/753108508 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1fd81009b0 msgr2=0x7f1fd8195d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:05.729 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.723+0000 7f1fdc94f700 1 --2- 172.21.15.38:0/753108508 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1fd81009b0 0x7f1fd8195d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.729 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.723+0000 7f1fdc94f700 1 -- 172.21.15.38:0/753108508 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1fd8105bf0 msgr2=0x7f1fd819d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:05.729 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.723+0000 7f1fdc94f700 1 --2- 172.21.15.38:0/753108508 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1fd8105bf0 0x7f1fd819d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.729 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.723+0000 7f1fdc94f700 1 -- 172.21.15.38:0/753108508 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fd0009d00 con 0x7f1fd81032d0 2023-03-09T14:46:05.730 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.723+0000 7f1fdd150700 1 --2- 172.21.15.38:0/753108508 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1fd81009b0 0x7f1fd8195d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:46:05.730 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.723+0000 7f1fdc94f700 1 --2- 172.21.15.38:0/753108508 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1fd81032d0 0x7f1fd8196290 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f1fc800d950 tx=0x7f1fc800dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:05.730 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.724+0000 7f1fce7fc700 1 -- 172.21.15.38:0/753108508 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc8009b40 con 0x7f1fd81032d0 2023-03-09T14:46:05.730 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.724+0000 7f1fdf3b4700 1 -- 172.21.15.38:0/753108508 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fd810b7f0 con 0x7f1fd81032d0 2023-03-09T14:46:05.730 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.724+0000 7f1fce7fc700 1 -- 172.21.15.38:0/753108508 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1fc8009ca0 con 0x7f1fd81032d0 2023-03-09T14:46:05.730 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.724+0000 7f1fce7fc700 1 -- 172.21.15.38:0/753108508 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc800f620 con 0x7f1fd81032d0 2023-03-09T14:46:05.730 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.724+0000 7f1fdf3b4700 1 -- 172.21.15.38:0/753108508 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fd810b660 con 0x7f1fd81032d0 2023-03-09T14:46:05.734 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.726+0000 7f1fce7fc700 1 -- 172.21.15.38:0/753108508 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 18) v1 ==== 133198+0+0 (secure 0 0 0) 0x7f1fc8018030 con 0x7f1fd81032d0 2023-03-09T14:46:05.735 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.726+0000 7f1fce7fc700 1 --2- 172.21.15.38:0/753108508 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f1fc409f3c0 0x7f1fc40a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:05.735 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.727+0000 7f1fce7fc700 1 -- 172.21.15.38:0/753108508 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f1fc80ca3c0 con 0x7f1fd81032d0 2023-03-09T14:46:05.735 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.727+0000 7f1fdd150700 1 --2- 172.21.15.38:0/753108508 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f1fc409f3c0 0x7f1fc40a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:05.735 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.727+0000 7f1fdd150700 1 --2- 172.21.15.38:0/753108508 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f1fc409f3c0 0x7f1fc40a1870 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f1fd8197220 tx=0x7f1fd400b2c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:05.735 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.728+0000 7f1feb53db80 1 Processor -- start 2023-03-09T14:46:05.735 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.728+0000 7f1feb53db80 1 -- start start 2023-03-09T14:46:05.735 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.728+0000 7f1feb53db80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55808b1c8a40 0x55808b1c8e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:05.736 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.728+0000 7f1feb53db80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55808b1c9390 0x55808b1c9cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:05.736 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.728+0000 7f1feb53db80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55808b1ca240 0x55808b1ca6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:05.736 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.728+0000 7f1feb53db80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55808b0b2da0 con 0x55808b1c9390 2023-03-09T14:46:05.736 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.728+0000 7f1feb53db80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55808b121780 con 0x55808b1c8a40 2023-03-09T14:46:05.736 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.728+0000 7f1feb53db80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55808b11acb0 con 0x55808b1ca240 2023-03-09T14:46:05.736 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.728+0000 7f1fdc94f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55808b1c9390 0x55808b1c9cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:05.736 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.728+0000 7f1fdc94f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55808b1c9390 0x55808b1c9cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38682/0 (socket says 172.21.15.38:38682) 2023-03-09T14:46:05.737 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.728+0000 7f1fdc94f700 1 -- 172.21.15.38:0/1053379093 learned_addr learned my addr 172.21.15.38:0/1053379093 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:46:05.737 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.729+0000 7f1fdd951700 1 --2- 172.21.15.38:0/1053379093 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55808b1ca240 0x55808b1ca6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:05.737 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.729+0000 7f1fdd150700 1 --2- 172.21.15.38:0/1053379093 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55808b1c8a40 0x55808b1c8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:05.737 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.729+0000 7f1fdc94f700 1 -- 172.21.15.38:0/1053379093 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55808b1ca240 msgr2=0x55808b1ca6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:46:05.737 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.729+0000 7f1fdc94f700 1 --2- 172.21.15.38:0/1053379093 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55808b1ca240 0x55808b1ca6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.737 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.729+0000 7f1fdc94f700 1 -- 172.21.15.38:0/1053379093 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55808b1c8a40 msgr2=0x55808b1c8e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:46:05.737 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.729+0000 7f1fdc94f700 1 --2- 172.21.15.38:0/1053379093 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55808b1c8a40 0x55808b1c8e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.737 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.729+0000 7f1fdc94f700 1 -- 172.21.15.38:0/1053379093 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55808b0e1190 con 0x55808b1c9390 2023-03-09T14:46:05.738 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.729+0000 7f1fdc94f700 1 --2- 172.21.15.38:0/1053379093 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55808b1c9390 0x55808b1c9cb0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f1fc80980c0 tx=0x7f1fc8097d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66b079bbfd5015ec server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:05.738 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.729+0000 7f1fc37fe700 1 -- 172.21.15.38:0/1053379093 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc80a0320 con 0x55808b1c9390 2023-03-09T14:46:05.738 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.729+0000 7f1fc37fe700 1 -- 172.21.15.38:0/1053379093 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1fc80964d0 con 0x55808b1c9390 2023-03-09T14:46:05.738 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.729+0000 7f1fc37fe700 1 -- 172.21.15.38:0/1053379093 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc809fe90 con 0x55808b1c9390 2023-03-09T14:46:05.738 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.730+0000 7f1feb53db80 1 -- 172.21.15.38:0/1053379093 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55808b1c9390 msgr2=0x55808b1c9cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:05.738 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.730+0000 7f1feb53db80 1 --2- 172.21.15.38:0/1053379093 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55808b1c9390 0x55808b1c9cb0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f1fc80980c0 tx=0x7f1fc8097d40 comp rx=0 tx=0).stop 2023-03-09T14:46:05.738 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.730+0000 7f1feb53db80 1 -- 172.21.15.38:0/1053379093 shutdown_connections 2023-03-09T14:46:05.739 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.730+0000 7f1feb53db80 1 --2- 172.21.15.38:0/1053379093 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55808b1ca240 0x55808b1ca6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.739 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.730+0000 7f1feb53db80 1 --2- 172.21.15.38:0/1053379093 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55808b1c8a40 0x55808b1c8e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.739 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.730+0000 7f1feb53db80 1 --2- 172.21.15.38:0/1053379093 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55808b1c9390 0x55808b1c9cb0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.739 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.730+0000 7f1feb53db80 1 -- 172.21.15.38:0/1053379093 >> 172.21.15.38:0/1053379093 conn(0x55808b1c3550 msgr2=0x55808b1c3940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:05.739 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.730+0000 7f1feb53db80 1 -- 172.21.15.38:0/1053379093 shutdown_connections 2023-03-09T14:46:05.739 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.730+0000 7f1feb53db80 1 -- 172.21.15.38:0/1053379093 wait complete. 2023-03-09T14:46:05.739 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.730+0000 7f1feb53db80 1 -- start start 2023-03-09T14:46:05.739 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.730+0000 7f1feb53db80 1 Processor -- start 2023-03-09T14:46:05.740 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.731+0000 7f1feb53db80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55808b1c8a40 0x55808b1d27e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:05.740 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.731+0000 7f1feb53db80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55808b1c9390 0x55808b1d2d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:05.740 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.731+0000 7f1feb53db80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55808b1ca240 0x55808b1d9d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:05.740 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.731+0000 7f1feb53db80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55808af9e920 con 0x55808b1c8a40 2023-03-09T14:46:05.740 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.731+0000 7f1feb53db80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55808b1da2a0 con 0x55808b1c9390 2023-03-09T14:46:05.740 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.731+0000 7f1feb53db80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55808b1da430 con 0x55808b1ca240 2023-03-09T14:46:05.740 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.731+0000 7f1feb53db80 10 client.0 _ll_register_callbacks cb 0x55808b0c0520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:46:05.740 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.731+0000 7f1fdd150700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55808b1c8a40 0x55808b1d27e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:05.741 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.731+0000 7f1fdd150700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55808b1c8a40 0x55808b1d27e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38686/0 (socket says 172.21.15.38:38686) 2023-03-09T14:46:05.741 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.731+0000 7f1fdd150700 1 -- 172.21.15.38:0/866989945 learned_addr learned my addr 172.21.15.38:0/866989945 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:46:05.741 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.731+0000 7f1fdc94f700 1 --2- 172.21.15.38:0/866989945 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55808b1c9390 0x55808b1d2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:05.741 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.731+0000 7f1fdd951700 1 --2- 172.21.15.38:0/866989945 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55808b1ca240 0x55808b1d9d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:05.741 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.731+0000 7f1fdd150700 1 -- 172.21.15.38:0/866989945 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55808b1ca240 msgr2=0x55808b1d9d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:46:05.741 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.731+0000 7f1fdd150700 1 --2- 172.21.15.38:0/866989945 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55808b1ca240 0x55808b1d9d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.741 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.731+0000 7f1fdd150700 1 -- 172.21.15.38:0/866989945 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55808b1c9390 msgr2=0x55808b1d2d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:46:05.742 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.731+0000 7f1fdd150700 1 --2- 172.21.15.38:0/866989945 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55808b1c9390 0x55808b1d2d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.742 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.731+0000 7f1fdd150700 1 -- 172.21.15.38:0/866989945 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fd810b660 con 0x55808b1c8a40 2023-03-09T14:46:05.742 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.731+0000 7f1fdd951700 1 --2- 172.21.15.38:0/866989945 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55808b1ca240 0x55808b1d9d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-09T14:46:05.742 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.731+0000 7f1fdd150700 1 --2- 172.21.15.38:0/866989945 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55808b1c8a40 0x55808b1d27e0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f1fd400bfa0 tx=0x7f1fd4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bafee206fb59b806 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:05.742 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.731+0000 7f1fc0ff9700 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2023-03-09T14:46:05.742 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.731+0000 7f1fc0ff9700 1 -- 172.21.15.38:0/866989945 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd401e620 con 0x55808b1c8a40 2023-03-09T14:46:05.742 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.731+0000 7f1fc0ff9700 1 -- 172.21.15.38:0/866989945 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f1fd4011e40 con 0x55808b1c8a40 2023-03-09T14:46:05.742 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.731+0000 7f1feb53db80 1 -- 172.21.15.38:0/866989945 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55808b0e8190 con 0x55808b1c8a40 2023-03-09T14:46:05.743 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.732+0000 7f1fc0ff9700 1 -- 172.21.15.38:0/866989945 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1fd401e780 con 0x55808b1c8a40 2023-03-09T14:46:05.743 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.732+0000 7f1feb53db80 10 client.15537 fetch_fsmap learned FSMap version 23 2023-03-09T14:46:05.743 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.732+0000 7f1feb53db80 1 -- 172.21.15.38:0/866989945 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55808af31c60 con 0x55808b1c8a40 2023-03-09T14:46:05.743 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.732+0000 7f1fc0ff9700 1 -- 172.21.15.38:0/866989945 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55808b0e8190 con 0x55808b1c8a40 2023-03-09T14:46:05.743 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.732+0000 7f1fc0ff9700 1 -- 172.21.15.38:0/866989945 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f1fd4024020 con 0x55808b1c8a40 2023-03-09T14:46:05.743 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.732+0000 7f1feb53db80 10 client.15537 fetch_fsmap finished waiting for FSMap version 23 2023-03-09T14:46:05.743 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.732+0000 7f1feb53db80 10 client.15537 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T14:46:05.743 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.732+0000 7f1feb53db80 20 client.15537 populate_metadata read hostname 'smithi038' 2023-03-09T14:46:05.744 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.732+0000 7f1feb53db80 1 --2- 172.21.15.38:0/866989945 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55808b1de820 0x55808b1e0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:05.744 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.732+0000 7f1feb53db80 4 client.15537 mds_command: new command op to 24530 tid=0[{"prefix": "get_command_descriptions"}] 2023-03-09T14:46:05.744 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.732+0000 7f1feb53db80 1 -- 172.21.15.38:0/866989945 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55808af9ef60 con 0x55808b1de820 2023-03-09T14:46:05.744 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.732+0000 7f1fdc94f700 1 --2- 172.21.15.38:0/866989945 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55808b1de820 0x55808b1e0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:05.745 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.733+0000 7f1fdc94f700 1 --2- 172.21.15.38:0/866989945 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55808b1de820 0x55808b1e0c30 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c87e703a27fd7ce9 server_cookie=1046899b4cceabcc in_seq=0 out_seq=0 2023-03-09T14:46:05.745 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.733+0000 7f1fc0ff9700 10 client.15537 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:46:05.745 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.733+0000 7f1fc0ff9700 1 -- 172.21.15.38:0/866989945 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18278 (crc 0 0 0) 0x55808af9ef60 con 0x55808b1de820 2023-03-09T14:46:05.745 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.733+0000 7f1fc0ff9700 10 client.15537 handle_command_reply: tid=0 2023-03-09T14:46:05.746 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.733+0000 7f1feb53db80 1 client.15537 shutdown 2023-03-09T14:46:05.746 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.734+0000 7f1feb53db80 1 -- 172.21.15.38:0/866989945 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55808b1c8a40 msgr2=0x55808b1d27e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:05.746 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.734+0000 7f1feb53db80 1 --2- 172.21.15.38:0/866989945 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55808b1c8a40 0x55808b1d27e0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f1fd400bfa0 tx=0x7f1fd4004750 comp rx=0 tx=0).stop 2023-03-09T14:46:05.746 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.734+0000 7f1feb53db80 1 -- 172.21.15.38:0/866989945 shutdown_connections 2023-03-09T14:46:05.746 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.734+0000 7f1feb53db80 1 --2- 172.21.15.38:0/866989945 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55808b1de820 0x55808b1e0c30 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.746 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.734+0000 7f1feb53db80 1 --2- 172.21.15.38:0/866989945 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55808b1c8a40 0x55808b1d27e0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.746 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.734+0000 7f1fc0ff9700 0 client.15537 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:46:05.747 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.734+0000 7f1feb53db80 1 --2- 172.21.15.38:0/866989945 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55808b1ca240 0x55808b1d9d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.747 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.734+0000 7f1feb53db80 1 --2- 172.21.15.38:0/866989945 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55808b1c9390 0x55808b1d2d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.747 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.734+0000 7f1feb53db80 1 -- 172.21.15.38:0/866989945 >> 172.21.15.38:0/866989945 conn(0x55808b1c3550 msgr2=0x55808b1c2c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:05.747 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.734+0000 7f1feb53db80 1 -- 172.21.15.38:0/866989945 shutdown_connections 2023-03-09T14:46:05.747 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.734+0000 7f1feb53db80 1 -- 172.21.15.38:0/866989945 wait complete. 2023-03-09T14:46:05.747 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.734+0000 7f1feb53db80 20 client.15537 trim_cache size 0 max 16384 2023-03-09T14:46:05.753 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.751+0000 7f1feb53db80 1 Processor -- start 2023-03-09T14:46:05.753 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.751+0000 7f1feb53db80 1 -- start start 2023-03-09T14:46:05.753 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.751+0000 7f1feb53db80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55808b1c8a40 0x55808b1e55b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:05.754 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.751+0000 7f1feb53db80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55808b1c9390 0x55808b1d9d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:05.754 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.751+0000 7f1feb53db80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55808b1ca240 0x55808b1c2c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:05.754 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.751+0000 7f1feb53db80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55808b1e5b20 con 0x55808b1c8a40 2023-03-09T14:46:05.754 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.751+0000 7f1feb53db80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55808b1cdd30 con 0x55808b1ca240 2023-03-09T14:46:05.754 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.751+0000 7f1feb53db80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55808b1c8e10 con 0x55808b1c9390 2023-03-09T14:46:05.755 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.751+0000 7f1fdc94f700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55808b1c9390 0x55808b1d9d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:05.756 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.752+0000 7f1fdc94f700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55808b1c9390 0x55808b1d9d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.38:46394/0 (socket says 172.21.15.38:46394) 2023-03-09T14:46:05.756 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.752+0000 7f1fdc94f700 1 -- 172.21.15.38:0/2009624761 learned_addr learned my addr 172.21.15.38:0/2009624761 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:46:05.756 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.752+0000 7f1fdd951700 1 --2- 172.21.15.38:0/2009624761 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55808b1ca240 0x55808b1c2c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:05.756 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.752+0000 7f1fdd150700 1 --2- 172.21.15.38:0/2009624761 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55808b1c8a40 0x55808b1e55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:05.756 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.752+0000 7f1fdc94f700 1 -- 172.21.15.38:0/2009624761 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55808b1ca240 msgr2=0x55808b1c2c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:46:05.756 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.752+0000 7f1fdc94f700 1 --2- 172.21.15.38:0/2009624761 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55808b1ca240 0x55808b1c2c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.756 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.752+0000 7f1fdc94f700 1 -- 172.21.15.38:0/2009624761 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55808b1c8a40 msgr2=0x55808b1e55b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:46:05.757 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.752+0000 7f1fdc94f700 1 --2- 172.21.15.38:0/2009624761 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55808b1c8a40 0x55808b1e55b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.757 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.752+0000 7f1fdc94f700 1 -- 172.21.15.38:0/2009624761 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55808af48de0 con 0x55808b1c9390 2023-03-09T14:46:05.757 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.752+0000 7f1fdc94f700 1 --2- 172.21.15.38:0/2009624761 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55808b1c9390 0x55808b1d9d20 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1fc8009520 tx=0x7f1fc801e000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7fbdbc60043fa0ac server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:05.757 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.752+0000 7f1fc3fff700 1 -- 172.21.15.38:0/2009624761 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc8097c70 con 0x55808b1c9390 2023-03-09T14:46:05.757 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.752+0000 7f1fc3fff700 1 -- 172.21.15.38:0/2009624761 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1fc80ca200 con 0x55808b1c9390 2023-03-09T14:46:05.757 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.752+0000 7f1fc3fff700 1 -- 172.21.15.38:0/2009624761 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc809fe90 con 0x55808b1c9390 2023-03-09T14:46:05.757 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.753+0000 7f1feb53db80 1 -- 172.21.15.38:0/2009624761 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55808b1c9390 msgr2=0x55808b1d9d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:05.758 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.753+0000 7f1feb53db80 1 --2- 172.21.15.38:0/2009624761 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55808b1c9390 0x55808b1d9d20 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1fc8009520 tx=0x7f1fc801e000 comp rx=0 tx=0).stop 2023-03-09T14:46:05.758 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.753+0000 7f1feb53db80 1 -- 172.21.15.38:0/2009624761 shutdown_connections 2023-03-09T14:46:05.758 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.753+0000 7f1feb53db80 1 --2- 172.21.15.38:0/2009624761 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55808b1c8a40 0x55808b1e55b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.758 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.753+0000 7f1feb53db80 1 --2- 172.21.15.38:0/2009624761 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55808b1ca240 0x55808b1c2c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.758 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.753+0000 7f1feb53db80 1 --2- 172.21.15.38:0/2009624761 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55808b1c9390 0x55808b1d9d20 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.758 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.753+0000 7f1feb53db80 1 -- 172.21.15.38:0/2009624761 >> 172.21.15.38:0/2009624761 conn(0x55808b1c3550 msgr2=0x55808b1c3920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:05.758 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.753+0000 7f1feb53db80 1 -- 172.21.15.38:0/2009624761 shutdown_connections 2023-03-09T14:46:05.758 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.753+0000 7f1feb53db80 1 -- 172.21.15.38:0/2009624761 wait complete. 2023-03-09T14:46:05.761 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.754+0000 7f1feb53db80 1 -- start start 2023-03-09T14:46:05.762 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.754+0000 7f1feb53db80 1 Processor -- start 2023-03-09T14:46:05.762 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.754+0000 7f1feb53db80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55808b1c8a40 0x55808b1d5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:05.762 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.754+0000 7f1feb53db80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55808b1c9390 0x55808b1d6290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:05.762 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.755+0000 7f1feb53db80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55808b1ca240 0x55808b1d92b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:05.762 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.755+0000 7f1feb53db80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55808b1d97f0 con 0x55808b1c8a40 2023-03-09T14:46:05.762 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.755+0000 7f1feb53db80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55808b1d9960 con 0x55808b1c9390 2023-03-09T14:46:05.762 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.755+0000 7f1feb53db80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55808b1d9b20 con 0x55808b1ca240 2023-03-09T14:46:05.762 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.755+0000 7f1feb53db80 10 client.0 _ll_register_callbacks cb 0x55808b0c0520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:46:05.763 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.755+0000 7f1fdd150700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55808b1c8a40 0x55808b1d5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:05.763 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.755+0000 7f1fdd150700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55808b1c8a40 0x55808b1d5d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38700/0 (socket says 172.21.15.38:38700) 2023-03-09T14:46:05.763 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.755+0000 7f1fdd150700 1 -- 172.21.15.38:0/3682640476 learned_addr learned my addr 172.21.15.38:0/3682640476 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:46:05.763 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.755+0000 7f1fdc94f700 1 --2- 172.21.15.38:0/3682640476 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55808b1c9390 0x55808b1d6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:05.763 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.755+0000 7f1fdd951700 1 --2- 172.21.15.38:0/3682640476 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55808b1ca240 0x55808b1d92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:05.763 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.755+0000 7f1fdd150700 1 -- 172.21.15.38:0/3682640476 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55808b1ca240 msgr2=0x55808b1d92b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:46:05.763 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.755+0000 7f1fdd150700 1 --2- 172.21.15.38:0/3682640476 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55808b1ca240 0x55808b1d92b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.764 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.755+0000 7f1fdd150700 1 -- 172.21.15.38:0/3682640476 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55808b1c9390 msgr2=0x55808b1d6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:46:05.764 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.755+0000 7f1fdd150700 1 --2- 172.21.15.38:0/3682640476 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55808b1c9390 0x55808b1d6290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.764 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.755+0000 7f1fdd150700 1 -- 172.21.15.38:0/3682640476 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55808b0e1190 con 0x55808b1c8a40 2023-03-09T14:46:05.764 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.755+0000 7f1fdd150700 1 --2- 172.21.15.38:0/3682640476 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55808b1c8a40 0x55808b1d5d50 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f1fd400bf70 tx=0x7f1fd40052f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d3553c8db4bcb5e server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:05.764 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.755+0000 7f1fc0ff9700 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2023-03-09T14:46:05.764 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.755+0000 7f1fc0ff9700 1 -- 172.21.15.38:0/3682640476 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd4005ee0 con 0x55808b1c8a40 2023-03-09T14:46:05.764 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.756+0000 7f1feb53db80 1 -- 172.21.15.38:0/3682640476 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55808b0d81b0 con 0x55808b1c8a40 2023-03-09T14:46:05.764 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.756+0000 7f1fc0ff9700 1 -- 172.21.15.38:0/3682640476 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f1fd4011e40 con 0x55808b1c8a40 2023-03-09T14:46:05.765 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.756+0000 7f1fc0ff9700 1 -- 172.21.15.38:0/3682640476 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1fd4011ab0 con 0x55808b1c8a40 2023-03-09T14:46:05.765 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.756+0000 7f1feb53db80 10 client.15543 fetch_fsmap learned FSMap version 23 2023-03-09T14:46:05.765 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.756+0000 7f1feb53db80 1 -- 172.21.15.38:0/3682640476 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55808af472b0 con 0x55808b1c8a40 2023-03-09T14:46:05.765 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.756+0000 7f1fc0ff9700 1 -- 172.21.15.38:0/3682640476 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55808b0d81b0 con 0x55808b1c8a40 2023-03-09T14:46:05.765 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.756+0000 7f1fc0ff9700 1 -- 172.21.15.38:0/3682640476 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f1fd4011440 con 0x55808b1c8a40 2023-03-09T14:46:05.765 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.756+0000 7f1feb53db80 10 client.15543 fetch_fsmap finished waiting for FSMap version 23 2023-03-09T14:46:05.765 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.756+0000 7f1feb53db80 10 client.15543 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T14:46:05.766 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.756+0000 7f1feb53db80 20 client.15543 populate_metadata read hostname 'smithi038' 2023-03-09T14:46:05.766 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.756+0000 7f1feb53db80 1 --2- 172.21.15.38:0/3682640476 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55808b233970 0x55808b235d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:05.766 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.756+0000 7f1feb53db80 4 client.15543 mds_command: new command op to 24530 tid=0[{"prefix": "scrub status"}] 2023-03-09T14:46:05.766 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.756+0000 7f1feb53db80 1 -- 172.21.15.38:0/3682640476 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55808b0c21c0 con 0x55808b233970 2023-03-09T14:46:05.766 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.757+0000 7f1fdc94f700 1 --2- 172.21.15.38:0/3682640476 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55808b233970 0x55808b235d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:05.766 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.757+0000 7f1fdc94f700 1 --2- 172.21.15.38:0/3682640476 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55808b233970 0x55808b235d80 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a7492036c6bb4f4 server_cookie=5b0fb7415d52e170 in_seq=0 out_seq=0 2023-03-09T14:46:05.766 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.757+0000 7f1fc0ff9700 10 client.15543 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:46:05.766 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.757+0000 7f1fc0ff9700 1 -- 172.21.15.38:0/3682640476 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55808b0c21c0 con 0x55808b233970 2023-03-09T14:46:05.767 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.757+0000 7f1fc0ff9700 10 client.15543 handle_command_reply: tid=0 2023-03-09T14:46:05.767 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.757+0000 7f1feb53db80 1 client.15543 shutdown 2023-03-09T14:46:05.767 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.757+0000 7f1feb53db80 1 -- 172.21.15.38:0/3682640476 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55808b1c8a40 msgr2=0x55808b1d5d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:05.767 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.757+0000 7f1feb53db80 1 --2- 172.21.15.38:0/3682640476 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55808b1c8a40 0x55808b1d5d50 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f1fd400bf70 tx=0x7f1fd40052f0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.767 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.757+0000 7f1feb53db80 1 -- 172.21.15.38:0/3682640476 shutdown_connections 2023-03-09T14:46:05.767 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.757+0000 7f1feb53db80 1 --2- 172.21.15.38:0/3682640476 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55808b233970 0x55808b235d80 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.767 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.757+0000 7f1feb53db80 1 --2- 172.21.15.38:0/3682640476 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55808b1c8a40 0x55808b1d5d50 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.768 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.757+0000 7f1feb53db80 1 --2- 172.21.15.38:0/3682640476 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55808b1ca240 0x55808b1d92b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.768 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.757+0000 7f1feb53db80 1 --2- 172.21.15.38:0/3682640476 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55808b1c9390 0x55808b1d6290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.768 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.757+0000 7f1feb53db80 1 -- 172.21.15.38:0/3682640476 >> 172.21.15.38:0/3682640476 conn(0x55808b1c3550 msgr2=0x55808b1c2c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:05.768 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.757+0000 7f1fc0ff9700 0 client.15543 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:46:05.768 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.757+0000 7f1feb53db80 1 -- 172.21.15.38:0/3682640476 shutdown_connections 2023-03-09T14:46:05.768 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.758+0000 7f1feb53db80 1 -- 172.21.15.38:0/3682640476 wait complete. 2023-03-09T14:46:05.768 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.758+0000 7f1feb53db80 20 client.15543 trim_cache size 0 max 16384 2023-03-09T14:46:05.769 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2023-03-09T14:46:05.769 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "IDLE (waiting for more scrubs)", 2023-03-09T14:46:05.769 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": { 2023-03-09T14:46:05.769 INFO:teuthology.orchestra.run.smithi038.stdout: "74d53bfa-6121-4cb0-882f-6502e2585fd1": { 2023-03-09T14:46:05.769 INFO:teuthology.orchestra.run.smithi038.stdout: "path": "/", 2023-03-09T14:46:05.769 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "74d53bfa-6121-4cb0-882f-6502e2585fd1", 2023-03-09T14:46:05.770 INFO:teuthology.orchestra.run.smithi038.stdout: "options": "recursive,force" 2023-03-09T14:46:05.770 INFO:teuthology.orchestra.run.smithi038.stdout: } 2023-03-09T14:46:05.770 INFO:teuthology.orchestra.run.smithi038.stdout: } 2023-03-09T14:46:05.770 INFO:teuthology.orchestra.run.smithi038.stdout:} 2023-03-09T14:46:05.770 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.758+0000 7f1fdf3b4700 1 -- 172.21.15.38:0/753108508 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f1fc409f3c0 msgr2=0x7f1fc40a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:05.771 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.758+0000 7f1fdf3b4700 1 --2- 172.21.15.38:0/753108508 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f1fc409f3c0 0x7f1fc40a1870 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f1fd8197220 tx=0x7f1fd400b2c0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.771 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.758+0000 7f1fdf3b4700 1 -- 172.21.15.38:0/753108508 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1fd81032d0 msgr2=0x7f1fd8196290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:05.771 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.758+0000 7f1fdf3b4700 1 --2- 172.21.15.38:0/753108508 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1fd81032d0 0x7f1fd8196290 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f1fc800d950 tx=0x7f1fc800dd10 comp rx=0 tx=0).stop 2023-03-09T14:46:05.771 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.758+0000 7f1fdf3b4700 1 -- 172.21.15.38:0/753108508 shutdown_connections 2023-03-09T14:46:05.771 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.758+0000 7f1fdf3b4700 1 --2- 172.21.15.38:0/753108508 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f1fc409f3c0 0x7f1fc40a1870 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.771 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.758+0000 7f1fdf3b4700 1 --2- 172.21.15.38:0/753108508 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1fd8105bf0 0x7f1fd819d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.771 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.758+0000 7f1fdf3b4700 1 --2- 172.21.15.38:0/753108508 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1fd81009b0 0x7f1fd8195d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.772 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.758+0000 7f1fdf3b4700 1 --2- 172.21.15.38:0/753108508 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1fd81032d0 0x7f1fd8196290 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.772 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.758+0000 7f1fdf3b4700 1 -- 172.21.15.38:0/753108508 >> 172.21.15.38:0/753108508 conn(0x7f1fd80fa3b0 msgr2=0x7f1fd8101610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:05.772 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.758+0000 7f1fdf3b4700 1 -- 172.21.15.38:0/753108508 shutdown_connections 2023-03-09T14:46:05.772 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.758+0000 7f1fdf3b4700 1 -- 172.21.15.38:0/753108508 wait complete. 2023-03-09T14:46:05.773 INFO:tasks.cephfs.filesystem:scrub status for tag:74d53bfa-6121-4cb0-882f-6502e2585fd1 - {'path': '/', 'tag': '74d53bfa-6121-4cb0-882f-6502e2585fd1', 'options': 'recursive,force'} 2023-03-09T14:46:05.820 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.818+0000 7f1babb43700 1 -- 172.21.15.38:0/1223986875 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f1ba41a0850 con 0x7f1ba40fffa0 2023-03-09T14:46:05.973 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.971+0000 7f1b9affd700 1 -- 172.21.15.38:0/1223986875 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mgrmap(e 19) v1 ==== 133215+0+0 (secure 0 0 0) 0x7f1b9400c040 con 0x7f1ba40fffa0 2023-03-09T14:46:05.974 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.972+0000 7f1b9affd700 1 -- 172.21.15.38:0/1223986875 <== mon.0 v2:172.21.15.38:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) v1 ==== 92+0+0 (secure 0 0 0) 0x7f1b94098d60 con 0x7f1ba40fffa0 2023-03-09T14:46:05.977 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.974+0000 7f1babb43700 1 -- 172.21.15.38:0/1223986875 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f1b9009f440 msgr2=0x7f1b900a18f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:05.977 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.974+0000 7f1babb43700 1 --2- 172.21.15.38:0/1223986875 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f1b9009f440 0x7f1b900a18f0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f1ba40ffe00 tx=0x7f1ba0007400 comp rx=0 tx=0).stop 2023-03-09T14:46:05.977 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.974+0000 7f1babb43700 1 -- 172.21.15.38:0/1223986875 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1ba40fffa0 msgr2=0x7f1ba4198330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:05.977 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.974+0000 7f1babb43700 1 --2- 172.21.15.38:0/1223986875 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1ba40fffa0 0x7f1ba4198330 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f1b940092e0 tx=0x7f1b94004730 comp rx=0 tx=0).stop 2023-03-09T14:46:05.978 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.974+0000 7f1babb43700 1 -- 172.21.15.38:0/1223986875 shutdown_connections 2023-03-09T14:46:05.978 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.974+0000 7f1babb43700 1 --2- 172.21.15.38:0/1223986875 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f1b9009f440 0x7f1b900a18f0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.978 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.974+0000 7f1babb43700 1 --2- 172.21.15.38:0/1223986875 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1ba4100930 0x7f1ba419f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.978 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.974+0000 7f1babb43700 1 --2- 172.21.15.38:0/1223986875 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1ba40fedb0 0x7f1ba4197df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.978 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.975+0000 7f1babb43700 1 --2- 172.21.15.38:0/1223986875 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1ba40fffa0 0x7f1ba4198330 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:05.978 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.975+0000 7f1babb43700 1 -- 172.21.15.38:0/1223986875 >> 172.21.15.38:0/1223986875 conn(0x7f1ba40fa3b0 msgr2=0x7f1ba4067d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:05.978 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.975+0000 7f1babb43700 1 -- 172.21.15.38:0/1223986875 shutdown_connections 2023-03-09T14:46:05.979 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:05.975+0000 7f1babb43700 1 -- 172.21.15.38:0/1223986875 wait complete. 2023-03-09T14:46:05.990 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2023-03-09T14:46:06.026 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:05 smithi038 ceph-mon[92422]: pgmap v314: 129 pgs: 129 active+clean; 641 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 682 B/s wr, 6 op/s 2023-03-09T14:46:06.026 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:05 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/656546350' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-03-09T14:46:06.026 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:05 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/460798957' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-03-09T14:46:06.026 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:05 smithi038 ceph-mon[92422]: scrub queued for path: / 2023-03-09T14:46:06.027 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:05 smithi038 ceph-mon[92422]: scrub summary: idle+waiting paths [/] 2023-03-09T14:46:06.027 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:05 smithi038 ceph-mon[92422]: scrub summary: active paths [/] 2023-03-09T14:46:06.027 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:05 smithi038 ceph-mon[92422]: scrub summary: idle+waiting paths [/] 2023-03-09T14:46:06.027 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:05 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/1223986875' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2023-03-09T14:46:06.125 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.122+0000 7f0b066ec700 1 -- 172.21.15.38:0/3280999632 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0b00105b70 msgr2=0x7f0b00107fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:06.125 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.122+0000 7f0b066ec700 1 --2- 172.21.15.38:0/3280999632 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0b00105b70 0x7f0b00107fd0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f0af800b0a0 tx=0x7f0af800b3b0 comp rx=0 tx=0).stop 2023-03-09T14:46:06.125 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.122+0000 7f0b066ec700 1 -- 172.21.15.38:0/3280999632 shutdown_connections 2023-03-09T14:46:06.125 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.122+0000 7f0b066ec700 1 --2- 172.21.15.38:0/3280999632 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0b00105b70 0x7f0b00107fd0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:06.125 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.122+0000 7f0b066ec700 1 --2- 172.21.15.38:0/3280999632 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b00100930 0x7f0b00102d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:06.126 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.122+0000 7f0b066ec700 1 --2- 172.21.15.38:0/3280999632 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b00103250 0x7f0b00105630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:06.126 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.122+0000 7f0b066ec700 1 -- 172.21.15.38:0/3280999632 >> 172.21.15.38:0/3280999632 conn(0x7f0b000fa3b0 msgr2=0x7f0b000fc800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:06.126 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.122+0000 7f0b066ec700 1 -- 172.21.15.38:0/3280999632 shutdown_connections 2023-03-09T14:46:06.126 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.122+0000 7f0b066ec700 1 -- 172.21.15.38:0/3280999632 wait complete. 2023-03-09T14:46:06.126 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.123+0000 7f0b066ec700 1 Processor -- start 2023-03-09T14:46:06.126 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.123+0000 7f0b066ec700 1 -- start start 2023-03-09T14:46:06.128 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.123+0000 7f0b066ec700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0b00100930 0x7f0b00195d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:06.128 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.123+0000 7f0b066ec700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b00103250 0x7f0b00196250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:06.128 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.123+0000 7f0b066ec700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b00105b70 0x7f0b0019d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:06.128 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.123+0000 7f0b066ec700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0019d850 con 0x7f0b00100930 2023-03-09T14:46:06.129 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.123+0000 7f0b066ec700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0019d9c0 con 0x7f0b00103250 2023-03-09T14:46:06.129 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.123+0000 7f0b066ec700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b0019db80 con 0x7f0b00105b70 2023-03-09T14:46:06.129 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.123+0000 7f0afffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0b00100930 0x7f0b00195d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:06.129 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.123+0000 7f0afffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0b00100930 0x7f0b00195d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38744/0 (socket says 172.21.15.38:38744) 2023-03-09T14:46:06.129 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.123+0000 7f0afffff700 1 -- 172.21.15.38:0/1557066563 learned_addr learned my addr 172.21.15.38:0/1557066563 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:46:06.129 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.123+0000 7f0b04c89700 1 --2- 172.21.15.38:0/1557066563 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b00105b70 0x7f0b0019d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:06.129 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.123+0000 7f0aff7fe700 1 --2- 172.21.15.38:0/1557066563 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b00103250 0x7f0b00196250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:06.130 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.124+0000 7f0afffff700 1 -- 172.21.15.38:0/1557066563 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b00105b70 msgr2=0x7f0b0019d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:06.130 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.124+0000 7f0afffff700 1 --2- 172.21.15.38:0/1557066563 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b00105b70 0x7f0b0019d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:06.130 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.124+0000 7f0afffff700 1 -- 172.21.15.38:0/1557066563 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b00103250 msgr2=0x7f0b00196250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:06.130 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.124+0000 7f0afffff700 1 --2- 172.21.15.38:0/1557066563 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b00103250 0x7f0b00196250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:06.130 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.124+0000 7f0afffff700 1 -- 172.21.15.38:0/1557066563 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0af4009520 con 0x7f0b00100930 2023-03-09T14:46:06.130 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.124+0000 7f0b04c89700 1 --2- 172.21.15.38:0/1557066563 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b00105b70 0x7f0b0019d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:46:06.130 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.124+0000 7f0afffff700 1 --2- 172.21.15.38:0/1557066563 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0b00100930 0x7f0b00195d10 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f0af000e9a0 tx=0x7f0af000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:06.130 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.124+0000 7f0afd7fa700 1 -- 172.21.15.38:0/1557066563 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0af0009790 con 0x7f0b00100930 2023-03-09T14:46:06.131 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.124+0000 7f0b066ec700 1 -- 172.21.15.38:0/1557066563 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0af8009d00 con 0x7f0b00100930 2023-03-09T14:46:06.131 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.124+0000 7f0afd7fa700 1 -- 172.21.15.38:0/1557066563 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0af0010430 con 0x7f0b00100930 2023-03-09T14:46:06.131 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.124+0000 7f0afd7fa700 1 -- 172.21.15.38:0/1557066563 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0af0018680 con 0x7f0b00100930 2023-03-09T14:46:06.131 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.124+0000 7f0b066ec700 1 -- 172.21.15.38:0/1557066563 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b0010b7f0 con 0x7f0b00100930 2023-03-09T14:46:06.131 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.126+0000 7f0b066ec700 1 -- 172.21.15.38:0/1557066563 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b000fbf20 con 0x7f0b00100930 2023-03-09T14:46:06.132 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.126+0000 7f0afd7fa700 1 -- 172.21.15.38:0/1557066563 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) v1 ==== 133215+0+0 (secure 0 0 0) 0x7f0af000c040 con 0x7f0b00100930 2023-03-09T14:46:06.132 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.127+0000 7f0afd7fa700 1 --2- 172.21.15.38:0/1557066563 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f0ae809f410 0x7f0ae80a18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:06.132 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.127+0000 7f0aff7fe700 1 -- 172.21.15.38:0/1557066563 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f0ae809f410 msgr2=0x7f0ae80a18c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/3328887328 2023-03-09T14:46:06.132 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.127+0000 7f0afd7fa700 1 -- 172.21.15.38:0/1557066563 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f0af0014070 con 0x7f0b00100930 2023-03-09T14:46:06.132 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.127+0000 7f0aff7fe700 1 --2- 172.21.15.38:0/1557066563 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f0ae809f410 0x7f0ae80a18c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-03-09T14:46:06.133 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.131+0000 7f0afd7fa700 1 -- 172.21.15.38:0/1557066563 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f0af00515b0 con 0x7f0b00100930 2023-03-09T14:46:06.221 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:05 smithi181 ceph-mon[94189]: pgmap v314: 129 pgs: 129 active+clean; 641 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 682 B/s wr, 6 op/s 2023-03-09T14:46:06.221 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:05 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/656546350' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-03-09T14:46:06.221 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:05 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/460798957' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-03-09T14:46:06.221 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:05 smithi181 ceph-mon[94189]: scrub queued for path: / 2023-03-09T14:46:06.221 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:05 smithi181 ceph-mon[94189]: scrub summary: idle+waiting paths [/] 2023-03-09T14:46:06.222 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:05 smithi181 ceph-mon[94189]: scrub summary: active paths [/] 2023-03-09T14:46:06.222 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:05 smithi181 ceph-mon[94189]: scrub summary: idle+waiting paths [/] 2023-03-09T14:46:06.222 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:05 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/1223986875' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2023-03-09T14:46:06.222 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 09 14:46:06 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-z[96215]: ignoring --setuser ceph since I am not root 2023-03-09T14:46:06.222 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 09 14:46:06 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-z[96215]: ignoring --setgroup ceph since I am not root 2023-03-09T14:46:06.222 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 09 14:46:06 smithi181 ceph-mgr[96301]: -- 172.21.15.181:0/1011586575 <== mon.1 v2:172.21.15.181:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d2250a4b60 con 0x55d225079000 2023-03-09T14:46:06.223 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 09 14:46:06 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-z[96215]: 2023-03-09T14:46:06.138+0000 7fbe3a67f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-03-09T14:46:06.226 INFO:journalctl@ceph.mgr.y.smithi122.stdout:Mar 09 14:46:06 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-y[95318]: ignoring --setuser ceph since I am not root 2023-03-09T14:46:06.227 INFO:journalctl@ceph.mgr.y.smithi122.stdout:Mar 09 14:46:06 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-y[95318]: ignoring --setgroup ceph since I am not root 2023-03-09T14:46:06.227 INFO:journalctl@ceph.mgr.y.smithi122.stdout:Mar 09 14:46:06 smithi122 ceph-mgr[95350]: -- 172.21.15.122:0/2656075054 <== mon.1 v2:172.21.15.181:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5570d6c14b60 con 0x5570d6be9000 2023-03-09T14:46:06.227 INFO:journalctl@ceph.mgr.y.smithi122.stdout:Mar 09 14:46:06 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-y[95318]: 2023-03-09T14:46:06.144+0000 7fc7c08d5200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-03-09T14:46:06.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:05 smithi122 ceph-mon[93869]: pgmap v314: 129 pgs: 129 active+clean; 641 KiB data, 82 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 682 B/s wr, 6 op/s 2023-03-09T14:46:06.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:05 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/656546350' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-03-09T14:46:06.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:05 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/460798957' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-03-09T14:46:06.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:05 smithi122 ceph-mon[93869]: scrub queued for path: / 2023-03-09T14:46:06.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:05 smithi122 ceph-mon[93869]: scrub summary: idle+waiting paths [/] 2023-03-09T14:46:06.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:05 smithi122 ceph-mon[93869]: scrub summary: active paths [/] 2023-03-09T14:46:06.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:05 smithi122 ceph-mon[93869]: scrub summary: idle+waiting paths [/] 2023-03-09T14:46:06.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:05 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/1223986875' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2023-03-09T14:46:06.293 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:46:06 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: ignoring --setuser ceph since I am not root 2023-03-09T14:46:06.293 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:46:06 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: ignoring --setgroup ceph since I am not root 2023-03-09T14:46:06.294 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:46:06 smithi038 ceph-mgr[92692]: -- 172.21.15.38:0/547817576 <== mon.2 v2:172.21.15.122:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558ff2cc6b60 con 0x558ff2c9ac00 2023-03-09T14:46:06.294 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:46:06 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:46:06.167+0000 7f40d86ca200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-03-09T14:46:06.294 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:46:06 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:46:06.264+0000 7f40d86ca200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-03-09T14:46:06.330 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.328+0000 7f0aff7fe700 1 -- 172.21.15.38:0/1557066563 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f0ae809f410 msgr2=0x7f0ae80a18c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/3328887328 2023-03-09T14:46:06.330 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.328+0000 7f0aff7fe700 1 --2- 172.21.15.38:0/1557066563 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f0ae809f410 0x7f0ae80a18c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-03-09T14:46:06.335 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.333+0000 7f0b066ec700 1 -- 172.21.15.38:0/1557066563 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f0b000614e0 con 0x7f0b00100930 2023-03-09T14:46:06.342 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.340+0000 7f0afd7fa700 1 -- 172.21.15.38:0/1557066563 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v23)=0 v23) v1 ==== 138+0+0 (secure 0 0 0) 0x7f0af00943b0 con 0x7f0b00100930 2023-03-09T14:46:06.345 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.342+0000 7f0b066ec700 1 -- 172.21.15.38:0/1557066563 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f0ae809f410 msgr2=0x7f0ae80a18c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-03-09T14:46:06.345 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.342+0000 7f0b066ec700 1 --2- 172.21.15.38:0/1557066563 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f0ae809f410 0x7f0ae80a18c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:06.345 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.342+0000 7f0b066ec700 1 -- 172.21.15.38:0/1557066563 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0b00100930 msgr2=0x7f0b00195d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:06.345 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.342+0000 7f0b066ec700 1 --2- 172.21.15.38:0/1557066563 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0b00100930 0x7f0b00195d10 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f0af000e9a0 tx=0x7f0af000ecb0 comp rx=0 tx=0).stop 2023-03-09T14:46:06.345 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.342+0000 7f0b066ec700 1 -- 172.21.15.38:0/1557066563 shutdown_connections 2023-03-09T14:46:06.345 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.342+0000 7f0b066ec700 1 --2- 172.21.15.38:0/1557066563 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0b00100930 0x7f0b00195d10 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:06.346 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.342+0000 7f0b066ec700 1 --2- 172.21.15.38:0/1557066563 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f0ae809f410 0x7f0ae80a18c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:06.346 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.342+0000 7f0b066ec700 1 --2- 172.21.15.38:0/1557066563 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b00105b70 0x7f0b0019d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:06.346 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.342+0000 7f0b066ec700 1 --2- 172.21.15.38:0/1557066563 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b00103250 0x7f0b00196250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:06.346 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.342+0000 7f0b066ec700 1 -- 172.21.15.38:0/1557066563 >> 172.21.15.38:0/1557066563 conn(0x7f0b000fa3b0 msgr2=0x7f0b001015b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:06.346 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.342+0000 7f0b066ec700 1 -- 172.21.15.38:0/1557066563 shutdown_connections 2023-03-09T14:46:06.346 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.342+0000 7f0b066ec700 1 -- 172.21.15.38:0/1557066563 wait complete. 2023-03-09T14:46:06.357 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2023-03-09T14:46:06.478 INFO:journalctl@ceph.mgr.y.smithi122.stdout:Mar 09 14:46:06 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-y[95318]: 2023-03-09T14:46:06.224+0000 7fc7c08d5200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-03-09T14:46:06.478 INFO:journalctl@ceph.mgr.y.smithi122.stdout:Mar 09 14:46:06 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-y[95318]: 2023-03-09T14:46:06.313+0000 7fc7c08d5200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-03-09T14:46:06.478 INFO:journalctl@ceph.mgr.y.smithi122.stdout:Mar 09 14:46:06 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-y[95318]: 2023-03-09T14:46:06.407+0000 7fc7c08d5200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-03-09T14:46:06.481 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 09 14:46:06 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-z[96215]: 2023-03-09T14:46:06.217+0000 7fbe3a67f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-03-09T14:46:06.481 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 09 14:46:06 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-z[96215]: 2023-03-09T14:46:06.307+0000 7fbe3a67f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-03-09T14:46:06.481 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 09 14:46:06 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-z[96215]: 2023-03-09T14:46:06.402+0000 7fbe3a67f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-03-09T14:46:06.527 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.524+0000 7feedaf09700 1 -- 172.21.15.38:0/1934802675 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feed40fe7a0 msgr2=0x7feed40febb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:06.527 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.524+0000 7feedaf09700 1 --2- 172.21.15.38:0/1934802675 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feed40fe7a0 0x7feed40febb0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7feec8009990 tx=0x7feec8009ca0 comp rx=0 tx=0).stop 2023-03-09T14:46:06.527 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.524+0000 7feedaf09700 1 -- 172.21.15.38:0/1934802675 shutdown_connections 2023-03-09T14:46:06.527 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.524+0000 7feedaf09700 1 --2- 172.21.15.38:0/1934802675 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feed40fe7a0 0x7feed40febb0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:06.527 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.524+0000 7feedaf09700 1 --2- 172.21.15.38:0/1934802675 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feed40ffaa0 0x7feed40fff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:06.528 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.524+0000 7feedaf09700 1 --2- 172.21.15.38:0/1934802675 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feed40ff0f0 0x7feed40ff560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:06.528 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.524+0000 7feedaf09700 1 -- 172.21.15.38:0/1934802675 >> 172.21.15.38:0/1934802675 conn(0x7feed40fa3b0 msgr2=0x7feed40fc800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:06.528 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.524+0000 7feedaf09700 1 -- 172.21.15.38:0/1934802675 shutdown_connections 2023-03-09T14:46:06.528 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.524+0000 7feedaf09700 1 -- 172.21.15.38:0/1934802675 wait complete. 2023-03-09T14:46:06.528 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.525+0000 7feedaf09700 1 Processor -- start 2023-03-09T14:46:06.528 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.525+0000 7feedaf09700 1 -- start start 2023-03-09T14:46:06.530 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.525+0000 7feedaf09700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feed40fe7a0 0x7feed4197e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:06.530 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.525+0000 7feedaf09700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feed40ff0f0 0x7feed4198340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:06.530 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.525+0000 7feedaf09700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feed40ffaa0 0x7feed419f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:06.530 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.525+0000 7feedaf09700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feed419f980 con 0x7feed40ffaa0 2023-03-09T14:46:06.530 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.525+0000 7feedaf09700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feed419fac0 con 0x7feed40fe7a0 2023-03-09T14:46:06.531 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.525+0000 7feedaf09700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feed419fc50 con 0x7feed40ff0f0 2023-03-09T14:46:06.531 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.525+0000 7feed94a6700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feed40ffaa0 0x7feed419f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:06.531 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.525+0000 7feed94a6700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feed40ffaa0 0x7feed419f3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38812/0 (socket says 172.21.15.38:38812) 2023-03-09T14:46:06.531 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.525+0000 7feed94a6700 1 -- 172.21.15.38:0/2348391980 learned_addr learned my addr 172.21.15.38:0/2348391980 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:46:06.531 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.525+0000 7feed3fff700 1 --2- 172.21.15.38:0/2348391980 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feed40ff0f0 0x7feed4198340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:06.531 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.525+0000 7feed8ca5700 1 --2- 172.21.15.38:0/2348391980 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feed40fe7a0 0x7feed4197e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:06.531 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.525+0000 7feed94a6700 1 -- 172.21.15.38:0/2348391980 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feed40ff0f0 msgr2=0x7feed4198340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:06.532 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.525+0000 7feed94a6700 1 --2- 172.21.15.38:0/2348391980 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feed40ff0f0 0x7feed4198340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:06.532 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.525+0000 7feed94a6700 1 -- 172.21.15.38:0/2348391980 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feed40fe7a0 msgr2=0x7feed4197e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:06.532 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.525+0000 7feed94a6700 1 --2- 172.21.15.38:0/2348391980 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feed40fe7a0 0x7feed4197e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:06.532 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.525+0000 7feed94a6700 1 -- 172.21.15.38:0/2348391980 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feec80095f0 con 0x7feed40ffaa0 2023-03-09T14:46:06.532 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.526+0000 7feed94a6700 1 --2- 172.21.15.38:0/2348391980 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feed40ffaa0 0x7feed419f3b0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7feecc007ee0 tx=0x7feecc00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:06.532 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.526+0000 7feed1ffb700 1 -- 172.21.15.38:0/2348391980 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feecc00dd20 con 0x7feed40ffaa0 2023-03-09T14:46:06.532 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.526+0000 7feed1ffb700 1 -- 172.21.15.38:0/2348391980 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7feecc00f040 con 0x7feed40ffaa0 2023-03-09T14:46:06.532 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.526+0000 7feedaf09700 1 -- 172.21.15.38:0/2348391980 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feed410da00 con 0x7feed40ffaa0 2023-03-09T14:46:06.533 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.526+0000 7feed1ffb700 1 -- 172.21.15.38:0/2348391980 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feecc0048a0 con 0x7feed40ffaa0 2023-03-09T14:46:06.533 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.526+0000 7feedaf09700 1 -- 172.21.15.38:0/2348391980 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feed410d870 con 0x7feed40ffaa0 2023-03-09T14:46:06.533 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.527+0000 7feedaf09700 1 -- 172.21.15.38:0/2348391980 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feed4109580 con 0x7feed40ffaa0 2023-03-09T14:46:06.534 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.529+0000 7feed1ffb700 1 -- 172.21.15.38:0/2348391980 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 19) v1 ==== 133215+0+0 (secure 0 0 0) 0x7feecc013070 con 0x7feed40ffaa0 2023-03-09T14:46:06.534 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.530+0000 7feed1ffb700 1 --2- 172.21.15.38:0/2348391980 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7feec409f490 0x7feec40a1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:06.534 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.530+0000 7feed8ca5700 1 -- 172.21.15.38:0/2348391980 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7feec409f490 msgr2=0x7feec40a1940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/3328887328 2023-03-09T14:46:06.534 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.530+0000 7feed1ffb700 1 -- 172.21.15.38:0/2348391980 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7feecc0c97f0 con 0x7feed40ffaa0 2023-03-09T14:46:06.534 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.530+0000 7feed8ca5700 1 --2- 172.21.15.38:0/2348391980 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7feec409f490 0x7feec40a1940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-03-09T14:46:06.540 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.538+0000 7feed1ffb700 1 -- 172.21.15.38:0/2348391980 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7feecc04e5b0 con 0x7feed40ffaa0 2023-03-09T14:46:06.711 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:46:06 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:46:06.377+0000 7f40d86ca200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-03-09T14:46:06.711 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:46:06 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:46:06.493+0000 7f40d86ca200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-03-09T14:46:06.733 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.731+0000 7feed8ca5700 1 -- 172.21.15.38:0/2348391980 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7feec409f490 msgr2=0x7feec40a1940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/3328887328 2023-03-09T14:46:06.733 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.731+0000 7feed8ca5700 1 --2- 172.21.15.38:0/2348391980 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7feec409f490 0x7feec40a1940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-03-09T14:46:06.746 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.744+0000 7feedaf09700 1 -- 172.21.15.38:0/2348391980 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7feed402cef0 con 0x7feed40ffaa0 2023-03-09T14:46:06.752 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.750+0000 7feed1ffb700 1 -- 172.21.15.38:0/2348391980 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v24)=0 v24) v1 ==== 133+0+0 (secure 0 0 0) 0x7feecc093840 con 0x7feed40ffaa0 2023-03-09T14:46:06.754 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.752+0000 7feedaf09700 1 -- 172.21.15.38:0/2348391980 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7feec409f490 msgr2=0x7feec40a1940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-03-09T14:46:06.755 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.752+0000 7feedaf09700 1 --2- 172.21.15.38:0/2348391980 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7feec409f490 0x7feec40a1940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:06.755 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.752+0000 7feedaf09700 1 -- 172.21.15.38:0/2348391980 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feed40ffaa0 msgr2=0x7feed419f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:06.755 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.752+0000 7feedaf09700 1 --2- 172.21.15.38:0/2348391980 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feed40ffaa0 0x7feed419f3b0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7feecc007ee0 tx=0x7feecc00d3b0 comp rx=0 tx=0).stop 2023-03-09T14:46:06.755 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.752+0000 7feedaf09700 1 -- 172.21.15.38:0/2348391980 shutdown_connections 2023-03-09T14:46:06.755 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.752+0000 7feedaf09700 1 --2- 172.21.15.38:0/2348391980 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7feed40ffaa0 0x7feed419f3b0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:06.756 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.752+0000 7feedaf09700 1 --2- 172.21.15.38:0/2348391980 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7feec409f490 0x7feec40a1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:06.756 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.752+0000 7feedaf09700 1 --2- 172.21.15.38:0/2348391980 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7feed40fe7a0 0x7feed4197e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:06.756 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.752+0000 7feedaf09700 1 --2- 172.21.15.38:0/2348391980 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7feed40ff0f0 0x7feed4198340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:06.756 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.752+0000 7feedaf09700 1 -- 172.21.15.38:0/2348391980 >> 172.21.15.38:0/2348391980 conn(0x7feed40fa3b0 msgr2=0x7feed40fc150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:06.756 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.752+0000 7feedaf09700 1 -- 172.21.15.38:0/2348391980 shutdown_connections 2023-03-09T14:46:06.757 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.752+0000 7feedaf09700 1 -- 172.21.15.38:0/2348391980 wait complete. 2023-03-09T14:46:06.767 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1M' 2023-03-09T14:46:06.898 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.896+0000 7f4b96761700 1 -- 172.21.15.38:0/1622649802 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b901032b0 msgr2=0x7f4b90105690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:06.899 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.896+0000 7f4b96761700 1 --2- 172.21.15.38:0/1622649802 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b901032b0 0x7f4b90105690 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f4b84009990 tx=0x7f4b84009ca0 comp rx=0 tx=0).stop 2023-03-09T14:46:06.899 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.896+0000 7f4b96761700 1 -- 172.21.15.38:0/1622649802 shutdown_connections 2023-03-09T14:46:06.899 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.896+0000 7f4b96761700 1 --2- 172.21.15.38:0/1622649802 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4b90105bd0 0x7f4b90107fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:06.899 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.896+0000 7f4b96761700 1 --2- 172.21.15.38:0/1622649802 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4b90100990 0x7f4b90102d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:06.899 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.896+0000 7f4b96761700 1 --2- 172.21.15.38:0/1622649802 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b901032b0 0x7f4b90105690 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:06.900 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.896+0000 7f4b96761700 1 -- 172.21.15.38:0/1622649802 >> 172.21.15.38:0/1622649802 conn(0x7f4b900fa3b0 msgr2=0x7f4b900fc800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:06.900 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.896+0000 7f4b96761700 1 -- 172.21.15.38:0/1622649802 shutdown_connections 2023-03-09T14:46:06.900 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.896+0000 7f4b96761700 1 -- 172.21.15.38:0/1622649802 wait complete. 2023-03-09T14:46:06.900 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.896+0000 7f4b96761700 1 Processor -- start 2023-03-09T14:46:06.900 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.896+0000 7f4b96761700 1 -- start start 2023-03-09T14:46:06.902 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.897+0000 7f4b96761700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4b90100990 0x7f4b90195c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:06.902 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.897+0000 7f4b96761700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b901032b0 0x7f4b90196190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:06.902 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.897+0000 7f4b96761700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4b90105bd0 0x7f4b9019d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:06.902 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.897+0000 7f4b96761700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b9019d790 con 0x7f4b901032b0 2023-03-09T14:46:06.902 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.897+0000 7f4b96761700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b9019d900 con 0x7f4b90105bd0 2023-03-09T14:46:06.903 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.897+0000 7f4b96761700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b9019dac0 con 0x7f4b90100990 2023-03-09T14:46:06.903 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.897+0000 7f4b8f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b901032b0 0x7f4b90196190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:06.903 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.897+0000 7f4b8ffff700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4b90100990 0x7f4b90195c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:06.903 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.897+0000 7f4b8f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b901032b0 0x7f4b90196190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:38874/0 (socket says 172.21.15.38:38874) 2023-03-09T14:46:06.903 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.897+0000 7f4b8f7fe700 1 -- 172.21.15.38:0/3638506349 learned_addr learned my addr 172.21.15.38:0/3638506349 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:46:06.903 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.897+0000 7f4b94cfe700 1 --2- 172.21.15.38:0/3638506349 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4b90105bd0 0x7f4b9019d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:06.903 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.897+0000 7f4b8ffff700 1 -- 172.21.15.38:0/3638506349 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4b90105bd0 msgr2=0x7f4b9019d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:06.904 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.897+0000 7f4b8ffff700 1 --2- 172.21.15.38:0/3638506349 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4b90105bd0 0x7f4b9019d1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:06.904 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.897+0000 7f4b8ffff700 1 -- 172.21.15.38:0/3638506349 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b901032b0 msgr2=0x7f4b90196190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:06.904 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.897+0000 7f4b8ffff700 1 --2- 172.21.15.38:0/3638506349 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b901032b0 0x7f4b90196190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:06.904 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.897+0000 7f4b8ffff700 1 -- 172.21.15.38:0/3638506349 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b840095f0 con 0x7f4b90100990 2023-03-09T14:46:06.904 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.897+0000 7f4b8f7fe700 1 --2- 172.21.15.38:0/3638506349 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b901032b0 0x7f4b90196190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:46:06.904 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.898+0000 7f4b8ffff700 1 --2- 172.21.15.38:0/3638506349 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4b90100990 0x7f4b90195c50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4b8000e9f0 tx=0x7f4b8000ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:06.904 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.898+0000 7f4b8d7fa700 1 -- 172.21.15.38:0/3638506349 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b8000c9c0 con 0x7f4b90100990 2023-03-09T14:46:06.904 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.898+0000 7f4b96761700 1 -- 172.21.15.38:0/3638506349 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b9010b850 con 0x7f4b90100990 2023-03-09T14:46:06.905 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.898+0000 7f4b8d7fa700 1 -- 172.21.15.38:0/3638506349 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4b80010430 con 0x7f4b90100990 2023-03-09T14:46:06.905 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.898+0000 7f4b8d7fa700 1 -- 172.21.15.38:0/3638506349 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b80018680 con 0x7f4b90100990 2023-03-09T14:46:06.905 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.898+0000 7f4b96761700 1 -- 172.21.15.38:0/3638506349 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b9010b6c0 con 0x7f4b90100990 2023-03-09T14:46:06.905 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.899+0000 7f4b96761700 1 -- 172.21.15.38:0/3638506349 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b9019e120 con 0x7f4b90100990 2023-03-09T14:46:06.906 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.901+0000 7f4b8d7fa700 1 -- 172.21.15.38:0/3638506349 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 19) v1 ==== 133215+0+0 (secure 0 0 0) 0x7f4b8000c040 con 0x7f4b90100990 2023-03-09T14:46:06.906 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.901+0000 7f4b8d7fa700 1 --2- 172.21.15.38:0/3638506349 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f4b7809f410 0x7f4b780a18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:06.906 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.901+0000 7f4b8f7fe700 1 -- 172.21.15.38:0/3638506349 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f4b7809f410 msgr2=0x7f4b780a18c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/3328887328 2023-03-09T14:46:06.906 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.901+0000 7f4b8d7fa700 1 -- 172.21.15.38:0/3638506349 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f4b80014070 con 0x7f4b90100990 2023-03-09T14:46:06.906 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.901+0000 7f4b8f7fe700 1 --2- 172.21.15.38:0/3638506349 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f4b7809f410 0x7f4b780a18c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-03-09T14:46:06.907 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:06.904+0000 7f4b8d7fa700 1 -- 172.21.15.38:0/3638506349 <== mon.1 v2:172.21.15.181:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f4b80094540 con 0x7f4b90100990 2023-03-09T14:46:06.967 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:46:06 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:46:06.709+0000 7f40d86ca200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-03-09T14:46:06.967 INFO:journalctl@ceph.mgr.y.smithi122.stdout:Mar 09 14:46:06 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-y[95318]: 2023-03-09T14:46:06.578+0000 7fc7c08d5200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-03-09T14:46:06.968 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 09 14:46:06 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-z[96215]: 2023-03-09T14:46:06.573+0000 7fbe3a67f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-03-09T14:46:07.104 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:07.102+0000 7f4b8f7fe700 1 -- 172.21.15.38:0/3638506349 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f4b7809f410 msgr2=0x7f4b780a18c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/3328887328 2023-03-09T14:46:07.104 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:07.102+0000 7f4b8f7fe700 1 --2- 172.21.15.38:0/3638506349 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f4b7809f410 0x7f4b780a18c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-03-09T14:46:07.137 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:07.135+0000 7f4b96761700 1 -- 172.21.15.38:0/3638506349 --> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f4b901024e0 con 0x7f4b7809f410 2023-03-09T14:46:07.228 INFO:journalctl@ceph.mgr.y.smithi122.stdout:Mar 09 14:46:07 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-y[95318]: 2023-03-09T14:46:07.183+0000 7fc7c08d5200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-03-09T14:46:07.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:06 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/1223986875' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2023-03-09T14:46:07.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:06 smithi122 ceph-mon[93869]: mgrmap e19: x(active, since 7m), standbys: z, y 2023-03-09T14:46:07.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:06 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/1557066563' entity='client.admin' 2023-03-09T14:46:07.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:06 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/2348391980' entity='client.admin' 2023-03-09T14:46:07.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:06 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/1223986875' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2023-03-09T14:46:07.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:06 smithi181 ceph-mon[94189]: mgrmap e19: x(active, since 7m), standbys: z, y 2023-03-09T14:46:07.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:06 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/1557066563' entity='client.admin' 2023-03-09T14:46:07.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:06 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/2348391980' entity='client.admin' 2023-03-09T14:46:07.232 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 09 14:46:07 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-z[96215]: 2023-03-09T14:46:07.175+0000 7fbe3a67f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-03-09T14:46:07.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:06 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/1223986875' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2023-03-09T14:46:07.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:06 smithi038 ceph-mon[92422]: mgrmap e19: x(active, since 7m), standbys: z, y 2023-03-09T14:46:07.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:06 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/1557066563' entity='client.admin' 2023-03-09T14:46:07.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:06 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/2348391980' entity='client.admin' 2023-03-09T14:46:07.505 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:07.503+0000 7f4b8f7fe700 1 -- 172.21.15.38:0/3638506349 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f4b7809f410 msgr2=0x7f4b780a18c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/3328887328 2023-03-09T14:46:07.506 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:07.503+0000 7f4b8f7fe700 1 --2- 172.21.15.38:0/3638506349 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f4b7809f410 0x7f4b780a18c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-03-09T14:46:07.727 INFO:journalctl@ceph.mgr.y.smithi122.stdout:Mar 09 14:46:07 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-y[95318]: 2023-03-09T14:46:07.272+0000 7fc7c08d5200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-03-09T14:46:07.728 INFO:journalctl@ceph.mgr.y.smithi122.stdout:Mar 09 14:46:07 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-y[95318]: 2023-03-09T14:46:07.362+0000 7fc7c08d5200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-03-09T14:46:07.728 INFO:journalctl@ceph.mgr.y.smithi122.stdout:Mar 09 14:46:07 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-y[95318]: 2023-03-09T14:46:07.515+0000 7fc7c08d5200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-03-09T14:46:07.731 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 09 14:46:07 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-z[96215]: 2023-03-09T14:46:07.265+0000 7fbe3a67f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-03-09T14:46:07.731 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 09 14:46:07 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-z[96215]: 2023-03-09T14:46:07.353+0000 7fbe3a67f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-03-09T14:46:07.731 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 09 14:46:07 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-z[96215]: 2023-03-09T14:46:07.507+0000 7fbe3a67f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-03-09T14:46:07.750 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:46:07 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:46:07.478+0000 7f40d86ca200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-03-09T14:46:07.751 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:46:07 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:46:07.591+0000 7f40d86ca200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-03-09T14:46:07.751 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:46:07 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:46:07.700+0000 7f40d86ca200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-03-09T14:46:08.228 INFO:journalctl@ceph.mgr.y.smithi122.stdout:Mar 09 14:46:07 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-y[95318]: 2023-03-09T14:46:07.951+0000 7fc7c08d5200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-03-09T14:46:08.231 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 09 14:46:07 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-z[96215]: 2023-03-09T14:46:07.943+0000 7fbe3a67f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-03-09T14:46:08.250 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:46:07 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:46:07.877+0000 7f40d86ca200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-03-09T14:46:08.306 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:08.304+0000 7f4b8f7fe700 1 -- 172.21.15.38:0/3638506349 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f4b7809f410 msgr2=0x7f4b780a18c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/3328887328 2023-03-09T14:46:08.306 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:08.304+0000 7f4b8f7fe700 1 --2- 172.21.15.38:0/3638506349 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f4b7809f410 0x7f4b780a18c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-03-09T14:46:08.750 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:46:08 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:46:08.406+0000 7f40d86ca200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-03-09T14:46:08.977 INFO:journalctl@ceph.mgr.y.smithi122.stdout:Mar 09 14:46:08 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-y[95318]: 2023-03-09T14:46:08.535+0000 7fc7c08d5200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-03-09T14:46:08.978 INFO:journalctl@ceph.mgr.y.smithi122.stdout:Mar 09 14:46:08 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-y[95318]: 2023-03-09T14:46:08.618+0000 7fc7c08d5200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-03-09T14:46:08.981 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 09 14:46:08 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-z[96215]: 2023-03-09T14:46:08.529+0000 7fbe3a67f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-03-09T14:46:08.981 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 09 14:46:08 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-z[96215]: 2023-03-09T14:46:08.612+0000 7fbe3a67f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-03-09T14:46:09.500 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:46:09 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:46:09.118+0000 7f40d86ca200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-03-09T14:46:09.501 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:46:09 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:46:09.219+0000 7f40d86ca200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-03-09T14:46:09.638 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 09 14:46:09 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-z[96215]: 2023-03-09T14:46:09.373+0000 7fbe3a67f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-03-09T14:46:09.639 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 09 14:46:09 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-z[96215]: 2023-03-09T14:46:09.465+0000 7fbe3a67f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-03-09T14:46:09.642 INFO:journalctl@ceph.mgr.y.smithi122.stdout:Mar 09 14:46:09 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-y[95318]: 2023-03-09T14:46:09.377+0000 7fc7c08d5200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-03-09T14:46:09.643 INFO:journalctl@ceph.mgr.y.smithi122.stdout:Mar 09 14:46:09 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-y[95318]: 2023-03-09T14:46:09.468+0000 7fc7c08d5200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-03-09T14:46:09.907 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:09.905+0000 7f4b8f7fe700 1 -- 172.21.15.38:0/3638506349 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f4b7809f410 msgr2=0x7f4b780a18c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/3328887328 2023-03-09T14:46:09.908 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:09.905+0000 7f4b8f7fe700 1 --2- 172.21.15.38:0/3638506349 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f4b7809f410 0x7f4b780a18c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-03-09T14:46:09.977 INFO:journalctl@ceph.mgr.y.smithi122.stdout:Mar 09 14:46:09 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-y[95318]: 2023-03-09T14:46:09.640+0000 7fc7c08d5200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-03-09T14:46:09.981 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 09 14:46:09 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-z[96215]: 2023-03-09T14:46:09.635+0000 7fbe3a67f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-03-09T14:46:10.478 INFO:journalctl@ceph.mgr.y.smithi122.stdout:Mar 09 14:46:10 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-y[95318]: 2023-03-09T14:46:10.057+0000 7fc7c08d5200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-03-09T14:46:10.478 INFO:journalctl@ceph.mgr.y.smithi122.stdout:Mar 09 14:46:10 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-y[95318]: 2023-03-09T14:46:10.145+0000 7fc7c08d5200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-03-09T14:46:10.478 INFO:journalctl@ceph.mgr.y.smithi122.stdout:Mar 09 14:46:10 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-y[95318]: 2023-03-09T14:46:10.293+0000 7fc7c08d5200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-03-09T14:46:10.481 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 09 14:46:10 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-z[96215]: 2023-03-09T14:46:10.054+0000 7fbe3a67f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-03-09T14:46:10.482 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 09 14:46:10 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-z[96215]: 2023-03-09T14:46:10.141+0000 7fbe3a67f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-03-09T14:46:10.482 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 09 14:46:10 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-z[96215]: 2023-03-09T14:46:10.288+0000 7fbe3a67f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-03-09T14:46:10.500 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:46:10 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:46:10.182+0000 7f40d86ca200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-03-09T14:46:10.501 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:46:10 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:46:10.295+0000 7f40d86ca200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-03-09T14:46:10.977 INFO:journalctl@ceph.mgr.y.smithi122.stdout:Mar 09 14:46:10 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-y[95318]: 2023-03-09T14:46:10.554+0000 7fc7c08d5200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-03-09T14:46:10.981 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 09 14:46:10 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-z[96215]: 2023-03-09T14:46:10.550+0000 7fbe3a67f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-03-09T14:46:11.000 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:46:10 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:46:10.516+0000 7f40d86ca200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-03-09T14:46:11.320 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:46:11 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:46:11.020+0000 7f40d86ca200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-03-09T14:46:11.320 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:46:11 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:46:11.131+0000 7f40d86ca200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-03-09T14:46:11.583 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 09 14:46:11 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-z[96215]: 2023-03-09T14:46:11.258+0000 7fbe3a67f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-03-09T14:46:11.583 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 09 14:46:11 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-z[96215]: 2023-03-09T14:46:11.389+0000 7fbe3a67f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-03-09T14:46:11.584 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 09 14:46:11 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-z[96215]: 2023-03-09T14:46:11.580+0000 7fbe3a67f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-03-09T14:46:11.584 INFO:journalctl@ceph.mgr.y.smithi122.stdout:Mar 09 14:46:11 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-y[95318]: 2023-03-09T14:46:11.260+0000 7fc7c08d5200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-03-09T14:46:11.585 INFO:journalctl@ceph.mgr.y.smithi122.stdout:Mar 09 14:46:11 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-y[95318]: 2023-03-09T14:46:11.391+0000 7fc7c08d5200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-03-09T14:46:11.656 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:46:11 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:46:11.317+0000 7f40d86ca200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-03-09T14:46:11.913 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 09 14:46:11 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-z[96215]: 2023-03-09T14:46:11.694+0000 7fbe3a67f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-03-09T14:46:11.914 INFO:journalctl@ceph.mgr.y.smithi122.stdout:Mar 09 14:46:11 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-y[95318]: 2023-03-09T14:46:11.582+0000 7fc7c08d5200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-03-09T14:46:11.914 INFO:journalctl@ceph.mgr.y.smithi122.stdout:Mar 09 14:46:11 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-y[95318]: 2023-03-09T14:46:11.696+0000 7fc7c08d5200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-03-09T14:46:12.000 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:46:11 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:46:11.652+0000 7f40d86ca200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-03-09T14:46:12.227 INFO:journalctl@ceph.mgr.y.smithi122.stdout:Mar 09 14:46:11 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-y[95318]: 2023-03-09T14:46:11.911+0000 7fc7c08d5200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-03-09T14:46:12.231 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 09 14:46:11 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-z[96215]: 2023-03-09T14:46:11.909+0000 7fbe3a67f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-03-09T14:46:12.751 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:46:12 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:46:12.495+0000 7f40d86ca200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-03-09T14:46:12.751 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:46:12 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:46:12.658+0000 7f40d86ca200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-03-09T14:46:13.111 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:13.108+0000 7f4b8f7fe700 1 -- 172.21.15.38:0/3638506349 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f4b7809f410 msgr2=0x7f4b780a18c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.38:6800/3328887328 2023-03-09T14:46:13.111 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:13.108+0000 7f4b8f7fe700 1 --2- 172.21.15.38:0/3638506349 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f4b7809f410 0x7f4b780a18c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2023-03-09T14:46:13.228 INFO:journalctl@ceph.mgr.y.smithi122.stdout:Mar 09 14:46:12 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-y[95318]: 2023-03-09T14:46:12.762+0000 7fc7c08d5200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-03-09T14:46:13.231 INFO:journalctl@ceph.mgr.z.smithi181.stdout:Mar 09 14:46:12 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-z[96215]: 2023-03-09T14:46:12.765+0000 7fbe3a67f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-03-09T14:46:13.250 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:46:12 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:46:12.907+0000 7f40d86ca200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-03-09T14:46:13.251 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:46:13 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:46:13.048+0000 7f40d86ca200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-03-09T14:46:13.538 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:13.535+0000 7f4b8d7fa700 1 -- 172.21.15.38:0/3638506349 <== mon.1 v2:172.21.15.181:3300/0 7 ==== mgrmap(e 20) v1 ==== 133215+0+0 (secure 0 0 0) 0x7f4b800105a0 con 0x7f4b90100990 2023-03-09T14:46:13.750 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:46:13 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:46:13.325+0000 7f40d86ca200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-03-09T14:46:13.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:13 smithi038 ceph-mon[92422]: Standby manager daemon y restarted 2023-03-09T14:46:13.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:13 smithi038 ceph-mon[92422]: Standby manager daemon y started 2023-03-09T14:46:13.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:13 smithi038 ceph-mon[92422]: from='mgr.? 172.21.15.122:0/898138531' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-03-09T14:46:13.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:13 smithi038 ceph-mon[92422]: from='mgr.? 172.21.15.122:0/898138531' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-03-09T14:46:13.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:13 smithi038 ceph-mon[92422]: from='mgr.? 172.21.15.122:0/898138531' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-03-09T14:46:13.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:13 smithi038 ceph-mon[92422]: from='mgr.? 172.21.15.122:0/898138531' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-03-09T14:46:13.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:13 smithi038 ceph-mon[92422]: Standby manager daemon z restarted 2023-03-09T14:46:13.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:13 smithi038 ceph-mon[92422]: Standby manager daemon z started 2023-03-09T14:46:13.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:13 smithi038 ceph-mon[92422]: from='mgr.? 172.21.15.181:0/2514881573' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-03-09T14:46:13.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:13 smithi038 ceph-mon[92422]: from='mgr.? 172.21.15.181:0/2514881573' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-03-09T14:46:13.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:13 smithi038 ceph-mon[92422]: from='mgr.? 172.21.15.181:0/2514881573' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-03-09T14:46:13.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:13 smithi038 ceph-mon[92422]: from='mgr.? 172.21.15.181:0/2514881573' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-03-09T14:46:13.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:13 smithi122 ceph-mon[93869]: Standby manager daemon y restarted 2023-03-09T14:46:13.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:13 smithi122 ceph-mon[93869]: Standby manager daemon y started 2023-03-09T14:46:13.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:13 smithi122 ceph-mon[93869]: from='mgr.? 172.21.15.122:0/898138531' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-03-09T14:46:13.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:13 smithi122 ceph-mon[93869]: from='mgr.? 172.21.15.122:0/898138531' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-03-09T14:46:13.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:13 smithi122 ceph-mon[93869]: from='mgr.? 172.21.15.122:0/898138531' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-03-09T14:46:13.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:13 smithi122 ceph-mon[93869]: from='mgr.? 172.21.15.122:0/898138531' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-03-09T14:46:13.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:13 smithi122 ceph-mon[93869]: Standby manager daemon z restarted 2023-03-09T14:46:13.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:13 smithi122 ceph-mon[93869]: Standby manager daemon z started 2023-03-09T14:46:13.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:13 smithi122 ceph-mon[93869]: from='mgr.? 172.21.15.181:0/2514881573' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-03-09T14:46:13.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:13 smithi122 ceph-mon[93869]: from='mgr.? 172.21.15.181:0/2514881573' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-03-09T14:46:13.980 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:13 smithi122 ceph-mon[93869]: from='mgr.? 172.21.15.181:0/2514881573' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-03-09T14:46:13.980 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:13 smithi122 ceph-mon[93869]: from='mgr.? 172.21.15.181:0/2514881573' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-03-09T14:46:13.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:13 smithi181 ceph-mon[94189]: Standby manager daemon y restarted 2023-03-09T14:46:13.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:13 smithi181 ceph-mon[94189]: Standby manager daemon y started 2023-03-09T14:46:13.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:13 smithi181 ceph-mon[94189]: from='mgr.? 172.21.15.122:0/898138531' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-03-09T14:46:13.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:13 smithi181 ceph-mon[94189]: from='mgr.? 172.21.15.122:0/898138531' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-03-09T14:46:13.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:13 smithi181 ceph-mon[94189]: from='mgr.? 172.21.15.122:0/898138531' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-03-09T14:46:13.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:13 smithi181 ceph-mon[94189]: from='mgr.? 172.21.15.122:0/898138531' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-03-09T14:46:13.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:13 smithi181 ceph-mon[94189]: Standby manager daemon z restarted 2023-03-09T14:46:13.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:13 smithi181 ceph-mon[94189]: Standby manager daemon z started 2023-03-09T14:46:13.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:13 smithi181 ceph-mon[94189]: from='mgr.? 172.21.15.181:0/2514881573' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-03-09T14:46:13.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:13 smithi181 ceph-mon[94189]: from='mgr.? 172.21.15.181:0/2514881573' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-03-09T14:46:13.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:13 smithi181 ceph-mon[94189]: from='mgr.? 172.21.15.181:0/2514881573' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-03-09T14:46:13.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:13 smithi181 ceph-mon[94189]: from='mgr.? 172.21.15.181:0/2514881573' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-03-09T14:46:14.543 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:14.540+0000 7f4b8d7fa700 1 -- 172.21.15.38:0/3638506349 <== mon.1 v2:172.21.15.181:3300/0 8 ==== mgrmap(e 21) v1 ==== 132942+0+0 (secure 0 0 0) 0x7f4b80010860 con 0x7f4b90100990 2023-03-09T14:46:14.543 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:14.540+0000 7f4b8d7fa700 1 -- 172.21.15.38:0/3638506349 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f4b7809f410 msgr2=0x7f4b780a18c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-03-09T14:46:14.544 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:14.540+0000 7f4b8d7fa700 1 --2- 172.21.15.38:0/3638506349 >> [v2:172.21.15.38:6800/3328887328,v1:172.21.15.38:6801/3328887328] conn(0x7f4b7809f410 0x7f4b780a18c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:14.714 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:14 smithi038 ceph-mon[92422]: mgrmap e20: x(active, since 7m), standbys: z, y 2023-03-09T14:46:14.714 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:14 smithi038 ceph-mon[92422]: Active manager daemon x restarted 2023-03-09T14:46:14.714 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:14 smithi038 ceph-mon[92422]: Activating manager daemon x 2023-03-09T14:46:14.715 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:14 smithi038 ceph-mon[92422]: osdmap e90: 12 total, 12 up, 12 in 2023-03-09T14:46:14.715 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:46:14 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:46:14.381+0000 7f40d86ca200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-03-09T14:46:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:14 smithi122 ceph-mon[93869]: mgrmap e20: x(active, since 7m), standbys: z, y 2023-03-09T14:46:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:14 smithi122 ceph-mon[93869]: Active manager daemon x restarted 2023-03-09T14:46:14.935 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:14 smithi122 ceph-mon[93869]: Activating manager daemon x 2023-03-09T14:46:14.936 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:14 smithi122 ceph-mon[93869]: osdmap e90: 12 total, 12 up, 12 in 2023-03-09T14:46:14.957 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:14 smithi181 ceph-mon[94189]: mgrmap e20: x(active, since 7m), standbys: z, y 2023-03-09T14:46:14.957 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:14 smithi181 ceph-mon[94189]: Active manager daemon x restarted 2023-03-09T14:46:14.957 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:14 smithi181 ceph-mon[94189]: Activating manager daemon x 2023-03-09T14:46:14.958 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:14 smithi181 ceph-mon[94189]: osdmap e90: 12 total, 12 up, 12 in 2023-03-09T14:46:14.967 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:46:14 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:46:14.756+0000 7f403d987700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-09T14:46:14.968 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:46:14 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:46:14.756+0000 7f403d987700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-09T14:46:14.968 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:46:14 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:46:14.756+0000 7f403d987700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-09T14:46:14.968 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:46:14 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:46:14.756+0000 7f403d987700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-09T14:46:14.968 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:46:14 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:46:14.756+0000 7f403d987700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-09T14:46:14.968 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:46:14 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:46:14.756+0000 7f403997f700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-09T14:46:14.968 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:46:14 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:46:14.756+0000 7f403997f700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-09T14:46:14.968 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:46:14 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:46:14.756+0000 7f403997f700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-09T14:46:14.969 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:46:14 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:46:14.756+0000 7f403997f700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-09T14:46:14.969 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:46:14 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:46:14.756+0000 7f403997f700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-09T14:46:15.564 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:15.560+0000 7f4b8d7fa700 1 -- 172.21.15.38:0/3638506349 <== mon.1 v2:172.21.15.181:3300/0 9 ==== mgrmap(e 22) v1 ==== 133052+0+0 (secure 0 0 0) 0x7f4b80094910 con 0x7f4b90100990 2023-03-09T14:46:15.564 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:15.561+0000 7f4b8d7fa700 1 --2- 172.21.15.38:0/3638506349 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f4b78030a70 0x7f4b78030e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:15.564 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:15.561+0000 7f4b8d7fa700 1 -- 172.21.15.38:0/3638506349 --> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f4b901024e0 con 0x7f4b78030a70 2023-03-09T14:46:15.566 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:15.563+0000 7f4b8f7fe700 1 --2- 172.21.15.38:0/3638506349 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f4b78030a70 0x7f4b78030e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:15.569 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:15.566+0000 7f4b8f7fe700 1 --2- 172.21.15.38:0/3638506349 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f4b78030a70 0x7f4b78030e40 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f4b8400b2e0 tx=0x7f4b84005fb0 comp rx=0 tx=0).ready entity=mgr.15558 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:15.639 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:15.635+0000 7f4b8d7fa700 1 -- 172.21.15.38:0/3638506349 <== mgr.15558 v2:172.21.15.38:6800/39994335 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f4b901024e0 con 0x7f4b78030a70 2023-03-09T14:46:15.639 INFO:teuthology.orchestra.run.smithi038.stdout:Schedule set for path / 2023-03-09T14:46:15.641 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:15.638+0000 7f4b96761700 1 -- 172.21.15.38:0/3638506349 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f4b78030a70 msgr2=0x7f4b78030e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:15.642 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:15.638+0000 7f4b96761700 1 --2- 172.21.15.38:0/3638506349 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f4b78030a70 0x7f4b78030e40 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f4b8400b2e0 tx=0x7f4b84005fb0 comp rx=0 tx=0).stop 2023-03-09T14:46:15.642 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:15.638+0000 7f4b96761700 1 -- 172.21.15.38:0/3638506349 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4b90100990 msgr2=0x7f4b90195c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:15.642 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:15.638+0000 7f4b96761700 1 --2- 172.21.15.38:0/3638506349 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4b90100990 0x7f4b90195c50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4b8000e9f0 tx=0x7f4b8000ed00 comp rx=0 tx=0).stop 2023-03-09T14:46:15.642 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:15.638+0000 7f4b94cfe700 1 -- 172.21.15.38:0/3638506349 reap_dead start 2023-03-09T14:46:15.642 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:15.638+0000 7f4b96761700 1 -- 172.21.15.38:0/3638506349 shutdown_connections 2023-03-09T14:46:15.643 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:15.638+0000 7f4b96761700 1 -- 172.21.15.38:0/3638506349 >> 172.21.15.38:0/3638506349 conn(0x7f4b900fa3b0 msgr2=0x7f4b90101dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:15.643 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:15.638+0000 7f4b96761700 1 -- 172.21.15.38:0/3638506349 shutdown_connections 2023-03-09T14:46:15.643 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:15.638+0000 7f4b96761700 1 -- 172.21.15.38:0/3638506349 wait complete. 2023-03-09T14:46:15.654 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6M3h' 2023-03-09T14:46:15.792 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:15.787+0000 7f1c07486700 1 -- 172.21.15.38:0/492159926 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c00101a00 msgr2=0x7f1c00108330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:15.792 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:15.787+0000 7f1c07486700 1 --2- 172.21.15.38:0/492159926 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c00101a00 0x7f1c00108330 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f1bf800b0a0 tx=0x7f1bf800b3b0 comp rx=0 tx=0).stop 2023-03-09T14:46:15.793 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:15.787+0000 7f1c07486700 1 -- 172.21.15.38:0/492159926 shutdown_connections 2023-03-09T14:46:15.793 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:15.787+0000 7f1c07486700 1 --2- 172.21.15.38:0/492159926 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c00101a00 0x7f1c00108330 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:15.793 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:15.787+0000 7f1c07486700 1 --2- 172.21.15.38:0/492159926 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1c000ffe70 0x7f1c00100280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:15.793 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:15.787+0000 7f1c07486700 1 --2- 172.21.15.38:0/492159926 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1c00101050 0x7f1c001014c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:15.793 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:15.787+0000 7f1c07486700 1 -- 172.21.15.38:0/492159926 >> 172.21.15.38:0/492159926 conn(0x7f1c000fb4e0 msgr2=0x7f1c000fd910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:15.793 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:15.787+0000 7f1c07486700 1 -- 172.21.15.38:0/492159926 shutdown_connections 2023-03-09T14:46:15.793 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:15.787+0000 7f1c07486700 1 -- 172.21.15.38:0/492159926 wait complete. 2023-03-09T14:46:15.794 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:15.788+0000 7f1c07486700 1 Processor -- start 2023-03-09T14:46:15.794 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:15.788+0000 7f1c07486700 1 -- start start 2023-03-09T14:46:15.794 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:15.788+0000 7f1c07486700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1c000ffe70 0x7f1c00074530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:15.794 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:15.788+0000 7f1c07486700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1c00101050 0x7f1c00072b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:15.794 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:15.788+0000 7f1c07486700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c00101a00 0x7f1c000730b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:15.794 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:15.788+0000 7f1c07486700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c00073850 con 0x7f1c00101a00 2023-03-09T14:46:15.794 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:15.788+0000 7f1c07486700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c000739c0 con 0x7f1c000ffe70 2023-03-09T14:46:15.795 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:15.788+0000 7f1c07486700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c0019fd80 con 0x7f1c00101050 2023-03-09T14:46:15.795 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:15.788+0000 7f1c05a23700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c00101a00 0x7f1c000730b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:15.795 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:15.788+0000 7f1c05a23700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c00101a00 0x7f1c000730b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39166/0 (socket says 172.21.15.38:39166) 2023-03-09T14:46:15.795 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:15.788+0000 7f1c05a23700 1 -- 172.21.15.38:0/1982303028 learned_addr learned my addr 172.21.15.38:0/1982303028 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:46:15.795 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:15.788+0000 7f1c05222700 1 --2- 172.21.15.38:0/1982303028 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1c000ffe70 0x7f1c00074530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:15.795 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:15.788+0000 7f1c04a21700 1 --2- 172.21.15.38:0/1982303028 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1c00101050 0x7f1c00072b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:15.796 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:15.788+0000 7f1c05a23700 1 -- 172.21.15.38:0/1982303028 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1c00101050 msgr2=0x7f1c00072b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:15.796 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:15.788+0000 7f1c05a23700 1 --2- 172.21.15.38:0/1982303028 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1c00101050 0x7f1c00072b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:15.796 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:15.788+0000 7f1c05a23700 1 -- 172.21.15.38:0/1982303028 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1c000ffe70 msgr2=0x7f1c00074530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:15.796 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:15.788+0000 7f1c05a23700 1 --2- 172.21.15.38:0/1982303028 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1c000ffe70 0x7f1c00074530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:15.796 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:15.788+0000 7f1c05a23700 1 -- 172.21.15.38:0/1982303028 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bf8009d00 con 0x7f1c00101a00 2023-03-09T14:46:15.796 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:15.789+0000 7f1c05a23700 1 --2- 172.21.15.38:0/1982303028 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c00101a00 0x7f1c000730b0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f1bf800b780 tx=0x7f1bf800bd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:15.796 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:15.789+0000 7f1bf67fc700 1 -- 172.21.15.38:0/1982303028 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf800e040 con 0x7f1c00101a00 2023-03-09T14:46:15.797 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:15.789+0000 7f1bf67fc700 1 -- 172.21.15.38:0/1982303028 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1bf80094a0 con 0x7f1c00101a00 2023-03-09T14:46:15.797 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:15.789+0000 7f1c07486700 1 -- 172.21.15.38:0/1982303028 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c0010b830 con 0x7f1c00101a00 2023-03-09T14:46:15.797 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:15.789+0000 7f1bf67fc700 1 -- 172.21.15.38:0/1982303028 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1bf8005390 con 0x7f1c00101a00 2023-03-09T14:46:15.797 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:15.789+0000 7f1c07486700 1 -- 172.21.15.38:0/1982303028 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c0010b6a0 con 0x7f1c00101a00 2023-03-09T14:46:15.798 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:15.790+0000 7f1c07486700 1 -- 172.21.15.38:0/1982303028 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c00106e20 con 0x7f1c00101a00 2023-03-09T14:46:15.798 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:15.791+0000 7f1bf67fc700 1 -- 172.21.15.38:0/1982303028 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 22) v1 ==== 133052+0+0 (secure 0 0 0) 0x7f1bf801d470 con 0x7f1c00101a00 2023-03-09T14:46:15.798 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:15.792+0000 7f1bf67fc700 1 --2- 172.21.15.38:0/1982303028 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f1bec09f280 0x7f1bec0a1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:15.799 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:15.792+0000 7f1c05222700 1 --2- 172.21.15.38:0/1982303028 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f1bec09f280 0x7f1bec0a1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:15.799 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:15.792+0000 7f1bf67fc700 1 -- 172.21.15.38:0/1982303028 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9278+0+0 (secure 0 0 0) 0x7f1bf80cb7f0 con 0x7f1c00101a00 2023-03-09T14:46:15.799 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:15.792+0000 7f1c05222700 1 --2- 172.21.15.38:0/1982303028 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f1bec09f280 0x7f1bec0a1730 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1c00100bc0 tx=0x7f1bf0009260 comp rx=0 tx=0).ready entity=mgr.15558 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:15.799 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:15.796+0000 7f1bf67fc700 1 -- 172.21.15.38:0/1982303028 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f1bf80958b0 con 0x7f1c00101a00 2023-03-09T14:46:15.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:15 smithi122 ceph-mon[93869]: mgrmap e21: x(active, starting, since 0.143436s), standbys: z, y 2023-03-09T14:46:15.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-03-09T14:46:15.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-03-09T14:46:15.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-03-09T14:46:15.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-03-09T14:46:15.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-03-09T14:46:15.980 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-03-09T14:46:15.980 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-03-09T14:46:15.980 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-03-09T14:46:15.981 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-03-09T14:46:15.981 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-03-09T14:46:15.981 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-03-09T14:46:15.981 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-03-09T14:46:15.981 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-03-09T14:46:15.982 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-03-09T14:46:15.982 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-03-09T14:46:15.982 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-03-09T14:46:15.982 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-03-09T14:46:15.982 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-03-09T14:46:15.982 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-03-09T14:46:15.982 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-03-09T14:46:15.983 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-03-09T14:46:15.983 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-03-09T14:46:15.983 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-03-09T14:46:15.983 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-03-09T14:46:15.983 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-03-09T14:46:15.983 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-03-09T14:46:15.984 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-03-09T14:46:15.984 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-03-09T14:46:15.984 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-03-09T14:46:15.984 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-03-09T14:46:15.984 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-03-09T14:46:15.985 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-03-09T14:46:15.986 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-03-09T14:46:15.986 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:15 smithi122 ceph-mon[93869]: Manager daemon x is now available 2023-03-09T14:46:15.986 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:46:15.986 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:46:15.986 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-03-09T14:46:15.986 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-03-09T14:46:15.987 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:15 smithi181 ceph-mon[94189]: mgrmap e21: x(active, starting, since 0.143436s), standbys: z, y 2023-03-09T14:46:15.987 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-03-09T14:46:15.987 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-03-09T14:46:15.988 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-03-09T14:46:15.988 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-03-09T14:46:15.988 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-03-09T14:46:15.988 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-03-09T14:46:15.988 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-03-09T14:46:15.988 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-03-09T14:46:15.988 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-03-09T14:46:15.989 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-03-09T14:46:15.989 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-03-09T14:46:15.989 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-03-09T14:46:15.989 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-03-09T14:46:15.989 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-03-09T14:46:15.989 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-03-09T14:46:15.989 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-03-09T14:46:15.990 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-03-09T14:46:15.990 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-03-09T14:46:15.990 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-03-09T14:46:15.990 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-03-09T14:46:15.990 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-03-09T14:46:15.990 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-03-09T14:46:15.990 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-03-09T14:46:15.990 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-03-09T14:46:15.991 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-03-09T14:46:15.991 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-03-09T14:46:15.991 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-03-09T14:46:15.991 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-03-09T14:46:15.991 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-03-09T14:46:15.991 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-03-09T14:46:15.991 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-03-09T14:46:15.992 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-03-09T14:46:15.992 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-03-09T14:46:15.992 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:15 smithi181 ceph-mon[94189]: Manager daemon x is now available 2023-03-09T14:46:15.992 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:46:15.992 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:46:15.992 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-03-09T14:46:15.992 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-03-09T14:46:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:15 smithi038 ceph-mon[92422]: mgrmap e21: x(active, starting, since 0.143436s), standbys: z, y 2023-03-09T14:46:16.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-03-09T14:46:16.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-03-09T14:46:16.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-03-09T14:46:16.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-03-09T14:46:16.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-03-09T14:46:16.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-03-09T14:46:16.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-03-09T14:46:16.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-03-09T14:46:16.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-03-09T14:46:16.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-03-09T14:46:16.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-03-09T14:46:16.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-03-09T14:46:16.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-03-09T14:46:16.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-03-09T14:46:16.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-03-09T14:46:16.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-03-09T14:46:16.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-03-09T14:46:16.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-03-09T14:46:16.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-03-09T14:46:16.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-03-09T14:46:16.004 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-03-09T14:46:16.004 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-03-09T14:46:16.004 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-03-09T14:46:16.004 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-03-09T14:46:16.004 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-03-09T14:46:16.004 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-03-09T14:46:16.004 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-03-09T14:46:16.004 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-03-09T14:46:16.005 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-03-09T14:46:16.005 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-03-09T14:46:16.005 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-03-09T14:46:16.005 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-03-09T14:46:16.005 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-03-09T14:46:16.005 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:15 smithi038 ceph-mon[92422]: Manager daemon x is now available 2023-03-09T14:46:16.005 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:46:16.006 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:46:16.006 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-03-09T14:46:16.006 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-03-09T14:46:16.034 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.031+0000 7f1c07486700 1 -- 172.21.15.38:0/1982303028 --> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6M3h", "target": ["mon-mgr", ""]}) v1 -- 0x7f1c00061650 con 0x7f1bec09f280 2023-03-09T14:46:16.089 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.086+0000 7f1bf67fc700 1 -- 172.21.15.38:0/1982303028 <== mgr.15558 v2:172.21.15.38:6800/39994335 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f1c00061650 con 0x7f1bec09f280 2023-03-09T14:46:16.090 INFO:teuthology.orchestra.run.smithi038.stdout:Retention added to path / 2023-03-09T14:46:16.091 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.088+0000 7f1c07486700 1 -- 172.21.15.38:0/1982303028 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f1bec09f280 msgr2=0x7f1bec0a1730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:16.092 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.088+0000 7f1c07486700 1 --2- 172.21.15.38:0/1982303028 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f1bec09f280 0x7f1bec0a1730 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1c00100bc0 tx=0x7f1bf0009260 comp rx=0 tx=0).stop 2023-03-09T14:46:16.092 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.088+0000 7f1c07486700 1 -- 172.21.15.38:0/1982303028 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c00101a00 msgr2=0x7f1c000730b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:16.092 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.088+0000 7f1c07486700 1 --2- 172.21.15.38:0/1982303028 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c00101a00 0x7f1c000730b0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f1bf800b780 tx=0x7f1bf800bd60 comp rx=0 tx=0).stop 2023-03-09T14:46:16.092 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.088+0000 7f1c07486700 1 -- 172.21.15.38:0/1982303028 shutdown_connections 2023-03-09T14:46:16.092 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.088+0000 7f1c07486700 1 --2- 172.21.15.38:0/1982303028 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f1bec09f280 0x7f1bec0a1730 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:16.092 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.088+0000 7f1c07486700 1 --2- 172.21.15.38:0/1982303028 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1c00101a00 0x7f1c000730b0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:16.092 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.088+0000 7f1c07486700 1 --2- 172.21.15.38:0/1982303028 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1c000ffe70 0x7f1c00074530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:16.093 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.088+0000 7f1c07486700 1 --2- 172.21.15.38:0/1982303028 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1c00101050 0x7f1c00072b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:16.093 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.088+0000 7f1c07486700 1 -- 172.21.15.38:0/1982303028 >> 172.21.15.38:0/1982303028 conn(0x7f1c000fb4e0 msgr2=0x7f1c000fd720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:16.093 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.088+0000 7f1c07486700 1 -- 172.21.15.38:0/1982303028 shutdown_connections 2023-03-09T14:46:16.093 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.088+0000 7f1c07486700 1 -- 172.21.15.38:0/1982303028 wait complete. 2023-03-09T14:46:16.104 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2023-03-09T14:46:16.235 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.231+0000 7f4b8cc10700 1 -- 172.21.15.38:0/4064532580 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b880ff070 msgr2=0x7f4b880ff4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:16.235 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.231+0000 7f4b8cc10700 1 --2- 172.21.15.38:0/4064532580 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b880ff070 0x7f4b880ff4e0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c009990 tx=0x7f4b7c009ca0 comp rx=0 tx=0).stop 2023-03-09T14:46:16.235 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.231+0000 7f4b8cc10700 1 -- 172.21.15.38:0/4064532580 shutdown_connections 2023-03-09T14:46:16.236 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.231+0000 7f4b8cc10700 1 --2- 172.21.15.38:0/4064532580 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4b880ffa20 0x7f4b880ffe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:16.236 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.231+0000 7f4b8cc10700 1 --2- 172.21.15.38:0/4064532580 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4b880fe720 0x7f4b880feb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:16.236 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.232+0000 7f4b8cc10700 1 --2- 172.21.15.38:0/4064532580 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b880ff070 0x7f4b880ff4e0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:16.236 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.232+0000 7f4b8cc10700 1 -- 172.21.15.38:0/4064532580 >> 172.21.15.38:0/4064532580 conn(0x7f4b880fa3b0 msgr2=0x7f4b880fc800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:16.236 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.232+0000 7f4b8cc10700 1 -- 172.21.15.38:0/4064532580 shutdown_connections 2023-03-09T14:46:16.236 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.232+0000 7f4b8cc10700 1 -- 172.21.15.38:0/4064532580 wait complete. 2023-03-09T14:46:16.238 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.232+0000 7f4b8cc10700 1 Processor -- start 2023-03-09T14:46:16.238 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.232+0000 7f4b8cc10700 1 -- start start 2023-03-09T14:46:16.238 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.233+0000 7f4b8cc10700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4b880fe720 0x7f4b88197e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:16.239 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.233+0000 7f4b8cc10700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b880ff070 0x7f4b88198380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:16.239 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.233+0000 7f4b8cc10700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4b880ffa20 0x7f4b8819f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:16.239 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.233+0000 7f4b8cc10700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b8819f9c0 con 0x7f4b880ff070 2023-03-09T14:46:16.239 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.233+0000 7f4b8cc10700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b8819fb00 con 0x7f4b880fe720 2023-03-09T14:46:16.239 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.233+0000 7f4b8cc10700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b8819fc90 con 0x7f4b880ffa20 2023-03-09T14:46:16.240 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.233+0000 7f4b85d9b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b880ff070 0x7f4b88198380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:16.240 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.233+0000 7f4b8659c700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4b880fe720 0x7f4b88197e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:16.240 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.233+0000 7f4b8659c700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4b880fe720 0x7f4b88197e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.38:48374/0 (socket says 172.21.15.38:48374) 2023-03-09T14:46:16.240 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.233+0000 7f4b8659c700 1 -- 172.21.15.38:0/797685211 learned_addr learned my addr 172.21.15.38:0/797685211 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:46:16.241 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.233+0000 7f4b85d9b700 1 --2- 172.21.15.38:0/797685211 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b880ff070 0x7f4b88198380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39186/0 (socket says 172.21.15.38:39186) 2023-03-09T14:46:16.241 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.233+0000 7f4b86d9d700 1 --2- 172.21.15.38:0/797685211 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4b880ffa20 0x7f4b8819f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:16.241 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.233+0000 7f4b8659c700 1 -- 172.21.15.38:0/797685211 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4b880ffa20 msgr2=0x7f4b8819f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:16.241 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.233+0000 7f4b8659c700 1 --2- 172.21.15.38:0/797685211 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4b880ffa20 0x7f4b8819f3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:16.241 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.233+0000 7f4b8659c700 1 -- 172.21.15.38:0/797685211 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b880ff070 msgr2=0x7f4b88198380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:16.241 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.233+0000 7f4b8659c700 1 --2- 172.21.15.38:0/797685211 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b880ff070 0x7f4b88198380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:16.241 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.233+0000 7f4b8659c700 1 -- 172.21.15.38:0/797685211 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b7c0095f0 con 0x7f4b880fe720 2023-03-09T14:46:16.242 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.233+0000 7f4b86d9d700 1 --2- 172.21.15.38:0/797685211 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4b880ffa20 0x7f4b8819f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:46:16.242 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.233+0000 7f4b85d9b700 1 --2- 172.21.15.38:0/797685211 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b880ff070 0x7f4b88198380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:46:16.242 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.233+0000 7f4b8659c700 1 --2- 172.21.15.38:0/797685211 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4b880fe720 0x7f4b88197e40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4b7800e9a0 tx=0x7f4b7800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:16.242 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.233+0000 7f4b777fe700 1 -- 172.21.15.38:0/797685211 <== mon.2 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b78009790 con 0x7f4b880fe720 2023-03-09T14:46:16.242 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.233+0000 7f4b8cc10700 1 -- 172.21.15.38:0/797685211 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b8810da70 con 0x7f4b880fe720 2023-03-09T14:46:16.242 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.233+0000 7f4b777fe700 1 -- 172.21.15.38:0/797685211 <== mon.2 v2:172.21.15.122:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4b7800cbd0 con 0x7f4b880fe720 2023-03-09T14:46:16.242 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.233+0000 7f4b777fe700 1 -- 172.21.15.38:0/797685211 <== mon.2 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b78018680 con 0x7f4b880fe720 2023-03-09T14:46:16.243 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.234+0000 7f4b8cc10700 1 -- 172.21.15.38:0/797685211 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b8810d8e0 con 0x7f4b880fe720 2023-03-09T14:46:16.243 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.234+0000 7f4b8cc10700 1 -- 172.21.15.38:0/797685211 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b88109500 con 0x7f4b880fe720 2023-03-09T14:46:16.244 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.236+0000 7f4b777fe700 1 -- 172.21.15.38:0/797685211 <== mon.2 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 22) v1 ==== 133052+0+0 (secure 0 0 0) 0x7f4b7800c040 con 0x7f4b880fe720 2023-03-09T14:46:16.244 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.236+0000 7f4b777fe700 1 --2- 172.21.15.38:0/797685211 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f4b7009f280 0x7f4b700a1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:16.244 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.236+0000 7f4b85d9b700 1 --2- 172.21.15.38:0/797685211 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f4b7009f280 0x7f4b700a1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:16.245 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.237+0000 7f4b777fe700 1 -- 172.21.15.38:0/797685211 <== mon.2 v2:172.21.15.122:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9278+0+0 (secure 0 0 0) 0x7f4b78014070 con 0x7f4b880fe720 2023-03-09T14:46:16.245 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.237+0000 7f4b85d9b700 1 --2- 172.21.15.38:0/797685211 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f4b7009f280 0x7f4b700a1730 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c00b2e0 tx=0x7f4b7c005fb0 comp rx=0 tx=0).ready entity=mgr.15558 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:16.245 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.240+0000 7f4b777fe700 1 -- 172.21.15.38:0/797685211 <== mon.2 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f4b780943a0 con 0x7f4b880fe720 2023-03-09T14:46:16.475 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.472+0000 7f4b8cc10700 1 -- 172.21.15.38:0/797685211 --> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) v1 -- 0x7f4b88061940 con 0x7f4b7009f280 2023-03-09T14:46:16.484 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.481+0000 7f4b777fe700 1 -- 172.21.15.38:0/797685211 <== mgr.15558 v2:172.21.15.38:6800/39994335 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+283 (secure 0 0 0) 0x7f4b88061940 con 0x7f4b7009f280 2023-03-09T14:46:16.485 INFO:teuthology.orchestra.run.smithi038.stdout:{"fs": "cephfs", "subvol": null, "path": "/", "rel_path": "/", "schedule": "1M", "retention": {"h": 3, "M": 6}, "start": "2023-03-09T00:00:00", "created": "2023-03-09T14:46:15", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2023-03-09T14:46:16.487 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.483+0000 7f4b8cc10700 1 -- 172.21.15.38:0/797685211 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f4b7009f280 msgr2=0x7f4b700a1730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:16.487 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.483+0000 7f4b8cc10700 1 --2- 172.21.15.38:0/797685211 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f4b7009f280 0x7f4b700a1730 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c00b2e0 tx=0x7f4b7c005fb0 comp rx=0 tx=0).stop 2023-03-09T14:46:16.487 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.483+0000 7f4b8cc10700 1 -- 172.21.15.38:0/797685211 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4b880fe720 msgr2=0x7f4b88197e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:16.487 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.483+0000 7f4b8cc10700 1 --2- 172.21.15.38:0/797685211 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4b880fe720 0x7f4b88197e40 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4b7800e9a0 tx=0x7f4b7800ecb0 comp rx=0 tx=0).stop 2023-03-09T14:46:16.487 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.483+0000 7f4b8cc10700 1 -- 172.21.15.38:0/797685211 shutdown_connections 2023-03-09T14:46:16.487 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.483+0000 7f4b8cc10700 1 --2- 172.21.15.38:0/797685211 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f4b7009f280 0x7f4b700a1730 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:16.488 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.483+0000 7f4b8cc10700 1 --2- 172.21.15.38:0/797685211 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4b880ffa20 0x7f4b8819f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:16.488 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.483+0000 7f4b8cc10700 1 --2- 172.21.15.38:0/797685211 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4b880fe720 0x7f4b88197e40 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:16.488 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.483+0000 7f4b8cc10700 1 --2- 172.21.15.38:0/797685211 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4b880ff070 0x7f4b88198380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:16.488 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.483+0000 7f4b8cc10700 1 -- 172.21.15.38:0/797685211 >> 172.21.15.38:0/797685211 conn(0x7f4b880fa3b0 msgr2=0x7f4b880fc020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:16.488 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.483+0000 7f4b8cc10700 1 -- 172.21.15.38:0/797685211 shutdown_connections 2023-03-09T14:46:16.488 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.483+0000 7f4b8cc10700 1 -- 172.21.15.38:0/797685211 wait complete. 2023-03-09T14:46:16.501 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2023-03-09T14:46:16.643 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.637+0000 7f10a6cc5700 1 -- 172.21.15.38:0/1355697666 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10a0100930 msgr2=0x7f10a01071f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:16.643 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.637+0000 7f10a6cc5700 1 --2- 172.21.15.38:0/1355697666 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10a0100930 0x7f10a01071f0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f109800b0a0 tx=0x7f109800b3b0 comp rx=0 tx=0).stop 2023-03-09T14:46:16.643 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.637+0000 7f10a6cc5700 1 -- 172.21.15.38:0/1355697666 shutdown_connections 2023-03-09T14:46:16.643 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.638+0000 7f10a6cc5700 1 --2- 172.21.15.38:0/1355697666 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10a0100930 0x7f10a01071f0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:16.644 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.638+0000 7f10a6cc5700 1 --2- 172.21.15.38:0/1355697666 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f10a00fedb0 0x7f10a00ff1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:16.644 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.638+0000 7f10a6cc5700 1 --2- 172.21.15.38:0/1355697666 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f10a00fffa0 0x7f10a01003f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:16.644 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.638+0000 7f10a6cc5700 1 -- 172.21.15.38:0/1355697666 >> 172.21.15.38:0/1355697666 conn(0x7f10a00fa3b0 msgr2=0x7f10a00fc800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:16.644 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.638+0000 7f10a6cc5700 1 -- 172.21.15.38:0/1355697666 shutdown_connections 2023-03-09T14:46:16.644 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.638+0000 7f10a6cc5700 1 -- 172.21.15.38:0/1355697666 wait complete. 2023-03-09T14:46:16.644 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.638+0000 7f10a6cc5700 1 Processor -- start 2023-03-09T14:46:16.644 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.638+0000 7f10a6cc5700 1 -- start start 2023-03-09T14:46:16.645 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.638+0000 7f10a6cc5700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10a00fedb0 0x7f10a0102d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:16.645 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.638+0000 7f10a6cc5700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f10a00fffa0 0x7f10a0101380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:16.645 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.638+0000 7f10a6cc5700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f10a0100930 0x7f10a01018c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:16.645 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.638+0000 7f10a6cc5700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10a0102060 con 0x7f10a00fedb0 2023-03-09T14:46:16.645 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.638+0000 7f10a6cc5700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10a01021d0 con 0x7f10a00fffa0 2023-03-09T14:46:16.645 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.638+0000 7f10a6cc5700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10a019ff10 con 0x7f10a0100930 2023-03-09T14:46:16.645 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.638+0000 7f10a4a61700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10a00fedb0 0x7f10a0102d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:16.646 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.639+0000 7f10a4a61700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10a00fedb0 0x7f10a0102d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39196/0 (socket says 172.21.15.38:39196) 2023-03-09T14:46:16.646 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.639+0000 7f10a4a61700 1 -- 172.21.15.38:0/393746741 learned_addr learned my addr 172.21.15.38:0/393746741 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:46:16.646 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.639+0000 7f109ffff700 1 --2- 172.21.15.38:0/393746741 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f10a00fffa0 0x7f10a0101380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:16.646 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.639+0000 7f10a5262700 1 --2- 172.21.15.38:0/393746741 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f10a0100930 0x7f10a01018c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:16.646 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.639+0000 7f10a4a61700 1 -- 172.21.15.38:0/393746741 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f10a0100930 msgr2=0x7f10a01018c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:16.647 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.639+0000 7f10a4a61700 1 --2- 172.21.15.38:0/393746741 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f10a0100930 0x7f10a01018c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:16.647 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.639+0000 7f10a4a61700 1 -- 172.21.15.38:0/393746741 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f10a00fffa0 msgr2=0x7f10a0101380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:16.647 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.639+0000 7f10a4a61700 1 --2- 172.21.15.38:0/393746741 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f10a00fffa0 0x7f10a0101380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:16.647 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.639+0000 7f10a4a61700 1 -- 172.21.15.38:0/393746741 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1094009520 con 0x7f10a00fedb0 2023-03-09T14:46:16.647 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.639+0000 7f109ffff700 1 --2- 172.21.15.38:0/393746741 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f10a00fffa0 0x7f10a0101380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:46:16.647 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.639+0000 7f10a5262700 1 --2- 172.21.15.38:0/393746741 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f10a0100930 0x7f10a01018c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-09T14:46:16.647 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.639+0000 7f10a4a61700 1 --2- 172.21.15.38:0/393746741 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10a00fedb0 0x7f10a0102d40 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f109400ea50 tx=0x7f109400ee10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:16.647 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.639+0000 7f109dffb700 1 -- 172.21.15.38:0/393746741 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f109400cc90 con 0x7f10a00fedb0 2023-03-09T14:46:16.648 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.639+0000 7f10a6cc5700 1 -- 172.21.15.38:0/393746741 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1098009d00 con 0x7f10a00fedb0 2023-03-09T14:46:16.648 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.639+0000 7f109dffb700 1 -- 172.21.15.38:0/393746741 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f109400cdf0 con 0x7f10a00fedb0 2023-03-09T14:46:16.648 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.639+0000 7f109dffb700 1 -- 172.21.15.38:0/393746741 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10940187a0 con 0x7f10a00fedb0 2023-03-09T14:46:16.648 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.639+0000 7f10a6cc5700 1 -- 172.21.15.38:0/393746741 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10a010a770 con 0x7f10a00fedb0 2023-03-09T14:46:16.649 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.640+0000 7f10a6cc5700 1 -- 172.21.15.38:0/393746741 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10a0066ff0 con 0x7f10a00fedb0 2023-03-09T14:46:16.649 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.643+0000 7f109dffb700 1 -- 172.21.15.38:0/393746741 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 23) v1 ==== 133335+0+0 (secure 0 0 0) 0x7f109400c040 con 0x7f10a00fedb0 2023-03-09T14:46:16.649 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.643+0000 7f109dffb700 1 --2- 172.21.15.38:0/393746741 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f109009f4e0 0x7f10900a1990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:16.650 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.644+0000 7f109dffb700 1 -- 172.21.15.38:0/393746741 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9278+0+0 (secure 0 0 0) 0x7f1094014070 con 0x7f10a00fedb0 2023-03-09T14:46:16.650 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.644+0000 7f109ffff700 1 --2- 172.21.15.38:0/393746741 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f109009f4e0 0x7f10900a1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:16.650 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.644+0000 7f109ffff700 1 --2- 172.21.15.38:0/393746741 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f109009f4e0 0x7f10900a1990 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f108c009cb0 tx=0x7f108c009260 comp rx=0 tx=0).ready entity=mgr.15558 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:16.650 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.647+0000 7f109dffb700 1 -- 172.21.15.38:0/393746741 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f1094095010 con 0x7f10a00fedb0 2023-03-09T14:46:16.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:16 smithi038 ceph-mon[92422]: mgrmap e22: x(active, since 1.16653s), standbys: z, y 2023-03-09T14:46:16.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:16 smithi038 ceph-mon[92422]: [09/Mar/2023:14:46:15] ENGINE Bus STARTING 2023-03-09T14:46:16.888 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.885+0000 7f10a6cc5700 1 -- 172.21.15.38:0/393746741 --> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f10a00fc5b0 con 0x7f109009f4e0 2023-03-09T14:46:16.895 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.892+0000 7f109dffb700 1 -- 172.21.15.38:0/393746741 <== mgr.15558 v2:172.21.15.38:6800/39994335 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+10 (secure 0 0 0) 0x7f10a00fc5b0 con 0x7f109009f4e0 2023-03-09T14:46:16.896 INFO:teuthology.orchestra.run.smithi038.stdout:/ 1M 6M3h 2023-03-09T14:46:16.897 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.894+0000 7f10a6cc5700 1 -- 172.21.15.38:0/393746741 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f109009f4e0 msgr2=0x7f10900a1990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:16.898 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.894+0000 7f10a6cc5700 1 --2- 172.21.15.38:0/393746741 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f109009f4e0 0x7f10900a1990 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f108c009cb0 tx=0x7f108c009260 comp rx=0 tx=0).stop 2023-03-09T14:46:16.898 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.894+0000 7f10a6cc5700 1 -- 172.21.15.38:0/393746741 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10a00fedb0 msgr2=0x7f10a0102d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:16.898 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.894+0000 7f10a6cc5700 1 --2- 172.21.15.38:0/393746741 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10a00fedb0 0x7f10a0102d40 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f109400ea50 tx=0x7f109400ee10 comp rx=0 tx=0).stop 2023-03-09T14:46:16.898 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.894+0000 7f10a6cc5700 1 -- 172.21.15.38:0/393746741 shutdown_connections 2023-03-09T14:46:16.898 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.894+0000 7f10a6cc5700 1 --2- 172.21.15.38:0/393746741 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f109009f4e0 0x7f10900a1990 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:16.898 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.894+0000 7f10a6cc5700 1 --2- 172.21.15.38:0/393746741 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f10a00fedb0 0x7f10a0102d40 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:16.899 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.894+0000 7f10a6cc5700 1 --2- 172.21.15.38:0/393746741 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f10a0100930 0x7f10a01018c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:16.899 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.894+0000 7f10a6cc5700 1 --2- 172.21.15.38:0/393746741 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f10a00fffa0 0x7f10a0101380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:16.899 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.894+0000 7f10a6cc5700 1 -- 172.21.15.38:0/393746741 >> 172.21.15.38:0/393746741 conn(0x7f10a00fa3b0 msgr2=0x7f10a00fbea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:16.899 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.894+0000 7f10a6cc5700 1 -- 172.21.15.38:0/393746741 shutdown_connections 2023-03-09T14:46:16.899 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:16.894+0000 7f10a6cc5700 1 -- 172.21.15.38:0/393746741 wait complete. 2023-03-09T14:46:16.910 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2023-03-09T14:46:16.941 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2023-03-09T14:46:16.954 INFO:teuthology.run_tasks:Running task exec... 2023-03-09T14:46:16.995 INFO:teuthology.task.exec:Executing custom commands... 2023-03-09T14:46:16.996 INFO:teuthology.task.exec:Running commands on role client.0 host ubuntu@smithi038.front.sepia.ceph.com 2023-03-09T14:46:16.996 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ls -l /tmp/cephfs/ && sudo df -h /tmp/cephfs/' 2023-03-09T14:46:16.997 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:16 smithi181 ceph-mon[94189]: mgrmap e22: x(active, since 1.16653s), standbys: z, y 2023-03-09T14:46:16.997 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:16 smithi181 ceph-mon[94189]: [09/Mar/2023:14:46:15] ENGINE Bus STARTING 2023-03-09T14:46:16.997 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:16 smithi122 ceph-mon[93869]: mgrmap e22: x(active, since 1.16653s), standbys: z, y 2023-03-09T14:46:16.998 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:16 smithi122 ceph-mon[93869]: [09/Mar/2023:14:46:15] ENGINE Bus STARTING 2023-03-09T14:46:17.035 INFO:teuthology.orchestra.run.smithi038.stdout:total 0 2023-03-09T14:46:17.071 INFO:teuthology.orchestra.run.smithi038.stdout:Filesystem Size Used Avail Use% Mounted on 2023-03-09T14:46:17.071 INFO:teuthology.orchestra.run.smithi038.stdout:172.21.15.38:6789,172.21.15.122:6789,172.21.15.181:6789:/volumes/_nogroup/sv_1/ba63a2c0-19ef-41bb-becc-0a0d0bb2d986 1.1T 84M 1.1T 1% /tmp/cephfs 2023-03-09T14:46:17.073 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo mkdir -m 755 --context=system_u:system_r:postgresql_t:s0 /tmp/cephfs/postgres && sudo chown postgres:postgres /tmp/cephfs/postgres' 2023-03-09T14:46:17.170 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- mkdir -m 700 --context=system_u:system_r:postgresql_t:s0 /tmp/cephfs/postgres/data' 2023-03-09T14:46:17.214 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo semanage fcontext -a -t postgresql_db_t "/tmp/cephfs/postgres(/.*)?"' 2023-03-09T14:46:17.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:17 smithi122 ceph-mon[93869]: from='client.15621 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6M3h", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:46:17.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:17 smithi122 ceph-mon[93869]: [09/Mar/2023:14:46:16] ENGINE Serving on https://172.21.15.38:7150 2023-03-09T14:46:17.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:17 smithi122 ceph-mon[93869]: [09/Mar/2023:14:46:16] ENGINE Serving on http://172.21.15.38:8765 2023-03-09T14:46:17.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:17 smithi122 ceph-mon[93869]: [09/Mar/2023:14:46:16] ENGINE Bus STARTED 2023-03-09T14:46:17.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:17 smithi122 ceph-mon[93869]: from='client.25304 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:46:17.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:17 smithi122 ceph-mon[93869]: pgmap v4: 129 pgs: 129 active+clean; 1.0 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail 2023-03-09T14:46:17.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:17 smithi122 ceph-mon[93869]: mgrmap e23: x(active, since 2s), standbys: z, y 2023-03-09T14:46:17.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:17 smithi122 ceph-mon[93869]: from='client.15633 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:46:17.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:17 smithi181 ceph-mon[94189]: from='client.15621 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6M3h", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:46:17.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:17 smithi181 ceph-mon[94189]: [09/Mar/2023:14:46:16] ENGINE Serving on https://172.21.15.38:7150 2023-03-09T14:46:17.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:17 smithi181 ceph-mon[94189]: [09/Mar/2023:14:46:16] ENGINE Serving on http://172.21.15.38:8765 2023-03-09T14:46:17.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:17 smithi181 ceph-mon[94189]: [09/Mar/2023:14:46:16] ENGINE Bus STARTED 2023-03-09T14:46:17.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:17 smithi181 ceph-mon[94189]: from='client.25304 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:46:17.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:17 smithi181 ceph-mon[94189]: pgmap v4: 129 pgs: 129 active+clean; 1.0 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail 2023-03-09T14:46:17.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:17 smithi181 ceph-mon[94189]: mgrmap e23: x(active, since 2s), standbys: z, y 2023-03-09T14:46:17.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:17 smithi181 ceph-mon[94189]: from='client.15633 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:46:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:17 smithi038 ceph-mon[92422]: from='client.15621 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6M3h", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:46:18.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:17 smithi038 ceph-mon[92422]: [09/Mar/2023:14:46:16] ENGINE Serving on https://172.21.15.38:7150 2023-03-09T14:46:18.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:17 smithi038 ceph-mon[92422]: [09/Mar/2023:14:46:16] ENGINE Serving on http://172.21.15.38:8765 2023-03-09T14:46:18.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:17 smithi038 ceph-mon[92422]: [09/Mar/2023:14:46:16] ENGINE Bus STARTED 2023-03-09T14:46:18.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:17 smithi038 ceph-mon[92422]: from='client.25304 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:46:18.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:17 smithi038 ceph-mon[92422]: pgmap v4: 129 pgs: 129 active+clean; 1.0 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail 2023-03-09T14:46:18.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:17 smithi038 ceph-mon[92422]: mgrmap e23: x(active, since 2s), standbys: z, y 2023-03-09T14:46:18.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:17 smithi038 ceph-mon[92422]: from='client.15633 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2023-03-09T14:46:18.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:18 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:46:18.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:18 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:46:18.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:18 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:46:18.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:18 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:46:18.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:18 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:46:18.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:18 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:46:18.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:18 smithi122 ceph-mon[93869]: scrub summary: idle 2023-03-09T14:46:18.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:18 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:46:18.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:18 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:46:18.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:18 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:46:18.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:18 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:46:18.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:18 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:46:18.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:18 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:46:18.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:18 smithi181 ceph-mon[94189]: scrub summary: idle 2023-03-09T14:46:19.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:18 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:46:19.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:18 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:46:19.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:18 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:46:19.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:18 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:46:19.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:18 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:46:19.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:18 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:46:19.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:18 smithi038 ceph-mon[92422]: scrub summary: idle 2023-03-09T14:46:20.092 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:19 smithi038 ceph-mon[92422]: pgmap v5: 129 pgs: 129 active+clean; 1.0 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail 2023-03-09T14:46:20.093 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:46:20.093 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:46:20.093 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:46:20.094 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:46:20.094 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:46:20.094 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:46:20.094 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:46:20.094 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:46:20.094 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:46:20.094 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:46:20.095 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:46:20.095 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:19 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T14:46:20.095 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:46:20.095 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:19 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T14:46:20.095 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:19 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T14:46:20.095 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:19 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T14:46:20.095 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:46:20.096 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:46:20.096 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:46:20.096 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:46:20.096 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:46:20.096 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:46:20.096 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:19 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi038 to 197.2M 2023-03-09T14:46:20.096 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:19 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi038 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-03-09T14:46:20.096 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:46:20.097 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:46:20.097 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:19 smithi038 ceph-mon[92422]: Updating smithi038:/etc/ceph/ceph.conf 2023-03-09T14:46:20.097 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:19 smithi038 ceph-mon[92422]: Updating smithi122:/etc/ceph/ceph.conf 2023-03-09T14:46:20.097 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:19 smithi038 ceph-mon[92422]: Updating smithi181:/etc/ceph/ceph.conf 2023-03-09T14:46:20.098 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo grep -i postgresql /etc/selinux/targeted/contexts/files/file_contexts.local' 2023-03-09T14:46:20.140 INFO:teuthology.orchestra.run.smithi038.stdout:/tmp/cephfs/postgres(/.*)? system_u:object_r:postgresql_db_t:s0 2023-03-09T14:46:20.143 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo restorecon -R -v /tmp/cephfs/postgres' 2023-03-09T14:46:20.226 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ls -lZaR /tmp/cephfs/postgres/' 2023-03-09T14:46:20.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:19 smithi122 ceph-mon[93869]: pgmap v5: 129 pgs: 129 active+clean; 1.0 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail 2023-03-09T14:46:20.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:46:20.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:46:20.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:46:20.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:46:20.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:46:20.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:46:20.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:46:20.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:46:20.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:46:20.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:46:20.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:46:20.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:19 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T14:46:20.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:46:20.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:19 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T14:46:20.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:19 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T14:46:20.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:19 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T14:46:20.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:46:20.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:46:20.231 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:46:20.231 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:46:20.231 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:46:20.231 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:46:20.231 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:19 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi038 to 197.2M 2023-03-09T14:46:20.231 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:19 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi038 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-03-09T14:46:20.231 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:46:20.232 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:46:20.232 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:19 smithi122 ceph-mon[93869]: Updating smithi038:/etc/ceph/ceph.conf 2023-03-09T14:46:20.232 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:19 smithi122 ceph-mon[93869]: Updating smithi122:/etc/ceph/ceph.conf 2023-03-09T14:46:20.232 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:19 smithi122 ceph-mon[93869]: Updating smithi181:/etc/ceph/ceph.conf 2023-03-09T14:46:20.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:19 smithi181 ceph-mon[94189]: pgmap v5: 129 pgs: 129 active+clean; 1.0 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail 2023-03-09T14:46:20.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:46:20.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:46:20.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:46:20.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:46:20.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:46:20.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:46:20.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:46:20.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:46:20.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:46:20.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:46:20.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:46:20.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:19 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T14:46:20.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:46:20.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:19 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T14:46:20.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:19 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T14:46:20.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:19 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T14:46:20.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:46:20.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:46:20.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:46:20.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:46:20.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:46:20.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:46:20.236 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:19 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi038 to 197.2M 2023-03-09T14:46:20.236 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:19 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi038 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-03-09T14:46:20.236 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:46:20.236 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:46:20.236 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:19 smithi181 ceph-mon[94189]: Updating smithi038:/etc/ceph/ceph.conf 2023-03-09T14:46:20.236 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:19 smithi181 ceph-mon[94189]: Updating smithi122:/etc/ceph/ceph.conf 2023-03-09T14:46:20.236 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:19 smithi181 ceph-mon[94189]: Updating smithi181:/etc/ceph/ceph.conf 2023-03-09T14:46:20.304 INFO:teuthology.orchestra.run.smithi038.stdout:/tmp/cephfs/postgres/: 2023-03-09T14:46:20.305 INFO:teuthology.orchestra.run.smithi038.stdout:total 0 2023-03-09T14:46:20.305 INFO:teuthology.orchestra.run.smithi038.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Mar 9 14:46 . 2023-03-09T14:46:20.305 INFO:teuthology.orchestra.run.smithi038.stdout:drwxrwxrwt. 3 root root system_u:object_r:postgresql_db_t:s0 1 Mar 9 14:46 .. 2023-03-09T14:46:20.305 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 data 2023-03-09T14:46:20.305 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:46:20.305 INFO:teuthology.orchestra.run.smithi038.stdout:/tmp/cephfs/postgres/data: 2023-03-09T14:46:20.306 INFO:teuthology.orchestra.run.smithi038.stdout:total 0 2023-03-09T14:46:20.306 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 . 2023-03-09T14:46:20.306 INFO:teuthology.orchestra.run.smithi038.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Mar 9 14:46 .. 2023-03-09T14:46:20.307 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo mkdir -p /etc/systemd/system/postgresql.service.d/ && printf '"'"'[Service]\nEnvironment=PGDATA=/tmp/cephfs/postgres/data\nEnvironment=PGLOG=/tmp/cephfs/postgres/pgstartup.log\n'"'"' | sudo tee /etc/systemd/system/postgresql.service.d/env.conf' 2023-03-09T14:46:20.395 INFO:teuthology.orchestra.run.smithi038.stdout:[Service] 2023-03-09T14:46:20.395 INFO:teuthology.orchestra.run.smithi038.stdout:Environment=PGDATA=/tmp/cephfs/postgres/data 2023-03-09T14:46:20.396 INFO:teuthology.orchestra.run.smithi038.stdout:Environment=PGLOG=/tmp/cephfs/postgres/pgstartup.log 2023-03-09T14:46:20.397 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- postgresql-setup --initdb' 2023-03-09T14:46:20.495 INFO:teuthology.orchestra.run.smithi038.stderr: * Initializing database in '/tmp/cephfs/postgres/data' 2023-03-09T14:46:21.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:20 smithi122 ceph-mon[93869]: Updating smithi122:/var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/config/ceph.conf 2023-03-09T14:46:21.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:20 smithi122 ceph-mon[93869]: Updating smithi181:/var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/config/ceph.conf 2023-03-09T14:46:21.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:20 smithi122 ceph-mon[93869]: Updating smithi038:/var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/config/ceph.conf 2023-03-09T14:46:21.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:20 smithi122 ceph-mon[93869]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2023-03-09T14:46:21.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:20 smithi122 ceph-mon[93869]: Updating smithi181:/etc/ceph/ceph.client.admin.keyring 2023-03-09T14:46:21.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:20 smithi122 ceph-mon[93869]: Updating smithi038:/etc/ceph/ceph.client.admin.keyring 2023-03-09T14:46:21.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:20 smithi122 ceph-mon[93869]: Updating smithi122:/var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/config/ceph.client.admin.keyring 2023-03-09T14:46:21.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:20 smithi122 ceph-mon[93869]: Updating smithi181:/var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/config/ceph.client.admin.keyring 2023-03-09T14:46:21.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:20 smithi122 ceph-mon[93869]: Updating smithi038:/var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/config/ceph.client.admin.keyring 2023-03-09T14:46:21.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:46:21.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:46:21.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:46:21.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:46:21.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:46:21.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:46:21.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:46:21.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:46:21.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:46:21.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:46:21.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:46:21.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:20 smithi181 ceph-mon[94189]: Updating smithi122:/var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/config/ceph.conf 2023-03-09T14:46:21.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:20 smithi181 ceph-mon[94189]: Updating smithi181:/var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/config/ceph.conf 2023-03-09T14:46:21.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:20 smithi181 ceph-mon[94189]: Updating smithi038:/var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/config/ceph.conf 2023-03-09T14:46:21.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:20 smithi181 ceph-mon[94189]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2023-03-09T14:46:21.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:20 smithi181 ceph-mon[94189]: Updating smithi181:/etc/ceph/ceph.client.admin.keyring 2023-03-09T14:46:21.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:20 smithi181 ceph-mon[94189]: Updating smithi038:/etc/ceph/ceph.client.admin.keyring 2023-03-09T14:46:21.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:20 smithi181 ceph-mon[94189]: Updating smithi122:/var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/config/ceph.client.admin.keyring 2023-03-09T14:46:21.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:20 smithi181 ceph-mon[94189]: Updating smithi181:/var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/config/ceph.client.admin.keyring 2023-03-09T14:46:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:20 smithi181 ceph-mon[94189]: Updating smithi038:/var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/config/ceph.client.admin.keyring 2023-03-09T14:46:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:46:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:46:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:46:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:46:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:46:21.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:46:21.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:46:21.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:46:21.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:46:21.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:46:21.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:46:21.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:20 smithi038 ceph-mon[92422]: Updating smithi122:/var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/config/ceph.conf 2023-03-09T14:46:21.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:20 smithi038 ceph-mon[92422]: Updating smithi181:/var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/config/ceph.conf 2023-03-09T14:46:21.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:20 smithi038 ceph-mon[92422]: Updating smithi038:/var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/config/ceph.conf 2023-03-09T14:46:21.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:20 smithi038 ceph-mon[92422]: Updating smithi122:/etc/ceph/ceph.client.admin.keyring 2023-03-09T14:46:21.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:20 smithi038 ceph-mon[92422]: Updating smithi181:/etc/ceph/ceph.client.admin.keyring 2023-03-09T14:46:21.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:20 smithi038 ceph-mon[92422]: Updating smithi038:/etc/ceph/ceph.client.admin.keyring 2023-03-09T14:46:21.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:20 smithi038 ceph-mon[92422]: Updating smithi122:/var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/config/ceph.client.admin.keyring 2023-03-09T14:46:21.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:20 smithi038 ceph-mon[92422]: Updating smithi181:/var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/config/ceph.client.admin.keyring 2023-03-09T14:46:21.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:20 smithi038 ceph-mon[92422]: Updating smithi038:/var/lib/ceph/c6733fe2-be87-11ed-9af2-001a4aab830c/config/ceph.client.admin.keyring 2023-03-09T14:46:21.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:46:21.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:46:21.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:46:21.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:46:21.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:46:21.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:46:21.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:46:21.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:46:21.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:46:21.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:46:21.254 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:46:22.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:21 smithi122 ceph-mon[93869]: pgmap v6: 129 pgs: 129 active+clean; 1.0 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail 2023-03-09T14:46:22.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:21 smithi181 ceph-mon[94189]: pgmap v6: 129 pgs: 129 active+clean; 1.0 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail 2023-03-09T14:46:22.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:21 smithi038 ceph-mon[92422]: pgmap v6: 129 pgs: 129 active+clean; 1.0 MiB data, 83 MiB used, 1.0 TiB / 1.0 TiB avail 2023-03-09T14:46:23.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:23 smithi122 ceph-mon[93869]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 134 KiB/s wr, 52 op/s 2023-03-09T14:46:23.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:23 smithi181 ceph-mon[94189]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 134 KiB/s wr, 52 op/s 2023-03-09T14:46:24.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:23 smithi038 ceph-mon[92422]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 134 KiB/s wr, 52 op/s 2023-03-09T14:46:24.219 INFO:teuthology.orchestra.run.smithi038.stderr: * Initialized, logs are in /var/lib/pgsql/initdb_postgresql.log 2023-03-09T14:46:24.222 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ls -lZaR /tmp/cephfs/postgres/' 2023-03-09T14:46:24.270 INFO:teuthology.orchestra.run.smithi038.stdout:/tmp/cephfs/postgres/: 2023-03-09T14:46:24.270 INFO:teuthology.orchestra.run.smithi038.stdout:total 0 2023-03-09T14:46:24.271 INFO:teuthology.orchestra.run.smithi038.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Mar 9 14:46 . 2023-03-09T14:46:24.271 INFO:teuthology.orchestra.run.smithi038.stdout:drwxrwxrwt. 3 root root system_u:object_r:postgresql_db_t:s0 1 Mar 9 14:46 .. 2023-03-09T14:46:24.271 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Mar 9 14:46 data 2023-03-09T14:46:24.271 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:46:24.271 INFO:teuthology.orchestra.run.smithi038.stdout:/tmp/cephfs/postgres/data: 2023-03-09T14:46:24.271 INFO:teuthology.orchestra.run.smithi038.stdout:total 30 2023-03-09T14:46:24.271 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Mar 9 14:46 . 2023-03-09T14:46:24.272 INFO:teuthology.orchestra.run.smithi038.stdout:drwxr-xr-x. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Mar 9 14:46 .. 2023-03-09T14:46:24.272 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Mar 9 14:46 base 2023-03-09T14:46:24.272 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 58 Mar 9 14:46 global 2023-03-09T14:46:24.272 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 log 2023-03-09T14:46:24.272 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 pg_commit_ts 2023-03-09T14:46:24.272 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 pg_dynshmem 2023-03-09T14:46:24.272 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 4269 Mar 9 14:46 pg_hba.conf 2023-03-09T14:46:24.272 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 1636 Mar 9 14:46 pg_ident.conf 2023-03-09T14:46:24.273 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Mar 9 14:46 pg_logical 2023-03-09T14:46:24.273 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Mar 9 14:46 pg_multixact 2023-03-09T14:46:24.273 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Mar 9 14:46 pg_notify 2023-03-09T14:46:24.273 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 pg_replslot 2023-03-09T14:46:24.273 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 pg_serial 2023-03-09T14:46:24.273 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 pg_snapshots 2023-03-09T14:46:24.273 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 pg_stat 2023-03-09T14:46:24.274 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 pg_stat_tmp 2023-03-09T14:46:24.274 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Mar 9 14:46 pg_subtrans 2023-03-09T14:46:24.274 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 pg_tblspc 2023-03-09T14:46:24.274 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 pg_twophase 2023-03-09T14:46:24.274 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Mar 9 14:46 PG_VERSION 2023-03-09T14:46:24.274 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Mar 9 14:46 pg_wal 2023-03-09T14:46:24.274 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Mar 9 14:46 pg_xact 2023-03-09T14:46:24.275 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 88 Mar 9 14:46 postgresql.auto.conf 2023-03-09T14:46:24.275 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 22989 Mar 9 14:46 postgresql.conf 2023-03-09T14:46:24.275 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:46:24.275 INFO:teuthology.orchestra.run.smithi038.stdout:/tmp/cephfs/postgres/data/base: 2023-03-09T14:46:24.275 INFO:teuthology.orchestra.run.smithi038.stdout:total 0 2023-03-09T14:46:24.275 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Mar 9 14:46 . 2023-03-09T14:46:24.275 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Mar 9 14:46 .. 2023-03-09T14:46:24.276 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 293 Mar 9 14:46 1 2023-03-09T14:46:24.276 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 293 Mar 9 14:46 12346 2023-03-09T14:46:24.276 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 293 Mar 9 14:46 12347 2023-03-09T14:46:24.276 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:46:24.276 INFO:teuthology.orchestra.run.smithi038.stdout:/tmp/cephfs/postgres/data/base/1: 2023-03-09T14:46:24.276 INFO:teuthology.orchestra.run.smithi038.stdout:total 7201 2023-03-09T14:46:24.276 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 293 Mar 9 14:46 . 2023-03-09T14:46:24.276 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Mar 9 14:46 .. 2023-03-09T14:46:24.277 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 112 2023-03-09T14:46:24.277 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 113 2023-03-09T14:46:24.277 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Mar 9 14:46 12182 2023-03-09T14:46:24.277 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 12182_fsm 2023-03-09T14:46:24.277 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 12182_vm 2023-03-09T14:46:24.277 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 12184 2023-03-09T14:46:24.277 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 12186 2023-03-09T14:46:24.278 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 12187 2023-03-09T14:46:24.278 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 12187_fsm 2023-03-09T14:46:24.278 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 12187_vm 2023-03-09T14:46:24.278 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 12189 2023-03-09T14:46:24.278 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 12191 2023-03-09T14:46:24.278 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 12192 2023-03-09T14:46:24.279 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 12192_fsm 2023-03-09T14:46:24.279 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 12192_vm 2023-03-09T14:46:24.279 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 12194 2023-03-09T14:46:24.279 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 12196 2023-03-09T14:46:24.280 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 12197 2023-03-09T14:46:24.280 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 12197_fsm 2023-03-09T14:46:24.280 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 12197_vm 2023-03-09T14:46:24.280 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 12199 2023-03-09T14:46:24.281 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 12201 2023-03-09T14:46:24.281 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 12202 2023-03-09T14:46:24.281 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 12202_fsm 2023-03-09T14:46:24.282 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 12202_vm 2023-03-09T14:46:24.282 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 12204 2023-03-09T14:46:24.282 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 12206 2023-03-09T14:46:24.282 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 12207 2023-03-09T14:46:24.283 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 12207_fsm 2023-03-09T14:46:24.283 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 12207_vm 2023-03-09T14:46:24.283 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 12209 2023-03-09T14:46:24.283 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 12211 2023-03-09T14:46:24.284 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 12212 2023-03-09T14:46:24.284 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 12214 2023-03-09T14:46:24.284 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 12216 2023-03-09T14:46:24.284 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 73728 Mar 9 14:46 1247 2023-03-09T14:46:24.284 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 1247_fsm 2023-03-09T14:46:24.284 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 1247_vm 2023-03-09T14:46:24.284 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 393216 Mar 9 14:46 1249 2023-03-09T14:46:24.284 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 1249_fsm 2023-03-09T14:46:24.285 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 1249_vm 2023-03-09T14:46:24.285 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 606208 Mar 9 14:46 1255 2023-03-09T14:46:24.285 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 1255_fsm 2023-03-09T14:46:24.285 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 1255_vm 2023-03-09T14:46:24.285 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Mar 9 14:46 1259 2023-03-09T14:46:24.285 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 1259_fsm 2023-03-09T14:46:24.285 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 1259_vm 2023-03-09T14:46:24.286 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 1417 2023-03-09T14:46:24.286 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 1417_vm 2023-03-09T14:46:24.286 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 1418 2023-03-09T14:46:24.286 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 1418_vm 2023-03-09T14:46:24.286 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 174 2023-03-09T14:46:24.286 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 175 2023-03-09T14:46:24.286 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2187 2023-03-09T14:46:24.287 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2224 2023-03-09T14:46:24.287 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2224_vm 2023-03-09T14:46:24.287 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2328 2023-03-09T14:46:24.287 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2328_vm 2023-03-09T14:46:24.287 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2336 2023-03-09T14:46:24.287 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2336_vm 2023-03-09T14:46:24.288 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2337 2023-03-09T14:46:24.288 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2600 2023-03-09T14:46:24.288 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 2600_fsm 2023-03-09T14:46:24.288 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2600_vm 2023-03-09T14:46:24.288 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2601 2023-03-09T14:46:24.289 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 2601_fsm 2023-03-09T14:46:24.289 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2601_vm 2023-03-09T14:46:24.289 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Mar 9 14:46 2602 2023-03-09T14:46:24.289 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 2602_fsm 2023-03-09T14:46:24.289 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2602_vm 2023-03-09T14:46:24.290 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 9 14:46 2603 2023-03-09T14:46:24.290 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 2603_fsm 2023-03-09T14:46:24.290 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2603_vm 2023-03-09T14:46:24.290 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2604 2023-03-09T14:46:24.290 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2604_vm 2023-03-09T14:46:24.290 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2605 2023-03-09T14:46:24.291 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 2605_fsm 2023-03-09T14:46:24.291 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2605_vm 2023-03-09T14:46:24.291 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2606 2023-03-09T14:46:24.291 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 2606_fsm 2023-03-09T14:46:24.291 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2606_vm 2023-03-09T14:46:24.291 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 2607 2023-03-09T14:46:24.291 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 2607_fsm 2023-03-09T14:46:24.292 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2607_vm 2023-03-09T14:46:24.292 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 442368 Mar 9 14:46 2608 2023-03-09T14:46:24.292 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 2608_fsm 2023-03-09T14:46:24.292 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2608_vm 2023-03-09T14:46:24.292 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 286720 Mar 9 14:46 2609 2023-03-09T14:46:24.292 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 2609_fsm 2023-03-09T14:46:24.292 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2609_vm 2023-03-09T14:46:24.293 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 2610 2023-03-09T14:46:24.293 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 2610_fsm 2023-03-09T14:46:24.293 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2610_vm 2023-03-09T14:46:24.293 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2611 2023-03-09T14:46:24.293 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2611_vm 2023-03-09T14:46:24.293 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2612 2023-03-09T14:46:24.293 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 2612_fsm 2023-03-09T14:46:24.294 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2612_vm 2023-03-09T14:46:24.294 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2613 2023-03-09T14:46:24.294 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2613_vm 2023-03-09T14:46:24.294 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2615 2023-03-09T14:46:24.294 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 2615_fsm 2023-03-09T14:46:24.294 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2615_vm 2023-03-09T14:46:24.294 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 2616 2023-03-09T14:46:24.295 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 2616_fsm 2023-03-09T14:46:24.295 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2616_vm 2023-03-09T14:46:24.295 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Mar 9 14:46 2617 2023-03-09T14:46:24.295 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 2617_fsm 2023-03-09T14:46:24.295 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2617_vm 2023-03-09T14:46:24.295 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Mar 9 14:46 2618 2023-03-09T14:46:24.295 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 2618_fsm 2023-03-09T14:46:24.295 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2618_vm 2023-03-09T14:46:24.296 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 131072 Mar 9 14:46 2619 2023-03-09T14:46:24.296 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 2619_fsm 2023-03-09T14:46:24.296 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2619_vm 2023-03-09T14:46:24.296 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2620 2023-03-09T14:46:24.296 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2620_vm 2023-03-09T14:46:24.296 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2650 2023-03-09T14:46:24.296 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2651 2023-03-09T14:46:24.297 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2652 2023-03-09T14:46:24.297 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 9 14:46 2653 2023-03-09T14:46:24.297 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 9 14:46 2654 2023-03-09T14:46:24.297 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 9 14:46 2655 2023-03-09T14:46:24.297 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2656 2023-03-09T14:46:24.297 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2657 2023-03-09T14:46:24.297 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Mar 9 14:46 2658 2023-03-09T14:46:24.298 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Mar 9 14:46 2659 2023-03-09T14:46:24.298 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2660 2023-03-09T14:46:24.298 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2661 2023-03-09T14:46:24.298 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 9 14:46 2662 2023-03-09T14:46:24.298 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Mar 9 14:46 2663 2023-03-09T14:46:24.298 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2664 2023-03-09T14:46:24.298 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2665 2023-03-09T14:46:24.298 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2666 2023-03-09T14:46:24.299 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2667 2023-03-09T14:46:24.299 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2668 2023-03-09T14:46:24.299 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2669 2023-03-09T14:46:24.299 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2670 2023-03-09T14:46:24.299 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 327680 Mar 9 14:46 2673 2023-03-09T14:46:24.299 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 376832 Mar 9 14:46 2674 2023-03-09T14:46:24.299 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 180224 Mar 9 14:46 2675 2023-03-09T14:46:24.300 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2678 2023-03-09T14:46:24.300 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2679 2023-03-09T14:46:24.300 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2680 2023-03-09T14:46:24.300 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2681 2023-03-09T14:46:24.300 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2682 2023-03-09T14:46:24.301 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2683 2023-03-09T14:46:24.301 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2684 2023-03-09T14:46:24.301 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2685 2023-03-09T14:46:24.302 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2686 2023-03-09T14:46:24.302 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2687 2023-03-09T14:46:24.302 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 9 14:46 2688 2023-03-09T14:46:24.302 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Mar 9 14:46 2689 2023-03-09T14:46:24.303 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Mar 9 14:46 2690 2023-03-09T14:46:24.303 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 270336 Mar 9 14:46 2691 2023-03-09T14:46:24.303 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2692 2023-03-09T14:46:24.303 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2693 2023-03-09T14:46:24.304 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2696 2023-03-09T14:46:24.304 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2699 2023-03-09T14:46:24.304 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2701 2023-03-09T14:46:24.304 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2702 2023-03-09T14:46:24.305 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2703 2023-03-09T14:46:24.305 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 9 14:46 2704 2023-03-09T14:46:24.305 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2753 2023-03-09T14:46:24.305 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 2753_fsm 2023-03-09T14:46:24.306 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2753_vm 2023-03-09T14:46:24.306 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2754 2023-03-09T14:46:24.306 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2755 2023-03-09T14:46:24.307 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 9 14:46 2756 2023-03-09T14:46:24.307 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 9 14:46 2757 2023-03-09T14:46:24.307 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2830 2023-03-09T14:46:24.307 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2830_vm 2023-03-09T14:46:24.307 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2831 2023-03-09T14:46:24.308 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2832 2023-03-09T14:46:24.308 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2832_vm 2023-03-09T14:46:24.308 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2833 2023-03-09T14:46:24.308 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2834 2023-03-09T14:46:24.308 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2834_vm 2023-03-09T14:46:24.308 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2835 2023-03-09T14:46:24.308 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2836 2023-03-09T14:46:24.308 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2836_vm 2023-03-09T14:46:24.309 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2837 2023-03-09T14:46:24.309 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 417792 Mar 9 14:46 2838 2023-03-09T14:46:24.309 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 2838_fsm 2023-03-09T14:46:24.309 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2838_vm 2023-03-09T14:46:24.309 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2839 2023-03-09T14:46:24.309 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 2840 2023-03-09T14:46:24.309 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 2840_fsm 2023-03-09T14:46:24.310 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2840_vm 2023-03-09T14:46:24.310 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2841 2023-03-09T14:46:24.310 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2995 2023-03-09T14:46:24.310 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2995_vm 2023-03-09T14:46:24.310 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2996 2023-03-09T14:46:24.311 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3079 2023-03-09T14:46:24.311 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 3079_fsm 2023-03-09T14:46:24.311 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3079_vm 2023-03-09T14:46:24.311 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 3080 2023-03-09T14:46:24.311 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 3081 2023-03-09T14:46:24.311 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 3085 2023-03-09T14:46:24.311 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 3118 2023-03-09T14:46:24.312 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 3118_vm 2023-03-09T14:46:24.312 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3119 2023-03-09T14:46:24.312 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 3164 2023-03-09T14:46:24.312 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 3256 2023-03-09T14:46:24.312 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 3256_vm 2023-03-09T14:46:24.313 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3257 2023-03-09T14:46:24.313 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3258 2023-03-09T14:46:24.313 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 3350 2023-03-09T14:46:24.313 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 3350_vm 2023-03-09T14:46:24.313 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3351 2023-03-09T14:46:24.313 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3379 2023-03-09T14:46:24.313 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3380 2023-03-09T14:46:24.314 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 3381 2023-03-09T14:46:24.314 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 3381_vm 2023-03-09T14:46:24.314 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 3394 2023-03-09T14:46:24.314 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 3394_fsm 2023-03-09T14:46:24.314 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3394_vm 2023-03-09T14:46:24.315 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 3395 2023-03-09T14:46:24.315 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 3439 2023-03-09T14:46:24.315 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 3439_vm 2023-03-09T14:46:24.315 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3440 2023-03-09T14:46:24.315 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 9 14:46 3455 2023-03-09T14:46:24.315 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 3456 2023-03-09T14:46:24.315 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 3456_fsm 2023-03-09T14:46:24.315 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3456_vm 2023-03-09T14:46:24.316 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 3466 2023-03-09T14:46:24.316 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 3466_vm 2023-03-09T14:46:24.317 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3467 2023-03-09T14:46:24.317 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3468 2023-03-09T14:46:24.317 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 3501 2023-03-09T14:46:24.318 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 3501_vm 2023-03-09T14:46:24.318 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3502 2023-03-09T14:46:24.318 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3503 2023-03-09T14:46:24.318 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3534 2023-03-09T14:46:24.319 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3541 2023-03-09T14:46:24.319 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 3541_fsm 2023-03-09T14:46:24.319 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3541_vm 2023-03-09T14:46:24.320 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 3542 2023-03-09T14:46:24.320 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3574 2023-03-09T14:46:24.320 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3575 2023-03-09T14:46:24.320 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 3576 2023-03-09T14:46:24.320 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 3576_vm 2023-03-09T14:46:24.321 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 3596 2023-03-09T14:46:24.321 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 3596_vm 2023-03-09T14:46:24.322 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3597 2023-03-09T14:46:24.322 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 3598 2023-03-09T14:46:24.322 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 3598_vm 2023-03-09T14:46:24.323 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3599 2023-03-09T14:46:24.323 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3600 2023-03-09T14:46:24.323 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 3600_fsm 2023-03-09T14:46:24.323 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3600_vm 2023-03-09T14:46:24.323 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3601 2023-03-09T14:46:24.324 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 3601_fsm 2023-03-09T14:46:24.324 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3601_vm 2023-03-09T14:46:24.324 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3602 2023-03-09T14:46:24.324 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 3602_fsm 2023-03-09T14:46:24.324 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3602_vm 2023-03-09T14:46:24.324 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 3603 2023-03-09T14:46:24.324 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 3603_fsm 2023-03-09T14:46:24.325 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3603_vm 2023-03-09T14:46:24.325 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 3604 2023-03-09T14:46:24.325 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 3605 2023-03-09T14:46:24.325 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 3606 2023-03-09T14:46:24.330 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 3607 2023-03-09T14:46:24.331 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 3608 2023-03-09T14:46:24.331 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 9 14:46 3609 2023-03-09T14:46:24.331 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 3712 2023-03-09T14:46:24.332 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3764 2023-03-09T14:46:24.332 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 3764_fsm 2023-03-09T14:46:24.332 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3764_vm 2023-03-09T14:46:24.332 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 3766 2023-03-09T14:46:24.332 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 3767 2023-03-09T14:46:24.333 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3997 2023-03-09T14:46:24.333 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 5002 2023-03-09T14:46:24.333 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 548 2023-03-09T14:46:24.333 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 549 2023-03-09T14:46:24.334 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 6102 2023-03-09T14:46:24.334 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 6102_vm 2023-03-09T14:46:24.334 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 6104 2023-03-09T14:46:24.335 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 6104_vm 2023-03-09T14:46:24.335 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 6106 2023-03-09T14:46:24.335 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 6106_vm 2023-03-09T14:46:24.335 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 6110 2023-03-09T14:46:24.336 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 6111 2023-03-09T14:46:24.336 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 6112 2023-03-09T14:46:24.336 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 6113 2023-03-09T14:46:24.336 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 6117 2023-03-09T14:46:24.336 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 826 2023-03-09T14:46:24.337 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 826_vm 2023-03-09T14:46:24.337 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 827 2023-03-09T14:46:24.337 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 828 2023-03-09T14:46:24.337 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Mar 9 14:46 pg_filenode.map 2023-03-09T14:46:24.338 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Mar 9 14:46 PG_VERSION 2023-03-09T14:46:24.338 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:46:24.338 INFO:teuthology.orchestra.run.smithi038.stdout:/tmp/cephfs/postgres/data/base/12346: 2023-03-09T14:46:24.338 INFO:teuthology.orchestra.run.smithi038.stdout:total 7201 2023-03-09T14:46:24.338 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 293 Mar 9 14:46 . 2023-03-09T14:46:24.339 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Mar 9 14:46 .. 2023-03-09T14:46:24.339 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 112 2023-03-09T14:46:24.339 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 113 2023-03-09T14:46:24.339 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Mar 9 14:46 12182 2023-03-09T14:46:24.340 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 12182_fsm 2023-03-09T14:46:24.340 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 12182_vm 2023-03-09T14:46:24.340 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 12184 2023-03-09T14:46:24.340 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 12186 2023-03-09T14:46:24.341 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 12187 2023-03-09T14:46:24.341 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 12187_fsm 2023-03-09T14:46:24.341 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 12187_vm 2023-03-09T14:46:24.341 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 12189 2023-03-09T14:46:24.342 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 12191 2023-03-09T14:46:24.342 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 12192 2023-03-09T14:46:24.342 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 12192_fsm 2023-03-09T14:46:24.342 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 12192_vm 2023-03-09T14:46:24.343 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 12194 2023-03-09T14:46:24.343 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 12196 2023-03-09T14:46:24.343 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 12197 2023-03-09T14:46:24.343 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 12197_fsm 2023-03-09T14:46:24.344 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 12197_vm 2023-03-09T14:46:24.344 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 12199 2023-03-09T14:46:24.344 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 12201 2023-03-09T14:46:24.344 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 12202 2023-03-09T14:46:24.345 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 12202_fsm 2023-03-09T14:46:24.345 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 12202_vm 2023-03-09T14:46:24.345 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 12204 2023-03-09T14:46:24.345 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 12206 2023-03-09T14:46:24.346 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 12207 2023-03-09T14:46:24.346 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 12207_fsm 2023-03-09T14:46:24.346 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 12207_vm 2023-03-09T14:46:24.346 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 12209 2023-03-09T14:46:24.346 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 12211 2023-03-09T14:46:24.346 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 12212 2023-03-09T14:46:24.346 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 12214 2023-03-09T14:46:24.347 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 12216 2023-03-09T14:46:24.347 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 73728 Mar 9 14:46 1247 2023-03-09T14:46:24.347 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 1247_fsm 2023-03-09T14:46:24.347 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 1247_vm 2023-03-09T14:46:24.347 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 393216 Mar 9 14:46 1249 2023-03-09T14:46:24.347 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 1249_fsm 2023-03-09T14:46:24.347 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 1249_vm 2023-03-09T14:46:24.348 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 606208 Mar 9 14:46 1255 2023-03-09T14:46:24.348 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 1255_fsm 2023-03-09T14:46:24.348 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 1255_vm 2023-03-09T14:46:24.348 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Mar 9 14:46 1259 2023-03-09T14:46:24.348 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 1259_fsm 2023-03-09T14:46:24.348 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 1259_vm 2023-03-09T14:46:24.348 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 1417 2023-03-09T14:46:24.349 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 1417_vm 2023-03-09T14:46:24.349 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 1418 2023-03-09T14:46:24.349 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 1418_vm 2023-03-09T14:46:24.349 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 174 2023-03-09T14:46:24.349 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 175 2023-03-09T14:46:24.349 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2187 2023-03-09T14:46:24.350 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2224 2023-03-09T14:46:24.350 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2224_vm 2023-03-09T14:46:24.350 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2328 2023-03-09T14:46:24.350 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2328_vm 2023-03-09T14:46:24.350 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2336 2023-03-09T14:46:24.350 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2336_vm 2023-03-09T14:46:24.351 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2337 2023-03-09T14:46:24.351 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2600 2023-03-09T14:46:24.351 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 2600_fsm 2023-03-09T14:46:24.351 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2600_vm 2023-03-09T14:46:24.351 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2601 2023-03-09T14:46:24.351 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 2601_fsm 2023-03-09T14:46:24.351 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2601_vm 2023-03-09T14:46:24.352 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Mar 9 14:46 2602 2023-03-09T14:46:24.352 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 2602_fsm 2023-03-09T14:46:24.352 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2602_vm 2023-03-09T14:46:24.352 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 9 14:46 2603 2023-03-09T14:46:24.353 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 2603_fsm 2023-03-09T14:46:24.353 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2603_vm 2023-03-09T14:46:24.353 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2604 2023-03-09T14:46:24.353 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2604_vm 2023-03-09T14:46:24.353 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2605 2023-03-09T14:46:24.353 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 2605_fsm 2023-03-09T14:46:24.354 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2605_vm 2023-03-09T14:46:24.354 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2606 2023-03-09T14:46:24.354 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 2606_fsm 2023-03-09T14:46:24.355 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2606_vm 2023-03-09T14:46:24.355 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 2607 2023-03-09T14:46:24.355 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 2607_fsm 2023-03-09T14:46:24.356 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2607_vm 2023-03-09T14:46:24.356 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 442368 Mar 9 14:46 2608 2023-03-09T14:46:24.356 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 2608_fsm 2023-03-09T14:46:24.356 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2608_vm 2023-03-09T14:46:24.357 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 286720 Mar 9 14:46 2609 2023-03-09T14:46:24.357 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 2609_fsm 2023-03-09T14:46:24.357 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2609_vm 2023-03-09T14:46:24.358 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 2610 2023-03-09T14:46:24.358 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 2610_fsm 2023-03-09T14:46:24.358 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2610_vm 2023-03-09T14:46:24.358 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2611 2023-03-09T14:46:24.359 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2611_vm 2023-03-09T14:46:24.359 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2612 2023-03-09T14:46:24.359 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 2612_fsm 2023-03-09T14:46:24.360 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2612_vm 2023-03-09T14:46:24.360 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2613 2023-03-09T14:46:24.360 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2613_vm 2023-03-09T14:46:24.360 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2615 2023-03-09T14:46:24.361 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 2615_fsm 2023-03-09T14:46:24.361 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2615_vm 2023-03-09T14:46:24.361 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 2616 2023-03-09T14:46:24.361 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 2616_fsm 2023-03-09T14:46:24.361 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2616_vm 2023-03-09T14:46:24.361 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Mar 9 14:46 2617 2023-03-09T14:46:24.361 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 2617_fsm 2023-03-09T14:46:24.362 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2617_vm 2023-03-09T14:46:24.362 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Mar 9 14:46 2618 2023-03-09T14:46:24.362 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 2618_fsm 2023-03-09T14:46:24.362 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2618_vm 2023-03-09T14:46:24.362 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 131072 Mar 9 14:46 2619 2023-03-09T14:46:24.362 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 2619_fsm 2023-03-09T14:46:24.362 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2619_vm 2023-03-09T14:46:24.363 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2620 2023-03-09T14:46:24.363 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2620_vm 2023-03-09T14:46:24.363 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2650 2023-03-09T14:46:24.363 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2651 2023-03-09T14:46:24.363 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2652 2023-03-09T14:46:24.363 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 9 14:46 2653 2023-03-09T14:46:24.364 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 9 14:46 2654 2023-03-09T14:46:24.364 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 9 14:46 2655 2023-03-09T14:46:24.364 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2656 2023-03-09T14:46:24.364 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2657 2023-03-09T14:46:24.364 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Mar 9 14:46 2658 2023-03-09T14:46:24.364 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Mar 9 14:46 2659 2023-03-09T14:46:24.364 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2660 2023-03-09T14:46:24.365 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2661 2023-03-09T14:46:24.365 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 9 14:46 2662 2023-03-09T14:46:24.365 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Mar 9 14:46 2663 2023-03-09T14:46:24.365 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2664 2023-03-09T14:46:24.365 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2665 2023-03-09T14:46:24.365 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2666 2023-03-09T14:46:24.365 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2667 2023-03-09T14:46:24.366 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2668 2023-03-09T14:46:24.366 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2669 2023-03-09T14:46:24.366 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2670 2023-03-09T14:46:24.366 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 327680 Mar 9 14:46 2673 2023-03-09T14:46:24.366 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 376832 Mar 9 14:46 2674 2023-03-09T14:46:24.366 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 180224 Mar 9 14:46 2675 2023-03-09T14:46:24.366 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2678 2023-03-09T14:46:24.367 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2679 2023-03-09T14:46:24.367 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2680 2023-03-09T14:46:24.367 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2681 2023-03-09T14:46:24.367 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2682 2023-03-09T14:46:24.367 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2683 2023-03-09T14:46:24.367 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2684 2023-03-09T14:46:24.367 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2685 2023-03-09T14:46:24.367 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2686 2023-03-09T14:46:24.368 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2687 2023-03-09T14:46:24.368 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 9 14:46 2688 2023-03-09T14:46:24.368 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Mar 9 14:46 2689 2023-03-09T14:46:24.368 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Mar 9 14:46 2690 2023-03-09T14:46:24.368 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 270336 Mar 9 14:46 2691 2023-03-09T14:46:24.368 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2692 2023-03-09T14:46:24.368 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2693 2023-03-09T14:46:24.368 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2696 2023-03-09T14:46:24.369 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2699 2023-03-09T14:46:24.369 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2701 2023-03-09T14:46:24.369 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2702 2023-03-09T14:46:24.369 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2703 2023-03-09T14:46:24.370 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 9 14:46 2704 2023-03-09T14:46:24.370 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2753 2023-03-09T14:46:24.370 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 2753_fsm 2023-03-09T14:46:24.370 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2753_vm 2023-03-09T14:46:24.371 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2754 2023-03-09T14:46:24.371 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2755 2023-03-09T14:46:24.371 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 9 14:46 2756 2023-03-09T14:46:24.371 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 9 14:46 2757 2023-03-09T14:46:24.372 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2830 2023-03-09T14:46:24.372 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2830_vm 2023-03-09T14:46:24.372 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2831 2023-03-09T14:46:24.373 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2832 2023-03-09T14:46:24.373 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2832_vm 2023-03-09T14:46:24.373 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2833 2023-03-09T14:46:24.373 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2834 2023-03-09T14:46:24.374 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2834_vm 2023-03-09T14:46:24.374 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2835 2023-03-09T14:46:24.374 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2836 2023-03-09T14:46:24.375 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2836_vm 2023-03-09T14:46:24.375 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2837 2023-03-09T14:46:24.375 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 417792 Mar 9 14:46 2838 2023-03-09T14:46:24.375 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 2838_fsm 2023-03-09T14:46:24.376 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2838_vm 2023-03-09T14:46:24.376 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2839 2023-03-09T14:46:24.377 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 2840 2023-03-09T14:46:24.377 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 2840_fsm 2023-03-09T14:46:24.377 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2840_vm 2023-03-09T14:46:24.377 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2841 2023-03-09T14:46:24.378 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2995 2023-03-09T14:46:24.378 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2995_vm 2023-03-09T14:46:24.378 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2996 2023-03-09T14:46:24.378 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3079 2023-03-09T14:46:24.378 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 3079_fsm 2023-03-09T14:46:24.379 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3079_vm 2023-03-09T14:46:24.379 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 3080 2023-03-09T14:46:24.379 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 3081 2023-03-09T14:46:24.380 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 3085 2023-03-09T14:46:24.380 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 3118 2023-03-09T14:46:24.380 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 3118_vm 2023-03-09T14:46:24.381 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3119 2023-03-09T14:46:24.381 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 3164 2023-03-09T14:46:24.381 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 3256 2023-03-09T14:46:24.381 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 3256_vm 2023-03-09T14:46:24.381 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3257 2023-03-09T14:46:24.382 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3258 2023-03-09T14:46:24.382 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 3350 2023-03-09T14:46:24.382 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 3350_vm 2023-03-09T14:46:24.382 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3351 2023-03-09T14:46:24.382 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3379 2023-03-09T14:46:24.382 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3380 2023-03-09T14:46:24.382 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 3381 2023-03-09T14:46:24.383 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 3381_vm 2023-03-09T14:46:24.383 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 3394 2023-03-09T14:46:24.383 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 3394_fsm 2023-03-09T14:46:24.383 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3394_vm 2023-03-09T14:46:24.383 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 3395 2023-03-09T14:46:24.383 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 3439 2023-03-09T14:46:24.383 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 3439_vm 2023-03-09T14:46:24.383 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3440 2023-03-09T14:46:24.384 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 9 14:46 3455 2023-03-09T14:46:24.384 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 3456 2023-03-09T14:46:24.384 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 3456_fsm 2023-03-09T14:46:24.384 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3456_vm 2023-03-09T14:46:24.384 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 3466 2023-03-09T14:46:24.384 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 3466_vm 2023-03-09T14:46:24.385 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3467 2023-03-09T14:46:24.385 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3468 2023-03-09T14:46:24.385 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 3501 2023-03-09T14:46:24.385 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 3501_vm 2023-03-09T14:46:24.385 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3502 2023-03-09T14:46:24.385 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3503 2023-03-09T14:46:24.386 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3534 2023-03-09T14:46:24.386 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3541 2023-03-09T14:46:24.386 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 3541_fsm 2023-03-09T14:46:24.386 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3541_vm 2023-03-09T14:46:24.386 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 3542 2023-03-09T14:46:24.386 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3574 2023-03-09T14:46:24.386 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3575 2023-03-09T14:46:24.387 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 3576 2023-03-09T14:46:24.387 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 3576_vm 2023-03-09T14:46:24.387 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 3596 2023-03-09T14:46:24.387 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 3596_vm 2023-03-09T14:46:24.387 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3597 2023-03-09T14:46:24.387 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 3598 2023-03-09T14:46:24.387 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 3598_vm 2023-03-09T14:46:24.388 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3599 2023-03-09T14:46:24.388 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3600 2023-03-09T14:46:24.388 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 3600_fsm 2023-03-09T14:46:24.388 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3600_vm 2023-03-09T14:46:24.388 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3601 2023-03-09T14:46:24.388 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 3601_fsm 2023-03-09T14:46:24.388 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3601_vm 2023-03-09T14:46:24.388 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3602 2023-03-09T14:46:24.389 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 3602_fsm 2023-03-09T14:46:24.389 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3602_vm 2023-03-09T14:46:24.389 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 3603 2023-03-09T14:46:24.389 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 3603_fsm 2023-03-09T14:46:24.389 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3603_vm 2023-03-09T14:46:24.389 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 3604 2023-03-09T14:46:24.390 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 3605 2023-03-09T14:46:24.390 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 3606 2023-03-09T14:46:24.390 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 3607 2023-03-09T14:46:24.390 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 3608 2023-03-09T14:46:24.390 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 9 14:46 3609 2023-03-09T14:46:24.390 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 3712 2023-03-09T14:46:24.390 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3764 2023-03-09T14:46:24.390 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 3764_fsm 2023-03-09T14:46:24.391 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3764_vm 2023-03-09T14:46:24.391 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 3766 2023-03-09T14:46:24.391 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 3767 2023-03-09T14:46:24.391 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3997 2023-03-09T14:46:24.391 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 5002 2023-03-09T14:46:24.391 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 548 2023-03-09T14:46:24.391 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 549 2023-03-09T14:46:24.391 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 6102 2023-03-09T14:46:24.392 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 6102_vm 2023-03-09T14:46:24.392 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 6104 2023-03-09T14:46:24.392 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 6104_vm 2023-03-09T14:46:24.392 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 6106 2023-03-09T14:46:24.392 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 6106_vm 2023-03-09T14:46:24.392 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 6110 2023-03-09T14:46:24.392 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 6111 2023-03-09T14:46:24.392 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 6112 2023-03-09T14:46:24.393 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 6113 2023-03-09T14:46:24.393 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 6117 2023-03-09T14:46:24.393 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 826 2023-03-09T14:46:24.394 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 826_vm 2023-03-09T14:46:24.394 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 827 2023-03-09T14:46:24.394 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 828 2023-03-09T14:46:24.394 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Mar 9 14:46 pg_filenode.map 2023-03-09T14:46:24.395 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Mar 9 14:46 PG_VERSION 2023-03-09T14:46:24.395 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:46:24.395 INFO:teuthology.orchestra.run.smithi038.stdout:/tmp/cephfs/postgres/data/base/12347: 2023-03-09T14:46:24.395 INFO:teuthology.orchestra.run.smithi038.stdout:total 7201 2023-03-09T14:46:24.396 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 293 Mar 9 14:46 . 2023-03-09T14:46:24.396 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 5 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Mar 9 14:46 .. 2023-03-09T14:46:24.396 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 112 2023-03-09T14:46:24.397 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 113 2023-03-09T14:46:24.397 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 57344 Mar 9 14:46 12182 2023-03-09T14:46:24.397 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 12182_fsm 2023-03-09T14:46:24.398 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 12182_vm 2023-03-09T14:46:24.398 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 12184 2023-03-09T14:46:24.398 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 12186 2023-03-09T14:46:24.399 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 12187 2023-03-09T14:46:24.399 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 12187_fsm 2023-03-09T14:46:24.399 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 12187_vm 2023-03-09T14:46:24.399 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 12189 2023-03-09T14:46:24.400 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 12191 2023-03-09T14:46:24.400 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 12192 2023-03-09T14:46:24.400 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 12192_fsm 2023-03-09T14:46:24.401 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 12192_vm 2023-03-09T14:46:24.401 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 12194 2023-03-09T14:46:24.401 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 12196 2023-03-09T14:46:24.401 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 12197 2023-03-09T14:46:24.401 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 12197_fsm 2023-03-09T14:46:24.401 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 12197_vm 2023-03-09T14:46:24.401 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 12199 2023-03-09T14:46:24.402 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 12201 2023-03-09T14:46:24.402 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 12202 2023-03-09T14:46:24.402 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 12202_fsm 2023-03-09T14:46:24.402 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 12202_vm 2023-03-09T14:46:24.402 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 12204 2023-03-09T14:46:24.402 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 12206 2023-03-09T14:46:24.402 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 12207 2023-03-09T14:46:24.402 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 12207_fsm 2023-03-09T14:46:24.403 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 12207_vm 2023-03-09T14:46:24.403 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 12209 2023-03-09T14:46:24.403 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 12211 2023-03-09T14:46:24.403 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 12212 2023-03-09T14:46:24.403 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 12214 2023-03-09T14:46:24.403 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 12216 2023-03-09T14:46:24.403 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 73728 Mar 9 14:46 1247 2023-03-09T14:46:24.404 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 1247_fsm 2023-03-09T14:46:24.404 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 1247_vm 2023-03-09T14:46:24.404 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 393216 Mar 9 14:46 1249 2023-03-09T14:46:24.404 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 1249_fsm 2023-03-09T14:46:24.404 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 1249_vm 2023-03-09T14:46:24.404 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 606208 Mar 9 14:46 1255 2023-03-09T14:46:24.404 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 1255_fsm 2023-03-09T14:46:24.405 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 1255_vm 2023-03-09T14:46:24.405 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 90112 Mar 9 14:46 1259 2023-03-09T14:46:24.405 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 1259_fsm 2023-03-09T14:46:24.405 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 1259_vm 2023-03-09T14:46:24.405 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 1417 2023-03-09T14:46:24.405 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 1417_vm 2023-03-09T14:46:24.405 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 1418 2023-03-09T14:46:24.405 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 1418_vm 2023-03-09T14:46:24.406 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 174 2023-03-09T14:46:24.406 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 175 2023-03-09T14:46:24.406 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2187 2023-03-09T14:46:24.406 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2224 2023-03-09T14:46:24.406 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2224_vm 2023-03-09T14:46:24.406 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2328 2023-03-09T14:46:24.406 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2328_vm 2023-03-09T14:46:24.406 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2336 2023-03-09T14:46:24.407 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2336_vm 2023-03-09T14:46:24.407 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2337 2023-03-09T14:46:24.407 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2600 2023-03-09T14:46:24.407 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 2600_fsm 2023-03-09T14:46:24.407 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2600_vm 2023-03-09T14:46:24.407 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2601 2023-03-09T14:46:24.407 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 2601_fsm 2023-03-09T14:46:24.407 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2601_vm 2023-03-09T14:46:24.408 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Mar 9 14:46 2602 2023-03-09T14:46:24.408 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 2602_fsm 2023-03-09T14:46:24.408 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2602_vm 2023-03-09T14:46:24.408 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 9 14:46 2603 2023-03-09T14:46:24.408 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 2603_fsm 2023-03-09T14:46:24.408 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2603_vm 2023-03-09T14:46:24.408 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2604 2023-03-09T14:46:24.408 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2604_vm 2023-03-09T14:46:24.409 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2605 2023-03-09T14:46:24.409 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 2605_fsm 2023-03-09T14:46:24.409 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2605_vm 2023-03-09T14:46:24.409 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2606 2023-03-09T14:46:24.409 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 2606_fsm 2023-03-09T14:46:24.409 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2606_vm 2023-03-09T14:46:24.409 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 2607 2023-03-09T14:46:24.410 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 2607_fsm 2023-03-09T14:46:24.410 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2607_vm 2023-03-09T14:46:24.410 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 442368 Mar 9 14:46 2608 2023-03-09T14:46:24.410 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 2608_fsm 2023-03-09T14:46:24.410 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2608_vm 2023-03-09T14:46:24.410 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 286720 Mar 9 14:46 2609 2023-03-09T14:46:24.410 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 2609_fsm 2023-03-09T14:46:24.410 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2609_vm 2023-03-09T14:46:24.411 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 2610 2023-03-09T14:46:24.411 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 2610_fsm 2023-03-09T14:46:24.411 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2610_vm 2023-03-09T14:46:24.411 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2611 2023-03-09T14:46:24.411 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2611_vm 2023-03-09T14:46:24.411 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2612 2023-03-09T14:46:24.411 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 2612_fsm 2023-03-09T14:46:24.411 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2612_vm 2023-03-09T14:46:24.412 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2613 2023-03-09T14:46:24.412 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2613_vm 2023-03-09T14:46:24.412 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2615 2023-03-09T14:46:24.412 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 2615_fsm 2023-03-09T14:46:24.412 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2615_vm 2023-03-09T14:46:24.412 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 2616 2023-03-09T14:46:24.412 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 2616_fsm 2023-03-09T14:46:24.413 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2616_vm 2023-03-09T14:46:24.413 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 122880 Mar 9 14:46 2617 2023-03-09T14:46:24.413 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 2617_fsm 2023-03-09T14:46:24.413 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2617_vm 2023-03-09T14:46:24.413 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 98304 Mar 9 14:46 2618 2023-03-09T14:46:24.413 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 2618_fsm 2023-03-09T14:46:24.413 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2618_vm 2023-03-09T14:46:24.414 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 131072 Mar 9 14:46 2619 2023-03-09T14:46:24.414 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 2619_fsm 2023-03-09T14:46:24.414 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2619_vm 2023-03-09T14:46:24.414 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2620 2023-03-09T14:46:24.414 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2620_vm 2023-03-09T14:46:24.414 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2650 2023-03-09T14:46:24.414 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2651 2023-03-09T14:46:24.414 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2652 2023-03-09T14:46:24.415 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 9 14:46 2653 2023-03-09T14:46:24.415 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 9 14:46 2654 2023-03-09T14:46:24.415 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 9 14:46 2655 2023-03-09T14:46:24.415 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2656 2023-03-09T14:46:24.415 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2657 2023-03-09T14:46:24.415 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 106496 Mar 9 14:46 2658 2023-03-09T14:46:24.415 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Mar 9 14:46 2659 2023-03-09T14:46:24.415 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2660 2023-03-09T14:46:24.416 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2661 2023-03-09T14:46:24.416 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 9 14:46 2662 2023-03-09T14:46:24.416 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Mar 9 14:46 2663 2023-03-09T14:46:24.416 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2664 2023-03-09T14:46:24.416 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2665 2023-03-09T14:46:24.416 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2666 2023-03-09T14:46:24.416 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2667 2023-03-09T14:46:24.416 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2668 2023-03-09T14:46:24.417 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2669 2023-03-09T14:46:24.417 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2670 2023-03-09T14:46:24.417 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 327680 Mar 9 14:46 2673 2023-03-09T14:46:24.417 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 376832 Mar 9 14:46 2674 2023-03-09T14:46:24.417 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 180224 Mar 9 14:46 2675 2023-03-09T14:46:24.417 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2678 2023-03-09T14:46:24.417 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2679 2023-03-09T14:46:24.418 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2680 2023-03-09T14:46:24.418 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2681 2023-03-09T14:46:24.418 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2682 2023-03-09T14:46:24.418 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2683 2023-03-09T14:46:24.418 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2684 2023-03-09T14:46:24.418 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2685 2023-03-09T14:46:24.418 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2686 2023-03-09T14:46:24.418 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2687 2023-03-09T14:46:24.419 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 9 14:46 2688 2023-03-09T14:46:24.419 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 49152 Mar 9 14:46 2689 2023-03-09T14:46:24.419 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 81920 Mar 9 14:46 2690 2023-03-09T14:46:24.419 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 270336 Mar 9 14:46 2691 2023-03-09T14:46:24.419 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2692 2023-03-09T14:46:24.419 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2693 2023-03-09T14:46:24.419 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2696 2023-03-09T14:46:24.419 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2699 2023-03-09T14:46:24.420 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2701 2023-03-09T14:46:24.420 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2702 2023-03-09T14:46:24.420 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2703 2023-03-09T14:46:24.420 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 9 14:46 2704 2023-03-09T14:46:24.420 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2753 2023-03-09T14:46:24.420 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 2753_fsm 2023-03-09T14:46:24.420 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2753_vm 2023-03-09T14:46:24.420 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2754 2023-03-09T14:46:24.421 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2755 2023-03-09T14:46:24.421 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 9 14:46 2756 2023-03-09T14:46:24.421 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 9 14:46 2757 2023-03-09T14:46:24.421 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2830 2023-03-09T14:46:24.421 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2830_vm 2023-03-09T14:46:24.421 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2831 2023-03-09T14:46:24.421 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2832 2023-03-09T14:46:24.422 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2832_vm 2023-03-09T14:46:24.422 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2833 2023-03-09T14:46:24.422 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2834 2023-03-09T14:46:24.422 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2834_vm 2023-03-09T14:46:24.422 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2835 2023-03-09T14:46:24.422 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2836 2023-03-09T14:46:24.422 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2836_vm 2023-03-09T14:46:24.422 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2837 2023-03-09T14:46:24.423 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 417792 Mar 9 14:46 2838 2023-03-09T14:46:24.423 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 2838_fsm 2023-03-09T14:46:24.423 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2838_vm 2023-03-09T14:46:24.423 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2839 2023-03-09T14:46:24.423 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 2840 2023-03-09T14:46:24.423 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 2840_fsm 2023-03-09T14:46:24.423 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2840_vm 2023-03-09T14:46:24.423 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2841 2023-03-09T14:46:24.424 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2995 2023-03-09T14:46:24.424 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2995_vm 2023-03-09T14:46:24.424 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2996 2023-03-09T14:46:24.424 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3079 2023-03-09T14:46:24.424 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 3079_fsm 2023-03-09T14:46:24.424 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3079_vm 2023-03-09T14:46:24.424 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 3080 2023-03-09T14:46:24.424 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 3081 2023-03-09T14:46:24.425 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 3085 2023-03-09T14:46:24.425 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 3118 2023-03-09T14:46:24.425 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 3118_vm 2023-03-09T14:46:24.425 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3119 2023-03-09T14:46:24.425 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 3164 2023-03-09T14:46:24.425 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 3256 2023-03-09T14:46:24.425 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 3256_vm 2023-03-09T14:46:24.426 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3257 2023-03-09T14:46:24.426 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3258 2023-03-09T14:46:24.426 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 3350 2023-03-09T14:46:24.426 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 3350_vm 2023-03-09T14:46:24.426 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3351 2023-03-09T14:46:24.426 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3379 2023-03-09T14:46:24.426 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3380 2023-03-09T14:46:24.426 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 3381 2023-03-09T14:46:24.427 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 3381_vm 2023-03-09T14:46:24.427 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 3394 2023-03-09T14:46:24.427 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 3394_fsm 2023-03-09T14:46:24.427 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3394_vm 2023-03-09T14:46:24.427 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 3395 2023-03-09T14:46:24.427 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 3439 2023-03-09T14:46:24.427 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 3439_vm 2023-03-09T14:46:24.427 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3440 2023-03-09T14:46:24.428 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 40960 Mar 9 14:46 3455 2023-03-09T14:46:24.428 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 3456 2023-03-09T14:46:24.428 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 3456_fsm 2023-03-09T14:46:24.428 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3456_vm 2023-03-09T14:46:24.428 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 3466 2023-03-09T14:46:24.428 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 3466_vm 2023-03-09T14:46:24.428 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3467 2023-03-09T14:46:24.428 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3468 2023-03-09T14:46:24.429 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 3501 2023-03-09T14:46:24.429 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 3501_vm 2023-03-09T14:46:24.429 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3502 2023-03-09T14:46:24.429 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3503 2023-03-09T14:46:24.429 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3534 2023-03-09T14:46:24.429 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3541 2023-03-09T14:46:24.429 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 3541_fsm 2023-03-09T14:46:24.429 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3541_vm 2023-03-09T14:46:24.430 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 3542 2023-03-09T14:46:24.430 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3574 2023-03-09T14:46:24.430 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3575 2023-03-09T14:46:24.430 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 3576 2023-03-09T14:46:24.430 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 3576_vm 2023-03-09T14:46:24.430 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 3596 2023-03-09T14:46:24.430 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 3596_vm 2023-03-09T14:46:24.430 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3597 2023-03-09T14:46:24.431 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 3598 2023-03-09T14:46:24.431 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 3598_vm 2023-03-09T14:46:24.431 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3599 2023-03-09T14:46:24.431 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3600 2023-03-09T14:46:24.431 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 3600_fsm 2023-03-09T14:46:24.431 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3600_vm 2023-03-09T14:46:24.431 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3601 2023-03-09T14:46:24.432 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 3601_fsm 2023-03-09T14:46:24.432 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3601_vm 2023-03-09T14:46:24.432 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3602 2023-03-09T14:46:24.432 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 3602_fsm 2023-03-09T14:46:24.432 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3602_vm 2023-03-09T14:46:24.432 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 3603 2023-03-09T14:46:24.432 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 3603_fsm 2023-03-09T14:46:24.432 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3603_vm 2023-03-09T14:46:24.433 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 3604 2023-03-09T14:46:24.433 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 3605 2023-03-09T14:46:24.433 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 3606 2023-03-09T14:46:24.433 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 3607 2023-03-09T14:46:24.433 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 3608 2023-03-09T14:46:24.433 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 32768 Mar 9 14:46 3609 2023-03-09T14:46:24.433 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 3712 2023-03-09T14:46:24.433 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3764 2023-03-09T14:46:24.434 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 3764_fsm 2023-03-09T14:46:24.434 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3764_vm 2023-03-09T14:46:24.434 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 3766 2023-03-09T14:46:24.434 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 3767 2023-03-09T14:46:24.434 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3997 2023-03-09T14:46:24.434 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 5002 2023-03-09T14:46:24.434 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 548 2023-03-09T14:46:24.434 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 549 2023-03-09T14:46:24.435 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 6102 2023-03-09T14:46:24.435 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 6102_vm 2023-03-09T14:46:24.435 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 6104 2023-03-09T14:46:24.435 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 6104_vm 2023-03-09T14:46:24.435 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 6106 2023-03-09T14:46:24.435 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 6106_vm 2023-03-09T14:46:24.435 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 6110 2023-03-09T14:46:24.436 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 6111 2023-03-09T14:46:24.436 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 6112 2023-03-09T14:46:24.436 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 6113 2023-03-09T14:46:24.436 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 6117 2023-03-09T14:46:24.436 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 826 2023-03-09T14:46:24.436 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 826_vm 2023-03-09T14:46:24.436 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 827 2023-03-09T14:46:24.436 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 828 2023-03-09T14:46:24.437 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Mar 9 14:46 pg_filenode.map 2023-03-09T14:46:24.437 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Mar 9 14:46 PG_VERSION 2023-03-09T14:46:24.437 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:46:24.437 INFO:teuthology.orchestra.run.smithi038.stdout:/tmp/cephfs/postgres/data/global: 2023-03-09T14:46:24.437 INFO:teuthology.orchestra.run.smithi038.stdout:total 553 2023-03-09T14:46:24.437 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 58 Mar 9 14:46 . 2023-03-09T14:46:24.437 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Mar 9 14:46 .. 2023-03-09T14:46:24.437 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 1136 2023-03-09T14:46:24.438 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 1136_fsm 2023-03-09T14:46:24.438 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 1136_vm 2023-03-09T14:46:24.438 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 1137 2023-03-09T14:46:24.438 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 1213 2023-03-09T14:46:24.438 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 1213_fsm 2023-03-09T14:46:24.438 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 1213_vm 2023-03-09T14:46:24.438 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 1214 2023-03-09T14:46:24.438 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 1214_fsm 2023-03-09T14:46:24.439 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 1214_vm 2023-03-09T14:46:24.439 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 1232 2023-03-09T14:46:24.439 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 1233 2023-03-09T14:46:24.439 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 1260 2023-03-09T14:46:24.439 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 1260_fsm 2023-03-09T14:46:24.439 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 1260_vm 2023-03-09T14:46:24.439 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 1261 2023-03-09T14:46:24.440 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 1261_fsm 2023-03-09T14:46:24.440 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 1261_vm 2023-03-09T14:46:24.440 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 1262 2023-03-09T14:46:24.440 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 1262_fsm 2023-03-09T14:46:24.440 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 1262_vm 2023-03-09T14:46:24.440 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2396 2023-03-09T14:46:24.440 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 24576 Mar 9 14:46 2396_fsm 2023-03-09T14:46:24.440 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2396_vm 2023-03-09T14:46:24.441 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2397 2023-03-09T14:46:24.441 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2671 2023-03-09T14:46:24.441 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2672 2023-03-09T14:46:24.441 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2676 2023-03-09T14:46:24.441 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2677 2023-03-09T14:46:24.441 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2694 2023-03-09T14:46:24.441 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2695 2023-03-09T14:46:24.441 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2697 2023-03-09T14:46:24.442 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16384 Mar 9 14:46 2698 2023-03-09T14:46:24.442 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2846 2023-03-09T14:46:24.442 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2846_vm 2023-03-09T14:46:24.442 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2847 2023-03-09T14:46:24.442 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2964 2023-03-09T14:46:24.442 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2964_vm 2023-03-09T14:46:24.442 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2965 2023-03-09T14:46:24.442 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2966 2023-03-09T14:46:24.443 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 2966_vm 2023-03-09T14:46:24.443 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 2967 2023-03-09T14:46:24.443 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 3592 2023-03-09T14:46:24.443 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 3592_vm 2023-03-09T14:46:24.443 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 3593 2023-03-09T14:46:24.443 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 4060 2023-03-09T14:46:24.443 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 4060_vm 2023-03-09T14:46:24.443 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 4061 2023-03-09T14:46:24.444 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 6000 2023-03-09T14:46:24.444 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 6000_vm 2023-03-09T14:46:24.444 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 6001 2023-03-09T14:46:24.444 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 6002 2023-03-09T14:46:24.444 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 6100 2023-03-09T14:46:24.444 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 6100_vm 2023-03-09T14:46:24.444 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 6114 2023-03-09T14:46:24.444 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 6115 2023-03-09T14:46:24.445 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 pg_control 2023-03-09T14:46:24.445 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 512 Mar 9 14:46 pg_filenode.map 2023-03-09T14:46:24.445 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:46:24.445 INFO:teuthology.orchestra.run.smithi038.stdout:/tmp/cephfs/postgres/data/log: 2023-03-09T14:46:24.445 INFO:teuthology.orchestra.run.smithi038.stdout:total 0 2023-03-09T14:46:24.445 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 . 2023-03-09T14:46:24.445 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Mar 9 14:46 .. 2023-03-09T14:46:24.446 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:46:24.446 INFO:teuthology.orchestra.run.smithi038.stdout:/tmp/cephfs/postgres/data/pg_commit_ts: 2023-03-09T14:46:24.446 INFO:teuthology.orchestra.run.smithi038.stdout:total 0 2023-03-09T14:46:24.446 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 . 2023-03-09T14:46:24.446 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Mar 9 14:46 .. 2023-03-09T14:46:24.446 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:46:24.446 INFO:teuthology.orchestra.run.smithi038.stdout:/tmp/cephfs/postgres/data/pg_dynshmem: 2023-03-09T14:46:24.446 INFO:teuthology.orchestra.run.smithi038.stdout:total 0 2023-03-09T14:46:24.447 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 . 2023-03-09T14:46:24.447 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Mar 9 14:46 .. 2023-03-09T14:46:24.447 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:46:24.447 INFO:teuthology.orchestra.run.smithi038.stdout:/tmp/cephfs/postgres/data/pg_logical: 2023-03-09T14:46:24.447 INFO:teuthology.orchestra.run.smithi038.stdout:total 1 2023-03-09T14:46:24.447 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Mar 9 14:46 . 2023-03-09T14:46:24.447 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Mar 9 14:46 .. 2023-03-09T14:46:24.448 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 mappings 2023-03-09T14:46:24.448 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8 Mar 9 14:46 replorigin_checkpoint 2023-03-09T14:46:24.448 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 snapshots 2023-03-09T14:46:24.448 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:46:24.448 INFO:teuthology.orchestra.run.smithi038.stdout:/tmp/cephfs/postgres/data/pg_logical/mappings: 2023-03-09T14:46:24.448 INFO:teuthology.orchestra.run.smithi038.stdout:total 0 2023-03-09T14:46:24.448 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 . 2023-03-09T14:46:24.448 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Mar 9 14:46 .. 2023-03-09T14:46:24.449 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:46:24.449 INFO:teuthology.orchestra.run.smithi038.stdout:/tmp/cephfs/postgres/data/pg_logical/snapshots: 2023-03-09T14:46:24.449 INFO:teuthology.orchestra.run.smithi038.stdout:total 0 2023-03-09T14:46:24.449 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 . 2023-03-09T14:46:24.449 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 3 Mar 9 14:46 .. 2023-03-09T14:46:24.449 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:46:24.449 INFO:teuthology.orchestra.run.smithi038.stdout:/tmp/cephfs/postgres/data/pg_multixact: 2023-03-09T14:46:24.449 INFO:teuthology.orchestra.run.smithi038.stdout:total 0 2023-03-09T14:46:24.450 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Mar 9 14:46 . 2023-03-09T14:46:24.450 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Mar 9 14:46 .. 2023-03-09T14:46:24.450 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Mar 9 14:46 members 2023-03-09T14:46:24.450 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Mar 9 14:46 offsets 2023-03-09T14:46:24.450 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:46:24.450 INFO:teuthology.orchestra.run.smithi038.stdout:/tmp/cephfs/postgres/data/pg_multixact/members: 2023-03-09T14:46:24.450 INFO:teuthology.orchestra.run.smithi038.stdout:total 8 2023-03-09T14:46:24.451 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Mar 9 14:46 . 2023-03-09T14:46:24.451 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Mar 9 14:46 .. 2023-03-09T14:46:24.451 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 0000 2023-03-09T14:46:24.451 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:46:24.451 INFO:teuthology.orchestra.run.smithi038.stdout:/tmp/cephfs/postgres/data/pg_multixact/offsets: 2023-03-09T14:46:24.451 INFO:teuthology.orchestra.run.smithi038.stdout:total 8 2023-03-09T14:46:24.451 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Mar 9 14:46 . 2023-03-09T14:46:24.451 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 4 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Mar 9 14:46 .. 2023-03-09T14:46:24.452 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 0000 2023-03-09T14:46:24.452 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:46:24.452 INFO:teuthology.orchestra.run.smithi038.stdout:/tmp/cephfs/postgres/data/pg_notify: 2023-03-09T14:46:24.452 INFO:teuthology.orchestra.run.smithi038.stdout:total 8 2023-03-09T14:46:24.452 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Mar 9 14:46 . 2023-03-09T14:46:24.452 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Mar 9 14:46 .. 2023-03-09T14:46:24.452 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 0000 2023-03-09T14:46:24.452 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:46:24.453 INFO:teuthology.orchestra.run.smithi038.stdout:/tmp/cephfs/postgres/data/pg_replslot: 2023-03-09T14:46:24.453 INFO:teuthology.orchestra.run.smithi038.stdout:total 0 2023-03-09T14:46:24.453 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 . 2023-03-09T14:46:24.453 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Mar 9 14:46 .. 2023-03-09T14:46:24.453 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:46:24.453 INFO:teuthology.orchestra.run.smithi038.stdout:/tmp/cephfs/postgres/data/pg_serial: 2023-03-09T14:46:24.453 INFO:teuthology.orchestra.run.smithi038.stdout:total 0 2023-03-09T14:46:24.453 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 . 2023-03-09T14:46:24.454 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Mar 9 14:46 .. 2023-03-09T14:46:24.454 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:46:24.454 INFO:teuthology.orchestra.run.smithi038.stdout:/tmp/cephfs/postgres/data/pg_snapshots: 2023-03-09T14:46:24.454 INFO:teuthology.orchestra.run.smithi038.stdout:total 0 2023-03-09T14:46:24.454 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 . 2023-03-09T14:46:24.454 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Mar 9 14:46 .. 2023-03-09T14:46:24.454 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:46:24.455 INFO:teuthology.orchestra.run.smithi038.stdout:/tmp/cephfs/postgres/data/pg_stat: 2023-03-09T14:46:24.455 INFO:teuthology.orchestra.run.smithi038.stdout:total 0 2023-03-09T14:46:24.455 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 . 2023-03-09T14:46:24.455 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Mar 9 14:46 .. 2023-03-09T14:46:24.455 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:46:24.455 INFO:teuthology.orchestra.run.smithi038.stdout:/tmp/cephfs/postgres/data/pg_stat_tmp: 2023-03-09T14:46:24.455 INFO:teuthology.orchestra.run.smithi038.stdout:total 0 2023-03-09T14:46:24.455 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 . 2023-03-09T14:46:24.456 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Mar 9 14:46 .. 2023-03-09T14:46:24.456 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:46:24.456 INFO:teuthology.orchestra.run.smithi038.stdout:/tmp/cephfs/postgres/data/pg_subtrans: 2023-03-09T14:46:24.456 INFO:teuthology.orchestra.run.smithi038.stdout:total 8 2023-03-09T14:46:24.456 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Mar 9 14:46 . 2023-03-09T14:46:24.456 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Mar 9 14:46 .. 2023-03-09T14:46:24.456 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 0000 2023-03-09T14:46:24.456 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:46:24.457 INFO:teuthology.orchestra.run.smithi038.stdout:/tmp/cephfs/postgres/data/pg_tblspc: 2023-03-09T14:46:24.457 INFO:teuthology.orchestra.run.smithi038.stdout:total 0 2023-03-09T14:46:24.457 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 . 2023-03-09T14:46:24.457 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Mar 9 14:46 .. 2023-03-09T14:46:24.457 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:46:24.457 INFO:teuthology.orchestra.run.smithi038.stdout:/tmp/cephfs/postgres/data/pg_twophase: 2023-03-09T14:46:24.457 INFO:teuthology.orchestra.run.smithi038.stdout:total 0 2023-03-09T14:46:24.457 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 . 2023-03-09T14:46:24.458 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Mar 9 14:46 .. 2023-03-09T14:46:24.458 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:46:24.458 INFO:teuthology.orchestra.run.smithi038.stdout:/tmp/cephfs/postgres/data/pg_wal: 2023-03-09T14:46:24.458 INFO:teuthology.orchestra.run.smithi038.stdout:total 16384 2023-03-09T14:46:24.458 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Mar 9 14:46 . 2023-03-09T14:46:24.458 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Mar 9 14:46 .. 2023-03-09T14:46:24.458 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 16777216 Mar 9 14:46 000000010000000000000001 2023-03-09T14:46:24.458 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 archive_status 2023-03-09T14:46:24.459 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:46:24.459 INFO:teuthology.orchestra.run.smithi038.stdout:/tmp/cephfs/postgres/data/pg_wal/archive_status: 2023-03-09T14:46:24.459 INFO:teuthology.orchestra.run.smithi038.stdout:total 0 2023-03-09T14:46:24.459 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 0 Mar 9 14:46 . 2023-03-09T14:46:24.459 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 3 postgres postgres system_u:object_r:postgresql_db_t:s0 2 Mar 9 14:46 .. 2023-03-09T14:46:24.459 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:46:24.459 INFO:teuthology.orchestra.run.smithi038.stdout:/tmp/cephfs/postgres/data/pg_xact: 2023-03-09T14:46:24.459 INFO:teuthology.orchestra.run.smithi038.stdout:total 8 2023-03-09T14:46:24.460 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 2 postgres postgres system_u:object_r:postgresql_db_t:s0 1 Mar 9 14:46 . 2023-03-09T14:46:24.460 INFO:teuthology.orchestra.run.smithi038.stdout:drwx------. 20 postgres postgres system_u:object_r:postgresql_db_t:s0 23 Mar 9 14:46 .. 2023-03-09T14:46:24.460 INFO:teuthology.orchestra.run.smithi038.stdout:-rw-------. 1 postgres postgres system_u:object_r:postgresql_db_t:s0 8192 Mar 9 14:46 0000 2023-03-09T14:46:24.460 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo systemctl start postgresql' 2023-03-09T14:46:24.591 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- pgbench -s 500 -i' 2023-03-09T14:46:24.638 INFO:teuthology.orchestra.run.smithi038.stderr:NOTICE: table "pgbench_history" does not exist, skipping 2023-03-09T14:46:24.643 INFO:teuthology.orchestra.run.smithi038.stderr:NOTICE: table "pgbench_tellers" does not exist, skipping 2023-03-09T14:46:24.646 INFO:teuthology.orchestra.run.smithi038.stderr:NOTICE: table "pgbench_accounts" does not exist, skipping 2023-03-09T14:46:24.650 INFO:teuthology.orchestra.run.smithi038.stderr:NOTICE: table "pgbench_branches" does not exist, skipping 2023-03-09T14:46:24.861 INFO:teuthology.orchestra.run.smithi038.stderr:creating tables... 2023-03-09T14:46:24.925 INFO:teuthology.orchestra.run.smithi038.stderr:100000 of 50000000 tuples (0%) done (elapsed 0.06 s, remaining 31.21 s) 2023-03-09T14:46:25.091 INFO:teuthology.orchestra.run.smithi038.stderr:200000 of 50000000 tuples (0%) done (elapsed 0.23 s, remaining 56.91 s) 2023-03-09T14:46:25.185 INFO:teuthology.orchestra.run.smithi038.stderr:300000 of 50000000 tuples (0%) done (elapsed 0.32 s, remaining 53.56 s) 2023-03-09T14:46:25.333 INFO:teuthology.orchestra.run.smithi038.stderr:400000 of 50000000 tuples (0%) done (elapsed 0.47 s, remaining 58.41 s) 2023-03-09T14:46:25.504 INFO:teuthology.orchestra.run.smithi038.stderr:500000 of 50000000 tuples (1%) done (elapsed 0.64 s, remaining 63.50 s) 2023-03-09T14:46:25.669 INFO:teuthology.orchestra.run.smithi038.stderr:600000 of 50000000 tuples (1%) done (elapsed 0.81 s, remaining 66.44 s) 2023-03-09T14:46:25.752 INFO:teuthology.orchestra.run.smithi038.stderr:700000 of 50000000 tuples (1%) done (elapsed 0.89 s, remaining 62.66 s) 2023-03-09T14:46:25.931 INFO:teuthology.orchestra.run.smithi038.stderr:800000 of 50000000 tuples (1%) done (elapsed 1.07 s, remaining 65.73 s) 2023-03-09T14:46:25.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:25 smithi122 ceph-mon[93869]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 104 KiB/s wr, 40 op/s 2023-03-09T14:46:25.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:25 smithi181 ceph-mon[94189]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 104 KiB/s wr, 40 op/s 2023-03-09T14:46:26.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:25 smithi038 ceph-mon[92422]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 86 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 104 KiB/s wr, 40 op/s 2023-03-09T14:46:26.095 INFO:teuthology.orchestra.run.smithi038.stderr:900000 of 50000000 tuples (1%) done (elapsed 1.23 s, remaining 67.29 s) 2023-03-09T14:46:26.170 INFO:teuthology.orchestra.run.smithi038.stderr:1000000 of 50000000 tuples (2%) done (elapsed 1.31 s, remaining 64.08 s) 2023-03-09T14:46:26.342 INFO:teuthology.orchestra.run.smithi038.stderr:1100000 of 50000000 tuples (2%) done (elapsed 1.48 s, remaining 65.80 s) 2023-03-09T14:46:26.408 INFO:teuthology.orchestra.run.smithi038.stderr:1200000 of 50000000 tuples (2%) done (elapsed 1.55 s, remaining 62.85 s) 2023-03-09T14:46:26.583 INFO:teuthology.orchestra.run.smithi038.stderr:1300000 of 50000000 tuples (2%) done (elapsed 1.72 s, remaining 64.47 s) 2023-03-09T14:46:26.752 INFO:teuthology.orchestra.run.smithi038.stderr:1400000 of 50000000 tuples (2%) done (elapsed 1.89 s, remaining 65.60 s) 2023-03-09T14:46:26.830 INFO:teuthology.orchestra.run.smithi038.stderr:1500000 of 50000000 tuples (3%) done (elapsed 1.97 s, remaining 63.63 s) 2023-03-09T14:46:27.017 INFO:teuthology.orchestra.run.smithi038.stderr:1600000 of 50000000 tuples (3%) done (elapsed 2.15 s, remaining 65.17 s) 2023-03-09T14:46:27.180 INFO:teuthology.orchestra.run.smithi038.stderr:1700000 of 50000000 tuples (3%) done (elapsed 2.32 s, remaining 65.85 s) 2023-03-09T14:46:27.255 INFO:teuthology.orchestra.run.smithi038.stderr:1800000 of 50000000 tuples (3%) done (elapsed 2.39 s, remaining 64.07 s) 2023-03-09T14:46:27.430 INFO:teuthology.orchestra.run.smithi038.stderr:1900000 of 50000000 tuples (3%) done (elapsed 2.57 s, remaining 65.00 s) 2023-03-09T14:46:27.594 INFO:teuthology.orchestra.run.smithi038.stderr:2000000 of 50000000 tuples (4%) done (elapsed 2.73 s, remaining 65.57 s) 2023-03-09T14:46:27.684 INFO:teuthology.orchestra.run.smithi038.stderr:2100000 of 50000000 tuples (4%) done (elapsed 2.82 s, remaining 64.36 s) 2023-03-09T14:46:27.845 INFO:teuthology.orchestra.run.smithi038.stderr:2200000 of 50000000 tuples (4%) done (elapsed 2.98 s, remaining 64.81 s) 2023-03-09T14:46:27.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:27 smithi122 ceph-mon[93869]: pgmap v9: 129 pgs: 129 active+clean; 91 MiB data, 221 MiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 13 MiB/s wr, 155 op/s 2023-03-09T14:46:27.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:27 smithi181 ceph-mon[94189]: pgmap v9: 129 pgs: 129 active+clean; 91 MiB data, 221 MiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 13 MiB/s wr, 155 op/s 2023-03-09T14:46:27.997 INFO:teuthology.orchestra.run.smithi038.stderr:2300000 of 50000000 tuples (4%) done (elapsed 3.13 s, remaining 65.01 s) 2023-03-09T14:46:28.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:27 smithi038 ceph-mon[92422]: pgmap v9: 129 pgs: 129 active+clean; 91 MiB data, 221 MiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 13 MiB/s wr, 155 op/s 2023-03-09T14:46:28.081 INFO:teuthology.orchestra.run.smithi038.stderr:2400000 of 50000000 tuples (4%) done (elapsed 3.22 s, remaining 63.84 s) 2023-03-09T14:46:28.228 INFO:teuthology.orchestra.run.smithi038.stderr:2500000 of 50000000 tuples (5%) done (elapsed 3.37 s, remaining 63.95 s) 2023-03-09T14:46:28.318 INFO:teuthology.orchestra.run.smithi038.stderr:2600000 of 50000000 tuples (5%) done (elapsed 3.46 s, remaining 63.01 s) 2023-03-09T14:46:28.464 INFO:teuthology.orchestra.run.smithi038.stderr:2700000 of 50000000 tuples (5%) done (elapsed 3.60 s, remaining 63.10 s) 2023-03-09T14:46:28.636 INFO:teuthology.orchestra.run.smithi038.stderr:2800000 of 50000000 tuples (5%) done (elapsed 3.77 s, remaining 63.61 s) 2023-03-09T14:46:28.727 INFO:teuthology.orchestra.run.smithi038.stderr:2900000 of 50000000 tuples (5%) done (elapsed 3.87 s, remaining 62.77 s) 2023-03-09T14:46:28.867 INFO:teuthology.orchestra.run.smithi038.stderr:3000000 of 50000000 tuples (6%) done (elapsed 4.01 s, remaining 62.75 s) 2023-03-09T14:46:28.961 INFO:teuthology.orchestra.run.smithi038.stderr:3100000 of 50000000 tuples (6%) done (elapsed 4.10 s, remaining 62.01 s) 2023-03-09T14:46:29.126 INFO:teuthology.orchestra.run.smithi038.stderr:3200000 of 50000000 tuples (6%) done (elapsed 4.26 s, remaining 62.36 s) 2023-03-09T14:46:29.268 INFO:teuthology.orchestra.run.smithi038.stderr:3300000 of 50000000 tuples (6%) done (elapsed 4.41 s, remaining 62.35 s) 2023-03-09T14:46:29.373 INFO:teuthology.orchestra.run.smithi038.stderr:3400000 of 50000000 tuples (6%) done (elapsed 4.51 s, remaining 61.82 s) 2023-03-09T14:46:29.511 INFO:teuthology.orchestra.run.smithi038.stderr:3500000 of 50000000 tuples (7%) done (elapsed 4.65 s, remaining 61.77 s) 2023-03-09T14:46:29.679 INFO:teuthology.orchestra.run.smithi038.stderr:3600000 of 50000000 tuples (7%) done (elapsed 4.82 s, remaining 62.09 s) 2023-03-09T14:46:29.787 INFO:teuthology.orchestra.run.smithi038.stderr:3700000 of 50000000 tuples (7%) done (elapsed 4.92 s, remaining 61.63 s) 2023-03-09T14:46:29.931 INFO:teuthology.orchestra.run.smithi038.stderr:3800000 of 50000000 tuples (7%) done (elapsed 5.07 s, remaining 61.63 s) 2023-03-09T14:46:29.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:29 smithi122 ceph-mon[93869]: pgmap v10: 129 pgs: 129 active+clean; 91 MiB data, 221 MiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 12 MiB/s wr, 141 op/s 2023-03-09T14:46:29.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:29 smithi181 ceph-mon[94189]: pgmap v10: 129 pgs: 129 active+clean; 91 MiB data, 221 MiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 12 MiB/s wr, 141 op/s 2023-03-09T14:46:30.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:29 smithi038 ceph-mon[92422]: pgmap v10: 129 pgs: 129 active+clean; 91 MiB data, 221 MiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 12 MiB/s wr, 141 op/s 2023-03-09T14:46:30.104 INFO:teuthology.orchestra.run.smithi038.stderr:3900000 of 50000000 tuples (7%) done (elapsed 5.24 s, remaining 61.96 s) 2023-03-09T14:46:30.200 INFO:teuthology.orchestra.run.smithi038.stderr:4000000 of 50000000 tuples (8%) done (elapsed 5.34 s, remaining 61.38 s) 2023-03-09T14:46:30.342 INFO:teuthology.orchestra.run.smithi038.stderr:4100000 of 50000000 tuples (8%) done (elapsed 5.48 s, remaining 61.35 s) 2023-03-09T14:46:30.430 INFO:teuthology.orchestra.run.smithi038.stderr:4200000 of 50000000 tuples (8%) done (elapsed 5.57 s, remaining 60.72 s) 2023-03-09T14:46:30.603 INFO:teuthology.orchestra.run.smithi038.stderr:4300000 of 50000000 tuples (8%) done (elapsed 5.74 s, remaining 61.01 s) 2023-03-09T14:46:30.736 INFO:teuthology.orchestra.run.smithi038.stderr:4400000 of 50000000 tuples (8%) done (elapsed 5.87 s, remaining 60.88 s) 2023-03-09T14:46:30.834 INFO:teuthology.orchestra.run.smithi038.stderr:4500000 of 50000000 tuples (9%) done (elapsed 5.97 s, remaining 60.38 s) 2023-03-09T14:46:30.969 INFO:teuthology.orchestra.run.smithi038.stderr:4600000 of 50000000 tuples (9%) done (elapsed 6.11 s, remaining 60.27 s) 2023-03-09T14:46:31.132 INFO:teuthology.orchestra.run.smithi038.stderr:4700000 of 50000000 tuples (9%) done (elapsed 6.27 s, remaining 60.43 s) 2023-03-09T14:46:31.223 INFO:teuthology.orchestra.run.smithi038.stderr:4800000 of 50000000 tuples (9%) done (elapsed 6.36 s, remaining 59.90 s) 2023-03-09T14:46:31.362 INFO:teuthology.orchestra.run.smithi038.stderr:4900000 of 50000000 tuples (9%) done (elapsed 6.50 s, remaining 59.82 s) 2023-03-09T14:46:31.518 INFO:teuthology.orchestra.run.smithi038.stderr:5000000 of 50000000 tuples (10%) done (elapsed 6.66 s, remaining 59.90 s) 2023-03-09T14:46:31.610 INFO:teuthology.orchestra.run.smithi038.stderr:5100000 of 50000000 tuples (10%) done (elapsed 6.75 s, remaining 59.41 s) 2023-03-09T14:46:31.763 INFO:teuthology.orchestra.run.smithi038.stderr:5200000 of 50000000 tuples (10%) done (elapsed 6.90 s, remaining 59.46 s) 2023-03-09T14:46:31.912 INFO:teuthology.orchestra.run.smithi038.stderr:5300000 of 50000000 tuples (10%) done (elapsed 7.05 s, remaining 59.46 s) 2023-03-09T14:46:31.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:31 smithi122 ceph-mon[93869]: pgmap v11: 129 pgs: 129 active+clean; 91 MiB data, 221 MiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 12 MiB/s wr, 141 op/s 2023-03-09T14:46:31.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:31 smithi181 ceph-mon[94189]: pgmap v11: 129 pgs: 129 active+clean; 91 MiB data, 221 MiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 12 MiB/s wr, 141 op/s 2023-03-09T14:46:31.993 INFO:teuthology.orchestra.run.smithi038.stderr:5400000 of 50000000 tuples (10%) done (elapsed 7.13 s, remaining 58.90 s) 2023-03-09T14:46:32.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:31 smithi038 ceph-mon[92422]: pgmap v11: 129 pgs: 129 active+clean; 91 MiB data, 221 MiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 12 MiB/s wr, 141 op/s 2023-03-09T14:46:32.168 INFO:teuthology.orchestra.run.smithi038.stderr:5500000 of 50000000 tuples (11%) done (elapsed 7.31 s, remaining 59.11 s) 2023-03-09T14:46:32.328 INFO:teuthology.orchestra.run.smithi038.stderr:5600000 of 50000000 tuples (11%) done (elapsed 7.47 s, remaining 59.19 s) 2023-03-09T14:46:32.413 INFO:teuthology.orchestra.run.smithi038.stderr:5700000 of 50000000 tuples (11%) done (elapsed 7.55 s, remaining 58.68 s) 2023-03-09T14:46:32.592 INFO:teuthology.orchestra.run.smithi038.stderr:5800000 of 50000000 tuples (11%) done (elapsed 7.73 s, remaining 58.91 s) 2023-03-09T14:46:33.252 INFO:teuthology.orchestra.run.smithi038.stderr:5900000 of 50000000 tuples (11%) done (elapsed 8.39 s, remaining 62.71 s) 2023-03-09T14:46:33.533 INFO:teuthology.orchestra.run.smithi038.stderr:6000000 of 50000000 tuples (12%) done (elapsed 8.67 s, remaining 63.59 s) 2023-03-09T14:46:33.669 INFO:teuthology.orchestra.run.smithi038.stderr:6100000 of 50000000 tuples (12%) done (elapsed 8.81 s, remaining 63.38 s) 2023-03-09T14:46:33.763 INFO:teuthology.orchestra.run.smithi038.stderr:6200000 of 50000000 tuples (12%) done (elapsed 8.90 s, remaining 62.88 s) 2023-03-09T14:46:33.902 INFO:teuthology.orchestra.run.smithi038.stderr:6300000 of 50000000 tuples (12%) done (elapsed 9.04 s, remaining 62.71 s) 2023-03-09T14:46:33.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:33 smithi122 ceph-mon[93869]: pgmap v12: 129 pgs: 129 active+clean; 475 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 75 MiB/s wr, 182 op/s 2023-03-09T14:46:33.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:33 smithi181 ceph-mon[94189]: pgmap v12: 129 pgs: 129 active+clean; 475 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 75 MiB/s wr, 182 op/s 2023-03-09T14:46:34.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:33 smithi038 ceph-mon[92422]: pgmap v12: 129 pgs: 129 active+clean; 475 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 75 MiB/s wr, 182 op/s 2023-03-09T14:46:34.068 INFO:teuthology.orchestra.run.smithi038.stderr:6400000 of 50000000 tuples (12%) done (elapsed 9.21 s, remaining 62.72 s) 2023-03-09T14:46:34.158 INFO:teuthology.orchestra.run.smithi038.stderr:6500000 of 50000000 tuples (13%) done (elapsed 9.30 s, remaining 62.21 s) 2023-03-09T14:46:34.299 INFO:teuthology.orchestra.run.smithi038.stderr:6600000 of 50000000 tuples (13%) done (elapsed 9.44 s, remaining 62.05 s) 2023-03-09T14:46:34.390 INFO:teuthology.orchestra.run.smithi038.stderr:6700000 of 50000000 tuples (13%) done (elapsed 9.53 s, remaining 61.57 s) 2023-03-09T14:46:34.554 INFO:teuthology.orchestra.run.smithi038.stderr:6800000 of 50000000 tuples (13%) done (elapsed 9.69 s, remaining 61.57 s) 2023-03-09T14:46:34.702 INFO:teuthology.orchestra.run.smithi038.stderr:6900000 of 50000000 tuples (13%) done (elapsed 9.84 s, remaining 61.47 s) 2023-03-09T14:46:34.804 INFO:teuthology.orchestra.run.smithi038.stderr:7000000 of 50000000 tuples (14%) done (elapsed 9.94 s, remaining 61.07 s) 2023-03-09T14:46:34.999 INFO:teuthology.orchestra.run.smithi038.stderr:7100000 of 50000000 tuples (14%) done (elapsed 10.14 s, remaining 61.25 s) 2023-03-09T14:46:35.143 INFO:teuthology.orchestra.run.smithi038.stderr:7200000 of 50000000 tuples (14%) done (elapsed 10.28 s, remaining 61.11 s) 2023-03-09T14:46:35.231 INFO:teuthology.orchestra.run.smithi038.stderr:7300000 of 50000000 tuples (14%) done (elapsed 10.37 s, remaining 60.65 s) 2023-03-09T14:46:35.368 INFO:teuthology.orchestra.run.smithi038.stderr:7400000 of 50000000 tuples (14%) done (elapsed 10.51 s, remaining 60.48 s) 2023-03-09T14:46:35.533 INFO:teuthology.orchestra.run.smithi038.stderr:7500000 of 50000000 tuples (15%) done (elapsed 10.67 s, remaining 60.47 s) 2023-03-09T14:46:35.629 INFO:teuthology.orchestra.run.smithi038.stderr:7600000 of 50000000 tuples (15%) done (elapsed 10.77 s, remaining 60.07 s) 2023-03-09T14:46:35.774 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-03-09T14:46:35.860 INFO:teuthology.orchestra.run.smithi038.stderr:7700000 of 50000000 tuples (15%) done (elapsed 11.00 s, remaining 60.42 s) 2023-03-09T14:46:35.925 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.922+0000 7fd19692b700 1 -- 172.21.15.38:0/2488742384 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd190104030 msgr2=0x7fd19010bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:35.926 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.922+0000 7fd19692b700 1 --2- 172.21.15.38:0/2488742384 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd190104030 0x7fd19010bc00 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fd18800b0d0 tx=0x7fd18800b3e0 comp rx=0 tx=0).stop 2023-03-09T14:46:35.926 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.922+0000 7fd19692b700 1 -- 172.21.15.38:0/2488742384 shutdown_connections 2023-03-09T14:46:35.926 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.922+0000 7fd19692b700 1 --2- 172.21.15.38:0/2488742384 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd190104030 0x7fd19010bc00 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:35.926 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.922+0000 7fd19692b700 1 --2- 172.21.15.38:0/2488742384 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd1900feff0 0x7fd1900ff3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:35.926 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.922+0000 7fd19692b700 1 --2- 172.21.15.38:0/2488742384 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd1900ff990 0x7fd190103950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:35.926 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.922+0000 7fd19692b700 1 -- 172.21.15.38:0/2488742384 >> 172.21.15.38:0/2488742384 conn(0x7fd1900fa840 msgr2=0x7fd1900fcc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:35.926 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.923+0000 7fd19692b700 1 -- 172.21.15.38:0/2488742384 shutdown_connections 2023-03-09T14:46:35.927 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.923+0000 7fd19692b700 1 -- 172.21.15.38:0/2488742384 wait complete. 2023-03-09T14:46:35.928 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.924+0000 7fd19692b700 1 Processor -- start 2023-03-09T14:46:35.928 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.924+0000 7fd19692b700 1 -- start start 2023-03-09T14:46:35.928 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.924+0000 7fd19692b700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd1900feff0 0x7fd19010c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:35.929 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.924+0000 7fd19692b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd1900ff990 0x7fd19010c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:35.929 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.924+0000 7fd19692b700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd190104030 0x7fd1901a4390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:35.929 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.924+0000 7fd19692b700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd19010d0c0 con 0x7fd1900ff990 2023-03-09T14:46:35.929 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.924+0000 7fd19692b700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1901a48d0 con 0x7fd190104030 2023-03-09T14:46:35.929 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.924+0000 7fd19692b700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1901a4a10 con 0x7fd1900feff0 2023-03-09T14:46:35.929 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.924+0000 7fd18f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd1900ff990 0x7fd19010c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:35.929 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.924+0000 7fd18f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd1900ff990 0x7fd19010c890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39240/0 (socket says 172.21.15.38:39240) 2023-03-09T14:46:35.930 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.924+0000 7fd18f7fe700 1 -- 172.21.15.38:0/3507952880 learned_addr learned my addr 172.21.15.38:0/3507952880 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:46:35.930 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.924+0000 7fd194ec8700 1 --2- 172.21.15.38:0/3507952880 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd190104030 0x7fd1901a4390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:35.930 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.924+0000 7fd18ffff700 1 --2- 172.21.15.38:0/3507952880 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd1900feff0 0x7fd19010c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:35.930 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.925+0000 7fd18f7fe700 1 -- 172.21.15.38:0/3507952880 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd1900feff0 msgr2=0x7fd19010c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:35.930 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.925+0000 7fd18f7fe700 1 --2- 172.21.15.38:0/3507952880 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd1900feff0 0x7fd19010c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:35.930 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.925+0000 7fd18f7fe700 1 -- 172.21.15.38:0/3507952880 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd190104030 msgr2=0x7fd1901a4390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:35.930 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.925+0000 7fd18f7fe700 1 --2- 172.21.15.38:0/3507952880 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd190104030 0x7fd1901a4390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:35.931 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.925+0000 7fd18f7fe700 1 -- 172.21.15.38:0/3507952880 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd188009d00 con 0x7fd1900ff990 2023-03-09T14:46:35.931 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.925+0000 7fd18f7fe700 1 --2- 172.21.15.38:0/3507952880 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd1900ff990 0x7fd19010c890 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fd18400d950 tx=0x7fd18400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:35.931 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.925+0000 7fd18d7fa700 1 -- 172.21.15.38:0/3507952880 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd18400f890 con 0x7fd1900ff990 2023-03-09T14:46:35.931 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.925+0000 7fd19692b700 1 -- 172.21.15.38:0/3507952880 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd19010ffc0 con 0x7fd1900ff990 2023-03-09T14:46:35.931 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.925+0000 7fd18d7fa700 1 -- 172.21.15.38:0/3507952880 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd184010460 con 0x7fd1900ff990 2023-03-09T14:46:35.931 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.925+0000 7fd18d7fa700 1 -- 172.21.15.38:0/3507952880 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd18400e610 con 0x7fd1900ff990 2023-03-09T14:46:35.931 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.925+0000 7fd19692b700 1 -- 172.21.15.38:0/3507952880 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd19010fe30 con 0x7fd1900ff990 2023-03-09T14:46:35.933 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.927+0000 7fd18d7fa700 1 -- 172.21.15.38:0/3507952880 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 23) v1 ==== 133335+0+0 (secure 0 0 0) 0x7fd184023070 con 0x7fd1900ff990 2023-03-09T14:46:35.933 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.928+0000 7fd18d7fa700 1 --2- 172.21.15.38:0/3507952880 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fd17809f4e0 0x7fd1780a1990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:35.934 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.928+0000 7fd18d7fa700 1 -- 172.21.15.38:0/3507952880 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9278+0+0 (secure 0 0 0) 0x7fd1840c9d20 con 0x7fd1900ff990 2023-03-09T14:46:35.934 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.929+0000 7fd1a2ab4b80 1 Processor -- start 2023-03-09T14:46:35.934 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.929+0000 7fd1a2ab4b80 1 -- start start 2023-03-09T14:46:35.934 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.929+0000 7fd18ffff700 1 --2- 172.21.15.38:0/3507952880 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fd17809f4e0 0x7fd1780a1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:35.934 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.929+0000 7fd1a2ab4b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55baeb44f830 0x55baeb44fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:35.934 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.929+0000 7fd1a2ab4b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55baeb450140 0x55baeb449aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:35.934 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.929+0000 7fd1a2ab4b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55baeb449fe0 0x55baeb44a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:35.935 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.929+0000 7fd1a2ab4b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55baeb333da0 con 0x55baeb44f830 2023-03-09T14:46:35.935 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.929+0000 7fd1a2ab4b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55baeb3a2780 con 0x55baeb450140 2023-03-09T14:46:35.935 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.929+0000 7fd1a2ab4b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55baeb39bcb0 con 0x55baeb449fe0 2023-03-09T14:46:35.935 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.930+0000 7fd18ffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55baeb44f830 0x55baeb44fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:35.935 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.930+0000 7fd18ffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55baeb44f830 0x55baeb44fc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39244/0 (socket says 172.21.15.38:39244) 2023-03-09T14:46:35.935 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.930+0000 7fd18f7fe700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55baeb450140 0x55baeb449aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:35.935 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.930+0000 7fd18ffff700 1 -- 172.21.15.38:0/1428331800 learned_addr learned my addr 172.21.15.38:0/1428331800 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:46:35.935 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.930+0000 7fd194ec8700 1 --2- 172.21.15.38:0/1428331800 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55baeb449fe0 0x55baeb44a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:35.936 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.930+0000 7fd18ffff700 1 --2- 172.21.15.38:0/3507952880 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fd17809f4e0 0x7fd1780a1990 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd180009a60 tx=0x7fd180009e20 comp rx=0 tx=0).ready entity=mgr.15558 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:35.936 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.930+0000 7fd18ffff700 1 -- 172.21.15.38:0/1428331800 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55baeb449fe0 msgr2=0x55baeb44a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:46:35.936 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.930+0000 7fd18ffff700 1 --2- 172.21.15.38:0/1428331800 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55baeb449fe0 0x55baeb44a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:35.936 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.930+0000 7fd18ffff700 1 -- 172.21.15.38:0/1428331800 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55baeb450140 msgr2=0x55baeb449aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:46:35.936 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.930+0000 7fd18ffff700 1 --2- 172.21.15.38:0/1428331800 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55baeb450140 0x55baeb449aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:35.936 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.930+0000 7fd18ffff700 1 -- 172.21.15.38:0/1428331800 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55baeb362190 con 0x55baeb44f830 2023-03-09T14:46:35.936 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.930+0000 7fd18f7fe700 1 --2- 172.21.15.38:0/1428331800 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55baeb450140 0x55baeb449aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:46:35.937 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.930+0000 7fd18ffff700 1 --2- 172.21.15.38:0/1428331800 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55baeb44f830 0x55baeb44fc00 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fd18000eb80 tx=0x7fd180019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f06a893f830af399 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:35.937 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.930+0000 7fd1767fc700 1 -- 172.21.15.38:0/1428331800 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd18000ee90 con 0x55baeb44f830 2023-03-09T14:46:35.937 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.930+0000 7fd1767fc700 1 -- 172.21.15.38:0/1428331800 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd18001e410 con 0x55baeb44f830 2023-03-09T14:46:35.937 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.930+0000 7fd1767fc700 1 -- 172.21.15.38:0/1428331800 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd180016660 con 0x55baeb44f830 2023-03-09T14:46:35.937 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.931+0000 7fd1a2ab4b80 1 -- 172.21.15.38:0/1428331800 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55baeb44f830 msgr2=0x55baeb44fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:35.937 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.931+0000 7fd1a2ab4b80 1 --2- 172.21.15.38:0/1428331800 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55baeb44f830 0x55baeb44fc00 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fd18000eb80 tx=0x7fd180019000 comp rx=0 tx=0).stop 2023-03-09T14:46:35.937 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.931+0000 7fd1a2ab4b80 1 -- 172.21.15.38:0/1428331800 shutdown_connections 2023-03-09T14:46:35.937 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.931+0000 7fd1a2ab4b80 1 --2- 172.21.15.38:0/1428331800 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55baeb44f830 0x55baeb44fc00 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:35.938 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.931+0000 7fd1a2ab4b80 1 --2- 172.21.15.38:0/1428331800 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55baeb449fe0 0x55baeb44a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:35.938 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.931+0000 7fd1a2ab4b80 1 --2- 172.21.15.38:0/1428331800 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55baeb450140 0x55baeb449aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:35.938 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.931+0000 7fd1a2ab4b80 1 -- 172.21.15.38:0/1428331800 >> 172.21.15.38:0/1428331800 conn(0x55baeb444a20 msgr2=0x55baeb444df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:35.938 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.931+0000 7fd1a2ab4b80 1 -- 172.21.15.38:0/1428331800 shutdown_connections 2023-03-09T14:46:35.938 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.931+0000 7fd1a2ab4b80 1 -- 172.21.15.38:0/1428331800 wait complete. 2023-03-09T14:46:35.938 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.931+0000 7fd1a2ab4b80 1 -- start start 2023-03-09T14:46:35.941 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.932+0000 7fd1a2ab4b80 1 Processor -- start 2023-03-09T14:46:35.941 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.932+0000 7fd1a2ab4b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55baeb449fe0 0x55baeb4560e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:35.941 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.932+0000 7fd1a2ab4b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55baeb44f830 0x55baeb4533d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:35.941 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.932+0000 7fd1a2ab4b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55baeb450140 0x55baeb453a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:35.942 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.932+0000 7fd1a2ab4b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55baeb21f920 con 0x55baeb450140 2023-03-09T14:46:35.942 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.932+0000 7fd1a2ab4b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55baeb454190 con 0x55baeb44f830 2023-03-09T14:46:35.942 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.932+0000 7fd1a2ab4b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55baeb45fc40 con 0x55baeb449fe0 2023-03-09T14:46:35.942 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.932+0000 7fd1a2ab4b80 10 client.0 _ll_register_callbacks cb 0x55baeb341520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:46:35.942 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.932+0000 7fd18ffff700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55baeb449fe0 0x55baeb4560e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:35.942 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.932+0000 7fd18ffff700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55baeb449fe0 0x55baeb4560e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.38:46948/0 (socket says 172.21.15.38:46948) 2023-03-09T14:46:35.942 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.932+0000 7fd18ffff700 1 -- 172.21.15.38:0/2924737974 learned_addr learned my addr 172.21.15.38:0/2924737974 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:46:35.943 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.932+0000 7fd194ec8700 1 --2- 172.21.15.38:0/2924737974 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55baeb450140 0x55baeb453a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:35.943 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.932+0000 7fd18f7fe700 1 --2- 172.21.15.38:0/2924737974 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55baeb44f830 0x55baeb4533d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:35.943 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.932+0000 7fd18ffff700 1 -- 172.21.15.38:0/2924737974 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55baeb44f830 msgr2=0x55baeb4533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:46:35.943 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.932+0000 7fd18ffff700 1 --2- 172.21.15.38:0/2924737974 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55baeb44f830 0x55baeb4533d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:35.943 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.932+0000 7fd18ffff700 1 -- 172.21.15.38:0/2924737974 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55baeb450140 msgr2=0x55baeb453a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:46:35.943 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.932+0000 7fd18ffff700 1 --2- 172.21.15.38:0/2924737974 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55baeb450140 0x55baeb453a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:35.943 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.932+0000 7fd18ffff700 1 -- 172.21.15.38:0/2924737974 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd19010fe30 con 0x55baeb449fe0 2023-03-09T14:46:35.943 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.932+0000 7fd18f7fe700 1 --2- 172.21.15.38:0/2924737974 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55baeb44f830 0x55baeb4533d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:46:35.944 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.933+0000 7fd18ffff700 1 --2- 172.21.15.38:0/2924737974 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55baeb449fe0 0x55baeb4560e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd1800097c0 tx=0x7fd180009870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=44622dd881d856d3 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:35.944 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.933+0000 7fd16f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2023-03-09T14:46:35.944 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.933+0000 7fd16f7fe700 1 -- 172.21.15.38:0/2924737974 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd18001e820 con 0x55baeb449fe0 2023-03-09T14:46:35.944 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.933+0000 7fd16f7fe700 1 -- 172.21.15.38:0/2924737974 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9278+0+0 (secure 0 0 0) 0x7fd180027450 con 0x55baeb449fe0 2023-03-09T14:46:35.944 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.933+0000 7fd1a2ab4b80 1 -- 172.21.15.38:0/2924737974 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55baeb369190 con 0x55baeb449fe0 2023-03-09T14:46:35.944 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.933+0000 7fd16f7fe700 1 -- 172.21.15.38:0/2924737974 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd18001eb30 con 0x55baeb449fe0 2023-03-09T14:46:35.944 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.933+0000 7fd1a2ab4b80 10 client.25276 fetch_fsmap learned FSMap version 23 2023-03-09T14:46:35.944 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.933+0000 7fd1a2ab4b80 1 -- 172.21.15.38:0/2924737974 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x7fd18000e830 con 0x55baeb449fe0 2023-03-09T14:46:35.945 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.933+0000 7fd16f7fe700 1 -- 172.21.15.38:0/2924737974 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55baeb369190 con 0x55baeb449fe0 2023-03-09T14:46:35.945 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.933+0000 7fd16f7fe700 1 -- 172.21.15.38:0/2924737974 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fd180031540 con 0x55baeb449fe0 2023-03-09T14:46:35.945 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.933+0000 7fd1a2ab4b80 10 client.25276 fetch_fsmap finished waiting for FSMap version 23 2023-03-09T14:46:35.945 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.933+0000 7fd1a2ab4b80 10 client.25276 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T14:46:35.945 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.933+0000 7fd1a2ab4b80 20 client.25276 populate_metadata read hostname 'smithi038' 2023-03-09T14:46:35.945 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.933+0000 7fd1a2ab4b80 1 --2- 172.21.15.38:0/2924737974 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55baeb463b90 0x55baeb465f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:35.945 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.933+0000 7fd1a2ab4b80 4 client.25276 mds_command: new command op to 24530 tid=0[{"prefix": "get_command_descriptions"}] 2023-03-09T14:46:35.946 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.933+0000 7fd1a2ab4b80 1 -- 172.21.15.38:0/2924737974 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55baeb21ff60 con 0x55baeb463b90 2023-03-09T14:46:35.946 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.934+0000 7fd18f7fe700 1 --2- 172.21.15.38:0/2924737974 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55baeb463b90 0x55baeb465f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:35.946 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.934+0000 7fd16f7fe700 10 client.25276 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:46:35.946 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.934+0000 7fd18f7fe700 1 --2- 172.21.15.38:0/2924737974 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55baeb463b90 0x55baeb465f70 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f6860ba7daba2592 server_cookie=c40dfe54d5341b84 in_seq=0 out_seq=0 2023-03-09T14:46:35.946 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.935+0000 7fd16f7fe700 1 -- 172.21.15.38:0/2924737974 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18278 (crc 0 0 0) 0x55baeb21ff60 con 0x55baeb463b90 2023-03-09T14:46:35.946 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.935+0000 7fd16f7fe700 10 client.25276 handle_command_reply: tid=0 2023-03-09T14:46:35.946 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.935+0000 7fd1a2ab4b80 1 client.25276 shutdown 2023-03-09T14:46:35.946 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.935+0000 7fd1a2ab4b80 1 -- 172.21.15.38:0/2924737974 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55baeb449fe0 msgr2=0x55baeb4560e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:35.947 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.935+0000 7fd1a2ab4b80 1 --2- 172.21.15.38:0/2924737974 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55baeb449fe0 0x55baeb4560e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd1800097c0 tx=0x7fd180009870 comp rx=0 tx=0).stop 2023-03-09T14:46:35.947 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.935+0000 7fd1a2ab4b80 1 -- 172.21.15.38:0/2924737974 shutdown_connections 2023-03-09T14:46:35.947 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.935+0000 7fd1a2ab4b80 1 --2- 172.21.15.38:0/2924737974 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55baeb463b90 0x55baeb465f70 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:35.947 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.935+0000 7fd1a2ab4b80 1 --2- 172.21.15.38:0/2924737974 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55baeb450140 0x55baeb453a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:35.947 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.935+0000 7fd16f7fe700 0 client.25276 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:46:35.947 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.935+0000 7fd1a2ab4b80 1 --2- 172.21.15.38:0/2924737974 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55baeb449fe0 0x55baeb4560e0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:35.947 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.935+0000 7fd1a2ab4b80 1 --2- 172.21.15.38:0/2924737974 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55baeb44f830 0x55baeb4533d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:35.947 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.935+0000 7fd1a2ab4b80 1 -- 172.21.15.38:0/2924737974 >> 172.21.15.38:0/2924737974 conn(0x55baeb444a20 msgr2=0x55baeb443c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:35.948 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.935+0000 7fd1a2ab4b80 1 -- 172.21.15.38:0/2924737974 shutdown_connections 2023-03-09T14:46:35.948 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.935+0000 7fd1a2ab4b80 1 -- 172.21.15.38:0/2924737974 wait complete. 2023-03-09T14:46:35.948 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.935+0000 7fd1a2ab4b80 20 client.25276 trim_cache size 0 max 16384 2023-03-09T14:46:35.955 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.952+0000 7fd1a2ab4b80 1 Processor -- start 2023-03-09T14:46:35.955 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.952+0000 7fd1a2ab4b80 1 -- start start 2023-03-09T14:46:35.955 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.952+0000 7fd1a2ab4b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55baeb449fe0 0x55baeb455c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:35.956 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.952+0000 7fd1a2ab4b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55baeb44f830 0x55baeb443c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:35.956 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.952+0000 7fd1a2ab4b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55baeb450140 0x55baeb44d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:35.956 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.952+0000 7fd1a2ab4b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55baeb44acc0 con 0x55baeb449fe0 2023-03-09T14:46:35.956 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.952+0000 7fd1a2ab4b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55baeb454940 con 0x55baeb450140 2023-03-09T14:46:35.956 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.952+0000 7fd1a2ab4b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55baeb451970 con 0x55baeb44f830 2023-03-09T14:46:35.956 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.953+0000 7fd18ffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55baeb449fe0 0x55baeb455c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:35.956 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.953+0000 7fd18ffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55baeb449fe0 0x55baeb455c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39258/0 (socket says 172.21.15.38:39258) 2023-03-09T14:46:35.957 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.953+0000 7fd18ffff700 1 -- 172.21.15.38:0/3584382299 learned_addr learned my addr 172.21.15.38:0/3584382299 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:46:35.957 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.953+0000 7fd194ec8700 1 --2- 172.21.15.38:0/3584382299 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55baeb450140 0x55baeb44d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:35.957 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.953+0000 7fd18f7fe700 1 --2- 172.21.15.38:0/3584382299 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55baeb44f830 0x55baeb443c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:35.957 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.953+0000 7fd18ffff700 1 -- 172.21.15.38:0/3584382299 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55baeb44f830 msgr2=0x55baeb443c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:46:35.957 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.953+0000 7fd18ffff700 1 --2- 172.21.15.38:0/3584382299 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55baeb44f830 0x55baeb443c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:35.957 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.953+0000 7fd18ffff700 1 -- 172.21.15.38:0/3584382299 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55baeb450140 msgr2=0x55baeb44d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:46:35.957 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.953+0000 7fd18ffff700 1 --2- 172.21.15.38:0/3584382299 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55baeb450140 0x55baeb44d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:35.958 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.953+0000 7fd18ffff700 1 -- 172.21.15.38:0/3584382299 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55baeb1b2c60 con 0x55baeb449fe0 2023-03-09T14:46:35.958 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.953+0000 7fd18ffff700 1 --2- 172.21.15.38:0/3584382299 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55baeb449fe0 0x55baeb455c80 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fd180004790 tx=0x7fd180009230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fd80733a023ce7a server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:35.958 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.953+0000 7fd176ffd700 1 -- 172.21.15.38:0/3584382299 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd180031640 con 0x55baeb449fe0 2023-03-09T14:46:35.958 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.953+0000 7fd176ffd700 1 -- 172.21.15.38:0/3584382299 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd18000ee90 con 0x55baeb449fe0 2023-03-09T14:46:35.958 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.953+0000 7fd176ffd700 1 -- 172.21.15.38:0/3584382299 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1800276b0 con 0x55baeb449fe0 2023-03-09T14:46:35.961 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.954+0000 7fd1a2ab4b80 1 -- 172.21.15.38:0/3584382299 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55baeb449fe0 msgr2=0x55baeb455c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:35.961 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.954+0000 7fd1a2ab4b80 1 --2- 172.21.15.38:0/3584382299 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55baeb449fe0 0x55baeb455c80 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fd180004790 tx=0x7fd180009230 comp rx=0 tx=0).stop 2023-03-09T14:46:35.961 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.954+0000 7fd1a2ab4b80 1 -- 172.21.15.38:0/3584382299 shutdown_connections 2023-03-09T14:46:35.962 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.954+0000 7fd1a2ab4b80 1 --2- 172.21.15.38:0/3584382299 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55baeb449fe0 0x55baeb455c80 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:35.962 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.954+0000 7fd1a2ab4b80 1 --2- 172.21.15.38:0/3584382299 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55baeb450140 0x55baeb44d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:35.962 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.954+0000 7fd1a2ab4b80 1 --2- 172.21.15.38:0/3584382299 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55baeb44f830 0x55baeb443c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:35.962 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.954+0000 7fd1a2ab4b80 1 -- 172.21.15.38:0/3584382299 >> 172.21.15.38:0/3584382299 conn(0x55baeb444a20 msgr2=0x55baeb460000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:35.962 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.954+0000 7fd1a2ab4b80 1 -- 172.21.15.38:0/3584382299 shutdown_connections 2023-03-09T14:46:35.962 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.954+0000 7fd1a2ab4b80 1 -- 172.21.15.38:0/3584382299 wait complete. 2023-03-09T14:46:35.962 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.954+0000 7fd1a2ab4b80 1 -- start start 2023-03-09T14:46:35.962 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.955+0000 7fd1a2ab4b80 1 Processor -- start 2023-03-09T14:46:35.963 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.955+0000 7fd1a2ab4b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55baeb449fe0 0x55baeb44b290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:35.963 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.955+0000 7fd1a2ab4b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55baeb44f830 0x55baeb44b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:35.963 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.955+0000 7fd1a2ab4b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55baeb450140 0x55baeb44bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:35.963 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.955+0000 7fd1a2ab4b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55baeb44c250 con 0x55baeb44f830 2023-03-09T14:46:35.963 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.955+0000 7fd1a2ab4b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55baeb44c3c0 con 0x55baeb450140 2023-03-09T14:46:35.963 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.955+0000 7fd1a2ab4b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55baeb44c530 con 0x55baeb449fe0 2023-03-09T14:46:35.963 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.955+0000 7fd1a2ab4b80 10 client.0 _ll_register_callbacks cb 0x55baeb341520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:46:35.964 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.955+0000 7fd18f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55baeb44f830 0x55baeb44b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:35.964 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.955+0000 7fd18ffff700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55baeb449fe0 0x55baeb44b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:35.964 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.955+0000 7fd18f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55baeb44f830 0x55baeb44b7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39266/0 (socket says 172.21.15.38:39266) 2023-03-09T14:46:35.964 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.955+0000 7fd18f7fe700 1 -- 172.21.15.38:0/1228337229 learned_addr learned my addr 172.21.15.38:0/1228337229 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:46:35.964 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.955+0000 7fd194ec8700 1 --2- 172.21.15.38:0/1228337229 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55baeb450140 0x55baeb44bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:35.964 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.955+0000 7fd18ffff700 1 -- 172.21.15.38:0/1228337229 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55baeb450140 msgr2=0x55baeb44bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:46:35.964 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.955+0000 7fd18ffff700 1 --2- 172.21.15.38:0/1228337229 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55baeb450140 0x55baeb44bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:35.964 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.955+0000 7fd18ffff700 1 -- 172.21.15.38:0/1228337229 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55baeb44f830 msgr2=0x55baeb44b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:46:35.965 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.955+0000 7fd18ffff700 1 --2- 172.21.15.38:0/1228337229 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55baeb44f830 0x55baeb44b7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:35.965 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.955+0000 7fd18ffff700 1 -- 172.21.15.38:0/1228337229 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd18000e830 con 0x55baeb449fe0 2023-03-09T14:46:35.965 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.956+0000 7fd18ffff700 1 --2- 172.21.15.38:0/1228337229 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55baeb449fe0 0x55baeb44b290 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd1800048a0 tx=0x7fd180006eb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=25b36e88376e3548 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:35.965 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.956+0000 7fd16f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2023-03-09T14:46:35.965 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.956+0000 7fd16f7fe700 1 -- 172.21.15.38:0/1228337229 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd180031cc0 con 0x55baeb449fe0 2023-03-09T14:46:35.965 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.956+0000 7fd16f7fe700 1 -- 172.21.15.38:0/1228337229 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9278+0+0 (secure 0 0 0) 0x7fd18001e830 con 0x55baeb449fe0 2023-03-09T14:46:35.965 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.956+0000 7fd1a2ab4b80 1 -- 172.21.15.38:0/1228337229 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55baeb3591b0 con 0x55baeb449fe0 2023-03-09T14:46:35.965 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.956+0000 7fd16f7fe700 1 -- 172.21.15.38:0/1228337229 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd180031e20 con 0x55baeb449fe0 2023-03-09T14:46:35.966 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.956+0000 7fd1a2ab4b80 10 client.25282 fetch_fsmap learned FSMap version 23 2023-03-09T14:46:35.966 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.956+0000 7fd1a2ab4b80 1 -- 172.21.15.38:0/1228337229 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55baeb1f0250 con 0x55baeb449fe0 2023-03-09T14:46:35.966 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.956+0000 7fd16f7fe700 1 -- 172.21.15.38:0/1228337229 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55baeb3591b0 con 0x55baeb449fe0 2023-03-09T14:46:35.966 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.956+0000 7fd16f7fe700 1 -- 172.21.15.38:0/1228337229 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fd18004f070 con 0x55baeb449fe0 2023-03-09T14:46:35.966 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.956+0000 7fd1a2ab4b80 10 client.25282 fetch_fsmap finished waiting for FSMap version 23 2023-03-09T14:46:35.966 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.956+0000 7fd1a2ab4b80 10 client.25282 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T14:46:35.966 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.956+0000 7fd1a2ab4b80 20 client.25282 populate_metadata read hostname 'smithi038' 2023-03-09T14:46:35.967 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.956+0000 7fd1a2ab4b80 1 --2- 172.21.15.38:0/1228337229 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55baeb463b90 0x55baeb4b9f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:35.967 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.956+0000 7fd1a2ab4b80 4 client.25282 mds_command: new command op to 24530 tid=0[{"prefix": "scrub status"}] 2023-03-09T14:46:35.967 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.956+0000 7fd1a2ab4b80 1 -- 172.21.15.38:0/1228337229 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55baeb3431c0 con 0x55baeb463b90 2023-03-09T14:46:35.967 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.957+0000 7fd18f7fe700 1 --2- 172.21.15.38:0/1228337229 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55baeb463b90 0x55baeb4b9f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:35.967 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.957+0000 7fd18f7fe700 1 --2- 172.21.15.38:0/1228337229 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55baeb463b90 0x55baeb4b9f20 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=74b664bd12479598 server_cookie=4cdf0f5fdd59e733 in_seq=0 out_seq=0 2023-03-09T14:46:35.967 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.957+0000 7fd16f7fe700 10 client.25282 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:46:35.967 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.957+0000 7fd16f7fe700 1 -- 172.21.15.38:0/1228337229 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55baeb3431c0 con 0x55baeb463b90 2023-03-09T14:46:35.968 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.957+0000 7fd16f7fe700 10 client.25282 handle_command_reply: tid=0 2023-03-09T14:46:35.968 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.957+0000 7fd1a2ab4b80 1 client.25282 shutdown 2023-03-09T14:46:35.968 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.957+0000 7fd1a2ab4b80 1 -- 172.21.15.38:0/1228337229 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55baeb449fe0 msgr2=0x55baeb44b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:35.968 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.957+0000 7fd1a2ab4b80 1 --2- 172.21.15.38:0/1228337229 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55baeb449fe0 0x55baeb44b290 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd1800048a0 tx=0x7fd180006eb0 comp rx=0 tx=0).stop 2023-03-09T14:46:35.968 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.957+0000 7fd1a2ab4b80 1 -- 172.21.15.38:0/1228337229 shutdown_connections 2023-03-09T14:46:35.968 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.957+0000 7fd1a2ab4b80 1 --2- 172.21.15.38:0/1228337229 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55baeb463b90 0x55baeb4b9f20 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:35.968 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.957+0000 7fd1a2ab4b80 1 --2- 172.21.15.38:0/1228337229 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55baeb450140 0x55baeb44bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:35.968 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.957+0000 7fd16f7fe700 0 client.25282 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:46:35.969 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.957+0000 7fd1a2ab4b80 1 --2- 172.21.15.38:0/1228337229 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55baeb449fe0 0x55baeb44b290 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:35.969 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.957+0000 7fd1a2ab4b80 1 --2- 172.21.15.38:0/1228337229 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55baeb44f830 0x55baeb44b7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:35.969 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.957+0000 7fd1a2ab4b80 1 -- 172.21.15.38:0/1228337229 >> 172.21.15.38:0/1228337229 conn(0x55baeb444a20 msgr2=0x55baeb44d570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:35.969 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.958+0000 7fd1a2ab4b80 1 -- 172.21.15.38:0/1228337229 shutdown_connections 2023-03-09T14:46:35.969 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.958+0000 7fd1a2ab4b80 1 -- 172.21.15.38:0/1228337229 wait complete. 2023-03-09T14:46:35.969 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.958+0000 7fd1a2ab4b80 20 client.25282 trim_cache size 0 max 16384 2023-03-09T14:46:35.970 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2023-03-09T14:46:35.970 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "no active scrubs running", 2023-03-09T14:46:35.970 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": {} 2023-03-09T14:46:35.970 INFO:teuthology.orchestra.run.smithi038.stdout:} 2023-03-09T14:46:35.971 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.958+0000 7fd19692b700 1 -- 172.21.15.38:0/3507952880 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fd17809f4e0 msgr2=0x7fd1780a1990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:35.971 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.958+0000 7fd19692b700 1 --2- 172.21.15.38:0/3507952880 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fd17809f4e0 0x7fd1780a1990 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fd180009a60 tx=0x7fd180009e20 comp rx=0 tx=0).stop 2023-03-09T14:46:35.972 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.958+0000 7fd19692b700 1 -- 172.21.15.38:0/3507952880 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd1900ff990 msgr2=0x7fd19010c890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:35.972 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.958+0000 7fd19692b700 1 --2- 172.21.15.38:0/3507952880 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd1900ff990 0x7fd19010c890 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fd18400d950 tx=0x7fd18400dc60 comp rx=0 tx=0).stop 2023-03-09T14:46:35.972 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.958+0000 7fd19692b700 1 -- 172.21.15.38:0/3507952880 shutdown_connections 2023-03-09T14:46:35.972 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.958+0000 7fd19692b700 1 --2- 172.21.15.38:0/3507952880 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fd17809f4e0 0x7fd1780a1990 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:35.973 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.958+0000 7fd19692b700 1 --2- 172.21.15.38:0/3507952880 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd190104030 0x7fd1901a4390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:35.973 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.958+0000 7fd19692b700 1 --2- 172.21.15.38:0/3507952880 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd1900feff0 0x7fd19010c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:35.973 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.958+0000 7fd19692b700 1 --2- 172.21.15.38:0/3507952880 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd1900ff990 0x7fd19010c890 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:35.973 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.958+0000 7fd19692b700 1 -- 172.21.15.38:0/3507952880 >> 172.21.15.38:0/3507952880 conn(0x7fd1900fa840 msgr2=0x7fd1900fb330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:35.973 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.958+0000 7fd19692b700 1 -- 172.21.15.38:0/3507952880 shutdown_connections 2023-03-09T14:46:35.973 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:35.958+0000 7fd19692b700 1 -- 172.21.15.38:0/3507952880 wait complete. 2023-03-09T14:46:35.974 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:35 smithi038 ceph-mon[92422]: pgmap v13: 129 pgs: 129 active+clean; 475 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 75 MiB/s wr, 151 op/s 2023-03-09T14:46:35.974 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-03-09T14:46:35.975 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-09T14:46:35.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:35 smithi122 ceph-mon[93869]: pgmap v13: 129 pgs: 129 active+clean; 475 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 75 MiB/s wr, 151 op/s 2023-03-09T14:46:35.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:35 smithi181 ceph-mon[94189]: pgmap v13: 129 pgs: 129 active+clean; 475 MiB data, 976 MiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 75 MiB/s wr, 151 op/s 2023-03-09T14:46:36.073 INFO:teuthology.orchestra.run.smithi038.stderr:7800000 of 50000000 tuples (15%) done (elapsed 11.21 s, remaining 60.65 s) 2023-03-09T14:46:36.179 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.111+0000 7f4967fec700 1 -- 172.21.15.38:0/3877816726 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f49600ff9a0 msgr2=0x7f4960103960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:36.179 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.111+0000 7f4967fec700 1 --2- 172.21.15.38:0/3877816726 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f49600ff9a0 0x7f4960103960 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f49500099c0 tx=0x7f4950009cd0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.179 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.111+0000 7f4967fec700 1 -- 172.21.15.38:0/3877816726 shutdown_connections 2023-03-09T14:46:36.179 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.111+0000 7f4967fec700 1 --2- 172.21.15.38:0/3877816726 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4960104040 0x7f496010bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.179 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.111+0000 7f4967fec700 1 --2- 172.21.15.38:0/3877816726 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f49600ff000 0x7f49600ff3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.180 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.111+0000 7f4967fec700 1 --2- 172.21.15.38:0/3877816726 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f49600ff9a0 0x7f4960103960 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.180 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.111+0000 7f4967fec700 1 -- 172.21.15.38:0/3877816726 >> 172.21.15.38:0/3877816726 conn(0x7f4960076030 msgr2=0x7f4960076430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:36.180 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.111+0000 7f4967fec700 1 -- 172.21.15.38:0/3877816726 shutdown_connections 2023-03-09T14:46:36.180 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.111+0000 7f4967fec700 1 -- 172.21.15.38:0/3877816726 wait complete. 2023-03-09T14:46:36.180 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.112+0000 7f4967fec700 1 Processor -- start 2023-03-09T14:46:36.180 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.112+0000 7f4967fec700 1 -- start start 2023-03-09T14:46:36.180 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.112+0000 7f4967fec700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f49600ff000 0x7f4960195fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:36.180 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.112+0000 7f4967fec700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f49600ff9a0 0x7f4960196510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:36.181 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.112+0000 7f4967fec700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4960104040 0x7f496019a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:36.181 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.112+0000 7f4967fec700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f496019afb0 con 0x7f49600ff000 2023-03-09T14:46:36.181 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.112+0000 7f4967fec700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f496019b120 con 0x7f4960104040 2023-03-09T14:46:36.182 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.112+0000 7f4967fec700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f496019b2e0 con 0x7f49600ff9a0 2023-03-09T14:46:36.182 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.112+0000 7f4965d88700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f49600ff000 0x7f4960195fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:36.182 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.112+0000 7f4965d88700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f49600ff000 0x7f4960195fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39278/0 (socket says 172.21.15.38:39278) 2023-03-09T14:46:36.182 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.112+0000 7f4965d88700 1 -- 172.21.15.38:0/2900950494 learned_addr learned my addr 172.21.15.38:0/2900950494 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:46:36.182 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.112+0000 7f4965587700 1 --2- 172.21.15.38:0/2900950494 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f49600ff9a0 0x7f4960196510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:36.182 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.112+0000 7f4966589700 1 --2- 172.21.15.38:0/2900950494 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4960104040 0x7f496019a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:36.182 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.112+0000 7f4965d88700 1 -- 172.21.15.38:0/2900950494 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f49600ff9a0 msgr2=0x7f4960196510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:36.182 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.112+0000 7f4965d88700 1 --2- 172.21.15.38:0/2900950494 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f49600ff9a0 0x7f4960196510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.183 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.112+0000 7f4965d88700 1 -- 172.21.15.38:0/2900950494 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4960104040 msgr2=0x7f496019a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:36.183 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.112+0000 7f4965d88700 1 --2- 172.21.15.38:0/2900950494 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4960104040 0x7f496019a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.183 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.112+0000 7f4965d88700 1 -- 172.21.15.38:0/2900950494 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49500095f0 con 0x7f49600ff000 2023-03-09T14:46:36.183 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.113+0000 7f4965d88700 1 --2- 172.21.15.38:0/2900950494 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f49600ff000 0x7f4960195fd0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f495c00d980 tx=0x7f495c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:36.183 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.113+0000 7f4956ffd700 1 -- 172.21.15.38:0/2900950494 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f495c004220 con 0x7f49600ff000 2023-03-09T14:46:36.183 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.113+0000 7f4967fec700 1 -- 172.21.15.38:0/2900950494 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f496010ff10 con 0x7f49600ff000 2023-03-09T14:46:36.183 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.113+0000 7f4956ffd700 1 -- 172.21.15.38:0/2900950494 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f495c007640 con 0x7f49600ff000 2023-03-09T14:46:36.184 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.113+0000 7f4956ffd700 1 -- 172.21.15.38:0/2900950494 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f495c005610 con 0x7f49600ff000 2023-03-09T14:46:36.184 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.113+0000 7f4967fec700 1 -- 172.21.15.38:0/2900950494 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f496010fd80 con 0x7f49600ff000 2023-03-09T14:46:36.184 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.114+0000 7f4967fec700 1 -- 172.21.15.38:0/2900950494 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49601100a0 con 0x7f49600ff000 2023-03-09T14:46:36.184 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.115+0000 7f4956ffd700 1 -- 172.21.15.38:0/2900950494 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 23) v1 ==== 133335+0+0 (secure 0 0 0) 0x7f495c021070 con 0x7f49600ff000 2023-03-09T14:46:36.184 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.116+0000 7f4956ffd700 1 --2- 172.21.15.38:0/2900950494 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f494c09f4e0 0x7f494c0a1990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:36.184 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.116+0000 7f4956ffd700 1 -- 172.21.15.38:0/2900950494 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9278+0+0 (secure 0 0 0) 0x7f495c0ca630 con 0x7f49600ff000 2023-03-09T14:46:36.184 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.116+0000 7f4965587700 1 --2- 172.21.15.38:0/2900950494 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f494c09f4e0 0x7f494c0a1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:36.184 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.117+0000 7f4965587700 1 --2- 172.21.15.38:0/2900950494 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f494c09f4e0 0x7f494c0a1990 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f49500098e0 tx=0x7f4950000bc0 comp rx=0 tx=0).ready entity=mgr.15558 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:36.185 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.119+0000 7f4956ffd700 1 -- 172.21.15.38:0/2900950494 <== mon.0 v2:172.21.15.38:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f495c094510 con 0x7f49600ff000 2023-03-09T14:46:36.185 INFO:teuthology.orchestra.run.smithi038.stderr:7900000 of 50000000 tuples (15%) done (elapsed 11.31 s, remaining 60.25 s) 2023-03-09T14:46:36.319 INFO:teuthology.orchestra.run.smithi038.stderr:8000000 of 50000000 tuples (16%) done (elapsed 11.46 s, remaining 60.15 s) 2023-03-09T14:46:36.394 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.391+0000 7f4967fec700 1 -- 172.21.15.38:0/2900950494 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4960067660 con 0x7f49600ff000 2023-03-09T14:46:36.394 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.392+0000 7f4956ffd700 1 -- 172.21.15.38:0/2900950494 <== mon.0 v2:172.21.15.38:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+10801 (secure 0 0 0) 0x7f495c016030 con 0x7f49600ff000 2023-03-09T14:46:36.395 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:46:36.395 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/768918602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":768918602},{"type":"v1","addr":"172.21.15.38:6835","nonce":768918602}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/18381540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":18381540},{"type":"v1","addr":"172.21.15.38:6837","nonce":18381540}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14610,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6839/4139020066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":4139020066},{"type":"v1","addr":"172.21.15.38:6839","nonce":4139020066}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14622,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6841/1397115322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":1397115322},{"type":"v1","addr":"172.21.15.38:6841","nonce":1397115322}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6833/415305571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":415305571},{"type":"v1","addr":"172.21.15.181:6833","nonce":415305571}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/659300520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":659300520},{"type":"v1","addr":"172.21.15.122:6833","nonce":659300520}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-03-09T14:44:19.083770+0000","modified":"2023-03-09T14:44:39.526150+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24530,"mds_1":24494,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24448":{"gid":24448,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6835/2576255861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":2576255861},{"type":"v1","addr":"172.21.15.181:6835","nonce":2576255861}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24460":{"gid":24460,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6837/2942560181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":2942560181},{"type":"v1","addr":"172.21.15.181:6837","nonce":2942560181}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24464":{"gid":24464,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.122:6835/3675537941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":3675537941},{"type":"v1","addr":"172.21.15.122:6835","nonce":3675537941}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24478":{"gid":24478,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":9,"addr":"172.21.15.181:6839/3902263660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":3902263660},{"type":"v1","addr":"172.21.15.181:6839","nonce":3902263660}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24494":{"gid":24494,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.122:6837/2210214596","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":2210214596},{"type":"v1","addr":"172.21.15.122:6837","nonce":2210214596}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24530":{"gid":24530,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6839/971050165","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6838","nonce":971050165},{"type":"v1","addr":"172.21.15.122:6839","nonce":971050165}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-09T14:46:36.397 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.395+0000 7f4967fec700 1 -- 172.21.15.38:0/2900950494 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f494c09f4e0 msgr2=0x7f494c0a1990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:36.397 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.395+0000 7f4967fec700 1 --2- 172.21.15.38:0/2900950494 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f494c09f4e0 0x7f494c0a1990 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f49500098e0 tx=0x7f4950000bc0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.397 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.395+0000 7f4967fec700 1 -- 172.21.15.38:0/2900950494 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f49600ff000 msgr2=0x7f4960195fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:36.398 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.395+0000 7f4967fec700 1 --2- 172.21.15.38:0/2900950494 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f49600ff000 0x7f4960195fd0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f495c00d980 tx=0x7f495c00dc90 comp rx=0 tx=0).stop 2023-03-09T14:46:36.398 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.395+0000 7f4967fec700 1 -- 172.21.15.38:0/2900950494 shutdown_connections 2023-03-09T14:46:36.398 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.395+0000 7f4967fec700 1 --2- 172.21.15.38:0/2900950494 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f494c09f4e0 0x7f494c0a1990 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.398 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.395+0000 7f4967fec700 1 --2- 172.21.15.38:0/2900950494 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f49600ff000 0x7f4960195fd0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.398 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.395+0000 7f4967fec700 1 --2- 172.21.15.38:0/2900950494 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4960104040 0x7f496019a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.398 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.395+0000 7f4967fec700 1 --2- 172.21.15.38:0/2900950494 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f49600ff9a0 0x7f4960196510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.398 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.395+0000 7f4967fec700 1 -- 172.21.15.38:0/2900950494 >> 172.21.15.38:0/2900950494 conn(0x7f4960076030 msgr2=0x7f49600fc660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:36.399 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.395+0000 7f4967fec700 1 -- 172.21.15.38:0/2900950494 shutdown_connections 2023-03-09T14:46:36.399 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.395+0000 7f4967fec700 1 -- 172.21.15.38:0/2900950494 wait complete. 2023-03-09T14:46:36.399 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 23 2023-03-09T14:46:36.413 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-03-09T14:46:36.475 INFO:teuthology.orchestra.run.smithi038.stderr:8100000 of 50000000 tuples (16%) done (elapsed 11.61 s, remaining 60.07 s) 2023-03-09T14:46:36.556 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.554+0000 7f4d8b840700 1 -- 172.21.15.38:0/2591216236 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d840ff100 msgr2=0x7f4d840ff4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:36.557 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.554+0000 7f4d8b840700 1 --2- 172.21.15.38:0/2591216236 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d840ff100 0x7f4d840ff4d0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f4d800099c0 tx=0x7f4d80009cd0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.557 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.554+0000 7f4d8b840700 1 -- 172.21.15.38:0/2591216236 shutdown_connections 2023-03-09T14:46:36.557 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.554+0000 7f4d8b840700 1 --2- 172.21.15.38:0/2591216236 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d840ff100 0x7f4d840ff4d0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.557 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.554+0000 7f4d8b840700 1 --2- 172.21.15.38:0/2591216236 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4d84104140 0x7f4d8410bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.557 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.554+0000 7f4d8b840700 1 --2- 172.21.15.38:0/2591216236 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4d840ffaa0 0x7f4d84103a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.558 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.554+0000 7f4d8b840700 1 -- 172.21.15.38:0/2591216236 >> 172.21.15.38:0/2591216236 conn(0x7f4d84075cc0 msgr2=0x7f4d840760c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:36.558 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.554+0000 7f4d8b840700 1 -- 172.21.15.38:0/2591216236 shutdown_connections 2023-03-09T14:46:36.558 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.554+0000 7f4d8b840700 1 -- 172.21.15.38:0/2591216236 wait complete. 2023-03-09T14:46:36.561 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.554+0000 7f4d8b840700 1 Processor -- start 2023-03-09T14:46:36.561 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.555+0000 7f4d8b840700 1 -- start start 2023-03-09T14:46:36.561 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.555+0000 7f4d8b840700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4d840ff100 0x7f4d84196060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:36.562 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.555+0000 7f4d8b840700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d840ffaa0 0x7f4d841965a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:36.562 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.555+0000 7f4d8b840700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4d84104140 0x7f4d8419ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:36.562 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.555+0000 7f4d8b840700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8419b250 con 0x7f4d840ffaa0 2023-03-09T14:46:36.562 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.555+0000 7f4d8b840700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8419b3c0 con 0x7f4d840ff100 2023-03-09T14:46:36.562 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.555+0000 7f4d8b840700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8419b580 con 0x7f4d84104140 2023-03-09T14:46:36.562 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.555+0000 7f4d88ddb700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d840ffaa0 0x7f4d841965a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:36.562 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.555+0000 7f4d88ddb700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d840ffaa0 0x7f4d841965a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39294/0 (socket says 172.21.15.38:39294) 2023-03-09T14:46:36.562 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.555+0000 7f4d88ddb700 1 -- 172.21.15.38:0/3463816598 learned_addr learned my addr 172.21.15.38:0/3463816598 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:46:36.563 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.555+0000 7f4d895dc700 1 --2- 172.21.15.38:0/3463816598 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4d840ff100 0x7f4d84196060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:36.563 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.555+0000 7f4d89ddd700 1 --2- 172.21.15.38:0/3463816598 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4d84104140 0x7f4d8419ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:36.563 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.555+0000 7f4d88ddb700 1 -- 172.21.15.38:0/3463816598 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4d84104140 msgr2=0x7f4d8419ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:36.563 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.555+0000 7f4d88ddb700 1 --2- 172.21.15.38:0/3463816598 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4d84104140 0x7f4d8419ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.563 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.555+0000 7f4d88ddb700 1 -- 172.21.15.38:0/3463816598 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4d840ff100 msgr2=0x7f4d84196060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:36.563 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.555+0000 7f4d88ddb700 1 --2- 172.21.15.38:0/3463816598 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4d840ff100 0x7f4d84196060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.563 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.555+0000 7f4d88ddb700 1 -- 172.21.15.38:0/3463816598 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d800095f0 con 0x7f4d840ffaa0 2023-03-09T14:46:36.564 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.556+0000 7f4d88ddb700 1 --2- 172.21.15.38:0/3463816598 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d840ffaa0 0x7f4d841965a0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f4d7400e940 tx=0x7f4d7400ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:36.564 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.556+0000 7f4d7a7fc700 1 -- 172.21.15.38:0/3463816598 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d74009730 con 0x7f4d840ffaa0 2023-03-09T14:46:36.564 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.556+0000 7f4d7a7fc700 1 -- 172.21.15.38:0/3463816598 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4d74010430 con 0x7f4d840ffaa0 2023-03-09T14:46:36.564 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.556+0000 7f4d8b840700 1 -- 172.21.15.38:0/3463816598 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d8410ef10 con 0x7f4d840ffaa0 2023-03-09T14:46:36.564 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.556+0000 7f4d7a7fc700 1 -- 172.21.15.38:0/3463816598 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d7400cea0 con 0x7f4d840ffaa0 2023-03-09T14:46:36.564 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.556+0000 7f4d8b840700 1 -- 172.21.15.38:0/3463816598 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d8410ed80 con 0x7f4d840ffaa0 2023-03-09T14:46:36.564 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.559+0000 7f4d7a7fc700 1 -- 172.21.15.38:0/3463816598 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 23) v1 ==== 133335+0+0 (secure 0 0 0) 0x7f4d7400c040 con 0x7f4d840ffaa0 2023-03-09T14:46:36.565 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.560+0000 7f4d979c9b80 1 Processor -- start 2023-03-09T14:46:36.565 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.560+0000 7f4d979c9b80 1 -- start start 2023-03-09T14:46:36.566 INFO:teuthology.orchestra.run.smithi038.stderr:8200000 of 50000000 tuples (16%) done (elapsed 11.70 s, remaining 59.64 s) 2023-03-09T14:46:36.570 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.560+0000 7f4d7a7fc700 1 --2- 172.21.15.38:0/3463816598 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f4d7009f4e0 0x7f4d700a1990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:36.570 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.560+0000 7f4d979c9b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562e1c994830 0x562e1c994c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:36.570 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.560+0000 7f4d979c9b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562e1c995140 0x562e1c98eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:36.570 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.560+0000 7f4d979c9b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e1c98efe0 0x562e1c98f410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:36.570 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.560+0000 7f4d979c9b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e1c878da0 con 0x562e1c98efe0 2023-03-09T14:46:36.570 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.560+0000 7f4d979c9b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e1c8e7780 con 0x562e1c995140 2023-03-09T14:46:36.571 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.560+0000 7f4d979c9b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e1c8e0cb0 con 0x562e1c994830 2023-03-09T14:46:36.571 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.560+0000 7f4d7a7fc700 1 -- 172.21.15.38:0/3463816598 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9278+0+0 (secure 0 0 0) 0x7f4d74014070 con 0x7f4d840ffaa0 2023-03-09T14:46:36.571 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.560+0000 7f4d89ddd700 1 --2- 172.21.15.38:0/3463816598 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f4d7009f4e0 0x7f4d700a1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:36.571 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.560+0000 7f4d895dc700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562e1c994830 0x562e1c994c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:36.571 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.560+0000 7f4d89ddd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e1c98efe0 0x562e1c98f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:36.571 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.560+0000 7f4d895dc700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562e1c994830 0x562e1c994c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.38:46998/0 (socket says 172.21.15.38:46998) 2023-03-09T14:46:36.571 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.560+0000 7f4d895dc700 1 -- 172.21.15.38:0/3892762166 learned_addr learned my addr 172.21.15.38:0/3892762166 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:46:36.571 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.561+0000 7f4d895dc700 1 -- 172.21.15.38:0/3892762166 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562e1c995140 msgr2=0x562e1c98eaa0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-03-09T14:46:36.572 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.561+0000 7f4d895dc700 1 --2- 172.21.15.38:0/3892762166 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562e1c995140 0x562e1c98eaa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.572 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.561+0000 7f4d895dc700 1 -- 172.21.15.38:0/3892762166 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e1c98efe0 msgr2=0x562e1c98f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:46:36.572 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.561+0000 7f4d895dc700 1 --2- 172.21.15.38:0/3892762166 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e1c98efe0 0x562e1c98f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.572 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.561+0000 7f4d895dc700 1 -- 172.21.15.38:0/3892762166 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562e1c8a7190 con 0x562e1c994830 2023-03-09T14:46:36.572 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.561+0000 7f4d89ddd700 1 --2- 172.21.15.38:0/3463816598 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f4d7009f4e0 0x7f4d700a1990 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c019bf0 tx=0x7f4d7c009bf0 comp rx=0 tx=0).ready entity=mgr.15558 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:36.572 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.561+0000 7f4d895dc700 1 --2- 172.21.15.38:0/3892762166 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562e1c994830 0x562e1c994c00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4d800098e0 tx=0x7f4d80005fb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c0bc5564162bd56 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:36.572 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.561+0000 7f4d6f7fe700 1 -- 172.21.15.38:0/3892762166 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d8001c050 con 0x562e1c994830 2023-03-09T14:46:36.573 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.561+0000 7f4d6f7fe700 1 -- 172.21.15.38:0/3892762166 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4d8001d040 con 0x562e1c994830 2023-03-09T14:46:36.573 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.561+0000 7f4d6f7fe700 1 -- 172.21.15.38:0/3892762166 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d8000b9f0 con 0x562e1c994830 2023-03-09T14:46:36.573 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.562+0000 7f4d979c9b80 1 -- 172.21.15.38:0/3892762166 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562e1c994830 msgr2=0x562e1c994c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:36.573 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.562+0000 7f4d979c9b80 1 --2- 172.21.15.38:0/3892762166 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562e1c994830 0x562e1c994c00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4d800098e0 tx=0x7f4d80005fb0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.573 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.562+0000 7f4d979c9b80 1 -- 172.21.15.38:0/3892762166 shutdown_connections 2023-03-09T14:46:36.573 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.562+0000 7f4d979c9b80 1 --2- 172.21.15.38:0/3892762166 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e1c98efe0 0x562e1c98f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.573 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.562+0000 7f4d979c9b80 1 --2- 172.21.15.38:0/3892762166 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562e1c994830 0x562e1c994c00 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.573 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.562+0000 7f4d979c9b80 1 --2- 172.21.15.38:0/3892762166 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562e1c995140 0x562e1c98eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.574 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.562+0000 7f4d979c9b80 1 -- 172.21.15.38:0/3892762166 >> 172.21.15.38:0/3892762166 conn(0x562e1c989a20 msgr2=0x562e1c989df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:36.574 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.562+0000 7f4d979c9b80 1 -- 172.21.15.38:0/3892762166 shutdown_connections 2023-03-09T14:46:36.574 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.562+0000 7f4d979c9b80 1 -- 172.21.15.38:0/3892762166 wait complete. 2023-03-09T14:46:36.574 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.562+0000 7f4d979c9b80 1 -- start start 2023-03-09T14:46:36.574 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.562+0000 7f4d979c9b80 1 Processor -- start 2023-03-09T14:46:36.574 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.563+0000 7f4d979c9b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562e1c98efe0 0x562e1c99b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:36.574 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.563+0000 7f4d979c9b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562e1c994830 0x562e1c998390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:36.575 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.563+0000 7f4d979c9b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e1c995140 0x562e1c9989a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:36.575 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.563+0000 7f4d979c9b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e1c764920 con 0x562e1c995140 2023-03-09T14:46:36.575 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.563+0000 7f4d979c9b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e1c999110 con 0x562e1c994830 2023-03-09T14:46:36.575 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.563+0000 7f4d979c9b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e1c9a4c10 con 0x562e1c98efe0 2023-03-09T14:46:36.575 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.563+0000 7f4d979c9b80 10 client.0 _ll_register_callbacks cb 0x562e1c886520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:46:36.575 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.563+0000 7f4d89ddd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e1c995140 0x562e1c9989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:36.575 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.563+0000 7f4d89ddd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e1c995140 0x562e1c9989a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39310/0 (socket says 172.21.15.38:39310) 2023-03-09T14:46:36.575 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.563+0000 7f4d89ddd700 1 -- 172.21.15.38:0/3536141348 learned_addr learned my addr 172.21.15.38:0/3536141348 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:46:36.576 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.563+0000 7f4d88ddb700 1 --2- 172.21.15.38:0/3536141348 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562e1c994830 0x562e1c998390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:36.576 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.563+0000 7f4d895dc700 1 --2- 172.21.15.38:0/3536141348 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562e1c98efe0 0x562e1c99b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:36.576 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.563+0000 7f4d89ddd700 1 -- 172.21.15.38:0/3536141348 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562e1c98efe0 msgr2=0x562e1c99b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:46:36.576 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.563+0000 7f4d89ddd700 1 --2- 172.21.15.38:0/3536141348 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562e1c98efe0 0x562e1c99b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.576 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.563+0000 7f4d89ddd700 1 -- 172.21.15.38:0/3536141348 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562e1c994830 msgr2=0x562e1c998390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:46:36.576 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.563+0000 7f4d89ddd700 1 --2- 172.21.15.38:0/3536141348 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562e1c994830 0x562e1c998390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.576 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.563+0000 7f4d89ddd700 1 -- 172.21.15.38:0/3536141348 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d8410c550 con 0x562e1c995140 2023-03-09T14:46:36.577 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.564+0000 7f4d89ddd700 1 --2- 172.21.15.38:0/3536141348 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e1c995140 0x562e1c9989a0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c009ab0 tx=0x7f4d7c009100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2049bdc511aada0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:36.577 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.564+0000 7f4d6cff9700 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2023-03-09T14:46:36.577 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.564+0000 7f4d6cff9700 1 -- 172.21.15.38:0/3536141348 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d7c0095e0 con 0x562e1c995140 2023-03-09T14:46:36.577 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.564+0000 7f4d6cff9700 1 -- 172.21.15.38:0/3536141348 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9278+0+0 (secure 0 0 0) 0x7f4d7c017cd0 con 0x562e1c995140 2023-03-09T14:46:36.577 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.564+0000 7f4d979c9b80 1 -- 172.21.15.38:0/3536141348 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562e1c8ae190 con 0x562e1c995140 2023-03-09T14:46:36.577 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.564+0000 7f4d6cff9700 1 -- 172.21.15.38:0/3536141348 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4d7c017e90 con 0x562e1c995140 2023-03-09T14:46:36.577 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.564+0000 7f4d979c9b80 10 client.15669 fetch_fsmap learned FSMap version 23 2023-03-09T14:46:36.577 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.564+0000 7f4d979c9b80 1 -- 172.21.15.38:0/3536141348 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x562e1c6f7c60 con 0x562e1c995140 2023-03-09T14:46:36.578 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.564+0000 7f4d6cff9700 1 -- 172.21.15.38:0/3536141348 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x562e1c8ae190 con 0x562e1c995140 2023-03-09T14:46:36.578 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.564+0000 7f4d6cff9700 1 -- 172.21.15.38:0/3536141348 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f4d7c0243b0 con 0x562e1c995140 2023-03-09T14:46:36.578 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.564+0000 7f4d979c9b80 10 client.15669 fetch_fsmap finished waiting for FSMap version 23 2023-03-09T14:46:36.578 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.564+0000 7f4d979c9b80 10 client.15669 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-03-09T14:46:36.578 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.564+0000 7f4d979c9b80 20 client.15669 populate_metadata read hostname 'smithi038' 2023-03-09T14:46:36.578 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.564+0000 7f4d979c9b80 1 --2- 172.21.15.38:0/3536141348 >> [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] conn(0x562e1c9a8c50 0x562e1c9ab030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:36.578 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.564+0000 7f4d979c9b80 4 client.15669 mds_command: new command op to 24478 tid=0[{"prefix": "get_command_descriptions"}] 2023-03-09T14:46:36.578 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.564+0000 7f4d979c9b80 1 -- 172.21.15.38:0/3536141348 --> [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562e1c764f60 con 0x562e1c9a8c50 2023-03-09T14:46:36.579 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.565+0000 7f4d895dc700 1 --2- 172.21.15.38:0/3536141348 >> [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] conn(0x562e1c9a8c50 0x562e1c9ab030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:36.579 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.565+0000 7f4d6cff9700 10 client.15669 ms_handle_connect on v2:172.21.15.181:6838/3902263660 2023-03-09T14:46:36.579 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.565+0000 7f4d895dc700 1 --2- 172.21.15.38:0/3536141348 >> [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] conn(0x562e1c9a8c50 0x562e1c9ab030 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5109c30c6ba320d6 server_cookie=7844029f91022436 in_seq=0 out_seq=0 2023-03-09T14:46:36.579 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.566+0000 7f4d6cff9700 1 -- 172.21.15.38:0/3536141348 <== mds.2 v2:172.21.15.181:6838/3902263660 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18278 (crc 0 0 0) 0x562e1c764f60 con 0x562e1c9a8c50 2023-03-09T14:46:36.579 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.566+0000 7f4d6cff9700 10 client.15669 handle_command_reply: tid=0 2023-03-09T14:46:36.579 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.566+0000 7f4d979c9b80 1 client.15669 shutdown 2023-03-09T14:46:36.579 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.566+0000 7f4d979c9b80 1 -- 172.21.15.38:0/3536141348 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e1c995140 msgr2=0x562e1c9989a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:36.580 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.566+0000 7f4d979c9b80 1 --2- 172.21.15.38:0/3536141348 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e1c995140 0x562e1c9989a0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c009ab0 tx=0x7f4d7c009100 comp rx=0 tx=0).stop 2023-03-09T14:46:36.580 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.566+0000 7f4d979c9b80 1 -- 172.21.15.38:0/3536141348 shutdown_connections 2023-03-09T14:46:36.580 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.566+0000 7f4d979c9b80 1 --2- 172.21.15.38:0/3536141348 >> [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] conn(0x562e1c9a8c50 0x562e1c9ab030 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.580 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.566+0000 7f4d979c9b80 1 --2- 172.21.15.38:0/3536141348 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e1c995140 0x562e1c9989a0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.580 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.566+0000 7f4d979c9b80 1 --2- 172.21.15.38:0/3536141348 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562e1c98efe0 0x562e1c99b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.580 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.566+0000 7f4d979c9b80 1 --2- 172.21.15.38:0/3536141348 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562e1c994830 0x562e1c998390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.580 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.566+0000 7f4d979c9b80 1 -- 172.21.15.38:0/3536141348 >> 172.21.15.38:0/3536141348 conn(0x562e1c989a20 msgr2=0x562e1c988c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:36.580 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.566+0000 7f4d6cff9700 0 client.15669 ms_handle_reset on v2:172.21.15.181:6838/3902263660 2023-03-09T14:46:36.581 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.566+0000 7f4d979c9b80 1 -- 172.21.15.38:0/3536141348 shutdown_connections 2023-03-09T14:46:36.581 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.566+0000 7f4d979c9b80 1 -- 172.21.15.38:0/3536141348 wait complete. 2023-03-09T14:46:36.581 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.566+0000 7f4d979c9b80 20 client.15669 trim_cache size 0 max 16384 2023-03-09T14:46:36.583 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.579+0000 7f4d979c9b80 1 Processor -- start 2023-03-09T14:46:36.583 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.579+0000 7f4d979c9b80 1 -- start start 2023-03-09T14:46:36.583 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.579+0000 7f4d979c9b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562e1c98efe0 0x562e1c99fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:36.583 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.579+0000 7f4d979c9b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562e1c994830 0x562e1c99acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:36.583 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.579+0000 7f4d979c9b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e1c995140 0x562e1c9a42f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:36.583 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.579+0000 7f4d979c9b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e1c989840 con 0x562e1c995140 2023-03-09T14:46:36.583 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.579+0000 7f4d979c9b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e1c985d70 con 0x562e1c98efe0 2023-03-09T14:46:36.584 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.579+0000 7f4d979c9b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e1c993060 con 0x562e1c994830 2023-03-09T14:46:36.584 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.580+0000 7f4d895dc700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562e1c98efe0 0x562e1c99fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:36.584 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.580+0000 7f4d88ddb700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562e1c994830 0x562e1c99acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:36.584 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.580+0000 7f4d89ddd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e1c995140 0x562e1c9a42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:36.584 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.580+0000 7f4d895dc700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562e1c98efe0 0x562e1c99fb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.38:48504/0 (socket says 172.21.15.38:48504) 2023-03-09T14:46:36.584 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.580+0000 7f4d89ddd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e1c995140 0x562e1c9a42f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39318/0 (socket says 172.21.15.38:39318) 2023-03-09T14:46:36.584 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.580+0000 7f4d88ddb700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562e1c994830 0x562e1c99acd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.38:47014/0 (socket says 172.21.15.38:47014) 2023-03-09T14:46:36.585 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.580+0000 7f4d895dc700 1 -- 172.21.15.38:0/3350714069 learned_addr learned my addr 172.21.15.38:0/3350714069 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:46:36.585 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.580+0000 7f4d895dc700 1 -- 172.21.15.38:0/3350714069 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562e1c994830 msgr2=0x562e1c99acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:46:36.585 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.580+0000 7f4d895dc700 1 --2- 172.21.15.38:0/3350714069 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562e1c994830 0x562e1c99acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.585 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.580+0000 7f4d895dc700 1 -- 172.21.15.38:0/3350714069 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e1c995140 msgr2=0x562e1c9a42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:46:36.585 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.580+0000 7f4d895dc700 1 --2- 172.21.15.38:0/3350714069 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e1c995140 0x562e1c9a42f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.585 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.580+0000 7f4d895dc700 1 -- 172.21.15.38:0/3350714069 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562e1c98a0c0 con 0x562e1c98efe0 2023-03-09T14:46:36.585 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.580+0000 7f4d88ddb700 1 --2- 172.21.15.38:0/3350714069 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562e1c994830 0x562e1c99acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:46:36.585 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.580+0000 7f4d895dc700 1 --2- 172.21.15.38:0/3350714069 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562e1c98efe0 0x562e1c99fb60 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f4d80005f50 tx=0x7f4d80009cd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5412da08f306fe70 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:36.586 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.580+0000 7f4d6ffff700 1 -- 172.21.15.38:0/3350714069 <== mon.2 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d8001c040 con 0x562e1c98efe0 2023-03-09T14:46:36.586 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.580+0000 7f4d6ffff700 1 -- 172.21.15.38:0/3350714069 <== mon.2 v2:172.21.15.122:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4d80032d10 con 0x562e1c98efe0 2023-03-09T14:46:36.586 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.580+0000 7f4d6ffff700 1 -- 172.21.15.38:0/3350714069 <== mon.2 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d80003c10 con 0x562e1c98efe0 2023-03-09T14:46:36.589 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.581+0000 7f4d979c9b80 1 -- 172.21.15.38:0/3350714069 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562e1c98efe0 msgr2=0x562e1c99fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:36.589 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.581+0000 7f4d979c9b80 1 --2- 172.21.15.38:0/3350714069 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562e1c98efe0 0x562e1c99fb60 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f4d80005f50 tx=0x7f4d80009cd0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.589 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.581+0000 7f4d979c9b80 1 -- 172.21.15.38:0/3350714069 shutdown_connections 2023-03-09T14:46:36.589 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.581+0000 7f4d979c9b80 1 --2- 172.21.15.38:0/3350714069 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e1c995140 0x562e1c9a42f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.590 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.581+0000 7f4d979c9b80 1 --2- 172.21.15.38:0/3350714069 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562e1c98efe0 0x562e1c99fb60 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.590 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.581+0000 7f4d979c9b80 1 --2- 172.21.15.38:0/3350714069 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562e1c994830 0x562e1c99acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.590 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.581+0000 7f4d979c9b80 1 -- 172.21.15.38:0/3350714069 >> 172.21.15.38:0/3350714069 conn(0x562e1c989a20 msgr2=0x562e1c987f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:36.590 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.581+0000 7f4d979c9b80 1 -- 172.21.15.38:0/3350714069 shutdown_connections 2023-03-09T14:46:36.590 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.581+0000 7f4d979c9b80 1 -- 172.21.15.38:0/3350714069 wait complete. 2023-03-09T14:46:36.590 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.581+0000 7f4d979c9b80 1 -- start start 2023-03-09T14:46:36.590 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.582+0000 7f4d979c9b80 1 Processor -- start 2023-03-09T14:46:36.590 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.582+0000 7f4d979c9b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e1c98efe0 0x562e1c99fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:36.591 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.582+0000 7f4d979c9b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562e1c994830 0x562e1c99db40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:36.591 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.582+0000 7f4d979c9b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562e1c995140 0x562e1c99e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:36.591 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.582+0000 7f4d979c9b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e1c99e950 con 0x562e1c98efe0 2023-03-09T14:46:36.591 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.582+0000 7f4d979c9b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e1c9fbac0 con 0x562e1c995140 2023-03-09T14:46:36.591 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.582+0000 7f4d979c9b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e1c9fbc00 con 0x562e1c994830 2023-03-09T14:46:36.591 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.582+0000 7f4d979c9b80 10 client.0 _ll_register_callbacks cb 0x562e1c886520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:46:36.591 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.582+0000 7f4d895dc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e1c98efe0 0x562e1c99fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:36.592 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.582+0000 7f4d895dc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e1c98efe0 0x562e1c99fd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39320/0 (socket says 172.21.15.38:39320) 2023-03-09T14:46:36.592 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.582+0000 7f4d895dc700 1 -- 172.21.15.38:0/3056641965 learned_addr learned my addr 172.21.15.38:0/3056641965 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:46:36.592 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.582+0000 7f4d88ddb700 1 --2- 172.21.15.38:0/3056641965 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562e1c994830 0x562e1c99db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:36.592 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.582+0000 7f4d89ddd700 1 --2- 172.21.15.38:0/3056641965 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562e1c995140 0x562e1c99e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:36.592 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.583+0000 7f4d89ddd700 1 -- 172.21.15.38:0/3056641965 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562e1c994830 msgr2=0x562e1c99db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:46:36.592 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.583+0000 7f4d89ddd700 1 --2- 172.21.15.38:0/3056641965 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562e1c994830 0x562e1c99db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.592 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.583+0000 7f4d89ddd700 1 -- 172.21.15.38:0/3056641965 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e1c98efe0 msgr2=0x562e1c99fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:46:36.592 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.583+0000 7f4d89ddd700 1 --2- 172.21.15.38:0/3056641965 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e1c98efe0 0x562e1c99fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.593 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.583+0000 7f4d89ddd700 1 -- 172.21.15.38:0/3056641965 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d8410ed80 con 0x562e1c995140 2023-03-09T14:46:36.593 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.583+0000 7f4d895dc700 1 --2- 172.21.15.38:0/3056641965 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e1c98efe0 0x562e1c99fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:46:36.593 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.583+0000 7f4d88ddb700 1 --2- 172.21.15.38:0/3056641965 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562e1c994830 0x562e1c99db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:46:36.593 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.583+0000 7f4d89ddd700 1 --2- 172.21.15.38:0/3056641965 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562e1c995140 0x562e1c99e280 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c0061e0 tx=0x7f4d7c009690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37c030fdf6a0dfe8 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:36.593 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.583+0000 7f4d6cff9700 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2023-03-09T14:46:36.593 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.583+0000 7f4d6cff9700 1 -- 172.21.15.38:0/3056641965 <== mon.2 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d7c024a30 con 0x562e1c995140 2023-03-09T14:46:36.593 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.583+0000 7f4d6cff9700 1 -- 172.21.15.38:0/3056641965 <== mon.2 v2:172.21.15.122:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9278+0+0 (secure 0 0 0) 0x7f4d7c024c70 con 0x562e1c995140 2023-03-09T14:46:36.593 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.583+0000 7f4d979c9b80 1 -- 172.21.15.38:0/3056641965 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562e1c89e1b0 con 0x562e1c995140 2023-03-09T14:46:36.594 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.583+0000 7f4d6cff9700 1 -- 172.21.15.38:0/3056641965 <== mon.2 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4d7c024e30 con 0x562e1c995140 2023-03-09T14:46:36.594 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.583+0000 7f4d979c9b80 10 client.25346 fetch_fsmap learned FSMap version 23 2023-03-09T14:46:36.594 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.583+0000 7f4d979c9b80 1 -- 172.21.15.38:0/3056641965 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x562e1c8a7190 con 0x562e1c995140 2023-03-09T14:46:36.594 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.583+0000 7f4d6cff9700 1 -- 172.21.15.38:0/3056641965 <== mon.2 v2:172.21.15.122:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x562e1c89e1b0 con 0x562e1c995140 2023-03-09T14:46:36.594 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.583+0000 7f4d6cff9700 1 -- 172.21.15.38:0/3056641965 <== mon.2 v2:172.21.15.122:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f4d7c025cc0 con 0x562e1c995140 2023-03-09T14:46:36.594 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.583+0000 7f4d979c9b80 10 client.25346 fetch_fsmap finished waiting for FSMap version 23 2023-03-09T14:46:36.594 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.583+0000 7f4d979c9b80 10 client.25346 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-03-09T14:46:36.595 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.583+0000 7f4d979c9b80 20 client.25346 populate_metadata read hostname 'smithi038' 2023-03-09T14:46:36.595 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.583+0000 7f4d979c9b80 1 --2- 172.21.15.38:0/3056641965 >> [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] conn(0x562e1c9fd130 0x562e1c9ff540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:36.595 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.583+0000 7f4d979c9b80 4 client.25346 mds_command: new command op to 24478 tid=0[{"prefix": "damage ls"}] 2023-03-09T14:46:36.595 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.583+0000 7f4d979c9b80 1 -- 172.21.15.38:0/3056641965 --> [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x562e1c8881c0 con 0x562e1c9fd130 2023-03-09T14:46:36.595 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.584+0000 7f4d895dc700 1 --2- 172.21.15.38:0/3056641965 >> [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] conn(0x562e1c9fd130 0x562e1c9ff540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:36.595 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.584+0000 7f4d6cff9700 10 client.25346 ms_handle_connect on v2:172.21.15.181:6838/3902263660 2023-03-09T14:46:36.595 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.584+0000 7f4d895dc700 1 --2- 172.21.15.38:0/3056641965 >> [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] conn(0x562e1c9fd130 0x562e1c9ff540 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1f1256df5dc1dca1 server_cookie=2018a8ee65934304 in_seq=0 out_seq=0 2023-03-09T14:46:36.595 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.584+0000 7f4d6cff9700 1 -- 172.21.15.38:0/3056641965 <== mds.2 v2:172.21.15.181:6838/3902263660 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x562e1c8881c0 con 0x562e1c9fd130 2023-03-09T14:46:36.596 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.584+0000 7f4d6cff9700 10 client.25346 handle_command_reply: tid=0 2023-03-09T14:46:36.596 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.584+0000 7f4d979c9b80 1 client.25346 shutdown 2023-03-09T14:46:36.596 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.584+0000 7f4d979c9b80 1 -- 172.21.15.38:0/3056641965 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562e1c995140 msgr2=0x562e1c99e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:36.596 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.584+0000 7f4d979c9b80 1 --2- 172.21.15.38:0/3056641965 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562e1c995140 0x562e1c99e280 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c0061e0 tx=0x7f4d7c009690 comp rx=0 tx=0).stop 2023-03-09T14:46:36.596 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.584+0000 7f4d979c9b80 1 -- 172.21.15.38:0/3056641965 shutdown_connections 2023-03-09T14:46:36.596 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.584+0000 7f4d979c9b80 1 --2- 172.21.15.38:0/3056641965 >> [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] conn(0x562e1c9fd130 0x562e1c9ff540 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.596 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.585+0000 7f4d979c9b80 1 --2- 172.21.15.38:0/3056641965 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e1c98efe0 0x562e1c99fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.596 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.585+0000 7f4d6cff9700 0 client.25346 ms_handle_reset on v2:172.21.15.181:6838/3902263660 2023-03-09T14:46:36.597 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.585+0000 7f4d979c9b80 1 --2- 172.21.15.38:0/3056641965 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562e1c995140 0x562e1c99e280 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.597 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.585+0000 7f4d979c9b80 1 --2- 172.21.15.38:0/3056641965 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562e1c994830 0x562e1c99db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.597 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.585+0000 7f4d979c9b80 1 -- 172.21.15.38:0/3056641965 >> 172.21.15.38:0/3056641965 conn(0x562e1c989a20 msgr2=0x562e1c9a42f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:36.597 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.585+0000 7f4d979c9b80 1 -- 172.21.15.38:0/3056641965 shutdown_connections 2023-03-09T14:46:36.597 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.585+0000 7f4d979c9b80 1 -- 172.21.15.38:0/3056641965 wait complete. 2023-03-09T14:46:36.597 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.585+0000 7f4d979c9b80 20 client.25346 trim_cache size 0 max 16384 2023-03-09T14:46:36.598 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2023-03-09T14:46:36.598 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.585+0000 7f4d8b840700 1 -- 172.21.15.38:0/3463816598 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f4d7009f4e0 msgr2=0x7f4d700a1990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:36.599 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.585+0000 7f4d8b840700 1 --2- 172.21.15.38:0/3463816598 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f4d7009f4e0 0x7f4d700a1990 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4d7c019bf0 tx=0x7f4d7c009bf0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.599 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.585+0000 7f4d8b840700 1 -- 172.21.15.38:0/3463816598 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d840ffaa0 msgr2=0x7f4d841965a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:36.599 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.585+0000 7f4d8b840700 1 --2- 172.21.15.38:0/3463816598 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d840ffaa0 0x7f4d841965a0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f4d7400e940 tx=0x7f4d7400ec50 comp rx=0 tx=0).stop 2023-03-09T14:46:36.599 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.585+0000 7f4d8b840700 1 -- 172.21.15.38:0/3463816598 shutdown_connections 2023-03-09T14:46:36.599 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.585+0000 7f4d8b840700 1 --2- 172.21.15.38:0/3463816598 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f4d7009f4e0 0x7f4d700a1990 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.599 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.585+0000 7f4d8b840700 1 --2- 172.21.15.38:0/3463816598 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4d84104140 0x7f4d8419ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.599 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.585+0000 7f4d8b840700 1 --2- 172.21.15.38:0/3463816598 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4d840ff100 0x7f4d84196060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.600 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.585+0000 7f4d8b840700 1 --2- 172.21.15.38:0/3463816598 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4d840ffaa0 0x7f4d841965a0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.600 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.585+0000 7f4d8b840700 1 -- 172.21.15.38:0/3463816598 >> 172.21.15.38:0/3463816598 conn(0x7f4d84075cc0 msgr2=0x7f4d84102c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:36.600 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.586+0000 7f4d8b840700 1 -- 172.21.15.38:0/3463816598 shutdown_connections 2023-03-09T14:46:36.600 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.586+0000 7f4d8b840700 1 -- 172.21.15.38:0/3463816598 wait complete. 2023-03-09T14:46:36.608 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-03-09T14:46:36.732 INFO:teuthology.orchestra.run.smithi038.stderr:8300000 of 50000000 tuples (16%) done (elapsed 11.87 s, remaining 59.64 s) 2023-03-09T14:46:36.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:36 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/2900950494' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-03-09T14:46:36.758 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.756+0000 7f6c9cbe7700 1 -- 172.21.15.38:0/4039630102 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c9810a720 msgr2=0x7f6c9810cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:36.759 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.756+0000 7f6c9cbe7700 1 --2- 172.21.15.38:0/4039630102 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c9810a720 0x7f6c9810cb00 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f6c9000b0a0 tx=0x7f6c9000b3b0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.759 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.756+0000 7f6c9cbe7700 1 -- 172.21.15.38:0/4039630102 shutdown_connections 2023-03-09T14:46:36.759 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.756+0000 7f6c9cbe7700 1 --2- 172.21.15.38:0/4039630102 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c9810a720 0x7f6c9810cb00 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.759 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.756+0000 7f6c9cbe7700 1 --2- 172.21.15.38:0/4039630102 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c980675f0 0x7f6c980ff300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.759 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.756+0000 7f6c9cbe7700 1 --2- 172.21.15.38:0/4039630102 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c980ff8d0 0x7f6c980ffd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.759 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.756+0000 7f6c9cbe7700 1 -- 172.21.15.38:0/4039630102 >> 172.21.15.38:0/4039630102 conn(0x7f6c980748f0 msgr2=0x7f6c98074cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:36.760 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.756+0000 7f6c9cbe7700 1 -- 172.21.15.38:0/4039630102 shutdown_connections 2023-03-09T14:46:36.760 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.756+0000 7f6c9cbe7700 1 -- 172.21.15.38:0/4039630102 wait complete. 2023-03-09T14:46:36.761 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.757+0000 7f6c9cbe7700 1 Processor -- start 2023-03-09T14:46:36.762 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.757+0000 7f6c9cbe7700 1 -- start start 2023-03-09T14:46:36.762 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.757+0000 7f6c9cbe7700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c980675f0 0x7f6c9819d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:36.762 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.757+0000 7f6c9cbe7700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c980ff8d0 0x7f6c9819d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:36.762 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.757+0000 7f6c9cbe7700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c9810a720 0x7f6c98198930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:36.762 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.757+0000 7f6c9cbe7700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c98199070 con 0x7f6c9810a720 2023-03-09T14:46:36.762 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.757+0000 7f6c9cbe7700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c981991e0 con 0x7f6c980675f0 2023-03-09T14:46:36.762 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.757+0000 7f6c9cbe7700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c981993a0 con 0x7f6c980ff8d0 2023-03-09T14:46:36.763 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.757+0000 7f6c96d9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c9810a720 0x7f6c98198930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:36.763 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.758+0000 7f6c95d9b700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c980ff8d0 0x7f6c9819d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:36.763 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.758+0000 7f6c96d9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c9810a720 0x7f6c98198930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39338/0 (socket says 172.21.15.38:39338) 2023-03-09T14:46:36.763 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.758+0000 7f6c95d9b700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c980ff8d0 0x7f6c9819d780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.38:47034/0 (socket says 172.21.15.38:47034) 2023-03-09T14:46:36.763 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.758+0000 7f6c96d9d700 1 -- 172.21.15.38:0/3084613543 learned_addr learned my addr 172.21.15.38:0/3084613543 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:46:36.763 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.758+0000 7f6c9659c700 1 --2- 172.21.15.38:0/3084613543 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c980675f0 0x7f6c9819d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:36.763 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.758+0000 7f6c96d9d700 1 -- 172.21.15.38:0/3084613543 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c980ff8d0 msgr2=0x7f6c9819d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:36.763 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.758+0000 7f6c96d9d700 1 --2- 172.21.15.38:0/3084613543 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c980ff8d0 0x7f6c9819d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.764 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.758+0000 7f6c96d9d700 1 -- 172.21.15.38:0/3084613543 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c980675f0 msgr2=0x7f6c9819d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:36.764 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.758+0000 7f6c96d9d700 1 --2- 172.21.15.38:0/3084613543 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c980675f0 0x7f6c9819d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.764 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.758+0000 7f6c96d9d700 1 -- 172.21.15.38:0/3084613543 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c90009d00 con 0x7f6c9810a720 2023-03-09T14:46:36.764 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.758+0000 7f6c9659c700 1 --2- 172.21.15.38:0/3084613543 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c980675f0 0x7f6c9819d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:46:36.764 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.758+0000 7f6c96d9d700 1 --2- 172.21.15.38:0/3084613543 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c9810a720 0x7f6c98198930 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f6c9000b780 tx=0x7f6c9000bd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:36.764 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.758+0000 7f6c877fe700 1 -- 172.21.15.38:0/3084613543 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c9000e040 con 0x7f6c9810a720 2023-03-09T14:46:36.764 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.758+0000 7f6c877fe700 1 -- 172.21.15.38:0/3084613543 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6c90004e60 con 0x7f6c9810a720 2023-03-09T14:46:36.765 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.758+0000 7f6c9cbe7700 1 -- 172.21.15.38:0/3084613543 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c98100ce0 con 0x7f6c9810a720 2023-03-09T14:46:36.765 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.758+0000 7f6c877fe700 1 -- 172.21.15.38:0/3084613543 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c900052a0 con 0x7f6c9810a720 2023-03-09T14:46:36.765 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.758+0000 7f6c9cbe7700 1 -- 172.21.15.38:0/3084613543 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c98100b50 con 0x7f6c9810a720 2023-03-09T14:46:36.767 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.761+0000 7f6c877fe700 1 -- 172.21.15.38:0/3084613543 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 23) v1 ==== 133335+0+0 (secure 0 0 0) 0x7f6c90005400 con 0x7f6c9810a720 2023-03-09T14:46:36.767 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.762+0000 7f6c877fe700 1 --2- 172.21.15.38:0/3084613543 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f6c8009f4e0 0x7f6c800a1990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:36.767 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.762+0000 7f6c877fe700 1 -- 172.21.15.38:0/3084613543 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9278+0+0 (secure 0 0 0) 0x7f6c900cba80 con 0x7f6c9810a720 2023-03-09T14:46:36.767 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.762+0000 7f6c9659c700 1 --2- 172.21.15.38:0/3084613543 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f6c8009f4e0 0x7f6c800a1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:36.768 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.762+0000 7f6ca8d70b80 1 Processor -- start 2023-03-09T14:46:36.768 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.763+0000 7f6ca8d70b80 1 -- start start 2023-03-09T14:46:36.768 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.763+0000 7f6c9659c700 1 --2- 172.21.15.38:0/3084613543 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f6c8009f4e0 0x7f6c800a1990 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6c88005950 tx=0x7f6c880058e0 comp rx=0 tx=0).ready entity=mgr.15558 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:36.768 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.763+0000 7f6ca8d70b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564196de8830 0x564196de8c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:36.768 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.763+0000 7f6ca8d70b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564196de9140 0x564196de2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:36.768 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.763+0000 7f6ca8d70b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564196de2fe0 0x564196de3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:36.768 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.763+0000 7f6ca8d70b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x564196cccda0 con 0x564196de2fe0 2023-03-09T14:46:36.769 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.763+0000 7f6ca8d70b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x564196d3b780 con 0x564196de8830 2023-03-09T14:46:36.769 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.763+0000 7f6ca8d70b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x564196d34cb0 con 0x564196de9140 2023-03-09T14:46:36.769 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.763+0000 7f6c96d9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564196de2fe0 0x564196de3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:36.769 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.763+0000 7f6c96d9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564196de2fe0 0x564196de3410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39346/0 (socket says 172.21.15.38:39346) 2023-03-09T14:46:36.769 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.763+0000 7f6c96d9d700 1 -- 172.21.15.38:0/3570291858 learned_addr learned my addr 172.21.15.38:0/3570291858 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:46:36.769 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.763+0000 7f6c95d9b700 1 --2- 172.21.15.38:0/3570291858 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564196de9140 0x564196de2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:36.769 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.763+0000 7f6c9659c700 1 --2- 172.21.15.38:0/3570291858 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564196de8830 0x564196de8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:36.769 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.764+0000 7f6c96d9d700 1 -- 172.21.15.38:0/3570291858 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564196de9140 msgr2=0x564196de2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:46:36.770 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.764+0000 7f6c96d9d700 1 --2- 172.21.15.38:0/3570291858 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564196de9140 0x564196de2aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.770 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.764+0000 7f6c96d9d700 1 -- 172.21.15.38:0/3570291858 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564196de8830 msgr2=0x564196de8c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:46:36.770 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.764+0000 7f6c96d9d700 1 --2- 172.21.15.38:0/3570291858 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564196de8830 0x564196de8c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.770 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.764+0000 7f6c96d9d700 1 -- 172.21.15.38:0/3570291858 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564196cfb190 con 0x564196de2fe0 2023-03-09T14:46:36.770 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.764+0000 7f6c95d9b700 1 --2- 172.21.15.38:0/3570291858 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564196de9140 0x564196de2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:46:36.770 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.764+0000 7f6c9659c700 1 --2- 172.21.15.38:0/3570291858 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564196de8830 0x564196de8c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:46:36.770 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.764+0000 7f6c96d9d700 1 --2- 172.21.15.38:0/3570291858 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564196de2fe0 0x564196de3410 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f6c90009fd0 tx=0x7f6c90095a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b662d5eb4c21ed0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:36.771 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.764+0000 7f6c84ff9700 1 -- 172.21.15.38:0/3570291858 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c90099010 con 0x564196de2fe0 2023-03-09T14:46:36.771 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.764+0000 7f6c84ff9700 1 -- 172.21.15.38:0/3570291858 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6c90011e60 con 0x564196de2fe0 2023-03-09T14:46:36.771 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.764+0000 7f6c84ff9700 1 -- 172.21.15.38:0/3570291858 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c90099b70 con 0x564196de2fe0 2023-03-09T14:46:36.771 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.765+0000 7f6ca8d70b80 1 -- 172.21.15.38:0/3570291858 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564196de2fe0 msgr2=0x564196de3410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:36.771 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.765+0000 7f6ca8d70b80 1 --2- 172.21.15.38:0/3570291858 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564196de2fe0 0x564196de3410 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f6c90009fd0 tx=0x7f6c90095a70 comp rx=0 tx=0).stop 2023-03-09T14:46:36.771 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.765+0000 7f6ca8d70b80 1 -- 172.21.15.38:0/3570291858 shutdown_connections 2023-03-09T14:46:36.771 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.765+0000 7f6ca8d70b80 1 --2- 172.21.15.38:0/3570291858 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564196de2fe0 0x564196de3410 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.771 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.765+0000 7f6ca8d70b80 1 --2- 172.21.15.38:0/3570291858 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564196de8830 0x564196de8c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.772 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.765+0000 7f6ca8d70b80 1 --2- 172.21.15.38:0/3570291858 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564196de9140 0x564196de2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.772 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.765+0000 7f6ca8d70b80 1 -- 172.21.15.38:0/3570291858 >> 172.21.15.38:0/3570291858 conn(0x564196ddda20 msgr2=0x564196ddddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:36.772 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.765+0000 7f6ca8d70b80 1 -- 172.21.15.38:0/3570291858 shutdown_connections 2023-03-09T14:46:36.772 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.765+0000 7f6ca8d70b80 1 -- 172.21.15.38:0/3570291858 wait complete. 2023-03-09T14:46:36.775 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.765+0000 7f6ca8d70b80 1 -- start start 2023-03-09T14:46:36.775 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.765+0000 7f6ca8d70b80 1 Processor -- start 2023-03-09T14:46:36.775 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.766+0000 7f6ca8d70b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564196de2fe0 0x564196def130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:36.775 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.766+0000 7f6ca8d70b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564196de8830 0x564196dec3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:36.775 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.766+0000 7f6ca8d70b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564196de9140 0x564196deca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:36.776 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.766+0000 7f6ca8d70b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x564196bb8920 con 0x564196de2fe0 2023-03-09T14:46:36.776 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.766+0000 7f6ca8d70b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x564196ded1c0 con 0x564196de9140 2023-03-09T14:46:36.776 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.766+0000 7f6ca8d70b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x564196df8a00 con 0x564196de8830 2023-03-09T14:46:36.776 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.766+0000 7f6ca8d70b80 10 client.0 _ll_register_callbacks cb 0x564196cda520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:46:36.776 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.766+0000 7f6c9659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564196de2fe0 0x564196def130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:36.776 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.766+0000 7f6c9659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564196de2fe0 0x564196def130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39348/0 (socket says 172.21.15.38:39348) 2023-03-09T14:46:36.776 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.766+0000 7f6c9659c700 1 -- 172.21.15.38:0/3494234969 learned_addr learned my addr 172.21.15.38:0/3494234969 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:46:36.776 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.766+0000 7f6c96d9d700 1 --2- 172.21.15.38:0/3494234969 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564196de9140 0x564196deca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:36.777 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.766+0000 7f6c95d9b700 1 --2- 172.21.15.38:0/3494234969 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564196de8830 0x564196dec3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:36.777 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.766+0000 7f6c9659c700 1 -- 172.21.15.38:0/3494234969 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564196de8830 msgr2=0x564196dec3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:46:36.777 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.766+0000 7f6c9659c700 1 --2- 172.21.15.38:0/3494234969 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564196de8830 0x564196dec3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.777 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.766+0000 7f6c9659c700 1 -- 172.21.15.38:0/3494234969 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564196de9140 msgr2=0x564196deca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:46:36.777 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.766+0000 7f6c9659c700 1 --2- 172.21.15.38:0/3494234969 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564196de9140 0x564196deca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.777 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.766+0000 7f6c9659c700 1 -- 172.21.15.38:0/3494234969 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c8c0098e0 con 0x564196de2fe0 2023-03-09T14:46:36.777 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.767+0000 7f6c9659c700 1 --2- 172.21.15.38:0/3494234969 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564196de2fe0 0x564196def130 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f6c8800ffe0 tx=0x7f6c88004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da268b19c855f3c6 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:36.778 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.767+0000 7f6c7dffb700 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2023-03-09T14:46:36.778 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.767+0000 7f6c7dffb700 1 -- 172.21.15.38:0/3494234969 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c88017040 con 0x564196de2fe0 2023-03-09T14:46:36.778 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.767+0000 7f6c7dffb700 1 -- 172.21.15.38:0/3494234969 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9278+0+0 (secure 0 0 0) 0x7f6c88003cd0 con 0x564196de2fe0 2023-03-09T14:46:36.778 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.767+0000 7f6ca8d70b80 1 -- 172.21.15.38:0/3494234969 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564196d02190 con 0x564196de2fe0 2023-03-09T14:46:36.778 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.767+0000 7f6c7dffb700 1 -- 172.21.15.38:0/3494234969 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6c88003e90 con 0x564196de2fe0 2023-03-09T14:46:36.778 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.767+0000 7f6ca8d70b80 10 client.15687 fetch_fsmap learned FSMap version 23 2023-03-09T14:46:36.778 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.767+0000 7f6ca8d70b80 1 -- 172.21.15.38:0/3494234969 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x7f6c98100b50 con 0x564196de2fe0 2023-03-09T14:46:36.778 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.767+0000 7f6c7dffb700 1 -- 172.21.15.38:0/3494234969 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x564196d02190 con 0x564196de2fe0 2023-03-09T14:46:36.779 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.767+0000 7f6c7dffb700 1 -- 172.21.15.38:0/3494234969 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f6c8801d050 con 0x564196de2fe0 2023-03-09T14:46:36.779 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.767+0000 7f6ca8d70b80 10 client.15687 fetch_fsmap finished waiting for FSMap version 23 2023-03-09T14:46:36.779 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.767+0000 7f6ca8d70b80 10 client.15687 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2023-03-09T14:46:36.779 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.767+0000 7f6ca8d70b80 20 client.15687 populate_metadata read hostname 'smithi038' 2023-03-09T14:46:36.779 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.767+0000 7f6ca8d70b80 1 --2- 172.21.15.38:0/3494234969 >> [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] conn(0x564196dfcb90 0x564196dfefa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:36.779 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.767+0000 7f6ca8d70b80 4 client.15687 mds_command: new command op to 24494 tid=0[{"prefix": "get_command_descriptions"}] 2023-03-09T14:46:36.779 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.767+0000 7f6ca8d70b80 1 -- 172.21.15.38:0/3494234969 --> [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564196bb8f60 con 0x564196dfcb90 2023-03-09T14:46:36.779 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.768+0000 7f6c95d9b700 1 --2- 172.21.15.38:0/3494234969 >> [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] conn(0x564196dfcb90 0x564196dfefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:36.780 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.768+0000 7f6c95d9b700 1 --2- 172.21.15.38:0/3494234969 >> [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] conn(0x564196dfcb90 0x564196dfefa0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=bb658a63212dfa6 server_cookie=fe838206dd92cdd in_seq=0 out_seq=0 2023-03-09T14:46:36.780 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.768+0000 7f6c7dffb700 10 client.15687 ms_handle_connect on v2:172.21.15.122:6836/2210214596 2023-03-09T14:46:36.780 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.769+0000 7f6c7dffb700 1 -- 172.21.15.38:0/3494234969 <== mds.1 v2:172.21.15.122:6836/2210214596 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18278 (crc 0 0 0) 0x564196bb8f60 con 0x564196dfcb90 2023-03-09T14:46:36.780 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.769+0000 7f6c7dffb700 10 client.15687 handle_command_reply: tid=0 2023-03-09T14:46:36.780 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.769+0000 7f6ca8d70b80 1 client.15687 shutdown 2023-03-09T14:46:36.780 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.769+0000 7f6ca8d70b80 1 -- 172.21.15.38:0/3494234969 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564196de2fe0 msgr2=0x564196def130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:36.780 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.769+0000 7f6ca8d70b80 1 --2- 172.21.15.38:0/3494234969 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564196de2fe0 0x564196def130 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f6c8800ffe0 tx=0x7f6c88004750 comp rx=0 tx=0).stop 2023-03-09T14:46:36.781 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.769+0000 7f6ca8d70b80 1 -- 172.21.15.38:0/3494234969 shutdown_connections 2023-03-09T14:46:36.781 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.769+0000 7f6ca8d70b80 1 --2- 172.21.15.38:0/3494234969 >> [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] conn(0x564196dfcb90 0x564196dfefa0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.781 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.769+0000 7f6ca8d70b80 1 --2- 172.21.15.38:0/3494234969 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564196de2fe0 0x564196def130 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.781 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.769+0000 7f6c7dffb700 0 client.15687 ms_handle_reset on v2:172.21.15.122:6836/2210214596 2023-03-09T14:46:36.781 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.769+0000 7f6ca8d70b80 1 --2- 172.21.15.38:0/3494234969 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564196de9140 0x564196deca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.781 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.769+0000 7f6ca8d70b80 1 --2- 172.21.15.38:0/3494234969 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564196de8830 0x564196dec3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.781 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.769+0000 7f6ca8d70b80 1 -- 172.21.15.38:0/3494234969 >> 172.21.15.38:0/3494234969 conn(0x564196ddda20 msgr2=0x564196ddcc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:36.782 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.769+0000 7f6ca8d70b80 1 -- 172.21.15.38:0/3494234969 shutdown_connections 2023-03-09T14:46:36.782 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.769+0000 7f6ca8d70b80 1 -- 172.21.15.38:0/3494234969 wait complete. 2023-03-09T14:46:36.782 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.769+0000 7f6ca8d70b80 20 client.15687 trim_cache size 0 max 16384 2023-03-09T14:46:36.785 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.782+0000 7f6ca8d70b80 1 Processor -- start 2023-03-09T14:46:36.785 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.782+0000 7f6ca8d70b80 1 -- start start 2023-03-09T14:46:36.785 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.782+0000 7f6ca8d70b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564196ddda20 0x564196deecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:36.785 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.782+0000 7f6ca8d70b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564196de2fe0 0x564196de6530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:36.786 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.782+0000 7f6ca8d70b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564196de8830 0x564196de6a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:36.786 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.782+0000 7f6ca8d70b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x564196de3cc0 con 0x564196de8830 2023-03-09T14:46:36.786 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.782+0000 7f6ca8d70b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x564196ddd760 con 0x564196de2fe0 2023-03-09T14:46:36.786 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.782+0000 7f6ca8d70b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x564196de8460 con 0x564196ddda20 2023-03-09T14:46:36.786 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.783+0000 7f6c96d9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564196de8830 0x564196de6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:36.786 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.783+0000 7f6c96d9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564196de8830 0x564196de6a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39360/0 (socket says 172.21.15.38:39360) 2023-03-09T14:46:36.786 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.783+0000 7f6c96d9d700 1 -- 172.21.15.38:0/1189653663 learned_addr learned my addr 172.21.15.38:0/1189653663 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:46:36.786 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.783+0000 7f6c9659c700 1 --2- 172.21.15.38:0/1189653663 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564196ddda20 0x564196deecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:36.787 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.783+0000 7f6c95d9b700 1 --2- 172.21.15.38:0/1189653663 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564196de2fe0 0x564196de6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:36.787 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.783+0000 7f6c96d9d700 1 -- 172.21.15.38:0/1189653663 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564196ddda20 msgr2=0x564196deecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:46:36.787 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.783+0000 7f6c96d9d700 1 --2- 172.21.15.38:0/1189653663 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564196ddda20 0x564196deecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.787 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.783+0000 7f6c96d9d700 1 -- 172.21.15.38:0/1189653663 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564196de2fe0 msgr2=0x564196de6530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:46:36.787 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.783+0000 7f6c96d9d700 1 --2- 172.21.15.38:0/1189653663 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564196de2fe0 0x564196de6530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.787 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.783+0000 7f6c96d9d700 1 -- 172.21.15.38:0/1189653663 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564196e03bd0 con 0x564196de8830 2023-03-09T14:46:36.787 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.783+0000 7f6c96d9d700 1 --2- 172.21.15.38:0/1189653663 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564196de8830 0x564196de6a70 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f6c90099640 tx=0x7f6c9001dcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d5ff673b8db09ba server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:36.790 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.783+0000 7f6c857fa700 1 -- 172.21.15.38:0/1189653663 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c90099010 con 0x564196de8830 2023-03-09T14:46:36.791 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.783+0000 7f6c857fa700 1 -- 172.21.15.38:0/1189653663 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6c90011e60 con 0x564196de8830 2023-03-09T14:46:36.791 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.783+0000 7f6c857fa700 1 -- 172.21.15.38:0/1189653663 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c90098ba0 con 0x564196de8830 2023-03-09T14:46:36.791 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.784+0000 7f6ca8d70b80 1 -- 172.21.15.38:0/1189653663 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564196de8830 msgr2=0x564196de6a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:36.791 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.784+0000 7f6ca8d70b80 1 --2- 172.21.15.38:0/1189653663 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564196de8830 0x564196de6a70 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f6c90099640 tx=0x7f6c9001dcd0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.791 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.784+0000 7f6ca8d70b80 1 -- 172.21.15.38:0/1189653663 shutdown_connections 2023-03-09T14:46:36.791 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.784+0000 7f6ca8d70b80 1 --2- 172.21.15.38:0/1189653663 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564196de8830 0x564196de6a70 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.791 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.784+0000 7f6ca8d70b80 1 --2- 172.21.15.38:0/1189653663 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564196ddda20 0x564196deecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.792 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.784+0000 7f6ca8d70b80 1 --2- 172.21.15.38:0/1189653663 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564196de2fe0 0x564196de6530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.792 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.784+0000 7f6ca8d70b80 1 -- 172.21.15.38:0/1189653663 >> 172.21.15.38:0/1189653663 conn(0x564196de3710 msgr2=0x564196ddcc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:36.792 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.784+0000 7f6ca8d70b80 1 -- 172.21.15.38:0/1189653663 shutdown_connections 2023-03-09T14:46:36.792 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.784+0000 7f6ca8d70b80 1 -- 172.21.15.38:0/1189653663 wait complete. 2023-03-09T14:46:36.792 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.784+0000 7f6ca8d70b80 1 -- start start 2023-03-09T14:46:36.792 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.784+0000 7f6ca8d70b80 1 Processor -- start 2023-03-09T14:46:36.792 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.785+0000 7f6ca8d70b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564196ddda20 0x564196de4890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:36.793 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.785+0000 7f6ca8d70b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564196de2fe0 0x564196de4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:36.793 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.785+0000 7f6ca8d70b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564196de8830 0x564196de5310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:36.793 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.785+0000 7f6ca8d70b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x564196df3fc0 con 0x564196ddda20 2023-03-09T14:46:36.793 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.785+0000 7f6ca8d70b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x564196e50a60 con 0x564196de8830 2023-03-09T14:46:36.793 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.785+0000 7f6ca8d70b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x564196e50c20 con 0x564196de2fe0 2023-03-09T14:46:36.793 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.785+0000 7f6ca8d70b80 10 client.0 _ll_register_callbacks cb 0x564196cda520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:46:36.794 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.785+0000 7f6c96d9d700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564196de8830 0x564196de5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:36.794 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.785+0000 7f6c9659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564196ddda20 0x564196de4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:36.794 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.785+0000 7f6c96d9d700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564196de8830 0x564196de5310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.38:48556/0 (socket says 172.21.15.38:48556) 2023-03-09T14:46:36.794 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.785+0000 7f6c9659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564196ddda20 0x564196de4890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39362/0 (socket says 172.21.15.38:39362) 2023-03-09T14:46:36.794 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.785+0000 7f6c96d9d700 1 -- 172.21.15.38:0/1665509419 learned_addr learned my addr 172.21.15.38:0/1665509419 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:46:36.794 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.785+0000 7f6c95d9b700 1 --2- 172.21.15.38:0/1665509419 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564196de2fe0 0x564196de4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:36.794 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.785+0000 7f6c9659c700 1 -- 172.21.15.38:0/1665509419 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564196de2fe0 msgr2=0x564196de4dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:46:36.795 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.785+0000 7f6c9659c700 1 --2- 172.21.15.38:0/1665509419 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564196de2fe0 0x564196de4dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.795 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.785+0000 7f6c9659c700 1 -- 172.21.15.38:0/1665509419 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564196de8830 msgr2=0x564196de5310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:46:36.795 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.785+0000 7f6c9659c700 1 --2- 172.21.15.38:0/1665509419 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564196de8830 0x564196de5310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.795 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.785+0000 7f6c9659c700 1 -- 172.21.15.38:0/1665509419 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564196cfb190 con 0x564196ddda20 2023-03-09T14:46:36.795 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.785+0000 7f6c9659c700 1 --2- 172.21.15.38:0/1665509419 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564196ddda20 0x564196de4890 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f6c880046c0 tx=0x7f6c88011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20d0465a103508f9 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:36.795 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.785+0000 7f6c7dffb700 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2023-03-09T14:46:36.795 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.786+0000 7f6c7dffb700 1 -- 172.21.15.38:0/1665509419 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c88003680 con 0x564196ddda20 2023-03-09T14:46:36.796 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.786+0000 7f6c7dffb700 1 -- 172.21.15.38:0/1665509419 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9278+0+0 (secure 0 0 0) 0x7f6c8800b4a0 con 0x564196ddda20 2023-03-09T14:46:36.796 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.786+0000 7f6ca8d70b80 1 -- 172.21.15.38:0/1665509419 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564196cf21b0 con 0x564196ddda20 2023-03-09T14:46:36.796 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.786+0000 7f6c7dffb700 1 -- 172.21.15.38:0/1665509419 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6c88028b90 con 0x564196ddda20 2023-03-09T14:46:36.796 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.786+0000 7f6ca8d70b80 10 client.15693 fetch_fsmap learned FSMap version 23 2023-03-09T14:46:36.796 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.786+0000 7f6ca8d70b80 1 -- 172.21.15.38:0/1665509419 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x564196b62de0 con 0x564196ddda20 2023-03-09T14:46:36.796 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.786+0000 7f6c7dffb700 1 -- 172.21.15.38:0/1665509419 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x564196cf21b0 con 0x564196ddda20 2023-03-09T14:46:36.796 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.786+0000 7f6c7dffb700 1 -- 172.21.15.38:0/1665509419 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f6c88004030 con 0x564196ddda20 2023-03-09T14:46:36.797 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.786+0000 7f6ca8d70b80 10 client.15693 fetch_fsmap finished waiting for FSMap version 23 2023-03-09T14:46:36.797 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.786+0000 7f6ca8d70b80 10 client.15693 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2023-03-09T14:46:36.797 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.786+0000 7f6ca8d70b80 20 client.15693 populate_metadata read hostname 'smithi038' 2023-03-09T14:46:36.797 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.786+0000 7f6ca8d70b80 1 --2- 172.21.15.38:0/1665509419 >> [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] conn(0x564196de9140 0x564196e54370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:36.797 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.786+0000 7f6ca8d70b80 4 client.15693 mds_command: new command op to 24494 tid=0[{"prefix": "damage ls"}] 2023-03-09T14:46:36.797 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.786+0000 7f6ca8d70b80 1 -- 172.21.15.38:0/1665509419 --> [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x564196cdc1c0 con 0x564196de9140 2023-03-09T14:46:36.797 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.786+0000 7f6c95d9b700 1 --2- 172.21.15.38:0/1665509419 >> [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] conn(0x564196de9140 0x564196e54370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:36.798 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.787+0000 7f6c7dffb700 10 client.15693 ms_handle_connect on v2:172.21.15.122:6836/2210214596 2023-03-09T14:46:36.798 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.787+0000 7f6c95d9b700 1 --2- 172.21.15.38:0/1665509419 >> [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] conn(0x564196de9140 0x564196e54370 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2800092194af7483 server_cookie=4d29bd8aaa0fff48 in_seq=0 out_seq=0 2023-03-09T14:46:36.798 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.787+0000 7f6c7dffb700 1 -- 172.21.15.38:0/1665509419 <== mds.1 v2:172.21.15.122:6836/2210214596 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x564196cdc1c0 con 0x564196de9140 2023-03-09T14:46:36.798 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.787+0000 7f6c7dffb700 10 client.15693 handle_command_reply: tid=0 2023-03-09T14:46:36.798 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.787+0000 7f6ca8d70b80 1 client.15693 shutdown 2023-03-09T14:46:36.798 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.787+0000 7f6ca8d70b80 1 -- 172.21.15.38:0/1665509419 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564196ddda20 msgr2=0x564196de4890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:36.798 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.787+0000 7f6ca8d70b80 1 --2- 172.21.15.38:0/1665509419 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564196ddda20 0x564196de4890 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f6c880046c0 tx=0x7f6c88011a50 comp rx=0 tx=0).stop 2023-03-09T14:46:36.798 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.787+0000 7f6ca8d70b80 1 -- 172.21.15.38:0/1665509419 shutdown_connections 2023-03-09T14:46:36.799 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.787+0000 7f6ca8d70b80 1 --2- 172.21.15.38:0/1665509419 >> [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] conn(0x564196de9140 0x564196e54370 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.799 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.787+0000 7f6ca8d70b80 1 --2- 172.21.15.38:0/1665509419 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564196ddda20 0x564196de4890 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.799 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.787+0000 7f6ca8d70b80 1 --2- 172.21.15.38:0/1665509419 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564196de8830 0x564196de5310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.799 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.787+0000 7f6ca8d70b80 1 --2- 172.21.15.38:0/1665509419 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564196de2fe0 0x564196de4dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.799 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.787+0000 7f6ca8d70b80 1 -- 172.21.15.38:0/1665509419 >> 172.21.15.38:0/1665509419 conn(0x564196de3710 msgr2=0x564196ddcc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:36.799 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.787+0000 7f6c7dffb700 0 client.15693 ms_handle_reset on v2:172.21.15.122:6836/2210214596 2023-03-09T14:46:36.799 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.787+0000 7f6ca8d70b80 1 -- 172.21.15.38:0/1665509419 shutdown_connections 2023-03-09T14:46:36.800 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.787+0000 7f6ca8d70b80 1 -- 172.21.15.38:0/1665509419 wait complete. 2023-03-09T14:46:36.800 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.787+0000 7f6ca8d70b80 20 client.15693 trim_cache size 0 max 16384 2023-03-09T14:46:36.800 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2023-03-09T14:46:36.801 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.788+0000 7f6c9cbe7700 1 -- 172.21.15.38:0/3084613543 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f6c8009f4e0 msgr2=0x7f6c800a1990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:36.801 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.788+0000 7f6c9cbe7700 1 --2- 172.21.15.38:0/3084613543 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f6c8009f4e0 0x7f6c800a1990 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6c88005950 tx=0x7f6c880058e0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.801 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.788+0000 7f6c9cbe7700 1 -- 172.21.15.38:0/3084613543 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c9810a720 msgr2=0x7f6c98198930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:36.801 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.788+0000 7f6c9cbe7700 1 --2- 172.21.15.38:0/3084613543 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c9810a720 0x7f6c98198930 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f6c9000b780 tx=0x7f6c9000bd60 comp rx=0 tx=0).stop 2023-03-09T14:46:36.801 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.788+0000 7f6c9cbe7700 1 -- 172.21.15.38:0/3084613543 shutdown_connections 2023-03-09T14:46:36.801 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.788+0000 7f6c9cbe7700 1 --2- 172.21.15.38:0/3084613543 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f6c8009f4e0 0x7f6c800a1990 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.802 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.788+0000 7f6c9cbe7700 1 --2- 172.21.15.38:0/3084613543 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6c9810a720 0x7f6c98198930 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.802 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.788+0000 7f6c9cbe7700 1 --2- 172.21.15.38:0/3084613543 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6c980675f0 0x7f6c9819d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.802 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.788+0000 7f6c9cbe7700 1 --2- 172.21.15.38:0/3084613543 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6c980ff8d0 0x7f6c9819d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.802 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.788+0000 7f6c9cbe7700 1 -- 172.21.15.38:0/3084613543 >> 172.21.15.38:0/3084613543 conn(0x7f6c980748f0 msgr2=0x7f6c98101590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:36.802 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.788+0000 7f6c9cbe7700 1 -- 172.21.15.38:0/3084613543 shutdown_connections 2023-03-09T14:46:36.802 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.788+0000 7f6c9cbe7700 1 -- 172.21.15.38:0/3084613543 wait complete. 2023-03-09T14:46:36.803 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-03-09T14:46:36.937 INFO:teuthology.orchestra.run.smithi038.stderr:8400000 of 50000000 tuples (16%) done (elapsed 12.07 s, remaining 59.79 s) 2023-03-09T14:46:36.954 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.951+0000 7f4108ac2700 1 -- 172.21.15.38:0/1333856557 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f41040ff840 msgr2=0x7f41040ffcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:36.954 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.951+0000 7f4108ac2700 1 --2- 172.21.15.38:0/1333856557 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f41040ff840 0x7f41040ffcb0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f40f4009990 tx=0x7f40f4009ca0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.954 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.951+0000 7f4108ac2700 1 -- 172.21.15.38:0/1333856557 shutdown_connections 2023-03-09T14:46:36.955 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.951+0000 7f4108ac2700 1 --2- 172.21.15.38:0/1333856557 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f410410a710 0x7f410410caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.955 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.951+0000 7f4108ac2700 1 --2- 172.21.15.38:0/1333856557 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4104075f80 0x7f41040ff300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.955 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.951+0000 7f4108ac2700 1 --2- 172.21.15.38:0/1333856557 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f41040ff840 0x7f41040ffcb0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.955 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.951+0000 7f4108ac2700 1 -- 172.21.15.38:0/1333856557 >> 172.21.15.38:0/1333856557 conn(0x7f41040748f0 msgr2=0x7f4104074cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:36.955 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.952+0000 7f4108ac2700 1 -- 172.21.15.38:0/1333856557 shutdown_connections 2023-03-09T14:46:36.955 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.952+0000 7f4108ac2700 1 -- 172.21.15.38:0/1333856557 wait complete. 2023-03-09T14:46:36.957 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.952+0000 7f4108ac2700 1 Processor -- start 2023-03-09T14:46:36.957 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.953+0000 7f4108ac2700 1 -- start start 2023-03-09T14:46:36.957 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.953+0000 7f4108ac2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4104075f80 0x7f4104196190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:36.958 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.953+0000 7f4108ac2700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f41040ff840 0x7f41041966d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:36.958 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.953+0000 7f4108ac2700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f410410a710 0x7f410419ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:36.958 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.953+0000 7f4108ac2700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f410419b170 con 0x7f4104075f80 2023-03-09T14:46:36.958 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.953+0000 7f4108ac2700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f410419b2e0 con 0x7f410410a710 2023-03-09T14:46:36.958 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.953+0000 7f4108ac2700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f410419b4a0 con 0x7f41040ff840 2023-03-09T14:46:36.958 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.953+0000 7f410259c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4104075f80 0x7f4104196190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:36.958 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.953+0000 7f410259c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4104075f80 0x7f4104196190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39376/0 (socket says 172.21.15.38:39376) 2023-03-09T14:46:36.958 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.953+0000 7f410259c700 1 -- 172.21.15.38:0/1532218431 learned_addr learned my addr 172.21.15.38:0/1532218431 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:46:36.959 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.953+0000 7f4101d9b700 1 --2- 172.21.15.38:0/1532218431 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f41040ff840 0x7f41041966d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:36.959 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.953+0000 7f4102d9d700 1 --2- 172.21.15.38:0/1532218431 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f410410a710 0x7f410419ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:36.959 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.953+0000 7f410259c700 1 -- 172.21.15.38:0/1532218431 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f41040ff840 msgr2=0x7f41041966d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:36.959 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.953+0000 7f410259c700 1 --2- 172.21.15.38:0/1532218431 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f41040ff840 0x7f41041966d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.959 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.953+0000 7f410259c700 1 -- 172.21.15.38:0/1532218431 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f410410a710 msgr2=0x7f410419ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:36.959 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.953+0000 7f410259c700 1 --2- 172.21.15.38:0/1532218431 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f410410a710 0x7f410419ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.960 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.953+0000 7f410259c700 1 -- 172.21.15.38:0/1532218431 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40f40095f0 con 0x7f4104075f80 2023-03-09T14:46:36.960 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.954+0000 7f410259c700 1 --2- 172.21.15.38:0/1532218431 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4104075f80 0x7f4104196190 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f40f800e9a0 tx=0x7f40f800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:36.960 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.954+0000 7f40f37fe700 1 -- 172.21.15.38:0/1532218431 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f8009790 con 0x7f4104075f80 2023-03-09T14:46:36.960 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.954+0000 7f4108ac2700 1 -- 172.21.15.38:0/1532218431 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4104110100 con 0x7f4104075f80 2023-03-09T14:46:36.961 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.954+0000 7f40f37fe700 1 -- 172.21.15.38:0/1532218431 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f40f800cbd0 con 0x7f4104075f80 2023-03-09T14:46:36.961 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.954+0000 7f40f37fe700 1 -- 172.21.15.38:0/1532218431 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f8018680 con 0x7f4104075f80 2023-03-09T14:46:36.961 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.954+0000 7f4108ac2700 1 -- 172.21.15.38:0/1532218431 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f410410ff70 con 0x7f4104075f80 2023-03-09T14:46:36.968 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.956+0000 7f40f37fe700 1 -- 172.21.15.38:0/1532218431 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 23) v1 ==== 133335+0+0 (secure 0 0 0) 0x7f40f800c040 con 0x7f4104075f80 2023-03-09T14:46:36.968 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.957+0000 7f40f37fe700 1 --2- 172.21.15.38:0/1532218431 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f40ec09f4e0 0x7f40ec0a1990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:36.968 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.957+0000 7f40f37fe700 1 -- 172.21.15.38:0/1532218431 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9278+0+0 (secure 0 0 0) 0x7f40f8014070 con 0x7f4104075f80 2023-03-09T14:46:36.968 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.957+0000 7f4101d9b700 1 --2- 172.21.15.38:0/1532218431 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f40ec09f4e0 0x7f40ec0a1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:36.968 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.957+0000 7f4101d9b700 1 --2- 172.21.15.38:0/1532218431 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f40ec09f4e0 0x7f40ec0a1990 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f41041977a0 tx=0x7f40f4005fd0 comp rx=0 tx=0).ready entity=mgr.15558 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:36.969 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.958+0000 7f4114c4bb80 1 Processor -- start 2023-03-09T14:46:36.969 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.958+0000 7f4114c4bb80 1 -- start start 2023-03-09T14:46:36.969 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.958+0000 7f4114c4bb80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562191917830 0x562191917c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:36.969 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.958+0000 7f4114c4bb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562191918140 0x562191911aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:36.969 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.958+0000 7f4114c4bb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562191911fe0 0x562191912410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:36.969 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.958+0000 7f4114c4bb80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621917fbda0 con 0x562191917830 2023-03-09T14:46:36.970 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.958+0000 7f4114c4bb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x56219186a780 con 0x562191911fe0 2023-03-09T14:46:36.970 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.958+0000 7f4114c4bb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x562191863cb0 con 0x562191918140 2023-03-09T14:46:36.970 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.959+0000 7f410259c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562191917830 0x562191917c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:36.970 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.959+0000 7f410259c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562191917830 0x562191917c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39386/0 (socket says 172.21.15.38:39386) 2023-03-09T14:46:36.970 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.959+0000 7f410259c700 1 -- 172.21.15.38:0/3809065216 learned_addr learned my addr 172.21.15.38:0/3809065216 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:46:36.970 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.959+0000 7f4102d9d700 1 --2- 172.21.15.38:0/3809065216 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562191911fe0 0x562191912410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:36.970 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.959+0000 7f4101d9b700 1 --2- 172.21.15.38:0/3809065216 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562191918140 0x562191911aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:36.970 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.959+0000 7f410259c700 1 -- 172.21.15.38:0/3809065216 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562191918140 msgr2=0x562191911aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:46:36.971 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.959+0000 7f410259c700 1 --2- 172.21.15.38:0/3809065216 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562191918140 0x562191911aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.971 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.959+0000 7f410259c700 1 -- 172.21.15.38:0/3809065216 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562191911fe0 msgr2=0x562191912410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:46:36.971 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.959+0000 7f410259c700 1 --2- 172.21.15.38:0/3809065216 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562191911fe0 0x562191912410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.971 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.959+0000 7f410259c700 1 -- 172.21.15.38:0/3809065216 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56219182a190 con 0x562191917830 2023-03-09T14:46:36.971 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.959+0000 7f410259c700 1 --2- 172.21.15.38:0/3809065216 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562191917830 0x562191917c00 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f40f8097ed0 tx=0x7f40f8097d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5e7590cb774d0a9 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:36.971 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.959+0000 7f40f0ff9700 1 -- 172.21.15.38:0/3809065216 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f80a03c0 con 0x562191917830 2023-03-09T14:46:36.971 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.959+0000 7f40f0ff9700 1 -- 172.21.15.38:0/3809065216 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f40f8096310 con 0x562191917830 2023-03-09T14:46:36.972 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.959+0000 7f40f0ff9700 1 -- 172.21.15.38:0/3809065216 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f80976c0 con 0x562191917830 2023-03-09T14:46:36.972 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.960+0000 7f4114c4bb80 1 -- 172.21.15.38:0/3809065216 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562191917830 msgr2=0x562191917c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:36.972 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.960+0000 7f4114c4bb80 1 --2- 172.21.15.38:0/3809065216 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562191917830 0x562191917c00 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f40f8097ed0 tx=0x7f40f8097d90 comp rx=0 tx=0).stop 2023-03-09T14:46:36.972 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.960+0000 7f4114c4bb80 1 -- 172.21.15.38:0/3809065216 shutdown_connections 2023-03-09T14:46:36.972 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.960+0000 7f4114c4bb80 1 --2- 172.21.15.38:0/3809065216 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562191917830 0x562191917c00 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.972 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.960+0000 7f4114c4bb80 1 --2- 172.21.15.38:0/3809065216 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562191911fe0 0x562191912410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.972 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.960+0000 7f4114c4bb80 1 --2- 172.21.15.38:0/3809065216 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562191918140 0x562191911aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.972 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.960+0000 7f4114c4bb80 1 -- 172.21.15.38:0/3809065216 >> 172.21.15.38:0/3809065216 conn(0x56219190ca20 msgr2=0x56219190cdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:36.973 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.960+0000 7f4114c4bb80 1 -- 172.21.15.38:0/3809065216 shutdown_connections 2023-03-09T14:46:36.973 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.960+0000 7f4114c4bb80 1 -- 172.21.15.38:0/3809065216 wait complete. 2023-03-09T14:46:36.973 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.960+0000 7f4114c4bb80 1 -- start start 2023-03-09T14:46:36.973 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.961+0000 7f4114c4bb80 1 Processor -- start 2023-03-09T14:46:36.973 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.961+0000 7f4114c4bb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562191911fe0 0x56219191e160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:36.973 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.961+0000 7f4114c4bb80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562191917830 0x56219191b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:36.973 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.961+0000 7f4114c4bb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562191918140 0x56219191bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:36.974 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.961+0000 7f4114c4bb80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621916e7920 con 0x562191917830 2023-03-09T14:46:36.974 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.961+0000 7f4114c4bb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x56219191c300 con 0x562191911fe0 2023-03-09T14:46:36.974 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.961+0000 7f4114c4bb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x562191927c70 con 0x562191918140 2023-03-09T14:46:36.974 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.961+0000 7f4114c4bb80 10 client.0 _ll_register_callbacks cb 0x562191809520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:46:36.974 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.961+0000 7f4101d9b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562191917830 0x56219191b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:36.974 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.961+0000 7f4101d9b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562191917830 0x56219191b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39392/0 (socket says 172.21.15.38:39392) 2023-03-09T14:46:36.974 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.961+0000 7f4101d9b700 1 -- 172.21.15.38:0/3190978788 learned_addr learned my addr 172.21.15.38:0/3190978788 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:46:36.974 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.961+0000 7f410259c700 1 --2- 172.21.15.38:0/3190978788 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562191911fe0 0x56219191e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:36.975 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.961+0000 7f4102d9d700 1 --2- 172.21.15.38:0/3190978788 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562191918140 0x56219191bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:36.975 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.961+0000 7f4101d9b700 1 -- 172.21.15.38:0/3190978788 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562191918140 msgr2=0x56219191bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:46:36.975 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.961+0000 7f4101d9b700 1 --2- 172.21.15.38:0/3190978788 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562191918140 0x56219191bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.975 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.961+0000 7f4101d9b700 1 -- 172.21.15.38:0/3190978788 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562191911fe0 msgr2=0x56219191e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:46:36.975 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.961+0000 7f4101d9b700 1 --2- 172.21.15.38:0/3190978788 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562191911fe0 0x56219191e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.975 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.961+0000 7f4101d9b700 1 -- 172.21.15.38:0/3190978788 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f410410ff70 con 0x562191917830 2023-03-09T14:46:36.975 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.961+0000 7f4102d9d700 1 --2- 172.21.15.38:0/3190978788 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562191918140 0x56219191bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-09T14:46:36.976 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.962+0000 7f4101d9b700 1 --2- 172.21.15.38:0/3190978788 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562191917830 0x56219191b450 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f40f4015000 tx=0x7f40f4009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d0086e50730895f server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:36.976 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.962+0000 7f40e9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2023-03-09T14:46:36.976 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.962+0000 7f40e9ffb700 1 -- 172.21.15.38:0/3190978788 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f4004ea0 con 0x562191917830 2023-03-09T14:46:36.976 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.962+0000 7f40e9ffb700 1 -- 172.21.15.38:0/3190978788 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9278+0+0 (secure 0 0 0) 0x7f40f400f460 con 0x562191917830 2023-03-09T14:46:36.976 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.962+0000 7f4114c4bb80 1 -- 172.21.15.38:0/3190978788 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562191831190 con 0x562191917830 2023-03-09T14:46:36.976 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.962+0000 7f40e9ffb700 1 -- 172.21.15.38:0/3190978788 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f40f400f620 con 0x562191917830 2023-03-09T14:46:36.976 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.962+0000 7f4114c4bb80 10 client.15705 fetch_fsmap learned FSMap version 23 2023-03-09T14:46:36.977 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.962+0000 7f4114c4bb80 1 -- 172.21.15.38:0/3190978788 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x56219167ac60 con 0x562191917830 2023-03-09T14:46:36.977 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.962+0000 7f40e9ffb700 1 -- 172.21.15.38:0/3190978788 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x562191831190 con 0x562191917830 2023-03-09T14:46:36.977 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.962+0000 7f40e9ffb700 1 -- 172.21.15.38:0/3190978788 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f40f4030910 con 0x562191917830 2023-03-09T14:46:36.977 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.962+0000 7f4114c4bb80 10 client.15705 fetch_fsmap finished waiting for FSMap version 23 2023-03-09T14:46:36.977 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.962+0000 7f4114c4bb80 10 client.15705 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T14:46:36.977 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.962+0000 7f4114c4bb80 20 client.15705 populate_metadata read hostname 'smithi038' 2023-03-09T14:46:36.977 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.962+0000 7f4114c4bb80 1 --2- 172.21.15.38:0/3190978788 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x56219192bda0 0x56219192e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:36.978 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.962+0000 7f4114c4bb80 4 client.15705 mds_command: new command op to 24530 tid=0[{"prefix": "get_command_descriptions"}] 2023-03-09T14:46:36.978 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.962+0000 7f4114c4bb80 1 -- 172.21.15.38:0/3190978788 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5621916e7f60 con 0x56219192bda0 2023-03-09T14:46:36.978 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.963+0000 7f410259c700 1 --2- 172.21.15.38:0/3190978788 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x56219192bda0 0x56219192e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:36.978 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.963+0000 7f40e9ffb700 10 client.15705 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:46:36.978 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.963+0000 7f410259c700 1 --2- 172.21.15.38:0/3190978788 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x56219192bda0 0x56219192e1b0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=868f15ac1d759d49 server_cookie=c08d7c4b8f83e653 in_seq=0 out_seq=0 2023-03-09T14:46:36.978 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.963+0000 7f40e9ffb700 1 -- 172.21.15.38:0/3190978788 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18278 (crc 0 0 0) 0x5621916e7f60 con 0x56219192bda0 2023-03-09T14:46:36.978 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.963+0000 7f40e9ffb700 10 client.15705 handle_command_reply: tid=0 2023-03-09T14:46:36.978 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.963+0000 7f4114c4bb80 1 client.15705 shutdown 2023-03-09T14:46:36.979 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.964+0000 7f4114c4bb80 1 -- 172.21.15.38:0/3190978788 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562191917830 msgr2=0x56219191b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:36.979 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.964+0000 7f4114c4bb80 1 --2- 172.21.15.38:0/3190978788 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562191917830 0x56219191b450 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f40f4015000 tx=0x7f40f4009ca0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.979 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.964+0000 7f4114c4bb80 1 -- 172.21.15.38:0/3190978788 shutdown_connections 2023-03-09T14:46:36.979 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.964+0000 7f4114c4bb80 1 --2- 172.21.15.38:0/3190978788 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x56219192bda0 0x56219192e1b0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.979 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.964+0000 7f4114c4bb80 1 --2- 172.21.15.38:0/3190978788 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562191918140 0x56219191bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.979 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.964+0000 7f4114c4bb80 1 --2- 172.21.15.38:0/3190978788 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562191911fe0 0x56219191e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.979 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.964+0000 7f40e9ffb700 0 client.15705 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:46:36.980 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.964+0000 7f4114c4bb80 1 --2- 172.21.15.38:0/3190978788 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562191917830 0x56219191b450 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.980 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.964+0000 7f4114c4bb80 1 -- 172.21.15.38:0/3190978788 >> 172.21.15.38:0/3190978788 conn(0x56219190ca20 msgr2=0x56219190bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:36.980 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.964+0000 7f4114c4bb80 1 -- 172.21.15.38:0/3190978788 shutdown_connections 2023-03-09T14:46:36.980 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.964+0000 7f4114c4bb80 1 -- 172.21.15.38:0/3190978788 wait complete. 2023-03-09T14:46:36.980 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.964+0000 7f4114c4bb80 20 client.15705 trim_cache size 0 max 16384 2023-03-09T14:46:36.981 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:36 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/2900950494' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-03-09T14:46:36.986 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.977+0000 7f4114c4bb80 1 Processor -- start 2023-03-09T14:46:36.987 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.977+0000 7f4114c4bb80 1 -- start start 2023-03-09T14:46:36.987 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.977+0000 7f4114c4bb80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562191911fe0 0x56219190bc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:36.987 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.978+0000 7f4114c4bb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562191917830 0x56219191dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:36.987 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.978+0000 7f4114c4bb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562191918140 0x56219191e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:36.987 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.978+0000 7f4114c4bb80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x56219191c9f0 con 0x562191911fe0 2023-03-09T14:46:36.988 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.978+0000 7f4114c4bb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x56219190c840 con 0x562191918140 2023-03-09T14:46:36.988 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.978+0000 7f4114c4bb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x562191917460 con 0x562191917830 2023-03-09T14:46:36.988 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.978+0000 7f410259c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562191911fe0 0x56219190bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:36.988 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.978+0000 7f410259c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562191911fe0 0x56219190bc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39398/0 (socket says 172.21.15.38:39398) 2023-03-09T14:46:36.989 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.978+0000 7f410259c700 1 -- 172.21.15.38:0/4203571812 learned_addr learned my addr 172.21.15.38:0/4203571812 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:46:36.989 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.978+0000 7f4101d9b700 1 --2- 172.21.15.38:0/4203571812 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562191917830 0x56219191dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:36.989 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.978+0000 7f4102d9d700 1 --2- 172.21.15.38:0/4203571812 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562191918140 0x56219191e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:36.989 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.978+0000 7f410259c700 1 -- 172.21.15.38:0/4203571812 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562191917830 msgr2=0x56219191dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:46:36.990 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.978+0000 7f410259c700 1 --2- 172.21.15.38:0/4203571812 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562191917830 0x56219191dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.990 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.978+0000 7f410259c700 1 -- 172.21.15.38:0/4203571812 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562191918140 msgr2=0x56219191e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:46:36.990 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.978+0000 7f410259c700 1 --2- 172.21.15.38:0/4203571812 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562191918140 0x56219191e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.990 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.978+0000 7f410259c700 1 -- 172.21.15.38:0/4203571812 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562191691de0 con 0x562191911fe0 2023-03-09T14:46:36.990 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.978+0000 7f4102d9d700 1 --2- 172.21.15.38:0/4203571812 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562191918140 0x56219191e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:46:36.991 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.978+0000 7f410259c700 1 --2- 172.21.15.38:0/4203571812 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562191911fe0 0x56219190bc20 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f40f800e970 tx=0x7f40f800cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9d3e58745b7cd89 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:36.991 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.978+0000 7f40f17fa700 1 -- 172.21.15.38:0/4203571812 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f80a03c0 con 0x562191911fe0 2023-03-09T14:46:36.991 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.978+0000 7f40f17fa700 1 -- 172.21.15.38:0/4203571812 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f40f8096310 con 0x562191911fe0 2023-03-09T14:46:36.991 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.978+0000 7f40f17fa700 1 -- 172.21.15.38:0/4203571812 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f80976c0 con 0x562191911fe0 2023-03-09T14:46:36.992 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.979+0000 7f4114c4bb80 1 -- 172.21.15.38:0/4203571812 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562191911fe0 msgr2=0x56219190bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:36.992 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.979+0000 7f4114c4bb80 1 --2- 172.21.15.38:0/4203571812 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562191911fe0 0x56219190bc20 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f40f800e970 tx=0x7f40f800cf50 comp rx=0 tx=0).stop 2023-03-09T14:46:36.992 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.979+0000 7f4114c4bb80 1 -- 172.21.15.38:0/4203571812 shutdown_connections 2023-03-09T14:46:36.992 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.979+0000 7f4114c4bb80 1 --2- 172.21.15.38:0/4203571812 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562191911fe0 0x56219190bc20 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.992 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.979+0000 7f4114c4bb80 1 --2- 172.21.15.38:0/4203571812 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562191918140 0x56219191e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.993 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.979+0000 7f4114c4bb80 1 --2- 172.21.15.38:0/4203571812 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562191917830 0x56219191dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.993 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.979+0000 7f4114c4bb80 1 -- 172.21.15.38:0/4203571812 >> 172.21.15.38:0/4203571812 conn(0x56219190ca20 msgr2=0x562191927320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:36.993 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.979+0000 7f4114c4bb80 1 -- 172.21.15.38:0/4203571812 shutdown_connections 2023-03-09T14:46:36.993 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.979+0000 7f4114c4bb80 1 -- 172.21.15.38:0/4203571812 wait complete. 2023-03-09T14:46:36.994 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.980+0000 7f4114c4bb80 1 -- start start 2023-03-09T14:46:36.994 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.980+0000 7f4114c4bb80 1 Processor -- start 2023-03-09T14:46:36.994 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.980+0000 7f4114c4bb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562191911fe0 0x562191914a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:36.994 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.980+0000 7f4114c4bb80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562191917830 0x562191913290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:36.995 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.980+0000 7f4114c4bb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562191918140 0x5621919137d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:36.995 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.980+0000 7f4114c4bb80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x562191923100 con 0x562191917830 2023-03-09T14:46:36.995 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.980+0000 7f4114c4bb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x562191913d10 con 0x562191918140 2023-03-09T14:46:36.995 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.980+0000 7f4114c4bb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x562191913ed0 con 0x562191911fe0 2023-03-09T14:46:36.995 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.980+0000 7f4114c4bb80 10 client.0 _ll_register_callbacks cb 0x562191809520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:46:36.996 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.980+0000 7f4101d9b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562191917830 0x562191913290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:36.996 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.980+0000 7f4101d9b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562191917830 0x562191913290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39406/0 (socket says 172.21.15.38:39406) 2023-03-09T14:46:36.996 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.980+0000 7f4101d9b700 1 -- 172.21.15.38:0/759776184 learned_addr learned my addr 172.21.15.38:0/759776184 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:46:36.997 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.980+0000 7f410259c700 1 --2- 172.21.15.38:0/759776184 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562191911fe0 0x562191914a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:36.997 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.980+0000 7f4102d9d700 1 --2- 172.21.15.38:0/759776184 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562191918140 0x5621919137d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:36.997 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.980+0000 7f4101d9b700 1 -- 172.21.15.38:0/759776184 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562191911fe0 msgr2=0x562191914a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:46:36.997 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.981+0000 7f4101d9b700 1 --2- 172.21.15.38:0/759776184 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562191911fe0 0x562191914a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.997 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.981+0000 7f4101d9b700 1 -- 172.21.15.38:0/759776184 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562191918140 msgr2=0x5621919137d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:46:36.998 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.981+0000 7f4101d9b700 1 --2- 172.21.15.38:0/759776184 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562191918140 0x5621919137d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:36.998 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.981+0000 7f4101d9b700 1 -- 172.21.15.38:0/759776184 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56219182a190 con 0x562191917830 2023-03-09T14:46:36.998 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.981+0000 7f4102d9d700 1 --2- 172.21.15.38:0/759776184 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562191918140 0x5621919137d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:46:36.998 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.981+0000 7f410259c700 1 --2- 172.21.15.38:0/759776184 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562191911fe0 0x562191914a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:46:36.998 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.981+0000 7f4101d9b700 1 --2- 172.21.15.38:0/759776184 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562191917830 0x562191913290 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f40f4030d20 tx=0x7f40f4004e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=226796aef40d8087 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:36.999 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.981+0000 7f40e9ffb700 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2023-03-09T14:46:36.999 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.981+0000 7f40e9ffb700 1 -- 172.21.15.38:0/759776184 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f4030840 con 0x562191917830 2023-03-09T14:46:36.999 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.981+0000 7f40e9ffb700 1 -- 172.21.15.38:0/759776184 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9278+0+0 (secure 0 0 0) 0x7f40f40309a0 con 0x562191917830 2023-03-09T14:46:36.999 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.981+0000 7f4114c4bb80 1 -- 172.21.15.38:0/759776184 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5621918211b0 con 0x562191917830 2023-03-09T14:46:36.999 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.981+0000 7f40e9ffb700 1 -- 172.21.15.38:0/759776184 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f40f4024410 con 0x562191917830 2023-03-09T14:46:36.999 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.981+0000 7f4114c4bb80 10 client.15711 fetch_fsmap learned FSMap version 23 2023-03-09T14:46:36.999 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.981+0000 7f4114c4bb80 1 -- 172.21.15.38:0/759776184 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5621916902b0 con 0x562191917830 2023-03-09T14:46:37.000 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.981+0000 7f40e9ffb700 1 -- 172.21.15.38:0/759776184 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5621918211b0 con 0x562191917830 2023-03-09T14:46:37.000 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.981+0000 7f40e9ffb700 1 -- 172.21.15.38:0/759776184 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f40f4004500 con 0x562191917830 2023-03-09T14:46:37.000 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.981+0000 7f4114c4bb80 10 client.15711 fetch_fsmap finished waiting for FSMap version 23 2023-03-09T14:46:37.000 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.981+0000 7f4114c4bb80 10 client.15711 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T14:46:37.000 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.981+0000 7f4114c4bb80 20 client.15711 populate_metadata read hostname 'smithi038' 2023-03-09T14:46:37.000 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.981+0000 7f4114c4bb80 1 --2- 172.21.15.38:0/759776184 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x562191980ac0 0x562191982ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:37.001 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.981+0000 7f4114c4bb80 4 client.15711 mds_command: new command op to 24530 tid=0[{"prefix": "damage ls"}] 2023-03-09T14:46:37.001 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.981+0000 7f4114c4bb80 1 -- 172.21.15.38:0/759776184 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56219180b1c0 con 0x562191980ac0 2023-03-09T14:46:37.001 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.982+0000 7f410259c700 1 --2- 172.21.15.38:0/759776184 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x562191980ac0 0x562191982ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:37.001 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.982+0000 7f410259c700 1 --2- 172.21.15.38:0/759776184 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x562191980ac0 0x562191982ed0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c7db67c50b936d6 server_cookie=a67e61c14c90bcc2 in_seq=0 out_seq=0 2023-03-09T14:46:37.001 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.982+0000 7f40e9ffb700 10 client.15711 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:46:37.002 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.982+0000 7f40e9ffb700 1 -- 172.21.15.38:0/759776184 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56219180b1c0 con 0x562191980ac0 2023-03-09T14:46:37.002 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.982+0000 7f40e9ffb700 10 client.15711 handle_command_reply: tid=0 2023-03-09T14:46:37.002 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.982+0000 7f4114c4bb80 1 client.15711 shutdown 2023-03-09T14:46:37.002 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.982+0000 7f4114c4bb80 1 -- 172.21.15.38:0/759776184 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562191917830 msgr2=0x562191913290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:37.002 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.982+0000 7f4114c4bb80 1 --2- 172.21.15.38:0/759776184 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562191917830 0x562191913290 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f40f4030d20 tx=0x7f40f4004e20 comp rx=0 tx=0).stop 2023-03-09T14:46:37.002 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.982+0000 7f4114c4bb80 1 -- 172.21.15.38:0/759776184 shutdown_connections 2023-03-09T14:46:37.002 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.982+0000 7f4114c4bb80 1 --2- 172.21.15.38:0/759776184 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x562191980ac0 0x562191982ed0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:37.003 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.982+0000 7f4114c4bb80 1 --2- 172.21.15.38:0/759776184 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562191918140 0x5621919137d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:37.003 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.982+0000 7f4114c4bb80 1 --2- 172.21.15.38:0/759776184 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562191911fe0 0x562191914a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:37.003 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.982+0000 7f40e9ffb700 0 client.15711 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:46:37.003 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.982+0000 7f4114c4bb80 1 --2- 172.21.15.38:0/759776184 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562191917830 0x562191913290 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:37.003 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.983+0000 7f4114c4bb80 1 -- 172.21.15.38:0/759776184 >> 172.21.15.38:0/759776184 conn(0x56219190ca20 msgr2=0x562191915590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:37.003 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.983+0000 7f4114c4bb80 1 -- 172.21.15.38:0/759776184 shutdown_connections 2023-03-09T14:46:37.003 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.983+0000 7f4114c4bb80 1 -- 172.21.15.38:0/759776184 wait complete. 2023-03-09T14:46:37.004 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.983+0000 7f4114c4bb80 20 client.15711 trim_cache size 0 max 16384 2023-03-09T14:46:37.004 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2023-03-09T14:46:37.005 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.983+0000 7f4108ac2700 1 -- 172.21.15.38:0/1532218431 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f40ec09f4e0 msgr2=0x7f40ec0a1990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:37.005 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.983+0000 7f4108ac2700 1 --2- 172.21.15.38:0/1532218431 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f40ec09f4e0 0x7f40ec0a1990 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f41041977a0 tx=0x7f40f4005fd0 comp rx=0 tx=0).stop 2023-03-09T14:46:37.005 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.983+0000 7f4108ac2700 1 -- 172.21.15.38:0/1532218431 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4104075f80 msgr2=0x7f4104196190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:37.006 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.983+0000 7f4108ac2700 1 --2- 172.21.15.38:0/1532218431 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4104075f80 0x7f4104196190 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f40f800e9a0 tx=0x7f40f800ecb0 comp rx=0 tx=0).stop 2023-03-09T14:46:37.006 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.983+0000 7f4108ac2700 1 -- 172.21.15.38:0/1532218431 shutdown_connections 2023-03-09T14:46:37.006 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.983+0000 7f4108ac2700 1 --2- 172.21.15.38:0/1532218431 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f40ec09f4e0 0x7f40ec0a1990 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:37.006 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.983+0000 7f4108ac2700 1 --2- 172.21.15.38:0/1532218431 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4104075f80 0x7f4104196190 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:37.006 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.983+0000 7f4108ac2700 1 --2- 172.21.15.38:0/1532218431 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f410410a710 0x7f410419ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:37.006 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.983+0000 7f4108ac2700 1 --2- 172.21.15.38:0/1532218431 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f41040ff840 0x7f41041966d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:37.006 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.983+0000 7f4108ac2700 1 -- 172.21.15.38:0/1532218431 >> 172.21.15.38:0/1532218431 conn(0x7f41040748f0 msgr2=0x7f4104102620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:37.007 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.983+0000 7f4108ac2700 1 -- 172.21.15.38:0/1532218431 shutdown_connections 2023-03-09T14:46:37.007 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:36.983+0000 7f4108ac2700 1 -- 172.21.15.38:0/1532218431 wait complete. 2023-03-09T14:46:37.007 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:36 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/2900950494' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-03-09T14:46:37.009 INFO:teuthology.orchestra.run.smithi038.stderr:8500000 of 50000000 tuples (17%) done (elapsed 12.15 s, remaining 59.31 s) 2023-03-09T14:46:37.195 INFO:teuthology.orchestra.run.smithi038.stderr:8600000 of 50000000 tuples (17%) done (elapsed 12.33 s, remaining 59.37 s) 2023-03-09T14:46:37.261 INFO:teuthology.orchestra.run.smithi038.stderr:8700000 of 50000000 tuples (17%) done (elapsed 12.40 s, remaining 58.86 s) 2023-03-09T14:46:37.420 INFO:teuthology.orchestra.run.smithi038.stderr:8800000 of 50000000 tuples (17%) done (elapsed 12.56 s, remaining 58.79 s) 2023-03-09T14:46:37.575 INFO:teuthology.orchestra.run.smithi038.stderr:8900000 of 50000000 tuples (17%) done (elapsed 12.71 s, remaining 58.71 s) 2023-03-09T14:46:37.661 INFO:teuthology.orchestra.run.smithi038.stderr:9000000 of 50000000 tuples (18%) done (elapsed 12.80 s, remaining 58.31 s) 2023-03-09T14:46:37.827 INFO:teuthology.orchestra.run.smithi038.stderr:9100000 of 50000000 tuples (18%) done (elapsed 12.96 s, remaining 58.27 s) 2023-03-09T14:46:37.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:37 smithi122 ceph-mon[93869]: pgmap v14: 129 pgs: 129 active+clean; 1.5 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 194 MiB/s wr, 202 op/s 2023-03-09T14:46:37.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:37 smithi181 ceph-mon[94189]: pgmap v14: 129 pgs: 129 active+clean; 1.5 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 194 MiB/s wr, 202 op/s 2023-03-09T14:46:37.986 INFO:teuthology.orchestra.run.smithi038.stderr:9200000 of 50000000 tuples (18%) done (elapsed 13.12 s, remaining 58.20 s) 2023-03-09T14:46:38.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:37 smithi038 ceph-mon[92422]: pgmap v14: 129 pgs: 129 active+clean; 1.5 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 194 MiB/s wr, 202 op/s 2023-03-09T14:46:38.008 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-03-09T14:46:38.008 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-03-09T14:46:38.071 INFO:teuthology.orchestra.run.smithi038.stderr:9300000 of 50000000 tuples (18%) done (elapsed 13.21 s, remaining 57.81 s) 2023-03-09T14:46:38.152 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.150+0000 7f8a47832700 1 -- 172.21.15.38:0/1679186521 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8a40101280 msgr2=0x7f8a40104ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:38.153 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.150+0000 7f8a47832700 1 --2- 172.21.15.38:0/1679186521 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8a40101280 0x7f8a40104ab0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f8a300099c0 tx=0x7f8a30009cd0 comp rx=0 tx=0).stop 2023-03-09T14:46:38.153 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.150+0000 7f8a47832700 1 -- 172.21.15.38:0/1679186521 shutdown_connections 2023-03-09T14:46:38.153 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.150+0000 7f8a47832700 1 --2- 172.21.15.38:0/1679186521 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a40105190 0x7f8a4010cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:38.153 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.150+0000 7f8a47832700 1 --2- 172.21.15.38:0/1679186521 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8a401008e0 0x7f8a40100cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:38.153 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.150+0000 7f8a47832700 1 --2- 172.21.15.38:0/1679186521 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8a40101280 0x7f8a40104ab0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:38.153 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.150+0000 7f8a47832700 1 -- 172.21.15.38:0/1679186521 >> 172.21.15.38:0/1679186521 conn(0x7f8a40077210 msgr2=0x7f8a40077610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:38.154 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.150+0000 7f8a47832700 1 -- 172.21.15.38:0/1679186521 shutdown_connections 2023-03-09T14:46:38.155 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.150+0000 7f8a47832700 1 -- 172.21.15.38:0/1679186521 wait complete. 2023-03-09T14:46:38.155 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.151+0000 7f8a47832700 1 Processor -- start 2023-03-09T14:46:38.155 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.151+0000 7f8a47832700 1 -- start start 2023-03-09T14:46:38.155 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.152+0000 7f8a47832700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a401008e0 0x7f8a40196140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:38.155 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.152+0000 7f8a47832700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8a40101280 0x7f8a40196680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:38.156 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.152+0000 7f8a47832700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8a40105190 0x7f8a4019aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:38.156 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.152+0000 7f8a47832700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a4019b120 con 0x7f8a40101280 2023-03-09T14:46:38.156 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.152+0000 7f8a47832700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a4019b290 con 0x7f8a40105190 2023-03-09T14:46:38.156 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.152+0000 7f8a47832700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a4019b450 con 0x7f8a401008e0 2023-03-09T14:46:38.156 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.152+0000 7f8a44dcd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8a40101280 0x7f8a40196680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:38.156 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.152+0000 7f8a44dcd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8a40101280 0x7f8a40196680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39420/0 (socket says 172.21.15.38:39420) 2023-03-09T14:46:38.156 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.152+0000 7f8a44dcd700 1 -- 172.21.15.38:0/4206804422 learned_addr learned my addr 172.21.15.38:0/4206804422 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:46:38.157 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.152+0000 7f8a44dcd700 1 -- 172.21.15.38:0/4206804422 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a401008e0 msgr2=0x7f8a40196140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:38.157 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.152+0000 7f8a44dcd700 1 --2- 172.21.15.38:0/4206804422 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a401008e0 0x7f8a40196140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:38.157 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.152+0000 7f8a44dcd700 1 -- 172.21.15.38:0/4206804422 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8a40105190 msgr2=0x7f8a4019aae0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-03-09T14:46:38.157 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.152+0000 7f8a44dcd700 1 --2- 172.21.15.38:0/4206804422 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8a40105190 0x7f8a4019aae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:38.157 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.152+0000 7f8a44dcd700 1 -- 172.21.15.38:0/4206804422 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a300095f0 con 0x7f8a40101280 2023-03-09T14:46:38.157 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.152+0000 7f8a44dcd700 1 --2- 172.21.15.38:0/4206804422 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8a40101280 0x7f8a40196680 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f8a30000c00 tx=0x7f8a30003750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:38.158 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.154+0000 7f8a367fc700 1 -- 172.21.15.38:0/4206804422 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a3001d070 con 0x7f8a40101280 2023-03-09T14:46:38.158 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.154+0000 7f8a367fc700 1 -- 172.21.15.38:0/4206804422 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8a30004150 con 0x7f8a40101280 2023-03-09T14:46:38.158 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.154+0000 7f8a47832700 1 -- 172.21.15.38:0/4206804422 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a4010ffe0 con 0x7f8a40101280 2023-03-09T14:46:38.158 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.154+0000 7f8a367fc700 1 -- 172.21.15.38:0/4206804422 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a3000f460 con 0x7f8a40101280 2023-03-09T14:46:38.158 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.154+0000 7f8a47832700 1 -- 172.21.15.38:0/4206804422 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a4010fe50 con 0x7f8a40101280 2023-03-09T14:46:38.159 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.157+0000 7f8a367fc700 1 -- 172.21.15.38:0/4206804422 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 23) v1 ==== 133335+0+0 (secure 0 0 0) 0x7f8a30029030 con 0x7f8a40101280 2023-03-09T14:46:38.160 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.158+0000 7f8a367fc700 1 --2- 172.21.15.38:0/4206804422 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f8a2c09f560 0x7f8a2c0a1a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:38.160 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.158+0000 7f8a455ce700 1 --2- 172.21.15.38:0/4206804422 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f8a2c09f560 0x7f8a2c0a1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:38.161 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.158+0000 7f8a367fc700 1 -- 172.21.15.38:0/4206804422 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9278+0+0 (secure 0 0 0) 0x7f8a300cb530 con 0x7f8a40101280 2023-03-09T14:46:38.162 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.159+0000 7f8a455ce700 1 --2- 172.21.15.38:0/4206804422 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f8a2c09f560 0x7f8a2c0a1a10 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c0097b0 tx=0x7f8a3c006d20 comp rx=0 tx=0).ready entity=mgr.15558 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:38.162 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.159+0000 7f8a539bbb80 1 Processor -- start 2023-03-09T14:46:38.162 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.159+0000 7f8a539bbb80 1 -- start start 2023-03-09T14:46:38.162 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.159+0000 7f8a539bbb80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ea35d28b0 0x561ea35d2c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:38.162 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.159+0000 7f8a539bbb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561ea35d31c0 0x561ea35ccb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:38.162 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.159+0000 7f8a539bbb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561ea35cd060 0x561ea35cd490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:38.162 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.159+0000 7f8a539bbb80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ea34b6e20 con 0x561ea35d28b0 2023-03-09T14:46:38.163 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.159+0000 7f8a539bbb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ea3525800 con 0x561ea35cd060 2023-03-09T14:46:38.163 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.159+0000 7f8a539bbb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ea351ed30 con 0x561ea35d31c0 2023-03-09T14:46:38.163 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.159+0000 7f8a455ce700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ea35d28b0 0x561ea35d2c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:38.163 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.159+0000 7f8a455ce700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ea35d28b0 0x561ea35d2c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39426/0 (socket says 172.21.15.38:39426) 2023-03-09T14:46:38.163 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.159+0000 7f8a455ce700 1 -- 172.21.15.38:0/2024296632 learned_addr learned my addr 172.21.15.38:0/2024296632 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:46:38.163 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.160+0000 7f8a45dcf700 1 --2- 172.21.15.38:0/2024296632 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561ea35cd060 0x561ea35cd490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:38.165 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.160+0000 7f8a455ce700 1 -- 172.21.15.38:0/2024296632 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561ea35d31c0 msgr2=0x561ea35ccb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:46:38.166 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.160+0000 7f8a455ce700 1 --2- 172.21.15.38:0/2024296632 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561ea35d31c0 0x561ea35ccb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:38.166 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.160+0000 7f8a455ce700 1 -- 172.21.15.38:0/2024296632 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561ea35cd060 msgr2=0x561ea35cd490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:46:38.166 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.160+0000 7f8a455ce700 1 --2- 172.21.15.38:0/2024296632 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561ea35cd060 0x561ea35cd490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:38.166 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.160+0000 7f8a455ce700 1 -- 172.21.15.38:0/2024296632 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561ea34e5210 con 0x561ea35d28b0 2023-03-09T14:46:38.166 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.160+0000 7f8a455ce700 1 --2- 172.21.15.38:0/2024296632 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ea35d28b0 0x561ea35d2c80 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c0158f0 tx=0x7f8a3c006f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e10d9b12844d03d5 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:38.166 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.160+0000 7f8a2b7fe700 1 -- 172.21.15.38:0/2024296632 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a3c01a070 con 0x561ea35d28b0 2023-03-09T14:46:38.166 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.160+0000 7f8a2b7fe700 1 -- 172.21.15.38:0/2024296632 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8a3c0128f0 con 0x561ea35d28b0 2023-03-09T14:46:38.167 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.160+0000 7f8a2b7fe700 1 -- 172.21.15.38:0/2024296632 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a3c013840 con 0x561ea35d28b0 2023-03-09T14:46:38.167 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.161+0000 7f8a539bbb80 1 -- 172.21.15.38:0/2024296632 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ea35d28b0 msgr2=0x561ea35d2c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:38.167 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.161+0000 7f8a539bbb80 1 --2- 172.21.15.38:0/2024296632 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ea35d28b0 0x561ea35d2c80 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c0158f0 tx=0x7f8a3c006f90 comp rx=0 tx=0).stop 2023-03-09T14:46:38.167 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.161+0000 7f8a539bbb80 1 -- 172.21.15.38:0/2024296632 shutdown_connections 2023-03-09T14:46:38.167 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.161+0000 7f8a539bbb80 1 --2- 172.21.15.38:0/2024296632 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ea35d28b0 0x561ea35d2c80 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:38.167 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.161+0000 7f8a539bbb80 1 --2- 172.21.15.38:0/2024296632 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561ea35cd060 0x561ea35cd490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:38.167 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.161+0000 7f8a539bbb80 1 --2- 172.21.15.38:0/2024296632 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561ea35d31c0 0x561ea35ccb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:38.167 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.161+0000 7f8a539bbb80 1 -- 172.21.15.38:0/2024296632 >> 172.21.15.38:0/2024296632 conn(0x561ea35c7aa0 msgr2=0x561ea35c7e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:38.168 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.161+0000 7f8a539bbb80 1 -- 172.21.15.38:0/2024296632 shutdown_connections 2023-03-09T14:46:38.168 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.161+0000 7f8a539bbb80 1 -- 172.21.15.38:0/2024296632 wait complete. 2023-03-09T14:46:38.168 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.162+0000 7f8a539bbb80 1 -- start start 2023-03-09T14:46:38.168 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.162+0000 7f8a539bbb80 1 Processor -- start 2023-03-09T14:46:38.168 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.162+0000 7f8a539bbb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561ea35cd060 0x561ea35d91b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:38.168 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.162+0000 7f8a539bbb80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ea35d28b0 0x561ea35d6430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:38.168 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.162+0000 7f8a539bbb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561ea35d31c0 0x561ea35d6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:38.169 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.162+0000 7f8a539bbb80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ea33a29a0 con 0x561ea35d28b0 2023-03-09T14:46:38.169 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.162+0000 7f8a539bbb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ea35d7240 con 0x561ea35d31c0 2023-03-09T14:46:38.169 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.162+0000 7f8a539bbb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ea35e2d50 con 0x561ea35cd060 2023-03-09T14:46:38.169 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.162+0000 7f8a539bbb80 10 client.0 _ll_register_callbacks cb 0x561ea34c45a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:46:38.169 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.162+0000 7f8a44dcd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ea35d28b0 0x561ea35d6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:38.169 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.162+0000 7f8a44dcd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ea35d28b0 0x561ea35d6430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39434/0 (socket says 172.21.15.38:39434) 2023-03-09T14:46:38.169 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.163+0000 7f8a44dcd700 1 -- 172.21.15.38:0/1297140019 learned_addr learned my addr 172.21.15.38:0/1297140019 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:46:38.169 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.163+0000 7f8a44dcd700 1 -- 172.21.15.38:0/1297140019 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561ea35cd060 msgr2=0x561ea35d91b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:46:38.170 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.163+0000 7f8a44dcd700 1 --2- 172.21.15.38:0/1297140019 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561ea35cd060 0x561ea35d91b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:38.170 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.163+0000 7f8a44dcd700 1 -- 172.21.15.38:0/1297140019 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561ea35d31c0 msgr2=0x561ea35d6ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:46:38.170 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.163+0000 7f8a44dcd700 1 --2- 172.21.15.38:0/1297140019 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561ea35d31c0 0x561ea35d6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:38.170 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.163+0000 7f8a44dcd700 1 -- 172.21.15.38:0/1297140019 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a3c0155d0 con 0x561ea35d28b0 2023-03-09T14:46:38.170 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.163+0000 7f8a44dcd700 1 --2- 172.21.15.38:0/1297140019 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ea35d28b0 0x561ea35d6430 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f8a30006010 tx=0x7f8a300996a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d88eba6da2b55c server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:38.170 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.163+0000 7f8a28ff9700 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2023-03-09T14:46:38.170 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.163+0000 7f8a28ff9700 1 -- 172.21.15.38:0/1297140019 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a300a1030 con 0x561ea35d28b0 2023-03-09T14:46:38.171 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.163+0000 7f8a28ff9700 1 -- 172.21.15.38:0/1297140019 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9278+0+0 (secure 0 0 0) 0x7f8a3009e070 con 0x561ea35d28b0 2023-03-09T14:46:38.171 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.163+0000 7f8a539bbb80 1 -- 172.21.15.38:0/1297140019 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561ea34ec210 con 0x561ea35d28b0 2023-03-09T14:46:38.171 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.163+0000 7f8a28ff9700 1 -- 172.21.15.38:0/1297140019 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8a300b1200 con 0x561ea35d28b0 2023-03-09T14:46:38.171 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.163+0000 7f8a539bbb80 10 client.15723 fetch_fsmap learned FSMap version 23 2023-03-09T14:46:38.171 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.163+0000 7f8a539bbb80 1 -- 172.21.15.38:0/1297140019 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x561ea3335ce0 con 0x561ea35d28b0 2023-03-09T14:46:38.171 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.164+0000 7f8a28ff9700 1 -- 172.21.15.38:0/1297140019 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x561ea34ec210 con 0x561ea35d28b0 2023-03-09T14:46:38.173 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.164+0000 7f8a28ff9700 1 -- 172.21.15.38:0/1297140019 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f8a300980c0 con 0x561ea35d28b0 2023-03-09T14:46:38.173 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.164+0000 7f8a539bbb80 10 client.15723 fetch_fsmap finished waiting for FSMap version 23 2023-03-09T14:46:38.173 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.164+0000 7f8a539bbb80 10 client.15723 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T14:46:38.173 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.164+0000 7f8a539bbb80 20 client.15723 populate_metadata read hostname 'smithi038' 2023-03-09T14:46:38.173 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.164+0000 7f8a539bbb80 1 --2- 172.21.15.38:0/1297140019 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x561ea35e6b30 0x561ea35e8f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:38.173 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.164+0000 7f8a539bbb80 4 client.15723 mds_command: new command op to 24530 tid=0[{"prefix": "get_command_descriptions"}] 2023-03-09T14:46:38.174 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.164+0000 7f8a539bbb80 1 -- 172.21.15.38:0/1297140019 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561ea33a2fe0 con 0x561ea35e6b30 2023-03-09T14:46:38.174 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.164+0000 7f8a455ce700 1 --2- 172.21.15.38:0/1297140019 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x561ea35e6b30 0x561ea35e8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:38.174 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.165+0000 7f8a28ff9700 10 client.15723 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:46:38.174 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.165+0000 7f8a455ce700 1 --2- 172.21.15.38:0/1297140019 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x561ea35e6b30 0x561ea35e8f40 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4a458e6e8fff0a39 server_cookie=77a480e3c38475e in_seq=0 out_seq=0 2023-03-09T14:46:38.174 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.165+0000 7f8a28ff9700 1 -- 172.21.15.38:0/1297140019 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18278 (crc 0 0 0) 0x561ea33a2fe0 con 0x561ea35e6b30 2023-03-09T14:46:38.174 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.165+0000 7f8a28ff9700 10 client.15723 handle_command_reply: tid=0 2023-03-09T14:46:38.174 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.166+0000 7f8a539bbb80 1 client.15723 shutdown 2023-03-09T14:46:38.174 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.166+0000 7f8a539bbb80 1 -- 172.21.15.38:0/1297140019 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ea35d28b0 msgr2=0x561ea35d6430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:38.175 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.166+0000 7f8a539bbb80 1 --2- 172.21.15.38:0/1297140019 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ea35d28b0 0x561ea35d6430 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f8a30006010 tx=0x7f8a300996a0 comp rx=0 tx=0).stop 2023-03-09T14:46:38.175 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.166+0000 7f8a539bbb80 1 -- 172.21.15.38:0/1297140019 shutdown_connections 2023-03-09T14:46:38.175 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.166+0000 7f8a539bbb80 1 --2- 172.21.15.38:0/1297140019 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x561ea35e6b30 0x561ea35e8f40 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:38.175 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.166+0000 7f8a539bbb80 1 --2- 172.21.15.38:0/1297140019 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561ea35d31c0 0x561ea35d6ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:38.175 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.166+0000 7f8a539bbb80 1 --2- 172.21.15.38:0/1297140019 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561ea35cd060 0x561ea35d91b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:38.175 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.166+0000 7f8a539bbb80 1 --2- 172.21.15.38:0/1297140019 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ea35d28b0 0x561ea35d6430 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:38.175 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.166+0000 7f8a539bbb80 1 -- 172.21.15.38:0/1297140019 >> 172.21.15.38:0/1297140019 conn(0x561ea35c7aa0 msgr2=0x561ea35c6ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:38.175 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.166+0000 7f8a28ff9700 0 client.15723 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:46:38.176 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.166+0000 7f8a539bbb80 1 -- 172.21.15.38:0/1297140019 shutdown_connections 2023-03-09T14:46:38.176 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.166+0000 7f8a539bbb80 1 -- 172.21.15.38:0/1297140019 wait complete. 2023-03-09T14:46:38.176 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.166+0000 7f8a539bbb80 20 client.15723 trim_cache size 0 max 16384 2023-03-09T14:46:38.188 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.185+0000 7f8a539bbb80 1 Processor -- start 2023-03-09T14:46:38.188 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.185+0000 7f8a539bbb80 1 -- start start 2023-03-09T14:46:38.188 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.185+0000 7f8a539bbb80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ea35cd060 0x561ea35d8d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:38.188 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.185+0000 7f8a539bbb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561ea35d28b0 0x561ea35c6ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:38.189 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.185+0000 7f8a539bbb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561ea35d31c0 0x561ea35c71e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:38.189 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.185+0000 7f8a539bbb80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ea35cdd40 con 0x561ea35cd060 2023-03-09T14:46:38.189 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.185+0000 7f8a539bbb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ea35d7a10 con 0x561ea35d31c0 2023-03-09T14:46:38.189 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.185+0000 7f8a539bbb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ea35d2500 con 0x561ea35d28b0 2023-03-09T14:46:38.189 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.185+0000 7f8a455ce700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ea35cd060 0x561ea35d8d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:38.189 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.185+0000 7f8a455ce700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ea35cd060 0x561ea35d8d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39440/0 (socket says 172.21.15.38:39440) 2023-03-09T14:46:38.189 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.185+0000 7f8a455ce700 1 -- 172.21.15.38:0/2551465318 learned_addr learned my addr 172.21.15.38:0/2551465318 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:46:38.190 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.185+0000 7f8a44dcd700 1 --2- 172.21.15.38:0/2551465318 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561ea35d28b0 0x561ea35c6ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:38.190 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.185+0000 7f8a45dcf700 1 --2- 172.21.15.38:0/2551465318 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561ea35d31c0 0x561ea35c71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:38.190 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.186+0000 7f8a455ce700 1 -- 172.21.15.38:0/2551465318 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561ea35d28b0 msgr2=0x561ea35c6ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:46:38.190 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.186+0000 7f8a455ce700 1 --2- 172.21.15.38:0/2551465318 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561ea35d28b0 0x561ea35c6ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:38.190 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.186+0000 7f8a455ce700 1 -- 172.21.15.38:0/2551465318 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561ea35d31c0 msgr2=0x561ea35c71e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:46:38.190 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.186+0000 7f8a455ce700 1 --2- 172.21.15.38:0/2551465318 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561ea35d31c0 0x561ea35c71e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:38.190 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.186+0000 7f8a455ce700 1 -- 172.21.15.38:0/2551465318 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561ea35edb30 con 0x561ea35cd060 2023-03-09T14:46:38.191 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.186+0000 7f8a455ce700 1 --2- 172.21.15.38:0/2551465318 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ea35cd060 0x561ea35d8d50 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c0158f0 tx=0x7f8a3c015f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb136a09a2e2e40a server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:38.191 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.186+0000 7f8a2bfff700 1 -- 172.21.15.38:0/2551465318 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a3c01a040 con 0x561ea35cd060 2023-03-09T14:46:38.191 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.186+0000 7f8a2bfff700 1 -- 172.21.15.38:0/2551465318 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8a3c02f070 con 0x561ea35cd060 2023-03-09T14:46:38.191 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.186+0000 7f8a2bfff700 1 -- 172.21.15.38:0/2551465318 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a3c013780 con 0x561ea35cd060 2023-03-09T14:46:38.194 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.187+0000 7f8a539bbb80 1 -- 172.21.15.38:0/2551465318 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ea35cd060 msgr2=0x561ea35d8d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:38.194 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.187+0000 7f8a539bbb80 1 --2- 172.21.15.38:0/2551465318 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ea35cd060 0x561ea35d8d50 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c0158f0 tx=0x7f8a3c015f90 comp rx=0 tx=0).stop 2023-03-09T14:46:38.194 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.187+0000 7f8a539bbb80 1 -- 172.21.15.38:0/2551465318 shutdown_connections 2023-03-09T14:46:38.194 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.187+0000 7f8a539bbb80 1 --2- 172.21.15.38:0/2551465318 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ea35cd060 0x561ea35d8d50 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:38.195 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.187+0000 7f8a539bbb80 1 --2- 172.21.15.38:0/2551465318 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561ea35d31c0 0x561ea35c71e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:38.195 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.187+0000 7f8a539bbb80 1 --2- 172.21.15.38:0/2551465318 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561ea35d28b0 0x561ea35c6ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:38.195 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.187+0000 7f8a539bbb80 1 -- 172.21.15.38:0/2551465318 >> 172.21.15.38:0/2551465318 conn(0x561ea35c7aa0 msgr2=0x561ea35d0550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:38.195 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.187+0000 7f8a539bbb80 1 -- 172.21.15.38:0/2551465318 shutdown_connections 2023-03-09T14:46:38.195 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.187+0000 7f8a539bbb80 1 -- 172.21.15.38:0/2551465318 wait complete. 2023-03-09T14:46:38.195 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.188+0000 7f8a539bbb80 1 -- start start 2023-03-09T14:46:38.195 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.188+0000 7f8a539bbb80 1 Processor -- start 2023-03-09T14:46:38.196 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.188+0000 7f8a539bbb80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ea35cd060 0x561ea35cea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:38.196 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.188+0000 7f8a539bbb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561ea35d28b0 0x561ea35cef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:38.196 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.188+0000 7f8a539bbb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561ea35d31c0 0x561ea35cf4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:38.196 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.188+0000 7f8a539bbb80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ea363aa30 con 0x561ea35cd060 2023-03-09T14:46:38.196 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.188+0000 7f8a539bbb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ea363aba0 con 0x561ea35d31c0 2023-03-09T14:46:38.196 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.188+0000 7f8a539bbb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x561ea363ad60 con 0x561ea35d28b0 2023-03-09T14:46:38.196 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.188+0000 7f8a539bbb80 10 client.0 _ll_register_callbacks cb 0x561ea34c45a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:46:38.196 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.188+0000 7f8a455ce700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ea35cd060 0x561ea35cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:38.197 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.188+0000 7f8a455ce700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ea35cd060 0x561ea35cea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39446/0 (socket says 172.21.15.38:39446) 2023-03-09T14:46:38.197 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.188+0000 7f8a455ce700 1 -- 172.21.15.38:0/2455026883 learned_addr learned my addr 172.21.15.38:0/2455026883 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:46:38.197 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.188+0000 7f8a45dcf700 1 --2- 172.21.15.38:0/2455026883 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561ea35d31c0 0x561ea35cf4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:38.197 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.189+0000 7f8a44dcd700 1 --2- 172.21.15.38:0/2455026883 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561ea35d28b0 0x561ea35cef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:38.197 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.189+0000 7f8a455ce700 1 -- 172.21.15.38:0/2455026883 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561ea35d28b0 msgr2=0x561ea35cef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:46:38.197 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.189+0000 7f8a455ce700 1 --2- 172.21.15.38:0/2455026883 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561ea35d28b0 0x561ea35cef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:38.197 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.189+0000 7f8a455ce700 1 -- 172.21.15.38:0/2455026883 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561ea35d31c0 msgr2=0x561ea35cf4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:46:38.198 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.189+0000 7f8a455ce700 1 --2- 172.21.15.38:0/2455026883 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561ea35d31c0 0x561ea35cf4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:38.198 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.189+0000 7f8a455ce700 1 -- 172.21.15.38:0/2455026883 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561ea34e5210 con 0x561ea35cd060 2023-03-09T14:46:38.198 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.189+0000 7f8a44dcd700 1 --2- 172.21.15.38:0/2455026883 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561ea35d28b0 0x561ea35cef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:46:38.198 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.189+0000 7f8a455ce700 1 --2- 172.21.15.38:0/2455026883 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ea35cd060 0x561ea35cea40 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c021040 tx=0x7f8a3c006f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73d2a8b8473cede2 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:38.198 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.189+0000 7f8a28ff9700 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2023-03-09T14:46:38.198 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.189+0000 7f8a28ff9700 1 -- 172.21.15.38:0/2455026883 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a3c01a040 con 0x561ea35cd060 2023-03-09T14:46:38.198 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.189+0000 7f8a28ff9700 1 -- 172.21.15.38:0/2455026883 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9278+0+0 (secure 0 0 0) 0x7f8a3c02b0c0 con 0x561ea35cd060 2023-03-09T14:46:38.198 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.189+0000 7f8a539bbb80 1 -- 172.21.15.38:0/2455026883 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561ea34dc230 con 0x561ea35cd060 2023-03-09T14:46:38.199 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.189+0000 7f8a28ff9700 1 -- 172.21.15.38:0/2455026883 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8a3c02bcf0 con 0x561ea35cd060 2023-03-09T14:46:38.199 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.189+0000 7f8a539bbb80 10 client.15729 fetch_fsmap learned FSMap version 23 2023-03-09T14:46:38.199 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.189+0000 7f8a539bbb80 1 -- 172.21.15.38:0/2455026883 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x561ea33732d0 con 0x561ea35cd060 2023-03-09T14:46:38.199 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.190+0000 7f8a28ff9700 1 -- 172.21.15.38:0/2455026883 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x561ea34dc230 con 0x561ea35cd060 2023-03-09T14:46:38.199 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.190+0000 7f8a28ff9700 1 -- 172.21.15.38:0/2455026883 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f8a3c040070 con 0x561ea35cd060 2023-03-09T14:46:38.199 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.190+0000 7f8a539bbb80 10 client.15729 fetch_fsmap finished waiting for FSMap version 23 2023-03-09T14:46:38.199 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.190+0000 7f8a539bbb80 10 client.15729 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T14:46:38.200 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.190+0000 7f8a539bbb80 20 client.15729 populate_metadata read hostname 'smithi038' 2023-03-09T14:46:38.200 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.190+0000 7f8a539bbb80 1 --2- 172.21.15.38:0/2455026883 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x561ea363c140 0x561ea363e550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:38.200 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.190+0000 7f8a539bbb80 4 client.15729 mds_command: new command op to 24530 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-03-09T14:46:38.200 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.190+0000 7f8a539bbb80 1 -- 172.21.15.38:0/2455026883 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x561ea34c6240 con 0x561ea363c140 2023-03-09T14:46:38.200 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.190+0000 7f8a44dcd700 1 --2- 172.21.15.38:0/2455026883 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x561ea363c140 0x561ea363e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:38.200 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.190+0000 7f8a28ff9700 10 client.15729 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:46:38.200 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.190+0000 7f8a44dcd700 1 --2- 172.21.15.38:0/2455026883 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x561ea363c140 0x561ea363e550 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a9f0e6a0e5a9c90c server_cookie=b714c33c0beaec54 in_seq=0 out_seq=0 2023-03-09T14:46:38.200 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.191+0000 7f8a28ff9700 1 -- 172.21.15.38:0/2455026883 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x561ea34c6240 con 0x561ea363c140 2023-03-09T14:46:38.201 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.191+0000 7f8a28ff9700 10 client.15729 handle_command_reply: tid=0 2023-03-09T14:46:38.201 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.191+0000 7f8a539bbb80 1 client.15729 shutdown 2023-03-09T14:46:38.201 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.191+0000 7f8a539bbb80 1 -- 172.21.15.38:0/2455026883 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ea35cd060 msgr2=0x561ea35cea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:38.201 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.191+0000 7f8a539bbb80 1 --2- 172.21.15.38:0/2455026883 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ea35cd060 0x561ea35cea40 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c021040 tx=0x7f8a3c006f10 comp rx=0 tx=0).stop 2023-03-09T14:46:38.201 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.191+0000 7f8a539bbb80 1 -- 172.21.15.38:0/2455026883 shutdown_connections 2023-03-09T14:46:38.201 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.191+0000 7f8a539bbb80 1 --2- 172.21.15.38:0/2455026883 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x561ea363c140 0x561ea363e550 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:38.201 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.191+0000 7f8a539bbb80 1 --2- 172.21.15.38:0/2455026883 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561ea35cd060 0x561ea35cea40 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:38.202 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.191+0000 7f8a539bbb80 1 --2- 172.21.15.38:0/2455026883 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561ea35d31c0 0x561ea35cf4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:38.202 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.191+0000 7f8a539bbb80 1 --2- 172.21.15.38:0/2455026883 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561ea35d28b0 0x561ea35cef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:38.202 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.191+0000 7f8a539bbb80 1 -- 172.21.15.38:0/2455026883 >> 172.21.15.38:0/2455026883 conn(0x561ea35c7aa0 msgr2=0x561ea35c6ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:38.202 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.191+0000 7f8a28ff9700 0 client.15729 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:46:38.202 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.192+0000 7f8a539bbb80 1 -- 172.21.15.38:0/2455026883 shutdown_connections 2023-03-09T14:46:38.202 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.192+0000 7f8a539bbb80 1 -- 172.21.15.38:0/2455026883 wait complete. 2023-03-09T14:46:38.202 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.192+0000 7f8a539bbb80 20 client.15729 trim_cache size 0 max 16384 2023-03-09T14:46:38.203 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2023-03-09T14:46:38.203 INFO:teuthology.orchestra.run.smithi038.stdout: "return_code": 0, 2023-03-09T14:46:38.203 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_tag": "2d9ba8a7-4044-4499-bc57-a6fd375da227", 2023-03-09T14:46:38.203 INFO:teuthology.orchestra.run.smithi038.stdout: "mode": "asynchronous" 2023-03-09T14:46:38.203 INFO:teuthology.orchestra.run.smithi038.stdout:} 2023-03-09T14:46:38.204 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.192+0000 7f8a47832700 1 -- 172.21.15.38:0/4206804422 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f8a2c09f560 msgr2=0x7f8a2c0a1a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:38.204 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.192+0000 7f8a47832700 1 --2- 172.21.15.38:0/4206804422 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f8a2c09f560 0x7f8a2c0a1a10 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c0097b0 tx=0x7f8a3c006d20 comp rx=0 tx=0).stop 2023-03-09T14:46:38.204 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.192+0000 7f8a47832700 1 -- 172.21.15.38:0/4206804422 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8a40101280 msgr2=0x7f8a40196680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:38.204 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.192+0000 7f8a47832700 1 --2- 172.21.15.38:0/4206804422 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8a40101280 0x7f8a40196680 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f8a30000c00 tx=0x7f8a30003750 comp rx=0 tx=0).stop 2023-03-09T14:46:38.205 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.192+0000 7f8a47832700 1 -- 172.21.15.38:0/4206804422 shutdown_connections 2023-03-09T14:46:38.205 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.192+0000 7f8a47832700 1 --2- 172.21.15.38:0/4206804422 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f8a2c09f560 0x7f8a2c0a1a10 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:38.205 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.192+0000 7f8a47832700 1 --2- 172.21.15.38:0/4206804422 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8a40105190 0x7f8a4019aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:38.205 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.192+0000 7f8a47832700 1 --2- 172.21.15.38:0/4206804422 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8a401008e0 0x7f8a40196140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:38.205 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.192+0000 7f8a47832700 1 --2- 172.21.15.38:0/4206804422 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8a40101280 0x7f8a40196680 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:38.205 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.192+0000 7f8a47832700 1 -- 172.21.15.38:0/4206804422 >> 172.21.15.38:0/4206804422 conn(0x7f8a40077210 msgr2=0x7f8a400fe7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:38.205 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.192+0000 7f8a47832700 1 -- 172.21.15.38:0/4206804422 shutdown_connections 2023-03-09T14:46:38.206 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.192+0000 7f8a47832700 1 -- 172.21.15.38:0/4206804422 wait complete. 2023-03-09T14:46:38.208 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-03-09T14:46:38.220 INFO:teuthology.orchestra.run.smithi038.stderr:9400000 of 50000000 tuples (18%) done (elapsed 13.36 s, remaining 57.70 s) 2023-03-09T14:46:38.330 INFO:teuthology.orchestra.run.smithi038.stderr:9500000 of 50000000 tuples (19%) done (elapsed 13.47 s, remaining 57.42 s) 2023-03-09T14:46:38.362 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.359+0000 7f8f5ca99700 1 -- 172.21.15.38:0/1357082864 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f580686d0 msgr2=0x7f8f58104a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:38.362 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.359+0000 7f8f5ca99700 1 --2- 172.21.15.38:0/1357082864 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f580686d0 0x7f8f58104a60 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f8f4c0099c0 tx=0x7f8f4c009cd0 comp rx=0 tx=0).stop 2023-03-09T14:46:38.362 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.359+0000 7f8f5ca99700 1 -- 172.21.15.38:0/1357082864 shutdown_connections 2023-03-09T14:46:38.362 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.359+0000 7f8f5ca99700 1 --2- 172.21.15.38:0/1357082864 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8f58105140 0x7f8f5810cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:38.362 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.359+0000 7f8f5ca99700 1 --2- 172.21.15.38:0/1357082864 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f58067d30 0x7f8f58068100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:38.363 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.359+0000 7f8f5ca99700 1 --2- 172.21.15.38:0/1357082864 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f580686d0 0x7f8f58104a60 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:38.363 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.359+0000 7f8f5ca99700 1 -- 172.21.15.38:0/1357082864 >> 172.21.15.38:0/1357082864 conn(0x7f8f58075c60 msgr2=0x7f8f58076060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:38.363 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.359+0000 7f8f5ca99700 1 -- 172.21.15.38:0/1357082864 shutdown_connections 2023-03-09T14:46:38.363 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.359+0000 7f8f5ca99700 1 -- 172.21.15.38:0/1357082864 wait complete. 2023-03-09T14:46:38.365 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.360+0000 7f8f5ca99700 1 Processor -- start 2023-03-09T14:46:38.365 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.360+0000 7f8f5ca99700 1 -- start start 2023-03-09T14:46:38.365 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.361+0000 7f8f5ca99700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8f58067d30 0x7f8f58196040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:38.365 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.361+0000 7f8f5ca99700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f580686d0 0x7f8f58196580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:38.365 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.361+0000 7f8f5ca99700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f58105140 0x7f8f5819a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:38.365 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.361+0000 7f8f5ca99700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f5819b020 con 0x7f8f580686d0 2023-03-09T14:46:38.366 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.361+0000 7f8f5ca99700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f5819b190 con 0x7f8f58067d30 2023-03-09T14:46:38.366 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.361+0000 7f8f5ca99700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f5819b350 con 0x7f8f58105140 2023-03-09T14:46:38.366 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.361+0000 7f8f55d9b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f580686d0 0x7f8f58196580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:38.366 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.361+0000 7f8f55d9b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f580686d0 0x7f8f58196580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39460/0 (socket says 172.21.15.38:39460) 2023-03-09T14:46:38.366 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.361+0000 7f8f55d9b700 1 -- 172.21.15.38:0/2452480527 learned_addr learned my addr 172.21.15.38:0/2452480527 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:46:38.366 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.361+0000 7f8f5659c700 1 --2- 172.21.15.38:0/2452480527 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8f58067d30 0x7f8f58196040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:38.366 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.361+0000 7f8f56d9d700 1 --2- 172.21.15.38:0/2452480527 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f58105140 0x7f8f5819a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:38.367 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.361+0000 7f8f55d9b700 1 -- 172.21.15.38:0/2452480527 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f58105140 msgr2=0x7f8f5819a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:38.367 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.361+0000 7f8f55d9b700 1 --2- 172.21.15.38:0/2452480527 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f58105140 0x7f8f5819a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:38.367 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.361+0000 7f8f55d9b700 1 -- 172.21.15.38:0/2452480527 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8f58067d30 msgr2=0x7f8f58196040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:38.367 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.361+0000 7f8f55d9b700 1 --2- 172.21.15.38:0/2452480527 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8f58067d30 0x7f8f58196040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:38.367 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.361+0000 7f8f55d9b700 1 -- 172.21.15.38:0/2452480527 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f4c0095f0 con 0x7f8f580686d0 2023-03-09T14:46:38.367 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.361+0000 7f8f5659c700 1 --2- 172.21.15.38:0/2452480527 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8f58067d30 0x7f8f58196040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:46:38.367 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.361+0000 7f8f55d9b700 1 --2- 172.21.15.38:0/2452480527 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f580686d0 0x7f8f58196580 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f8f4c0098e0 tx=0x7f8f4c00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:38.367 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.361+0000 7f8f477fe700 1 -- 172.21.15.38:0/2452480527 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f4c01d070 con 0x7f8f580686d0 2023-03-09T14:46:38.368 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.361+0000 7f8f477fe700 1 -- 172.21.15.38:0/2452480527 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8f4c022470 con 0x7f8f580686d0 2023-03-09T14:46:38.368 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.361+0000 7f8f5ca99700 1 -- 172.21.15.38:0/2452480527 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f5810ff10 con 0x7f8f580686d0 2023-03-09T14:46:38.368 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.362+0000 7f8f477fe700 1 -- 172.21.15.38:0/2452480527 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f4c017650 con 0x7f8f580686d0 2023-03-09T14:46:38.368 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.362+0000 7f8f5ca99700 1 -- 172.21.15.38:0/2452480527 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f5810fd80 con 0x7f8f580686d0 2023-03-09T14:46:38.370 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.364+0000 7f8f477fe700 1 -- 172.21.15.38:0/2452480527 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 23) v1 ==== 133335+0+0 (secure 0 0 0) 0x7f8f4c029030 con 0x7f8f580686d0 2023-03-09T14:46:38.370 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.365+0000 7f8f477fe700 1 --2- 172.21.15.38:0/2452480527 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f8f4009f560 0x7f8f400a1a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:38.370 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.365+0000 7f8f477fe700 1 -- 172.21.15.38:0/2452480527 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9278+0+0 (secure 0 0 0) 0x7f8f4c0cb1b0 con 0x7f8f580686d0 2023-03-09T14:46:38.370 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.365+0000 7f8f5659c700 1 --2- 172.21.15.38:0/2452480527 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f8f4009f560 0x7f8f400a1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:38.371 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.366+0000 7f8f5659c700 1 --2- 172.21.15.38:0/2452480527 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f8f4009f560 0x7f8f400a1a10 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8f48011490 tx=0x7f8f48009260 comp rx=0 tx=0).ready entity=mgr.15558 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:38.371 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.366+0000 7f8f68c22b80 1 Processor -- start 2023-03-09T14:46:38.371 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.366+0000 7f8f68c22b80 1 -- start start 2023-03-09T14:46:38.371 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.366+0000 7f8f68c22b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557e50431830 0x557e50431c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:38.371 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.366+0000 7f8f68c22b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557e50432140 0x557e5042baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:38.371 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.366+0000 7f8f68c22b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557e5042bfe0 0x557e5042c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:38.371 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.366+0000 7f8f68c22b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e50315da0 con 0x557e50431830 2023-03-09T14:46:38.372 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.366+0000 7f8f68c22b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e50384780 con 0x557e5042bfe0 2023-03-09T14:46:38.372 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.366+0000 7f8f68c22b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e5037dcb0 con 0x557e50432140 2023-03-09T14:46:38.372 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.366+0000 7f8f5659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557e50431830 0x557e50431c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:38.372 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.366+0000 7f8f56d9d700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557e5042bfe0 0x557e5042c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:38.372 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.366+0000 7f8f55d9b700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557e50432140 0x557e5042baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:38.372 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.366+0000 7f8f5659c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557e50431830 0x557e50431c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39466/0 (socket says 172.21.15.38:39466) 2023-03-09T14:46:38.372 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.366+0000 7f8f5659c700 1 -- 172.21.15.38:0/3604218559 learned_addr learned my addr 172.21.15.38:0/3604218559 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:46:38.372 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.366+0000 7f8f5659c700 1 -- 172.21.15.38:0/3604218559 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557e50432140 msgr2=0x557e5042baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:46:38.373 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.366+0000 7f8f5659c700 1 --2- 172.21.15.38:0/3604218559 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557e50432140 0x557e5042baa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:38.373 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.366+0000 7f8f5659c700 1 -- 172.21.15.38:0/3604218559 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557e5042bfe0 msgr2=0x557e5042c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:46:38.373 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.366+0000 7f8f5659c700 1 --2- 172.21.15.38:0/3604218559 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557e5042bfe0 0x557e5042c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:38.373 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.367+0000 7f8f5659c700 1 -- 172.21.15.38:0/3604218559 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557e50344190 con 0x557e50431830 2023-03-09T14:46:38.373 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.367+0000 7f8f5659c700 1 --2- 172.21.15.38:0/3604218559 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557e50431830 0x557e50431c00 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f8f480043c0 tx=0x7f8f48014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f26bf99dd3f5cc79 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:38.373 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.367+0000 7f8f44ff9700 1 -- 172.21.15.38:0/3604218559 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f48003d10 con 0x557e50431830 2023-03-09T14:46:38.373 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.367+0000 7f8f44ff9700 1 -- 172.21.15.38:0/3604218559 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8f48003e70 con 0x557e50431830 2023-03-09T14:46:38.374 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.367+0000 7f8f44ff9700 1 -- 172.21.15.38:0/3604218559 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f4800e7a0 con 0x557e50431830 2023-03-09T14:46:38.374 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.368+0000 7f8f68c22b80 1 -- 172.21.15.38:0/3604218559 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557e50431830 msgr2=0x557e50431c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:38.374 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.368+0000 7f8f68c22b80 1 --2- 172.21.15.38:0/3604218559 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557e50431830 0x557e50431c00 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f8f480043c0 tx=0x7f8f48014000 comp rx=0 tx=0).stop 2023-03-09T14:46:38.374 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.368+0000 7f8f68c22b80 1 -- 172.21.15.38:0/3604218559 shutdown_connections 2023-03-09T14:46:38.374 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.368+0000 7f8f68c22b80 1 --2- 172.21.15.38:0/3604218559 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557e50431830 0x557e50431c00 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:38.374 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.368+0000 7f8f68c22b80 1 --2- 172.21.15.38:0/3604218559 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557e5042bfe0 0x557e5042c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:38.374 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.368+0000 7f8f68c22b80 1 --2- 172.21.15.38:0/3604218559 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557e50432140 0x557e5042baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:38.374 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.368+0000 7f8f68c22b80 1 -- 172.21.15.38:0/3604218559 >> 172.21.15.38:0/3604218559 conn(0x557e50426a20 msgr2=0x557e50426df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:38.375 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.368+0000 7f8f68c22b80 1 -- 172.21.15.38:0/3604218559 shutdown_connections 2023-03-09T14:46:38.375 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.368+0000 7f8f68c22b80 1 -- 172.21.15.38:0/3604218559 wait complete. 2023-03-09T14:46:38.375 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.368+0000 7f8f68c22b80 1 -- start start 2023-03-09T14:46:38.375 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.368+0000 7f8f68c22b80 1 Processor -- start 2023-03-09T14:46:38.377 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.368+0000 7f8f68c22b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557e5042bfe0 0x557e50438120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:38.378 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.368+0000 7f8f68c22b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557e50431830 0x557e504353a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:38.378 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.368+0000 7f8f68c22b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557e50432140 0x557e50435a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:38.378 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.368+0000 7f8f68c22b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e50201920 con 0x557e50431830 2023-03-09T14:46:38.378 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.368+0000 7f8f68c22b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e504361b0 con 0x557e50432140 2023-03-09T14:46:38.378 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.368+0000 7f8f68c22b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e50441c40 con 0x557e5042bfe0 2023-03-09T14:46:38.378 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.369+0000 7f8f68c22b80 10 client.0 _ll_register_callbacks cb 0x557e50323520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:46:38.378 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.369+0000 7f8f55d9b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557e50431830 0x557e504353a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:38.379 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.369+0000 7f8f56d9d700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557e50432140 0x557e50435a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:38.379 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.369+0000 7f8f55d9b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557e50431830 0x557e504353a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39474/0 (socket says 172.21.15.38:39474) 2023-03-09T14:46:38.379 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.369+0000 7f8f56d9d700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557e50432140 0x557e50435a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.38:48666/0 (socket says 172.21.15.38:48666) 2023-03-09T14:46:38.379 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.369+0000 7f8f55d9b700 1 -- 172.21.15.38:0/478577111 learned_addr learned my addr 172.21.15.38:0/478577111 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:46:38.379 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.369+0000 7f8f5659c700 1 --2- 172.21.15.38:0/478577111 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557e5042bfe0 0x557e50438120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:38.379 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.369+0000 7f8f55d9b700 1 -- 172.21.15.38:0/478577111 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557e5042bfe0 msgr2=0x557e50438120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:46:38.379 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.369+0000 7f8f55d9b700 1 --2- 172.21.15.38:0/478577111 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557e5042bfe0 0x557e50438120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:38.380 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.369+0000 7f8f55d9b700 1 -- 172.21.15.38:0/478577111 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557e50432140 msgr2=0x557e50435a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:46:38.380 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.369+0000 7f8f55d9b700 1 --2- 172.21.15.38:0/478577111 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557e50432140 0x557e50435a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:38.380 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.369+0000 7f8f55d9b700 1 -- 172.21.15.38:0/478577111 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f48017070 con 0x557e50431830 2023-03-09T14:46:38.380 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.369+0000 7f8f55d9b700 1 --2- 172.21.15.38:0/478577111 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557e50431830 0x557e504353a0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f8f4c095280 tx=0x7f8f4c022b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4ac347e18a7110f server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:38.380 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.369+0000 7f8f3dffb700 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2023-03-09T14:46:38.380 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.369+0000 7f8f3dffb700 1 -- 172.21.15.38:0/478577111 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f4c098d80 con 0x557e50431830 2023-03-09T14:46:38.380 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.369+0000 7f8f3dffb700 1 -- 172.21.15.38:0/478577111 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9278+0+0 (secure 0 0 0) 0x7f8f4c0a0ee0 con 0x557e50431830 2023-03-09T14:46:38.380 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.369+0000 7f8f68c22b80 1 -- 172.21.15.38:0/478577111 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557e5034b190 con 0x557e50431830 2023-03-09T14:46:38.381 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.369+0000 7f8f3dffb700 1 -- 172.21.15.38:0/478577111 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8f4c098270 con 0x557e50431830 2023-03-09T14:46:38.381 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.369+0000 7f8f68c22b80 10 client.15741 fetch_fsmap learned FSMap version 23 2023-03-09T14:46:38.381 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.370+0000 7f8f68c22b80 1 -- 172.21.15.38:0/478577111 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x557e50194c60 con 0x557e50431830 2023-03-09T14:46:38.381 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.370+0000 7f8f3dffb700 1 -- 172.21.15.38:0/478577111 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x557e5034b190 con 0x557e50431830 2023-03-09T14:46:38.381 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.370+0000 7f8f3dffb700 1 -- 172.21.15.38:0/478577111 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f8f4c00fb40 con 0x557e50431830 2023-03-09T14:46:38.381 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.370+0000 7f8f68c22b80 10 client.15741 fetch_fsmap finished waiting for FSMap version 23 2023-03-09T14:46:38.381 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.370+0000 7f8f68c22b80 10 client.15741 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T14:46:38.382 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.370+0000 7f8f68c22b80 20 client.15741 populate_metadata read hostname 'smithi038' 2023-03-09T14:46:38.382 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.370+0000 7f8f68c22b80 1 --2- 172.21.15.38:0/478577111 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x557e50445970 0x557e50447d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:38.382 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.370+0000 7f8f68c22b80 4 client.15741 mds_command: new command op to 24530 tid=0[{"prefix": "get_command_descriptions"}] 2023-03-09T14:46:38.382 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.370+0000 7f8f68c22b80 1 -- 172.21.15.38:0/478577111 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557e50201f60 con 0x557e50445970 2023-03-09T14:46:38.382 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.370+0000 7f8f5659c700 1 --2- 172.21.15.38:0/478577111 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x557e50445970 0x557e50447d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:38.382 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.371+0000 7f8f3dffb700 10 client.15741 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:46:38.382 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.371+0000 7f8f5659c700 1 --2- 172.21.15.38:0/478577111 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x557e50445970 0x557e50447d80 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f3bc4385cafce2e2 server_cookie=80deca964213a117 in_seq=0 out_seq=0 2023-03-09T14:46:38.382 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.372+0000 7f8f3dffb700 1 -- 172.21.15.38:0/478577111 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18278 (crc 0 0 0) 0x557e50201f60 con 0x557e50445970 2023-03-09T14:46:38.383 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.372+0000 7f8f3dffb700 10 client.15741 handle_command_reply: tid=0 2023-03-09T14:46:38.383 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.372+0000 7f8f68c22b80 1 client.15741 shutdown 2023-03-09T14:46:38.383 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.372+0000 7f8f68c22b80 1 -- 172.21.15.38:0/478577111 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557e50431830 msgr2=0x557e504353a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:38.383 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.372+0000 7f8f68c22b80 1 --2- 172.21.15.38:0/478577111 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557e50431830 0x557e504353a0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f8f4c095280 tx=0x7f8f4c022b30 comp rx=0 tx=0).stop 2023-03-09T14:46:38.383 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.372+0000 7f8f68c22b80 1 -- 172.21.15.38:0/478577111 shutdown_connections 2023-03-09T14:46:38.383 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.372+0000 7f8f68c22b80 1 --2- 172.21.15.38:0/478577111 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x557e50445970 0x557e50447d80 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:38.383 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.372+0000 7f8f68c22b80 1 --2- 172.21.15.38:0/478577111 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557e50432140 0x557e50435a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:38.383 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.372+0000 7f8f68c22b80 1 --2- 172.21.15.38:0/478577111 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557e5042bfe0 0x557e50438120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:38.384 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.372+0000 7f8f3dffb700 0 client.15741 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:46:38.384 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.372+0000 7f8f68c22b80 1 --2- 172.21.15.38:0/478577111 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557e50431830 0x557e504353a0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:38.384 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.372+0000 7f8f68c22b80 1 -- 172.21.15.38:0/478577111 >> 172.21.15.38:0/478577111 conn(0x557e50426a20 msgr2=0x557e50425c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:38.384 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.372+0000 7f8f68c22b80 1 -- 172.21.15.38:0/478577111 shutdown_connections 2023-03-09T14:46:38.384 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.372+0000 7f8f68c22b80 1 -- 172.21.15.38:0/478577111 wait complete. 2023-03-09T14:46:38.384 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.372+0000 7f8f68c22b80 20 client.15741 trim_cache size 0 max 16384 2023-03-09T14:46:38.395 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.393+0000 7f8f68c22b80 1 Processor -- start 2023-03-09T14:46:38.396 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.393+0000 7f8f68c22b80 1 -- start start 2023-03-09T14:46:38.396 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.393+0000 7f8f68c22b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557e5042bfe0 0x557e50437cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:38.396 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.393+0000 7f8f68c22b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557e50431830 0x557e5042f530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:38.396 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.393+0000 7f8f68c22b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557e50432140 0x557e5042fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:38.396 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.393+0000 7f8f68c22b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e50427f00 con 0x557e50431830 2023-03-09T14:46:38.396 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.393+0000 7f8f68c22b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e50442760 con 0x557e5042bfe0 2023-03-09T14:46:38.397 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.393+0000 7f8f68c22b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e504362f0 con 0x557e50432140 2023-03-09T14:46:38.400 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.394+0000 7f8f55d9b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557e50431830 0x557e5042f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:38.400 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.394+0000 7f8f5659c700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557e5042bfe0 0x557e50437cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:38.400 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.394+0000 7f8f55d9b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557e50431830 0x557e5042f530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39482/0 (socket says 172.21.15.38:39482) 2023-03-09T14:46:38.401 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.394+0000 7f8f55d9b700 1 -- 172.21.15.38:0/3476112990 learned_addr learned my addr 172.21.15.38:0/3476112990 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:46:38.401 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.394+0000 7f8f56d9d700 1 --2- 172.21.15.38:0/3476112990 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557e50432140 0x557e5042fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:38.401 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.394+0000 7f8f55d9b700 1 -- 172.21.15.38:0/3476112990 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557e50432140 msgr2=0x557e5042fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:46:38.401 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.394+0000 7f8f55d9b700 1 --2- 172.21.15.38:0/3476112990 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557e50432140 0x557e5042fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:38.401 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.394+0000 7f8f55d9b700 1 -- 172.21.15.38:0/3476112990 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557e5042bfe0 msgr2=0x557e50437cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:46:38.401 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.394+0000 7f8f55d9b700 1 --2- 172.21.15.38:0/3476112990 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557e5042bfe0 0x557e50437cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:38.401 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.394+0000 7f8f55d9b700 1 -- 172.21.15.38:0/3476112990 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557e5044c9b0 con 0x557e50431830 2023-03-09T14:46:38.402 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.394+0000 7f8f5659c700 1 --2- 172.21.15.38:0/3476112990 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557e5042bfe0 0x557e50437cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:46:38.402 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.394+0000 7f8f55d9b700 1 --2- 172.21.15.38:0/3476112990 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557e50431830 0x557e5042f530 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f8f4c0957a0 tx=0x7f8f4c00b5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88e3a3648e6c0082 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:38.402 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.394+0000 7f8f457fa700 1 -- 172.21.15.38:0/3476112990 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f4c097db0 con 0x557e50431830 2023-03-09T14:46:38.402 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.394+0000 7f8f457fa700 1 -- 172.21.15.38:0/3476112990 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f8f4c0a0ee0 con 0x557e50431830 2023-03-09T14:46:38.402 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.394+0000 7f8f457fa700 1 -- 172.21.15.38:0/3476112990 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f4c0a0cb0 con 0x557e50431830 2023-03-09T14:46:38.402 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.395+0000 7f8f68c22b80 1 -- 172.21.15.38:0/3476112990 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557e50431830 msgr2=0x557e5042f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:38.402 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.395+0000 7f8f68c22b80 1 --2- 172.21.15.38:0/3476112990 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557e50431830 0x557e5042f530 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f8f4c0957a0 tx=0x7f8f4c00b5c0 comp rx=0 tx=0).stop 2023-03-09T14:46:38.402 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.395+0000 7f8f68c22b80 1 -- 172.21.15.38:0/3476112990 shutdown_connections 2023-03-09T14:46:38.403 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.395+0000 7f8f68c22b80 1 --2- 172.21.15.38:0/3476112990 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557e50432140 0x557e5042fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:38.403 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.395+0000 7f8f68c22b80 1 --2- 172.21.15.38:0/3476112990 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557e5042bfe0 0x557e50437cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:38.403 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.395+0000 7f8f68c22b80 1 --2- 172.21.15.38:0/3476112990 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557e50431830 0x557e5042f530 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:38.403 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.395+0000 7f8f68c22b80 1 -- 172.21.15.38:0/3476112990 >> 172.21.15.38:0/3476112990 conn(0x557e50426a20 msgr2=0x557e50442050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:38.403 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.395+0000 7f8f68c22b80 1 -- 172.21.15.38:0/3476112990 shutdown_connections 2023-03-09T14:46:38.403 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.395+0000 7f8f68c22b80 1 -- 172.21.15.38:0/3476112990 wait complete. 2023-03-09T14:46:38.403 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.396+0000 7f8f68c22b80 1 -- start start 2023-03-09T14:46:38.404 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.396+0000 7f8f68c22b80 1 Processor -- start 2023-03-09T14:46:38.404 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.396+0000 7f8f68c22b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557e5042bfe0 0x557e5042e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:38.404 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.396+0000 7f8f68c22b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557e50431830 0x557e5042d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:38.404 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.396+0000 7f8f68c22b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557e50432140 0x557e5042d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:38.404 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.396+0000 7f8f68c22b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e5042ee60 con 0x557e50432140 2023-03-09T14:46:38.404 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.396+0000 7f8f68c22b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e5042efd0 con 0x557e50431830 2023-03-09T14:46:38.404 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.396+0000 7f8f68c22b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x557e5042dd10 con 0x557e5042bfe0 2023-03-09T14:46:38.404 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.396+0000 7f8f68c22b80 10 client.0 _ll_register_callbacks cb 0x557e50323520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:46:38.405 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.396+0000 7f8f56d9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557e50432140 0x557e5042d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:38.405 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.396+0000 7f8f56d9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557e50432140 0x557e5042d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39490/0 (socket says 172.21.15.38:39490) 2023-03-09T14:46:38.405 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.396+0000 7f8f5659c700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557e5042bfe0 0x557e5042e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:38.405 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.396+0000 7f8f56d9d700 1 -- 172.21.15.38:0/3234224331 learned_addr learned my addr 172.21.15.38:0/3234224331 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:46:38.405 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.396+0000 7f8f55d9b700 1 --2- 172.21.15.38:0/3234224331 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557e50431830 0x557e5042d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:38.405 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.397+0000 7f8f56d9d700 1 -- 172.21.15.38:0/3234224331 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557e5042bfe0 msgr2=0x557e5042e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:46:38.405 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.397+0000 7f8f56d9d700 1 --2- 172.21.15.38:0/3234224331 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557e5042bfe0 0x557e5042e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:38.406 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.397+0000 7f8f56d9d700 1 -- 172.21.15.38:0/3234224331 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557e50431830 msgr2=0x557e5042d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:46:38.406 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.397+0000 7f8f56d9d700 1 --2- 172.21.15.38:0/3234224331 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557e50431830 0x557e5042d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:38.406 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.397+0000 7f8f56d9d700 1 -- 172.21.15.38:0/3234224331 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557e50344190 con 0x557e50432140 2023-03-09T14:46:38.406 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.397+0000 7f8f56d9d700 1 --2- 172.21.15.38:0/3234224331 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557e50432140 0x557e5042d7d0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f8f50009790 tx=0x7f8f50007e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=648b92fefc92aa17 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:46:38.406 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.397+0000 7f8f3dffb700 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2023-03-09T14:46:38.406 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.397+0000 7f8f3dffb700 1 -- 172.21.15.38:0/3234224331 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f5001c070 con 0x557e50432140 2023-03-09T14:46:38.406 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.397+0000 7f8f3dffb700 1 -- 172.21.15.38:0/3234224331 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9278+0+0 (secure 0 0 0) 0x7f8f50003a00 con 0x557e50432140 2023-03-09T14:46:38.406 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.397+0000 7f8f68c22b80 1 -- 172.21.15.38:0/3234224331 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557e5033b1b0 con 0x557e50432140 2023-03-09T14:46:38.407 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.397+0000 7f8f3dffb700 1 -- 172.21.15.38:0/3234224331 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8f5000e040 con 0x557e50432140 2023-03-09T14:46:38.407 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.397+0000 7f8f68c22b80 10 client.15747 fetch_fsmap learned FSMap version 23 2023-03-09T14:46:38.407 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.397+0000 7f8f68c22b80 1 -- 172.21.15.38:0/3234224331 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x557e50194c60 con 0x557e50432140 2023-03-09T14:46:38.407 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.398+0000 7f8f3dffb700 1 -- 172.21.15.38:0/3234224331 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x557e5033b1b0 con 0x557e50432140 2023-03-09T14:46:38.407 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.398+0000 7f8f3dffb700 1 -- 172.21.15.38:0/3234224331 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f8f50020b60 con 0x557e50432140 2023-03-09T14:46:38.407 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.398+0000 7f8f68c22b80 10 client.15747 fetch_fsmap finished waiting for FSMap version 23 2023-03-09T14:46:38.407 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.398+0000 7f8f68c22b80 10 client.15747 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T14:46:38.408 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.398+0000 7f8f68c22b80 20 client.15747 populate_metadata read hostname 'smithi038' 2023-03-09T14:46:38.408 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.398+0000 7f8f68c22b80 1 --2- 172.21.15.38:0/3234224331 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x557e5049ee20 0x557e504a1230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:46:38.408 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.398+0000 7f8f68c22b80 4 client.15747 mds_command: new command op to 24530 tid=0[{"prefix": "scrub status"}] 2023-03-09T14:46:38.408 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.398+0000 7f8f68c22b80 1 -- 172.21.15.38:0/3234224331 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x557e503251c0 con 0x557e5049ee20 2023-03-09T14:46:38.409 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.398+0000 7f8f5659c700 1 --2- 172.21.15.38:0/3234224331 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x557e5049ee20 0x557e504a1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:46:38.409 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.398+0000 7f8f3dffb700 10 client.15747 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:46:38.409 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.398+0000 7f8f5659c700 1 --2- 172.21.15.38:0/3234224331 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x557e5049ee20 0x557e504a1230 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=baa858c69a1a4b2e server_cookie=721362996510ca21 in_seq=0 out_seq=0 2023-03-09T14:46:38.409 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.398+0000 7f8f3dffb700 1 -- 172.21.15.38:0/3234224331 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x557e503251c0 con 0x557e5049ee20 2023-03-09T14:46:38.410 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.399+0000 7f8f3dffb700 10 client.15747 handle_command_reply: tid=0 2023-03-09T14:46:38.410 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.399+0000 7f8f68c22b80 1 client.15747 shutdown 2023-03-09T14:46:38.410 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.399+0000 7f8f68c22b80 1 -- 172.21.15.38:0/3234224331 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557e50432140 msgr2=0x557e5042d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:38.410 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.399+0000 7f8f68c22b80 1 --2- 172.21.15.38:0/3234224331 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557e50432140 0x557e5042d7d0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f8f50009790 tx=0x7f8f50007e30 comp rx=0 tx=0).stop 2023-03-09T14:46:38.410 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.399+0000 7f8f68c22b80 1 -- 172.21.15.38:0/3234224331 shutdown_connections 2023-03-09T14:46:38.410 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.399+0000 7f8f68c22b80 1 --2- 172.21.15.38:0/3234224331 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x557e5049ee20 0x557e504a1230 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:38.410 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.399+0000 7f8f68c22b80 1 --2- 172.21.15.38:0/3234224331 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557e50432140 0x557e5042d7d0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:38.411 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.399+0000 7f8f68c22b80 1 --2- 172.21.15.38:0/3234224331 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557e5042bfe0 0x557e5042e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:38.411 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.399+0000 7f8f3dffb700 0 client.15747 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:46:38.411 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.399+0000 7f8f68c22b80 1 --2- 172.21.15.38:0/3234224331 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557e50431830 0x557e5042d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:38.411 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.399+0000 7f8f68c22b80 1 -- 172.21.15.38:0/3234224331 >> 172.21.15.38:0/3234224331 conn(0x557e50426a20 msgr2=0x557e5042f530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:38.411 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.399+0000 7f8f68c22b80 1 -- 172.21.15.38:0/3234224331 shutdown_connections 2023-03-09T14:46:38.411 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.399+0000 7f8f68c22b80 1 -- 172.21.15.38:0/3234224331 wait complete. 2023-03-09T14:46:38.411 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.399+0000 7f8f68c22b80 20 client.15747 trim_cache size 0 max 16384 2023-03-09T14:46:38.412 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2023-03-09T14:46:38.412 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "IDLE (waiting for more scrubs)", 2023-03-09T14:46:38.412 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": { 2023-03-09T14:46:38.412 INFO:teuthology.orchestra.run.smithi038.stdout: "2d9ba8a7-4044-4499-bc57-a6fd375da227": { 2023-03-09T14:46:38.412 INFO:teuthology.orchestra.run.smithi038.stdout: "path": "/", 2023-03-09T14:46:38.412 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "2d9ba8a7-4044-4499-bc57-a6fd375da227", 2023-03-09T14:46:38.413 INFO:teuthology.orchestra.run.smithi038.stdout: "options": "recursive,force" 2023-03-09T14:46:38.413 INFO:teuthology.orchestra.run.smithi038.stdout: } 2023-03-09T14:46:38.413 INFO:teuthology.orchestra.run.smithi038.stdout: } 2023-03-09T14:46:38.413 INFO:teuthology.orchestra.run.smithi038.stdout:} 2023-03-09T14:46:38.414 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.399+0000 7f8f5ca99700 1 -- 172.21.15.38:0/2452480527 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f8f4009f560 msgr2=0x7f8f400a1a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:38.414 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.399+0000 7f8f5ca99700 1 --2- 172.21.15.38:0/2452480527 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f8f4009f560 0x7f8f400a1a10 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8f48011490 tx=0x7f8f48009260 comp rx=0 tx=0).stop 2023-03-09T14:46:38.414 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.399+0000 7f8f5ca99700 1 -- 172.21.15.38:0/2452480527 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f580686d0 msgr2=0x7f8f58196580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:46:38.414 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.399+0000 7f8f5ca99700 1 --2- 172.21.15.38:0/2452480527 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f580686d0 0x7f8f58196580 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f8f4c0098e0 tx=0x7f8f4c00f690 comp rx=0 tx=0).stop 2023-03-09T14:46:38.415 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.399+0000 7f8f5ca99700 1 -- 172.21.15.38:0/2452480527 shutdown_connections 2023-03-09T14:46:38.415 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.399+0000 7f8f5ca99700 1 --2- 172.21.15.38:0/2452480527 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f8f4009f560 0x7f8f400a1a10 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:38.415 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.399+0000 7f8f5ca99700 1 --2- 172.21.15.38:0/2452480527 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8f58105140 0x7f8f5819a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:38.415 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.399+0000 7f8f5ca99700 1 --2- 172.21.15.38:0/2452480527 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8f58067d30 0x7f8f58196040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:38.415 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.399+0000 7f8f5ca99700 1 --2- 172.21.15.38:0/2452480527 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8f580686d0 0x7f8f58196580 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:46:38.415 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.399+0000 7f8f5ca99700 1 -- 172.21.15.38:0/2452480527 >> 172.21.15.38:0/2452480527 conn(0x7f8f58075c60 msgr2=0x7f8f580fc940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:46:38.415 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.400+0000 7f8f5ca99700 1 -- 172.21.15.38:0/2452480527 shutdown_connections 2023-03-09T14:46:38.416 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:46:38.400+0000 7f8f5ca99700 1 -- 172.21.15.38:0/2452480527 wait complete. 2023-03-09T14:46:38.416 INFO:tasks.cephfs.filesystem:scrub status for tag:2d9ba8a7-4044-4499-bc57-a6fd375da227 - {'path': '/', 'tag': '2d9ba8a7-4044-4499-bc57-a6fd375da227', 'options': 'recursive,force'} 2023-03-09T14:46:38.499 INFO:teuthology.orchestra.run.smithi038.stderr:9600000 of 50000000 tuples (19%) done (elapsed 13.64 s, remaining 57.39 s) 2023-03-09T14:46:38.645 INFO:teuthology.orchestra.run.smithi038.stderr:9700000 of 50000000 tuples (19%) done (elapsed 13.78 s, remaining 57.26 s) 2023-03-09T14:46:38.744 INFO:teuthology.orchestra.run.smithi038.stderr:9800000 of 50000000 tuples (19%) done (elapsed 13.88 s, remaining 56.95 s) 2023-03-09T14:46:38.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:38 smithi038 ceph-mon[92422]: scrub queued for path: / 2023-03-09T14:46:38.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:38 smithi038 ceph-mon[92422]: scrub summary: idle+waiting paths [/] 2023-03-09T14:46:38.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:38 smithi038 ceph-mon[92422]: scrub summary: active paths [/] 2023-03-09T14:46:38.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:38 smithi038 ceph-mon[92422]: scrub summary: idle+waiting paths [/] 2023-03-09T14:46:38.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:38 smithi038 ceph-mon[92422]: scrub summary: idle 2023-03-09T14:46:38.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:38 smithi122 ceph-mon[93869]: scrub queued for path: / 2023-03-09T14:46:38.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:38 smithi122 ceph-mon[93869]: scrub summary: idle+waiting paths [/] 2023-03-09T14:46:38.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:38 smithi122 ceph-mon[93869]: scrub summary: active paths [/] 2023-03-09T14:46:38.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:38 smithi122 ceph-mon[93869]: scrub summary: idle+waiting paths [/] 2023-03-09T14:46:38.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:38 smithi122 ceph-mon[93869]: scrub summary: idle 2023-03-09T14:46:38.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:38 smithi181 ceph-mon[94189]: scrub queued for path: / 2023-03-09T14:46:38.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:38 smithi181 ceph-mon[94189]: scrub summary: idle+waiting paths [/] 2023-03-09T14:46:38.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:38 smithi181 ceph-mon[94189]: scrub summary: active paths [/] 2023-03-09T14:46:38.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:38 smithi181 ceph-mon[94189]: scrub summary: idle+waiting paths [/] 2023-03-09T14:46:38.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:38 smithi181 ceph-mon[94189]: scrub summary: idle 2023-03-09T14:46:39.525 INFO:teuthology.orchestra.run.smithi038.stderr:9900000 of 50000000 tuples (19%) done (elapsed 14.66 s, remaining 59.39 s) 2023-03-09T14:46:39.631 INFO:teuthology.orchestra.run.smithi038.stderr:10000000 of 50000000 tuples (20%) done (elapsed 14.77 s, remaining 59.08 s) 2023-03-09T14:46:39.722 INFO:teuthology.orchestra.run.smithi038.stderr:10100000 of 50000000 tuples (20%) done (elapsed 14.86 s, remaining 58.70 s) 2023-03-09T14:46:39.813 INFO:teuthology.orchestra.run.smithi038.stderr:10200000 of 50000000 tuples (20%) done (elapsed 14.95 s, remaining 58.34 s) 2023-03-09T14:46:39.939 INFO:teuthology.orchestra.run.smithi038.stderr:10300000 of 50000000 tuples (20%) done (elapsed 15.08 s, remaining 58.11 s) 2023-03-09T14:46:39.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:39 smithi122 ceph-mon[93869]: pgmap v15: 129 pgs: 129 active+clean; 1.5 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 183 MiB/s wr, 91 op/s 2023-03-09T14:46:39.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:39 smithi181 ceph-mon[94189]: pgmap v15: 129 pgs: 129 active+clean; 1.5 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 183 MiB/s wr, 91 op/s 2023-03-09T14:46:40.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:39 smithi038 ceph-mon[92422]: pgmap v15: 129 pgs: 129 active+clean; 1.5 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 183 MiB/s wr, 91 op/s 2023-03-09T14:46:40.007 INFO:teuthology.orchestra.run.smithi038.stderr:10400000 of 50000000 tuples (20%) done (elapsed 15.14 s, remaining 57.67 s) 2023-03-09T14:46:40.101 INFO:teuthology.orchestra.run.smithi038.stderr:10500000 of 50000000 tuples (21%) done (elapsed 15.24 s, remaining 57.33 s) 2023-03-09T14:46:40.201 INFO:teuthology.orchestra.run.smithi038.stderr:10600000 of 50000000 tuples (21%) done (elapsed 15.34 s, remaining 57.01 s) 2023-03-09T14:46:40.313 INFO:teuthology.orchestra.run.smithi038.stderr:10700000 of 50000000 tuples (21%) done (elapsed 15.45 s, remaining 56.75 s) 2023-03-09T14:46:40.407 INFO:teuthology.orchestra.run.smithi038.stderr:10800000 of 50000000 tuples (21%) done (elapsed 15.54 s, remaining 56.42 s) 2023-03-09T14:46:40.517 INFO:teuthology.orchestra.run.smithi038.stderr:10900000 of 50000000 tuples (21%) done (elapsed 15.66 s, remaining 56.16 s) 2023-03-09T14:46:40.613 INFO:teuthology.orchestra.run.smithi038.stderr:11000000 of 50000000 tuples (22%) done (elapsed 15.75 s, remaining 55.84 s) 2023-03-09T14:46:40.713 INFO:teuthology.orchestra.run.smithi038.stderr:11100000 of 50000000 tuples (22%) done (elapsed 15.85 s, remaining 55.55 s) 2023-03-09T14:46:40.814 INFO:teuthology.orchestra.run.smithi038.stderr:11200000 of 50000000 tuples (22%) done (elapsed 15.95 s, remaining 55.26 s) 2023-03-09T14:46:40.887 INFO:teuthology.orchestra.run.smithi038.stderr:11300000 of 50000000 tuples (22%) done (elapsed 16.02 s, remaining 54.88 s) 2023-03-09T14:46:41.004 INFO:teuthology.orchestra.run.smithi038.stderr:11400000 of 50000000 tuples (22%) done (elapsed 16.14 s, remaining 54.66 s) 2023-03-09T14:46:41.090 INFO:teuthology.orchestra.run.smithi038.stderr:11500000 of 50000000 tuples (23%) done (elapsed 16.23 s, remaining 54.33 s) 2023-03-09T14:46:41.190 INFO:teuthology.orchestra.run.smithi038.stderr:11600000 of 50000000 tuples (23%) done (elapsed 16.33 s, remaining 54.05 s) 2023-03-09T14:46:41.287 INFO:teuthology.orchestra.run.smithi038.stderr:11700000 of 50000000 tuples (23%) done (elapsed 16.42 s, remaining 53.77 s) 2023-03-09T14:46:41.396 INFO:teuthology.orchestra.run.smithi038.stderr:11800000 of 50000000 tuples (23%) done (elapsed 16.53 s, remaining 53.53 s) 2023-03-09T14:46:41.493 INFO:teuthology.orchestra.run.smithi038.stderr:11900000 of 50000000 tuples (23%) done (elapsed 16.63 s, remaining 53.25 s) 2023-03-09T14:46:41.600 INFO:teuthology.orchestra.run.smithi038.stderr:12000000 of 50000000 tuples (24%) done (elapsed 16.74 s, remaining 53.00 s) 2023-03-09T14:46:41.690 INFO:teuthology.orchestra.run.smithi038.stderr:12100000 of 50000000 tuples (24%) done (elapsed 16.83 s, remaining 52.71 s) 2023-03-09T14:46:41.780 INFO:teuthology.orchestra.run.smithi038.stderr:12200000 of 50000000 tuples (24%) done (elapsed 16.92 s, remaining 52.42 s) 2023-03-09T14:46:41.877 INFO:teuthology.orchestra.run.smithi038.stderr:12300000 of 50000000 tuples (24%) done (elapsed 17.02 s, remaining 52.15 s) 2023-03-09T14:46:41.968 INFO:teuthology.orchestra.run.smithi038.stderr:12400000 of 50000000 tuples (24%) done (elapsed 17.11 s, remaining 51.87 s) 2023-03-09T14:46:41.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:41 smithi122 ceph-mon[93869]: pgmap v16: 129 pgs: 129 active+clean; 1.5 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 183 MiB/s wr, 91 op/s 2023-03-09T14:46:41.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:41 smithi181 ceph-mon[94189]: pgmap v16: 129 pgs: 129 active+clean; 1.5 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 183 MiB/s wr, 91 op/s 2023-03-09T14:46:42.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:41 smithi038 ceph-mon[92422]: pgmap v16: 129 pgs: 129 active+clean; 1.5 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 183 MiB/s wr, 91 op/s 2023-03-09T14:46:42.080 INFO:teuthology.orchestra.run.smithi038.stderr:12500000 of 50000000 tuples (25%) done (elapsed 17.22 s, remaining 51.65 s) 2023-03-09T14:46:42.167 INFO:teuthology.orchestra.run.smithi038.stderr:12600000 of 50000000 tuples (25%) done (elapsed 17.30 s, remaining 51.36 s) 2023-03-09T14:46:42.262 INFO:teuthology.orchestra.run.smithi038.stderr:12700000 of 50000000 tuples (25%) done (elapsed 17.40 s, remaining 51.10 s) 2023-03-09T14:46:42.373 INFO:teuthology.orchestra.run.smithi038.stderr:12800000 of 50000000 tuples (25%) done (elapsed 17.51 s, remaining 50.89 s) 2023-03-09T14:46:42.467 INFO:teuthology.orchestra.run.smithi038.stderr:12900000 of 50000000 tuples (25%) done (elapsed 17.61 s, remaining 50.63 s) 2023-03-09T14:46:42.567 INFO:teuthology.orchestra.run.smithi038.stderr:13000000 of 50000000 tuples (26%) done (elapsed 17.70 s, remaining 50.39 s) 2023-03-09T14:46:42.669 INFO:teuthology.orchestra.run.smithi038.stderr:13100000 of 50000000 tuples (26%) done (elapsed 17.81 s, remaining 50.16 s) 2023-03-09T14:46:42.767 INFO:teuthology.orchestra.run.smithi038.stderr:13200000 of 50000000 tuples (26%) done (elapsed 17.90 s, remaining 49.92 s) 2023-03-09T14:46:42.848 INFO:teuthology.orchestra.run.smithi038.stderr:13300000 of 50000000 tuples (26%) done (elapsed 17.99 s, remaining 49.63 s) 2023-03-09T14:46:42.943 INFO:teuthology.orchestra.run.smithi038.stderr:13400000 of 50000000 tuples (26%) done (elapsed 18.08 s, remaining 49.38 s) 2023-03-09T14:46:43.038 INFO:teuthology.orchestra.run.smithi038.stderr:13500000 of 50000000 tuples (27%) done (elapsed 18.18 s, remaining 49.14 s) 2023-03-09T14:46:43.150 INFO:teuthology.orchestra.run.smithi038.stderr:13600000 of 50000000 tuples (27%) done (elapsed 18.29 s, remaining 48.95 s) 2023-03-09T14:46:43.243 INFO:teuthology.orchestra.run.smithi038.stderr:13700000 of 50000000 tuples (27%) done (elapsed 18.38 s, remaining 48.70 s) 2023-03-09T14:46:43.861 INFO:teuthology.orchestra.run.smithi038.stderr:13800000 of 50000000 tuples (27%) done (elapsed 19.00 s, remaining 49.84 s) 2023-03-09T14:46:43.952 INFO:teuthology.orchestra.run.smithi038.stderr:13900000 of 50000000 tuples (27%) done (elapsed 19.09 s, remaining 49.58 s) 2023-03-09T14:46:43.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:43 smithi122 ceph-mon[93869]: pgmap v17: 129 pgs: 129 active+clean; 2.3 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 275 MiB/s wr, 137 op/s 2023-03-09T14:46:43.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:43 smithi181 ceph-mon[94189]: pgmap v17: 129 pgs: 129 active+clean; 2.3 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 275 MiB/s wr, 137 op/s 2023-03-09T14:46:44.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:43 smithi038 ceph-mon[92422]: pgmap v17: 129 pgs: 129 active+clean; 2.3 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 275 MiB/s wr, 137 op/s 2023-03-09T14:46:44.050 INFO:teuthology.orchestra.run.smithi038.stderr:14000000 of 50000000 tuples (28%) done (elapsed 19.19 s, remaining 49.34 s) 2023-03-09T14:46:44.135 INFO:teuthology.orchestra.run.smithi038.stderr:14100000 of 50000000 tuples (28%) done (elapsed 19.27 s, remaining 49.07 s) 2023-03-09T14:46:44.256 INFO:teuthology.orchestra.run.smithi038.stderr:14200000 of 50000000 tuples (28%) done (elapsed 19.39 s, remaining 48.89 s) 2023-03-09T14:46:44.336 INFO:teuthology.orchestra.run.smithi038.stderr:14300000 of 50000000 tuples (28%) done (elapsed 19.47 s, remaining 48.62 s) 2023-03-09T14:46:44.434 INFO:teuthology.orchestra.run.smithi038.stderr:14400000 of 50000000 tuples (28%) done (elapsed 19.57 s, remaining 48.39 s) 2023-03-09T14:46:44.538 INFO:teuthology.orchestra.run.smithi038.stderr:14500000 of 50000000 tuples (29%) done (elapsed 19.68 s, remaining 48.17 s) 2023-03-09T14:46:44.653 INFO:teuthology.orchestra.run.smithi038.stderr:14600000 of 50000000 tuples (29%) done (elapsed 19.79 s, remaining 47.99 s) 2023-03-09T14:46:44.746 INFO:teuthology.orchestra.run.smithi038.stderr:14700000 of 50000000 tuples (29%) done (elapsed 19.88 s, remaining 47.75 s) 2023-03-09T14:46:44.855 INFO:teuthology.orchestra.run.smithi038.stderr:14800000 of 50000000 tuples (29%) done (elapsed 19.99 s, remaining 47.55 s) 2023-03-09T14:46:44.959 INFO:teuthology.orchestra.run.smithi038.stderr:14900000 of 50000000 tuples (29%) done (elapsed 20.10 s, remaining 47.34 s) 2023-03-09T14:46:45.046 INFO:teuthology.orchestra.run.smithi038.stderr:15000000 of 50000000 tuples (30%) done (elapsed 20.18 s, remaining 47.09 s) 2023-03-09T14:46:45.137 INFO:teuthology.orchestra.run.smithi038.stderr:15100000 of 50000000 tuples (30%) done (elapsed 20.27 s, remaining 46.86 s) 2023-03-09T14:46:45.216 INFO:teuthology.orchestra.run.smithi038.stderr:15200000 of 50000000 tuples (30%) done (elapsed 20.35 s, remaining 46.60 s) 2023-03-09T14:46:45.327 INFO:teuthology.orchestra.run.smithi038.stderr:15300000 of 50000000 tuples (30%) done (elapsed 20.46 s, remaining 46.41 s) 2023-03-09T14:46:45.414 INFO:teuthology.orchestra.run.smithi038.stderr:15400000 of 50000000 tuples (30%) done (elapsed 20.55 s, remaining 46.17 s) 2023-03-09T14:46:45.501 INFO:teuthology.orchestra.run.smithi038.stderr:15500000 of 50000000 tuples (31%) done (elapsed 20.64 s, remaining 45.94 s) 2023-03-09T14:46:45.612 INFO:teuthology.orchestra.run.smithi038.stderr:15600000 of 50000000 tuples (31%) done (elapsed 20.75 s, remaining 45.76 s) 2023-03-09T14:46:45.708 INFO:teuthology.orchestra.run.smithi038.stderr:15700000 of 50000000 tuples (31%) done (elapsed 20.85 s, remaining 45.54 s) 2023-03-09T14:46:45.795 INFO:teuthology.orchestra.run.smithi038.stderr:15800000 of 50000000 tuples (31%) done (elapsed 20.93 s, remaining 45.31 s) 2023-03-09T14:46:45.881 INFO:teuthology.orchestra.run.smithi038.stderr:15900000 of 50000000 tuples (31%) done (elapsed 21.02 s, remaining 45.08 s) 2023-03-09T14:46:45.975 INFO:teuthology.orchestra.run.smithi038.stderr:16000000 of 50000000 tuples (32%) done (elapsed 21.11 s, remaining 44.86 s) 2023-03-09T14:46:45.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:45 smithi122 ceph-mon[93869]: pgmap v18: 129 pgs: 129 active+clean; 2.3 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 212 MiB/s wr, 97 op/s 2023-03-09T14:46:45.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:45 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T14:46:45.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:45 smithi181 ceph-mon[94189]: pgmap v18: 129 pgs: 129 active+clean; 2.3 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 212 MiB/s wr, 97 op/s 2023-03-09T14:46:45.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:45 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T14:46:46.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:45 smithi038 ceph-mon[92422]: pgmap v18: 129 pgs: 129 active+clean; 2.3 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 212 MiB/s wr, 97 op/s 2023-03-09T14:46:46.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:45 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T14:46:46.089 INFO:teuthology.orchestra.run.smithi038.stderr:16100000 of 50000000 tuples (32%) done (elapsed 21.23 s, remaining 44.70 s) 2023-03-09T14:46:46.183 INFO:teuthology.orchestra.run.smithi038.stderr:16200000 of 50000000 tuples (32%) done (elapsed 21.32 s, remaining 44.48 s) 2023-03-09T14:46:46.280 INFO:teuthology.orchestra.run.smithi038.stderr:16300000 of 50000000 tuples (32%) done (elapsed 21.42 s, remaining 44.28 s) 2023-03-09T14:46:46.384 INFO:teuthology.orchestra.run.smithi038.stderr:16400000 of 50000000 tuples (32%) done (elapsed 21.52 s, remaining 44.09 s) 2023-03-09T14:46:46.483 INFO:teuthology.orchestra.run.smithi038.stderr:16500000 of 50000000 tuples (33%) done (elapsed 21.62 s, remaining 43.89 s) 2023-03-09T14:46:46.565 INFO:teuthology.orchestra.run.smithi038.stderr:16600000 of 50000000 tuples (33%) done (elapsed 21.70 s, remaining 43.67 s) 2023-03-09T14:46:46.671 INFO:teuthology.orchestra.run.smithi038.stderr:16700000 of 50000000 tuples (33%) done (elapsed 21.81 s, remaining 43.49 s) 2023-03-09T14:46:46.764 INFO:teuthology.orchestra.run.smithi038.stderr:16800000 of 50000000 tuples (33%) done (elapsed 21.90 s, remaining 43.28 s) 2023-03-09T14:46:46.859 INFO:teuthology.orchestra.run.smithi038.stderr:16900000 of 50000000 tuples (33%) done (elapsed 22.00 s, remaining 43.08 s) 2023-03-09T14:46:46.951 INFO:teuthology.orchestra.run.smithi038.stderr:17000000 of 50000000 tuples (34%) done (elapsed 22.09 s, remaining 42.88 s) 2023-03-09T14:46:47.066 INFO:teuthology.orchestra.run.smithi038.stderr:17100000 of 50000000 tuples (34%) done (elapsed 22.20 s, remaining 42.72 s) 2023-03-09T14:46:47.161 INFO:teuthology.orchestra.run.smithi038.stderr:17200000 of 50000000 tuples (34%) done (elapsed 22.30 s, remaining 42.52 s) 2023-03-09T14:46:47.265 INFO:teuthology.orchestra.run.smithi038.stderr:17300000 of 50000000 tuples (34%) done (elapsed 22.40 s, remaining 42.35 s) 2023-03-09T14:46:47.333 INFO:teuthology.orchestra.run.smithi038.stderr:17400000 of 50000000 tuples (34%) done (elapsed 22.47 s, remaining 42.10 s) 2023-03-09T14:46:47.441 INFO:teuthology.orchestra.run.smithi038.stderr:17500000 of 50000000 tuples (35%) done (elapsed 22.58 s, remaining 41.93 s) 2023-03-09T14:46:47.708 INFO:teuthology.orchestra.run.smithi038.stderr:17600000 of 50000000 tuples (35%) done (elapsed 22.85 s, remaining 42.06 s) 2023-03-09T14:46:47.849 INFO:teuthology.orchestra.run.smithi038.stderr:17700000 of 50000000 tuples (35%) done (elapsed 22.99 s, remaining 41.95 s) 2023-03-09T14:46:47.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:47 smithi122 ceph-mon[93869]: pgmap v19: 129 pgs: 129 active+clean; 2.7 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 295 MiB/s wr, 132 op/s 2023-03-09T14:46:47.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:47 smithi181 ceph-mon[94189]: pgmap v19: 129 pgs: 129 active+clean; 2.7 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 295 MiB/s wr, 132 op/s 2023-03-09T14:46:48.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:47 smithi038 ceph-mon[92422]: pgmap v19: 129 pgs: 129 active+clean; 2.7 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 295 MiB/s wr, 132 op/s 2023-03-09T14:46:48.215 INFO:teuthology.orchestra.run.smithi038.stderr:17800000 of 50000000 tuples (35%) done (elapsed 23.35 s, remaining 42.24 s) 2023-03-09T14:46:48.313 INFO:teuthology.orchestra.run.smithi038.stderr:17900000 of 50000000 tuples (35%) done (elapsed 23.45 s, remaining 42.05 s) 2023-03-09T14:46:48.387 INFO:teuthology.orchestra.run.smithi038.stderr:18000000 of 50000000 tuples (36%) done (elapsed 23.52 s, remaining 41.82 s) 2023-03-09T14:46:48.479 INFO:teuthology.orchestra.run.smithi038.stderr:18100000 of 50000000 tuples (36%) done (elapsed 23.62 s, remaining 41.62 s) 2023-03-09T14:46:48.610 INFO:teuthology.orchestra.run.smithi038.stderr:18200000 of 50000000 tuples (36%) done (elapsed 23.75 s, remaining 41.49 s) 2023-03-09T14:46:48.686 INFO:teuthology.orchestra.run.smithi038.stderr:18300000 of 50000000 tuples (36%) done (elapsed 23.82 s, remaining 41.27 s) 2023-03-09T14:46:48.779 INFO:teuthology.orchestra.run.smithi038.stderr:18400000 of 50000000 tuples (36%) done (elapsed 23.92 s, remaining 41.07 s) 2023-03-09T14:46:48.879 INFO:teuthology.orchestra.run.smithi038.stderr:18500000 of 50000000 tuples (37%) done (elapsed 24.02 s, remaining 40.89 s) 2023-03-09T14:46:48.979 INFO:teuthology.orchestra.run.smithi038.stderr:18600000 of 50000000 tuples (37%) done (elapsed 24.12 s, remaining 40.71 s) 2023-03-09T14:46:49.080 INFO:teuthology.orchestra.run.smithi038.stderr:18700000 of 50000000 tuples (37%) done (elapsed 24.22 s, remaining 40.54 s) 2023-03-09T14:46:49.166 INFO:teuthology.orchestra.run.smithi038.stderr:18800000 of 50000000 tuples (37%) done (elapsed 24.30 s, remaining 40.33 s) 2023-03-09T14:46:49.275 INFO:teuthology.orchestra.run.smithi038.stderr:18900000 of 50000000 tuples (37%) done (elapsed 24.41 s, remaining 40.17 s) 2023-03-09T14:46:49.384 INFO:teuthology.orchestra.run.smithi038.stderr:19000000 of 50000000 tuples (38%) done (elapsed 24.52 s, remaining 40.01 s) 2023-03-09T14:46:49.465 INFO:teuthology.orchestra.run.smithi038.stderr:19100000 of 50000000 tuples (38%) done (elapsed 24.60 s, remaining 39.80 s) 2023-03-09T14:46:49.549 INFO:teuthology.orchestra.run.smithi038.stderr:19200000 of 50000000 tuples (38%) done (elapsed 24.69 s, remaining 39.60 s) 2023-03-09T14:46:49.671 INFO:teuthology.orchestra.run.smithi038.stderr:19300000 of 50000000 tuples (38%) done (elapsed 24.81 s, remaining 39.46 s) 2023-03-09T14:46:49.749 INFO:teuthology.orchestra.run.smithi038.stderr:19400000 of 50000000 tuples (38%) done (elapsed 24.89 s, remaining 39.25 s) 2023-03-09T14:46:49.842 INFO:teuthology.orchestra.run.smithi038.stderr:19500000 of 50000000 tuples (39%) done (elapsed 24.98 s, remaining 39.07 s) 2023-03-09T14:46:49.953 INFO:teuthology.orchestra.run.smithi038.stderr:19600000 of 50000000 tuples (39%) done (elapsed 25.09 s, remaining 38.92 s) 2023-03-09T14:46:49.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:49 smithi122 ceph-mon[93869]: pgmap v20: 129 pgs: 129 active+clean; 2.7 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 176 MiB/s wr, 81 op/s 2023-03-09T14:46:49.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:49 smithi181 ceph-mon[94189]: pgmap v20: 129 pgs: 129 active+clean; 2.7 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 176 MiB/s wr, 81 op/s 2023-03-09T14:46:50.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:49 smithi038 ceph-mon[92422]: pgmap v20: 129 pgs: 129 active+clean; 2.7 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 176 MiB/s wr, 81 op/s 2023-03-09T14:46:50.051 INFO:teuthology.orchestra.run.smithi038.stderr:19700000 of 50000000 tuples (39%) done (elapsed 25.19 s, remaining 38.74 s) 2023-03-09T14:46:50.149 INFO:teuthology.orchestra.run.smithi038.stderr:19800000 of 50000000 tuples (39%) done (elapsed 25.29 s, remaining 38.57 s) 2023-03-09T14:46:50.232 INFO:teuthology.orchestra.run.smithi038.stderr:19900000 of 50000000 tuples (39%) done (elapsed 25.37 s, remaining 38.37 s) 2023-03-09T14:46:50.351 INFO:teuthology.orchestra.run.smithi038.stderr:20000000 of 50000000 tuples (40%) done (elapsed 25.49 s, remaining 38.23 s) 2023-03-09T14:46:50.419 INFO:teuthology.orchestra.run.smithi038.stderr:20100000 of 50000000 tuples (40%) done (elapsed 25.56 s, remaining 38.02 s) 2023-03-09T14:46:50.518 INFO:teuthology.orchestra.run.smithi038.stderr:20200000 of 50000000 tuples (40%) done (elapsed 25.66 s, remaining 37.85 s) 2023-03-09T14:46:50.618 INFO:teuthology.orchestra.run.smithi038.stderr:20300000 of 50000000 tuples (40%) done (elapsed 25.76 s, remaining 37.68 s) 2023-03-09T14:46:50.720 INFO:teuthology.orchestra.run.smithi038.stderr:20400000 of 50000000 tuples (40%) done (elapsed 25.86 s, remaining 37.52 s) 2023-03-09T14:46:50.813 INFO:teuthology.orchestra.run.smithi038.stderr:20500000 of 50000000 tuples (41%) done (elapsed 25.95 s, remaining 37.34 s) 2023-03-09T14:46:50.914 INFO:teuthology.orchestra.run.smithi038.stderr:20600000 of 50000000 tuples (41%) done (elapsed 26.05 s, remaining 37.18 s) 2023-03-09T14:46:50.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:50 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T14:46:50.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:50 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T14:46:51.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:50 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T14:46:51.003 INFO:teuthology.orchestra.run.smithi038.stderr:20700000 of 50000000 tuples (41%) done (elapsed 26.14 s, remaining 37.00 s) 2023-03-09T14:46:51.094 INFO:teuthology.orchestra.run.smithi038.stderr:20800000 of 50000000 tuples (41%) done (elapsed 26.23 s, remaining 36.83 s) 2023-03-09T14:46:51.191 INFO:teuthology.orchestra.run.smithi038.stderr:20900000 of 50000000 tuples (41%) done (elapsed 26.33 s, remaining 36.66 s) 2023-03-09T14:46:51.304 INFO:teuthology.orchestra.run.smithi038.stderr:21000000 of 50000000 tuples (42%) done (elapsed 26.44 s, remaining 36.51 s) 2023-03-09T14:46:51.384 INFO:teuthology.orchestra.run.smithi038.stderr:21100000 of 50000000 tuples (42%) done (elapsed 26.52 s, remaining 36.33 s) 2023-03-09T14:46:51.482 INFO:teuthology.orchestra.run.smithi038.stderr:21200000 of 50000000 tuples (42%) done (elapsed 26.62 s, remaining 36.16 s) 2023-03-09T14:46:51.574 INFO:teuthology.orchestra.run.smithi038.stderr:21300000 of 50000000 tuples (42%) done (elapsed 26.71 s, remaining 35.99 s) 2023-03-09T14:46:51.681 INFO:teuthology.orchestra.run.smithi038.stderr:21400000 of 50000000 tuples (42%) done (elapsed 26.82 s, remaining 35.84 s) 2023-03-09T14:46:51.776 INFO:teuthology.orchestra.run.smithi038.stderr:21500000 of 50000000 tuples (43%) done (elapsed 26.91 s, remaining 35.68 s) 2023-03-09T14:46:51.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:51 smithi122 ceph-mon[93869]: pgmap v21: 129 pgs: 129 active+clean; 2.7 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 176 MiB/s wr, 81 op/s 2023-03-09T14:46:51.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:51 smithi181 ceph-mon[94189]: pgmap v21: 129 pgs: 129 active+clean; 2.7 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 176 MiB/s wr, 81 op/s 2023-03-09T14:46:52.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:51 smithi038 ceph-mon[92422]: pgmap v21: 129 pgs: 129 active+clean; 2.7 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 176 MiB/s wr, 81 op/s 2023-03-09T14:46:52.321 INFO:teuthology.orchestra.run.smithi038.stderr:21600000 of 50000000 tuples (43%) done (elapsed 27.46 s, remaining 36.10 s) 2023-03-09T14:46:52.431 INFO:teuthology.orchestra.run.smithi038.stderr:21700000 of 50000000 tuples (43%) done (elapsed 27.57 s, remaining 35.95 s) 2023-03-09T14:46:52.525 INFO:teuthology.orchestra.run.smithi038.stderr:21800000 of 50000000 tuples (43%) done (elapsed 27.66 s, remaining 35.78 s) 2023-03-09T14:46:52.624 INFO:teuthology.orchestra.run.smithi038.stderr:21900000 of 50000000 tuples (43%) done (elapsed 27.76 s, remaining 35.62 s) 2023-03-09T14:46:52.737 INFO:teuthology.orchestra.run.smithi038.stderr:22000000 of 50000000 tuples (44%) done (elapsed 27.87 s, remaining 35.48 s) 2023-03-09T14:46:52.839 INFO:teuthology.orchestra.run.smithi038.stderr:22100000 of 50000000 tuples (44%) done (elapsed 27.98 s, remaining 35.32 s) 2023-03-09T14:46:52.942 INFO:teuthology.orchestra.run.smithi038.stderr:22200000 of 50000000 tuples (44%) done (elapsed 28.08 s, remaining 35.16 s) 2023-03-09T14:46:53.036 INFO:teuthology.orchestra.run.smithi038.stderr:22300000 of 50000000 tuples (44%) done (elapsed 28.17 s, remaining 35.00 s) 2023-03-09T14:46:53.125 INFO:teuthology.orchestra.run.smithi038.stderr:22400000 of 50000000 tuples (44%) done (elapsed 28.26 s, remaining 34.82 s) 2023-03-09T14:46:53.240 INFO:teuthology.orchestra.run.smithi038.stderr:22500000 of 50000000 tuples (45%) done (elapsed 28.38 s, remaining 34.68 s) 2023-03-09T14:46:53.332 INFO:teuthology.orchestra.run.smithi038.stderr:22600000 of 50000000 tuples (45%) done (elapsed 28.47 s, remaining 34.52 s) 2023-03-09T14:46:53.443 INFO:teuthology.orchestra.run.smithi038.stderr:22700000 of 50000000 tuples (45%) done (elapsed 28.58 s, remaining 34.37 s) 2023-03-09T14:46:53.542 INFO:teuthology.orchestra.run.smithi038.stderr:22800000 of 50000000 tuples (45%) done (elapsed 28.68 s, remaining 34.21 s) 2023-03-09T14:46:53.655 INFO:teuthology.orchestra.run.smithi038.stderr:22900000 of 50000000 tuples (45%) done (elapsed 28.79 s, remaining 34.07 s) 2023-03-09T14:46:53.741 INFO:teuthology.orchestra.run.smithi038.stderr:23000000 of 50000000 tuples (46%) done (elapsed 28.88 s, remaining 33.90 s) 2023-03-09T14:46:53.857 INFO:teuthology.orchestra.run.smithi038.stderr:23100000 of 50000000 tuples (46%) done (elapsed 28.99 s, remaining 33.76 s) 2023-03-09T14:46:53.961 INFO:teuthology.orchestra.run.smithi038.stderr:23200000 of 50000000 tuples (46%) done (elapsed 29.10 s, remaining 33.61 s) 2023-03-09T14:46:53.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:53 smithi122 ceph-mon[93869]: pgmap v22: 129 pgs: 129 active+clean; 3.3 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 261 MiB/s wr, 122 op/s 2023-03-09T14:46:53.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:53 smithi181 ceph-mon[94189]: pgmap v22: 129 pgs: 129 active+clean; 3.3 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 261 MiB/s wr, 122 op/s 2023-03-09T14:46:54.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:53 smithi038 ceph-mon[92422]: pgmap v22: 129 pgs: 129 active+clean; 3.3 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 261 MiB/s wr, 122 op/s 2023-03-09T14:46:54.039 INFO:teuthology.orchestra.run.smithi038.stderr:23300000 of 50000000 tuples (46%) done (elapsed 29.18 s, remaining 33.43 s) 2023-03-09T14:46:54.131 INFO:teuthology.orchestra.run.smithi038.stderr:23400000 of 50000000 tuples (46%) done (elapsed 29.27 s, remaining 33.27 s) 2023-03-09T14:46:54.225 INFO:teuthology.orchestra.run.smithi038.stderr:23500000 of 50000000 tuples (47%) done (elapsed 29.36 s, remaining 33.11 s) 2023-03-09T14:46:54.327 INFO:teuthology.orchestra.run.smithi038.stderr:23600000 of 50000000 tuples (47%) done (elapsed 29.47 s, remaining 32.96 s) 2023-03-09T14:46:54.438 INFO:teuthology.orchestra.run.smithi038.stderr:23700000 of 50000000 tuples (47%) done (elapsed 29.58 s, remaining 32.82 s) 2023-03-09T14:46:54.533 INFO:teuthology.orchestra.run.smithi038.stderr:23800000 of 50000000 tuples (47%) done (elapsed 29.67 s, remaining 32.66 s) 2023-03-09T14:46:54.635 INFO:teuthology.orchestra.run.smithi038.stderr:23900000 of 50000000 tuples (47%) done (elapsed 29.77 s, remaining 32.51 s) 2023-03-09T14:46:54.736 INFO:teuthology.orchestra.run.smithi038.stderr:24000000 of 50000000 tuples (48%) done (elapsed 29.87 s, remaining 32.36 s) 2023-03-09T14:46:54.821 INFO:teuthology.orchestra.run.smithi038.stderr:24100000 of 50000000 tuples (48%) done (elapsed 29.96 s, remaining 32.20 s) 2023-03-09T14:46:54.942 INFO:teuthology.orchestra.run.smithi038.stderr:24200000 of 50000000 tuples (48%) done (elapsed 30.08 s, remaining 32.07 s) 2023-03-09T14:46:55.009 INFO:teuthology.orchestra.run.smithi038.stderr:24300000 of 50000000 tuples (48%) done (elapsed 30.15 s, remaining 31.88 s) 2023-03-09T14:46:55.115 INFO:teuthology.orchestra.run.smithi038.stderr:24400000 of 50000000 tuples (48%) done (elapsed 30.25 s, remaining 31.74 s) 2023-03-09T14:46:55.213 INFO:teuthology.orchestra.run.smithi038.stderr:24500000 of 50000000 tuples (49%) done (elapsed 30.35 s, remaining 31.59 s) 2023-03-09T14:46:55.320 INFO:teuthology.orchestra.run.smithi038.stderr:24600000 of 50000000 tuples (49%) done (elapsed 30.46 s, remaining 31.45 s) 2023-03-09T14:46:55.396 INFO:teuthology.orchestra.run.smithi038.stderr:24700000 of 50000000 tuples (49%) done (elapsed 30.53 s, remaining 31.28 s) 2023-03-09T14:46:55.505 INFO:teuthology.orchestra.run.smithi038.stderr:24800000 of 50000000 tuples (49%) done (elapsed 30.64 s, remaining 31.14 s) 2023-03-09T14:46:55.610 INFO:teuthology.orchestra.run.smithi038.stderr:24900000 of 50000000 tuples (49%) done (elapsed 30.75 s, remaining 30.99 s) 2023-03-09T14:46:55.703 INFO:teuthology.orchestra.run.smithi038.stderr:25000000 of 50000000 tuples (50%) done (elapsed 30.84 s, remaining 30.84 s) 2023-03-09T14:46:55.811 INFO:teuthology.orchestra.run.smithi038.stderr:25100000 of 50000000 tuples (50%) done (elapsed 30.95 s, remaining 30.70 s) 2023-03-09T14:46:55.907 INFO:teuthology.orchestra.run.smithi038.stderr:25200000 of 50000000 tuples (50%) done (elapsed 31.04 s, remaining 30.55 s) 2023-03-09T14:46:55.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:55 smithi122 ceph-mon[93869]: pgmap v23: 129 pgs: 129 active+clean; 3.3 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 168 MiB/s wr, 75 op/s 2023-03-09T14:46:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:55 smithi181 ceph-mon[94189]: pgmap v23: 129 pgs: 129 active+clean; 3.3 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 168 MiB/s wr, 75 op/s 2023-03-09T14:46:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:55 smithi038 ceph-mon[92422]: pgmap v23: 129 pgs: 129 active+clean; 3.3 GiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 168 MiB/s wr, 75 op/s 2023-03-09T14:46:56.022 INFO:teuthology.orchestra.run.smithi038.stderr:25300000 of 50000000 tuples (50%) done (elapsed 31.16 s, remaining 30.42 s) 2023-03-09T14:46:56.342 INFO:teuthology.orchestra.run.smithi038.stderr:25400000 of 50000000 tuples (50%) done (elapsed 31.48 s, remaining 30.49 s) 2023-03-09T14:46:56.682 INFO:teuthology.orchestra.run.smithi038.stderr:25500000 of 50000000 tuples (51%) done (elapsed 31.82 s, remaining 30.57 s) 2023-03-09T14:46:56.757 INFO:teuthology.orchestra.run.smithi038.stderr:25600000 of 50000000 tuples (51%) done (elapsed 31.89 s, remaining 30.40 s) 2023-03-09T14:46:56.877 INFO:teuthology.orchestra.run.smithi038.stderr:25700000 of 50000000 tuples (51%) done (elapsed 32.01 s, remaining 30.27 s) 2023-03-09T14:46:56.955 INFO:teuthology.orchestra.run.smithi038.stderr:25800000 of 50000000 tuples (51%) done (elapsed 32.09 s, remaining 30.10 s) 2023-03-09T14:46:57.061 INFO:teuthology.orchestra.run.smithi038.stderr:25900000 of 50000000 tuples (51%) done (elapsed 32.20 s, remaining 29.96 s) 2023-03-09T14:46:57.178 INFO:teuthology.orchestra.run.smithi038.stderr:26000000 of 50000000 tuples (52%) done (elapsed 32.32 s, remaining 29.83 s) 2023-03-09T14:46:57.286 INFO:teuthology.orchestra.run.smithi038.stderr:26100000 of 50000000 tuples (52%) done (elapsed 32.42 s, remaining 29.69 s) 2023-03-09T14:46:57.374 INFO:teuthology.orchestra.run.smithi038.stderr:26200000 of 50000000 tuples (52%) done (elapsed 32.51 s, remaining 29.53 s) 2023-03-09T14:46:57.473 INFO:teuthology.orchestra.run.smithi038.stderr:26300000 of 50000000 tuples (52%) done (elapsed 32.61 s, remaining 29.39 s) 2023-03-09T14:46:57.574 INFO:teuthology.orchestra.run.smithi038.stderr:26400000 of 50000000 tuples (52%) done (elapsed 32.71 s, remaining 29.24 s) 2023-03-09T14:46:57.670 INFO:teuthology.orchestra.run.smithi038.stderr:26500000 of 50000000 tuples (53%) done (elapsed 32.81 s, remaining 29.09 s) 2023-03-09T14:46:57.771 INFO:teuthology.orchestra.run.smithi038.stderr:26600000 of 50000000 tuples (53%) done (elapsed 32.91 s, remaining 28.95 s) 2023-03-09T14:46:57.862 INFO:teuthology.orchestra.run.smithi038.stderr:26700000 of 50000000 tuples (53%) done (elapsed 33.00 s, remaining 28.80 s) 2023-03-09T14:46:57.956 INFO:teuthology.orchestra.run.smithi038.stderr:26800000 of 50000000 tuples (53%) done (elapsed 33.09 s, remaining 28.65 s) 2023-03-09T14:46:57.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:57 smithi122 ceph-mon[93869]: pgmap v24: 129 pgs: 129 active+clean; 3.7 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 248 MiB/s wr, 112 op/s 2023-03-09T14:46:57.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:57 smithi181 ceph-mon[94189]: pgmap v24: 129 pgs: 129 active+clean; 3.7 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 248 MiB/s wr, 112 op/s 2023-03-09T14:46:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:57 smithi038 ceph-mon[92422]: pgmap v24: 129 pgs: 129 active+clean; 3.7 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 248 MiB/s wr, 112 op/s 2023-03-09T14:46:58.030 INFO:teuthology.orchestra.run.smithi038.stderr:26900000 of 50000000 tuples (53%) done (elapsed 33.17 s, remaining 28.48 s) 2023-03-09T14:46:58.136 INFO:teuthology.orchestra.run.smithi038.stderr:27000000 of 50000000 tuples (54%) done (elapsed 33.27 s, remaining 28.34 s) 2023-03-09T14:46:58.232 INFO:teuthology.orchestra.run.smithi038.stderr:27100000 of 50000000 tuples (54%) done (elapsed 33.37 s, remaining 28.20 s) 2023-03-09T14:46:58.325 INFO:teuthology.orchestra.run.smithi038.stderr:27200000 of 50000000 tuples (54%) done (elapsed 33.46 s, remaining 28.05 s) 2023-03-09T14:46:58.432 INFO:teuthology.orchestra.run.smithi038.stderr:27300000 of 50000000 tuples (54%) done (elapsed 33.57 s, remaining 27.91 s) 2023-03-09T14:46:58.509 INFO:teuthology.orchestra.run.smithi038.stderr:27400000 of 50000000 tuples (54%) done (elapsed 33.65 s, remaining 27.75 s) 2023-03-09T14:46:58.631 INFO:teuthology.orchestra.run.smithi038.stderr:27500000 of 50000000 tuples (55%) done (elapsed 33.77 s, remaining 27.63 s) 2023-03-09T14:46:58.700 INFO:teuthology.orchestra.run.smithi038.stderr:27600000 of 50000000 tuples (55%) done (elapsed 33.84 s, remaining 27.46 s) 2023-03-09T14:46:58.808 INFO:teuthology.orchestra.run.smithi038.stderr:27700000 of 50000000 tuples (55%) done (elapsed 33.95 s, remaining 27.33 s) 2023-03-09T14:46:58.915 INFO:teuthology.orchestra.run.smithi038.stderr:27800000 of 50000000 tuples (55%) done (elapsed 34.05 s, remaining 27.19 s) 2023-03-09T14:46:59.013 INFO:teuthology.orchestra.run.smithi038.stderr:27900000 of 50000000 tuples (55%) done (elapsed 34.15 s, remaining 27.05 s) 2023-03-09T14:46:59.124 INFO:teuthology.orchestra.run.smithi038.stderr:28000000 of 50000000 tuples (56%) done (elapsed 34.26 s, remaining 26.92 s) 2023-03-09T14:46:59.221 INFO:teuthology.orchestra.run.smithi038.stderr:28100000 of 50000000 tuples (56%) done (elapsed 34.36 s, remaining 26.78 s) 2023-03-09T14:46:59.313 INFO:teuthology.orchestra.run.smithi038.stderr:28200000 of 50000000 tuples (56%) done (elapsed 34.45 s, remaining 26.63 s) 2023-03-09T14:46:59.421 INFO:teuthology.orchestra.run.smithi038.stderr:28300000 of 50000000 tuples (56%) done (elapsed 34.56 s, remaining 26.50 s) 2023-03-09T14:46:59.519 INFO:teuthology.orchestra.run.smithi038.stderr:28400000 of 50000000 tuples (56%) done (elapsed 34.66 s, remaining 26.36 s) 2023-03-09T14:46:59.591 INFO:teuthology.orchestra.run.smithi038.stderr:28500000 of 50000000 tuples (57%) done (elapsed 34.73 s, remaining 26.20 s) 2023-03-09T14:46:59.699 INFO:teuthology.orchestra.run.smithi038.stderr:28600000 of 50000000 tuples (57%) done (elapsed 34.84 s, remaining 26.07 s) 2023-03-09T14:46:59.791 INFO:teuthology.orchestra.run.smithi038.stderr:28700000 of 50000000 tuples (57%) done (elapsed 34.93 s, remaining 25.92 s) 2023-03-09T14:46:59.880 INFO:teuthology.orchestra.run.smithi038.stderr:28800000 of 50000000 tuples (57%) done (elapsed 35.02 s, remaining 25.78 s) 2023-03-09T14:46:59.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:46:59 smithi122 ceph-mon[93869]: pgmap v25: 129 pgs: 129 active+clean; 3.7 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 165 MiB/s wr, 76 op/s 2023-03-09T14:46:59.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:46:59 smithi181 ceph-mon[94189]: pgmap v25: 129 pgs: 129 active+clean; 3.7 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 165 MiB/s wr, 76 op/s 2023-03-09T14:46:59.990 INFO:teuthology.orchestra.run.smithi038.stderr:28900000 of 50000000 tuples (57%) done (elapsed 35.13 s, remaining 25.65 s) 2023-03-09T14:47:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:46:59 smithi038 ceph-mon[92422]: pgmap v25: 129 pgs: 129 active+clean; 3.7 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 165 MiB/s wr, 76 op/s 2023-03-09T14:47:00.083 INFO:teuthology.orchestra.run.smithi038.stderr:29000000 of 50000000 tuples (58%) done (elapsed 35.22 s, remaining 25.50 s) 2023-03-09T14:47:00.500 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:47:00 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:47:00.097+0000 7f401be04700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-09T14:47:00.501 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:47:00 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:47:00.097+0000 7f401be04700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-09T14:47:00.501 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:47:00 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:47:00.097+0000 7f401be04700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-09T14:47:00.501 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:47:00 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:47:00.097+0000 7f401be04700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-09T14:47:00.501 INFO:journalctl@ceph.mgr.x.smithi038.stdout:Mar 09 14:47:00 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-x[92668]: 2023-03-09T14:47:00.097+0000 7f401be04700 -1 client.0 error registering admin socket command: (17) File exists 2023-03-09T14:47:01.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:47:01 smithi122 ceph-mon[93869]: pgmap v26: 129 pgs: 129 active+clean; 3.7 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 165 MiB/s wr, 76 op/s 2023-03-09T14:47:01.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:47:01 smithi181 ceph-mon[94189]: pgmap v26: 129 pgs: 129 active+clean; 3.7 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 165 MiB/s wr, 76 op/s 2023-03-09T14:47:02.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:47:01 smithi038 ceph-mon[92422]: pgmap v26: 129 pgs: 129 active+clean; 3.7 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 165 MiB/s wr, 76 op/s 2023-03-09T14:47:02.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:47:02 smithi122 ceph-mon[93869]: mgrmap e24: x(active, since 47s), standbys: z, y 2023-03-09T14:47:02.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:47:02 smithi181 ceph-mon[94189]: mgrmap e24: x(active, since 47s), standbys: z, y 2023-03-09T14:47:03.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:47:02 smithi038 ceph-mon[92422]: mgrmap e24: x(active, since 47s), standbys: z, y 2023-03-09T14:47:03.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:47:03 smithi122 ceph-mon[93869]: pgmap v27: 129 pgs: 129 active+clean; 4.2 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 245 MiB/s wr, 134 op/s 2023-03-09T14:47:03.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:47:03 smithi181 ceph-mon[94189]: pgmap v27: 129 pgs: 129 active+clean; 4.2 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 245 MiB/s wr, 134 op/s 2023-03-09T14:47:04.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:47:03 smithi038 ceph-mon[92422]: pgmap v27: 129 pgs: 129 active+clean; 4.2 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 245 MiB/s wr, 134 op/s 2023-03-09T14:47:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:47:05 smithi122 ceph-mon[93869]: pgmap v28: 129 pgs: 129 active+clean; 4.2 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 160 MiB/s wr, 94 op/s 2023-03-09T14:47:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:47:05 smithi181 ceph-mon[94189]: pgmap v28: 129 pgs: 129 active+clean; 4.2 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 160 MiB/s wr, 94 op/s 2023-03-09T14:47:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:47:05 smithi038 ceph-mon[92422]: pgmap v28: 129 pgs: 129 active+clean; 4.2 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 160 MiB/s wr, 94 op/s 2023-03-09T14:47:06.314 INFO:teuthology.orchestra.run.smithi038.stderr:29100000 of 50000000 tuples (58%) done (elapsed 41.45 s, remaining 29.77 s) 2023-03-09T14:47:07.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:47:07 smithi122 ceph-mon[93869]: pgmap v29: 129 pgs: 129 active+clean; 4.2 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 161 MiB/s wr, 238 op/s 2023-03-09T14:47:07.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:47:07 smithi181 ceph-mon[94189]: pgmap v29: 129 pgs: 129 active+clean; 4.2 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 161 MiB/s wr, 238 op/s 2023-03-09T14:47:08.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:47:07 smithi038 ceph-mon[92422]: pgmap v29: 129 pgs: 129 active+clean; 4.2 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 161 MiB/s wr, 238 op/s 2023-03-09T14:47:08.417 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-03-09T14:47:08.585 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.581+0000 7f4410f9b700 1 -- 172.21.15.38:0/3699127790 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f440c106820 msgr2=0x7f440c106bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:47:08.585 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.581+0000 7f4410f9b700 1 --2- 172.21.15.38:0/3699127790 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f440c106820 0x7f440c106bf0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f43fc009990 tx=0x7f43fc009ca0 comp rx=0 tx=0).stop 2023-03-09T14:47:08.585 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.581+0000 7f4410f9b700 1 -- 172.21.15.38:0/3699127790 shutdown_connections 2023-03-09T14:47:08.585 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.581+0000 7f4410f9b700 1 --2- 172.21.15.38:0/3699127790 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f440c10b880 0x7f440c113400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:08.585 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.581+0000 7f4410f9b700 1 --2- 172.21.15.38:0/3699127790 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f440c106820 0x7f440c106bf0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:08.586 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.581+0000 7f4410f9b700 1 --2- 172.21.15.38:0/3699127790 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f440c1071c0 0x7f440c10b150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:08.586 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.581+0000 7f4410f9b700 1 -- 172.21.15.38:0/3699127790 >> 172.21.15.38:0/3699127790 conn(0x7f440c0744c0 msgr2=0x7f440c0768d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:47:08.586 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.581+0000 7f4410f9b700 1 -- 172.21.15.38:0/3699127790 shutdown_connections 2023-03-09T14:47:08.586 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.581+0000 7f4410f9b700 1 -- 172.21.15.38:0/3699127790 wait complete. 2023-03-09T14:47:08.588 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.582+0000 7f4410f9b700 1 Processor -- start 2023-03-09T14:47:08.588 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.582+0000 7f4410f9b700 1 -- start start 2023-03-09T14:47:08.588 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.582+0000 7f4410f9b700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f440c106820 0x7f440c19c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:08.588 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.582+0000 7f4410f9b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f440c1071c0 0x7f440c19cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:08.589 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.582+0000 7f4410f9b700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f440c10b880 0x7f440c1a0f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:08.589 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.582+0000 7f4410f9b700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f440c1a1560 con 0x7f440c1071c0 2023-03-09T14:47:08.589 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.582+0000 7f4410f9b700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f440c1a16d0 con 0x7f440c10b880 2023-03-09T14:47:08.589 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.582+0000 7f4410f9b700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f440c1a1890 con 0x7f440c106820 2023-03-09T14:47:08.589 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.582+0000 7f440a59c700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f440c106820 0x7f440c19c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:08.589 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.582+0000 7f4409d9b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f440c1071c0 0x7f440c19cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:08.590 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.582+0000 7f440a59c700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f440c106820 0x7f440c19c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.38:47216/0 (socket says 172.21.15.38:47216) 2023-03-09T14:47:08.590 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.582+0000 7f440a59c700 1 -- 172.21.15.38:0/2685954334 learned_addr learned my addr 172.21.15.38:0/2685954334 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:47:08.590 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.583+0000 7f440ad9d700 1 --2- 172.21.15.38:0/2685954334 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f440c10b880 0x7f440c1a0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:08.590 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.583+0000 7f440a59c700 1 -- 172.21.15.38:0/2685954334 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f440c10b880 msgr2=0x7f440c1a0f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:47:08.590 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.583+0000 7f440a59c700 1 --2- 172.21.15.38:0/2685954334 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f440c10b880 0x7f440c1a0f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:08.590 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.583+0000 7f440a59c700 1 -- 172.21.15.38:0/2685954334 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f440c1071c0 msgr2=0x7f440c19cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:47:08.591 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.583+0000 7f440a59c700 1 --2- 172.21.15.38:0/2685954334 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f440c1071c0 0x7f440c19cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:08.591 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.583+0000 7f440a59c700 1 -- 172.21.15.38:0/2685954334 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43fc0095f0 con 0x7f440c106820 2023-03-09T14:47:08.591 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.583+0000 7f440a59c700 1 --2- 172.21.15.38:0/2685954334 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f440c106820 0x7f440c19c620 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f43fc000c00 tx=0x7f43fc004790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:47:08.591 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.583+0000 7f44037fe700 1 -- 172.21.15.38:0/2685954334 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43fc01d070 con 0x7f440c106820 2023-03-09T14:47:08.591 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.583+0000 7f4410f9b700 1 -- 172.21.15.38:0/2685954334 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f440c116570 con 0x7f440c106820 2023-03-09T14:47:08.592 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.583+0000 7f44037fe700 1 -- 172.21.15.38:0/2685954334 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f43fc021410 con 0x7f440c106820 2023-03-09T14:47:08.592 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.583+0000 7f44037fe700 1 -- 172.21.15.38:0/2685954334 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43fc017650 con 0x7f440c106820 2023-03-09T14:47:08.592 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.583+0000 7f4410f9b700 1 -- 172.21.15.38:0/2685954334 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f440c1163e0 con 0x7f440c106820 2023-03-09T14:47:08.596 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.586+0000 7f44037fe700 1 -- 172.21.15.38:0/2685954334 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 24) v1 ==== 133375+0+0 (secure 0 0 0) 0x7f43fc028030 con 0x7f440c106820 2023-03-09T14:47:08.596 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.586+0000 7f44037fe700 1 --2- 172.21.15.38:0/2685954334 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f43f809f530 0x7f43f80a19e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:08.596 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.587+0000 7f4409d9b700 1 --2- 172.21.15.38:0/2685954334 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f43f809f530 0x7f43f80a19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:08.597 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.587+0000 7f44037fe700 1 -- 172.21.15.38:0/2685954334 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9278+0+0 (secure 0 0 0) 0x7f43fc0cb620 con 0x7f440c106820 2023-03-09T14:47:08.597 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.587+0000 7f4409d9b700 1 --2- 172.21.15.38:0/2685954334 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f43f809f530 0x7f43f80a19e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f440c19dc30 tx=0x7f43f4009260 comp rx=0 tx=0).ready entity=mgr.15558 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:47:08.597 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.588+0000 7f441d124b80 1 Processor -- start 2023-03-09T14:47:08.597 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.588+0000 7f441d124b80 1 -- start start 2023-03-09T14:47:08.597 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.588+0000 7f441d124b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557948d91830 0x557948d91c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:08.597 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.588+0000 7f441d124b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557948d92140 0x557948d8baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:08.598 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.588+0000 7f441d124b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557948d8bfe0 0x557948d8c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:08.598 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.588+0000 7f441d124b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x557948c75da0 con 0x557948d92140 2023-03-09T14:47:08.598 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.588+0000 7f441d124b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x557948ce4780 con 0x557948d8bfe0 2023-03-09T14:47:08.598 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.588+0000 7f441d124b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x557948cddcb0 con 0x557948d91830 2023-03-09T14:47:08.598 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.588+0000 7f440a59c700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557948d91830 0x557948d91c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:08.599 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.588+0000 7f4409d9b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557948d92140 0x557948d8baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:08.599 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.588+0000 7f440a59c700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557948d91830 0x557948d91c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.38:47224/0 (socket says 172.21.15.38:47224) 2023-03-09T14:47:08.599 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.588+0000 7f440a59c700 1 -- 172.21.15.38:0/888035280 learned_addr learned my addr 172.21.15.38:0/888035280 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:47:08.599 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.588+0000 7f440ad9d700 1 --2- 172.21.15.38:0/888035280 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557948d8bfe0 0x557948d8c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:08.599 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.589+0000 7f4409d9b700 1 -- 172.21.15.38:0/888035280 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557948d91830 msgr2=0x557948d91c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:47:08.599 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.589+0000 7f4409d9b700 1 --2- 172.21.15.38:0/888035280 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557948d91830 0x557948d91c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:08.599 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.589+0000 7f4409d9b700 1 -- 172.21.15.38:0/888035280 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557948d8bfe0 msgr2=0x557948d8c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:47:08.600 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.589+0000 7f4409d9b700 1 --2- 172.21.15.38:0/888035280 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557948d8bfe0 0x557948d8c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:08.600 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.589+0000 7f4409d9b700 1 -- 172.21.15.38:0/888035280 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557948ca4190 con 0x557948d92140 2023-03-09T14:47:08.600 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.589+0000 7f440a59c700 1 --2- 172.21.15.38:0/888035280 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557948d91830 0x557948d91c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:47:08.600 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.589+0000 7f4409d9b700 1 --2- 172.21.15.38:0/888035280 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557948d92140 0x557948d8baa0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f43f4011c60 tx=0x7f43f4014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35b9ea0caa83092b server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:47:08.600 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.589+0000 7f4400ff9700 1 -- 172.21.15.38:0/888035280 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f4004230 con 0x557948d92140 2023-03-09T14:47:08.600 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.589+0000 7f4400ff9700 1 -- 172.21.15.38:0/888035280 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f43f4021070 con 0x557948d92140 2023-03-09T14:47:08.601 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.589+0000 7f4400ff9700 1 -- 172.21.15.38:0/888035280 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f4003bf0 con 0x557948d92140 2023-03-09T14:47:08.601 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.590+0000 7f441d124b80 1 -- 172.21.15.38:0/888035280 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557948d92140 msgr2=0x557948d8baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:47:08.601 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.590+0000 7f441d124b80 1 --2- 172.21.15.38:0/888035280 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557948d92140 0x557948d8baa0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f43f4011c60 tx=0x7f43f4014000 comp rx=0 tx=0).stop 2023-03-09T14:47:08.601 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.590+0000 7f441d124b80 1 -- 172.21.15.38:0/888035280 shutdown_connections 2023-03-09T14:47:08.601 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.590+0000 7f441d124b80 1 --2- 172.21.15.38:0/888035280 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557948d8bfe0 0x557948d8c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:08.601 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.590+0000 7f441d124b80 1 --2- 172.21.15.38:0/888035280 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557948d91830 0x557948d91c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:08.602 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.590+0000 7f441d124b80 1 --2- 172.21.15.38:0/888035280 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557948d92140 0x557948d8baa0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:08.602 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.590+0000 7f441d124b80 1 -- 172.21.15.38:0/888035280 >> 172.21.15.38:0/888035280 conn(0x557948d86a20 msgr2=0x557948d86df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:47:08.602 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.590+0000 7f441d124b80 1 -- 172.21.15.38:0/888035280 shutdown_connections 2023-03-09T14:47:08.602 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.590+0000 7f441d124b80 1 -- 172.21.15.38:0/888035280 wait complete. 2023-03-09T14:47:08.602 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.591+0000 7f441d124b80 1 -- start start 2023-03-09T14:47:08.602 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.591+0000 7f441d124b80 1 Processor -- start 2023-03-09T14:47:08.602 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.591+0000 7f441d124b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557948d8bfe0 0x557948d98150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:08.603 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.591+0000 7f441d124b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557948d91830 0x557948d953d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:08.603 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.591+0000 7f441d124b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557948d92140 0x557948d95b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:08.603 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.591+0000 7f441d124b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x557948b61920 con 0x557948d92140 2023-03-09T14:47:08.603 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.591+0000 7f441d124b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x557948d96280 con 0x557948d91830 2023-03-09T14:47:08.603 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.591+0000 7f441d124b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x557948da1c70 con 0x557948d8bfe0 2023-03-09T14:47:08.603 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.591+0000 7f441d124b80 10 client.0 _ll_register_callbacks cb 0x557948c83520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:47:08.604 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.591+0000 7f440ad9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557948d92140 0x557948d95b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:08.604 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.591+0000 7f440ad9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557948d92140 0x557948d95b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39536/0 (socket says 172.21.15.38:39536) 2023-03-09T14:47:08.604 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.591+0000 7f440ad9d700 1 -- 172.21.15.38:0/234829980 learned_addr learned my addr 172.21.15.38:0/234829980 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:47:08.604 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.591+0000 7f4409d9b700 1 --2- 172.21.15.38:0/234829980 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557948d91830 0x557948d953d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:08.604 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.591+0000 7f440a59c700 1 --2- 172.21.15.38:0/234829980 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557948d8bfe0 0x557948d98150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:08.604 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.591+0000 7f440ad9d700 1 -- 172.21.15.38:0/234829980 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557948d8bfe0 msgr2=0x557948d98150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:47:08.605 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.591+0000 7f440ad9d700 1 --2- 172.21.15.38:0/234829980 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557948d8bfe0 0x557948d98150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:08.605 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.591+0000 7f440ad9d700 1 -- 172.21.15.38:0/234829980 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557948d91830 msgr2=0x557948d953d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:47:08.605 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.591+0000 7f440ad9d700 1 --2- 172.21.15.38:0/234829980 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557948d91830 0x557948d953d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:08.605 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.591+0000 7f440ad9d700 1 -- 172.21.15.38:0/234829980 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f440c1163e0 con 0x557948d92140 2023-03-09T14:47:08.605 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.591+0000 7f440ad9d700 1 --2- 172.21.15.38:0/234829980 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557948d92140 0x557948d95b10 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f4404009550 tx=0x7f4404007e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=389a82cf6d9abbd7 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:47:08.605 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.591+0000 7f43f1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2023-03-09T14:47:08.605 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.592+0000 7f43f1ffb700 1 -- 172.21.15.38:0/234829980 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f440400e070 con 0x557948d92140 2023-03-09T14:47:08.606 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.592+0000 7f43f1ffb700 1 -- 172.21.15.38:0/234829980 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9278+0+0 (secure 0 0 0) 0x7f4404003c00 con 0x557948d92140 2023-03-09T14:47:08.606 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.592+0000 7f441d124b80 1 -- 172.21.15.38:0/234829980 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557948cab190 con 0x557948d92140 2023-03-09T14:47:08.606 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.592+0000 7f43f1ffb700 1 -- 172.21.15.38:0/234829980 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4404003e00 con 0x557948d92140 2023-03-09T14:47:08.606 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.592+0000 7f441d124b80 10 client.15765 fetch_fsmap learned FSMap version 25 2023-03-09T14:47:08.606 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.592+0000 7f441d124b80 1 -- 172.21.15.38:0/234829980 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x7f43f4017070 con 0x557948d92140 2023-03-09T14:47:08.606 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.592+0000 7f43f1ffb700 1 -- 172.21.15.38:0/234829980 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x557948cab190 con 0x557948d92140 2023-03-09T14:47:08.606 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.592+0000 7f43f1ffb700 1 -- 172.21.15.38:0/234829980 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 25) v1 ==== 7478+0+0 (secure 0 0 0) 0x7f4404014040 con 0x557948d92140 2023-03-09T14:47:08.606 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.592+0000 7f441d124b80 10 client.15765 fetch_fsmap finished waiting for FSMap version 25 2023-03-09T14:47:08.607 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.592+0000 7f441d124b80 10 client.15765 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T14:47:08.607 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.592+0000 7f441d124b80 20 client.15765 populate_metadata read hostname 'smithi038' 2023-03-09T14:47:08.607 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.592+0000 7f441d124b80 1 --2- 172.21.15.38:0/234829980 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x557948da5e70 0x557948da8280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:08.607 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.592+0000 7f441d124b80 4 client.15765 mds_command: new command op to 24530 tid=0[{"prefix": "get_command_descriptions"}] 2023-03-09T14:47:08.607 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.592+0000 7f441d124b80 1 -- 172.21.15.38:0/234829980 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557948b61f60 con 0x557948da5e70 2023-03-09T14:47:08.607 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.592+0000 7f440a59c700 1 --2- 172.21.15.38:0/234829980 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x557948da5e70 0x557948da8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:08.607 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.593+0000 7f43f1ffb700 10 client.15765 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:47:08.608 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.593+0000 7f440a59c700 1 --2- 172.21.15.38:0/234829980 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x557948da5e70 0x557948da8280 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fe6825c5d5731454 server_cookie=7f7278dd6af21380 in_seq=0 out_seq=0 2023-03-09T14:47:08.609 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.593+0000 7f43f1ffb700 1 -- 172.21.15.38:0/234829980 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18278 (crc 0 0 0) 0x557948b61f60 con 0x557948da5e70 2023-03-09T14:47:08.609 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.593+0000 7f43f1ffb700 10 client.15765 handle_command_reply: tid=0 2023-03-09T14:47:08.609 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.593+0000 7f441d124b80 1 client.15765 shutdown 2023-03-09T14:47:08.610 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.593+0000 7f441d124b80 1 -- 172.21.15.38:0/234829980 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557948d92140 msgr2=0x557948d95b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:47:08.610 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.593+0000 7f441d124b80 1 --2- 172.21.15.38:0/234829980 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557948d92140 0x557948d95b10 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f4404009550 tx=0x7f4404007e30 comp rx=0 tx=0).stop 2023-03-09T14:47:08.610 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.593+0000 7f441d124b80 1 -- 172.21.15.38:0/234829980 shutdown_connections 2023-03-09T14:47:08.610 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.594+0000 7f441d124b80 1 --2- 172.21.15.38:0/234829980 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x557948da5e70 0x557948da8280 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:08.610 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.594+0000 7f441d124b80 1 --2- 172.21.15.38:0/234829980 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557948d92140 0x557948d95b10 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:08.610 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.594+0000 7f441d124b80 1 --2- 172.21.15.38:0/234829980 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557948d8bfe0 0x557948d98150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:08.610 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.594+0000 7f441d124b80 1 --2- 172.21.15.38:0/234829980 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557948d91830 0x557948d953d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:08.610 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.594+0000 7f43f1ffb700 0 client.15765 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:47:08.611 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.594+0000 7f441d124b80 1 -- 172.21.15.38:0/234829980 >> 172.21.15.38:0/234829980 conn(0x557948d86a20 msgr2=0x557948d85c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:47:08.611 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.594+0000 7f441d124b80 1 -- 172.21.15.38:0/234829980 shutdown_connections 2023-03-09T14:47:08.611 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.594+0000 7f441d124b80 1 -- 172.21.15.38:0/234829980 wait complete. 2023-03-09T14:47:08.611 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.594+0000 7f441d124b80 20 client.15765 trim_cache size 0 max 16384 2023-03-09T14:47:08.620 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.615+0000 7f441d124b80 1 Processor -- start 2023-03-09T14:47:08.620 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.615+0000 7f441d124b80 1 -- start start 2023-03-09T14:47:08.621 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.616+0000 7f441d124b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557948d8bfe0 0x557948da1350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:08.621 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.616+0000 7f441d124b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557948d91830 0x557948d85c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:08.621 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.616+0000 7f441d124b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557948d92140 0x557948d8f530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:08.622 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.616+0000 7f441d124b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x557948da34c0 con 0x557948d91830 2023-03-09T14:47:08.622 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.616+0000 7f441d124b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x557948d87f20 con 0x557948d92140 2023-03-09T14:47:08.622 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.616+0000 7f441d124b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x557948d98490 con 0x557948d8bfe0 2023-03-09T14:47:08.622 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.616+0000 7f440a59c700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557948d8bfe0 0x557948da1350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:08.622 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.616+0000 7f4409d9b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557948d91830 0x557948d85c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:08.623 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.616+0000 7f440a59c700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557948d8bfe0 0x557948da1350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.38:47238/0 (socket says 172.21.15.38:47238) 2023-03-09T14:47:08.623 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.616+0000 7f440a59c700 1 -- 172.21.15.38:0/258726036 learned_addr learned my addr 172.21.15.38:0/258726036 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:47:08.623 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.616+0000 7f440ad9d700 1 --2- 172.21.15.38:0/258726036 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557948d92140 0x557948d8f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:08.623 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.616+0000 7f440a59c700 1 -- 172.21.15.38:0/258726036 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557948d92140 msgr2=0x557948d8f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:47:08.624 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.616+0000 7f440a59c700 1 --2- 172.21.15.38:0/258726036 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557948d92140 0x557948d8f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:08.624 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.616+0000 7f440a59c700 1 -- 172.21.15.38:0/258726036 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557948d91830 msgr2=0x557948d85c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:47:08.624 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.616+0000 7f440a59c700 1 --2- 172.21.15.38:0/258726036 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557948d91830 0x557948d85c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:08.625 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.616+0000 7f440a59c700 1 -- 172.21.15.38:0/258726036 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557948af4c60 con 0x557948d8bfe0 2023-03-09T14:47:08.625 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.616+0000 7f440a59c700 1 --2- 172.21.15.38:0/258726036 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557948d8bfe0 0x557948da1350 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f43fc0c8fb0 tx=0x7f43fc009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2935d52675ca3a7 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:47:08.625 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.617+0000 7f44017fa700 1 -- 172.21.15.38:0/258726036 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43fc099200 con 0x557948d8bfe0 2023-03-09T14:47:08.625 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.617+0000 7f44017fa700 1 -- 172.21.15.38:0/258726036 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f43fc0a1360 con 0x557948d8bfe0 2023-03-09T14:47:08.626 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.617+0000 7f44017fa700 1 -- 172.21.15.38:0/258726036 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43fc0216e0 con 0x557948d8bfe0 2023-03-09T14:47:08.629 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.617+0000 7f441d124b80 1 -- 172.21.15.38:0/258726036 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557948d8bfe0 msgr2=0x557948da1350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:47:08.629 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.617+0000 7f441d124b80 1 --2- 172.21.15.38:0/258726036 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557948d8bfe0 0x557948da1350 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f43fc0c8fb0 tx=0x7f43fc009310 comp rx=0 tx=0).stop 2023-03-09T14:47:08.629 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.617+0000 7f441d124b80 1 -- 172.21.15.38:0/258726036 shutdown_connections 2023-03-09T14:47:08.629 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.617+0000 7f441d124b80 1 --2- 172.21.15.38:0/258726036 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557948d92140 0x557948d8f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:08.630 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.617+0000 7f441d124b80 1 --2- 172.21.15.38:0/258726036 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557948d8bfe0 0x557948da1350 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:08.630 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.617+0000 7f441d124b80 1 --2- 172.21.15.38:0/258726036 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557948d91830 0x557948d85c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:08.630 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.617+0000 7f441d124b80 1 -- 172.21.15.38:0/258726036 >> 172.21.15.38:0/258726036 conn(0x557948d86a20 msgr2=0x557948d97d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:47:08.630 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.618+0000 7f441d124b80 1 -- 172.21.15.38:0/258726036 shutdown_connections 2023-03-09T14:47:08.630 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.618+0000 7f441d124b80 1 -- 172.21.15.38:0/258726036 wait complete. 2023-03-09T14:47:08.630 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.618+0000 7f441d124b80 1 -- start start 2023-03-09T14:47:08.631 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.618+0000 7f441d124b80 1 Processor -- start 2023-03-09T14:47:08.631 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.618+0000 7f441d124b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557948d8bfe0 0x557948d8e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:08.631 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.618+0000 7f441d124b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557948d91830 0x557948d8d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:08.631 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.618+0000 7f441d124b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557948d92140 0x557948d8d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:08.631 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.618+0000 7f441d124b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x557948d8efd0 con 0x557948d8bfe0 2023-03-09T14:47:08.631 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.618+0000 7f441d124b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x557948d8dd10 con 0x557948d92140 2023-03-09T14:47:08.631 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.618+0000 7f441d124b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x557948d8ded0 con 0x557948d91830 2023-03-09T14:47:08.632 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.618+0000 7f441d124b80 10 client.0 _ll_register_callbacks cb 0x557948c83520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:47:08.632 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.618+0000 7f440a59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557948d8bfe0 0x557948d8e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:08.632 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.618+0000 7f440a59c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557948d8bfe0 0x557948d8e9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39546/0 (socket says 172.21.15.38:39546) 2023-03-09T14:47:08.632 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.618+0000 7f440a59c700 1 -- 172.21.15.38:0/2750525248 learned_addr learned my addr 172.21.15.38:0/2750525248 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:47:08.632 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.618+0000 7f440ad9d700 1 --2- 172.21.15.38:0/2750525248 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557948d92140 0x557948d8d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:08.632 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.618+0000 7f4409d9b700 1 --2- 172.21.15.38:0/2750525248 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557948d91830 0x557948d8d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:08.633 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.619+0000 7f440ad9d700 1 -- 172.21.15.38:0/2750525248 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557948d91830 msgr2=0x557948d8d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:47:08.633 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.619+0000 7f440ad9d700 1 --2- 172.21.15.38:0/2750525248 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557948d91830 0x557948d8d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:08.633 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.619+0000 7f440ad9d700 1 -- 172.21.15.38:0/2750525248 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557948d8bfe0 msgr2=0x557948d8e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:47:08.633 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.619+0000 7f440ad9d700 1 --2- 172.21.15.38:0/2750525248 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557948d8bfe0 0x557948d8e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:08.633 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.619+0000 7f440ad9d700 1 -- 172.21.15.38:0/2750525248 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f440c116570 con 0x557948d92140 2023-03-09T14:47:08.633 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.619+0000 7f440a59c700 1 --2- 172.21.15.38:0/2750525248 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557948d8bfe0 0x557948d8e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:47:08.633 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.619+0000 7f4409d9b700 1 --2- 172.21.15.38:0/2750525248 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557948d91830 0x557948d8d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:47:08.634 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.619+0000 7f440ad9d700 1 --2- 172.21.15.38:0/2750525248 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557948d92140 0x557948d8d7d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4404009010 tx=0x7f4404012fc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cdd6b815e047dbfd server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:47:08.634 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.619+0000 7f43f1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2023-03-09T14:47:08.634 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.619+0000 7f43f1ffb700 1 -- 172.21.15.38:0/2750525248 <== mon.2 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f440400e040 con 0x557948d92140 2023-03-09T14:47:08.634 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.619+0000 7f441d124b80 1 -- 172.21.15.38:0/2750525248 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557948c9b1b0 con 0x557948d92140 2023-03-09T14:47:08.634 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.619+0000 7f43f1ffb700 1 -- 172.21.15.38:0/2750525248 <== mon.2 v2:172.21.15.122:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9278+0+0 (secure 0 0 0) 0x7f4404004c90 con 0x557948d92140 2023-03-09T14:47:08.634 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.619+0000 7f43f1ffb700 1 -- 172.21.15.38:0/2750525248 <== mon.2 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f44040060a0 con 0x557948d92140 2023-03-09T14:47:08.634 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.619+0000 7f441d124b80 10 client.25433 fetch_fsmap learned FSMap version 25 2023-03-09T14:47:08.634 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.619+0000 7f441d124b80 1 -- 172.21.15.38:0/2750525248 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x557948b32250 con 0x557948d92140 2023-03-09T14:47:08.635 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.619+0000 7f43f1ffb700 1 -- 172.21.15.38:0/2750525248 <== mon.2 v2:172.21.15.122:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x557948c9b1b0 con 0x557948d92140 2023-03-09T14:47:08.635 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.619+0000 7f43f1ffb700 1 -- 172.21.15.38:0/2750525248 <== mon.2 v2:172.21.15.122:3300/0 5 ==== fsmap(e 25) v1 ==== 7478+0+0 (secure 0 0 0) 0x7f4404007b50 con 0x557948d92140 2023-03-09T14:47:08.635 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.619+0000 7f441d124b80 10 client.25433 fetch_fsmap finished waiting for FSMap version 25 2023-03-09T14:47:08.635 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.619+0000 7f441d124b80 10 client.25433 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T14:47:08.635 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.619+0000 7f441d124b80 20 client.25433 populate_metadata read hostname 'smithi038' 2023-03-09T14:47:08.635 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.620+0000 7f441d124b80 1 --2- 172.21.15.38:0/2750525248 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x557948dfaf90 0x557948dfd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:08.636 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.620+0000 7f441d124b80 4 client.25433 mds_command: new command op to 24530 tid=0[{"prefix": "scrub status"}] 2023-03-09T14:47:08.636 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.620+0000 7f441d124b80 1 -- 172.21.15.38:0/2750525248 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x557948c851c0 con 0x557948dfaf90 2023-03-09T14:47:08.636 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.620+0000 7f440a59c700 1 --2- 172.21.15.38:0/2750525248 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x557948dfaf90 0x557948dfd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:08.636 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.620+0000 7f43f1ffb700 10 client.25433 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:47:08.637 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.620+0000 7f440a59c700 1 --2- 172.21.15.38:0/2750525248 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x557948dfaf90 0x557948dfd3a0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc39185c9ffc76bd server_cookie=39abb24a07de25d8 in_seq=0 out_seq=0 2023-03-09T14:47:08.637 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.620+0000 7f43f1ffb700 1 -- 172.21.15.38:0/2750525248 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x557948c851c0 con 0x557948dfaf90 2023-03-09T14:47:08.637 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.620+0000 7f43f1ffb700 10 client.25433 handle_command_reply: tid=0 2023-03-09T14:47:08.637 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.620+0000 7f441d124b80 1 client.25433 shutdown 2023-03-09T14:47:08.638 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.620+0000 7f441d124b80 1 -- 172.21.15.38:0/2750525248 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557948d92140 msgr2=0x557948d8d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:47:08.638 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.620+0000 7f441d124b80 1 --2- 172.21.15.38:0/2750525248 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557948d92140 0x557948d8d7d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4404009010 tx=0x7f4404012fc0 comp rx=0 tx=0).stop 2023-03-09T14:47:08.638 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.620+0000 7f441d124b80 1 -- 172.21.15.38:0/2750525248 shutdown_connections 2023-03-09T14:47:08.639 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.620+0000 7f441d124b80 1 --2- 172.21.15.38:0/2750525248 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x557948dfaf90 0x557948dfd3a0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:08.639 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.620+0000 7f441d124b80 1 --2- 172.21.15.38:0/2750525248 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557948d8bfe0 0x557948d8e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:08.639 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.620+0000 7f43f1ffb700 0 client.25433 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:47:08.639 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.620+0000 7f441d124b80 1 --2- 172.21.15.38:0/2750525248 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557948d92140 0x557948d8d7d0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:08.640 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.620+0000 7f441d124b80 1 --2- 172.21.15.38:0/2750525248 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557948d91830 0x557948d8d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:08.640 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.620+0000 7f441d124b80 1 -- 172.21.15.38:0/2750525248 >> 172.21.15.38:0/2750525248 conn(0x557948d86a20 msgr2=0x557948da1350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:47:08.640 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.621+0000 7f441d124b80 1 -- 172.21.15.38:0/2750525248 shutdown_connections 2023-03-09T14:47:08.640 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.621+0000 7f441d124b80 1 -- 172.21.15.38:0/2750525248 wait complete. 2023-03-09T14:47:08.641 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.621+0000 7f441d124b80 20 client.25433 trim_cache size 0 max 16384 2023-03-09T14:47:08.642 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2023-03-09T14:47:08.642 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "no active scrubs running", 2023-03-09T14:47:08.642 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": {} 2023-03-09T14:47:08.642 INFO:teuthology.orchestra.run.smithi038.stdout:} 2023-03-09T14:47:08.644 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.621+0000 7f4410f9b700 1 -- 172.21.15.38:0/2685954334 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f43f809f530 msgr2=0x7f43f80a19e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:47:08.645 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.621+0000 7f4410f9b700 1 --2- 172.21.15.38:0/2685954334 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f43f809f530 0x7f43f80a19e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f440c19dc30 tx=0x7f43f4009260 comp rx=0 tx=0).stop 2023-03-09T14:47:08.645 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.621+0000 7f4410f9b700 1 -- 172.21.15.38:0/2685954334 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f440c106820 msgr2=0x7f440c19c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:47:08.645 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.621+0000 7f4410f9b700 1 --2- 172.21.15.38:0/2685954334 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f440c106820 0x7f440c19c620 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f43fc000c00 tx=0x7f43fc004790 comp rx=0 tx=0).stop 2023-03-09T14:47:08.645 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.621+0000 7f4410f9b700 1 -- 172.21.15.38:0/2685954334 shutdown_connections 2023-03-09T14:47:08.646 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.621+0000 7f4410f9b700 1 --2- 172.21.15.38:0/2685954334 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f43f809f530 0x7f43f80a19e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:08.646 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.621+0000 7f4410f9b700 1 --2- 172.21.15.38:0/2685954334 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f440c10b880 0x7f440c1a0f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:08.646 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.621+0000 7f4410f9b700 1 --2- 172.21.15.38:0/2685954334 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f440c106820 0x7f440c19c620 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:08.646 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.621+0000 7f4410f9b700 1 --2- 172.21.15.38:0/2685954334 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f440c1071c0 0x7f440c19cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:08.647 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.621+0000 7f4410f9b700 1 -- 172.21.15.38:0/2685954334 >> 172.21.15.38:0/2685954334 conn(0x7f440c0744c0 msgr2=0x7f440c075010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:47:08.647 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.621+0000 7f4410f9b700 1 -- 172.21.15.38:0/2685954334 shutdown_connections 2023-03-09T14:47:08.647 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.621+0000 7f4410f9b700 1 -- 172.21.15.38:0/2685954334 wait complete. 2023-03-09T14:47:08.648 INFO:tasks.cephfs.filesystem:all active scrubs completed 2023-03-09T14:47:08.648 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-09T14:47:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.796+0000 7f1d3a3c7700 1 -- 172.21.15.38:0/1881938966 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1d340ff000 msgr2=0x7f1d340ff3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:47:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.796+0000 7f1d3a3c7700 1 --2- 172.21.15.38:0/1881938966 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1d340ff000 0x7f1d340ff3d0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f1d240099c0 tx=0x7f1d24009cd0 comp rx=0 tx=0).stop 2023-03-09T14:47:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.796+0000 7f1d3a3c7700 1 -- 172.21.15.38:0/1881938966 shutdown_connections 2023-03-09T14:47:08.801 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.796+0000 7f1d3a3c7700 1 --2- 172.21.15.38:0/1881938966 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1d340ff000 0x7f1d340ff3d0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:08.801 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.796+0000 7f1d3a3c7700 1 --2- 172.21.15.38:0/1881938966 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d34104040 0x7f1d3410bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:08.801 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.796+0000 7f1d3a3c7700 1 --2- 172.21.15.38:0/1881938966 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d340ff9a0 0x7f1d34103960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:08.801 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.796+0000 7f1d3a3c7700 1 -- 172.21.15.38:0/1881938966 >> 172.21.15.38:0/1881938966 conn(0x7f1d34076030 msgr2=0x7f1d34076430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:47:08.801 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.796+0000 7f1d3a3c7700 1 -- 172.21.15.38:0/1881938966 shutdown_connections 2023-03-09T14:47:08.801 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.797+0000 7f1d3a3c7700 1 -- 172.21.15.38:0/1881938966 wait complete. 2023-03-09T14:47:08.803 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.797+0000 7f1d3a3c7700 1 Processor -- start 2023-03-09T14:47:08.803 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.797+0000 7f1d3a3c7700 1 -- start start 2023-03-09T14:47:08.803 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.798+0000 7f1d3a3c7700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d340ff000 0x7f1d34100a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:08.803 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.798+0000 7f1d3a3c7700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d340ff9a0 0x7f1d34100f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:08.803 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.798+0000 7f1d3a3c7700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1d34104040 0x7f1d341a41c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:08.803 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.798+0000 7f1d3a3c7700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d34101770 con 0x7f1d34104040 2023-03-09T14:47:08.804 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.798+0000 7f1d3a3c7700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d341a4700 con 0x7f1d340ff000 2023-03-09T14:47:08.804 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.798+0000 7f1d3a3c7700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d341a4840 con 0x7f1d340ff9a0 2023-03-09T14:47:08.804 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.798+0000 7f1d33fff700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d340ff000 0x7f1d34100a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:08.804 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.798+0000 7f1d38964700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1d34104040 0x7f1d341a41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:08.804 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.798+0000 7f1d33fff700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d340ff000 0x7f1d34100a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.38:48750/0 (socket says 172.21.15.38:48750) 2023-03-09T14:47:08.804 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.798+0000 7f1d33fff700 1 -- 172.21.15.38:0/428990921 learned_addr learned my addr 172.21.15.38:0/428990921 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:47:08.804 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.798+0000 7f1d337fe700 1 --2- 172.21.15.38:0/428990921 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d340ff9a0 0x7f1d34100f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:08.805 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.798+0000 7f1d33fff700 1 -- 172.21.15.38:0/428990921 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d340ff9a0 msgr2=0x7f1d34100f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:47:08.805 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.798+0000 7f1d33fff700 1 --2- 172.21.15.38:0/428990921 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d340ff9a0 0x7f1d34100f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:08.805 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.798+0000 7f1d33fff700 1 -- 172.21.15.38:0/428990921 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1d34104040 msgr2=0x7f1d341a41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:47:08.805 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.798+0000 7f1d33fff700 1 --2- 172.21.15.38:0/428990921 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1d34104040 0x7f1d341a41c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:08.805 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.798+0000 7f1d33fff700 1 -- 172.21.15.38:0/428990921 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d240095f0 con 0x7f1d340ff000 2023-03-09T14:47:08.805 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.798+0000 7f1d38964700 1 --2- 172.21.15.38:0/428990921 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1d34104040 0x7f1d341a41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:47:08.805 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.798+0000 7f1d33fff700 1 --2- 172.21.15.38:0/428990921 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d340ff000 0x7f1d34100a00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1d2400b580 tx=0x7f1d2400bfd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:47:08.806 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.799+0000 7f1d317fa700 1 -- 172.21.15.38:0/428990921 <== mon.2 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d2401d070 con 0x7f1d340ff000 2023-03-09T14:47:08.806 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.799+0000 7f1d317fa700 1 -- 172.21.15.38:0/428990921 <== mon.2 v2:172.21.15.122:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1d24022470 con 0x7f1d340ff000 2023-03-09T14:47:08.806 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.799+0000 7f1d3a3c7700 1 -- 172.21.15.38:0/428990921 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d3410ee00 con 0x7f1d340ff000 2023-03-09T14:47:08.806 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.799+0000 7f1d317fa700 1 -- 172.21.15.38:0/428990921 <== mon.2 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d24017650 con 0x7f1d340ff000 2023-03-09T14:47:08.806 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.799+0000 7f1d3a3c7700 1 -- 172.21.15.38:0/428990921 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d3410ec70 con 0x7f1d340ff000 2023-03-09T14:47:08.806 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.800+0000 7f1d3a3c7700 1 -- 172.21.15.38:0/428990921 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d3410ef90 con 0x7f1d340ff000 2023-03-09T14:47:08.807 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.801+0000 7f1d317fa700 1 -- 172.21.15.38:0/428990921 <== mon.2 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 24) v1 ==== 133375+0+0 (secure 0 0 0) 0x7f1d24029030 con 0x7f1d340ff000 2023-03-09T14:47:08.807 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.801+0000 7f1d317fa700 1 --2- 172.21.15.38:0/428990921 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f1d1c09f530 0x7f1d1c0a19e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:08.807 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.802+0000 7f1d337fe700 1 --2- 172.21.15.38:0/428990921 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f1d1c09f530 0x7f1d1c0a19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:08.807 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.802+0000 7f1d317fa700 1 -- 172.21.15.38:0/428990921 <== mon.2 v2:172.21.15.122:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9278+0+0 (secure 0 0 0) 0x7f1d240cb2c0 con 0x7f1d340ff000 2023-03-09T14:47:08.807 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.802+0000 7f1d337fe700 1 --2- 172.21.15.38:0/428990921 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f1d1c09f530 0x7f1d1c0a19e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1d341021f0 tx=0x7f1d28009260 comp rx=0 tx=0).ready entity=mgr.15558 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:47:08.808 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:08.805+0000 7f1d317fa700 1 -- 172.21.15.38:0/428990921 <== mon.2 v2:172.21.15.122:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+177564 (secure 0 0 0) 0x7f1d24095180 con 0x7f1d340ff000 2023-03-09T14:47:09.119 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.116+0000 7f1d3a3c7700 1 -- 172.21.15.38:0/428990921 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1d34067660 con 0x7f1d340ff000 2023-03-09T14:47:09.120 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.117+0000 7f1d317fa700 1 -- 172.21.15.38:0/428990921 <== mon.2 v2:172.21.15.122:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+10804 (secure 0 0 0) 0x7f1d240987a0 con 0x7f1d340ff000 2023-03-09T14:47:09.120 INFO:teuthology.orchestra.run.smithi038.stdout: 2023-03-09T14:47:09.121 INFO:teuthology.orchestra.run.smithi038.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14544,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6835/768918602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6834","nonce":768918602},{"type":"v1","addr":"172.21.15.38:6835","nonce":768918602}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":14550,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6837/18381540","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6836","nonce":18381540},{"type":"v1","addr":"172.21.15.38:6837","nonce":18381540}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14610,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6839/4139020066","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6838","nonce":4139020066},{"type":"v1","addr":"172.21.15.38:6839","nonce":4139020066}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14622,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.38:6841/1397115322","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.38:6840","nonce":1397115322},{"type":"v1","addr":"172.21.15.38:6841","nonce":1397115322}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.181:6833/415305571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6832","nonce":415305571},{"type":"v1","addr":"172.21.15.181:6833","nonce":415305571}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.122:6833/659300520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6832","nonce":659300520},{"type":"v1","addr":"172.21.15.122:6833","nonce":659300520}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-03-09T14:44:19.083770+0000","modified":"2023-03-09T14:46:49.586911+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24530,"mds_1":24494,"mds_2":24478},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24448":{"gid":24448,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6835/2576255861","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6834","nonce":2576255861},{"type":"v1","addr":"172.21.15.181:6835","nonce":2576255861}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24460":{"gid":24460,"name":"c","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.181:6837/2942560181","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6836","nonce":2942560181},{"type":"v1","addr":"172.21.15.181:6837","nonce":2942560181}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24464":{"gid":24464,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.122:6835/3675537941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6834","nonce":3675537941},{"type":"v1","addr":"172.21.15.122:6835","nonce":3675537941}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24478":{"gid":24478,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":9,"addr":"172.21.15.181:6839/3902263660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.181:6838","nonce":3902263660},{"type":"v1","addr":"172.21.15.181:6839","nonce":3902263660}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24494":{"gid":24494,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":10,"addr":"172.21.15.122:6837/2210214596","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6836","nonce":2210214596},{"type":"v1","addr":"172.21.15.122:6837","nonce":2210214596}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24530":{"gid":24530,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.122:6839/971050165","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.122:6838","nonce":971050165},{"type":"v1","addr":"172.21.15.122:6839","nonce":971050165}]},"join_fscid":-1,"export_targets":[1,2],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-03-09T14:47:09.123 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.120+0000 7f1d3a3c7700 1 -- 172.21.15.38:0/428990921 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f1d1c09f530 msgr2=0x7f1d1c0a19e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:47:09.124 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.120+0000 7f1d3a3c7700 1 --2- 172.21.15.38:0/428990921 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f1d1c09f530 0x7f1d1c0a19e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1d341021f0 tx=0x7f1d28009260 comp rx=0 tx=0).stop 2023-03-09T14:47:09.124 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.120+0000 7f1d3a3c7700 1 -- 172.21.15.38:0/428990921 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d340ff000 msgr2=0x7f1d34100a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:47:09.124 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.120+0000 7f1d3a3c7700 1 --2- 172.21.15.38:0/428990921 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d340ff000 0x7f1d34100a00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1d2400b580 tx=0x7f1d2400bfd0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.124 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.120+0000 7f1d3a3c7700 1 -- 172.21.15.38:0/428990921 shutdown_connections 2023-03-09T14:47:09.124 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.120+0000 7f1d3a3c7700 1 --2- 172.21.15.38:0/428990921 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f1d1c09f530 0x7f1d1c0a19e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.124 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.120+0000 7f1d3a3c7700 1 --2- 172.21.15.38:0/428990921 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1d34104040 0x7f1d341a41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.125 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.120+0000 7f1d3a3c7700 1 --2- 172.21.15.38:0/428990921 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d340ff000 0x7f1d34100a00 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.125 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.120+0000 7f1d3a3c7700 1 --2- 172.21.15.38:0/428990921 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d340ff9a0 0x7f1d34100f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.125 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.120+0000 7f1d3a3c7700 1 -- 172.21.15.38:0/428990921 >> 172.21.15.38:0/428990921 conn(0x7f1d34076030 msgr2=0x7f1d34102c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:47:09.125 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.120+0000 7f1d3a3c7700 1 -- 172.21.15.38:0/428990921 shutdown_connections 2023-03-09T14:47:09.125 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.120+0000 7f1d3a3c7700 1 -- 172.21.15.38:0/428990921 wait complete. 2023-03-09T14:47:09.126 INFO:teuthology.orchestra.run.smithi038.stderr:dumped fsmap epoch 25 2023-03-09T14:47:09.137 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2023-03-09T14:47:09.290 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.286+0000 7fb472b46700 1 -- 172.21.15.38:0/2195623413 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb46c0ff020 msgr2=0x7fb46c0ff3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:47:09.290 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.286+0000 7fb472b46700 1 --2- 172.21.15.38:0/2195623413 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb46c0ff020 0x7fb46c0ff3f0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fb4600099c0 tx=0x7fb460009cd0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.291 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.286+0000 7fb472b46700 1 -- 172.21.15.38:0/2195623413 shutdown_connections 2023-03-09T14:47:09.291 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.286+0000 7fb472b46700 1 --2- 172.21.15.38:0/2195623413 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb46c0ff020 0x7fb46c0ff3f0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.291 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.286+0000 7fb472b46700 1 --2- 172.21.15.38:0/2195623413 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb46c104010 0x7fb46c10bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.291 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.286+0000 7fb472b46700 1 --2- 172.21.15.38:0/2195623413 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb46c0ff9c0 0x7fb46c103930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.291 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.286+0000 7fb472b46700 1 -- 172.21.15.38:0/2195623413 >> 172.21.15.38:0/2195623413 conn(0x7fb46c075ea0 msgr2=0x7fb46c0762a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:47:09.291 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.286+0000 7fb472b46700 1 -- 172.21.15.38:0/2195623413 shutdown_connections 2023-03-09T14:47:09.292 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.286+0000 7fb472b46700 1 -- 172.21.15.38:0/2195623413 wait complete. 2023-03-09T14:47:09.294 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.287+0000 7fb472b46700 1 Processor -- start 2023-03-09T14:47:09.294 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.287+0000 7fb472b46700 1 -- start start 2023-03-09T14:47:09.294 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.288+0000 7fb472b46700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb46c0ff020 0x7fb46c196e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:09.294 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.288+0000 7fb472b46700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb46c0ff9c0 0x7fb46c197370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:09.295 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.288+0000 7fb472b46700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb46c104010 0x7fb46c1a1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:09.295 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.288+0000 7fb472b46700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb46c197c80 con 0x7fb46c104010 2023-03-09T14:47:09.295 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.288+0000 7fb472b46700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb46c1a23f0 con 0x7fb46c0ff020 2023-03-09T14:47:09.295 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.288+0000 7fb472b46700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb46c1a2560 con 0x7fb46c0ff9c0 2023-03-09T14:47:09.295 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.288+0000 7fb4710e3700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb46c104010 0x7fb46c1a1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:09.295 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.288+0000 7fb46bfff700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb46c0ff9c0 0x7fb46c197370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:09.295 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.288+0000 7fb4710e3700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb46c104010 0x7fb46c1a1eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39578/0 (socket says 172.21.15.38:39578) 2023-03-09T14:47:09.295 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.288+0000 7fb4710e3700 1 -- 172.21.15.38:0/2029124768 learned_addr learned my addr 172.21.15.38:0/2029124768 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:47:09.296 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.288+0000 7fb4708e2700 1 --2- 172.21.15.38:0/2029124768 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb46c0ff020 0x7fb46c196e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:09.296 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.288+0000 7fb46bfff700 1 -- 172.21.15.38:0/2029124768 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb46c0ff020 msgr2=0x7fb46c196e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:47:09.296 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.288+0000 7fb46bfff700 1 --2- 172.21.15.38:0/2029124768 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb46c0ff020 0x7fb46c196e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.296 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.288+0000 7fb46bfff700 1 -- 172.21.15.38:0/2029124768 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb46c104010 msgr2=0x7fb46c1a1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:47:09.296 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.288+0000 7fb46bfff700 1 --2- 172.21.15.38:0/2029124768 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb46c104010 0x7fb46c1a1eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.296 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.288+0000 7fb46bfff700 1 -- 172.21.15.38:0/2029124768 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4600095f0 con 0x7fb46c0ff9c0 2023-03-09T14:47:09.296 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.288+0000 7fb4710e3700 1 --2- 172.21.15.38:0/2029124768 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb46c104010 0x7fb46c1a1eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:47:09.297 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.288+0000 7fb46bfff700 1 --2- 172.21.15.38:0/2029124768 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb46c0ff9c0 0x7fb46c197370 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb46c0ffd90 tx=0x7fb45800eca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:47:09.297 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.289+0000 7fb469ffb700 1 -- 172.21.15.38:0/2029124768 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb458010900 con 0x7fb46c0ff9c0 2023-03-09T14:47:09.297 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.289+0000 7fb472b46700 1 -- 172.21.15.38:0/2029124768 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb46c10ee20 con 0x7fb46c0ff9c0 2023-03-09T14:47:09.297 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.289+0000 7fb469ffb700 1 -- 172.21.15.38:0/2029124768 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb45800c430 con 0x7fb46c0ff9c0 2023-03-09T14:47:09.297 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.289+0000 7fb469ffb700 1 -- 172.21.15.38:0/2029124768 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb458018680 con 0x7fb46c0ff9c0 2023-03-09T14:47:09.297 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.289+0000 7fb472b46700 1 -- 172.21.15.38:0/2029124768 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb46c10ec90 con 0x7fb46c0ff9c0 2023-03-09T14:47:09.302 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.291+0000 7fb469ffb700 1 -- 172.21.15.38:0/2029124768 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 24) v1 ==== 133375+0+0 (secure 0 0 0) 0x7fb45800c040 con 0x7fb46c0ff9c0 2023-03-09T14:47:09.302 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.292+0000 7fb469ffb700 1 --2- 172.21.15.38:0/2029124768 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fb45c09f530 0x7fb45c0a19e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:09.302 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.292+0000 7fb469ffb700 1 -- 172.21.15.38:0/2029124768 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9278+0+0 (secure 0 0 0) 0x7fb458014070 con 0x7fb46c0ff9c0 2023-03-09T14:47:09.303 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.292+0000 7fb4708e2700 1 --2- 172.21.15.38:0/2029124768 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fb45c09f530 0x7fb45c0a19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:09.303 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.292+0000 7fb4708e2700 1 --2- 172.21.15.38:0/2029124768 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fb45c09f530 0x7fb45c0a19e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb460009990 tx=0x7fb460005fb0 comp rx=0 tx=0).ready entity=mgr.15558 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:47:09.303 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.293+0000 7fb47eccfb80 1 Processor -- start 2023-03-09T14:47:09.303 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.293+0000 7fb47eccfb80 1 -- start start 2023-03-09T14:47:09.303 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.293+0000 7fb47eccfb80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564f9c527830 0x564f9c527c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:09.303 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.293+0000 7fb47eccfb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564f9c528140 0x564f9c521aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:09.304 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.293+0000 7fb47eccfb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564f9c521fe0 0x564f9c522410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:09.304 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.293+0000 7fb47eccfb80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f9c40bda0 con 0x564f9c527830 2023-03-09T14:47:09.304 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.293+0000 7fb47eccfb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f9c47a780 con 0x564f9c528140 2023-03-09T14:47:09.304 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.293+0000 7fb47eccfb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f9c473cb0 con 0x564f9c521fe0 2023-03-09T14:47:09.304 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.293+0000 7fb4708e2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564f9c527830 0x564f9c527c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:09.304 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.293+0000 7fb4708e2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564f9c527830 0x564f9c527c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39582/0 (socket says 172.21.15.38:39582) 2023-03-09T14:47:09.305 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.293+0000 7fb4708e2700 1 -- 172.21.15.38:0/189285630 learned_addr learned my addr 172.21.15.38:0/189285630 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:47:09.305 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.293+0000 7fb4710e3700 1 --2- 172.21.15.38:0/189285630 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564f9c521fe0 0x564f9c522410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:09.305 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.293+0000 7fb46bfff700 1 --2- 172.21.15.38:0/189285630 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564f9c528140 0x564f9c521aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:09.305 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.293+0000 7fb4708e2700 1 -- 172.21.15.38:0/189285630 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564f9c521fe0 msgr2=0x564f9c522410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:47:09.305 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.293+0000 7fb4708e2700 1 --2- 172.21.15.38:0/189285630 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564f9c521fe0 0x564f9c522410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.305 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.293+0000 7fb4708e2700 1 -- 172.21.15.38:0/189285630 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564f9c528140 msgr2=0x564f9c521aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:47:09.306 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.293+0000 7fb4708e2700 1 --2- 172.21.15.38:0/189285630 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564f9c528140 0x564f9c521aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.306 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.293+0000 7fb4708e2700 1 -- 172.21.15.38:0/189285630 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564f9c43a190 con 0x564f9c527830 2023-03-09T14:47:09.306 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.293+0000 7fb4710e3700 1 --2- 172.21.15.38:0/189285630 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564f9c521fe0 0x564f9c522410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:47:09.306 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.293+0000 7fb4708e2700 1 --2- 172.21.15.38:0/189285630 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564f9c527830 0x564f9c527c00 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fb4600095a0 tx=0x7fb46001f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=376c81e10bd46a30 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:47:09.306 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.294+0000 7fb456ffd700 1 -- 172.21.15.38:0/189285630 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb460003880 con 0x564f9c527830 2023-03-09T14:47:09.306 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.294+0000 7fb456ffd700 1 -- 172.21.15.38:0/189285630 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb46001e040 con 0x564f9c527830 2023-03-09T14:47:09.306 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.294+0000 7fb456ffd700 1 -- 172.21.15.38:0/189285630 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb460004800 con 0x564f9c527830 2023-03-09T14:47:09.307 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.294+0000 7fb47eccfb80 1 -- 172.21.15.38:0/189285630 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564f9c527830 msgr2=0x564f9c527c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:47:09.307 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.294+0000 7fb47eccfb80 1 --2- 172.21.15.38:0/189285630 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564f9c527830 0x564f9c527c00 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fb4600095a0 tx=0x7fb46001f000 comp rx=0 tx=0).stop 2023-03-09T14:47:09.307 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.294+0000 7fb47eccfb80 1 -- 172.21.15.38:0/189285630 shutdown_connections 2023-03-09T14:47:09.307 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.294+0000 7fb47eccfb80 1 --2- 172.21.15.38:0/189285630 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564f9c527830 0x564f9c527c00 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.307 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.294+0000 7fb47eccfb80 1 --2- 172.21.15.38:0/189285630 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564f9c521fe0 0x564f9c522410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.307 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.294+0000 7fb47eccfb80 1 --2- 172.21.15.38:0/189285630 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564f9c528140 0x564f9c521aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.307 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.294+0000 7fb47eccfb80 1 -- 172.21.15.38:0/189285630 >> 172.21.15.38:0/189285630 conn(0x564f9c51ca20 msgr2=0x564f9c51cdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:47:09.307 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.295+0000 7fb47eccfb80 1 -- 172.21.15.38:0/189285630 shutdown_connections 2023-03-09T14:47:09.308 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.295+0000 7fb47eccfb80 1 -- 172.21.15.38:0/189285630 wait complete. 2023-03-09T14:47:09.308 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.295+0000 7fb47eccfb80 1 -- start start 2023-03-09T14:47:09.308 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.295+0000 7fb47eccfb80 1 Processor -- start 2023-03-09T14:47:09.308 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.295+0000 7fb47eccfb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564f9c521fe0 0x564f9c52e160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:09.308 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.295+0000 7fb47eccfb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564f9c527830 0x564f9c52b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:09.308 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.295+0000 7fb47eccfb80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564f9c528140 0x564f9c52bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:09.308 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.295+0000 7fb47eccfb80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f9c2f7920 con 0x564f9c528140 2023-03-09T14:47:09.308 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.295+0000 7fb47eccfb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f9c52c300 con 0x564f9c521fe0 2023-03-09T14:47:09.309 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.295+0000 7fb47eccfb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f9c537c10 con 0x564f9c527830 2023-03-09T14:47:09.309 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.296+0000 7fb47eccfb80 10 client.0 _ll_register_callbacks cb 0x564f9c419520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:47:09.309 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.296+0000 7fb46bfff700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564f9c527830 0x564f9c52b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:09.309 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.296+0000 7fb4708e2700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564f9c521fe0 0x564f9c52e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:09.309 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.296+0000 7fb4708e2700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564f9c521fe0 0x564f9c52e160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.38:48778/0 (socket says 172.21.15.38:48778) 2023-03-09T14:47:09.309 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.296+0000 7fb46bfff700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564f9c527830 0x564f9c52b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.38:47288/0 (socket says 172.21.15.38:47288) 2023-03-09T14:47:09.309 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.296+0000 7fb4708e2700 1 -- 172.21.15.38:0/3575588700 learned_addr learned my addr 172.21.15.38:0/3575588700 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:47:09.310 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.296+0000 7fb4710e3700 1 --2- 172.21.15.38:0/3575588700 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564f9c528140 0x564f9c52bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:09.310 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.296+0000 7fb4708e2700 1 -- 172.21.15.38:0/3575588700 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564f9c527830 msgr2=0x564f9c52b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:47:09.310 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.296+0000 7fb4708e2700 1 --2- 172.21.15.38:0/3575588700 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564f9c527830 0x564f9c52b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.310 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.296+0000 7fb4708e2700 1 -- 172.21.15.38:0/3575588700 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564f9c528140 msgr2=0x564f9c52bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:47:09.310 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.296+0000 7fb4708e2700 1 --2- 172.21.15.38:0/3575588700 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564f9c528140 0x564f9c52bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.310 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.296+0000 7fb4708e2700 1 -- 172.21.15.38:0/3575588700 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb464009110 con 0x564f9c521fe0 2023-03-09T14:47:09.310 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.296+0000 7fb4708e2700 1 --2- 172.21.15.38:0/3575588700 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564f9c521fe0 0x564f9c52e160 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb4600098e0 tx=0x7fb460004000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f03e3f354c5cff1 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:47:09.310 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.296+0000 7fb44bfff700 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2023-03-09T14:47:09.311 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.296+0000 7fb44bfff700 1 -- 172.21.15.38:0/3575588700 <== mon.2 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb460030a40 con 0x564f9c521fe0 2023-03-09T14:47:09.311 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.296+0000 7fb44bfff700 1 -- 172.21.15.38:0/3575588700 <== mon.2 v2:172.21.15.122:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9278+0+0 (secure 0 0 0) 0x7fb460030ba0 con 0x564f9c521fe0 2023-03-09T14:47:09.311 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.296+0000 7fb47eccfb80 1 -- 172.21.15.38:0/3575588700 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564f9c441190 con 0x564f9c521fe0 2023-03-09T14:47:09.311 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.297+0000 7fb44bfff700 1 -- 172.21.15.38:0/3575588700 <== mon.2 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb460030d60 con 0x564f9c521fe0 2023-03-09T14:47:09.311 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.297+0000 7fb47eccfb80 10 client.25451 fetch_fsmap learned FSMap version 25 2023-03-09T14:47:09.311 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.297+0000 7fb47eccfb80 1 -- 172.21.15.38:0/3575588700 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x7fb46c10c420 con 0x564f9c521fe0 2023-03-09T14:47:09.311 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.297+0000 7fb44bfff700 1 -- 172.21.15.38:0/3575588700 <== mon.2 v2:172.21.15.122:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x564f9c441190 con 0x564f9c521fe0 2023-03-09T14:47:09.312 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.297+0000 7fb44bfff700 1 -- 172.21.15.38:0/3575588700 <== mon.2 v2:172.21.15.122:3300/0 5 ==== fsmap(e 25) v1 ==== 7478+0+0 (secure 0 0 0) 0x7fb4600398f0 con 0x564f9c521fe0 2023-03-09T14:47:09.312 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.297+0000 7fb47eccfb80 10 client.25451 fetch_fsmap finished waiting for FSMap version 25 2023-03-09T14:47:09.312 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.297+0000 7fb47eccfb80 10 client.25451 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-03-09T14:47:09.312 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.297+0000 7fb47eccfb80 20 client.25451 populate_metadata read hostname 'smithi038' 2023-03-09T14:47:09.312 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.297+0000 7fb47eccfb80 1 --2- 172.21.15.38:0/3575588700 >> [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] conn(0x564f9c53b810 0x564f9c53dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:09.313 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.297+0000 7fb47eccfb80 4 client.25451 mds_command: new command op to 24478 tid=0[{"prefix": "get_command_descriptions"}] 2023-03-09T14:47:09.313 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.297+0000 7fb47eccfb80 1 -- 172.21.15.38:0/3575588700 --> [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564f9c2f7f60 con 0x564f9c53b810 2023-03-09T14:47:09.313 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.297+0000 7fb46bfff700 1 --2- 172.21.15.38:0/3575588700 >> [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] conn(0x564f9c53b810 0x564f9c53dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:09.313 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.298+0000 7fb44bfff700 10 client.25451 ms_handle_connect on v2:172.21.15.181:6838/3902263660 2023-03-09T14:47:09.313 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.298+0000 7fb46bfff700 1 --2- 172.21.15.38:0/3575588700 >> [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] conn(0x564f9c53b810 0x564f9c53dbf0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=86dcaeee6950f252 server_cookie=926cad74702f22ea in_seq=0 out_seq=0 2023-03-09T14:47:09.313 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.298+0000 7fb44bfff700 1 -- 172.21.15.38:0/3575588700 <== mds.2 v2:172.21.15.181:6838/3902263660 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18278 (crc 0 0 0) 0x564f9c2f7f60 con 0x564f9c53b810 2023-03-09T14:47:09.314 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.298+0000 7fb44bfff700 10 client.25451 handle_command_reply: tid=0 2023-03-09T14:47:09.314 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.299+0000 7fb47eccfb80 1 client.25451 shutdown 2023-03-09T14:47:09.314 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.299+0000 7fb47eccfb80 1 -- 172.21.15.38:0/3575588700 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564f9c521fe0 msgr2=0x564f9c52e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:47:09.314 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.299+0000 7fb47eccfb80 1 --2- 172.21.15.38:0/3575588700 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564f9c521fe0 0x564f9c52e160 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb4600098e0 tx=0x7fb460004000 comp rx=0 tx=0).stop 2023-03-09T14:47:09.315 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.299+0000 7fb47eccfb80 1 -- 172.21.15.38:0/3575588700 shutdown_connections 2023-03-09T14:47:09.315 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.299+0000 7fb47eccfb80 1 --2- 172.21.15.38:0/3575588700 >> [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] conn(0x564f9c53b810 0x564f9c53dbf0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.315 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.299+0000 7fb47eccfb80 1 --2- 172.21.15.38:0/3575588700 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564f9c528140 0x564f9c52bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.315 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.299+0000 7fb47eccfb80 1 --2- 172.21.15.38:0/3575588700 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564f9c521fe0 0x564f9c52e160 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.315 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.299+0000 7fb47eccfb80 1 --2- 172.21.15.38:0/3575588700 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564f9c527830 0x564f9c52b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.315 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.299+0000 7fb44bfff700 0 client.25451 ms_handle_reset on v2:172.21.15.181:6838/3902263660 2023-03-09T14:47:09.315 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.299+0000 7fb47eccfb80 1 -- 172.21.15.38:0/3575588700 >> 172.21.15.38:0/3575588700 conn(0x564f9c51ca20 msgr2=0x564f9c51bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:47:09.316 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.299+0000 7fb47eccfb80 1 -- 172.21.15.38:0/3575588700 shutdown_connections 2023-03-09T14:47:09.316 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.299+0000 7fb47eccfb80 1 -- 172.21.15.38:0/3575588700 wait complete. 2023-03-09T14:47:09.316 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.299+0000 7fb47eccfb80 20 client.25451 trim_cache size 0 max 16384 2023-03-09T14:47:09.318 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.314+0000 7fb47eccfb80 1 Processor -- start 2023-03-09T14:47:09.318 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.315+0000 7fb47eccfb80 1 -- start start 2023-03-09T14:47:09.318 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.315+0000 7fb47eccfb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564f9c521fe0 0x564f9c51bc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:09.321 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.315+0000 7fb47eccfb80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564f9c527830 0x564f9c52dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:09.321 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.315+0000 7fb47eccfb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564f9c528140 0x564f9c52e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:09.321 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.315+0000 7fb47eccfb80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f9c522cc0 con 0x564f9c527830 2023-03-09T14:47:09.322 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.315+0000 7fb47eccfb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f9c52c9c0 con 0x564f9c528140 2023-03-09T14:47:09.322 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.315+0000 7fb47eccfb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f9c527460 con 0x564f9c521fe0 2023-03-09T14:47:09.322 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.315+0000 7fb4708e2700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564f9c521fe0 0x564f9c51bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:09.323 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.315+0000 7fb4708e2700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564f9c521fe0 0x564f9c51bc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.38:47294/0 (socket says 172.21.15.38:47294) 2023-03-09T14:47:09.323 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.315+0000 7fb4708e2700 1 -- 172.21.15.38:0/931157588 learned_addr learned my addr 172.21.15.38:0/931157588 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:47:09.323 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.315+0000 7fb4710e3700 1 --2- 172.21.15.38:0/931157588 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564f9c528140 0x564f9c52e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:09.323 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.315+0000 7fb46bfff700 1 --2- 172.21.15.38:0/931157588 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564f9c527830 0x564f9c52dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:09.324 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.315+0000 7fb4708e2700 1 -- 172.21.15.38:0/931157588 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564f9c528140 msgr2=0x564f9c52e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:47:09.324 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.315+0000 7fb4708e2700 1 --2- 172.21.15.38:0/931157588 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564f9c528140 0x564f9c52e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.324 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.315+0000 7fb4708e2700 1 -- 172.21.15.38:0/931157588 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564f9c527830 msgr2=0x564f9c52dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:47:09.324 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.315+0000 7fb4708e2700 1 --2- 172.21.15.38:0/931157588 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564f9c527830 0x564f9c52dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.325 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.315+0000 7fb4708e2700 1 -- 172.21.15.38:0/931157588 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564f9c28ac60 con 0x564f9c521fe0 2023-03-09T14:47:09.325 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.316+0000 7fb4708e2700 1 --2- 172.21.15.38:0/931157588 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564f9c521fe0 0x564f9c51bc20 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb4600099c0 tx=0x7fb4600173f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9629865127beeda2 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:47:09.325 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.316+0000 7fb4577fe700 1 -- 172.21.15.38:0/931157588 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb460039680 con 0x564f9c521fe0 2023-03-09T14:47:09.325 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.316+0000 7fb4577fe700 1 -- 172.21.15.38:0/931157588 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb46001e070 con 0x564f9c521fe0 2023-03-09T14:47:09.326 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.316+0000 7fb4577fe700 1 -- 172.21.15.38:0/931157588 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb46000b970 con 0x564f9c521fe0 2023-03-09T14:47:09.326 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.317+0000 7fb47eccfb80 1 -- 172.21.15.38:0/931157588 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564f9c521fe0 msgr2=0x564f9c51bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:47:09.326 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.317+0000 7fb47eccfb80 1 --2- 172.21.15.38:0/931157588 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564f9c521fe0 0x564f9c51bc20 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb4600099c0 tx=0x7fb4600173f0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.327 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.317+0000 7fb47eccfb80 1 -- 172.21.15.38:0/931157588 shutdown_connections 2023-03-09T14:47:09.327 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.317+0000 7fb47eccfb80 1 --2- 172.21.15.38:0/931157588 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564f9c528140 0x564f9c52e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.327 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.317+0000 7fb47eccfb80 1 --2- 172.21.15.38:0/931157588 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564f9c521fe0 0x564f9c51bc20 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.327 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.317+0000 7fb47eccfb80 1 --2- 172.21.15.38:0/931157588 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564f9c527830 0x564f9c52dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.328 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.317+0000 7fb47eccfb80 1 -- 172.21.15.38:0/931157588 >> 172.21.15.38:0/931157588 conn(0x564f9c51ca20 msgr2=0x564f9c51af70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:47:09.328 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.317+0000 7fb47eccfb80 1 -- 172.21.15.38:0/931157588 shutdown_connections 2023-03-09T14:47:09.328 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.317+0000 7fb47eccfb80 1 -- 172.21.15.38:0/931157588 wait complete. 2023-03-09T14:47:09.328 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.317+0000 7fb47eccfb80 1 -- start start 2023-03-09T14:47:09.328 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.317+0000 7fb47eccfb80 1 Processor -- start 2023-03-09T14:47:09.332 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.318+0000 7fb47eccfb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564f9c521fe0 0x564f9c524740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:09.332 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.318+0000 7fb47eccfb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564f9c527830 0x564f9c524c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:09.332 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.318+0000 7fb47eccfb80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564f9c528140 0x564f9c523290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:09.333 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.318+0000 7fb47eccfb80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f9c523800 con 0x564f9c528140 2023-03-09T14:47:09.333 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.318+0000 7fb47eccfb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f9c523970 con 0x564f9c521fe0 2023-03-09T14:47:09.333 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.318+0000 7fb47eccfb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f9c523b30 con 0x564f9c527830 2023-03-09T14:47:09.334 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.318+0000 7fb47eccfb80 10 client.0 _ll_register_callbacks cb 0x564f9c419520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:47:09.334 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.318+0000 7fb4710e3700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564f9c528140 0x564f9c523290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:09.334 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.318+0000 7fb46bfff700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564f9c527830 0x564f9c524c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:09.334 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.318+0000 7fb4710e3700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564f9c528140 0x564f9c523290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39604/0 (socket says 172.21.15.38:39604) 2023-03-09T14:47:09.335 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.318+0000 7fb46bfff700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564f9c527830 0x564f9c524c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.38:47300/0 (socket says 172.21.15.38:47300) 2023-03-09T14:47:09.335 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.318+0000 7fb4710e3700 1 -- 172.21.15.38:0/3734553062 learned_addr learned my addr 172.21.15.38:0/3734553062 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:47:09.335 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.318+0000 7fb46bfff700 1 -- 172.21.15.38:0/3734553062 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564f9c521fe0 msgr2=0x564f9c524740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-03-09T14:47:09.335 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.318+0000 7fb46bfff700 1 --2- 172.21.15.38:0/3734553062 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564f9c521fe0 0x564f9c524740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.336 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.318+0000 7fb46bfff700 1 -- 172.21.15.38:0/3734553062 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564f9c528140 msgr2=0x564f9c523290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:47:09.336 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.318+0000 7fb46bfff700 1 --2- 172.21.15.38:0/3734553062 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564f9c528140 0x564f9c523290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.336 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.318+0000 7fb46bfff700 1 -- 172.21.15.38:0/3734553062 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb46c10c420 con 0x564f9c527830 2023-03-09T14:47:09.336 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.319+0000 7fb46bfff700 1 --2- 172.21.15.38:0/3734553062 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564f9c527830 0x564f9c524c80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb458097d40 tx=0x7fb45801a040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3896402e3e471649 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:47:09.337 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.319+0000 7fb44bfff700 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2023-03-09T14:47:09.337 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.319+0000 7fb44bfff700 1 -- 172.21.15.38:0/3734553062 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4580b0170 con 0x564f9c527830 2023-03-09T14:47:09.337 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.319+0000 7fb44bfff700 1 -- 172.21.15.38:0/3734553062 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9278+0+0 (secure 0 0 0) 0x7fb4580b02d0 con 0x564f9c527830 2023-03-09T14:47:09.337 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.319+0000 7fb47eccfb80 1 -- 172.21.15.38:0/3734553062 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564f9c4311b0 con 0x564f9c527830 2023-03-09T14:47:09.338 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.319+0000 7fb44bfff700 1 -- 172.21.15.38:0/3734553062 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb4580c7350 con 0x564f9c527830 2023-03-09T14:47:09.338 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.319+0000 7fb47eccfb80 10 client.25408 fetch_fsmap learned FSMap version 25 2023-03-09T14:47:09.338 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.319+0000 7fb47eccfb80 1 -- 172.21.15.38:0/3734553062 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x564f9c2c8250 con 0x564f9c527830 2023-03-09T14:47:09.338 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.319+0000 7fb44bfff700 1 -- 172.21.15.38:0/3734553062 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb458062020 con 0x564f9c527830 2023-03-09T14:47:09.338 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.320+0000 7fb44bfff700 1 -- 172.21.15.38:0/3734553062 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 25) v1 ==== 7478+0+0 (secure 0 0 0) 0x7fb458097530 con 0x564f9c527830 2023-03-09T14:47:09.338 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.320+0000 7fb47eccfb80 10 client.25408 fetch_fsmap finished waiting for FSMap version 25 2023-03-09T14:47:09.339 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.320+0000 7fb47eccfb80 10 client.25408 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2023-03-09T14:47:09.339 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.320+0000 7fb47eccfb80 20 client.25408 populate_metadata read hostname 'smithi038' 2023-03-09T14:47:09.339 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.320+0000 7fb47eccfb80 1 --2- 172.21.15.38:0/3734553062 >> [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] conn(0x564f9c598e40 0x564f9c59b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:09.339 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.320+0000 7fb47eccfb80 4 client.25408 mds_command: new command op to 24478 tid=0[{"prefix": "damage ls"}] 2023-03-09T14:47:09.339 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.320+0000 7fb47eccfb80 1 -- 172.21.15.38:0/3734553062 --> [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x564f9c41b1c0 con 0x564f9c598e40 2023-03-09T14:47:09.339 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.320+0000 7fb4708e2700 1 --2- 172.21.15.38:0/3734553062 >> [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] conn(0x564f9c598e40 0x564f9c59b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:09.339 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.320+0000 7fb44bfff700 10 client.25408 ms_handle_connect on v2:172.21.15.181:6838/3902263660 2023-03-09T14:47:09.339 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.320+0000 7fb4708e2700 1 --2- 172.21.15.38:0/3734553062 >> [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] conn(0x564f9c598e40 0x564f9c59b250 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4a625cb2d5df75fe server_cookie=521749eaa58fb816 in_seq=0 out_seq=0 2023-03-09T14:47:09.340 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.320+0000 7fb44bfff700 1 -- 172.21.15.38:0/3734553062 <== mds.2 v2:172.21.15.181:6838/3902263660 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x564f9c41b1c0 con 0x564f9c598e40 2023-03-09T14:47:09.340 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.320+0000 7fb44bfff700 10 client.25408 handle_command_reply: tid=0 2023-03-09T14:47:09.340 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.320+0000 7fb47eccfb80 1 client.25408 shutdown 2023-03-09T14:47:09.340 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.320+0000 7fb47eccfb80 1 -- 172.21.15.38:0/3734553062 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564f9c527830 msgr2=0x564f9c524c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:47:09.340 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.320+0000 7fb47eccfb80 1 --2- 172.21.15.38:0/3734553062 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564f9c527830 0x564f9c524c80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb458097d40 tx=0x7fb45801a040 comp rx=0 tx=0).stop 2023-03-09T14:47:09.340 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.320+0000 7fb47eccfb80 1 -- 172.21.15.38:0/3734553062 shutdown_connections 2023-03-09T14:47:09.340 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.320+0000 7fb47eccfb80 1 --2- 172.21.15.38:0/3734553062 >> [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] conn(0x564f9c598e40 0x564f9c59b250 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.341 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.320+0000 7fb47eccfb80 1 --2- 172.21.15.38:0/3734553062 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564f9c528140 0x564f9c523290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.341 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.320+0000 7fb44bfff700 0 client.25408 ms_handle_reset on v2:172.21.15.181:6838/3902263660 2023-03-09T14:47:09.341 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.320+0000 7fb47eccfb80 1 --2- 172.21.15.38:0/3734553062 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564f9c521fe0 0x564f9c524740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.341 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.321+0000 7fb47eccfb80 1 --2- 172.21.15.38:0/3734553062 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564f9c527830 0x564f9c524c80 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.341 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.321+0000 7fb47eccfb80 1 -- 172.21.15.38:0/3734553062 >> 172.21.15.38:0/3734553062 conn(0x564f9c51ca20 msgr2=0x564f9c51bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:47:09.341 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.321+0000 7fb47eccfb80 1 -- 172.21.15.38:0/3734553062 shutdown_connections 2023-03-09T14:47:09.341 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.321+0000 7fb47eccfb80 1 -- 172.21.15.38:0/3734553062 wait complete. 2023-03-09T14:47:09.342 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.321+0000 7fb47eccfb80 20 client.25408 trim_cache size 0 max 16384 2023-03-09T14:47:09.342 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2023-03-09T14:47:09.343 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.321+0000 7fb472b46700 1 -- 172.21.15.38:0/2029124768 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fb45c09f530 msgr2=0x7fb45c0a19e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:47:09.343 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.321+0000 7fb472b46700 1 --2- 172.21.15.38:0/2029124768 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fb45c09f530 0x7fb45c0a19e0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb460009990 tx=0x7fb460005fb0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.344 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.321+0000 7fb472b46700 1 -- 172.21.15.38:0/2029124768 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb46c0ff9c0 msgr2=0x7fb46c197370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:47:09.344 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.321+0000 7fb472b46700 1 --2- 172.21.15.38:0/2029124768 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb46c0ff9c0 0x7fb46c197370 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fb46c0ffd90 tx=0x7fb45800eca0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.344 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.321+0000 7fb472b46700 1 -- 172.21.15.38:0/2029124768 shutdown_connections 2023-03-09T14:47:09.344 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.321+0000 7fb472b46700 1 --2- 172.21.15.38:0/2029124768 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fb45c09f530 0x7fb45c0a19e0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.344 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.321+0000 7fb472b46700 1 --2- 172.21.15.38:0/2029124768 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb46c104010 0x7fb46c1a1eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.344 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.321+0000 7fb472b46700 1 --2- 172.21.15.38:0/2029124768 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb46c0ff020 0x7fb46c196e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.344 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.321+0000 7fb472b46700 1 --2- 172.21.15.38:0/2029124768 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb46c0ff9c0 0x7fb46c197370 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.345 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.321+0000 7fb472b46700 1 -- 172.21.15.38:0/2029124768 >> 172.21.15.38:0/2029124768 conn(0x7fb46c075ea0 msgr2=0x7fb46c0fc5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:47:09.345 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.322+0000 7fb472b46700 1 -- 172.21.15.38:0/2029124768 shutdown_connections 2023-03-09T14:47:09.345 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.322+0000 7fb472b46700 1 -- 172.21.15.38:0/2029124768 wait complete. 2023-03-09T14:47:09.345 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2023-03-09T14:47:09.504 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.500+0000 7f65f5354700 1 -- 172.21.15.38:0/1176016350 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f65f0100410 msgr2=0x7f65f01007e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:47:09.504 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.500+0000 7f65f5354700 1 --2- 172.21.15.38:0/1176016350 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f65f0100410 0x7f65f01007e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f65e0009990 tx=0x7f65e0009ca0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.505 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.500+0000 7f65f5354700 1 -- 172.21.15.38:0/1176016350 shutdown_connections 2023-03-09T14:47:09.505 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.500+0000 7f65f5354700 1 --2- 172.21.15.38:0/1176016350 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f65f01099c0 0x7f65f010bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.505 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.500+0000 7f65f5354700 1 --2- 172.21.15.38:0/1176016350 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f65f0100410 0x7f65f01007e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.506 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.500+0000 7f65f5354700 1 --2- 172.21.15.38:0/1176016350 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f65f0100d20 0x7f65f0109480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.506 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.500+0000 7f65f5354700 1 -- 172.21.15.38:0/1176016350 >> 172.21.15.38:0/1176016350 conn(0x7f65f00748f0 msgr2=0x7f65f0074cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:47:09.506 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.500+0000 7f65f5354700 1 -- 172.21.15.38:0/1176016350 shutdown_connections 2023-03-09T14:47:09.506 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.500+0000 7f65f5354700 1 -- 172.21.15.38:0/1176016350 wait complete. 2023-03-09T14:47:09.507 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.501+0000 7f65f5354700 1 Processor -- start 2023-03-09T14:47:09.507 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.501+0000 7f65f5354700 1 -- start start 2023-03-09T14:47:09.513 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.501+0000 7f65f5354700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f65f0100410 0x7f65f019d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:09.514 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.501+0000 7f65f5354700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f65f0100d20 0x7f65f019d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:09.514 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.501+0000 7f65f5354700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f65f01099c0 0x7f65f0108190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:09.514 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.501+0000 7f65f5354700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65f01088a0 con 0x7f65f0100410 2023-03-09T14:47:09.515 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.501+0000 7f65f5354700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65f0108a10 con 0x7f65f0100d20 2023-03-09T14:47:09.515 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.501+0000 7f65f5354700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65f0108bd0 con 0x7f65f01099c0 2023-03-09T14:47:09.515 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.501+0000 7f65eeffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f65f0100410 0x7f65f019d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:09.515 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.501+0000 7f65ef7fe700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f65f01099c0 0x7f65f0108190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:09.515 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.501+0000 7f65eeffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f65f0100410 0x7f65f019d240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39614/0 (socket says 172.21.15.38:39614) 2023-03-09T14:47:09.515 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.501+0000 7f65eeffd700 1 -- 172.21.15.38:0/2631757914 learned_addr learned my addr 172.21.15.38:0/2631757914 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:47:09.516 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.501+0000 7f65ee7fc700 1 --2- 172.21.15.38:0/2631757914 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f65f0100d20 0x7f65f019d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:09.516 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.502+0000 7f65eeffd700 1 -- 172.21.15.38:0/2631757914 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f65f01099c0 msgr2=0x7f65f0108190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:47:09.516 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.502+0000 7f65eeffd700 1 --2- 172.21.15.38:0/2631757914 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f65f01099c0 0x7f65f0108190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.516 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.502+0000 7f65eeffd700 1 -- 172.21.15.38:0/2631757914 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f65f0100d20 msgr2=0x7f65f019d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:47:09.517 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.502+0000 7f65eeffd700 1 --2- 172.21.15.38:0/2631757914 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f65f0100d20 0x7f65f019d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.517 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.502+0000 7f65eeffd700 1 -- 172.21.15.38:0/2631757914 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65e00095f0 con 0x7f65f0100410 2023-03-09T14:47:09.517 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.502+0000 7f65ef7fe700 1 --2- 172.21.15.38:0/2631757914 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f65f01099c0 0x7f65f0108190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:47:09.517 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.502+0000 7f65ee7fc700 1 --2- 172.21.15.38:0/2631757914 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f65f0100d20 0x7f65f019d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:47:09.518 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.502+0000 7f65eeffd700 1 --2- 172.21.15.38:0/2631757914 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f65f0100410 0x7f65f019d240 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f65e0005740 tx=0x7f65e0005770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:47:09.518 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.502+0000 7f65d7fff700 1 -- 172.21.15.38:0/2631757914 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65e001d070 con 0x7f65f0100410 2023-03-09T14:47:09.518 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.502+0000 7f65f5354700 1 -- 172.21.15.38:0/2631757914 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65f00ffc10 con 0x7f65f0100410 2023-03-09T14:47:09.518 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.502+0000 7f65d7fff700 1 -- 172.21.15.38:0/2631757914 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f65e0022470 con 0x7f65f0100410 2023-03-09T14:47:09.519 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.502+0000 7f65d7fff700 1 -- 172.21.15.38:0/2631757914 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65e0017650 con 0x7f65f0100410 2023-03-09T14:47:09.519 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.502+0000 7f65f5354700 1 -- 172.21.15.38:0/2631757914 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65f00ffa80 con 0x7f65f0100410 2023-03-09T14:47:09.519 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.506+0000 7f65d7fff700 1 -- 172.21.15.38:0/2631757914 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 24) v1 ==== 133375+0+0 (secure 0 0 0) 0x7f65e0029030 con 0x7f65f0100410 2023-03-09T14:47:09.519 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.506+0000 7f65d7fff700 1 --2- 172.21.15.38:0/2631757914 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f65d809f530 0x7f65d80a19e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:09.520 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.507+0000 7f65d7fff700 1 -- 172.21.15.38:0/2631757914 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9278+0+0 (secure 0 0 0) 0x7f65e00cb720 con 0x7f65f0100410 2023-03-09T14:47:09.520 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.507+0000 7f65ee7fc700 1 --2- 172.21.15.38:0/2631757914 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f65d809f530 0x7f65d80a19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:09.520 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.507+0000 7f66014ddb80 1 Processor -- start 2023-03-09T14:47:09.520 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.507+0000 7f66014ddb80 1 -- start start 2023-03-09T14:47:09.521 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.507+0000 7f65ee7fc700 1 --2- 172.21.15.38:0/2631757914 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f65d809f530 0x7f65d80a19e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f65f0199c40 tx=0x7f65e4009190 comp rx=0 tx=0).ready entity=mgr.15558 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:47:09.521 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.507+0000 7f66014ddb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561451ba8830 0x561451ba8c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:09.521 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.507+0000 7f66014ddb80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561451ba9140 0x561451ba2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:09.521 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.507+0000 7f66014ddb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561451ba2fe0 0x561451ba3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:09.522 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.507+0000 7f66014ddb80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x561451a8cda0 con 0x561451ba9140 2023-03-09T14:47:09.522 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.507+0000 7f66014ddb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x561451afb780 con 0x561451ba8830 2023-03-09T14:47:09.522 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.507+0000 7f66014ddb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x561451af4cb0 con 0x561451ba2fe0 2023-03-09T14:47:09.522 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.507+0000 7f65ee7fc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561451ba9140 0x561451ba2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:09.522 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.507+0000 7f65ee7fc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561451ba9140 0x561451ba2aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39624/0 (socket says 172.21.15.38:39624) 2023-03-09T14:47:09.522 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.507+0000 7f65ee7fc700 1 -- 172.21.15.38:0/4260547069 learned_addr learned my addr 172.21.15.38:0/4260547069 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:47:09.523 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.507+0000 7f65eeffd700 1 --2- 172.21.15.38:0/4260547069 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561451ba8830 0x561451ba8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:09.523 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.507+0000 7f65ef7fe700 1 --2- 172.21.15.38:0/4260547069 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561451ba2fe0 0x561451ba3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:09.523 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.508+0000 7f65eeffd700 1 -- 172.21.15.38:0/4260547069 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561451ba2fe0 msgr2=0x561451ba3410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:47:09.523 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.508+0000 7f65eeffd700 1 --2- 172.21.15.38:0/4260547069 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561451ba2fe0 0x561451ba3410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.523 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.508+0000 7f65eeffd700 1 -- 172.21.15.38:0/4260547069 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561451ba9140 msgr2=0x561451ba2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:47:09.524 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.508+0000 7f65eeffd700 1 --2- 172.21.15.38:0/4260547069 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561451ba9140 0x561451ba2aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.524 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.508+0000 7f65eeffd700 1 -- 172.21.15.38:0/4260547069 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561451abb190 con 0x561451ba8830 2023-03-09T14:47:09.524 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.508+0000 7f65ee7fc700 1 --2- 172.21.15.38:0/4260547069 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561451ba9140 0x561451ba2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:47:09.524 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.508+0000 7f65eeffd700 1 --2- 172.21.15.38:0/4260547069 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561451ba8830 0x561451ba8c00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f65e0098fa0 tx=0x7f65e0022f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=44f6eb1149aaf340 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:47:09.524 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.508+0000 7f65d57fa700 1 -- 172.21.15.38:0/4260547069 <== mon.2 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65e0098de0 con 0x561451ba8830 2023-03-09T14:47:09.524 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.508+0000 7f65d57fa700 1 -- 172.21.15.38:0/4260547069 <== mon.2 v2:172.21.15.122:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f65e0098c00 con 0x561451ba8830 2023-03-09T14:47:09.524 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.508+0000 7f65d57fa700 1 -- 172.21.15.38:0/4260547069 <== mon.2 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65e00229e0 con 0x561451ba8830 2023-03-09T14:47:09.525 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.509+0000 7f66014ddb80 1 -- 172.21.15.38:0/4260547069 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561451ba8830 msgr2=0x561451ba8c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:47:09.525 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.509+0000 7f66014ddb80 1 --2- 172.21.15.38:0/4260547069 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561451ba8830 0x561451ba8c00 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f65e0098fa0 tx=0x7f65e0022f90 comp rx=0 tx=0).stop 2023-03-09T14:47:09.525 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.509+0000 7f66014ddb80 1 -- 172.21.15.38:0/4260547069 shutdown_connections 2023-03-09T14:47:09.525 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.509+0000 7f66014ddb80 1 --2- 172.21.15.38:0/4260547069 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561451ba2fe0 0x561451ba3410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.525 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.509+0000 7f66014ddb80 1 --2- 172.21.15.38:0/4260547069 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561451ba8830 0x561451ba8c00 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.525 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.509+0000 7f66014ddb80 1 --2- 172.21.15.38:0/4260547069 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561451ba9140 0x561451ba2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.526 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.509+0000 7f66014ddb80 1 -- 172.21.15.38:0/4260547069 >> 172.21.15.38:0/4260547069 conn(0x561451b9da20 msgr2=0x561451b9ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:47:09.526 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.509+0000 7f66014ddb80 1 -- 172.21.15.38:0/4260547069 shutdown_connections 2023-03-09T14:47:09.526 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.509+0000 7f66014ddb80 1 -- 172.21.15.38:0/4260547069 wait complete. 2023-03-09T14:47:09.526 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.509+0000 7f66014ddb80 1 -- start start 2023-03-09T14:47:09.526 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.509+0000 7f66014ddb80 1 Processor -- start 2023-03-09T14:47:09.526 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.510+0000 7f66014ddb80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561451ba2fe0 0x561451baf0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:09.526 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.510+0000 7f66014ddb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561451ba8830 0x561451bac330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:09.527 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.510+0000 7f66014ddb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561451ba9140 0x561451bac960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:09.527 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.510+0000 7f66014ddb80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x561451978920 con 0x561451ba2fe0 2023-03-09T14:47:09.527 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.510+0000 7f66014ddb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x561451bad0d0 con 0x561451ba8830 2023-03-09T14:47:09.527 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.510+0000 7f66014ddb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x561451bb8930 con 0x561451ba9140 2023-03-09T14:47:09.527 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.510+0000 7f66014ddb80 10 client.0 _ll_register_callbacks cb 0x561451a9a520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:47:09.528 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.510+0000 7f65ee7fc700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561451ba8830 0x561451bac330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:09.528 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.510+0000 7f65ee7fc700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561451ba8830 0x561451bac330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.38:48818/0 (socket says 172.21.15.38:48818) 2023-03-09T14:47:09.528 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.510+0000 7f65ee7fc700 1 -- 172.21.15.38:0/3814850461 learned_addr learned my addr 172.21.15.38:0/3814850461 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:47:09.528 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.510+0000 7f65eeffd700 1 --2- 172.21.15.38:0/3814850461 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561451ba2fe0 0x561451baf0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:09.528 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.510+0000 7f65ef7fe700 1 --2- 172.21.15.38:0/3814850461 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561451ba9140 0x561451bac960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:09.529 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.510+0000 7f65ee7fc700 1 -- 172.21.15.38:0/3814850461 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561451ba9140 msgr2=0x561451bac960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:47:09.529 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.510+0000 7f65ee7fc700 1 --2- 172.21.15.38:0/3814850461 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561451ba9140 0x561451bac960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.529 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.510+0000 7f65ee7fc700 1 -- 172.21.15.38:0/3814850461 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561451ba2fe0 msgr2=0x561451baf0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:47:09.529 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.510+0000 7f65ee7fc700 1 --2- 172.21.15.38:0/3814850461 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561451ba2fe0 0x561451baf0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.529 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.510+0000 7f65ee7fc700 1 -- 172.21.15.38:0/3814850461 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65e400ea80 con 0x561451ba8830 2023-03-09T14:47:09.529 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.510+0000 7f65eeffd700 1 --2- 172.21.15.38:0/3814850461 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561451ba2fe0 0x561451baf0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:47:09.530 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.510+0000 7f65ef7fe700 1 --2- 172.21.15.38:0/3814850461 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561451ba9140 0x561451bac960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-09T14:47:09.530 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.510+0000 7f65ee7fc700 1 --2- 172.21.15.38:0/3814850461 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561451ba8830 0x561451bac330 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f65e4015960 tx=0x7f65e40047e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d7de8c77485cd42 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:47:09.530 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.510+0000 7f65d2ff5700 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2023-03-09T14:47:09.530 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.511+0000 7f65d2ff5700 1 -- 172.21.15.38:0/3814850461 <== mon.2 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65e4013660 con 0x561451ba8830 2023-03-09T14:47:09.532 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.511+0000 7f65d2ff5700 1 -- 172.21.15.38:0/3814850461 <== mon.2 v2:172.21.15.122:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9278+0+0 (secure 0 0 0) 0x7f65e40137c0 con 0x561451ba8830 2023-03-09T14:47:09.532 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.511+0000 7f66014ddb80 1 -- 172.21.15.38:0/3814850461 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561451ac2190 con 0x561451ba8830 2023-03-09T14:47:09.532 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.511+0000 7f65d2ff5700 1 -- 172.21.15.38:0/3814850461 <== mon.2 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f65e4013980 con 0x561451ba8830 2023-03-09T14:47:09.532 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.511+0000 7f66014ddb80 10 client.25466 fetch_fsmap learned FSMap version 25 2023-03-09T14:47:09.532 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.511+0000 7f66014ddb80 1 -- 172.21.15.38:0/3814850461 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x7f65f00ffa80 con 0x561451ba8830 2023-03-09T14:47:09.533 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.511+0000 7f65d2ff5700 1 -- 172.21.15.38:0/3814850461 <== mon.2 v2:172.21.15.122:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x561451ac2190 con 0x561451ba8830 2023-03-09T14:47:09.533 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.511+0000 7f65d2ff5700 1 -- 172.21.15.38:0/3814850461 <== mon.2 v2:172.21.15.122:3300/0 5 ==== fsmap(e 25) v1 ==== 7478+0+0 (secure 0 0 0) 0x7f65e4021ca0 con 0x561451ba8830 2023-03-09T14:47:09.533 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.511+0000 7f66014ddb80 10 client.25466 fetch_fsmap finished waiting for FSMap version 25 2023-03-09T14:47:09.533 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.511+0000 7f66014ddb80 10 client.25466 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2023-03-09T14:47:09.533 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.511+0000 7f66014ddb80 20 client.25466 populate_metadata read hostname 'smithi038' 2023-03-09T14:47:09.533 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.511+0000 7f66014ddb80 1 --2- 172.21.15.38:0/3814850461 >> [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] conn(0x561451bbc920 0x561451bbed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:09.534 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.511+0000 7f66014ddb80 4 client.25466 mds_command: new command op to 24494 tid=0[{"prefix": "get_command_descriptions"}] 2023-03-09T14:47:09.534 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.511+0000 7f66014ddb80 1 -- 172.21.15.38:0/3814850461 --> [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561451978f60 con 0x561451bbc920 2023-03-09T14:47:09.534 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.512+0000 7f65eeffd700 1 --2- 172.21.15.38:0/3814850461 >> [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] conn(0x561451bbc920 0x561451bbed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:09.534 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.512+0000 7f65eeffd700 1 --2- 172.21.15.38:0/3814850461 >> [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] conn(0x561451bbc920 0x561451bbed30 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=79ef2d8aa3731080 server_cookie=efae79149068456a in_seq=0 out_seq=0 2023-03-09T14:47:09.535 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.512+0000 7f65d2ff5700 10 client.25466 ms_handle_connect on v2:172.21.15.122:6836/2210214596 2023-03-09T14:47:09.535 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.513+0000 7f65d2ff5700 1 -- 172.21.15.38:0/3814850461 <== mds.1 v2:172.21.15.122:6836/2210214596 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18278 (crc 0 0 0) 0x561451978f60 con 0x561451bbc920 2023-03-09T14:47:09.535 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.513+0000 7f65d2ff5700 10 client.25466 handle_command_reply: tid=0 2023-03-09T14:47:09.535 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.513+0000 7f66014ddb80 1 client.25466 shutdown 2023-03-09T14:47:09.536 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.513+0000 7f66014ddb80 1 -- 172.21.15.38:0/3814850461 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561451ba8830 msgr2=0x561451bac330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:47:09.536 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.513+0000 7f66014ddb80 1 --2- 172.21.15.38:0/3814850461 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561451ba8830 0x561451bac330 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f65e4015960 tx=0x7f65e40047e0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.536 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.513+0000 7f66014ddb80 1 -- 172.21.15.38:0/3814850461 shutdown_connections 2023-03-09T14:47:09.536 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.513+0000 7f66014ddb80 1 --2- 172.21.15.38:0/3814850461 >> [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] conn(0x561451bbc920 0x561451bbed30 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.536 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.513+0000 7f66014ddb80 1 --2- 172.21.15.38:0/3814850461 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561451ba2fe0 0x561451baf0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.536 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.513+0000 7f66014ddb80 1 --2- 172.21.15.38:0/3814850461 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561451ba9140 0x561451bac960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.536 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.513+0000 7f66014ddb80 1 --2- 172.21.15.38:0/3814850461 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561451ba8830 0x561451bac330 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.537 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.513+0000 7f65d2ff5700 0 client.25466 ms_handle_reset on v2:172.21.15.122:6836/2210214596 2023-03-09T14:47:09.537 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.513+0000 7f66014ddb80 1 -- 172.21.15.38:0/3814850461 >> 172.21.15.38:0/3814850461 conn(0x561451b9da20 msgr2=0x561451b9cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:47:09.537 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.513+0000 7f66014ddb80 1 -- 172.21.15.38:0/3814850461 shutdown_connections 2023-03-09T14:47:09.537 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.513+0000 7f66014ddb80 1 -- 172.21.15.38:0/3814850461 wait complete. 2023-03-09T14:47:09.537 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.513+0000 7f66014ddb80 20 client.25466 trim_cache size 0 max 16384 2023-03-09T14:47:09.541 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.531+0000 7f66014ddb80 1 Processor -- start 2023-03-09T14:47:09.541 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.531+0000 7f66014ddb80 1 -- start start 2023-03-09T14:47:09.541 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.531+0000 7f66014ddb80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561451ba2fe0 0x561451bb8090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:09.542 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.531+0000 7f66014ddb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561451ba8830 0x561451bb8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:09.542 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.531+0000 7f66014ddb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561451ba9140 0x561451baec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:09.542 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.531+0000 7f66014ddb80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x561451b99e00 con 0x561451ba2fe0 2023-03-09T14:47:09.542 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.531+0000 7f66014ddb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x561451b9c680 con 0x561451ba9140 2023-03-09T14:47:09.543 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.531+0000 7f66014ddb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x561451ba6e00 con 0x561451ba8830 2023-03-09T14:47:09.543 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.531+0000 7f65eeffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561451ba2fe0 0x561451bb8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:09.543 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.531+0000 7f65eeffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561451ba2fe0 0x561451bb8090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39636/0 (socket says 172.21.15.38:39636) 2023-03-09T14:47:09.543 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.531+0000 7f65eeffd700 1 -- 172.21.15.38:0/4263554557 learned_addr learned my addr 172.21.15.38:0/4263554557 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:47:09.543 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.531+0000 7f65ee7fc700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561451ba8830 0x561451bb8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:09.543 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.531+0000 7f65ef7fe700 1 --2- 172.21.15.38:0/4263554557 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561451ba9140 0x561451baec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:09.543 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.532+0000 7f65eeffd700 1 -- 172.21.15.38:0/4263554557 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561451ba8830 msgr2=0x561451bb8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:47:09.544 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.532+0000 7f65eeffd700 1 --2- 172.21.15.38:0/4263554557 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561451ba8830 0x561451bb8cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.544 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.532+0000 7f65eeffd700 1 -- 172.21.15.38:0/4263554557 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561451ba9140 msgr2=0x561451baec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:47:09.544 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.532+0000 7f65eeffd700 1 --2- 172.21.15.38:0/4263554557 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561451ba9140 0x561451baec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.544 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.532+0000 7f65eeffd700 1 -- 172.21.15.38:0/4263554557 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56145190bc60 con 0x561451ba2fe0 2023-03-09T14:47:09.544 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.532+0000 7f65eeffd700 1 --2- 172.21.15.38:0/4263554557 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561451ba2fe0 0x561451bb8090 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f65e0022cd0 tx=0x7f65e0009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0c5846e6be03a14 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:47:09.544 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.532+0000 7f65d5ffb700 1 -- 172.21.15.38:0/4263554557 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65e0099320 con 0x561451ba2fe0 2023-03-09T14:47:09.545 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.532+0000 7f65d5ffb700 1 -- 172.21.15.38:0/4263554557 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f65e0097640 con 0x561451ba2fe0 2023-03-09T14:47:09.545 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.532+0000 7f65d5ffb700 1 -- 172.21.15.38:0/4263554557 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65e00cb380 con 0x561451ba2fe0 2023-03-09T14:47:09.545 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.533+0000 7f66014ddb80 1 -- 172.21.15.38:0/4263554557 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561451ba2fe0 msgr2=0x561451bb8090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:47:09.545 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.533+0000 7f66014ddb80 1 --2- 172.21.15.38:0/4263554557 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561451ba2fe0 0x561451bb8090 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f65e0022cd0 tx=0x7f65e0009ca0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.545 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.533+0000 7f66014ddb80 1 -- 172.21.15.38:0/4263554557 shutdown_connections 2023-03-09T14:47:09.546 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.533+0000 7f66014ddb80 1 --2- 172.21.15.38:0/4263554557 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561451ba2fe0 0x561451bb8090 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.546 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.533+0000 7f66014ddb80 1 --2- 172.21.15.38:0/4263554557 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561451ba9140 0x561451baec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.546 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.533+0000 7f66014ddb80 1 --2- 172.21.15.38:0/4263554557 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561451ba8830 0x561451bb8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.546 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.533+0000 7f66014ddb80 1 -- 172.21.15.38:0/4263554557 >> 172.21.15.38:0/4263554557 conn(0x561451b9da20 msgr2=0x561451b9bf70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:47:09.547 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.533+0000 7f66014ddb80 1 -- 172.21.15.38:0/4263554557 shutdown_connections 2023-03-09T14:47:09.547 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.533+0000 7f66014ddb80 1 -- 172.21.15.38:0/4263554557 wait complete. 2023-03-09T14:47:09.547 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.533+0000 7f66014ddb80 1 -- start start 2023-03-09T14:47:09.548 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.533+0000 7f66014ddb80 1 Processor -- start 2023-03-09T14:47:09.548 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.534+0000 7f66014ddb80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561451ba2fe0 0x561451ba53f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:09.548 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.534+0000 7f66014ddb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561451ba8830 0x561451ba5930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:09.548 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.534+0000 7f66014ddb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561451ba9140 0x561451ba4290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:09.549 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.534+0000 7f66014ddb80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x561451ba5f30 con 0x561451ba2fe0 2023-03-09T14:47:09.549 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.534+0000 7f66014ddb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x561451ba47d0 con 0x561451ba8830 2023-03-09T14:47:09.549 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.534+0000 7f66014ddb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x561451ba4990 con 0x561451ba9140 2023-03-09T14:47:09.550 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.534+0000 7f66014ddb80 10 client.0 _ll_register_callbacks cb 0x561451a9a520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:47:09.550 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.534+0000 7f65eeffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561451ba2fe0 0x561451ba53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:09.550 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.534+0000 7f65eeffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561451ba2fe0 0x561451ba53f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39642/0 (socket says 172.21.15.38:39642) 2023-03-09T14:47:09.550 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.534+0000 7f65eeffd700 1 -- 172.21.15.38:0/3212094636 learned_addr learned my addr 172.21.15.38:0/3212094636 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:47:09.551 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.534+0000 7f65ee7fc700 1 --2- 172.21.15.38:0/3212094636 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561451ba8830 0x561451ba5930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:09.551 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.534+0000 7f65ef7fe700 1 --2- 172.21.15.38:0/3212094636 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561451ba9140 0x561451ba4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:09.551 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.534+0000 7f65ee7fc700 1 -- 172.21.15.38:0/3212094636 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561451ba9140 msgr2=0x561451ba4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:47:09.552 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.534+0000 7f65ee7fc700 1 --2- 172.21.15.38:0/3212094636 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561451ba9140 0x561451ba4290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.552 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.534+0000 7f65ee7fc700 1 -- 172.21.15.38:0/3212094636 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561451ba2fe0 msgr2=0x561451ba53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:47:09.552 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.534+0000 7f65ee7fc700 1 --2- 172.21.15.38:0/3212094636 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561451ba2fe0 0x561451ba53f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.552 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.534+0000 7f65ee7fc700 1 -- 172.21.15.38:0/3212094636 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561451abb190 con 0x561451ba8830 2023-03-09T14:47:09.553 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.535+0000 7f65ee7fc700 1 --2- 172.21.15.38:0/3212094636 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561451ba8830 0x561451ba5930 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f65e4009fd0 tx=0x7f65e40042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b5e01b7a40e1553 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:47:09.553 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.535+0000 7f65d2ff5700 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2023-03-09T14:47:09.553 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.535+0000 7f65d2ff5700 1 -- 172.21.15.38:0/3212094636 <== mon.2 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65e4020a60 con 0x561451ba8830 2023-03-09T14:47:09.553 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.535+0000 7f65d2ff5700 1 -- 172.21.15.38:0/3212094636 <== mon.2 v2:172.21.15.122:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9278+0+0 (secure 0 0 0) 0x7f65e4020620 con 0x561451ba8830 2023-03-09T14:47:09.554 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.535+0000 7f66014ddb80 1 -- 172.21.15.38:0/3212094636 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561451ab21b0 con 0x561451ba8830 2023-03-09T14:47:09.554 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.535+0000 7f65d2ff5700 1 -- 172.21.15.38:0/3212094636 <== mon.2 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f65e4020bc0 con 0x561451ba8830 2023-03-09T14:47:09.554 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.535+0000 7f66014ddb80 10 client.25472 fetch_fsmap learned FSMap version 25 2023-03-09T14:47:09.554 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.535+0000 7f66014ddb80 1 -- 172.21.15.38:0/3212094636 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x561451949250 con 0x561451ba8830 2023-03-09T14:47:09.555 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.535+0000 7f65d2ff5700 1 -- 172.21.15.38:0/3212094636 <== mon.2 v2:172.21.15.122:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x561451ab21b0 con 0x561451ba8830 2023-03-09T14:47:09.555 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.535+0000 7f65d2ff5700 1 -- 172.21.15.38:0/3212094636 <== mon.2 v2:172.21.15.122:3300/0 5 ==== fsmap(e 25) v1 ==== 7478+0+0 (secure 0 0 0) 0x7f65e4013660 con 0x561451ba8830 2023-03-09T14:47:09.555 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.536+0000 7f66014ddb80 10 client.25472 fetch_fsmap finished waiting for FSMap version 25 2023-03-09T14:47:09.555 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.536+0000 7f66014ddb80 10 client.25472 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2023-03-09T14:47:09.556 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.536+0000 7f66014ddb80 20 client.25472 populate_metadata read hostname 'smithi038' 2023-03-09T14:47:09.556 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.536+0000 7f66014ddb80 1 --2- 172.21.15.38:0/3212094636 >> [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] conn(0x561451c11550 0x561451c13960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:09.556 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.536+0000 7f66014ddb80 4 client.25472 mds_command: new command op to 24494 tid=0[{"prefix": "damage ls"}] 2023-03-09T14:47:09.556 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.536+0000 7f66014ddb80 1 -- 172.21.15.38:0/3212094636 --> [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x561451a9c1c0 con 0x561451c11550 2023-03-09T14:47:09.556 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.536+0000 7f65eeffd700 1 --2- 172.21.15.38:0/3212094636 >> [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] conn(0x561451c11550 0x561451c13960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:09.557 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.536+0000 7f65d2ff5700 10 client.25472 ms_handle_connect on v2:172.21.15.122:6836/2210214596 2023-03-09T14:47:09.557 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.536+0000 7f65eeffd700 1 --2- 172.21.15.38:0/3212094636 >> [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] conn(0x561451c11550 0x561451c13960 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f7853846b852698e server_cookie=aaf2e8f3b76ead6d in_seq=0 out_seq=0 2023-03-09T14:47:09.557 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.536+0000 7f65d2ff5700 1 -- 172.21.15.38:0/3212094636 <== mds.1 v2:172.21.15.122:6836/2210214596 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x561451a9c1c0 con 0x561451c11550 2023-03-09T14:47:09.557 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.536+0000 7f65d2ff5700 10 client.25472 handle_command_reply: tid=0 2023-03-09T14:47:09.558 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.536+0000 7f66014ddb80 1 client.25472 shutdown 2023-03-09T14:47:09.558 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.537+0000 7f66014ddb80 1 -- 172.21.15.38:0/3212094636 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561451ba8830 msgr2=0x561451ba5930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:47:09.558 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.537+0000 7f66014ddb80 1 --2- 172.21.15.38:0/3212094636 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561451ba8830 0x561451ba5930 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f65e4009fd0 tx=0x7f65e40042f0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.558 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.537+0000 7f66014ddb80 1 -- 172.21.15.38:0/3212094636 shutdown_connections 2023-03-09T14:47:09.559 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.537+0000 7f66014ddb80 1 --2- 172.21.15.38:0/3212094636 >> [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] conn(0x561451c11550 0x561451c13960 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.559 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.537+0000 7f66014ddb80 1 --2- 172.21.15.38:0/3212094636 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x561451ba2fe0 0x561451ba53f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.559 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.537+0000 7f66014ddb80 1 --2- 172.21.15.38:0/3212094636 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x561451ba9140 0x561451ba4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.559 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.537+0000 7f66014ddb80 1 --2- 172.21.15.38:0/3212094636 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x561451ba8830 0x561451ba5930 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.560 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.537+0000 7f66014ddb80 1 -- 172.21.15.38:0/3212094636 >> 172.21.15.38:0/3212094636 conn(0x561451b9da20 msgr2=0x561451bb8090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:47:09.560 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.537+0000 7f65d2ff5700 0 client.25472 ms_handle_reset on v2:172.21.15.122:6836/2210214596 2023-03-09T14:47:09.560 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.537+0000 7f66014ddb80 1 -- 172.21.15.38:0/3212094636 shutdown_connections 2023-03-09T14:47:09.561 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.537+0000 7f66014ddb80 1 -- 172.21.15.38:0/3212094636 wait complete. 2023-03-09T14:47:09.561 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.537+0000 7f66014ddb80 20 client.25472 trim_cache size 0 max 16384 2023-03-09T14:47:09.562 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2023-03-09T14:47:09.564 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.538+0000 7f65f5354700 1 -- 172.21.15.38:0/2631757914 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f65d809f530 msgr2=0x7f65d80a19e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:47:09.564 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.538+0000 7f65f5354700 1 --2- 172.21.15.38:0/2631757914 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f65d809f530 0x7f65d80a19e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f65f0199c40 tx=0x7f65e4009190 comp rx=0 tx=0).stop 2023-03-09T14:47:09.564 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.538+0000 7f65f5354700 1 -- 172.21.15.38:0/2631757914 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f65f0100410 msgr2=0x7f65f019d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:47:09.565 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.538+0000 7f65f5354700 1 --2- 172.21.15.38:0/2631757914 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f65f0100410 0x7f65f019d240 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f65e0005740 tx=0x7f65e0005770 comp rx=0 tx=0).stop 2023-03-09T14:47:09.565 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.538+0000 7f65f5354700 1 -- 172.21.15.38:0/2631757914 shutdown_connections 2023-03-09T14:47:09.565 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.538+0000 7f65f5354700 1 --2- 172.21.15.38:0/2631757914 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f65d809f530 0x7f65d80a19e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.565 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.538+0000 7f65f5354700 1 --2- 172.21.15.38:0/2631757914 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f65f0100410 0x7f65f019d240 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.566 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.538+0000 7f65f5354700 1 --2- 172.21.15.38:0/2631757914 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f65f01099c0 0x7f65f0108190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.566 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.538+0000 7f65f5354700 1 --2- 172.21.15.38:0/2631757914 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f65f0100d20 0x7f65f019d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.566 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.538+0000 7f65f5354700 1 -- 172.21.15.38:0/2631757914 >> 172.21.15.38:0/2631757914 conn(0x7f65f00748f0 msgr2=0x7f65f0109d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:47:09.567 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.538+0000 7f65f5354700 1 -- 172.21.15.38:0/2631757914 shutdown_connections 2023-03-09T14:47:09.567 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.538+0000 7f65f5354700 1 -- 172.21.15.38:0/2631757914 wait complete. 2023-03-09T14:47:09.568 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2023-03-09T14:47:09.724 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.720+0000 7fcdee6c0700 1 -- 172.21.15.38:0/2859733902 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcde8104040 msgr2=0x7fcde810bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:47:09.725 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.720+0000 7fcdee6c0700 1 --2- 172.21.15.38:0/2859733902 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcde8104040 0x7fcde810bc10 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fcde000d210 tx=0x7fcde000d520 comp rx=0 tx=0).stop 2023-03-09T14:47:09.725 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.720+0000 7fcdee6c0700 1 -- 172.21.15.38:0/2859733902 shutdown_connections 2023-03-09T14:47:09.726 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.720+0000 7fcdee6c0700 1 --2- 172.21.15.38:0/2859733902 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcde80ff000 0x7fcde80ff3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.726 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.720+0000 7fcdee6c0700 1 --2- 172.21.15.38:0/2859733902 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcde8104040 0x7fcde810bc10 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.726 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.720+0000 7fcdee6c0700 1 --2- 172.21.15.38:0/2859733902 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcde80ff9a0 0x7fcde8103960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.727 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.720+0000 7fcdee6c0700 1 -- 172.21.15.38:0/2859733902 >> 172.21.15.38:0/2859733902 conn(0x7fcde8075c60 msgr2=0x7fcde8076060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:47:09.727 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.720+0000 7fcdee6c0700 1 -- 172.21.15.38:0/2859733902 shutdown_connections 2023-03-09T14:47:09.727 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.720+0000 7fcdee6c0700 1 -- 172.21.15.38:0/2859733902 wait complete. 2023-03-09T14:47:09.727 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.721+0000 7fcdee6c0700 1 Processor -- start 2023-03-09T14:47:09.728 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.721+0000 7fcdee6c0700 1 -- start start 2023-03-09T14:47:09.740 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.722+0000 7fcdee6c0700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcde80ff000 0x7fcde8193c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:09.740 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.722+0000 7fcdee6c0700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcde80ff9a0 0x7fcde8194150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:09.740 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.722+0000 7fcdee6c0700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcde8104040 0x7fcde8194690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:09.741 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.722+0000 7fcdee6c0700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcde81a88b0 con 0x7fcde80ff000 2023-03-09T14:47:09.741 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.722+0000 7fcdee6c0700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcde81a8a20 con 0x7fcde80ff9a0 2023-03-09T14:47:09.741 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.722+0000 7fcdee6c0700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcde81a8be0 con 0x7fcde8104040 2023-03-09T14:47:09.742 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.722+0000 7fcde7fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcde80ff000 0x7fcde8193c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:09.742 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.722+0000 7fcde77fe700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcde80ff9a0 0x7fcde8194150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:09.742 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.722+0000 7fcde77fe700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcde80ff9a0 0x7fcde8194150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.38:48846/0 (socket says 172.21.15.38:48846) 2023-03-09T14:47:09.742 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.722+0000 7fcde77fe700 1 -- 172.21.15.38:0/3665682464 learned_addr learned my addr 172.21.15.38:0/3665682464 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:47:09.743 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.722+0000 7fcdecc5d700 1 --2- 172.21.15.38:0/3665682464 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcde8104040 0x7fcde8194690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:09.743 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.722+0000 7fcde77fe700 1 -- 172.21.15.38:0/3665682464 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcde8104040 msgr2=0x7fcde8194690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:47:09.743 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.722+0000 7fcde77fe700 1 --2- 172.21.15.38:0/3665682464 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcde8104040 0x7fcde8194690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.744 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.722+0000 7fcde77fe700 1 -- 172.21.15.38:0/3665682464 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcde80ff000 msgr2=0x7fcde8193c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:47:09.744 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.722+0000 7fcde77fe700 1 --2- 172.21.15.38:0/3665682464 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcde80ff000 0x7fcde8193c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.744 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.722+0000 7fcde77fe700 1 -- 172.21.15.38:0/3665682464 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcde000be50 con 0x7fcde80ff9a0 2023-03-09T14:47:09.744 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.722+0000 7fcde7fff700 1 --2- 172.21.15.38:0/3665682464 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcde80ff000 0x7fcde8193c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:47:09.744 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.722+0000 7fcdecc5d700 1 --2- 172.21.15.38:0/3665682464 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcde8104040 0x7fcde8194690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:47:09.745 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.723+0000 7fcde77fe700 1 --2- 172.21.15.38:0/3665682464 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcde80ff9a0 0x7fcde8194150 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fcddc00d950 tx=0x7fcddc00dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:47:09.745 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.723+0000 7fcde57fa700 1 -- 172.21.15.38:0/3665682464 <== mon.2 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcddc009930 con 0x7fcde80ff9a0 2023-03-09T14:47:09.745 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.723+0000 7fcdee6c0700 1 -- 172.21.15.38:0/3665682464 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcde810ddc0 con 0x7fcde80ff9a0 2023-03-09T14:47:09.745 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.723+0000 7fcde57fa700 1 -- 172.21.15.38:0/3665682464 <== mon.2 v2:172.21.15.122:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcddc00b640 con 0x7fcde80ff9a0 2023-03-09T14:47:09.745 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.723+0000 7fcde57fa700 1 -- 172.21.15.38:0/3665682464 <== mon.2 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcddc00f3c0 con 0x7fcde80ff9a0 2023-03-09T14:47:09.745 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.723+0000 7fcdee6c0700 1 -- 172.21.15.38:0/3665682464 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcde810dc30 con 0x7fcde80ff9a0 2023-03-09T14:47:09.746 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.726+0000 7fcde57fa700 1 -- 172.21.15.38:0/3665682464 <== mon.2 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 24) v1 ==== 133375+0+0 (secure 0 0 0) 0x7fcddc023070 con 0x7fcde80ff9a0 2023-03-09T14:47:09.746 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.727+0000 7fcde57fa700 1 --2- 172.21.15.38:0/3665682464 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fcdd009f530 0x7fcdd00a19e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:09.746 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.727+0000 7fcde57fa700 1 -- 172.21.15.38:0/3665682464 <== mon.2 v2:172.21.15.122:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9278+0+0 (secure 0 0 0) 0x7fcddc0c9d30 con 0x7fcde80ff9a0 2023-03-09T14:47:09.746 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.727+0000 7fcde7fff700 1 --2- 172.21.15.38:0/3665682464 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fcdd009f530 0x7fcdd00a19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:09.746 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.727+0000 7fcde7fff700 1 --2- 172.21.15.38:0/3665682464 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fcdd009f530 0x7fcdd00a19e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fcdd8005fd0 tx=0x7fcdd8009310 comp rx=0 tx=0).ready entity=mgr.15558 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:47:09.746 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.728+0000 7fcdfa849b80 1 Processor -- start 2023-03-09T14:47:09.747 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.728+0000 7fcdfa849b80 1 -- start start 2023-03-09T14:47:09.747 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.728+0000 7fcdfa849b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558738eaf830 0x558738eafc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:09.747 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.728+0000 7fcdfa849b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558738eb0140 0x558738ea9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:09.747 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.728+0000 7fcdfa849b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558738ea9fe0 0x558738eaa410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:09.748 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.728+0000 7fcdfa849b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x558738d93da0 con 0x558738eaf830 2023-03-09T14:47:09.748 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.728+0000 7fcdfa849b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x558738e02780 con 0x558738eb0140 2023-03-09T14:47:09.748 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.728+0000 7fcdfa849b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x558738dfbcb0 con 0x558738ea9fe0 2023-03-09T14:47:09.749 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.728+0000 7fcde7fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558738eaf830 0x558738eafc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:09.749 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.728+0000 7fcde77fe700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558738eb0140 0x558738ea9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:09.749 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.728+0000 7fcde7fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558738eaf830 0x558738eafc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39662/0 (socket says 172.21.15.38:39662) 2023-03-09T14:47:09.749 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.728+0000 7fcde7fff700 1 -- 172.21.15.38:0/621138723 learned_addr learned my addr 172.21.15.38:0/621138723 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:47:09.749 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.728+0000 7fcdecc5d700 1 --2- 172.21.15.38:0/621138723 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558738ea9fe0 0x558738eaa410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:09.749 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.729+0000 7fcde77fe700 1 -- 172.21.15.38:0/621138723 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558738ea9fe0 msgr2=0x558738eaa410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:47:09.750 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.729+0000 7fcde77fe700 1 --2- 172.21.15.38:0/621138723 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558738ea9fe0 0x558738eaa410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.750 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.729+0000 7fcde77fe700 1 -- 172.21.15.38:0/621138723 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558738eaf830 msgr2=0x558738eafc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:47:09.750 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.729+0000 7fcde77fe700 1 --2- 172.21.15.38:0/621138723 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558738eaf830 0x558738eafc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.750 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.729+0000 7fcde77fe700 1 -- 172.21.15.38:0/621138723 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558738dc2190 con 0x558738eb0140 2023-03-09T14:47:09.750 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.729+0000 7fcdecc5d700 1 --2- 172.21.15.38:0/621138723 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558738ea9fe0 0x558738eaa410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:47:09.750 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.729+0000 7fcde77fe700 1 --2- 172.21.15.38:0/621138723 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558738eb0140 0x558738ea9aa0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fcddc093d80 tx=0x7fcddc097720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e32640863c29ac4 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:47:09.750 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.729+0000 7fcdce7fc700 1 -- 172.21.15.38:0/621138723 <== mon.2 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcddc097880 con 0x558738eb0140 2023-03-09T14:47:09.750 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.729+0000 7fcdce7fc700 1 -- 172.21.15.38:0/621138723 <== mon.2 v2:172.21.15.122:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcddc09fa70 con 0x558738eb0140 2023-03-09T14:47:09.751 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.729+0000 7fcdce7fc700 1 -- 172.21.15.38:0/621138723 <== mon.2 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcddc09f840 con 0x558738eb0140 2023-03-09T14:47:09.751 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.730+0000 7fcdfa849b80 1 -- 172.21.15.38:0/621138723 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558738eb0140 msgr2=0x558738ea9aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:47:09.751 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.730+0000 7fcdfa849b80 1 --2- 172.21.15.38:0/621138723 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558738eb0140 0x558738ea9aa0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fcddc093d80 tx=0x7fcddc097720 comp rx=0 tx=0).stop 2023-03-09T14:47:09.751 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.730+0000 7fcdfa849b80 1 -- 172.21.15.38:0/621138723 shutdown_connections 2023-03-09T14:47:09.751 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.730+0000 7fcdfa849b80 1 --2- 172.21.15.38:0/621138723 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558738eaf830 0x558738eafc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.752 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.730+0000 7fcdfa849b80 1 --2- 172.21.15.38:0/621138723 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558738ea9fe0 0x558738eaa410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.752 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.730+0000 7fcdfa849b80 1 --2- 172.21.15.38:0/621138723 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558738eb0140 0x558738ea9aa0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.752 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.730+0000 7fcdfa849b80 1 -- 172.21.15.38:0/621138723 >> 172.21.15.38:0/621138723 conn(0x558738ea4a20 msgr2=0x558738ea4df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:47:09.752 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.730+0000 7fcdfa849b80 1 -- 172.21.15.38:0/621138723 shutdown_connections 2023-03-09T14:47:09.752 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.730+0000 7fcdfa849b80 1 -- 172.21.15.38:0/621138723 wait complete. 2023-03-09T14:47:09.752 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.731+0000 7fcdfa849b80 1 -- start start 2023-03-09T14:47:09.752 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.731+0000 7fcdfa849b80 1 Processor -- start 2023-03-09T14:47:09.753 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.731+0000 7fcdfa849b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558738ea9fe0 0x558738eb6140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:09.753 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.731+0000 7fcdfa849b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558738eaf830 0x558738eb33c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:09.753 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.731+0000 7fcdfa849b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558738eb0140 0x558738eb3b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:09.753 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.731+0000 7fcdfa849b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x558738c7f920 con 0x558738eb0140 2023-03-09T14:47:09.753 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.731+0000 7fcdfa849b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x558738eb4270 con 0x558738ea9fe0 2023-03-09T14:47:09.754 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.731+0000 7fcdfa849b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x558738ebfb20 con 0x558738eaf830 2023-03-09T14:47:09.754 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.731+0000 7fcdfa849b80 10 client.0 _ll_register_callbacks cb 0x558738da1520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:47:09.754 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.731+0000 7fcde77fe700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558738eaf830 0x558738eb33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:09.754 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.731+0000 7fcde7fff700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558738ea9fe0 0x558738eb6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:09.754 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.731+0000 7fcde77fe700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558738eaf830 0x558738eb33c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.38:47368/0 (socket says 172.21.15.38:47368) 2023-03-09T14:47:09.754 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.731+0000 7fcde77fe700 1 -- 172.21.15.38:0/3502522736 learned_addr learned my addr 172.21.15.38:0/3502522736 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:47:09.754 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.731+0000 7fcdecc5d700 1 --2- 172.21.15.38:0/3502522736 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558738eb0140 0x558738eb3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:09.755 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.732+0000 7fcde7fff700 1 -- 172.21.15.38:0/3502522736 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558738eaf830 msgr2=0x558738eb33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:47:09.755 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.732+0000 7fcde7fff700 1 --2- 172.21.15.38:0/3502522736 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558738eaf830 0x558738eb33c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.755 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.732+0000 7fcde7fff700 1 -- 172.21.15.38:0/3502522736 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558738eb0140 msgr2=0x558738eb3b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:47:09.755 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.732+0000 7fcde7fff700 1 --2- 172.21.15.38:0/3502522736 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558738eb0140 0x558738eb3b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.755 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.732+0000 7fcde7fff700 1 -- 172.21.15.38:0/3502522736 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcde810c450 con 0x558738ea9fe0 2023-03-09T14:47:09.755 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.732+0000 7fcdecc5d700 1 --2- 172.21.15.38:0/3502522736 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558738eb0140 0x558738eb3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:47:09.756 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.732+0000 7fcde7fff700 1 --2- 172.21.15.38:0/3502522736 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558738ea9fe0 0x558738eb6140 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fcdd8011c70 tx=0x7fcdd80047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa187a9e31d50756 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:47:09.756 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.732+0000 7fcdc77fe700 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2023-03-09T14:47:09.756 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.732+0000 7fcdc77fe700 1 -- 172.21.15.38:0/3502522736 <== mon.2 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd801fa90 con 0x558738ea9fe0 2023-03-09T14:47:09.756 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.732+0000 7fcdc77fe700 1 -- 172.21.15.38:0/3502522736 <== mon.2 v2:172.21.15.122:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9278+0+0 (secure 0 0 0) 0x7fcdd801fbf0 con 0x558738ea9fe0 2023-03-09T14:47:09.756 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.732+0000 7fcdfa849b80 1 -- 172.21.15.38:0/3502522736 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558738dc9190 con 0x558738ea9fe0 2023-03-09T14:47:09.756 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.732+0000 7fcdc77fe700 1 -- 172.21.15.38:0/3502522736 <== mon.2 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcdd801fdb0 con 0x558738ea9fe0 2023-03-09T14:47:09.756 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.732+0000 7fcdfa849b80 10 client.25484 fetch_fsmap learned FSMap version 25 2023-03-09T14:47:09.757 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.732+0000 7fcdfa849b80 1 -- 172.21.15.38:0/3502522736 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x7fcde810dc30 con 0x558738ea9fe0 2023-03-09T14:47:09.757 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.732+0000 7fcdc77fe700 1 -- 172.21.15.38:0/3502522736 <== mon.2 v2:172.21.15.122:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x558738dc9190 con 0x558738ea9fe0 2023-03-09T14:47:09.757 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.733+0000 7fcdc77fe700 1 -- 172.21.15.38:0/3502522736 <== mon.2 v2:172.21.15.122:3300/0 5 ==== fsmap(e 25) v1 ==== 7478+0+0 (secure 0 0 0) 0x7fcdd80273e0 con 0x558738ea9fe0 2023-03-09T14:47:09.757 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.733+0000 7fcdfa849b80 10 client.25484 fetch_fsmap finished waiting for FSMap version 25 2023-03-09T14:47:09.757 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.733+0000 7fcdfa849b80 10 client.25484 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T14:47:09.757 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.733+0000 7fcdfa849b80 20 client.25484 populate_metadata read hostname 'smithi038' 2023-03-09T14:47:09.758 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.733+0000 7fcdfa849b80 1 --2- 172.21.15.38:0/3502522736 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x558738ec3d20 0x558738ec6130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:09.758 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.733+0000 7fcdfa849b80 4 client.25484 mds_command: new command op to 24530 tid=0[{"prefix": "get_command_descriptions"}] 2023-03-09T14:47:09.758 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.733+0000 7fcdfa849b80 1 -- 172.21.15.38:0/3502522736 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558738c7ff60 con 0x558738ec3d20 2023-03-09T14:47:09.758 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.733+0000 7fcde77fe700 1 --2- 172.21.15.38:0/3502522736 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x558738ec3d20 0x558738ec6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:09.758 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.733+0000 7fcdc77fe700 10 client.25484 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:47:09.758 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.733+0000 7fcde77fe700 1 --2- 172.21.15.38:0/3502522736 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x558738ec3d20 0x558738ec6130 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4fed4f188a13bf2a server_cookie=f0b8c42f21d08ba4 in_seq=0 out_seq=0 2023-03-09T14:47:09.758 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.734+0000 7fcdc77fe700 1 -- 172.21.15.38:0/3502522736 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18278 (crc 0 0 0) 0x558738c7ff60 con 0x558738ec3d20 2023-03-09T14:47:09.759 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.734+0000 7fcdc77fe700 10 client.25484 handle_command_reply: tid=0 2023-03-09T14:47:09.759 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.734+0000 7fcdfa849b80 1 client.25484 shutdown 2023-03-09T14:47:09.759 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.734+0000 7fcdfa849b80 1 -- 172.21.15.38:0/3502522736 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558738ea9fe0 msgr2=0x558738eb6140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:47:09.759 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.734+0000 7fcdfa849b80 1 --2- 172.21.15.38:0/3502522736 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558738ea9fe0 0x558738eb6140 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fcdd8011c70 tx=0x7fcdd80047c0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.759 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.734+0000 7fcdfa849b80 1 -- 172.21.15.38:0/3502522736 shutdown_connections 2023-03-09T14:47:09.759 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.734+0000 7fcdfa849b80 1 --2- 172.21.15.38:0/3502522736 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x558738ec3d20 0x558738ec6130 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.759 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.734+0000 7fcdfa849b80 1 --2- 172.21.15.38:0/3502522736 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558738eb0140 0x558738eb3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.759 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.734+0000 7fcdfa849b80 1 --2- 172.21.15.38:0/3502522736 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558738ea9fe0 0x558738eb6140 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.760 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.734+0000 7fcdc77fe700 0 client.25484 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:47:09.760 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.734+0000 7fcdfa849b80 1 --2- 172.21.15.38:0/3502522736 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558738eaf830 0x558738eb33c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.760 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.734+0000 7fcdfa849b80 1 -- 172.21.15.38:0/3502522736 >> 172.21.15.38:0/3502522736 conn(0x558738ea4a20 msgr2=0x558738ea3c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:47:09.760 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.734+0000 7fcdfa849b80 1 -- 172.21.15.38:0/3502522736 shutdown_connections 2023-03-09T14:47:09.760 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.735+0000 7fcdfa849b80 1 -- 172.21.15.38:0/3502522736 wait complete. 2023-03-09T14:47:09.760 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.735+0000 7fcdfa849b80 20 client.25484 trim_cache size 0 max 16384 2023-03-09T14:47:09.765 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.754+0000 7fcdfa849b80 1 Processor -- start 2023-03-09T14:47:09.765 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.754+0000 7fcdfa849b80 1 -- start start 2023-03-09T14:47:09.766 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.754+0000 7fcdfa849b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558738ea9fe0 0x558738ea3c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:09.766 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.754+0000 7fcdfa849b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558738eaf830 0x558738eb5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:09.766 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.754+0000 7fcdfa849b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558738eb0140 0x558738eb6220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:09.766 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.754+0000 7fcdfa849b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x558738ea46f0 con 0x558738eb0140 2023-03-09T14:47:09.766 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.754+0000 7fcdfa849b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x558738eb1920 con 0x558738ea9fe0 2023-03-09T14:47:09.766 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.754+0000 7fcdfa849b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x558738eaaa30 con 0x558738eaf830 2023-03-09T14:47:09.767 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.754+0000 7fcdecc5d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558738eb0140 0x558738eb6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:09.767 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.754+0000 7fcde77fe700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558738eaf830 0x558738eb5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:09.767 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.754+0000 7fcde77fe700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558738eaf830 0x558738eb5ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.38:47376/0 (socket says 172.21.15.38:47376) 2023-03-09T14:47:09.767 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.754+0000 7fcdecc5d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558738eb0140 0x558738eb6220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39680/0 (socket says 172.21.15.38:39680) 2023-03-09T14:47:09.767 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.754+0000 7fcde77fe700 1 -- 172.21.15.38:0/3977643942 learned_addr learned my addr 172.21.15.38:0/3977643942 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:47:09.768 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.754+0000 7fcde7fff700 1 --2- 172.21.15.38:0/3977643942 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558738ea9fe0 0x558738ea3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:09.768 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.754+0000 7fcde77fe700 1 -- 172.21.15.38:0/3977643942 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558738ea9fe0 msgr2=0x558738ea3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:47:09.768 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.754+0000 7fcde77fe700 1 --2- 172.21.15.38:0/3977643942 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558738ea9fe0 0x558738ea3c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.768 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.754+0000 7fcde77fe700 1 -- 172.21.15.38:0/3977643942 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558738eb0140 msgr2=0x558738eb6220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:47:09.768 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.754+0000 7fcde77fe700 1 --2- 172.21.15.38:0/3977643942 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558738eb0140 0x558738eb6220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.768 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.754+0000 7fcde77fe700 1 -- 172.21.15.38:0/3977643942 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558738c12c60 con 0x558738eaf830 2023-03-09T14:47:09.769 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.754+0000 7fcde7fff700 1 --2- 172.21.15.38:0/3977643942 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558738ea9fe0 0x558738ea3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:47:09.769 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.754+0000 7fcdecc5d700 1 --2- 172.21.15.38:0/3977643942 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558738eb0140 0x558738eb6220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:47:09.769 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.755+0000 7fcde77fe700 1 --2- 172.21.15.38:0/3977643942 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558738eaf830 0x558738eb5ce0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fcddc097810 tx=0x7fcddc005880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=58e18ebb9d3ed81d server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:47:09.769 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.755+0000 7fcdceffd700 1 -- 172.21.15.38:0/3977643942 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcddc0c7050 con 0x558738eaf830 2023-03-09T14:47:09.769 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.755+0000 7fcdceffd700 1 -- 172.21.15.38:0/3977643942 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fcddc097880 con 0x558738eaf830 2023-03-09T14:47:09.769 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.755+0000 7fcdceffd700 1 -- 172.21.15.38:0/3977643942 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcddc096d70 con 0x558738eaf830 2023-03-09T14:47:09.769 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.756+0000 7fcdfa849b80 1 -- 172.21.15.38:0/3977643942 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558738eaf830 msgr2=0x558738eb5ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:47:09.770 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.756+0000 7fcdfa849b80 1 --2- 172.21.15.38:0/3977643942 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558738eaf830 0x558738eb5ce0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fcddc097810 tx=0x7fcddc005880 comp rx=0 tx=0).stop 2023-03-09T14:47:09.770 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.756+0000 7fcdfa849b80 1 -- 172.21.15.38:0/3977643942 shutdown_connections 2023-03-09T14:47:09.770 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.756+0000 7fcdfa849b80 1 --2- 172.21.15.38:0/3977643942 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558738eb0140 0x558738eb6220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.770 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.756+0000 7fcdfa849b80 1 --2- 172.21.15.38:0/3977643942 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558738ea9fe0 0x558738ea3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.770 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.756+0000 7fcdfa849b80 1 --2- 172.21.15.38:0/3977643942 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558738eaf830 0x558738eb5ce0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.770 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.756+0000 7fcdfa849b80 1 -- 172.21.15.38:0/3977643942 >> 172.21.15.38:0/3977643942 conn(0x558738ea4a20 msgr2=0x558738ebf200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:47:09.770 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.756+0000 7fcdfa849b80 1 -- 172.21.15.38:0/3977643942 shutdown_connections 2023-03-09T14:47:09.770 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.756+0000 7fcdfa849b80 1 -- 172.21.15.38:0/3977643942 wait complete. 2023-03-09T14:47:09.771 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.756+0000 7fcdfa849b80 1 -- start start 2023-03-09T14:47:09.771 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.756+0000 7fcdfa849b80 1 Processor -- start 2023-03-09T14:47:09.772 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.757+0000 7fcdfa849b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558738ea9fe0 0x558738ebaa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:09.772 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.757+0000 7fcdfa849b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558738eaf830 0x558738eab6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:09.772 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.757+0000 7fcdfa849b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558738eb0140 0x558738eabc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:09.772 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.757+0000 7fcdfa849b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x558738eac2c0 con 0x558738eaf830 2023-03-09T14:47:09.772 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.757+0000 7fcdfa849b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x558738eac430 con 0x558738ea9fe0 2023-03-09T14:47:09.772 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.757+0000 7fcdfa849b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x558738f17bc0 con 0x558738eb0140 2023-03-09T14:47:09.772 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.757+0000 7fcdfa849b80 10 client.0 _ll_register_callbacks cb 0x558738da1520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:47:09.772 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.757+0000 7fcdecc5d700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558738eb0140 0x558738eabc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:09.773 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.757+0000 7fcde77fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558738eaf830 0x558738eab6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:09.773 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.757+0000 7fcdecc5d700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558738eb0140 0x558738eabc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.38:47382/0 (socket says 172.21.15.38:47382) 2023-03-09T14:47:09.773 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.757+0000 7fcdecc5d700 1 -- 172.21.15.38:0/2801968367 learned_addr learned my addr 172.21.15.38:0/2801968367 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:47:09.773 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.757+0000 7fcde77fe700 1 -- 172.21.15.38:0/2801968367 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558738eb0140 msgr2=0x558738eabc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:47:09.774 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.757+0000 7fcde77fe700 1 --2- 172.21.15.38:0/2801968367 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558738eb0140 0x558738eabc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.774 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.757+0000 7fcde77fe700 1 -- 172.21.15.38:0/2801968367 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558738ea9fe0 msgr2=0x558738ebaa70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-03-09T14:47:09.774 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.757+0000 7fcde77fe700 1 --2- 172.21.15.38:0/2801968367 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558738ea9fe0 0x558738ebaa70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.774 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.757+0000 7fcde77fe700 1 -- 172.21.15.38:0/2801968367 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcde0009440 con 0x558738eaf830 2023-03-09T14:47:09.774 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.757+0000 7fcdecc5d700 1 --2- 172.21.15.38:0/2801968367 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558738eb0140 0x558738eabc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:47:09.774 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.757+0000 7fcde77fe700 1 --2- 172.21.15.38:0/2801968367 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558738eaf830 0x558738eab6e0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x558738e9f6b0 tx=0x7fcddc000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77af6a23dc06dd6b server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:47:09.775 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.757+0000 7fcdc77fe700 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2023-03-09T14:47:09.775 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.757+0000 7fcdc77fe700 1 -- 172.21.15.38:0/2801968367 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcddc0c7050 con 0x558738eaf830 2023-03-09T14:47:09.775 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.758+0000 7fcdc77fe700 1 -- 172.21.15.38:0/2801968367 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9278+0+0 (secure 0 0 0) 0x7fcddc0c71b0 con 0x558738eaf830 2023-03-09T14:47:09.775 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.758+0000 7fcdfa849b80 1 -- 172.21.15.38:0/2801968367 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558738db91b0 con 0x558738eaf830 2023-03-09T14:47:09.775 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.758+0000 7fcdc77fe700 1 -- 172.21.15.38:0/2801968367 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcddc096d70 con 0x558738eaf830 2023-03-09T14:47:09.775 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.758+0000 7fcdfa849b80 10 client.15831 fetch_fsmap learned FSMap version 25 2023-03-09T14:47:09.775 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.758+0000 7fcdfa849b80 1 -- 172.21.15.38:0/2801968367 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x558738dc2190 con 0x558738eaf830 2023-03-09T14:47:09.776 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.758+0000 7fcdc77fe700 1 -- 172.21.15.38:0/2801968367 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x558738db91b0 con 0x558738eaf830 2023-03-09T14:47:09.776 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.758+0000 7fcdc77fe700 1 -- 172.21.15.38:0/2801968367 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 25) v1 ==== 7478+0+0 (secure 0 0 0) 0x7fcddc009a90 con 0x558738eaf830 2023-03-09T14:47:09.776 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.758+0000 7fcdfa849b80 10 client.15831 fetch_fsmap finished waiting for FSMap version 25 2023-03-09T14:47:09.776 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.758+0000 7fcdfa849b80 10 client.15831 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T14:47:09.776 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.758+0000 7fcdfa849b80 20 client.15831 populate_metadata read hostname 'smithi038' 2023-03-09T14:47:09.777 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.758+0000 7fcdfa849b80 1 --2- 172.21.15.38:0/2801968367 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x558738f18ce0 0x558738f1b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:09.777 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.758+0000 7fcdfa849b80 4 client.15831 mds_command: new command op to 24530 tid=0[{"prefix": "damage ls"}] 2023-03-09T14:47:09.777 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.758+0000 7fcdfa849b80 1 -- 172.21.15.38:0/2801968367 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x558738da31c0 con 0x558738f18ce0 2023-03-09T14:47:09.777 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.759+0000 7fcde7fff700 1 --2- 172.21.15.38:0/2801968367 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x558738f18ce0 0x558738f1b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:09.777 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.759+0000 7fcdc77fe700 10 client.15831 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:47:09.777 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.759+0000 7fcde7fff700 1 --2- 172.21.15.38:0/2801968367 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x558738f18ce0 0x558738f1b0f0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb793bf65abdd9f5 server_cookie=97dd16dd889e5bd8 in_seq=0 out_seq=0 2023-03-09T14:47:09.777 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.759+0000 7fcdc77fe700 1 -- 172.21.15.38:0/2801968367 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x558738da31c0 con 0x558738f18ce0 2023-03-09T14:47:09.778 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.759+0000 7fcdc77fe700 10 client.15831 handle_command_reply: tid=0 2023-03-09T14:47:09.778 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.759+0000 7fcdfa849b80 1 client.15831 shutdown 2023-03-09T14:47:09.778 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.760+0000 7fcdfa849b80 1 -- 172.21.15.38:0/2801968367 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558738eaf830 msgr2=0x558738eab6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:47:09.778 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.760+0000 7fcdfa849b80 1 --2- 172.21.15.38:0/2801968367 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558738eaf830 0x558738eab6e0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x558738e9f6b0 tx=0x7fcddc000bc0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.778 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.760+0000 7fcdfa849b80 1 -- 172.21.15.38:0/2801968367 shutdown_connections 2023-03-09T14:47:09.778 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.760+0000 7fcdfa849b80 1 --2- 172.21.15.38:0/2801968367 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x558738f18ce0 0x558738f1b0f0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.779 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.760+0000 7fcdfa849b80 1 --2- 172.21.15.38:0/2801968367 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558738eb0140 0x558738eabc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.779 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.760+0000 7fcdc77fe700 0 client.15831 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:47:09.779 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.760+0000 7fcdfa849b80 1 --2- 172.21.15.38:0/2801968367 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558738ea9fe0 0x558738ebaa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.779 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.760+0000 7fcdfa849b80 1 --2- 172.21.15.38:0/2801968367 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558738eaf830 0x558738eab6e0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.779 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.760+0000 7fcdfa849b80 1 -- 172.21.15.38:0/2801968367 >> 172.21.15.38:0/2801968367 conn(0x558738ea4a20 msgr2=0x558738ea3c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:47:09.779 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.760+0000 7fcdfa849b80 1 -- 172.21.15.38:0/2801968367 shutdown_connections 2023-03-09T14:47:09.779 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.760+0000 7fcdfa849b80 1 -- 172.21.15.38:0/2801968367 wait complete. 2023-03-09T14:47:09.780 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.760+0000 7fcdfa849b80 20 client.15831 trim_cache size 0 max 16384 2023-03-09T14:47:09.780 INFO:teuthology.orchestra.run.smithi038.stdout:[] 2023-03-09T14:47:09.781 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.761+0000 7fcdee6c0700 1 -- 172.21.15.38:0/3665682464 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fcdd009f530 msgr2=0x7fcdd00a19e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:47:09.781 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.761+0000 7fcdee6c0700 1 --2- 172.21.15.38:0/3665682464 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fcdd009f530 0x7fcdd00a19e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fcdd8005fd0 tx=0x7fcdd8009310 comp rx=0 tx=0).stop 2023-03-09T14:47:09.782 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.761+0000 7fcdee6c0700 1 -- 172.21.15.38:0/3665682464 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcde80ff9a0 msgr2=0x7fcde8194150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:47:09.782 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.761+0000 7fcdee6c0700 1 --2- 172.21.15.38:0/3665682464 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcde80ff9a0 0x7fcde8194150 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fcddc00d950 tx=0x7fcddc00dc60 comp rx=0 tx=0).stop 2023-03-09T14:47:09.782 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.761+0000 7fcdee6c0700 1 -- 172.21.15.38:0/3665682464 shutdown_connections 2023-03-09T14:47:09.782 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.761+0000 7fcdee6c0700 1 --2- 172.21.15.38:0/3665682464 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fcdd009f530 0x7fcdd00a19e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.782 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.761+0000 7fcdee6c0700 1 --2- 172.21.15.38:0/3665682464 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fcde80ff000 0x7fcde8193c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.782 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.761+0000 7fcdee6c0700 1 --2- 172.21.15.38:0/3665682464 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fcde8104040 0x7fcde8194690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.782 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.761+0000 7fcdee6c0700 1 --2- 172.21.15.38:0/3665682464 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fcde80ff9a0 0x7fcde8194150 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:09.782 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.761+0000 7fcdee6c0700 1 -- 172.21.15.38:0/3665682464 >> 172.21.15.38:0/3665682464 conn(0x7fcde8075c60 msgr2=0x7fcde80fc940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:47:09.783 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.761+0000 7fcdee6c0700 1 -- 172.21.15.38:0/3665682464 shutdown_connections 2023-03-09T14:47:09.783 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:09.761+0000 7fcdee6c0700 1 -- 172.21.15.38:0/3665682464 wait complete. 2023-03-09T14:47:09.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:47:09 smithi122 ceph-mon[93869]: pgmap v30: 129 pgs: 129 active+clean; 4.2 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 80 MiB/s wr, 202 op/s 2023-03-09T14:47:09.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:47:09 smithi122 ceph-mon[93869]: from='client.? 172.21.15.38:0/428990921' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-03-09T14:47:09.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:47:09 smithi181 ceph-mon[94189]: pgmap v30: 129 pgs: 129 active+clean; 4.2 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 80 MiB/s wr, 202 op/s 2023-03-09T14:47:09.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:47:09 smithi181 ceph-mon[94189]: from='client.? 172.21.15.38:0/428990921' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-03-09T14:47:10.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:47:09 smithi038 ceph-mon[92422]: pgmap v30: 129 pgs: 129 active+clean; 4.2 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 80 MiB/s wr, 202 op/s 2023-03-09T14:47:10.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:47:09 smithi038 ceph-mon[92422]: from='client.? 172.21.15.38:0/428990921' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-03-09T14:47:10.784 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2023-03-09T14:47:10.785 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2023-03-09T14:47:10.922 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.918+0000 7fb505ba8700 1 -- 172.21.15.38:0/770706118 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb500104c00 msgr2=0x7fb50010b5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:47:10.922 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.918+0000 7fb505ba8700 1 --2- 172.21.15.38:0/770706118 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb500104c00 0x7fb50010b5e0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fb4f800b0d0 tx=0x7fb4f800b3e0 comp rx=0 tx=0).stop 2023-03-09T14:47:10.922 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.918+0000 7fb505ba8700 1 -- 172.21.15.38:0/770706118 shutdown_connections 2023-03-09T14:47:10.923 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.918+0000 7fb505ba8700 1 --2- 172.21.15.38:0/770706118 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb500104c00 0x7fb50010b5e0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:10.923 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.918+0000 7fb505ba8700 1 --2- 172.21.15.38:0/770706118 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb5000ff570 0x7fb5000ff940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:10.923 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.918+0000 7fb505ba8700 1 --2- 172.21.15.38:0/770706118 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb5000fff10 0x7fb5001044d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:10.923 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.918+0000 7fb505ba8700 1 -- 172.21.15.38:0/770706118 >> 172.21.15.38:0/770706118 conn(0x7fb5000748f0 msgr2=0x7fb500074cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:47:10.923 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.918+0000 7fb505ba8700 1 -- 172.21.15.38:0/770706118 shutdown_connections 2023-03-09T14:47:10.924 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.919+0000 7fb505ba8700 1 -- 172.21.15.38:0/770706118 wait complete. 2023-03-09T14:47:10.925 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.919+0000 7fb505ba8700 1 Processor -- start 2023-03-09T14:47:10.926 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.919+0000 7fb505ba8700 1 -- start start 2023-03-09T14:47:10.926 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.920+0000 7fb505ba8700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb5000ff570 0x7fb500196370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:10.926 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.920+0000 7fb505ba8700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb5000fff10 0x7fb5001968b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:10.926 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.920+0000 7fb505ba8700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb500104c00 0x7fb50019acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:10.926 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.920+0000 7fb505ba8700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb50019b300 con 0x7fb5000fff10 2023-03-09T14:47:10.926 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.920+0000 7fb505ba8700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb50019b470 con 0x7fb500104c00 2023-03-09T14:47:10.927 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.920+0000 7fb505ba8700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb50019b630 con 0x7fb5000ff570 2023-03-09T14:47:10.927 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.920+0000 7fb4fffff700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb500104c00 0x7fb50019acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:10.927 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.920+0000 7fb4fffff700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb500104c00 0x7fb50019acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.38:48888/0 (socket says 172.21.15.38:48888) 2023-03-09T14:47:10.927 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.920+0000 7fb4fffff700 1 -- 172.21.15.38:0/1832808534 learned_addr learned my addr 172.21.15.38:0/1832808534 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:47:10.927 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.920+0000 7fb4ff7fe700 1 --2- 172.21.15.38:0/1832808534 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb5000ff570 0x7fb500196370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:10.927 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.920+0000 7fb4feffd700 1 --2- 172.21.15.38:0/1832808534 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb5000fff10 0x7fb5001968b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:10.927 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.920+0000 7fb4fffff700 1 -- 172.21.15.38:0/1832808534 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb5000ff570 msgr2=0x7fb500196370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:47:10.928 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.920+0000 7fb4fffff700 1 --2- 172.21.15.38:0/1832808534 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb5000ff570 0x7fb500196370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:10.928 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.920+0000 7fb4fffff700 1 -- 172.21.15.38:0/1832808534 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb5000fff10 msgr2=0x7fb5001968b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:47:10.928 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.920+0000 7fb4fffff700 1 --2- 172.21.15.38:0/1832808534 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb5000fff10 0x7fb5001968b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:10.928 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.920+0000 7fb4fffff700 1 -- 172.21.15.38:0/1832808534 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4f8009d00 con 0x7fb500104c00 2023-03-09T14:47:10.928 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.920+0000 7fb4feffd700 1 --2- 172.21.15.38:0/1832808534 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb5000fff10 0x7fb5001968b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:47:10.928 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.921+0000 7fb4fffff700 1 --2- 172.21.15.38:0/1832808534 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb500104c00 0x7fb50019acc0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb4f8007820 tx=0x7fb4f8007850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:47:10.929 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.921+0000 7fb4fcff9700 1 -- 172.21.15.38:0/1832808534 <== mon.2 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4f800e040 con 0x7fb500104c00 2023-03-09T14:47:10.929 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.921+0000 7fb505ba8700 1 -- 172.21.15.38:0/1832808534 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb50010e000 con 0x7fb500104c00 2023-03-09T14:47:10.929 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.921+0000 7fb4fcff9700 1 -- 172.21.15.38:0/1832808534 <== mon.2 v2:172.21.15.122:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb4f800bd30 con 0x7fb500104c00 2023-03-09T14:47:10.929 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.921+0000 7fb4fcff9700 1 -- 172.21.15.38:0/1832808534 <== mon.2 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4f801d470 con 0x7fb500104c00 2023-03-09T14:47:10.929 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.921+0000 7fb505ba8700 1 -- 172.21.15.38:0/1832808534 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb50010de70 con 0x7fb500104c00 2023-03-09T14:47:10.932 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.923+0000 7fb4fcff9700 1 -- 172.21.15.38:0/1832808534 <== mon.2 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 24) v1 ==== 133375+0+0 (secure 0 0 0) 0x7fb4f8004a50 con 0x7fb500104c00 2023-03-09T14:47:10.933 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.924+0000 7fb4fcff9700 1 --2- 172.21.15.38:0/1832808534 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fb4e809f530 0x7fb4e80a19e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:10.933 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.924+0000 7fb4fcff9700 1 -- 172.21.15.38:0/1832808534 <== mon.2 v2:172.21.15.122:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9278+0+0 (secure 0 0 0) 0x7fb4f80cb3b0 con 0x7fb500104c00 2023-03-09T14:47:10.933 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.924+0000 7fb4ff7fe700 1 --2- 172.21.15.38:0/1832808534 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fb4e809f530 0x7fb4e80a19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:10.933 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.925+0000 7fb4ff7fe700 1 --2- 172.21.15.38:0/1832808534 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fb4e809f530 0x7fb4e80a19e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb4f000acd0 tx=0x7fb4f0010040 comp rx=0 tx=0).ready entity=mgr.15558 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:47:10.933 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.925+0000 7fb511d31b80 1 Processor -- start 2023-03-09T14:47:10.933 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.925+0000 7fb511d31b80 1 -- start start 2023-03-09T14:47:10.933 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.925+0000 7fb511d31b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558e4cb058b0 0x558e4cb05c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:10.934 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.925+0000 7fb511d31b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558e4cb061c0 0x558e4caffb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:10.934 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.925+0000 7fb511d31b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558e4cb00060 0x558e4cb00490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:10.934 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.925+0000 7fb511d31b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e4c9e9e20 con 0x558e4cb058b0 2023-03-09T14:47:10.934 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.925+0000 7fb511d31b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e4ca58800 con 0x558e4cb061c0 2023-03-09T14:47:10.934 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.925+0000 7fb511d31b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e4ca51d30 con 0x558e4cb00060 2023-03-09T14:47:10.934 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.926+0000 7fb4ff7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558e4cb058b0 0x558e4cb05c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:10.934 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.926+0000 7fb4ff7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558e4cb058b0 0x558e4cb05c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39704/0 (socket says 172.21.15.38:39704) 2023-03-09T14:47:10.934 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.926+0000 7fb4feffd700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558e4cb061c0 0x558e4caffb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:10.935 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.926+0000 7fb4ff7fe700 1 -- 172.21.15.38:0/3797509680 learned_addr learned my addr 172.21.15.38:0/3797509680 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:47:10.935 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.926+0000 7fb4fffff700 1 --2- 172.21.15.38:0/3797509680 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558e4cb00060 0x558e4cb00490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:10.935 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.926+0000 7fb4ff7fe700 1 -- 172.21.15.38:0/3797509680 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558e4cb00060 msgr2=0x558e4cb00490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:47:10.935 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.926+0000 7fb4ff7fe700 1 --2- 172.21.15.38:0/3797509680 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558e4cb00060 0x558e4cb00490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:10.935 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.926+0000 7fb4ff7fe700 1 -- 172.21.15.38:0/3797509680 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558e4cb061c0 msgr2=0x558e4caffb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:47:10.935 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.926+0000 7fb4ff7fe700 1 --2- 172.21.15.38:0/3797509680 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558e4cb061c0 0x558e4caffb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:10.935 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.926+0000 7fb4ff7fe700 1 -- 172.21.15.38:0/3797509680 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558e4ca18210 con 0x558e4cb058b0 2023-03-09T14:47:10.936 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.926+0000 7fb4ff7fe700 1 --2- 172.21.15.38:0/3797509680 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558e4cb058b0 0x558e4cb05c80 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fb4f000efa0 tx=0x7fb4f0014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ffc9e57a40e46e8 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:47:10.936 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.926+0000 7fb4e5ffb700 1 -- 172.21.15.38:0/3797509680 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4f0020070 con 0x558e4cb058b0 2023-03-09T14:47:10.936 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.926+0000 7fb4e5ffb700 1 -- 172.21.15.38:0/3797509680 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb4f001b0d0 con 0x558e4cb058b0 2023-03-09T14:47:10.936 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.926+0000 7fb4e5ffb700 1 -- 172.21.15.38:0/3797509680 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4f0013040 con 0x558e4cb058b0 2023-03-09T14:47:10.936 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.927+0000 7fb511d31b80 1 -- 172.21.15.38:0/3797509680 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558e4cb058b0 msgr2=0x558e4cb05c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:47:10.936 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.927+0000 7fb511d31b80 1 --2- 172.21.15.38:0/3797509680 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558e4cb058b0 0x558e4cb05c80 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fb4f000efa0 tx=0x7fb4f0014000 comp rx=0 tx=0).stop 2023-03-09T14:47:10.936 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.927+0000 7fb511d31b80 1 -- 172.21.15.38:0/3797509680 shutdown_connections 2023-03-09T14:47:10.936 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.927+0000 7fb511d31b80 1 --2- 172.21.15.38:0/3797509680 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558e4cb058b0 0x558e4cb05c80 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:10.937 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.927+0000 7fb511d31b80 1 --2- 172.21.15.38:0/3797509680 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558e4cb00060 0x558e4cb00490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:10.937 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.927+0000 7fb511d31b80 1 --2- 172.21.15.38:0/3797509680 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558e4cb061c0 0x558e4caffb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:10.937 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.927+0000 7fb511d31b80 1 -- 172.21.15.38:0/3797509680 >> 172.21.15.38:0/3797509680 conn(0x558e4cafaaa0 msgr2=0x558e4cafae70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:47:10.937 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.927+0000 7fb511d31b80 1 -- 172.21.15.38:0/3797509680 shutdown_connections 2023-03-09T14:47:10.937 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.927+0000 7fb511d31b80 1 -- 172.21.15.38:0/3797509680 wait complete. 2023-03-09T14:47:10.937 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.927+0000 7fb511d31b80 1 -- start start 2023-03-09T14:47:10.937 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.928+0000 7fb511d31b80 1 Processor -- start 2023-03-09T14:47:10.938 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.928+0000 7fb511d31b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558e4cb00060 0x558e4cb0c100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:10.938 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.928+0000 7fb511d31b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558e4cb058b0 0x558e4cb09380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:10.938 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.928+0000 7fb511d31b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558e4cb061c0 0x558e4cb099b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:10.938 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.928+0000 7fb511d31b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e4c8d59a0 con 0x558e4cb058b0 2023-03-09T14:47:10.938 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.928+0000 7fb511d31b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e4cb0a0e0 con 0x558e4cb00060 2023-03-09T14:47:10.938 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.928+0000 7fb511d31b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e4cb15c60 con 0x558e4cb061c0 2023-03-09T14:47:10.938 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.928+0000 7fb511d31b80 10 client.0 _ll_register_callbacks cb 0x558e4c9f75a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:47:10.939 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.928+0000 7fb4ff7fe700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558e4cb00060 0x558e4cb0c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:10.939 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.928+0000 7fb4ff7fe700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558e4cb00060 0x558e4cb0c100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.38:48900/0 (socket says 172.21.15.38:48900) 2023-03-09T14:47:10.939 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.928+0000 7fb4ff7fe700 1 -- 172.21.15.38:0/441248574 learned_addr learned my addr 172.21.15.38:0/441248574 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:47:10.939 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.928+0000 7fb4feffd700 1 --2- 172.21.15.38:0/441248574 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558e4cb058b0 0x558e4cb09380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:10.939 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.928+0000 7fb4fffff700 1 --2- 172.21.15.38:0/441248574 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558e4cb061c0 0x558e4cb099b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:10.939 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.928+0000 7fb4ff7fe700 1 -- 172.21.15.38:0/441248574 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558e4cb061c0 msgr2=0x558e4cb099b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:47:10.939 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.928+0000 7fb4ff7fe700 1 --2- 172.21.15.38:0/441248574 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558e4cb061c0 0x558e4cb099b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:10.939 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.929+0000 7fb4ff7fe700 1 -- 172.21.15.38:0/441248574 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558e4cb058b0 msgr2=0x558e4cb09380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:47:10.940 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.929+0000 7fb4ff7fe700 1 --2- 172.21.15.38:0/441248574 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558e4cb058b0 0x558e4cb09380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:10.940 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.929+0000 7fb4ff7fe700 1 -- 172.21.15.38:0/441248574 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4f000ecd0 con 0x558e4cb00060 2023-03-09T14:47:10.940 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.929+0000 7fb4ff7fe700 1 --2- 172.21.15.38:0/441248574 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558e4cb00060 0x558e4cb0c100 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb4f0026040 tx=0x7fb4f001bd10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6cbe645edc675840 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:47:10.940 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.929+0000 7fb4deffd700 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2023-03-09T14:47:10.940 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.929+0000 7fb4deffd700 1 -- 172.21.15.38:0/441248574 <== mon.2 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4f0020040 con 0x558e4cb00060 2023-03-09T14:47:10.940 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.929+0000 7fb4deffd700 1 -- 172.21.15.38:0/441248574 <== mon.2 v2:172.21.15.122:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9278+0+0 (secure 0 0 0) 0x7fb4f0005260 con 0x558e4cb00060 2023-03-09T14:47:10.940 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.929+0000 7fb511d31b80 1 -- 172.21.15.38:0/441248574 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558e4ca1f210 con 0x558e4cb00060 2023-03-09T14:47:10.941 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.929+0000 7fb4deffd700 1 -- 172.21.15.38:0/441248574 <== mon.2 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb4f0013040 con 0x558e4cb00060 2023-03-09T14:47:10.941 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.929+0000 7fb511d31b80 10 client.25499 fetch_fsmap learned FSMap version 25 2023-03-09T14:47:10.941 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.929+0000 7fb511d31b80 1 -- 172.21.15.38:0/441248574 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x558e4c868ce0 con 0x558e4cb00060 2023-03-09T14:47:10.942 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.930+0000 7fb4deffd700 1 -- 172.21.15.38:0/441248574 <== mon.2 v2:172.21.15.122:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x558e4ca1f210 con 0x558e4cb00060 2023-03-09T14:47:10.942 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.930+0000 7fb4deffd700 1 -- 172.21.15.38:0/441248574 <== mon.2 v2:172.21.15.122:3300/0 5 ==== fsmap(e 25) v1 ==== 7478+0+0 (secure 0 0 0) 0x7fb4f0003680 con 0x558e4cb00060 2023-03-09T14:47:10.942 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.930+0000 7fb511d31b80 10 client.25499 fetch_fsmap finished waiting for FSMap version 25 2023-03-09T14:47:10.942 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.930+0000 7fb511d31b80 10 client.25499 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T14:47:10.942 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.930+0000 7fb511d31b80 20 client.25499 populate_metadata read hostname 'smithi038' 2023-03-09T14:47:10.943 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.930+0000 7fb511d31b80 1 --2- 172.21.15.38:0/441248574 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x558e4cb19c90 0x558e4cb1c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:10.943 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.930+0000 7fb511d31b80 4 client.25499 mds_command: new command op to 24530 tid=0[{"prefix": "get_command_descriptions"}] 2023-03-09T14:47:10.943 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.930+0000 7fb511d31b80 1 -- 172.21.15.38:0/441248574 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558e4c8d5fe0 con 0x558e4cb19c90 2023-03-09T14:47:10.943 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.930+0000 7fb4feffd700 1 --2- 172.21.15.38:0/441248574 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x558e4cb19c90 0x558e4cb1c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:10.943 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.930+0000 7fb4deffd700 10 client.25499 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:47:10.943 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.930+0000 7fb4feffd700 1 --2- 172.21.15.38:0/441248574 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x558e4cb19c90 0x558e4cb1c0a0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6513c35054df2a5 server_cookie=735e4d089aaf5c8a in_seq=0 out_seq=0 2023-03-09T14:47:10.943 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.931+0000 7fb4deffd700 1 -- 172.21.15.38:0/441248574 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18278 (crc 0 0 0) 0x558e4c8d5fe0 con 0x558e4cb19c90 2023-03-09T14:47:10.944 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.931+0000 7fb4deffd700 10 client.25499 handle_command_reply: tid=0 2023-03-09T14:47:10.944 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.931+0000 7fb511d31b80 1 client.25499 shutdown 2023-03-09T14:47:10.944 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.931+0000 7fb511d31b80 1 -- 172.21.15.38:0/441248574 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558e4cb00060 msgr2=0x558e4cb0c100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:47:10.944 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.931+0000 7fb511d31b80 1 --2- 172.21.15.38:0/441248574 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558e4cb00060 0x558e4cb0c100 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb4f0026040 tx=0x7fb4f001bd10 comp rx=0 tx=0).stop 2023-03-09T14:47:10.944 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.931+0000 7fb511d31b80 1 -- 172.21.15.38:0/441248574 shutdown_connections 2023-03-09T14:47:10.944 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.931+0000 7fb511d31b80 1 --2- 172.21.15.38:0/441248574 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x558e4cb19c90 0x558e4cb1c0a0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:10.944 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.931+0000 7fb511d31b80 1 --2- 172.21.15.38:0/441248574 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558e4cb061c0 0x558e4cb099b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:10.944 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.931+0000 7fb4deffd700 0 client.25499 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:47:10.945 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.931+0000 7fb511d31b80 1 --2- 172.21.15.38:0/441248574 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558e4cb00060 0x558e4cb0c100 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:10.945 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.931+0000 7fb511d31b80 1 --2- 172.21.15.38:0/441248574 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558e4cb058b0 0x558e4cb09380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:10.945 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.931+0000 7fb511d31b80 1 -- 172.21.15.38:0/441248574 >> 172.21.15.38:0/441248574 conn(0x558e4cafaaa0 msgr2=0x558e4caf9ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:47:10.945 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.931+0000 7fb511d31b80 1 -- 172.21.15.38:0/441248574 shutdown_connections 2023-03-09T14:47:10.945 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.931+0000 7fb511d31b80 1 -- 172.21.15.38:0/441248574 wait complete. 2023-03-09T14:47:10.945 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.932+0000 7fb511d31b80 20 client.25499 trim_cache size 0 max 16384 2023-03-09T14:47:10.958 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.953+0000 7fb511d31b80 1 Processor -- start 2023-03-09T14:47:10.958 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.953+0000 7fb511d31b80 1 -- start start 2023-03-09T14:47:10.958 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.953+0000 7fb511d31b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558e4cb00060 0x558e4cb0bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:10.958 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.953+0000 7fb511d31b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558e4cb058b0 0x558e4cb06680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:10.958 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.953+0000 7fb511d31b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558e4cb061c0 0x558e4cb06bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:10.958 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.953+0000 7fb511d31b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e4caffb20 con 0x558e4cb00060 2023-03-09T14:47:10.959 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.953+0000 7fb511d31b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e4cb00d40 con 0x558e4cb061c0 2023-03-09T14:47:10.959 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.953+0000 7fb511d31b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e4cb0c6e0 con 0x558e4cb058b0 2023-03-09T14:47:10.959 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.954+0000 7fb4ff7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558e4cb00060 0x558e4cb0bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:10.959 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.954+0000 7fb4ff7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558e4cb00060 0x558e4cb0bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39718/0 (socket says 172.21.15.38:39718) 2023-03-09T14:47:10.959 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.954+0000 7fb4ff7fe700 1 -- 172.21.15.38:0/2367600247 learned_addr learned my addr 172.21.15.38:0/2367600247 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:47:10.959 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.954+0000 7fb4fffff700 1 --2- 172.21.15.38:0/2367600247 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558e4cb061c0 0x558e4cb06bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:10.959 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.954+0000 7fb4feffd700 1 --2- 172.21.15.38:0/2367600247 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558e4cb058b0 0x558e4cb06680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:10.960 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.954+0000 7fb4ff7fe700 1 -- 172.21.15.38:0/2367600247 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558e4cb058b0 msgr2=0x558e4cb06680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:47:10.960 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.954+0000 7fb4ff7fe700 1 --2- 172.21.15.38:0/2367600247 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558e4cb058b0 0x558e4cb06680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:10.960 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.954+0000 7fb4ff7fe700 1 -- 172.21.15.38:0/2367600247 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558e4cb061c0 msgr2=0x558e4cb06bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:47:10.960 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.954+0000 7fb4ff7fe700 1 --2- 172.21.15.38:0/2367600247 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558e4cb061c0 0x558e4cb06bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:10.960 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.954+0000 7fb4ff7fe700 1 -- 172.21.15.38:0/2367600247 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558e4c87fe60 con 0x558e4cb00060 2023-03-09T14:47:10.960 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.954+0000 7fb4ff7fe700 1 --2- 172.21.15.38:0/2367600247 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558e4cb00060 0x558e4cb0bca0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fb4f00042f0 tx=0x7fb4f0003c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f251ebe4f76829e4 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:47:10.960 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.954+0000 7fb4e67fc700 1 -- 172.21.15.38:0/2367600247 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4f0013070 con 0x558e4cb00060 2023-03-09T14:47:10.960 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.954+0000 7fb4e67fc700 1 -- 172.21.15.38:0/2367600247 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb4f0004470 con 0x558e4cb00060 2023-03-09T14:47:10.961 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.954+0000 7fb4e67fc700 1 -- 172.21.15.38:0/2367600247 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4f0020070 con 0x558e4cb00060 2023-03-09T14:47:10.964 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.955+0000 7fb511d31b80 1 -- 172.21.15.38:0/2367600247 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558e4cb00060 msgr2=0x558e4cb0bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:47:10.965 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.955+0000 7fb511d31b80 1 --2- 172.21.15.38:0/2367600247 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558e4cb00060 0x558e4cb0bca0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fb4f00042f0 tx=0x7fb4f0003c30 comp rx=0 tx=0).stop 2023-03-09T14:47:10.965 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.955+0000 7fb511d31b80 1 -- 172.21.15.38:0/2367600247 shutdown_connections 2023-03-09T14:47:10.965 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.955+0000 7fb511d31b80 1 --2- 172.21.15.38:0/2367600247 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558e4cb00060 0x558e4cb0bca0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:10.965 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.955+0000 7fb511d31b80 1 --2- 172.21.15.38:0/2367600247 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558e4cb061c0 0x558e4cb06bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:10.965 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.955+0000 7fb511d31b80 1 --2- 172.21.15.38:0/2367600247 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558e4cb058b0 0x558e4cb06680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:10.965 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.955+0000 7fb511d31b80 1 -- 172.21.15.38:0/2367600247 >> 172.21.15.38:0/2367600247 conn(0x558e4cafaaa0 msgr2=0x558e4cb03510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:47:10.966 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.955+0000 7fb511d31b80 1 -- 172.21.15.38:0/2367600247 shutdown_connections 2023-03-09T14:47:10.966 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.955+0000 7fb511d31b80 1 -- 172.21.15.38:0/2367600247 wait complete. 2023-03-09T14:47:10.966 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.955+0000 7fb511d31b80 1 -- start start 2023-03-09T14:47:10.966 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.956+0000 7fb511d31b80 1 Processor -- start 2023-03-09T14:47:10.966 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.956+0000 7fb511d31b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558e4cb00060 0x558e4cb10a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:10.966 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.956+0000 7fb511d31b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558e4cb058b0 0x558e4cb0dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:10.966 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.956+0000 7fb511d31b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558e4cb061c0 0x558e4cb0e200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:10.967 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.956+0000 7fb511d31b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e4cb0e740 con 0x558e4cb00060 2023-03-09T14:47:10.967 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.956+0000 7fb511d31b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e4cb0e8b0 con 0x558e4cb061c0 2023-03-09T14:47:10.967 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.956+0000 7fb511d31b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x558e4cb0ea70 con 0x558e4cb058b0 2023-03-09T14:47:10.967 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.956+0000 7fb511d31b80 10 client.0 _ll_register_callbacks cb 0x558e4c9f75a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:47:10.967 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.956+0000 7fb4ff7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558e4cb00060 0x558e4cb10a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:10.967 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.956+0000 7fb4ff7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558e4cb00060 0x558e4cb10a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39722/0 (socket says 172.21.15.38:39722) 2023-03-09T14:47:10.967 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.956+0000 7fb4ff7fe700 1 -- 172.21.15.38:0/4048008268 learned_addr learned my addr 172.21.15.38:0/4048008268 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:47:10.967 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.956+0000 7fb4feffd700 1 --2- 172.21.15.38:0/4048008268 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558e4cb058b0 0x558e4cb0dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:10.968 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.957+0000 7fb4fffff700 1 --2- 172.21.15.38:0/4048008268 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558e4cb061c0 0x558e4cb0e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:10.968 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.957+0000 7fb4ff7fe700 1 -- 172.21.15.38:0/4048008268 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558e4cb058b0 msgr2=0x558e4cb0dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:47:10.968 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.957+0000 7fb4ff7fe700 1 --2- 172.21.15.38:0/4048008268 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558e4cb058b0 0x558e4cb0dcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:10.968 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.957+0000 7fb4ff7fe700 1 -- 172.21.15.38:0/4048008268 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558e4cb061c0 msgr2=0x558e4cb0e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:47:10.968 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.957+0000 7fb4ff7fe700 1 --2- 172.21.15.38:0/4048008268 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558e4cb061c0 0x558e4cb0e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:10.968 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.957+0000 7fb4ff7fe700 1 -- 172.21.15.38:0/4048008268 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558e4c868ce0 con 0x558e4cb00060 2023-03-09T14:47:10.968 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.957+0000 7fb4ff7fe700 1 --2- 172.21.15.38:0/4048008268 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558e4cb00060 0x558e4cb10a10 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fb4f0003b30 tx=0x7fb4f000eb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7573e5d1421c63c8 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:47:10.969 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.957+0000 7fb4deffd700 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2023-03-09T14:47:10.969 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.957+0000 7fb4deffd700 1 -- 172.21.15.38:0/4048008268 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4f0013070 con 0x558e4cb00060 2023-03-09T14:47:10.969 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.957+0000 7fb4deffd700 1 -- 172.21.15.38:0/4048008268 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9278+0+0 (secure 0 0 0) 0x7fb4f0005260 con 0x558e4cb00060 2023-03-09T14:47:10.969 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.957+0000 7fb511d31b80 1 -- 172.21.15.38:0/4048008268 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558e4ca0f230 con 0x558e4cb00060 2023-03-09T14:47:10.969 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.957+0000 7fb4deffd700 1 -- 172.21.15.38:0/4048008268 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb4f0020070 con 0x558e4cb00060 2023-03-09T14:47:10.969 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.957+0000 7fb511d31b80 10 client.15849 fetch_fsmap learned FSMap version 25 2023-03-09T14:47:10.969 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.958+0000 7fb511d31b80 1 -- 172.21.15.38:0/4048008268 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x558e4c87e330 con 0x558e4cb00060 2023-03-09T14:47:10.969 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.958+0000 7fb4deffd700 1 -- 172.21.15.38:0/4048008268 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x558e4ca0f230 con 0x558e4cb00060 2023-03-09T14:47:10.970 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.958+0000 7fb4deffd700 1 -- 172.21.15.38:0/4048008268 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 25) v1 ==== 7478+0+0 (secure 0 0 0) 0x7fb4f001b700 con 0x558e4cb00060 2023-03-09T14:47:10.970 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.958+0000 7fb511d31b80 10 client.15849 fetch_fsmap finished waiting for FSMap version 25 2023-03-09T14:47:10.970 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.958+0000 7fb511d31b80 10 client.15849 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T14:47:10.970 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.958+0000 7fb511d31b80 20 client.15849 populate_metadata read hostname 'smithi038' 2023-03-09T14:47:10.970 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.958+0000 7fb511d31b80 1 --2- 172.21.15.38:0/4048008268 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x558e4cb6fbe0 0x558e4cb71ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:10.970 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.958+0000 7fb511d31b80 4 client.15849 mds_command: new command op to 24530 tid=0[{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2023-03-09T14:47:10.970 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.958+0000 7fb511d31b80 1 -- 172.21.15.38:0/4048008268 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x558e4c9f9240 con 0x558e4cb6fbe0 2023-03-09T14:47:10.971 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.958+0000 7fb4feffd700 1 --2- 172.21.15.38:0/4048008268 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x558e4cb6fbe0 0x558e4cb71ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:10.971 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.959+0000 7fb4deffd700 10 client.15849 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:47:10.971 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.959+0000 7fb4feffd700 1 --2- 172.21.15.38:0/4048008268 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x558e4cb6fbe0 0x558e4cb71ff0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1cd853f5897f44b4 server_cookie=b0bbf8b2c69c3f38 in_seq=0 out_seq=0 2023-03-09T14:47:10.971 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.960+0000 7fb4deffd700 1 -- 172.21.15.38:0/4048008268 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x558e4c9f9240 con 0x558e4cb6fbe0 2023-03-09T14:47:10.971 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.960+0000 7fb4deffd700 10 client.15849 handle_command_reply: tid=0 2023-03-09T14:47:10.971 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.960+0000 7fb511d31b80 1 client.15849 shutdown 2023-03-09T14:47:10.971 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.960+0000 7fb511d31b80 1 -- 172.21.15.38:0/4048008268 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558e4cb00060 msgr2=0x558e4cb10a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:47:10.971 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.960+0000 7fb511d31b80 1 --2- 172.21.15.38:0/4048008268 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558e4cb00060 0x558e4cb10a10 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fb4f0003b30 tx=0x7fb4f000eb30 comp rx=0 tx=0).stop 2023-03-09T14:47:10.972 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.960+0000 7fb511d31b80 1 -- 172.21.15.38:0/4048008268 shutdown_connections 2023-03-09T14:47:10.972 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.960+0000 7fb511d31b80 1 --2- 172.21.15.38:0/4048008268 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x558e4cb6fbe0 0x558e4cb71ff0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:10.972 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.960+0000 7fb511d31b80 1 --2- 172.21.15.38:0/4048008268 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558e4cb00060 0x558e4cb10a10 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:10.972 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.960+0000 7fb511d31b80 1 --2- 172.21.15.38:0/4048008268 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558e4cb061c0 0x558e4cb0e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:10.972 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.960+0000 7fb511d31b80 1 --2- 172.21.15.38:0/4048008268 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558e4cb058b0 0x558e4cb0dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:10.972 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.960+0000 7fb511d31b80 1 -- 172.21.15.38:0/4048008268 >> 172.21.15.38:0/4048008268 conn(0x558e4cafaaa0 msgr2=0x558e4cb06680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:47:10.972 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.960+0000 7fb4deffd700 0 client.15849 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:47:10.973 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.960+0000 7fb511d31b80 1 -- 172.21.15.38:0/4048008268 shutdown_connections 2023-03-09T14:47:10.973 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.960+0000 7fb511d31b80 1 -- 172.21.15.38:0/4048008268 wait complete. 2023-03-09T14:47:10.973 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.960+0000 7fb511d31b80 20 client.15849 trim_cache size 0 max 16384 2023-03-09T14:47:10.973 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2023-03-09T14:47:10.973 INFO:teuthology.orchestra.run.smithi038.stdout: "return_code": 0, 2023-03-09T14:47:10.973 INFO:teuthology.orchestra.run.smithi038.stdout: "scrub_tag": "a14659f4-d7ea-4c5a-b15b-6138af3f909f", 2023-03-09T14:47:10.974 INFO:teuthology.orchestra.run.smithi038.stdout: "mode": "asynchronous" 2023-03-09T14:47:10.974 INFO:teuthology.orchestra.run.smithi038.stdout:} 2023-03-09T14:47:10.974 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.961+0000 7fb505ba8700 1 -- 172.21.15.38:0/1832808534 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fb4e809f530 msgr2=0x7fb4e80a19e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:47:10.975 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.961+0000 7fb505ba8700 1 --2- 172.21.15.38:0/1832808534 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fb4e809f530 0x7fb4e80a19e0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fb4f000acd0 tx=0x7fb4f0010040 comp rx=0 tx=0).stop 2023-03-09T14:47:10.975 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.961+0000 7fb505ba8700 1 -- 172.21.15.38:0/1832808534 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb500104c00 msgr2=0x7fb50019acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:47:10.975 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.961+0000 7fb505ba8700 1 --2- 172.21.15.38:0/1832808534 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb500104c00 0x7fb50019acc0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb4f8007820 tx=0x7fb4f8007850 comp rx=0 tx=0).stop 2023-03-09T14:47:10.975 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.961+0000 7fb505ba8700 1 -- 172.21.15.38:0/1832808534 shutdown_connections 2023-03-09T14:47:10.975 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.961+0000 7fb505ba8700 1 --2- 172.21.15.38:0/1832808534 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fb4e809f530 0x7fb4e80a19e0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:10.975 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.961+0000 7fb505ba8700 1 --2- 172.21.15.38:0/1832808534 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb500104c00 0x7fb50019acc0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:10.975 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.961+0000 7fb505ba8700 1 --2- 172.21.15.38:0/1832808534 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb5000ff570 0x7fb500196370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:10.976 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.961+0000 7fb505ba8700 1 --2- 172.21.15.38:0/1832808534 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb5000fff10 0x7fb5001968b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:10.976 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.961+0000 7fb505ba8700 1 -- 172.21.15.38:0/1832808534 >> 172.21.15.38:0/1832808534 conn(0x7fb5000748f0 msgr2=0x7fb5000fe310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:47:10.976 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.961+0000 7fb505ba8700 1 -- 172.21.15.38:0/1832808534 shutdown_connections 2023-03-09T14:47:10.976 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:10.961+0000 7fb505ba8700 1 -- 172.21.15.38:0/1832808534 wait complete. 2023-03-09T14:47:10.978 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-03-09T14:47:11.137 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.134+0000 7fe448712700 1 -- 172.21.15.38:0/272089629 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe4400ff9a0 msgr2=0x7fe440103960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:47:11.138 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.134+0000 7fe448712700 1 --2- 172.21.15.38:0/272089629 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe4400ff9a0 0x7fe440103960 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fe4300099c0 tx=0x7fe430009cd0 comp rx=0 tx=0).stop 2023-03-09T14:47:11.138 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.134+0000 7fe448712700 1 -- 172.21.15.38:0/272089629 shutdown_connections 2023-03-09T14:47:11.138 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.134+0000 7fe448712700 1 --2- 172.21.15.38:0/272089629 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe440104040 0x7fe44010bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:11.138 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.134+0000 7fe448712700 1 --2- 172.21.15.38:0/272089629 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe4400ff000 0x7fe4400ff3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:11.138 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.134+0000 7fe448712700 1 --2- 172.21.15.38:0/272089629 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe4400ff9a0 0x7fe440103960 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:11.138 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.134+0000 7fe448712700 1 -- 172.21.15.38:0/272089629 >> 172.21.15.38:0/272089629 conn(0x7fe440075c60 msgr2=0x7fe440076060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:47:11.139 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.134+0000 7fe448712700 1 -- 172.21.15.38:0/272089629 shutdown_connections 2023-03-09T14:47:11.139 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.134+0000 7fe448712700 1 -- 172.21.15.38:0/272089629 wait complete. 2023-03-09T14:47:11.140 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.135+0000 7fe448712700 1 Processor -- start 2023-03-09T14:47:11.140 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.135+0000 7fe448712700 1 -- start start 2023-03-09T14:47:11.140 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.135+0000 7fe448712700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe4400ff000 0x7fe440196140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:11.141 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.135+0000 7fe448712700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe4400ff9a0 0x7fe440196680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:11.141 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.135+0000 7fe448712700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe440104040 0x7fe44019aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:11.141 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.135+0000 7fe448712700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe44019b120 con 0x7fe440104040 2023-03-09T14:47:11.141 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.135+0000 7fe448712700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe44019b290 con 0x7fe4400ff000 2023-03-09T14:47:11.141 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.135+0000 7fe448712700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe44019b450 con 0x7fe4400ff9a0 2023-03-09T14:47:11.141 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.135+0000 7fe446caf700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe440104040 0x7fe44019aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:11.141 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.135+0000 7fe4464ae700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe4400ff000 0x7fe440196140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:11.142 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.135+0000 7fe4464ae700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe4400ff000 0x7fe440196140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.38:48926/0 (socket says 172.21.15.38:48926) 2023-03-09T14:47:11.142 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.135+0000 7fe446caf700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe440104040 0x7fe44019aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39740/0 (socket says 172.21.15.38:39740) 2023-03-09T14:47:11.142 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.135+0000 7fe445cad700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe4400ff9a0 0x7fe440196680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:11.142 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.135+0000 7fe4464ae700 1 -- 172.21.15.38:0/4293717686 learned_addr learned my addr 172.21.15.38:0/4293717686 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:47:11.142 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.135+0000 7fe446caf700 1 -- 172.21.15.38:0/4293717686 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe4400ff9a0 msgr2=0x7fe440196680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:47:11.142 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.135+0000 7fe446caf700 1 --2- 172.21.15.38:0/4293717686 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe4400ff9a0 0x7fe440196680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:11.143 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.135+0000 7fe446caf700 1 -- 172.21.15.38:0/4293717686 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe4400ff000 msgr2=0x7fe440196140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:47:11.143 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.135+0000 7fe446caf700 1 --2- 172.21.15.38:0/4293717686 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe4400ff000 0x7fe440196140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:11.143 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.136+0000 7fe446caf700 1 -- 172.21.15.38:0/4293717686 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4300095f0 con 0x7fe440104040 2023-03-09T14:47:11.143 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.136+0000 7fe445cad700 1 --2- 172.21.15.38:0/4293717686 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe4400ff9a0 0x7fe440196680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:47:11.143 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.136+0000 7fe446caf700 1 --2- 172.21.15.38:0/4293717686 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe440104040 0x7fe44019aae0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fe43800efc0 tx=0x7fe43800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:47:11.143 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.136+0000 7fe4377fe700 1 -- 172.21.15.38:0/4293717686 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe438018070 con 0x7fe440104040 2023-03-09T14:47:11.143 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.136+0000 7fe4377fe700 1 -- 172.21.15.38:0/4293717686 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe43800f040 con 0x7fe440104040 2023-03-09T14:47:11.143 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.136+0000 7fe448712700 1 -- 172.21.15.38:0/4293717686 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe44010ee10 con 0x7fe440104040 2023-03-09T14:47:11.144 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.136+0000 7fe4377fe700 1 -- 172.21.15.38:0/4293717686 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe43800cd40 con 0x7fe440104040 2023-03-09T14:47:11.144 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.136+0000 7fe448712700 1 -- 172.21.15.38:0/4293717686 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe44010ec80 con 0x7fe440104040 2023-03-09T14:47:11.145 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.139+0000 7fe4377fe700 1 -- 172.21.15.38:0/4293717686 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 24) v1 ==== 133375+0+0 (secure 0 0 0) 0x7fe438003a00 con 0x7fe440104040 2023-03-09T14:47:11.145 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.140+0000 7fe4377fe700 1 --2- 172.21.15.38:0/4293717686 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fe42c09f5b0 0x7fe42c0a1a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:11.145 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.140+0000 7fe45489bb80 1 Processor -- start 2023-03-09T14:47:11.146 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.140+0000 7fe4377fe700 1 -- 172.21.15.38:0/4293717686 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9278+0+0 (secure 0 0 0) 0x7fe4380ca050 con 0x7fe440104040 2023-03-09T14:47:11.146 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.140+0000 7fe4464ae700 1 --2- 172.21.15.38:0/4293717686 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fe42c09f5b0 0x7fe42c0a1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:11.146 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.140+0000 7fe45489bb80 1 -- start start 2023-03-09T14:47:11.146 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.140+0000 7fe45489bb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5556c59b9900 0x5556c59b5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:11.146 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.140+0000 7fe45489bb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5556c59b5fe0 0x5556c59b6430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:11.146 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.140+0000 7fe45489bb80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5556c59c8700 0x5556c59caae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:11.146 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.140+0000 7fe45489bb80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556c589fda0 con 0x5556c59c8700 2023-03-09T14:47:11.147 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.140+0000 7fe45489bb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556c590e780 con 0x5556c59b9900 2023-03-09T14:47:11.147 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.140+0000 7fe45489bb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556c5907cb0 con 0x5556c59b5fe0 2023-03-09T14:47:11.147 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.141+0000 7fe4464ae700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5556c59b9900 0x5556c59b5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:11.147 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.141+0000 7fe446caf700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5556c59c8700 0x5556c59caae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:11.147 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.141+0000 7fe4464ae700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5556c59b9900 0x5556c59b5aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.38:48934/0 (socket says 172.21.15.38:48934) 2023-03-09T14:47:11.147 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.141+0000 7fe4464ae700 1 -- 172.21.15.38:0/628038691 learned_addr learned my addr 172.21.15.38:0/628038691 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:47:11.147 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.141+0000 7fe445cad700 1 --2- 172.21.15.38:0/628038691 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5556c59b5fe0 0x5556c59b6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:11.147 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.141+0000 7fe4464ae700 1 --2- 172.21.15.38:0/4293717686 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fe42c09f5b0 0x7fe42c0a1a60 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe43c009d20 tx=0x7fe43c009260 comp rx=0 tx=0).ready entity=mgr.15558 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:47:11.148 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.141+0000 7fe4464ae700 1 -- 172.21.15.38:0/628038691 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5556c59b5fe0 msgr2=0x5556c59b6430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:47:11.148 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.141+0000 7fe4464ae700 1 --2- 172.21.15.38:0/628038691 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5556c59b5fe0 0x5556c59b6430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:11.148 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.141+0000 7fe4464ae700 1 -- 172.21.15.38:0/628038691 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5556c59c8700 msgr2=0x5556c59caae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:47:11.148 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.141+0000 7fe4464ae700 1 --2- 172.21.15.38:0/628038691 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5556c59c8700 0x5556c59caae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:11.148 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.141+0000 7fe4464ae700 1 -- 172.21.15.38:0/628038691 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5556c58ce190 con 0x5556c59b9900 2023-03-09T14:47:11.148 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.141+0000 7fe4464ae700 1 --2- 172.21.15.38:0/628038691 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5556c59b9900 0x5556c59b5aa0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe43c01c5b0 tx=0x7fe43c01b000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b533ca1f616952aa server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:47:11.148 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.141+0000 7fe434ff9700 1 -- 172.21.15.38:0/628038691 <== mon.2 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe43c004a30 con 0x5556c59b9900 2023-03-09T14:47:11.149 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.141+0000 7fe434ff9700 1 -- 172.21.15.38:0/628038691 <== mon.2 v2:172.21.15.122:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe43c0054e0 con 0x5556c59b9900 2023-03-09T14:47:11.149 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.142+0000 7fe434ff9700 1 -- 172.21.15.38:0/628038691 <== mon.2 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe43c00e910 con 0x5556c59b9900 2023-03-09T14:47:11.151 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.142+0000 7fe45489bb80 1 -- 172.21.15.38:0/628038691 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5556c59b9900 msgr2=0x5556c59b5aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:47:11.152 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.142+0000 7fe45489bb80 1 --2- 172.21.15.38:0/628038691 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5556c59b9900 0x5556c59b5aa0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe43c01c5b0 tx=0x7fe43c01b000 comp rx=0 tx=0).stop 2023-03-09T14:47:11.152 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.143+0000 7fe45489bb80 1 -- 172.21.15.38:0/628038691 shutdown_connections 2023-03-09T14:47:11.152 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.143+0000 7fe45489bb80 1 --2- 172.21.15.38:0/628038691 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5556c59c8700 0x5556c59caae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:11.152 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.143+0000 7fe45489bb80 1 --2- 172.21.15.38:0/628038691 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5556c59b9900 0x5556c59b5aa0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:11.153 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.143+0000 7fe45489bb80 1 --2- 172.21.15.38:0/628038691 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5556c59b5fe0 0x5556c59b6430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:11.153 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.143+0000 7fe45489bb80 1 -- 172.21.15.38:0/628038691 >> 172.21.15.38:0/628038691 conn(0x5556c59b0a20 msgr2=0x5556c59b0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:47:11.153 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.143+0000 7fe45489bb80 1 -- 172.21.15.38:0/628038691 shutdown_connections 2023-03-09T14:47:11.153 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.143+0000 7fe45489bb80 1 -- 172.21.15.38:0/628038691 wait complete. 2023-03-09T14:47:11.153 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.143+0000 7fe45489bb80 1 -- start start 2023-03-09T14:47:11.153 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.143+0000 7fe45489bb80 1 Processor -- start 2023-03-09T14:47:11.153 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.144+0000 7fe45489bb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5556c59b5fe0 0x5556c59c8110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:11.154 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.144+0000 7fe45489bb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5556c59b9900 0x5556c59c5320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:11.154 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.144+0000 7fe45489bb80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5556c59c8700 0x5556c59c5950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:11.154 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.144+0000 7fe45489bb80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556c578b920 con 0x5556c59c8700 2023-03-09T14:47:11.154 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.144+0000 7fe45489bb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556c59c6090 con 0x5556c59b9900 2023-03-09T14:47:11.154 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.144+0000 7fe45489bb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556c59d4450 con 0x5556c59b5fe0 2023-03-09T14:47:11.154 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.144+0000 7fe45489bb80 10 client.0 _ll_register_callbacks cb 0x5556c58ad520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:47:11.154 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.144+0000 7fe4464ae700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5556c59b5fe0 0x5556c59c8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:11.155 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.144+0000 7fe4464ae700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5556c59b5fe0 0x5556c59c8110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.38:47448/0 (socket says 172.21.15.38:47448) 2023-03-09T14:47:11.155 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.144+0000 7fe4464ae700 1 -- 172.21.15.38:0/2046103683 learned_addr learned my addr 172.21.15.38:0/2046103683 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:47:11.155 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.144+0000 7fe446caf700 1 --2- 172.21.15.38:0/2046103683 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5556c59c8700 0x5556c59c5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:11.155 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.144+0000 7fe4464ae700 1 -- 172.21.15.38:0/2046103683 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5556c59b9900 msgr2=0x5556c59c5320 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-03-09T14:47:11.155 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.144+0000 7fe4464ae700 1 --2- 172.21.15.38:0/2046103683 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5556c59b9900 0x5556c59c5320 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:11.155 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.144+0000 7fe4464ae700 1 -- 172.21.15.38:0/2046103683 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5556c59c8700 msgr2=0x5556c59c5950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:47:11.155 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.144+0000 7fe4464ae700 1 --2- 172.21.15.38:0/2046103683 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5556c59c8700 0x5556c59c5950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:11.155 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.144+0000 7fe4464ae700 1 -- 172.21.15.38:0/2046103683 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe43c017070 con 0x5556c59b5fe0 2023-03-09T14:47:11.156 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.144+0000 7fe4464ae700 1 --2- 172.21.15.38:0/2046103683 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5556c59b5fe0 0x5556c59c8110 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe43c00e3f0 tx=0x7fe43c0052f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=311a2f1797ab58f5 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:47:11.156 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.144+0000 7fe429ffb700 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2023-03-09T14:47:11.156 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.145+0000 7fe429ffb700 1 -- 172.21.15.38:0/2046103683 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe43c004750 con 0x5556c59b5fe0 2023-03-09T14:47:11.156 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.145+0000 7fe429ffb700 1 -- 172.21.15.38:0/2046103683 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9278+0+0 (secure 0 0 0) 0x7fe43c0048b0 con 0x5556c59b5fe0 2023-03-09T14:47:11.156 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.145+0000 7fe45489bb80 1 -- 172.21.15.38:0/2046103683 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5556c58d5190 con 0x5556c59b5fe0 2023-03-09T14:47:11.156 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.145+0000 7fe429ffb700 1 -- 172.21.15.38:0/2046103683 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe43c004a70 con 0x5556c59b5fe0 2023-03-09T14:47:11.156 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.145+0000 7fe45489bb80 10 client.25465 fetch_fsmap learned FSMap version 25 2023-03-09T14:47:11.156 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.145+0000 7fe45489bb80 1 -- 172.21.15.38:0/2046103683 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5556c571ec60 con 0x5556c59b5fe0 2023-03-09T14:47:11.157 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.145+0000 7fe429ffb700 1 -- 172.21.15.38:0/2046103683 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5556c58d5190 con 0x5556c59b5fe0 2023-03-09T14:47:11.157 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.145+0000 7fe429ffb700 1 -- 172.21.15.38:0/2046103683 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 25) v1 ==== 7478+0+0 (secure 0 0 0) 0x7fe43c02c450 con 0x5556c59b5fe0 2023-03-09T14:47:11.157 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.145+0000 7fe45489bb80 10 client.25465 fetch_fsmap finished waiting for FSMap version 25 2023-03-09T14:47:11.157 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.145+0000 7fe45489bb80 10 client.25465 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T14:47:11.157 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.145+0000 7fe45489bb80 20 client.25465 populate_metadata read hostname 'smithi038' 2023-03-09T14:47:11.157 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.145+0000 7fe45489bb80 1 --2- 172.21.15.38:0/2046103683 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5556c59d84c0 0x5556c59da8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:11.157 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.145+0000 7fe45489bb80 4 client.25465 mds_command: new command op to 24530 tid=0[{"prefix": "get_command_descriptions"}] 2023-03-09T14:47:11.158 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.145+0000 7fe45489bb80 1 -- 172.21.15.38:0/2046103683 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5556c578bf60 con 0x5556c59d84c0 2023-03-09T14:47:11.158 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.146+0000 7fe445cad700 1 --2- 172.21.15.38:0/2046103683 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5556c59d84c0 0x5556c59da8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:11.158 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.146+0000 7fe429ffb700 10 client.25465 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:47:11.158 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.146+0000 7fe445cad700 1 --2- 172.21.15.38:0/2046103683 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5556c59d84c0 0x5556c59da8d0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d8103548bff73038 server_cookie=cacc5d08b2b19e77 in_seq=0 out_seq=0 2023-03-09T14:47:11.158 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.147+0000 7fe429ffb700 1 -- 172.21.15.38:0/2046103683 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18278 (crc 0 0 0) 0x5556c578bf60 con 0x5556c59d84c0 2023-03-09T14:47:11.158 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.147+0000 7fe429ffb700 10 client.25465 handle_command_reply: tid=0 2023-03-09T14:47:11.158 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.147+0000 7fe45489bb80 1 client.25465 shutdown 2023-03-09T14:47:11.158 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.147+0000 7fe45489bb80 1 -- 172.21.15.38:0/2046103683 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5556c59b5fe0 msgr2=0x5556c59c8110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:47:11.159 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.147+0000 7fe45489bb80 1 --2- 172.21.15.38:0/2046103683 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5556c59b5fe0 0x5556c59c8110 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe43c00e3f0 tx=0x7fe43c0052f0 comp rx=0 tx=0).stop 2023-03-09T14:47:11.159 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.147+0000 7fe45489bb80 1 -- 172.21.15.38:0/2046103683 shutdown_connections 2023-03-09T14:47:11.159 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.147+0000 7fe45489bb80 1 --2- 172.21.15.38:0/2046103683 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5556c59d84c0 0x5556c59da8d0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:11.159 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.147+0000 7fe45489bb80 1 --2- 172.21.15.38:0/2046103683 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5556c59c8700 0x5556c59c5950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:11.159 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.147+0000 7fe45489bb80 1 --2- 172.21.15.38:0/2046103683 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5556c59b5fe0 0x5556c59c8110 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:11.159 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.147+0000 7fe429ffb700 0 client.25465 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:47:11.159 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.147+0000 7fe45489bb80 1 --2- 172.21.15.38:0/2046103683 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5556c59b9900 0x5556c59c5320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:11.160 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.147+0000 7fe45489bb80 1 -- 172.21.15.38:0/2046103683 >> 172.21.15.38:0/2046103683 conn(0x5556c59b0a20 msgr2=0x5556c59afc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:47:11.160 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.147+0000 7fe45489bb80 1 -- 172.21.15.38:0/2046103683 shutdown_connections 2023-03-09T14:47:11.160 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.147+0000 7fe45489bb80 1 -- 172.21.15.38:0/2046103683 wait complete. 2023-03-09T14:47:11.160 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.147+0000 7fe45489bb80 20 client.25465 trim_cache size 0 max 16384 2023-03-09T14:47:11.172 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.169+0000 7fe45489bb80 1 Processor -- start 2023-03-09T14:47:11.174 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.169+0000 7fe45489bb80 1 -- start start 2023-03-09T14:47:11.174 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.169+0000 7fe45489bb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5556c59b5fe0 0x5556c59cf3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:11.174 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.169+0000 7fe45489bb80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5556c59b9900 0x5556c59d4750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:11.174 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.169+0000 7fe45489bb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5556c59c8700 0x5556c59afc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:11.174 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.169+0000 7fe45489bb80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556c59b9d40 con 0x5556c59b9900 2023-03-09T14:47:11.174 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.170+0000 7fe45489bb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556c59b1f20 con 0x5556c59b5fe0 2023-03-09T14:47:11.175 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.170+0000 7fe45489bb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556c59c38c0 con 0x5556c59c8700 2023-03-09T14:47:11.175 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.170+0000 7fe445cad700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5556c59b9900 0x5556c59d4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:11.175 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.170+0000 7fe446caf700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5556c59c8700 0x5556c59afc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:11.175 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.170+0000 7fe445cad700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5556c59b9900 0x5556c59d4750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39756/0 (socket says 172.21.15.38:39756) 2023-03-09T14:47:11.175 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.170+0000 7fe446caf700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5556c59c8700 0x5556c59afc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.38:47458/0 (socket says 172.21.15.38:47458) 2023-03-09T14:47:11.175 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.170+0000 7fe445cad700 1 -- 172.21.15.38:0/2847564591 learned_addr learned my addr 172.21.15.38:0/2847564591 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:47:11.175 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.170+0000 7fe4464ae700 1 --2- 172.21.15.38:0/2847564591 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5556c59b5fe0 0x5556c59cf3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:11.175 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.170+0000 7fe446caf700 1 -- 172.21.15.38:0/2847564591 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5556c59b5fe0 msgr2=0x5556c59cf3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:47:11.176 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.170+0000 7fe446caf700 1 --2- 172.21.15.38:0/2847564591 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5556c59b5fe0 0x5556c59cf3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:11.176 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.170+0000 7fe446caf700 1 -- 172.21.15.38:0/2847564591 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5556c59b9900 msgr2=0x5556c59d4750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:47:11.176 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.170+0000 7fe446caf700 1 --2- 172.21.15.38:0/2847564591 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5556c59b9900 0x5556c59d4750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:11.176 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.170+0000 7fe446caf700 1 -- 172.21.15.38:0/2847564591 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5556c5735de0 con 0x5556c59c8700 2023-03-09T14:47:11.176 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.170+0000 7fe445cad700 1 --2- 172.21.15.38:0/2847564591 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5556c59b9900 0x5556c59d4750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:47:11.176 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.170+0000 7fe446caf700 1 --2- 172.21.15.38:0/2847564591 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5556c59c8700 0x5556c59afc20 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe4380946a0 tx=0x7fe4380c9040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=52a63462f0dcb2f server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:47:11.176 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.170+0000 7fe4357fa700 1 -- 172.21.15.38:0/2847564591 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe43809ffe0 con 0x5556c59c8700 2023-03-09T14:47:11.177 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.171+0000 7fe4357fa700 1 -- 172.21.15.38:0/2847564591 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe438004ad0 con 0x5556c59c8700 2023-03-09T14:47:11.177 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.171+0000 7fe4357fa700 1 -- 172.21.15.38:0/2847564591 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe438097380 con 0x5556c59c8700 2023-03-09T14:47:11.179 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.172+0000 7fe45489bb80 1 -- 172.21.15.38:0/2847564591 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5556c59c8700 msgr2=0x5556c59afc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:47:11.179 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.172+0000 7fe45489bb80 1 --2- 172.21.15.38:0/2847564591 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5556c59c8700 0x5556c59afc20 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe4380946a0 tx=0x7fe4380c9040 comp rx=0 tx=0).stop 2023-03-09T14:47:11.180 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.172+0000 7fe45489bb80 1 -- 172.21.15.38:0/2847564591 shutdown_connections 2023-03-09T14:47:11.180 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.172+0000 7fe45489bb80 1 --2- 172.21.15.38:0/2847564591 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5556c59c8700 0x5556c59afc20 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:11.180 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.172+0000 7fe45489bb80 1 --2- 172.21.15.38:0/2847564591 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5556c59b5fe0 0x5556c59cf3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:11.180 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.172+0000 7fe45489bb80 1 --2- 172.21.15.38:0/2847564591 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5556c59b9900 0x5556c59d4750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:11.180 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.172+0000 7fe45489bb80 1 -- 172.21.15.38:0/2847564591 >> 172.21.15.38:0/2847564591 conn(0x5556c59b0a20 msgr2=0x5556c59aef70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:47:11.180 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.172+0000 7fe45489bb80 1 -- 172.21.15.38:0/2847564591 shutdown_connections 2023-03-09T14:47:11.180 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.172+0000 7fe45489bb80 1 -- 172.21.15.38:0/2847564591 wait complete. 2023-03-09T14:47:11.181 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.172+0000 7fe45489bb80 1 -- start start 2023-03-09T14:47:11.181 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.172+0000 7fe45489bb80 1 Processor -- start 2023-03-09T14:47:11.181 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.172+0000 7fe45489bb80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5556c59b5fe0 0x5556c5a28a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:11.181 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.173+0000 7fe45489bb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5556c59b9900 0x5556c59ce540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:11.181 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.173+0000 7fe45489bb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5556c59c8700 0x5556c59cea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:11.181 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.173+0000 7fe45489bb80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556c59cefc0 con 0x5556c59b5fe0 2023-03-09T14:47:11.182 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.173+0000 7fe45489bb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556c59cf100 con 0x5556c59c8700 2023-03-09T14:47:11.182 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.173+0000 7fe45489bb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556c59cf2c0 con 0x5556c59b9900 2023-03-09T14:47:11.182 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.173+0000 7fe45489bb80 10 client.0 _ll_register_callbacks cb 0x5556c58ad520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:47:11.182 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.173+0000 7fe4464ae700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5556c59b5fe0 0x5556c5a28a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:11.182 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.173+0000 7fe446caf700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5556c59c8700 0x5556c59cea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:11.182 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.173+0000 7fe4464ae700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5556c59b5fe0 0x5556c5a28a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39762/0 (socket says 172.21.15.38:39762) 2023-03-09T14:47:11.182 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.173+0000 7fe446caf700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5556c59c8700 0x5556c59cea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.38:48954/0 (socket says 172.21.15.38:48954) 2023-03-09T14:47:11.183 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.173+0000 7fe4464ae700 1 -- 172.21.15.38:0/352268620 learned_addr learned my addr 172.21.15.38:0/352268620 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:47:11.183 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.173+0000 7fe445cad700 1 --2- 172.21.15.38:0/352268620 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5556c59b9900 0x5556c59ce540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:11.183 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.173+0000 7fe4464ae700 1 -- 172.21.15.38:0/352268620 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5556c59b9900 msgr2=0x5556c59ce540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:47:11.183 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.173+0000 7fe4464ae700 1 --2- 172.21.15.38:0/352268620 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5556c59b9900 0x5556c59ce540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:11.183 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.173+0000 7fe4464ae700 1 -- 172.21.15.38:0/352268620 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5556c59c8700 msgr2=0x5556c59cea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:47:11.183 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.173+0000 7fe4464ae700 1 --2- 172.21.15.38:0/352268620 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5556c59c8700 0x5556c59cea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:11.183 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.173+0000 7fe4464ae700 1 -- 172.21.15.38:0/352268620 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe44010c450 con 0x5556c59b5fe0 2023-03-09T14:47:11.183 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.173+0000 7fe445cad700 1 --2- 172.21.15.38:0/352268620 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5556c59b9900 0x5556c59ce540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:47:11.184 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.173+0000 7fe4464ae700 1 --2- 172.21.15.38:0/352268620 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5556c59b5fe0 0x5556c5a28a60 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fe43c02c7b0 tx=0x7fe43c01cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b91c345f70f6cd2c server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:47:11.184 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.173+0000 7fe429ffb700 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2023-03-09T14:47:11.184 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.174+0000 7fe429ffb700 1 -- 172.21.15.38:0/352268620 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe43c011ae0 con 0x5556c59b5fe0 2023-03-09T14:47:11.184 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.174+0000 7fe429ffb700 1 -- 172.21.15.38:0/352268620 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 9278+0+0 (secure 0 0 0) 0x7fe43c020030 con 0x5556c59b5fe0 2023-03-09T14:47:11.184 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.174+0000 7fe45489bb80 1 -- 172.21.15.38:0/352268620 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5556c58c51b0 con 0x5556c59b5fe0 2023-03-09T14:47:11.184 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.174+0000 7fe429ffb700 1 -- 172.21.15.38:0/352268620 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe43c02cbe0 con 0x5556c59b5fe0 2023-03-09T14:47:11.184 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.174+0000 7fe45489bb80 10 client.15867 fetch_fsmap learned FSMap version 25 2023-03-09T14:47:11.185 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.174+0000 7fe45489bb80 1 -- 172.21.15.38:0/352268620 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x7fe44010ec80 con 0x5556c59b5fe0 2023-03-09T14:47:11.185 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.174+0000 7fe429ffb700 1 -- 172.21.15.38:0/352268620 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5556c58c51b0 con 0x5556c59b5fe0 2023-03-09T14:47:11.185 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.174+0000 7fe429ffb700 1 -- 172.21.15.38:0/352268620 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 25) v1 ==== 7478+0+0 (secure 0 0 0) 0x7fe43c00e420 con 0x5556c59b5fe0 2023-03-09T14:47:11.185 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.174+0000 7fe45489bb80 10 client.15867 fetch_fsmap finished waiting for FSMap version 25 2023-03-09T14:47:11.185 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.174+0000 7fe45489bb80 10 client.15867 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T14:47:11.185 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.174+0000 7fe45489bb80 20 client.15867 populate_metadata read hostname 'smithi038' 2023-03-09T14:47:11.185 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.174+0000 7fe45489bb80 1 --2- 172.21.15.38:0/352268620 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5556c59bf0e0 0x5556c59c14f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:11.185 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.174+0000 7fe45489bb80 4 client.15867 mds_command: new command op to 24530 tid=0[{"prefix": "scrub status"}] 2023-03-09T14:47:11.186 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.174+0000 7fe45489bb80 1 -- 172.21.15.38:0/352268620 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5556c58af1c0 con 0x5556c59bf0e0 2023-03-09T14:47:11.186 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.175+0000 7fe445cad700 1 --2- 172.21.15.38:0/352268620 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5556c59bf0e0 0x5556c59c14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:11.186 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.175+0000 7fe429ffb700 10 client.15867 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:47:11.186 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.175+0000 7fe445cad700 1 --2- 172.21.15.38:0/352268620 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5556c59bf0e0 0x5556c59c14f0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e5d6a7c0c7e7bd09 server_cookie=63ecbdc1878a8d2d in_seq=0 out_seq=0 2023-03-09T14:47:11.186 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.175+0000 7fe429ffb700 1 -- 172.21.15.38:0/352268620 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+265 (crc 0 0 0) 0x5556c58af1c0 con 0x5556c59bf0e0 2023-03-09T14:47:11.186 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.176+0000 7fe429ffb700 10 client.15867 handle_command_reply: tid=0 2023-03-09T14:47:11.186 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.176+0000 7fe45489bb80 1 client.15867 shutdown 2023-03-09T14:47:11.187 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.176+0000 7fe45489bb80 1 -- 172.21.15.38:0/352268620 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5556c59b5fe0 msgr2=0x5556c5a28a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:47:11.187 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.176+0000 7fe45489bb80 1 --2- 172.21.15.38:0/352268620 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5556c59b5fe0 0x5556c5a28a60 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fe43c02c7b0 tx=0x7fe43c01cc30 comp rx=0 tx=0).stop 2023-03-09T14:47:11.187 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.176+0000 7fe45489bb80 1 -- 172.21.15.38:0/352268620 shutdown_connections 2023-03-09T14:47:11.187 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.176+0000 7fe45489bb80 1 --2- 172.21.15.38:0/352268620 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5556c59bf0e0 0x5556c59c14f0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:11.187 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.176+0000 7fe45489bb80 1 --2- 172.21.15.38:0/352268620 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5556c59b5fe0 0x5556c5a28a60 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:11.187 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.176+0000 7fe45489bb80 1 --2- 172.21.15.38:0/352268620 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5556c59c8700 0x5556c59cea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:11.187 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.176+0000 7fe429ffb700 0 client.15867 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:47:11.187 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.176+0000 7fe45489bb80 1 --2- 172.21.15.38:0/352268620 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5556c59b9900 0x5556c59ce540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:11.188 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.176+0000 7fe45489bb80 1 -- 172.21.15.38:0/352268620 >> 172.21.15.38:0/352268620 conn(0x5556c59b0a20 msgr2=0x5556c59afc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:47:11.188 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.176+0000 7fe45489bb80 1 -- 172.21.15.38:0/352268620 shutdown_connections 2023-03-09T14:47:11.188 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.176+0000 7fe45489bb80 1 -- 172.21.15.38:0/352268620 wait complete. 2023-03-09T14:47:11.188 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.176+0000 7fe45489bb80 20 client.15867 trim_cache size 0 max 16384 2023-03-09T14:47:11.188 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2023-03-09T14:47:11.188 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "scrub active (3 inodes in the stack)", 2023-03-09T14:47:11.189 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": { 2023-03-09T14:47:11.189 INFO:teuthology.orchestra.run.smithi038.stdout: "a14659f4-d7ea-4c5a-b15b-6138af3f909f": { 2023-03-09T14:47:11.189 INFO:teuthology.orchestra.run.smithi038.stdout: "path": "/", 2023-03-09T14:47:11.189 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "a14659f4-d7ea-4c5a-b15b-6138af3f909f", 2023-03-09T14:47:11.189 INFO:teuthology.orchestra.run.smithi038.stdout: "options": "recursive,force" 2023-03-09T14:47:11.189 INFO:teuthology.orchestra.run.smithi038.stdout: } 2023-03-09T14:47:11.189 INFO:teuthology.orchestra.run.smithi038.stdout: } 2023-03-09T14:47:11.190 INFO:teuthology.orchestra.run.smithi038.stdout:} 2023-03-09T14:47:11.190 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.177+0000 7fe448712700 1 -- 172.21.15.38:0/4293717686 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fe42c09f5b0 msgr2=0x7fe42c0a1a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:47:11.190 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.177+0000 7fe448712700 1 --2- 172.21.15.38:0/4293717686 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fe42c09f5b0 0x7fe42c0a1a60 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe43c009d20 tx=0x7fe43c009260 comp rx=0 tx=0).stop 2023-03-09T14:47:11.191 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.177+0000 7fe448712700 1 -- 172.21.15.38:0/4293717686 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe440104040 msgr2=0x7fe44019aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:47:11.191 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.177+0000 7fe448712700 1 --2- 172.21.15.38:0/4293717686 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe440104040 0x7fe44019aae0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fe43800efc0 tx=0x7fe43800c3a0 comp rx=0 tx=0).stop 2023-03-09T14:47:11.191 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.177+0000 7fe448712700 1 -- 172.21.15.38:0/4293717686 shutdown_connections 2023-03-09T14:47:11.191 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.177+0000 7fe448712700 1 --2- 172.21.15.38:0/4293717686 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fe42c09f5b0 0x7fe42c0a1a60 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:11.191 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.177+0000 7fe448712700 1 --2- 172.21.15.38:0/4293717686 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe440104040 0x7fe44019aae0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:11.191 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.177+0000 7fe448712700 1 --2- 172.21.15.38:0/4293717686 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe4400ff000 0x7fe440196140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:11.191 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.177+0000 7fe448712700 1 --2- 172.21.15.38:0/4293717686 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe4400ff9a0 0x7fe440196680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:11.191 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.177+0000 7fe448712700 1 -- 172.21.15.38:0/4293717686 >> 172.21.15.38:0/4293717686 conn(0x7fe440075c60 msgr2=0x7fe440102b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:47:11.192 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.177+0000 7fe448712700 1 -- 172.21.15.38:0/4293717686 shutdown_connections 2023-03-09T14:47:11.192 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:11.177+0000 7fe448712700 1 -- 172.21.15.38:0/4293717686 wait complete. 2023-03-09T14:47:11.194 INFO:tasks.cephfs.filesystem:scrub status for tag:a14659f4-d7ea-4c5a-b15b-6138af3f909f - {'path': '/', 'tag': 'a14659f4-d7ea-4c5a-b15b-6138af3f909f', 'options': 'recursive,force'} 2023-03-09T14:47:11.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:47:11 smithi122 ceph-mon[93869]: pgmap v31: 129 pgs: 129 active+clean; 4.2 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 80 MiB/s wr, 202 op/s 2023-03-09T14:47:11.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:47:11 smithi122 ceph-mon[93869]: scrub queued for path: / 2023-03-09T14:47:11.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:47:11 smithi122 ceph-mon[93869]: scrub summary: idle+waiting paths [/] 2023-03-09T14:47:11.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:47:11 smithi122 ceph-mon[93869]: scrub summary: active paths [/] 2023-03-09T14:47:11.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:47:11 smithi122 ceph-mon[93869]: scrub summary: active paths [/] 2023-03-09T14:47:11.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:47:11 smithi122 ceph-mon[93869]: scrub summary: idle+waiting paths [/] 2023-03-09T14:47:11.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:47:11 smithi181 ceph-mon[94189]: pgmap v31: 129 pgs: 129 active+clean; 4.2 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 80 MiB/s wr, 202 op/s 2023-03-09T14:47:11.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:47:11 smithi181 ceph-mon[94189]: scrub queued for path: / 2023-03-09T14:47:11.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:47:11 smithi181 ceph-mon[94189]: scrub summary: idle+waiting paths [/] 2023-03-09T14:47:11.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:47:11 smithi181 ceph-mon[94189]: scrub summary: active paths [/] 2023-03-09T14:47:11.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:47:11 smithi181 ceph-mon[94189]: scrub summary: active paths [/] 2023-03-09T14:47:11.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:47:11 smithi181 ceph-mon[94189]: scrub summary: idle+waiting paths [/] 2023-03-09T14:47:12.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:47:11 smithi038 ceph-mon[92422]: pgmap v31: 129 pgs: 129 active+clean; 4.2 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 80 MiB/s wr, 202 op/s 2023-03-09T14:47:12.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:47:11 smithi038 ceph-mon[92422]: scrub queued for path: / 2023-03-09T14:47:12.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:47:11 smithi038 ceph-mon[92422]: scrub summary: idle+waiting paths [/] 2023-03-09T14:47:12.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:47:11 smithi038 ceph-mon[92422]: scrub summary: active paths [/] 2023-03-09T14:47:12.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:47:11 smithi038 ceph-mon[92422]: scrub summary: active paths [/] 2023-03-09T14:47:12.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:47:11 smithi038 ceph-mon[92422]: scrub summary: idle+waiting paths [/] 2023-03-09T14:47:12.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:47:12 smithi122 ceph-mon[93869]: scrub summary: active paths [/] 2023-03-09T14:47:12.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:47:12 smithi122 ceph-mon[93869]: scrub summary: idle+waiting paths [/] 2023-03-09T14:47:12.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:47:12 smithi122 ceph-mon[93869]: scrub summary: active paths [/] 2023-03-09T14:47:12.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:47:12 smithi122 ceph-mon[93869]: scrub summary: idle+waiting paths [/] 2023-03-09T14:47:12.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:47:12 smithi122 ceph-mon[93869]: scrub summary: active paths [/] 2023-03-09T14:47:12.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:47:12 smithi122 ceph-mon[93869]: scrub summary: idle+waiting paths [/] 2023-03-09T14:47:12.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:47:12 smithi181 ceph-mon[94189]: scrub summary: active paths [/] 2023-03-09T14:47:12.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:47:12 smithi181 ceph-mon[94189]: scrub summary: idle+waiting paths [/] 2023-03-09T14:47:12.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:47:12 smithi181 ceph-mon[94189]: scrub summary: active paths [/] 2023-03-09T14:47:12.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:47:12 smithi181 ceph-mon[94189]: scrub summary: idle+waiting paths [/] 2023-03-09T14:47:12.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:47:12 smithi181 ceph-mon[94189]: scrub summary: active paths [/] 2023-03-09T14:47:12.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:47:12 smithi181 ceph-mon[94189]: scrub summary: idle+waiting paths [/] 2023-03-09T14:47:13.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:47:12 smithi038 ceph-mon[92422]: scrub summary: active paths [/] 2023-03-09T14:47:13.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:47:12 smithi038 ceph-mon[92422]: scrub summary: idle+waiting paths [/] 2023-03-09T14:47:13.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:47:12 smithi038 ceph-mon[92422]: scrub summary: active paths [/] 2023-03-09T14:47:13.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:47:12 smithi038 ceph-mon[92422]: scrub summary: idle+waiting paths [/] 2023-03-09T14:47:13.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:47:12 smithi038 ceph-mon[92422]: scrub summary: active paths [/] 2023-03-09T14:47:13.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:47:12 smithi038 ceph-mon[92422]: scrub summary: idle+waiting paths [/] 2023-03-09T14:47:13.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:47:13 smithi122 ceph-mon[93869]: pgmap v32: 129 pgs: 129 active+clean; 4.2 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 81 MiB/s wr, 315 op/s 2023-03-09T14:47:13.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:47:13 smithi181 ceph-mon[94189]: pgmap v32: 129 pgs: 129 active+clean; 4.2 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 81 MiB/s wr, 315 op/s 2023-03-09T14:47:14.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:47:13 smithi038 ceph-mon[92422]: pgmap v32: 129 pgs: 129 active+clean; 4.2 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 81 MiB/s wr, 315 op/s 2023-03-09T14:47:14.618 INFO:teuthology.orchestra.run.smithi038.stderr:29200000 of 50000000 tuples (58%) done (elapsed 49.76 s, remaining 35.44 s) 2023-03-09T14:47:14.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:47:14 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]: dispatch 2023-03-09T14:47:14.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:47:14 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]: dispatch 2023-03-09T14:47:14.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:47:14 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [3, 8]}]: dispatch 2023-03-09T14:47:14.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:47:14 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]: dispatch 2023-03-09T14:47:14.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:47:14 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 7]}]: dispatch 2023-03-09T14:47:14.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:47:14 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]: dispatch 2023-03-09T14:47:14.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:47:14 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10, 5, 1]}]: dispatch 2023-03-09T14:47:14.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:47:14 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]: dispatch 2023-03-09T14:47:14.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:47:14 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]: dispatch 2023-03-09T14:47:14.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:47:14 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]: dispatch 2023-03-09T14:47:14.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:47:14 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [3, 8]}]: dispatch 2023-03-09T14:47:14.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:47:14 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]: dispatch 2023-03-09T14:47:14.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:47:14 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 7]}]: dispatch 2023-03-09T14:47:14.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:47:14 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]: dispatch 2023-03-09T14:47:14.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:47:14 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10, 5, 1]}]: dispatch 2023-03-09T14:47:14.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:47:14 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]: dispatch 2023-03-09T14:47:15.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:47:14 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]: dispatch 2023-03-09T14:47:15.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:47:14 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]: dispatch 2023-03-09T14:47:15.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:47:14 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [3, 8]}]: dispatch 2023-03-09T14:47:15.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:47:14 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]: dispatch 2023-03-09T14:47:15.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:47:14 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 7]}]: dispatch 2023-03-09T14:47:15.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:47:14 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]: dispatch 2023-03-09T14:47:15.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:47:14 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10, 5, 1]}]: dispatch 2023-03-09T14:47:15.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:47:14 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]: dispatch 2023-03-09T14:47:15.124 INFO:teuthology.orchestra.run.smithi038.stderr:29300000 of 50000000 tuples (58%) done (elapsed 50.26 s, remaining 35.51 s) 2023-03-09T14:47:15.228 INFO:teuthology.orchestra.run.smithi038.stderr:29400000 of 50000000 tuples (58%) done (elapsed 50.37 s, remaining 35.29 s) 2023-03-09T14:47:15.335 INFO:teuthology.orchestra.run.smithi038.stderr:29500000 of 50000000 tuples (59%) done (elapsed 50.47 s, remaining 35.07 s) 2023-03-09T14:47:15.514 INFO:teuthology.orchestra.run.smithi038.stderr:29600000 of 50000000 tuples (59%) done (elapsed 50.65 s, remaining 34.91 s) 2023-03-09T14:47:15.652 INFO:teuthology.orchestra.run.smithi038.stderr:29700000 of 50000000 tuples (59%) done (elapsed 50.79 s, remaining 34.72 s) 2023-03-09T14:47:15.754 INFO:teuthology.orchestra.run.smithi038.stderr:29800000 of 50000000 tuples (59%) done (elapsed 50.89 s, remaining 34.50 s) 2023-03-09T14:47:15.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:47:15 smithi122 ceph-mon[93869]: pgmap v33: 129 pgs: 129 active+clean; 4.2 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 1.7 MiB/s wr, 257 op/s 2023-03-09T14:47:15.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:47:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]': finished 2023-03-09T14:47:15.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:47:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]': finished 2023-03-09T14:47:15.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:47:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [3, 8]}]': finished 2023-03-09T14:47:15.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:47:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]': finished 2023-03-09T14:47:15.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:47:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 7]}]': finished 2023-03-09T14:47:15.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:47:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]': finished 2023-03-09T14:47:15.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:47:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10, 5, 1]}]': finished 2023-03-09T14:47:15.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:47:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]': finished 2023-03-09T14:47:15.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:47:15 smithi122 ceph-mon[93869]: osdmap e91: 12 total, 12 up, 12 in 2023-03-09T14:47:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:47:15 smithi181 ceph-mon[94189]: pgmap v33: 129 pgs: 129 active+clean; 4.2 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 1.7 MiB/s wr, 257 op/s 2023-03-09T14:47:15.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:47:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]': finished 2023-03-09T14:47:15.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:47:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]': finished 2023-03-09T14:47:15.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:47:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [3, 8]}]': finished 2023-03-09T14:47:15.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:47:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]': finished 2023-03-09T14:47:15.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:47:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 7]}]': finished 2023-03-09T14:47:15.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:47:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]': finished 2023-03-09T14:47:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:47:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10, 5, 1]}]': finished 2023-03-09T14:47:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:47:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]': finished 2023-03-09T14:47:15.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:47:15 smithi181 ceph-mon[94189]: osdmap e91: 12 total, 12 up, 12 in 2023-03-09T14:47:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:47:15 smithi038 ceph-mon[92422]: pgmap v33: 129 pgs: 129 active+clean; 4.2 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 1.7 MiB/s wr, 257 op/s 2023-03-09T14:47:16.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:47:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]': finished 2023-03-09T14:47:16.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:47:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.10", "id": [5, 9]}]': finished 2023-03-09T14:47:16.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:47:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [3, 8]}]': finished 2023-03-09T14:47:16.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:47:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 8]}]': finished 2023-03-09T14:47:16.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:47:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 7]}]': finished 2023-03-09T14:47:16.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:47:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]': finished 2023-03-09T14:47:16.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:47:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10, 5, 1]}]': finished 2023-03-09T14:47:16.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:47:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]': finished 2023-03-09T14:47:16.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:47:15 smithi038 ceph-mon[92422]: osdmap e91: 12 total, 12 up, 12 in 2023-03-09T14:47:16.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:47:16 smithi122 ceph-mon[93869]: osdmap e92: 12 total, 12 up, 12 in 2023-03-09T14:47:16.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:47:16 smithi122 ceph-mon[93869]: osdmap e93: 12 total, 12 up, 12 in 2023-03-09T14:47:16.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:47:16 smithi181 ceph-mon[94189]: osdmap e92: 12 total, 12 up, 12 in 2023-03-09T14:47:16.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:47:16 smithi181 ceph-mon[94189]: osdmap e93: 12 total, 12 up, 12 in 2023-03-09T14:47:17.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:47:16 smithi038 ceph-mon[92422]: osdmap e92: 12 total, 12 up, 12 in 2023-03-09T14:47:17.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:47:16 smithi038 ceph-mon[92422]: osdmap e93: 12 total, 12 up, 12 in 2023-03-09T14:47:17.072 INFO:teuthology.orchestra.run.smithi038.stderr:29900000 of 50000000 tuples (59%) done (elapsed 52.21 s, remaining 35.10 s) 2023-03-09T14:47:17.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:47:17 smithi122 ceph-mon[93869]: pgmap v36: 129 pgs: 8 peering, 121 active+clean; 4.7 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 66 MiB/s wr, 325 op/s 2023-03-09T14:47:17.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:47:17 smithi181 ceph-mon[94189]: pgmap v36: 129 pgs: 8 peering, 121 active+clean; 4.7 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 66 MiB/s wr, 325 op/s 2023-03-09T14:47:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:47:17 smithi038 ceph-mon[92422]: pgmap v36: 129 pgs: 8 peering, 121 active+clean; 4.7 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 66 MiB/s wr, 325 op/s 2023-03-09T14:47:19.360 INFO:teuthology.orchestra.run.smithi038.stderr:30000000 of 50000000 tuples (60%) done (elapsed 54.50 s, remaining 36.33 s) 2023-03-09T14:47:19.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:47:19 smithi122 ceph-mon[93869]: pgmap v38: 129 pgs: 8 peering, 121 active+clean; 4.7 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 86 MiB/s wr, 207 op/s 2023-03-09T14:47:19.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:47:19 smithi181 ceph-mon[94189]: pgmap v38: 129 pgs: 8 peering, 121 active+clean; 4.7 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 86 MiB/s wr, 207 op/s 2023-03-09T14:47:20.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:47:19 smithi038 ceph-mon[92422]: pgmap v38: 129 pgs: 8 peering, 121 active+clean; 4.7 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 86 MiB/s wr, 207 op/s 2023-03-09T14:47:21.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:47:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:47:21.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:47:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:47:21.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:47:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:47:21.853 INFO:teuthology.orchestra.run.smithi038.stderr:30100000 of 50000000 tuples (60%) done (elapsed 56.99 s, remaining 37.68 s) 2023-03-09T14:47:21.938 INFO:teuthology.orchestra.run.smithi038.stderr:30200000 of 50000000 tuples (60%) done (elapsed 57.08 s, remaining 37.42 s) 2023-03-09T14:47:22.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:47:21 smithi038 ceph-mon[92422]: pgmap v39: 129 pgs: 8 peering, 121 active+clean; 4.7 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 86 MiB/s wr, 207 op/s 2023-03-09T14:47:22.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:47:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:47:22.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:47:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:47:22.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:47:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:47:22.046 INFO:teuthology.orchestra.run.smithi038.stderr:30300000 of 50000000 tuples (60%) done (elapsed 57.18 s, remaining 37.18 s) 2023-03-09T14:47:22.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:47:21 smithi122 ceph-mon[93869]: pgmap v39: 129 pgs: 8 peering, 121 active+clean; 4.7 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 86 MiB/s wr, 207 op/s 2023-03-09T14:47:22.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:47:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:47:22.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:47:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:47:22.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:47:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:47:22.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:47:21 smithi181 ceph-mon[94189]: pgmap v39: 129 pgs: 8 peering, 121 active+clean; 4.7 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 86 MiB/s wr, 207 op/s 2023-03-09T14:47:22.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:47:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:47:22.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:47:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:47:22.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:47:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:47:22.790 INFO:teuthology.orchestra.run.smithi038.stderr:30400000 of 50000000 tuples (60%) done (elapsed 57.93 s, remaining 37.35 s) 2023-03-09T14:47:22.881 INFO:teuthology.orchestra.run.smithi038.stderr:30500000 of 50000000 tuples (61%) done (elapsed 58.02 s, remaining 37.09 s) 2023-03-09T14:47:22.995 INFO:teuthology.orchestra.run.smithi038.stderr:30600000 of 50000000 tuples (61%) done (elapsed 58.13 s, remaining 36.86 s) 2023-03-09T14:47:23.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:47:22 smithi038 ceph-mon[92422]: osdmap e94: 12 total, 12 up, 12 in 2023-03-09T14:47:23.093 INFO:teuthology.orchestra.run.smithi038.stderr:30700000 of 50000000 tuples (61%) done (elapsed 58.23 s, remaining 36.61 s) 2023-03-09T14:47:23.174 INFO:teuthology.orchestra.run.smithi038.stderr:30800000 of 50000000 tuples (61%) done (elapsed 58.31 s, remaining 36.35 s) 2023-03-09T14:47:23.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:47:22 smithi122 ceph-mon[93869]: osdmap e94: 12 total, 12 up, 12 in 2023-03-09T14:47:23.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:47:22 smithi181 ceph-mon[94189]: osdmap e94: 12 total, 12 up, 12 in 2023-03-09T14:47:23.272 INFO:teuthology.orchestra.run.smithi038.stderr:30900000 of 50000000 tuples (61%) done (elapsed 58.41 s, remaining 36.10 s) 2023-03-09T14:47:23.363 INFO:teuthology.orchestra.run.smithi038.stderr:31000000 of 50000000 tuples (62%) done (elapsed 58.50 s, remaining 35.86 s) 2023-03-09T14:47:23.475 INFO:teuthology.orchestra.run.smithi038.stderr:31100000 of 50000000 tuples (62%) done (elapsed 58.61 s, remaining 35.62 s) 2023-03-09T14:47:23.581 INFO:teuthology.orchestra.run.smithi038.stderr:31200000 of 50000000 tuples (62%) done (elapsed 58.72 s, remaining 35.38 s) 2023-03-09T14:47:23.683 INFO:teuthology.orchestra.run.smithi038.stderr:31300000 of 50000000 tuples (62%) done (elapsed 58.82 s, remaining 35.14 s) 2023-03-09T14:47:23.786 INFO:teuthology.orchestra.run.smithi038.stderr:31400000 of 50000000 tuples (62%) done (elapsed 58.92 s, remaining 34.90 s) 2023-03-09T14:47:23.892 INFO:teuthology.orchestra.run.smithi038.stderr:31500000 of 50000000 tuples (63%) done (elapsed 59.03 s, remaining 34.67 s) 2023-03-09T14:47:23.990 INFO:teuthology.orchestra.run.smithi038.stderr:31600000 of 50000000 tuples (63%) done (elapsed 59.13 s, remaining 34.43 s) 2023-03-09T14:47:24.102 INFO:teuthology.orchestra.run.smithi038.stderr:31700000 of 50000000 tuples (63%) done (elapsed 59.24 s, remaining 34.20 s) 2023-03-09T14:47:24.211 INFO:teuthology.orchestra.run.smithi038.stderr:31800000 of 50000000 tuples (63%) done (elapsed 59.35 s, remaining 33.97 s) 2023-03-09T14:47:24.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:47:23 smithi122 ceph-mon[93869]: pgmap v41: 129 pgs: 1 active+recovering+undersized+degraded+remapped, 128 active+clean; 4.9 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 34 MiB/s wr, 193 op/s; 2/8356 objects degraded (0.024%); 11/8356 objects misplaced (0.132%); 32 MiB/s, 15 objects/s recovering 2023-03-09T14:47:24.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:47:23 smithi122 ceph-mon[93869]: osdmap e95: 12 total, 12 up, 12 in 2023-03-09T14:47:24.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:47:23 smithi122 ceph-mon[93869]: Health check failed: Degraded data redundancy: 2/8356 objects degraded (0.024%), 1 pg degraded (PG_DEGRADED) 2023-03-09T14:47:24.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:47:23 smithi181 ceph-mon[94189]: pgmap v41: 129 pgs: 1 active+recovering+undersized+degraded+remapped, 128 active+clean; 4.9 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 34 MiB/s wr, 193 op/s; 2/8356 objects degraded (0.024%); 11/8356 objects misplaced (0.132%); 32 MiB/s, 15 objects/s recovering 2023-03-09T14:47:24.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:47:23 smithi181 ceph-mon[94189]: osdmap e95: 12 total, 12 up, 12 in 2023-03-09T14:47:24.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:47:23 smithi181 ceph-mon[94189]: Health check failed: Degraded data redundancy: 2/8356 objects degraded (0.024%), 1 pg degraded (PG_DEGRADED) 2023-03-09T14:47:24.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:47:23 smithi038 ceph-mon[92422]: pgmap v41: 129 pgs: 1 active+recovering+undersized+degraded+remapped, 128 active+clean; 4.9 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 34 MiB/s wr, 193 op/s; 2/8356 objects degraded (0.024%); 11/8356 objects misplaced (0.132%); 32 MiB/s, 15 objects/s recovering 2023-03-09T14:47:24.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:47:23 smithi038 ceph-mon[92422]: osdmap e95: 12 total, 12 up, 12 in 2023-03-09T14:47:24.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:47:23 smithi038 ceph-mon[92422]: Health check failed: Degraded data redundancy: 2/8356 objects degraded (0.024%), 1 pg degraded (PG_DEGRADED) 2023-03-09T14:47:24.324 INFO:teuthology.orchestra.run.smithi038.stderr:31900000 of 50000000 tuples (63%) done (elapsed 59.46 s, remaining 33.74 s) 2023-03-09T14:47:24.426 INFO:teuthology.orchestra.run.smithi038.stderr:32000000 of 50000000 tuples (64%) done (elapsed 59.56 s, remaining 33.50 s) 2023-03-09T14:47:24.496 INFO:teuthology.orchestra.run.smithi038.stderr:32100000 of 50000000 tuples (64%) done (elapsed 59.63 s, remaining 33.25 s) 2023-03-09T14:47:24.604 INFO:teuthology.orchestra.run.smithi038.stderr:32200000 of 50000000 tuples (64%) done (elapsed 59.74 s, remaining 33.03 s) 2023-03-09T14:47:24.706 INFO:teuthology.orchestra.run.smithi038.stderr:32300000 of 50000000 tuples (64%) done (elapsed 59.84 s, remaining 32.79 s) 2023-03-09T14:47:24.780 INFO:teuthology.orchestra.run.smithi038.stderr:32400000 of 50000000 tuples (64%) done (elapsed 59.92 s, remaining 32.55 s) 2023-03-09T14:47:24.913 INFO:teuthology.orchestra.run.smithi038.stderr:32500000 of 50000000 tuples (65%) done (elapsed 60.05 s, remaining 32.34 s) 2023-03-09T14:47:25.013 INFO:teuthology.orchestra.run.smithi038.stderr:32600000 of 50000000 tuples (65%) done (elapsed 60.15 s, remaining 32.11 s) 2023-03-09T14:47:25.081 INFO:teuthology.orchestra.run.smithi038.stderr:32700000 of 50000000 tuples (65%) done (elapsed 60.22 s, remaining 31.86 s) 2023-03-09T14:47:25.191 INFO:teuthology.orchestra.run.smithi038.stderr:32800000 of 50000000 tuples (65%) done (elapsed 60.33 s, remaining 31.64 s) 2023-03-09T14:47:25.285 INFO:teuthology.orchestra.run.smithi038.stderr:32900000 of 50000000 tuples (65%) done (elapsed 60.42 s, remaining 31.41 s) 2023-03-09T14:47:25.381 INFO:teuthology.orchestra.run.smithi038.stderr:33000000 of 50000000 tuples (66%) done (elapsed 60.52 s, remaining 31.18 s) 2023-03-09T14:47:25.483 INFO:teuthology.orchestra.run.smithi038.stderr:33100000 of 50000000 tuples (66%) done (elapsed 60.62 s, remaining 30.95 s) 2023-03-09T14:47:25.586 INFO:teuthology.orchestra.run.smithi038.stderr:33200000 of 50000000 tuples (66%) done (elapsed 60.72 s, remaining 30.73 s) 2023-03-09T14:47:25.672 INFO:teuthology.orchestra.run.smithi038.stderr:33300000 of 50000000 tuples (66%) done (elapsed 60.81 s, remaining 30.50 s) 2023-03-09T14:47:25.791 INFO:teuthology.orchestra.run.smithi038.stderr:33400000 of 50000000 tuples (66%) done (elapsed 60.93 s, remaining 30.28 s) 2023-03-09T14:47:26.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:47:25 smithi122 ceph-mon[93869]: pgmap v43: 129 pgs: 1 active+recovering+undersized+degraded+remapped, 128 active+clean; 4.9 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 30 MiB/s wr, 168 op/s; 2/8356 objects degraded (0.024%); 11/8356 objects misplaced (0.132%); 28 MiB/s, 13 objects/s recovering 2023-03-09T14:47:26.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:47:25 smithi181 ceph-mon[94189]: pgmap v43: 129 pgs: 1 active+recovering+undersized+degraded+remapped, 128 active+clean; 4.9 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 30 MiB/s wr, 168 op/s; 2/8356 objects degraded (0.024%); 11/8356 objects misplaced (0.132%); 28 MiB/s, 13 objects/s recovering 2023-03-09T14:47:26.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:47:25 smithi038 ceph-mon[92422]: pgmap v43: 129 pgs: 1 active+recovering+undersized+degraded+remapped, 128 active+clean; 4.9 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 30 MiB/s wr, 168 op/s; 2/8356 objects degraded (0.024%); 11/8356 objects misplaced (0.132%); 28 MiB/s, 13 objects/s recovering 2023-03-09T14:47:27.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:47:26 smithi122 ceph-mon[93869]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/8356 objects degraded (0.024%), 1 pg degraded) 2023-03-09T14:47:27.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:47:26 smithi122 ceph-mon[93869]: Cluster is now healthy 2023-03-09T14:47:27.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:47:26 smithi181 ceph-mon[94189]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/8356 objects degraded (0.024%), 1 pg degraded) 2023-03-09T14:47:27.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:47:26 smithi181 ceph-mon[94189]: Cluster is now healthy 2023-03-09T14:47:27.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:47:26 smithi038 ceph-mon[92422]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/8356 objects degraded (0.024%), 1 pg degraded) 2023-03-09T14:47:27.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:47:26 smithi038 ceph-mon[92422]: Cluster is now healthy 2023-03-09T14:47:28.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:47:27 smithi122 ceph-mon[93869]: pgmap v44: 129 pgs: 129 active+clean; 5.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 69 MiB/s wr, 232 op/s; 34 MiB/s, 14 objects/s recovering 2023-03-09T14:47:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:47:27 smithi181 ceph-mon[94189]: pgmap v44: 129 pgs: 129 active+clean; 5.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 69 MiB/s wr, 232 op/s; 34 MiB/s, 14 objects/s recovering 2023-03-09T14:47:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:47:27 smithi038 ceph-mon[92422]: pgmap v44: 129 pgs: 129 active+clean; 5.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 69 MiB/s wr, 232 op/s; 34 MiB/s, 14 objects/s recovering 2023-03-09T14:47:28.751 INFO:teuthology.orchestra.run.smithi038.stderr:33500000 of 50000000 tuples (67%) done (elapsed 63.89 s, remaining 31.47 s) 2023-03-09T14:47:30.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:47:29 smithi122 ceph-mon[93869]: pgmap v45: 129 pgs: 129 active+clean; 5.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 69 MiB/s wr, 232 op/s; 34 MiB/s, 14 objects/s recovering 2023-03-09T14:47:30.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:47:29 smithi181 ceph-mon[94189]: pgmap v45: 129 pgs: 129 active+clean; 5.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 69 MiB/s wr, 232 op/s; 34 MiB/s, 14 objects/s recovering 2023-03-09T14:47:30.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:47:29 smithi038 ceph-mon[92422]: pgmap v45: 129 pgs: 129 active+clean; 5.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 69 MiB/s wr, 232 op/s; 34 MiB/s, 14 objects/s recovering 2023-03-09T14:47:32.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:47:31 smithi122 ceph-mon[93869]: pgmap v46: 129 pgs: 129 active+clean; 5.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 36 MiB/s wr, 59 op/s; 5.9 MiB/s, 1 objects/s recovering 2023-03-09T14:47:32.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:47:31 smithi181 ceph-mon[94189]: pgmap v46: 129 pgs: 129 active+clean; 5.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 36 MiB/s wr, 59 op/s; 5.9 MiB/s, 1 objects/s recovering 2023-03-09T14:47:32.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:47:31 smithi038 ceph-mon[92422]: pgmap v46: 129 pgs: 129 active+clean; 5.2 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 36 MiB/s wr, 59 op/s; 5.9 MiB/s, 1 objects/s recovering 2023-03-09T14:47:34.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:47:33 smithi122 ceph-mon[93869]: pgmap v47: 129 pgs: 129 active+clean; 5.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 84 MiB/s wr, 145 op/s; 5.2 MiB/s, 1 objects/s recovering 2023-03-09T14:47:34.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:47:33 smithi181 ceph-mon[94189]: pgmap v47: 129 pgs: 129 active+clean; 5.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 84 MiB/s wr, 145 op/s; 5.2 MiB/s, 1 objects/s recovering 2023-03-09T14:47:34.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:47:33 smithi038 ceph-mon[92422]: pgmap v47: 129 pgs: 129 active+clean; 5.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 84 MiB/s wr, 145 op/s; 5.2 MiB/s, 1 objects/s recovering 2023-03-09T14:47:35.674 INFO:teuthology.orchestra.run.smithi038.stderr:33600000 of 50000000 tuples (67%) done (elapsed 70.81 s, remaining 34.56 s) 2023-03-09T14:47:36.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:47:35 smithi122 ceph-mon[93869]: pgmap v48: 129 pgs: 129 active+clean; 5.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 71 MiB/s wr, 123 op/s; 4.4 MiB/s, 1 objects/s recovering 2023-03-09T14:47:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:47:35 smithi181 ceph-mon[94189]: pgmap v48: 129 pgs: 129 active+clean; 5.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 71 MiB/s wr, 123 op/s; 4.4 MiB/s, 1 objects/s recovering 2023-03-09T14:47:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:47:35 smithi038 ceph-mon[92422]: pgmap v48: 129 pgs: 129 active+clean; 5.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 71 MiB/s wr, 123 op/s; 4.4 MiB/s, 1 objects/s recovering 2023-03-09T14:47:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:47:37 smithi122 ceph-mon[93869]: pgmap v49: 129 pgs: 129 active+clean; 5.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 71 MiB/s wr, 257 op/s; 4.3 MiB/s, 1 objects/s recovering 2023-03-09T14:47:38.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:47:37 smithi181 ceph-mon[94189]: pgmap v49: 129 pgs: 129 active+clean; 5.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 71 MiB/s wr, 257 op/s; 4.3 MiB/s, 1 objects/s recovering 2023-03-09T14:47:38.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:47:37 smithi038 ceph-mon[92422]: pgmap v49: 129 pgs: 129 active+clean; 5.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 71 MiB/s wr, 257 op/s; 4.3 MiB/s, 1 objects/s recovering 2023-03-09T14:47:40.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:47:39 smithi122 ceph-mon[93869]: pgmap v50: 129 pgs: 129 active+clean; 5.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 44 MiB/s wr, 213 op/s 2023-03-09T14:47:40.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:47:39 smithi181 ceph-mon[94189]: pgmap v50: 129 pgs: 129 active+clean; 5.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 44 MiB/s wr, 213 op/s 2023-03-09T14:47:40.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:47:39 smithi038 ceph-mon[92422]: pgmap v50: 129 pgs: 129 active+clean; 5.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 44 MiB/s wr, 213 op/s 2023-03-09T14:47:41.195 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-03-09T14:47:41.347 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.345+0000 7fa90df79700 1 -- 172.21.15.38:0/1461423036 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9080686d0 msgr2=0x7fa908104a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:47:41.348 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.345+0000 7fa90df79700 1 --2- 172.21.15.38:0/1461423036 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9080686d0 0x7fa908104a60 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc0099c0 tx=0x7fa8fc009cd0 comp rx=0 tx=0).stop 2023-03-09T14:47:41.348 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.345+0000 7fa90df79700 1 -- 172.21.15.38:0/1461423036 shutdown_connections 2023-03-09T14:47:41.348 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.345+0000 7fa90df79700 1 --2- 172.21.15.38:0/1461423036 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa908105140 0x7fa90810cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:41.348 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.345+0000 7fa90df79700 1 --2- 172.21.15.38:0/1461423036 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa908067d30 0x7fa908068100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:41.348 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.345+0000 7fa90df79700 1 --2- 172.21.15.38:0/1461423036 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9080686d0 0x7fa908104a60 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:41.348 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.345+0000 7fa90df79700 1 -- 172.21.15.38:0/1461423036 >> 172.21.15.38:0/1461423036 conn(0x7fa908075c60 msgr2=0x7fa908076060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:47:41.348 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.345+0000 7fa90df79700 1 -- 172.21.15.38:0/1461423036 shutdown_connections 2023-03-09T14:47:41.349 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.345+0000 7fa90df79700 1 -- 172.21.15.38:0/1461423036 wait complete. 2023-03-09T14:47:41.350 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.346+0000 7fa90df79700 1 Processor -- start 2023-03-09T14:47:41.351 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.346+0000 7fa90df79700 1 -- start start 2023-03-09T14:47:41.351 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.346+0000 7fa90df79700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa908067d30 0x7fa908196040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:41.351 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.346+0000 7fa90df79700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9080686d0 0x7fa908196580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:41.351 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.346+0000 7fa90df79700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa908105140 0x7fa90819a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:41.351 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.346+0000 7fa90df79700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa90819b020 con 0x7fa9080686d0 2023-03-09T14:47:41.352 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.346+0000 7fa90df79700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa90819b190 con 0x7fa908105140 2023-03-09T14:47:41.352 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.346+0000 7fa90df79700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa90819b350 con 0x7fa908067d30 2023-03-09T14:47:41.352 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.346+0000 7fa906ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9080686d0 0x7fa908196580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:41.352 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.346+0000 7fa9077fe700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa908067d30 0x7fa908196040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:41.352 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.346+0000 7fa9077fe700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa908067d30 0x7fa908196040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.38:47472/0 (socket says 172.21.15.38:47472) 2023-03-09T14:47:41.352 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.346+0000 7fa906ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9080686d0 0x7fa908196580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39776/0 (socket says 172.21.15.38:39776) 2023-03-09T14:47:41.352 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.346+0000 7fa9077fe700 1 -- 172.21.15.38:0/3057953204 learned_addr learned my addr 172.21.15.38:0/3057953204 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:47:41.353 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.346+0000 7fa907fff700 1 --2- 172.21.15.38:0/3057953204 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa908105140 0x7fa90819a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:41.353 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.347+0000 7fa9077fe700 1 -- 172.21.15.38:0/3057953204 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa908105140 msgr2=0x7fa90819a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:47:41.353 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.347+0000 7fa9077fe700 1 --2- 172.21.15.38:0/3057953204 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa908105140 0x7fa90819a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:41.353 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.347+0000 7fa9077fe700 1 -- 172.21.15.38:0/3057953204 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9080686d0 msgr2=0x7fa908196580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:47:41.353 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.347+0000 7fa9077fe700 1 --2- 172.21.15.38:0/3057953204 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9080686d0 0x7fa908196580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:41.353 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.347+0000 7fa9077fe700 1 -- 172.21.15.38:0/3057953204 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8fc0095f0 con 0x7fa908067d30 2023-03-09T14:47:41.353 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.347+0000 7fa906ffd700 1 --2- 172.21.15.38:0/3057953204 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9080686d0 0x7fa908196580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:47:41.354 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.347+0000 7fa9077fe700 1 --2- 172.21.15.38:0/3057953204 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa908067d30 0x7fa908196040 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa8f800e9a0 tx=0x7fa8f800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:47:41.354 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.347+0000 7fa904ff9700 1 -- 172.21.15.38:0/3057953204 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8f8009790 con 0x7fa908067d30 2023-03-09T14:47:41.354 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.347+0000 7fa90df79700 1 -- 172.21.15.38:0/3057953204 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa90810fec0 con 0x7fa908067d30 2023-03-09T14:47:41.354 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.347+0000 7fa904ff9700 1 -- 172.21.15.38:0/3057953204 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa8f8010430 con 0x7fa908067d30 2023-03-09T14:47:41.354 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.347+0000 7fa90df79700 1 -- 172.21.15.38:0/3057953204 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa90810fd30 con 0x7fa908067d30 2023-03-09T14:47:41.354 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.347+0000 7fa904ff9700 1 -- 172.21.15.38:0/3057953204 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8f8018680 con 0x7fa908067d30 2023-03-09T14:47:41.356 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.350+0000 7fa904ff9700 1 -- 172.21.15.38:0/3057953204 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 24) v1 ==== 133375+0+0 (secure 0 0 0) 0x7fa8f800c040 con 0x7fa908067d30 2023-03-09T14:47:41.356 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.351+0000 7fa904ff9700 1 --2- 172.21.15.38:0/3057953204 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fa8f009f530 0x7fa8f00a19e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:41.356 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.351+0000 7fa904ff9700 1 -- 172.21.15.38:0/3057953204 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9489+0+0 (secure 0 0 0) 0x7fa8f8014070 con 0x7fa908067d30 2023-03-09T14:47:41.356 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.351+0000 7fa906ffd700 1 --2- 172.21.15.38:0/3057953204 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fa8f009f530 0x7fa8f00a19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:41.356 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.352+0000 7fa91a102b80 1 Processor -- start 2023-03-09T14:47:41.357 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.352+0000 7fa91a102b80 1 -- start start 2023-03-09T14:47:41.357 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.352+0000 7fa906ffd700 1 --2- 172.21.15.38:0/3057953204 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fa8f009f530 0x7fa8f00a19e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc0098e0 tx=0x7fa8fc01a040 comp rx=0 tx=0).ready entity=mgr.15558 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:47:41.357 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.352+0000 7fa91a102b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56509be152c0 0x56509be15690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:41.357 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.352+0000 7fa91a102b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56509be15bd0 0x56509be1e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:41.357 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.352+0000 7fa91a102b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56509be1e870 0x56509be20c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:41.357 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.352+0000 7fa91a102b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x56509bcf5da0 con 0x56509be152c0 2023-03-09T14:47:41.358 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.352+0000 7fa91a102b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x56509bd64780 con 0x56509be15bd0 2023-03-09T14:47:41.358 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.352+0000 7fa91a102b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x56509bd5dcb0 con 0x56509be1e870 2023-03-09T14:47:41.358 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.352+0000 7fa9077fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56509be152c0 0x56509be15690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:41.358 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.352+0000 7fa9077fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56509be152c0 0x56509be15690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39782/0 (socket says 172.21.15.38:39782) 2023-03-09T14:47:41.358 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.352+0000 7fa9077fe700 1 -- 172.21.15.38:0/4232125766 learned_addr learned my addr 172.21.15.38:0/4232125766 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:47:41.358 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.352+0000 7fa906ffd700 1 --2- 172.21.15.38:0/4232125766 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56509be15bd0 0x56509be1e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:41.358 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.352+0000 7fa907fff700 1 --2- 172.21.15.38:0/4232125766 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56509be1e870 0x56509be20c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:41.358 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.352+0000 7fa9077fe700 1 -- 172.21.15.38:0/4232125766 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56509be1e870 msgr2=0x56509be20c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:47:41.359 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.352+0000 7fa9077fe700 1 --2- 172.21.15.38:0/4232125766 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56509be1e870 0x56509be20c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:41.359 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.352+0000 7fa9077fe700 1 -- 172.21.15.38:0/4232125766 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56509be15bd0 msgr2=0x56509be1e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:47:41.359 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.352+0000 7fa9077fe700 1 --2- 172.21.15.38:0/4232125766 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56509be15bd0 0x56509be1e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:41.359 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.352+0000 7fa9077fe700 1 -- 172.21.15.38:0/4232125766 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56509bd24190 con 0x56509be152c0 2023-03-09T14:47:41.359 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.352+0000 7fa907fff700 1 --2- 172.21.15.38:0/4232125766 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56509be1e870 0x56509be20c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-09T14:47:41.359 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.353+0000 7fa9077fe700 1 --2- 172.21.15.38:0/4232125766 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56509be152c0 0x56509be15690 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fa8f8010c30 tx=0x7fa8f8097d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32cda175e910c585 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:47:41.359 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.353+0000 7fa8edffb700 1 -- 172.21.15.38:0/4232125766 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8f8096310 con 0x56509be152c0 2023-03-09T14:47:41.360 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.353+0000 7fa8edffb700 1 -- 172.21.15.38:0/4232125766 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa8f8010880 con 0x56509be152c0 2023-03-09T14:47:41.360 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.353+0000 7fa8edffb700 1 -- 172.21.15.38:0/4232125766 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8f8097600 con 0x56509be152c0 2023-03-09T14:47:41.363 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.354+0000 7fa91a102b80 1 -- 172.21.15.38:0/4232125766 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56509be152c0 msgr2=0x56509be15690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:47:41.363 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.354+0000 7fa91a102b80 1 --2- 172.21.15.38:0/4232125766 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56509be152c0 0x56509be15690 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fa8f8010c30 tx=0x7fa8f8097d80 comp rx=0 tx=0).stop 2023-03-09T14:47:41.363 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.354+0000 7fa91a102b80 1 -- 172.21.15.38:0/4232125766 shutdown_connections 2023-03-09T14:47:41.363 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.354+0000 7fa91a102b80 1 --2- 172.21.15.38:0/4232125766 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56509be152c0 0x56509be15690 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:41.363 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.354+0000 7fa91a102b80 1 --2- 172.21.15.38:0/4232125766 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56509be1e870 0x56509be20c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:41.363 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.354+0000 7fa91a102b80 1 --2- 172.21.15.38:0/4232125766 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56509be15bd0 0x56509be1e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:41.363 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.354+0000 7fa91a102b80 1 -- 172.21.15.38:0/4232125766 >> 172.21.15.38:0/4232125766 conn(0x56509be065a0 msgr2=0x56509be06970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:47:41.364 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.354+0000 7fa91a102b80 1 -- 172.21.15.38:0/4232125766 shutdown_connections 2023-03-09T14:47:41.364 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.354+0000 7fa91a102b80 1 -- 172.21.15.38:0/4232125766 wait complete. 2023-03-09T14:47:41.364 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.354+0000 7fa91a102b80 1 -- start start 2023-03-09T14:47:41.364 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.355+0000 7fa91a102b80 1 Processor -- start 2023-03-09T14:47:41.364 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.355+0000 7fa91a102b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56509be152c0 0x56509be14ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:41.364 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.355+0000 7fa91a102b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56509be15bd0 0x56509be11f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:41.364 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.355+0000 7fa91a102b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56509be1e870 0x56509be12580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:41.365 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.355+0000 7fa91a102b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x56509bbe1920 con 0x56509be152c0 2023-03-09T14:47:41.365 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.355+0000 7fa91a102b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x56509be12cf0 con 0x56509be1e870 2023-03-09T14:47:41.365 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.355+0000 7fa91a102b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x56509be1d2b0 con 0x56509be15bd0 2023-03-09T14:47:41.365 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.355+0000 7fa91a102b80 10 client.0 _ll_register_callbacks cb 0x56509bd03520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:47:41.365 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.355+0000 7fa9077fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56509be152c0 0x56509be14ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:41.365 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.355+0000 7fa907fff700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56509be1e870 0x56509be12580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:41.365 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.355+0000 7fa9077fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56509be152c0 0x56509be14ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39790/0 (socket says 172.21.15.38:39790) 2023-03-09T14:47:41.365 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.355+0000 7fa9077fe700 1 -- 172.21.15.38:0/1501294100 learned_addr learned my addr 172.21.15.38:0/1501294100 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:47:41.366 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.356+0000 7fa906ffd700 1 --2- 172.21.15.38:0/1501294100 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56509be15bd0 0x56509be11f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:41.366 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.356+0000 7fa9077fe700 1 -- 172.21.15.38:0/1501294100 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56509be15bd0 msgr2=0x56509be11f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:47:41.366 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.356+0000 7fa9077fe700 1 --2- 172.21.15.38:0/1501294100 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56509be15bd0 0x56509be11f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:41.366 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.356+0000 7fa9077fe700 1 -- 172.21.15.38:0/1501294100 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56509be1e870 msgr2=0x56509be12580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:47:41.366 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.356+0000 7fa9077fe700 1 --2- 172.21.15.38:0/1501294100 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56509be1e870 0x56509be12580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:41.366 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.356+0000 7fa9077fe700 1 -- 172.21.15.38:0/1501294100 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa90810fd30 con 0x56509be152c0 2023-03-09T14:47:41.366 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.356+0000 7fa9077fe700 1 --2- 172.21.15.38:0/1501294100 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56509be152c0 0x56509be14ca0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fa8f8000c00 tx=0x7fa8f80a0480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e40f2ebc31517269 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:47:41.367 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.356+0000 7fa8e6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2023-03-09T14:47:41.367 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.356+0000 7fa8e6ffd700 1 -- 172.21.15.38:0/1501294100 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8f8097600 con 0x56509be152c0 2023-03-09T14:47:41.367 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.356+0000 7fa8e6ffd700 1 -- 172.21.15.38:0/1501294100 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9489+0+0 (secure 0 0 0) 0x7fa8f80c3020 con 0x56509be152c0 2023-03-09T14:47:41.367 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.356+0000 7fa91a102b80 1 -- 172.21.15.38:0/1501294100 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56509bd2b190 con 0x56509be152c0 2023-03-09T14:47:41.367 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.356+0000 7fa8e6ffd700 1 -- 172.21.15.38:0/1501294100 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa8f8097760 con 0x56509be152c0 2023-03-09T14:47:41.367 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.356+0000 7fa91a102b80 10 client.15879 fetch_fsmap learned FSMap version 25 2023-03-09T14:47:41.367 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.356+0000 7fa91a102b80 1 -- 172.21.15.38:0/1501294100 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x56509bb74c60 con 0x56509be152c0 2023-03-09T14:47:41.368 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.357+0000 7fa8e6ffd700 1 -- 172.21.15.38:0/1501294100 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56509bd2b190 con 0x56509be152c0 2023-03-09T14:47:41.368 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.357+0000 7fa8e6ffd700 1 -- 172.21.15.38:0/1501294100 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 25) v1 ==== 7478+0+0 (secure 0 0 0) 0x7fa8f809ff70 con 0x56509be152c0 2023-03-09T14:47:41.368 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.357+0000 7fa91a102b80 10 client.15879 fetch_fsmap finished waiting for FSMap version 25 2023-03-09T14:47:41.368 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.357+0000 7fa91a102b80 10 client.15879 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T14:47:41.368 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.357+0000 7fa91a102b80 20 client.15879 populate_metadata read hostname 'smithi038' 2023-03-09T14:47:41.368 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.357+0000 7fa91a102b80 1 --2- 172.21.15.38:0/1501294100 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x56509be369a0 0x56509be38d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:41.368 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.357+0000 7fa91a102b80 4 client.15879 mds_command: new command op to 24530 tid=0[{"prefix": "get_command_descriptions"}] 2023-03-09T14:47:41.368 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.357+0000 7fa91a102b80 1 -- 172.21.15.38:0/1501294100 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56509bbe1f60 con 0x56509be369a0 2023-03-09T14:47:41.369 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.357+0000 7fa906ffd700 1 --2- 172.21.15.38:0/1501294100 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x56509be369a0 0x56509be38d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:41.369 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.357+0000 7fa8e6ffd700 10 client.15879 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:47:41.369 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.357+0000 7fa906ffd700 1 --2- 172.21.15.38:0/1501294100 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x56509be369a0 0x56509be38d80 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c1b95684d6dc2700 server_cookie=b5c97cd92723e962 in_seq=0 out_seq=0 2023-03-09T14:47:41.369 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.358+0000 7fa8e6ffd700 1 -- 172.21.15.38:0/1501294100 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18278 (crc 0 0 0) 0x56509bbe1f60 con 0x56509be369a0 2023-03-09T14:47:41.369 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.358+0000 7fa8e6ffd700 10 client.15879 handle_command_reply: tid=0 2023-03-09T14:47:41.369 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.358+0000 7fa91a102b80 1 client.15879 shutdown 2023-03-09T14:47:41.369 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.358+0000 7fa91a102b80 1 -- 172.21.15.38:0/1501294100 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56509be152c0 msgr2=0x56509be14ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:47:41.370 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.358+0000 7fa91a102b80 1 --2- 172.21.15.38:0/1501294100 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56509be152c0 0x56509be14ca0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fa8f8000c00 tx=0x7fa8f80a0480 comp rx=0 tx=0).stop 2023-03-09T14:47:41.370 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.358+0000 7fa91a102b80 1 -- 172.21.15.38:0/1501294100 shutdown_connections 2023-03-09T14:47:41.370 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.358+0000 7fa91a102b80 1 --2- 172.21.15.38:0/1501294100 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x56509be369a0 0x56509be38d80 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:41.370 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.358+0000 7fa91a102b80 1 --2- 172.21.15.38:0/1501294100 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56509be152c0 0x56509be14ca0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:41.370 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.358+0000 7fa8e6ffd700 0 client.15879 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:47:41.370 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.358+0000 7fa91a102b80 1 --2- 172.21.15.38:0/1501294100 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56509be1e870 0x56509be12580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:41.370 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.358+0000 7fa91a102b80 1 --2- 172.21.15.38:0/1501294100 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56509be15bd0 0x56509be11f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:41.370 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.358+0000 7fa91a102b80 1 -- 172.21.15.38:0/1501294100 >> 172.21.15.38:0/1501294100 conn(0x56509be065a0 msgr2=0x56509be05c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:47:41.371 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.358+0000 7fa91a102b80 1 -- 172.21.15.38:0/1501294100 shutdown_connections 2023-03-09T14:47:41.371 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.358+0000 7fa91a102b80 1 -- 172.21.15.38:0/1501294100 wait complete. 2023-03-09T14:47:41.371 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.359+0000 7fa91a102b80 20 client.15879 trim_cache size 0 max 16384 2023-03-09T14:47:41.382 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.380+0000 7fa91a102b80 1 Processor -- start 2023-03-09T14:47:41.382 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.380+0000 7fa91a102b80 1 -- start start 2023-03-09T14:47:41.383 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.380+0000 7fa91a102b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56509be152c0 0x56509be1a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:41.383 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.380+0000 7fa91a102b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56509be15bd0 0x56509be1d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:41.384 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.380+0000 7fa91a102b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56509be1e870 0x56509be0d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:41.384 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.380+0000 7fa91a102b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x56509be0c310 con 0x56509be15bd0 2023-03-09T14:47:41.384 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.380+0000 7fa91a102b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x56509be0c640 con 0x56509be1e870 2023-03-09T14:47:41.384 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.380+0000 7fa91a102b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x56509be104e0 con 0x56509be152c0 2023-03-09T14:47:41.384 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.380+0000 7fa906ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56509be15bd0 0x56509be1d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:41.384 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.380+0000 7fa9077fe700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56509be152c0 0x56509be1a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:41.384 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.380+0000 7fa9077fe700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56509be152c0 0x56509be1a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.38:47494/0 (socket says 172.21.15.38:47494) 2023-03-09T14:47:41.385 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.380+0000 7fa906ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56509be15bd0 0x56509be1d750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39798/0 (socket says 172.21.15.38:39798) 2023-03-09T14:47:41.385 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.380+0000 7fa9077fe700 1 -- 172.21.15.38:0/1350320095 learned_addr learned my addr 172.21.15.38:0/1350320095 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:47:41.385 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.380+0000 7fa907fff700 1 --2- 172.21.15.38:0/1350320095 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56509be1e870 0x56509be0d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:41.385 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.380+0000 7fa9077fe700 1 -- 172.21.15.38:0/1350320095 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56509be1e870 msgr2=0x56509be0d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:47:41.385 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.380+0000 7fa9077fe700 1 --2- 172.21.15.38:0/1350320095 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56509be1e870 0x56509be0d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:41.385 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.380+0000 7fa9077fe700 1 -- 172.21.15.38:0/1350320095 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56509be15bd0 msgr2=0x56509be1d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:47:41.385 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.380+0000 7fa9077fe700 1 --2- 172.21.15.38:0/1350320095 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56509be15bd0 0x56509be1d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:41.385 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.381+0000 7fa9077fe700 1 -- 172.21.15.38:0/1350320095 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56509bb8bde0 con 0x56509be152c0 2023-03-09T14:47:41.386 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.381+0000 7fa9077fe700 1 --2- 172.21.15.38:0/1350320095 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56509be152c0 0x56509be1a230 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa8f8097e10 tx=0x7fa8f8097d80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e9ef9dbf3d197b8 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:47:41.386 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.381+0000 7fa8ee7fc700 1 -- 172.21.15.38:0/1350320095 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8f80ca430 con 0x56509be152c0 2023-03-09T14:47:41.386 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.381+0000 7fa8ee7fc700 1 -- 172.21.15.38:0/1350320095 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa8f8098110 con 0x56509be152c0 2023-03-09T14:47:41.386 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.381+0000 7fa8ee7fc700 1 -- 172.21.15.38:0/1350320095 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8f8097860 con 0x56509be152c0 2023-03-09T14:47:41.389 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.382+0000 7fa91a102b80 1 -- 172.21.15.38:0/1350320095 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56509be152c0 msgr2=0x56509be1a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:47:41.389 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.382+0000 7fa91a102b80 1 --2- 172.21.15.38:0/1350320095 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56509be152c0 0x56509be1a230 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa8f8097e10 tx=0x7fa8f8097d80 comp rx=0 tx=0).stop 2023-03-09T14:47:41.389 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.382+0000 7fa91a102b80 1 -- 172.21.15.38:0/1350320095 shutdown_connections 2023-03-09T14:47:41.389 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.382+0000 7fa91a102b80 1 --2- 172.21.15.38:0/1350320095 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56509be1e870 0x56509be0d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:41.389 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.382+0000 7fa91a102b80 1 --2- 172.21.15.38:0/1350320095 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56509be152c0 0x56509be1a230 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:41.390 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.382+0000 7fa91a102b80 1 --2- 172.21.15.38:0/1350320095 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56509be15bd0 0x56509be1d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:41.390 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.382+0000 7fa91a102b80 1 -- 172.21.15.38:0/1350320095 >> 172.21.15.38:0/1350320095 conn(0x56509be065a0 msgr2=0x56509be05c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:47:41.390 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.382+0000 7fa91a102b80 1 -- 172.21.15.38:0/1350320095 shutdown_connections 2023-03-09T14:47:41.390 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.382+0000 7fa91a102b80 1 -- 172.21.15.38:0/1350320095 wait complete. 2023-03-09T14:47:41.390 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.382+0000 7fa91a102b80 1 -- start start 2023-03-09T14:47:41.390 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.383+0000 7fa91a102b80 1 Processor -- start 2023-03-09T14:47:41.390 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.383+0000 7fa91a102b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56509be152c0 0x56509be18850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:41.391 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.383+0000 7fa91a102b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56509be15bd0 0x56509be18d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:41.391 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.383+0000 7fa91a102b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56509be1e870 0x56509be192d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:41.391 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.383+0000 7fa91a102b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x56509be8a760 con 0x56509be15bd0 2023-03-09T14:47:41.391 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.383+0000 7fa91a102b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x56509be8a8a0 con 0x56509be1e870 2023-03-09T14:47:41.391 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.383+0000 7fa91a102b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x56509be8a9e0 con 0x56509be152c0 2023-03-09T14:47:41.391 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.383+0000 7fa91a102b80 10 client.0 _ll_register_callbacks cb 0x56509bd03520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:47:41.391 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.383+0000 7fa907fff700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56509be1e870 0x56509be192d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:41.391 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.383+0000 7fa9077fe700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56509be152c0 0x56509be18850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:41.392 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.383+0000 7fa907fff700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56509be1e870 0x56509be192d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.38:48996/0 (socket says 172.21.15.38:48996) 2023-03-09T14:47:41.392 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.383+0000 7fa907fff700 1 -- 172.21.15.38:0/2442213612 learned_addr learned my addr 172.21.15.38:0/2442213612 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:47:41.392 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.383+0000 7fa906ffd700 1 --2- 172.21.15.38:0/2442213612 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56509be15bd0 0x56509be18d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:41.392 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.384+0000 7fa9077fe700 1 -- 172.21.15.38:0/2442213612 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56509be1e870 msgr2=0x56509be192d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:47:41.392 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.384+0000 7fa9077fe700 1 --2- 172.21.15.38:0/2442213612 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56509be1e870 0x56509be192d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:41.392 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.384+0000 7fa9077fe700 1 -- 172.21.15.38:0/2442213612 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56509be15bd0 msgr2=0x56509be18d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:47:41.392 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.384+0000 7fa9077fe700 1 --2- 172.21.15.38:0/2442213612 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56509be15bd0 0x56509be18d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:41.393 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.384+0000 7fa9077fe700 1 -- 172.21.15.38:0/2442213612 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56509bb74c60 con 0x56509be152c0 2023-03-09T14:47:41.393 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.384+0000 7fa907fff700 1 --2- 172.21.15.38:0/2442213612 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56509be1e870 0x56509be192d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:47:41.393 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.384+0000 7fa9077fe700 1 --2- 172.21.15.38:0/2442213612 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56509be152c0 0x56509be18850 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa8f80980a0 tx=0x7fa8f8010c80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=99eb3d585e666e82 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:47:41.393 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.384+0000 7fa8e6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2023-03-09T14:47:41.393 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.384+0000 7fa8e6ffd700 1 -- 172.21.15.38:0/2442213612 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8f80a0120 con 0x56509be152c0 2023-03-09T14:47:41.393 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.384+0000 7fa8e6ffd700 1 -- 172.21.15.38:0/2442213612 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9489+0+0 (secure 0 0 0) 0x7fa8f80c3070 con 0x56509be152c0 2023-03-09T14:47:41.393 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.384+0000 7fa91a102b80 1 -- 172.21.15.38:0/2442213612 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56509bd1b1b0 con 0x56509be152c0 2023-03-09T14:47:41.394 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.384+0000 7fa8e6ffd700 1 -- 172.21.15.38:0/2442213612 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa8f80c7e00 con 0x56509be152c0 2023-03-09T14:47:41.394 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.384+0000 7fa91a102b80 10 client.25486 fetch_fsmap learned FSMap version 25 2023-03-09T14:47:41.394 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.384+0000 7fa91a102b80 1 -- 172.21.15.38:0/2442213612 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x56509bb8a2b0 con 0x56509be152c0 2023-03-09T14:47:41.394 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.385+0000 7fa8e6ffd700 1 -- 172.21.15.38:0/2442213612 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56509bd1b1b0 con 0x56509be152c0 2023-03-09T14:47:41.394 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.385+0000 7fa8e6ffd700 1 -- 172.21.15.38:0/2442213612 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 25) v1 ==== 7478+0+0 (secure 0 0 0) 0x7fa8f8097100 con 0x56509be152c0 2023-03-09T14:47:41.394 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.385+0000 7fa91a102b80 10 client.25486 fetch_fsmap finished waiting for FSMap version 25 2023-03-09T14:47:41.394 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.385+0000 7fa91a102b80 10 client.25486 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T14:47:41.394 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.385+0000 7fa91a102b80 20 client.25486 populate_metadata read hostname 'smithi038' 2023-03-09T14:47:41.395 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.385+0000 7fa91a102b80 1 --2- 172.21.15.38:0/2442213612 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x56509be8bbc0 0x56509be8dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:47:41.395 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.385+0000 7fa91a102b80 4 client.25486 mds_command: new command op to 24530 tid=0[{"prefix": "scrub status"}] 2023-03-09T14:47:41.395 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.385+0000 7fa91a102b80 1 -- 172.21.15.38:0/2442213612 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x56509bd051c0 con 0x56509be8bbc0 2023-03-09T14:47:41.395 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.385+0000 7fa906ffd700 1 --2- 172.21.15.38:0/2442213612 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x56509be8bbc0 0x56509be8dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:47:41.395 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.385+0000 7fa8e6ffd700 10 client.25486 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:47:41.395 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.385+0000 7fa906ffd700 1 --2- 172.21.15.38:0/2442213612 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x56509be8bbc0 0x56509be8dfa0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5381f801257b8f8b server_cookie=63064c8458fb57f9 in_seq=0 out_seq=0 2023-03-09T14:47:41.395 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.386+0000 7fa8e6ffd700 1 -- 172.21.15.38:0/2442213612 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+265 (crc 0 0 0) 0x56509bd051c0 con 0x56509be8bbc0 2023-03-09T14:47:41.396 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.386+0000 7fa8e6ffd700 10 client.25486 handle_command_reply: tid=0 2023-03-09T14:47:41.396 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.386+0000 7fa91a102b80 1 client.25486 shutdown 2023-03-09T14:47:41.396 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.386+0000 7fa91a102b80 1 -- 172.21.15.38:0/2442213612 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56509be152c0 msgr2=0x56509be18850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:47:41.396 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.386+0000 7fa91a102b80 1 --2- 172.21.15.38:0/2442213612 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56509be152c0 0x56509be18850 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa8f80980a0 tx=0x7fa8f8010c80 comp rx=0 tx=0).stop 2023-03-09T14:47:41.396 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.386+0000 7fa91a102b80 1 -- 172.21.15.38:0/2442213612 shutdown_connections 2023-03-09T14:47:41.396 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.386+0000 7fa91a102b80 1 --2- 172.21.15.38:0/2442213612 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x56509be8bbc0 0x56509be8dfa0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:41.396 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.386+0000 7fa91a102b80 1 --2- 172.21.15.38:0/2442213612 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56509be1e870 0x56509be192d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:41.397 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.386+0000 7fa91a102b80 1 --2- 172.21.15.38:0/2442213612 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56509be152c0 0x56509be18850 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:41.397 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.386+0000 7fa91a102b80 1 --2- 172.21.15.38:0/2442213612 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56509be15bd0 0x56509be18d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:41.397 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.386+0000 7fa8e6ffd700 0 client.25486 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:47:41.397 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.386+0000 7fa91a102b80 1 -- 172.21.15.38:0/2442213612 >> 172.21.15.38:0/2442213612 conn(0x56509be065a0 msgr2=0x56509be1d750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:47:41.397 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.386+0000 7fa91a102b80 1 -- 172.21.15.38:0/2442213612 shutdown_connections 2023-03-09T14:47:41.397 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.386+0000 7fa91a102b80 1 -- 172.21.15.38:0/2442213612 wait complete. 2023-03-09T14:47:41.397 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.386+0000 7fa91a102b80 20 client.25486 trim_cache size 0 max 16384 2023-03-09T14:47:41.398 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2023-03-09T14:47:41.398 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "scrub active (3 inodes in the stack)", 2023-03-09T14:47:41.398 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": { 2023-03-09T14:47:41.398 INFO:teuthology.orchestra.run.smithi038.stdout: "a14659f4-d7ea-4c5a-b15b-6138af3f909f": { 2023-03-09T14:47:41.398 INFO:teuthology.orchestra.run.smithi038.stdout: "path": "/", 2023-03-09T14:47:41.398 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "a14659f4-d7ea-4c5a-b15b-6138af3f909f", 2023-03-09T14:47:41.399 INFO:teuthology.orchestra.run.smithi038.stdout: "options": "recursive,force" 2023-03-09T14:47:41.399 INFO:teuthology.orchestra.run.smithi038.stdout: } 2023-03-09T14:47:41.399 INFO:teuthology.orchestra.run.smithi038.stdout: } 2023-03-09T14:47:41.399 INFO:teuthology.orchestra.run.smithi038.stdout:} 2023-03-09T14:47:41.400 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.387+0000 7fa90df79700 1 -- 172.21.15.38:0/3057953204 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fa8f009f530 msgr2=0x7fa8f00a19e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:47:41.400 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.387+0000 7fa90df79700 1 --2- 172.21.15.38:0/3057953204 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fa8f009f530 0x7fa8f00a19e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc0098e0 tx=0x7fa8fc01a040 comp rx=0 tx=0).stop 2023-03-09T14:47:41.400 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.387+0000 7fa90df79700 1 -- 172.21.15.38:0/3057953204 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa908067d30 msgr2=0x7fa908196040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:47:41.400 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.387+0000 7fa90df79700 1 --2- 172.21.15.38:0/3057953204 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa908067d30 0x7fa908196040 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa8f800e9a0 tx=0x7fa8f800ecb0 comp rx=0 tx=0).stop 2023-03-09T14:47:41.400 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.387+0000 7fa90df79700 1 -- 172.21.15.38:0/3057953204 shutdown_connections 2023-03-09T14:47:41.400 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.387+0000 7fa90df79700 1 --2- 172.21.15.38:0/3057953204 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fa8f009f530 0x7fa8f00a19e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:41.401 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.387+0000 7fa90df79700 1 --2- 172.21.15.38:0/3057953204 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa908105140 0x7fa90819a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:41.401 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.387+0000 7fa90df79700 1 --2- 172.21.15.38:0/3057953204 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa908067d30 0x7fa908196040 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:41.401 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.387+0000 7fa90df79700 1 --2- 172.21.15.38:0/3057953204 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa9080686d0 0x7fa908196580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:47:41.401 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.387+0000 7fa90df79700 1 -- 172.21.15.38:0/3057953204 >> 172.21.15.38:0/3057953204 conn(0x7fa908075c60 msgr2=0x7fa9080fc940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:47:41.401 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.387+0000 7fa90df79700 1 -- 172.21.15.38:0/3057953204 shutdown_connections 2023-03-09T14:47:41.401 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:47:41.387+0000 7fa90df79700 1 -- 172.21.15.38:0/3057953204 wait complete. 2023-03-09T14:47:41.403 INFO:tasks.cephfs.filesystem:scrub status for tag:a14659f4-d7ea-4c5a-b15b-6138af3f909f - {'path': '/', 'tag': 'a14659f4-d7ea-4c5a-b15b-6138af3f909f', 'options': 'recursive,force'} 2023-03-09T14:47:42.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:47:41 smithi122 ceph-mon[93869]: pgmap v51: 129 pgs: 129 active+clean; 5.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 44 MiB/s wr, 213 op/s 2023-03-09T14:47:42.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:47:41 smithi181 ceph-mon[94189]: pgmap v51: 129 pgs: 129 active+clean; 5.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 44 MiB/s wr, 213 op/s 2023-03-09T14:47:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:47:41 smithi038 ceph-mon[92422]: pgmap v51: 129 pgs: 129 active+clean; 5.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 44 MiB/s wr, 213 op/s 2023-03-09T14:47:42.425 INFO:teuthology.orchestra.run.smithi038.stderr:33700000 of 50000000 tuples (67%) done (elapsed 77.56 s, remaining 37.52 s) 2023-03-09T14:47:42.505 INFO:teuthology.orchestra.run.smithi038.stderr:33800000 of 50000000 tuples (67%) done (elapsed 77.64 s, remaining 37.21 s) 2023-03-09T14:47:42.640 INFO:teuthology.orchestra.run.smithi038.stderr:33900000 of 50000000 tuples (67%) done (elapsed 77.78 s, remaining 36.94 s) 2023-03-09T14:47:42.746 INFO:teuthology.orchestra.run.smithi038.stderr:34000000 of 50000000 tuples (68%) done (elapsed 77.88 s, remaining 36.65 s) 2023-03-09T14:47:42.824 INFO:teuthology.orchestra.run.smithi038.stderr:34100000 of 50000000 tuples (68%) done (elapsed 77.96 s, remaining 36.35 s) 2023-03-09T14:47:42.933 INFO:teuthology.orchestra.run.smithi038.stderr:34200000 of 50000000 tuples (68%) done (elapsed 78.07 s, remaining 36.07 s) 2023-03-09T14:47:43.027 INFO:teuthology.orchestra.run.smithi038.stderr:34300000 of 50000000 tuples (68%) done (elapsed 78.17 s, remaining 35.78 s) 2023-03-09T14:47:43.129 INFO:teuthology.orchestra.run.smithi038.stderr:34400000 of 50000000 tuples (68%) done (elapsed 78.27 s, remaining 35.49 s) 2023-03-09T14:47:43.226 INFO:teuthology.orchestra.run.smithi038.stderr:34500000 of 50000000 tuples (69%) done (elapsed 78.36 s, remaining 35.21 s) 2023-03-09T14:47:43.335 INFO:teuthology.orchestra.run.smithi038.stderr:34600000 of 50000000 tuples (69%) done (elapsed 78.47 s, remaining 34.93 s) 2023-03-09T14:47:43.429 INFO:teuthology.orchestra.run.smithi038.stderr:34700000 of 50000000 tuples (69%) done (elapsed 78.57 s, remaining 34.64 s) 2023-03-09T14:47:43.536 INFO:teuthology.orchestra.run.smithi038.stderr:34800000 of 50000000 tuples (69%) done (elapsed 78.67 s, remaining 34.36 s) 2023-03-09T14:47:43.605 INFO:teuthology.orchestra.run.smithi038.stderr:34900000 of 50000000 tuples (69%) done (elapsed 78.74 s, remaining 34.07 s) 2023-03-09T14:47:43.722 INFO:teuthology.orchestra.run.smithi038.stderr:35000000 of 50000000 tuples (70%) done (elapsed 78.86 s, remaining 33.80 s) 2023-03-09T14:47:43.824 INFO:teuthology.orchestra.run.smithi038.stderr:35100000 of 50000000 tuples (70%) done (elapsed 78.96 s, remaining 33.52 s) 2023-03-09T14:47:43.907 INFO:teuthology.orchestra.run.smithi038.stderr:35200000 of 50000000 tuples (70%) done (elapsed 79.05 s, remaining 33.23 s) 2023-03-09T14:47:44.013 INFO:teuthology.orchestra.run.smithi038.stderr:35300000 of 50000000 tuples (70%) done (elapsed 79.15 s, remaining 32.96 s) 2023-03-09T14:47:44.106 INFO:teuthology.orchestra.run.smithi038.stderr:35400000 of 50000000 tuples (70%) done (elapsed 79.24 s, remaining 32.68 s) 2023-03-09T14:47:44.205 INFO:teuthology.orchestra.run.smithi038.stderr:35500000 of 50000000 tuples (71%) done (elapsed 79.34 s, remaining 32.41 s) 2023-03-09T14:47:44.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:47:43 smithi122 ceph-mon[93869]: pgmap v52: 129 pgs: 129 active+clean; 5.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 45 MiB/s wr, 354 op/s 2023-03-09T14:47:44.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:47:43 smithi181 ceph-mon[94189]: pgmap v52: 129 pgs: 129 active+clean; 5.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 45 MiB/s wr, 354 op/s 2023-03-09T14:47:44.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:47:43 smithi038 ceph-mon[92422]: pgmap v52: 129 pgs: 129 active+clean; 5.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 45 MiB/s wr, 354 op/s 2023-03-09T14:47:44.307 INFO:teuthology.orchestra.run.smithi038.stderr:35600000 of 50000000 tuples (71%) done (elapsed 79.44 s, remaining 32.13 s) 2023-03-09T14:47:44.406 INFO:teuthology.orchestra.run.smithi038.stderr:35700000 of 50000000 tuples (71%) done (elapsed 79.54 s, remaining 31.86 s) 2023-03-09T14:47:44.510 INFO:teuthology.orchestra.run.smithi038.stderr:35800000 of 50000000 tuples (71%) done (elapsed 79.65 s, remaining 31.59 s) 2023-03-09T14:47:44.613 INFO:teuthology.orchestra.run.smithi038.stderr:35900000 of 50000000 tuples (71%) done (elapsed 79.75 s, remaining 31.32 s) 2023-03-09T14:47:44.682 INFO:teuthology.orchestra.run.smithi038.stderr:36000000 of 50000000 tuples (72%) done (elapsed 79.82 s, remaining 31.04 s) 2023-03-09T14:47:44.800 INFO:teuthology.orchestra.run.smithi038.stderr:36100000 of 50000000 tuples (72%) done (elapsed 79.94 s, remaining 30.78 s) 2023-03-09T14:47:44.898 INFO:teuthology.orchestra.run.smithi038.stderr:36200000 of 50000000 tuples (72%) done (elapsed 80.04 s, remaining 30.51 s) 2023-03-09T14:47:44.999 INFO:teuthology.orchestra.run.smithi038.stderr:36300000 of 50000000 tuples (72%) done (elapsed 80.14 s, remaining 30.24 s) 2023-03-09T14:47:45.106 INFO:teuthology.orchestra.run.smithi038.stderr:36400000 of 50000000 tuples (72%) done (elapsed 80.24 s, remaining 29.98 s) 2023-03-09T14:47:45.200 INFO:teuthology.orchestra.run.smithi038.stderr:36500000 of 50000000 tuples (73%) done (elapsed 80.34 s, remaining 29.71 s) 2023-03-09T14:47:45.288 INFO:teuthology.orchestra.run.smithi038.stderr:36600000 of 50000000 tuples (73%) done (elapsed 80.43 s, remaining 29.45 s) 2023-03-09T14:47:45.396 INFO:teuthology.orchestra.run.smithi038.stderr:36700000 of 50000000 tuples (73%) done (elapsed 80.53 s, remaining 29.19 s) 2023-03-09T14:47:45.494 INFO:teuthology.orchestra.run.smithi038.stderr:36800000 of 50000000 tuples (73%) done (elapsed 80.63 s, remaining 28.92 s) 2023-03-09T14:47:45.577 INFO:teuthology.orchestra.run.smithi038.stderr:36900000 of 50000000 tuples (73%) done (elapsed 80.71 s, remaining 28.65 s) 2023-03-09T14:47:45.671 INFO:teuthology.orchestra.run.smithi038.stderr:37000000 of 50000000 tuples (74%) done (elapsed 80.81 s, remaining 28.39 s) 2023-03-09T14:47:45.775 INFO:teuthology.orchestra.run.smithi038.stderr:37100000 of 50000000 tuples (74%) done (elapsed 80.91 s, remaining 28.13 s) 2023-03-09T14:47:46.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:47:45 smithi122 ceph-mon[93869]: pgmap v53: 129 pgs: 129 active+clean; 5.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 1.9 MiB/s wr, 277 op/s 2023-03-09T14:47:46.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:47:45 smithi181 ceph-mon[94189]: pgmap v53: 129 pgs: 129 active+clean; 5.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 1.9 MiB/s wr, 277 op/s 2023-03-09T14:47:46.257 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:47:45 smithi038 ceph-mon[92422]: pgmap v53: 129 pgs: 129 active+clean; 5.7 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 1.9 MiB/s wr, 277 op/s 2023-03-09T14:47:46.784 INFO:teuthology.orchestra.run.smithi038.stderr:37200000 of 50000000 tuples (74%) done (elapsed 81.92 s, remaining 28.19 s) 2023-03-09T14:47:47.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:47:46 smithi122 ceph-mon[93869]: mgrmap e25: x(active, since 91s), standbys: z, y 2023-03-09T14:47:47.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:47:46 smithi181 ceph-mon[94189]: mgrmap e25: x(active, since 91s), standbys: z, y 2023-03-09T14:47:47.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:47:46 smithi038 ceph-mon[92422]: mgrmap e25: x(active, since 91s), standbys: z, y 2023-03-09T14:47:48.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:47:47 smithi122 ceph-mon[93869]: pgmap v54: 129 pgs: 129 active+clean; 6.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 29 MiB/s wr, 330 op/s 2023-03-09T14:47:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:47:47 smithi181 ceph-mon[94189]: pgmap v54: 129 pgs: 129 active+clean; 6.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 29 MiB/s wr, 330 op/s 2023-03-09T14:47:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:47:47 smithi038 ceph-mon[92422]: pgmap v54: 129 pgs: 129 active+clean; 6.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 29 MiB/s wr, 330 op/s 2023-03-09T14:47:50.160 INFO:teuthology.orchestra.run.smithi038.stderr:37300000 of 50000000 tuples (74%) done (elapsed 85.30 s, remaining 29.04 s) 2023-03-09T14:47:50.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:47:49 smithi122 ceph-mon[93869]: pgmap v55: 129 pgs: 129 active+clean; 6.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 28 MiB/s wr, 194 op/s 2023-03-09T14:47:50.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:47:49 smithi181 ceph-mon[94189]: pgmap v55: 129 pgs: 129 active+clean; 6.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 28 MiB/s wr, 194 op/s 2023-03-09T14:47:50.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:47:49 smithi038 ceph-mon[92422]: pgmap v55: 129 pgs: 129 active+clean; 6.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 28 MiB/s wr, 194 op/s 2023-03-09T14:47:52.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:47:51 smithi122 ceph-mon[93869]: pgmap v56: 129 pgs: 129 active+clean; 6.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 28 MiB/s wr, 194 op/s 2023-03-09T14:47:52.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:47:51 smithi181 ceph-mon[94189]: pgmap v56: 129 pgs: 129 active+clean; 6.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 28 MiB/s wr, 194 op/s 2023-03-09T14:47:52.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:47:51 smithi038 ceph-mon[92422]: pgmap v56: 129 pgs: 129 active+clean; 6.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 28 MiB/s wr, 194 op/s 2023-03-09T14:47:53.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:47:53 smithi122 ceph-mon[93869]: pgmap v57: 129 pgs: 129 active+clean; 6.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 74 MiB/s wr, 321 op/s 2023-03-09T14:47:53.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:47:53 smithi181 ceph-mon[94189]: pgmap v57: 129 pgs: 129 active+clean; 6.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 74 MiB/s wr, 321 op/s 2023-03-09T14:47:54.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:47:53 smithi038 ceph-mon[92422]: pgmap v57: 129 pgs: 129 active+clean; 6.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 74 MiB/s wr, 321 op/s 2023-03-09T14:47:55.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:47:55 smithi122 ceph-mon[93869]: pgmap v58: 129 pgs: 129 active+clean; 6.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 73 MiB/s wr, 180 op/s 2023-03-09T14:47:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:47:55 smithi181 ceph-mon[94189]: pgmap v58: 129 pgs: 129 active+clean; 6.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 73 MiB/s wr, 180 op/s 2023-03-09T14:47:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:47:55 smithi038 ceph-mon[92422]: pgmap v58: 129 pgs: 129 active+clean; 6.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 73 MiB/s wr, 180 op/s 2023-03-09T14:47:57.050 INFO:teuthology.orchestra.run.smithi038.stderr:37400000 of 50000000 tuples (74%) done (elapsed 92.19 s, remaining 31.06 s) 2023-03-09T14:47:57.131 INFO:teuthology.orchestra.run.smithi038.stderr:37500000 of 50000000 tuples (75%) done (elapsed 92.27 s, remaining 30.76 s) 2023-03-09T14:47:57.225 INFO:teuthology.orchestra.run.smithi038.stderr:37600000 of 50000000 tuples (75%) done (elapsed 92.36 s, remaining 30.46 s) 2023-03-09T14:47:57.315 INFO:teuthology.orchestra.run.smithi038.stderr:37700000 of 50000000 tuples (75%) done (elapsed 92.45 s, remaining 30.16 s) 2023-03-09T14:47:57.427 INFO:teuthology.orchestra.run.smithi038.stderr:37800000 of 50000000 tuples (75%) done (elapsed 92.56 s, remaining 29.88 s) 2023-03-09T14:47:57.525 INFO:teuthology.orchestra.run.smithi038.stderr:37900000 of 50000000 tuples (75%) done (elapsed 92.66 s, remaining 29.58 s) 2023-03-09T14:47:57.634 INFO:teuthology.orchestra.run.smithi038.stderr:38000000 of 50000000 tuples (76%) done (elapsed 92.77 s, remaining 29.30 s) 2023-03-09T14:47:57.732 INFO:teuthology.orchestra.run.smithi038.stderr:38100000 of 50000000 tuples (76%) done (elapsed 92.87 s, remaining 29.01 s) 2023-03-09T14:47:57.836 INFO:teuthology.orchestra.run.smithi038.stderr:38200000 of 50000000 tuples (76%) done (elapsed 92.97 s, remaining 28.72 s) 2023-03-09T14:47:57.912 INFO:teuthology.orchestra.run.smithi038.stderr:38300000 of 50000000 tuples (76%) done (elapsed 93.05 s, remaining 28.43 s) 2023-03-09T14:47:57.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:47:57 smithi122 ceph-mon[93869]: pgmap v59: 129 pgs: 129 active+clean; 6.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 76 MiB/s wr, 300 op/s 2023-03-09T14:47:57.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:47:57 smithi181 ceph-mon[94189]: pgmap v59: 129 pgs: 129 active+clean; 6.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 76 MiB/s wr, 300 op/s 2023-03-09T14:47:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:47:57 smithi038 ceph-mon[92422]: pgmap v59: 129 pgs: 129 active+clean; 6.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 76 MiB/s wr, 300 op/s 2023-03-09T14:47:58.020 INFO:teuthology.orchestra.run.smithi038.stderr:38400000 of 50000000 tuples (76%) done (elapsed 93.16 s, remaining 28.14 s) 2023-03-09T14:47:58.122 INFO:teuthology.orchestra.run.smithi038.stderr:38500000 of 50000000 tuples (77%) done (elapsed 93.26 s, remaining 27.86 s) 2023-03-09T14:47:58.227 INFO:teuthology.orchestra.run.smithi038.stderr:38600000 of 50000000 tuples (77%) done (elapsed 93.36 s, remaining 27.57 s) 2023-03-09T14:47:58.321 INFO:teuthology.orchestra.run.smithi038.stderr:38700000 of 50000000 tuples (77%) done (elapsed 93.46 s, remaining 27.29 s) 2023-03-09T14:47:58.412 INFO:teuthology.orchestra.run.smithi038.stderr:38800000 of 50000000 tuples (77%) done (elapsed 93.55 s, remaining 27.00 s) 2023-03-09T14:47:58.523 INFO:teuthology.orchestra.run.smithi038.stderr:38900000 of 50000000 tuples (77%) done (elapsed 93.66 s, remaining 26.73 s) 2023-03-09T14:47:59.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:47:59 smithi122 ceph-mon[93869]: pgmap v60: 129 pgs: 129 active+clean; 6.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 49 MiB/s wr, 246 op/s 2023-03-09T14:47:59.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:47:59 smithi181 ceph-mon[94189]: pgmap v60: 129 pgs: 129 active+clean; 6.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 49 MiB/s wr, 246 op/s 2023-03-09T14:48:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:47:59 smithi038 ceph-mon[92422]: pgmap v60: 129 pgs: 129 active+clean; 6.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 49 MiB/s wr, 246 op/s 2023-03-09T14:48:00.146 INFO:teuthology.orchestra.run.smithi038.stderr:39000000 of 50000000 tuples (78%) done (elapsed 95.28 s, remaining 26.88 s) 2023-03-09T14:48:01.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:48:01 smithi122 ceph-mon[93869]: pgmap v61: 129 pgs: 129 active+clean; 6.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 49 MiB/s wr, 246 op/s 2023-03-09T14:48:01.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:48:01 smithi181 ceph-mon[94189]: pgmap v61: 129 pgs: 129 active+clean; 6.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 49 MiB/s wr, 246 op/s 2023-03-09T14:48:02.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:48:01 smithi038 ceph-mon[92422]: pgmap v61: 129 pgs: 129 active+clean; 6.5 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 49 MiB/s wr, 246 op/s 2023-03-09T14:48:03.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:48:03 smithi122 ceph-mon[93869]: pgmap v62: 129 pgs: 129 active+clean; 6.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 63 MiB/s wr, 353 op/s 2023-03-09T14:48:03.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:48:03 smithi181 ceph-mon[94189]: pgmap v62: 129 pgs: 129 active+clean; 6.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 63 MiB/s wr, 353 op/s 2023-03-09T14:48:04.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:48:03 smithi038 ceph-mon[92422]: pgmap v62: 129 pgs: 129 active+clean; 6.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 63 MiB/s wr, 353 op/s 2023-03-09T14:48:05.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:48:05 smithi122 ceph-mon[93869]: pgmap v63: 129 pgs: 129 active+clean; 6.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 17 MiB/s wr, 227 op/s 2023-03-09T14:48:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:48:05 smithi181 ceph-mon[94189]: pgmap v63: 129 pgs: 129 active+clean; 6.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 17 MiB/s wr, 227 op/s 2023-03-09T14:48:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:48:05 smithi038 ceph-mon[92422]: pgmap v63: 129 pgs: 129 active+clean; 6.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 17 MiB/s wr, 227 op/s 2023-03-09T14:48:07.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:48:07 smithi122 ceph-mon[93869]: pgmap v64: 129 pgs: 129 active+clean; 6.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 18 MiB/s wr, 358 op/s 2023-03-09T14:48:07.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:48:07 smithi181 ceph-mon[94189]: pgmap v64: 129 pgs: 129 active+clean; 6.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 18 MiB/s wr, 358 op/s 2023-03-09T14:48:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:48:07 smithi038 ceph-mon[92422]: pgmap v64: 129 pgs: 129 active+clean; 6.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 18 MiB/s wr, 358 op/s 2023-03-09T14:48:09.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:48:09 smithi122 ceph-mon[93869]: pgmap v65: 129 pgs: 129 active+clean; 6.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 15 MiB/s wr, 238 op/s 2023-03-09T14:48:09.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:48:09 smithi181 ceph-mon[94189]: pgmap v65: 129 pgs: 129 active+clean; 6.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 15 MiB/s wr, 238 op/s 2023-03-09T14:48:10.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:48:09 smithi038 ceph-mon[92422]: pgmap v65: 129 pgs: 129 active+clean; 6.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 15 MiB/s wr, 238 op/s 2023-03-09T14:48:10.759 INFO:teuthology.orchestra.run.smithi038.stderr:39100000 of 50000000 tuples (78%) done (elapsed 105.90 s, remaining 29.52 s) 2023-03-09T14:48:11.404 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-03-09T14:48:11.544 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.540+0000 7f4169d87700 1 -- 172.21.15.38:0/771487278 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f41640ff000 msgr2=0x7f41640ff3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:48:11.544 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.540+0000 7f4169d87700 1 --2- 172.21.15.38:0/771487278 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f41640ff000 0x7f41640ff3d0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f41540099c0 tx=0x7f4154009cd0 comp rx=0 tx=0).stop 2023-03-09T14:48:11.545 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.540+0000 7f4169d87700 1 -- 172.21.15.38:0/771487278 shutdown_connections 2023-03-09T14:48:11.545 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.540+0000 7f4169d87700 1 --2- 172.21.15.38:0/771487278 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f41640ff000 0x7f41640ff3d0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:48:11.545 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.540+0000 7f4169d87700 1 --2- 172.21.15.38:0/771487278 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4164104040 0x7f416410bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:48:11.545 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.540+0000 7f4169d87700 1 --2- 172.21.15.38:0/771487278 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f41640ff9a0 0x7f4164103960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:48:11.546 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.540+0000 7f4169d87700 1 -- 172.21.15.38:0/771487278 >> 172.21.15.38:0/771487278 conn(0x7f4164075c60 msgr2=0x7f4164076060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:48:11.546 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.540+0000 7f4169d87700 1 -- 172.21.15.38:0/771487278 shutdown_connections 2023-03-09T14:48:11.546 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.540+0000 7f4169d87700 1 -- 172.21.15.38:0/771487278 wait complete. 2023-03-09T14:48:11.547 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.541+0000 7f4169d87700 1 Processor -- start 2023-03-09T14:48:11.556 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.541+0000 7f4169d87700 1 -- start start 2023-03-09T14:48:11.557 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.541+0000 7f4169d87700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f41640ff000 0x7f4164196e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:48:11.557 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.541+0000 7f4169d87700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f41640ff9a0 0x7f41641973b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:48:11.557 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.541+0000 7f4169d87700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4164104040 0x7f41641a1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:48:11.557 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.541+0000 7f4169d87700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41641a2430 con 0x7f41640ff000 2023-03-09T14:48:11.558 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.541+0000 7f4169d87700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41641a2570 con 0x7f4164104040 2023-03-09T14:48:11.558 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.541+0000 7f4169d87700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41641a26b0 con 0x7f41640ff9a0 2023-03-09T14:48:11.558 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.541+0000 7f41637fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f41640ff000 0x7f4164196e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:48:11.558 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.541+0000 7f41637fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f41640ff000 0x7f4164196e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39816/0 (socket says 172.21.15.38:39816) 2023-03-09T14:48:11.558 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.541+0000 7f41637fe700 1 -- 172.21.15.38:0/3616544273 learned_addr learned my addr 172.21.15.38:0/3616544273 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:48:11.558 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.541+0000 7f4162ffd700 1 --2- 172.21.15.38:0/3616544273 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f41640ff9a0 0x7f41641973b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:48:11.559 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.542+0000 7f41637fe700 1 -- 172.21.15.38:0/3616544273 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f41640ff9a0 msgr2=0x7f41641973b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:48:11.559 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.542+0000 7f41637fe700 1 --2- 172.21.15.38:0/3616544273 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f41640ff9a0 0x7f41641973b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:48:11.559 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.542+0000 7f41637fe700 1 -- 172.21.15.38:0/3616544273 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4164104040 msgr2=0x7f41641a1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:48:11.559 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.542+0000 7f41637fe700 1 --2- 172.21.15.38:0/3616544273 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4164104040 0x7f41641a1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:48:11.559 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.542+0000 7f41637fe700 1 -- 172.21.15.38:0/3616544273 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41540095f0 con 0x7f41640ff000 2023-03-09T14:48:11.559 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.542+0000 7f41637fe700 1 --2- 172.21.15.38:0/3616544273 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f41640ff000 0x7f4164196e70 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f4154000c00 tx=0x7f415400bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:48:11.559 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.542+0000 7f4160ff9700 1 -- 172.21.15.38:0/3616544273 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f415401d070 con 0x7f41640ff000 2023-03-09T14:48:11.560 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.542+0000 7f4160ff9700 1 -- 172.21.15.38:0/3616544273 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4154005050 con 0x7f41640ff000 2023-03-09T14:48:11.560 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.542+0000 7f4169d87700 1 -- 172.21.15.38:0/3616544273 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f416410ee00 con 0x7f41640ff000 2023-03-09T14:48:11.560 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.542+0000 7f4160ff9700 1 -- 172.21.15.38:0/3616544273 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4154004b30 con 0x7f41640ff000 2023-03-09T14:48:11.560 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.542+0000 7f4169d87700 1 -- 172.21.15.38:0/3616544273 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f416410ec70 con 0x7f41640ff000 2023-03-09T14:48:11.560 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.545+0000 7f4160ff9700 1 -- 172.21.15.38:0/3616544273 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 25) v1 ==== 133295+0+0 (secure 0 0 0) 0x7f4154028030 con 0x7f41640ff000 2023-03-09T14:48:11.560 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.546+0000 7f4160ff9700 1 --2- 172.21.15.38:0/3616544273 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f414c09f4a0 0x7f414c0a1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:48:11.561 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.546+0000 7f4160ff9700 1 -- 172.21.15.38:0/3616544273 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9489+0+0 (secure 0 0 0) 0x7f41540caec0 con 0x7f41640ff000 2023-03-09T14:48:11.561 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.546+0000 7f4162ffd700 1 --2- 172.21.15.38:0/3616544273 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f414c09f4a0 0x7f414c0a1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:48:11.561 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.546+0000 7f4175f10b80 1 Processor -- start 2023-03-09T14:48:11.561 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.546+0000 7f4175f10b80 1 -- start start 2023-03-09T14:48:11.561 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.546+0000 7f4175f10b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55683e6aec40 0x55683e6af010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:48:11.562 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.546+0000 7f4175f10b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55683e6af550 0x55683e6a8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:48:11.562 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.546+0000 7f4175f10b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55683e6a93f0 0x55683e6a9820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:48:11.562 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.546+0000 7f4175f10b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55683e58eda0 con 0x55683e6aec40 2023-03-09T14:48:11.562 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.546+0000 7f4175f10b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55683e5fd780 con 0x55683e6a93f0 2023-03-09T14:48:11.563 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.546+0000 7f4175f10b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55683e5f6cb0 con 0x55683e6af550 2023-03-09T14:48:11.563 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.547+0000 7f4162ffd700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55683e6af550 0x55683e6a8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:48:11.563 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.547+0000 7f41637fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55683e6aec40 0x55683e6af010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:48:11.563 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.547+0000 7f4162ffd700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55683e6af550 0x55683e6a8eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.38:47524/0 (socket says 172.21.15.38:47524) 2023-03-09T14:48:11.563 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.547+0000 7f4162ffd700 1 -- 172.21.15.38:0/687564298 learned_addr learned my addr 172.21.15.38:0/687564298 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:48:11.564 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.547+0000 7f4163fff700 1 --2- 172.21.15.38:0/687564298 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55683e6a93f0 0x55683e6a9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:48:11.564 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.547+0000 7f4162ffd700 1 --2- 172.21.15.38:0/3616544273 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f414c09f4a0 0x7f414c0a1950 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4164069fc0 tx=0x7f4158009260 comp rx=0 tx=0).ready entity=mgr.15558 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:48:11.564 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.547+0000 7f4162ffd700 1 -- 172.21.15.38:0/687564298 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55683e6a93f0 msgr2=0x55683e6a9820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:48:11.564 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.547+0000 7f4162ffd700 1 --2- 172.21.15.38:0/687564298 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55683e6a93f0 0x55683e6a9820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:48:11.564 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.547+0000 7f4162ffd700 1 -- 172.21.15.38:0/687564298 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55683e6aec40 msgr2=0x55683e6af010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:48:11.564 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.547+0000 7f4162ffd700 1 --2- 172.21.15.38:0/687564298 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55683e6aec40 0x55683e6af010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:48:11.564 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.547+0000 7f4162ffd700 1 -- 172.21.15.38:0/687564298 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55683e5bd190 con 0x55683e6af550 2023-03-09T14:48:11.565 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.547+0000 7f4162ffd700 1 --2- 172.21.15.38:0/687564298 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55683e6af550 0x55683e6a8eb0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4158011ce0 tx=0x7f4158014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d558b3f40fbd4bdc server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:48:11.565 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.547+0000 7f4149ffb700 1 -- 172.21.15.38:0/687564298 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f415800eb00 con 0x55683e6af550 2023-03-09T14:48:11.565 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.547+0000 7f4149ffb700 1 -- 172.21.15.38:0/687564298 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4158021070 con 0x55683e6af550 2023-03-09T14:48:11.565 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.547+0000 7f4149ffb700 1 -- 172.21.15.38:0/687564298 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41580054e0 con 0x55683e6af550 2023-03-09T14:48:11.565 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.548+0000 7f4175f10b80 1 -- 172.21.15.38:0/687564298 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55683e6af550 msgr2=0x55683e6a8eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:48:11.565 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.548+0000 7f4175f10b80 1 --2- 172.21.15.38:0/687564298 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55683e6af550 0x55683e6a8eb0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4158011ce0 tx=0x7f4158014000 comp rx=0 tx=0).stop 2023-03-09T14:48:11.565 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.548+0000 7f4175f10b80 1 -- 172.21.15.38:0/687564298 shutdown_connections 2023-03-09T14:48:11.566 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.548+0000 7f4175f10b80 1 --2- 172.21.15.38:0/687564298 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55683e6aec40 0x55683e6af010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:48:11.566 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.548+0000 7f4175f10b80 1 --2- 172.21.15.38:0/687564298 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55683e6a93f0 0x55683e6a9820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:48:11.566 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.548+0000 7f4175f10b80 1 --2- 172.21.15.38:0/687564298 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55683e6af550 0x55683e6a8eb0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:48:11.566 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.548+0000 7f4175f10b80 1 -- 172.21.15.38:0/687564298 >> 172.21.15.38:0/687564298 conn(0x55683e69fa20 msgr2=0x55683e69fdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:48:11.566 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.548+0000 7f4175f10b80 1 -- 172.21.15.38:0/687564298 shutdown_connections 2023-03-09T14:48:11.566 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.548+0000 7f4175f10b80 1 -- 172.21.15.38:0/687564298 wait complete. 2023-03-09T14:48:11.566 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.548+0000 7f4175f10b80 1 -- start start 2023-03-09T14:48:11.566 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.549+0000 7f4175f10b80 1 Processor -- start 2023-03-09T14:48:11.567 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.549+0000 7f4175f10b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55683e6a93f0 0x55683e6b5540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:48:11.567 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.549+0000 7f4175f10b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55683e6aec40 0x55683e6b27c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:48:11.567 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.549+0000 7f4175f10b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55683e6af550 0x55683e6b2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:48:11.567 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.549+0000 7f4175f10b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55683e47a920 con 0x55683e6af550 2023-03-09T14:48:11.567 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.549+0000 7f4175f10b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55683e6b35d0 con 0x55683e6a93f0 2023-03-09T14:48:11.568 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.549+0000 7f4175f10b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55683e6bf0b0 con 0x55683e6aec40 2023-03-09T14:48:11.568 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.549+0000 7f4175f10b80 10 client.0 _ll_register_callbacks cb 0x55683e59c520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:48:11.568 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.549+0000 7f41637fe700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55683e6a93f0 0x55683e6b5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:48:11.568 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.549+0000 7f4162ffd700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55683e6aec40 0x55683e6b27c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:48:11.569 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.549+0000 7f4163fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55683e6af550 0x55683e6b2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:48:11.569 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.549+0000 7f41637fe700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55683e6a93f0 0x55683e6b5540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.38:49020/0 (socket says 172.21.15.38:49020) 2023-03-09T14:48:11.569 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.549+0000 7f4162ffd700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55683e6aec40 0x55683e6b27c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.38:47530/0 (socket says 172.21.15.38:47530) 2023-03-09T14:48:11.570 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.549+0000 7f41637fe700 1 -- 172.21.15.38:0/781766989 learned_addr learned my addr 172.21.15.38:0/781766989 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:48:11.570 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.549+0000 7f4162ffd700 1 -- 172.21.15.38:0/781766989 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55683e6a93f0 msgr2=0x55683e6b5540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:48:11.570 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.549+0000 7f4162ffd700 1 --2- 172.21.15.38:0/781766989 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55683e6a93f0 0x55683e6b5540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:48:11.570 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.549+0000 7f4162ffd700 1 -- 172.21.15.38:0/781766989 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55683e6af550 msgr2=0x55683e6b2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:48:11.571 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.549+0000 7f4162ffd700 1 --2- 172.21.15.38:0/781766989 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55683e6af550 0x55683e6b2e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:48:11.571 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.550+0000 7f4162ffd700 1 -- 172.21.15.38:0/781766989 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4158017070 con 0x55683e6aec40 2023-03-09T14:48:11.571 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.550+0000 7f41637fe700 1 --2- 172.21.15.38:0/781766989 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55683e6a93f0 0x55683e6b5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:48:11.571 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.550+0000 7f4162ffd700 1 --2- 172.21.15.38:0/781766989 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55683e6aec40 0x55683e6b27c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f415800e690 tx=0x7f415800eb30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=191b78b6d0b6e54e server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:48:11.571 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.550+0000 7f4142ffd700 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2023-03-09T14:48:11.571 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.550+0000 7f4142ffd700 1 -- 172.21.15.38:0/781766989 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41580055f0 con 0x55683e6aec40 2023-03-09T14:48:11.571 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.550+0000 7f4175f10b80 1 -- 172.21.15.38:0/781766989 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55683e5c4190 con 0x55683e6aec40 2023-03-09T14:48:11.572 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.550+0000 7f4142ffd700 1 -- 172.21.15.38:0/781766989 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9489+0+0 (secure 0 0 0) 0x7f4158004750 con 0x55683e6aec40 2023-03-09T14:48:11.572 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.550+0000 7f4142ffd700 1 -- 172.21.15.38:0/781766989 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4158004910 con 0x55683e6aec40 2023-03-09T14:48:11.572 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.550+0000 7f4175f10b80 10 client.25498 fetch_fsmap learned FSMap version 25 2023-03-09T14:48:11.572 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.550+0000 7f4175f10b80 1 -- 172.21.15.38:0/781766989 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55683e40dc60 con 0x55683e6aec40 2023-03-09T14:48:11.572 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.550+0000 7f4142ffd700 1 -- 172.21.15.38:0/781766989 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55683e5c4190 con 0x55683e6aec40 2023-03-09T14:48:11.573 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.551+0000 7f4142ffd700 1 -- 172.21.15.38:0/781766989 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 25) v1 ==== 7478+0+0 (secure 0 0 0) 0x7f4158026b00 con 0x55683e6aec40 2023-03-09T14:48:11.573 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.551+0000 7f4175f10b80 10 client.25498 fetch_fsmap finished waiting for FSMap version 25 2023-03-09T14:48:11.573 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.551+0000 7f4175f10b80 10 client.25498 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T14:48:11.573 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.551+0000 7f4175f10b80 20 client.25498 populate_metadata read hostname 'smithi038' 2023-03-09T14:48:11.573 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.551+0000 7f4175f10b80 1 --2- 172.21.15.38:0/781766989 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55683e6c30c0 0x55683e6c54d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:48:11.573 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.551+0000 7f4175f10b80 4 client.25498 mds_command: new command op to 24530 tid=0[{"prefix": "get_command_descriptions"}] 2023-03-09T14:48:11.573 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.551+0000 7f4175f10b80 1 -- 172.21.15.38:0/781766989 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55683e47af60 con 0x55683e6c30c0 2023-03-09T14:48:11.573 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.551+0000 7f41637fe700 1 --2- 172.21.15.38:0/781766989 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55683e6c30c0 0x55683e6c54d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:48:11.574 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.551+0000 7f4142ffd700 10 client.25498 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:48:11.574 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.551+0000 7f41637fe700 1 --2- 172.21.15.38:0/781766989 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55683e6c30c0 0x55683e6c54d0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ef39b4cdae40f21 server_cookie=5ab98e1ead5e559f in_seq=0 out_seq=0 2023-03-09T14:48:11.574 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.552+0000 7f4142ffd700 1 -- 172.21.15.38:0/781766989 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18278 (crc 0 0 0) 0x55683e47af60 con 0x55683e6c30c0 2023-03-09T14:48:11.574 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.552+0000 7f4142ffd700 10 client.25498 handle_command_reply: tid=0 2023-03-09T14:48:11.574 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.552+0000 7f4175f10b80 1 client.25498 shutdown 2023-03-09T14:48:11.574 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.553+0000 7f4175f10b80 1 -- 172.21.15.38:0/781766989 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55683e6aec40 msgr2=0x55683e6b27c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:48:11.574 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.553+0000 7f4175f10b80 1 --2- 172.21.15.38:0/781766989 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55683e6aec40 0x55683e6b27c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f415800e690 tx=0x7f415800eb30 comp rx=0 tx=0).stop 2023-03-09T14:48:11.575 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.553+0000 7f4175f10b80 1 -- 172.21.15.38:0/781766989 shutdown_connections 2023-03-09T14:48:11.575 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.553+0000 7f4175f10b80 1 --2- 172.21.15.38:0/781766989 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55683e6c30c0 0x55683e6c54d0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:48:11.575 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.553+0000 7f4175f10b80 1 --2- 172.21.15.38:0/781766989 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55683e6af550 0x55683e6b2e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:48:11.575 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.553+0000 7f4175f10b80 1 --2- 172.21.15.38:0/781766989 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55683e6a93f0 0x55683e6b5540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:48:11.575 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.553+0000 7f4175f10b80 1 --2- 172.21.15.38:0/781766989 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55683e6aec40 0x55683e6b27c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:48:11.575 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.553+0000 7f4175f10b80 1 -- 172.21.15.38:0/781766989 >> 172.21.15.38:0/781766989 conn(0x55683e69fa20 msgr2=0x55683e69fdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:48:11.575 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.553+0000 7f4142ffd700 0 client.25498 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:48:11.575 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.553+0000 7f4175f10b80 1 -- 172.21.15.38:0/781766989 shutdown_connections 2023-03-09T14:48:11.576 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.553+0000 7f4175f10b80 1 -- 172.21.15.38:0/781766989 wait complete. 2023-03-09T14:48:11.576 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.553+0000 7f4175f10b80 20 client.25498 trim_cache size 0 max 16384 2023-03-09T14:48:11.579 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.571+0000 7f4175f10b80 1 Processor -- start 2023-03-09T14:48:11.579 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.571+0000 7f4175f10b80 1 -- start start 2023-03-09T14:48:11.579 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.572+0000 7f4175f10b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55683e6a93f0 0x55683e6b1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:48:11.579 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.572+0000 7f4175f10b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55683e6aec40 0x55683e6be790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:48:11.580 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.572+0000 7f4175f10b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55683e6af550 0x55683e6c50d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:48:11.580 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.572+0000 7f4175f10b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55683e69e680 con 0x55683e6af550 2023-03-09T14:48:11.580 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.572+0000 7f4175f10b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55683e69f810 con 0x55683e6a93f0 2023-03-09T14:48:11.580 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.572+0000 7f4175f10b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55683e6b3710 con 0x55683e6aec40 2023-03-09T14:48:11.580 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.572+0000 7f4163fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55683e6af550 0x55683e6c50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:48:11.580 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.572+0000 7f4162ffd700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55683e6aec40 0x55683e6be790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:48:11.580 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.572+0000 7f41637fe700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55683e6a93f0 0x55683e6b1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:48:11.581 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.572+0000 7f4163fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55683e6af550 0x55683e6c50d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39842/0 (socket says 172.21.15.38:39842) 2023-03-09T14:48:11.581 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.572+0000 7f4163fff700 1 -- 172.21.15.38:0/1065609817 learned_addr learned my addr 172.21.15.38:0/1065609817 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:48:11.581 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.572+0000 7f4162ffd700 1 --2- 172.21.15.38:0/1065609817 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55683e6aec40 0x55683e6be790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.38:47538/0 (socket says 172.21.15.38:47538) 2023-03-09T14:48:11.581 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.572+0000 7f4163fff700 1 -- 172.21.15.38:0/1065609817 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55683e6aec40 msgr2=0x55683e6be790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:48:11.581 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.572+0000 7f4163fff700 1 --2- 172.21.15.38:0/1065609817 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55683e6aec40 0x55683e6be790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:48:11.581 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.572+0000 7f4163fff700 1 -- 172.21.15.38:0/1065609817 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55683e6a93f0 msgr2=0x55683e6b1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:48:11.581 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.572+0000 7f4163fff700 1 --2- 172.21.15.38:0/1065609817 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55683e6a93f0 0x55683e6b1fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:48:11.581 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.572+0000 7f4163fff700 1 -- 172.21.15.38:0/1065609817 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55683e424de0 con 0x55683e6af550 2023-03-09T14:48:11.582 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.572+0000 7f4163fff700 1 --2- 172.21.15.38:0/1065609817 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55683e6af550 0x55683e6c50d0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f415c014660 tx=0x7f415c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=753aa576e5a179e4 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:48:11.582 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.572+0000 7f414a7fc700 1 -- 172.21.15.38:0/1065609817 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f415c010040 con 0x55683e6af550 2023-03-09T14:48:11.582 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.572+0000 7f414a7fc700 1 -- 172.21.15.38:0/1065609817 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f415c004050 con 0x55683e6af550 2023-03-09T14:48:11.582 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.573+0000 7f414a7fc700 1 -- 172.21.15.38:0/1065609817 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f415c007760 con 0x55683e6af550 2023-03-09T14:48:11.582 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.573+0000 7f4175f10b80 1 -- 172.21.15.38:0/1065609817 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55683e6af550 msgr2=0x55683e6c50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:48:11.582 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.573+0000 7f4175f10b80 1 --2- 172.21.15.38:0/1065609817 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55683e6af550 0x55683e6c50d0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f415c014660 tx=0x7f415c00b040 comp rx=0 tx=0).stop 2023-03-09T14:48:11.582 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.573+0000 7f4175f10b80 1 -- 172.21.15.38:0/1065609817 shutdown_connections 2023-03-09T14:48:11.583 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.573+0000 7f4175f10b80 1 --2- 172.21.15.38:0/1065609817 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55683e6af550 0x55683e6c50d0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:48:11.583 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.573+0000 7f4175f10b80 1 --2- 172.21.15.38:0/1065609817 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55683e6a93f0 0x55683e6b1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:48:11.583 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.573+0000 7f4175f10b80 1 --2- 172.21.15.38:0/1065609817 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55683e6aec40 0x55683e6be790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:48:11.583 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.573+0000 7f4175f10b80 1 -- 172.21.15.38:0/1065609817 >> 172.21.15.38:0/1065609817 conn(0x55683e69fa20 msgr2=0x55683e69df70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:48:11.583 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.573+0000 7f4175f10b80 1 -- 172.21.15.38:0/1065609817 shutdown_connections 2023-03-09T14:48:11.583 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.573+0000 7f4175f10b80 1 -- 172.21.15.38:0/1065609817 wait complete. 2023-03-09T14:48:11.583 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.574+0000 7f4175f10b80 1 -- start start 2023-03-09T14:48:11.583 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.574+0000 7f4175f10b80 1 Processor -- start 2023-03-09T14:48:11.584 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.574+0000 7f4175f10b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55683e6a93f0 0x55683e6ab870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:48:11.584 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.574+0000 7f4175f10b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55683e6aec40 0x55683e6abdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:48:11.584 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.574+0000 7f4175f10b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55683e6af550 0x55683e6aa6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:48:11.584 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.574+0000 7f4175f10b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55683e6ac4c0 con 0x55683e6aec40 2023-03-09T14:48:11.584 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.574+0000 7f4175f10b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55683e6aabe0 con 0x55683e6af550 2023-03-09T14:48:11.584 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.574+0000 7f4175f10b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55683e6aada0 con 0x55683e6a93f0 2023-03-09T14:48:11.584 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.574+0000 7f4175f10b80 10 client.0 _ll_register_callbacks cb 0x55683e59c520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:48:11.585 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.575+0000 7f41637fe700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55683e6a93f0 0x55683e6ab870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:48:11.585 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.575+0000 7f4162ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55683e6aec40 0x55683e6abdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:48:11.585 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.575+0000 7f41637fe700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55683e6a93f0 0x55683e6ab870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.38:47542/0 (socket says 172.21.15.38:47542) 2023-03-09T14:48:11.585 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.575+0000 7f41637fe700 1 -- 172.21.15.38:0/74528041 learned_addr learned my addr 172.21.15.38:0/74528041 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:48:11.585 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.575+0000 7f4163fff700 1 --2- 172.21.15.38:0/74528041 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55683e6af550 0x55683e6aa6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:48:11.585 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.575+0000 7f41637fe700 1 -- 172.21.15.38:0/74528041 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55683e6af550 msgr2=0x55683e6aa6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:48:11.585 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.575+0000 7f41637fe700 1 --2- 172.21.15.38:0/74528041 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55683e6af550 0x55683e6aa6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:48:11.585 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.575+0000 7f41637fe700 1 -- 172.21.15.38:0/74528041 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55683e6aec40 msgr2=0x55683e6abdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:48:11.586 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.575+0000 7f41637fe700 1 --2- 172.21.15.38:0/74528041 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55683e6aec40 0x55683e6abdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:48:11.586 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.575+0000 7f41637fe700 1 -- 172.21.15.38:0/74528041 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f415c014340 con 0x55683e6a93f0 2023-03-09T14:48:11.586 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.575+0000 7f4162ffd700 1 --2- 172.21.15.38:0/74528041 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55683e6aec40 0x55683e6abdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:48:11.586 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.576+0000 7f41637fe700 1 --2- 172.21.15.38:0/74528041 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55683e6a93f0 0x55683e6ab870 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f41540036c0 tx=0x7f41540ca040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d123d767cbb304a server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:48:11.586 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.576+0000 7f4142ffd700 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2023-03-09T14:48:11.586 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.576+0000 7f4142ffd700 1 -- 172.21.15.38:0/74528041 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4154098f50 con 0x55683e6a93f0 2023-03-09T14:48:11.586 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.576+0000 7f4175f10b80 1 -- 172.21.15.38:0/74528041 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55683e5b41b0 con 0x55683e6a93f0 2023-03-09T14:48:11.587 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.576+0000 7f4142ffd700 1 -- 172.21.15.38:0/74528041 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9489+0+0 (secure 0 0 0) 0x7f4154097fe0 con 0x55683e6a93f0 2023-03-09T14:48:11.588 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.576+0000 7f4142ffd700 1 -- 172.21.15.38:0/74528041 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f41540981a0 con 0x55683e6a93f0 2023-03-09T14:48:11.588 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.576+0000 7f4175f10b80 10 client.25504 fetch_fsmap learned FSMap version 25 2023-03-09T14:48:11.589 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.576+0000 7f4175f10b80 1 -- 172.21.15.38:0/74528041 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55683e4232b0 con 0x55683e6a93f0 2023-03-09T14:48:11.589 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.576+0000 7f4142ffd700 1 -- 172.21.15.38:0/74528041 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55683e5b41b0 con 0x55683e6a93f0 2023-03-09T14:48:11.589 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.576+0000 7f4142ffd700 1 -- 172.21.15.38:0/74528041 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 25) v1 ==== 7478+0+0 (secure 0 0 0) 0x7f41540c8180 con 0x55683e6a93f0 2023-03-09T14:48:11.589 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.576+0000 7f4175f10b80 10 client.25504 fetch_fsmap finished waiting for FSMap version 25 2023-03-09T14:48:11.589 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.576+0000 7f4175f10b80 10 client.25504 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T14:48:11.589 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.576+0000 7f4175f10b80 20 client.25504 populate_metadata read hostname 'smithi038' 2023-03-09T14:48:11.589 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.576+0000 7f4175f10b80 1 --2- 172.21.15.38:0/74528041 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55683e717930 0x55683e719d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:48:11.590 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.576+0000 7f4175f10b80 4 client.25504 mds_command: new command op to 24530 tid=0[{"prefix": "scrub status"}] 2023-03-09T14:48:11.590 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.576+0000 7f4175f10b80 1 -- 172.21.15.38:0/74528041 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55683e59e1c0 con 0x55683e717930 2023-03-09T14:48:11.590 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.577+0000 7f4162ffd700 1 --2- 172.21.15.38:0/74528041 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55683e717930 0x55683e719d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:48:11.590 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.577+0000 7f4162ffd700 1 --2- 172.21.15.38:0/74528041 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55683e717930 0x55683e719d40 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6996f10b42fc02bb server_cookie=90ec005a28688836 in_seq=0 out_seq=0 2023-03-09T14:48:11.590 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.577+0000 7f4142ffd700 10 client.25504 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:48:11.590 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.578+0000 7f4142ffd700 1 -- 172.21.15.38:0/74528041 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+265 (crc 0 0 0) 0x55683e59e1c0 con 0x55683e717930 2023-03-09T14:48:11.590 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.578+0000 7f4142ffd700 10 client.25504 handle_command_reply: tid=0 2023-03-09T14:48:11.591 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.578+0000 7f4175f10b80 1 client.25504 shutdown 2023-03-09T14:48:11.591 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.578+0000 7f4175f10b80 1 -- 172.21.15.38:0/74528041 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55683e6a93f0 msgr2=0x55683e6ab870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:48:11.591 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.578+0000 7f4175f10b80 1 --2- 172.21.15.38:0/74528041 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55683e6a93f0 0x55683e6ab870 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f41540036c0 tx=0x7f41540ca040 comp rx=0 tx=0).stop 2023-03-09T14:48:11.591 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.578+0000 7f4175f10b80 1 -- 172.21.15.38:0/74528041 shutdown_connections 2023-03-09T14:48:11.591 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.578+0000 7f4175f10b80 1 --2- 172.21.15.38:0/74528041 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55683e717930 0x55683e719d40 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:48:11.591 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.578+0000 7f4175f10b80 1 --2- 172.21.15.38:0/74528041 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55683e6af550 0x55683e6aa6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:48:11.591 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.578+0000 7f4175f10b80 1 --2- 172.21.15.38:0/74528041 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55683e6a93f0 0x55683e6ab870 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:48:11.592 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.578+0000 7f4142ffd700 0 client.25504 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:48:11.592 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.578+0000 7f4175f10b80 1 --2- 172.21.15.38:0/74528041 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55683e6aec40 0x55683e6abdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:48:11.592 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.578+0000 7f4175f10b80 1 -- 172.21.15.38:0/74528041 >> 172.21.15.38:0/74528041 conn(0x55683e69fa20 msgr2=0x55683e6be790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:48:11.592 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.578+0000 7f4175f10b80 1 -- 172.21.15.38:0/74528041 shutdown_connections 2023-03-09T14:48:11.592 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.578+0000 7f4175f10b80 1 -- 172.21.15.38:0/74528041 wait complete. 2023-03-09T14:48:11.592 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.578+0000 7f4175f10b80 20 client.25504 trim_cache size 0 max 16384 2023-03-09T14:48:11.593 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2023-03-09T14:48:11.593 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "scrub active (3 inodes in the stack)", 2023-03-09T14:48:11.593 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": { 2023-03-09T14:48:11.593 INFO:teuthology.orchestra.run.smithi038.stdout: "a14659f4-d7ea-4c5a-b15b-6138af3f909f": { 2023-03-09T14:48:11.593 INFO:teuthology.orchestra.run.smithi038.stdout: "path": "/", 2023-03-09T14:48:11.593 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "a14659f4-d7ea-4c5a-b15b-6138af3f909f", 2023-03-09T14:48:11.594 INFO:teuthology.orchestra.run.smithi038.stdout: "options": "recursive,force" 2023-03-09T14:48:11.594 INFO:teuthology.orchestra.run.smithi038.stdout: } 2023-03-09T14:48:11.594 INFO:teuthology.orchestra.run.smithi038.stdout: } 2023-03-09T14:48:11.594 INFO:teuthology.orchestra.run.smithi038.stdout:} 2023-03-09T14:48:11.595 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.579+0000 7f4169d87700 1 -- 172.21.15.38:0/3616544273 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f414c09f4a0 msgr2=0x7f414c0a1950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:48:11.595 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.579+0000 7f4169d87700 1 --2- 172.21.15.38:0/3616544273 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f414c09f4a0 0x7f414c0a1950 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4164069fc0 tx=0x7f4158009260 comp rx=0 tx=0).stop 2023-03-09T14:48:11.595 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.579+0000 7f4169d87700 1 -- 172.21.15.38:0/3616544273 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f41640ff000 msgr2=0x7f4164196e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:48:11.595 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.579+0000 7f4169d87700 1 --2- 172.21.15.38:0/3616544273 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f41640ff000 0x7f4164196e70 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f4154000c00 tx=0x7f415400bfd0 comp rx=0 tx=0).stop 2023-03-09T14:48:11.595 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.579+0000 7f4169d87700 1 -- 172.21.15.38:0/3616544273 shutdown_connections 2023-03-09T14:48:11.596 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.579+0000 7f4169d87700 1 --2- 172.21.15.38:0/3616544273 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f414c09f4a0 0x7f414c0a1950 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:48:11.596 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.579+0000 7f4169d87700 1 --2- 172.21.15.38:0/3616544273 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f41640ff000 0x7f4164196e70 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:48:11.596 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.579+0000 7f4169d87700 1 --2- 172.21.15.38:0/3616544273 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4164104040 0x7f41641a1ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:48:11.596 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.579+0000 7f4169d87700 1 --2- 172.21.15.38:0/3616544273 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f41640ff9a0 0x7f41641973b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:48:11.596 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.579+0000 7f4169d87700 1 -- 172.21.15.38:0/3616544273 >> 172.21.15.38:0/3616544273 conn(0x7f4164075c60 msgr2=0x7f41640fc940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:48:11.596 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.579+0000 7f4169d87700 1 -- 172.21.15.38:0/3616544273 shutdown_connections 2023-03-09T14:48:11.596 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:11.579+0000 7f4169d87700 1 -- 172.21.15.38:0/3616544273 wait complete. 2023-03-09T14:48:11.597 INFO:tasks.cephfs.filesystem:scrub status for tag:a14659f4-d7ea-4c5a-b15b-6138af3f909f - {'path': '/', 'tag': 'a14659f4-d7ea-4c5a-b15b-6138af3f909f', 'options': 'recursive,force'} 2023-03-09T14:48:11.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:48:11 smithi038 ceph-mon[92422]: pgmap v66: 129 pgs: 129 active+clean; 6.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 15 MiB/s wr, 238 op/s 2023-03-09T14:48:11.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:48:11 smithi122 ceph-mon[93869]: pgmap v66: 129 pgs: 129 active+clean; 6.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 15 MiB/s wr, 238 op/s 2023-03-09T14:48:11.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:48:11 smithi181 ceph-mon[94189]: pgmap v66: 129 pgs: 129 active+clean; 6.6 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 15 MiB/s wr, 238 op/s 2023-03-09T14:48:13.841 INFO:teuthology.orchestra.run.smithi038.stderr:39200000 of 50000000 tuples (78%) done (elapsed 108.98 s, remaining 30.02 s) 2023-03-09T14:48:13.927 INFO:teuthology.orchestra.run.smithi038.stderr:39300000 of 50000000 tuples (78%) done (elapsed 109.07 s, remaining 29.69 s) 2023-03-09T14:48:13.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:48:13 smithi122 ceph-mon[93869]: pgmap v67: 129 pgs: 129 active+clean; 6.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 16 MiB/s wr, 375 op/s 2023-03-09T14:48:13.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:48:13 smithi181 ceph-mon[94189]: pgmap v67: 129 pgs: 129 active+clean; 6.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 16 MiB/s wr, 375 op/s 2023-03-09T14:48:14.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:48:13 smithi038 ceph-mon[92422]: pgmap v67: 129 pgs: 129 active+clean; 6.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 16 MiB/s wr, 375 op/s 2023-03-09T14:48:14.041 INFO:teuthology.orchestra.run.smithi038.stderr:39400000 of 50000000 tuples (78%) done (elapsed 109.18 s, remaining 29.37 s) 2023-03-09T14:48:14.153 INFO:teuthology.orchestra.run.smithi038.stderr:39500000 of 50000000 tuples (79%) done (elapsed 109.29 s, remaining 29.05 s) 2023-03-09T14:48:14.275 INFO:teuthology.orchestra.run.smithi038.stderr:39600000 of 50000000 tuples (79%) done (elapsed 109.41 s, remaining 28.73 s) 2023-03-09T14:48:14.360 INFO:teuthology.orchestra.run.smithi038.stderr:39700000 of 50000000 tuples (79%) done (elapsed 109.50 s, remaining 28.41 s) 2023-03-09T14:48:14.460 INFO:teuthology.orchestra.run.smithi038.stderr:39800000 of 50000000 tuples (79%) done (elapsed 109.60 s, remaining 28.09 s) 2023-03-09T14:48:14.567 INFO:teuthology.orchestra.run.smithi038.stderr:39900000 of 50000000 tuples (79%) done (elapsed 109.71 s, remaining 27.77 s) 2023-03-09T14:48:14.659 INFO:teuthology.orchestra.run.smithi038.stderr:40000000 of 50000000 tuples (80%) done (elapsed 109.80 s, remaining 27.45 s) 2023-03-09T14:48:14.761 INFO:teuthology.orchestra.run.smithi038.stderr:40100000 of 50000000 tuples (80%) done (elapsed 109.90 s, remaining 27.13 s) 2023-03-09T14:48:14.833 INFO:teuthology.orchestra.run.smithi038.stderr:40200000 of 50000000 tuples (80%) done (elapsed 109.97 s, remaining 26.81 s) 2023-03-09T14:48:14.974 INFO:teuthology.orchestra.run.smithi038.stderr:40300000 of 50000000 tuples (80%) done (elapsed 110.11 s, remaining 26.50 s) 2023-03-09T14:48:15.042 INFO:teuthology.orchestra.run.smithi038.stderr:40400000 of 50000000 tuples (80%) done (elapsed 110.18 s, remaining 26.18 s) 2023-03-09T14:48:15.147 INFO:teuthology.orchestra.run.smithi038.stderr:40500000 of 50000000 tuples (81%) done (elapsed 110.29 s, remaining 25.87 s) 2023-03-09T14:48:15.248 INFO:teuthology.orchestra.run.smithi038.stderr:40600000 of 50000000 tuples (81%) done (elapsed 110.39 s, remaining 25.56 s) 2023-03-09T14:48:15.349 INFO:teuthology.orchestra.run.smithi038.stderr:40700000 of 50000000 tuples (81%) done (elapsed 110.49 s, remaining 25.25 s) 2023-03-09T14:48:15.445 INFO:teuthology.orchestra.run.smithi038.stderr:40800000 of 50000000 tuples (81%) done (elapsed 110.58 s, remaining 24.94 s) 2023-03-09T14:48:15.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:48:15 smithi122 ceph-mon[93869]: pgmap v68: 129 pgs: 129 active+clean; 6.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 1.8 MiB/s wr, 268 op/s 2023-03-09T14:48:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:48:15 smithi181 ceph-mon[94189]: pgmap v68: 129 pgs: 129 active+clean; 6.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 1.8 MiB/s wr, 268 op/s 2023-03-09T14:48:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:48:15 smithi038 ceph-mon[92422]: pgmap v68: 129 pgs: 129 active+clean; 6.7 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 1.8 MiB/s wr, 268 op/s 2023-03-09T14:48:17.641 INFO:teuthology.orchestra.run.smithi038.stderr:40900000 of 50000000 tuples (81%) done (elapsed 112.78 s, remaining 25.09 s) 2023-03-09T14:48:17.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:48:17 smithi122 ceph-mon[93869]: pgmap v69: 129 pgs: 129 active+clean; 6.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 17 MiB/s wr, 354 op/s 2023-03-09T14:48:17.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:48:17 smithi181 ceph-mon[94189]: pgmap v69: 129 pgs: 129 active+clean; 6.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 17 MiB/s wr, 354 op/s 2023-03-09T14:48:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:48:17 smithi038 ceph-mon[92422]: pgmap v69: 129 pgs: 129 active+clean; 6.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 17 MiB/s wr, 354 op/s 2023-03-09T14:48:19.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:48:19 smithi122 ceph-mon[93869]: pgmap v70: 129 pgs: 129 active+clean; 6.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 16 MiB/s wr, 223 op/s 2023-03-09T14:48:19.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:48:19 smithi181 ceph-mon[94189]: pgmap v70: 129 pgs: 129 active+clean; 6.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 16 MiB/s wr, 223 op/s 2023-03-09T14:48:20.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:48:19 smithi038 ceph-mon[92422]: pgmap v70: 129 pgs: 129 active+clean; 6.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 16 MiB/s wr, 223 op/s 2023-03-09T14:48:21.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:48:21 smithi122 ceph-mon[93869]: pgmap v71: 129 pgs: 129 active+clean; 6.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 17 MiB/s wr, 263 op/s 2023-03-09T14:48:21.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:48:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:48:21.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:48:21 smithi181 ceph-mon[94189]: pgmap v71: 129 pgs: 129 active+clean; 6.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 17 MiB/s wr, 263 op/s 2023-03-09T14:48:21.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:48:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:48:22.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:48:21 smithi038 ceph-mon[92422]: pgmap v71: 129 pgs: 129 active+clean; 6.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 17 MiB/s wr, 263 op/s 2023-03-09T14:48:22.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:48:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:48:22.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:48:22 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:48:22.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:48:22 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:48:22.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:48:22 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:48:22.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:48:22 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:48:22.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:48:22 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:48:22.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:48:22 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:48:22.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:48:22 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:48:22.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:48:22 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:48:22.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:48:22 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:48:22.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:48:22 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:48:23.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:48:22 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:48:23.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:48:22 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:48:23.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:48:22 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:48:23.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:48:22 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:48:23.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:48:22 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:48:23.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:48:23 smithi122 ceph-mon[93869]: pgmap v72: 129 pgs: 129 active+clean; 6.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 17 MiB/s wr, 351 op/s 2023-03-09T14:48:23.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:48:23 smithi181 ceph-mon[94189]: pgmap v72: 129 pgs: 129 active+clean; 6.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 17 MiB/s wr, 351 op/s 2023-03-09T14:48:24.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:48:23 smithi038 ceph-mon[92422]: pgmap v72: 129 pgs: 129 active+clean; 6.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 17 MiB/s wr, 351 op/s 2023-03-09T14:48:24.129 INFO:teuthology.orchestra.run.smithi038.stderr:41000000 of 50000000 tuples (82%) done (elapsed 119.27 s, remaining 26.18 s) 2023-03-09T14:48:24.225 INFO:teuthology.orchestra.run.smithi038.stderr:41100000 of 50000000 tuples (82%) done (elapsed 119.36 s, remaining 25.85 s) 2023-03-09T14:48:25.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:48:25 smithi122 ceph-mon[93869]: pgmap v73: 129 pgs: 129 active+clean; 6.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 16 MiB/s wr, 213 op/s 2023-03-09T14:48:25.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:48:25 smithi181 ceph-mon[94189]: pgmap v73: 129 pgs: 129 active+clean; 6.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 16 MiB/s wr, 213 op/s 2023-03-09T14:48:26.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:48:25 smithi038 ceph-mon[92422]: pgmap v73: 129 pgs: 129 active+clean; 6.9 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 16 MiB/s wr, 213 op/s 2023-03-09T14:48:27.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:48:27 smithi122 ceph-mon[93869]: pgmap v74: 129 pgs: 129 active+clean; 7.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 59 MiB/s wr, 333 op/s 2023-03-09T14:48:27.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:48:27 smithi181 ceph-mon[94189]: pgmap v74: 129 pgs: 129 active+clean; 7.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 59 MiB/s wr, 333 op/s 2023-03-09T14:48:28.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:48:27 smithi038 ceph-mon[92422]: pgmap v74: 129 pgs: 129 active+clean; 7.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 59 MiB/s wr, 333 op/s 2023-03-09T14:48:29.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:48:29 smithi122 ceph-mon[93869]: pgmap v75: 129 pgs: 129 active+clean; 7.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 44 MiB/s wr, 246 op/s 2023-03-09T14:48:29.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:48:29 smithi181 ceph-mon[94189]: pgmap v75: 129 pgs: 129 active+clean; 7.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 44 MiB/s wr, 246 op/s 2023-03-09T14:48:30.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:48:29 smithi038 ceph-mon[92422]: pgmap v75: 129 pgs: 129 active+clean; 7.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 44 MiB/s wr, 246 op/s 2023-03-09T14:48:31.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:48:31 smithi122 ceph-mon[93869]: pgmap v76: 129 pgs: 129 active+clean; 7.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 44 MiB/s wr, 274 op/s 2023-03-09T14:48:31.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:48:31 smithi181 ceph-mon[94189]: pgmap v76: 129 pgs: 129 active+clean; 7.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 44 MiB/s wr, 274 op/s 2023-03-09T14:48:32.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:48:31 smithi038 ceph-mon[92422]: pgmap v76: 129 pgs: 129 active+clean; 7.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 44 MiB/s wr, 274 op/s 2023-03-09T14:48:33.010 INFO:teuthology.orchestra.run.smithi038.stderr:41200000 of 50000000 tuples (82%) done (elapsed 128.15 s, remaining 27.37 s) 2023-03-09T14:48:33.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:48:33 smithi122 ceph-mon[93869]: pgmap v77: 129 pgs: 129 active+clean; 7.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 44 MiB/s wr, 330 op/s 2023-03-09T14:48:33.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:48:33 smithi181 ceph-mon[94189]: pgmap v77: 129 pgs: 129 active+clean; 7.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 44 MiB/s wr, 330 op/s 2023-03-09T14:48:34.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:48:33 smithi038 ceph-mon[92422]: pgmap v77: 129 pgs: 129 active+clean; 7.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 44 MiB/s wr, 330 op/s 2023-03-09T14:48:35.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:48:35 smithi122 ceph-mon[93869]: pgmap v78: 129 pgs: 129 active+clean; 7.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 44 MiB/s wr, 243 op/s 2023-03-09T14:48:35.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:48:35 smithi181 ceph-mon[94189]: pgmap v78: 129 pgs: 129 active+clean; 7.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 44 MiB/s wr, 243 op/s 2023-03-09T14:48:36.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:48:35 smithi038 ceph-mon[92422]: pgmap v78: 129 pgs: 129 active+clean; 7.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 44 MiB/s wr, 243 op/s 2023-03-09T14:48:37.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:48:37 smithi122 ceph-mon[93869]: pgmap v79: 129 pgs: 129 active+clean; 7.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 45 MiB/s wr, 373 op/s 2023-03-09T14:48:37.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:48:37 smithi181 ceph-mon[94189]: pgmap v79: 129 pgs: 129 active+clean; 7.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 45 MiB/s wr, 373 op/s 2023-03-09T14:48:38.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:48:37 smithi038 ceph-mon[92422]: pgmap v79: 129 pgs: 129 active+clean; 7.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 45 MiB/s wr, 373 op/s 2023-03-09T14:48:38.961 INFO:teuthology.orchestra.run.smithi038.stderr:41300000 of 50000000 tuples (82%) done (elapsed 134.10 s, remaining 28.25 s) 2023-03-09T14:48:39.178 INFO:teuthology.orchestra.run.smithi038.stderr:41400000 of 50000000 tuples (82%) done (elapsed 134.32 s, remaining 27.90 s) 2023-03-09T14:48:39.732 INFO:teuthology.orchestra.run.smithi038.stderr:41500000 of 50000000 tuples (83%) done (elapsed 134.87 s, remaining 27.62 s) 2023-03-09T14:48:39.876 INFO:teuthology.orchestra.run.smithi038.stderr:41600000 of 50000000 tuples (83%) done (elapsed 135.01 s, remaining 27.26 s) 2023-03-09T14:48:39.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:48:39 smithi122 ceph-mon[93869]: pgmap v80: 129 pgs: 129 active+clean; 7.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 1.7 MiB/s wr, 254 op/s 2023-03-09T14:48:39.981 INFO:teuthology.orchestra.run.smithi038.stderr:41700000 of 50000000 tuples (83%) done (elapsed 135.12 s, remaining 26.89 s) 2023-03-09T14:48:39.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:48:39 smithi181 ceph-mon[94189]: pgmap v80: 129 pgs: 129 active+clean; 7.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 1.7 MiB/s wr, 254 op/s 2023-03-09T14:48:40.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:48:39 smithi038 ceph-mon[92422]: pgmap v80: 129 pgs: 129 active+clean; 7.3 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 1.7 MiB/s wr, 254 op/s 2023-03-09T14:48:40.069 INFO:teuthology.orchestra.run.smithi038.stderr:41800000 of 50000000 tuples (83%) done (elapsed 135.21 s, remaining 26.52 s) 2023-03-09T14:48:40.464 INFO:teuthology.orchestra.run.smithi038.stderr:41900000 of 50000000 tuples (83%) done (elapsed 135.60 s, remaining 26.21 s) 2023-03-09T14:48:40.915 INFO:teuthology.orchestra.run.smithi038.stderr:42000000 of 50000000 tuples (84%) done (elapsed 136.05 s, remaining 25.91 s) 2023-03-09T14:48:41.012 INFO:teuthology.orchestra.run.smithi038.stderr:42100000 of 50000000 tuples (84%) done (elapsed 136.15 s, remaining 25.55 s) 2023-03-09T14:48:41.095 INFO:teuthology.orchestra.run.smithi038.stderr:42200000 of 50000000 tuples (84%) done (elapsed 136.23 s, remaining 25.18 s) 2023-03-09T14:48:41.191 INFO:teuthology.orchestra.run.smithi038.stderr:42300000 of 50000000 tuples (84%) done (elapsed 136.33 s, remaining 24.82 s) 2023-03-09T14:48:41.288 INFO:teuthology.orchestra.run.smithi038.stderr:42400000 of 50000000 tuples (84%) done (elapsed 136.43 s, remaining 24.45 s) 2023-03-09T14:48:41.403 INFO:teuthology.orchestra.run.smithi038.stderr:42500000 of 50000000 tuples (85%) done (elapsed 136.54 s, remaining 24.10 s) 2023-03-09T14:48:41.501 INFO:teuthology.orchestra.run.smithi038.stderr:42600000 of 50000000 tuples (85%) done (elapsed 136.64 s, remaining 23.74 s) 2023-03-09T14:48:41.598 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-03-09T14:48:41.600 INFO:teuthology.orchestra.run.smithi038.stderr:42700000 of 50000000 tuples (85%) done (elapsed 136.74 s, remaining 23.38 s) 2023-03-09T14:48:41.703 INFO:teuthology.orchestra.run.smithi038.stderr:42800000 of 50000000 tuples (85%) done (elapsed 136.84 s, remaining 23.02 s) 2023-03-09T14:48:41.741 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.738+0000 7fc7e9122700 1 -- 172.21.15.38:0/3893649671 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc7e4067ac0 msgr2=0x7fc7e4067e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:48:41.742 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.738+0000 7fc7e9122700 1 --2- 172.21.15.38:0/3893649671 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc7e4067ac0 0x7fc7e4067e90 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc7d4009990 tx=0x7fc7d4009ca0 comp rx=0 tx=0).stop 2023-03-09T14:48:41.742 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.739+0000 7fc7e9122700 1 -- 172.21.15.38:0/3893649671 shutdown_connections 2023-03-09T14:48:41.742 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.739+0000 7fc7e9122700 1 --2- 172.21.15.38:0/3893649671 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7e410c8c0 0x7fc7e410eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:48:41.742 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.739+0000 7fc7e9122700 1 --2- 172.21.15.38:0/3893649671 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc7e4067ac0 0x7fc7e4067e90 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:48:41.742 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.739+0000 7fc7e9122700 1 --2- 172.21.15.38:0/3893649671 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7e40683d0 0x7fc7e410c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:48:41.742 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.739+0000 7fc7e9122700 1 -- 172.21.15.38:0/3893649671 >> 172.21.15.38:0/3893649671 conn(0x7fc7e4075c60 msgr2=0x7fc7e4076060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:48:41.742 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.739+0000 7fc7e9122700 1 -- 172.21.15.38:0/3893649671 shutdown_connections 2023-03-09T14:48:41.743 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.739+0000 7fc7e9122700 1 -- 172.21.15.38:0/3893649671 wait complete. 2023-03-09T14:48:41.744 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.740+0000 7fc7e9122700 1 Processor -- start 2023-03-09T14:48:41.744 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.740+0000 7fc7e9122700 1 -- start start 2023-03-09T14:48:41.745 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.740+0000 7fc7e9122700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7e4067ac0 0x7fc7e41982a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:48:41.745 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.740+0000 7fc7e9122700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc7e40683d0 0x7fc7e41987e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:48:41.745 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.740+0000 7fc7e9122700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7e410c8c0 0x7fc7e419cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:48:41.745 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.740+0000 7fc7e9122700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7e419d230 con 0x7fc7e410c8c0 2023-03-09T14:48:41.746 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.740+0000 7fc7e9122700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7e419d3a0 con 0x7fc7e4067ac0 2023-03-09T14:48:41.746 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.740+0000 7fc7e9122700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7e419d560 con 0x7fc7e40683d0 2023-03-09T14:48:41.746 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.740+0000 7fc7e2d9d700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7e4067ac0 0x7fc7e41982a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:48:41.746 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.740+0000 7fc7e2d9d700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7e4067ac0 0x7fc7e41982a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.38:49046/0 (socket says 172.21.15.38:49046) 2023-03-09T14:48:41.746 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.740+0000 7fc7e2d9d700 1 -- 172.21.15.38:0/1892583511 learned_addr learned my addr 172.21.15.38:0/1892583511 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:48:41.746 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.740+0000 7fc7e359e700 1 --2- 172.21.15.38:0/1892583511 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7e410c8c0 0x7fc7e419cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:48:41.746 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.740+0000 7fc7e259c700 1 --2- 172.21.15.38:0/1892583511 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc7e40683d0 0x7fc7e41987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:48:41.747 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.741+0000 7fc7e2d9d700 1 -- 172.21.15.38:0/1892583511 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc7e40683d0 msgr2=0x7fc7e41987e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:48:41.747 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.741+0000 7fc7e2d9d700 1 --2- 172.21.15.38:0/1892583511 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc7e40683d0 0x7fc7e41987e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:48:41.747 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.741+0000 7fc7e2d9d700 1 -- 172.21.15.38:0/1892583511 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7e410c8c0 msgr2=0x7fc7e419cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:48:41.747 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.741+0000 7fc7e2d9d700 1 --2- 172.21.15.38:0/1892583511 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7e410c8c0 0x7fc7e419cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:48:41.747 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.741+0000 7fc7e2d9d700 1 -- 172.21.15.38:0/1892583511 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7d40095f0 con 0x7fc7e4067ac0 2023-03-09T14:48:41.747 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.741+0000 7fc7e2d9d700 1 --2- 172.21.15.38:0/1892583511 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7e4067ac0 0x7fc7e41982a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc7d4005910 tx=0x7fc7d4005720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:48:41.747 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.741+0000 7fc7cbfff700 1 -- 172.21.15.38:0/1892583511 <== mon.2 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7d401d070 con 0x7fc7e4067ac0 2023-03-09T14:48:41.747 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.741+0000 7fc7e9122700 1 -- 172.21.15.38:0/1892583511 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7e4112130 con 0x7fc7e4067ac0 2023-03-09T14:48:41.748 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.741+0000 7fc7cbfff700 1 -- 172.21.15.38:0/1892583511 <== mon.2 v2:172.21.15.122:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc7d4022470 con 0x7fc7e4067ac0 2023-03-09T14:48:41.748 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.741+0000 7fc7cbfff700 1 -- 172.21.15.38:0/1892583511 <== mon.2 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7d4017650 con 0x7fc7e4067ac0 2023-03-09T14:48:41.748 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.741+0000 7fc7e9122700 1 -- 172.21.15.38:0/1892583511 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7e4111fa0 con 0x7fc7e4067ac0 2023-03-09T14:48:41.752 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.744+0000 7fc7cbfff700 1 -- 172.21.15.38:0/1892583511 <== mon.2 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 25) v1 ==== 133295+0+0 (secure 0 0 0) 0x7fc7d4029030 con 0x7fc7e4067ac0 2023-03-09T14:48:41.752 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.744+0000 7fc7cbfff700 1 --2- 172.21.15.38:0/1892583511 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fc7cc09f4a0 0x7fc7cc0a1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:48:41.752 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.744+0000 7fc7cbfff700 1 -- 172.21.15.38:0/1892583511 <== mon.2 v2:172.21.15.122:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9489+0+0 (secure 0 0 0) 0x7fc7d40cb590 con 0x7fc7e4067ac0 2023-03-09T14:48:41.752 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.744+0000 7fc7e259c700 1 --2- 172.21.15.38:0/1892583511 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fc7cc09f4a0 0x7fc7cc0a1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:48:41.753 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.745+0000 7fc7f52abb80 1 Processor -- start 2023-03-09T14:48:41.753 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.745+0000 7fc7f52abb80 1 -- start start 2023-03-09T14:48:41.753 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.745+0000 7fc7e259c700 1 --2- 172.21.15.38:0/1892583511 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fc7cc09f4a0 0x7fc7cc0a1950 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc7e4199860 tx=0x7fc7d8009260 comp rx=0 tx=0).ready entity=mgr.15558 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:48:41.753 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.745+0000 7fc7f52abb80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563013c33830 0x563013c33c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:48:41.753 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.745+0000 7fc7f52abb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563013c34140 0x563013c2daa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:48:41.753 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.745+0000 7fc7f52abb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563013c2dfe0 0x563013c2e410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:48:41.754 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.745+0000 7fc7f52abb80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x563013b17da0 con 0x563013c33830 2023-03-09T14:48:41.754 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.745+0000 7fc7f52abb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x563013b86780 con 0x563013c34140 2023-03-09T14:48:41.754 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.745+0000 7fc7f52abb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x563013b7fcb0 con 0x563013c2dfe0 2023-03-09T14:48:41.754 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.745+0000 7fc7e2d9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563013c33830 0x563013c33c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:48:41.754 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.745+0000 7fc7e2d9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563013c33830 0x563013c33c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39864/0 (socket says 172.21.15.38:39864) 2023-03-09T14:48:41.754 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.745+0000 7fc7e259c700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563013c34140 0x563013c2daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:48:41.754 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.745+0000 7fc7e2d9d700 1 -- 172.21.15.38:0/621187636 learned_addr learned my addr 172.21.15.38:0/621187636 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:48:41.755 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.745+0000 7fc7e359e700 1 --2- 172.21.15.38:0/621187636 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563013c2dfe0 0x563013c2e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:48:41.755 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.745+0000 7fc7e2d9d700 1 -- 172.21.15.38:0/621187636 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563013c2dfe0 msgr2=0x563013c2e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:48:41.755 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.745+0000 7fc7e2d9d700 1 --2- 172.21.15.38:0/621187636 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563013c2dfe0 0x563013c2e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:48:41.755 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.745+0000 7fc7e2d9d700 1 -- 172.21.15.38:0/621187636 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563013c34140 msgr2=0x563013c2daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:48:41.755 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.745+0000 7fc7e2d9d700 1 --2- 172.21.15.38:0/621187636 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563013c34140 0x563013c2daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:48:41.755 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.745+0000 7fc7e2d9d700 1 -- 172.21.15.38:0/621187636 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563013b46190 con 0x563013c33830 2023-03-09T14:48:41.755 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.745+0000 7fc7e259c700 1 --2- 172.21.15.38:0/621187636 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563013c34140 0x563013c2daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:48:41.756 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.746+0000 7fc7e2d9d700 1 --2- 172.21.15.38:0/621187636 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563013c33830 0x563013c33c00 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fc7d400b520 tx=0x7fc7d4022c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc2aba6f41c3c0ef server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:48:41.756 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.746+0000 7fc7c97fa700 1 -- 172.21.15.38:0/621187636 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7d4098d00 con 0x563013c33830 2023-03-09T14:48:41.756 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.746+0000 7fc7c97fa700 1 -- 172.21.15.38:0/621187636 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc7d4098b20 con 0x563013c33830 2023-03-09T14:48:41.756 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.746+0000 7fc7c97fa700 1 -- 172.21.15.38:0/621187636 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7d40cb170 con 0x563013c33830 2023-03-09T14:48:41.756 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.746+0000 7fc7f52abb80 1 -- 172.21.15.38:0/621187636 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563013c33830 msgr2=0x563013c33c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:48:41.756 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.746+0000 7fc7f52abb80 1 --2- 172.21.15.38:0/621187636 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563013c33830 0x563013c33c00 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fc7d400b520 tx=0x7fc7d4022c90 comp rx=0 tx=0).stop 2023-03-09T14:48:41.756 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.747+0000 7fc7f52abb80 1 -- 172.21.15.38:0/621187636 shutdown_connections 2023-03-09T14:48:41.757 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.747+0000 7fc7f52abb80 1 --2- 172.21.15.38:0/621187636 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563013c33830 0x563013c33c00 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:48:41.757 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.747+0000 7fc7f52abb80 1 --2- 172.21.15.38:0/621187636 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563013c2dfe0 0x563013c2e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:48:41.757 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.747+0000 7fc7f52abb80 1 --2- 172.21.15.38:0/621187636 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563013c34140 0x563013c2daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:48:41.757 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.747+0000 7fc7f52abb80 1 -- 172.21.15.38:0/621187636 >> 172.21.15.38:0/621187636 conn(0x563013c28a20 msgr2=0x563013c28df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:48:41.757 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.747+0000 7fc7f52abb80 1 -- 172.21.15.38:0/621187636 shutdown_connections 2023-03-09T14:48:41.757 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.747+0000 7fc7f52abb80 1 -- 172.21.15.38:0/621187636 wait complete. 2023-03-09T14:48:41.757 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.747+0000 7fc7f52abb80 1 -- start start 2023-03-09T14:48:41.757 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.747+0000 7fc7f52abb80 1 Processor -- start 2023-03-09T14:48:41.758 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.747+0000 7fc7f52abb80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563013c2dfe0 0x563013c3a080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:48:41.758 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.747+0000 7fc7f52abb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563013c33830 0x563013c37300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:48:41.758 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.747+0000 7fc7f52abb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563013c34140 0x563013c37930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:48:41.758 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.747+0000 7fc7f52abb80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x563013a03920 con 0x563013c2dfe0 2023-03-09T14:48:41.758 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.747+0000 7fc7f52abb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x563013c38060 con 0x563013c33830 2023-03-09T14:48:41.758 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.747+0000 7fc7f52abb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x563013c43930 con 0x563013c34140 2023-03-09T14:48:41.758 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.747+0000 7fc7f52abb80 10 client.0 _ll_register_callbacks cb 0x563013b25520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:48:41.759 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.748+0000 7fc7e2d9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563013c2dfe0 0x563013c3a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:48:41.759 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.748+0000 7fc7e259c700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563013c33830 0x563013c37300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:48:41.759 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.748+0000 7fc7e2d9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563013c2dfe0 0x563013c3a080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39870/0 (socket says 172.21.15.38:39870) 2023-03-09T14:48:41.759 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.748+0000 7fc7e2d9d700 1 -- 172.21.15.38:0/3241135210 learned_addr learned my addr 172.21.15.38:0/3241135210 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:48:41.759 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.748+0000 7fc7e359e700 1 --2- 172.21.15.38:0/3241135210 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563013c34140 0x563013c37930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:48:41.759 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.748+0000 7fc7e2d9d700 1 -- 172.21.15.38:0/3241135210 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563013c34140 msgr2=0x563013c37930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:48:41.759 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.748+0000 7fc7e2d9d700 1 --2- 172.21.15.38:0/3241135210 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563013c34140 0x563013c37930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:48:41.759 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.748+0000 7fc7e2d9d700 1 -- 172.21.15.38:0/3241135210 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563013c33830 msgr2=0x563013c37300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:48:41.760 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.748+0000 7fc7e2d9d700 1 --2- 172.21.15.38:0/3241135210 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563013c33830 0x563013c37300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:48:41.760 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.748+0000 7fc7e2d9d700 1 -- 172.21.15.38:0/3241135210 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7d8017070 con 0x563013c2dfe0 2023-03-09T14:48:41.760 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.748+0000 7fc7e259c700 1 --2- 172.21.15.38:0/3241135210 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563013c33830 0x563013c37300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:48:41.760 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.748+0000 7fc7e2d9d700 1 --2- 172.21.15.38:0/3241135210 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563013c2dfe0 0x563013c3a080 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fc7d4099820 tx=0x7fc7d40cb4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9622b922aac8d2aa server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:48:41.760 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.748+0000 7fc7c6ff5700 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2023-03-09T14:48:41.760 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.748+0000 7fc7c6ff5700 1 -- 172.21.15.38:0/3241135210 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7d40b7120 con 0x563013c2dfe0 2023-03-09T14:48:41.760 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.748+0000 7fc7c6ff5700 1 -- 172.21.15.38:0/3241135210 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9489+0+0 (secure 0 0 0) 0x7fc7d40a1320 con 0x563013c2dfe0 2023-03-09T14:48:41.760 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.748+0000 7fc7f52abb80 1 -- 172.21.15.38:0/3241135210 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563013b4d190 con 0x563013c2dfe0 2023-03-09T14:48:41.761 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.748+0000 7fc7c6ff5700 1 -- 172.21.15.38:0/3241135210 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc7d40991c0 con 0x563013c2dfe0 2023-03-09T14:48:41.761 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.748+0000 7fc7f52abb80 10 client.15915 fetch_fsmap learned FSMap version 25 2023-03-09T14:48:41.761 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.748+0000 7fc7f52abb80 1 -- 172.21.15.38:0/3241135210 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x7fc7e4111fa0 con 0x563013c2dfe0 2023-03-09T14:48:41.761 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.749+0000 7fc7c6ff5700 1 -- 172.21.15.38:0/3241135210 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x563013b4d190 con 0x563013c2dfe0 2023-03-09T14:48:41.761 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.749+0000 7fc7c6ff5700 1 -- 172.21.15.38:0/3241135210 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 25) v1 ==== 7478+0+0 (secure 0 0 0) 0x7fc7d40a0f70 con 0x563013c2dfe0 2023-03-09T14:48:41.761 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.749+0000 7fc7f52abb80 10 client.15915 fetch_fsmap finished waiting for FSMap version 25 2023-03-09T14:48:41.761 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.749+0000 7fc7f52abb80 10 client.15915 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T14:48:41.762 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.749+0000 7fc7f52abb80 20 client.15915 populate_metadata read hostname 'smithi038' 2023-03-09T14:48:41.762 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.749+0000 7fc7f52abb80 1 --2- 172.21.15.38:0/3241135210 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x563013c47590 0x563013c499a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:48:41.762 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.749+0000 7fc7f52abb80 4 client.15915 mds_command: new command op to 24530 tid=0[{"prefix": "get_command_descriptions"}] 2023-03-09T14:48:41.762 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.749+0000 7fc7f52abb80 1 -- 172.21.15.38:0/3241135210 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563013a03f60 con 0x563013c47590 2023-03-09T14:48:41.762 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.749+0000 7fc7e259c700 1 --2- 172.21.15.38:0/3241135210 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x563013c47590 0x563013c499a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:48:41.763 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.750+0000 7fc7e259c700 1 --2- 172.21.15.38:0/3241135210 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x563013c47590 0x563013c499a0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c79c8e8ed039ccd server_cookie=a8a2969e72155f7c in_seq=0 out_seq=0 2023-03-09T14:48:41.763 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.750+0000 7fc7c6ff5700 10 client.15915 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:48:41.764 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.751+0000 7fc7c6ff5700 1 -- 172.21.15.38:0/3241135210 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18278 (crc 0 0 0) 0x563013a03f60 con 0x563013c47590 2023-03-09T14:48:41.764 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.751+0000 7fc7c6ff5700 10 client.15915 handle_command_reply: tid=0 2023-03-09T14:48:41.764 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.751+0000 7fc7f52abb80 1 client.15915 shutdown 2023-03-09T14:48:41.764 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.751+0000 7fc7f52abb80 1 -- 172.21.15.38:0/3241135210 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563013c2dfe0 msgr2=0x563013c3a080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:48:41.764 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.751+0000 7fc7f52abb80 1 --2- 172.21.15.38:0/3241135210 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563013c2dfe0 0x563013c3a080 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fc7d4099820 tx=0x7fc7d40cb4e0 comp rx=0 tx=0).stop 2023-03-09T14:48:41.764 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.751+0000 7fc7f52abb80 1 -- 172.21.15.38:0/3241135210 shutdown_connections 2023-03-09T14:48:41.764 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.751+0000 7fc7f52abb80 1 --2- 172.21.15.38:0/3241135210 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x563013c47590 0x563013c499a0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:48:41.764 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.751+0000 7fc7f52abb80 1 --2- 172.21.15.38:0/3241135210 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563013c2dfe0 0x563013c3a080 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:48:41.765 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.751+0000 7fc7c6ff5700 0 client.15915 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:48:41.765 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.751+0000 7fc7f52abb80 1 --2- 172.21.15.38:0/3241135210 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563013c34140 0x563013c37930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:48:41.765 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.751+0000 7fc7f52abb80 1 --2- 172.21.15.38:0/3241135210 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563013c33830 0x563013c37300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:48:41.765 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.751+0000 7fc7f52abb80 1 -- 172.21.15.38:0/3241135210 >> 172.21.15.38:0/3241135210 conn(0x563013c28a20 msgr2=0x563013c27c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:48:41.765 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.752+0000 7fc7f52abb80 1 -- 172.21.15.38:0/3241135210 shutdown_connections 2023-03-09T14:48:41.765 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.752+0000 7fc7f52abb80 1 -- 172.21.15.38:0/3241135210 wait complete. 2023-03-09T14:48:41.765 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.752+0000 7fc7f52abb80 20 client.15915 trim_cache size 0 max 16384 2023-03-09T14:48:41.775 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.772+0000 7fc7f52abb80 1 Processor -- start 2023-03-09T14:48:41.776 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.772+0000 7fc7f52abb80 1 -- start start 2023-03-09T14:48:41.776 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.772+0000 7fc7f52abb80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563013c2dfe0 0x563013c3e900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:48:41.776 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.772+0000 7fc7f52abb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563013c33830 0x563013c43c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:48:41.776 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.772+0000 7fc7f52abb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563013c34140 0x563013c39c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:48:41.776 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.772+0000 7fc7f52abb80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x563013c284d0 con 0x563013c2dfe0 2023-03-09T14:48:41.777 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.772+0000 7fc7f52abb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x563013c286f0 con 0x563013c34140 2023-03-09T14:48:41.777 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.772+0000 7fc7f52abb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x563013c2ea30 con 0x563013c33830 2023-03-09T14:48:41.777 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.773+0000 7fc7e2d9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563013c2dfe0 0x563013c3e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:48:41.777 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.773+0000 7fc7e2d9d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563013c2dfe0 0x563013c3e900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39878/0 (socket says 172.21.15.38:39878) 2023-03-09T14:48:41.777 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.773+0000 7fc7e2d9d700 1 -- 172.21.15.38:0/110041725 learned_addr learned my addr 172.21.15.38:0/110041725 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:48:41.777 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.773+0000 7fc7e259c700 1 --2- 172.21.15.38:0/110041725 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563013c33830 0x563013c43c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:48:41.777 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.773+0000 7fc7e359e700 1 --2- 172.21.15.38:0/110041725 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563013c34140 0x563013c39c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:48:41.778 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.773+0000 7fc7e259c700 1 -- 172.21.15.38:0/110041725 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563013c34140 msgr2=0x563013c39c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:48:41.778 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.773+0000 7fc7e259c700 1 --2- 172.21.15.38:0/110041725 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563013c34140 0x563013c39c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:48:41.778 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.773+0000 7fc7e259c700 1 -- 172.21.15.38:0/110041725 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563013c2dfe0 msgr2=0x563013c3e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:48:41.778 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.773+0000 7fc7e259c700 1 --2- 172.21.15.38:0/110041725 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563013c2dfe0 0x563013c3e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:48:41.778 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.773+0000 7fc7e259c700 1 -- 172.21.15.38:0/110041725 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563013996c60 con 0x563013c33830 2023-03-09T14:48:41.778 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.773+0000 7fc7e2d9d700 1 --2- 172.21.15.38:0/110041725 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563013c2dfe0 0x563013c3e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:48:41.778 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.773+0000 7fc7e359e700 1 --2- 172.21.15.38:0/110041725 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563013c34140 0x563013c39c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:48:41.779 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.773+0000 7fc7e259c700 1 --2- 172.21.15.38:0/110041725 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563013c33830 0x563013c43c90 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x563013c4a2b0 tx=0x7fc7d8014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b6701c621af122a7 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:48:41.779 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.773+0000 7fc7c9ffb700 1 -- 172.21.15.38:0/110041725 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7d8022070 con 0x563013c33830 2023-03-09T14:48:41.779 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.773+0000 7fc7c9ffb700 1 -- 172.21.15.38:0/110041725 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc7d8023040 con 0x563013c33830 2023-03-09T14:48:41.779 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.774+0000 7fc7c9ffb700 1 -- 172.21.15.38:0/110041725 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7d80045b0 con 0x563013c33830 2023-03-09T14:48:41.782 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.774+0000 7fc7f52abb80 1 -- 172.21.15.38:0/110041725 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563013c33830 msgr2=0x563013c43c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:48:41.782 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.774+0000 7fc7f52abb80 1 --2- 172.21.15.38:0/110041725 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563013c33830 0x563013c43c90 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x563013c4a2b0 tx=0x7fc7d8014000 comp rx=0 tx=0).stop 2023-03-09T14:48:41.783 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.774+0000 7fc7f52abb80 1 -- 172.21.15.38:0/110041725 shutdown_connections 2023-03-09T14:48:41.783 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.774+0000 7fc7f52abb80 1 --2- 172.21.15.38:0/110041725 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563013c2dfe0 0x563013c3e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:48:41.783 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.774+0000 7fc7f52abb80 1 --2- 172.21.15.38:0/110041725 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563013c34140 0x563013c39c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:48:41.783 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.774+0000 7fc7f52abb80 1 --2- 172.21.15.38:0/110041725 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563013c33830 0x563013c43c90 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:48:41.783 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.774+0000 7fc7f52abb80 1 -- 172.21.15.38:0/110041725 >> 172.21.15.38:0/110041725 conn(0x563013c28a20 msgr2=0x563013c43090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:48:41.783 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.774+0000 7fc7f52abb80 1 -- 172.21.15.38:0/110041725 shutdown_connections 2023-03-09T14:48:41.783 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.774+0000 7fc7f52abb80 1 -- 172.21.15.38:0/110041725 wait complete. 2023-03-09T14:48:41.784 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.775+0000 7fc7f52abb80 1 -- start start 2023-03-09T14:48:41.784 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.775+0000 7fc7f52abb80 1 Processor -- start 2023-03-09T14:48:41.784 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.775+0000 7fc7f52abb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563013c2dfe0 0x563013c3e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:48:41.784 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.775+0000 7fc7f52abb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563013c33830 0x563013c99430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:48:41.784 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.775+0000 7fc7f52abb80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563013c34140 0x563013c3cd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:48:41.784 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.775+0000 7fc7f52abb80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x563013c3d270 con 0x563013c34140 2023-03-09T14:48:41.784 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.775+0000 7fc7f52abb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x563013c3d3e0 con 0x563013c2dfe0 2023-03-09T14:48:41.784 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.775+0000 7fc7f52abb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x563013c3d5a0 con 0x563013c33830 2023-03-09T14:48:41.785 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.775+0000 7fc7f52abb80 10 client.0 _ll_register_callbacks cb 0x563013b25520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:48:41.785 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.775+0000 7fc7e2d9d700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563013c2dfe0 0x563013c3e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:48:41.785 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.775+0000 7fc7e259c700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563013c33830 0x563013c99430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:48:41.785 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.775+0000 7fc7e2d9d700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563013c2dfe0 0x563013c3e8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.38:49074/0 (socket says 172.21.15.38:49074) 2023-03-09T14:48:41.785 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.775+0000 7fc7e2d9d700 1 -- 172.21.15.38:0/2782055695 learned_addr learned my addr 172.21.15.38:0/2782055695 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:48:41.785 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.775+0000 7fc7e359e700 1 --2- 172.21.15.38:0/2782055695 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563013c34140 0x563013c3cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:48:41.785 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.776+0000 7fc7e259c700 1 -- 172.21.15.38:0/2782055695 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563013c2dfe0 msgr2=0x563013c3e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:48:41.786 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.776+0000 7fc7e259c700 1 --2- 172.21.15.38:0/2782055695 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563013c2dfe0 0x563013c3e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:48:41.786 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.776+0000 7fc7e259c700 1 -- 172.21.15.38:0/2782055695 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563013c34140 msgr2=0x563013c3cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:48:41.786 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.776+0000 7fc7e259c700 1 --2- 172.21.15.38:0/2782055695 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563013c34140 0x563013c3cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:48:41.786 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.776+0000 7fc7e259c700 1 -- 172.21.15.38:0/2782055695 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7e4111fa0 con 0x563013c33830 2023-03-09T14:48:41.786 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.776+0000 7fc7e259c700 1 --2- 172.21.15.38:0/2782055695 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563013c33830 0x563013c99430 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x563013c235c0 tx=0x7fc7d8004050 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa1b56f4496a12dc server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:48:41.786 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.776+0000 7fc7c6ff5700 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2023-03-09T14:48:41.786 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.776+0000 7fc7c6ff5700 1 -- 172.21.15.38:0/2782055695 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7d8022040 con 0x563013c33830 2023-03-09T14:48:41.786 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.776+0000 7fc7f52abb80 1 -- 172.21.15.38:0/2782055695 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563013b3d1b0 con 0x563013c33830 2023-03-09T14:48:41.787 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.776+0000 7fc7c6ff5700 1 -- 172.21.15.38:0/2782055695 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9489+0+0 (secure 0 0 0) 0x7fc7d801dd10 con 0x563013c33830 2023-03-09T14:48:41.787 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.776+0000 7fc7c6ff5700 1 -- 172.21.15.38:0/2782055695 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc7d80041d0 con 0x563013c33830 2023-03-09T14:48:41.787 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.776+0000 7fc7f52abb80 10 client.25522 fetch_fsmap learned FSMap version 25 2023-03-09T14:48:41.787 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.776+0000 7fc7f52abb80 1 -- 172.21.15.38:0/2782055695 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x7fc7d80299c0 con 0x563013c33830 2023-03-09T14:48:41.787 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.776+0000 7fc7c6ff5700 1 -- 172.21.15.38:0/2782055695 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x563013b3d1b0 con 0x563013c33830 2023-03-09T14:48:41.787 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.777+0000 7fc7c6ff5700 1 -- 172.21.15.38:0/2782055695 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 25) v1 ==== 7478+0+0 (secure 0 0 0) 0x7fc7d8031070 con 0x563013c33830 2023-03-09T14:48:41.787 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.777+0000 7fc7f52abb80 10 client.25522 fetch_fsmap finished waiting for FSMap version 25 2023-03-09T14:48:41.788 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.777+0000 7fc7f52abb80 10 client.25522 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T14:48:41.788 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.777+0000 7fc7f52abb80 20 client.25522 populate_metadata read hostname 'smithi038' 2023-03-09T14:48:41.788 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.777+0000 7fc7f52abb80 1 --2- 172.21.15.38:0/2782055695 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x563013cad990 0x563013cafda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:48:41.788 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.777+0000 7fc7f52abb80 4 client.25522 mds_command: new command op to 24530 tid=0[{"prefix": "scrub status"}] 2023-03-09T14:48:41.788 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.777+0000 7fc7f52abb80 1 -- 172.21.15.38:0/2782055695 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x563013b271c0 con 0x563013cad990 2023-03-09T14:48:41.788 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.777+0000 7fc7e2d9d700 1 --2- 172.21.15.38:0/2782055695 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x563013cad990 0x563013cafda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:48:41.788 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.777+0000 7fc7c6ff5700 10 client.25522 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:48:41.789 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.777+0000 7fc7e2d9d700 1 --2- 172.21.15.38:0/2782055695 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x563013cad990 0x563013cafda0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=307fa5ce56bf4e3d server_cookie=c9c4ae49f80e223 in_seq=0 out_seq=0 2023-03-09T14:48:41.789 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.778+0000 7fc7c6ff5700 1 -- 172.21.15.38:0/2782055695 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+265 (crc 0 0 0) 0x563013b271c0 con 0x563013cad990 2023-03-09T14:48:41.789 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.778+0000 7fc7c6ff5700 10 client.25522 handle_command_reply: tid=0 2023-03-09T14:48:41.789 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.778+0000 7fc7f52abb80 1 client.25522 shutdown 2023-03-09T14:48:41.789 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.778+0000 7fc7f52abb80 1 -- 172.21.15.38:0/2782055695 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563013c33830 msgr2=0x563013c99430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:48:41.789 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.778+0000 7fc7f52abb80 1 --2- 172.21.15.38:0/2782055695 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563013c33830 0x563013c99430 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x563013c235c0 tx=0x7fc7d8004050 comp rx=0 tx=0).stop 2023-03-09T14:48:41.789 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.778+0000 7fc7f52abb80 1 -- 172.21.15.38:0/2782055695 shutdown_connections 2023-03-09T14:48:41.789 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.778+0000 7fc7f52abb80 1 --2- 172.21.15.38:0/2782055695 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x563013cad990 0x563013cafda0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:48:41.790 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.778+0000 7fc7f52abb80 1 --2- 172.21.15.38:0/2782055695 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x563013c34140 0x563013c3cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:48:41.790 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.778+0000 7fc7c6ff5700 0 client.25522 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:48:41.790 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.778+0000 7fc7f52abb80 1 --2- 172.21.15.38:0/2782055695 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x563013c2dfe0 0x563013c3e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:48:41.790 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.778+0000 7fc7f52abb80 1 --2- 172.21.15.38:0/2782055695 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x563013c33830 0x563013c99430 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:48:41.790 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.778+0000 7fc7f52abb80 1 -- 172.21.15.38:0/2782055695 >> 172.21.15.38:0/2782055695 conn(0x563013c28a20 msgr2=0x563013c43c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:48:41.790 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.778+0000 7fc7f52abb80 1 -- 172.21.15.38:0/2782055695 shutdown_connections 2023-03-09T14:48:41.790 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.778+0000 7fc7f52abb80 1 -- 172.21.15.38:0/2782055695 wait complete. 2023-03-09T14:48:41.790 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.778+0000 7fc7f52abb80 20 client.25522 trim_cache size 0 max 16384 2023-03-09T14:48:41.791 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2023-03-09T14:48:41.791 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "scrub active (3 inodes in the stack)", 2023-03-09T14:48:41.791 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": { 2023-03-09T14:48:41.791 INFO:teuthology.orchestra.run.smithi038.stdout: "a14659f4-d7ea-4c5a-b15b-6138af3f909f": { 2023-03-09T14:48:41.791 INFO:teuthology.orchestra.run.smithi038.stdout: "path": "/", 2023-03-09T14:48:41.792 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "a14659f4-d7ea-4c5a-b15b-6138af3f909f", 2023-03-09T14:48:41.792 INFO:teuthology.orchestra.run.smithi038.stdout: "options": "recursive,force" 2023-03-09T14:48:41.792 INFO:teuthology.orchestra.run.smithi038.stdout: } 2023-03-09T14:48:41.792 INFO:teuthology.orchestra.run.smithi038.stdout: } 2023-03-09T14:48:41.792 INFO:teuthology.orchestra.run.smithi038.stdout:} 2023-03-09T14:48:41.793 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.779+0000 7fc7e9122700 1 -- 172.21.15.38:0/1892583511 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fc7cc09f4a0 msgr2=0x7fc7cc0a1950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:48:41.794 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.779+0000 7fc7e9122700 1 --2- 172.21.15.38:0/1892583511 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fc7cc09f4a0 0x7fc7cc0a1950 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc7e4199860 tx=0x7fc7d8009260 comp rx=0 tx=0).stop 2023-03-09T14:48:41.794 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.779+0000 7fc7e9122700 1 -- 172.21.15.38:0/1892583511 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7e4067ac0 msgr2=0x7fc7e41982a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:48:41.794 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.779+0000 7fc7e9122700 1 --2- 172.21.15.38:0/1892583511 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7e4067ac0 0x7fc7e41982a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc7d4005910 tx=0x7fc7d4005720 comp rx=0 tx=0).stop 2023-03-09T14:48:41.794 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.779+0000 7fc7e9122700 1 -- 172.21.15.38:0/1892583511 shutdown_connections 2023-03-09T14:48:41.794 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.779+0000 7fc7e9122700 1 --2- 172.21.15.38:0/1892583511 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fc7cc09f4a0 0x7fc7cc0a1950 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:48:41.794 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.779+0000 7fc7e9122700 1 --2- 172.21.15.38:0/1892583511 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc7e410c8c0 0x7fc7e419cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:48:41.795 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.779+0000 7fc7e9122700 1 --2- 172.21.15.38:0/1892583511 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc7e4067ac0 0x7fc7e41982a0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:48:41.795 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.779+0000 7fc7e9122700 1 --2- 172.21.15.38:0/1892583511 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc7e40683d0 0x7fc7e41987e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:48:41.795 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.779+0000 7fc7e9122700 1 -- 172.21.15.38:0/1892583511 >> 172.21.15.38:0/1892583511 conn(0x7fc7e4075c60 msgr2=0x7fc7e40fc960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:48:41.795 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.779+0000 7fc7e9122700 1 -- 172.21.15.38:0/1892583511 shutdown_connections 2023-03-09T14:48:41.795 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:48:41.779+0000 7fc7e9122700 1 -- 172.21.15.38:0/1892583511 wait complete. 2023-03-09T14:48:41.798 INFO:tasks.cephfs.filesystem:scrub status for tag:a14659f4-d7ea-4c5a-b15b-6138af3f909f - {'path': '/', 'tag': 'a14659f4-d7ea-4c5a-b15b-6138af3f909f', 'options': 'recursive,force'} 2023-03-09T14:48:41.811 INFO:teuthology.orchestra.run.smithi038.stderr:42900000 of 50000000 tuples (85%) done (elapsed 136.95 s, remaining 22.67 s) 2023-03-09T14:48:41.919 INFO:teuthology.orchestra.run.smithi038.stderr:43000000 of 50000000 tuples (86%) done (elapsed 137.06 s, remaining 22.31 s) 2023-03-09T14:48:41.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:48:41 smithi122 ceph-mon[93869]: pgmap v81: 129 pgs: 129 active+clean; 7.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 5.7 MiB/s wr, 293 op/s 2023-03-09T14:48:41.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:48:41 smithi181 ceph-mon[94189]: pgmap v81: 129 pgs: 129 active+clean; 7.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 5.7 MiB/s wr, 293 op/s 2023-03-09T14:48:42.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:48:41 smithi038 ceph-mon[92422]: pgmap v81: 129 pgs: 129 active+clean; 7.4 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 5.7 MiB/s wr, 293 op/s 2023-03-09T14:48:42.027 INFO:teuthology.orchestra.run.smithi038.stderr:43100000 of 50000000 tuples (86%) done (elapsed 137.17 s, remaining 21.96 s) 2023-03-09T14:48:42.124 INFO:teuthology.orchestra.run.smithi038.stderr:43200000 of 50000000 tuples (86%) done (elapsed 137.26 s, remaining 21.61 s) 2023-03-09T14:48:42.218 INFO:teuthology.orchestra.run.smithi038.stderr:43300000 of 50000000 tuples (86%) done (elapsed 137.36 s, remaining 21.25 s) 2023-03-09T14:48:42.333 INFO:teuthology.orchestra.run.smithi038.stderr:43400000 of 50000000 tuples (86%) done (elapsed 137.47 s, remaining 20.91 s) 2023-03-09T14:48:42.432 INFO:teuthology.orchestra.run.smithi038.stderr:43500000 of 50000000 tuples (87%) done (elapsed 137.57 s, remaining 20.56 s) 2023-03-09T14:48:42.514 INFO:teuthology.orchestra.run.smithi038.stderr:43600000 of 50000000 tuples (87%) done (elapsed 137.65 s, remaining 20.21 s) 2023-03-09T14:48:42.608 INFO:teuthology.orchestra.run.smithi038.stderr:43700000 of 50000000 tuples (87%) done (elapsed 137.75 s, remaining 19.86 s) 2023-03-09T14:48:42.695 INFO:teuthology.orchestra.run.smithi038.stderr:43800000 of 50000000 tuples (87%) done (elapsed 137.83 s, remaining 19.51 s) 2023-03-09T14:48:42.803 INFO:teuthology.orchestra.run.smithi038.stderr:43900000 of 50000000 tuples (87%) done (elapsed 137.94 s, remaining 19.17 s) 2023-03-09T14:48:42.905 INFO:teuthology.orchestra.run.smithi038.stderr:44000000 of 50000000 tuples (88%) done (elapsed 138.04 s, remaining 18.82 s) 2023-03-09T14:48:42.999 INFO:teuthology.orchestra.run.smithi038.stderr:44100000 of 50000000 tuples (88%) done (elapsed 138.14 s, remaining 18.48 s) 2023-03-09T14:48:43.100 INFO:teuthology.orchestra.run.smithi038.stderr:44200000 of 50000000 tuples (88%) done (elapsed 138.24 s, remaining 18.14 s) 2023-03-09T14:48:43.206 INFO:teuthology.orchestra.run.smithi038.stderr:44300000 of 50000000 tuples (88%) done (elapsed 138.34 s, remaining 17.80 s) 2023-03-09T14:48:43.288 INFO:teuthology.orchestra.run.smithi038.stderr:44400000 of 50000000 tuples (88%) done (elapsed 138.43 s, remaining 17.46 s) 2023-03-09T14:48:43.390 INFO:teuthology.orchestra.run.smithi038.stderr:44500000 of 50000000 tuples (89%) done (elapsed 138.53 s, remaining 17.12 s) 2023-03-09T14:48:43.489 INFO:teuthology.orchestra.run.smithi038.stderr:44600000 of 50000000 tuples (89%) done (elapsed 138.63 s, remaining 16.78 s) 2023-03-09T14:48:43.592 INFO:teuthology.orchestra.run.smithi038.stderr:44700000 of 50000000 tuples (89%) done (elapsed 138.73 s, remaining 16.45 s) 2023-03-09T14:48:43.687 INFO:teuthology.orchestra.run.smithi038.stderr:44800000 of 50000000 tuples (89%) done (elapsed 138.82 s, remaining 16.11 s) 2023-03-09T14:48:43.773 INFO:teuthology.orchestra.run.smithi038.stderr:44900000 of 50000000 tuples (89%) done (elapsed 138.91 s, remaining 15.78 s) 2023-03-09T14:48:43.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:48:43 smithi122 ceph-mon[93869]: pgmap v82: 129 pgs: 129 active+clean; 7.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 7.5 MiB/s wr, 358 op/s 2023-03-09T14:48:43.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:48:43 smithi181 ceph-mon[94189]: pgmap v82: 129 pgs: 129 active+clean; 7.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 7.5 MiB/s wr, 358 op/s 2023-03-09T14:48:44.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:48:43 smithi038 ceph-mon[92422]: pgmap v82: 129 pgs: 129 active+clean; 7.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 7.5 MiB/s wr, 358 op/s 2023-03-09T14:48:44.439 INFO:teuthology.orchestra.run.smithi038.stderr:45000000 of 50000000 tuples (90%) done (elapsed 139.58 s, remaining 15.51 s) 2023-03-09T14:48:44.640 INFO:teuthology.orchestra.run.smithi038.stderr:45100000 of 50000000 tuples (90%) done (elapsed 139.78 s, remaining 15.19 s) 2023-03-09T14:48:45.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:48:45 smithi122 ceph-mon[93869]: pgmap v83: 129 pgs: 129 active+clean; 7.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 6.9 MiB/s wr, 261 op/s 2023-03-09T14:48:45.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:48:45 smithi181 ceph-mon[94189]: pgmap v83: 129 pgs: 129 active+clean; 7.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 6.9 MiB/s wr, 261 op/s 2023-03-09T14:48:46.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:48:45 smithi038 ceph-mon[92422]: pgmap v83: 129 pgs: 129 active+clean; 7.4 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 6.9 MiB/s wr, 261 op/s 2023-03-09T14:48:46.382 INFO:teuthology.orchestra.run.smithi038.stderr:45200000 of 50000000 tuples (90%) done (elapsed 141.52 s, remaining 15.03 s) 2023-03-09T14:48:47.686 INFO:teuthology.orchestra.run.smithi038.stderr:45300000 of 50000000 tuples (90%) done (elapsed 142.82 s, remaining 14.82 s) 2023-03-09T14:48:47.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:48:47 smithi122 ceph-mon[93869]: pgmap v84: 129 pgs: 129 active+clean; 8.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 83 MiB/s wr, 327 op/s 2023-03-09T14:48:47.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:48:47 smithi181 ceph-mon[94189]: pgmap v84: 129 pgs: 129 active+clean; 8.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 83 MiB/s wr, 327 op/s 2023-03-09T14:48:48.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:48:47 smithi038 ceph-mon[92422]: pgmap v84: 129 pgs: 129 active+clean; 8.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 83 MiB/s wr, 327 op/s 2023-03-09T14:48:49.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:48:49 smithi122 ceph-mon[93869]: pgmap v85: 129 pgs: 129 active+clean; 8.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 82 MiB/s wr, 197 op/s 2023-03-09T14:48:49.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:48:49 smithi181 ceph-mon[94189]: pgmap v85: 129 pgs: 129 active+clean; 8.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 82 MiB/s wr, 197 op/s 2023-03-09T14:48:50.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:48:49 smithi038 ceph-mon[92422]: pgmap v85: 129 pgs: 129 active+clean; 8.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 82 MiB/s wr, 197 op/s 2023-03-09T14:48:50.124 INFO:teuthology.orchestra.run.smithi038.stderr:45400000 of 50000000 tuples (90%) done (elapsed 145.26 s, remaining 14.72 s) 2023-03-09T14:48:50.239 INFO:teuthology.orchestra.run.smithi038.stderr:45500000 of 50000000 tuples (91%) done (elapsed 145.38 s, remaining 14.38 s) 2023-03-09T14:48:50.348 INFO:teuthology.orchestra.run.smithi038.stderr:45600000 of 50000000 tuples (91%) done (elapsed 145.49 s, remaining 14.04 s) 2023-03-09T14:48:50.856 INFO:teuthology.orchestra.run.smithi038.stderr:45700000 of 50000000 tuples (91%) done (elapsed 145.99 s, remaining 13.74 s) 2023-03-09T14:48:51.117 INFO:teuthology.orchestra.run.smithi038.stderr:45800000 of 50000000 tuples (91%) done (elapsed 146.25 s, remaining 13.41 s) 2023-03-09T14:48:51.224 INFO:teuthology.orchestra.run.smithi038.stderr:45900000 of 50000000 tuples (91%) done (elapsed 146.36 s, remaining 13.07 s) 2023-03-09T14:48:51.335 INFO:teuthology.orchestra.run.smithi038.stderr:46000000 of 50000000 tuples (92%) done (elapsed 146.47 s, remaining 12.74 s) 2023-03-09T14:48:51.514 INFO:teuthology.orchestra.run.smithi038.stderr:46100000 of 50000000 tuples (92%) done (elapsed 146.65 s, remaining 12.41 s) 2023-03-09T14:48:51.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:48:51 smithi122 ceph-mon[93869]: pgmap v86: 129 pgs: 129 active+clean; 8.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 84 MiB/s wr, 217 op/s 2023-03-09T14:48:51.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:48:51 smithi181 ceph-mon[94189]: pgmap v86: 129 pgs: 129 active+clean; 8.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 84 MiB/s wr, 217 op/s 2023-03-09T14:48:52.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:48:51 smithi038 ceph-mon[92422]: pgmap v86: 129 pgs: 129 active+clean; 8.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 84 MiB/s wr, 217 op/s 2023-03-09T14:48:52.150 INFO:teuthology.orchestra.run.smithi038.stderr:46200000 of 50000000 tuples (92%) done (elapsed 147.29 s, remaining 12.11 s) 2023-03-09T14:48:52.218 INFO:teuthology.orchestra.run.smithi038.stderr:46300000 of 50000000 tuples (92%) done (elapsed 147.36 s, remaining 11.78 s) 2023-03-09T14:48:52.336 INFO:teuthology.orchestra.run.smithi038.stderr:46400000 of 50000000 tuples (92%) done (elapsed 147.47 s, remaining 11.44 s) 2023-03-09T14:48:52.421 INFO:teuthology.orchestra.run.smithi038.stderr:46500000 of 50000000 tuples (93%) done (elapsed 147.56 s, remaining 11.11 s) 2023-03-09T14:48:52.530 INFO:teuthology.orchestra.run.smithi038.stderr:46600000 of 50000000 tuples (93%) done (elapsed 147.67 s, remaining 10.77 s) 2023-03-09T14:48:52.652 INFO:teuthology.orchestra.run.smithi038.stderr:46700000 of 50000000 tuples (93%) done (elapsed 147.79 s, remaining 10.44 s) 2023-03-09T14:48:52.749 INFO:teuthology.orchestra.run.smithi038.stderr:46800000 of 50000000 tuples (93%) done (elapsed 147.89 s, remaining 10.11 s) 2023-03-09T14:48:52.854 INFO:teuthology.orchestra.run.smithi038.stderr:46900000 of 50000000 tuples (93%) done (elapsed 147.99 s, remaining 9.78 s) 2023-03-09T14:48:52.959 INFO:teuthology.orchestra.run.smithi038.stderr:47000000 of 50000000 tuples (94%) done (elapsed 148.10 s, remaining 9.45 s) 2023-03-09T14:48:53.061 INFO:teuthology.orchestra.run.smithi038.stderr:47100000 of 50000000 tuples (94%) done (elapsed 148.20 s, remaining 9.12 s) 2023-03-09T14:48:53.479 INFO:teuthology.orchestra.run.smithi038.stderr:47200000 of 50000000 tuples (94%) done (elapsed 148.62 s, remaining 8.82 s) 2023-03-09T14:48:53.925 INFO:teuthology.orchestra.run.smithi038.stderr:47300000 of 50000000 tuples (94%) done (elapsed 149.06 s, remaining 8.51 s) 2023-03-09T14:48:53.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:48:53 smithi122 ceph-mon[93869]: pgmap v87: 129 pgs: 129 active+clean; 8.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 84 MiB/s wr, 269 op/s 2023-03-09T14:48:53.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:48:53 smithi181 ceph-mon[94189]: pgmap v87: 129 pgs: 129 active+clean; 8.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 84 MiB/s wr, 269 op/s 2023-03-09T14:48:53.994 INFO:teuthology.orchestra.run.smithi038.stderr:47400000 of 50000000 tuples (94%) done (elapsed 149.13 s, remaining 8.18 s) 2023-03-09T14:48:54.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:48:53 smithi038 ceph-mon[92422]: pgmap v87: 129 pgs: 129 active+clean; 8.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 84 MiB/s wr, 269 op/s 2023-03-09T14:48:54.138 INFO:teuthology.orchestra.run.smithi038.stderr:47500000 of 50000000 tuples (95%) done (elapsed 149.28 s, remaining 7.86 s) 2023-03-09T14:48:54.305 INFO:teuthology.orchestra.run.smithi038.stderr:47600000 of 50000000 tuples (95%) done (elapsed 149.44 s, remaining 7.53 s) 2023-03-09T14:48:54.387 INFO:teuthology.orchestra.run.smithi038.stderr:47700000 of 50000000 tuples (95%) done (elapsed 149.53 s, remaining 7.21 s) 2023-03-09T14:48:54.500 INFO:teuthology.orchestra.run.smithi038.stderr:47800000 of 50000000 tuples (95%) done (elapsed 149.64 s, remaining 6.89 s) 2023-03-09T14:48:54.596 INFO:teuthology.orchestra.run.smithi038.stderr:47900000 of 50000000 tuples (95%) done (elapsed 149.73 s, remaining 6.56 s) 2023-03-09T14:48:54.685 INFO:teuthology.orchestra.run.smithi038.stderr:48000000 of 50000000 tuples (96%) done (elapsed 149.82 s, remaining 6.24 s) 2023-03-09T14:48:54.799 INFO:teuthology.orchestra.run.smithi038.stderr:48100000 of 50000000 tuples (96%) done (elapsed 149.94 s, remaining 5.92 s) 2023-03-09T14:48:54.892 INFO:teuthology.orchestra.run.smithi038.stderr:48200000 of 50000000 tuples (96%) done (elapsed 150.03 s, remaining 5.60 s) 2023-03-09T14:48:54.977 INFO:teuthology.orchestra.run.smithi038.stderr:48300000 of 50000000 tuples (96%) done (elapsed 150.11 s, remaining 5.28 s) 2023-03-09T14:48:55.156 INFO:teuthology.orchestra.run.smithi038.stderr:48400000 of 50000000 tuples (96%) done (elapsed 150.29 s, remaining 4.97 s) 2023-03-09T14:48:55.375 INFO:teuthology.orchestra.run.smithi038.stderr:48500000 of 50000000 tuples (97%) done (elapsed 150.51 s, remaining 4.66 s) 2023-03-09T14:48:55.473 INFO:teuthology.orchestra.run.smithi038.stderr:48600000 of 50000000 tuples (97%) done (elapsed 150.61 s, remaining 4.34 s) 2023-03-09T14:48:55.575 INFO:teuthology.orchestra.run.smithi038.stderr:48700000 of 50000000 tuples (97%) done (elapsed 150.71 s, remaining 4.02 s) 2023-03-09T14:48:55.660 INFO:teuthology.orchestra.run.smithi038.stderr:48800000 of 50000000 tuples (97%) done (elapsed 150.80 s, remaining 3.71 s) 2023-03-09T14:48:55.779 INFO:teuthology.orchestra.run.smithi038.stderr:48900000 of 50000000 tuples (97%) done (elapsed 150.92 s, remaining 3.39 s) 2023-03-09T14:48:55.896 INFO:teuthology.orchestra.run.smithi038.stderr:49000000 of 50000000 tuples (98%) done (elapsed 151.03 s, remaining 3.08 s) 2023-03-09T14:48:55.980 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:48:55 smithi122 ceph-mon[93869]: pgmap v88: 129 pgs: 129 active+clean; 8.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 82 MiB/s wr, 176 op/s 2023-03-09T14:48:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:48:55 smithi181 ceph-mon[94189]: pgmap v88: 129 pgs: 129 active+clean; 8.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 82 MiB/s wr, 176 op/s 2023-03-09T14:48:56.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:48:55 smithi038 ceph-mon[92422]: pgmap v88: 129 pgs: 129 active+clean; 8.3 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 82 MiB/s wr, 176 op/s 2023-03-09T14:48:56.569 INFO:teuthology.orchestra.run.smithi038.stderr:49100000 of 50000000 tuples (98%) done (elapsed 151.71 s, remaining 2.78 s) 2023-03-09T14:48:57.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:48:57 smithi122 ceph-mon[93869]: pgmap v89: 129 pgs: 129 active+clean; 8.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 108 MiB/s wr, 243 op/s 2023-03-09T14:48:57.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:48:57 smithi181 ceph-mon[94189]: pgmap v89: 129 pgs: 129 active+clean; 8.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 108 MiB/s wr, 243 op/s 2023-03-09T14:48:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:48:57 smithi038 ceph-mon[92422]: pgmap v89: 129 pgs: 129 active+clean; 8.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 108 MiB/s wr, 243 op/s 2023-03-09T14:48:59.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:48:59 smithi122 ceph-mon[93869]: pgmap v90: 129 pgs: 129 active+clean; 8.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 32 MiB/s wr, 177 op/s 2023-03-09T14:48:59.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:48:59 smithi181 ceph-mon[94189]: pgmap v90: 129 pgs: 129 active+clean; 8.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 32 MiB/s wr, 177 op/s 2023-03-09T14:49:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:48:59 smithi038 ceph-mon[92422]: pgmap v90: 129 pgs: 129 active+clean; 8.6 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 32 MiB/s wr, 177 op/s 2023-03-09T14:49:01.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:01 smithi122 ceph-mon[93869]: pgmap v91: 129 pgs: 129 active+clean; 9.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 68 MiB/s wr, 246 op/s 2023-03-09T14:49:01.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:01 smithi181 ceph-mon[94189]: pgmap v91: 129 pgs: 129 active+clean; 9.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 68 MiB/s wr, 246 op/s 2023-03-09T14:49:02.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:01 smithi038 ceph-mon[92422]: pgmap v91: 129 pgs: 129 active+clean; 9.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 68 MiB/s wr, 246 op/s 2023-03-09T14:49:02.108 INFO:teuthology.orchestra.run.smithi038.stderr:49200000 of 50000000 tuples (98%) done (elapsed 157.25 s, remaining 2.56 s) 2023-03-09T14:49:03.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:03 smithi122 ceph-mon[93869]: pgmap v92: 129 pgs: 129 active+clean; 9.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 79 MiB/s wr, 288 op/s 2023-03-09T14:49:03.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:03 smithi181 ceph-mon[94189]: pgmap v92: 129 pgs: 129 active+clean; 9.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 79 MiB/s wr, 288 op/s 2023-03-09T14:49:04.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:03 smithi038 ceph-mon[92422]: pgmap v92: 129 pgs: 129 active+clean; 9.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 79 MiB/s wr, 288 op/s 2023-03-09T14:49:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:05 smithi122 ceph-mon[93869]: pgmap v93: 129 pgs: 129 active+clean; 9.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 75 MiB/s wr, 198 op/s 2023-03-09T14:49:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:05 smithi181 ceph-mon[94189]: pgmap v93: 129 pgs: 129 active+clean; 9.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 75 MiB/s wr, 198 op/s 2023-03-09T14:49:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:05 smithi038 ceph-mon[92422]: pgmap v93: 129 pgs: 129 active+clean; 9.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 75 MiB/s wr, 198 op/s 2023-03-09T14:49:07.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:07 smithi122 ceph-mon[93869]: pgmap v94: 129 pgs: 129 active+clean; 9.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 76 MiB/s wr, 308 op/s 2023-03-09T14:49:07.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:07 smithi181 ceph-mon[94189]: pgmap v94: 129 pgs: 129 active+clean; 9.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 76 MiB/s wr, 308 op/s 2023-03-09T14:49:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:07 smithi038 ceph-mon[92422]: pgmap v94: 129 pgs: 129 active+clean; 9.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 76 MiB/s wr, 308 op/s 2023-03-09T14:49:08.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T14:49:08.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:08 smithi122 ceph-mon[93869]: scrub summary: active paths [/] 2023-03-09T14:49:08.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:08 smithi122 ceph-mon[93869]: scrub summary: active paths [/] 2023-03-09T14:49:08.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:08 smithi122 ceph-mon[93869]: scrub summary: idle+waiting paths [/] 2023-03-09T14:49:08.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:08 smithi122 ceph-mon[93869]: scrub summary: idle+waiting paths [/] 2023-03-09T14:49:08.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T14:49:08.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:08 smithi181 ceph-mon[94189]: scrub summary: active paths [/] 2023-03-09T14:49:08.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:08 smithi181 ceph-mon[94189]: scrub summary: active paths [/] 2023-03-09T14:49:08.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:08 smithi181 ceph-mon[94189]: scrub summary: idle+waiting paths [/] 2023-03-09T14:49:08.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:08 smithi181 ceph-mon[94189]: scrub summary: idle+waiting paths [/] 2023-03-09T14:49:09.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T14:49:09.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:08 smithi038 ceph-mon[92422]: scrub summary: active paths [/] 2023-03-09T14:49:09.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:08 smithi038 ceph-mon[92422]: scrub summary: active paths [/] 2023-03-09T14:49:09.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:08 smithi038 ceph-mon[92422]: scrub summary: idle+waiting paths [/] 2023-03-09T14:49:09.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:08 smithi038 ceph-mon[92422]: scrub summary: idle+waiting paths [/] 2023-03-09T14:49:09.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:09 smithi122 ceph-mon[93869]: pgmap v95: 129 pgs: 129 active+clean; 9.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 50 MiB/s wr, 241 op/s 2023-03-09T14:49:09.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:09 smithi181 ceph-mon[94189]: pgmap v95: 129 pgs: 129 active+clean; 9.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 50 MiB/s wr, 241 op/s 2023-03-09T14:49:10.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:09 smithi038 ceph-mon[92422]: pgmap v95: 129 pgs: 129 active+clean; 9.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 50 MiB/s wr, 241 op/s 2023-03-09T14:49:11.799 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-03-09T14:49:11.971 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.968+0000 7f83719e6700 1 -- 172.21.15.38:0/1677319268 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f836c104080 msgr2=0x7f836c10bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:49:11.971 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.968+0000 7f83719e6700 1 --2- 172.21.15.38:0/1677319268 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f836c104080 0x7f836c10bc50 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f836400b0d0 tx=0x7f836400b3e0 comp rx=0 tx=0).stop 2023-03-09T14:49:11.972 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.968+0000 7f83719e6700 1 -- 172.21.15.38:0/1677319268 shutdown_connections 2023-03-09T14:49:11.972 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.968+0000 7f83719e6700 1 --2- 172.21.15.38:0/1677319268 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f836c104080 0x7f836c10bc50 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:49:11.972 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.968+0000 7f83719e6700 1 --2- 172.21.15.38:0/1677319268 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f836c0ff090 0x7f836c0ff460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:49:11.972 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.968+0000 7f83719e6700 1 --2- 172.21.15.38:0/1677319268 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f836c0ffa30 0x7f836c1039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:49:11.972 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.968+0000 7f83719e6700 1 -- 172.21.15.38:0/1677319268 >> 172.21.15.38:0/1677319268 conn(0x7f836c075e80 msgr2=0x7f836c076280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:49:11.972 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.968+0000 7f83719e6700 1 -- 172.21.15.38:0/1677319268 shutdown_connections 2023-03-09T14:49:11.973 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.968+0000 7f83719e6700 1 -- 172.21.15.38:0/1677319268 wait complete. 2023-03-09T14:49:11.973 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.969+0000 7f83719e6700 1 Processor -- start 2023-03-09T14:49:11.974 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.969+0000 7f83719e6700 1 -- start start 2023-03-09T14:49:11.974 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.969+0000 7f83719e6700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f836c0ff090 0x7f836c100a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:49:11.974 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.969+0000 7f83719e6700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f836c0ffa30 0x7f836c100f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:49:11.975 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.969+0000 7f83719e6700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f836c104080 0x7f836c1014c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:49:11.975 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.970+0000 7f83719e6700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f836c1a8ad0 con 0x7f836c0ffa30 2023-03-09T14:49:11.975 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.970+0000 7f83719e6700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f836c1a8c10 con 0x7f836c0ff090 2023-03-09T14:49:11.975 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.970+0000 7f83719e6700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f836c1a8d50 con 0x7f836c104080 2023-03-09T14:49:11.975 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.970+0000 7f836affd700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f836c0ff090 0x7f836c100a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:49:11.975 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.970+0000 7f836affd700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f836c0ff090 0x7f836c100a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.38:49088/0 (socket says 172.21.15.38:49088) 2023-03-09T14:49:11.976 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.970+0000 7f836affd700 1 -- 172.21.15.38:0/3165301711 learned_addr learned my addr 172.21.15.38:0/3165301711 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:49:11.976 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.970+0000 7f836b7fe700 1 --2- 172.21.15.38:0/3165301711 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f836c104080 0x7f836c1014c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:49:11.976 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.970+0000 7f836affd700 1 -- 172.21.15.38:0/3165301711 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f836c104080 msgr2=0x7f836c1014c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:49:11.976 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.970+0000 7f836affd700 1 --2- 172.21.15.38:0/3165301711 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f836c104080 0x7f836c1014c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:49:11.976 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.970+0000 7f836affd700 1 -- 172.21.15.38:0/3165301711 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f836c0ffa30 msgr2=0x7f836c100f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:49:11.976 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.970+0000 7f836affd700 1 --2- 172.21.15.38:0/3165301711 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f836c0ffa30 0x7f836c100f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:49:11.976 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.970+0000 7f836affd700 1 -- 172.21.15.38:0/3165301711 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8364009d00 con 0x7f836c0ff090 2023-03-09T14:49:11.977 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.970+0000 7f836b7fe700 1 --2- 172.21.15.38:0/3165301711 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f836c104080 0x7f836c1014c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:49:11.977 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.970+0000 7f836affd700 1 --2- 172.21.15.38:0/3165301711 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f836c0ff090 0x7f836c100a40 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f835c00e9a0 tx=0x7f835c00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:49:11.977 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.970+0000 7f83709e4700 1 -- 172.21.15.38:0/3165301711 <== mon.2 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f835c009790 con 0x7f836c0ff090 2023-03-09T14:49:11.977 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.971+0000 7f83719e6700 1 -- 172.21.15.38:0/3165301711 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f836c10ee40 con 0x7f836c0ff090 2023-03-09T14:49:11.977 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.971+0000 7f83709e4700 1 -- 172.21.15.38:0/3165301711 <== mon.2 v2:172.21.15.122:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f835c00cbd0 con 0x7f836c0ff090 2023-03-09T14:49:11.977 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.971+0000 7f83709e4700 1 -- 172.21.15.38:0/3165301711 <== mon.2 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f835c018680 con 0x7f836c0ff090 2023-03-09T14:49:11.977 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.971+0000 7f83719e6700 1 -- 172.21.15.38:0/3165301711 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f836c10ecb0 con 0x7f836c0ff090 2023-03-09T14:49:11.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:11 smithi122 ceph-mon[93869]: pgmap v96: 129 pgs: 129 active+clean; 9.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 51 MiB/s wr, 372 op/s 2023-03-09T14:49:11.979 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.974+0000 7f83709e4700 1 -- 172.21.15.38:0/3165301711 <== mon.2 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 25) v1 ==== 133295+0+0 (secure 0 0 0) 0x7f835c00c040 con 0x7f836c0ff090 2023-03-09T14:49:11.979 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.975+0000 7f83709e4700 1 --2- 172.21.15.38:0/3165301711 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f835409f4a0 0x7f83540a1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:49:11.980 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.975+0000 7f83709e4700 1 -- 172.21.15.38:0/3165301711 <== mon.2 v2:172.21.15.122:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9489+0+0 (secure 0 0 0) 0x7f835c014070 con 0x7f836c0ff090 2023-03-09T14:49:11.980 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.975+0000 7f836a7fc700 1 --2- 172.21.15.38:0/3165301711 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f835409f4a0 0x7f83540a1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:49:11.980 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.975+0000 7f837db6fb80 1 Processor -- start 2023-03-09T14:49:11.980 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.975+0000 7f837db6fb80 1 -- start start 2023-03-09T14:49:11.980 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.975+0000 7f836a7fc700 1 --2- 172.21.15.38:0/3165301711 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f835409f4a0 0x7f83540a1950 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f836c102230 tx=0x7f8360009620 comp rx=0 tx=0).ready entity=mgr.15558 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:49:11.980 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.976+0000 7f837db6fb80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db31036830 0x55db31036c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:49:11.980 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.976+0000 7f837db6fb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55db31037140 0x55db31030aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:49:11.981 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.976+0000 7f837db6fb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55db31030fe0 0x55db31031410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:49:11.981 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.976+0000 7f837db6fb80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db30f1ada0 con 0x55db31036830 2023-03-09T14:49:11.981 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.976+0000 7f837db6fb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db30f89780 con 0x55db31037140 2023-03-09T14:49:11.981 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.976+0000 7f837db6fb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db30f82cb0 con 0x55db31030fe0 2023-03-09T14:49:11.981 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.976+0000 7f836a7fc700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55db31037140 0x55db31030aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:49:11.981 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.976+0000 7f836affd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db31036830 0x55db31036c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:49:11.982 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.976+0000 7f836affd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db31036830 0x55db31036c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39908/0 (socket says 172.21.15.38:39908) 2023-03-09T14:49:11.982 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.976+0000 7f836a7fc700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55db31037140 0x55db31030aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.38:49096/0 (socket says 172.21.15.38:49096) 2023-03-09T14:49:11.982 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.976+0000 7f836affd700 1 -- 172.21.15.38:0/3477430804 learned_addr learned my addr 172.21.15.38:0/3477430804 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:49:11.982 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.976+0000 7f836a7fc700 1 -- 172.21.15.38:0/3477430804 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55db31030fe0 msgr2=0x55db31031410 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-03-09T14:49:11.982 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.976+0000 7f836a7fc700 1 --2- 172.21.15.38:0/3477430804 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55db31030fe0 0x55db31031410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:49:11.982 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.976+0000 7f836a7fc700 1 -- 172.21.15.38:0/3477430804 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db31036830 msgr2=0x55db31036c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:49:11.982 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.976+0000 7f836a7fc700 1 --2- 172.21.15.38:0/3477430804 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db31036830 0x55db31036c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:49:11.983 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.976+0000 7f836a7fc700 1 -- 172.21.15.38:0/3477430804 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55db30f49190 con 0x55db31037140 2023-03-09T14:49:11.983 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.977+0000 7f836a7fc700 1 --2- 172.21.15.38:0/3477430804 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55db31037140 0x55db31030aa0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f836c0679c0 tx=0x7f8360011000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d252326d5b940461 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:49:11.983 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.977+0000 7f8351ffb700 1 -- 172.21.15.38:0/3477430804 <== mon.2 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8360004a30 con 0x55db31037140 2023-03-09T14:49:11.983 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.977+0000 7f8351ffb700 1 -- 172.21.15.38:0/3477430804 <== mon.2 v2:172.21.15.122:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f83600054e0 con 0x55db31037140 2023-03-09T14:49:11.983 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.977+0000 7f8351ffb700 1 -- 172.21.15.38:0/3477430804 <== mon.2 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8360010a70 con 0x55db31037140 2023-03-09T14:49:11.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:11 smithi181 ceph-mon[94189]: pgmap v96: 129 pgs: 129 active+clean; 9.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 51 MiB/s wr, 372 op/s 2023-03-09T14:49:11.987 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.978+0000 7f837db6fb80 1 -- 172.21.15.38:0/3477430804 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55db31037140 msgr2=0x55db31030aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:49:11.987 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.978+0000 7f837db6fb80 1 --2- 172.21.15.38:0/3477430804 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55db31037140 0x55db31030aa0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f836c0679c0 tx=0x7f8360011000 comp rx=0 tx=0).stop 2023-03-09T14:49:11.988 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.978+0000 7f837db6fb80 1 -- 172.21.15.38:0/3477430804 shutdown_connections 2023-03-09T14:49:11.988 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.978+0000 7f837db6fb80 1 --2- 172.21.15.38:0/3477430804 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db31036830 0x55db31036c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:49:11.988 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.978+0000 7f837db6fb80 1 --2- 172.21.15.38:0/3477430804 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55db31030fe0 0x55db31031410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:49:11.988 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.978+0000 7f837db6fb80 1 --2- 172.21.15.38:0/3477430804 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55db31037140 0x55db31030aa0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:49:11.988 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.978+0000 7f837db6fb80 1 -- 172.21.15.38:0/3477430804 >> 172.21.15.38:0/3477430804 conn(0x55db3102ba20 msgr2=0x55db3102bdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:49:11.988 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.978+0000 7f837db6fb80 1 -- 172.21.15.38:0/3477430804 shutdown_connections 2023-03-09T14:49:11.988 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.978+0000 7f837db6fb80 1 -- 172.21.15.38:0/3477430804 wait complete. 2023-03-09T14:49:11.989 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.978+0000 7f837db6fb80 1 -- start start 2023-03-09T14:49:11.989 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.978+0000 7f837db6fb80 1 Processor -- start 2023-03-09T14:49:11.989 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.979+0000 7f837db6fb80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db31030fe0 0x55db3103d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:49:11.989 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.979+0000 7f837db6fb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55db31036830 0x55db3103a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:49:11.989 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.979+0000 7f837db6fb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55db31037140 0x55db3103aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:49:11.989 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.979+0000 7f837db6fb80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db30e06920 con 0x55db31030fe0 2023-03-09T14:49:11.989 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.979+0000 7f837db6fb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db3103b1b0 con 0x55db31037140 2023-03-09T14:49:11.990 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.979+0000 7f837db6fb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db31046c40 con 0x55db31036830 2023-03-09T14:49:11.990 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.979+0000 7f837db6fb80 10 client.0 _ll_register_callbacks cb 0x55db30f28520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:49:11.990 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.979+0000 7f836affd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db31030fe0 0x55db3103d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:49:11.990 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.979+0000 7f836a7fc700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55db31036830 0x55db3103a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:49:11.990 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.979+0000 7f836a7fc700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55db31036830 0x55db3103a3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.38:47610/0 (socket says 172.21.15.38:47610) 2023-03-09T14:49:11.990 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.979+0000 7f836b7fe700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55db31037140 0x55db3103aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:49:11.990 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.979+0000 7f836a7fc700 1 -- 172.21.15.38:0/2195825366 learned_addr learned my addr 172.21.15.38:0/2195825366 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:49:11.990 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.979+0000 7f836affd700 1 -- 172.21.15.38:0/2195825366 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55db31036830 msgr2=0x55db3103a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:49:11.991 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.979+0000 7f836affd700 1 --2- 172.21.15.38:0/2195825366 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55db31036830 0x55db3103a3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:49:11.991 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.979+0000 7f836affd700 1 -- 172.21.15.38:0/2195825366 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55db31037140 msgr2=0x55db3103aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:49:11.991 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.979+0000 7f836affd700 1 --2- 172.21.15.38:0/2195825366 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55db31037140 0x55db3103aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:49:11.991 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.979+0000 7f836affd700 1 -- 172.21.15.38:0/2195825366 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f836000e710 con 0x55db31030fe0 2023-03-09T14:49:11.991 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.979+0000 7f836b7fe700 1 --2- 172.21.15.38:0/2195825366 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55db31037140 0x55db3103aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:49:11.991 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.979+0000 7f836a7fc700 1 --2- 172.21.15.38:0/2195825366 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55db31036830 0x55db3103a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:49:11.991 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.979+0000 7f836affd700 1 --2- 172.21.15.38:0/2195825366 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db31030fe0 0x55db3103d120 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f835c09fe20 tx=0x7f835c0986a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1bd48113c9b93a9 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:49:11.992 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.979+0000 7f834affd700 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2023-03-09T14:49:11.992 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.979+0000 7f834affd700 1 -- 172.21.15.38:0/2195825366 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f835c0af070 con 0x55db31030fe0 2023-03-09T14:49:11.992 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.979+0000 7f834affd700 1 -- 172.21.15.38:0/2195825366 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9489+0+0 (secure 0 0 0) 0x7f835c0a0350 con 0x55db31030fe0 2023-03-09T14:49:11.992 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.979+0000 7f837db6fb80 1 -- 172.21.15.38:0/2195825366 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55db30f50190 con 0x55db31030fe0 2023-03-09T14:49:11.992 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.980+0000 7f834affd700 1 -- 172.21.15.38:0/2195825366 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f835c09fec0 con 0x55db31030fe0 2023-03-09T14:49:11.992 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.980+0000 7f837db6fb80 10 client.15930 fetch_fsmap learned FSMap version 27 2023-03-09T14:49:11.992 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.980+0000 7f837db6fb80 1 -- 172.21.15.38:0/2195825366 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55db30d99c60 con 0x55db31030fe0 2023-03-09T14:49:11.992 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.980+0000 7f834affd700 1 -- 172.21.15.38:0/2195825366 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f835c0b4020 con 0x55db31030fe0 2023-03-09T14:49:11.993 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.980+0000 7f834affd700 1 -- 172.21.15.38:0/2195825366 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 27) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f835c0ca110 con 0x55db31030fe0 2023-03-09T14:49:11.993 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.980+0000 7f837db6fb80 10 client.15930 fetch_fsmap finished waiting for FSMap version 27 2023-03-09T14:49:11.993 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.980+0000 7f837db6fb80 10 client.15930 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T14:49:11.993 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.980+0000 7f837db6fb80 20 client.15930 populate_metadata read hostname 'smithi038' 2023-03-09T14:49:11.993 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.980+0000 7f837db6fb80 1 --2- 172.21.15.38:0/2195825366 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55db3104ae00 0x55db3104d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:49:11.993 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.980+0000 7f837db6fb80 4 client.15930 mds_command: new command op to 24530 tid=0[{"prefix": "get_command_descriptions"}] 2023-03-09T14:49:11.993 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.980+0000 7f837db6fb80 1 -- 172.21.15.38:0/2195825366 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55db30e06f60 con 0x55db3104ae00 2023-03-09T14:49:11.994 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.980+0000 7f836a7fc700 1 --2- 172.21.15.38:0/2195825366 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55db3104ae00 0x55db3104d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:49:11.994 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.981+0000 7f834affd700 10 client.15930 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:49:11.994 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.981+0000 7f836a7fc700 1 --2- 172.21.15.38:0/2195825366 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55db3104ae00 0x55db3104d210 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=71e928d0bd4a7c81 server_cookie=1174a8fbbe322067 in_seq=0 out_seq=0 2023-03-09T14:49:11.994 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.982+0000 7f834affd700 1 -- 172.21.15.38:0/2195825366 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18278 (crc 0 0 0) 0x55db30e06f60 con 0x55db3104ae00 2023-03-09T14:49:11.994 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.982+0000 7f834affd700 10 client.15930 handle_command_reply: tid=0 2023-03-09T14:49:11.994 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.982+0000 7f837db6fb80 1 client.15930 shutdown 2023-03-09T14:49:11.994 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.982+0000 7f837db6fb80 1 -- 172.21.15.38:0/2195825366 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db31030fe0 msgr2=0x55db3103d120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:49:11.994 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.982+0000 7f837db6fb80 1 --2- 172.21.15.38:0/2195825366 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db31030fe0 0x55db3103d120 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f835c09fe20 tx=0x7f835c0986a0 comp rx=0 tx=0).stop 2023-03-09T14:49:11.995 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.982+0000 7f837db6fb80 1 -- 172.21.15.38:0/2195825366 shutdown_connections 2023-03-09T14:49:11.995 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.982+0000 7f837db6fb80 1 --2- 172.21.15.38:0/2195825366 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55db3104ae00 0x55db3104d210 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:49:11.995 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.982+0000 7f837db6fb80 1 --2- 172.21.15.38:0/2195825366 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db31030fe0 0x55db3103d120 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:49:11.995 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.982+0000 7f837db6fb80 1 --2- 172.21.15.38:0/2195825366 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55db31037140 0x55db3103aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:49:11.995 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.982+0000 7f837db6fb80 1 --2- 172.21.15.38:0/2195825366 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55db31036830 0x55db3103a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:49:11.995 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.982+0000 7f837db6fb80 1 -- 172.21.15.38:0/2195825366 >> 172.21.15.38:0/2195825366 conn(0x55db3102ba20 msgr2=0x55db3102ac20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:49:11.995 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.982+0000 7f834affd700 0 client.15930 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:49:11.995 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.982+0000 7f837db6fb80 1 -- 172.21.15.38:0/2195825366 shutdown_connections 2023-03-09T14:49:11.996 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.982+0000 7f837db6fb80 1 -- 172.21.15.38:0/2195825366 wait complete. 2023-03-09T14:49:11.996 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:11.982+0000 7f837db6fb80 20 client.15930 trim_cache size 0 max 16384 2023-03-09T14:49:12.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:11 smithi038 ceph-mon[92422]: pgmap v96: 129 pgs: 129 active+clean; 9.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 51 MiB/s wr, 372 op/s 2023-03-09T14:49:12.008 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.004+0000 7f837db6fb80 1 Processor -- start 2023-03-09T14:49:12.008 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.004+0000 7f837db6fb80 1 -- start start 2023-03-09T14:49:12.008 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.005+0000 7f837db6fb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55db31030fe0 0x55db3103ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:49:12.008 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.005+0000 7f837db6fb80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db31036830 0x55db310344f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:49:12.008 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.005+0000 7f837db6fb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55db31037140 0x55db31034a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:49:12.009 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.005+0000 7f837db6fb80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db31031d00 con 0x55db31036830 2023-03-09T14:49:12.009 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.005+0000 7f837db6fb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db3103b980 con 0x55db31037140 2023-03-09T14:49:12.009 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.005+0000 7f837db6fb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db31025040 con 0x55db31030fe0 2023-03-09T14:49:12.009 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.005+0000 7f836a7fc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db31036830 0x55db310344f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:49:12.009 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.005+0000 7f836affd700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55db31030fe0 0x55db3103ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:49:12.009 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.005+0000 7f836a7fc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db31036830 0x55db310344f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39920/0 (socket says 172.21.15.38:39920) 2023-03-09T14:49:12.009 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.005+0000 7f836a7fc700 1 -- 172.21.15.38:0/1622881311 learned_addr learned my addr 172.21.15.38:0/1622881311 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:49:12.010 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.005+0000 7f836b7fe700 1 --2- 172.21.15.38:0/1622881311 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55db31037140 0x55db31034a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:49:12.010 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.005+0000 7f836affd700 1 -- 172.21.15.38:0/1622881311 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55db31037140 msgr2=0x55db31034a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:49:12.010 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.005+0000 7f836affd700 1 --2- 172.21.15.38:0/1622881311 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55db31037140 0x55db31034a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:49:12.010 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.005+0000 7f836affd700 1 -- 172.21.15.38:0/1622881311 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db31036830 msgr2=0x55db310344f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:49:12.010 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.005+0000 7f836affd700 1 --2- 172.21.15.38:0/1622881311 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db31036830 0x55db310344f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:49:12.010 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.005+0000 7f836affd700 1 -- 172.21.15.38:0/1622881311 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55db30db0de0 con 0x55db31030fe0 2023-03-09T14:49:12.010 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.005+0000 7f836a7fc700 1 --2- 172.21.15.38:0/1622881311 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db31036830 0x55db310344f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:49:12.011 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.005+0000 7f836affd700 1 --2- 172.21.15.38:0/1622881311 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55db31030fe0 0x55db3103ccc0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f835c094b80 tx=0x7f835c098050 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6cbec6928ec415a3 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:49:12.011 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.006+0000 7f83527fc700 1 -- 172.21.15.38:0/1622881311 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f835c0af070 con 0x55db31030fe0 2023-03-09T14:49:12.011 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.006+0000 7f83527fc700 1 -- 172.21.15.38:0/1622881311 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f835c0b4070 con 0x55db31030fe0 2023-03-09T14:49:12.011 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.006+0000 7f83527fc700 1 -- 172.21.15.38:0/1622881311 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f835c0aba00 con 0x55db31030fe0 2023-03-09T14:49:12.014 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.006+0000 7f837db6fb80 1 -- 172.21.15.38:0/1622881311 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55db31030fe0 msgr2=0x55db3103ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:49:12.014 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.006+0000 7f837db6fb80 1 --2- 172.21.15.38:0/1622881311 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55db31030fe0 0x55db3103ccc0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f835c094b80 tx=0x7f835c098050 comp rx=0 tx=0).stop 2023-03-09T14:49:12.014 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.006+0000 7f837db6fb80 1 -- 172.21.15.38:0/1622881311 shutdown_connections 2023-03-09T14:49:12.014 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.006+0000 7f837db6fb80 1 --2- 172.21.15.38:0/1622881311 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55db31037140 0x55db31034a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:49:12.015 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.006+0000 7f837db6fb80 1 --2- 172.21.15.38:0/1622881311 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55db31030fe0 0x55db3103ccc0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:49:12.015 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.006+0000 7f837db6fb80 1 --2- 172.21.15.38:0/1622881311 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db31036830 0x55db310344f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:49:12.015 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.006+0000 7f837db6fb80 1 -- 172.21.15.38:0/1622881311 >> 172.21.15.38:0/1622881311 conn(0x55db3102ba20 msgr2=0x55db3104ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:49:12.015 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.006+0000 7f837db6fb80 1 -- 172.21.15.38:0/1622881311 shutdown_connections 2023-03-09T14:49:12.015 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.006+0000 7f837db6fb80 1 -- 172.21.15.38:0/1622881311 wait complete. 2023-03-09T14:49:12.015 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.007+0000 7f837db6fb80 1 -- start start 2023-03-09T14:49:12.015 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.007+0000 7f837db6fb80 1 Processor -- start 2023-03-09T14:49:12.016 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.007+0000 7f837db6fb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55db31030fe0 0x55db31032290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:49:12.016 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.007+0000 7f837db6fb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55db31036830 0x55db310327d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:49:12.016 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.007+0000 7f837db6fb80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db31037140 0x55db31032d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:49:12.016 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.007+0000 7f837db6fb80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db310333c0 con 0x55db31037140 2023-03-09T14:49:12.016 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.007+0000 7f837db6fb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db31033530 con 0x55db31030fe0 2023-03-09T14:49:12.016 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.007+0000 7f837db6fb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db310336f0 con 0x55db31036830 2023-03-09T14:49:12.016 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.007+0000 7f837db6fb80 10 client.0 _ll_register_callbacks cb 0x55db30f28520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:49:12.016 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.007+0000 7f836b7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db31037140 0x55db31032d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:49:12.017 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.008+0000 7f836b7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db31037140 0x55db31032d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39926/0 (socket says 172.21.15.38:39926) 2023-03-09T14:49:12.017 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.008+0000 7f836b7fe700 1 -- 172.21.15.38:0/1273920949 learned_addr learned my addr 172.21.15.38:0/1273920949 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:49:12.017 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.008+0000 7f836a7fc700 1 --2- 172.21.15.38:0/1273920949 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55db31036830 0x55db310327d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:49:12.017 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.008+0000 7f836b7fe700 1 -- 172.21.15.38:0/1273920949 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55db31036830 msgr2=0x55db310327d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:49:12.017 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.008+0000 7f836b7fe700 1 --2- 172.21.15.38:0/1273920949 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55db31036830 0x55db310327d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:49:12.017 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.008+0000 7f836b7fe700 1 -- 172.21.15.38:0/1273920949 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55db31030fe0 msgr2=0x55db31032290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:49:12.017 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.008+0000 7f836b7fe700 1 --2- 172.21.15.38:0/1273920949 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55db31030fe0 0x55db31032290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:49:12.018 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.008+0000 7f836b7fe700 1 -- 172.21.15.38:0/1273920949 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55db30f49190 con 0x55db31037140 2023-03-09T14:49:12.018 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.008+0000 7f836a7fc700 1 --2- 172.21.15.38:0/1273920949 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55db31036830 0x55db310327d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:49:12.018 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.008+0000 7f836b7fe700 1 --2- 172.21.15.38:0/1273920949 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db31037140 0x55db31032d10 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f836400b0d0 tx=0x7f8364007710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90120452dd6e1851 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:49:12.018 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.008+0000 7f834affd700 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2023-03-09T14:49:12.018 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.008+0000 7f834affd700 1 -- 172.21.15.38:0/1273920949 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8364021070 con 0x55db31037140 2023-03-09T14:49:12.018 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.008+0000 7f837db6fb80 1 -- 172.21.15.38:0/1273920949 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55db30f401b0 con 0x55db31037140 2023-03-09T14:49:12.018 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.008+0000 7f834affd700 1 -- 172.21.15.38:0/1273920949 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9489+0+0 (secure 0 0 0) 0x7f8364003a00 con 0x55db31037140 2023-03-09T14:49:12.019 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.009+0000 7f834affd700 1 -- 172.21.15.38:0/1273920949 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f836400e040 con 0x55db31037140 2023-03-09T14:49:12.019 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.009+0000 7f837db6fb80 10 client.15936 fetch_fsmap learned FSMap version 27 2023-03-09T14:49:12.019 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.009+0000 7f837db6fb80 1 -- 172.21.15.38:0/1273920949 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55db30d99c60 con 0x55db31037140 2023-03-09T14:49:12.019 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.009+0000 7f834affd700 1 -- 172.21.15.38:0/1273920949 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55db30f401b0 con 0x55db31037140 2023-03-09T14:49:12.019 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.009+0000 7f834affd700 1 -- 172.21.15.38:0/1273920949 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 27) v1 ==== 7486+0+0 (secure 0 0 0) 0x7f836400b100 con 0x55db31037140 2023-03-09T14:49:12.019 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.009+0000 7f837db6fb80 10 client.15936 fetch_fsmap finished waiting for FSMap version 27 2023-03-09T14:49:12.019 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.009+0000 7f837db6fb80 10 client.15936 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T14:49:12.019 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.009+0000 7f837db6fb80 20 client.15936 populate_metadata read hostname 'smithi038' 2023-03-09T14:49:12.020 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.009+0000 7f837db6fb80 1 --2- 172.21.15.38:0/1273920949 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55db3109f8c0 0x55db310a1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:49:12.020 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.009+0000 7f837db6fb80 4 client.15936 mds_command: new command op to 24530 tid=0[{"prefix": "scrub status"}] 2023-03-09T14:49:12.020 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.009+0000 7f837db6fb80 1 -- 172.21.15.38:0/1273920949 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55db30f2a1c0 con 0x55db3109f8c0 2023-03-09T14:49:12.020 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.009+0000 7f836affd700 1 --2- 172.21.15.38:0/1273920949 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55db3109f8c0 0x55db310a1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:49:12.020 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.010+0000 7f834affd700 10 client.15936 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:49:12.020 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.010+0000 7f836affd700 1 --2- 172.21.15.38:0/1273920949 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55db3109f8c0 0x55db310a1cd0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a949f113c5bf64f9 server_cookie=f71cef1a3cbb7cc6 in_seq=0 out_seq=0 2023-03-09T14:49:12.020 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.010+0000 7f834affd700 1 -- 172.21.15.38:0/1273920949 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+265 (crc 0 0 0) 0x55db30f2a1c0 con 0x55db3109f8c0 2023-03-09T14:49:12.021 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.010+0000 7f834affd700 10 client.15936 handle_command_reply: tid=0 2023-03-09T14:49:12.021 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.010+0000 7f837db6fb80 1 client.15936 shutdown 2023-03-09T14:49:12.021 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.011+0000 7f837db6fb80 1 -- 172.21.15.38:0/1273920949 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db31037140 msgr2=0x55db31032d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:49:12.021 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.011+0000 7f837db6fb80 1 --2- 172.21.15.38:0/1273920949 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db31037140 0x55db31032d10 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f836400b0d0 tx=0x7f8364007710 comp rx=0 tx=0).stop 2023-03-09T14:49:12.021 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.011+0000 7f837db6fb80 1 -- 172.21.15.38:0/1273920949 shutdown_connections 2023-03-09T14:49:12.021 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.011+0000 7f837db6fb80 1 --2- 172.21.15.38:0/1273920949 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55db3109f8c0 0x55db310a1cd0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:49:12.021 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.011+0000 7f837db6fb80 1 --2- 172.21.15.38:0/1273920949 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55db31037140 0x55db31032d10 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:49:12.022 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.011+0000 7f837db6fb80 1 --2- 172.21.15.38:0/1273920949 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55db31030fe0 0x55db31032290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:49:12.022 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.011+0000 7f837db6fb80 1 --2- 172.21.15.38:0/1273920949 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55db31036830 0x55db310327d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:49:12.022 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.011+0000 7f837db6fb80 1 -- 172.21.15.38:0/1273920949 >> 172.21.15.38:0/1273920949 conn(0x55db3102ba20 msgr2=0x55db310344f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:49:12.022 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.011+0000 7f834affd700 0 client.15936 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:49:12.022 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.011+0000 7f837db6fb80 1 -- 172.21.15.38:0/1273920949 shutdown_connections 2023-03-09T14:49:12.022 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.011+0000 7f837db6fb80 1 -- 172.21.15.38:0/1273920949 wait complete. 2023-03-09T14:49:12.022 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.011+0000 7f837db6fb80 20 client.15936 trim_cache size 0 max 16384 2023-03-09T14:49:12.023 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2023-03-09T14:49:12.023 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "scrub active (3 inodes in the stack)", 2023-03-09T14:49:12.023 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": { 2023-03-09T14:49:12.023 INFO:teuthology.orchestra.run.smithi038.stdout: "a14659f4-d7ea-4c5a-b15b-6138af3f909f": { 2023-03-09T14:49:12.023 INFO:teuthology.orchestra.run.smithi038.stdout: "path": "/", 2023-03-09T14:49:12.023 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "a14659f4-d7ea-4c5a-b15b-6138af3f909f", 2023-03-09T14:49:12.023 INFO:teuthology.orchestra.run.smithi038.stdout: "options": "recursive,force" 2023-03-09T14:49:12.024 INFO:teuthology.orchestra.run.smithi038.stdout: } 2023-03-09T14:49:12.024 INFO:teuthology.orchestra.run.smithi038.stdout: } 2023-03-09T14:49:12.024 INFO:teuthology.orchestra.run.smithi038.stdout:} 2023-03-09T14:49:12.024 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.012+0000 7f83719e6700 1 -- 172.21.15.38:0/3165301711 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f835409f4a0 msgr2=0x7f83540a1950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:49:12.025 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.012+0000 7f83719e6700 1 --2- 172.21.15.38:0/3165301711 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f835409f4a0 0x7f83540a1950 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f836c102230 tx=0x7f8360009620 comp rx=0 tx=0).stop 2023-03-09T14:49:12.025 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.012+0000 7f83719e6700 1 -- 172.21.15.38:0/3165301711 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f836c0ff090 msgr2=0x7f836c100a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:49:12.025 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.012+0000 7f83719e6700 1 --2- 172.21.15.38:0/3165301711 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f836c0ff090 0x7f836c100a40 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f835c00e9a0 tx=0x7f835c00ecb0 comp rx=0 tx=0).stop 2023-03-09T14:49:12.025 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.012+0000 7f83719e6700 1 -- 172.21.15.38:0/3165301711 shutdown_connections 2023-03-09T14:49:12.025 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.012+0000 7f83719e6700 1 --2- 172.21.15.38:0/3165301711 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f835409f4a0 0x7f83540a1950 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:49:12.026 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.012+0000 7f83719e6700 1 --2- 172.21.15.38:0/3165301711 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f836c104080 0x7f836c1014c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:49:12.026 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.012+0000 7f83719e6700 1 --2- 172.21.15.38:0/3165301711 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f836c0ff090 0x7f836c100a40 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:49:12.026 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.012+0000 7f83719e6700 1 --2- 172.21.15.38:0/3165301711 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f836c0ffa30 0x7f836c100f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:49:12.026 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.012+0000 7f83719e6700 1 -- 172.21.15.38:0/3165301711 >> 172.21.15.38:0/3165301711 conn(0x7f836c075e80 msgr2=0x7f836c0689f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:49:12.026 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.012+0000 7f83719e6700 1 -- 172.21.15.38:0/3165301711 shutdown_connections 2023-03-09T14:49:12.026 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:12.012+0000 7f83719e6700 1 -- 172.21.15.38:0/3165301711 wait complete. 2023-03-09T14:49:12.029 INFO:tasks.cephfs.filesystem:scrub status for tag:a14659f4-d7ea-4c5a-b15b-6138af3f909f - {'path': '/', 'tag': 'a14659f4-d7ea-4c5a-b15b-6138af3f909f', 'options': 'recursive,force'} 2023-03-09T14:49:12.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T14:49:12.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T14:49:13.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T14:49:13.717 INFO:teuthology.orchestra.run.smithi038.stderr:49300000 of 50000000 tuples (98%) done (elapsed 168.85 s, remaining 2.40 s) 2023-03-09T14:49:13.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:13 smithi122 ceph-mon[93869]: pgmap v97: 129 pgs: 129 active+clean; 9.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 15 MiB/s wr, 302 op/s 2023-03-09T14:49:13.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:13 smithi181 ceph-mon[94189]: pgmap v97: 129 pgs: 129 active+clean; 9.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 15 MiB/s wr, 302 op/s 2023-03-09T14:49:14.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:13 smithi038 ceph-mon[92422]: pgmap v97: 129 pgs: 129 active+clean; 9.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 15 MiB/s wr, 302 op/s 2023-03-09T14:49:15.341 INFO:teuthology.orchestra.run.smithi038.stderr:49400000 of 50000000 tuples (98%) done (elapsed 170.48 s, remaining 2.07 s) 2023-03-09T14:49:15.447 INFO:teuthology.orchestra.run.smithi038.stderr:49500000 of 50000000 tuples (99%) done (elapsed 170.58 s, remaining 1.72 s) 2023-03-09T14:49:15.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:15 smithi122 ceph-mon[93869]: pgmap v98: 129 pgs: 129 active+clean; 9.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 1.6 MiB/s wr, 240 op/s 2023-03-09T14:49:15.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:15 smithi181 ceph-mon[94189]: pgmap v98: 129 pgs: 129 active+clean; 9.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 1.6 MiB/s wr, 240 op/s 2023-03-09T14:49:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:15 smithi038 ceph-mon[92422]: pgmap v98: 129 pgs: 129 active+clean; 9.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 1.6 MiB/s wr, 240 op/s 2023-03-09T14:49:16.070 INFO:teuthology.orchestra.run.smithi038.stderr:49600000 of 50000000 tuples (99%) done (elapsed 171.21 s, remaining 1.38 s) 2023-03-09T14:49:16.328 INFO:teuthology.orchestra.run.smithi038.stderr:49700000 of 50000000 tuples (99%) done (elapsed 171.47 s, remaining 1.04 s) 2023-03-09T14:49:16.423 INFO:teuthology.orchestra.run.smithi038.stderr:49800000 of 50000000 tuples (99%) done (elapsed 171.56 s, remaining 0.69 s) 2023-03-09T14:49:16.524 INFO:teuthology.orchestra.run.smithi038.stderr:49900000 of 50000000 tuples (99%) done (elapsed 171.66 s, remaining 0.34 s) 2023-03-09T14:49:16.634 INFO:teuthology.orchestra.run.smithi038.stderr:50000000 of 50000000 tuples (100%) done (elapsed 171.77 s, remaining 0.00 s) 2023-03-09T14:49:16.658 INFO:teuthology.orchestra.run.smithi038.stderr:vacuum... 2023-03-09T14:49:17.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:17 smithi122 ceph-mon[93869]: pgmap v99: 129 pgs: 129 active+clean; 9.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 3.7 MiB/s wr, 357 op/s 2023-03-09T14:49:17.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:17 smithi181 ceph-mon[94189]: pgmap v99: 129 pgs: 129 active+clean; 9.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 3.7 MiB/s wr, 357 op/s 2023-03-09T14:49:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:17 smithi038 ceph-mon[92422]: pgmap v99: 129 pgs: 129 active+clean; 9.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 3.7 MiB/s wr, 357 op/s 2023-03-09T14:49:19.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:19 smithi122 ceph-mon[93869]: pgmap v100: 129 pgs: 129 active+clean; 9.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 3.0 MiB/s wr, 248 op/s 2023-03-09T14:49:19.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:19 smithi181 ceph-mon[94189]: pgmap v100: 129 pgs: 129 active+clean; 9.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 3.0 MiB/s wr, 248 op/s 2023-03-09T14:49:20.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:19 smithi038 ceph-mon[92422]: pgmap v100: 129 pgs: 129 active+clean; 9.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 3.0 MiB/s wr, 248 op/s 2023-03-09T14:49:21.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:21 smithi122 ceph-mon[93869]: pgmap v101: 129 pgs: 129 active+clean; 9.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 143 MiB/s rd, 8.6 MiB/s wr, 333 op/s 2023-03-09T14:49:21.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:21 smithi181 ceph-mon[94189]: pgmap v101: 129 pgs: 129 active+clean; 9.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 143 MiB/s rd, 8.6 MiB/s wr, 333 op/s 2023-03-09T14:49:22.008 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:21 smithi038 ceph-mon[92422]: pgmap v101: 129 pgs: 129 active+clean; 9.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 143 MiB/s rd, 8.6 MiB/s wr, 333 op/s 2023-03-09T14:49:23.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:22 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:49:23.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:22 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:49:23.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:22 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:49:24.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:23 smithi038 ceph-mon[92422]: pgmap v102: 129 pgs: 129 active+clean; 9.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 167 MiB/s rd, 8.0 MiB/s wr, 211 op/s 2023-03-09T14:49:24.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:23 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:49:24.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:23 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:49:24.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:23 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:49:24.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:23 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:49:24.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:23 smithi122 ceph-mon[93869]: pgmap v102: 129 pgs: 129 active+clean; 9.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 167 MiB/s rd, 8.0 MiB/s wr, 211 op/s 2023-03-09T14:49:24.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:23 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:49:24.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:23 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:49:24.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:23 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:49:24.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:23 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:49:24.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:23 smithi181 ceph-mon[94189]: pgmap v102: 129 pgs: 129 active+clean; 9.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 167 MiB/s rd, 8.0 MiB/s wr, 211 op/s 2023-03-09T14:49:24.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:23 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:49:24.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:23 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:49:24.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:23 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:49:24.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:23 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:49:25.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:49:25.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:49:25.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:49:25.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:49:25.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:49:25.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:49:25.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:49:25.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:49:25.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:49:26.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:25 smithi122 ceph-mon[93869]: pgmap v103: 129 pgs: 129 active+clean; 9.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 167 MiB/s rd, 8.0 MiB/s wr, 211 op/s 2023-03-09T14:49:26.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:25 smithi181 ceph-mon[94189]: pgmap v103: 129 pgs: 129 active+clean; 9.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 167 MiB/s rd, 8.0 MiB/s wr, 211 op/s 2023-03-09T14:49:26.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:25 smithi038 ceph-mon[92422]: pgmap v103: 129 pgs: 129 active+clean; 9.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 167 MiB/s rd, 8.0 MiB/s wr, 211 op/s 2023-03-09T14:49:28.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:27 smithi122 ceph-mon[93869]: pgmap v104: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 336 MiB/s rd, 153 MiB/s wr, 319 op/s 2023-03-09T14:49:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:27 smithi181 ceph-mon[94189]: pgmap v104: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 336 MiB/s rd, 153 MiB/s wr, 319 op/s 2023-03-09T14:49:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:27 smithi038 ceph-mon[92422]: pgmap v104: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 336 MiB/s rd, 153 MiB/s wr, 319 op/s 2023-03-09T14:49:30.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:29 smithi122 ceph-mon[93869]: pgmap v105: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 336 MiB/s rd, 151 MiB/s wr, 202 op/s 2023-03-09T14:49:30.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:29 smithi181 ceph-mon[94189]: pgmap v105: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 336 MiB/s rd, 151 MiB/s wr, 202 op/s 2023-03-09T14:49:30.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:29 smithi038 ceph-mon[92422]: pgmap v105: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 336 MiB/s rd, 151 MiB/s wr, 202 op/s 2023-03-09T14:49:32.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:31 smithi122 ceph-mon[93869]: pgmap v106: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 359 MiB/s rd, 174 MiB/s wr, 219 op/s 2023-03-09T14:49:32.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:31 smithi181 ceph-mon[94189]: pgmap v106: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 359 MiB/s rd, 174 MiB/s wr, 219 op/s 2023-03-09T14:49:32.256 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:31 smithi038 ceph-mon[92422]: pgmap v106: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 359 MiB/s rd, 174 MiB/s wr, 219 op/s 2023-03-09T14:49:34.136 INFO:teuthology.orchestra.run.smithi038.stderr:set primary keys... 2023-03-09T14:49:34.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:33 smithi122 ceph-mon[93869]: pgmap v107: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 338 MiB/s rd, 325 MiB/s wr, 225 op/s 2023-03-09T14:49:34.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:33 smithi181 ceph-mon[94189]: pgmap v107: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 338 MiB/s rd, 325 MiB/s wr, 225 op/s 2023-03-09T14:49:34.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:33 smithi038 ceph-mon[92422]: pgmap v107: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 338 MiB/s rd, 325 MiB/s wr, 225 op/s 2023-03-09T14:49:36.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:35 smithi122 ceph-mon[93869]: pgmap v108: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 313 MiB/s rd, 324 MiB/s wr, 216 op/s 2023-03-09T14:49:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:35 smithi181 ceph-mon[94189]: pgmap v108: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 313 MiB/s rd, 324 MiB/s wr, 216 op/s 2023-03-09T14:49:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:35 smithi038 ceph-mon[92422]: pgmap v108: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 313 MiB/s rd, 324 MiB/s wr, 216 op/s 2023-03-09T14:49:38.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:37 smithi122 ceph-mon[93869]: pgmap v109: 129 pgs: 129 active+clean; 15 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 340 MiB/s rd, 487 MiB/s wr, 275 op/s 2023-03-09T14:49:38.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:37 smithi181 ceph-mon[94189]: pgmap v109: 129 pgs: 129 active+clean; 15 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 340 MiB/s rd, 487 MiB/s wr, 275 op/s 2023-03-09T14:49:38.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:37 smithi038 ceph-mon[92422]: pgmap v109: 129 pgs: 129 active+clean; 15 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 340 MiB/s rd, 487 MiB/s wr, 275 op/s 2023-03-09T14:49:40.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:39 smithi122 ceph-mon[93869]: pgmap v110: 129 pgs: 129 active+clean; 15 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 176 MiB/s rd, 351 MiB/s wr, 171 op/s 2023-03-09T14:49:40.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:39 smithi181 ceph-mon[94189]: pgmap v110: 129 pgs: 129 active+clean; 15 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 176 MiB/s rd, 351 MiB/s wr, 171 op/s 2023-03-09T14:49:40.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:39 smithi038 ceph-mon[92422]: pgmap v110: 129 pgs: 129 active+clean; 15 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 176 MiB/s rd, 351 MiB/s wr, 171 op/s 2023-03-09T14:49:42.030 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-03-09T14:49:42.166 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.161+0000 7ff4b97c4700 1 -- 172.21.15.38:0/904298891 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4b4101ad0 msgr2=0x7ff4b4109fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:49:42.166 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.161+0000 7ff4b97c4700 1 --2- 172.21.15.38:0/904298891 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4b4101ad0 0x7ff4b4109fc0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff4a80098f0 tx=0x7ff4a8009c00 comp rx=0 tx=0).stop 2023-03-09T14:49:42.167 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.161+0000 7ff4b97c4700 1 -- 172.21.15.38:0/904298891 shutdown_connections 2023-03-09T14:49:42.167 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.161+0000 7ff4b97c4700 1 --2- 172.21.15.38:0/904298891 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4b410a590 0x7ff4b410c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:49:42.167 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.161+0000 7ff4b97c4700 1 --2- 172.21.15.38:0/904298891 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4b41011c0 0x7ff4b4101590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:49:42.167 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.161+0000 7ff4b97c4700 1 --2- 172.21.15.38:0/904298891 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4b4101ad0 0x7ff4b4109fc0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:49:42.167 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.161+0000 7ff4b97c4700 1 -- 172.21.15.38:0/904298891 >> 172.21.15.38:0/904298891 conn(0x7ff4b4075cc0 msgr2=0x7ff4b40760c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:49:42.167 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.161+0000 7ff4b97c4700 1 -- 172.21.15.38:0/904298891 shutdown_connections 2023-03-09T14:49:42.168 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.161+0000 7ff4b97c4700 1 -- 172.21.15.38:0/904298891 wait complete. 2023-03-09T14:49:42.168 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.161+0000 7ff4b97c4700 1 Processor -- start 2023-03-09T14:49:42.168 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.162+0000 7ff4b97c4700 1 -- start start 2023-03-09T14:49:42.168 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.162+0000 7ff4b97c4700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4b41011c0 0x7ff4b4195ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:49:42.168 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.162+0000 7ff4b97c4700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4b4101ad0 0x7ff4b4196530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:49:42.168 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.162+0000 7ff4b97c4700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4b410a590 0x7ff4b419aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:49:42.168 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.162+0000 7ff4b97c4700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b419b1e0 con 0x7ff4b410a590 2023-03-09T14:49:42.169 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.162+0000 7ff4b97c4700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b419b350 con 0x7ff4b4101ad0 2023-03-09T14:49:42.169 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.162+0000 7ff4b97c4700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b419b510 con 0x7ff4b41011c0 2023-03-09T14:49:42.169 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.162+0000 7ff4b27fc700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4b4101ad0 0x7ff4b4196530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:49:42.169 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.162+0000 7ff4b2ffd700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4b41011c0 0x7ff4b4195ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:49:42.169 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.162+0000 7ff4b37fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4b410a590 0x7ff4b419aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:49:42.169 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.162+0000 7ff4b27fc700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4b4101ad0 0x7ff4b4196530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.38:49128/0 (socket says 172.21.15.38:49128) 2023-03-09T14:49:42.170 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.162+0000 7ff4b2ffd700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4b41011c0 0x7ff4b4195ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.38:47634/0 (socket says 172.21.15.38:47634) 2023-03-09T14:49:42.170 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.162+0000 7ff4b27fc700 1 -- 172.21.15.38:0/250481682 learned_addr learned my addr 172.21.15.38:0/250481682 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:49:42.170 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.162+0000 7ff4b37fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4b410a590 0x7ff4b419aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39940/0 (socket says 172.21.15.38:39940) 2023-03-09T14:49:42.170 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.162+0000 7ff4b37fe700 1 -- 172.21.15.38:0/250481682 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4b41011c0 msgr2=0x7ff4b4195ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:49:42.170 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.162+0000 7ff4b37fe700 1 --2- 172.21.15.38:0/250481682 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4b41011c0 0x7ff4b4195ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:49:42.170 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.162+0000 7ff4b37fe700 1 -- 172.21.15.38:0/250481682 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4b4101ad0 msgr2=0x7ff4b4196530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:49:42.170 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.162+0000 7ff4b37fe700 1 --2- 172.21.15.38:0/250481682 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4b4101ad0 0x7ff4b4196530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:49:42.171 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.162+0000 7ff4b37fe700 1 -- 172.21.15.38:0/250481682 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4a8009520 con 0x7ff4b410a590 2023-03-09T14:49:42.171 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.162+0000 7ff4b27fc700 1 --2- 172.21.15.38:0/250481682 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4b4101ad0 0x7ff4b4196530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:49:42.171 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.162+0000 7ff4b2ffd700 1 --2- 172.21.15.38:0/250481682 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4b41011c0 0x7ff4b4195ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:49:42.171 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.162+0000 7ff4b37fe700 1 --2- 172.21.15.38:0/250481682 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4b410a590 0x7ff4b419aba0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac00efc0 tx=0x7ff4ac00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:49:42.171 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.163+0000 7ff49bfff700 1 -- 172.21.15.38:0/250481682 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4ac018070 con 0x7ff4b410a590 2023-03-09T14:49:42.171 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.163+0000 7ff49bfff700 1 -- 172.21.15.38:0/250481682 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff4ac00f040 con 0x7ff4b410a590 2023-03-09T14:49:42.172 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.163+0000 7ff4b97c4700 1 -- 172.21.15.38:0/250481682 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4b4110030 con 0x7ff4b410a590 2023-03-09T14:49:42.172 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.163+0000 7ff49bfff700 1 -- 172.21.15.38:0/250481682 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4ac00ccb0 con 0x7ff4b410a590 2023-03-09T14:49:42.172 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.163+0000 7ff4b97c4700 1 -- 172.21.15.38:0/250481682 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4b410fea0 con 0x7ff4b410a590 2023-03-09T14:49:42.180 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.166+0000 7ff49bfff700 1 -- 172.21.15.38:0/250481682 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 25) v1 ==== 133295+0+0 (secure 0 0 0) 0x7ff4ac004ad0 con 0x7ff4b410a590 2023-03-09T14:49:42.180 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.166+0000 7ff4c594db80 1 Processor -- start 2023-03-09T14:49:42.180 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.166+0000 7ff4c594db80 1 -- start start 2023-03-09T14:49:42.180 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.166+0000 7ff49bfff700 1 --2- 172.21.15.38:0/250481682 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7ff49c09f4a0 0x7ff49c0a1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:49:42.181 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.167+0000 7ff49bfff700 1 -- 172.21.15.38:0/250481682 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 9489+0+0 (secure 0 0 0) 0x7ff4ac0c9f80 con 0x7ff4b410a590 2023-03-09T14:49:42.181 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.167+0000 7ff4c594db80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560100465830 0x560100465c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:49:42.181 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.167+0000 7ff4c594db80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560100466140 0x56010045faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:49:42.181 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.167+0000 7ff4c594db80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56010045ffe0 0x560100460410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:49:42.181 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.167+0000 7ff4c594db80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x560100349da0 con 0x560100465830 2023-03-09T14:49:42.181 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.167+0000 7ff4c594db80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601003b8780 con 0x56010045ffe0 2023-03-09T14:49:42.182 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.167+0000 7ff4c594db80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601003b1cb0 con 0x560100466140 2023-03-09T14:49:42.182 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.167+0000 7ff4b27fc700 1 --2- 172.21.15.38:0/250481682 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7ff49c09f4a0 0x7ff49c0a1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:49:42.182 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.167+0000 7ff4b27fc700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560100466140 0x56010045faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:49:42.182 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.167+0000 7ff4b37fe700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56010045ffe0 0x560100460410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:49:42.182 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.167+0000 7ff4b27fc700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560100466140 0x56010045faa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.38:47642/0 (socket says 172.21.15.38:47642) 2023-03-09T14:49:42.182 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.167+0000 7ff4b37fe700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56010045ffe0 0x560100460410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.38:49136/0 (socket says 172.21.15.38:49136) 2023-03-09T14:49:42.182 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.167+0000 7ff4b27fc700 1 -- 172.21.15.38:0/2581648014 learned_addr learned my addr 172.21.15.38:0/2581648014 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:49:42.183 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.167+0000 7ff4b2ffd700 1 --2- 172.21.15.38:0/2581648014 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560100465830 0x560100465c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:49:42.183 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.167+0000 7ff4b37fe700 1 -- 172.21.15.38:0/2581648014 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560100466140 msgr2=0x56010045faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:49:42.183 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.167+0000 7ff4b37fe700 1 --2- 172.21.15.38:0/2581648014 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560100466140 0x56010045faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:49:42.183 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.167+0000 7ff4b37fe700 1 -- 172.21.15.38:0/2581648014 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560100465830 msgr2=0x560100465c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:49:42.183 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.167+0000 7ff4b37fe700 1 --2- 172.21.15.38:0/2581648014 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560100465830 0x560100465c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:49:42.183 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.167+0000 7ff4b37fe700 1 -- 172.21.15.38:0/2581648014 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560100378190 con 0x56010045ffe0 2023-03-09T14:49:42.183 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.167+0000 7ff4b2ffd700 1 --2- 172.21.15.38:0/2581648014 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560100465830 0x560100465c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:49:42.184 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.167+0000 7ff4b27fc700 1 --2- 172.21.15.38:0/2581648014 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560100466140 0x56010045faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:49:42.184 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.167+0000 7ff4b27fc700 1 --2- 172.21.15.38:0/250481682 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7ff49c09f4a0 0x7ff49c0a1950 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff4a8009fd0 tx=0x7ff4a800f4e0 comp rx=0 tx=0).ready entity=mgr.15558 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:49:42.184 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.167+0000 7ff4b37fe700 1 --2- 172.21.15.38:0/2581648014 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56010045ffe0 0x560100460410 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac009390 tx=0x7ff4ac0977c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de2258cc4961c4f5 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:49:42.184 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.168+0000 7ff4997fa700 1 -- 172.21.15.38:0/2581648014 <== mon.2 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4ac097b50 con 0x56010045ffe0 2023-03-09T14:49:42.184 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.168+0000 7ff4997fa700 1 -- 172.21.15.38:0/2581648014 <== mon.2 v2:172.21.15.122:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff4ac09fcb0 con 0x56010045ffe0 2023-03-09T14:49:42.184 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.168+0000 7ff4997fa700 1 -- 172.21.15.38:0/2581648014 <== mon.2 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4ac09fa80 con 0x56010045ffe0 2023-03-09T14:49:42.184 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.168+0000 7ff4c594db80 1 -- 172.21.15.38:0/2581648014 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56010045ffe0 msgr2=0x560100460410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:49:42.185 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.168+0000 7ff4c594db80 1 --2- 172.21.15.38:0/2581648014 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56010045ffe0 0x560100460410 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac009390 tx=0x7ff4ac0977c0 comp rx=0 tx=0).stop 2023-03-09T14:49:42.185 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.168+0000 7ff4c594db80 1 -- 172.21.15.38:0/2581648014 shutdown_connections 2023-03-09T14:49:42.185 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.168+0000 7ff4c594db80 1 --2- 172.21.15.38:0/2581648014 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560100465830 0x560100465c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:49:42.185 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.168+0000 7ff4c594db80 1 --2- 172.21.15.38:0/2581648014 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56010045ffe0 0x560100460410 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:49:42.185 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.168+0000 7ff4c594db80 1 --2- 172.21.15.38:0/2581648014 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560100466140 0x56010045faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:49:42.186 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.168+0000 7ff4c594db80 1 -- 172.21.15.38:0/2581648014 >> 172.21.15.38:0/2581648014 conn(0x56010045aa20 msgr2=0x56010045adf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:49:42.186 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.168+0000 7ff4c594db80 1 -- 172.21.15.38:0/2581648014 shutdown_connections 2023-03-09T14:49:42.186 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.168+0000 7ff4c594db80 1 -- 172.21.15.38:0/2581648014 wait complete. 2023-03-09T14:49:42.186 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.169+0000 7ff4c594db80 1 -- start start 2023-03-09T14:49:42.186 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.169+0000 7ff4c594db80 1 Processor -- start 2023-03-09T14:49:42.186 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.169+0000 7ff4c594db80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56010045ffe0 0x56010046c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:49:42.186 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.169+0000 7ff4c594db80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560100465830 0x560100469430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:49:42.187 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.169+0000 7ff4c594db80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560100466140 0x560100469b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:49:42.188 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.169+0000 7ff4c594db80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x560100235920 con 0x56010045ffe0 2023-03-09T14:49:42.188 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.169+0000 7ff4c594db80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x56010046a2b0 con 0x560100466140 2023-03-09T14:49:42.188 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.169+0000 7ff4c594db80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x56010047a080 con 0x560100465830 2023-03-09T14:49:42.188 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.169+0000 7ff4c594db80 10 client.0 _ll_register_callbacks cb 0x560100357520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:49:42.188 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.169+0000 7ff4b2ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56010045ffe0 0x56010046c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:49:42.188 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.169+0000 7ff4b2ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56010045ffe0 0x56010046c1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39950/0 (socket says 172.21.15.38:39950) 2023-03-09T14:49:42.189 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.169+0000 7ff4b27fc700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560100465830 0x560100469430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:49:42.189 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.169+0000 7ff4b2ffd700 1 -- 172.21.15.38:0/2460633343 learned_addr learned my addr 172.21.15.38:0/2460633343 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:49:42.189 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.169+0000 7ff4b37fe700 1 --2- 172.21.15.38:0/2460633343 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560100466140 0x560100469b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:49:42.189 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.170+0000 7ff4b27fc700 1 -- 172.21.15.38:0/2460633343 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560100466140 msgr2=0x560100469b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:49:42.189 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.170+0000 7ff4b27fc700 1 --2- 172.21.15.38:0/2460633343 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560100466140 0x560100469b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:49:42.189 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.170+0000 7ff4b27fc700 1 -- 172.21.15.38:0/2460633343 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56010045ffe0 msgr2=0x56010046c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:49:42.189 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.170+0000 7ff4b27fc700 1 --2- 172.21.15.38:0/2460633343 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56010045ffe0 0x56010046c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:49:42.189 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.170+0000 7ff4b27fc700 1 -- 172.21.15.38:0/2460633343 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4a801f040 con 0x560100465830 2023-03-09T14:49:42.190 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.170+0000 7ff4b2ffd700 1 --2- 172.21.15.38:0/2460633343 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56010045ffe0 0x56010046c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:49:42.190 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.170+0000 7ff4b27fc700 1 --2- 172.21.15.38:0/2460633343 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560100465830 0x560100469430 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff4a80098c0 tx=0x7ff4a8009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=acd79154c5b0fe68 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:49:42.190 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.170+0000 7ff496ff5700 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2023-03-09T14:49:42.190 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.170+0000 7ff496ff5700 1 -- 172.21.15.38:0/2460633343 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4a801c040 con 0x560100465830 2023-03-09T14:49:42.190 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.170+0000 7ff4c594db80 1 -- 172.21.15.38:0/2460633343 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56010037f190 con 0x560100465830 2023-03-09T14:49:42.190 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.170+0000 7ff496ff5700 1 -- 172.21.15.38:0/2460633343 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9489+0+0 (secure 0 0 0) 0x7ff4a8005420 con 0x560100465830 2023-03-09T14:49:42.190 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.170+0000 7ff496ff5700 1 -- 172.21.15.38:0/2460633343 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff4a80055e0 con 0x560100465830 2023-03-09T14:49:42.191 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.170+0000 7ff4c594db80 10 client.25549 fetch_fsmap learned FSMap version 27 2023-03-09T14:49:42.191 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.170+0000 7ff4c594db80 1 -- 172.21.15.38:0/2460633343 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x7ff4b410fea0 con 0x560100465830 2023-03-09T14:49:42.191 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.171+0000 7ff496ff5700 1 -- 172.21.15.38:0/2460633343 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x56010037f190 con 0x560100465830 2023-03-09T14:49:42.191 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.171+0000 7ff496ff5700 1 -- 172.21.15.38:0/2460633343 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 27) v1 ==== 7486+0+0 (secure 0 0 0) 0x7ff4a8028500 con 0x560100465830 2023-03-09T14:49:42.191 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.171+0000 7ff4c594db80 10 client.25549 fetch_fsmap finished waiting for FSMap version 27 2023-03-09T14:49:42.191 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.171+0000 7ff4c594db80 10 client.25549 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T14:49:42.191 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.171+0000 7ff4c594db80 20 client.25549 populate_metadata read hostname 'smithi038' 2023-03-09T14:49:42.192 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.171+0000 7ff4c594db80 1 --2- 172.21.15.38:0/2460633343 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x56010047e000 0x560100480410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:49:42.192 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.171+0000 7ff4c594db80 4 client.25549 mds_command: new command op to 24530 tid=0[{"prefix": "get_command_descriptions"}] 2023-03-09T14:49:42.192 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.171+0000 7ff4c594db80 1 -- 172.21.15.38:0/2460633343 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560100235f60 con 0x56010047e000 2023-03-09T14:49:42.192 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.171+0000 7ff4b2ffd700 1 --2- 172.21.15.38:0/2460633343 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x56010047e000 0x560100480410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:49:42.192 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.171+0000 7ff496ff5700 10 client.25549 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:49:42.192 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.171+0000 7ff4b2ffd700 1 --2- 172.21.15.38:0/2460633343 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x56010047e000 0x560100480410 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=933e34d7e5083104 server_cookie=f468a89c348c9f94 in_seq=0 out_seq=0 2023-03-09T14:49:42.193 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.172+0000 7ff496ff5700 1 -- 172.21.15.38:0/2460633343 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18278 (crc 0 0 0) 0x560100235f60 con 0x56010047e000 2023-03-09T14:49:42.193 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.172+0000 7ff496ff5700 10 client.25549 handle_command_reply: tid=0 2023-03-09T14:49:42.193 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.172+0000 7ff4c594db80 1 client.25549 shutdown 2023-03-09T14:49:42.193 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.172+0000 7ff4c594db80 1 -- 172.21.15.38:0/2460633343 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560100465830 msgr2=0x560100469430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:49:42.193 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.172+0000 7ff4c594db80 1 --2- 172.21.15.38:0/2460633343 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560100465830 0x560100469430 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff4a80098c0 tx=0x7ff4a8009c00 comp rx=0 tx=0).stop 2023-03-09T14:49:42.193 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.172+0000 7ff4c594db80 1 -- 172.21.15.38:0/2460633343 shutdown_connections 2023-03-09T14:49:42.193 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.172+0000 7ff4c594db80 1 --2- 172.21.15.38:0/2460633343 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x56010047e000 0x560100480410 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:49:42.194 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.172+0000 7ff4c594db80 1 --2- 172.21.15.38:0/2460633343 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56010045ffe0 0x56010046c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:49:42.194 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.172+0000 7ff496ff5700 0 client.25549 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:49:42.194 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.172+0000 7ff4c594db80 1 --2- 172.21.15.38:0/2460633343 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560100466140 0x560100469b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:49:42.194 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.172+0000 7ff4c594db80 1 --2- 172.21.15.38:0/2460633343 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560100465830 0x560100469430 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:49:42.194 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.172+0000 7ff4c594db80 1 -- 172.21.15.38:0/2460633343 >> 172.21.15.38:0/2460633343 conn(0x56010045aa20 msgr2=0x560100459c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:49:42.194 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.172+0000 7ff4c594db80 1 -- 172.21.15.38:0/2460633343 shutdown_connections 2023-03-09T14:49:42.194 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.172+0000 7ff4c594db80 1 -- 172.21.15.38:0/2460633343 wait complete. 2023-03-09T14:49:42.195 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.172+0000 7ff4c594db80 20 client.25549 trim_cache size 0 max 16384 2023-03-09T14:49:42.199 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.189+0000 7ff4c594db80 1 Processor -- start 2023-03-09T14:49:42.199 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.189+0000 7ff4c594db80 1 -- start start 2023-03-09T14:49:42.199 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.190+0000 7ff4c594db80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56010045ffe0 0x560100468bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:49:42.199 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.190+0000 7ff4c594db80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560100465830 0x56010046bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:49:42.200 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.190+0000 7ff4c594db80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560100466140 0x560100463570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:49:42.200 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.190+0000 7ff4c594db80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x56010037f190 con 0x560100466140 2023-03-09T14:49:42.200 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.190+0000 7ff4c594db80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x560100460cc0 con 0x56010045ffe0 2023-03-09T14:49:42.200 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.190+0000 7ff4c594db80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x560100456a50 con 0x560100465830 2023-03-09T14:49:42.200 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.190+0000 7ff4b2ffd700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56010045ffe0 0x560100468bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:49:42.200 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.190+0000 7ff4b37fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560100466140 0x560100463570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:49:42.200 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.190+0000 7ff4b2ffd700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56010045ffe0 0x560100468bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.38:49148/0 (socket says 172.21.15.38:49148) 2023-03-09T14:49:42.200 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.190+0000 7ff4b2ffd700 1 -- 172.21.15.38:0/88420002 learned_addr learned my addr 172.21.15.38:0/88420002 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:49:42.201 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.190+0000 7ff4b27fc700 1 --2- 172.21.15.38:0/88420002 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560100465830 0x56010046bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:49:42.201 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.190+0000 7ff4b27fc700 1 -- 172.21.15.38:0/88420002 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56010045ffe0 msgr2=0x560100468bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:49:42.201 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.190+0000 7ff4b27fc700 1 --2- 172.21.15.38:0/88420002 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56010045ffe0 0x560100468bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:49:42.201 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.190+0000 7ff4b27fc700 1 -- 172.21.15.38:0/88420002 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560100466140 msgr2=0x560100463570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:49:42.201 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.190+0000 7ff4b27fc700 1 --2- 172.21.15.38:0/88420002 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560100466140 0x560100463570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:49:42.201 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.190+0000 7ff4b27fc700 1 -- 172.21.15.38:0/88420002 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5601001c8c60 con 0x560100465830 2023-03-09T14:49:42.201 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.190+0000 7ff4b37fe700 1 --2- 172.21.15.38:0/88420002 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560100466140 0x560100463570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:49:42.202 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.190+0000 7ff4b2ffd700 1 --2- 172.21.15.38:0/88420002 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56010045ffe0 0x560100468bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:49:42.202 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.190+0000 7ff4b27fc700 1 --2- 172.21.15.38:0/88420002 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560100465830 0x56010046bd50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff4a800fab0 tx=0x7ff4a8028a00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=561a5374dc525dbd server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:49:42.202 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.191+0000 7ff499ffb700 1 -- 172.21.15.38:0/88420002 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4a801c040 con 0x560100465830 2023-03-09T14:49:42.202 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.191+0000 7ff499ffb700 1 -- 172.21.15.38:0/88420002 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff4a8005b60 con 0x560100465830 2023-03-09T14:49:42.202 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.191+0000 7ff499ffb700 1 -- 172.21.15.38:0/88420002 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4a80038c0 con 0x560100465830 2023-03-09T14:49:42.202 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.191+0000 7ff4c594db80 1 -- 172.21.15.38:0/88420002 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560100465830 msgr2=0x56010046bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:49:42.202 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.191+0000 7ff4c594db80 1 --2- 172.21.15.38:0/88420002 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560100465830 0x56010046bd50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff4a800fab0 tx=0x7ff4a8028a00 comp rx=0 tx=0).stop 2023-03-09T14:49:42.203 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.191+0000 7ff4c594db80 1 -- 172.21.15.38:0/88420002 shutdown_connections 2023-03-09T14:49:42.203 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.191+0000 7ff4c594db80 1 --2- 172.21.15.38:0/88420002 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x560100466140 0x560100463570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:49:42.203 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.191+0000 7ff4c594db80 1 --2- 172.21.15.38:0/88420002 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56010045ffe0 0x560100468bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:49:42.203 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.191+0000 7ff4c594db80 1 --2- 172.21.15.38:0/88420002 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560100465830 0x56010046bd50 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:49:42.203 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.191+0000 7ff4c594db80 1 -- 172.21.15.38:0/88420002 >> 172.21.15.38:0/88420002 conn(0x56010045aa20 msgr2=0x5601004603d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:49:42.203 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.191+0000 7ff4c594db80 1 -- 172.21.15.38:0/88420002 shutdown_connections 2023-03-09T14:49:42.203 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.192+0000 7ff4c594db80 1 -- 172.21.15.38:0/88420002 wait complete. 2023-03-09T14:49:42.203 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.192+0000 7ff4c594db80 1 -- start start 2023-03-09T14:49:42.204 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.192+0000 7ff4c594db80 1 Processor -- start 2023-03-09T14:49:42.204 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.192+0000 7ff4c594db80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56010045ffe0 0x560100462a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:49:42.204 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.192+0000 7ff4c594db80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560100465830 0x560100461290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:49:42.204 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.192+0000 7ff4c594db80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560100466140 0x5601004617d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:49:42.204 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.192+0000 7ff4c594db80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x560100475490 con 0x56010045ffe0 2023-03-09T14:49:42.204 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.192+0000 7ff4c594db80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x560100461d10 con 0x560100465830 2023-03-09T14:49:42.204 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.192+0000 7ff4c594db80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x560100461ed0 con 0x560100466140 2023-03-09T14:49:42.205 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.192+0000 7ff4c594db80 10 client.0 _ll_register_callbacks cb 0x560100357520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:49:42.205 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.192+0000 7ff4b2ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56010045ffe0 0x560100462a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:49:42.205 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.192+0000 7ff4b27fc700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560100465830 0x560100461290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:49:42.205 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.192+0000 7ff4b2ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56010045ffe0 0x560100462a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:39962/0 (socket says 172.21.15.38:39962) 2023-03-09T14:49:42.205 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.192+0000 7ff4b2ffd700 1 -- 172.21.15.38:0/1215612220 learned_addr learned my addr 172.21.15.38:0/1215612220 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:49:42.205 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.192+0000 7ff4b37fe700 1 --2- 172.21.15.38:0/1215612220 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560100466140 0x5601004617d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:49:42.206 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.193+0000 7ff4b2ffd700 1 -- 172.21.15.38:0/1215612220 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560100466140 msgr2=0x5601004617d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:49:42.206 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.193+0000 7ff4b2ffd700 1 --2- 172.21.15.38:0/1215612220 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560100466140 0x5601004617d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:49:42.206 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.193+0000 7ff4b2ffd700 1 -- 172.21.15.38:0/1215612220 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560100465830 msgr2=0x560100461290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:49:42.206 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.193+0000 7ff4b2ffd700 1 --2- 172.21.15.38:0/1215612220 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560100465830 0x560100461290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:49:42.206 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.193+0000 7ff4b2ffd700 1 -- 172.21.15.38:0/1215612220 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4a40121d0 con 0x56010045ffe0 2023-03-09T14:49:42.206 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.193+0000 7ff4b2ffd700 1 --2- 172.21.15.38:0/1215612220 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56010045ffe0 0x560100462a20 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7ff4a401d6d0 tx=0x7ff4a401d700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e32684c35f15e480 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:49:42.206 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.193+0000 7ff496ff5700 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2023-03-09T14:49:42.206 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.193+0000 7ff496ff5700 1 -- 172.21.15.38:0/1215612220 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4a401dd60 con 0x56010045ffe0 2023-03-09T14:49:42.207 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.193+0000 7ff496ff5700 1 -- 172.21.15.38:0/1215612220 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 9489+0+0 (secure 0 0 0) 0x7ff4a4013880 con 0x56010045ffe0 2023-03-09T14:49:42.207 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.193+0000 7ff4c594db80 1 -- 172.21.15.38:0/1215612220 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5601003591c0 con 0x56010045ffe0 2023-03-09T14:49:42.207 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.193+0000 7ff496ff5700 1 -- 172.21.15.38:0/1215612220 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff4a4013a40 con 0x56010045ffe0 2023-03-09T14:49:42.207 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.193+0000 7ff4c594db80 10 client.15954 fetch_fsmap learned FSMap version 27 2023-03-09T14:49:42.207 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.193+0000 7ff4c594db80 1 -- 172.21.15.38:0/1215612220 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x560100378190 con 0x56010045ffe0 2023-03-09T14:49:42.207 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.193+0000 7ff496ff5700 1 -- 172.21.15.38:0/1215612220 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5601003591c0 con 0x56010045ffe0 2023-03-09T14:49:42.207 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.193+0000 7ff496ff5700 1 -- 172.21.15.38:0/1215612220 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 27) v1 ==== 7486+0+0 (secure 0 0 0) 0x7ff4a4010430 con 0x56010045ffe0 2023-03-09T14:49:42.208 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.194+0000 7ff4c594db80 10 client.15954 fetch_fsmap finished waiting for FSMap version 27 2023-03-09T14:49:42.208 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.194+0000 7ff4c594db80 10 client.15954 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T14:49:42.208 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.194+0000 7ff4c594db80 20 client.15954 populate_metadata read hostname 'smithi038' 2023-03-09T14:49:42.208 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.194+0000 7ff4c594db80 1 --2- 172.21.15.38:0/1215612220 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x56010047e000 0x5601004d51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:49:42.208 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.194+0000 7ff4c594db80 4 client.15954 mds_command: new command op to 24530 tid=0[{"prefix": "scrub status"}] 2023-03-09T14:49:42.208 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.194+0000 7ff4c594db80 1 -- 172.21.15.38:0/1215612220 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5601003805a0 con 0x56010047e000 2023-03-09T14:49:42.208 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.194+0000 7ff4b27fc700 1 --2- 172.21.15.38:0/1215612220 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x56010047e000 0x5601004d51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:49:42.208 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.194+0000 7ff496ff5700 10 client.15954 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:49:42.209 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.194+0000 7ff4b27fc700 1 --2- 172.21.15.38:0/1215612220 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x56010047e000 0x5601004d51f0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b48e99df2a12860 server_cookie=27eff78b570c68b in_seq=0 out_seq=0 2023-03-09T14:49:42.209 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.194+0000 7ff496ff5700 1 -- 172.21.15.38:0/1215612220 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+265 (crc 0 0 0) 0x5601003805a0 con 0x56010047e000 2023-03-09T14:49:42.209 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.194+0000 7ff496ff5700 10 client.15954 handle_command_reply: tid=0 2023-03-09T14:49:42.209 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.194+0000 7ff4c594db80 1 client.15954 shutdown 2023-03-09T14:49:42.209 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.194+0000 7ff4c594db80 1 -- 172.21.15.38:0/1215612220 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56010045ffe0 msgr2=0x560100462a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:49:42.209 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.194+0000 7ff4c594db80 1 --2- 172.21.15.38:0/1215612220 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56010045ffe0 0x560100462a20 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7ff4a401d6d0 tx=0x7ff4a401d700 comp rx=0 tx=0).stop 2023-03-09T14:49:42.209 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.195+0000 7ff4c594db80 1 -- 172.21.15.38:0/1215612220 shutdown_connections 2023-03-09T14:49:42.210 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.195+0000 7ff4c594db80 1 --2- 172.21.15.38:0/1215612220 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x56010047e000 0x5601004d51f0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:49:42.210 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.195+0000 7ff4c594db80 1 --2- 172.21.15.38:0/1215612220 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56010045ffe0 0x560100462a20 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:49:42.210 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.195+0000 7ff496ff5700 0 client.15954 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:49:42.210 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.195+0000 7ff4c594db80 1 --2- 172.21.15.38:0/1215612220 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x560100466140 0x5601004617d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:49:42.210 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.195+0000 7ff4c594db80 1 --2- 172.21.15.38:0/1215612220 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x560100465830 0x560100461290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:49:42.210 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.195+0000 7ff4c594db80 1 -- 172.21.15.38:0/1215612220 >> 172.21.15.38:0/1215612220 conn(0x56010045aa20 msgr2=0x560100463570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:49:42.210 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.195+0000 7ff4c594db80 1 -- 172.21.15.38:0/1215612220 shutdown_connections 2023-03-09T14:49:42.211 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.195+0000 7ff4c594db80 1 -- 172.21.15.38:0/1215612220 wait complete. 2023-03-09T14:49:42.211 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.195+0000 7ff4c594db80 20 client.15954 trim_cache size 0 max 16384 2023-03-09T14:49:42.211 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2023-03-09T14:49:42.211 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "scrub active (3 inodes in the stack)", 2023-03-09T14:49:42.212 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": { 2023-03-09T14:49:42.212 INFO:teuthology.orchestra.run.smithi038.stdout: "a14659f4-d7ea-4c5a-b15b-6138af3f909f": { 2023-03-09T14:49:42.212 INFO:teuthology.orchestra.run.smithi038.stdout: "path": "/", 2023-03-09T14:49:42.212 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "a14659f4-d7ea-4c5a-b15b-6138af3f909f", 2023-03-09T14:49:42.212 INFO:teuthology.orchestra.run.smithi038.stdout: "options": "recursive,force" 2023-03-09T14:49:42.212 INFO:teuthology.orchestra.run.smithi038.stdout: } 2023-03-09T14:49:42.212 INFO:teuthology.orchestra.run.smithi038.stdout: } 2023-03-09T14:49:42.213 INFO:teuthology.orchestra.run.smithi038.stdout:} 2023-03-09T14:49:42.213 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.195+0000 7ff4b97c4700 1 -- 172.21.15.38:0/250481682 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7ff49c09f4a0 msgr2=0x7ff49c0a1950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:49:42.213 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.195+0000 7ff4b97c4700 1 --2- 172.21.15.38:0/250481682 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7ff49c09f4a0 0x7ff49c0a1950 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff4a8009fd0 tx=0x7ff4a800f4e0 comp rx=0 tx=0).stop 2023-03-09T14:49:42.214 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.195+0000 7ff4b97c4700 1 -- 172.21.15.38:0/250481682 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4b410a590 msgr2=0x7ff4b419aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:49:42.214 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.195+0000 7ff4b97c4700 1 --2- 172.21.15.38:0/250481682 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4b410a590 0x7ff4b419aba0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac00efc0 tx=0x7ff4ac00c3a0 comp rx=0 tx=0).stop 2023-03-09T14:49:42.214 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.195+0000 7ff4b97c4700 1 -- 172.21.15.38:0/250481682 shutdown_connections 2023-03-09T14:49:42.214 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.195+0000 7ff4b97c4700 1 --2- 172.21.15.38:0/250481682 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7ff49c09f4a0 0x7ff49c0a1950 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:49:42.214 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.195+0000 7ff4b97c4700 1 --2- 172.21.15.38:0/250481682 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff4b410a590 0x7ff4b419aba0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:49:42.214 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.195+0000 7ff4b97c4700 1 --2- 172.21.15.38:0/250481682 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff4b41011c0 0x7ff4b4195ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:49:42.214 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.195+0000 7ff4b97c4700 1 --2- 172.21.15.38:0/250481682 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff4b4101ad0 0x7ff4b4196530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:49:42.215 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.195+0000 7ff4b97c4700 1 -- 172.21.15.38:0/250481682 >> 172.21.15.38:0/250481682 conn(0x7ff4b4075cc0 msgr2=0x7ff4b40fd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:49:42.215 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.195+0000 7ff4b97c4700 1 -- 172.21.15.38:0/250481682 shutdown_connections 2023-03-09T14:49:42.215 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:49:42.195+0000 7ff4b97c4700 1 -- 172.21.15.38:0/250481682 wait complete. 2023-03-09T14:49:42.215 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:41 smithi038 ceph-mon[92422]: pgmap v111: 129 pgs: 129 active+clean; 15 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 176 MiB/s rd, 351 MiB/s wr, 175 op/s 2023-03-09T14:49:42.216 INFO:tasks.cephfs.filesystem:scrub status for tag:a14659f4-d7ea-4c5a-b15b-6138af3f909f - {'path': '/', 'tag': 'a14659f4-d7ea-4c5a-b15b-6138af3f909f', 'options': 'recursive,force'} 2023-03-09T14:49:42.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:41 smithi122 ceph-mon[93869]: pgmap v111: 129 pgs: 129 active+clean; 15 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 176 MiB/s rd, 351 MiB/s wr, 175 op/s 2023-03-09T14:49:42.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:41 smithi181 ceph-mon[94189]: pgmap v111: 129 pgs: 129 active+clean; 15 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 176 MiB/s rd, 351 MiB/s wr, 175 op/s 2023-03-09T14:49:43.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:43 smithi122 ceph-mon[93869]: pgmap v112: 129 pgs: 129 active+clean; 15 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 153 MiB/s rd, 327 MiB/s wr, 159 op/s 2023-03-09T14:49:43.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:43 smithi181 ceph-mon[94189]: pgmap v112: 129 pgs: 129 active+clean; 15 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 153 MiB/s rd, 327 MiB/s wr, 159 op/s 2023-03-09T14:49:44.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:43 smithi038 ceph-mon[92422]: pgmap v112: 129 pgs: 129 active+clean; 15 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 153 MiB/s rd, 327 MiB/s wr, 159 op/s 2023-03-09T14:49:45.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:45 smithi122 ceph-mon[93869]: pgmap v113: 129 pgs: 129 active+clean; 15 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 31 MiB/s rd, 171 MiB/s wr, 67 op/s 2023-03-09T14:49:45.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:45 smithi181 ceph-mon[94189]: pgmap v113: 129 pgs: 129 active+clean; 15 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 31 MiB/s rd, 171 MiB/s wr, 67 op/s 2023-03-09T14:49:46.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:45 smithi038 ceph-mon[92422]: pgmap v113: 129 pgs: 129 active+clean; 15 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 31 MiB/s rd, 171 MiB/s wr, 67 op/s 2023-03-09T14:49:47.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:47 smithi122 ceph-mon[93869]: pgmap v114: 129 pgs: 129 active+clean; 15 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 31 MiB/s rd, 197 MiB/s wr, 89 op/s 2023-03-09T14:49:47.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:47 smithi181 ceph-mon[94189]: pgmap v114: 129 pgs: 129 active+clean; 15 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 31 MiB/s rd, 197 MiB/s wr, 89 op/s 2023-03-09T14:49:48.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:47 smithi038 ceph-mon[92422]: pgmap v114: 129 pgs: 129 active+clean; 15 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 31 MiB/s rd, 197 MiB/s wr, 89 op/s 2023-03-09T14:49:49.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:49 smithi122 ceph-mon[93869]: pgmap v115: 129 pgs: 129 active+clean; 15 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 40 MiB/s wr, 70 op/s 2023-03-09T14:49:49.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:49 smithi181 ceph-mon[94189]: pgmap v115: 129 pgs: 129 active+clean; 15 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 40 MiB/s wr, 70 op/s 2023-03-09T14:49:50.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:49 smithi038 ceph-mon[92422]: pgmap v115: 129 pgs: 129 active+clean; 15 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 40 MiB/s wr, 70 op/s 2023-03-09T14:49:51.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:51 smithi122 ceph-mon[93869]: pgmap v116: 129 pgs: 129 active+clean; 15 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 50 MiB/s wr, 121 op/s 2023-03-09T14:49:51.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:51 smithi181 ceph-mon[94189]: pgmap v116: 129 pgs: 129 active+clean; 15 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 50 MiB/s wr, 121 op/s 2023-03-09T14:49:52.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:51 smithi038 ceph-mon[92422]: pgmap v116: 129 pgs: 129 active+clean; 15 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 50 MiB/s wr, 121 op/s 2023-03-09T14:49:52.455 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 2023-03-09T14:49:52.020+0000 7fda9b5ce700 -1 osd.2 pg_epoch: 95 pg[4.1s0( v 95'2133 (0'0,95'2133] local-lis/les=85/86 n=159 ec=85/85 lis/c=85/85 les/c/f=86/86/0 sis=85) [2,1,4,7]p2(0) r=0 lpr=85 luod=95'2128 crt=95'2128 lcod 95'2126 mlcod 95'2126 active+clean] bad op order, already applied 42479 > this 42476 2023-03-09T14:49:52.456 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7fda9b5ce700 time 2023-03-09T14:49:52.020481+0000 2023-03-09T14:49:52.456 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op") 2023-03-09T14:49:52.456 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:49:52.456 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x55668ac3ac85] 2023-03-09T14:49:52.456 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 2: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55668aef36e4] 2023-03-09T14:49:52.456 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55668aef9d5e] 2023-03-09T14:49:52.457 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 4: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55668ad49986] 2023-03-09T14:49:52.457 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 5: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55668b05f729] 2023-03-09T14:49:52.457 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55668ad5eebf] 2023-03-09T14:49:52.457 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55668b410d85] 2023-03-09T14:49:52.457 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55668b413f84] 2023-03-09T14:49:52.457 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7fdac2b371ca] 2023-03-09T14:49:52.457 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 10: clone() 2023-03-09T14:49:52.458 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 2023-03-09T14:49:52.023+0000 7fda9b5ce700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7fda9b5ce700 time 2023-03-09T14:49:52.020481+0000 2023-03-09T14:49:52.458 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op") 2023-03-09T14:49:52.458 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 2023-03-09T14:49:52.458 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:49:52.458 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x55668ac3ac85] 2023-03-09T14:49:52.458 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 2: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55668aef36e4] 2023-03-09T14:49:52.458 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55668aef9d5e] 2023-03-09T14:49:52.459 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 4: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55668ad49986] 2023-03-09T14:49:52.459 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 5: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55668b05f729] 2023-03-09T14:49:52.459 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55668ad5eebf] 2023-03-09T14:49:52.459 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55668b410d85] 2023-03-09T14:49:52.459 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55668b413f84] 2023-03-09T14:49:52.459 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7fdac2b371ca] 2023-03-09T14:49:52.459 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 10: clone() 2023-03-09T14:49:52.460 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 2023-03-09T14:49:52.460 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: *** Caught signal (Aborted) ** 2023-03-09T14:49:52.460 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: in thread 7fda9b5ce700 thread_name:tp_osd_tp 2023-03-09T14:49:52.460 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:49:52.460 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7fdac2b41cf0] 2023-03-09T14:49:52.460 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 2: gsignal() 2023-03-09T14:49:52.460 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 3: abort() 2023-03-09T14:49:52.460 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x55668ac3ad45] 2023-03-09T14:49:52.461 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 5: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55668aef36e4] 2023-03-09T14:49:52.461 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 6: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55668aef9d5e] 2023-03-09T14:49:52.461 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55668ad49986] 2023-03-09T14:49:52.461 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 8: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55668b05f729] 2023-03-09T14:49:52.461 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55668ad5eebf] 2023-03-09T14:49:52.461 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55668b410d85] 2023-03-09T14:49:52.462 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55668b413f84] 2023-03-09T14:49:52.462 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fdac2b371ca] 2023-03-09T14:49:52.462 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 13: clone() 2023-03-09T14:49:52.463 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 2023-03-09T14:49:52.026+0000 7fda9b5ce700 -1 *** Caught signal (Aborted) ** 2023-03-09T14:49:52.463 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: in thread 7fda9b5ce700 thread_name:tp_osd_tp 2023-03-09T14:49:52.463 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 2023-03-09T14:49:52.463 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:49:52.463 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7fdac2b41cf0] 2023-03-09T14:49:52.463 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 2: gsignal() 2023-03-09T14:49:52.463 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 3: abort() 2023-03-09T14:49:52.463 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x55668ac3ad45] 2023-03-09T14:49:52.464 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 5: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55668aef36e4] 2023-03-09T14:49:52.464 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 6: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55668aef9d5e] 2023-03-09T14:49:52.464 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55668ad49986] 2023-03-09T14:49:52.464 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 8: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55668b05f729] 2023-03-09T14:49:52.464 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55668ad5eebf] 2023-03-09T14:49:52.464 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55668b410d85] 2023-03-09T14:49:52.464 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55668b413f84] 2023-03-09T14:49:52.465 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fdac2b371ca] 2023-03-09T14:49:52.465 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 13: clone() 2023-03-09T14:49:52.465 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-03-09T14:49:52.465 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 2023-03-09T14:49:52.465 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: -352> 2023-03-09T14:49:52.020+0000 7fda9b5ce700 -1 osd.2 pg_epoch: 95 pg[4.1s0( v 95'2133 (0'0,95'2133] local-lis/les=85/86 n=159 ec=85/85 lis/c=85/85 les/c/f=86/86/0 sis=85) [2,1,4,7]p2(0) r=0 lpr=85 luod=95'2128 crt=95'2128 lcod 95'2126 mlcod 95'2126 active+clean] bad op order, already applied 42479 > this 42476 2023-03-09T14:49:52.465 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: -37> 2023-03-09T14:49:52.023+0000 7fda9b5ce700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7fda9b5ce700 time 2023-03-09T14:49:52.020481+0000 2023-03-09T14:49:52.465 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op") 2023-03-09T14:49:52.466 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 2023-03-09T14:49:52.466 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:49:52.466 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x55668ac3ac85] 2023-03-09T14:49:52.466 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 2: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55668aef36e4] 2023-03-09T14:49:52.466 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55668aef9d5e] 2023-03-09T14:49:52.466 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 4: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55668ad49986] 2023-03-09T14:49:52.466 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 5: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55668b05f729] 2023-03-09T14:49:52.466 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55668ad5eebf] 2023-03-09T14:49:52.467 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55668b410d85] 2023-03-09T14:49:52.467 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55668b413f84] 2023-03-09T14:49:52.467 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7fdac2b371ca] 2023-03-09T14:49:52.467 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 10: clone() 2023-03-09T14:49:52.467 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 2023-03-09T14:49:52.467 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 0> 2023-03-09T14:49:52.026+0000 7fda9b5ce700 -1 *** Caught signal (Aborted) ** 2023-03-09T14:49:52.467 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: in thread 7fda9b5ce700 thread_name:tp_osd_tp 2023-03-09T14:49:52.468 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 2023-03-09T14:49:52.468 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:49:52.468 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7fdac2b41cf0] 2023-03-09T14:49:52.468 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 2: gsignal() 2023-03-09T14:49:52.468 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 3: abort() 2023-03-09T14:49:52.468 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x55668ac3ad45] 2023-03-09T14:49:52.468 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 5: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55668aef36e4] 2023-03-09T14:49:52.468 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 6: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55668aef9d5e] 2023-03-09T14:49:52.469 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55668ad49986] 2023-03-09T14:49:52.469 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 8: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55668b05f729] 2023-03-09T14:49:52.469 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55668ad5eebf] 2023-03-09T14:49:52.469 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55668b410d85] 2023-03-09T14:49:52.469 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55668b413f84] 2023-03-09T14:49:52.469 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fdac2b371ca] 2023-03-09T14:49:52.469 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 13: clone() 2023-03-09T14:49:52.470 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-03-09T14:49:52.470 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 2023-03-09T14:49:52.470 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: -352> 2023-03-09T14:49:52.020+0000 7fda9b5ce700 -1 osd.2 pg_epoch: 95 pg[4.1s0( v 95'2133 (0'0,95'2133] local-lis/les=85/86 n=159 ec=85/85 lis/c=85/85 les/c/f=86/86/0 sis=85) [2,1,4,7]p2(0) r=0 lpr=85 luod=95'2128 crt=95'2128 lcod 95'2126 mlcod 95'2126 active+clean] bad op order, already applied 42479 > this 42476 2023-03-09T14:49:52.470 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: -37> 2023-03-09T14:49:52.023+0000 7fda9b5ce700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7fda9b5ce700 time 2023-03-09T14:49:52.020481+0000 2023-03-09T14:49:52.470 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op") 2023-03-09T14:49:52.470 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 2023-03-09T14:49:52.470 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:49:52.471 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x55668ac3ac85] 2023-03-09T14:49:52.471 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 2: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55668aef36e4] 2023-03-09T14:49:52.471 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55668aef9d5e] 2023-03-09T14:49:52.471 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 4: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55668ad49986] 2023-03-09T14:49:52.471 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 5: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55668b05f729] 2023-03-09T14:49:52.471 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55668ad5eebf] 2023-03-09T14:49:52.471 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55668b410d85] 2023-03-09T14:49:52.471 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55668b413f84] 2023-03-09T14:49:52.472 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7fdac2b371ca] 2023-03-09T14:49:52.472 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 10: clone() 2023-03-09T14:49:52.472 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 2023-03-09T14:49:52.472 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 0> 2023-03-09T14:49:52.026+0000 7fda9b5ce700 -1 *** Caught signal (Aborted) ** 2023-03-09T14:49:52.472 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: in thread 7fda9b5ce700 thread_name:tp_osd_tp 2023-03-09T14:49:52.472 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 2023-03-09T14:49:52.472 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:49:52.472 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7fdac2b41cf0] 2023-03-09T14:49:52.473 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 2: gsignal() 2023-03-09T14:49:52.473 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 3: abort() 2023-03-09T14:49:52.473 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x55668ac3ad45] 2023-03-09T14:49:52.473 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 5: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55668aef36e4] 2023-03-09T14:49:52.473 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 6: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55668aef9d5e] 2023-03-09T14:49:52.473 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55668ad49986] 2023-03-09T14:49:52.473 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 8: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55668b05f729] 2023-03-09T14:49:52.474 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55668ad5eebf] 2023-03-09T14:49:52.474 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55668b410d85] 2023-03-09T14:49:52.474 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55668b413f84] 2023-03-09T14:49:52.474 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fdac2b371ca] 2023-03-09T14:49:52.474 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 13: clone() 2023-03-09T14:49:52.474 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-03-09T14:49:52.474 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[99987]: 2023-03-09T14:49:52.474 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:52 smithi181 systemd[1]: ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.2.service: Main process exited, code=exited, status=134/n/a 2023-03-09T14:49:52.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-mon[94189]: osd.2 reported immediately failed by osd.11 2023-03-09T14:49:52.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-mon[94189]: osd.2 failed (root=default,host=smithi181) (connection refused reported by osd.11) 2023-03-09T14:49:52.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-mon[94189]: osd.2 reported immediately failed by osd.8 2023-03-09T14:49:52.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-mon[94189]: osd.2 reported immediately failed by osd.0 2023-03-09T14:49:52.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-mon[94189]: osd.2 reported immediately failed by osd.3 2023-03-09T14:49:52.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-mon[94189]: osd.2 reported immediately failed by osd.6 2023-03-09T14:49:52.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-mon[94189]: osd.2 reported immediately failed by osd.9 2023-03-09T14:49:52.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-mon[94189]: osd.2 reported immediately failed by osd.5 2023-03-09T14:49:52.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-mon[94189]: osd.2 reported immediately failed by osd.4 2023-03-09T14:49:52.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-mon[94189]: osd.2 reported immediately failed by osd.0 2023-03-09T14:49:52.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-mon[94189]: osd.2 reported immediately failed by osd.3 2023-03-09T14:49:52.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-mon[94189]: osd.2 reported immediately failed by osd.6 2023-03-09T14:49:52.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-mon[94189]: osd.2 reported immediately failed by osd.9 2023-03-09T14:49:52.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-mon[94189]: osd.2 reported immediately failed by osd.11 2023-03-09T14:49:52.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-mon[94189]: osd.2 reported immediately failed by osd.10 2023-03-09T14:49:52.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-mon[94189]: osd.2 reported immediately failed by osd.4 2023-03-09T14:49:52.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-mon[94189]: osd.2 reported immediately failed by osd.10 2023-03-09T14:49:52.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-mon[94189]: osd.2 reported immediately failed by osd.7 2023-03-09T14:49:52.736 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-mon[94189]: osd.2 reported immediately failed by osd.5 2023-03-09T14:49:52.736 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-mon[94189]: osd.2 reported immediately failed by osd.1 2023-03-09T14:49:52.736 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-mon[94189]: osd.2 reported immediately failed by osd.8 2023-03-09T14:49:52.736 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-mon[94189]: osd.2 reported immediately failed by osd.7 2023-03-09T14:49:52.736 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:52 smithi181 ceph-mon[94189]: osd.2 reported immediately failed by osd.1 2023-03-09T14:49:52.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:52 smithi122 ceph-mon[93869]: osd.2 reported immediately failed by osd.11 2023-03-09T14:49:52.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:52 smithi122 ceph-mon[93869]: osd.2 failed (root=default,host=smithi181) (connection refused reported by osd.11) 2023-03-09T14:49:52.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:52 smithi122 ceph-mon[93869]: osd.2 reported immediately failed by osd.8 2023-03-09T14:49:52.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:52 smithi122 ceph-mon[93869]: osd.2 reported immediately failed by osd.0 2023-03-09T14:49:52.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:52 smithi122 ceph-mon[93869]: osd.2 reported immediately failed by osd.3 2023-03-09T14:49:52.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:52 smithi122 ceph-mon[93869]: osd.2 reported immediately failed by osd.6 2023-03-09T14:49:52.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:52 smithi122 ceph-mon[93869]: osd.2 reported immediately failed by osd.9 2023-03-09T14:49:52.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:52 smithi122 ceph-mon[93869]: osd.2 reported immediately failed by osd.5 2023-03-09T14:49:52.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:52 smithi122 ceph-mon[93869]: osd.2 reported immediately failed by osd.4 2023-03-09T14:49:52.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:52 smithi122 ceph-mon[93869]: osd.2 reported immediately failed by osd.0 2023-03-09T14:49:52.980 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:52 smithi122 ceph-mon[93869]: osd.2 reported immediately failed by osd.3 2023-03-09T14:49:52.980 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:52 smithi122 ceph-mon[93869]: osd.2 reported immediately failed by osd.6 2023-03-09T14:49:52.980 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:52 smithi122 ceph-mon[93869]: osd.2 reported immediately failed by osd.9 2023-03-09T14:49:52.980 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:52 smithi122 ceph-mon[93869]: osd.2 reported immediately failed by osd.11 2023-03-09T14:49:52.980 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:52 smithi122 ceph-mon[93869]: osd.2 reported immediately failed by osd.10 2023-03-09T14:49:52.980 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:52 smithi122 ceph-mon[93869]: osd.2 reported immediately failed by osd.4 2023-03-09T14:49:52.980 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:52 smithi122 ceph-mon[93869]: osd.2 reported immediately failed by osd.10 2023-03-09T14:49:52.980 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:52 smithi122 ceph-mon[93869]: osd.2 reported immediately failed by osd.7 2023-03-09T14:49:52.981 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:52 smithi122 ceph-mon[93869]: osd.2 reported immediately failed by osd.5 2023-03-09T14:49:52.981 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:52 smithi122 ceph-mon[93869]: osd.2 reported immediately failed by osd.1 2023-03-09T14:49:52.981 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:52 smithi122 ceph-mon[93869]: osd.2 reported immediately failed by osd.8 2023-03-09T14:49:52.981 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:52 smithi122 ceph-mon[93869]: osd.2 reported immediately failed by osd.7 2023-03-09T14:49:52.981 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:52 smithi122 ceph-mon[93869]: osd.2 reported immediately failed by osd.1 2023-03-09T14:49:53.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:52 smithi038 ceph-mon[92422]: osd.2 reported immediately failed by osd.11 2023-03-09T14:49:53.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:52 smithi038 ceph-mon[92422]: osd.2 failed (root=default,host=smithi181) (connection refused reported by osd.11) 2023-03-09T14:49:53.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:52 smithi038 ceph-mon[92422]: osd.2 reported immediately failed by osd.8 2023-03-09T14:49:53.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:52 smithi038 ceph-mon[92422]: osd.2 reported immediately failed by osd.0 2023-03-09T14:49:53.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:52 smithi038 ceph-mon[92422]: osd.2 reported immediately failed by osd.3 2023-03-09T14:49:53.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:52 smithi038 ceph-mon[92422]: osd.2 reported immediately failed by osd.6 2023-03-09T14:49:53.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:52 smithi038 ceph-mon[92422]: osd.2 reported immediately failed by osd.9 2023-03-09T14:49:53.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:52 smithi038 ceph-mon[92422]: osd.2 reported immediately failed by osd.5 2023-03-09T14:49:53.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:52 smithi038 ceph-mon[92422]: osd.2 reported immediately failed by osd.4 2023-03-09T14:49:53.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:52 smithi038 ceph-mon[92422]: osd.2 reported immediately failed by osd.0 2023-03-09T14:49:53.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:52 smithi038 ceph-mon[92422]: osd.2 reported immediately failed by osd.3 2023-03-09T14:49:53.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:52 smithi038 ceph-mon[92422]: osd.2 reported immediately failed by osd.6 2023-03-09T14:49:53.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:52 smithi038 ceph-mon[92422]: osd.2 reported immediately failed by osd.9 2023-03-09T14:49:53.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:52 smithi038 ceph-mon[92422]: osd.2 reported immediately failed by osd.11 2023-03-09T14:49:53.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:52 smithi038 ceph-mon[92422]: osd.2 reported immediately failed by osd.10 2023-03-09T14:49:53.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:52 smithi038 ceph-mon[92422]: osd.2 reported immediately failed by osd.4 2023-03-09T14:49:53.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:52 smithi038 ceph-mon[92422]: osd.2 reported immediately failed by osd.10 2023-03-09T14:49:53.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:52 smithi038 ceph-mon[92422]: osd.2 reported immediately failed by osd.7 2023-03-09T14:49:53.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:52 smithi038 ceph-mon[92422]: osd.2 reported immediately failed by osd.5 2023-03-09T14:49:53.004 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:52 smithi038 ceph-mon[92422]: osd.2 reported immediately failed by osd.1 2023-03-09T14:49:53.004 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:52 smithi038 ceph-mon[92422]: osd.2 reported immediately failed by osd.8 2023-03-09T14:49:53.004 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:52 smithi038 ceph-mon[92422]: osd.2 reported immediately failed by osd.7 2023-03-09T14:49:53.004 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:52 smithi038 ceph-mon[92422]: osd.2 reported immediately failed by osd.1 2023-03-09T14:49:53.859 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:53 smithi181 ceph-mon[94189]: pgmap v117: 129 pgs: 129 active+clean; 15 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 60 MiB/s wr, 163 op/s 2023-03-09T14:49:53.859 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:53 smithi181 ceph-mon[94189]: Health check failed: 1 osds down (OSD_DOWN) 2023-03-09T14:49:53.860 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:53 smithi181 ceph-mon[94189]: osdmap e96: 12 total, 11 up, 12 in 2023-03-09T14:49:53.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-mon[93869]: pgmap v117: 129 pgs: 129 active+clean; 15 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 60 MiB/s wr, 163 op/s 2023-03-09T14:49:53.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-mon[93869]: Health check failed: 1 osds down (OSD_DOWN) 2023-03-09T14:49:53.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-mon[93869]: osdmap e96: 12 total, 11 up, 12 in 2023-03-09T14:49:53.979 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 2023-03-09T14:49:53.682+0000 7f8bad507700 -1 osd.1 pg_epoch: 97 pg[4.1s1( v 97'2135 (0'0,97'2135] local-lis/les=96/97 n=159 ec=85/85 lis/c=96/85 les/c/f=97/86/0 sis=96) [NONE,1,4,7]p1(1) r=1 lpr=96 pi=[85,96)/1 luod=97'2134 crt=95'2133 lcod 95'2133 mlcod 95'2133 active+undersized+degraded mbc={}] bad op order, already applied 42480 > this 42478 2023-03-09T14:49:53.980 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7f8bad507700 time 2023-03-09T14:49:53.683494+0000 2023-03-09T14:49:53.980 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op") 2023-03-09T14:49:53.980 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:49:53.980 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x56344dabdc85] 2023-03-09T14:49:53.980 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 2: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x56344dd766e4] 2023-03-09T14:49:53.980 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x56344dd7cd5e] 2023-03-09T14:49:53.980 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 4: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x56344dbcc986] 2023-03-09T14:49:53.981 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 5: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x56344dee2729] 2023-03-09T14:49:53.981 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x56344dbe1ebf] 2023-03-09T14:49:53.981 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x56344e293d85] 2023-03-09T14:49:53.981 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x56344e296f84] 2023-03-09T14:49:53.981 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f8bd326d1ca] 2023-03-09T14:49:53.981 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 10: clone() 2023-03-09T14:49:53.981 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 2023-03-09T14:49:53.686+0000 7f8bad507700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7f8bad507700 time 2023-03-09T14:49:53.683494+0000 2023-03-09T14:49:53.982 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op") 2023-03-09T14:49:53.982 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 2023-03-09T14:49:53.982 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:49:53.982 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x56344dabdc85] 2023-03-09T14:49:53.982 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 2: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x56344dd766e4] 2023-03-09T14:49:53.982 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x56344dd7cd5e] 2023-03-09T14:49:53.983 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 4: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x56344dbcc986] 2023-03-09T14:49:53.983 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 5: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x56344dee2729] 2023-03-09T14:49:53.983 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x56344dbe1ebf] 2023-03-09T14:49:53.983 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x56344e293d85] 2023-03-09T14:49:53.983 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x56344e296f84] 2023-03-09T14:49:53.983 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f8bd326d1ca] 2023-03-09T14:49:53.983 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 10: clone() 2023-03-09T14:49:53.984 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 2023-03-09T14:49:53.984 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: *** Caught signal (Aborted) ** 2023-03-09T14:49:53.984 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: in thread 7f8bad507700 thread_name:tp_osd_tp 2023-03-09T14:49:53.984 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:49:53.984 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f8bd3277cf0] 2023-03-09T14:49:53.984 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 2: gsignal() 2023-03-09T14:49:53.984 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 3: abort() 2023-03-09T14:49:53.985 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x56344dabdd45] 2023-03-09T14:49:53.985 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 5: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x56344dd766e4] 2023-03-09T14:49:53.985 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 6: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x56344dd7cd5e] 2023-03-09T14:49:53.986 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x56344dbcc986] 2023-03-09T14:49:53.986 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 8: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x56344dee2729] 2023-03-09T14:49:53.986 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x56344dbe1ebf] 2023-03-09T14:49:53.986 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x56344e293d85] 2023-03-09T14:49:53.986 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x56344e296f84] 2023-03-09T14:49:53.986 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f8bd326d1ca] 2023-03-09T14:49:53.987 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 13: clone() 2023-03-09T14:49:53.987 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 2023-03-09T14:49:53.690+0000 7f8bad507700 -1 *** Caught signal (Aborted) ** 2023-03-09T14:49:53.987 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: in thread 7f8bad507700 thread_name:tp_osd_tp 2023-03-09T14:49:53.987 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 2023-03-09T14:49:53.987 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:49:53.987 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f8bd3277cf0] 2023-03-09T14:49:53.987 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 2: gsignal() 2023-03-09T14:49:53.987 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 3: abort() 2023-03-09T14:49:53.988 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x56344dabdd45] 2023-03-09T14:49:53.988 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 5: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x56344dd766e4] 2023-03-09T14:49:53.988 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 6: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x56344dd7cd5e] 2023-03-09T14:49:53.988 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x56344dbcc986] 2023-03-09T14:49:53.988 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 8: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x56344dee2729] 2023-03-09T14:49:53.988 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x56344dbe1ebf] 2023-03-09T14:49:53.988 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x56344e293d85] 2023-03-09T14:49:53.989 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x56344e296f84] 2023-03-09T14:49:53.989 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f8bd326d1ca] 2023-03-09T14:49:53.989 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 13: clone() 2023-03-09T14:49:53.989 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-03-09T14:49:53.989 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 2023-03-09T14:49:53.989 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: -22> 2023-03-09T14:49:53.682+0000 7f8bad507700 -1 osd.1 pg_epoch: 97 pg[4.1s1( v 97'2135 (0'0,97'2135] local-lis/les=96/97 n=159 ec=85/85 lis/c=96/85 les/c/f=97/86/0 sis=96) [NONE,1,4,7]p1(1) r=1 lpr=96 pi=[85,96)/1 luod=97'2134 crt=95'2133 lcod 95'2133 mlcod 95'2133 active+undersized+degraded mbc={}] bad op order, already applied 42480 > this 42478 2023-03-09T14:49:53.990 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: -1> 2023-03-09T14:49:53.686+0000 7f8bad507700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7f8bad507700 time 2023-03-09T14:49:53.683494+0000 2023-03-09T14:49:53.990 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op") 2023-03-09T14:49:53.990 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 2023-03-09T14:49:53.991 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:49:53.991 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x56344dabdc85] 2023-03-09T14:49:53.991 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 2: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x56344dd766e4] 2023-03-09T14:49:53.991 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x56344dd7cd5e] 2023-03-09T14:49:53.991 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 4: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x56344dbcc986] 2023-03-09T14:49:53.992 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 5: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x56344dee2729] 2023-03-09T14:49:53.992 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x56344dbe1ebf] 2023-03-09T14:49:53.992 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x56344e293d85] 2023-03-09T14:49:53.992 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x56344e296f84] 2023-03-09T14:49:53.992 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f8bd326d1ca] 2023-03-09T14:49:53.992 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 10: clone() 2023-03-09T14:49:53.992 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 2023-03-09T14:49:53.993 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 0> 2023-03-09T14:49:53.690+0000 7f8bad507700 -1 *** Caught signal (Aborted) ** 2023-03-09T14:49:53.993 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: in thread 7f8bad507700 thread_name:tp_osd_tp 2023-03-09T14:49:53.993 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 2023-03-09T14:49:53.993 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:49:53.993 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f8bd3277cf0] 2023-03-09T14:49:53.993 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 2: gsignal() 2023-03-09T14:49:53.994 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 3: abort() 2023-03-09T14:49:53.994 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x56344dabdd45] 2023-03-09T14:49:53.994 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 5: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x56344dd766e4] 2023-03-09T14:49:53.994 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 6: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x56344dd7cd5e] 2023-03-09T14:49:53.994 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x56344dbcc986] 2023-03-09T14:49:53.994 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 8: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x56344dee2729] 2023-03-09T14:49:53.994 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x56344dbe1ebf] 2023-03-09T14:49:53.995 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x56344e293d85] 2023-03-09T14:49:53.995 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x56344e296f84] 2023-03-09T14:49:53.995 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f8bd326d1ca] 2023-03-09T14:49:53.995 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 13: clone() 2023-03-09T14:49:53.995 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-03-09T14:49:53.995 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 2023-03-09T14:49:53.995 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: -1023> 2023-03-09T14:49:53.682+0000 7f8bad507700 -1 osd.1 pg_epoch: 97 pg[4.1s1( v 97'2135 (0'0,97'2135] local-lis/les=96/97 n=159 ec=85/85 lis/c=96/85 les/c/f=97/86/0 sis=96) [NONE,1,4,7]p1(1) r=1 lpr=96 pi=[85,96)/1 luod=97'2134 crt=95'2133 lcod 95'2133 mlcod 95'2133 active+undersized+degraded mbc={}] bad op order, already applied 42480 > this 42478 2023-03-09T14:49:53.995 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: -1002> 2023-03-09T14:49:53.686+0000 7f8bad507700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7f8bad507700 time 2023-03-09T14:49:53.683494+0000 2023-03-09T14:49:53.996 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op") 2023-03-09T14:49:53.996 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 2023-03-09T14:49:53.996 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:49:53.996 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x56344dabdc85] 2023-03-09T14:49:53.996 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 2: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x56344dd766e4] 2023-03-09T14:49:53.997 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x56344dd7cd5e] 2023-03-09T14:49:53.997 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 4: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x56344dbcc986] 2023-03-09T14:49:53.997 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 5: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x56344dee2729] 2023-03-09T14:49:53.997 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x56344dbe1ebf] 2023-03-09T14:49:53.997 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x56344e293d85] 2023-03-09T14:49:53.997 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x56344e296f84] 2023-03-09T14:49:53.998 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f8bd326d1ca] 2023-03-09T14:49:53.998 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 10: clone() 2023-03-09T14:49:53.998 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 2023-03-09T14:49:53.998 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: -1001> 2023-03-09T14:49:53.690+0000 7f8bad507700 -1 *** Caught signal (Aborted) ** 2023-03-09T14:49:53.998 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: in thread 7f8bad507700 thread_name:tp_osd_tp 2023-03-09T14:49:53.998 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 2023-03-09T14:49:53.998 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:49:53.998 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f8bd3277cf0] 2023-03-09T14:49:53.999 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 2: gsignal() 2023-03-09T14:49:53.999 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 3: abort() 2023-03-09T14:49:53.999 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x56344dabdd45] 2023-03-09T14:49:53.999 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 5: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x56344dd766e4] 2023-03-09T14:49:53.999 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 6: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x56344dd7cd5e] 2023-03-09T14:49:53.999 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x56344dbcc986] 2023-03-09T14:49:53.999 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 8: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x56344dee2729] 2023-03-09T14:49:54.000 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x56344dbe1ebf] 2023-03-09T14:49:54.000 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x56344e293d85] 2023-03-09T14:49:54.000 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x56344e296f84] 2023-03-09T14:49:54.000 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f8bd326d1ca] 2023-03-09T14:49:54.000 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 13: clone() 2023-03-09T14:49:54.000 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-03-09T14:49:54.000 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[98835]: 2023-03-09T14:49:54.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:53 smithi038 ceph-mon[92422]: pgmap v117: 129 pgs: 129 active+clean; 15 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 60 MiB/s wr, 163 op/s 2023-03-09T14:49:54.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:53 smithi038 ceph-mon[92422]: Health check failed: 1 osds down (OSD_DOWN) 2023-03-09T14:49:54.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:53 smithi038 ceph-mon[92422]: osdmap e96: 12 total, 11 up, 12 in 2023-03-09T14:49:54.231 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:49:53 smithi181 systemd[1]: ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.2.service: Failed with result 'exit-code'. 2023-03-09T14:49:54.676 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:54 smithi122 ceph-mon[93869]: osdmap e97: 12 total, 11 up, 12 in 2023-03-09T14:49:54.676 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.11 2023-03-09T14:49:54.677 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:54 smithi122 ceph-mon[93869]: osd.1 failed (root=default,host=smithi122) (connection refused reported by osd.11) 2023-03-09T14:49:54.677 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.11 2023-03-09T14:49:54.677 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.5 2023-03-09T14:49:54.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.7 2023-03-09T14:49:54.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.0 2023-03-09T14:49:54.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.3 2023-03-09T14:49:54.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.6 2023-03-09T14:49:54.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.9 2023-03-09T14:49:54.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.8 2023-03-09T14:49:54.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.4 2023-03-09T14:49:54.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.0 2023-03-09T14:49:54.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.3 2023-03-09T14:49:54.980 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.6 2023-03-09T14:49:54.980 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.9 2023-03-09T14:49:54.980 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.5 2023-03-09T14:49:54.980 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.7 2023-03-09T14:49:54.980 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.6 2023-03-09T14:49:54.980 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.8 2023-03-09T14:49:54.980 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.4 2023-03-09T14:49:54.981 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.5 2023-03-09T14:49:54.981 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.7 2023-03-09T14:49:54.981 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.10 2023-03-09T14:49:54.981 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.10 2023-03-09T14:49:54.981 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.3 2023-03-09T14:49:54.982 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.3 2023-03-09T14:49:54.982 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.11 2023-03-09T14:49:54.982 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.8 2023-03-09T14:49:54.982 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.0 2023-03-09T14:49:54.982 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.6 2023-03-09T14:49:54.982 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.9 2023-03-09T14:49:54.983 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.11 2023-03-09T14:49:54.983 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.5 2023-03-09T14:49:54.983 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.4 2023-03-09T14:49:54.983 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.0 2023-03-09T14:49:54.983 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.6 2023-03-09T14:49:54.983 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.9 2023-03-09T14:49:54.983 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.8 2023-03-09T14:49:54.984 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.4 2023-03-09T14:49:54.984 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.6 2023-03-09T14:49:54.984 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.5 2023-03-09T14:49:54.984 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.7 2023-03-09T14:49:54.984 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.5 2023-03-09T14:49:54.985 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.7 2023-03-09T14:49:54.985 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.10 2023-03-09T14:49:54.985 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.7 2023-03-09T14:49:54.985 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.10 2023-03-09T14:49:54.986 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:54 smithi181 ceph-mon[94189]: osdmap e97: 12 total, 11 up, 12 in 2023-03-09T14:49:54.987 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.11 2023-03-09T14:49:54.987 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:54 smithi181 ceph-mon[94189]: osd.1 failed (root=default,host=smithi122) (connection refused reported by osd.11) 2023-03-09T14:49:54.987 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.11 2023-03-09T14:49:54.988 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.5 2023-03-09T14:49:54.988 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.7 2023-03-09T14:49:54.988 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.0 2023-03-09T14:49:54.988 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.3 2023-03-09T14:49:54.989 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.6 2023-03-09T14:49:54.989 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.9 2023-03-09T14:49:54.989 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.8 2023-03-09T14:49:54.990 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.4 2023-03-09T14:49:54.990 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.0 2023-03-09T14:49:54.990 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.3 2023-03-09T14:49:54.991 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.6 2023-03-09T14:49:54.991 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.9 2023-03-09T14:49:54.991 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.5 2023-03-09T14:49:54.991 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.7 2023-03-09T14:49:54.992 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.6 2023-03-09T14:49:54.992 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.8 2023-03-09T14:49:54.992 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.4 2023-03-09T14:49:54.993 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.5 2023-03-09T14:49:54.993 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.7 2023-03-09T14:49:54.993 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.10 2023-03-09T14:49:54.993 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.10 2023-03-09T14:49:54.994 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.3 2023-03-09T14:49:54.994 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.3 2023-03-09T14:49:54.994 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.11 2023-03-09T14:49:54.995 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.8 2023-03-09T14:49:54.995 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.0 2023-03-09T14:49:54.995 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.6 2023-03-09T14:49:54.995 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.9 2023-03-09T14:49:54.996 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.11 2023-03-09T14:49:54.996 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.5 2023-03-09T14:49:54.996 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.4 2023-03-09T14:49:54.996 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.0 2023-03-09T14:49:54.997 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.6 2023-03-09T14:49:54.997 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.9 2023-03-09T14:49:54.997 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.8 2023-03-09T14:49:54.998 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.4 2023-03-09T14:49:54.998 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.6 2023-03-09T14:49:54.998 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.5 2023-03-09T14:49:54.998 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.7 2023-03-09T14:49:54.999 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.5 2023-03-09T14:49:54.999 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.7 2023-03-09T14:49:54.999 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.10 2023-03-09T14:49:54.999 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.7 2023-03-09T14:49:55.000 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.10 2023-03-09T14:49:55.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:54 smithi038 ceph-mon[92422]: osdmap e97: 12 total, 11 up, 12 in 2023-03-09T14:49:55.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.11 2023-03-09T14:49:55.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:54 smithi038 ceph-mon[92422]: osd.1 failed (root=default,host=smithi122) (connection refused reported by osd.11) 2023-03-09T14:49:55.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.11 2023-03-09T14:49:55.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.5 2023-03-09T14:49:55.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.7 2023-03-09T14:49:55.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.0 2023-03-09T14:49:55.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.3 2023-03-09T14:49:55.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.6 2023-03-09T14:49:55.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.9 2023-03-09T14:49:55.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.8 2023-03-09T14:49:55.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.4 2023-03-09T14:49:55.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.0 2023-03-09T14:49:55.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.3 2023-03-09T14:49:55.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.6 2023-03-09T14:49:55.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.9 2023-03-09T14:49:55.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.5 2023-03-09T14:49:55.004 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.7 2023-03-09T14:49:55.004 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.6 2023-03-09T14:49:55.004 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.8 2023-03-09T14:49:55.004 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.4 2023-03-09T14:49:55.004 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.5 2023-03-09T14:49:55.004 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.7 2023-03-09T14:49:55.004 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.10 2023-03-09T14:49:55.005 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.10 2023-03-09T14:49:55.005 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.3 2023-03-09T14:49:55.005 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.3 2023-03-09T14:49:55.005 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.11 2023-03-09T14:49:55.005 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.8 2023-03-09T14:49:55.005 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.0 2023-03-09T14:49:55.005 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.6 2023-03-09T14:49:55.006 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.9 2023-03-09T14:49:55.006 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.11 2023-03-09T14:49:55.006 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.5 2023-03-09T14:49:55.006 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.4 2023-03-09T14:49:55.006 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.0 2023-03-09T14:49:55.006 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.6 2023-03-09T14:49:55.006 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.9 2023-03-09T14:49:55.007 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.8 2023-03-09T14:49:55.007 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.4 2023-03-09T14:49:55.007 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.6 2023-03-09T14:49:55.007 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.5 2023-03-09T14:49:55.007 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.7 2023-03-09T14:49:55.007 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.5 2023-03-09T14:49:55.007 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.7 2023-03-09T14:49:55.008 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.10 2023-03-09T14:49:55.008 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.7 2023-03-09T14:49:55.008 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.10 2023-03-09T14:49:55.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:55 smithi122 ceph-mon[93869]: pgmap v120: 129 pgs: 10 stale+active+clean, 119 active+clean; 15 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 50 MiB/s wr, 211 op/s 2023-03-09T14:49:55.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:55 smithi122 ceph-mon[93869]: osdmap e98: 12 total, 10 up, 12 in 2023-03-09T14:49:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:55 smithi181 ceph-mon[94189]: pgmap v120: 129 pgs: 10 stale+active+clean, 119 active+clean; 15 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 50 MiB/s wr, 211 op/s 2023-03-09T14:49:55.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:55 smithi181 ceph-mon[94189]: osdmap e98: 12 total, 10 up, 12 in 2023-03-09T14:49:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:55 smithi038 ceph-mon[92422]: pgmap v120: 129 pgs: 10 stale+active+clean, 119 active+clean; 15 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 50 MiB/s wr, 211 op/s 2023-03-09T14:49:56.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:55 smithi038 ceph-mon[92422]: osdmap e98: 12 total, 10 up, 12 in 2023-03-09T14:49:56.687 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:49:56 smithi122 systemd[1]: ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.1.service: Main process exited, code=exited, status=134/n/a 2023-03-09T14:49:56.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:56 smithi122 ceph-mon[93869]: osdmap e99: 12 total, 10 up, 12 in 2023-03-09T14:49:56.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:56 smithi181 ceph-mon[94189]: osdmap e99: 12 total, 10 up, 12 in 2023-03-09T14:49:57.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:56 smithi038 ceph-mon[92422]: osdmap e99: 12 total, 10 up, 12 in 2023-03-09T14:49:57.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:57 smithi122 ceph-mon[93869]: pgmap v123: 129 pgs: 13 active+undersized+degraded, 26 peering, 2 stale+active+clean, 13 active+undersized, 75 active+clean; 16 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 MiB/s wr, 33 op/s; 895/19587 objects degraded (4.569%) 2023-03-09T14:49:57.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:57 smithi122 ceph-mon[93869]: Health check failed: Reduced data availability: 6 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2023-03-09T14:49:57.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:57 smithi122 ceph-mon[93869]: Health check failed: Degraded data redundancy: 895/19587 objects degraded (4.569%), 13 pgs degraded (PG_DEGRADED) 2023-03-09T14:49:57.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:57 smithi181 ceph-mon[94189]: pgmap v123: 129 pgs: 13 active+undersized+degraded, 26 peering, 2 stale+active+clean, 13 active+undersized, 75 active+clean; 16 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 MiB/s wr, 33 op/s; 895/19587 objects degraded (4.569%) 2023-03-09T14:49:57.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:57 smithi181 ceph-mon[94189]: Health check failed: Reduced data availability: 6 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2023-03-09T14:49:57.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:57 smithi181 ceph-mon[94189]: Health check failed: Degraded data redundancy: 895/19587 objects degraded (4.569%), 13 pgs degraded (PG_DEGRADED) 2023-03-09T14:49:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:57 smithi038 ceph-mon[92422]: pgmap v123: 129 pgs: 13 active+undersized+degraded, 26 peering, 2 stale+active+clean, 13 active+undersized, 75 active+clean; 16 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 MiB/s wr, 33 op/s; 895/19587 objects degraded (4.569%) 2023-03-09T14:49:58.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:57 smithi038 ceph-mon[92422]: Health check failed: Reduced data availability: 6 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2023-03-09T14:49:58.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:57 smithi038 ceph-mon[92422]: Health check failed: Degraded data redundancy: 895/19587 objects degraded (4.569%), 13 pgs degraded (PG_DEGRADED) 2023-03-09T14:49:59.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:49:59 smithi122 ceph-mon[93869]: pgmap v124: 129 pgs: 66 active+clean, 1 undersized+peered, 17 active+undersized, 26 peering, 18 active+undersized+degraded, 1 undersized+degraded+peered; 16 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 11 MiB/s wr, 24 op/s; 1487/19603 objects degraded (7.586%) 2023-03-09T14:49:59.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:49:59 smithi181 ceph-mon[94189]: pgmap v124: 129 pgs: 66 active+clean, 1 undersized+peered, 17 active+undersized, 26 peering, 18 active+undersized+degraded, 1 undersized+degraded+peered; 16 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 11 MiB/s wr, 24 op/s; 1487/19603 objects degraded (7.586%) 2023-03-09T14:50:00.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:49:59 smithi038 ceph-mon[92422]: pgmap v124: 129 pgs: 66 active+clean, 1 undersized+peered, 17 active+undersized, 26 peering, 18 active+undersized+degraded, 1 undersized+degraded+peered; 16 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 11 MiB/s wr, 24 op/s; 1487/19603 objects degraded (7.586%) 2023-03-09T14:50:00.478 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:50:00 smithi122 systemd[1]: ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.1.service: Failed with result 'exit-code'. 2023-03-09T14:50:00.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:00 smithi122 ceph-mon[93869]: Health detail: HEALTH_WARN 2 osds down; Reduced data availability: 6 pgs inactive, 6 pgs peering; Degraded data redundancy: 1487/19603 objects degraded (7.586%), 19 pgs degraded 2023-03-09T14:50:00.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:00 smithi122 ceph-mon[93869]: [WRN] OSD_DOWN: 2 osds down 2023-03-09T14:50:00.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:00 smithi122 ceph-mon[93869]: osd.1 (root=default,host=smithi122) is down 2023-03-09T14:50:00.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:00 smithi122 ceph-mon[93869]: osd.2 (root=default,host=smithi181) is down 2023-03-09T14:50:00.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:00 smithi122 ceph-mon[93869]: [WRN] PG_AVAILABILITY: Reduced data availability: 6 pgs inactive, 6 pgs peering 2023-03-09T14:50:00.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:00 smithi122 ceph-mon[93869]: pg 3.14 is stuck peering for 5m, current state peering, last acting [8,4] 2023-03-09T14:50:00.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:00 smithi122 ceph-mon[93869]: pg 3.16 is stuck peering for 5m, current state peering, last acting [10,9] 2023-03-09T14:50:00.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:00 smithi122 ceph-mon[93869]: pg 3.1c is stuck peering for 5m, current state peering, last acting [10,6] 2023-03-09T14:50:00.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:00 smithi122 ceph-mon[93869]: pg 3.2a is stuck peering for 5m, current state peering, last acting [7,3] 2023-03-09T14:50:00.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:00 smithi122 ceph-mon[93869]: pg 3.30 is stuck peering for 5m, current state peering, last acting [7,0] 2023-03-09T14:50:00.980 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:00 smithi122 ceph-mon[93869]: pg 3.37 is stuck peering for 5m, current state peering, last acting [5,11] 2023-03-09T14:50:00.980 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:00 smithi122 ceph-mon[93869]: [WRN] PG_DEGRADED: Degraded data redundancy: 1487/19603 objects degraded (7.586%), 19 pgs degraded 2023-03-09T14:50:00.980 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:00 smithi122 ceph-mon[93869]: pg 2.4 is active+undersized+degraded, acting [9,0] 2023-03-09T14:50:00.980 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:00 smithi122 ceph-mon[93869]: pg 2.8 is undersized+degraded+peered, acting [0] 2023-03-09T14:50:00.980 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:00 smithi122 ceph-mon[93869]: pg 2.a is active+undersized+degraded, acting [3,8] 2023-03-09T14:50:00.980 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:00 smithi122 ceph-mon[93869]: pg 2.b is active+undersized+degraded, acting [8,7] 2023-03-09T14:50:00.980 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:00 smithi122 ceph-mon[93869]: pg 2.e is active+undersized+degraded, acting [6,3] 2023-03-09T14:50:00.981 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:00 smithi122 ceph-mon[93869]: pg 2.10 is active+undersized+degraded, acting [8,0] 2023-03-09T14:50:00.981 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:00 smithi122 ceph-mon[93869]: pg 2.13 is active+undersized+degraded, acting [10,3] 2023-03-09T14:50:00.981 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:00 smithi122 ceph-mon[93869]: pg 2.1d is active+undersized+degraded, acting [8,7] 2023-03-09T14:50:00.981 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:00 smithi122 ceph-mon[93869]: pg 2.1f is active+undersized+degraded, acting [9,7] 2023-03-09T14:50:00.981 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:00 smithi122 ceph-mon[93869]: pg 4.4 is active+undersized+degraded, acting [6,0,NONE,8] 2023-03-09T14:50:00.982 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:00 smithi122 ceph-mon[93869]: pg 4.b is active+undersized+degraded, acting [0,9,NONE,4] 2023-03-09T14:50:00.982 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:00 smithi122 ceph-mon[93869]: pg 4.d is active+undersized+degraded, acting [4,NONE,10,3] 2023-03-09T14:50:00.982 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:00 smithi122 ceph-mon[93869]: pg 4.f is active+undersized+degraded, acting [9,3,NONE,5] 2023-03-09T14:50:00.982 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:00 smithi122 ceph-mon[93869]: pg 4.10 is active+undersized+degraded, acting [9,0,NONE,4] 2023-03-09T14:50:00.982 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:00 smithi122 ceph-mon[93869]: pg 4.11 is active+undersized+degraded, acting [3,7,NONE,5] 2023-03-09T14:50:00.982 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:00 smithi122 ceph-mon[93869]: pg 4.19 is active+undersized+degraded, acting [NONE,3,8,0] 2023-03-09T14:50:00.983 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:00 smithi122 ceph-mon[93869]: pg 4.1c is active+undersized+degraded, acting [NONE,6,9,5] 2023-03-09T14:50:00.983 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:00 smithi122 ceph-mon[93869]: pg 4.1d is active+undersized+degraded, acting [NONE,5,7,11] 2023-03-09T14:50:00.983 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:00 smithi122 ceph-mon[93869]: pg 4.1e is active+undersized+degraded, acting [0,7,6,NONE] 2023-03-09T14:50:00.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:00 smithi181 ceph-mon[94189]: Health detail: HEALTH_WARN 2 osds down; Reduced data availability: 6 pgs inactive, 6 pgs peering; Degraded data redundancy: 1487/19603 objects degraded (7.586%), 19 pgs degraded 2023-03-09T14:50:00.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:00 smithi181 ceph-mon[94189]: [WRN] OSD_DOWN: 2 osds down 2023-03-09T14:50:00.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:00 smithi181 ceph-mon[94189]: osd.1 (root=default,host=smithi122) is down 2023-03-09T14:50:00.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:00 smithi181 ceph-mon[94189]: osd.2 (root=default,host=smithi181) is down 2023-03-09T14:50:00.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:00 smithi181 ceph-mon[94189]: [WRN] PG_AVAILABILITY: Reduced data availability: 6 pgs inactive, 6 pgs peering 2023-03-09T14:50:00.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:00 smithi181 ceph-mon[94189]: pg 3.14 is stuck peering for 5m, current state peering, last acting [8,4] 2023-03-09T14:50:00.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:00 smithi181 ceph-mon[94189]: pg 3.16 is stuck peering for 5m, current state peering, last acting [10,9] 2023-03-09T14:50:00.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:00 smithi181 ceph-mon[94189]: pg 3.1c is stuck peering for 5m, current state peering, last acting [10,6] 2023-03-09T14:50:00.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:00 smithi181 ceph-mon[94189]: pg 3.2a is stuck peering for 5m, current state peering, last acting [7,3] 2023-03-09T14:50:00.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:00 smithi181 ceph-mon[94189]: pg 3.30 is stuck peering for 5m, current state peering, last acting [7,0] 2023-03-09T14:50:00.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:00 smithi181 ceph-mon[94189]: pg 3.37 is stuck peering for 5m, current state peering, last acting [5,11] 2023-03-09T14:50:00.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:00 smithi181 ceph-mon[94189]: [WRN] PG_DEGRADED: Degraded data redundancy: 1487/19603 objects degraded (7.586%), 19 pgs degraded 2023-03-09T14:50:00.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:00 smithi181 ceph-mon[94189]: pg 2.4 is active+undersized+degraded, acting [9,0] 2023-03-09T14:50:00.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:00 smithi181 ceph-mon[94189]: pg 2.8 is undersized+degraded+peered, acting [0] 2023-03-09T14:50:00.986 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:00 smithi181 ceph-mon[94189]: pg 2.a is active+undersized+degraded, acting [3,8] 2023-03-09T14:50:00.986 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:00 smithi181 ceph-mon[94189]: pg 2.b is active+undersized+degraded, acting [8,7] 2023-03-09T14:50:00.986 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:00 smithi181 ceph-mon[94189]: pg 2.e is active+undersized+degraded, acting [6,3] 2023-03-09T14:50:00.986 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:00 smithi181 ceph-mon[94189]: pg 2.10 is active+undersized+degraded, acting [8,0] 2023-03-09T14:50:00.986 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:00 smithi181 ceph-mon[94189]: pg 2.13 is active+undersized+degraded, acting [10,3] 2023-03-09T14:50:00.986 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:00 smithi181 ceph-mon[94189]: pg 2.1d is active+undersized+degraded, acting [8,7] 2023-03-09T14:50:00.986 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:00 smithi181 ceph-mon[94189]: pg 2.1f is active+undersized+degraded, acting [9,7] 2023-03-09T14:50:00.986 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:00 smithi181 ceph-mon[94189]: pg 4.4 is active+undersized+degraded, acting [6,0,NONE,8] 2023-03-09T14:50:00.987 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:00 smithi181 ceph-mon[94189]: pg 4.b is active+undersized+degraded, acting [0,9,NONE,4] 2023-03-09T14:50:00.987 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:00 smithi181 ceph-mon[94189]: pg 4.d is active+undersized+degraded, acting [4,NONE,10,3] 2023-03-09T14:50:00.987 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:00 smithi181 ceph-mon[94189]: pg 4.f is active+undersized+degraded, acting [9,3,NONE,5] 2023-03-09T14:50:00.987 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:00 smithi181 ceph-mon[94189]: pg 4.10 is active+undersized+degraded, acting [9,0,NONE,4] 2023-03-09T14:50:00.987 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:00 smithi181 ceph-mon[94189]: pg 4.11 is active+undersized+degraded, acting [3,7,NONE,5] 2023-03-09T14:50:00.987 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:00 smithi181 ceph-mon[94189]: pg 4.19 is active+undersized+degraded, acting [NONE,3,8,0] 2023-03-09T14:50:00.987 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:00 smithi181 ceph-mon[94189]: pg 4.1c is active+undersized+degraded, acting [NONE,6,9,5] 2023-03-09T14:50:00.988 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:00 smithi181 ceph-mon[94189]: pg 4.1d is active+undersized+degraded, acting [NONE,5,7,11] 2023-03-09T14:50:00.988 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:00 smithi181 ceph-mon[94189]: pg 4.1e is active+undersized+degraded, acting [0,7,6,NONE] 2023-03-09T14:50:01.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:00 smithi038 ceph-mon[92422]: Health detail: HEALTH_WARN 2 osds down; Reduced data availability: 6 pgs inactive, 6 pgs peering; Degraded data redundancy: 1487/19603 objects degraded (7.586%), 19 pgs degraded 2023-03-09T14:50:01.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:00 smithi038 ceph-mon[92422]: [WRN] OSD_DOWN: 2 osds down 2023-03-09T14:50:01.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:00 smithi038 ceph-mon[92422]: osd.1 (root=default,host=smithi122) is down 2023-03-09T14:50:01.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:00 smithi038 ceph-mon[92422]: osd.2 (root=default,host=smithi181) is down 2023-03-09T14:50:01.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:00 smithi038 ceph-mon[92422]: [WRN] PG_AVAILABILITY: Reduced data availability: 6 pgs inactive, 6 pgs peering 2023-03-09T14:50:01.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:00 smithi038 ceph-mon[92422]: pg 3.14 is stuck peering for 5m, current state peering, last acting [8,4] 2023-03-09T14:50:01.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:00 smithi038 ceph-mon[92422]: pg 3.16 is stuck peering for 5m, current state peering, last acting [10,9] 2023-03-09T14:50:01.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:00 smithi038 ceph-mon[92422]: pg 3.1c is stuck peering for 5m, current state peering, last acting [10,6] 2023-03-09T14:50:01.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:00 smithi038 ceph-mon[92422]: pg 3.2a is stuck peering for 5m, current state peering, last acting [7,3] 2023-03-09T14:50:01.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:00 smithi038 ceph-mon[92422]: pg 3.30 is stuck peering for 5m, current state peering, last acting [7,0] 2023-03-09T14:50:01.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:00 smithi038 ceph-mon[92422]: pg 3.37 is stuck peering for 5m, current state peering, last acting [5,11] 2023-03-09T14:50:01.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:00 smithi038 ceph-mon[92422]: [WRN] PG_DEGRADED: Degraded data redundancy: 1487/19603 objects degraded (7.586%), 19 pgs degraded 2023-03-09T14:50:01.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:00 smithi038 ceph-mon[92422]: pg 2.4 is active+undersized+degraded, acting [9,0] 2023-03-09T14:50:01.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:00 smithi038 ceph-mon[92422]: pg 2.8 is undersized+degraded+peered, acting [0] 2023-03-09T14:50:01.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:00 smithi038 ceph-mon[92422]: pg 2.a is active+undersized+degraded, acting [3,8] 2023-03-09T14:50:01.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:00 smithi038 ceph-mon[92422]: pg 2.b is active+undersized+degraded, acting [8,7] 2023-03-09T14:50:01.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:00 smithi038 ceph-mon[92422]: pg 2.e is active+undersized+degraded, acting [6,3] 2023-03-09T14:50:01.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:00 smithi038 ceph-mon[92422]: pg 2.10 is active+undersized+degraded, acting [8,0] 2023-03-09T14:50:01.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:00 smithi038 ceph-mon[92422]: pg 2.13 is active+undersized+degraded, acting [10,3] 2023-03-09T14:50:01.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:00 smithi038 ceph-mon[92422]: pg 2.1d is active+undersized+degraded, acting [8,7] 2023-03-09T14:50:01.004 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:00 smithi038 ceph-mon[92422]: pg 2.1f is active+undersized+degraded, acting [9,7] 2023-03-09T14:50:01.004 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:00 smithi038 ceph-mon[92422]: pg 4.4 is active+undersized+degraded, acting [6,0,NONE,8] 2023-03-09T14:50:01.004 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:00 smithi038 ceph-mon[92422]: pg 4.b is active+undersized+degraded, acting [0,9,NONE,4] 2023-03-09T14:50:01.004 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:00 smithi038 ceph-mon[92422]: pg 4.d is active+undersized+degraded, acting [4,NONE,10,3] 2023-03-09T14:50:01.004 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:00 smithi038 ceph-mon[92422]: pg 4.f is active+undersized+degraded, acting [9,3,NONE,5] 2023-03-09T14:50:01.004 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:00 smithi038 ceph-mon[92422]: pg 4.10 is active+undersized+degraded, acting [9,0,NONE,4] 2023-03-09T14:50:01.004 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:00 smithi038 ceph-mon[92422]: pg 4.11 is active+undersized+degraded, acting [3,7,NONE,5] 2023-03-09T14:50:01.005 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:00 smithi038 ceph-mon[92422]: pg 4.19 is active+undersized+degraded, acting [NONE,3,8,0] 2023-03-09T14:50:01.005 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:00 smithi038 ceph-mon[92422]: pg 4.1c is active+undersized+degraded, acting [NONE,6,9,5] 2023-03-09T14:50:01.005 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:00 smithi038 ceph-mon[92422]: pg 4.1d is active+undersized+degraded, acting [NONE,5,7,11] 2023-03-09T14:50:01.005 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:00 smithi038 ceph-mon[92422]: pg 4.1e is active+undersized+degraded, acting [0,7,6,NONE] 2023-03-09T14:50:02.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:01 smithi038 ceph-mon[92422]: pgmap v125: 129 pgs: 66 active+clean, 2 undersized+peered, 30 active+undersized, 26 active+undersized+degraded, 5 undersized+degraded+peered; 16 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 MiB/s wr, 55 op/s; 2871/19623 objects degraded (14.631%) 2023-03-09T14:50:02.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:01 smithi038 ceph-mon[92422]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 6 pgs peering) 2023-03-09T14:50:02.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:01 smithi122 ceph-mon[93869]: pgmap v125: 129 pgs: 66 active+clean, 2 undersized+peered, 30 active+undersized, 26 active+undersized+degraded, 5 undersized+degraded+peered; 16 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 MiB/s wr, 55 op/s; 2871/19623 objects degraded (14.631%) 2023-03-09T14:50:02.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:01 smithi122 ceph-mon[93869]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 6 pgs peering) 2023-03-09T14:50:02.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:01 smithi181 ceph-mon[94189]: pgmap v125: 129 pgs: 66 active+clean, 2 undersized+peered, 30 active+undersized, 26 active+undersized+degraded, 5 undersized+degraded+peered; 16 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 MiB/s wr, 55 op/s; 2871/19623 objects degraded (14.631%) 2023-03-09T14:50:02.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:01 smithi181 ceph-mon[94189]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 6 pgs peering) 2023-03-09T14:50:03.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:02 smithi038 ceph-mon[92422]: Health check update: 2 osds down (OSD_DOWN) 2023-03-09T14:50:03.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:02 smithi038 ceph-mon[92422]: Health check update: Degraded data redundancy: 2871/19623 objects degraded (14.631%), 31 pgs degraded (PG_DEGRADED) 2023-03-09T14:50:03.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:02 smithi122 ceph-mon[93869]: Health check update: 2 osds down (OSD_DOWN) 2023-03-09T14:50:03.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:02 smithi122 ceph-mon[93869]: Health check update: Degraded data redundancy: 2871/19623 objects degraded (14.631%), 31 pgs degraded (PG_DEGRADED) 2023-03-09T14:50:03.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:02 smithi181 ceph-mon[94189]: Health check update: 2 osds down (OSD_DOWN) 2023-03-09T14:50:03.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:02 smithi181 ceph-mon[94189]: Health check update: Degraded data redundancy: 2871/19623 objects degraded (14.631%), 31 pgs degraded (PG_DEGRADED) 2023-03-09T14:50:03.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:03 smithi181 ceph-mon[94189]: pgmap v126: 129 pgs: 66 active+clean, 2 undersized+peered, 30 active+undersized, 26 active+undersized+degraded, 5 undersized+degraded+peered; 16 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 28 MiB/s wr, 55 op/s; 2891/19751 objects degraded (14.637%) 2023-03-09T14:50:03.985 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:50:03 smithi181 systemd[1]: ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.2.service: Service RestartSec=10s expired, scheduling restart. 2023-03-09T14:50:03.986 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:50:03 smithi181 systemd[1]: ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.2.service: Scheduled restart job, restart counter is at 1. 2023-03-09T14:50:03.986 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:50:03 smithi181 systemd[1]: Stopped Ceph osd.2 for c6733fe2-be87-11ed-9af2-001a4aab830c. 2023-03-09T14:50:03.986 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:50:03 smithi181 systemd[1]: Starting Ceph osd.2 for c6733fe2-be87-11ed-9af2-001a4aab830c... 2023-03-09T14:50:04.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:03 smithi038 ceph-mon[92422]: pgmap v126: 129 pgs: 66 active+clean, 2 undersized+peered, 30 active+undersized, 26 active+undersized+degraded, 5 undersized+degraded+peered; 16 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 28 MiB/s wr, 55 op/s; 2891/19751 objects degraded (14.637%) 2023-03-09T14:50:04.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:03 smithi122 ceph-mon[93869]: pgmap v126: 129 pgs: 66 active+clean, 2 undersized+peered, 30 active+undersized, 26 active+undersized+degraded, 5 undersized+degraded+peered; 16 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 28 MiB/s wr, 55 op/s; 2891/19751 objects degraded (14.637%) 2023-03-09T14:50:05.231 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:50:04 smithi181 bash[124179]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-03-09T14:50:05.232 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:50:04 smithi181 bash[124179]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2023-03-09T14:50:05.232 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:50:04 smithi181 bash[124179]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2023-03-09T14:50:05.232 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:50:04 smithi181 bash[124179]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2023-03-09T14:50:05.232 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:50:04 smithi181 bash[124179]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-2/block 2023-03-09T14:50:05.232 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:50:04 smithi181 bash[124179]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-03-09T14:50:05.232 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:50:04 smithi181 bash[124179]: --> ceph-volume raw activate successful for osd ID: 2 2023-03-09T14:50:05.969 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:05 smithi181 ceph-mon[94189]: pgmap v127: 129 pgs: 66 active+clean, 2 undersized+peered, 30 active+undersized, 26 active+undersized+degraded, 5 undersized+degraded+peered; 16 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 MiB/s wr, 44 op/s; 2891/19751 objects degraded (14.637%) 2023-03-09T14:50:05.969 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:50:05 smithi181 podman[124441]: 2023-03-09T14:50:05.969 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:50:05 smithi181 bash[124441]: 7faaa6cfa1fc9875270aaa23410458166abf246cdf9062e18f3f37630abd1589 2023-03-09T14:50:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:05 smithi038 ceph-mon[92422]: pgmap v127: 129 pgs: 66 active+clean, 2 undersized+peered, 30 active+undersized, 26 active+undersized+degraded, 5 undersized+degraded+peered; 16 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 MiB/s wr, 44 op/s; 2891/19751 objects degraded (14.637%) 2023-03-09T14:50:06.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:05 smithi122 ceph-mon[93869]: pgmap v127: 129 pgs: 66 active+clean, 2 undersized+peered, 30 active+undersized, 26 active+undersized+degraded, 5 undersized+degraded+peered; 16 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 MiB/s wr, 44 op/s; 2891/19751 objects degraded (14.637%) 2023-03-09T14:50:06.231 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:50:05 smithi181 systemd[1]: Started Ceph osd.2 for c6733fe2-be87-11ed-9af2-001a4aab830c. 2023-03-09T14:50:06.731 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:50:06 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[124457]: 2023-03-09T14:50:06.394+0000 7f3c223ed640 -1 Falling back to public interface 2023-03-09T14:50:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:07 smithi038 ceph-mon[92422]: pgmap v128: 129 pgs: 66 active+clean, 2 undersized+peered, 30 active+undersized, 26 active+undersized+degraded, 5 undersized+degraded+peered; 16 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 42 MiB/s wr, 60 op/s; 2917/20027 objects degraded (14.565%) 2023-03-09T14:50:08.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:07 smithi038 ceph-mon[92422]: Health check update: Degraded data redundancy: 2917/20027 objects degraded (14.565%), 31 pgs degraded (PG_DEGRADED) 2023-03-09T14:50:08.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:07 smithi122 ceph-mon[93869]: pgmap v128: 129 pgs: 66 active+clean, 2 undersized+peered, 30 active+undersized, 26 active+undersized+degraded, 5 undersized+degraded+peered; 16 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 42 MiB/s wr, 60 op/s; 2917/20027 objects degraded (14.565%) 2023-03-09T14:50:08.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:07 smithi122 ceph-mon[93869]: Health check update: Degraded data redundancy: 2917/20027 objects degraded (14.565%), 31 pgs degraded (PG_DEGRADED) 2023-03-09T14:50:08.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:07 smithi181 ceph-mon[94189]: pgmap v128: 129 pgs: 66 active+clean, 2 undersized+peered, 30 active+undersized, 26 active+undersized+degraded, 5 undersized+degraded+peered; 16 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 42 MiB/s wr, 60 op/s; 2917/20027 objects degraded (14.565%) 2023-03-09T14:50:08.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:07 smithi181 ceph-mon[94189]: Health check update: Degraded data redundancy: 2917/20027 objects degraded (14.565%), 31 pgs degraded (PG_DEGRADED) 2023-03-09T14:50:10.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:09 smithi038 ceph-mon[92422]: pgmap v129: 129 pgs: 66 active+clean, 2 undersized+peered, 30 active+undersized, 26 active+undersized+degraded, 5 undersized+degraded+peered; 16 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 47 MiB/s wr, 52 op/s; 2935/20155 objects degraded (14.562%) 2023-03-09T14:50:10.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:09 smithi122 ceph-mon[93869]: pgmap v129: 129 pgs: 66 active+clean, 2 undersized+peered, 30 active+undersized, 26 active+undersized+degraded, 5 undersized+degraded+peered; 16 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 47 MiB/s wr, 52 op/s; 2935/20155 objects degraded (14.562%) 2023-03-09T14:50:10.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:09 smithi181 ceph-mon[94189]: pgmap v129: 129 pgs: 66 active+clean, 2 undersized+peered, 30 active+undersized, 26 active+undersized+degraded, 5 undersized+degraded+peered; 16 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 47 MiB/s wr, 52 op/s; 2935/20155 objects degraded (14.562%) 2023-03-09T14:50:10.728 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:50:10 smithi122 systemd[1]: ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.1.service: Service RestartSec=10s expired, scheduling restart. 2023-03-09T14:50:10.728 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:50:10 smithi122 systemd[1]: ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.1.service: Scheduled restart job, restart counter is at 1. 2023-03-09T14:50:10.728 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:50:10 smithi122 systemd[1]: Stopped Ceph osd.1 for c6733fe2-be87-11ed-9af2-001a4aab830c. 2023-03-09T14:50:10.728 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:50:10 smithi122 systemd[1]: Starting Ceph osd.1 for c6733fe2-be87-11ed-9af2-001a4aab830c... 2023-03-09T14:50:12.218 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-03-09T14:50:12.219 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:11 smithi122 ceph-mon[93869]: pgmap v130: 129 pgs: 66 active+clean, 2 undersized+peered, 30 active+undersized, 26 active+undersized+degraded, 5 undersized+degraded+peered; 16 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 70 MiB/s wr, 60 op/s; 2968/20455 objects degraded (14.510%) 2023-03-09T14:50:12.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:11 smithi181 ceph-mon[94189]: pgmap v130: 129 pgs: 66 active+clean, 2 undersized+peered, 30 active+undersized, 26 active+undersized+degraded, 5 undersized+degraded+peered; 16 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 70 MiB/s wr, 60 op/s; 2968/20455 objects degraded (14.510%) 2023-03-09T14:50:12.232 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:50:11 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[124457]: 2023-03-09T14:50:11.917+0000 7f3c223ed640 -1 osd.2 95 log_to_monitors true 2023-03-09T14:50:12.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:11 smithi038 ceph-mon[92422]: pgmap v130: 129 pgs: 66 active+clean, 2 undersized+peered, 30 active+undersized, 26 active+undersized+degraded, 5 undersized+degraded+peered; 16 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 70 MiB/s wr, 60 op/s; 2968/20455 objects degraded (14.510%) 2023-03-09T14:50:12.364 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.361+0000 7fdfd3a74700 1 -- 172.21.15.38:0/1726290543 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdfcc0ff9a0 msgr2=0x7fdfcc103960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:50:12.364 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.361+0000 7fdfd3a74700 1 --2- 172.21.15.38:0/1726290543 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdfcc0ff9a0 0x7fdfcc103960 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fdfc00099c0 tx=0x7fdfc0009cd0 comp rx=0 tx=0).stop 2023-03-09T14:50:12.364 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.361+0000 7fdfd3a74700 1 -- 172.21.15.38:0/1726290543 shutdown_connections 2023-03-09T14:50:12.364 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.361+0000 7fdfd3a74700 1 --2- 172.21.15.38:0/1726290543 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfcc104040 0x7fdfcc10bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:50:12.365 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.361+0000 7fdfd3a74700 1 --2- 172.21.15.38:0/1726290543 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdfcc0ff000 0x7fdfcc0ff3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:50:12.365 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.361+0000 7fdfd3a74700 1 --2- 172.21.15.38:0/1726290543 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdfcc0ff9a0 0x7fdfcc103960 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:50:12.365 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.361+0000 7fdfd3a74700 1 -- 172.21.15.38:0/1726290543 >> 172.21.15.38:0/1726290543 conn(0x7fdfcc075c60 msgr2=0x7fdfcc076060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:50:12.365 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.361+0000 7fdfd3a74700 1 -- 172.21.15.38:0/1726290543 shutdown_connections 2023-03-09T14:50:12.365 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.361+0000 7fdfd3a74700 1 -- 172.21.15.38:0/1726290543 wait complete. 2023-03-09T14:50:12.365 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.362+0000 7fdfd3a74700 1 Processor -- start 2023-03-09T14:50:12.365 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.362+0000 7fdfd3a74700 1 -- start start 2023-03-09T14:50:12.367 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.362+0000 7fdfd3a74700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfcc0ff000 0x7fdfcc196e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:50:12.367 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.362+0000 7fdfd3a74700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdfcc0ff9a0 0x7fdfcc1973c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:50:12.367 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.362+0000 7fdfd3a74700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdfcc104040 0x7fdfcc1a1f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:50:12.367 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.362+0000 7fdfd3a74700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfcc197d00 con 0x7fdfcc0ff9a0 2023-03-09T14:50:12.368 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.362+0000 7fdfd3a74700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfcc1a2440 con 0x7fdfcc104040 2023-03-09T14:50:12.368 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.362+0000 7fdfd3a74700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfcc1a2600 con 0x7fdfcc0ff000 2023-03-09T14:50:12.368 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.362+0000 7fdfd100f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdfcc0ff9a0 0x7fdfcc1973c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:50:12.368 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.362+0000 7fdfd1810700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfcc0ff000 0x7fdfcc196e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:50:12.368 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.362+0000 7fdfd100f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdfcc0ff9a0 0x7fdfcc1973c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:40092/0 (socket says 172.21.15.38:40092) 2023-03-09T14:50:12.368 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.362+0000 7fdfd100f700 1 -- 172.21.15.38:0/2930161160 learned_addr learned my addr 172.21.15.38:0/2930161160 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:50:12.368 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.362+0000 7fdfd2011700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdfcc104040 0x7fdfcc1a1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:50:12.368 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.363+0000 7fdfd100f700 1 -- 172.21.15.38:0/2930161160 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfcc0ff000 msgr2=0x7fdfcc196e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:50:12.369 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.363+0000 7fdfd100f700 1 --2- 172.21.15.38:0/2930161160 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfcc0ff000 0x7fdfcc196e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:50:12.369 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.363+0000 7fdfd100f700 1 -- 172.21.15.38:0/2930161160 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdfcc104040 msgr2=0x7fdfcc1a1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:50:12.369 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.363+0000 7fdfd100f700 1 --2- 172.21.15.38:0/2930161160 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdfcc104040 0x7fdfcc1a1f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:50:12.369 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.363+0000 7fdfd100f700 1 -- 172.21.15.38:0/2930161160 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfc8009520 con 0x7fdfcc0ff9a0 2023-03-09T14:50:12.369 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.363+0000 7fdfd2011700 1 --2- 172.21.15.38:0/2930161160 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdfcc104040 0x7fdfcc1a1f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:50:12.369 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.363+0000 7fdfd1810700 1 --2- 172.21.15.38:0/2930161160 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfcc0ff000 0x7fdfcc196e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:50:12.369 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.363+0000 7fdfd100f700 1 --2- 172.21.15.38:0/2930161160 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdfcc0ff9a0 0x7fdfcc1973c0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fdfc00098e0 tx=0x7fdfc000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:50:12.370 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.363+0000 7fdfbeffd700 1 -- 172.21.15.38:0/2930161160 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfc001d070 con 0x7fdfcc0ff9a0 2023-03-09T14:50:12.370 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.363+0000 7fdfbeffd700 1 -- 172.21.15.38:0/2930161160 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdfc0022470 con 0x7fdfcc0ff9a0 2023-03-09T14:50:12.370 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.363+0000 7fdfd3a74700 1 -- 172.21.15.38:0/2930161160 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfc00095f0 con 0x7fdfcc0ff9a0 2023-03-09T14:50:12.370 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.363+0000 7fdfbeffd700 1 -- 172.21.15.38:0/2930161160 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfc0017650 con 0x7fdfcc0ff9a0 2023-03-09T14:50:12.370 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.363+0000 7fdfd3a74700 1 -- 172.21.15.38:0/2930161160 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfcc10edc0 con 0x7fdfcc0ff9a0 2023-03-09T14:50:12.374 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.366+0000 7fdfbeffd700 1 -- 172.21.15.38:0/2930161160 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 25) v1 ==== 133295+0+0 (secure 0 0 0) 0x7fdfc000c040 con 0x7fdfcc0ff9a0 2023-03-09T14:50:12.374 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.366+0000 7fdfbeffd700 1 --2- 172.21.15.38:0/2930161160 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fdfb809f4a0 0x7fdfb80a1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:50:12.374 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.367+0000 7fdfd1810700 1 --2- 172.21.15.38:0/2930161160 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fdfb809f4a0 0x7fdfb80a1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:50:12.374 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.367+0000 7fdfbeffd700 1 -- 172.21.15.38:0/2930161160 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 9489+0+0 (secure 0 0 0) 0x7fdfc00caef0 con 0x7fdfcc0ff9a0 2023-03-09T14:50:12.375 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.367+0000 7fdfd1810700 1 --2- 172.21.15.38:0/2930161160 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fdfb809f4a0 0x7fdfb80a1950 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fdfc8009fd0 tx=0x7fdfc8009260 comp rx=0 tx=0).ready entity=mgr.15558 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:50:12.375 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.367+0000 7fdfdfbfdb80 1 Processor -- start 2023-03-09T14:50:12.375 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.367+0000 7fdfdfbfdb80 1 -- start start 2023-03-09T14:50:12.375 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.367+0000 7fdfdfbfdb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5627ef20c830 0x5627ef20cc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:50:12.375 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.367+0000 7fdfdfbfdb80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5627ef20d140 0x5627ef206aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:50:12.375 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.367+0000 7fdfdfbfdb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5627ef206fe0 0x5627ef207410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:50:12.375 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.367+0000 7fdfdfbfdb80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627ef0f0da0 con 0x5627ef20d140 2023-03-09T14:50:12.375 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.367+0000 7fdfdfbfdb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627ef15f780 con 0x5627ef206fe0 2023-03-09T14:50:12.376 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.367+0000 7fdfdfbfdb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627ef158cb0 con 0x5627ef20c830 2023-03-09T14:50:12.376 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.368+0000 7fdfd100f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5627ef20d140 0x5627ef206aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:50:12.376 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.368+0000 7fdfd2011700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5627ef206fe0 0x5627ef207410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:50:12.376 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.368+0000 7fdfd100f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5627ef20d140 0x5627ef206aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:40100/0 (socket says 172.21.15.38:40100) 2023-03-09T14:50:12.376 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.368+0000 7fdfd1810700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5627ef20c830 0x5627ef20cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:50:12.376 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.368+0000 7fdfd100f700 1 -- 172.21.15.38:0/3098868981 learned_addr learned my addr 172.21.15.38:0/3098868981 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:50:12.376 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.368+0000 7fdfd2011700 1 -- 172.21.15.38:0/3098868981 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5627ef20c830 msgr2=0x5627ef20cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:50:12.377 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.368+0000 7fdfd2011700 1 --2- 172.21.15.38:0/3098868981 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5627ef20c830 0x5627ef20cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:50:12.377 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.368+0000 7fdfd2011700 1 -- 172.21.15.38:0/3098868981 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5627ef20d140 msgr2=0x5627ef206aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:50:12.377 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.368+0000 7fdfd2011700 1 --2- 172.21.15.38:0/3098868981 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5627ef20d140 0x5627ef206aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:50:12.377 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.368+0000 7fdfd2011700 1 -- 172.21.15.38:0/3098868981 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5627ef11f190 con 0x5627ef206fe0 2023-03-09T14:50:12.377 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.368+0000 7fdfd100f700 1 --2- 172.21.15.38:0/3098868981 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5627ef20d140 0x5627ef206aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:50:12.377 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.368+0000 7fdfd2011700 1 --2- 172.21.15.38:0/3098868981 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5627ef206fe0 0x5627ef207410 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fdfc4003db0 tx=0x7fdfc4009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba497170ee4a69f0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:50:12.377 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.368+0000 7fdfb7fff700 1 -- 172.21.15.38:0/3098868981 <== mon.2 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfc4004bb0 con 0x5627ef206fe0 2023-03-09T14:50:12.378 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.368+0000 7fdfb7fff700 1 -- 172.21.15.38:0/3098868981 <== mon.2 v2:172.21.15.122:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdfc400f040 con 0x5627ef206fe0 2023-03-09T14:50:12.378 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.368+0000 7fdfb7fff700 1 -- 172.21.15.38:0/3098868981 <== mon.2 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfc4007cb0 con 0x5627ef206fe0 2023-03-09T14:50:12.378 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.369+0000 7fdfdfbfdb80 1 -- 172.21.15.38:0/3098868981 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5627ef206fe0 msgr2=0x5627ef207410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:50:12.378 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.369+0000 7fdfdfbfdb80 1 --2- 172.21.15.38:0/3098868981 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5627ef206fe0 0x5627ef207410 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fdfc4003db0 tx=0x7fdfc4009920 comp rx=0 tx=0).stop 2023-03-09T14:50:12.378 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.369+0000 7fdfdfbfdb80 1 -- 172.21.15.38:0/3098868981 shutdown_connections 2023-03-09T14:50:12.378 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.369+0000 7fdfdfbfdb80 1 --2- 172.21.15.38:0/3098868981 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5627ef206fe0 0x5627ef207410 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:50:12.378 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.369+0000 7fdfdfbfdb80 1 --2- 172.21.15.38:0/3098868981 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5627ef20c830 0x5627ef20cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:50:12.378 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.369+0000 7fdfdfbfdb80 1 --2- 172.21.15.38:0/3098868981 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5627ef20d140 0x5627ef206aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:50:12.379 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.369+0000 7fdfdfbfdb80 1 -- 172.21.15.38:0/3098868981 >> 172.21.15.38:0/3098868981 conn(0x5627ef201a20 msgr2=0x5627ef201df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:50:12.379 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.369+0000 7fdfdfbfdb80 1 -- 172.21.15.38:0/3098868981 shutdown_connections 2023-03-09T14:50:12.379 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.369+0000 7fdfdfbfdb80 1 -- 172.21.15.38:0/3098868981 wait complete. 2023-03-09T14:50:12.379 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.369+0000 7fdfdfbfdb80 1 -- start start 2023-03-09T14:50:12.379 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.370+0000 7fdfdfbfdb80 1 Processor -- start 2023-03-09T14:50:12.379 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.370+0000 7fdfdfbfdb80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5627ef206fe0 0x5627ef213170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:50:12.379 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.370+0000 7fdfdfbfdb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5627ef20c830 0x5627ef2103f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:50:12.379 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.370+0000 7fdfdfbfdb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5627ef20d140 0x5627ef210a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:50:12.380 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.370+0000 7fdfdfbfdb80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627eefdc920 con 0x5627ef206fe0 2023-03-09T14:50:12.380 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.370+0000 7fdfdfbfdb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627ef2111c0 con 0x5627ef20d140 2023-03-09T14:50:12.380 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.370+0000 7fdfdfbfdb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627ef21cc40 con 0x5627ef20c830 2023-03-09T14:50:12.380 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.370+0000 7fdfdfbfdb80 10 client.0 _ll_register_callbacks cb 0x5627ef0fe520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:50:12.380 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.370+0000 7fdfd1810700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5627ef206fe0 0x5627ef213170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:50:12.380 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.370+0000 7fdfd1810700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5627ef206fe0 0x5627ef213170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:40104/0 (socket says 172.21.15.38:40104) 2023-03-09T14:50:12.380 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.370+0000 7fdfd1810700 1 -- 172.21.15.38:0/3305969577 learned_addr learned my addr 172.21.15.38:0/3305969577 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:50:12.381 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.370+0000 7fdfd2011700 1 --2- 172.21.15.38:0/3305969577 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5627ef20d140 0x5627ef210a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:50:12.381 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.370+0000 7fdfd100f700 1 --2- 172.21.15.38:0/3305969577 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5627ef20c830 0x5627ef2103f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:50:12.381 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.370+0000 7fdfd1810700 1 -- 172.21.15.38:0/3305969577 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5627ef20c830 msgr2=0x5627ef2103f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:50:12.381 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.370+0000 7fdfd1810700 1 --2- 172.21.15.38:0/3305969577 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5627ef20c830 0x5627ef2103f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:50:12.381 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.370+0000 7fdfd1810700 1 -- 172.21.15.38:0/3305969577 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5627ef20d140 msgr2=0x5627ef210a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:50:12.381 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.370+0000 7fdfd1810700 1 --2- 172.21.15.38:0/3305969577 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5627ef20d140 0x5627ef210a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:50:12.381 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.370+0000 7fdfd1810700 1 -- 172.21.15.38:0/3305969577 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfcc10edc0 con 0x5627ef206fe0 2023-03-09T14:50:12.381 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.371+0000 7fdfd1810700 1 --2- 172.21.15.38:0/3305969577 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5627ef206fe0 0x5627ef213170 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fdfc800ebd0 tx=0x7fdfc8013040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d3d0f731290aca9 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:50:12.382 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.371+0000 7fdfb57fa700 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2023-03-09T14:50:12.382 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.371+0000 7fdfb57fa700 1 -- 172.21.15.38:0/3305969577 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfc8021040 con 0x5627ef206fe0 2023-03-09T14:50:12.382 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.371+0000 7fdfb57fa700 1 -- 172.21.15.38:0/3305969577 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 9489+0+0 (secure 0 0 0) 0x7fdfc801cab0 con 0x5627ef206fe0 2023-03-09T14:50:12.382 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.371+0000 7fdfdfbfdb80 1 -- 172.21.15.38:0/3305969577 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5627ef126190 con 0x5627ef206fe0 2023-03-09T14:50:12.382 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.371+0000 7fdfb57fa700 1 -- 172.21.15.38:0/3305969577 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdfc801ccb0 con 0x5627ef206fe0 2023-03-09T14:50:12.382 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.371+0000 7fdfdfbfdb80 10 client.15972 fetch_fsmap learned FSMap version 27 2023-03-09T14:50:12.382 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.371+0000 7fdfdfbfdb80 1 -- 172.21.15.38:0/3305969577 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x7fdfc4003a60 con 0x5627ef206fe0 2023-03-09T14:50:12.382 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.371+0000 7fdfb57fa700 1 -- 172.21.15.38:0/3305969577 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5627ef126190 con 0x5627ef206fe0 2023-03-09T14:50:12.383 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.371+0000 7fdfb57fa700 1 -- 172.21.15.38:0/3305969577 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 27) v1 ==== 7486+0+0 (secure 0 0 0) 0x7fdfc801a390 con 0x5627ef206fe0 2023-03-09T14:50:12.383 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.371+0000 7fdfdfbfdb80 10 client.15972 fetch_fsmap finished waiting for FSMap version 27 2023-03-09T14:50:12.383 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.371+0000 7fdfdfbfdb80 10 client.15972 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T14:50:12.383 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.371+0000 7fdfdfbfdb80 20 client.15972 populate_metadata read hostname 'smithi038' 2023-03-09T14:50:12.383 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.371+0000 7fdfdfbfdb80 1 --2- 172.21.15.38:0/3305969577 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5627ef220dc0 0x5627ef2231d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:50:12.383 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.371+0000 7fdfdfbfdb80 4 client.15972 mds_command: new command op to 24530 tid=0[{"prefix": "get_command_descriptions"}] 2023-03-09T14:50:12.383 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.371+0000 7fdfdfbfdb80 1 -- 172.21.15.38:0/3305969577 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5627eefdcf60 con 0x5627ef220dc0 2023-03-09T14:50:12.384 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.372+0000 7fdfd100f700 1 --2- 172.21.15.38:0/3305969577 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5627ef220dc0 0x5627ef2231d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:50:12.385 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.372+0000 7fdfb57fa700 10 client.15972 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:50:12.385 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.372+0000 7fdfd100f700 1 --2- 172.21.15.38:0/3305969577 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5627ef220dc0 0x5627ef2231d0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6617efd8c2a8cf0c server_cookie=d89aaa5541e26be3 in_seq=0 out_seq=0 2023-03-09T14:50:12.385 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.373+0000 7fdfb57fa700 1 -- 172.21.15.38:0/3305969577 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18278 (crc 0 0 0) 0x5627eefdcf60 con 0x5627ef220dc0 2023-03-09T14:50:12.385 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.373+0000 7fdfb57fa700 10 client.15972 handle_command_reply: tid=0 2023-03-09T14:50:12.385 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.373+0000 7fdfdfbfdb80 1 client.15972 shutdown 2023-03-09T14:50:12.386 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.373+0000 7fdfdfbfdb80 1 -- 172.21.15.38:0/3305969577 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5627ef206fe0 msgr2=0x5627ef213170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:50:12.386 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.373+0000 7fdfdfbfdb80 1 --2- 172.21.15.38:0/3305969577 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5627ef206fe0 0x5627ef213170 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fdfc800ebd0 tx=0x7fdfc8013040 comp rx=0 tx=0).stop 2023-03-09T14:50:12.386 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.373+0000 7fdfdfbfdb80 1 -- 172.21.15.38:0/3305969577 shutdown_connections 2023-03-09T14:50:12.386 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.373+0000 7fdfdfbfdb80 1 --2- 172.21.15.38:0/3305969577 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5627ef220dc0 0x5627ef2231d0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:50:12.386 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.373+0000 7fdfdfbfdb80 1 --2- 172.21.15.38:0/3305969577 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5627ef206fe0 0x5627ef213170 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:50:12.386 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.373+0000 7fdfb57fa700 0 client.15972 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:50:12.386 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.373+0000 7fdfdfbfdb80 1 --2- 172.21.15.38:0/3305969577 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5627ef20d140 0x5627ef210a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:50:12.386 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.373+0000 7fdfdfbfdb80 1 --2- 172.21.15.38:0/3305969577 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5627ef20c830 0x5627ef2103f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:50:12.387 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.373+0000 7fdfdfbfdb80 1 -- 172.21.15.38:0/3305969577 >> 172.21.15.38:0/3305969577 conn(0x5627ef201a20 msgr2=0x5627ef200c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:50:12.387 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.373+0000 7fdfdfbfdb80 1 -- 172.21.15.38:0/3305969577 shutdown_connections 2023-03-09T14:50:12.387 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.373+0000 7fdfdfbfdb80 1 -- 172.21.15.38:0/3305969577 wait complete. 2023-03-09T14:50:12.387 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.374+0000 7fdfdfbfdb80 20 client.15972 trim_cache size 0 max 16384 2023-03-09T14:50:12.394 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.390+0000 7fdfdfbfdb80 1 Processor -- start 2023-03-09T14:50:12.394 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.391+0000 7fdfdfbfdb80 1 -- start start 2023-03-09T14:50:12.394 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.391+0000 7fdfdfbfdb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5627ef206fe0 0x5627ef212d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:50:12.394 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.391+0000 7fdfdfbfdb80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5627ef20c830 0x5627ef200c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:50:12.394 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.391+0000 7fdfdfbfdb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5627ef20d140 0x5627ef20a530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:50:12.395 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.391+0000 7fdfdfbfdb80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627ef207cc0 con 0x5627ef20c830 2023-03-09T14:50:12.395 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.391+0000 7fdfdfbfdb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627ef207a30 con 0x5627ef206fe0 2023-03-09T14:50:12.395 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.391+0000 7fdfdfbfdb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627ef21ca30 con 0x5627ef20d140 2023-03-09T14:50:12.395 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.391+0000 7fdfd100f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5627ef20c830 0x5627ef200c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:50:12.395 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.391+0000 7fdfd1810700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5627ef206fe0 0x5627ef212d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:50:12.395 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.391+0000 7fdfd1810700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5627ef206fe0 0x5627ef212d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.38:49302/0 (socket says 172.21.15.38:49302) 2023-03-09T14:50:12.395 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.391+0000 7fdfd100f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5627ef20c830 0x5627ef200c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:40114/0 (socket says 172.21.15.38:40114) 2023-03-09T14:50:12.395 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.391+0000 7fdfd1810700 1 -- 172.21.15.38:0/2328740223 learned_addr learned my addr 172.21.15.38:0/2328740223 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:50:12.396 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.391+0000 7fdfd2011700 1 --2- 172.21.15.38:0/2328740223 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5627ef20d140 0x5627ef20a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:50:12.396 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.391+0000 7fdfd1810700 1 -- 172.21.15.38:0/2328740223 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5627ef20d140 msgr2=0x5627ef20a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:50:12.396 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.391+0000 7fdfd1810700 1 --2- 172.21.15.38:0/2328740223 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5627ef20d140 0x5627ef20a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:50:12.396 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.391+0000 7fdfd1810700 1 -- 172.21.15.38:0/2328740223 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5627ef20c830 msgr2=0x5627ef200c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:50:12.396 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.391+0000 7fdfd1810700 1 --2- 172.21.15.38:0/2328740223 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5627ef20c830 0x5627ef200c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:50:12.396 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.391+0000 7fdfd1810700 1 -- 172.21.15.38:0/2328740223 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5627eef6fc60 con 0x5627ef206fe0 2023-03-09T14:50:12.396 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.391+0000 7fdfd100f700 1 --2- 172.21.15.38:0/2328740223 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5627ef20c830 0x5627ef200c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:50:12.397 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.391+0000 7fdfd1810700 1 --2- 172.21.15.38:0/2328740223 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5627ef206fe0 0x5627ef212d10 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fdfc8012fd0 tx=0x7fdfc801a8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c02c2b6e272490f server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:50:12.397 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.392+0000 7fdfbcdaf700 1 -- 172.21.15.38:0/2328740223 <== mon.2 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfc8021040 con 0x5627ef206fe0 2023-03-09T14:50:12.397 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.392+0000 7fdfbcdaf700 1 -- 172.21.15.38:0/2328740223 <== mon.2 v2:172.21.15.122:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdfc801c6f0 con 0x5627ef206fe0 2023-03-09T14:50:12.397 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.392+0000 7fdfbcdaf700 1 -- 172.21.15.38:0/2328740223 <== mon.2 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfc801a330 con 0x5627ef206fe0 2023-03-09T14:50:12.400 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.392+0000 7fdfdfbfdb80 1 -- 172.21.15.38:0/2328740223 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5627ef206fe0 msgr2=0x5627ef212d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:50:12.400 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.392+0000 7fdfdfbfdb80 1 --2- 172.21.15.38:0/2328740223 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5627ef206fe0 0x5627ef212d10 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fdfc8012fd0 tx=0x7fdfc801a8c0 comp rx=0 tx=0).stop 2023-03-09T14:50:12.400 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.392+0000 7fdfdfbfdb80 1 -- 172.21.15.38:0/2328740223 shutdown_connections 2023-03-09T14:50:12.400 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.392+0000 7fdfdfbfdb80 1 --2- 172.21.15.38:0/2328740223 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5627ef20d140 0x5627ef20a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:50:12.400 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.392+0000 7fdfdfbfdb80 1 --2- 172.21.15.38:0/2328740223 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5627ef206fe0 0x5627ef212d10 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:50:12.401 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.392+0000 7fdfdfbfdb80 1 --2- 172.21.15.38:0/2328740223 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5627ef20c830 0x5627ef200c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:50:12.401 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.392+0000 7fdfdfbfdb80 1 -- 172.21.15.38:0/2328740223 >> 172.21.15.38:0/2328740223 conn(0x5627ef201a20 msgr2=0x5627ef21c320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:50:12.401 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.392+0000 7fdfdfbfdb80 1 -- 172.21.15.38:0/2328740223 shutdown_connections 2023-03-09T14:50:12.401 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.392+0000 7fdfdfbfdb80 1 -- 172.21.15.38:0/2328740223 wait complete. 2023-03-09T14:50:12.401 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.393+0000 7fdfdfbfdb80 1 -- start start 2023-03-09T14:50:12.401 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.393+0000 7fdfdfbfdb80 1 Processor -- start 2023-03-09T14:50:12.401 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.393+0000 7fdfdfbfdb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5627ef206fe0 0x5627ef208920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:50:12.402 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.393+0000 7fdfdfbfdb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5627ef20c830 0x5627ef208e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:50:12.402 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.393+0000 7fdfdfbfdb80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5627ef20d140 0x5627ef2093a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:50:12.402 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.393+0000 7fdfdfbfdb80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627ef274c60 con 0x5627ef20d140 2023-03-09T14:50:12.402 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.393+0000 7fdfdfbfdb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627ef274da0 con 0x5627ef20c830 2023-03-09T14:50:12.402 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.393+0000 7fdfdfbfdb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627ef274f60 con 0x5627ef206fe0 2023-03-09T14:50:12.402 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.393+0000 7fdfdfbfdb80 10 client.0 _ll_register_callbacks cb 0x5627ef0fe520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:50:12.402 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.393+0000 7fdfd2011700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5627ef20d140 0x5627ef2093a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:50:12.402 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.393+0000 7fdfd1810700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5627ef206fe0 0x5627ef208920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:50:12.403 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.393+0000 7fdfd2011700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5627ef20d140 0x5627ef2093a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:40122/0 (socket says 172.21.15.38:40122) 2023-03-09T14:50:12.403 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.393+0000 7fdfd2011700 1 -- 172.21.15.38:0/3001901440 learned_addr learned my addr 172.21.15.38:0/3001901440 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:50:12.403 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.393+0000 7fdfd100f700 1 --2- 172.21.15.38:0/3001901440 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5627ef20c830 0x5627ef208e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:50:12.403 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.394+0000 7fdfd2011700 1 -- 172.21.15.38:0/3001901440 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5627ef206fe0 msgr2=0x5627ef208920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:50:12.403 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.394+0000 7fdfd2011700 1 --2- 172.21.15.38:0/3001901440 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5627ef206fe0 0x5627ef208920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:50:12.403 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.394+0000 7fdfd2011700 1 -- 172.21.15.38:0/3001901440 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5627ef20c830 msgr2=0x5627ef208e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:50:12.403 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.394+0000 7fdfd2011700 1 --2- 172.21.15.38:0/3001901440 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5627ef20c830 0x5627ef208e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:50:12.403 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.394+0000 7fdfd2011700 1 -- 172.21.15.38:0/3001901440 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfc00095f0 con 0x5627ef20d140 2023-03-09T14:50:12.404 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.394+0000 7fdfd1810700 1 --2- 172.21.15.38:0/3001901440 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5627ef206fe0 0x5627ef208920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:50:12.404 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.394+0000 7fdfd2011700 1 --2- 172.21.15.38:0/3001901440 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5627ef20d140 0x5627ef2093a0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x5627ef20d5d0 tx=0x7fdfc4004bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40b1f9cdc647a6dc server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:50:12.404 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.394+0000 7fdfb57fa700 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2023-03-09T14:50:12.404 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.394+0000 7fdfb57fa700 1 -- 172.21.15.38:0/3001901440 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdfc4007500 con 0x5627ef20d140 2023-03-09T14:50:12.404 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.394+0000 7fdfb57fa700 1 -- 172.21.15.38:0/3001901440 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 9489+0+0 (secure 0 0 0) 0x7fdfc4007660 con 0x5627ef20d140 2023-03-09T14:50:12.404 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.394+0000 7fdfdfbfdb80 1 -- 172.21.15.38:0/3001901440 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5627ef1161b0 con 0x5627ef20d140 2023-03-09T14:50:12.404 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.394+0000 7fdfb57fa700 1 -- 172.21.15.38:0/3001901440 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdfc4007820 con 0x5627ef20d140 2023-03-09T14:50:12.405 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.394+0000 7fdfdfbfdb80 10 client.15978 fetch_fsmap learned FSMap version 27 2023-03-09T14:50:12.405 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.394+0000 7fdfdfbfdb80 1 -- 172.21.15.38:0/3001901440 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x7fdfc4003a60 con 0x5627ef20d140 2023-03-09T14:50:12.405 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.394+0000 7fdfb57fa700 1 -- 172.21.15.38:0/3001901440 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5627ef1161b0 con 0x5627ef20d140 2023-03-09T14:50:12.405 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.394+0000 7fdfb57fa700 1 -- 172.21.15.38:0/3001901440 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 27) v1 ==== 7486+0+0 (secure 0 0 0) 0x7fdfc400ec20 con 0x5627ef20d140 2023-03-09T14:50:12.405 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.394+0000 7fdfdfbfdb80 10 client.15978 fetch_fsmap finished waiting for FSMap version 27 2023-03-09T14:50:12.405 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.394+0000 7fdfdfbfdb80 10 client.15978 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T14:50:12.405 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.394+0000 7fdfdfbfdb80 20 client.15978 populate_metadata read hostname 'smithi038' 2023-03-09T14:50:12.405 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.394+0000 7fdfdfbfdb80 1 --2- 172.21.15.38:0/3001901440 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5627ef2766e0 0x5627ef278af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:50:12.406 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.394+0000 7fdfdfbfdb80 4 client.15978 mds_command: new command op to 24530 tid=0[{"prefix": "scrub status"}] 2023-03-09T14:50:12.406 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.394+0000 7fdfdfbfdb80 1 -- 172.21.15.38:0/3001901440 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5627ef1001c0 con 0x5627ef2766e0 2023-03-09T14:50:12.406 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.395+0000 7fdfd1810700 1 --2- 172.21.15.38:0/3001901440 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5627ef2766e0 0x5627ef278af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:50:12.406 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.395+0000 7fdfd1810700 1 --2- 172.21.15.38:0/3001901440 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5627ef2766e0 0x5627ef278af0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d1020cb6f71cb8c7 server_cookie=9feb86153fb4324c in_seq=0 out_seq=0 2023-03-09T14:50:12.406 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.395+0000 7fdfb57fa700 10 client.15978 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:50:12.406 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.395+0000 7fdfb57fa700 1 -- 172.21.15.38:0/3001901440 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+265 (crc 0 0 0) 0x5627ef1001c0 con 0x5627ef2766e0 2023-03-09T14:50:12.406 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.395+0000 7fdfb57fa700 10 client.15978 handle_command_reply: tid=0 2023-03-09T14:50:12.407 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.395+0000 7fdfdfbfdb80 1 client.15978 shutdown 2023-03-09T14:50:12.407 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.396+0000 7fdfdfbfdb80 1 -- 172.21.15.38:0/3001901440 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5627ef20d140 msgr2=0x5627ef2093a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:50:12.407 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.396+0000 7fdfdfbfdb80 1 --2- 172.21.15.38:0/3001901440 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5627ef20d140 0x5627ef2093a0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x5627ef20d5d0 tx=0x7fdfc4004bd0 comp rx=0 tx=0).stop 2023-03-09T14:50:12.407 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.396+0000 7fdfdfbfdb80 1 -- 172.21.15.38:0/3001901440 shutdown_connections 2023-03-09T14:50:12.407 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.396+0000 7fdfdfbfdb80 1 --2- 172.21.15.38:0/3001901440 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5627ef2766e0 0x5627ef278af0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:50:12.407 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.396+0000 7fdfdfbfdb80 1 --2- 172.21.15.38:0/3001901440 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5627ef20d140 0x5627ef2093a0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:50:12.407 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.396+0000 7fdfb57fa700 0 client.15978 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:50:12.408 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.396+0000 7fdfdfbfdb80 1 --2- 172.21.15.38:0/3001901440 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5627ef206fe0 0x5627ef208920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:50:12.408 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.396+0000 7fdfdfbfdb80 1 --2- 172.21.15.38:0/3001901440 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5627ef20c830 0x5627ef208e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:50:12.408 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.396+0000 7fdfdfbfdb80 1 -- 172.21.15.38:0/3001901440 >> 172.21.15.38:0/3001901440 conn(0x5627ef201a20 msgr2=0x5627ef200c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:50:12.408 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.396+0000 7fdfdfbfdb80 1 -- 172.21.15.38:0/3001901440 shutdown_connections 2023-03-09T14:50:12.408 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.396+0000 7fdfdfbfdb80 1 -- 172.21.15.38:0/3001901440 wait complete. 2023-03-09T14:50:12.409 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.396+0000 7fdfdfbfdb80 20 client.15978 trim_cache size 0 max 16384 2023-03-09T14:50:12.409 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2023-03-09T14:50:12.409 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "scrub active (3 inodes in the stack)", 2023-03-09T14:50:12.410 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": { 2023-03-09T14:50:12.410 INFO:teuthology.orchestra.run.smithi038.stdout: "a14659f4-d7ea-4c5a-b15b-6138af3f909f": { 2023-03-09T14:50:12.410 INFO:teuthology.orchestra.run.smithi038.stdout: "path": "/", 2023-03-09T14:50:12.410 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "a14659f4-d7ea-4c5a-b15b-6138af3f909f", 2023-03-09T14:50:12.410 INFO:teuthology.orchestra.run.smithi038.stdout: "options": "recursive,force" 2023-03-09T14:50:12.410 INFO:teuthology.orchestra.run.smithi038.stdout: } 2023-03-09T14:50:12.410 INFO:teuthology.orchestra.run.smithi038.stdout: } 2023-03-09T14:50:12.411 INFO:teuthology.orchestra.run.smithi038.stdout:} 2023-03-09T14:50:12.412 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.396+0000 7fdfd3a74700 1 -- 172.21.15.38:0/2930161160 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fdfb809f4a0 msgr2=0x7fdfb80a1950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:50:12.412 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.396+0000 7fdfd3a74700 1 --2- 172.21.15.38:0/2930161160 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fdfb809f4a0 0x7fdfb80a1950 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fdfc8009fd0 tx=0x7fdfc8009260 comp rx=0 tx=0).stop 2023-03-09T14:50:12.412 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.396+0000 7fdfd3a74700 1 -- 172.21.15.38:0/2930161160 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdfcc0ff9a0 msgr2=0x7fdfcc1973c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:50:12.412 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.396+0000 7fdfd3a74700 1 --2- 172.21.15.38:0/2930161160 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdfcc0ff9a0 0x7fdfcc1973c0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fdfc00098e0 tx=0x7fdfc000f690 comp rx=0 tx=0).stop 2023-03-09T14:50:12.412 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.396+0000 7fdfd3a74700 1 -- 172.21.15.38:0/2930161160 shutdown_connections 2023-03-09T14:50:12.413 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.396+0000 7fdfd3a74700 1 --2- 172.21.15.38:0/2930161160 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fdfb809f4a0 0x7fdfb80a1950 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:50:12.413 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.396+0000 7fdfd3a74700 1 --2- 172.21.15.38:0/2930161160 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdfcc104040 0x7fdfcc1a1f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:50:12.413 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.396+0000 7fdfd3a74700 1 --2- 172.21.15.38:0/2930161160 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdfcc0ff000 0x7fdfcc196e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:50:12.413 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.396+0000 7fdfd3a74700 1 --2- 172.21.15.38:0/2930161160 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdfcc0ff9a0 0x7fdfcc1973c0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:50:12.413 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.396+0000 7fdfd3a74700 1 -- 172.21.15.38:0/2930161160 >> 172.21.15.38:0/2930161160 conn(0x7fdfcc075c60 msgr2=0x7fdfcc0fc940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:50:12.413 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.396+0000 7fdfd3a74700 1 -- 172.21.15.38:0/2930161160 shutdown_connections 2023-03-09T14:50:12.413 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:12.396+0000 7fdfd3a74700 1 -- 172.21.15.38:0/2930161160 wait complete. 2023-03-09T14:50:12.414 INFO:tasks.cephfs.filesystem:scrub status for tag:a14659f4-d7ea-4c5a-b15b-6138af3f909f - {'path': '/', 'tag': 'a14659f4-d7ea-4c5a-b15b-6138af3f909f', 'options': 'recursive,force'} 2023-03-09T14:50:13.033 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:12 smithi122 ceph-mon[93869]: from='osd.2 [v2:172.21.15.181:6800/933290453,v1:172.21.15.181:6801/933290453]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-03-09T14:50:13.033 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:12 smithi122 ceph-mon[93869]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-03-09T14:50:13.034 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:12 smithi122 ceph-mon[93869]: Health check update: Degraded data redundancy: 2968/20455 objects degraded (14.510%), 31 pgs degraded (PG_DEGRADED) 2023-03-09T14:50:13.231 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 09 14:50:12 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-2[124457]: 2023-03-09T14:50:12.787+0000 7f3c11966700 -1 osd.2 95 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-09T14:50:13.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:12 smithi181 ceph-mon[94189]: from='osd.2 [v2:172.21.15.181:6800/933290453,v1:172.21.15.181:6801/933290453]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-03-09T14:50:13.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:12 smithi181 ceph-mon[94189]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-03-09T14:50:13.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:12 smithi181 ceph-mon[94189]: Health check update: Degraded data redundancy: 2968/20455 objects degraded (14.510%), 31 pgs degraded (PG_DEGRADED) 2023-03-09T14:50:13.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:12 smithi038 ceph-mon[92422]: from='osd.2 [v2:172.21.15.181:6800/933290453,v1:172.21.15.181:6801/933290453]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-03-09T14:50:13.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:12 smithi038 ceph-mon[92422]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-03-09T14:50:13.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:12 smithi038 ceph-mon[92422]: Health check update: Degraded data redundancy: 2968/20455 objects degraded (14.510%), 31 pgs degraded (PG_DEGRADED) 2023-03-09T14:50:13.478 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:50:13 smithi122 bash[122959]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-03-09T14:50:13.478 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:50:13 smithi122 bash[122959]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2023-03-09T14:50:13.478 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:50:13 smithi122 bash[122959]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2023-03-09T14:50:13.478 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:50:13 smithi122 bash[122959]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2023-03-09T14:50:13.478 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:50:13 smithi122 bash[122959]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-1/block 2023-03-09T14:50:13.479 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:50:13 smithi122 bash[122959]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-03-09T14:50:13.479 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:50:13 smithi122 bash[122959]: --> ceph-volume raw activate successful for osd ID: 1 2023-03-09T14:50:13.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:13 smithi122 ceph-mon[93869]: pgmap v131: 129 pgs: 66 active+clean, 2 undersized+peered, 30 active+undersized, 26 active+undersized+degraded, 5 undersized+degraded+peered; 16 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 76 MiB/s wr, 43 op/s; 2984/20559 objects degraded (14.514%) 2023-03-09T14:50:13.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:13 smithi122 ceph-mon[93869]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-03-09T14:50:13.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:13 smithi122 ceph-mon[93869]: osdmap e100: 12 total, 10 up, 12 in 2023-03-09T14:50:13.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:13 smithi122 ceph-mon[93869]: from='osd.2 [v2:172.21.15.181:6800/933290453,v1:172.21.15.181:6801/933290453]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2023-03-09T14:50:13.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:13 smithi122 ceph-mon[93869]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2023-03-09T14:50:13.979 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:50:13 smithi122 podman[123217]: 2023-03-09T14:50:13.979 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:50:13 smithi122 bash[123217]: 12bfbcccdf8ad2c5ecab4aed96e259ef46ddbf14cd9f29a8292d856b525f7ce5 2023-03-09T14:50:13.979 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:50:13 smithi122 ceph-osd[123258]: -- 172.21.15.122:0/1424354189 <== mon.2 v2:172.21.15.122:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x561dc23709c0 con 0x561dc2338c00 2023-03-09T14:50:13.979 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:50:13 smithi122 systemd[1]: Started Ceph osd.1 for c6733fe2-be87-11ed-9af2-001a4aab830c. 2023-03-09T14:50:14.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:13 smithi181 ceph-mon[94189]: pgmap v131: 129 pgs: 66 active+clean, 2 undersized+peered, 30 active+undersized, 26 active+undersized+degraded, 5 undersized+degraded+peered; 16 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 76 MiB/s wr, 43 op/s; 2984/20559 objects degraded (14.514%) 2023-03-09T14:50:14.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:13 smithi181 ceph-mon[94189]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-03-09T14:50:14.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:13 smithi181 ceph-mon[94189]: osdmap e100: 12 total, 10 up, 12 in 2023-03-09T14:50:14.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:13 smithi181 ceph-mon[94189]: from='osd.2 [v2:172.21.15.181:6800/933290453,v1:172.21.15.181:6801/933290453]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2023-03-09T14:50:14.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:13 smithi181 ceph-mon[94189]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2023-03-09T14:50:14.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:13 smithi038 ceph-mon[92422]: pgmap v131: 129 pgs: 66 active+clean, 2 undersized+peered, 30 active+undersized, 26 active+undersized+degraded, 5 undersized+degraded+peered; 16 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 76 MiB/s wr, 43 op/s; 2984/20559 objects degraded (14.514%) 2023-03-09T14:50:14.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:13 smithi038 ceph-mon[92422]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-03-09T14:50:14.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:13 smithi038 ceph-mon[92422]: osdmap e100: 12 total, 10 up, 12 in 2023-03-09T14:50:14.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:13 smithi038 ceph-mon[92422]: from='osd.2 [v2:172.21.15.181:6800/933290453,v1:172.21.15.181:6801/933290453]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2023-03-09T14:50:14.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:13 smithi038 ceph-mon[92422]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2023-03-09T14:50:14.728 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:50:14 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 2023-03-09T14:50:14.415+0000 7f48636a8640 -1 Falling back to public interface 2023-03-09T14:50:15.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:14 smithi122 ceph-mon[93869]: Health check update: 1 osds down (OSD_DOWN) 2023-03-09T14:50:15.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:14 smithi122 ceph-mon[93869]: osd.2 [v2:172.21.15.181:6800/933290453,v1:172.21.15.181:6801/933290453] boot 2023-03-09T14:50:15.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:14 smithi122 ceph-mon[93869]: osdmap e101: 12 total, 11 up, 12 in 2023-03-09T14:50:15.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:14 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-03-09T14:50:15.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:14 smithi122 ceph-mon[93869]: osdmap e102: 12 total, 11 up, 12 in 2023-03-09T14:50:15.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:14 smithi181 ceph-mon[94189]: Health check update: 1 osds down (OSD_DOWN) 2023-03-09T14:50:15.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:14 smithi181 ceph-mon[94189]: osd.2 [v2:172.21.15.181:6800/933290453,v1:172.21.15.181:6801/933290453] boot 2023-03-09T14:50:15.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:14 smithi181 ceph-mon[94189]: osdmap e101: 12 total, 11 up, 12 in 2023-03-09T14:50:15.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:14 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-03-09T14:50:15.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:14 smithi181 ceph-mon[94189]: osdmap e102: 12 total, 11 up, 12 in 2023-03-09T14:50:15.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:14 smithi038 ceph-mon[92422]: Health check update: 1 osds down (OSD_DOWN) 2023-03-09T14:50:15.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:14 smithi038 ceph-mon[92422]: osd.2 [v2:172.21.15.181:6800/933290453,v1:172.21.15.181:6801/933290453] boot 2023-03-09T14:50:15.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:14 smithi038 ceph-mon[92422]: osdmap e101: 12 total, 11 up, 12 in 2023-03-09T14:50:15.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:14 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-03-09T14:50:15.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:14 smithi038 ceph-mon[92422]: osdmap e102: 12 total, 11 up, 12 in 2023-03-09T14:50:16.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:15 smithi122 ceph-mon[93869]: pgmap v134: 129 pgs: 66 active+clean, 2 undersized+peered, 30 active+undersized, 26 active+undersized+degraded, 5 undersized+degraded+peered; 16 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 66 MiB/s wr, 23 op/s; 2984/20559 objects degraded (14.514%) 2023-03-09T14:50:16.245 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:15 smithi181 ceph-mon[94189]: pgmap v134: 129 pgs: 66 active+clean, 2 undersized+peered, 30 active+undersized, 26 active+undersized+degraded, 5 undersized+degraded+peered; 16 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 66 MiB/s wr, 23 op/s; 2984/20559 objects degraded (14.514%) 2023-03-09T14:50:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:15 smithi038 ceph-mon[92422]: pgmap v134: 129 pgs: 66 active+clean, 2 undersized+peered, 30 active+undersized, 26 active+undersized+degraded, 5 undersized+degraded+peered; 16 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 66 MiB/s wr, 23 op/s; 2984/20559 objects degraded (14.514%) 2023-03-09T14:50:18.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:17 smithi122 ceph-mon[93869]: pgmap v136: 129 pgs: 81 active+clean, 2 undersized+peered, 22 active+undersized, 22 active+undersized+degraded, 2 undersized+degraded+peered; 17 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 64 MiB/s wr, 34 op/s; 2267/20827 objects degraded (10.885%); 15 MiB/s, 3 objects/s recovering 2023-03-09T14:50:18.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:17 smithi122 ceph-mon[93869]: Health check update: Degraded data redundancy: 2267/20827 objects degraded (10.885%), 24 pgs degraded (PG_DEGRADED) 2023-03-09T14:50:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:17 smithi181 ceph-mon[94189]: pgmap v136: 129 pgs: 81 active+clean, 2 undersized+peered, 22 active+undersized, 22 active+undersized+degraded, 2 undersized+degraded+peered; 17 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 64 MiB/s wr, 34 op/s; 2267/20827 objects degraded (10.885%); 15 MiB/s, 3 objects/s recovering 2023-03-09T14:50:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:17 smithi181 ceph-mon[94189]: Health check update: Degraded data redundancy: 2267/20827 objects degraded (10.885%), 24 pgs degraded (PG_DEGRADED) 2023-03-09T14:50:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:17 smithi038 ceph-mon[92422]: pgmap v136: 129 pgs: 81 active+clean, 2 undersized+peered, 22 active+undersized, 22 active+undersized+degraded, 2 undersized+degraded+peered; 17 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 64 MiB/s wr, 34 op/s; 2267/20827 objects degraded (10.885%); 15 MiB/s, 3 objects/s recovering 2023-03-09T14:50:18.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:17 smithi038 ceph-mon[92422]: Health check update: Degraded data redundancy: 2267/20827 objects degraded (10.885%), 24 pgs degraded (PG_DEGRADED) 2023-03-09T14:50:20.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:19 smithi122 ceph-mon[93869]: pgmap v137: 129 pgs: 93 active+clean, 18 active+undersized, 18 active+undersized+degraded; 17 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 64 MiB/s wr, 49 op/s; 1672/20944 objects degraded (7.983%); 35 MiB/s, 13 keys/s, 9 objects/s recovering 2023-03-09T14:50:20.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:19 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T14:50:20.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:19 smithi181 ceph-mon[94189]: pgmap v137: 129 pgs: 93 active+clean, 18 active+undersized, 18 active+undersized+degraded; 17 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 64 MiB/s wr, 49 op/s; 1672/20944 objects degraded (7.983%); 35 MiB/s, 13 keys/s, 9 objects/s recovering 2023-03-09T14:50:20.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:19 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T14:50:20.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:19 smithi038 ceph-mon[92422]: pgmap v137: 129 pgs: 93 active+clean, 18 active+undersized, 18 active+undersized+degraded; 17 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 64 MiB/s wr, 49 op/s; 1672/20944 objects degraded (7.983%); 35 MiB/s, 13 keys/s, 9 objects/s recovering 2023-03-09T14:50:20.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:19 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T14:50:22.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:21 smithi122 ceph-mon[93869]: pgmap v138: 129 pgs: 93 active+clean, 18 active+undersized, 18 active+undersized+degraded; 17 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 69 MiB/s wr, 139 op/s; 1689/21152 objects degraded (7.985%); 27 MiB/s, 10 keys/s, 6 objects/s recovering 2023-03-09T14:50:22.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:21 smithi122 ceph-mon[93869]: from='osd.1 [v2:172.21.15.122:6800/2980663525,v1:172.21.15.122:6801/2980663525]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-03-09T14:50:22.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:21 smithi122 ceph-mon[93869]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-03-09T14:50:22.228 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:50:21 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 2023-03-09T14:50:21.752+0000 7f48636a8640 -1 osd.1 97 log_to_monitors true 2023-03-09T14:50:22.229 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:50:21 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 2023-03-09T14:50:21.849+0000 7f4851c1f700 -1 osd.1 97 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-09T14:50:22.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:21 smithi181 ceph-mon[94189]: pgmap v138: 129 pgs: 93 active+clean, 18 active+undersized, 18 active+undersized+degraded; 17 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 69 MiB/s wr, 139 op/s; 1689/21152 objects degraded (7.985%); 27 MiB/s, 10 keys/s, 6 objects/s recovering 2023-03-09T14:50:22.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:21 smithi181 ceph-mon[94189]: from='osd.1 [v2:172.21.15.122:6800/2980663525,v1:172.21.15.122:6801/2980663525]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-03-09T14:50:22.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:21 smithi181 ceph-mon[94189]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-03-09T14:50:22.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:21 smithi038 ceph-mon[92422]: pgmap v138: 129 pgs: 93 active+clean, 18 active+undersized, 18 active+undersized+degraded; 17 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 69 MiB/s wr, 139 op/s; 1689/21152 objects degraded (7.985%); 27 MiB/s, 10 keys/s, 6 objects/s recovering 2023-03-09T14:50:22.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:21 smithi038 ceph-mon[92422]: from='osd.1 [v2:172.21.15.122:6800/2980663525,v1:172.21.15.122:6801/2980663525]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-03-09T14:50:22.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:21 smithi038 ceph-mon[92422]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-03-09T14:50:23.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:22 smithi122 ceph-mon[93869]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-03-09T14:50:23.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:22 smithi122 ceph-mon[93869]: osdmap e103: 12 total, 11 up, 12 in 2023-03-09T14:50:23.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:22 smithi122 ceph-mon[93869]: from='osd.1 [v2:172.21.15.122:6800/2980663525,v1:172.21.15.122:6801/2980663525]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-03-09T14:50:23.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:22 smithi122 ceph-mon[93869]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-03-09T14:50:23.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:22 smithi122 ceph-mon[93869]: Health check update: Degraded data redundancy: 1689/21152 objects degraded (7.985%), 18 pgs degraded (PG_DEGRADED) 2023-03-09T14:50:23.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:22 smithi181 ceph-mon[94189]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-03-09T14:50:23.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:22 smithi181 ceph-mon[94189]: osdmap e103: 12 total, 11 up, 12 in 2023-03-09T14:50:23.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:22 smithi181 ceph-mon[94189]: from='osd.1 [v2:172.21.15.122:6800/2980663525,v1:172.21.15.122:6801/2980663525]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-03-09T14:50:23.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:22 smithi181 ceph-mon[94189]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-03-09T14:50:23.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:22 smithi181 ceph-mon[94189]: Health check update: Degraded data redundancy: 1689/21152 objects degraded (7.985%), 18 pgs degraded (PG_DEGRADED) 2023-03-09T14:50:23.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:22 smithi038 ceph-mon[92422]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-03-09T14:50:23.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:22 smithi038 ceph-mon[92422]: osdmap e103: 12 total, 11 up, 12 in 2023-03-09T14:50:23.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:22 smithi038 ceph-mon[92422]: from='osd.1 [v2:172.21.15.122:6800/2980663525,v1:172.21.15.122:6801/2980663525]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-03-09T14:50:23.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:22 smithi038 ceph-mon[92422]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-03-09T14:50:23.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:22 smithi038 ceph-mon[92422]: Health check update: Degraded data redundancy: 1689/21152 objects degraded (7.985%), 18 pgs degraded (PG_DEGRADED) 2023-03-09T14:50:24.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:23 smithi122 ceph-mon[93869]: pgmap v140: 129 pgs: 93 active+clean, 18 active+undersized, 18 active+undersized+degraded; 17 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 83 MiB/s wr, 189 op/s; 1707/21314 objects degraded (8.009%); 27 MiB/s, 10 keys/s, 6 objects/s recovering 2023-03-09T14:50:24.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:23 smithi122 ceph-mon[93869]: Health check cleared: OSD_DOWN (was: 1 osds down) 2023-03-09T14:50:24.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:23 smithi122 ceph-mon[93869]: osd.1 [v2:172.21.15.122:6800/2980663525,v1:172.21.15.122:6801/2980663525] boot 2023-03-09T14:50:24.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:23 smithi122 ceph-mon[93869]: osdmap e104: 12 total, 12 up, 12 in 2023-03-09T14:50:24.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:23 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-03-09T14:50:24.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:23 smithi122 ceph-mon[93869]: osdmap e105: 12 total, 12 up, 12 in 2023-03-09T14:50:24.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:23 smithi181 ceph-mon[94189]: pgmap v140: 129 pgs: 93 active+clean, 18 active+undersized, 18 active+undersized+degraded; 17 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 83 MiB/s wr, 189 op/s; 1707/21314 objects degraded (8.009%); 27 MiB/s, 10 keys/s, 6 objects/s recovering 2023-03-09T14:50:24.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:23 smithi181 ceph-mon[94189]: Health check cleared: OSD_DOWN (was: 1 osds down) 2023-03-09T14:50:24.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:23 smithi181 ceph-mon[94189]: osd.1 [v2:172.21.15.122:6800/2980663525,v1:172.21.15.122:6801/2980663525] boot 2023-03-09T14:50:24.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:23 smithi181 ceph-mon[94189]: osdmap e104: 12 total, 12 up, 12 in 2023-03-09T14:50:24.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:23 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-03-09T14:50:24.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:23 smithi181 ceph-mon[94189]: osdmap e105: 12 total, 12 up, 12 in 2023-03-09T14:50:24.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:23 smithi038 ceph-mon[92422]: pgmap v140: 129 pgs: 93 active+clean, 18 active+undersized, 18 active+undersized+degraded; 17 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 83 MiB/s wr, 189 op/s; 1707/21314 objects degraded (8.009%); 27 MiB/s, 10 keys/s, 6 objects/s recovering 2023-03-09T14:50:24.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:23 smithi038 ceph-mon[92422]: Health check cleared: OSD_DOWN (was: 1 osds down) 2023-03-09T14:50:24.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:23 smithi038 ceph-mon[92422]: osd.1 [v2:172.21.15.122:6800/2980663525,v1:172.21.15.122:6801/2980663525] boot 2023-03-09T14:50:24.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:23 smithi038 ceph-mon[92422]: osdmap e104: 12 total, 12 up, 12 in 2023-03-09T14:50:24.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:23 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-03-09T14:50:24.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:23 smithi038 ceph-mon[92422]: osdmap e105: 12 total, 12 up, 12 in 2023-03-09T14:50:25.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:50:25.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:24 smithi122 ceph-mon[93869]: osdmap e106: 12 total, 12 up, 12 in 2023-03-09T14:50:25.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:50:25.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:24 smithi181 ceph-mon[94189]: osdmap e106: 12 total, 12 up, 12 in 2023-03-09T14:50:25.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:50:25.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:24 smithi038 ceph-mon[92422]: osdmap e106: 12 total, 12 up, 12 in 2023-03-09T14:50:26.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:25 smithi122 ceph-mon[93869]: pgmap v143: 129 pgs: 93 active+clean, 18 active+undersized, 18 active+undersized+degraded; 17 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 56 MiB/s wr, 217 op/s; 1712/21382 objects degraded (8.007%) 2023-03-09T14:50:26.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:50:26.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:50:26.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:50:26.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:25 smithi181 ceph-mon[94189]: pgmap v143: 129 pgs: 93 active+clean, 18 active+undersized, 18 active+undersized+degraded; 17 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 56 MiB/s wr, 217 op/s; 1712/21382 objects degraded (8.007%) 2023-03-09T14:50:26.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:50:26.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:50:26.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:50:26.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:25 smithi038 ceph-mon[92422]: pgmap v143: 129 pgs: 93 active+clean, 18 active+undersized, 18 active+undersized+degraded; 17 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 56 MiB/s wr, 217 op/s; 1712/21382 objects degraded (8.007%) 2023-03-09T14:50:26.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:50:26.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:50:26.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:50:27.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:26 smithi122 ceph-mon[93869]: osdmap e107: 12 total, 12 up, 12 in 2023-03-09T14:50:27.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:26 smithi181 ceph-mon[94189]: osdmap e107: 12 total, 12 up, 12 in 2023-03-09T14:50:27.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:26 smithi038 ceph-mon[92422]: osdmap e107: 12 total, 12 up, 12 in 2023-03-09T14:50:28.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:27 smithi122 ceph-mon[93869]: pgmap v146: 129 pgs: 1 peering, 4 active+recovery_wait+degraded, 108 active+clean, 9 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+undersized+degraded; 17 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 86 MiB/s wr, 98 op/s; 971/21694 objects degraded (4.476%); 14 MiB/s, 5 keys/s, 4 objects/s recovering 2023-03-09T14:50:28.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:27 smithi122 ceph-mon[93869]: osdmap e108: 12 total, 12 up, 12 in 2023-03-09T14:50:28.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:27 smithi122 ceph-mon[93869]: Health check update: Degraded data redundancy: 971/21694 objects degraded (4.476%), 11 pgs degraded (PG_DEGRADED) 2023-03-09T14:50:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:27 smithi181 ceph-mon[94189]: pgmap v146: 129 pgs: 1 peering, 4 active+recovery_wait+degraded, 108 active+clean, 9 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+undersized+degraded; 17 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 86 MiB/s wr, 98 op/s; 971/21694 objects degraded (4.476%); 14 MiB/s, 5 keys/s, 4 objects/s recovering 2023-03-09T14:50:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:27 smithi181 ceph-mon[94189]: osdmap e108: 12 total, 12 up, 12 in 2023-03-09T14:50:28.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:27 smithi181 ceph-mon[94189]: Health check update: Degraded data redundancy: 971/21694 objects degraded (4.476%), 11 pgs degraded (PG_DEGRADED) 2023-03-09T14:50:28.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:27 smithi038 ceph-mon[92422]: pgmap v146: 129 pgs: 1 peering, 4 active+recovery_wait+degraded, 108 active+clean, 9 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+undersized+degraded; 17 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 86 MiB/s wr, 98 op/s; 971/21694 objects degraded (4.476%); 14 MiB/s, 5 keys/s, 4 objects/s recovering 2023-03-09T14:50:28.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:27 smithi038 ceph-mon[92422]: osdmap e108: 12 total, 12 up, 12 in 2023-03-09T14:50:28.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:27 smithi038 ceph-mon[92422]: Health check update: Degraded data redundancy: 971/21694 objects degraded (4.476%), 11 pgs degraded (PG_DEGRADED) 2023-03-09T14:50:28.675 INFO:teuthology.orchestra.run.smithi038.stderr:done. 2023-03-09T14:50:28.679 DEBUG:teuthology.orchestra.run.smithi038:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo -u postgres -- pgbench -c 100 -j 4 --progress=5 --time=900' 2023-03-09T14:50:28.795 INFO:teuthology.orchestra.run.smithi038.stderr:starting vacuum...end. 2023-03-09T14:50:30.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:29 smithi122 ceph-mon[93869]: pgmap v148: 129 pgs: 1 peering, 6 active+recovery_wait+degraded, 121 active+clean, 1 active+recovery_wait+undersized+degraded+remapped; 18 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 99 MiB/s wr, 88 op/s; 165/21862 objects degraded (0.755%); 71 MiB/s, 4 keys/s, 19 objects/s recovering 2023-03-09T14:50:30.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:29 smithi181 ceph-mon[94189]: pgmap v148: 129 pgs: 1 peering, 6 active+recovery_wait+degraded, 121 active+clean, 1 active+recovery_wait+undersized+degraded+remapped; 18 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 99 MiB/s wr, 88 op/s; 165/21862 objects degraded (0.755%); 71 MiB/s, 4 keys/s, 19 objects/s recovering 2023-03-09T14:50:30.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:29 smithi038 ceph-mon[92422]: pgmap v148: 129 pgs: 1 peering, 6 active+recovery_wait+degraded, 121 active+clean, 1 active+recovery_wait+undersized+degraded+remapped; 18 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 99 MiB/s wr, 88 op/s; 165/21862 objects degraded (0.755%); 71 MiB/s, 4 keys/s, 19 objects/s recovering 2023-03-09T14:50:32.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:31 smithi122 ceph-mon[93869]: pgmap v149: 129 pgs: 1 active+undersized+remapped, 1 peering, 4 active+recovery_wait+degraded, 123 active+clean; 18 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 158 MiB/s wr, 485 op/s; 84/22282 objects degraded (0.377%); 108 MiB/s, 3 keys/s, 28 objects/s recovering 2023-03-09T14:50:32.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:31 smithi122 ceph-mon[93869]: osdmap e109: 12 total, 12 up, 12 in 2023-03-09T14:50:32.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:31 smithi181 ceph-mon[94189]: pgmap v149: 129 pgs: 1 active+undersized+remapped, 1 peering, 4 active+recovery_wait+degraded, 123 active+clean; 18 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 158 MiB/s wr, 485 op/s; 84/22282 objects degraded (0.377%); 108 MiB/s, 3 keys/s, 28 objects/s recovering 2023-03-09T14:50:32.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:31 smithi181 ceph-mon[94189]: osdmap e109: 12 total, 12 up, 12 in 2023-03-09T14:50:32.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:31 smithi038 ceph-mon[92422]: pgmap v149: 129 pgs: 1 active+undersized+remapped, 1 peering, 4 active+recovery_wait+degraded, 123 active+clean; 18 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 158 MiB/s wr, 485 op/s; 84/22282 objects degraded (0.377%); 108 MiB/s, 3 keys/s, 28 objects/s recovering 2023-03-09T14:50:32.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:31 smithi038 ceph-mon[92422]: osdmap e109: 12 total, 12 up, 12 in 2023-03-09T14:50:33.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:32 smithi122 ceph-mon[93869]: osdmap e110: 12 total, 12 up, 12 in 2023-03-09T14:50:33.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:32 smithi122 ceph-mon[93869]: Health check update: Degraded data redundancy: 84/22282 objects degraded (0.377%), 4 pgs degraded (PG_DEGRADED) 2023-03-09T14:50:33.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:32 smithi181 ceph-mon[94189]: osdmap e110: 12 total, 12 up, 12 in 2023-03-09T14:50:33.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:32 smithi181 ceph-mon[94189]: Health check update: Degraded data redundancy: 84/22282 objects degraded (0.377%), 4 pgs degraded (PG_DEGRADED) 2023-03-09T14:50:33.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:32 smithi038 ceph-mon[92422]: osdmap e110: 12 total, 12 up, 12 in 2023-03-09T14:50:33.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:32 smithi038 ceph-mon[92422]: Health check update: Degraded data redundancy: 84/22282 objects degraded (0.377%), 4 pgs degraded (PG_DEGRADED) 2023-03-09T14:50:33.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5.0 s, 755.4 tps, lat 125.270 ms stddev 172.826 2023-03-09T14:50:34.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:33 smithi122 ceph-mon[93869]: pgmap v152: 129 pgs: 1 active+undersized+remapped, 1 active+recovering+degraded, 2 active+recovery_wait+degraded, 125 active+clean; 18 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 148 MiB/s wr, 754 op/s; 52/22470 objects degraded (0.231%); 118 MiB/s, 31 objects/s recovering 2023-03-09T14:50:34.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:33 smithi181 ceph-mon[94189]: pgmap v152: 129 pgs: 1 active+undersized+remapped, 1 active+recovering+degraded, 2 active+recovery_wait+degraded, 125 active+clean; 18 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 148 MiB/s wr, 754 op/s; 52/22470 objects degraded (0.231%); 118 MiB/s, 31 objects/s recovering 2023-03-09T14:50:34.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:33 smithi038 ceph-mon[92422]: pgmap v152: 129 pgs: 1 active+undersized+remapped, 1 active+recovering+degraded, 2 active+recovery_wait+degraded, 125 active+clean; 18 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 148 MiB/s wr, 754 op/s; 52/22470 objects degraded (0.231%); 118 MiB/s, 31 objects/s recovering 2023-03-09T14:50:36.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:35 smithi122 ceph-mon[93869]: pgmap v153: 129 pgs: 1 active+undersized+remapped, 1 active+recovering+degraded, 2 active+recovery_wait+degraded, 125 active+clean; 18 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 122 MiB/s wr, 697 op/s; 52/22526 objects degraded (0.231%); 82 MiB/s, 21 objects/s recovering 2023-03-09T14:50:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:35 smithi181 ceph-mon[94189]: pgmap v153: 129 pgs: 1 active+undersized+remapped, 1 active+recovering+degraded, 2 active+recovery_wait+degraded, 125 active+clean; 18 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 122 MiB/s wr, 697 op/s; 52/22526 objects degraded (0.231%); 82 MiB/s, 21 objects/s recovering 2023-03-09T14:50:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:35 smithi038 ceph-mon[92422]: pgmap v153: 129 pgs: 1 active+undersized+remapped, 1 active+recovering+degraded, 2 active+recovery_wait+degraded, 125 active+clean; 18 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 122 MiB/s wr, 697 op/s; 52/22526 objects degraded (0.231%); 82 MiB/s, 21 objects/s recovering 2023-03-09T14:50:38.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:37 smithi122 ceph-mon[93869]: pgmap v154: 129 pgs: 1 active+recovering+degraded, 1 active+recovery_wait+degraded, 127 active+clean; 18 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 102 MiB/s wr, 1.55k op/s; 26/23162 objects degraded (0.112%); 66 MiB/s, 17 objects/s recovering 2023-03-09T14:50:38.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:37 smithi122 ceph-mon[93869]: Health check update: Degraded data redundancy: 26/23162 objects degraded (0.112%), 2 pgs degraded (PG_DEGRADED) 2023-03-09T14:50:38.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:37 smithi181 ceph-mon[94189]: pgmap v154: 129 pgs: 1 active+recovering+degraded, 1 active+recovery_wait+degraded, 127 active+clean; 18 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 102 MiB/s wr, 1.55k op/s; 26/23162 objects degraded (0.112%); 66 MiB/s, 17 objects/s recovering 2023-03-09T14:50:38.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:37 smithi181 ceph-mon[94189]: Health check update: Degraded data redundancy: 26/23162 objects degraded (0.112%), 2 pgs degraded (PG_DEGRADED) 2023-03-09T14:50:38.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:37 smithi038 ceph-mon[92422]: pgmap v154: 129 pgs: 1 active+recovering+degraded, 1 active+recovery_wait+degraded, 127 active+clean; 18 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 102 MiB/s wr, 1.55k op/s; 26/23162 objects degraded (0.112%); 66 MiB/s, 17 objects/s recovering 2023-03-09T14:50:38.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:37 smithi038 ceph-mon[92422]: Health check update: Degraded data redundancy: 26/23162 objects degraded (0.112%), 2 pgs degraded (PG_DEGRADED) 2023-03-09T14:50:38.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10.0 s, 1244.2 tps, lat 82.950 ms stddev 43.120 2023-03-09T14:50:39.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:39 smithi122 ceph-mon[93869]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 26/23162 objects degraded (0.112%), 2 pgs degraded) 2023-03-09T14:50:39.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:39 smithi122 ceph-mon[93869]: Cluster is now healthy 2023-03-09T14:50:39.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:39 smithi181 ceph-mon[94189]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 26/23162 objects degraded (0.112%), 2 pgs degraded) 2023-03-09T14:50:39.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:39 smithi181 ceph-mon[94189]: Cluster is now healthy 2023-03-09T14:50:39.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-mon[92422]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 26/23162 objects degraded (0.112%), 2 pgs degraded) 2023-03-09T14:50:39.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-mon[92422]: Cluster is now healthy 2023-03-09T14:50:39.645 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 2023-03-09T14:50:39.594+0000 7fcb88337700 -1 osd.3 pg_epoch: 110 pg[4.11s0( v 110'349 (0'0,110'349] local-lis/les=104/105 n=204 ec=85/85 lis/c=104/104 les/c/f=105/108/0 sis=104) [3,7,1,5]p3(0) r=0 lpr=104 luod=110'343 crt=110'343 lcod 110'342 mlcod 110'342 active+clean] bad op order, already applied 68875 > this 68874 2023-03-09T14:50:39.646 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7fcb88337700 time 2023-03-09T14:50:39.596015+0000 2023-03-09T14:50:39.646 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op") 2023-03-09T14:50:39.646 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:50:39.646 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x55d5080dbc85] 2023-03-09T14:50:39.646 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 2: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55d5083946e4] 2023-03-09T14:50:39.647 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55d50839ad5e] 2023-03-09T14:50:39.647 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 4: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55d5081ea986] 2023-03-09T14:50:39.647 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 5: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55d508500729] 2023-03-09T14:50:39.647 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55d5081ffebf] 2023-03-09T14:50:39.647 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55d5088b1d85] 2023-03-09T14:50:39.647 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55d5088b4f84] 2023-03-09T14:50:39.647 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7fcbaa0951ca] 2023-03-09T14:50:39.648 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 10: clone() 2023-03-09T14:50:39.648 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 2023-03-09T14:50:39.597+0000 7fcb88337700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7fcb88337700 time 2023-03-09T14:50:39.596015+0000 2023-03-09T14:50:39.648 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op") 2023-03-09T14:50:39.648 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 2023-03-09T14:50:39.648 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:50:39.649 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x55d5080dbc85] 2023-03-09T14:50:39.649 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 2: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55d5083946e4] 2023-03-09T14:50:39.649 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55d50839ad5e] 2023-03-09T14:50:39.649 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 4: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55d5081ea986] 2023-03-09T14:50:39.649 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 5: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55d508500729] 2023-03-09T14:50:39.650 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55d5081ffebf] 2023-03-09T14:50:39.650 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55d5088b1d85] 2023-03-09T14:50:39.650 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55d5088b4f84] 2023-03-09T14:50:39.650 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7fcbaa0951ca] 2023-03-09T14:50:39.650 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 10: clone() 2023-03-09T14:50:39.650 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 2023-03-09T14:50:39.651 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: *** Caught signal (Aborted) ** 2023-03-09T14:50:39.651 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: in thread 7fcb88337700 thread_name:tp_osd_tp 2023-03-09T14:50:39.651 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:50:39.651 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7fcbaa09fcf0] 2023-03-09T14:50:39.651 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 2: gsignal() 2023-03-09T14:50:39.651 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 3: abort() 2023-03-09T14:50:39.651 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x55d5080dbd45] 2023-03-09T14:50:39.652 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 5: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55d5083946e4] 2023-03-09T14:50:39.652 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 6: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55d50839ad5e] 2023-03-09T14:50:39.652 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55d5081ea986] 2023-03-09T14:50:39.652 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 8: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55d508500729] 2023-03-09T14:50:39.652 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55d5081ffebf] 2023-03-09T14:50:39.653 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55d5088b1d85] 2023-03-09T14:50:39.653 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55d5088b4f84] 2023-03-09T14:50:39.653 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fcbaa0951ca] 2023-03-09T14:50:39.653 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 13: clone() 2023-03-09T14:50:39.654 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 2023-03-09T14:50:39.602+0000 7fcb88337700 -1 *** Caught signal (Aborted) ** 2023-03-09T14:50:39.654 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: in thread 7fcb88337700 thread_name:tp_osd_tp 2023-03-09T14:50:39.654 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 2023-03-09T14:50:39.654 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:50:39.654 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7fcbaa09fcf0] 2023-03-09T14:50:39.654 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 2: gsignal() 2023-03-09T14:50:39.655 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 3: abort() 2023-03-09T14:50:39.655 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x55d5080dbd45] 2023-03-09T14:50:39.655 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 5: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55d5083946e4] 2023-03-09T14:50:39.655 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 6: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55d50839ad5e] 2023-03-09T14:50:39.655 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55d5081ea986] 2023-03-09T14:50:39.655 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 8: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55d508500729] 2023-03-09T14:50:39.655 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55d5081ffebf] 2023-03-09T14:50:39.656 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55d5088b1d85] 2023-03-09T14:50:39.656 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55d5088b4f84] 2023-03-09T14:50:39.656 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fcbaa0951ca] 2023-03-09T14:50:39.656 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 13: clone() 2023-03-09T14:50:39.656 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-03-09T14:50:39.656 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 2023-03-09T14:50:39.656 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: -277> 2023-03-09T14:50:39.594+0000 7fcb88337700 -1 osd.3 pg_epoch: 110 pg[4.11s0( v 110'349 (0'0,110'349] local-lis/les=104/105 n=204 ec=85/85 lis/c=104/104 les/c/f=105/108/0 sis=104) [3,7,1,5]p3(0) r=0 lpr=104 luod=110'343 crt=110'343 lcod 110'342 mlcod 110'342 active+clean] bad op order, already applied 68875 > this 68874 2023-03-09T14:50:39.657 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: -174> 2023-03-09T14:50:39.597+0000 7fcb88337700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7fcb88337700 time 2023-03-09T14:50:39.596015+0000 2023-03-09T14:50:39.657 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op") 2023-03-09T14:50:39.657 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 2023-03-09T14:50:39.657 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:50:39.657 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x55d5080dbc85] 2023-03-09T14:50:39.657 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 2: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55d5083946e4] 2023-03-09T14:50:39.657 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55d50839ad5e] 2023-03-09T14:50:39.658 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 4: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55d5081ea986] 2023-03-09T14:50:39.658 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 5: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55d508500729] 2023-03-09T14:50:39.658 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55d5081ffebf] 2023-03-09T14:50:39.658 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55d5088b1d85] 2023-03-09T14:50:39.658 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55d5088b4f84] 2023-03-09T14:50:39.658 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7fcbaa0951ca] 2023-03-09T14:50:39.658 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 10: clone() 2023-03-09T14:50:39.659 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 2023-03-09T14:50:39.659 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 0> 2023-03-09T14:50:39.602+0000 7fcb88337700 -1 *** Caught signal (Aborted) ** 2023-03-09T14:50:39.659 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: in thread 7fcb88337700 thread_name:tp_osd_tp 2023-03-09T14:50:39.659 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 2023-03-09T14:50:39.660 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:50:39.660 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7fcbaa09fcf0] 2023-03-09T14:50:39.660 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 2: gsignal() 2023-03-09T14:50:39.660 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 3: abort() 2023-03-09T14:50:39.660 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x55d5080dbd45] 2023-03-09T14:50:39.660 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 5: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55d5083946e4] 2023-03-09T14:50:39.661 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 6: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55d50839ad5e] 2023-03-09T14:50:39.661 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55d5081ea986] 2023-03-09T14:50:39.661 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 8: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55d508500729] 2023-03-09T14:50:39.661 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55d5081ffebf] 2023-03-09T14:50:39.661 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55d5088b1d85] 2023-03-09T14:50:39.661 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55d5088b4f84] 2023-03-09T14:50:39.661 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fcbaa0951ca] 2023-03-09T14:50:39.662 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 13: clone() 2023-03-09T14:50:39.662 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-03-09T14:50:39.662 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 2023-03-09T14:50:39.662 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: -1278> 2023-03-09T14:50:39.594+0000 7fcb88337700 -1 osd.3 pg_epoch: 110 pg[4.11s0( v 110'349 (0'0,110'349] local-lis/les=104/105 n=204 ec=85/85 lis/c=104/104 les/c/f=105/108/0 sis=104) [3,7,1,5]p3(0) r=0 lpr=104 luod=110'343 crt=110'343 lcod 110'342 mlcod 110'342 active+clean] bad op order, already applied 68875 > this 68874 2023-03-09T14:50:39.662 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: -1175> 2023-03-09T14:50:39.597+0000 7fcb88337700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7fcb88337700 time 2023-03-09T14:50:39.596015+0000 2023-03-09T14:50:39.662 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op") 2023-03-09T14:50:39.662 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 2023-03-09T14:50:39.663 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:50:39.663 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x55d5080dbc85] 2023-03-09T14:50:39.663 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 2: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55d5083946e4] 2023-03-09T14:50:39.663 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55d50839ad5e] 2023-03-09T14:50:39.663 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 4: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55d5081ea986] 2023-03-09T14:50:39.664 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 5: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55d508500729] 2023-03-09T14:50:39.664 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55d5081ffebf] 2023-03-09T14:50:39.664 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55d5088b1d85] 2023-03-09T14:50:39.664 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55d5088b4f84] 2023-03-09T14:50:39.664 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7fcbaa0951ca] 2023-03-09T14:50:39.664 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 10: clone() 2023-03-09T14:50:39.665 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 2023-03-09T14:50:39.665 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: -1001> 2023-03-09T14:50:39.602+0000 7fcb88337700 -1 *** Caught signal (Aborted) ** 2023-03-09T14:50:39.665 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: in thread 7fcb88337700 thread_name:tp_osd_tp 2023-03-09T14:50:39.665 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 2023-03-09T14:50:39.665 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:50:39.665 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7fcbaa09fcf0] 2023-03-09T14:50:39.666 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 2: gsignal() 2023-03-09T14:50:39.666 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 3: abort() 2023-03-09T14:50:39.666 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x55d5080dbd45] 2023-03-09T14:50:39.666 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 5: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55d5083946e4] 2023-03-09T14:50:39.998 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 6: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55d50839ad5e] 2023-03-09T14:50:39.998 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55d5081ea986] 2023-03-09T14:50:39.998 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 8: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55d508500729] 2023-03-09T14:50:39.998 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55d5081ffebf] 2023-03-09T14:50:39.998 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55d5088b1d85] 2023-03-09T14:50:39.998 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55d5088b4f84] 2023-03-09T14:50:39.999 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fcbaa0951ca] 2023-03-09T14:50:39.999 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 13: clone() 2023-03-09T14:50:39.999 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-03-09T14:50:39.999 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[110661]: 2023-03-09T14:50:40.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:40 smithi122 ceph-mon[93869]: pgmap v155: 129 pgs: 129 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 44 MiB/s wr, 1.72k op/s; 41 MiB/s, 10 objects/s recovering 2023-03-09T14:50:40.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:40 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.5 2023-03-09T14:50:40.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:40 smithi122 ceph-mon[93869]: osd.3 failed (root=default,host=smithi038) (connection refused reported by osd.5) 2023-03-09T14:50:40.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:40 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.9 2023-03-09T14:50:40.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:40 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.9 2023-03-09T14:50:40.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:40 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.0 2023-03-09T14:50:40.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:40 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.6 2023-03-09T14:50:40.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:40 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.10 2023-03-09T14:50:40.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:40 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.4 2023-03-09T14:50:40.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:40 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.0 2023-03-09T14:50:40.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:40 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.6 2023-03-09T14:50:40.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:40 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.11 2023-03-09T14:50:40.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:40 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.10 2023-03-09T14:50:40.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:40 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.7 2023-03-09T14:50:40.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:40 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.6 2023-03-09T14:50:40.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:40 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.9 2023-03-09T14:50:40.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:40 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.11 2023-03-09T14:50:40.481 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:40 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.1 2023-03-09T14:50:40.481 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:40 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.4 2023-03-09T14:50:40.481 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:40 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.1 2023-03-09T14:50:40.481 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:40 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.4 2023-03-09T14:50:40.481 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:40 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.8 2023-03-09T14:50:40.481 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:40 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.7 2023-03-09T14:50:40.481 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:40 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.2 2023-03-09T14:50:40.482 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:40 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.7 2023-03-09T14:50:40.482 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:40 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.5 2023-03-09T14:50:40.482 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:40 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.8 2023-03-09T14:50:40.482 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:40 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.2 2023-03-09T14:50:40.482 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:40 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.5 2023-03-09T14:50:40.482 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:40 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.1 2023-03-09T14:50:40.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:40 smithi181 ceph-mon[94189]: pgmap v155: 129 pgs: 129 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 44 MiB/s wr, 1.72k op/s; 41 MiB/s, 10 objects/s recovering 2023-03-09T14:50:40.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:40 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.5 2023-03-09T14:50:40.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:40 smithi181 ceph-mon[94189]: osd.3 failed (root=default,host=smithi038) (connection refused reported by osd.5) 2023-03-09T14:50:40.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:40 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.9 2023-03-09T14:50:40.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:40 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.9 2023-03-09T14:50:40.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:40 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.0 2023-03-09T14:50:40.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:40 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.6 2023-03-09T14:50:40.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:40 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.10 2023-03-09T14:50:40.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:40 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.4 2023-03-09T14:50:40.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:40 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.0 2023-03-09T14:50:40.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:40 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.6 2023-03-09T14:50:40.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:40 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.11 2023-03-09T14:50:40.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:40 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.10 2023-03-09T14:50:40.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:40 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.7 2023-03-09T14:50:40.485 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:40 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.6 2023-03-09T14:50:40.485 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:40 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.9 2023-03-09T14:50:40.485 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:40 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.11 2023-03-09T14:50:40.485 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:40 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.1 2023-03-09T14:50:40.485 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:40 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.4 2023-03-09T14:50:40.485 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:40 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.1 2023-03-09T14:50:40.485 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:40 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.4 2023-03-09T14:50:40.486 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:40 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.8 2023-03-09T14:50:40.486 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:40 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.7 2023-03-09T14:50:40.486 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:40 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.2 2023-03-09T14:50:40.486 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:40 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.7 2023-03-09T14:50:40.486 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:40 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.5 2023-03-09T14:50:40.486 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:40 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.8 2023-03-09T14:50:40.486 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:40 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.2 2023-03-09T14:50:40.487 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:40 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.5 2023-03-09T14:50:40.487 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:40 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.1 2023-03-09T14:50:40.500 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:40 smithi038 systemd[1]: ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.3.service: Main process exited, code=exited, status=134/n/a 2023-03-09T14:50:40.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:40 smithi038 ceph-mon[92422]: pgmap v155: 129 pgs: 129 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 44 MiB/s wr, 1.72k op/s; 41 MiB/s, 10 objects/s recovering 2023-03-09T14:50:40.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:40 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.5 2023-03-09T14:50:40.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:40 smithi038 ceph-mon[92422]: osd.3 failed (root=default,host=smithi038) (connection refused reported by osd.5) 2023-03-09T14:50:40.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:40 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.9 2023-03-09T14:50:40.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:40 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.9 2023-03-09T14:50:40.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:40 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.0 2023-03-09T14:50:40.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:40 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.6 2023-03-09T14:50:40.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:40 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.10 2023-03-09T14:50:40.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:40 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.4 2023-03-09T14:50:40.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:40 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.0 2023-03-09T14:50:40.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:40 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.6 2023-03-09T14:50:40.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:40 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.11 2023-03-09T14:50:40.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:40 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.10 2023-03-09T14:50:40.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:40 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.7 2023-03-09T14:50:40.503 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:40 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.6 2023-03-09T14:50:40.503 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:40 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.9 2023-03-09T14:50:40.503 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:40 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.11 2023-03-09T14:50:40.503 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:40 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.1 2023-03-09T14:50:40.503 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:40 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.4 2023-03-09T14:50:40.503 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:40 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.1 2023-03-09T14:50:40.503 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:40 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.4 2023-03-09T14:50:40.503 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:40 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.8 2023-03-09T14:50:40.504 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:40 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.7 2023-03-09T14:50:40.504 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:40 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.2 2023-03-09T14:50:40.504 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:40 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.7 2023-03-09T14:50:40.504 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:40 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.5 2023-03-09T14:50:40.504 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:40 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.8 2023-03-09T14:50:40.504 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:40 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.2 2023-03-09T14:50:40.504 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:40 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.5 2023-03-09T14:50:40.504 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:40 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.1 2023-03-09T14:50:41.447 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:41 smithi038 ceph-mon[92422]: Health check failed: 1 osds down (OSD_DOWN) 2023-03-09T14:50:41.448 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:41 smithi038 ceph-mon[92422]: osdmap e111: 12 total, 11 up, 12 in 2023-03-09T14:50:41.448 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:41 smithi038 ceph-mon[92422]: osdmap e112: 12 total, 11 up, 12 in 2023-03-09T14:50:41.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:41 smithi122 ceph-mon[93869]: Health check failed: 1 osds down (OSD_DOWN) 2023-03-09T14:50:41.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:41 smithi122 ceph-mon[93869]: osdmap e111: 12 total, 11 up, 12 in 2023-03-09T14:50:41.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:41 smithi122 ceph-mon[93869]: osdmap e112: 12 total, 11 up, 12 in 2023-03-09T14:50:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-mon[94189]: Health check failed: 1 osds down (OSD_DOWN) 2023-03-09T14:50:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-mon[94189]: osdmap e111: 12 total, 11 up, 12 in 2023-03-09T14:50:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-mon[94189]: osdmap e112: 12 total, 11 up, 12 in 2023-03-09T14:50:41.750 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:41 smithi038 systemd[1]: ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.3.service: Failed with result 'exit-code'. 2023-03-09T14:50:42.048 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 2023-03-09T14:50:41.782+0000 7f7fb3342700 -1 osd.8 pg_epoch: 112 pg[4.2s0( v 112'641 (0'0,112'641] local-lis/les=85/86 n=194 ec=85/85 lis/c=85/85 les/c/f=86/86/0 sis=85) [8,10,11,9]p8(0) r=0 lpr=85 luod=112'638 crt=112'638 lcod 112'637 mlcod 112'637 active+clean] bad op order, already applied 70876 > this 70875 2023-03-09T14:50:42.048 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7f7fb3342700 time 2023-03-09T14:50:41.782333+0000 2023-03-09T14:50:42.048 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op") 2023-03-09T14:50:42.048 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:50:42.048 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x55c007a84c85] 2023-03-09T14:50:42.049 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 2: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55c007d3d6e4] 2023-03-09T14:50:42.049 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55c007d43d5e] 2023-03-09T14:50:42.049 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 4: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55c007b93986] 2023-03-09T14:50:42.049 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 5: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55c007ea9729] 2023-03-09T14:50:42.049 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55c007ba8ebf] 2023-03-09T14:50:42.050 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55c00825ad85] 2023-03-09T14:50:42.050 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55c00825df84] 2023-03-09T14:50:42.050 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f7fdb0ac1ca] 2023-03-09T14:50:42.050 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 10: clone() 2023-03-09T14:50:42.050 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: *** Caught signal (Aborted) ** 2023-03-09T14:50:42.050 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: in thread 7f7fb3342700 thread_name:tp_osd_tp 2023-03-09T14:50:42.051 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 2023-03-09T14:50:41.785+0000 7f7fb3342700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7f7fb3342700 time 2023-03-09T14:50:41.782333+0000 2023-03-09T14:50:42.051 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op") 2023-03-09T14:50:42.051 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 2023-03-09T14:50:42.051 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:50:42.051 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x55c007a84c85] 2023-03-09T14:50:42.051 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 2: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55c007d3d6e4] 2023-03-09T14:50:42.051 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55c007d43d5e] 2023-03-09T14:50:42.052 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 4: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55c007b93986] 2023-03-09T14:50:42.052 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 5: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55c007ea9729] 2023-03-09T14:50:42.052 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55c007ba8ebf] 2023-03-09T14:50:42.052 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55c00825ad85] 2023-03-09T14:50:42.052 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55c00825df84] 2023-03-09T14:50:42.052 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f7fdb0ac1ca] 2023-03-09T14:50:42.052 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 10: clone() 2023-03-09T14:50:42.053 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 2023-03-09T14:50:42.053 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:50:42.053 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f7fdb0b6cf0] 2023-03-09T14:50:42.053 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 2: gsignal() 2023-03-09T14:50:42.053 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 3: abort() 2023-03-09T14:50:42.053 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x55c007a84d45] 2023-03-09T14:50:42.053 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 5: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55c007d3d6e4] 2023-03-09T14:50:42.054 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 6: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55c007d43d5e] 2023-03-09T14:50:42.054 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55c007b93986] 2023-03-09T14:50:42.054 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 8: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55c007ea9729] 2023-03-09T14:50:42.054 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55c007ba8ebf] 2023-03-09T14:50:42.054 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55c00825ad85] 2023-03-09T14:50:42.054 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55c00825df84] 2023-03-09T14:50:42.054 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f7fdb0ac1ca] 2023-03-09T14:50:42.055 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 13: clone() 2023-03-09T14:50:42.055 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 2023-03-09T14:50:41.790+0000 7f7fb3342700 -1 *** Caught signal (Aborted) ** 2023-03-09T14:50:42.055 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: in thread 7f7fb3342700 thread_name:tp_osd_tp 2023-03-09T14:50:42.055 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 2023-03-09T14:50:42.055 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:50:42.055 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f7fdb0b6cf0] 2023-03-09T14:50:42.055 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 2: gsignal() 2023-03-09T14:50:42.056 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 3: abort() 2023-03-09T14:50:42.056 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x55c007a84d45] 2023-03-09T14:50:42.056 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 5: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55c007d3d6e4] 2023-03-09T14:50:42.056 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 6: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55c007d43d5e] 2023-03-09T14:50:42.056 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55c007b93986] 2023-03-09T14:50:42.056 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 8: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55c007ea9729] 2023-03-09T14:50:42.056 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55c007ba8ebf] 2023-03-09T14:50:42.057 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55c00825ad85] 2023-03-09T14:50:42.057 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55c00825df84] 2023-03-09T14:50:42.057 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f7fdb0ac1ca] 2023-03-09T14:50:42.057 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 13: clone() 2023-03-09T14:50:42.057 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-03-09T14:50:42.057 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 2023-03-09T14:50:42.057 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: -686> 2023-03-09T14:50:41.782+0000 7f7fb3342700 -1 osd.8 pg_epoch: 112 pg[4.2s0( v 112'641 (0'0,112'641] local-lis/les=85/86 n=194 ec=85/85 lis/c=85/85 les/c/f=86/86/0 sis=85) [8,10,11,9]p8(0) r=0 lpr=85 luod=112'638 crt=112'638 lcod 112'637 mlcod 112'637 active+clean] bad op order, already applied 70876 > this 70875 2023-03-09T14:50:42.057 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: -101> 2023-03-09T14:50:41.785+0000 7f7fb3342700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7f7fb3342700 time 2023-03-09T14:50:41.782333+0000 2023-03-09T14:50:42.058 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op") 2023-03-09T14:50:42.058 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 2023-03-09T14:50:42.058 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:50:42.058 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x55c007a84c85] 2023-03-09T14:50:42.058 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 2: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55c007d3d6e4] 2023-03-09T14:50:42.058 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55c007d43d5e] 2023-03-09T14:50:42.058 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 4: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55c007b93986] 2023-03-09T14:50:42.058 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 5: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55c007ea9729] 2023-03-09T14:50:42.059 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55c007ba8ebf] 2023-03-09T14:50:42.059 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55c00825ad85] 2023-03-09T14:50:42.059 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55c00825df84] 2023-03-09T14:50:42.059 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f7fdb0ac1ca] 2023-03-09T14:50:42.059 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 10: clone() 2023-03-09T14:50:42.059 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 2023-03-09T14:50:42.059 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 0> 2023-03-09T14:50:41.790+0000 7f7fb3342700 -1 *** Caught signal (Aborted) ** 2023-03-09T14:50:42.060 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: in thread 7f7fb3342700 thread_name:tp_osd_tp 2023-03-09T14:50:42.060 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 2023-03-09T14:50:42.060 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:50:42.060 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f7fdb0b6cf0] 2023-03-09T14:50:42.060 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 2: gsignal() 2023-03-09T14:50:42.060 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 3: abort() 2023-03-09T14:50:42.060 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x55c007a84d45] 2023-03-09T14:50:42.061 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 5: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55c007d3d6e4] 2023-03-09T14:50:42.061 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 6: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55c007d43d5e] 2023-03-09T14:50:42.062 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55c007b93986] 2023-03-09T14:50:42.062 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 8: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55c007ea9729] 2023-03-09T14:50:42.062 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55c007ba8ebf] 2023-03-09T14:50:42.062 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55c00825ad85] 2023-03-09T14:50:42.062 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55c00825df84] 2023-03-09T14:50:42.062 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f7fdb0ac1ca] 2023-03-09T14:50:42.063 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 13: clone() 2023-03-09T14:50:42.063 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-03-09T14:50:42.063 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 2023-03-09T14:50:42.063 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: -1687> 2023-03-09T14:50:41.782+0000 7f7fb3342700 -1 osd.8 pg_epoch: 112 pg[4.2s0( v 112'641 (0'0,112'641] local-lis/les=85/86 n=194 ec=85/85 lis/c=85/85 les/c/f=86/86/0 sis=85) [8,10,11,9]p8(0) r=0 lpr=85 luod=112'638 crt=112'638 lcod 112'637 mlcod 112'637 active+clean] bad op order, already applied 70876 > this 70875 2023-03-09T14:50:42.063 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: -1102> 2023-03-09T14:50:41.785+0000 7f7fb3342700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7f7fb3342700 time 2023-03-09T14:50:41.782333+0000 2023-03-09T14:50:42.063 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op") 2023-03-09T14:50:42.063 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 2023-03-09T14:50:42.064 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:50:42.064 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x55c007a84c85] 2023-03-09T14:50:42.064 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 2: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55c007d3d6e4] 2023-03-09T14:50:42.064 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55c007d43d5e] 2023-03-09T14:50:42.064 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 4: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55c007b93986] 2023-03-09T14:50:42.064 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 5: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55c007ea9729] 2023-03-09T14:50:42.064 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55c007ba8ebf] 2023-03-09T14:50:42.064 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55c00825ad85] 2023-03-09T14:50:42.065 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55c00825df84] 2023-03-09T14:50:42.065 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f7fdb0ac1ca] 2023-03-09T14:50:42.065 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 10: clone() 2023-03-09T14:50:42.065 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 2023-03-09T14:50:42.065 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: -1001> 2023-03-09T14:50:41.790+0000 7f7fb3342700 -1 *** Caught signal (Aborted) ** 2023-03-09T14:50:42.065 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: in thread 7f7fb3342700 thread_name:tp_osd_tp 2023-03-09T14:50:42.065 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 2023-03-09T14:50:42.066 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:50:42.066 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f7fdb0b6cf0] 2023-03-09T14:50:42.066 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 2: gsignal() 2023-03-09T14:50:42.066 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 3: abort() 2023-03-09T14:50:42.066 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x55c007a84d45] 2023-03-09T14:50:42.066 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 5: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55c007d3d6e4] 2023-03-09T14:50:42.066 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 6: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55c007d43d5e] 2023-03-09T14:50:42.066 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55c007b93986] 2023-03-09T14:50:42.067 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 8: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55c007ea9729] 2023-03-09T14:50:42.067 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55c007ba8ebf] 2023-03-09T14:50:42.067 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55c00825ad85] 2023-03-09T14:50:42.067 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55c00825df84] 2023-03-09T14:50:42.067 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f7fdb0ac1ca] 2023-03-09T14:50:42.067 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 13: clone() 2023-03-09T14:50:42.067 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-03-09T14:50:42.067 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:41 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[110914]: 2023-03-09T14:50:42.176 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:42 smithi038 ceph-mon[92422]: pgmap v157: 129 pgs: 19 peering, 5 stale+active+clean, 105 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 31 MiB/s wr, 2.17k op/s; 33 MiB/s, 8 objects/s recovering 2023-03-09T14:50:42.176 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:42 smithi038 ceph-mon[92422]: Health check failed: Reduced data availability: 5 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2023-03-09T14:50:42.414 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-03-09T14:50:42.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:42 smithi122 ceph-mon[93869]: pgmap v157: 129 pgs: 19 peering, 5 stale+active+clean, 105 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 31 MiB/s wr, 2.17k op/s; 33 MiB/s, 8 objects/s recovering 2023-03-09T14:50:42.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:42 smithi122 ceph-mon[93869]: Health check failed: Reduced data availability: 5 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2023-03-09T14:50:42.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:42 smithi181 ceph-mon[94189]: pgmap v157: 129 pgs: 19 peering, 5 stale+active+clean, 105 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 31 MiB/s wr, 2.17k op/s; 33 MiB/s, 8 objects/s recovering 2023-03-09T14:50:42.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:42 smithi181 ceph-mon[94189]: Health check failed: Reduced data availability: 5 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2023-03-09T14:50:42.564 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.560+0000 7fea88328700 1 -- 172.21.15.38:0/3744383233 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea800ff9a0 msgr2=0x7fea80103960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:50:42.564 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.560+0000 7fea88328700 1 --2- 172.21.15.38:0/3744383233 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea800ff9a0 0x7fea80103960 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fea700099c0 tx=0x7fea70009cd0 comp rx=0 tx=0).stop 2023-03-09T14:50:42.564 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.560+0000 7fea88328700 1 -- 172.21.15.38:0/3744383233 shutdown_connections 2023-03-09T14:50:42.564 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.560+0000 7fea88328700 1 --2- 172.21.15.38:0/3744383233 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fea80104040 0x7fea8010bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:50:42.565 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.560+0000 7fea88328700 1 --2- 172.21.15.38:0/3744383233 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fea800ff000 0x7fea800ff3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:50:42.565 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.560+0000 7fea88328700 1 --2- 172.21.15.38:0/3744383233 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea800ff9a0 0x7fea80103960 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:50:42.565 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.560+0000 7fea88328700 1 -- 172.21.15.38:0/3744383233 >> 172.21.15.38:0/3744383233 conn(0x7fea80075c60 msgr2=0x7fea80076060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:50:42.565 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.560+0000 7fea88328700 1 -- 172.21.15.38:0/3744383233 shutdown_connections 2023-03-09T14:50:42.565 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.560+0000 7fea88328700 1 -- 172.21.15.38:0/3744383233 wait complete. 2023-03-09T14:50:42.567 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.561+0000 7fea88328700 1 Processor -- start 2023-03-09T14:50:42.567 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.561+0000 7fea88328700 1 -- start start 2023-03-09T14:50:42.567 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.561+0000 7fea88328700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fea800ff000 0x7fea801960c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:50:42.567 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.561+0000 7fea88328700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fea800ff9a0 0x7fea80196600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:50:42.567 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.561+0000 7fea88328700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea80104040 0x7fea8019aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:50:42.567 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.561+0000 7fea88328700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea8019b0a0 con 0x7fea80104040 2023-03-09T14:50:42.568 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.561+0000 7fea88328700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea8019b210 con 0x7fea800ff9a0 2023-03-09T14:50:42.568 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.561+0000 7fea88328700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea8019b3d0 con 0x7fea800ff000 2023-03-09T14:50:42.568 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.561+0000 7fea868c5700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea80104040 0x7fea8019aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:50:42.568 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.562+0000 7fea868c5700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea80104040 0x7fea8019aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:40266/0 (socket says 172.21.15.38:40266) 2023-03-09T14:50:42.568 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.562+0000 7fea868c5700 1 -- 172.21.15.38:0/1415220167 learned_addr learned my addr 172.21.15.38:0/1415220167 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:50:42.568 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.562+0000 7fea860c4700 1 --2- 172.21.15.38:0/1415220167 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fea800ff000 0x7fea801960c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:50:42.568 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.562+0000 7fea858c3700 1 --2- 172.21.15.38:0/1415220167 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fea800ff9a0 0x7fea80196600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:50:42.568 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.562+0000 7fea868c5700 1 -- 172.21.15.38:0/1415220167 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fea800ff000 msgr2=0x7fea801960c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:50:42.569 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.562+0000 7fea868c5700 1 --2- 172.21.15.38:0/1415220167 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fea800ff000 0x7fea801960c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:50:42.569 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.562+0000 7fea868c5700 1 -- 172.21.15.38:0/1415220167 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fea800ff9a0 msgr2=0x7fea80196600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:50:42.569 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.562+0000 7fea868c5700 1 --2- 172.21.15.38:0/1415220167 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fea800ff9a0 0x7fea80196600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:50:42.569 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.562+0000 7fea868c5700 1 -- 172.21.15.38:0/1415220167 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea700095f0 con 0x7fea80104040 2023-03-09T14:50:42.569 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.562+0000 7fea858c3700 1 --2- 172.21.15.38:0/1415220167 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fea800ff9a0 0x7fea80196600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:50:42.569 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.562+0000 7fea868c5700 1 --2- 172.21.15.38:0/1415220167 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea80104040 0x7fea8019aa60 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fea7800ef30 tx=0x7fea7800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:50:42.570 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.562+0000 7fea777fe700 1 -- 172.21.15.38:0/1415220167 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea78018070 con 0x7fea80104040 2023-03-09T14:50:42.570 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.562+0000 7fea777fe700 1 -- 172.21.15.38:0/1415220167 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fea7800f040 con 0x7fea80104040 2023-03-09T14:50:42.570 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.562+0000 7fea777fe700 1 -- 172.21.15.38:0/1415220167 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea7800cd00 con 0x7fea80104040 2023-03-09T14:50:42.570 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.562+0000 7fea88328700 1 -- 172.21.15.38:0/1415220167 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea8010edc0 con 0x7fea80104040 2023-03-09T14:50:42.570 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.562+0000 7fea88328700 1 -- 172.21.15.38:0/1415220167 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea8010ec30 con 0x7fea80104040 2023-03-09T14:50:42.574 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.565+0000 7fea777fe700 1 -- 172.21.15.38:0/1415220167 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 25) v1 ==== 133295+0+0 (secure 0 0 0) 0x7fea78003a00 con 0x7fea80104040 2023-03-09T14:50:42.574 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.565+0000 7fea777fe700 1 --2- 172.21.15.38:0/1415220167 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fea6c09f4a0 0x7fea6c0a1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:50:42.574 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.565+0000 7fea860c4700 1 --2- 172.21.15.38:0/1415220167 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fea6c09f4a0 0x7fea6c0a1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:50:42.574 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.565+0000 7fea777fe700 1 -- 172.21.15.38:0/1415220167 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 9489+0+0 (secure 0 0 0) 0x7fea780c9fd0 con 0x7fea80104040 2023-03-09T14:50:42.574 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.566+0000 7fea860c4700 1 --2- 172.21.15.38:0/1415220167 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fea6c09f4a0 0x7fea6c0a1950 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fea7c009c60 tx=0x7fea7c009260 comp rx=0 tx=0).ready entity=mgr.15558 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:50:42.574 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.566+0000 7fea944b1b80 1 Processor -- start 2023-03-09T14:50:42.574 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.566+0000 7fea944b1b80 1 -- start start 2023-03-09T14:50:42.575 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.566+0000 7fea944b1b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e2a3bd830 0x562e2a3bdc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:50:42.575 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.566+0000 7fea944b1b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562e2a3be140 0x562e2a3b7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:50:42.575 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.566+0000 7fea944b1b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562e2a3b7fe0 0x562e2a3b8410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:50:42.575 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.566+0000 7fea944b1b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e2a2a1da0 con 0x562e2a3bd830 2023-03-09T14:50:42.575 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.566+0000 7fea944b1b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e2a310780 con 0x562e2a3be140 2023-03-09T14:50:42.575 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.566+0000 7fea944b1b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e2a309cb0 con 0x562e2a3b7fe0 2023-03-09T14:50:42.575 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.566+0000 7fea860c4700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e2a3bd830 0x562e2a3bdc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:50:42.576 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.567+0000 7fea858c3700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562e2a3be140 0x562e2a3b7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:50:42.576 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.567+0000 7fea868c5700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562e2a3b7fe0 0x562e2a3b8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:50:42.576 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.567+0000 7fea858c3700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562e2a3be140 0x562e2a3b7aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.38:49462/0 (socket says 172.21.15.38:49462) 2023-03-09T14:50:42.576 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.567+0000 7fea858c3700 1 -- 172.21.15.38:0/1173080646 learned_addr learned my addr 172.21.15.38:0/1173080646 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:50:42.576 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.567+0000 7fea860c4700 1 -- 172.21.15.38:0/1173080646 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562e2a3b7fe0 msgr2=0x562e2a3b8410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:50:42.576 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.567+0000 7fea860c4700 1 --2- 172.21.15.38:0/1173080646 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562e2a3b7fe0 0x562e2a3b8410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:50:42.576 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.567+0000 7fea860c4700 1 -- 172.21.15.38:0/1173080646 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562e2a3be140 msgr2=0x562e2a3b7aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:50:42.577 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.567+0000 7fea860c4700 1 --2- 172.21.15.38:0/1173080646 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562e2a3be140 0x562e2a3b7aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:50:42.577 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.567+0000 7fea860c4700 1 -- 172.21.15.38:0/1173080646 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562e2a2d0190 con 0x562e2a3bd830 2023-03-09T14:50:42.577 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.567+0000 7fea858c3700 1 --2- 172.21.15.38:0/1173080646 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562e2a3be140 0x562e2a3b7aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:50:42.577 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.567+0000 7fea860c4700 1 --2- 172.21.15.38:0/1173080646 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e2a3bd830 0x562e2a3bdc00 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fea7c00e3b0 tx=0x7fea7c014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=addaa9e9262a17df server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:50:42.577 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.567+0000 7fea74daf700 1 -- 172.21.15.38:0/1173080646 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea7c00ed50 con 0x562e2a3bd830 2023-03-09T14:50:42.577 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.567+0000 7fea74daf700 1 -- 172.21.15.38:0/1173080646 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fea7c021070 con 0x562e2a3bd830 2023-03-09T14:50:42.577 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.567+0000 7fea74daf700 1 -- 172.21.15.38:0/1173080646 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea7c003ea0 con 0x562e2a3bd830 2023-03-09T14:50:42.577 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.568+0000 7fea944b1b80 1 -- 172.21.15.38:0/1173080646 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e2a3bd830 msgr2=0x562e2a3bdc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:50:42.578 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.568+0000 7fea944b1b80 1 --2- 172.21.15.38:0/1173080646 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e2a3bd830 0x562e2a3bdc00 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fea7c00e3b0 tx=0x7fea7c014000 comp rx=0 tx=0).stop 2023-03-09T14:50:42.578 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.568+0000 7fea944b1b80 1 -- 172.21.15.38:0/1173080646 shutdown_connections 2023-03-09T14:50:42.578 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.568+0000 7fea944b1b80 1 --2- 172.21.15.38:0/1173080646 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e2a3bd830 0x562e2a3bdc00 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:50:42.578 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.568+0000 7fea944b1b80 1 --2- 172.21.15.38:0/1173080646 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562e2a3b7fe0 0x562e2a3b8410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:50:42.578 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.568+0000 7fea944b1b80 1 --2- 172.21.15.38:0/1173080646 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562e2a3be140 0x562e2a3b7aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:50:42.578 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.568+0000 7fea944b1b80 1 -- 172.21.15.38:0/1173080646 >> 172.21.15.38:0/1173080646 conn(0x562e2a3b2a20 msgr2=0x562e2a3b2df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:50:42.578 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.568+0000 7fea944b1b80 1 -- 172.21.15.38:0/1173080646 shutdown_connections 2023-03-09T14:50:42.578 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.568+0000 7fea944b1b80 1 -- 172.21.15.38:0/1173080646 wait complete. 2023-03-09T14:50:42.579 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.568+0000 7fea944b1b80 1 -- start start 2023-03-09T14:50:42.579 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.569+0000 7fea944b1b80 1 Processor -- start 2023-03-09T14:50:42.579 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.569+0000 7fea944b1b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e2a3b7fe0 0x562e2a3c40a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:50:42.579 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.569+0000 7fea944b1b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562e2a3bd830 0x562e2a3c1320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:50:42.579 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.569+0000 7fea944b1b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562e2a3be140 0x562e2a3c1930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:50:42.579 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.569+0000 7fea944b1b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e2a18d920 con 0x562e2a3b7fe0 2023-03-09T14:50:42.579 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.569+0000 7fea944b1b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e2a3c20a0 con 0x562e2a3be140 2023-03-09T14:50:42.580 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.569+0000 7fea944b1b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e2a3cdc40 con 0x562e2a3bd830 2023-03-09T14:50:42.580 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.569+0000 7fea860c4700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e2a3b7fe0 0x562e2a3c40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:50:42.580 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.569+0000 7fea944b1b80 10 client.0 _ll_register_callbacks cb 0x562e2a2af520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:50:42.580 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.569+0000 7fea860c4700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e2a3b7fe0 0x562e2a3c40a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:40276/0 (socket says 172.21.15.38:40276) 2023-03-09T14:50:42.580 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.569+0000 7fea860c4700 1 -- 172.21.15.38:0/1987570193 learned_addr learned my addr 172.21.15.38:0/1987570193 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:50:42.580 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.569+0000 7fea858c3700 1 --2- 172.21.15.38:0/1987570193 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562e2a3bd830 0x562e2a3c1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:50:42.580 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.569+0000 7fea868c5700 1 --2- 172.21.15.38:0/1987570193 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562e2a3be140 0x562e2a3c1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:50:42.580 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.569+0000 7fea860c4700 1 -- 172.21.15.38:0/1987570193 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562e2a3bd830 msgr2=0x562e2a3c1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:50:42.581 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.569+0000 7fea860c4700 1 --2- 172.21.15.38:0/1987570193 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562e2a3bd830 0x562e2a3c1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:50:42.581 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.569+0000 7fea860c4700 1 -- 172.21.15.38:0/1987570193 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562e2a3be140 msgr2=0x562e2a3c1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:50:42.581 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.569+0000 7fea860c4700 1 --2- 172.21.15.38:0/1987570193 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562e2a3be140 0x562e2a3c1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:50:42.581 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.569+0000 7fea860c4700 1 -- 172.21.15.38:0/1987570193 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea8010c450 con 0x562e2a3b7fe0 2023-03-09T14:50:42.581 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.570+0000 7fea860c4700 1 --2- 172.21.15.38:0/1987570193 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e2a3b7fe0 0x562e2a3c40a0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fea7c009b20 tx=0x7fea7c003850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d527b7b570806619 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:50:42.581 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.570+0000 7fea69ffb700 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2023-03-09T14:50:42.581 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.570+0000 7fea69ffb700 1 -- 172.21.15.38:0/1987570193 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea7c0052c0 con 0x562e2a3b7fe0 2023-03-09T14:50:42.582 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.570+0000 7fea944b1b80 1 -- 172.21.15.38:0/1987570193 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562e2a2d7190 con 0x562e2a3b7fe0 2023-03-09T14:50:42.582 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.570+0000 7fea69ffb700 1 -- 172.21.15.38:0/1987570193 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(112..112 src has 1..112) v4 ==== 9489+0+0 (secure 0 0 0) 0x7fea7c003e30 con 0x562e2a3b7fe0 2023-03-09T14:50:42.582 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.570+0000 7fea69ffb700 1 -- 172.21.15.38:0/1987570193 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fea7c005420 con 0x562e2a3b7fe0 2023-03-09T14:50:42.582 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.570+0000 7fea944b1b80 10 client.15996 fetch_fsmap learned FSMap version 28 2023-03-09T14:50:42.582 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.570+0000 7fea944b1b80 1 -- 172.21.15.38:0/1987570193 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x7fea7c017070 con 0x562e2a3b7fe0 2023-03-09T14:50:42.582 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.570+0000 7fea69ffb700 1 -- 172.21.15.38:0/1987570193 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x562e2a2d7190 con 0x562e2a3b7fe0 2023-03-09T14:50:42.582 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.570+0000 7fea69ffb700 1 -- 172.21.15.38:0/1987570193 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) v1 ==== 7482+0+0 (secure 0 0 0) 0x7fea7c02d030 con 0x562e2a3b7fe0 2023-03-09T14:50:42.582 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.570+0000 7fea944b1b80 10 client.15996 fetch_fsmap finished waiting for FSMap version 28 2023-03-09T14:50:42.583 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.570+0000 7fea944b1b80 10 client.15996 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T14:50:42.583 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.570+0000 7fea944b1b80 20 client.15996 populate_metadata read hostname 'smithi038' 2023-03-09T14:50:42.583 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.570+0000 7fea944b1b80 1 --2- 172.21.15.38:0/1987570193 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x562e2a3d1a90 0x562e2a3d3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:50:42.583 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.570+0000 7fea944b1b80 4 client.15996 mds_command: new command op to 24530 tid=0[{"prefix": "get_command_descriptions"}] 2023-03-09T14:50:42.583 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.570+0000 7fea944b1b80 1 -- 172.21.15.38:0/1987570193 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562e2a18df60 con 0x562e2a3d1a90 2023-03-09T14:50:42.583 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.571+0000 7fea858c3700 1 --2- 172.21.15.38:0/1987570193 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x562e2a3d1a90 0x562e2a3d3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:50:42.584 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.571+0000 7fea69ffb700 10 client.15996 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:50:42.584 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.571+0000 7fea858c3700 1 --2- 172.21.15.38:0/1987570193 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x562e2a3d1a90 0x562e2a3d3ea0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2bfa6334d7257cbb server_cookie=c172966b8e1bd47e in_seq=0 out_seq=0 2023-03-09T14:50:42.585 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.572+0000 7fea69ffb700 1 -- 172.21.15.38:0/1987570193 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18278 (crc 0 0 0) 0x562e2a18df60 con 0x562e2a3d1a90 2023-03-09T14:50:42.585 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.572+0000 7fea69ffb700 10 client.15996 handle_command_reply: tid=0 2023-03-09T14:50:42.585 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.572+0000 7fea944b1b80 1 client.15996 shutdown 2023-03-09T14:50:42.585 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.572+0000 7fea944b1b80 1 -- 172.21.15.38:0/1987570193 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e2a3b7fe0 msgr2=0x562e2a3c40a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:50:42.585 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.572+0000 7fea944b1b80 1 --2- 172.21.15.38:0/1987570193 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e2a3b7fe0 0x562e2a3c40a0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fea7c009b20 tx=0x7fea7c003850 comp rx=0 tx=0).stop 2023-03-09T14:50:42.585 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.572+0000 7fea944b1b80 1 -- 172.21.15.38:0/1987570193 shutdown_connections 2023-03-09T14:50:42.585 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.572+0000 7fea944b1b80 1 --2- 172.21.15.38:0/1987570193 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x562e2a3d1a90 0x562e2a3d3ea0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:50:42.586 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.572+0000 7fea944b1b80 1 --2- 172.21.15.38:0/1987570193 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e2a3b7fe0 0x562e2a3c40a0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:50:42.586 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.572+0000 7fea69ffb700 0 client.15996 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:50:42.586 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.572+0000 7fea944b1b80 1 --2- 172.21.15.38:0/1987570193 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562e2a3be140 0x562e2a3c1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:50:42.586 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.572+0000 7fea944b1b80 1 --2- 172.21.15.38:0/1987570193 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562e2a3bd830 0x562e2a3c1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:50:42.586 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.572+0000 7fea944b1b80 1 -- 172.21.15.38:0/1987570193 >> 172.21.15.38:0/1987570193 conn(0x562e2a3b2a20 msgr2=0x562e2a3b1c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:50:42.586 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.572+0000 7fea944b1b80 1 -- 172.21.15.38:0/1987570193 shutdown_connections 2023-03-09T14:50:42.586 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.572+0000 7fea944b1b80 1 -- 172.21.15.38:0/1987570193 wait complete. 2023-03-09T14:50:42.587 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.572+0000 7fea944b1b80 20 client.15996 trim_cache size 0 max 16384 2023-03-09T14:50:42.594 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.589+0000 7fea944b1b80 1 Processor -- start 2023-03-09T14:50:42.594 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.590+0000 7fea944b1b80 1 -- start start 2023-03-09T14:50:42.594 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.590+0000 7fea944b1b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e2a3b7fe0 0x562e2a3c3c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:50:42.594 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.590+0000 7fea944b1b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562e2a3bd830 0x562e2a3b1c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:50:42.594 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.590+0000 7fea944b1b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562e2a3be140 0x562e2a3bb490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:50:42.595 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.590+0000 7fea944b1b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e2a3ce6b0 con 0x562e2a3b7fe0 2023-03-09T14:50:42.595 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.590+0000 7fea944b1b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e2a3b8cc0 con 0x562e2a3be140 2023-03-09T14:50:42.595 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.590+0000 7fea944b1b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e2a3c2900 con 0x562e2a3bd830 2023-03-09T14:50:42.595 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.590+0000 7fea860c4700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e2a3b7fe0 0x562e2a3c3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:50:42.595 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.590+0000 7fea860c4700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e2a3b7fe0 0x562e2a3c3c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:40284/0 (socket says 172.21.15.38:40284) 2023-03-09T14:50:42.595 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.590+0000 7fea860c4700 1 -- 172.21.15.38:0/1193568691 learned_addr learned my addr 172.21.15.38:0/1193568691 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:50:42.595 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.590+0000 7fea858c3700 1 --2- 172.21.15.38:0/1193568691 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562e2a3bd830 0x562e2a3b1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:50:42.595 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.590+0000 7fea868c5700 1 --2- 172.21.15.38:0/1193568691 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562e2a3be140 0x562e2a3bb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:50:42.596 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.590+0000 7fea860c4700 1 -- 172.21.15.38:0/1193568691 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562e2a3bd830 msgr2=0x562e2a3b1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:50:42.596 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.590+0000 7fea860c4700 1 --2- 172.21.15.38:0/1193568691 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562e2a3bd830 0x562e2a3b1c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:50:42.596 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.590+0000 7fea860c4700 1 -- 172.21.15.38:0/1193568691 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562e2a3be140 msgr2=0x562e2a3bb490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:50:42.596 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.590+0000 7fea860c4700 1 --2- 172.21.15.38:0/1193568691 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562e2a3be140 0x562e2a3bb490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:50:42.596 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.590+0000 7fea860c4700 1 -- 172.21.15.38:0/1193568691 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562e2a120c60 con 0x562e2a3b7fe0 2023-03-09T14:50:42.596 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.590+0000 7fea860c4700 1 --2- 172.21.15.38:0/1193568691 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e2a3b7fe0 0x562e2a3c3c40 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fea7c0097c0 tx=0x7fea7c005ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c877514d0125875 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:50:42.596 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.591+0000 7fea755b0700 1 -- 172.21.15.38:0/1193568691 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea7c003ac0 con 0x562e2a3b7fe0 2023-03-09T14:50:42.597 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.591+0000 7fea755b0700 1 -- 172.21.15.38:0/1193568691 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fea7c021070 con 0x562e2a3b7fe0 2023-03-09T14:50:42.597 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.591+0000 7fea755b0700 1 -- 172.21.15.38:0/1193568691 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea7c005310 con 0x562e2a3b7fe0 2023-03-09T14:50:42.600 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.591+0000 7fea944b1b80 1 -- 172.21.15.38:0/1193568691 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e2a3b7fe0 msgr2=0x562e2a3c3c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:50:42.600 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.591+0000 7fea944b1b80 1 --2- 172.21.15.38:0/1193568691 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e2a3b7fe0 0x562e2a3c3c40 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fea7c0097c0 tx=0x7fea7c005ce0 comp rx=0 tx=0).stop 2023-03-09T14:50:42.600 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.591+0000 7fea944b1b80 1 -- 172.21.15.38:0/1193568691 shutdown_connections 2023-03-09T14:50:42.600 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.591+0000 7fea944b1b80 1 --2- 172.21.15.38:0/1193568691 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e2a3b7fe0 0x562e2a3c3c40 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:50:42.600 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.591+0000 7fea944b1b80 1 --2- 172.21.15.38:0/1193568691 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562e2a3be140 0x562e2a3bb490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:50:42.600 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.591+0000 7fea944b1b80 1 --2- 172.21.15.38:0/1193568691 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562e2a3bd830 0x562e2a3b1c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:50:42.601 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.591+0000 7fea944b1b80 1 -- 172.21.15.38:0/1193568691 >> 172.21.15.38:0/1193568691 conn(0x562e2a3b2a20 msgr2=0x562e2a3b0f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:50:42.601 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.591+0000 7fea944b1b80 1 -- 172.21.15.38:0/1193568691 shutdown_connections 2023-03-09T14:50:42.601 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.591+0000 7fea944b1b80 1 -- 172.21.15.38:0/1193568691 wait complete. 2023-03-09T14:50:42.601 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.592+0000 7fea944b1b80 1 -- start start 2023-03-09T14:50:42.601 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.592+0000 7fea944b1b80 1 Processor -- start 2023-03-09T14:50:42.601 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.592+0000 7fea944b1b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e2a3b7fe0 0x562e2a3c8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:50:42.601 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.592+0000 7fea944b1b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562e2a3bd830 0x562e2a3b93f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:50:42.602 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.592+0000 7fea944b1b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562e2a3be140 0x562e2a3b9930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:50:42.602 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.592+0000 7fea944b1b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e2a3b9fd0 con 0x562e2a3b7fe0 2023-03-09T14:50:42.602 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.592+0000 7fea944b1b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e2a3ba140 con 0x562e2a3bd830 2023-03-09T14:50:42.602 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.592+0000 7fea944b1b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x562e2a425930 con 0x562e2a3be140 2023-03-09T14:50:42.602 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.592+0000 7fea944b1b80 10 client.0 _ll_register_callbacks cb 0x562e2a2af520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:50:42.602 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.592+0000 7fea860c4700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e2a3b7fe0 0x562e2a3c8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:50:42.602 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.592+0000 7fea860c4700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e2a3b7fe0 0x562e2a3c8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:40290/0 (socket says 172.21.15.38:40290) 2023-03-09T14:50:42.602 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.592+0000 7fea860c4700 1 -- 172.21.15.38:0/2081181427 learned_addr learned my addr 172.21.15.38:0/2081181427 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:50:42.603 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.592+0000 7fea858c3700 1 --2- 172.21.15.38:0/2081181427 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562e2a3bd830 0x562e2a3b93f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:50:42.603 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.592+0000 7fea868c5700 1 --2- 172.21.15.38:0/2081181427 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562e2a3be140 0x562e2a3b9930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:50:42.603 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.592+0000 7fea860c4700 1 -- 172.21.15.38:0/2081181427 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562e2a3be140 msgr2=0x562e2a3b9930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:50:42.603 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.592+0000 7fea860c4700 1 --2- 172.21.15.38:0/2081181427 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562e2a3be140 0x562e2a3b9930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:50:42.603 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.592+0000 7fea860c4700 1 -- 172.21.15.38:0/2081181427 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562e2a3bd830 msgr2=0x562e2a3b93f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:50:42.603 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.592+0000 7fea860c4700 1 --2- 172.21.15.38:0/2081181427 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562e2a3bd830 0x562e2a3b93f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:50:42.603 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.592+0000 7fea860c4700 1 -- 172.21.15.38:0/2081181427 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea7c017070 con 0x562e2a3b7fe0 2023-03-09T14:50:42.604 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.592+0000 7fea858c3700 1 --2- 172.21.15.38:0/2081181427 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562e2a3bd830 0x562e2a3b93f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:50:42.604 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.593+0000 7fea860c4700 1 --2- 172.21.15.38:0/2081181427 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e2a3b7fe0 0x562e2a3c8b90 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fea7c01d4d0 tx=0x7fea7c0039f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b49438f60c0539f server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:50:42.604 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.593+0000 7fea69ffb700 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2023-03-09T14:50:42.604 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.593+0000 7fea69ffb700 1 -- 172.21.15.38:0/2081181427 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea7c01dca0 con 0x562e2a3b7fe0 2023-03-09T14:50:42.604 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.593+0000 7fea69ffb700 1 -- 172.21.15.38:0/2081181427 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(112..112 src has 1..112) v4 ==== 9489+0+0 (secure 0 0 0) 0x7fea7c01de00 con 0x562e2a3b7fe0 2023-03-09T14:50:42.604 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.593+0000 7fea944b1b80 1 -- 172.21.15.38:0/2081181427 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562e2a2c71b0 con 0x562e2a3b7fe0 2023-03-09T14:50:42.604 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.593+0000 7fea69ffb700 1 -- 172.21.15.38:0/2081181427 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fea7c0092d0 con 0x562e2a3b7fe0 2023-03-09T14:50:42.604 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.593+0000 7fea944b1b80 10 client.16002 fetch_fsmap learned FSMap version 28 2023-03-09T14:50:42.605 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.593+0000 7fea944b1b80 1 -- 172.21.15.38:0/2081181427 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x562e2a15e250 con 0x562e2a3b7fe0 2023-03-09T14:50:42.605 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.593+0000 7fea69ffb700 1 -- 172.21.15.38:0/2081181427 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x562e2a2c71b0 con 0x562e2a3b7fe0 2023-03-09T14:50:42.605 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.593+0000 7fea69ffb700 1 -- 172.21.15.38:0/2081181427 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 28) v1 ==== 7482+0+0 (secure 0 0 0) 0x7fea7c02d070 con 0x562e2a3b7fe0 2023-03-09T14:50:42.605 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.593+0000 7fea944b1b80 10 client.16002 fetch_fsmap finished waiting for FSMap version 28 2023-03-09T14:50:42.605 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.593+0000 7fea944b1b80 10 client.16002 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T14:50:42.605 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.593+0000 7fea944b1b80 20 client.16002 populate_metadata read hostname 'smithi038' 2023-03-09T14:50:42.605 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.593+0000 7fea944b1b80 1 --2- 172.21.15.38:0/2081181427 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x562e2a42ade0 0x562e2a42d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:50:42.606 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.593+0000 7fea944b1b80 4 client.16002 mds_command: new command op to 24530 tid=0[{"prefix": "scrub status"}] 2023-03-09T14:50:42.606 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.593+0000 7fea944b1b80 1 -- 172.21.15.38:0/2081181427 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x562e2a2b11c0 con 0x562e2a42ade0 2023-03-09T14:50:42.606 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.594+0000 7fea858c3700 1 --2- 172.21.15.38:0/2081181427 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x562e2a42ade0 0x562e2a42d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:50:42.606 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.594+0000 7fea858c3700 1 --2- 172.21.15.38:0/2081181427 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x562e2a42ade0 0x562e2a42d1c0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b31161f7282a7b44 server_cookie=7a5353bdfff234a4 in_seq=0 out_seq=0 2023-03-09T14:50:42.606 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.594+0000 7fea69ffb700 10 client.16002 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:50:42.606 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.594+0000 7fea69ffb700 1 -- 172.21.15.38:0/2081181427 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+265 (crc 0 0 0) 0x562e2a2b11c0 con 0x562e2a42ade0 2023-03-09T14:50:42.606 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.594+0000 7fea69ffb700 10 client.16002 handle_command_reply: tid=0 2023-03-09T14:50:42.606 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.594+0000 7fea944b1b80 1 client.16002 shutdown 2023-03-09T14:50:42.607 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.594+0000 7fea944b1b80 1 -- 172.21.15.38:0/2081181427 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e2a3b7fe0 msgr2=0x562e2a3c8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:50:42.607 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.594+0000 7fea944b1b80 1 --2- 172.21.15.38:0/2081181427 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e2a3b7fe0 0x562e2a3c8b90 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fea7c01d4d0 tx=0x7fea7c0039f0 comp rx=0 tx=0).stop 2023-03-09T14:50:42.607 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.595+0000 7fea944b1b80 1 -- 172.21.15.38:0/2081181427 shutdown_connections 2023-03-09T14:50:42.607 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.595+0000 7fea944b1b80 1 --2- 172.21.15.38:0/2081181427 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x562e2a42ade0 0x562e2a42d1c0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:50:42.607 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.595+0000 7fea944b1b80 1 --2- 172.21.15.38:0/2081181427 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562e2a3b7fe0 0x562e2a3c8b90 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:50:42.607 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.595+0000 7fea944b1b80 1 --2- 172.21.15.38:0/2081181427 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562e2a3be140 0x562e2a3b9930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:50:42.607 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.595+0000 7fea69ffb700 0 client.16002 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:50:42.607 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.595+0000 7fea944b1b80 1 --2- 172.21.15.38:0/2081181427 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562e2a3bd830 0x562e2a3b93f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:50:42.608 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.595+0000 7fea944b1b80 1 -- 172.21.15.38:0/2081181427 >> 172.21.15.38:0/2081181427 conn(0x562e2a3b2a20 msgr2=0x562e2a3b1c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:50:42.608 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.595+0000 7fea944b1b80 1 -- 172.21.15.38:0/2081181427 shutdown_connections 2023-03-09T14:50:42.608 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.595+0000 7fea944b1b80 1 -- 172.21.15.38:0/2081181427 wait complete. 2023-03-09T14:50:42.608 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.595+0000 7fea944b1b80 20 client.16002 trim_cache size 0 max 16384 2023-03-09T14:50:42.608 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2023-03-09T14:50:42.609 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "scrub active (3 inodes in the stack)", 2023-03-09T14:50:42.609 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": { 2023-03-09T14:50:42.609 INFO:teuthology.orchestra.run.smithi038.stdout: "a14659f4-d7ea-4c5a-b15b-6138af3f909f": { 2023-03-09T14:50:42.609 INFO:teuthology.orchestra.run.smithi038.stdout: "path": "/", 2023-03-09T14:50:42.609 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "a14659f4-d7ea-4c5a-b15b-6138af3f909f", 2023-03-09T14:50:42.609 INFO:teuthology.orchestra.run.smithi038.stdout: "options": "recursive,force" 2023-03-09T14:50:42.609 INFO:teuthology.orchestra.run.smithi038.stdout: } 2023-03-09T14:50:42.609 INFO:teuthology.orchestra.run.smithi038.stdout: } 2023-03-09T14:50:42.610 INFO:teuthology.orchestra.run.smithi038.stdout:} 2023-03-09T14:50:42.611 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.595+0000 7fea88328700 1 -- 172.21.15.38:0/1415220167 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fea6c09f4a0 msgr2=0x7fea6c0a1950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:50:42.611 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.595+0000 7fea88328700 1 --2- 172.21.15.38:0/1415220167 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fea6c09f4a0 0x7fea6c0a1950 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fea7c009c60 tx=0x7fea7c009260 comp rx=0 tx=0).stop 2023-03-09T14:50:42.611 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.595+0000 7fea88328700 1 -- 172.21.15.38:0/1415220167 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea80104040 msgr2=0x7fea8019aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:50:42.611 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.595+0000 7fea88328700 1 --2- 172.21.15.38:0/1415220167 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea80104040 0x7fea8019aa60 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fea7800ef30 tx=0x7fea7800c3a0 comp rx=0 tx=0).stop 2023-03-09T14:50:42.611 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.595+0000 7fea88328700 1 -- 172.21.15.38:0/1415220167 shutdown_connections 2023-03-09T14:50:42.611 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.595+0000 7fea88328700 1 --2- 172.21.15.38:0/1415220167 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fea6c09f4a0 0x7fea6c0a1950 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:50:42.612 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.595+0000 7fea88328700 1 --2- 172.21.15.38:0/1415220167 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fea80104040 0x7fea8019aa60 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:50:42.612 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.595+0000 7fea88328700 1 --2- 172.21.15.38:0/1415220167 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fea800ff000 0x7fea801960c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:50:42.612 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.595+0000 7fea88328700 1 --2- 172.21.15.38:0/1415220167 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fea800ff9a0 0x7fea80196600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:50:42.612 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.595+0000 7fea88328700 1 -- 172.21.15.38:0/1415220167 >> 172.21.15.38:0/1415220167 conn(0x7fea80075c60 msgr2=0x7fea80067b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:50:42.612 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.595+0000 7fea88328700 1 -- 172.21.15.38:0/1415220167 shutdown_connections 2023-03-09T14:50:42.612 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:50:42.595+0000 7fea88328700 1 -- 172.21.15.38:0/1415220167 wait complete. 2023-03-09T14:50:42.612 INFO:tasks.cephfs.filesystem:scrub status for tag:a14659f4-d7ea-4c5a-b15b-6138af3f909f - {'path': '/', 'tag': 'a14659f4-d7ea-4c5a-b15b-6138af3f909f', 'options': 'recursive,force'} 2023-03-09T14:50:43.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:43 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.11 2023-03-09T14:50:43.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:43 smithi122 ceph-mon[93869]: osd.8 failed (root=default,host=smithi181) (connection refused reported by osd.11) 2023-03-09T14:50:43.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:43 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.11 2023-03-09T14:50:43.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:43 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.2 2023-03-09T14:50:43.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:43 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.4 2023-03-09T14:50:43.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:43 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.0 2023-03-09T14:50:43.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:43 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.6 2023-03-09T14:50:43.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:43 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.9 2023-03-09T14:50:43.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:43 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.5 2023-03-09T14:50:43.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:43 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.4 2023-03-09T14:50:43.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:43 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.0 2023-03-09T14:50:43.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:43 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.6 2023-03-09T14:50:43.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:43 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.9 2023-03-09T14:50:43.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:43 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.2 2023-03-09T14:50:43.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:43 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.7 2023-03-09T14:50:43.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:43 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.5 2023-03-09T14:50:43.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:43 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.7 2023-03-09T14:50:43.481 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:43 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.1 2023-03-09T14:50:43.481 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:43 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.1 2023-03-09T14:50:43.481 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:43 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.10 2023-03-09T14:50:43.481 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:43 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.10 2023-03-09T14:50:43.481 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:43 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.11 2023-03-09T14:50:43.481 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:43 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.11 2023-03-09T14:50:43.482 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:43 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.2 2023-03-09T14:50:43.482 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:43 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.0 2023-03-09T14:50:43.482 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:43 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.6 2023-03-09T14:50:43.482 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:43 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.9 2023-03-09T14:50:43.482 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:43 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.2 2023-03-09T14:50:43.482 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:43 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.7 2023-03-09T14:50:43.482 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:43 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.0 2023-03-09T14:50:43.482 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:43 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.6 2023-03-09T14:50:43.483 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:43 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.9 2023-03-09T14:50:43.483 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:43 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.5 2023-03-09T14:50:43.483 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:43 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.7 2023-03-09T14:50:43.483 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:43 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.5 2023-03-09T14:50:43.483 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:43 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.4 2023-03-09T14:50:43.483 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:43 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.1 2023-03-09T14:50:43.483 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:43 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.4 2023-03-09T14:50:43.483 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:43 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.1 2023-03-09T14:50:43.484 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:43 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.10 2023-03-09T14:50:43.484 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:43 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.10 2023-03-09T14:50:43.484 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:43 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.7 2023-03-09T14:50:43.484 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:43 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.7 2023-03-09T14:50:43.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:43 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.11 2023-03-09T14:50:43.485 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:43 smithi181 ceph-mon[94189]: osd.8 failed (root=default,host=smithi181) (connection refused reported by osd.11) 2023-03-09T14:50:43.485 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:43 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.11 2023-03-09T14:50:43.485 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:43 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.2 2023-03-09T14:50:43.485 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:43 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.4 2023-03-09T14:50:43.485 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:43 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.0 2023-03-09T14:50:43.485 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:43 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.6 2023-03-09T14:50:43.486 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:43 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.9 2023-03-09T14:50:43.486 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:43 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.5 2023-03-09T14:50:43.486 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:43 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.4 2023-03-09T14:50:43.486 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:43 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.0 2023-03-09T14:50:43.486 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:43 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.6 2023-03-09T14:50:43.486 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:43 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.9 2023-03-09T14:50:43.486 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:43 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.2 2023-03-09T14:50:43.486 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:43 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.7 2023-03-09T14:50:43.487 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:43 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.5 2023-03-09T14:50:43.487 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:43 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.7 2023-03-09T14:50:43.487 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:43 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.1 2023-03-09T14:50:43.487 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:43 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.1 2023-03-09T14:50:43.487 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:43 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.10 2023-03-09T14:50:43.487 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:43 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.10 2023-03-09T14:50:43.487 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:43 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.11 2023-03-09T14:50:43.487 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:43 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.11 2023-03-09T14:50:43.488 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:43 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.2 2023-03-09T14:50:43.488 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:43 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.0 2023-03-09T14:50:43.488 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:43 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.6 2023-03-09T14:50:43.488 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:43 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.9 2023-03-09T14:50:43.488 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:43 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.2 2023-03-09T14:50:43.488 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:43 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.7 2023-03-09T14:50:43.488 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:43 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.0 2023-03-09T14:50:43.488 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:43 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.6 2023-03-09T14:50:43.489 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:43 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.9 2023-03-09T14:50:43.489 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:43 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.5 2023-03-09T14:50:43.489 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:43 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.7 2023-03-09T14:50:43.489 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:43 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.5 2023-03-09T14:50:43.489 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:43 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.4 2023-03-09T14:50:43.489 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:43 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.1 2023-03-09T14:50:43.489 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:43 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.4 2023-03-09T14:50:43.490 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:43 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.1 2023-03-09T14:50:43.490 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:43 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.10 2023-03-09T14:50:43.490 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:43 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.10 2023-03-09T14:50:43.490 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:43 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.7 2023-03-09T14:50:43.490 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:43 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.7 2023-03-09T14:50:43.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:43 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.11 2023-03-09T14:50:43.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:43 smithi038 ceph-mon[92422]: osd.8 failed (root=default,host=smithi181) (connection refused reported by osd.11) 2023-03-09T14:50:43.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:43 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.11 2023-03-09T14:50:43.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:43 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.2 2023-03-09T14:50:43.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:43 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.4 2023-03-09T14:50:43.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:43 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.0 2023-03-09T14:50:43.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:43 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.6 2023-03-09T14:50:43.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:43 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.9 2023-03-09T14:50:43.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:43 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.5 2023-03-09T14:50:43.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:43 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.4 2023-03-09T14:50:43.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:43 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.0 2023-03-09T14:50:43.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:43 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.6 2023-03-09T14:50:43.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:43 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.9 2023-03-09T14:50:43.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:43 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.2 2023-03-09T14:50:43.503 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:43 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.7 2023-03-09T14:50:43.503 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:43 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.5 2023-03-09T14:50:43.503 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:43 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.7 2023-03-09T14:50:43.503 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:43 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.1 2023-03-09T14:50:43.503 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:43 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.1 2023-03-09T14:50:43.503 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:43 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.10 2023-03-09T14:50:43.503 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:43 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.10 2023-03-09T14:50:43.503 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:43 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.11 2023-03-09T14:50:43.504 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:43 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.11 2023-03-09T14:50:43.504 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:43 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.2 2023-03-09T14:50:43.504 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:43 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.0 2023-03-09T14:50:43.504 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:43 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.6 2023-03-09T14:50:43.504 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:43 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.9 2023-03-09T14:50:43.504 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:43 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.2 2023-03-09T14:50:43.504 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:43 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.7 2023-03-09T14:50:43.504 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:43 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.0 2023-03-09T14:50:43.505 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:43 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.6 2023-03-09T14:50:43.505 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:43 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.9 2023-03-09T14:50:43.505 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:43 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.5 2023-03-09T14:50:43.505 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:43 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.7 2023-03-09T14:50:43.505 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:43 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.5 2023-03-09T14:50:43.505 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:43 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.4 2023-03-09T14:50:43.505 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:43 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.1 2023-03-09T14:50:43.506 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:43 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.4 2023-03-09T14:50:43.506 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:43 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.1 2023-03-09T14:50:43.506 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:43 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.10 2023-03-09T14:50:43.506 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:43 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.10 2023-03-09T14:50:43.506 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:43 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.7 2023-03-09T14:50:43.506 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:43 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.7 2023-03-09T14:50:43.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15.0 s, 391.6 tps, lat 153.515 ms stddev 325.306 2023-03-09T14:50:44.322 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:44 smithi181 ceph-mon[94189]: pgmap v159: 129 pgs: 5 active+undersized, 8 active+undersized+degraded, 19 peering, 2 stale+active+clean, 95 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 19 MiB/s wr, 2.49k op/s; 605/24050 objects degraded (2.516%); 26 MiB/s, 6 objects/s recovering 2023-03-09T14:50:44.322 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:44 smithi181 ceph-mon[94189]: osdmap e113: 12 total, 10 up, 12 in 2023-03-09T14:50:44.322 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:44 smithi181 ceph-mon[94189]: Health check failed: Degraded data redundancy: 605/24050 objects degraded (2.516%), 8 pgs degraded (PG_DEGRADED) 2023-03-09T14:50:44.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:44 smithi122 ceph-mon[93869]: pgmap v159: 129 pgs: 5 active+undersized, 8 active+undersized+degraded, 19 peering, 2 stale+active+clean, 95 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 19 MiB/s wr, 2.49k op/s; 605/24050 objects degraded (2.516%); 26 MiB/s, 6 objects/s recovering 2023-03-09T14:50:44.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:44 smithi122 ceph-mon[93869]: osdmap e113: 12 total, 10 up, 12 in 2023-03-09T14:50:44.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:44 smithi122 ceph-mon[93869]: Health check failed: Degraded data redundancy: 605/24050 objects degraded (2.516%), 8 pgs degraded (PG_DEGRADED) 2023-03-09T14:50:44.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:44 smithi038 ceph-mon[92422]: pgmap v159: 129 pgs: 5 active+undersized, 8 active+undersized+degraded, 19 peering, 2 stale+active+clean, 95 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 17 MiB/s rd, 19 MiB/s wr, 2.49k op/s; 605/24050 objects degraded (2.516%); 26 MiB/s, 6 objects/s recovering 2023-03-09T14:50:44.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:44 smithi038 ceph-mon[92422]: osdmap e113: 12 total, 10 up, 12 in 2023-03-09T14:50:44.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:44 smithi038 ceph-mon[92422]: Health check failed: Degraded data redundancy: 605/24050 objects degraded (2.516%), 8 pgs degraded (PG_DEGRADED) 2023-03-09T14:50:44.732 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:44 smithi181 systemd[1]: ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.8.service: Main process exited, code=exited, status=134/n/a 2023-03-09T14:50:45.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:45 smithi181 ceph-mon[94189]: osdmap e114: 12 total, 10 up, 12 in 2023-03-09T14:50:45.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:45 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T14:50:45.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:45 smithi122 ceph-mon[93869]: osdmap e114: 12 total, 10 up, 12 in 2023-03-09T14:50:45.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:45 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T14:50:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:45 smithi038 ceph-mon[92422]: osdmap e114: 12 total, 10 up, 12 in 2023-03-09T14:50:45.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:45 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T14:50:45.981 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:45 smithi181 systemd[1]: ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.8.service: Failed with result 'exit-code'. 2023-03-09T14:50:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:46 smithi122 ceph-mon[93869]: pgmap v162: 129 pgs: 1 stale+peering, 9 active+undersized, 11 active+undersized+degraded, 18 peering, 10 stale+active+clean, 80 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 19 MiB/s wr, 2.10k op/s; 767/24106 objects degraded (3.182%) 2023-03-09T14:50:46.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:46 smithi181 ceph-mon[94189]: pgmap v162: 129 pgs: 1 stale+peering, 9 active+undersized, 11 active+undersized+degraded, 18 peering, 10 stale+active+clean, 80 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 19 MiB/s wr, 2.10k op/s; 767/24106 objects degraded (3.182%) 2023-03-09T14:50:46.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:46 smithi038 ceph-mon[92422]: pgmap v162: 129 pgs: 1 stale+peering, 9 active+undersized, 11 active+undersized+degraded, 18 peering, 10 stale+active+clean, 80 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 19 MiB/s wr, 2.10k op/s; 767/24106 objects degraded (3.182%) 2023-03-09T14:50:47.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:47 smithi122 ceph-mon[93869]: pgmap v163: 129 pgs: 1 stale+peering, 28 active+undersized, 2 undersized+degraded+peered, 24 active+undersized+degraded, 5 stale+active+clean, 69 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 6.5 MiB/s wr, 813 op/s; 3078/24146 objects degraded (12.747%) 2023-03-09T14:50:47.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:47 smithi122 ceph-mon[93869]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 5 pgs peering) 2023-03-09T14:50:47.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:47 smithi181 ceph-mon[94189]: pgmap v163: 129 pgs: 1 stale+peering, 28 active+undersized, 2 undersized+degraded+peered, 24 active+undersized+degraded, 5 stale+active+clean, 69 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 6.5 MiB/s wr, 813 op/s; 3078/24146 objects degraded (12.747%) 2023-03-09T14:50:47.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:47 smithi181 ceph-mon[94189]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 5 pgs peering) 2023-03-09T14:50:47.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:47 smithi038 ceph-mon[92422]: pgmap v163: 129 pgs: 1 stale+peering, 28 active+undersized, 2 undersized+degraded+peered, 24 active+undersized+degraded, 5 stale+active+clean, 69 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 6.5 MiB/s wr, 813 op/s; 3078/24146 objects degraded (12.747%) 2023-03-09T14:50:47.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:47 smithi038 ceph-mon[92422]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 5 pgs peering) 2023-03-09T14:50:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:48 smithi122 ceph-mon[93869]: Health check update: 2 osds down (OSD_DOWN) 2023-03-09T14:50:48.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:48 smithi181 ceph-mon[94189]: Health check update: 2 osds down (OSD_DOWN) 2023-03-09T14:50:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:48 smithi038 ceph-mon[92422]: Health check update: 2 osds down (OSD_DOWN) 2023-03-09T14:50:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20.0 s, 7.0 tps, lat 2397.998 ms stddev 23.207 2023-03-09T14:50:49.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:49 smithi122 ceph-mon[93869]: pgmap v164: 129 pgs: 1 undersized+peered, 31 active+undersized, 4 undersized+degraded+peered, 28 active+undersized+degraded, 65 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 5.1 MiB/s wr, 638 op/s; 3455/24146 objects degraded (14.309%) 2023-03-09T14:50:49.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:49 smithi181 ceph-mon[94189]: pgmap v164: 129 pgs: 1 undersized+peered, 31 active+undersized, 4 undersized+degraded+peered, 28 active+undersized+degraded, 65 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 5.1 MiB/s wr, 638 op/s; 3455/24146 objects degraded (14.309%) 2023-03-09T14:50:49.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:49 smithi038 ceph-mon[92422]: pgmap v164: 129 pgs: 1 undersized+peered, 31 active+undersized, 4 undersized+degraded+peered, 28 active+undersized+degraded, 65 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 5.1 MiB/s wr, 638 op/s; 3455/24146 objects degraded (14.309%) 2023-03-09T14:50:50.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:50 smithi122 ceph-mon[93869]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2023-03-09T14:50:50.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:50 smithi122 ceph-mon[93869]: Health check update: Degraded data redundancy: 3455/24146 objects degraded (14.309%), 32 pgs degraded (PG_DEGRADED) 2023-03-09T14:50:50.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:50 smithi181 ceph-mon[94189]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2023-03-09T14:50:50.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:50 smithi181 ceph-mon[94189]: Health check update: Degraded data redundancy: 3455/24146 objects degraded (14.309%), 32 pgs degraded (PG_DEGRADED) 2023-03-09T14:50:50.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:50 smithi038 ceph-mon[92422]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2023-03-09T14:50:50.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:50 smithi038 ceph-mon[92422]: Health check update: Degraded data redundancy: 3455/24146 objects degraded (14.309%), 32 pgs degraded (PG_DEGRADED) 2023-03-09T14:50:51.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:51 smithi122 ceph-mon[93869]: pgmap v165: 129 pgs: 1 undersized+peered, 31 active+undersized, 4 undersized+degraded+peered, 28 active+undersized+degraded, 65 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 896 KiB/s wr, 274 op/s; 3455/24146 objects degraded (14.309%) 2023-03-09T14:50:51.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:51 smithi181 ceph-mon[94189]: pgmap v165: 129 pgs: 1 undersized+peered, 31 active+undersized, 4 undersized+degraded+peered, 28 active+undersized+degraded, 65 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 896 KiB/s wr, 274 op/s; 3455/24146 objects degraded (14.309%) 2023-03-09T14:50:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:51 smithi038 ceph-mon[92422]: pgmap v165: 129 pgs: 1 undersized+peered, 31 active+undersized, 4 undersized+degraded+peered, 28 active+undersized+degraded, 65 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 896 KiB/s wr, 274 op/s; 3455/24146 objects degraded (14.309%) 2023-03-09T14:50:51.911 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:51 smithi038 systemd[1]: ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.3.service: Service RestartSec=10s expired, scheduling restart. 2023-03-09T14:50:51.912 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:51 smithi038 systemd[1]: ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.3.service: Scheduled restart job, restart counter is at 1. 2023-03-09T14:50:51.912 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:51 smithi038 systemd[1]: Stopped Ceph osd.3 for c6733fe2-be87-11ed-9af2-001a4aab830c. 2023-03-09T14:50:51.912 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:51 smithi038 systemd[1]: Starting Ceph osd.3 for c6733fe2-be87-11ed-9af2-001a4aab830c... 2023-03-09T14:50:52.751 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:52 smithi038 bash[147659]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-03-09T14:50:52.751 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:52 smithi038 bash[147659]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2023-03-09T14:50:52.751 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:52 smithi038 bash[147659]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2023-03-09T14:50:52.751 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:52 smithi038 bash[147659]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2023-03-09T14:50:52.751 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:52 smithi038 bash[147659]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-3/block 2023-03-09T14:50:52.752 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:52 smithi038 bash[147659]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-03-09T14:50:52.752 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:52 smithi038 bash[147659]: --> ceph-volume raw activate successful for osd ID: 3 2023-03-09T14:50:53.500 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:53 smithi038 podman[147925]: 2023-03-09T14:50:53.501 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:53 smithi038 bash[147925]: f3a22513b3bf14e47ac1fa99a4b59b40e1b558d5d2afe7ea1de62f5b5252d9e1 2023-03-09T14:50:53.501 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:53 smithi038 ceph-osd[147965]: -- 172.21.15.38:0/105841876 <== mon.1 v2:172.21.15.181:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x563b28a8e9c0 con 0x563b28a56c00 2023-03-09T14:50:53.501 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:53 smithi038 systemd[1]: Started Ceph osd.3 for c6733fe2-be87-11ed-9af2-001a4aab830c. 2023-03-09T14:50:53.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T14:50:53.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:53 smithi122 ceph-mon[93869]: pgmap v166: 129 pgs: 1 undersized+peered, 31 active+undersized, 4 undersized+degraded+peered, 28 active+undersized+degraded, 65 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1011 KiB/s rd, 703 KiB/s wr, 148 op/s; 3455/24146 objects degraded (14.309%) 2023-03-09T14:50:53.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:53 smithi181 ceph-mon[94189]: pgmap v166: 129 pgs: 1 undersized+peered, 31 active+undersized, 4 undersized+degraded+peered, 28 active+undersized+degraded, 65 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1011 KiB/s rd, 703 KiB/s wr, 148 op/s; 3455/24146 objects degraded (14.309%) 2023-03-09T14:50:54.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:53 smithi038 ceph-mon[92422]: pgmap v166: 129 pgs: 1 undersized+peered, 31 active+undersized, 4 undersized+degraded+peered, 28 active+undersized+degraded, 65 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1011 KiB/s rd, 703 KiB/s wr, 148 op/s; 3455/24146 objects degraded (14.309%) 2023-03-09T14:50:54.001 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:50:53 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 2023-03-09T14:50:53.762+0000 7fd11dd54640 -1 Falling back to public interface 2023-03-09T14:50:55.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:55 smithi122 ceph-mon[93869]: pgmap v167: 129 pgs: 1 undersized+peered, 31 active+undersized, 4 undersized+degraded+peered, 28 active+undersized+degraded, 65 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 917 KiB/s rd, 638 KiB/s wr, 134 op/s; 3455/24146 objects degraded (14.309%) 2023-03-09T14:50:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:55 smithi181 ceph-mon[94189]: pgmap v167: 129 pgs: 1 undersized+peered, 31 active+undersized, 4 undersized+degraded+peered, 28 active+undersized+degraded, 65 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 917 KiB/s rd, 638 KiB/s wr, 134 op/s; 3455/24146 objects degraded (14.309%) 2023-03-09T14:50:55.981 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:55 smithi181 systemd[1]: ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.8.service: Service RestartSec=10s expired, scheduling restart. 2023-03-09T14:50:55.982 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:55 smithi181 systemd[1]: ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.8.service: Scheduled restart job, restart counter is at 1. 2023-03-09T14:50:55.982 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:55 smithi181 systemd[1]: Stopped Ceph osd.8 for c6733fe2-be87-11ed-9af2-001a4aab830c. 2023-03-09T14:50:55.982 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:55 smithi181 systemd[1]: Starting Ceph osd.8 for c6733fe2-be87-11ed-9af2-001a4aab830c... 2023-03-09T14:50:56.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:55 smithi038 ceph-mon[92422]: pgmap v167: 129 pgs: 1 undersized+peered, 31 active+undersized, 4 undersized+degraded+peered, 28 active+undersized+degraded, 65 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 917 KiB/s rd, 638 KiB/s wr, 134 op/s; 3455/24146 objects degraded (14.309%) 2023-03-09T14:50:56.981 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:56 smithi181 bash[125723]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-8 2023-03-09T14:50:56.982 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:56 smithi181 bash[125723]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-8 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2023-03-09T14:50:56.982 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:56 smithi181 bash[125723]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2023-03-09T14:50:56.982 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:56 smithi181 bash[125723]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2023-03-09T14:50:56.982 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:56 smithi181 bash[125723]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-8/block 2023-03-09T14:50:56.982 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:56 smithi181 bash[125723]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-8 2023-03-09T14:50:56.983 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:56 smithi181 bash[125723]: --> ceph-volume raw activate successful for osd ID: 8 2023-03-09T14:50:57.818 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:57 smithi181 ceph-mon[94189]: pgmap v168: 129 pgs: 1 undersized+peered, 31 active+undersized, 4 undersized+degraded+peered, 28 active+undersized+degraded, 65 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 809 KiB/s rd, 563 KiB/s wr, 121 op/s; 3455/24146 objects degraded (14.309%) 2023-03-09T14:50:57.819 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:57 smithi181 podman[125980]: 2023-03-09T14:50:57.819 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:57 smithi181 ceph-osd[126020]: -- 172.21.15.181:0/2727304695 <== mon.1 v2:172.21.15.181:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56533415e9c0 con 0x565334126c00 2023-03-09T14:50:57.819 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:57 smithi181 bash[125980]: d29a872113c8b9efb614147f248179d6f6a61c52c9e298a8e1bbbce607d9f8b2 2023-03-09T14:50:57.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:57 smithi122 ceph-mon[93869]: pgmap v168: 129 pgs: 1 undersized+peered, 31 active+undersized, 4 undersized+degraded+peered, 28 active+undersized+degraded, 65 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 809 KiB/s rd, 563 KiB/s wr, 121 op/s; 3455/24146 objects degraded (14.309%) 2023-03-09T14:50:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:57 smithi038 ceph-mon[92422]: pgmap v168: 129 pgs: 1 undersized+peered, 31 active+undersized, 4 undersized+degraded+peered, 28 active+undersized+degraded, 65 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 809 KiB/s rd, 563 KiB/s wr, 121 op/s; 3455/24146 objects degraded (14.309%) 2023-03-09T14:50:58.231 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:57 smithi181 systemd[1]: Started Ceph osd.8 for c6733fe2-be87-11ed-9af2-001a4aab830c. 2023-03-09T14:50:58.732 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:50:58 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 2023-03-09T14:50:58.254+0000 7fbe379b3640 -1 Falling back to public interface 2023-03-09T14:50:58.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T14:50:59.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:50:59 smithi122 ceph-mon[93869]: pgmap v169: 129 pgs: 1 undersized+peered, 31 active+undersized, 4 undersized+degraded+peered, 28 active+undersized+degraded, 65 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s rd, 20 KiB/s wr, 5 op/s; 3455/24146 objects degraded (14.309%) 2023-03-09T14:50:59.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:50:59 smithi181 ceph-mon[94189]: pgmap v169: 129 pgs: 1 undersized+peered, 31 active+undersized, 4 undersized+degraded+peered, 28 active+undersized+degraded, 65 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s rd, 20 KiB/s wr, 5 op/s; 3455/24146 objects degraded (14.309%) 2023-03-09T14:51:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:50:59 smithi038 ceph-mon[92422]: pgmap v169: 129 pgs: 1 undersized+peered, 31 active+undersized, 4 undersized+degraded+peered, 28 active+undersized+degraded, 65 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s rd, 20 KiB/s wr, 5 op/s; 3455/24146 objects degraded (14.309%) 2023-03-09T14:51:01.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:00 smithi122 ceph-mon[93869]: Health check update: Degraded data redundancy: 3457/24162 objects degraded (14.308%), 32 pgs degraded (PG_DEGRADED) 2023-03-09T14:51:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:00 smithi181 ceph-mon[94189]: Health check update: Degraded data redundancy: 3457/24162 objects degraded (14.308%), 32 pgs degraded (PG_DEGRADED) 2023-03-09T14:51:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:00 smithi038 ceph-mon[92422]: Health check update: Degraded data redundancy: 3457/24162 objects degraded (14.308%), 32 pgs degraded (PG_DEGRADED) 2023-03-09T14:51:01.251 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:00 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 2023-03-09T14:51:00.871+0000 7fd11dd54640 -1 osd.3 110 log_to_monitors true 2023-03-09T14:51:02.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:01 smithi122 ceph-mon[93869]: pgmap v170: 129 pgs: 1 undersized+peered, 31 active+undersized, 4 undersized+degraded+peered, 28 active+undersized+degraded, 65 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 38 KiB/s wr, 12 op/s; 3457/24162 objects degraded (14.308%) 2023-03-09T14:51:02.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:01 smithi122 ceph-mon[93869]: from='osd.3 [v2:172.21.15.38:6810/1746992773,v1:172.21.15.38:6811/1746992773]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-03-09T14:51:02.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:01 smithi181 ceph-mon[94189]: pgmap v170: 129 pgs: 1 undersized+peered, 31 active+undersized, 4 undersized+degraded+peered, 28 active+undersized+degraded, 65 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 38 KiB/s wr, 12 op/s; 3457/24162 objects degraded (14.308%) 2023-03-09T14:51:02.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:01 smithi181 ceph-mon[94189]: from='osd.3 [v2:172.21.15.38:6810/1746992773,v1:172.21.15.38:6811/1746992773]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-03-09T14:51:02.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:01 smithi038 ceph-mon[92422]: pgmap v170: 129 pgs: 1 undersized+peered, 31 active+undersized, 4 undersized+degraded+peered, 28 active+undersized+degraded, 65 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 38 KiB/s wr, 12 op/s; 3457/24162 objects degraded (14.308%) 2023-03-09T14:51:02.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:01 smithi038 ceph-mon[92422]: from='osd.3 [v2:172.21.15.38:6810/1746992773,v1:172.21.15.38:6811/1746992773]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-03-09T14:51:02.251 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:01 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 2023-03-09T14:51:01.785+0000 7fd10b2c9700 -1 osd.3 110 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-09T14:51:03.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:02 smithi122 ceph-mon[93869]: from='osd.3 [v2:172.21.15.38:6810/1746992773,v1:172.21.15.38:6811/1746992773]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-03-09T14:51:03.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:02 smithi122 ceph-mon[93869]: osdmap e115: 12 total, 10 up, 12 in 2023-03-09T14:51:03.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:02 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T14:51:03.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:02 smithi122 ceph-mon[93869]: from='osd.3 [v2:172.21.15.38:6810/1746992773,v1:172.21.15.38:6811/1746992773]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2023-03-09T14:51:03.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:02 smithi181 ceph-mon[94189]: from='osd.3 [v2:172.21.15.38:6810/1746992773,v1:172.21.15.38:6811/1746992773]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-03-09T14:51:03.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:02 smithi181 ceph-mon[94189]: osdmap e115: 12 total, 10 up, 12 in 2023-03-09T14:51:03.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:02 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T14:51:03.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:02 smithi181 ceph-mon[94189]: from='osd.3 [v2:172.21.15.38:6810/1746992773,v1:172.21.15.38:6811/1746992773]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2023-03-09T14:51:03.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:02 smithi038 ceph-mon[92422]: from='osd.3 [v2:172.21.15.38:6810/1746992773,v1:172.21.15.38:6811/1746992773]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-03-09T14:51:03.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:02 smithi038 ceph-mon[92422]: osdmap e115: 12 total, 10 up, 12 in 2023-03-09T14:51:03.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:02 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T14:51:03.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:02 smithi038 ceph-mon[92422]: from='osd.3 [v2:172.21.15.38:6810/1746992773,v1:172.21.15.38:6811/1746992773]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2023-03-09T14:51:03.801 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T14:51:04.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:03 smithi122 ceph-mon[93869]: pgmap v172: 129 pgs: 1 undersized+peered, 31 active+undersized, 4 undersized+degraded+peered, 28 active+undersized+degraded, 65 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 69 KiB/s wr, 16 op/s; 3457/24162 objects degraded (14.308%) 2023-03-09T14:51:04.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:03 smithi122 ceph-mon[93869]: Health check update: 1 osds down (OSD_DOWN) 2023-03-09T14:51:04.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:03 smithi122 ceph-mon[93869]: osd.3 [v2:172.21.15.38:6810/1746992773,v1:172.21.15.38:6811/1746992773] boot 2023-03-09T14:51:04.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:03 smithi122 ceph-mon[93869]: osdmap e116: 12 total, 11 up, 12 in 2023-03-09T14:51:04.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:03 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-03-09T14:51:04.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:03 smithi122 ceph-mon[93869]: osdmap e117: 12 total, 11 up, 12 in 2023-03-09T14:51:04.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:03 smithi181 ceph-mon[94189]: pgmap v172: 129 pgs: 1 undersized+peered, 31 active+undersized, 4 undersized+degraded+peered, 28 active+undersized+degraded, 65 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 69 KiB/s wr, 16 op/s; 3457/24162 objects degraded (14.308%) 2023-03-09T14:51:04.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:03 smithi181 ceph-mon[94189]: Health check update: 1 osds down (OSD_DOWN) 2023-03-09T14:51:04.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:03 smithi181 ceph-mon[94189]: osd.3 [v2:172.21.15.38:6810/1746992773,v1:172.21.15.38:6811/1746992773] boot 2023-03-09T14:51:04.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:03 smithi181 ceph-mon[94189]: osdmap e116: 12 total, 11 up, 12 in 2023-03-09T14:51:04.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:03 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-03-09T14:51:04.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:03 smithi181 ceph-mon[94189]: osdmap e117: 12 total, 11 up, 12 in 2023-03-09T14:51:04.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:03 smithi038 ceph-mon[92422]: pgmap v172: 129 pgs: 1 undersized+peered, 31 active+undersized, 4 undersized+degraded+peered, 28 active+undersized+degraded, 65 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 69 KiB/s wr, 16 op/s; 3457/24162 objects degraded (14.308%) 2023-03-09T14:51:04.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:03 smithi038 ceph-mon[92422]: Health check update: 1 osds down (OSD_DOWN) 2023-03-09T14:51:04.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:03 smithi038 ceph-mon[92422]: osd.3 [v2:172.21.15.38:6810/1746992773,v1:172.21.15.38:6811/1746992773] boot 2023-03-09T14:51:04.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:03 smithi038 ceph-mon[92422]: osdmap e116: 12 total, 11 up, 12 in 2023-03-09T14:51:04.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:03 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-03-09T14:51:04.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:03 smithi038 ceph-mon[92422]: osdmap e117: 12 total, 11 up, 12 in 2023-03-09T14:51:04.481 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:04 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 2023-03-09T14:51:04.227+0000 7fbe379b3640 -1 osd.8 112 log_to_monitors true 2023-03-09T14:51:04.730 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 2023-03-09T14:51:04.327+0000 7f83a9408700 -1 osd.4 pg_epoch: 117 pg[4.13s0( v 117'848 (0'0,117'848] local-lis/les=109/110 n=215 ec=85/85 lis/c=109/109 les/c/f=110/110/0 sis=109) [4,1,2,9]p4(0) r=0 lpr=109 luod=117'847 crt=117'846 lcod 117'846 mlcod 117'846 active+clean] bad op order, already applied 71167 > this 71166 2023-03-09T14:51:04.730 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7f83a9408700 time 2023-03-09T14:51:04.327829+0000 2023-03-09T14:51:04.730 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op") 2023-03-09T14:51:04.731 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:51:04.731 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x55d7546a6c85] 2023-03-09T14:51:04.731 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 2: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55d75495f6e4] 2023-03-09T14:51:04.731 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55d754965d5e] 2023-03-09T14:51:04.731 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 4: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55d7547b5986] 2023-03-09T14:51:04.731 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 5: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55d754acb729] 2023-03-09T14:51:04.731 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55d7547caebf] 2023-03-09T14:51:04.732 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55d754e7cd85] 2023-03-09T14:51:04.732 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55d754e7ff84] 2023-03-09T14:51:04.732 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f83cc1681ca] 2023-03-09T14:51:04.732 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 10: clone() 2023-03-09T14:51:04.732 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: *** Caught signal (Aborted) ** 2023-03-09T14:51:04.732 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: in thread 7f83a9408700 thread_name:tp_osd_tp 2023-03-09T14:51:04.733 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 2023-03-09T14:51:04.330+0000 7f83a9408700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7f83a9408700 time 2023-03-09T14:51:04.327829+0000 2023-03-09T14:51:04.733 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op") 2023-03-09T14:51:04.733 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 2023-03-09T14:51:04.733 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:51:04.733 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x55d7546a6c85] 2023-03-09T14:51:04.733 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 2: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55d75495f6e4] 2023-03-09T14:51:04.734 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55d754965d5e] 2023-03-09T14:51:04.734 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 4: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55d7547b5986] 2023-03-09T14:51:04.734 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 5: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55d754acb729] 2023-03-09T14:51:04.734 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55d7547caebf] 2023-03-09T14:51:04.734 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55d754e7cd85] 2023-03-09T14:51:04.734 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55d754e7ff84] 2023-03-09T14:51:04.734 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f83cc1681ca] 2023-03-09T14:51:04.735 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 10: clone() 2023-03-09T14:51:04.735 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 2023-03-09T14:51:04.735 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:51:04.735 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f83cc172cf0] 2023-03-09T14:51:04.735 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 2: gsignal() 2023-03-09T14:51:04.735 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 3: abort() 2023-03-09T14:51:04.735 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x55d7546a6d45] 2023-03-09T14:51:04.735 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 5: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55d75495f6e4] 2023-03-09T14:51:04.736 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 6: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55d754965d5e] 2023-03-09T14:51:04.736 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55d7547b5986] 2023-03-09T14:51:04.736 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 8: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55d754acb729] 2023-03-09T14:51:04.736 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55d7547caebf] 2023-03-09T14:51:04.737 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55d754e7cd85] 2023-03-09T14:51:04.737 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55d754e7ff84] 2023-03-09T14:51:04.737 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f83cc1681ca] 2023-03-09T14:51:04.737 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 13: clone() 2023-03-09T14:51:04.737 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 2023-03-09T14:51:04.334+0000 7f83a9408700 -1 *** Caught signal (Aborted) ** 2023-03-09T14:51:04.737 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: in thread 7f83a9408700 thread_name:tp_osd_tp 2023-03-09T14:51:04.738 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 2023-03-09T14:51:04.738 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:51:04.738 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f83cc172cf0] 2023-03-09T14:51:04.738 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 2: gsignal() 2023-03-09T14:51:04.738 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 3: abort() 2023-03-09T14:51:04.738 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x55d7546a6d45] 2023-03-09T14:51:04.738 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 5: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55d75495f6e4] 2023-03-09T14:51:04.738 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 6: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55d754965d5e] 2023-03-09T14:51:04.739 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55d7547b5986] 2023-03-09T14:51:04.739 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 8: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55d754acb729] 2023-03-09T14:51:04.739 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55d7547caebf] 2023-03-09T14:51:04.739 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55d754e7cd85] 2023-03-09T14:51:04.739 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55d754e7ff84] 2023-03-09T14:51:04.739 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f83cc1681ca] 2023-03-09T14:51:04.739 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 13: clone() 2023-03-09T14:51:04.739 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-03-09T14:51:04.740 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 2023-03-09T14:51:04.740 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: -3158> 2023-03-09T14:51:04.327+0000 7f83a9408700 -1 osd.4 pg_epoch: 117 pg[4.13s0( v 117'848 (0'0,117'848] local-lis/les=109/110 n=215 ec=85/85 lis/c=109/109 les/c/f=110/110/0 sis=109) [4,1,2,9]p4(0) r=0 lpr=109 luod=117'847 crt=117'846 lcod 117'846 mlcod 117'846 active+clean] bad op order, already applied 71167 > this 71166 2023-03-09T14:51:04.740 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: -2663> 2023-03-09T14:51:04.330+0000 7f83a9408700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7f83a9408700 time 2023-03-09T14:51:04.327829+0000 2023-03-09T14:51:04.740 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op") 2023-03-09T14:51:04.740 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 2023-03-09T14:51:04.740 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:51:04.740 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x55d7546a6c85] 2023-03-09T14:51:04.741 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 2: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55d75495f6e4] 2023-03-09T14:51:04.741 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55d754965d5e] 2023-03-09T14:51:04.741 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 4: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55d7547b5986] 2023-03-09T14:51:04.741 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 5: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55d754acb729] 2023-03-09T14:51:04.741 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55d7547caebf] 2023-03-09T14:51:04.741 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55d754e7cd85] 2023-03-09T14:51:04.741 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55d754e7ff84] 2023-03-09T14:51:04.742 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f83cc1681ca] 2023-03-09T14:51:04.742 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 10: clone() 2023-03-09T14:51:04.742 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 2023-03-09T14:51:04.742 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: -984> 2023-03-09T14:51:04.334+0000 7f83a9408700 -1 *** Caught signal (Aborted) ** 2023-03-09T14:51:04.742 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: in thread 7f83a9408700 thread_name:tp_osd_tp 2023-03-09T14:51:04.742 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 2023-03-09T14:51:04.742 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:51:04.742 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f83cc172cf0] 2023-03-09T14:51:04.743 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 2: gsignal() 2023-03-09T14:51:04.743 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 3: abort() 2023-03-09T14:51:04.743 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x55d7546a6d45] 2023-03-09T14:51:04.743 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 5: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55d75495f6e4] 2023-03-09T14:51:04.743 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 6: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55d754965d5e] 2023-03-09T14:51:04.743 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55d7547b5986] 2023-03-09T14:51:04.743 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 8: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55d754acb729] 2023-03-09T14:51:04.743 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55d7547caebf] 2023-03-09T14:51:04.744 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55d754e7cd85] 2023-03-09T14:51:04.744 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55d754e7ff84] 2023-03-09T14:51:04.744 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f83cc1681ca] 2023-03-09T14:51:04.744 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 13: clone() 2023-03-09T14:51:04.744 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-03-09T14:51:04.744 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 2023-03-09T14:51:04.744 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: -4159> 2023-03-09T14:51:04.327+0000 7f83a9408700 -1 osd.4 pg_epoch: 117 pg[4.13s0( v 117'848 (0'0,117'848] local-lis/les=109/110 n=215 ec=85/85 lis/c=109/109 les/c/f=110/110/0 sis=109) [4,1,2,9]p4(0) r=0 lpr=109 luod=117'847 crt=117'846 lcod 117'846 mlcod 117'846 active+clean] bad op order, already applied 71167 > this 71166 2023-03-09T14:51:04.745 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: -3664> 2023-03-09T14:51:04.330+0000 7f83a9408700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7f83a9408700 time 2023-03-09T14:51:04.327829+0000 2023-03-09T14:51:04.745 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op") 2023-03-09T14:51:04.745 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 2023-03-09T14:51:04.745 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:51:04.745 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x55d7546a6c85] 2023-03-09T14:51:04.745 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 2: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55d75495f6e4] 2023-03-09T14:51:04.745 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55d754965d5e] 2023-03-09T14:51:04.746 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 4: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55d7547b5986] 2023-03-09T14:51:04.746 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 5: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55d754acb729] 2023-03-09T14:51:04.746 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55d7547caebf] 2023-03-09T14:51:04.746 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55d754e7cd85] 2023-03-09T14:51:04.746 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55d754e7ff84] 2023-03-09T14:51:04.746 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f83cc1681ca] 2023-03-09T14:51:04.747 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 10: clone() 2023-03-09T14:51:04.747 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 2023-03-09T14:51:04.747 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: -1985> 2023-03-09T14:51:04.334+0000 7f83a9408700 -1 *** Caught signal (Aborted) ** 2023-03-09T14:51:04.747 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: in thread 7f83a9408700 thread_name:tp_osd_tp 2023-03-09T14:51:04.747 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 2023-03-09T14:51:04.747 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:51:04.747 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f83cc172cf0] 2023-03-09T14:51:04.748 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 2: gsignal() 2023-03-09T14:51:04.748 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 3: abort() 2023-03-09T14:51:04.748 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x55d7546a6d45] 2023-03-09T14:51:04.748 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 5: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55d75495f6e4] 2023-03-09T14:51:04.748 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 6: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55d754965d5e] 2023-03-09T14:51:04.748 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55d7547b5986] 2023-03-09T14:51:04.748 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 8: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55d754acb729] 2023-03-09T14:51:04.748 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55d7547caebf] 2023-03-09T14:51:04.749 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55d754e7cd85] 2023-03-09T14:51:04.749 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55d754e7ff84] 2023-03-09T14:51:04.749 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f83cc1681ca] 2023-03-09T14:51:04.749 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 13: clone() 2023-03-09T14:51:04.749 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-03-09T14:51:04.749 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[104307]: 2023-03-09T14:51:05.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-mon[93869]: from='osd.8 [v2:172.21.15.181:6816/1026160644,v1:172.21.15.181:6817/1026160644]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-03-09T14:51:05.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-mon[93869]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-03-09T14:51:05.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-mon[93869]: osd.4 reported immediately failed by osd.11 2023-03-09T14:51:05.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-mon[93869]: osd.4 failed (root=default,host=smithi122) (connection refused reported by osd.11) 2023-03-09T14:51:05.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-mon[93869]: osd.4 reported immediately failed by osd.2 2023-03-09T14:51:05.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-mon[93869]: osd.4 reported immediately failed by osd.0 2023-03-09T14:51:05.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-mon[93869]: osd.4 reported immediately failed by osd.11 2023-03-09T14:51:05.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-mon[93869]: osd.4 reported immediately failed by osd.5 2023-03-09T14:51:05.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-mon[93869]: osd.4 reported immediately failed by osd.2 2023-03-09T14:51:05.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-mon[93869]: osd.4 reported immediately failed by osd.5 2023-03-09T14:51:05.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-mon[93869]: osd.4 reported immediately failed by osd.2 2023-03-09T14:51:05.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-mon[93869]: osd.4 reported immediately failed by osd.5 2023-03-09T14:51:05.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-mon[93869]: osd.4 reported immediately failed by osd.10 2023-03-09T14:51:05.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-mon[93869]: osd.4 reported immediately failed by osd.1 2023-03-09T14:51:05.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-mon[93869]: osd.4 reported immediately failed by osd.10 2023-03-09T14:51:05.231 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-mon[93869]: osd.4 reported immediately failed by osd.1 2023-03-09T14:51:05.231 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-mon[93869]: osd.4 reported immediately failed by osd.10 2023-03-09T14:51:05.231 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-mon[93869]: osd.4 reported immediately failed by osd.7 2023-03-09T14:51:05.231 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-mon[93869]: osd.4 reported immediately failed by osd.6 2023-03-09T14:51:05.231 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-mon[93869]: osd.4 reported immediately failed by osd.7 2023-03-09T14:51:05.231 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-mon[93869]: osd.4 reported immediately failed by osd.6 2023-03-09T14:51:05.232 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-mon[93869]: osd.4 reported immediately failed by osd.7 2023-03-09T14:51:05.232 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-mon[93869]: osd.4 reported immediately failed by osd.3 2023-03-09T14:51:05.232 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-mon[93869]: osd.4 reported immediately failed by osd.6 2023-03-09T14:51:05.232 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-mon[93869]: osd.4 reported immediately failed by osd.3 2023-03-09T14:51:05.232 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-mon[93869]: osd.4 reported immediately failed by osd.9 2023-03-09T14:51:05.232 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-mon[93869]: osd.4 reported immediately failed by osd.3 2023-03-09T14:51:05.232 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-mon[93869]: osd.4 reported immediately failed by osd.9 2023-03-09T14:51:05.232 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-mon[93869]: osd.4 reported immediately failed by osd.9 2023-03-09T14:51:05.233 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:04 smithi122 ceph-mon[93869]: osd.4 reported immediately failed by osd.0 2023-03-09T14:51:05.233 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:04 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 2023-03-09T14:51:04.824+0000 7fbe23f26700 -1 osd.8 112 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-09T14:51:05.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:04 smithi181 ceph-mon[94189]: from='osd.8 [v2:172.21.15.181:6816/1026160644,v1:172.21.15.181:6817/1026160644]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-03-09T14:51:05.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:04 smithi181 ceph-mon[94189]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-03-09T14:51:05.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:04 smithi181 ceph-mon[94189]: osd.4 reported immediately failed by osd.11 2023-03-09T14:51:05.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:04 smithi181 ceph-mon[94189]: osd.4 failed (root=default,host=smithi122) (connection refused reported by osd.11) 2023-03-09T14:51:05.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:04 smithi181 ceph-mon[94189]: osd.4 reported immediately failed by osd.2 2023-03-09T14:51:05.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:04 smithi181 ceph-mon[94189]: osd.4 reported immediately failed by osd.0 2023-03-09T14:51:05.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:04 smithi181 ceph-mon[94189]: osd.4 reported immediately failed by osd.11 2023-03-09T14:51:05.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:04 smithi181 ceph-mon[94189]: osd.4 reported immediately failed by osd.5 2023-03-09T14:51:05.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:04 smithi181 ceph-mon[94189]: osd.4 reported immediately failed by osd.2 2023-03-09T14:51:05.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:04 smithi181 ceph-mon[94189]: osd.4 reported immediately failed by osd.5 2023-03-09T14:51:05.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:04 smithi181 ceph-mon[94189]: osd.4 reported immediately failed by osd.2 2023-03-09T14:51:05.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:04 smithi181 ceph-mon[94189]: osd.4 reported immediately failed by osd.5 2023-03-09T14:51:05.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:04 smithi181 ceph-mon[94189]: osd.4 reported immediately failed by osd.10 2023-03-09T14:51:05.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:04 smithi181 ceph-mon[94189]: osd.4 reported immediately failed by osd.1 2023-03-09T14:51:05.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:04 smithi181 ceph-mon[94189]: osd.4 reported immediately failed by osd.10 2023-03-09T14:51:05.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:04 smithi181 ceph-mon[94189]: osd.4 reported immediately failed by osd.1 2023-03-09T14:51:05.236 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:04 smithi181 ceph-mon[94189]: osd.4 reported immediately failed by osd.10 2023-03-09T14:51:05.236 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:04 smithi181 ceph-mon[94189]: osd.4 reported immediately failed by osd.7 2023-03-09T14:51:05.236 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:04 smithi181 ceph-mon[94189]: osd.4 reported immediately failed by osd.6 2023-03-09T14:51:05.236 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:04 smithi181 ceph-mon[94189]: osd.4 reported immediately failed by osd.7 2023-03-09T14:51:05.236 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:04 smithi181 ceph-mon[94189]: osd.4 reported immediately failed by osd.6 2023-03-09T14:51:05.236 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:04 smithi181 ceph-mon[94189]: osd.4 reported immediately failed by osd.7 2023-03-09T14:51:05.236 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:04 smithi181 ceph-mon[94189]: osd.4 reported immediately failed by osd.3 2023-03-09T14:51:05.236 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:04 smithi181 ceph-mon[94189]: osd.4 reported immediately failed by osd.6 2023-03-09T14:51:05.237 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:04 smithi181 ceph-mon[94189]: osd.4 reported immediately failed by osd.3 2023-03-09T14:51:05.237 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:04 smithi181 ceph-mon[94189]: osd.4 reported immediately failed by osd.9 2023-03-09T14:51:05.237 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:04 smithi181 ceph-mon[94189]: osd.4 reported immediately failed by osd.3 2023-03-09T14:51:05.237 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:04 smithi181 ceph-mon[94189]: osd.4 reported immediately failed by osd.9 2023-03-09T14:51:05.237 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:04 smithi181 ceph-mon[94189]: osd.4 reported immediately failed by osd.9 2023-03-09T14:51:05.237 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:04 smithi181 ceph-mon[94189]: osd.4 reported immediately failed by osd.0 2023-03-09T14:51:05.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:04 smithi038 ceph-mon[92422]: from='osd.8 [v2:172.21.15.181:6816/1026160644,v1:172.21.15.181:6817/1026160644]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-03-09T14:51:05.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:04 smithi038 ceph-mon[92422]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-03-09T14:51:05.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:04 smithi038 ceph-mon[92422]: osd.4 reported immediately failed by osd.11 2023-03-09T14:51:05.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:04 smithi038 ceph-mon[92422]: osd.4 failed (root=default,host=smithi122) (connection refused reported by osd.11) 2023-03-09T14:51:05.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:04 smithi038 ceph-mon[92422]: osd.4 reported immediately failed by osd.2 2023-03-09T14:51:05.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:04 smithi038 ceph-mon[92422]: osd.4 reported immediately failed by osd.0 2023-03-09T14:51:05.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:04 smithi038 ceph-mon[92422]: osd.4 reported immediately failed by osd.11 2023-03-09T14:51:05.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:04 smithi038 ceph-mon[92422]: osd.4 reported immediately failed by osd.5 2023-03-09T14:51:05.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:04 smithi038 ceph-mon[92422]: osd.4 reported immediately failed by osd.2 2023-03-09T14:51:05.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:04 smithi038 ceph-mon[92422]: osd.4 reported immediately failed by osd.5 2023-03-09T14:51:05.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:04 smithi038 ceph-mon[92422]: osd.4 reported immediately failed by osd.2 2023-03-09T14:51:05.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:04 smithi038 ceph-mon[92422]: osd.4 reported immediately failed by osd.5 2023-03-09T14:51:05.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:04 smithi038 ceph-mon[92422]: osd.4 reported immediately failed by osd.10 2023-03-09T14:51:05.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:04 smithi038 ceph-mon[92422]: osd.4 reported immediately failed by osd.1 2023-03-09T14:51:05.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:04 smithi038 ceph-mon[92422]: osd.4 reported immediately failed by osd.10 2023-03-09T14:51:05.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:04 smithi038 ceph-mon[92422]: osd.4 reported immediately failed by osd.1 2023-03-09T14:51:05.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:04 smithi038 ceph-mon[92422]: osd.4 reported immediately failed by osd.10 2023-03-09T14:51:05.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:04 smithi038 ceph-mon[92422]: osd.4 reported immediately failed by osd.7 2023-03-09T14:51:05.254 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:04 smithi038 ceph-mon[92422]: osd.4 reported immediately failed by osd.6 2023-03-09T14:51:05.254 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:04 smithi038 ceph-mon[92422]: osd.4 reported immediately failed by osd.7 2023-03-09T14:51:05.254 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:04 smithi038 ceph-mon[92422]: osd.4 reported immediately failed by osd.6 2023-03-09T14:51:05.254 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:04 smithi038 ceph-mon[92422]: osd.4 reported immediately failed by osd.7 2023-03-09T14:51:05.254 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:04 smithi038 ceph-mon[92422]: osd.4 reported immediately failed by osd.3 2023-03-09T14:51:05.254 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:04 smithi038 ceph-mon[92422]: osd.4 reported immediately failed by osd.6 2023-03-09T14:51:05.254 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:04 smithi038 ceph-mon[92422]: osd.4 reported immediately failed by osd.3 2023-03-09T14:51:05.254 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:04 smithi038 ceph-mon[92422]: osd.4 reported immediately failed by osd.9 2023-03-09T14:51:05.255 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:04 smithi038 ceph-mon[92422]: osd.4 reported immediately failed by osd.3 2023-03-09T14:51:05.255 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:04 smithi038 ceph-mon[92422]: osd.4 reported immediately failed by osd.9 2023-03-09T14:51:05.255 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:04 smithi038 ceph-mon[92422]: osd.4 reported immediately failed by osd.9 2023-03-09T14:51:05.255 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:04 smithi038 ceph-mon[92422]: osd.4 reported immediately failed by osd.0 2023-03-09T14:51:06.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:05 smithi122 ceph-mon[93869]: pgmap v175: 129 pgs: 5 peering, 1 undersized+peered, 29 active+undersized, 2 undersized+degraded+peered, 27 active+undersized+degraded, 65 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 142 KiB/s wr, 23 op/s; 3139/24162 objects degraded (12.991%) 2023-03-09T14:51:06.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:05 smithi122 ceph-mon[93869]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-03-09T14:51:06.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:05 smithi122 ceph-mon[93869]: osdmap e118: 12 total, 10 up, 12 in 2023-03-09T14:51:06.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:05 smithi122 ceph-mon[93869]: from='osd.8 [v2:172.21.15.181:6816/1026160644,v1:172.21.15.181:6817/1026160644]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2023-03-09T14:51:06.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:05 smithi122 ceph-mon[93869]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2023-03-09T14:51:06.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:05 smithi181 ceph-mon[94189]: pgmap v175: 129 pgs: 5 peering, 1 undersized+peered, 29 active+undersized, 2 undersized+degraded+peered, 27 active+undersized+degraded, 65 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 142 KiB/s wr, 23 op/s; 3139/24162 objects degraded (12.991%) 2023-03-09T14:51:06.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:05 smithi181 ceph-mon[94189]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-03-09T14:51:06.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:05 smithi181 ceph-mon[94189]: osdmap e118: 12 total, 10 up, 12 in 2023-03-09T14:51:06.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:05 smithi181 ceph-mon[94189]: from='osd.8 [v2:172.21.15.181:6816/1026160644,v1:172.21.15.181:6817/1026160644]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2023-03-09T14:51:06.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:05 smithi181 ceph-mon[94189]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2023-03-09T14:51:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:05 smithi038 ceph-mon[92422]: pgmap v175: 129 pgs: 5 peering, 1 undersized+peered, 29 active+undersized, 2 undersized+degraded+peered, 27 active+undersized+degraded, 65 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 142 KiB/s wr, 23 op/s; 3139/24162 objects degraded (12.991%) 2023-03-09T14:51:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:05 smithi038 ceph-mon[92422]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-03-09T14:51:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:05 smithi038 ceph-mon[92422]: osdmap e118: 12 total, 10 up, 12 in 2023-03-09T14:51:06.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:05 smithi038 ceph-mon[92422]: from='osd.8 [v2:172.21.15.181:6816/1026160644,v1:172.21.15.181:6817/1026160644]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2023-03-09T14:51:06.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:05 smithi038 ceph-mon[92422]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2023-03-09T14:51:07.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:06 smithi122 ceph-mon[93869]: osd.8 [v2:172.21.15.181:6816/1026160644,v1:172.21.15.181:6817/1026160644] boot 2023-03-09T14:51:07.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:06 smithi122 ceph-mon[93869]: osdmap e119: 12 total, 11 up, 12 in 2023-03-09T14:51:07.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:06 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-03-09T14:51:07.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:06 smithi122 ceph-mon[93869]: osdmap e120: 12 total, 11 up, 12 in 2023-03-09T14:51:07.229 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:06 smithi122 systemd[1]: ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.4.service: Main process exited, code=exited, status=134/n/a 2023-03-09T14:51:07.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:06 smithi181 ceph-mon[94189]: osd.8 [v2:172.21.15.181:6816/1026160644,v1:172.21.15.181:6817/1026160644] boot 2023-03-09T14:51:07.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:06 smithi181 ceph-mon[94189]: osdmap e119: 12 total, 11 up, 12 in 2023-03-09T14:51:07.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:06 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-03-09T14:51:07.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:06 smithi181 ceph-mon[94189]: osdmap e120: 12 total, 11 up, 12 in 2023-03-09T14:51:07.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:06 smithi038 ceph-mon[92422]: osd.8 [v2:172.21.15.181:6816/1026160644,v1:172.21.15.181:6817/1026160644] boot 2023-03-09T14:51:07.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:06 smithi038 ceph-mon[92422]: osdmap e119: 12 total, 11 up, 12 in 2023-03-09T14:51:07.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:06 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-03-09T14:51:07.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:06 smithi038 ceph-mon[92422]: osdmap e120: 12 total, 11 up, 12 in 2023-03-09T14:51:08.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:07 smithi122 ceph-mon[93869]: pgmap v178: 129 pgs: 4 stale+active+undersized+degraded, 2 stale+active+undersized, 1 stale+active+clean, 25 peering, 1 undersized+peered, 20 active+undersized, 18 active+undersized+degraded, 58 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 8.4 MiB/s wr, 1.02k op/s; 2488/26946 objects degraded (9.233%); 15 MiB/s, 3 objects/s recovering 2023-03-09T14:51:08.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:07 smithi122 ceph-mon[93869]: Health check update: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2023-03-09T14:51:08.095 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:07 smithi122 ceph-mon[93869]: Health check update: Degraded data redundancy: 2488/26946 objects degraded (9.233%), 22 pgs degraded (PG_DEGRADED) 2023-03-09T14:51:08.096 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:07 smithi122 ceph-mon[93869]: osdmap e121: 12 total, 11 up, 12 in 2023-03-09T14:51:08.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:07 smithi181 ceph-mon[94189]: pgmap v178: 129 pgs: 4 stale+active+undersized+degraded, 2 stale+active+undersized, 1 stale+active+clean, 25 peering, 1 undersized+peered, 20 active+undersized, 18 active+undersized+degraded, 58 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 8.4 MiB/s wr, 1.02k op/s; 2488/26946 objects degraded (9.233%); 15 MiB/s, 3 objects/s recovering 2023-03-09T14:51:08.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:07 smithi181 ceph-mon[94189]: Health check update: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2023-03-09T14:51:08.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:07 smithi181 ceph-mon[94189]: Health check update: Degraded data redundancy: 2488/26946 objects degraded (9.233%), 22 pgs degraded (PG_DEGRADED) 2023-03-09T14:51:08.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:07 smithi181 ceph-mon[94189]: osdmap e121: 12 total, 11 up, 12 in 2023-03-09T14:51:08.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:07 smithi038 ceph-mon[92422]: pgmap v178: 129 pgs: 4 stale+active+undersized+degraded, 2 stale+active+undersized, 1 stale+active+clean, 25 peering, 1 undersized+peered, 20 active+undersized, 18 active+undersized+degraded, 58 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 8.4 MiB/s wr, 1.02k op/s; 2488/26946 objects degraded (9.233%); 15 MiB/s, 3 objects/s recovering 2023-03-09T14:51:08.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:07 smithi038 ceph-mon[92422]: Health check update: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2023-03-09T14:51:08.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:07 smithi038 ceph-mon[92422]: Health check update: Degraded data redundancy: 2488/26946 objects degraded (9.233%), 22 pgs degraded (PG_DEGRADED) 2023-03-09T14:51:08.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:07 smithi038 ceph-mon[92422]: osdmap e121: 12 total, 11 up, 12 in 2023-03-09T14:51:08.477 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:08 smithi122 systemd[1]: ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.4.service: Failed with result 'exit-code'. 2023-03-09T14:51:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40.0 s, 4.4 tps, lat 22726.040 ms stddev 1274.956 2023-03-09T14:51:10.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:09 smithi122 ceph-mon[93869]: pgmap v181: 129 pgs: 1 active+recovery_wait+degraded, 1 remapped+peering, 1 stale+active+undersized, 20 peering, 1 undersized+peered, 15 active+undersized, 18 active+undersized+degraded, 72 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 18 MiB/s wr, 2.12k op/s; 2649/30110 objects degraded (8.798%); 52 MiB/s, 13 keys/s, 14 objects/s recovering 2023-03-09T14:51:10.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:09 smithi181 ceph-mon[94189]: pgmap v181: 129 pgs: 1 active+recovery_wait+degraded, 1 remapped+peering, 1 stale+active+undersized, 20 peering, 1 undersized+peered, 15 active+undersized, 18 active+undersized+degraded, 72 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 18 MiB/s wr, 2.12k op/s; 2649/30110 objects degraded (8.798%); 52 MiB/s, 13 keys/s, 14 objects/s recovering 2023-03-09T14:51:10.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:09 smithi038 ceph-mon[92422]: pgmap v181: 129 pgs: 1 active+recovery_wait+degraded, 1 remapped+peering, 1 stale+active+undersized, 20 peering, 1 undersized+peered, 15 active+undersized, 18 active+undersized+degraded, 72 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 18 MiB/s wr, 2.12k op/s; 2649/30110 objects degraded (8.798%); 52 MiB/s, 13 keys/s, 14 objects/s recovering 2023-03-09T14:51:11.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:10 smithi122 ceph-mon[93869]: osdmap e122: 12 total, 11 up, 12 in 2023-03-09T14:51:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:10 smithi181 ceph-mon[94189]: osdmap e122: 12 total, 11 up, 12 in 2023-03-09T14:51:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:10 smithi038 ceph-mon[92422]: osdmap e122: 12 total, 11 up, 12 in 2023-03-09T14:51:12.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:11 smithi122 ceph-mon[93869]: pgmap v183: 129 pgs: 2 active+recovery_wait+degraded, 1 remapped+peering, 2 peering, 18 active+undersized, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 18 active+undersized+degraded, 86 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 9.1 MiB/s wr, 1.12k op/s; 2492/30110 objects degraded (8.276%); 257 MiB/s, 23 keys/s, 65 objects/s recovering 2023-03-09T14:51:12.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:11 smithi122 ceph-mon[93869]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2023-03-09T14:51:12.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:11 smithi122 ceph-mon[93869]: osdmap e123: 12 total, 11 up, 12 in 2023-03-09T14:51:12.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:11 smithi181 ceph-mon[94189]: pgmap v183: 129 pgs: 2 active+recovery_wait+degraded, 1 remapped+peering, 2 peering, 18 active+undersized, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 18 active+undersized+degraded, 86 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 9.1 MiB/s wr, 1.12k op/s; 2492/30110 objects degraded (8.276%); 257 MiB/s, 23 keys/s, 65 objects/s recovering 2023-03-09T14:51:12.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:11 smithi181 ceph-mon[94189]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2023-03-09T14:51:12.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:11 smithi181 ceph-mon[94189]: osdmap e123: 12 total, 11 up, 12 in 2023-03-09T14:51:12.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:11 smithi038 ceph-mon[92422]: pgmap v183: 129 pgs: 2 active+recovery_wait+degraded, 1 remapped+peering, 2 peering, 18 active+undersized, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 18 active+undersized+degraded, 86 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 9.1 MiB/s wr, 1.12k op/s; 2492/30110 objects degraded (8.276%); 257 MiB/s, 23 keys/s, 65 objects/s recovering 2023-03-09T14:51:12.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:11 smithi038 ceph-mon[92422]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2023-03-09T14:51:12.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:11 smithi038 ceph-mon[92422]: osdmap e123: 12 total, 11 up, 12 in 2023-03-09T14:51:12.613 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-03-09T14:51:12.767 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.764+0000 7fd115f2d700 1 -- 172.21.15.38:0/2785054361 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd11010c8c0 msgr2=0x7fd11010eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:51:12.767 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.764+0000 7fd115f2d700 1 --2- 172.21.15.38:0/2785054361 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd11010c8c0 0x7fd11010eca0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fd10800b0a0 tx=0x7fd10800b3b0 comp rx=0 tx=0).stop 2023-03-09T14:51:12.767 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.764+0000 7fd115f2d700 1 -- 172.21.15.38:0/2785054361 shutdown_connections 2023-03-09T14:51:12.768 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.764+0000 7fd115f2d700 1 --2- 172.21.15.38:0/2785054361 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd11010c8c0 0x7fd11010eca0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:51:12.768 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.764+0000 7fd115f2d700 1 --2- 172.21.15.38:0/2785054361 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd110067ac0 0x7fd110067e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:51:12.768 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.764+0000 7fd115f2d700 1 --2- 172.21.15.38:0/2785054361 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd1100683d0 0x7fd11010c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:51:12.768 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.764+0000 7fd115f2d700 1 -- 172.21.15.38:0/2785054361 >> 172.21.15.38:0/2785054361 conn(0x7fd110075c60 msgr2=0x7fd110076060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:51:12.768 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.764+0000 7fd115f2d700 1 -- 172.21.15.38:0/2785054361 shutdown_connections 2023-03-09T14:51:12.768 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.764+0000 7fd115f2d700 1 -- 172.21.15.38:0/2785054361 wait complete. 2023-03-09T14:51:12.770 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.765+0000 7fd115f2d700 1 Processor -- start 2023-03-09T14:51:12.770 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.765+0000 7fd115f2d700 1 -- start start 2023-03-09T14:51:12.770 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.766+0000 7fd115f2d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd110067ac0 0x7fd110198360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:51:12.771 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.766+0000 7fd115f2d700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd1100683d0 0x7fd1101988a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:51:12.771 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.766+0000 7fd115f2d700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd11010c8c0 0x7fd11019cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:51:12.771 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.766+0000 7fd115f2d700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd11019d340 con 0x7fd110067ac0 2023-03-09T14:51:12.771 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.766+0000 7fd115f2d700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd11019d4b0 con 0x7fd1100683d0 2023-03-09T14:51:12.771 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.766+0000 7fd115f2d700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd11019d670 con 0x7fd11010c8c0 2023-03-09T14:51:12.771 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.766+0000 7fd10f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd110067ac0 0x7fd110198360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:51:12.771 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.766+0000 7fd10f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd110067ac0 0x7fd110198360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:40458/0 (socket says 172.21.15.38:40458) 2023-03-09T14:51:12.771 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.766+0000 7fd10effd700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd1100683d0 0x7fd1101988a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:51:12.772 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.766+0000 7fd10f7fe700 1 -- 172.21.15.38:0/3236840731 learned_addr learned my addr 172.21.15.38:0/3236840731 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:51:12.772 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.766+0000 7fd10ffff700 1 --2- 172.21.15.38:0/3236840731 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd11010c8c0 0x7fd11019cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:51:12.772 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.766+0000 7fd10f7fe700 1 -- 172.21.15.38:0/3236840731 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd11010c8c0 msgr2=0x7fd11019cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:51:12.772 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.766+0000 7fd10f7fe700 1 --2- 172.21.15.38:0/3236840731 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd11010c8c0 0x7fd11019cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:51:12.772 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.766+0000 7fd10f7fe700 1 -- 172.21.15.38:0/3236840731 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd1100683d0 msgr2=0x7fd1101988a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:51:12.772 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.766+0000 7fd10f7fe700 1 --2- 172.21.15.38:0/3236840731 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd1100683d0 0x7fd1101988a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:51:12.772 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.766+0000 7fd10f7fe700 1 -- 172.21.15.38:0/3236840731 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd108009d00 con 0x7fd110067ac0 2023-03-09T14:51:12.773 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.766+0000 7fd10f7fe700 1 --2- 172.21.15.38:0/3236840731 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd110067ac0 0x7fd110198360 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fd10000e9a0 tx=0x7fd10000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:51:12.773 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.766+0000 7fd10cff9700 1 -- 172.21.15.38:0/3236840731 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd100009790 con 0x7fd110067ac0 2023-03-09T14:51:12.773 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.766+0000 7fd115f2d700 1 -- 172.21.15.38:0/3236840731 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd110112100 con 0x7fd110067ac0 2023-03-09T14:51:12.773 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.766+0000 7fd10cff9700 1 -- 172.21.15.38:0/3236840731 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd10000cbd0 con 0x7fd110067ac0 2023-03-09T14:51:12.773 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.766+0000 7fd10cff9700 1 -- 172.21.15.38:0/3236840731 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd100018680 con 0x7fd110067ac0 2023-03-09T14:51:12.773 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.767+0000 7fd115f2d700 1 -- 172.21.15.38:0/3236840731 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd110111f70 con 0x7fd110067ac0 2023-03-09T14:51:12.775 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.769+0000 7fd10cff9700 1 -- 172.21.15.38:0/3236840731 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 25) v1 ==== 133295+0+0 (secure 0 0 0) 0x7fd10000c040 con 0x7fd110067ac0 2023-03-09T14:51:12.776 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.770+0000 7fd10cff9700 1 --2- 172.21.15.38:0/3236840731 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fd0f809f4a0 0x7fd0f80a1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:51:12.776 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.770+0000 7fd10cff9700 1 -- 172.21.15.38:0/3236840731 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 9563+0+0 (secure 0 0 0) 0x7fd100014070 con 0x7fd110067ac0 2023-03-09T14:51:12.776 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.770+0000 7fd10effd700 1 --2- 172.21.15.38:0/3236840731 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fd0f809f4a0 0x7fd0f80a1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:51:12.776 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.770+0000 7fd10effd700 1 --2- 172.21.15.38:0/3236840731 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fd0f809f4a0 0x7fd0f80a1950 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd110199970 tx=0x7fd104009260 comp rx=0 tx=0).ready entity=mgr.15558 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:51:12.776 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.770+0000 7fd1220b6b80 1 Processor -- start 2023-03-09T14:51:12.776 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.770+0000 7fd1220b6b80 1 -- start start 2023-03-09T14:51:12.776 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.771+0000 7fd1220b6b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56258504d830 0x56258504dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:51:12.777 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.771+0000 7fd1220b6b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56258504e140 0x562585047aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:51:12.777 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.771+0000 7fd1220b6b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562585047fe0 0x562585048410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:51:12.777 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.771+0000 7fd1220b6b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x562584f31da0 con 0x56258504d830 2023-03-09T14:51:12.777 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.771+0000 7fd1220b6b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x562584fa0780 con 0x56258504e140 2023-03-09T14:51:12.777 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.771+0000 7fd1220b6b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x562584f99cb0 con 0x562585047fe0 2023-03-09T14:51:12.777 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.771+0000 7fd10f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56258504d830 0x56258504dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:51:12.777 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.771+0000 7fd10f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56258504d830 0x56258504dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:40466/0 (socket says 172.21.15.38:40466) 2023-03-09T14:51:12.778 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.771+0000 7fd10f7fe700 1 -- 172.21.15.38:0/3095218187 learned_addr learned my addr 172.21.15.38:0/3095218187 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:51:12.778 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.771+0000 7fd10ffff700 1 --2- 172.21.15.38:0/3095218187 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562585047fe0 0x562585048410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:51:12.778 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.771+0000 7fd10f7fe700 1 -- 172.21.15.38:0/3095218187 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562585047fe0 msgr2=0x562585048410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:51:12.778 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.771+0000 7fd10f7fe700 1 --2- 172.21.15.38:0/3095218187 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562585047fe0 0x562585048410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:51:12.778 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.771+0000 7fd10f7fe700 1 -- 172.21.15.38:0/3095218187 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56258504e140 msgr2=0x562585047aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:51:12.778 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.771+0000 7fd10f7fe700 1 --2- 172.21.15.38:0/3095218187 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56258504e140 0x562585047aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:51:12.778 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.771+0000 7fd10f7fe700 1 -- 172.21.15.38:0/3095218187 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562584f60190 con 0x56258504d830 2023-03-09T14:51:12.778 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.771+0000 7fd10ffff700 1 --2- 172.21.15.38:0/3095218187 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562585047fe0 0x562585048410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:51:12.779 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.772+0000 7fd10f7fe700 1 --2- 172.21.15.38:0/3095218187 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56258504d830 0x56258504dc00 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fd100094610 tx=0x7fd100097b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94a12f4371c99803 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:51:12.779 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.772+0000 7fd0f5ffb700 1 -- 172.21.15.38:0/3095218187 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1000a0350 con 0x56258504d830 2023-03-09T14:51:12.779 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.772+0000 7fd0f5ffb700 1 -- 172.21.15.38:0/3095218187 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd10000cd40 con 0x56258504d830 2023-03-09T14:51:12.779 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.772+0000 7fd0f5ffb700 1 -- 172.21.15.38:0/3095218187 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd10009fec0 con 0x56258504d830 2023-03-09T14:51:12.779 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.773+0000 7fd1220b6b80 1 -- 172.21.15.38:0/3095218187 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56258504d830 msgr2=0x56258504dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:51:12.779 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.773+0000 7fd1220b6b80 1 --2- 172.21.15.38:0/3095218187 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56258504d830 0x56258504dc00 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fd100094610 tx=0x7fd100097b40 comp rx=0 tx=0).stop 2023-03-09T14:51:12.779 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.773+0000 7fd1220b6b80 1 -- 172.21.15.38:0/3095218187 shutdown_connections 2023-03-09T14:51:12.780 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.773+0000 7fd1220b6b80 1 --2- 172.21.15.38:0/3095218187 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56258504d830 0x56258504dc00 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:51:12.780 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.773+0000 7fd1220b6b80 1 --2- 172.21.15.38:0/3095218187 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x562585047fe0 0x562585048410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:51:12.780 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.773+0000 7fd1220b6b80 1 --2- 172.21.15.38:0/3095218187 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56258504e140 0x562585047aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:51:12.780 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.773+0000 7fd1220b6b80 1 -- 172.21.15.38:0/3095218187 >> 172.21.15.38:0/3095218187 conn(0x562585042a20 msgr2=0x562585042df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:51:12.780 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.773+0000 7fd1220b6b80 1 -- 172.21.15.38:0/3095218187 shutdown_connections 2023-03-09T14:51:12.780 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.773+0000 7fd1220b6b80 1 -- 172.21.15.38:0/3095218187 wait complete. 2023-03-09T14:51:12.780 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.773+0000 7fd1220b6b80 1 -- start start 2023-03-09T14:51:12.780 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.773+0000 7fd1220b6b80 1 Processor -- start 2023-03-09T14:51:12.783 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.774+0000 7fd1220b6b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562585047fe0 0x562585054100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:51:12.783 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.774+0000 7fd1220b6b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56258504d830 0x5625850513f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:51:12.784 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.774+0000 7fd1220b6b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56258504e140 0x562585051a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:51:12.784 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.774+0000 7fd1220b6b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x562584e1d920 con 0x562585047fe0 2023-03-09T14:51:12.784 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.774+0000 7fd1220b6b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625850521e0 con 0x56258504e140 2023-03-09T14:51:12.784 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.774+0000 7fd1220b6b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x56258505dc40 con 0x56258504d830 2023-03-09T14:51:12.784 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.774+0000 7fd1220b6b80 10 client.0 _ll_register_callbacks cb 0x562584f3f520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:51:12.784 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.774+0000 7fd10f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562585047fe0 0x562585054100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:51:12.784 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.774+0000 7fd10f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562585047fe0 0x562585054100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:40472/0 (socket says 172.21.15.38:40472) 2023-03-09T14:51:12.785 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.774+0000 7fd10effd700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56258504d830 0x5625850513f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:51:12.785 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.774+0000 7fd10f7fe700 1 -- 172.21.15.38:0/166626002 learned_addr learned my addr 172.21.15.38:0/166626002 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:51:12.785 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.774+0000 7fd10ffff700 1 --2- 172.21.15.38:0/166626002 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56258504e140 0x562585051a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:51:12.785 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.774+0000 7fd10f7fe700 1 -- 172.21.15.38:0/166626002 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56258504d830 msgr2=0x5625850513f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:51:12.785 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.774+0000 7fd10f7fe700 1 --2- 172.21.15.38:0/166626002 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56258504d830 0x5625850513f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:51:12.785 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.774+0000 7fd10f7fe700 1 -- 172.21.15.38:0/166626002 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56258504e140 msgr2=0x562585051a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:51:12.785 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.774+0000 7fd10f7fe700 1 --2- 172.21.15.38:0/166626002 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56258504e140 0x562585051a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:51:12.785 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.774+0000 7fd10f7fe700 1 -- 172.21.15.38:0/166626002 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd11010f760 con 0x562585047fe0 2023-03-09T14:51:12.786 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.774+0000 7fd10ffff700 1 --2- 172.21.15.38:0/166626002 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56258504e140 0x562585051a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:51:12.786 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.774+0000 7fd10f7fe700 1 --2- 172.21.15.38:0/166626002 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562585047fe0 0x562585054100 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fd1000987c0 tx=0x7fd1000a03e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ea0a1ce6d3dcf09 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:51:12.786 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.774+0000 7fd0eeffd700 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2023-03-09T14:51:12.786 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.775+0000 7fd0eeffd700 1 -- 172.21.15.38:0/166626002 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1000b0470 con 0x562585047fe0 2023-03-09T14:51:12.786 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.775+0000 7fd0eeffd700 1 -- 172.21.15.38:0/166626002 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(124..124 src has 1..124) v4 ==== 9563+0+0 (secure 0 0 0) 0x7fd10009fec0 con 0x562585047fe0 2023-03-09T14:51:12.787 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.775+0000 7fd1220b6b80 1 -- 172.21.15.38:0/166626002 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562584f67190 con 0x562585047fe0 2023-03-09T14:51:12.787 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.775+0000 7fd0eeffd700 1 -- 172.21.15.38:0/166626002 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd1000a0080 con 0x562585047fe0 2023-03-09T14:51:12.787 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.775+0000 7fd1220b6b80 10 client.16026 fetch_fsmap learned FSMap version 30 2023-03-09T14:51:12.787 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.775+0000 7fd1220b6b80 1 -- 172.21.15.38:0/166626002 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x7fd110111f70 con 0x562585047fe0 2023-03-09T14:51:12.787 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.775+0000 7fd0eeffd700 1 -- 172.21.15.38:0/166626002 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fd10007b080 con 0x562585047fe0 2023-03-09T14:51:12.787 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.775+0000 7fd0eeffd700 1 -- 172.21.15.38:0/166626002 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 30) v1 ==== 7474+0+0 (secure 0 0 0) 0x7fd1000aba00 con 0x562585047fe0 2023-03-09T14:51:12.788 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.775+0000 7fd1220b6b80 10 client.16026 fetch_fsmap finished waiting for FSMap version 30 2023-03-09T14:51:12.788 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.775+0000 7fd1220b6b80 10 client.16026 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T14:51:12.788 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.775+0000 7fd1220b6b80 20 client.16026 populate_metadata read hostname 'smithi038' 2023-03-09T14:51:12.788 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.775+0000 7fd1220b6b80 1 --2- 172.21.15.38:0/166626002 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x562585061880 0x562585063c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:51:12.788 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.775+0000 7fd1220b6b80 4 client.16026 mds_command: new command op to 24530 tid=0[{"prefix": "get_command_descriptions"}] 2023-03-09T14:51:12.788 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.775+0000 7fd1220b6b80 1 -- 172.21.15.38:0/166626002 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562584e1df60 con 0x562585061880 2023-03-09T14:51:12.788 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.776+0000 7fd10effd700 1 --2- 172.21.15.38:0/166626002 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x562585061880 0x562585063c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:51:12.788 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.776+0000 7fd0eeffd700 10 client.16026 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:51:12.789 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.776+0000 7fd10effd700 1 --2- 172.21.15.38:0/166626002 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x562585061880 0x562585063c90 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48e9ee18c876adf8 server_cookie=564d80ba9b49d02 in_seq=0 out_seq=0 2023-03-09T14:51:12.789 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.777+0000 7fd0eeffd700 1 -- 172.21.15.38:0/166626002 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18278 (crc 0 0 0) 0x562584e1df60 con 0x562585061880 2023-03-09T14:51:12.789 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.777+0000 7fd0eeffd700 10 client.16026 handle_command_reply: tid=0 2023-03-09T14:51:12.789 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.777+0000 7fd1220b6b80 1 client.16026 shutdown 2023-03-09T14:51:12.789 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.777+0000 7fd1220b6b80 1 -- 172.21.15.38:0/166626002 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562585047fe0 msgr2=0x562585054100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:51:12.789 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.777+0000 7fd1220b6b80 1 --2- 172.21.15.38:0/166626002 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562585047fe0 0x562585054100 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fd1000987c0 tx=0x7fd1000a03e0 comp rx=0 tx=0).stop 2023-03-09T14:51:12.789 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.777+0000 7fd1220b6b80 1 -- 172.21.15.38:0/166626002 shutdown_connections 2023-03-09T14:51:12.790 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.777+0000 7fd1220b6b80 1 --2- 172.21.15.38:0/166626002 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x562585061880 0x562585063c90 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:51:12.790 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.777+0000 7fd1220b6b80 1 --2- 172.21.15.38:0/166626002 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x562585047fe0 0x562585054100 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:51:12.790 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.777+0000 7fd0eeffd700 0 client.16026 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:51:12.790 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.777+0000 7fd1220b6b80 1 --2- 172.21.15.38:0/166626002 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x56258504e140 0x562585051a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:51:12.790 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.777+0000 7fd1220b6b80 1 --2- 172.21.15.38:0/166626002 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56258504d830 0x5625850513f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:51:12.790 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.777+0000 7fd1220b6b80 1 -- 172.21.15.38:0/166626002 >> 172.21.15.38:0/166626002 conn(0x562585042a20 msgr2=0x562585041c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:51:12.790 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.777+0000 7fd1220b6b80 1 -- 172.21.15.38:0/166626002 shutdown_connections 2023-03-09T14:51:12.790 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.777+0000 7fd1220b6b80 1 -- 172.21.15.38:0/166626002 wait complete. 2023-03-09T14:51:12.791 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.777+0000 7fd1220b6b80 20 client.16026 trim_cache size 0 max 16384 2023-03-09T14:51:12.799 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.796+0000 7fd1220b6b80 1 Processor -- start 2023-03-09T14:51:12.800 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.796+0000 7fd1220b6b80 1 -- start start 2023-03-09T14:51:12.800 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.796+0000 7fd1220b6b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56258504d830 0x562585041c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:51:12.800 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.796+0000 7fd1220b6b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56258504e140 0x562585053ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:51:12.800 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.796+0000 7fd1220b6b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562585042160 0x56258504b530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:51:12.800 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.796+0000 7fd1220b6b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x562585052960 con 0x56258504d830 2023-03-09T14:51:12.800 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.796+0000 7fd1220b6b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x56258504f950 con 0x562585042160 2023-03-09T14:51:12.800 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.796+0000 7fd1220b6b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x562585048a30 con 0x56258504e140 2023-03-09T14:51:12.801 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.796+0000 7fd10f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56258504d830 0x562585041c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:51:12.801 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.796+0000 7fd10f7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56258504d830 0x562585041c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:40480/0 (socket says 172.21.15.38:40480) 2023-03-09T14:51:12.801 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.796+0000 7fd10f7fe700 1 -- 172.21.15.38:0/272370410 learned_addr learned my addr 172.21.15.38:0/272370410 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:51:12.801 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.796+0000 7fd10ffff700 1 --2- 172.21.15.38:0/272370410 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562585042160 0x56258504b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:51:12.801 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.797+0000 7fd10effd700 1 --2- 172.21.15.38:0/272370410 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56258504e140 0x562585053ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:51:12.801 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.797+0000 7fd10f7fe700 1 -- 172.21.15.38:0/272370410 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56258504e140 msgr2=0x562585053ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:51:12.801 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.797+0000 7fd10f7fe700 1 --2- 172.21.15.38:0/272370410 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56258504e140 0x562585053ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:51:12.802 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.797+0000 7fd10f7fe700 1 -- 172.21.15.38:0/272370410 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562585042160 msgr2=0x56258504b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:51:12.802 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.797+0000 7fd10f7fe700 1 --2- 172.21.15.38:0/272370410 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562585042160 0x56258504b530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:51:12.802 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.797+0000 7fd10f7fe700 1 -- 172.21.15.38:0/272370410 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562584db0c60 con 0x56258504d830 2023-03-09T14:51:12.802 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.797+0000 7fd10effd700 1 --2- 172.21.15.38:0/272370410 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56258504e140 0x562585053ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:51:12.802 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.797+0000 7fd10f7fe700 1 --2- 172.21.15.38:0/272370410 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56258504d830 0x562585041c20 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fd10009fe20 tx=0x7fd10000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=597206c3346d615c server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:51:12.802 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.797+0000 7fd0f67fc700 1 -- 172.21.15.38:0/272370410 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1000abc00 con 0x56258504d830 2023-03-09T14:51:12.802 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.797+0000 7fd0f67fc700 1 -- 172.21.15.38:0/272370410 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x562584f67190 con 0x56258504d830 2023-03-09T14:51:12.802 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.797+0000 7fd0f67fc700 1 -- 172.21.15.38:0/272370410 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1000970c0 con 0x56258504d830 2023-03-09T14:51:12.806 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.798+0000 7fd1220b6b80 1 -- 172.21.15.38:0/272370410 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56258504d830 msgr2=0x562585041c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:51:12.806 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.798+0000 7fd1220b6b80 1 --2- 172.21.15.38:0/272370410 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56258504d830 0x562585041c20 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fd10009fe20 tx=0x7fd10000cf90 comp rx=0 tx=0).stop 2023-03-09T14:51:12.806 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.798+0000 7fd1220b6b80 1 -- 172.21.15.38:0/272370410 shutdown_connections 2023-03-09T14:51:12.806 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.798+0000 7fd1220b6b80 1 --2- 172.21.15.38:0/272370410 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56258504d830 0x562585041c20 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:51:12.806 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.798+0000 7fd1220b6b80 1 --2- 172.21.15.38:0/272370410 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562585042160 0x56258504b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:51:12.806 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.798+0000 7fd1220b6b80 1 --2- 172.21.15.38:0/272370410 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56258504e140 0x562585053ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:51:12.806 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.798+0000 7fd1220b6b80 1 -- 172.21.15.38:0/272370410 >> 172.21.15.38:0/272370410 conn(0x562585042a20 msgr2=0x56258505e030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:51:12.807 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.798+0000 7fd1220b6b80 1 -- 172.21.15.38:0/272370410 shutdown_connections 2023-03-09T14:51:12.807 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.798+0000 7fd1220b6b80 1 -- 172.21.15.38:0/272370410 wait complete. 2023-03-09T14:51:12.807 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.798+0000 7fd1220b6b80 1 -- start start 2023-03-09T14:51:12.807 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.798+0000 7fd1220b6b80 1 Processor -- start 2023-03-09T14:51:12.807 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.799+0000 7fd1220b6b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562585042160 0x562585058d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:51:12.807 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.799+0000 7fd1220b6b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56258504d830 0x562585049700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:51:12.807 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.799+0000 7fd1220b6b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56258504e140 0x562585049c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:51:12.808 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.799+0000 7fd1220b6b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x56258504a380 con 0x56258504e140 2023-03-09T14:51:12.808 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.799+0000 7fd1220b6b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x56258504a4f0 con 0x562585042160 2023-03-09T14:51:12.808 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.799+0000 7fd1220b6b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625850b56e0 con 0x56258504d830 2023-03-09T14:51:12.808 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.799+0000 7fd1220b6b80 10 client.0 _ll_register_callbacks cb 0x562584f3f520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:51:12.808 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.799+0000 7fd10ffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56258504e140 0x562585049c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:51:12.808 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.799+0000 7fd10effd700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56258504d830 0x562585049700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:51:12.808 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.799+0000 7fd10ffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56258504e140 0x562585049c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:40490/0 (socket says 172.21.15.38:40490) 2023-03-09T14:51:12.808 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.799+0000 7fd10effd700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56258504d830 0x562585049700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.38:48186/0 (socket says 172.21.15.38:48186) 2023-03-09T14:51:12.809 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.799+0000 7fd10ffff700 1 -- 172.21.15.38:0/1204410831 learned_addr learned my addr 172.21.15.38:0/1204410831 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:51:12.809 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.799+0000 7fd10f7fe700 1 --2- 172.21.15.38:0/1204410831 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562585042160 0x562585058d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:51:12.809 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.799+0000 7fd10ffff700 1 -- 172.21.15.38:0/1204410831 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56258504d830 msgr2=0x562585049700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:51:12.809 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.799+0000 7fd10ffff700 1 --2- 172.21.15.38:0/1204410831 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56258504d830 0x562585049700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:51:12.809 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.799+0000 7fd10ffff700 1 -- 172.21.15.38:0/1204410831 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562585042160 msgr2=0x562585058d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:51:12.809 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.799+0000 7fd10ffff700 1 --2- 172.21.15.38:0/1204410831 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562585042160 0x562585058d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:51:12.809 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.799+0000 7fd10ffff700 1 -- 172.21.15.38:0/1204410831 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd110111f70 con 0x56258504e140 2023-03-09T14:51:12.810 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.799+0000 7fd10ffff700 1 --2- 172.21.15.38:0/1204410831 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56258504e140 0x562585049c40 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fd110069d30 tx=0x7fd108004e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f91b888b1912a2 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:51:12.810 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.799+0000 7fd0eeffd700 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2023-03-09T14:51:12.810 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.800+0000 7fd0eeffd700 1 -- 172.21.15.38:0/1204410831 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd10800e040 con 0x56258504e140 2023-03-09T14:51:12.810 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.800+0000 7fd0eeffd700 1 -- 172.21.15.38:0/1204410831 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(124..124 src has 1..124) v4 ==== 9563+0+0 (secure 0 0 0) 0x7fd1080076f0 con 0x56258504e140 2023-03-09T14:51:12.810 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.800+0000 7fd1220b6b80 1 -- 172.21.15.38:0/1204410831 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562584f571b0 con 0x56258504e140 2023-03-09T14:51:12.810 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.800+0000 7fd0eeffd700 1 -- 172.21.15.38:0/1204410831 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd1080078f0 con 0x56258504e140 2023-03-09T14:51:12.810 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.800+0000 7fd1220b6b80 10 client.16032 fetch_fsmap learned FSMap version 30 2023-03-09T14:51:12.810 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.800+0000 7fd1220b6b80 1 -- 172.21.15.38:0/1204410831 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x562584dee250 con 0x56258504e140 2023-03-09T14:51:12.811 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.800+0000 7fd0eeffd700 1 -- 172.21.15.38:0/1204410831 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x562584f571b0 con 0x56258504e140 2023-03-09T14:51:12.811 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.800+0000 7fd0eeffd700 1 -- 172.21.15.38:0/1204410831 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 30) v1 ==== 7474+0+0 (secure 0 0 0) 0x7fd10800b0d0 con 0x56258504e140 2023-03-09T14:51:12.811 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.800+0000 7fd1220b6b80 10 client.16032 fetch_fsmap finished waiting for FSMap version 30 2023-03-09T14:51:12.811 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.800+0000 7fd1220b6b80 10 client.16032 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T14:51:12.811 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.800+0000 7fd1220b6b80 20 client.16032 populate_metadata read hostname 'smithi038' 2023-03-09T14:51:12.811 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.800+0000 7fd1220b6b80 1 --2- 172.21.15.38:0/1204410831 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5625850b6e40 0x5625850b9250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:51:12.811 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.800+0000 7fd1220b6b80 4 client.16032 mds_command: new command op to 24530 tid=0[{"prefix": "scrub status"}] 2023-03-09T14:51:12.811 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.800+0000 7fd1220b6b80 1 -- 172.21.15.38:0/1204410831 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x562584f411c0 con 0x5625850b6e40 2023-03-09T14:51:12.812 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.800+0000 7fd10f7fe700 1 --2- 172.21.15.38:0/1204410831 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5625850b6e40 0x5625850b9250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:51:12.812 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.801+0000 7fd0eeffd700 10 client.16032 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:51:12.812 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.801+0000 7fd10f7fe700 1 --2- 172.21.15.38:0/1204410831 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5625850b6e40 0x5625850b9250 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=911034fd39cf2879 server_cookie=84a91aa0e18634c in_seq=0 out_seq=0 2023-03-09T14:51:12.812 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.801+0000 7fd0eeffd700 1 -- 172.21.15.38:0/1204410831 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+265 (crc 0 0 0) 0x562584f411c0 con 0x5625850b6e40 2023-03-09T14:51:12.812 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.801+0000 7fd0eeffd700 10 client.16032 handle_command_reply: tid=0 2023-03-09T14:51:12.813 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.801+0000 7fd1220b6b80 1 client.16032 shutdown 2023-03-09T14:51:12.813 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.801+0000 7fd1220b6b80 1 -- 172.21.15.38:0/1204410831 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56258504e140 msgr2=0x562585049c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:51:12.813 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.801+0000 7fd1220b6b80 1 --2- 172.21.15.38:0/1204410831 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56258504e140 0x562585049c40 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fd110069d30 tx=0x7fd108004e70 comp rx=0 tx=0).stop 2023-03-09T14:51:12.813 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.801+0000 7fd1220b6b80 1 -- 172.21.15.38:0/1204410831 shutdown_connections 2023-03-09T14:51:12.813 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.801+0000 7fd1220b6b80 1 --2- 172.21.15.38:0/1204410831 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5625850b6e40 0x5625850b9250 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:51:12.813 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.801+0000 7fd1220b6b80 1 --2- 172.21.15.38:0/1204410831 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x56258504e140 0x562585049c40 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:51:12.813 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.801+0000 7fd1220b6b80 1 --2- 172.21.15.38:0/1204410831 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x562585042160 0x562585058d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:51:12.814 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.801+0000 7fd1220b6b80 1 --2- 172.21.15.38:0/1204410831 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x56258504d830 0x562585049700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:51:12.814 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.801+0000 7fd0eeffd700 0 client.16032 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:51:12.814 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.801+0000 7fd1220b6b80 1 -- 172.21.15.38:0/1204410831 >> 172.21.15.38:0/1204410831 conn(0x562585042a20 msgr2=0x56258504b530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:51:12.814 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.801+0000 7fd1220b6b80 1 -- 172.21.15.38:0/1204410831 shutdown_connections 2023-03-09T14:51:12.814 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.801+0000 7fd1220b6b80 1 -- 172.21.15.38:0/1204410831 wait complete. 2023-03-09T14:51:12.814 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.802+0000 7fd1220b6b80 20 client.16032 trim_cache size 0 max 16384 2023-03-09T14:51:12.815 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2023-03-09T14:51:12.815 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "scrub active (3 inodes in the stack)", 2023-03-09T14:51:12.815 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": { 2023-03-09T14:51:12.815 INFO:teuthology.orchestra.run.smithi038.stdout: "a14659f4-d7ea-4c5a-b15b-6138af3f909f": { 2023-03-09T14:51:12.815 INFO:teuthology.orchestra.run.smithi038.stdout: "path": "/", 2023-03-09T14:51:12.815 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "a14659f4-d7ea-4c5a-b15b-6138af3f909f", 2023-03-09T14:51:12.816 INFO:teuthology.orchestra.run.smithi038.stdout: "options": "recursive,force" 2023-03-09T14:51:12.816 INFO:teuthology.orchestra.run.smithi038.stdout: } 2023-03-09T14:51:12.816 INFO:teuthology.orchestra.run.smithi038.stdout: } 2023-03-09T14:51:12.816 INFO:teuthology.orchestra.run.smithi038.stdout:} 2023-03-09T14:51:12.817 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.802+0000 7fd115f2d700 1 -- 172.21.15.38:0/3236840731 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fd0f809f4a0 msgr2=0x7fd0f80a1950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:51:12.817 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.802+0000 7fd115f2d700 1 --2- 172.21.15.38:0/3236840731 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fd0f809f4a0 0x7fd0f80a1950 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd110199970 tx=0x7fd104009260 comp rx=0 tx=0).stop 2023-03-09T14:51:12.817 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.802+0000 7fd115f2d700 1 -- 172.21.15.38:0/3236840731 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd110067ac0 msgr2=0x7fd110198360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:51:12.817 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.802+0000 7fd115f2d700 1 --2- 172.21.15.38:0/3236840731 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd110067ac0 0x7fd110198360 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fd10000e9a0 tx=0x7fd10000ecb0 comp rx=0 tx=0).stop 2023-03-09T14:51:12.817 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.802+0000 7fd115f2d700 1 -- 172.21.15.38:0/3236840731 shutdown_connections 2023-03-09T14:51:12.817 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.802+0000 7fd115f2d700 1 --2- 172.21.15.38:0/3236840731 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fd0f809f4a0 0x7fd0f80a1950 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:51:12.818 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.802+0000 7fd115f2d700 1 --2- 172.21.15.38:0/3236840731 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fd110067ac0 0x7fd110198360 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:51:12.818 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.802+0000 7fd115f2d700 1 --2- 172.21.15.38:0/3236840731 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fd11010c8c0 0x7fd11019cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:51:12.818 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.802+0000 7fd115f2d700 1 --2- 172.21.15.38:0/3236840731 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fd1100683d0 0x7fd1101988a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:51:12.818 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.802+0000 7fd115f2d700 1 -- 172.21.15.38:0/3236840731 >> 172.21.15.38:0/3236840731 conn(0x7fd110075c60 msgr2=0x7fd1100fc940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:51:12.818 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.802+0000 7fd115f2d700 1 -- 172.21.15.38:0/3236840731 shutdown_connections 2023-03-09T14:51:12.818 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:12.802+0000 7fd115f2d700 1 -- 172.21.15.38:0/3236840731 wait complete. 2023-03-09T14:51:12.819 INFO:tasks.cephfs.filesystem:scrub status for tag:a14659f4-d7ea-4c5a-b15b-6138af3f909f - {'path': '/', 'tag': 'a14659f4-d7ea-4c5a-b15b-6138af3f909f', 'options': 'recursive,force'} 2023-03-09T14:51:13.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:12 smithi038 ceph-mon[92422]: osdmap e124: 12 total, 11 up, 12 in 2023-03-09T14:51:13.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:12 smithi038 ceph-mon[92422]: Health check update: Degraded data redundancy: 2492/30110 objects degraded (8.276%), 22 pgs degraded (PG_DEGRADED) 2023-03-09T14:51:13.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:12 smithi122 ceph-mon[93869]: osdmap e124: 12 total, 11 up, 12 in 2023-03-09T14:51:13.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:12 smithi122 ceph-mon[93869]: Health check update: Degraded data redundancy: 2492/30110 objects degraded (8.276%), 22 pgs degraded (PG_DEGRADED) 2023-03-09T14:51:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:12 smithi181 ceph-mon[94189]: osdmap e124: 12 total, 11 up, 12 in 2023-03-09T14:51:13.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:12 smithi181 ceph-mon[94189]: Health check update: Degraded data redundancy: 2492/30110 objects degraded (8.276%), 22 pgs degraded (PG_DEGRADED) 2023-03-09T14:51:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 45.0 s, 234.3 tps, lat 2103.041 ms stddev 7038.117 2023-03-09T14:51:14.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:13 smithi122 ceph-mon[93869]: pgmap v186: 129 pgs: 2 active+recovery_wait+degraded, 1 peering, 18 active+undersized, 2 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 18 active+undersized+degraded, 87 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.0 MiB/s wr, 205 op/s; 2532/30162 objects degraded (8.395%); 287 MiB/s, 12 keys/s, 72 objects/s recovering 2023-03-09T14:51:14.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:13 smithi122 ceph-mon[93869]: osdmap e125: 12 total, 11 up, 12 in 2023-03-09T14:51:14.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:13 smithi181 ceph-mon[94189]: pgmap v186: 129 pgs: 2 active+recovery_wait+degraded, 1 peering, 18 active+undersized, 2 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 18 active+undersized+degraded, 87 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.0 MiB/s wr, 205 op/s; 2532/30162 objects degraded (8.395%); 287 MiB/s, 12 keys/s, 72 objects/s recovering 2023-03-09T14:51:14.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:13 smithi181 ceph-mon[94189]: osdmap e125: 12 total, 11 up, 12 in 2023-03-09T14:51:14.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:13 smithi038 ceph-mon[92422]: pgmap v186: 129 pgs: 2 active+recovery_wait+degraded, 1 peering, 18 active+undersized, 2 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 18 active+undersized+degraded, 87 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.0 MiB/s wr, 205 op/s; 2532/30162 objects degraded (8.395%); 287 MiB/s, 12 keys/s, 72 objects/s recovering 2023-03-09T14:51:14.254 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:13 smithi038 ceph-mon[92422]: osdmap e125: 12 total, 11 up, 12 in 2023-03-09T14:51:15.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:14 smithi122 ceph-mon[93869]: osdmap e126: 12 total, 11 up, 12 in 2023-03-09T14:51:15.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:14 smithi181 ceph-mon[94189]: osdmap e126: 12 total, 11 up, 12 in 2023-03-09T14:51:15.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:14 smithi038 ceph-mon[92422]: osdmap e126: 12 total, 11 up, 12 in 2023-03-09T14:51:16.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:15 smithi122 ceph-mon[93869]: pgmap v189: 129 pgs: 2 active+recovery_wait+degraded, 1 peering, 18 active+undersized, 2 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 18 active+undersized+degraded, 87 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.2 MiB/s wr, 322 op/s; 2533/30190 objects degraded (8.390%); 82 MiB/s, 20 objects/s recovering 2023-03-09T14:51:16.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:15 smithi122 ceph-mon[93869]: osdmap e127: 12 total, 11 up, 12 in 2023-03-09T14:51:16.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:15 smithi181 ceph-mon[94189]: pgmap v189: 129 pgs: 2 active+recovery_wait+degraded, 1 peering, 18 active+undersized, 2 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 18 active+undersized+degraded, 87 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.2 MiB/s wr, 322 op/s; 2533/30190 objects degraded (8.390%); 82 MiB/s, 20 objects/s recovering 2023-03-09T14:51:16.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:15 smithi181 ceph-mon[94189]: osdmap e127: 12 total, 11 up, 12 in 2023-03-09T14:51:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:15 smithi038 ceph-mon[92422]: pgmap v189: 129 pgs: 2 active+recovery_wait+degraded, 1 peering, 18 active+undersized, 2 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 18 active+undersized+degraded, 87 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.2 MiB/s wr, 322 op/s; 2533/30190 objects degraded (8.390%); 82 MiB/s, 20 objects/s recovering 2023-03-09T14:51:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:15 smithi038 ceph-mon[92422]: osdmap e127: 12 total, 11 up, 12 in 2023-03-09T14:51:17.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:16 smithi122 ceph-mon[93869]: osdmap e128: 12 total, 11 up, 12 in 2023-03-09T14:51:17.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:16 smithi181 ceph-mon[94189]: osdmap e128: 12 total, 11 up, 12 in 2023-03-09T14:51:17.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:16 smithi038 ceph-mon[92422]: osdmap e128: 12 total, 11 up, 12 in 2023-03-09T14:51:17.555 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 2023-03-09T14:51:17.247+0000 7f483b3f2700 -1 osd.1 pg_epoch: 128 pg[4.as0( v 128'3043 (0'0,128'3043] local-lis/les=104/105 n=283 ec=85/85 lis/c=104/104 les/c/f=105/106/0 sis=104) [1,5,6,7]p1(0) r=0 lpr=104 luod=128'3016 crt=128'3016 lcod 128'3014 mlcod 128'3014 active+clean] bad op order, already applied 79621 > this 79620 2023-03-09T14:51:17.556 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7f483b3f2700 time 2023-03-09T14:51:17.247728+0000 2023-03-09T14:51:17.556 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op") 2023-03-09T14:51:17.556 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:51:17.557 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x561dbf53cc85] 2023-03-09T14:51:17.557 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 2: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x561dbf7f56e4] 2023-03-09T14:51:17.557 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x561dbf7fbd5e] 2023-03-09T14:51:17.557 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 4: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x561dbf64b986] 2023-03-09T14:51:17.558 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 5: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x561dbf961729] 2023-03-09T14:51:17.558 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x561dbf660ebf] 2023-03-09T14:51:17.558 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x561dbfd12d85] 2023-03-09T14:51:17.559 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x561dbfd15f84] 2023-03-09T14:51:17.559 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f48611581ca] 2023-03-09T14:51:17.559 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 10: clone() 2023-03-09T14:51:17.560 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 2023-03-09T14:51:17.250+0000 7f483b3f2700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7f483b3f2700 time 2023-03-09T14:51:17.247728+0000 2023-03-09T14:51:17.560 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op") 2023-03-09T14:51:17.560 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 2023-03-09T14:51:17.560 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:51:17.561 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x561dbf53cc85] 2023-03-09T14:51:17.561 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 2: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x561dbf7f56e4] 2023-03-09T14:51:17.561 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x561dbf7fbd5e] 2023-03-09T14:51:17.562 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 4: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x561dbf64b986] 2023-03-09T14:51:17.562 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 5: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x561dbf961729] 2023-03-09T14:51:17.562 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x561dbf660ebf] 2023-03-09T14:51:17.562 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x561dbfd12d85] 2023-03-09T14:51:17.563 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x561dbfd15f84] 2023-03-09T14:51:17.563 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f48611581ca] 2023-03-09T14:51:17.563 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 10: clone() 2023-03-09T14:51:17.563 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 2023-03-09T14:51:17.564 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: *** Caught signal (Aborted) ** 2023-03-09T14:51:17.564 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: in thread 7f483b3f2700 thread_name:tp_osd_tp 2023-03-09T14:51:17.564 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:51:17.564 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f4861162cf0] 2023-03-09T14:51:17.565 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 2: gsignal() 2023-03-09T14:51:17.565 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 3: abort() 2023-03-09T14:51:17.565 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x561dbf53cd45] 2023-03-09T14:51:17.565 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 5: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x561dbf7f56e4] 2023-03-09T14:51:17.566 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 6: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x561dbf7fbd5e] 2023-03-09T14:51:17.566 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x561dbf64b986] 2023-03-09T14:51:17.566 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 8: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x561dbf961729] 2023-03-09T14:51:17.566 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x561dbf660ebf] 2023-03-09T14:51:17.566 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x561dbfd12d85] 2023-03-09T14:51:17.567 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x561dbfd15f84] 2023-03-09T14:51:17.567 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f48611581ca] 2023-03-09T14:51:17.567 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 13: clone() 2023-03-09T14:51:17.567 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 2023-03-09T14:51:17.257+0000 7f483b3f2700 -1 *** Caught signal (Aborted) ** 2023-03-09T14:51:17.567 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: in thread 7f483b3f2700 thread_name:tp_osd_tp 2023-03-09T14:51:17.567 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 2023-03-09T14:51:17.568 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:51:17.568 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f4861162cf0] 2023-03-09T14:51:17.568 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 2: gsignal() 2023-03-09T14:51:17.569 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 3: abort() 2023-03-09T14:51:17.569 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x561dbf53cd45] 2023-03-09T14:51:17.569 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 5: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x561dbf7f56e4] 2023-03-09T14:51:17.569 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 6: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x561dbf7fbd5e] 2023-03-09T14:51:17.570 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x561dbf64b986] 2023-03-09T14:51:17.570 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 8: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x561dbf961729] 2023-03-09T14:51:17.570 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x561dbf660ebf] 2023-03-09T14:51:17.570 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x561dbfd12d85] 2023-03-09T14:51:17.571 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x561dbfd15f84] 2023-03-09T14:51:17.571 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f48611581ca] 2023-03-09T14:51:17.571 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 13: clone() 2023-03-09T14:51:17.571 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-03-09T14:51:17.572 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 2023-03-09T14:51:17.572 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: -7580> 2023-03-09T14:51:17.247+0000 7f483b3f2700 -1 osd.1 pg_epoch: 128 pg[4.as0( v 128'3043 (0'0,128'3043] local-lis/les=104/105 n=283 ec=85/85 lis/c=104/104 les/c/f=105/106/0 sis=104) [1,5,6,7]p1(0) r=0 lpr=104 luod=128'3016 crt=128'3016 lcod 128'3014 mlcod 128'3014 active+clean] bad op order, already applied 79621 > this 79620 2023-03-09T14:51:17.572 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: -6960> 2023-03-09T14:51:17.250+0000 7f483b3f2700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7f483b3f2700 time 2023-03-09T14:51:17.247728+0000 2023-03-09T14:51:17.572 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op") 2023-03-09T14:51:17.573 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 2023-03-09T14:51:17.573 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:51:17.573 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x561dbf53cc85] 2023-03-09T14:51:17.574 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 2: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x561dbf7f56e4] 2023-03-09T14:51:17.574 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x561dbf7fbd5e] 2023-03-09T14:51:17.574 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 4: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x561dbf64b986] 2023-03-09T14:51:17.574 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 5: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x561dbf961729] 2023-03-09T14:51:17.574 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x561dbf660ebf] 2023-03-09T14:51:17.574 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x561dbfd12d85] 2023-03-09T14:51:17.574 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x561dbfd15f84] 2023-03-09T14:51:17.575 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f48611581ca] 2023-03-09T14:51:17.575 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 10: clone() 2023-03-09T14:51:17.575 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 2023-03-09T14:51:17.575 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: -4803> 2023-03-09T14:51:17.257+0000 7f483b3f2700 -1 *** Caught signal (Aborted) ** 2023-03-09T14:51:17.575 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: in thread 7f483b3f2700 thread_name:tp_osd_tp 2023-03-09T14:51:17.575 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 2023-03-09T14:51:17.575 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:51:17.576 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f4861162cf0] 2023-03-09T14:51:17.576 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 2: gsignal() 2023-03-09T14:51:17.576 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 3: abort() 2023-03-09T14:51:17.576 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x561dbf53cd45] 2023-03-09T14:51:17.576 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 5: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x561dbf7f56e4] 2023-03-09T14:51:17.576 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 6: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x561dbf7fbd5e] 2023-03-09T14:51:17.576 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x561dbf64b986] 2023-03-09T14:51:17.576 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 8: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x561dbf961729] 2023-03-09T14:51:17.577 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x561dbf660ebf] 2023-03-09T14:51:17.577 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x561dbfd12d85] 2023-03-09T14:51:17.577 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x561dbfd15f84] 2023-03-09T14:51:17.577 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f48611581ca] 2023-03-09T14:51:17.577 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 13: clone() 2023-03-09T14:51:17.578 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-03-09T14:51:17.578 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 2023-03-09T14:51:17.578 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: -8581> 2023-03-09T14:51:17.247+0000 7f483b3f2700 -1 osd.1 pg_epoch: 128 pg[4.as0( v 128'3043 (0'0,128'3043] local-lis/les=104/105 n=283 ec=85/85 lis/c=104/104 les/c/f=105/106/0 sis=104) [1,5,6,7]p1(0) r=0 lpr=104 luod=128'3016 crt=128'3016 lcod 128'3014 mlcod 128'3014 active+clean] bad op order, already applied 79621 > this 79620 2023-03-09T14:51:17.578 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: -7961> 2023-03-09T14:51:17.250+0000 7f483b3f2700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7f483b3f2700 time 2023-03-09T14:51:17.247728+0000 2023-03-09T14:51:17.578 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op") 2023-03-09T14:51:17.579 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 2023-03-09T14:51:17.579 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:51:17.579 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x561dbf53cc85] 2023-03-09T14:51:17.579 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 2: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x561dbf7f56e4] 2023-03-09T14:51:17.579 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x561dbf7fbd5e] 2023-03-09T14:51:17.579 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 4: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x561dbf64b986] 2023-03-09T14:51:17.579 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 5: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x561dbf961729] 2023-03-09T14:51:17.580 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x561dbf660ebf] 2023-03-09T14:51:17.580 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x561dbfd12d85] 2023-03-09T14:51:17.580 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x561dbfd15f84] 2023-03-09T14:51:17.580 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f48611581ca] 2023-03-09T14:51:17.580 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 10: clone() 2023-03-09T14:51:17.580 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 2023-03-09T14:51:17.580 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: -5804> 2023-03-09T14:51:17.257+0000 7f483b3f2700 -1 *** Caught signal (Aborted) ** 2023-03-09T14:51:17.580 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: in thread 7f483b3f2700 thread_name:tp_osd_tp 2023-03-09T14:51:17.581 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 2023-03-09T14:51:17.581 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:51:17.581 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f4861162cf0] 2023-03-09T14:51:17.581 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 2: gsignal() 2023-03-09T14:51:17.582 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 3: abort() 2023-03-09T14:51:17.582 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x561dbf53cd45] 2023-03-09T14:51:17.582 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 5: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x561dbf7f56e4] 2023-03-09T14:51:17.582 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 6: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x561dbf7fbd5e] 2023-03-09T14:51:17.582 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x561dbf64b986] 2023-03-09T14:51:17.583 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 8: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x561dbf961729] 2023-03-09T14:51:17.583 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x561dbf660ebf] 2023-03-09T14:51:17.583 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x561dbfd12d85] 2023-03-09T14:51:17.583 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x561dbfd15f84] 2023-03-09T14:51:17.583 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f48611581ca] 2023-03-09T14:51:17.583 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 13: clone() 2023-03-09T14:51:17.583 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-03-09T14:51:17.584 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[123232]: 2023-03-09T14:51:17.654 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 2023-03-09T14:51:17.250+0000 7fd0f82a3700 -1 osd.3 pg_epoch: 128 pg[4.1bs1( v 128'749 (0'0,128'749] local-lis/les=118/119 n=262 ec=85/85 lis/c=118/85 les/c/f=119/86/0 sis=118) [NONE,3,9,5]p3(1) r=1 lpr=118 pi=[85,118)/2 luod=128'715 crt=128'715 lcod 128'713 mlcod 128'713 active+undersized+degraded mbc={0={},1={},2={},3={}}] bad op order, already applied 79720 > this 79719 2023-03-09T14:51:17.655 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7fd0f82a3700 time 2023-03-09T14:51:17.251178+0000 2023-03-09T14:51:17.655 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op") 2023-03-09T14:51:17.655 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:51:17.655 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x563b268e5c85] 2023-03-09T14:51:17.655 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 2: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x563b26b9e6e4] 2023-03-09T14:51:17.655 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x563b26ba4d5e] 2023-03-09T14:51:17.655 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 4: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x563b269f4986] 2023-03-09T14:51:17.656 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 5: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x563b26d0a729] 2023-03-09T14:51:17.656 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x563b26a09ebf] 2023-03-09T14:51:17.656 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x563b270bbd85] 2023-03-09T14:51:17.656 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x563b270bef84] 2023-03-09T14:51:17.656 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7fd11b8041ca] 2023-03-09T14:51:17.656 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 10: clone() 2023-03-09T14:51:17.656 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: *** Caught signal (Aborted) ** 2023-03-09T14:51:17.657 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: in thread 7fd0f82a3700 thread_name:tp_osd_tp 2023-03-09T14:51:17.657 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 2023-03-09T14:51:17.254+0000 7fd0f82a3700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7fd0f82a3700 time 2023-03-09T14:51:17.251178+0000 2023-03-09T14:51:17.657 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op") 2023-03-09T14:51:17.657 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 2023-03-09T14:51:17.657 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:51:17.657 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x563b268e5c85] 2023-03-09T14:51:17.657 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 2: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x563b26b9e6e4] 2023-03-09T14:51:17.658 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x563b26ba4d5e] 2023-03-09T14:51:17.658 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 4: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x563b269f4986] 2023-03-09T14:51:17.658 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 5: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x563b26d0a729] 2023-03-09T14:51:17.658 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x563b26a09ebf] 2023-03-09T14:51:17.658 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x563b270bbd85] 2023-03-09T14:51:17.658 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x563b270bef84] 2023-03-09T14:51:17.658 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7fd11b8041ca] 2023-03-09T14:51:17.659 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 10: clone() 2023-03-09T14:51:17.659 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 2023-03-09T14:51:17.659 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:51:17.659 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7fd11b80ecf0] 2023-03-09T14:51:17.659 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 2: gsignal() 2023-03-09T14:51:17.659 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 3: abort() 2023-03-09T14:51:17.659 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x563b268e5d45] 2023-03-09T14:51:17.659 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 5: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x563b26b9e6e4] 2023-03-09T14:51:17.660 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 6: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x563b26ba4d5e] 2023-03-09T14:51:17.660 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x563b269f4986] 2023-03-09T14:51:17.660 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 8: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x563b26d0a729] 2023-03-09T14:51:17.660 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x563b26a09ebf] 2023-03-09T14:51:17.660 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x563b270bbd85] 2023-03-09T14:51:17.660 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x563b270bef84] 2023-03-09T14:51:17.660 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fd11b8041ca] 2023-03-09T14:51:17.661 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 13: clone() 2023-03-09T14:51:17.661 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 2023-03-09T14:51:17.258+0000 7fd0f82a3700 -1 *** Caught signal (Aborted) ** 2023-03-09T14:51:17.661 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: in thread 7fd0f82a3700 thread_name:tp_osd_tp 2023-03-09T14:51:17.661 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 2023-03-09T14:51:17.661 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:51:17.661 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7fd11b80ecf0] 2023-03-09T14:51:17.661 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 2: gsignal() 2023-03-09T14:51:17.661 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 3: abort() 2023-03-09T14:51:17.662 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x563b268e5d45] 2023-03-09T14:51:17.662 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 5: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x563b26b9e6e4] 2023-03-09T14:51:17.662 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 6: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x563b26ba4d5e] 2023-03-09T14:51:17.662 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x563b269f4986] 2023-03-09T14:51:17.662 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 8: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x563b26d0a729] 2023-03-09T14:51:17.662 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x563b26a09ebf] 2023-03-09T14:51:17.662 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x563b270bbd85] 2023-03-09T14:51:17.663 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x563b270bef84] 2023-03-09T14:51:17.663 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fd11b8041ca] 2023-03-09T14:51:17.663 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 13: clone() 2023-03-09T14:51:17.663 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-03-09T14:51:17.663 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 2023-03-09T14:51:17.663 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: -2928> 2023-03-09T14:51:17.250+0000 7fd0f82a3700 -1 osd.3 pg_epoch: 128 pg[4.1bs1( v 128'749 (0'0,128'749] local-lis/les=118/119 n=262 ec=85/85 lis/c=118/85 les/c/f=119/86/0 sis=118) [NONE,3,9,5]p3(1) r=1 lpr=118 pi=[85,118)/2 luod=128'715 crt=128'715 lcod 128'713 mlcod 128'713 active+undersized+degraded mbc={0={},1={},2={},3={}}] bad op order, already applied 79720 > this 79719 2023-03-09T14:51:17.664 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: -1669> 2023-03-09T14:51:17.254+0000 7fd0f82a3700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7fd0f82a3700 time 2023-03-09T14:51:17.251178+0000 2023-03-09T14:51:17.664 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op") 2023-03-09T14:51:17.664 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 2023-03-09T14:51:17.664 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:51:17.664 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x563b268e5c85] 2023-03-09T14:51:17.665 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 2: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x563b26b9e6e4] 2023-03-09T14:51:17.665 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x563b26ba4d5e] 2023-03-09T14:51:17.665 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 4: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x563b269f4986] 2023-03-09T14:51:17.665 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 5: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x563b26d0a729] 2023-03-09T14:51:17.665 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x563b26a09ebf] 2023-03-09T14:51:17.665 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x563b270bbd85] 2023-03-09T14:51:17.665 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x563b270bef84] 2023-03-09T14:51:17.666 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7fd11b8041ca] 2023-03-09T14:51:17.666 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 10: clone() 2023-03-09T14:51:17.666 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 2023-03-09T14:51:17.666 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: -983> 2023-03-09T14:51:17.258+0000 7fd0f82a3700 -1 *** Caught signal (Aborted) ** 2023-03-09T14:51:17.666 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: in thread 7fd0f82a3700 thread_name:tp_osd_tp 2023-03-09T14:51:17.666 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 2023-03-09T14:51:17.666 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:51:17.666 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7fd11b80ecf0] 2023-03-09T14:51:17.667 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 2: gsignal() 2023-03-09T14:51:17.667 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 3: abort() 2023-03-09T14:51:17.667 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x563b268e5d45] 2023-03-09T14:51:17.667 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 5: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x563b26b9e6e4] 2023-03-09T14:51:17.667 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 6: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x563b26ba4d5e] 2023-03-09T14:51:17.668 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x563b269f4986] 2023-03-09T14:51:17.668 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 8: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x563b26d0a729] 2023-03-09T14:51:17.668 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x563b26a09ebf] 2023-03-09T14:51:17.668 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x563b270bbd85] 2023-03-09T14:51:17.669 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x563b270bef84] 2023-03-09T14:51:17.669 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fd11b8041ca] 2023-03-09T14:51:17.669 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 13: clone() 2023-03-09T14:51:17.669 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-03-09T14:51:17.669 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 2023-03-09T14:51:17.669 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: -3429> 2023-03-09T14:51:17.250+0000 7fd0f82a3700 -1 osd.3 pg_epoch: 128 pg[4.1bs1( v 128'749 (0'0,128'749] local-lis/les=118/119 n=262 ec=85/85 lis/c=118/85 les/c/f=119/86/0 sis=118) [NONE,3,9,5]p3(1) r=1 lpr=118 pi=[85,118)/2 luod=128'715 crt=128'715 lcod 128'713 mlcod 128'713 active+undersized+degraded mbc={0={},1={},2={},3={}}] bad op order, already applied 79720 > this 79719 2023-03-09T14:51:17.669 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: -2170> 2023-03-09T14:51:17.254+0000 7fd0f82a3700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7fd0f82a3700 time 2023-03-09T14:51:17.251178+0000 2023-03-09T14:51:17.670 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op") 2023-03-09T14:51:17.670 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 2023-03-09T14:51:17.670 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:51:17.670 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x563b268e5c85] 2023-03-09T14:51:17.670 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 2: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x563b26b9e6e4] 2023-03-09T14:51:17.670 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x563b26ba4d5e] 2023-03-09T14:51:17.670 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 4: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x563b269f4986] 2023-03-09T14:51:17.671 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 5: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x563b26d0a729] 2023-03-09T14:51:17.671 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x563b26a09ebf] 2023-03-09T14:51:17.671 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x563b270bbd85] 2023-03-09T14:51:17.671 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x563b270bef84] 2023-03-09T14:51:17.671 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7fd11b8041ca] 2023-03-09T14:51:17.671 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 10: clone() 2023-03-09T14:51:17.672 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 2023-03-09T14:51:17.672 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: -1484> 2023-03-09T14:51:17.258+0000 7fd0f82a3700 -1 *** Caught signal (Aborted) ** 2023-03-09T14:51:17.672 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: in thread 7fd0f82a3700 thread_name:tp_osd_tp 2023-03-09T14:51:17.672 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 2023-03-09T14:51:17.672 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:51:17.673 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7fd11b80ecf0] 2023-03-09T14:51:17.673 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 2: gsignal() 2023-03-09T14:51:17.673 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 3: abort() 2023-03-09T14:51:17.673 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x563b268e5d45] 2023-03-09T14:51:17.673 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 5: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x563b26b9e6e4] 2023-03-09T14:51:17.673 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 6: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x563b26ba4d5e] 2023-03-09T14:51:17.674 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x563b269f4986] 2023-03-09T14:51:17.674 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 8: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x563b26d0a729] 2023-03-09T14:51:17.674 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x563b26a09ebf] 2023-03-09T14:51:17.674 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x563b270bbd85] 2023-03-09T14:51:17.674 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x563b270bef84] 2023-03-09T14:51:17.674 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fd11b8041ca] 2023-03-09T14:51:17.674 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 13: clone() 2023-03-09T14:51:17.675 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-03-09T14:51:17.675 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[147941]: 2023-03-09T14:51:17.808 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:17 smithi122 systemd[1]: ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.1.service: Main process exited, code=exited, status=134/n/a 2023-03-09T14:51:17.953 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-mon[92422]: pgmap v192: 129 pgs: 1 active+recovering+degraded, 1 active+recovery_wait+degraded, 18 active+undersized, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 19 active+undersized+degraded, 88 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 6.1 MiB/s wr, 645 op/s; 2778/30558 objects degraded (9.091%); 27 MiB/s, 6 objects/s recovering 2023-03-09T14:51:17.954 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.9 2023-03-09T14:51:17.954 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-mon[92422]: osd.1 failed (root=default,host=smithi122) (connection refused reported by osd.9) 2023-03-09T14:51:17.954 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.6 2023-03-09T14:51:17.954 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.6 2023-03-09T14:51:17.954 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.0 2023-03-09T14:51:17.954 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.5 2023-03-09T14:51:17.955 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.8 2023-03-09T14:51:17.955 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.2 2023-03-09T14:51:17.955 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.8 2023-03-09T14:51:17.955 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.10 2023-03-09T14:51:17.955 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.7 2023-03-09T14:51:17.955 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.10 2023-03-09T14:51:17.955 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.7 2023-03-09T14:51:17.955 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.10 2023-03-09T14:51:17.956 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.7 2023-03-09T14:51:17.956 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.11 2023-03-09T14:51:17.956 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.5 2023-03-09T14:51:17.956 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.11 2023-03-09T14:51:17.956 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.5 2023-03-09T14:51:17.956 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.2 2023-03-09T14:51:17.956 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.2 2023-03-09T14:51:17.956 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.9 2023-03-09T14:51:17.957 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.9 2023-03-09T14:51:17.957 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.0 2023-03-09T14:51:17.957 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.6 2023-03-09T14:51:17.957 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.0 2023-03-09T14:51:17.957 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.0 2023-03-09T14:51:17.957 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-mon[92422]: osd.3 failed (root=default,host=smithi038) (connection refused reported by osd.0) 2023-03-09T14:51:17.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.9 2023-03-09T14:51:17.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.11 2023-03-09T14:51:17.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.2 2023-03-09T14:51:17.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.8 2023-03-09T14:51:17.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.6 2023-03-09T14:51:17.958 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.9 2023-03-09T14:51:17.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.11 2023-03-09T14:51:17.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.2 2023-03-09T14:51:17.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.7 2023-03-09T14:51:17.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.6 2023-03-09T14:51:17.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.9 2023-03-09T14:51:17.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.10 2023-03-09T14:51:17.959 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.7 2023-03-09T14:51:17.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.6 2023-03-09T14:51:17.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.10 2023-03-09T14:51:17.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.8 2023-03-09T14:51:17.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.5 2023-03-09T14:51:17.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.2 2023-03-09T14:51:17.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.5 2023-03-09T14:51:17.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.5 2023-03-09T14:51:17.960 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.0 2023-03-09T14:51:17.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-mon[92422]: Health check update: Degraded data redundancy: 2778/30558 objects degraded (9.091%), 23 pgs degraded (PG_DEGRADED) 2023-03-09T14:51:17.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.9 2023-03-09T14:51:17.961 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:17 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.0 2023-03-09T14:51:18.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-mon[93869]: pgmap v192: 129 pgs: 1 active+recovering+degraded, 1 active+recovery_wait+degraded, 18 active+undersized, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 19 active+undersized+degraded, 88 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 6.1 MiB/s wr, 645 op/s; 2778/30558 objects degraded (9.091%); 27 MiB/s, 6 objects/s recovering 2023-03-09T14:51:18.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.9 2023-03-09T14:51:18.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-mon[93869]: osd.1 failed (root=default,host=smithi122) (connection refused reported by osd.9) 2023-03-09T14:51:18.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.6 2023-03-09T14:51:18.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.6 2023-03-09T14:51:18.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.0 2023-03-09T14:51:18.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.5 2023-03-09T14:51:18.231 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.8 2023-03-09T14:51:18.231 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.2 2023-03-09T14:51:18.231 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.8 2023-03-09T14:51:18.231 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.10 2023-03-09T14:51:18.231 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.7 2023-03-09T14:51:18.231 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.10 2023-03-09T14:51:18.231 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.7 2023-03-09T14:51:18.231 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.10 2023-03-09T14:51:18.232 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.7 2023-03-09T14:51:18.232 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.11 2023-03-09T14:51:18.232 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.5 2023-03-09T14:51:18.232 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.11 2023-03-09T14:51:18.232 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.5 2023-03-09T14:51:18.232 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.2 2023-03-09T14:51:18.232 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.2 2023-03-09T14:51:18.232 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.9 2023-03-09T14:51:18.233 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.9 2023-03-09T14:51:18.233 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.0 2023-03-09T14:51:18.233 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.6 2023-03-09T14:51:18.233 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.0 2023-03-09T14:51:18.233 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.0 2023-03-09T14:51:18.233 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-mon[93869]: osd.3 failed (root=default,host=smithi038) (connection refused reported by osd.0) 2023-03-09T14:51:18.233 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.9 2023-03-09T14:51:18.234 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.11 2023-03-09T14:51:18.234 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.2 2023-03-09T14:51:18.234 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.8 2023-03-09T14:51:18.234 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.6 2023-03-09T14:51:18.234 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.9 2023-03-09T14:51:18.234 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.11 2023-03-09T14:51:18.234 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.2 2023-03-09T14:51:18.235 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.7 2023-03-09T14:51:18.235 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.6 2023-03-09T14:51:18.235 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.9 2023-03-09T14:51:18.235 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.10 2023-03-09T14:51:18.235 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.7 2023-03-09T14:51:18.235 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.6 2023-03-09T14:51:18.235 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.10 2023-03-09T14:51:18.235 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.8 2023-03-09T14:51:18.236 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.5 2023-03-09T14:51:18.236 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.2 2023-03-09T14:51:18.236 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.5 2023-03-09T14:51:18.236 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.5 2023-03-09T14:51:18.236 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.0 2023-03-09T14:51:18.236 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-mon[93869]: Health check update: Degraded data redundancy: 2778/30558 objects degraded (9.091%), 23 pgs degraded (PG_DEGRADED) 2023-03-09T14:51:18.236 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.9 2023-03-09T14:51:18.236 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:17 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.0 2023-03-09T14:51:18.237 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:17 smithi181 ceph-mon[94189]: pgmap v192: 129 pgs: 1 active+recovering+degraded, 1 active+recovery_wait+degraded, 18 active+undersized, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 19 active+undersized+degraded, 88 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 6.1 MiB/s wr, 645 op/s; 2778/30558 objects degraded (9.091%); 27 MiB/s, 6 objects/s recovering 2023-03-09T14:51:18.237 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:17 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.9 2023-03-09T14:51:18.238 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:17 smithi181 ceph-mon[94189]: osd.1 failed (root=default,host=smithi122) (connection refused reported by osd.9) 2023-03-09T14:51:18.238 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:17 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.6 2023-03-09T14:51:18.238 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:17 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.6 2023-03-09T14:51:18.238 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:17 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.0 2023-03-09T14:51:18.238 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:17 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.5 2023-03-09T14:51:18.238 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:17 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.8 2023-03-09T14:51:18.238 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:17 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.2 2023-03-09T14:51:18.239 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:17 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.8 2023-03-09T14:51:18.239 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:17 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.10 2023-03-09T14:51:18.239 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:17 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.7 2023-03-09T14:51:18.239 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:17 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.10 2023-03-09T14:51:18.239 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:17 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.7 2023-03-09T14:51:18.240 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:17 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.10 2023-03-09T14:51:18.240 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:17 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.7 2023-03-09T14:51:18.240 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:17 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.11 2023-03-09T14:51:18.240 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:17 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.5 2023-03-09T14:51:18.240 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:17 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.11 2023-03-09T14:51:18.240 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:17 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.5 2023-03-09T14:51:18.240 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:17 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.2 2023-03-09T14:51:18.240 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:17 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.2 2023-03-09T14:51:18.241 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:17 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.9 2023-03-09T14:51:18.241 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:17 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.9 2023-03-09T14:51:18.241 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:17 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.0 2023-03-09T14:51:18.241 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:17 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.6 2023-03-09T14:51:18.241 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:17 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.0 2023-03-09T14:51:18.241 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:17 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.0 2023-03-09T14:51:18.241 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:17 smithi181 ceph-mon[94189]: osd.3 failed (root=default,host=smithi038) (connection refused reported by osd.0) 2023-03-09T14:51:18.242 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:17 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.9 2023-03-09T14:51:18.242 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:17 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.11 2023-03-09T14:51:18.242 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:17 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.2 2023-03-09T14:51:18.242 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:17 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.8 2023-03-09T14:51:18.242 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:17 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.6 2023-03-09T14:51:18.242 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:17 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.9 2023-03-09T14:51:18.242 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:17 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.11 2023-03-09T14:51:18.242 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:17 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.2 2023-03-09T14:51:18.243 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:17 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.7 2023-03-09T14:51:18.243 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:17 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.6 2023-03-09T14:51:18.243 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:17 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.9 2023-03-09T14:51:18.243 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:17 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.10 2023-03-09T14:51:18.243 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:17 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.7 2023-03-09T14:51:18.243 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:17 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.6 2023-03-09T14:51:18.243 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:17 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.10 2023-03-09T14:51:18.243 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:17 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.8 2023-03-09T14:51:18.244 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:17 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.5 2023-03-09T14:51:18.244 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:17 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.2 2023-03-09T14:51:18.244 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:17 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.5 2023-03-09T14:51:18.244 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:17 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.5 2023-03-09T14:51:18.244 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:17 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.0 2023-03-09T14:51:18.244 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:17 smithi181 ceph-mon[94189]: Health check update: Degraded data redundancy: 2778/30558 objects degraded (9.091%), 23 pgs degraded (PG_DEGRADED) 2023-03-09T14:51:18.244 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:17 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.9 2023-03-09T14:51:18.244 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:17 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.0 2023-03-09T14:51:18.728 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:18 smithi122 systemd[1]: ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.4.service: Service RestartSec=10s expired, scheduling restart. 2023-03-09T14:51:18.729 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:18 smithi122 systemd[1]: ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.4.service: Scheduled restart job, restart counter is at 1. 2023-03-09T14:51:18.729 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:18 smithi122 systemd[1]: Stopped Ceph osd.4 for c6733fe2-be87-11ed-9af2-001a4aab830c. 2023-03-09T14:51:18.729 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:18 smithi122 systemd[1]: Starting Ceph osd.4 for c6733fe2-be87-11ed-9af2-001a4aab830c... 2023-03-09T14:51:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 50.0 s, 313.9 tps, lat 299.761 ms stddev 478.741 2023-03-09T14:51:19.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:18 smithi122 ceph-mon[93869]: Health check update: 3 osds down (OSD_DOWN) 2023-03-09T14:51:19.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:18 smithi122 ceph-mon[93869]: osdmap e129: 12 total, 9 up, 12 in 2023-03-09T14:51:19.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:18 smithi181 ceph-mon[94189]: Health check update: 3 osds down (OSD_DOWN) 2023-03-09T14:51:19.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:18 smithi181 ceph-mon[94189]: osdmap e129: 12 total, 9 up, 12 in 2023-03-09T14:51:19.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:18 smithi038 ceph-mon[92422]: Health check update: 3 osds down (OSD_DOWN) 2023-03-09T14:51:19.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:18 smithi038 ceph-mon[92422]: osdmap e129: 12 total, 9 up, 12 in 2023-03-09T14:51:19.590 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:19 smithi122 systemd[1]: ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.1.service: Failed with result 'exit-code'. 2023-03-09T14:51:19.941 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:19 smithi122 bash[124639]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-03-09T14:51:19.941 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:19 smithi122 bash[124639]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2023-03-09T14:51:19.942 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:19 smithi122 bash[124639]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2023-03-09T14:51:19.942 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:19 smithi122 bash[124639]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2023-03-09T14:51:19.942 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:19 smithi122 bash[124639]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-4/block 2023-03-09T14:51:19.942 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:19 smithi122 bash[124639]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-03-09T14:51:19.943 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:19 smithi122 bash[124639]: --> ceph-volume raw activate successful for osd ID: 4 2023-03-09T14:51:19.943 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:19 smithi122 ceph-mon[93869]: pgmap v194: 129 pgs: 2 stale+active+undersized+degraded, 2 stale+active+undersized, 18 stale+active+clean, 1 active+recovering+degraded, 16 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 17 active+undersized+degraded, 72 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 8.8 MiB/s wr, 1.03k op/s; 2801/31846 objects degraded (8.795%); 117 MiB/s, 29 objects/s recovering 2023-03-09T14:51:19.944 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:19 smithi122 ceph-mon[93869]: osdmap e130: 12 total, 9 up, 12 in 2023-03-09T14:51:20.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:19 smithi181 ceph-mon[94189]: pgmap v194: 129 pgs: 2 stale+active+undersized+degraded, 2 stale+active+undersized, 18 stale+active+clean, 1 active+recovering+degraded, 16 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 17 active+undersized+degraded, 72 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 8.8 MiB/s wr, 1.03k op/s; 2801/31846 objects degraded (8.795%); 117 MiB/s, 29 objects/s recovering 2023-03-09T14:51:20.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:19 smithi181 ceph-mon[94189]: osdmap e130: 12 total, 9 up, 12 in 2023-03-09T14:51:20.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:19 smithi038 ceph-mon[92422]: pgmap v194: 129 pgs: 2 stale+active+undersized+degraded, 2 stale+active+undersized, 18 stale+active+clean, 1 active+recovering+degraded, 16 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 17 active+undersized+degraded, 72 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 8.8 MiB/s wr, 1.03k op/s; 2801/31846 objects degraded (8.795%); 117 MiB/s, 29 objects/s recovering 2023-03-09T14:51:20.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:19 smithi038 ceph-mon[92422]: osdmap e130: 12 total, 9 up, 12 in 2023-03-09T14:51:20.251 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:20 smithi038 systemd[1]: ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.3.service: Main process exited, code=exited, status=134/n/a 2023-03-09T14:51:20.597 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:20 smithi122 podman[124940]: 2023-03-09T14:51:20.597 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:20 smithi122 ceph-osd[124979]: -- 172.21.15.122:0/874227963 <== mon.1 v2:172.21.15.181:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558d801529c0 con 0x558d8011ac00 2023-03-09T14:51:20.978 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:20 smithi122 bash[124940]: 5d9c0d98bbebdf824f8c058a5b6dd8a213ffb11bdc5da540ed733625859ebbbe 2023-03-09T14:51:20.978 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:20 smithi122 systemd[1]: Started Ceph osd.4 for c6733fe2-be87-11ed-9af2-001a4aab830c. 2023-03-09T14:51:21.478 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:21 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[124955]: 2023-03-09T14:51:21.114+0000 7f2735866640 -1 Falling back to public interface 2023-03-09T14:51:21.750 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:21 smithi038 systemd[1]: ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.3.service: Failed with result 'exit-code'. 2023-03-09T14:51:22.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:21 smithi122 ceph-mon[93869]: pgmap v196: 129 pgs: 2 stale+active+undersized+degraded, 1 stale+active+undersized, 6 stale+active+clean, 4 undersized+peered, 32 active+undersized, 3 undersized+degraded+peered, 28 active+undersized+degraded, 53 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 10 MiB/s wr, 1.25k op/s; 5168/33074 objects degraded (15.626%); 194 MiB/s, 48 objects/s recovering 2023-03-09T14:51:22.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:21 smithi181 ceph-mon[94189]: pgmap v196: 129 pgs: 2 stale+active+undersized+degraded, 1 stale+active+undersized, 6 stale+active+clean, 4 undersized+peered, 32 active+undersized, 3 undersized+degraded+peered, 28 active+undersized+degraded, 53 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 10 MiB/s wr, 1.25k op/s; 5168/33074 objects degraded (15.626%); 194 MiB/s, 48 objects/s recovering 2023-03-09T14:51:22.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:21 smithi038 ceph-mon[92422]: pgmap v196: 129 pgs: 2 stale+active+undersized+degraded, 1 stale+active+undersized, 6 stale+active+clean, 4 undersized+peered, 32 active+undersized, 3 undersized+degraded+peered, 28 active+undersized+degraded, 53 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 10 MiB/s wr, 1.25k op/s; 5168/33074 objects degraded (15.626%); 194 MiB/s, 48 objects/s recovering 2023-03-09T14:51:23.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:22 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T14:51:23.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:22 smithi122 ceph-mon[93869]: Health check update: Degraded data redundancy: 5168/33074 objects degraded (15.626%), 33 pgs degraded (PG_DEGRADED) 2023-03-09T14:51:23.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:22 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T14:51:23.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:22 smithi181 ceph-mon[94189]: Health check update: Degraded data redundancy: 5168/33074 objects degraded (15.626%), 33 pgs degraded (PG_DEGRADED) 2023-03-09T14:51:23.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:22 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T14:51:23.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:22 smithi038 ceph-mon[92422]: Health check update: Degraded data redundancy: 5168/33074 objects degraded (15.626%), 33 pgs degraded (PG_DEGRADED) 2023-03-09T14:51:23.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 55.0 s, 1.8 tps, lat 2048.640 ms stddev 17.478 2023-03-09T14:51:24.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:23 smithi122 ceph-mon[93869]: pgmap v197: 129 pgs: 1 stale+active+undersized+degraded, 1 stale+active+undersized, 3 stale+active+clean, 5 undersized+peered, 37 active+undersized, 7 undersized+degraded+peered, 36 active+undersized+degraded, 39 active+clean; 18 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 7.1 MiB/s wr, 958 op/s; 7599/33446 objects degraded (22.720%); 150 MiB/s, 37 objects/s recovering 2023-03-09T14:51:24.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:23 smithi181 ceph-mon[94189]: pgmap v197: 129 pgs: 1 stale+active+undersized+degraded, 1 stale+active+undersized, 3 stale+active+clean, 5 undersized+peered, 37 active+undersized, 7 undersized+degraded+peered, 36 active+undersized+degraded, 39 active+clean; 18 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 7.1 MiB/s wr, 958 op/s; 7599/33446 objects degraded (22.720%); 150 MiB/s, 37 objects/s recovering 2023-03-09T14:51:24.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:23 smithi038 ceph-mon[92422]: pgmap v197: 129 pgs: 1 stale+active+undersized+degraded, 1 stale+active+undersized, 3 stale+active+clean, 5 undersized+peered, 37 active+undersized, 7 undersized+degraded+peered, 36 active+undersized+degraded, 39 active+clean; 18 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 7.1 MiB/s wr, 958 op/s; 7599/33446 objects degraded (22.720%); 150 MiB/s, 37 objects/s recovering 2023-03-09T14:51:25.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:25 smithi122 ceph-mon[93869]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2023-03-09T14:51:25.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:25 smithi181 ceph-mon[94189]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2023-03-09T14:51:25.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:25 smithi038 ceph-mon[92422]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2023-03-09T14:51:26.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:26 smithi122 ceph-mon[93869]: pgmap v198: 129 pgs: 6 undersized+peered, 42 active+undersized, 10 undersized+degraded+peered, 38 active+undersized+degraded, 33 active+clean; 18 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.8 MiB/s wr, 799 op/s; 8617/33564 objects degraded (25.673%); 126 MiB/s, 31 objects/s recovering 2023-03-09T14:51:26.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:26 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:51:26.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:26 smithi181 ceph-mon[94189]: pgmap v198: 129 pgs: 6 undersized+peered, 42 active+undersized, 10 undersized+degraded+peered, 38 active+undersized+degraded, 33 active+clean; 18 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.8 MiB/s wr, 799 op/s; 8617/33564 objects degraded (25.673%); 126 MiB/s, 31 objects/s recovering 2023-03-09T14:51:26.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:26 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:51:26.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:26 smithi038 ceph-mon[92422]: pgmap v198: 129 pgs: 6 undersized+peered, 42 active+undersized, 10 undersized+degraded+peered, 38 active+undersized+degraded, 33 active+clean; 18 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 5.8 MiB/s wr, 799 op/s; 8617/33564 objects degraded (25.673%); 126 MiB/s, 31 objects/s recovering 2023-03-09T14:51:26.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:26 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:51:27.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:51:27.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:51:27.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:51:27.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:51:27.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:51:27.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:51:27.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:51:27.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:51:27.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:51:28.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:28 smithi122 ceph-mon[93869]: pgmap v199: 129 pgs: 6 undersized+peered, 42 active+undersized, 10 undersized+degraded+peered, 38 active+undersized+degraded, 33 active+clean; 18 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 588 KiB/s rd, 3.3 MiB/s wr, 432 op/s; 8620/33579 objects degraded (25.671%); 64 MiB/s, 16 objects/s recovering 2023-03-09T14:51:28.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:28 smithi122 ceph-mon[93869]: Health check update: Degraded data redundancy: 8620/33579 objects degraded (25.671%), 48 pgs degraded (PG_DEGRADED) 2023-03-09T14:51:28.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:28 smithi181 ceph-mon[94189]: pgmap v199: 129 pgs: 6 undersized+peered, 42 active+undersized, 10 undersized+degraded+peered, 38 active+undersized+degraded, 33 active+clean; 18 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 588 KiB/s rd, 3.3 MiB/s wr, 432 op/s; 8620/33579 objects degraded (25.671%); 64 MiB/s, 16 objects/s recovering 2023-03-09T14:51:28.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:28 smithi181 ceph-mon[94189]: Health check update: Degraded data redundancy: 8620/33579 objects degraded (25.671%), 48 pgs degraded (PG_DEGRADED) 2023-03-09T14:51:28.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:28 smithi038 ceph-mon[92422]: pgmap v199: 129 pgs: 6 undersized+peered, 42 active+undersized, 10 undersized+degraded+peered, 38 active+undersized+degraded, 33 active+clean; 18 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 588 KiB/s rd, 3.3 MiB/s wr, 432 op/s; 8620/33579 objects degraded (25.671%); 64 MiB/s, 16 objects/s recovering 2023-03-09T14:51:28.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:28 smithi038 ceph-mon[92422]: Health check update: Degraded data redundancy: 8620/33579 objects degraded (25.671%), 48 pgs degraded (PG_DEGRADED) 2023-03-09T14:51:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 60.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T14:51:29.978 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:29 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[124955]: 2023-03-09T14:51:29.678+0000 7f2735866640 -1 osd.4 117 log_to_monitors true 2023-03-09T14:51:29.978 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:29 smithi122 systemd[1]: ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.1.service: Service RestartSec=10s expired, scheduling restart. 2023-03-09T14:51:29.979 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:29 smithi122 systemd[1]: ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.1.service: Scheduled restart job, restart counter is at 2. 2023-03-09T14:51:29.979 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:29 smithi122 systemd[1]: Stopped Ceph osd.1 for c6733fe2-be87-11ed-9af2-001a4aab830c. 2023-03-09T14:51:29.979 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:29 smithi122 systemd[1]: Starting Ceph osd.1 for c6733fe2-be87-11ed-9af2-001a4aab830c... 2023-03-09T14:51:30.478 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 14:51:30 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[124955]: 2023-03-09T14:51:30.106+0000 7f27255e0700 -1 osd.4 117 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-09T14:51:30.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:30 smithi122 ceph-mon[93869]: pgmap v200: 129 pgs: 6 undersized+peered, 42 active+undersized, 10 undersized+degraded+peered, 38 active+undersized+degraded, 33 active+clean; 18 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 514 KiB/s rd, 2.9 MiB/s wr, 378 op/s; 8620/33579 objects degraded (25.671%); 56 MiB/s, 13 objects/s recovering 2023-03-09T14:51:30.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:30 smithi122 ceph-mon[93869]: from='osd.4 [v2:172.21.15.122:6800/4029744437,v1:172.21.15.122:6801/4029744437]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-03-09T14:51:30.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:30 smithi122 ceph-mon[93869]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-03-09T14:51:30.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:30 smithi181 ceph-mon[94189]: pgmap v200: 129 pgs: 6 undersized+peered, 42 active+undersized, 10 undersized+degraded+peered, 38 active+undersized+degraded, 33 active+clean; 18 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 514 KiB/s rd, 2.9 MiB/s wr, 378 op/s; 8620/33579 objects degraded (25.671%); 56 MiB/s, 13 objects/s recovering 2023-03-09T14:51:30.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:30 smithi181 ceph-mon[94189]: from='osd.4 [v2:172.21.15.122:6800/4029744437,v1:172.21.15.122:6801/4029744437]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-03-09T14:51:30.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:30 smithi181 ceph-mon[94189]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-03-09T14:51:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:30 smithi038 ceph-mon[92422]: pgmap v200: 129 pgs: 6 undersized+peered, 42 active+undersized, 10 undersized+degraded+peered, 38 active+undersized+degraded, 33 active+clean; 18 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 514 KiB/s rd, 2.9 MiB/s wr, 378 op/s; 8620/33579 objects degraded (25.671%); 56 MiB/s, 13 objects/s recovering 2023-03-09T14:51:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:30 smithi038 ceph-mon[92422]: from='osd.4 [v2:172.21.15.122:6800/4029744437,v1:172.21.15.122:6801/4029744437]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-03-09T14:51:30.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:30 smithi038 ceph-mon[92422]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-03-09T14:51:30.978 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:30 smithi122 bash[125997]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-03-09T14:51:30.978 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:30 smithi122 bash[125997]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2023-03-09T14:51:30.979 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:30 smithi122 bash[125997]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2023-03-09T14:51:30.979 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:30 smithi122 bash[125997]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2023-03-09T14:51:30.979 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:30 smithi122 bash[125997]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-1/block 2023-03-09T14:51:30.979 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:30 smithi122 bash[125997]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-03-09T14:51:30.979 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:30 smithi122 bash[125997]: --> ceph-volume raw activate successful for osd ID: 1 2023-03-09T14:51:31.409 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:31 smithi122 ceph-mon[93869]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-03-09T14:51:31.410 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:31 smithi122 ceph-mon[93869]: osdmap e131: 12 total, 9 up, 12 in 2023-03-09T14:51:31.410 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:31 smithi122 ceph-mon[93869]: from='osd.4 [v2:172.21.15.122:6800/4029744437,v1:172.21.15.122:6801/4029744437]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-03-09T14:51:31.410 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:31 smithi122 ceph-mon[93869]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-03-09T14:51:31.411 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:31 smithi122 podman[126275]: 2023-03-09T14:51:31.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:31 smithi181 ceph-mon[94189]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-03-09T14:51:31.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:31 smithi181 ceph-mon[94189]: osdmap e131: 12 total, 9 up, 12 in 2023-03-09T14:51:31.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:31 smithi181 ceph-mon[94189]: from='osd.4 [v2:172.21.15.122:6800/4029744437,v1:172.21.15.122:6801/4029744437]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-03-09T14:51:31.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:31 smithi181 ceph-mon[94189]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-03-09T14:51:31.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:31 smithi038 ceph-mon[92422]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-03-09T14:51:31.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:31 smithi038 ceph-mon[92422]: osdmap e131: 12 total, 9 up, 12 in 2023-03-09T14:51:31.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:31 smithi038 ceph-mon[92422]: from='osd.4 [v2:172.21.15.122:6800/4029744437,v1:172.21.15.122:6801/4029744437]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-03-09T14:51:31.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:31 smithi038 ceph-mon[92422]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-03-09T14:51:31.728 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:31 smithi122 bash[126275]: 17a3996eb6a80d35faadc53acdfe6dacb1535e35c1045930852659826b6c9a3b 2023-03-09T14:51:31.728 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:31 smithi122 ceph-osd[126315]: -- 172.21.15.122:0/2657078159 <== mon.1 v2:172.21.15.181:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c2d40929c0 con 0x55c2d405b400 2023-03-09T14:51:31.728 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:31 smithi122 systemd[1]: Started Ceph osd.1 for c6733fe2-be87-11ed-9af2-001a4aab830c. 2023-03-09T14:51:31.914 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:31 smithi038 systemd[1]: ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.3.service: Service RestartSec=10s expired, scheduling restart. 2023-03-09T14:51:31.914 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:31 smithi038 systemd[1]: ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.3.service: Scheduled restart job, restart counter is at 2. 2023-03-09T14:51:31.914 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:31 smithi038 systemd[1]: Stopped Ceph osd.3 for c6733fe2-be87-11ed-9af2-001a4aab830c. 2023-03-09T14:51:31.914 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:31 smithi038 systemd[1]: Starting Ceph osd.3 for c6733fe2-be87-11ed-9af2-001a4aab830c... 2023-03-09T14:51:32.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:32 smithi038 ceph-mon[92422]: pgmap v202: 129 pgs: 6 undersized+peered, 42 active+undersized, 10 undersized+degraded+peered, 38 active+undersized+degraded, 33 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 1.3 MiB/s wr, 155 op/s; 8621/33587 objects degraded (25.668%) 2023-03-09T14:51:32.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:32 smithi038 ceph-mon[92422]: Health check update: 2 osds down (OSD_DOWN) 2023-03-09T14:51:32.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:32 smithi038 ceph-mon[92422]: osd.4 [v2:172.21.15.122:6800/4029744437,v1:172.21.15.122:6801/4029744437] boot 2023-03-09T14:51:32.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:32 smithi038 ceph-mon[92422]: osdmap e132: 12 total, 10 up, 12 in 2023-03-09T14:51:32.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-03-09T14:51:32.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:32 smithi038 ceph-mon[92422]: osdmap e133: 12 total, 10 up, 12 in 2023-03-09T14:51:32.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:32 smithi122 ceph-mon[93869]: pgmap v202: 129 pgs: 6 undersized+peered, 42 active+undersized, 10 undersized+degraded+peered, 38 active+undersized+degraded, 33 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 1.3 MiB/s wr, 155 op/s; 8621/33587 objects degraded (25.668%) 2023-03-09T14:51:32.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:32 smithi122 ceph-mon[93869]: Health check update: 2 osds down (OSD_DOWN) 2023-03-09T14:51:32.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:32 smithi122 ceph-mon[93869]: osd.4 [v2:172.21.15.122:6800/4029744437,v1:172.21.15.122:6801/4029744437] boot 2023-03-09T14:51:32.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:32 smithi122 ceph-mon[93869]: osdmap e132: 12 total, 10 up, 12 in 2023-03-09T14:51:32.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-03-09T14:51:32.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:32 smithi122 ceph-mon[93869]: osdmap e133: 12 total, 10 up, 12 in 2023-03-09T14:51:32.479 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:32 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 2023-03-09T14:51:32.094+0000 7ff220838640 -1 Falling back to public interface 2023-03-09T14:51:32.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:32 smithi181 ceph-mon[94189]: pgmap v202: 129 pgs: 6 undersized+peered, 42 active+undersized, 10 undersized+degraded+peered, 38 active+undersized+degraded, 33 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 1.3 MiB/s wr, 155 op/s; 8621/33587 objects degraded (25.668%) 2023-03-09T14:51:32.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:32 smithi181 ceph-mon[94189]: Health check update: 2 osds down (OSD_DOWN) 2023-03-09T14:51:32.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:32 smithi181 ceph-mon[94189]: osd.4 [v2:172.21.15.122:6800/4029744437,v1:172.21.15.122:6801/4029744437] boot 2023-03-09T14:51:32.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:32 smithi181 ceph-mon[94189]: osdmap e132: 12 total, 10 up, 12 in 2023-03-09T14:51:32.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-03-09T14:51:32.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:32 smithi181 ceph-mon[94189]: osdmap e133: 12 total, 10 up, 12 in 2023-03-09T14:51:33.001 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:32 smithi038 bash[149258]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-03-09T14:51:33.066 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:32 smithi038 bash[149258]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2023-03-09T14:51:33.066 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:32 smithi038 bash[149258]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2023-03-09T14:51:33.067 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:32 smithi038 bash[149258]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2023-03-09T14:51:33.067 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:32 smithi038 bash[149258]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-3/block 2023-03-09T14:51:33.067 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:32 smithi038 bash[149258]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-03-09T14:51:33.067 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:32 smithi038 bash[149258]: --> ceph-volume raw activate successful for osd ID: 3 2023-03-09T14:51:33.405 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:33 smithi038 podman[149545]: 2023-03-09T14:51:33.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:33 smithi038 ceph-mon[92422]: Health check update: Degraded data redundancy: 8621/33587 objects degraded (25.668%), 48 pgs degraded (PG_DEGRADED) 2023-03-09T14:51:33.405 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:33 smithi038 ceph-mon[92422]: pgmap v205: 129 pgs: 1 remapped+peering, 1 peering, 1 active+recovering+undersized+degraded, 5 undersized+peered, 39 active+undersized, 8 undersized+degraded+peered, 36 active+undersized+degraded, 38 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 6.5 KiB/s wr, 8 op/s; 7603/33587 objects degraded (22.637%); 2.7 MiB/s, 0 objects/s recovering 2023-03-09T14:51:33.406 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:33 smithi038 ceph-mon[92422]: osdmap e134: 12 total, 10 up, 12 in 2023-03-09T14:51:33.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:33 smithi122 ceph-mon[93869]: Health check update: Degraded data redundancy: 8621/33587 objects degraded (25.668%), 48 pgs degraded (PG_DEGRADED) 2023-03-09T14:51:33.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:33 smithi122 ceph-mon[93869]: pgmap v205: 129 pgs: 1 remapped+peering, 1 peering, 1 active+recovering+undersized+degraded, 5 undersized+peered, 39 active+undersized, 8 undersized+degraded+peered, 36 active+undersized+degraded, 38 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 6.5 KiB/s wr, 8 op/s; 7603/33587 objects degraded (22.637%); 2.7 MiB/s, 0 objects/s recovering 2023-03-09T14:51:33.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:33 smithi122 ceph-mon[93869]: osdmap e134: 12 total, 10 up, 12 in 2023-03-09T14:51:33.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:33 smithi181 ceph-mon[94189]: Health check update: Degraded data redundancy: 8621/33587 objects degraded (25.668%), 48 pgs degraded (PG_DEGRADED) 2023-03-09T14:51:33.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:33 smithi181 ceph-mon[94189]: pgmap v205: 129 pgs: 1 remapped+peering, 1 peering, 1 active+recovering+undersized+degraded, 5 undersized+peered, 39 active+undersized, 8 undersized+degraded+peered, 36 active+undersized+degraded, 38 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 6.5 KiB/s wr, 8 op/s; 7603/33587 objects degraded (22.637%); 2.7 MiB/s, 0 objects/s recovering 2023-03-09T14:51:33.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:33 smithi181 ceph-mon[94189]: osdmap e134: 12 total, 10 up, 12 in 2023-03-09T14:51:33.751 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:33 smithi038 ceph-osd[149585]: -- 172.21.15.38:0/2276926453 <== mon.0 v2:172.21.15.38:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b58b5089c0 con 0x55b58b4d0c00 2023-03-09T14:51:33.751 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:33 smithi038 bash[149545]: 01d5b2b681e4b7a04583fe119bebbe153be068b0738dee2e19b98204ccb98f0d 2023-03-09T14:51:33.751 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:33 smithi038 systemd[1]: Started Ceph osd.3 for c6733fe2-be87-11ed-9af2-001a4aab830c. 2023-03-09T14:51:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 65.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T14:51:34.250 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:33 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 2023-03-09T14:51:33.920+0000 7f8c96d68640 -1 Falling back to public interface 2023-03-09T14:51:35.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:35 smithi122 ceph-mon[93869]: pgmap v207: 129 pgs: 2 remapped+peering, 1 peering, 1 active+recovery_wait+undersized+degraded, 1 active+recovering+undersized+degraded, 5 undersized+peered, 38 active+undersized, 6 undersized+degraded+peered, 36 active+undersized+degraded, 39 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.4 KiB/s wr, 9 op/s; 6843/33587 objects degraded (20.374%); 14 MiB/s, 3 objects/s recovering 2023-03-09T14:51:35.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:35 smithi181 ceph-mon[94189]: pgmap v207: 129 pgs: 2 remapped+peering, 1 peering, 1 active+recovery_wait+undersized+degraded, 1 active+recovering+undersized+degraded, 5 undersized+peered, 38 active+undersized, 6 undersized+degraded+peered, 36 active+undersized+degraded, 39 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.4 KiB/s wr, 9 op/s; 6843/33587 objects degraded (20.374%); 14 MiB/s, 3 objects/s recovering 2023-03-09T14:51:36.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:35 smithi038 ceph-mon[92422]: pgmap v207: 129 pgs: 2 remapped+peering, 1 peering, 1 active+recovery_wait+undersized+degraded, 1 active+recovering+undersized+degraded, 5 undersized+peered, 38 active+undersized, 6 undersized+degraded+peered, 36 active+undersized+degraded, 39 active+clean; 18 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.4 KiB/s wr, 9 op/s; 6843/33587 objects degraded (20.374%); 14 MiB/s, 3 objects/s recovering 2023-03-09T14:51:36.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:36 smithi122 ceph-mon[93869]: osdmap e135: 12 total, 10 up, 12 in 2023-03-09T14:51:36.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:36 smithi181 ceph-mon[94189]: osdmap e135: 12 total, 10 up, 12 in 2023-03-09T14:51:37.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:36 smithi038 ceph-mon[92422]: osdmap e135: 12 total, 10 up, 12 in 2023-03-09T14:51:37.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:37 smithi122 ceph-mon[93869]: pgmap v209: 129 pgs: 2 active+recovery_wait+degraded, 2 remapped+peering, 1 active+recovery_wait+undersized+degraded, 1 active+recovering+undersized+degraded, 2 undersized+peered, 32 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait, 3 undersized+degraded+peered, 30 active+undersized+degraded, 53 active+clean; 18 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 23 KiB/s wr, 9 op/s; 5797/33587 objects degraded (17.260%); 115 MiB/s, 9 keys/s, 29 objects/s recovering 2023-03-09T14:51:37.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:37 smithi122 ceph-mon[93869]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2023-03-09T14:51:37.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:37 smithi122 ceph-mon[93869]: osdmap e136: 12 total, 10 up, 12 in 2023-03-09T14:51:37.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:37 smithi122 ceph-mon[93869]: Health check update: Degraded data redundancy: 5797/33587 objects degraded (17.260%), 38 pgs degraded (PG_DEGRADED) 2023-03-09T14:51:37.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:37 smithi181 ceph-mon[94189]: pgmap v209: 129 pgs: 2 active+recovery_wait+degraded, 2 remapped+peering, 1 active+recovery_wait+undersized+degraded, 1 active+recovering+undersized+degraded, 2 undersized+peered, 32 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait, 3 undersized+degraded+peered, 30 active+undersized+degraded, 53 active+clean; 18 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 23 KiB/s wr, 9 op/s; 5797/33587 objects degraded (17.260%); 115 MiB/s, 9 keys/s, 29 objects/s recovering 2023-03-09T14:51:37.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:37 smithi181 ceph-mon[94189]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2023-03-09T14:51:37.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:37 smithi181 ceph-mon[94189]: osdmap e136: 12 total, 10 up, 12 in 2023-03-09T14:51:37.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:37 smithi181 ceph-mon[94189]: Health check update: Degraded data redundancy: 5797/33587 objects degraded (17.260%), 38 pgs degraded (PG_DEGRADED) 2023-03-09T14:51:38.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:37 smithi038 ceph-mon[92422]: pgmap v209: 129 pgs: 2 active+recovery_wait+degraded, 2 remapped+peering, 1 active+recovery_wait+undersized+degraded, 1 active+recovering+undersized+degraded, 2 undersized+peered, 32 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait, 3 undersized+degraded+peered, 30 active+undersized+degraded, 53 active+clean; 18 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 23 KiB/s wr, 9 op/s; 5797/33587 objects degraded (17.260%); 115 MiB/s, 9 keys/s, 29 objects/s recovering 2023-03-09T14:51:38.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:37 smithi038 ceph-mon[92422]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2023-03-09T14:51:38.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:37 smithi038 ceph-mon[92422]: osdmap e136: 12 total, 10 up, 12 in 2023-03-09T14:51:38.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:37 smithi038 ceph-mon[92422]: Health check update: Degraded data redundancy: 5797/33587 objects degraded (17.260%), 38 pgs degraded (PG_DEGRADED) 2023-03-09T14:51:38.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 70.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T14:51:39.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:39 smithi122 ceph-mon[93869]: pgmap v211: 129 pgs: 2 active+recovery_wait+degraded, 2 undersized+peered, 32 active+undersized, 1 recovering+undersized+degraded+remapped+peered, 1 recovery_wait+undersized+degraded+remapped+peered, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait, 3 undersized+degraded+peered, 32 active+undersized+degraded, 53 active+clean; 18 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 17 KiB/s wr, 7 op/s; 6431/33587 objects degraded (19.147%); 183 MiB/s, 8 keys/s, 46 objects/s recovering 2023-03-09T14:51:39.978 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:39 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 2023-03-09T14:51:39.887+0000 7ff220838640 -1 osd.1 128 log_to_monitors true 2023-03-09T14:51:39.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:39 smithi181 ceph-mon[94189]: pgmap v211: 129 pgs: 2 active+recovery_wait+degraded, 2 undersized+peered, 32 active+undersized, 1 recovering+undersized+degraded+remapped+peered, 1 recovery_wait+undersized+degraded+remapped+peered, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait, 3 undersized+degraded+peered, 32 active+undersized+degraded, 53 active+clean; 18 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 17 KiB/s wr, 7 op/s; 6431/33587 objects degraded (19.147%); 183 MiB/s, 8 keys/s, 46 objects/s recovering 2023-03-09T14:51:40.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:39 smithi038 ceph-mon[92422]: pgmap v211: 129 pgs: 2 active+recovery_wait+degraded, 2 undersized+peered, 32 active+undersized, 1 recovering+undersized+degraded+remapped+peered, 1 recovery_wait+undersized+degraded+remapped+peered, 1 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait, 3 undersized+degraded+peered, 32 active+undersized+degraded, 53 active+clean; 18 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 17 KiB/s wr, 7 op/s; 6431/33587 objects degraded (19.147%); 183 MiB/s, 8 keys/s, 46 objects/s recovering 2023-03-09T14:51:40.000 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:39 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 2023-03-09T14:51:39.905+0000 7f8c96d68640 -1 osd.3 128 log_to_monitors true 2023-03-09T14:51:40.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:40 smithi122 ceph-mon[93869]: from='osd.1 [v2:172.21.15.122:6808/216712231,v1:172.21.15.122:6809/216712231]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-03-09T14:51:40.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:40 smithi122 ceph-mon[93869]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-03-09T14:51:40.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:40 smithi122 ceph-mon[93869]: from='osd.3 [v2:172.21.15.38:6810/911235164,v1:172.21.15.38:6811/911235164]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-03-09T14:51:40.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:40 smithi122 ceph-mon[93869]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-03-09T14:51:40.979 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:40 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 2023-03-09T14:51:40.759+0000 7ff20c5aa700 -1 osd.1 128 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-09T14:51:40.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:40 smithi181 ceph-mon[94189]: from='osd.1 [v2:172.21.15.122:6808/216712231,v1:172.21.15.122:6809/216712231]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-03-09T14:51:40.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:40 smithi181 ceph-mon[94189]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-03-09T14:51:40.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:40 smithi181 ceph-mon[94189]: from='osd.3 [v2:172.21.15.38:6810/911235164,v1:172.21.15.38:6811/911235164]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-03-09T14:51:40.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:40 smithi181 ceph-mon[94189]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-03-09T14:51:41.001 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:40 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 2023-03-09T14:51:40.759+0000 7f8c82ada700 -1 osd.3 128 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-09T14:51:41.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:40 smithi038 ceph-mon[92422]: from='osd.1 [v2:172.21.15.122:6808/216712231,v1:172.21.15.122:6809/216712231]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-03-09T14:51:41.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:40 smithi038 ceph-mon[92422]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-03-09T14:51:41.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:40 smithi038 ceph-mon[92422]: from='osd.3 [v2:172.21.15.38:6810/911235164,v1:172.21.15.38:6811/911235164]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-03-09T14:51:41.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:40 smithi038 ceph-mon[92422]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-03-09T14:51:42.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:41 smithi122 ceph-mon[93869]: pgmap v212: 129 pgs: 2 active+recovery_wait+degraded, 2 undersized+peered, 32 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 1 recovering+undersized+degraded+remapped+peered, 1 recovery_wait+undersized+degraded+remapped+peered, 2 active+recovery_wait, 2 undersized+degraded+peered, 33 active+undersized+degraded, 53 active+clean; 18 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 76 KiB/s wr, 16 op/s; 6171/33595 objects degraded (18.369%); 139 MiB/s, 7 keys/s, 35 objects/s recovering 2023-03-09T14:51:42.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:41 smithi122 ceph-mon[93869]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-03-09T14:51:42.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:41 smithi122 ceph-mon[93869]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-03-09T14:51:42.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:41 smithi122 ceph-mon[93869]: osdmap e137: 12 total, 10 up, 12 in 2023-03-09T14:51:42.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:41 smithi122 ceph-mon[93869]: from='osd.1 [v2:172.21.15.122:6808/216712231,v1:172.21.15.122:6809/216712231]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-03-09T14:51:42.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:41 smithi122 ceph-mon[93869]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-03-09T14:51:42.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:41 smithi122 ceph-mon[93869]: from='osd.3 [v2:172.21.15.38:6810/911235164,v1:172.21.15.38:6811/911235164]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2023-03-09T14:51:42.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:41 smithi122 ceph-mon[93869]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2023-03-09T14:51:42.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:41 smithi181 ceph-mon[94189]: pgmap v212: 129 pgs: 2 active+recovery_wait+degraded, 2 undersized+peered, 32 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 1 recovering+undersized+degraded+remapped+peered, 1 recovery_wait+undersized+degraded+remapped+peered, 2 active+recovery_wait, 2 undersized+degraded+peered, 33 active+undersized+degraded, 53 active+clean; 18 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 76 KiB/s wr, 16 op/s; 6171/33595 objects degraded (18.369%); 139 MiB/s, 7 keys/s, 35 objects/s recovering 2023-03-09T14:51:42.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:41 smithi181 ceph-mon[94189]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-03-09T14:51:42.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:41 smithi181 ceph-mon[94189]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-03-09T14:51:42.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:41 smithi181 ceph-mon[94189]: osdmap e137: 12 total, 10 up, 12 in 2023-03-09T14:51:42.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:41 smithi181 ceph-mon[94189]: from='osd.1 [v2:172.21.15.122:6808/216712231,v1:172.21.15.122:6809/216712231]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-03-09T14:51:42.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:41 smithi181 ceph-mon[94189]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-03-09T14:51:42.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:41 smithi181 ceph-mon[94189]: from='osd.3 [v2:172.21.15.38:6810/911235164,v1:172.21.15.38:6811/911235164]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2023-03-09T14:51:42.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:41 smithi181 ceph-mon[94189]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2023-03-09T14:51:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:41 smithi038 ceph-mon[92422]: pgmap v212: 129 pgs: 2 active+recovery_wait+degraded, 2 undersized+peered, 32 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 1 recovering+undersized+degraded+remapped+peered, 1 recovery_wait+undersized+degraded+remapped+peered, 2 active+recovery_wait, 2 undersized+degraded+peered, 33 active+undersized+degraded, 53 active+clean; 18 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 76 KiB/s wr, 16 op/s; 6171/33595 objects degraded (18.369%); 139 MiB/s, 7 keys/s, 35 objects/s recovering 2023-03-09T14:51:42.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:41 smithi038 ceph-mon[92422]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-03-09T14:51:42.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:41 smithi038 ceph-mon[92422]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-03-09T14:51:42.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:41 smithi038 ceph-mon[92422]: osdmap e137: 12 total, 10 up, 12 in 2023-03-09T14:51:42.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:41 smithi038 ceph-mon[92422]: from='osd.1 [v2:172.21.15.122:6808/216712231,v1:172.21.15.122:6809/216712231]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-03-09T14:51:42.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:41 smithi038 ceph-mon[92422]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-03-09T14:51:42.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:41 smithi038 ceph-mon[92422]: from='osd.3 [v2:172.21.15.38:6810/911235164,v1:172.21.15.38:6811/911235164]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2023-03-09T14:51:42.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:41 smithi038 ceph-mon[92422]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2023-03-09T14:51:42.820 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-03-09T14:51:42.981 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.977+0000 7f4c67c26700 1 -- 172.21.15.38:0/3631568239 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c60100b20 msgr2=0x7f4c60104ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:51:42.981 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.977+0000 7f4c67c26700 1 --2- 172.21.15.38:0/3631568239 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c60100b20 0x7f4c60104ae0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f4c50009920 tx=0x7f4c50009c30 comp rx=0 tx=0).stop 2023-03-09T14:51:42.982 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.977+0000 7f4c67c26700 1 -- 172.21.15.38:0/3631568239 shutdown_connections 2023-03-09T14:51:42.982 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.977+0000 7f4c67c26700 1 --2- 172.21.15.38:0/3631568239 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4c601051c0 0x7f4c6010cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:51:42.982 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.977+0000 7f4c67c26700 1 --2- 172.21.15.38:0/3631568239 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c60100180 0x7f4c60100550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:51:42.982 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.977+0000 7f4c67c26700 1 --2- 172.21.15.38:0/3631568239 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c60100b20 0x7f4c60104ae0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:51:42.982 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.977+0000 7f4c67c26700 1 -- 172.21.15.38:0/3631568239 >> 172.21.15.38:0/3631568239 conn(0x7f4c600fb9d0 msgr2=0x7f4c600fdde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:51:42.983 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.977+0000 7f4c67c26700 1 -- 172.21.15.38:0/3631568239 shutdown_connections 2023-03-09T14:51:42.983 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.977+0000 7f4c67c26700 1 -- 172.21.15.38:0/3631568239 wait complete. 2023-03-09T14:51:42.983 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.978+0000 7f4c67c26700 1 Processor -- start 2023-03-09T14:51:42.983 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.978+0000 7f4c67c26700 1 -- start start 2023-03-09T14:51:42.984 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.978+0000 7f4c67c26700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c60100180 0x7f4c60196110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:51:42.985 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.978+0000 7f4c67c26700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4c60100b20 0x7f4c60196650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:51:42.985 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.978+0000 7f4c67c26700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c601051c0 0x7f4c6019aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:51:42.985 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.978+0000 7f4c67c26700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c6019b0f0 con 0x7f4c60100180 2023-03-09T14:51:42.985 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.978+0000 7f4c67c26700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c6019b260 con 0x7f4c60100b20 2023-03-09T14:51:42.985 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.978+0000 7f4c67c26700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c6019b420 con 0x7f4c601051c0 2023-03-09T14:51:42.985 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.978+0000 7f4c659c2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c60100180 0x7f4c60196110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:51:42.985 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.979+0000 7f4c651c1700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4c60100b20 0x7f4c60196650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:51:42.986 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.979+0000 7f4c659c2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c60100180 0x7f4c60196110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:40714/0 (socket says 172.21.15.38:40714) 2023-03-09T14:51:42.986 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.979+0000 7f4c661c3700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c601051c0 0x7f4c6019aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:51:42.986 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.979+0000 7f4c659c2700 1 -- 172.21.15.38:0/554054776 learned_addr learned my addr 172.21.15.38:0/554054776 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:51:42.986 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.979+0000 7f4c651c1700 1 -- 172.21.15.38:0/554054776 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c601051c0 msgr2=0x7f4c6019aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:51:42.986 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.979+0000 7f4c651c1700 1 --2- 172.21.15.38:0/554054776 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c601051c0 0x7f4c6019aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:51:42.986 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.979+0000 7f4c651c1700 1 -- 172.21.15.38:0/554054776 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c60100180 msgr2=0x7f4c60196110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:51:42.986 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.979+0000 7f4c651c1700 1 --2- 172.21.15.38:0/554054776 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c60100180 0x7f4c60196110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:51:42.987 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.979+0000 7f4c651c1700 1 -- 172.21.15.38:0/554054776 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c5c0095f0 con 0x7f4c60100b20 2023-03-09T14:51:42.987 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.979+0000 7f4c659c2700 1 --2- 172.21.15.38:0/554054776 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c60100180 0x7f4c60196110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:51:42.987 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.979+0000 7f4c661c3700 1 --2- 172.21.15.38:0/554054776 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c601051c0 0x7f4c6019aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:51:42.987 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.979+0000 7f4c651c1700 1 --2- 172.21.15.38:0/554054776 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4c60100b20 0x7f4c60196650 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4c50000c00 tx=0x7f4c5000fa00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:51:42.987 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.979+0000 7f4c56ffd700 1 -- 172.21.15.38:0/554054776 <== mon.2 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c5001c070 con 0x7f4c60100b20 2023-03-09T14:51:42.987 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.979+0000 7f4c67c26700 1 -- 172.21.15.38:0/554054776 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c50009520 con 0x7f4c60100b20 2023-03-09T14:51:42.987 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.979+0000 7f4c56ffd700 1 -- 172.21.15.38:0/554054776 <== mon.2 v2:172.21.15.122:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4c5000bbe0 con 0x7f4c60100b20 2023-03-09T14:51:42.987 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.979+0000 7f4c56ffd700 1 -- 172.21.15.38:0/554054776 <== mon.2 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c500038e0 con 0x7f4c60100b20 2023-03-09T14:51:42.988 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.979+0000 7f4c67c26700 1 -- 172.21.15.38:0/554054776 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c6010ff40 con 0x7f4c60100b20 2023-03-09T14:51:42.990 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.983+0000 7f4c56ffd700 1 -- 172.21.15.38:0/554054776 <== mon.2 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 25) v1 ==== 133295+0+0 (secure 0 0 0) 0x7f4c50029020 con 0x7f4c60100b20 2023-03-09T14:51:42.990 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.983+0000 7f4c73dafb80 1 Processor -- start 2023-03-09T14:51:42.990 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.983+0000 7f4c73dafb80 1 -- start start 2023-03-09T14:51:42.990 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.984+0000 7f4c73dafb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555cdea8c830 0x555cdea8cc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:51:42.990 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.984+0000 7f4c73dafb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555cdea8d140 0x555cdea86aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:51:42.991 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.984+0000 7f4c73dafb80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555cdea86fe0 0x555cdea87410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:51:42.991 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.984+0000 7f4c73dafb80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cde970da0 con 0x555cdea86fe0 2023-03-09T14:51:42.991 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.984+0000 7f4c73dafb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cde9df780 con 0x555cdea8d140 2023-03-09T14:51:42.991 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.984+0000 7f4c73dafb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cde9d8cb0 con 0x555cdea8c830 2023-03-09T14:51:42.991 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.984+0000 7f4c56ffd700 1 --2- 172.21.15.38:0/554054776 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f4c4c09f4a0 0x7f4c4c0a1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:51:42.992 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.984+0000 7f4c659c2700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555cdea8d140 0x555cdea86aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:51:42.992 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.984+0000 7f4c659c2700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555cdea8d140 0x555cdea86aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.38:49912/0 (socket says 172.21.15.38:49912) 2023-03-09T14:51:42.992 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.984+0000 7f4c659c2700 1 -- 172.21.15.38:0/1083476215 learned_addr learned my addr 172.21.15.38:0/1083476215 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:51:42.992 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.984+0000 7f4c661c3700 1 --2- 172.21.15.38:0/1083476215 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555cdea8c830 0x555cdea8cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:51:42.992 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.984+0000 7f4c56ffd700 1 -- 172.21.15.38:0/554054776 <== mon.2 v2:172.21.15.122:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 9526+0+0 (secure 0 0 0) 0x7f4c500cb330 con 0x7f4c60100b20 2023-03-09T14:51:42.992 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.984+0000 7f4c659c2700 1 -- 172.21.15.38:0/1083476215 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555cdea8c830 msgr2=0x555cdea8cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:51:42.993 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.984+0000 7f4c659c2700 1 --2- 172.21.15.38:0/1083476215 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555cdea8c830 0x555cdea8cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:51:42.993 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.984+0000 7f4c659c2700 1 -- 172.21.15.38:0/1083476215 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555cdea86fe0 msgr2=0x555cdea87410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:51:42.993 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.984+0000 7f4c659c2700 1 --2- 172.21.15.38:0/1083476215 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555cdea86fe0 0x555cdea87410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:51:42.993 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.984+0000 7f4c659c2700 1 -- 172.21.15.38:0/1083476215 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555cde99f190 con 0x555cdea8d140 2023-03-09T14:51:42.993 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.984+0000 7f4c661c3700 1 --2- 172.21.15.38:0/554054776 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f4c4c09f4a0 0x7f4c4c0a1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:51:42.993 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.984+0000 7f4c659c2700 1 --2- 172.21.15.38:0/1083476215 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555cdea8d140 0x555cdea86aa0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c0118d0 tx=0x7f4c5c011be0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d710d49847aaeaad server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:51:42.993 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.985+0000 7f4c4bfff700 1 -- 172.21.15.38:0/1083476215 <== mon.2 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c5c011250 con 0x555cdea8d140 2023-03-09T14:51:42.994 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.985+0000 7f4c661c3700 1 --2- 172.21.15.38:0/554054776 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f4c4c09f4a0 0x7f4c4c0a1950 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f4c58009fd0 tx=0x7f4c58004ad0 comp rx=0 tx=0).ready entity=mgr.15558 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:51:42.994 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.985+0000 7f4c4bfff700 1 -- 172.21.15.38:0/1083476215 <== mon.2 v2:172.21.15.122:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4c5c004cf0 con 0x555cdea8d140 2023-03-09T14:51:42.994 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.985+0000 7f4c4bfff700 1 -- 172.21.15.38:0/1083476215 <== mon.2 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c5c00f950 con 0x555cdea8d140 2023-03-09T14:51:42.994 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.985+0000 7f4c73dafb80 1 -- 172.21.15.38:0/1083476215 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555cdea8d140 msgr2=0x555cdea86aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:51:42.994 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.985+0000 7f4c73dafb80 1 --2- 172.21.15.38:0/1083476215 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555cdea8d140 0x555cdea86aa0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c0118d0 tx=0x7f4c5c011be0 comp rx=0 tx=0).stop 2023-03-09T14:51:42.994 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.986+0000 7f4c73dafb80 1 -- 172.21.15.38:0/1083476215 shutdown_connections 2023-03-09T14:51:42.994 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.986+0000 7f4c73dafb80 1 --2- 172.21.15.38:0/1083476215 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555cdea86fe0 0x555cdea87410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:51:42.995 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.986+0000 7f4c73dafb80 1 --2- 172.21.15.38:0/1083476215 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555cdea8c830 0x555cdea8cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:51:42.995 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.986+0000 7f4c73dafb80 1 --2- 172.21.15.38:0/1083476215 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555cdea8d140 0x555cdea86aa0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:51:42.995 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.986+0000 7f4c73dafb80 1 -- 172.21.15.38:0/1083476215 >> 172.21.15.38:0/1083476215 conn(0x555cdea81a20 msgr2=0x555cdea81df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:51:42.995 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.986+0000 7f4c73dafb80 1 -- 172.21.15.38:0/1083476215 shutdown_connections 2023-03-09T14:51:42.995 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.986+0000 7f4c73dafb80 1 -- 172.21.15.38:0/1083476215 wait complete. 2023-03-09T14:51:42.995 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.986+0000 7f4c73dafb80 1 -- start start 2023-03-09T14:51:42.995 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.986+0000 7f4c73dafb80 1 Processor -- start 2023-03-09T14:51:42.995 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.987+0000 7f4c73dafb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555cdea86fe0 0x555cdea93120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:51:42.996 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.987+0000 7f4c73dafb80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555cdea8c830 0x555cdea903a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:51:42.996 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.987+0000 7f4c73dafb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555cdea8d140 0x555cdea90ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:51:42.996 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.987+0000 7f4c73dafb80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cde85c920 con 0x555cdea8c830 2023-03-09T14:51:42.996 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.987+0000 7f4c73dafb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cdea91250 con 0x555cdea86fe0 2023-03-09T14:51:42.996 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.987+0000 7f4c73dafb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cdeaa1050 con 0x555cdea8d140 2023-03-09T14:51:42.996 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.987+0000 7f4c73dafb80 10 client.0 _ll_register_callbacks cb 0x555cde97e520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:51:42.998 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.987+0000 7f4c651c1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555cdea8c830 0x555cdea903a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:51:42.999 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.987+0000 7f4c659c2700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555cdea86fe0 0x555cdea93120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:51:42.999 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.987+0000 7f4c659c2700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555cdea86fe0 0x555cdea93120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.38:49918/0 (socket says 172.21.15.38:49918) 2023-03-09T14:51:42.999 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.987+0000 7f4c659c2700 1 -- 172.21.15.38:0/2559049460 learned_addr learned my addr 172.21.15.38:0/2559049460 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:51:42.999 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.987+0000 7f4c651c1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555cdea8c830 0x555cdea903a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:40728/0 (socket says 172.21.15.38:40728) 2023-03-09T14:51:42.999 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.987+0000 7f4c661c3700 1 --2- 172.21.15.38:0/2559049460 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555cdea8d140 0x555cdea90ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:51:42.999 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.987+0000 7f4c659c2700 1 -- 172.21.15.38:0/2559049460 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555cdea8d140 msgr2=0x555cdea90ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:51:43.000 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.987+0000 7f4c659c2700 1 --2- 172.21.15.38:0/2559049460 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555cdea8d140 0x555cdea90ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:51:43.000 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.987+0000 7f4c659c2700 1 -- 172.21.15.38:0/2559049460 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555cdea8c830 msgr2=0x555cdea903a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:51:43.000 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.987+0000 7f4c659c2700 1 --2- 172.21.15.38:0/2559049460 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555cdea8c830 0x555cdea903a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:51:43.000 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.987+0000 7f4c659c2700 1 -- 172.21.15.38:0/2559049460 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c5c011500 con 0x555cdea86fe0 2023-03-09T14:51:43.000 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.987+0000 7f4c651c1700 1 --2- 172.21.15.38:0/2559049460 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555cdea8c830 0x555cdea903a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:51:43.000 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.988+0000 7f4c659c2700 1 --2- 172.21.15.38:0/2559049460 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555cdea86fe0 0x555cdea93120 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c005f50 tx=0x7f4c5c011300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e16aed93607e9ed9 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:51:43.000 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.988+0000 7f4c497fa700 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2023-03-09T14:51:43.000 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.988+0000 7f4c497fa700 1 -- 172.21.15.38:0/2559049460 <== mon.2 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c5c004220 con 0x555cdea86fe0 2023-03-09T14:51:43.001 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.988+0000 7f4c497fa700 1 -- 172.21.15.38:0/2559049460 <== mon.2 v2:172.21.15.122:3300/0 2 ==== osd_map(139..139 src has 1..139) v4 ==== 9526+0+0 (secure 0 0 0) 0x7f4c5c004380 con 0x555cdea86fe0 2023-03-09T14:51:43.001 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.988+0000 7f4c73dafb80 1 -- 172.21.15.38:0/2559049460 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555cde9a6190 con 0x555cdea86fe0 2023-03-09T14:51:43.001 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.988+0000 7f4c497fa700 1 -- 172.21.15.38:0/2559049460 <== mon.2 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4c5c004540 con 0x555cdea86fe0 2023-03-09T14:51:43.001 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.988+0000 7f4c73dafb80 10 client.25706 fetch_fsmap learned FSMap version 31 2023-03-09T14:51:43.001 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.988+0000 7f4c73dafb80 1 -- 172.21.15.38:0/2559049460 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x555cde7efc60 con 0x555cdea86fe0 2023-03-09T14:51:43.001 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.988+0000 7f4c497fa700 1 -- 172.21.15.38:0/2559049460 <== mon.2 v2:172.21.15.122:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x555cde9a6190 con 0x555cdea86fe0 2023-03-09T14:51:43.002 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.989+0000 7f4c497fa700 1 -- 172.21.15.38:0/2559049460 <== mon.2 v2:172.21.15.122:3300/0 5 ==== fsmap(e 31) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f4c5c021040 con 0x555cdea86fe0 2023-03-09T14:51:43.002 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.989+0000 7f4c73dafb80 10 client.25706 fetch_fsmap finished waiting for FSMap version 31 2023-03-09T14:51:43.002 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.989+0000 7f4c73dafb80 10 client.25706 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T14:51:43.002 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.989+0000 7f4c73dafb80 20 client.25706 populate_metadata read hostname 'smithi038' 2023-03-09T14:51:43.002 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.989+0000 7f4c73dafb80 1 --2- 172.21.15.38:0/2559049460 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x555cdeaa50a0 0x555cdeaa7480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:51:43.002 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.989+0000 7f4c73dafb80 4 client.25706 mds_command: new command op to 24530 tid=0[{"prefix": "get_command_descriptions"}] 2023-03-09T14:51:43.003 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.989+0000 7f4c73dafb80 1 -- 172.21.15.38:0/2559049460 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555cde85cf60 con 0x555cdeaa50a0 2023-03-09T14:51:43.003 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.989+0000 7f4c651c1700 1 --2- 172.21.15.38:0/2559049460 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x555cdeaa50a0 0x555cdeaa7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:51:43.003 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.989+0000 7f4c497fa700 10 client.25706 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:51:43.003 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.989+0000 7f4c651c1700 1 --2- 172.21.15.38:0/2559049460 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x555cdeaa50a0 0x555cdeaa7480 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4609166de51bd296 server_cookie=97f0db49abd8a8ca in_seq=0 out_seq=0 2023-03-09T14:51:43.003 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.990+0000 7f4c497fa700 1 -- 172.21.15.38:0/2559049460 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18278 (crc 0 0 0) 0x555cde85cf60 con 0x555cdeaa50a0 2023-03-09T14:51:43.003 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.990+0000 7f4c497fa700 10 client.25706 handle_command_reply: tid=0 2023-03-09T14:51:43.004 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.990+0000 7f4c73dafb80 1 client.25706 shutdown 2023-03-09T14:51:43.004 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.991+0000 7f4c73dafb80 1 -- 172.21.15.38:0/2559049460 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555cdea86fe0 msgr2=0x555cdea93120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:51:43.004 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.991+0000 7f4c73dafb80 1 --2- 172.21.15.38:0/2559049460 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555cdea86fe0 0x555cdea93120 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c005f50 tx=0x7f4c5c011300 comp rx=0 tx=0).stop 2023-03-09T14:51:43.004 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.991+0000 7f4c73dafb80 1 -- 172.21.15.38:0/2559049460 shutdown_connections 2023-03-09T14:51:43.004 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.991+0000 7f4c73dafb80 1 --2- 172.21.15.38:0/2559049460 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x555cdeaa50a0 0x555cdeaa7480 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:51:43.004 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.991+0000 7f4c73dafb80 1 --2- 172.21.15.38:0/2559049460 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555cdea8d140 0x555cdea90ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:51:43.004 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.991+0000 7f4c497fa700 0 client.25706 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:51:43.005 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.991+0000 7f4c73dafb80 1 --2- 172.21.15.38:0/2559049460 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555cdea86fe0 0x555cdea93120 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:51:43.005 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.991+0000 7f4c73dafb80 1 --2- 172.21.15.38:0/2559049460 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555cdea8c830 0x555cdea903a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:51:43.005 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.991+0000 7f4c73dafb80 1 -- 172.21.15.38:0/2559049460 >> 172.21.15.38:0/2559049460 conn(0x555cdea81a20 msgr2=0x555cdea80c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:51:43.005 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.991+0000 7f4c73dafb80 1 -- 172.21.15.38:0/2559049460 shutdown_connections 2023-03-09T14:51:43.005 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.991+0000 7f4c73dafb80 1 -- 172.21.15.38:0/2559049460 wait complete. 2023-03-09T14:51:43.005 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:42.991+0000 7f4c73dafb80 20 client.25706 trim_cache size 0 max 16384 2023-03-09T14:51:43.015 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.011+0000 7f4c73dafb80 1 Processor -- start 2023-03-09T14:51:43.015 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.011+0000 7f4c73dafb80 1 -- start start 2023-03-09T14:51:43.015 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.011+0000 7f4c73dafb80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555cdea86fe0 0x555cdea92cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:51:43.016 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.011+0000 7f4c73dafb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555cdea8c830 0x555cdea80c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:51:43.016 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.011+0000 7f4c73dafb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555cdea8d140 0x555cdea8a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:51:43.016 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.011+0000 7f4c73dafb80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cdeaa4ed0 con 0x555cdea86fe0 2023-03-09T14:51:43.016 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.011+0000 7f4c73dafb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cdea7de00 con 0x555cdea8d140 2023-03-09T14:51:43.016 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.011+0000 7f4c73dafb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cdea87cc0 con 0x555cdea8c830 2023-03-09T14:51:43.016 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.012+0000 7f4c659c2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555cdea86fe0 0x555cdea92cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:51:43.016 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.012+0000 7f4c661c3700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555cdea8d140 0x555cdea8a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:51:43.017 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.012+0000 7f4c659c2700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555cdea86fe0 0x555cdea92cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:40736/0 (socket says 172.21.15.38:40736) 2023-03-09T14:51:43.017 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.012+0000 7f4c659c2700 1 -- 172.21.15.38:0/3750713601 learned_addr learned my addr 172.21.15.38:0/3750713601 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:51:43.017 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.012+0000 7f4c661c3700 1 --2- 172.21.15.38:0/3750713601 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555cdea8d140 0x555cdea8a4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.38:49928/0 (socket says 172.21.15.38:49928) 2023-03-09T14:51:43.017 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.012+0000 7f4c651c1700 1 --2- 172.21.15.38:0/3750713601 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555cdea8c830 0x555cdea80c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:51:43.017 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.012+0000 7f4c659c2700 1 -- 172.21.15.38:0/3750713601 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555cdea8c830 msgr2=0x555cdea80c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:51:43.017 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.012+0000 7f4c659c2700 1 --2- 172.21.15.38:0/3750713601 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555cdea8c830 0x555cdea80c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:51:43.018 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.012+0000 7f4c659c2700 1 -- 172.21.15.38:0/3750713601 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555cdea8d140 msgr2=0x555cdea8a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:51:43.018 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.012+0000 7f4c659c2700 1 --2- 172.21.15.38:0/3750713601 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555cdea8d140 0x555cdea8a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:51:43.018 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.012+0000 7f4c659c2700 1 -- 172.21.15.38:0/3750713601 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555cde806de0 con 0x555cdea86fe0 2023-03-09T14:51:43.018 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.012+0000 7f4c659c2700 1 --2- 172.21.15.38:0/3750713601 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555cdea86fe0 0x555cdea92cc0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c009af0 tx=0x7f4c5c011be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9acbf2661c00ef75 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:51:43.021 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.012+0000 7f4c54daf700 1 -- 172.21.15.38:0/3750713601 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c5c019cb0 con 0x555cdea86fe0 2023-03-09T14:51:43.021 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.012+0000 7f4c54daf700 1 -- 172.21.15.38:0/3750713601 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f4c5c0198f0 con 0x555cdea86fe0 2023-03-09T14:51:43.022 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.012+0000 7f4c54daf700 1 -- 172.21.15.38:0/3750713601 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c5c00fd80 con 0x555cdea86fe0 2023-03-09T14:51:43.022 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.013+0000 7f4c73dafb80 1 -- 172.21.15.38:0/3750713601 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555cdea86fe0 msgr2=0x555cdea92cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:51:43.022 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.013+0000 7f4c73dafb80 1 --2- 172.21.15.38:0/3750713601 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555cdea86fe0 0x555cdea92cc0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c009af0 tx=0x7f4c5c011be0 comp rx=0 tx=0).stop 2023-03-09T14:51:43.022 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.013+0000 7f4c73dafb80 1 -- 172.21.15.38:0/3750713601 shutdown_connections 2023-03-09T14:51:43.022 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.013+0000 7f4c73dafb80 1 --2- 172.21.15.38:0/3750713601 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555cdea86fe0 0x555cdea92cc0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:51:43.022 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.013+0000 7f4c73dafb80 1 --2- 172.21.15.38:0/3750713601 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555cdea8d140 0x555cdea8a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:51:43.023 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.013+0000 7f4c73dafb80 1 --2- 172.21.15.38:0/3750713601 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555cdea8c830 0x555cdea80c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:51:43.023 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.013+0000 7f4c73dafb80 1 -- 172.21.15.38:0/3750713601 >> 172.21.15.38:0/3750713601 conn(0x555cdea81a20 msgr2=0x555cdeaa1490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:51:43.023 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.013+0000 7f4c73dafb80 1 -- 172.21.15.38:0/3750713601 shutdown_connections 2023-03-09T14:51:43.023 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.013+0000 7f4c73dafb80 1 -- 172.21.15.38:0/3750713601 wait complete. 2023-03-09T14:51:43.023 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.013+0000 7f4c73dafb80 1 -- start start 2023-03-09T14:51:43.023 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.014+0000 7f4c73dafb80 1 Processor -- start 2023-03-09T14:51:43.024 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.014+0000 7f4c73dafb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555cdea86fe0 0x555cdea894d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:51:43.024 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.014+0000 7f4c73dafb80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555cdea8c830 0x555cdea89a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:51:43.024 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.014+0000 7f4c73dafb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555cdea8d140 0x555cdea88290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:51:43.024 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.014+0000 7f4c73dafb80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cdea9c540 con 0x555cdea8c830 2023-03-09T14:51:43.024 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.014+0000 7f4c73dafb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cdea887d0 con 0x555cdea86fe0 2023-03-09T14:51:43.024 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.014+0000 7f4c73dafb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cdea88990 con 0x555cdea8d140 2023-03-09T14:51:43.025 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.014+0000 7f4c73dafb80 10 client.0 _ll_register_callbacks cb 0x555cde97e520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:51:43.025 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.014+0000 7f4c651c1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555cdea8c830 0x555cdea89a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:51:43.025 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.014+0000 7f4c659c2700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555cdea86fe0 0x555cdea894d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:51:43.025 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.014+0000 7f4c651c1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555cdea8c830 0x555cdea89a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:40742/0 (socket says 172.21.15.38:40742) 2023-03-09T14:51:43.025 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.014+0000 7f4c651c1700 1 -- 172.21.15.38:0/1311722676 learned_addr learned my addr 172.21.15.38:0/1311722676 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:51:43.025 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.015+0000 7f4c661c3700 1 --2- 172.21.15.38:0/1311722676 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555cdea8d140 0x555cdea88290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:51:43.025 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.015+0000 7f4c651c1700 1 -- 172.21.15.38:0/1311722676 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555cdea8d140 msgr2=0x555cdea88290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:51:43.025 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.015+0000 7f4c651c1700 1 --2- 172.21.15.38:0/1311722676 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555cdea8d140 0x555cdea88290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:51:43.026 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.015+0000 7f4c651c1700 1 -- 172.21.15.38:0/1311722676 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555cdea86fe0 msgr2=0x555cdea894d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:51:43.026 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.015+0000 7f4c651c1700 1 --2- 172.21.15.38:0/1311722676 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555cdea86fe0 0x555cdea894d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:51:43.026 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.015+0000 7f4c651c1700 1 -- 172.21.15.38:0/1311722676 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555cde7efc60 con 0x555cdea8c830 2023-03-09T14:51:43.026 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.015+0000 7f4c661c3700 1 --2- 172.21.15.38:0/1311722676 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555cdea8d140 0x555cdea88290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:51:43.026 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.015+0000 7f4c659c2700 1 --2- 172.21.15.38:0/1311722676 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555cdea86fe0 0x555cdea894d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:51:43.026 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.015+0000 7f4c651c1700 1 --2- 172.21.15.38:0/1311722676 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555cdea8c830 0x555cdea89a10 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f4c5000bd50 tx=0x7f4c50009c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fe901dd154f5553 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:51:43.026 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.015+0000 7f4c497fa700 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2023-03-09T14:51:43.027 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.015+0000 7f4c497fa700 1 -- 172.21.15.38:0/1311722676 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c500a10a0 con 0x555cdea8c830 2023-03-09T14:51:43.027 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.015+0000 7f4c497fa700 1 -- 172.21.15.38:0/1311722676 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(139..139 src has 1..139) v4 ==== 9526+0+0 (secure 0 0 0) 0x7f4c50005140 con 0x555cdea8c830 2023-03-09T14:51:43.027 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.015+0000 7f4c73dafb80 1 -- 172.21.15.38:0/1311722676 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555cde9961b0 con 0x555cdea8c830 2023-03-09T14:51:43.027 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.015+0000 7f4c497fa700 1 -- 172.21.15.38:0/1311722676 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4c50005300 con 0x555cdea8c830 2023-03-09T14:51:43.027 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.015+0000 7f4c73dafb80 10 client.16065 fetch_fsmap learned FSMap version 31 2023-03-09T14:51:43.027 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.015+0000 7f4c73dafb80 1 -- 172.21.15.38:0/1311722676 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x555cde8052b0 con 0x555cdea8c830 2023-03-09T14:51:43.027 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.015+0000 7f4c497fa700 1 -- 172.21.15.38:0/1311722676 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x555cde9961b0 con 0x555cdea8c830 2023-03-09T14:51:43.027 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.016+0000 7f4c497fa700 1 -- 172.21.15.38:0/1311722676 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 31) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f4c50098430 con 0x555cdea8c830 2023-03-09T14:51:43.028 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.016+0000 7f4c73dafb80 10 client.16065 fetch_fsmap finished waiting for FSMap version 31 2023-03-09T14:51:43.028 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.016+0000 7f4c73dafb80 10 client.16065 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T14:51:43.028 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.016+0000 7f4c73dafb80 20 client.16065 populate_metadata read hostname 'smithi038' 2023-03-09T14:51:43.028 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.016+0000 7f4c73dafb80 1 --2- 172.21.15.38:0/1311722676 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x555cdeaa50a0 0x555cdeafb990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:51:43.028 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.016+0000 7f4c73dafb80 4 client.16065 mds_command: new command op to 24530 tid=0[{"prefix": "scrub status"}] 2023-03-09T14:51:43.028 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.016+0000 7f4c73dafb80 1 -- 172.21.15.38:0/1311722676 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x555cde9801c0 con 0x555cdeaa50a0 2023-03-09T14:51:43.028 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.016+0000 7f4c659c2700 1 --2- 172.21.15.38:0/1311722676 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x555cdeaa50a0 0x555cdeafb990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:51:43.029 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.016+0000 7f4c497fa700 10 client.16065 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:51:43.029 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.016+0000 7f4c659c2700 1 --2- 172.21.15.38:0/1311722676 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x555cdeaa50a0 0x555cdeafb990 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4248dfbe7390319 server_cookie=a5b722ae1db1c778 in_seq=0 out_seq=0 2023-03-09T14:51:43.029 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.017+0000 7f4c497fa700 1 -- 172.21.15.38:0/1311722676 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+265 (crc 0 0 0) 0x555cde9801c0 con 0x555cdeaa50a0 2023-03-09T14:51:43.029 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.017+0000 7f4c497fa700 10 client.16065 handle_command_reply: tid=0 2023-03-09T14:51:43.029 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.017+0000 7f4c73dafb80 1 client.16065 shutdown 2023-03-09T14:51:43.029 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.017+0000 7f4c73dafb80 1 -- 172.21.15.38:0/1311722676 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555cdea8c830 msgr2=0x555cdea89a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:51:43.029 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.017+0000 7f4c73dafb80 1 --2- 172.21.15.38:0/1311722676 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555cdea8c830 0x555cdea89a10 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f4c5000bd50 tx=0x7f4c50009c30 comp rx=0 tx=0).stop 2023-03-09T14:51:43.029 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.017+0000 7f4c73dafb80 1 -- 172.21.15.38:0/1311722676 shutdown_connections 2023-03-09T14:51:43.030 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.017+0000 7f4c73dafb80 1 --2- 172.21.15.38:0/1311722676 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x555cdeaa50a0 0x555cdeafb990 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:51:43.030 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.017+0000 7f4c73dafb80 1 --2- 172.21.15.38:0/1311722676 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555cdea8d140 0x555cdea88290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:51:43.030 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.017+0000 7f4c73dafb80 1 --2- 172.21.15.38:0/1311722676 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555cdea86fe0 0x555cdea894d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:51:43.030 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.017+0000 7f4c497fa700 0 client.16065 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:51:43.030 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.017+0000 7f4c73dafb80 1 --2- 172.21.15.38:0/1311722676 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555cdea8c830 0x555cdea89a10 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:51:43.030 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.017+0000 7f4c73dafb80 1 -- 172.21.15.38:0/1311722676 >> 172.21.15.38:0/1311722676 conn(0x555cdea81a20 msgr2=0x555cdea8a4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:51:43.030 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.017+0000 7f4c73dafb80 1 -- 172.21.15.38:0/1311722676 shutdown_connections 2023-03-09T14:51:43.030 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.017+0000 7f4c73dafb80 1 -- 172.21.15.38:0/1311722676 wait complete. 2023-03-09T14:51:43.031 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.017+0000 7f4c73dafb80 20 client.16065 trim_cache size 0 max 16384 2023-03-09T14:51:43.031 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2023-03-09T14:51:43.031 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "scrub active (3 inodes in the stack)", 2023-03-09T14:51:43.031 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": { 2023-03-09T14:51:43.031 INFO:teuthology.orchestra.run.smithi038.stdout: "a14659f4-d7ea-4c5a-b15b-6138af3f909f": { 2023-03-09T14:51:43.032 INFO:teuthology.orchestra.run.smithi038.stdout: "path": "/", 2023-03-09T14:51:43.032 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "a14659f4-d7ea-4c5a-b15b-6138af3f909f", 2023-03-09T14:51:43.032 INFO:teuthology.orchestra.run.smithi038.stdout: "options": "recursive,force" 2023-03-09T14:51:43.032 INFO:teuthology.orchestra.run.smithi038.stdout: } 2023-03-09T14:51:43.032 INFO:teuthology.orchestra.run.smithi038.stdout: } 2023-03-09T14:51:43.032 INFO:teuthology.orchestra.run.smithi038.stdout:} 2023-03-09T14:51:43.033 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.018+0000 7f4c67c26700 1 -- 172.21.15.38:0/554054776 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f4c4c09f4a0 msgr2=0x7f4c4c0a1950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:51:43.033 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.018+0000 7f4c67c26700 1 --2- 172.21.15.38:0/554054776 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f4c4c09f4a0 0x7f4c4c0a1950 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f4c58009fd0 tx=0x7f4c58004ad0 comp rx=0 tx=0).stop 2023-03-09T14:51:43.034 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.018+0000 7f4c67c26700 1 -- 172.21.15.38:0/554054776 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4c60100b20 msgr2=0x7f4c60196650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:51:43.034 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.018+0000 7f4c67c26700 1 --2- 172.21.15.38:0/554054776 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4c60100b20 0x7f4c60196650 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4c50000c00 tx=0x7f4c5000fa00 comp rx=0 tx=0).stop 2023-03-09T14:51:43.034 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.018+0000 7f4c67c26700 1 -- 172.21.15.38:0/554054776 shutdown_connections 2023-03-09T14:51:43.034 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.018+0000 7f4c67c26700 1 --2- 172.21.15.38:0/554054776 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f4c4c09f4a0 0x7f4c4c0a1950 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:51:43.035 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.018+0000 7f4c67c26700 1 --2- 172.21.15.38:0/554054776 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f4c60100180 0x7f4c60196110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:51:43.035 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.018+0000 7f4c67c26700 1 --2- 172.21.15.38:0/554054776 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f4c601051c0 0x7f4c6019aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:51:43.035 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.018+0000 7f4c67c26700 1 --2- 172.21.15.38:0/554054776 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f4c60100b20 0x7f4c60196650 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:51:43.035 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.018+0000 7f4c67c26700 1 -- 172.21.15.38:0/554054776 >> 172.21.15.38:0/554054776 conn(0x7f4c600fb9d0 msgr2=0x7f4c600fc680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:51:43.035 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.018+0000 7f4c67c26700 1 -- 172.21.15.38:0/554054776 shutdown_connections 2023-03-09T14:51:43.035 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:51:43.018+0000 7f4c67c26700 1 -- 172.21.15.38:0/554054776 wait complete. 2023-03-09T14:51:43.036 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:42 smithi038 ceph-mon[92422]: Health check cleared: OSD_DOWN (was: 2 osds down) 2023-03-09T14:51:43.036 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:42 smithi038 ceph-mon[92422]: osd.1 [v2:172.21.15.122:6808/216712231,v1:172.21.15.122:6809/216712231] boot 2023-03-09T14:51:43.036 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:42 smithi038 ceph-mon[92422]: osd.3 [v2:172.21.15.38:6810/911235164,v1:172.21.15.38:6811/911235164] boot 2023-03-09T14:51:43.036 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:42 smithi038 ceph-mon[92422]: osdmap e138: 12 total, 12 up, 12 in 2023-03-09T14:51:43.036 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-03-09T14:51:43.036 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-03-09T14:51:43.036 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:42 smithi038 ceph-mon[92422]: Health check update: Degraded data redundancy: 6171/33595 objects degraded (18.369%), 40 pgs degraded (PG_DEGRADED) 2023-03-09T14:51:43.037 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:42 smithi038 ceph-mon[92422]: osdmap e139: 12 total, 12 up, 12 in 2023-03-09T14:51:43.037 INFO:tasks.cephfs.filesystem:scrub status for tag:a14659f4-d7ea-4c5a-b15b-6138af3f909f - {'path': '/', 'tag': 'a14659f4-d7ea-4c5a-b15b-6138af3f909f', 'options': 'recursive,force'} 2023-03-09T14:51:43.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:42 smithi122 ceph-mon[93869]: Health check cleared: OSD_DOWN (was: 2 osds down) 2023-03-09T14:51:43.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:42 smithi122 ceph-mon[93869]: osd.1 [v2:172.21.15.122:6808/216712231,v1:172.21.15.122:6809/216712231] boot 2023-03-09T14:51:43.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:42 smithi122 ceph-mon[93869]: osd.3 [v2:172.21.15.38:6810/911235164,v1:172.21.15.38:6811/911235164] boot 2023-03-09T14:51:43.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:42 smithi122 ceph-mon[93869]: osdmap e138: 12 total, 12 up, 12 in 2023-03-09T14:51:43.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-03-09T14:51:43.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-03-09T14:51:43.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:42 smithi122 ceph-mon[93869]: Health check update: Degraded data redundancy: 6171/33595 objects degraded (18.369%), 40 pgs degraded (PG_DEGRADED) 2023-03-09T14:51:43.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:42 smithi122 ceph-mon[93869]: osdmap e139: 12 total, 12 up, 12 in 2023-03-09T14:51:43.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:42 smithi181 ceph-mon[94189]: Health check cleared: OSD_DOWN (was: 2 osds down) 2023-03-09T14:51:43.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:42 smithi181 ceph-mon[94189]: osd.1 [v2:172.21.15.122:6808/216712231,v1:172.21.15.122:6809/216712231] boot 2023-03-09T14:51:43.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:42 smithi181 ceph-mon[94189]: osd.3 [v2:172.21.15.38:6810/911235164,v1:172.21.15.38:6811/911235164] boot 2023-03-09T14:51:43.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:42 smithi181 ceph-mon[94189]: osdmap e138: 12 total, 12 up, 12 in 2023-03-09T14:51:43.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-03-09T14:51:43.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-03-09T14:51:43.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:42 smithi181 ceph-mon[94189]: Health check update: Degraded data redundancy: 6171/33595 objects degraded (18.369%), 40 pgs degraded (PG_DEGRADED) 2023-03-09T14:51:43.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:42 smithi181 ceph-mon[94189]: osdmap e139: 12 total, 12 up, 12 in 2023-03-09T14:51:43.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 75.0 s, 9.4 tps, lat 25933.473 ms stddev 132.883 2023-03-09T14:51:44.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:43 smithi122 ceph-mon[93869]: pgmap v216: 129 pgs: 1 remapped+peering, 10 peering, 2 active+recovery_wait+degraded, 2 undersized+peered, 28 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 1 recovery_wait+undersized+degraded+remapped+peered, 2 active+recovery_wait, 2 undersized+degraded+peered, 27 active+undersized+degraded, 53 active+clean; 18 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 97 KiB/s wr, 16 op/s; 4754/33595 objects degraded (14.151%); 113 MiB/s, 28 objects/s recovering 2023-03-09T14:51:44.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:43 smithi122 ceph-mon[93869]: osdmap e140: 12 total, 12 up, 12 in 2023-03-09T14:51:44.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:43 smithi181 ceph-mon[94189]: pgmap v216: 129 pgs: 1 remapped+peering, 10 peering, 2 active+recovery_wait+degraded, 2 undersized+peered, 28 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 1 recovery_wait+undersized+degraded+remapped+peered, 2 active+recovery_wait, 2 undersized+degraded+peered, 27 active+undersized+degraded, 53 active+clean; 18 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 97 KiB/s wr, 16 op/s; 4754/33595 objects degraded (14.151%); 113 MiB/s, 28 objects/s recovering 2023-03-09T14:51:44.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:43 smithi181 ceph-mon[94189]: osdmap e140: 12 total, 12 up, 12 in 2023-03-09T14:51:44.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:43 smithi038 ceph-mon[92422]: pgmap v216: 129 pgs: 1 remapped+peering, 10 peering, 2 active+recovery_wait+degraded, 2 undersized+peered, 28 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 1 recovery_wait+undersized+degraded+remapped+peered, 2 active+recovery_wait, 2 undersized+degraded+peered, 27 active+undersized+degraded, 53 active+clean; 18 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 97 KiB/s wr, 16 op/s; 4754/33595 objects degraded (14.151%); 113 MiB/s, 28 objects/s recovering 2023-03-09T14:51:44.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:43 smithi038 ceph-mon[92422]: osdmap e140: 12 total, 12 up, 12 in 2023-03-09T14:51:46.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:45 smithi122 ceph-mon[93869]: pgmap v218: 129 pgs: 1 remapped+peering, 10 peering, 5 active+recovery_wait+degraded, 2 undersized+peered, 25 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 1 recovery_wait+undersized+degraded+remapped+peered, 2 active+recovery_wait, 2 undersized+degraded+peered, 23 active+undersized+degraded, 57 active+clean; 18 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 26 KiB/s wr, 3 op/s; 4294/33595 objects degraded (12.782%); 49 MiB/s, 12 keys/s, 12 objects/s recovering 2023-03-09T14:51:46.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:45 smithi181 ceph-mon[94189]: pgmap v218: 129 pgs: 1 remapped+peering, 10 peering, 5 active+recovery_wait+degraded, 2 undersized+peered, 25 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 1 recovery_wait+undersized+degraded+remapped+peered, 2 active+recovery_wait, 2 undersized+degraded+peered, 23 active+undersized+degraded, 57 active+clean; 18 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 26 KiB/s wr, 3 op/s; 4294/33595 objects degraded (12.782%); 49 MiB/s, 12 keys/s, 12 objects/s recovering 2023-03-09T14:51:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:45 smithi038 ceph-mon[92422]: pgmap v218: 129 pgs: 1 remapped+peering, 10 peering, 5 active+recovery_wait+degraded, 2 undersized+peered, 25 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 1 recovery_wait+undersized+degraded+remapped+peered, 2 active+recovery_wait, 2 undersized+degraded+peered, 23 active+undersized+degraded, 57 active+clean; 18 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 26 KiB/s wr, 3 op/s; 4294/33595 objects degraded (12.782%); 49 MiB/s, 12 keys/s, 12 objects/s recovering 2023-03-09T14:51:48.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:47 smithi122 ceph-mon[93869]: pgmap v219: 129 pgs: 10 peering, 10 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 108 active+clean; 18 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 383 op/s; 221/34238 objects degraded (0.645%); 139 MiB/s, 25 keys/s, 36 objects/s recovering 2023-03-09T14:51:48.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:47 smithi122 ceph-mon[93869]: osdmap e141: 12 total, 12 up, 12 in 2023-03-09T14:51:48.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:47 smithi122 ceph-mon[93869]: Health check update: Degraded data redundancy: 221/34238 objects degraded (0.645%), 11 pgs degraded (PG_DEGRADED) 2023-03-09T14:51:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:47 smithi181 ceph-mon[94189]: pgmap v219: 129 pgs: 10 peering, 10 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 108 active+clean; 18 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 383 op/s; 221/34238 objects degraded (0.645%); 139 MiB/s, 25 keys/s, 36 objects/s recovering 2023-03-09T14:51:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:47 smithi181 ceph-mon[94189]: osdmap e141: 12 total, 12 up, 12 in 2023-03-09T14:51:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:47 smithi181 ceph-mon[94189]: Health check update: Degraded data redundancy: 221/34238 objects degraded (0.645%), 11 pgs degraded (PG_DEGRADED) 2023-03-09T14:51:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:47 smithi038 ceph-mon[92422]: pgmap v219: 129 pgs: 10 peering, 10 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 108 active+clean; 18 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.3 MiB/s wr, 383 op/s; 221/34238 objects degraded (0.645%); 139 MiB/s, 25 keys/s, 36 objects/s recovering 2023-03-09T14:51:48.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:47 smithi038 ceph-mon[92422]: osdmap e141: 12 total, 12 up, 12 in 2023-03-09T14:51:48.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:47 smithi038 ceph-mon[92422]: Health check update: Degraded data redundancy: 221/34238 objects degraded (0.645%), 11 pgs degraded (PG_DEGRADED) 2023-03-09T14:51:48.557 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7f8c712b7700 time 2023-03-09T14:51:48.287070+0000 2023-03-09T14:51:48.558 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op") 2023-03-09T14:51:48.558 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:51:48.558 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x55b588f7dc85] 2023-03-09T14:51:48.558 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 2: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55b5892366e4] 2023-03-09T14:51:48.559 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55b58923cd5e] 2023-03-09T14:51:48.559 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 4: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55b58908c986] 2023-03-09T14:51:48.559 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 5: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55b5893a2729] 2023-03-09T14:51:48.559 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55b5890a1ebf] 2023-03-09T14:51:48.559 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55b589753d85] 2023-03-09T14:51:48.559 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55b589756f84] 2023-03-09T14:51:48.559 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f8c948181ca] 2023-03-09T14:51:48.560 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 10: clone() 2023-03-09T14:51:48.560 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: *** Caught signal (Aborted) ** 2023-03-09T14:51:48.560 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: in thread 7f8c712b7700 thread_name:tp_osd_tp 2023-03-09T14:51:48.560 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 2023-03-09T14:51:48.286+0000 7f8c712b7700 -1 osd.3 pg_epoch: 142 pg[4.0s0( v 142'2572 (0'0,142'2572] local-lis/les=138/139 n=299 ec=85/85 lis/c=138/138 les/c/f=139/140/0 sis=138) [3,10,0,7]p3(0) r=0 lpr=138 luod=140'2570 crt=140'2570 lcod 140'2569 mlcod 140'2569 active+clean] bad op order, already applied 85983 > this 85982 2023-03-09T14:51:48.560 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 2023-03-09T14:51:48.293+0000 7f8c712b7700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7f8c712b7700 time 2023-03-09T14:51:48.287070+0000 2023-03-09T14:51:48.560 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op") 2023-03-09T14:51:48.561 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 2023-03-09T14:51:48.561 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:51:48.561 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x55b588f7dc85] 2023-03-09T14:51:48.561 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 2: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55b5892366e4] 2023-03-09T14:51:48.561 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55b58923cd5e] 2023-03-09T14:51:48.561 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 4: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55b58908c986] 2023-03-09T14:51:48.561 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 5: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55b5893a2729] 2023-03-09T14:51:48.562 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55b5890a1ebf] 2023-03-09T14:51:48.562 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55b589753d85] 2023-03-09T14:51:48.562 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55b589756f84] 2023-03-09T14:51:48.562 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f8c948181ca] 2023-03-09T14:51:48.562 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 10: clone() 2023-03-09T14:51:48.563 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 2023-03-09T14:51:48.563 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:51:48.563 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f8c94822cf0] 2023-03-09T14:51:48.563 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 2: gsignal() 2023-03-09T14:51:48.563 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 3: abort() 2023-03-09T14:51:48.563 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x55b588f7dd45] 2023-03-09T14:51:48.563 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 5: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55b5892366e4] 2023-03-09T14:51:48.563 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 6: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55b58923cd5e] 2023-03-09T14:51:48.564 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55b58908c986] 2023-03-09T14:51:48.564 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 8: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55b5893a2729] 2023-03-09T14:51:48.564 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55b5890a1ebf] 2023-03-09T14:51:48.564 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55b589753d85] 2023-03-09T14:51:48.564 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55b589756f84] 2023-03-09T14:51:48.564 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f8c948181ca] 2023-03-09T14:51:48.564 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 13: clone() 2023-03-09T14:51:48.564 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 2023-03-09T14:51:48.299+0000 7f8c712b7700 -1 *** Caught signal (Aborted) ** 2023-03-09T14:51:48.565 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: in thread 7f8c712b7700 thread_name:tp_osd_tp 2023-03-09T14:51:48.565 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 2023-03-09T14:51:48.565 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:51:48.565 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f8c94822cf0] 2023-03-09T14:51:48.565 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 2: gsignal() 2023-03-09T14:51:48.565 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 3: abort() 2023-03-09T14:51:48.565 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x55b588f7dd45] 2023-03-09T14:51:48.566 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 5: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55b5892366e4] 2023-03-09T14:51:48.566 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 6: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55b58923cd5e] 2023-03-09T14:51:48.566 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55b58908c986] 2023-03-09T14:51:48.566 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 8: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55b5893a2729] 2023-03-09T14:51:48.566 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55b5890a1ebf] 2023-03-09T14:51:48.566 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55b589753d85] 2023-03-09T14:51:48.566 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55b589756f84] 2023-03-09T14:51:48.567 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f8c948181ca] 2023-03-09T14:51:48.567 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 13: clone() 2023-03-09T14:51:48.567 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-03-09T14:51:48.568 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 2023-03-09T14:51:48.568 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: -1365> 2023-03-09T14:51:48.286+0000 7f8c712b7700 -1 osd.3 pg_epoch: 142 pg[4.0s0( v 142'2572 (0'0,142'2572] local-lis/les=138/139 n=299 ec=85/85 lis/c=138/138 les/c/f=139/140/0 sis=138) [3,10,0,7]p3(0) r=0 lpr=138 luod=140'2570 crt=140'2570 lcod 140'2569 mlcod 140'2569 active+clean] bad op order, already applied 85983 > this 85982 2023-03-09T14:51:48.568 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: -659> 2023-03-09T14:51:48.293+0000 7f8c712b7700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7f8c712b7700 time 2023-03-09T14:51:48.287070+0000 2023-03-09T14:51:48.568 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op") 2023-03-09T14:51:48.568 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 2023-03-09T14:51:48.568 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:51:48.568 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x55b588f7dc85] 2023-03-09T14:51:48.569 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 2: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55b5892366e4] 2023-03-09T14:51:48.569 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55b58923cd5e] 2023-03-09T14:51:48.569 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 4: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55b58908c986] 2023-03-09T14:51:48.569 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 5: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55b5893a2729] 2023-03-09T14:51:48.569 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55b5890a1ebf] 2023-03-09T14:51:48.569 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55b589753d85] 2023-03-09T14:51:48.569 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55b589756f84] 2023-03-09T14:51:48.570 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f8c948181ca] 2023-03-09T14:51:48.570 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 10: clone() 2023-03-09T14:51:48.570 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 2023-03-09T14:51:48.570 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: -230> 2023-03-09T14:51:48.299+0000 7f8c712b7700 -1 *** Caught signal (Aborted) ** 2023-03-09T14:51:48.570 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: in thread 7f8c712b7700 thread_name:tp_osd_tp 2023-03-09T14:51:48.570 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 2023-03-09T14:51:48.570 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:51:48.571 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f8c94822cf0] 2023-03-09T14:51:48.571 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 2: gsignal() 2023-03-09T14:51:48.571 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 3: abort() 2023-03-09T14:51:48.571 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x55b588f7dd45] 2023-03-09T14:51:48.571 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 5: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55b5892366e4] 2023-03-09T14:51:48.571 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 6: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55b58923cd5e] 2023-03-09T14:51:48.571 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55b58908c986] 2023-03-09T14:51:48.571 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 8: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55b5893a2729] 2023-03-09T14:51:48.572 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55b5890a1ebf] 2023-03-09T14:51:48.572 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55b589753d85] 2023-03-09T14:51:48.572 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55b589756f84] 2023-03-09T14:51:48.572 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f8c948181ca] 2023-03-09T14:51:48.572 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 13: clone() 2023-03-09T14:51:48.573 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-03-09T14:51:48.573 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 2023-03-09T14:51:48.573 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: -2366> 2023-03-09T14:51:48.286+0000 7f8c712b7700 -1 osd.3 pg_epoch: 142 pg[4.0s0( v 142'2572 (0'0,142'2572] local-lis/les=138/139 n=299 ec=85/85 lis/c=138/138 les/c/f=139/140/0 sis=138) [3,10,0,7]p3(0) r=0 lpr=138 luod=140'2570 crt=140'2570 lcod 140'2569 mlcod 140'2569 active+clean] bad op order, already applied 85983 > this 85982 2023-03-09T14:51:48.573 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: -1660> 2023-03-09T14:51:48.293+0000 7f8c712b7700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7f8c712b7700 time 2023-03-09T14:51:48.287070+0000 2023-03-09T14:51:48.573 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op") 2023-03-09T14:51:48.573 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 2023-03-09T14:51:48.573 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:51:48.574 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x55b588f7dc85] 2023-03-09T14:51:48.574 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 2: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55b5892366e4] 2023-03-09T14:51:48.574 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55b58923cd5e] 2023-03-09T14:51:48.574 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 4: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55b58908c986] 2023-03-09T14:51:48.574 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 5: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55b5893a2729] 2023-03-09T14:51:48.574 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55b5890a1ebf] 2023-03-09T14:51:48.574 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55b589753d85] 2023-03-09T14:51:48.575 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55b589756f84] 2023-03-09T14:51:48.575 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f8c948181ca] 2023-03-09T14:51:48.575 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 10: clone() 2023-03-09T14:51:48.575 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 2023-03-09T14:51:48.575 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: -1231> 2023-03-09T14:51:48.299+0000 7f8c712b7700 -1 *** Caught signal (Aborted) ** 2023-03-09T14:51:48.575 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: in thread 7f8c712b7700 thread_name:tp_osd_tp 2023-03-09T14:51:48.575 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 2023-03-09T14:51:48.575 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:51:48.576 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f8c94822cf0] 2023-03-09T14:51:48.576 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 2: gsignal() 2023-03-09T14:51:48.576 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 3: abort() 2023-03-09T14:51:48.576 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x55b588f7dd45] 2023-03-09T14:51:48.576 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 5: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55b5892366e4] 2023-03-09T14:51:48.576 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 6: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55b58923cd5e] 2023-03-09T14:51:48.577 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55b58908c986] 2023-03-09T14:51:48.577 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 8: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55b5893a2729] 2023-03-09T14:51:48.577 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55b5890a1ebf] 2023-03-09T14:51:48.578 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55b589753d85] 2023-03-09T14:51:48.578 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55b589756f84] 2023-03-09T14:51:48.578 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f8c948181ca] 2023-03-09T14:51:48.578 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 13: clone() 2023-03-09T14:51:48.578 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-03-09T14:51:48.578 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[149561]: 2023-03-09T14:51:48.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 80.0 s, 537.4 tps, lat 700.466 ms stddev 3752.158 2023-03-09T14:51:48.817 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:48 smithi038 systemd[1]: ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.3.service: Main process exited, code=exited, status=134/n/a 2023-03-09T14:51:49.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:48 smithi122 ceph-mon[93869]: osdmap e142: 12 total, 12 up, 12 in 2023-03-09T14:51:49.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:48 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.9 2023-03-09T14:51:49.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:48 smithi122 ceph-mon[93869]: osd.3 failed (root=default,host=smithi038) (connection refused reported by osd.9) 2023-03-09T14:51:49.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:48 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.9 2023-03-09T14:51:49.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:48 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.0 2023-03-09T14:51:49.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:48 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.0 2023-03-09T14:51:49.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:48 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.6 2023-03-09T14:51:49.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:48 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.6 2023-03-09T14:51:49.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:48 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.6 2023-03-09T14:51:49.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:48 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.7 2023-03-09T14:51:49.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:48 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.7 2023-03-09T14:51:49.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:48 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.1 2023-03-09T14:51:49.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:48 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.11 2023-03-09T14:51:49.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:48 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.4 2023-03-09T14:51:49.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:48 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.7 2023-03-09T14:51:49.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:48 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.11 2023-03-09T14:51:49.231 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:48 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.10 2023-03-09T14:51:49.231 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:48 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.1 2023-03-09T14:51:49.231 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:48 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.4 2023-03-09T14:51:49.231 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:48 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.1 2023-03-09T14:51:49.231 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:48 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.10 2023-03-09T14:51:49.231 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:48 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.8 2023-03-09T14:51:49.232 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:48 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.4 2023-03-09T14:51:49.232 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:48 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.8 2023-03-09T14:51:49.232 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:48 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.5 2023-03-09T14:51:49.232 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:48 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.2 2023-03-09T14:51:49.232 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:48 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.5 2023-03-09T14:51:49.232 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:48 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.2 2023-03-09T14:51:49.233 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:48 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.5 2023-03-09T14:51:49.233 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:48 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.9 2023-03-09T14:51:49.233 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:48 smithi122 ceph-mon[93869]: osd.3 reported immediately failed by osd.9 2023-03-09T14:51:49.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:48 smithi181 ceph-mon[94189]: osdmap e142: 12 total, 12 up, 12 in 2023-03-09T14:51:49.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:48 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.9 2023-03-09T14:51:49.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:48 smithi181 ceph-mon[94189]: osd.3 failed (root=default,host=smithi038) (connection refused reported by osd.9) 2023-03-09T14:51:49.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:48 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.9 2023-03-09T14:51:49.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:48 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.0 2023-03-09T14:51:49.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:48 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.0 2023-03-09T14:51:49.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:48 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.6 2023-03-09T14:51:49.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:48 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.6 2023-03-09T14:51:49.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:48 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.6 2023-03-09T14:51:49.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:48 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.7 2023-03-09T14:51:49.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:48 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.7 2023-03-09T14:51:49.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:48 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.1 2023-03-09T14:51:49.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:48 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.11 2023-03-09T14:51:49.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:48 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.4 2023-03-09T14:51:49.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:48 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.7 2023-03-09T14:51:49.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:48 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.11 2023-03-09T14:51:49.236 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:48 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.10 2023-03-09T14:51:49.236 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:48 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.1 2023-03-09T14:51:49.236 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:48 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.4 2023-03-09T14:51:49.236 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:48 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.1 2023-03-09T14:51:49.236 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:48 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.10 2023-03-09T14:51:49.236 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:48 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.8 2023-03-09T14:51:49.236 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:48 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.4 2023-03-09T14:51:49.237 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:48 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.8 2023-03-09T14:51:49.237 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:48 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.5 2023-03-09T14:51:49.237 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:48 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.2 2023-03-09T14:51:49.237 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:48 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.5 2023-03-09T14:51:49.237 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:48 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.2 2023-03-09T14:51:49.237 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:48 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.5 2023-03-09T14:51:49.237 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:48 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.9 2023-03-09T14:51:49.238 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:48 smithi181 ceph-mon[94189]: osd.3 reported immediately failed by osd.9 2023-03-09T14:51:49.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-mon[92422]: osdmap e142: 12 total, 12 up, 12 in 2023-03-09T14:51:49.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.9 2023-03-09T14:51:49.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-mon[92422]: osd.3 failed (root=default,host=smithi038) (connection refused reported by osd.9) 2023-03-09T14:51:49.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.9 2023-03-09T14:51:49.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.0 2023-03-09T14:51:49.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.0 2023-03-09T14:51:49.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.6 2023-03-09T14:51:49.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.6 2023-03-09T14:51:49.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.6 2023-03-09T14:51:49.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.7 2023-03-09T14:51:49.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.7 2023-03-09T14:51:49.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.1 2023-03-09T14:51:49.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.11 2023-03-09T14:51:49.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.4 2023-03-09T14:51:49.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.7 2023-03-09T14:51:49.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.11 2023-03-09T14:51:49.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.10 2023-03-09T14:51:49.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.1 2023-03-09T14:51:49.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.4 2023-03-09T14:51:49.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.1 2023-03-09T14:51:49.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.10 2023-03-09T14:51:49.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.8 2023-03-09T14:51:49.254 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.4 2023-03-09T14:51:49.254 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.8 2023-03-09T14:51:49.254 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.5 2023-03-09T14:51:49.254 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.2 2023-03-09T14:51:49.254 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.5 2023-03-09T14:51:49.254 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.2 2023-03-09T14:51:49.254 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.5 2023-03-09T14:51:49.254 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.9 2023-03-09T14:51:49.255 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:48 smithi038 ceph-mon[92422]: osd.3 reported immediately failed by osd.9 2023-03-09T14:51:50.112 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:49 smithi038 ceph-mon[92422]: pgmap v222: 129 pgs: 7 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 121 active+clean; 18 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.7 MiB/s wr, 610 op/s; 180/34474 objects degraded (0.522%); 163 MiB/s, 27 keys/s, 43 objects/s recovering 2023-03-09T14:51:50.112 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:49 smithi038 ceph-mon[92422]: Health check failed: 1 osds down (OSD_DOWN) 2023-03-09T14:51:50.112 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:49 smithi038 ceph-mon[92422]: osdmap e143: 12 total, 11 up, 12 in 2023-03-09T14:51:50.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:49 smithi122 ceph-mon[93869]: pgmap v222: 129 pgs: 7 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 121 active+clean; 18 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.7 MiB/s wr, 610 op/s; 180/34474 objects degraded (0.522%); 163 MiB/s, 27 keys/s, 43 objects/s recovering 2023-03-09T14:51:50.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:49 smithi122 ceph-mon[93869]: Health check failed: 1 osds down (OSD_DOWN) 2023-03-09T14:51:50.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:49 smithi122 ceph-mon[93869]: osdmap e143: 12 total, 11 up, 12 in 2023-03-09T14:51:50.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:49 smithi181 ceph-mon[94189]: pgmap v222: 129 pgs: 7 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 121 active+clean; 18 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 4.7 MiB/s wr, 610 op/s; 180/34474 objects degraded (0.522%); 163 MiB/s, 27 keys/s, 43 objects/s recovering 2023-03-09T14:51:50.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:49 smithi181 ceph-mon[94189]: Health check failed: 1 osds down (OSD_DOWN) 2023-03-09T14:51:50.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:49 smithi181 ceph-mon[94189]: osdmap e143: 12 total, 11 up, 12 in 2023-03-09T14:51:50.374 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 2023-03-09T14:51:50.336+0000 7f265dc04700 -1 osd.0 pg_epoch: 144 pg[4.1es0( v 144'1530 (0'0,144'1530] local-lis/les=138/139 n=258 ec=85/85 lis/c=138/138 les/c/f=139/139/0 sis=138) [0,7,6,1]p0(0) r=0 lpr=138 luod=144'1528 crt=144'1528 lcod 144'1527 mlcod 144'1527 active+clean] bad op order, already applied 87406 > this 87403 2023-03-09T14:51:50.374 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7f265dc04700 time 2023-03-09T14:51:50.337190+0000 2023-03-09T14:51:50.375 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op") 2023-03-09T14:51:50.375 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:51:50.375 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x55c9f568cc85] 2023-03-09T14:51:50.375 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 2: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55c9f59456e4] 2023-03-09T14:51:50.375 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55c9f594bd5e] 2023-03-09T14:51:50.375 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 4: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55c9f579b986] 2023-03-09T14:51:50.375 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 5: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55c9f5ab1729] 2023-03-09T14:51:50.376 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55c9f57b0ebf] 2023-03-09T14:51:50.376 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55c9f5e62d85] 2023-03-09T14:51:50.376 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55c9f5e65f84] 2023-03-09T14:51:50.376 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f26821671ca] 2023-03-09T14:51:50.376 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 10: clone() 2023-03-09T14:51:50.376 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: *** Caught signal (Aborted) ** 2023-03-09T14:51:50.376 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: in thread 7f265dc04700 thread_name:tp_osd_tp 2023-03-09T14:51:50.377 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 2023-03-09T14:51:50.340+0000 7f265dc04700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7f265dc04700 time 2023-03-09T14:51:50.337190+0000 2023-03-09T14:51:50.377 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op") 2023-03-09T14:51:50.377 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 2023-03-09T14:51:50.377 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:51:50.377 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x55c9f568cc85] 2023-03-09T14:51:50.377 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 2: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55c9f59456e4] 2023-03-09T14:51:50.377 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55c9f594bd5e] 2023-03-09T14:51:50.378 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 4: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55c9f579b986] 2023-03-09T14:51:50.378 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 5: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55c9f5ab1729] 2023-03-09T14:51:50.378 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55c9f57b0ebf] 2023-03-09T14:51:50.378 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55c9f5e62d85] 2023-03-09T14:51:50.378 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55c9f5e65f84] 2023-03-09T14:51:50.378 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f26821671ca] 2023-03-09T14:51:50.378 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 10: clone() 2023-03-09T14:51:50.378 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 2023-03-09T14:51:50.379 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:51:50.379 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f2682171cf0] 2023-03-09T14:51:50.379 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 2: gsignal() 2023-03-09T14:51:50.379 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 3: abort() 2023-03-09T14:51:50.379 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x55c9f568cd45] 2023-03-09T14:51:50.379 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 5: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55c9f59456e4] 2023-03-09T14:51:50.379 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 6: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55c9f594bd5e] 2023-03-09T14:51:50.379 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55c9f579b986] 2023-03-09T14:51:50.380 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 8: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55c9f5ab1729] 2023-03-09T14:51:50.380 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55c9f57b0ebf] 2023-03-09T14:51:50.380 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55c9f5e62d85] 2023-03-09T14:51:50.380 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55c9f5e65f84] 2023-03-09T14:51:50.380 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f26821671ca] 2023-03-09T14:51:50.380 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 13: clone() 2023-03-09T14:51:50.380 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 2023-03-09T14:51:50.344+0000 7f265dc04700 -1 *** Caught signal (Aborted) ** 2023-03-09T14:51:50.381 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: in thread 7f265dc04700 thread_name:tp_osd_tp 2023-03-09T14:51:50.381 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 2023-03-09T14:51:50.381 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:51:50.381 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f2682171cf0] 2023-03-09T14:51:50.381 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 2: gsignal() 2023-03-09T14:51:50.381 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 3: abort() 2023-03-09T14:51:50.381 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x55c9f568cd45] 2023-03-09T14:51:50.381 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 5: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55c9f59456e4] 2023-03-09T14:51:50.382 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 6: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55c9f594bd5e] 2023-03-09T14:51:50.382 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55c9f579b986] 2023-03-09T14:51:50.382 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 8: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55c9f5ab1729] 2023-03-09T14:51:50.382 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55c9f57b0ebf] 2023-03-09T14:51:50.382 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55c9f5e62d85] 2023-03-09T14:51:50.382 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55c9f5e65f84] 2023-03-09T14:51:50.382 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f26821671ca] 2023-03-09T14:51:50.383 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 13: clone() 2023-03-09T14:51:50.383 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-03-09T14:51:50.383 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 2023-03-09T14:51:50.383 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: -223> 2023-03-09T14:51:50.336+0000 7f265dc04700 -1 osd.0 pg_epoch: 144 pg[4.1es0( v 144'1530 (0'0,144'1530] local-lis/les=138/139 n=258 ec=85/85 lis/c=138/138 les/c/f=139/139/0 sis=138) [0,7,6,1]p0(0) r=0 lpr=138 luod=144'1528 crt=144'1528 lcod 144'1527 mlcod 144'1527 active+clean] bad op order, already applied 87406 > this 87403 2023-03-09T14:51:50.383 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: -165> 2023-03-09T14:51:50.340+0000 7f265dc04700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7f265dc04700 time 2023-03-09T14:51:50.337190+0000 2023-03-09T14:51:50.383 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op") 2023-03-09T14:51:50.383 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 2023-03-09T14:51:50.383 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:51:50.384 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x55c9f568cc85] 2023-03-09T14:51:50.384 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 2: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55c9f59456e4] 2023-03-09T14:51:50.384 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55c9f594bd5e] 2023-03-09T14:51:50.384 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 4: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55c9f579b986] 2023-03-09T14:51:50.384 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 5: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55c9f5ab1729] 2023-03-09T14:51:50.384 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55c9f57b0ebf] 2023-03-09T14:51:50.384 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55c9f5e62d85] 2023-03-09T14:51:50.385 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55c9f5e65f84] 2023-03-09T14:51:50.385 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f26821671ca] 2023-03-09T14:51:50.385 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 10: clone() 2023-03-09T14:51:50.385 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 2023-03-09T14:51:50.385 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 0> 2023-03-09T14:51:50.344+0000 7f265dc04700 -1 *** Caught signal (Aborted) ** 2023-03-09T14:51:50.385 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: in thread 7f265dc04700 thread_name:tp_osd_tp 2023-03-09T14:51:50.385 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 2023-03-09T14:51:50.385 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:51:50.386 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:51:50 smithi038 systemd[1]: ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.3.service: Failed with result 'exit-code'. 2023-03-09T14:51:50.728 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 2023-03-09T14:51:50.361+0000 7ff1f5d7d700 -1 osd.1 pg_epoch: 144 pg[4.as0( v 144'3138 (0'0,144'3138] local-lis/les=138/139 n=297 ec=85/85 lis/c=138/138 les/c/f=139/139/0 sis=138) [1,5,6,7]p1(0) r=0 lpr=138 luod=144'3136 crt=144'3136 lcod 144'3135 mlcod 144'3135 active+clean] bad op order, already applied 87436 > this 87434 2023-03-09T14:51:50.728 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7ff1f5d7d700 time 2023-03-09T14:51:50.362099+0000 2023-03-09T14:51:50.729 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op") 2023-03-09T14:51:50.729 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:51:50.729 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x55c2d1196c85] 2023-03-09T14:51:50.729 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 2: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55c2d144f6e4] 2023-03-09T14:51:50.729 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55c2d1455d5e] 2023-03-09T14:51:50.729 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 4: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55c2d12a5986] 2023-03-09T14:51:50.729 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 5: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55c2d15bb729] 2023-03-09T14:51:50.730 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55c2d12baebf] 2023-03-09T14:51:50.730 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55c2d196cd85] 2023-03-09T14:51:50.730 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55c2d196ff84] 2023-03-09T14:51:50.730 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7ff21e2e81ca] 2023-03-09T14:51:50.730 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 10: clone() 2023-03-09T14:51:50.730 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: *** Caught signal (Aborted) ** 2023-03-09T14:51:50.730 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: in thread 7ff1f5d7d700 thread_name:tp_osd_tp 2023-03-09T14:51:50.731 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:51:50.731 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7ff21e2f2cf0] 2023-03-09T14:51:50.731 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 2: gsignal() 2023-03-09T14:51:50.731 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 3: abort() 2023-03-09T14:51:50.731 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x55c2d1196d45] 2023-03-09T14:51:50.731 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 5: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55c2d144f6e4] 2023-03-09T14:51:50.731 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 6: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55c2d1455d5e] 2023-03-09T14:51:50.731 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55c2d12a5986] 2023-03-09T14:51:50.732 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 8: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55c2d15bb729] 2023-03-09T14:51:50.732 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55c2d12baebf] 2023-03-09T14:51:50.732 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55c2d196cd85] 2023-03-09T14:51:50.732 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55c2d196ff84] 2023-03-09T14:51:50.732 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7ff21e2e81ca] 2023-03-09T14:51:50.732 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 13: clone() 2023-03-09T14:51:50.732 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 2023-03-09T14:51:50.367+0000 7ff1f5d7d700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7ff1f5d7d700 time 2023-03-09T14:51:50.362099+0000 2023-03-09T14:51:50.732 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op") 2023-03-09T14:51:50.733 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 2023-03-09T14:51:50.733 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:51:50.733 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x55c2d1196c85] 2023-03-09T14:51:50.733 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 2: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55c2d144f6e4] 2023-03-09T14:51:50.733 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55c2d1455d5e] 2023-03-09T14:51:50.733 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 4: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55c2d12a5986] 2023-03-09T14:51:50.733 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 5: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55c2d15bb729] 2023-03-09T14:51:50.734 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55c2d12baebf] 2023-03-09T14:51:50.734 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55c2d196cd85] 2023-03-09T14:51:50.735 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55c2d196ff84] 2023-03-09T14:51:50.735 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7ff21e2e81ca] 2023-03-09T14:51:50.735 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 10: clone() 2023-03-09T14:51:50.735 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 2023-03-09T14:51:50.735 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 2023-03-09T14:51:50.371+0000 7ff1f5d7d700 -1 *** Caught signal (Aborted) ** 2023-03-09T14:51:50.736 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: in thread 7ff1f5d7d700 thread_name:tp_osd_tp 2023-03-09T14:51:50.736 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 2023-03-09T14:51:50.736 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:51:50.736 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7ff21e2f2cf0] 2023-03-09T14:51:50.736 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 2: gsignal() 2023-03-09T14:51:50.736 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 3: abort() 2023-03-09T14:51:50.736 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x55c2d1196d45] 2023-03-09T14:51:50.737 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 5: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55c2d144f6e4] 2023-03-09T14:51:50.737 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 6: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55c2d1455d5e] 2023-03-09T14:51:50.737 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55c2d12a5986] 2023-03-09T14:51:50.737 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 8: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55c2d15bb729] 2023-03-09T14:51:50.737 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55c2d12baebf] 2023-03-09T14:51:50.737 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55c2d196cd85] 2023-03-09T14:51:50.737 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55c2d196ff84] 2023-03-09T14:51:50.737 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7ff21e2e81ca] 2023-03-09T14:51:50.738 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 13: clone() 2023-03-09T14:51:50.738 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-03-09T14:51:50.738 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 2023-03-09T14:51:50.738 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: -1657> 2023-03-09T14:51:50.361+0000 7ff1f5d7d700 -1 osd.1 pg_epoch: 144 pg[4.as0( v 144'3138 (0'0,144'3138] local-lis/les=138/139 n=297 ec=85/85 lis/c=138/138 les/c/f=139/139/0 sis=138) [1,5,6,7]p1(0) r=0 lpr=138 luod=144'3136 crt=144'3136 lcod 144'3135 mlcod 144'3135 active+clean] bad op order, already applied 87436 > this 87434 2023-03-09T14:51:50.738 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: -963> 2023-03-09T14:51:50.367+0000 7ff1f5d7d700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7ff1f5d7d700 time 2023-03-09T14:51:50.362099+0000 2023-03-09T14:51:50.738 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op") 2023-03-09T14:51:50.738 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 2023-03-09T14:51:50.739 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:51:50.739 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x55c2d1196c85] 2023-03-09T14:51:50.739 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 2: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55c2d144f6e4] 2023-03-09T14:51:50.739 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55c2d1455d5e] 2023-03-09T14:51:50.739 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 4: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55c2d12a5986] 2023-03-09T14:51:50.739 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 5: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55c2d15bb729] 2023-03-09T14:51:50.739 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55c2d12baebf] 2023-03-09T14:51:50.739 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55c2d196cd85] 2023-03-09T14:51:50.740 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55c2d196ff84] 2023-03-09T14:51:50.740 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7ff21e2e81ca] 2023-03-09T14:51:50.740 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 10: clone() 2023-03-09T14:51:50.740 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 2023-03-09T14:51:50.740 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: -223> 2023-03-09T14:51:50.371+0000 7ff1f5d7d700 -1 *** Caught signal (Aborted) ** 2023-03-09T14:51:50.740 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: in thread 7ff1f5d7d700 thread_name:tp_osd_tp 2023-03-09T14:51:50.740 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 2023-03-09T14:51:50.740 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:51:50.741 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7ff21e2f2cf0] 2023-03-09T14:51:50.741 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 2: gsignal() 2023-03-09T14:51:50.741 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 3: abort() 2023-03-09T14:51:50.741 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x55c2d1196d45] 2023-03-09T14:51:50.741 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 5: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55c2d144f6e4] 2023-03-09T14:51:50.741 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 6: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55c2d1455d5e] 2023-03-09T14:51:50.741 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55c2d12a5986] 2023-03-09T14:51:50.742 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 8: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55c2d15bb729] 2023-03-09T14:51:50.742 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55c2d12baebf] 2023-03-09T14:51:50.742 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55c2d196cd85] 2023-03-09T14:51:50.742 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55c2d196ff84] 2023-03-09T14:51:50.742 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7ff21e2e81ca] 2023-03-09T14:51:50.742 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 13: clone() 2023-03-09T14:51:50.742 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-03-09T14:51:50.742 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 2023-03-09T14:51:50.743 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: -2658> 2023-03-09T14:51:50.361+0000 7ff1f5d7d700 -1 osd.1 pg_epoch: 144 pg[4.as0( v 144'3138 (0'0,144'3138] local-lis/les=138/139 n=297 ec=85/85 lis/c=138/138 les/c/f=139/139/0 sis=138) [1,5,6,7]p1(0) r=0 lpr=138 luod=144'3136 crt=144'3136 lcod 144'3135 mlcod 144'3135 active+clean] bad op order, already applied 87436 > this 87434 2023-03-09T14:51:50.743 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: -1964> 2023-03-09T14:51:50.367+0000 7ff1f5d7d700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7ff1f5d7d700 time 2023-03-09T14:51:50.362099+0000 2023-03-09T14:51:50.743 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op") 2023-03-09T14:51:50.743 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 2023-03-09T14:51:50.743 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:51:50.743 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x55c2d1196c85] 2023-03-09T14:51:50.743 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 2: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55c2d144f6e4] 2023-03-09T14:51:50.744 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55c2d1455d5e] 2023-03-09T14:51:50.744 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 4: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55c2d12a5986] 2023-03-09T14:51:50.744 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 5: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55c2d15bb729] 2023-03-09T14:51:50.744 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55c2d12baebf] 2023-03-09T14:51:50.744 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55c2d196cd85] 2023-03-09T14:51:50.744 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55c2d196ff84] 2023-03-09T14:51:50.744 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7ff21e2e81ca] 2023-03-09T14:51:50.744 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 10: clone() 2023-03-09T14:51:50.745 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 2023-03-09T14:51:50.745 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: -1224> 2023-03-09T14:51:50.371+0000 7ff1f5d7d700 -1 *** Caught signal (Aborted) ** 2023-03-09T14:51:50.745 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: in thread 7ff1f5d7d700 thread_name:tp_osd_tp 2023-03-09T14:51:50.745 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 2023-03-09T14:51:50.745 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:51:50.745 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7ff21e2f2cf0] 2023-03-09T14:51:50.745 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 2: gsignal() 2023-03-09T14:51:50.745 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 3: abort() 2023-03-09T14:51:50.746 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x55c2d1196d45] 2023-03-09T14:51:50.746 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 5: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55c2d144f6e4] 2023-03-09T14:51:50.746 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 6: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55c2d1455d5e] 2023-03-09T14:51:50.746 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55c2d12a5986] 2023-03-09T14:51:50.746 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 8: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55c2d15bb729] 2023-03-09T14:51:50.746 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55c2d12baebf] 2023-03-09T14:51:50.746 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55c2d196cd85] 2023-03-09T14:51:50.746 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55c2d196ff84] 2023-03-09T14:51:50.747 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7ff21e2e81ca] 2023-03-09T14:51:50.747 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 13: clone() 2023-03-09T14:51:50.747 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-03-09T14:51:50.747 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[126291]: 2023-03-09T14:51:50.748 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f2682171cf0] 2023-03-09T14:51:50.748 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 2: gsignal() 2023-03-09T14:51:50.748 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 3: abort() 2023-03-09T14:51:50.748 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x55c9f568cd45] 2023-03-09T14:51:50.748 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 5: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55c9f59456e4] 2023-03-09T14:51:50.748 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 6: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55c9f594bd5e] 2023-03-09T14:51:50.749 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55c9f579b986] 2023-03-09T14:51:50.749 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 8: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55c9f5ab1729] 2023-03-09T14:51:50.749 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55c9f57b0ebf] 2023-03-09T14:51:50.749 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55c9f5e62d85] 2023-03-09T14:51:50.749 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55c9f5e65f84] 2023-03-09T14:51:50.749 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f26821671ca] 2023-03-09T14:51:50.749 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 13: clone() 2023-03-09T14:51:50.749 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-03-09T14:51:50.750 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 2023-03-09T14:51:50.750 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: -1051> 2023-03-09T14:51:50.336+0000 7f265dc04700 -1 osd.0 pg_epoch: 144 pg[4.1es0( v 144'1530 (0'0,144'1530] local-lis/les=138/139 n=258 ec=85/85 lis/c=138/138 les/c/f=139/139/0 sis=138) [0,7,6,1]p0(0) r=0 lpr=138 luod=144'1528 crt=144'1528 lcod 144'1527 mlcod 144'1527 active+clean] bad op order, already applied 87406 > this 87403 2023-03-09T14:51:50.750 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: -993> 2023-03-09T14:51:50.340+0000 7f265dc04700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7f265dc04700 time 2023-03-09T14:51:50.337190+0000 2023-03-09T14:51:50.750 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op") 2023-03-09T14:51:50.750 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 2023-03-09T14:51:50.750 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:51:50.750 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x55c9f568cc85] 2023-03-09T14:51:50.751 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 2: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55c9f59456e4] 2023-03-09T14:51:50.751 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55c9f594bd5e] 2023-03-09T14:51:50.751 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 4: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55c9f579b986] 2023-03-09T14:51:50.751 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 5: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55c9f5ab1729] 2023-03-09T14:51:50.751 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55c9f57b0ebf] 2023-03-09T14:51:50.751 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55c9f5e62d85] 2023-03-09T14:51:50.751 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55c9f5e65f84] 2023-03-09T14:51:50.751 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f26821671ca] 2023-03-09T14:51:50.752 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 10: clone() 2023-03-09T14:51:50.752 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 2023-03-09T14:51:50.752 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: -828> 2023-03-09T14:51:50.344+0000 7f265dc04700 -1 *** Caught signal (Aborted) ** 2023-03-09T14:51:50.752 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: in thread 7f265dc04700 thread_name:tp_osd_tp 2023-03-09T14:51:50.752 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 2023-03-09T14:51:50.752 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:51:50.752 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f2682171cf0] 2023-03-09T14:51:50.752 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 2: gsignal() 2023-03-09T14:51:50.753 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 3: abort() 2023-03-09T14:51:50.753 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x55c9f568cd45] 2023-03-09T14:51:50.753 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 5: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55c9f59456e4] 2023-03-09T14:51:50.753 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 6: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55c9f594bd5e] 2023-03-09T14:51:50.753 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55c9f579b986] 2023-03-09T14:51:50.753 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 8: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55c9f5ab1729] 2023-03-09T14:51:50.753 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55c9f57b0ebf] 2023-03-09T14:51:50.753 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55c9f5e62d85] 2023-03-09T14:51:50.754 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55c9f5e65f84] 2023-03-09T14:51:50.754 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f26821671ca] 2023-03-09T14:51:50.754 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 13: clone() 2023-03-09T14:51:50.754 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-03-09T14:51:50.754 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[105229]: 2023-03-09T14:51:51.166 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-mon[93869]: osdmap e144: 12 total, 11 up, 12 in 2023-03-09T14:51:51.166 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.8 2023-03-09T14:51:51.166 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-mon[93869]: osd.1 failed (root=default,host=smithi122) (connection refused reported by osd.8) 2023-03-09T14:51:51.167 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.11 2023-03-09T14:51:51.167 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.11 2023-03-09T14:51:51.167 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-mon[93869]: osd.0 failed (root=default,host=smithi038) (connection refused reported by osd.11) 2023-03-09T14:51:51.167 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.9 2023-03-09T14:51:51.167 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.6 2023-03-09T14:51:51.167 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.9 2023-03-09T14:51:51.167 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.11 2023-03-09T14:51:51.168 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.11 2023-03-09T14:51:51.168 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.10 2023-03-09T14:51:51.168 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.8 2023-03-09T14:51:51.168 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.6 2023-03-09T14:51:51.168 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.2 2023-03-09T14:51:51.168 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.8 2023-03-09T14:51:51.168 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.2 2023-03-09T14:51:51.168 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.5 2023-03-09T14:51:51.169 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.5 2023-03-09T14:51:51.169 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.10 2023-03-09T14:51:51.169 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.4 2023-03-09T14:51:51.169 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.10 2023-03-09T14:51:51.169 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.4 2023-03-09T14:51:51.169 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.7 2023-03-09T14:51:51.169 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.7 2023-03-09T14:51:51.169 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.11 2023-03-09T14:51:51.170 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.4 2023-03-09T14:51:51.170 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.10 2023-03-09T14:51:51.170 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.7 2023-03-09T14:51:51.170 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.10 2023-03-09T14:51:51.170 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.7 2023-03-09T14:51:51.170 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.11 2023-03-09T14:51:51.170 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.4 2023-03-09T14:51:51.170 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.2 2023-03-09T14:51:51.171 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.8 2023-03-09T14:51:51.171 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.2 2023-03-09T14:51:51.171 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.8 2023-03-09T14:51:51.171 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.5 2023-03-09T14:51:51.171 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.5 2023-03-09T14:51:51.171 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.9 2023-03-09T14:51:51.171 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.6 2023-03-09T14:51:51.171 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.9 2023-03-09T14:51:51.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.6 2023-03-09T14:51:51.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.7 2023-03-09T14:51:51.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.6 2023-03-09T14:51:51.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.6 2023-03-09T14:51:51.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.5 2023-03-09T14:51:51.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.2 2023-03-09T14:51:51.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.9 2023-03-09T14:51:51.172 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:50 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.9 2023-03-09T14:51:51.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:50 smithi181 ceph-mon[94189]: osdmap e144: 12 total, 11 up, 12 in 2023-03-09T14:51:51.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:50 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.8 2023-03-09T14:51:51.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:50 smithi181 ceph-mon[94189]: osd.1 failed (root=default,host=smithi122) (connection refused reported by osd.8) 2023-03-09T14:51:51.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:50 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.11 2023-03-09T14:51:51.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:50 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.11 2023-03-09T14:51:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:50 smithi181 ceph-mon[94189]: osd.0 failed (root=default,host=smithi038) (connection refused reported by osd.11) 2023-03-09T14:51:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:50 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.9 2023-03-09T14:51:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:50 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.6 2023-03-09T14:51:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:50 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.9 2023-03-09T14:51:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:50 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.11 2023-03-09T14:51:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:50 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.11 2023-03-09T14:51:51.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:50 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.10 2023-03-09T14:51:51.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:50 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.8 2023-03-09T14:51:51.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:50 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.6 2023-03-09T14:51:51.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:50 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.2 2023-03-09T14:51:51.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:50 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.8 2023-03-09T14:51:51.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:50 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.2 2023-03-09T14:51:51.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:50 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.5 2023-03-09T14:51:51.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:50 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.5 2023-03-09T14:51:51.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:50 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.10 2023-03-09T14:51:51.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:50 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.4 2023-03-09T14:51:51.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:50 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.10 2023-03-09T14:51:51.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:50 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.4 2023-03-09T14:51:51.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:50 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.7 2023-03-09T14:51:51.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:50 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.7 2023-03-09T14:51:51.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:50 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.11 2023-03-09T14:51:51.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:50 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.4 2023-03-09T14:51:51.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:50 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.10 2023-03-09T14:51:51.236 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:50 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.7 2023-03-09T14:51:51.236 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:50 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.10 2023-03-09T14:51:51.236 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:50 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.7 2023-03-09T14:51:51.236 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:50 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.11 2023-03-09T14:51:51.236 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:50 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.4 2023-03-09T14:51:51.236 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:50 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.2 2023-03-09T14:51:51.236 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:50 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.8 2023-03-09T14:51:51.237 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:50 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.2 2023-03-09T14:51:51.237 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:50 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.8 2023-03-09T14:51:51.237 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:50 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.5 2023-03-09T14:51:51.237 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:50 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.5 2023-03-09T14:51:51.237 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:50 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.9 2023-03-09T14:51:51.237 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:50 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.6 2023-03-09T14:51:51.237 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:50 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.9 2023-03-09T14:51:51.237 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:50 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.6 2023-03-09T14:51:51.238 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:50 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.7 2023-03-09T14:51:51.238 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:50 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.6 2023-03-09T14:51:51.238 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:50 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.6 2023-03-09T14:51:51.238 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:50 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.5 2023-03-09T14:51:51.238 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:50 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.2 2023-03-09T14:51:51.238 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:50 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.9 2023-03-09T14:51:51.238 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:50 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.9 2023-03-09T14:51:51.250 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:50 smithi038 systemd[1]: ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.0.service: Main process exited, code=exited, status=134/n/a 2023-03-09T14:51:51.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-mon[92422]: osdmap e144: 12 total, 11 up, 12 in 2023-03-09T14:51:51.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.8 2023-03-09T14:51:51.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-mon[92422]: osd.1 failed (root=default,host=smithi122) (connection refused reported by osd.8) 2023-03-09T14:51:51.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.11 2023-03-09T14:51:51.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.11 2023-03-09T14:51:51.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-mon[92422]: osd.0 failed (root=default,host=smithi038) (connection refused reported by osd.11) 2023-03-09T14:51:51.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.9 2023-03-09T14:51:51.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.6 2023-03-09T14:51:51.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.9 2023-03-09T14:51:51.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.11 2023-03-09T14:51:51.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.11 2023-03-09T14:51:51.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.10 2023-03-09T14:51:51.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.8 2023-03-09T14:51:51.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.6 2023-03-09T14:51:51.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.2 2023-03-09T14:51:51.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.8 2023-03-09T14:51:51.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.2 2023-03-09T14:51:51.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.5 2023-03-09T14:51:51.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.5 2023-03-09T14:51:51.254 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.10 2023-03-09T14:51:51.254 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.4 2023-03-09T14:51:51.254 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.10 2023-03-09T14:51:51.254 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.4 2023-03-09T14:51:51.254 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.7 2023-03-09T14:51:51.254 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.7 2023-03-09T14:51:51.254 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.11 2023-03-09T14:51:51.254 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.4 2023-03-09T14:51:51.255 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.10 2023-03-09T14:51:51.255 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.7 2023-03-09T14:51:51.255 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.10 2023-03-09T14:51:51.255 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.7 2023-03-09T14:51:51.255 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.11 2023-03-09T14:51:51.255 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.4 2023-03-09T14:51:51.255 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.2 2023-03-09T14:51:51.255 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.8 2023-03-09T14:51:51.256 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.2 2023-03-09T14:51:51.256 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.8 2023-03-09T14:51:51.256 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.5 2023-03-09T14:51:51.256 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.5 2023-03-09T14:51:51.256 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.9 2023-03-09T14:51:51.256 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.6 2023-03-09T14:51:51.256 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.9 2023-03-09T14:51:51.256 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.6 2023-03-09T14:51:51.257 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.7 2023-03-09T14:51:51.257 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.6 2023-03-09T14:51:51.257 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.6 2023-03-09T14:51:51.257 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.5 2023-03-09T14:51:51.257 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.2 2023-03-09T14:51:51.257 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.9 2023-03-09T14:51:51.257 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:50 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.9 2023-03-09T14:51:52.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:51 smithi122 ceph-mon[93869]: pgmap v225: 129 pgs: 1 peering, 8 active+undersized, 5 stale+active+clean, 2 active+recovery_wait+degraded, 10 active+undersized+degraded, 103 active+clean; 18 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 9.9 MiB/s wr, 1.09k op/s; 2238/34562 objects degraded (6.475%); 192 MiB/s, 2 keys/s, 48 objects/s recovering 2023-03-09T14:51:52.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:51 smithi122 ceph-mon[93869]: osdmap e145: 12 total, 9 up, 12 in 2023-03-09T14:51:52.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:51 smithi181 ceph-mon[94189]: pgmap v225: 129 pgs: 1 peering, 8 active+undersized, 5 stale+active+clean, 2 active+recovery_wait+degraded, 10 active+undersized+degraded, 103 active+clean; 18 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 9.9 MiB/s wr, 1.09k op/s; 2238/34562 objects degraded (6.475%); 192 MiB/s, 2 keys/s, 48 objects/s recovering 2023-03-09T14:51:52.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:51 smithi181 ceph-mon[94189]: osdmap e145: 12 total, 9 up, 12 in 2023-03-09T14:51:52.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:51 smithi038 ceph-mon[92422]: pgmap v225: 129 pgs: 1 peering, 8 active+undersized, 5 stale+active+clean, 2 active+recovery_wait+degraded, 10 active+undersized+degraded, 103 active+clean; 18 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 9.9 MiB/s wr, 1.09k op/s; 2238/34562 objects degraded (6.475%); 192 MiB/s, 2 keys/s, 48 objects/s recovering 2023-03-09T14:51:52.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:51 smithi038 ceph-mon[92422]: osdmap e145: 12 total, 9 up, 12 in 2023-03-09T14:51:53.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:52 smithi122 ceph-mon[93869]: osdmap e146: 12 total, 9 up, 12 in 2023-03-09T14:51:53.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:52 smithi122 ceph-mon[93869]: Health check update: Degraded data redundancy: 2238/34562 objects degraded (6.475%), 12 pgs degraded (PG_DEGRADED) 2023-03-09T14:51:53.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:52 smithi181 ceph-mon[94189]: osdmap e146: 12 total, 9 up, 12 in 2023-03-09T14:51:53.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:52 smithi181 ceph-mon[94189]: Health check update: Degraded data redundancy: 2238/34562 objects degraded (6.475%), 12 pgs degraded (PG_DEGRADED) 2023-03-09T14:51:53.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:52 smithi038 ceph-mon[92422]: osdmap e146: 12 total, 9 up, 12 in 2023-03-09T14:51:53.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:52 smithi038 ceph-mon[92422]: Health check update: Degraded data redundancy: 2238/34562 objects degraded (6.475%), 12 pgs degraded (PG_DEGRADED) 2023-03-09T14:51:53.728 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:53 smithi122 systemd[1]: ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.1.service: Main process exited, code=exited, status=134/n/a 2023-03-09T14:51:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 85.0 s, 114.1 tps, lat 117.941 ms stddev 364.942 2023-03-09T14:51:54.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:53 smithi122 ceph-mon[93869]: pgmap v228: 129 pgs: 8 peering, 13 active+undersized, 19 stale+active+clean, 2 undersized+degraded+peered, 16 active+undersized+degraded, 71 active+clean; 18 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 6.4 MiB/s wr, 864 op/s; 3552/34574 objects degraded (10.274%); 179 MiB/s, 44 objects/s recovering 2023-03-09T14:51:54.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:53 smithi122 ceph-mon[93869]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2023-03-09T14:51:54.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:53 smithi181 ceph-mon[94189]: pgmap v228: 129 pgs: 8 peering, 13 active+undersized, 19 stale+active+clean, 2 undersized+degraded+peered, 16 active+undersized+degraded, 71 active+clean; 18 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 6.4 MiB/s wr, 864 op/s; 3552/34574 objects degraded (10.274%); 179 MiB/s, 44 objects/s recovering 2023-03-09T14:51:54.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:53 smithi181 ceph-mon[94189]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2023-03-09T14:51:54.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:53 smithi038 ceph-mon[92422]: pgmap v228: 129 pgs: 8 peering, 13 active+undersized, 19 stale+active+clean, 2 undersized+degraded+peered, 16 active+undersized+degraded, 71 active+clean; 18 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 6.4 MiB/s wr, 864 op/s; 3552/34574 objects degraded (10.274%); 179 MiB/s, 44 objects/s recovering 2023-03-09T14:51:54.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:53 smithi038 ceph-mon[92422]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2023-03-09T14:51:54.750 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:51:54 smithi038 systemd[1]: ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.0.service: Failed with result 'exit-code'. 2023-03-09T14:51:55.232 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7fbe0d6f9700 time 2023-03-09T14:51:54.875325+0000 2023-03-09T14:51:55.233 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op") 2023-03-09T14:51:55.233 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:51:55.233 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x565330533c85] 2023-03-09T14:51:55.234 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 2: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x5653307ec6e4] 2023-03-09T14:51:55.234 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x5653307f2d5e] 2023-03-09T14:51:55.234 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 4: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x565330642986] 2023-03-09T14:51:55.235 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 5: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x565330958729] 2023-03-09T14:51:55.235 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x565330657ebf] 2023-03-09T14:51:55.235 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x565330d09d85] 2023-03-09T14:51:55.235 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x565330d0cf84] 2023-03-09T14:51:55.236 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7fbe354631ca] 2023-03-09T14:51:55.236 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 10: clone() 2023-03-09T14:51:55.236 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: *** Caught signal (Aborted) ** 2023-03-09T14:51:55.236 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: in thread 7fbe0d6f9700 thread_name:tp_osd_tp 2023-03-09T14:51:55.237 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 2023-03-09T14:51:54.874+0000 7fbe0d6f9700 -1 osd.8 pg_epoch: 146 pg[4.2s0( v 146'839 (0'0,146'839] local-lis/les=127/128 n=278 ec=85/85 lis/c=127/127 les/c/f=128/128/0 sis=127) [8,10,11,9]p8(0) r=0 lpr=127 luod=146'835 crt=146'833 lcod 146'833 mlcod 146'833 active+clean] bad op order, already applied 87665 > this 87664 2023-03-09T14:51:55.237 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 2023-03-09T14:51:54.878+0000 7fbe0d6f9700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7fbe0d6f9700 time 2023-03-09T14:51:54.875325+0000 2023-03-09T14:51:55.237 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op") 2023-03-09T14:51:55.237 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 2023-03-09T14:51:55.237 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:51:55.238 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x565330533c85] 2023-03-09T14:51:55.238 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 2: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x5653307ec6e4] 2023-03-09T14:51:55.238 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x5653307f2d5e] 2023-03-09T14:51:55.238 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 4: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x565330642986] 2023-03-09T14:51:55.238 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 5: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x565330958729] 2023-03-09T14:51:55.238 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x565330657ebf] 2023-03-09T14:51:55.238 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x565330d09d85] 2023-03-09T14:51:55.239 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x565330d0cf84] 2023-03-09T14:51:55.239 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7fbe354631ca] 2023-03-09T14:51:55.239 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 10: clone() 2023-03-09T14:51:55.239 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 2023-03-09T14:51:55.239 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:51:55.239 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7fbe3546dcf0] 2023-03-09T14:51:55.239 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 2: gsignal() 2023-03-09T14:51:55.240 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 3: abort() 2023-03-09T14:51:55.240 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x565330533d45] 2023-03-09T14:51:55.240 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 5: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x5653307ec6e4] 2023-03-09T14:51:55.240 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 6: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x5653307f2d5e] 2023-03-09T14:51:55.240 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x565330642986] 2023-03-09T14:51:55.240 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 8: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x565330958729] 2023-03-09T14:51:55.241 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x565330657ebf] 2023-03-09T14:51:55.241 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x565330d09d85] 2023-03-09T14:51:55.241 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x565330d0cf84] 2023-03-09T14:51:55.241 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fbe354631ca] 2023-03-09T14:51:55.241 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 13: clone() 2023-03-09T14:51:55.241 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 2023-03-09T14:51:54.882+0000 7fbe0d6f9700 -1 *** Caught signal (Aborted) ** 2023-03-09T14:51:55.241 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: in thread 7fbe0d6f9700 thread_name:tp_osd_tp 2023-03-09T14:51:55.242 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 2023-03-09T14:51:55.242 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:51:55.242 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7fbe3546dcf0] 2023-03-09T14:51:55.242 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 2: gsignal() 2023-03-09T14:51:55.242 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 3: abort() 2023-03-09T14:51:55.242 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x565330533d45] 2023-03-09T14:51:55.242 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 5: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x5653307ec6e4] 2023-03-09T14:51:55.243 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 6: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x5653307f2d5e] 2023-03-09T14:51:55.243 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x565330642986] 2023-03-09T14:51:55.243 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 8: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x565330958729] 2023-03-09T14:51:55.243 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x565330657ebf] 2023-03-09T14:51:55.243 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x565330d09d85] 2023-03-09T14:51:55.243 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x565330d0cf84] 2023-03-09T14:51:55.243 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fbe354631ca] 2023-03-09T14:51:55.243 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 13: clone() 2023-03-09T14:51:55.244 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-03-09T14:51:55.244 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 2023-03-09T14:51:55.244 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: -1425> 2023-03-09T14:51:54.874+0000 7fbe0d6f9700 -1 osd.8 pg_epoch: 146 pg[4.2s0( v 146'839 (0'0,146'839] local-lis/les=127/128 n=278 ec=85/85 lis/c=127/127 les/c/f=128/128/0 sis=127) [8,10,11,9]p8(0) r=0 lpr=127 luod=146'835 crt=146'833 lcod 146'833 mlcod 146'833 active+clean] bad op order, already applied 87665 > this 87664 2023-03-09T14:51:55.244 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: -916> 2023-03-09T14:51:54.878+0000 7fbe0d6f9700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7fbe0d6f9700 time 2023-03-09T14:51:54.875325+0000 2023-03-09T14:51:55.244 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op") 2023-03-09T14:51:55.244 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 2023-03-09T14:51:55.245 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:51:55.245 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x565330533c85] 2023-03-09T14:51:55.245 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 2: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x5653307ec6e4] 2023-03-09T14:51:55.245 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x5653307f2d5e] 2023-03-09T14:51:55.245 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 4: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x565330642986] 2023-03-09T14:51:55.245 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 5: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x565330958729] 2023-03-09T14:51:55.245 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x565330657ebf] 2023-03-09T14:51:55.246 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x565330d09d85] 2023-03-09T14:51:55.246 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x565330d0cf84] 2023-03-09T14:51:55.246 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7fbe354631ca] 2023-03-09T14:51:55.246 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 10: clone() 2023-03-09T14:51:55.246 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 2023-03-09T14:51:55.247 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 0> 2023-03-09T14:51:54.882+0000 7fbe0d6f9700 -1 *** Caught signal (Aborted) ** 2023-03-09T14:51:55.247 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: in thread 7fbe0d6f9700 thread_name:tp_osd_tp 2023-03-09T14:51:55.247 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 2023-03-09T14:51:55.247 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:51:55.247 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7fbe3546dcf0] 2023-03-09T14:51:55.248 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 2: gsignal() 2023-03-09T14:51:55.248 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 3: abort() 2023-03-09T14:51:55.248 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x565330533d45] 2023-03-09T14:51:55.248 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 5: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x5653307ec6e4] 2023-03-09T14:51:55.248 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 6: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x5653307f2d5e] 2023-03-09T14:51:55.248 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x565330642986] 2023-03-09T14:51:55.249 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 8: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x565330958729] 2023-03-09T14:51:55.249 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x565330657ebf] 2023-03-09T14:51:55.249 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x565330d09d85] 2023-03-09T14:51:55.249 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x565330d0cf84] 2023-03-09T14:51:55.249 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fbe354631ca] 2023-03-09T14:51:55.250 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 13: clone() 2023-03-09T14:51:55.250 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-03-09T14:51:55.250 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 2023-03-09T14:51:55.250 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: -1773> 2023-03-09T14:51:54.874+0000 7fbe0d6f9700 -1 osd.8 pg_epoch: 146 pg[4.2s0( v 146'839 (0'0,146'839] local-lis/les=127/128 n=278 ec=85/85 lis/c=127/127 les/c/f=128/128/0 sis=127) [8,10,11,9]p8(0) r=0 lpr=127 luod=146'835 crt=146'833 lcod 146'833 mlcod 146'833 active+clean] bad op order, already applied 87665 > this 87664 2023-03-09T14:51:55.250 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: -1264> 2023-03-09T14:51:54.878+0000 7fbe0d6f9700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7fbe0d6f9700 time 2023-03-09T14:51:54.875325+0000 2023-03-09T14:51:55.251 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op") 2023-03-09T14:51:55.251 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 2023-03-09T14:51:55.251 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:51:55.251 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x565330533c85] 2023-03-09T14:51:55.251 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 2: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x5653307ec6e4] 2023-03-09T14:51:55.252 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x5653307f2d5e] 2023-03-09T14:51:55.252 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 4: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x565330642986] 2023-03-09T14:51:55.252 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 5: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x565330958729] 2023-03-09T14:51:55.252 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x565330657ebf] 2023-03-09T14:51:55.252 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x565330d09d85] 2023-03-09T14:51:55.252 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x565330d0cf84] 2023-03-09T14:51:55.252 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7fbe354631ca] 2023-03-09T14:51:55.253 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 10: clone() 2023-03-09T14:51:55.253 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 2023-03-09T14:51:55.253 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: -348> 2023-03-09T14:51:54.882+0000 7fbe0d6f9700 -1 *** Caught signal (Aborted) ** 2023-03-09T14:51:55.253 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: in thread 7fbe0d6f9700 thread_name:tp_osd_tp 2023-03-09T14:51:55.253 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 2023-03-09T14:51:55.253 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:51:55.254 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7fbe3546dcf0] 2023-03-09T14:51:55.254 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 2: gsignal() 2023-03-09T14:51:55.254 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 3: abort() 2023-03-09T14:51:55.254 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x565330533d45] 2023-03-09T14:51:55.254 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 5: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x5653307ec6e4] 2023-03-09T14:51:55.254 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 6: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x5653307f2d5e] 2023-03-09T14:51:55.254 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x565330642986] 2023-03-09T14:51:55.254 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 8: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x565330958729] 2023-03-09T14:51:55.255 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x565330657ebf] 2023-03-09T14:51:55.255 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x565330d09d85] 2023-03-09T14:51:55.255 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x565330d0cf84] 2023-03-09T14:51:55.255 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fbe354631ca] 2023-03-09T14:51:55.255 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 13: clone() 2023-03-09T14:51:55.255 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-03-09T14:51:55.255 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[125996]: 2023-03-09T14:51:55.732 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:55 smithi181 systemd[1]: ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.8.service: Main process exited, code=exited, status=134/n/a 2023-03-09T14:51:56.072 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:55 smithi122 ceph-mon[93869]: pgmap v229: 129 pgs: 8 peering, 2 undersized+peered, 17 active+undersized, 15 stale+active+clean, 4 undersized+degraded+peered, 19 active+undersized+degraded, 64 active+clean; 18 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.5 MiB/s wr, 647 op/s; 4295/34606 objects degraded (12.411%); 122 MiB/s, 30 objects/s recovering 2023-03-09T14:51:56.072 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:55 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.9 2023-03-09T14:51:56.073 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:55 smithi122 ceph-mon[93869]: osd.8 failed (root=default,host=smithi181) (connection refused reported by osd.9) 2023-03-09T14:51:56.073 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:55 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.9 2023-03-09T14:51:56.073 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:55 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.11 2023-03-09T14:51:56.073 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:55 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.11 2023-03-09T14:51:56.073 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:55 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.5 2023-03-09T14:51:56.073 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:55 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.7 2023-03-09T14:51:56.073 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:55 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.6 2023-03-09T14:51:56.074 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:55 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.2 2023-03-09T14:51:56.074 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:55 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.7 2023-03-09T14:51:56.074 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:55 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.6 2023-03-09T14:51:56.074 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:55 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.5 2023-03-09T14:51:56.074 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:55 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.2 2023-03-09T14:51:56.074 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:55 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.10 2023-03-09T14:51:56.075 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:55 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.4 2023-03-09T14:51:56.075 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:55 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.10 2023-03-09T14:51:56.075 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:55 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.4 2023-03-09T14:51:56.075 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:55 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.9 2023-03-09T14:51:56.075 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:55 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.9 2023-03-09T14:51:56.075 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:55 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.11 2023-03-09T14:51:56.075 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:55 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.11 2023-03-09T14:51:56.075 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:55 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.2 2023-03-09T14:51:56.076 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:55 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.7 2023-03-09T14:51:56.076 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:55 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.6 2023-03-09T14:51:56.076 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:55 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.5 2023-03-09T14:51:56.076 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:55 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.7 2023-03-09T14:51:56.076 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:55 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.6 2023-03-09T14:51:56.076 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:55 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.2 2023-03-09T14:51:56.077 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:55 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.5 2023-03-09T14:51:56.077 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:55 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.10 2023-03-09T14:51:56.077 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:55 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.10 2023-03-09T14:51:56.077 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:55 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.4 2023-03-09T14:51:56.077 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:55 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.4 2023-03-09T14:51:56.077 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:55 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.9 2023-03-09T14:51:56.077 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:55 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.9 2023-03-09T14:51:56.078 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:55 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.10 2023-03-09T14:51:56.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:55 smithi181 ceph-mon[94189]: pgmap v229: 129 pgs: 8 peering, 2 undersized+peered, 17 active+undersized, 15 stale+active+clean, 4 undersized+degraded+peered, 19 active+undersized+degraded, 64 active+clean; 18 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.5 MiB/s wr, 647 op/s; 4295/34606 objects degraded (12.411%); 122 MiB/s, 30 objects/s recovering 2023-03-09T14:51:56.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:55 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.9 2023-03-09T14:51:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:55 smithi181 ceph-mon[94189]: osd.8 failed (root=default,host=smithi181) (connection refused reported by osd.9) 2023-03-09T14:51:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:55 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.9 2023-03-09T14:51:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:55 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.11 2023-03-09T14:51:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:55 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.11 2023-03-09T14:51:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:55 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.5 2023-03-09T14:51:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:55 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.7 2023-03-09T14:51:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:55 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.6 2023-03-09T14:51:56.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:55 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.2 2023-03-09T14:51:56.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:55 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.7 2023-03-09T14:51:56.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:55 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.6 2023-03-09T14:51:56.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:55 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.5 2023-03-09T14:51:56.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:55 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.2 2023-03-09T14:51:56.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:55 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.10 2023-03-09T14:51:56.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:55 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.4 2023-03-09T14:51:56.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:55 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.10 2023-03-09T14:51:56.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:55 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.4 2023-03-09T14:51:56.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:55 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.9 2023-03-09T14:51:56.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:55 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.9 2023-03-09T14:51:56.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:55 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.11 2023-03-09T14:51:56.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:55 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.11 2023-03-09T14:51:56.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:55 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.2 2023-03-09T14:51:56.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:55 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.7 2023-03-09T14:51:56.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:55 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.6 2023-03-09T14:51:56.236 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:55 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.5 2023-03-09T14:51:56.236 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:55 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.7 2023-03-09T14:51:56.236 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:55 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.6 2023-03-09T14:51:56.236 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:55 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.2 2023-03-09T14:51:56.236 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:55 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.5 2023-03-09T14:51:56.236 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:55 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.10 2023-03-09T14:51:56.236 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:55 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.10 2023-03-09T14:51:56.236 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:55 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.4 2023-03-09T14:51:56.237 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:55 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.4 2023-03-09T14:51:56.237 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:55 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.9 2023-03-09T14:51:56.237 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:55 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.9 2023-03-09T14:51:56.237 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:55 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.10 2023-03-09T14:51:56.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:55 smithi038 ceph-mon[92422]: pgmap v229: 129 pgs: 8 peering, 2 undersized+peered, 17 active+undersized, 15 stale+active+clean, 4 undersized+degraded+peered, 19 active+undersized+degraded, 64 active+clean; 18 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.5 MiB/s wr, 647 op/s; 4295/34606 objects degraded (12.411%); 122 MiB/s, 30 objects/s recovering 2023-03-09T14:51:56.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:55 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.9 2023-03-09T14:51:56.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:55 smithi038 ceph-mon[92422]: osd.8 failed (root=default,host=smithi181) (connection refused reported by osd.9) 2023-03-09T14:51:56.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:55 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.9 2023-03-09T14:51:56.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:55 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.11 2023-03-09T14:51:56.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:55 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.11 2023-03-09T14:51:56.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:55 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.5 2023-03-09T14:51:56.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:55 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.7 2023-03-09T14:51:56.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:55 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.6 2023-03-09T14:51:56.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:55 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.2 2023-03-09T14:51:56.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:55 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.7 2023-03-09T14:51:56.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:55 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.6 2023-03-09T14:51:56.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:55 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.5 2023-03-09T14:51:56.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:55 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.2 2023-03-09T14:51:56.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:55 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.10 2023-03-09T14:51:56.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:55 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.4 2023-03-09T14:51:56.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:55 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.10 2023-03-09T14:51:56.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:55 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.4 2023-03-09T14:51:56.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:55 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.9 2023-03-09T14:51:56.254 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:55 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.9 2023-03-09T14:51:56.254 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:55 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.11 2023-03-09T14:51:56.254 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:55 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.11 2023-03-09T14:51:56.254 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:55 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.2 2023-03-09T14:51:56.254 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:55 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.7 2023-03-09T14:51:56.254 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:55 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.6 2023-03-09T14:51:56.254 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:55 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.5 2023-03-09T14:51:56.254 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:55 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.7 2023-03-09T14:51:56.255 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:55 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.6 2023-03-09T14:51:56.255 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:55 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.2 2023-03-09T14:51:56.255 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:55 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.5 2023-03-09T14:51:56.255 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:55 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.10 2023-03-09T14:51:56.255 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:55 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.10 2023-03-09T14:51:56.255 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:55 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.4 2023-03-09T14:51:56.256 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:55 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.4 2023-03-09T14:51:56.256 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:55 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.9 2023-03-09T14:51:56.256 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:55 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.9 2023-03-09T14:51:56.257 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:55 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.10 2023-03-09T14:51:56.849 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:56 smithi122 ceph-mon[93869]: Health check update: 4 osds down (OSD_DOWN) 2023-03-09T14:51:56.849 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:56 smithi122 ceph-mon[93869]: osdmap e147: 12 total, 8 up, 12 in 2023-03-09T14:51:56.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:56 smithi181 ceph-mon[94189]: Health check update: 4 osds down (OSD_DOWN) 2023-03-09T14:51:56.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:56 smithi181 ceph-mon[94189]: osdmap e147: 12 total, 8 up, 12 in 2023-03-09T14:51:56.982 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:51:56 smithi181 systemd[1]: ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.8.service: Failed with result 'exit-code'. 2023-03-09T14:51:57.227 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:51:56 smithi122 systemd[1]: ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.1.service: Failed with result 'exit-code'. 2023-03-09T14:51:57.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:56 smithi038 ceph-mon[92422]: Health check update: 4 osds down (OSD_DOWN) 2023-03-09T14:51:57.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:56 smithi038 ceph-mon[92422]: osdmap e147: 12 total, 8 up, 12 in 2023-03-09T14:51:58.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:57 smithi122 ceph-mon[93869]: pgmap v231: 129 pgs: 1 stale+active+undersized, 2 stale+active+undersized+degraded, 1 down, 3 peering, 10 undersized+peered, 32 active+undersized, 6 stale+active+clean, 10 undersized+degraded+peered, 30 active+undersized+degraded, 34 active+clean; 18 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 251 op/s; 7944/34726 objects degraded (22.876%); 49 MiB/s, 12 objects/s recovering 2023-03-09T14:51:58.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:57 smithi122 ceph-mon[93869]: osdmap e148: 12 total, 8 up, 12 in 2023-03-09T14:51:58.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:57 smithi122 ceph-mon[93869]: Health check update: Degraded data redundancy: 7944/34726 objects degraded (22.876%), 42 pgs degraded (PG_DEGRADED) 2023-03-09T14:51:58.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:57 smithi181 ceph-mon[94189]: pgmap v231: 129 pgs: 1 stale+active+undersized, 2 stale+active+undersized+degraded, 1 down, 3 peering, 10 undersized+peered, 32 active+undersized, 6 stale+active+clean, 10 undersized+degraded+peered, 30 active+undersized+degraded, 34 active+clean; 18 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 251 op/s; 7944/34726 objects degraded (22.876%); 49 MiB/s, 12 objects/s recovering 2023-03-09T14:51:58.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:57 smithi181 ceph-mon[94189]: osdmap e148: 12 total, 8 up, 12 in 2023-03-09T14:51:58.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:57 smithi181 ceph-mon[94189]: Health check update: Degraded data redundancy: 7944/34726 objects degraded (22.876%), 42 pgs degraded (PG_DEGRADED) 2023-03-09T14:51:58.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:57 smithi038 ceph-mon[92422]: pgmap v231: 129 pgs: 1 stale+active+undersized, 2 stale+active+undersized+degraded, 1 down, 3 peering, 10 undersized+peered, 32 active+undersized, 6 stale+active+clean, 10 undersized+degraded+peered, 30 active+undersized+degraded, 34 active+clean; 18 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 251 op/s; 7944/34726 objects degraded (22.876%); 49 MiB/s, 12 objects/s recovering 2023-03-09T14:51:58.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:57 smithi038 ceph-mon[92422]: osdmap e148: 12 total, 8 up, 12 in 2023-03-09T14:51:58.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:57 smithi038 ceph-mon[92422]: Health check update: Degraded data redundancy: 7944/34726 objects degraded (22.876%), 42 pgs degraded (PG_DEGRADED) 2023-03-09T14:51:58.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 90.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T14:51:59.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:58 smithi122 ceph-mon[93869]: Health check update: Reduced data availability: 5 pgs inactive, 3 pgs down (PG_AVAILABILITY) 2023-03-09T14:51:59.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:58 smithi181 ceph-mon[94189]: Health check update: Reduced data availability: 5 pgs inactive, 3 pgs down (PG_AVAILABILITY) 2023-03-09T14:51:59.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:58 smithi038 ceph-mon[92422]: Health check update: Reduced data availability: 5 pgs inactive, 3 pgs down (PG_AVAILABILITY) 2023-03-09T14:52:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:51:59 smithi122 ceph-mon[93869]: pgmap v233: 129 pgs: 2 stale+active+undersized, 1 stale+active+undersized+degraded, 3 down, 3 peering, 12 undersized+peered, 29 active+undersized, 4 stale+active+clean, 10 undersized+degraded+peered, 31 active+undersized+degraded, 34 active+clean; 18 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 913 KiB/s rd, 1.2 MiB/s wr, 180 op/s; 6957/34774 objects degraded (20.006%) 2023-03-09T14:52:00.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:51:59 smithi181 ceph-mon[94189]: pgmap v233: 129 pgs: 2 stale+active+undersized, 1 stale+active+undersized+degraded, 3 down, 3 peering, 12 undersized+peered, 29 active+undersized, 4 stale+active+clean, 10 undersized+degraded+peered, 31 active+undersized+degraded, 34 active+clean; 18 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 913 KiB/s rd, 1.2 MiB/s wr, 180 op/s; 6957/34774 objects degraded (20.006%) 2023-03-09T14:52:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:51:59 smithi038 ceph-mon[92422]: pgmap v233: 129 pgs: 2 stale+active+undersized, 1 stale+active+undersized+degraded, 3 down, 3 peering, 12 undersized+peered, 29 active+undersized, 4 stale+active+clean, 10 undersized+degraded+peered, 31 active+undersized+degraded, 34 active+clean; 18 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 913 KiB/s rd, 1.2 MiB/s wr, 180 op/s; 6957/34774 objects degraded (20.006%) 2023-03-09T14:52:00.645 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:52:00 smithi038 systemd[1]: ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.3.service: Service RestartSec=10s expired, scheduling restart. 2023-03-09T14:52:00.645 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:52:00 smithi038 systemd[1]: ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.3.service: Scheduled restart job, restart counter is at 3. 2023-03-09T14:52:00.645 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:52:00 smithi038 systemd[1]: Stopped Ceph osd.3 for c6733fe2-be87-11ed-9af2-001a4aab830c. 2023-03-09T14:52:00.646 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:52:00 smithi038 systemd[1]: Starting Ceph osd.3 for c6733fe2-be87-11ed-9af2-001a4aab830c... 2023-03-09T14:52:01.500 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:52:01 smithi038 bash[150911]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-03-09T14:52:01.501 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:52:01 smithi038 bash[150911]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2023-03-09T14:52:01.501 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:52:01 smithi038 bash[150911]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2023-03-09T14:52:01.501 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:52:01 smithi038 bash[150911]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2023-03-09T14:52:01.501 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:52:01 smithi038 bash[150911]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-3/block 2023-03-09T14:52:01.501 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:52:01 smithi038 bash[150911]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-03-09T14:52:01.502 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:52:01 smithi038 bash[150911]: --> ceph-volume raw activate successful for osd ID: 3 2023-03-09T14:52:02.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:01 smithi122 ceph-mon[93869]: pgmap v234: 129 pgs: 3 down, 3 peering, 16 undersized+peered, 35 active+undersized, 13 undersized+degraded+peered, 33 active+undersized+degraded, 26 active+clean; 18 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 615 KiB/s rd, 1.2 MiB/s wr, 160 op/s; 8108/34894 objects degraded (23.236%) 2023-03-09T14:52:02.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:01 smithi181 ceph-mon[94189]: pgmap v234: 129 pgs: 3 down, 3 peering, 16 undersized+peered, 35 active+undersized, 13 undersized+degraded+peered, 33 active+undersized+degraded, 26 active+clean; 18 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 615 KiB/s rd, 1.2 MiB/s wr, 160 op/s; 8108/34894 objects degraded (23.236%) 2023-03-09T14:52:02.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:01 smithi038 ceph-mon[92422]: pgmap v234: 129 pgs: 3 down, 3 peering, 16 undersized+peered, 35 active+undersized, 13 undersized+degraded+peered, 33 active+undersized+degraded, 26 active+clean; 18 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 615 KiB/s rd, 1.2 MiB/s wr, 160 op/s; 8108/34894 objects degraded (23.236%) 2023-03-09T14:52:02.251 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:52:01 smithi038 podman[151175]: 2023-03-09T14:52:02.251 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:52:02 smithi038 ceph-osd[151215]: -- 172.21.15.38:0/1508021221 <== mon.0 v2:172.21.15.38:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5605758449c0 con 0x56057580d400 2023-03-09T14:52:02.252 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:52:02 smithi038 bash[151175]: bd6f3e34b546be2e6832ebc4a4adefe04f8cbfe81083c508a53fa27a1ba7d15a 2023-03-09T14:52:02.252 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:52:02 smithi038 systemd[1]: Started Ceph osd.3 for c6733fe2-be87-11ed-9af2-001a4aab830c. 2023-03-09T14:52:02.918 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:52:02 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[151191]: 2023-03-09T14:52:02.548+0000 7f8fda660640 -1 Falling back to public interface 2023-03-09T14:52:03.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:02 smithi122 ceph-mon[93869]: Health check update: Degraded data redundancy: 8108/34894 objects degraded (23.236%), 46 pgs degraded (PG_DEGRADED) 2023-03-09T14:52:03.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:02 smithi181 ceph-mon[94189]: Health check update: Degraded data redundancy: 8108/34894 objects degraded (23.236%), 46 pgs degraded (PG_DEGRADED) 2023-03-09T14:52:03.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:02 smithi038 ceph-mon[92422]: Health check update: Degraded data redundancy: 8108/34894 objects degraded (23.236%), 46 pgs degraded (PG_DEGRADED) 2023-03-09T14:52:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 95.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T14:52:04.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:03 smithi122 ceph-mon[93869]: pgmap v235: 129 pgs: 3 down, 16 undersized+peered, 36 active+undersized, 15 undersized+degraded+peered, 33 active+undersized+degraded, 26 active+clean; 18 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 1.2 MiB/s wr, 128 op/s; 8129/34930 objects degraded (23.272%) 2023-03-09T14:52:04.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:03 smithi181 ceph-mon[94189]: pgmap v235: 129 pgs: 3 down, 16 undersized+peered, 36 active+undersized, 15 undersized+degraded+peered, 33 active+undersized+degraded, 26 active+clean; 18 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 1.2 MiB/s wr, 128 op/s; 8129/34930 objects degraded (23.272%) 2023-03-09T14:52:04.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:03 smithi038 ceph-mon[92422]: pgmap v235: 129 pgs: 3 down, 16 undersized+peered, 36 active+undersized, 15 undersized+degraded+peered, 33 active+undersized+degraded, 26 active+clean; 18 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 1.2 MiB/s wr, 128 op/s; 8129/34930 objects degraded (23.272%) 2023-03-09T14:52:04.750 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:04 smithi038 systemd[1]: ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.0.service: Service RestartSec=10s expired, scheduling restart. 2023-03-09T14:52:04.751 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:04 smithi038 systemd[1]: ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.0.service: Scheduled restart job, restart counter is at 1. 2023-03-09T14:52:04.751 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:04 smithi038 systemd[1]: Stopped Ceph osd.0 for c6733fe2-be87-11ed-9af2-001a4aab830c. 2023-03-09T14:52:04.751 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:04 smithi038 systemd[1]: Starting Ceph osd.0 for c6733fe2-be87-11ed-9af2-001a4aab830c... 2023-03-09T14:52:06.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:05 smithi122 ceph-mon[93869]: pgmap v236: 129 pgs: 3 down, 16 undersized+peered, 36 active+undersized, 15 undersized+degraded+peered, 33 active+undersized+degraded, 26 active+clean; 18 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 391 KiB/s wr, 52 op/s; 8129/34946 objects degraded (23.262%) 2023-03-09T14:52:06.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:05 smithi181 ceph-mon[94189]: pgmap v236: 129 pgs: 3 down, 16 undersized+peered, 36 active+undersized, 15 undersized+degraded+peered, 33 active+undersized+degraded, 26 active+clean; 18 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 391 KiB/s wr, 52 op/s; 8129/34946 objects degraded (23.262%) 2023-03-09T14:52:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:05 smithi038 ceph-mon[92422]: pgmap v236: 129 pgs: 3 down, 16 undersized+peered, 36 active+undersized, 15 undersized+degraded+peered, 33 active+undersized+degraded, 26 active+clean; 18 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 391 KiB/s wr, 52 op/s; 8129/34946 objects degraded (23.262%) 2023-03-09T14:52:07.136 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:06 smithi181 systemd[1]: ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.8.service: Service RestartSec=10s expired, scheduling restart. 2023-03-09T14:52:07.136 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:06 smithi181 systemd[1]: ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.8.service: Scheduled restart job, restart counter is at 2. 2023-03-09T14:52:07.136 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:06 smithi181 systemd[1]: Stopped Ceph osd.8 for c6733fe2-be87-11ed-9af2-001a4aab830c. 2023-03-09T14:52:07.136 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:06 smithi181 systemd[1]: Starting Ceph osd.8 for c6733fe2-be87-11ed-9af2-001a4aab830c... 2023-03-09T14:52:07.431 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:06 smithi122 systemd[1]: ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.1.service: Service RestartSec=10s expired, scheduling restart. 2023-03-09T14:52:07.431 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:06 smithi122 systemd[1]: ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.1.service: Scheduled restart job, restart counter is at 3. 2023-03-09T14:52:07.432 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:06 smithi122 systemd[1]: Stopped Ceph osd.1 for c6733fe2-be87-11ed-9af2-001a4aab830c. 2023-03-09T14:52:07.432 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:06 smithi122 systemd[1]: Starting Ceph osd.1 for c6733fe2-be87-11ed-9af2-001a4aab830c... 2023-03-09T14:52:07.500 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:07 smithi038 bash[151313]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-03-09T14:52:07.501 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:07 smithi038 bash[151313]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2023-03-09T14:52:07.501 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:07 smithi038 bash[151313]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2023-03-09T14:52:07.501 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:07 smithi038 bash[151313]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2023-03-09T14:52:07.501 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:07 smithi038 bash[151313]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2023-03-09T14:52:07.501 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:07 smithi038 bash[151313]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-03-09T14:52:07.501 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:07 smithi038 bash[151313]: --> ceph-volume raw activate successful for osd ID: 0 2023-03-09T14:52:07.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:07 smithi181 ceph-mon[94189]: pgmap v237: 129 pgs: 3 down, 16 undersized+peered, 36 active+undersized, 15 undersized+degraded+peered, 33 active+undersized+degraded, 26 active+clean; 18 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 1.2 MiB/s wr, 156 op/s; 8131/34962 objects degraded (23.257%) 2023-03-09T14:52:07.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:07 smithi181 ceph-mon[94189]: Health check update: Reduced data availability: 7 pgs inactive, 3 pgs down (PG_AVAILABILITY) 2023-03-09T14:52:07.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:07 smithi181 ceph-mon[94189]: Health check update: Degraded data redundancy: 8131/34962 objects degraded (23.257%), 48 pgs degraded (PG_DEGRADED) 2023-03-09T14:52:07.982 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:07 smithi181 bash[127264]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-8 2023-03-09T14:52:07.982 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:07 smithi181 bash[127264]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-8 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2023-03-09T14:52:07.982 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:07 smithi181 bash[127264]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2023-03-09T14:52:07.983 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:07 smithi181 bash[127264]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2023-03-09T14:52:07.983 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:07 smithi181 bash[127264]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-8/block 2023-03-09T14:52:07.983 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:07 smithi181 bash[127264]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-8 2023-03-09T14:52:07.983 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:07 smithi181 bash[127264]: --> ceph-volume raw activate successful for osd ID: 8 2023-03-09T14:52:08.228 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:08 smithi122 bash[127394]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-03-09T14:52:08.228 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:08 smithi122 bash[127394]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2023-03-09T14:52:08.228 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:08 smithi122 bash[127394]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2023-03-09T14:52:08.229 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:08 smithi122 bash[127394]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2023-03-09T14:52:08.229 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:08 smithi122 bash[127394]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-1/block 2023-03-09T14:52:08.229 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:08 smithi122 bash[127394]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-03-09T14:52:08.229 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:08 smithi122 bash[127394]: --> ceph-volume raw activate successful for osd ID: 1 2023-03-09T14:52:08.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:07 smithi122 ceph-mon[93869]: pgmap v237: 129 pgs: 3 down, 16 undersized+peered, 36 active+undersized, 15 undersized+degraded+peered, 33 active+undersized+degraded, 26 active+clean; 18 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 1.2 MiB/s wr, 156 op/s; 8131/34962 objects degraded (23.257%) 2023-03-09T14:52:08.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:07 smithi122 ceph-mon[93869]: Health check update: Reduced data availability: 7 pgs inactive, 3 pgs down (PG_AVAILABILITY) 2023-03-09T14:52:08.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:07 smithi122 ceph-mon[93869]: Health check update: Degraded data redundancy: 8131/34962 objects degraded (23.257%), 48 pgs degraded (PG_DEGRADED) 2023-03-09T14:52:08.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:07 smithi038 ceph-mon[92422]: pgmap v237: 129 pgs: 3 down, 16 undersized+peered, 36 active+undersized, 15 undersized+degraded+peered, 33 active+undersized+degraded, 26 active+clean; 18 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 1.2 MiB/s wr, 156 op/s; 8131/34962 objects degraded (23.257%) 2023-03-09T14:52:08.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:07 smithi038 ceph-mon[92422]: Health check update: Reduced data availability: 7 pgs inactive, 3 pgs down (PG_AVAILABILITY) 2023-03-09T14:52:08.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:07 smithi038 ceph-mon[92422]: Health check update: Degraded data redundancy: 8131/34962 objects degraded (23.257%), 48 pgs degraded (PG_DEGRADED) 2023-03-09T14:52:08.251 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:07 smithi038 podman[151994]: 2023-03-09T14:52:08.252 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:07 smithi038 bash[151994]: 09d4202b03e611cb375ca81ace00ab2d220dcf4c4c89b50bf8a59f7f56afaa9e 2023-03-09T14:52:08.252 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:07 smithi038 ceph-osd[152218]: -- 172.21.15.38:0/977380448 <== mon.0 v2:172.21.15.38:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x563c24b1c9c0 con 0x563c24ae4c00 2023-03-09T14:52:08.252 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:07 smithi038 systemd[1]: Started Ceph osd.0 for c6733fe2-be87-11ed-9af2-001a4aab830c. 2023-03-09T14:52:08.721 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:08 smithi181 podman[127519]: 2023-03-09T14:52:08.721 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:08 smithi181 ceph-osd[127567]: -- 172.21.15.181:0/1844522297 <== mon.1 v2:172.21.15.181:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a0b84b49c0 con 0x55a0b847cc00 2023-03-09T14:52:08.721 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:08 smithi181 bash[127519]: 53b4338924c82fe7b8ecebf9c1ac37c269fe7335573f4ea9578d2c506132739f 2023-03-09T14:52:08.750 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:08 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 2023-03-09T14:52:08.486+0000 7f4ead873640 -1 Falling back to public interface 2023-03-09T14:52:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 100.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T14:52:08.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:08 smithi181 ceph-mon[94189]: from='osd.3 [v2:172.21.15.38:6802/4278169458,v1:172.21.15.38:6803/4278169458]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-03-09T14:52:08.982 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:08 smithi181 systemd[1]: Started Ceph osd.8 for c6733fe2-be87-11ed-9af2-001a4aab830c. 2023-03-09T14:52:09.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:08 smithi122 ceph-mon[93869]: from='osd.3 [v2:172.21.15.38:6802/4278169458,v1:172.21.15.38:6803/4278169458]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-03-09T14:52:09.228 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:08 smithi122 podman[127642]: 2023-03-09T14:52:09.228 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:09 smithi122 bash[127642]: 4ff49dd35b31b28e01ed9b15df91a738831cb5e2592f2ed4b8c5337007f5b240 2023-03-09T14:52:09.228 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:09 smithi122 systemd[1]: Started Ceph osd.1 for c6733fe2-be87-11ed-9af2-001a4aab830c. 2023-03-09T14:52:09.228 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:09 smithi122 ceph-osd[127682]: -- 172.21.15.122:0/3720277229 <== mon.2 v2:172.21.15.122:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55fd30de89c0 con 0x55fd30db1000 2023-03-09T14:52:09.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:08 smithi038 ceph-mon[92422]: from='osd.3 [v2:172.21.15.38:6802/4278169458,v1:172.21.15.38:6803/4278169458]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-03-09T14:52:09.251 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:52:08 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[151191]: 2023-03-09T14:52:08.796+0000 7f8fda660640 -1 osd.3 142 log_to_monitors true 2023-03-09T14:52:09.251 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 09 14:52:08 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-3[151191]: 2023-03-09T14:52:08.968+0000 7f8fc83d6700 -1 osd.3 142 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-09T14:52:09.481 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:09 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 2023-03-09T14:52:09.195+0000 7f627c68e640 -1 Falling back to public interface 2023-03-09T14:52:09.975 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:09 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 2023-03-09T14:52:09.575+0000 7fa3dc4f8640 -1 Falling back to public interface 2023-03-09T14:52:10.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:09 smithi122 ceph-mon[93869]: pgmap v238: 129 pgs: 3 down, 16 undersized+peered, 36 active+undersized, 15 undersized+degraded+peered, 33 active+undersized+degraded, 26 active+clean; 18 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 1.2 MiB/s wr, 160 op/s; 8131/34962 objects degraded (23.257%) 2023-03-09T14:52:10.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:09 smithi122 ceph-mon[93869]: from='osd.3 [v2:172.21.15.38:6802/4278169458,v1:172.21.15.38:6803/4278169458]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-03-09T14:52:10.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:09 smithi122 ceph-mon[93869]: osdmap e149: 12 total, 8 up, 12 in 2023-03-09T14:52:10.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:09 smithi122 ceph-mon[93869]: from='osd.3 [v2:172.21.15.38:6802/4278169458,v1:172.21.15.38:6803/4278169458]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2023-03-09T14:52:10.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:09 smithi181 ceph-mon[94189]: pgmap v238: 129 pgs: 3 down, 16 undersized+peered, 36 active+undersized, 15 undersized+degraded+peered, 33 active+undersized+degraded, 26 active+clean; 18 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 1.2 MiB/s wr, 160 op/s; 8131/34962 objects degraded (23.257%) 2023-03-09T14:52:10.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:09 smithi181 ceph-mon[94189]: from='osd.3 [v2:172.21.15.38:6802/4278169458,v1:172.21.15.38:6803/4278169458]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-03-09T14:52:10.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:09 smithi181 ceph-mon[94189]: osdmap e149: 12 total, 8 up, 12 in 2023-03-09T14:52:10.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:09 smithi181 ceph-mon[94189]: from='osd.3 [v2:172.21.15.38:6802/4278169458,v1:172.21.15.38:6803/4278169458]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2023-03-09T14:52:10.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:09 smithi038 ceph-mon[92422]: pgmap v238: 129 pgs: 3 down, 16 undersized+peered, 36 active+undersized, 15 undersized+degraded+peered, 33 active+undersized+degraded, 26 active+clean; 18 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 1.2 MiB/s wr, 160 op/s; 8131/34962 objects degraded (23.257%) 2023-03-09T14:52:10.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:09 smithi038 ceph-mon[92422]: from='osd.3 [v2:172.21.15.38:6802/4278169458,v1:172.21.15.38:6803/4278169458]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-03-09T14:52:10.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:09 smithi038 ceph-mon[92422]: osdmap e149: 12 total, 8 up, 12 in 2023-03-09T14:52:10.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:09 smithi038 ceph-mon[92422]: from='osd.3 [v2:172.21.15.38:6802/4278169458,v1:172.21.15.38:6803/4278169458]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2023-03-09T14:52:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:10 smithi122 ceph-mon[93869]: Health check update: 3 osds down (OSD_DOWN) 2023-03-09T14:52:11.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:10 smithi122 ceph-mon[93869]: osd.3 [v2:172.21.15.38:6802/4278169458,v1:172.21.15.38:6803/4278169458] boot 2023-03-09T14:52:11.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:10 smithi122 ceph-mon[93869]: osdmap e150: 12 total, 9 up, 12 in 2023-03-09T14:52:11.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-03-09T14:52:11.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:10 smithi122 ceph-mon[93869]: osdmap e151: 12 total, 9 up, 12 in 2023-03-09T14:52:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:10 smithi181 ceph-mon[94189]: Health check update: 3 osds down (OSD_DOWN) 2023-03-09T14:52:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:10 smithi181 ceph-mon[94189]: osd.3 [v2:172.21.15.38:6802/4278169458,v1:172.21.15.38:6803/4278169458] boot 2023-03-09T14:52:11.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:10 smithi181 ceph-mon[94189]: osdmap e150: 12 total, 9 up, 12 in 2023-03-09T14:52:11.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-03-09T14:52:11.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:10 smithi181 ceph-mon[94189]: osdmap e151: 12 total, 9 up, 12 in 2023-03-09T14:52:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:10 smithi038 ceph-mon[92422]: Health check update: 3 osds down (OSD_DOWN) 2023-03-09T14:52:11.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:10 smithi038 ceph-mon[92422]: osd.3 [v2:172.21.15.38:6802/4278169458,v1:172.21.15.38:6803/4278169458] boot 2023-03-09T14:52:11.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:10 smithi038 ceph-mon[92422]: osdmap e150: 12 total, 9 up, 12 in 2023-03-09T14:52:11.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-03-09T14:52:11.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:10 smithi038 ceph-mon[92422]: osdmap e151: 12 total, 9 up, 12 in 2023-03-09T14:52:12.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:11 smithi038 ceph-mon[92422]: pgmap v241: 129 pgs: 11 peering, 3 down, 15 undersized+peered, 33 active+undersized, 14 undersized+degraded+peered, 27 active+undersized+degraded, 26 active+clean; 18 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.9 MiB/s wr, 241 op/s; 6543/35190 objects degraded (18.593%) 2023-03-09T14:52:12.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:11 smithi038 ceph-mon[92422]: osdmap e152: 12 total, 9 up, 12 in 2023-03-09T14:52:12.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:11 smithi122 ceph-mon[93869]: pgmap v241: 129 pgs: 11 peering, 3 down, 15 undersized+peered, 33 active+undersized, 14 undersized+degraded+peered, 27 active+undersized+degraded, 26 active+clean; 18 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.9 MiB/s wr, 241 op/s; 6543/35190 objects degraded (18.593%) 2023-03-09T14:52:12.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:11 smithi122 ceph-mon[93869]: osdmap e152: 12 total, 9 up, 12 in 2023-03-09T14:52:12.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:11 smithi181 ceph-mon[94189]: pgmap v241: 129 pgs: 11 peering, 3 down, 15 undersized+peered, 33 active+undersized, 14 undersized+degraded+peered, 27 active+undersized+degraded, 26 active+clean; 18 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.9 MiB/s wr, 241 op/s; 6543/35190 objects degraded (18.593%) 2023-03-09T14:52:12.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:11 smithi181 ceph-mon[94189]: osdmap e152: 12 total, 9 up, 12 in 2023-03-09T14:52:13.038 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-03-09T14:52:13.204 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.202+0000 7f288be14700 1 -- 172.21.15.38:0/2992423488 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f28841051c0 msgr2=0x7f288410cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:52:13.204 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.202+0000 7f288be14700 1 --2- 172.21.15.38:0/2992423488 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f28841051c0 0x7f288410cd90 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f287c00b0d0 tx=0x7f287c00b3e0 comp rx=0 tx=0).stop 2023-03-09T14:52:13.205 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.202+0000 7f288be14700 1 -- 172.21.15.38:0/2992423488 shutdown_connections 2023-03-09T14:52:13.205 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.202+0000 7f288be14700 1 --2- 172.21.15.38:0/2992423488 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f28841051c0 0x7f288410cd90 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:52:13.205 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.202+0000 7f288be14700 1 --2- 172.21.15.38:0/2992423488 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f2884100180 0x7f2884100550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:52:13.205 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.202+0000 7f288be14700 1 --2- 172.21.15.38:0/2992423488 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2884100b20 0x7f2884104ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:52:13.205 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.202+0000 7f288be14700 1 -- 172.21.15.38:0/2992423488 >> 172.21.15.38:0/2992423488 conn(0x7f28840fb9d0 msgr2=0x7f28840fdde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:52:13.207 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.202+0000 7f288be14700 1 -- 172.21.15.38:0/2992423488 shutdown_connections 2023-03-09T14:52:13.208 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.203+0000 7f288be14700 1 -- 172.21.15.38:0/2992423488 wait complete. 2023-03-09T14:52:13.208 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.203+0000 7f288be14700 1 Processor -- start 2023-03-09T14:52:13.208 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.203+0000 7f288be14700 1 -- start start 2023-03-09T14:52:13.208 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.204+0000 7f288be14700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2884100180 0x7f28841a25b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:52:13.208 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.204+0000 7f288be14700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2884100b20 0x7f28841a2af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:52:13.208 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.204+0000 7f288be14700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f28841051c0 0x7f288410bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:52:13.209 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.204+0000 7f288be14700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f288410c2b0 con 0x7f2884100180 2023-03-09T14:52:13.209 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.204+0000 7f288be14700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f288410c420 con 0x7f2884100b20 2023-03-09T14:52:13.209 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.204+0000 7f288be14700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f288410c5e0 con 0x7f28841051c0 2023-03-09T14:52:13.209 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.204+0000 7f2889bb0700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2884100180 0x7f28841a25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:52:13.209 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.204+0000 7f2889bb0700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2884100180 0x7f28841a25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:40936/0 (socket says 172.21.15.38:40936) 2023-03-09T14:52:13.209 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.204+0000 7f2889bb0700 1 -- 172.21.15.38:0/1512988120 learned_addr learned my addr 172.21.15.38:0/1512988120 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:52:13.209 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.204+0000 7f288a3b1700 1 --2- 172.21.15.38:0/1512988120 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f28841051c0 0x7f288410bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:52:13.209 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.204+0000 7f28893af700 1 --2- 172.21.15.38:0/1512988120 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2884100b20 0x7f28841a2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:52:13.210 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.204+0000 7f2889bb0700 1 -- 172.21.15.38:0/1512988120 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f28841051c0 msgr2=0x7f288410bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:52:13.210 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.204+0000 7f2889bb0700 1 --2- 172.21.15.38:0/1512988120 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f28841051c0 0x7f288410bce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:52:13.210 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.204+0000 7f2889bb0700 1 -- 172.21.15.38:0/1512988120 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2884100b20 msgr2=0x7f28841a2af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:52:13.210 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.204+0000 7f2889bb0700 1 --2- 172.21.15.38:0/1512988120 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2884100b20 0x7f28841a2af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:52:13.210 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.204+0000 7f2889bb0700 1 -- 172.21.15.38:0/1512988120 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f287c009d00 con 0x7f2884100180 2023-03-09T14:52:13.210 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.204+0000 7f28893af700 1 --2- 172.21.15.38:0/1512988120 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2884100b20 0x7f28841a2af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-09T14:52:13.210 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.204+0000 7f2889bb0700 1 --2- 172.21.15.38:0/1512988120 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2884100180 0x7f28841a25b0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f288000e9c0 tx=0x7f288000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:52:13.211 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.205+0000 7f287affd700 1 -- 172.21.15.38:0/1512988120 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28800097b0 con 0x7f2884100180 2023-03-09T14:52:13.211 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.205+0000 7f287affd700 1 -- 172.21.15.38:0/1512988120 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2880004cf0 con 0x7f2884100180 2023-03-09T14:52:13.211 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.205+0000 7f288be14700 1 -- 172.21.15.38:0/1512988120 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f288410ffb0 con 0x7f2884100180 2023-03-09T14:52:13.211 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.205+0000 7f287affd700 1 -- 172.21.15.38:0/1512988120 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2880005600 con 0x7f2884100180 2023-03-09T14:52:13.211 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.205+0000 7f288be14700 1 -- 172.21.15.38:0/1512988120 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f288410fe20 con 0x7f2884100180 2023-03-09T14:52:13.213 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.207+0000 7f287affd700 1 -- 172.21.15.38:0/1512988120 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 25) v1 ==== 133295+0+0 (secure 0 0 0) 0x7f288001e020 con 0x7f2884100180 2023-03-09T14:52:13.213 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.208+0000 7f287affd700 1 --2- 172.21.15.38:0/1512988120 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f287009f4a0 0x7f28700a1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:52:13.214 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.208+0000 7f28893af700 1 --2- 172.21.15.38:0/1512988120 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f287009f4a0 0x7f28700a1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:52:13.214 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.208+0000 7f287affd700 1 -- 172.21.15.38:0/1512988120 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 9526+0+0 (secure 0 0 0) 0x7f2880014070 con 0x7f2884100180 2023-03-09T14:52:13.214 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.209+0000 7f28893af700 1 --2- 172.21.15.38:0/1512988120 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f287009f4a0 0x7f28700a1950 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f288410d150 tx=0x7f2874009550 comp rx=0 tx=0).ready entity=mgr.15558 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:52:13.214 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.209+0000 7f2897f9db80 1 Processor -- start 2023-03-09T14:52:13.214 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.210+0000 7f2897f9db80 1 -- start start 2023-03-09T14:52:13.214 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.210+0000 7f2897f9db80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55be56298830 0x55be56298c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:52:13.214 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.210+0000 7f2897f9db80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55be56299140 0x55be56292aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:52:13.214 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.210+0000 7f2897f9db80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55be56292fe0 0x55be56293410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:52:13.215 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.210+0000 7f2897f9db80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be5617cda0 con 0x55be56292fe0 2023-03-09T14:52:13.215 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.210+0000 7f2897f9db80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be561eb780 con 0x55be56298830 2023-03-09T14:52:13.215 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.210+0000 7f2897f9db80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be561e4cb0 con 0x55be56299140 2023-03-09T14:52:13.215 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.210+0000 7f28893af700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55be56299140 0x55be56292aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:52:13.215 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.210+0000 7f28893af700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55be56299140 0x55be56292aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.38:48644/0 (socket says 172.21.15.38:48644) 2023-03-09T14:52:13.215 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.210+0000 7f28893af700 1 -- 172.21.15.38:0/3909816537 learned_addr learned my addr 172.21.15.38:0/3909816537 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:52:13.215 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.210+0000 7f2889bb0700 1 --2- 172.21.15.38:0/3909816537 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55be56298830 0x55be56298c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:52:13.216 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.210+0000 7f288a3b1700 1 --2- 172.21.15.38:0/3909816537 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55be56292fe0 0x55be56293410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:52:13.216 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.210+0000 7f28893af700 1 -- 172.21.15.38:0/3909816537 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55be56298830 msgr2=0x55be56298c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:52:13.216 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.210+0000 7f28893af700 1 --2- 172.21.15.38:0/3909816537 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55be56298830 0x55be56298c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:52:13.216 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.210+0000 7f28893af700 1 -- 172.21.15.38:0/3909816537 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55be56292fe0 msgr2=0x55be56293410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:52:13.216 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.210+0000 7f28893af700 1 --2- 172.21.15.38:0/3909816537 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55be56292fe0 0x55be56293410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:52:13.216 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.210+0000 7f28893af700 1 -- 172.21.15.38:0/3909816537 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55be561ab190 con 0x55be56299140 2023-03-09T14:52:13.216 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.210+0000 7f288a3b1700 1 --2- 172.21.15.38:0/3909816537 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55be56292fe0 0x55be56293410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:52:13.216 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.211+0000 7f28893af700 1 --2- 172.21.15.38:0/3909816537 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55be56299140 0x55be56292aa0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f287400ed00 tx=0x7f2874009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=63fb8e6ea09b663e server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:52:13.217 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.211+0000 7f286ffff700 1 -- 172.21.15.38:0/3909816537 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2874011b80 con 0x55be56299140 2023-03-09T14:52:13.217 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.211+0000 7f286ffff700 1 -- 172.21.15.38:0/3909816537 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f2874011ce0 con 0x55be56299140 2023-03-09T14:52:13.217 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.211+0000 7f286ffff700 1 -- 172.21.15.38:0/3909816537 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2874010610 con 0x55be56299140 2023-03-09T14:52:13.220 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.212+0000 7f2897f9db80 1 -- 172.21.15.38:0/3909816537 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55be56299140 msgr2=0x55be56292aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:52:13.220 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.212+0000 7f2897f9db80 1 --2- 172.21.15.38:0/3909816537 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55be56299140 0x55be56292aa0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f287400ed00 tx=0x7f2874009f90 comp rx=0 tx=0).stop 2023-03-09T14:52:13.220 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.212+0000 7f2897f9db80 1 -- 172.21.15.38:0/3909816537 shutdown_connections 2023-03-09T14:52:13.221 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.212+0000 7f2897f9db80 1 --2- 172.21.15.38:0/3909816537 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55be56292fe0 0x55be56293410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:52:13.221 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.212+0000 7f2897f9db80 1 --2- 172.21.15.38:0/3909816537 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55be56298830 0x55be56298c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:52:13.221 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.212+0000 7f2897f9db80 1 --2- 172.21.15.38:0/3909816537 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55be56299140 0x55be56292aa0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:52:13.221 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.212+0000 7f2897f9db80 1 -- 172.21.15.38:0/3909816537 >> 172.21.15.38:0/3909816537 conn(0x55be5628da20 msgr2=0x55be5628ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:52:13.221 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.212+0000 7f2897f9db80 1 -- 172.21.15.38:0/3909816537 shutdown_connections 2023-03-09T14:52:13.221 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.212+0000 7f2897f9db80 1 -- 172.21.15.38:0/3909816537 wait complete. 2023-03-09T14:52:13.222 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.212+0000 7f2897f9db80 1 -- start start 2023-03-09T14:52:13.222 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.212+0000 7f2897f9db80 1 Processor -- start 2023-03-09T14:52:13.222 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.213+0000 7f2897f9db80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55be56292fe0 0x55be5629f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:52:13.222 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.213+0000 7f2897f9db80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55be56298830 0x55be5629c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:52:13.222 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.213+0000 7f2897f9db80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55be56299140 0x55be5629c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:52:13.222 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.213+0000 7f2897f9db80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be56068920 con 0x55be56299140 2023-03-09T14:52:13.222 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.213+0000 7f2897f9db80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be5629d140 con 0x55be56292fe0 2023-03-09T14:52:13.223 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.213+0000 7f2897f9db80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be562a8c40 con 0x55be56298830 2023-03-09T14:52:13.223 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.213+0000 7f2897f9db80 10 client.0 _ll_register_callbacks cb 0x55be5618a520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:52:13.223 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.213+0000 7f2889bb0700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55be56292fe0 0x55be5629f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:52:13.223 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.213+0000 7f2889bb0700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55be56292fe0 0x55be5629f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.38:50140/0 (socket says 172.21.15.38:50140) 2023-03-09T14:52:13.223 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.213+0000 7f2889bb0700 1 -- 172.21.15.38:0/2401985101 learned_addr learned my addr 172.21.15.38:0/2401985101 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:52:13.223 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.213+0000 7f288a3b1700 1 --2- 172.21.15.38:0/2401985101 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55be56299140 0x55be5629c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:52:13.223 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.213+0000 7f28893af700 1 --2- 172.21.15.38:0/2401985101 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55be56298830 0x55be5629c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:52:13.224 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.213+0000 7f2889bb0700 1 -- 172.21.15.38:0/2401985101 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55be56298830 msgr2=0x55be5629c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:52:13.224 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.213+0000 7f2889bb0700 1 --2- 172.21.15.38:0/2401985101 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55be56298830 0x55be5629c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:52:13.224 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.213+0000 7f2889bb0700 1 -- 172.21.15.38:0/2401985101 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55be56299140 msgr2=0x55be5629c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:52:13.224 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.213+0000 7f2889bb0700 1 --2- 172.21.15.38:0/2401985101 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55be56299140 0x55be5629c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:52:13.224 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.213+0000 7f2889bb0700 1 -- 172.21.15.38:0/2401985101 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f287400e9b0 con 0x55be56292fe0 2023-03-09T14:52:13.224 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.214+0000 7f2889bb0700 1 --2- 172.21.15.38:0/2401985101 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55be56292fe0 0x55be5629f0f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f28800c7ef0 tx=0x7f28800981e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=919f74f730f49550 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:52:13.224 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.214+0000 7f286d7fa700 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2023-03-09T14:52:13.224 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.214+0000 7f286d7fa700 1 -- 172.21.15.38:0/2401985101 <== mon.2 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2880098440 con 0x55be56292fe0 2023-03-09T14:52:13.225 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.214+0000 7f286d7fa700 1 -- 172.21.15.38:0/2401985101 <== mon.2 v2:172.21.15.122:3300/0 2 ==== osd_map(152..152 src has 1..152) v4 ==== 9526+0+0 (secure 0 0 0) 0x7f28800985a0 con 0x55be56292fe0 2023-03-09T14:52:13.225 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.214+0000 7f2897f9db80 1 -- 172.21.15.38:0/2401985101 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55be561b2190 con 0x55be56292fe0 2023-03-09T14:52:13.225 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.214+0000 7f286d7fa700 1 -- 172.21.15.38:0/2401985101 <== mon.2 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f28800976d0 con 0x55be56292fe0 2023-03-09T14:52:13.225 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.215+0000 7f2897f9db80 10 client.25736 fetch_fsmap learned FSMap version 31 2023-03-09T14:52:13.225 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.215+0000 7f2897f9db80 1 -- 172.21.15.38:0/2401985101 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55be55ffbc60 con 0x55be56292fe0 2023-03-09T14:52:13.225 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.215+0000 7f286d7fa700 1 -- 172.21.15.38:0/2401985101 <== mon.2 v2:172.21.15.122:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55be561b2190 con 0x55be56292fe0 2023-03-09T14:52:13.225 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.215+0000 7f286d7fa700 1 -- 172.21.15.38:0/2401985101 <== mon.2 v2:172.21.15.122:3300/0 5 ==== fsmap(e 31) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f28800b0250 con 0x55be56292fe0 2023-03-09T14:52:13.226 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.215+0000 7f2897f9db80 10 client.25736 fetch_fsmap finished waiting for FSMap version 31 2023-03-09T14:52:13.226 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.215+0000 7f2897f9db80 10 client.25736 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T14:52:13.226 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.215+0000 7f2897f9db80 20 client.25736 populate_metadata read hostname 'smithi038' 2023-03-09T14:52:13.226 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.215+0000 7f2897f9db80 1 --2- 172.21.15.38:0/2401985101 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55be562acc30 0x55be562af040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:52:13.226 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.215+0000 7f2897f9db80 4 client.25736 mds_command: new command op to 24530 tid=0[{"prefix": "get_command_descriptions"}] 2023-03-09T14:52:13.226 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.215+0000 7f2897f9db80 1 -- 172.21.15.38:0/2401985101 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55be56068f60 con 0x55be562acc30 2023-03-09T14:52:13.226 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.215+0000 7f28893af700 1 --2- 172.21.15.38:0/2401985101 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55be562acc30 0x55be562af040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:52:13.226 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.216+0000 7f28893af700 1 --2- 172.21.15.38:0/2401985101 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55be562acc30 0x55be562af040 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5c5de41d3ee06e5f server_cookie=b8395a8f99ee6623 in_seq=0 out_seq=0 2023-03-09T14:52:13.227 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.216+0000 7f286d7fa700 10 client.25736 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:52:13.227 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.217+0000 7f286d7fa700 1 -- 172.21.15.38:0/2401985101 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18278 (crc 0 0 0) 0x55be56068f60 con 0x55be562acc30 2023-03-09T14:52:13.227 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.217+0000 7f286d7fa700 10 client.25736 handle_command_reply: tid=0 2023-03-09T14:52:13.227 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.217+0000 7f2897f9db80 1 client.25736 shutdown 2023-03-09T14:52:13.227 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.217+0000 7f2897f9db80 1 -- 172.21.15.38:0/2401985101 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55be56292fe0 msgr2=0x55be5629f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:52:13.227 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.217+0000 7f2897f9db80 1 --2- 172.21.15.38:0/2401985101 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55be56292fe0 0x55be5629f0f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f28800c7ef0 tx=0x7f28800981e0 comp rx=0 tx=0).stop 2023-03-09T14:52:13.227 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.217+0000 7f2897f9db80 1 -- 172.21.15.38:0/2401985101 shutdown_connections 2023-03-09T14:52:13.227 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.217+0000 7f2897f9db80 1 --2- 172.21.15.38:0/2401985101 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55be562acc30 0x55be562af040 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:52:13.228 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.217+0000 7f2897f9db80 1 --2- 172.21.15.38:0/2401985101 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55be56299140 0x55be5629c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:52:13.228 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.217+0000 7f2897f9db80 1 --2- 172.21.15.38:0/2401985101 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55be56292fe0 0x55be5629f0f0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:52:13.228 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.217+0000 7f286d7fa700 0 client.25736 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:52:13.228 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.217+0000 7f2897f9db80 1 --2- 172.21.15.38:0/2401985101 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55be56298830 0x55be5629c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:52:13.228 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.217+0000 7f2897f9db80 1 -- 172.21.15.38:0/2401985101 >> 172.21.15.38:0/2401985101 conn(0x55be5628da20 msgr2=0x55be5628cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:52:13.228 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.217+0000 7f2897f9db80 1 -- 172.21.15.38:0/2401985101 shutdown_connections 2023-03-09T14:52:13.228 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.218+0000 7f2897f9db80 1 -- 172.21.15.38:0/2401985101 wait complete. 2023-03-09T14:52:13.229 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.218+0000 7f2897f9db80 20 client.25736 trim_cache size 0 max 16384 2023-03-09T14:52:13.238 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.235+0000 7f2897f9db80 1 Processor -- start 2023-03-09T14:52:13.238 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.235+0000 7f2897f9db80 1 -- start start 2023-03-09T14:52:13.238 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.236+0000 7f2897f9db80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55be56292fe0 0x55be5629ec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:52:13.238 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.236+0000 7f2897f9db80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55be56298830 0x55be5628cc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:52:13.238 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.236+0000 7f2897f9db80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55be56299140 0x55be562964d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:52:13.239 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.236+0000 7f2897f9db80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be56293d00 con 0x55be56298830 2023-03-09T14:52:13.239 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.236+0000 7f2897f9db80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be56293a30 con 0x55be56292fe0 2023-03-09T14:52:13.239 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.236+0000 7f2897f9db80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be562a8a30 con 0x55be56299140 2023-03-09T14:52:13.239 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.236+0000 7f288a3b1700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55be56299140 0x55be562964d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:52:13.239 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.236+0000 7f288a3b1700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55be56299140 0x55be562964d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.38:48658/0 (socket says 172.21.15.38:48658) 2023-03-09T14:52:13.239 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.236+0000 7f288a3b1700 1 -- 172.21.15.38:0/1893991387 learned_addr learned my addr 172.21.15.38:0/1893991387 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:52:13.242 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.236+0000 7f2889bb0700 1 --2- 172.21.15.38:0/1893991387 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55be56292fe0 0x55be5629ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:52:13.242 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.236+0000 7f288a3b1700 1 -- 172.21.15.38:0/1893991387 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55be56292fe0 msgr2=0x55be5629ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:52:13.242 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.236+0000 7f288a3b1700 1 --2- 172.21.15.38:0/1893991387 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55be56292fe0 0x55be5629ec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:52:13.242 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.236+0000 7f288a3b1700 1 -- 172.21.15.38:0/1893991387 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55be56298830 msgr2=0x55be5628cc20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-03-09T14:52:13.242 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.236+0000 7f288a3b1700 1 --2- 172.21.15.38:0/1893991387 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55be56298830 0x55be5628cc20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:52:13.242 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.236+0000 7f288a3b1700 1 -- 172.21.15.38:0/1893991387 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55be56012de0 con 0x55be56299140 2023-03-09T14:52:13.243 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.236+0000 7f288a3b1700 1 --2- 172.21.15.38:0/1893991387 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55be56299140 0x55be562964d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f287c015000 tx=0x7f287c00b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2d65c9353562fa2f server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:52:13.243 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.237+0000 7f2878ff9700 1 -- 172.21.15.38:0/1893991387 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f287c00e040 con 0x55be56299140 2023-03-09T14:52:13.243 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.237+0000 7f2878ff9700 1 -- 172.21.15.38:0/1893991387 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f287c0074e0 con 0x55be56299140 2023-03-09T14:52:13.243 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.237+0000 7f2878ff9700 1 -- 172.21.15.38:0/1893991387 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f287c003e80 con 0x55be56299140 2023-03-09T14:52:13.243 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.238+0000 7f2897f9db80 1 -- 172.21.15.38:0/1893991387 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55be56299140 msgr2=0x55be562964d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:52:13.244 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.238+0000 7f2897f9db80 1 --2- 172.21.15.38:0/1893991387 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55be56299140 0x55be562964d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f287c015000 tx=0x7f287c00b040 comp rx=0 tx=0).stop 2023-03-09T14:52:13.244 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.238+0000 7f2897f9db80 1 -- 172.21.15.38:0/1893991387 shutdown_connections 2023-03-09T14:52:13.244 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.238+0000 7f2897f9db80 1 --2- 172.21.15.38:0/1893991387 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55be56299140 0x55be562964d0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:52:13.244 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.238+0000 7f2897f9db80 1 --2- 172.21.15.38:0/1893991387 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55be56292fe0 0x55be5629ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:52:13.244 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.238+0000 7f2897f9db80 1 --2- 172.21.15.38:0/1893991387 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55be56298830 0x55be5628cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:52:13.244 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.238+0000 7f2897f9db80 1 -- 172.21.15.38:0/1893991387 >> 172.21.15.38:0/1893991387 conn(0x55be5628da20 msgr2=0x55be562a8320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:52:13.244 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.238+0000 7f2897f9db80 1 -- 172.21.15.38:0/1893991387 shutdown_connections 2023-03-09T14:52:13.244 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.238+0000 7f2897f9db80 1 -- 172.21.15.38:0/1893991387 wait complete. 2023-03-09T14:52:13.245 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.238+0000 7f2897f9db80 1 -- start start 2023-03-09T14:52:13.245 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.238+0000 7f2897f9db80 1 Processor -- start 2023-03-09T14:52:13.245 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.239+0000 7f2897f9db80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55be56292fe0 0x55be562948e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:52:13.245 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.239+0000 7f2897f9db80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55be56298830 0x55be56294e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:52:13.245 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.239+0000 7f2897f9db80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55be56299140 0x55be56295360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:52:13.245 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.239+0000 7f2897f9db80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be56300ad0 con 0x55be56292fe0 2023-03-09T14:52:13.245 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.239+0000 7f2897f9db80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be56300c10 con 0x55be56299140 2023-03-09T14:52:13.246 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.239+0000 7f2897f9db80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be56300dd0 con 0x55be56298830 2023-03-09T14:52:13.246 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.239+0000 7f2897f9db80 10 client.0 _ll_register_callbacks cb 0x55be5618a520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:52:13.246 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.239+0000 7f28893af700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55be56298830 0x55be56294e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:52:13.246 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.239+0000 7f2889bb0700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55be56292fe0 0x55be562948e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:52:13.246 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.239+0000 7f28893af700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55be56298830 0x55be56294e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.38:48662/0 (socket says 172.21.15.38:48662) 2023-03-09T14:52:13.246 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.239+0000 7f28893af700 1 -- 172.21.15.38:0/1353512463 learned_addr learned my addr 172.21.15.38:0/1353512463 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:52:13.246 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.239+0000 7f288a3b1700 1 --2- 172.21.15.38:0/1353512463 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55be56299140 0x55be56295360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:52:13.246 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.239+0000 7f28893af700 1 -- 172.21.15.38:0/1353512463 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55be56299140 msgr2=0x55be56295360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:52:13.247 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.239+0000 7f28893af700 1 --2- 172.21.15.38:0/1353512463 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55be56299140 0x55be56295360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:52:13.247 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.239+0000 7f28893af700 1 -- 172.21.15.38:0/1353512463 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55be56292fe0 msgr2=0x55be562948e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:52:13.247 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.239+0000 7f28893af700 1 --2- 172.21.15.38:0/1353512463 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55be56292fe0 0x55be562948e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:52:13.247 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.239+0000 7f28893af700 1 -- 172.21.15.38:0/1353512463 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f288410d5d0 con 0x55be56298830 2023-03-09T14:52:13.247 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.239+0000 7f28893af700 1 --2- 172.21.15.38:0/1353512463 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55be56298830 0x55be56294e20 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f287400f010 tx=0x7f287400e920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1ed1c0f53e5fc8bb server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:52:13.247 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.239+0000 7f286d7fa700 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2023-03-09T14:52:13.247 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.240+0000 7f286d7fa700 1 -- 172.21.15.38:0/1353512463 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f287402f040 con 0x55be56298830 2023-03-09T14:52:13.247 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.240+0000 7f286d7fa700 1 -- 172.21.15.38:0/1353512463 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(152..152 src has 1..152) v4 ==== 9526+0+0 (secure 0 0 0) 0x7f2874011e40 con 0x55be56298830 2023-03-09T14:52:13.248 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.240+0000 7f2897f9db80 1 -- 172.21.15.38:0/1353512463 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55be561a21b0 con 0x55be56298830 2023-03-09T14:52:13.249 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.240+0000 7f286d7fa700 1 -- 172.21.15.38:0/1353512463 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f28740312c0 con 0x55be56298830 2023-03-09T14:52:13.250 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.240+0000 7f2897f9db80 10 client.25699 fetch_fsmap learned FSMap version 31 2023-03-09T14:52:13.250 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.240+0000 7f2897f9db80 1 -- 172.21.15.38:0/1353512463 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55be560112b0 con 0x55be56298830 2023-03-09T14:52:13.250 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.240+0000 7f286d7fa700 1 -- 172.21.15.38:0/1353512463 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55be561a21b0 con 0x55be56298830 2023-03-09T14:52:13.250 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.240+0000 7f286d7fa700 1 -- 172.21.15.38:0/1353512463 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 31) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f2874037020 con 0x55be56298830 2023-03-09T14:52:13.250 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.240+0000 7f2897f9db80 10 client.25699 fetch_fsmap finished waiting for FSMap version 31 2023-03-09T14:52:13.250 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.240+0000 7f2897f9db80 10 client.25699 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T14:52:13.250 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.240+0000 7f2897f9db80 20 client.25699 populate_metadata read hostname 'smithi038' 2023-03-09T14:52:13.250 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.240+0000 7f2897f9db80 1 --2- 172.21.15.38:0/1353512463 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55be563023c0 0x55be563047d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:52:13.251 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.240+0000 7f2897f9db80 4 client.25699 mds_command: new command op to 24530 tid=0[{"prefix": "scrub status"}] 2023-03-09T14:52:13.251 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.240+0000 7f2897f9db80 1 -- 172.21.15.38:0/1353512463 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55be5618c1c0 con 0x55be563023c0 2023-03-09T14:52:13.251 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.240+0000 7f2889bb0700 1 --2- 172.21.15.38:0/1353512463 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55be563023c0 0x55be563047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:52:13.251 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.241+0000 7f286d7fa700 10 client.25699 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:52:13.251 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.241+0000 7f2889bb0700 1 --2- 172.21.15.38:0/1353512463 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55be563023c0 0x55be563047d0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=543144958c57f26 server_cookie=c32e6d80ad77c157 in_seq=0 out_seq=0 2023-03-09T14:52:13.251 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.241+0000 7f286d7fa700 1 -- 172.21.15.38:0/1353512463 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+265 (crc 0 0 0) 0x55be5618c1c0 con 0x55be563023c0 2023-03-09T14:52:13.251 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.241+0000 7f286d7fa700 10 client.25699 handle_command_reply: tid=0 2023-03-09T14:52:13.252 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.241+0000 7f2897f9db80 1 client.25699 shutdown 2023-03-09T14:52:13.252 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.241+0000 7f2897f9db80 1 -- 172.21.15.38:0/1353512463 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55be56298830 msgr2=0x55be56294e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:52:13.252 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.241+0000 7f2897f9db80 1 --2- 172.21.15.38:0/1353512463 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55be56298830 0x55be56294e20 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f287400f010 tx=0x7f287400e920 comp rx=0 tx=0).stop 2023-03-09T14:52:13.252 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.242+0000 7f2897f9db80 1 -- 172.21.15.38:0/1353512463 shutdown_connections 2023-03-09T14:52:13.252 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.242+0000 7f2897f9db80 1 --2- 172.21.15.38:0/1353512463 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55be563023c0 0x55be563047d0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:52:13.252 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.242+0000 7f2897f9db80 1 --2- 172.21.15.38:0/1353512463 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55be56292fe0 0x55be562948e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:52:13.253 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.242+0000 7f2897f9db80 1 --2- 172.21.15.38:0/1353512463 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55be56299140 0x55be56295360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:52:13.253 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.242+0000 7f2897f9db80 1 --2- 172.21.15.38:0/1353512463 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55be56298830 0x55be56294e20 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:52:13.253 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.242+0000 7f286d7fa700 0 client.25699 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:52:13.253 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.242+0000 7f2897f9db80 1 -- 172.21.15.38:0/1353512463 >> 172.21.15.38:0/1353512463 conn(0x55be5628da20 msgr2=0x55be5628cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:52:13.253 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.242+0000 7f2897f9db80 1 -- 172.21.15.38:0/1353512463 shutdown_connections 2023-03-09T14:52:13.253 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.242+0000 7f2897f9db80 1 -- 172.21.15.38:0/1353512463 wait complete. 2023-03-09T14:52:13.253 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.242+0000 7f2897f9db80 20 client.25699 trim_cache size 0 max 16384 2023-03-09T14:52:13.254 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2023-03-09T14:52:13.254 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "scrub active (3 inodes in the stack)", 2023-03-09T14:52:13.254 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": { 2023-03-09T14:52:13.254 INFO:teuthology.orchestra.run.smithi038.stdout: "a14659f4-d7ea-4c5a-b15b-6138af3f909f": { 2023-03-09T14:52:13.254 INFO:teuthology.orchestra.run.smithi038.stdout: "path": "/", 2023-03-09T14:52:13.254 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "a14659f4-d7ea-4c5a-b15b-6138af3f909f", 2023-03-09T14:52:13.255 INFO:teuthology.orchestra.run.smithi038.stdout: "options": "recursive,force" 2023-03-09T14:52:13.255 INFO:teuthology.orchestra.run.smithi038.stdout: } 2023-03-09T14:52:13.255 INFO:teuthology.orchestra.run.smithi038.stdout: } 2023-03-09T14:52:13.255 INFO:teuthology.orchestra.run.smithi038.stdout:} 2023-03-09T14:52:13.256 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.242+0000 7f288be14700 1 -- 172.21.15.38:0/1512988120 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f287009f4a0 msgr2=0x7f28700a1950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:52:13.256 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.242+0000 7f288be14700 1 --2- 172.21.15.38:0/1512988120 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f287009f4a0 0x7f28700a1950 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f288410d150 tx=0x7f2874009550 comp rx=0 tx=0).stop 2023-03-09T14:52:13.256 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.242+0000 7f288be14700 1 -- 172.21.15.38:0/1512988120 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2884100180 msgr2=0x7f28841a25b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:52:13.256 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.242+0000 7f288be14700 1 --2- 172.21.15.38:0/1512988120 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2884100180 0x7f28841a25b0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f288000e9c0 tx=0x7f288000ecd0 comp rx=0 tx=0).stop 2023-03-09T14:52:13.256 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.242+0000 7f288be14700 1 -- 172.21.15.38:0/1512988120 shutdown_connections 2023-03-09T14:52:13.256 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.242+0000 7f288be14700 1 --2- 172.21.15.38:0/1512988120 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f287009f4a0 0x7f28700a1950 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:52:13.256 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.242+0000 7f288be14700 1 --2- 172.21.15.38:0/1512988120 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f2884100180 0x7f28841a25b0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:52:13.257 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.242+0000 7f288be14700 1 --2- 172.21.15.38:0/1512988120 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f28841051c0 0x7f288410bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:52:13.257 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.242+0000 7f288be14700 1 --2- 172.21.15.38:0/1512988120 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f2884100b20 0x7f28841a2af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:52:13.257 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.242+0000 7f288be14700 1 -- 172.21.15.38:0/1512988120 >> 172.21.15.38:0/1512988120 conn(0x7f28840fb9d0 msgr2=0x7f28840fc460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:52:13.257 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.242+0000 7f288be14700 1 -- 172.21.15.38:0/1512988120 shutdown_connections 2023-03-09T14:52:13.257 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:13.242+0000 7f288be14700 1 -- 172.21.15.38:0/1512988120 wait complete. 2023-03-09T14:52:13.257 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:12 smithi038 ceph-mon[92422]: Health check update: Degraded data redundancy: 6543/35190 objects degraded (18.593%), 41 pgs degraded (PG_DEGRADED) 2023-03-09T14:52:13.259 INFO:tasks.cephfs.filesystem:scrub status for tag:a14659f4-d7ea-4c5a-b15b-6138af3f909f - {'path': '/', 'tag': 'a14659f4-d7ea-4c5a-b15b-6138af3f909f', 'options': 'recursive,force'} 2023-03-09T14:52:13.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:12 smithi122 ceph-mon[93869]: Health check update: Degraded data redundancy: 6543/35190 objects degraded (18.593%), 41 pgs degraded (PG_DEGRADED) 2023-03-09T14:52:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:12 smithi181 ceph-mon[94189]: Health check update: Degraded data redundancy: 6543/35190 objects degraded (18.593%), 41 pgs degraded (PG_DEGRADED) 2023-03-09T14:52:13.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 105.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T14:52:14.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:13 smithi038 ceph-mon[92422]: pgmap v244: 129 pgs: 1 active+recovery_wait+degraded, 11 peering, 3 down, 1 active+recovery_wait+undersized+degraded, 15 undersized+peered, 31 active+undersized, 11 undersized+degraded+peered, 26 active+undersized+degraded, 30 active+clean; 18 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 1.2 MiB/s wr, 187 op/s; 5731/35262 objects degraded (16.253%); 34 MiB/s, 9 keys/s, 8 objects/s recovering 2023-03-09T14:52:14.329 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:13 smithi181 ceph-mon[94189]: pgmap v244: 129 pgs: 1 active+recovery_wait+degraded, 11 peering, 3 down, 1 active+recovery_wait+undersized+degraded, 15 undersized+peered, 31 active+undersized, 11 undersized+degraded+peered, 26 active+undersized+degraded, 30 active+clean; 18 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 1.2 MiB/s wr, 187 op/s; 5731/35262 objects degraded (16.253%); 34 MiB/s, 9 keys/s, 8 objects/s recovering 2023-03-09T14:52:14.330 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:14 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 2023-03-09T14:52:14.326+0000 7f627c68e640 -1 osd.8 146 log_to_monitors true 2023-03-09T14:52:14.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:13 smithi122 ceph-mon[93869]: pgmap v244: 129 pgs: 1 active+recovery_wait+degraded, 11 peering, 3 down, 1 active+recovery_wait+undersized+degraded, 15 undersized+peered, 31 active+undersized, 11 undersized+degraded+peered, 26 active+undersized+degraded, 30 active+clean; 18 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 1.2 MiB/s wr, 187 op/s; 5731/35262 objects degraded (16.253%); 34 MiB/s, 9 keys/s, 8 objects/s recovering 2023-03-09T14:52:15.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:14 smithi038 ceph-mon[92422]: from='osd.8 [v2:172.21.15.181:6816/496864406,v1:172.21.15.181:6817/496864406]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-03-09T14:52:15.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:14 smithi038 ceph-mon[92422]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-03-09T14:52:15.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:14 smithi122 ceph-mon[93869]: from='osd.8 [v2:172.21.15.181:6816/496864406,v1:172.21.15.181:6817/496864406]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-03-09T14:52:15.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:14 smithi122 ceph-mon[93869]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-03-09T14:52:15.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:14 smithi181 ceph-mon[94189]: from='osd.8 [v2:172.21.15.181:6816/496864406,v1:172.21.15.181:6817/496864406]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-03-09T14:52:15.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:14 smithi181 ceph-mon[94189]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-03-09T14:52:15.482 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:15 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 2023-03-09T14:52:15.035+0000 7f6268400700 -1 osd.8 146 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-09T14:52:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:15 smithi038 ceph-mon[92422]: pgmap v245: 129 pgs: 1 active+recovery_wait+degraded, 11 peering, 3 down, 1 active+recovery_wait+undersized+degraded, 10 undersized+peered, 29 active+undersized, 7 undersized+degraded+peered, 28 active+undersized+degraded, 39 active+clean; 18 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 1.3 MiB/s wr, 189 op/s; 5142/35394 objects degraded (14.528%); 29 MiB/s, 17 keys/s, 8 objects/s recovering 2023-03-09T14:52:16.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:15 smithi038 ceph-mon[92422]: Health check update: Reduced data availability: 4 pgs inactive, 3 pgs down (PG_AVAILABILITY) 2023-03-09T14:52:16.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:15 smithi038 ceph-mon[92422]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-03-09T14:52:16.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:15 smithi038 ceph-mon[92422]: osdmap e153: 12 total, 9 up, 12 in 2023-03-09T14:52:16.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:15 smithi038 ceph-mon[92422]: from='osd.8 [v2:172.21.15.181:6816/496864406,v1:172.21.15.181:6817/496864406]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2023-03-09T14:52:16.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:15 smithi038 ceph-mon[92422]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2023-03-09T14:52:16.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:15 smithi122 ceph-mon[93869]: pgmap v245: 129 pgs: 1 active+recovery_wait+degraded, 11 peering, 3 down, 1 active+recovery_wait+undersized+degraded, 10 undersized+peered, 29 active+undersized, 7 undersized+degraded+peered, 28 active+undersized+degraded, 39 active+clean; 18 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 1.3 MiB/s wr, 189 op/s; 5142/35394 objects degraded (14.528%); 29 MiB/s, 17 keys/s, 8 objects/s recovering 2023-03-09T14:52:16.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:15 smithi122 ceph-mon[93869]: Health check update: Reduced data availability: 4 pgs inactive, 3 pgs down (PG_AVAILABILITY) 2023-03-09T14:52:16.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:15 smithi122 ceph-mon[93869]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-03-09T14:52:16.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:15 smithi122 ceph-mon[93869]: osdmap e153: 12 total, 9 up, 12 in 2023-03-09T14:52:16.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:15 smithi122 ceph-mon[93869]: from='osd.8 [v2:172.21.15.181:6816/496864406,v1:172.21.15.181:6817/496864406]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2023-03-09T14:52:16.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:15 smithi122 ceph-mon[93869]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2023-03-09T14:52:16.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:15 smithi181 ceph-mon[94189]: pgmap v245: 129 pgs: 1 active+recovery_wait+degraded, 11 peering, 3 down, 1 active+recovery_wait+undersized+degraded, 10 undersized+peered, 29 active+undersized, 7 undersized+degraded+peered, 28 active+undersized+degraded, 39 active+clean; 18 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 1.3 MiB/s wr, 189 op/s; 5142/35394 objects degraded (14.528%); 29 MiB/s, 17 keys/s, 8 objects/s recovering 2023-03-09T14:52:16.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:15 smithi181 ceph-mon[94189]: Health check update: Reduced data availability: 4 pgs inactive, 3 pgs down (PG_AVAILABILITY) 2023-03-09T14:52:16.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:15 smithi181 ceph-mon[94189]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-03-09T14:52:16.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:15 smithi181 ceph-mon[94189]: osdmap e153: 12 total, 9 up, 12 in 2023-03-09T14:52:16.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:15 smithi181 ceph-mon[94189]: from='osd.8 [v2:172.21.15.181:6816/496864406,v1:172.21.15.181:6817/496864406]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2023-03-09T14:52:16.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:15 smithi181 ceph-mon[94189]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2023-03-09T14:52:17.250 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:16 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 2023-03-09T14:52:16.982+0000 7f4ead873640 -1 osd.0 144 log_to_monitors true 2023-03-09T14:52:17.251 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:17 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 2023-03-09T14:52:17.035+0000 7f4e995e5700 -1 osd.0 144 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-09T14:52:17.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:16 smithi038 ceph-mon[92422]: Health check update: 2 osds down (OSD_DOWN) 2023-03-09T14:52:17.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:16 smithi038 ceph-mon[92422]: osd.8 [v2:172.21.15.181:6816/496864406,v1:172.21.15.181:6817/496864406] boot 2023-03-09T14:52:17.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:16 smithi038 ceph-mon[92422]: osdmap e154: 12 total, 10 up, 12 in 2023-03-09T14:52:17.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:16 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-03-09T14:52:17.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:16 smithi038 ceph-mon[92422]: from='osd.0 [v2:172.21.15.38:6810/549013282,v1:172.21.15.38:6811/549013282]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-03-09T14:52:17.478 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:17 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 2023-03-09T14:52:17.084+0000 7fa3dc4f8640 -1 osd.1 144 log_to_monitors true 2023-03-09T14:52:17.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:16 smithi122 ceph-mon[93869]: Health check update: 2 osds down (OSD_DOWN) 2023-03-09T14:52:17.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:16 smithi122 ceph-mon[93869]: osd.8 [v2:172.21.15.181:6816/496864406,v1:172.21.15.181:6817/496864406] boot 2023-03-09T14:52:17.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:16 smithi122 ceph-mon[93869]: osdmap e154: 12 total, 10 up, 12 in 2023-03-09T14:52:17.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:16 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-03-09T14:52:17.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:16 smithi122 ceph-mon[93869]: from='osd.0 [v2:172.21.15.38:6810/549013282,v1:172.21.15.38:6811/549013282]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-03-09T14:52:17.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:16 smithi181 ceph-mon[94189]: Health check update: 2 osds down (OSD_DOWN) 2023-03-09T14:52:17.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:16 smithi181 ceph-mon[94189]: osd.8 [v2:172.21.15.181:6816/496864406,v1:172.21.15.181:6817/496864406] boot 2023-03-09T14:52:17.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:16 smithi181 ceph-mon[94189]: osdmap e154: 12 total, 10 up, 12 in 2023-03-09T14:52:17.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:16 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-03-09T14:52:17.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:16 smithi181 ceph-mon[94189]: from='osd.0 [v2:172.21.15.38:6810/549013282,v1:172.21.15.38:6811/549013282]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-03-09T14:52:18.477 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:18 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 2023-03-09T14:52:18.042+0000 7fa3c826a700 -1 osd.1 144 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-09T14:52:18.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:18 smithi122 ceph-mon[93869]: pgmap v248: 129 pgs: 2 peering, 3 down, 10 undersized+peered, 30 active+undersized, 7 undersized+degraded+peered, 29 active+undersized+degraded, 48 active+clean; 18 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 2.5 MiB/s wr, 339 op/s; 5490/36218 objects degraded (15.158%); 144 MiB/s, 17 keys/s, 38 objects/s recovering 2023-03-09T14:52:18.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:18 smithi122 ceph-mon[93869]: from='osd.0 [v2:172.21.15.38:6810/549013282,v1:172.21.15.38:6811/549013282]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-03-09T14:52:18.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:18 smithi122 ceph-mon[93869]: osdmap e155: 12 total, 10 up, 12 in 2023-03-09T14:52:18.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:18 smithi122 ceph-mon[93869]: from='osd.0 [v2:172.21.15.38:6810/549013282,v1:172.21.15.38:6811/549013282]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2023-03-09T14:52:18.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:18 smithi122 ceph-mon[93869]: from='osd.1 [v2:172.21.15.122:6808/2161602167,v1:172.21.15.122:6809/2161602167]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-03-09T14:52:18.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:18 smithi122 ceph-mon[93869]: Health check update: Degraded data redundancy: 5490/36218 objects degraded (15.158%), 36 pgs degraded (PG_DEGRADED) 2023-03-09T14:52:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:18 smithi181 ceph-mon[94189]: pgmap v248: 129 pgs: 2 peering, 3 down, 10 undersized+peered, 30 active+undersized, 7 undersized+degraded+peered, 29 active+undersized+degraded, 48 active+clean; 18 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 2.5 MiB/s wr, 339 op/s; 5490/36218 objects degraded (15.158%); 144 MiB/s, 17 keys/s, 38 objects/s recovering 2023-03-09T14:52:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:18 smithi181 ceph-mon[94189]: from='osd.0 [v2:172.21.15.38:6810/549013282,v1:172.21.15.38:6811/549013282]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-03-09T14:52:18.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:18 smithi181 ceph-mon[94189]: osdmap e155: 12 total, 10 up, 12 in 2023-03-09T14:52:18.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:18 smithi181 ceph-mon[94189]: from='osd.0 [v2:172.21.15.38:6810/549013282,v1:172.21.15.38:6811/549013282]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2023-03-09T14:52:18.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:18 smithi181 ceph-mon[94189]: from='osd.1 [v2:172.21.15.122:6808/2161602167,v1:172.21.15.122:6809/2161602167]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-03-09T14:52:18.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:18 smithi181 ceph-mon[94189]: Health check update: Degraded data redundancy: 5490/36218 objects degraded (15.158%), 36 pgs degraded (PG_DEGRADED) 2023-03-09T14:52:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:18 smithi038 ceph-mon[92422]: pgmap v248: 129 pgs: 2 peering, 3 down, 10 undersized+peered, 30 active+undersized, 7 undersized+degraded+peered, 29 active+undersized+degraded, 48 active+clean; 18 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 2.5 MiB/s wr, 339 op/s; 5490/36218 objects degraded (15.158%); 144 MiB/s, 17 keys/s, 38 objects/s recovering 2023-03-09T14:52:18.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:18 smithi038 ceph-mon[92422]: from='osd.0 [v2:172.21.15.38:6810/549013282,v1:172.21.15.38:6811/549013282]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-03-09T14:52:18.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:18 smithi038 ceph-mon[92422]: osdmap e155: 12 total, 10 up, 12 in 2023-03-09T14:52:18.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:18 smithi038 ceph-mon[92422]: from='osd.0 [v2:172.21.15.38:6810/549013282,v1:172.21.15.38:6811/549013282]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2023-03-09T14:52:18.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:18 smithi038 ceph-mon[92422]: from='osd.1 [v2:172.21.15.122:6808/2161602167,v1:172.21.15.122:6809/2161602167]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-03-09T14:52:18.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:18 smithi038 ceph-mon[92422]: Health check update: Degraded data redundancy: 5490/36218 objects degraded (15.158%), 36 pgs degraded (PG_DEGRADED) 2023-03-09T14:52:18.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 110.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T14:52:19.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:19 smithi122 ceph-mon[93869]: osd.0 [v2:172.21.15.38:6810/549013282,v1:172.21.15.38:6811/549013282] boot 2023-03-09T14:52:19.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:19 smithi122 ceph-mon[93869]: from='osd.1 [v2:172.21.15.122:6808/2161602167,v1:172.21.15.122:6809/2161602167]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-03-09T14:52:19.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:19 smithi122 ceph-mon[93869]: osdmap e156: 12 total, 11 up, 12 in 2023-03-09T14:52:19.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-03-09T14:52:19.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:19 smithi122 ceph-mon[93869]: from='osd.1 [v2:172.21.15.122:6808/2161602167,v1:172.21.15.122:6809/2161602167]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-03-09T14:52:19.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:19 smithi181 ceph-mon[94189]: osd.0 [v2:172.21.15.38:6810/549013282,v1:172.21.15.38:6811/549013282] boot 2023-03-09T14:52:19.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:19 smithi181 ceph-mon[94189]: from='osd.1 [v2:172.21.15.122:6808/2161602167,v1:172.21.15.122:6809/2161602167]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-03-09T14:52:19.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:19 smithi181 ceph-mon[94189]: osdmap e156: 12 total, 11 up, 12 in 2023-03-09T14:52:19.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-03-09T14:52:19.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:19 smithi181 ceph-mon[94189]: from='osd.1 [v2:172.21.15.122:6808/2161602167,v1:172.21.15.122:6809/2161602167]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-03-09T14:52:19.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:19 smithi038 ceph-mon[92422]: osd.0 [v2:172.21.15.38:6810/549013282,v1:172.21.15.38:6811/549013282] boot 2023-03-09T14:52:19.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:19 smithi038 ceph-mon[92422]: from='osd.1 [v2:172.21.15.122:6808/2161602167,v1:172.21.15.122:6809/2161602167]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-03-09T14:52:19.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:19 smithi038 ceph-mon[92422]: osdmap e156: 12 total, 11 up, 12 in 2023-03-09T14:52:19.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-03-09T14:52:19.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:19 smithi038 ceph-mon[92422]: from='osd.1 [v2:172.21.15.122:6808/2161602167,v1:172.21.15.122:6809/2161602167]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-03-09T14:52:20.344 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-mon[92422]: pgmap v251: 129 pgs: 2 peering, 1 down, 9 undersized+peered, 31 active+undersized, 8 undersized+degraded+peered, 29 active+undersized+degraded, 49 active+clean; 18 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 3.4 MiB/s wr, 448 op/s; 6322/36530 objects degraded (17.306%); 168 MiB/s, 43 objects/s recovering 2023-03-09T14:52:20.344 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-mon[92422]: Health check cleared: OSD_DOWN (was: 1 osds down) 2023-03-09T14:52:20.344 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-mon[92422]: osd.1 [v2:172.21.15.122:6808/2161602167,v1:172.21.15.122:6809/2161602167] boot 2023-03-09T14:52:20.344 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-mon[92422]: osdmap e157: 12 total, 12 up, 12 in 2023-03-09T14:52:20.344 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-03-09T14:52:20.353 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7f4e865bf700 time 2023-03-09T14:52:20.131903+0000 2023-03-09T14:52:20.353 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op") 2023-03-09T14:52:20.354 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:52:20.354 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x563c21e6ac85] 2023-03-09T14:52:20.354 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 2: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x563c221236e4] 2023-03-09T14:52:20.354 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x563c22129d5e] 2023-03-09T14:52:20.354 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 4: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x563c21f79986] 2023-03-09T14:52:20.354 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 5: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x563c2228f729] 2023-03-09T14:52:20.354 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x563c21f8eebf] 2023-03-09T14:52:20.355 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x563c22640d85] 2023-03-09T14:52:20.355 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x563c22643f84] 2023-03-09T14:52:20.355 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f4eab3231ca] 2023-03-09T14:52:20.355 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 10: clone() 2023-03-09T14:52:20.355 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 2023-03-09T14:52:20.131+0000 7f4e865bf700 -1 osd.0 pg_epoch: 158 pg[4.bs0( v 158'2161 (0'0,158'2161] local-lis/les=157/158 n=260 ec=85/85 lis/c=157/157 les/c/f=158/158/0 sis=157) [0,9,1,4]p0(0) r=0 lpr=157 luod=144'2158 crt=144'2158 lcod 0'0 mlcod 0'0 active+clean] bad op order, already applied 87681 > this 87680 2023-03-09T14:52:20.355 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: *** Caught signal (Aborted) ** 2023-03-09T14:52:20.355 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: in thread 7f4e865bf700 thread_name:tp_osd_tp 2023-03-09T14:52:20.355 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 2023-03-09T14:52:20.135+0000 7f4e865bf700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7f4e865bf700 time 2023-03-09T14:52:20.131903+0000 2023-03-09T14:52:20.356 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op") 2023-03-09T14:52:20.357 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 2023-03-09T14:52:20.357 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:52:20.357 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x563c21e6ac85] 2023-03-09T14:52:20.357 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 2: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x563c221236e4] 2023-03-09T14:52:20.357 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x563c22129d5e] 2023-03-09T14:52:20.357 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 4: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x563c21f79986] 2023-03-09T14:52:20.357 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 5: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x563c2228f729] 2023-03-09T14:52:20.358 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x563c21f8eebf] 2023-03-09T14:52:20.358 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x563c22640d85] 2023-03-09T14:52:20.358 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x563c22643f84] 2023-03-09T14:52:20.358 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f4eab3231ca] 2023-03-09T14:52:20.358 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 10: clone() 2023-03-09T14:52:20.358 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 2023-03-09T14:52:20.358 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:52:20.358 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f4eab32dcf0] 2023-03-09T14:52:20.359 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 2: gsignal() 2023-03-09T14:52:20.359 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 3: abort() 2023-03-09T14:52:20.359 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x563c21e6ad45] 2023-03-09T14:52:20.359 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 5: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x563c221236e4] 2023-03-09T14:52:20.359 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 6: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x563c22129d5e] 2023-03-09T14:52:20.359 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x563c21f79986] 2023-03-09T14:52:20.359 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 8: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x563c2228f729] 2023-03-09T14:52:20.360 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x563c21f8eebf] 2023-03-09T14:52:20.360 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x563c22640d85] 2023-03-09T14:52:20.360 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x563c22643f84] 2023-03-09T14:52:20.361 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f4eab3231ca] 2023-03-09T14:52:20.361 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 13: clone() 2023-03-09T14:52:20.361 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 2023-03-09T14:52:20.148+0000 7f4e865bf700 -1 *** Caught signal (Aborted) ** 2023-03-09T14:52:20.361 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: in thread 7f4e865bf700 thread_name:tp_osd_tp 2023-03-09T14:52:20.361 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 2023-03-09T14:52:20.361 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:52:20.361 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f4eab32dcf0] 2023-03-09T14:52:20.362 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 2: gsignal() 2023-03-09T14:52:20.362 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 3: abort() 2023-03-09T14:52:20.362 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x563c21e6ad45] 2023-03-09T14:52:20.362 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 5: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x563c221236e4] 2023-03-09T14:52:20.362 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 6: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x563c22129d5e] 2023-03-09T14:52:20.362 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x563c21f79986] 2023-03-09T14:52:20.362 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 8: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x563c2228f729] 2023-03-09T14:52:20.363 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x563c21f8eebf] 2023-03-09T14:52:20.363 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x563c22640d85] 2023-03-09T14:52:20.363 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x563c22643f84] 2023-03-09T14:52:20.363 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f4eab3231ca] 2023-03-09T14:52:20.363 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 13: clone() 2023-03-09T14:52:20.363 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-03-09T14:52:20.363 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 2023-03-09T14:52:20.363 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: -7543> 2023-03-09T14:52:20.131+0000 7f4e865bf700 -1 osd.0 pg_epoch: 158 pg[4.bs0( v 158'2161 (0'0,158'2161] local-lis/les=157/158 n=260 ec=85/85 lis/c=157/157 les/c/f=158/158/0 sis=157) [0,9,1,4]p0(0) r=0 lpr=157 luod=144'2158 crt=144'2158 lcod 0'0 mlcod 0'0 active+clean] bad op order, already applied 87681 > this 87680 2023-03-09T14:52:20.364 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: -5472> 2023-03-09T14:52:20.135+0000 7f4e865bf700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7f4e865bf700 time 2023-03-09T14:52:20.131903+0000 2023-03-09T14:52:20.364 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op") 2023-03-09T14:52:20.364 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 2023-03-09T14:52:20.364 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:52:20.364 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x563c21e6ac85] 2023-03-09T14:52:20.364 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 2: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x563c221236e4] 2023-03-09T14:52:20.364 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x563c22129d5e] 2023-03-09T14:52:20.365 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 4: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x563c21f79986] 2023-03-09T14:52:20.365 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 5: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x563c2228f729] 2023-03-09T14:52:20.365 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x563c21f8eebf] 2023-03-09T14:52:20.365 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x563c22640d85] 2023-03-09T14:52:20.365 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x563c22643f84] 2023-03-09T14:52:20.365 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f4eab3231ca] 2023-03-09T14:52:20.365 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 10: clone() 2023-03-09T14:52:20.365 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 2023-03-09T14:52:20.366 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: -4523> 2023-03-09T14:52:20.148+0000 7f4e865bf700 -1 *** Caught signal (Aborted) ** 2023-03-09T14:52:20.366 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: in thread 7f4e865bf700 thread_name:tp_osd_tp 2023-03-09T14:52:20.366 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 2023-03-09T14:52:20.366 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:52:20.366 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f4eab32dcf0] 2023-03-09T14:52:20.366 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 2: gsignal() 2023-03-09T14:52:20.366 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 3: abort() 2023-03-09T14:52:20.366 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x563c21e6ad45] 2023-03-09T14:52:20.367 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 5: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x563c221236e4] 2023-03-09T14:52:20.367 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 6: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x563c22129d5e] 2023-03-09T14:52:20.367 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x563c21f79986] 2023-03-09T14:52:20.367 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 8: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x563c2228f729] 2023-03-09T14:52:20.367 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x563c21f8eebf] 2023-03-09T14:52:20.367 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x563c22640d85] 2023-03-09T14:52:20.367 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x563c22643f84] 2023-03-09T14:52:20.368 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f4eab3231ca] 2023-03-09T14:52:20.368 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 13: clone() 2023-03-09T14:52:20.368 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-03-09T14:52:20.368 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 2023-03-09T14:52:20.368 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: -8544> 2023-03-09T14:52:20.131+0000 7f4e865bf700 -1 osd.0 pg_epoch: 158 pg[4.bs0( v 158'2161 (0'0,158'2161] local-lis/les=157/158 n=260 ec=85/85 lis/c=157/157 les/c/f=158/158/0 sis=157) [0,9,1,4]p0(0) r=0 lpr=157 luod=144'2158 crt=144'2158 lcod 0'0 mlcod 0'0 active+clean] bad op order, already applied 87681 > this 87680 2023-03-09T14:52:20.368 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: -6473> 2023-03-09T14:52:20.135+0000 7f4e865bf700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7f4e865bf700 time 2023-03-09T14:52:20.131903+0000 2023-03-09T14:52:20.368 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op") 2023-03-09T14:52:20.368 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 2023-03-09T14:52:20.369 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:52:20.369 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x563c21e6ac85] 2023-03-09T14:52:20.369 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 2: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x563c221236e4] 2023-03-09T14:52:20.369 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x563c22129d5e] 2023-03-09T14:52:20.369 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 4: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x563c21f79986] 2023-03-09T14:52:20.369 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 5: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x563c2228f729] 2023-03-09T14:52:20.369 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x563c21f8eebf] 2023-03-09T14:52:20.370 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x563c22640d85] 2023-03-09T14:52:20.370 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x563c22643f84] 2023-03-09T14:52:20.370 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f4eab3231ca] 2023-03-09T14:52:20.370 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 10: clone() 2023-03-09T14:52:20.370 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 2023-03-09T14:52:20.370 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: -5524> 2023-03-09T14:52:20.148+0000 7f4e865bf700 -1 *** Caught signal (Aborted) ** 2023-03-09T14:52:20.371 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: in thread 7f4e865bf700 thread_name:tp_osd_tp 2023-03-09T14:52:20.371 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 2023-03-09T14:52:20.371 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:52:20.371 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f4eab32dcf0] 2023-03-09T14:52:20.371 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 2: gsignal() 2023-03-09T14:52:20.371 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 3: abort() 2023-03-09T14:52:20.371 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x563c21e6ad45] 2023-03-09T14:52:20.371 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 5: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x563c221236e4] 2023-03-09T14:52:20.372 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 6: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x563c22129d5e] 2023-03-09T14:52:20.372 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x563c21f79986] 2023-03-09T14:52:20.372 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 8: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x563c2228f729] 2023-03-09T14:52:20.372 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x563c21f8eebf] 2023-03-09T14:52:20.372 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x563c22640d85] 2023-03-09T14:52:20.372 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x563c22643f84] 2023-03-09T14:52:20.372 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f4eab3231ca] 2023-03-09T14:52:20.373 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 13: clone() 2023-03-09T14:52:20.373 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-03-09T14:52:20.373 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[152194]: 2023-03-09T14:52:20.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:20 smithi122 ceph-mon[93869]: pgmap v251: 129 pgs: 2 peering, 1 down, 9 undersized+peered, 31 active+undersized, 8 undersized+degraded+peered, 29 active+undersized+degraded, 49 active+clean; 18 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 3.4 MiB/s wr, 448 op/s; 6322/36530 objects degraded (17.306%); 168 MiB/s, 43 objects/s recovering 2023-03-09T14:52:20.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:20 smithi122 ceph-mon[93869]: Health check cleared: OSD_DOWN (was: 1 osds down) 2023-03-09T14:52:20.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:20 smithi122 ceph-mon[93869]: osd.1 [v2:172.21.15.122:6808/2161602167,v1:172.21.15.122:6809/2161602167] boot 2023-03-09T14:52:20.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:20 smithi122 ceph-mon[93869]: osdmap e157: 12 total, 12 up, 12 in 2023-03-09T14:52:20.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-03-09T14:52:20.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:20 smithi181 ceph-mon[94189]: pgmap v251: 129 pgs: 2 peering, 1 down, 9 undersized+peered, 31 active+undersized, 8 undersized+degraded+peered, 29 active+undersized+degraded, 49 active+clean; 18 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 3.4 MiB/s wr, 448 op/s; 6322/36530 objects degraded (17.306%); 168 MiB/s, 43 objects/s recovering 2023-03-09T14:52:20.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:20 smithi181 ceph-mon[94189]: Health check cleared: OSD_DOWN (was: 1 osds down) 2023-03-09T14:52:20.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:20 smithi181 ceph-mon[94189]: osd.1 [v2:172.21.15.122:6808/2161602167,v1:172.21.15.122:6809/2161602167] boot 2023-03-09T14:52:20.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:20 smithi181 ceph-mon[94189]: osdmap e157: 12 total, 12 up, 12 in 2023-03-09T14:52:20.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-03-09T14:52:21.051 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:20 smithi038 systemd[1]: ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.0.service: Main process exited, code=exited, status=134/n/a 2023-03-09T14:52:21.345 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:21 smithi038 ceph-mon[92422]: osdmap e158: 12 total, 12 up, 12 in 2023-03-09T14:52:21.346 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:21 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.11 2023-03-09T14:52:21.346 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:21 smithi038 ceph-mon[92422]: osd.0 failed (root=default,host=smithi038) (connection refused reported by osd.11) 2023-03-09T14:52:21.346 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:21 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.11 2023-03-09T14:52:21.346 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:21 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.11 2023-03-09T14:52:21.346 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:21 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.3 2023-03-09T14:52:21.346 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:21 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.1 2023-03-09T14:52:21.346 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:21 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.2 2023-03-09T14:52:21.347 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:21 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.7 2023-03-09T14:52:21.347 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:21 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.6 2023-03-09T14:52:21.347 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:21 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.9 2023-03-09T14:52:21.347 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:21 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.3 2023-03-09T14:52:21.347 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:21 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.1 2023-03-09T14:52:21.347 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:21 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.5 2023-03-09T14:52:21.347 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:21 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.7 2023-03-09T14:52:21.347 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:21 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.6 2023-03-09T14:52:21.348 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:21 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.9 2023-03-09T14:52:21.348 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:21 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.3 2023-03-09T14:52:21.348 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:21 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.1 2023-03-09T14:52:21.348 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:21 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.2 2023-03-09T14:52:21.348 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:21 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.6 2023-03-09T14:52:21.348 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:21 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.9 2023-03-09T14:52:21.348 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:21 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.8 2023-03-09T14:52:21.348 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:21 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.5 2023-03-09T14:52:21.349 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:21 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.10 2023-03-09T14:52:21.349 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:21 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.2 2023-03-09T14:52:21.349 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:21 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.4 2023-03-09T14:52:21.349 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:21 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.8 2023-03-09T14:52:21.349 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:21 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.5 2023-03-09T14:52:21.349 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:21 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.10 2023-03-09T14:52:21.349 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:21 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.4 2023-03-09T14:52:21.350 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:21 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.11 2023-03-09T14:52:21.350 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:21 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.5 2023-03-09T14:52:21.350 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:21 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.6 2023-03-09T14:52:21.350 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:21 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.11 2023-03-09T14:52:21.350 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:21 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.3 2023-03-09T14:52:21.350 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:21 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.1 2023-03-09T14:52:21.350 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:21 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.5 2023-03-09T14:52:21.350 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:21 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.7 2023-03-09T14:52:21.351 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:21 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.6 2023-03-09T14:52:21.351 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:21 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.9 2023-03-09T14:52:21.351 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:21 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.11 2023-03-09T14:52:21.351 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:21 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.3 2023-03-09T14:52:21.351 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:21 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.1 2023-03-09T14:52:21.351 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:21 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.2 2023-03-09T14:52:21.351 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:21 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.7 2023-03-09T14:52:21.351 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:21 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.6 2023-03-09T14:52:21.352 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:21 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.9 2023-03-09T14:52:21.352 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:21 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.3 2023-03-09T14:52:21.352 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:21 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.1 2023-03-09T14:52:21.352 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:21 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.8 2023-03-09T14:52:21.352 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:21 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.9 2023-03-09T14:52:21.352 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:21 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.5 2023-03-09T14:52:21.352 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:21 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.2 2023-03-09T14:52:21.353 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:21 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.4 2023-03-09T14:52:21.353 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:21 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.8 2023-03-09T14:52:21.353 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:21 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.10 2023-03-09T14:52:21.353 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:21 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.2 2023-03-09T14:52:21.353 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:21 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.4 2023-03-09T14:52:21.353 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:21 smithi038 ceph-mon[92422]: osd.0 reported immediately failed by osd.10 2023-03-09T14:52:21.353 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:21 smithi038 ceph-mon[92422]: Health check failed: 1 osds down (OSD_DOWN) 2023-03-09T14:52:21.353 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:21 smithi038 ceph-mon[92422]: osdmap e159: 12 total, 11 up, 12 in 2023-03-09T14:52:21.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:21 smithi122 ceph-mon[93869]: osdmap e158: 12 total, 12 up, 12 in 2023-03-09T14:52:21.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:21 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.11 2023-03-09T14:52:21.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:21 smithi122 ceph-mon[93869]: osd.0 failed (root=default,host=smithi038) (connection refused reported by osd.11) 2023-03-09T14:52:21.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:21 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.11 2023-03-09T14:52:21.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:21 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.11 2023-03-09T14:52:21.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:21 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.3 2023-03-09T14:52:21.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:21 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.1 2023-03-09T14:52:21.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:21 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.2 2023-03-09T14:52:21.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:21 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.7 2023-03-09T14:52:21.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:21 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.6 2023-03-09T14:52:21.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:21 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.9 2023-03-09T14:52:21.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:21 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.3 2023-03-09T14:52:21.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:21 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.1 2023-03-09T14:52:21.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:21 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.5 2023-03-09T14:52:21.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:21 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.7 2023-03-09T14:52:21.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:21 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.6 2023-03-09T14:52:21.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:21 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.9 2023-03-09T14:52:21.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:21 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.3 2023-03-09T14:52:21.481 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:21 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.1 2023-03-09T14:52:21.481 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:21 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.2 2023-03-09T14:52:21.481 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:21 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.6 2023-03-09T14:52:21.481 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:21 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.9 2023-03-09T14:52:21.481 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:21 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.8 2023-03-09T14:52:21.481 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:21 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.5 2023-03-09T14:52:21.481 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:21 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.10 2023-03-09T14:52:21.481 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:21 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.2 2023-03-09T14:52:21.482 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:21 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.4 2023-03-09T14:52:21.482 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:21 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.8 2023-03-09T14:52:21.482 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:21 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.5 2023-03-09T14:52:21.482 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:21 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.10 2023-03-09T14:52:21.482 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:21 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.4 2023-03-09T14:52:21.482 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:21 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.11 2023-03-09T14:52:21.482 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:21 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.5 2023-03-09T14:52:21.483 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:21 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.6 2023-03-09T14:52:21.483 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:21 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.11 2023-03-09T14:52:21.483 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:21 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.3 2023-03-09T14:52:21.483 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:21 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.1 2023-03-09T14:52:21.483 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:21 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.5 2023-03-09T14:52:21.483 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:21 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.7 2023-03-09T14:52:21.483 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:21 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.6 2023-03-09T14:52:21.483 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:21 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.9 2023-03-09T14:52:21.484 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:21 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.11 2023-03-09T14:52:21.484 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:21 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.3 2023-03-09T14:52:21.484 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:21 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.1 2023-03-09T14:52:21.484 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:21 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.2 2023-03-09T14:52:21.484 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:21 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.7 2023-03-09T14:52:21.484 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:21 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.6 2023-03-09T14:52:21.484 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:21 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.9 2023-03-09T14:52:21.484 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:21 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.3 2023-03-09T14:52:21.485 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:21 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.1 2023-03-09T14:52:21.485 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:21 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.8 2023-03-09T14:52:21.485 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:21 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.9 2023-03-09T14:52:21.485 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:21 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.5 2023-03-09T14:52:21.485 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:21 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.2 2023-03-09T14:52:21.485 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:21 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.4 2023-03-09T14:52:21.485 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:21 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.8 2023-03-09T14:52:21.485 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:21 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.10 2023-03-09T14:52:21.486 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:21 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.2 2023-03-09T14:52:21.486 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:21 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.4 2023-03-09T14:52:21.486 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:21 smithi122 ceph-mon[93869]: osd.0 reported immediately failed by osd.10 2023-03-09T14:52:21.486 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:21 smithi122 ceph-mon[93869]: Health check failed: 1 osds down (OSD_DOWN) 2023-03-09T14:52:21.486 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:21 smithi122 ceph-mon[93869]: osdmap e159: 12 total, 11 up, 12 in 2023-03-09T14:52:21.487 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:21 smithi181 ceph-mon[94189]: osdmap e158: 12 total, 12 up, 12 in 2023-03-09T14:52:21.487 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:21 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.11 2023-03-09T14:52:21.487 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:21 smithi181 ceph-mon[94189]: osd.0 failed (root=default,host=smithi038) (connection refused reported by osd.11) 2023-03-09T14:52:21.487 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:21 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.11 2023-03-09T14:52:21.487 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:21 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.11 2023-03-09T14:52:21.488 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:21 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.3 2023-03-09T14:52:21.488 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:21 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.1 2023-03-09T14:52:21.488 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:21 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.2 2023-03-09T14:52:21.488 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:21 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.7 2023-03-09T14:52:21.488 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:21 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.6 2023-03-09T14:52:21.488 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:21 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.9 2023-03-09T14:52:21.488 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:21 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.3 2023-03-09T14:52:21.488 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:21 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.1 2023-03-09T14:52:21.489 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:21 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.5 2023-03-09T14:52:21.489 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:21 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.7 2023-03-09T14:52:21.489 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:21 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.6 2023-03-09T14:52:21.489 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:21 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.9 2023-03-09T14:52:21.489 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:21 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.3 2023-03-09T14:52:21.489 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:21 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.1 2023-03-09T14:52:21.489 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:21 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.2 2023-03-09T14:52:21.490 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:21 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.6 2023-03-09T14:52:21.490 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:21 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.9 2023-03-09T14:52:21.490 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:21 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.8 2023-03-09T14:52:21.490 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:21 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.5 2023-03-09T14:52:21.490 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:21 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.10 2023-03-09T14:52:21.490 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:21 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.2 2023-03-09T14:52:21.490 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:21 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.4 2023-03-09T14:52:21.490 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:21 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.8 2023-03-09T14:52:21.491 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:21 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.5 2023-03-09T14:52:21.491 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:21 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.10 2023-03-09T14:52:21.491 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:21 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.4 2023-03-09T14:52:21.491 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:21 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.11 2023-03-09T14:52:21.491 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:21 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.5 2023-03-09T14:52:21.491 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:21 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.6 2023-03-09T14:52:21.491 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:21 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.11 2023-03-09T14:52:21.491 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:21 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.3 2023-03-09T14:52:21.492 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:21 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.1 2023-03-09T14:52:21.492 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:21 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.5 2023-03-09T14:52:21.492 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:21 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.7 2023-03-09T14:52:21.492 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:21 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.6 2023-03-09T14:52:21.492 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:21 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.9 2023-03-09T14:52:21.492 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:21 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.11 2023-03-09T14:52:21.492 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:21 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.3 2023-03-09T14:52:21.493 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:21 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.1 2023-03-09T14:52:21.493 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:21 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.2 2023-03-09T14:52:21.493 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:21 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.7 2023-03-09T14:52:21.493 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:21 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.6 2023-03-09T14:52:21.493 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:21 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.9 2023-03-09T14:52:21.493 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:21 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.3 2023-03-09T14:52:21.493 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:21 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.1 2023-03-09T14:52:21.493 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:21 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.8 2023-03-09T14:52:21.494 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:21 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.9 2023-03-09T14:52:21.494 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:21 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.5 2023-03-09T14:52:21.494 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:21 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.2 2023-03-09T14:52:21.494 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:21 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.4 2023-03-09T14:52:21.494 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:21 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.8 2023-03-09T14:52:21.494 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:21 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.10 2023-03-09T14:52:21.494 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:21 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.2 2023-03-09T14:52:21.494 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:21 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.4 2023-03-09T14:52:21.495 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:21 smithi181 ceph-mon[94189]: osd.0 reported immediately failed by osd.10 2023-03-09T14:52:21.495 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:21 smithi181 ceph-mon[94189]: Health check failed: 1 osds down (OSD_DOWN) 2023-03-09T14:52:21.495 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:21 smithi181 ceph-mon[94189]: osdmap e159: 12 total, 11 up, 12 in 2023-03-09T14:52:22.462 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-mon[92422]: pgmap v254: 129 pgs: 1 remapped+peering, 2 active+recovering+degraded, 5 active+recovery_wait+degraded, 5 peering, 5 undersized+peered, 15 active+undersized, 4 undersized+degraded+peered, 9 active+undersized+degraded, 83 active+clean; 18 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 4.0 MiB/s wr, 453 op/s; 3626/37030 objects degraded (9.792%); 173 MiB/s, 16 keys/s, 44 objects/s recovering 2023-03-09T14:52:22.462 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-mon[92422]: Health check update: Reduced data availability: 4 pgs inactive (PG_AVAILABILITY) 2023-03-09T14:52:22.462 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-mon[92422]: osdmap e160: 12 total, 11 up, 12 in 2023-03-09T14:52:22.463 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:22 smithi038 systemd[1]: ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.0.service: Failed with result 'exit-code'. 2023-03-09T14:52:22.464 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 2023-03-09T14:52:22.153+0000 7f9ff0619700 -1 osd.9 pg_epoch: 160 pg[4.bs1( v 160'2164 (0'0,160'2164] local-lis/les=159/160 n=260 ec=85/85 lis/c=159/157 les/c/f=160/158/0 sis=159) [NONE,9,1,4]p9(1) r=1 lpr=159 pi=[157,159)/1 luod=160'2162 crt=158'2161 lcod 158'2161 mlcod 158'2161 active+undersized+degraded mbc={}] bad op order, already applied 87682 > this 87680 2023-03-09T14:52:22.464 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7f9ff0619700 time 2023-03-09T14:52:22.155180+0000 2023-03-09T14:52:22.464 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op") 2023-03-09T14:52:22.465 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:52:22.465 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x562e4ad1ac85] 2023-03-09T14:52:22.465 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 2: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x562e4afd36e4] 2023-03-09T14:52:22.465 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x562e4afd9d5e] 2023-03-09T14:52:22.466 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 4: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x562e4ae29986] 2023-03-09T14:52:22.466 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 5: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x562e4b13f729] 2023-03-09T14:52:22.466 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x562e4ae3eebf] 2023-03-09T14:52:22.467 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x562e4b4f0d85] 2023-03-09T14:52:22.467 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x562e4b4f3f84] 2023-03-09T14:52:22.467 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7fa0133791ca] 2023-03-09T14:52:22.467 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 10: clone() 2023-03-09T14:52:22.467 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 2023-03-09T14:52:22.156+0000 7f9ff0619700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7f9ff0619700 time 2023-03-09T14:52:22.155180+0000 2023-03-09T14:52:22.467 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op") 2023-03-09T14:52:22.468 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 2023-03-09T14:52:22.468 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:52:22.468 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x562e4ad1ac85] 2023-03-09T14:52:22.468 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 2: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x562e4afd36e4] 2023-03-09T14:52:22.468 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x562e4afd9d5e] 2023-03-09T14:52:22.468 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 4: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x562e4ae29986] 2023-03-09T14:52:22.468 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 5: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x562e4b13f729] 2023-03-09T14:52:22.468 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x562e4ae3eebf] 2023-03-09T14:52:22.469 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x562e4b4f0d85] 2023-03-09T14:52:22.469 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x562e4b4f3f84] 2023-03-09T14:52:22.469 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7fa0133791ca] 2023-03-09T14:52:22.469 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 10: clone() 2023-03-09T14:52:22.469 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 2023-03-09T14:52:22.469 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: *** Caught signal (Aborted) ** 2023-03-09T14:52:22.469 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: in thread 7f9ff0619700 thread_name:tp_osd_tp 2023-03-09T14:52:22.470 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:52:22.470 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7fa013383cf0] 2023-03-09T14:52:22.470 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 2: gsignal() 2023-03-09T14:52:22.470 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 3: abort() 2023-03-09T14:52:22.470 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x562e4ad1ad45] 2023-03-09T14:52:22.470 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 5: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x562e4afd36e4] 2023-03-09T14:52:22.470 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 6: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x562e4afd9d5e] 2023-03-09T14:52:22.471 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x562e4ae29986] 2023-03-09T14:52:22.471 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 8: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x562e4b13f729] 2023-03-09T14:52:22.471 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x562e4ae3eebf] 2023-03-09T14:52:22.471 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x562e4b4f0d85] 2023-03-09T14:52:22.472 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x562e4b4f3f84] 2023-03-09T14:52:22.472 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fa0133791ca] 2023-03-09T14:52:22.472 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 13: clone() 2023-03-09T14:52:22.472 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 2023-03-09T14:52:22.159+0000 7f9ff0619700 -1 *** Caught signal (Aborted) ** 2023-03-09T14:52:22.472 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: in thread 7f9ff0619700 thread_name:tp_osd_tp 2023-03-09T14:52:22.472 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 2023-03-09T14:52:22.472 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:52:22.473 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7fa013383cf0] 2023-03-09T14:52:22.473 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 2: gsignal() 2023-03-09T14:52:22.473 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 3: abort() 2023-03-09T14:52:22.473 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x562e4ad1ad45] 2023-03-09T14:52:22.473 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 5: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x562e4afd36e4] 2023-03-09T14:52:22.473 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 6: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x562e4afd9d5e] 2023-03-09T14:52:22.473 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x562e4ae29986] 2023-03-09T14:52:22.474 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 8: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x562e4b13f729] 2023-03-09T14:52:22.474 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x562e4ae3eebf] 2023-03-09T14:52:22.474 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x562e4b4f0d85] 2023-03-09T14:52:22.474 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x562e4b4f3f84] 2023-03-09T14:52:22.474 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fa0133791ca] 2023-03-09T14:52:22.474 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 13: clone() 2023-03-09T14:52:22.474 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-03-09T14:52:22.474 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 2023-03-09T14:52:22.475 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: -14> 2023-03-09T14:52:22.153+0000 7f9ff0619700 -1 osd.9 pg_epoch: 160 pg[4.bs1( v 160'2164 (0'0,160'2164] local-lis/les=159/160 n=260 ec=85/85 lis/c=159/157 les/c/f=160/158/0 sis=159) [NONE,9,1,4]p9(1) r=1 lpr=159 pi=[157,159)/1 luod=160'2162 crt=158'2161 lcod 158'2161 mlcod 158'2161 active+undersized+degraded mbc={}] bad op order, already applied 87682 > this 87680 2023-03-09T14:52:22.475 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: -7> 2023-03-09T14:52:22.156+0000 7f9ff0619700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7f9ff0619700 time 2023-03-09T14:52:22.155180+0000 2023-03-09T14:52:22.475 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op") 2023-03-09T14:52:22.475 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 2023-03-09T14:52:22.475 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:52:22.475 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x562e4ad1ac85] 2023-03-09T14:52:22.475 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 2: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x562e4afd36e4] 2023-03-09T14:52:22.475 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x562e4afd9d5e] 2023-03-09T14:52:22.476 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 4: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x562e4ae29986] 2023-03-09T14:52:22.476 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 5: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x562e4b13f729] 2023-03-09T14:52:22.476 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x562e4ae3eebf] 2023-03-09T14:52:22.476 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x562e4b4f0d85] 2023-03-09T14:52:22.476 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x562e4b4f3f84] 2023-03-09T14:52:22.476 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7fa0133791ca] 2023-03-09T14:52:22.476 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 10: clone() 2023-03-09T14:52:22.477 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 2023-03-09T14:52:22.477 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 0> 2023-03-09T14:52:22.159+0000 7f9ff0619700 -1 *** Caught signal (Aborted) ** 2023-03-09T14:52:22.477 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: in thread 7f9ff0619700 thread_name:tp_osd_tp 2023-03-09T14:52:22.477 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 2023-03-09T14:52:22.477 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:52:22.477 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7fa013383cf0] 2023-03-09T14:52:22.477 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 2: gsignal() 2023-03-09T14:52:22.477 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 3: abort() 2023-03-09T14:52:22.478 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x562e4ad1ad45] 2023-03-09T14:52:22.478 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 5: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x562e4afd36e4] 2023-03-09T14:52:22.478 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 6: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x562e4afd9d5e] 2023-03-09T14:52:22.478 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x562e4ae29986] 2023-03-09T14:52:22.478 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 8: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x562e4b13f729] 2023-03-09T14:52:22.478 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x562e4ae3eebf] 2023-03-09T14:52:22.478 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x562e4b4f0d85] 2023-03-09T14:52:22.479 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x562e4b4f3f84] 2023-03-09T14:52:22.479 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fa0133791ca] 2023-03-09T14:52:22.479 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 13: clone() 2023-03-09T14:52:22.479 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-03-09T14:52:22.479 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 2023-03-09T14:52:22.479 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: -15> 2023-03-09T14:52:22.153+0000 7f9ff0619700 -1 osd.9 pg_epoch: 160 pg[4.bs1( v 160'2164 (0'0,160'2164] local-lis/les=159/160 n=260 ec=85/85 lis/c=159/157 les/c/f=160/158/0 sis=159) [NONE,9,1,4]p9(1) r=1 lpr=159 pi=[157,159)/1 luod=160'2162 crt=158'2161 lcod 158'2161 mlcod 158'2161 active+undersized+degraded mbc={}] bad op order, already applied 87682 > this 87680 2023-03-09T14:52:22.479 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: -8> 2023-03-09T14:52:22.156+0000 7f9ff0619700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7f9ff0619700 time 2023-03-09T14:52:22.155180+0000 2023-03-09T14:52:22.479 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op") 2023-03-09T14:52:22.480 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 2023-03-09T14:52:22.480 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:52:22.480 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x562e4ad1ac85] 2023-03-09T14:52:22.480 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 2: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x562e4afd36e4] 2023-03-09T14:52:22.480 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x562e4afd9d5e] 2023-03-09T14:52:22.480 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 4: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x562e4ae29986] 2023-03-09T14:52:22.480 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 5: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x562e4b13f729] 2023-03-09T14:52:22.481 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x562e4ae3eebf] 2023-03-09T14:52:22.481 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x562e4b4f0d85] 2023-03-09T14:52:22.481 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x562e4b4f3f84] 2023-03-09T14:52:22.481 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7fa0133791ca] 2023-03-09T14:52:22.481 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 10: clone() 2023-03-09T14:52:22.481 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 2023-03-09T14:52:22.481 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: -1> 2023-03-09T14:52:22.159+0000 7f9ff0619700 -1 *** Caught signal (Aborted) ** 2023-03-09T14:52:22.481 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: in thread 7f9ff0619700 thread_name:tp_osd_tp 2023-03-09T14:52:22.482 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 2023-03-09T14:52:22.482 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:52:22.482 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7fa013383cf0] 2023-03-09T14:52:22.482 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 2: gsignal() 2023-03-09T14:52:22.482 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 3: abort() 2023-03-09T14:52:22.482 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x562e4ad1ad45] 2023-03-09T14:52:22.482 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 5: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x562e4afd36e4] 2023-03-09T14:52:22.482 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 6: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x562e4afd9d5e] 2023-03-09T14:52:22.483 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x562e4ae29986] 2023-03-09T14:52:22.483 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 8: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x562e4b13f729] 2023-03-09T14:52:22.483 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x562e4ae3eebf] 2023-03-09T14:52:22.483 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x562e4b4f0d85] 2023-03-09T14:52:22.483 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x562e4b4f3f84] 2023-03-09T14:52:22.483 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fa0133791ca] 2023-03-09T14:52:22.483 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 13: clone() 2023-03-09T14:52:22.484 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-03-09T14:52:22.484 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[121618]: 2023-03-09T14:52:22.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:22 smithi181 ceph-mon[94189]: pgmap v254: 129 pgs: 1 remapped+peering, 2 active+recovering+degraded, 5 active+recovery_wait+degraded, 5 peering, 5 undersized+peered, 15 active+undersized, 4 undersized+degraded+peered, 9 active+undersized+degraded, 83 active+clean; 18 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 4.0 MiB/s wr, 453 op/s; 3626/37030 objects degraded (9.792%); 173 MiB/s, 16 keys/s, 44 objects/s recovering 2023-03-09T14:52:22.485 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:22 smithi181 ceph-mon[94189]: Health check update: Reduced data availability: 4 pgs inactive (PG_AVAILABILITY) 2023-03-09T14:52:22.485 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:22 smithi181 ceph-mon[94189]: osdmap e160: 12 total, 11 up, 12 in 2023-03-09T14:52:22.485 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-mon[93869]: pgmap v254: 129 pgs: 1 remapped+peering, 2 active+recovering+degraded, 5 active+recovery_wait+degraded, 5 peering, 5 undersized+peered, 15 active+undersized, 4 undersized+degraded+peered, 9 active+undersized+degraded, 83 active+clean; 18 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 4.0 MiB/s wr, 453 op/s; 3626/37030 objects degraded (9.792%); 173 MiB/s, 16 keys/s, 44 objects/s recovering 2023-03-09T14:52:22.485 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-mon[93869]: Health check update: Reduced data availability: 4 pgs inactive (PG_AVAILABILITY) 2023-03-09T14:52:22.485 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-mon[93869]: osdmap e160: 12 total, 11 up, 12 in 2023-03-09T14:52:22.486 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 2023-03-09T14:52:22.188+0000 7f4f6b6b8700 -1 osd.7 pg_epoch: 160 pg[4.1es1( v 160'1564 (0'0,160'1564] local-lis/les=159/160 n=269 ec=85/85 lis/c=159/157 les/c/f=160/158/0 sis=159) [NONE,7,6,1]p7(1) r=1 lpr=159 pi=[157,159)/1 luod=160'1562 crt=160'1558 lcod 160'1561 mlcod 160'1561 active+undersized+degraded mbc={}] bad op order, already applied 89757 > this 87516 2023-03-09T14:52:22.486 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7f4f6b6b8700 time 2023-03-09T14:52:22.190022+0000 2023-03-09T14:52:22.487 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op") 2023-03-09T14:52:22.487 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:52:22.487 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x55f762036c85] 2023-03-09T14:52:22.487 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 2: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55f7622ef6e4] 2023-03-09T14:52:22.487 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55f7622f5d5e] 2023-03-09T14:52:22.487 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 4: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55f762145986] 2023-03-09T14:52:22.487 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 5: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55f76245b729] 2023-03-09T14:52:22.488 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55f76215aebf] 2023-03-09T14:52:22.488 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55f76280cd85] 2023-03-09T14:52:22.488 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55f76280ff84] 2023-03-09T14:52:22.488 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f4f9041c1ca] 2023-03-09T14:52:22.488 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 10: clone() 2023-03-09T14:52:22.488 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 2023-03-09T14:52:22.191+0000 7f4f6b6b8700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7f4f6b6b8700 time 2023-03-09T14:52:22.190022+0000 2023-03-09T14:52:22.488 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op") 2023-03-09T14:52:22.488 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 2023-03-09T14:52:22.489 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:52:22.489 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x55f762036c85] 2023-03-09T14:52:22.489 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 2: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55f7622ef6e4] 2023-03-09T14:52:22.489 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55f7622f5d5e] 2023-03-09T14:52:22.489 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 4: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55f762145986] 2023-03-09T14:52:22.489 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 5: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55f76245b729] 2023-03-09T14:52:22.489 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55f76215aebf] 2023-03-09T14:52:22.490 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55f76280cd85] 2023-03-09T14:52:22.490 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55f76280ff84] 2023-03-09T14:52:22.490 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f4f9041c1ca] 2023-03-09T14:52:22.490 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 10: clone() 2023-03-09T14:52:22.490 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 2023-03-09T14:52:22.490 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: *** Caught signal (Aborted) ** 2023-03-09T14:52:22.490 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: in thread 7f4f6b6b8700 thread_name:tp_osd_tp 2023-03-09T14:52:22.490 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:52:22.491 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f4f90426cf0] 2023-03-09T14:52:22.491 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 2: gsignal() 2023-03-09T14:52:22.491 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 3: abort() 2023-03-09T14:52:22.491 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x55f762036d45] 2023-03-09T14:52:22.491 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 5: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55f7622ef6e4] 2023-03-09T14:52:22.491 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 6: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55f7622f5d5e] 2023-03-09T14:52:22.491 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55f762145986] 2023-03-09T14:52:22.492 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 8: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55f76245b729] 2023-03-09T14:52:22.492 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55f76215aebf] 2023-03-09T14:52:22.492 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55f76280cd85] 2023-03-09T14:52:22.492 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55f76280ff84] 2023-03-09T14:52:22.492 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f4f9041c1ca] 2023-03-09T14:52:22.492 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 13: clone() 2023-03-09T14:52:22.492 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 2023-03-09T14:52:22.195+0000 7f4f6b6b8700 -1 *** Caught signal (Aborted) ** 2023-03-09T14:52:22.492 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: in thread 7f4f6b6b8700 thread_name:tp_osd_tp 2023-03-09T14:52:22.493 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 2023-03-09T14:52:22.493 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:52:22.493 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f4f90426cf0] 2023-03-09T14:52:22.493 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 2: gsignal() 2023-03-09T14:52:22.493 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 3: abort() 2023-03-09T14:52:22.493 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x55f762036d45] 2023-03-09T14:52:22.493 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 5: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55f7622ef6e4] 2023-03-09T14:52:22.494 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 6: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55f7622f5d5e] 2023-03-09T14:52:22.494 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55f762145986] 2023-03-09T14:52:22.494 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 8: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55f76245b729] 2023-03-09T14:52:22.494 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55f76215aebf] 2023-03-09T14:52:22.494 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55f76280cd85] 2023-03-09T14:52:22.494 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55f76280ff84] 2023-03-09T14:52:22.494 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f4f9041c1ca] 2023-03-09T14:52:22.494 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 13: clone() 2023-03-09T14:52:22.495 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-03-09T14:52:22.495 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 2023-03-09T14:52:22.495 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: -9> 2023-03-09T14:52:22.188+0000 7f4f6b6b8700 -1 osd.7 pg_epoch: 160 pg[4.1es1( v 160'1564 (0'0,160'1564] local-lis/les=159/160 n=269 ec=85/85 lis/c=159/157 les/c/f=160/158/0 sis=159) [NONE,7,6,1]p7(1) r=1 lpr=159 pi=[157,159)/1 luod=160'1562 crt=160'1558 lcod 160'1561 mlcod 160'1561 active+undersized+degraded mbc={}] bad op order, already applied 89757 > this 87516 2023-03-09T14:52:22.495 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: -1> 2023-03-09T14:52:22.191+0000 7f4f6b6b8700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7f4f6b6b8700 time 2023-03-09T14:52:22.190022+0000 2023-03-09T14:52:22.495 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op") 2023-03-09T14:52:22.495 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 2023-03-09T14:52:22.495 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:52:22.496 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x55f762036c85] 2023-03-09T14:52:22.496 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 2: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55f7622ef6e4] 2023-03-09T14:52:22.496 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55f7622f5d5e] 2023-03-09T14:52:22.496 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 4: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55f762145986] 2023-03-09T14:52:22.496 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 5: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55f76245b729] 2023-03-09T14:52:22.496 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55f76215aebf] 2023-03-09T14:52:22.496 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55f76280cd85] 2023-03-09T14:52:22.496 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55f76280ff84] 2023-03-09T14:52:22.497 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f4f9041c1ca] 2023-03-09T14:52:22.497 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 10: clone() 2023-03-09T14:52:22.497 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 2023-03-09T14:52:22.497 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 0> 2023-03-09T14:52:22.195+0000 7f4f6b6b8700 -1 *** Caught signal (Aborted) ** 2023-03-09T14:52:22.497 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: in thread 7f4f6b6b8700 thread_name:tp_osd_tp 2023-03-09T14:52:22.497 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 2023-03-09T14:52:22.497 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:52:22.498 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f4f90426cf0] 2023-03-09T14:52:22.498 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 2: gsignal() 2023-03-09T14:52:22.498 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 3: abort() 2023-03-09T14:52:22.498 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x55f762036d45] 2023-03-09T14:52:22.498 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 5: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55f7622ef6e4] 2023-03-09T14:52:22.498 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 6: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55f7622f5d5e] 2023-03-09T14:52:22.499 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55f762145986] 2023-03-09T14:52:22.499 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 8: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55f76245b729] 2023-03-09T14:52:22.499 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55f76215aebf] 2023-03-09T14:52:22.499 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55f76280cd85] 2023-03-09T14:52:22.499 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55f76280ff84] 2023-03-09T14:52:22.499 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f4f9041c1ca] 2023-03-09T14:52:22.499 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 13: clone() 2023-03-09T14:52:22.499 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-03-09T14:52:22.500 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 2023-03-09T14:52:22.500 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: -9> 2023-03-09T14:52:22.188+0000 7f4f6b6b8700 -1 osd.7 pg_epoch: 160 pg[4.1es1( v 160'1564 (0'0,160'1564] local-lis/les=159/160 n=269 ec=85/85 lis/c=159/157 les/c/f=160/158/0 sis=159) [NONE,7,6,1]p7(1) r=1 lpr=159 pi=[157,159)/1 luod=160'1562 crt=160'1558 lcod 160'1561 mlcod 160'1561 active+undersized+degraded mbc={}] bad op order, already applied 89757 > this 87516 2023-03-09T14:52:22.500 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: -1> 2023-03-09T14:52:22.191+0000 7f4f6b6b8700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7f4f6b6b8700 time 2023-03-09T14:52:22.190022+0000 2023-03-09T14:52:22.500 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op") 2023-03-09T14:52:22.500 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 2023-03-09T14:52:22.500 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:52:22.500 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x55f762036c85] 2023-03-09T14:52:22.501 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 2: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55f7622ef6e4] 2023-03-09T14:52:22.501 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55f7622f5d5e] 2023-03-09T14:52:22.501 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 4: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55f762145986] 2023-03-09T14:52:22.501 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 5: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55f76245b729] 2023-03-09T14:52:22.501 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55f76215aebf] 2023-03-09T14:52:22.501 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55f76280cd85] 2023-03-09T14:52:22.501 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55f76280ff84] 2023-03-09T14:52:22.502 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f4f9041c1ca] 2023-03-09T14:52:22.502 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 10: clone() 2023-03-09T14:52:22.502 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 2023-03-09T14:52:22.502 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 0> 2023-03-09T14:52:22.195+0000 7f4f6b6b8700 -1 *** Caught signal (Aborted) ** 2023-03-09T14:52:22.502 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: in thread 7f4f6b6b8700 thread_name:tp_osd_tp 2023-03-09T14:52:22.502 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 2023-03-09T14:52:22.502 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:52:22.503 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f4f90426cf0] 2023-03-09T14:52:22.503 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 2: gsignal() 2023-03-09T14:52:22.503 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 3: abort() 2023-03-09T14:52:22.503 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x55f762036d45] 2023-03-09T14:52:22.503 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 5: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55f7622ef6e4] 2023-03-09T14:52:22.504 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 6: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55f7622f5d5e] 2023-03-09T14:52:22.504 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55f762145986] 2023-03-09T14:52:22.504 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 8: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55f76245b729] 2023-03-09T14:52:22.504 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55f76215aebf] 2023-03-09T14:52:22.504 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55f76280cd85] 2023-03-09T14:52:22.504 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55f76280ff84] 2023-03-09T14:52:22.504 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f4f9041c1ca] 2023-03-09T14:52:22.504 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 13: clone() 2023-03-09T14:52:22.505 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-03-09T14:52:22.505 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[109811]: 2023-03-09T14:52:22.750 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:22 smithi038 systemd[1]: ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.9.service: Main process exited, code=exited, status=134/n/a 2023-03-09T14:52:23.356 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.7 reported immediately failed by osd.11 2023-03-09T14:52:23.356 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.7 failed (root=default,host=smithi122) (connection refused reported by osd.11) 2023-03-09T14:52:23.356 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.7 reported immediately failed by osd.11 2023-03-09T14:52:23.356 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.7 reported immediately failed by osd.5 2023-03-09T14:52:23.356 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.7 reported immediately failed by osd.6 2023-03-09T14:52:23.357 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.7 reported immediately failed by osd.3 2023-03-09T14:52:23.357 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.7 reported immediately failed by osd.1 2023-03-09T14:52:23.357 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.7 reported immediately failed by osd.2 2023-03-09T14:52:23.357 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.7 reported immediately failed by osd.6 2023-03-09T14:52:23.357 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.7 reported immediately failed by osd.3 2023-03-09T14:52:23.357 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.7 reported immediately failed by osd.1 2023-03-09T14:52:23.357 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.7 reported immediately failed by osd.8 2023-03-09T14:52:23.358 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.7 reported immediately failed by osd.5 2023-03-09T14:52:23.358 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.7 reported immediately failed by osd.2 2023-03-09T14:52:23.358 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.7 reported immediately failed by osd.8 2023-03-09T14:52:23.358 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.7 reported immediately failed by osd.10 2023-03-09T14:52:23.358 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.7 reported immediately failed by osd.4 2023-03-09T14:52:23.358 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.7 reported immediately failed by osd.10 2023-03-09T14:52:23.358 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.7 reported immediately failed by osd.4 2023-03-09T14:52:23.359 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.9 reported immediately failed by osd.1 2023-03-09T14:52:23.359 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.9 failed (root=default,host=smithi038) (connection refused reported by osd.1) 2023-03-09T14:52:23.359 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.9 reported immediately failed by osd.6 2023-03-09T14:52:23.359 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.9 reported immediately failed by osd.3 2023-03-09T14:52:23.359 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.9 reported immediately failed by osd.6 2023-03-09T14:52:23.359 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.9 reported immediately failed by osd.11 2023-03-09T14:52:23.359 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.9 reported immediately failed by osd.11 2023-03-09T14:52:23.359 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.9 reported immediately failed by osd.5 2023-03-09T14:52:23.360 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.9 reported immediately failed by osd.2 2023-03-09T14:52:23.360 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.9 reported immediately failed by osd.3 2023-03-09T14:52:23.360 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.9 reported immediately failed by osd.1 2023-03-09T14:52:23.360 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.9 reported immediately failed by osd.8 2023-03-09T14:52:23.360 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.9 reported immediately failed by osd.5 2023-03-09T14:52:23.360 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.9 reported immediately failed by osd.2 2023-03-09T14:52:23.360 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.9 reported immediately failed by osd.8 2023-03-09T14:52:23.360 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.9 reported immediately failed by osd.5 2023-03-09T14:52:23.361 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.9 reported immediately failed by osd.10 2023-03-09T14:52:23.361 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.9 reported immediately failed by osd.4 2023-03-09T14:52:23.361 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.9 reported immediately failed by osd.10 2023-03-09T14:52:23.361 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.9 reported immediately failed by osd.4 2023-03-09T14:52:23.361 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.9 reported immediately failed by osd.4 2023-03-09T14:52:23.361 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: Health check update: Degraded data redundancy: 3626/37030 objects degraded (9.792%), 20 pgs degraded (PG_DEGRADED) 2023-03-09T14:52:23.361 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.9 reported immediately failed by osd.6 2023-03-09T14:52:23.362 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.9 reported immediately failed by osd.10 2023-03-09T14:52:23.362 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.7 reported immediately failed by osd.2 2023-03-09T14:52:23.362 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.9 reported immediately failed by osd.2 2023-03-09T14:52:23.362 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.9 reported immediately failed by osd.8 2023-03-09T14:52:23.362 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.7 reported immediately failed by osd.11 2023-03-09T14:52:23.362 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.7 reported immediately failed by osd.11 2023-03-09T14:52:23.362 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.7 reported immediately failed by osd.2 2023-03-09T14:52:23.363 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.7 reported immediately failed by osd.2 2023-03-09T14:52:23.363 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.7 reported immediately failed by osd.6 2023-03-09T14:52:23.363 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.7 reported immediately failed by osd.3 2023-03-09T14:52:23.363 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.7 reported immediately failed by osd.1 2023-03-09T14:52:23.363 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.7 reported immediately failed by osd.8 2023-03-09T14:52:23.363 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.7 reported immediately failed by osd.6 2023-03-09T14:52:23.363 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.7 reported immediately failed by osd.3 2023-03-09T14:52:23.363 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.7 reported immediately failed by osd.1 2023-03-09T14:52:23.364 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.7 reported immediately failed by osd.5 2023-03-09T14:52:23.364 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.7 reported immediately failed by osd.8 2023-03-09T14:52:23.364 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.7 reported immediately failed by osd.5 2023-03-09T14:52:23.364 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.7 reported immediately failed by osd.10 2023-03-09T14:52:23.364 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.7 reported immediately failed by osd.4 2023-03-09T14:52:23.364 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.7 reported immediately failed by osd.10 2023-03-09T14:52:23.364 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.7 reported immediately failed by osd.4 2023-03-09T14:52:23.364 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.9 reported immediately failed by osd.1 2023-03-09T14:52:23.365 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.9 reported immediately failed by osd.3 2023-03-09T14:52:23.365 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.9 reported immediately failed by osd.6 2023-03-09T14:52:23.365 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.9 reported immediately failed by osd.6 2023-03-09T14:52:23.365 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.9 reported immediately failed by osd.11 2023-03-09T14:52:23.365 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.9 reported immediately failed by osd.11 2023-03-09T14:52:23.366 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.9 reported immediately failed by osd.2 2023-03-09T14:52:23.366 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.9 reported immediately failed by osd.8 2023-03-09T14:52:23.366 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.9 reported immediately failed by osd.3 2023-03-09T14:52:23.366 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.9 reported immediately failed by osd.1 2023-03-09T14:52:23.367 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.9 reported immediately failed by osd.5 2023-03-09T14:52:23.367 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.9 reported immediately failed by osd.2 2023-03-09T14:52:23.367 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.9 reported immediately failed by osd.8 2023-03-09T14:52:23.367 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.9 reported immediately failed by osd.5 2023-03-09T14:52:23.368 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.9 reported immediately failed by osd.5 2023-03-09T14:52:23.368 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.9 reported immediately failed by osd.4 2023-03-09T14:52:23.368 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.9 reported immediately failed by osd.4 2023-03-09T14:52:23.368 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.9 reported immediately failed by osd.4 2023-03-09T14:52:23.369 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.9 reported immediately failed by osd.10 2023-03-09T14:52:23.369 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.9 reported immediately failed by osd.10 2023-03-09T14:52:23.369 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.9 reported immediately failed by osd.6 2023-03-09T14:52:23.370 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.9 reported immediately failed by osd.10 2023-03-09T14:52:23.370 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.9 reported immediately failed by osd.2 2023-03-09T14:52:23.370 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:23 smithi038 ceph-mon[92422]: osd.9 reported immediately failed by osd.8 2023-03-09T14:52:23.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.7 reported immediately failed by osd.11 2023-03-09T14:52:23.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.7 failed (root=default,host=smithi122) (connection refused reported by osd.11) 2023-03-09T14:52:23.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.7 reported immediately failed by osd.11 2023-03-09T14:52:23.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.7 reported immediately failed by osd.5 2023-03-09T14:52:23.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.7 reported immediately failed by osd.6 2023-03-09T14:52:23.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.7 reported immediately failed by osd.3 2023-03-09T14:52:23.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.7 reported immediately failed by osd.1 2023-03-09T14:52:23.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.7 reported immediately failed by osd.2 2023-03-09T14:52:23.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.7 reported immediately failed by osd.6 2023-03-09T14:52:23.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.7 reported immediately failed by osd.3 2023-03-09T14:52:23.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.7 reported immediately failed by osd.1 2023-03-09T14:52:23.481 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.7 reported immediately failed by osd.8 2023-03-09T14:52:23.481 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.7 reported immediately failed by osd.5 2023-03-09T14:52:23.481 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.7 reported immediately failed by osd.2 2023-03-09T14:52:23.482 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.7 reported immediately failed by osd.8 2023-03-09T14:52:23.482 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.7 reported immediately failed by osd.10 2023-03-09T14:52:23.482 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.7 reported immediately failed by osd.4 2023-03-09T14:52:23.482 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.7 reported immediately failed by osd.10 2023-03-09T14:52:23.483 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.7 reported immediately failed by osd.4 2023-03-09T14:52:23.483 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.9 reported immediately failed by osd.1 2023-03-09T14:52:23.483 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.9 failed (root=default,host=smithi038) (connection refused reported by osd.1) 2023-03-09T14:52:23.483 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.9 reported immediately failed by osd.6 2023-03-09T14:52:23.484 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.9 reported immediately failed by osd.3 2023-03-09T14:52:23.484 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.9 reported immediately failed by osd.6 2023-03-09T14:52:23.484 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.9 reported immediately failed by osd.11 2023-03-09T14:52:23.485 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.9 reported immediately failed by osd.11 2023-03-09T14:52:23.485 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.9 reported immediately failed by osd.5 2023-03-09T14:52:23.485 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.9 reported immediately failed by osd.2 2023-03-09T14:52:23.486 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.9 reported immediately failed by osd.3 2023-03-09T14:52:23.486 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.9 reported immediately failed by osd.1 2023-03-09T14:52:23.486 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.9 reported immediately failed by osd.8 2023-03-09T14:52:23.486 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.9 reported immediately failed by osd.5 2023-03-09T14:52:23.487 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.9 reported immediately failed by osd.2 2023-03-09T14:52:23.487 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.9 reported immediately failed by osd.8 2023-03-09T14:52:23.487 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.9 reported immediately failed by osd.5 2023-03-09T14:52:23.488 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.9 reported immediately failed by osd.10 2023-03-09T14:52:23.488 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.9 reported immediately failed by osd.4 2023-03-09T14:52:23.488 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.9 reported immediately failed by osd.10 2023-03-09T14:52:23.488 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.9 reported immediately failed by osd.4 2023-03-09T14:52:23.488 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.9 reported immediately failed by osd.4 2023-03-09T14:52:23.489 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: Health check update: Degraded data redundancy: 3626/37030 objects degraded (9.792%), 20 pgs degraded (PG_DEGRADED) 2023-03-09T14:52:23.489 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.9 reported immediately failed by osd.6 2023-03-09T14:52:23.489 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.9 reported immediately failed by osd.10 2023-03-09T14:52:23.489 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.7 reported immediately failed by osd.2 2023-03-09T14:52:23.490 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.9 reported immediately failed by osd.2 2023-03-09T14:52:23.490 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.9 reported immediately failed by osd.8 2023-03-09T14:52:23.490 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.7 reported immediately failed by osd.11 2023-03-09T14:52:23.490 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.7 reported immediately failed by osd.11 2023-03-09T14:52:23.491 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.7 reported immediately failed by osd.2 2023-03-09T14:52:23.491 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.7 reported immediately failed by osd.2 2023-03-09T14:52:23.491 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.7 reported immediately failed by osd.6 2023-03-09T14:52:23.491 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.7 reported immediately failed by osd.3 2023-03-09T14:52:23.491 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.7 reported immediately failed by osd.1 2023-03-09T14:52:23.492 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.7 reported immediately failed by osd.8 2023-03-09T14:52:23.492 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.7 reported immediately failed by osd.6 2023-03-09T14:52:23.492 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.7 reported immediately failed by osd.3 2023-03-09T14:52:23.492 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.7 reported immediately failed by osd.1 2023-03-09T14:52:23.492 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.7 reported immediately failed by osd.5 2023-03-09T14:52:23.492 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.7 reported immediately failed by osd.8 2023-03-09T14:52:23.492 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.7 reported immediately failed by osd.5 2023-03-09T14:52:23.493 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.7 reported immediately failed by osd.10 2023-03-09T14:52:23.493 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.7 reported immediately failed by osd.4 2023-03-09T14:52:23.493 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.7 reported immediately failed by osd.10 2023-03-09T14:52:23.493 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.7 reported immediately failed by osd.4 2023-03-09T14:52:23.493 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.9 reported immediately failed by osd.1 2023-03-09T14:52:23.493 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.9 reported immediately failed by osd.3 2023-03-09T14:52:23.493 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.9 reported immediately failed by osd.6 2023-03-09T14:52:23.494 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.9 reported immediately failed by osd.6 2023-03-09T14:52:23.494 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.9 reported immediately failed by osd.11 2023-03-09T14:52:23.494 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.9 reported immediately failed by osd.11 2023-03-09T14:52:23.494 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.9 reported immediately failed by osd.2 2023-03-09T14:52:23.494 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.9 reported immediately failed by osd.8 2023-03-09T14:52:23.494 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.9 reported immediately failed by osd.3 2023-03-09T14:52:23.494 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.9 reported immediately failed by osd.1 2023-03-09T14:52:23.494 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.9 reported immediately failed by osd.5 2023-03-09T14:52:23.495 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.9 reported immediately failed by osd.2 2023-03-09T14:52:23.495 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.9 reported immediately failed by osd.8 2023-03-09T14:52:23.495 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.9 reported immediately failed by osd.5 2023-03-09T14:52:23.495 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.9 reported immediately failed by osd.5 2023-03-09T14:52:23.495 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.9 reported immediately failed by osd.4 2023-03-09T14:52:23.495 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.9 reported immediately failed by osd.4 2023-03-09T14:52:23.495 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.9 reported immediately failed by osd.4 2023-03-09T14:52:23.495 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.9 reported immediately failed by osd.10 2023-03-09T14:52:23.496 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.9 reported immediately failed by osd.10 2023-03-09T14:52:23.496 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.9 reported immediately failed by osd.6 2023-03-09T14:52:23.496 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.9 reported immediately failed by osd.10 2023-03-09T14:52:23.496 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.9 reported immediately failed by osd.2 2023-03-09T14:52:23.496 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:23 smithi122 ceph-mon[93869]: osd.9 reported immediately failed by osd.8 2023-03-09T14:52:23.497 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.7 reported immediately failed by osd.11 2023-03-09T14:52:23.497 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.7 failed (root=default,host=smithi122) (connection refused reported by osd.11) 2023-03-09T14:52:23.497 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.7 reported immediately failed by osd.11 2023-03-09T14:52:23.497 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.7 reported immediately failed by osd.5 2023-03-09T14:52:23.497 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.7 reported immediately failed by osd.6 2023-03-09T14:52:23.498 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.7 reported immediately failed by osd.3 2023-03-09T14:52:23.498 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.7 reported immediately failed by osd.1 2023-03-09T14:52:23.498 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.7 reported immediately failed by osd.2 2023-03-09T14:52:23.498 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.7 reported immediately failed by osd.6 2023-03-09T14:52:23.498 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.7 reported immediately failed by osd.3 2023-03-09T14:52:23.498 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.7 reported immediately failed by osd.1 2023-03-09T14:52:23.498 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.7 reported immediately failed by osd.8 2023-03-09T14:52:23.498 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.7 reported immediately failed by osd.5 2023-03-09T14:52:23.499 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.7 reported immediately failed by osd.2 2023-03-09T14:52:23.499 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.7 reported immediately failed by osd.8 2023-03-09T14:52:23.499 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.7 reported immediately failed by osd.10 2023-03-09T14:52:23.499 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.7 reported immediately failed by osd.4 2023-03-09T14:52:23.499 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.7 reported immediately failed by osd.10 2023-03-09T14:52:23.499 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.7 reported immediately failed by osd.4 2023-03-09T14:52:23.499 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.9 reported immediately failed by osd.1 2023-03-09T14:52:23.500 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.9 failed (root=default,host=smithi038) (connection refused reported by osd.1) 2023-03-09T14:52:23.500 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.9 reported immediately failed by osd.6 2023-03-09T14:52:23.500 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.9 reported immediately failed by osd.3 2023-03-09T14:52:23.500 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.9 reported immediately failed by osd.6 2023-03-09T14:52:23.500 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.9 reported immediately failed by osd.11 2023-03-09T14:52:23.500 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.9 reported immediately failed by osd.11 2023-03-09T14:52:23.500 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.9 reported immediately failed by osd.5 2023-03-09T14:52:23.500 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.9 reported immediately failed by osd.2 2023-03-09T14:52:23.501 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.9 reported immediately failed by osd.3 2023-03-09T14:52:23.501 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.9 reported immediately failed by osd.1 2023-03-09T14:52:23.501 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.9 reported immediately failed by osd.8 2023-03-09T14:52:23.501 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.9 reported immediately failed by osd.5 2023-03-09T14:52:23.501 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.9 reported immediately failed by osd.2 2023-03-09T14:52:23.501 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.9 reported immediately failed by osd.8 2023-03-09T14:52:23.501 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.9 reported immediately failed by osd.5 2023-03-09T14:52:23.501 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.9 reported immediately failed by osd.10 2023-03-09T14:52:23.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.9 reported immediately failed by osd.4 2023-03-09T14:52:23.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.9 reported immediately failed by osd.10 2023-03-09T14:52:23.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.9 reported immediately failed by osd.4 2023-03-09T14:52:23.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.9 reported immediately failed by osd.4 2023-03-09T14:52:23.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: Health check update: Degraded data redundancy: 3626/37030 objects degraded (9.792%), 20 pgs degraded (PG_DEGRADED) 2023-03-09T14:52:23.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.9 reported immediately failed by osd.6 2023-03-09T14:52:23.502 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.9 reported immediately failed by osd.10 2023-03-09T14:52:23.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.7 reported immediately failed by osd.2 2023-03-09T14:52:23.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.9 reported immediately failed by osd.2 2023-03-09T14:52:23.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.9 reported immediately failed by osd.8 2023-03-09T14:52:23.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.7 reported immediately failed by osd.11 2023-03-09T14:52:23.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.7 reported immediately failed by osd.11 2023-03-09T14:52:23.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.7 reported immediately failed by osd.2 2023-03-09T14:52:23.504 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.7 reported immediately failed by osd.2 2023-03-09T14:52:23.504 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.7 reported immediately failed by osd.6 2023-03-09T14:52:23.504 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.7 reported immediately failed by osd.3 2023-03-09T14:52:23.504 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.7 reported immediately failed by osd.1 2023-03-09T14:52:23.504 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.7 reported immediately failed by osd.8 2023-03-09T14:52:23.504 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.7 reported immediately failed by osd.6 2023-03-09T14:52:23.504 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.7 reported immediately failed by osd.3 2023-03-09T14:52:23.504 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.7 reported immediately failed by osd.1 2023-03-09T14:52:23.505 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.7 reported immediately failed by osd.5 2023-03-09T14:52:23.505 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.7 reported immediately failed by osd.8 2023-03-09T14:52:23.505 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.7 reported immediately failed by osd.5 2023-03-09T14:52:23.505 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.7 reported immediately failed by osd.10 2023-03-09T14:52:23.505 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.7 reported immediately failed by osd.4 2023-03-09T14:52:23.505 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.7 reported immediately failed by osd.10 2023-03-09T14:52:23.506 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.7 reported immediately failed by osd.4 2023-03-09T14:52:23.506 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.9 reported immediately failed by osd.1 2023-03-09T14:52:23.506 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.9 reported immediately failed by osd.3 2023-03-09T14:52:23.506 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.9 reported immediately failed by osd.6 2023-03-09T14:52:23.506 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.9 reported immediately failed by osd.6 2023-03-09T14:52:23.506 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.9 reported immediately failed by osd.11 2023-03-09T14:52:23.506 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.9 reported immediately failed by osd.11 2023-03-09T14:52:23.507 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.9 reported immediately failed by osd.2 2023-03-09T14:52:23.507 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.9 reported immediately failed by osd.8 2023-03-09T14:52:23.507 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.9 reported immediately failed by osd.3 2023-03-09T14:52:23.507 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.9 reported immediately failed by osd.1 2023-03-09T14:52:23.507 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.9 reported immediately failed by osd.5 2023-03-09T14:52:23.507 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.9 reported immediately failed by osd.2 2023-03-09T14:52:23.507 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.9 reported immediately failed by osd.8 2023-03-09T14:52:23.508 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.9 reported immediately failed by osd.5 2023-03-09T14:52:23.508 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.9 reported immediately failed by osd.5 2023-03-09T14:52:23.508 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.9 reported immediately failed by osd.4 2023-03-09T14:52:23.508 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.9 reported immediately failed by osd.4 2023-03-09T14:52:23.508 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.9 reported immediately failed by osd.4 2023-03-09T14:52:23.508 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.9 reported immediately failed by osd.10 2023-03-09T14:52:23.508 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.9 reported immediately failed by osd.10 2023-03-09T14:52:23.508 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.9 reported immediately failed by osd.6 2023-03-09T14:52:23.509 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.9 reported immediately failed by osd.10 2023-03-09T14:52:23.509 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.9 reported immediately failed by osd.2 2023-03-09T14:52:23.509 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:23 smithi181 ceph-mon[94189]: osd.9 reported immediately failed by osd.8 2023-03-09T14:52:23.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 115.0 s, 6.0 tps, lat 30817.088 ms stddev 5718.205 2023-03-09T14:52:24.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:24 smithi122 ceph-mon[93869]: pgmap v257: 129 pgs: 1 remapped+peering, 2 active+recovering+degraded, 5 active+recovery_wait+degraded, 8 peering, 5 undersized+peered, 14 active+undersized, 2 undersized+degraded+peered, 6 active+undersized+degraded, 86 active+clean; 18 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 3.9 MiB/s wr, 440 op/s; 2251/37218 objects degraded (6.048%); 169 MiB/s, 25 keys/s, 44 objects/s recovering 2023-03-09T14:52:24.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:24 smithi122 ceph-mon[93869]: osdmap e161: 12 total, 9 up, 12 in 2023-03-09T14:52:24.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:24 smithi181 ceph-mon[94189]: pgmap v257: 129 pgs: 1 remapped+peering, 2 active+recovering+degraded, 5 active+recovery_wait+degraded, 8 peering, 5 undersized+peered, 14 active+undersized, 2 undersized+degraded+peered, 6 active+undersized+degraded, 86 active+clean; 18 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 3.9 MiB/s wr, 440 op/s; 2251/37218 objects degraded (6.048%); 169 MiB/s, 25 keys/s, 44 objects/s recovering 2023-03-09T14:52:24.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:24 smithi181 ceph-mon[94189]: osdmap e161: 12 total, 9 up, 12 in 2023-03-09T14:52:24.500 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:24 smithi038 systemd[1]: ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.9.service: Failed with result 'exit-code'. 2023-03-09T14:52:24.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:24 smithi038 ceph-mon[92422]: pgmap v257: 129 pgs: 1 remapped+peering, 2 active+recovering+degraded, 5 active+recovery_wait+degraded, 8 peering, 5 undersized+peered, 14 active+undersized, 2 undersized+degraded+peered, 6 active+undersized+degraded, 86 active+clean; 18 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 3.9 MiB/s wr, 440 op/s; 2251/37218 objects degraded (6.048%); 169 MiB/s, 25 keys/s, 44 objects/s recovering 2023-03-09T14:52:24.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:24 smithi038 ceph-mon[92422]: osdmap e161: 12 total, 9 up, 12 in 2023-03-09T14:52:25.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:25 smithi122 ceph-mon[93869]: osdmap e162: 12 total, 9 up, 12 in 2023-03-09T14:52:25.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:25 smithi122 ceph-mon[93869]: pgmap v260: 129 pgs: 1 stale+undersized+degraded+peered, 3 stale+peering, 14 stale+active+clean, 2 remapped+peering, 2 active+recovering+degraded, 5 active+recovery_wait+degraded, 1 down, 7 peering, 1 undersized+peered, 12 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+undersized+degraded, 74 active+clean; 19 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 1.6 MiB/s wr, 208 op/s; 1698/37330 objects degraded (4.549%); 3.1 MiB/s, 8 keys/s, 1 objects/s recovering 2023-03-09T14:52:25.479 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:25 smithi122 systemd[1]: ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.7.service: Main process exited, code=exited, status=134/n/a 2023-03-09T14:52:25.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:25 smithi181 ceph-mon[94189]: osdmap e162: 12 total, 9 up, 12 in 2023-03-09T14:52:25.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:25 smithi181 ceph-mon[94189]: pgmap v260: 129 pgs: 1 stale+undersized+degraded+peered, 3 stale+peering, 14 stale+active+clean, 2 remapped+peering, 2 active+recovering+degraded, 5 active+recovery_wait+degraded, 1 down, 7 peering, 1 undersized+peered, 12 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+undersized+degraded, 74 active+clean; 19 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 1.6 MiB/s wr, 208 op/s; 1698/37330 objects degraded (4.549%); 3.1 MiB/s, 8 keys/s, 1 objects/s recovering 2023-03-09T14:52:25.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:25 smithi038 ceph-mon[92422]: osdmap e162: 12 total, 9 up, 12 in 2023-03-09T14:52:25.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:25 smithi038 ceph-mon[92422]: pgmap v260: 129 pgs: 1 stale+undersized+degraded+peered, 3 stale+peering, 14 stale+active+clean, 2 remapped+peering, 2 active+recovering+degraded, 5 active+recovery_wait+degraded, 1 down, 7 peering, 1 undersized+peered, 12 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 6 active+undersized+degraded, 74 active+clean; 19 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 1.6 MiB/s wr, 208 op/s; 1698/37330 objects degraded (4.549%); 3.1 MiB/s, 8 keys/s, 1 objects/s recovering 2023-03-09T14:52:26.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:26 smithi122 ceph-mon[93869]: osdmap e163: 12 total, 9 up, 12 in 2023-03-09T14:52:26.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:26 smithi181 ceph-mon[94189]: osdmap e163: 12 total, 9 up, 12 in 2023-03-09T14:52:26.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:26 smithi038 ceph-mon[92422]: osdmap e163: 12 total, 9 up, 12 in 2023-03-09T14:52:27.452 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:52:27.453 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:27 smithi122 ceph-mon[93869]: pgmap v262: 129 pgs: 1 stale+undersized+degraded+peered, 3 stale+peering, 5 stale+active+clean, 2 remapped+peering, 1 down, 6 peering, 1 active+recovering+undersized+degraded, 2 undersized+peered, 26 active+undersized, 4 undersized+degraded+peered, 1 active+recovery_wait+undersized+degraded+remapped, 29 active+undersized+degraded, 48 active+clean; 19 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 1.7 MiB/s wr, 213 op/s; 6103/37438 objects degraded (16.302%); 111 MiB/s, 27 objects/s recovering 2023-03-09T14:52:27.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:52:27.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:27 smithi181 ceph-mon[94189]: pgmap v262: 129 pgs: 1 stale+undersized+degraded+peered, 3 stale+peering, 5 stale+active+clean, 2 remapped+peering, 1 down, 6 peering, 1 active+recovering+undersized+degraded, 2 undersized+peered, 26 active+undersized, 4 undersized+degraded+peered, 1 active+recovery_wait+undersized+degraded+remapped, 29 active+undersized+degraded, 48 active+clean; 19 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 1.7 MiB/s wr, 213 op/s; 6103/37438 objects degraded (16.302%); 111 MiB/s, 27 objects/s recovering 2023-03-09T14:52:27.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:52:27.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:27 smithi038 ceph-mon[92422]: pgmap v262: 129 pgs: 1 stale+undersized+degraded+peered, 3 stale+peering, 5 stale+active+clean, 2 remapped+peering, 1 down, 6 peering, 1 active+recovering+undersized+degraded, 2 undersized+peered, 26 active+undersized, 4 undersized+degraded+peered, 1 active+recovery_wait+undersized+degraded+remapped, 29 active+undersized+degraded, 48 active+clean; 19 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 1.7 MiB/s wr, 213 op/s; 6103/37438 objects degraded (16.302%); 111 MiB/s, 27 objects/s recovering 2023-03-09T14:52:28.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:28 smithi122 ceph-mon[93869]: Health check update: Reduced data availability: 4 pgs inactive, 1 pg down, 2 pgs peering (PG_AVAILABILITY) 2023-03-09T14:52:28.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:28 smithi122 ceph-mon[93869]: Health check update: 3 osds down (OSD_DOWN) 2023-03-09T14:52:28.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:28 smithi122 ceph-mon[93869]: Health check update: Degraded data redundancy: 6103/37438 objects degraded (16.302%), 36 pgs degraded (PG_DEGRADED) 2023-03-09T14:52:28.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:28 smithi181 ceph-mon[94189]: Health check update: Reduced data availability: 4 pgs inactive, 1 pg down, 2 pgs peering (PG_AVAILABILITY) 2023-03-09T14:52:28.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:28 smithi181 ceph-mon[94189]: Health check update: 3 osds down (OSD_DOWN) 2023-03-09T14:52:28.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:28 smithi181 ceph-mon[94189]: Health check update: Degraded data redundancy: 6103/37438 objects degraded (16.302%), 36 pgs degraded (PG_DEGRADED) 2023-03-09T14:52:28.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:28 smithi038 ceph-mon[92422]: Health check update: Reduced data availability: 4 pgs inactive, 1 pg down, 2 pgs peering (PG_AVAILABILITY) 2023-03-09T14:52:28.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:28 smithi038 ceph-mon[92422]: Health check update: 3 osds down (OSD_DOWN) 2023-03-09T14:52:28.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:28 smithi038 ceph-mon[92422]: Health check update: Degraded data redundancy: 6103/37438 objects degraded (16.302%), 36 pgs degraded (PG_DEGRADED) 2023-03-09T14:52:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 120.0 s, 0.2 tps, lat 1948.883 ms stddev -nan 2023-03-09T14:52:28.978 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:28 smithi122 systemd[1]: ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.7.service: Failed with result 'exit-code'. 2023-03-09T14:52:29.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:52:29.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:52:29.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:52:29.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:29 smithi122 ceph-mon[93869]: pgmap v263: 129 pgs: 1 remapped+peering, 1 down, 6 peering, 2 active+recovering+undersized+degraded, 6 undersized+peered, 29 active+undersized, 9 undersized+degraded+peered, 35 active+undersized+degraded, 40 active+clean; 19 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 1024 KiB/s wr, 123 op/s; 8658/37486 objects degraded (23.097%); 158 MiB/s, 39 objects/s recovering 2023-03-09T14:52:29.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:52:29.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:52:29.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:52:29.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:29 smithi181 ceph-mon[94189]: pgmap v263: 129 pgs: 1 remapped+peering, 1 down, 6 peering, 2 active+recovering+undersized+degraded, 6 undersized+peered, 29 active+undersized, 9 undersized+degraded+peered, 35 active+undersized+degraded, 40 active+clean; 19 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 1024 KiB/s wr, 123 op/s; 8658/37486 objects degraded (23.097%); 158 MiB/s, 39 objects/s recovering 2023-03-09T14:52:29.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:52:29.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:52:29.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:52:29.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:29 smithi038 ceph-mon[92422]: pgmap v263: 129 pgs: 1 remapped+peering, 1 down, 6 peering, 2 active+recovering+undersized+degraded, 6 undersized+peered, 29 active+undersized, 9 undersized+degraded+peered, 35 active+undersized+degraded, 40 active+clean; 19 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 1024 KiB/s wr, 123 op/s; 8658/37486 objects degraded (23.097%); 158 MiB/s, 39 objects/s recovering 2023-03-09T14:52:32.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:31 smithi038 ceph-mon[92422]: pgmap v264: 129 pgs: 1 down, 1 active+recovery_wait+undersized+degraded, 1 active+recovering+undersized+degraded, 6 undersized+peered, 32 active+undersized, 10 undersized+degraded+peered, 38 active+undersized+degraded, 40 active+clean; 19 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 600 KiB/s wr, 74 op/s; 9976/37490 objects degraded (26.610%); 140 MiB/s, 34 objects/s recovering 2023-03-09T14:52:32.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:31 smithi122 ceph-mon[93869]: pgmap v264: 129 pgs: 1 down, 1 active+recovery_wait+undersized+degraded, 1 active+recovering+undersized+degraded, 6 undersized+peered, 32 active+undersized, 10 undersized+degraded+peered, 38 active+undersized+degraded, 40 active+clean; 19 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 600 KiB/s wr, 74 op/s; 9976/37490 objects degraded (26.610%); 140 MiB/s, 34 objects/s recovering 2023-03-09T14:52:32.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:31 smithi181 ceph-mon[94189]: pgmap v264: 129 pgs: 1 down, 1 active+recovery_wait+undersized+degraded, 1 active+recovering+undersized+degraded, 6 undersized+peered, 32 active+undersized, 10 undersized+degraded+peered, 38 active+undersized+degraded, 40 active+clean; 19 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 600 KiB/s wr, 74 op/s; 9976/37490 objects degraded (26.610%); 140 MiB/s, 34 objects/s recovering 2023-03-09T14:52:32.682 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:32 smithi038 systemd[1]: ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.0.service: Service RestartSec=10s expired, scheduling restart. 2023-03-09T14:52:32.682 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:32 smithi038 systemd[1]: ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.0.service: Scheduled restart job, restart counter is at 2. 2023-03-09T14:52:32.683 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:32 smithi038 systemd[1]: Stopped Ceph osd.0 for c6733fe2-be87-11ed-9af2-001a4aab830c. 2023-03-09T14:52:32.683 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:32 smithi038 systemd[1]: Starting Ceph osd.0 for c6733fe2-be87-11ed-9af2-001a4aab830c... 2023-03-09T14:52:33.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:32 smithi038 ceph-mon[92422]: Health check update: Reduced data availability: 2 pgs inactive, 1 pg down (PG_AVAILABILITY) 2023-03-09T14:52:33.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:32 smithi038 ceph-mon[92422]: Health check update: Degraded data redundancy: 9976/37490 objects degraded (26.610%), 50 pgs degraded (PG_DEGRADED) 2023-03-09T14:52:33.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:32 smithi122 ceph-mon[93869]: Health check update: Reduced data availability: 2 pgs inactive, 1 pg down (PG_AVAILABILITY) 2023-03-09T14:52:33.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:32 smithi122 ceph-mon[93869]: Health check update: Degraded data redundancy: 9976/37490 objects degraded (26.610%), 50 pgs degraded (PG_DEGRADED) 2023-03-09T14:52:33.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:32 smithi181 ceph-mon[94189]: Health check update: Reduced data availability: 2 pgs inactive, 1 pg down (PG_AVAILABILITY) 2023-03-09T14:52:33.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:32 smithi181 ceph-mon[94189]: Health check update: Degraded data redundancy: 9976/37490 objects degraded (26.610%), 50 pgs degraded (PG_DEGRADED) 2023-03-09T14:52:33.741 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:33 smithi038 bash[153905]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-03-09T14:52:33.741 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:33 smithi038 bash[153905]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2023-03-09T14:52:33.741 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:33 smithi038 bash[153905]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2023-03-09T14:52:33.741 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:33 smithi038 bash[153905]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2023-03-09T14:52:33.742 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:33 smithi038 bash[153905]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2023-03-09T14:52:33.742 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:33 smithi038 bash[153905]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-03-09T14:52:33.742 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:33 smithi038 bash[153905]: --> ceph-volume raw activate successful for osd ID: 0 2023-03-09T14:52:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 125.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T14:52:34.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:33 smithi038 ceph-mon[92422]: pgmap v265: 129 pgs: 1 down, 1 active+recovery_wait+undersized+degraded, 1 active+recovering+undersized+degraded, 6 undersized+peered, 32 active+undersized, 10 undersized+degraded+peered, 38 active+undersized+degraded, 40 active+clean; 19 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 386 KiB/s wr, 48 op/s; 9977/37494 objects degraded (26.610%); 123 MiB/s, 30 objects/s recovering 2023-03-09T14:52:34.001 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:33 smithi038 podman[154163]: 2023-03-09T14:52:34.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:33 smithi122 ceph-mon[93869]: pgmap v265: 129 pgs: 1 down, 1 active+recovery_wait+undersized+degraded, 1 active+recovering+undersized+degraded, 6 undersized+peered, 32 active+undersized, 10 undersized+degraded+peered, 38 active+undersized+degraded, 40 active+clean; 19 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 386 KiB/s wr, 48 op/s; 9977/37494 objects degraded (26.610%); 123 MiB/s, 30 objects/s recovering 2023-03-09T14:52:34.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:33 smithi181 ceph-mon[94189]: pgmap v265: 129 pgs: 1 down, 1 active+recovery_wait+undersized+degraded, 1 active+recovering+undersized+degraded, 6 undersized+peered, 32 active+undersized, 10 undersized+degraded+peered, 38 active+undersized+degraded, 40 active+clean; 19 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 386 KiB/s wr, 48 op/s; 9977/37494 objects degraded (26.610%); 123 MiB/s, 30 objects/s recovering 2023-03-09T14:52:34.501 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:34 smithi038 bash[154163]: 880015feaa0ea070de19a351c9a510300cf2623325bc888f31d8bd8aa782a96f 2023-03-09T14:52:34.501 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:34 smithi038 ceph-osd[154203]: -- 172.21.15.38:0/2105718085 <== mon.0 v2:172.21.15.38:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a71f8289c0 con 0x55a71f7f0c00 2023-03-09T14:52:34.501 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:34 smithi038 systemd[1]: Started Ceph osd.0 for c6733fe2-be87-11ed-9af2-001a4aab830c. 2023-03-09T14:52:34.502 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:34 smithi038 systemd[1]: ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.9.service: Service RestartSec=10s expired, scheduling restart. 2023-03-09T14:52:34.502 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:34 smithi038 systemd[1]: ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.9.service: Scheduled restart job, restart counter is at 1. 2023-03-09T14:52:34.502 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:34 smithi038 systemd[1]: Stopped Ceph osd.9 for c6733fe2-be87-11ed-9af2-001a4aab830c. 2023-03-09T14:52:34.502 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:34 smithi038 systemd[1]: Starting Ceph osd.9 for c6733fe2-be87-11ed-9af2-001a4aab830c... 2023-03-09T14:52:35.000 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:34 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[154179]: 2023-03-09T14:52:34.667+0000 7ff873ec6640 -1 Falling back to public interface 2023-03-09T14:52:35.500 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:35 smithi038 bash[154271]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-9 2023-03-09T14:52:35.501 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:35 smithi038 bash[154271]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-9 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2023-03-09T14:52:35.501 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:35 smithi038 bash[154271]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2023-03-09T14:52:35.501 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:35 smithi038 bash[154271]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2023-03-09T14:52:35.501 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:35 smithi038 bash[154271]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-9/block 2023-03-09T14:52:35.501 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:35 smithi038 bash[154271]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-9 2023-03-09T14:52:35.501 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:35 smithi038 bash[154271]: --> ceph-volume raw activate successful for osd ID: 9 2023-03-09T14:52:35.964 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:35 smithi038 ceph-mon[92422]: pgmap v266: 129 pgs: 1 down, 1 active+recovering+undersized+degraded, 6 undersized+peered, 32 active+undersized, 10 undersized+degraded+peered, 39 active+undersized+degraded, 40 active+clean; 19 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 331 KiB/s wr, 42 op/s; 9848/37494 objects degraded (26.266%); 156 MiB/s, 38 objects/s recovering 2023-03-09T14:52:35.965 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:35 smithi038 podman[154578]: 2023-03-09T14:52:35.965 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:35 smithi038 ceph-osd[154622]: -- 172.21.15.38:0/392107677 <== mon.0 v2:172.21.15.38:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559e4a0f89c0 con 0x559e4a0c1000 2023-03-09T14:52:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:35 smithi122 ceph-mon[93869]: pgmap v266: 129 pgs: 1 down, 1 active+recovering+undersized+degraded, 6 undersized+peered, 32 active+undersized, 10 undersized+degraded+peered, 39 active+undersized+degraded, 40 active+clean; 19 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 331 KiB/s wr, 42 op/s; 9848/37494 objects degraded (26.266%); 156 MiB/s, 38 objects/s recovering 2023-03-09T14:52:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:35 smithi181 ceph-mon[94189]: pgmap v266: 129 pgs: 1 down, 1 active+recovering+undersized+degraded, 6 undersized+peered, 32 active+undersized, 10 undersized+degraded+peered, 39 active+undersized+degraded, 40 active+clean; 19 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 331 KiB/s wr, 42 op/s; 9848/37494 objects degraded (26.266%); 156 MiB/s, 38 objects/s recovering 2023-03-09T14:52:36.250 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:35 smithi038 bash[154578]: 1972ecd88c7df8b932649cf69a267af8445ad4386af740a43bff9687da61c701 2023-03-09T14:52:36.250 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:36 smithi038 systemd[1]: Started Ceph osd.9 for c6733fe2-be87-11ed-9af2-001a4aab830c. 2023-03-09T14:52:36.751 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:36 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[154598]: 2023-03-09T14:52:36.476+0000 7f6ab05cc640 -1 Falling back to public interface 2023-03-09T14:52:38.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:37 smithi038 ceph-mon[92422]: pgmap v267: 129 pgs: 1 down, 1 active+recovering+undersized+degraded, 6 undersized+peered, 32 active+undersized, 10 undersized+degraded+peered, 39 active+undersized+degraded, 40 active+clean; 19 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 287 KiB/s wr, 37 op/s; 9848/37494 objects degraded (26.266%); 134 MiB/s, 33 objects/s recovering 2023-03-09T14:52:38.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:37 smithi038 ceph-mon[92422]: Health check update: Degraded data redundancy: 9848/37494 objects degraded (26.266%), 50 pgs degraded (PG_DEGRADED) 2023-03-09T14:52:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:37 smithi122 ceph-mon[93869]: pgmap v267: 129 pgs: 1 down, 1 active+recovering+undersized+degraded, 6 undersized+peered, 32 active+undersized, 10 undersized+degraded+peered, 39 active+undersized+degraded, 40 active+clean; 19 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 287 KiB/s wr, 37 op/s; 9848/37494 objects degraded (26.266%); 134 MiB/s, 33 objects/s recovering 2023-03-09T14:52:38.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:37 smithi122 ceph-mon[93869]: Health check update: Degraded data redundancy: 9848/37494 objects degraded (26.266%), 50 pgs degraded (PG_DEGRADED) 2023-03-09T14:52:38.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:37 smithi181 ceph-mon[94189]: pgmap v267: 129 pgs: 1 down, 1 active+recovering+undersized+degraded, 6 undersized+peered, 32 active+undersized, 10 undersized+degraded+peered, 39 active+undersized+degraded, 40 active+clean; 19 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 287 KiB/s wr, 37 op/s; 9848/37494 objects degraded (26.266%); 134 MiB/s, 33 objects/s recovering 2023-03-09T14:52:38.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:37 smithi181 ceph-mon[94189]: Health check update: Degraded data redundancy: 9848/37494 objects degraded (26.266%), 50 pgs degraded (PG_DEGRADED) 2023-03-09T14:52:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 130.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T14:52:39.479 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:38 smithi122 systemd[1]: ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.7.service: Service RestartSec=10s expired, scheduling restart. 2023-03-09T14:52:39.479 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:38 smithi122 systemd[1]: ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.7.service: Scheduled restart job, restart counter is at 1. 2023-03-09T14:52:39.479 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:38 smithi122 systemd[1]: Stopped Ceph osd.7 for c6733fe2-be87-11ed-9af2-001a4aab830c. 2023-03-09T14:52:39.479 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:38 smithi122 systemd[1]: Starting Ceph osd.7 for c6733fe2-be87-11ed-9af2-001a4aab830c... 2023-03-09T14:52:40.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:39 smithi122 ceph-mon[93869]: pgmap v268: 129 pgs: 1 down, 1 active+recovering+undersized+degraded, 6 undersized+peered, 32 active+undersized, 10 undersized+degraded+peered, 39 active+undersized+degraded, 40 active+clean; 19 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 81 KiB/s wr, 15 op/s; 9848/37494 objects degraded (26.266%); 88 MiB/s, 22 objects/s recovering 2023-03-09T14:52:40.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:39 smithi181 ceph-mon[94189]: pgmap v268: 129 pgs: 1 down, 1 active+recovering+undersized+degraded, 6 undersized+peered, 32 active+undersized, 10 undersized+degraded+peered, 39 active+undersized+degraded, 40 active+clean; 19 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 81 KiB/s wr, 15 op/s; 9848/37494 objects degraded (26.266%); 88 MiB/s, 22 objects/s recovering 2023-03-09T14:52:40.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:39 smithi038 ceph-mon[92422]: pgmap v268: 129 pgs: 1 down, 1 active+recovering+undersized+degraded, 6 undersized+peered, 32 active+undersized, 10 undersized+degraded+peered, 39 active+undersized+degraded, 40 active+clean; 19 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 81 KiB/s wr, 15 op/s; 9848/37494 objects degraded (26.266%); 88 MiB/s, 22 objects/s recovering 2023-03-09T14:52:40.750 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:40 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[154179]: 2023-03-09T14:52:40.463+0000 7ff873ec6640 -1 osd.0 158 log_to_monitors true 2023-03-09T14:52:41.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:40 smithi122 ceph-mon[93869]: from='osd.0 [v2:172.21.15.38:6810/1721245517,v1:172.21.15.38:6811/1721245517]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-03-09T14:52:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:40 smithi181 ceph-mon[94189]: from='osd.0 [v2:172.21.15.38:6810/1721245517,v1:172.21.15.38:6811/1721245517]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-03-09T14:52:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:40 smithi038 ceph-mon[92422]: from='osd.0 [v2:172.21.15.38:6810/1721245517,v1:172.21.15.38:6811/1721245517]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-03-09T14:52:41.251 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 09 14:52:40 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-0[154179]: 2023-03-09T14:52:40.791+0000 7ff86243d700 -1 osd.0 158 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-09T14:52:41.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:41 smithi122 ceph-mon[93869]: pgmap v269: 129 pgs: 1 down, 6 undersized+peered, 32 active+undersized, 10 undersized+degraded+peered, 40 active+undersized+degraded, 40 active+clean; 19 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 3.2 KiB/s wr, 6 op/s; 9844/37494 objects degraded (26.255%); 54 MiB/s, 13 objects/s recovering 2023-03-09T14:52:41.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:41 smithi122 ceph-mon[93869]: from='osd.0 [v2:172.21.15.38:6810/1721245517,v1:172.21.15.38:6811/1721245517]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-03-09T14:52:41.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:41 smithi122 ceph-mon[93869]: osdmap e164: 12 total, 9 up, 12 in 2023-03-09T14:52:41.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:41 smithi122 ceph-mon[93869]: from='osd.0 [v2:172.21.15.38:6810/1721245517,v1:172.21.15.38:6811/1721245517]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2023-03-09T14:52:42.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:41 smithi181 ceph-mon[94189]: pgmap v269: 129 pgs: 1 down, 6 undersized+peered, 32 active+undersized, 10 undersized+degraded+peered, 40 active+undersized+degraded, 40 active+clean; 19 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 3.2 KiB/s wr, 6 op/s; 9844/37494 objects degraded (26.255%); 54 MiB/s, 13 objects/s recovering 2023-03-09T14:52:42.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:41 smithi181 ceph-mon[94189]: from='osd.0 [v2:172.21.15.38:6810/1721245517,v1:172.21.15.38:6811/1721245517]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-03-09T14:52:42.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:41 smithi181 ceph-mon[94189]: osdmap e164: 12 total, 9 up, 12 in 2023-03-09T14:52:42.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:41 smithi181 ceph-mon[94189]: from='osd.0 [v2:172.21.15.38:6810/1721245517,v1:172.21.15.38:6811/1721245517]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2023-03-09T14:52:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:41 smithi038 ceph-mon[92422]: pgmap v269: 129 pgs: 1 down, 6 undersized+peered, 32 active+undersized, 10 undersized+degraded+peered, 40 active+undersized+degraded, 40 active+clean; 19 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 3.2 KiB/s wr, 6 op/s; 9844/37494 objects degraded (26.255%); 54 MiB/s, 13 objects/s recovering 2023-03-09T14:52:42.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:41 smithi038 ceph-mon[92422]: from='osd.0 [v2:172.21.15.38:6810/1721245517,v1:172.21.15.38:6811/1721245517]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-03-09T14:52:42.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:41 smithi038 ceph-mon[92422]: osdmap e164: 12 total, 9 up, 12 in 2023-03-09T14:52:42.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:41 smithi038 ceph-mon[92422]: from='osd.0 [v2:172.21.15.38:6810/1721245517,v1:172.21.15.38:6811/1721245517]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2023-03-09T14:52:43.007 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:42 smithi122 ceph-mon[93869]: Health check update: 2 osds down (OSD_DOWN) 2023-03-09T14:52:43.007 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:42 smithi122 ceph-mon[93869]: osd.0 [v2:172.21.15.38:6810/1721245517,v1:172.21.15.38:6811/1721245517] boot 2023-03-09T14:52:43.008 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:42 smithi122 ceph-mon[93869]: osdmap e165: 12 total, 10 up, 12 in 2023-03-09T14:52:43.008 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-03-09T14:52:43.008 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:42 smithi122 ceph-mon[93869]: Health check update: Degraded data redundancy: 9844/37494 objects degraded (26.255%), 50 pgs degraded (PG_DEGRADED) 2023-03-09T14:52:43.008 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:42 smithi122 ceph-mon[93869]: osdmap e166: 12 total, 10 up, 12 in 2023-03-09T14:52:43.008 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:42 smithi122 bash[128884]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-03-09T14:52:43.009 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:42 smithi122 bash[128884]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2023-03-09T14:52:43.009 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:42 smithi122 bash[128884]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2023-03-09T14:52:43.009 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:42 smithi122 bash[128884]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2023-03-09T14:52:43.009 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:42 smithi122 bash[128884]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-7/block 2023-03-09T14:52:43.009 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:42 smithi122 bash[128884]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-03-09T14:52:43.009 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:42 smithi122 bash[128884]: --> ceph-volume raw activate successful for osd ID: 7 2023-03-09T14:52:43.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:42 smithi181 ceph-mon[94189]: Health check update: 2 osds down (OSD_DOWN) 2023-03-09T14:52:43.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:42 smithi181 ceph-mon[94189]: osd.0 [v2:172.21.15.38:6810/1721245517,v1:172.21.15.38:6811/1721245517] boot 2023-03-09T14:52:43.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:42 smithi181 ceph-mon[94189]: osdmap e165: 12 total, 10 up, 12 in 2023-03-09T14:52:43.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-03-09T14:52:43.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:42 smithi181 ceph-mon[94189]: Health check update: Degraded data redundancy: 9844/37494 objects degraded (26.255%), 50 pgs degraded (PG_DEGRADED) 2023-03-09T14:52:43.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:42 smithi181 ceph-mon[94189]: osdmap e166: 12 total, 10 up, 12 in 2023-03-09T14:52:43.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:42 smithi038 ceph-mon[92422]: Health check update: 2 osds down (OSD_DOWN) 2023-03-09T14:52:43.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:42 smithi038 ceph-mon[92422]: osd.0 [v2:172.21.15.38:6810/1721245517,v1:172.21.15.38:6811/1721245517] boot 2023-03-09T14:52:43.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:42 smithi038 ceph-mon[92422]: osdmap e165: 12 total, 10 up, 12 in 2023-03-09T14:52:43.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-03-09T14:52:43.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:42 smithi038 ceph-mon[92422]: Health check update: Degraded data redundancy: 9844/37494 objects degraded (26.255%), 50 pgs degraded (PG_DEGRADED) 2023-03-09T14:52:43.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:42 smithi038 ceph-mon[92422]: osdmap e166: 12 total, 10 up, 12 in 2023-03-09T14:52:43.260 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-03-09T14:52:43.598 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.588+0000 7f0b3ac20700 1 -- 172.21.15.38:0/516109906 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b34068000 msgr2=0x7f0b34068470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:52:43.598 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.588+0000 7f0b3ac20700 1 --2- 172.21.15.38:0/516109906 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b34068000 0x7f0b34068470 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0b200098f0 tx=0x7f0b20009c00 comp rx=0 tx=0).stop 2023-03-09T14:52:43.598 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.588+0000 7f0b3ac20700 1 -- 172.21.15.38:0/516109906 shutdown_connections 2023-03-09T14:52:43.598 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.588+0000 7f0b3ac20700 1 --2- 172.21.15.38:0/516109906 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0b34102850 0x7f0b34102cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:52:43.599 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.588+0000 7f0b3ac20700 1 --2- 172.21.15.38:0/516109906 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b34104d50 0x7f0b34067ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:52:43.599 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.588+0000 7f0b3ac20700 1 --2- 172.21.15.38:0/516109906 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b34068000 0x7f0b34068470 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:52:43.599 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.588+0000 7f0b3ac20700 1 -- 172.21.15.38:0/516109906 >> 172.21.15.38:0/516109906 conn(0x7f0b34075c60 msgr2=0x7f0b34076060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:52:43.605 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.602+0000 7f0b3ac20700 1 -- 172.21.15.38:0/516109906 shutdown_connections 2023-03-09T14:52:43.605 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.603+0000 7f0b3ac20700 1 -- 172.21.15.38:0/516109906 wait complete. 2023-03-09T14:52:43.606 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.604+0000 7f0b3ac20700 1 Processor -- start 2023-03-09T14:52:43.607 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.604+0000 7f0b3ac20700 1 -- start start 2023-03-09T14:52:43.608 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.604+0000 7f0b3ac20700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0b34068000 0x7f0b3419c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:52:43.608 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.604+0000 7f0b3ac20700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b34102850 0x7f0b3419cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:52:43.608 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.604+0000 7f0b3ac20700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b34104d50 0x7f0b341a0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:52:43.608 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.604+0000 7f0b3ac20700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b341a1630 con 0x7f0b34068000 2023-03-09T14:52:43.608 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.605+0000 7f0b3ac20700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b341a17a0 con 0x7f0b34104d50 2023-03-09T14:52:43.608 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.605+0000 7f0b3ac20700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b341a1960 con 0x7f0b34102850 2023-03-09T14:52:43.608 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.605+0000 7f0b389bc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0b34068000 0x7f0b3419c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:52:43.615 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.613+0000 7f0b389bc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0b34068000 0x7f0b3419c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:41340/0 (socket says 172.21.15.38:41340) 2023-03-09T14:52:43.615 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.613+0000 7f0b391bd700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b34104d50 0x7f0b341a0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:52:43.615 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.613+0000 7f0b389bc700 1 -- 172.21.15.38:0/2647211259 learned_addr learned my addr 172.21.15.38:0/2647211259 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:52:43.626 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.614+0000 7f0b33fff700 1 --2- 172.21.15.38:0/2647211259 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b34102850 0x7f0b3419cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:52:43.626 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.615+0000 7f0b33fff700 1 -- 172.21.15.38:0/2647211259 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b34104d50 msgr2=0x7f0b341a0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:52:43.627 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.615+0000 7f0b33fff700 1 --2- 172.21.15.38:0/2647211259 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b34104d50 0x7f0b341a0ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:52:43.627 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.615+0000 7f0b33fff700 1 -- 172.21.15.38:0/2647211259 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0b34068000 msgr2=0x7f0b3419c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:52:43.627 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.615+0000 7f0b33fff700 1 --2- 172.21.15.38:0/2647211259 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0b34068000 0x7f0b3419c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:52:43.627 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.615+0000 7f0b33fff700 1 -- 172.21.15.38:0/2647211259 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b20009520 con 0x7f0b34102850 2023-03-09T14:52:43.627 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.615+0000 7f0b33fff700 1 --2- 172.21.15.38:0/2647211259 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b34102850 0x7f0b3419cc30 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f0b2000b2e0 tx=0x7f0b20010f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:52:43.627 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.615+0000 7f0b31ffb700 1 -- 172.21.15.38:0/2647211259 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b2001c070 con 0x7f0b34102850 2023-03-09T14:52:43.627 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.615+0000 7f0b31ffb700 1 -- 172.21.15.38:0/2647211259 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0b20005530 con 0x7f0b34102850 2023-03-09T14:52:43.628 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.615+0000 7f0b31ffb700 1 -- 172.21.15.38:0/2647211259 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b20003b50 con 0x7f0b34102850 2023-03-09T14:52:43.628 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.616+0000 7f0b3ac20700 1 -- 172.21.15.38:0/2647211259 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b341165c0 con 0x7f0b34102850 2023-03-09T14:52:43.628 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.617+0000 7f0b3ac20700 1 -- 172.21.15.38:0/2647211259 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b34116430 con 0x7f0b34102850 2023-03-09T14:52:43.628 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.620+0000 7f0b31ffb700 1 -- 172.21.15.38:0/2647211259 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 25) v1 ==== 133295+0+0 (secure 0 0 0) 0x7f0b2000c040 con 0x7f0b34102850 2023-03-09T14:52:43.628 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.621+0000 7f0b31ffb700 1 --2- 172.21.15.38:0/2647211259 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f0b2409f5a0 0x7f0b240a1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:52:43.628 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.621+0000 7f0b31ffb700 1 -- 172.21.15.38:0/2647211259 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(166..166 src has 1..166) v4 ==== 9563+0+0 (secure 0 0 0) 0x7f0b200cb580 con 0x7f0b34102850 2023-03-09T14:52:43.629 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.623+0000 7f0b46da9b80 1 Processor -- start 2023-03-09T14:52:43.629 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.623+0000 7f0b46da9b80 1 -- start start 2023-03-09T14:52:43.629 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.623+0000 7f0b46da9b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5574c6d74840 0x5574c6d74c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:52:43.629 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.623+0000 7f0b46da9b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5574c6d75150 0x5574c6d6eab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:52:43.629 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.623+0000 7f0b46da9b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5574c6d6eff0 0x5574c6d6f420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:52:43.629 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.623+0000 7f0b46da9b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574c6c36da0 con 0x5574c6d74840 2023-03-09T14:52:43.630 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.623+0000 7f0b46da9b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574c6ca5780 con 0x5574c6d6eff0 2023-03-09T14:52:43.630 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.623+0000 7f0b46da9b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574c6c9ecb0 con 0x5574c6d75150 2023-03-09T14:52:43.630 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.624+0000 7f0b33fff700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5574c6d75150 0x5574c6d6eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:52:43.630 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.624+0000 7f0b33fff700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5574c6d75150 0x5574c6d6eab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.38:49044/0 (socket says 172.21.15.38:49044) 2023-03-09T14:52:43.630 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.624+0000 7f0b33fff700 1 -- 172.21.15.38:0/3253409684 learned_addr learned my addr 172.21.15.38:0/3253409684 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:52:43.630 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.624+0000 7f0b391bd700 1 --2- 172.21.15.38:0/3253409684 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5574c6d6eff0 0x5574c6d6f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:52:43.630 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.624+0000 7f0b33fff700 1 -- 172.21.15.38:0/3253409684 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5574c6d6eff0 msgr2=0x5574c6d6f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:52:43.630 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.624+0000 7f0b33fff700 1 --2- 172.21.15.38:0/3253409684 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5574c6d6eff0 0x5574c6d6f420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:52:43.631 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.624+0000 7f0b33fff700 1 -- 172.21.15.38:0/3253409684 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5574c6d74840 msgr2=0x5574c6d74c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:52:43.631 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.624+0000 7f0b33fff700 1 --2- 172.21.15.38:0/3253409684 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5574c6d74840 0x5574c6d74c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:52:43.631 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.624+0000 7f0b33fff700 1 -- 172.21.15.38:0/3253409684 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5574c6c65190 con 0x5574c6d75150 2023-03-09T14:52:43.632 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.625+0000 7f0b33fff700 1 --2- 172.21.15.38:0/3253409684 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5574c6d75150 0x5574c6d6eab0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f0b2000b520 tx=0x7f0b20099010 comp rx=0 tx=0).ready entity=mon.1 client_cookie=38072dd0de85ee2f server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:52:43.632 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.626+0000 7f0b1effd700 1 -- 172.21.15.38:0/3253409684 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b20099110 con 0x5574c6d75150 2023-03-09T14:52:43.632 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.626+0000 7f0b1effd700 1 -- 172.21.15.38:0/3253409684 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0b2009e070 con 0x5574c6d75150 2023-03-09T14:52:43.632 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.626+0000 7f0b1effd700 1 -- 172.21.15.38:0/3253409684 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b2000b9c0 con 0x5574c6d75150 2023-03-09T14:52:43.632 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.627+0000 7f0b389bc700 1 --2- 172.21.15.38:0/2647211259 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f0b2409f5a0 0x7f0b240a1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:52:43.632 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.627+0000 7f0b46da9b80 1 -- 172.21.15.38:0/3253409684 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5574c6d75150 msgr2=0x5574c6d6eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:52:43.633 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.627+0000 7f0b46da9b80 1 --2- 172.21.15.38:0/3253409684 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5574c6d75150 0x5574c6d6eab0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f0b2000b520 tx=0x7f0b20099010 comp rx=0 tx=0).stop 2023-03-09T14:52:43.633 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.627+0000 7f0b46da9b80 1 -- 172.21.15.38:0/3253409684 shutdown_connections 2023-03-09T14:52:43.633 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.627+0000 7f0b46da9b80 1 --2- 172.21.15.38:0/3253409684 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5574c6d74840 0x5574c6d74c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:52:43.633 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.627+0000 7f0b46da9b80 1 --2- 172.21.15.38:0/3253409684 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5574c6d6eff0 0x5574c6d6f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:52:43.633 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.627+0000 7f0b46da9b80 1 --2- 172.21.15.38:0/3253409684 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5574c6d75150 0x5574c6d6eab0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:52:43.634 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.627+0000 7f0b46da9b80 1 -- 172.21.15.38:0/3253409684 >> 172.21.15.38:0/3253409684 conn(0x5574c6d47910 msgr2=0x5574c6d47ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:52:43.634 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.628+0000 7f0b389bc700 1 --2- 172.21.15.38:0/2647211259 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f0b2409f5a0 0x7f0b240a1a50 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0b2800bc90 tx=0x7f0b280047c0 comp rx=0 tx=0).ready entity=mgr.15558 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:52:43.637 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.635+0000 7f0b46da9b80 1 -- 172.21.15.38:0/3253409684 shutdown_connections 2023-03-09T14:52:43.640 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.637+0000 7f0b46da9b80 1 -- 172.21.15.38:0/3253409684 wait complete. 2023-03-09T14:52:43.640 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.638+0000 7f0b46da9b80 1 -- start start 2023-03-09T14:52:43.640 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.638+0000 7f0b46da9b80 1 Processor -- start 2023-03-09T14:52:43.641 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.638+0000 7f0b46da9b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5574c6d6eff0 0x5574c6d7b150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:52:43.641 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.639+0000 7f0b46da9b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5574c6d74840 0x5574c6d78440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:52:43.641 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.639+0000 7f0b46da9b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5574c6d75150 0x5574c6d78b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:52:43.641 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.639+0000 7f0b46da9b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574c6b22920 con 0x5574c6d75150 2023-03-09T14:52:43.641 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.639+0000 7f0b46da9b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574c6d792f0 con 0x5574c6d74840 2023-03-09T14:52:43.642 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.639+0000 7f0b46da9b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574c6d84940 con 0x5574c6d6eff0 2023-03-09T14:52:43.642 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.639+0000 7f0b46da9b80 10 client.0 _ll_register_callbacks cb 0x5574c6c44520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:52:43.642 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.640+0000 7f0b33fff700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5574c6d74840 0x5574c6d78440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:52:43.643 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.641+0000 7f0b389bc700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5574c6d6eff0 0x5574c6d7b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:52:43.643 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.641+0000 7f0b389bc700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5574c6d6eff0 0x5574c6d7b150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.38:49056/0 (socket says 172.21.15.38:49056) 2023-03-09T14:52:43.643 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.641+0000 7f0b389bc700 1 -- 172.21.15.38:0/380494171 learned_addr learned my addr 172.21.15.38:0/380494171 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:52:43.644 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.641+0000 7f0b389bc700 1 -- 172.21.15.38:0/380494171 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5574c6d74840 msgr2=0x5574c6d78440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:52:43.644 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.641+0000 7f0b389bc700 1 --2- 172.21.15.38:0/380494171 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5574c6d74840 0x5574c6d78440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:52:43.644 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.641+0000 7f0b389bc700 1 -- 172.21.15.38:0/380494171 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5574c6d75150 msgr2=0x5574c6d78b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:52:43.644 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.641+0000 7f0b389bc700 1 --2- 172.21.15.38:0/380494171 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5574c6d75150 0x5574c6d78b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:52:43.644 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.641+0000 7f0b389bc700 1 -- 172.21.15.38:0/380494171 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b34116430 con 0x5574c6d6eff0 2023-03-09T14:52:43.651 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.649+0000 7f0b389bc700 1 --2- 172.21.15.38:0/380494171 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5574c6d6eff0 0x5574c6d7b150 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f0b2800bff0 tx=0x7f0b2800b4e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=326f0680e46c9b52 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:52:43.651 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.649+0000 7f0b13fff700 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2023-03-09T14:52:43.651 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.649+0000 7f0b13fff700 1 -- 172.21.15.38:0/380494171 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b2801e620 con 0x5574c6d6eff0 2023-03-09T14:52:43.652 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.649+0000 7f0b13fff700 1 -- 172.21.15.38:0/380494171 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(166..166 src has 1..166) v4 ==== 9563+0+0 (secure 0 0 0) 0x7f0b28017070 con 0x5574c6d6eff0 2023-03-09T14:52:43.652 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.649+0000 7f0b46da9b80 1 -- 172.21.15.38:0/380494171 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5574c6c6c190 con 0x5574c6d6eff0 2023-03-09T14:52:43.653 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.651+0000 7f0b13fff700 1 -- 172.21.15.38:0/380494171 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0b2801e780 con 0x5574c6d6eff0 2023-03-09T14:52:43.660 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.658+0000 7f0b46da9b80 10 client.25729 fetch_fsmap learned FSMap version 31 2023-03-09T14:52:43.660 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.658+0000 7f0b46da9b80 1 -- 172.21.15.38:0/380494171 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x5574c6ab5c60 con 0x5574c6d6eff0 2023-03-09T14:52:43.665 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.663+0000 7f0b13fff700 1 -- 172.21.15.38:0/380494171 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5574c6c6c190 con 0x5574c6d6eff0 2023-03-09T14:52:43.665 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.663+0000 7f0b13fff700 1 -- 172.21.15.38:0/380494171 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 31) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f0b28029d00 con 0x5574c6d6eff0 2023-03-09T14:52:43.665 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.663+0000 7f0b46da9b80 10 client.25729 fetch_fsmap finished waiting for FSMap version 31 2023-03-09T14:52:43.666 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.663+0000 7f0b46da9b80 10 client.25729 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T14:52:43.666 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.663+0000 7f0b46da9b80 20 client.25729 populate_metadata read hostname 'smithi038' 2023-03-09T14:52:43.666 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.663+0000 7f0b46da9b80 1 --2- 172.21.15.38:0/380494171 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5574c6d88ac0 0x5574c6d8aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:52:43.666 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.663+0000 7f0b46da9b80 4 client.25729 mds_command: new command op to 24530 tid=0[{"prefix": "get_command_descriptions"}] 2023-03-09T14:52:43.666 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.663+0000 7f0b46da9b80 1 -- 172.21.15.38:0/380494171 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5574c6b22f60 con 0x5574c6d88ac0 2023-03-09T14:52:43.666 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.663+0000 7f0b33fff700 1 --2- 172.21.15.38:0/380494171 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5574c6d88ac0 0x5574c6d8aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:52:43.668 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.665+0000 7f0b33fff700 1 --2- 172.21.15.38:0/380494171 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5574c6d88ac0 0x5574c6d8aed0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7c52b74843b4dc3a server_cookie=11d9ee402de9f276 in_seq=0 out_seq=0 2023-03-09T14:52:43.671 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.669+0000 7f0b13fff700 10 client.25729 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:52:43.671 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.669+0000 7f0b13fff700 1 -- 172.21.15.38:0/380494171 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18278 (crc 0 0 0) 0x5574c6b22f60 con 0x5574c6d88ac0 2023-03-09T14:52:43.671 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.669+0000 7f0b13fff700 10 client.25729 handle_command_reply: tid=0 2023-03-09T14:52:43.672 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.669+0000 7f0b46da9b80 1 client.25729 shutdown 2023-03-09T14:52:43.679 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.674+0000 7f0b46da9b80 1 -- 172.21.15.38:0/380494171 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5574c6d6eff0 msgr2=0x5574c6d7b150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:52:43.679 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.674+0000 7f0b46da9b80 1 --2- 172.21.15.38:0/380494171 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5574c6d6eff0 0x5574c6d7b150 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f0b2800bff0 tx=0x7f0b2800b4e0 comp rx=0 tx=0).stop 2023-03-09T14:52:43.679 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.675+0000 7f0b46da9b80 1 -- 172.21.15.38:0/380494171 shutdown_connections 2023-03-09T14:52:43.680 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.675+0000 7f0b46da9b80 1 --2- 172.21.15.38:0/380494171 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5574c6d88ac0 0x5574c6d8aed0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:52:43.680 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.675+0000 7f0b46da9b80 1 --2- 172.21.15.38:0/380494171 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5574c6d75150 0x5574c6d78b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:52:43.680 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.675+0000 7f0b46da9b80 1 --2- 172.21.15.38:0/380494171 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5574c6d6eff0 0x5574c6d7b150 secure :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f0b2800bff0 tx=0x7f0b2800b4e0 comp rx=0 tx=0).stop 2023-03-09T14:52:43.680 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.675+0000 7f0b46da9b80 1 --2- 172.21.15.38:0/380494171 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5574c6d74840 0x5574c6d78440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:52:43.680 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.675+0000 7f0b46da9b80 1 -- 172.21.15.38:0/380494171 >> 172.21.15.38:0/380494171 conn(0x5574c6d47910 msgr2=0x5574c6d46c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:52:43.681 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.675+0000 7f0b13fff700 0 client.25729 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:52:43.684 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.682+0000 7f0b46da9b80 1 -- 172.21.15.38:0/380494171 shutdown_connections 2023-03-09T14:52:43.684 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.682+0000 7f0b46da9b80 1 -- 172.21.15.38:0/380494171 wait complete. 2023-03-09T14:52:43.684 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.682+0000 7f0b46da9b80 20 client.25729 trim_cache size 0 max 16384 2023-03-09T14:52:43.718 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.715+0000 7f0b46da9b80 1 Processor -- start 2023-03-09T14:52:43.718 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.715+0000 7f0b46da9b80 1 -- start start 2023-03-09T14:52:43.719 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.715+0000 7f0b46da9b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5574c6d74840 0x5574c6d7f910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:52:43.719 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.715+0000 7f0b46da9b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5574c6d75150 0x5574c6d8aad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:52:43.719 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.715+0000 7f0b46da9b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5574c6d8b010 0x5574c6d46c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:52:43.719 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.715+0000 7f0b46da9b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574c6d79180 con 0x5574c6d74840 2023-03-09T14:52:43.719 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.715+0000 7f0b46da9b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574c6d861a0 con 0x5574c6d8b010 2023-03-09T14:52:43.719 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.715+0000 7f0b46da9b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574c6d74490 con 0x5574c6d75150 2023-03-09T14:52:43.719 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.715+0000 7f0b389bc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5574c6d74840 0x5574c6d7f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:52:43.720 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.715+0000 7f0b389bc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5574c6d74840 0x5574c6d7f910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:41362/0 (socket says 172.21.15.38:41362) 2023-03-09T14:52:43.720 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.715+0000 7f0b389bc700 1 -- 172.21.15.38:0/1967485330 learned_addr learned my addr 172.21.15.38:0/1967485330 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:52:43.720 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.716+0000 7f0b391bd700 1 --2- 172.21.15.38:0/1967485330 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5574c6d8b010 0x5574c6d46c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:52:43.720 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.716+0000 7f0b33fff700 1 --2- 172.21.15.38:0/1967485330 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5574c6d75150 0x5574c6d8aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:52:43.721 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.718+0000 7f0b391bd700 1 -- 172.21.15.38:0/1967485330 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5574c6d75150 msgr2=0x5574c6d8aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:52:43.721 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.718+0000 7f0b391bd700 1 --2- 172.21.15.38:0/1967485330 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5574c6d75150 0x5574c6d8aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:52:43.721 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.718+0000 7f0b391bd700 1 -- 172.21.15.38:0/1967485330 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5574c6d74840 msgr2=0x5574c6d7f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:52:43.721 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.718+0000 7f0b391bd700 1 --2- 172.21.15.38:0/1967485330 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5574c6d74840 0x5574c6d7f910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:52:43.721 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.718+0000 7f0b391bd700 1 -- 172.21.15.38:0/1967485330 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b34116750 con 0x5574c6d8b010 2023-03-09T14:52:43.722 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.718+0000 7f0b391bd700 1 --2- 172.21.15.38:0/1967485330 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5574c6d8b010 0x5574c6d46c20 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c009fa0 tx=0x7f0b2c00b410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d879b5695b06db89 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:52:43.722 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.718+0000 7f0b1f7fe700 1 -- 172.21.15.38:0/1967485330 <== mon.2 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b2c00bea0 con 0x5574c6d8b010 2023-03-09T14:52:43.722 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.718+0000 7f0b1f7fe700 1 -- 172.21.15.38:0/1967485330 <== mon.2 v2:172.21.15.122:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0b2c012040 con 0x5574c6d8b010 2023-03-09T14:52:43.722 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.718+0000 7f0b1f7fe700 1 -- 172.21.15.38:0/1967485330 <== mon.2 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b2c015810 con 0x5574c6d8b010 2023-03-09T14:52:43.722 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.719+0000 7f0b46da9b80 1 -- 172.21.15.38:0/1967485330 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5574c6d8b010 msgr2=0x5574c6d46c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:52:43.723 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.719+0000 7f0b46da9b80 1 --2- 172.21.15.38:0/1967485330 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5574c6d8b010 0x5574c6d46c20 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c009fa0 tx=0x7f0b2c00b410 comp rx=0 tx=0).stop 2023-03-09T14:52:43.723 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.719+0000 7f0b46da9b80 1 -- 172.21.15.38:0/1967485330 shutdown_connections 2023-03-09T14:52:43.723 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.719+0000 7f0b46da9b80 1 --2- 172.21.15.38:0/1967485330 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5574c6d74840 0x5574c6d7f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:52:43.723 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.719+0000 7f0b46da9b80 1 --2- 172.21.15.38:0/1967485330 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5574c6d8b010 0x5574c6d46c20 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:52:43.723 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.719+0000 7f0b46da9b80 1 --2- 172.21.15.38:0/1967485330 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5574c6d75150 0x5574c6d8aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:52:43.723 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.719+0000 7f0b46da9b80 1 -- 172.21.15.38:0/1967485330 >> 172.21.15.38:0/1967485330 conn(0x5574c6d47910 msgr2=0x5574c6d840a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:52:43.724 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.720+0000 7f0b46da9b80 1 -- 172.21.15.38:0/1967485330 shutdown_connections 2023-03-09T14:52:43.724 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.721+0000 7f0b46da9b80 1 -- 172.21.15.38:0/1967485330 wait complete. 2023-03-09T14:52:43.724 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.722+0000 7f0b46da9b80 1 -- start start 2023-03-09T14:52:43.724 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.722+0000 7f0b46da9b80 1 Processor -- start 2023-03-09T14:52:43.733 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.724+0000 7f0b46da9b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5574c6d8b010 0x5574c6d7fa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:52:43.733 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.724+0000 7f0b46da9b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5574c6d7dd40 0x5574c6d7e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:52:43.733 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.724+0000 7f0b46da9b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5574c6d7e6f0 0x5574c6ddc970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:52:43.733 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.724+0000 7f0b46da9b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574c6ddceb0 con 0x5574c6d8b010 2023-03-09T14:52:43.733 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.724+0000 7f0b46da9b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574c6ddcff0 con 0x5574c6d7e6f0 2023-03-09T14:52:43.733 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.724+0000 7f0b46da9b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574c6ddd130 con 0x5574c6d7dd40 2023-03-09T14:52:43.734 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.724+0000 7f0b46da9b80 10 client.0 _ll_register_callbacks cb 0x5574c6c44520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:52:43.734 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.727+0000 7f0b389bc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5574c6d8b010 0x5574c6d7fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:52:43.734 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.727+0000 7f0b389bc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5574c6d8b010 0x5574c6d7fa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:41370/0 (socket says 172.21.15.38:41370) 2023-03-09T14:52:43.734 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.727+0000 7f0b389bc700 1 -- 172.21.15.38:0/1859252871 learned_addr learned my addr 172.21.15.38:0/1859252871 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:52:43.734 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.728+0000 7f0b389bc700 1 -- 172.21.15.38:0/1859252871 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5574c6d7dd40 msgr2=0x5574c6d7e1b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-03-09T14:52:43.734 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.728+0000 7f0b389bc700 1 --2- 172.21.15.38:0/1859252871 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5574c6d7dd40 0x5574c6d7e1b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:52:43.734 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.728+0000 7f0b389bc700 1 -- 172.21.15.38:0/1859252871 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5574c6d7e6f0 msgr2=0x5574c6ddc970 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-03-09T14:52:43.735 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.728+0000 7f0b389bc700 1 --2- 172.21.15.38:0/1859252871 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5574c6d7e6f0 0x5574c6ddc970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:52:43.735 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.728+0000 7f0b389bc700 1 -- 172.21.15.38:0/1859252871 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b2c009c80 con 0x5574c6d8b010 2023-03-09T14:52:43.735 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.728+0000 7f0b389bc700 1 --2- 172.21.15.38:0/1859252871 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5574c6d8b010 0x5574c6d7fa00 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f0b28011b60 tx=0x7f0b28032d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51e9ddea5a2a7fe8 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:52:43.735 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.728+0000 7f0b13fff700 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2023-03-09T14:52:43.735 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.728+0000 7f0b13fff700 1 -- 172.21.15.38:0/1859252871 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b2800be80 con 0x5574c6d8b010 2023-03-09T14:52:43.735 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.728+0000 7f0b46da9b80 1 -- 172.21.15.38:0/1859252871 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5574c6c5c1b0 con 0x5574c6d8b010 2023-03-09T14:52:43.735 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.729+0000 7f0b13fff700 1 -- 172.21.15.38:0/1859252871 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(167..167 src has 1..167) v4 ==== 9563+0+0 (secure 0 0 0) 0x7f0b28017070 con 0x5574c6d8b010 2023-03-09T14:52:43.735 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.729+0000 7f0b13fff700 1 -- 172.21.15.38:0/1859252871 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0b280327c0 con 0x5574c6d8b010 2023-03-09T14:52:43.736 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.729+0000 7f0b46da9b80 10 client.16119 fetch_fsmap learned FSMap version 31 2023-03-09T14:52:43.736 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.729+0000 7f0b46da9b80 1 -- 172.21.15.38:0/1859252871 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x5574c6af3250 con 0x5574c6d8b010 2023-03-09T14:52:43.736 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.729+0000 7f0b13fff700 1 -- 172.21.15.38:0/1859252871 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5574c6c5c1b0 con 0x5574c6d8b010 2023-03-09T14:52:43.736 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.729+0000 7f0b13fff700 1 -- 172.21.15.38:0/1859252871 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 31) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f0b28026b80 con 0x5574c6d8b010 2023-03-09T14:52:43.736 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.729+0000 7f0b46da9b80 10 client.16119 fetch_fsmap finished waiting for FSMap version 31 2023-03-09T14:52:43.736 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.729+0000 7f0b46da9b80 10 client.16119 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T14:52:43.736 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.729+0000 7f0b46da9b80 20 client.16119 populate_metadata read hostname 'smithi038' 2023-03-09T14:52:43.736 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.729+0000 7f0b46da9b80 1 --2- 172.21.15.38:0/1859252871 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5574c6dde8b0 0x5574c6de0d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:52:43.737 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.729+0000 7f0b46da9b80 4 client.16119 mds_command: new command op to 24530 tid=0[{"prefix": "scrub status"}] 2023-03-09T14:52:43.737 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.729+0000 7f0b46da9b80 1 -- 172.21.15.38:0/1859252871 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5574c6c461c0 con 0x5574c6dde8b0 2023-03-09T14:52:43.737 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.730+0000 7f0b33fff700 1 --2- 172.21.15.38:0/1859252871 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5574c6dde8b0 0x5574c6de0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:52:43.737 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.730+0000 7f0b13fff700 10 client.16119 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:52:43.737 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.730+0000 7f0b33fff700 1 --2- 172.21.15.38:0/1859252871 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5574c6dde8b0 0x5574c6de0d40 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=39d1322e43270b8c server_cookie=9b66db060dd1ccae in_seq=0 out_seq=0 2023-03-09T14:52:43.738 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.732+0000 7f0b13fff700 1 -- 172.21.15.38:0/1859252871 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+265 (crc 0 0 0) 0x5574c6c461c0 con 0x5574c6dde8b0 2023-03-09T14:52:43.738 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.732+0000 7f0b13fff700 10 client.16119 handle_command_reply: tid=0 2023-03-09T14:52:43.738 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.732+0000 7f0b46da9b80 1 client.16119 shutdown 2023-03-09T14:52:43.738 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.732+0000 7f0b46da9b80 1 -- 172.21.15.38:0/1859252871 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5574c6d8b010 msgr2=0x5574c6d7fa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:52:43.738 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.732+0000 7f0b46da9b80 1 --2- 172.21.15.38:0/1859252871 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5574c6d8b010 0x5574c6d7fa00 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f0b28011b60 tx=0x7f0b28032d20 comp rx=0 tx=0).stop 2023-03-09T14:52:43.738 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.733+0000 7f0b46da9b80 1 -- 172.21.15.38:0/1859252871 shutdown_connections 2023-03-09T14:52:43.738 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.733+0000 7f0b46da9b80 1 --2- 172.21.15.38:0/1859252871 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5574c6dde8b0 0x5574c6de0d40 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:52:43.739 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.733+0000 7f0b46da9b80 1 --2- 172.21.15.38:0/1859252871 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5574c6d8b010 0x5574c6d7fa00 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:52:43.739 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.733+0000 7f0b46da9b80 1 --2- 172.21.15.38:0/1859252871 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5574c6d7e6f0 0x5574c6ddc970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:52:43.739 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.733+0000 7f0b46da9b80 1 --2- 172.21.15.38:0/1859252871 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5574c6d7dd40 0x5574c6d7e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:52:43.739 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.733+0000 7f0b46da9b80 1 -- 172.21.15.38:0/1859252871 >> 172.21.15.38:0/1859252871 conn(0x5574c6d47910 msgr2=0x5574c6d46c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:52:43.739 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.733+0000 7f0b13fff700 0 client.16119 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:52:43.740 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.736+0000 7f0b46da9b80 1 -- 172.21.15.38:0/1859252871 shutdown_connections 2023-03-09T14:52:43.740 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.736+0000 7f0b46da9b80 1 -- 172.21.15.38:0/1859252871 wait complete. 2023-03-09T14:52:43.744 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2023-03-09T14:52:43.744 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "scrub active (3 inodes in the stack)", 2023-03-09T14:52:43.745 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": { 2023-03-09T14:52:43.745 INFO:teuthology.orchestra.run.smithi038.stdout: "a14659f4-d7ea-4c5a-b15b-6138af3f909f": { 2023-03-09T14:52:43.745 INFO:teuthology.orchestra.run.smithi038.stdout: "path": "/", 2023-03-09T14:52:43.745 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "a14659f4-d7ea-4c5a-b15b-6138af3f909f", 2023-03-09T14:52:43.745 INFO:teuthology.orchestra.run.smithi038.stdout: "options": "recursive,force" 2023-03-09T14:52:43.745 INFO:teuthology.orchestra.run.smithi038.stdout: } 2023-03-09T14:52:43.745 INFO:teuthology.orchestra.run.smithi038.stdout: } 2023-03-09T14:52:43.746 INFO:teuthology.orchestra.run.smithi038.stdout:} 2023-03-09T14:52:43.746 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.741+0000 7f0b46da9b80 20 client.16119 trim_cache size 0 max 16384 2023-03-09T14:52:43.749 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.743+0000 7f0b3ac20700 1 -- 172.21.15.38:0/2647211259 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f0b2409f5a0 msgr2=0x7f0b240a1a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:52:43.749 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.743+0000 7f0b3ac20700 1 --2- 172.21.15.38:0/2647211259 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f0b2409f5a0 0x7f0b240a1a50 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0b2800bc90 tx=0x7f0b280047c0 comp rx=0 tx=0).stop 2023-03-09T14:52:43.749 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.743+0000 7f0b3ac20700 1 -- 172.21.15.38:0/2647211259 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b34102850 msgr2=0x7f0b3419cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:52:43.749 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.743+0000 7f0b3ac20700 1 --2- 172.21.15.38:0/2647211259 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b34102850 0x7f0b3419cc30 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f0b2000b2e0 tx=0x7f0b20010f80 comp rx=0 tx=0).stop 2023-03-09T14:52:43.749 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.747+0000 7f0b3ac20700 1 -- 172.21.15.38:0/2647211259 shutdown_connections 2023-03-09T14:52:43.750 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.747+0000 7f0b3ac20700 1 --2- 172.21.15.38:0/2647211259 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f0b2409f5a0 0x7f0b240a1a50 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:52:43.750 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.747+0000 7f0b3ac20700 1 --2- 172.21.15.38:0/2647211259 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f0b34068000 0x7f0b3419c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:52:43.750 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.747+0000 7f0b3ac20700 1 --2- 172.21.15.38:0/2647211259 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f0b34104d50 0x7f0b341a0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:52:43.750 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.747+0000 7f0b3ac20700 1 --2- 172.21.15.38:0/2647211259 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f0b34102850 0x7f0b3419cc30 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:52:43.750 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.747+0000 7f0b3ac20700 1 -- 172.21.15.38:0/2647211259 >> 172.21.15.38:0/2647211259 conn(0x7f0b34075c60 msgr2=0x7f0b34101830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:52:43.755 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.753+0000 7f0b3ac20700 1 -- 172.21.15.38:0/2647211259 shutdown_connections 2023-03-09T14:52:43.765 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:52:43.756+0000 7f0b3ac20700 1 -- 172.21.15.38:0/2647211259 wait complete. 2023-03-09T14:52:43.786 INFO:tasks.cephfs.filesystem:scrub status for tag:a14659f4-d7ea-4c5a-b15b-6138af3f909f - {'path': '/', 'tag': 'a14659f4-d7ea-4c5a-b15b-6138af3f909f', 'options': 'recursive,force'} 2023-03-09T14:52:43.804 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 135.0 s, 0.2 tps, lat 53421.387 ms stddev 0.003 2023-03-09T14:52:44.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:43 smithi122 ceph-mon[93869]: pgmap v273: 129 pgs: 5 peering, 1 down, 5 undersized+peered, 31 active+undersized, 10 undersized+degraded+peered, 37 active+undersized+degraded, 40 active+clean; 19 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s; 9221/37494 objects degraded (24.593%); 2.7 MiB/s, 0 objects/s recovering 2023-03-09T14:52:44.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:43 smithi122 ceph-mon[93869]: osdmap e167: 12 total, 10 up, 12 in 2023-03-09T14:52:44.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:43 smithi181 ceph-mon[94189]: pgmap v273: 129 pgs: 5 peering, 1 down, 5 undersized+peered, 31 active+undersized, 10 undersized+degraded+peered, 37 active+undersized+degraded, 40 active+clean; 19 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s; 9221/37494 objects degraded (24.593%); 2.7 MiB/s, 0 objects/s recovering 2023-03-09T14:52:44.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:43 smithi181 ceph-mon[94189]: osdmap e167: 12 total, 10 up, 12 in 2023-03-09T14:52:44.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:43 smithi038 ceph-mon[92422]: pgmap v273: 129 pgs: 5 peering, 1 down, 5 undersized+peered, 31 active+undersized, 10 undersized+degraded+peered, 37 active+undersized+degraded, 40 active+clean; 19 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 3 op/s; 9221/37494 objects degraded (24.593%); 2.7 MiB/s, 0 objects/s recovering 2023-03-09T14:52:44.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:43 smithi038 ceph-mon[92422]: osdmap e167: 12 total, 10 up, 12 in 2023-03-09T14:52:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:45 smithi122 ceph-mon[93869]: pgmap v275: 129 pgs: 1 active+recovering+degraded, 5 peering, 1 down, 1 active+recovering+undersized+degraded, 4 undersized+peered, 30 active+undersized, 1 active+recovery_wait, 8 undersized+degraded+peered, 35 active+undersized+degraded, 43 active+clean; 19 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 626 KiB/s wr, 79 op/s; 8833/37830 objects degraded (23.349%); 31 MiB/s, 7 objects/s recovering 2023-03-09T14:52:46.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:45 smithi181 ceph-mon[94189]: pgmap v275: 129 pgs: 1 active+recovering+degraded, 5 peering, 1 down, 1 active+recovering+undersized+degraded, 4 undersized+peered, 30 active+undersized, 1 active+recovery_wait, 8 undersized+degraded+peered, 35 active+undersized+degraded, 43 active+clean; 19 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 626 KiB/s wr, 79 op/s; 8833/37830 objects degraded (23.349%); 31 MiB/s, 7 objects/s recovering 2023-03-09T14:52:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:45 smithi038 ceph-mon[92422]: pgmap v275: 129 pgs: 1 active+recovering+degraded, 5 peering, 1 down, 1 active+recovering+undersized+degraded, 4 undersized+peered, 30 active+undersized, 1 active+recovery_wait, 8 undersized+degraded+peered, 35 active+undersized+degraded, 43 active+clean; 19 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 626 KiB/s wr, 79 op/s; 8833/37830 objects degraded (23.349%); 31 MiB/s, 7 objects/s recovering 2023-03-09T14:52:47.750 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:47 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[154598]: 2023-03-09T14:52:47.293+0000 7f6ab05cc640 -1 osd.9 160 log_to_monitors true 2023-03-09T14:52:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:47 smithi122 ceph-mon[93869]: pgmap v276: 129 pgs: 1 active+recovering+degraded, 1 down, 5 peering, 2 active+recovery_wait+undersized+degraded, 2 active+recovering+undersized+degraded, 2 undersized+peered, 22 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 undersized+degraded+peered, 33 active+undersized+degraded, 56 active+clean; 19 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 2.0 MiB/s wr, 255 op/s; 7953/39026 objects degraded (20.379%); 55 MiB/s, 14 objects/s recovering 2023-03-09T14:52:48.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:47 smithi122 ceph-mon[93869]: from='osd.9 [v2:172.21.15.38:6826/77860270,v1:172.21.15.38:6827/77860270]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-03-09T14:52:48.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:47 smithi122 ceph-mon[93869]: Health check update: Degraded data redundancy: 7953/39026 objects degraded (20.379%), 42 pgs degraded (PG_DEGRADED) 2023-03-09T14:52:48.228 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:47 smithi122 podman[129167]: 2023-03-09T14:52:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:47 smithi181 ceph-mon[94189]: pgmap v276: 129 pgs: 1 active+recovering+degraded, 1 down, 5 peering, 2 active+recovery_wait+undersized+degraded, 2 active+recovering+undersized+degraded, 2 undersized+peered, 22 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 undersized+degraded+peered, 33 active+undersized+degraded, 56 active+clean; 19 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 2.0 MiB/s wr, 255 op/s; 7953/39026 objects degraded (20.379%); 55 MiB/s, 14 objects/s recovering 2023-03-09T14:52:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:47 smithi181 ceph-mon[94189]: from='osd.9 [v2:172.21.15.38:6826/77860270,v1:172.21.15.38:6827/77860270]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-03-09T14:52:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:47 smithi181 ceph-mon[94189]: Health check update: Degraded data redundancy: 7953/39026 objects degraded (20.379%), 42 pgs degraded (PG_DEGRADED) 2023-03-09T14:52:48.250 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 09 14:52:47 smithi038 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-9[154598]: 2023-03-09T14:52:47.831+0000 7f6aa0346700 -1 osd.9 160 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-09T14:52:48.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:47 smithi038 ceph-mon[92422]: pgmap v276: 129 pgs: 1 active+recovering+degraded, 1 down, 5 peering, 2 active+recovery_wait+undersized+degraded, 2 active+recovering+undersized+degraded, 2 undersized+peered, 22 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 3 undersized+degraded+peered, 33 active+undersized+degraded, 56 active+clean; 19 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 2.0 MiB/s wr, 255 op/s; 7953/39026 objects degraded (20.379%); 55 MiB/s, 14 objects/s recovering 2023-03-09T14:52:48.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:47 smithi038 ceph-mon[92422]: from='osd.9 [v2:172.21.15.38:6826/77860270,v1:172.21.15.38:6827/77860270]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-03-09T14:52:48.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:47 smithi038 ceph-mon[92422]: Health check update: Degraded data redundancy: 7953/39026 objects degraded (20.379%), 42 pgs degraded (PG_DEGRADED) 2023-03-09T14:52:48.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 140.0 s, 0.4 tps, lat 37573.666 ms stddev 15937.988 2023-03-09T14:52:49.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:48 smithi122 ceph-mon[93869]: from='osd.9 [v2:172.21.15.38:6826/77860270,v1:172.21.15.38:6827/77860270]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-03-09T14:52:49.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:48 smithi122 ceph-mon[93869]: osdmap e168: 12 total, 10 up, 12 in 2023-03-09T14:52:49.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:48 smithi122 ceph-mon[93869]: from='osd.9 [v2:172.21.15.38:6826/77860270,v1:172.21.15.38:6827/77860270]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2023-03-09T14:52:49.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:48 smithi181 ceph-mon[94189]: from='osd.9 [v2:172.21.15.38:6826/77860270,v1:172.21.15.38:6827/77860270]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-03-09T14:52:49.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:48 smithi181 ceph-mon[94189]: osdmap e168: 12 total, 10 up, 12 in 2023-03-09T14:52:49.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:48 smithi181 ceph-mon[94189]: from='osd.9 [v2:172.21.15.38:6826/77860270,v1:172.21.15.38:6827/77860270]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2023-03-09T14:52:49.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:48 smithi038 ceph-mon[92422]: from='osd.9 [v2:172.21.15.38:6826/77860270,v1:172.21.15.38:6827/77860270]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-03-09T14:52:49.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:48 smithi038 ceph-mon[92422]: osdmap e168: 12 total, 10 up, 12 in 2023-03-09T14:52:49.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:48 smithi038 ceph-mon[92422]: from='osd.9 [v2:172.21.15.38:6826/77860270,v1:172.21.15.38:6827/77860270]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi038", "root=default"]}]: dispatch 2023-03-09T14:52:49.728 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:49 smithi122 bash[129167]: d2785eaffe00fc198e8b70ed71acdb6c5d0aa184a6462e4b9ac32a1b62bb84bb 2023-03-09T14:52:49.728 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:49 smithi122 ceph-osd[129206]: -- 172.21.15.122:0/2737030317 <== mon.2 v2:172.21.15.122:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5626e888a9c0 con 0x5626e8853000 2023-03-09T14:52:49.728 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:49 smithi122 systemd[1]: Started Ceph osd.7 for c6733fe2-be87-11ed-9af2-001a4aab830c. 2023-03-09T14:52:50.084 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:52:50 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 2023-03-09T14:52:50.080+0000 7f12b15b7640 -1 Falling back to public interface 2023-03-09T14:52:50.084 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:49 smithi122 ceph-mon[93869]: pgmap v278: 129 pgs: 1 active+recovery_wait+degraded, 1 down, 2 active+recovery_wait+undersized+degraded, 2 active+recovering+undersized+degraded, 2 undersized+peered, 23 active+undersized, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait, 3 undersized+degraded+peered, 33 active+undersized+degraded, 58 active+clean; 19 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 2.7 MiB/s wr, 336 op/s; 8143/39594 objects degraded (20.566%); 100 MiB/s, 25 objects/s recovering 2023-03-09T14:52:50.085 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:49 smithi122 ceph-mon[93869]: Health check update: 1 osds down (OSD_DOWN) 2023-03-09T14:52:50.085 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:49 smithi122 ceph-mon[93869]: osd.9 [v2:172.21.15.38:6826/77860270,v1:172.21.15.38:6827/77860270] boot 2023-03-09T14:52:50.085 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:49 smithi122 ceph-mon[93869]: osdmap e169: 12 total, 11 up, 12 in 2023-03-09T14:52:50.085 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-03-09T14:52:50.085 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:49 smithi122 ceph-mon[93869]: osdmap e170: 12 total, 11 up, 12 in 2023-03-09T14:52:50.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:49 smithi181 ceph-mon[94189]: pgmap v278: 129 pgs: 1 active+recovery_wait+degraded, 1 down, 2 active+recovery_wait+undersized+degraded, 2 active+recovering+undersized+degraded, 2 undersized+peered, 23 active+undersized, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait, 3 undersized+degraded+peered, 33 active+undersized+degraded, 58 active+clean; 19 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 2.7 MiB/s wr, 336 op/s; 8143/39594 objects degraded (20.566%); 100 MiB/s, 25 objects/s recovering 2023-03-09T14:52:50.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:49 smithi181 ceph-mon[94189]: Health check update: 1 osds down (OSD_DOWN) 2023-03-09T14:52:50.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:49 smithi181 ceph-mon[94189]: osd.9 [v2:172.21.15.38:6826/77860270,v1:172.21.15.38:6827/77860270] boot 2023-03-09T14:52:50.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:49 smithi181 ceph-mon[94189]: osdmap e169: 12 total, 11 up, 12 in 2023-03-09T14:52:50.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-03-09T14:52:50.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:49 smithi181 ceph-mon[94189]: osdmap e170: 12 total, 11 up, 12 in 2023-03-09T14:52:50.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:49 smithi038 ceph-mon[92422]: pgmap v278: 129 pgs: 1 active+recovery_wait+degraded, 1 down, 2 active+recovery_wait+undersized+degraded, 2 active+recovering+undersized+degraded, 2 undersized+peered, 23 active+undersized, 2 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait, 3 undersized+degraded+peered, 33 active+undersized+degraded, 58 active+clean; 19 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 2.7 MiB/s wr, 336 op/s; 8143/39594 objects degraded (20.566%); 100 MiB/s, 25 objects/s recovering 2023-03-09T14:52:50.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:49 smithi038 ceph-mon[92422]: Health check update: 1 osds down (OSD_DOWN) 2023-03-09T14:52:50.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:49 smithi038 ceph-mon[92422]: osd.9 [v2:172.21.15.38:6826/77860270,v1:172.21.15.38:6827/77860270] boot 2023-03-09T14:52:50.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:49 smithi038 ceph-mon[92422]: osdmap e169: 12 total, 11 up, 12 in 2023-03-09T14:52:50.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-03-09T14:52:50.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:49 smithi038 ceph-mon[92422]: osdmap e170: 12 total, 11 up, 12 in 2023-03-09T14:52:52.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:51 smithi122 ceph-mon[93869]: pgmap v281: 129 pgs: 1 active+recovering+degraded, 7 active+recovery_wait+degraded, 1 peering, 1 down, 1 active+recovery_wait+undersized+degraded, 1 active+recovering+undersized+degraded, 2 undersized+peered, 18 active+undersized, 1 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 1 undersized+degraded+peered, 26 active+undersized+degraded, 67 active+clean; 19 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 2.5 MiB/s wr, 320 op/s; 5503/39786 objects degraded (13.831%); 206 MiB/s, 52 objects/s recovering 2023-03-09T14:52:52.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:51 smithi122 ceph-mon[93869]: osdmap e171: 12 total, 11 up, 12 in 2023-03-09T14:52:52.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:51 smithi181 ceph-mon[94189]: pgmap v281: 129 pgs: 1 active+recovering+degraded, 7 active+recovery_wait+degraded, 1 peering, 1 down, 1 active+recovery_wait+undersized+degraded, 1 active+recovering+undersized+degraded, 2 undersized+peered, 18 active+undersized, 1 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 1 undersized+degraded+peered, 26 active+undersized+degraded, 67 active+clean; 19 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 2.5 MiB/s wr, 320 op/s; 5503/39786 objects degraded (13.831%); 206 MiB/s, 52 objects/s recovering 2023-03-09T14:52:52.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:51 smithi181 ceph-mon[94189]: osdmap e171: 12 total, 11 up, 12 in 2023-03-09T14:52:52.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:51 smithi038 ceph-mon[92422]: pgmap v281: 129 pgs: 1 active+recovering+degraded, 7 active+recovery_wait+degraded, 1 peering, 1 down, 1 active+recovery_wait+undersized+degraded, 1 active+recovering+undersized+degraded, 2 undersized+peered, 18 active+undersized, 1 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait, 1 undersized+degraded+peered, 26 active+undersized+degraded, 67 active+clean; 19 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 2.5 MiB/s wr, 320 op/s; 5503/39786 objects degraded (13.831%); 206 MiB/s, 52 objects/s recovering 2023-03-09T14:52:52.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:51 smithi038 ceph-mon[92422]: osdmap e171: 12 total, 11 up, 12 in 2023-03-09T14:52:53.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:52 smithi122 ceph-mon[93869]: osdmap e172: 12 total, 11 up, 12 in 2023-03-09T14:52:53.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:52 smithi122 ceph-mon[93869]: Health check update: Degraded data redundancy: 5503/39786 objects degraded (13.831%), 38 pgs degraded (PG_DEGRADED) 2023-03-09T14:52:53.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:52 smithi122 ceph-mon[93869]: osdmap e173: 12 total, 11 up, 12 in 2023-03-09T14:52:53.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:52 smithi181 ceph-mon[94189]: osdmap e172: 12 total, 11 up, 12 in 2023-03-09T14:52:53.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:52 smithi181 ceph-mon[94189]: Health check update: Degraded data redundancy: 5503/39786 objects degraded (13.831%), 38 pgs degraded (PG_DEGRADED) 2023-03-09T14:52:53.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:52 smithi181 ceph-mon[94189]: osdmap e173: 12 total, 11 up, 12 in 2023-03-09T14:52:53.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:52 smithi038 ceph-mon[92422]: osdmap e172: 12 total, 11 up, 12 in 2023-03-09T14:52:53.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:52 smithi038 ceph-mon[92422]: Health check update: Degraded data redundancy: 5503/39786 objects degraded (13.831%), 38 pgs degraded (PG_DEGRADED) 2023-03-09T14:52:53.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:52 smithi038 ceph-mon[92422]: osdmap e173: 12 total, 11 up, 12 in 2023-03-09T14:52:53.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 145.0 s, 4.0 tps, lat 23699.178 ms stddev 16959.713 2023-03-09T14:52:54.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:53 smithi122 ceph-mon[93869]: pgmap v285: 129 pgs: 1 remapped+peering, 1 active+recovering+degraded, 8 active+recovery_wait+degraded, 1 peering, 1 down, 1 active+recovering+undersized+degraded, 2 undersized+peered, 16 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 1 undersized+degraded+peered, 24 active+undersized+degraded, 71 active+clean; 19 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 401 KiB/s wr, 57 op/s; 4921/39794 objects degraded (12.366%); 209 MiB/s, 52 objects/s recovering 2023-03-09T14:52:54.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:53 smithi122 ceph-mon[93869]: osdmap e174: 12 total, 11 up, 12 in 2023-03-09T14:52:54.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:53 smithi181 ceph-mon[94189]: pgmap v285: 129 pgs: 1 remapped+peering, 1 active+recovering+degraded, 8 active+recovery_wait+degraded, 1 peering, 1 down, 1 active+recovering+undersized+degraded, 2 undersized+peered, 16 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 1 undersized+degraded+peered, 24 active+undersized+degraded, 71 active+clean; 19 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 401 KiB/s wr, 57 op/s; 4921/39794 objects degraded (12.366%); 209 MiB/s, 52 objects/s recovering 2023-03-09T14:52:54.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:53 smithi181 ceph-mon[94189]: osdmap e174: 12 total, 11 up, 12 in 2023-03-09T14:52:54.254 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:53 smithi038 ceph-mon[92422]: pgmap v285: 129 pgs: 1 remapped+peering, 1 active+recovering+degraded, 8 active+recovery_wait+degraded, 1 peering, 1 down, 1 active+recovering+undersized+degraded, 2 undersized+peered, 16 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 1 undersized+degraded+peered, 24 active+undersized+degraded, 71 active+clean; 19 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 401 KiB/s wr, 57 op/s; 4921/39794 objects degraded (12.366%); 209 MiB/s, 52 objects/s recovering 2023-03-09T14:52:54.254 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:53 smithi038 ceph-mon[92422]: osdmap e174: 12 total, 11 up, 12 in 2023-03-09T14:52:54.594 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 2023-03-09T14:52:54.268+0000 7f6251bd3700 -1 osd.8 pg_epoch: 174 pg[4.2s0( v 174'1058 (0'0,174'1058] local-lis/les=169/170 n=345 ec=85/85 lis/c=169/169 les/c/f=170/173/0 sis=169) [8,10,11,9]p8(0) r=0 lpr=169 luod=174'1005 crt=174'1005 lcod 174'1004 mlcod 174'1004 active+clean] bad op order, already applied 91326 > this 91322 2023-03-09T14:52:54.594 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7f6251bd3700 time 2023-03-09T14:52:54.269817+0000 2023-03-09T14:52:54.594 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op") 2023-03-09T14:52:54.595 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:52:54.595 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x55a0b5f23c85] 2023-03-09T14:52:54.595 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 2: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55a0b61dc6e4] 2023-03-09T14:52:54.595 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55a0b61e2d5e] 2023-03-09T14:52:54.595 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 4: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55a0b6032986] 2023-03-09T14:52:54.595 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 5: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55a0b6348729] 2023-03-09T14:52:54.595 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55a0b6047ebf] 2023-03-09T14:52:54.595 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55a0b66f9d85] 2023-03-09T14:52:54.596 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55a0b66fcf84] 2023-03-09T14:52:54.596 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f627a13e1ca] 2023-03-09T14:52:54.596 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 10: clone() 2023-03-09T14:52:54.596 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: *** Caught signal (Aborted) ** 2023-03-09T14:52:54.596 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: in thread 7f6251bd3700 thread_name:tp_osd_tp 2023-03-09T14:52:54.596 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 2023-03-09T14:52:54.272+0000 7f6251bd3700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7f6251bd3700 time 2023-03-09T14:52:54.269817+0000 2023-03-09T14:52:54.596 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op") 2023-03-09T14:52:54.597 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 2023-03-09T14:52:54.597 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:52:54.597 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x55a0b5f23c85] 2023-03-09T14:52:54.597 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 2: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55a0b61dc6e4] 2023-03-09T14:52:54.597 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55a0b61e2d5e] 2023-03-09T14:52:54.597 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 4: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55a0b6032986] 2023-03-09T14:52:54.597 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 5: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55a0b6348729] 2023-03-09T14:52:54.597 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55a0b6047ebf] 2023-03-09T14:52:54.598 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55a0b66f9d85] 2023-03-09T14:52:54.598 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55a0b66fcf84] 2023-03-09T14:52:54.598 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f627a13e1ca] 2023-03-09T14:52:54.598 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 10: clone() 2023-03-09T14:52:54.598 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 2023-03-09T14:52:54.598 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:52:54.598 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f627a148cf0] 2023-03-09T14:52:54.599 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 2: gsignal() 2023-03-09T14:52:54.599 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 3: abort() 2023-03-09T14:52:54.599 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x55a0b5f23d45] 2023-03-09T14:52:54.599 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 5: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55a0b61dc6e4] 2023-03-09T14:52:54.599 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 6: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55a0b61e2d5e] 2023-03-09T14:52:54.599 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55a0b6032986] 2023-03-09T14:52:54.599 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 8: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55a0b6348729] 2023-03-09T14:52:54.600 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55a0b6047ebf] 2023-03-09T14:52:54.600 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55a0b66f9d85] 2023-03-09T14:52:54.600 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55a0b66fcf84] 2023-03-09T14:52:54.600 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f627a13e1ca] 2023-03-09T14:52:54.600 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 13: clone() 2023-03-09T14:52:54.600 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 2023-03-09T14:52:54.276+0000 7f6251bd3700 -1 *** Caught signal (Aborted) ** 2023-03-09T14:52:54.600 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: in thread 7f6251bd3700 thread_name:tp_osd_tp 2023-03-09T14:52:54.600 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 2023-03-09T14:52:54.601 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:52:54.601 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f627a148cf0] 2023-03-09T14:52:54.601 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 2: gsignal() 2023-03-09T14:52:54.601 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 3: abort() 2023-03-09T14:52:54.601 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x55a0b5f23d45] 2023-03-09T14:52:54.601 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 5: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55a0b61dc6e4] 2023-03-09T14:52:54.601 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 6: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55a0b61e2d5e] 2023-03-09T14:52:54.602 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55a0b6032986] 2023-03-09T14:52:54.602 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 8: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55a0b6348729] 2023-03-09T14:52:54.602 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55a0b6047ebf] 2023-03-09T14:52:54.602 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55a0b66f9d85] 2023-03-09T14:52:54.602 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55a0b66fcf84] 2023-03-09T14:52:54.602 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f627a13e1ca] 2023-03-09T14:52:54.602 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 13: clone() 2023-03-09T14:52:54.602 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-03-09T14:52:54.603 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 2023-03-09T14:52:54.603 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: -904> 2023-03-09T14:52:54.268+0000 7f6251bd3700 -1 osd.8 pg_epoch: 174 pg[4.2s0( v 174'1058 (0'0,174'1058] local-lis/les=169/170 n=345 ec=85/85 lis/c=169/169 les/c/f=170/173/0 sis=169) [8,10,11,9]p8(0) r=0 lpr=169 luod=174'1005 crt=174'1005 lcod 174'1004 mlcod 174'1004 active+clean] bad op order, already applied 91326 > this 91322 2023-03-09T14:52:54.603 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: -518> 2023-03-09T14:52:54.272+0000 7f6251bd3700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7f6251bd3700 time 2023-03-09T14:52:54.269817+0000 2023-03-09T14:52:54.603 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op") 2023-03-09T14:52:54.603 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 2023-03-09T14:52:54.603 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:52:54.603 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x55a0b5f23c85] 2023-03-09T14:52:54.603 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 2: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55a0b61dc6e4] 2023-03-09T14:52:54.604 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55a0b61e2d5e] 2023-03-09T14:52:54.604 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 4: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55a0b6032986] 2023-03-09T14:52:54.604 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 5: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55a0b6348729] 2023-03-09T14:52:54.604 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55a0b6047ebf] 2023-03-09T14:52:54.604 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55a0b66f9d85] 2023-03-09T14:52:54.604 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55a0b66fcf84] 2023-03-09T14:52:54.604 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f627a13e1ca] 2023-03-09T14:52:54.605 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 10: clone() 2023-03-09T14:52:54.605 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 2023-03-09T14:52:54.605 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 0> 2023-03-09T14:52:54.276+0000 7f6251bd3700 -1 *** Caught signal (Aborted) ** 2023-03-09T14:52:54.605 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: in thread 7f6251bd3700 thread_name:tp_osd_tp 2023-03-09T14:52:54.605 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 2023-03-09T14:52:54.605 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:52:54.605 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f627a148cf0] 2023-03-09T14:52:54.605 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 2: gsignal() 2023-03-09T14:52:54.606 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 3: abort() 2023-03-09T14:52:54.606 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x55a0b5f23d45] 2023-03-09T14:52:54.606 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 5: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55a0b61dc6e4] 2023-03-09T14:52:54.606 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 6: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55a0b61e2d5e] 2023-03-09T14:52:54.606 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55a0b6032986] 2023-03-09T14:52:54.606 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 8: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55a0b6348729] 2023-03-09T14:52:54.606 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55a0b6047ebf] 2023-03-09T14:52:54.607 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55a0b66f9d85] 2023-03-09T14:52:54.607 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55a0b66fcf84] 2023-03-09T14:52:54.607 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f627a13e1ca] 2023-03-09T14:52:54.607 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 13: clone() 2023-03-09T14:52:54.607 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-03-09T14:52:54.607 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 2023-03-09T14:52:54.607 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: -1905> 2023-03-09T14:52:54.268+0000 7f6251bd3700 -1 osd.8 pg_epoch: 174 pg[4.2s0( v 174'1058 (0'0,174'1058] local-lis/les=169/170 n=345 ec=85/85 lis/c=169/169 les/c/f=170/173/0 sis=169) [8,10,11,9]p8(0) r=0 lpr=169 luod=174'1005 crt=174'1005 lcod 174'1004 mlcod 174'1004 active+clean] bad op order, already applied 91326 > this 91322 2023-03-09T14:52:54.607 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: -1519> 2023-03-09T14:52:54.272+0000 7f6251bd3700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7f6251bd3700 time 2023-03-09T14:52:54.269817+0000 2023-03-09T14:52:54.608 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op") 2023-03-09T14:52:54.608 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 2023-03-09T14:52:54.608 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:52:54.608 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x55a0b5f23c85] 2023-03-09T14:52:54.608 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 2: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55a0b61dc6e4] 2023-03-09T14:52:54.608 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55a0b61e2d5e] 2023-03-09T14:52:54.608 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 4: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55a0b6032986] 2023-03-09T14:52:54.609 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 5: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55a0b6348729] 2023-03-09T14:52:54.609 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55a0b6047ebf] 2023-03-09T14:52:54.609 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55a0b66f9d85] 2023-03-09T14:52:54.609 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55a0b66fcf84] 2023-03-09T14:52:54.609 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f627a13e1ca] 2023-03-09T14:52:54.609 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 10: clone() 2023-03-09T14:52:54.609 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 2023-03-09T14:52:54.609 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: -1001> 2023-03-09T14:52:54.276+0000 7f6251bd3700 -1 *** Caught signal (Aborted) ** 2023-03-09T14:52:54.610 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: in thread 7f6251bd3700 thread_name:tp_osd_tp 2023-03-09T14:52:54.610 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 2023-03-09T14:52:54.610 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:52:54.610 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f627a148cf0] 2023-03-09T14:52:54.610 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 2: gsignal() 2023-03-09T14:52:54.611 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 3: abort() 2023-03-09T14:52:54.611 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x55a0b5f23d45] 2023-03-09T14:52:54.611 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 5: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55a0b61dc6e4] 2023-03-09T14:52:54.611 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 6: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55a0b61e2d5e] 2023-03-09T14:52:54.611 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55a0b6032986] 2023-03-09T14:52:54.611 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 8: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55a0b6348729] 2023-03-09T14:52:54.611 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55a0b6047ebf] 2023-03-09T14:52:54.612 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55a0b66f9d85] 2023-03-09T14:52:54.612 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55a0b66fcf84] 2023-03-09T14:52:54.612 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f627a13e1ca] 2023-03-09T14:52:54.612 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 13: clone() 2023-03-09T14:52:54.612 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-03-09T14:52:54.612 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[127543]: 2023-03-09T14:52:54.733 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 2023-03-09T14:52:54.387+0000 7fa3b1a3d700 -1 osd.1 pg_epoch: 174 pg[4.as0( v 174'3384 (0'0,174'3384] local-lis/les=162/163 n=372 ec=85/85 lis/c=162/138 les/c/f=163/139/0 sis=162) [1,5,6,NONE]p1(0) r=0 lpr=162 pi=[138,162)/1 luod=174'3330 crt=174'3330 lcod 174'3329 mlcod 174'3329 active+undersized+degraded mbc={0={},1={},2={},3={}}] bad op order, already applied 91167 > this 91165 2023-03-09T14:52:54.733 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7fa3b1a3d700 time 2023-03-09T14:52:54.388687+0000 2023-03-09T14:52:54.733 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op") 2023-03-09T14:52:54.734 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:52:54.734 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x55fd2db88c85] 2023-03-09T14:52:54.734 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 2: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55fd2de416e4] 2023-03-09T14:52:54.734 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55fd2de47d5e] 2023-03-09T14:52:54.734 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 4: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55fd2dc97986] 2023-03-09T14:52:54.734 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 5: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55fd2dfad729] 2023-03-09T14:52:54.734 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55fd2dcacebf] 2023-03-09T14:52:54.735 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55fd2e35ed85] 2023-03-09T14:52:54.735 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55fd2e361f84] 2023-03-09T14:52:54.735 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7fa3d9fa81ca] 2023-03-09T14:52:54.735 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 10: clone() 2023-03-09T14:52:54.735 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 2023-03-09T14:52:54.390+0000 7fa3b1a3d700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7fa3b1a3d700 time 2023-03-09T14:52:54.388687+0000 2023-03-09T14:52:54.735 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op") 2023-03-09T14:52:54.735 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 2023-03-09T14:52:54.736 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:52:54.736 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x55fd2db88c85] 2023-03-09T14:52:54.736 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 2: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55fd2de416e4] 2023-03-09T14:52:54.736 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55fd2de47d5e] 2023-03-09T14:52:54.736 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 4: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55fd2dc97986] 2023-03-09T14:52:54.736 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 5: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55fd2dfad729] 2023-03-09T14:52:54.736 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55fd2dcacebf] 2023-03-09T14:52:54.736 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55fd2e35ed85] 2023-03-09T14:52:54.737 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55fd2e361f84] 2023-03-09T14:52:54.737 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7fa3d9fa81ca] 2023-03-09T14:52:54.737 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 10: clone() 2023-03-09T14:52:54.737 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 2023-03-09T14:52:54.737 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: *** Caught signal (Aborted) ** 2023-03-09T14:52:54.737 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: in thread 7fa3b1a3d700 thread_name:tp_osd_tp 2023-03-09T14:52:54.737 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:52:54.737 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7fa3d9fb2cf0] 2023-03-09T14:52:54.738 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 2: gsignal() 2023-03-09T14:52:54.738 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 3: abort() 2023-03-09T14:52:54.738 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x55fd2db88d45] 2023-03-09T14:52:54.738 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 5: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55fd2de416e4] 2023-03-09T14:52:54.738 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 6: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55fd2de47d5e] 2023-03-09T14:52:54.738 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55fd2dc97986] 2023-03-09T14:52:54.738 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 8: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55fd2dfad729] 2023-03-09T14:52:54.739 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55fd2dcacebf] 2023-03-09T14:52:54.739 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55fd2e35ed85] 2023-03-09T14:52:54.739 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55fd2e361f84] 2023-03-09T14:52:54.739 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fa3d9fa81ca] 2023-03-09T14:52:54.739 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 13: clone() 2023-03-09T14:52:54.740 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 2023-03-09T14:52:54.393+0000 7fa3b1a3d700 -1 *** Caught signal (Aborted) ** 2023-03-09T14:52:54.740 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: in thread 7fa3b1a3d700 thread_name:tp_osd_tp 2023-03-09T14:52:54.740 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 2023-03-09T14:52:54.740 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:52:54.740 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7fa3d9fb2cf0] 2023-03-09T14:52:54.740 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 2: gsignal() 2023-03-09T14:52:54.741 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 3: abort() 2023-03-09T14:52:54.741 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x55fd2db88d45] 2023-03-09T14:52:54.741 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 5: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55fd2de416e4] 2023-03-09T14:52:54.741 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 6: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55fd2de47d5e] 2023-03-09T14:52:54.741 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55fd2dc97986] 2023-03-09T14:52:54.741 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 8: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55fd2dfad729] 2023-03-09T14:52:54.741 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55fd2dcacebf] 2023-03-09T14:52:54.742 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55fd2e35ed85] 2023-03-09T14:52:54.742 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55fd2e361f84] 2023-03-09T14:52:54.742 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fa3d9fa81ca] 2023-03-09T14:52:54.742 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 13: clone() 2023-03-09T14:52:54.742 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-03-09T14:52:54.742 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 2023-03-09T14:52:54.743 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: -698> 2023-03-09T14:52:54.387+0000 7fa3b1a3d700 -1 osd.1 pg_epoch: 174 pg[4.as0( v 174'3384 (0'0,174'3384] local-lis/les=162/163 n=372 ec=85/85 lis/c=162/138 les/c/f=163/139/0 sis=162) [1,5,6,NONE]p1(0) r=0 lpr=162 pi=[138,162)/1 luod=174'3330 crt=174'3330 lcod 174'3329 mlcod 174'3329 active+undersized+degraded mbc={0={},1={},2={},3={}}] bad op order, already applied 91167 > this 91165 2023-03-09T14:52:54.743 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: -75> 2023-03-09T14:52:54.390+0000 7fa3b1a3d700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7fa3b1a3d700 time 2023-03-09T14:52:54.388687+0000 2023-03-09T14:52:54.743 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op") 2023-03-09T14:52:54.743 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 2023-03-09T14:52:54.743 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:52:54.743 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x55fd2db88c85] 2023-03-09T14:52:54.743 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 2: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55fd2de416e4] 2023-03-09T14:52:54.744 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55fd2de47d5e] 2023-03-09T14:52:54.744 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 4: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55fd2dc97986] 2023-03-09T14:52:54.744 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 5: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55fd2dfad729] 2023-03-09T14:52:54.744 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55fd2dcacebf] 2023-03-09T14:52:54.744 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55fd2e35ed85] 2023-03-09T14:52:54.744 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55fd2e361f84] 2023-03-09T14:52:54.744 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7fa3d9fa81ca] 2023-03-09T14:52:54.744 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 10: clone() 2023-03-09T14:52:54.745 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 2023-03-09T14:52:54.745 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 0> 2023-03-09T14:52:54.393+0000 7fa3b1a3d700 -1 *** Caught signal (Aborted) ** 2023-03-09T14:52:54.745 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: in thread 7fa3b1a3d700 thread_name:tp_osd_tp 2023-03-09T14:52:54.745 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 2023-03-09T14:52:54.745 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:52:54.745 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7fa3d9fb2cf0] 2023-03-09T14:52:54.745 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 2: gsignal() 2023-03-09T14:52:54.745 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 3: abort() 2023-03-09T14:52:54.746 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x55fd2db88d45] 2023-03-09T14:52:54.746 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 5: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55fd2de416e4] 2023-03-09T14:52:54.746 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 6: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55fd2de47d5e] 2023-03-09T14:52:54.746 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55fd2dc97986] 2023-03-09T14:52:54.746 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 8: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55fd2dfad729] 2023-03-09T14:52:54.746 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55fd2dcacebf] 2023-03-09T14:52:54.746 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55fd2e35ed85] 2023-03-09T14:52:54.747 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55fd2e361f84] 2023-03-09T14:52:54.747 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fa3d9fa81ca] 2023-03-09T14:52:54.747 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 13: clone() 2023-03-09T14:52:54.747 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-03-09T14:52:54.747 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 2023-03-09T14:52:54.747 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: -1699> 2023-03-09T14:52:54.387+0000 7fa3b1a3d700 -1 osd.1 pg_epoch: 174 pg[4.as0( v 174'3384 (0'0,174'3384] local-lis/les=162/163 n=372 ec=85/85 lis/c=162/138 les/c/f=163/139/0 sis=162) [1,5,6,NONE]p1(0) r=0 lpr=162 pi=[138,162)/1 luod=174'3330 crt=174'3330 lcod 174'3329 mlcod 174'3329 active+undersized+degraded mbc={0={},1={},2={},3={}}] bad op order, already applied 91167 > this 91165 2023-03-09T14:52:54.747 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: -1076> 2023-03-09T14:52:54.390+0000 7fa3b1a3d700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7fa3b1a3d700 time 2023-03-09T14:52:54.388687+0000 2023-03-09T14:52:54.747 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op") 2023-03-09T14:52:54.748 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 2023-03-09T14:52:54.748 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:52:54.748 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x55fd2db88c85] 2023-03-09T14:52:54.748 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 2: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55fd2de416e4] 2023-03-09T14:52:54.748 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55fd2de47d5e] 2023-03-09T14:52:54.748 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 4: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55fd2dc97986] 2023-03-09T14:52:54.748 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 5: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55fd2dfad729] 2023-03-09T14:52:54.749 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55fd2dcacebf] 2023-03-09T14:52:54.749 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55fd2e35ed85] 2023-03-09T14:52:54.749 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55fd2e361f84] 2023-03-09T14:52:54.749 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7fa3d9fa81ca] 2023-03-09T14:52:54.749 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 10: clone() 2023-03-09T14:52:54.749 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 2023-03-09T14:52:54.749 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: -1001> 2023-03-09T14:52:54.393+0000 7fa3b1a3d700 -1 *** Caught signal (Aborted) ** 2023-03-09T14:52:54.749 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: in thread 7fa3b1a3d700 thread_name:tp_osd_tp 2023-03-09T14:52:54.750 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 2023-03-09T14:52:54.750 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:52:54.750 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7fa3d9fb2cf0] 2023-03-09T14:52:54.750 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 2: gsignal() 2023-03-09T14:52:54.750 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 3: abort() 2023-03-09T14:52:54.750 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x55fd2db88d45] 2023-03-09T14:52:54.751 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 5: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x55fd2de416e4] 2023-03-09T14:52:54.751 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 6: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x55fd2de47d5e] 2023-03-09T14:52:54.751 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x55fd2dc97986] 2023-03-09T14:52:54.751 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 8: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x55fd2dfad729] 2023-03-09T14:52:54.752 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x55fd2dcacebf] 2023-03-09T14:52:54.752 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x55fd2e35ed85] 2023-03-09T14:52:54.752 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x55fd2e361f84] 2023-03-09T14:52:54.752 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7fa3d9fa81ca] 2023-03-09T14:52:54.752 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 13: clone() 2023-03-09T14:52:54.752 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-03-09T14:52:54.752 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[127657]: 2023-03-09T14:52:54.869 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.11 2023-03-09T14:52:54.869 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.8 failed (root=default,host=smithi181) (connection refused reported by osd.11) 2023-03-09T14:52:54.869 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.2 2023-03-09T14:52:54.869 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.2 2023-03-09T14:52:54.869 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.9 2023-03-09T14:52:54.870 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.9 2023-03-09T14:52:54.870 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.3 2023-03-09T14:52:54.870 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.3 2023-03-09T14:52:54.870 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.0 2023-03-09T14:52:54.870 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.0 2023-03-09T14:52:54.870 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.4 2023-03-09T14:52:54.870 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.4 2023-03-09T14:52:54.871 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.4 2023-03-09T14:52:54.871 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.9 2023-03-09T14:52:54.871 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.2 2023-03-09T14:52:54.871 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.1 failed (root=default,host=smithi122) (connection refused reported by osd.2) 2023-03-09T14:52:55.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.11 2023-03-09T14:52:55.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.8 failed (root=default,host=smithi181) (connection refused reported by osd.11) 2023-03-09T14:52:55.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.2 2023-03-09T14:52:55.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.2 2023-03-09T14:52:55.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.9 2023-03-09T14:52:55.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.9 2023-03-09T14:52:55.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.3 2023-03-09T14:52:55.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.3 2023-03-09T14:52:55.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.0 2023-03-09T14:52:55.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.0 2023-03-09T14:52:55.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.4 2023-03-09T14:52:55.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.4 2023-03-09T14:52:55.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.4 2023-03-09T14:52:55.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.9 2023-03-09T14:52:55.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.2 2023-03-09T14:52:55.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.1 failed (root=default,host=smithi122) (connection refused reported by osd.2) 2023-03-09T14:52:55.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.2 2023-03-09T14:52:55.231 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.5 2023-03-09T14:52:55.231 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.11 2023-03-09T14:52:55.231 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.5 2023-03-09T14:52:55.231 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.11 2023-03-09T14:52:55.231 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.0 2023-03-09T14:52:55.231 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.0 2023-03-09T14:52:55.231 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.0 2023-03-09T14:52:55.231 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.2 2023-03-09T14:52:55.232 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.3 2023-03-09T14:52:55.232 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.3 2023-03-09T14:52:55.232 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.10 2023-03-09T14:52:55.232 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.2 2023-03-09T14:52:55.232 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.10 2023-03-09T14:52:55.232 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.2 2023-03-09T14:52:55.232 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.2 2023-03-09T14:52:55.232 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.11 2023-03-09T14:52:55.233 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.11 2023-03-09T14:52:55.233 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.11 2023-03-09T14:52:55.233 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.5 2023-03-09T14:52:55.233 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.10 2023-03-09T14:52:55.233 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.0 2023-03-09T14:52:55.233 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.5 2023-03-09T14:52:55.233 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.5 2023-03-09T14:52:55.234 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.9 2023-03-09T14:52:55.234 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.9 2023-03-09T14:52:55.234 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.6 2023-03-09T14:52:55.234 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.6 2023-03-09T14:52:55.234 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.6 2023-03-09T14:52:55.234 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.0 2023-03-09T14:52:55.234 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.4 2023-03-09T14:52:55.235 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.3 2023-03-09T14:52:55.235 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.3 2023-03-09T14:52:55.235 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.4 2023-03-09T14:52:55.235 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.4 2023-03-09T14:52:55.235 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.8 reported immediately failed by osd.9 2023-03-09T14:52:55.235 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.9 2023-03-09T14:52:55.235 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.4 2023-03-09T14:52:55.235 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.9 2023-03-09T14:52:55.236 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.10 2023-03-09T14:52:55.236 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.2 2023-03-09T14:52:55.236 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.4 2023-03-09T14:52:55.237 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.10 2023-03-09T14:52:55.237 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.2 2023-03-09T14:52:55.237 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.4 2023-03-09T14:52:55.237 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.6 2023-03-09T14:52:55.238 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.10 2023-03-09T14:52:55.238 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.5 2023-03-09T14:52:55.238 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.5 2023-03-09T14:52:55.238 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.5 2023-03-09T14:52:55.239 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.6 2023-03-09T14:52:55.239 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.11 2023-03-09T14:52:55.239 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.11 2023-03-09T14:52:55.239 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.2 2023-03-09T14:52:55.240 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.0 2023-03-09T14:52:55.240 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.3 2023-03-09T14:52:55.240 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.0 2023-03-09T14:52:55.240 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.0 2023-03-09T14:52:55.241 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.6 2023-03-09T14:52:55.241 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.3 2023-03-09T14:52:55.241 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:54 smithi122 ceph-mon[93869]: osd.1 reported immediately failed by osd.3 2023-03-09T14:52:55.242 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:54 smithi181 systemd[1]: ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.8.service: Main process exited, code=exited, status=134/n/a 2023-03-09T14:52:55.243 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.2 2023-03-09T14:52:55.243 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.5 2023-03-09T14:52:55.244 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.11 2023-03-09T14:52:55.244 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.5 2023-03-09T14:52:55.244 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.11 2023-03-09T14:52:55.244 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.0 2023-03-09T14:52:55.245 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.0 2023-03-09T14:52:55.245 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.0 2023-03-09T14:52:55.245 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.2 2023-03-09T14:52:55.245 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.3 2023-03-09T14:52:55.245 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.3 2023-03-09T14:52:55.246 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.10 2023-03-09T14:52:55.246 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.2 2023-03-09T14:52:55.246 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.10 2023-03-09T14:52:55.246 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.2 2023-03-09T14:52:55.247 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.2 2023-03-09T14:52:55.247 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.11 2023-03-09T14:52:55.247 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.11 2023-03-09T14:52:55.247 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.11 2023-03-09T14:52:55.248 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.5 2023-03-09T14:52:55.248 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.10 2023-03-09T14:52:55.248 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.0 2023-03-09T14:52:55.248 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.5 2023-03-09T14:52:55.248 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.5 2023-03-09T14:52:55.248 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.9 2023-03-09T14:52:55.248 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.9 2023-03-09T14:52:55.249 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.6 2023-03-09T14:52:55.249 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.6 2023-03-09T14:52:55.249 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.6 2023-03-09T14:52:55.249 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.0 2023-03-09T14:52:55.249 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.4 2023-03-09T14:52:55.249 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.3 2023-03-09T14:52:55.249 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.3 2023-03-09T14:52:55.249 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.4 2023-03-09T14:52:55.250 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.4 2023-03-09T14:52:55.250 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.8 reported immediately failed by osd.9 2023-03-09T14:52:55.250 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.9 2023-03-09T14:52:55.250 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.4 2023-03-09T14:52:55.250 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.9 2023-03-09T14:52:55.250 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.10 2023-03-09T14:52:55.250 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.2 2023-03-09T14:52:55.251 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.4 2023-03-09T14:52:55.251 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.10 2023-03-09T14:52:55.251 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.2 2023-03-09T14:52:55.251 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.4 2023-03-09T14:52:55.251 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.6 2023-03-09T14:52:55.251 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.10 2023-03-09T14:52:55.251 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.5 2023-03-09T14:52:55.251 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.5 2023-03-09T14:52:55.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.5 2023-03-09T14:52:55.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.6 2023-03-09T14:52:55.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.11 2023-03-09T14:52:55.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.11 2023-03-09T14:52:55.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.2 2023-03-09T14:52:55.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.0 2023-03-09T14:52:55.252 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.3 2023-03-09T14:52:55.253 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.0 2023-03-09T14:52:55.253 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.0 2023-03-09T14:52:55.253 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.6 2023-03-09T14:52:55.253 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.3 2023-03-09T14:52:55.253 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:54 smithi181 ceph-mon[94189]: osd.1 reported immediately failed by osd.3 2023-03-09T14:52:55.254 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.11 2023-03-09T14:52:55.254 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.8 failed (root=default,host=smithi181) (connection refused reported by osd.11) 2023-03-09T14:52:55.254 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.2 2023-03-09T14:52:55.254 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.2 2023-03-09T14:52:55.255 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.9 2023-03-09T14:52:55.255 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.9 2023-03-09T14:52:55.255 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.3 2023-03-09T14:52:55.255 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.3 2023-03-09T14:52:55.255 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.0 2023-03-09T14:52:55.255 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.0 2023-03-09T14:52:55.255 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.4 2023-03-09T14:52:55.255 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.4 2023-03-09T14:52:55.256 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.4 2023-03-09T14:52:55.256 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.9 2023-03-09T14:52:55.256 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.2 2023-03-09T14:52:55.256 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.1 failed (root=default,host=smithi122) (connection refused reported by osd.2) 2023-03-09T14:52:55.256 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.2 2023-03-09T14:52:55.256 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.5 2023-03-09T14:52:55.256 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.11 2023-03-09T14:52:55.257 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.5 2023-03-09T14:52:55.257 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.11 2023-03-09T14:52:55.257 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.0 2023-03-09T14:52:55.257 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.0 2023-03-09T14:52:55.257 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.0 2023-03-09T14:52:55.257 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.2 2023-03-09T14:52:55.257 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.3 2023-03-09T14:52:55.257 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.3 2023-03-09T14:52:55.258 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.10 2023-03-09T14:52:55.258 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.2 2023-03-09T14:52:55.258 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.10 2023-03-09T14:52:55.258 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.2 2023-03-09T14:52:55.258 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.2 2023-03-09T14:52:55.258 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.11 2023-03-09T14:52:55.258 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.11 2023-03-09T14:52:55.258 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.11 2023-03-09T14:52:55.259 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.5 2023-03-09T14:52:55.259 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.10 2023-03-09T14:52:55.259 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.0 2023-03-09T14:52:55.259 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.5 2023-03-09T14:52:55.259 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.5 2023-03-09T14:52:55.259 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.9 2023-03-09T14:52:55.259 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.9 2023-03-09T14:52:55.259 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.6 2023-03-09T14:52:55.260 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.6 2023-03-09T14:52:55.260 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.6 2023-03-09T14:52:55.260 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.0 2023-03-09T14:52:55.260 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.4 2023-03-09T14:52:55.260 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.3 2023-03-09T14:52:55.260 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.3 2023-03-09T14:52:55.260 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.4 2023-03-09T14:52:55.260 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.4 2023-03-09T14:52:55.261 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.8 reported immediately failed by osd.9 2023-03-09T14:52:55.261 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.9 2023-03-09T14:52:55.261 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.4 2023-03-09T14:52:55.261 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.9 2023-03-09T14:52:55.261 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.10 2023-03-09T14:52:55.261 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.2 2023-03-09T14:52:55.261 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.4 2023-03-09T14:52:55.262 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.10 2023-03-09T14:52:55.262 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.2 2023-03-09T14:52:55.262 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.4 2023-03-09T14:52:55.262 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.6 2023-03-09T14:52:55.262 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.10 2023-03-09T14:52:55.262 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.5 2023-03-09T14:52:55.262 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.5 2023-03-09T14:52:55.262 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.5 2023-03-09T14:52:55.263 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.6 2023-03-09T14:52:55.263 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.11 2023-03-09T14:52:55.263 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.11 2023-03-09T14:52:55.263 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.2 2023-03-09T14:52:55.263 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.0 2023-03-09T14:52:55.263 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.3 2023-03-09T14:52:55.263 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.0 2023-03-09T14:52:55.263 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.0 2023-03-09T14:52:55.264 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.6 2023-03-09T14:52:55.264 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.3 2023-03-09T14:52:55.264 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:54 smithi038 ceph-mon[92422]: osd.1 reported immediately failed by osd.3 2023-03-09T14:52:55.867 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:55 smithi122 ceph-mon[93869]: pgmap v287: 129 pgs: 1 remapped+peering, 1 active+recovering+degraded, 11 active+recovery_wait+degraded, 1 peering, 15 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 21 active+undersized+degraded, 77 active+clean; 19 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 691 KiB/s wr, 90 op/s; 4155/39974 objects degraded (10.394%); 138 MiB/s, 34 objects/s recovering 2023-03-09T14:52:55.867 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:55 smithi122 ceph-mon[93869]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 1 pg down) 2023-03-09T14:52:55.868 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:55 smithi122 ceph-mon[93869]: Health check update: 3 osds down (OSD_DOWN) 2023-03-09T14:52:55.868 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:55 smithi122 ceph-mon[93869]: osdmap e175: 12 total, 9 up, 12 in 2023-03-09T14:52:55.868 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:55 smithi181 ceph-mon[94189]: pgmap v287: 129 pgs: 1 remapped+peering, 1 active+recovering+degraded, 11 active+recovery_wait+degraded, 1 peering, 15 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 21 active+undersized+degraded, 77 active+clean; 19 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 691 KiB/s wr, 90 op/s; 4155/39974 objects degraded (10.394%); 138 MiB/s, 34 objects/s recovering 2023-03-09T14:52:55.868 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:55 smithi181 ceph-mon[94189]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 1 pg down) 2023-03-09T14:52:55.868 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:55 smithi181 ceph-mon[94189]: Health check update: 3 osds down (OSD_DOWN) 2023-03-09T14:52:55.869 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:55 smithi181 ceph-mon[94189]: osdmap e175: 12 total, 9 up, 12 in 2023-03-09T14:52:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:55 smithi038 ceph-mon[92422]: pgmap v287: 129 pgs: 1 remapped+peering, 1 active+recovering+degraded, 11 active+recovery_wait+degraded, 1 peering, 15 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 21 active+undersized+degraded, 77 active+clean; 19 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 691 KiB/s wr, 90 op/s; 4155/39974 objects degraded (10.394%); 138 MiB/s, 34 objects/s recovering 2023-03-09T14:52:56.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:55 smithi038 ceph-mon[92422]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 1 pg down) 2023-03-09T14:52:56.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:55 smithi038 ceph-mon[92422]: Health check update: 3 osds down (OSD_DOWN) 2023-03-09T14:52:56.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:55 smithi038 ceph-mon[92422]: osdmap e175: 12 total, 9 up, 12 in 2023-03-09T14:52:56.731 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:52:56 smithi181 systemd[1]: ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.8.service: Failed with result 'exit-code'. 2023-03-09T14:52:57.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:56 smithi122 ceph-mon[93869]: osdmap e176: 12 total, 9 up, 12 in 2023-03-09T14:52:57.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:56 smithi181 ceph-mon[94189]: osdmap e176: 12 total, 9 up, 12 in 2023-03-09T14:52:57.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:56 smithi038 ceph-mon[92422]: osdmap e176: 12 total, 9 up, 12 in 2023-03-09T14:52:58.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:57 smithi122 ceph-mon[93869]: pgmap v290: 129 pgs: 1 remapped+peering, 3 stale+active+undersized, 2 stale+active+undersized+degraded, 3 stale+active+clean, 1 active+recovering+degraded, 9 active+recovery_wait+degraded, 41 peering, 8 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 12 active+undersized+degraded, 48 active+clean; 19 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 4.3 MiB/s wr, 521 op/s; 2884/41967 objects degraded (6.872%); 176 MiB/s, 44 objects/s recovering 2023-03-09T14:52:58.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:57 smithi122 ceph-mon[93869]: Health check failed: Reduced data availability: 4 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2023-03-09T14:52:58.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:57 smithi122 ceph-mon[93869]: osdmap e177: 12 total, 9 up, 12 in 2023-03-09T14:52:58.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:57 smithi122 ceph-mon[93869]: Health check update: Degraded data redundancy: 2884/41967 objects degraded (6.872%), 25 pgs degraded (PG_DEGRADED) 2023-03-09T14:52:58.228 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:52:57 smithi122 systemd[1]: ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.1.service: Main process exited, code=exited, status=134/n/a 2023-03-09T14:52:58.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:57 smithi181 ceph-mon[94189]: pgmap v290: 129 pgs: 1 remapped+peering, 3 stale+active+undersized, 2 stale+active+undersized+degraded, 3 stale+active+clean, 1 active+recovering+degraded, 9 active+recovery_wait+degraded, 41 peering, 8 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 12 active+undersized+degraded, 48 active+clean; 19 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 4.3 MiB/s wr, 521 op/s; 2884/41967 objects degraded (6.872%); 176 MiB/s, 44 objects/s recovering 2023-03-09T14:52:58.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:57 smithi181 ceph-mon[94189]: Health check failed: Reduced data availability: 4 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2023-03-09T14:52:58.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:57 smithi181 ceph-mon[94189]: osdmap e177: 12 total, 9 up, 12 in 2023-03-09T14:52:58.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:57 smithi181 ceph-mon[94189]: Health check update: Degraded data redundancy: 2884/41967 objects degraded (6.872%), 25 pgs degraded (PG_DEGRADED) 2023-03-09T14:52:58.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:57 smithi038 ceph-mon[92422]: pgmap v290: 129 pgs: 1 remapped+peering, 3 stale+active+undersized, 2 stale+active+undersized+degraded, 3 stale+active+clean, 1 active+recovering+degraded, 9 active+recovery_wait+degraded, 41 peering, 8 active+undersized, 1 active+recovery_wait+undersized+degraded+remapped, 12 active+undersized+degraded, 48 active+clean; 19 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 4.3 MiB/s wr, 521 op/s; 2884/41967 objects degraded (6.872%); 176 MiB/s, 44 objects/s recovering 2023-03-09T14:52:58.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:57 smithi038 ceph-mon[92422]: Health check failed: Reduced data availability: 4 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2023-03-09T14:52:58.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:57 smithi038 ceph-mon[92422]: osdmap e177: 12 total, 9 up, 12 in 2023-03-09T14:52:58.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:57 smithi038 ceph-mon[92422]: Health check update: Degraded data redundancy: 2884/41967 objects degraded (6.872%), 25 pgs degraded (PG_DEGRADED) 2023-03-09T14:52:58.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 150.0 s, 4.8 tps, lat 23918.368 ms stddev 12449.145 2023-03-09T14:53:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:52:59 smithi122 ceph-mon[93869]: pgmap v292: 129 pgs: 1 down+remapped, 1 remapped+peering, 2 stale+active+undersized, 1 stale+active+clean, 1 active+recovering+degraded, 7 active+recovery_wait+degraded, 40 peering, 2 undersized+peered, 10 active+undersized, 4 undersized+degraded+peered, 15 active+undersized+degraded, 45 active+clean; 19 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 4.4 MiB/s wr, 538 op/s; 3737/42934 objects degraded (8.704%); 160 MiB/s, 40 objects/s recovering 2023-03-09T14:53:00.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:52:59 smithi181 ceph-mon[94189]: pgmap v292: 129 pgs: 1 down+remapped, 1 remapped+peering, 2 stale+active+undersized, 1 stale+active+clean, 1 active+recovering+degraded, 7 active+recovery_wait+degraded, 40 peering, 2 undersized+peered, 10 active+undersized, 4 undersized+degraded+peered, 15 active+undersized+degraded, 45 active+clean; 19 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 4.4 MiB/s wr, 538 op/s; 3737/42934 objects degraded (8.704%); 160 MiB/s, 40 objects/s recovering 2023-03-09T14:53:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:52:59 smithi038 ceph-mon[92422]: pgmap v292: 129 pgs: 1 down+remapped, 1 remapped+peering, 2 stale+active+undersized, 1 stale+active+clean, 1 active+recovering+degraded, 7 active+recovery_wait+degraded, 40 peering, 2 undersized+peered, 10 active+undersized, 4 undersized+degraded+peered, 15 active+undersized+degraded, 45 active+clean; 19 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 4.4 MiB/s wr, 538 op/s; 3737/42934 objects degraded (8.704%); 160 MiB/s, 40 objects/s recovering 2023-03-09T14:53:01.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:01 smithi122 ceph-mon[93869]: pgmap v293: 129 pgs: 1 down+remapped, 1 remapped+peering, 1 stale+active+undersized, 1 active+recovering+degraded, 3 active+recovery_wait+degraded, 7 peering, 8 undersized+peered, 24 active+undersized, 13 undersized+degraded+peered, 22 active+undersized+degraded, 48 active+clean; 19 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 4.2 MiB/s wr, 489 op/s; 8159/43182 objects degraded (18.894%); 245 MiB/s, 62 objects/s recovering 2023-03-09T14:53:01.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:01 smithi122 ceph-mon[93869]: from='osd.7 [v2:172.21.15.122:6816/2149044862,v1:172.21.15.122:6817/2149044862]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-03-09T14:53:01.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:01 smithi122 ceph-mon[93869]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-03-09T14:53:01.979 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:01 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 2023-03-09T14:53:01.715+0000 7f12b15b7640 -1 osd.7 160 log_to_monitors true 2023-03-09T14:53:01.979 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:01 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 2023-03-09T14:53:01.921+0000 7f129e32b700 -1 osd.7 160 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-09T14:53:02.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:01 smithi181 ceph-mon[94189]: pgmap v293: 129 pgs: 1 down+remapped, 1 remapped+peering, 1 stale+active+undersized, 1 active+recovering+degraded, 3 active+recovery_wait+degraded, 7 peering, 8 undersized+peered, 24 active+undersized, 13 undersized+degraded+peered, 22 active+undersized+degraded, 48 active+clean; 19 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 4.2 MiB/s wr, 489 op/s; 8159/43182 objects degraded (18.894%); 245 MiB/s, 62 objects/s recovering 2023-03-09T14:53:02.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:01 smithi181 ceph-mon[94189]: from='osd.7 [v2:172.21.15.122:6816/2149044862,v1:172.21.15.122:6817/2149044862]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-03-09T14:53:02.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:01 smithi181 ceph-mon[94189]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-03-09T14:53:02.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:01 smithi038 ceph-mon[92422]: pgmap v293: 129 pgs: 1 down+remapped, 1 remapped+peering, 1 stale+active+undersized, 1 active+recovering+degraded, 3 active+recovery_wait+degraded, 7 peering, 8 undersized+peered, 24 active+undersized, 13 undersized+degraded+peered, 22 active+undersized+degraded, 48 active+clean; 19 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 4.2 MiB/s wr, 489 op/s; 8159/43182 objects degraded (18.894%); 245 MiB/s, 62 objects/s recovering 2023-03-09T14:53:02.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:01 smithi038 ceph-mon[92422]: from='osd.7 [v2:172.21.15.122:6816/2149044862,v1:172.21.15.122:6817/2149044862]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-03-09T14:53:02.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:01 smithi038 ceph-mon[92422]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-03-09T14:53:02.477 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:53:02 smithi122 systemd[1]: ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.1.service: Failed with result 'exit-code'. 2023-03-09T14:53:03.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:02 smithi122 ceph-mon[93869]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-03-09T14:53:03.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:02 smithi122 ceph-mon[93869]: osdmap e178: 12 total, 9 up, 12 in 2023-03-09T14:53:03.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:02 smithi122 ceph-mon[93869]: from='osd.7 [v2:172.21.15.122:6816/2149044862,v1:172.21.15.122:6817/2149044862]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-03-09T14:53:03.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:02 smithi122 ceph-mon[93869]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-03-09T14:53:03.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:02 smithi122 ceph-mon[93869]: Health check update: Reduced data availability: 6 pgs inactive, 1 pg down, 2 pgs peering (PG_AVAILABILITY) 2023-03-09T14:53:03.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:02 smithi122 ceph-mon[93869]: Health check update: Degraded data redundancy: 8159/43182 objects degraded (18.894%), 39 pgs degraded (PG_DEGRADED) 2023-03-09T14:53:03.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:02 smithi181 ceph-mon[94189]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-03-09T14:53:03.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:02 smithi181 ceph-mon[94189]: osdmap e178: 12 total, 9 up, 12 in 2023-03-09T14:53:03.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:02 smithi181 ceph-mon[94189]: from='osd.7 [v2:172.21.15.122:6816/2149044862,v1:172.21.15.122:6817/2149044862]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-03-09T14:53:03.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:02 smithi181 ceph-mon[94189]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-03-09T14:53:03.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:02 smithi181 ceph-mon[94189]: Health check update: Reduced data availability: 6 pgs inactive, 1 pg down, 2 pgs peering (PG_AVAILABILITY) 2023-03-09T14:53:03.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:02 smithi181 ceph-mon[94189]: Health check update: Degraded data redundancy: 8159/43182 objects degraded (18.894%), 39 pgs degraded (PG_DEGRADED) 2023-03-09T14:53:03.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:02 smithi038 ceph-mon[92422]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-03-09T14:53:03.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:02 smithi038 ceph-mon[92422]: osdmap e178: 12 total, 9 up, 12 in 2023-03-09T14:53:03.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:02 smithi038 ceph-mon[92422]: from='osd.7 [v2:172.21.15.122:6816/2149044862,v1:172.21.15.122:6817/2149044862]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-03-09T14:53:03.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:02 smithi038 ceph-mon[92422]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-03-09T14:53:03.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:02 smithi038 ceph-mon[92422]: Health check update: Reduced data availability: 6 pgs inactive, 1 pg down, 2 pgs peering (PG_AVAILABILITY) 2023-03-09T14:53:03.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:02 smithi038 ceph-mon[92422]: Health check update: Degraded data redundancy: 8159/43182 objects degraded (18.894%), 39 pgs degraded (PG_DEGRADED) 2023-03-09T14:53:03.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 155.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T14:53:04.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:03 smithi122 ceph-mon[93869]: pgmap v295: 129 pgs: 1 down+remapped, 1 stale+active+undersized, 1 active+recovering+degraded, 2 active+recovery_wait+degraded, 9 undersized+peered, 27 active+undersized, 14 undersized+degraded+peered, 25 active+undersized+degraded, 49 active+clean; 19 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 1.6 MiB/s wr, 170 op/s; 9117/43134 objects degraded (21.136%); 256 MiB/s, 65 objects/s recovering 2023-03-09T14:53:04.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:03 smithi122 ceph-mon[93869]: Health check update: 2 osds down (OSD_DOWN) 2023-03-09T14:53:04.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:03 smithi122 ceph-mon[93869]: osd.7 [v2:172.21.15.122:6816/2149044862,v1:172.21.15.122:6817/2149044862] boot 2023-03-09T14:53:04.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:03 smithi122 ceph-mon[93869]: osdmap e179: 12 total, 10 up, 12 in 2023-03-09T14:53:04.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:03 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-03-09T14:53:04.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:03 smithi181 ceph-mon[94189]: pgmap v295: 129 pgs: 1 down+remapped, 1 stale+active+undersized, 1 active+recovering+degraded, 2 active+recovery_wait+degraded, 9 undersized+peered, 27 active+undersized, 14 undersized+degraded+peered, 25 active+undersized+degraded, 49 active+clean; 19 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 1.6 MiB/s wr, 170 op/s; 9117/43134 objects degraded (21.136%); 256 MiB/s, 65 objects/s recovering 2023-03-09T14:53:04.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:03 smithi181 ceph-mon[94189]: Health check update: 2 osds down (OSD_DOWN) 2023-03-09T14:53:04.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:03 smithi181 ceph-mon[94189]: osd.7 [v2:172.21.15.122:6816/2149044862,v1:172.21.15.122:6817/2149044862] boot 2023-03-09T14:53:04.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:03 smithi181 ceph-mon[94189]: osdmap e179: 12 total, 10 up, 12 in 2023-03-09T14:53:04.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:03 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-03-09T14:53:04.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:03 smithi038 ceph-mon[92422]: pgmap v295: 129 pgs: 1 down+remapped, 1 stale+active+undersized, 1 active+recovering+degraded, 2 active+recovery_wait+degraded, 9 undersized+peered, 27 active+undersized, 14 undersized+degraded+peered, 25 active+undersized+degraded, 49 active+clean; 19 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 1.6 MiB/s wr, 170 op/s; 9117/43134 objects degraded (21.136%); 256 MiB/s, 65 objects/s recovering 2023-03-09T14:53:04.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:03 smithi038 ceph-mon[92422]: Health check update: 2 osds down (OSD_DOWN) 2023-03-09T14:53:04.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:03 smithi038 ceph-mon[92422]: osd.7 [v2:172.21.15.122:6816/2149044862,v1:172.21.15.122:6817/2149044862] boot 2023-03-09T14:53:04.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:03 smithi038 ceph-mon[92422]: osdmap e179: 12 total, 10 up, 12 in 2023-03-09T14:53:04.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:03 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-03-09T14:53:05.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:04 smithi122 ceph-mon[93869]: osdmap e180: 12 total, 10 up, 12 in 2023-03-09T14:53:05.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:04 smithi122 ceph-mon[93869]: osdmap e181: 12 total, 10 up, 12 in 2023-03-09T14:53:05.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:04 smithi181 ceph-mon[94189]: osdmap e180: 12 total, 10 up, 12 in 2023-03-09T14:53:05.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:04 smithi181 ceph-mon[94189]: osdmap e181: 12 total, 10 up, 12 in 2023-03-09T14:53:05.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:04 smithi038 ceph-mon[92422]: osdmap e180: 12 total, 10 up, 12 in 2023-03-09T14:53:05.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:04 smithi038 ceph-mon[92422]: osdmap e181: 12 total, 10 up, 12 in 2023-03-09T14:53:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:05 smithi122 ceph-mon[93869]: pgmap v298: 129 pgs: 2 remapped+peering, 1 down+remapped, 1 stale+active+undersized, 1 active+recovering+degraded, 9 undersized+peered, 26 active+undersized, 13 undersized+degraded+peered, 24 active+undersized+degraded, 52 active+clean; 19 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 623 KiB/s wr, 43 op/s; 7974/43134 objects degraded (18.487%); 218 MiB/s, 55 objects/s recovering 2023-03-09T14:53:06.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:05 smithi181 ceph-mon[94189]: pgmap v298: 129 pgs: 2 remapped+peering, 1 down+remapped, 1 stale+active+undersized, 1 active+recovering+degraded, 9 undersized+peered, 26 active+undersized, 13 undersized+degraded+peered, 24 active+undersized+degraded, 52 active+clean; 19 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 623 KiB/s wr, 43 op/s; 7974/43134 objects degraded (18.487%); 218 MiB/s, 55 objects/s recovering 2023-03-09T14:53:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:05 smithi038 ceph-mon[92422]: pgmap v298: 129 pgs: 2 remapped+peering, 1 down+remapped, 1 stale+active+undersized, 1 active+recovering+degraded, 9 undersized+peered, 26 active+undersized, 13 undersized+degraded+peered, 24 active+undersized+degraded, 52 active+clean; 19 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 623 KiB/s wr, 43 op/s; 7974/43134 objects degraded (18.487%); 218 MiB/s, 55 objects/s recovering 2023-03-09T14:53:06.981 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:53:06 smithi181 systemd[1]: ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.8.service: Service RestartSec=10s expired, scheduling restart. 2023-03-09T14:53:06.982 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:53:06 smithi181 systemd[1]: ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.8.service: Scheduled restart job, restart counter is at 3. 2023-03-09T14:53:06.982 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:53:06 smithi181 systemd[1]: Stopped Ceph osd.8 for c6733fe2-be87-11ed-9af2-001a4aab830c. 2023-03-09T14:53:06.982 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:53:06 smithi181 systemd[1]: Starting Ceph osd.8 for c6733fe2-be87-11ed-9af2-001a4aab830c... 2023-03-09T14:53:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:07 smithi122 ceph-mon[93869]: pgmap v300: 129 pgs: 6 activating+undersized+degraded+remapped, 2 remapped+peering, 1 down+remapped, 1 stale+active+undersized, 8 undersized+peered, 26 active+undersized, 1 recovery_wait+undersized+degraded+remapped+peered, 8 undersized+degraded+peered, 19 active+undersized+degraded, 57 active+clean; 19 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 5.6 KiB/s wr, 6 op/s; 6238/43134 objects degraded (14.462%); 38 MiB/s, 4 keys/s, 9 objects/s recovering 2023-03-09T14:53:08.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:07 smithi122 ceph-mon[93869]: Health check update: Reduced data availability: 6 pgs inactive, 1 pg down (PG_AVAILABILITY) 2023-03-09T14:53:08.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:07 smithi122 ceph-mon[93869]: Health check update: Degraded data redundancy: 6238/43134 objects degraded (14.462%), 34 pgs degraded (PG_DEGRADED) 2023-03-09T14:53:08.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:07 smithi122 ceph-mon[93869]: osdmap e182: 12 total, 10 up, 12 in 2023-03-09T14:53:08.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:07 smithi181 ceph-mon[94189]: pgmap v300: 129 pgs: 6 activating+undersized+degraded+remapped, 2 remapped+peering, 1 down+remapped, 1 stale+active+undersized, 8 undersized+peered, 26 active+undersized, 1 recovery_wait+undersized+degraded+remapped+peered, 8 undersized+degraded+peered, 19 active+undersized+degraded, 57 active+clean; 19 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 5.6 KiB/s wr, 6 op/s; 6238/43134 objects degraded (14.462%); 38 MiB/s, 4 keys/s, 9 objects/s recovering 2023-03-09T14:53:08.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:07 smithi181 ceph-mon[94189]: Health check update: Reduced data availability: 6 pgs inactive, 1 pg down (PG_AVAILABILITY) 2023-03-09T14:53:08.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:07 smithi181 ceph-mon[94189]: Health check update: Degraded data redundancy: 6238/43134 objects degraded (14.462%), 34 pgs degraded (PG_DEGRADED) 2023-03-09T14:53:08.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:07 smithi181 ceph-mon[94189]: osdmap e182: 12 total, 10 up, 12 in 2023-03-09T14:53:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:07 smithi038 ceph-mon[92422]: pgmap v300: 129 pgs: 6 activating+undersized+degraded+remapped, 2 remapped+peering, 1 down+remapped, 1 stale+active+undersized, 8 undersized+peered, 26 active+undersized, 1 recovery_wait+undersized+degraded+remapped+peered, 8 undersized+degraded+peered, 19 active+undersized+degraded, 57 active+clean; 19 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 5.6 KiB/s wr, 6 op/s; 6238/43134 objects degraded (14.462%); 38 MiB/s, 4 keys/s, 9 objects/s recovering 2023-03-09T14:53:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:07 smithi038 ceph-mon[92422]: Health check update: Reduced data availability: 6 pgs inactive, 1 pg down (PG_AVAILABILITY) 2023-03-09T14:53:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:07 smithi038 ceph-mon[92422]: Health check update: Degraded data redundancy: 6238/43134 objects degraded (14.462%), 34 pgs degraded (PG_DEGRADED) 2023-03-09T14:53:08.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:07 smithi038 ceph-mon[92422]: osdmap e182: 12 total, 10 up, 12 in 2023-03-09T14:53:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 160.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T14:53:09.679 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:53:09 smithi181 bash[128796]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-8 2023-03-09T14:53:09.679 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:53:09 smithi181 bash[128796]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-8 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2023-03-09T14:53:09.679 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:53:09 smithi181 bash[128796]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2023-03-09T14:53:09.679 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:53:09 smithi181 bash[128796]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2023-03-09T14:53:09.680 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:53:09 smithi181 bash[128796]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-8/block 2023-03-09T14:53:09.680 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:53:09 smithi181 bash[128796]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-8 2023-03-09T14:53:09.680 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:53:09 smithi181 bash[128796]: --> ceph-volume raw activate successful for osd ID: 8 2023-03-09T14:53:09.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:09 smithi122 ceph-mon[93869]: osdmap e183: 12 total, 10 up, 12 in 2023-03-09T14:53:09.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:09 smithi122 ceph-mon[93869]: pgmap v303: 129 pgs: 1 down, 6 activating+undersized+degraded+remapped, 1 down+remapped, 2 remapped+peering, 3 undersized+peered, 25 active+undersized, 2 active+recovery_wait+undersized+degraded+remapped, 2 recovery_wait+undersized+degraded+remapped+peered, 2 undersized+degraded+peered, 21 active+undersized+degraded, 64 active+clean; 19 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 33 KiB/s wr, 8 op/s; 5376/43131 objects degraded (12.464%); 5.9 MiB/s, 8 keys/s, 2 objects/s recovering 2023-03-09T14:53:09.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:09 smithi181 ceph-mon[94189]: osdmap e183: 12 total, 10 up, 12 in 2023-03-09T14:53:09.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:09 smithi181 ceph-mon[94189]: pgmap v303: 129 pgs: 1 down, 6 activating+undersized+degraded+remapped, 1 down+remapped, 2 remapped+peering, 3 undersized+peered, 25 active+undersized, 2 active+recovery_wait+undersized+degraded+remapped, 2 recovery_wait+undersized+degraded+remapped+peered, 2 undersized+degraded+peered, 21 active+undersized+degraded, 64 active+clean; 19 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 33 KiB/s wr, 8 op/s; 5376/43131 objects degraded (12.464%); 5.9 MiB/s, 8 keys/s, 2 objects/s recovering 2023-03-09T14:53:10.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:09 smithi038 ceph-mon[92422]: osdmap e183: 12 total, 10 up, 12 in 2023-03-09T14:53:10.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:09 smithi038 ceph-mon[92422]: pgmap v303: 129 pgs: 1 down, 6 activating+undersized+degraded+remapped, 1 down+remapped, 2 remapped+peering, 3 undersized+peered, 25 active+undersized, 2 active+recovery_wait+undersized+degraded+remapped, 2 recovery_wait+undersized+degraded+remapped+peered, 2 undersized+degraded+peered, 21 active+undersized+degraded, 64 active+clean; 19 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 33 KiB/s wr, 8 op/s; 5376/43131 objects degraded (12.464%); 5.9 MiB/s, 8 keys/s, 2 objects/s recovering 2023-03-09T14:53:10.296 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:53:10 smithi181 podman[129078]: 2023-03-09T14:53:10.297 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:53:10 smithi181 ceph-osd[129119]: -- 172.21.15.181:0/1339198443 <== mon.1 v2:172.21.15.181:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x563edd5c89c0 con 0x563edd591400 2023-03-09T14:53:10.297 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:53:10 smithi181 bash[129078]: cb161d5f516085b1820358ae09f751a1f07a46f6a03c9c08a6cbe6c939d5288b 2023-03-09T14:53:10.680 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:53:10 smithi181 systemd[1]: Started Ceph osd.8 for c6733fe2-be87-11ed-9af2-001a4aab830c. 2023-03-09T14:53:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:10 smithi122 ceph-mon[93869]: osdmap e184: 12 total, 10 up, 12 in 2023-03-09T14:53:10.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:10 smithi181 ceph-mon[94189]: osdmap e184: 12 total, 10 up, 12 in 2023-03-09T14:53:10.981 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:53:10 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[129095]: 2023-03-09T14:53:10.747+0000 7fbb316b7640 -1 Falling back to public interface 2023-03-09T14:53:11.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:10 smithi038 ceph-mon[92422]: osdmap e184: 12 total, 10 up, 12 in 2023-03-09T14:53:11.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:11 smithi122 ceph-mon[93869]: osdmap e185: 12 total, 10 up, 12 in 2023-03-09T14:53:11.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:11 smithi122 ceph-mon[93869]: pgmap v306: 129 pgs: 1 peering, 1 down, 1 down+remapped, 3 undersized+peered, 25 active+undersized, 1 recovering+undersized+degraded+remapped+peered, 6 active+recovery_wait+undersized+degraded+remapped, 3 recovery_wait+undersized+degraded+remapped+peered, 2 undersized+degraded+peered, 22 active+undersized+degraded, 64 active+clean; 19 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 43 KiB/s wr, 11 op/s; 5355/43131 objects degraded (12.416%); 203 MiB/s, 4 keys/s, 51 objects/s recovering 2023-03-09T14:53:11.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:11 smithi181 ceph-mon[94189]: osdmap e185: 12 total, 10 up, 12 in 2023-03-09T14:53:11.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:11 smithi181 ceph-mon[94189]: pgmap v306: 129 pgs: 1 peering, 1 down, 1 down+remapped, 3 undersized+peered, 25 active+undersized, 1 recovering+undersized+degraded+remapped+peered, 6 active+recovery_wait+undersized+degraded+remapped, 3 recovery_wait+undersized+degraded+remapped+peered, 2 undersized+degraded+peered, 22 active+undersized+degraded, 64 active+clean; 19 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 43 KiB/s wr, 11 op/s; 5355/43131 objects degraded (12.416%); 203 MiB/s, 4 keys/s, 51 objects/s recovering 2023-03-09T14:53:12.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:11 smithi038 ceph-mon[92422]: osdmap e185: 12 total, 10 up, 12 in 2023-03-09T14:53:12.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:11 smithi038 ceph-mon[92422]: pgmap v306: 129 pgs: 1 peering, 1 down, 1 down+remapped, 3 undersized+peered, 25 active+undersized, 1 recovering+undersized+degraded+remapped+peered, 6 active+recovery_wait+undersized+degraded+remapped, 3 recovery_wait+undersized+degraded+remapped+peered, 2 undersized+degraded+peered, 22 active+undersized+degraded, 64 active+clean; 19 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 43 KiB/s wr, 11 op/s; 5355/43131 objects degraded (12.416%); 203 MiB/s, 4 keys/s, 51 objects/s recovering 2023-03-09T14:53:12.690 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:53:12 smithi122 systemd[1]: ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.1.service: Service RestartSec=10s expired, scheduling restart. 2023-03-09T14:53:12.691 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:53:12 smithi122 systemd[1]: ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.1.service: Scheduled restart job, restart counter is at 4. 2023-03-09T14:53:12.691 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:53:12 smithi122 systemd[1]: Stopped Ceph osd.1 for c6733fe2-be87-11ed-9af2-001a4aab830c. 2023-03-09T14:53:12.691 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:53:12 smithi122 systemd[1]: Starting Ceph osd.1 for c6733fe2-be87-11ed-9af2-001a4aab830c... 2023-03-09T14:53:12.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:12 smithi122 ceph-mon[93869]: osdmap e186: 12 total, 10 up, 12 in 2023-03-09T14:53:12.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:12 smithi122 ceph-mon[93869]: Health check update: Reduced data availability: 2 pgs inactive, 2 pgs down (PG_AVAILABILITY) 2023-03-09T14:53:12.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:12 smithi122 ceph-mon[93869]: Health check update: Degraded data redundancy: 5355/43131 objects degraded (12.416%), 34 pgs degraded (PG_DEGRADED) 2023-03-09T14:53:12.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:12 smithi181 ceph-mon[94189]: osdmap e186: 12 total, 10 up, 12 in 2023-03-09T14:53:12.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:12 smithi181 ceph-mon[94189]: Health check update: Reduced data availability: 2 pgs inactive, 2 pgs down (PG_AVAILABILITY) 2023-03-09T14:53:12.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:12 smithi181 ceph-mon[94189]: Health check update: Degraded data redundancy: 5355/43131 objects degraded (12.416%), 34 pgs degraded (PG_DEGRADED) 2023-03-09T14:53:13.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:12 smithi038 ceph-mon[92422]: osdmap e186: 12 total, 10 up, 12 in 2023-03-09T14:53:13.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:12 smithi038 ceph-mon[92422]: Health check update: Reduced data availability: 2 pgs inactive, 2 pgs down (PG_AVAILABILITY) 2023-03-09T14:53:13.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:12 smithi038 ceph-mon[92422]: Health check update: Degraded data redundancy: 5355/43131 objects degraded (12.416%), 34 pgs degraded (PG_DEGRADED) 2023-03-09T14:53:13.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:13 smithi122 ceph-mon[93869]: osdmap e187: 12 total, 10 up, 12 in 2023-03-09T14:53:13.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:13 smithi122 ceph-mon[93869]: pgmap v309: 129 pgs: 1 down+remapped, 1 down, 1 peering, 3 undersized+peered, 25 active+undersized, 1 recovering+undersized+degraded+remapped+peered, 6 active+recovery_wait+undersized+degraded+remapped, 3 recovery_wait+undersized+degraded+remapped+peered, 2 undersized+degraded+peered, 22 active+undersized+degraded, 64 active+clean; 19 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 16 KiB/s wr, 8 op/s; 5355/43131 objects degraded (12.416%); 203 MiB/s, 50 objects/s recovering 2023-03-09T14:53:13.728 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:53:13 smithi122 bash[130268]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-03-09T14:53:13.728 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:53:13 smithi122 bash[130268]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2023-03-09T14:53:13.728 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:53:13 smithi122 bash[130268]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2023-03-09T14:53:13.728 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:53:13 smithi122 bash[130268]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2023-03-09T14:53:13.729 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:53:13 smithi122 bash[130268]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-1/block 2023-03-09T14:53:13.729 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:53:13 smithi122 bash[130268]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-03-09T14:53:13.729 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:53:13 smithi122 bash[130268]: --> ceph-volume raw activate successful for osd ID: 1 2023-03-09T14:53:13.788 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-03-09T14:53:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 165.0 s, 0.2 tps, lat 15994.975 ms stddev -nan 2023-03-09T14:53:13.927 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.922+0000 7f3dee8b7700 1 -- 172.21.15.38:0/1030162540 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3de8104040 msgr2=0x7f3de810bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:53:13.927 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.922+0000 7f3dee8b7700 1 --2- 172.21.15.38:0/1030162540 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3de8104040 0x7f3de810bc10 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f3de000b0d0 tx=0x7f3de000b3e0 comp rx=0 tx=0).stop 2023-03-09T14:53:13.927 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.922+0000 7f3dee8b7700 1 -- 172.21.15.38:0/1030162540 shutdown_connections 2023-03-09T14:53:13.927 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.922+0000 7f3dee8b7700 1 --2- 172.21.15.38:0/1030162540 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3de8104040 0x7f3de810bc10 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:53:13.928 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.922+0000 7f3dee8b7700 1 --2- 172.21.15.38:0/1030162540 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3de80ff000 0x7f3de80ff3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:53:13.928 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.922+0000 7f3dee8b7700 1 --2- 172.21.15.38:0/1030162540 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3de80ff9a0 0x7f3de8103960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:53:13.928 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.922+0000 7f3dee8b7700 1 -- 172.21.15.38:0/1030162540 >> 172.21.15.38:0/1030162540 conn(0x7f3de8075c60 msgr2=0x7f3de8076060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:53:13.928 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.922+0000 7f3dee8b7700 1 -- 172.21.15.38:0/1030162540 shutdown_connections 2023-03-09T14:53:13.928 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.922+0000 7f3dee8b7700 1 -- 172.21.15.38:0/1030162540 wait complete. 2023-03-09T14:53:13.928 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.923+0000 7f3dee8b7700 1 Processor -- start 2023-03-09T14:53:13.929 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.923+0000 7f3dee8b7700 1 -- start start 2023-03-09T14:53:13.929 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.923+0000 7f3dee8b7700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3de80ff000 0x7f3de81950b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:53:13.929 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.923+0000 7f3dee8b7700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3de80ff9a0 0x7f3de81955f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:53:13.930 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.923+0000 7f3dee8b7700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3de8104040 0x7f3de8199a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:53:13.930 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.923+0000 7f3dee8b7700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3de819a090 con 0x7f3de8104040 2023-03-09T14:53:13.930 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.923+0000 7f3dee8b7700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3de819a200 con 0x7f3de80ff9a0 2023-03-09T14:53:13.931 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.923+0000 7f3dee8b7700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3de819a3c0 con 0x7f3de80ff000 2023-03-09T14:53:13.931 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.923+0000 7f3de7fff700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3de80ff000 0x7f3de81950b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:53:13.931 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.923+0000 7f3de7fff700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3de80ff000 0x7f3de81950b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.38:49296/0 (socket says 172.21.15.38:49296) 2023-03-09T14:53:13.931 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.923+0000 7f3de7fff700 1 -- 172.21.15.38:0/942193768 learned_addr learned my addr 172.21.15.38:0/942193768 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:53:13.932 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.923+0000 7f3dece54700 1 --2- 172.21.15.38:0/942193768 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3de8104040 0x7f3de8199a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:53:13.932 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.923+0000 7f3de77fe700 1 --2- 172.21.15.38:0/942193768 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3de80ff9a0 0x7f3de81955f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:53:13.932 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.923+0000 7f3de7fff700 1 -- 172.21.15.38:0/942193768 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3de80ff9a0 msgr2=0x7f3de81955f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:53:13.932 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.923+0000 7f3de7fff700 1 --2- 172.21.15.38:0/942193768 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3de80ff9a0 0x7f3de81955f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:53:13.932 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.923+0000 7f3de7fff700 1 -- 172.21.15.38:0/942193768 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3de8104040 msgr2=0x7f3de8199a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:53:13.932 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.923+0000 7f3de7fff700 1 --2- 172.21.15.38:0/942193768 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3de8104040 0x7f3de8199a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:53:13.933 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.923+0000 7f3de7fff700 1 -- 172.21.15.38:0/942193768 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3de0009d00 con 0x7f3de80ff000 2023-03-09T14:53:13.933 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.923+0000 7f3dece54700 1 --2- 172.21.15.38:0/942193768 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3de8104040 0x7f3de8199a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:53:13.933 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.924+0000 7f3de7fff700 1 --2- 172.21.15.38:0/942193768 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3de80ff000 0x7f3de81950b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f3dd800e9a0 tx=0x7f3dd800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:53:13.933 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.924+0000 7f3de57fa700 1 -- 172.21.15.38:0/942193768 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dd8009790 con 0x7f3de80ff000 2023-03-09T14:53:13.933 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.924+0000 7f3dee8b7700 1 -- 172.21.15.38:0/942193768 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3de810ee60 con 0x7f3de80ff000 2023-03-09T14:53:13.933 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.924+0000 7f3dee8b7700 1 -- 172.21.15.38:0/942193768 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3de810ecd0 con 0x7f3de80ff000 2023-03-09T14:53:13.934 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.924+0000 7f3de57fa700 1 -- 172.21.15.38:0/942193768 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3dd8010430 con 0x7f3de80ff000 2023-03-09T14:53:13.934 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.924+0000 7f3de57fa700 1 -- 172.21.15.38:0/942193768 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dd8018680 con 0x7f3de80ff000 2023-03-09T14:53:13.939 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.926+0000 7f3de57fa700 1 -- 172.21.15.38:0/942193768 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 25) v1 ==== 133295+0+0 (secure 0 0 0) 0x7f3dd800c040 con 0x7f3de80ff000 2023-03-09T14:53:13.939 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.927+0000 7f3de57fa700 1 --2- 172.21.15.38:0/942193768 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f3dd009f4a0 0x7f3dd00a1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:53:13.939 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.927+0000 7f3de57fa700 1 -- 172.21.15.38:0/942193768 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 9785+0+0 (secure 0 0 0) 0x7f3dd8014070 con 0x7f3de80ff000 2023-03-09T14:53:13.939 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.927+0000 7f3de77fe700 1 --2- 172.21.15.38:0/942193768 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f3dd009f4a0 0x7f3dd00a1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:53:13.940 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.927+0000 7f3dfaa40b80 1 Processor -- start 2023-03-09T14:53:13.940 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.927+0000 7f3dfaa40b80 1 -- start start 2023-03-09T14:53:13.940 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.927+0000 7f3dfaa40b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c8e0719ea0 0x55c8e071a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:53:13.940 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.928+0000 7f3dfaa40b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c8e071a7b0 0x55c8e071e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:53:13.941 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.928+0000 7f3dfaa40b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c8e071ec10 0x55c8e071f020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:53:13.941 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.928+0000 7f3dfaa40b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8e0601da0 con 0x55c8e071a7b0 2023-03-09T14:53:13.941 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.928+0000 7f3dfaa40b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8e0670780 con 0x55c8e0719ea0 2023-03-09T14:53:13.942 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.928+0000 7f3dfaa40b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8e0669cb0 con 0x55c8e071ec10 2023-03-09T14:53:13.942 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.928+0000 7f3de77fe700 1 --2- 172.21.15.38:0/942193768 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f3dd009f4a0 0x7f3dd00a1950 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3de81966c0 tx=0x7f3ddc009260 comp rx=0 tx=0).ready entity=mgr.15558 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:53:13.942 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.928+0000 7f3de7fff700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c8e0719ea0 0x55c8e071a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:53:13.943 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.928+0000 7f3de77fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c8e071a7b0 0x55c8e071e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:53:13.943 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.928+0000 7f3de7fff700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c8e0719ea0 0x55c8e071a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.38:50796/0 (socket says 172.21.15.38:50796) 2023-03-09T14:53:13.943 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.928+0000 7f3de7fff700 1 -- 172.21.15.38:0/1004421148 learned_addr learned my addr 172.21.15.38:0/1004421148 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:53:13.943 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.928+0000 7f3dece54700 1 --2- 172.21.15.38:0/1004421148 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c8e071ec10 0x55c8e071f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:53:13.943 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.928+0000 7f3de77fe700 1 -- 172.21.15.38:0/1004421148 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c8e071ec10 msgr2=0x55c8e071f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:53:13.944 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.928+0000 7f3de77fe700 1 --2- 172.21.15.38:0/1004421148 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c8e071ec10 0x55c8e071f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:53:13.944 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.928+0000 7f3de77fe700 1 -- 172.21.15.38:0/1004421148 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c8e0719ea0 msgr2=0x55c8e071a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:53:13.944 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.928+0000 7f3de77fe700 1 --2- 172.21.15.38:0/1004421148 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c8e0719ea0 0x55c8e071a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:53:13.944 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.928+0000 7f3de77fe700 1 -- 172.21.15.38:0/1004421148 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c8e0630190 con 0x55c8e071a7b0 2023-03-09T14:53:13.945 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.928+0000 7f3de7fff700 1 --2- 172.21.15.38:0/1004421148 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c8e0719ea0 0x55c8e071a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:53:13.945 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.928+0000 7f3de77fe700 1 --2- 172.21.15.38:0/1004421148 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c8e071a7b0 0x55c8e071e6d0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f3de80ffd70 tx=0x7f3ddc014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7398b7d69039a652 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:53:13.945 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.928+0000 7f3dce7fc700 1 -- 172.21.15.38:0/1004421148 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ddc004050 con 0x55c8e071a7b0 2023-03-09T14:53:13.946 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.928+0000 7f3dce7fc700 1 -- 172.21.15.38:0/1004421148 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3ddc00e840 con 0x55c8e071a7b0 2023-03-09T14:53:13.946 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.928+0000 7f3dce7fc700 1 -- 172.21.15.38:0/1004421148 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ddc003a80 con 0x55c8e071a7b0 2023-03-09T14:53:13.946 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.929+0000 7f3dfaa40b80 1 -- 172.21.15.38:0/1004421148 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c8e071a7b0 msgr2=0x55c8e071e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:53:13.947 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.929+0000 7f3dfaa40b80 1 --2- 172.21.15.38:0/1004421148 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c8e071a7b0 0x55c8e071e6d0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f3de80ffd70 tx=0x7f3ddc014000 comp rx=0 tx=0).stop 2023-03-09T14:53:13.947 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.929+0000 7f3dfaa40b80 1 -- 172.21.15.38:0/1004421148 shutdown_connections 2023-03-09T14:53:13.947 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.929+0000 7f3dfaa40b80 1 --2- 172.21.15.38:0/1004421148 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c8e071ec10 0x55c8e071f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:53:13.947 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.929+0000 7f3dfaa40b80 1 --2- 172.21.15.38:0/1004421148 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c8e0719ea0 0x55c8e071a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:53:13.948 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.929+0000 7f3dfaa40b80 1 --2- 172.21.15.38:0/1004421148 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c8e071a7b0 0x55c8e071e6d0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:53:13.948 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.929+0000 7f3dfaa40b80 1 -- 172.21.15.38:0/1004421148 >> 172.21.15.38:0/1004421148 conn(0x55c8e0712900 msgr2=0x55c8e0712cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:53:13.948 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.929+0000 7f3dfaa40b80 1 -- 172.21.15.38:0/1004421148 shutdown_connections 2023-03-09T14:53:13.948 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.929+0000 7f3dfaa40b80 1 -- 172.21.15.38:0/1004421148 wait complete. 2023-03-09T14:53:13.949 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.929+0000 7f3dfaa40b80 1 -- start start 2023-03-09T14:53:13.949 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.930+0000 7f3dfaa40b80 1 Processor -- start 2023-03-09T14:53:13.949 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.930+0000 7f3dfaa40b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c8e0719ea0 0x55c8e0726320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:53:13.950 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.930+0000 7f3dfaa40b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c8e071a7b0 0x55c8e0723530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:53:13.950 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.930+0000 7f3dfaa40b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c8e071ec10 0x55c8e0723af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:53:13.950 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.930+0000 7f3dfaa40b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8e04ed920 con 0x55c8e0719ea0 2023-03-09T14:53:13.951 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.930+0000 7f3dfaa40b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8e0724230 con 0x55c8e071ec10 2023-03-09T14:53:13.951 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.930+0000 7f3dfaa40b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8e07243f0 con 0x55c8e071a7b0 2023-03-09T14:53:13.951 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.930+0000 7f3dfaa40b80 10 client.0 _ll_register_callbacks cb 0x55c8e060f520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:53:13.951 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.930+0000 7f3de7fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c8e0719ea0 0x55c8e0726320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:53:13.951 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.930+0000 7f3de7fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c8e0719ea0 0x55c8e0726320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:41612/0 (socket says 172.21.15.38:41612) 2023-03-09T14:53:13.951 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.930+0000 7f3de7fff700 1 -- 172.21.15.38:0/3527960246 learned_addr learned my addr 172.21.15.38:0/3527960246 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:53:13.952 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.930+0000 7f3dece54700 1 --2- 172.21.15.38:0/3527960246 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c8e071ec10 0x55c8e0723af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:53:13.952 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.930+0000 7f3de77fe700 1 --2- 172.21.15.38:0/3527960246 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c8e071a7b0 0x55c8e0723530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:53:13.952 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.930+0000 7f3de7fff700 1 -- 172.21.15.38:0/3527960246 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c8e071a7b0 msgr2=0x55c8e0723530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:53:13.952 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.930+0000 7f3de7fff700 1 --2- 172.21.15.38:0/3527960246 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c8e071a7b0 0x55c8e0723530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:53:13.952 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.930+0000 7f3de7fff700 1 -- 172.21.15.38:0/3527960246 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c8e071ec10 msgr2=0x55c8e0723af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:53:13.953 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.930+0000 7f3de7fff700 1 --2- 172.21.15.38:0/3527960246 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c8e071ec10 0x55c8e0723af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:53:13.953 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.930+0000 7f3de7fff700 1 -- 172.21.15.38:0/3527960246 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3de810ecd0 con 0x55c8e0719ea0 2023-03-09T14:53:13.953 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.931+0000 7f3de7fff700 1 --2- 172.21.15.38:0/3527960246 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c8e0719ea0 0x55c8e0726320 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f3dd8098030 tx=0x7f3dd8010af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3292097e5913313d server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:53:13.953 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.931+0000 7f3dc77fe700 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2023-03-09T14:53:13.953 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.931+0000 7f3dc77fe700 1 -- 172.21.15.38:0/3527960246 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dd80b04b0 con 0x55c8e0719ea0 2023-03-09T14:53:13.954 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.931+0000 7f3dc77fe700 1 -- 172.21.15.38:0/3527960246 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(188..188 src has 1..188) v4 ==== 9785+0+0 (secure 0 0 0) 0x7f3dd80975e0 con 0x55c8e0719ea0 2023-03-09T14:53:13.954 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.931+0000 7f3dfaa40b80 1 -- 172.21.15.38:0/3527960246 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c8e0637190 con 0x55c8e0719ea0 2023-03-09T14:53:13.954 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.931+0000 7f3dc77fe700 1 -- 172.21.15.38:0/3527960246 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3dd80977a0 con 0x55c8e0719ea0 2023-03-09T14:53:13.954 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.931+0000 7f3dfaa40b80 10 client.16140 fetch_fsmap learned FSMap version 31 2023-03-09T14:53:13.954 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.931+0000 7f3dfaa40b80 1 -- 172.21.15.38:0/3527960246 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x7f3ddc017070 con 0x55c8e0719ea0 2023-03-09T14:53:13.954 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.931+0000 7f3dc77fe700 1 -- 172.21.15.38:0/3527960246 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55c8e0637190 con 0x55c8e0719ea0 2023-03-09T14:53:13.955 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.931+0000 7f3dc77fe700 1 -- 172.21.15.38:0/3527960246 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 31) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f3dd80940c0 con 0x55c8e0719ea0 2023-03-09T14:53:13.955 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.931+0000 7f3dfaa40b80 10 client.16140 fetch_fsmap finished waiting for FSMap version 31 2023-03-09T14:53:13.955 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.931+0000 7f3dfaa40b80 10 client.16140 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T14:53:13.955 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.931+0000 7f3dfaa40b80 20 client.16140 populate_metadata read hostname 'smithi038' 2023-03-09T14:53:13.955 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.931+0000 7f3dfaa40b80 1 --2- 172.21.15.38:0/3527960246 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55c8e0733f00 0x55c8e0736310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:53:13.956 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.931+0000 7f3dfaa40b80 4 client.16140 mds_command: new command op to 24530 tid=0[{"prefix": "get_command_descriptions"}] 2023-03-09T14:53:13.956 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.931+0000 7f3dfaa40b80 1 -- 172.21.15.38:0/3527960246 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c8e04edf60 con 0x55c8e0733f00 2023-03-09T14:53:13.956 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.932+0000 7f3de77fe700 1 --2- 172.21.15.38:0/3527960246 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55c8e0733f00 0x55c8e0736310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:53:13.956 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.932+0000 7f3de77fe700 1 --2- 172.21.15.38:0/3527960246 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55c8e0733f00 0x55c8e0736310 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=39decabdbc269395 server_cookie=d528847bd9852afb in_seq=0 out_seq=0 2023-03-09T14:53:13.956 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.932+0000 7f3dc77fe700 10 client.16140 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:53:13.957 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.935+0000 7f3dc77fe700 1 -- 172.21.15.38:0/3527960246 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18278 (crc 0 0 0) 0x55c8e04edf60 con 0x55c8e0733f00 2023-03-09T14:53:13.957 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.935+0000 7f3dc77fe700 10 client.16140 handle_command_reply: tid=0 2023-03-09T14:53:13.957 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.935+0000 7f3dfaa40b80 1 client.16140 shutdown 2023-03-09T14:53:13.957 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.935+0000 7f3dfaa40b80 1 -- 172.21.15.38:0/3527960246 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c8e0719ea0 msgr2=0x55c8e0726320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:53:13.957 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.935+0000 7f3dfaa40b80 1 --2- 172.21.15.38:0/3527960246 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c8e0719ea0 0x55c8e0726320 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f3dd8098030 tx=0x7f3dd8010af0 comp rx=0 tx=0).stop 2023-03-09T14:53:13.958 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.935+0000 7f3dfaa40b80 1 -- 172.21.15.38:0/3527960246 shutdown_connections 2023-03-09T14:53:13.958 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.935+0000 7f3dfaa40b80 1 --2- 172.21.15.38:0/3527960246 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55c8e0733f00 0x55c8e0736310 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:53:13.958 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.935+0000 7f3dfaa40b80 1 --2- 172.21.15.38:0/3527960246 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c8e0719ea0 0x55c8e0726320 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:53:13.958 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.935+0000 7f3dc77fe700 0 client.16140 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:53:13.959 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.935+0000 7f3dfaa40b80 1 --2- 172.21.15.38:0/3527960246 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c8e071ec10 0x55c8e0723af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:53:13.959 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.935+0000 7f3dfaa40b80 1 --2- 172.21.15.38:0/3527960246 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c8e071a7b0 0x55c8e0723530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:53:13.959 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.935+0000 7f3dfaa40b80 1 -- 172.21.15.38:0/3527960246 >> 172.21.15.38:0/3527960246 conn(0x55c8e0712900 msgr2=0x55c8e0711c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:53:13.959 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.935+0000 7f3dfaa40b80 1 -- 172.21.15.38:0/3527960246 shutdown_connections 2023-03-09T14:53:13.959 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.935+0000 7f3dfaa40b80 1 -- 172.21.15.38:0/3527960246 wait complete. 2023-03-09T14:53:13.959 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.935+0000 7f3dfaa40b80 20 client.16140 trim_cache size 0 max 16384 2023-03-09T14:53:13.964 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.953+0000 7f3dfaa40b80 1 Processor -- start 2023-03-09T14:53:13.964 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.953+0000 7f3dfaa40b80 1 -- start start 2023-03-09T14:53:13.964 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.953+0000 7f3dfaa40b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c8e0719ea0 0x55c8e072add0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:53:13.965 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.953+0000 7f3dfaa40b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c8e071a7b0 0x55c8e0720e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:53:13.965 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.953+0000 7f3dfaa40b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c8e071ec10 0x55c8e0725e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:53:13.965 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.953+0000 7f3dfaa40b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8e0733d60 con 0x55c8e071a7b0 2023-03-09T14:53:13.965 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.953+0000 7f3dfaa40b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8e0714190 con 0x55c8e0719ea0 2023-03-09T14:53:13.965 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.953+0000 7f3dfaa40b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8e0721a90 con 0x55c8e071ec10 2023-03-09T14:53:13.965 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.953+0000 7f3de77fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c8e071a7b0 0x55c8e0720e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:53:13.966 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.953+0000 7f3de77fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c8e071a7b0 0x55c8e0720e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:41622/0 (socket says 172.21.15.38:41622) 2023-03-09T14:53:13.966 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.953+0000 7f3de77fe700 1 -- 172.21.15.38:0/4071290679 learned_addr learned my addr 172.21.15.38:0/4071290679 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:53:13.966 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.953+0000 7f3de7fff700 1 --2- 172.21.15.38:0/4071290679 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c8e0719ea0 0x55c8e072add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:53:13.966 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.953+0000 7f3dece54700 1 --2- 172.21.15.38:0/4071290679 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c8e071ec10 0x55c8e0725e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:53:13.966 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.954+0000 7f3de77fe700 1 -- 172.21.15.38:0/4071290679 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c8e071ec10 msgr2=0x55c8e0725e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:53:13.966 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.954+0000 7f3de77fe700 1 --2- 172.21.15.38:0/4071290679 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c8e071ec10 0x55c8e0725e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:53:13.967 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.954+0000 7f3de77fe700 1 -- 172.21.15.38:0/4071290679 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c8e0719ea0 msgr2=0x55c8e072add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:53:13.967 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.954+0000 7f3de77fe700 1 --2- 172.21.15.38:0/4071290679 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c8e0719ea0 0x55c8e072add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:53:13.967 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.954+0000 7f3de77fe700 1 -- 172.21.15.38:0/4071290679 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c8e0480c60 con 0x55c8e071a7b0 2023-03-09T14:53:13.967 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.954+0000 7f3de77fe700 1 --2- 172.21.15.38:0/4071290679 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c8e071a7b0 0x55c8e0720e30 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f3ddc00ef90 tx=0x7f3ddc00e740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fdae0ea365e6c35 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:53:13.967 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.954+0000 7f3dceffd700 1 -- 172.21.15.38:0/4071290679 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ddc02d030 con 0x55c8e071a7b0 2023-03-09T14:53:13.967 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.954+0000 7f3dceffd700 1 -- 172.21.15.38:0/4071290679 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3ddc02e040 con 0x55c8e071a7b0 2023-03-09T14:53:13.967 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.954+0000 7f3dceffd700 1 -- 172.21.15.38:0/4071290679 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ddc003bb0 con 0x55c8e071a7b0 2023-03-09T14:53:13.968 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.955+0000 7f3dfaa40b80 1 -- 172.21.15.38:0/4071290679 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c8e071a7b0 msgr2=0x55c8e0720e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:53:13.968 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.955+0000 7f3dfaa40b80 1 --2- 172.21.15.38:0/4071290679 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c8e071a7b0 0x55c8e0720e30 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f3ddc00ef90 tx=0x7f3ddc00e740 comp rx=0 tx=0).stop 2023-03-09T14:53:13.968 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.955+0000 7f3dfaa40b80 1 -- 172.21.15.38:0/4071290679 shutdown_connections 2023-03-09T14:53:13.968 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.955+0000 7f3dfaa40b80 1 --2- 172.21.15.38:0/4071290679 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c8e071ec10 0x55c8e0725e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:53:13.968 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.955+0000 7f3dfaa40b80 1 --2- 172.21.15.38:0/4071290679 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c8e0719ea0 0x55c8e072add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:53:13.968 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.955+0000 7f3dfaa40b80 1 --2- 172.21.15.38:0/4071290679 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c8e071a7b0 0x55c8e0720e30 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:53:13.968 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.955+0000 7f3dfaa40b80 1 -- 172.21.15.38:0/4071290679 >> 172.21.15.38:0/4071290679 conn(0x55c8e0712900 msgr2=0x55c8e072f570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:53:13.969 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.955+0000 7f3dfaa40b80 1 -- 172.21.15.38:0/4071290679 shutdown_connections 2023-03-09T14:53:13.969 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.955+0000 7f3dfaa40b80 1 -- 172.21.15.38:0/4071290679 wait complete. 2023-03-09T14:53:13.969 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.955+0000 7f3dfaa40b80 1 -- start start 2023-03-09T14:53:13.969 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.955+0000 7f3dfaa40b80 1 Processor -- start 2023-03-09T14:53:13.969 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.956+0000 7f3dfaa40b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c8e0719ea0 0x55c8e0725000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:53:13.969 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.956+0000 7f3dfaa40b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c8e071a7b0 0x55c8e0725540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:53:13.970 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.956+0000 7f3dfaa40b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c8e071ec10 0x55c8e0725a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:53:13.970 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.956+0000 7f3dfaa40b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8e0787de0 con 0x55c8e071ec10 2023-03-09T14:53:13.970 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.956+0000 7f3dfaa40b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8e0787f50 con 0x55c8e071a7b0 2023-03-09T14:53:13.970 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.956+0000 7f3dfaa40b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8e0788110 con 0x55c8e0719ea0 2023-03-09T14:53:13.970 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.956+0000 7f3dfaa40b80 10 client.0 _ll_register_callbacks cb 0x55c8e060f520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:53:13.970 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.956+0000 7f3dece54700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c8e071ec10 0x55c8e0725a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:53:13.970 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.956+0000 7f3dece54700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c8e071ec10 0x55c8e0725a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:41630/0 (socket says 172.21.15.38:41630) 2023-03-09T14:53:13.971 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.956+0000 7f3dece54700 1 -- 172.21.15.38:0/355935641 learned_addr learned my addr 172.21.15.38:0/355935641 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:53:13.971 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.956+0000 7f3de7fff700 1 --2- 172.21.15.38:0/355935641 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c8e0719ea0 0x55c8e0725000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:53:13.971 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.956+0000 7f3de77fe700 1 --2- 172.21.15.38:0/355935641 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c8e071a7b0 0x55c8e0725540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:53:13.971 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.956+0000 7f3dece54700 1 -- 172.21.15.38:0/355935641 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c8e0719ea0 msgr2=0x55c8e0725000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:53:13.971 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.956+0000 7f3dece54700 1 --2- 172.21.15.38:0/355935641 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c8e0719ea0 0x55c8e0725000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:53:13.971 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.956+0000 7f3dece54700 1 -- 172.21.15.38:0/355935641 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c8e071a7b0 msgr2=0x55c8e0725540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:53:13.972 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.956+0000 7f3dece54700 1 --2- 172.21.15.38:0/355935641 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c8e071a7b0 0x55c8e0725540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:53:13.972 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.956+0000 7f3dece54700 1 -- 172.21.15.38:0/355935641 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c8e0630190 con 0x55c8e071ec10 2023-03-09T14:53:13.972 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.956+0000 7f3dece54700 1 --2- 172.21.15.38:0/355935641 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c8e071ec10 0x55c8e0725a80 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f3de000b0d0 tx=0x7f3de0007710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79d8f6393eb85e5a server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:53:13.972 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.956+0000 7f3dc77fe700 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2023-03-09T14:53:13.972 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.956+0000 7f3dc77fe700 1 -- 172.21.15.38:0/355935641 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3de0021040 con 0x55c8e071ec10 2023-03-09T14:53:13.972 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.956+0000 7f3dc77fe700 1 -- 172.21.15.38:0/355935641 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(188..188 src has 1..188) v4 ==== 9785+0+0 (secure 0 0 0) 0x7f3de0003c00 con 0x55c8e071ec10 2023-03-09T14:53:13.972 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.956+0000 7f3dfaa40b80 1 -- 172.21.15.38:0/355935641 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c8e06271b0 con 0x55c8e071ec10 2023-03-09T14:53:13.973 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.957+0000 7f3dc77fe700 1 -- 172.21.15.38:0/355935641 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3de000e040 con 0x55c8e071ec10 2023-03-09T14:53:13.973 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.957+0000 7f3dfaa40b80 10 client.16146 fetch_fsmap learned FSMap version 31 2023-03-09T14:53:13.973 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.957+0000 7f3dfaa40b80 1 -- 172.21.15.38:0/355935641 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55c8e04be250 con 0x55c8e071ec10 2023-03-09T14:53:13.973 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.957+0000 7f3dc77fe700 1 -- 172.21.15.38:0/355935641 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55c8e06271b0 con 0x55c8e071ec10 2023-03-09T14:53:13.973 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.957+0000 7f3dc77fe700 1 -- 172.21.15.38:0/355935641 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 31) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f3de000b100 con 0x55c8e071ec10 2023-03-09T14:53:13.974 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.957+0000 7f3dfaa40b80 10 client.16146 fetch_fsmap finished waiting for FSMap version 31 2023-03-09T14:53:13.974 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.957+0000 7f3dfaa40b80 10 client.16146 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T14:53:13.974 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.957+0000 7f3dfaa40b80 20 client.16146 populate_metadata read hostname 'smithi038' 2023-03-09T14:53:13.974 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.957+0000 7f3dfaa40b80 1 --2- 172.21.15.38:0/355935641 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55c8e0789890 0x55c8e078bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:53:13.974 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.957+0000 7f3dfaa40b80 4 client.16146 mds_command: new command op to 24530 tid=0[{"prefix": "scrub status"}] 2023-03-09T14:53:13.974 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.957+0000 7f3dfaa40b80 1 -- 172.21.15.38:0/355935641 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55c8e06111c0 con 0x55c8e0789890 2023-03-09T14:53:13.975 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.957+0000 7f3de7fff700 1 --2- 172.21.15.38:0/355935641 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55c8e0789890 0x55c8e078bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:53:13.975 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.958+0000 7f3dc77fe700 10 client.16146 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:53:13.975 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.958+0000 7f3de7fff700 1 --2- 172.21.15.38:0/355935641 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55c8e0789890 0x55c8e078bca0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=943da5f365bff67d server_cookie=21a6407f4d104786 in_seq=0 out_seq=0 2023-03-09T14:53:13.975 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.958+0000 7f3dc77fe700 1 -- 172.21.15.38:0/355935641 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+265 (crc 0 0 0) 0x55c8e06111c0 con 0x55c8e0789890 2023-03-09T14:53:13.975 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.958+0000 7f3dc77fe700 10 client.16146 handle_command_reply: tid=0 2023-03-09T14:53:13.975 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.958+0000 7f3dfaa40b80 1 client.16146 shutdown 2023-03-09T14:53:13.975 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.958+0000 7f3dfaa40b80 1 -- 172.21.15.38:0/355935641 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c8e071ec10 msgr2=0x55c8e0725a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:53:13.976 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.958+0000 7f3dfaa40b80 1 --2- 172.21.15.38:0/355935641 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c8e071ec10 0x55c8e0725a80 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f3de000b0d0 tx=0x7f3de0007710 comp rx=0 tx=0).stop 2023-03-09T14:53:13.976 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.958+0000 7f3dfaa40b80 1 -- 172.21.15.38:0/355935641 shutdown_connections 2023-03-09T14:53:13.976 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.958+0000 7f3dfaa40b80 1 --2- 172.21.15.38:0/355935641 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55c8e0789890 0x55c8e078bca0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:53:13.976 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.958+0000 7f3dfaa40b80 1 --2- 172.21.15.38:0/355935641 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55c8e071ec10 0x55c8e0725a80 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:53:13.976 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.958+0000 7f3dfaa40b80 1 --2- 172.21.15.38:0/355935641 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55c8e0719ea0 0x55c8e0725000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:53:13.976 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.958+0000 7f3dfaa40b80 1 --2- 172.21.15.38:0/355935641 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55c8e071a7b0 0x55c8e0725540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:53:13.977 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.958+0000 7f3dfaa40b80 1 -- 172.21.15.38:0/355935641 >> 172.21.15.38:0/355935641 conn(0x55c8e0712900 msgr2=0x55c8e071cc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:53:13.977 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.958+0000 7f3dc77fe700 0 client.16146 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:53:13.977 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.958+0000 7f3dfaa40b80 1 -- 172.21.15.38:0/355935641 shutdown_connections 2023-03-09T14:53:13.977 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.958+0000 7f3dfaa40b80 1 -- 172.21.15.38:0/355935641 wait complete. 2023-03-09T14:53:13.977 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.958+0000 7f3dfaa40b80 20 client.16146 trim_cache size 0 max 16384 2023-03-09T14:53:13.978 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2023-03-09T14:53:13.978 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "scrub active (3 inodes in the stack)", 2023-03-09T14:53:13.978 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": { 2023-03-09T14:53:13.978 INFO:teuthology.orchestra.run.smithi038.stdout: "a14659f4-d7ea-4c5a-b15b-6138af3f909f": { 2023-03-09T14:53:13.978 INFO:teuthology.orchestra.run.smithi038.stdout: "path": "/", 2023-03-09T14:53:13.978 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "a14659f4-d7ea-4c5a-b15b-6138af3f909f", 2023-03-09T14:53:13.979 INFO:teuthology.orchestra.run.smithi038.stdout: "options": "recursive,force" 2023-03-09T14:53:13.979 INFO:teuthology.orchestra.run.smithi038.stdout: } 2023-03-09T14:53:13.979 INFO:teuthology.orchestra.run.smithi038.stdout: } 2023-03-09T14:53:13.979 INFO:teuthology.orchestra.run.smithi038.stdout:} 2023-03-09T14:53:13.980 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.959+0000 7f3dee8b7700 1 -- 172.21.15.38:0/942193768 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f3dd009f4a0 msgr2=0x7f3dd00a1950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:53:13.980 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.959+0000 7f3dee8b7700 1 --2- 172.21.15.38:0/942193768 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f3dd009f4a0 0x7f3dd00a1950 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3de81966c0 tx=0x7f3ddc009260 comp rx=0 tx=0).stop 2023-03-09T14:53:13.980 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.959+0000 7f3dee8b7700 1 -- 172.21.15.38:0/942193768 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3de80ff000 msgr2=0x7f3de81950b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:53:13.981 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.959+0000 7f3dee8b7700 1 --2- 172.21.15.38:0/942193768 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3de80ff000 0x7f3de81950b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f3dd800e9a0 tx=0x7f3dd800ecb0 comp rx=0 tx=0).stop 2023-03-09T14:53:13.981 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.959+0000 7f3dee8b7700 1 -- 172.21.15.38:0/942193768 shutdown_connections 2023-03-09T14:53:13.981 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.959+0000 7f3dee8b7700 1 --2- 172.21.15.38:0/942193768 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f3dd009f4a0 0x7f3dd00a1950 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:53:13.981 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.959+0000 7f3dee8b7700 1 --2- 172.21.15.38:0/942193768 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3de8104040 0x7f3de8199a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:53:13.981 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.959+0000 7f3dee8b7700 1 --2- 172.21.15.38:0/942193768 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3de80ff000 0x7f3de81950b0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:53:13.981 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.959+0000 7f3dee8b7700 1 --2- 172.21.15.38:0/942193768 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3de80ff9a0 0x7f3de81955f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:53:13.982 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.959+0000 7f3dee8b7700 1 -- 172.21.15.38:0/942193768 >> 172.21.15.38:0/942193768 conn(0x7f3de8075c60 msgr2=0x7f3de80fc940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:53:13.982 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.959+0000 7f3dee8b7700 1 -- 172.21.15.38:0/942193768 shutdown_connections 2023-03-09T14:53:13.982 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:13.959+0000 7f3dee8b7700 1 -- 172.21.15.38:0/942193768 wait complete. 2023-03-09T14:53:13.982 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:13 smithi038 ceph-mon[92422]: osdmap e187: 12 total, 10 up, 12 in 2023-03-09T14:53:13.983 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:13 smithi038 ceph-mon[92422]: pgmap v309: 129 pgs: 1 down+remapped, 1 down, 1 peering, 3 undersized+peered, 25 active+undersized, 1 recovering+undersized+degraded+remapped+peered, 6 active+recovery_wait+undersized+degraded+remapped, 3 recovery_wait+undersized+degraded+remapped+peered, 2 undersized+degraded+peered, 22 active+undersized+degraded, 64 active+clean; 19 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 16 KiB/s wr, 8 op/s; 5355/43131 objects degraded (12.416%); 203 MiB/s, 50 objects/s recovering 2023-03-09T14:53:13.983 INFO:tasks.cephfs.filesystem:scrub status for tag:a14659f4-d7ea-4c5a-b15b-6138af3f909f - {'path': '/', 'tag': 'a14659f4-d7ea-4c5a-b15b-6138af3f909f', 'options': 'recursive,force'} 2023-03-09T14:53:13.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:13 smithi181 ceph-mon[94189]: osdmap e187: 12 total, 10 up, 12 in 2023-03-09T14:53:13.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:13 smithi181 ceph-mon[94189]: pgmap v309: 129 pgs: 1 down+remapped, 1 down, 1 peering, 3 undersized+peered, 25 active+undersized, 1 recovering+undersized+degraded+remapped+peered, 6 active+recovery_wait+undersized+degraded+remapped, 3 recovery_wait+undersized+degraded+remapped+peered, 2 undersized+degraded+peered, 22 active+undersized+degraded, 64 active+clean; 19 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 16 KiB/s wr, 8 op/s; 5355/43131 objects degraded (12.416%); 203 MiB/s, 50 objects/s recovering 2023-03-09T14:53:14.551 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:53:14 smithi122 podman[130519]: 2023-03-09T14:53:14.552 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:53:14 smithi122 bash[130519]: 4f392f1f2ddb731489d0f182ec409db33f565e227958947b723a75206c4c10a1 2023-03-09T14:53:14.552 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:53:14 smithi122 ceph-osd[130560]: -- 172.21.15.122:0/174950664 <== mon.2 v2:172.21.15.122:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559e497149c0 con 0x559e496dd000 2023-03-09T14:53:14.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:14 smithi122 ceph-mon[93869]: osdmap e188: 12 total, 10 up, 12 in 2023-03-09T14:53:14.978 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:53:14 smithi122 systemd[1]: Started Ceph osd.1 for c6733fe2-be87-11ed-9af2-001a4aab830c. 2023-03-09T14:53:14.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:14 smithi181 ceph-mon[94189]: osdmap e188: 12 total, 10 up, 12 in 2023-03-09T14:53:15.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:14 smithi038 ceph-mon[92422]: osdmap e188: 12 total, 10 up, 12 in 2023-03-09T14:53:15.477 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:53:14 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[130534]: 2023-03-09T14:53:14.997+0000 7fd778435640 -1 Falling back to public interface 2023-03-09T14:53:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:15 smithi122 ceph-mon[93869]: pgmap v311: 129 pgs: 1 down+remapped, 1 down, 3 undersized+peered, 25 active+undersized, 1 recovering+undersized+degraded+remapped+peered, 6 active+recovery_wait+undersized+degraded+remapped, 2 recovery_wait+undersized+degraded+remapped+peered, 2 undersized+degraded+peered, 24 active+undersized+degraded, 64 active+clean; 19 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 74 KiB/s wr, 12 op/s; 5624/43171 objects degraded (13.027%); 102 MiB/s, 25 objects/s recovering 2023-03-09T14:53:15.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:15 smithi122 ceph-mon[93869]: osdmap e189: 12 total, 10 up, 12 in 2023-03-09T14:53:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:15 smithi181 ceph-mon[94189]: pgmap v311: 129 pgs: 1 down+remapped, 1 down, 3 undersized+peered, 25 active+undersized, 1 recovering+undersized+degraded+remapped+peered, 6 active+recovery_wait+undersized+degraded+remapped, 2 recovery_wait+undersized+degraded+remapped+peered, 2 undersized+degraded+peered, 24 active+undersized+degraded, 64 active+clean; 19 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 74 KiB/s wr, 12 op/s; 5624/43171 objects degraded (13.027%); 102 MiB/s, 25 objects/s recovering 2023-03-09T14:53:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:15 smithi181 ceph-mon[94189]: osdmap e189: 12 total, 10 up, 12 in 2023-03-09T14:53:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:15 smithi038 ceph-mon[92422]: pgmap v311: 129 pgs: 1 down+remapped, 1 down, 3 undersized+peered, 25 active+undersized, 1 recovering+undersized+degraded+remapped+peered, 6 active+recovery_wait+undersized+degraded+remapped, 2 recovery_wait+undersized+degraded+remapped+peered, 2 undersized+degraded+peered, 24 active+undersized+degraded, 64 active+clean; 19 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 74 KiB/s wr, 12 op/s; 5624/43171 objects degraded (13.027%); 102 MiB/s, 25 objects/s recovering 2023-03-09T14:53:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:15 smithi038 ceph-mon[92422]: osdmap e189: 12 total, 10 up, 12 in 2023-03-09T14:53:16.481 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:53:16 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[129095]: 2023-03-09T14:53:16.059+0000 7fbb316b7640 -1 osd.8 174 log_to_monitors true 2023-03-09T14:53:16.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:16 smithi181 ceph-mon[94189]: osdmap e190: 12 total, 10 up, 12 in 2023-03-09T14:53:16.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:16 smithi181 ceph-mon[94189]: from='osd.8 [v2:172.21.15.181:6816/1445888559,v1:172.21.15.181:6817/1445888559]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-03-09T14:53:16.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:16 smithi181 ceph-mon[94189]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-03-09T14:53:16.982 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 09 14:53:16 smithi181 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-8[129095]: 2023-03-09T14:53:16.749+0000 7fbb1f42d700 -1 osd.8 174 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-09T14:53:17.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:16 smithi038 ceph-mon[92422]: osdmap e190: 12 total, 10 up, 12 in 2023-03-09T14:53:17.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:16 smithi038 ceph-mon[92422]: from='osd.8 [v2:172.21.15.181:6816/1445888559,v1:172.21.15.181:6817/1445888559]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-03-09T14:53:17.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:16 smithi038 ceph-mon[92422]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-03-09T14:53:17.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:16 smithi122 ceph-mon[93869]: osdmap e190: 12 total, 10 up, 12 in 2023-03-09T14:53:17.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:16 smithi122 ceph-mon[93869]: from='osd.8 [v2:172.21.15.181:6816/1445888559,v1:172.21.15.181:6817/1445888559]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-03-09T14:53:17.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:16 smithi122 ceph-mon[93869]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-03-09T14:53:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:17 smithi038 ceph-mon[92422]: pgmap v314: 129 pgs: 1 down+remapped, 1 peering, 1 down, 3 undersized+peered, 25 active+undersized, 6 active+recovery_wait+undersized+degraded+remapped, 2 undersized+degraded+peered, 26 active+undersized+degraded, 64 active+clean; 19 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 250 KiB/s wr, 32 op/s; 5025/43235 objects degraded (11.623%); 344 MiB/s, 86 objects/s recovering 2023-03-09T14:53:18.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:17 smithi038 ceph-mon[92422]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-03-09T14:53:18.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:17 smithi038 ceph-mon[92422]: osdmap e191: 12 total, 10 up, 12 in 2023-03-09T14:53:18.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:17 smithi038 ceph-mon[92422]: from='osd.8 [v2:172.21.15.181:6816/1445888559,v1:172.21.15.181:6817/1445888559]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2023-03-09T14:53:18.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:17 smithi038 ceph-mon[92422]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2023-03-09T14:53:18.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:17 smithi038 ceph-mon[92422]: Health check update: Degraded data redundancy: 5025/43235 objects degraded (11.623%), 34 pgs degraded (PG_DEGRADED) 2023-03-09T14:53:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:17 smithi122 ceph-mon[93869]: pgmap v314: 129 pgs: 1 down+remapped, 1 peering, 1 down, 3 undersized+peered, 25 active+undersized, 6 active+recovery_wait+undersized+degraded+remapped, 2 undersized+degraded+peered, 26 active+undersized+degraded, 64 active+clean; 19 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 250 KiB/s wr, 32 op/s; 5025/43235 objects degraded (11.623%); 344 MiB/s, 86 objects/s recovering 2023-03-09T14:53:18.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:17 smithi122 ceph-mon[93869]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-03-09T14:53:18.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:17 smithi122 ceph-mon[93869]: osdmap e191: 12 total, 10 up, 12 in 2023-03-09T14:53:18.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:17 smithi122 ceph-mon[93869]: from='osd.8 [v2:172.21.15.181:6816/1445888559,v1:172.21.15.181:6817/1445888559]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2023-03-09T14:53:18.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:17 smithi122 ceph-mon[93869]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2023-03-09T14:53:18.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:17 smithi122 ceph-mon[93869]: Health check update: Degraded data redundancy: 5025/43235 objects degraded (11.623%), 34 pgs degraded (PG_DEGRADED) 2023-03-09T14:53:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:17 smithi181 ceph-mon[94189]: pgmap v314: 129 pgs: 1 down+remapped, 1 peering, 1 down, 3 undersized+peered, 25 active+undersized, 6 active+recovery_wait+undersized+degraded+remapped, 2 undersized+degraded+peered, 26 active+undersized+degraded, 64 active+clean; 19 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 250 KiB/s wr, 32 op/s; 5025/43235 objects degraded (11.623%); 344 MiB/s, 86 objects/s recovering 2023-03-09T14:53:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:17 smithi181 ceph-mon[94189]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-03-09T14:53:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:17 smithi181 ceph-mon[94189]: osdmap e191: 12 total, 10 up, 12 in 2023-03-09T14:53:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:17 smithi181 ceph-mon[94189]: from='osd.8 [v2:172.21.15.181:6816/1445888559,v1:172.21.15.181:6817/1445888559]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2023-03-09T14:53:18.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:17 smithi181 ceph-mon[94189]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi181", "root=default"]}]: dispatch 2023-03-09T14:53:18.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:17 smithi181 ceph-mon[94189]: Health check update: Degraded data redundancy: 5025/43235 objects degraded (11.623%), 34 pgs degraded (PG_DEGRADED) 2023-03-09T14:53:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 170.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T14:53:19.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:18 smithi038 ceph-mon[92422]: Health check update: 1 osds down (OSD_DOWN) 2023-03-09T14:53:19.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:18 smithi038 ceph-mon[92422]: osd.8 [v2:172.21.15.181:6816/1445888559,v1:172.21.15.181:6817/1445888559] boot 2023-03-09T14:53:19.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:18 smithi038 ceph-mon[92422]: osdmap e192: 12 total, 11 up, 12 in 2023-03-09T14:53:19.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:18 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-03-09T14:53:19.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:18 smithi038 ceph-mon[92422]: osdmap e193: 12 total, 11 up, 12 in 2023-03-09T14:53:19.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:18 smithi122 ceph-mon[93869]: Health check update: 1 osds down (OSD_DOWN) 2023-03-09T14:53:19.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:18 smithi122 ceph-mon[93869]: osd.8 [v2:172.21.15.181:6816/1445888559,v1:172.21.15.181:6817/1445888559] boot 2023-03-09T14:53:19.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:18 smithi122 ceph-mon[93869]: osdmap e192: 12 total, 11 up, 12 in 2023-03-09T14:53:19.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:18 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-03-09T14:53:19.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:18 smithi122 ceph-mon[93869]: osdmap e193: 12 total, 11 up, 12 in 2023-03-09T14:53:19.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:18 smithi181 ceph-mon[94189]: Health check update: 1 osds down (OSD_DOWN) 2023-03-09T14:53:19.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:18 smithi181 ceph-mon[94189]: osd.8 [v2:172.21.15.181:6816/1445888559,v1:172.21.15.181:6817/1445888559] boot 2023-03-09T14:53:19.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:18 smithi181 ceph-mon[94189]: osdmap e192: 12 total, 11 up, 12 in 2023-03-09T14:53:19.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:18 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-03-09T14:53:19.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:18 smithi181 ceph-mon[94189]: osdmap e193: 12 total, 11 up, 12 in 2023-03-09T14:53:20.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:19 smithi122 ceph-mon[93869]: pgmap v317: 129 pgs: 1 down+remapped, 1 down, 5 peering, 2 undersized+peered, 25 active+undersized, 5 active+recovery_wait+undersized+degraded+remapped, 2 undersized+degraded+peered, 23 active+undersized+degraded, 65 active+clean; 19 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 267 KiB/s wr, 62 op/s; 4657/43235 objects degraded (10.771%); 340 MiB/s, 85 objects/s recovering 2023-03-09T14:53:20.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:19 smithi122 ceph-mon[93869]: osdmap e194: 12 total, 11 up, 12 in 2023-03-09T14:53:20.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:19 smithi181 ceph-mon[94189]: pgmap v317: 129 pgs: 1 down+remapped, 1 down, 5 peering, 2 undersized+peered, 25 active+undersized, 5 active+recovery_wait+undersized+degraded+remapped, 2 undersized+degraded+peered, 23 active+undersized+degraded, 65 active+clean; 19 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 267 KiB/s wr, 62 op/s; 4657/43235 objects degraded (10.771%); 340 MiB/s, 85 objects/s recovering 2023-03-09T14:53:20.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:19 smithi181 ceph-mon[94189]: osdmap e194: 12 total, 11 up, 12 in 2023-03-09T14:53:20.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:19 smithi038 ceph-mon[92422]: pgmap v317: 129 pgs: 1 down+remapped, 1 down, 5 peering, 2 undersized+peered, 25 active+undersized, 5 active+recovery_wait+undersized+degraded+remapped, 2 undersized+degraded+peered, 23 active+undersized+degraded, 65 active+clean; 19 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 267 KiB/s wr, 62 op/s; 4657/43235 objects degraded (10.771%); 340 MiB/s, 85 objects/s recovering 2023-03-09T14:53:20.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:19 smithi038 ceph-mon[92422]: osdmap e194: 12 total, 11 up, 12 in 2023-03-09T14:53:22.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:21 smithi038 ceph-mon[92422]: pgmap v320: 129 pgs: 2 active+clean+snaptrim, 2 active+recovery_wait+degraded, 5 active+clean+snaptrim_wait, 1 down+remapped, 1 down, 5 peering, 1 undersized+peered, 21 active+undersized, 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 undersized+degraded+peered, 23 active+undersized+degraded, 63 active+clean; 18 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 350 KiB/s wr, 80 op/s; 3822/43199 objects degraded (8.847%); 350 MiB/s, 87 objects/s recovering 2023-03-09T14:53:22.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:21 smithi038 ceph-mon[92422]: osdmap e195: 12 total, 11 up, 12 in 2023-03-09T14:53:22.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:21 smithi122 ceph-mon[93869]: pgmap v320: 129 pgs: 2 active+clean+snaptrim, 2 active+recovery_wait+degraded, 5 active+clean+snaptrim_wait, 1 down+remapped, 1 down, 5 peering, 1 undersized+peered, 21 active+undersized, 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 undersized+degraded+peered, 23 active+undersized+degraded, 63 active+clean; 18 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 350 KiB/s wr, 80 op/s; 3822/43199 objects degraded (8.847%); 350 MiB/s, 87 objects/s recovering 2023-03-09T14:53:22.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:21 smithi122 ceph-mon[93869]: osdmap e195: 12 total, 11 up, 12 in 2023-03-09T14:53:22.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:21 smithi181 ceph-mon[94189]: pgmap v320: 129 pgs: 2 active+clean+snaptrim, 2 active+recovery_wait+degraded, 5 active+clean+snaptrim_wait, 1 down+remapped, 1 down, 5 peering, 1 undersized+peered, 21 active+undersized, 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 1 undersized+degraded+peered, 23 active+undersized+degraded, 63 active+clean; 18 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 350 KiB/s wr, 80 op/s; 3822/43199 objects degraded (8.847%); 350 MiB/s, 87 objects/s recovering 2023-03-09T14:53:22.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:21 smithi181 ceph-mon[94189]: osdmap e195: 12 total, 11 up, 12 in 2023-03-09T14:53:22.728 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7f12862fb700 time 2023-03-09T14:53:22.256489+0000 2023-03-09T14:53:22.729 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op") 2023-03-09T14:53:22.729 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:53:22.729 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x5626e4a06c85] 2023-03-09T14:53:22.729 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 2: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x5626e4cbf6e4] 2023-03-09T14:53:22.729 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x5626e4cc5d5e] 2023-03-09T14:53:22.729 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 4: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x5626e4b15986] 2023-03-09T14:53:22.730 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 5: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x5626e4e2b729] 2023-03-09T14:53:22.730 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x5626e4b2aebf] 2023-03-09T14:53:22.730 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x5626e51dcd85] 2023-03-09T14:53:22.730 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x5626e51dff84] 2023-03-09T14:53:22.730 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f12af0671ca] 2023-03-09T14:53:22.730 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 10: clone() 2023-03-09T14:53:22.731 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: *** Caught signal (Aborted) ** 2023-03-09T14:53:22.731 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: in thread 7f12862fb700 thread_name:tp_osd_tp 2023-03-09T14:53:22.731 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 2023-03-09T14:53:22.255+0000 7f12862fb700 -1 osd.7 pg_epoch: 196 pg[4.5s0( v 196'2558 (0'0,196'2558] local-lis/les=195/196 n=391 ec=85/85 lis/c=195/195 les/c/f=196/196/0 sis=195) [7,5,11,9]p7(0) r=0 lpr=195 luod=196'2553 crt=196'2553 lcod 196'2551 mlcod 196'2551 active+clean ps=[2~1,9~1]] bad op order, already applied 94789 > this 94787 2023-03-09T14:53:22.731 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 2023-03-09T14:53:22.259+0000 7f12862fb700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7f12862fb700 time 2023-03-09T14:53:22.256489+0000 2023-03-09T14:53:22.732 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op") 2023-03-09T14:53:22.732 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 2023-03-09T14:53:22.732 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:53:22.732 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x5626e4a06c85] 2023-03-09T14:53:22.732 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 2: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x5626e4cbf6e4] 2023-03-09T14:53:22.732 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x5626e4cc5d5e] 2023-03-09T14:53:22.733 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 4: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x5626e4b15986] 2023-03-09T14:53:22.733 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 5: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x5626e4e2b729] 2023-03-09T14:53:22.733 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x5626e4b2aebf] 2023-03-09T14:53:22.733 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x5626e51dcd85] 2023-03-09T14:53:22.733 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x5626e51dff84] 2023-03-09T14:53:22.733 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f12af0671ca] 2023-03-09T14:53:22.733 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 10: clone() 2023-03-09T14:53:22.734 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 2023-03-09T14:53:22.734 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:53:22.734 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f12af071cf0] 2023-03-09T14:53:22.734 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 2: gsignal() 2023-03-09T14:53:22.734 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 3: abort() 2023-03-09T14:53:22.734 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x5626e4a06d45] 2023-03-09T14:53:22.734 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 5: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x5626e4cbf6e4] 2023-03-09T14:53:22.734 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 6: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x5626e4cc5d5e] 2023-03-09T14:53:22.735 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x5626e4b15986] 2023-03-09T14:53:22.735 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 8: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x5626e4e2b729] 2023-03-09T14:53:22.735 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x5626e4b2aebf] 2023-03-09T14:53:22.735 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x5626e51dcd85] 2023-03-09T14:53:22.735 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x5626e51dff84] 2023-03-09T14:53:22.735 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f12af0671ca] 2023-03-09T14:53:22.735 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 13: clone() 2023-03-09T14:53:22.735 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 2023-03-09T14:53:22.266+0000 7f12862fb700 -1 *** Caught signal (Aborted) ** 2023-03-09T14:53:22.736 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: in thread 7f12862fb700 thread_name:tp_osd_tp 2023-03-09T14:53:22.736 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 2023-03-09T14:53:22.736 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:53:22.736 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f12af071cf0] 2023-03-09T14:53:22.736 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 2: gsignal() 2023-03-09T14:53:22.736 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 3: abort() 2023-03-09T14:53:22.736 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x5626e4a06d45] 2023-03-09T14:53:22.736 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 5: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x5626e4cbf6e4] 2023-03-09T14:53:22.737 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 6: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x5626e4cc5d5e] 2023-03-09T14:53:22.737 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x5626e4b15986] 2023-03-09T14:53:22.737 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 8: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x5626e4e2b729] 2023-03-09T14:53:22.737 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x5626e4b2aebf] 2023-03-09T14:53:22.737 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x5626e51dcd85] 2023-03-09T14:53:22.737 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x5626e51dff84] 2023-03-09T14:53:22.737 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f12af0671ca] 2023-03-09T14:53:22.738 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 13: clone() 2023-03-09T14:53:22.738 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-03-09T14:53:22.738 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 2023-03-09T14:53:22.738 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: -2512> 2023-03-09T14:53:22.255+0000 7f12862fb700 -1 osd.7 pg_epoch: 196 pg[4.5s0( v 196'2558 (0'0,196'2558] local-lis/les=195/196 n=391 ec=85/85 lis/c=195/195 les/c/f=196/196/0 sis=195) [7,5,11,9]p7(0) r=0 lpr=195 luod=196'2553 crt=196'2553 lcod 196'2551 mlcod 196'2551 active+clean ps=[2~1,9~1]] bad op order, already applied 94789 > this 94787 2023-03-09T14:53:22.738 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: -1164> 2023-03-09T14:53:22.259+0000 7f12862fb700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7f12862fb700 time 2023-03-09T14:53:22.256489+0000 2023-03-09T14:53:22.738 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op") 2023-03-09T14:53:22.738 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 2023-03-09T14:53:22.738 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:53:22.739 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x5626e4a06c85] 2023-03-09T14:53:22.739 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 2: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x5626e4cbf6e4] 2023-03-09T14:53:22.739 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x5626e4cc5d5e] 2023-03-09T14:53:22.739 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 4: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x5626e4b15986] 2023-03-09T14:53:22.739 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 5: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x5626e4e2b729] 2023-03-09T14:53:22.739 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x5626e4b2aebf] 2023-03-09T14:53:22.739 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x5626e51dcd85] 2023-03-09T14:53:22.740 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x5626e51dff84] 2023-03-09T14:53:22.740 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f12af0671ca] 2023-03-09T14:53:22.740 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 10: clone() 2023-03-09T14:53:22.740 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 2023-03-09T14:53:22.740 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: -32> 2023-03-09T14:53:22.266+0000 7f12862fb700 -1 *** Caught signal (Aborted) ** 2023-03-09T14:53:22.740 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: in thread 7f12862fb700 thread_name:tp_osd_tp 2023-03-09T14:53:22.740 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 2023-03-09T14:53:22.740 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:53:22.741 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f12af071cf0] 2023-03-09T14:53:22.741 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 2: gsignal() 2023-03-09T14:53:22.741 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 3: abort() 2023-03-09T14:53:22.741 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x5626e4a06d45] 2023-03-09T14:53:22.741 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 5: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x5626e4cbf6e4] 2023-03-09T14:53:22.741 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 6: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x5626e4cc5d5e] 2023-03-09T14:53:22.741 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x5626e4b15986] 2023-03-09T14:53:22.742 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 8: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x5626e4e2b729] 2023-03-09T14:53:22.742 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x5626e4b2aebf] 2023-03-09T14:53:22.742 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x5626e51dcd85] 2023-03-09T14:53:22.742 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x5626e51dff84] 2023-03-09T14:53:22.742 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f12af0671ca] 2023-03-09T14:53:22.742 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 13: clone() 2023-03-09T14:53:22.742 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-03-09T14:53:22.742 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 2023-03-09T14:53:22.743 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: -3513> 2023-03-09T14:53:22.255+0000 7f12862fb700 -1 osd.7 pg_epoch: 196 pg[4.5s0( v 196'2558 (0'0,196'2558] local-lis/les=195/196 n=391 ec=85/85 lis/c=195/195 les/c/f=196/196/0 sis=195) [7,5,11,9]p7(0) r=0 lpr=195 luod=196'2553 crt=196'2553 lcod 196'2551 mlcod 196'2551 active+clean ps=[2~1,9~1]] bad op order, already applied 94789 > this 94787 2023-03-09T14:53:22.743 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: -2165> 2023-03-09T14:53:22.259+0000 7f12862fb700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: In function 'void PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)' thread 7f12862fb700 time 2023-03-09T14:53:22.256489+0000 2023-03-09T14:53:22.743 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/18.0.0-2872-ge0ab30a2/rpm/el8/BUILD/ceph-18.0.0-2872-ge0ab30a2/src/osd/PrimaryLogPG.cc: 4284: ceph_abort_msg("out of order op") 2023-03-09T14:53:22.743 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 2023-03-09T14:53:22.743 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:53:22.743 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0xd7) [0x5626e4a06c85] 2023-03-09T14:53:22.743 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 2: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x5626e4cbf6e4] 2023-03-09T14:53:22.743 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 3: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x5626e4cc5d5e] 2023-03-09T14:53:22.744 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 4: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x5626e4b15986] 2023-03-09T14:53:22.744 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 5: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x5626e4e2b729] 2023-03-09T14:53:22.744 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 6: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x5626e4b2aebf] 2023-03-09T14:53:22.744 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 7: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x5626e51dcd85] 2023-03-09T14:53:22.744 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 8: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x5626e51dff84] 2023-03-09T14:53:22.744 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 9: /lib64/libpthread.so.0(+0x81ca) [0x7f12af0671ca] 2023-03-09T14:53:22.744 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 10: clone() 2023-03-09T14:53:22.744 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 2023-03-09T14:53:22.745 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: -1033> 2023-03-09T14:53:22.266+0000 7f12862fb700 -1 *** Caught signal (Aborted) ** 2023-03-09T14:53:22.745 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: in thread 7f12862fb700 thread_name:tp_osd_tp 2023-03-09T14:53:22.745 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 2023-03-09T14:53:22.745 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: ceph version 18.0.0-2872-ge0ab30a2 (e0ab30a2acead6db62b144ba6b6aeb23635adae7) reef (dev) 2023-03-09T14:53:22.745 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 1: /lib64/libpthread.so.0(+0x12cf0) [0x7f12af071cf0] 2023-03-09T14:53:22.745 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 2: gsignal() 2023-03-09T14:53:22.745 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 3: abort() 2023-03-09T14:53:22.746 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 4: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string, std::allocator > const&)+0x197) [0x5626e4a06d45] 2023-03-09T14:53:22.746 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 5: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x13f4) [0x5626e4cbf6e4] 2023-03-09T14:53:22.746 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 6: (PrimaryLogPG::do_op(boost::intrusive_ptr&)+0x281e) [0x5626e4cc5d5e] 2023-03-09T14:53:22.746 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 7: (OSD::dequeue_op(boost::intrusive_ptr, boost::intrusive_ptr, ThreadPool::TPHandle&)+0x1c6) [0x5626e4b15986] 2023-03-09T14:53:22.746 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 8: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&, ThreadPool::TPHandle&)+0x69) [0x5626e4e2b729] 2023-03-09T14:53:22.746 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 9: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x115f) [0x5626e4b2aebf] 2023-03-09T14:53:22.746 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 10: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x435) [0x5626e51dcd85] 2023-03-09T14:53:22.746 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 11: (ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x5626e51dff84] 2023-03-09T14:53:22.747 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 12: /lib64/libpthread.so.0(+0x81ca) [0x7f12af0671ca] 2023-03-09T14:53:22.747 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 13: clone() 2023-03-09T14:53:22.747 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2023-03-09T14:53:22.747 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[129182]: 2023-03-09T14:53:23.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:22 smithi038 ceph-mon[92422]: osdmap e196: 12 total, 11 up, 12 in 2023-03-09T14:53:23.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:22 smithi038 ceph-mon[92422]: osd.7 reported immediately failed by osd.11 2023-03-09T14:53:23.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:22 smithi038 ceph-mon[92422]: osd.7 failed (root=default,host=smithi122) (connection refused reported by osd.11) 2023-03-09T14:53:23.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:22 smithi038 ceph-mon[92422]: osd.7 reported immediately failed by osd.11 2023-03-09T14:53:23.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:22 smithi038 ceph-mon[92422]: osd.7 reported immediately failed by osd.11 2023-03-09T14:53:23.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:22 smithi038 ceph-mon[92422]: osd.7 reported immediately failed by osd.8 2023-03-09T14:53:23.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:22 smithi038 ceph-mon[92422]: osd.7 reported immediately failed by osd.6 2023-03-09T14:53:23.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:22 smithi038 ceph-mon[92422]: osd.7 reported immediately failed by osd.3 2023-03-09T14:53:23.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:22 smithi038 ceph-mon[92422]: osd.7 reported immediately failed by osd.0 2023-03-09T14:53:23.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:22 smithi038 ceph-mon[92422]: osd.7 reported immediately failed by osd.9 2023-03-09T14:53:23.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:22 smithi038 ceph-mon[92422]: osd.7 reported immediately failed by osd.8 2023-03-09T14:53:23.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:22 smithi038 ceph-mon[92422]: osd.7 reported immediately failed by osd.6 2023-03-09T14:53:23.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:22 smithi038 ceph-mon[92422]: osd.7 reported immediately failed by osd.3 2023-03-09T14:53:23.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:22 smithi038 ceph-mon[92422]: osd.7 reported immediately failed by osd.0 2023-03-09T14:53:23.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:22 smithi038 ceph-mon[92422]: osd.7 reported immediately failed by osd.9 2023-03-09T14:53:23.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:22 smithi038 ceph-mon[92422]: osd.7 reported immediately failed by osd.5 2023-03-09T14:53:23.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:22 smithi038 ceph-mon[92422]: osd.7 reported immediately failed by osd.3 2023-03-09T14:53:23.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:22 smithi038 ceph-mon[92422]: osd.7 reported immediately failed by osd.9 2023-03-09T14:53:23.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:22 smithi038 ceph-mon[92422]: osd.7 reported immediately failed by osd.2 2023-03-09T14:53:23.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:22 smithi038 ceph-mon[92422]: osd.7 reported immediately failed by osd.4 2023-03-09T14:53:23.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:22 smithi038 ceph-mon[92422]: osd.7 reported immediately failed by osd.5 2023-03-09T14:53:23.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:22 smithi038 ceph-mon[92422]: osd.7 reported immediately failed by osd.10 2023-03-09T14:53:23.004 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:22 smithi038 ceph-mon[92422]: osd.7 reported immediately failed by osd.4 2023-03-09T14:53:23.004 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:22 smithi038 ceph-mon[92422]: osd.7 reported immediately failed by osd.10 2023-03-09T14:53:23.004 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:22 smithi038 ceph-mon[92422]: osd.7 reported immediately failed by osd.2 2023-03-09T14:53:23.004 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:22 smithi038 ceph-mon[92422]: Health check update: Degraded data redundancy: 3822/43199 objects degraded (8.847%), 29 pgs degraded (PG_DEGRADED) 2023-03-09T14:53:23.004 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:22 smithi038 ceph-mon[92422]: osdmap e197: 12 total, 10 up, 12 in 2023-03-09T14:53:23.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-mon[93869]: osdmap e196: 12 total, 11 up, 12 in 2023-03-09T14:53:23.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-mon[93869]: osd.7 reported immediately failed by osd.11 2023-03-09T14:53:23.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-mon[93869]: osd.7 failed (root=default,host=smithi122) (connection refused reported by osd.11) 2023-03-09T14:53:23.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-mon[93869]: osd.7 reported immediately failed by osd.11 2023-03-09T14:53:23.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-mon[93869]: osd.7 reported immediately failed by osd.11 2023-03-09T14:53:23.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-mon[93869]: osd.7 reported immediately failed by osd.8 2023-03-09T14:53:23.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-mon[93869]: osd.7 reported immediately failed by osd.6 2023-03-09T14:53:23.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-mon[93869]: osd.7 reported immediately failed by osd.3 2023-03-09T14:53:23.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-mon[93869]: osd.7 reported immediately failed by osd.0 2023-03-09T14:53:23.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-mon[93869]: osd.7 reported immediately failed by osd.9 2023-03-09T14:53:23.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-mon[93869]: osd.7 reported immediately failed by osd.8 2023-03-09T14:53:23.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-mon[93869]: osd.7 reported immediately failed by osd.6 2023-03-09T14:53:23.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-mon[93869]: osd.7 reported immediately failed by osd.3 2023-03-09T14:53:23.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-mon[93869]: osd.7 reported immediately failed by osd.0 2023-03-09T14:53:23.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-mon[93869]: osd.7 reported immediately failed by osd.9 2023-03-09T14:53:23.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-mon[93869]: osd.7 reported immediately failed by osd.5 2023-03-09T14:53:23.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-mon[93869]: osd.7 reported immediately failed by osd.3 2023-03-09T14:53:23.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-mon[93869]: osd.7 reported immediately failed by osd.9 2023-03-09T14:53:23.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-mon[93869]: osd.7 reported immediately failed by osd.2 2023-03-09T14:53:23.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-mon[93869]: osd.7 reported immediately failed by osd.4 2023-03-09T14:53:23.231 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-mon[93869]: osd.7 reported immediately failed by osd.5 2023-03-09T14:53:23.231 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-mon[93869]: osd.7 reported immediately failed by osd.10 2023-03-09T14:53:23.231 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-mon[93869]: osd.7 reported immediately failed by osd.4 2023-03-09T14:53:23.231 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-mon[93869]: osd.7 reported immediately failed by osd.10 2023-03-09T14:53:23.231 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-mon[93869]: osd.7 reported immediately failed by osd.2 2023-03-09T14:53:23.231 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-mon[93869]: Health check update: Degraded data redundancy: 3822/43199 objects degraded (8.847%), 29 pgs degraded (PG_DEGRADED) 2023-03-09T14:53:23.231 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:22 smithi122 ceph-mon[93869]: osdmap e197: 12 total, 10 up, 12 in 2023-03-09T14:53:23.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:22 smithi181 ceph-mon[94189]: osdmap e196: 12 total, 11 up, 12 in 2023-03-09T14:53:23.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:22 smithi181 ceph-mon[94189]: osd.7 reported immediately failed by osd.11 2023-03-09T14:53:23.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:22 smithi181 ceph-mon[94189]: osd.7 failed (root=default,host=smithi122) (connection refused reported by osd.11) 2023-03-09T14:53:23.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:22 smithi181 ceph-mon[94189]: osd.7 reported immediately failed by osd.11 2023-03-09T14:53:23.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:22 smithi181 ceph-mon[94189]: osd.7 reported immediately failed by osd.11 2023-03-09T14:53:23.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:22 smithi181 ceph-mon[94189]: osd.7 reported immediately failed by osd.8 2023-03-09T14:53:23.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:22 smithi181 ceph-mon[94189]: osd.7 reported immediately failed by osd.6 2023-03-09T14:53:23.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:22 smithi181 ceph-mon[94189]: osd.7 reported immediately failed by osd.3 2023-03-09T14:53:23.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:22 smithi181 ceph-mon[94189]: osd.7 reported immediately failed by osd.0 2023-03-09T14:53:23.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:22 smithi181 ceph-mon[94189]: osd.7 reported immediately failed by osd.9 2023-03-09T14:53:23.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:22 smithi181 ceph-mon[94189]: osd.7 reported immediately failed by osd.8 2023-03-09T14:53:23.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:22 smithi181 ceph-mon[94189]: osd.7 reported immediately failed by osd.6 2023-03-09T14:53:23.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:22 smithi181 ceph-mon[94189]: osd.7 reported immediately failed by osd.3 2023-03-09T14:53:23.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:22 smithi181 ceph-mon[94189]: osd.7 reported immediately failed by osd.0 2023-03-09T14:53:23.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:22 smithi181 ceph-mon[94189]: osd.7 reported immediately failed by osd.9 2023-03-09T14:53:23.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:22 smithi181 ceph-mon[94189]: osd.7 reported immediately failed by osd.5 2023-03-09T14:53:23.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:22 smithi181 ceph-mon[94189]: osd.7 reported immediately failed by osd.3 2023-03-09T14:53:23.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:22 smithi181 ceph-mon[94189]: osd.7 reported immediately failed by osd.9 2023-03-09T14:53:23.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:22 smithi181 ceph-mon[94189]: osd.7 reported immediately failed by osd.2 2023-03-09T14:53:23.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:22 smithi181 ceph-mon[94189]: osd.7 reported immediately failed by osd.4 2023-03-09T14:53:23.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:22 smithi181 ceph-mon[94189]: osd.7 reported immediately failed by osd.5 2023-03-09T14:53:23.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:22 smithi181 ceph-mon[94189]: osd.7 reported immediately failed by osd.10 2023-03-09T14:53:23.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:22 smithi181 ceph-mon[94189]: osd.7 reported immediately failed by osd.4 2023-03-09T14:53:23.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:22 smithi181 ceph-mon[94189]: osd.7 reported immediately failed by osd.10 2023-03-09T14:53:23.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:22 smithi181 ceph-mon[94189]: osd.7 reported immediately failed by osd.2 2023-03-09T14:53:23.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:22 smithi181 ceph-mon[94189]: Health check update: Degraded data redundancy: 3822/43199 objects degraded (8.847%), 29 pgs degraded (PG_DEGRADED) 2023-03-09T14:53:23.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:22 smithi181 ceph-mon[94189]: osdmap e197: 12 total, 10 up, 12 in 2023-03-09T14:53:23.709 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:53:23 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[130534]: 2023-03-09T14:53:23.263+0000 7fd778435640 -1 osd.1 174 log_to_monitors true 2023-03-09T14:53:23.710 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 14:53:23 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[130534]: 2023-03-09T14:53:23.707+0000 7fd7671ad700 -1 osd.1 174 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-09T14:53:23.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 175.0 s, 84.3 tps, lat 17143.781 ms stddev 33557.541 2023-03-09T14:53:23.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:23 smithi122 ceph-mon[93869]: pgmap v324: 129 pgs: 5 stale+active+undersized, 1 stale+active+undersized+degraded, 7 stale+active+clean, 3 active+clean+snaptrim, 2 active+recovery_wait+degraded, 5 active+clean+snaptrim_wait, 5 peering, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 17 active+undersized+degraded, 69 active+clean; 18 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 1.3 MiB/s wr, 104 op/s; 3818/43291 objects degraded (8.819%); 386 MiB/s, 14 keys/s, 97 objects/s recovering 2023-03-09T14:53:23.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:23 smithi122 ceph-mon[93869]: from='osd.1 [v2:172.21.15.122:6808/2232324510,v1:172.21.15.122:6809/2232324510]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-03-09T14:53:23.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:23 smithi122 ceph-mon[93869]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-03-09T14:53:23.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:23 smithi122 ceph-mon[93869]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-03-09T14:53:23.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:23 smithi122 ceph-mon[93869]: osdmap e198: 12 total, 10 up, 12 in 2023-03-09T14:53:23.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:23 smithi122 ceph-mon[93869]: from='osd.1 [v2:172.21.15.122:6808/2232324510,v1:172.21.15.122:6809/2232324510]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-03-09T14:53:23.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:23 smithi122 ceph-mon[93869]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-03-09T14:53:23.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:23 smithi122 ceph-mon[93869]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs down) 2023-03-09T14:53:24.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:23 smithi181 ceph-mon[94189]: pgmap v324: 129 pgs: 5 stale+active+undersized, 1 stale+active+undersized+degraded, 7 stale+active+clean, 3 active+clean+snaptrim, 2 active+recovery_wait+degraded, 5 active+clean+snaptrim_wait, 5 peering, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 17 active+undersized+degraded, 69 active+clean; 18 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 1.3 MiB/s wr, 104 op/s; 3818/43291 objects degraded (8.819%); 386 MiB/s, 14 keys/s, 97 objects/s recovering 2023-03-09T14:53:24.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:23 smithi181 ceph-mon[94189]: from='osd.1 [v2:172.21.15.122:6808/2232324510,v1:172.21.15.122:6809/2232324510]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-03-09T14:53:24.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:23 smithi181 ceph-mon[94189]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-03-09T14:53:24.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:23 smithi181 ceph-mon[94189]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-03-09T14:53:24.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:23 smithi181 ceph-mon[94189]: osdmap e198: 12 total, 10 up, 12 in 2023-03-09T14:53:24.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:23 smithi181 ceph-mon[94189]: from='osd.1 [v2:172.21.15.122:6808/2232324510,v1:172.21.15.122:6809/2232324510]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-03-09T14:53:24.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:23 smithi181 ceph-mon[94189]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-03-09T14:53:24.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:23 smithi181 ceph-mon[94189]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs down) 2023-03-09T14:53:24.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:23 smithi038 ceph-mon[92422]: pgmap v324: 129 pgs: 5 stale+active+undersized, 1 stale+active+undersized+degraded, 7 stale+active+clean, 3 active+clean+snaptrim, 2 active+recovery_wait+degraded, 5 active+clean+snaptrim_wait, 5 peering, 12 active+undersized, 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 17 active+undersized+degraded, 69 active+clean; 18 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 1.3 MiB/s wr, 104 op/s; 3818/43291 objects degraded (8.819%); 386 MiB/s, 14 keys/s, 97 objects/s recovering 2023-03-09T14:53:24.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:23 smithi038 ceph-mon[92422]: from='osd.1 [v2:172.21.15.122:6808/2232324510,v1:172.21.15.122:6809/2232324510]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-03-09T14:53:24.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:23 smithi038 ceph-mon[92422]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-03-09T14:53:24.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:23 smithi038 ceph-mon[92422]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-03-09T14:53:24.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:23 smithi038 ceph-mon[92422]: osdmap e198: 12 total, 10 up, 12 in 2023-03-09T14:53:24.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:23 smithi038 ceph-mon[92422]: from='osd.1 [v2:172.21.15.122:6808/2232324510,v1:172.21.15.122:6809/2232324510]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-03-09T14:53:24.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:23 smithi038 ceph-mon[92422]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-03-09T14:53:24.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:23 smithi038 ceph-mon[92422]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs down) 2023-03-09T14:53:25.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:24 smithi038 ceph-mon[92422]: Health check update: 1 osds down (OSD_DOWN) 2023-03-09T14:53:25.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:24 smithi038 ceph-mon[92422]: osd.1 [v2:172.21.15.122:6808/2232324510,v1:172.21.15.122:6809/2232324510] boot 2023-03-09T14:53:25.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:24 smithi038 ceph-mon[92422]: osdmap e199: 12 total, 11 up, 12 in 2023-03-09T14:53:25.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-03-09T14:53:25.073 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:25 smithi122 systemd[1]: ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.7.service: Main process exited, code=exited, status=134/n/a 2023-03-09T14:53:25.074 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:24 smithi122 ceph-mon[93869]: Health check update: 1 osds down (OSD_DOWN) 2023-03-09T14:53:25.074 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:24 smithi122 ceph-mon[93869]: osd.1 [v2:172.21.15.122:6808/2232324510,v1:172.21.15.122:6809/2232324510] boot 2023-03-09T14:53:25.074 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:24 smithi122 ceph-mon[93869]: osdmap e199: 12 total, 11 up, 12 in 2023-03-09T14:53:25.075 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-03-09T14:53:25.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:24 smithi181 ceph-mon[94189]: Health check update: 1 osds down (OSD_DOWN) 2023-03-09T14:53:25.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:24 smithi181 ceph-mon[94189]: osd.1 [v2:172.21.15.122:6808/2232324510,v1:172.21.15.122:6809/2232324510] boot 2023-03-09T14:53:25.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:24 smithi181 ceph-mon[94189]: osdmap e199: 12 total, 11 up, 12 in 2023-03-09T14:53:25.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-03-09T14:53:26.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:25 smithi122 ceph-mon[93869]: pgmap v326: 129 pgs: 1 stale+active+clean+snaptrim, 4 stale+active+undersized, 1 stale+active+undersized+degraded, 4 stale+active+clean, 2 active+clean+snaptrim, 2 active+recovery_wait+degraded, 5 active+clean+snaptrim_wait, 6 peering, 1 undersized+peered, 15 active+undersized, 1 undersized+degraded+peered, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 15 active+undersized+degraded, 70 active+clean; 18 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 2.7 MiB/s wr, 218 op/s; 4101/43363 objects degraded (9.457%); 173 MiB/s, 26 keys/s, 44 objects/s recovering 2023-03-09T14:53:26.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:25 smithi122 ceph-mon[93869]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2023-03-09T14:53:26.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:25 smithi122 ceph-mon[93869]: osdmap e200: 12 total, 11 up, 12 in 2023-03-09T14:53:26.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:25 smithi181 ceph-mon[94189]: pgmap v326: 129 pgs: 1 stale+active+clean+snaptrim, 4 stale+active+undersized, 1 stale+active+undersized+degraded, 4 stale+active+clean, 2 active+clean+snaptrim, 2 active+recovery_wait+degraded, 5 active+clean+snaptrim_wait, 6 peering, 1 undersized+peered, 15 active+undersized, 1 undersized+degraded+peered, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 15 active+undersized+degraded, 70 active+clean; 18 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 2.7 MiB/s wr, 218 op/s; 4101/43363 objects degraded (9.457%); 173 MiB/s, 26 keys/s, 44 objects/s recovering 2023-03-09T14:53:26.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:25 smithi181 ceph-mon[94189]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2023-03-09T14:53:26.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:25 smithi181 ceph-mon[94189]: osdmap e200: 12 total, 11 up, 12 in 2023-03-09T14:53:26.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:25 smithi038 ceph-mon[92422]: pgmap v326: 129 pgs: 1 stale+active+clean+snaptrim, 4 stale+active+undersized, 1 stale+active+undersized+degraded, 4 stale+active+clean, 2 active+clean+snaptrim, 2 active+recovery_wait+degraded, 5 active+clean+snaptrim_wait, 6 peering, 1 undersized+peered, 15 active+undersized, 1 undersized+degraded+peered, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 15 active+undersized+degraded, 70 active+clean; 18 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 2.7 MiB/s wr, 218 op/s; 4101/43363 objects degraded (9.457%); 173 MiB/s, 26 keys/s, 44 objects/s recovering 2023-03-09T14:53:26.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:25 smithi038 ceph-mon[92422]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2023-03-09T14:53:26.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:25 smithi038 ceph-mon[92422]: osdmap e200: 12 total, 11 up, 12 in 2023-03-09T14:53:27.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:27 smithi122 ceph-mon[93869]: pgmap v329: 129 pgs: 1 remapped+peering, 4 stale+active+undersized, 1 stale+active+undersized+degraded, 18 peering, 1 active+recovery_wait+undersized+degraded, 1 undersized+peered, 12 active+undersized, 1 undersized+degraded+peered, 14 active+undersized+degraded, 76 active+clean; 18 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 KiB/s rd, 2.8 MiB/s wr, 347 op/s; 3675/43631 objects degraded (8.423%); 170 MiB/s, 12 keys/s, 43 objects/s recovering 2023-03-09T14:53:27.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:27 smithi122 ceph-mon[93869]: osdmap e201: 12 total, 11 up, 12 in 2023-03-09T14:53:27.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:27 smithi122 ceph-mon[93869]: Health check update: Degraded data redundancy: 3675/43631 objects degraded (8.423%), 17 pgs degraded (PG_DEGRADED) 2023-03-09T14:53:27.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:27 smithi181 ceph-mon[94189]: pgmap v329: 129 pgs: 1 remapped+peering, 4 stale+active+undersized, 1 stale+active+undersized+degraded, 18 peering, 1 active+recovery_wait+undersized+degraded, 1 undersized+peered, 12 active+undersized, 1 undersized+degraded+peered, 14 active+undersized+degraded, 76 active+clean; 18 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 KiB/s rd, 2.8 MiB/s wr, 347 op/s; 3675/43631 objects degraded (8.423%); 170 MiB/s, 12 keys/s, 43 objects/s recovering 2023-03-09T14:53:27.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:27 smithi181 ceph-mon[94189]: osdmap e201: 12 total, 11 up, 12 in 2023-03-09T14:53:27.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:27 smithi181 ceph-mon[94189]: Health check update: Degraded data redundancy: 3675/43631 objects degraded (8.423%), 17 pgs degraded (PG_DEGRADED) 2023-03-09T14:53:28.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:27 smithi038 ceph-mon[92422]: pgmap v329: 129 pgs: 1 remapped+peering, 4 stale+active+undersized, 1 stale+active+undersized+degraded, 18 peering, 1 active+recovery_wait+undersized+degraded, 1 undersized+peered, 12 active+undersized, 1 undersized+degraded+peered, 14 active+undersized+degraded, 76 active+clean; 18 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 KiB/s rd, 2.8 MiB/s wr, 347 op/s; 3675/43631 objects degraded (8.423%); 170 MiB/s, 12 keys/s, 43 objects/s recovering 2023-03-09T14:53:28.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:27 smithi038 ceph-mon[92422]: osdmap e201: 12 total, 11 up, 12 in 2023-03-09T14:53:28.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:27 smithi038 ceph-mon[92422]: Health check update: Degraded data redundancy: 3675/43631 objects degraded (8.423%), 17 pgs degraded (PG_DEGRADED) 2023-03-09T14:53:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 180.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T14:53:28.977 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:28 smithi122 systemd[1]: ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.7.service: Failed with result 'exit-code'. 2023-03-09T14:53:28.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:53:28.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:53:29.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:53:30.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:29 smithi038 ceph-mon[92422]: pgmap v331: 129 pgs: 1 active+recovery_wait+degraded, 1 remapped+peering, 12 peering, 1 active+recovery_wait+undersized+degraded, 16 active+undersized, 1 undersized+degraded+peered, 18 active+undersized+degraded, 79 active+clean; 18 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 675 KiB/s rd, 1.1 MiB/s wr, 203 op/s; 4530/43667 objects degraded (10.374%); 155 MiB/s, 39 objects/s recovering 2023-03-09T14:53:30.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:29 smithi038 ceph-mon[92422]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2023-03-09T14:53:30.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:29 smithi038 ceph-mon[92422]: osdmap e202: 12 total, 11 up, 12 in 2023-03-09T14:53:30.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:53:30.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:53:30.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:53:30.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:29 smithi122 ceph-mon[93869]: pgmap v331: 129 pgs: 1 active+recovery_wait+degraded, 1 remapped+peering, 12 peering, 1 active+recovery_wait+undersized+degraded, 16 active+undersized, 1 undersized+degraded+peered, 18 active+undersized+degraded, 79 active+clean; 18 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 675 KiB/s rd, 1.1 MiB/s wr, 203 op/s; 4530/43667 objects degraded (10.374%); 155 MiB/s, 39 objects/s recovering 2023-03-09T14:53:30.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:29 smithi122 ceph-mon[93869]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2023-03-09T14:53:30.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:29 smithi122 ceph-mon[93869]: osdmap e202: 12 total, 11 up, 12 in 2023-03-09T14:53:30.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:53:30.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:53:30.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:53:30.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:29 smithi181 ceph-mon[94189]: pgmap v331: 129 pgs: 1 active+recovery_wait+degraded, 1 remapped+peering, 12 peering, 1 active+recovery_wait+undersized+degraded, 16 active+undersized, 1 undersized+degraded+peered, 18 active+undersized+degraded, 79 active+clean; 18 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 675 KiB/s rd, 1.1 MiB/s wr, 203 op/s; 4530/43667 objects degraded (10.374%); 155 MiB/s, 39 objects/s recovering 2023-03-09T14:53:30.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:29 smithi181 ceph-mon[94189]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2023-03-09T14:53:30.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:29 smithi181 ceph-mon[94189]: osdmap e202: 12 total, 11 up, 12 in 2023-03-09T14:53:30.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:53:30.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:53:30.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:53:31.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:30 smithi038 ceph-mon[92422]: osdmap e203: 12 total, 11 up, 12 in 2023-03-09T14:53:31.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:30 smithi122 ceph-mon[93869]: osdmap e203: 12 total, 11 up, 12 in 2023-03-09T14:53:31.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:30 smithi181 ceph-mon[94189]: osdmap e203: 12 total, 11 up, 12 in 2023-03-09T14:53:32.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:31 smithi038 ceph-mon[92422]: pgmap v334: 129 pgs: 2 active+recovery_wait+degraded, 1 remapped+peering, 1 peering, 1 active+recovery_wait+undersized+degraded, 15 active+undersized, 1 active+recovering+undersized+degraded+remapped, 20 active+undersized+degraded, 88 active+clean; 18 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 746 KiB/s wr, 170 op/s; 4148/43659 objects degraded (9.501%); 201 MiB/s, 51 objects/s recovering 2023-03-09T14:53:32.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:31 smithi038 ceph-mon[92422]: osdmap e204: 12 total, 11 up, 12 in 2023-03-09T14:53:32.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:31 smithi122 ceph-mon[93869]: pgmap v334: 129 pgs: 2 active+recovery_wait+degraded, 1 remapped+peering, 1 peering, 1 active+recovery_wait+undersized+degraded, 15 active+undersized, 1 active+recovering+undersized+degraded+remapped, 20 active+undersized+degraded, 88 active+clean; 18 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 746 KiB/s wr, 170 op/s; 4148/43659 objects degraded (9.501%); 201 MiB/s, 51 objects/s recovering 2023-03-09T14:53:32.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:31 smithi122 ceph-mon[93869]: osdmap e204: 12 total, 11 up, 12 in 2023-03-09T14:53:32.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:31 smithi181 ceph-mon[94189]: pgmap v334: 129 pgs: 2 active+recovery_wait+degraded, 1 remapped+peering, 1 peering, 1 active+recovery_wait+undersized+degraded, 15 active+undersized, 1 active+recovering+undersized+degraded+remapped, 20 active+undersized+degraded, 88 active+clean; 18 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 746 KiB/s wr, 170 op/s; 4148/43659 objects degraded (9.501%); 201 MiB/s, 51 objects/s recovering 2023-03-09T14:53:32.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:31 smithi181 ceph-mon[94189]: osdmap e204: 12 total, 11 up, 12 in 2023-03-09T14:53:33.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:32 smithi038 ceph-mon[92422]: osdmap e205: 12 total, 11 up, 12 in 2023-03-09T14:53:33.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:32 smithi038 ceph-mon[92422]: Health check update: Degraded data redundancy: 4148/43659 objects degraded (9.501%), 24 pgs degraded (PG_DEGRADED) 2023-03-09T14:53:33.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:32 smithi122 ceph-mon[93869]: osdmap e205: 12 total, 11 up, 12 in 2023-03-09T14:53:33.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:32 smithi122 ceph-mon[93869]: Health check update: Degraded data redundancy: 4148/43659 objects degraded (9.501%), 24 pgs degraded (PG_DEGRADED) 2023-03-09T14:53:33.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:32 smithi181 ceph-mon[94189]: osdmap e205: 12 total, 11 up, 12 in 2023-03-09T14:53:33.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:32 smithi181 ceph-mon[94189]: Health check update: Degraded data redundancy: 4148/43659 objects degraded (9.501%), 24 pgs degraded (PG_DEGRADED) 2023-03-09T14:53:33.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 185.0 s, 345.8 tps, lat 668.109 ms stddev 1790.863 2023-03-09T14:53:34.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:33 smithi122 ceph-mon[93869]: pgmap v337: 129 pgs: 1 active+recovery_wait+degraded, 1 peering, 15 active+undersized, 2 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 89 active+clean; 18 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.7 MiB/s wr, 269 op/s; 4083/43631 objects degraded (9.358%); 290 MiB/s, 73 objects/s recovering 2023-03-09T14:53:34.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:33 smithi122 ceph-mon[93869]: osdmap e206: 12 total, 11 up, 12 in 2023-03-09T14:53:34.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:33 smithi181 ceph-mon[94189]: pgmap v337: 129 pgs: 1 active+recovery_wait+degraded, 1 peering, 15 active+undersized, 2 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 89 active+clean; 18 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.7 MiB/s wr, 269 op/s; 4083/43631 objects degraded (9.358%); 290 MiB/s, 73 objects/s recovering 2023-03-09T14:53:34.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:33 smithi181 ceph-mon[94189]: osdmap e206: 12 total, 11 up, 12 in 2023-03-09T14:53:34.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:33 smithi038 ceph-mon[92422]: pgmap v337: 129 pgs: 1 active+recovery_wait+degraded, 1 peering, 15 active+undersized, 2 active+recovering+undersized+degraded+remapped, 21 active+undersized+degraded, 89 active+clean; 18 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.7 MiB/s wr, 269 op/s; 4083/43631 objects degraded (9.358%); 290 MiB/s, 73 objects/s recovering 2023-03-09T14:53:34.254 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:33 smithi038 ceph-mon[92422]: osdmap e206: 12 total, 11 up, 12 in 2023-03-09T14:53:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:35 smithi122 ceph-mon[93869]: pgmap v339: 129 pgs: 1 active+recovery_wait+degraded, 15 active+undersized, 2 active+recovering+undersized+degraded+remapped, 22 active+undersized+degraded, 89 active+clean; 18 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.9 MiB/s wr, 410 op/s; 4467/43663 objects degraded (10.231%); 191 MiB/s, 48 objects/s recovering 2023-03-09T14:53:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:35 smithi181 ceph-mon[94189]: pgmap v339: 129 pgs: 1 active+recovery_wait+degraded, 15 active+undersized, 2 active+recovering+undersized+degraded+remapped, 22 active+undersized+degraded, 89 active+clean; 18 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.9 MiB/s wr, 410 op/s; 4467/43663 objects degraded (10.231%); 191 MiB/s, 48 objects/s recovering 2023-03-09T14:53:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:35 smithi038 ceph-mon[92422]: pgmap v339: 129 pgs: 1 active+recovery_wait+degraded, 15 active+undersized, 2 active+recovering+undersized+degraded+remapped, 22 active+undersized+degraded, 89 active+clean; 18 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.9 MiB/s wr, 410 op/s; 4467/43663 objects degraded (10.231%); 191 MiB/s, 48 objects/s recovering 2023-03-09T14:53:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:37 smithi122 ceph-mon[93869]: pgmap v340: 129 pgs: 15 active+undersized, 22 active+undersized+degraded, 92 active+clean; 18 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 9.3 MiB/s wr, 1.06k op/s; 4413/43631 objects degraded (10.114%); 106 MiB/s, 26 objects/s recovering 2023-03-09T14:53:38.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:37 smithi122 ceph-mon[93869]: Health check update: Degraded data redundancy: 4413/43631 objects degraded (10.114%), 22 pgs degraded (PG_DEGRADED) 2023-03-09T14:53:38.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:37 smithi181 ceph-mon[94189]: pgmap v340: 129 pgs: 15 active+undersized, 22 active+undersized+degraded, 92 active+clean; 18 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 9.3 MiB/s wr, 1.06k op/s; 4413/43631 objects degraded (10.114%); 106 MiB/s, 26 objects/s recovering 2023-03-09T14:53:38.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:37 smithi181 ceph-mon[94189]: Health check update: Degraded data redundancy: 4413/43631 objects degraded (10.114%), 22 pgs degraded (PG_DEGRADED) 2023-03-09T14:53:38.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:37 smithi038 ceph-mon[92422]: pgmap v340: 129 pgs: 15 active+undersized, 22 active+undersized+degraded, 92 active+clean; 18 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 9.3 MiB/s wr, 1.06k op/s; 4413/43631 objects degraded (10.114%); 106 MiB/s, 26 objects/s recovering 2023-03-09T14:53:38.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:37 smithi038 ceph-mon[92422]: Health check update: Degraded data redundancy: 4413/43631 objects degraded (10.114%), 22 pgs degraded (PG_DEGRADED) 2023-03-09T14:53:38.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 190.0 s, 999.4 tps, lat 95.352 ms stddev 64.717 2023-03-09T14:53:39.228 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:38 smithi122 systemd[1]: ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.7.service: Service RestartSec=10s expired, scheduling restart. 2023-03-09T14:53:39.228 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:38 smithi122 systemd[1]: ceph-c6733fe2-be87-11ed-9af2-001a4aab830c@osd.7.service: Scheduled restart job, restart counter is at 2. 2023-03-09T14:53:39.228 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:38 smithi122 systemd[1]: Stopped Ceph osd.7 for c6733fe2-be87-11ed-9af2-001a4aab830c. 2023-03-09T14:53:39.228 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:38 smithi122 systemd[1]: Starting Ceph osd.7 for c6733fe2-be87-11ed-9af2-001a4aab830c... 2023-03-09T14:53:40.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:39 smithi122 ceph-mon[93869]: pgmap v341: 129 pgs: 15 active+undersized, 22 active+undersized+degraded, 92 active+clean; 18 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 13 MiB/s wr, 1.40k op/s; 4421/43727 objects degraded (10.110%); 80 MiB/s, 20 objects/s recovering 2023-03-09T14:53:40.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:39 smithi181 ceph-mon[94189]: pgmap v341: 129 pgs: 15 active+undersized, 22 active+undersized+degraded, 92 active+clean; 18 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 13 MiB/s wr, 1.40k op/s; 4421/43727 objects degraded (10.110%); 80 MiB/s, 20 objects/s recovering 2023-03-09T14:53:40.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:39 smithi038 ceph-mon[92422]: pgmap v341: 129 pgs: 15 active+undersized, 22 active+undersized+degraded, 92 active+clean; 18 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 13 MiB/s wr, 1.40k op/s; 4421/43727 objects degraded (10.110%); 80 MiB/s, 20 objects/s recovering 2023-03-09T14:53:41.228 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:40 smithi122 bash[131762]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-03-09T14:53:41.228 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:40 smithi122 bash[131762]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2023-03-09T14:53:41.228 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:40 smithi122 bash[131762]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2023-03-09T14:53:41.228 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:40 smithi122 bash[131762]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2023-03-09T14:53:41.229 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:40 smithi122 bash[131762]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-7/block 2023-03-09T14:53:41.229 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:40 smithi122 bash[131762]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-03-09T14:53:41.229 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:40 smithi122 bash[131762]: --> ceph-volume raw activate successful for osd ID: 7 2023-03-09T14:53:42.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:41 smithi122 ceph-mon[93869]: pgmap v342: 129 pgs: 15 active+undersized, 22 active+undersized+degraded, 92 active+clean; 18 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 20 MiB/s wr, 2.07k op/s; 4453/43983 objects degraded (10.124%); 48 MiB/s, 12 objects/s recovering 2023-03-09T14:53:42.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:41 smithi181 ceph-mon[94189]: pgmap v342: 129 pgs: 15 active+undersized, 22 active+undersized+degraded, 92 active+clean; 18 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 20 MiB/s wr, 2.07k op/s; 4453/43983 objects degraded (10.124%); 48 MiB/s, 12 objects/s recovering 2023-03-09T14:53:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:41 smithi038 ceph-mon[92422]: pgmap v342: 129 pgs: 15 active+undersized, 22 active+undersized+degraded, 92 active+clean; 18 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 20 MiB/s wr, 2.07k op/s; 4453/43983 objects degraded (10.124%); 48 MiB/s, 12 objects/s recovering 2023-03-09T14:53:43.227 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:42 smithi122 podman[132019]: 2023-03-09T14:53:43.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:42 smithi122 ceph-mon[93869]: Health check update: Degraded data redundancy: 4453/43983 objects degraded (10.124%), 22 pgs degraded (PG_DEGRADED) 2023-03-09T14:53:43.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:42 smithi181 ceph-mon[94189]: Health check update: Degraded data redundancy: 4453/43983 objects degraded (10.124%), 22 pgs degraded (PG_DEGRADED) 2023-03-09T14:53:43.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:42 smithi038 ceph-mon[92422]: Health check update: Degraded data redundancy: 4453/43983 objects degraded (10.124%), 22 pgs degraded (PG_DEGRADED) 2023-03-09T14:53:43.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 195.0 s, 1152.8 tps, lat 91.659 ms stddev 63.524 2023-03-09T14:53:43.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:43 smithi122 ceph-mon[93869]: pgmap v343: 129 pgs: 15 active+undersized, 22 active+undersized+degraded, 92 active+clean; 18 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 22 MiB/s wr, 2.05k op/s; 4456/44143 objects degraded (10.094%); 22 MiB/s, 5 objects/s recovering 2023-03-09T14:53:43.984 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-03-09T14:53:44.229 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.224+0000 7fdf22649700 1 -- 172.21.15.38:0/3605122335 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdf1c107780 msgr2=0x7fdf1c10e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:53:44.229 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.224+0000 7fdf22649700 1 --2- 172.21.15.38:0/3605122335 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdf1c107780 0x7fdf1c10e7e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fdf1001a9a0 tx=0x7fdf1001acb0 comp rx=0 tx=0).stop 2023-03-09T14:53:44.229 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.225+0000 7fdf22649700 1 -- 172.21.15.38:0/3605122335 shutdown_connections 2023-03-09T14:53:44.229 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.225+0000 7fdf22649700 1 --2- 172.21.15.38:0/3605122335 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf1c1196d0 0x7fdf1c11bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:53:44.230 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.225+0000 7fdf22649700 1 --2- 172.21.15.38:0/3605122335 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdf1c107780 0x7fdf1c10e7e0 secure :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fdf1001a9a0 tx=0x7fdf1001acb0 comp rx=0 tx=0).stop 2023-03-09T14:53:44.230 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.225+0000 7fdf22649700 1 --2- 172.21.15.38:0/3605122335 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdf1c107b50 0x7fdf1c107fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:53:44.230 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.225+0000 7fdf22649700 1 -- 172.21.15.38:0/3605122335 >> 172.21.15.38:0/3605122335 conn(0x7fdf1c06b450 msgr2=0x7fdf1c0712c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:53:44.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:43 smithi181 ceph-mon[94189]: pgmap v343: 129 pgs: 15 active+undersized, 22 active+undersized+degraded, 92 active+clean; 18 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 22 MiB/s wr, 2.05k op/s; 4456/44143 objects degraded (10.094%); 22 MiB/s, 5 objects/s recovering 2023-03-09T14:53:44.234 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.230+0000 7fdf22649700 1 -- 172.21.15.38:0/3605122335 shutdown_connections 2023-03-09T14:53:44.234 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.232+0000 7fdf22649700 1 -- 172.21.15.38:0/3605122335 wait complete. 2023-03-09T14:53:44.236 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.233+0000 7fdf22649700 1 Processor -- start 2023-03-09T14:53:44.236 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.233+0000 7fdf22649700 1 -- start start 2023-03-09T14:53:44.236 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.234+0000 7fdf22649700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdf1c107b50 0x7fdf1c118b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:53:44.236 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.234+0000 7fdf22649700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdf1c1196d0 0x7fdf1c1190a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:53:44.236 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.234+0000 7fdf22649700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf1c114bb0 0x7fdf1c115000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:53:44.236 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.234+0000 7fdf22649700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf1c1157d0 con 0x7fdf1c114bb0 2023-03-09T14:53:44.236 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.234+0000 7fdf22649700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf1c115940 con 0x7fdf1c107b50 2023-03-09T14:53:44.237 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.234+0000 7fdf22649700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf1c1a92d0 con 0x7fdf1c1196d0 2023-03-09T14:53:44.237 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.234+0000 7fdf1bfff700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdf1c107b50 0x7fdf1c118b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:53:44.238 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.234+0000 7fdf1bfff700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdf1c107b50 0x7fdf1c118b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.38:50944/0 (socket says 172.21.15.38:50944) 2023-03-09T14:53:44.238 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.234+0000 7fdf1bfff700 1 -- 172.21.15.38:0/2579016759 learned_addr learned my addr 172.21.15.38:0/2579016759 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:53:44.238 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.234+0000 7fdf20be6700 1 --2- 172.21.15.38:0/2579016759 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf1c114bb0 0x7fdf1c115000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:53:44.238 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.235+0000 7fdf1bfff700 1 -- 172.21.15.38:0/2579016759 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdf1c1196d0 msgr2=0x7fdf1c1190a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:53:44.238 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.235+0000 7fdf1bfff700 1 --2- 172.21.15.38:0/2579016759 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdf1c1196d0 0x7fdf1c1190a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:53:44.238 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.235+0000 7fdf1bfff700 1 -- 172.21.15.38:0/2579016759 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf1c114bb0 msgr2=0x7fdf1c115000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:53:44.238 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.235+0000 7fdf1bfff700 1 --2- 172.21.15.38:0/2579016759 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf1c114bb0 0x7fdf1c115000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:53:44.239 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.235+0000 7fdf1bfff700 1 -- 172.21.15.38:0/2579016759 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf1001a600 con 0x7fdf1c107b50 2023-03-09T14:53:44.239 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.235+0000 7fdf1bfff700 1 --2- 172.21.15.38:0/2579016759 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdf1c107b50 0x7fdf1c118b60 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fdf10000c00 tx=0x7fdf100055e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:53:44.239 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.236+0000 7fdf197fa700 1 -- 172.21.15.38:0/2579016759 <== mon.2 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf1002d070 con 0x7fdf1c107b50 2023-03-09T14:53:44.239 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.236+0000 7fdf197fa700 1 -- 172.21.15.38:0/2579016759 <== mon.2 v2:172.21.15.122:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdf10003680 con 0x7fdf1c107b50 2023-03-09T14:53:44.239 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.236+0000 7fdf197fa700 1 -- 172.21.15.38:0/2579016759 <== mon.2 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf10003f70 con 0x7fdf1c107b50 2023-03-09T14:53:44.239 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.236+0000 7fdf22649700 1 -- 172.21.15.38:0/2579016759 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf1c1117b0 con 0x7fdf1c107b50 2023-03-09T14:53:44.240 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.236+0000 7fdf22649700 1 -- 172.21.15.38:0/2579016759 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf1c111620 con 0x7fdf1c107b50 2023-03-09T14:53:44.242 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.240+0000 7fdf2e7d2b80 1 Processor -- start 2023-03-09T14:53:44.243 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.240+0000 7fdf2e7d2b80 1 -- start start 2023-03-09T14:53:44.243 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.241+0000 7fdf2e7d2b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bdbd538830 0x55bdbd538c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:53:44.243 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.241+0000 7fdf2e7d2b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bdbd539140 0x55bdbd532aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:53:44.243 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.241+0000 7fdf2e7d2b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bdbd532fe0 0x55bdbd533410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:53:44.244 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.241+0000 7fdf2e7d2b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdbd41cda0 con 0x55bdbd539140 2023-03-09T14:53:44.244 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.241+0000 7fdf2e7d2b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdbd48b780 con 0x55bdbd532fe0 2023-03-09T14:53:44.244 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.241+0000 7fdf2e7d2b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdbd484cb0 con 0x55bdbd538830 2023-03-09T14:53:44.244 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.241+0000 7fdf1bfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bdbd539140 0x55bdbd532aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:53:44.244 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.241+0000 7fdf1bfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bdbd539140 0x55bdbd532aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:41756/0 (socket says 172.21.15.38:41756) 2023-03-09T14:53:44.244 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.241+0000 7fdf1bfff700 1 -- 172.21.15.38:0/4165657833 learned_addr learned my addr 172.21.15.38:0/4165657833 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:53:44.245 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.242+0000 7fdf1bfff700 1 -- 172.21.15.38:0/4165657833 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bdbd538830 msgr2=0x55bdbd538c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:53:44.245 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.242+0000 7fdf1bfff700 1 --2- 172.21.15.38:0/4165657833 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bdbd538830 0x55bdbd538c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:53:44.245 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.242+0000 7fdf1bfff700 1 -- 172.21.15.38:0/4165657833 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bdbd532fe0 msgr2=0x55bdbd533410 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-03-09T14:53:44.245 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.242+0000 7fdf1bfff700 1 --2- 172.21.15.38:0/4165657833 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bdbd532fe0 0x55bdbd533410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:53:44.246 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.242+0000 7fdf1bfff700 1 -- 172.21.15.38:0/4165657833 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bdbd44b190 con 0x55bdbd539140 2023-03-09T14:53:44.246 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.242+0000 7fdf1bfff700 1 --2- 172.21.15.38:0/4165657833 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bdbd539140 0x55bdbd532aa0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fdf10032000 tx=0x7fdf1003b440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcd1f228839f955 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:53:44.246 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.242+0000 7fdf027fc700 1 -- 172.21.15.38:0/4165657833 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf1003be80 con 0x55bdbd539140 2023-03-09T14:53:44.246 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.242+0000 7fdf027fc700 1 -- 172.21.15.38:0/4165657833 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdf10041040 con 0x55bdbd539140 2023-03-09T14:53:44.246 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.242+0000 7fdf027fc700 1 -- 172.21.15.38:0/4165657833 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf100328b0 con 0x55bdbd539140 2023-03-09T14:53:44.247 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.243+0000 7fdf2e7d2b80 1 -- 172.21.15.38:0/4165657833 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bdbd539140 msgr2=0x55bdbd532aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:53:44.248 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.243+0000 7fdf2e7d2b80 1 --2- 172.21.15.38:0/4165657833 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bdbd539140 0x55bdbd532aa0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fdf10032000 tx=0x7fdf1003b440 comp rx=0 tx=0).stop 2023-03-09T14:53:44.248 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.245+0000 7fdf2e7d2b80 1 -- 172.21.15.38:0/4165657833 shutdown_connections 2023-03-09T14:53:44.248 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.245+0000 7fdf2e7d2b80 1 --2- 172.21.15.38:0/4165657833 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bdbd532fe0 0x55bdbd533410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:53:44.248 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.245+0000 7fdf2e7d2b80 1 --2- 172.21.15.38:0/4165657833 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bdbd538830 0x55bdbd538c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:53:44.248 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.245+0000 7fdf2e7d2b80 1 --2- 172.21.15.38:0/4165657833 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bdbd539140 0x55bdbd532aa0 secure :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fdf10032000 tx=0x7fdf1003b440 comp rx=0 tx=0).stop 2023-03-09T14:53:44.248 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.245+0000 7fdf2e7d2b80 1 -- 172.21.15.38:0/4165657833 >> 172.21.15.38:0/4165657833 conn(0x55bdbd52da20 msgr2=0x55bdbd52ddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:53:44.254 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:43 smithi038 ceph-mon[92422]: pgmap v343: 129 pgs: 15 active+undersized, 22 active+undersized+degraded, 92 active+clean; 18 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 22 MiB/s wr, 2.05k op/s; 4456/44143 objects degraded (10.094%); 22 MiB/s, 5 objects/s recovering 2023-03-09T14:53:44.259 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.256+0000 7fdf197fa700 1 -- 172.21.15.38:0/2579016759 <== mon.2 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 25) v1 ==== 133295+0+0 (secure 0 0 0) 0x7fdf10038020 con 0x7fdf1c107b50 2023-03-09T14:53:44.259 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.256+0000 7fdf197fa700 1 --2- 172.21.15.38:0/2579016759 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fdf0409f6a0 0x7fdf040a1b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:53:44.259 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.257+0000 7fdf197fa700 1 -- 172.21.15.38:0/2579016759 <== mon.2 v2:172.21.15.122:3300/0 5 ==== osd_map(206..206 src has 1..206) v4 ==== 9577+0+0 (secure 0 0 0) 0x7fdf100de910 con 0x7fdf1c107b50 2023-03-09T14:53:44.266 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.259+0000 7fdf2e7d2b80 1 -- 172.21.15.38:0/4165657833 shutdown_connections 2023-03-09T14:53:44.267 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.260+0000 7fdf1b7fe700 1 --2- 172.21.15.38:0/2579016759 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fdf0409f6a0 0x7fdf040a1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:53:44.267 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.263+0000 7fdf1b7fe700 1 --2- 172.21.15.38:0/2579016759 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fdf0409f6a0 0x7fdf040a1b50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x55bdbd533950 tx=0x7fdf140091a0 comp rx=0 tx=0).ready entity=mgr.15558 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:53:44.267 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.264+0000 7fdf2e7d2b80 1 -- 172.21.15.38:0/4165657833 wait complete. 2023-03-09T14:53:44.272 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.265+0000 7fdf2e7d2b80 1 -- start start 2023-03-09T14:53:44.273 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.265+0000 7fdf2e7d2b80 1 Processor -- start 2023-03-09T14:53:44.273 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.268+0000 7fdf2e7d2b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bdbd538830 0x55bdbd53f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:53:44.273 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.268+0000 7fdf2e7d2b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bdbd53c470 0x55bdbd53c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:53:44.273 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.268+0000 7fdf2e7d2b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bdbd53cf80 0x55bdbd548930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:53:44.273 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.268+0000 7fdf2e7d2b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdbd308920 con 0x55bdbd538830 2023-03-09T14:53:44.273 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.268+0000 7fdf2e7d2b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdbd548f40 con 0x55bdbd53c470 2023-03-09T14:53:44.274 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.268+0000 7fdf2e7d2b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdbd549100 con 0x55bdbd53cf80 2023-03-09T14:53:44.274 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.268+0000 7fdf2e7d2b80 10 client.0 _ll_register_callbacks cb 0x55bdbd42a520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:53:44.274 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.268+0000 7fdf1bfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bdbd538830 0x55bdbd53f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:53:44.274 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.268+0000 7fdf1bfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bdbd538830 0x55bdbd53f1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:41762/0 (socket says 172.21.15.38:41762) 2023-03-09T14:53:44.274 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.268+0000 7fdf1bfff700 1 -- 172.21.15.38:0/1503807217 learned_addr learned my addr 172.21.15.38:0/1503807217 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:53:44.274 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.269+0000 7fdf1bfff700 1 -- 172.21.15.38:0/1503807217 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bdbd53cf80 msgr2=0x55bdbd548930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:53:44.274 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.269+0000 7fdf1bfff700 1 --2- 172.21.15.38:0/1503807217 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bdbd53cf80 0x55bdbd548930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:53:44.274 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.269+0000 7fdf1bfff700 1 -- 172.21.15.38:0/1503807217 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bdbd53c470 msgr2=0x55bdbd53c8e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-03-09T14:53:44.275 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.269+0000 7fdf1bfff700 1 --2- 172.21.15.38:0/1503807217 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bdbd53c470 0x55bdbd53c8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:53:44.275 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.269+0000 7fdf1bfff700 1 -- 172.21.15.38:0/1503807217 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf1c111620 con 0x55bdbd538830 2023-03-09T14:53:44.275 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.269+0000 7fdf1bfff700 1 --2- 172.21.15.38:0/1503807217 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bdbd538830 0x55bdbd53f1f0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fdf100b40c0 tx=0x7fdf100a8900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d70bd8c0395a9638 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:53:44.275 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.269+0000 7fdeffff7700 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2023-03-09T14:53:44.275 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.270+0000 7fdeffff7700 1 -- 172.21.15.38:0/1503807217 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf100dc020 con 0x55bdbd538830 2023-03-09T14:53:44.275 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.270+0000 7fdeffff7700 1 -- 172.21.15.38:0/1503807217 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(206..206 src has 1..206) v4 ==== 9577+0+0 (secure 0 0 0) 0x7fdf100b20d0 con 0x55bdbd538830 2023-03-09T14:53:44.275 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.270+0000 7fdf2e7d2b80 1 -- 172.21.15.38:0/1503807217 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55bdbd452190 con 0x55bdbd538830 2023-03-09T14:53:44.276 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.270+0000 7fdeffff7700 1 -- 172.21.15.38:0/1503807217 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdf100ac490 con 0x55bdbd538830 2023-03-09T14:53:44.276 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.270+0000 7fdf2e7d2b80 10 client.16164 fetch_fsmap learned FSMap version 31 2023-03-09T14:53:44.276 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.270+0000 7fdf2e7d2b80 1 -- 172.21.15.38:0/1503807217 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55bdbd29bc60 con 0x55bdbd538830 2023-03-09T14:53:44.276 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.270+0000 7fdeffff7700 1 -- 172.21.15.38:0/1503807217 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55bdbd452190 con 0x55bdbd538830 2023-03-09T14:53:44.276 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.272+0000 7fdeffff7700 1 -- 172.21.15.38:0/1503807217 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 31) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fdf100ac6f0 con 0x55bdbd538830 2023-03-09T14:53:44.276 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.272+0000 7fdf2e7d2b80 10 client.16164 fetch_fsmap finished waiting for FSMap version 31 2023-03-09T14:53:44.277 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.272+0000 7fdf2e7d2b80 10 client.16164 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T14:53:44.277 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.272+0000 7fdf2e7d2b80 20 client.16164 populate_metadata read hostname 'smithi038' 2023-03-09T14:53:44.277 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.272+0000 7fdf2e7d2b80 1 --2- 172.21.15.38:0/1503807217 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55bdbd54ce30 0x55bdbd54f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:53:44.277 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.272+0000 7fdf2e7d2b80 4 client.16164 mds_command: new command op to 24530 tid=0[{"prefix": "get_command_descriptions"}] 2023-03-09T14:53:44.277 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.272+0000 7fdf2e7d2b80 1 -- 172.21.15.38:0/1503807217 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bdbd308f60 con 0x55bdbd54ce30 2023-03-09T14:53:44.284 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.279+0000 7fdf1b7fe700 1 --2- 172.21.15.38:0/1503807217 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55bdbd54ce30 0x55bdbd54f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:53:44.305 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.303+0000 7fdeffff7700 10 client.16164 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:53:44.305 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.303+0000 7fdf1b7fe700 1 --2- 172.21.15.38:0/1503807217 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55bdbd54ce30 0x55bdbd54f2e0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f6a61d5e4d32883 server_cookie=2e2fa5ffbbb86d07 in_seq=0 out_seq=0 2023-03-09T14:53:44.311 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.309+0000 7fdeffff7700 1 -- 172.21.15.38:0/1503807217 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18278 (crc 0 0 0) 0x55bdbd308f60 con 0x55bdbd54ce30 2023-03-09T14:53:44.311 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.309+0000 7fdeffff7700 10 client.16164 handle_command_reply: tid=0 2023-03-09T14:53:44.312 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.309+0000 7fdf2e7d2b80 1 client.16164 shutdown 2023-03-09T14:53:44.319 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.318+0000 7fdf2e7d2b80 1 -- 172.21.15.38:0/1503807217 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bdbd538830 msgr2=0x55bdbd53f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:53:44.320 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.318+0000 7fdf2e7d2b80 1 --2- 172.21.15.38:0/1503807217 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bdbd538830 0x55bdbd53f1f0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fdf100b40c0 tx=0x7fdf100a8900 comp rx=0 tx=0).stop 2023-03-09T14:53:44.325 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.323+0000 7fdf2e7d2b80 1 -- 172.21.15.38:0/1503807217 shutdown_connections 2023-03-09T14:53:44.325 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.323+0000 7fdf2e7d2b80 1 --2- 172.21.15.38:0/1503807217 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55bdbd54ce30 0x55bdbd54f2e0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:53:44.325 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.323+0000 7fdf2e7d2b80 1 --2- 172.21.15.38:0/1503807217 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bdbd538830 0x55bdbd53f1f0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:53:44.330 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.323+0000 7fdf2e7d2b80 1 --2- 172.21.15.38:0/1503807217 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bdbd53cf80 0x55bdbd548930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:53:44.330 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.323+0000 7fdf2e7d2b80 1 --2- 172.21.15.38:0/1503807217 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bdbd53c470 0x55bdbd53c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:53:44.330 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.323+0000 7fdf2e7d2b80 1 -- 172.21.15.38:0/1503807217 >> 172.21.15.38:0/1503807217 conn(0x55bdbd52da20 msgr2=0x55bdbd52cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:53:44.330 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.323+0000 7fdeffff7700 0 client.16164 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:53:44.330 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.323+0000 7fdf2e7d2b80 1 -- 172.21.15.38:0/1503807217 shutdown_connections 2023-03-09T14:53:44.331 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.323+0000 7fdf2e7d2b80 1 -- 172.21.15.38:0/1503807217 wait complete. 2023-03-09T14:53:44.331 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.323+0000 7fdf2e7d2b80 20 client.16164 trim_cache size 0 max 16384 2023-03-09T14:53:44.357 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.355+0000 7fdf2e7d2b80 1 Processor -- start 2023-03-09T14:53:44.357 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.355+0000 7fdf2e7d2b80 1 -- start start 2023-03-09T14:53:44.358 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.355+0000 7fdf2e7d2b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bdbd538830 0x55bdbd53bc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:53:44.358 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.355+0000 7fdf2e7d2b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bdbd53c470 0x55bdbd53ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:53:44.358 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.355+0000 7fdf2e7d2b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bdbd53cf80 0x55bdbd52cc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:53:44.358 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.355+0000 7fdf2e7d2b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdbd533a00 con 0x55bdbd53c470 2023-03-09T14:53:44.358 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.355+0000 7fdf2e7d2b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdbd52d870 con 0x55bdbd53cf80 2023-03-09T14:53:44.358 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.355+0000 7fdf2e7d2b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdbd538480 con 0x55bdbd538830 2023-03-09T14:53:44.358 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.355+0000 7fdf1b7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bdbd53c470 0x55bdbd53ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:53:44.359 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.356+0000 7fdf1b7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bdbd53c470 0x55bdbd53ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:41774/0 (socket says 172.21.15.38:41774) 2023-03-09T14:53:44.359 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.356+0000 7fdf1b7fe700 1 -- 172.21.15.38:0/3017148068 learned_addr learned my addr 172.21.15.38:0/3017148068 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:53:44.359 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.357+0000 7fdf1b7fe700 1 -- 172.21.15.38:0/3017148068 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bdbd538830 msgr2=0x55bdbd53bc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:53:44.359 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.357+0000 7fdf1b7fe700 1 --2- 172.21.15.38:0/3017148068 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bdbd538830 0x55bdbd53bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:53:44.360 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.357+0000 7fdf1b7fe700 1 -- 172.21.15.38:0/3017148068 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bdbd53cf80 msgr2=0x55bdbd52cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:53:44.360 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.357+0000 7fdf1b7fe700 1 --2- 172.21.15.38:0/3017148068 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bdbd53cf80 0x55bdbd52cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:53:44.360 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.357+0000 7fdf1b7fe700 1 -- 172.21.15.38:0/3017148068 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bdbd2b2de0 con 0x55bdbd53c470 2023-03-09T14:53:44.362 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.360+0000 7fdf1b7fe700 1 --2- 172.21.15.38:0/3017148068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bdbd53c470 0x55bdbd53ed90 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c116350 tx=0x7fdf1400f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6ef9c6abe5164e4 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:53:44.364 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.361+0000 7fdf02ffd700 1 -- 172.21.15.38:0/3017148068 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf14021330 con 0x55bdbd53c470 2023-03-09T14:53:44.365 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.361+0000 7fdf02ffd700 1 -- 172.21.15.38:0/3017148068 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fdf14021490 con 0x55bdbd53c470 2023-03-09T14:53:44.365 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.361+0000 7fdf02ffd700 1 -- 172.21.15.38:0/3017148068 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf1402a6d0 con 0x55bdbd53c470 2023-03-09T14:53:44.365 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.363+0000 7fdf2e7d2b80 1 -- 172.21.15.38:0/3017148068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bdbd53c470 msgr2=0x55bdbd53ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:53:44.365 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.363+0000 7fdf2e7d2b80 1 --2- 172.21.15.38:0/3017148068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bdbd53c470 0x55bdbd53ed90 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fdf1c116350 tx=0x7fdf1400f000 comp rx=0 tx=0).stop 2023-03-09T14:53:44.374 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.372+0000 7fdf2e7d2b80 1 -- 172.21.15.38:0/3017148068 shutdown_connections 2023-03-09T14:53:44.375 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.372+0000 7fdf2e7d2b80 1 --2- 172.21.15.38:0/3017148068 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bdbd53cf80 0x55bdbd52cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:53:44.375 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.372+0000 7fdf2e7d2b80 1 --2- 172.21.15.38:0/3017148068 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bdbd538830 0x55bdbd53bc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:53:44.375 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.372+0000 7fdf2e7d2b80 1 --2- 172.21.15.38:0/3017148068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bdbd53c470 0x55bdbd53ed90 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:53:44.375 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.372+0000 7fdf2e7d2b80 1 -- 172.21.15.38:0/3017148068 >> 172.21.15.38:0/3017148068 conn(0x55bdbd52da20 msgr2=0x55bdbd54ee40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:53:44.377 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.376+0000 7fdf2e7d2b80 1 -- 172.21.15.38:0/3017148068 shutdown_connections 2023-03-09T14:53:44.381 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.379+0000 7fdf2e7d2b80 1 -- 172.21.15.38:0/3017148068 wait complete. 2023-03-09T14:53:44.382 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.379+0000 7fdf2e7d2b80 1 -- start start 2023-03-09T14:53:44.382 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.379+0000 7fdf2e7d2b80 1 Processor -- start 2023-03-09T14:53:44.382 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.380+0000 7fdf2e7d2b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bdbd538830 0x55bdbd535740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:53:44.382 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.380+0000 7fdf2e7d2b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bdbd53c470 0x55bdbd534290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:53:44.382 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.380+0000 7fdf2e7d2b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bdbd53cf80 0x55bdbd5347d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:53:44.383 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.380+0000 7fdf2e7d2b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdbd535e80 con 0x55bdbd53cf80 2023-03-09T14:53:44.383 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.380+0000 7fdf2e7d2b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdbd535ff0 con 0x55bdbd53c470 2023-03-09T14:53:44.383 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.380+0000 7fdf2e7d2b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdbd534d10 con 0x55bdbd538830 2023-03-09T14:53:44.383 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.380+0000 7fdf2e7d2b80 10 client.0 _ll_register_callbacks cb 0x55bdbd42a520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:53:44.383 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.380+0000 7fdf20be6700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bdbd53cf80 0x55bdbd5347d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:53:44.384 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.380+0000 7fdf20be6700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bdbd53cf80 0x55bdbd5347d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:41776/0 (socket says 172.21.15.38:41776) 2023-03-09T14:53:44.384 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.380+0000 7fdf20be6700 1 -- 172.21.15.38:0/3264285483 learned_addr learned my addr 172.21.15.38:0/3264285483 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:53:44.384 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.382+0000 7fdf1b7fe700 1 --2- 172.21.15.38:0/3264285483 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bdbd53c470 0x55bdbd534290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:53:44.384 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.382+0000 7fdf20be6700 1 -- 172.21.15.38:0/3264285483 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bdbd538830 msgr2=0x55bdbd535740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:53:44.385 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.382+0000 7fdf20be6700 1 --2- 172.21.15.38:0/3264285483 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bdbd538830 0x55bdbd535740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:53:44.385 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.382+0000 7fdf20be6700 1 -- 172.21.15.38:0/3264285483 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bdbd53c470 msgr2=0x55bdbd534290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:53:44.385 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.382+0000 7fdf20be6700 1 --2- 172.21.15.38:0/3264285483 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bdbd53c470 0x55bdbd534290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:53:44.385 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.382+0000 7fdf20be6700 1 -- 172.21.15.38:0/3264285483 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bdbd2b12b0 con 0x55bdbd53cf80 2023-03-09T14:53:44.385 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.382+0000 7fdf20be6700 1 --2- 172.21.15.38:0/3264285483 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bdbd53cf80 0x55bdbd5347d0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fdf0c009f70 tx=0x7fdf0c00e560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b68fa4e4c6e9a1a6 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:53:44.385 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.382+0000 7fdeffff7700 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2023-03-09T14:53:44.386 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.383+0000 7fdeffff7700 1 -- 172.21.15.38:0/3264285483 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf0c00ed60 con 0x55bdbd53cf80 2023-03-09T14:53:44.386 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.383+0000 7fdeffff7700 1 -- 172.21.15.38:0/3264285483 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(206..206 src has 1..206) v4 ==== 9577+0+0 (secure 0 0 0) 0x7fdf0c013470 con 0x55bdbd53cf80 2023-03-09T14:53:44.388 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.386+0000 7fdf2e7d2b80 1 -- 172.21.15.38:0/3264285483 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55bdbd4421b0 con 0x55bdbd53cf80 2023-03-09T14:53:44.390 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.389+0000 7fdeffff7700 1 -- 172.21.15.38:0/3264285483 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdf0c00eec0 con 0x55bdbd53cf80 2023-03-09T14:53:44.391 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.389+0000 7fdf2e7d2b80 10 client.16170 fetch_fsmap learned FSMap version 31 2023-03-09T14:53:44.391 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.389+0000 7fdf2e7d2b80 1 -- 172.21.15.38:0/3264285483 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55bdbd2d9250 con 0x55bdbd53cf80 2023-03-09T14:53:44.391 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.389+0000 7fdeffff7700 1 -- 172.21.15.38:0/3264285483 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55bdbd4421b0 con 0x55bdbd53cf80 2023-03-09T14:53:44.391 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.389+0000 7fdeffff7700 1 -- 172.21.15.38:0/3264285483 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 31) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fdf0c011070 con 0x55bdbd53cf80 2023-03-09T14:53:44.392 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.390+0000 7fdf2e7d2b80 10 client.16170 fetch_fsmap finished waiting for FSMap version 31 2023-03-09T14:53:44.392 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.390+0000 7fdf2e7d2b80 10 client.16170 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T14:53:44.392 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.390+0000 7fdf2e7d2b80 20 client.16170 populate_metadata read hostname 'smithi038' 2023-03-09T14:53:44.392 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.390+0000 7fdf2e7d2b80 1 --2- 172.21.15.38:0/3264285483 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55bdbd5a2010 0x55bdbd5a4420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:53:44.392 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.390+0000 7fdf2e7d2b80 4 client.16170 mds_command: new command op to 24530 tid=0[{"prefix": "scrub status"}] 2023-03-09T14:53:44.393 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.390+0000 7fdf2e7d2b80 1 -- 172.21.15.38:0/3264285483 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55bdbd42c1c0 con 0x55bdbd5a2010 2023-03-09T14:53:44.393 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.391+0000 7fdf1bfff700 1 --2- 172.21.15.38:0/3264285483 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55bdbd5a2010 0x55bdbd5a4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:53:44.397 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.395+0000 7fdeffff7700 10 client.16170 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:53:44.397 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.395+0000 7fdf1bfff700 1 --2- 172.21.15.38:0/3264285483 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55bdbd5a2010 0x55bdbd5a4420 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=265b69525d521258 server_cookie=672d8f67a38f1c7f in_seq=0 out_seq=0 2023-03-09T14:53:44.407 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.406+0000 7fdeffff7700 1 -- 172.21.15.38:0/3264285483 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+265 (crc 0 0 0) 0x55bdbd42c1c0 con 0x55bdbd5a2010 2023-03-09T14:53:44.408 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.406+0000 7fdeffff7700 10 client.16170 handle_command_reply: tid=0 2023-03-09T14:53:44.414 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.413+0000 7fdf2e7d2b80 1 client.16170 shutdown 2023-03-09T14:53:44.418 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.416+0000 7fdf2e7d2b80 1 -- 172.21.15.38:0/3264285483 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bdbd53cf80 msgr2=0x55bdbd5347d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:53:44.418 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.416+0000 7fdf2e7d2b80 1 --2- 172.21.15.38:0/3264285483 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bdbd53cf80 0x55bdbd5347d0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fdf0c009f70 tx=0x7fdf0c00e560 comp rx=0 tx=0).stop 2023-03-09T14:53:44.418 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.416+0000 7fdf2e7d2b80 1 -- 172.21.15.38:0/3264285483 shutdown_connections 2023-03-09T14:53:44.418 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.416+0000 7fdf2e7d2b80 1 --2- 172.21.15.38:0/3264285483 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55bdbd5a2010 0x55bdbd5a4420 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:53:44.419 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.416+0000 7fdf2e7d2b80 1 --2- 172.21.15.38:0/3264285483 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55bdbd53cf80 0x55bdbd5347d0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:53:44.419 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.416+0000 7fdf2e7d2b80 1 --2- 172.21.15.38:0/3264285483 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55bdbd538830 0x55bdbd535740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:53:44.419 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.416+0000 7fdf2e7d2b80 1 --2- 172.21.15.38:0/3264285483 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55bdbd53c470 0x55bdbd534290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:53:44.419 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.416+0000 7fdf2e7d2b80 1 -- 172.21.15.38:0/3264285483 >> 172.21.15.38:0/3264285483 conn(0x55bdbd52da20 msgr2=0x55bdbd536490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:53:44.419 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.416+0000 7fdeffff7700 0 client.16170 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:53:44.431 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2023-03-09T14:53:44.431 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "scrub active (3 inodes in the stack)", 2023-03-09T14:53:44.431 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": { 2023-03-09T14:53:44.431 INFO:teuthology.orchestra.run.smithi038.stdout: "a14659f4-d7ea-4c5a-b15b-6138af3f909f": { 2023-03-09T14:53:44.431 INFO:teuthology.orchestra.run.smithi038.stdout: "path": "/", 2023-03-09T14:53:44.431 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "a14659f4-d7ea-4c5a-b15b-6138af3f909f", 2023-03-09T14:53:44.432 INFO:teuthology.orchestra.run.smithi038.stdout: "options": "recursive,force" 2023-03-09T14:53:44.432 INFO:teuthology.orchestra.run.smithi038.stdout: } 2023-03-09T14:53:44.432 INFO:teuthology.orchestra.run.smithi038.stdout: } 2023-03-09T14:53:44.432 INFO:teuthology.orchestra.run.smithi038.stdout:} 2023-03-09T14:53:44.435 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.428+0000 7fdf2e7d2b80 1 -- 172.21.15.38:0/3264285483 shutdown_connections 2023-03-09T14:53:44.435 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.429+0000 7fdf2e7d2b80 1 -- 172.21.15.38:0/3264285483 wait complete. 2023-03-09T14:53:44.435 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.429+0000 7fdf2e7d2b80 20 client.16170 trim_cache size 0 max 16384 2023-03-09T14:53:44.435 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.429+0000 7fdf22649700 1 -- 172.21.15.38:0/2579016759 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fdf0409f6a0 msgr2=0x7fdf040a1b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:53:44.435 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.429+0000 7fdf22649700 1 --2- 172.21.15.38:0/2579016759 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fdf0409f6a0 0x7fdf040a1b50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x55bdbd533950 tx=0x7fdf140091a0 comp rx=0 tx=0).stop 2023-03-09T14:53:44.435 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.429+0000 7fdf22649700 1 -- 172.21.15.38:0/2579016759 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdf1c107b50 msgr2=0x7fdf1c118b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:53:44.435 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.429+0000 7fdf22649700 1 --2- 172.21.15.38:0/2579016759 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdf1c107b50 0x7fdf1c118b60 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fdf10000c00 tx=0x7fdf100055e0 comp rx=0 tx=0).stop 2023-03-09T14:53:44.436 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.429+0000 7fdf22649700 1 -- 172.21.15.38:0/2579016759 shutdown_connections 2023-03-09T14:53:44.436 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.429+0000 7fdf22649700 1 --2- 172.21.15.38:0/2579016759 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fdf0409f6a0 0x7fdf040a1b50 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:53:44.436 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.429+0000 7fdf22649700 1 --2- 172.21.15.38:0/2579016759 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fdf1c114bb0 0x7fdf1c115000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:53:44.436 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.429+0000 7fdf22649700 1 --2- 172.21.15.38:0/2579016759 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fdf1c107b50 0x7fdf1c118b60 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:53:44.436 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.429+0000 7fdf22649700 1 --2- 172.21.15.38:0/2579016759 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fdf1c1196d0 0x7fdf1c1190a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:53:44.436 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.429+0000 7fdf22649700 1 -- 172.21.15.38:0/2579016759 >> 172.21.15.38:0/2579016759 conn(0x7fdf1c06b450 msgr2=0x7fdf1c11b880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:53:44.436 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.430+0000 7fdf22649700 1 -- 172.21.15.38:0/2579016759 shutdown_connections 2023-03-09T14:53:44.437 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:53:44.430+0000 7fdf22649700 1 -- 172.21.15.38:0/2579016759 wait complete. 2023-03-09T14:53:44.455 INFO:tasks.cephfs.filesystem:scrub status for tag:a14659f4-d7ea-4c5a-b15b-6138af3f909f - {'path': '/', 'tag': 'a14659f4-d7ea-4c5a-b15b-6138af3f909f', 'options': 'recursive,force'} 2023-03-09T14:53:44.490 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:44 smithi122 bash[132019]: 05ae404b68020028ffd68cb142916d3129f70c653ad460dc6d775f1cf7fb28a6 2023-03-09T14:53:44.490 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:44 smithi122 systemd[1]: Started Ceph osd.7 for c6733fe2-be87-11ed-9af2-001a4aab830c. 2023-03-09T14:53:44.490 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:44 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[132036]: 2023-03-09T14:53:44.484+0000 7fef45588640 -1 Falling back to public interface 2023-03-09T14:53:46.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:45 smithi122 ceph-mon[93869]: pgmap v344: 129 pgs: 15 active+undersized, 22 active+undersized+degraded, 92 active+clean; 18 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 24 MiB/s wr, 2.05k op/s; 4519/44467 objects degraded (10.163%); 19 MiB/s, 4 objects/s recovering 2023-03-09T14:53:46.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:45 smithi181 ceph-mon[94189]: pgmap v344: 129 pgs: 15 active+undersized, 22 active+undersized+degraded, 92 active+clean; 18 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 24 MiB/s wr, 2.05k op/s; 4519/44467 objects degraded (10.163%); 19 MiB/s, 4 objects/s recovering 2023-03-09T14:53:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:45 smithi038 ceph-mon[92422]: pgmap v344: 129 pgs: 15 active+undersized, 22 active+undersized+degraded, 92 active+clean; 18 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 24 MiB/s wr, 2.05k op/s; 4519/44467 objects degraded (10.163%); 19 MiB/s, 4 objects/s recovering 2023-03-09T14:53:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:47 smithi122 ceph-mon[93869]: pgmap v345: 129 pgs: 15 active+undersized, 22 active+undersized+degraded, 92 active+clean; 18 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 30 MiB/s wr, 2.74k op/s; 4675/46203 objects degraded (10.118%); 19 MiB/s, 4 objects/s recovering 2023-03-09T14:53:48.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:47 smithi122 ceph-mon[93869]: Health check update: Degraded data redundancy: 4675/46203 objects degraded (10.118%), 22 pgs degraded (PG_DEGRADED) 2023-03-09T14:53:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:47 smithi181 ceph-mon[94189]: pgmap v345: 129 pgs: 15 active+undersized, 22 active+undersized+degraded, 92 active+clean; 18 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 30 MiB/s wr, 2.74k op/s; 4675/46203 objects degraded (10.118%); 19 MiB/s, 4 objects/s recovering 2023-03-09T14:53:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:47 smithi181 ceph-mon[94189]: Health check update: Degraded data redundancy: 4675/46203 objects degraded (10.118%), 22 pgs degraded (PG_DEGRADED) 2023-03-09T14:53:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:47 smithi038 ceph-mon[92422]: pgmap v345: 129 pgs: 15 active+undersized, 22 active+undersized+degraded, 92 active+clean; 18 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 30 MiB/s wr, 2.74k op/s; 4675/46203 objects degraded (10.118%); 19 MiB/s, 4 objects/s recovering 2023-03-09T14:53:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:47 smithi038 ceph-mon[92422]: Health check update: Degraded data redundancy: 4675/46203 objects degraded (10.118%), 22 pgs degraded (PG_DEGRADED) 2023-03-09T14:53:48.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 200.0 s, 593.6 tps, lat 168.697 ms stddev 336.292 2023-03-09T14:53:50.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:49 smithi122 ceph-mon[93869]: pgmap v346: 129 pgs: 15 active+undersized, 22 active+undersized+degraded, 92 active+clean; 18 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 32 MiB/s wr, 2.85k op/s; 4830/48163 objects degraded (10.028%) 2023-03-09T14:53:50.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:49 smithi181 ceph-mon[94189]: pgmap v346: 129 pgs: 15 active+undersized, 22 active+undersized+degraded, 92 active+clean; 18 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 32 MiB/s wr, 2.85k op/s; 4830/48163 objects degraded (10.028%) 2023-03-09T14:53:50.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:49 smithi038 ceph-mon[92422]: pgmap v346: 129 pgs: 15 active+undersized, 22 active+undersized+degraded, 92 active+clean; 18 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 32 MiB/s wr, 2.85k op/s; 4830/48163 objects degraded (10.028%) 2023-03-09T14:53:52.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:51 smithi122 ceph-mon[93869]: pgmap v347: 129 pgs: 15 active+undersized, 22 active+undersized+degraded, 92 active+clean; 18 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 33 MiB/s wr, 3.01k op/s; 5029/49766 objects degraded (10.105%) 2023-03-09T14:53:52.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:51 smithi181 ceph-mon[94189]: pgmap v347: 129 pgs: 15 active+undersized, 22 active+undersized+degraded, 92 active+clean; 18 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 33 MiB/s wr, 3.01k op/s; 5029/49766 objects degraded (10.105%) 2023-03-09T14:53:52.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:51 smithi038 ceph-mon[92422]: pgmap v347: 129 pgs: 15 active+undersized, 22 active+undersized+degraded, 92 active+clean; 18 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 33 MiB/s wr, 3.01k op/s; 5029/49766 objects degraded (10.105%) 2023-03-09T14:53:53.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:52 smithi122 ceph-mon[93869]: Health check update: Degraded data redundancy: 5029/49766 objects degraded (10.105%), 22 pgs degraded (PG_DEGRADED) 2023-03-09T14:53:53.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:52 smithi181 ceph-mon[94189]: Health check update: Degraded data redundancy: 5029/49766 objects degraded (10.105%), 22 pgs degraded (PG_DEGRADED) 2023-03-09T14:53:53.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:52 smithi038 ceph-mon[92422]: Health check update: Degraded data redundancy: 5029/49766 objects degraded (10.105%), 22 pgs degraded (PG_DEGRADED) 2023-03-09T14:53:53.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 205.0 s, 1307.2 tps, lat 76.239 ms stddev 42.974 2023-03-09T14:53:53.861 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[132036]: 2023-03-09T14:53:53.565+0000 7fef45588640 -1 osd.7 196 log_to_monitors true 2023-03-09T14:53:53.862 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:53 smithi122 ceph-mon[93869]: pgmap v348: 129 pgs: 15 active+undersized, 22 active+undersized+degraded, 92 active+clean; 18 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 29 MiB/s wr, 2.75k op/s; 5096/50290 objects degraded (10.133%) 2023-03-09T14:53:53.862 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:53 smithi122 ceph-mon[93869]: from='osd.7 [v2:172.21.15.122:6816/2859629451,v1:172.21.15.122:6817/2859629451]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-03-09T14:53:53.862 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:53 smithi122 ceph-mon[93869]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-03-09T14:53:54.228 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 14:53:53 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[132036]: 2023-03-09T14:53:53.873+0000 7fef33aff700 -1 osd.7 196 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-09T14:53:54.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:53 smithi181 ceph-mon[94189]: pgmap v348: 129 pgs: 15 active+undersized, 22 active+undersized+degraded, 92 active+clean; 18 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 29 MiB/s wr, 2.75k op/s; 5096/50290 objects degraded (10.133%) 2023-03-09T14:53:54.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:53 smithi181 ceph-mon[94189]: from='osd.7 [v2:172.21.15.122:6816/2859629451,v1:172.21.15.122:6817/2859629451]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-03-09T14:53:54.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:53 smithi181 ceph-mon[94189]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-03-09T14:53:54.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:53 smithi038 ceph-mon[92422]: pgmap v348: 129 pgs: 15 active+undersized, 22 active+undersized+degraded, 92 active+clean; 18 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 29 MiB/s wr, 2.75k op/s; 5096/50290 objects degraded (10.133%) 2023-03-09T14:53:54.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:53 smithi038 ceph-mon[92422]: from='osd.7 [v2:172.21.15.122:6816/2859629451,v1:172.21.15.122:6817/2859629451]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-03-09T14:53:54.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:53 smithi038 ceph-mon[92422]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-03-09T14:53:55.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:54 smithi122 ceph-mon[93869]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-03-09T14:53:55.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:54 smithi122 ceph-mon[93869]: osdmap e207: 12 total, 11 up, 12 in 2023-03-09T14:53:55.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:54 smithi122 ceph-mon[93869]: from='osd.7 [v2:172.21.15.122:6816/2859629451,v1:172.21.15.122:6817/2859629451]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-03-09T14:53:55.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:54 smithi122 ceph-mon[93869]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-03-09T14:53:55.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:54 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T14:53:55.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:54 smithi181 ceph-mon[94189]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-03-09T14:53:55.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:54 smithi181 ceph-mon[94189]: osdmap e207: 12 total, 11 up, 12 in 2023-03-09T14:53:55.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:54 smithi181 ceph-mon[94189]: from='osd.7 [v2:172.21.15.122:6816/2859629451,v1:172.21.15.122:6817/2859629451]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-03-09T14:53:55.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:54 smithi181 ceph-mon[94189]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-03-09T14:53:55.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:54 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T14:53:55.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:54 smithi038 ceph-mon[92422]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-03-09T14:53:55.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:54 smithi038 ceph-mon[92422]: osdmap e207: 12 total, 11 up, 12 in 2023-03-09T14:53:55.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:54 smithi038 ceph-mon[92422]: from='osd.7 [v2:172.21.15.122:6816/2859629451,v1:172.21.15.122:6817/2859629451]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-03-09T14:53:55.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:54 smithi038 ceph-mon[92422]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi122", "root=default"]}]: dispatch 2023-03-09T14:53:55.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:54 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T14:53:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:55 smithi122 ceph-mon[93869]: pgmap v350: 129 pgs: 15 active+undersized, 22 active+undersized+degraded, 92 active+clean; 18 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 25 MiB/s wr, 3.06k op/s; 5098/50310 objects degraded (10.133%) 2023-03-09T14:53:56.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:55 smithi122 ceph-mon[93869]: Health check cleared: OSD_DOWN (was: 1 osds down) 2023-03-09T14:53:56.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:55 smithi122 ceph-mon[93869]: osd.7 [v2:172.21.15.122:6816/2859629451,v1:172.21.15.122:6817/2859629451] boot 2023-03-09T14:53:56.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:55 smithi122 ceph-mon[93869]: osdmap e208: 12 total, 12 up, 12 in 2023-03-09T14:53:56.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:55 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-03-09T14:53:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:55 smithi181 ceph-mon[94189]: pgmap v350: 129 pgs: 15 active+undersized, 22 active+undersized+degraded, 92 active+clean; 18 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 25 MiB/s wr, 3.06k op/s; 5098/50310 objects degraded (10.133%) 2023-03-09T14:53:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:55 smithi181 ceph-mon[94189]: Health check cleared: OSD_DOWN (was: 1 osds down) 2023-03-09T14:53:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:55 smithi181 ceph-mon[94189]: osd.7 [v2:172.21.15.122:6816/2859629451,v1:172.21.15.122:6817/2859629451] boot 2023-03-09T14:53:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:55 smithi181 ceph-mon[94189]: osdmap e208: 12 total, 12 up, 12 in 2023-03-09T14:53:56.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:55 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-03-09T14:53:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:55 smithi038 ceph-mon[92422]: pgmap v350: 129 pgs: 15 active+undersized, 22 active+undersized+degraded, 92 active+clean; 18 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 25 MiB/s wr, 3.06k op/s; 5098/50310 objects degraded (10.133%) 2023-03-09T14:53:56.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:55 smithi038 ceph-mon[92422]: Health check cleared: OSD_DOWN (was: 1 osds down) 2023-03-09T14:53:56.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:55 smithi038 ceph-mon[92422]: osd.7 [v2:172.21.15.122:6816/2859629451,v1:172.21.15.122:6817/2859629451] boot 2023-03-09T14:53:56.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:55 smithi038 ceph-mon[92422]: osdmap e208: 12 total, 12 up, 12 in 2023-03-09T14:53:56.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:55 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-03-09T14:53:57.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:56 smithi122 ceph-mon[93869]: osdmap e209: 12 total, 12 up, 12 in 2023-03-09T14:53:57.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:56 smithi122 ceph-mon[93869]: osdmap e210: 12 total, 12 up, 12 in 2023-03-09T14:53:57.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:56 smithi181 ceph-mon[94189]: osdmap e209: 12 total, 12 up, 12 in 2023-03-09T14:53:57.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:56 smithi181 ceph-mon[94189]: osdmap e210: 12 total, 12 up, 12 in 2023-03-09T14:53:57.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:56 smithi038 ceph-mon[92422]: osdmap e209: 12 total, 12 up, 12 in 2023-03-09T14:53:57.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:56 smithi038 ceph-mon[92422]: osdmap e210: 12 total, 12 up, 12 in 2023-03-09T14:53:58.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:57 smithi122 ceph-mon[93869]: pgmap v353: 129 pgs: 10 peering, 14 active+undersized, 13 active+undersized+degraded, 92 active+clean; 18 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 17 MiB/s wr, 2.24k op/s; 3165/50362 objects degraded (6.285%) 2023-03-09T14:53:58.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:57 smithi122 ceph-mon[93869]: Health check update: Degraded data redundancy: 3165/50362 objects degraded (6.285%), 13 pgs degraded (PG_DEGRADED) 2023-03-09T14:53:58.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:57 smithi122 ceph-mon[93869]: osdmap e211: 12 total, 12 up, 12 in 2023-03-09T14:53:58.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:57 smithi181 ceph-mon[94189]: pgmap v353: 129 pgs: 10 peering, 14 active+undersized, 13 active+undersized+degraded, 92 active+clean; 18 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 17 MiB/s wr, 2.24k op/s; 3165/50362 objects degraded (6.285%) 2023-03-09T14:53:58.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:57 smithi181 ceph-mon[94189]: Health check update: Degraded data redundancy: 3165/50362 objects degraded (6.285%), 13 pgs degraded (PG_DEGRADED) 2023-03-09T14:53:58.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:57 smithi181 ceph-mon[94189]: osdmap e211: 12 total, 12 up, 12 in 2023-03-09T14:53:58.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:57 smithi038 ceph-mon[92422]: pgmap v353: 129 pgs: 10 peering, 14 active+undersized, 13 active+undersized+degraded, 92 active+clean; 18 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 17 MiB/s wr, 2.24k op/s; 3165/50362 objects degraded (6.285%) 2023-03-09T14:53:58.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:57 smithi038 ceph-mon[92422]: Health check update: Degraded data redundancy: 3165/50362 objects degraded (6.285%), 13 pgs degraded (PG_DEGRADED) 2023-03-09T14:53:58.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:57 smithi038 ceph-mon[92422]: osdmap e211: 12 total, 12 up, 12 in 2023-03-09T14:53:58.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 210.0 s, 695.6 tps, lat 143.171 ms stddev 351.859 2023-03-09T14:53:59.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:59 smithi122 ceph-mon[93869]: osdmap e212: 12 total, 12 up, 12 in 2023-03-09T14:53:59.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:59 smithi122 ceph-mon[93869]: pgmap v357: 129 pgs: 11 peering, 11 active+undersized, 3 active+recovery_wait+undersized+degraded+remapped, 7 active+undersized+degraded, 97 active+clean; 18 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 21 MiB/s wr, 1.96k op/s; 2254/50402 objects degraded (4.472%); 345 KiB/s, 0 objects/s recovering 2023-03-09T14:53:59.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:53:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T14:53:59.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:59 smithi181 ceph-mon[94189]: osdmap e212: 12 total, 12 up, 12 in 2023-03-09T14:53:59.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:59 smithi181 ceph-mon[94189]: pgmap v357: 129 pgs: 11 peering, 11 active+undersized, 3 active+recovery_wait+undersized+degraded+remapped, 7 active+undersized+degraded, 97 active+clean; 18 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 21 MiB/s wr, 1.96k op/s; 2254/50402 objects degraded (4.472%); 345 KiB/s, 0 objects/s recovering 2023-03-09T14:53:59.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:53:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T14:54:00.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:59 smithi038 ceph-mon[92422]: osdmap e212: 12 total, 12 up, 12 in 2023-03-09T14:54:00.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:59 smithi038 ceph-mon[92422]: pgmap v357: 129 pgs: 11 peering, 11 active+undersized, 3 active+recovery_wait+undersized+degraded+remapped, 7 active+undersized+degraded, 97 active+clean; 18 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 21 MiB/s wr, 1.96k op/s; 2254/50402 objects degraded (4.472%); 345 KiB/s, 0 objects/s recovering 2023-03-09T14:54:00.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:53:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T14:54:01.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:54:00 smithi122 ceph-mon[93869]: osdmap e213: 12 total, 12 up, 12 in 2023-03-09T14:54:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:54:00 smithi181 ceph-mon[94189]: osdmap e213: 12 total, 12 up, 12 in 2023-03-09T14:54:01.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:54:00 smithi038 ceph-mon[92422]: osdmap e213: 12 total, 12 up, 12 in 2023-03-09T14:54:02.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:54:01 smithi122 ceph-mon[93869]: pgmap v359: 129 pgs: 8 peering, 1 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded, 112 active+clean; 18 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 12 MiB/s wr, 1.20k op/s; 1504/50436 objects degraded (2.982%); 182 MiB/s, 47 objects/s recovering 2023-03-09T14:54:02.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:54:01 smithi122 ceph-mon[93869]: osdmap e214: 12 total, 12 up, 12 in 2023-03-09T14:54:02.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:54:01 smithi181 ceph-mon[94189]: pgmap v359: 129 pgs: 8 peering, 1 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded, 112 active+clean; 18 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 12 MiB/s wr, 1.20k op/s; 1504/50436 objects degraded (2.982%); 182 MiB/s, 47 objects/s recovering 2023-03-09T14:54:02.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:54:01 smithi181 ceph-mon[94189]: osdmap e214: 12 total, 12 up, 12 in 2023-03-09T14:54:02.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:54:01 smithi038 ceph-mon[92422]: pgmap v359: 129 pgs: 8 peering, 1 active+recovering+undersized+degraded+remapped, 7 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+degraded, 112 active+clean; 18 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 12 MiB/s wr, 1.20k op/s; 1504/50436 objects degraded (2.982%); 182 MiB/s, 47 objects/s recovering 2023-03-09T14:54:02.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:54:01 smithi038 ceph-mon[92422]: osdmap e214: 12 total, 12 up, 12 in 2023-03-09T14:54:03.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:54:02 smithi122 ceph-mon[93869]: Health check update: Degraded data redundancy: 1504/50436 objects degraded (2.982%), 9 pgs degraded (PG_DEGRADED) 2023-03-09T14:54:03.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:54:02 smithi122 ceph-mon[93869]: osdmap e215: 12 total, 12 up, 12 in 2023-03-09T14:54:03.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:54:02 smithi181 ceph-mon[94189]: Health check update: Degraded data redundancy: 1504/50436 objects degraded (2.982%), 9 pgs degraded (PG_DEGRADED) 2023-03-09T14:54:03.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:54:02 smithi181 ceph-mon[94189]: osdmap e215: 12 total, 12 up, 12 in 2023-03-09T14:54:03.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:54:02 smithi038 ceph-mon[92422]: Health check update: Degraded data redundancy: 1504/50436 objects degraded (2.982%), 9 pgs degraded (PG_DEGRADED) 2023-03-09T14:54:03.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:54:02 smithi038 ceph-mon[92422]: osdmap e215: 12 total, 12 up, 12 in 2023-03-09T14:54:03.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 215.0 s, 195.9 tps, lat 255.758 ms stddev 594.059 2023-03-09T14:54:04.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:54:03 smithi122 ceph-mon[93869]: pgmap v362: 129 pgs: 2 peering, 1 active+recovering+undersized+degraded+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 18 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 10 MiB/s wr, 1.41k op/s; 1393/50471 objects degraded (2.760%); 180 MiB/s, 46 objects/s recovering 2023-03-09T14:54:04.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:54:03 smithi122 ceph-mon[93869]: osdmap e216: 12 total, 12 up, 12 in 2023-03-09T14:54:04.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:54:03 smithi181 ceph-mon[94189]: pgmap v362: 129 pgs: 2 peering, 1 active+recovering+undersized+degraded+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 18 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 10 MiB/s wr, 1.41k op/s; 1393/50471 objects degraded (2.760%); 180 MiB/s, 46 objects/s recovering 2023-03-09T14:54:04.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:54:03 smithi181 ceph-mon[94189]: osdmap e216: 12 total, 12 up, 12 in 2023-03-09T14:54:04.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:54:03 smithi038 ceph-mon[92422]: pgmap v362: 129 pgs: 2 peering, 1 active+recovering+undersized+degraded+remapped, 10 active+recovery_wait+undersized+degraded+remapped, 116 active+clean; 18 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 10 MiB/s wr, 1.41k op/s; 1393/50471 objects degraded (2.760%); 180 MiB/s, 46 objects/s recovering 2023-03-09T14:54:04.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:54:03 smithi038 ceph-mon[92422]: osdmap e216: 12 total, 12 up, 12 in 2023-03-09T14:54:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:54:05 smithi122 ceph-mon[93869]: osdmap e217: 12 total, 12 up, 12 in 2023-03-09T14:54:05.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:54:05 smithi122 ceph-mon[93869]: pgmap v365: 129 pgs: 1 peering, 2 active+recovering+undersized+degraded+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 18 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 13 MiB/s wr, 1.72k op/s; 1332/50490 objects degraded (2.638%); 59 MiB/s, 15 objects/s recovering 2023-03-09T14:54:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:54:05 smithi181 ceph-mon[94189]: osdmap e217: 12 total, 12 up, 12 in 2023-03-09T14:54:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:54:05 smithi181 ceph-mon[94189]: pgmap v365: 129 pgs: 1 peering, 2 active+recovering+undersized+degraded+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 18 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 13 MiB/s wr, 1.72k op/s; 1332/50490 objects degraded (2.638%); 59 MiB/s, 15 objects/s recovering 2023-03-09T14:54:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:54:05 smithi038 ceph-mon[92422]: osdmap e217: 12 total, 12 up, 12 in 2023-03-09T14:54:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:54:05 smithi038 ceph-mon[92422]: pgmap v365: 129 pgs: 1 peering, 2 active+recovering+undersized+degraded+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 117 active+clean; 18 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 13 MiB/s wr, 1.72k op/s; 1332/50490 objects degraded (2.638%); 59 MiB/s, 15 objects/s recovering 2023-03-09T14:54:06.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:54:06 smithi122 ceph-mon[93869]: osdmap e218: 12 total, 12 up, 12 in 2023-03-09T14:54:06.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:54:06 smithi181 ceph-mon[94189]: osdmap e218: 12 total, 12 up, 12 in 2023-03-09T14:54:07.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:54:06 smithi038 ceph-mon[92422]: osdmap e218: 12 total, 12 up, 12 in 2023-03-09T14:54:07.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:54:07 smithi122 ceph-mon[93869]: osdmap e219: 12 total, 12 up, 12 in 2023-03-09T14:54:07.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:54:07 smithi122 ceph-mon[93869]: pgmap v368: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 1 active+recovering+undersized+degraded+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 18 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 16 MiB/s wr, 1.97k op/s; 1248/50151 objects degraded (2.488%); 143 MiB/s, 36 objects/s recovering 2023-03-09T14:54:07.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:54:07 smithi122 ceph-mon[93869]: Health check update: Degraded data redundancy: 1332/50490 objects degraded (2.638%), 11 pgs degraded (PG_DEGRADED) 2023-03-09T14:54:07.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:54:07 smithi181 ceph-mon[94189]: osdmap e219: 12 total, 12 up, 12 in 2023-03-09T14:54:07.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:54:07 smithi181 ceph-mon[94189]: pgmap v368: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 1 active+recovering+undersized+degraded+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 18 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 16 MiB/s wr, 1.97k op/s; 1248/50151 objects degraded (2.488%); 143 MiB/s, 36 objects/s recovering 2023-03-09T14:54:07.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:54:07 smithi181 ceph-mon[94189]: Health check update: Degraded data redundancy: 1332/50490 objects degraded (2.638%), 11 pgs degraded (PG_DEGRADED) 2023-03-09T14:54:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:54:07 smithi038 ceph-mon[92422]: osdmap e219: 12 total, 12 up, 12 in 2023-03-09T14:54:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:54:07 smithi038 ceph-mon[92422]: pgmap v368: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 1 active+recovering+undersized+degraded+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 18 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 16 MiB/s wr, 1.97k op/s; 1248/50151 objects degraded (2.488%); 143 MiB/s, 36 objects/s recovering 2023-03-09T14:54:08.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:54:07 smithi038 ceph-mon[92422]: Health check update: Degraded data redundancy: 1332/50490 objects degraded (2.638%), 11 pgs degraded (PG_DEGRADED) 2023-03-09T14:54:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 220.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T14:54:09.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:54:09 smithi181 ceph-mon[94189]: osdmap e220: 12 total, 12 up, 12 in 2023-03-09T14:54:09.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:54:09 smithi181 ceph-mon[94189]: pgmap v370: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 1 active+recovering+undersized+degraded+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 18 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 452 KiB/s rd, 8.0 MiB/s wr, 1.05k op/s; 1263/50051 objects degraded (2.523%); 83 MiB/s, 20 objects/s recovering 2023-03-09T14:54:10.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:54:09 smithi038 ceph-mon[92422]: osdmap e220: 12 total, 12 up, 12 in 2023-03-09T14:54:10.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:54:09 smithi038 ceph-mon[92422]: pgmap v370: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 1 active+recovering+undersized+degraded+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 18 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 452 KiB/s rd, 8.0 MiB/s wr, 1.05k op/s; 1263/50051 objects degraded (2.523%); 83 MiB/s, 20 objects/s recovering 2023-03-09T14:54:10.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:54:09 smithi122 ceph-mon[93869]: osdmap e220: 12 total, 12 up, 12 in 2023-03-09T14:54:10.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:54:09 smithi122 ceph-mon[93869]: pgmap v370: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 1 active+recovering+undersized+degraded+remapped, 9 active+recovery_wait+undersized+degraded+remapped, 110 active+clean; 18 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 452 KiB/s rd, 8.0 MiB/s wr, 1.05k op/s; 1263/50051 objects degraded (2.523%); 83 MiB/s, 20 objects/s recovering 2023-03-09T14:54:11.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:54:10 smithi038 ceph-mon[92422]: osdmap e221: 12 total, 12 up, 12 in 2023-03-09T14:54:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:54:10 smithi122 ceph-mon[93869]: osdmap e221: 12 total, 12 up, 12 in 2023-03-09T14:54:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:54:10 smithi181 ceph-mon[94189]: osdmap e221: 12 total, 12 up, 12 in 2023-03-09T14:54:12.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:54:11 smithi038 ceph-mon[92422]: pgmap v372: 129 pgs: 1 peering, 8 active+recovery_wait+undersized+degraded+remapped, 120 active+clean; 18 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 3.7 MiB/s wr, 493 op/s; 1068/50007 objects degraded (2.136%); 231 MiB/s, 57 objects/s recovering 2023-03-09T14:54:12.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:54:11 smithi122 ceph-mon[93869]: pgmap v372: 129 pgs: 1 peering, 8 active+recovery_wait+undersized+degraded+remapped, 120 active+clean; 18 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 3.7 MiB/s wr, 493 op/s; 1068/50007 objects degraded (2.136%); 231 MiB/s, 57 objects/s recovering 2023-03-09T14:54:12.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:54:11 smithi181 ceph-mon[94189]: pgmap v372: 129 pgs: 1 peering, 8 active+recovery_wait+undersized+degraded+remapped, 120 active+clean; 18 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 3.7 MiB/s wr, 493 op/s; 1068/50007 objects degraded (2.136%); 231 MiB/s, 57 objects/s recovering 2023-03-09T14:54:13.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:54:12 smithi038 ceph-mon[92422]: osdmap e222: 12 total, 12 up, 12 in 2023-03-09T14:54:13.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:54:12 smithi038 ceph-mon[92422]: Health check update: Degraded data redundancy: 1068/50007 objects degraded (2.136%), 8 pgs degraded (PG_DEGRADED) 2023-03-09T14:54:13.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:54:12 smithi038 ceph-mon[92422]: osdmap e223: 12 total, 12 up, 12 in 2023-03-09T14:54:13.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:54:12 smithi122 ceph-mon[93869]: osdmap e222: 12 total, 12 up, 12 in 2023-03-09T14:54:13.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:54:12 smithi122 ceph-mon[93869]: Health check update: Degraded data redundancy: 1068/50007 objects degraded (2.136%), 8 pgs degraded (PG_DEGRADED) 2023-03-09T14:54:13.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:54:12 smithi122 ceph-mon[93869]: osdmap e223: 12 total, 12 up, 12 in 2023-03-09T14:54:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:54:12 smithi181 ceph-mon[94189]: osdmap e222: 12 total, 12 up, 12 in 2023-03-09T14:54:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:54:12 smithi181 ceph-mon[94189]: Health check update: Degraded data redundancy: 1068/50007 objects degraded (2.136%), 8 pgs degraded (PG_DEGRADED) 2023-03-09T14:54:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:54:12 smithi181 ceph-mon[94189]: osdmap e223: 12 total, 12 up, 12 in 2023-03-09T14:54:13.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 225.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T14:54:14.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:54:13 smithi038 ceph-mon[92422]: pgmap v375: 129 pgs: 1 peering, 8 active+recovery_wait+undersized+degraded+remapped, 120 active+clean; 18 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 326 KiB/s wr, 42 op/s; 1081/50139 objects degraded (2.156%); 206 MiB/s, 51 objects/s recovering 2023-03-09T14:54:14.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:54:13 smithi122 ceph-mon[93869]: pgmap v375: 129 pgs: 1 peering, 8 active+recovery_wait+undersized+degraded+remapped, 120 active+clean; 18 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 326 KiB/s wr, 42 op/s; 1081/50139 objects degraded (2.156%); 206 MiB/s, 51 objects/s recovering 2023-03-09T14:54:14.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:54:13 smithi181 ceph-mon[94189]: pgmap v375: 129 pgs: 1 peering, 8 active+recovery_wait+undersized+degraded+remapped, 120 active+clean; 18 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 326 KiB/s wr, 42 op/s; 1081/50139 objects degraded (2.156%); 206 MiB/s, 51 objects/s recovering 2023-03-09T14:54:14.456 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-03-09T14:54:14.600 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.597+0000 7f155eb28700 1 -- 172.21.15.38:0/2780825609 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f155810a720 msgr2=0x7f155810cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:54:14.601 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.597+0000 7f155eb28700 1 --2- 172.21.15.38:0/2780825609 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f155810a720 0x7f155810cb00 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f155000b0a0 tx=0x7f155000b3b0 comp rx=0 tx=0).stop 2023-03-09T14:54:14.601 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.597+0000 7f155eb28700 1 -- 172.21.15.38:0/2780825609 shutdown_connections 2023-03-09T14:54:14.601 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.597+0000 7f155eb28700 1 --2- 172.21.15.38:0/2780825609 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f155810a720 0x7f155810cb00 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:54:14.601 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.597+0000 7f155eb28700 1 --2- 172.21.15.38:0/2780825609 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f15580675f0 0x7f15580ff300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:54:14.602 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.597+0000 7f155eb28700 1 --2- 172.21.15.38:0/2780825609 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15580ff8d0 0x7f15580ffd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:54:14.602 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.597+0000 7f155eb28700 1 -- 172.21.15.38:0/2780825609 >> 172.21.15.38:0/2780825609 conn(0x7f15580748f0 msgr2=0x7f1558074cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:54:14.602 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.597+0000 7f155eb28700 1 -- 172.21.15.38:0/2780825609 shutdown_connections 2023-03-09T14:54:14.602 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.597+0000 7f155eb28700 1 -- 172.21.15.38:0/2780825609 wait complete. 2023-03-09T14:54:14.602 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.598+0000 7f155eb28700 1 Processor -- start 2023-03-09T14:54:14.602 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.598+0000 7f155eb28700 1 -- start start 2023-03-09T14:54:14.603 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.598+0000 7f155eb28700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f15580675f0 0x7f15581004e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:54:14.604 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.598+0000 7f155eb28700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15580ff8d0 0x7f1558100a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:54:14.604 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.598+0000 7f155eb28700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f155810a720 0x7f15581a4560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:54:14.604 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.598+0000 7f155eb28700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1558101250 con 0x7f15580ff8d0 2023-03-09T14:54:14.604 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.598+0000 7f155eb28700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15581a4aa0 con 0x7f155810a720 2023-03-09T14:54:14.604 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.598+0000 7f155eb28700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15581a4be0 con 0x7f15580675f0 2023-03-09T14:54:14.604 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.598+0000 7f1557fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15580ff8d0 0x7f1558100a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:54:14.604 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.598+0000 7f1557fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15580ff8d0 0x7f1558100a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:41814/0 (socket says 172.21.15.38:41814) 2023-03-09T14:54:14.605 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.598+0000 7f1557fff700 1 -- 172.21.15.38:0/1504508299 learned_addr learned my addr 172.21.15.38:0/1504508299 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:54:14.605 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.598+0000 7f155d0c5700 1 --2- 172.21.15.38:0/1504508299 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f155810a720 0x7f15581a4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:54:14.605 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.598+0000 7f155c8c4700 1 --2- 172.21.15.38:0/1504508299 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f15580675f0 0x7f15581004e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:54:14.605 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.598+0000 7f1557fff700 1 -- 172.21.15.38:0/1504508299 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f15580675f0 msgr2=0x7f15581004e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:54:14.605 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.598+0000 7f1557fff700 1 --2- 172.21.15.38:0/1504508299 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f15580675f0 0x7f15581004e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:54:14.605 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.598+0000 7f1557fff700 1 -- 172.21.15.38:0/1504508299 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f155810a720 msgr2=0x7f15581a4560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:54:14.605 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.598+0000 7f1557fff700 1 --2- 172.21.15.38:0/1504508299 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f155810a720 0x7f15581a4560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:54:14.606 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.598+0000 7f1557fff700 1 -- 172.21.15.38:0/1504508299 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1550009d00 con 0x7f15580ff8d0 2023-03-09T14:54:14.606 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.599+0000 7f1557fff700 1 --2- 172.21.15.38:0/1504508299 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15580ff8d0 0x7f1558100a20 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f154400e970 tx=0x7f154400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:54:14.606 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.599+0000 7f1555ffb700 1 -- 172.21.15.38:0/1504508299 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1544009790 con 0x7f15580ff8d0 2023-03-09T14:54:14.606 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.599+0000 7f1555ffb700 1 -- 172.21.15.38:0/1504508299 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f154400c7e0 con 0x7f15580ff8d0 2023-03-09T14:54:14.606 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.599+0000 7f155eb28700 1 -- 172.21.15.38:0/1504508299 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15581100e0 con 0x7f15580ff8d0 2023-03-09T14:54:14.606 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.599+0000 7f1555ffb700 1 -- 172.21.15.38:0/1504508299 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1544003c70 con 0x7f15580ff8d0 2023-03-09T14:54:14.606 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.599+0000 7f155eb28700 1 -- 172.21.15.38:0/1504508299 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f155810ff50 con 0x7f15580ff8d0 2023-03-09T14:54:14.608 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.601+0000 7f1555ffb700 1 -- 172.21.15.38:0/1504508299 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 25) v1 ==== 133295+0+0 (secure 0 0 0) 0x7f154401e020 con 0x7f15580ff8d0 2023-03-09T14:54:14.609 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.602+0000 7f1555ffb700 1 --2- 172.21.15.38:0/1504508299 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f154809f4a0 0x7f15480a1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:54:14.609 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.602+0000 7f155c8c4700 1 --2- 172.21.15.38:0/1504508299 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f154809f4a0 0x7f15480a1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:54:14.609 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.602+0000 7f1555ffb700 1 -- 172.21.15.38:0/1504508299 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(224..224 src has 1..224) v4 ==== 9771+0+0 (secure 0 0 0) 0x7f1544014070 con 0x7f15580ff8d0 2023-03-09T14:54:14.609 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.603+0000 7f155c8c4700 1 --2- 172.21.15.38:0/1504508299 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f154809f4a0 0x7f15480a1950 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f154c009c60 tx=0x7f154c009260 comp rx=0 tx=0).ready entity=mgr.15558 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:54:14.609 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.603+0000 7f156acb1b80 1 Processor -- start 2023-03-09T14:54:14.609 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.603+0000 7f156acb1b80 1 -- start start 2023-03-09T14:54:14.609 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.603+0000 7f156acb1b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f9e4f80830 0x55f9e4f80c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:54:14.610 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.603+0000 7f156acb1b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9e4f81140 0x55f9e4f7aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:54:14.610 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.603+0000 7f156acb1b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f9e4f7afe0 0x55f9e4f7b410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:54:14.610 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.603+0000 7f156acb1b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9e4e64da0 con 0x55f9e4f81140 2023-03-09T14:54:14.610 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.603+0000 7f156acb1b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9e4ed3780 con 0x55f9e4f80830 2023-03-09T14:54:14.610 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.603+0000 7f156acb1b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9e4ecccb0 con 0x55f9e4f7afe0 2023-03-09T14:54:14.610 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.603+0000 7f1557fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9e4f81140 0x55f9e4f7aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:54:14.610 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.603+0000 7f155c8c4700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f9e4f80830 0x55f9e4f80c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:54:14.610 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.603+0000 7f155c8c4700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f9e4f80830 0x55f9e4f80c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.38:51010/0 (socket says 172.21.15.38:51010) 2023-03-09T14:54:14.611 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.603+0000 7f155d0c5700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f9e4f7afe0 0x55f9e4f7b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:54:14.611 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.603+0000 7f155c8c4700 1 -- 172.21.15.38:0/857792228 learned_addr learned my addr 172.21.15.38:0/857792228 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:54:14.611 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.603+0000 7f1557fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9e4f81140 0x55f9e4f7aaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:41822/0 (socket says 172.21.15.38:41822) 2023-03-09T14:54:14.611 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.604+0000 7f155c8c4700 1 -- 172.21.15.38:0/857792228 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f9e4f7afe0 msgr2=0x55f9e4f7b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:54:14.611 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.604+0000 7f155c8c4700 1 --2- 172.21.15.38:0/857792228 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f9e4f7afe0 0x55f9e4f7b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:54:14.611 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.604+0000 7f155c8c4700 1 -- 172.21.15.38:0/857792228 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9e4f81140 msgr2=0x55f9e4f7aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:54:14.611 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.604+0000 7f155c8c4700 1 --2- 172.21.15.38:0/857792228 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9e4f81140 0x55f9e4f7aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:54:14.612 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.604+0000 7f155c8c4700 1 -- 172.21.15.38:0/857792228 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f9e4e93190 con 0x55f9e4f80830 2023-03-09T14:54:14.612 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.604+0000 7f1557fff700 1 --2- 172.21.15.38:0/857792228 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9e4f81140 0x55f9e4f7aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:54:14.612 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.604+0000 7f155c8c4700 1 --2- 172.21.15.38:0/857792228 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f9e4f80830 0x55f9e4f80c00 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f154c00e4e0 tx=0x7f154c014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab00b145c8ec8b2f server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:54:14.612 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.604+0000 7f1542ffd700 1 -- 172.21.15.38:0/857792228 <== mon.2 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f154c00ee00 con 0x55f9e4f80830 2023-03-09T14:54:14.612 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.604+0000 7f1542ffd700 1 -- 172.21.15.38:0/857792228 <== mon.2 v2:172.21.15.122:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f154c021070 con 0x55f9e4f80830 2023-03-09T14:54:14.612 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.604+0000 7f1542ffd700 1 -- 172.21.15.38:0/857792228 <== mon.2 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f154c01d810 con 0x55f9e4f80830 2023-03-09T14:54:14.612 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.605+0000 7f156acb1b80 1 -- 172.21.15.38:0/857792228 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f9e4f80830 msgr2=0x55f9e4f80c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:54:14.612 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.605+0000 7f156acb1b80 1 --2- 172.21.15.38:0/857792228 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f9e4f80830 0x55f9e4f80c00 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f154c00e4e0 tx=0x7f154c014000 comp rx=0 tx=0).stop 2023-03-09T14:54:14.613 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.605+0000 7f156acb1b80 1 -- 172.21.15.38:0/857792228 shutdown_connections 2023-03-09T14:54:14.613 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.605+0000 7f156acb1b80 1 --2- 172.21.15.38:0/857792228 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f9e4f7afe0 0x55f9e4f7b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:54:14.613 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.605+0000 7f156acb1b80 1 --2- 172.21.15.38:0/857792228 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f9e4f80830 0x55f9e4f80c00 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:54:14.613 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.605+0000 7f156acb1b80 1 --2- 172.21.15.38:0/857792228 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9e4f81140 0x55f9e4f7aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:54:14.613 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.605+0000 7f156acb1b80 1 -- 172.21.15.38:0/857792228 >> 172.21.15.38:0/857792228 conn(0x55f9e4f75a20 msgr2=0x55f9e4f75df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:54:14.613 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.605+0000 7f156acb1b80 1 -- 172.21.15.38:0/857792228 shutdown_connections 2023-03-09T14:54:14.613 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.605+0000 7f156acb1b80 1 -- 172.21.15.38:0/857792228 wait complete. 2023-03-09T14:54:14.614 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.605+0000 7f156acb1b80 1 -- start start 2023-03-09T14:54:14.614 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.605+0000 7f156acb1b80 1 Processor -- start 2023-03-09T14:54:14.616 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.606+0000 7f156acb1b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f9e4f7afe0 0x55f9e4f871a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:54:14.616 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.606+0000 7f156acb1b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9e4f80830 0x55f9e4f84420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:54:14.617 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.606+0000 7f156acb1b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f9e4f81140 0x55f9e4f84ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:54:14.617 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.606+0000 7f156acb1b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9e4d50920 con 0x55f9e4f80830 2023-03-09T14:54:14.617 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.606+0000 7f156acb1b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9e4f85230 con 0x55f9e4f7afe0 2023-03-09T14:54:14.617 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.606+0000 7f156acb1b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9e4f90c00 con 0x55f9e4f81140 2023-03-09T14:54:14.617 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.606+0000 7f156acb1b80 10 client.0 _ll_register_callbacks cb 0x55f9e4e72520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:54:14.617 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.606+0000 7f1557fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9e4f80830 0x55f9e4f84420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:54:14.617 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.606+0000 7f155c8c4700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f9e4f7afe0 0x55f9e4f871a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:54:14.618 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.606+0000 7f155c8c4700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f9e4f7afe0 0x55f9e4f871a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.38:51016/0 (socket says 172.21.15.38:51016) 2023-03-09T14:54:14.618 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.606+0000 7f155c8c4700 1 -- 172.21.15.38:0/2702140413 learned_addr learned my addr 172.21.15.38:0/2702140413 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:54:14.618 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.606+0000 7f1557fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9e4f80830 0x55f9e4f84420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:41828/0 (socket says 172.21.15.38:41828) 2023-03-09T14:54:14.618 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.606+0000 7f155d0c5700 1 --2- 172.21.15.38:0/2702140413 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f9e4f81140 0x55f9e4f84ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:54:14.618 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.606+0000 7f1557fff700 1 -- 172.21.15.38:0/2702140413 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f9e4f81140 msgr2=0x55f9e4f84ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:54:14.618 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.606+0000 7f1557fff700 1 --2- 172.21.15.38:0/2702140413 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f9e4f81140 0x55f9e4f84ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:54:14.618 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.606+0000 7f1557fff700 1 -- 172.21.15.38:0/2702140413 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f9e4f7afe0 msgr2=0x55f9e4f871a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:54:14.618 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.606+0000 7f1557fff700 1 --2- 172.21.15.38:0/2702140413 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f9e4f7afe0 0x55f9e4f871a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:54:14.619 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.606+0000 7f1557fff700 1 -- 172.21.15.38:0/2702140413 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f155810ff50 con 0x55f9e4f80830 2023-03-09T14:54:14.619 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.607+0000 7f1557fff700 1 --2- 172.21.15.38:0/2702140413 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9e4f80830 0x55f9e4f84420 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f1544094af0 tx=0x7f1544003dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d12b7f5fb8c6e95 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:54:14.619 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.607+0000 7f1537fff700 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2023-03-09T14:54:14.619 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.607+0000 7f1537fff700 1 -- 172.21.15.38:0/2702140413 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15440b03d0 con 0x55f9e4f80830 2023-03-09T14:54:14.619 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.607+0000 7f1537fff700 1 -- 172.21.15.38:0/2702140413 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(224..224 src has 1..224) v4 ==== 9771+0+0 (secure 0 0 0) 0x7f15440982e0 con 0x55f9e4f80830 2023-03-09T14:54:14.619 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.607+0000 7f156acb1b80 1 -- 172.21.15.38:0/2702140413 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f9e4e9a190 con 0x55f9e4f80830 2023-03-09T14:54:14.619 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.607+0000 7f1537fff700 1 -- 172.21.15.38:0/2702140413 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f15440984a0 con 0x55f9e4f80830 2023-03-09T14:54:14.620 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.607+0000 7f156acb1b80 10 client.16185 fetch_fsmap learned FSMap version 33 2023-03-09T14:54:14.620 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.607+0000 7f156acb1b80 1 -- 172.21.15.38:0/2702140413 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x7f154c017070 con 0x55f9e4f80830 2023-03-09T14:54:14.620 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.607+0000 7f1537fff700 1 -- 172.21.15.38:0/2702140413 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1544064030 con 0x55f9e4f80830 2023-03-09T14:54:14.620 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.607+0000 7f1537fff700 1 -- 172.21.15.38:0/2702140413 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 33) v1 ==== 7478+0+0 (secure 0 0 0) 0x7f154409ff00 con 0x55f9e4f80830 2023-03-09T14:54:14.620 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.607+0000 7f156acb1b80 10 client.16185 fetch_fsmap finished waiting for FSMap version 33 2023-03-09T14:54:14.620 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.607+0000 7f156acb1b80 10 client.16185 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T14:54:14.620 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.607+0000 7f156acb1b80 20 client.16185 populate_metadata read hostname 'smithi038' 2023-03-09T14:54:14.620 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.607+0000 7f156acb1b80 1 --2- 172.21.15.38:0/2702140413 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55f9e4f94d20 0x55f9e4f97130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:54:14.621 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.607+0000 7f156acb1b80 4 client.16185 mds_command: new command op to 24530 tid=0[{"prefix": "get_command_descriptions"}] 2023-03-09T14:54:14.621 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.607+0000 7f156acb1b80 1 -- 172.21.15.38:0/2702140413 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f9e4d50f60 con 0x55f9e4f94d20 2023-03-09T14:54:14.621 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.607+0000 7f155c8c4700 1 --2- 172.21.15.38:0/2702140413 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55f9e4f94d20 0x55f9e4f97130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:54:14.621 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.608+0000 7f1537fff700 10 client.16185 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:54:14.621 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.608+0000 7f155c8c4700 1 --2- 172.21.15.38:0/2702140413 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55f9e4f94d20 0x55f9e4f97130 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=86ad6780aa190255 server_cookie=24b602e040161db4 in_seq=0 out_seq=0 2023-03-09T14:54:14.621 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.609+0000 7f1537fff700 1 -- 172.21.15.38:0/2702140413 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18278 (crc 0 0 0) 0x55f9e4d50f60 con 0x55f9e4f94d20 2023-03-09T14:54:14.621 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.609+0000 7f1537fff700 10 client.16185 handle_command_reply: tid=0 2023-03-09T14:54:14.622 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.609+0000 7f156acb1b80 1 client.16185 shutdown 2023-03-09T14:54:14.622 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.609+0000 7f156acb1b80 1 -- 172.21.15.38:0/2702140413 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9e4f80830 msgr2=0x55f9e4f84420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:54:14.622 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.609+0000 7f156acb1b80 1 --2- 172.21.15.38:0/2702140413 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9e4f80830 0x55f9e4f84420 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f1544094af0 tx=0x7f1544003dd0 comp rx=0 tx=0).stop 2023-03-09T14:54:14.622 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.609+0000 7f156acb1b80 1 -- 172.21.15.38:0/2702140413 shutdown_connections 2023-03-09T14:54:14.622 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.609+0000 7f156acb1b80 1 --2- 172.21.15.38:0/2702140413 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55f9e4f94d20 0x55f9e4f97130 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:54:14.622 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.609+0000 7f156acb1b80 1 --2- 172.21.15.38:0/2702140413 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f9e4f81140 0x55f9e4f84ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:54:14.622 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.609+0000 7f156acb1b80 1 --2- 172.21.15.38:0/2702140413 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f9e4f7afe0 0x55f9e4f871a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:54:14.622 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.609+0000 7f1537fff700 0 client.16185 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:54:14.623 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.609+0000 7f156acb1b80 1 --2- 172.21.15.38:0/2702140413 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9e4f80830 0x55f9e4f84420 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:54:14.623 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.609+0000 7f156acb1b80 1 -- 172.21.15.38:0/2702140413 >> 172.21.15.38:0/2702140413 conn(0x55f9e4f75a20 msgr2=0x55f9e4f74c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:54:14.623 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.609+0000 7f156acb1b80 1 -- 172.21.15.38:0/2702140413 shutdown_connections 2023-03-09T14:54:14.623 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.609+0000 7f156acb1b80 1 -- 172.21.15.38:0/2702140413 wait complete. 2023-03-09T14:54:14.623 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.609+0000 7f156acb1b80 20 client.16185 trim_cache size 0 max 16384 2023-03-09T14:54:14.630 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.626+0000 7f156acb1b80 1 Processor -- start 2023-03-09T14:54:14.630 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.626+0000 7f156acb1b80 1 -- start start 2023-03-09T14:54:14.630 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.626+0000 7f156acb1b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9e4f7afe0 0x55f9e4f8bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:54:14.631 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.627+0000 7f156acb1b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f9e4f80830 0x55f9e4f86d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:54:14.631 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.627+0000 7f156acb1b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f9e4f81140 0x55f9e4f7e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:54:14.631 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.627+0000 7f156acb1b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9e4f757d0 con 0x55f9e4f7afe0 2023-03-09T14:54:14.631 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.627+0000 7f156acb1b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9e4f80430 con 0x55f9e4f80830 2023-03-09T14:54:14.631 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.627+0000 7f156acb1b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9e4f745f0 con 0x55f9e4f81140 2023-03-09T14:54:14.631 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.627+0000 7f155c8c4700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9e4f7afe0 0x55f9e4f8bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:54:14.631 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.627+0000 7f155c8c4700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9e4f7afe0 0x55f9e4f8bb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:41834/0 (socket says 172.21.15.38:41834) 2023-03-09T14:54:14.632 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.627+0000 7f155c8c4700 1 -- 172.21.15.38:0/197241863 learned_addr learned my addr 172.21.15.38:0/197241863 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:54:14.632 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.627+0000 7f1557fff700 1 --2- 172.21.15.38:0/197241863 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f9e4f80830 0x55f9e4f86d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:54:14.632 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.627+0000 7f155d0c5700 1 --2- 172.21.15.38:0/197241863 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f9e4f81140 0x55f9e4f7e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:54:14.632 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.627+0000 7f155c8c4700 1 -- 172.21.15.38:0/197241863 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f9e4f81140 msgr2=0x55f9e4f7e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:54:14.632 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.627+0000 7f155c8c4700 1 --2- 172.21.15.38:0/197241863 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f9e4f81140 0x55f9e4f7e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:54:14.632 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.627+0000 7f155c8c4700 1 -- 172.21.15.38:0/197241863 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f9e4f80830 msgr2=0x55f9e4f86d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:54:14.632 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.627+0000 7f155c8c4700 1 --2- 172.21.15.38:0/197241863 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f9e4f80830 0x55f9e4f86d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:54:14.632 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.627+0000 7f155c8c4700 1 -- 172.21.15.38:0/197241863 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f9e4ce3c60 con 0x55f9e4f7afe0 2023-03-09T14:54:14.633 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.627+0000 7f1557fff700 1 --2- 172.21.15.38:0/197241863 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f9e4f80830 0x55f9e4f86d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:54:14.633 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.627+0000 7f155c8c4700 1 --2- 172.21.15.38:0/197241863 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9e4f7afe0 0x55f9e4f8bb90 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f154c011ab0 tx=0x7f154c01f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c346c16fa77623b server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:54:14.633 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.627+0000 7f15437fe700 1 -- 172.21.15.38:0/197241863 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f154c00ead0 con 0x55f9e4f7afe0 2023-03-09T14:54:14.633 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.627+0000 7f15437fe700 1 -- 172.21.15.38:0/197241863 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f154c021070 con 0x55f9e4f7afe0 2023-03-09T14:54:14.633 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.627+0000 7f15437fe700 1 -- 172.21.15.38:0/197241863 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f154c003680 con 0x55f9e4f7afe0 2023-03-09T14:54:14.636 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.628+0000 7f156acb1b80 1 -- 172.21.15.38:0/197241863 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9e4f7afe0 msgr2=0x55f9e4f8bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:54:14.636 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.628+0000 7f156acb1b80 1 --2- 172.21.15.38:0/197241863 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9e4f7afe0 0x55f9e4f8bb90 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f154c011ab0 tx=0x7f154c01f040 comp rx=0 tx=0).stop 2023-03-09T14:54:14.637 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.628+0000 7f156acb1b80 1 -- 172.21.15.38:0/197241863 shutdown_connections 2023-03-09T14:54:14.637 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.628+0000 7f156acb1b80 1 --2- 172.21.15.38:0/197241863 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9e4f7afe0 0x55f9e4f8bb90 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:54:14.637 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.628+0000 7f156acb1b80 1 --2- 172.21.15.38:0/197241863 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f9e4f81140 0x55f9e4f7e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:54:14.637 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.628+0000 7f156acb1b80 1 --2- 172.21.15.38:0/197241863 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f9e4f80830 0x55f9e4f86d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:54:14.637 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.628+0000 7f156acb1b80 1 -- 172.21.15.38:0/197241863 >> 172.21.15.38:0/197241863 conn(0x55f9e4f75a20 msgr2=0x55f9e4f75df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:54:14.637 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.628+0000 7f156acb1b80 1 -- 172.21.15.38:0/197241863 shutdown_connections 2023-03-09T14:54:14.637 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.628+0000 7f156acb1b80 1 -- 172.21.15.38:0/197241863 wait complete. 2023-03-09T14:54:14.638 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.628+0000 7f156acb1b80 1 -- start start 2023-03-09T14:54:14.638 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.628+0000 7f156acb1b80 1 Processor -- start 2023-03-09T14:54:14.638 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.629+0000 7f156acb1b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f9e4f7afe0 0x55f9e4f8a480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:54:14.638 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.629+0000 7f156acb1b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f9e4f80830 0x55f9e4f8a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:54:14.638 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.629+0000 7f156acb1b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9e4f81140 0x55f9e4f8af00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:54:14.638 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.629+0000 7f156acb1b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9e4fe8d50 con 0x55f9e4f81140 2023-03-09T14:54:14.638 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.629+0000 7f156acb1b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9e4fe8ec0 con 0x55f9e4f7afe0 2023-03-09T14:54:14.639 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.629+0000 7f156acb1b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9e4fe9080 con 0x55f9e4f80830 2023-03-09T14:54:14.639 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.629+0000 7f156acb1b80 10 client.0 _ll_register_callbacks cb 0x55f9e4e72520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:54:14.639 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.629+0000 7f155d0c5700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9e4f81140 0x55f9e4f8af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:54:14.639 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.629+0000 7f155d0c5700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9e4f81140 0x55f9e4f8af00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:41844/0 (socket says 172.21.15.38:41844) 2023-03-09T14:54:14.639 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.629+0000 7f155d0c5700 1 -- 172.21.15.38:0/3526201881 learned_addr learned my addr 172.21.15.38:0/3526201881 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:54:14.639 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.629+0000 7f155c8c4700 1 --2- 172.21.15.38:0/3526201881 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f9e4f7afe0 0x55f9e4f8a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:54:14.639 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.629+0000 7f1557fff700 1 --2- 172.21.15.38:0/3526201881 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f9e4f80830 0x55f9e4f8a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:54:14.639 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.629+0000 7f155d0c5700 1 -- 172.21.15.38:0/3526201881 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f9e4f80830 msgr2=0x55f9e4f8a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:54:14.640 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.629+0000 7f155d0c5700 1 --2- 172.21.15.38:0/3526201881 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f9e4f80830 0x55f9e4f8a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:54:14.640 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.629+0000 7f155d0c5700 1 -- 172.21.15.38:0/3526201881 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f9e4f7afe0 msgr2=0x55f9e4f8a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:54:14.640 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.629+0000 7f155d0c5700 1 --2- 172.21.15.38:0/3526201881 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f9e4f7afe0 0x55f9e4f8a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:54:14.640 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.629+0000 7f155d0c5700 1 -- 172.21.15.38:0/3526201881 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f154c02a040 con 0x55f9e4f81140 2023-03-09T14:54:14.640 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.629+0000 7f155d0c5700 1 --2- 172.21.15.38:0/3526201881 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9e4f81140 0x55f9e4f8af00 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f155000b0a0 tx=0x7f1550007710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d3d13bc54c1dbb1 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:54:14.640 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.629+0000 7f1537fff700 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2023-03-09T14:54:14.640 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.629+0000 7f1537fff700 1 -- 172.21.15.38:0/3526201881 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f155000e040 con 0x55f9e4f81140 2023-03-09T14:54:14.640 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.629+0000 7f1537fff700 1 -- 172.21.15.38:0/3526201881 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(224..224 src has 1..224) v4 ==== 9771+0+0 (secure 0 0 0) 0x7f1550003c00 con 0x55f9e4f81140 2023-03-09T14:54:14.641 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.629+0000 7f156acb1b80 1 -- 172.21.15.38:0/3526201881 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f9e4e8a1b0 con 0x55f9e4f81140 2023-03-09T14:54:14.641 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.630+0000 7f1537fff700 1 -- 172.21.15.38:0/3526201881 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1550007930 con 0x55f9e4f81140 2023-03-09T14:54:14.641 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.630+0000 7f156acb1b80 10 client.16191 fetch_fsmap learned FSMap version 33 2023-03-09T14:54:14.641 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.630+0000 7f156acb1b80 1 -- 172.21.15.38:0/3526201881 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) v3 -- 0x55f9e4d21250 con 0x55f9e4f81140 2023-03-09T14:54:14.641 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.630+0000 7f1537fff700 1 -- 172.21.15.38:0/3526201881 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55f9e4e8a1b0 con 0x55f9e4f81140 2023-03-09T14:54:14.641 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.630+0000 7f1537fff700 1 -- 172.21.15.38:0/3526201881 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 33) v1 ==== 7478+0+0 (secure 0 0 0) 0x7f155000b0d0 con 0x55f9e4f81140 2023-03-09T14:54:14.641 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.630+0000 7f156acb1b80 10 client.16191 fetch_fsmap finished waiting for FSMap version 33 2023-03-09T14:54:14.642 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.630+0000 7f156acb1b80 10 client.16191 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T14:54:14.642 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.630+0000 7f156acb1b80 20 client.16191 populate_metadata read hostname 'smithi038' 2023-03-09T14:54:14.642 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.630+0000 7f156acb1b80 1 --2- 172.21.15.38:0/3526201881 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55f9e4fea820 0x55f9e4fecc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:54:14.642 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.630+0000 7f156acb1b80 4 client.16191 mds_command: new command op to 24530 tid=0[{"prefix": "scrub status"}] 2023-03-09T14:54:14.642 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.630+0000 7f156acb1b80 1 -- 172.21.15.38:0/3526201881 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55f9e4e741c0 con 0x55f9e4fea820 2023-03-09T14:54:14.642 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.630+0000 7f155c8c4700 1 --2- 172.21.15.38:0/3526201881 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55f9e4fea820 0x55f9e4fecc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:54:14.642 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.631+0000 7f1537fff700 10 client.16191 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:54:14.642 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.631+0000 7f155c8c4700 1 --2- 172.21.15.38:0/3526201881 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55f9e4fea820 0x55f9e4fecc30 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dba6faadf09182a server_cookie=32cbccab78a5a37a in_seq=0 out_seq=0 2023-03-09T14:54:14.643 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.631+0000 7f1537fff700 1 -- 172.21.15.38:0/3526201881 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+265 (crc 0 0 0) 0x55f9e4e741c0 con 0x55f9e4fea820 2023-03-09T14:54:14.643 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.631+0000 7f1537fff700 10 client.16191 handle_command_reply: tid=0 2023-03-09T14:54:14.643 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.631+0000 7f156acb1b80 1 client.16191 shutdown 2023-03-09T14:54:14.643 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.631+0000 7f156acb1b80 1 -- 172.21.15.38:0/3526201881 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9e4f81140 msgr2=0x55f9e4f8af00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:54:14.643 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.631+0000 7f156acb1b80 1 --2- 172.21.15.38:0/3526201881 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9e4f81140 0x55f9e4f8af00 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f155000b0a0 tx=0x7f1550007710 comp rx=0 tx=0).stop 2023-03-09T14:54:14.643 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.631+0000 7f156acb1b80 1 -- 172.21.15.38:0/3526201881 shutdown_connections 2023-03-09T14:54:14.643 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.631+0000 7f156acb1b80 1 --2- 172.21.15.38:0/3526201881 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55f9e4fea820 0x55f9e4fecc30 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:54:14.644 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.631+0000 7f156acb1b80 1 --2- 172.21.15.38:0/3526201881 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55f9e4f81140 0x55f9e4f8af00 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:54:14.644 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.631+0000 7f1537fff700 0 client.16191 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:54:14.644 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.631+0000 7f156acb1b80 1 --2- 172.21.15.38:0/3526201881 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55f9e4f7afe0 0x55f9e4f8a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:54:14.644 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.631+0000 7f156acb1b80 1 --2- 172.21.15.38:0/3526201881 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55f9e4f80830 0x55f9e4f8a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:54:14.644 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.631+0000 7f156acb1b80 1 -- 172.21.15.38:0/3526201881 >> 172.21.15.38:0/3526201881 conn(0x55f9e4f75a20 msgr2=0x55f9e4f7e5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:54:14.644 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.631+0000 7f156acb1b80 1 -- 172.21.15.38:0/3526201881 shutdown_connections 2023-03-09T14:54:14.644 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.631+0000 7f156acb1b80 1 -- 172.21.15.38:0/3526201881 wait complete. 2023-03-09T14:54:14.644 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.631+0000 7f156acb1b80 20 client.16191 trim_cache size 0 max 16384 2023-03-09T14:54:14.645 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2023-03-09T14:54:14.645 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "scrub active (3 inodes in the stack)", 2023-03-09T14:54:14.645 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": { 2023-03-09T14:54:14.645 INFO:teuthology.orchestra.run.smithi038.stdout: "a14659f4-d7ea-4c5a-b15b-6138af3f909f": { 2023-03-09T14:54:14.645 INFO:teuthology.orchestra.run.smithi038.stdout: "path": "/", 2023-03-09T14:54:14.645 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "a14659f4-d7ea-4c5a-b15b-6138af3f909f", 2023-03-09T14:54:14.646 INFO:teuthology.orchestra.run.smithi038.stdout: "options": "recursive,force" 2023-03-09T14:54:14.646 INFO:teuthology.orchestra.run.smithi038.stdout: } 2023-03-09T14:54:14.646 INFO:teuthology.orchestra.run.smithi038.stdout: } 2023-03-09T14:54:14.646 INFO:teuthology.orchestra.run.smithi038.stdout:} 2023-03-09T14:54:14.647 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.632+0000 7f155eb28700 1 -- 172.21.15.38:0/1504508299 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f154809f4a0 msgr2=0x7f15480a1950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:54:14.647 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.632+0000 7f155eb28700 1 --2- 172.21.15.38:0/1504508299 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f154809f4a0 0x7f15480a1950 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f154c009c60 tx=0x7f154c009260 comp rx=0 tx=0).stop 2023-03-09T14:54:14.647 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.632+0000 7f155eb28700 1 -- 172.21.15.38:0/1504508299 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15580ff8d0 msgr2=0x7f1558100a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:54:14.648 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.632+0000 7f155eb28700 1 --2- 172.21.15.38:0/1504508299 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15580ff8d0 0x7f1558100a20 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f154400e970 tx=0x7f154400ec80 comp rx=0 tx=0).stop 2023-03-09T14:54:14.648 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.632+0000 7f155eb28700 1 -- 172.21.15.38:0/1504508299 shutdown_connections 2023-03-09T14:54:14.648 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.632+0000 7f155eb28700 1 --2- 172.21.15.38:0/1504508299 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f154809f4a0 0x7f15480a1950 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:54:14.648 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.632+0000 7f155eb28700 1 --2- 172.21.15.38:0/1504508299 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f155810a720 0x7f15581a4560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:54:14.648 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.632+0000 7f155eb28700 1 --2- 172.21.15.38:0/1504508299 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f15580675f0 0x7f15581004e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:54:14.648 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.632+0000 7f155eb28700 1 --2- 172.21.15.38:0/1504508299 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f15580ff8d0 0x7f1558100a20 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:54:14.648 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.632+0000 7f155eb28700 1 -- 172.21.15.38:0/1504508299 >> 172.21.15.38:0/1504508299 conn(0x7f15580748f0 msgr2=0x7f1558102620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:54:14.648 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.632+0000 7f155eb28700 1 -- 172.21.15.38:0/1504508299 shutdown_connections 2023-03-09T14:54:14.649 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:14.632+0000 7f155eb28700 1 -- 172.21.15.38:0/1504508299 wait complete. 2023-03-09T14:54:14.649 INFO:tasks.cephfs.filesystem:scrub status for tag:a14659f4-d7ea-4c5a-b15b-6138af3f909f - {'path': '/', 'tag': 'a14659f4-d7ea-4c5a-b15b-6138af3f909f', 'options': 'recursive,force'} 2023-03-09T14:54:15.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:54:14 smithi122 ceph-mon[93869]: osdmap e224: 12 total, 12 up, 12 in 2023-03-09T14:54:15.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:54:14 smithi181 ceph-mon[94189]: osdmap e224: 12 total, 12 up, 12 in 2023-03-09T14:54:15.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:54:14 smithi038 ceph-mon[92422]: osdmap e224: 12 total, 12 up, 12 in 2023-03-09T14:54:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:54:15 smithi122 ceph-mon[93869]: pgmap v377: 129 pgs: 1 active+clean+snaptrim, 7 active+recovery_wait+undersized+degraded+remapped, 121 active+clean; 18 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 265 KiB/s wr, 34 op/s; 886/50059 objects degraded (1.770%); 154 MiB/s, 39 objects/s recovering 2023-03-09T14:54:16.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:54:15 smithi122 ceph-mon[93869]: osdmap e225: 12 total, 12 up, 12 in 2023-03-09T14:54:16.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:54:15 smithi181 ceph-mon[94189]: pgmap v377: 129 pgs: 1 active+clean+snaptrim, 7 active+recovery_wait+undersized+degraded+remapped, 121 active+clean; 18 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 265 KiB/s wr, 34 op/s; 886/50059 objects degraded (1.770%); 154 MiB/s, 39 objects/s recovering 2023-03-09T14:54:16.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:54:15 smithi181 ceph-mon[94189]: osdmap e225: 12 total, 12 up, 12 in 2023-03-09T14:54:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:54:15 smithi038 ceph-mon[92422]: pgmap v377: 129 pgs: 1 active+clean+snaptrim, 7 active+recovery_wait+undersized+degraded+remapped, 121 active+clean; 18 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 265 KiB/s wr, 34 op/s; 886/50059 objects degraded (1.770%); 154 MiB/s, 39 objects/s recovering 2023-03-09T14:54:16.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:54:15 smithi038 ceph-mon[92422]: osdmap e225: 12 total, 12 up, 12 in 2023-03-09T14:54:17.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:54:16 smithi122 ceph-mon[93869]: osdmap e226: 12 total, 12 up, 12 in 2023-03-09T14:54:17.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:54:16 smithi181 ceph-mon[94189]: osdmap e226: 12 total, 12 up, 12 in 2023-03-09T14:54:17.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:54:16 smithi038 ceph-mon[92422]: osdmap e226: 12 total, 12 up, 12 in 2023-03-09T14:54:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:54:17 smithi122 ceph-mon[93869]: pgmap v380: 129 pgs: 1 active+clean+snaptrim, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 122 active+clean; 18 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 149 KiB/s wr, 22 op/s; 595/49979 objects degraded (1.191%); 478 MiB/s, 120 objects/s recovering 2023-03-09T14:54:18.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:54:17 smithi122 ceph-mon[93869]: osdmap e227: 12 total, 12 up, 12 in 2023-03-09T14:54:18.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:54:17 smithi122 ceph-mon[93869]: Health check update: Degraded data redundancy: 595/49979 objects degraded (1.191%), 5 pgs degraded (PG_DEGRADED) 2023-03-09T14:54:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:54:17 smithi181 ceph-mon[94189]: pgmap v380: 129 pgs: 1 active+clean+snaptrim, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 122 active+clean; 18 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 149 KiB/s wr, 22 op/s; 595/49979 objects degraded (1.191%); 478 MiB/s, 120 objects/s recovering 2023-03-09T14:54:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:54:17 smithi181 ceph-mon[94189]: osdmap e227: 12 total, 12 up, 12 in 2023-03-09T14:54:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:54:17 smithi181 ceph-mon[94189]: Health check update: Degraded data redundancy: 595/49979 objects degraded (1.191%), 5 pgs degraded (PG_DEGRADED) 2023-03-09T14:54:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:54:17 smithi038 ceph-mon[92422]: pgmap v380: 129 pgs: 1 active+clean+snaptrim, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 122 active+clean; 18 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 149 KiB/s wr, 22 op/s; 595/49979 objects degraded (1.191%); 478 MiB/s, 120 objects/s recovering 2023-03-09T14:54:18.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:54:17 smithi038 ceph-mon[92422]: osdmap e227: 12 total, 12 up, 12 in 2023-03-09T14:54:18.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:54:17 smithi038 ceph-mon[92422]: Health check update: Degraded data redundancy: 595/49979 objects degraded (1.191%), 5 pgs degraded (PG_DEGRADED) 2023-03-09T14:54:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 230.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T14:54:19.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:54:18 smithi122 ceph-mon[93869]: osdmap e228: 12 total, 12 up, 12 in 2023-03-09T14:54:19.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:54:18 smithi181 ceph-mon[94189]: osdmap e228: 12 total, 12 up, 12 in 2023-03-09T14:54:19.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:54:18 smithi038 ceph-mon[92422]: osdmap e228: 12 total, 12 up, 12 in 2023-03-09T14:54:20.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:54:19 smithi122 ceph-mon[93869]: pgmap v383: 129 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 123 active+clean; 17 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 118 KiB/s wr, 19 op/s; 595/49907 objects degraded (1.192%); 291 MiB/s, 72 objects/s recovering 2023-03-09T14:54:20.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:54:19 smithi181 ceph-mon[94189]: pgmap v383: 129 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 123 active+clean; 17 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 118 KiB/s wr, 19 op/s; 595/49907 objects degraded (1.192%); 291 MiB/s, 72 objects/s recovering 2023-03-09T14:54:20.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:54:19 smithi038 ceph-mon[92422]: pgmap v383: 129 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 123 active+clean; 17 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 118 KiB/s wr, 19 op/s; 595/49907 objects degraded (1.192%); 291 MiB/s, 72 objects/s recovering 2023-03-09T14:54:21.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:54:20 smithi122 ceph-mon[93869]: osdmap e229: 12 total, 12 up, 12 in 2023-03-09T14:54:21.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:54:20 smithi181 ceph-mon[94189]: osdmap e229: 12 total, 12 up, 12 in 2023-03-09T14:54:21.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:54:20 smithi038 ceph-mon[92422]: osdmap e229: 12 total, 12 up, 12 in 2023-03-09T14:54:22.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:54:21 smithi122 ceph-mon[93869]: pgmap v385: 129 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 124 active+clean; 17 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 1.7 MiB/s wr, 169 op/s; 595/49843 objects degraded (1.194%); 138 MiB/s, 34 objects/s recovering 2023-03-09T14:54:22.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:54:21 smithi122 ceph-mon[93869]: osdmap e230: 12 total, 12 up, 12 in 2023-03-09T14:54:22.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:54:21 smithi181 ceph-mon[94189]: pgmap v385: 129 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 124 active+clean; 17 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 1.7 MiB/s wr, 169 op/s; 595/49843 objects degraded (1.194%); 138 MiB/s, 34 objects/s recovering 2023-03-09T14:54:22.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:54:21 smithi181 ceph-mon[94189]: osdmap e230: 12 total, 12 up, 12 in 2023-03-09T14:54:22.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:54:21 smithi038 ceph-mon[92422]: pgmap v385: 129 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 124 active+clean; 17 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 1.7 MiB/s wr, 169 op/s; 595/49843 objects degraded (1.194%); 138 MiB/s, 34 objects/s recovering 2023-03-09T14:54:22.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:54:21 smithi038 ceph-mon[92422]: osdmap e230: 12 total, 12 up, 12 in 2023-03-09T14:54:23.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:54:22 smithi122 ceph-mon[93869]: osdmap e231: 12 total, 12 up, 12 in 2023-03-09T14:54:23.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:54:22 smithi122 ceph-mon[93869]: Health check update: Degraded data redundancy: 595/49843 objects degraded (1.194%), 5 pgs degraded (PG_DEGRADED) 2023-03-09T14:54:23.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:54:22 smithi181 ceph-mon[94189]: osdmap e231: 12 total, 12 up, 12 in 2023-03-09T14:54:23.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:54:22 smithi181 ceph-mon[94189]: Health check update: Degraded data redundancy: 595/49843 objects degraded (1.194%), 5 pgs degraded (PG_DEGRADED) 2023-03-09T14:54:23.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:54:22 smithi038 ceph-mon[92422]: osdmap e231: 12 total, 12 up, 12 in 2023-03-09T14:54:23.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:54:22 smithi038 ceph-mon[92422]: Health check update: Degraded data redundancy: 595/49843 objects degraded (1.194%), 5 pgs degraded (PG_DEGRADED) 2023-03-09T14:54:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 235.0 s, 286.3 tps, lat 1571.520 ms stddev 4651.219 2023-03-09T14:54:24.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:54:23 smithi122 ceph-mon[93869]: pgmap v388: 129 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 125 active+clean; 17 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 666 KiB/s rd, 2.6 MiB/s wr, 346 op/s; 549/49805 objects degraded (1.102%); 37 MiB/s, 9 objects/s recovering 2023-03-09T14:54:24.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:54:23 smithi122 ceph-mon[93869]: osdmap e232: 12 total, 12 up, 12 in 2023-03-09T14:54:24.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:54:23 smithi181 ceph-mon[94189]: pgmap v388: 129 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 125 active+clean; 17 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 666 KiB/s rd, 2.6 MiB/s wr, 346 op/s; 549/49805 objects degraded (1.102%); 37 MiB/s, 9 objects/s recovering 2023-03-09T14:54:24.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:54:23 smithi181 ceph-mon[94189]: osdmap e232: 12 total, 12 up, 12 in 2023-03-09T14:54:24.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:54:23 smithi038 ceph-mon[92422]: pgmap v388: 129 pgs: 4 active+recovery_wait+undersized+degraded+remapped, 125 active+clean; 17 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 666 KiB/s rd, 2.6 MiB/s wr, 346 op/s; 549/49805 objects degraded (1.102%); 37 MiB/s, 9 objects/s recovering 2023-03-09T14:54:24.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:54:23 smithi038 ceph-mon[92422]: osdmap e232: 12 total, 12 up, 12 in 2023-03-09T14:54:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:54:25 smithi122 ceph-mon[93869]: pgmap v390: 129 pgs: 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 126 active+clean; 17 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.5 MiB/s wr, 714 op/s; 234/50044 objects degraded (0.468%); 304 MiB/s, 76 objects/s recovering 2023-03-09T14:54:26.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:54:25 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T14:54:26.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:54:25 smithi122 ceph-mon[93869]: scrub summary: active paths [/] 2023-03-09T14:54:26.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:54:25 smithi122 ceph-mon[93869]: scrub summary: idle+waiting paths [/] 2023-03-09T14:54:26.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:54:25 smithi181 ceph-mon[94189]: pgmap v390: 129 pgs: 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 126 active+clean; 17 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.5 MiB/s wr, 714 op/s; 234/50044 objects degraded (0.468%); 304 MiB/s, 76 objects/s recovering 2023-03-09T14:54:26.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:54:25 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T14:54:26.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:54:25 smithi181 ceph-mon[94189]: scrub summary: active paths [/] 2023-03-09T14:54:26.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:54:25 smithi181 ceph-mon[94189]: scrub summary: idle+waiting paths [/] 2023-03-09T14:54:26.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:54:25 smithi038 ceph-mon[92422]: pgmap v390: 129 pgs: 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 126 active+clean; 17 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 4.5 MiB/s wr, 714 op/s; 234/50044 objects degraded (0.468%); 304 MiB/s, 76 objects/s recovering 2023-03-09T14:54:26.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:54:25 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T14:54:26.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:54:25 smithi038 ceph-mon[92422]: scrub summary: active paths [/] 2023-03-09T14:54:26.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:54:25 smithi038 ceph-mon[92422]: scrub summary: idle+waiting paths [/] 2023-03-09T14:54:27.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:54:26 smithi122 ceph-mon[93869]: osdmap e233: 12 total, 12 up, 12 in 2023-03-09T14:54:27.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:54:26 smithi181 ceph-mon[94189]: osdmap e233: 12 total, 12 up, 12 in 2023-03-09T14:54:27.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:54:26 smithi038 ceph-mon[92422]: osdmap e233: 12 total, 12 up, 12 in 2023-03-09T14:54:28.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:54:27 smithi122 ceph-mon[93869]: pgmap v392: 129 pgs: 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 126 active+clean; 17 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 8.0 MiB/s wr, 1.03k op/s; 234/51076 objects degraded (0.458%); 220 MiB/s, 55 objects/s recovering 2023-03-09T14:54:28.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:54:27 smithi122 ceph-mon[93869]: osdmap e234: 12 total, 12 up, 12 in 2023-03-09T14:54:28.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:54:27 smithi122 ceph-mon[93869]: Health check update: Degraded data redundancy: 234/51076 objects degraded (0.458%), 2 pgs degraded (PG_DEGRADED) 2023-03-09T14:54:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:54:27 smithi181 ceph-mon[94189]: pgmap v392: 129 pgs: 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 126 active+clean; 17 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 8.0 MiB/s wr, 1.03k op/s; 234/51076 objects degraded (0.458%); 220 MiB/s, 55 objects/s recovering 2023-03-09T14:54:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:54:27 smithi181 ceph-mon[94189]: osdmap e234: 12 total, 12 up, 12 in 2023-03-09T14:54:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:54:27 smithi181 ceph-mon[94189]: Health check update: Degraded data redundancy: 234/51076 objects degraded (0.458%), 2 pgs degraded (PG_DEGRADED) 2023-03-09T14:54:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:54:27 smithi038 ceph-mon[92422]: pgmap v392: 129 pgs: 1 peering, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering+undersized+degraded+remapped, 126 active+clean; 17 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 8.0 MiB/s wr, 1.03k op/s; 234/51076 objects degraded (0.458%); 220 MiB/s, 55 objects/s recovering 2023-03-09T14:54:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:54:27 smithi038 ceph-mon[92422]: osdmap e234: 12 total, 12 up, 12 in 2023-03-09T14:54:28.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:54:27 smithi038 ceph-mon[92422]: Health check update: Degraded data redundancy: 234/51076 objects degraded (0.458%), 2 pgs degraded (PG_DEGRADED) 2023-03-09T14:54:28.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 240.0 s, 305.4 tps, lat 321.494 ms stddev 351.095 2023-03-09T14:54:30.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:54:29 smithi122 ceph-mon[93869]: pgmap v394: 129 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 128 active+clean; 17 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 12 MiB/s wr, 1.37k op/s; 144/51892 objects degraded (0.277%); 278 MiB/s, 70 objects/s recovering 2023-03-09T14:54:30.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:54:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:54:30.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:54:29 smithi181 ceph-mon[94189]: pgmap v394: 129 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 128 active+clean; 17 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 12 MiB/s wr, 1.37k op/s; 144/51892 objects degraded (0.277%); 278 MiB/s, 70 objects/s recovering 2023-03-09T14:54:30.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:54:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:54:30.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:54:29 smithi038 ceph-mon[92422]: pgmap v394: 129 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 128 active+clean; 17 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 12 MiB/s wr, 1.37k op/s; 144/51892 objects degraded (0.277%); 278 MiB/s, 70 objects/s recovering 2023-03-09T14:54:30.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:54:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:54:32.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:54:31 smithi122 ceph-mon[93869]: pgmap v395: 129 pgs: 1 active+recovering+undersized+degraded+remapped, 128 active+clean; 17 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 11 MiB/s wr, 1.35k op/s; 25/52968 objects degraded (0.047%); 183 MiB/s, 46 objects/s recovering 2023-03-09T14:54:32.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:54:31 smithi122 ceph-mon[93869]: osdmap e235: 12 total, 12 up, 12 in 2023-03-09T14:54:32.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:54:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:54:32.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:54:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:54:32.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:54:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:54:32.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:54:31 smithi181 ceph-mon[94189]: pgmap v395: 129 pgs: 1 active+recovering+undersized+degraded+remapped, 128 active+clean; 17 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 11 MiB/s wr, 1.35k op/s; 25/52968 objects degraded (0.047%); 183 MiB/s, 46 objects/s recovering 2023-03-09T14:54:32.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:54:31 smithi181 ceph-mon[94189]: osdmap e235: 12 total, 12 up, 12 in 2023-03-09T14:54:32.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:54:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:54:32.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:54:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:54:32.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:54:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:54:32.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:54:31 smithi038 ceph-mon[92422]: pgmap v395: 129 pgs: 1 active+recovering+undersized+degraded+remapped, 128 active+clean; 17 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 11 MiB/s wr, 1.35k op/s; 25/52968 objects degraded (0.047%); 183 MiB/s, 46 objects/s recovering 2023-03-09T14:54:32.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:54:31 smithi038 ceph-mon[92422]: osdmap e235: 12 total, 12 up, 12 in 2023-03-09T14:54:32.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:54:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:54:32.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:54:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:54:32.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:54:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:54:33.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:54:32 smithi122 ceph-mon[93869]: osdmap e236: 12 total, 12 up, 12 in 2023-03-09T14:54:33.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:54:32 smithi122 ceph-mon[93869]: Health check update: Degraded data redundancy: 25/52968 objects degraded (0.047%), 1 pg degraded (PG_DEGRADED) 2023-03-09T14:54:33.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:54:32 smithi181 ceph-mon[94189]: osdmap e236: 12 total, 12 up, 12 in 2023-03-09T14:54:33.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:54:32 smithi181 ceph-mon[94189]: Health check update: Degraded data redundancy: 25/52968 objects degraded (0.047%), 1 pg degraded (PG_DEGRADED) 2023-03-09T14:54:33.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:54:32 smithi038 ceph-mon[92422]: osdmap e236: 12 total, 12 up, 12 in 2023-03-09T14:54:33.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:54:32 smithi038 ceph-mon[92422]: Health check update: Degraded data redundancy: 25/52968 objects degraded (0.047%), 1 pg degraded (PG_DEGRADED) 2023-03-09T14:54:33.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 245.0 s, 418.8 tps, lat 244.487 ms stddev 239.766 2023-03-09T14:54:34.258 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:54:33 smithi038 ceph-mon[92422]: pgmap v398: 129 pgs: 1 active+recovering+undersized+degraded+remapped, 128 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 11 MiB/s wr, 1.27k op/s; 25/53584 objects degraded (0.047%); 147 MiB/s, 37 objects/s recovering 2023-03-09T14:54:34.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:54:33 smithi122 ceph-mon[93869]: pgmap v398: 129 pgs: 1 active+recovering+undersized+degraded+remapped, 128 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 11 MiB/s wr, 1.27k op/s; 25/53584 objects degraded (0.047%); 147 MiB/s, 37 objects/s recovering 2023-03-09T14:54:34.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:54:33 smithi181 ceph-mon[94189]: pgmap v398: 129 pgs: 1 active+recovering+undersized+degraded+remapped, 128 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 11 MiB/s wr, 1.27k op/s; 25/53584 objects degraded (0.047%); 147 MiB/s, 37 objects/s recovering 2023-03-09T14:54:35.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:54:35 smithi122 ceph-mon[93869]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 25/53584 objects degraded (0.047%), 1 pg degraded) 2023-03-09T14:54:35.481 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:54:35 smithi122 ceph-mon[93869]: Cluster is now healthy 2023-03-09T14:54:35.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:54:35 smithi181 ceph-mon[94189]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 25/53584 objects degraded (0.047%), 1 pg degraded) 2023-03-09T14:54:35.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:54:35 smithi181 ceph-mon[94189]: Cluster is now healthy 2023-03-09T14:54:35.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:54:35 smithi038 ceph-mon[92422]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 25/53584 objects degraded (0.047%), 1 pg degraded) 2023-03-09T14:54:35.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:54:35 smithi038 ceph-mon[92422]: Cluster is now healthy 2023-03-09T14:54:36.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:54:36 smithi181 ceph-mon[94189]: pgmap v399: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 10 MiB/s wr, 1.23k op/s; 126 MiB/s, 31 objects/s recovering 2023-03-09T14:54:36.483 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:54:36 smithi122 ceph-mon[93869]: pgmap v399: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 10 MiB/s wr, 1.23k op/s; 126 MiB/s, 31 objects/s recovering 2023-03-09T14:54:36.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:54:36 smithi038 ceph-mon[92422]: pgmap v399: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 10 MiB/s wr, 1.23k op/s; 126 MiB/s, 31 objects/s recovering 2023-03-09T14:54:38.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:54:38 smithi122 ceph-mon[93869]: pgmap v400: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 15 MiB/s wr, 1.83k op/s; 76 MiB/s, 19 objects/s recovering 2023-03-09T14:54:38.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:54:38 smithi122 ceph-mon[93869]: osdmap e237: 12 total, 12 up, 12 in 2023-03-09T14:54:38.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:54:38 smithi181 ceph-mon[94189]: pgmap v400: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 15 MiB/s wr, 1.83k op/s; 76 MiB/s, 19 objects/s recovering 2023-03-09T14:54:38.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:54:38 smithi181 ceph-mon[94189]: osdmap e237: 12 total, 12 up, 12 in 2023-03-09T14:54:38.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:54:38 smithi038 ceph-mon[92422]: pgmap v400: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 15 MiB/s wr, 1.83k op/s; 76 MiB/s, 19 objects/s recovering 2023-03-09T14:54:38.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:54:38 smithi038 ceph-mon[92422]: osdmap e237: 12 total, 12 up, 12 in 2023-03-09T14:54:38.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 250.0 s, 318.8 tps, lat 310.282 ms stddev 710.014 2023-03-09T14:54:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:54:40 smithi122 ceph-mon[93869]: pgmap v402: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 19 MiB/s wr, 2.21k op/s; 13 MiB/s, 3 objects/s recovering 2023-03-09T14:54:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:54:40 smithi181 ceph-mon[94189]: pgmap v402: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 19 MiB/s wr, 2.21k op/s; 13 MiB/s, 3 objects/s recovering 2023-03-09T14:54:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:54:40 smithi038 ceph-mon[92422]: pgmap v402: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 19 MiB/s wr, 2.21k op/s; 13 MiB/s, 3 objects/s recovering 2023-03-09T14:54:42.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:54:42 smithi122 ceph-mon[93869]: pgmap v403: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 23 MiB/s wr, 2.85k op/s; 11 MiB/s, 2 objects/s recovering 2023-03-09T14:54:42.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:54:42 smithi181 ceph-mon[94189]: pgmap v403: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 23 MiB/s wr, 2.85k op/s; 11 MiB/s, 2 objects/s recovering 2023-03-09T14:54:42.503 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:54:42 smithi038 ceph-mon[92422]: pgmap v403: 129 pgs: 129 active+clean; 17 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 23 MiB/s wr, 2.85k op/s; 11 MiB/s, 2 objects/s recovering 2023-03-09T14:54:43.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 255.0 s, 547.8 tps, lat 169.651 ms stddev 443.179 2023-03-09T14:54:44.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:54:43 smithi038 ceph-mon[92422]: pgmap v404: 129 pgs: 129 active+clean; 18 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 23 MiB/s wr, 2.69k op/s; 10 MiB/s, 2 objects/s recovering 2023-03-09T14:54:44.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:54:43 smithi122 ceph-mon[93869]: pgmap v404: 129 pgs: 129 active+clean; 18 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 23 MiB/s wr, 2.69k op/s; 10 MiB/s, 2 objects/s recovering 2023-03-09T14:54:44.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:54:43 smithi181 ceph-mon[94189]: pgmap v404: 129 pgs: 129 active+clean; 18 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 23 MiB/s wr, 2.69k op/s; 10 MiB/s, 2 objects/s recovering 2023-03-09T14:54:44.650 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-03-09T14:54:44.849 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.846+0000 7fee29555700 1 -- 172.21.15.38:0/2809458984 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee240ffa60 msgr2=0x7fee24103a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:54:44.849 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.846+0000 7fee29555700 1 --2- 172.21.15.38:0/2809458984 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee240ffa60 0x7fee24103a20 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fee180099c0 tx=0x7fee18009cd0 comp rx=0 tx=0).stop 2023-03-09T14:54:44.850 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.846+0000 7fee29555700 1 -- 172.21.15.38:0/2809458984 shutdown_connections 2023-03-09T14:54:44.850 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.846+0000 7fee29555700 1 --2- 172.21.15.38:0/2809458984 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fee24104100 0x7fee2410bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:54:44.850 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.846+0000 7fee29555700 1 --2- 172.21.15.38:0/2809458984 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fee240ff0c0 0x7fee240ff490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:54:44.851 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.846+0000 7fee29555700 1 --2- 172.21.15.38:0/2809458984 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee240ffa60 0x7fee24103a20 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:54:44.851 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.846+0000 7fee29555700 1 -- 172.21.15.38:0/2809458984 >> 172.21.15.38:0/2809458984 conn(0x7fee24075c90 msgr2=0x7fee24076090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:54:44.851 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.846+0000 7fee29555700 1 -- 172.21.15.38:0/2809458984 shutdown_connections 2023-03-09T14:54:44.852 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.846+0000 7fee29555700 1 -- 172.21.15.38:0/2809458984 wait complete. 2023-03-09T14:54:44.854 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.847+0000 7fee29555700 1 Processor -- start 2023-03-09T14:54:44.855 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.848+0000 7fee29555700 1 -- start start 2023-03-09T14:54:44.855 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.848+0000 7fee29555700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fee240ff0c0 0x7fee241a12f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:54:44.856 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.848+0000 7fee29555700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee240ffa60 0x7fee241a1830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:54:44.856 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.848+0000 7fee29555700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fee24104100 0x7fee2419cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:54:44.856 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.848+0000 7fee29555700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee2419d320 con 0x7fee240ffa60 2023-03-09T14:54:44.856 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.848+0000 7fee29555700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee2419d490 con 0x7fee240ff0c0 2023-03-09T14:54:44.857 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.848+0000 7fee29555700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee2419d650 con 0x7fee24104100 2023-03-09T14:54:44.857 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.848+0000 7fee227fc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee240ffa60 0x7fee241a1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:54:44.857 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.848+0000 7fee22ffd700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fee240ff0c0 0x7fee241a12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:54:44.858 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.848+0000 7fee22ffd700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fee240ff0c0 0x7fee241a12f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.38:51050/0 (socket says 172.21.15.38:51050) 2023-03-09T14:54:44.858 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.848+0000 7fee227fc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee240ffa60 0x7fee241a1830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:41862/0 (socket says 172.21.15.38:41862) 2023-03-09T14:54:44.858 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.848+0000 7fee22ffd700 1 -- 172.21.15.38:0/998517344 learned_addr learned my addr 172.21.15.38:0/998517344 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:54:44.858 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.848+0000 7fee237fe700 1 --2- 172.21.15.38:0/998517344 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fee24104100 0x7fee2419cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:54:44.859 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.849+0000 7fee237fe700 1 -- 172.21.15.38:0/998517344 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fee240ff0c0 msgr2=0x7fee241a12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:54:44.859 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.849+0000 7fee237fe700 1 --2- 172.21.15.38:0/998517344 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fee240ff0c0 0x7fee241a12f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:54:44.859 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.849+0000 7fee237fe700 1 -- 172.21.15.38:0/998517344 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee240ffa60 msgr2=0x7fee241a1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:54:44.859 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.849+0000 7fee237fe700 1 --2- 172.21.15.38:0/998517344 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee240ffa60 0x7fee241a1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:54:44.859 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.849+0000 7fee237fe700 1 -- 172.21.15.38:0/998517344 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee180095f0 con 0x7fee24104100 2023-03-09T14:54:44.860 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.849+0000 7fee237fe700 1 --2- 172.21.15.38:0/998517344 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fee24104100 0x7fee2419cb80 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fee1c00efc0 tx=0x7fee1c00c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:54:44.860 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.849+0000 7fee0bfff700 1 -- 172.21.15.38:0/998517344 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee1c018070 con 0x7fee24104100 2023-03-09T14:54:44.860 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.850+0000 7fee0bfff700 1 -- 172.21.15.38:0/998517344 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fee1c00f040 con 0x7fee24104100 2023-03-09T14:54:44.860 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.850+0000 7fee29555700 1 -- 172.21.15.38:0/998517344 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee2410ee80 con 0x7fee24104100 2023-03-09T14:54:44.861 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.850+0000 7fee0bfff700 1 -- 172.21.15.38:0/998517344 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee1c00cc80 con 0x7fee24104100 2023-03-09T14:54:44.861 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.850+0000 7fee29555700 1 -- 172.21.15.38:0/998517344 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee2410ecf0 con 0x7fee24104100 2023-03-09T14:54:44.861 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.853+0000 7fee0bfff700 1 -- 172.21.15.38:0/998517344 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 25) v1 ==== 133295+0+0 (secure 0 0 0) 0x7fee1c003a00 con 0x7fee24104100 2023-03-09T14:54:44.865 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.853+0000 7fee0bfff700 1 --2- 172.21.15.38:0/998517344 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fee0c09f520 0x7fee0c0a19d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:54:44.865 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.853+0000 7fee0bfff700 1 -- 172.21.15.38:0/998517344 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 9549+0+0 (secure 0 0 0) 0x7fee1c0c9f70 con 0x7fee24104100 2023-03-09T14:54:44.865 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.853+0000 7fee22ffd700 1 --2- 172.21.15.38:0/998517344 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fee0c09f520 0x7fee0c0a19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:54:44.866 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.854+0000 7fee356deb80 1 Processor -- start 2023-03-09T14:54:44.866 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.854+0000 7fee356deb80 1 -- start start 2023-03-09T14:54:44.866 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.854+0000 7fee22ffd700 1 --2- 172.21.15.38:0/998517344 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fee0c09f520 0x7fee0c0a19d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fee2410bc30 tx=0x7fee14009260 comp rx=0 tx=0).ready entity=mgr.15558 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:54:44.866 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.854+0000 7fee356deb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556d8cd2d840 0x556d8cd2dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:54:44.866 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.854+0000 7fee356deb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556d8cd2e150 0x556d8cd27ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:54:44.866 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.854+0000 7fee356deb80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556d8cd27ff0 0x556d8cd28420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:54:44.866 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.854+0000 7fee356deb80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d8cc00da0 con 0x556d8cd27ff0 2023-03-09T14:54:44.867 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.854+0000 7fee356deb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d8cc6f780 con 0x556d8cd2e150 2023-03-09T14:54:44.867 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.854+0000 7fee356deb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d8cc68cb0 con 0x556d8cd2d840 2023-03-09T14:54:44.867 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.854+0000 7fee22ffd700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556d8cd2d840 0x556d8cd2dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:54:44.867 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.854+0000 7fee227fc700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556d8cd2e150 0x556d8cd27ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:54:44.867 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.854+0000 7fee22ffd700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556d8cd2d840 0x556d8cd2dc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.38:49566/0 (socket says 172.21.15.38:49566) 2023-03-09T14:54:44.867 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.854+0000 7fee22ffd700 1 -- 172.21.15.38:0/2491088502 learned_addr learned my addr 172.21.15.38:0/2491088502 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:54:44.867 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.855+0000 7fee237fe700 1 --2- 172.21.15.38:0/2491088502 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556d8cd27ff0 0x556d8cd28420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:54:44.868 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.855+0000 7fee22ffd700 1 -- 172.21.15.38:0/2491088502 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556d8cd2e150 msgr2=0x556d8cd27ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:54:44.868 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.855+0000 7fee22ffd700 1 --2- 172.21.15.38:0/2491088502 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556d8cd2e150 0x556d8cd27ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:54:44.868 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.855+0000 7fee22ffd700 1 -- 172.21.15.38:0/2491088502 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556d8cd27ff0 msgr2=0x556d8cd28420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:54:44.868 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.855+0000 7fee22ffd700 1 --2- 172.21.15.38:0/2491088502 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556d8cd27ff0 0x556d8cd28420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:54:44.868 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.855+0000 7fee22ffd700 1 -- 172.21.15.38:0/2491088502 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556d8cc2f190 con 0x556d8cd2d840 2023-03-09T14:54:44.868 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.855+0000 7fee227fc700 1 --2- 172.21.15.38:0/2491088502 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556d8cd2e150 0x556d8cd27ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:54:44.868 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.855+0000 7fee22ffd700 1 --2- 172.21.15.38:0/2491088502 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556d8cd2d840 0x556d8cd2dc10 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fee1403e540 tx=0x7fee14014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d8f63066b1f221d0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:54:44.869 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.855+0000 7fee097fa700 1 -- 172.21.15.38:0/2491088502 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee14004830 con 0x556d8cd2d840 2023-03-09T14:54:44.869 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.855+0000 7fee097fa700 1 -- 172.21.15.38:0/2491088502 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fee14043070 con 0x556d8cd2d840 2023-03-09T14:54:44.869 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.855+0000 7fee097fa700 1 -- 172.21.15.38:0/2491088502 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee1403f680 con 0x556d8cd2d840 2023-03-09T14:54:44.869 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.856+0000 7fee356deb80 1 -- 172.21.15.38:0/2491088502 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556d8cd2d840 msgr2=0x556d8cd2dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:54:44.869 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.856+0000 7fee356deb80 1 --2- 172.21.15.38:0/2491088502 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556d8cd2d840 0x556d8cd2dc10 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fee1403e540 tx=0x7fee14014000 comp rx=0 tx=0).stop 2023-03-09T14:54:44.870 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.856+0000 7fee356deb80 1 -- 172.21.15.38:0/2491088502 shutdown_connections 2023-03-09T14:54:44.870 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.856+0000 7fee356deb80 1 --2- 172.21.15.38:0/2491088502 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556d8cd27ff0 0x556d8cd28420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:54:44.870 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.856+0000 7fee356deb80 1 --2- 172.21.15.38:0/2491088502 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556d8cd2d840 0x556d8cd2dc10 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:54:44.870 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.856+0000 7fee356deb80 1 --2- 172.21.15.38:0/2491088502 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556d8cd2e150 0x556d8cd27ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:54:44.870 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.856+0000 7fee356deb80 1 -- 172.21.15.38:0/2491088502 >> 172.21.15.38:0/2491088502 conn(0x556d8cd11a20 msgr2=0x556d8cd11df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:54:44.870 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.856+0000 7fee356deb80 1 -- 172.21.15.38:0/2491088502 shutdown_connections 2023-03-09T14:54:44.871 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.856+0000 7fee356deb80 1 -- 172.21.15.38:0/2491088502 wait complete. 2023-03-09T14:54:44.871 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.857+0000 7fee356deb80 1 -- start start 2023-03-09T14:54:44.871 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.857+0000 7fee356deb80 1 Processor -- start 2023-03-09T14:54:44.871 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.857+0000 7fee356deb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556d8cd27ff0 0x556d8cd31840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:54:44.871 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.857+0000 7fee356deb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556d8cd2d840 0x556d8cd31d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:54:44.871 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.857+0000 7fee356deb80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556d8cd2e150 0x556d8cd1fea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:54:44.872 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.857+0000 7fee356deb80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d8caec920 con 0x556d8cd2e150 2023-03-09T14:54:44.872 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.857+0000 7fee356deb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d8cd32590 con 0x556d8cd27ff0 2023-03-09T14:54:44.872 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.857+0000 7fee356deb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d8cd203e0 con 0x556d8cd2d840 2023-03-09T14:54:44.872 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.857+0000 7fee356deb80 10 client.0 _ll_register_callbacks cb 0x556d8cc0e520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:54:44.872 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.857+0000 7fee227fc700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556d8cd2d840 0x556d8cd31d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:54:44.872 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.857+0000 7fee227fc700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556d8cd2d840 0x556d8cd31d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.38:49572/0 (socket says 172.21.15.38:49572) 2023-03-09T14:54:44.872 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.857+0000 7fee227fc700 1 -- 172.21.15.38:0/1891536720 learned_addr learned my addr 172.21.15.38:0/1891536720 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:54:44.873 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.857+0000 7fee237fe700 1 --2- 172.21.15.38:0/1891536720 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556d8cd2e150 0x556d8cd1fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:54:44.873 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.857+0000 7fee22ffd700 1 --2- 172.21.15.38:0/1891536720 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556d8cd27ff0 0x556d8cd31840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:54:44.873 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.858+0000 7fee227fc700 1 -- 172.21.15.38:0/1891536720 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556d8cd27ff0 msgr2=0x556d8cd31840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:54:44.873 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.858+0000 7fee227fc700 1 --2- 172.21.15.38:0/1891536720 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556d8cd27ff0 0x556d8cd31840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:54:44.873 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.858+0000 7fee227fc700 1 -- 172.21.15.38:0/1891536720 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556d8cd2e150 msgr2=0x556d8cd1fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:54:44.874 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.858+0000 7fee227fc700 1 --2- 172.21.15.38:0/1891536720 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556d8cd2e150 0x556d8cd1fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:54:44.874 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.858+0000 7fee227fc700 1 -- 172.21.15.38:0/1891536720 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee2410c510 con 0x556d8cd2d840 2023-03-09T14:54:44.874 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.858+0000 7fee22ffd700 1 --2- 172.21.15.38:0/1891536720 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556d8cd27ff0 0x556d8cd31840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:54:44.874 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.858+0000 7fee237fe700 1 --2- 172.21.15.38:0/1891536720 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556d8cd2e150 0x556d8cd1fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:54:44.874 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.858+0000 7fee227fc700 1 --2- 172.21.15.38:0/1891536720 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556d8cd2d840 0x556d8cd31d80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fee18006010 tx=0x7fee18005c60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=28ed2596aec98e48 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:54:44.874 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.858+0000 7fee06ff5700 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2023-03-09T14:54:44.874 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.858+0000 7fee06ff5700 1 -- 172.21.15.38:0/1891536720 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee18005600 con 0x556d8cd2d840 2023-03-09T14:54:44.875 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.858+0000 7fee356deb80 1 -- 172.21.15.38:0/1891536720 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556d8cc36190 con 0x556d8cd2d840 2023-03-09T14:54:44.875 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.858+0000 7fee06ff5700 1 -- 172.21.15.38:0/1891536720 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(237..237 src has 1..237) v4 ==== 9549+0+0 (secure 0 0 0) 0x7fee1800b790 con 0x556d8cd2d840 2023-03-09T14:54:44.875 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.859+0000 7fee06ff5700 1 -- 172.21.15.38:0/1891536720 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7fee1800b950 con 0x556d8cd2d840 2023-03-09T14:54:44.875 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.859+0000 7fee356deb80 10 client.25807 fetch_fsmap learned FSMap version 34 2023-03-09T14:54:44.875 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.859+0000 7fee356deb80 1 -- 172.21.15.38:0/1891536720 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x7fee14017070 con 0x556d8cd2d840 2023-03-09T14:54:44.875 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.859+0000 7fee06ff5700 1 -- 172.21.15.38:0/1891536720 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x556d8cc36190 con 0x556d8cd2d840 2023-03-09T14:54:44.875 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.859+0000 7fee06ff5700 1 -- 172.21.15.38:0/1891536720 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 34) v1 ==== 7482+0+0 (secure 0 0 0) 0x7fee1800faa0 con 0x556d8cd2d840 2023-03-09T14:54:44.875 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.859+0000 7fee356deb80 10 client.25807 fetch_fsmap finished waiting for FSMap version 34 2023-03-09T14:54:44.876 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.859+0000 7fee356deb80 10 client.25807 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T14:54:44.876 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.859+0000 7fee356deb80 20 client.25807 populate_metadata read hostname 'smithi038' 2023-03-09T14:54:44.876 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.859+0000 7fee356deb80 1 --2- 172.21.15.38:0/1891536720 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x556d8cd244f0 0x556d8cd26900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:54:44.876 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.859+0000 7fee356deb80 4 client.25807 mds_command: new command op to 24530 tid=0[{"prefix": "get_command_descriptions"}] 2023-03-09T14:54:44.876 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.859+0000 7fee356deb80 1 -- 172.21.15.38:0/1891536720 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556d8caecf60 con 0x556d8cd244f0 2023-03-09T14:54:44.876 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.859+0000 7fee22ffd700 1 --2- 172.21.15.38:0/1891536720 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x556d8cd244f0 0x556d8cd26900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:54:44.876 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.863+0000 7fee22ffd700 1 --2- 172.21.15.38:0/1891536720 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x556d8cd244f0 0x556d8cd26900 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=479b4556275e5ce8 server_cookie=a1f5a1ab82be3165 in_seq=0 out_seq=0 2023-03-09T14:54:44.877 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.863+0000 7fee06ff5700 10 client.25807 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:54:44.878 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.864+0000 7fee06ff5700 1 -- 172.21.15.38:0/1891536720 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18278 (crc 0 0 0) 0x556d8caecf60 con 0x556d8cd244f0 2023-03-09T14:54:44.878 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.864+0000 7fee06ff5700 10 client.25807 handle_command_reply: tid=0 2023-03-09T14:54:44.878 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.864+0000 7fee356deb80 1 client.25807 shutdown 2023-03-09T14:54:44.878 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.864+0000 7fee356deb80 1 -- 172.21.15.38:0/1891536720 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556d8cd2d840 msgr2=0x556d8cd31d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:54:44.878 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.864+0000 7fee356deb80 1 --2- 172.21.15.38:0/1891536720 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556d8cd2d840 0x556d8cd31d80 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fee18006010 tx=0x7fee18005c60 comp rx=0 tx=0).stop 2023-03-09T14:54:44.878 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.864+0000 7fee356deb80 1 -- 172.21.15.38:0/1891536720 shutdown_connections 2023-03-09T14:54:44.879 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.864+0000 7fee356deb80 1 --2- 172.21.15.38:0/1891536720 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x556d8cd244f0 0x556d8cd26900 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:54:44.879 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.864+0000 7fee356deb80 1 --2- 172.21.15.38:0/1891536720 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556d8cd2e150 0x556d8cd1fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:54:44.879 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.864+0000 7fee06ff5700 0 client.25807 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:54:44.879 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.864+0000 7fee356deb80 1 --2- 172.21.15.38:0/1891536720 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556d8cd27ff0 0x556d8cd31840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:54:44.879 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.864+0000 7fee356deb80 1 --2- 172.21.15.38:0/1891536720 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556d8cd2d840 0x556d8cd31d80 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:54:44.880 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.864+0000 7fee356deb80 1 -- 172.21.15.38:0/1891536720 >> 172.21.15.38:0/1891536720 conn(0x556d8cd11a20 msgr2=0x556d8cd11df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:54:44.880 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.864+0000 7fee356deb80 1 -- 172.21.15.38:0/1891536720 shutdown_connections 2023-03-09T14:54:44.880 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.864+0000 7fee356deb80 1 -- 172.21.15.38:0/1891536720 wait complete. 2023-03-09T14:54:44.880 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.865+0000 7fee356deb80 20 client.25807 trim_cache size 0 max 16384 2023-03-09T14:54:44.890 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.887+0000 7fee356deb80 1 Processor -- start 2023-03-09T14:54:44.890 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.887+0000 7fee356deb80 1 -- start start 2023-03-09T14:54:44.890 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.887+0000 7fee356deb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556d8cd27ff0 0x556d8cd1adf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:54:44.890 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.887+0000 7fee356deb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556d8cd2d840 0x556d8cd2c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:54:44.890 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.887+0000 7fee356deb80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556d8cd2e150 0x556d8cd2ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:54:44.891 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.887+0000 7fee356deb80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d8cd28ae0 con 0x556d8cd2e150 2023-03-09T14:54:44.891 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.887+0000 7fee356deb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d8cd34440 con 0x556d8cd27ff0 2023-03-09T14:54:44.891 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.887+0000 7fee356deb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d8cd28d20 con 0x556d8cd2d840 2023-03-09T14:54:44.891 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.887+0000 7fee237fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556d8cd2e150 0x556d8cd2ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:54:44.891 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.887+0000 7fee227fc700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556d8cd2d840 0x556d8cd2c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:54:44.891 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.887+0000 7fee22ffd700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556d8cd27ff0 0x556d8cd1adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:54:44.891 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.887+0000 7fee237fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556d8cd2e150 0x556d8cd2ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:41886/0 (socket says 172.21.15.38:41886) 2023-03-09T14:54:44.892 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.887+0000 7fee227fc700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556d8cd2d840 0x556d8cd2c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.38:49582/0 (socket says 172.21.15.38:49582) 2023-03-09T14:54:44.892 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.887+0000 7fee237fe700 1 -- 172.21.15.38:0/3977358273 learned_addr learned my addr 172.21.15.38:0/3977358273 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:54:44.892 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.888+0000 7fee237fe700 1 -- 172.21.15.38:0/3977358273 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556d8cd2d840 msgr2=0x556d8cd2c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:54:44.892 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.888+0000 7fee237fe700 1 --2- 172.21.15.38:0/3977358273 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556d8cd2d840 0x556d8cd2c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:54:44.892 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.888+0000 7fee237fe700 1 -- 172.21.15.38:0/3977358273 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556d8cd27ff0 msgr2=0x556d8cd1adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:54:44.892 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.888+0000 7fee237fe700 1 --2- 172.21.15.38:0/3977358273 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556d8cd27ff0 0x556d8cd1adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:54:44.892 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.888+0000 7fee237fe700 1 -- 172.21.15.38:0/3977358273 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556d8cd38bd0 con 0x556d8cd2e150 2023-03-09T14:54:44.893 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.888+0000 7fee237fe700 1 --2- 172.21.15.38:0/3977358273 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556d8cd2e150 0x556d8cd2ce80 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fee1c00ef90 tx=0x7fee1c0c9040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ac4bbf1690b456b server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:54:44.898 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.888+0000 7fee09ffb700 1 -- 172.21.15.38:0/3977358273 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee1c097430 con 0x556d8cd2e150 2023-03-09T14:54:44.899 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.888+0000 7fee09ffb700 1 -- 172.21.15.38:0/3977358273 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fee1c004ad0 con 0x556d8cd2e150 2023-03-09T14:54:44.899 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.888+0000 7fee09ffb700 1 -- 172.21.15.38:0/3977358273 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee1c09f7a0 con 0x556d8cd2e150 2023-03-09T14:54:44.899 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.889+0000 7fee356deb80 1 -- 172.21.15.38:0/3977358273 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556d8cd2e150 msgr2=0x556d8cd2ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:54:44.899 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.889+0000 7fee356deb80 1 --2- 172.21.15.38:0/3977358273 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556d8cd2e150 0x556d8cd2ce80 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fee1c00ef90 tx=0x7fee1c0c9040 comp rx=0 tx=0).stop 2023-03-09T14:54:44.900 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.889+0000 7fee356deb80 1 -- 172.21.15.38:0/3977358273 shutdown_connections 2023-03-09T14:54:44.900 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.889+0000 7fee356deb80 1 --2- 172.21.15.38:0/3977358273 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556d8cd2e150 0x556d8cd2ce80 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:54:44.900 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.889+0000 7fee356deb80 1 --2- 172.21.15.38:0/3977358273 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556d8cd27ff0 0x556d8cd1adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:54:44.901 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.889+0000 7fee356deb80 1 --2- 172.21.15.38:0/3977358273 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556d8cd2d840 0x556d8cd2c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:54:44.901 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.889+0000 7fee356deb80 1 -- 172.21.15.38:0/3977358273 >> 172.21.15.38:0/3977358273 conn(0x556d8cd11a20 msgr2=0x556d8cd2b9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:54:44.901 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.889+0000 7fee356deb80 1 -- 172.21.15.38:0/3977358273 shutdown_connections 2023-03-09T14:54:44.901 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.889+0000 7fee356deb80 1 -- 172.21.15.38:0/3977358273 wait complete. 2023-03-09T14:54:44.902 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.889+0000 7fee356deb80 1 -- start start 2023-03-09T14:54:44.902 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.889+0000 7fee356deb80 1 Processor -- start 2023-03-09T14:54:44.902 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.890+0000 7fee356deb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556d8cd27ff0 0x556d8cd1a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:54:44.903 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.890+0000 7fee356deb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556d8cd2d840 0x556d8cd1a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:54:44.903 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.890+0000 7fee356deb80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556d8cd2e150 0x556d8cd7ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:54:44.903 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.890+0000 7fee356deb80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d8cd7d260 con 0x556d8cd2e150 2023-03-09T14:54:44.903 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.890+0000 7fee356deb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d8cd7d3d0 con 0x556d8cd2d840 2023-03-09T14:54:44.904 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.890+0000 7fee356deb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d8cd7d590 con 0x556d8cd27ff0 2023-03-09T14:54:44.904 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.890+0000 7fee356deb80 10 client.0 _ll_register_callbacks cb 0x556d8cc0e520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:54:44.904 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.890+0000 7fee227fc700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556d8cd2d840 0x556d8cd1a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:54:44.905 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.890+0000 7fee22ffd700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556d8cd27ff0 0x556d8cd1a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:54:44.905 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.890+0000 7fee227fc700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556d8cd2d840 0x556d8cd1a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.38:51078/0 (socket says 172.21.15.38:51078) 2023-03-09T14:54:44.905 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.890+0000 7fee227fc700 1 -- 172.21.15.38:0/93603194 learned_addr learned my addr 172.21.15.38:0/93603194 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:54:44.905 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.890+0000 7fee22ffd700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556d8cd27ff0 0x556d8cd1a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.38:49588/0 (socket says 172.21.15.38:49588) 2023-03-09T14:54:44.906 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.890+0000 7fee237fe700 1 --2- 172.21.15.38:0/93603194 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556d8cd2e150 0x556d8cd7ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:54:44.906 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.890+0000 7fee227fc700 1 -- 172.21.15.38:0/93603194 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556d8cd27ff0 msgr2=0x556d8cd1a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:54:44.906 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.890+0000 7fee227fc700 1 --2- 172.21.15.38:0/93603194 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556d8cd27ff0 0x556d8cd1a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:54:44.906 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.890+0000 7fee227fc700 1 -- 172.21.15.38:0/93603194 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556d8cd2e150 msgr2=0x556d8cd7ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:54:44.907 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.890+0000 7fee227fc700 1 --2- 172.21.15.38:0/93603194 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556d8cd2e150 0x556d8cd7ccf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:54:44.907 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.891+0000 7fee227fc700 1 -- 172.21.15.38:0/93603194 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee2410ecf0 con 0x556d8cd2d840 2023-03-09T14:54:44.908 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.891+0000 7fee237fe700 1 --2- 172.21.15.38:0/93603194 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556d8cd2e150 0x556d8cd7ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:54:44.908 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.891+0000 7fee227fc700 1 --2- 172.21.15.38:0/93603194 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556d8cd2d840 0x556d8cd1a740 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fee180098e0 tx=0x7fee18009cd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f933db719263697 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:54:44.908 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.891+0000 7fee06ff5700 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2023-03-09T14:54:44.908 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.891+0000 7fee06ff5700 1 -- 172.21.15.38:0/93603194 <== mon.2 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee18017540 con 0x556d8cd2d840 2023-03-09T14:54:44.909 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.891+0000 7fee356deb80 1 -- 172.21.15.38:0/93603194 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556d8cc261b0 con 0x556d8cd2d840 2023-03-09T14:54:44.909 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.891+0000 7fee06ff5700 1 -- 172.21.15.38:0/93603194 <== mon.2 v2:172.21.15.122:3300/0 2 ==== osd_map(237..237 src has 1..237) v4 ==== 9549+0+0 (secure 0 0 0) 0x7fee18005940 con 0x556d8cd2d840 2023-03-09T14:54:44.909 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.892+0000 7fee06ff5700 1 -- 172.21.15.38:0/93603194 <== mon.2 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7fee180167b0 con 0x556d8cd2d840 2023-03-09T14:54:44.910 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.892+0000 7fee356deb80 10 client.25853 fetch_fsmap learned FSMap version 34 2023-03-09T14:54:44.910 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.892+0000 7fee356deb80 1 -- 172.21.15.38:0/93603194 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x556d8ca96de0 con 0x556d8cd2d840 2023-03-09T14:54:44.910 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.892+0000 7fee06ff5700 1 -- 172.21.15.38:0/93603194 <== mon.2 v2:172.21.15.122:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x556d8cc261b0 con 0x556d8cd2d840 2023-03-09T14:54:44.910 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.892+0000 7fee06ff5700 1 -- 172.21.15.38:0/93603194 <== mon.2 v2:172.21.15.122:3300/0 5 ==== fsmap(e 34) v1 ==== 7482+0+0 (secure 0 0 0) 0x7fee1800b260 con 0x556d8cd2d840 2023-03-09T14:54:44.911 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.892+0000 7fee356deb80 10 client.25853 fetch_fsmap finished waiting for FSMap version 34 2023-03-09T14:54:44.911 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.892+0000 7fee356deb80 10 client.25853 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T14:54:44.911 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.892+0000 7fee356deb80 20 client.25853 populate_metadata read hostname 'smithi038' 2023-03-09T14:54:44.911 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.892+0000 7fee356deb80 1 --2- 172.21.15.38:0/93603194 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x556d8cd85b20 0x556d8cd87f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:54:44.912 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.892+0000 7fee356deb80 4 client.25853 mds_command: new command op to 24530 tid=0[{"prefix": "scrub status"}] 2023-03-09T14:54:44.912 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.892+0000 7fee356deb80 1 -- 172.21.15.38:0/93603194 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x556d8cc101c0 con 0x556d8cd85b20 2023-03-09T14:54:44.912 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.893+0000 7fee22ffd700 1 --2- 172.21.15.38:0/93603194 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x556d8cd85b20 0x556d8cd87f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:54:44.913 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.893+0000 7fee06ff5700 10 client.25853 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:54:44.913 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.893+0000 7fee22ffd700 1 --2- 172.21.15.38:0/93603194 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x556d8cd85b20 0x556d8cd87f30 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=128aabd4d203ff3e server_cookie=70ad18e6a5c8e1f3 in_seq=0 out_seq=0 2023-03-09T14:54:44.913 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.894+0000 7fee06ff5700 1 -- 172.21.15.38:0/93603194 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+265 (crc 0 0 0) 0x556d8cc101c0 con 0x556d8cd85b20 2023-03-09T14:54:44.913 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.894+0000 7fee06ff5700 10 client.25853 handle_command_reply: tid=0 2023-03-09T14:54:44.914 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.894+0000 7fee356deb80 1 client.25853 shutdown 2023-03-09T14:54:44.914 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.894+0000 7fee356deb80 1 -- 172.21.15.38:0/93603194 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556d8cd2d840 msgr2=0x556d8cd1a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:54:44.914 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.894+0000 7fee356deb80 1 --2- 172.21.15.38:0/93603194 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556d8cd2d840 0x556d8cd1a740 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fee180098e0 tx=0x7fee18009cd0 comp rx=0 tx=0).stop 2023-03-09T14:54:44.915 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.894+0000 7fee356deb80 1 -- 172.21.15.38:0/93603194 shutdown_connections 2023-03-09T14:54:44.915 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.894+0000 7fee356deb80 1 --2- 172.21.15.38:0/93603194 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x556d8cd85b20 0x556d8cd87f30 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:54:44.915 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.894+0000 7fee356deb80 1 --2- 172.21.15.38:0/93603194 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x556d8cd2e150 0x556d8cd7ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:54:44.915 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.894+0000 7fee356deb80 1 --2- 172.21.15.38:0/93603194 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x556d8cd27ff0 0x556d8cd1a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:54:44.916 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.894+0000 7fee356deb80 1 --2- 172.21.15.38:0/93603194 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x556d8cd2d840 0x556d8cd1a740 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:54:44.916 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.894+0000 7fee356deb80 1 -- 172.21.15.38:0/93603194 >> 172.21.15.38:0/93603194 conn(0x556d8cd11a20 msgr2=0x556d8cd2b9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:54:44.916 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.894+0000 7fee06ff5700 0 client.25853 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:54:44.917 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.894+0000 7fee356deb80 1 -- 172.21.15.38:0/93603194 shutdown_connections 2023-03-09T14:54:44.917 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.894+0000 7fee356deb80 1 -- 172.21.15.38:0/93603194 wait complete. 2023-03-09T14:54:44.917 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.894+0000 7fee356deb80 20 client.25853 trim_cache size 0 max 16384 2023-03-09T14:54:44.918 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2023-03-09T14:54:44.918 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "scrub active (3 inodes in the stack)", 2023-03-09T14:54:44.918 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": { 2023-03-09T14:54:44.919 INFO:teuthology.orchestra.run.smithi038.stdout: "a14659f4-d7ea-4c5a-b15b-6138af3f909f": { 2023-03-09T14:54:44.919 INFO:teuthology.orchestra.run.smithi038.stdout: "path": "/", 2023-03-09T14:54:44.919 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "a14659f4-d7ea-4c5a-b15b-6138af3f909f", 2023-03-09T14:54:44.920 INFO:teuthology.orchestra.run.smithi038.stdout: "options": "recursive,force" 2023-03-09T14:54:44.920 INFO:teuthology.orchestra.run.smithi038.stdout: } 2023-03-09T14:54:44.920 INFO:teuthology.orchestra.run.smithi038.stdout: } 2023-03-09T14:54:44.920 INFO:teuthology.orchestra.run.smithi038.stdout:} 2023-03-09T14:54:44.922 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.895+0000 7fee29555700 1 -- 172.21.15.38:0/998517344 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fee0c09f520 msgr2=0x7fee0c0a19d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:54:44.922 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.895+0000 7fee29555700 1 --2- 172.21.15.38:0/998517344 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fee0c09f520 0x7fee0c0a19d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fee2410bc30 tx=0x7fee14009260 comp rx=0 tx=0).stop 2023-03-09T14:54:44.922 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.895+0000 7fee29555700 1 -- 172.21.15.38:0/998517344 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fee24104100 msgr2=0x7fee2419cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:54:44.923 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.895+0000 7fee29555700 1 --2- 172.21.15.38:0/998517344 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fee24104100 0x7fee2419cb80 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fee1c00efc0 tx=0x7fee1c00c3a0 comp rx=0 tx=0).stop 2023-03-09T14:54:44.923 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.895+0000 7fee29555700 1 -- 172.21.15.38:0/998517344 shutdown_connections 2023-03-09T14:54:44.923 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.895+0000 7fee29555700 1 --2- 172.21.15.38:0/998517344 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fee0c09f520 0x7fee0c0a19d0 secure :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fee2410bc30 tx=0x7fee14009260 comp rx=0 tx=0).stop 2023-03-09T14:54:44.923 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.895+0000 7fee29555700 1 --2- 172.21.15.38:0/998517344 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fee24104100 0x7fee2419cb80 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:54:44.924 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.895+0000 7fee29555700 1 --2- 172.21.15.38:0/998517344 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fee240ff0c0 0x7fee241a12f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:54:44.924 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.895+0000 7fee29555700 1 --2- 172.21.15.38:0/998517344 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fee240ffa60 0x7fee241a1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:54:44.924 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.895+0000 7fee29555700 1 -- 172.21.15.38:0/998517344 >> 172.21.15.38:0/998517344 conn(0x7fee24075c90 msgr2=0x7fee24069370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:54:44.925 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.897+0000 7fee29555700 1 -- 172.21.15.38:0/998517344 shutdown_connections 2023-03-09T14:54:44.925 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:54:44.897+0000 7fee29555700 1 -- 172.21.15.38:0/998517344 wait complete. 2023-03-09T14:54:44.925 INFO:tasks.cephfs.filesystem:scrub status for tag:a14659f4-d7ea-4c5a-b15b-6138af3f909f - {'path': '/', 'tag': 'a14659f4-d7ea-4c5a-b15b-6138af3f909f', 'options': 'recursive,force'} 2023-03-09T14:54:46.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:54:45 smithi122 ceph-mon[93869]: pgmap v405: 129 pgs: 129 active+clean; 18 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 25 MiB/s wr, 2.96k op/s 2023-03-09T14:54:46.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:54:45 smithi181 ceph-mon[94189]: pgmap v405: 129 pgs: 129 active+clean; 18 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 25 MiB/s wr, 2.96k op/s 2023-03-09T14:54:46.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:54:45 smithi038 ceph-mon[92422]: pgmap v405: 129 pgs: 129 active+clean; 18 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 25 MiB/s wr, 2.96k op/s 2023-03-09T14:54:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:54:47 smithi122 ceph-mon[93869]: pgmap v406: 129 pgs: 129 active+clean; 18 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 26 MiB/s wr, 2.86k op/s 2023-03-09T14:54:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:54:47 smithi181 ceph-mon[94189]: pgmap v406: 129 pgs: 129 active+clean; 18 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 26 MiB/s wr, 2.86k op/s 2023-03-09T14:54:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:54:47 smithi038 ceph-mon[92422]: pgmap v406: 129 pgs: 129 active+clean; 18 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 26 MiB/s wr, 2.86k op/s 2023-03-09T14:54:48.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 260.0 s, 799.7 tps, lat 129.523 ms stddev 185.095 2023-03-09T14:54:50.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:54:49 smithi122 ceph-mon[93869]: pgmap v407: 129 pgs: 129 active+clean; 18 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 23 MiB/s wr, 2.81k op/s 2023-03-09T14:54:50.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:54:49 smithi181 ceph-mon[94189]: pgmap v407: 129 pgs: 129 active+clean; 18 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 23 MiB/s wr, 2.81k op/s 2023-03-09T14:54:50.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:54:49 smithi038 ceph-mon[92422]: pgmap v407: 129 pgs: 129 active+clean; 18 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 23 MiB/s wr, 2.81k op/s 2023-03-09T14:54:52.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:54:51 smithi122 ceph-mon[93869]: pgmap v408: 129 pgs: 129 active+clean; 18 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 22 MiB/s wr, 2.63k op/s 2023-03-09T14:54:52.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:54:51 smithi181 ceph-mon[94189]: pgmap v408: 129 pgs: 129 active+clean; 18 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 22 MiB/s wr, 2.63k op/s 2023-03-09T14:54:52.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:54:51 smithi038 ceph-mon[92422]: pgmap v408: 129 pgs: 129 active+clean; 18 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 22 MiB/s wr, 2.63k op/s 2023-03-09T14:54:53.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 265.0 s, 280.0 tps, lat 313.520 ms stddev 639.828 2023-03-09T14:54:54.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:54:53 smithi122 ceph-mon[93869]: pgmap v409: 129 pgs: 129 active+clean; 18 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 19 MiB/s wr, 2.08k op/s 2023-03-09T14:54:54.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:54:53 smithi181 ceph-mon[94189]: pgmap v409: 129 pgs: 129 active+clean; 18 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 19 MiB/s wr, 2.08k op/s 2023-03-09T14:54:54.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:54:53 smithi038 ceph-mon[92422]: pgmap v409: 129 pgs: 129 active+clean; 18 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 19 MiB/s wr, 2.08k op/s 2023-03-09T14:54:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:54:55 smithi122 ceph-mon[93869]: pgmap v410: 129 pgs: 129 active+clean; 18 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 17 MiB/s wr, 2.06k op/s 2023-03-09T14:54:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:54:55 smithi181 ceph-mon[94189]: pgmap v410: 129 pgs: 129 active+clean; 18 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 17 MiB/s wr, 2.06k op/s 2023-03-09T14:54:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:54:55 smithi038 ceph-mon[92422]: pgmap v410: 129 pgs: 129 active+clean; 18 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 17 MiB/s wr, 2.06k op/s 2023-03-09T14:54:58.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:54:57 smithi181 ceph-mon[94189]: pgmap v411: 129 pgs: 129 active+clean; 18 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 18 MiB/s wr, 2.02k op/s 2023-03-09T14:54:58.232 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:54:57 smithi122 ceph-mon[93869]: pgmap v411: 129 pgs: 129 active+clean; 18 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 18 MiB/s wr, 2.02k op/s 2023-03-09T14:54:58.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:54:57 smithi038 ceph-mon[92422]: pgmap v411: 129 pgs: 129 active+clean; 18 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 18 MiB/s wr, 2.02k op/s 2023-03-09T14:54:58.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 270.0 s, 854.3 tps, lat 137.266 ms stddev 254.315 2023-03-09T14:55:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:54:59 smithi122 ceph-mon[93869]: pgmap v412: 129 pgs: 129 active+clean; 18 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 16 MiB/s wr, 1.84k op/s 2023-03-09T14:55:00.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:54:59 smithi181 ceph-mon[94189]: pgmap v412: 129 pgs: 129 active+clean; 18 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 16 MiB/s wr, 1.84k op/s 2023-03-09T14:55:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:54:59 smithi038 ceph-mon[92422]: pgmap v412: 129 pgs: 129 active+clean; 18 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 16 MiB/s wr, 1.84k op/s 2023-03-09T14:55:02.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:55:01 smithi122 ceph-mon[93869]: pgmap v413: 129 pgs: 129 active+clean; 18 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 19 MiB/s wr, 1.95k op/s 2023-03-09T14:55:02.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:55:01 smithi181 ceph-mon[94189]: pgmap v413: 129 pgs: 129 active+clean; 18 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 19 MiB/s wr, 1.95k op/s 2023-03-09T14:55:02.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:55:01 smithi038 ceph-mon[92422]: pgmap v413: 129 pgs: 129 active+clean; 18 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 19 MiB/s wr, 1.95k op/s 2023-03-09T14:55:03.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 275.0 s, 1319.2 tps, lat 74.888 ms stddev 43.186 2023-03-09T14:55:03.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:55:03 smithi181 ceph-mon[94189]: pgmap v414: 129 pgs: 129 active+clean; 18 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 17 MiB/s wr, 1.74k op/s 2023-03-09T14:55:04.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:55:03 smithi038 ceph-mon[92422]: pgmap v414: 129 pgs: 129 active+clean; 18 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 17 MiB/s wr, 1.74k op/s 2023-03-09T14:55:04.232 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:55:03 smithi122 ceph-mon[93869]: pgmap v414: 129 pgs: 129 active+clean; 18 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 17 MiB/s wr, 1.74k op/s 2023-03-09T14:55:05.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:55:04 smithi038 ceph-mon[92422]: osdmap e238: 12 total, 12 up, 12 in 2023-03-09T14:55:05.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:55:04 smithi122 ceph-mon[93869]: osdmap e238: 12 total, 12 up, 12 in 2023-03-09T14:55:05.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:55:04 smithi181 ceph-mon[94189]: osdmap e238: 12 total, 12 up, 12 in 2023-03-09T14:55:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:55:05 smithi122 ceph-mon[93869]: pgmap v416: 129 pgs: 4 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 117 active+clean; 18 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 21 MiB/s wr, 2.28k op/s 2023-03-09T14:55:06.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:55:05 smithi181 ceph-mon[94189]: pgmap v416: 129 pgs: 4 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 117 active+clean; 18 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 21 MiB/s wr, 2.28k op/s 2023-03-09T14:55:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:55:05 smithi038 ceph-mon[92422]: pgmap v416: 129 pgs: 4 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 117 active+clean; 18 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 21 MiB/s wr, 2.28k op/s 2023-03-09T14:55:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:55:07 smithi122 ceph-mon[93869]: pgmap v417: 129 pgs: 9 active+clean+snaptrim, 22 active+clean+snaptrim_wait, 98 active+clean; 16 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 21 MiB/s wr, 2.23k op/s 2023-03-09T14:55:08.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:55:07 smithi181 ceph-mon[94189]: pgmap v417: 129 pgs: 9 active+clean+snaptrim, 22 active+clean+snaptrim_wait, 98 active+clean; 16 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 21 MiB/s wr, 2.23k op/s 2023-03-09T14:55:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:55:07 smithi038 ceph-mon[92422]: pgmap v417: 129 pgs: 9 active+clean+snaptrim, 22 active+clean+snaptrim_wait, 98 active+clean; 16 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 21 MiB/s wr, 2.23k op/s 2023-03-09T14:55:08.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 280.0 s, 614.6 tps, lat 163.826 ms stddev 236.150 2023-03-09T14:55:10.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:55:09 smithi122 ceph-mon[93869]: pgmap v418: 129 pgs: 9 active+clean+snaptrim, 22 active+clean+snaptrim_wait, 98 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 15 MiB/s wr, 1.99k op/s 2023-03-09T14:55:10.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:55:09 smithi181 ceph-mon[94189]: pgmap v418: 129 pgs: 9 active+clean+snaptrim, 22 active+clean+snaptrim_wait, 98 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 15 MiB/s wr, 1.99k op/s 2023-03-09T14:55:10.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:55:09 smithi038 ceph-mon[92422]: pgmap v418: 129 pgs: 9 active+clean+snaptrim, 22 active+clean+snaptrim_wait, 98 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 15 MiB/s wr, 1.99k op/s 2023-03-09T14:55:12.231 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:55:11 smithi122 ceph-mon[93869]: pgmap v419: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 15 MiB/s wr, 1.79k op/s 2023-03-09T14:55:12.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:55:11 smithi181 ceph-mon[94189]: pgmap v419: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 15 MiB/s wr, 1.79k op/s 2023-03-09T14:55:12.258 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:55:11 smithi038 ceph-mon[92422]: pgmap v419: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 15 MiB/s wr, 1.79k op/s 2023-03-09T14:55:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 285.0 s, 551.0 tps, lat 165.837 ms stddev 163.578 2023-03-09T14:55:14.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:55:13 smithi038 ceph-mon[92422]: osdmap e239: 12 total, 12 up, 12 in 2023-03-09T14:55:14.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:55:13 smithi038 ceph-mon[92422]: pgmap v421: 129 pgs: 129 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 12 MiB/s wr, 1.48k op/s 2023-03-09T14:55:14.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:55:13 smithi122 ceph-mon[93869]: osdmap e239: 12 total, 12 up, 12 in 2023-03-09T14:55:14.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:55:13 smithi122 ceph-mon[93869]: pgmap v421: 129 pgs: 129 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 12 MiB/s wr, 1.48k op/s 2023-03-09T14:55:14.237 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:55:13 smithi181 ceph-mon[94189]: osdmap e239: 12 total, 12 up, 12 in 2023-03-09T14:55:14.238 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:55:13 smithi181 ceph-mon[94189]: pgmap v421: 129 pgs: 129 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 12 MiB/s wr, 1.48k op/s 2023-03-09T14:55:14.927 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-03-09T14:55:15.295 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.285+0000 7f03b7155700 1 -- 172.21.15.38:0/2583487740 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f03b01196d0 msgr2=0x7f03b011bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:55:15.295 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.285+0000 7f03b7155700 1 --2- 172.21.15.38:0/2583487740 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f03b01196d0 0x7f03b011bab0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f03a0009f90 tx=0x7f03a000b330 comp rx=0 tx=0).stop 2023-03-09T14:55:15.295 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.287+0000 7f03b7155700 1 -- 172.21.15.38:0/2583487740 shutdown_connections 2023-03-09T14:55:15.295 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.287+0000 7f03b7155700 1 --2- 172.21.15.38:0/2583487740 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f03b01196d0 0x7f03b011bab0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:55:15.295 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.287+0000 7f03b7155700 1 --2- 172.21.15.38:0/2583487740 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f03b0107780 0x7f03b010e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:55:15.296 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.287+0000 7f03b7155700 1 --2- 172.21.15.38:0/2583487740 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f03b0107b50 0x7f03b0107fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:55:15.296 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.287+0000 7f03b7155700 1 -- 172.21.15.38:0/2583487740 >> 172.21.15.38:0/2583487740 conn(0x7f03b006b450 msgr2=0x7f03b00712c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:55:15.302 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.297+0000 7f03b7155700 1 -- 172.21.15.38:0/2583487740 shutdown_connections 2023-03-09T14:55:15.302 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.298+0000 7f03b7155700 1 -- 172.21.15.38:0/2583487740 wait complete. 2023-03-09T14:55:15.302 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.298+0000 7f03b7155700 1 Processor -- start 2023-03-09T14:55:15.302 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.299+0000 7f03b7155700 1 -- start start 2023-03-09T14:55:15.303 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.299+0000 7f03b7155700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f03b0107780 0x7f03b010f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:55:15.303 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.299+0000 7f03b7155700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f03b0107b50 0x7f03b010fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:55:15.303 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.299+0000 7f03b7155700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f03b01196d0 0x7f03b01b9310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:55:15.303 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.299+0000 7f03b7155700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03b01b9850 con 0x7f03b0107780 2023-03-09T14:55:15.303 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.299+0000 7f03b7155700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03b01b99c0 con 0x7f03b01196d0 2023-03-09T14:55:15.304 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.299+0000 7f03b7155700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03b01b9b80 con 0x7f03b0107b50 2023-03-09T14:55:15.304 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.300+0000 7f03b56f2700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f03b01196d0 0x7f03b01b9310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:55:15.304 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.300+0000 7f03b56f2700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f03b01196d0 0x7f03b01b9310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.38:51098/0 (socket says 172.21.15.38:51098) 2023-03-09T14:55:15.304 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.300+0000 7f03b56f2700 1 -- 172.21.15.38:0/3513253830 learned_addr learned my addr 172.21.15.38:0/3513253830 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:55:15.312 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.301+0000 7f03b4ef1700 1 --2- 172.21.15.38:0/3513253830 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f03b0107780 0x7f03b010f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:55:15.312 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.301+0000 7f03b4ef1700 1 -- 172.21.15.38:0/3513253830 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f03b0107b50 msgr2=0x7f03b010fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:55:15.312 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.301+0000 7f03b4ef1700 1 --2- 172.21.15.38:0/3513253830 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f03b0107b50 0x7f03b010fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:55:15.312 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.301+0000 7f03b4ef1700 1 -- 172.21.15.38:0/3513253830 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f03b01196d0 msgr2=0x7f03b01b9310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:55:15.312 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.301+0000 7f03b4ef1700 1 --2- 172.21.15.38:0/3513253830 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f03b01196d0 0x7f03b01b9310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:55:15.312 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.301+0000 7f03b4ef1700 1 -- 172.21.15.38:0/3513253830 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03a0009c40 con 0x7f03b0107780 2023-03-09T14:55:15.312 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.301+0000 7f03b4ef1700 1 --2- 172.21.15.38:0/3513253830 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f03b0107780 0x7f03b010f940 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f03a800e9f0 tx=0x7f03a800edb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:55:15.313 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.302+0000 7f03adffb700 1 -- 172.21.15.38:0/3513253830 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03a800cbd0 con 0x7f03b0107780 2023-03-09T14:55:15.313 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.302+0000 7f03adffb700 1 -- 172.21.15.38:0/3513253830 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f03a800cd30 con 0x7f03b0107780 2023-03-09T14:55:15.313 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.302+0000 7f03b7155700 1 -- 172.21.15.38:0/3513253830 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03b011ed80 con 0x7f03b0107780 2023-03-09T14:55:15.313 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.302+0000 7f03adffb700 1 -- 172.21.15.38:0/3513253830 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03a8018680 con 0x7f03b0107780 2023-03-09T14:55:15.313 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.302+0000 7f03b7155700 1 -- 172.21.15.38:0/3513253830 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03b011ebf0 con 0x7f03b0107780 2023-03-09T14:55:15.314 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.310+0000 7f03c32deb80 1 Processor -- start 2023-03-09T14:55:15.314 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.310+0000 7f03c32deb80 1 -- start start 2023-03-09T14:55:15.314 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.310+0000 7f03c32deb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5596e1ff1830 0x5596e1ff1c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:55:15.314 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.310+0000 7f03c32deb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5596e1ff2140 0x5596e1febaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:55:15.314 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.310+0000 7f03c32deb80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5596e1febfe0 0x5596e1fec410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:55:15.315 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.310+0000 7f03c32deb80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596e1ed5da0 con 0x5596e1febfe0 2023-03-09T14:55:15.315 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.310+0000 7f03c32deb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596e1f44780 con 0x5596e1ff1830 2023-03-09T14:55:15.315 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.310+0000 7f03c32deb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596e1f3dcb0 con 0x5596e1ff2140 2023-03-09T14:55:15.315 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.311+0000 7f03affff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5596e1febfe0 0x5596e1fec410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:55:15.315 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.311+0000 7f03affff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5596e1febfe0 0x5596e1fec410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:41912/0 (socket says 172.21.15.38:41912) 2023-03-09T14:55:15.315 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.311+0000 7f03affff700 1 -- 172.21.15.38:0/2625398477 learned_addr learned my addr 172.21.15.38:0/2625398477 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:55:15.316 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.312+0000 7f03affff700 1 -- 172.21.15.38:0/2625398477 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5596e1ff2140 msgr2=0x5596e1febaa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-03-09T14:55:15.316 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.312+0000 7f03affff700 1 --2- 172.21.15.38:0/2625398477 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5596e1ff2140 0x5596e1febaa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:55:15.316 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.312+0000 7f03affff700 1 -- 172.21.15.38:0/2625398477 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5596e1ff1830 msgr2=0x5596e1ff1c00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-03-09T14:55:15.317 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.312+0000 7f03affff700 1 --2- 172.21.15.38:0/2625398477 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5596e1ff1830 0x5596e1ff1c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:55:15.317 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.312+0000 7f03affff700 1 -- 172.21.15.38:0/2625398477 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5596e1f04190 con 0x5596e1febfe0 2023-03-09T14:55:15.317 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.312+0000 7f03affff700 1 --2- 172.21.15.38:0/2625398477 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5596e1febfe0 0x5596e1fec410 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f039800ba90 tx=0x7f039800bda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0a89ade17ffc4b1 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:55:15.317 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.312+0000 7f03adffb700 1 -- 172.21.15.38:0/3513253830 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 25) v1 ==== 133295+0+0 (secure 0 0 0) 0x7f03a800c040 con 0x7f03b0107780 2023-03-09T14:55:15.317 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.313+0000 7f03a6ffd700 1 -- 172.21.15.38:0/2625398477 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0398004a30 con 0x5596e1febfe0 2023-03-09T14:55:15.317 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.313+0000 7f03a6ffd700 1 -- 172.21.15.38:0/2625398477 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f039800dc90 con 0x5596e1febfe0 2023-03-09T14:55:15.319 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.313+0000 7f03a6ffd700 1 -- 172.21.15.38:0/2625398477 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03980054d0 con 0x5596e1febfe0 2023-03-09T14:55:15.320 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.313+0000 7f03adffb700 1 --2- 172.21.15.38:0/3513253830 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f039c09f5d0 0x7f039c0a1a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:55:15.320 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.313+0000 7f03adffb700 1 -- 172.21.15.38:0/3513253830 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(239..239 src has 1..239) v4 ==== 9589+0+0 (secure 0 0 0) 0x7f03a8014070 con 0x7f03b0107780 2023-03-09T14:55:15.321 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.313+0000 7f03b56f2700 1 --2- 172.21.15.38:0/3513253830 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f039c09f5d0 0x7f039c0a1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:55:15.321 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.314+0000 7f03c32deb80 1 -- 172.21.15.38:0/2625398477 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5596e1febfe0 msgr2=0x5596e1fec410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:55:15.321 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.314+0000 7f03c32deb80 1 --2- 172.21.15.38:0/2625398477 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5596e1febfe0 0x5596e1fec410 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f039800ba90 tx=0x7f039800bda0 comp rx=0 tx=0).stop 2023-03-09T14:55:15.322 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.314+0000 7f03c32deb80 1 -- 172.21.15.38:0/2625398477 shutdown_connections 2023-03-09T14:55:15.322 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.314+0000 7f03c32deb80 1 --2- 172.21.15.38:0/2625398477 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5596e1febfe0 0x5596e1fec410 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:55:15.322 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.314+0000 7f03c32deb80 1 --2- 172.21.15.38:0/2625398477 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5596e1ff1830 0x5596e1ff1c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:55:15.322 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.314+0000 7f03c32deb80 1 --2- 172.21.15.38:0/2625398477 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5596e1ff2140 0x5596e1febaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:55:15.322 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.314+0000 7f03c32deb80 1 -- 172.21.15.38:0/2625398477 >> 172.21.15.38:0/2625398477 conn(0x5596e1fe6a20 msgr2=0x5596e1fe6df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:55:15.322 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.314+0000 7f03c32deb80 1 -- 172.21.15.38:0/2625398477 shutdown_connections 2023-03-09T14:55:15.322 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.314+0000 7f03c32deb80 1 -- 172.21.15.38:0/2625398477 wait complete. 2023-03-09T14:55:15.323 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.315+0000 7f03c32deb80 1 -- start start 2023-03-09T14:55:15.323 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.315+0000 7f03c32deb80 1 Processor -- start 2023-03-09T14:55:15.323 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.315+0000 7f03c32deb80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5596e1febfe0 0x5596e1ff8120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:55:15.323 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.315+0000 7f03c32deb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5596e1ff1830 0x5596e1ff53a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:55:15.323 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.315+0000 7f03c32deb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5596e1ff2140 0x5596e1ff5a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:55:15.323 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.315+0000 7f03c32deb80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596e1dc1920 con 0x5596e1febfe0 2023-03-09T14:55:15.323 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.315+0000 7f03c32deb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596e1ff61b0 con 0x5596e1ff2140 2023-03-09T14:55:15.323 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.315+0000 7f03c32deb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596e2001c70 con 0x5596e1ff1830 2023-03-09T14:55:15.324 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.316+0000 7f03c32deb80 10 client.0 _ll_register_callbacks cb 0x5596e1ee3520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:55:15.324 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.316+0000 7f03b56f2700 1 --2- 172.21.15.38:0/3513253830 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f039c09f5d0 0x7f039c0a1a80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f03a00060b0 tx=0x7f03a0006000 comp rx=0 tx=0).ready entity=mgr.15558 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:55:15.324 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.317+0000 7f03b4ef1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5596e1febfe0 0x5596e1ff8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:55:15.324 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.317+0000 7f03b4ef1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5596e1febfe0 0x5596e1ff8120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:41918/0 (socket says 172.21.15.38:41918) 2023-03-09T14:55:15.324 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.317+0000 7f03b4ef1700 1 -- 172.21.15.38:0/1653250391 learned_addr learned my addr 172.21.15.38:0/1653250391 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:55:15.324 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.318+0000 7f03b4ef1700 1 -- 172.21.15.38:0/1653250391 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5596e1ff1830 msgr2=0x5596e1ff53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:55:15.325 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.318+0000 7f03b4ef1700 1 --2- 172.21.15.38:0/1653250391 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5596e1ff1830 0x5596e1ff53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:55:15.325 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.318+0000 7f03b4ef1700 1 -- 172.21.15.38:0/1653250391 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5596e1ff2140 msgr2=0x5596e1ff5a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:55:15.325 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.318+0000 7f03b4ef1700 1 --2- 172.21.15.38:0/1653250391 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5596e1ff2140 0x5596e1ff5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:55:15.325 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.318+0000 7f03b4ef1700 1 -- 172.21.15.38:0/1653250391 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f039800b6c0 con 0x5596e1febfe0 2023-03-09T14:55:15.325 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.318+0000 7f03b4ef1700 1 --2- 172.21.15.38:0/1653250391 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5596e1febfe0 0x5596e1ff8120 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f03a8098150 tx=0x7f03a8097c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d31b751292dbb7e1 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:55:15.325 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.318+0000 7f038ffff700 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2023-03-09T14:55:15.326 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.319+0000 7f038ffff700 1 -- 172.21.15.38:0/1653250391 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03a80981c0 con 0x5596e1febfe0 2023-03-09T14:55:15.326 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.319+0000 7f038ffff700 1 -- 172.21.15.38:0/1653250391 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(239..239 src has 1..239) v4 ==== 9589+0+0 (secure 0 0 0) 0x7f03a80ca050 con 0x5596e1febfe0 2023-03-09T14:55:15.326 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.319+0000 7f03c32deb80 1 -- 172.21.15.38:0/1653250391 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5596e1f0b190 con 0x5596e1febfe0 2023-03-09T14:55:15.326 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.322+0000 7f038ffff700 1 -- 172.21.15.38:0/1653250391 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7f03a80a03b0 con 0x5596e1febfe0 2023-03-09T14:55:15.332 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.329+0000 7f03c32deb80 10 client.16221 fetch_fsmap learned FSMap version 34 2023-03-09T14:55:15.333 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.330+0000 7f03c32deb80 1 -- 172.21.15.38:0/1653250391 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x5596e1d54c60 con 0x5596e1febfe0 2023-03-09T14:55:15.333 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.330+0000 7f038ffff700 1 -- 172.21.15.38:0/1653250391 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5596e1f0b190 con 0x5596e1febfe0 2023-03-09T14:55:15.333 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.330+0000 7f038ffff700 1 -- 172.21.15.38:0/1653250391 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 34) v1 ==== 7482+0+0 (secure 0 0 0) 0x7f03a80b5f70 con 0x5596e1febfe0 2023-03-09T14:55:15.334 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.330+0000 7f03c32deb80 10 client.16221 fetch_fsmap finished waiting for FSMap version 34 2023-03-09T14:55:15.334 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.330+0000 7f03c32deb80 10 client.16221 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T14:55:15.334 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.330+0000 7f03c32deb80 20 client.16221 populate_metadata read hostname 'smithi038' 2023-03-09T14:55:15.334 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.331+0000 7f03c32deb80 1 --2- 172.21.15.38:0/1653250391 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5596e2005df0 0x5596e2008200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:55:15.334 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.331+0000 7f03c32deb80 4 client.16221 mds_command: new command op to 24530 tid=0[{"prefix": "get_command_descriptions"}] 2023-03-09T14:55:15.334 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.331+0000 7f03c32deb80 1 -- 172.21.15.38:0/1653250391 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5596e1dc1f60 con 0x5596e2005df0 2023-03-09T14:55:15.337 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.333+0000 7f03affff700 1 --2- 172.21.15.38:0/1653250391 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5596e2005df0 0x5596e2008200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:55:15.370 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.367+0000 7f038ffff700 10 client.16221 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:55:15.371 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.367+0000 7f03affff700 1 --2- 172.21.15.38:0/1653250391 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5596e2005df0 0x5596e2008200 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dceaad61729275c7 server_cookie=4db84441c23295c0 in_seq=0 out_seq=0 2023-03-09T14:55:15.392 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.389+0000 7f038ffff700 1 -- 172.21.15.38:0/1653250391 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18278 (crc 0 0 0) 0x5596e1dc1f60 con 0x5596e2005df0 2023-03-09T14:55:15.392 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.389+0000 7f038ffff700 10 client.16221 handle_command_reply: tid=0 2023-03-09T14:55:15.392 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.389+0000 7f03c32deb80 1 client.16221 shutdown 2023-03-09T14:55:15.399 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.396+0000 7f03c32deb80 1 -- 172.21.15.38:0/1653250391 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5596e1febfe0 msgr2=0x5596e1ff8120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:55:15.400 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.396+0000 7f03c32deb80 1 --2- 172.21.15.38:0/1653250391 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5596e1febfe0 0x5596e1ff8120 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f03a8098150 tx=0x7f03a8097c50 comp rx=0 tx=0).stop 2023-03-09T14:55:15.401 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.399+0000 7f03c32deb80 1 -- 172.21.15.38:0/1653250391 shutdown_connections 2023-03-09T14:55:15.402 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.399+0000 7f03c32deb80 1 --2- 172.21.15.38:0/1653250391 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5596e2005df0 0x5596e2008200 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:55:15.402 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.399+0000 7f03c32deb80 1 --2- 172.21.15.38:0/1653250391 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5596e1febfe0 0x5596e1ff8120 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:55:15.402 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.399+0000 7f03c32deb80 1 --2- 172.21.15.38:0/1653250391 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5596e1ff2140 0x5596e1ff5a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:55:15.402 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.399+0000 7f03c32deb80 1 --2- 172.21.15.38:0/1653250391 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5596e1ff1830 0x5596e1ff53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:55:15.402 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.399+0000 7f03c32deb80 1 -- 172.21.15.38:0/1653250391 >> 172.21.15.38:0/1653250391 conn(0x5596e1fe6a20 msgr2=0x5596e1fe6df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:55:15.402 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.399+0000 7f038ffff700 0 client.16221 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:55:15.412 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.406+0000 7f03c32deb80 1 -- 172.21.15.38:0/1653250391 shutdown_connections 2023-03-09T14:55:15.416 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.411+0000 7f03c32deb80 1 -- 172.21.15.38:0/1653250391 wait complete. 2023-03-09T14:55:15.417 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.413+0000 7f03c32deb80 20 client.16221 trim_cache size 0 max 16384 2023-03-09T14:55:15.457 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.445+0000 7f03c32deb80 1 Processor -- start 2023-03-09T14:55:15.458 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.445+0000 7f03c32deb80 1 -- start start 2023-03-09T14:55:15.458 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.445+0000 7f03c32deb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5596e1febfe0 0x5596e1ff7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:55:15.458 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.445+0000 7f03c32deb80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5596e1ff1830 0x5596e1fe6df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:55:15.458 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.445+0000 7f03c32deb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5596e1ff2140 0x5596e1fef570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:55:15.458 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.445+0000 7f03c32deb80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596e1ff6340 con 0x5596e1ff1830 2023-03-09T14:55:15.459 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.445+0000 7f03c32deb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596e1feccc0 con 0x5596e1febfe0 2023-03-09T14:55:15.459 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.445+0000 7f03c32deb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596e2003460 con 0x5596e1ff2140 2023-03-09T14:55:15.459 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.448+0000 7f03affff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5596e1ff1830 0x5596e1fe6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:55:15.459 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.449+0000 7f03affff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5596e1ff1830 0x5596e1fe6df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:41930/0 (socket says 172.21.15.38:41930) 2023-03-09T14:55:15.459 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.449+0000 7f03affff700 1 -- 172.21.15.38:0/2204477500 learned_addr learned my addr 172.21.15.38:0/2204477500 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:55:15.459 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.449+0000 7f03affff700 1 -- 172.21.15.38:0/2204477500 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5596e1ff2140 msgr2=0x5596e1fef570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:55:15.459 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.449+0000 7f03affff700 1 --2- 172.21.15.38:0/2204477500 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5596e1ff2140 0x5596e1fef570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:55:15.460 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.449+0000 7f03affff700 1 -- 172.21.15.38:0/2204477500 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5596e1febfe0 msgr2=0x5596e1ff7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:55:15.460 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.449+0000 7f03affff700 1 --2- 172.21.15.38:0/2204477500 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5596e1febfe0 0x5596e1ff7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:55:15.460 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.449+0000 7f03affff700 1 -- 172.21.15.38:0/2204477500 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5596e1d6bde0 con 0x5596e1ff1830 2023-03-09T14:55:15.460 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.451+0000 7f03affff700 1 --2- 172.21.15.38:0/2204477500 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5596e1ff1830 0x5596e1fe6df0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f039800b9b0 tx=0x7f0398004dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65c6eb98b610d439 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:55:15.461 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.456+0000 7f03a77fe700 1 -- 172.21.15.38:0/2204477500 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0398021070 con 0x5596e1ff1830 2023-03-09T14:55:15.461 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.456+0000 7f03a77fe700 1 -- 172.21.15.38:0/2204477500 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f0398014040 con 0x5596e1ff1830 2023-03-09T14:55:15.461 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.456+0000 7f03a77fe700 1 -- 172.21.15.38:0/2204477500 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0398025770 con 0x5596e1ff1830 2023-03-09T14:55:15.461 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.457+0000 7f03c32deb80 1 -- 172.21.15.38:0/2204477500 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5596e1ff1830 msgr2=0x5596e1fe6df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:55:15.461 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.457+0000 7f03c32deb80 1 --2- 172.21.15.38:0/2204477500 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5596e1ff1830 0x5596e1fe6df0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f039800b9b0 tx=0x7f0398004dc0 comp rx=0 tx=0).stop 2023-03-09T14:55:15.462 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.457+0000 7f03c32deb80 1 -- 172.21.15.38:0/2204477500 shutdown_connections 2023-03-09T14:55:15.462 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.457+0000 7f03c32deb80 1 --2- 172.21.15.38:0/2204477500 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5596e1ff2140 0x5596e1fef570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:55:15.462 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.457+0000 7f03c32deb80 1 --2- 172.21.15.38:0/2204477500 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5596e1febfe0 0x5596e1ff7cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:55:15.462 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.457+0000 7f03c32deb80 1 --2- 172.21.15.38:0/2204477500 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5596e1ff1830 0x5596e1fe6df0 secure :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f039800b9b0 tx=0x7f0398004dc0 comp rx=0 tx=0).stop 2023-03-09T14:55:15.462 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.457+0000 7f03c32deb80 1 -- 172.21.15.38:0/2204477500 >> 172.21.15.38:0/2204477500 conn(0x5596e1fe6a20 msgr2=0x5596e2001fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:55:15.465 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.462+0000 7f03c32deb80 1 -- 172.21.15.38:0/2204477500 shutdown_connections 2023-03-09T14:55:15.467 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.464+0000 7f03c32deb80 1 -- 172.21.15.38:0/2204477500 wait complete. 2023-03-09T14:55:15.467 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.464+0000 7f03c32deb80 1 -- start start 2023-03-09T14:55:15.467 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.464+0000 7f03c32deb80 1 Processor -- start 2023-03-09T14:55:15.471 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.467+0000 7f03c32deb80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5596e1febfe0 0x5596e1fed9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:55:15.472 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.467+0000 7f03c32deb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5596e1ff2140 0x5596e1fedf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:55:15.472 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.467+0000 7f03c32deb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5596e1fee450 0x5596e2059c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:55:15.472 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.467+0000 7f03c32deb80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596e205a1b0 con 0x5596e1febfe0 2023-03-09T14:55:15.472 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.467+0000 7f03c32deb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596e205a2f0 con 0x5596e1ff2140 2023-03-09T14:55:15.472 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.467+0000 7f03c32deb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x5596e205a430 con 0x5596e1fee450 2023-03-09T14:55:15.472 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.467+0000 7f03c32deb80 10 client.0 _ll_register_callbacks cb 0x5596e1ee3520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:55:15.477 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.474+0000 7f03affff700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5596e1ff2140 0x5596e1fedf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:55:15.477 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.474+0000 7f03b4ef1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5596e1febfe0 0x5596e1fed9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:55:15.477 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.474+0000 7f03b4ef1700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5596e1febfe0 0x5596e1fed9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:41932/0 (socket says 172.21.15.38:41932) 2023-03-09T14:55:15.478 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.474+0000 7f03b4ef1700 1 -- 172.21.15.38:0/2417141946 learned_addr learned my addr 172.21.15.38:0/2417141946 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:55:15.488 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.480+0000 7f03b56f2700 1 --2- 172.21.15.38:0/2417141946 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5596e1fee450 0x5596e2059c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:55:15.488 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.484+0000 7f03b56f2700 1 -- 172.21.15.38:0/2417141946 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5596e1ff2140 msgr2=0x5596e1fedf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:55:15.489 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.484+0000 7f03b56f2700 1 --2- 172.21.15.38:0/2417141946 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5596e1ff2140 0x5596e1fedf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:55:15.489 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.484+0000 7f03b56f2700 1 -- 172.21.15.38:0/2417141946 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5596e1febfe0 msgr2=0x5596e1fed9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:55:15.489 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.484+0000 7f03b56f2700 1 --2- 172.21.15.38:0/2417141946 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5596e1febfe0 0x5596e1fed9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:55:15.489 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.484+0000 7f03b56f2700 1 -- 172.21.15.38:0/2417141946 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5596e1f04190 con 0x5596e1fee450 2023-03-09T14:55:15.505 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.501+0000 7f03b56f2700 1 --2- 172.21.15.38:0/2417141946 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5596e1fee450 0x5596e2059c70 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f03a0009af0 tx=0x7f03a0007660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d033a8969e37250e server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:55:15.505 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.502+0000 7f038ffff700 10 client.0 ms_handle_connect on v2:172.21.15.181:3300/0 2023-03-09T14:55:15.513 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.508+0000 7f038ffff700 1 -- 172.21.15.38:0/2417141946 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03a001b630 con 0x5596e1fee450 2023-03-09T14:55:15.514 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.508+0000 7f038ffff700 1 -- 172.21.15.38:0/2417141946 <== mon.1 v2:172.21.15.181:3300/0 2 ==== osd_map(239..239 src has 1..239) v4 ==== 9589+0+0 (secure 0 0 0) 0x7f03a0022050 con 0x5596e1fee450 2023-03-09T14:55:15.514 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.508+0000 7f03c32deb80 1 -- 172.21.15.38:0/2417141946 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5596e1efb1b0 con 0x5596e1fee450 2023-03-09T14:55:15.522 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.519+0000 7f038ffff700 1 -- 172.21.15.38:0/2417141946 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7f03a001b790 con 0x5596e1fee450 2023-03-09T14:55:15.522 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.519+0000 7f03c32deb80 10 client.25819 fetch_fsmap learned FSMap version 34 2023-03-09T14:55:15.522 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.519+0000 7f03c32deb80 1 -- 172.21.15.38:0/2417141946 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x5596e1d6a2b0 con 0x5596e1fee450 2023-03-09T14:55:15.529 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.526+0000 7f038ffff700 1 -- 172.21.15.38:0/2417141946 <== mon.1 v2:172.21.15.181:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f03a0034030 con 0x5596e1fee450 2023-03-09T14:55:15.529 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.526+0000 7f038ffff700 1 -- 172.21.15.38:0/2417141946 <== mon.1 v2:172.21.15.181:3300/0 5 ==== fsmap(e 34) v1 ==== 7482+0+0 (secure 0 0 0) 0x7f03a0027b50 con 0x5596e1fee450 2023-03-09T14:55:15.530 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.526+0000 7f03c32deb80 10 client.25819 fetch_fsmap finished waiting for FSMap version 34 2023-03-09T14:55:15.530 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.526+0000 7f03c32deb80 10 client.25819 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T14:55:15.530 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.526+0000 7f03c32deb80 20 client.25819 populate_metadata read hostname 'smithi038' 2023-03-09T14:55:15.530 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.526+0000 7f03c32deb80 1 --2- 172.21.15.38:0/2417141946 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5596e205bbb0 0x5596e205dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:55:15.530 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.526+0000 7f03c32deb80 4 client.25819 mds_command: new command op to 24530 tid=0[{"prefix": "scrub status"}] 2023-03-09T14:55:15.530 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.526+0000 7f03c32deb80 1 -- 172.21.15.38:0/2417141946 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5596e1ee51c0 con 0x5596e205bbb0 2023-03-09T14:55:15.532 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.529+0000 7f03b4ef1700 1 --2- 172.21.15.38:0/2417141946 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5596e205bbb0 0x5596e205dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:55:15.552 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.549+0000 7f03b4ef1700 1 --2- 172.21.15.38:0/2417141946 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5596e205bbb0 0x5596e205dfc0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cdefcd113d445da4 server_cookie=58b27efefa96944a in_seq=0 out_seq=0 2023-03-09T14:55:15.559 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.556+0000 7f038ffff700 10 client.25819 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:55:15.564 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.561+0000 7f038ffff700 1 -- 172.21.15.38:0/2417141946 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+265 (crc 0 0 0) 0x5596e1ee51c0 con 0x5596e205bbb0 2023-03-09T14:55:15.564 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.561+0000 7f038ffff700 10 client.25819 handle_command_reply: tid=0 2023-03-09T14:55:15.564 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.562+0000 7f03c32deb80 1 client.25819 shutdown 2023-03-09T14:55:15.577 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2023-03-09T14:55:15.577 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "scrub active (3 inodes in the stack)", 2023-03-09T14:55:15.577 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": { 2023-03-09T14:55:15.577 INFO:teuthology.orchestra.run.smithi038.stdout: "a14659f4-d7ea-4c5a-b15b-6138af3f909f": { 2023-03-09T14:55:15.577 INFO:teuthology.orchestra.run.smithi038.stdout: "path": "/", 2023-03-09T14:55:15.578 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "a14659f4-d7ea-4c5a-b15b-6138af3f909f", 2023-03-09T14:55:15.578 INFO:teuthology.orchestra.run.smithi038.stdout: "options": "recursive,force" 2023-03-09T14:55:15.578 INFO:teuthology.orchestra.run.smithi038.stdout: } 2023-03-09T14:55:15.578 INFO:teuthology.orchestra.run.smithi038.stdout: } 2023-03-09T14:55:15.578 INFO:teuthology.orchestra.run.smithi038.stdout:} 2023-03-09T14:55:15.578 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.572+0000 7f03c32deb80 1 -- 172.21.15.38:0/2417141946 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5596e1fee450 msgr2=0x5596e2059c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:55:15.579 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.572+0000 7f03c32deb80 1 --2- 172.21.15.38:0/2417141946 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5596e1fee450 0x5596e2059c70 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f03a0009af0 tx=0x7f03a0007660 comp rx=0 tx=0).stop 2023-03-09T14:55:15.579 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.572+0000 7f03c32deb80 1 -- 172.21.15.38:0/2417141946 shutdown_connections 2023-03-09T14:55:15.579 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.572+0000 7f03c32deb80 1 --2- 172.21.15.38:0/2417141946 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5596e205bbb0 0x5596e205dfc0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:55:15.579 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.572+0000 7f03c32deb80 1 --2- 172.21.15.38:0/2417141946 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5596e1febfe0 0x5596e1fed9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:55:15.579 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.572+0000 7f03c32deb80 1 --2- 172.21.15.38:0/2417141946 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5596e1fee450 0x5596e2059c70 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:55:15.579 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.572+0000 7f03c32deb80 1 --2- 172.21.15.38:0/2417141946 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5596e1ff2140 0x5596e1fedf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:55:15.579 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.572+0000 7f03c32deb80 1 -- 172.21.15.38:0/2417141946 >> 172.21.15.38:0/2417141946 conn(0x5596e1fe6a20 msgr2=0x5596e2001fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:55:15.579 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.572+0000 7f038ffff700 0 client.25819 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:55:15.580 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.572+0000 7f03c32deb80 1 -- 172.21.15.38:0/2417141946 shutdown_connections 2023-03-09T14:55:15.580 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.572+0000 7f03c32deb80 1 -- 172.21.15.38:0/2417141946 wait complete. 2023-03-09T14:55:15.580 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.572+0000 7f03c32deb80 20 client.25819 trim_cache size 0 max 16384 2023-03-09T14:55:15.584 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.577+0000 7f03b7155700 1 -- 172.21.15.38:0/3513253830 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f039c09f5d0 msgr2=0x7f039c0a1a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:55:15.584 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.577+0000 7f03b7155700 1 --2- 172.21.15.38:0/3513253830 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f039c09f5d0 0x7f039c0a1a80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f03a00060b0 tx=0x7f03a0006000 comp rx=0 tx=0).stop 2023-03-09T14:55:15.585 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.577+0000 7f03b7155700 1 -- 172.21.15.38:0/3513253830 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f03b0107780 msgr2=0x7f03b010f940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:55:15.585 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.577+0000 7f03b7155700 1 --2- 172.21.15.38:0/3513253830 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f03b0107780 0x7f03b010f940 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f03a800e9f0 tx=0x7f03a800edb0 comp rx=0 tx=0).stop 2023-03-09T14:55:15.585 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.577+0000 7f03b7155700 1 -- 172.21.15.38:0/3513253830 shutdown_connections 2023-03-09T14:55:15.585 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.577+0000 7f03b7155700 1 --2- 172.21.15.38:0/3513253830 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f039c09f5d0 0x7f039c0a1a80 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:55:15.585 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.577+0000 7f03b7155700 1 --2- 172.21.15.38:0/3513253830 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f03b0107780 0x7f03b010f940 secure :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f03a800e9f0 tx=0x7f03a800edb0 comp rx=0 tx=0).stop 2023-03-09T14:55:15.585 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.577+0000 7f03b7155700 1 --2- 172.21.15.38:0/3513253830 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f03b01196d0 0x7f03b01b9310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:55:15.585 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.577+0000 7f03b7155700 1 --2- 172.21.15.38:0/3513253830 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f03b0107b50 0x7f03b010fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:55:15.586 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.577+0000 7f03b7155700 1 -- 172.21.15.38:0/3513253830 >> 172.21.15.38:0/3513253830 conn(0x7f03b006b450 msgr2=0x7f03b00712a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:55:15.588 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.586+0000 7f03b7155700 1 -- 172.21.15.38:0/3513253830 shutdown_connections 2023-03-09T14:55:15.594 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:15.588+0000 7f03b7155700 1 -- 172.21.15.38:0/3513253830 wait complete. 2023-03-09T14:55:15.618 INFO:tasks.cephfs.filesystem:scrub status for tag:a14659f4-d7ea-4c5a-b15b-6138af3f909f - {'path': '/', 'tag': 'a14659f4-d7ea-4c5a-b15b-6138af3f909f', 'options': 'recursive,force'} 2023-03-09T14:55:16.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:55:15 smithi038 ceph-mon[92422]: pgmap v422: 129 pgs: 129 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 12 MiB/s wr, 1.59k op/s 2023-03-09T14:55:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:55:15 smithi122 ceph-mon[93869]: pgmap v422: 129 pgs: 129 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 12 MiB/s wr, 1.59k op/s 2023-03-09T14:55:16.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:55:15 smithi181 ceph-mon[94189]: pgmap v422: 129 pgs: 129 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 12 MiB/s wr, 1.59k op/s 2023-03-09T14:55:18.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:55:17 smithi122 ceph-mon[93869]: pgmap v423: 129 pgs: 129 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 16 MiB/s wr, 2.03k op/s 2023-03-09T14:55:18.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:55:17 smithi122 ceph-mon[93869]: osdmap e240: 12 total, 12 up, 12 in 2023-03-09T14:55:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:55:17 smithi181 ceph-mon[94189]: pgmap v423: 129 pgs: 129 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 16 MiB/s wr, 2.03k op/s 2023-03-09T14:55:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:55:17 smithi181 ceph-mon[94189]: osdmap e240: 12 total, 12 up, 12 in 2023-03-09T14:55:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:55:17 smithi038 ceph-mon[92422]: pgmap v423: 129 pgs: 129 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 16 MiB/s wr, 2.03k op/s 2023-03-09T14:55:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:55:17 smithi038 ceph-mon[92422]: osdmap e240: 12 total, 12 up, 12 in 2023-03-09T14:55:18.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 290.0 s, 580.4 tps, lat 186.189 ms stddev 525.375 2023-03-09T14:55:20.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:55:19 smithi122 ceph-mon[93869]: pgmap v425: 129 pgs: 129 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 21 MiB/s wr, 2.50k op/s 2023-03-09T14:55:20.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:55:19 smithi181 ceph-mon[94189]: pgmap v425: 129 pgs: 129 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 21 MiB/s wr, 2.50k op/s 2023-03-09T14:55:20.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:55:19 smithi038 ceph-mon[92422]: pgmap v425: 129 pgs: 129 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 21 MiB/s wr, 2.50k op/s 2023-03-09T14:55:22.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:55:21 smithi122 ceph-mon[93869]: pgmap v426: 129 pgs: 129 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 21 MiB/s wr, 2.69k op/s 2023-03-09T14:55:22.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:55:21 smithi181 ceph-mon[94189]: pgmap v426: 129 pgs: 129 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 21 MiB/s wr, 2.69k op/s 2023-03-09T14:55:22.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:55:21 smithi038 ceph-mon[92422]: pgmap v426: 129 pgs: 129 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 21 MiB/s wr, 2.69k op/s 2023-03-09T14:55:23.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 295.0 s, 907.2 tps, lat 111.592 ms stddev 139.494 2023-03-09T14:55:24.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:55:23 smithi038 ceph-mon[92422]: pgmap v427: 129 pgs: 129 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 23 MiB/s wr, 2.75k op/s 2023-03-09T14:55:24.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:55:23 smithi122 ceph-mon[93869]: pgmap v427: 129 pgs: 129 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 23 MiB/s wr, 2.75k op/s 2023-03-09T14:55:24.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:55:23 smithi181 ceph-mon[94189]: pgmap v427: 129 pgs: 129 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 23 MiB/s wr, 2.75k op/s 2023-03-09T14:55:26.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:55:25 smithi122 ceph-mon[93869]: pgmap v428: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 25 MiB/s wr, 2.98k op/s 2023-03-09T14:55:26.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:55:25 smithi181 ceph-mon[94189]: pgmap v428: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 25 MiB/s wr, 2.98k op/s 2023-03-09T14:55:26.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:55:25 smithi038 ceph-mon[92422]: pgmap v428: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 25 MiB/s wr, 2.98k op/s 2023-03-09T14:55:28.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:55:27 smithi122 ceph-mon[93869]: pgmap v429: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 24 MiB/s wr, 2.89k op/s 2023-03-09T14:55:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:55:27 smithi181 ceph-mon[94189]: pgmap v429: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 24 MiB/s wr, 2.89k op/s 2023-03-09T14:55:28.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:55:27 smithi038 ceph-mon[92422]: pgmap v429: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 24 MiB/s wr, 2.89k op/s 2023-03-09T14:55:28.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 300.0 s, 383.8 tps, lat 150.592 ms stddev 293.650 2023-03-09T14:55:30.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:55:29 smithi122 ceph-mon[93869]: pgmap v430: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 26 MiB/s wr, 3.22k op/s 2023-03-09T14:55:30.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:55:29 smithi181 ceph-mon[94189]: pgmap v430: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 26 MiB/s wr, 3.22k op/s 2023-03-09T14:55:30.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:55:29 smithi038 ceph-mon[92422]: pgmap v430: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 26 MiB/s wr, 3.22k op/s 2023-03-09T14:55:32.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:55:31 smithi122 ceph-mon[93869]: pgmap v431: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 24 MiB/s wr, 2.95k op/s 2023-03-09T14:55:32.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:55:31 smithi181 ceph-mon[94189]: pgmap v431: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 24 MiB/s wr, 2.95k op/s 2023-03-09T14:55:32.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:55:31 smithi038 ceph-mon[92422]: pgmap v431: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 24 MiB/s wr, 2.95k op/s 2023-03-09T14:55:33.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:55:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:55:33.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:55:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:55:33.257 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:55:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:55:33.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 305.0 s, 484.2 tps, lat 279.007 ms stddev 677.596 2023-03-09T14:55:34.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:55:33 smithi122 ceph-mon[93869]: pgmap v432: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 29 MiB/s wr, 3.46k op/s 2023-03-09T14:55:34.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:55:33 smithi181 ceph-mon[94189]: pgmap v432: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 29 MiB/s wr, 3.46k op/s 2023-03-09T14:55:34.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:55:33 smithi038 ceph-mon[92422]: pgmap v432: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 29 MiB/s wr, 3.46k op/s 2023-03-09T14:55:35.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:55:34 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:55:35.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:55:34 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:55:35.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:55:34 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:55:35.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:55:34 smithi122 ceph-mon[93869]: scrub summary: active paths [/] 2023-03-09T14:55:35.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:55:34 smithi122 ceph-mon[93869]: scrub summary: idle+waiting paths [/] 2023-03-09T14:55:35.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:55:34 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:55:35.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:55:34 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:55:35.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:55:34 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:55:35.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:55:34 smithi181 ceph-mon[94189]: scrub summary: active paths [/] 2023-03-09T14:55:35.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:55:34 smithi181 ceph-mon[94189]: scrub summary: idle+waiting paths [/] 2023-03-09T14:55:35.651 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:55:34 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:55:35.652 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:55:34 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:55:35.652 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:55:34 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:55:35.652 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:55:34 smithi038 ceph-mon[92422]: scrub summary: active paths [/] 2023-03-09T14:55:35.652 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:55:34 smithi038 ceph-mon[92422]: scrub summary: idle+waiting paths [/] 2023-03-09T14:55:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:55:36 smithi122 ceph-mon[93869]: pgmap v433: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 30 MiB/s wr, 3.55k op/s 2023-03-09T14:55:36.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:55:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T14:55:36.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:55:36 smithi181 ceph-mon[94189]: pgmap v433: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 30 MiB/s wr, 3.55k op/s 2023-03-09T14:55:36.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:55:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T14:55:36.524 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:55:36 smithi038 ceph-mon[92422]: pgmap v433: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 30 MiB/s wr, 3.55k op/s 2023-03-09T14:55:36.524 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:55:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T14:55:38.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:55:38 smithi122 ceph-mon[93869]: pgmap v434: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 31 MiB/s wr, 3.57k op/s 2023-03-09T14:55:38.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:55:38 smithi181 ceph-mon[94189]: pgmap v434: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 31 MiB/s wr, 3.57k op/s 2023-03-09T14:55:38.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:55:38 smithi038 ceph-mon[92422]: pgmap v434: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 31 MiB/s wr, 3.57k op/s 2023-03-09T14:55:38.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 310.0 s, 1178.0 tps, lat 90.630 ms stddev 89.163 2023-03-09T14:55:40.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:55:40 smithi122 ceph-mon[93869]: pgmap v435: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 28 MiB/s wr, 3.24k op/s 2023-03-09T14:55:40.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:55:40 smithi181 ceph-mon[94189]: pgmap v435: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 28 MiB/s wr, 3.24k op/s 2023-03-09T14:55:40.503 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:55:40 smithi038 ceph-mon[92422]: pgmap v435: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 28 MiB/s wr, 3.24k op/s 2023-03-09T14:55:42.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:55:42 smithi122 ceph-mon[93869]: pgmap v436: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 25 MiB/s wr, 2.88k op/s 2023-03-09T14:55:42.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:55:42 smithi181 ceph-mon[94189]: pgmap v436: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 25 MiB/s wr, 2.88k op/s 2023-03-09T14:55:42.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:55:42 smithi038 ceph-mon[92422]: pgmap v436: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 25 MiB/s wr, 2.88k op/s 2023-03-09T14:55:43.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 315.0 s, 901.3 tps, lat 111.481 ms stddev 219.751 2023-03-09T14:55:44.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:55:43 smithi122 ceph-mon[93869]: pgmap v437: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 27 MiB/s wr, 3.07k op/s 2023-03-09T14:55:44.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:55:43 smithi181 ceph-mon[94189]: pgmap v437: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 27 MiB/s wr, 3.07k op/s 2023-03-09T14:55:44.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:55:43 smithi038 ceph-mon[92422]: pgmap v437: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 27 MiB/s wr, 3.07k op/s 2023-03-09T14:55:45.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:55:44 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T14:55:45.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:55:44 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T14:55:45.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:55:44 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T14:55:45.619 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-03-09T14:55:46.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:55:45 smithi122 ceph-mon[93869]: pgmap v438: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 25 MiB/s wr, 2.83k op/s 2023-03-09T14:55:46.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:55:45 smithi181 ceph-mon[94189]: pgmap v438: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 25 MiB/s wr, 2.83k op/s 2023-03-09T14:55:46.258 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:55:45 smithi038 ceph-mon[92422]: pgmap v438: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 25 MiB/s wr, 2.83k op/s 2023-03-09T14:55:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:55:47 smithi122 ceph-mon[93869]: pgmap v439: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 22 MiB/s wr, 2.48k op/s 2023-03-09T14:55:48.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:55:47 smithi122 ceph-mon[93869]: scrub summary: active paths [/] 2023-03-09T14:55:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:55:47 smithi181 ceph-mon[94189]: pgmap v439: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 22 MiB/s wr, 2.48k op/s 2023-03-09T14:55:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:55:47 smithi181 ceph-mon[94189]: scrub summary: active paths [/] 2023-03-09T14:55:48.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:55:47 smithi038 ceph-mon[92422]: pgmap v439: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 22 MiB/s wr, 2.48k op/s 2023-03-09T14:55:48.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:55:47 smithi038 ceph-mon[92422]: scrub summary: active paths [/] 2023-03-09T14:55:48.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 320.0 s, 831.7 tps, lat 119.851 ms stddev 281.704 2023-03-09T14:55:49.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:55:48 smithi122 ceph-mon[93869]: scrub summary: idle+waiting paths [/] 2023-03-09T14:55:49.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:55:48 smithi181 ceph-mon[94189]: scrub summary: idle+waiting paths [/] 2023-03-09T14:55:49.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:55:48 smithi038 ceph-mon[92422]: scrub summary: idle+waiting paths [/] 2023-03-09T14:55:50.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:55:49 smithi122 ceph-mon[93869]: pgmap v440: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 26 MiB/s wr, 2.93k op/s 2023-03-09T14:55:50.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:55:49 smithi181 ceph-mon[94189]: pgmap v440: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 26 MiB/s wr, 2.93k op/s 2023-03-09T14:55:50.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:55:49 smithi038 ceph-mon[92422]: pgmap v440: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 15 MiB/s rd, 26 MiB/s wr, 2.93k op/s 2023-03-09T14:55:52.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:55:51 smithi181 ceph-mon[94189]: pgmap v441: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 26 MiB/s wr, 2.94k op/s 2023-03-09T14:55:52.235 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:55:51 smithi122 ceph-mon[93869]: pgmap v441: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 26 MiB/s wr, 2.94k op/s 2023-03-09T14:55:52.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:55:51 smithi038 ceph-mon[92422]: pgmap v441: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 26 MiB/s wr, 2.94k op/s 2023-03-09T14:55:53.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 325.0 s, 845.6 tps, lat 118.113 ms stddev 63.144 2023-03-09T14:55:54.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:55:53 smithi122 ceph-mon[93869]: pgmap v442: 129 pgs: 129 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 29 MiB/s wr, 3.20k op/s 2023-03-09T14:55:54.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:55:53 smithi181 ceph-mon[94189]: pgmap v442: 129 pgs: 129 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 29 MiB/s wr, 3.20k op/s 2023-03-09T14:55:54.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:55:53 smithi038 ceph-mon[92422]: pgmap v442: 129 pgs: 129 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 29 MiB/s wr, 3.20k op/s 2023-03-09T14:55:54.798 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:54.795+0000 7f6a8aa8b700 1 -- 172.21.15.38:0/2746792052 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6a841046f0 msgr2=0x7f6a84104ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:55:54.799 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:54.795+0000 7f6a8aa8b700 1 --2- 172.21.15.38:0/2746792052 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6a841046f0 0x7f6a84104ac0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f6a7800aaa0 tx=0x7f6a7800adb0 comp rx=0 tx=0).stop 2023-03-09T14:55:54.799 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:54.796+0000 7f6a8aa8b700 1 -- 172.21.15.38:0/2746792052 shutdown_connections 2023-03-09T14:55:54.800 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:54.796+0000 7f6a8aa8b700 1 --2- 172.21.15.38:0/2746792052 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6a840ffa50 0x7f6a8410b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:55:54.800 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:54.796+0000 7f6a8aa8b700 1 --2- 172.21.15.38:0/2746792052 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6a841046f0 0x7f6a84104ac0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:55:54.800 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:54.796+0000 7f6a8aa8b700 1 --2- 172.21.15.38:0/2746792052 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a840ff0a0 0x7f6a840ff510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:55:54.800 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:54.796+0000 7f6a8aa8b700 1 -- 172.21.15.38:0/2746792052 >> 172.21.15.38:0/2746792052 conn(0x7f6a840755f0 msgr2=0x7f6a840759f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:55:54.800 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:54.797+0000 7f6a8aa8b700 1 -- 172.21.15.38:0/2746792052 shutdown_connections 2023-03-09T14:55:54.800 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:54.797+0000 7f6a8aa8b700 1 -- 172.21.15.38:0/2746792052 wait complete. 2023-03-09T14:55:54.801 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:54.797+0000 7f6a8aa8b700 1 Processor -- start 2023-03-09T14:55:54.801 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:54.798+0000 7f6a8aa8b700 1 -- start start 2023-03-09T14:55:54.801 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:54.798+0000 7f6a8aa8b700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a840ff0a0 0x7f6a8410b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:55:54.802 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:54.798+0000 7f6a8aa8b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6a840ffa50 0x7f6a84108010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:55:54.802 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:54.798+0000 7f6a8aa8b700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6a841046f0 0x7f6a84108550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:55:54.802 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:54.798+0000 7f6a8aa8b700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a84108b50 con 0x7f6a840ffa50 2023-03-09T14:55:54.802 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:54.798+0000 7f6a8aa8b700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a84108cc0 con 0x7f6a841046f0 2023-03-09T14:55:54.802 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:54.798+0000 7f6a8aa8b700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a84108e80 con 0x7f6a840ff0a0 2023-03-09T14:55:54.802 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:54.798+0000 7f6a89028700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6a841046f0 0x7f6a84108550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:55:54.803 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:54.798+0000 7f6a89028700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6a841046f0 0x7f6a84108550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.38:51140/0 (socket says 172.21.15.38:51140) 2023-03-09T14:55:54.803 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:54.798+0000 7f6a89028700 1 -- 172.21.15.38:0/402439513 learned_addr learned my addr 172.21.15.38:0/402439513 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:55:54.803 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:54.798+0000 7f6a88827700 1 --2- 172.21.15.38:0/402439513 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a840ff0a0 0x7f6a8410b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:55:54.803 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:54.798+0000 7f6a83fff700 1 --2- 172.21.15.38:0/402439513 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6a840ffa50 0x7f6a84108010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:55:54.803 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:54.799+0000 7f6a89028700 1 -- 172.21.15.38:0/402439513 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a840ff0a0 msgr2=0x7f6a8410b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:55:54.803 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:54.799+0000 7f6a89028700 1 --2- 172.21.15.38:0/402439513 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a840ff0a0 0x7f6a8410b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:55:54.804 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:54.799+0000 7f6a89028700 1 -- 172.21.15.38:0/402439513 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6a840ffa50 msgr2=0x7f6a84108010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:55:54.804 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:54.799+0000 7f6a89028700 1 --2- 172.21.15.38:0/402439513 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6a840ffa50 0x7f6a84108010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:55:54.804 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:54.799+0000 7f6a89028700 1 -- 172.21.15.38:0/402439513 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a7800a700 con 0x7f6a841046f0 2023-03-09T14:55:54.804 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:54.799+0000 7f6a88827700 1 --2- 172.21.15.38:0/402439513 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a840ff0a0 0x7f6a8410b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:55:54.804 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:54.799+0000 7f6a89028700 1 --2- 172.21.15.38:0/402439513 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6a841046f0 0x7f6a84108550 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6a7c00ef30 tx=0x7f6a7c00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:55:54.804 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:54.799+0000 7f6a81ffb700 1 -- 172.21.15.38:0/402439513 <== mon.2 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a7c018070 con 0x7f6a841046f0 2023-03-09T14:55:54.804 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:54.799+0000 7f6a81ffb700 1 -- 172.21.15.38:0/402439513 <== mon.2 v2:172.21.15.122:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6a7c00f040 con 0x7f6a841046f0 2023-03-09T14:55:54.805 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:54.800+0000 7f6a8aa8b700 1 -- 172.21.15.38:0/402439513 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a8410db40 con 0x7f6a841046f0 2023-03-09T14:55:54.805 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:54.800+0000 7f6a8aa8b700 1 -- 172.21.15.38:0/402439513 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a8410d9b0 con 0x7f6a841046f0 2023-03-09T14:55:54.805 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:54.800+0000 7f6a81ffb700 1 -- 172.21.15.38:0/402439513 <== mon.2 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a7c00cc80 con 0x7f6a841046f0 2023-03-09T14:55:54.807 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:54.803+0000 7f6a81ffb700 1 -- 172.21.15.38:0/402439513 <== mon.2 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 25) v1 ==== 133295+0+0 (secure 0 0 0) 0x7f6a7c004ad0 con 0x7f6a841046f0 2023-03-09T14:55:54.807 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:54.804+0000 7f6a81ffb700 1 --2- 172.21.15.38:0/402439513 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f6a7409f6a0 0x7f6a740a1b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:55:54.807 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:54.804+0000 7f6a81ffb700 1 -- 172.21.15.38:0/402439513 <== mon.2 v2:172.21.15.122:3300/0 5 ==== osd_map(240..240 src has 1..240) v4 ==== 9561+0+0 (secure 0 0 0) 0x7f6a7c0c9d70 con 0x7f6a841046f0 2023-03-09T14:55:54.808 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:54.805+0000 7f6a88827700 1 --2- 172.21.15.38:0/402439513 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f6a7409f6a0 0x7f6a740a1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:55:54.809 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:54.806+0000 7f6a88827700 1 --2- 172.21.15.38:0/402439513 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f6a7409f6a0 0x7f6a740a1b50 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6a78005cd0 tx=0x7f6a78005c40 comp rx=0 tx=0).ready entity=mgr.15558 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:55:55.006 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.001+0000 7f6a96c14b80 1 Processor -- start 2023-03-09T14:55:55.011 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.008+0000 7f6a96c14b80 1 -- start start 2023-03-09T14:55:55.012 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.009+0000 7f6a96c14b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b9f055f830 0x55b9f055fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:55:55.012 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.009+0000 7f6a96c14b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b9f0560140 0x55b9f0559aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:55:55.012 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.009+0000 7f6a96c14b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b9f0559fe0 0x55b9f055a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:55:55.012 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.009+0000 7f6a96c14b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9f0443da0 con 0x55b9f0559fe0 2023-03-09T14:55:55.013 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.009+0000 7f6a96c14b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9f04b2780 con 0x55b9f055f830 2023-03-09T14:55:55.013 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.009+0000 7f6a96c14b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9f04abcb0 con 0x55b9f0560140 2023-03-09T14:55:55.013 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.009+0000 7f6a83fff700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b9f0560140 0x55b9f0559aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:55:55.013 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.009+0000 7f6a83fff700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b9f0560140 0x55b9f0559aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.38:49654/0 (socket says 172.21.15.38:49654) 2023-03-09T14:55:55.013 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.009+0000 7f6a83fff700 1 -- 172.21.15.38:0/1602994238 learned_addr learned my addr 172.21.15.38:0/1602994238 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:55:55.014 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.010+0000 7f6a83fff700 1 -- 172.21.15.38:0/1602994238 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b9f055f830 msgr2=0x55b9f055fc00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-03-09T14:55:55.014 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.010+0000 7f6a83fff700 1 --2- 172.21.15.38:0/1602994238 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b9f055f830 0x55b9f055fc00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:55:55.014 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.010+0000 7f6a83fff700 1 -- 172.21.15.38:0/1602994238 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b9f0559fe0 msgr2=0x55b9f055a410 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-03-09T14:55:55.014 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.010+0000 7f6a83fff700 1 --2- 172.21.15.38:0/1602994238 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b9f0559fe0 0x55b9f055a410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:55:55.014 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.010+0000 7f6a83fff700 1 -- 172.21.15.38:0/1602994238 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b9f0472190 con 0x55b9f0560140 2023-03-09T14:55:55.014 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.010+0000 7f6a83fff700 1 --2- 172.21.15.38:0/1602994238 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b9f0560140 0x55b9f0559aa0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6a7000bcf0 tx=0x7f6a700103d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=63f7584477acc37 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:55:55.015 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.011+0000 7f6a6effd700 1 -- 172.21.15.38:0/1602994238 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a70010dc0 con 0x55b9f0560140 2023-03-09T14:55:55.015 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.011+0000 7f6a6effd700 1 -- 172.21.15.38:0/1602994238 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6a70015070 con 0x55b9f0560140 2023-03-09T14:55:55.015 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.011+0000 7f6a6effd700 1 -- 172.21.15.38:0/1602994238 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a70011440 con 0x55b9f0560140 2023-03-09T14:55:55.016 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.013+0000 7f6a96c14b80 1 -- 172.21.15.38:0/1602994238 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b9f0560140 msgr2=0x55b9f0559aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:55:55.016 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.013+0000 7f6a96c14b80 1 --2- 172.21.15.38:0/1602994238 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b9f0560140 0x55b9f0559aa0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6a7000bcf0 tx=0x7f6a700103d0 comp rx=0 tx=0).stop 2023-03-09T14:55:55.017 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.014+0000 7f6a96c14b80 1 -- 172.21.15.38:0/1602994238 shutdown_connections 2023-03-09T14:55:55.017 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.014+0000 7f6a96c14b80 1 --2- 172.21.15.38:0/1602994238 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b9f0559fe0 0x55b9f055a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:55:55.017 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.014+0000 7f6a96c14b80 1 --2- 172.21.15.38:0/1602994238 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b9f055f830 0x55b9f055fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:55:55.017 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.014+0000 7f6a96c14b80 1 --2- 172.21.15.38:0/1602994238 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b9f0560140 0x55b9f0559aa0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:55:55.017 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.014+0000 7f6a96c14b80 1 -- 172.21.15.38:0/1602994238 >> 172.21.15.38:0/1602994238 conn(0x55b9f0554a20 msgr2=0x55b9f0554df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:55:55.018 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.014+0000 7f6a96c14b80 1 -- 172.21.15.38:0/1602994238 shutdown_connections 2023-03-09T14:55:55.020 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.017+0000 7f6a96c14b80 1 -- 172.21.15.38:0/1602994238 wait complete. 2023-03-09T14:55:55.021 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.018+0000 7f6a96c14b80 1 -- start start 2023-03-09T14:55:55.021 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.018+0000 7f6a96c14b80 1 Processor -- start 2023-03-09T14:55:55.021 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.018+0000 7f6a96c14b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b9f0559fe0 0x55b9f0566170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:55:55.022 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.018+0000 7f6a96c14b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b9f055f830 0x55b9f05633f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:55:55.022 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.018+0000 7f6a96c14b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b9f0560140 0x55b9f0563b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:55:55.022 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.018+0000 7f6a96c14b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9f032f920 con 0x55b9f0560140 2023-03-09T14:55:55.022 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.018+0000 7f6a96c14b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9f05642a0 con 0x55b9f055f830 2023-03-09T14:55:55.022 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.018+0000 7f6a96c14b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9f056fd00 con 0x55b9f0559fe0 2023-03-09T14:55:55.022 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.018+0000 7f6a96c14b80 10 client.0 _ll_register_callbacks cb 0x55b9f0451520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:55:55.023 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.019+0000 7f6a88827700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b9f0559fe0 0x55b9f0566170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:55:55.023 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.019+0000 7f6a83fff700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b9f055f830 0x55b9f05633f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:55:55.023 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.019+0000 7f6a88827700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b9f0559fe0 0x55b9f0566170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.38:49656/0 (socket says 172.21.15.38:49656) 2023-03-09T14:55:55.023 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.019+0000 7f6a88827700 1 -- 172.21.15.38:0/4086114128 learned_addr learned my addr 172.21.15.38:0/4086114128 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:55:55.024 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.019+0000 7f6a83fff700 1 -- 172.21.15.38:0/4086114128 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b9f0559fe0 msgr2=0x55b9f0566170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:55:55.024 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.019+0000 7f6a83fff700 1 --2- 172.21.15.38:0/4086114128 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b9f0559fe0 0x55b9f0566170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:55:55.024 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.019+0000 7f6a83fff700 1 -- 172.21.15.38:0/4086114128 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b9f0560140 msgr2=0x55b9f0563b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:55:55.024 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.019+0000 7f6a83fff700 1 --2- 172.21.15.38:0/4086114128 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b9f0560140 0x55b9f0563b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:55:55.024 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.019+0000 7f6a83fff700 1 -- 172.21.15.38:0/4086114128 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a7000b920 con 0x55b9f055f830 2023-03-09T14:55:55.024 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.020+0000 7f6a83fff700 1 --2- 172.21.15.38:0/4086114128 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b9f055f830 0x55b9f05633f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6a70000c00 tx=0x7f6a70010ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94dc92ec66f3e21a server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:55:55.024 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.020+0000 7f6a63fff700 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2023-03-09T14:55:55.027 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.024+0000 7f6a63fff700 1 -- 172.21.15.38:0/4086114128 <== mon.2 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a7001f040 con 0x55b9f055f830 2023-03-09T14:55:55.027 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.024+0000 7f6a63fff700 1 -- 172.21.15.38:0/4086114128 <== mon.2 v2:172.21.15.122:3300/0 2 ==== osd_map(240..240 src has 1..240) v4 ==== 9561+0+0 (secure 0 0 0) 0x7f6a70017070 con 0x55b9f055f830 2023-03-09T14:55:55.030 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.027+0000 7f6a96c14b80 1 -- 172.21.15.38:0/4086114128 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b9f0479190 con 0x55b9f055f830 2023-03-09T14:55:55.031 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.028+0000 7f6a63fff700 1 -- 172.21.15.38:0/4086114128 <== mon.2 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6a70005450 con 0x55b9f055f830 2023-03-09T14:55:55.034 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.031+0000 7f6a96c14b80 10 client.25871 fetch_fsmap learned FSMap version 36 2023-03-09T14:55:55.034 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.031+0000 7f6a96c14b80 1 -- 172.21.15.38:0/4086114128 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x55b9f02c2c60 con 0x55b9f055f830 2023-03-09T14:55:55.037 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.032+0000 7f6a63fff700 1 -- 172.21.15.38:0/4086114128 <== mon.2 v2:172.21.15.122:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55b9f0479190 con 0x55b9f055f830 2023-03-09T14:55:55.038 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.032+0000 7f6a63fff700 1 -- 172.21.15.38:0/4086114128 <== mon.2 v2:172.21.15.122:3300/0 5 ==== fsmap(e 36) v1 ==== 7482+0+0 (secure 0 0 0) 0x7f6a7002b050 con 0x55b9f055f830 2023-03-09T14:55:55.038 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.034+0000 7f6a96c14b80 10 client.25871 fetch_fsmap finished waiting for FSMap version 36 2023-03-09T14:55:55.038 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.034+0000 7f6a96c14b80 10 client.25871 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T14:55:55.038 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.035+0000 7f6a96c14b80 20 client.25871 populate_metadata read hostname 'smithi038' 2023-03-09T14:55:55.039 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.035+0000 7f6a96c14b80 1 --2- 172.21.15.38:0/4086114128 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55b9f0573c50 0x55b9f0576060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:55:55.039 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.035+0000 7f6a96c14b80 4 client.25871 mds_command: new command op to 24530 tid=0[{"prefix": "get_command_descriptions"}] 2023-03-09T14:55:55.039 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.035+0000 7f6a96c14b80 1 -- 172.21.15.38:0/4086114128 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b9f032ff60 con 0x55b9f0573c50 2023-03-09T14:55:55.039 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.036+0000 7f6a88827700 1 --2- 172.21.15.38:0/4086114128 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55b9f0573c50 0x55b9f0576060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:55:55.040 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.037+0000 7f6a88827700 1 --2- 172.21.15.38:0/4086114128 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55b9f0573c50 0x55b9f0576060 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cb5de7a656b28ab9 server_cookie=ae90b0370001d526 in_seq=0 out_seq=0 2023-03-09T14:55:55.040 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.037+0000 7f6a63fff700 10 client.25871 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:55:55.041 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.038+0000 7f6a63fff700 1 -- 172.21.15.38:0/4086114128 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18278 (crc 0 0 0) 0x55b9f032ff60 con 0x55b9f0573c50 2023-03-09T14:55:55.041 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.038+0000 7f6a63fff700 10 client.25871 handle_command_reply: tid=0 2023-03-09T14:55:55.041 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.039+0000 7f6a96c14b80 1 client.25871 shutdown 2023-03-09T14:55:55.044 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.040+0000 7f6a96c14b80 1 -- 172.21.15.38:0/4086114128 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b9f055f830 msgr2=0x55b9f05633f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:55:55.044 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.040+0000 7f6a96c14b80 1 --2- 172.21.15.38:0/4086114128 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b9f055f830 0x55b9f05633f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6a70000c00 tx=0x7f6a70010ea0 comp rx=0 tx=0).stop 2023-03-09T14:55:55.044 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.040+0000 7f6a96c14b80 1 -- 172.21.15.38:0/4086114128 shutdown_connections 2023-03-09T14:55:55.044 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.040+0000 7f6a96c14b80 1 --2- 172.21.15.38:0/4086114128 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55b9f0573c50 0x55b9f0576060 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:55:55.048 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.040+0000 7f6a96c14b80 1 --2- 172.21.15.38:0/4086114128 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b9f0560140 0x55b9f0563b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:55:55.048 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.040+0000 7f6a96c14b80 1 --2- 172.21.15.38:0/4086114128 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b9f0559fe0 0x55b9f0566170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:55:55.048 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.040+0000 7f6a96c14b80 1 --2- 172.21.15.38:0/4086114128 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b9f055f830 0x55b9f05633f0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:55:55.048 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.040+0000 7f6a96c14b80 1 -- 172.21.15.38:0/4086114128 >> 172.21.15.38:0/4086114128 conn(0x55b9f0554a20 msgr2=0x55b9f0553c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:55:55.048 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.041+0000 7f6a63fff700 0 client.25871 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:55:55.049 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.041+0000 7f6a96c14b80 1 -- 172.21.15.38:0/4086114128 shutdown_connections 2023-03-09T14:55:55.049 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.041+0000 7f6a96c14b80 1 -- 172.21.15.38:0/4086114128 wait complete. 2023-03-09T14:55:55.049 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.041+0000 7f6a96c14b80 20 client.25871 trim_cache size 0 max 16384 2023-03-09T14:55:55.070 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.067+0000 7f6a96c14b80 1 Processor -- start 2023-03-09T14:55:55.070 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.067+0000 7f6a96c14b80 1 -- start start 2023-03-09T14:55:55.071 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.067+0000 7f6a96c14b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b9f0554a20 0x55b9f0554df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:55:55.071 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.067+0000 7f6a96c14b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b9f0559fe0 0x55b9f0555330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:55:55.072 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.067+0000 7f6a96c14b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b9f055f830 0x55b9f055d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:55:55.072 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.067+0000 7f6a96c14b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9f0479190 con 0x55b9f0559fe0 2023-03-09T14:55:55.072 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.067+0000 7f6a96c14b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9f055aad0 con 0x55b9f055f830 2023-03-09T14:55:55.072 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.067+0000 7f6a96c14b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9f05547d0 con 0x55b9f0554a20 2023-03-09T14:55:55.072 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.068+0000 7f6a83fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b9f0559fe0 0x55b9f0555330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:55:55.072 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.068+0000 7f6a83fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b9f0559fe0 0x55b9f0555330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:41966/0 (socket says 172.21.15.38:41966) 2023-03-09T14:55:55.072 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.068+0000 7f6a83fff700 1 -- 172.21.15.38:0/2420414010 learned_addr learned my addr 172.21.15.38:0/2420414010 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:55:55.073 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.068+0000 7f6a88827700 1 --2- 172.21.15.38:0/2420414010 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b9f0554a20 0x55b9f0554df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:55:55.073 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.068+0000 7f6a89028700 1 --2- 172.21.15.38:0/2420414010 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b9f055f830 0x55b9f055d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:55:55.073 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.068+0000 7f6a88827700 1 -- 172.21.15.38:0/2420414010 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b9f055f830 msgr2=0x55b9f055d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:55:55.073 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.068+0000 7f6a88827700 1 --2- 172.21.15.38:0/2420414010 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b9f055f830 0x55b9f055d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:55:55.073 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.068+0000 7f6a88827700 1 -- 172.21.15.38:0/2420414010 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b9f0559fe0 msgr2=0x55b9f0555330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:55:55.073 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.068+0000 7f6a88827700 1 --2- 172.21.15.38:0/2420414010 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b9f0559fe0 0x55b9f0555330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:55:55.073 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.068+0000 7f6a88827700 1 -- 172.21.15.38:0/2420414010 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b9f057ac90 con 0x55b9f0554a20 2023-03-09T14:55:55.074 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.068+0000 7f6a88827700 1 --2- 172.21.15.38:0/2420414010 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b9f0554a20 0x55b9f0554df0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6a70003990 tx=0x7f6a78024000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=37c961302cf2a60b server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:55:55.074 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.069+0000 7f6a6f7fe700 1 -- 172.21.15.38:0/2420414010 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a78028040 con 0x55b9f0554a20 2023-03-09T14:55:55.074 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.069+0000 7f6a6f7fe700 1 -- 172.21.15.38:0/2420414010 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f6a78005470 con 0x55b9f0554a20 2023-03-09T14:55:55.075 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.069+0000 7f6a6f7fe700 1 -- 172.21.15.38:0/2420414010 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a7800ead0 con 0x55b9f0554a20 2023-03-09T14:55:55.075 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.070+0000 7f6a96c14b80 1 -- 172.21.15.38:0/2420414010 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b9f0554a20 msgr2=0x55b9f0554df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:55:55.075 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.070+0000 7f6a96c14b80 1 --2- 172.21.15.38:0/2420414010 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b9f0554a20 0x55b9f0554df0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6a70003990 tx=0x7f6a78024000 comp rx=0 tx=0).stop 2023-03-09T14:55:55.075 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.070+0000 7f6a96c14b80 1 -- 172.21.15.38:0/2420414010 shutdown_connections 2023-03-09T14:55:55.075 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.070+0000 7f6a96c14b80 1 --2- 172.21.15.38:0/2420414010 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b9f055f830 0x55b9f055d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:55:55.075 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.070+0000 7f6a96c14b80 1 --2- 172.21.15.38:0/2420414010 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b9f0554a20 0x55b9f0554df0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:55:55.075 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.070+0000 7f6a96c14b80 1 --2- 172.21.15.38:0/2420414010 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b9f0559fe0 0x55b9f0555330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:55:55.075 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.070+0000 7f6a96c14b80 1 -- 172.21.15.38:0/2420414010 >> 172.21.15.38:0/2420414010 conn(0x55b9f05532d0 msgr2=0x55b9f0565da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:55:55.076 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.070+0000 7f6a96c14b80 1 -- 172.21.15.38:0/2420414010 shutdown_connections 2023-03-09T14:55:55.076 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.071+0000 7f6a96c14b80 1 -- 172.21.15.38:0/2420414010 wait complete. 2023-03-09T14:55:55.076 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.071+0000 7f6a96c14b80 1 -- start start 2023-03-09T14:55:55.076 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.071+0000 7f6a96c14b80 1 Processor -- start 2023-03-09T14:55:55.077 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.072+0000 7f6a96c14b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b9f0554a20 0x55b9f056ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:55:55.077 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.072+0000 7f6a96c14b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b9f0559fe0 0x55b9f055b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:55:55.077 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.072+0000 7f6a96c14b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b9f055f830 0x55b9f055bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:55:55.077 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.072+0000 7f6a96c14b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9f055c340 con 0x55b9f055f830 2023-03-09T14:55:55.078 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.072+0000 7f6a96c14b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9f055c4b0 con 0x55b9f0554a20 2023-03-09T14:55:55.078 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.072+0000 7f6a96c14b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9f05c7af0 con 0x55b9f0559fe0 2023-03-09T14:55:55.078 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.072+0000 7f6a96c14b80 10 client.0 _ll_register_callbacks cb 0x55b9f0451520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:55:55.078 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.072+0000 7f6a83fff700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b9f0559fe0 0x55b9f055b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:55:55.078 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.072+0000 7f6a83fff700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b9f0559fe0 0x55b9f055b6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.38:49670/0 (socket says 172.21.15.38:49670) 2023-03-09T14:55:55.078 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.072+0000 7f6a83fff700 1 -- 172.21.15.38:0/3571532896 learned_addr learned my addr 172.21.15.38:0/3571532896 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:55:55.078 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.073+0000 7f6a89028700 1 --2- 172.21.15.38:0/3571532896 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b9f055f830 0x55b9f055bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:55:55.078 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.073+0000 7f6a88827700 1 --2- 172.21.15.38:0/3571532896 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b9f0554a20 0x55b9f056ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:55:55.079 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.073+0000 7f6a89028700 1 -- 172.21.15.38:0/3571532896 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b9f0559fe0 msgr2=0x55b9f055b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:55:55.079 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.073+0000 7f6a89028700 1 --2- 172.21.15.38:0/3571532896 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b9f0559fe0 0x55b9f055b6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:55:55.079 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.073+0000 7f6a89028700 1 -- 172.21.15.38:0/3571532896 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b9f0554a20 msgr2=0x55b9f056ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:55:55.079 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.073+0000 7f6a89028700 1 --2- 172.21.15.38:0/3571532896 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b9f0554a20 0x55b9f056ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:55:55.079 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.073+0000 7f6a89028700 1 -- 172.21.15.38:0/3571532896 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a7801a040 con 0x55b9f055f830 2023-03-09T14:55:55.079 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.074+0000 7f6a89028700 1 --2- 172.21.15.38:0/3571532896 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b9f055f830 0x55b9f055bc40 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f6a7c097600 tx=0x7f6a7c097e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac6dba177df48f5a server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:55:55.079 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.074+0000 7f6a63fff700 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2023-03-09T14:55:55.081 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.074+0000 7f6a63fff700 1 -- 172.21.15.38:0/3571532896 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a7c09fa60 con 0x55b9f055f830 2023-03-09T14:55:55.081 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.074+0000 7f6a63fff700 1 -- 172.21.15.38:0/3571532896 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(240..240 src has 1..240) v4 ==== 9561+0+0 (secure 0 0 0) 0x7f6a7c09f5d0 con 0x55b9f055f830 2023-03-09T14:55:55.081 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.075+0000 7f6a96c14b80 1 -- 172.21.15.38:0/3571532896 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b9f04691b0 con 0x55b9f055f830 2023-03-09T14:55:55.081 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.075+0000 7f6a63fff700 1 -- 172.21.15.38:0/3571532896 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6a7c0c72b0 con 0x55b9f055f830 2023-03-09T14:55:55.082 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.075+0000 7f6a96c14b80 10 client.16239 fetch_fsmap learned FSMap version 36 2023-03-09T14:55:55.082 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.075+0000 7f6a96c14b80 1 -- 172.21.15.38:0/3571532896 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x55b9f0300250 con 0x55b9f055f830 2023-03-09T14:55:55.082 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.075+0000 7f6a63fff700 1 -- 172.21.15.38:0/3571532896 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55b9f04691b0 con 0x55b9f055f830 2023-03-09T14:55:55.082 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.075+0000 7f6a63fff700 1 -- 172.21.15.38:0/3571532896 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 36) v1 ==== 7482+0+0 (secure 0 0 0) 0x7f6a7c0af020 con 0x55b9f055f830 2023-03-09T14:55:55.082 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.075+0000 7f6a96c14b80 10 client.16239 fetch_fsmap finished waiting for FSMap version 36 2023-03-09T14:55:55.082 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.075+0000 7f6a96c14b80 10 client.16239 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T14:55:55.082 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.075+0000 7f6a96c14b80 20 client.16239 populate_metadata read hostname 'smithi038' 2023-03-09T14:55:55.083 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.075+0000 7f6a96c14b80 1 --2- 172.21.15.38:0/3571532896 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55b9f0560140 0x55b9f05cb120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:55:55.083 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.075+0000 7f6a96c14b80 4 client.16239 mds_command: new command op to 24530 tid=0[{"prefix": "scrub status"}] 2023-03-09T14:55:55.083 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.075+0000 7f6a96c14b80 1 -- 172.21.15.38:0/3571532896 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55b9f04531c0 con 0x55b9f0560140 2023-03-09T14:55:55.083 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.076+0000 7f6a88827700 1 --2- 172.21.15.38:0/3571532896 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55b9f0560140 0x55b9f05cb120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:55:55.083 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.076+0000 7f6a63fff700 10 client.16239 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:55:55.083 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.076+0000 7f6a88827700 1 --2- 172.21.15.38:0/3571532896 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55b9f0560140 0x55b9f05cb120 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cf781fb2f0f5280b server_cookie=6732c87dd147dee0 in_seq=0 out_seq=0 2023-03-09T14:55:55.083 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.077+0000 7f6a63fff700 1 -- 172.21.15.38:0/3571532896 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+265 (crc 0 0 0) 0x55b9f04531c0 con 0x55b9f0560140 2023-03-09T14:55:55.083 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.077+0000 7f6a63fff700 10 client.16239 handle_command_reply: tid=0 2023-03-09T14:55:55.084 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.077+0000 7f6a96c14b80 1 client.16239 shutdown 2023-03-09T14:55:55.084 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.077+0000 7f6a96c14b80 1 -- 172.21.15.38:0/3571532896 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b9f055f830 msgr2=0x55b9f055bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:55:55.084 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.077+0000 7f6a96c14b80 1 --2- 172.21.15.38:0/3571532896 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b9f055f830 0x55b9f055bc40 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f6a7c097600 tx=0x7f6a7c097e40 comp rx=0 tx=0).stop 2023-03-09T14:55:55.084 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.077+0000 7f6a96c14b80 1 -- 172.21.15.38:0/3571532896 shutdown_connections 2023-03-09T14:55:55.084 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.077+0000 7f6a96c14b80 1 --2- 172.21.15.38:0/3571532896 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55b9f0560140 0x55b9f05cb120 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:55:55.084 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.077+0000 7f6a96c14b80 1 --2- 172.21.15.38:0/3571532896 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55b9f055f830 0x55b9f055bc40 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:55:55.084 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.077+0000 7f6a63fff700 0 client.16239 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:55:55.085 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.077+0000 7f6a96c14b80 1 --2- 172.21.15.38:0/3571532896 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55b9f0554a20 0x55b9f056ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:55:55.085 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.077+0000 7f6a96c14b80 1 --2- 172.21.15.38:0/3571532896 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55b9f0559fe0 0x55b9f055b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:55:55.085 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.077+0000 7f6a96c14b80 1 -- 172.21.15.38:0/3571532896 >> 172.21.15.38:0/3571532896 conn(0x55b9f05532d0 msgr2=0x55b9f0554df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:55:55.085 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.077+0000 7f6a96c14b80 1 -- 172.21.15.38:0/3571532896 shutdown_connections 2023-03-09T14:55:55.085 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.077+0000 7f6a96c14b80 1 -- 172.21.15.38:0/3571532896 wait complete. 2023-03-09T14:55:55.085 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.077+0000 7f6a96c14b80 20 client.16239 trim_cache size 0 max 16384 2023-03-09T14:55:55.085 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2023-03-09T14:55:55.086 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "scrub active (3 inodes in the stack)", 2023-03-09T14:55:55.086 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": { 2023-03-09T14:55:55.086 INFO:teuthology.orchestra.run.smithi038.stdout: "a14659f4-d7ea-4c5a-b15b-6138af3f909f": { 2023-03-09T14:55:55.086 INFO:teuthology.orchestra.run.smithi038.stdout: "path": "/", 2023-03-09T14:55:55.086 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "a14659f4-d7ea-4c5a-b15b-6138af3f909f", 2023-03-09T14:55:55.086 INFO:teuthology.orchestra.run.smithi038.stdout: "options": "recursive,force" 2023-03-09T14:55:55.086 INFO:teuthology.orchestra.run.smithi038.stdout: } 2023-03-09T14:55:55.087 INFO:teuthology.orchestra.run.smithi038.stdout: } 2023-03-09T14:55:55.087 INFO:teuthology.orchestra.run.smithi038.stdout:} 2023-03-09T14:55:55.087 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.078+0000 7f6a8aa8b700 1 -- 172.21.15.38:0/402439513 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f6a7409f6a0 msgr2=0x7f6a740a1b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:55:55.088 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.078+0000 7f6a8aa8b700 1 --2- 172.21.15.38:0/402439513 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f6a7409f6a0 0x7f6a740a1b50 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f6a78005cd0 tx=0x7f6a78005c40 comp rx=0 tx=0).stop 2023-03-09T14:55:55.088 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.078+0000 7f6a8aa8b700 1 -- 172.21.15.38:0/402439513 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6a841046f0 msgr2=0x7f6a84108550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:55:55.088 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.078+0000 7f6a8aa8b700 1 --2- 172.21.15.38:0/402439513 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6a841046f0 0x7f6a84108550 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6a7c00ef30 tx=0x7f6a7c00c3a0 comp rx=0 tx=0).stop 2023-03-09T14:55:55.088 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.078+0000 7f6a8aa8b700 1 -- 172.21.15.38:0/402439513 shutdown_connections 2023-03-09T14:55:55.088 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.078+0000 7f6a8aa8b700 1 --2- 172.21.15.38:0/402439513 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f6a7409f6a0 0x7f6a740a1b50 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:55:55.088 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.078+0000 7f6a8aa8b700 1 --2- 172.21.15.38:0/402439513 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f6a841046f0 0x7f6a84108550 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:55:55.088 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.078+0000 7f6a8aa8b700 1 --2- 172.21.15.38:0/402439513 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f6a840ff0a0 0x7f6a8410b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:55:55.089 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.078+0000 7f6a8aa8b700 1 --2- 172.21.15.38:0/402439513 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f6a840ffa50 0x7f6a84108010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:55:55.089 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.078+0000 7f6a8aa8b700 1 -- 172.21.15.38:0/402439513 >> 172.21.15.38:0/402439513 conn(0x7f6a840755f0 msgr2=0x7f6a840fcf10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:55:55.089 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.078+0000 7f6a8aa8b700 1 -- 172.21.15.38:0/402439513 shutdown_connections 2023-03-09T14:55:55.089 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:55:55.078+0000 7f6a8aa8b700 1 -- 172.21.15.38:0/402439513 wait complete. 2023-03-09T14:55:55.110 INFO:tasks.cephfs.filesystem:scrub status for tag:a14659f4-d7ea-4c5a-b15b-6138af3f909f - {'path': '/', 'tag': 'a14659f4-d7ea-4c5a-b15b-6138af3f909f', 'options': 'recursive,force'} 2023-03-09T14:55:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:55:55 smithi122 ceph-mon[93869]: pgmap v443: 129 pgs: 129 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 26 MiB/s wr, 2.96k op/s 2023-03-09T14:55:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:55:55 smithi181 ceph-mon[94189]: pgmap v443: 129 pgs: 129 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 26 MiB/s wr, 2.96k op/s 2023-03-09T14:55:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:55:55 smithi038 ceph-mon[92422]: pgmap v443: 129 pgs: 129 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 26 MiB/s wr, 2.96k op/s 2023-03-09T14:55:58.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:55:57 smithi122 ceph-mon[93869]: pgmap v444: 129 pgs: 129 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 23 MiB/s wr, 2.58k op/s 2023-03-09T14:55:58.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:55:57 smithi181 ceph-mon[94189]: pgmap v444: 129 pgs: 129 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 23 MiB/s wr, 2.58k op/s 2023-03-09T14:55:58.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:55:57 smithi038 ceph-mon[92422]: pgmap v444: 129 pgs: 129 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 23 MiB/s wr, 2.58k op/s 2023-03-09T14:55:58.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 330.0 s, 770.2 tps, lat 130.099 ms stddev 272.958 2023-03-09T14:56:00.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:55:59 smithi122 ceph-mon[93869]: pgmap v445: 129 pgs: 129 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 28 MiB/s wr, 3.19k op/s 2023-03-09T14:56:00.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:55:59 smithi181 ceph-mon[94189]: pgmap v445: 129 pgs: 129 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 28 MiB/s wr, 3.19k op/s 2023-03-09T14:56:00.254 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:55:59 smithi038 ceph-mon[92422]: pgmap v445: 129 pgs: 129 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 28 MiB/s wr, 3.19k op/s 2023-03-09T14:56:02.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:56:01 smithi122 ceph-mon[93869]: pgmap v446: 129 pgs: 129 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 22 MiB/s wr, 2.49k op/s 2023-03-09T14:56:02.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:56:01 smithi181 ceph-mon[94189]: pgmap v446: 129 pgs: 129 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 22 MiB/s wr, 2.49k op/s 2023-03-09T14:56:02.376 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:56:01 smithi038 ceph-mon[92422]: pgmap v446: 129 pgs: 129 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 22 MiB/s wr, 2.49k op/s 2023-03-09T14:56:03.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 335.0 s, 510.0 tps, lat 169.440 ms stddev 300.312 2023-03-09T14:56:04.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:56:03 smithi122 ceph-mon[93869]: pgmap v447: 129 pgs: 129 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 21 MiB/s wr, 2.35k op/s 2023-03-09T14:56:04.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:56:03 smithi181 ceph-mon[94189]: pgmap v447: 129 pgs: 129 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 21 MiB/s wr, 2.35k op/s 2023-03-09T14:56:04.259 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:56:03 smithi038 ceph-mon[92422]: pgmap v447: 129 pgs: 129 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 21 MiB/s wr, 2.35k op/s 2023-03-09T14:56:06.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:56:05 smithi122 ceph-mon[93869]: pgmap v448: 129 pgs: 129 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 24 MiB/s wr, 2.90k op/s 2023-03-09T14:56:06.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:56:05 smithi181 ceph-mon[94189]: pgmap v448: 129 pgs: 129 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 24 MiB/s wr, 2.90k op/s 2023-03-09T14:56:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:56:05 smithi038 ceph-mon[92422]: pgmap v448: 129 pgs: 129 active+clean; 13 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 24 MiB/s wr, 2.90k op/s 2023-03-09T14:56:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:56:07 smithi122 ceph-mon[93869]: pgmap v449: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 24 MiB/s wr, 2.71k op/s 2023-03-09T14:56:08.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:56:07 smithi181 ceph-mon[94189]: pgmap v449: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 24 MiB/s wr, 2.71k op/s 2023-03-09T14:56:08.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:56:07 smithi038 ceph-mon[92422]: pgmap v449: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 24 MiB/s wr, 2.71k op/s 2023-03-09T14:56:08.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 340.0 s, 381.4 tps, lat 280.368 ms stddev 450.742 2023-03-09T14:56:10.231 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:56:09 smithi122 ceph-mon[93869]: pgmap v450: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 25 MiB/s wr, 2.87k op/s 2023-03-09T14:56:10.232 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:56:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T14:56:10.232 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:56:09 smithi122 ceph-mon[93869]: osdmap e241: 12 total, 12 up, 12 in 2023-03-09T14:56:10.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:56:09 smithi181 ceph-mon[94189]: pgmap v450: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 25 MiB/s wr, 2.87k op/s 2023-03-09T14:56:10.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:56:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T14:56:10.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:56:09 smithi181 ceph-mon[94189]: osdmap e241: 12 total, 12 up, 12 in 2023-03-09T14:56:10.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:56:09 smithi038 ceph-mon[92422]: pgmap v450: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 25 MiB/s wr, 2.87k op/s 2023-03-09T14:56:10.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:56:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T14:56:10.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:56:09 smithi038 ceph-mon[92422]: osdmap e241: 12 total, 12 up, 12 in 2023-03-09T14:56:12.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:56:11 smithi122 ceph-mon[93869]: pgmap v452: 129 pgs: 6 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 104 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 19 MiB/s wr, 2.29k op/s 2023-03-09T14:56:12.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:56:11 smithi181 ceph-mon[94189]: pgmap v452: 129 pgs: 6 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 104 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 19 MiB/s wr, 2.29k op/s 2023-03-09T14:56:12.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:56:11 smithi038 ceph-mon[92422]: pgmap v452: 129 pgs: 6 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 104 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 19 MiB/s wr, 2.29k op/s 2023-03-09T14:56:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 345.0 s, 199.7 tps, lat 516.899 ms stddev 362.811 2023-03-09T14:56:14.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:56:13 smithi122 ceph-mon[93869]: pgmap v453: 129 pgs: 6 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 104 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 17 MiB/s wr, 2.12k op/s 2023-03-09T14:56:14.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:56:13 smithi181 ceph-mon[94189]: pgmap v453: 129 pgs: 6 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 104 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 17 MiB/s wr, 2.12k op/s 2023-03-09T14:56:14.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:56:13 smithi038 ceph-mon[92422]: pgmap v453: 129 pgs: 6 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 104 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 17 MiB/s wr, 2.12k op/s 2023-03-09T14:56:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:56:15 smithi122 ceph-mon[93869]: pgmap v454: 129 pgs: 4 active+clean+snaptrim, 15 active+clean+snaptrim_wait, 110 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 12 MiB/s wr, 1.16k op/s 2023-03-09T14:56:16.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:56:15 smithi181 ceph-mon[94189]: pgmap v454: 129 pgs: 4 active+clean+snaptrim, 15 active+clean+snaptrim_wait, 110 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 12 MiB/s wr, 1.16k op/s 2023-03-09T14:56:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:56:15 smithi038 ceph-mon[92422]: pgmap v454: 129 pgs: 4 active+clean+snaptrim, 15 active+clean+snaptrim_wait, 110 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 12 MiB/s wr, 1.16k op/s 2023-03-09T14:56:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:56:17 smithi122 ceph-mon[93869]: pgmap v455: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 10 MiB/s wr, 1.02k op/s 2023-03-09T14:56:18.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:56:17 smithi122 ceph-mon[93869]: osdmap e242: 12 total, 12 up, 12 in 2023-03-09T14:56:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:56:17 smithi181 ceph-mon[94189]: pgmap v455: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 10 MiB/s wr, 1.02k op/s 2023-03-09T14:56:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:56:17 smithi181 ceph-mon[94189]: osdmap e242: 12 total, 12 up, 12 in 2023-03-09T14:56:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:56:17 smithi038 ceph-mon[92422]: pgmap v455: 129 pgs: 2 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 120 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 MiB/s rd, 10 MiB/s wr, 1.02k op/s 2023-03-09T14:56:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:56:17 smithi038 ceph-mon[92422]: osdmap e242: 12 total, 12 up, 12 in 2023-03-09T14:56:18.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 350.0 s, 527.6 tps, lat 196.152 ms stddev 473.506 2023-03-09T14:56:20.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:56:19 smithi122 ceph-mon[93869]: pgmap v457: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 11 MiB/s wr, 904 op/s 2023-03-09T14:56:20.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:56:19 smithi181 ceph-mon[94189]: pgmap v457: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 11 MiB/s wr, 904 op/s 2023-03-09T14:56:20.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:56:19 smithi038 ceph-mon[92422]: pgmap v457: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 11 MiB/s wr, 904 op/s 2023-03-09T14:56:22.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:56:21 smithi122 ceph-mon[93869]: pgmap v458: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 12 MiB/s wr, 1.02k op/s 2023-03-09T14:56:22.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:56:21 smithi181 ceph-mon[94189]: pgmap v458: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 12 MiB/s wr, 1.02k op/s 2023-03-09T14:56:22.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:56:21 smithi038 ceph-mon[92422]: pgmap v458: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 12 MiB/s wr, 1.02k op/s 2023-03-09T14:56:23.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 355.0 s, 735.0 tps, lat 129.427 ms stddev 371.063 2023-03-09T14:56:24.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:56:23 smithi122 ceph-mon[93869]: osdmap e243: 12 total, 12 up, 12 in 2023-03-09T14:56:24.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:56:23 smithi122 ceph-mon[93869]: pgmap v460: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 10 MiB/s wr, 1.01k op/s 2023-03-09T14:56:24.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:56:23 smithi181 ceph-mon[94189]: osdmap e243: 12 total, 12 up, 12 in 2023-03-09T14:56:24.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:56:23 smithi181 ceph-mon[94189]: pgmap v460: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 10 MiB/s wr, 1.01k op/s 2023-03-09T14:56:24.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:56:23 smithi038 ceph-mon[92422]: osdmap e243: 12 total, 12 up, 12 in 2023-03-09T14:56:24.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:56:23 smithi038 ceph-mon[92422]: pgmap v460: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 10 MiB/s wr, 1.01k op/s 2023-03-09T14:56:25.111 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-03-09T14:56:25.410 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.408+0000 7fe74782c700 1 -- 172.21.15.38:0/102695454 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe7400ff570 msgr2=0x7fe7400ff940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:56:25.410 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.408+0000 7fe74782c700 1 --2- 172.21.15.38:0/102695454 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe7400ff570 0x7fe7400ff940 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe73c01a9a0 tx=0x7fe73c01acb0 comp rx=0 tx=0).stop 2023-03-09T14:56:25.411 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.409+0000 7fe74782c700 1 -- 172.21.15.38:0/102695454 shutdown_connections 2023-03-09T14:56:25.411 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.409+0000 7fe74782c700 1 --2- 172.21.15.38:0/102695454 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe740104c00 0x7fe74010b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:56:25.411 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.409+0000 7fe74782c700 1 --2- 172.21.15.38:0/102695454 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe7400ff570 0x7fe7400ff940 secure :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe73c01a9a0 tx=0x7fe73c01acb0 comp rx=0 tx=0).stop 2023-03-09T14:56:25.411 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.409+0000 7fe74782c700 1 --2- 172.21.15.38:0/102695454 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe7400fff10 0x7fe7401044d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:56:25.411 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.409+0000 7fe74782c700 1 -- 172.21.15.38:0/102695454 >> 172.21.15.38:0/102695454 conn(0x7fe7400748f0 msgr2=0x7fe740074cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:56:25.414 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.413+0000 7fe74782c700 1 -- 172.21.15.38:0/102695454 shutdown_connections 2023-03-09T14:56:25.414 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.413+0000 7fe74782c700 1 -- 172.21.15.38:0/102695454 wait complete. 2023-03-09T14:56:25.415 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.414+0000 7fe74782c700 1 Processor -- start 2023-03-09T14:56:25.415 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.414+0000 7fe74782c700 1 -- start start 2023-03-09T14:56:25.416 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.414+0000 7fe74782c700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe7400fff10 0x7fe740107690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:56:25.416 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.414+0000 7fe74782c700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe740104c00 0x7fe740107bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:56:25.416 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.414+0000 7fe74782c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe740108110 0x7fe74019c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:56:25.417 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.414+0000 7fe74782c700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe740109640 con 0x7fe740108110 2023-03-09T14:56:25.417 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.414+0000 7fe74782c700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe74019c710 con 0x7fe740104c00 2023-03-09T14:56:25.417 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.414+0000 7fe74782c700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe74019c8d0 con 0x7fe7400fff10 2023-03-09T14:56:25.417 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.414+0000 7fe745dc9700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe740108110 0x7fe74019c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:56:25.417 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.414+0000 7fe7455c8700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe7400fff10 0x7fe740107690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:56:25.417 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.414+0000 7fe745dc9700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe740108110 0x7fe74019c1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:41988/0 (socket says 172.21.15.38:41988) 2023-03-09T14:56:25.417 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.414+0000 7fe745dc9700 1 -- 172.21.15.38:0/2210105790 learned_addr learned my addr 172.21.15.38:0/2210105790 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:56:25.418 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.415+0000 7fe7455c8700 1 -- 172.21.15.38:0/2210105790 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe740104c00 msgr2=0x7fe740107bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:56:25.418 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.415+0000 7fe7455c8700 1 --2- 172.21.15.38:0/2210105790 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe740104c00 0x7fe740107bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:56:25.419 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.415+0000 7fe7455c8700 1 -- 172.21.15.38:0/2210105790 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe740108110 msgr2=0x7fe74019c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:56:25.419 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.415+0000 7fe7455c8700 1 --2- 172.21.15.38:0/2210105790 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe740108110 0x7fe74019c1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:56:25.419 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.415+0000 7fe7455c8700 1 -- 172.21.15.38:0/2210105790 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe73c01a600 con 0x7fe7400fff10 2023-03-09T14:56:25.419 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.415+0000 7fe7455c8700 1 --2- 172.21.15.38:0/2210105790 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe7400fff10 0x7fe740107690 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe73c019000 tx=0x7fe73c01a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:56:25.419 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.415+0000 7fe7367fc700 1 -- 172.21.15.38:0/2210105790 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe73c02d070 con 0x7fe7400fff10 2023-03-09T14:56:25.419 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.415+0000 7fe7367fc700 1 -- 172.21.15.38:0/2210105790 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe73c0056b0 con 0x7fe7400fff10 2023-03-09T14:56:25.419 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.415+0000 7fe74782c700 1 -- 172.21.15.38:0/2210105790 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe74006ca30 con 0x7fe7400fff10 2023-03-09T14:56:25.419 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.415+0000 7fe7367fc700 1 -- 172.21.15.38:0/2210105790 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe73c003fd0 con 0x7fe7400fff10 2023-03-09T14:56:25.420 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.415+0000 7fe74782c700 1 -- 172.21.15.38:0/2210105790 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe74006c8a0 con 0x7fe7400fff10 2023-03-09T14:56:25.420 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.419+0000 7fe7367fc700 1 -- 172.21.15.38:0/2210105790 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 25) v1 ==== 133295+0+0 (secure 0 0 0) 0x7fe73c038020 con 0x7fe7400fff10 2023-03-09T14:56:25.421 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.419+0000 7fe7367fc700 1 --2- 172.21.15.38:0/2210105790 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fe72c09f520 0x7fe72c0a19d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:56:25.421 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.420+0000 7fe7367fc700 1 -- 172.21.15.38:0/2210105790 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 9561+0+0 (secure 0 0 0) 0x7fe73c0dc090 con 0x7fe7400fff10 2023-03-09T14:56:25.439 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.420+0000 7fe744dc7700 1 --2- 172.21.15.38:0/2210105790 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fe72c09f520 0x7fe72c0a19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:56:25.440 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.438+0000 7fe744dc7700 1 --2- 172.21.15.38:0/2210105790 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fe72c09f520 0x7fe72c0a19d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fe740073110 tx=0x7fe730006cd0 comp rx=0 tx=0).ready entity=mgr.15558 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:56:25.441 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.439+0000 7fe7539b5b80 1 Processor -- start 2023-03-09T14:56:25.444 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.440+0000 7fe7539b5b80 1 -- start start 2023-03-09T14:56:25.444 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.440+0000 7fe7539b5b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642ab34c040 0x5642ab34c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:56:25.445 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.440+0000 7fe7539b5b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5642ab34c950 0x5642ab3462b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:56:25.445 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.440+0000 7fe7539b5b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5642ab3467f0 0x5642ab346c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:56:25.445 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.440+0000 7fe7539b5b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x5642ab227da0 con 0x5642ab34c040 2023-03-09T14:56:25.445 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.440+0000 7fe7539b5b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x5642ab296780 con 0x5642ab34c950 2023-03-09T14:56:25.446 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.440+0000 7fe7539b5b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x5642ab28fcb0 con 0x5642ab3467f0 2023-03-09T14:56:25.446 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.440+0000 7fe744dc7700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5642ab34c950 0x5642ab3462b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:56:25.446 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.440+0000 7fe744dc7700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5642ab34c950 0x5642ab3462b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.38:51184/0 (socket says 172.21.15.38:51184) 2023-03-09T14:56:25.447 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.440+0000 7fe744dc7700 1 -- 172.21.15.38:0/2097057890 learned_addr learned my addr 172.21.15.38:0/2097057890 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:56:25.447 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.440+0000 7fe745dc9700 1 --2- 172.21.15.38:0/2097057890 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5642ab3467f0 0x5642ab346c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:56:25.447 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.441+0000 7fe7455c8700 1 --2- 172.21.15.38:0/2097057890 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642ab34c040 0x5642ab34c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:56:25.448 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.441+0000 7fe744dc7700 1 -- 172.21.15.38:0/2097057890 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5642ab3467f0 msgr2=0x5642ab346c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:56:25.448 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.441+0000 7fe744dc7700 1 --2- 172.21.15.38:0/2097057890 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5642ab3467f0 0x5642ab346c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:56:25.448 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.441+0000 7fe744dc7700 1 -- 172.21.15.38:0/2097057890 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642ab34c040 msgr2=0x5642ab34c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:56:25.449 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.441+0000 7fe744dc7700 1 --2- 172.21.15.38:0/2097057890 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642ab34c040 0x5642ab34c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:56:25.449 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.441+0000 7fe744dc7700 1 -- 172.21.15.38:0/2097057890 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5642ab256190 con 0x5642ab34c950 2023-03-09T14:56:25.449 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.441+0000 7fe7455c8700 1 --2- 172.21.15.38:0/2097057890 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642ab34c040 0x5642ab34c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:56:25.450 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.441+0000 7fe744dc7700 1 --2- 172.21.15.38:0/2097057890 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5642ab34c950 0x5642ab3462b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fe730015920 tx=0x7fe730009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad8cd2d93a9cbcef server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:56:25.450 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.442+0000 7fe72b7fe700 1 -- 172.21.15.38:0/2097057890 <== mon.2 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe73001a070 con 0x5642ab34c950 2023-03-09T14:56:25.450 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.442+0000 7fe72b7fe700 1 -- 172.21.15.38:0/2097057890 <== mon.2 v2:172.21.15.122:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe730012950 con 0x5642ab34c950 2023-03-09T14:56:25.451 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.442+0000 7fe72b7fe700 1 -- 172.21.15.38:0/2097057890 <== mon.2 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe730013980 con 0x5642ab34c950 2023-03-09T14:56:25.455 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.443+0000 7fe7539b5b80 1 -- 172.21.15.38:0/2097057890 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5642ab34c950 msgr2=0x5642ab3462b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:56:25.456 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.443+0000 7fe7539b5b80 1 --2- 172.21.15.38:0/2097057890 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5642ab34c950 0x5642ab3462b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fe730015920 tx=0x7fe730009f90 comp rx=0 tx=0).stop 2023-03-09T14:56:25.456 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.443+0000 7fe7539b5b80 1 -- 172.21.15.38:0/2097057890 shutdown_connections 2023-03-09T14:56:25.456 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.443+0000 7fe7539b5b80 1 --2- 172.21.15.38:0/2097057890 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642ab34c040 0x5642ab34c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:56:25.456 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.443+0000 7fe7539b5b80 1 --2- 172.21.15.38:0/2097057890 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5642ab3467f0 0x5642ab346c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:56:25.456 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.443+0000 7fe7539b5b80 1 --2- 172.21.15.38:0/2097057890 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5642ab34c950 0x5642ab3462b0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:56:25.456 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.443+0000 7fe7539b5b80 1 -- 172.21.15.38:0/2097057890 >> 172.21.15.38:0/2097057890 conn(0x5642ab338910 msgr2=0x5642ab338ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:56:25.457 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.443+0000 7fe7539b5b80 1 -- 172.21.15.38:0/2097057890 shutdown_connections 2023-03-09T14:56:25.457 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.444+0000 7fe7539b5b80 1 -- 172.21.15.38:0/2097057890 wait complete. 2023-03-09T14:56:25.457 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.444+0000 7fe7539b5b80 1 -- start start 2023-03-09T14:56:25.457 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.444+0000 7fe7539b5b80 1 Processor -- start 2023-03-09T14:56:25.457 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.445+0000 7fe7539b5b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5642ab3467f0 0x5642ab352980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:56:25.457 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.445+0000 7fe7539b5b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5642ab34c040 0x5642ab34fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:56:25.458 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.445+0000 7fe7539b5b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642ab34c950 0x5642ab350340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:56:25.458 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.445+0000 7fe7539b5b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x5642ab113920 con 0x5642ab34c950 2023-03-09T14:56:25.458 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.445+0000 7fe7539b5b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x5642ab350ab0 con 0x5642ab34c040 2023-03-09T14:56:25.458 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.445+0000 7fe7539b5b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x5642ab35c480 con 0x5642ab3467f0 2023-03-09T14:56:25.458 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.445+0000 7fe7539b5b80 10 client.0 _ll_register_callbacks cb 0x5642ab235520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:56:25.458 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.445+0000 7fe7455c8700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5642ab3467f0 0x5642ab352980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:56:25.458 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.445+0000 7fe7455c8700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5642ab3467f0 0x5642ab352980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.38:49698/0 (socket says 172.21.15.38:49698) 2023-03-09T14:56:25.459 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.445+0000 7fe7455c8700 1 -- 172.21.15.38:0/2842040008 learned_addr learned my addr 172.21.15.38:0/2842040008 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:56:25.459 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.445+0000 7fe744dc7700 1 --2- 172.21.15.38:0/2842040008 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5642ab34c040 0x5642ab34fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:56:25.459 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.445+0000 7fe745dc9700 1 --2- 172.21.15.38:0/2842040008 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642ab34c950 0x5642ab350340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:56:25.459 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.445+0000 7fe744dc7700 1 -- 172.21.15.38:0/2842040008 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5642ab3467f0 msgr2=0x5642ab352980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:56:25.459 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.445+0000 7fe744dc7700 1 --2- 172.21.15.38:0/2842040008 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5642ab3467f0 0x5642ab352980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:56:25.459 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.445+0000 7fe744dc7700 1 -- 172.21.15.38:0/2842040008 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642ab34c950 msgr2=0x5642ab350340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:56:25.459 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.445+0000 7fe744dc7700 1 --2- 172.21.15.38:0/2842040008 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642ab34c950 0x5642ab350340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:56:25.460 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.445+0000 7fe744dc7700 1 -- 172.21.15.38:0/2842040008 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7300155d0 con 0x5642ab34c040 2023-03-09T14:56:25.460 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.446+0000 7fe744dc7700 1 --2- 172.21.15.38:0/2842040008 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5642ab34c040 0x5642ab34fc00 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe730015920 tx=0x7fe730012b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8a5256b61e59e2a server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:56:25.460 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.446+0000 7fe728ff9700 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2023-03-09T14:56:25.460 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.446+0000 7fe745dc9700 1 --2- 172.21.15.38:0/2842040008 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642ab34c950 0x5642ab350340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:56:25.460 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.446+0000 7fe728ff9700 1 -- 172.21.15.38:0/2842040008 <== mon.2 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe73001a040 con 0x5642ab34c040 2023-03-09T14:56:25.460 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.446+0000 7fe7539b5b80 1 -- 172.21.15.38:0/2842040008 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5642ab25d190 con 0x5642ab34c040 2023-03-09T14:56:25.460 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.446+0000 7fe728ff9700 1 -- 172.21.15.38:0/2842040008 <== mon.2 v2:172.21.15.122:3300/0 2 ==== osd_map(243..243 src has 1..243) v4 ==== 9561+0+0 (secure 0 0 0) 0x7fe7300136c0 con 0x5642ab34c040 2023-03-09T14:56:25.461 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.446+0000 7fe728ff9700 1 -- 172.21.15.38:0/2842040008 <== mon.2 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe7300152e0 con 0x5642ab34c040 2023-03-09T14:56:25.461 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.446+0000 7fe7539b5b80 10 client.25886 fetch_fsmap learned FSMap version 37 2023-03-09T14:56:25.461 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.446+0000 7fe7539b5b80 1 -- 172.21.15.38:0/2842040008 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) v3 -- 0x5642ab0a6c60 con 0x5642ab34c040 2023-03-09T14:56:25.461 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.447+0000 7fe728ff9700 1 -- 172.21.15.38:0/2842040008 <== mon.2 v2:172.21.15.122:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fe730036080 con 0x5642ab34c040 2023-03-09T14:56:25.461 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.447+0000 7fe728ff9700 1 -- 172.21.15.38:0/2842040008 <== mon.2 v2:172.21.15.122:3300/0 5 ==== fsmap(e 37) v1 ==== 7478+0+0 (secure 0 0 0) 0x7fe73001ea00 con 0x5642ab34c040 2023-03-09T14:56:25.461 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.447+0000 7fe7539b5b80 10 client.25886 fetch_fsmap finished waiting for FSMap version 37 2023-03-09T14:56:25.461 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.447+0000 7fe7539b5b80 10 client.25886 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T14:56:25.462 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.447+0000 7fe7539b5b80 20 client.25886 populate_metadata read hostname 'smithi038' 2023-03-09T14:56:25.462 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.447+0000 7fe7539b5b80 1 --2- 172.21.15.38:0/2842040008 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5642ab360470 0x5642ab362880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:56:25.462 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.447+0000 7fe7539b5b80 4 client.25886 mds_command: new command op to 24530 tid=0[{"prefix": "get_command_descriptions"}] 2023-03-09T14:56:25.462 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.447+0000 7fe7539b5b80 1 -- 172.21.15.38:0/2842040008 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5642ab113f60 con 0x5642ab360470 2023-03-09T14:56:25.462 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.447+0000 7fe7455c8700 1 --2- 172.21.15.38:0/2842040008 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5642ab360470 0x5642ab362880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:56:25.462 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.448+0000 7fe728ff9700 10 client.25886 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:56:25.463 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.448+0000 7fe7455c8700 1 --2- 172.21.15.38:0/2842040008 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5642ab360470 0x5642ab362880 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9cc6e95b9aebf4f4 server_cookie=7a3c20d29a2f3b9f in_seq=0 out_seq=0 2023-03-09T14:56:25.463 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.448+0000 7fe728ff9700 1 -- 172.21.15.38:0/2842040008 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18278 (crc 0 0 0) 0x5642ab113f60 con 0x5642ab360470 2023-03-09T14:56:25.463 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.448+0000 7fe728ff9700 10 client.25886 handle_command_reply: tid=0 2023-03-09T14:56:25.463 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.448+0000 7fe7539b5b80 1 client.25886 shutdown 2023-03-09T14:56:25.463 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.449+0000 7fe7539b5b80 1 -- 172.21.15.38:0/2842040008 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5642ab34c040 msgr2=0x5642ab34fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:56:25.463 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.449+0000 7fe7539b5b80 1 --2- 172.21.15.38:0/2842040008 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5642ab34c040 0x5642ab34fc00 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe730015920 tx=0x7fe730012b60 comp rx=0 tx=0).stop 2023-03-09T14:56:25.463 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.449+0000 7fe7539b5b80 1 -- 172.21.15.38:0/2842040008 shutdown_connections 2023-03-09T14:56:25.464 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.449+0000 7fe7539b5b80 1 --2- 172.21.15.38:0/2842040008 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5642ab360470 0x5642ab362880 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:56:25.464 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.449+0000 7fe7539b5b80 1 --2- 172.21.15.38:0/2842040008 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642ab34c950 0x5642ab350340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:56:25.464 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.449+0000 7fe728ff9700 0 client.25886 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:56:25.464 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.449+0000 7fe7539b5b80 1 --2- 172.21.15.38:0/2842040008 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5642ab3467f0 0x5642ab352980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:56:25.464 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.449+0000 7fe7539b5b80 1 --2- 172.21.15.38:0/2842040008 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5642ab34c040 0x5642ab34fc00 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:56:25.464 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.449+0000 7fe7539b5b80 1 -- 172.21.15.38:0/2842040008 >> 172.21.15.38:0/2842040008 conn(0x5642ab338910 msgr2=0x5642ab337c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:56:25.465 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.449+0000 7fe7539b5b80 1 -- 172.21.15.38:0/2842040008 shutdown_connections 2023-03-09T14:56:25.465 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.449+0000 7fe7539b5b80 1 -- 172.21.15.38:0/2842040008 wait complete. 2023-03-09T14:56:25.465 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.449+0000 7fe7539b5b80 20 client.25886 trim_cache size 0 max 16384 2023-03-09T14:56:25.479 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.476+0000 7fe7539b5b80 1 Processor -- start 2023-03-09T14:56:25.480 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.476+0000 7fe7539b5b80 1 -- start start 2023-03-09T14:56:25.480 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.476+0000 7fe7539b5b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5642ab3467f0 0x5642ab3573a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:56:25.480 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.476+0000 7fe7539b5b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5642ab34c040 0x5642ab34f400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:56:25.480 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.476+0000 7fe7539b5b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642ab34c950 0x5642ab337c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:56:25.480 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.476+0000 7fe7539b5b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x5642ab338690 con 0x5642ab34c950 2023-03-09T14:56:25.481 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.476+0000 7fe7539b5b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x5642ab347500 con 0x5642ab3467f0 2023-03-09T14:56:25.481 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.476+0000 7fe7539b5b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x5642ab3396d0 con 0x5642ab34c040 2023-03-09T14:56:25.481 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.477+0000 7fe745dc9700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642ab34c950 0x5642ab337c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:56:25.481 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.477+0000 7fe744dc7700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5642ab34c040 0x5642ab34f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:56:25.481 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.477+0000 7fe745dc9700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642ab34c950 0x5642ab337c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42012/0 (socket says 172.21.15.38:42012) 2023-03-09T14:56:25.482 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.477+0000 7fe745dc9700 1 -- 172.21.15.38:0/4013112100 learned_addr learned my addr 172.21.15.38:0/4013112100 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:56:25.482 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.477+0000 7fe7455c8700 1 --2- 172.21.15.38:0/4013112100 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5642ab3467f0 0x5642ab3573a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:56:25.482 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.477+0000 7fe744dc7700 1 -- 172.21.15.38:0/4013112100 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5642ab3467f0 msgr2=0x5642ab3573a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:56:25.482 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.477+0000 7fe744dc7700 1 --2- 172.21.15.38:0/4013112100 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5642ab3467f0 0x5642ab3573a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:56:25.482 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.477+0000 7fe744dc7700 1 -- 172.21.15.38:0/4013112100 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642ab34c950 msgr2=0x5642ab337c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:56:25.483 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.477+0000 7fe744dc7700 1 --2- 172.21.15.38:0/4013112100 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642ab34c950 0x5642ab337c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:56:25.483 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.477+0000 7fe744dc7700 1 -- 172.21.15.38:0/4013112100 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5642ab0bdde0 con 0x5642ab34c040 2023-03-09T14:56:25.483 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.477+0000 7fe7455c8700 1 --2- 172.21.15.38:0/4013112100 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5642ab3467f0 0x5642ab3573a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:56:25.483 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.477+0000 7fe744dc7700 1 --2- 172.21.15.38:0/4013112100 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5642ab34c040 0x5642ab34f400 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe73001ede0 tx=0x7fe73000a040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3142eb267bad6275 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:56:25.483 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.478+0000 7fe72bfff700 1 -- 172.21.15.38:0/4013112100 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe73001a040 con 0x5642ab34c040 2023-03-09T14:56:25.487 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.478+0000 7fe72bfff700 1 -- 172.21.15.38:0/4013112100 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5642ab25d190 con 0x5642ab34c040 2023-03-09T14:56:25.487 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.478+0000 7fe72bfff700 1 -- 172.21.15.38:0/4013112100 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe730013680 con 0x5642ab34c040 2023-03-09T14:56:25.487 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.479+0000 7fe7539b5b80 1 -- 172.21.15.38:0/4013112100 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5642ab34c040 msgr2=0x5642ab34f400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:56:25.487 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.479+0000 7fe7539b5b80 1 --2- 172.21.15.38:0/4013112100 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5642ab34c040 0x5642ab34f400 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fe73001ede0 tx=0x7fe73000a040 comp rx=0 tx=0).stop 2023-03-09T14:56:25.488 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.479+0000 7fe7539b5b80 1 -- 172.21.15.38:0/4013112100 shutdown_connections 2023-03-09T14:56:25.488 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.479+0000 7fe7539b5b80 1 --2- 172.21.15.38:0/4013112100 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642ab34c950 0x5642ab337c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:56:25.488 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.479+0000 7fe7539b5b80 1 --2- 172.21.15.38:0/4013112100 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5642ab3467f0 0x5642ab3573a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:56:25.488 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.479+0000 7fe7539b5b80 1 --2- 172.21.15.38:0/4013112100 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5642ab34c040 0x5642ab34f400 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:56:25.488 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.479+0000 7fe7539b5b80 1 -- 172.21.15.38:0/4013112100 >> 172.21.15.38:0/4013112100 conn(0x5642ab338910 msgr2=0x5642ab346be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:56:25.488 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.479+0000 7fe7539b5b80 1 -- 172.21.15.38:0/4013112100 shutdown_connections 2023-03-09T14:56:25.488 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.479+0000 7fe7539b5b80 1 -- 172.21.15.38:0/4013112100 wait complete. 2023-03-09T14:56:25.489 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.479+0000 7fe7539b5b80 1 -- start start 2023-03-09T14:56:25.489 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.479+0000 7fe7539b5b80 1 Processor -- start 2023-03-09T14:56:25.489 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.480+0000 7fe7539b5b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5642ab3467f0 0x5642ab3570d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:56:25.489 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.480+0000 7fe7539b5b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5642ab34c040 0x5642ab3553c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:56:25.489 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.480+0000 7fe7539b5b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642ab34c950 0x5642ab355900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:56:25.490 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.480+0000 7fe7539b5b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x5642ab355e40 con 0x5642ab34c950 2023-03-09T14:56:25.490 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.480+0000 7fe7539b5b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x5642ab355fb0 con 0x5642ab34c040 2023-03-09T14:56:25.490 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.480+0000 7fe7539b5b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x5642ab356170 con 0x5642ab3467f0 2023-03-09T14:56:25.490 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.480+0000 7fe7539b5b80 10 client.0 _ll_register_callbacks cb 0x5642ab235520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:56:25.490 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.480+0000 7fe744dc7700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5642ab34c040 0x5642ab3553c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:56:25.490 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.480+0000 7fe7455c8700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5642ab3467f0 0x5642ab3570d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:56:25.490 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.480+0000 7fe744dc7700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5642ab34c040 0x5642ab3553c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.38:51206/0 (socket says 172.21.15.38:51206) 2023-03-09T14:56:25.491 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.480+0000 7fe744dc7700 1 -- 172.21.15.38:0/2224898611 learned_addr learned my addr 172.21.15.38:0/2224898611 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:56:25.491 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.480+0000 7fe745dc9700 1 --2- 172.21.15.38:0/2224898611 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642ab34c950 0x5642ab355900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:56:25.491 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.480+0000 7fe744dc7700 1 -- 172.21.15.38:0/2224898611 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5642ab3467f0 msgr2=0x5642ab3570d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:56:25.491 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.480+0000 7fe744dc7700 1 --2- 172.21.15.38:0/2224898611 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5642ab3467f0 0x5642ab3570d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:56:25.491 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.480+0000 7fe744dc7700 1 -- 172.21.15.38:0/2224898611 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642ab34c950 msgr2=0x5642ab355900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:56:25.491 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.480+0000 7fe744dc7700 1 --2- 172.21.15.38:0/2224898611 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642ab34c950 0x5642ab355900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:56:25.491 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.480+0000 7fe744dc7700 1 -- 172.21.15.38:0/2224898611 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe74006c8a0 con 0x5642ab34c040 2023-03-09T14:56:25.492 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.480+0000 7fe7455c8700 1 --2- 172.21.15.38:0/2224898611 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5642ab3467f0 0x5642ab3570d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:56:25.492 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.480+0000 7fe744dc7700 1 --2- 172.21.15.38:0/2224898611 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5642ab34c040 0x5642ab3553c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe730019010 tx=0x7fe73001e6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca6aee4f3ae0bd3 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:56:25.492 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.480+0000 7fe728ff9700 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2023-03-09T14:56:25.492 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.481+0000 7fe728ff9700 1 -- 172.21.15.38:0/2224898611 <== mon.2 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe73001a050 con 0x5642ab34c040 2023-03-09T14:56:25.492 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.481+0000 7fe7539b5b80 1 -- 172.21.15.38:0/2224898611 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5642ab24d1b0 con 0x5642ab34c040 2023-03-09T14:56:25.492 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.481+0000 7fe728ff9700 1 -- 172.21.15.38:0/2224898611 <== mon.2 v2:172.21.15.122:3300/0 2 ==== osd_map(243..243 src has 1..243) v4 ==== 9561+0+0 (secure 0 0 0) 0x7fe73003e670 con 0x5642ab34c040 2023-03-09T14:56:25.492 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.481+0000 7fe728ff9700 1 -- 172.21.15.38:0/2224898611 <== mon.2 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe730012ee0 con 0x5642ab34c040 2023-03-09T14:56:25.493 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.481+0000 7fe7539b5b80 10 client.25892 fetch_fsmap learned FSMap version 37 2023-03-09T14:56:25.493 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.481+0000 7fe7539b5b80 1 -- 172.21.15.38:0/2224898611 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) v3 -- 0x5642ab0a6c60 con 0x5642ab34c040 2023-03-09T14:56:25.493 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.481+0000 7fe728ff9700 1 -- 172.21.15.38:0/2224898611 <== mon.2 v2:172.21.15.122:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5642ab24d1b0 con 0x5642ab34c040 2023-03-09T14:56:25.493 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.481+0000 7fe728ff9700 1 -- 172.21.15.38:0/2224898611 <== mon.2 v2:172.21.15.122:3300/0 5 ==== fsmap(e 37) v1 ==== 7478+0+0 (secure 0 0 0) 0x7fe7300135c0 con 0x5642ab34c040 2023-03-09T14:56:25.493 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.481+0000 7fe7539b5b80 10 client.25892 fetch_fsmap finished waiting for FSMap version 37 2023-03-09T14:56:25.493 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.481+0000 7fe7539b5b80 10 client.25892 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T14:56:25.493 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.481+0000 7fe7539b5b80 20 client.25892 populate_metadata read hostname 'smithi038' 2023-03-09T14:56:25.494 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.481+0000 7fe7539b5b80 1 --2- 172.21.15.38:0/2224898611 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5642ab3b5210 0x5642ab3b75f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:56:25.494 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.481+0000 7fe7539b5b80 4 client.25892 mds_command: new command op to 24530 tid=0[{"prefix": "scrub status"}] 2023-03-09T14:56:25.494 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.481+0000 7fe7539b5b80 1 -- 172.21.15.38:0/2224898611 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5642ab2371c0 con 0x5642ab3b5210 2023-03-09T14:56:25.494 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.482+0000 7fe7455c8700 1 --2- 172.21.15.38:0/2224898611 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5642ab3b5210 0x5642ab3b75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:56:25.494 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.482+0000 7fe728ff9700 10 client.25892 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:56:25.494 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.482+0000 7fe7455c8700 1 --2- 172.21.15.38:0/2224898611 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5642ab3b5210 0x5642ab3b75f0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=520d9ccb2f2c8e71 server_cookie=280246a03accf696 in_seq=0 out_seq=0 2023-03-09T14:56:25.495 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.482+0000 7fe728ff9700 1 -- 172.21.15.38:0/2224898611 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+265 (crc 0 0 0) 0x5642ab2371c0 con 0x5642ab3b5210 2023-03-09T14:56:25.495 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.482+0000 7fe728ff9700 10 client.25892 handle_command_reply: tid=0 2023-03-09T14:56:25.495 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.482+0000 7fe7539b5b80 1 client.25892 shutdown 2023-03-09T14:56:25.495 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.483+0000 7fe7539b5b80 1 -- 172.21.15.38:0/2224898611 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5642ab34c040 msgr2=0x5642ab3553c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:56:25.495 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.483+0000 7fe7539b5b80 1 --2- 172.21.15.38:0/2224898611 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5642ab34c040 0x5642ab3553c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe730019010 tx=0x7fe73001e6a0 comp rx=0 tx=0).stop 2023-03-09T14:56:25.496 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.483+0000 7fe7539b5b80 1 -- 172.21.15.38:0/2224898611 shutdown_connections 2023-03-09T14:56:25.496 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.483+0000 7fe7539b5b80 1 --2- 172.21.15.38:0/2224898611 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5642ab3b5210 0x5642ab3b75f0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:56:25.496 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.483+0000 7fe7539b5b80 1 --2- 172.21.15.38:0/2224898611 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5642ab34c950 0x5642ab355900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:56:25.496 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.483+0000 7fe7539b5b80 1 --2- 172.21.15.38:0/2224898611 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5642ab3467f0 0x5642ab3570d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:56:25.496 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.483+0000 7fe7539b5b80 1 --2- 172.21.15.38:0/2224898611 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5642ab34c040 0x5642ab3553c0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:56:25.496 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.483+0000 7fe7539b5b80 1 -- 172.21.15.38:0/2224898611 >> 172.21.15.38:0/2224898611 conn(0x5642ab338910 msgr2=0x5642ab337c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:56:25.496 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.483+0000 7fe728ff9700 0 client.25892 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:56:25.497 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.483+0000 7fe7539b5b80 1 -- 172.21.15.38:0/2224898611 shutdown_connections 2023-03-09T14:56:25.497 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.483+0000 7fe7539b5b80 1 -- 172.21.15.38:0/2224898611 wait complete. 2023-03-09T14:56:25.497 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.483+0000 7fe7539b5b80 20 client.25892 trim_cache size 0 max 16384 2023-03-09T14:56:25.497 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2023-03-09T14:56:25.498 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "scrub active (3 inodes in the stack)", 2023-03-09T14:56:25.498 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": { 2023-03-09T14:56:25.498 INFO:teuthology.orchestra.run.smithi038.stdout: "a14659f4-d7ea-4c5a-b15b-6138af3f909f": { 2023-03-09T14:56:25.498 INFO:teuthology.orchestra.run.smithi038.stdout: "path": "/", 2023-03-09T14:56:25.498 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "a14659f4-d7ea-4c5a-b15b-6138af3f909f", 2023-03-09T14:56:25.498 INFO:teuthology.orchestra.run.smithi038.stdout: "options": "recursive,force" 2023-03-09T14:56:25.498 INFO:teuthology.orchestra.run.smithi038.stdout: } 2023-03-09T14:56:25.499 INFO:teuthology.orchestra.run.smithi038.stdout: } 2023-03-09T14:56:25.499 INFO:teuthology.orchestra.run.smithi038.stdout:} 2023-03-09T14:56:25.499 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.484+0000 7fe74782c700 1 -- 172.21.15.38:0/2210105790 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fe72c09f520 msgr2=0x7fe72c0a19d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:56:25.500 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.484+0000 7fe74782c700 1 --2- 172.21.15.38:0/2210105790 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fe72c09f520 0x7fe72c0a19d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fe740073110 tx=0x7fe730006cd0 comp rx=0 tx=0).stop 2023-03-09T14:56:25.500 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.484+0000 7fe74782c700 1 -- 172.21.15.38:0/2210105790 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe7400fff10 msgr2=0x7fe740107690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:56:25.500 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.484+0000 7fe74782c700 1 --2- 172.21.15.38:0/2210105790 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe7400fff10 0x7fe740107690 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe73c019000 tx=0x7fe73c01a180 comp rx=0 tx=0).stop 2023-03-09T14:56:25.500 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.484+0000 7fe74782c700 1 -- 172.21.15.38:0/2210105790 shutdown_connections 2023-03-09T14:56:25.500 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.484+0000 7fe74782c700 1 --2- 172.21.15.38:0/2210105790 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fe72c09f520 0x7fe72c0a19d0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:56:25.500 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.484+0000 7fe74782c700 1 --2- 172.21.15.38:0/2210105790 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fe740108110 0x7fe74019c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:56:25.501 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.484+0000 7fe74782c700 1 --2- 172.21.15.38:0/2210105790 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fe7400fff10 0x7fe740107690 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:56:25.501 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.484+0000 7fe74782c700 1 --2- 172.21.15.38:0/2210105790 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fe740104c00 0x7fe740107bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:56:25.501 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.484+0000 7fe74782c700 1 -- 172.21.15.38:0/2210105790 >> 172.21.15.38:0/2210105790 conn(0x7fe7400748f0 msgr2=0x7fe7400fe2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:56:25.501 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.484+0000 7fe74782c700 1 -- 172.21.15.38:0/2210105790 shutdown_connections 2023-03-09T14:56:25.501 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:25.484+0000 7fe74782c700 1 -- 172.21.15.38:0/2210105790 wait complete. 2023-03-09T14:56:25.504 INFO:tasks.cephfs.filesystem:scrub status for tag:a14659f4-d7ea-4c5a-b15b-6138af3f909f - {'path': '/', 'tag': 'a14659f4-d7ea-4c5a-b15b-6138af3f909f', 'options': 'recursive,force'} 2023-03-09T14:56:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:56:25 smithi122 ceph-mon[93869]: pgmap v461: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 24 MiB/s wr, 2.41k op/s 2023-03-09T14:56:26.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:56:25 smithi181 ceph-mon[94189]: pgmap v461: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 24 MiB/s wr, 2.41k op/s 2023-03-09T14:56:26.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:56:25 smithi038 ceph-mon[92422]: pgmap v461: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 24 MiB/s wr, 2.41k op/s 2023-03-09T14:56:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:56:27 smithi122 ceph-mon[93869]: pgmap v462: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 25 MiB/s wr, 2.41k op/s 2023-03-09T14:56:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:56:27 smithi181 ceph-mon[94189]: pgmap v462: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 25 MiB/s wr, 2.41k op/s 2023-03-09T14:56:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:56:27 smithi038 ceph-mon[92422]: pgmap v462: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 25 MiB/s wr, 2.41k op/s 2023-03-09T14:56:28.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 360.0 s, 1114.7 tps, lat 93.729 ms stddev 114.711 2023-03-09T14:56:30.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:56:29 smithi122 ceph-mon[93869]: pgmap v463: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 22 MiB/s wr, 2.46k op/s 2023-03-09T14:56:30.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:56:29 smithi181 ceph-mon[94189]: pgmap v463: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 22 MiB/s wr, 2.46k op/s 2023-03-09T14:56:30.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:56:29 smithi038 ceph-mon[92422]: pgmap v463: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 22 MiB/s wr, 2.46k op/s 2023-03-09T14:56:32.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:56:31 smithi122 ceph-mon[93869]: pgmap v464: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 24 MiB/s wr, 2.60k op/s 2023-03-09T14:56:32.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:56:31 smithi181 ceph-mon[94189]: pgmap v464: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 24 MiB/s wr, 2.60k op/s 2023-03-09T14:56:32.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:56:31 smithi038 ceph-mon[92422]: pgmap v464: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 24 MiB/s wr, 2.60k op/s 2023-03-09T14:56:33.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 365.0 s, 978.2 tps, lat 101.500 ms stddev 76.172 2023-03-09T14:56:34.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:56:33 smithi122 ceph-mon[93869]: pgmap v465: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 25 MiB/s wr, 2.60k op/s 2023-03-09T14:56:34.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:56:33 smithi181 ceph-mon[94189]: pgmap v465: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 25 MiB/s wr, 2.60k op/s 2023-03-09T14:56:34.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:56:33 smithi038 ceph-mon[92422]: pgmap v465: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 25 MiB/s wr, 2.60k op/s 2023-03-09T14:56:35.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:56:34 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:56:35.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:56:34 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:56:35.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:56:34 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:56:36.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:56:35 smithi122 ceph-mon[93869]: pgmap v466: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 27 MiB/s wr, 2.88k op/s 2023-03-09T14:56:36.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:56:35 smithi181 ceph-mon[94189]: pgmap v466: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 27 MiB/s wr, 2.88k op/s 2023-03-09T14:56:36.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:56:35 smithi038 ceph-mon[92422]: pgmap v466: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 16 MiB/s rd, 27 MiB/s wr, 2.88k op/s 2023-03-09T14:56:37.922 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:56:37 smithi038 ceph-mon[92422]: pgmap v467: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 20 MiB/s wr, 2.19k op/s 2023-03-09T14:56:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:56:37 smithi122 ceph-mon[93869]: pgmap v467: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 20 MiB/s wr, 2.19k op/s 2023-03-09T14:56:38.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:56:37 smithi181 ceph-mon[94189]: pgmap v467: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 20 MiB/s wr, 2.19k op/s 2023-03-09T14:56:38.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 370.0 s, 241.4 tps, lat 412.295 ms stddev 897.964 2023-03-09T14:56:40.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:56:39 smithi122 ceph-mon[93869]: pgmap v468: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 25 MiB/s wr, 2.96k op/s 2023-03-09T14:56:40.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:56:39 smithi181 ceph-mon[94189]: pgmap v468: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 25 MiB/s wr, 2.96k op/s 2023-03-09T14:56:40.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:56:39 smithi038 ceph-mon[92422]: pgmap v468: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 25 MiB/s wr, 2.96k op/s 2023-03-09T14:56:42.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:56:42 smithi122 ceph-mon[93869]: pgmap v469: 129 pgs: 129 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 25 MiB/s wr, 2.82k op/s 2023-03-09T14:56:42.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:56:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:56:42.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:56:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:56:42.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:56:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:56:42.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:56:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:56:42.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:56:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:56:42.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:56:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:56:42.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:56:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:56:42.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:56:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:56:42.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:56:42 smithi181 ceph-mon[94189]: pgmap v469: 129 pgs: 129 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 25 MiB/s wr, 2.82k op/s 2023-03-09T14:56:42.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:56:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:56:42.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:56:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:56:42.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:56:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:56:42.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:56:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:56:42.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:56:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:56:42.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:56:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:56:42.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:56:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:56:42.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:56:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:56:42.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:56:42 smithi038 ceph-mon[92422]: pgmap v469: 129 pgs: 129 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 25 MiB/s wr, 2.82k op/s 2023-03-09T14:56:42.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:56:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:56:42.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:56:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:56:42.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:56:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:56:42.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:56:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:56:42.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:56:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:56:42.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:56:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:56:42.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:56:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:56:42.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:56:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:56:43.505 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:56:43 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T14:56:43.506 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:56:43 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T14:56:43.506 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:56:43 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T14:56:43.506 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:56:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:56:43.507 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:56:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:56:43.507 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:56:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:56:43.507 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:56:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:56:43.507 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:56:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:56:43.508 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:56:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:56:43.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:56:43 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T14:56:43.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:56:43 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T14:56:43.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:56:43 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T14:56:43.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:56:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:56:43.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:56:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:56:43.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:56:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:56:43.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:56:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:56:43.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:56:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:56:43.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:56:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:56:43.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:56:43 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T14:56:43.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:56:43 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T14:56:43.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:56:43 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T14:56:43.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:56:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:56:43.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:56:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:56:43.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:56:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:56:43.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:56:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:56:43.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:56:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:56:43.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:56:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:56:43.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 375.0 s, 712.5 tps, lat 124.394 ms stddev 116.831 2023-03-09T14:56:44.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:56:44 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi038 to 197.2M 2023-03-09T14:56:44.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:56:44 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi038 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-03-09T14:56:44.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:56:44 smithi038 ceph-mon[92422]: pgmap v470: 129 pgs: 129 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 22 MiB/s wr, 2.60k op/s 2023-03-09T14:56:44.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:56:44 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:56:44.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:56:44 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:56:44.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:56:44 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:56:44.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:56:44 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:56:44.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:56:44 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:56:44.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:56:44 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:56:44.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:56:44 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:56:44.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:56:44 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi038 to 197.2M 2023-03-09T14:56:44.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:56:44 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi038 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-03-09T14:56:44.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:56:44 smithi122 ceph-mon[93869]: pgmap v470: 129 pgs: 129 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 22 MiB/s wr, 2.60k op/s 2023-03-09T14:56:44.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:56:44 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:56:44.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:56:44 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:56:44.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:56:44 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:56:44.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:56:44 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:56:44.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:56:44 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:56:44.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:56:44 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:56:44.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:56:44 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:56:44.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:56:44 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi038 to 197.2M 2023-03-09T14:56:44.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:56:44 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi038 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-03-09T14:56:44.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:56:44 smithi181 ceph-mon[94189]: pgmap v470: 129 pgs: 129 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 22 MiB/s wr, 2.60k op/s 2023-03-09T14:56:44.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:56:44 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:56:44.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:56:44 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:56:44.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:56:44 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:56:44.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:56:44 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T14:56:44.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:56:44 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:56:44.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:56:44 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:56:44.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:56:44 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:56:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:56:45 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T14:56:45.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:56:45 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T14:56:45.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:56:45 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T14:56:45.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:56:45 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T14:56:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:56:45 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T14:56:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:56:45 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T14:56:46.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:56:46 smithi122 ceph-mon[93869]: pgmap v471: 129 pgs: 129 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 27 MiB/s wr, 3.21k op/s 2023-03-09T14:56:46.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:56:46 smithi181 ceph-mon[94189]: pgmap v471: 129 pgs: 129 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 27 MiB/s wr, 3.21k op/s 2023-03-09T14:56:46.756 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:56:46 smithi038 ceph-mon[92422]: pgmap v471: 129 pgs: 129 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 27 MiB/s wr, 3.21k op/s 2023-03-09T14:56:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:56:48 smithi122 ceph-mon[93869]: pgmap v472: 129 pgs: 129 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 24 MiB/s wr, 2.71k op/s 2023-03-09T14:56:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:56:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T14:56:48.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:56:48 smithi181 ceph-mon[94189]: pgmap v472: 129 pgs: 129 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 24 MiB/s wr, 2.71k op/s 2023-03-09T14:56:48.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:56:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T14:56:48.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:56:48 smithi038 ceph-mon[92422]: pgmap v472: 129 pgs: 129 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 24 MiB/s wr, 2.71k op/s 2023-03-09T14:56:48.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:56:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T14:56:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 380.0 s, 1469.9 tps, lat 76.705 ms stddev 101.017 2023-03-09T14:56:49.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:56:49 smithi122 ceph-mon[93869]: pgmap v473: 129 pgs: 129 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 25 MiB/s wr, 2.89k op/s 2023-03-09T14:56:49.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:56:49 smithi181 ceph-mon[94189]: pgmap v473: 129 pgs: 129 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 25 MiB/s wr, 2.89k op/s 2023-03-09T14:56:49.756 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:56:49 smithi038 ceph-mon[92422]: pgmap v473: 129 pgs: 129 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 25 MiB/s wr, 2.89k op/s 2023-03-09T14:56:52.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:56:51 smithi122 ceph-mon[93869]: pgmap v474: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 21 MiB/s wr, 2.33k op/s 2023-03-09T14:56:52.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:56:51 smithi181 ceph-mon[94189]: pgmap v474: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 21 MiB/s wr, 2.33k op/s 2023-03-09T14:56:52.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:56:51 smithi038 ceph-mon[92422]: pgmap v474: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 21 MiB/s wr, 2.33k op/s 2023-03-09T14:56:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 385.0 s, 344.7 tps, lat 228.382 ms stddev 366.044 2023-03-09T14:56:54.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:56:53 smithi122 ceph-mon[93869]: pgmap v475: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 20 MiB/s wr, 2.24k op/s 2023-03-09T14:56:54.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:56:53 smithi181 ceph-mon[94189]: pgmap v475: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 20 MiB/s wr, 2.24k op/s 2023-03-09T14:56:54.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:56:53 smithi038 ceph-mon[92422]: pgmap v475: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 20 MiB/s wr, 2.24k op/s 2023-03-09T14:56:55.506 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-03-09T14:56:55.724 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.721+0000 7fb6ede82700 1 -- 172.21.15.38:0/2820401000 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb6e8104040 msgr2=0x7fb6e810bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:56:55.724 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.721+0000 7fb6ede82700 1 --2- 172.21.15.38:0/2820401000 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb6e8104040 0x7fb6e810bc10 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb6d8009990 tx=0x7fb6d8009ca0 comp rx=0 tx=0).stop 2023-03-09T14:56:55.724 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.721+0000 7fb6ede82700 1 -- 172.21.15.38:0/2820401000 shutdown_connections 2023-03-09T14:56:55.725 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.721+0000 7fb6ede82700 1 --2- 172.21.15.38:0/2820401000 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb6e8104040 0x7fb6e810bc10 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:56:55.725 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.721+0000 7fb6ede82700 1 --2- 172.21.15.38:0/2820401000 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6e80ff000 0x7fb6e80ff3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:56:55.725 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.721+0000 7fb6ede82700 1 --2- 172.21.15.38:0/2820401000 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb6e80ff9a0 0x7fb6e8103960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:56:55.725 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.721+0000 7fb6ede82700 1 -- 172.21.15.38:0/2820401000 >> 172.21.15.38:0/2820401000 conn(0x7fb6e8075c60 msgr2=0x7fb6e8076060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:56:55.725 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.721+0000 7fb6ede82700 1 -- 172.21.15.38:0/2820401000 shutdown_connections 2023-03-09T14:56:55.726 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.721+0000 7fb6ede82700 1 -- 172.21.15.38:0/2820401000 wait complete. 2023-03-09T14:56:55.727 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.722+0000 7fb6ede82700 1 Processor -- start 2023-03-09T14:56:55.727 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.722+0000 7fb6ede82700 1 -- start start 2023-03-09T14:56:55.727 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.722+0000 7fb6ede82700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6e80ff000 0x7fb6e810c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:56:55.727 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.722+0000 7fb6ede82700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb6e80ff9a0 0x7fb6e810c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:56:55.728 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.722+0000 7fb6ede82700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb6e8104040 0x7fb6e810cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:56:55.728 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.722+0000 7fb6ede82700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6e81a4440 con 0x7fb6e80ff9a0 2023-03-09T14:56:55.728 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.722+0000 7fb6ede82700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6e81a45b0 con 0x7fb6e8104040 2023-03-09T14:56:55.728 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.722+0000 7fb6ede82700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6e81a4770 con 0x7fb6e80ff000 2023-03-09T14:56:55.728 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.722+0000 7fb6e6ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb6e80ff9a0 0x7fb6e810c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:56:55.728 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.722+0000 7fb6e7fff700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb6e8104040 0x7fb6e810cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:56:55.728 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.722+0000 7fb6e77fe700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6e80ff000 0x7fb6e810c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:56:55.729 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.722+0000 7fb6e6ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb6e80ff9a0 0x7fb6e810c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42036/0 (socket says 172.21.15.38:42036) 2023-03-09T14:56:55.729 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.722+0000 7fb6e6ffd700 1 -- 172.21.15.38:0/1745868277 learned_addr learned my addr 172.21.15.38:0/1745868277 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:56:55.729 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.722+0000 7fb6e7fff700 1 --2- 172.21.15.38:0/1745868277 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb6e8104040 0x7fb6e810cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.38:51224/0 (socket says 172.21.15.38:51224) 2023-03-09T14:56:55.729 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.723+0000 7fb6e77fe700 1 -- 172.21.15.38:0/1745868277 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb6e8104040 msgr2=0x7fb6e810cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:56:55.730 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.723+0000 7fb6e77fe700 1 --2- 172.21.15.38:0/1745868277 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb6e8104040 0x7fb6e810cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:56:55.730 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.723+0000 7fb6e77fe700 1 -- 172.21.15.38:0/1745868277 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb6e80ff9a0 msgr2=0x7fb6e810c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:56:55.730 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.723+0000 7fb6e77fe700 1 --2- 172.21.15.38:0/1745868277 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb6e80ff9a0 0x7fb6e810c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:56:55.730 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.723+0000 7fb6e77fe700 1 -- 172.21.15.38:0/1745868277 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6d80095f0 con 0x7fb6e80ff000 2023-03-09T14:56:55.730 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.723+0000 7fb6e77fe700 1 --2- 172.21.15.38:0/1745868277 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6e80ff000 0x7fb6e810c220 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb6e000ef10 tx=0x7fb6e000c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:56:55.730 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.723+0000 7fb6e4ff9700 1 -- 172.21.15.38:0/1745868277 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6e000c9c0 con 0x7fb6e80ff000 2023-03-09T14:56:55.730 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.723+0000 7fb6ede82700 1 -- 172.21.15.38:0/1745868277 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6e810ffb0 con 0x7fb6e80ff000 2023-03-09T14:56:55.731 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.723+0000 7fb6e4ff9700 1 -- 172.21.15.38:0/1745868277 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb6e000f040 con 0x7fb6e80ff000 2023-03-09T14:56:55.731 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.723+0000 7fb6e4ff9700 1 -- 172.21.15.38:0/1745868277 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6e0014480 con 0x7fb6e80ff000 2023-03-09T14:56:55.731 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.723+0000 7fb6ede82700 1 -- 172.21.15.38:0/1745868277 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6e810fe20 con 0x7fb6e80ff000 2023-03-09T14:56:55.733 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.726+0000 7fb6e4ff9700 1 -- 172.21.15.38:0/1745868277 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 25) v1 ==== 133295+0+0 (secure 0 0 0) 0x7fb6e000cb20 con 0x7fb6e80ff000 2023-03-09T14:56:55.733 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.727+0000 7fb6e4ff9700 1 --2- 172.21.15.38:0/1745868277 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fb6d009f4a0 0x7fb6d00a1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:56:55.733 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.727+0000 7fb6e4ff9700 1 -- 172.21.15.38:0/1745868277 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(243..243 src has 1..243) v4 ==== 9561+0+0 (secure 0 0 0) 0x7fb6e00ca7f0 con 0x7fb6e80ff000 2023-03-09T14:56:55.733 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.727+0000 7fb6e6ffd700 1 --2- 172.21.15.38:0/1745868277 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fb6d009f4a0 0x7fb6d00a1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:56:55.734 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.728+0000 7fb6e6ffd700 1 --2- 172.21.15.38:0/1745868277 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fb6d009f4a0 0x7fb6d00a1950 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb6e810da10 tx=0x7fb6dc009260 comp rx=0 tx=0).ready entity=mgr.15558 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:56:55.734 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.728+0000 7fb6fa00bb80 1 Processor -- start 2023-03-09T14:56:55.734 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.728+0000 7fb6fa00bb80 1 -- start start 2023-03-09T14:56:55.734 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.729+0000 7fb6fa00bb80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564e42ecf830 0x564e42ecfc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:56:55.734 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.729+0000 7fb6fa00bb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564e42ed0140 0x564e42ec9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:56:55.734 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.729+0000 7fb6fa00bb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564e42ec9fe0 0x564e42eca410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:56:55.734 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.729+0000 7fb6fa00bb80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e42db3da0 con 0x564e42ecf830 2023-03-09T14:56:55.734 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.729+0000 7fb6fa00bb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e42e22780 con 0x564e42ec9fe0 2023-03-09T14:56:55.735 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.729+0000 7fb6fa00bb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e42e1bcb0 con 0x564e42ed0140 2023-03-09T14:56:55.735 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.729+0000 7fb6e77fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564e42ecf830 0x564e42ecfc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:56:55.735 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.729+0000 7fb6e77fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564e42ecf830 0x564e42ecfc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42040/0 (socket says 172.21.15.38:42040) 2023-03-09T14:56:55.735 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.729+0000 7fb6e77fe700 1 -- 172.21.15.38:0/2844448575 learned_addr learned my addr 172.21.15.38:0/2844448575 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:56:55.735 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.729+0000 7fb6e6ffd700 1 --2- 172.21.15.38:0/2844448575 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564e42ed0140 0x564e42ec9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:56:55.735 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.729+0000 7fb6e7fff700 1 --2- 172.21.15.38:0/2844448575 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564e42ec9fe0 0x564e42eca410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:56:55.735 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.729+0000 7fb6e77fe700 1 -- 172.21.15.38:0/2844448575 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564e42ed0140 msgr2=0x564e42ec9aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:56:55.736 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.729+0000 7fb6e77fe700 1 --2- 172.21.15.38:0/2844448575 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564e42ed0140 0x564e42ec9aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:56:55.736 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.729+0000 7fb6e77fe700 1 -- 172.21.15.38:0/2844448575 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564e42ec9fe0 msgr2=0x564e42eca410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:56:55.736 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.729+0000 7fb6e77fe700 1 --2- 172.21.15.38:0/2844448575 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564e42ec9fe0 0x564e42eca410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:56:55.736 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.729+0000 7fb6e77fe700 1 -- 172.21.15.38:0/2844448575 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564e42de2190 con 0x564e42ecf830 2023-03-09T14:56:55.736 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.729+0000 7fb6e77fe700 1 --2- 172.21.15.38:0/2844448575 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564e42ecf830 0x564e42ecfc00 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fb6e00a0010 tx=0x7fb6e0098250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e207489f90485697 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:56:55.736 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.729+0000 7fb6cdffb700 1 -- 172.21.15.38:0/2844448575 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6e0097cb0 con 0x564e42ecf830 2023-03-09T14:56:55.736 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.730+0000 7fb6cdffb700 1 -- 172.21.15.38:0/2844448575 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb6e0098350 con 0x564e42ecf830 2023-03-09T14:56:55.736 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.730+0000 7fb6cdffb700 1 -- 172.21.15.38:0/2844448575 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6e0097aa0 con 0x564e42ecf830 2023-03-09T14:56:55.740 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.731+0000 7fb6fa00bb80 1 -- 172.21.15.38:0/2844448575 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564e42ecf830 msgr2=0x564e42ecfc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:56:55.740 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.731+0000 7fb6fa00bb80 1 --2- 172.21.15.38:0/2844448575 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564e42ecf830 0x564e42ecfc00 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fb6e00a0010 tx=0x7fb6e0098250 comp rx=0 tx=0).stop 2023-03-09T14:56:55.740 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.731+0000 7fb6fa00bb80 1 -- 172.21.15.38:0/2844448575 shutdown_connections 2023-03-09T14:56:55.740 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.731+0000 7fb6fa00bb80 1 --2- 172.21.15.38:0/2844448575 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564e42ecf830 0x564e42ecfc00 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:56:55.740 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.731+0000 7fb6fa00bb80 1 --2- 172.21.15.38:0/2844448575 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564e42ec9fe0 0x564e42eca410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:56:55.740 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.731+0000 7fb6fa00bb80 1 --2- 172.21.15.38:0/2844448575 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564e42ed0140 0x564e42ec9aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:56:55.741 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.731+0000 7fb6fa00bb80 1 -- 172.21.15.38:0/2844448575 >> 172.21.15.38:0/2844448575 conn(0x564e42ec4a20 msgr2=0x564e42ec4df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:56:55.741 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.731+0000 7fb6fa00bb80 1 -- 172.21.15.38:0/2844448575 shutdown_connections 2023-03-09T14:56:55.741 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.731+0000 7fb6fa00bb80 1 -- 172.21.15.38:0/2844448575 wait complete. 2023-03-09T14:56:55.741 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.731+0000 7fb6fa00bb80 1 -- start start 2023-03-09T14:56:55.741 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.731+0000 7fb6fa00bb80 1 Processor -- start 2023-03-09T14:56:55.741 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.732+0000 7fb6fa00bb80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564e42ec9fe0 0x564e42ed6100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:56:55.741 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.732+0000 7fb6fa00bb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564e42ecf830 0x564e42ed33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:56:55.742 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.732+0000 7fb6fa00bb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564e42ed0140 0x564e42ed3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:56:55.742 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.732+0000 7fb6fa00bb80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e42c9f920 con 0x564e42ec9fe0 2023-03-09T14:56:55.742 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.732+0000 7fb6fa00bb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e42ed41e0 con 0x564e42ed0140 2023-03-09T14:56:55.742 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.732+0000 7fb6fa00bb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e42edfc40 con 0x564e42ecf830 2023-03-09T14:56:55.742 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.732+0000 7fb6fa00bb80 10 client.0 _ll_register_callbacks cb 0x564e42dc1520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:56:55.742 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.732+0000 7fb6e77fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564e42ec9fe0 0x564e42ed6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:56:55.742 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.732+0000 7fb6e77fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564e42ec9fe0 0x564e42ed6100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42046/0 (socket says 172.21.15.38:42046) 2023-03-09T14:56:55.742 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.732+0000 7fb6e77fe700 1 -- 172.21.15.38:0/3679202505 learned_addr learned my addr 172.21.15.38:0/3679202505 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:56:55.743 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.732+0000 7fb6e6ffd700 1 --2- 172.21.15.38:0/3679202505 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564e42ecf830 0x564e42ed33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:56:55.743 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.732+0000 7fb6e7fff700 1 --2- 172.21.15.38:0/3679202505 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564e42ed0140 0x564e42ed3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:56:55.743 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.732+0000 7fb6e77fe700 1 -- 172.21.15.38:0/3679202505 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564e42ecf830 msgr2=0x564e42ed33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:56:55.743 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.732+0000 7fb6e77fe700 1 --2- 172.21.15.38:0/3679202505 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564e42ecf830 0x564e42ed33f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:56:55.743 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.732+0000 7fb6e77fe700 1 -- 172.21.15.38:0/3679202505 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564e42ed0140 msgr2=0x564e42ed3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:56:55.743 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.732+0000 7fb6e77fe700 1 --2- 172.21.15.38:0/3679202505 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564e42ed0140 0x564e42ed3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:56:55.743 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.732+0000 7fb6e77fe700 1 -- 172.21.15.38:0/3679202505 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6e810fe20 con 0x564e42ec9fe0 2023-03-09T14:56:55.744 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.732+0000 7fb6e6ffd700 1 --2- 172.21.15.38:0/3679202505 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564e42ecf830 0x564e42ed33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:56:55.744 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.732+0000 7fb6e77fe700 1 --2- 172.21.15.38:0/3679202505 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564e42ec9fe0 0x564e42ed6100 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fb6e00940d0 tx=0x7fb6e0097cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fae8f571bcf41c76 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:56:55.744 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.732+0000 7fb6c6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2023-03-09T14:56:55.744 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.732+0000 7fb6c6ffd700 1 -- 172.21.15.38:0/3679202505 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6e00a0540 con 0x564e42ec9fe0 2023-03-09T14:56:55.744 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.732+0000 7fb6c6ffd700 1 -- 172.21.15.38:0/3679202505 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(243..243 src has 1..243) v4 ==== 9561+0+0 (secure 0 0 0) 0x7fb6e00b62c0 con 0x564e42ec9fe0 2023-03-09T14:56:55.744 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.732+0000 7fb6fa00bb80 1 -- 172.21.15.38:0/3679202505 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564e42de9190 con 0x564e42ec9fe0 2023-03-09T14:56:55.744 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.733+0000 7fb6c6ffd700 1 -- 172.21.15.38:0/3679202505 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb6e0007690 con 0x564e42ec9fe0 2023-03-09T14:56:55.744 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.733+0000 7fb6fa00bb80 10 client.16266 fetch_fsmap learned FSMap version 39 2023-03-09T14:56:55.745 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.733+0000 7fb6fa00bb80 1 -- 172.21.15.38:0/3679202505 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x564e42c32c60 con 0x564e42ec9fe0 2023-03-09T14:56:55.745 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.733+0000 7fb6c6ffd700 1 -- 172.21.15.38:0/3679202505 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x564e42de9190 con 0x564e42ec9fe0 2023-03-09T14:56:55.745 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.733+0000 7fb6c6ffd700 1 -- 172.21.15.38:0/3679202505 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 39) v1 ==== 7486+0+0 (secure 0 0 0) 0x7fb6e00ca390 con 0x564e42ec9fe0 2023-03-09T14:56:55.745 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.733+0000 7fb6fa00bb80 10 client.16266 fetch_fsmap finished waiting for FSMap version 39 2023-03-09T14:56:55.745 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.733+0000 7fb6fa00bb80 10 client.16266 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T14:56:55.745 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.733+0000 7fb6fa00bb80 20 client.16266 populate_metadata read hostname 'smithi038' 2023-03-09T14:56:55.745 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.733+0000 7fb6fa00bb80 1 --2- 172.21.15.38:0/3679202505 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x564e42ee3970 0x564e42ee5d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:56:55.746 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.733+0000 7fb6fa00bb80 4 client.16266 mds_command: new command op to 24530 tid=0[{"prefix": "get_command_descriptions"}] 2023-03-09T14:56:55.746 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.733+0000 7fb6fa00bb80 1 -- 172.21.15.38:0/3679202505 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564e42c9ff60 con 0x564e42ee3970 2023-03-09T14:56:55.746 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.733+0000 7fb6e6ffd700 1 --2- 172.21.15.38:0/3679202505 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x564e42ee3970 0x564e42ee5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:56:55.746 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.734+0000 7fb6c6ffd700 10 client.16266 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:56:55.746 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.734+0000 7fb6e6ffd700 1 --2- 172.21.15.38:0/3679202505 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x564e42ee3970 0x564e42ee5d50 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d8025b3fa575820 server_cookie=6cb428b69ceae000 in_seq=0 out_seq=0 2023-03-09T14:56:55.746 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.734+0000 7fb6c6ffd700 1 -- 172.21.15.38:0/3679202505 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18278 (crc 0 0 0) 0x564e42c9ff60 con 0x564e42ee3970 2023-03-09T14:56:55.746 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.734+0000 7fb6c6ffd700 10 client.16266 handle_command_reply: tid=0 2023-03-09T14:56:55.746 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.734+0000 7fb6fa00bb80 1 client.16266 shutdown 2023-03-09T14:56:55.747 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.734+0000 7fb6fa00bb80 1 -- 172.21.15.38:0/3679202505 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564e42ec9fe0 msgr2=0x564e42ed6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:56:55.747 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.734+0000 7fb6fa00bb80 1 --2- 172.21.15.38:0/3679202505 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564e42ec9fe0 0x564e42ed6100 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fb6e00940d0 tx=0x7fb6e0097cb0 comp rx=0 tx=0).stop 2023-03-09T14:56:55.747 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.734+0000 7fb6fa00bb80 1 -- 172.21.15.38:0/3679202505 shutdown_connections 2023-03-09T14:56:55.747 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.734+0000 7fb6fa00bb80 1 --2- 172.21.15.38:0/3679202505 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x564e42ee3970 0x564e42ee5d50 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:56:55.747 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.735+0000 7fb6fa00bb80 1 --2- 172.21.15.38:0/3679202505 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564e42ec9fe0 0x564e42ed6100 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:56:55.747 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.735+0000 7fb6fa00bb80 1 --2- 172.21.15.38:0/3679202505 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564e42ed0140 0x564e42ed3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:56:55.747 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.735+0000 7fb6c6ffd700 0 client.16266 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:56:55.748 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.735+0000 7fb6fa00bb80 1 --2- 172.21.15.38:0/3679202505 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564e42ecf830 0x564e42ed33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:56:55.748 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.735+0000 7fb6fa00bb80 1 -- 172.21.15.38:0/3679202505 >> 172.21.15.38:0/3679202505 conn(0x564e42ec4a20 msgr2=0x564e42ec3c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:56:55.748 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.735+0000 7fb6fa00bb80 1 -- 172.21.15.38:0/3679202505 shutdown_connections 2023-03-09T14:56:55.748 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.735+0000 7fb6fa00bb80 1 -- 172.21.15.38:0/3679202505 wait complete. 2023-03-09T14:56:55.748 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.735+0000 7fb6fa00bb80 20 client.16266 trim_cache size 0 max 16384 2023-03-09T14:56:55.755 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.752+0000 7fb6fa00bb80 1 Processor -- start 2023-03-09T14:56:55.756 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.752+0000 7fb6fa00bb80 1 -- start start 2023-03-09T14:56:55.756 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.752+0000 7fb6fa00bb80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564e42ec9fe0 0x564e42ed5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:56:55.756 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.752+0000 7fb6fa00bb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564e42ecf830 0x564e42ec3c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:56:55.756 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.752+0000 7fb6fa00bb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564e42ed0140 0x564e42ecd570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:56:55.756 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.752+0000 7fb6fa00bb80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e42ee0770 con 0x564e42ec9fe0 2023-03-09T14:56:55.756 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.752+0000 7fb6fa00bb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e42ed1970 con 0x564e42ecf830 2023-03-09T14:56:55.757 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.752+0000 7fb6fa00bb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e42ec0d70 con 0x564e42ed0140 2023-03-09T14:56:55.757 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.752+0000 7fb6e77fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564e42ec9fe0 0x564e42ed5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:56:55.757 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.752+0000 7fb6e77fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564e42ec9fe0 0x564e42ed5ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42054/0 (socket says 172.21.15.38:42054) 2023-03-09T14:56:55.757 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.752+0000 7fb6e6ffd700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564e42ecf830 0x564e42ec3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:56:55.757 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.752+0000 7fb6e77fe700 1 -- 172.21.15.38:0/1883812419 learned_addr learned my addr 172.21.15.38:0/1883812419 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:56:55.757 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.752+0000 7fb6e7fff700 1 --2- 172.21.15.38:0/1883812419 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564e42ed0140 0x564e42ecd570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:56:55.757 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.753+0000 7fb6e77fe700 1 -- 172.21.15.38:0/1883812419 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564e42ed0140 msgr2=0x564e42ecd570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:56:55.758 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.753+0000 7fb6e77fe700 1 --2- 172.21.15.38:0/1883812419 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564e42ed0140 0x564e42ecd570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:56:55.758 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.753+0000 7fb6e77fe700 1 -- 172.21.15.38:0/1883812419 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564e42ecf830 msgr2=0x564e42ec3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:56:55.758 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.753+0000 7fb6e77fe700 1 --2- 172.21.15.38:0/1883812419 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564e42ecf830 0x564e42ec3c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:56:55.758 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.753+0000 7fb6e77fe700 1 -- 172.21.15.38:0/1883812419 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564e42c49de0 con 0x564e42ec9fe0 2023-03-09T14:56:55.758 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.753+0000 7fb6e6ffd700 1 --2- 172.21.15.38:0/1883812419 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564e42ecf830 0x564e42ec3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:56:55.758 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.753+0000 7fb6e77fe700 1 --2- 172.21.15.38:0/1883812419 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564e42ec9fe0 0x564e42ed5ca0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fb6e009c2e0 tx=0x7fb6e0098250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0c4edcf5ea1b596 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:56:55.758 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.753+0000 7fb6ce7fc700 1 -- 172.21.15.38:0/1883812419 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6e00b62c0 con 0x564e42ec9fe0 2023-03-09T14:56:55.759 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.753+0000 7fb6ce7fc700 1 -- 172.21.15.38:0/1883812419 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fb6e00a0540 con 0x564e42ec9fe0 2023-03-09T14:56:55.759 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.753+0000 7fb6ce7fc700 1 -- 172.21.15.38:0/1883812419 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6e005ebc0 con 0x564e42ec9fe0 2023-03-09T14:56:55.762 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.753+0000 7fb6fa00bb80 1 -- 172.21.15.38:0/1883812419 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564e42ec9fe0 msgr2=0x564e42ed5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:56:55.762 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.753+0000 7fb6fa00bb80 1 --2- 172.21.15.38:0/1883812419 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564e42ec9fe0 0x564e42ed5ca0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fb6e009c2e0 tx=0x7fb6e0098250 comp rx=0 tx=0).stop 2023-03-09T14:56:55.762 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.753+0000 7fb6fa00bb80 1 -- 172.21.15.38:0/1883812419 shutdown_connections 2023-03-09T14:56:55.762 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.753+0000 7fb6fa00bb80 1 --2- 172.21.15.38:0/1883812419 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564e42ec9fe0 0x564e42ed5ca0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:56:55.763 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.753+0000 7fb6fa00bb80 1 --2- 172.21.15.38:0/1883812419 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564e42ed0140 0x564e42ecd570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:56:55.763 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.753+0000 7fb6fa00bb80 1 --2- 172.21.15.38:0/1883812419 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564e42ecf830 0x564e42ec3c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:56:55.763 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.754+0000 7fb6fa00bb80 1 -- 172.21.15.38:0/1883812419 >> 172.21.15.38:0/1883812419 conn(0x564e42ec4a20 msgr2=0x564e42ee0030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:56:55.763 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.754+0000 7fb6fa00bb80 1 -- 172.21.15.38:0/1883812419 shutdown_connections 2023-03-09T14:56:55.763 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.754+0000 7fb6fa00bb80 1 -- 172.21.15.38:0/1883812419 wait complete. 2023-03-09T14:56:55.763 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.754+0000 7fb6fa00bb80 1 -- start start 2023-03-09T14:56:55.763 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.754+0000 7fb6fa00bb80 1 Processor -- start 2023-03-09T14:56:55.763 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.754+0000 7fb6fa00bb80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564e42ec9fe0 0x564e42ed6b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:56:55.764 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.754+0000 7fb6fa00bb80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564e42ecf830 0x564e42ed70a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:56:55.764 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.754+0000 7fb6fa00bb80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564e42ed0140 0x564e42f367d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:56:55.764 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.754+0000 7fb6fa00bb80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e42ed7810 con 0x564e42ec9fe0 2023-03-09T14:56:55.764 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.754+0000 7fb6fa00bb80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e42ed7980 con 0x564e42ecf830 2023-03-09T14:56:55.764 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.754+0000 7fb6fa00bb80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e42f36d10 con 0x564e42ed0140 2023-03-09T14:56:55.764 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.754+0000 7fb6fa00bb80 10 client.0 _ll_register_callbacks cb 0x564e42dc1520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:56:55.764 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.754+0000 7fb6e77fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564e42ec9fe0 0x564e42ed6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:56:55.765 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.754+0000 7fb6e77fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564e42ec9fe0 0x564e42ed6b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42060/0 (socket says 172.21.15.38:42060) 2023-03-09T14:56:55.765 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.754+0000 7fb6e77fe700 1 -- 172.21.15.38:0/1173994894 learned_addr learned my addr 172.21.15.38:0/1173994894 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:56:55.765 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.754+0000 7fb6e6ffd700 1 --2- 172.21.15.38:0/1173994894 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564e42ecf830 0x564e42ed70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:56:55.765 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.755+0000 7fb6e7fff700 1 --2- 172.21.15.38:0/1173994894 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564e42ed0140 0x564e42f367d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:56:55.765 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.755+0000 7fb6e6ffd700 1 -- 172.21.15.38:0/1173994894 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564e42ed0140 msgr2=0x564e42f367d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:56:55.765 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.755+0000 7fb6e6ffd700 1 --2- 172.21.15.38:0/1173994894 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564e42ed0140 0x564e42f367d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:56:55.765 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.755+0000 7fb6e6ffd700 1 -- 172.21.15.38:0/1173994894 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564e42ec9fe0 msgr2=0x564e42ed6b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:56:55.765 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.755+0000 7fb6e6ffd700 1 --2- 172.21.15.38:0/1173994894 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564e42ec9fe0 0x564e42ed6b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:56:55.766 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.755+0000 7fb6e6ffd700 1 -- 172.21.15.38:0/1173994894 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6dc017070 con 0x564e42ecf830 2023-03-09T14:56:55.766 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.755+0000 7fb6e77fe700 1 --2- 172.21.15.38:0/1173994894 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564e42ec9fe0 0x564e42ed6b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:56:55.766 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.755+0000 7fb6e6ffd700 1 --2- 172.21.15.38:0/1173994894 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564e42ecf830 0x564e42ed70a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x564e42ed66e0 tx=0x7fb6dc0047c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8bd730e60a3c2f78 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:56:55.766 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.755+0000 7fb6c6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2023-03-09T14:56:55.766 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.755+0000 7fb6c6ffd700 1 -- 172.21.15.38:0/1173994894 <== mon.2 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6dc024040 con 0x564e42ecf830 2023-03-09T14:56:55.766 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.755+0000 7fb6fa00bb80 1 -- 172.21.15.38:0/1173994894 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564e42dd91b0 con 0x564e42ecf830 2023-03-09T14:56:55.766 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.755+0000 7fb6c6ffd700 1 -- 172.21.15.38:0/1173994894 <== mon.2 v2:172.21.15.122:3300/0 2 ==== osd_map(243..243 src has 1..243) v4 ==== 9561+0+0 (secure 0 0 0) 0x7fb6dc01fdd0 con 0x564e42ecf830 2023-03-09T14:56:55.767 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.755+0000 7fb6c6ffd700 1 -- 172.21.15.38:0/1173994894 <== mon.2 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb6dc004020 con 0x564e42ecf830 2023-03-09T14:56:55.767 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.755+0000 7fb6fa00bb80 10 client.25910 fetch_fsmap learned FSMap version 39 2023-03-09T14:56:55.767 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.755+0000 7fb6fa00bb80 1 -- 172.21.15.38:0/1173994894 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x564e42c32c60 con 0x564e42ecf830 2023-03-09T14:56:55.767 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.756+0000 7fb6c6ffd700 1 -- 172.21.15.38:0/1173994894 <== mon.2 v2:172.21.15.122:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x564e42dd91b0 con 0x564e42ecf830 2023-03-09T14:56:55.767 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.756+0000 7fb6c6ffd700 1 -- 172.21.15.38:0/1173994894 <== mon.2 v2:172.21.15.122:3300/0 5 ==== fsmap(e 39) v1 ==== 7486+0+0 (secure 0 0 0) 0x7fb6dc005020 con 0x564e42ecf830 2023-03-09T14:56:55.767 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.756+0000 7fb6fa00bb80 10 client.25910 fetch_fsmap finished waiting for FSMap version 39 2023-03-09T14:56:55.767 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.756+0000 7fb6fa00bb80 10 client.25910 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T14:56:55.767 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.756+0000 7fb6fa00bb80 20 client.25910 populate_metadata read hostname 'smithi038' 2023-03-09T14:56:55.768 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.756+0000 7fb6fa00bb80 1 --2- 172.21.15.38:0/1173994894 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x564e42f384a0 0x564e42f3a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:56:55.768 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.756+0000 7fb6fa00bb80 4 client.25910 mds_command: new command op to 24530 tid=0[{"prefix": "scrub status"}] 2023-03-09T14:56:55.768 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.756+0000 7fb6fa00bb80 1 -- 172.21.15.38:0/1173994894 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x564e42dc31c0 con 0x564e42f384a0 2023-03-09T14:56:55.768 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.756+0000 7fb6e77fe700 1 --2- 172.21.15.38:0/1173994894 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x564e42f384a0 0x564e42f3a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:56:55.768 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.756+0000 7fb6c6ffd700 10 client.25910 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:56:55.768 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.756+0000 7fb6e77fe700 1 --2- 172.21.15.38:0/1173994894 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x564e42f384a0 0x564e42f3a8b0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a017292ca0c65dc4 server_cookie=7e74ebb5626bccec in_seq=0 out_seq=0 2023-03-09T14:56:55.768 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.756+0000 7fb6c6ffd700 1 -- 172.21.15.38:0/1173994894 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+265 (crc 0 0 0) 0x564e42dc31c0 con 0x564e42f384a0 2023-03-09T14:56:55.769 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.756+0000 7fb6c6ffd700 10 client.25910 handle_command_reply: tid=0 2023-03-09T14:56:55.769 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.756+0000 7fb6fa00bb80 1 client.25910 shutdown 2023-03-09T14:56:55.769 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.756+0000 7fb6fa00bb80 1 -- 172.21.15.38:0/1173994894 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564e42ecf830 msgr2=0x564e42ed70a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:56:55.769 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.756+0000 7fb6fa00bb80 1 --2- 172.21.15.38:0/1173994894 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564e42ecf830 0x564e42ed70a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x564e42ed66e0 tx=0x7fb6dc0047c0 comp rx=0 tx=0).stop 2023-03-09T14:56:55.769 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.757+0000 7fb6fa00bb80 1 -- 172.21.15.38:0/1173994894 shutdown_connections 2023-03-09T14:56:55.769 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.757+0000 7fb6fa00bb80 1 --2- 172.21.15.38:0/1173994894 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x564e42f384a0 0x564e42f3a8b0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:56:55.769 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.757+0000 7fb6fa00bb80 1 --2- 172.21.15.38:0/1173994894 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564e42ec9fe0 0x564e42ed6b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:56:55.770 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.757+0000 7fb6c6ffd700 0 client.25910 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:56:55.770 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.757+0000 7fb6fa00bb80 1 --2- 172.21.15.38:0/1173994894 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564e42ed0140 0x564e42f367d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:56:55.770 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.757+0000 7fb6fa00bb80 1 --2- 172.21.15.38:0/1173994894 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564e42ecf830 0x564e42ed70a0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:56:55.770 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.757+0000 7fb6fa00bb80 1 -- 172.21.15.38:0/1173994894 >> 172.21.15.38:0/1173994894 conn(0x564e42ec4a20 msgr2=0x564e42ec3c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:56:55.770 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.757+0000 7fb6fa00bb80 1 -- 172.21.15.38:0/1173994894 shutdown_connections 2023-03-09T14:56:55.770 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.757+0000 7fb6fa00bb80 1 -- 172.21.15.38:0/1173994894 wait complete. 2023-03-09T14:56:55.770 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.757+0000 7fb6fa00bb80 20 client.25910 trim_cache size 0 max 16384 2023-03-09T14:56:55.771 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2023-03-09T14:56:55.771 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "scrub active (1 inodes in the stack)", 2023-03-09T14:56:55.771 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": { 2023-03-09T14:56:55.771 INFO:teuthology.orchestra.run.smithi038.stdout: "a14659f4-d7ea-4c5a-b15b-6138af3f909f": { 2023-03-09T14:56:55.771 INFO:teuthology.orchestra.run.smithi038.stdout: "path": "/", 2023-03-09T14:56:55.771 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "a14659f4-d7ea-4c5a-b15b-6138af3f909f", 2023-03-09T14:56:55.772 INFO:teuthology.orchestra.run.smithi038.stdout: "options": "recursive,force" 2023-03-09T14:56:55.772 INFO:teuthology.orchestra.run.smithi038.stdout: } 2023-03-09T14:56:55.772 INFO:teuthology.orchestra.run.smithi038.stdout: } 2023-03-09T14:56:55.772 INFO:teuthology.orchestra.run.smithi038.stdout:} 2023-03-09T14:56:55.773 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.757+0000 7fb6ede82700 1 -- 172.21.15.38:0/1745868277 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fb6d009f4a0 msgr2=0x7fb6d00a1950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:56:55.774 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.757+0000 7fb6ede82700 1 --2- 172.21.15.38:0/1745868277 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fb6d009f4a0 0x7fb6d00a1950 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb6e810da10 tx=0x7fb6dc009260 comp rx=0 tx=0).stop 2023-03-09T14:56:55.774 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.757+0000 7fb6ede82700 1 -- 172.21.15.38:0/1745868277 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6e80ff000 msgr2=0x7fb6e810c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:56:55.774 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.757+0000 7fb6ede82700 1 --2- 172.21.15.38:0/1745868277 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6e80ff000 0x7fb6e810c220 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb6e000ef10 tx=0x7fb6e000c3a0 comp rx=0 tx=0).stop 2023-03-09T14:56:55.774 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.757+0000 7fb6ede82700 1 -- 172.21.15.38:0/1745868277 shutdown_connections 2023-03-09T14:56:55.774 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.757+0000 7fb6ede82700 1 --2- 172.21.15.38:0/1745868277 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fb6d009f4a0 0x7fb6d00a1950 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:56:55.774 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.757+0000 7fb6ede82700 1 --2- 172.21.15.38:0/1745868277 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fb6e8104040 0x7fb6e810cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:56:55.774 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.757+0000 7fb6ede82700 1 --2- 172.21.15.38:0/1745868277 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fb6e80ff000 0x7fb6e810c220 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:56:55.774 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.757+0000 7fb6ede82700 1 --2- 172.21.15.38:0/1745868277 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fb6e80ff9a0 0x7fb6e810c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:56:55.775 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.757+0000 7fb6ede82700 1 -- 172.21.15.38:0/1745868277 >> 172.21.15.38:0/1745868277 conn(0x7fb6e8075c60 msgr2=0x7fb6e80fc940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:56:55.775 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.757+0000 7fb6ede82700 1 -- 172.21.15.38:0/1745868277 shutdown_connections 2023-03-09T14:56:55.775 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:56:55.757+0000 7fb6ede82700 1 -- 172.21.15.38:0/1745868277 wait complete. 2023-03-09T14:56:55.775 INFO:tasks.cephfs.filesystem:scrub status for tag:a14659f4-d7ea-4c5a-b15b-6138af3f909f - {'path': '/', 'tag': 'a14659f4-d7ea-4c5a-b15b-6138af3f909f', 'options': 'recursive,force'} 2023-03-09T14:56:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:56:55 smithi122 ceph-mon[93869]: pgmap v476: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 27 MiB/s wr, 3.06k op/s 2023-03-09T14:56:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:56:55 smithi181 ceph-mon[94189]: pgmap v476: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 27 MiB/s wr, 3.06k op/s 2023-03-09T14:56:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:56:55 smithi038 ceph-mon[92422]: pgmap v476: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 27 MiB/s wr, 3.06k op/s 2023-03-09T14:56:58.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:56:57 smithi122 ceph-mon[93869]: pgmap v477: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 21 MiB/s wr, 2.44k op/s 2023-03-09T14:56:58.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:56:57 smithi181 ceph-mon[94189]: pgmap v477: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 21 MiB/s wr, 2.44k op/s 2023-03-09T14:56:58.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:56:57 smithi038 ceph-mon[92422]: pgmap v477: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 21 MiB/s wr, 2.44k op/s 2023-03-09T14:56:58.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 390.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T14:57:00.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:56:59 smithi181 ceph-mon[94189]: pgmap v478: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 20 MiB/s wr, 2.30k op/s 2023-03-09T14:57:00.236 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:56:59 smithi122 ceph-mon[93869]: pgmap v478: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 20 MiB/s wr, 2.30k op/s 2023-03-09T14:57:00.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:56:59 smithi038 ceph-mon[92422]: pgmap v478: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 20 MiB/s wr, 2.30k op/s 2023-03-09T14:57:02.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:57:01 smithi122 ceph-mon[93869]: pgmap v479: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 16 MiB/s wr, 1.85k op/s 2023-03-09T14:57:02.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:57:01 smithi181 ceph-mon[94189]: pgmap v479: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 16 MiB/s wr, 1.85k op/s 2023-03-09T14:57:02.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:57:01 smithi038 ceph-mon[92422]: pgmap v479: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 16 MiB/s wr, 1.85k op/s 2023-03-09T14:57:03.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 395.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T14:57:04.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:57:03 smithi122 ceph-mon[93869]: pgmap v480: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 12 MiB/s wr, 1.50k op/s 2023-03-09T14:57:04.237 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:57:03 smithi181 ceph-mon[94189]: pgmap v480: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 12 MiB/s wr, 1.50k op/s 2023-03-09T14:57:04.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:57:03 smithi038 ceph-mon[92422]: pgmap v480: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 12 MiB/s wr, 1.50k op/s 2023-03-09T14:57:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:57:05 smithi122 ceph-mon[93869]: pgmap v481: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 11 MiB/s wr, 1.42k op/s 2023-03-09T14:57:06.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:57:05 smithi181 ceph-mon[94189]: pgmap v481: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 11 MiB/s wr, 1.42k op/s 2023-03-09T14:57:06.258 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:57:05 smithi038 ceph-mon[92422]: pgmap v481: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 11 MiB/s wr, 1.42k op/s 2023-03-09T14:57:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:57:07 smithi122 ceph-mon[93869]: pgmap v482: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 3.4 MiB/s wr, 485 op/s 2023-03-09T14:57:08.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:57:07 smithi181 ceph-mon[94189]: pgmap v482: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 3.4 MiB/s wr, 485 op/s 2023-03-09T14:57:08.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:57:07 smithi038 ceph-mon[92422]: pgmap v482: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 3.4 MiB/s wr, 485 op/s 2023-03-09T14:57:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 400.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T14:57:10.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:57:09 smithi122 ceph-mon[93869]: pgmap v483: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 3.2 MiB/s wr, 427 op/s 2023-03-09T14:57:10.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:57:09 smithi181 ceph-mon[94189]: pgmap v483: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 3.2 MiB/s wr, 427 op/s 2023-03-09T14:57:10.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:57:09 smithi038 ceph-mon[92422]: pgmap v483: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 3.2 MiB/s wr, 427 op/s 2023-03-09T14:57:12.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:57:11 smithi038 ceph-mon[92422]: pgmap v484: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 2.9 MiB/s wr, 387 op/s 2023-03-09T14:57:12.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:57:11 smithi122 ceph-mon[93869]: pgmap v484: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 2.9 MiB/s wr, 387 op/s 2023-03-09T14:57:12.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:57:11 smithi181 ceph-mon[94189]: pgmap v484: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 2.9 MiB/s wr, 387 op/s 2023-03-09T14:57:13.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 405.0 s, 1144.9 tps, lat 367.752 ms stddev 2224.669 2023-03-09T14:57:14.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:57:13 smithi122 ceph-mon[93869]: pgmap v485: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 731 KiB/s rd, 3.9 MiB/s wr, 439 op/s 2023-03-09T14:57:14.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:57:13 smithi181 ceph-mon[94189]: pgmap v485: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 731 KiB/s rd, 3.9 MiB/s wr, 439 op/s 2023-03-09T14:57:14.255 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:57:13 smithi038 ceph-mon[92422]: pgmap v485: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 731 KiB/s rd, 3.9 MiB/s wr, 439 op/s 2023-03-09T14:57:15.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:57:14 smithi122 ceph-mon[93869]: osdmap e244: 12 total, 12 up, 12 in 2023-03-09T14:57:15.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:57:14 smithi181 ceph-mon[94189]: osdmap e244: 12 total, 12 up, 12 in 2023-03-09T14:57:15.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:57:14 smithi038 ceph-mon[92422]: osdmap e244: 12 total, 12 up, 12 in 2023-03-09T14:57:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:57:15 smithi122 ceph-mon[93869]: pgmap v487: 129 pgs: 6 active+clean+snaptrim, 18 active+clean+snaptrim_wait, 105 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 8.9 MiB/s wr, 808 op/s 2023-03-09T14:57:16.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:57:15 smithi181 ceph-mon[94189]: pgmap v487: 129 pgs: 6 active+clean+snaptrim, 18 active+clean+snaptrim_wait, 105 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 8.9 MiB/s wr, 808 op/s 2023-03-09T14:57:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:57:15 smithi038 ceph-mon[92422]: pgmap v487: 129 pgs: 6 active+clean+snaptrim, 18 active+clean+snaptrim_wait, 105 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 8.9 MiB/s wr, 808 op/s 2023-03-09T14:57:18.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:57:17 smithi122 ceph-mon[93869]: pgmap v488: 129 pgs: 14 active+clean+snaptrim, 39 active+clean+snaptrim_wait, 76 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 11 MiB/s wr, 1.08k op/s 2023-03-09T14:57:18.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:57:17 smithi181 ceph-mon[94189]: pgmap v488: 129 pgs: 14 active+clean+snaptrim, 39 active+clean+snaptrim_wait, 76 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 11 MiB/s wr, 1.08k op/s 2023-03-09T14:57:18.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:57:17 smithi038 ceph-mon[92422]: pgmap v488: 129 pgs: 14 active+clean+snaptrim, 39 active+clean+snaptrim_wait, 76 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 11 MiB/s wr, 1.08k op/s 2023-03-09T14:57:18.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 410.0 s, 43.6 tps, lat 1416.956 ms stddev 845.540 2023-03-09T14:57:20.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:57:20 smithi122 ceph-mon[93869]: pgmap v489: 129 pgs: 20 active+clean+snaptrim, 57 active+clean+snaptrim_wait, 52 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 14 MiB/s wr, 1.37k op/s 2023-03-09T14:57:20.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:57:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T14:57:20.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:57:20 smithi181 ceph-mon[94189]: pgmap v489: 129 pgs: 20 active+clean+snaptrim, 57 active+clean+snaptrim_wait, 52 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 14 MiB/s wr, 1.37k op/s 2023-03-09T14:57:20.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:57:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T14:57:20.504 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:57:20 smithi038 ceph-mon[92422]: pgmap v489: 129 pgs: 20 active+clean+snaptrim, 57 active+clean+snaptrim_wait, 52 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 14 MiB/s wr, 1.37k op/s 2023-03-09T14:57:20.504 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:57:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T14:57:22.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:57:22 smithi122 ceph-mon[93869]: pgmap v490: 129 pgs: 14 active+clean+snaptrim, 33 active+clean+snaptrim_wait, 82 active+clean; 12 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 15 MiB/s wr, 1.52k op/s 2023-03-09T14:57:22.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:57:22 smithi181 ceph-mon[94189]: pgmap v490: 129 pgs: 14 active+clean+snaptrim, 33 active+clean+snaptrim_wait, 82 active+clean; 12 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 15 MiB/s wr, 1.52k op/s 2023-03-09T14:57:22.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:57:22 smithi038 ceph-mon[92422]: pgmap v490: 129 pgs: 14 active+clean+snaptrim, 33 active+clean+snaptrim_wait, 82 active+clean; 12 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 15 MiB/s wr, 1.52k op/s 2023-03-09T14:57:23.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 415.0 s, 672.7 tps, lat 207.057 ms stddev 486.751 2023-03-09T14:57:24.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:57:23 smithi122 ceph-mon[93869]: pgmap v491: 129 pgs: 9 active+clean+snaptrim, 21 active+clean+snaptrim_wait, 99 active+clean; 12 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 14 MiB/s wr, 1.53k op/s 2023-03-09T14:57:24.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:57:23 smithi181 ceph-mon[94189]: pgmap v491: 129 pgs: 9 active+clean+snaptrim, 21 active+clean+snaptrim_wait, 99 active+clean; 12 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 14 MiB/s wr, 1.53k op/s 2023-03-09T14:57:24.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:57:23 smithi038 ceph-mon[92422]: pgmap v491: 129 pgs: 9 active+clean+snaptrim, 21 active+clean+snaptrim_wait, 99 active+clean; 12 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 14 MiB/s wr, 1.53k op/s 2023-03-09T14:57:25.776 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-03-09T14:57:26.011 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.009+0000 7f9245e51700 1 -- 172.21.15.38:0/1185646351 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f924010ce50 msgr2=0x7f924010d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:57:26.012 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.009+0000 7f9245e51700 1 --2- 172.21.15.38:0/1185646351 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f924010ce50 0x7f924010d240 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9230009f90 tx=0x7f923000b330 comp rx=0 tx=0).stop 2023-03-09T14:57:26.012 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.009+0000 7f9245e51700 1 -- 172.21.15.38:0/1185646351 shutdown_connections 2023-03-09T14:57:26.012 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.009+0000 7f9245e51700 1 --2- 172.21.15.38:0/1185646351 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f9240108130 0x7f9240113bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:57:26.012 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.009+0000 7f9245e51700 1 --2- 172.21.15.38:0/1185646351 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f924010ce50 0x7f924010d240 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:57:26.012 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.009+0000 7f9245e51700 1 --2- 172.21.15.38:0/1185646351 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9240107780 0x7f9240107bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:57:26.012 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.009+0000 7f9245e51700 1 -- 172.21.15.38:0/1185646351 >> 172.21.15.38:0/1185646351 conn(0x7f924006b450 msgr2=0x7f92400712c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:57:26.013 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.009+0000 7f9245e51700 1 -- 172.21.15.38:0/1185646351 shutdown_connections 2023-03-09T14:57:26.013 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.010+0000 7f9245e51700 1 -- 172.21.15.38:0/1185646351 wait complete. 2023-03-09T14:57:26.013 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.010+0000 7f9245e51700 1 Processor -- start 2023-03-09T14:57:26.014 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.010+0000 7f9245e51700 1 -- start start 2023-03-09T14:57:26.014 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.011+0000 7f9245e51700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9240107780 0x7f92401115f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:57:26.014 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.011+0000 7f9245e51700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9240108130 0x7f9240122630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:57:26.014 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.011+0000 7f9245e51700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f924010ce50 0x7f9240111bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:57:26.014 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.011+0000 7f9245e51700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f924011d570 con 0x7f924010ce50 2023-03-09T14:57:26.014 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.011+0000 7f9245e51700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f924011d6e0 con 0x7f9240107780 2023-03-09T14:57:26.014 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.011+0000 7f9245e51700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f924011d8a0 con 0x7f9240108130 2023-03-09T14:57:26.015 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.011+0000 7f923f7fe700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9240107780 0x7f92401115f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:57:26.015 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.011+0000 7f9236ffd700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9240108130 0x7f9240122630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:57:26.015 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.011+0000 7f923ffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f924010ce50 0x7f9240111bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:57:26.015 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.011+0000 7f923f7fe700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9240107780 0x7f92401115f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.38:51268/0 (socket says 172.21.15.38:51268) 2023-03-09T14:57:26.015 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.011+0000 7f923f7fe700 1 -- 172.21.15.38:0/1551858936 learned_addr learned my addr 172.21.15.38:0/1551858936 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:57:26.015 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.011+0000 7f9236ffd700 1 --2- 172.21.15.38:0/1551858936 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9240108130 0x7f9240122630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.38:49776/0 (socket says 172.21.15.38:49776) 2023-03-09T14:57:26.015 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.011+0000 7f923f7fe700 1 -- 172.21.15.38:0/1551858936 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9240108130 msgr2=0x7f9240122630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:57:26.016 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.011+0000 7f923f7fe700 1 --2- 172.21.15.38:0/1551858936 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9240108130 0x7f9240122630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:57:26.016 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.011+0000 7f923f7fe700 1 -- 172.21.15.38:0/1551858936 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f924010ce50 msgr2=0x7f9240111bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:57:26.016 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.011+0000 7f923f7fe700 1 --2- 172.21.15.38:0/1551858936 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f924010ce50 0x7f9240111bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:57:26.016 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.011+0000 7f923f7fe700 1 -- 172.21.15.38:0/1551858936 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9230009c40 con 0x7f9240107780 2023-03-09T14:57:26.016 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.011+0000 7f923ffff700 1 --2- 172.21.15.38:0/1551858936 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f924010ce50 0x7f9240111bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:57:26.016 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.012+0000 7f923f7fe700 1 --2- 172.21.15.38:0/1551858936 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9240107780 0x7f92401115f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f9230000f80 tx=0x7f9230007580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:57:26.017 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.012+0000 7f923d7fa700 1 -- 172.21.15.38:0/1551858936 <== mon.2 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f923000e040 con 0x7f9240107780 2023-03-09T14:57:26.017 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.012+0000 7f923d7fa700 1 -- 172.21.15.38:0/1551858936 <== mon.2 v2:172.21.15.122:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f923000bbf0 con 0x7f9240107780 2023-03-09T14:57:26.017 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.012+0000 7f9245e51700 1 -- 172.21.15.38:0/1551858936 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9240115bb0 con 0x7f9240107780 2023-03-09T14:57:26.018 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.012+0000 7f923d7fa700 1 -- 172.21.15.38:0/1551858936 <== mon.2 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92300052a0 con 0x7f9240107780 2023-03-09T14:57:26.018 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.014+0000 7f9245e51700 1 -- 172.21.15.38:0/1551858936 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9240115a20 con 0x7f9240107780 2023-03-09T14:57:26.018 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.017+0000 7f923d7fa700 1 -- 172.21.15.38:0/1551858936 <== mon.2 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 25) v1 ==== 133295+0+0 (secure 0 0 0) 0x7f923001d470 con 0x7f9240107780 2023-03-09T14:57:26.019 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.018+0000 7f923d7fa700 1 --2- 172.21.15.38:0/1551858936 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f922c09f510 0x7f922c0a19c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:57:26.020 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.018+0000 7f923d7fa700 1 -- 172.21.15.38:0/1551858936 <== mon.2 v2:172.21.15.122:3300/0 5 ==== osd_map(244..244 src has 1..244) v4 ==== 9665+0+0 (secure 0 0 0) 0x7f92300cb490 con 0x7f9240107780 2023-03-09T14:57:26.020 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.018+0000 7f9236ffd700 1 --2- 172.21.15.38:0/1551858936 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f922c09f510 0x7f922c0a19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:57:26.021 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.019+0000 7f9236ffd700 1 --2- 172.21.15.38:0/1551858936 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f922c09f510 0x7f922c0a19c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9240112c60 tx=0x7f9228010040 comp rx=0 tx=0).ready entity=mgr.15558 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:57:26.021 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.019+0000 7f9251fdab80 1 Processor -- start 2023-03-09T14:57:26.021 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.019+0000 7f9251fdab80 1 -- start start 2023-03-09T14:57:26.022 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.020+0000 7f9251fdab80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e488b93830 0x55e488b93c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:57:26.022 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.020+0000 7f9251fdab80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e488b94140 0x55e488b8daa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:57:26.022 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.020+0000 7f9251fdab80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e488b8dfe0 0x55e488b8e410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:57:26.022 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.020+0000 7f9251fdab80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e488a77da0 con 0x55e488b8dfe0 2023-03-09T14:57:26.023 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.020+0000 7f9251fdab80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e488ae6780 con 0x55e488b94140 2023-03-09T14:57:26.023 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.020+0000 7f9251fdab80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e488adfcb0 con 0x55e488b93830 2023-03-09T14:57:26.025 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.020+0000 7f923f7fe700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e488b93830 0x55e488b93c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:57:26.025 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.020+0000 7f923f7fe700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e488b93830 0x55e488b93c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.38:49784/0 (socket says 172.21.15.38:49784) 2023-03-09T14:57:26.025 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.020+0000 7f923f7fe700 1 -- 172.21.15.38:0/2439501142 learned_addr learned my addr 172.21.15.38:0/2439501142 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:57:26.025 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.020+0000 7f923f7fe700 1 -- 172.21.15.38:0/2439501142 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e488b94140 msgr2=0x55e488b8daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:57:26.025 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.020+0000 7f923f7fe700 1 --2- 172.21.15.38:0/2439501142 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e488b94140 0x55e488b8daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:57:26.026 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.020+0000 7f923f7fe700 1 -- 172.21.15.38:0/2439501142 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e488b8dfe0 msgr2=0x55e488b8e410 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-03-09T14:57:26.026 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.020+0000 7f923f7fe700 1 --2- 172.21.15.38:0/2439501142 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e488b8dfe0 0x55e488b8e410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:57:26.026 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.020+0000 7f923f7fe700 1 -- 172.21.15.38:0/2439501142 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e488aa6190 con 0x55e488b93830 2023-03-09T14:57:26.026 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.021+0000 7f923f7fe700 1 --2- 172.21.15.38:0/2439501142 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e488b93830 0x55e488b93c00 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f92300a0c40 tx=0x7f923000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc87380230c246cc server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:57:26.026 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.021+0000 7f9235ffb700 1 -- 172.21.15.38:0/2439501142 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92300b0070 con 0x55e488b93830 2023-03-09T14:57:26.026 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.021+0000 7f9235ffb700 1 -- 172.21.15.38:0/2439501142 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9230099010 con 0x55e488b93830 2023-03-09T14:57:26.026 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.021+0000 7f9235ffb700 1 -- 172.21.15.38:0/2439501142 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9230099270 con 0x55e488b93830 2023-03-09T14:57:26.027 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.022+0000 7f9251fdab80 1 -- 172.21.15.38:0/2439501142 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e488b93830 msgr2=0x55e488b93c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:57:26.027 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.022+0000 7f9251fdab80 1 --2- 172.21.15.38:0/2439501142 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e488b93830 0x55e488b93c00 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f92300a0c40 tx=0x7f923000b330 comp rx=0 tx=0).stop 2023-03-09T14:57:26.027 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.022+0000 7f9251fdab80 1 -- 172.21.15.38:0/2439501142 shutdown_connections 2023-03-09T14:57:26.027 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.022+0000 7f9251fdab80 1 --2- 172.21.15.38:0/2439501142 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e488b8dfe0 0x55e488b8e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:57:26.027 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.022+0000 7f9251fdab80 1 --2- 172.21.15.38:0/2439501142 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e488b93830 0x55e488b93c00 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:57:26.027 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.022+0000 7f9251fdab80 1 --2- 172.21.15.38:0/2439501142 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e488b94140 0x55e488b8daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:57:26.027 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.022+0000 7f9251fdab80 1 -- 172.21.15.38:0/2439501142 >> 172.21.15.38:0/2439501142 conn(0x55e488b88a20 msgr2=0x55e488b88df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:57:26.027 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.022+0000 7f9251fdab80 1 -- 172.21.15.38:0/2439501142 shutdown_connections 2023-03-09T14:57:26.028 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.022+0000 7f9251fdab80 1 -- 172.21.15.38:0/2439501142 wait complete. 2023-03-09T14:57:26.028 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.022+0000 7f9251fdab80 1 -- start start 2023-03-09T14:57:26.028 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.022+0000 7f9251fdab80 1 Processor -- start 2023-03-09T14:57:26.028 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.023+0000 7f9251fdab80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e488b8dfe0 0x55e488b9a190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:57:26.028 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.023+0000 7f9251fdab80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e488b93830 0x55e488b97410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:57:26.029 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.023+0000 7f9251fdab80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e488b94140 0x55e488b97b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:57:26.029 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.023+0000 7f9251fdab80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e488963920 con 0x55e488b94140 2023-03-09T14:57:26.029 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.023+0000 7f9251fdab80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e488b98270 con 0x55e488b8dfe0 2023-03-09T14:57:26.029 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.023+0000 7f9251fdab80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e488ba39f0 con 0x55e488b93830 2023-03-09T14:57:26.029 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.023+0000 7f9251fdab80 10 client.0 _ll_register_callbacks cb 0x55e488a85520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:57:26.029 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.023+0000 7f923f7fe700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e488b8dfe0 0x55e488b9a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:57:26.029 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.023+0000 7f923f7fe700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e488b8dfe0 0x55e488b9a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.38:51280/0 (socket says 172.21.15.38:51280) 2023-03-09T14:57:26.030 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.023+0000 7f923f7fe700 1 -- 172.21.15.38:0/486903974 learned_addr learned my addr 172.21.15.38:0/486903974 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:57:26.030 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.023+0000 7f923ffff700 1 --2- 172.21.15.38:0/486903974 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e488b94140 0x55e488b97b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:57:26.030 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.023+0000 7f9236ffd700 1 --2- 172.21.15.38:0/486903974 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e488b93830 0x55e488b97410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:57:26.030 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.023+0000 7f923f7fe700 1 -- 172.21.15.38:0/486903974 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e488b93830 msgr2=0x55e488b97410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:57:26.030 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.023+0000 7f923f7fe700 1 --2- 172.21.15.38:0/486903974 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e488b93830 0x55e488b97410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:57:26.030 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.023+0000 7f923f7fe700 1 -- 172.21.15.38:0/486903974 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e488b94140 msgr2=0x55e488b97b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:57:26.030 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.023+0000 7f923f7fe700 1 --2- 172.21.15.38:0/486903974 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e488b94140 0x55e488b97b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:57:26.031 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.023+0000 7f923f7fe700 1 -- 172.21.15.38:0/486903974 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9240115a20 con 0x55e488b8dfe0 2023-03-09T14:57:26.031 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.023+0000 7f9236ffd700 1 --2- 172.21.15.38:0/486903974 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e488b93830 0x55e488b97410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:57:26.032 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.023+0000 7f923f7fe700 1 --2- 172.21.15.38:0/486903974 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e488b8dfe0 0x55e488b9a190 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f923000b6c0 tx=0x7f92300a1170 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9bee83352018b975 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:57:26.032 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.023+0000 7f921effd700 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2023-03-09T14:57:26.032 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.024+0000 7f921effd700 1 -- 172.21.15.38:0/486903974 <== mon.2 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92300b0030 con 0x55e488b8dfe0 2023-03-09T14:57:26.032 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.024+0000 7f921effd700 1 -- 172.21.15.38:0/486903974 <== mon.2 v2:172.21.15.122:3300/0 2 ==== osd_map(244..244 src has 1..244) v4 ==== 9665+0+0 (secure 0 0 0) 0x7f92300b1070 con 0x55e488b8dfe0 2023-03-09T14:57:26.032 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.024+0000 7f9251fdab80 1 -- 172.21.15.38:0/486903974 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e488aad190 con 0x55e488b8dfe0 2023-03-09T14:57:26.032 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.024+0000 7f921effd700 1 -- 172.21.15.38:0/486903974 <== mon.2 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9230099270 con 0x55e488b8dfe0 2023-03-09T14:57:26.033 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.024+0000 7f9251fdab80 10 client.25919 fetch_fsmap learned FSMap version 41 2023-03-09T14:57:26.033 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.024+0000 7f9251fdab80 1 -- 172.21.15.38:0/486903974 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) v3 -- 0x55e4888f6c60 con 0x55e488b8dfe0 2023-03-09T14:57:26.033 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.024+0000 7f921effd700 1 -- 172.21.15.38:0/486903974 <== mon.2 v2:172.21.15.122:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f923007c080 con 0x55e488b8dfe0 2023-03-09T14:57:26.033 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.024+0000 7f921effd700 1 -- 172.21.15.38:0/486903974 <== mon.2 v2:172.21.15.122:3300/0 5 ==== fsmap(e 41) v1 ==== 7490+0+0 (secure 0 0 0) 0x7f923001d700 con 0x55e488b8dfe0 2023-03-09T14:57:26.033 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.024+0000 7f9251fdab80 10 client.25919 fetch_fsmap finished waiting for FSMap version 41 2023-03-09T14:57:26.033 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.024+0000 7f9251fdab80 10 client.25919 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T14:57:26.033 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.024+0000 7f9251fdab80 20 client.25919 populate_metadata read hostname 'smithi038' 2023-03-09T14:57:26.034 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.024+0000 7f9251fdab80 1 --2- 172.21.15.38:0/486903974 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55e488ba76b0 0x55e488ba9ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:57:26.034 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.024+0000 7f9251fdab80 4 client.25919 mds_command: new command op to 24530 tid=0[{"prefix": "get_command_descriptions"}] 2023-03-09T14:57:26.034 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.024+0000 7f9251fdab80 1 -- 172.21.15.38:0/486903974 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e488963f60 con 0x55e488ba76b0 2023-03-09T14:57:26.034 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.025+0000 7f9236ffd700 1 --2- 172.21.15.38:0/486903974 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55e488ba76b0 0x55e488ba9ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:57:26.034 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.025+0000 7f921effd700 10 client.25919 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:57:26.034 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.025+0000 7f9236ffd700 1 --2- 172.21.15.38:0/486903974 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55e488ba76b0 0x55e488ba9ac0 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=42a2024129fd34bc server_cookie=9f9095aa1c63f201 in_seq=0 out_seq=0 2023-03-09T14:57:26.034 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.026+0000 7f921effd700 1 -- 172.21.15.38:0/486903974 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18278 (crc 0 0 0) 0x55e488963f60 con 0x55e488ba76b0 2023-03-09T14:57:26.035 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.026+0000 7f921effd700 10 client.25919 handle_command_reply: tid=0 2023-03-09T14:57:26.035 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.026+0000 7f9251fdab80 1 client.25919 shutdown 2023-03-09T14:57:26.035 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.026+0000 7f9251fdab80 1 -- 172.21.15.38:0/486903974 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e488b8dfe0 msgr2=0x55e488b9a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:57:26.035 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.026+0000 7f9251fdab80 1 --2- 172.21.15.38:0/486903974 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e488b8dfe0 0x55e488b9a190 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f923000b6c0 tx=0x7f92300a1170 comp rx=0 tx=0).stop 2023-03-09T14:57:26.035 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.026+0000 7f9251fdab80 1 -- 172.21.15.38:0/486903974 shutdown_connections 2023-03-09T14:57:26.035 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.026+0000 7f9251fdab80 1 --2- 172.21.15.38:0/486903974 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55e488ba76b0 0x55e488ba9ac0 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:57:26.035 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.026+0000 7f9251fdab80 1 --2- 172.21.15.38:0/486903974 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e488b94140 0x55e488b97b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:57:26.036 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.026+0000 7f921effd700 0 client.25919 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:57:26.036 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.026+0000 7f9251fdab80 1 --2- 172.21.15.38:0/486903974 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e488b8dfe0 0x55e488b9a190 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:57:26.036 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.026+0000 7f9251fdab80 1 --2- 172.21.15.38:0/486903974 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e488b93830 0x55e488b97410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:57:26.036 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.026+0000 7f9251fdab80 1 -- 172.21.15.38:0/486903974 >> 172.21.15.38:0/486903974 conn(0x55e488b88a20 msgr2=0x55e488b88df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:57:26.036 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.026+0000 7f9251fdab80 1 -- 172.21.15.38:0/486903974 shutdown_connections 2023-03-09T14:57:26.036 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.026+0000 7f9251fdab80 1 -- 172.21.15.38:0/486903974 wait complete. 2023-03-09T14:57:26.037 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.026+0000 7f9251fdab80 20 client.25919 trim_cache size 0 max 16384 2023-03-09T14:57:26.060 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.055+0000 7f9251fdab80 1 Processor -- start 2023-03-09T14:57:26.060 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.055+0000 7f9251fdab80 1 -- start start 2023-03-09T14:57:26.060 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.056+0000 7f9251fdab80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e488b88a20 0x55e488b86f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:57:26.061 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.056+0000 7f9251fdab80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e488b8dfe0 0x55e488b88df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:57:26.061 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.056+0000 7f9251fdab80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e488b93830 0x55e488b99d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:57:26.061 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.056+0000 7f9251fdab80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e488b89fb0 con 0x55e488b8dfe0 2023-03-09T14:57:26.061 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.056+0000 7f9251fdab80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e488b920d0 con 0x55e488b93830 2023-03-09T14:57:26.061 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.056+0000 7f9251fdab80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e488b97ac0 con 0x55e488b88a20 2023-03-09T14:57:26.061 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.056+0000 7f9236ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e488b8dfe0 0x55e488b88df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:57:26.061 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.056+0000 7f923ffff700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e488b93830 0x55e488b99d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:57:26.062 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.056+0000 7f9236ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e488b8dfe0 0x55e488b88df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42100/0 (socket says 172.21.15.38:42100) 2023-03-09T14:57:26.062 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.056+0000 7f9236ffd700 1 -- 172.21.15.38:0/3833933257 learned_addr learned my addr 172.21.15.38:0/3833933257 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:57:26.062 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.056+0000 7f9236ffd700 1 -- 172.21.15.38:0/3833933257 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e488b88a20 msgr2=0x55e488b86f70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-03-09T14:57:26.062 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.056+0000 7f9236ffd700 1 --2- 172.21.15.38:0/3833933257 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e488b88a20 0x55e488b86f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:57:26.062 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.056+0000 7f9236ffd700 1 -- 172.21.15.38:0/3833933257 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e488b93830 msgr2=0x55e488b99d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:57:26.062 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.056+0000 7f9236ffd700 1 --2- 172.21.15.38:0/3833933257 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e488b93830 0x55e488b99d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:57:26.062 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.056+0000 7f9236ffd700 1 -- 172.21.15.38:0/3833933257 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e48890dde0 con 0x55e488b8dfe0 2023-03-09T14:57:26.063 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.057+0000 7f9236ffd700 1 --2- 172.21.15.38:0/3833933257 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e488b8dfe0 0x55e488b88df0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x55e488baa3d0 tx=0x7f9228014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34a7d4519a29cd5e server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:57:26.063 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.057+0000 7f92367fc700 1 -- 172.21.15.38:0/3833933257 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9228013040 con 0x55e488b8dfe0 2023-03-09T14:57:26.063 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.057+0000 7f92367fc700 1 -- 172.21.15.38:0/3833933257 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f9228021040 con 0x55e488b8dfe0 2023-03-09T14:57:26.063 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.057+0000 7f92367fc700 1 -- 172.21.15.38:0/3833933257 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f922801a7f0 con 0x55e488b8dfe0 2023-03-09T14:57:26.063 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.058+0000 7f9251fdab80 1 -- 172.21.15.38:0/3833933257 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e488b8dfe0 msgr2=0x55e488b88df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:57:26.063 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.058+0000 7f9251fdab80 1 --2- 172.21.15.38:0/3833933257 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e488b8dfe0 0x55e488b88df0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x55e488baa3d0 tx=0x7f9228014000 comp rx=0 tx=0).stop 2023-03-09T14:57:26.064 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.060+0000 7f9251fdab80 1 -- 172.21.15.38:0/3833933257 shutdown_connections 2023-03-09T14:57:26.064 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.060+0000 7f9251fdab80 1 --2- 172.21.15.38:0/3833933257 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e488b93830 0x55e488b99d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:57:26.064 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.060+0000 7f9251fdab80 1 --2- 172.21.15.38:0/3833933257 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e488b88a20 0x55e488b86f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:57:26.064 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.060+0000 7f9251fdab80 1 --2- 172.21.15.38:0/3833933257 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e488b8dfe0 0x55e488b88df0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:57:26.064 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.060+0000 7f9251fdab80 1 -- 172.21.15.38:0/3833933257 >> 172.21.15.38:0/3833933257 conn(0x55e488b87ef0 msgr2=0x55e488ba3e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:57:26.065 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.061+0000 7f9251fdab80 1 -- 172.21.15.38:0/3833933257 shutdown_connections 2023-03-09T14:57:26.065 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.062+0000 7f9251fdab80 1 -- 172.21.15.38:0/3833933257 wait complete. 2023-03-09T14:57:26.065 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.062+0000 7f9251fdab80 1 -- start start 2023-03-09T14:57:26.065 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.062+0000 7f9251fdab80 1 Processor -- start 2023-03-09T14:57:26.066 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.062+0000 7f9251fdab80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e488b88a20 0x55e488b96220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:57:26.066 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.062+0000 7f9251fdab80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e488b8dfe0 0x55e488b96760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:57:26.066 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.062+0000 7f9251fdab80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e488b93830 0x55e488b96ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:57:26.067 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.062+0000 7f9251fdab80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e488bfb700 con 0x55e488b88a20 2023-03-09T14:57:26.067 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.062+0000 7f9251fdab80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e488bfb870 con 0x55e488b93830 2023-03-09T14:57:26.067 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.062+0000 7f9251fdab80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e488bfba30 con 0x55e488b8dfe0 2023-03-09T14:57:26.067 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.063+0000 7f9251fdab80 10 client.0 _ll_register_callbacks cb 0x55e488a85520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:57:26.067 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.063+0000 7f923ffff700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e488b93830 0x55e488b96ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:57:26.067 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.063+0000 7f923ffff700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e488b93830 0x55e488b96ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.38:51292/0 (socket says 172.21.15.38:51292) 2023-03-09T14:57:26.067 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.063+0000 7f923ffff700 1 -- 172.21.15.38:0/3571098167 learned_addr learned my addr 172.21.15.38:0/3571098167 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:57:26.067 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.063+0000 7f923f7fe700 1 --2- 172.21.15.38:0/3571098167 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e488b88a20 0x55e488b96220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:57:26.068 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.063+0000 7f923f7fe700 1 -- 172.21.15.38:0/3571098167 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e488b8dfe0 msgr2=0x55e488b96760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-03-09T14:57:26.068 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.063+0000 7f923f7fe700 1 --2- 172.21.15.38:0/3571098167 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e488b8dfe0 0x55e488b96760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:57:26.068 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.063+0000 7f923f7fe700 1 -- 172.21.15.38:0/3571098167 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e488b93830 msgr2=0x55e488b96ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:57:26.068 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.063+0000 7f923f7fe700 1 --2- 172.21.15.38:0/3571098167 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e488b93830 0x55e488b96ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:57:26.068 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.063+0000 7f923f7fe700 1 -- 172.21.15.38:0/3571098167 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e48890c2b0 con 0x55e488b88a20 2023-03-09T14:57:26.069 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.063+0000 7f923ffff700 1 --2- 172.21.15.38:0/3571098167 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e488b93830 0x55e488b96ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:57:26.069 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.064+0000 7f923f7fe700 1 --2- 172.21.15.38:0/3571098167 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e488b88a20 0x55e488b96220 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f9230095510 tx=0x7f923001db80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f8f94ef98b83f67 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:57:26.069 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.064+0000 7f921effd700 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2023-03-09T14:57:26.069 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.064+0000 7f921effd700 1 -- 172.21.15.38:0/3571098167 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9230019320 con 0x55e488b88a20 2023-03-09T14:57:26.069 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.064+0000 7f921effd700 1 -- 172.21.15.38:0/3571098167 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(244..244 src has 1..244) v4 ==== 9665+0+0 (secure 0 0 0) 0x7f9230098500 con 0x55e488b88a20 2023-03-09T14:57:26.069 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.064+0000 7f9251fdab80 1 -- 172.21.15.38:0/3571098167 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e488a9d1b0 con 0x55e488b88a20 2023-03-09T14:57:26.070 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.064+0000 7f921effd700 1 -- 172.21.15.38:0/3571098167 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9230097f70 con 0x55e488b88a20 2023-03-09T14:57:26.070 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.064+0000 7f9251fdab80 10 client.16284 fetch_fsmap learned FSMap version 41 2023-03-09T14:57:26.070 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.064+0000 7f9251fdab80 1 -- 172.21.15.38:0/3571098167 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) v3 -- 0x55e488934250 con 0x55e488b88a20 2023-03-09T14:57:26.071 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.065+0000 7f921effd700 1 -- 172.21.15.38:0/3571098167 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55e488a9d1b0 con 0x55e488b88a20 2023-03-09T14:57:26.071 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.065+0000 7f921effd700 1 -- 172.21.15.38:0/3571098167 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 41) v1 ==== 7490+0+0 (secure 0 0 0) 0x7f9230099f00 con 0x55e488b88a20 2023-03-09T14:57:26.072 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.065+0000 7f9251fdab80 10 client.16284 fetch_fsmap finished waiting for FSMap version 41 2023-03-09T14:57:26.072 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.065+0000 7f9251fdab80 10 client.16284 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T14:57:26.072 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.065+0000 7f9251fdab80 20 client.16284 populate_metadata read hostname 'smithi038' 2023-03-09T14:57:26.072 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.065+0000 7f9251fdab80 1 --2- 172.21.15.38:0/3571098167 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55e488b94140 0x55e488bfec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:57:26.072 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.065+0000 7f9251fdab80 4 client.16284 mds_command: new command op to 24530 tid=0[{"prefix": "scrub status"}] 2023-03-09T14:57:26.072 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.065+0000 7f9251fdab80 1 -- 172.21.15.38:0/3571098167 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55e488a871c0 con 0x55e488b94140 2023-03-09T14:57:26.072 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.065+0000 7f9236ffd700 1 --2- 172.21.15.38:0/3571098167 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55e488b94140 0x55e488bfec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:57:26.073 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.066+0000 7f921effd700 10 client.16284 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:57:26.073 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.066+0000 7f9236ffd700 1 --2- 172.21.15.38:0/3571098167 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55e488b94140 0x55e488bfec00 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ad5638c33a1b41 server_cookie=5e1330ed8397143f in_seq=0 out_seq=0 2023-03-09T14:57:26.073 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.066+0000 7f921effd700 1 -- 172.21.15.38:0/3571098167 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+265 (crc 0 0 0) 0x55e488a871c0 con 0x55e488b94140 2023-03-09T14:57:26.073 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.066+0000 7f921effd700 10 client.16284 handle_command_reply: tid=0 2023-03-09T14:57:26.073 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.066+0000 7f9251fdab80 1 client.16284 shutdown 2023-03-09T14:57:26.073 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.067+0000 7f9251fdab80 1 -- 172.21.15.38:0/3571098167 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e488b88a20 msgr2=0x55e488b96220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:57:26.074 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.067+0000 7f9251fdab80 1 --2- 172.21.15.38:0/3571098167 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e488b88a20 0x55e488b96220 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f9230095510 tx=0x7f923001db80 comp rx=0 tx=0).stop 2023-03-09T14:57:26.074 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.068+0000 7f9251fdab80 1 -- 172.21.15.38:0/3571098167 shutdown_connections 2023-03-09T14:57:26.074 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.068+0000 7f9251fdab80 1 --2- 172.21.15.38:0/3571098167 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55e488b94140 0x55e488bfec00 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:57:26.074 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.068+0000 7f9251fdab80 1 --2- 172.21.15.38:0/3571098167 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55e488b88a20 0x55e488b96220 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:57:26.074 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.068+0000 7f9251fdab80 1 --2- 172.21.15.38:0/3571098167 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55e488b93830 0x55e488b96ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:57:26.074 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.068+0000 7f9251fdab80 1 --2- 172.21.15.38:0/3571098167 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55e488b8dfe0 0x55e488b96760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:57:26.074 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.068+0000 7f9251fdab80 1 -- 172.21.15.38:0/3571098167 >> 172.21.15.38:0/3571098167 conn(0x55e488b87ef0 msgr2=0x55e488b8e3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:57:26.074 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.068+0000 7f921effd700 0 client.16284 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:57:26.075 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.069+0000 7f9251fdab80 1 -- 172.21.15.38:0/3571098167 shutdown_connections 2023-03-09T14:57:26.075 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.069+0000 7f9251fdab80 1 -- 172.21.15.38:0/3571098167 wait complete. 2023-03-09T14:57:26.075 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.069+0000 7f9251fdab80 20 client.16284 trim_cache size 0 max 16384 2023-03-09T14:57:26.075 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2023-03-09T14:57:26.075 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "scrub active (1 inodes in the stack)", 2023-03-09T14:57:26.076 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": { 2023-03-09T14:57:26.076 INFO:teuthology.orchestra.run.smithi038.stdout: "a14659f4-d7ea-4c5a-b15b-6138af3f909f": { 2023-03-09T14:57:26.076 INFO:teuthology.orchestra.run.smithi038.stdout: "path": "/", 2023-03-09T14:57:26.076 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "a14659f4-d7ea-4c5a-b15b-6138af3f909f", 2023-03-09T14:57:26.076 INFO:teuthology.orchestra.run.smithi038.stdout: "options": "recursive,force" 2023-03-09T14:57:26.076 INFO:teuthology.orchestra.run.smithi038.stdout: } 2023-03-09T14:57:26.076 INFO:teuthology.orchestra.run.smithi038.stdout: } 2023-03-09T14:57:26.076 INFO:teuthology.orchestra.run.smithi038.stdout:} 2023-03-09T14:57:26.077 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.069+0000 7f9245e51700 1 -- 172.21.15.38:0/1551858936 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f922c09f510 msgr2=0x7f922c0a19c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:57:26.077 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.069+0000 7f9245e51700 1 --2- 172.21.15.38:0/1551858936 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f922c09f510 0x7f922c0a19c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9240112c60 tx=0x7f9228010040 comp rx=0 tx=0).stop 2023-03-09T14:57:26.077 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.069+0000 7f9245e51700 1 -- 172.21.15.38:0/1551858936 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9240107780 msgr2=0x7f92401115f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:57:26.078 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.069+0000 7f9245e51700 1 --2- 172.21.15.38:0/1551858936 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9240107780 0x7f92401115f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f9230000f80 tx=0x7f9230007580 comp rx=0 tx=0).stop 2023-03-09T14:57:26.078 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.070+0000 7f9245e51700 1 -- 172.21.15.38:0/1551858936 shutdown_connections 2023-03-09T14:57:26.078 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.070+0000 7f9245e51700 1 --2- 172.21.15.38:0/1551858936 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f922c09f510 0x7f922c0a19c0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:57:26.078 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.070+0000 7f9245e51700 1 --2- 172.21.15.38:0/1551858936 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f924010ce50 0x7f9240111bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:57:26.078 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.070+0000 7f9245e51700 1 --2- 172.21.15.38:0/1551858936 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f9240107780 0x7f92401115f0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:57:26.078 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.070+0000 7f9245e51700 1 --2- 172.21.15.38:0/1551858936 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f9240108130 0x7f9240122630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:57:26.078 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.070+0000 7f9245e51700 1 -- 172.21.15.38:0/1551858936 >> 172.21.15.38:0/1551858936 conn(0x7f924006b450 msgr2=0x7f9240118530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:57:26.079 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.070+0000 7f9245e51700 1 -- 172.21.15.38:0/1551858936 shutdown_connections 2023-03-09T14:57:26.079 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:26.070+0000 7f9245e51700 1 -- 172.21.15.38:0/1551858936 wait complete. 2023-03-09T14:57:26.092 INFO:tasks.cephfs.filesystem:scrub status for tag:a14659f4-d7ea-4c5a-b15b-6138af3f909f - {'path': '/', 'tag': 'a14659f4-d7ea-4c5a-b15b-6138af3f909f', 'options': 'recursive,force'} 2023-03-09T14:57:26.093 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:57:25 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T14:57:26.093 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:57:25 smithi038 ceph-mon[92422]: pgmap v492: 129 pgs: 3 active+clean+snaptrim, 3 active+clean+snaptrim_wait, 123 active+clean; 12 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 12 MiB/s wr, 1.40k op/s 2023-03-09T14:57:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:57:25 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T14:57:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:57:25 smithi122 ceph-mon[93869]: pgmap v492: 129 pgs: 3 active+clean+snaptrim, 3 active+clean+snaptrim_wait, 123 active+clean; 12 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 12 MiB/s wr, 1.40k op/s 2023-03-09T14:57:26.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:57:25 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T14:57:26.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:57:25 smithi181 ceph-mon[94189]: pgmap v492: 129 pgs: 3 active+clean+snaptrim, 3 active+clean+snaptrim_wait, 123 active+clean; 12 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 12 MiB/s wr, 1.40k op/s 2023-03-09T14:57:28.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:57:27 smithi122 ceph-mon[93869]: pgmap v493: 129 pgs: 129 active+clean; 12 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 13 MiB/s wr, 1.52k op/s 2023-03-09T14:57:28.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:57:27 smithi122 ceph-mon[93869]: osdmap e245: 12 total, 12 up, 12 in 2023-03-09T14:57:28.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:57:27 smithi181 ceph-mon[94189]: pgmap v493: 129 pgs: 129 active+clean; 12 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 13 MiB/s wr, 1.52k op/s 2023-03-09T14:57:28.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:57:27 smithi181 ceph-mon[94189]: osdmap e245: 12 total, 12 up, 12 in 2023-03-09T14:57:28.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:57:27 smithi038 ceph-mon[92422]: pgmap v493: 129 pgs: 129 active+clean; 12 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 13 MiB/s wr, 1.52k op/s 2023-03-09T14:57:28.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:57:27 smithi038 ceph-mon[92422]: osdmap e245: 12 total, 12 up, 12 in 2023-03-09T14:57:28.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 420.0 s, 915.7 tps, lat 103.733 ms stddev 196.187 2023-03-09T14:57:30.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:57:29 smithi122 ceph-mon[93869]: pgmap v495: 129 pgs: 129 active+clean; 12 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 12 MiB/s wr, 1.52k op/s 2023-03-09T14:57:30.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:57:29 smithi181 ceph-mon[94189]: pgmap v495: 129 pgs: 129 active+clean; 12 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 12 MiB/s wr, 1.52k op/s 2023-03-09T14:57:30.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:57:29 smithi038 ceph-mon[92422]: pgmap v495: 129 pgs: 129 active+clean; 12 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 12 MiB/s wr, 1.52k op/s 2023-03-09T14:57:32.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:57:31 smithi122 ceph-mon[93869]: pgmap v496: 129 pgs: 129 active+clean; 12 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 17 MiB/s wr, 2.05k op/s 2023-03-09T14:57:32.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:57:31 smithi181 ceph-mon[94189]: pgmap v496: 129 pgs: 129 active+clean; 12 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 17 MiB/s wr, 2.05k op/s 2023-03-09T14:57:32.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:57:31 smithi038 ceph-mon[92422]: pgmap v496: 129 pgs: 129 active+clean; 12 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 17 MiB/s wr, 2.05k op/s 2023-03-09T14:57:33.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 425.0 s, 683.3 tps, lat 152.396 ms stddev 119.128 2023-03-09T14:57:34.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:57:33 smithi122 ceph-mon[93869]: pgmap v497: 129 pgs: 129 active+clean; 12 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 19 MiB/s wr, 2.13k op/s 2023-03-09T14:57:34.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:57:33 smithi181 ceph-mon[94189]: pgmap v497: 129 pgs: 129 active+clean; 12 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 19 MiB/s wr, 2.13k op/s 2023-03-09T14:57:34.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:57:33 smithi038 ceph-mon[92422]: pgmap v497: 129 pgs: 129 active+clean; 12 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 19 MiB/s wr, 2.13k op/s 2023-03-09T14:57:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:57:36 smithi122 ceph-mon[93869]: pgmap v498: 129 pgs: 129 active+clean; 12 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 19 MiB/s wr, 2.23k op/s 2023-03-09T14:57:36.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:57:36 smithi181 ceph-mon[94189]: pgmap v498: 129 pgs: 129 active+clean; 12 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 19 MiB/s wr, 2.23k op/s 2023-03-09T14:57:36.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:57:36 smithi038 ceph-mon[92422]: pgmap v498: 129 pgs: 129 active+clean; 12 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 19 MiB/s wr, 2.23k op/s 2023-03-09T14:57:38.481 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:57:38 smithi122 ceph-mon[93869]: pgmap v499: 129 pgs: 129 active+clean; 12 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 20 MiB/s wr, 2.22k op/s 2023-03-09T14:57:38.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:57:38 smithi181 ceph-mon[94189]: pgmap v499: 129 pgs: 129 active+clean; 12 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 20 MiB/s wr, 2.22k op/s 2023-03-09T14:57:38.504 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:57:38 smithi038 ceph-mon[92422]: pgmap v499: 129 pgs: 129 active+clean; 12 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 20 MiB/s wr, 2.22k op/s 2023-03-09T14:57:38.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 430.0 s, 699.1 tps, lat 130.868 ms stddev 203.550 2023-03-09T14:57:40.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:57:40 smithi122 ceph-mon[93869]: pgmap v500: 129 pgs: 129 active+clean; 12 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 20 MiB/s wr, 2.17k op/s 2023-03-09T14:57:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:57:40 smithi181 ceph-mon[94189]: pgmap v500: 129 pgs: 129 active+clean; 12 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 20 MiB/s wr, 2.17k op/s 2023-03-09T14:57:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:57:40 smithi038 ceph-mon[92422]: pgmap v500: 129 pgs: 129 active+clean; 12 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 20 MiB/s wr, 2.17k op/s 2023-03-09T14:57:42.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:57:42 smithi122 ceph-mon[93869]: pgmap v501: 129 pgs: 129 active+clean; 12 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 24 MiB/s wr, 2.68k op/s 2023-03-09T14:57:42.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:57:42 smithi181 ceph-mon[94189]: pgmap v501: 129 pgs: 129 active+clean; 12 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 24 MiB/s wr, 2.68k op/s 2023-03-09T14:57:42.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:57:42 smithi038 ceph-mon[92422]: pgmap v501: 129 pgs: 129 active+clean; 12 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 24 MiB/s wr, 2.68k op/s 2023-03-09T14:57:43.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 435.0 s, 549.6 tps, lat 174.366 ms stddev 457.557 2023-03-09T14:57:44.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:57:43 smithi122 ceph-mon[93869]: pgmap v502: 129 pgs: 129 active+clean; 12 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 22 MiB/s wr, 2.38k op/s 2023-03-09T14:57:44.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:57:43 smithi181 ceph-mon[94189]: pgmap v502: 129 pgs: 129 active+clean; 12 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 22 MiB/s wr, 2.38k op/s 2023-03-09T14:57:44.257 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:57:43 smithi038 ceph-mon[92422]: pgmap v502: 129 pgs: 129 active+clean; 12 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 22 MiB/s wr, 2.38k op/s 2023-03-09T14:57:45.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:57:44 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:57:45.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:57:44 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:57:45.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:57:44 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:57:46.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:57:45 smithi122 ceph-mon[93869]: pgmap v503: 129 pgs: 129 active+clean; 12 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 21 MiB/s wr, 2.45k op/s 2023-03-09T14:57:46.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:57:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:57:46.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:57:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:57:46.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:57:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:57:46.254 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:57:45 smithi038 ceph-mon[92422]: pgmap v503: 129 pgs: 129 active+clean; 12 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 21 MiB/s wr, 2.45k op/s 2023-03-09T14:57:46.254 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:57:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:57:46.254 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:57:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:57:46.254 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:57:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:57:46.261 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:57:45 smithi181 ceph-mon[94189]: pgmap v503: 129 pgs: 129 active+clean; 12 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 21 MiB/s wr, 2.45k op/s 2023-03-09T14:57:46.261 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:57:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:57:46.261 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:57:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:57:46.261 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:57:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:57:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:57:47 smithi122 ceph-mon[93869]: pgmap v504: 129 pgs: 129 active+clean; 12 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 25 MiB/s wr, 2.83k op/s 2023-03-09T14:57:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:57:47 smithi038 ceph-mon[92422]: pgmap v504: 129 pgs: 129 active+clean; 12 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 25 MiB/s wr, 2.83k op/s 2023-03-09T14:57:48.614 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:57:47 smithi181 ceph-mon[94189]: pgmap v504: 129 pgs: 129 active+clean; 12 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 25 MiB/s wr, 2.83k op/s 2023-03-09T14:57:48.803 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 440.0 s, 771.3 tps, lat 145.811 ms stddev 323.354 2023-03-09T14:57:50.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:57:50 smithi122 ceph-mon[93869]: pgmap v505: 129 pgs: 129 active+clean; 12 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 20 MiB/s wr, 2.42k op/s 2023-03-09T14:57:50.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:57:50 smithi181 ceph-mon[94189]: pgmap v505: 129 pgs: 129 active+clean; 12 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 20 MiB/s wr, 2.42k op/s 2023-03-09T14:57:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:57:50 smithi038 ceph-mon[92422]: pgmap v505: 129 pgs: 129 active+clean; 12 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 20 MiB/s wr, 2.42k op/s 2023-03-09T14:57:52.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:57:52 smithi122 ceph-mon[93869]: pgmap v506: 129 pgs: 129 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 24 MiB/s wr, 2.94k op/s 2023-03-09T14:57:52.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:57:52 smithi181 ceph-mon[94189]: pgmap v506: 129 pgs: 129 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 24 MiB/s wr, 2.94k op/s 2023-03-09T14:57:52.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:57:52 smithi038 ceph-mon[92422]: pgmap v506: 129 pgs: 129 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 24 MiB/s wr, 2.94k op/s 2023-03-09T14:57:53.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 445.0 s, 413.3 tps, lat 213.496 ms stddev 359.855 2023-03-09T14:57:54.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:57:53 smithi122 ceph-mon[93869]: pgmap v507: 129 pgs: 129 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 20 MiB/s wr, 2.27k op/s 2023-03-09T14:57:54.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:57:53 smithi181 ceph-mon[94189]: pgmap v507: 129 pgs: 129 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 20 MiB/s wr, 2.27k op/s 2023-03-09T14:57:54.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:57:53 smithi038 ceph-mon[92422]: pgmap v507: 129 pgs: 129 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 20 MiB/s wr, 2.27k op/s 2023-03-09T14:57:56.094 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-03-09T14:57:56.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:57:55 smithi038 ceph-mon[92422]: pgmap v508: 129 pgs: 129 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 18 MiB/s wr, 2.10k op/s 2023-03-09T14:57:56.408 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.401+0000 7f80095f0700 1 -- 172.21.15.38:0/1492522878 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80041196d0 msgr2=0x7f800411bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:57:56.408 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.401+0000 7f80095f0700 1 --2- 172.21.15.38:0/1492522878 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80041196d0 0x7f800411bab0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f7fec0098f0 tx=0x7f7fec009c00 comp rx=0 tx=0).stop 2023-03-09T14:57:56.408 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.405+0000 7f80095f0700 1 -- 172.21.15.38:0/1492522878 shutdown_connections 2023-03-09T14:57:56.409 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.405+0000 7f80095f0700 1 --2- 172.21.15.38:0/1492522878 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f80041196d0 0x7f800411bab0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:57:56.409 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.405+0000 7f80095f0700 1 --2- 172.21.15.38:0/1492522878 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f8004107780 0x7f800410e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:57:56.409 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.405+0000 7f80095f0700 1 --2- 172.21.15.38:0/1492522878 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8004107b50 0x7f8004107fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:57:56.409 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.405+0000 7f80095f0700 1 -- 172.21.15.38:0/1492522878 >> 172.21.15.38:0/1492522878 conn(0x7f800406b450 msgr2=0x7f80040712c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:57:56.411 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.408+0000 7f80095f0700 1 -- 172.21.15.38:0/1492522878 shutdown_connections 2023-03-09T14:57:56.412 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.408+0000 7f80095f0700 1 -- 172.21.15.38:0/1492522878 wait complete. 2023-03-09T14:57:56.412 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.409+0000 7f80095f0700 1 Processor -- start 2023-03-09T14:57:56.412 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.409+0000 7f80095f0700 1 -- start start 2023-03-09T14:57:56.412 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.409+0000 7f80095f0700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8004107780 0x7f80041a4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:57:56.413 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.409+0000 7f80095f0700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8004107b50 0x7f80041a5360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:57:56.413 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.409+0000 7f80095f0700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f80041196d0 0x7f80041a97c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:57:56.413 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.409+0000 7f80095f0700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80041a9e00 con 0x7f8004107b50 2023-03-09T14:57:56.413 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.409+0000 7f80095f0700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80041a9f70 con 0x7f8004107780 2023-03-09T14:57:56.413 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.409+0000 7f80095f0700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80041aa130 con 0x7f80041196d0 2023-03-09T14:57:56.414 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.410+0000 7f8002ffd700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8004107780 0x7f80041a4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:57:56.414 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.410+0000 7f8002ffd700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8004107780 0x7f80041a4e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.38:51306/0 (socket says 172.21.15.38:51306) 2023-03-09T14:57:56.415 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.410+0000 7f80027fc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8004107b50 0x7f80041a5360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:57:56.415 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.410+0000 7f8002ffd700 1 -- 172.21.15.38:0/3395206744 learned_addr learned my addr 172.21.15.38:0/3395206744 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:57:56.415 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.410+0000 7f80037fe700 1 --2- 172.21.15.38:0/3395206744 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f80041196d0 0x7f80041a97c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:57:56.415 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.410+0000 7f8002ffd700 1 -- 172.21.15.38:0/3395206744 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f80041196d0 msgr2=0x7f80041a97c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:57:56.415 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.410+0000 7f8002ffd700 1 --2- 172.21.15.38:0/3395206744 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f80041196d0 0x7f80041a97c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:57:56.415 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.410+0000 7f8002ffd700 1 -- 172.21.15.38:0/3395206744 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8004107b50 msgr2=0x7f80041a5360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:57:56.415 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.410+0000 7f8002ffd700 1 --2- 172.21.15.38:0/3395206744 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8004107b50 0x7f80041a5360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:57:56.416 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.410+0000 7f8002ffd700 1 -- 172.21.15.38:0/3395206744 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fec009520 con 0x7f8004107780 2023-03-09T14:57:56.416 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.410+0000 7f80027fc700 1 --2- 172.21.15.38:0/3395206744 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8004107b50 0x7f80041a5360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:57:56.416 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.411+0000 7f8002ffd700 1 --2- 172.21.15.38:0/3395206744 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8004107780 0x7f80041a4e20 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc00d900 tx=0x7f7ffc00dc10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:57:56.416 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.411+0000 7f7ffbfff700 1 -- 172.21.15.38:0/3395206744 <== mon.2 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ffc00f890 con 0x7f8004107780 2023-03-09T14:57:56.416 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.411+0000 7f80095f0700 1 -- 172.21.15.38:0/3395206744 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f800411ed80 con 0x7f8004107780 2023-03-09T14:57:56.416 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.411+0000 7f7ffbfff700 1 -- 172.21.15.38:0/3395206744 <== mon.2 v2:172.21.15.122:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7ffc010460 con 0x7f8004107780 2023-03-09T14:57:56.417 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.411+0000 7f7ffbfff700 1 -- 172.21.15.38:0/3395206744 <== mon.2 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ffc0195c0 con 0x7f8004107780 2023-03-09T14:57:56.417 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.411+0000 7f80095f0700 1 -- 172.21.15.38:0/3395206744 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f800411ebf0 con 0x7f8004107780 2023-03-09T14:57:56.419 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.413+0000 7f7ffbfff700 1 -- 172.21.15.38:0/3395206744 <== mon.2 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 25) v1 ==== 133295+0+0 (secure 0 0 0) 0x7f7ffc022070 con 0x7f8004107780 2023-03-09T14:57:56.419 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.414+0000 7f7ffbfff700 1 --2- 172.21.15.38:0/3395206744 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f7ff009f4d0 0x7f7ff00a1980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:57:56.419 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.414+0000 7f80027fc700 1 --2- 172.21.15.38:0/3395206744 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f7ff009f4d0 0x7f7ff00a1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:57:56.419 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.414+0000 7f7ffbfff700 1 -- 172.21.15.38:0/3395206744 <== mon.2 v2:172.21.15.122:3300/0 5 ==== osd_map(245..245 src has 1..245) v4 ==== 9577+0+0 (secure 0 0 0) 0x7f7ffc0c9fb0 con 0x7f8004107780 2023-03-09T14:57:56.419 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.414+0000 7f80027fc700 1 --2- 172.21.15.38:0/3395206744 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f7ff009f4d0 0x7f7ff00a1980 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f80041a6430 tx=0x7f7ff400d040 comp rx=0 tx=0).ready entity=mgr.15558 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:57:56.419 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.415+0000 7f8015779b80 1 Processor -- start 2023-03-09T14:57:56.419 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.415+0000 7f8015779b80 1 -- start start 2023-03-09T14:57:56.420 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.415+0000 7f8015779b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557d3a870830 0x557d3a870c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:57:56.420 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.415+0000 7f8015779b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557d3a871140 0x557d3a86aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:57:56.420 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.415+0000 7f8015779b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d3a86afe0 0x557d3a86b410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:57:56.420 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.415+0000 7f8015779b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d3a754da0 con 0x557d3a871140 2023-03-09T14:57:56.420 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.415+0000 7f8015779b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d3a7c3780 con 0x557d3a870830 2023-03-09T14:57:56.420 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.415+0000 7f8015779b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d3a7bccb0 con 0x557d3a86afe0 2023-03-09T14:57:56.420 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.415+0000 7f80027fc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557d3a871140 0x557d3a86aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:57:56.421 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.415+0000 7f8002ffd700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557d3a870830 0x557d3a870c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:57:56.421 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.415+0000 7f80027fc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557d3a871140 0x557d3a86aaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42126/0 (socket says 172.21.15.38:42126) 2023-03-09T14:57:56.421 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.415+0000 7f80027fc700 1 -- 172.21.15.38:0/386746540 learned_addr learned my addr 172.21.15.38:0/386746540 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:57:56.421 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.415+0000 7f80037fe700 1 --2- 172.21.15.38:0/386746540 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d3a86afe0 0x557d3a86b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:57:56.421 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.415+0000 7f80027fc700 1 -- 172.21.15.38:0/386746540 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d3a86afe0 msgr2=0x557d3a86b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:57:56.421 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.415+0000 7f80027fc700 1 --2- 172.21.15.38:0/386746540 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d3a86afe0 0x557d3a86b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:57:56.421 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.415+0000 7f80027fc700 1 -- 172.21.15.38:0/386746540 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557d3a870830 msgr2=0x557d3a870c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:57:56.421 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.415+0000 7f80027fc700 1 --2- 172.21.15.38:0/386746540 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557d3a870830 0x557d3a870c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:57:56.422 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.416+0000 7f80027fc700 1 -- 172.21.15.38:0/386746540 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557d3a783190 con 0x557d3a871140 2023-03-09T14:57:56.422 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.416+0000 7f8002ffd700 1 --2- 172.21.15.38:0/386746540 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557d3a870830 0x557d3a870c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:57:56.422 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.416+0000 7f80027fc700 1 --2- 172.21.15.38:0/386746540 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557d3a871140 0x557d3a86aaa0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f7ff400a9a0 tx=0x7f7ff4011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9031e245356d324 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:57:56.422 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.416+0000 7f7ff97fa700 1 -- 172.21.15.38:0/386746540 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff4010040 con 0x557d3a871140 2023-03-09T14:57:56.422 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.416+0000 7f7ff97fa700 1 -- 172.21.15.38:0/386746540 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7ff401c040 con 0x557d3a871140 2023-03-09T14:57:56.422 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.416+0000 7f7ff97fa700 1 -- 172.21.15.38:0/386746540 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff4020d80 con 0x557d3a871140 2023-03-09T14:57:56.426 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.417+0000 7f8015779b80 1 -- 172.21.15.38:0/386746540 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557d3a871140 msgr2=0x557d3a86aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:57:56.426 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.417+0000 7f8015779b80 1 --2- 172.21.15.38:0/386746540 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557d3a871140 0x557d3a86aaa0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f7ff400a9a0 tx=0x7f7ff4011000 comp rx=0 tx=0).stop 2023-03-09T14:57:56.426 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.417+0000 7f8015779b80 1 -- 172.21.15.38:0/386746540 shutdown_connections 2023-03-09T14:57:56.427 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.417+0000 7f8015779b80 1 --2- 172.21.15.38:0/386746540 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d3a86afe0 0x557d3a86b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:57:56.427 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.417+0000 7f8015779b80 1 --2- 172.21.15.38:0/386746540 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557d3a870830 0x557d3a870c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:57:56.427 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.417+0000 7f8015779b80 1 --2- 172.21.15.38:0/386746540 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557d3a871140 0x557d3a86aaa0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:57:56.427 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.417+0000 7f8015779b80 1 -- 172.21.15.38:0/386746540 >> 172.21.15.38:0/386746540 conn(0x557d3a865a20 msgr2=0x557d3a865df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:57:56.427 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.417+0000 7f8015779b80 1 -- 172.21.15.38:0/386746540 shutdown_connections 2023-03-09T14:57:56.427 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.417+0000 7f8015779b80 1 -- 172.21.15.38:0/386746540 wait complete. 2023-03-09T14:57:56.427 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.417+0000 7f8015779b80 1 -- start start 2023-03-09T14:57:56.428 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.417+0000 7f8015779b80 1 Processor -- start 2023-03-09T14:57:56.428 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.418+0000 7f8015779b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557d3a86afe0 0x557d3a8770d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:57:56.428 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.418+0000 7f8015779b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d3a870830 0x557d3a874350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:57:56.428 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.418+0000 7f8015779b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557d3a871140 0x557d3a8749d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:57:56.428 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.418+0000 7f8015779b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d3a640920 con 0x557d3a871140 2023-03-09T14:57:56.428 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.418+0000 7f8015779b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d3a875100 con 0x557d3a86afe0 2023-03-09T14:57:56.428 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.418+0000 7f8015779b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d3a880c40 con 0x557d3a870830 2023-03-09T14:57:56.428 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.418+0000 7f8015779b80 10 client.0 _ll_register_callbacks cb 0x557d3a762520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:57:56.429 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.418+0000 7f8002ffd700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557d3a86afe0 0x557d3a8770d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:57:56.429 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.418+0000 7f80037fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557d3a871140 0x557d3a8749d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:57:56.429 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.418+0000 7f8002ffd700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557d3a86afe0 0x557d3a8770d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.38:51320/0 (socket says 172.21.15.38:51320) 2023-03-09T14:57:56.429 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.418+0000 7f80027fc700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d3a870830 0x557d3a874350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:57:56.429 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.418+0000 7f8002ffd700 1 -- 172.21.15.38:0/206616468 learned_addr learned my addr 172.21.15.38:0/206616468 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:57:56.429 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.418+0000 7f8002ffd700 1 -- 172.21.15.38:0/206616468 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d3a870830 msgr2=0x557d3a874350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:57:56.429 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.418+0000 7f8002ffd700 1 --2- 172.21.15.38:0/206616468 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d3a870830 0x557d3a874350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:57:56.430 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.418+0000 7f8002ffd700 1 -- 172.21.15.38:0/206616468 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557d3a871140 msgr2=0x557d3a8749d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:57:56.430 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.418+0000 7f8002ffd700 1 --2- 172.21.15.38:0/206616468 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557d3a871140 0x557d3a8749d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:57:56.430 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.418+0000 7f8002ffd700 1 -- 172.21.15.38:0/206616468 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f800411ebf0 con 0x557d3a86afe0 2023-03-09T14:57:56.430 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.418+0000 7f80037fe700 1 --2- 172.21.15.38:0/206616468 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557d3a871140 0x557d3a8749d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:57:56.430 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.418+0000 7f80027fc700 1 --2- 172.21.15.38:0/206616468 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d3a870830 0x557d3a874350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:57:56.430 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.418+0000 7f8002ffd700 1 --2- 172.21.15.38:0/206616468 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557d3a86afe0 0x557d3a8770d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc09f7c0 tx=0x7f7ffc09fde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c81ab04f8005385 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:57:56.431 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.418+0000 7f7fea7fc700 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2023-03-09T14:57:56.431 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.418+0000 7f7fea7fc700 1 -- 172.21.15.38:0/206616468 <== mon.2 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ffc0971f0 con 0x557d3a86afe0 2023-03-09T14:57:56.431 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.419+0000 7f8015779b80 1 -- 172.21.15.38:0/206616468 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557d3a78a190 con 0x557d3a86afe0 2023-03-09T14:57:56.431 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.419+0000 7f7fea7fc700 1 -- 172.21.15.38:0/206616468 <== mon.2 v2:172.21.15.122:3300/0 2 ==== osd_map(245..245 src has 1..245) v4 ==== 9577+0+0 (secure 0 0 0) 0x7f7ffc010690 con 0x557d3a86afe0 2023-03-09T14:57:56.431 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.419+0000 7f7fea7fc700 1 -- 172.21.15.38:0/206616468 <== mon.2 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7ffc09f860 con 0x557d3a86afe0 2023-03-09T14:57:56.431 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.419+0000 7f8015779b80 10 client.25934 fetch_fsmap learned FSMap version 41 2023-03-09T14:57:56.431 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.419+0000 7f8015779b80 1 -- 172.21.15.38:0/206616468 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) v3 -- 0x7f7ff400a710 con 0x557d3a86afe0 2023-03-09T14:57:56.432 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.419+0000 7f7fea7fc700 1 -- 172.21.15.38:0/206616468 <== mon.2 v2:172.21.15.122:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x557d3a78a190 con 0x557d3a86afe0 2023-03-09T14:57:56.432 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.419+0000 7f7fea7fc700 1 -- 172.21.15.38:0/206616468 <== mon.2 v2:172.21.15.122:3300/0 5 ==== fsmap(e 41) v1 ==== 7490+0+0 (secure 0 0 0) 0x7f7ffc096a60 con 0x557d3a86afe0 2023-03-09T14:57:56.432 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.419+0000 7f8015779b80 10 client.25934 fetch_fsmap finished waiting for FSMap version 41 2023-03-09T14:57:56.432 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.419+0000 7f8015779b80 10 client.25934 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T14:57:56.432 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.419+0000 7f8015779b80 20 client.25934 populate_metadata read hostname 'smithi038' 2023-03-09T14:57:56.432 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.419+0000 7f8015779b80 1 --2- 172.21.15.38:0/206616468 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x557d3a884d40 0x557d3a887150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:57:56.432 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.419+0000 7f8015779b80 4 client.25934 mds_command: new command op to 24530 tid=0[{"prefix": "get_command_descriptions"}] 2023-03-09T14:57:56.433 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.419+0000 7f8015779b80 1 -- 172.21.15.38:0/206616468 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557d3a640f60 con 0x557d3a884d40 2023-03-09T14:57:56.433 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.419+0000 7f80027fc700 1 --2- 172.21.15.38:0/206616468 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x557d3a884d40 0x557d3a887150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:57:56.433 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.420+0000 7f7fea7fc700 10 client.25934 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:57:56.433 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.420+0000 7f80027fc700 1 --2- 172.21.15.38:0/206616468 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x557d3a884d40 0x557d3a887150 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e09c35105c523f27 server_cookie=756f482d641be65b in_seq=0 out_seq=0 2023-03-09T14:57:56.433 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.420+0000 7f7fea7fc700 1 -- 172.21.15.38:0/206616468 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18278 (crc 0 0 0) 0x557d3a640f60 con 0x557d3a884d40 2023-03-09T14:57:56.433 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.420+0000 7f7fea7fc700 10 client.25934 handle_command_reply: tid=0 2023-03-09T14:57:56.434 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.420+0000 7f8015779b80 1 client.25934 shutdown 2023-03-09T14:57:56.434 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.420+0000 7f8015779b80 1 -- 172.21.15.38:0/206616468 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557d3a86afe0 msgr2=0x557d3a8770d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:57:56.434 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.420+0000 7f8015779b80 1 --2- 172.21.15.38:0/206616468 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557d3a86afe0 0x557d3a8770d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc09f7c0 tx=0x7f7ffc09fde0 comp rx=0 tx=0).stop 2023-03-09T14:57:56.434 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.420+0000 7f8015779b80 1 -- 172.21.15.38:0/206616468 shutdown_connections 2023-03-09T14:57:56.434 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.420+0000 7f8015779b80 1 --2- 172.21.15.38:0/206616468 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x557d3a884d40 0x557d3a887150 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:57:56.434 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.420+0000 7f8015779b80 1 --2- 172.21.15.38:0/206616468 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557d3a871140 0x557d3a8749d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:57:56.434 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.420+0000 7f8015779b80 1 --2- 172.21.15.38:0/206616468 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557d3a86afe0 0x557d3a8770d0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:57:56.435 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.420+0000 7f8015779b80 1 --2- 172.21.15.38:0/206616468 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d3a870830 0x557d3a874350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:57:56.435 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.420+0000 7f7fea7fc700 0 client.25934 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:57:56.435 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.420+0000 7f8015779b80 1 -- 172.21.15.38:0/206616468 >> 172.21.15.38:0/206616468 conn(0x557d3a865a20 msgr2=0x557d3a864c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:57:56.435 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.421+0000 7f8015779b80 1 -- 172.21.15.38:0/206616468 shutdown_connections 2023-03-09T14:57:56.435 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.421+0000 7f8015779b80 1 -- 172.21.15.38:0/206616468 wait complete. 2023-03-09T14:57:56.435 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.421+0000 7f8015779b80 20 client.25934 trim_cache size 0 max 16384 2023-03-09T14:57:56.441 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.437+0000 7f8015779b80 1 Processor -- start 2023-03-09T14:57:56.441 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.437+0000 7f8015779b80 1 -- start start 2023-03-09T14:57:56.442 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.438+0000 7f8015779b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d3a86afe0 0x557d3a876c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:57:56.442 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.438+0000 7f8015779b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557d3a870830 0x557d3a864c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:57:56.442 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.438+0000 7f8015779b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557d3a871140 0x557d3a86e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:57:56.442 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.438+0000 7f8015779b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d3a872900 con 0x557d3a870830 2023-03-09T14:57:56.442 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.438+0000 7f8015779b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d3a870430 con 0x557d3a871140 2023-03-09T14:57:56.442 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.438+0000 7f8015779b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d3a861d70 con 0x557d3a86afe0 2023-03-09T14:57:56.442 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.438+0000 7f80027fc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557d3a870830 0x557d3a864c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:57:56.443 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.438+0000 7f80037fe700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557d3a871140 0x557d3a86e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:57:56.443 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.438+0000 7f80037fe700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557d3a871140 0x557d3a86e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.38:51332/0 (socket says 172.21.15.38:51332) 2023-03-09T14:57:56.443 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.438+0000 7f80037fe700 1 -- 172.21.15.38:0/2878289175 learned_addr learned my addr 172.21.15.38:0/2878289175 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:57:56.443 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.438+0000 7f80027fc700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557d3a870830 0x557d3a864c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42140/0 (socket says 172.21.15.38:42140) 2023-03-09T14:57:56.443 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.438+0000 7f8002ffd700 1 --2- 172.21.15.38:0/2878289175 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d3a86afe0 0x557d3a876c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:57:56.443 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.438+0000 7f80037fe700 1 -- 172.21.15.38:0/2878289175 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d3a86afe0 msgr2=0x557d3a876c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:57:56.444 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.438+0000 7f80037fe700 1 --2- 172.21.15.38:0/2878289175 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d3a86afe0 0x557d3a876c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:57:56.444 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.438+0000 7f80037fe700 1 -- 172.21.15.38:0/2878289175 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557d3a870830 msgr2=0x557d3a864c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:57:56.444 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.438+0000 7f80037fe700 1 --2- 172.21.15.38:0/2878289175 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557d3a870830 0x557d3a864c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:57:56.444 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.438+0000 7f80037fe700 1 -- 172.21.15.38:0/2878289175 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557d3a5d3c60 con 0x557d3a871140 2023-03-09T14:57:56.444 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.438+0000 7f80027fc700 1 --2- 172.21.15.38:0/2878289175 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557d3a870830 0x557d3a864c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:57:56.444 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.438+0000 7f8002ffd700 1 --2- 172.21.15.38:0/2878289175 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d3a86afe0 0x557d3a876c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:57:56.444 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.439+0000 7f80037fe700 1 --2- 172.21.15.38:0/2878289175 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557d3a871140 0x557d3a86e590 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f7fec0098f0 tx=0x7f7fec00bfa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=59d71c8d8bcc0c27 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:57:56.444 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.439+0000 7f7ff9ffb700 1 -- 172.21.15.38:0/2878289175 <== mon.2 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fec01c040 con 0x557d3a871140 2023-03-09T14:57:56.445 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.439+0000 7f7ff9ffb700 1 -- 172.21.15.38:0/2878289175 <== mon.2 v2:172.21.15.122:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f7fec005530 con 0x557d3a871140 2023-03-09T14:57:56.445 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.439+0000 7f7ff9ffb700 1 -- 172.21.15.38:0/2878289175 <== mon.2 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fec003680 con 0x557d3a871140 2023-03-09T14:57:56.448 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.439+0000 7f8015779b80 1 -- 172.21.15.38:0/2878289175 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557d3a871140 msgr2=0x557d3a86e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:57:56.448 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.439+0000 7f8015779b80 1 --2- 172.21.15.38:0/2878289175 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557d3a871140 0x557d3a86e590 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f7fec0098f0 tx=0x7f7fec00bfa0 comp rx=0 tx=0).stop 2023-03-09T14:57:56.448 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.439+0000 7f8015779b80 1 -- 172.21.15.38:0/2878289175 shutdown_connections 2023-03-09T14:57:56.448 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.439+0000 7f8015779b80 1 --2- 172.21.15.38:0/2878289175 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557d3a871140 0x557d3a86e590 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:57:56.449 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.439+0000 7f8015779b80 1 --2- 172.21.15.38:0/2878289175 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d3a86afe0 0x557d3a876c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:57:56.449 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.439+0000 7f8015779b80 1 --2- 172.21.15.38:0/2878289175 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557d3a870830 0x557d3a864c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:57:56.449 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.439+0000 7f8015779b80 1 -- 172.21.15.38:0/2878289175 >> 172.21.15.38:0/2878289175 conn(0x557d3a865a20 msgr2=0x557d3a880f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:57:56.449 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.439+0000 7f8015779b80 1 -- 172.21.15.38:0/2878289175 shutdown_connections 2023-03-09T14:57:56.449 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.439+0000 7f8015779b80 1 -- 172.21.15.38:0/2878289175 wait complete. 2023-03-09T14:57:56.449 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.440+0000 7f8015779b80 1 -- start start 2023-03-09T14:57:56.449 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.440+0000 7f8015779b80 1 Processor -- start 2023-03-09T14:57:56.450 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.440+0000 7f8015779b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557d3a86afe0 0x557d3a87bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:57:56.450 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.440+0000 7f8015779b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557d3a870830 0x557d3a86c450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:57:56.450 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.440+0000 7f8015779b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d3a871140 0x557d3a86c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:57:56.450 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.440+0000 7f8015779b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d3a86cf20 con 0x557d3a86afe0 2023-03-09T14:57:56.450 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.440+0000 7f8015779b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d3a86d060 con 0x557d3a870830 2023-03-09T14:57:56.450 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.440+0000 7f8015779b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d3a86d220 con 0x557d3a871140 2023-03-09T14:57:56.450 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.440+0000 7f8015779b80 10 client.0 _ll_register_callbacks cb 0x557d3a762520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:57:56.451 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.440+0000 7f8002ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557d3a86afe0 0x557d3a87bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:57:56.451 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.440+0000 7f80027fc700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557d3a870830 0x557d3a86c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:57:56.451 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.440+0000 7f8002ffd700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557d3a86afe0 0x557d3a87bb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42144/0 (socket says 172.21.15.38:42144) 2023-03-09T14:57:56.451 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.440+0000 7f80027fc700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557d3a870830 0x557d3a86c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.38:51336/0 (socket says 172.21.15.38:51336) 2023-03-09T14:57:56.451 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.440+0000 7f8002ffd700 1 -- 172.21.15.38:0/3009717243 learned_addr learned my addr 172.21.15.38:0/3009717243 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:57:56.451 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.440+0000 7f80037fe700 1 --2- 172.21.15.38:0/3009717243 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d3a871140 0x557d3a86c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:57:56.451 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.440+0000 7f80027fc700 1 -- 172.21.15.38:0/3009717243 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d3a871140 msgr2=0x557d3a86c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:57:56.451 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.441+0000 7f80027fc700 1 --2- 172.21.15.38:0/3009717243 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d3a871140 0x557d3a86c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:57:56.452 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.441+0000 7f80027fc700 1 -- 172.21.15.38:0/3009717243 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557d3a86afe0 msgr2=0x557d3a87bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:57:56.452 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.441+0000 7f80027fc700 1 --2- 172.21.15.38:0/3009717243 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557d3a86afe0 0x557d3a87bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:57:56.452 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.441+0000 7f80027fc700 1 -- 172.21.15.38:0/3009717243 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557d3a783190 con 0x557d3a870830 2023-03-09T14:57:56.452 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.441+0000 7f8002ffd700 1 --2- 172.21.15.38:0/3009717243 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557d3a86afe0 0x557d3a87bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:57:56.452 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.441+0000 7f80027fc700 1 --2- 172.21.15.38:0/3009717243 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557d3a870830 0x557d3a86c450 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7ff4009fd0 tx=0x7f7ff4020700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb9d0640759a93f6 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:57:56.452 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.441+0000 7f7fea7fc700 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2023-03-09T14:57:56.452 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.441+0000 7f7fea7fc700 1 -- 172.21.15.38:0/3009717243 <== mon.2 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ff402f070 con 0x557d3a870830 2023-03-09T14:57:56.452 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.441+0000 7f7fea7fc700 1 -- 172.21.15.38:0/3009717243 <== mon.2 v2:172.21.15.122:3300/0 2 ==== osd_map(245..245 src has 1..245) v4 ==== 9577+0+0 (secure 0 0 0) 0x7f7ff4020ba0 con 0x557d3a870830 2023-03-09T14:57:56.453 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.441+0000 7f8015779b80 1 -- 172.21.15.38:0/3009717243 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557d3a77a1b0 con 0x557d3a870830 2023-03-09T14:57:56.453 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.441+0000 7f7fea7fc700 1 -- 172.21.15.38:0/3009717243 <== mon.2 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7ff4010040 con 0x557d3a870830 2023-03-09T14:57:56.453 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.441+0000 7f8015779b80 10 client.25940 fetch_fsmap learned FSMap version 41 2023-03-09T14:57:56.453 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.441+0000 7f8015779b80 1 -- 172.21.15.38:0/3009717243 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) v3 -- 0x7f800411c320 con 0x557d3a870830 2023-03-09T14:57:56.453 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.441+0000 7f7fea7fc700 1 -- 172.21.15.38:0/3009717243 <== mon.2 v2:172.21.15.122:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x557d3a77a1b0 con 0x557d3a870830 2023-03-09T14:57:56.453 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.441+0000 7f7fea7fc700 1 -- 172.21.15.38:0/3009717243 <== mon.2 v2:172.21.15.122:3300/0 5 ==== fsmap(e 41) v1 ==== 7490+0+0 (secure 0 0 0) 0x7f7ff401f070 con 0x557d3a870830 2023-03-09T14:57:56.454 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.441+0000 7f8015779b80 10 client.25940 fetch_fsmap finished waiting for FSMap version 41 2023-03-09T14:57:56.454 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.441+0000 7f8015779b80 10 client.25940 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T14:57:56.454 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.441+0000 7f8015779b80 20 client.25940 populate_metadata read hostname 'smithi038' 2023-03-09T14:57:56.454 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.442+0000 7f8015779b80 1 --2- 172.21.15.38:0/3009717243 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x557d3a8da140 0x557d3a8dc550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:57:56.454 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.442+0000 7f8015779b80 4 client.25940 mds_command: new command op to 24530 tid=0[{"prefix": "scrub status"}] 2023-03-09T14:57:56.454 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.442+0000 7f8015779b80 1 -- 172.21.15.38:0/3009717243 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x557d3a7641c0 con 0x557d3a8da140 2023-03-09T14:57:56.454 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.442+0000 7f8002ffd700 1 --2- 172.21.15.38:0/3009717243 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x557d3a8da140 0x557d3a8dc550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:57:56.455 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.442+0000 7f7fea7fc700 10 client.25940 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:57:56.455 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.442+0000 7f8002ffd700 1 --2- 172.21.15.38:0/3009717243 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x557d3a8da140 0x557d3a8dc550 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7dc09b9de9505d5 server_cookie=c2e06a7c2b5f943a in_seq=0 out_seq=0 2023-03-09T14:57:56.455 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.442+0000 7f7fea7fc700 1 -- 172.21.15.38:0/3009717243 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+265 (crc 0 0 0) 0x557d3a7641c0 con 0x557d3a8da140 2023-03-09T14:57:56.455 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.442+0000 7f7fea7fc700 10 client.25940 handle_command_reply: tid=0 2023-03-09T14:57:56.455 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.442+0000 7f8015779b80 1 client.25940 shutdown 2023-03-09T14:57:56.455 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.442+0000 7f8015779b80 1 -- 172.21.15.38:0/3009717243 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557d3a870830 msgr2=0x557d3a86c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:57:56.456 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.442+0000 7f8015779b80 1 --2- 172.21.15.38:0/3009717243 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557d3a870830 0x557d3a86c450 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7ff4009fd0 tx=0x7f7ff4020700 comp rx=0 tx=0).stop 2023-03-09T14:57:56.456 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.442+0000 7f8015779b80 1 -- 172.21.15.38:0/3009717243 shutdown_connections 2023-03-09T14:57:56.456 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.442+0000 7f8015779b80 1 --2- 172.21.15.38:0/3009717243 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x557d3a8da140 0x557d3a8dc550 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:57:56.456 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.442+0000 7f8015779b80 1 --2- 172.21.15.38:0/3009717243 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x557d3a86afe0 0x557d3a87bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:57:56.456 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.442+0000 7f8015779b80 1 --2- 172.21.15.38:0/3009717243 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557d3a871140 0x557d3a86c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:57:56.457 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.442+0000 7f7fea7fc700 0 client.25940 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:57:56.457 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.442+0000 7f8015779b80 1 --2- 172.21.15.38:0/3009717243 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557d3a870830 0x557d3a86c450 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:57:56.457 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.442+0000 7f8015779b80 1 -- 172.21.15.38:0/3009717243 >> 172.21.15.38:0/3009717243 conn(0x557d3a865a20 msgr2=0x557d3a864c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:57:56.457 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.443+0000 7f8015779b80 1 -- 172.21.15.38:0/3009717243 shutdown_connections 2023-03-09T14:57:56.458 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.443+0000 7f8015779b80 1 -- 172.21.15.38:0/3009717243 wait complete. 2023-03-09T14:57:56.458 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.443+0000 7f8015779b80 20 client.25940 trim_cache size 0 max 16384 2023-03-09T14:57:56.459 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2023-03-09T14:57:56.459 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "scrub active (1 inodes in the stack)", 2023-03-09T14:57:56.459 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": { 2023-03-09T14:57:56.459 INFO:teuthology.orchestra.run.smithi038.stdout: "a14659f4-d7ea-4c5a-b15b-6138af3f909f": { 2023-03-09T14:57:56.460 INFO:teuthology.orchestra.run.smithi038.stdout: "path": "/", 2023-03-09T14:57:56.460 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "a14659f4-d7ea-4c5a-b15b-6138af3f909f", 2023-03-09T14:57:56.460 INFO:teuthology.orchestra.run.smithi038.stdout: "options": "recursive,force" 2023-03-09T14:57:56.460 INFO:teuthology.orchestra.run.smithi038.stdout: } 2023-03-09T14:57:56.461 INFO:teuthology.orchestra.run.smithi038.stdout: } 2023-03-09T14:57:56.461 INFO:teuthology.orchestra.run.smithi038.stdout:} 2023-03-09T14:57:56.463 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.443+0000 7f80095f0700 1 -- 172.21.15.38:0/3395206744 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f7ff009f4d0 msgr2=0x7f7ff00a1980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:57:56.463 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.443+0000 7f80095f0700 1 --2- 172.21.15.38:0/3395206744 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f7ff009f4d0 0x7f7ff00a1980 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f80041a6430 tx=0x7f7ff400d040 comp rx=0 tx=0).stop 2023-03-09T14:57:56.464 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.443+0000 7f80095f0700 1 -- 172.21.15.38:0/3395206744 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8004107780 msgr2=0x7f80041a4e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:57:56.464 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.443+0000 7f80095f0700 1 --2- 172.21.15.38:0/3395206744 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8004107780 0x7f80041a4e20 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7ffc00d900 tx=0x7f7ffc00dc10 comp rx=0 tx=0).stop 2023-03-09T14:57:56.464 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.443+0000 7f80095f0700 1 -- 172.21.15.38:0/3395206744 shutdown_connections 2023-03-09T14:57:56.464 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.443+0000 7f80095f0700 1 --2- 172.21.15.38:0/3395206744 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f7ff009f4d0 0x7f7ff00a1980 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:57:56.464 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.443+0000 7f80095f0700 1 --2- 172.21.15.38:0/3395206744 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f80041196d0 0x7f80041a97c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:57:56.465 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.443+0000 7f80095f0700 1 --2- 172.21.15.38:0/3395206744 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f8004107780 0x7f80041a4e20 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:57:56.465 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.443+0000 7f80095f0700 1 --2- 172.21.15.38:0/3395206744 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f8004107b50 0x7f80041a5360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:57:56.465 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.443+0000 7f80095f0700 1 -- 172.21.15.38:0/3395206744 >> 172.21.15.38:0/3395206744 conn(0x7f800406b450 msgr2=0x7f80040712a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:57:56.465 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.443+0000 7f80095f0700 1 -- 172.21.15.38:0/3395206744 shutdown_connections 2023-03-09T14:57:56.466 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:57:56.443+0000 7f80095f0700 1 -- 172.21.15.38:0/3395206744 wait complete. 2023-03-09T14:57:56.466 INFO:tasks.cephfs.filesystem:scrub status for tag:a14659f4-d7ea-4c5a-b15b-6138af3f909f - {'path': '/', 'tag': 'a14659f4-d7ea-4c5a-b15b-6138af3f909f', 'options': 'recursive,force'} 2023-03-09T14:57:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:57:55 smithi122 ceph-mon[93869]: pgmap v508: 129 pgs: 129 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 18 MiB/s wr, 2.10k op/s 2023-03-09T14:57:56.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:57:55 smithi181 ceph-mon[94189]: pgmap v508: 129 pgs: 129 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 18 MiB/s wr, 2.10k op/s 2023-03-09T14:57:58.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:57:58 smithi122 ceph-mon[93869]: pgmap v509: 129 pgs: 129 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 20 MiB/s wr, 2.29k op/s 2023-03-09T14:57:58.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:57:58 smithi181 ceph-mon[94189]: pgmap v509: 129 pgs: 129 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 20 MiB/s wr, 2.29k op/s 2023-03-09T14:57:58.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:57:58 smithi038 ceph-mon[92422]: pgmap v509: 129 pgs: 129 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 20 MiB/s wr, 2.29k op/s 2023-03-09T14:57:58.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 450.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T14:58:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:58:00 smithi122 ceph-mon[93869]: pgmap v510: 129 pgs: 129 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 13 MiB/s wr, 1.48k op/s 2023-03-09T14:58:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:58:00 smithi181 ceph-mon[94189]: pgmap v510: 129 pgs: 129 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 13 MiB/s wr, 1.48k op/s 2023-03-09T14:58:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:58:00 smithi038 ceph-mon[92422]: pgmap v510: 129 pgs: 129 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 13 MiB/s wr, 1.48k op/s 2023-03-09T14:58:02.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:58:02 smithi122 ceph-mon[93869]: pgmap v511: 129 pgs: 129 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 13 MiB/s wr, 1.55k op/s 2023-03-09T14:58:02.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:58:02 smithi181 ceph-mon[94189]: pgmap v511: 129 pgs: 129 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 13 MiB/s wr, 1.55k op/s 2023-03-09T14:58:02.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:58:02 smithi038 ceph-mon[92422]: pgmap v511: 129 pgs: 129 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 13 MiB/s wr, 1.55k op/s 2023-03-09T14:58:03.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 455.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T14:58:04.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:58:03 smithi181 ceph-mon[94189]: pgmap v512: 129 pgs: 129 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 8.3 MiB/s wr, 950 op/s 2023-03-09T14:58:04.232 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:58:03 smithi122 ceph-mon[93869]: pgmap v512: 129 pgs: 129 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 8.3 MiB/s wr, 950 op/s 2023-03-09T14:58:04.254 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:58:03 smithi038 ceph-mon[92422]: pgmap v512: 129 pgs: 129 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 MiB/s rd, 8.3 MiB/s wr, 950 op/s 2023-03-09T14:58:05.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:58:04 smithi122 ceph-mon[93869]: osdmap e246: 12 total, 12 up, 12 in 2023-03-09T14:58:05.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:58:04 smithi181 ceph-mon[94189]: osdmap e246: 12 total, 12 up, 12 in 2023-03-09T14:58:05.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:58:04 smithi038 ceph-mon[92422]: osdmap e246: 12 total, 12 up, 12 in 2023-03-09T14:58:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:58:05 smithi122 ceph-mon[93869]: pgmap v514: 129 pgs: 3 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 120 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 6.1 MiB/s wr, 721 op/s 2023-03-09T14:58:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:58:05 smithi181 ceph-mon[94189]: pgmap v514: 129 pgs: 3 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 120 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 6.1 MiB/s wr, 721 op/s 2023-03-09T14:58:06.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:58:05 smithi038 ceph-mon[92422]: pgmap v514: 129 pgs: 3 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 120 active+clean; 12 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 6.1 MiB/s wr, 721 op/s 2023-03-09T14:58:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:58:08 smithi122 ceph-mon[93869]: pgmap v515: 129 pgs: 18 active+clean+snaptrim, 51 active+clean+snaptrim_wait, 60 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 4.5 MiB/s wr, 388 op/s 2023-03-09T14:58:08.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:58:08 smithi181 ceph-mon[94189]: pgmap v515: 129 pgs: 18 active+clean+snaptrim, 51 active+clean+snaptrim_wait, 60 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 4.5 MiB/s wr, 388 op/s 2023-03-09T14:58:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:58:08 smithi038 ceph-mon[92422]: pgmap v515: 129 pgs: 18 active+clean+snaptrim, 51 active+clean+snaptrim_wait, 60 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 4.5 MiB/s wr, 388 op/s 2023-03-09T14:58:08.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 460.0 s, 172.8 tps, lat 1809.704 ms stddev 4139.069 2023-03-09T14:58:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:58:10 smithi122 ceph-mon[93869]: pgmap v516: 129 pgs: 18 active+clean+snaptrim, 51 active+clean+snaptrim_wait, 60 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 4.5 MiB/s wr, 388 op/s 2023-03-09T14:58:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:58:10 smithi181 ceph-mon[94189]: pgmap v516: 129 pgs: 18 active+clean+snaptrim, 51 active+clean+snaptrim_wait, 60 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 4.5 MiB/s wr, 388 op/s 2023-03-09T14:58:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:58:10 smithi038 ceph-mon[92422]: pgmap v516: 129 pgs: 18 active+clean+snaptrim, 51 active+clean+snaptrim_wait, 60 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 4.5 MiB/s wr, 388 op/s 2023-03-09T14:58:12.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:58:12 smithi122 ceph-mon[93869]: pgmap v517: 129 pgs: 6 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 104 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.9 MiB/s wr, 580 op/s 2023-03-09T14:58:12.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:58:12 smithi181 ceph-mon[94189]: pgmap v517: 129 pgs: 6 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 104 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.9 MiB/s wr, 580 op/s 2023-03-09T14:58:12.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:58:12 smithi038 ceph-mon[92422]: pgmap v517: 129 pgs: 6 active+clean+snaptrim, 19 active+clean+snaptrim_wait, 104 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 5.9 MiB/s wr, 580 op/s 2023-03-09T14:58:13.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 465.0 s, 1363.0 tps, lat 73.537 ms stddev 44.992 2023-03-09T14:58:14.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:58:13 smithi122 ceph-mon[93869]: pgmap v518: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 9.1 MiB/s wr, 738 op/s 2023-03-09T14:58:14.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:58:13 smithi181 ceph-mon[94189]: pgmap v518: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 9.1 MiB/s wr, 738 op/s 2023-03-09T14:58:14.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:58:13 smithi038 ceph-mon[92422]: pgmap v518: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 9.1 MiB/s wr, 738 op/s 2023-03-09T14:58:16.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:58:16 smithi122 ceph-mon[93869]: pgmap v519: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 9.4 MiB/s wr, 799 op/s 2023-03-09T14:58:16.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:58:16 smithi181 ceph-mon[94189]: pgmap v519: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 9.4 MiB/s wr, 799 op/s 2023-03-09T14:58:16.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:58:16 smithi038 ceph-mon[92422]: pgmap v519: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 9.4 MiB/s wr, 799 op/s 2023-03-09T14:58:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:58:18 smithi122 ceph-mon[93869]: pgmap v520: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 12 MiB/s wr, 1.28k op/s 2023-03-09T14:58:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:58:18 smithi122 ceph-mon[93869]: osdmap e247: 12 total, 12 up, 12 in 2023-03-09T14:58:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:58:18 smithi181 ceph-mon[94189]: pgmap v520: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 12 MiB/s wr, 1.28k op/s 2023-03-09T14:58:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:58:18 smithi181 ceph-mon[94189]: osdmap e247: 12 total, 12 up, 12 in 2023-03-09T14:58:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:58:18 smithi038 ceph-mon[92422]: pgmap v520: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 12 MiB/s wr, 1.28k op/s 2023-03-09T14:58:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:58:18 smithi038 ceph-mon[92422]: osdmap e247: 12 total, 12 up, 12 in 2023-03-09T14:58:18.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 470.0 s, 198.6 tps, lat 501.069 ms stddev 1211.887 2023-03-09T14:58:20.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:58:20 smithi122 ceph-mon[93869]: pgmap v522: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 15 MiB/s wr, 1.86k op/s 2023-03-09T14:58:20.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:58:20 smithi181 ceph-mon[94189]: pgmap v522: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 15 MiB/s wr, 1.86k op/s 2023-03-09T14:58:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:58:20 smithi038 ceph-mon[92422]: pgmap v522: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 15 MiB/s wr, 1.86k op/s 2023-03-09T14:58:22.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:58:22 smithi122 ceph-mon[93869]: pgmap v523: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 15 MiB/s wr, 1.81k op/s 2023-03-09T14:58:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:58:22 smithi181 ceph-mon[94189]: pgmap v523: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 15 MiB/s wr, 1.81k op/s 2023-03-09T14:58:22.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:58:22 smithi038 ceph-mon[92422]: pgmap v523: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 15 MiB/s wr, 1.81k op/s 2023-03-09T14:58:23.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 475.0 s, 1377.8 tps, lat 73.154 ms stddev 46.444 2023-03-09T14:58:24.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:58:23 smithi181 ceph-mon[94189]: pgmap v524: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 16 MiB/s wr, 2.23k op/s 2023-03-09T14:58:24.231 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:58:23 smithi122 ceph-mon[93869]: pgmap v524: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 16 MiB/s wr, 2.23k op/s 2023-03-09T14:58:24.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:58:23 smithi038 ceph-mon[92422]: pgmap v524: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 13 MiB/s rd, 16 MiB/s wr, 2.23k op/s 2023-03-09T14:58:26.467 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-03-09T14:58:26.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:58:26 smithi122 ceph-mon[93869]: pgmap v525: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 18 MiB/s wr, 2.33k op/s 2023-03-09T14:58:26.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:58:26 smithi181 ceph-mon[94189]: pgmap v525: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 18 MiB/s wr, 2.33k op/s 2023-03-09T14:58:26.505 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:58:26 smithi038 ceph-mon[92422]: pgmap v525: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 14 MiB/s rd, 18 MiB/s wr, 2.33k op/s 2023-03-09T14:58:27.071 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.069+0000 7ff696a88700 1 -- 172.21.15.38:0/2799541434 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff690100b20 msgr2=0x7ff690104ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:58:27.071 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.069+0000 7ff696a88700 1 --2- 172.21.15.38:0/2799541434 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff690100b20 0x7ff690104ae0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7ff67c0099c0 tx=0x7ff67c009cd0 comp rx=0 tx=0).stop 2023-03-09T14:58:27.072 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.069+0000 7ff696a88700 1 -- 172.21.15.38:0/2799541434 shutdown_connections 2023-03-09T14:58:27.072 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.069+0000 7ff696a88700 1 --2- 172.21.15.38:0/2799541434 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff6901051c0 0x7ff69010cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:58:27.072 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.069+0000 7ff696a88700 1 --2- 172.21.15.38:0/2799541434 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff690100180 0x7ff690100550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:58:27.072 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.069+0000 7ff696a88700 1 --2- 172.21.15.38:0/2799541434 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff690100b20 0x7ff690104ae0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:58:27.072 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.069+0000 7ff696a88700 1 -- 172.21.15.38:0/2799541434 >> 172.21.15.38:0/2799541434 conn(0x7ff6900fb9d0 msgr2=0x7ff6900fdde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:58:27.072 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.069+0000 7ff696a88700 1 -- 172.21.15.38:0/2799541434 shutdown_connections 2023-03-09T14:58:27.072 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.069+0000 7ff696a88700 1 -- 172.21.15.38:0/2799541434 wait complete. 2023-03-09T14:58:27.074 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.070+0000 7ff696a88700 1 Processor -- start 2023-03-09T14:58:27.074 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.070+0000 7ff696a88700 1 -- start start 2023-03-09T14:58:27.074 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.070+0000 7ff696a88700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff690100b20 0x7ff6901962f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:58:27.074 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.070+0000 7ff696a88700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff6901051c0 0x7ff690196830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:58:27.075 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.070+0000 7ff696a88700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff69019ac90 0x7ff69019b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:58:27.075 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.070+0000 7ff696a88700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff69019b700 con 0x7ff690100b20 2023-03-09T14:58:27.075 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.070+0000 7ff696a88700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff69019b870 con 0x7ff69019ac90 2023-03-09T14:58:27.075 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.070+0000 7ff696a88700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff69019ba30 con 0x7ff6901051c0 2023-03-09T14:58:27.075 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.071+0000 7ff694824700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff690100b20 0x7ff6901962f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:58:27.075 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.071+0000 7ff694824700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff690100b20 0x7ff6901962f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42160/0 (socket says 172.21.15.38:42160) 2023-03-09T14:58:27.075 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.071+0000 7ff694824700 1 -- 172.21.15.38:0/4132310035 learned_addr learned my addr 172.21.15.38:0/4132310035 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:58:27.076 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.071+0000 7ff695025700 1 --2- 172.21.15.38:0/4132310035 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff69019ac90 0x7ff69019b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:58:27.076 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.071+0000 7ff68ffff700 1 --2- 172.21.15.38:0/4132310035 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff6901051c0 0x7ff690196830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:58:27.076 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.071+0000 7ff695025700 1 -- 172.21.15.38:0/4132310035 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff6901051c0 msgr2=0x7ff690196830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:58:27.076 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.071+0000 7ff695025700 1 --2- 172.21.15.38:0/4132310035 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff6901051c0 0x7ff690196830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:58:27.076 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.071+0000 7ff695025700 1 -- 172.21.15.38:0/4132310035 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff690100b20 msgr2=0x7ff6901962f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:58:27.076 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.071+0000 7ff695025700 1 --2- 172.21.15.38:0/4132310035 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff690100b20 0x7ff6901962f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:58:27.076 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.071+0000 7ff695025700 1 -- 172.21.15.38:0/4132310035 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff67c0095f0 con 0x7ff69019ac90 2023-03-09T14:58:27.076 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.071+0000 7ff695025700 1 --2- 172.21.15.38:0/4132310035 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff69019ac90 0x7ff69019b100 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff68800efc0 tx=0x7ff68800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:58:27.077 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.071+0000 7ff68dffb700 1 -- 172.21.15.38:0/4132310035 <== mon.2 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff688018070 con 0x7ff69019ac90 2023-03-09T14:58:27.077 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.072+0000 7ff696a88700 1 -- 172.21.15.38:0/4132310035 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff690110030 con 0x7ff69019ac90 2023-03-09T14:58:27.077 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.072+0000 7ff68dffb700 1 -- 172.21.15.38:0/4132310035 <== mon.2 v2:172.21.15.122:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff68800f040 con 0x7ff69019ac90 2023-03-09T14:58:27.077 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.072+0000 7ff68dffb700 1 -- 172.21.15.38:0/4132310035 <== mon.2 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff68800ce10 con 0x7ff69019ac90 2023-03-09T14:58:27.077 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.072+0000 7ff696a88700 1 -- 172.21.15.38:0/4132310035 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff69010fea0 con 0x7ff69019ac90 2023-03-09T14:58:27.078 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.074+0000 7ff68dffb700 1 -- 172.21.15.38:0/4132310035 <== mon.2 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 25) v1 ==== 133295+0+0 (secure 0 0 0) 0x7ff688003a00 con 0x7ff69019ac90 2023-03-09T14:58:27.078 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.075+0000 7ff68dffb700 1 --2- 172.21.15.38:0/4132310035 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7ff68009f4a0 0x7ff6800a1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:58:27.078 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.075+0000 7ff68dffb700 1 -- 172.21.15.38:0/4132310035 <== mon.2 v2:172.21.15.122:3300/0 5 ==== osd_map(247..247 src has 1..247) v4 ==== 9577+0+0 (secure 0 0 0) 0x7ff6880ca190 con 0x7ff69019ac90 2023-03-09T14:58:27.078 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.076+0000 7ff694824700 1 --2- 172.21.15.38:0/4132310035 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7ff68009f4a0 0x7ff6800a1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:58:27.080 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.076+0000 7ff694824700 1 --2- 172.21.15.38:0/4132310035 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7ff68009f4a0 0x7ff6800a1950 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff684009a20 tx=0x7ff684009210 comp rx=0 tx=0).ready entity=mgr.15558 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:58:27.080 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.076+0000 7ff6a2c11b80 1 Processor -- start 2023-03-09T14:58:27.080 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.077+0000 7ff6a2c11b80 1 -- start start 2023-03-09T14:58:27.080 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.077+0000 7ff6a2c11b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e661702c0 0x555e66170690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:58:27.080 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.077+0000 7ff6a2c11b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e66170bd0 0x555e66179330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:58:27.080 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.077+0000 7ff6a2c11b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555e66179870 0x555e6617bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:58:27.080 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.077+0000 7ff6a2c11b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e66050da0 con 0x555e66170bd0 2023-03-09T14:58:27.081 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.077+0000 7ff6a2c11b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e660bf780 con 0x555e661702c0 2023-03-09T14:58:27.081 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.077+0000 7ff6a2c11b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e660b8cb0 con 0x555e66179870 2023-03-09T14:58:27.081 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.077+0000 7ff68ffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e66170bd0 0x555e66179330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:58:27.081 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.077+0000 7ff694824700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e661702c0 0x555e66170690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:58:27.081 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.077+0000 7ff68ffff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e66170bd0 0x555e66179330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42170/0 (socket says 172.21.15.38:42170) 2023-03-09T14:58:27.081 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.077+0000 7ff68ffff700 1 -- 172.21.15.38:0/4059585488 learned_addr learned my addr 172.21.15.38:0/4059585488 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:58:27.081 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.077+0000 7ff695025700 1 --2- 172.21.15.38:0/4059585488 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555e66179870 0x555e6617bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:58:27.082 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.077+0000 7ff694824700 1 -- 172.21.15.38:0/4059585488 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555e66179870 msgr2=0x555e6617bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:58:27.082 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.077+0000 7ff694824700 1 --2- 172.21.15.38:0/4059585488 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555e66179870 0x555e6617bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:58:27.082 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.077+0000 7ff694824700 1 -- 172.21.15.38:0/4059585488 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e66170bd0 msgr2=0x555e66179330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:58:27.082 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.077+0000 7ff694824700 1 --2- 172.21.15.38:0/4059585488 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e66170bd0 0x555e66179330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:58:27.082 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.077+0000 7ff694824700 1 -- 172.21.15.38:0/4059585488 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555e6607f190 con 0x555e661702c0 2023-03-09T14:58:27.082 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.077+0000 7ff68ffff700 1 --2- 172.21.15.38:0/4059585488 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e66170bd0 0x555e66179330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:58:27.082 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.078+0000 7ff694824700 1 --2- 172.21.15.38:0/4059585488 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e661702c0 0x555e66170690 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff68400ea10 tx=0x7ff684011000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8a033ba18ca9075 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:58:27.082 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.078+0000 7ff67affd700 1 -- 172.21.15.38:0/4059585488 <== mon.2 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff68400ee50 con 0x555e661702c0 2023-03-09T14:58:27.083 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.078+0000 7ff67affd700 1 -- 172.21.15.38:0/4059585488 <== mon.2 v2:172.21.15.122:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff684015940 con 0x555e661702c0 2023-03-09T14:58:27.083 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.078+0000 7ff67affd700 1 -- 172.21.15.38:0/4059585488 <== mon.2 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6840139d0 con 0x555e661702c0 2023-03-09T14:58:27.086 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.079+0000 7ff6a2c11b80 1 -- 172.21.15.38:0/4059585488 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e661702c0 msgr2=0x555e66170690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:58:27.086 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.079+0000 7ff6a2c11b80 1 --2- 172.21.15.38:0/4059585488 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e661702c0 0x555e66170690 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff68400ea10 tx=0x7ff684011000 comp rx=0 tx=0).stop 2023-03-09T14:58:27.086 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.079+0000 7ff6a2c11b80 1 -- 172.21.15.38:0/4059585488 shutdown_connections 2023-03-09T14:58:27.086 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.079+0000 7ff6a2c11b80 1 --2- 172.21.15.38:0/4059585488 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555e66179870 0x555e6617bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:58:27.086 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.079+0000 7ff6a2c11b80 1 --2- 172.21.15.38:0/4059585488 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e661702c0 0x555e66170690 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:58:27.086 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.079+0000 7ff6a2c11b80 1 --2- 172.21.15.38:0/4059585488 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e66170bd0 0x555e66179330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:58:27.087 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.079+0000 7ff6a2c11b80 1 -- 172.21.15.38:0/4059585488 >> 172.21.15.38:0/4059585488 conn(0x555e661615a0 msgr2=0x555e66161970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:58:27.087 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.079+0000 7ff6a2c11b80 1 -- 172.21.15.38:0/4059585488 shutdown_connections 2023-03-09T14:58:27.087 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.079+0000 7ff6a2c11b80 1 -- 172.21.15.38:0/4059585488 wait complete. 2023-03-09T14:58:27.087 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.079+0000 7ff6a2c11b80 1 -- start start 2023-03-09T14:58:27.087 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.080+0000 7ff6a2c11b80 1 Processor -- start 2023-03-09T14:58:27.087 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.080+0000 7ff6a2c11b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e661702c0 0x555e66179170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:58:27.087 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.080+0000 7ff6a2c11b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555e66170bd0 0x555e661763f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:58:27.088 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.080+0000 7ff6a2c11b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e66179870 0x555e66176970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:58:27.088 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.080+0000 7ff6a2c11b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e65f3c920 con 0x555e66179870 2023-03-09T14:58:27.088 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.080+0000 7ff6a2c11b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e661770e0 con 0x555e661702c0 2023-03-09T14:58:27.088 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.080+0000 7ff6a2c11b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e661772a0 con 0x555e66170bd0 2023-03-09T14:58:27.088 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.080+0000 7ff6a2c11b80 10 client.0 _ll_register_callbacks cb 0x555e6605e520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:58:27.088 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.080+0000 7ff694824700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e661702c0 0x555e66179170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:58:27.088 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.080+0000 7ff694824700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e661702c0 0x555e66179170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.38:51364/0 (socket says 172.21.15.38:51364) 2023-03-09T14:58:27.088 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.080+0000 7ff694824700 1 -- 172.21.15.38:0/798230904 learned_addr learned my addr 172.21.15.38:0/798230904 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:58:27.089 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.080+0000 7ff695025700 1 --2- 172.21.15.38:0/798230904 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e66179870 0x555e66176970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:58:27.089 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.080+0000 7ff68ffff700 1 --2- 172.21.15.38:0/798230904 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555e66170bd0 0x555e661763f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:58:27.089 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.080+0000 7ff694824700 1 -- 172.21.15.38:0/798230904 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555e66170bd0 msgr2=0x555e661763f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:58:27.089 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.080+0000 7ff694824700 1 --2- 172.21.15.38:0/798230904 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555e66170bd0 0x555e661763f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:58:27.089 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.080+0000 7ff694824700 1 -- 172.21.15.38:0/798230904 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e66179870 msgr2=0x555e66176970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:58:27.089 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.080+0000 7ff694824700 1 --2- 172.21.15.38:0/798230904 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e66179870 0x555e66176970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:58:27.089 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.080+0000 7ff694824700 1 -- 172.21.15.38:0/798230904 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff69010d5d0 con 0x555e661702c0 2023-03-09T14:58:27.090 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.080+0000 7ff68ffff700 1 --2- 172.21.15.38:0/798230904 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555e66170bd0 0x555e661763f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:58:27.090 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.081+0000 7ff694824700 1 --2- 172.21.15.38:0/798230904 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e661702c0 0x555e66179170 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff684013c00 tx=0x7ff684013450 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b321b84a891029f8 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:58:27.090 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.081+0000 7ff66ffff700 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2023-03-09T14:58:27.090 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.081+0000 7ff66ffff700 1 -- 172.21.15.38:0/798230904 <== mon.2 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff684015890 con 0x555e661702c0 2023-03-09T14:58:27.090 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.081+0000 7ff6a2c11b80 1 -- 172.21.15.38:0/798230904 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555e66086190 con 0x555e661702c0 2023-03-09T14:58:27.090 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.081+0000 7ff66ffff700 1 -- 172.21.15.38:0/798230904 <== mon.2 v2:172.21.15.122:3300/0 2 ==== osd_map(247..247 src has 1..247) v4 ==== 9577+0+0 (secure 0 0 0) 0x7ff6840159f0 con 0x555e661702c0 2023-03-09T14:58:27.090 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.081+0000 7ff66ffff700 1 -- 172.21.15.38:0/798230904 <== mon.2 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff684015bb0 con 0x555e661702c0 2023-03-09T14:58:27.090 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.081+0000 7ff6a2c11b80 10 client.25949 fetch_fsmap learned FSMap version 41 2023-03-09T14:58:27.091 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.081+0000 7ff6a2c11b80 1 -- 172.21.15.38:0/798230904 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) v3 -- 0x7ff68400e6c0 con 0x555e661702c0 2023-03-09T14:58:27.091 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.081+0000 7ff66ffff700 1 -- 172.21.15.38:0/798230904 <== mon.2 v2:172.21.15.122:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x555e66086190 con 0x555e661702c0 2023-03-09T14:58:27.091 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.081+0000 7ff66ffff700 1 -- 172.21.15.38:0/798230904 <== mon.2 v2:172.21.15.122:3300/0 5 ==== fsmap(e 41) v1 ==== 7490+0+0 (secure 0 0 0) 0x7ff6840323f0 con 0x555e661702c0 2023-03-09T14:58:27.091 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.082+0000 7ff6a2c11b80 10 client.25949 fetch_fsmap finished waiting for FSMap version 41 2023-03-09T14:58:27.091 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.082+0000 7ff6a2c11b80 10 client.25949 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T14:58:27.091 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.082+0000 7ff6a2c11b80 20 client.25949 populate_metadata read hostname 'smithi038' 2023-03-09T14:58:27.091 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.082+0000 7ff6a2c11b80 1 --2- 172.21.15.38:0/798230904 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x555e66189450 0x555e6618b860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:58:27.092 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.082+0000 7ff6a2c11b80 4 client.25949 mds_command: new command op to 24530 tid=0[{"prefix": "get_command_descriptions"}] 2023-03-09T14:58:27.092 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.082+0000 7ff6a2c11b80 1 -- 172.21.15.38:0/798230904 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555e65f3cf60 con 0x555e66189450 2023-03-09T14:58:27.092 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.082+0000 7ff68ffff700 1 --2- 172.21.15.38:0/798230904 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x555e66189450 0x555e6618b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:58:27.092 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.082+0000 7ff66ffff700 10 client.25949 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:58:27.092 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.082+0000 7ff68ffff700 1 --2- 172.21.15.38:0/798230904 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x555e66189450 0x555e6618b860 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e8fe974221024035 server_cookie=b09c365d7cfc51e0 in_seq=0 out_seq=0 2023-03-09T14:58:27.092 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.083+0000 7ff66ffff700 1 -- 172.21.15.38:0/798230904 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18278 (crc 0 0 0) 0x555e65f3cf60 con 0x555e66189450 2023-03-09T14:58:27.092 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.083+0000 7ff66ffff700 10 client.25949 handle_command_reply: tid=0 2023-03-09T14:58:27.092 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.083+0000 7ff6a2c11b80 1 client.25949 shutdown 2023-03-09T14:58:27.093 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.083+0000 7ff6a2c11b80 1 -- 172.21.15.38:0/798230904 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e661702c0 msgr2=0x555e66179170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:58:27.093 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.083+0000 7ff6a2c11b80 1 --2- 172.21.15.38:0/798230904 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e661702c0 0x555e66179170 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff684013c00 tx=0x7ff684013450 comp rx=0 tx=0).stop 2023-03-09T14:58:27.093 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.083+0000 7ff6a2c11b80 1 -- 172.21.15.38:0/798230904 shutdown_connections 2023-03-09T14:58:27.093 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.083+0000 7ff6a2c11b80 1 --2- 172.21.15.38:0/798230904 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x555e66189450 0x555e6618b860 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:58:27.093 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.083+0000 7ff6a2c11b80 1 --2- 172.21.15.38:0/798230904 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e66179870 0x555e66176970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:58:27.093 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.083+0000 7ff6a2c11b80 1 --2- 172.21.15.38:0/798230904 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e661702c0 0x555e66179170 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:58:27.093 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.083+0000 7ff6a2c11b80 1 --2- 172.21.15.38:0/798230904 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555e66170bd0 0x555e661763f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:58:27.094 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.083+0000 7ff66ffff700 0 client.25949 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:58:27.094 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.083+0000 7ff6a2c11b80 1 -- 172.21.15.38:0/798230904 >> 172.21.15.38:0/798230904 conn(0x555e661615a0 msgr2=0x555e66160c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:58:27.094 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.083+0000 7ff6a2c11b80 1 -- 172.21.15.38:0/798230904 shutdown_connections 2023-03-09T14:58:27.094 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.083+0000 7ff6a2c11b80 1 -- 172.21.15.38:0/798230904 wait complete. 2023-03-09T14:58:27.094 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.084+0000 7ff6a2c11b80 20 client.25949 trim_cache size 0 max 16384 2023-03-09T14:58:27.108 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.107+0000 7ff6a2c11b80 1 Processor -- start 2023-03-09T14:58:27.112 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.109+0000 7ff6a2c11b80 1 -- start start 2023-03-09T14:58:27.112 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.110+0000 7ff6a2c11b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e661702c0 0x555e661803f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:58:27.112 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.110+0000 7ff6a2c11b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555e66170bd0 0x555e66184b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:58:27.112 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.110+0000 7ff6a2c11b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e66179870 0x555e66171780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:58:27.112 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.110+0000 7ff6a2c11b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e661622c0 con 0x555e661702c0 2023-03-09T14:58:27.112 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.110+0000 7ff6a2c11b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e66163190 con 0x555e66179870 2023-03-09T14:58:27.113 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.110+0000 7ff6a2c11b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e661722c0 con 0x555e66170bd0 2023-03-09T14:58:27.113 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.110+0000 7ff694824700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e661702c0 0x555e661803f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:58:27.113 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.110+0000 7ff694824700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e661702c0 0x555e661803f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42182/0 (socket says 172.21.15.38:42182) 2023-03-09T14:58:27.113 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.110+0000 7ff694824700 1 -- 172.21.15.38:0/1342018428 learned_addr learned my addr 172.21.15.38:0/1342018428 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:58:27.113 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.110+0000 7ff68ffff700 1 --2- 172.21.15.38:0/1342018428 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555e66170bd0 0x555e66184b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:58:27.114 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.110+0000 7ff694824700 1 -- 172.21.15.38:0/1342018428 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555e66170bd0 msgr2=0x555e66184b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:58:27.114 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.110+0000 7ff694824700 1 --2- 172.21.15.38:0/1342018428 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555e66170bd0 0x555e66184b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:58:27.114 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.110+0000 7ff695025700 1 --2- 172.21.15.38:0/1342018428 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e66179870 0x555e66171780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:58:27.115 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.110+0000 7ff694824700 1 -- 172.21.15.38:0/1342018428 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e66179870 msgr2=0x555e66171780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:58:27.115 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.110+0000 7ff694824700 1 --2- 172.21.15.38:0/1342018428 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e66179870 0x555e66171780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:58:27.115 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.110+0000 7ff694824700 1 -- 172.21.15.38:0/1342018428 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555e65ecfc60 con 0x555e661702c0 2023-03-09T14:58:27.115 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.110+0000 7ff695025700 1 --2- 172.21.15.38:0/1342018428 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e66179870 0x555e66171780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-09T14:58:27.115 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.111+0000 7ff694824700 1 --2- 172.21.15.38:0/1342018428 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e661702c0 0x555e661803f0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7ff684010010 tx=0x7ff684006b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=785ace63c1de989b server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:58:27.115 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.111+0000 7ff67b7fe700 1 -- 172.21.15.38:0/1342018428 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff684013ce0 con 0x555e661702c0 2023-03-09T14:58:27.115 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.111+0000 7ff67b7fe700 1 -- 172.21.15.38:0/1342018428 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7ff684032e30 con 0x555e661702c0 2023-03-09T14:58:27.115 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.111+0000 7ff67b7fe700 1 -- 172.21.15.38:0/1342018428 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff684040480 con 0x555e661702c0 2023-03-09T14:58:27.116 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.112+0000 7ff6a2c11b80 1 -- 172.21.15.38:0/1342018428 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e661702c0 msgr2=0x555e661803f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:58:27.116 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.112+0000 7ff6a2c11b80 1 --2- 172.21.15.38:0/1342018428 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e661702c0 0x555e661803f0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7ff684010010 tx=0x7ff684006b20 comp rx=0 tx=0).stop 2023-03-09T14:58:27.116 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.112+0000 7ff6a2c11b80 1 -- 172.21.15.38:0/1342018428 shutdown_connections 2023-03-09T14:58:27.116 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.112+0000 7ff6a2c11b80 1 --2- 172.21.15.38:0/1342018428 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e661702c0 0x555e661803f0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:58:27.116 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.112+0000 7ff6a2c11b80 1 --2- 172.21.15.38:0/1342018428 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e66179870 0x555e66171780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:58:27.116 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.112+0000 7ff6a2c11b80 1 --2- 172.21.15.38:0/1342018428 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555e66170bd0 0x555e66184b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:58:27.116 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.112+0000 7ff6a2c11b80 1 -- 172.21.15.38:0/1342018428 >> 172.21.15.38:0/1342018428 conn(0x555e661615a0 msgr2=0x555e66160c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:58:27.116 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.112+0000 7ff6a2c11b80 1 -- 172.21.15.38:0/1342018428 shutdown_connections 2023-03-09T14:58:27.117 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.116+0000 7ff6a2c11b80 1 -- 172.21.15.38:0/1342018428 wait complete. 2023-03-09T14:58:27.118 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.116+0000 7ff6a2c11b80 1 -- start start 2023-03-09T14:58:27.118 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.116+0000 7ff6a2c11b80 1 Processor -- start 2023-03-09T14:58:27.119 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.117+0000 7ff6a2c11b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e661702c0 0x555e661d7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:58:27.120 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.117+0000 7ff6a2c11b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e66170bd0 0x555e661d1240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:58:27.120 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.117+0000 7ff6a2c11b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555e66179870 0x555e661d1980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:58:27.120 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.117+0000 7ff6a2c11b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e661d2090 con 0x555e661702c0 2023-03-09T14:58:27.120 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.117+0000 7ff6a2c11b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e661d57d0 con 0x555e66170bd0 2023-03-09T14:58:27.120 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.117+0000 7ff6a2c11b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e661d5940 con 0x555e66179870 2023-03-09T14:58:27.120 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.117+0000 7ff6a2c11b80 10 client.0 _ll_register_callbacks cb 0x555e6605e520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:58:27.120 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.117+0000 7ff68ffff700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e66170bd0 0x555e661d1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:58:27.120 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.117+0000 7ff695025700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555e66179870 0x555e661d1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:58:27.121 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.117+0000 7ff68ffff700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e66170bd0 0x555e661d1240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.38:51380/0 (socket says 172.21.15.38:51380) 2023-03-09T14:58:27.121 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.117+0000 7ff68ffff700 1 -- 172.21.15.38:0/2463120946 learned_addr learned my addr 172.21.15.38:0/2463120946 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:58:27.121 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.117+0000 7ff695025700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555e66179870 0x555e661d1980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.38:49886/0 (socket says 172.21.15.38:49886) 2023-03-09T14:58:27.121 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.117+0000 7ff694824700 1 --2- 172.21.15.38:0/2463120946 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e661702c0 0x555e661d7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:58:27.121 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.117+0000 7ff68ffff700 1 -- 172.21.15.38:0/2463120946 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555e66179870 msgr2=0x555e661d1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:58:27.121 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.117+0000 7ff68ffff700 1 --2- 172.21.15.38:0/2463120946 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555e66179870 0x555e661d1980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:58:27.121 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.117+0000 7ff68ffff700 1 -- 172.21.15.38:0/2463120946 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e661702c0 msgr2=0x555e661d7cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:58:27.122 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.117+0000 7ff68ffff700 1 --2- 172.21.15.38:0/2463120946 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e661702c0 0x555e661d7cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:58:27.122 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.117+0000 7ff68ffff700 1 -- 172.21.15.38:0/2463120946 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff68400e6c0 con 0x555e66170bd0 2023-03-09T14:58:27.122 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.117+0000 7ff695025700 1 --2- 172.21.15.38:0/2463120946 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555e66179870 0x555e661d1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:58:27.122 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.118+0000 7ff68ffff700 1 --2- 172.21.15.38:0/2463120946 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e66170bd0 0x555e661d1240 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff67c000c00 tx=0x7ff67c005590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36a467326c556772 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:58:27.122 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.118+0000 7ff66ffff700 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2023-03-09T14:58:27.122 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.118+0000 7ff66ffff700 1 -- 172.21.15.38:0/2463120946 <== mon.2 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff67c027050 con 0x555e66170bd0 2023-03-09T14:58:27.122 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.118+0000 7ff66ffff700 1 -- 172.21.15.38:0/2463120946 <== mon.2 v2:172.21.15.122:3300/0 2 ==== osd_map(247..247 src has 1..247) v4 ==== 9577+0+0 (secure 0 0 0) 0x7ff67c0271b0 con 0x555e66170bd0 2023-03-09T14:58:27.123 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.118+0000 7ff6a2c11b80 1 -- 172.21.15.38:0/2463120946 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555e660761b0 con 0x555e66170bd0 2023-03-09T14:58:27.124 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.118+0000 7ff66ffff700 1 -- 172.21.15.38:0/2463120946 <== mon.2 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff67c027370 con 0x555e66170bd0 2023-03-09T14:58:27.124 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.118+0000 7ff6a2c11b80 10 client.25952 fetch_fsmap learned FSMap version 41 2023-03-09T14:58:27.124 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.118+0000 7ff6a2c11b80 1 -- 172.21.15.38:0/2463120946 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) v3 -- 0x555e65f0d250 con 0x555e66170bd0 2023-03-09T14:58:27.125 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.119+0000 7ff66ffff700 1 -- 172.21.15.38:0/2463120946 <== mon.2 v2:172.21.15.122:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x555e660761b0 con 0x555e66170bd0 2023-03-09T14:58:27.125 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.119+0000 7ff66ffff700 1 -- 172.21.15.38:0/2463120946 <== mon.2 v2:172.21.15.122:3300/0 5 ==== fsmap(e 41) v1 ==== 7490+0+0 (secure 0 0 0) 0x7ff67c00b940 con 0x555e66170bd0 2023-03-09T14:58:27.125 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.119+0000 7ff6a2c11b80 10 client.25952 fetch_fsmap finished waiting for FSMap version 41 2023-03-09T14:58:27.125 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.119+0000 7ff6a2c11b80 10 client.25952 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T14:58:27.125 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.119+0000 7ff6a2c11b80 20 client.25952 populate_metadata read hostname 'smithi038' 2023-03-09T14:58:27.125 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.119+0000 7ff6a2c11b80 1 --2- 172.21.15.38:0/2463120946 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x555e6616e210 0x555e6616e610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:58:27.125 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.119+0000 7ff6a2c11b80 4 client.25952 mds_command: new command op to 24530 tid=0[{"prefix": "scrub status"}] 2023-03-09T14:58:27.126 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.119+0000 7ff6a2c11b80 1 -- 172.21.15.38:0/2463120946 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x555e660601c0 con 0x555e6616e210 2023-03-09T14:58:27.126 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.119+0000 7ff694824700 1 --2- 172.21.15.38:0/2463120946 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x555e6616e210 0x555e6616e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:58:27.126 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.119+0000 7ff66ffff700 10 client.25952 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:58:27.126 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.120+0000 7ff694824700 1 --2- 172.21.15.38:0/2463120946 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x555e6616e210 0x555e6616e610 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ddc2204470c9045 server_cookie=1e18b141d5d8639a in_seq=0 out_seq=0 2023-03-09T14:58:27.126 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.120+0000 7ff66ffff700 1 -- 172.21.15.38:0/2463120946 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+265 (crc 0 0 0) 0x555e660601c0 con 0x555e6616e210 2023-03-09T14:58:27.126 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.120+0000 7ff66ffff700 10 client.25952 handle_command_reply: tid=0 2023-03-09T14:58:27.126 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.120+0000 7ff6a2c11b80 1 client.25952 shutdown 2023-03-09T14:58:27.126 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.120+0000 7ff6a2c11b80 1 -- 172.21.15.38:0/2463120946 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e66170bd0 msgr2=0x555e661d1240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:58:27.127 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.120+0000 7ff6a2c11b80 1 --2- 172.21.15.38:0/2463120946 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e66170bd0 0x555e661d1240 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff67c000c00 tx=0x7ff67c005590 comp rx=0 tx=0).stop 2023-03-09T14:58:27.127 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.120+0000 7ff6a2c11b80 1 -- 172.21.15.38:0/2463120946 shutdown_connections 2023-03-09T14:58:27.127 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.120+0000 7ff6a2c11b80 1 --2- 172.21.15.38:0/2463120946 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x555e6616e210 0x555e6616e610 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:58:27.127 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.120+0000 7ff6a2c11b80 1 --2- 172.21.15.38:0/2463120946 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x555e661702c0 0x555e661d7cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:58:27.127 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.120+0000 7ff66ffff700 0 client.25952 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:58:27.127 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.120+0000 7ff6a2c11b80 1 --2- 172.21.15.38:0/2463120946 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x555e66179870 0x555e661d1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:58:27.127 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.120+0000 7ff6a2c11b80 1 --2- 172.21.15.38:0/2463120946 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x555e66170bd0 0x555e661d1240 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:58:27.128 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.120+0000 7ff6a2c11b80 1 -- 172.21.15.38:0/2463120946 >> 172.21.15.38:0/2463120946 conn(0x555e661615a0 msgr2=0x555e66184b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:58:27.128 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.120+0000 7ff6a2c11b80 1 -- 172.21.15.38:0/2463120946 shutdown_connections 2023-03-09T14:58:27.128 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.120+0000 7ff6a2c11b80 1 -- 172.21.15.38:0/2463120946 wait complete. 2023-03-09T14:58:27.128 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.120+0000 7ff6a2c11b80 20 client.25952 trim_cache size 0 max 16384 2023-03-09T14:58:27.128 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2023-03-09T14:58:27.128 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "scrub active (1 inodes in the stack)", 2023-03-09T14:58:27.129 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": { 2023-03-09T14:58:27.129 INFO:teuthology.orchestra.run.smithi038.stdout: "a14659f4-d7ea-4c5a-b15b-6138af3f909f": { 2023-03-09T14:58:27.129 INFO:teuthology.orchestra.run.smithi038.stdout: "path": "/", 2023-03-09T14:58:27.129 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "a14659f4-d7ea-4c5a-b15b-6138af3f909f", 2023-03-09T14:58:27.129 INFO:teuthology.orchestra.run.smithi038.stdout: "options": "recursive,force" 2023-03-09T14:58:27.129 INFO:teuthology.orchestra.run.smithi038.stdout: } 2023-03-09T14:58:27.129 INFO:teuthology.orchestra.run.smithi038.stdout: } 2023-03-09T14:58:27.129 INFO:teuthology.orchestra.run.smithi038.stdout:} 2023-03-09T14:58:27.130 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.121+0000 7ff696a88700 1 -- 172.21.15.38:0/4132310035 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7ff68009f4a0 msgr2=0x7ff6800a1950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:58:27.130 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.121+0000 7ff696a88700 1 --2- 172.21.15.38:0/4132310035 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7ff68009f4a0 0x7ff6800a1950 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff684009a20 tx=0x7ff684009210 comp rx=0 tx=0).stop 2023-03-09T14:58:27.130 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.121+0000 7ff696a88700 1 -- 172.21.15.38:0/4132310035 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff69019ac90 msgr2=0x7ff69019b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:58:27.131 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.121+0000 7ff696a88700 1 --2- 172.21.15.38:0/4132310035 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff69019ac90 0x7ff69019b100 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff68800efc0 tx=0x7ff68800c3a0 comp rx=0 tx=0).stop 2023-03-09T14:58:27.131 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.121+0000 7ff696a88700 1 -- 172.21.15.38:0/4132310035 shutdown_connections 2023-03-09T14:58:27.131 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.121+0000 7ff696a88700 1 --2- 172.21.15.38:0/4132310035 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7ff68009f4a0 0x7ff6800a1950 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:58:27.131 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.121+0000 7ff696a88700 1 --2- 172.21.15.38:0/4132310035 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7ff690100b20 0x7ff6901962f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:58:27.131 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.121+0000 7ff696a88700 1 --2- 172.21.15.38:0/4132310035 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7ff69019ac90 0x7ff69019b100 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:58:27.131 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.121+0000 7ff696a88700 1 --2- 172.21.15.38:0/4132310035 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7ff6901051c0 0x7ff690196830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:58:27.131 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.121+0000 7ff696a88700 1 -- 172.21.15.38:0/4132310035 >> 172.21.15.38:0/4132310035 conn(0x7ff6900fb9d0 msgr2=0x7ff690104350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:58:27.131 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.121+0000 7ff696a88700 1 -- 172.21.15.38:0/4132310035 shutdown_connections 2023-03-09T14:58:27.132 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:27.121+0000 7ff696a88700 1 -- 172.21.15.38:0/4132310035 wait complete. 2023-03-09T14:58:27.139 INFO:tasks.cephfs.filesystem:scrub status for tag:a14659f4-d7ea-4c5a-b15b-6138af3f909f - {'path': '/', 'tag': 'a14659f4-d7ea-4c5a-b15b-6138af3f909f', 'options': 'recursive,force'} 2023-03-09T14:58:28.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:58:28 smithi122 ceph-mon[93869]: pgmap v526: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 20 MiB/s wr, 2.42k op/s 2023-03-09T14:58:28.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:58:28 smithi181 ceph-mon[94189]: pgmap v526: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 20 MiB/s wr, 2.42k op/s 2023-03-09T14:58:28.506 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:58:28 smithi038 ceph-mon[92422]: pgmap v526: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 20 MiB/s wr, 2.42k op/s 2023-03-09T14:58:28.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 480.0 s, 566.6 tps, lat 173.837 ms stddev 511.019 2023-03-09T14:58:30.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:58:30 smithi122 ceph-mon[93869]: pgmap v527: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 20 MiB/s wr, 2.36k op/s 2023-03-09T14:58:30.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:58:30 smithi181 ceph-mon[94189]: pgmap v527: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 20 MiB/s wr, 2.36k op/s 2023-03-09T14:58:30.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:58:30 smithi038 ceph-mon[92422]: pgmap v527: 129 pgs: 129 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 20 MiB/s wr, 2.36k op/s 2023-03-09T14:58:31.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:58:31 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T14:58:31.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:58:31 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T14:58:31.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:58:31 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T14:58:32.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:58:32 smithi122 ceph-mon[93869]: pgmap v528: 129 pgs: 129 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 19 MiB/s wr, 2.38k op/s 2023-03-09T14:58:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:58:32 smithi181 ceph-mon[94189]: pgmap v528: 129 pgs: 129 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 19 MiB/s wr, 2.38k op/s 2023-03-09T14:58:32.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:58:32 smithi038 ceph-mon[92422]: pgmap v528: 129 pgs: 129 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 19 MiB/s wr, 2.38k op/s 2023-03-09T14:58:33.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 485.0 s, 629.8 tps, lat 160.832 ms stddev 482.630 2023-03-09T14:58:34.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:58:33 smithi122 ceph-mon[93869]: pgmap v529: 129 pgs: 129 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 26 MiB/s wr, 3.10k op/s 2023-03-09T14:58:34.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:58:33 smithi181 ceph-mon[94189]: pgmap v529: 129 pgs: 129 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 26 MiB/s wr, 3.10k op/s 2023-03-09T14:58:34.254 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:58:33 smithi038 ceph-mon[92422]: pgmap v529: 129 pgs: 129 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 26 MiB/s wr, 3.10k op/s 2023-03-09T14:58:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:58:36 smithi122 ceph-mon[93869]: pgmap v530: 129 pgs: 129 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 24 MiB/s wr, 2.83k op/s 2023-03-09T14:58:36.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:58:36 smithi181 ceph-mon[94189]: pgmap v530: 129 pgs: 129 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 24 MiB/s wr, 2.83k op/s 2023-03-09T14:58:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:58:36 smithi038 ceph-mon[92422]: pgmap v530: 129 pgs: 129 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 24 MiB/s wr, 2.83k op/s 2023-03-09T14:58:38.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:58:38 smithi122 ceph-mon[93869]: pgmap v531: 129 pgs: 129 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 25 MiB/s wr, 3.08k op/s 2023-03-09T14:58:38.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:58:38 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T14:58:38.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:58:38 smithi181 ceph-mon[94189]: pgmap v531: 129 pgs: 129 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 25 MiB/s wr, 3.08k op/s 2023-03-09T14:58:38.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:58:38 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T14:58:38.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:58:38 smithi038 ceph-mon[92422]: pgmap v531: 129 pgs: 129 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 25 MiB/s wr, 3.08k op/s 2023-03-09T14:58:38.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:58:38 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T14:58:38.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 490.0 s, 912.3 tps, lat 108.012 ms stddev 273.249 2023-03-09T14:58:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:58:40 smithi181 ceph-mon[94189]: pgmap v532: 129 pgs: 129 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 23 MiB/s wr, 2.80k op/s 2023-03-09T14:58:40.489 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:58:40 smithi122 ceph-mon[93869]: pgmap v532: 129 pgs: 129 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 23 MiB/s wr, 2.80k op/s 2023-03-09T14:58:40.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:58:40 smithi038 ceph-mon[92422]: pgmap v532: 129 pgs: 129 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 23 MiB/s wr, 2.80k op/s 2023-03-09T14:58:42.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:58:42 smithi122 ceph-mon[93869]: pgmap v533: 129 pgs: 129 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 22 MiB/s wr, 2.61k op/s 2023-03-09T14:58:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:58:42 smithi181 ceph-mon[94189]: pgmap v533: 129 pgs: 129 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 22 MiB/s wr, 2.61k op/s 2023-03-09T14:58:42.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:58:42 smithi038 ceph-mon[92422]: pgmap v533: 129 pgs: 129 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 22 MiB/s wr, 2.61k op/s 2023-03-09T14:58:43.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 495.0 s, 804.4 tps, lat 123.008 ms stddev 354.408 2023-03-09T14:58:44.234 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:58:43 smithi122 ceph-mon[93869]: pgmap v534: 129 pgs: 129 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 26 MiB/s wr, 3.06k op/s 2023-03-09T14:58:44.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:58:43 smithi181 ceph-mon[94189]: pgmap v534: 129 pgs: 129 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 26 MiB/s wr, 3.06k op/s 2023-03-09T14:58:44.258 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:58:43 smithi038 ceph-mon[92422]: pgmap v534: 129 pgs: 129 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 26 MiB/s wr, 3.06k op/s 2023-03-09T14:58:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:58:46 smithi122 ceph-mon[93869]: pgmap v535: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 21 MiB/s wr, 2.43k op/s 2023-03-09T14:58:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:58:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:58:46.485 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:58:46 smithi181 ceph-mon[94189]: pgmap v535: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 21 MiB/s wr, 2.43k op/s 2023-03-09T14:58:46.485 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:58:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:58:46.509 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:58:46 smithi038 ceph-mon[92422]: pgmap v535: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 21 MiB/s wr, 2.43k op/s 2023-03-09T14:58:46.509 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:58:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:58:48.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:58:48 smithi122 ceph-mon[93869]: pgmap v536: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 22 MiB/s wr, 2.52k op/s 2023-03-09T14:58:48.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:58:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:58:48.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:58:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:58:48.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:58:48 smithi181 ceph-mon[94189]: pgmap v536: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 22 MiB/s wr, 2.52k op/s 2023-03-09T14:58:48.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:58:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:58:48.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:58:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:58:48.503 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:58:48 smithi038 ceph-mon[92422]: pgmap v536: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 22 MiB/s wr, 2.52k op/s 2023-03-09T14:58:48.504 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:58:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:58:48.504 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:58:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:58:48.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 500.0 s, 262.1 tps, lat 306.204 ms stddev 448.240 2023-03-09T14:58:49.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:58:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:58:49.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:58:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:58:49.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:58:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:58:49.483 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:58:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:58:49.483 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:58:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:58:49.483 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:58:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:58:49.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:58:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:58:49.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:58:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:58:49.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:58:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:58:50.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:58:50 smithi122 ceph-mon[93869]: pgmap v537: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 22 MiB/s wr, 2.58k op/s 2023-03-09T14:58:50.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:58:50 smithi181 ceph-mon[94189]: pgmap v537: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 22 MiB/s wr, 2.58k op/s 2023-03-09T14:58:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:58:50 smithi038 ceph-mon[92422]: pgmap v537: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 22 MiB/s wr, 2.58k op/s 2023-03-09T14:58:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:58:52 smithi122 ceph-mon[93869]: pgmap v538: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 19 MiB/s wr, 2.30k op/s 2023-03-09T14:58:52.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:58:52 smithi181 ceph-mon[94189]: pgmap v538: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 19 MiB/s wr, 2.30k op/s 2023-03-09T14:58:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:58:52 smithi038 ceph-mon[92422]: pgmap v538: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 19 MiB/s wr, 2.30k op/s 2023-03-09T14:58:53.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 505.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T14:58:54.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:58:53 smithi122 ceph-mon[93869]: pgmap v539: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 18 MiB/s wr, 2.16k op/s 2023-03-09T14:58:54.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:58:53 smithi181 ceph-mon[94189]: pgmap v539: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 18 MiB/s wr, 2.16k op/s 2023-03-09T14:58:54.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:58:53 smithi038 ceph-mon[92422]: pgmap v539: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 18 MiB/s wr, 2.16k op/s 2023-03-09T14:58:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:58:56 smithi122 ceph-mon[93869]: pgmap v540: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 12 MiB/s wr, 1.44k op/s 2023-03-09T14:58:56.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:58:56 smithi181 ceph-mon[94189]: pgmap v540: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 12 MiB/s wr, 1.44k op/s 2023-03-09T14:58:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:58:56 smithi038 ceph-mon[92422]: pgmap v540: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 12 MiB/s wr, 1.44k op/s 2023-03-09T14:58:57.140 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-03-09T14:58:57.341 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.335+0000 7f3dbc590700 1 -- 172.21.15.38:0/1753555600 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3db4107780 msgr2=0x7f3db410e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:58:57.341 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.335+0000 7f3dbc590700 1 --2- 172.21.15.38:0/1753555600 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3db4107780 0x7f3db410e7e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f3da4009990 tx=0x7f3da4009ca0 comp rx=0 tx=0).stop 2023-03-09T14:58:57.344 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.342+0000 7f3dbc590700 1 -- 172.21.15.38:0/1753555600 shutdown_connections 2023-03-09T14:58:57.344 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.342+0000 7f3dbc590700 1 --2- 172.21.15.38:0/1753555600 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3db41196d0 0x7f3db411bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:58:57.344 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.342+0000 7f3dbc590700 1 --2- 172.21.15.38:0/1753555600 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3db4107780 0x7f3db410e7e0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:58:57.344 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.342+0000 7f3dbc590700 1 --2- 172.21.15.38:0/1753555600 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3db4107b50 0x7f3db4107fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:58:57.344 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.342+0000 7f3dbc590700 1 -- 172.21.15.38:0/1753555600 >> 172.21.15.38:0/1753555600 conn(0x7f3db406b450 msgr2=0x7f3db40712c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:58:57.347 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.346+0000 7f3dbc590700 1 -- 172.21.15.38:0/1753555600 shutdown_connections 2023-03-09T14:58:57.352 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.350+0000 7f3dbc590700 1 -- 172.21.15.38:0/1753555600 wait complete. 2023-03-09T14:58:57.353 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.351+0000 7f3dbc590700 1 Processor -- start 2023-03-09T14:58:57.354 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.351+0000 7f3dbc590700 1 -- start start 2023-03-09T14:58:57.354 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.352+0000 7f3dbc590700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3db4107780 0x7f3db410f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:58:57.354 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.352+0000 7f3dbc590700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3db4107b50 0x7f3db410ff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:58:57.354 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.352+0000 7f3dbc590700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3db41196d0 0x7f3db41b93c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:58:57.354 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.352+0000 7f3dbc590700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3db41b9990 con 0x7f3db4107780 2023-03-09T14:58:57.354 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.352+0000 7f3dbc590700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3db41b9ad0 con 0x7f3db4107b50 2023-03-09T14:58:57.355 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.352+0000 7f3dbc590700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3db41b9c60 con 0x7f3db41196d0 2023-03-09T14:58:57.355 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.352+0000 7f3dbab2d700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3db41196d0 0x7f3db41b93c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:58:57.355 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.352+0000 7f3dbab2d700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3db41196d0 0x7f3db41b93c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.181:3300/0 says I am v2:172.21.15.38:49902/0 (socket says 172.21.15.38:49902) 2023-03-09T14:58:57.355 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.352+0000 7f3dbab2d700 1 -- 172.21.15.38:0/133867986 learned_addr learned my addr 172.21.15.38:0/133867986 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:58:57.355 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.352+0000 7f3dba32c700 1 --2- 172.21.15.38:0/133867986 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3db4107780 0x7f3db410f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:58:57.355 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.352+0000 7f3db9b2b700 1 --2- 172.21.15.38:0/133867986 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3db4107b50 0x7f3db410ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:58:57.356 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.352+0000 7f3dbab2d700 1 -- 172.21.15.38:0/133867986 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3db4107b50 msgr2=0x7f3db410ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:58:57.357 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.352+0000 7f3dbab2d700 1 --2- 172.21.15.38:0/133867986 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3db4107b50 0x7f3db410ff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:58:57.357 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.352+0000 7f3dbab2d700 1 -- 172.21.15.38:0/133867986 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3db4107780 msgr2=0x7f3db410f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:58:57.357 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.352+0000 7f3dbab2d700 1 --2- 172.21.15.38:0/133867986 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3db4107780 0x7f3db410f9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:58:57.357 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.352+0000 7f3dbab2d700 1 -- 172.21.15.38:0/133867986 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3da40095f0 con 0x7f3db41196d0 2023-03-09T14:58:57.357 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.352+0000 7f3dba32c700 1 --2- 172.21.15.38:0/133867986 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3db4107780 0x7f3db410f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:58:57.357 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.354+0000 7f3dbab2d700 1 --2- 172.21.15.38:0/133867986 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3db41196d0 0x7f3db41b93c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f3db000e9a0 tx=0x7f3db000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:58:57.358 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.354+0000 7f3dab7fe700 1 -- 172.21.15.38:0/133867986 <== mon.1 v2:172.21.15.181:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3db0009750 con 0x7f3db41196d0 2023-03-09T14:58:57.358 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.354+0000 7f3dab7fe700 1 -- 172.21.15.38:0/133867986 <== mon.1 v2:172.21.15.181:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3db0005020 con 0x7f3db41196d0 2023-03-09T14:58:57.358 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.354+0000 7f3dbc590700 1 -- 172.21.15.38:0/133867986 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3db411edd0 con 0x7f3db41196d0 2023-03-09T14:58:57.358 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.354+0000 7f3dab7fe700 1 -- 172.21.15.38:0/133867986 <== mon.1 v2:172.21.15.181:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3db000cc10 con 0x7f3db41196d0 2023-03-09T14:58:57.358 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.354+0000 7f3dbc590700 1 -- 172.21.15.38:0/133867986 --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3db411ec40 con 0x7f3db41196d0 2023-03-09T14:58:57.359 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.356+0000 7f3dab7fe700 1 -- 172.21.15.38:0/133867986 <== mon.1 v2:172.21.15.181:3300/0 4 ==== mgrmap(e 25) v1 ==== 133295+0+0 (secure 0 0 0) 0x7f3db001e020 con 0x7f3db41196d0 2023-03-09T14:58:57.359 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.357+0000 7f3dab7fe700 1 --2- 172.21.15.38:0/133867986 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f3da009f4d0 0x7f3da00a1980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:58:57.359 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.357+0000 7f3dab7fe700 1 -- 172.21.15.38:0/133867986 <== mon.1 v2:172.21.15.181:3300/0 5 ==== osd_map(247..247 src has 1..247) v4 ==== 9577+0+0 (secure 0 0 0) 0x7f3db0014070 con 0x7f3db41196d0 2023-03-09T14:58:57.359 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.357+0000 7f3dba32c700 1 --2- 172.21.15.38:0/133867986 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f3da009f4d0 0x7f3da00a1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:58:57.361 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.358+0000 7f3dba32c700 1 --2- 172.21.15.38:0/133867986 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f3da009f4d0 0x7f3da00a1980 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3da400b2e0 tx=0x7f3da4005fd0 comp rx=0 tx=0).ready entity=mgr.15558 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:58:57.361 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.358+0000 7f3dc8719b80 1 Processor -- start 2023-03-09T14:58:57.361 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.358+0000 7f3dc8719b80 1 -- start start 2023-03-09T14:58:57.361 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.358+0000 7f3dc8719b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558bf8fa3830 0x558bf8fa3c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:58:57.362 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.358+0000 7f3dc8719b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558bf8fa4140 0x558bf8f9daa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:58:57.362 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.358+0000 7f3dc8719b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bf8f9dfe0 0x558bf8f9e410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:58:57.362 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.358+0000 7f3dc8719b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x558bf8e87da0 con 0x558bf8f9dfe0 2023-03-09T14:58:57.362 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.358+0000 7f3dc8719b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x558bf8ef6780 con 0x558bf8fa4140 2023-03-09T14:58:57.362 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.358+0000 7f3dc8719b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x558bf8eefcb0 con 0x558bf8fa3830 2023-03-09T14:58:57.362 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.358+0000 7f3dbab2d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bf8f9dfe0 0x558bf8f9e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:58:57.362 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.358+0000 7f3dba32c700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558bf8fa3830 0x558bf8fa3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:58:57.362 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.358+0000 7f3dbab2d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bf8f9dfe0 0x558bf8f9e410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42214/0 (socket says 172.21.15.38:42214) 2023-03-09T14:58:57.363 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.358+0000 7f3db9b2b700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558bf8fa4140 0x558bf8f9daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:58:57.363 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.358+0000 7f3dbab2d700 1 -- 172.21.15.38:0/586776992 learned_addr learned my addr 172.21.15.38:0/586776992 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:58:57.363 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.359+0000 7f3dbab2d700 1 -- 172.21.15.38:0/586776992 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558bf8fa3830 msgr2=0x558bf8fa3c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:58:57.363 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.359+0000 7f3dbab2d700 1 --2- 172.21.15.38:0/586776992 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558bf8fa3830 0x558bf8fa3c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:58:57.363 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.359+0000 7f3dbab2d700 1 -- 172.21.15.38:0/586776992 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558bf8fa4140 msgr2=0x558bf8f9daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:58:57.363 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.359+0000 7f3dbab2d700 1 --2- 172.21.15.38:0/586776992 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558bf8fa4140 0x558bf8f9daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:58:57.363 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.359+0000 7f3dbab2d700 1 -- 172.21.15.38:0/586776992 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558bf8eb6190 con 0x558bf8f9dfe0 2023-03-09T14:58:57.364 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.359+0000 7f3db9b2b700 1 --2- 172.21.15.38:0/586776992 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558bf8fa4140 0x558bf8f9daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:58:57.364 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.359+0000 7f3dbab2d700 1 --2- 172.21.15.38:0/586776992 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bf8f9dfe0 0x558bf8f9e410 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f3db009fc70 tx=0x7f3db00979e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8adb3a0cd2158716 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:58:57.364 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.359+0000 7f3da8ff9700 1 -- 172.21.15.38:0/586776992 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3db00a01a0 con 0x558bf8f9dfe0 2023-03-09T14:58:57.364 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.359+0000 7f3da8ff9700 1 -- 172.21.15.38:0/586776992 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3db00b0270 con 0x558bf8f9dfe0 2023-03-09T14:58:57.364 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.359+0000 7f3da8ff9700 1 -- 172.21.15.38:0/586776992 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3db0098310 con 0x558bf8f9dfe0 2023-03-09T14:58:57.367 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.360+0000 7f3dc8719b80 1 -- 172.21.15.38:0/586776992 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bf8f9dfe0 msgr2=0x558bf8f9e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:58:57.367 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.360+0000 7f3dc8719b80 1 --2- 172.21.15.38:0/586776992 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bf8f9dfe0 0x558bf8f9e410 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f3db009fc70 tx=0x7f3db00979e0 comp rx=0 tx=0).stop 2023-03-09T14:58:57.367 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.360+0000 7f3dc8719b80 1 -- 172.21.15.38:0/586776992 shutdown_connections 2023-03-09T14:58:57.368 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.360+0000 7f3dc8719b80 1 --2- 172.21.15.38:0/586776992 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bf8f9dfe0 0x558bf8f9e410 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:58:57.368 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.360+0000 7f3dc8719b80 1 --2- 172.21.15.38:0/586776992 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558bf8fa3830 0x558bf8fa3c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:58:57.368 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.360+0000 7f3dc8719b80 1 --2- 172.21.15.38:0/586776992 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558bf8fa4140 0x558bf8f9daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:58:57.368 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.360+0000 7f3dc8719b80 1 -- 172.21.15.38:0/586776992 >> 172.21.15.38:0/586776992 conn(0x558bf8f98a20 msgr2=0x558bf8f98df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:58:57.368 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.360+0000 7f3dc8719b80 1 -- 172.21.15.38:0/586776992 shutdown_connections 2023-03-09T14:58:57.368 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.360+0000 7f3dc8719b80 1 -- 172.21.15.38:0/586776992 wait complete. 2023-03-09T14:58:57.368 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.360+0000 7f3dc8719b80 1 -- start start 2023-03-09T14:58:57.368 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.360+0000 7f3dc8719b80 1 Processor -- start 2023-03-09T14:58:57.369 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.360+0000 7f3dc8719b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bf8f9dfe0 0x558bf8faa0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:58:57.369 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.360+0000 7f3dc8719b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558bf8fa3830 0x558bf8fa7370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:58:57.369 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.360+0000 7f3dc8719b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558bf8fa4140 0x558bf8fa79d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:58:57.369 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.360+0000 7f3dc8719b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x558bf8d73920 con 0x558bf8f9dfe0 2023-03-09T14:58:57.369 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.360+0000 7f3dc8719b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x558bf8fa8140 con 0x558bf8fa3830 2023-03-09T14:58:57.369 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.360+0000 7f3dc8719b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x558bf8fb3c70 con 0x558bf8fa4140 2023-03-09T14:58:57.369 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.360+0000 7f3dc8719b80 10 client.0 _ll_register_callbacks cb 0x558bf8e95520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:58:57.370 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.360+0000 7f3dba32c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bf8f9dfe0 0x558bf8faa0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:58:57.370 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.360+0000 7f3dba32c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bf8f9dfe0 0x558bf8faa0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42216/0 (socket says 172.21.15.38:42216) 2023-03-09T14:58:57.370 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.361+0000 7f3dba32c700 1 -- 172.21.15.38:0/2832028042 learned_addr learned my addr 172.21.15.38:0/2832028042 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:58:57.370 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.361+0000 7f3db9b2b700 1 --2- 172.21.15.38:0/2832028042 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558bf8fa3830 0x558bf8fa7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:58:57.370 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.361+0000 7f3dbab2d700 1 --2- 172.21.15.38:0/2832028042 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558bf8fa4140 0x558bf8fa79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:58:57.370 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.361+0000 7f3dba32c700 1 -- 172.21.15.38:0/2832028042 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558bf8fa4140 msgr2=0x558bf8fa79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:58:57.370 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.361+0000 7f3dba32c700 1 --2- 172.21.15.38:0/2832028042 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558bf8fa4140 0x558bf8fa79d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:58:57.370 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.361+0000 7f3dba32c700 1 -- 172.21.15.38:0/2832028042 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558bf8fa3830 msgr2=0x558bf8fa7370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:58:57.371 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.361+0000 7f3dba32c700 1 --2- 172.21.15.38:0/2832028042 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558bf8fa3830 0x558bf8fa7370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:58:57.371 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.361+0000 7f3dba32c700 1 -- 172.21.15.38:0/2832028042 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dac009c40 con 0x558bf8f9dfe0 2023-03-09T14:58:57.371 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.361+0000 7f3db9b2b700 1 --2- 172.21.15.38:0/2832028042 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558bf8fa3830 0x558bf8fa7370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:58:57.371 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.361+0000 7f3dbab2d700 1 --2- 172.21.15.38:0/2832028042 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558bf8fa4140 0x558bf8fa79d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:58:57.371 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.361+0000 7f3dba32c700 1 --2- 172.21.15.38:0/2832028042 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bf8f9dfe0 0x558bf8faa0f0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f3da4004850 tx=0x7f3da4009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ca438241034b026 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:58:57.371 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.361+0000 7f3d9dffb700 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2023-03-09T14:58:57.371 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.361+0000 7f3d9dffb700 1 -- 172.21.15.38:0/2832028042 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3da400f7b0 con 0x558bf8f9dfe0 2023-03-09T14:58:57.372 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.361+0000 7f3d9dffb700 1 -- 172.21.15.38:0/2832028042 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(247..247 src has 1..247) v4 ==== 9577+0+0 (secure 0 0 0) 0x7f3da400f910 con 0x558bf8f9dfe0 2023-03-09T14:58:57.372 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.361+0000 7f3dc8719b80 1 -- 172.21.15.38:0/2832028042 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558bf8ebd190 con 0x558bf8f9dfe0 2023-03-09T14:58:57.372 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.361+0000 7f3d9dffb700 1 -- 172.21.15.38:0/2832028042 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3da400fad0 con 0x558bf8f9dfe0 2023-03-09T14:58:57.372 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.361+0000 7f3dc8719b80 10 client.16329 fetch_fsmap learned FSMap version 43 2023-03-09T14:58:57.372 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.361+0000 7f3dc8719b80 1 -- 172.21.15.38:0/2832028042 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) v3 -- 0x7f3db411ec40 con 0x558bf8f9dfe0 2023-03-09T14:58:57.372 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.362+0000 7f3d9dffb700 1 -- 172.21.15.38:0/2832028042 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x558bf8ebd190 con 0x558bf8f9dfe0 2023-03-09T14:58:57.372 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.362+0000 7f3d9dffb700 1 -- 172.21.15.38:0/2832028042 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 43) v1 ==== 7482+0+0 (secure 0 0 0) 0x7f3da402f070 con 0x558bf8f9dfe0 2023-03-09T14:58:57.372 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.362+0000 7f3dc8719b80 10 client.16329 fetch_fsmap finished waiting for FSMap version 43 2023-03-09T14:58:57.373 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.362+0000 7f3dc8719b80 10 client.16329 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T14:58:57.373 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.362+0000 7f3dc8719b80 20 client.16329 populate_metadata read hostname 'smithi038' 2023-03-09T14:58:57.373 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.362+0000 7f3dc8719b80 1 --2- 172.21.15.38:0/2832028042 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x558bf8fb7e40 0x558bf8fba250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:58:57.373 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.362+0000 7f3dc8719b80 4 client.16329 mds_command: new command op to 24530 tid=0[{"prefix": "get_command_descriptions"}] 2023-03-09T14:58:57.373 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.362+0000 7f3dc8719b80 1 -- 172.21.15.38:0/2832028042 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558bf8d73f60 con 0x558bf8fb7e40 2023-03-09T14:58:57.373 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.362+0000 7f3db9b2b700 1 --2- 172.21.15.38:0/2832028042 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x558bf8fb7e40 0x558bf8fba250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:58:57.373 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.362+0000 7f3db9b2b700 1 --2- 172.21.15.38:0/2832028042 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x558bf8fb7e40 0x558bf8fba250 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2beec7c4e3403e4f server_cookie=174e2441392458fe in_seq=0 out_seq=0 2023-03-09T14:58:57.374 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.362+0000 7f3d9dffb700 10 client.16329 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:58:57.374 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.363+0000 7f3d9dffb700 1 -- 172.21.15.38:0/2832028042 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18278 (crc 0 0 0) 0x558bf8d73f60 con 0x558bf8fb7e40 2023-03-09T14:58:57.374 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.363+0000 7f3d9dffb700 10 client.16329 handle_command_reply: tid=0 2023-03-09T14:58:57.374 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.363+0000 7f3dc8719b80 1 client.16329 shutdown 2023-03-09T14:58:57.374 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.363+0000 7f3dc8719b80 1 -- 172.21.15.38:0/2832028042 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bf8f9dfe0 msgr2=0x558bf8faa0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:58:57.374 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.363+0000 7f3dc8719b80 1 --2- 172.21.15.38:0/2832028042 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bf8f9dfe0 0x558bf8faa0f0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f3da4004850 tx=0x7f3da4009ca0 comp rx=0 tx=0).stop 2023-03-09T14:58:57.374 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.363+0000 7f3dc8719b80 1 -- 172.21.15.38:0/2832028042 shutdown_connections 2023-03-09T14:58:57.374 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.363+0000 7f3dc8719b80 1 --2- 172.21.15.38:0/2832028042 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x558bf8fb7e40 0x558bf8fba250 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:58:57.375 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.363+0000 7f3dc8719b80 1 --2- 172.21.15.38:0/2832028042 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bf8f9dfe0 0x558bf8faa0f0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:58:57.375 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.363+0000 7f3d9dffb700 0 client.16329 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:58:57.375 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.363+0000 7f3dc8719b80 1 --2- 172.21.15.38:0/2832028042 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558bf8fa4140 0x558bf8fa79d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:58:57.375 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.363+0000 7f3dc8719b80 1 --2- 172.21.15.38:0/2832028042 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558bf8fa3830 0x558bf8fa7370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:58:57.375 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.363+0000 7f3dc8719b80 1 -- 172.21.15.38:0/2832028042 >> 172.21.15.38:0/2832028042 conn(0x558bf8f98a20 msgr2=0x558bf8f98df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:58:57.375 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.363+0000 7f3dc8719b80 1 -- 172.21.15.38:0/2832028042 shutdown_connections 2023-03-09T14:58:57.375 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.363+0000 7f3dc8719b80 1 -- 172.21.15.38:0/2832028042 wait complete. 2023-03-09T14:58:57.376 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.363+0000 7f3dc8719b80 20 client.16329 trim_cache size 0 max 16384 2023-03-09T14:58:57.383 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.380+0000 7f3dc8719b80 1 Processor -- start 2023-03-09T14:58:57.383 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.380+0000 7f3dc8719b80 1 -- start start 2023-03-09T14:58:57.383 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.380+0000 7f3dc8719b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bf8f9dfe0 0x558bf8fa9c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:58:57.383 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.380+0000 7f3dc8719b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558bf8fa3830 0x558bf8f98df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:58:57.383 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.380+0000 7f3dc8719b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558bf8fa4140 0x558bf8fa1530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:58:57.383 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.380+0000 7f3dc8719b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x558bf8fb53f0 con 0x558bf8f9dfe0 2023-03-09T14:58:57.383 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.380+0000 7f3dc8719b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x558bf8f99f20 con 0x558bf8fa4140 2023-03-09T14:58:57.384 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.380+0000 7f3dc8719b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x558bf8fa2040 con 0x558bf8fa3830 2023-03-09T14:58:57.384 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.381+0000 7f3dba32c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bf8f9dfe0 0x558bf8fa9c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:58:57.384 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.381+0000 7f3dbab2d700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558bf8fa4140 0x558bf8fa1530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:58:57.384 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.381+0000 7f3dbab2d700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558bf8fa4140 0x558bf8fa1530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.38:51418/0 (socket says 172.21.15.38:51418) 2023-03-09T14:58:57.384 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.381+0000 7f3dba32c700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bf8f9dfe0 0x558bf8fa9c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42224/0 (socket says 172.21.15.38:42224) 2023-03-09T14:58:57.384 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.381+0000 7f3dbab2d700 1 -- 172.21.15.38:0/1351987952 learned_addr learned my addr 172.21.15.38:0/1351987952 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:58:57.384 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.381+0000 7f3db9b2b700 1 --2- 172.21.15.38:0/1351987952 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558bf8fa3830 0x558bf8f98df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:58:57.385 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.381+0000 7f3dba32c700 1 -- 172.21.15.38:0/1351987952 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558bf8fa3830 msgr2=0x558bf8f98df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:58:57.385 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.381+0000 7f3dba32c700 1 --2- 172.21.15.38:0/1351987952 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558bf8fa3830 0x558bf8f98df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:58:57.385 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.381+0000 7f3dba32c700 1 -- 172.21.15.38:0/1351987952 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558bf8fa4140 msgr2=0x558bf8fa1530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:58:57.385 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.381+0000 7f3dba32c700 1 --2- 172.21.15.38:0/1351987952 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558bf8fa4140 0x558bf8fa1530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:58:57.385 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.381+0000 7f3dba32c700 1 -- 172.21.15.38:0/1351987952 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558bf8d06c60 con 0x558bf8f9dfe0 2023-03-09T14:58:57.385 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.381+0000 7f3dba32c700 1 --2- 172.21.15.38:0/1351987952 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bf8f9dfe0 0x558bf8fa9c90 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f3da400bca0 tx=0x7f3da4009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a6972f3184e31a2 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:58:57.388 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.381+0000 7f3da97fa700 1 -- 172.21.15.38:0/1351987952 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3da400be80 con 0x558bf8f9dfe0 2023-03-09T14:58:57.388 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.381+0000 7f3da97fa700 1 -- 172.21.15.38:0/1351987952 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3da401e040 con 0x558bf8f9dfe0 2023-03-09T14:58:57.388 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.381+0000 7f3da97fa700 1 -- 172.21.15.38:0/1351987952 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3da4024890 con 0x558bf8f9dfe0 2023-03-09T14:58:57.389 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.382+0000 7f3dc8719b80 1 -- 172.21.15.38:0/1351987952 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bf8f9dfe0 msgr2=0x558bf8fa9c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:58:57.389 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.382+0000 7f3dc8719b80 1 --2- 172.21.15.38:0/1351987952 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bf8f9dfe0 0x558bf8fa9c90 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f3da400bca0 tx=0x7f3da4009310 comp rx=0 tx=0).stop 2023-03-09T14:58:57.389 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.382+0000 7f3dc8719b80 1 -- 172.21.15.38:0/1351987952 shutdown_connections 2023-03-09T14:58:57.389 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.382+0000 7f3dc8719b80 1 --2- 172.21.15.38:0/1351987952 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bf8f9dfe0 0x558bf8fa9c90 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:58:57.389 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.382+0000 7f3dc8719b80 1 --2- 172.21.15.38:0/1351987952 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558bf8fa4140 0x558bf8fa1530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:58:57.389 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.382+0000 7f3dc8719b80 1 --2- 172.21.15.38:0/1351987952 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558bf8fa3830 0x558bf8f98df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:58:57.389 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.382+0000 7f3dc8719b80 1 -- 172.21.15.38:0/1351987952 >> 172.21.15.38:0/1351987952 conn(0x558bf8f98a20 msgr2=0x558bf8f96f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:58:57.390 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.382+0000 7f3dc8719b80 1 -- 172.21.15.38:0/1351987952 shutdown_connections 2023-03-09T14:58:57.390 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.382+0000 7f3dc8719b80 1 -- 172.21.15.38:0/1351987952 wait complete. 2023-03-09T14:58:57.390 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.382+0000 7f3dc8719b80 1 -- start start 2023-03-09T14:58:57.390 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.382+0000 7f3dc8719b80 1 Processor -- start 2023-03-09T14:58:57.390 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.382+0000 7f3dc8719b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558bf8f9dfe0 0x558bf8fa0a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:58:57.390 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.382+0000 7f3dc8719b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bf8fa3830 0x558bf8f9f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:58:57.390 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.382+0000 7f3dc8719b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558bf8fa4140 0x558bf8f9f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:58:57.390 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.382+0000 7f3dc8719b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x558bf8fa1040 con 0x558bf8fa3830 2023-03-09T14:58:57.391 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.382+0000 7f3dc8719b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x558bf8f9fd10 con 0x558bf8fa4140 2023-03-09T14:58:57.391 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.383+0000 7f3dc8719b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x558bf8f9fed0 con 0x558bf8f9dfe0 2023-03-09T14:58:57.391 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.383+0000 7f3dc8719b80 10 client.0 _ll_register_callbacks cb 0x558bf8e95520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:58:57.391 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.383+0000 7f3db9b2b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bf8fa3830 0x558bf8f9f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:58:57.391 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.383+0000 7f3db9b2b700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bf8fa3830 0x558bf8f9f290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42232/0 (socket says 172.21.15.38:42232) 2023-03-09T14:58:57.391 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.383+0000 7f3db9b2b700 1 -- 172.21.15.38:0/637236809 learned_addr learned my addr 172.21.15.38:0/637236809 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:58:57.391 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.383+0000 7f3dba32c700 1 --2- 172.21.15.38:0/637236809 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558bf8f9dfe0 0x558bf8fa0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:58:57.392 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.383+0000 7f3dbab2d700 1 --2- 172.21.15.38:0/637236809 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558bf8fa4140 0x558bf8f9f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:58:57.392 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.383+0000 7f3db9b2b700 1 -- 172.21.15.38:0/637236809 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558bf8f9dfe0 msgr2=0x558bf8fa0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:58:57.392 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.383+0000 7f3db9b2b700 1 --2- 172.21.15.38:0/637236809 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558bf8f9dfe0 0x558bf8fa0a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:58:57.392 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.383+0000 7f3db9b2b700 1 -- 172.21.15.38:0/637236809 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558bf8fa4140 msgr2=0x558bf8f9f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:58:57.392 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.383+0000 7f3db9b2b700 1 --2- 172.21.15.38:0/637236809 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558bf8fa4140 0x558bf8f9f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:58:57.392 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.383+0000 7f3db9b2b700 1 -- 172.21.15.38:0/637236809 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3db411ec40 con 0x558bf8fa3830 2023-03-09T14:58:57.392 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.383+0000 7f3db9b2b700 1 --2- 172.21.15.38:0/637236809 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bf8fa3830 0x558bf8f9f290 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f3db4111030 tx=0x7f3dac00bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58f588905bdccd1e server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:58:57.393 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.383+0000 7f3d9dffb700 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2023-03-09T14:58:57.393 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.383+0000 7f3d9dffb700 1 -- 172.21.15.38:0/637236809 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dac01e040 con 0x558bf8fa3830 2023-03-09T14:58:57.393 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.383+0000 7f3d9dffb700 1 -- 172.21.15.38:0/637236809 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(247..247 src has 1..247) v4 ==== 9577+0+0 (secure 0 0 0) 0x7f3dac0175f0 con 0x558bf8fa3830 2023-03-09T14:58:57.393 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.383+0000 7f3dc8719b80 1 -- 172.21.15.38:0/637236809 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558bf8ead1b0 con 0x558bf8fa3830 2023-03-09T14:58:57.393 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.383+0000 7f3d9dffb700 1 -- 172.21.15.38:0/637236809 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3dac010040 con 0x558bf8fa3830 2023-03-09T14:58:57.393 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.384+0000 7f3dc8719b80 10 client.16335 fetch_fsmap learned FSMap version 43 2023-03-09T14:58:57.393 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.384+0000 7f3dc8719b80 1 -- 172.21.15.38:0/637236809 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) v3 -- 0x558bf8d44250 con 0x558bf8fa3830 2023-03-09T14:58:57.394 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.384+0000 7f3d9dffb700 1 -- 172.21.15.38:0/637236809 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x558bf8ead1b0 con 0x558bf8fa3830 2023-03-09T14:58:57.394 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.384+0000 7f3d9dffb700 1 -- 172.21.15.38:0/637236809 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 43) v1 ==== 7482+0+0 (secure 0 0 0) 0x7f3dac00aa50 con 0x558bf8fa3830 2023-03-09T14:58:57.394 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.384+0000 7f3dc8719b80 10 client.16335 fetch_fsmap finished waiting for FSMap version 43 2023-03-09T14:58:57.394 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.384+0000 7f3dc8719b80 10 client.16335 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T14:58:57.394 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.384+0000 7f3dc8719b80 20 client.16335 populate_metadata read hostname 'smithi038' 2023-03-09T14:58:57.394 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.384+0000 7f3dc8719b80 1 --2- 172.21.15.38:0/637236809 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x558bf900d150 0x558bf900f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:58:57.394 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.384+0000 7f3dc8719b80 4 client.16335 mds_command: new command op to 24530 tid=0[{"prefix": "scrub status"}] 2023-03-09T14:58:57.394 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.384+0000 7f3dc8719b80 1 -- 172.21.15.38:0/637236809 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x558bf8e971c0 con 0x558bf900d150 2023-03-09T14:58:57.395 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.384+0000 7f3dba32c700 1 --2- 172.21.15.38:0/637236809 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x558bf900d150 0x558bf900f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:58:57.395 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.384+0000 7f3d9dffb700 10 client.16335 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:58:57.395 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.384+0000 7f3dba32c700 1 --2- 172.21.15.38:0/637236809 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x558bf900d150 0x558bf900f560 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d5c611d526649256 server_cookie=7e632050423252a8 in_seq=0 out_seq=0 2023-03-09T14:58:57.395 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.385+0000 7f3d9dffb700 1 -- 172.21.15.38:0/637236809 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+265 (crc 0 0 0) 0x558bf8e971c0 con 0x558bf900d150 2023-03-09T14:58:57.395 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.385+0000 7f3d9dffb700 10 client.16335 handle_command_reply: tid=0 2023-03-09T14:58:57.395 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.385+0000 7f3dc8719b80 1 client.16335 shutdown 2023-03-09T14:58:57.395 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.385+0000 7f3dc8719b80 1 -- 172.21.15.38:0/637236809 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bf8fa3830 msgr2=0x558bf8f9f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:58:57.395 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.385+0000 7f3dc8719b80 1 --2- 172.21.15.38:0/637236809 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bf8fa3830 0x558bf8f9f290 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f3db4111030 tx=0x7f3dac00bf90 comp rx=0 tx=0).stop 2023-03-09T14:58:57.396 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.385+0000 7f3dc8719b80 1 -- 172.21.15.38:0/637236809 shutdown_connections 2023-03-09T14:58:57.396 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.385+0000 7f3dc8719b80 1 --2- 172.21.15.38:0/637236809 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x558bf900d150 0x558bf900f560 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:58:57.396 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.385+0000 7f3dc8719b80 1 --2- 172.21.15.38:0/637236809 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x558bf8fa4140 0x558bf8f9f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:58:57.396 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.385+0000 7f3dc8719b80 1 --2- 172.21.15.38:0/637236809 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x558bf8f9dfe0 0x558bf8fa0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:58:57.396 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.385+0000 7f3d9dffb700 0 client.16335 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:58:57.396 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.385+0000 7f3dc8719b80 1 --2- 172.21.15.38:0/637236809 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x558bf8fa3830 0x558bf8f9f290 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:58:57.396 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.385+0000 7f3dc8719b80 1 -- 172.21.15.38:0/637236809 >> 172.21.15.38:0/637236809 conn(0x558bf8f98a20 msgr2=0x558bf8f98df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:58:57.397 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.385+0000 7f3dc8719b80 1 -- 172.21.15.38:0/637236809 shutdown_connections 2023-03-09T14:58:57.397 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.385+0000 7f3dc8719b80 1 -- 172.21.15.38:0/637236809 wait complete. 2023-03-09T14:58:57.397 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.385+0000 7f3dc8719b80 20 client.16335 trim_cache size 0 max 16384 2023-03-09T14:58:57.397 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2023-03-09T14:58:57.397 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "scrub active (1 inodes in the stack)", 2023-03-09T14:58:57.397 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": { 2023-03-09T14:58:57.398 INFO:teuthology.orchestra.run.smithi038.stdout: "a14659f4-d7ea-4c5a-b15b-6138af3f909f": { 2023-03-09T14:58:57.398 INFO:teuthology.orchestra.run.smithi038.stdout: "path": "/", 2023-03-09T14:58:57.398 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "a14659f4-d7ea-4c5a-b15b-6138af3f909f", 2023-03-09T14:58:57.398 INFO:teuthology.orchestra.run.smithi038.stdout: "options": "recursive,force" 2023-03-09T14:58:57.398 INFO:teuthology.orchestra.run.smithi038.stdout: } 2023-03-09T14:58:57.398 INFO:teuthology.orchestra.run.smithi038.stdout: } 2023-03-09T14:58:57.398 INFO:teuthology.orchestra.run.smithi038.stdout:} 2023-03-09T14:58:57.400 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.386+0000 7f3dbc590700 1 -- 172.21.15.38:0/133867986 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f3da009f4d0 msgr2=0x7f3da00a1980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:58:57.400 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.386+0000 7f3dbc590700 1 --2- 172.21.15.38:0/133867986 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f3da009f4d0 0x7f3da00a1980 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3da400b2e0 tx=0x7f3da4005fd0 comp rx=0 tx=0).stop 2023-03-09T14:58:57.400 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.386+0000 7f3dbc590700 1 -- 172.21.15.38:0/133867986 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3db41196d0 msgr2=0x7f3db41b93c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:58:57.400 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.386+0000 7f3dbc590700 1 --2- 172.21.15.38:0/133867986 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3db41196d0 0x7f3db41b93c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f3db000e9a0 tx=0x7f3db000ecb0 comp rx=0 tx=0).stop 2023-03-09T14:58:57.400 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.386+0000 7f3dbc590700 1 -- 172.21.15.38:0/133867986 shutdown_connections 2023-03-09T14:58:57.400 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.386+0000 7f3dbc590700 1 --2- 172.21.15.38:0/133867986 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f3da009f4d0 0x7f3da00a1980 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:58:57.401 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.386+0000 7f3dbc590700 1 --2- 172.21.15.38:0/133867986 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f3db4107780 0x7f3db410f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:58:57.401 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.386+0000 7f3dbc590700 1 --2- 172.21.15.38:0/133867986 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3db41196d0 0x7f3db41b93c0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:58:57.401 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.386+0000 7f3dbc590700 1 --2- 172.21.15.38:0/133867986 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3db4107b50 0x7f3db410ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:58:57.401 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.386+0000 7f3dbc590700 1 -- 172.21.15.38:0/133867986 >> 172.21.15.38:0/133867986 conn(0x7f3db406b450 msgr2=0x7f3db40712a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:58:57.401 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.386+0000 7f3dbc590700 1 -- 172.21.15.38:0/133867986 shutdown_connections 2023-03-09T14:58:57.401 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:58:57.386+0000 7f3dbc590700 1 -- 172.21.15.38:0/133867986 wait complete. 2023-03-09T14:58:57.402 INFO:tasks.cephfs.filesystem:scrub status for tag:a14659f4-d7ea-4c5a-b15b-6138af3f909f - {'path': '/', 'tag': 'a14659f4-d7ea-4c5a-b15b-6138af3f909f', 'options': 'recursive,force'} 2023-03-09T14:58:58.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:58:58 smithi122 ceph-mon[93869]: pgmap v541: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 9.4 MiB/s wr, 1.20k op/s 2023-03-09T14:58:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:58:58 smithi181 ceph-mon[94189]: pgmap v541: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 9.4 MiB/s wr, 1.20k op/s 2023-03-09T14:58:58.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:58:58 smithi038 ceph-mon[92422]: pgmap v541: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 9.4 MiB/s wr, 1.20k op/s 2023-03-09T14:58:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 510.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T14:59:00.485 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:59:00 smithi122 ceph-mon[93869]: pgmap v542: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 6.2 MiB/s wr, 841 op/s 2023-03-09T14:59:00.487 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:59:00 smithi181 ceph-mon[94189]: pgmap v542: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 6.2 MiB/s wr, 841 op/s 2023-03-09T14:59:00.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:59:00 smithi038 ceph-mon[92422]: pgmap v542: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 6.2 MiB/s wr, 841 op/s 2023-03-09T14:59:02.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:59:02 smithi122 ceph-mon[93869]: pgmap v543: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 904 KiB/s rd, 2.6 MiB/s wr, 335 op/s 2023-03-09T14:59:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:59:02 smithi181 ceph-mon[94189]: pgmap v543: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 904 KiB/s rd, 2.6 MiB/s wr, 335 op/s 2023-03-09T14:59:02.505 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:59:02 smithi038 ceph-mon[92422]: pgmap v543: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 904 KiB/s rd, 2.6 MiB/s wr, 335 op/s 2023-03-09T14:59:03.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 515.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T14:59:04.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:59:03 smithi122 ceph-mon[93869]: pgmap v544: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 531 KiB/s rd, 3.0 MiB/s wr, 337 op/s 2023-03-09T14:59:04.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:59:03 smithi181 ceph-mon[94189]: pgmap v544: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 531 KiB/s rd, 3.0 MiB/s wr, 337 op/s 2023-03-09T14:59:04.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:59:03 smithi038 ceph-mon[92422]: pgmap v544: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 531 KiB/s rd, 3.0 MiB/s wr, 337 op/s 2023-03-09T14:59:06.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:59:06 smithi122 ceph-mon[93869]: pgmap v545: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 1.7 MiB/s wr, 223 op/s 2023-03-09T14:59:06.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:59:06 smithi181 ceph-mon[94189]: pgmap v545: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 1.7 MiB/s wr, 223 op/s 2023-03-09T14:59:06.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:59:06 smithi038 ceph-mon[92422]: pgmap v545: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 1.7 MiB/s wr, 223 op/s 2023-03-09T14:59:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:59:08 smithi122 ceph-mon[93869]: pgmap v546: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 1.7 MiB/s wr, 228 op/s 2023-03-09T14:59:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:59:08 smithi181 ceph-mon[94189]: pgmap v546: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 1.7 MiB/s wr, 228 op/s 2023-03-09T14:59:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:59:08 smithi038 ceph-mon[92422]: pgmap v546: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 1.7 MiB/s wr, 228 op/s 2023-03-09T14:59:08.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 520.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T14:59:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:59:10 smithi122 ceph-mon[93869]: pgmap v547: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 1.7 MiB/s wr, 227 op/s 2023-03-09T14:59:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:59:10 smithi122 ceph-mon[93869]: osdmap e248: 12 total, 12 up, 12 in 2023-03-09T14:59:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:59:10 smithi181 ceph-mon[94189]: pgmap v547: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 1.7 MiB/s wr, 227 op/s 2023-03-09T14:59:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:59:10 smithi181 ceph-mon[94189]: osdmap e248: 12 total, 12 up, 12 in 2023-03-09T14:59:10.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:59:10 smithi038 ceph-mon[92422]: pgmap v547: 129 pgs: 129 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 1.7 MiB/s wr, 227 op/s 2023-03-09T14:59:10.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:59:10 smithi038 ceph-mon[92422]: osdmap e248: 12 total, 12 up, 12 in 2023-03-09T14:59:12.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:59:12 smithi122 ceph-mon[93869]: pgmap v549: 129 pgs: 5 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 110 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 1.9 MiB/s wr, 252 op/s 2023-03-09T14:59:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:59:12 smithi181 ceph-mon[94189]: pgmap v549: 129 pgs: 5 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 110 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 1.9 MiB/s wr, 252 op/s 2023-03-09T14:59:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:59:12 smithi038 ceph-mon[92422]: pgmap v549: 129 pgs: 5 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 110 active+clean; 13 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 1.9 MiB/s wr, 252 op/s 2023-03-09T14:59:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 525.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T14:59:14.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:59:13 smithi122 ceph-mon[93869]: pgmap v550: 129 pgs: 13 active+clean+snaptrim, 43 active+clean+snaptrim_wait, 73 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 1.2 MiB/s wr, 167 op/s 2023-03-09T14:59:14.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:59:13 smithi181 ceph-mon[94189]: pgmap v550: 129 pgs: 13 active+clean+snaptrim, 43 active+clean+snaptrim_wait, 73 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 1.2 MiB/s wr, 167 op/s 2023-03-09T14:59:14.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:59:13 smithi038 ceph-mon[92422]: pgmap v550: 129 pgs: 13 active+clean+snaptrim, 43 active+clean+snaptrim_wait, 73 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 1.2 MiB/s wr, 167 op/s 2023-03-09T14:59:16.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:59:16 smithi122 ceph-mon[93869]: pgmap v551: 129 pgs: 11 active+clean+snaptrim, 37 active+clean+snaptrim_wait, 81 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 906 KiB/s wr, 117 op/s 2023-03-09T14:59:16.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:59:16 smithi181 ceph-mon[94189]: pgmap v551: 129 pgs: 11 active+clean+snaptrim, 37 active+clean+snaptrim_wait, 81 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 906 KiB/s wr, 117 op/s 2023-03-09T14:59:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:59:16 smithi038 ceph-mon[92422]: pgmap v551: 129 pgs: 11 active+clean+snaptrim, 37 active+clean+snaptrim_wait, 81 active+clean; 13 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 906 KiB/s wr, 117 op/s 2023-03-09T14:59:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:59:18 smithi122 ceph-mon[93869]: pgmap v552: 129 pgs: 4 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 111 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 607 KiB/s wr, 82 op/s 2023-03-09T14:59:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:59:18 smithi122 ceph-mon[93869]: osdmap e249: 12 total, 12 up, 12 in 2023-03-09T14:59:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:59:18 smithi181 ceph-mon[94189]: pgmap v552: 129 pgs: 4 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 111 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 607 KiB/s wr, 82 op/s 2023-03-09T14:59:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:59:18 smithi181 ceph-mon[94189]: osdmap e249: 12 total, 12 up, 12 in 2023-03-09T14:59:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:59:18 smithi038 ceph-mon[92422]: pgmap v552: 129 pgs: 4 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 111 active+clean; 13 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 607 KiB/s wr, 82 op/s 2023-03-09T14:59:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:59:18 smithi038 ceph-mon[92422]: osdmap e249: 12 total, 12 up, 12 in 2023-03-09T14:59:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 530.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T14:59:20.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:59:20 smithi122 ceph-mon[93869]: pgmap v554: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 349 KiB/s wr, 50 op/s 2023-03-09T14:59:20.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:59:20 smithi181 ceph-mon[94189]: pgmap v554: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 349 KiB/s wr, 50 op/s 2023-03-09T14:59:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:59:20 smithi038 ceph-mon[92422]: pgmap v554: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 349 KiB/s wr, 50 op/s 2023-03-09T14:59:22.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:59:22 smithi122 ceph-mon[93869]: pgmap v555: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 264 KiB/s wr, 40 op/s 2023-03-09T14:59:22.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:59:22 smithi181 ceph-mon[94189]: pgmap v555: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 264 KiB/s wr, 40 op/s 2023-03-09T14:59:22.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:59:22 smithi038 ceph-mon[92422]: pgmap v555: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 264 KiB/s wr, 40 op/s 2023-03-09T14:59:23.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 535.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T14:59:24.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:59:23 smithi122 ceph-mon[93869]: osdmap e250: 12 total, 12 up, 12 in 2023-03-09T14:59:24.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:59:23 smithi122 ceph-mon[93869]: pgmap v557: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 84 KiB/s wr, 17 op/s 2023-03-09T14:59:24.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:59:23 smithi181 ceph-mon[94189]: osdmap e250: 12 total, 12 up, 12 in 2023-03-09T14:59:24.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:59:23 smithi181 ceph-mon[94189]: pgmap v557: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 84 KiB/s wr, 17 op/s 2023-03-09T14:59:24.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:59:23 smithi038 ceph-mon[92422]: osdmap e250: 12 total, 12 up, 12 in 2023-03-09T14:59:24.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:59:23 smithi038 ceph-mon[92422]: pgmap v557: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 84 KiB/s wr, 17 op/s 2023-03-09T14:59:26.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:59:26 smithi122 ceph-mon[93869]: pgmap v558: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 83 KiB/s wr, 14 op/s 2023-03-09T14:59:26.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:59:26 smithi181 ceph-mon[94189]: pgmap v558: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 83 KiB/s wr, 14 op/s 2023-03-09T14:59:26.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:59:26 smithi038 ceph-mon[92422]: pgmap v558: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 83 KiB/s wr, 14 op/s 2023-03-09T14:59:27.403 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-03-09T14:59:27.569 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.566+0000 7f1d82a5d700 1 -- 172.21.15.38:0/1429732378 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1d7c0686d0 msgr2=0x7f1d7c104a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:59:27.570 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.566+0000 7f1d82a5d700 1 --2- 172.21.15.38:0/1429732378 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1d7c0686d0 0x7f1d7c104a60 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f1d640099c0 tx=0x7f1d64009cd0 comp rx=0 tx=0).stop 2023-03-09T14:59:27.570 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.566+0000 7f1d82a5d700 1 -- 172.21.15.38:0/1429732378 shutdown_connections 2023-03-09T14:59:27.570 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.566+0000 7f1d82a5d700 1 --2- 172.21.15.38:0/1429732378 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d7c105140 0x7f1d7c10cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:59:27.570 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.566+0000 7f1d82a5d700 1 --2- 172.21.15.38:0/1429732378 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d7c067d30 0x7f1d7c068100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:59:27.570 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.566+0000 7f1d82a5d700 1 --2- 172.21.15.38:0/1429732378 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1d7c0686d0 0x7f1d7c104a60 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:59:27.570 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.566+0000 7f1d82a5d700 1 -- 172.21.15.38:0/1429732378 >> 172.21.15.38:0/1429732378 conn(0x7f1d7c075c60 msgr2=0x7f1d7c076060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:59:27.571 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.566+0000 7f1d82a5d700 1 -- 172.21.15.38:0/1429732378 shutdown_connections 2023-03-09T14:59:27.571 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.566+0000 7f1d82a5d700 1 -- 172.21.15.38:0/1429732378 wait complete. 2023-03-09T14:59:27.571 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.567+0000 7f1d82a5d700 1 Processor -- start 2023-03-09T14:59:27.571 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.567+0000 7f1d82a5d700 1 -- start start 2023-03-09T14:59:27.572 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.567+0000 7f1d82a5d700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d7c067d30 0x7f1d7c196040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:59:27.573 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.567+0000 7f1d82a5d700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d7c0686d0 0x7f1d7c196580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:59:27.573 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.567+0000 7f1d82a5d700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1d7c105140 0x7f1d7c19a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:59:27.573 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.567+0000 7f1d82a5d700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7c19b020 con 0x7f1d7c105140 2023-03-09T14:59:27.573 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.567+0000 7f1d82a5d700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7c19b190 con 0x7f1d7c0686d0 2023-03-09T14:59:27.573 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.567+0000 7f1d82a5d700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d7c19b350 con 0x7f1d7c067d30 2023-03-09T14:59:27.573 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.567+0000 7f1d80ffa700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1d7c105140 0x7f1d7c19a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:59:27.574 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.567+0000 7f1d7b7fe700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d7c0686d0 0x7f1d7c196580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:59:27.574 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.567+0000 7f1d80ffa700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1d7c105140 0x7f1d7c19a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42252/0 (socket says 172.21.15.38:42252) 2023-03-09T14:59:27.574 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.567+0000 7f1d80ffa700 1 -- 172.21.15.38:0/1235464132 learned_addr learned my addr 172.21.15.38:0/1235464132 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:59:27.574 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.567+0000 7f1d7bfff700 1 --2- 172.21.15.38:0/1235464132 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d7c067d30 0x7f1d7c196040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:59:27.574 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.568+0000 7f1d7b7fe700 1 -- 172.21.15.38:0/1235464132 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d7c067d30 msgr2=0x7f1d7c196040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:59:27.574 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.568+0000 7f1d7b7fe700 1 --2- 172.21.15.38:0/1235464132 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d7c067d30 0x7f1d7c196040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:59:27.574 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.568+0000 7f1d7b7fe700 1 -- 172.21.15.38:0/1235464132 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1d7c105140 msgr2=0x7f1d7c19a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:59:27.574 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.568+0000 7f1d7b7fe700 1 --2- 172.21.15.38:0/1235464132 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1d7c105140 0x7f1d7c19a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:59:27.575 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.568+0000 7f1d7b7fe700 1 -- 172.21.15.38:0/1235464132 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d6c009520 con 0x7f1d7c0686d0 2023-03-09T14:59:27.575 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.568+0000 7f1d80ffa700 1 --2- 172.21.15.38:0/1235464132 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1d7c105140 0x7f1d7c19a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:59:27.575 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.568+0000 7f1d7bfff700 1 --2- 172.21.15.38:0/1235464132 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d7c067d30 0x7f1d7c196040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:59:27.575 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.568+0000 7f1d7b7fe700 1 --2- 172.21.15.38:0/1235464132 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d7c0686d0 0x7f1d7c196580 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1d640058e0 tx=0x7f1d6400f690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:59:27.575 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.568+0000 7f1d797fa700 1 -- 172.21.15.38:0/1235464132 <== mon.2 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d6401d070 con 0x7f1d7c0686d0 2023-03-09T14:59:27.575 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.568+0000 7f1d797fa700 1 -- 172.21.15.38:0/1235464132 <== mon.2 v2:172.21.15.122:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1d64005020 con 0x7f1d7c0686d0 2023-03-09T14:59:27.575 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.568+0000 7f1d797fa700 1 -- 172.21.15.38:0/1235464132 <== mon.2 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d64004ac0 con 0x7f1d7c0686d0 2023-03-09T14:59:27.576 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.568+0000 7f1d82a5d700 1 -- 172.21.15.38:0/1235464132 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d640095f0 con 0x7f1d7c0686d0 2023-03-09T14:59:27.576 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.568+0000 7f1d82a5d700 1 -- 172.21.15.38:0/1235464132 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d7c10fec0 con 0x7f1d7c0686d0 2023-03-09T14:59:27.580 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.570+0000 7f1d797fa700 1 -- 172.21.15.38:0/1235464132 <== mon.2 v2:172.21.15.122:3300/0 4 ==== mgrmap(e 25) v1 ==== 133295+0+0 (secure 0 0 0) 0x7f1d6400c040 con 0x7f1d7c0686d0 2023-03-09T14:59:27.580 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.571+0000 7f1d797fa700 1 --2- 172.21.15.38:0/1235464132 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f1d6809f4a0 0x7f1d680a1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:59:27.580 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.571+0000 7f1d7bfff700 1 --2- 172.21.15.38:0/1235464132 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f1d6809f4a0 0x7f1d680a1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:59:27.581 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.571+0000 7f1d797fa700 1 -- 172.21.15.38:0/1235464132 <== mon.2 v2:172.21.15.122:3300/0 5 ==== osd_map(250..250 src has 1..250) v4 ==== 9561+0+0 (secure 0 0 0) 0x7f1d640cad30 con 0x7f1d7c0686d0 2023-03-09T14:59:27.581 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.572+0000 7f1d7bfff700 1 --2- 172.21.15.38:0/1235464132 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f1d6809f4a0 0x7f1d680a1950 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c009e40 tx=0x7f1d6c009260 comp rx=0 tx=0).ready entity=mgr.15558 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:59:27.581 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.572+0000 7f1d8ebe6b80 1 Processor -- start 2023-03-09T14:59:27.581 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.572+0000 7f1d8ebe6b80 1 -- start start 2023-03-09T14:59:27.581 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.572+0000 7f1d8ebe6b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55727827a830 0x55727827ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:59:27.581 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.572+0000 7f1d8ebe6b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55727827b140 0x557278274aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:59:27.581 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.572+0000 7f1d8ebe6b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557278274fe0 0x557278275410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:59:27.582 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.572+0000 7f1d8ebe6b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55727815eda0 con 0x55727827a830 2023-03-09T14:59:27.582 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.572+0000 7f1d8ebe6b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572781cd780 con 0x55727827b140 2023-03-09T14:59:27.582 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.572+0000 7f1d8ebe6b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572781c6cb0 con 0x557278274fe0 2023-03-09T14:59:27.582 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.572+0000 7f1d7bfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55727827a830 0x55727827ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:59:27.582 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.572+0000 7f1d7bfff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55727827a830 0x55727827ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42256/0 (socket says 172.21.15.38:42256) 2023-03-09T14:59:27.582 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.572+0000 7f1d7bfff700 1 -- 172.21.15.38:0/493857742 learned_addr learned my addr 172.21.15.38:0/493857742 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:59:27.582 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.572+0000 7f1d7b7fe700 1 --2- 172.21.15.38:0/493857742 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55727827b140 0x557278274aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:59:27.583 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.572+0000 7f1d80ffa700 1 --2- 172.21.15.38:0/493857742 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557278274fe0 0x557278275410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:59:27.583 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.572+0000 7f1d7bfff700 1 -- 172.21.15.38:0/493857742 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557278274fe0 msgr2=0x557278275410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:59:27.583 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.572+0000 7f1d7bfff700 1 --2- 172.21.15.38:0/493857742 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557278274fe0 0x557278275410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:59:27.583 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.573+0000 7f1d7bfff700 1 -- 172.21.15.38:0/493857742 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55727827b140 msgr2=0x557278274aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:59:27.583 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.573+0000 7f1d7bfff700 1 --2- 172.21.15.38:0/493857742 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55727827b140 0x557278274aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:59:27.583 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.573+0000 7f1d7bfff700 1 -- 172.21.15.38:0/493857742 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55727818d190 con 0x55727827a830 2023-03-09T14:59:27.583 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.573+0000 7f1d80ffa700 1 --2- 172.21.15.38:0/493857742 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557278274fe0 0x557278275410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:59:27.583 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.573+0000 7f1d7bfff700 1 --2- 172.21.15.38:0/493857742 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55727827a830 0x55727827ac00 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c0094f0 tx=0x7f1d6c014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30869121e32b0ee7 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:59:27.584 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.573+0000 7f1d727fc700 1 -- 172.21.15.38:0/493857742 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d6c004220 con 0x55727827a830 2023-03-09T14:59:27.584 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.573+0000 7f1d727fc700 1 -- 172.21.15.38:0/493857742 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1d6c016070 con 0x55727827a830 2023-03-09T14:59:27.584 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.573+0000 7f1d727fc700 1 -- 172.21.15.38:0/493857742 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d6c01f710 con 0x55727827a830 2023-03-09T14:59:27.584 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.573+0000 7f1d8ebe6b80 1 -- 172.21.15.38:0/493857742 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55727827a830 msgr2=0x55727827ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:59:27.584 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.573+0000 7f1d8ebe6b80 1 --2- 172.21.15.38:0/493857742 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55727827a830 0x55727827ac00 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c0094f0 tx=0x7f1d6c014000 comp rx=0 tx=0).stop 2023-03-09T14:59:27.584 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.574+0000 7f1d8ebe6b80 1 -- 172.21.15.38:0/493857742 shutdown_connections 2023-03-09T14:59:27.584 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.574+0000 7f1d8ebe6b80 1 --2- 172.21.15.38:0/493857742 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55727827a830 0x55727827ac00 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:59:27.584 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.574+0000 7f1d8ebe6b80 1 --2- 172.21.15.38:0/493857742 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557278274fe0 0x557278275410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:59:27.585 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.574+0000 7f1d8ebe6b80 1 --2- 172.21.15.38:0/493857742 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55727827b140 0x557278274aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:59:27.585 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.574+0000 7f1d8ebe6b80 1 -- 172.21.15.38:0/493857742 >> 172.21.15.38:0/493857742 conn(0x55727826fa20 msgr2=0x55727826fdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:59:27.585 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.574+0000 7f1d8ebe6b80 1 -- 172.21.15.38:0/493857742 shutdown_connections 2023-03-09T14:59:27.585 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.574+0000 7f1d8ebe6b80 1 -- 172.21.15.38:0/493857742 wait complete. 2023-03-09T14:59:27.585 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.574+0000 7f1d8ebe6b80 1 -- start start 2023-03-09T14:59:27.585 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.574+0000 7f1d8ebe6b80 1 Processor -- start 2023-03-09T14:59:27.585 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.574+0000 7f1d8ebe6b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557278274fe0 0x557278281100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:59:27.586 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.574+0000 7f1d8ebe6b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55727827a830 0x55727827e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:59:27.586 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.574+0000 7f1d8ebe6b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55727827b140 0x55727827ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:59:27.586 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.574+0000 7f1d8ebe6b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55727804a920 con 0x55727827a830 2023-03-09T14:59:27.586 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.574+0000 7f1d8ebe6b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55727827f1e0 con 0x557278274fe0 2023-03-09T14:59:27.586 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.574+0000 7f1d8ebe6b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55727828ac40 con 0x55727827b140 2023-03-09T14:59:27.586 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.574+0000 7f1d8ebe6b80 10 client.0 _ll_register_callbacks cb 0x55727816c520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:59:27.586 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.574+0000 7f1d7b7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55727827a830 0x55727827e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:59:27.586 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.574+0000 7f1d7bfff700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557278274fe0 0x557278281100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:59:27.587 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.574+0000 7f1d7bfff700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557278274fe0 0x557278281100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.38:51452/0 (socket says 172.21.15.38:51452) 2023-03-09T14:59:27.587 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.574+0000 7f1d7bfff700 1 -- 172.21.15.38:0/3760696882 learned_addr learned my addr 172.21.15.38:0/3760696882 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:59:27.587 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.575+0000 7f1d80ffa700 1 --2- 172.21.15.38:0/3760696882 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55727827b140 0x55727827ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:59:27.587 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.575+0000 7f1d7bfff700 1 -- 172.21.15.38:0/3760696882 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55727827b140 msgr2=0x55727827ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:59:27.587 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.575+0000 7f1d7bfff700 1 --2- 172.21.15.38:0/3760696882 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55727827b140 0x55727827ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:59:27.587 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.575+0000 7f1d7bfff700 1 -- 172.21.15.38:0/3760696882 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55727827a830 msgr2=0x55727827e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:59:27.587 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.575+0000 7f1d7bfff700 1 --2- 172.21.15.38:0/3760696882 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55727827a830 0x55727827e3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:59:27.588 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.575+0000 7f1d7bfff700 1 -- 172.21.15.38:0/3760696882 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d6c011da0 con 0x557278274fe0 2023-03-09T14:59:27.588 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.575+0000 7f1d80ffa700 1 --2- 172.21.15.38:0/3760696882 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55727827b140 0x55727827ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:59:27.588 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.575+0000 7f1d7bfff700 1 --2- 172.21.15.38:0/3760696882 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557278274fe0 0x557278281100 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c0094c0 tx=0x7f1d6c0042e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17de79c63ea02104 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:59:27.588 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.575+0000 7f1d5b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2023-03-09T14:59:27.588 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.575+0000 7f1d5b7fe700 1 -- 172.21.15.38:0/3760696882 <== mon.2 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d6c02e070 con 0x557278274fe0 2023-03-09T14:59:27.588 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.575+0000 7f1d8ebe6b80 1 -- 172.21.15.38:0/3760696882 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557278194190 con 0x557278274fe0 2023-03-09T14:59:27.588 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.575+0000 7f1d5b7fe700 1 -- 172.21.15.38:0/3760696882 <== mon.2 v2:172.21.15.122:3300/0 2 ==== osd_map(250..250 src has 1..250) v4 ==== 9561+0+0 (secure 0 0 0) 0x7f1d6c018070 con 0x557278274fe0 2023-03-09T14:59:27.588 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.575+0000 7f1d5b7fe700 1 -- 172.21.15.38:0/3760696882 <== mon.2 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1d6c005260 con 0x557278274fe0 2023-03-09T14:59:27.589 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.575+0000 7f1d8ebe6b80 10 client.25982 fetch_fsmap learned FSMap version 43 2023-03-09T14:59:27.589 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.575+0000 7f1d8ebe6b80 1 -- 172.21.15.38:0/3760696882 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) v3 -- 0x557277fddc60 con 0x557278274fe0 2023-03-09T14:59:27.589 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.576+0000 7f1d5b7fe700 1 -- 172.21.15.38:0/3760696882 <== mon.2 v2:172.21.15.122:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x557278194190 con 0x557278274fe0 2023-03-09T14:59:27.589 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.576+0000 7f1d5b7fe700 1 -- 172.21.15.38:0/3760696882 <== mon.2 v2:172.21.15.122:3300/0 5 ==== fsmap(e 43) v1 ==== 7482+0+0 (secure 0 0 0) 0x7f1d6c025070 con 0x557278274fe0 2023-03-09T14:59:27.589 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.576+0000 7f1d8ebe6b80 10 client.25982 fetch_fsmap finished waiting for FSMap version 43 2023-03-09T14:59:27.589 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.576+0000 7f1d8ebe6b80 10 client.25982 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T14:59:27.589 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.576+0000 7f1d8ebe6b80 20 client.25982 populate_metadata read hostname 'smithi038' 2023-03-09T14:59:27.590 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.576+0000 7f1d8ebe6b80 1 --2- 172.21.15.38:0/3760696882 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55727828ec00 0x557278290fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:59:27.590 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.576+0000 7f1d8ebe6b80 4 client.25982 mds_command: new command op to 24530 tid=0[{"prefix": "get_command_descriptions"}] 2023-03-09T14:59:27.590 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.576+0000 7f1d8ebe6b80 1 -- 172.21.15.38:0/3760696882 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55727804af60 con 0x55727828ec00 2023-03-09T14:59:27.590 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.576+0000 7f1d7b7fe700 1 --2- 172.21.15.38:0/3760696882 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55727828ec00 0x557278290fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:59:27.590 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.576+0000 7f1d5b7fe700 10 client.25982 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:59:27.590 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.576+0000 7f1d7b7fe700 1 --2- 172.21.15.38:0/3760696882 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55727828ec00 0x557278290fe0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=965321257e002b14 server_cookie=9aa7f768a82c752 in_seq=0 out_seq=0 2023-03-09T14:59:27.590 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.577+0000 7f1d5b7fe700 1 -- 172.21.15.38:0/3760696882 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18278 (crc 0 0 0) 0x55727804af60 con 0x55727828ec00 2023-03-09T14:59:27.590 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.577+0000 7f1d5b7fe700 10 client.25982 handle_command_reply: tid=0 2023-03-09T14:59:27.591 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.577+0000 7f1d8ebe6b80 1 client.25982 shutdown 2023-03-09T14:59:27.591 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.577+0000 7f1d8ebe6b80 1 -- 172.21.15.38:0/3760696882 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557278274fe0 msgr2=0x557278281100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:59:27.591 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.577+0000 7f1d8ebe6b80 1 --2- 172.21.15.38:0/3760696882 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557278274fe0 0x557278281100 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c0094c0 tx=0x7f1d6c0042e0 comp rx=0 tx=0).stop 2023-03-09T14:59:27.591 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.577+0000 7f1d8ebe6b80 1 -- 172.21.15.38:0/3760696882 shutdown_connections 2023-03-09T14:59:27.591 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.577+0000 7f1d8ebe6b80 1 --2- 172.21.15.38:0/3760696882 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55727828ec00 0x557278290fe0 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:59:27.591 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.577+0000 7f1d8ebe6b80 1 --2- 172.21.15.38:0/3760696882 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55727827b140 0x55727827ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:59:27.591 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.577+0000 7f1d8ebe6b80 1 --2- 172.21.15.38:0/3760696882 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557278274fe0 0x557278281100 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:59:27.592 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.577+0000 7f1d8ebe6b80 1 --2- 172.21.15.38:0/3760696882 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55727827a830 0x55727827e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:59:27.592 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.577+0000 7f1d5b7fe700 0 client.25982 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:59:27.592 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.577+0000 7f1d8ebe6b80 1 -- 172.21.15.38:0/3760696882 >> 172.21.15.38:0/3760696882 conn(0x55727826fa20 msgr2=0x55727826ec20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:59:27.592 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.577+0000 7f1d8ebe6b80 1 -- 172.21.15.38:0/3760696882 shutdown_connections 2023-03-09T14:59:27.592 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.577+0000 7f1d8ebe6b80 1 -- 172.21.15.38:0/3760696882 wait complete. 2023-03-09T14:59:27.592 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.577+0000 7f1d8ebe6b80 20 client.25982 trim_cache size 0 max 16384 2023-03-09T14:59:27.598 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.595+0000 7f1d8ebe6b80 1 Processor -- start 2023-03-09T14:59:27.598 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.595+0000 7f1d8ebe6b80 1 -- start start 2023-03-09T14:59:27.598 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.595+0000 7f1d8ebe6b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557278274fe0 0x55727828b030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:59:27.600 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.595+0000 7f1d8ebe6b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55727827a830 0x557278280ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:59:27.600 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.595+0000 7f1d8ebe6b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55727827b140 0x55727826ec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:59:27.601 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.595+0000 7f1d8ebe6b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55727826bd70 con 0x55727827a830 2023-03-09T14:59:27.601 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.595+0000 7f1d8ebe6b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572782790e0 con 0x55727827b140 2023-03-09T14:59:27.601 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.595+0000 7f1d8ebe6b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x557278274da0 con 0x557278274fe0 2023-03-09T14:59:27.601 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.595+0000 7f1d7b7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55727827a830 0x557278280ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:59:27.601 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.596+0000 7f1d7b7fe700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55727827a830 0x557278280ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42272/0 (socket says 172.21.15.38:42272) 2023-03-09T14:59:27.601 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.596+0000 7f1d7b7fe700 1 -- 172.21.15.38:0/1958998150 learned_addr learned my addr 172.21.15.38:0/1958998150 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:59:27.601 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.596+0000 7f1d80ffa700 1 --2- 172.21.15.38:0/1958998150 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55727827b140 0x55727826ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:59:27.602 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.596+0000 7f1d7bfff700 1 --2- 172.21.15.38:0/1958998150 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557278274fe0 0x55727828b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:59:27.602 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.596+0000 7f1d7b7fe700 1 -- 172.21.15.38:0/1958998150 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557278274fe0 msgr2=0x55727828b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:59:27.602 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.596+0000 7f1d7b7fe700 1 --2- 172.21.15.38:0/1958998150 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557278274fe0 0x55727828b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:59:27.602 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.596+0000 7f1d7b7fe700 1 -- 172.21.15.38:0/1958998150 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55727827b140 msgr2=0x55727826ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:59:27.602 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.596+0000 7f1d7b7fe700 1 --2- 172.21.15.38:0/1958998150 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55727827b140 0x55727826ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:59:27.602 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.596+0000 7f1d7b7fe700 1 -- 172.21.15.38:0/1958998150 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557277ff4de0 con 0x55727827a830 2023-03-09T14:59:27.602 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.596+0000 7f1d7bfff700 1 --2- 172.21.15.38:0/1958998150 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557278274fe0 0x55727828b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-09T14:59:27.603 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.596+0000 7f1d80ffa700 1 --2- 172.21.15.38:0/1958998150 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55727827b140 0x55727826ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:59:27.603 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.596+0000 7f1d7b7fe700 1 --2- 172.21.15.38:0/1958998150 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55727827a830 0x557278280ca0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f1d640985c0 tx=0x7f1d6409b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=685f2e6860449703 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:59:27.603 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.596+0000 7f1d72ffd700 1 -- 172.21.15.38:0/1958998150 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d64096a00 con 0x55727827a830 2023-03-09T14:59:27.603 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.596+0000 7f1d72ffd700 1 -- 172.21.15.38:0/1958998150 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f1d640ae040 con 0x55727827a830 2023-03-09T14:59:27.603 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.596+0000 7f1d72ffd700 1 -- 172.21.15.38:0/1958998150 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d64098b20 con 0x55727827a830 2023-03-09T14:59:27.603 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.597+0000 7f1d8ebe6b80 1 -- 172.21.15.38:0/1958998150 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55727827a830 msgr2=0x557278280ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:59:27.603 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.597+0000 7f1d8ebe6b80 1 --2- 172.21.15.38:0/1958998150 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55727827a830 0x557278280ca0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f1d640985c0 tx=0x7f1d6409b040 comp rx=0 tx=0).stop 2023-03-09T14:59:27.603 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.597+0000 7f1d8ebe6b80 1 -- 172.21.15.38:0/1958998150 shutdown_connections 2023-03-09T14:59:27.604 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.597+0000 7f1d8ebe6b80 1 --2- 172.21.15.38:0/1958998150 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55727827b140 0x55727826ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:59:27.604 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.597+0000 7f1d8ebe6b80 1 --2- 172.21.15.38:0/1958998150 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x557278274fe0 0x55727828b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:59:27.604 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.597+0000 7f1d8ebe6b80 1 --2- 172.21.15.38:0/1958998150 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55727827a830 0x557278280ca0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:59:27.604 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.597+0000 7f1d8ebe6b80 1 -- 172.21.15.38:0/1958998150 >> 172.21.15.38:0/1958998150 conn(0x55727826fa20 msgr2=0x55727826df70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:59:27.604 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.597+0000 7f1d8ebe6b80 1 -- 172.21.15.38:0/1958998150 shutdown_connections 2023-03-09T14:59:27.604 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.597+0000 7f1d8ebe6b80 1 -- 172.21.15.38:0/1958998150 wait complete. 2023-03-09T14:59:27.604 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.597+0000 7f1d8ebe6b80 1 -- start start 2023-03-09T14:59:27.605 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.597+0000 7f1d8ebe6b80 1 Processor -- start 2023-03-09T14:59:27.605 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.597+0000 7f1d8ebe6b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557278274fe0 0x557278277a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:59:27.605 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.597+0000 7f1d8ebe6b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55727827a830 0x557278276290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:59:27.605 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.597+0000 7f1d8ebe6b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55727827b140 0x5572782767d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:59:27.605 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.597+0000 7f1d8ebe6b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x557278278050 con 0x55727827b140 2023-03-09T14:59:27.605 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.597+0000 7f1d8ebe6b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x557278276d10 con 0x557278274fe0 2023-03-09T14:59:27.605 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.598+0000 7f1d8ebe6b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x557278276ed0 con 0x55727827a830 2023-03-09T14:59:27.606 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.598+0000 7f1d8ebe6b80 10 client.0 _ll_register_callbacks cb 0x55727816c520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:59:27.606 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.598+0000 7f1d80ffa700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55727827b140 0x5572782767d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:59:27.606 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.598+0000 7f1d7bfff700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557278274fe0 0x557278277a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:59:27.606 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.598+0000 7f1d7bfff700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557278274fe0 0x557278277a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.38:51466/0 (socket says 172.21.15.38:51466) 2023-03-09T14:59:27.606 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.598+0000 7f1d7bfff700 1 -- 172.21.15.38:0/1667718845 learned_addr learned my addr 172.21.15.38:0/1667718845 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:59:27.606 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.598+0000 7f1d80ffa700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55727827b140 0x5572782767d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42280/0 (socket says 172.21.15.38:42280) 2023-03-09T14:59:27.606 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.598+0000 7f1d7b7fe700 1 --2- 172.21.15.38:0/1667718845 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55727827a830 0x557278276290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:59:27.608 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.598+0000 7f1d7bfff700 1 -- 172.21.15.38:0/1667718845 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55727827a830 msgr2=0x557278276290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:59:27.609 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.598+0000 7f1d7bfff700 1 --2- 172.21.15.38:0/1667718845 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55727827a830 0x557278276290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:59:27.609 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.598+0000 7f1d7bfff700 1 -- 172.21.15.38:0/1667718845 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55727827b140 msgr2=0x5572782767d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:59:27.609 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.598+0000 7f1d7bfff700 1 --2- 172.21.15.38:0/1667718845 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55727827b140 0x5572782767d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:59:27.609 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.598+0000 7f1d7bfff700 1 -- 172.21.15.38:0/1667718845 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d74007500 con 0x557278274fe0 2023-03-09T14:59:27.609 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.598+0000 7f1d80ffa700 1 --2- 172.21.15.38:0/1667718845 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55727827b140 0x5572782767d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:59:27.609 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.598+0000 7f1d7b7fe700 1 --2- 172.21.15.38:0/1667718845 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55727827a830 0x557278276290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T14:59:27.609 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.598+0000 7f1d7bfff700 1 --2- 172.21.15.38:0/1667718845 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557278274fe0 0x557278277a30 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c003a10 tx=0x7f1d6c005960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd668b4fb8f10498 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:59:27.610 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.598+0000 7f1d5b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2023-03-09T14:59:27.610 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.598+0000 7f1d5b7fe700 1 -- 172.21.15.38:0/1667718845 <== mon.2 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d6c02e030 con 0x557278274fe0 2023-03-09T14:59:27.610 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.598+0000 7f1d8ebe6b80 1 -- 172.21.15.38:0/1667718845 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5572781841b0 con 0x557278274fe0 2023-03-09T14:59:27.610 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.598+0000 7f1d5b7fe700 1 -- 172.21.15.38:0/1667718845 <== mon.2 v2:172.21.15.122:3300/0 2 ==== osd_map(250..250 src has 1..250) v4 ==== 9561+0+0 (secure 0 0 0) 0x7f1d6c02f070 con 0x557278274fe0 2023-03-09T14:59:27.610 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.598+0000 7f1d5b7fe700 1 -- 172.21.15.38:0/1667718845 <== mon.2 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1d6c0056c0 con 0x557278274fe0 2023-03-09T14:59:27.610 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.599+0000 7f1d8ebe6b80 10 client.25988 fetch_fsmap learned FSMap version 43 2023-03-09T14:59:27.610 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.599+0000 7f1d8ebe6b80 1 -- 172.21.15.38:0/1667718845 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) v3 -- 0x7f1d7c10fec0 con 0x557278274fe0 2023-03-09T14:59:27.610 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.599+0000 7f1d5b7fe700 1 -- 172.21.15.38:0/1667718845 <== mon.2 v2:172.21.15.122:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5572781841b0 con 0x557278274fe0 2023-03-09T14:59:27.611 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.599+0000 7f1d5b7fe700 1 -- 172.21.15.38:0/1667718845 <== mon.2 v2:172.21.15.122:3300/0 5 ==== fsmap(e 43) v1 ==== 7482+0+0 (secure 0 0 0) 0x7f1d6c012830 con 0x557278274fe0 2023-03-09T14:59:27.611 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.599+0000 7f1d8ebe6b80 10 client.25988 fetch_fsmap finished waiting for FSMap version 43 2023-03-09T14:59:27.611 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.599+0000 7f1d8ebe6b80 10 client.25988 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T14:59:27.611 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.599+0000 7f1d8ebe6b80 20 client.25988 populate_metadata read hostname 'smithi038' 2023-03-09T14:59:27.611 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.599+0000 7f1d8ebe6b80 1 --2- 172.21.15.38:0/1667718845 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5572782e37f0 0x5572782e5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:59:27.611 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.599+0000 7f1d8ebe6b80 4 client.25988 mds_command: new command op to 24530 tid=0[{"prefix": "scrub status"}] 2023-03-09T14:59:27.611 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.599+0000 7f1d8ebe6b80 1 -- 172.21.15.38:0/1667718845 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55727816e1c0 con 0x5572782e37f0 2023-03-09T14:59:27.611 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.599+0000 7f1d7b7fe700 1 --2- 172.21.15.38:0/1667718845 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5572782e37f0 0x5572782e5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:59:27.612 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.599+0000 7f1d5b7fe700 10 client.25988 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:59:27.612 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.599+0000 7f1d7b7fe700 1 --2- 172.21.15.38:0/1667718845 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5572782e37f0 0x5572782e5bd0 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8938648485bf3419 server_cookie=d0a0203ebeccf5b3 in_seq=0 out_seq=0 2023-03-09T14:59:27.612 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.600+0000 7f1d5b7fe700 1 -- 172.21.15.38:0/1667718845 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+265 (crc 0 0 0) 0x55727816e1c0 con 0x5572782e37f0 2023-03-09T14:59:27.612 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.600+0000 7f1d5b7fe700 10 client.25988 handle_command_reply: tid=0 2023-03-09T14:59:27.612 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.600+0000 7f1d8ebe6b80 1 client.25988 shutdown 2023-03-09T14:59:27.612 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.600+0000 7f1d8ebe6b80 1 -- 172.21.15.38:0/1667718845 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557278274fe0 msgr2=0x557278277a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:59:27.612 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.600+0000 7f1d8ebe6b80 1 --2- 172.21.15.38:0/1667718845 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557278274fe0 0x557278277a30 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c003a10 tx=0x7f1d6c005960 comp rx=0 tx=0).stop 2023-03-09T14:59:27.613 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.600+0000 7f1d8ebe6b80 1 -- 172.21.15.38:0/1667718845 shutdown_connections 2023-03-09T14:59:27.613 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.600+0000 7f1d8ebe6b80 1 --2- 172.21.15.38:0/1667718845 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5572782e37f0 0x5572782e5bd0 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:59:27.613 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.600+0000 7f1d8ebe6b80 1 --2- 172.21.15.38:0/1667718845 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55727827b140 0x5572782767d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:59:27.613 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.600+0000 7f1d8ebe6b80 1 --2- 172.21.15.38:0/1667718845 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x557278274fe0 0x557278277a30 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:59:27.613 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.600+0000 7f1d8ebe6b80 1 --2- 172.21.15.38:0/1667718845 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55727827a830 0x557278276290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:59:27.613 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.600+0000 7f1d5b7fe700 0 client.25988 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:59:27.613 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.600+0000 7f1d8ebe6b80 1 -- 172.21.15.38:0/1667718845 >> 172.21.15.38:0/1667718845 conn(0x55727826fa20 msgr2=0x5572782753b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:59:27.613 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.600+0000 7f1d8ebe6b80 1 -- 172.21.15.38:0/1667718845 shutdown_connections 2023-03-09T14:59:27.614 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.600+0000 7f1d8ebe6b80 1 -- 172.21.15.38:0/1667718845 wait complete. 2023-03-09T14:59:27.614 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.600+0000 7f1d8ebe6b80 20 client.25988 trim_cache size 0 max 16384 2023-03-09T14:59:27.614 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2023-03-09T14:59:27.614 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "scrub active (1 inodes in the stack)", 2023-03-09T14:59:27.614 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": { 2023-03-09T14:59:27.615 INFO:teuthology.orchestra.run.smithi038.stdout: "a14659f4-d7ea-4c5a-b15b-6138af3f909f": { 2023-03-09T14:59:27.615 INFO:teuthology.orchestra.run.smithi038.stdout: "path": "/", 2023-03-09T14:59:27.615 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "a14659f4-d7ea-4c5a-b15b-6138af3f909f", 2023-03-09T14:59:27.615 INFO:teuthology.orchestra.run.smithi038.stdout: "options": "recursive,force" 2023-03-09T14:59:27.615 INFO:teuthology.orchestra.run.smithi038.stdout: } 2023-03-09T14:59:27.615 INFO:teuthology.orchestra.run.smithi038.stdout: } 2023-03-09T14:59:27.615 INFO:teuthology.orchestra.run.smithi038.stdout:} 2023-03-09T14:59:27.616 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.600+0000 7f1d82a5d700 1 -- 172.21.15.38:0/1235464132 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f1d6809f4a0 msgr2=0x7f1d680a1950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:59:27.616 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.600+0000 7f1d82a5d700 1 --2- 172.21.15.38:0/1235464132 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f1d6809f4a0 0x7f1d680a1950 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1d6c009e40 tx=0x7f1d6c009260 comp rx=0 tx=0).stop 2023-03-09T14:59:27.617 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.600+0000 7f1d82a5d700 1 -- 172.21.15.38:0/1235464132 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d7c0686d0 msgr2=0x7f1d7c196580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:59:27.617 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.600+0000 7f1d82a5d700 1 --2- 172.21.15.38:0/1235464132 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d7c0686d0 0x7f1d7c196580 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1d640058e0 tx=0x7f1d6400f690 comp rx=0 tx=0).stop 2023-03-09T14:59:27.617 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.600+0000 7f1d82a5d700 1 -- 172.21.15.38:0/1235464132 shutdown_connections 2023-03-09T14:59:27.617 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.600+0000 7f1d82a5d700 1 --2- 172.21.15.38:0/1235464132 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f1d6809f4a0 0x7f1d680a1950 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:59:27.617 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.600+0000 7f1d82a5d700 1 --2- 172.21.15.38:0/1235464132 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f1d7c105140 0x7f1d7c19a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:59:27.617 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.600+0000 7f1d82a5d700 1 --2- 172.21.15.38:0/1235464132 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f1d7c067d30 0x7f1d7c196040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:59:27.617 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.600+0000 7f1d82a5d700 1 --2- 172.21.15.38:0/1235464132 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f1d7c0686d0 0x7f1d7c196580 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:59:27.618 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.601+0000 7f1d82a5d700 1 -- 172.21.15.38:0/1235464132 >> 172.21.15.38:0/1235464132 conn(0x7f1d7c075c60 msgr2=0x7f1d7c0fc940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:59:27.618 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.601+0000 7f1d82a5d700 1 -- 172.21.15.38:0/1235464132 shutdown_connections 2023-03-09T14:59:27.618 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:27.601+0000 7f1d82a5d700 1 -- 172.21.15.38:0/1235464132 wait complete. 2023-03-09T14:59:27.618 INFO:tasks.cephfs.filesystem:scrub status for tag:a14659f4-d7ea-4c5a-b15b-6138af3f909f - {'path': '/', 'tag': 'a14659f4-d7ea-4c5a-b15b-6138af3f909f', 'options': 'recursive,force'} 2023-03-09T14:59:28.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:59:28 smithi122 ceph-mon[93869]: pgmap v559: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 95 KiB/s wr, 17 op/s 2023-03-09T14:59:28.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:59:28 smithi181 ceph-mon[94189]: pgmap v559: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 95 KiB/s wr, 17 op/s 2023-03-09T14:59:28.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:59:28 smithi038 ceph-mon[92422]: pgmap v559: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 95 KiB/s wr, 17 op/s 2023-03-09T14:59:28.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 540.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T14:59:30.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:59:30 smithi122 ceph-mon[93869]: pgmap v560: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 129 KiB/s wr, 20 op/s 2023-03-09T14:59:30.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:59:30 smithi181 ceph-mon[94189]: pgmap v560: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 129 KiB/s wr, 20 op/s 2023-03-09T14:59:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:59:30 smithi038 ceph-mon[92422]: pgmap v560: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 129 KiB/s wr, 20 op/s 2023-03-09T14:59:32.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:59:32 smithi122 ceph-mon[93869]: pgmap v561: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 137 KiB/s wr, 21 op/s 2023-03-09T14:59:32.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:59:32 smithi181 ceph-mon[94189]: pgmap v561: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 137 KiB/s wr, 21 op/s 2023-03-09T14:59:32.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:59:32 smithi038 ceph-mon[92422]: pgmap v561: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 137 KiB/s wr, 21 op/s 2023-03-09T14:59:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 545.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T14:59:34.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:59:33 smithi122 ceph-mon[93869]: pgmap v562: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 117 KiB/s wr, 19 op/s 2023-03-09T14:59:34.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:59:33 smithi181 ceph-mon[94189]: pgmap v562: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 117 KiB/s wr, 19 op/s 2023-03-09T14:59:34.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:59:33 smithi038 ceph-mon[92422]: pgmap v562: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 117 KiB/s wr, 19 op/s 2023-03-09T14:59:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:59:36 smithi122 ceph-mon[93869]: pgmap v563: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 99 KiB/s wr, 16 op/s 2023-03-09T14:59:36.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:59:36 smithi181 ceph-mon[94189]: pgmap v563: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 99 KiB/s wr, 16 op/s 2023-03-09T14:59:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:59:36 smithi038 ceph-mon[92422]: pgmap v563: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 99 KiB/s wr, 16 op/s 2023-03-09T14:59:38.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:59:38 smithi122 ceph-mon[93869]: pgmap v564: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 96 KiB/s wr, 18 op/s 2023-03-09T14:59:38.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:59:38 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T14:59:38.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:59:38 smithi181 ceph-mon[94189]: pgmap v564: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 96 KiB/s wr, 18 op/s 2023-03-09T14:59:38.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:59:38 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T14:59:38.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:59:38 smithi038 ceph-mon[92422]: pgmap v564: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 96 KiB/s wr, 18 op/s 2023-03-09T14:59:38.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:59:38 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T14:59:38.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 550.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T14:59:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:59:40 smithi122 ceph-mon[93869]: pgmap v565: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 89 KiB/s wr, 14 op/s 2023-03-09T14:59:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:59:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T14:59:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:59:40 smithi181 ceph-mon[94189]: pgmap v565: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 89 KiB/s wr, 14 op/s 2023-03-09T14:59:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:59:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T14:59:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:59:40 smithi038 ceph-mon[92422]: pgmap v565: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 89 KiB/s wr, 14 op/s 2023-03-09T14:59:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:59:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T14:59:42.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:59:42 smithi122 ceph-mon[93869]: pgmap v566: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 53 KiB/s wr, 10 op/s 2023-03-09T14:59:42.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:59:42 smithi181 ceph-mon[94189]: pgmap v566: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 53 KiB/s wr, 10 op/s 2023-03-09T14:59:42.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:59:42 smithi038 ceph-mon[92422]: pgmap v566: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 53 KiB/s wr, 10 op/s 2023-03-09T14:59:43.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 555.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T14:59:44.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:59:43 smithi122 ceph-mon[93869]: pgmap v567: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 53 KiB/s wr, 12 op/s 2023-03-09T14:59:44.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:59:43 smithi181 ceph-mon[94189]: pgmap v567: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 53 KiB/s wr, 12 op/s 2023-03-09T14:59:44.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:59:43 smithi038 ceph-mon[92422]: pgmap v567: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 53 KiB/s wr, 12 op/s 2023-03-09T14:59:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:59:46 smithi122 ceph-mon[93869]: pgmap v568: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 22 KiB/s wr, 6 op/s 2023-03-09T14:59:46.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:59:46 smithi181 ceph-mon[94189]: pgmap v568: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 22 KiB/s wr, 6 op/s 2023-03-09T14:59:46.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:59:46 smithi038 ceph-mon[92422]: pgmap v568: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 22 KiB/s wr, 6 op/s 2023-03-09T14:59:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:59:48 smithi122 ceph-mon[93869]: pgmap v569: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 22 KiB/s wr, 8 op/s 2023-03-09T14:59:48.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:59:48 smithi181 ceph-mon[94189]: pgmap v569: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 22 KiB/s wr, 8 op/s 2023-03-09T14:59:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:59:48 smithi038 ceph-mon[92422]: pgmap v569: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 22 KiB/s wr, 8 op/s 2023-03-09T14:59:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 560.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T14:59:49.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:59:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:59:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:59:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:59:49.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:59:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T14:59:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:59:50 smithi122 ceph-mon[93869]: pgmap v570: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T14:59:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:59:50 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:59:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:59:50 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:59:50.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:59:50 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:59:50.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:59:50 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:59:50.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:59:50 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:59:50.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:59:50 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:59:50.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:59:50 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:59:50.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:59:50 smithi181 ceph-mon[94189]: pgmap v570: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T14:59:50.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:59:50 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:59:50.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:59:50 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:59:50.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:59:50 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:59:50.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:59:50 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:59:50.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:59:50 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:59:50.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:59:50 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:59:50.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:59:50 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:59:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:59:50 smithi038 ceph-mon[92422]: pgmap v570: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T14:59:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:59:50 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:59:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:59:50 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:59:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:59:50 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:59:50.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:59:50 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:59:50.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:59:50 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T14:59:50.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:59:50 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T14:59:50.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:59:50 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T14:59:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:59:52 smithi122 ceph-mon[93869]: pgmap v571: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 15 KiB/s wr, 6 op/s 2023-03-09T14:59:52.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:59:52 smithi181 ceph-mon[94189]: pgmap v571: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 15 KiB/s wr, 6 op/s 2023-03-09T14:59:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:59:52 smithi038 ceph-mon[92422]: pgmap v571: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 15 KiB/s wr, 6 op/s 2023-03-09T14:59:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 565.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T14:59:54.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:59:53 smithi122 ceph-mon[93869]: pgmap v572: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 23 KiB/s wr, 9 op/s 2023-03-09T14:59:54.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:59:53 smithi181 ceph-mon[94189]: pgmap v572: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 23 KiB/s wr, 9 op/s 2023-03-09T14:59:54.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:59:53 smithi038 ceph-mon[92422]: pgmap v572: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 23 KiB/s wr, 9 op/s 2023-03-09T14:59:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:59:56 smithi122 ceph-mon[93869]: pgmap v573: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 5 op/s 2023-03-09T14:59:56.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:59:56 smithi181 ceph-mon[94189]: pgmap v573: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 5 op/s 2023-03-09T14:59:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:59:56 smithi038 ceph-mon[92422]: pgmap v573: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 5 op/s 2023-03-09T14:59:57.619 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-03-09T14:59:57.764 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.761+0000 7fa3c2a99700 1 -- 172.21.15.38:0/1208243017 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa3bc0ff9a0 msgr2=0x7fa3bc103960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:59:57.764 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.761+0000 7fa3c2a99700 1 --2- 172.21.15.38:0/1208243017 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa3bc0ff9a0 0x7fa3bc103960 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fa3a80099c0 tx=0x7fa3a8009cd0 comp rx=0 tx=0).stop 2023-03-09T14:59:57.764 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.761+0000 7fa3c2a99700 1 -- 172.21.15.38:0/1208243017 shutdown_connections 2023-03-09T14:59:57.764 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.761+0000 7fa3c2a99700 1 --2- 172.21.15.38:0/1208243017 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa3bc104040 0x7fa3bc10bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:59:57.764 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.761+0000 7fa3c2a99700 1 --2- 172.21.15.38:0/1208243017 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa3bc0ff000 0x7fa3bc0ff3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:59:57.765 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.761+0000 7fa3c2a99700 1 --2- 172.21.15.38:0/1208243017 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa3bc0ff9a0 0x7fa3bc103960 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:59:57.765 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.761+0000 7fa3c2a99700 1 -- 172.21.15.38:0/1208243017 >> 172.21.15.38:0/1208243017 conn(0x7fa3bc075c60 msgr2=0x7fa3bc076060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:59:57.765 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.761+0000 7fa3c2a99700 1 -- 172.21.15.38:0/1208243017 shutdown_connections 2023-03-09T14:59:57.765 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.762+0000 7fa3c2a99700 1 -- 172.21.15.38:0/1208243017 wait complete. 2023-03-09T14:59:57.765 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.762+0000 7fa3c2a99700 1 Processor -- start 2023-03-09T14:59:57.765 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.762+0000 7fa3c2a99700 1 -- start start 2023-03-09T14:59:57.767 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.762+0000 7fa3c2a99700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa3bc0ff000 0x7fa3bc194f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:59:57.767 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.762+0000 7fa3c2a99700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa3bc0ff9a0 0x7fa3bc195470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:59:57.767 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.762+0000 7fa3c2a99700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa3bc104040 0x7fa3bc1998d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:59:57.767 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.762+0000 7fa3c2a99700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3bc199f10 con 0x7fa3bc0ff000 2023-03-09T14:59:57.768 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.762+0000 7fa3c2a99700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3bc19a080 con 0x7fa3bc0ff9a0 2023-03-09T14:59:57.768 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.762+0000 7fa3c2a99700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3bc19a240 con 0x7fa3bc104040 2023-03-09T14:59:57.768 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.763+0000 7fa3c0835700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa3bc0ff000 0x7fa3bc194f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:59:57.768 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.763+0000 7fa3bbfff700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa3bc0ff9a0 0x7fa3bc195470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:59:57.768 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.763+0000 7fa3c0835700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa3bc0ff000 0x7fa3bc194f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42292/0 (socket says 172.21.15.38:42292) 2023-03-09T14:59:57.768 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.763+0000 7fa3c0835700 1 -- 172.21.15.38:0/702903233 learned_addr learned my addr 172.21.15.38:0/702903233 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:59:57.768 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.763+0000 7fa3c0835700 1 -- 172.21.15.38:0/702903233 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa3bc104040 msgr2=0x7fa3bc1998d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:59:57.769 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.763+0000 7fa3c1036700 1 --2- 172.21.15.38:0/702903233 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa3bc104040 0x7fa3bc1998d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:59:57.769 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.763+0000 7fa3c0835700 1 --2- 172.21.15.38:0/702903233 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa3bc104040 0x7fa3bc1998d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:59:57.769 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.763+0000 7fa3c0835700 1 -- 172.21.15.38:0/702903233 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa3bc0ff9a0 msgr2=0x7fa3bc195470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:59:57.769 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.763+0000 7fa3c0835700 1 --2- 172.21.15.38:0/702903233 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa3bc0ff9a0 0x7fa3bc195470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:59:57.769 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.763+0000 7fa3c0835700 1 -- 172.21.15.38:0/702903233 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3a80095f0 con 0x7fa3bc0ff000 2023-03-09T14:59:57.769 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.763+0000 7fa3c1036700 1 --2- 172.21.15.38:0/702903233 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa3bc104040 0x7fa3bc1998d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-09T14:59:57.769 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.763+0000 7fa3bbfff700 1 --2- 172.21.15.38:0/702903233 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa3bc0ff9a0 0x7fa3bc195470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:59:57.770 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.763+0000 7fa3c0835700 1 --2- 172.21.15.38:0/702903233 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa3bc0ff000 0x7fa3bc194f30 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fa3b000e9a0 tx=0x7fa3b000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:59:57.770 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.763+0000 7fa3b9ffb700 1 -- 172.21.15.38:0/702903233 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3b0009790 con 0x7fa3bc0ff000 2023-03-09T14:59:57.770 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.763+0000 7fa3c2a99700 1 -- 172.21.15.38:0/702903233 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3bc10edc0 con 0x7fa3bc0ff000 2023-03-09T14:59:57.770 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.763+0000 7fa3b9ffb700 1 -- 172.21.15.38:0/702903233 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa3b000cbd0 con 0x7fa3bc0ff000 2023-03-09T14:59:57.770 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.763+0000 7fa3b9ffb700 1 -- 172.21.15.38:0/702903233 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3b0018680 con 0x7fa3bc0ff000 2023-03-09T14:59:57.770 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.763+0000 7fa3c2a99700 1 -- 172.21.15.38:0/702903233 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3bc10ec30 con 0x7fa3bc0ff000 2023-03-09T14:59:57.774 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.766+0000 7fa3b9ffb700 1 -- 172.21.15.38:0/702903233 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 25) v1 ==== 133295+0+0 (secure 0 0 0) 0x7fa3b001f060 con 0x7fa3bc0ff000 2023-03-09T14:59:57.774 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.767+0000 7fa3b9ffb700 1 --2- 172.21.15.38:0/702903233 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fa3ac09f4a0 0x7fa3ac0a1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:59:57.774 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.767+0000 7fa3b9ffb700 1 -- 172.21.15.38:0/702903233 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(250..250 src has 1..250) v4 ==== 9561+0+0 (secure 0 0 0) 0x7fa3b0014070 con 0x7fa3bc0ff000 2023-03-09T14:59:57.774 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.767+0000 7fa3bbfff700 1 --2- 172.21.15.38:0/702903233 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fa3ac09f4a0 0x7fa3ac0a1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:59:57.774 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.767+0000 7fa3cec22b80 1 Processor -- start 2023-03-09T14:59:57.775 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.767+0000 7fa3cec22b80 1 -- start start 2023-03-09T14:59:57.775 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.767+0000 7fa3cec22b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5577617cab90 0x5577617caf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:59:57.775 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.767+0000 7fa3cec22b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5577617cb4a0 0x5577617c4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:59:57.775 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.767+0000 7fa3cec22b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5577617c51f0 0x5577617c5660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:59:57.775 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.767+0000 7fa3cec22b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577616acda0 con 0x5577617cab90 2023-03-09T14:59:57.775 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.767+0000 7fa3cec22b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55776171b780 con 0x5577617cb4a0 2023-03-09T14:59:57.775 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.767+0000 7fa3cec22b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x557761714cb0 con 0x5577617c51f0 2023-03-09T14:59:57.776 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.767+0000 7fa3bbfff700 1 --2- 172.21.15.38:0/702903233 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fa3ac09f4a0 0x7fa3ac0a1950 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa3a80098e0 tx=0x7fa3a801a040 comp rx=0 tx=0).ready entity=mgr.15558 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:59:57.776 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.767+0000 7fa3c0835700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5577617cab90 0x5577617caf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:59:57.776 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.767+0000 7fa3bbfff700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5577617cb4a0 0x5577617c4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:59:57.776 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.767+0000 7fa3c0835700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5577617cab90 0x5577617caf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42300/0 (socket says 172.21.15.38:42300) 2023-03-09T14:59:57.776 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.768+0000 7fa3c0835700 1 -- 172.21.15.38:0/4127132744 learned_addr learned my addr 172.21.15.38:0/4127132744 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:59:57.776 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.768+0000 7fa3c1036700 1 --2- 172.21.15.38:0/4127132744 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5577617c51f0 0x5577617c5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:59:57.776 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.768+0000 7fa3c0835700 1 -- 172.21.15.38:0/4127132744 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5577617c51f0 msgr2=0x5577617c5660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:59:57.776 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.768+0000 7fa3c0835700 1 --2- 172.21.15.38:0/4127132744 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5577617c51f0 0x5577617c5660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:59:57.777 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.768+0000 7fa3c0835700 1 -- 172.21.15.38:0/4127132744 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5577617cb4a0 msgr2=0x5577617c4cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:59:57.777 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.768+0000 7fa3c0835700 1 --2- 172.21.15.38:0/4127132744 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5577617cb4a0 0x5577617c4cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:59:57.777 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.768+0000 7fa3c0835700 1 -- 172.21.15.38:0/4127132744 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5577616db190 con 0x5577617cab90 2023-03-09T14:59:57.777 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.768+0000 7fa3bbfff700 1 --2- 172.21.15.38:0/4127132744 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5577617cb4a0 0x5577617c4cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:59:57.777 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.768+0000 7fa3c0835700 1 --2- 172.21.15.38:0/4127132744 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5577617cab90 0x5577617caf60 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fa3b0097e80 tx=0x7fa3b00983a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ae6923c437da56c server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:59:57.777 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.768+0000 7fa3a6ffd700 1 -- 172.21.15.38:0/4127132744 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3b00a0690 con 0x5577617cab90 2023-03-09T14:59:57.777 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.768+0000 7fa3a6ffd700 1 -- 172.21.15.38:0/4127132744 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa3b000cd40 con 0x5577617cab90 2023-03-09T14:59:57.778 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.768+0000 7fa3a6ffd700 1 -- 172.21.15.38:0/4127132744 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3b0097b90 con 0x5577617cab90 2023-03-09T14:59:57.778 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.769+0000 7fa3cec22b80 1 -- 172.21.15.38:0/4127132744 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5577617cab90 msgr2=0x5577617caf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:59:57.778 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.769+0000 7fa3cec22b80 1 --2- 172.21.15.38:0/4127132744 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5577617cab90 0x5577617caf60 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fa3b0097e80 tx=0x7fa3b00983a0 comp rx=0 tx=0).stop 2023-03-09T14:59:57.778 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.769+0000 7fa3cec22b80 1 -- 172.21.15.38:0/4127132744 shutdown_connections 2023-03-09T14:59:57.778 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.769+0000 7fa3cec22b80 1 --2- 172.21.15.38:0/4127132744 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5577617cab90 0x5577617caf60 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:59:57.778 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.769+0000 7fa3cec22b80 1 --2- 172.21.15.38:0/4127132744 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5577617c51f0 0x5577617c5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:59:57.778 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.769+0000 7fa3cec22b80 1 --2- 172.21.15.38:0/4127132744 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5577617cb4a0 0x5577617c4cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:59:57.778 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.769+0000 7fa3cec22b80 1 -- 172.21.15.38:0/4127132744 >> 172.21.15.38:0/4127132744 conn(0x5577617bd8d0 msgr2=0x5577617bdca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:59:57.779 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.769+0000 7fa3cec22b80 1 -- 172.21.15.38:0/4127132744 shutdown_connections 2023-03-09T14:59:57.779 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.769+0000 7fa3cec22b80 1 -- 172.21.15.38:0/4127132744 wait complete. 2023-03-09T14:59:57.779 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.769+0000 7fa3cec22b80 1 -- start start 2023-03-09T14:59:57.779 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.769+0000 7fa3cec22b80 1 Processor -- start 2023-03-09T14:59:57.779 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.770+0000 7fa3cec22b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5577617c51f0 0x5577617d13e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:59:57.779 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.770+0000 7fa3cec22b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5577617cab90 0x5577617ce6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:59:57.779 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.770+0000 7fa3cec22b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5577617cb4a0 0x5577617cedc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:59:57.780 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.770+0000 7fa3cec22b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x557761598920 con 0x5577617c51f0 2023-03-09T14:59:57.780 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.770+0000 7fa3cec22b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577617cf530 con 0x5577617cab90 2023-03-09T14:59:57.780 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.770+0000 7fa3cec22b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577617dae50 con 0x5577617cb4a0 2023-03-09T14:59:57.780 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.770+0000 7fa3cec22b80 10 client.0 _ll_register_callbacks cb 0x5577616ba520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:59:57.780 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.770+0000 7fa3c0835700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5577617c51f0 0x5577617d13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:59:57.780 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.770+0000 7fa3bbfff700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5577617cab90 0x5577617ce6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:59:57.780 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.770+0000 7fa3c0835700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5577617c51f0 0x5577617d13e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42306/0 (socket says 172.21.15.38:42306) 2023-03-09T14:59:57.780 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.770+0000 7fa3bbfff700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5577617cab90 0x5577617ce6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.38:51498/0 (socket says 172.21.15.38:51498) 2023-03-09T14:59:57.781 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.770+0000 7fa3c0835700 1 -- 172.21.15.38:0/3135611842 learned_addr learned my addr 172.21.15.38:0/3135611842 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:59:57.781 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.770+0000 7fa3c1036700 1 --2- 172.21.15.38:0/3135611842 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5577617cb4a0 0x5577617cedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:59:57.781 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.770+0000 7fa3bbfff700 1 -- 172.21.15.38:0/3135611842 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5577617cb4a0 msgr2=0x5577617cedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:59:57.781 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.770+0000 7fa3bbfff700 1 --2- 172.21.15.38:0/3135611842 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5577617cb4a0 0x5577617cedc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:59:57.781 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.770+0000 7fa3bbfff700 1 -- 172.21.15.38:0/3135611842 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5577617c51f0 msgr2=0x5577617d13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:59:57.781 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.770+0000 7fa3bbfff700 1 --2- 172.21.15.38:0/3135611842 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5577617c51f0 0x5577617d13e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:59:57.781 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.770+0000 7fa3bbfff700 1 -- 172.21.15.38:0/3135611842 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3bc10c450 con 0x5577617cab90 2023-03-09T14:59:57.782 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.770+0000 7fa3bbfff700 1 --2- 172.21.15.38:0/3135611842 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5577617cab90 0x5577617ce6d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fa3a80058e0 tx=0x7fa3a8009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=22779c10ab7034ec server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:59:57.782 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.770+0000 7fa39bfff700 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2023-03-09T14:59:57.782 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.770+0000 7fa39bfff700 1 -- 172.21.15.38:0/3135611842 <== mon.2 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3a802b040 con 0x5577617cab90 2023-03-09T14:59:57.782 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.771+0000 7fa39bfff700 1 -- 172.21.15.38:0/3135611842 <== mon.2 v2:172.21.15.122:3300/0 2 ==== osd_map(250..250 src has 1..250) v4 ==== 9561+0+0 (secure 0 0 0) 0x7fa3a8024420 con 0x5577617cab90 2023-03-09T14:59:57.782 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.771+0000 7fa3cec22b80 1 -- 172.21.15.38:0/3135611842 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5577616e2190 con 0x5577617cab90 2023-03-09T14:59:57.782 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.771+0000 7fa39bfff700 1 -- 172.21.15.38:0/3135611842 <== mon.2 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa3a8024620 con 0x5577617cab90 2023-03-09T14:59:57.782 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.771+0000 7fa3cec22b80 10 client.26000 fetch_fsmap learned FSMap version 45 2023-03-09T14:59:57.782 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.771+0000 7fa3cec22b80 1 -- 172.21.15.38:0/3135611842 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) v3 -- 0x7fa3bc10ec30 con 0x5577617cab90 2023-03-09T14:59:57.783 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.771+0000 7fa39bfff700 1 -- 172.21.15.38:0/3135611842 <== mon.2 v2:172.21.15.122:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5577616e2190 con 0x5577617cab90 2023-03-09T14:59:57.783 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.771+0000 7fa39bfff700 1 -- 172.21.15.38:0/3135611842 <== mon.2 v2:172.21.15.122:3300/0 5 ==== fsmap(e 45) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fa3a8030760 con 0x5577617cab90 2023-03-09T14:59:57.783 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.771+0000 7fa3cec22b80 10 client.26000 fetch_fsmap finished waiting for FSMap version 45 2023-03-09T14:59:57.783 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.771+0000 7fa3cec22b80 10 client.26000 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T14:59:57.783 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.771+0000 7fa3cec22b80 20 client.26000 populate_metadata read hostname 'smithi038' 2023-03-09T14:59:57.783 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.771+0000 7fa3cec22b80 1 --2- 172.21.15.38:0/3135611842 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5577617def00 0x5577617e1310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:59:57.783 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.771+0000 7fa3cec22b80 4 client.26000 mds_command: new command op to 24530 tid=0[{"prefix": "get_command_descriptions"}] 2023-03-09T14:59:57.784 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.771+0000 7fa3cec22b80 1 -- 172.21.15.38:0/3135611842 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557761598f60 con 0x5577617def00 2023-03-09T14:59:57.784 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.772+0000 7fa3c0835700 1 --2- 172.21.15.38:0/3135611842 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5577617def00 0x5577617e1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:59:57.784 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.772+0000 7fa39bfff700 10 client.26000 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:59:57.784 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.772+0000 7fa3c0835700 1 --2- 172.21.15.38:0/3135611842 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5577617def00 0x5577617e1310 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2c177d8ec40ae737 server_cookie=1ba8807e38763e17 in_seq=0 out_seq=0 2023-03-09T14:59:57.785 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.773+0000 7fa39bfff700 1 -- 172.21.15.38:0/3135611842 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18278 (crc 0 0 0) 0x557761598f60 con 0x5577617def00 2023-03-09T14:59:57.785 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.773+0000 7fa39bfff700 10 client.26000 handle_command_reply: tid=0 2023-03-09T14:59:57.785 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.773+0000 7fa3cec22b80 1 client.26000 shutdown 2023-03-09T14:59:57.785 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.773+0000 7fa3cec22b80 1 -- 172.21.15.38:0/3135611842 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5577617cab90 msgr2=0x5577617ce6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:59:57.785 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.773+0000 7fa3cec22b80 1 --2- 172.21.15.38:0/3135611842 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5577617cab90 0x5577617ce6d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fa3a80058e0 tx=0x7fa3a8009310 comp rx=0 tx=0).stop 2023-03-09T14:59:57.786 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.773+0000 7fa3cec22b80 1 -- 172.21.15.38:0/3135611842 shutdown_connections 2023-03-09T14:59:57.786 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.773+0000 7fa3cec22b80 1 --2- 172.21.15.38:0/3135611842 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5577617def00 0x5577617e1310 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:59:57.786 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.773+0000 7fa3cec22b80 1 --2- 172.21.15.38:0/3135611842 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5577617c51f0 0x5577617d13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:59:57.786 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.773+0000 7fa3cec22b80 1 --2- 172.21.15.38:0/3135611842 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5577617cb4a0 0x5577617cedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:59:57.786 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.773+0000 7fa3cec22b80 1 --2- 172.21.15.38:0/3135611842 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5577617cab90 0x5577617ce6d0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:59:57.786 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.773+0000 7fa39bfff700 0 client.26000 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:59:57.786 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.773+0000 7fa3cec22b80 1 -- 172.21.15.38:0/3135611842 >> 172.21.15.38:0/3135611842 conn(0x5577617bd8d0 msgr2=0x5577617bdca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:59:57.787 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.773+0000 7fa3cec22b80 1 -- 172.21.15.38:0/3135611842 shutdown_connections 2023-03-09T14:59:57.787 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.773+0000 7fa3cec22b80 1 -- 172.21.15.38:0/3135611842 wait complete. 2023-03-09T14:59:57.787 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.773+0000 7fa3cec22b80 20 client.26000 trim_cache size 0 max 16384 2023-03-09T14:59:57.793 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.790+0000 7fa3cec22b80 1 Processor -- start 2023-03-09T14:59:57.793 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.790+0000 7fa3cec22b80 1 -- start start 2023-03-09T14:59:57.793 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.790+0000 7fa3cec22b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5577617c51f0 0x5577617d5de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:59:57.793 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.790+0000 7fa3cec22b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5577617cab90 0x5577617cde90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:59:57.793 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.790+0000 7fa3cec22b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5577617cb4a0 0x5577617c55c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:59:57.793 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.790+0000 7fa3cec22b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577617ccc60 con 0x5577617c51f0 2023-03-09T14:59:57.794 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.790+0000 7fa3cec22b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577617bd660 con 0x5577617cab90 2023-03-09T14:59:57.794 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.790+0000 7fa3cec22b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577617c93b0 con 0x5577617cb4a0 2023-03-09T14:59:57.794 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.791+0000 7fa3c0835700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5577617c51f0 0x5577617d5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:59:57.794 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.791+0000 7fa3bbfff700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5577617cab90 0x5577617cde90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:59:57.794 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.791+0000 7fa3c0835700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5577617c51f0 0x5577617d5de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42314/0 (socket says 172.21.15.38:42314) 2023-03-09T14:59:57.794 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.791+0000 7fa3c0835700 1 -- 172.21.15.38:0/2718749519 learned_addr learned my addr 172.21.15.38:0/2718749519 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:59:57.794 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.791+0000 7fa3c1036700 1 --2- 172.21.15.38:0/2718749519 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5577617cb4a0 0x5577617c55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:59:57.795 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.791+0000 7fa3c0835700 1 -- 172.21.15.38:0/2718749519 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5577617cb4a0 msgr2=0x5577617c55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:59:57.795 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.791+0000 7fa3c0835700 1 --2- 172.21.15.38:0/2718749519 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5577617cb4a0 0x5577617c55c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:59:57.795 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.791+0000 7fa3c0835700 1 -- 172.21.15.38:0/2718749519 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5577617cab90 msgr2=0x5577617cde90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:59:57.795 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.791+0000 7fa3c0835700 1 --2- 172.21.15.38:0/2718749519 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5577617cab90 0x5577617cde90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:59:57.795 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.791+0000 7fa3c0835700 1 -- 172.21.15.38:0/2718749519 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5577617bcef0 con 0x5577617c51f0 2023-03-09T14:59:57.795 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.791+0000 7fa3bbfff700 1 --2- 172.21.15.38:0/2718749519 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5577617cab90 0x5577617cde90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T14:59:57.795 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.791+0000 7fa3c1036700 1 --2- 172.21.15.38:0/2718749519 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5577617cb4a0 0x5577617c55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-09T14:59:57.795 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.791+0000 7fa3c0835700 1 --2- 172.21.15.38:0/2718749519 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5577617c51f0 0x5577617d5de0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fa3b00989e0 tx=0x7fa3b001a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e04acb031b84f012 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:59:57.796 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.791+0000 7fa3a77fe700 1 -- 172.21.15.38:0/2718749519 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3b0062070 con 0x5577617c51f0 2023-03-09T14:59:57.796 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.791+0000 7fa3a77fe700 1 -- 172.21.15.38:0/2718749519 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fa3b00a0710 con 0x5577617c51f0 2023-03-09T14:59:57.796 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.791+0000 7fa3a77fe700 1 -- 172.21.15.38:0/2718749519 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3b009c170 con 0x5577617c51f0 2023-03-09T14:59:57.799 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.792+0000 7fa3cec22b80 1 -- 172.21.15.38:0/2718749519 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5577617c51f0 msgr2=0x5577617d5de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:59:57.799 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.792+0000 7fa3cec22b80 1 --2- 172.21.15.38:0/2718749519 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5577617c51f0 0x5577617d5de0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fa3b00989e0 tx=0x7fa3b001a040 comp rx=0 tx=0).stop 2023-03-09T14:59:57.799 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.792+0000 7fa3cec22b80 1 -- 172.21.15.38:0/2718749519 shutdown_connections 2023-03-09T14:59:57.799 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.792+0000 7fa3cec22b80 1 --2- 172.21.15.38:0/2718749519 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5577617c51f0 0x5577617d5de0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:59:57.799 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.792+0000 7fa3cec22b80 1 --2- 172.21.15.38:0/2718749519 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5577617cb4a0 0x5577617c55c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:59:57.799 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.792+0000 7fa3cec22b80 1 --2- 172.21.15.38:0/2718749519 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5577617cab90 0x5577617cde90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:59:57.800 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.792+0000 7fa3cec22b80 1 -- 172.21.15.38:0/2718749519 >> 172.21.15.38:0/2718749519 conn(0x5577617bd8d0 msgr2=0x5577617db230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:59:57.800 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.792+0000 7fa3cec22b80 1 -- 172.21.15.38:0/2718749519 shutdown_connections 2023-03-09T14:59:57.800 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.792+0000 7fa3cec22b80 1 -- 172.21.15.38:0/2718749519 wait complete. 2023-03-09T14:59:57.800 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.792+0000 7fa3cec22b80 1 -- start start 2023-03-09T14:59:57.800 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.792+0000 7fa3cec22b80 1 Processor -- start 2023-03-09T14:59:57.800 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.792+0000 7fa3cec22b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5577617c51f0 0x5577617d5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:59:57.800 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.793+0000 7fa3cec22b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5577617cab90 0x5577617c6950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:59:57.801 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.793+0000 7fa3cec22b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5577617cb4a0 0x5577617c6f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:59:57.801 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.793+0000 7fa3cec22b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577617c7660 con 0x5577617c51f0 2023-03-09T14:59:57.801 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.793+0000 7fa3cec22b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x5577617c77a0 con 0x5577617cb4a0 2023-03-09T14:59:57.801 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.793+0000 7fa3cec22b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x557761835190 con 0x5577617cab90 2023-03-09T14:59:57.801 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.793+0000 7fa3cec22b80 10 client.0 _ll_register_callbacks cb 0x5577616ba520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T14:59:57.801 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.793+0000 7fa3c0835700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5577617c51f0 0x5577617d5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:59:57.801 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.793+0000 7fa3c0835700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5577617c51f0 0x5577617d5ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42320/0 (socket says 172.21.15.38:42320) 2023-03-09T14:59:57.801 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.793+0000 7fa3c0835700 1 -- 172.21.15.38:0/2538114055 learned_addr learned my addr 172.21.15.38:0/2538114055 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T14:59:57.802 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.793+0000 7fa3c1036700 1 --2- 172.21.15.38:0/2538114055 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5577617cb4a0 0x5577617c6f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:59:57.802 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.793+0000 7fa3bbfff700 1 --2- 172.21.15.38:0/2538114055 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5577617cab90 0x5577617c6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:59:57.802 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.793+0000 7fa3c0835700 1 -- 172.21.15.38:0/2538114055 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5577617cab90 msgr2=0x5577617c6950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:59:57.802 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.793+0000 7fa3c0835700 1 --2- 172.21.15.38:0/2538114055 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5577617cab90 0x5577617c6950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:59:57.802 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.793+0000 7fa3c0835700 1 -- 172.21.15.38:0/2538114055 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5577617cb4a0 msgr2=0x5577617c6f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T14:59:57.802 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.793+0000 7fa3c0835700 1 --2- 172.21.15.38:0/2538114055 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5577617cb4a0 0x5577617c6f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:59:57.802 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.793+0000 7fa3c0835700 1 -- 172.21.15.38:0/2538114055 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3bc10ec30 con 0x5577617c51f0 2023-03-09T14:59:57.803 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.793+0000 7fa3bbfff700 1 --2- 172.21.15.38:0/2538114055 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5577617cab90 0x5577617c6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-09T14:59:57.803 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.793+0000 7fa3c0835700 1 --2- 172.21.15.38:0/2538114055 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5577617c51f0 0x5577617d5ff0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fa3b0094f10 tx=0x7fa3b0097f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd36d6a3c1b3cd47 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T14:59:57.803 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.793+0000 7fa39bfff700 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2023-03-09T14:59:57.803 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.793+0000 7fa39bfff700 1 -- 172.21.15.38:0/2538114055 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3b0062030 con 0x5577617c51f0 2023-03-09T14:59:57.803 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.793+0000 7fa39bfff700 1 -- 172.21.15.38:0/2538114055 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(250..250 src has 1..250) v4 ==== 9561+0+0 (secure 0 0 0) 0x7fa3b0063070 con 0x5577617c51f0 2023-03-09T14:59:57.803 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.793+0000 7fa3cec22b80 1 -- 172.21.15.38:0/2538114055 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5577616d21b0 con 0x5577617c51f0 2023-03-09T14:59:57.803 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.794+0000 7fa39bfff700 1 -- 172.21.15.38:0/2538114055 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa3b009c170 con 0x5577617c51f0 2023-03-09T14:59:57.803 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.794+0000 7fa3cec22b80 10 client.16371 fetch_fsmap learned FSMap version 45 2023-03-09T14:59:57.804 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.794+0000 7fa3cec22b80 1 -- 172.21.15.38:0/2538114055 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) v3 -- 0x7fa3b00c6040 con 0x5577617c51f0 2023-03-09T14:59:57.804 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.794+0000 7fa39bfff700 1 -- 172.21.15.38:0/2538114055 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5577616d21b0 con 0x5577617c51f0 2023-03-09T14:59:57.804 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.794+0000 7fa39bfff700 1 -- 172.21.15.38:0/2538114055 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 45) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fa3b0080070 con 0x5577617c51f0 2023-03-09T14:59:57.804 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.794+0000 7fa3cec22b80 10 client.16371 fetch_fsmap finished waiting for FSMap version 45 2023-03-09T14:59:57.804 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.794+0000 7fa3cec22b80 10 client.16371 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T14:59:57.804 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.794+0000 7fa3cec22b80 20 client.16371 populate_metadata read hostname 'smithi038' 2023-03-09T14:59:57.804 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.794+0000 7fa3cec22b80 1 --2- 172.21.15.38:0/2538114055 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x557761836420 0x557761838830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T14:59:57.805 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.794+0000 7fa3cec22b80 4 client.16371 mds_command: new command op to 24530 tid=0[{"prefix": "scrub status"}] 2023-03-09T14:59:57.805 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.794+0000 7fa3cec22b80 1 -- 172.21.15.38:0/2538114055 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5577616bc1c0 con 0x557761836420 2023-03-09T14:59:57.805 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.794+0000 7fa3bbfff700 1 --2- 172.21.15.38:0/2538114055 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x557761836420 0x557761838830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T14:59:57.805 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.795+0000 7fa39bfff700 10 client.16371 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T14:59:57.805 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.795+0000 7fa3bbfff700 1 --2- 172.21.15.38:0/2538114055 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x557761836420 0x557761838830 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=624dd751bfb09f23 server_cookie=77d95043d8e5c2d6 in_seq=0 out_seq=0 2023-03-09T14:59:57.805 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.795+0000 7fa39bfff700 1 -- 172.21.15.38:0/2538114055 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+265 (crc 0 0 0) 0x5577616bc1c0 con 0x557761836420 2023-03-09T14:59:57.805 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.795+0000 7fa39bfff700 10 client.16371 handle_command_reply: tid=0 2023-03-09T14:59:57.805 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.795+0000 7fa3cec22b80 1 client.16371 shutdown 2023-03-09T14:59:57.806 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.795+0000 7fa3cec22b80 1 -- 172.21.15.38:0/2538114055 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5577617c51f0 msgr2=0x5577617d5ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:59:57.806 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.795+0000 7fa3cec22b80 1 --2- 172.21.15.38:0/2538114055 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5577617c51f0 0x5577617d5ff0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fa3b0094f10 tx=0x7fa3b0097f00 comp rx=0 tx=0).stop 2023-03-09T14:59:57.806 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.795+0000 7fa3cec22b80 1 -- 172.21.15.38:0/2538114055 shutdown_connections 2023-03-09T14:59:57.806 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.795+0000 7fa3cec22b80 1 --2- 172.21.15.38:0/2538114055 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x557761836420 0x557761838830 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:59:57.806 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.795+0000 7fa3cec22b80 1 --2- 172.21.15.38:0/2538114055 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5577617c51f0 0x5577617d5ff0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:59:57.806 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.795+0000 7fa39bfff700 0 client.16371 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T14:59:57.806 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.795+0000 7fa3cec22b80 1 --2- 172.21.15.38:0/2538114055 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5577617cb4a0 0x5577617c6f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:59:57.807 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.795+0000 7fa3cec22b80 1 --2- 172.21.15.38:0/2538114055 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5577617cab90 0x5577617c6950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:59:57.807 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.795+0000 7fa3cec22b80 1 -- 172.21.15.38:0/2538114055 >> 172.21.15.38:0/2538114055 conn(0x5577617bd8d0 msgr2=0x5577617c55c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:59:57.807 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.795+0000 7fa3cec22b80 1 -- 172.21.15.38:0/2538114055 shutdown_connections 2023-03-09T14:59:57.807 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.795+0000 7fa3cec22b80 1 -- 172.21.15.38:0/2538114055 wait complete. 2023-03-09T14:59:57.807 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.795+0000 7fa3cec22b80 20 client.16371 trim_cache size 0 max 16384 2023-03-09T14:59:57.807 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2023-03-09T14:59:57.808 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "scrub active (1 inodes in the stack)", 2023-03-09T14:59:57.808 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": { 2023-03-09T14:59:57.808 INFO:teuthology.orchestra.run.smithi038.stdout: "a14659f4-d7ea-4c5a-b15b-6138af3f909f": { 2023-03-09T14:59:57.808 INFO:teuthology.orchestra.run.smithi038.stdout: "path": "/", 2023-03-09T14:59:57.808 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "a14659f4-d7ea-4c5a-b15b-6138af3f909f", 2023-03-09T14:59:57.808 INFO:teuthology.orchestra.run.smithi038.stdout: "options": "recursive,force" 2023-03-09T14:59:57.808 INFO:teuthology.orchestra.run.smithi038.stdout: } 2023-03-09T14:59:57.809 INFO:teuthology.orchestra.run.smithi038.stdout: } 2023-03-09T14:59:57.809 INFO:teuthology.orchestra.run.smithi038.stdout:} 2023-03-09T14:59:57.810 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.796+0000 7fa3c2a99700 1 -- 172.21.15.38:0/702903233 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fa3ac09f4a0 msgr2=0x7fa3ac0a1950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:59:57.810 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.796+0000 7fa3c2a99700 1 --2- 172.21.15.38:0/702903233 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fa3ac09f4a0 0x7fa3ac0a1950 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fa3a80098e0 tx=0x7fa3a801a040 comp rx=0 tx=0).stop 2023-03-09T14:59:57.810 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.796+0000 7fa3c2a99700 1 -- 172.21.15.38:0/702903233 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa3bc0ff000 msgr2=0x7fa3bc194f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T14:59:57.810 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.796+0000 7fa3c2a99700 1 --2- 172.21.15.38:0/702903233 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa3bc0ff000 0x7fa3bc194f30 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fa3b000e9a0 tx=0x7fa3b000ecb0 comp rx=0 tx=0).stop 2023-03-09T14:59:57.810 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.796+0000 7fa3c2a99700 1 -- 172.21.15.38:0/702903233 shutdown_connections 2023-03-09T14:59:57.811 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.796+0000 7fa3c2a99700 1 --2- 172.21.15.38:0/702903233 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fa3ac09f4a0 0x7fa3ac0a1950 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:59:57.811 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.796+0000 7fa3c2a99700 1 --2- 172.21.15.38:0/702903233 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fa3bc0ff000 0x7fa3bc194f30 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:59:57.811 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.796+0000 7fa3c2a99700 1 --2- 172.21.15.38:0/702903233 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fa3bc104040 0x7fa3bc1998d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:59:57.811 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.796+0000 7fa3c2a99700 1 --2- 172.21.15.38:0/702903233 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fa3bc0ff9a0 0x7fa3bc195470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T14:59:57.811 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.796+0000 7fa3c2a99700 1 -- 172.21.15.38:0/702903233 >> 172.21.15.38:0/702903233 conn(0x7fa3bc075c60 msgr2=0x7fa3bc0fc940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T14:59:57.811 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.796+0000 7fa3c2a99700 1 -- 172.21.15.38:0/702903233 shutdown_connections 2023-03-09T14:59:57.811 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T14:59:57.796+0000 7fa3c2a99700 1 -- 172.21.15.38:0/702903233 wait complete. 2023-03-09T14:59:57.812 INFO:tasks.cephfs.filesystem:scrub status for tag:a14659f4-d7ea-4c5a-b15b-6138af3f909f - {'path': '/', 'tag': 'a14659f4-d7ea-4c5a-b15b-6138af3f909f', 'options': 'recursive,force'} 2023-03-09T14:59:58.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 14:59:58 smithi122 ceph-mon[93869]: pgmap v574: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 15 KiB/s wr, 7 op/s 2023-03-09T14:59:58.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 14:59:58 smithi181 ceph-mon[94189]: pgmap v574: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 15 KiB/s wr, 7 op/s 2023-03-09T14:59:58.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 14:59:58 smithi038 ceph-mon[92422]: pgmap v574: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 15 KiB/s wr, 7 op/s 2023-03-09T14:59:58.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 570.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:00:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:00:00 smithi122 ceph-mon[93869]: pgmap v575: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 15 KiB/s wr, 6 op/s 2023-03-09T15:00:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:00:00 smithi122 ceph-mon[93869]: overall HEALTH_OK 2023-03-09T15:00:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:00:00 smithi181 ceph-mon[94189]: pgmap v575: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 15 KiB/s wr, 6 op/s 2023-03-09T15:00:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:00:00 smithi181 ceph-mon[94189]: overall HEALTH_OK 2023-03-09T15:00:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:00:00 smithi038 ceph-mon[92422]: pgmap v575: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 15 KiB/s wr, 6 op/s 2023-03-09T15:00:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:00:00 smithi038 ceph-mon[92422]: overall HEALTH_OK 2023-03-09T15:00:02.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:00:02 smithi122 ceph-mon[93869]: pgmap v576: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.7 KiB/s wr, 5 op/s 2023-03-09T15:00:02.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:00:02 smithi181 ceph-mon[94189]: pgmap v576: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.7 KiB/s wr, 5 op/s 2023-03-09T15:00:02.503 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:00:02 smithi038 ceph-mon[92422]: pgmap v576: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.7 KiB/s wr, 5 op/s 2023-03-09T15:00:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 575.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:00:04.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:00:03 smithi122 ceph-mon[93869]: pgmap v577: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 49 KiB/s wr, 11 op/s 2023-03-09T15:00:04.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:00:03 smithi181 ceph-mon[94189]: pgmap v577: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 49 KiB/s wr, 11 op/s 2023-03-09T15:00:04.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:00:03 smithi038 ceph-mon[92422]: pgmap v577: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 49 KiB/s wr, 11 op/s 2023-03-09T15:00:06.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:00:06 smithi122 ceph-mon[93869]: pgmap v578: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 63 KiB/s wr, 21 op/s 2023-03-09T15:00:06.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:00:06 smithi181 ceph-mon[94189]: pgmap v578: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 63 KiB/s wr, 21 op/s 2023-03-09T15:00:06.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:00:06 smithi038 ceph-mon[92422]: pgmap v578: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 63 KiB/s wr, 21 op/s 2023-03-09T15:00:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:00:08 smithi122 ceph-mon[93869]: pgmap v579: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 72 KiB/s wr, 24 op/s 2023-03-09T15:00:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:00:08 smithi181 ceph-mon[94189]: pgmap v579: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 72 KiB/s wr, 24 op/s 2023-03-09T15:00:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:00:08 smithi038 ceph-mon[92422]: pgmap v579: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 72 KiB/s wr, 24 op/s 2023-03-09T15:00:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 580.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:00:10.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:00:10 smithi122 ceph-mon[93869]: pgmap v580: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 90 KiB/s wr, 24 op/s 2023-03-09T15:00:10.481 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:00:10 smithi122 ceph-mon[93869]: osdmap e251: 12 total, 12 up, 12 in 2023-03-09T15:00:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:00:10 smithi181 ceph-mon[94189]: pgmap v580: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 90 KiB/s wr, 24 op/s 2023-03-09T15:00:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:00:10 smithi181 ceph-mon[94189]: osdmap e251: 12 total, 12 up, 12 in 2023-03-09T15:00:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:00:10 smithi038 ceph-mon[92422]: pgmap v580: 129 pgs: 129 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 90 KiB/s wr, 24 op/s 2023-03-09T15:00:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:00:10 smithi038 ceph-mon[92422]: osdmap e251: 12 total, 12 up, 12 in 2023-03-09T15:00:12.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:00:12 smithi181 ceph-mon[94189]: pgmap v582: 129 pgs: 5 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 110 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 108 KiB/s wr, 27 op/s 2023-03-09T15:00:12.486 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:00:12 smithi122 ceph-mon[93869]: pgmap v582: 129 pgs: 5 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 110 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 108 KiB/s wr, 27 op/s 2023-03-09T15:00:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:00:12 smithi038 ceph-mon[92422]: pgmap v582: 129 pgs: 5 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 110 active+clean; 13 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 108 KiB/s wr, 27 op/s 2023-03-09T15:00:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 585.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:00:14.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:00:13 smithi122 ceph-mon[93869]: pgmap v583: 129 pgs: 11 active+clean+snaptrim, 30 active+clean+snaptrim_wait, 88 active+clean; 12 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 75 KiB/s wr, 24 op/s 2023-03-09T15:00:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:00:13 smithi181 ceph-mon[94189]: pgmap v583: 129 pgs: 11 active+clean+snaptrim, 30 active+clean+snaptrim_wait, 88 active+clean; 12 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 75 KiB/s wr, 24 op/s 2023-03-09T15:00:14.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:00:13 smithi038 ceph-mon[92422]: pgmap v583: 129 pgs: 11 active+clean+snaptrim, 30 active+clean+snaptrim_wait, 88 active+clean; 12 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 75 KiB/s wr, 24 op/s 2023-03-09T15:00:16.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:00:16 smithi122 ceph-mon[93869]: pgmap v584: 129 pgs: 10 active+clean+snaptrim, 24 active+clean+snaptrim_wait, 95 active+clean; 12 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 49 KiB/s wr, 10 op/s 2023-03-09T15:00:16.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:00:16 smithi181 ceph-mon[94189]: pgmap v584: 129 pgs: 10 active+clean+snaptrim, 24 active+clean+snaptrim_wait, 95 active+clean; 12 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 49 KiB/s wr, 10 op/s 2023-03-09T15:00:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:00:16 smithi038 ceph-mon[92422]: pgmap v584: 129 pgs: 10 active+clean+snaptrim, 24 active+clean+snaptrim_wait, 95 active+clean; 12 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 49 KiB/s wr, 10 op/s 2023-03-09T15:00:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:00:18 smithi122 ceph-mon[93869]: pgmap v585: 129 pgs: 5 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 116 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 40 KiB/s wr, 9 op/s 2023-03-09T15:00:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:00:18 smithi122 ceph-mon[93869]: osdmap e252: 12 total, 12 up, 12 in 2023-03-09T15:00:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:00:18 smithi181 ceph-mon[94189]: pgmap v585: 129 pgs: 5 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 116 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 40 KiB/s wr, 9 op/s 2023-03-09T15:00:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:00:18 smithi181 ceph-mon[94189]: osdmap e252: 12 total, 12 up, 12 in 2023-03-09T15:00:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:00:18 smithi038 ceph-mon[92422]: pgmap v585: 129 pgs: 5 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 116 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 40 KiB/s wr, 9 op/s 2023-03-09T15:00:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:00:18 smithi038 ceph-mon[92422]: osdmap e252: 12 total, 12 up, 12 in 2023-03-09T15:00:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 590.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:00:20.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:00:20 smithi122 ceph-mon[93869]: pgmap v587: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 34 KiB/s wr, 9 op/s 2023-03-09T15:00:20.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:00:20 smithi181 ceph-mon[94189]: pgmap v587: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 34 KiB/s wr, 9 op/s 2023-03-09T15:00:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:00:20 smithi038 ceph-mon[92422]: pgmap v587: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 34 KiB/s wr, 9 op/s 2023-03-09T15:00:22.399 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:00:22 smithi181 ceph-mon[94189]: pgmap v588: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 33 KiB/s wr, 9 op/s 2023-03-09T15:00:22.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:00:22 smithi122 ceph-mon[93869]: pgmap v588: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 33 KiB/s wr, 9 op/s 2023-03-09T15:00:22.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:00:22 smithi038 ceph-mon[92422]: pgmap v588: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 33 KiB/s wr, 9 op/s 2023-03-09T15:00:23.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 595.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:00:24.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:00:23 smithi122 ceph-mon[93869]: osdmap e253: 12 total, 12 up, 12 in 2023-03-09T15:00:24.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:00:23 smithi122 ceph-mon[93869]: pgmap v590: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 26 KiB/s wr, 10 op/s 2023-03-09T15:00:24.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:00:23 smithi181 ceph-mon[94189]: osdmap e253: 12 total, 12 up, 12 in 2023-03-09T15:00:24.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:00:23 smithi181 ceph-mon[94189]: pgmap v590: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 26 KiB/s wr, 10 op/s 2023-03-09T15:00:24.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:00:23 smithi038 ceph-mon[92422]: osdmap e253: 12 total, 12 up, 12 in 2023-03-09T15:00:24.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:00:23 smithi038 ceph-mon[92422]: pgmap v590: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 26 KiB/s wr, 10 op/s 2023-03-09T15:00:26.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:00:26 smithi122 ceph-mon[93869]: pgmap v591: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T15:00:26.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:00:26 smithi181 ceph-mon[94189]: pgmap v591: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T15:00:26.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:00:26 smithi038 ceph-mon[92422]: pgmap v591: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T15:00:27.813 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-03-09T15:00:27.960 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.957+0000 7f325f54e700 1 -- 172.21.15.38:0/576720728 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f32580ff000 msgr2=0x7f32580ff3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T15:00:27.961 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.957+0000 7f325f54e700 1 --2- 172.21.15.38:0/576720728 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f32580ff000 0x7f32580ff3d0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f32480099c0 tx=0x7f3248009cd0 comp rx=0 tx=0).stop 2023-03-09T15:00:27.961 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.957+0000 7f325f54e700 1 -- 172.21.15.38:0/576720728 shutdown_connections 2023-03-09T15:00:27.961 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.957+0000 7f325f54e700 1 --2- 172.21.15.38:0/576720728 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f32580ff000 0x7f32580ff3d0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:00:27.961 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.957+0000 7f325f54e700 1 --2- 172.21.15.38:0/576720728 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f3258104040 0x7f325810bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:00:27.962 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.957+0000 7f325f54e700 1 --2- 172.21.15.38:0/576720728 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f32580ff9a0 0x7f3258103960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:00:27.962 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.957+0000 7f325f54e700 1 -- 172.21.15.38:0/576720728 >> 172.21.15.38:0/576720728 conn(0x7f3258075c60 msgr2=0x7f3258076060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T15:00:27.962 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.957+0000 7f325f54e700 1 -- 172.21.15.38:0/576720728 shutdown_connections 2023-03-09T15:00:27.962 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.957+0000 7f325f54e700 1 -- 172.21.15.38:0/576720728 wait complete. 2023-03-09T15:00:27.962 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.958+0000 7f325f54e700 1 Processor -- start 2023-03-09T15:00:27.962 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.958+0000 7f325f54e700 1 -- start start 2023-03-09T15:00:27.964 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.958+0000 7f325f54e700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f32580ff000 0x7f3258194f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T15:00:27.965 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.958+0000 7f325f54e700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f32580ff9a0 0x7f3258195480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T15:00:27.965 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.958+0000 7f325f54e700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3258104040 0x7f32581998e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T15:00:27.965 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.958+0000 7f325f54e700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3258199f20 con 0x7f32580ff9a0 2023-03-09T15:00:27.966 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.958+0000 7f325f54e700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f325819a090 con 0x7f32580ff000 2023-03-09T15:00:27.966 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.958+0000 7f325f54e700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f325819a250 con 0x7f3258104040 2023-03-09T15:00:27.966 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.958+0000 7f325cae9700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f32580ff9a0 0x7f3258195480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T15:00:27.966 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.958+0000 7f325d2ea700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f32580ff000 0x7f3258194f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T15:00:27.967 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.958+0000 7f325cae9700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f32580ff9a0 0x7f3258195480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42342/0 (socket says 172.21.15.38:42342) 2023-03-09T15:00:27.967 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.958+0000 7f325cae9700 1 -- 172.21.15.38:0/2116953909 learned_addr learned my addr 172.21.15.38:0/2116953909 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T15:00:27.968 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.958+0000 7f325daeb700 1 --2- 172.21.15.38:0/2116953909 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3258104040 0x7f32581998e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T15:00:27.968 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.958+0000 7f325cae9700 1 -- 172.21.15.38:0/2116953909 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3258104040 msgr2=0x7f32581998e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T15:00:27.968 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.959+0000 7f325cae9700 1 --2- 172.21.15.38:0/2116953909 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3258104040 0x7f32581998e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:00:27.968 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.959+0000 7f325cae9700 1 -- 172.21.15.38:0/2116953909 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f32580ff000 msgr2=0x7f3258194f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T15:00:27.969 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.959+0000 7f325cae9700 1 --2- 172.21.15.38:0/2116953909 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f32580ff000 0x7f3258194f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:00:27.969 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.959+0000 7f325cae9700 1 -- 172.21.15.38:0/2116953909 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32480095f0 con 0x7f32580ff9a0 2023-03-09T15:00:27.969 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.959+0000 7f325cae9700 1 --2- 172.21.15.38:0/2116953909 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f32580ff9a0 0x7f3258195480 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f3254007e50 tx=0x7f325400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T15:00:27.970 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.959+0000 7f324e7fc700 1 -- 172.21.15.38:0/2116953909 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3254004220 con 0x7f32580ff9a0 2023-03-09T15:00:27.970 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.959+0000 7f324e7fc700 1 -- 172.21.15.38:0/2116953909 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3254014070 con 0x7f32580ff9a0 2023-03-09T15:00:27.970 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.959+0000 7f325f54e700 1 -- 172.21.15.38:0/2116953909 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f325810ee00 con 0x7f32580ff9a0 2023-03-09T15:00:27.970 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.959+0000 7f324e7fc700 1 -- 172.21.15.38:0/2116953909 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3254005520 con 0x7f32580ff9a0 2023-03-09T15:00:27.970 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.959+0000 7f325f54e700 1 -- 172.21.15.38:0/2116953909 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f325810ec70 con 0x7f32580ff9a0 2023-03-09T15:00:27.970 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.962+0000 7f324e7fc700 1 -- 172.21.15.38:0/2116953909 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 25) v1 ==== 133295+0+0 (secure 0 0 0) 0x7f3254021070 con 0x7f32580ff9a0 2023-03-09T15:00:27.977 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.962+0000 7f324e7fc700 1 --2- 172.21.15.38:0/2116953909 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f324409f4a0 0x7f32440a1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T15:00:27.977 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.962+0000 7f325d2ea700 1 --2- 172.21.15.38:0/2116953909 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f324409f4a0 0x7f32440a1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T15:00:27.977 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.962+0000 7f324e7fc700 1 -- 172.21.15.38:0/2116953909 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 9561+0+0 (secure 0 0 0) 0x7f3254016070 con 0x7f32580ff9a0 2023-03-09T15:00:27.977 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.963+0000 7f325d2ea700 1 --2- 172.21.15.38:0/2116953909 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f324409f4a0 0x7f32440a1950 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3248009990 tx=0x7f3248005fb0 comp rx=0 tx=0).ready entity=mgr.15558 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T15:00:27.977 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.963+0000 7f326b6d7b80 1 Processor -- start 2023-03-09T15:00:27.977 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.963+0000 7f326b6d7b80 1 -- start start 2023-03-09T15:00:27.978 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.963+0000 7f326b6d7b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dddc43ec40 0x55dddc43f010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T15:00:27.978 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.963+0000 7f326b6d7b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dddc43f550 0x55dddc438eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T15:00:27.978 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.963+0000 7f326b6d7b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dddc4393f0 0x55dddc439820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T15:00:27.978 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.963+0000 7f326b6d7b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dddc31eda0 con 0x55dddc43f550 2023-03-09T15:00:27.978 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.963+0000 7f326b6d7b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dddc38d780 con 0x55dddc4393f0 2023-03-09T15:00:27.978 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.963+0000 7f326b6d7b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dddc386cb0 con 0x55dddc43ec40 2023-03-09T15:00:27.978 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.963+0000 7f325cae9700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dddc43f550 0x55dddc438eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T15:00:27.979 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.963+0000 7f325daeb700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dddc4393f0 0x55dddc439820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T15:00:27.979 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.963+0000 7f325cae9700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dddc43f550 0x55dddc438eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42350/0 (socket says 172.21.15.38:42350) 2023-03-09T15:00:27.979 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.963+0000 7f325cae9700 1 -- 172.21.15.38:0/3791771758 learned_addr learned my addr 172.21.15.38:0/3791771758 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T15:00:27.979 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.964+0000 7f325d2ea700 1 --2- 172.21.15.38:0/3791771758 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dddc43ec40 0x55dddc43f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T15:00:27.979 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.964+0000 7f325cae9700 1 -- 172.21.15.38:0/3791771758 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dddc43ec40 msgr2=0x55dddc43f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T15:00:27.979 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.964+0000 7f325cae9700 1 --2- 172.21.15.38:0/3791771758 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dddc43ec40 0x55dddc43f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:00:27.979 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.964+0000 7f325cae9700 1 -- 172.21.15.38:0/3791771758 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dddc4393f0 msgr2=0x55dddc439820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T15:00:27.979 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.964+0000 7f325cae9700 1 --2- 172.21.15.38:0/3791771758 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dddc4393f0 0x55dddc439820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:00:27.980 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.964+0000 7f325cae9700 1 -- 172.21.15.38:0/3791771758 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dddc34d190 con 0x55dddc43f550 2023-03-09T15:00:27.980 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.964+0000 7f325daeb700 1 --2- 172.21.15.38:0/3791771758 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dddc4393f0 0x55dddc439820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T15:00:27.980 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.964+0000 7f325cae9700 1 --2- 172.21.15.38:0/3791771758 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dddc43f550 0x55dddc438eb0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f3254004fa0 tx=0x7f3254098fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45aee6b5a49edf0a server_cookie=0 in_seq=0 out_seq=0 2023-03-09T15:00:27.980 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.964+0000 7f32437fe700 1 -- 172.21.15.38:0/3791771758 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32540a17c0 con 0x55dddc43f550 2023-03-09T15:00:27.980 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.964+0000 7f32437fe700 1 -- 172.21.15.38:0/3791771758 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f3254005680 con 0x55dddc43f550 2023-03-09T15:00:27.980 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.964+0000 7f32437fe700 1 -- 172.21.15.38:0/3791771758 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3254098ac0 con 0x55dddc43f550 2023-03-09T15:00:27.980 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.965+0000 7f326b6d7b80 1 -- 172.21.15.38:0/3791771758 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dddc43f550 msgr2=0x55dddc438eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T15:00:27.980 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.965+0000 7f326b6d7b80 1 --2- 172.21.15.38:0/3791771758 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dddc43f550 0x55dddc438eb0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f3254004fa0 tx=0x7f3254098fb0 comp rx=0 tx=0).stop 2023-03-09T15:00:27.981 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.965+0000 7f326b6d7b80 1 -- 172.21.15.38:0/3791771758 shutdown_connections 2023-03-09T15:00:27.981 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.965+0000 7f326b6d7b80 1 --2- 172.21.15.38:0/3791771758 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dddc4393f0 0x55dddc439820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:00:27.981 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.965+0000 7f326b6d7b80 1 --2- 172.21.15.38:0/3791771758 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dddc43ec40 0x55dddc43f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:00:27.981 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.965+0000 7f326b6d7b80 1 --2- 172.21.15.38:0/3791771758 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dddc43f550 0x55dddc438eb0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:00:27.981 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.965+0000 7f326b6d7b80 1 -- 172.21.15.38:0/3791771758 >> 172.21.15.38:0/3791771758 conn(0x55dddc42fa20 msgr2=0x55dddc42fdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T15:00:27.981 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.965+0000 7f326b6d7b80 1 -- 172.21.15.38:0/3791771758 shutdown_connections 2023-03-09T15:00:27.981 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.965+0000 7f326b6d7b80 1 -- 172.21.15.38:0/3791771758 wait complete. 2023-03-09T15:00:27.982 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.965+0000 7f326b6d7b80 1 -- start start 2023-03-09T15:00:27.982 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.965+0000 7f326b6d7b80 1 Processor -- start 2023-03-09T15:00:27.982 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.966+0000 7f326b6d7b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dddc4393f0 0x55dddc4454e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T15:00:27.982 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.966+0000 7f326b6d7b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dddc43ec40 0x55dddc442760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T15:00:27.983 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.966+0000 7f326b6d7b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dddc43f550 0x55dddc442de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T15:00:27.983 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.966+0000 7f326b6d7b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dddc20a920 con 0x55dddc43ec40 2023-03-09T15:00:27.983 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.966+0000 7f326b6d7b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dddc443510 con 0x55dddc4393f0 2023-03-09T15:00:27.983 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.966+0000 7f326b6d7b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dddc44f020 con 0x55dddc43f550 2023-03-09T15:00:27.983 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.966+0000 7f326b6d7b80 10 client.0 _ll_register_callbacks cb 0x55dddc32c520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T15:00:27.983 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.966+0000 7f325d2ea700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dddc4393f0 0x55dddc4454e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T15:00:27.984 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.966+0000 7f325cae9700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dddc43ec40 0x55dddc442760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T15:00:27.984 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.966+0000 7f325d2ea700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dddc4393f0 0x55dddc4454e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.38:51544/0 (socket says 172.21.15.38:51544) 2023-03-09T15:00:27.984 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.966+0000 7f325d2ea700 1 -- 172.21.15.38:0/2824248204 learned_addr learned my addr 172.21.15.38:0/2824248204 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T15:00:27.984 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.966+0000 7f325daeb700 1 --2- 172.21.15.38:0/2824248204 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dddc43f550 0x55dddc442de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T15:00:27.984 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.966+0000 7f325cae9700 1 -- 172.21.15.38:0/2824248204 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dddc43f550 msgr2=0x55dddc442de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T15:00:27.984 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.966+0000 7f325cae9700 1 --2- 172.21.15.38:0/2824248204 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dddc43f550 0x55dddc442de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:00:27.984 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.966+0000 7f325cae9700 1 -- 172.21.15.38:0/2824248204 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dddc4393f0 msgr2=0x55dddc4454e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T15:00:27.984 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.966+0000 7f325cae9700 1 --2- 172.21.15.38:0/2824248204 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dddc4393f0 0x55dddc4454e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:00:27.985 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.966+0000 7f325cae9700 1 -- 172.21.15.38:0/2824248204 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3250009050 con 0x55dddc43ec40 2023-03-09T15:00:27.985 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.966+0000 7f325d2ea700 1 --2- 172.21.15.38:0/2824248204 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dddc4393f0 0x55dddc4454e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T15:00:27.985 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.966+0000 7f325cae9700 1 --2- 172.21.15.38:0/2824248204 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dddc43ec40 0x55dddc442760 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f3254096040 tx=0x7f325400df60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=801910e9dae8aeef server_cookie=0 in_seq=0 out_seq=0 2023-03-09T15:00:27.985 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.966+0000 7f3240ff9700 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2023-03-09T15:00:27.985 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.967+0000 7f3240ff9700 1 -- 172.21.15.38:0/2824248204 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32540b5070 con 0x55dddc43ec40 2023-03-09T15:00:27.985 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.967+0000 7f3240ff9700 1 -- 172.21.15.38:0/2824248204 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(253..253 src has 1..253) v4 ==== 9561+0+0 (secure 0 0 0) 0x7f3254005680 con 0x55dddc43ec40 2023-03-09T15:00:27.985 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.967+0000 7f326b6d7b80 1 -- 172.21.15.38:0/2824248204 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55dddc354190 con 0x55dddc43ec40 2023-03-09T15:00:27.986 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.967+0000 7f3240ff9700 1 -- 172.21.15.38:0/2824248204 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) v2 ==== 24+0+0 (secure 0 0 0) 0x7f32540a17c0 con 0x55dddc43ec40 2023-03-09T15:00:27.986 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.967+0000 7f326b6d7b80 10 client.16383 fetch_fsmap learned FSMap version 45 2023-03-09T15:00:27.986 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.967+0000 7f326b6d7b80 1 -- 172.21.15.38:0/2824248204 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) v3 -- 0x7f325810ec70 con 0x55dddc43ec40 2023-03-09T15:00:27.986 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.967+0000 7f3240ff9700 1 -- 172.21.15.38:0/2824248204 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55dddc354190 con 0x55dddc43ec40 2023-03-09T15:00:27.986 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.967+0000 7f3240ff9700 1 -- 172.21.15.38:0/2824248204 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 45) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f32540b72c0 con 0x55dddc43ec40 2023-03-09T15:00:27.986 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.967+0000 7f326b6d7b80 10 client.16383 fetch_fsmap finished waiting for FSMap version 45 2023-03-09T15:00:27.986 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.967+0000 7f326b6d7b80 10 client.16383 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T15:00:27.986 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.967+0000 7f326b6d7b80 20 client.16383 populate_metadata read hostname 'smithi038' 2023-03-09T15:00:27.987 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.967+0000 7f326b6d7b80 1 --2- 172.21.15.38:0/2824248204 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55dddc452c60 0x55dddc455070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T15:00:27.987 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.967+0000 7f326b6d7b80 4 client.16383 mds_command: new command op to 24530 tid=0[{"prefix": "get_command_descriptions"}] 2023-03-09T15:00:27.987 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.967+0000 7f326b6d7b80 1 -- 172.21.15.38:0/2824248204 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55dddc20af60 con 0x55dddc452c60 2023-03-09T15:00:27.987 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.967+0000 7f325d2ea700 1 --2- 172.21.15.38:0/2824248204 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55dddc452c60 0x55dddc455070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T15:00:27.987 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.968+0000 7f3240ff9700 10 client.16383 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T15:00:27.987 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.968+0000 7f325d2ea700 1 --2- 172.21.15.38:0/2824248204 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55dddc452c60 0x55dddc455070 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e81e9d7d43743d5d server_cookie=8618e07b7e126b4b in_seq=0 out_seq=0 2023-03-09T15:00:27.987 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.968+0000 7f3240ff9700 1 -- 172.21.15.38:0/2824248204 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18278 (crc 0 0 0) 0x55dddc20af60 con 0x55dddc452c60 2023-03-09T15:00:27.988 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.968+0000 7f3240ff9700 10 client.16383 handle_command_reply: tid=0 2023-03-09T15:00:27.988 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.968+0000 7f326b6d7b80 1 client.16383 shutdown 2023-03-09T15:00:27.988 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.968+0000 7f326b6d7b80 1 -- 172.21.15.38:0/2824248204 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dddc43ec40 msgr2=0x55dddc442760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T15:00:27.988 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.968+0000 7f326b6d7b80 1 --2- 172.21.15.38:0/2824248204 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dddc43ec40 0x55dddc442760 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f3254096040 tx=0x7f325400df60 comp rx=0 tx=0).stop 2023-03-09T15:00:27.988 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.969+0000 7f326b6d7b80 1 -- 172.21.15.38:0/2824248204 shutdown_connections 2023-03-09T15:00:27.988 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.969+0000 7f326b6d7b80 1 --2- 172.21.15.38:0/2824248204 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55dddc452c60 0x55dddc455070 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:00:27.988 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.969+0000 7f326b6d7b80 1 --2- 172.21.15.38:0/2824248204 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dddc43f550 0x55dddc442de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:00:27.988 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.969+0000 7f3240ff9700 0 client.16383 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T15:00:27.989 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.969+0000 7f326b6d7b80 1 --2- 172.21.15.38:0/2824248204 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dddc4393f0 0x55dddc4454e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:00:27.989 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.969+0000 7f326b6d7b80 1 --2- 172.21.15.38:0/2824248204 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dddc43ec40 0x55dddc442760 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:00:27.989 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.969+0000 7f326b6d7b80 1 -- 172.21.15.38:0/2824248204 >> 172.21.15.38:0/2824248204 conn(0x55dddc42fa20 msgr2=0x55dddc42ec20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T15:00:27.989 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.969+0000 7f326b6d7b80 1 -- 172.21.15.38:0/2824248204 shutdown_connections 2023-03-09T15:00:27.989 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.969+0000 7f326b6d7b80 1 -- 172.21.15.38:0/2824248204 wait complete. 2023-03-09T15:00:27.989 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.969+0000 7f326b6d7b80 20 client.16383 trim_cache size 0 max 16384 2023-03-09T15:00:27.994 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.986+0000 7f326b6d7b80 1 Processor -- start 2023-03-09T15:00:27.994 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.986+0000 7f326b6d7b80 1 -- start start 2023-03-09T15:00:27.994 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.986+0000 7f326b6d7b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dddc4393f0 0x55dddc441f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T15:00:27.995 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.986+0000 7f326b6d7b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dddc43ec40 0x55dddc445110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T15:00:27.995 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.986+0000 7f326b6d7b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dddc43f550 0x55dddc44e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T15:00:27.995 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.986+0000 7f326b6d7b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dddc443d40 con 0x55dddc4393f0 2023-03-09T15:00:27.995 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.986+0000 7f326b6d7b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dddc44fa90 con 0x55dddc43f550 2023-03-09T15:00:27.995 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.986+0000 7f326b6d7b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dddc430f20 con 0x55dddc43ec40 2023-03-09T15:00:27.995 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.986+0000 7f325d2ea700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dddc4393f0 0x55dddc441f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T15:00:27.995 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.986+0000 7f325d2ea700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dddc4393f0 0x55dddc441f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42362/0 (socket says 172.21.15.38:42362) 2023-03-09T15:00:27.995 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.986+0000 7f325d2ea700 1 -- 172.21.15.38:0/3882831033 learned_addr learned my addr 172.21.15.38:0/3882831033 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T15:00:27.996 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.986+0000 7f325daeb700 1 --2- 172.21.15.38:0/3882831033 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dddc43f550 0x55dddc44e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T15:00:27.996 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.986+0000 7f325cae9700 1 --2- 172.21.15.38:0/3882831033 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dddc43ec40 0x55dddc445110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T15:00:27.996 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.986+0000 7f325d2ea700 1 -- 172.21.15.38:0/3882831033 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dddc43ec40 msgr2=0x55dddc445110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T15:00:27.996 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.986+0000 7f325d2ea700 1 --2- 172.21.15.38:0/3882831033 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dddc43ec40 0x55dddc445110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:00:27.996 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.986+0000 7f325d2ea700 1 -- 172.21.15.38:0/3882831033 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dddc43f550 msgr2=0x55dddc44e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T15:00:27.996 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.986+0000 7f325d2ea700 1 --2- 172.21.15.38:0/3882831033 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dddc43f550 0x55dddc44e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:00:27.996 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.986+0000 7f325d2ea700 1 -- 172.21.15.38:0/3882831033 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dddc19dc60 con 0x55dddc4393f0 2023-03-09T15:00:27.997 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.987+0000 7f325d2ea700 1 --2- 172.21.15.38:0/3882831033 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dddc4393f0 0x55dddc441f40 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f32480036c0 tx=0x7f3248022000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53ee4f94359d5b1e server_cookie=0 in_seq=0 out_seq=0 2023-03-09T15:00:27.997 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.987+0000 7f3243fff700 1 -- 172.21.15.38:0/3882831033 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f324802c070 con 0x55dddc4393f0 2023-03-09T15:00:27.997 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.987+0000 7f3243fff700 1 -- 172.21.15.38:0/3882831033 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f324802d040 con 0x55dddc4393f0 2023-03-09T15:00:27.997 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.987+0000 7f3243fff700 1 -- 172.21.15.38:0/3882831033 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32480040e0 con 0x55dddc4393f0 2023-03-09T15:00:27.997 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.987+0000 7f326b6d7b80 1 -- 172.21.15.38:0/3882831033 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dddc4393f0 msgr2=0x55dddc441f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T15:00:27.997 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.987+0000 7f326b6d7b80 1 --2- 172.21.15.38:0/3882831033 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dddc4393f0 0x55dddc441f40 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f32480036c0 tx=0x7f3248022000 comp rx=0 tx=0).stop 2023-03-09T15:00:27.997 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.987+0000 7f326b6d7b80 1 -- 172.21.15.38:0/3882831033 shutdown_connections 2023-03-09T15:00:27.998 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.987+0000 7f326b6d7b80 1 --2- 172.21.15.38:0/3882831033 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dddc4393f0 0x55dddc441f40 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:00:27.998 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.987+0000 7f326b6d7b80 1 --2- 172.21.15.38:0/3882831033 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dddc43f550 0x55dddc44e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:00:27.998 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.987+0000 7f326b6d7b80 1 --2- 172.21.15.38:0/3882831033 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dddc43ec40 0x55dddc445110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:00:27.998 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.987+0000 7f326b6d7b80 1 -- 172.21.15.38:0/3882831033 >> 172.21.15.38:0/3882831033 conn(0x55dddc42fa20 msgr2=0x55dddc44f380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T15:00:27.998 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.987+0000 7f326b6d7b80 1 -- 172.21.15.38:0/3882831033 shutdown_connections 2023-03-09T15:00:27.998 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.987+0000 7f326b6d7b80 1 -- 172.21.15.38:0/3882831033 wait complete. 2023-03-09T15:00:27.998 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.988+0000 7f326b6d7b80 1 -- start start 2023-03-09T15:00:27.998 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.988+0000 7f326b6d7b80 1 Processor -- start 2023-03-09T15:00:27.999 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.988+0000 7f326b6d7b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dddc4393f0 0x55dddc43be40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T15:00:27.999 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.988+0000 7f326b6d7b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dddc43ec40 0x55dddc43a770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T15:00:27.999 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.988+0000 7f326b6d7b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dddc43f550 0x55dddc43acb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T15:00:27.999 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.988+0000 7f326b6d7b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dddc43b1f0 con 0x55dddc43f550 2023-03-09T15:00:27.999 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.988+0000 7f326b6d7b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dddc43b360 con 0x55dddc4393f0 2023-03-09T15:00:27.999 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.988+0000 7f326b6d7b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dddc43b520 con 0x55dddc43ec40 2023-03-09T15:00:27.999 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.988+0000 7f326b6d7b80 10 client.0 _ll_register_callbacks cb 0x55dddc32c520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T15:00:27.999 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.988+0000 7f325daeb700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dddc43f550 0x55dddc43acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T15:00:28.000 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.988+0000 7f325d2ea700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dddc4393f0 0x55dddc43be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T15:00:28.000 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.988+0000 7f325daeb700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dddc43f550 0x55dddc43acb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42372/0 (socket says 172.21.15.38:42372) 2023-03-09T15:00:28.000 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.988+0000 7f325daeb700 1 -- 172.21.15.38:0/970711420 learned_addr learned my addr 172.21.15.38:0/970711420 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T15:00:28.000 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.988+0000 7f325cae9700 1 --2- 172.21.15.38:0/970711420 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dddc43ec40 0x55dddc43a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T15:00:28.000 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.989+0000 7f325daeb700 1 -- 172.21.15.38:0/970711420 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dddc43ec40 msgr2=0x55dddc43a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T15:00:28.000 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.989+0000 7f325daeb700 1 --2- 172.21.15.38:0/970711420 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dddc43ec40 0x55dddc43a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:00:28.000 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.989+0000 7f325daeb700 1 -- 172.21.15.38:0/970711420 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dddc4393f0 msgr2=0x55dddc43be40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T15:00:28.001 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.989+0000 7f325daeb700 1 --2- 172.21.15.38:0/970711420 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dddc4393f0 0x55dddc43be40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:00:28.001 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.989+0000 7f325daeb700 1 -- 172.21.15.38:0/970711420 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f325810c450 con 0x55dddc43f550 2023-03-09T15:00:28.001 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.989+0000 7f325daeb700 1 --2- 172.21.15.38:0/970711420 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dddc43f550 0x55dddc43acb0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f3250018e90 tx=0x7f3250018ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a277ea8653b84bf server_cookie=0 in_seq=0 out_seq=0 2023-03-09T15:00:28.002 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.989+0000 7f3240ff9700 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2023-03-09T15:00:28.002 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.989+0000 7f3240ff9700 1 -- 172.21.15.38:0/970711420 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f325000e070 con 0x55dddc43f550 2023-03-09T15:00:28.002 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.989+0000 7f3240ff9700 1 -- 172.21.15.38:0/970711420 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(253..253 src has 1..253) v4 ==== 9561+0+0 (secure 0 0 0) 0x7f3250008cd0 con 0x55dddc43f550 2023-03-09T15:00:28.002 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.989+0000 7f326b6d7b80 1 -- 172.21.15.38:0/970711420 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55dddc3441b0 con 0x55dddc43f550 2023-03-09T15:00:28.003 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.989+0000 7f3240ff9700 1 -- 172.21.15.38:0/970711420 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3250008e90 con 0x55dddc43f550 2023-03-09T15:00:28.003 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.989+0000 7f326b6d7b80 10 client.16389 fetch_fsmap learned FSMap version 45 2023-03-09T15:00:28.003 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.989+0000 7f326b6d7b80 1 -- 172.21.15.38:0/970711420 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) v3 -- 0x55dddc1db250 con 0x55dddc43f550 2023-03-09T15:00:28.004 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.989+0000 7f3240ff9700 1 -- 172.21.15.38:0/970711420 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x55dddc3441b0 con 0x55dddc43f550 2023-03-09T15:00:28.004 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.989+0000 7f3240ff9700 1 -- 172.21.15.38:0/970711420 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 45) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f32500073e0 con 0x55dddc43f550 2023-03-09T15:00:28.004 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.989+0000 7f326b6d7b80 10 client.16389 fetch_fsmap finished waiting for FSMap version 45 2023-03-09T15:00:28.004 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.989+0000 7f326b6d7b80 10 client.16389 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T15:00:28.004 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.989+0000 7f326b6d7b80 20 client.16389 populate_metadata read hostname 'smithi038' 2023-03-09T15:00:28.004 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.989+0000 7f326b6d7b80 1 --2- 172.21.15.38:0/970711420 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55dddc4a7f70 0x55dddc4aa380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T15:00:28.005 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.989+0000 7f326b6d7b80 4 client.16389 mds_command: new command op to 24530 tid=0[{"prefix": "scrub status"}] 2023-03-09T15:00:28.005 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.989+0000 7f326b6d7b80 1 -- 172.21.15.38:0/970711420 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55dddc32e1c0 con 0x55dddc4a7f70 2023-03-09T15:00:28.005 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.990+0000 7f325d2ea700 1 --2- 172.21.15.38:0/970711420 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55dddc4a7f70 0x55dddc4aa380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T15:00:28.005 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.990+0000 7f3240ff9700 10 client.16389 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T15:00:28.005 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.990+0000 7f325d2ea700 1 --2- 172.21.15.38:0/970711420 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55dddc4a7f70 0x55dddc4aa380 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=384964695399c008 server_cookie=2207b4abc7aea247 in_seq=0 out_seq=0 2023-03-09T15:00:28.005 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.990+0000 7f3240ff9700 1 -- 172.21.15.38:0/970711420 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+265 (crc 0 0 0) 0x55dddc32e1c0 con 0x55dddc4a7f70 2023-03-09T15:00:28.005 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.990+0000 7f3240ff9700 10 client.16389 handle_command_reply: tid=0 2023-03-09T15:00:28.006 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.990+0000 7f326b6d7b80 1 client.16389 shutdown 2023-03-09T15:00:28.006 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.990+0000 7f326b6d7b80 1 -- 172.21.15.38:0/970711420 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dddc43f550 msgr2=0x55dddc43acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T15:00:28.006 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.990+0000 7f326b6d7b80 1 --2- 172.21.15.38:0/970711420 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dddc43f550 0x55dddc43acb0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f3250018e90 tx=0x7f3250018ec0 comp rx=0 tx=0).stop 2023-03-09T15:00:28.006 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.990+0000 7f326b6d7b80 1 -- 172.21.15.38:0/970711420 shutdown_connections 2023-03-09T15:00:28.006 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.990+0000 7f326b6d7b80 1 --2- 172.21.15.38:0/970711420 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x55dddc4a7f70 0x55dddc4aa380 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:00:28.006 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.990+0000 7f326b6d7b80 1 --2- 172.21.15.38:0/970711420 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x55dddc43f550 0x55dddc43acb0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:00:28.006 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.990+0000 7f326b6d7b80 1 --2- 172.21.15.38:0/970711420 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x55dddc4393f0 0x55dddc43be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:00:28.006 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.990+0000 7f3240ff9700 0 client.16389 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T15:00:28.007 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.990+0000 7f326b6d7b80 1 --2- 172.21.15.38:0/970711420 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x55dddc43ec40 0x55dddc43a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:00:28.007 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.990+0000 7f326b6d7b80 1 -- 172.21.15.38:0/970711420 >> 172.21.15.38:0/970711420 conn(0x55dddc42fa20 msgr2=0x55dddc44e700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T15:00:28.007 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.990+0000 7f326b6d7b80 1 -- 172.21.15.38:0/970711420 shutdown_connections 2023-03-09T15:00:28.007 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.990+0000 7f326b6d7b80 1 -- 172.21.15.38:0/970711420 wait complete. 2023-03-09T15:00:28.007 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.991+0000 7f326b6d7b80 20 client.16389 trim_cache size 0 max 16384 2023-03-09T15:00:28.008 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2023-03-09T15:00:28.008 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "scrub active (1 inodes in the stack)", 2023-03-09T15:00:28.008 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": { 2023-03-09T15:00:28.008 INFO:teuthology.orchestra.run.smithi038.stdout: "a14659f4-d7ea-4c5a-b15b-6138af3f909f": { 2023-03-09T15:00:28.008 INFO:teuthology.orchestra.run.smithi038.stdout: "path": "/", 2023-03-09T15:00:28.008 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "a14659f4-d7ea-4c5a-b15b-6138af3f909f", 2023-03-09T15:00:28.009 INFO:teuthology.orchestra.run.smithi038.stdout: "options": "recursive,force" 2023-03-09T15:00:28.009 INFO:teuthology.orchestra.run.smithi038.stdout: } 2023-03-09T15:00:28.009 INFO:teuthology.orchestra.run.smithi038.stdout: } 2023-03-09T15:00:28.009 INFO:teuthology.orchestra.run.smithi038.stdout:} 2023-03-09T15:00:28.010 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.991+0000 7f325f54e700 1 -- 172.21.15.38:0/2116953909 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f324409f4a0 msgr2=0x7f32440a1950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T15:00:28.010 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.991+0000 7f325f54e700 1 --2- 172.21.15.38:0/2116953909 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f324409f4a0 0x7f32440a1950 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f3248009990 tx=0x7f3248005fb0 comp rx=0 tx=0).stop 2023-03-09T15:00:28.011 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.991+0000 7f325f54e700 1 -- 172.21.15.38:0/2116953909 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f32580ff9a0 msgr2=0x7f3258195480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T15:00:28.011 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.991+0000 7f325f54e700 1 --2- 172.21.15.38:0/2116953909 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f32580ff9a0 0x7f3258195480 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f3254007e50 tx=0x7f325400dc30 comp rx=0 tx=0).stop 2023-03-09T15:00:28.011 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.991+0000 7f325f54e700 1 -- 172.21.15.38:0/2116953909 shutdown_connections 2023-03-09T15:00:28.011 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.991+0000 7f325f54e700 1 --2- 172.21.15.38:0/2116953909 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f324409f4a0 0x7f32440a1950 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:00:28.011 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.991+0000 7f325f54e700 1 --2- 172.21.15.38:0/2116953909 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f3258104040 0x7f32581998e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:00:28.011 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.991+0000 7f325f54e700 1 --2- 172.21.15.38:0/2116953909 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f32580ff000 0x7f3258194f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:00:28.011 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.991+0000 7f325f54e700 1 --2- 172.21.15.38:0/2116953909 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f32580ff9a0 0x7f3258195480 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:00:28.011 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.991+0000 7f325f54e700 1 -- 172.21.15.38:0/2116953909 >> 172.21.15.38:0/2116953909 conn(0x7f3258075c60 msgr2=0x7f32581029a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T15:00:28.012 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.991+0000 7f325f54e700 1 -- 172.21.15.38:0/2116953909 shutdown_connections 2023-03-09T15:00:28.012 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:27.991+0000 7f325f54e700 1 -- 172.21.15.38:0/2116953909 wait complete. 2023-03-09T15:00:28.012 INFO:tasks.cephfs.filesystem:scrub status for tag:a14659f4-d7ea-4c5a-b15b-6138af3f909f - {'path': '/', 'tag': 'a14659f4-d7ea-4c5a-b15b-6138af3f909f', 'options': 'recursive,force'} 2023-03-09T15:00:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:00:28 smithi038 ceph-mon[92422]: pgmap v592: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 9.3 KiB/s wr, 8 op/s 2023-03-09T15:00:28.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:00:28 smithi122 ceph-mon[93869]: pgmap v592: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 9.3 KiB/s wr, 8 op/s 2023-03-09T15:00:28.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:00:28 smithi181 ceph-mon[94189]: pgmap v592: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 9.3 KiB/s wr, 8 op/s 2023-03-09T15:00:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 600.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:00:30.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:00:30 smithi122 ceph-mon[93869]: pgmap v593: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 25 KiB/s wr, 8 op/s 2023-03-09T15:00:30.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:00:30 smithi181 ceph-mon[94189]: pgmap v593: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 25 KiB/s wr, 8 op/s 2023-03-09T15:00:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:00:30 smithi038 ceph-mon[92422]: pgmap v593: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 25 KiB/s wr, 8 op/s 2023-03-09T15:00:32.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:00:32 smithi122 ceph-mon[93869]: pgmap v594: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 26 KiB/s wr, 9 op/s 2023-03-09T15:00:32.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:00:32 smithi181 ceph-mon[94189]: pgmap v594: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 26 KiB/s wr, 9 op/s 2023-03-09T15:00:32.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:00:32 smithi038 ceph-mon[92422]: pgmap v594: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 26 KiB/s wr, 9 op/s 2023-03-09T15:00:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 605.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:00:34.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:00:33 smithi122 ceph-mon[93869]: pgmap v595: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 57 KiB/s wr, 10 op/s 2023-03-09T15:00:34.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:00:33 smithi181 ceph-mon[94189]: pgmap v595: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 57 KiB/s wr, 10 op/s 2023-03-09T15:00:34.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:00:33 smithi038 ceph-mon[92422]: pgmap v595: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 57 KiB/s wr, 10 op/s 2023-03-09T15:00:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:00:36 smithi122 ceph-mon[93869]: pgmap v596: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 48 KiB/s wr, 9 op/s 2023-03-09T15:00:36.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:00:36 smithi181 ceph-mon[94189]: pgmap v596: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 48 KiB/s wr, 9 op/s 2023-03-09T15:00:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:00:36 smithi038 ceph-mon[92422]: pgmap v596: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 48 KiB/s wr, 9 op/s 2023-03-09T15:00:38.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:00:38 smithi122 ceph-mon[93869]: pgmap v597: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 50 KiB/s wr, 11 op/s 2023-03-09T15:00:38.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:00:38 smithi181 ceph-mon[94189]: pgmap v597: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 50 KiB/s wr, 11 op/s 2023-03-09T15:00:38.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:00:38 smithi038 ceph-mon[92422]: pgmap v597: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 50 KiB/s wr, 11 op/s 2023-03-09T15:00:38.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 610.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:00:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:00:40 smithi122 ceph-mon[93869]: pgmap v598: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 57 KiB/s wr, 9 op/s 2023-03-09T15:00:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:00:40 smithi181 ceph-mon[94189]: pgmap v598: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 57 KiB/s wr, 9 op/s 2023-03-09T15:00:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:00:40 smithi038 ceph-mon[92422]: pgmap v598: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 57 KiB/s wr, 9 op/s 2023-03-09T15:00:42.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:00:42 smithi122 ceph-mon[93869]: pgmap v599: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 8 op/s 2023-03-09T15:00:42.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:00:42 smithi181 ceph-mon[94189]: pgmap v599: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 8 op/s 2023-03-09T15:00:42.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:00:42 smithi038 ceph-mon[92422]: pgmap v599: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 8 op/s 2023-03-09T15:00:43.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 615.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:00:44.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:00:43 smithi122 ceph-mon[93869]: pgmap v600: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 49 KiB/s wr, 10 op/s 2023-03-09T15:00:44.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:00:43 smithi181 ceph-mon[94189]: pgmap v600: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 49 KiB/s wr, 10 op/s 2023-03-09T15:00:44.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:00:43 smithi038 ceph-mon[92422]: pgmap v600: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 49 KiB/s wr, 10 op/s 2023-03-09T15:00:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:00:46 smithi122 ceph-mon[93869]: pgmap v601: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 21 KiB/s wr, 6 op/s 2023-03-09T15:00:46.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:00:46 smithi181 ceph-mon[94189]: pgmap v601: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 21 KiB/s wr, 6 op/s 2023-03-09T15:00:46.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:00:46 smithi038 ceph-mon[92422]: pgmap v601: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 21 KiB/s wr, 6 op/s 2023-03-09T15:00:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:00:48 smithi122 ceph-mon[93869]: pgmap v602: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 21 KiB/s wr, 8 op/s 2023-03-09T15:00:48.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:00:48 smithi181 ceph-mon[94189]: pgmap v602: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 21 KiB/s wr, 8 op/s 2023-03-09T15:00:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:00:48 smithi038 ceph-mon[92422]: pgmap v602: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 21 KiB/s wr, 8 op/s 2023-03-09T15:00:48.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 620.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:00:50.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:00:50 smithi122 ceph-mon[93869]: pgmap v603: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T15:00:50.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:00:50 smithi181 ceph-mon[94189]: pgmap v603: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T15:00:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:00:50 smithi038 ceph-mon[92422]: pgmap v603: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T15:00:51.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:00:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:00:51.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:00:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:00:51.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:00:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:00:51.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:00:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:00:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:00:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:00:51.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:00:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:00:51.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:00:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:00:51.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:00:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:00:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:00:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:00:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:00:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:00:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:00:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:00:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:00:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:00:52.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:00:52 smithi122 ceph-mon[93869]: pgmap v604: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 5 op/s 2023-03-09T15:00:52.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:00:52 smithi181 ceph-mon[94189]: pgmap v604: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 5 op/s 2023-03-09T15:00:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:00:52 smithi038 ceph-mon[92422]: pgmap v604: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 5 op/s 2023-03-09T15:00:53.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 625.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:00:54.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:00:53 smithi122 ceph-mon[93869]: pgmap v605: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 8 op/s 2023-03-09T15:00:54.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:00:53 smithi181 ceph-mon[94189]: pgmap v605: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 8 op/s 2023-03-09T15:00:54.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:00:53 smithi038 ceph-mon[92422]: pgmap v605: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 8 op/s 2023-03-09T15:00:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:00:56 smithi122 ceph-mon[93869]: pgmap v606: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 4.7 KiB/s wr, 5 op/s 2023-03-09T15:00:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:00:56 smithi181 ceph-mon[94189]: pgmap v606: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 4.7 KiB/s wr, 5 op/s 2023-03-09T15:00:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:00:56 smithi038 ceph-mon[92422]: pgmap v606: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 4.7 KiB/s wr, 5 op/s 2023-03-09T15:00:58.013 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-03-09T15:00:58.159 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.155+0000 7f69c43d4700 1 -- 172.21.15.38:0/2855153117 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69bc100410 msgr2=0x7f69bc1007e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T15:00:58.159 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.155+0000 7f69c43d4700 1 --2- 172.21.15.38:0/2855153117 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69bc100410 0x7f69bc1007e0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f69b8009990 tx=0x7f69b8009ca0 comp rx=0 tx=0).stop 2023-03-09T15:00:58.159 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.155+0000 7f69c43d4700 1 -- 172.21.15.38:0/2855153117 shutdown_connections 2023-03-09T15:00:58.160 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.155+0000 7f69c43d4700 1 --2- 172.21.15.38:0/2855153117 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69bc100410 0x7f69bc1007e0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:00:58.160 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.155+0000 7f69c43d4700 1 --2- 172.21.15.38:0/2855153117 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69bc1099c0 0x7f69bc10bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:00:58.160 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.155+0000 7f69c43d4700 1 --2- 172.21.15.38:0/2855153117 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69bc100d20 0x7f69bc109480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:00:58.160 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.155+0000 7f69c43d4700 1 -- 172.21.15.38:0/2855153117 >> 172.21.15.38:0/2855153117 conn(0x7f69bc0748f0 msgr2=0x7f69bc074cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T15:00:58.160 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.155+0000 7f69c43d4700 1 -- 172.21.15.38:0/2855153117 shutdown_connections 2023-03-09T15:00:58.160 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.155+0000 7f69c43d4700 1 -- 172.21.15.38:0/2855153117 wait complete. 2023-03-09T15:00:58.161 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.156+0000 7f69c43d4700 1 Processor -- start 2023-03-09T15:00:58.161 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.156+0000 7f69c43d4700 1 -- start start 2023-03-09T15:00:58.161 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.156+0000 7f69c43d4700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69bc100410 0x7f69bc195210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T15:00:58.161 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.156+0000 7f69c43d4700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69bc100d20 0x7f69bc195750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T15:00:58.161 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.156+0000 7f69c43d4700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69bc1099c0 0x7f69bc199bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T15:00:58.161 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.156+0000 7f69c43d4700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69bc19a1f0 con 0x7f69bc100d20 2023-03-09T15:00:58.161 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.156+0000 7f69c43d4700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69bc19a360 con 0x7f69bc1099c0 2023-03-09T15:00:58.162 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.156+0000 7f69c43d4700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69bc19a520 con 0x7f69bc100410 2023-03-09T15:00:58.162 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.156+0000 7f69c196f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69bc100d20 0x7f69bc195750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T15:00:58.162 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.156+0000 7f69c196f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69bc100d20 0x7f69bc195750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42388/0 (socket says 172.21.15.38:42388) 2023-03-09T15:00:58.162 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.156+0000 7f69c196f700 1 -- 172.21.15.38:0/3431037089 learned_addr learned my addr 172.21.15.38:0/3431037089 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T15:00:58.162 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.156+0000 7f69c2170700 1 --2- 172.21.15.38:0/3431037089 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69bc100410 0x7f69bc195210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T15:00:58.162 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.156+0000 7f69c2971700 1 --2- 172.21.15.38:0/3431037089 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69bc1099c0 0x7f69bc199bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T15:00:58.162 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.157+0000 7f69c196f700 1 -- 172.21.15.38:0/3431037089 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69bc100410 msgr2=0x7f69bc195210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T15:00:58.163 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.157+0000 7f69c196f700 1 --2- 172.21.15.38:0/3431037089 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69bc100410 0x7f69bc195210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:00:58.163 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.157+0000 7f69c196f700 1 -- 172.21.15.38:0/3431037089 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69bc1099c0 msgr2=0x7f69bc199bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T15:00:58.163 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.157+0000 7f69c196f700 1 --2- 172.21.15.38:0/3431037089 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69bc1099c0 0x7f69bc199bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:00:58.163 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.157+0000 7f69c196f700 1 -- 172.21.15.38:0/3431037089 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69b80095f0 con 0x7f69bc100d20 2023-03-09T15:00:58.163 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.157+0000 7f69c2971700 1 --2- 172.21.15.38:0/3431037089 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69bc1099c0 0x7f69bc199bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T15:00:58.163 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.157+0000 7f69c196f700 1 --2- 172.21.15.38:0/3431037089 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69bc100d20 0x7f69bc195750 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f69ac00c970 tx=0x7f69ac00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T15:00:58.164 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.157+0000 7f69b37fe700 1 -- 172.21.15.38:0/3431037089 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69ac0078d0 con 0x7f69bc100d20 2023-03-09T15:00:58.164 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.157+0000 7f69b37fe700 1 -- 172.21.15.38:0/3431037089 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f69ac018480 con 0x7f69bc100d20 2023-03-09T15:00:58.164 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.157+0000 7f69c43d4700 1 -- 172.21.15.38:0/3431037089 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69bc10f040 con 0x7f69bc100d20 2023-03-09T15:00:58.164 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.157+0000 7f69b37fe700 1 -- 172.21.15.38:0/3431037089 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69ac00f690 con 0x7f69bc100d20 2023-03-09T15:00:58.164 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.157+0000 7f69c43d4700 1 -- 172.21.15.38:0/3431037089 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69bc10eeb0 con 0x7f69bc100d20 2023-03-09T15:00:58.169 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.159+0000 7f69b37fe700 1 -- 172.21.15.38:0/3431037089 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 25) v1 ==== 133295+0+0 (secure 0 0 0) 0x7f69ac0188d0 con 0x7f69bc100d20 2023-03-09T15:00:58.170 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.160+0000 7f69b37fe700 1 --2- 172.21.15.38:0/3431037089 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f69a809f520 0x7f69a80a19d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T15:00:58.170 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.160+0000 7f69c2170700 1 --2- 172.21.15.38:0/3431037089 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f69a809f520 0x7f69a80a19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T15:00:58.170 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.160+0000 7f69b37fe700 1 -- 172.21.15.38:0/3431037089 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 9561+0+0 (secure 0 0 0) 0x7f69ac013070 con 0x7f69bc100d20 2023-03-09T15:00:58.170 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.160+0000 7f69c2170700 1 --2- 172.21.15.38:0/3431037089 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f69a809f520 0x7f69a80a19d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f69b800b2e0 tx=0x7f69b8005fd0 comp rx=0 tx=0).ready entity=mgr.15558 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T15:00:58.170 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.161+0000 7f69d055db80 1 Processor -- start 2023-03-09T15:00:58.171 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.161+0000 7f69d055db80 1 -- start start 2023-03-09T15:00:58.171 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.161+0000 7f69d055db80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559512b9da40 0x559512b9de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T15:00:58.171 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.161+0000 7f69d055db80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559512b9e350 0x559512b97cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T15:00:58.171 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.161+0000 7f69d055db80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559512b981f0 0x559512b98620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T15:00:58.171 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.161+0000 7f69d055db80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x559512a7fda0 con 0x559512b9da40 2023-03-09T15:00:58.171 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.161+0000 7f69d055db80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x559512aee780 con 0x559512b981f0 2023-03-09T15:00:58.171 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.161+0000 7f69d055db80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x559512ae7cb0 con 0x559512b9e350 2023-03-09T15:00:58.172 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.161+0000 7f69c2170700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559512b9da40 0x559512b9de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T15:00:58.172 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.161+0000 7f69c2170700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559512b9da40 0x559512b9de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42394/0 (socket says 172.21.15.38:42394) 2023-03-09T15:00:58.172 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.161+0000 7f69c2170700 1 -- 172.21.15.38:0/624674430 learned_addr learned my addr 172.21.15.38:0/624674430 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T15:00:58.172 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.161+0000 7f69c2971700 1 --2- 172.21.15.38:0/624674430 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559512b981f0 0x559512b98620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T15:00:58.172 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.161+0000 7f69c196f700 1 --2- 172.21.15.38:0/624674430 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559512b9e350 0x559512b97cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T15:00:58.172 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.161+0000 7f69c2170700 1 -- 172.21.15.38:0/624674430 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559512b9e350 msgr2=0x559512b97cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T15:00:58.172 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.161+0000 7f69c2170700 1 --2- 172.21.15.38:0/624674430 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559512b9e350 0x559512b97cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:00:58.173 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.161+0000 7f69c2170700 1 -- 172.21.15.38:0/624674430 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559512b981f0 msgr2=0x559512b98620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T15:00:58.173 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.161+0000 7f69c2170700 1 --2- 172.21.15.38:0/624674430 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559512b981f0 0x559512b98620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:00:58.173 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.162+0000 7f69c2170700 1 -- 172.21.15.38:0/624674430 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559512aae190 con 0x559512b9da40 2023-03-09T15:00:58.173 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.162+0000 7f69c2170700 1 --2- 172.21.15.38:0/624674430 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559512b9da40 0x559512b9de10 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f69b8015000 tx=0x7f69b8020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=932cc09f7183af1d server_cookie=0 in_seq=0 out_seq=0 2023-03-09T15:00:58.173 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.162+0000 7f69b0daf700 1 -- 172.21.15.38:0/624674430 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b802b070 con 0x559512b9da40 2023-03-09T15:00:58.173 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.162+0000 7f69b0daf700 1 -- 172.21.15.38:0/624674430 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f69b801e040 con 0x559512b9da40 2023-03-09T15:00:58.173 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.162+0000 7f69b0daf700 1 -- 172.21.15.38:0/624674430 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b800f7b0 con 0x559512b9da40 2023-03-09T15:00:58.174 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.162+0000 7f69d055db80 1 -- 172.21.15.38:0/624674430 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559512b9da40 msgr2=0x559512b9de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T15:00:58.174 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.162+0000 7f69d055db80 1 --2- 172.21.15.38:0/624674430 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559512b9da40 0x559512b9de10 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f69b8015000 tx=0x7f69b8020000 comp rx=0 tx=0).stop 2023-03-09T15:00:58.174 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.163+0000 7f69d055db80 1 -- 172.21.15.38:0/624674430 shutdown_connections 2023-03-09T15:00:58.174 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.163+0000 7f69d055db80 1 --2- 172.21.15.38:0/624674430 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559512b9da40 0x559512b9de10 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:00:58.174 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.163+0000 7f69d055db80 1 --2- 172.21.15.38:0/624674430 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559512b981f0 0x559512b98620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:00:58.174 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.163+0000 7f69d055db80 1 --2- 172.21.15.38:0/624674430 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559512b9e350 0x559512b97cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:00:58.174 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.163+0000 7f69d055db80 1 -- 172.21.15.38:0/624674430 >> 172.21.15.38:0/624674430 conn(0x559512b90910 msgr2=0x559512b90ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T15:00:58.175 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.163+0000 7f69d055db80 1 -- 172.21.15.38:0/624674430 shutdown_connections 2023-03-09T15:00:58.175 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.163+0000 7f69d055db80 1 -- 172.21.15.38:0/624674430 wait complete. 2023-03-09T15:00:58.175 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.163+0000 7f69d055db80 1 -- start start 2023-03-09T15:00:58.175 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.163+0000 7f69d055db80 1 Processor -- start 2023-03-09T15:00:58.175 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.164+0000 7f69d055db80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559512b981f0 0x559512ba42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T15:00:58.175 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.164+0000 7f69d055db80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559512b9da40 0x559512ba1520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T15:00:58.175 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.164+0000 7f69d055db80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559512b9e350 0x559512ba1b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T15:00:58.176 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.164+0000 7f69d055db80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x55951296b920 con 0x559512b981f0 2023-03-09T15:00:58.176 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.164+0000 7f69d055db80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x559512ba2290 con 0x559512b9da40 2023-03-09T15:00:58.176 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.164+0000 7f69d055db80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x559512baddf0 con 0x559512b9e350 2023-03-09T15:00:58.176 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.164+0000 7f69d055db80 10 client.0 _ll_register_callbacks cb 0x559512a8d520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T15:00:58.176 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.164+0000 7f69c2170700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559512b981f0 0x559512ba42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T15:00:58.176 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.164+0000 7f69c2170700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559512b981f0 0x559512ba42a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42398/0 (socket says 172.21.15.38:42398) 2023-03-09T15:00:58.176 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.164+0000 7f69c2170700 1 -- 172.21.15.38:0/336119972 learned_addr learned my addr 172.21.15.38:0/336119972 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T15:00:58.177 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.164+0000 7f69c196f700 1 --2- 172.21.15.38:0/336119972 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559512b9da40 0x559512ba1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T15:00:58.177 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.164+0000 7f69c2170700 1 -- 172.21.15.38:0/336119972 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559512b9e350 msgr2=0x559512ba1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T15:00:58.177 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.164+0000 7f69c2971700 1 --2- 172.21.15.38:0/336119972 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559512b9e350 0x559512ba1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T15:00:58.177 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.164+0000 7f69c2170700 1 --2- 172.21.15.38:0/336119972 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559512b9e350 0x559512ba1b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:00:58.177 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.164+0000 7f69c2170700 1 -- 172.21.15.38:0/336119972 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559512b9da40 msgr2=0x559512ba1520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T15:00:58.178 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.164+0000 7f69c2170700 1 --2- 172.21.15.38:0/336119972 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559512b9da40 0x559512ba1520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:00:58.178 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.164+0000 7f69c2170700 1 -- 172.21.15.38:0/336119972 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69bc10c690 con 0x559512b981f0 2023-03-09T15:00:58.178 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.164+0000 7f69c196f700 1 --2- 172.21.15.38:0/336119972 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559512b9da40 0x559512ba1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T15:00:58.178 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.164+0000 7f69c2170700 1 --2- 172.21.15.38:0/336119972 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559512b981f0 0x559512ba42a0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f69b80098e0 tx=0x7f69b8004cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90e547b1b5f8edfa server_cookie=0 in_seq=0 out_seq=0 2023-03-09T15:00:58.178 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.164+0000 7f69a5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2023-03-09T15:00:58.178 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.164+0000 7f69a5ffb700 1 -- 172.21.15.38:0/336119972 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b802b040 con 0x559512b981f0 2023-03-09T15:00:58.178 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.164+0000 7f69a5ffb700 1 -- 172.21.15.38:0/336119972 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(253..253 src has 1..253) v4 ==== 9561+0+0 (secure 0 0 0) 0x7f69b800f870 con 0x559512b981f0 2023-03-09T15:00:58.179 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.164+0000 7f69d055db80 1 -- 172.21.15.38:0/336119972 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559512ab5190 con 0x559512b981f0 2023-03-09T15:00:58.179 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.165+0000 7f69a5ffb700 1 -- 172.21.15.38:0/336119972 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) v2 ==== 24+0+0 (secure 0 0 0) 0x7f69b800fa30 con 0x559512b981f0 2023-03-09T15:00:58.179 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.165+0000 7f69d055db80 10 client.16401 fetch_fsmap learned FSMap version 45 2023-03-09T15:00:58.179 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.165+0000 7f69d055db80 1 -- 172.21.15.38:0/336119972 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) v3 -- 0x5595128fec60 con 0x559512b981f0 2023-03-09T15:00:58.179 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.165+0000 7f69a5ffb700 1 -- 172.21.15.38:0/336119972 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x559512ab5190 con 0x559512b981f0 2023-03-09T15:00:58.179 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.165+0000 7f69a5ffb700 1 -- 172.21.15.38:0/336119972 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 45) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f69b8024bb0 con 0x559512b981f0 2023-03-09T15:00:58.179 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.165+0000 7f69d055db80 10 client.16401 fetch_fsmap finished waiting for FSMap version 45 2023-03-09T15:00:58.180 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.165+0000 7f69d055db80 10 client.16401 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T15:00:58.180 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.165+0000 7f69d055db80 20 client.16401 populate_metadata read hostname 'smithi038' 2023-03-09T15:00:58.180 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.165+0000 7f69d055db80 1 --2- 172.21.15.38:0/336119972 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x559512bb1ae0 0x559512bb3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T15:00:58.180 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.165+0000 7f69d055db80 4 client.16401 mds_command: new command op to 24530 tid=0[{"prefix": "get_command_descriptions"}] 2023-03-09T15:00:58.180 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.165+0000 7f69d055db80 1 -- 172.21.15.38:0/336119972 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55951296bf60 con 0x559512bb1ae0 2023-03-09T15:00:58.180 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.165+0000 7f69c196f700 1 --2- 172.21.15.38:0/336119972 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x559512bb1ae0 0x559512bb3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T15:00:58.180 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.166+0000 7f69a5ffb700 10 client.16401 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T15:00:58.181 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.166+0000 7f69c196f700 1 --2- 172.21.15.38:0/336119972 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x559512bb1ae0 0x559512bb3ec0 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d33de3bb3b702fdf server_cookie=b6f228acccd311db in_seq=0 out_seq=0 2023-03-09T15:00:58.181 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.166+0000 7f69a5ffb700 1 -- 172.21.15.38:0/336119972 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18278 (crc 0 0 0) 0x55951296bf60 con 0x559512bb1ae0 2023-03-09T15:00:58.181 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.166+0000 7f69a5ffb700 10 client.16401 handle_command_reply: tid=0 2023-03-09T15:00:58.181 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.166+0000 7f69d055db80 1 client.16401 shutdown 2023-03-09T15:00:58.181 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.166+0000 7f69d055db80 1 -- 172.21.15.38:0/336119972 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559512b981f0 msgr2=0x559512ba42a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T15:00:58.181 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.166+0000 7f69d055db80 1 --2- 172.21.15.38:0/336119972 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559512b981f0 0x559512ba42a0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f69b80098e0 tx=0x7f69b8004cd0 comp rx=0 tx=0).stop 2023-03-09T15:00:58.181 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.166+0000 7f69d055db80 1 -- 172.21.15.38:0/336119972 shutdown_connections 2023-03-09T15:00:58.182 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.166+0000 7f69d055db80 1 --2- 172.21.15.38:0/336119972 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x559512bb1ae0 0x559512bb3ec0 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:00:58.182 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.166+0000 7f69d055db80 1 --2- 172.21.15.38:0/336119972 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559512b981f0 0x559512ba42a0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:00:58.182 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.166+0000 7f69a5ffb700 0 client.16401 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T15:00:58.182 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.166+0000 7f69d055db80 1 --2- 172.21.15.38:0/336119972 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559512b9e350 0x559512ba1b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:00:58.182 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.166+0000 7f69d055db80 1 --2- 172.21.15.38:0/336119972 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559512b9da40 0x559512ba1520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:00:58.182 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.166+0000 7f69d055db80 1 -- 172.21.15.38:0/336119972 >> 172.21.15.38:0/336119972 conn(0x559512b90910 msgr2=0x559512b90ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T15:00:58.182 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.166+0000 7f69d055db80 1 -- 172.21.15.38:0/336119972 shutdown_connections 2023-03-09T15:00:58.183 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.166+0000 7f69d055db80 1 -- 172.21.15.38:0/336119972 wait complete. 2023-03-09T15:00:58.183 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.167+0000 7f69d055db80 20 client.16401 trim_cache size 0 max 16384 2023-03-09T15:00:58.186 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.184+0000 7f69d055db80 1 Processor -- start 2023-03-09T15:00:58.186 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.184+0000 7f69d055db80 1 -- start start 2023-03-09T15:00:58.187 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.184+0000 7f69d055db80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559512b981f0 0x559512bad4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T15:00:58.187 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.184+0000 7f69d055db80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559512b9da40 0x559512b9b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T15:00:58.187 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.184+0000 7f69d055db80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559512b9e350 0x559512ba3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T15:00:58.187 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.184+0000 7f69d055db80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x559512b921b0 con 0x559512b9da40 2023-03-09T15:00:58.187 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.184+0000 7f69d055db80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x559512ba24a0 con 0x559512b9e350 2023-03-09T15:00:58.187 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.184+0000 7f69d055db80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x559512b90570 con 0x559512b981f0 2023-03-09T15:00:58.188 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.184+0000 7f69c196f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559512b9da40 0x559512b9b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T15:00:58.188 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.184+0000 7f69c2971700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559512b9e350 0x559512ba3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T15:00:58.188 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.184+0000 7f69c196f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559512b9da40 0x559512b9b660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42408/0 (socket says 172.21.15.38:42408) 2023-03-09T15:00:58.188 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.184+0000 7f69c196f700 1 -- 172.21.15.38:0/81001380 learned_addr learned my addr 172.21.15.38:0/81001380 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T15:00:58.188 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.184+0000 7f69c2170700 1 --2- 172.21.15.38:0/81001380 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559512b981f0 0x559512bad4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T15:00:58.188 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.184+0000 7f69c2971700 1 -- 172.21.15.38:0/81001380 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559512b981f0 msgr2=0x559512bad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T15:00:58.188 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.184+0000 7f69c2971700 1 --2- 172.21.15.38:0/81001380 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559512b981f0 0x559512bad4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:00:58.189 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.184+0000 7f69c2971700 1 -- 172.21.15.38:0/81001380 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559512b9da40 msgr2=0x559512b9b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T15:00:58.189 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.184+0000 7f69c2971700 1 --2- 172.21.15.38:0/81001380 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559512b9da40 0x559512b9b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:00:58.189 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.184+0000 7f69c2971700 1 -- 172.21.15.38:0/81001380 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559512915de0 con 0x559512b9e350 2023-03-09T15:00:58.189 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.184+0000 7f69c196f700 1 --2- 172.21.15.38:0/81001380 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559512b9da40 0x559512b9b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T15:00:58.189 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.184+0000 7f69c2971700 1 --2- 172.21.15.38:0/81001380 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559512b9e350 0x559512ba3e40 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f69bc073110 tx=0x7f69b4009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0811ad2760de887 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T15:00:58.189 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.185+0000 7f69b15b0700 1 -- 172.21.15.38:0/81001380 <== mon.2 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b4004db0 con 0x559512b9e350 2023-03-09T15:00:58.189 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.185+0000 7f69b15b0700 1 -- 172.21.15.38:0/81001380 <== mon.2 v2:172.21.15.122:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7f69b400f040 con 0x559512b9e350 2023-03-09T15:00:58.190 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.185+0000 7f69b15b0700 1 -- 172.21.15.38:0/81001380 <== mon.2 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b400e4f0 con 0x559512b9e350 2023-03-09T15:00:58.193 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.185+0000 7f69d055db80 1 -- 172.21.15.38:0/81001380 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559512b9e350 msgr2=0x559512ba3e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T15:00:58.193 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.185+0000 7f69d055db80 1 --2- 172.21.15.38:0/81001380 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559512b9e350 0x559512ba3e40 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f69bc073110 tx=0x7f69b4009920 comp rx=0 tx=0).stop 2023-03-09T15:00:58.193 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.185+0000 7f69d055db80 1 -- 172.21.15.38:0/81001380 shutdown_connections 2023-03-09T15:00:58.194 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.185+0000 7f69d055db80 1 --2- 172.21.15.38:0/81001380 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559512b9e350 0x559512ba3e40 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:00:58.194 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.185+0000 7f69d055db80 1 --2- 172.21.15.38:0/81001380 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559512b981f0 0x559512bad4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:00:58.194 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.185+0000 7f69d055db80 1 --2- 172.21.15.38:0/81001380 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559512b9da40 0x559512b9b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:00:58.194 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.185+0000 7f69d055db80 1 -- 172.21.15.38:0/81001380 >> 172.21.15.38:0/81001380 conn(0x559512b90910 msgr2=0x559512bae150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T15:00:58.194 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.185+0000 7f69d055db80 1 -- 172.21.15.38:0/81001380 shutdown_connections 2023-03-09T15:00:58.194 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.185+0000 7f69d055db80 1 -- 172.21.15.38:0/81001380 wait complete. 2023-03-09T15:00:58.194 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.186+0000 7f69d055db80 1 -- start start 2023-03-09T15:00:58.195 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.186+0000 7f69d055db80 1 Processor -- start 2023-03-09T15:00:58.195 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.186+0000 7f69d055db80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559512b981f0 0x559512ba8f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T15:00:58.195 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.186+0000 7f69d055db80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559512b9da40 0x559512b998b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T15:00:58.195 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.186+0000 7f69d055db80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559512b9e350 0x559512b99ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T15:00:58.195 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.186+0000 7f69d055db80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x559512b9a730 con 0x559512b9da40 2023-03-09T15:00:58.195 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.186+0000 7f69d055db80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x559512c058f0 con 0x559512b981f0 2023-03-09T15:00:58.195 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.186+0000 7f69d055db80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x559512c05a30 con 0x559512b9e350 2023-03-09T15:00:58.196 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.186+0000 7f69d055db80 10 client.0 _ll_register_callbacks cb 0x559512a8d520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T15:00:58.196 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.186+0000 7f69c196f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559512b9da40 0x559512b998b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T15:00:58.196 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.186+0000 7f69c2170700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559512b981f0 0x559512ba8f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T15:00:58.196 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.186+0000 7f69c196f700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559512b9da40 0x559512b998b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42414/0 (socket says 172.21.15.38:42414) 2023-03-09T15:00:58.196 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.186+0000 7f69c196f700 1 -- 172.21.15.38:0/1967967113 learned_addr learned my addr 172.21.15.38:0/1967967113 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T15:00:58.196 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.186+0000 7f69c2971700 1 --2- 172.21.15.38:0/1967967113 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559512b9e350 0x559512b99ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T15:00:58.196 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.186+0000 7f69c196f700 1 -- 172.21.15.38:0/1967967113 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559512b9e350 msgr2=0x559512b99ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T15:00:58.197 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.186+0000 7f69c196f700 1 --2- 172.21.15.38:0/1967967113 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559512b9e350 0x559512b99ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:00:58.197 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.186+0000 7f69c196f700 1 -- 172.21.15.38:0/1967967113 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559512b981f0 msgr2=0x559512ba8f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T15:00:58.197 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.186+0000 7f69c196f700 1 --2- 172.21.15.38:0/1967967113 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559512b981f0 0x559512ba8f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:00:58.197 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.186+0000 7f69c196f700 1 -- 172.21.15.38:0/1967967113 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69bc10eeb0 con 0x559512b9da40 2023-03-09T15:00:58.197 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.186+0000 7f69c2971700 1 --2- 172.21.15.38:0/1967967113 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559512b9e350 0x559512b99ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T15:00:58.197 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.187+0000 7f69c196f700 1 --2- 172.21.15.38:0/1967967113 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559512b9da40 0x559512b998b0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f69ac0947a0 tx=0x7f69ac09c1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e80781ec69ab5401 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T15:00:58.197 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.187+0000 7f69a5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2023-03-09T15:00:58.198 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.187+0000 7f69a5ffb700 1 -- 172.21.15.38:0/1967967113 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69ac0c7430 con 0x559512b9da40 2023-03-09T15:00:58.198 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.187+0000 7f69a5ffb700 1 -- 172.21.15.38:0/1967967113 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(253..253 src has 1..253) v4 ==== 9561+0+0 (secure 0 0 0) 0x7f69ac0b0040 con 0x559512b9da40 2023-03-09T15:00:58.198 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.187+0000 7f69d055db80 1 -- 172.21.15.38:0/1967967113 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559512aa51b0 con 0x559512b9da40 2023-03-09T15:00:58.198 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.187+0000 7f69a5ffb700 1 -- 172.21.15.38:0/1967967113 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) v2 ==== 24+0+0 (secure 0 0 0) 0x7f69ac0c7590 con 0x559512b9da40 2023-03-09T15:00:58.198 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.187+0000 7f69d055db80 10 client.16407 fetch_fsmap learned FSMap version 45 2023-03-09T15:00:58.198 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.187+0000 7f69d055db80 1 -- 172.21.15.38:0/1967967113 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) v3 -- 0x7f69b4003af0 con 0x559512b9da40 2023-03-09T15:00:58.198 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.187+0000 7f69a5ffb700 1 -- 172.21.15.38:0/1967967113 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x559512aa51b0 con 0x559512b9da40 2023-03-09T15:00:58.199 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.187+0000 7f69a5ffb700 1 -- 172.21.15.38:0/1967967113 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 45) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f69ac0b6020 con 0x559512b9da40 2023-03-09T15:00:58.199 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.187+0000 7f69d055db80 10 client.16407 fetch_fsmap finished waiting for FSMap version 45 2023-03-09T15:00:58.199 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.187+0000 7f69d055db80 10 client.16407 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T15:00:58.199 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.187+0000 7f69d055db80 20 client.16407 populate_metadata read hostname 'smithi038' 2023-03-09T15:00:58.199 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.187+0000 7f69d055db80 1 --2- 172.21.15.38:0/1967967113 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x559512c07180 0x559512c09590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T15:00:58.199 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.187+0000 7f69d055db80 4 client.16407 mds_command: new command op to 24530 tid=0[{"prefix": "scrub status"}] 2023-03-09T15:00:58.199 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.187+0000 7f69d055db80 1 -- 172.21.15.38:0/1967967113 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x559512a8f1c0 con 0x559512c07180 2023-03-09T15:00:58.200 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.187+0000 7f69c2170700 1 --2- 172.21.15.38:0/1967967113 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x559512c07180 0x559512c09590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T15:00:58.200 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.188+0000 7f69a5ffb700 10 client.16407 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T15:00:58.200 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.188+0000 7f69c2170700 1 --2- 172.21.15.38:0/1967967113 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x559512c07180 0x559512c09590 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c28117d84899b7e server_cookie=75e8dd003e6b756 in_seq=0 out_seq=0 2023-03-09T15:00:58.200 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.188+0000 7f69a5ffb700 1 -- 172.21.15.38:0/1967967113 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+265 (crc 0 0 0) 0x559512a8f1c0 con 0x559512c07180 2023-03-09T15:00:58.200 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.188+0000 7f69a5ffb700 10 client.16407 handle_command_reply: tid=0 2023-03-09T15:00:58.200 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.188+0000 7f69d055db80 1 client.16407 shutdown 2023-03-09T15:00:58.200 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.188+0000 7f69d055db80 1 -- 172.21.15.38:0/1967967113 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559512b9da40 msgr2=0x559512b998b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T15:00:58.201 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.188+0000 7f69d055db80 1 --2- 172.21.15.38:0/1967967113 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559512b9da40 0x559512b998b0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f69ac0947a0 tx=0x7f69ac09c1f0 comp rx=0 tx=0).stop 2023-03-09T15:00:58.201 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.188+0000 7f69d055db80 1 -- 172.21.15.38:0/1967967113 shutdown_connections 2023-03-09T15:00:58.201 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.188+0000 7f69d055db80 1 --2- 172.21.15.38:0/1967967113 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x559512c07180 0x559512c09590 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:00:58.201 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.188+0000 7f69d055db80 1 --2- 172.21.15.38:0/1967967113 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x559512b9e350 0x559512b99ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:00:58.201 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.188+0000 7f69a5ffb700 0 client.16407 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T15:00:58.201 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.188+0000 7f69d055db80 1 --2- 172.21.15.38:0/1967967113 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x559512b981f0 0x559512ba8f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:00:58.201 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.188+0000 7f69d055db80 1 --2- 172.21.15.38:0/1967967113 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x559512b9da40 0x559512b998b0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:00:58.202 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.188+0000 7f69d055db80 1 -- 172.21.15.38:0/1967967113 >> 172.21.15.38:0/1967967113 conn(0x559512b90910 msgr2=0x559512bad4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T15:00:58.202 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.188+0000 7f69d055db80 1 -- 172.21.15.38:0/1967967113 shutdown_connections 2023-03-09T15:00:58.202 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.188+0000 7f69d055db80 1 -- 172.21.15.38:0/1967967113 wait complete. 2023-03-09T15:00:58.202 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.188+0000 7f69d055db80 20 client.16407 trim_cache size 0 max 16384 2023-03-09T15:00:58.202 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2023-03-09T15:00:58.203 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "scrub active (1 inodes in the stack)", 2023-03-09T15:00:58.203 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": { 2023-03-09T15:00:58.203 INFO:teuthology.orchestra.run.smithi038.stdout: "a14659f4-d7ea-4c5a-b15b-6138af3f909f": { 2023-03-09T15:00:58.203 INFO:teuthology.orchestra.run.smithi038.stdout: "path": "/", 2023-03-09T15:00:58.203 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "a14659f4-d7ea-4c5a-b15b-6138af3f909f", 2023-03-09T15:00:58.203 INFO:teuthology.orchestra.run.smithi038.stdout: "options": "recursive,force" 2023-03-09T15:00:58.204 INFO:teuthology.orchestra.run.smithi038.stdout: } 2023-03-09T15:00:58.204 INFO:teuthology.orchestra.run.smithi038.stdout: } 2023-03-09T15:00:58.204 INFO:teuthology.orchestra.run.smithi038.stdout:} 2023-03-09T15:00:58.205 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.189+0000 7f69c43d4700 1 -- 172.21.15.38:0/3431037089 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f69a809f520 msgr2=0x7f69a80a19d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T15:00:58.205 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.189+0000 7f69c43d4700 1 --2- 172.21.15.38:0/3431037089 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f69a809f520 0x7f69a80a19d0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f69b800b2e0 tx=0x7f69b8005fd0 comp rx=0 tx=0).stop 2023-03-09T15:00:58.205 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.189+0000 7f69c43d4700 1 -- 172.21.15.38:0/3431037089 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69bc100d20 msgr2=0x7f69bc195750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T15:00:58.206 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.189+0000 7f69c43d4700 1 --2- 172.21.15.38:0/3431037089 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69bc100d20 0x7f69bc195750 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f69ac00c970 tx=0x7f69ac00cc80 comp rx=0 tx=0).stop 2023-03-09T15:00:58.206 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.189+0000 7f69c43d4700 1 -- 172.21.15.38:0/3431037089 shutdown_connections 2023-03-09T15:00:58.206 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.189+0000 7f69c43d4700 1 --2- 172.21.15.38:0/3431037089 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7f69a809f520 0x7f69a80a19d0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:00:58.206 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.189+0000 7f69c43d4700 1 --2- 172.21.15.38:0/3431037089 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7f69bc1099c0 0x7f69bc199bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:00:58.206 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.189+0000 7f69c43d4700 1 --2- 172.21.15.38:0/3431037089 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7f69bc100410 0x7f69bc195210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:00:58.206 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.189+0000 7f69c43d4700 1 --2- 172.21.15.38:0/3431037089 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7f69bc100d20 0x7f69bc195750 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:00:58.207 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.189+0000 7f69c43d4700 1 -- 172.21.15.38:0/3431037089 >> 172.21.15.38:0/3431037089 conn(0x7f69bc0748f0 msgr2=0x7f69bc0fd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T15:00:58.207 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.189+0000 7f69c43d4700 1 -- 172.21.15.38:0/3431037089 shutdown_connections 2023-03-09T15:00:58.207 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:00:58.189+0000 7f69c43d4700 1 -- 172.21.15.38:0/3431037089 wait complete. 2023-03-09T15:00:58.207 INFO:tasks.cephfs.filesystem:scrub status for tag:a14659f4-d7ea-4c5a-b15b-6138af3f909f - {'path': '/', 'tag': 'a14659f4-d7ea-4c5a-b15b-6138af3f909f', 'options': 'recursive,force'} 2023-03-09T15:00:58.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:00:58 smithi122 ceph-mon[93869]: pgmap v607: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.5 KiB/s wr, 7 op/s 2023-03-09T15:00:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:00:58 smithi181 ceph-mon[94189]: pgmap v607: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.5 KiB/s wr, 7 op/s 2023-03-09T15:00:58.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:00:58 smithi038 ceph-mon[92422]: pgmap v607: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.5 KiB/s wr, 7 op/s 2023-03-09T15:00:58.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 630.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:01:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:01:00 smithi122 ceph-mon[93869]: pgmap v608: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 17 KiB/s wr, 7 op/s 2023-03-09T15:01:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:01:00 smithi181 ceph-mon[94189]: pgmap v608: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 17 KiB/s wr, 7 op/s 2023-03-09T15:01:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:01:00 smithi038 ceph-mon[92422]: pgmap v608: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 17 KiB/s wr, 7 op/s 2023-03-09T15:01:02.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:01:02 smithi122 ceph-mon[93869]: pgmap v609: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 7 op/s 2023-03-09T15:01:02.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:01:02 smithi181 ceph-mon[94189]: pgmap v609: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 7 op/s 2023-03-09T15:01:02.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:01:02 smithi038 ceph-mon[92422]: pgmap v609: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 7 op/s 2023-03-09T15:01:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 635.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:01:04.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:01:03 smithi122 ceph-mon[93869]: pgmap v610: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 52 KiB/s wr, 13 op/s 2023-03-09T15:01:04.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:01:03 smithi181 ceph-mon[94189]: pgmap v610: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 52 KiB/s wr, 13 op/s 2023-03-09T15:01:04.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:01:03 smithi038 ceph-mon[92422]: pgmap v610: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 52 KiB/s wr, 13 op/s 2023-03-09T15:01:05.231 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:01:04 smithi122 ceph-mon[93869]: osdmap e254: 12 total, 12 up, 12 in 2023-03-09T15:01:05.236 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:01:04 smithi181 ceph-mon[94189]: osdmap e254: 12 total, 12 up, 12 in 2023-03-09T15:01:05.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:01:04 smithi038 ceph-mon[92422]: osdmap e254: 12 total, 12 up, 12 in 2023-03-09T15:01:06.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:01:05 smithi181 ceph-mon[94189]: pgmap v612: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 88 KiB/s wr, 27 op/s 2023-03-09T15:01:06.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:01:05 smithi038 ceph-mon[92422]: pgmap v612: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 88 KiB/s wr, 27 op/s 2023-03-09T15:01:06.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:01:05 smithi122 ceph-mon[93869]: pgmap v612: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 88 KiB/s wr, 27 op/s 2023-03-09T15:01:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:01:08 smithi122 ceph-mon[93869]: pgmap v613: 129 pgs: 8 active+clean+snaptrim, 21 active+clean+snaptrim_wait, 100 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 85 KiB/s wr, 27 op/s 2023-03-09T15:01:08.487 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:01:08 smithi181 ceph-mon[94189]: pgmap v613: 129 pgs: 8 active+clean+snaptrim, 21 active+clean+snaptrim_wait, 100 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 85 KiB/s wr, 27 op/s 2023-03-09T15:01:08.503 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:01:08 smithi038 ceph-mon[92422]: pgmap v613: 129 pgs: 8 active+clean+snaptrim, 21 active+clean+snaptrim_wait, 100 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 85 KiB/s wr, 27 op/s 2023-03-09T15:01:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 640.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:01:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:01:10 smithi122 ceph-mon[93869]: pgmap v614: 129 pgs: 14 active+clean+snaptrim, 35 active+clean+snaptrim_wait, 80 active+clean; 11 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 75 KiB/s wr, 26 op/s 2023-03-09T15:01:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:01:10 smithi122 ceph-mon[93869]: 2 slow requests, 2 included below; oldest blocked for > 180.934694 secs 2023-03-09T15:01:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:01:10 smithi122 ceph-mon[93869]: slow request 180.934694 seconds old, received at 2023-03-09T14:58:08.564667+0000: peer_request:mds.2:599 currently acquired locks 2023-03-09T15:01:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:01:10 smithi122 ceph-mon[93869]: slow request 180.923993 seconds old, received at 2023-03-09T14:58:08.575367+0000: peer_request(client.15375:6773.0 authpin) currently dispatched 2023-03-09T15:01:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:01:10 smithi181 ceph-mon[94189]: pgmap v614: 129 pgs: 14 active+clean+snaptrim, 35 active+clean+snaptrim_wait, 80 active+clean; 11 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 75 KiB/s wr, 26 op/s 2023-03-09T15:01:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:01:10 smithi181 ceph-mon[94189]: 2 slow requests, 2 included below; oldest blocked for > 180.934694 secs 2023-03-09T15:01:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:01:10 smithi181 ceph-mon[94189]: slow request 180.934694 seconds old, received at 2023-03-09T14:58:08.564667+0000: peer_request:mds.2:599 currently acquired locks 2023-03-09T15:01:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:01:10 smithi181 ceph-mon[94189]: slow request 180.923993 seconds old, received at 2023-03-09T14:58:08.575367+0000: peer_request(client.15375:6773.0 authpin) currently dispatched 2023-03-09T15:01:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:01:10 smithi038 ceph-mon[92422]: pgmap v614: 129 pgs: 14 active+clean+snaptrim, 35 active+clean+snaptrim_wait, 80 active+clean; 11 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 75 KiB/s wr, 26 op/s 2023-03-09T15:01:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:01:10 smithi038 ceph-mon[92422]: 2 slow requests, 2 included below; oldest blocked for > 180.934694 secs 2023-03-09T15:01:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:01:10 smithi038 ceph-mon[92422]: slow request 180.934694 seconds old, received at 2023-03-09T14:58:08.564667+0000: peer_request:mds.2:599 currently acquired locks 2023-03-09T15:01:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:01:10 smithi038 ceph-mon[92422]: slow request 180.923993 seconds old, received at 2023-03-09T14:58:08.575367+0000: peer_request(client.15375:6773.0 authpin) currently dispatched 2023-03-09T15:01:11.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:01:11 smithi122 ceph-mon[93869]: Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:01:11.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:01:11 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] up:active 2023-03-09T15:01:11.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:01:11 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:01:11.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:01:11 smithi181 ceph-mon[94189]: Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:01:11.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:01:11 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] up:active 2023-03-09T15:01:11.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:01:11 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:01:11.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:01:11 smithi038 ceph-mon[92422]: Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:01:11.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:01:11 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] up:active 2023-03-09T15:01:11.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:01:11 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:01:12.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:01:12 smithi122 ceph-mon[93869]: pgmap v615: 129 pgs: 9 active+clean+snaptrim, 21 active+clean+snaptrim_wait, 99 active+clean; 11 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 74 KiB/s wr, 26 op/s 2023-03-09T15:01:12.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:01:12 smithi122 ceph-mon[93869]: 2 slow requests, 2 included below; oldest blocked for > 183.122690 secs 2023-03-09T15:01:12.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:01:12 smithi122 ceph-mon[93869]: slow request 183.122690 seconds old, received at 2023-03-09T14:58:08.560137+0000: client_request(mds.2:599 rename #0x10000000007/0000000100000001000000EB #0x60f/1000000062a caller_uid=0, caller_gid=0{}) currently acquired locks 2023-03-09T15:01:12.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:01:12 smithi122 ceph-mon[93869]: slow request 183.108310 seconds old, received at 2023-03-09T14:58:08.574517+0000: client_request(client.15375:6773 unlink #0x10000000007/0000000100000001000000EB 2023-03-09T14:58:08.569481+0000 caller_uid=26, caller_gid=26{26,}) currently requesting remote authpins 2023-03-09T15:01:12.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:01:12 smithi181 ceph-mon[94189]: pgmap v615: 129 pgs: 9 active+clean+snaptrim, 21 active+clean+snaptrim_wait, 99 active+clean; 11 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 74 KiB/s wr, 26 op/s 2023-03-09T15:01:12.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:01:12 smithi181 ceph-mon[94189]: 2 slow requests, 2 included below; oldest blocked for > 183.122690 secs 2023-03-09T15:01:12.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:01:12 smithi181 ceph-mon[94189]: slow request 183.122690 seconds old, received at 2023-03-09T14:58:08.560137+0000: client_request(mds.2:599 rename #0x10000000007/0000000100000001000000EB #0x60f/1000000062a caller_uid=0, caller_gid=0{}) currently acquired locks 2023-03-09T15:01:12.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:01:12 smithi181 ceph-mon[94189]: slow request 183.108310 seconds old, received at 2023-03-09T14:58:08.574517+0000: client_request(client.15375:6773 unlink #0x10000000007/0000000100000001000000EB 2023-03-09T14:58:08.569481+0000 caller_uid=26, caller_gid=26{26,}) currently requesting remote authpins 2023-03-09T15:01:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:01:12 smithi038 ceph-mon[92422]: pgmap v615: 129 pgs: 9 active+clean+snaptrim, 21 active+clean+snaptrim_wait, 99 active+clean; 11 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 74 KiB/s wr, 26 op/s 2023-03-09T15:01:12.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:01:12 smithi038 ceph-mon[92422]: 2 slow requests, 2 included below; oldest blocked for > 183.122690 secs 2023-03-09T15:01:12.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:01:12 smithi038 ceph-mon[92422]: slow request 183.122690 seconds old, received at 2023-03-09T14:58:08.560137+0000: client_request(mds.2:599 rename #0x10000000007/0000000100000001000000EB #0x60f/1000000062a caller_uid=0, caller_gid=0{}) currently acquired locks 2023-03-09T15:01:12.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:01:12 smithi038 ceph-mon[92422]: slow request 183.108310 seconds old, received at 2023-03-09T14:58:08.574517+0000: client_request(client.15375:6773 unlink #0x10000000007/0000000100000001000000EB 2023-03-09T14:58:08.569481+0000 caller_uid=26, caller_gid=26{26,}) currently requesting remote authpins 2023-03-09T15:01:13.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 645.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:01:14.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:01:13 smithi122 ceph-mon[93869]: pgmap v616: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 39 KiB/s wr, 22 op/s 2023-03-09T15:01:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:01:13 smithi181 ceph-mon[94189]: pgmap v616: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 39 KiB/s wr, 22 op/s 2023-03-09T15:01:14.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:01:13 smithi038 ceph-mon[92422]: pgmap v616: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 39 KiB/s wr, 22 op/s 2023-03-09T15:01:15.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:01:14 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 185.934772 secs 2023-03-09T15:01:15.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:01:14 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 185.934772 secs 2023-03-09T15:01:15.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:01:14 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 185.934772 secs 2023-03-09T15:01:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:01:15 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:01:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:01:15 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:01:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:01:15 smithi122 ceph-mon[93869]: pgmap v617: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 7 op/s 2023-03-09T15:01:16.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:01:15 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:01:16.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:01:15 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:01:16.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:01:15 smithi181 ceph-mon[94189]: pgmap v617: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 7 op/s 2023-03-09T15:01:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:01:15 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:01:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:01:15 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:01:16.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:01:15 smithi038 ceph-mon[92422]: pgmap v617: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 7 op/s 2023-03-09T15:01:17.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:01:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 188.122785 secs 2023-03-09T15:01:17.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:01:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 188.122785 secs 2023-03-09T15:01:17.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:01:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 188.122785 secs 2023-03-09T15:01:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:01:18 smithi122 ceph-mon[93869]: pgmap v618: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 8 op/s 2023-03-09T15:01:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:01:18 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:01:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:01:18 smithi122 ceph-mon[93869]: osdmap e255: 12 total, 12 up, 12 in 2023-03-09T15:01:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:01:18 smithi181 ceph-mon[94189]: pgmap v618: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 8 op/s 2023-03-09T15:01:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:01:18 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:01:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:01:18 smithi181 ceph-mon[94189]: osdmap e255: 12 total, 12 up, 12 in 2023-03-09T15:01:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:01:18 smithi038 ceph-mon[92422]: pgmap v618: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 8 op/s 2023-03-09T15:01:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:01:18 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:01:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:01:18 smithi038 ceph-mon[92422]: osdmap e255: 12 total, 12 up, 12 in 2023-03-09T15:01:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 650.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:01:20.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:01:20 smithi122 ceph-mon[93869]: pgmap v620: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T15:01:20.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:01:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 190.934833 secs 2023-03-09T15:01:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:01:20 smithi181 ceph-mon[94189]: pgmap v620: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T15:01:20.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:01:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 190.934833 secs 2023-03-09T15:01:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:01:20 smithi038 ceph-mon[92422]: pgmap v620: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T15:01:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:01:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 190.934833 secs 2023-03-09T15:01:22.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:01:22 smithi122 ceph-mon[93869]: pgmap v621: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T15:01:22.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:01:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 193.122902 secs 2023-03-09T15:01:22.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:01:22 smithi181 ceph-mon[94189]: pgmap v621: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T15:01:22.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:01:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 193.122902 secs 2023-03-09T15:01:22.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:01:22 smithi038 ceph-mon[92422]: pgmap v621: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T15:01:22.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:01:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 193.122902 secs 2023-03-09T15:01:23.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 655.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:01:24.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:01:23 smithi122 ceph-mon[93869]: pgmap v622: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 6 op/s 2023-03-09T15:01:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:01:23 smithi181 ceph-mon[94189]: pgmap v622: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 6 op/s 2023-03-09T15:01:24.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:01:23 smithi038 ceph-mon[92422]: pgmap v622: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 6 op/s 2023-03-09T15:01:25.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:01:24 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 195.934896 secs 2023-03-09T15:01:25.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:01:24 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 195.934896 secs 2023-03-09T15:01:25.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:01:24 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 195.934896 secs 2023-03-09T15:01:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:01:25 smithi122 ceph-mon[93869]: pgmap v623: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 6 op/s 2023-03-09T15:01:26.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:01:25 smithi181 ceph-mon[94189]: pgmap v623: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 6 op/s 2023-03-09T15:01:26.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:01:25 smithi038 ceph-mon[92422]: pgmap v623: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 6 op/s 2023-03-09T15:01:27.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:01:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 198.123020 secs 2023-03-09T15:01:27.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:01:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 198.123020 secs 2023-03-09T15:01:27.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:01:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 198.123020 secs 2023-03-09T15:01:28.208 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-03-09T15:01:28.369 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.366+0000 7fc9af20a700 1 -- 172.21.15.38:0/2470887752 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc9a80ff000 msgr2=0x7fc9a80ff3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T15:01:28.369 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.366+0000 7fc9af20a700 1 --2- 172.21.15.38:0/2470887752 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc9a80ff000 0x7fc9a80ff3d0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fc99c0099c0 tx=0x7fc99c009cd0 comp rx=0 tx=0).stop 2023-03-09T15:01:28.369 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.366+0000 7fc9af20a700 1 -- 172.21.15.38:0/2470887752 shutdown_connections 2023-03-09T15:01:28.369 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.366+0000 7fc9af20a700 1 --2- 172.21.15.38:0/2470887752 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc9a80ff000 0x7fc9a80ff3d0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:01:28.370 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.366+0000 7fc9af20a700 1 --2- 172.21.15.38:0/2470887752 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc9a8104040 0x7fc9a810bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:01:28.370 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.366+0000 7fc9af20a700 1 --2- 172.21.15.38:0/2470887752 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc9a80ff9a0 0x7fc9a8103960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:01:28.370 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.366+0000 7fc9af20a700 1 -- 172.21.15.38:0/2470887752 >> 172.21.15.38:0/2470887752 conn(0x7fc9a8075c60 msgr2=0x7fc9a8076060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T15:01:28.370 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.366+0000 7fc9af20a700 1 -- 172.21.15.38:0/2470887752 shutdown_connections 2023-03-09T15:01:28.370 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.366+0000 7fc9af20a700 1 -- 172.21.15.38:0/2470887752 wait complete. 2023-03-09T15:01:28.370 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.366+0000 7fc9af20a700 1 Processor -- start 2023-03-09T15:01:28.370 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.366+0000 7fc9af20a700 1 -- start start 2023-03-09T15:01:28.372 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.367+0000 7fc9af20a700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc9a80ff000 0x7fc9a8196dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T15:01:28.372 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.367+0000 7fc9af20a700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc9a80ff9a0 0x7fc9a8197310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T15:01:28.372 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.367+0000 7fc9af20a700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc9a8104040 0x7fc9a81a1e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T15:01:28.372 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.367+0000 7fc9af20a700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a8197c50 con 0x7fc9a80ff000 2023-03-09T15:01:28.372 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.367+0000 7fc9af20a700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a81a2390 con 0x7fc9a80ff9a0 2023-03-09T15:01:28.372 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.367+0000 7fc9af20a700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9a81a2550 con 0x7fc9a8104040 2023-03-09T15:01:28.373 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.367+0000 7fc9acfa6700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc9a80ff000 0x7fc9a8196dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T15:01:28.373 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.367+0000 7fc9acfa6700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc9a80ff000 0x7fc9a8196dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42450/0 (socket says 172.21.15.38:42450) 2023-03-09T15:01:28.373 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.367+0000 7fc9acfa6700 1 -- 172.21.15.38:0/2707832422 learned_addr learned my addr 172.21.15.38:0/2707832422 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T15:01:28.373 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.367+0000 7fc9a7fff700 1 --2- 172.21.15.38:0/2707832422 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc9a80ff9a0 0x7fc9a8197310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T15:01:28.373 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.367+0000 7fc9ad7a7700 1 --2- 172.21.15.38:0/2707832422 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc9a8104040 0x7fc9a81a1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T15:01:28.373 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.367+0000 7fc9acfa6700 1 -- 172.21.15.38:0/2707832422 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc9a8104040 msgr2=0x7fc9a81a1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T15:01:28.373 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.367+0000 7fc9acfa6700 1 --2- 172.21.15.38:0/2707832422 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc9a8104040 0x7fc9a81a1e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:01:28.374 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.367+0000 7fc9acfa6700 1 -- 172.21.15.38:0/2707832422 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc9a80ff9a0 msgr2=0x7fc9a8197310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T15:01:28.374 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.367+0000 7fc9acfa6700 1 --2- 172.21.15.38:0/2707832422 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc9a80ff9a0 0x7fc9a8197310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:01:28.374 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.367+0000 7fc9acfa6700 1 -- 172.21.15.38:0/2707832422 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc99c0095f0 con 0x7fc9a80ff000 2023-03-09T15:01:28.374 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.367+0000 7fc9a7fff700 1 --2- 172.21.15.38:0/2707832422 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc9a80ff9a0 0x7fc9a8197310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T15:01:28.374 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.367+0000 7fc9acfa6700 1 --2- 172.21.15.38:0/2707832422 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc9a80ff000 0x7fc9a8196dd0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fc99c00b580 tx=0x7fc99c00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T15:01:28.374 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.367+0000 7fc9a5ffb700 1 -- 172.21.15.38:0/2707832422 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc99c01d070 con 0x7fc9a80ff000 2023-03-09T15:01:28.374 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.367+0000 7fc9a5ffb700 1 -- 172.21.15.38:0/2707832422 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc99c005050 con 0x7fc9a80ff000 2023-03-09T15:01:28.374 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.367+0000 7fc9af20a700 1 -- 172.21.15.38:0/2707832422 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9a810ee50 con 0x7fc9a80ff000 2023-03-09T15:01:28.375 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.367+0000 7fc9a5ffb700 1 -- 172.21.15.38:0/2707832422 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc99c004a40 con 0x7fc9a80ff000 2023-03-09T15:01:28.375 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.368+0000 7fc9af20a700 1 -- 172.21.15.38:0/2707832422 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9a810ecc0 con 0x7fc9a80ff000 2023-03-09T15:01:28.379 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.370+0000 7fc9a5ffb700 1 -- 172.21.15.38:0/2707832422 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 25) v1 ==== 133295+0+0 (secure 0 0 0) 0x7fc99c028030 con 0x7fc9a80ff000 2023-03-09T15:01:28.379 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.370+0000 7fc9a5ffb700 1 --2- 172.21.15.38:0/2707832422 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fc99809f4a0 0x7fc9980a1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T15:01:28.379 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.371+0000 7fc9a5ffb700 1 -- 172.21.15.38:0/2707832422 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(255..255 src has 1..255) v4 ==== 9577+0+0 (secure 0 0 0) 0x7fc99c0cb0d0 con 0x7fc9a80ff000 2023-03-09T15:01:28.379 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.371+0000 7fc9a7fff700 1 --2- 172.21.15.38:0/2707832422 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fc99809f4a0 0x7fc9980a1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T15:01:28.379 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.371+0000 7fc9a7fff700 1 --2- 172.21.15.38:0/2707832422 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fc99809f4a0 0x7fc9980a1950 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc9a8068310 tx=0x7fc99400a380 comp rx=0 tx=0).ready entity=mgr.15558 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T15:01:28.379 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.371+0000 7fc9bb393b80 1 Processor -- start 2023-03-09T15:01:28.379 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.371+0000 7fc9bb393b80 1 -- start start 2023-03-09T15:01:28.380 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.371+0000 7fc9bb393b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564d0bfdd830 0x564d0bfddc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T15:01:28.380 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.371+0000 7fc9bb393b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d0bfde140 0x564d0bfd7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T15:01:28.380 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.371+0000 7fc9bb393b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564d0bfd7fe0 0x564d0bfd8410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T15:01:28.380 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.371+0000 7fc9bb393b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d0bec1da0 con 0x564d0bfde140 2023-03-09T15:01:28.380 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.371+0000 7fc9bb393b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d0bf30780 con 0x564d0bfdd830 2023-03-09T15:01:28.380 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.371+0000 7fc9bb393b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d0bf29cb0 con 0x564d0bfd7fe0 2023-03-09T15:01:28.380 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.372+0000 7fc9a7fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d0bfde140 0x564d0bfd7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T15:01:28.381 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.372+0000 7fc9acfa6700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564d0bfdd830 0x564d0bfddc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T15:01:28.381 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.372+0000 7fc9a7fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d0bfde140 0x564d0bfd7aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42460/0 (socket says 172.21.15.38:42460) 2023-03-09T15:01:28.381 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.372+0000 7fc9acfa6700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564d0bfdd830 0x564d0bfddc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.38:51648/0 (socket says 172.21.15.38:51648) 2023-03-09T15:01:28.381 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.372+0000 7fc9a7fff700 1 -- 172.21.15.38:0/2133636347 learned_addr learned my addr 172.21.15.38:0/2133636347 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T15:01:28.381 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.372+0000 7fc9ad7a7700 1 --2- 172.21.15.38:0/2133636347 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564d0bfd7fe0 0x564d0bfd8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T15:01:28.381 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.372+0000 7fc9acfa6700 1 -- 172.21.15.38:0/2133636347 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564d0bfd7fe0 msgr2=0x564d0bfd8410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T15:01:28.381 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.372+0000 7fc9acfa6700 1 --2- 172.21.15.38:0/2133636347 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564d0bfd7fe0 0x564d0bfd8410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:01:28.381 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.372+0000 7fc9acfa6700 1 -- 172.21.15.38:0/2133636347 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d0bfde140 msgr2=0x564d0bfd7aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T15:01:28.382 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.372+0000 7fc9acfa6700 1 --2- 172.21.15.38:0/2133636347 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d0bfde140 0x564d0bfd7aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:01:28.382 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.372+0000 7fc9acfa6700 1 -- 172.21.15.38:0/2133636347 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564d0bef0190 con 0x564d0bfdd830 2023-03-09T15:01:28.382 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.372+0000 7fc9a7fff700 1 --2- 172.21.15.38:0/2133636347 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d0bfde140 0x564d0bfd7aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T15:01:28.382 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.372+0000 7fc9acfa6700 1 --2- 172.21.15.38:0/2133636347 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564d0bfdd830 0x564d0bfddc00 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fc99c098c40 tx=0x7fc99c098d30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=22898935dcf8b0b0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T15:01:28.382 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.372+0000 7fc992ffd700 1 -- 172.21.15.38:0/2133636347 <== mon.2 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc99c097050 con 0x564d0bfdd830 2023-03-09T15:01:28.382 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.372+0000 7fc992ffd700 1 -- 172.21.15.38:0/2133636347 <== mon.2 v2:172.21.15.122:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc99c004c00 con 0x564d0bfdd830 2023-03-09T15:01:28.382 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.372+0000 7fc992ffd700 1 -- 172.21.15.38:0/2133636347 <== mon.2 v2:172.21.15.122:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc99c0983c0 con 0x564d0bfdd830 2023-03-09T15:01:28.383 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.373+0000 7fc9bb393b80 1 -- 172.21.15.38:0/2133636347 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564d0bfdd830 msgr2=0x564d0bfddc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T15:01:28.383 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.373+0000 7fc9bb393b80 1 --2- 172.21.15.38:0/2133636347 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564d0bfdd830 0x564d0bfddc00 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fc99c098c40 tx=0x7fc99c098d30 comp rx=0 tx=0).stop 2023-03-09T15:01:28.383 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.373+0000 7fc9bb393b80 1 -- 172.21.15.38:0/2133636347 shutdown_connections 2023-03-09T15:01:28.383 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.373+0000 7fc9bb393b80 1 --2- 172.21.15.38:0/2133636347 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564d0bfd7fe0 0x564d0bfd8410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:01:28.383 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.373+0000 7fc9bb393b80 1 --2- 172.21.15.38:0/2133636347 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564d0bfdd830 0x564d0bfddc00 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:01:28.383 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.373+0000 7fc9bb393b80 1 --2- 172.21.15.38:0/2133636347 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d0bfde140 0x564d0bfd7aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:01:28.383 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.373+0000 7fc9bb393b80 1 -- 172.21.15.38:0/2133636347 >> 172.21.15.38:0/2133636347 conn(0x564d0bfd2a20 msgr2=0x564d0bfd2df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T15:01:28.383 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.373+0000 7fc9bb393b80 1 -- 172.21.15.38:0/2133636347 shutdown_connections 2023-03-09T15:01:28.384 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.373+0000 7fc9bb393b80 1 -- 172.21.15.38:0/2133636347 wait complete. 2023-03-09T15:01:28.384 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.373+0000 7fc9bb393b80 1 -- start start 2023-03-09T15:01:28.384 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.374+0000 7fc9bb393b80 1 Processor -- start 2023-03-09T15:01:28.384 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.374+0000 7fc9bb393b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564d0bfd7fe0 0x564d0bfe40b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T15:01:28.384 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.374+0000 7fc9bb393b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564d0bfdd830 0x564d0bfe1330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T15:01:28.384 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.374+0000 7fc9bb393b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d0bfde140 0x564d0bfe1960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T15:01:28.384 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.374+0000 7fc9bb393b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d0bdad920 con 0x564d0bfde140 2023-03-09T15:01:28.384 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.374+0000 7fc9bb393b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d0bfe20d0 con 0x564d0bfd7fe0 2023-03-09T15:01:28.385 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.374+0000 7fc9bb393b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d0bfed930 con 0x564d0bfdd830 2023-03-09T15:01:28.385 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.374+0000 7fc9bb393b80 10 client.0 _ll_register_callbacks cb 0x564d0becf520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T15:01:28.385 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.374+0000 7fc9ad7a7700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d0bfde140 0x564d0bfe1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T15:01:28.385 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.374+0000 7fc9acfa6700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564d0bfd7fe0 0x564d0bfe40b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T15:01:28.385 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.374+0000 7fc9ad7a7700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d0bfde140 0x564d0bfe1960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42468/0 (socket says 172.21.15.38:42468) 2023-03-09T15:01:28.385 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.374+0000 7fc9ad7a7700 1 -- 172.21.15.38:0/1955838068 learned_addr learned my addr 172.21.15.38:0/1955838068 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T15:01:28.385 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.374+0000 7fc9a7fff700 1 --2- 172.21.15.38:0/1955838068 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564d0bfdd830 0x564d0bfe1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T15:01:28.386 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.374+0000 7fc9acfa6700 1 -- 172.21.15.38:0/1955838068 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564d0bfdd830 msgr2=0x564d0bfe1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T15:01:28.386 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.374+0000 7fc9acfa6700 1 --2- 172.21.15.38:0/1955838068 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564d0bfdd830 0x564d0bfe1330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:01:28.386 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.374+0000 7fc9acfa6700 1 -- 172.21.15.38:0/1955838068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d0bfde140 msgr2=0x564d0bfe1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T15:01:28.386 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.374+0000 7fc9acfa6700 1 --2- 172.21.15.38:0/1955838068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d0bfde140 0x564d0bfe1960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:01:28.386 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.374+0000 7fc9acfa6700 1 -- 172.21.15.38:0/1955838068 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc99400fe30 con 0x564d0bfd7fe0 2023-03-09T15:01:28.386 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.374+0000 7fc9a7fff700 1 --2- 172.21.15.38:0/1955838068 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564d0bfdd830 0x564d0bfe1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-09T15:01:28.386 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.374+0000 7fc9ad7a7700 1 --2- 172.21.15.38:0/1955838068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d0bfde140 0x564d0bfe1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T15:01:28.386 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.374+0000 7fc9acfa6700 1 --2- 172.21.15.38:0/1955838068 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564d0bfd7fe0 0x564d0bfe40b0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fc99c009fd0 tx=0x7fc99c097050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf228b0c0d26ba6d server_cookie=0 in_seq=0 out_seq=0 2023-03-09T15:01:28.387 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.374+0000 7fc987fff700 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2023-03-09T15:01:28.387 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.375+0000 7fc987fff700 1 -- 172.21.15.38:0/1955838068 <== mon.2 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc99c098f10 con 0x564d0bfd7fe0 2023-03-09T15:01:28.387 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.375+0000 7fc987fff700 1 -- 172.21.15.38:0/1955838068 <== mon.2 v2:172.21.15.122:3300/0 2 ==== osd_map(255..255 src has 1..255) v4 ==== 9577+0+0 (secure 0 0 0) 0x7fc99c099070 con 0x564d0bfd7fe0 2023-03-09T15:01:28.387 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.375+0000 7fc9bb393b80 1 -- 172.21.15.38:0/1955838068 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564d0bef7190 con 0x564d0bfd7fe0 2023-03-09T15:01:28.387 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.375+0000 7fc987fff700 1 -- 172.21.15.38:0/1955838068 <== mon.2 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc99c0a0980 con 0x564d0bfd7fe0 2023-03-09T15:01:28.387 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.375+0000 7fc9bb393b80 10 client.26054 fetch_fsmap learned FSMap version 47 2023-03-09T15:01:28.387 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.375+0000 7fc9bb393b80 1 -- 172.21.15.38:0/1955838068 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x7fc9a810ecc0 con 0x564d0bfd7fe0 2023-03-09T15:01:28.387 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.375+0000 7fc987fff700 1 -- 172.21.15.38:0/1955838068 <== mon.2 v2:172.21.15.122:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc99c07a070 con 0x564d0bfd7fe0 2023-03-09T15:01:28.388 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.375+0000 7fc987fff700 1 -- 172.21.15.38:0/1955838068 <== mon.2 v2:172.21.15.122:3300/0 5 ==== fsmap(e 47) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fc99c0c8c70 con 0x564d0bfd7fe0 2023-03-09T15:01:28.388 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.375+0000 7fc9bb393b80 10 client.26054 fetch_fsmap finished waiting for FSMap version 47 2023-03-09T15:01:28.388 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.375+0000 7fc9bb393b80 10 client.26054 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T15:01:28.388 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.375+0000 7fc9bb393b80 20 client.26054 populate_metadata read hostname 'smithi038' 2023-03-09T15:01:28.388 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.375+0000 7fc9bb393b80 1 --2- 172.21.15.38:0/1955838068 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x564d0bff1510 0x564d0bff38f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T15:01:28.388 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.375+0000 7fc9bb393b80 4 client.26054 mds_command: new command op to 24530 tid=0[{"prefix": "get_command_descriptions"}] 2023-03-09T15:01:28.388 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.375+0000 7fc9bb393b80 1 -- 172.21.15.38:0/1955838068 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564d0bdadf60 con 0x564d0bff1510 2023-03-09T15:01:28.389 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.375+0000 7fc9a7fff700 1 --2- 172.21.15.38:0/1955838068 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x564d0bff1510 0x564d0bff38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T15:01:28.389 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.376+0000 7fc9a7fff700 1 --2- 172.21.15.38:0/1955838068 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x564d0bff1510 0x564d0bff38f0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=626436b5b8f85e22 server_cookie=dc0b493461bc78e0 in_seq=0 out_seq=0 2023-03-09T15:01:28.389 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.376+0000 7fc987fff700 10 client.26054 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T15:01:28.390 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.376+0000 7fc987fff700 1 -- 172.21.15.38:0/1955838068 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18278 (crc 0 0 0) 0x564d0bdadf60 con 0x564d0bff1510 2023-03-09T15:01:28.390 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.376+0000 7fc987fff700 10 client.26054 handle_command_reply: tid=0 2023-03-09T15:01:28.390 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.377+0000 7fc9bb393b80 1 client.26054 shutdown 2023-03-09T15:01:28.390 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.377+0000 7fc9bb393b80 1 -- 172.21.15.38:0/1955838068 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564d0bfd7fe0 msgr2=0x564d0bfe40b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T15:01:28.391 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.377+0000 7fc9bb393b80 1 --2- 172.21.15.38:0/1955838068 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564d0bfd7fe0 0x564d0bfe40b0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fc99c009fd0 tx=0x7fc99c097050 comp rx=0 tx=0).stop 2023-03-09T15:01:28.391 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.377+0000 7fc9bb393b80 1 -- 172.21.15.38:0/1955838068 shutdown_connections 2023-03-09T15:01:28.391 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.377+0000 7fc9bb393b80 1 --2- 172.21.15.38:0/1955838068 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x564d0bff1510 0x564d0bff38f0 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:01:28.391 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.377+0000 7fc9bb393b80 1 --2- 172.21.15.38:0/1955838068 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d0bfde140 0x564d0bfe1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:01:28.391 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.377+0000 7fc987fff700 0 client.26054 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T15:01:28.391 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.377+0000 7fc9bb393b80 1 --2- 172.21.15.38:0/1955838068 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564d0bfd7fe0 0x564d0bfe40b0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:01:28.391 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.377+0000 7fc9bb393b80 1 --2- 172.21.15.38:0/1955838068 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564d0bfdd830 0x564d0bfe1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:01:28.392 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.377+0000 7fc9bb393b80 1 -- 172.21.15.38:0/1955838068 >> 172.21.15.38:0/1955838068 conn(0x564d0bfd2a20 msgr2=0x564d0bfd1c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T15:01:28.392 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.377+0000 7fc9bb393b80 1 -- 172.21.15.38:0/1955838068 shutdown_connections 2023-03-09T15:01:28.392 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.377+0000 7fc9bb393b80 1 -- 172.21.15.38:0/1955838068 wait complete. 2023-03-09T15:01:28.392 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.377+0000 7fc9bb393b80 20 client.26054 trim_cache size 0 max 16384 2023-03-09T15:01:28.397 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.394+0000 7fc9bb393b80 1 Processor -- start 2023-03-09T15:01:28.397 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.394+0000 7fc9bb393b80 1 -- start start 2023-03-09T15:01:28.398 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.394+0000 7fc9bb393b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564d0bfd7fe0 0x564d0bfed090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T15:01:28.398 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.394+0000 7fc9bb393b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564d0bfdd830 0x564d0bfedcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T15:01:28.398 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.394+0000 7fc9bb393b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d0bfde140 0x564d0bfe3c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T15:01:28.398 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.394+0000 7fc9bb393b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d0bfcee00 con 0x564d0bfde140 2023-03-09T15:01:28.398 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.394+0000 7fc9bb393b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d0bfd1680 con 0x564d0bfd7fe0 2023-03-09T15:01:28.398 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.394+0000 7fc9bb393b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d0bfd8cc0 con 0x564d0bfdd830 2023-03-09T15:01:28.399 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.394+0000 7fc9ad7a7700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d0bfde140 0x564d0bfe3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T15:01:28.399 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.394+0000 7fc9ad7a7700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d0bfde140 0x564d0bfe3c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42476/0 (socket says 172.21.15.38:42476) 2023-03-09T15:01:28.399 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.394+0000 7fc9ad7a7700 1 -- 172.21.15.38:0/45613975 learned_addr learned my addr 172.21.15.38:0/45613975 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T15:01:28.399 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.394+0000 7fc9acfa6700 1 --2- 172.21.15.38:0/45613975 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564d0bfd7fe0 0x564d0bfed090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T15:01:28.399 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.394+0000 7fc9a7fff700 1 --2- 172.21.15.38:0/45613975 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564d0bfdd830 0x564d0bfedcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T15:01:28.399 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.394+0000 7fc9ad7a7700 1 -- 172.21.15.38:0/45613975 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564d0bfdd830 msgr2=0x564d0bfedcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T15:01:28.399 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.394+0000 7fc9ad7a7700 1 --2- 172.21.15.38:0/45613975 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564d0bfdd830 0x564d0bfedcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:01:28.399 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.395+0000 7fc9ad7a7700 1 -- 172.21.15.38:0/45613975 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564d0bfd7fe0 msgr2=0x564d0bfed090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T15:01:28.400 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.395+0000 7fc9ad7a7700 1 --2- 172.21.15.38:0/45613975 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564d0bfd7fe0 0x564d0bfed090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:01:28.400 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.395+0000 7fc9ad7a7700 1 -- 172.21.15.38:0/45613975 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564d0bd40c60 con 0x564d0bfde140 2023-03-09T15:01:28.400 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.395+0000 7fc9a7fff700 1 --2- 172.21.15.38:0/45613975 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564d0bfdd830 0x564d0bfedcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-09T15:01:28.400 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.395+0000 7fc9ad7a7700 1 --2- 172.21.15.38:0/45613975 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d0bfde140 0x564d0bfe3c50 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fc9a0003ee0 tx=0x7fc9a0009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e51aa8d0de31cc0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T15:01:28.400 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.395+0000 7fc9937fe700 1 -- 172.21.15.38:0/45613975 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9a000e040 con 0x564d0bfde140 2023-03-09T15:01:28.400 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.395+0000 7fc9937fe700 1 -- 172.21.15.38:0/45613975 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc9a000f040 con 0x564d0bfde140 2023-03-09T15:01:28.400 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.395+0000 7fc9937fe700 1 -- 172.21.15.38:0/45613975 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9a0017810 con 0x564d0bfde140 2023-03-09T15:01:28.403 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.395+0000 7fc9bb393b80 1 -- 172.21.15.38:0/45613975 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d0bfde140 msgr2=0x564d0bfe3c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T15:01:28.404 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.395+0000 7fc9bb393b80 1 --2- 172.21.15.38:0/45613975 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d0bfde140 0x564d0bfe3c50 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fc9a0003ee0 tx=0x7fc9a0009920 comp rx=0 tx=0).stop 2023-03-09T15:01:28.404 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.395+0000 7fc9bb393b80 1 -- 172.21.15.38:0/45613975 shutdown_connections 2023-03-09T15:01:28.404 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.395+0000 7fc9bb393b80 1 --2- 172.21.15.38:0/45613975 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d0bfde140 0x564d0bfe3c50 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:01:28.404 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.395+0000 7fc9bb393b80 1 --2- 172.21.15.38:0/45613975 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564d0bfd7fe0 0x564d0bfed090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:01:28.404 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.395+0000 7fc9bb393b80 1 --2- 172.21.15.38:0/45613975 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564d0bfdd830 0x564d0bfedcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:01:28.404 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.395+0000 7fc9bb393b80 1 -- 172.21.15.38:0/45613975 >> 172.21.15.38:0/45613975 conn(0x564d0bfd2a20 msgr2=0x564d0bfd0f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T15:01:28.404 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.395+0000 7fc9bb393b80 1 -- 172.21.15.38:0/45613975 shutdown_connections 2023-03-09T15:01:28.405 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.395+0000 7fc9bb393b80 1 -- 172.21.15.38:0/45613975 wait complete. 2023-03-09T15:01:28.405 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.396+0000 7fc9bb393b80 1 -- start start 2023-03-09T15:01:28.405 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.396+0000 7fc9bb393b80 1 Processor -- start 2023-03-09T15:01:28.405 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.396+0000 7fc9bb393b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564d0bfd7fe0 0x564d0bfda9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T15:01:28.405 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.396+0000 7fc9bb393b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d0bfdd830 0x564d0bfd9290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T15:01:28.405 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.396+0000 7fc9bb393b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564d0bfde140 0x564d0bfd97d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T15:01:28.405 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.396+0000 7fc9bb393b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d0bfe8ea0 con 0x564d0bfdd830 2023-03-09T15:01:28.406 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.396+0000 7fc9bb393b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d0bfd9d10 con 0x564d0bfde140 2023-03-09T15:01:28.406 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.396+0000 7fc9bb393b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d0bfd9ed0 con 0x564d0bfd7fe0 2023-03-09T15:01:28.406 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.396+0000 7fc9bb393b80 10 client.0 _ll_register_callbacks cb 0x564d0becf520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T15:01:28.406 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.396+0000 7fc9a7fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d0bfdd830 0x564d0bfd9290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T15:01:28.406 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.396+0000 7fc9a7fff700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d0bfdd830 0x564d0bfd9290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42480/0 (socket says 172.21.15.38:42480) 2023-03-09T15:01:28.406 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.396+0000 7fc9a7fff700 1 -- 172.21.15.38:0/3266034585 learned_addr learned my addr 172.21.15.38:0/3266034585 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T15:01:28.406 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.396+0000 7fc9ad7a7700 1 --2- 172.21.15.38:0/3266034585 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564d0bfde140 0x564d0bfd97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T15:01:28.406 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.397+0000 7fc9acfa6700 1 --2- 172.21.15.38:0/3266034585 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564d0bfd7fe0 0x564d0bfda9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T15:01:28.407 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.397+0000 7fc9a7fff700 1 -- 172.21.15.38:0/3266034585 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564d0bfd7fe0 msgr2=0x564d0bfda9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T15:01:28.407 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.397+0000 7fc9a7fff700 1 --2- 172.21.15.38:0/3266034585 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564d0bfd7fe0 0x564d0bfda9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:01:28.407 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.397+0000 7fc9a7fff700 1 -- 172.21.15.38:0/3266034585 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564d0bfde140 msgr2=0x564d0bfd97d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T15:01:28.407 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.397+0000 7fc9a7fff700 1 --2- 172.21.15.38:0/3266034585 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564d0bfde140 0x564d0bfd97d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:01:28.407 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.397+0000 7fc9a7fff700 1 -- 172.21.15.38:0/3266034585 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9a0003b90 con 0x564d0bfdd830 2023-03-09T15:01:28.407 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.397+0000 7fc9ad7a7700 1 --2- 172.21.15.38:0/3266034585 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564d0bfde140 0x564d0bfd97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T15:01:28.407 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.397+0000 7fc9a7fff700 1 --2- 172.21.15.38:0/3266034585 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d0bfdd830 0x564d0bfd9290 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x564d0bfd2990 tx=0x7fc9940047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2bea08fb747e4c2 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T15:01:28.408 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.397+0000 7fc987fff700 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2023-03-09T15:01:28.408 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.397+0000 7fc987fff700 1 -- 172.21.15.38:0/3266034585 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc994021040 con 0x564d0bfdd830 2023-03-09T15:01:28.408 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.397+0000 7fc987fff700 1 -- 172.21.15.38:0/3266034585 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(255..255 src has 1..255) v4 ==== 9577+0+0 (secure 0 0 0) 0x7fc994010b60 con 0x564d0bfdd830 2023-03-09T15:01:28.408 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.397+0000 7fc9bb393b80 1 -- 172.21.15.38:0/3266034585 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564d0bee71b0 con 0x564d0bfdd830 2023-03-09T15:01:28.408 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.397+0000 7fc987fff700 1 -- 172.21.15.38:0/3266034585 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc9940265e0 con 0x564d0bfdd830 2023-03-09T15:01:28.408 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.397+0000 7fc9bb393b80 10 client.16425 fetch_fsmap learned FSMap version 47 2023-03-09T15:01:28.408 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.397+0000 7fc9bb393b80 1 -- 172.21.15.38:0/3266034585 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x564d0bd7e250 con 0x564d0bfdd830 2023-03-09T15:01:28.408 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.397+0000 7fc987fff700 1 -- 172.21.15.38:0/3266034585 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x564d0bee71b0 con 0x564d0bfdd830 2023-03-09T15:01:28.409 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.398+0000 7fc987fff700 1 -- 172.21.15.38:0/3266034585 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 47) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fc994004c10 con 0x564d0bfdd830 2023-03-09T15:01:28.409 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.398+0000 7fc9bb393b80 10 client.16425 fetch_fsmap finished waiting for FSMap version 47 2023-03-09T15:01:28.409 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.398+0000 7fc9bb393b80 10 client.16425 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T15:01:28.409 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.398+0000 7fc9bb393b80 20 client.16425 populate_metadata read hostname 'smithi038' 2023-03-09T15:01:28.409 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.398+0000 7fc9bb393b80 1 --2- 172.21.15.38:0/3266034585 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x564d0c046720 0x564d0c048b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T15:01:28.409 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.398+0000 7fc9bb393b80 4 client.16425 mds_command: new command op to 24530 tid=0[{"prefix": "scrub status"}] 2023-03-09T15:01:28.409 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.398+0000 7fc9bb393b80 1 -- 172.21.15.38:0/3266034585 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x564d0bed11c0 con 0x564d0c046720 2023-03-09T15:01:28.409 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.398+0000 7fc9acfa6700 1 --2- 172.21.15.38:0/3266034585 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x564d0c046720 0x564d0c048b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T15:01:28.410 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.398+0000 7fc9acfa6700 1 --2- 172.21.15.38:0/3266034585 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x564d0c046720 0x564d0c048b30 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=63fc6d4392e2ced1 server_cookie=a9226beb812f6f41 in_seq=0 out_seq=0 2023-03-09T15:01:28.410 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.398+0000 7fc987fff700 10 client.16425 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T15:01:28.410 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.398+0000 7fc987fff700 1 -- 172.21.15.38:0/3266034585 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+265 (crc 0 0 0) 0x564d0bed11c0 con 0x564d0c046720 2023-03-09T15:01:28.410 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.398+0000 7fc987fff700 10 client.16425 handle_command_reply: tid=0 2023-03-09T15:01:28.410 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.398+0000 7fc9bb393b80 1 client.16425 shutdown 2023-03-09T15:01:28.410 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.398+0000 7fc9bb393b80 1 -- 172.21.15.38:0/3266034585 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d0bfdd830 msgr2=0x564d0bfd9290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T15:01:28.410 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.398+0000 7fc9bb393b80 1 --2- 172.21.15.38:0/3266034585 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d0bfdd830 0x564d0bfd9290 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x564d0bfd2990 tx=0x7fc9940047c0 comp rx=0 tx=0).stop 2023-03-09T15:01:28.411 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.398+0000 7fc9bb393b80 1 -- 172.21.15.38:0/3266034585 shutdown_connections 2023-03-09T15:01:28.411 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.398+0000 7fc9bb393b80 1 --2- 172.21.15.38:0/3266034585 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x564d0c046720 0x564d0c048b30 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:01:28.411 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.398+0000 7fc9bb393b80 1 --2- 172.21.15.38:0/3266034585 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x564d0bfde140 0x564d0bfd97d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:01:28.411 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.398+0000 7fc9bb393b80 1 --2- 172.21.15.38:0/3266034585 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x564d0bfd7fe0 0x564d0bfda9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:01:28.411 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.398+0000 7fc9bb393b80 1 --2- 172.21.15.38:0/3266034585 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x564d0bfdd830 0x564d0bfd9290 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:01:28.411 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.398+0000 7fc987fff700 0 client.16425 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T15:01:28.411 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.398+0000 7fc9bb393b80 1 -- 172.21.15.38:0/3266034585 >> 172.21.15.38:0/3266034585 conn(0x564d0bfd2a20 msgr2=0x564d0bfed090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T15:01:28.411 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.399+0000 7fc9bb393b80 1 -- 172.21.15.38:0/3266034585 shutdown_connections 2023-03-09T15:01:28.412 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.399+0000 7fc9bb393b80 1 -- 172.21.15.38:0/3266034585 wait complete. 2023-03-09T15:01:28.412 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.399+0000 7fc9bb393b80 20 client.16425 trim_cache size 0 max 16384 2023-03-09T15:01:28.412 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2023-03-09T15:01:28.412 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "scrub active (1 inodes in the stack)", 2023-03-09T15:01:28.412 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": { 2023-03-09T15:01:28.413 INFO:teuthology.orchestra.run.smithi038.stdout: "a14659f4-d7ea-4c5a-b15b-6138af3f909f": { 2023-03-09T15:01:28.413 INFO:teuthology.orchestra.run.smithi038.stdout: "path": "/", 2023-03-09T15:01:28.413 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "a14659f4-d7ea-4c5a-b15b-6138af3f909f", 2023-03-09T15:01:28.413 INFO:teuthology.orchestra.run.smithi038.stdout: "options": "recursive,force" 2023-03-09T15:01:28.413 INFO:teuthology.orchestra.run.smithi038.stdout: } 2023-03-09T15:01:28.413 INFO:teuthology.orchestra.run.smithi038.stdout: } 2023-03-09T15:01:28.413 INFO:teuthology.orchestra.run.smithi038.stdout:} 2023-03-09T15:01:28.414 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.399+0000 7fc9af20a700 1 -- 172.21.15.38:0/2707832422 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fc99809f4a0 msgr2=0x7fc9980a1950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T15:01:28.414 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.399+0000 7fc9af20a700 1 --2- 172.21.15.38:0/2707832422 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fc99809f4a0 0x7fc9980a1950 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc9a8068310 tx=0x7fc99400a380 comp rx=0 tx=0).stop 2023-03-09T15:01:28.414 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.399+0000 7fc9af20a700 1 -- 172.21.15.38:0/2707832422 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc9a80ff000 msgr2=0x7fc9a8196dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T15:01:28.414 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.399+0000 7fc9af20a700 1 --2- 172.21.15.38:0/2707832422 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc9a80ff000 0x7fc9a8196dd0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fc99c00b580 tx=0x7fc99c00f690 comp rx=0 tx=0).stop 2023-03-09T15:01:28.415 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.399+0000 7fc9af20a700 1 -- 172.21.15.38:0/2707832422 shutdown_connections 2023-03-09T15:01:28.415 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.399+0000 7fc9af20a700 1 --2- 172.21.15.38:0/2707832422 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fc99809f4a0 0x7fc9980a1950 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:01:28.415 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.399+0000 7fc9af20a700 1 --2- 172.21.15.38:0/2707832422 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc9a80ff000 0x7fc9a8196dd0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:01:28.415 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.399+0000 7fc9af20a700 1 --2- 172.21.15.38:0/2707832422 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc9a8104040 0x7fc9a81a1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:01:28.415 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.399+0000 7fc9af20a700 1 --2- 172.21.15.38:0/2707832422 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc9a80ff9a0 0x7fc9a8197310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:01:28.415 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.399+0000 7fc9af20a700 1 -- 172.21.15.38:0/2707832422 >> 172.21.15.38:0/2707832422 conn(0x7fc9a8075c60 msgr2=0x7fc9a80fc940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T15:01:28.415 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.399+0000 7fc9af20a700 1 -- 172.21.15.38:0/2707832422 shutdown_connections 2023-03-09T15:01:28.416 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:28.399+0000 7fc9af20a700 1 -- 172.21.15.38:0/2707832422 wait complete. 2023-03-09T15:01:28.416 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:01:28 smithi038 ceph-mon[92422]: pgmap v624: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 17 KiB/s wr, 7 op/s 2023-03-09T15:01:28.417 INFO:tasks.cephfs.filesystem:scrub status for tag:a14659f4-d7ea-4c5a-b15b-6138af3f909f - {'path': '/', 'tag': 'a14659f4-d7ea-4c5a-b15b-6138af3f909f', 'options': 'recursive,force'} 2023-03-09T15:01:28.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:01:28 smithi122 ceph-mon[93869]: pgmap v624: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 17 KiB/s wr, 7 op/s 2023-03-09T15:01:28.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:01:28 smithi181 ceph-mon[94189]: pgmap v624: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 17 KiB/s wr, 7 op/s 2023-03-09T15:01:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 660.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:01:30.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:01:30 smithi122 ceph-mon[93869]: pgmap v625: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 12 KiB/s wr, 6 op/s 2023-03-09T15:01:30.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:01:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 200.934957 secs 2023-03-09T15:01:30.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:01:30 smithi181 ceph-mon[94189]: pgmap v625: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 12 KiB/s wr, 6 op/s 2023-03-09T15:01:30.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:01:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 200.934957 secs 2023-03-09T15:01:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:01:30 smithi038 ceph-mon[92422]: pgmap v625: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 12 KiB/s wr, 6 op/s 2023-03-09T15:01:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:01:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 200.934957 secs 2023-03-09T15:01:32.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:01:32 smithi122 ceph-mon[93869]: pgmap v626: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 11 KiB/s wr, 6 op/s 2023-03-09T15:01:32.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:01:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 203.123137 secs 2023-03-09T15:01:32.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:01:32 smithi181 ceph-mon[94189]: pgmap v626: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 11 KiB/s wr, 6 op/s 2023-03-09T15:01:32.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:01:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 203.123137 secs 2023-03-09T15:01:32.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:01:32 smithi038 ceph-mon[92422]: pgmap v626: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 11 KiB/s wr, 6 op/s 2023-03-09T15:01:32.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:01:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 203.123137 secs 2023-03-09T15:01:33.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 665.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:01:34.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:01:33 smithi122 ceph-mon[93869]: pgmap v627: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 34 KiB/s wr, 10 op/s 2023-03-09T15:01:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:01:33 smithi181 ceph-mon[94189]: pgmap v627: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 34 KiB/s wr, 10 op/s 2023-03-09T15:01:34.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:01:33 smithi038 ceph-mon[92422]: pgmap v627: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 34 KiB/s wr, 10 op/s 2023-03-09T15:01:35.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:01:34 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 205.934984 secs 2023-03-09T15:01:35.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:01:34 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 205.934984 secs 2023-03-09T15:01:35.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:01:34 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 205.934984 secs 2023-03-09T15:01:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:01:35 smithi122 ceph-mon[93869]: pgmap v628: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 33 KiB/s wr, 8 op/s 2023-03-09T15:01:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:01:35 smithi181 ceph-mon[94189]: pgmap v628: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 33 KiB/s wr, 8 op/s 2023-03-09T15:01:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:01:35 smithi038 ceph-mon[92422]: pgmap v628: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 33 KiB/s wr, 8 op/s 2023-03-09T15:01:37.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:01:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 208.123269 secs 2023-03-09T15:01:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:01:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 208.123269 secs 2023-03-09T15:01:37.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:01:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 208.123269 secs 2023-03-09T15:01:38.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:01:38 smithi122 ceph-mon[93869]: pgmap v629: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 33 KiB/s wr, 11 op/s 2023-03-09T15:01:38.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:01:38 smithi181 ceph-mon[94189]: pgmap v629: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 33 KiB/s wr, 11 op/s 2023-03-09T15:01:38.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:01:38 smithi038 ceph-mon[92422]: pgmap v629: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 33 KiB/s wr, 11 op/s 2023-03-09T15:01:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 670.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:01:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:01:40 smithi122 ceph-mon[93869]: pgmap v630: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 34 KiB/s wr, 8 op/s 2023-03-09T15:01:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:01:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 210.935044 secs 2023-03-09T15:01:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:01:40 smithi181 ceph-mon[94189]: pgmap v630: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 34 KiB/s wr, 8 op/s 2023-03-09T15:01:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:01:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 210.935044 secs 2023-03-09T15:01:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:01:40 smithi038 ceph-mon[92422]: pgmap v630: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 34 KiB/s wr, 8 op/s 2023-03-09T15:01:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:01:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 210.935044 secs 2023-03-09T15:01:42.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:01:42 smithi122 ceph-mon[93869]: pgmap v631: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 28 KiB/s wr, 8 op/s 2023-03-09T15:01:42.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:01:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 213.123335 secs 2023-03-09T15:01:42.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:01:42 smithi181 ceph-mon[94189]: pgmap v631: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 28 KiB/s wr, 8 op/s 2023-03-09T15:01:42.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:01:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 213.123335 secs 2023-03-09T15:01:42.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:01:42 smithi038 ceph-mon[92422]: pgmap v631: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 28 KiB/s wr, 8 op/s 2023-03-09T15:01:42.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:01:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 213.123335 secs 2023-03-09T15:01:43.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 675.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:01:44.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:01:43 smithi122 ceph-mon[93869]: pgmap v632: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 37 KiB/s wr, 10 op/s 2023-03-09T15:01:44.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:01:43 smithi181 ceph-mon[94189]: pgmap v632: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 37 KiB/s wr, 10 op/s 2023-03-09T15:01:44.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:01:43 smithi038 ceph-mon[92422]: pgmap v632: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 37 KiB/s wr, 10 op/s 2023-03-09T15:01:45.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:01:44 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 215.935124 secs 2023-03-09T15:01:45.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:01:44 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 215.935124 secs 2023-03-09T15:01:45.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:01:44 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 215.935124 secs 2023-03-09T15:01:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:01:45 smithi122 ceph-mon[93869]: pgmap v633: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 14 KiB/s wr, 6 op/s 2023-03-09T15:01:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:01:45 smithi181 ceph-mon[94189]: pgmap v633: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 14 KiB/s wr, 6 op/s 2023-03-09T15:01:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:01:45 smithi038 ceph-mon[92422]: pgmap v633: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 14 KiB/s wr, 6 op/s 2023-03-09T15:01:47.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:01:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 218.123450 secs 2023-03-09T15:01:47.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:01:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 218.123450 secs 2023-03-09T15:01:47.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:01:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 218.123450 secs 2023-03-09T15:01:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:01:48 smithi122 ceph-mon[93869]: pgmap v634: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 16 KiB/s wr, 8 op/s 2023-03-09T15:01:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:01:48 smithi181 ceph-mon[94189]: pgmap v634: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 16 KiB/s wr, 8 op/s 2023-03-09T15:01:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:01:48 smithi038 ceph-mon[92422]: pgmap v634: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 16 KiB/s wr, 8 op/s 2023-03-09T15:01:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 680.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:01:50.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:01:50 smithi122 ceph-mon[93869]: pgmap v635: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 15 KiB/s wr, 6 op/s 2023-03-09T15:01:50.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:01:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 220.935187 secs 2023-03-09T15:01:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:01:50 smithi181 ceph-mon[94189]: pgmap v635: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 15 KiB/s wr, 6 op/s 2023-03-09T15:01:50.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:01:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 220.935187 secs 2023-03-09T15:01:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:01:50 smithi038 ceph-mon[92422]: pgmap v635: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 15 KiB/s wr, 6 op/s 2023-03-09T15:01:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:01:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 220.935187 secs 2023-03-09T15:01:51.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:01:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:01:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:01:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:01:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:01:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:01:52.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:01:52 smithi122 ceph-mon[93869]: pgmap v636: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 10 KiB/s wr, 6 op/s 2023-03-09T15:01:52.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:01:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 223.123574 secs 2023-03-09T15:01:52.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:01:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:01:52.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:01:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:01:52.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:01:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:01:52.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:01:52 smithi181 ceph-mon[94189]: pgmap v636: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 10 KiB/s wr, 6 op/s 2023-03-09T15:01:52.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:01:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 223.123574 secs 2023-03-09T15:01:52.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:01:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:01:52.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:01:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:01:52.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:01:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:01:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:01:52 smithi038 ceph-mon[92422]: pgmap v636: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 10 KiB/s wr, 6 op/s 2023-03-09T15:01:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:01:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 223.123574 secs 2023-03-09T15:01:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:01:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:01:52.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:01:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:01:52.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:01:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:01:53.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 685.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:01:54.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:01:53 smithi122 ceph-mon[93869]: pgmap v637: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T15:01:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:01:53 smithi181 ceph-mon[94189]: pgmap v637: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T15:01:54.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:01:53 smithi038 ceph-mon[92422]: pgmap v637: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T15:01:55.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:01:54 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 225.935249 secs 2023-03-09T15:01:55.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:01:54 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 225.935249 secs 2023-03-09T15:01:55.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:01:54 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 225.935249 secs 2023-03-09T15:01:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:01:55 smithi122 ceph-mon[93869]: pgmap v638: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 7.4 KiB/s wr, 5 op/s 2023-03-09T15:01:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:01:55 smithi181 ceph-mon[94189]: pgmap v638: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 7.4 KiB/s wr, 5 op/s 2023-03-09T15:01:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:01:55 smithi038 ceph-mon[92422]: pgmap v638: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 7.4 KiB/s wr, 5 op/s 2023-03-09T15:01:57.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:01:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 228.123694 secs 2023-03-09T15:01:57.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:01:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 228.123694 secs 2023-03-09T15:01:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:01:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 228.123694 secs 2023-03-09T15:01:58.418 DEBUG:teuthology.orchestra.run.smithi038:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 scrub status 2023-03-09T15:01:58.450 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:01:58 smithi038 ceph-mon[92422]: pgmap v639: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.5 KiB/s wr, 8 op/s 2023-03-09T15:01:58.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:01:58 smithi122 ceph-mon[93869]: pgmap v639: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.5 KiB/s wr, 8 op/s 2023-03-09T15:01:58.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:01:58 smithi181 ceph-mon[94189]: pgmap v639: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.5 KiB/s wr, 8 op/s 2023-03-09T15:01:58.563 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.561+0000 7fc424497700 1 -- 172.21.15.38:0/2560225959 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc41c0ff9a0 msgr2=0x7fc41c103960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T15:01:58.563 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.561+0000 7fc424497700 1 --2- 172.21.15.38:0/2560225959 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc41c0ff9a0 0x7fc41c103960 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fc4180099c0 tx=0x7fc418009cd0 comp rx=0 tx=0).stop 2023-03-09T15:01:58.564 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.561+0000 7fc424497700 1 -- 172.21.15.38:0/2560225959 shutdown_connections 2023-03-09T15:01:58.564 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.561+0000 7fc424497700 1 --2- 172.21.15.38:0/2560225959 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc41c104040 0x7fc41c10bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:01:58.564 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.561+0000 7fc424497700 1 --2- 172.21.15.38:0/2560225959 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc41c0ff000 0x7fc41c0ff3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:01:58.564 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.561+0000 7fc424497700 1 --2- 172.21.15.38:0/2560225959 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc41c0ff9a0 0x7fc41c103960 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:01:58.564 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.561+0000 7fc424497700 1 -- 172.21.15.38:0/2560225959 >> 172.21.15.38:0/2560225959 conn(0x7fc41c075c60 msgr2=0x7fc41c076060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T15:01:58.564 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.561+0000 7fc424497700 1 -- 172.21.15.38:0/2560225959 shutdown_connections 2023-03-09T15:01:58.564 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.561+0000 7fc424497700 1 -- 172.21.15.38:0/2560225959 wait complete. 2023-03-09T15:01:58.566 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.562+0000 7fc424497700 1 Processor -- start 2023-03-09T15:01:58.567 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.562+0000 7fc424497700 1 -- start start 2023-03-09T15:01:58.567 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.562+0000 7fc424497700 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc41c0ff000 0x7fc41c196ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T15:01:58.567 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.562+0000 7fc424497700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc41c0ff9a0 0x7fc41c197430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T15:01:58.568 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.562+0000 7fc424497700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc41c104040 0x7fc41c1a1f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T15:01:58.568 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.562+0000 7fc424497700 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc41c197d70 con 0x7fc41c0ff9a0 2023-03-09T15:01:58.568 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.562+0000 7fc424497700 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc41c1a24b0 con 0x7fc41c104040 2023-03-09T15:01:58.568 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.562+0000 7fc424497700 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc41c1a2670 con 0x7fc41c0ff000 2023-03-09T15:01:58.569 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.562+0000 7fc421a32700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc41c0ff9a0 0x7fc41c197430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T15:01:58.569 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.562+0000 7fc422a34700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc41c104040 0x7fc41c1a1f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T15:01:58.569 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.562+0000 7fc421a32700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc41c0ff9a0 0x7fc41c197430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42502/0 (socket says 172.21.15.38:42502) 2023-03-09T15:01:58.570 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.562+0000 7fc421a32700 1 -- 172.21.15.38:0/185176896 learned_addr learned my addr 172.21.15.38:0/185176896 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T15:01:58.570 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.562+0000 7fc422233700 1 --2- 172.21.15.38:0/185176896 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc41c0ff000 0x7fc41c196ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T15:01:58.570 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.562+0000 7fc421a32700 1 -- 172.21.15.38:0/185176896 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc41c0ff000 msgr2=0x7fc41c196ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T15:01:58.571 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.563+0000 7fc421a32700 1 --2- 172.21.15.38:0/185176896 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc41c0ff000 0x7fc41c196ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:01:58.571 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.563+0000 7fc421a32700 1 -- 172.21.15.38:0/185176896 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc41c104040 msgr2=0x7fc41c1a1f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T15:01:58.571 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.563+0000 7fc421a32700 1 --2- 172.21.15.38:0/185176896 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc41c104040 0x7fc41c1a1f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:01:58.572 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.563+0000 7fc421a32700 1 -- 172.21.15.38:0/185176896 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc40c009520 con 0x7fc41c0ff9a0 2023-03-09T15:01:58.572 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.563+0000 7fc422a34700 1 --2- 172.21.15.38:0/185176896 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc41c104040 0x7fc41c1a1f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T15:01:58.572 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.563+0000 7fc421a32700 1 --2- 172.21.15.38:0/185176896 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc41c0ff9a0 0x7fc41c197430 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fc418009fd0 tx=0x7fc418009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T15:01:58.572 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.563+0000 7fc4137fe700 1 -- 172.21.15.38:0/185176896 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc41801d070 con 0x7fc41c0ff9a0 2023-03-09T15:01:58.573 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.563+0000 7fc4137fe700 1 -- 172.21.15.38:0/185176896 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc418005020 con 0x7fc41c0ff9a0 2023-03-09T15:01:58.573 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.563+0000 7fc424497700 1 -- 172.21.15.38:0/185176896 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4180095f0 con 0x7fc41c0ff9a0 2023-03-09T15:01:58.573 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.563+0000 7fc4137fe700 1 -- 172.21.15.38:0/185176896 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc418004a90 con 0x7fc41c0ff9a0 2023-03-09T15:01:58.573 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.563+0000 7fc424497700 1 -- 172.21.15.38:0/185176896 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc41c10edc0 con 0x7fc41c0ff9a0 2023-03-09T15:01:58.579 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.565+0000 7fc4137fe700 1 -- 172.21.15.38:0/185176896 <== mon.0 v2:172.21.15.38:3300/0 4 ==== mgrmap(e 25) v1 ==== 133295+0+0 (secure 0 0 0) 0x7fc41800c040 con 0x7fc41c0ff9a0 2023-03-09T15:01:58.579 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.566+0000 7fc4137fe700 1 --2- 172.21.15.38:0/185176896 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fc40809f520 0x7fc4080a19d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T15:01:58.579 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.566+0000 7fc422233700 1 --2- 172.21.15.38:0/185176896 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fc40809f520 0x7fc4080a19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T15:01:58.579 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.566+0000 7fc4137fe700 1 -- 172.21.15.38:0/185176896 <== mon.0 v2:172.21.15.38:3300/0 5 ==== osd_map(255..255 src has 1..255) v4 ==== 9577+0+0 (secure 0 0 0) 0x7fc4180cac40 con 0x7fc41c0ff9a0 2023-03-09T15:01:58.579 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.566+0000 7fc422233700 1 --2- 172.21.15.38:0/185176896 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fc40809f520 0x7fc4080a19d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fc40c009dd0 tx=0x7fc40c009260 comp rx=0 tx=0).ready entity=mgr.15558 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T15:01:58.579 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.566+0000 7fc430620b80 1 Processor -- start 2023-03-09T15:01:58.580 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.567+0000 7fc430620b80 1 -- start start 2023-03-09T15:01:58.580 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.567+0000 7fc430620b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631fb449c40 0x5631fb44a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T15:01:58.580 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.567+0000 7fc430620b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5631fb44a550 0x5631fb443eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T15:01:58.580 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.567+0000 7fc430620b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5631fb4443f0 0x5631fb444820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T15:01:58.580 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.567+0000 7fc430620b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631fb329da0 con 0x5631fb449c40 2023-03-09T15:01:58.580 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.567+0000 7fc430620b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631fb398780 con 0x5631fb4443f0 2023-03-09T15:01:58.580 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.567+0000 7fc430620b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631fb391cb0 con 0x5631fb44a550 2023-03-09T15:01:58.581 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.567+0000 7fc422233700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631fb449c40 0x5631fb44a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T15:01:58.581 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.567+0000 7fc422233700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631fb449c40 0x5631fb44a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42508/0 (socket says 172.21.15.38:42508) 2023-03-09T15:01:58.581 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.567+0000 7fc422233700 1 -- 172.21.15.38:0/3796656783 learned_addr learned my addr 172.21.15.38:0/3796656783 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T15:01:58.581 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.567+0000 7fc422a34700 1 --2- 172.21.15.38:0/3796656783 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5631fb4443f0 0x5631fb444820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T15:01:58.581 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.567+0000 7fc421a32700 1 --2- 172.21.15.38:0/3796656783 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5631fb44a550 0x5631fb443eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T15:01:58.581 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.567+0000 7fc422233700 1 -- 172.21.15.38:0/3796656783 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5631fb44a550 msgr2=0x5631fb443eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T15:01:58.581 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.567+0000 7fc422233700 1 --2- 172.21.15.38:0/3796656783 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5631fb44a550 0x5631fb443eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:01:58.581 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.567+0000 7fc422233700 1 -- 172.21.15.38:0/3796656783 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5631fb4443f0 msgr2=0x5631fb444820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T15:01:58.582 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.567+0000 7fc422233700 1 --2- 172.21.15.38:0/3796656783 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5631fb4443f0 0x5631fb444820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:01:58.582 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.567+0000 7fc422233700 1 -- 172.21.15.38:0/3796656783 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5631fb358190 con 0x5631fb449c40 2023-03-09T15:01:58.582 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.567+0000 7fc421a32700 1 --2- 172.21.15.38:0/3796656783 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5631fb44a550 0x5631fb443eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-09T15:01:58.582 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.567+0000 7fc422a34700 1 --2- 172.21.15.38:0/3796656783 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5631fb4443f0 0x5631fb444820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T15:01:58.582 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.567+0000 7fc422233700 1 --2- 172.21.15.38:0/3796656783 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631fb449c40 0x5631fb44a010 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fc40c00e600 tx=0x7fc40c014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16253a307389fbe3 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T15:01:58.582 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.567+0000 7fc410daf700 1 -- 172.21.15.38:0/3796656783 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc40c026430 con 0x5631fb449c40 2023-03-09T15:01:58.582 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.568+0000 7fc410daf700 1 -- 172.21.15.38:0/3796656783 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc40c021070 con 0x5631fb449c40 2023-03-09T15:01:58.583 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.568+0000 7fc410daf700 1 -- 172.21.15.38:0/3796656783 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc40c01d8b0 con 0x5631fb449c40 2023-03-09T15:01:58.583 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.568+0000 7fc430620b80 1 -- 172.21.15.38:0/3796656783 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631fb449c40 msgr2=0x5631fb44a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T15:01:58.583 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.568+0000 7fc430620b80 1 --2- 172.21.15.38:0/3796656783 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631fb449c40 0x5631fb44a010 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7fc40c00e600 tx=0x7fc40c014000 comp rx=0 tx=0).stop 2023-03-09T15:01:58.583 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.568+0000 7fc430620b80 1 -- 172.21.15.38:0/3796656783 shutdown_connections 2023-03-09T15:01:58.583 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.568+0000 7fc430620b80 1 --2- 172.21.15.38:0/3796656783 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631fb449c40 0x5631fb44a010 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:01:58.583 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.568+0000 7fc430620b80 1 --2- 172.21.15.38:0/3796656783 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5631fb4443f0 0x5631fb444820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:01:58.583 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.568+0000 7fc430620b80 1 --2- 172.21.15.38:0/3796656783 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5631fb44a550 0x5631fb443eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:01:58.584 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.568+0000 7fc430620b80 1 -- 172.21.15.38:0/3796656783 >> 172.21.15.38:0/3796656783 conn(0x5631fb43aa20 msgr2=0x5631fb43adf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T15:01:58.584 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.568+0000 7fc430620b80 1 -- 172.21.15.38:0/3796656783 shutdown_connections 2023-03-09T15:01:58.584 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.568+0000 7fc430620b80 1 -- 172.21.15.38:0/3796656783 wait complete. 2023-03-09T15:01:58.584 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.569+0000 7fc430620b80 1 -- start start 2023-03-09T15:01:58.584 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.569+0000 7fc430620b80 1 Processor -- start 2023-03-09T15:01:58.584 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.569+0000 7fc430620b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5631fb4443f0 0x5631fb450510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T15:01:58.584 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.569+0000 7fc430620b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5631fb449c40 0x5631fb44d800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T15:01:58.584 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.569+0000 7fc430620b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631fb44a550 0x5631fb44de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T15:01:58.585 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.569+0000 7fc430620b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631fb215920 con 0x5631fb44a550 2023-03-09T15:01:58.585 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.569+0000 7fc430620b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631fb44e5f0 con 0x5631fb4443f0 2023-03-09T15:01:58.585 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.569+0000 7fc430620b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631fb45a050 con 0x5631fb449c40 2023-03-09T15:01:58.585 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.569+0000 7fc430620b80 10 client.0 _ll_register_callbacks cb 0x5631fb337520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T15:01:58.585 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.569+0000 7fc422a34700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631fb44a550 0x5631fb44de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T15:01:58.585 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.569+0000 7fc422233700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5631fb4443f0 0x5631fb450510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T15:01:58.585 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.569+0000 7fc422233700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5631fb4443f0 0x5631fb450510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.122:3300/0 says I am v2:172.21.15.38:51704/0 (socket says 172.21.15.38:51704) 2023-03-09T15:01:58.586 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.569+0000 7fc422233700 1 -- 172.21.15.38:0/3255278951 learned_addr learned my addr 172.21.15.38:0/3255278951 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T15:01:58.586 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.569+0000 7fc421a32700 1 --2- 172.21.15.38:0/3255278951 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5631fb449c40 0x5631fb44d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T15:01:58.586 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.569+0000 7fc422233700 1 -- 172.21.15.38:0/3255278951 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5631fb449c40 msgr2=0x5631fb44d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T15:01:58.586 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.569+0000 7fc422233700 1 --2- 172.21.15.38:0/3255278951 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5631fb449c40 0x5631fb44d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:01:58.586 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.569+0000 7fc422233700 1 -- 172.21.15.38:0/3255278951 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631fb44a550 msgr2=0x5631fb44de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T15:01:58.586 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.570+0000 7fc422233700 1 --2- 172.21.15.38:0/3255278951 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631fb44a550 0x5631fb44de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:01:58.587 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.570+0000 7fc422233700 1 -- 172.21.15.38:0/3255278951 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc414009d10 con 0x5631fb4443f0 2023-03-09T15:01:58.587 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.570+0000 7fc422a34700 1 --2- 172.21.15.38:0/3255278951 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631fb44a550 0x5631fb44de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T15:01:58.587 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.570+0000 7fc422233700 1 --2- 172.21.15.38:0/3255278951 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5631fb4443f0 0x5631fb450510 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fc40c00ee80 tx=0x7fc40c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ea28497c4d5e614 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T15:01:58.587 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.570+0000 7fc405ffb700 10 client.0 ms_handle_connect on v2:172.21.15.122:3300/0 2023-03-09T15:01:58.587 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.570+0000 7fc405ffb700 1 -- 172.21.15.38:0/3255278951 <== mon.2 v2:172.21.15.122:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc40c005410 con 0x5631fb4443f0 2023-03-09T15:01:58.587 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.570+0000 7fc430620b80 1 -- 172.21.15.38:0/3255278951 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5631fb35f190 con 0x5631fb4443f0 2023-03-09T15:01:58.587 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.570+0000 7fc405ffb700 1 -- 172.21.15.38:0/3255278951 <== mon.2 v2:172.21.15.122:3300/0 2 ==== osd_map(255..255 src has 1..255) v4 ==== 9577+0+0 (secure 0 0 0) 0x7fc40c005570 con 0x5631fb4443f0 2023-03-09T15:01:58.587 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.570+0000 7fc405ffb700 1 -- 172.21.15.38:0/3255278951 <== mon.2 v2:172.21.15.122:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc40c01c410 con 0x5631fb4443f0 2023-03-09T15:01:58.588 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.570+0000 7fc430620b80 10 client.26072 fetch_fsmap learned FSMap version 47 2023-03-09T15:01:58.588 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.570+0000 7fc430620b80 1 -- 172.21.15.38:0/3255278951 --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x7fc40c017070 con 0x5631fb4443f0 2023-03-09T15:01:58.588 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.570+0000 7fc405ffb700 1 -- 172.21.15.38:0/3255278951 <== mon.2 v2:172.21.15.122:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5631fb35f190 con 0x5631fb4443f0 2023-03-09T15:01:58.588 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.570+0000 7fc405ffb700 1 -- 172.21.15.38:0/3255278951 <== mon.2 v2:172.21.15.122:3300/0 5 ==== fsmap(e 47) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fc40c003b00 con 0x5631fb4443f0 2023-03-09T15:01:58.588 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.571+0000 7fc430620b80 10 client.26072 fetch_fsmap finished waiting for FSMap version 47 2023-03-09T15:01:58.588 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.571+0000 7fc430620b80 10 client.26072 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T15:01:58.588 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.571+0000 7fc430620b80 20 client.26072 populate_metadata read hostname 'smithi038' 2023-03-09T15:01:58.588 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.571+0000 7fc430620b80 1 --2- 172.21.15.38:0/3255278951 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5631fb45df80 0x5631fb460360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T15:01:58.589 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.571+0000 7fc430620b80 4 client.26072 mds_command: new command op to 24530 tid=0[{"prefix": "get_command_descriptions"}] 2023-03-09T15:01:58.589 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.571+0000 7fc430620b80 1 -- 172.21.15.38:0/3255278951 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5631fb215f60 con 0x5631fb45df80 2023-03-09T15:01:58.589 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.571+0000 7fc421a32700 1 --2- 172.21.15.38:0/3255278951 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5631fb45df80 0x5631fb460360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T15:01:58.589 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.571+0000 7fc405ffb700 10 client.26072 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T15:01:58.589 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.571+0000 7fc421a32700 1 --2- 172.21.15.38:0/3255278951 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5631fb45df80 0x5631fb460360 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3f397672ae39da65 server_cookie=76b36c90206cd3a0 in_seq=0 out_seq=0 2023-03-09T15:01:58.589 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.572+0000 7fc405ffb700 1 -- 172.21.15.38:0/3255278951 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18278 (crc 0 0 0) 0x5631fb215f60 con 0x5631fb45df80 2023-03-09T15:01:58.590 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.572+0000 7fc405ffb700 10 client.26072 handle_command_reply: tid=0 2023-03-09T15:01:58.590 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.572+0000 7fc430620b80 1 client.26072 shutdown 2023-03-09T15:01:58.590 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.572+0000 7fc430620b80 1 -- 172.21.15.38:0/3255278951 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5631fb4443f0 msgr2=0x5631fb450510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T15:01:58.590 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.572+0000 7fc430620b80 1 --2- 172.21.15.38:0/3255278951 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5631fb4443f0 0x5631fb450510 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fc40c00ee80 tx=0x7fc40c00eeb0 comp rx=0 tx=0).stop 2023-03-09T15:01:58.590 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.572+0000 7fc430620b80 1 -- 172.21.15.38:0/3255278951 shutdown_connections 2023-03-09T15:01:58.590 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.572+0000 7fc430620b80 1 --2- 172.21.15.38:0/3255278951 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5631fb45df80 0x5631fb460360 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:01:58.590 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.572+0000 7fc430620b80 1 --2- 172.21.15.38:0/3255278951 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631fb44a550 0x5631fb44de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:01:58.591 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.572+0000 7fc430620b80 1 --2- 172.21.15.38:0/3255278951 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5631fb4443f0 0x5631fb450510 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:01:58.591 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.572+0000 7fc430620b80 1 --2- 172.21.15.38:0/3255278951 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5631fb449c40 0x5631fb44d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:01:58.591 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.572+0000 7fc405ffb700 0 client.26072 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T15:01:58.591 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.572+0000 7fc430620b80 1 -- 172.21.15.38:0/3255278951 >> 172.21.15.38:0/3255278951 conn(0x5631fb43aa20 msgr2=0x5631fb439c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T15:01:58.591 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.572+0000 7fc430620b80 1 -- 172.21.15.38:0/3255278951 shutdown_connections 2023-03-09T15:01:58.591 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.572+0000 7fc430620b80 1 -- 172.21.15.38:0/3255278951 wait complete. 2023-03-09T15:01:58.591 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.572+0000 7fc430620b80 20 client.26072 trim_cache size 0 max 16384 2023-03-09T15:01:58.595 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.589+0000 7fc430620b80 1 Processor -- start 2023-03-09T15:01:58.596 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.589+0000 7fc430620b80 1 -- start start 2023-03-09T15:01:58.596 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.589+0000 7fc430620b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5631fb4443f0 0x5631fb454fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T15:01:58.596 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.589+0000 7fc430620b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5631fb449c40 0x5631fb4500b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T15:01:58.596 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.589+0000 7fc430620b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631fb44a550 0x5631fb439c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T15:01:58.597 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.589+0000 7fc430620b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631fb444ec0 con 0x5631fb44a550 2023-03-09T15:01:58.597 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.589+0000 7fc430620b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631fb4484f0 con 0x5631fb449c40 2023-03-09T15:01:58.597 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.589+0000 7fc430620b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631fb4554e0 con 0x5631fb4443f0 2023-03-09T15:01:58.597 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.590+0000 7fc422a34700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631fb44a550 0x5631fb439c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T15:01:58.598 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.590+0000 7fc421a32700 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5631fb449c40 0x5631fb4500b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T15:01:58.598 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.590+0000 7fc422a34700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631fb44a550 0x5631fb439c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42526/0 (socket says 172.21.15.38:42526) 2023-03-09T15:01:58.598 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.590+0000 7fc422a34700 1 -- 172.21.15.38:0/2825718109 learned_addr learned my addr 172.21.15.38:0/2825718109 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T15:01:58.599 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.590+0000 7fc422233700 1 --2- 172.21.15.38:0/2825718109 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5631fb4443f0 0x5631fb454fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T15:01:58.599 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.590+0000 7fc422a34700 1 -- 172.21.15.38:0/2825718109 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5631fb4443f0 msgr2=0x5631fb454fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T15:01:58.599 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.590+0000 7fc422a34700 1 --2- 172.21.15.38:0/2825718109 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5631fb4443f0 0x5631fb454fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:01:58.599 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.590+0000 7fc422a34700 1 -- 172.21.15.38:0/2825718109 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5631fb449c40 msgr2=0x5631fb4500b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T15:01:58.600 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.590+0000 7fc422a34700 1 --2- 172.21.15.38:0/2825718109 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5631fb449c40 0x5631fb4500b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:01:58.600 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.590+0000 7fc422a34700 1 -- 172.21.15.38:0/2825718109 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5631fb1a8c60 con 0x5631fb44a550 2023-03-09T15:01:58.600 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.590+0000 7fc422233700 1 --2- 172.21.15.38:0/2825718109 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5631fb4443f0 0x5631fb454fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-09T15:01:58.600 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.590+0000 7fc421a32700 1 --2- 172.21.15.38:0/2825718109 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5631fb449c40 0x5631fb4500b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-09T15:01:58.601 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.590+0000 7fc422a34700 1 --2- 172.21.15.38:0/2825718109 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631fb44a550 0x5631fb439c20 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fc414003e20 tx=0x7fc414003e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94a1341c744609bd server_cookie=0 in_seq=0 out_seq=0 2023-03-09T15:01:58.601 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.590+0000 7fc4115b0700 1 -- 172.21.15.38:0/2825718109 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc414004240 con 0x5631fb44a550 2023-03-09T15:01:58.601 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.590+0000 7fc4115b0700 1 -- 172.21.15.38:0/2825718109 <== mon.0 v2:172.21.15.38:3300/0 2 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x7fc41400f040 con 0x5631fb44a550 2023-03-09T15:01:58.602 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.590+0000 7fc4115b0700 1 -- 172.21.15.38:0/2825718109 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc414020d20 con 0x5631fb44a550 2023-03-09T15:01:58.602 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.591+0000 7fc430620b80 1 -- 172.21.15.38:0/2825718109 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631fb44a550 msgr2=0x5631fb439c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T15:01:58.602 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.591+0000 7fc430620b80 1 --2- 172.21.15.38:0/2825718109 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631fb44a550 0x5631fb439c20 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fc414003e20 tx=0x7fc414003e50 comp rx=0 tx=0).stop 2023-03-09T15:01:58.602 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.591+0000 7fc430620b80 1 -- 172.21.15.38:0/2825718109 shutdown_connections 2023-03-09T15:01:58.603 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.591+0000 7fc430620b80 1 --2- 172.21.15.38:0/2825718109 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631fb44a550 0x5631fb439c20 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:01:58.603 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.591+0000 7fc430620b80 1 --2- 172.21.15.38:0/2825718109 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5631fb4443f0 0x5631fb454fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:01:58.603 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.591+0000 7fc430620b80 1 --2- 172.21.15.38:0/2825718109 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5631fb449c40 0x5631fb4500b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:01:58.603 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.591+0000 7fc430620b80 1 -- 172.21.15.38:0/2825718109 >> 172.21.15.38:0/2825718109 conn(0x5631fb43aa20 msgr2=0x5631fb45a440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T15:01:58.603 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.591+0000 7fc430620b80 1 -- 172.21.15.38:0/2825718109 shutdown_connections 2023-03-09T15:01:58.603 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.591+0000 7fc430620b80 1 -- 172.21.15.38:0/2825718109 wait complete. 2023-03-09T15:01:58.603 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.591+0000 7fc430620b80 1 -- start start 2023-03-09T15:01:58.604 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.591+0000 7fc430620b80 1 Processor -- start 2023-03-09T15:01:58.604 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.591+0000 7fc430620b80 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631fb4443f0 0x5631fb454d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T15:01:58.604 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.591+0000 7fc430620b80 1 --2- >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5631fb449c40 0x5631fb4530c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T15:01:58.604 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.591+0000 7fc430620b80 1 --2- >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5631fb44a550 0x5631fb453600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T15:01:58.604 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.591+0000 7fc430620b80 1 -- --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631fb453d10 con 0x5631fb4443f0 2023-03-09T15:01:58.604 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.591+0000 7fc430620b80 1 -- --> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631fb453e80 con 0x5631fb449c40 2023-03-09T15:01:58.604 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.591+0000 7fc430620b80 1 -- --> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631fb4b1dc0 con 0x5631fb44a550 2023-03-09T15:01:58.605 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.592+0000 7fc430620b80 10 client.0 _ll_register_callbacks cb 0x5631fb337520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-03-09T15:01:58.605 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.592+0000 7fc422233700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631fb4443f0 0x5631fb454d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T15:01:58.605 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.592+0000 7fc422233700 1 --2- >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631fb4443f0 0x5631fb454d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.38:3300/0 says I am v2:172.21.15.38:42528/0 (socket says 172.21.15.38:42528) 2023-03-09T15:01:58.605 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.592+0000 7fc422233700 1 -- 172.21.15.38:0/3534672837 learned_addr learned my addr 172.21.15.38:0/3534672837 (peer_addr_for_me v2:172.21.15.38:0/0) 2023-03-09T15:01:58.605 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.592+0000 7fc421a32700 1 --2- 172.21.15.38:0/3534672837 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5631fb449c40 0x5631fb4530c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T15:01:58.605 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.592+0000 7fc422a34700 1 --2- 172.21.15.38:0/3534672837 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5631fb44a550 0x5631fb453600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T15:01:58.605 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.592+0000 7fc422233700 1 -- 172.21.15.38:0/3534672837 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5631fb44a550 msgr2=0x5631fb453600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T15:01:58.606 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.592+0000 7fc422233700 1 --2- 172.21.15.38:0/3534672837 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5631fb44a550 0x5631fb453600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:01:58.606 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.592+0000 7fc422233700 1 -- 172.21.15.38:0/3534672837 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5631fb449c40 msgr2=0x5631fb4530c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-09T15:01:58.606 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.592+0000 7fc422233700 1 --2- 172.21.15.38:0/3534672837 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5631fb449c40 0x5631fb4530c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:01:58.606 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.592+0000 7fc422233700 1 -- 172.21.15.38:0/3534672837 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc41c10edc0 con 0x5631fb4443f0 2023-03-09T15:01:58.606 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.592+0000 7fc422233700 1 --2- 172.21.15.38:0/3534672837 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631fb4443f0 0x5631fb454d90 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fc40c003e80 tx=0x7fc40c01cfb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f13c7ebfb05c6b70 server_cookie=0 in_seq=0 out_seq=0 2023-03-09T15:01:58.606 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.592+0000 7fc405ffb700 10 client.0 ms_handle_connect on v2:172.21.15.38:3300/0 2023-03-09T15:01:58.606 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.592+0000 7fc405ffb700 1 -- 172.21.15.38:0/3534672837 <== mon.0 v2:172.21.15.38:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc40c003910 con 0x5631fb4443f0 2023-03-09T15:01:58.607 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.592+0000 7fc405ffb700 1 -- 172.21.15.38:0/3534672837 <== mon.0 v2:172.21.15.38:3300/0 2 ==== osd_map(255..255 src has 1..255) v4 ==== 9577+0+0 (secure 0 0 0) 0x7fc40c003a70 con 0x5631fb4443f0 2023-03-09T15:01:58.607 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.592+0000 7fc430620b80 1 -- 172.21.15.38:0/3534672837 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5631fb34f1b0 con 0x5631fb4443f0 2023-03-09T15:01:58.607 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.592+0000 7fc405ffb700 1 -- 172.21.15.38:0/3534672837 <== mon.0 v2:172.21.15.38:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc40c003c30 con 0x5631fb4443f0 2023-03-09T15:01:58.607 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.592+0000 7fc430620b80 10 client.16443 fetch_fsmap learned FSMap version 47 2023-03-09T15:01:58.607 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.592+0000 7fc430620b80 1 -- 172.21.15.38:0/3534672837 --> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x7fc414003ad0 con 0x5631fb4443f0 2023-03-09T15:01:58.607 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.593+0000 7fc405ffb700 1 -- 172.21.15.38:0/3534672837 <== mon.0 v2:172.21.15.38:3300/0 4 ==== config(26 keys) v1 ==== 1105+0+0 (secure 0 0 0) 0x5631fb34f1b0 con 0x5631fb4443f0 2023-03-09T15:01:58.607 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.593+0000 7fc405ffb700 1 -- 172.21.15.38:0/3534672837 <== mon.0 v2:172.21.15.38:3300/0 5 ==== fsmap(e 47) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fc40c0267b0 con 0x5631fb4443f0 2023-03-09T15:01:58.608 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.593+0000 7fc430620b80 10 client.16443 fetch_fsmap finished waiting for FSMap version 47 2023-03-09T15:01:58.608 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.593+0000 7fc430620b80 10 client.16443 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2023-03-09T15:01:58.608 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.593+0000 7fc430620b80 20 client.16443 populate_metadata read hostname 'smithi038' 2023-03-09T15:01:58.608 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.593+0000 7fc430620b80 1 --2- 172.21.15.38:0/3534672837 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5631fb4b3130 0x5631fb4b5540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-09T15:01:58.608 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.593+0000 7fc430620b80 4 client.16443 mds_command: new command op to 24530 tid=0[{"prefix": "scrub status"}] 2023-03-09T15:01:58.609 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.593+0000 7fc430620b80 1 -- 172.21.15.38:0/3534672837 --> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5631fb3391c0 con 0x5631fb4b3130 2023-03-09T15:01:58.609 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.593+0000 7fc421a32700 1 --2- 172.21.15.38:0/3534672837 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5631fb4b3130 0x5631fb4b5540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-09T15:01:58.609 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.593+0000 7fc405ffb700 10 client.16443 ms_handle_connect on v2:172.21.15.122:6838/971050165 2023-03-09T15:01:58.609 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.593+0000 7fc421a32700 1 --2- 172.21.15.38:0/3534672837 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5631fb4b3130 0x5631fb4b5540 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6bd8065e37b5a809 server_cookie=9173dc67d5494ece in_seq=0 out_seq=0 2023-03-09T15:01:58.609 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.593+0000 7fc405ffb700 1 -- 172.21.15.38:0/3534672837 <== mds.0 v2:172.21.15.122:6838/971050165 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+265 (crc 0 0 0) 0x5631fb3391c0 con 0x5631fb4b3130 2023-03-09T15:01:58.609 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.593+0000 7fc405ffb700 10 client.16443 handle_command_reply: tid=0 2023-03-09T15:01:58.610 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.593+0000 7fc430620b80 1 client.16443 shutdown 2023-03-09T15:01:58.610 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.594+0000 7fc430620b80 1 -- 172.21.15.38:0/3534672837 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631fb4443f0 msgr2=0x5631fb454d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T15:01:58.610 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.594+0000 7fc430620b80 1 --2- 172.21.15.38:0/3534672837 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631fb4443f0 0x5631fb454d90 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7fc40c003e80 tx=0x7fc40c01cfb0 comp rx=0 tx=0).stop 2023-03-09T15:01:58.610 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.594+0000 7fc430620b80 1 -- 172.21.15.38:0/3534672837 shutdown_connections 2023-03-09T15:01:58.610 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.594+0000 7fc430620b80 1 --2- 172.21.15.38:0/3534672837 >> [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] conn(0x5631fb4b3130 0x5631fb4b5540 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:01:58.610 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.594+0000 7fc430620b80 1 --2- 172.21.15.38:0/3534672837 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x5631fb4443f0 0x5631fb454d90 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:01:58.610 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.594+0000 7fc430620b80 1 --2- 172.21.15.38:0/3534672837 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x5631fb44a550 0x5631fb453600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:01:58.610 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.594+0000 7fc405ffb700 0 client.16443 ms_handle_reset on v2:172.21.15.122:6838/971050165 2023-03-09T15:01:58.611 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.594+0000 7fc430620b80 1 --2- 172.21.15.38:0/3534672837 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x5631fb449c40 0x5631fb4530c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:01:58.611 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.594+0000 7fc430620b80 1 -- 172.21.15.38:0/3534672837 >> 172.21.15.38:0/3534672837 conn(0x5631fb43aa20 msgr2=0x5631fb439c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T15:01:58.611 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.594+0000 7fc430620b80 1 -- 172.21.15.38:0/3534672837 shutdown_connections 2023-03-09T15:01:58.611 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.594+0000 7fc430620b80 1 -- 172.21.15.38:0/3534672837 wait complete. 2023-03-09T15:01:58.611 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.594+0000 7fc430620b80 20 client.16443 trim_cache size 0 max 16384 2023-03-09T15:01:58.612 INFO:teuthology.orchestra.run.smithi038.stdout:{ 2023-03-09T15:01:58.612 INFO:teuthology.orchestra.run.smithi038.stdout: "status": "scrub active (1 inodes in the stack)", 2023-03-09T15:01:58.612 INFO:teuthology.orchestra.run.smithi038.stdout: "scrubs": { 2023-03-09T15:01:58.612 INFO:teuthology.orchestra.run.smithi038.stdout: "a14659f4-d7ea-4c5a-b15b-6138af3f909f": { 2023-03-09T15:01:58.612 INFO:teuthology.orchestra.run.smithi038.stdout: "path": "/", 2023-03-09T15:01:58.612 INFO:teuthology.orchestra.run.smithi038.stdout: "tag": "a14659f4-d7ea-4c5a-b15b-6138af3f909f", 2023-03-09T15:01:58.612 INFO:teuthology.orchestra.run.smithi038.stdout: "options": "recursive,force" 2023-03-09T15:01:58.613 INFO:teuthology.orchestra.run.smithi038.stdout: } 2023-03-09T15:01:58.613 INFO:teuthology.orchestra.run.smithi038.stdout: } 2023-03-09T15:01:58.613 INFO:teuthology.orchestra.run.smithi038.stdout:} 2023-03-09T15:01:58.614 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.594+0000 7fc424497700 1 -- 172.21.15.38:0/185176896 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fc40809f520 msgr2=0x7fc4080a19d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T15:01:58.614 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.594+0000 7fc424497700 1 --2- 172.21.15.38:0/185176896 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fc40809f520 0x7fc4080a19d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fc40c009dd0 tx=0x7fc40c009260 comp rx=0 tx=0).stop 2023-03-09T15:01:58.614 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.594+0000 7fc424497700 1 -- 172.21.15.38:0/185176896 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc41c0ff9a0 msgr2=0x7fc41c197430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-09T15:01:58.614 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.594+0000 7fc424497700 1 --2- 172.21.15.38:0/185176896 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc41c0ff9a0 0x7fc41c197430 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fc418009fd0 tx=0x7fc418009580 comp rx=0 tx=0).stop 2023-03-09T15:01:58.614 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.594+0000 7fc424497700 1 -- 172.21.15.38:0/185176896 shutdown_connections 2023-03-09T15:01:58.615 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.594+0000 7fc424497700 1 --2- 172.21.15.38:0/185176896 >> [v2:172.21.15.38:6800/39994335,v1:172.21.15.38:6801/39994335] conn(0x7fc40809f520 0x7fc4080a19d0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:01:58.615 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.594+0000 7fc424497700 1 --2- 172.21.15.38:0/185176896 >> [v2:172.21.15.122:3300/0,v1:172.21.15.122:6789/0] conn(0x7fc41c104040 0x7fc41c1a1f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:01:58.615 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.594+0000 7fc424497700 1 --2- 172.21.15.38:0/185176896 >> [v2:172.21.15.181:3300/0,v1:172.21.15.181:6789/0] conn(0x7fc41c0ff000 0x7fc41c196ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:01:58.615 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.594+0000 7fc424497700 1 --2- 172.21.15.38:0/185176896 >> [v2:172.21.15.38:3300/0,v1:172.21.15.38:6789/0] conn(0x7fc41c0ff9a0 0x7fc41c197430 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-09T15:01:58.615 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.594+0000 7fc424497700 1 -- 172.21.15.38:0/185176896 >> 172.21.15.38:0/185176896 conn(0x7fc41c075c60 msgr2=0x7fc41c0fc940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-09T15:01:58.615 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.594+0000 7fc424497700 1 -- 172.21.15.38:0/185176896 shutdown_connections 2023-03-09T15:01:58.615 INFO:teuthology.orchestra.run.smithi038.stderr:2023-03-09T15:01:58.594+0000 7fc424497700 1 -- 172.21.15.38:0/185176896 wait complete. 2023-03-09T15:01:58.616 INFO:tasks.cephfs.filesystem:scrub status for tag:a14659f4-d7ea-4c5a-b15b-6138af3f909f - {'path': '/', 'tag': 'a14659f4-d7ea-4c5a-b15b-6138af3f909f', 'options': 'recursive,force'} 2023-03-09T15:01:58.616 ERROR:tasks.fwd_scrub.fs.[cephfs]:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_e0ab30a2acead6db62b144ba6b6aeb23635adae7/qa/tasks/fwd_scrub.py", line 38, in _run self.do_scrub() File "/home/teuthworker/src/git.ceph.com_ceph-c_e0ab30a2acead6db62b144ba6b6aeb23635adae7/qa/tasks/fwd_scrub.py", line 55, in do_scrub self._scrub() File "/home/teuthworker/src/git.ceph.com_ceph-c_e0ab30a2acead6db62b144ba6b6aeb23635adae7/qa/tasks/fwd_scrub.py", line 76, in _scrub done = self.fs.wait_until_scrub_complete(tag=tag, sleep=30, timeout=self.scrub_timeout) File "/home/teuthworker/src/git.ceph.com_ceph-c_e0ab30a2acead6db62b144ba6b6aeb23635adae7/qa/tasks/cephfs/filesystem.py", line 1703, in wait_until_scrub_complete while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_a2dfb0599b24ec4a1585108a2792a4567ce0a378/teuthology/contextutil.py", line 135, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (30) after waiting for 900 seconds 2023-03-09T15:01:58.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 690.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:02:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:02:00 smithi122 ceph-mon[93869]: pgmap v640: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 8.9 KiB/s wr, 6 op/s 2023-03-09T15:02:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:02:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 230.935317 secs 2023-03-09T15:02:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:02:00 smithi181 ceph-mon[94189]: pgmap v640: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 8.9 KiB/s wr, 6 op/s 2023-03-09T15:02:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:02:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 230.935317 secs 2023-03-09T15:02:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:02:00 smithi038 ceph-mon[92422]: pgmap v640: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 8.9 KiB/s wr, 6 op/s 2023-03-09T15:02:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:02:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 230.935317 secs 2023-03-09T15:02:02.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:02:02 smithi122 ceph-mon[93869]: pgmap v641: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.2 KiB/s wr, 6 op/s 2023-03-09T15:02:02.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:02:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 233.123787 secs 2023-03-09T15:02:02.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:02:02 smithi181 ceph-mon[94189]: pgmap v641: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.2 KiB/s wr, 6 op/s 2023-03-09T15:02:02.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:02:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 233.123787 secs 2023-03-09T15:02:02.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:02:02 smithi038 ceph-mon[92422]: pgmap v641: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 9.2 KiB/s wr, 6 op/s 2023-03-09T15:02:02.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:02:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 233.123787 secs 2023-03-09T15:02:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 695.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:02:04.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:02:03 smithi122 ceph-mon[93869]: pgmap v642: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 41 KiB/s wr, 11 op/s 2023-03-09T15:02:04.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:02:03 smithi181 ceph-mon[94189]: pgmap v642: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 41 KiB/s wr, 11 op/s 2023-03-09T15:02:04.257 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:02:03 smithi038 ceph-mon[92422]: pgmap v642: 129 pgs: 129 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 41 KiB/s wr, 11 op/s 2023-03-09T15:02:05.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:02:04 smithi038 ceph-mon[92422]: osdmap e256: 12 total, 12 up, 12 in 2023-03-09T15:02:05.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:02:04 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 235.935990 secs 2023-03-09T15:02:05.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:02:04 smithi122 ceph-mon[93869]: osdmap e256: 12 total, 12 up, 12 in 2023-03-09T15:02:05.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:02:04 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 235.935990 secs 2023-03-09T15:02:05.485 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:02:04 smithi181 ceph-mon[94189]: osdmap e256: 12 total, 12 up, 12 in 2023-03-09T15:02:05.486 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:02:04 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 235.935990 secs 2023-03-09T15:02:06.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:02:06 smithi122 ceph-mon[93869]: pgmap v644: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 67 KiB/s wr, 25 op/s 2023-03-09T15:02:06.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:02:06 smithi181 ceph-mon[94189]: pgmap v644: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 67 KiB/s wr, 25 op/s 2023-03-09T15:02:06.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:02:06 smithi038 ceph-mon[92422]: pgmap v644: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 67 KiB/s wr, 25 op/s 2023-03-09T15:02:07.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:02:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 238.123854 secs 2023-03-09T15:02:07.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:02:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 238.123854 secs 2023-03-09T15:02:07.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:02:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 238.123854 secs 2023-03-09T15:02:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:02:08 smithi122 ceph-mon[93869]: pgmap v645: 129 pgs: 11 active+clean+snaptrim, 35 active+clean+snaptrim_wait, 83 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 65 KiB/s wr, 25 op/s 2023-03-09T15:02:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:02:08 smithi181 ceph-mon[94189]: pgmap v645: 129 pgs: 11 active+clean+snaptrim, 35 active+clean+snaptrim_wait, 83 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 65 KiB/s wr, 25 op/s 2023-03-09T15:02:08.505 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:02:08 smithi038 ceph-mon[92422]: pgmap v645: 129 pgs: 11 active+clean+snaptrim, 35 active+clean+snaptrim_wait, 83 active+clean; 11 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 65 KiB/s wr, 25 op/s 2023-03-09T15:02:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 700.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:02:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:02:10 smithi122 ceph-mon[93869]: pgmap v646: 129 pgs: 18 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 58 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 64 KiB/s wr, 26 op/s 2023-03-09T15:02:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:02:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 240.936099 secs 2023-03-09T15:02:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:02:10 smithi181 ceph-mon[94189]: pgmap v646: 129 pgs: 18 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 58 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 64 KiB/s wr, 26 op/s 2023-03-09T15:02:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:02:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 240.936099 secs 2023-03-09T15:02:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:02:10 smithi038 ceph-mon[92422]: pgmap v646: 129 pgs: 18 active+clean+snaptrim, 53 active+clean+snaptrim_wait, 58 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 64 KiB/s wr, 26 op/s 2023-03-09T15:02:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:02:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 240.936099 secs 2023-03-09T15:02:12.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:02:12 smithi122 ceph-mon[93869]: pgmap v647: 129 pgs: 11 active+clean+snaptrim, 33 active+clean+snaptrim_wait, 85 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 64 KiB/s wr, 25 op/s 2023-03-09T15:02:12.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:02:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 243.124014 secs 2023-03-09T15:02:12.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:02:12 smithi181 ceph-mon[94189]: pgmap v647: 129 pgs: 11 active+clean+snaptrim, 33 active+clean+snaptrim_wait, 85 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 64 KiB/s wr, 25 op/s 2023-03-09T15:02:12.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:02:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 243.124014 secs 2023-03-09T15:02:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:02:12 smithi038 ceph-mon[92422]: pgmap v647: 129 pgs: 11 active+clean+snaptrim, 33 active+clean+snaptrim_wait, 85 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 64 KiB/s wr, 25 op/s 2023-03-09T15:02:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:02:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 243.124014 secs 2023-03-09T15:02:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 705.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:02:14.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:02:13 smithi122 ceph-mon[93869]: pgmap v648: 129 pgs: 2 active+clean+snaptrim, 3 active+clean+snaptrim_wait, 124 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 KiB/s wr, 21 op/s 2023-03-09T15:02:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:02:13 smithi181 ceph-mon[94189]: pgmap v648: 129 pgs: 2 active+clean+snaptrim, 3 active+clean+snaptrim_wait, 124 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 KiB/s wr, 21 op/s 2023-03-09T15:02:14.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:02:13 smithi038 ceph-mon[92422]: pgmap v648: 129 pgs: 2 active+clean+snaptrim, 3 active+clean+snaptrim_wait, 124 active+clean; 10 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 KiB/s wr, 21 op/s 2023-03-09T15:02:15.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:02:14 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 245.936159 secs 2023-03-09T15:02:15.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:02:14 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 245.936159 secs 2023-03-09T15:02:15.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:02:14 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 245.936159 secs 2023-03-09T15:02:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:02:15 smithi181 ceph-mon[94189]: pgmap v649: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 460 B/s wr, 6 op/s 2023-03-09T15:02:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:02:15 smithi038 ceph-mon[92422]: pgmap v649: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 460 B/s wr, 6 op/s 2023-03-09T15:02:16.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:02:15 smithi122 ceph-mon[93869]: pgmap v649: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 460 B/s wr, 6 op/s 2023-03-09T15:02:17.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:02:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 248.124169 secs 2023-03-09T15:02:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:02:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 248.124169 secs 2023-03-09T15:02:17.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:02:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 248.124169 secs 2023-03-09T15:02:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:02:18 smithi122 ceph-mon[93869]: pgmap v650: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 426 B/s wr, 7 op/s 2023-03-09T15:02:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:02:18 smithi122 ceph-mon[93869]: osdmap e257: 12 total, 12 up, 12 in 2023-03-09T15:02:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:02:18 smithi181 ceph-mon[94189]: pgmap v650: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 426 B/s wr, 7 op/s 2023-03-09T15:02:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:02:18 smithi181 ceph-mon[94189]: osdmap e257: 12 total, 12 up, 12 in 2023-03-09T15:02:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:02:18 smithi038 ceph-mon[92422]: pgmap v650: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 426 B/s wr, 7 op/s 2023-03-09T15:02:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:02:18 smithi038 ceph-mon[92422]: osdmap e257: 12 total, 12 up, 12 in 2023-03-09T15:02:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 710.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:02:20.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:02:20 smithi122 ceph-mon[93869]: pgmap v652: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:02:20.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:02:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 250.936224 secs 2023-03-09T15:02:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:02:20 smithi181 ceph-mon[94189]: pgmap v652: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:02:20.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:02:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 250.936224 secs 2023-03-09T15:02:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:02:20 smithi038 ceph-mon[92422]: pgmap v652: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:02:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:02:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 250.936224 secs 2023-03-09T15:02:22.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:02:22 smithi122 ceph-mon[93869]: pgmap v653: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:02:22.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:02:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 253.124311 secs 2023-03-09T15:02:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:02:22 smithi181 ceph-mon[94189]: pgmap v653: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:02:22.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:02:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 253.124311 secs 2023-03-09T15:02:22.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:02:22 smithi038 ceph-mon[92422]: pgmap v653: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:02:22.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:02:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 253.124311 secs 2023-03-09T15:02:23.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 715.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:02:24.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:02:23 smithi122 ceph-mon[93869]: pgmap v654: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-09T15:02:24.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:02:23 smithi181 ceph-mon[94189]: pgmap v654: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-09T15:02:24.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:02:23 smithi038 ceph-mon[92422]: pgmap v654: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-09T15:02:25.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:02:24 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 255.936295 secs 2023-03-09T15:02:25.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:02:24 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 255.936295 secs 2023-03-09T15:02:25.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:02:24 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 255.936295 secs 2023-03-09T15:02:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:02:25 smithi122 ceph-mon[93869]: pgmap v655: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-09T15:02:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:02:25 smithi181 ceph-mon[94189]: pgmap v655: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-09T15:02:26.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:02:25 smithi038 ceph-mon[92422]: pgmap v655: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-09T15:02:27.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:02:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 258.124445 secs 2023-03-09T15:02:27.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:02:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 258.124445 secs 2023-03-09T15:02:27.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:02:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 258.124445 secs 2023-03-09T15:02:28.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:02:28 smithi122 ceph-mon[93869]: pgmap v656: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 4.8 KiB/s wr, 7 op/s 2023-03-09T15:02:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:02:28 smithi181 ceph-mon[94189]: pgmap v656: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 4.8 KiB/s wr, 7 op/s 2023-03-09T15:02:28.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:02:28 smithi038 ceph-mon[92422]: pgmap v656: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 4.8 KiB/s wr, 7 op/s 2023-03-09T15:02:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 720.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:02:30.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:02:30 smithi122 ceph-mon[93869]: pgmap v657: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 4.3 KiB/s wr, 6 op/s 2023-03-09T15:02:30.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:02:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 260.936357 secs 2023-03-09T15:02:30.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:02:30 smithi181 ceph-mon[94189]: pgmap v657: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 4.3 KiB/s wr, 6 op/s 2023-03-09T15:02:30.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:02:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 260.936357 secs 2023-03-09T15:02:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:02:30 smithi038 ceph-mon[92422]: pgmap v657: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 4.3 KiB/s wr, 6 op/s 2023-03-09T15:02:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:02:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 260.936357 secs 2023-03-09T15:02:32.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:02:32 smithi122 ceph-mon[93869]: pgmap v658: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 4.1 KiB/s wr, 6 op/s 2023-03-09T15:02:32.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:02:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 263.124569 secs 2023-03-09T15:02:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:02:32 smithi181 ceph-mon[94189]: pgmap v658: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 4.1 KiB/s wr, 6 op/s 2023-03-09T15:02:32.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:02:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 263.124569 secs 2023-03-09T15:02:32.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:02:32 smithi038 ceph-mon[92422]: pgmap v658: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 4.1 KiB/s wr, 6 op/s 2023-03-09T15:02:32.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:02:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 263.124569 secs 2023-03-09T15:02:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 725.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:02:34.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:02:33 smithi122 ceph-mon[93869]: pgmap v659: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 22 KiB/s wr, 9 op/s 2023-03-09T15:02:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:02:33 smithi181 ceph-mon[94189]: pgmap v659: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 22 KiB/s wr, 9 op/s 2023-03-09T15:02:34.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:02:33 smithi038 ceph-mon[92422]: pgmap v659: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 22 KiB/s wr, 9 op/s 2023-03-09T15:02:35.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:02:34 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 265.936416 secs 2023-03-09T15:02:35.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:02:34 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 265.936416 secs 2023-03-09T15:02:35.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:02:34 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 265.936416 secs 2023-03-09T15:02:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:02:35 smithi038 ceph-mon[92422]: pgmap v660: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 22 KiB/s wr, 7 op/s 2023-03-09T15:02:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:02:35 smithi122 ceph-mon[93869]: pgmap v660: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 22 KiB/s wr, 7 op/s 2023-03-09T15:02:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:02:35 smithi181 ceph-mon[94189]: pgmap v660: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 22 KiB/s wr, 7 op/s 2023-03-09T15:02:37.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:02:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 268.124709 secs 2023-03-09T15:02:37.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:02:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 268.124709 secs 2023-03-09T15:02:37.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:02:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 268.124709 secs 2023-03-09T15:02:38.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:02:38 smithi122 ceph-mon[93869]: pgmap v661: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 22 KiB/s wr, 10 op/s 2023-03-09T15:02:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:02:38 smithi181 ceph-mon[94189]: pgmap v661: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 22 KiB/s wr, 10 op/s 2023-03-09T15:02:38.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:02:38 smithi038 ceph-mon[92422]: pgmap v661: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 22 KiB/s wr, 10 op/s 2023-03-09T15:02:38.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 730.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:02:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:02:40 smithi122 ceph-mon[93869]: pgmap v662: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 7 op/s 2023-03-09T15:02:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:02:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 270.936476 secs 2023-03-09T15:02:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:02:40 smithi181 ceph-mon[94189]: pgmap v662: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 7 op/s 2023-03-09T15:02:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:02:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 270.936476 secs 2023-03-09T15:02:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:02:40 smithi038 ceph-mon[92422]: pgmap v662: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 7 op/s 2023-03-09T15:02:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:02:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 270.936476 secs 2023-03-09T15:02:42.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:02:42 smithi122 ceph-mon[93869]: pgmap v663: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 18 KiB/s wr, 7 op/s 2023-03-09T15:02:42.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:02:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 273.124823 secs 2023-03-09T15:02:42.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:02:42 smithi181 ceph-mon[94189]: pgmap v663: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 18 KiB/s wr, 7 op/s 2023-03-09T15:02:42.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:02:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 273.124823 secs 2023-03-09T15:02:42.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:02:42 smithi038 ceph-mon[92422]: pgmap v663: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 18 KiB/s wr, 7 op/s 2023-03-09T15:02:42.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:02:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 273.124823 secs 2023-03-09T15:02:43.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 735.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:02:44.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:02:43 smithi122 ceph-mon[93869]: pgmap v664: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 9 op/s 2023-03-09T15:02:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:02:43 smithi181 ceph-mon[94189]: pgmap v664: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 9 op/s 2023-03-09T15:02:44.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:02:43 smithi038 ceph-mon[92422]: pgmap v664: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 9 op/s 2023-03-09T15:02:45.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:02:44 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 275.936544 secs 2023-03-09T15:02:45.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:02:44 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 275.936544 secs 2023-03-09T15:02:45.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:02:44 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 275.936544 secs 2023-03-09T15:02:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:02:45 smithi038 ceph-mon[92422]: pgmap v665: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 5.3 KiB/s wr, 5 op/s 2023-03-09T15:02:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:02:45 smithi122 ceph-mon[93869]: pgmap v665: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 5.3 KiB/s wr, 5 op/s 2023-03-09T15:02:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:02:45 smithi181 ceph-mon[94189]: pgmap v665: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 5.3 KiB/s wr, 5 op/s 2023-03-09T15:02:47.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:02:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 278.124955 secs 2023-03-09T15:02:47.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:02:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 278.124955 secs 2023-03-09T15:02:47.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:02:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 278.124955 secs 2023-03-09T15:02:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:02:48 smithi122 ceph-mon[93869]: pgmap v666: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 5.2 KiB/s wr, 7 op/s 2023-03-09T15:02:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:02:48 smithi181 ceph-mon[94189]: pgmap v666: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 5.2 KiB/s wr, 7 op/s 2023-03-09T15:02:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:02:48 smithi038 ceph-mon[92422]: pgmap v666: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 5.2 KiB/s wr, 7 op/s 2023-03-09T15:02:48.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 740.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:02:50.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:02:50 smithi122 ceph-mon[93869]: pgmap v667: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 5.1 KiB/s wr, 5 op/s 2023-03-09T15:02:50.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:02:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 280.936624 secs 2023-03-09T15:02:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:02:50 smithi181 ceph-mon[94189]: pgmap v667: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 5.1 KiB/s wr, 5 op/s 2023-03-09T15:02:50.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:02:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 280.936624 secs 2023-03-09T15:02:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:02:50 smithi038 ceph-mon[92422]: pgmap v667: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 5.1 KiB/s wr, 5 op/s 2023-03-09T15:02:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:02:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 280.936624 secs 2023-03-09T15:02:52.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:02:52 smithi122 ceph-mon[93869]: pgmap v668: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 5.1 KiB/s wr, 5 op/s 2023-03-09T15:02:52.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:02:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 283.125073 secs 2023-03-09T15:02:52.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:02:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:02:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:02:52 smithi181 ceph-mon[94189]: pgmap v668: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 5.1 KiB/s wr, 5 op/s 2023-03-09T15:02:52.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:02:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 283.125073 secs 2023-03-09T15:02:52.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:02:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:02:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:02:52 smithi038 ceph-mon[92422]: pgmap v668: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 5.1 KiB/s wr, 5 op/s 2023-03-09T15:02:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:02:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 283.125073 secs 2023-03-09T15:02:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:02:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:02:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:02:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:02:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:02:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:02:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:02:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:02:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:02:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:02:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:02:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:02:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:02:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:02:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:02:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:02:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:02:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:02:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:02:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:02:53.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 745.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:02:54.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:02:54 smithi122 ceph-mon[93869]: pgmap v669: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 5.1 KiB/s wr, 7 op/s 2023-03-09T15:02:54.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:02:54 smithi181 ceph-mon[94189]: pgmap v669: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 5.1 KiB/s wr, 7 op/s 2023-03-09T15:02:54.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:02:54 smithi038 ceph-mon[92422]: pgmap v669: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 5.1 KiB/s wr, 7 op/s 2023-03-09T15:02:55.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:02:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 285.936685 secs 2023-03-09T15:02:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:02:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 285.936685 secs 2023-03-09T15:02:55.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:02:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 285.936685 secs 2023-03-09T15:02:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:02:56 smithi122 ceph-mon[93869]: pgmap v670: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T15:02:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:02:56 smithi181 ceph-mon[94189]: pgmap v670: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T15:02:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:02:56 smithi038 ceph-mon[92422]: pgmap v670: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T15:02:57.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:02:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 288.125191 secs 2023-03-09T15:02:57.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:02:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 288.125191 secs 2023-03-09T15:02:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:02:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 288.125191 secs 2023-03-09T15:02:58.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:02:58 smithi122 ceph-mon[93869]: pgmap v671: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2023-03-09T15:02:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:02:58 smithi181 ceph-mon[94189]: pgmap v671: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2023-03-09T15:02:58.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:02:58 smithi038 ceph-mon[92422]: pgmap v671: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2023-03-09T15:02:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 750.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:03:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:03:00 smithi122 ceph-mon[93869]: pgmap v672: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 2.2 KiB/s wr, 6 op/s 2023-03-09T15:03:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:03:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 290.936752 secs 2023-03-09T15:03:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:03:00 smithi181 ceph-mon[94189]: pgmap v672: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 2.2 KiB/s wr, 6 op/s 2023-03-09T15:03:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:03:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 290.936752 secs 2023-03-09T15:03:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:03:00 smithi038 ceph-mon[92422]: pgmap v672: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 2.2 KiB/s wr, 6 op/s 2023-03-09T15:03:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:03:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 290.936752 secs 2023-03-09T15:03:02.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:03:02 smithi122 ceph-mon[93869]: pgmap v673: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 3.3 KiB/s wr, 6 op/s 2023-03-09T15:03:02.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:03:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 293.125273 secs 2023-03-09T15:03:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:03:02 smithi181 ceph-mon[94189]: pgmap v673: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 3.3 KiB/s wr, 6 op/s 2023-03-09T15:03:02.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:03:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 293.125273 secs 2023-03-09T15:03:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:03:02 smithi038 ceph-mon[92422]: pgmap v673: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 3.3 KiB/s wr, 6 op/s 2023-03-09T15:03:02.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:03:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 293.125273 secs 2023-03-09T15:03:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 755.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:03:04.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:03:03 smithi181 ceph-mon[94189]: pgmap v674: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 35 KiB/s wr, 11 op/s 2023-03-09T15:03:04.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:03:03 smithi038 ceph-mon[92422]: pgmap v674: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 35 KiB/s wr, 11 op/s 2023-03-09T15:03:04.483 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:03:03 smithi122 ceph-mon[93869]: pgmap v674: 129 pgs: 129 active+clean; 10 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 35 KiB/s wr, 11 op/s 2023-03-09T15:03:05.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:03:04 smithi181 ceph-mon[94189]: osdmap e258: 12 total, 12 up, 12 in 2023-03-09T15:03:05.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:03:04 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 295.942758 secs 2023-03-09T15:03:05.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:03:04 smithi038 ceph-mon[92422]: osdmap e258: 12 total, 12 up, 12 in 2023-03-09T15:03:05.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:03:04 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 295.942758 secs 2023-03-09T15:03:05.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:03:04 smithi122 ceph-mon[93869]: osdmap e258: 12 total, 12 up, 12 in 2023-03-09T15:03:05.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:03:04 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 295.942758 secs 2023-03-09T15:03:06.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:03:06 smithi122 ceph-mon[93869]: pgmap v676: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 9.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 68 KiB/s wr, 26 op/s 2023-03-09T15:03:06.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:03:06 smithi181 ceph-mon[94189]: pgmap v676: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 9.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 68 KiB/s wr, 26 op/s 2023-03-09T15:03:06.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:03:06 smithi038 ceph-mon[92422]: pgmap v676: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 9.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 68 KiB/s wr, 26 op/s 2023-03-09T15:03:07.482 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:03:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 298.125344 secs 2023-03-09T15:03:07.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:03:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 298.125344 secs 2023-03-09T15:03:07.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:03:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 298.125344 secs 2023-03-09T15:03:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:03:08 smithi122 ceph-mon[93869]: pgmap v677: 129 pgs: 12 active+clean+snaptrim, 36 active+clean+snaptrim_wait, 81 active+clean; 9.8 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 65 KiB/s wr, 25 op/s 2023-03-09T15:03:08.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:03:08 smithi181 ceph-mon[94189]: pgmap v677: 129 pgs: 12 active+clean+snaptrim, 36 active+clean+snaptrim_wait, 81 active+clean; 9.8 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 65 KiB/s wr, 25 op/s 2023-03-09T15:03:08.507 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:03:08 smithi038 ceph-mon[92422]: pgmap v677: 129 pgs: 12 active+clean+snaptrim, 36 active+clean+snaptrim_wait, 81 active+clean; 9.8 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 65 KiB/s wr, 25 op/s 2023-03-09T15:03:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 760.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:03:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:03:10 smithi122 ceph-mon[93869]: pgmap v678: 129 pgs: 19 active+clean+snaptrim, 54 active+clean+snaptrim_wait, 56 active+clean; 9.6 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 66 KiB/s wr, 26 op/s 2023-03-09T15:03:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:03:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 300.942822 secs 2023-03-09T15:03:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:03:10 smithi181 ceph-mon[94189]: pgmap v678: 129 pgs: 19 active+clean+snaptrim, 54 active+clean+snaptrim_wait, 56 active+clean; 9.6 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 66 KiB/s wr, 26 op/s 2023-03-09T15:03:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:03:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 300.942822 secs 2023-03-09T15:03:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:03:10 smithi038 ceph-mon[92422]: pgmap v678: 129 pgs: 19 active+clean+snaptrim, 54 active+clean+snaptrim_wait, 56 active+clean; 9.6 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 66 KiB/s wr, 26 op/s 2023-03-09T15:03:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:03:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 300.942822 secs 2023-03-09T15:03:12.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:03:12 smithi122 ceph-mon[93869]: pgmap v679: 129 pgs: 12 active+clean+snaptrim, 33 active+clean+snaptrim_wait, 84 active+clean; 9.4 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 64 KiB/s wr, 26 op/s 2023-03-09T15:03:12.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:03:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 303.125482 secs 2023-03-09T15:03:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:03:12 smithi181 ceph-mon[94189]: pgmap v679: 129 pgs: 12 active+clean+snaptrim, 33 active+clean+snaptrim_wait, 84 active+clean; 9.4 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 64 KiB/s wr, 26 op/s 2023-03-09T15:03:12.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:03:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 303.125482 secs 2023-03-09T15:03:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:03:12 smithi038 ceph-mon[92422]: pgmap v679: 129 pgs: 12 active+clean+snaptrim, 33 active+clean+snaptrim_wait, 84 active+clean; 9.4 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 64 KiB/s wr, 26 op/s 2023-03-09T15:03:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:03:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 303.125482 secs 2023-03-09T15:03:13.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 765.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:03:14.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:03:13 smithi038 ceph-mon[92422]: pgmap v680: 129 pgs: 2 active+clean+snaptrim, 3 active+clean+snaptrim_wait, 124 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 KiB/s wr, 21 op/s 2023-03-09T15:03:14.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:03:13 smithi122 ceph-mon[93869]: pgmap v680: 129 pgs: 2 active+clean+snaptrim, 3 active+clean+snaptrim_wait, 124 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 KiB/s wr, 21 op/s 2023-03-09T15:03:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:03:13 smithi181 ceph-mon[94189]: pgmap v680: 129 pgs: 2 active+clean+snaptrim, 3 active+clean+snaptrim_wait, 124 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 KiB/s wr, 21 op/s 2023-03-09T15:03:15.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:03:14 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 305.942881 secs 2023-03-09T15:03:15.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:03:14 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 305.942881 secs 2023-03-09T15:03:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:03:14 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 305.942881 secs 2023-03-09T15:03:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:03:15 smithi038 ceph-mon[92422]: pgmap v681: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 368 B/s wr, 6 op/s 2023-03-09T15:03:16.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:03:15 smithi122 ceph-mon[93869]: pgmap v681: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 368 B/s wr, 6 op/s 2023-03-09T15:03:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:03:15 smithi181 ceph-mon[94189]: pgmap v681: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 368 B/s wr, 6 op/s 2023-03-09T15:03:17.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:03:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 308.125573 secs 2023-03-09T15:03:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:03:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 308.125573 secs 2023-03-09T15:03:17.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:03:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 308.125573 secs 2023-03-09T15:03:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:03:18 smithi122 ceph-mon[93869]: pgmap v682: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 341 B/s wr, 8 op/s 2023-03-09T15:03:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:03:18 smithi122 ceph-mon[93869]: osdmap e259: 12 total, 12 up, 12 in 2023-03-09T15:03:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:03:18 smithi181 ceph-mon[94189]: pgmap v682: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 341 B/s wr, 8 op/s 2023-03-09T15:03:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:03:18 smithi181 ceph-mon[94189]: osdmap e259: 12 total, 12 up, 12 in 2023-03-09T15:03:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:03:18 smithi038 ceph-mon[92422]: pgmap v682: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 341 B/s wr, 8 op/s 2023-03-09T15:03:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:03:18 smithi038 ceph-mon[92422]: osdmap e259: 12 total, 12 up, 12 in 2023-03-09T15:03:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 770.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:03:20.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:03:20 smithi122 ceph-mon[93869]: pgmap v684: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:03:20.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:03:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 310.942959 secs 2023-03-09T15:03:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:03:20 smithi181 ceph-mon[94189]: pgmap v684: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:03:20.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:03:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 310.942959 secs 2023-03-09T15:03:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:03:20 smithi038 ceph-mon[92422]: pgmap v684: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:03:20.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:03:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 310.942959 secs 2023-03-09T15:03:22.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:03:22 smithi122 ceph-mon[93869]: pgmap v685: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:03:22.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:03:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 313.125695 secs 2023-03-09T15:03:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:03:22 smithi181 ceph-mon[94189]: pgmap v685: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:03:22.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:03:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 313.125695 secs 2023-03-09T15:03:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:03:22 smithi038 ceph-mon[92422]: pgmap v685: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:03:22.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:03:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 313.125695 secs 2023-03-09T15:03:23.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 775.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:03:24.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:03:23 smithi038 ceph-mon[92422]: pgmap v686: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-09T15:03:24.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:03:23 smithi122 ceph-mon[93869]: pgmap v686: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-09T15:03:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:03:23 smithi181 ceph-mon[94189]: pgmap v686: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-09T15:03:25.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:03:24 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 315.943017 secs 2023-03-09T15:03:25.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:03:24 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 315.943017 secs 2023-03-09T15:03:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:03:24 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 315.943017 secs 2023-03-09T15:03:26.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:03:26 smithi122 ceph-mon[93869]: pgmap v687: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-09T15:03:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:03:26 smithi181 ceph-mon[94189]: pgmap v687: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-09T15:03:26.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:03:26 smithi038 ceph-mon[92422]: pgmap v687: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-09T15:03:27.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:03:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 318.125812 secs 2023-03-09T15:03:27.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:03:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 318.125812 secs 2023-03-09T15:03:27.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:03:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 318.125812 secs 2023-03-09T15:03:28.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:03:28 smithi122 ceph-mon[93869]: pgmap v688: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 6.8 KiB/s wr, 7 op/s 2023-03-09T15:03:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:03:28 smithi181 ceph-mon[94189]: pgmap v688: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 6.8 KiB/s wr, 7 op/s 2023-03-09T15:03:28.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:03:28 smithi038 ceph-mon[92422]: pgmap v688: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 6.8 KiB/s wr, 7 op/s 2023-03-09T15:03:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 780.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:03:30.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:03:30 smithi122 ceph-mon[93869]: pgmap v689: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 13 KiB/s wr, 7 op/s 2023-03-09T15:03:30.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:03:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 320.943077 secs 2023-03-09T15:03:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:03:30 smithi181 ceph-mon[94189]: pgmap v689: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 13 KiB/s wr, 7 op/s 2023-03-09T15:03:30.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:03:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 320.943077 secs 2023-03-09T15:03:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:03:30 smithi038 ceph-mon[92422]: pgmap v689: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 13 KiB/s wr, 7 op/s 2023-03-09T15:03:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:03:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 320.943077 secs 2023-03-09T15:03:32.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:03:32 smithi122 ceph-mon[93869]: pgmap v690: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 12 KiB/s wr, 7 op/s 2023-03-09T15:03:32.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:03:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 323.125947 secs 2023-03-09T15:03:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:03:32 smithi181 ceph-mon[94189]: pgmap v690: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 12 KiB/s wr, 7 op/s 2023-03-09T15:03:32.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:03:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 323.125947 secs 2023-03-09T15:03:32.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:03:32 smithi038 ceph-mon[92422]: pgmap v690: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 12 KiB/s wr, 7 op/s 2023-03-09T15:03:32.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:03:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 323.125947 secs 2023-03-09T15:03:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 785.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:03:34.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:03:33 smithi038 ceph-mon[92422]: pgmap v691: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 19 KiB/s wr, 9 op/s 2023-03-09T15:03:34.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:03:33 smithi122 ceph-mon[93869]: pgmap v691: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 19 KiB/s wr, 9 op/s 2023-03-09T15:03:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:03:33 smithi181 ceph-mon[94189]: pgmap v691: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 19 KiB/s wr, 9 op/s 2023-03-09T15:03:35.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:03:34 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 325.943137 secs 2023-03-09T15:03:35.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:03:34 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 325.943137 secs 2023-03-09T15:03:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:03:34 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 325.943137 secs 2023-03-09T15:03:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:03:36 smithi122 ceph-mon[93869]: pgmap v692: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 22 KiB/s wr, 8 op/s 2023-03-09T15:03:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:03:36 smithi181 ceph-mon[94189]: pgmap v692: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 22 KiB/s wr, 8 op/s 2023-03-09T15:03:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:03:36 smithi038 ceph-mon[92422]: pgmap v692: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 22 KiB/s wr, 8 op/s 2023-03-09T15:03:37.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:03:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 328.126030 secs 2023-03-09T15:03:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:03:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 328.126030 secs 2023-03-09T15:03:37.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:03:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 328.126030 secs 2023-03-09T15:03:38.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:03:38 smithi122 ceph-mon[93869]: pgmap v693: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 25 KiB/s wr, 11 op/s 2023-03-09T15:03:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:03:38 smithi181 ceph-mon[94189]: pgmap v693: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 25 KiB/s wr, 11 op/s 2023-03-09T15:03:38.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:03:38 smithi038 ceph-mon[92422]: pgmap v693: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 25 KiB/s wr, 11 op/s 2023-03-09T15:03:38.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 790.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:03:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:03:40 smithi122 ceph-mon[93869]: pgmap v694: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 8 op/s 2023-03-09T15:03:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:03:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 330.943197 secs 2023-03-09T15:03:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:03:40 smithi181 ceph-mon[94189]: pgmap v694: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 8 op/s 2023-03-09T15:03:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:03:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 330.943197 secs 2023-03-09T15:03:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:03:40 smithi038 ceph-mon[92422]: pgmap v694: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 8 op/s 2023-03-09T15:03:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:03:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 330.943197 secs 2023-03-09T15:03:42.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:03:42 smithi122 ceph-mon[93869]: pgmap v695: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 19 KiB/s wr, 8 op/s 2023-03-09T15:03:42.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:03:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 333.126148 secs 2023-03-09T15:03:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:03:42 smithi181 ceph-mon[94189]: pgmap v695: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 19 KiB/s wr, 8 op/s 2023-03-09T15:03:42.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:03:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 333.126148 secs 2023-03-09T15:03:42.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:03:42 smithi038 ceph-mon[92422]: pgmap v695: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 19 KiB/s wr, 8 op/s 2023-03-09T15:03:42.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:03:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 333.126148 secs 2023-03-09T15:03:43.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 795.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:03:44.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:03:43 smithi038 ceph-mon[92422]: pgmap v696: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 10 op/s 2023-03-09T15:03:44.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:03:43 smithi122 ceph-mon[93869]: pgmap v696: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 10 op/s 2023-03-09T15:03:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:03:43 smithi181 ceph-mon[94189]: pgmap v696: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 10 op/s 2023-03-09T15:03:45.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:03:44 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 335.943260 secs 2023-03-09T15:03:45.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:03:44 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 335.943260 secs 2023-03-09T15:03:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:03:44 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 335.943260 secs 2023-03-09T15:03:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:03:45 smithi038 ceph-mon[92422]: pgmap v697: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 21 KiB/s wr, 8 op/s 2023-03-09T15:03:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:03:46 smithi122 ceph-mon[93869]: pgmap v697: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 21 KiB/s wr, 8 op/s 2023-03-09T15:03:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:03:45 smithi181 ceph-mon[94189]: pgmap v697: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 21 KiB/s wr, 8 op/s 2023-03-09T15:03:47.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:03:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 338.126292 secs 2023-03-09T15:03:47.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:03:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 338.126292 secs 2023-03-09T15:03:47.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:03:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 338.126292 secs 2023-03-09T15:03:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:03:48 smithi122 ceph-mon[93869]: pgmap v698: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 17 KiB/s wr, 10 op/s 2023-03-09T15:03:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:03:48 smithi181 ceph-mon[94189]: pgmap v698: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 17 KiB/s wr, 10 op/s 2023-03-09T15:03:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:03:48 smithi038 ceph-mon[92422]: pgmap v698: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 17 KiB/s wr, 10 op/s 2023-03-09T15:03:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 800.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:03:50.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:03:50 smithi122 ceph-mon[93869]: pgmap v699: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 15 KiB/s wr, 7 op/s 2023-03-09T15:03:50.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:03:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 340.943338 secs 2023-03-09T15:03:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:03:50 smithi181 ceph-mon[94189]: pgmap v699: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 15 KiB/s wr, 7 op/s 2023-03-09T15:03:50.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:03:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 340.943338 secs 2023-03-09T15:03:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:03:50 smithi038 ceph-mon[92422]: pgmap v699: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 15 KiB/s wr, 7 op/s 2023-03-09T15:03:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:03:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 340.943338 secs 2023-03-09T15:03:52.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:03:52 smithi122 ceph-mon[93869]: pgmap v700: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T15:03:52.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:03:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 343.126435 secs 2023-03-09T15:03:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:03:52 smithi181 ceph-mon[94189]: pgmap v700: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T15:03:52.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:03:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 343.126435 secs 2023-03-09T15:03:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:03:52 smithi038 ceph-mon[92422]: pgmap v700: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T15:03:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:03:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 343.126435 secs 2023-03-09T15:03:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:03:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:03:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:03:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:03:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:03:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:03:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 805.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:03:54.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:03:54 smithi122 ceph-mon[93869]: pgmap v701: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 8.5 KiB/s wr, 8 op/s 2023-03-09T15:03:54.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:03:54 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:03:54.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:03:54 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:03:54.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:03:54 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:03:54.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:03:54 smithi181 ceph-mon[94189]: pgmap v701: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 8.5 KiB/s wr, 8 op/s 2023-03-09T15:03:54.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:03:54 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:03:54.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:03:54 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:03:54.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:03:54 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:03:54.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:03:54 smithi038 ceph-mon[92422]: pgmap v701: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 8.5 KiB/s wr, 8 op/s 2023-03-09T15:03:54.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:03:54 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:03:54.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:03:54 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:03:54.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:03:54 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:03:55.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:03:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 345.943397 secs 2023-03-09T15:03:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:03:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 345.943397 secs 2023-03-09T15:03:55.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:03:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 345.943397 secs 2023-03-09T15:03:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:03:56 smithi122 ceph-mon[93869]: pgmap v702: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.3 KiB/s wr, 5 op/s 2023-03-09T15:03:56.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:03:56 smithi181 ceph-mon[94189]: pgmap v702: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.3 KiB/s wr, 5 op/s 2023-03-09T15:03:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:03:56 smithi038 ceph-mon[92422]: pgmap v702: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.3 KiB/s wr, 5 op/s 2023-03-09T15:03:57.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:03:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 348.126575 secs 2023-03-09T15:03:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:03:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 348.126575 secs 2023-03-09T15:03:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:03:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 348.126575 secs 2023-03-09T15:03:58.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:03:58 smithi122 ceph-mon[93869]: pgmap v703: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 7 op/s 2023-03-09T15:03:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:03:58 smithi181 ceph-mon[94189]: pgmap v703: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 7 op/s 2023-03-09T15:03:58.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:03:58 smithi038 ceph-mon[92422]: pgmap v703: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 7 op/s 2023-03-09T15:03:58.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 810.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:04:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:04:00 smithi122 ceph-mon[93869]: pgmap v704: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2023-03-09T15:04:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:04:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 350.943456 secs 2023-03-09T15:04:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:04:00 smithi181 ceph-mon[94189]: pgmap v704: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2023-03-09T15:04:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:04:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 350.943456 secs 2023-03-09T15:04:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:04:00 smithi038 ceph-mon[92422]: pgmap v704: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2023-03-09T15:04:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:04:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 350.943456 secs 2023-03-09T15:04:02.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:04:02 smithi122 ceph-mon[93869]: pgmap v705: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-09T15:04:02.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:04:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 353.126694 secs 2023-03-09T15:04:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:04:02 smithi181 ceph-mon[94189]: pgmap v705: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-09T15:04:02.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:04:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 353.126694 secs 2023-03-09T15:04:02.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:04:02 smithi038 ceph-mon[92422]: pgmap v705: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-09T15:04:02.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:04:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 353.126694 secs 2023-03-09T15:04:03.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 815.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:04:04.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:04:03 smithi038 ceph-mon[92422]: pgmap v706: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 22 KiB/s wr, 8 op/s 2023-03-09T15:04:04.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:04:03 smithi122 ceph-mon[93869]: pgmap v706: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 22 KiB/s wr, 8 op/s 2023-03-09T15:04:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:04:03 smithi181 ceph-mon[94189]: pgmap v706: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 22 KiB/s wr, 8 op/s 2023-03-09T15:04:05.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:04:04 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 355.943520 secs 2023-03-09T15:04:05.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:04:04 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 355.943520 secs 2023-03-09T15:04:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:04:04 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 355.943520 secs 2023-03-09T15:04:06.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:04:06 smithi122 ceph-mon[93869]: pgmap v707: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 20 op/s 2023-03-09T15:04:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:04:06 smithi181 ceph-mon[94189]: pgmap v707: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 20 op/s 2023-03-09T15:04:06.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:04:06 smithi038 ceph-mon[92422]: pgmap v707: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 20 op/s 2023-03-09T15:04:07.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:04:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 358.126812 secs 2023-03-09T15:04:07.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:04:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 358.126812 secs 2023-03-09T15:04:07.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:04:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 358.126812 secs 2023-03-09T15:04:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:04:08 smithi122 ceph-mon[93869]: pgmap v708: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2023-03-09T15:04:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:04:08 smithi181 ceph-mon[94189]: pgmap v708: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2023-03-09T15:04:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:04:08 smithi038 ceph-mon[92422]: pgmap v708: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2023-03-09T15:04:08.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 820.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:04:10.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:04:10 smithi122 ceph-mon[93869]: pgmap v709: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 20 op/s 2023-03-09T15:04:10.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:04:10 smithi122 ceph-mon[93869]: osdmap e260: 12 total, 12 up, 12 in 2023-03-09T15:04:10.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:04:10 smithi122 ceph-mon[93869]: 2 slow requests, 2 included below; oldest blocked for > 360.943588 secs 2023-03-09T15:04:10.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:04:10 smithi122 ceph-mon[93869]: slow request 360.943588 seconds old, received at 2023-03-09T14:58:08.564667+0000: peer_request:mds.2:599 currently acquired locks 2023-03-09T15:04:10.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:04:10 smithi122 ceph-mon[93869]: slow request 360.932887 seconds old, received at 2023-03-09T14:58:08.575367+0000: peer_request(client.15375:6773.0 authpin) currently dispatched 2023-03-09T15:04:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:04:10 smithi181 ceph-mon[94189]: pgmap v709: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 20 op/s 2023-03-09T15:04:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:04:10 smithi181 ceph-mon[94189]: osdmap e260: 12 total, 12 up, 12 in 2023-03-09T15:04:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:04:10 smithi181 ceph-mon[94189]: 2 slow requests, 2 included below; oldest blocked for > 360.943588 secs 2023-03-09T15:04:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:04:10 smithi181 ceph-mon[94189]: slow request 360.943588 seconds old, received at 2023-03-09T14:58:08.564667+0000: peer_request:mds.2:599 currently acquired locks 2023-03-09T15:04:10.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:04:10 smithi181 ceph-mon[94189]: slow request 360.932887 seconds old, received at 2023-03-09T14:58:08.575367+0000: peer_request(client.15375:6773.0 authpin) currently dispatched 2023-03-09T15:04:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:04:10 smithi038 ceph-mon[92422]: pgmap v709: 129 pgs: 129 active+clean; 9.2 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 20 op/s 2023-03-09T15:04:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:04:10 smithi038 ceph-mon[92422]: osdmap e260: 12 total, 12 up, 12 in 2023-03-09T15:04:10.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:04:10 smithi038 ceph-mon[92422]: 2 slow requests, 2 included below; oldest blocked for > 360.943588 secs 2023-03-09T15:04:10.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:04:10 smithi038 ceph-mon[92422]: slow request 360.943588 seconds old, received at 2023-03-09T14:58:08.564667+0000: peer_request:mds.2:599 currently acquired locks 2023-03-09T15:04:10.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:04:10 smithi038 ceph-mon[92422]: slow request 360.932887 seconds old, received at 2023-03-09T14:58:08.575367+0000: peer_request(client.15375:6773.0 authpin) currently dispatched 2023-03-09T15:04:12.503 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:04:12 smithi038 ceph-mon[92422]: pgmap v711: 129 pgs: 5 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 111 active+clean; 9.1 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 24 op/s 2023-03-09T15:04:12.503 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:04:12 smithi038 ceph-mon[92422]: 2 slow requests, 2 included below; oldest blocked for > 363.126883 secs 2023-03-09T15:04:12.503 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:04:12 smithi038 ceph-mon[92422]: slow request 363.126883 seconds old, received at 2023-03-09T14:58:08.560137+0000: client_request(mds.2:599 rename #0x10000000007/0000000100000001000000EB #0x60f/1000000062a caller_uid=0, caller_gid=0{}) currently acquired locks 2023-03-09T15:04:12.503 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:04:12 smithi038 ceph-mon[92422]: slow request 363.112503 seconds old, received at 2023-03-09T14:58:08.574517+0000: client_request(client.15375:6773 unlink #0x10000000007/0000000100000001000000EB 2023-03-09T14:58:08.569481+0000 caller_uid=26, caller_gid=26{26,}) currently requesting remote authpins 2023-03-09T15:04:12.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:04:12 smithi122 ceph-mon[93869]: pgmap v711: 129 pgs: 5 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 111 active+clean; 9.1 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 24 op/s 2023-03-09T15:04:12.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:04:12 smithi122 ceph-mon[93869]: 2 slow requests, 2 included below; oldest blocked for > 363.126883 secs 2023-03-09T15:04:12.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:04:12 smithi122 ceph-mon[93869]: slow request 363.126883 seconds old, received at 2023-03-09T14:58:08.560137+0000: client_request(mds.2:599 rename #0x10000000007/0000000100000001000000EB #0x60f/1000000062a caller_uid=0, caller_gid=0{}) currently acquired locks 2023-03-09T15:04:12.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:04:12 smithi122 ceph-mon[93869]: slow request 363.112503 seconds old, received at 2023-03-09T14:58:08.574517+0000: client_request(client.15375:6773 unlink #0x10000000007/0000000100000001000000EB 2023-03-09T14:58:08.569481+0000 caller_uid=26, caller_gid=26{26,}) currently requesting remote authpins 2023-03-09T15:04:12.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:04:12 smithi181 ceph-mon[94189]: pgmap v711: 129 pgs: 5 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 111 active+clean; 9.1 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 24 op/s 2023-03-09T15:04:12.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:04:12 smithi181 ceph-mon[94189]: 2 slow requests, 2 included below; oldest blocked for > 363.126883 secs 2023-03-09T15:04:12.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:04:12 smithi181 ceph-mon[94189]: slow request 363.126883 seconds old, received at 2023-03-09T14:58:08.560137+0000: client_request(mds.2:599 rename #0x10000000007/0000000100000001000000EB #0x60f/1000000062a caller_uid=0, caller_gid=0{}) currently acquired locks 2023-03-09T15:04:12.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:04:12 smithi181 ceph-mon[94189]: slow request 363.112503 seconds old, received at 2023-03-09T14:58:08.574517+0000: client_request(client.15375:6773 unlink #0x10000000007/0000000100000001000000EB 2023-03-09T14:58:08.569481+0000 caller_uid=26, caller_gid=26{26,}) currently requesting remote authpins 2023-03-09T15:04:13.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 825.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:04:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:04:14 smithi181 ceph-mon[94189]: pgmap v712: 129 pgs: 17 active+clean+snaptrim, 49 active+clean+snaptrim_wait, 63 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 27 KiB/s wr, 21 op/s 2023-03-09T15:04:14.482 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:04:14 smithi122 ceph-mon[93869]: pgmap v712: 129 pgs: 17 active+clean+snaptrim, 49 active+clean+snaptrim_wait, 63 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 27 KiB/s wr, 21 op/s 2023-03-09T15:04:14.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:04:14 smithi038 ceph-mon[92422]: pgmap v712: 129 pgs: 17 active+clean+snaptrim, 49 active+clean+snaptrim_wait, 63 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 27 KiB/s wr, 21 op/s 2023-03-09T15:04:15.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:04:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 365.943661 secs 2023-03-09T15:04:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:04:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 365.943661 secs 2023-03-09T15:04:15.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:04:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 365.943661 secs 2023-03-09T15:04:16.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:04:16 smithi122 ceph-mon[93869]: pgmap v713: 129 pgs: 17 active+clean+snaptrim, 46 active+clean+snaptrim_wait, 66 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 306 B/s wr, 6 op/s 2023-03-09T15:04:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:04:16 smithi181 ceph-mon[94189]: pgmap v713: 129 pgs: 17 active+clean+snaptrim, 46 active+clean+snaptrim_wait, 66 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 306 B/s wr, 6 op/s 2023-03-09T15:04:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:04:16 smithi038 ceph-mon[92422]: pgmap v713: 129 pgs: 17 active+clean+snaptrim, 46 active+clean+snaptrim_wait, 66 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 306 B/s wr, 6 op/s 2023-03-09T15:04:17.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:04:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 368.126998 secs 2023-03-09T15:04:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:04:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 368.126998 secs 2023-03-09T15:04:17.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:04:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 368.126998 secs 2023-03-09T15:04:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:04:18 smithi122 ceph-mon[93869]: pgmap v714: 129 pgs: 7 active+clean+snaptrim, 16 active+clean+snaptrim_wait, 106 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:04:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:04:18 smithi181 ceph-mon[94189]: pgmap v714: 129 pgs: 7 active+clean+snaptrim, 16 active+clean+snaptrim_wait, 106 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:04:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:04:18 smithi038 ceph-mon[92422]: pgmap v714: 129 pgs: 7 active+clean+snaptrim, 16 active+clean+snaptrim_wait, 106 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:04:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 830.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:04:20.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:04:20 smithi122 ceph-mon[93869]: pgmap v715: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:04:20.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:04:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 370.943720 secs 2023-03-09T15:04:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:04:20 smithi181 ceph-mon[94189]: pgmap v715: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:04:20.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:04:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 370.943720 secs 2023-03-09T15:04:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:04:20 smithi038 ceph-mon[92422]: pgmap v715: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:04:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:04:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 370.943720 secs 2023-03-09T15:04:22.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:04:22 smithi122 ceph-mon[93869]: pgmap v716: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 171 B/s wr, 6 op/s 2023-03-09T15:04:22.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:04:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 373.127121 secs 2023-03-09T15:04:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:04:22 smithi181 ceph-mon[94189]: pgmap v716: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 171 B/s wr, 6 op/s 2023-03-09T15:04:22.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:04:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 373.127121 secs 2023-03-09T15:04:22.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:04:22 smithi038 ceph-mon[92422]: pgmap v716: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 171 B/s wr, 6 op/s 2023-03-09T15:04:22.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:04:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 373.127121 secs 2023-03-09T15:04:23.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 835.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:04:24.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:04:24 smithi122 ceph-mon[93869]: osdmap e261: 12 total, 12 up, 12 in 2023-03-09T15:04:24.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:04:24 smithi122 ceph-mon[93869]: pgmap v718: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:04:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:04:24 smithi181 ceph-mon[94189]: osdmap e261: 12 total, 12 up, 12 in 2023-03-09T15:04:24.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:04:24 smithi181 ceph-mon[94189]: pgmap v718: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:04:24.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:04:24 smithi038 ceph-mon[92422]: osdmap e261: 12 total, 12 up, 12 in 2023-03-09T15:04:24.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:04:24 smithi038 ceph-mon[92422]: pgmap v718: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:04:25.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:04:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 375.943779 secs 2023-03-09T15:04:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:04:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 375.943779 secs 2023-03-09T15:04:25.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:04:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 375.943779 secs 2023-03-09T15:04:26.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:04:26 smithi122 ceph-mon[93869]: pgmap v719: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-09T15:04:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:04:26 smithi181 ceph-mon[94189]: pgmap v719: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-09T15:04:26.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:04:26 smithi038 ceph-mon[92422]: pgmap v719: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-09T15:04:27.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:04:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 378.127191 secs 2023-03-09T15:04:27.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:04:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 378.127191 secs 2023-03-09T15:04:27.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:04:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 378.127191 secs 2023-03-09T15:04:28.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:04:28 smithi122 ceph-mon[93869]: pgmap v720: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 7.2 KiB/s wr, 7 op/s 2023-03-09T15:04:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:04:28 smithi181 ceph-mon[94189]: pgmap v720: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 7.2 KiB/s wr, 7 op/s 2023-03-09T15:04:28.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:04:28 smithi038 ceph-mon[92422]: pgmap v720: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 7.2 KiB/s wr, 7 op/s 2023-03-09T15:04:28.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 840.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:04:30.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:04:30 smithi122 ceph-mon[93869]: pgmap v721: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 7.2 KiB/s wr, 8 op/s 2023-03-09T15:04:30.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:04:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 380.943865 secs 2023-03-09T15:04:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:04:30 smithi181 ceph-mon[94189]: pgmap v721: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 7.2 KiB/s wr, 8 op/s 2023-03-09T15:04:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:04:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 380.943865 secs 2023-03-09T15:04:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:04:30 smithi038 ceph-mon[92422]: pgmap v721: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 7.2 KiB/s wr, 8 op/s 2023-03-09T15:04:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:04:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 380.943865 secs 2023-03-09T15:04:32.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:04:32 smithi122 ceph-mon[93869]: pgmap v722: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 7.6 KiB/s wr, 8 op/s 2023-03-09T15:04:32.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:04:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 383.127278 secs 2023-03-09T15:04:32.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:04:32 smithi181 ceph-mon[94189]: pgmap v722: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 7.6 KiB/s wr, 8 op/s 2023-03-09T15:04:32.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:04:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 383.127278 secs 2023-03-09T15:04:32.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:04:32 smithi038 ceph-mon[92422]: pgmap v722: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 7.6 KiB/s wr, 8 op/s 2023-03-09T15:04:32.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:04:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 383.127278 secs 2023-03-09T15:04:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 845.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:04:34.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:04:34 smithi122 ceph-mon[93869]: pgmap v723: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 8.6 KiB/s wr, 8 op/s 2023-03-09T15:04:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:04:34 smithi181 ceph-mon[94189]: pgmap v723: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 8.6 KiB/s wr, 8 op/s 2023-03-09T15:04:34.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:04:34 smithi038 ceph-mon[92422]: pgmap v723: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 8.6 KiB/s wr, 8 op/s 2023-03-09T15:04:35.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:04:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 385.943957 secs 2023-03-09T15:04:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:04:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 385.943957 secs 2023-03-09T15:04:35.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:04:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 385.943957 secs 2023-03-09T15:04:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:04:36 smithi122 ceph-mon[93869]: pgmap v724: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 17 KiB/s wr, 8 op/s 2023-03-09T15:04:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:04:36 smithi181 ceph-mon[94189]: pgmap v724: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 17 KiB/s wr, 8 op/s 2023-03-09T15:04:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:04:36 smithi038 ceph-mon[92422]: pgmap v724: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 17 KiB/s wr, 8 op/s 2023-03-09T15:04:37.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:04:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 388.127346 secs 2023-03-09T15:04:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:04:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 388.127346 secs 2023-03-09T15:04:37.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:04:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 388.127346 secs 2023-03-09T15:04:38.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:04:38 smithi122 ceph-mon[93869]: pgmap v725: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 17 KiB/s wr, 10 op/s 2023-03-09T15:04:38.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:04:38 smithi181 ceph-mon[94189]: pgmap v725: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 17 KiB/s wr, 10 op/s 2023-03-09T15:04:38.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:04:38 smithi038 ceph-mon[92422]: pgmap v725: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 17 KiB/s wr, 10 op/s 2023-03-09T15:04:38.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 850.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:04:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:04:40 smithi122 ceph-mon[93869]: pgmap v726: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 11 KiB/s wr, 6 op/s 2023-03-09T15:04:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:04:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 390.943984 secs 2023-03-09T15:04:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:04:40 smithi181 ceph-mon[94189]: pgmap v726: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 11 KiB/s wr, 6 op/s 2023-03-09T15:04:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:04:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 390.943984 secs 2023-03-09T15:04:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:04:40 smithi038 ceph-mon[92422]: pgmap v726: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 11 KiB/s wr, 6 op/s 2023-03-09T15:04:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:04:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 390.943984 secs 2023-03-09T15:04:42.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:04:42 smithi122 ceph-mon[93869]: pgmap v727: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 KiB/s wr, 7 op/s 2023-03-09T15:04:42.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:04:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 393.127424 secs 2023-03-09T15:04:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:04:42 smithi181 ceph-mon[94189]: pgmap v727: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 KiB/s wr, 7 op/s 2023-03-09T15:04:42.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:04:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 393.127424 secs 2023-03-09T15:04:42.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:04:42 smithi038 ceph-mon[92422]: pgmap v727: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 KiB/s wr, 7 op/s 2023-03-09T15:04:42.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:04:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 393.127424 secs 2023-03-09T15:04:43.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 855.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:04:44.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:04:44 smithi122 ceph-mon[93869]: pgmap v728: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T15:04:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:04:44 smithi181 ceph-mon[94189]: pgmap v728: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T15:04:44.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:04:44 smithi038 ceph-mon[92422]: pgmap v728: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T15:04:45.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:04:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 395.944044 secs 2023-03-09T15:04:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:04:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 395.944044 secs 2023-03-09T15:04:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:04:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 395.944044 secs 2023-03-09T15:04:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:04:46 smithi122 ceph-mon[93869]: pgmap v729: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 15 KiB/s wr, 6 op/s 2023-03-09T15:04:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:04:46 smithi181 ceph-mon[94189]: pgmap v729: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 15 KiB/s wr, 6 op/s 2023-03-09T15:04:46.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:04:46 smithi038 ceph-mon[92422]: pgmap v729: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 15 KiB/s wr, 6 op/s 2023-03-09T15:04:47.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:04:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 398.127559 secs 2023-03-09T15:04:47.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:04:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 398.127559 secs 2023-03-09T15:04:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:04:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 398.127559 secs 2023-03-09T15:04:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:04:48 smithi122 ceph-mon[93869]: pgmap v730: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.7 KiB/s wr, 7 op/s 2023-03-09T15:04:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:04:48 smithi181 ceph-mon[94189]: pgmap v730: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.7 KiB/s wr, 7 op/s 2023-03-09T15:04:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:04:48 smithi038 ceph-mon[92422]: pgmap v730: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.7 KiB/s wr, 7 op/s 2023-03-09T15:04:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 860.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:04:50.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:04:50 smithi122 ceph-mon[93869]: pgmap v731: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 5.5 KiB/s wr, 5 op/s 2023-03-09T15:04:50.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:04:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 400.944114 secs 2023-03-09T15:04:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:04:50 smithi181 ceph-mon[94189]: pgmap v731: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 5.5 KiB/s wr, 5 op/s 2023-03-09T15:04:50.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:04:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 400.944114 secs 2023-03-09T15:04:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:04:50 smithi038 ceph-mon[92422]: pgmap v731: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 5.5 KiB/s wr, 5 op/s 2023-03-09T15:04:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:04:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 400.944114 secs 2023-03-09T15:04:52.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:04:52 smithi122 ceph-mon[93869]: pgmap v732: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 5.5 KiB/s wr, 6 op/s 2023-03-09T15:04:52.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:04:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 403.127688 secs 2023-03-09T15:04:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:04:52 smithi181 ceph-mon[94189]: pgmap v732: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 5.5 KiB/s wr, 6 op/s 2023-03-09T15:04:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:04:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 403.127688 secs 2023-03-09T15:04:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:04:52 smithi038 ceph-mon[92422]: pgmap v732: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 5.5 KiB/s wr, 6 op/s 2023-03-09T15:04:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:04:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 403.127688 secs 2023-03-09T15:04:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 865.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:04:54.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:04:54 smithi122 ceph-mon[93869]: pgmap v733: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2023-03-09T15:04:54.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:04:54 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:04:54.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:04:54 smithi181 ceph-mon[94189]: pgmap v733: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2023-03-09T15:04:54.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:04:54 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:04:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:04:54 smithi038 ceph-mon[92422]: pgmap v733: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2023-03-09T15:04:54.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:04:54 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:04:55.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:04:55 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:04:55.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:04:55 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:04:55.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:04:55 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:04:55.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:04:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 405.944175 secs 2023-03-09T15:04:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:04:55 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:04:55.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:04:55 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:04:55.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:04:55 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:04:55.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:04:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 405.944175 secs 2023-03-09T15:04:55.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:04:55 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:04:55.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:04:55 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:04:55.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:04:55 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:04:55.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:04:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 405.944175 secs 2023-03-09T15:04:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:04:56 smithi122 ceph-mon[93869]: pgmap v734: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T15:04:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:04:56 smithi181 ceph-mon[94189]: pgmap v734: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T15:04:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:04:56 smithi038 ceph-mon[92422]: pgmap v734: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T15:04:57.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:04:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 408.127824 secs 2023-03-09T15:04:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:04:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 408.127824 secs 2023-03-09T15:04:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:04:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 408.127824 secs 2023-03-09T15:04:58.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:04:58 smithi122 ceph-mon[93869]: pgmap v735: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 2.1 KiB/s wr, 7 op/s 2023-03-09T15:04:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:04:58 smithi181 ceph-mon[94189]: pgmap v735: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 2.1 KiB/s wr, 7 op/s 2023-03-09T15:04:58.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:04:58 smithi038 ceph-mon[92422]: pgmap v735: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 2.1 KiB/s wr, 7 op/s 2023-03-09T15:04:58.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 870.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:05:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:05:00 smithi122 ceph-mon[93869]: pgmap v736: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 3.4 KiB/s wr, 5 op/s 2023-03-09T15:05:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:05:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 410.944237 secs 2023-03-09T15:05:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:05:00 smithi181 ceph-mon[94189]: pgmap v736: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 3.4 KiB/s wr, 5 op/s 2023-03-09T15:05:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:05:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 410.944237 secs 2023-03-09T15:05:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:05:00 smithi038 ceph-mon[92422]: pgmap v736: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 3.4 KiB/s wr, 5 op/s 2023-03-09T15:05:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:05:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 410.944237 secs 2023-03-09T15:05:02.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:05:02 smithi122 ceph-mon[93869]: pgmap v737: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 3.5 KiB/s wr, 6 op/s 2023-03-09T15:05:02.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:05:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 413.127894 secs 2023-03-09T15:05:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:05:02 smithi181 ceph-mon[94189]: pgmap v737: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 3.5 KiB/s wr, 6 op/s 2023-03-09T15:05:02.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:05:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 413.127894 secs 2023-03-09T15:05:02.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:05:02 smithi038 ceph-mon[92422]: pgmap v737: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 3.5 KiB/s wr, 6 op/s 2023-03-09T15:05:02.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:05:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 413.127894 secs 2023-03-09T15:05:03.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 875.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:05:04.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:05:04 smithi122 ceph-mon[93869]: pgmap v738: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 16 KiB/s wr, 9 op/s 2023-03-09T15:05:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:05:04 smithi181 ceph-mon[94189]: pgmap v738: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 16 KiB/s wr, 9 op/s 2023-03-09T15:05:04.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:05:04 smithi038 ceph-mon[92422]: pgmap v738: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 16 KiB/s wr, 9 op/s 2023-03-09T15:05:05.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:05:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 415.944299 secs 2023-03-09T15:05:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:05:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 415.944299 secs 2023-03-09T15:05:05.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:05:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 415.944299 secs 2023-03-09T15:05:06.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:05:06 smithi122 ceph-mon[93869]: pgmap v739: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 35 KiB/s wr, 19 op/s 2023-03-09T15:05:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:05:06 smithi181 ceph-mon[94189]: pgmap v739: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 35 KiB/s wr, 19 op/s 2023-03-09T15:05:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:05:06 smithi038 ceph-mon[92422]: pgmap v739: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 35 KiB/s wr, 19 op/s 2023-03-09T15:05:07.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:05:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 418.128037 secs 2023-03-09T15:05:07.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:05:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 418.128037 secs 2023-03-09T15:05:07.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:05:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 418.128037 secs 2023-03-09T15:05:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:05:08 smithi122 ceph-mon[93869]: pgmap v740: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 35 KiB/s wr, 21 op/s 2023-03-09T15:05:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:05:08 smithi181 ceph-mon[94189]: pgmap v740: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 35 KiB/s wr, 21 op/s 2023-03-09T15:05:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:05:08 smithi038 ceph-mon[92422]: pgmap v740: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 35 KiB/s wr, 21 op/s 2023-03-09T15:05:08.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 880.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:05:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:05:10 smithi122 ceph-mon[93869]: pgmap v741: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 33 KiB/s wr, 19 op/s 2023-03-09T15:05:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:05:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 420.944361 secs 2023-03-09T15:05:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:05:10 smithi181 ceph-mon[94189]: pgmap v741: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 33 KiB/s wr, 19 op/s 2023-03-09T15:05:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:05:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 420.944361 secs 2023-03-09T15:05:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:05:10 smithi038 ceph-mon[92422]: pgmap v741: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 33 KiB/s wr, 19 op/s 2023-03-09T15:05:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:05:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 420.944361 secs 2023-03-09T15:05:12.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:05:12 smithi122 ceph-mon[93869]: pgmap v742: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 32 KiB/s wr, 20 op/s 2023-03-09T15:05:12.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:05:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 423.128101 secs 2023-03-09T15:05:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:05:12 smithi181 ceph-mon[94189]: pgmap v742: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 32 KiB/s wr, 20 op/s 2023-03-09T15:05:12.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:05:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 423.128101 secs 2023-03-09T15:05:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:05:12 smithi038 ceph-mon[92422]: pgmap v742: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 32 KiB/s wr, 20 op/s 2023-03-09T15:05:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:05:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 423.128101 secs 2023-03-09T15:05:13.801 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 885.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:05:14.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:05:14 smithi122 ceph-mon[93869]: pgmap v743: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 32 KiB/s wr, 21 op/s 2023-03-09T15:05:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:05:14 smithi181 ceph-mon[94189]: pgmap v743: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 32 KiB/s wr, 21 op/s 2023-03-09T15:05:14.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:05:14 smithi038 ceph-mon[92422]: pgmap v743: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 32 KiB/s wr, 21 op/s 2023-03-09T15:05:15.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:05:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 425.944423 secs 2023-03-09T15:05:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:05:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 425.944423 secs 2023-03-09T15:05:15.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:05:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 425.944423 secs 2023-03-09T15:05:16.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:05:16 smithi122 ceph-mon[93869]: pgmap v744: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 19 KiB/s wr, 17 op/s 2023-03-09T15:05:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:05:16 smithi181 ceph-mon[94189]: pgmap v744: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 19 KiB/s wr, 17 op/s 2023-03-09T15:05:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:05:16 smithi038 ceph-mon[92422]: pgmap v744: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 19 KiB/s wr, 17 op/s 2023-03-09T15:05:17.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:05:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 428.128249 secs 2023-03-09T15:05:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:05:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 428.128249 secs 2023-03-09T15:05:17.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:05:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 428.128249 secs 2023-03-09T15:05:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:05:18 smithi122 ceph-mon[93869]: pgmap v745: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 170 B/s wr, 7 op/s 2023-03-09T15:05:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:05:18 smithi181 ceph-mon[94189]: pgmap v745: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 170 B/s wr, 7 op/s 2023-03-09T15:05:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:05:18 smithi038 ceph-mon[92422]: pgmap v745: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 170 B/s wr, 7 op/s 2023-03-09T15:05:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 890.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:05:20.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:05:20 smithi122 ceph-mon[93869]: pgmap v746: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T15:05:20.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:05:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 430.944484 secs 2023-03-09T15:05:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:05:20 smithi181 ceph-mon[94189]: pgmap v746: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T15:05:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:05:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 430.944484 secs 2023-03-09T15:05:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:05:20 smithi038 ceph-mon[92422]: pgmap v746: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T15:05:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:05:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 430.944484 secs 2023-03-09T15:05:22.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:05:22 smithi122 ceph-mon[93869]: pgmap v747: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T15:05:22.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:05:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 433.128398 secs 2023-03-09T15:05:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:05:22 smithi181 ceph-mon[94189]: pgmap v747: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T15:05:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:05:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 433.128398 secs 2023-03-09T15:05:22.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:05:22 smithi038 ceph-mon[92422]: pgmap v747: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T15:05:22.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:05:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 433.128398 secs 2023-03-09T15:05:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 895.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:05:24.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:05:24 smithi122 ceph-mon[93869]: pgmap v748: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T15:05:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:05:24 smithi181 ceph-mon[94189]: pgmap v748: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T15:05:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:05:24 smithi038 ceph-mon[92422]: pgmap v748: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T15:05:25.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:05:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 435.944544 secs 2023-03-09T15:05:25.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:05:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 435.944544 secs 2023-03-09T15:05:25.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:05:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 435.944544 secs 2023-03-09T15:05:26.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:05:26 smithi122 ceph-mon[93869]: pgmap v749: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T15:05:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:05:26 smithi181 ceph-mon[94189]: pgmap v749: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T15:05:26.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:05:26 smithi038 ceph-mon[92422]: pgmap v749: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T15:05:27.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:05:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 438.128517 secs 2023-03-09T15:05:27.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:05:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 438.128517 secs 2023-03-09T15:05:27.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:05:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 438.128517 secs 2023-03-09T15:05:28.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:05:28 smithi122 ceph-mon[93869]: pgmap v750: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 4.7 KiB/s wr, 7 op/s 2023-03-09T15:05:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:05:28 smithi181 ceph-mon[94189]: pgmap v750: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 4.7 KiB/s wr, 7 op/s 2023-03-09T15:05:28.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:05:28 smithi038 ceph-mon[92422]: pgmap v750: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 4.7 KiB/s wr, 7 op/s 2023-03-09T15:05:28.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 900.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:05:30.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:05:30 smithi122 ceph-mon[93869]: pgmap v751: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 4.7 KiB/s wr, 6 op/s 2023-03-09T15:05:30.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:05:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 440.944615 secs 2023-03-09T15:05:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:05:30 smithi181 ceph-mon[94189]: pgmap v751: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 4.7 KiB/s wr, 6 op/s 2023-03-09T15:05:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:05:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 440.944615 secs 2023-03-09T15:05:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:05:30 smithi038 ceph-mon[92422]: pgmap v751: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 4.7 KiB/s wr, 6 op/s 2023-03-09T15:05:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:05:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 440.944615 secs 2023-03-09T15:05:32.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:05:32 smithi122 ceph-mon[93869]: pgmap v752: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 4.7 KiB/s wr, 6 op/s 2023-03-09T15:05:32.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:05:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 443.128645 secs 2023-03-09T15:05:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:05:32 smithi181 ceph-mon[94189]: pgmap v752: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 4.7 KiB/s wr, 6 op/s 2023-03-09T15:05:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:05:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 443.128645 secs 2023-03-09T15:05:32.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:05:32 smithi038 ceph-mon[92422]: pgmap v752: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 4.7 KiB/s wr, 6 op/s 2023-03-09T15:05:32.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:05:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 443.128645 secs 2023-03-09T15:05:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 905.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:05:34.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:05:34 smithi122 ceph-mon[93869]: pgmap v753: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 18 KiB/s wr, 9 op/s 2023-03-09T15:05:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:05:34 smithi181 ceph-mon[94189]: pgmap v753: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 18 KiB/s wr, 9 op/s 2023-03-09T15:05:34.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:05:34 smithi038 ceph-mon[92422]: pgmap v753: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 18 KiB/s wr, 9 op/s 2023-03-09T15:05:35.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:05:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 445.944677 secs 2023-03-09T15:05:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:05:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 445.944677 secs 2023-03-09T15:05:35.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:05:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 445.944677 secs 2023-03-09T15:05:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:05:36 smithi122 ceph-mon[93869]: pgmap v754: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 18 KiB/s wr, 7 op/s 2023-03-09T15:05:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:05:36 smithi181 ceph-mon[94189]: pgmap v754: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 18 KiB/s wr, 7 op/s 2023-03-09T15:05:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:05:36 smithi038 ceph-mon[92422]: pgmap v754: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 18 KiB/s wr, 7 op/s 2023-03-09T15:05:37.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:05:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 448.128770 secs 2023-03-09T15:05:37.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:05:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 448.128770 secs 2023-03-09T15:05:37.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:05:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 448.128770 secs 2023-03-09T15:05:38.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:05:38 smithi122 ceph-mon[93869]: pgmap v755: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 18 KiB/s wr, 10 op/s 2023-03-09T15:05:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:05:38 smithi181 ceph-mon[94189]: pgmap v755: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 18 KiB/s wr, 10 op/s 2023-03-09T15:05:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:05:38 smithi038 ceph-mon[92422]: pgmap v755: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 18 KiB/s wr, 10 op/s 2023-03-09T15:05:38.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 910.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:05:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:05:40 smithi122 ceph-mon[93869]: pgmap v756: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T15:05:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:05:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 450.944738 secs 2023-03-09T15:05:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:05:40 smithi181 ceph-mon[94189]: pgmap v756: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T15:05:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:05:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 450.944738 secs 2023-03-09T15:05:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:05:40 smithi038 ceph-mon[92422]: pgmap v756: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T15:05:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:05:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 450.944738 secs 2023-03-09T15:05:42.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:05:42 smithi122 ceph-mon[93869]: pgmap v757: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T15:05:42.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:05:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 453.128888 secs 2023-03-09T15:05:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:05:42 smithi181 ceph-mon[94189]: pgmap v757: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T15:05:42.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:05:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 453.128888 secs 2023-03-09T15:05:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:05:42 smithi038 ceph-mon[92422]: pgmap v757: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T15:05:42.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:05:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 453.128888 secs 2023-03-09T15:05:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 915.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:05:44.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:05:44 smithi122 ceph-mon[93869]: pgmap v758: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 9 op/s 2023-03-09T15:05:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:05:44 smithi181 ceph-mon[94189]: pgmap v758: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 9 op/s 2023-03-09T15:05:44.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:05:44 smithi038 ceph-mon[92422]: pgmap v758: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 9 op/s 2023-03-09T15:05:45.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:05:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 455.944798 secs 2023-03-09T15:05:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:05:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 455.944798 secs 2023-03-09T15:05:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:05:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 455.944798 secs 2023-03-09T15:05:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:05:46 smithi122 ceph-mon[93869]: pgmap v759: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.0 KiB/s wr, 5 op/s 2023-03-09T15:05:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:05:46 smithi181 ceph-mon[94189]: pgmap v759: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.0 KiB/s wr, 5 op/s 2023-03-09T15:05:46.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:05:46 smithi038 ceph-mon[92422]: pgmap v759: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 6.0 KiB/s wr, 5 op/s 2023-03-09T15:05:47.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:05:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 458.129026 secs 2023-03-09T15:05:47.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:05:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 458.129026 secs 2023-03-09T15:05:47.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:05:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 458.129026 secs 2023-03-09T15:05:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:05:48 smithi122 ceph-mon[93869]: pgmap v760: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 5.9 KiB/s wr, 7 op/s 2023-03-09T15:05:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:05:48 smithi181 ceph-mon[94189]: pgmap v760: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 5.9 KiB/s wr, 7 op/s 2023-03-09T15:05:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:05:48 smithi038 ceph-mon[92422]: pgmap v760: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 5.9 KiB/s wr, 7 op/s 2023-03-09T15:05:48.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 920.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:05:50.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:05:50 smithi122 ceph-mon[93869]: pgmap v761: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 5.7 KiB/s wr, 5 op/s 2023-03-09T15:05:50.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:05:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 460.944860 secs 2023-03-09T15:05:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:05:50 smithi181 ceph-mon[94189]: pgmap v761: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 5.7 KiB/s wr, 5 op/s 2023-03-09T15:05:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:05:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 460.944860 secs 2023-03-09T15:05:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:05:50 smithi038 ceph-mon[92422]: pgmap v761: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 5.7 KiB/s wr, 5 op/s 2023-03-09T15:05:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:05:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 460.944860 secs 2023-03-09T15:05:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:05:52 smithi038 ceph-mon[92422]: pgmap v762: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 5.7 KiB/s wr, 5 op/s 2023-03-09T15:05:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:05:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 463.129138 secs 2023-03-09T15:05:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:05:52 smithi122 ceph-mon[93869]: pgmap v762: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 5.7 KiB/s wr, 5 op/s 2023-03-09T15:05:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:05:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 463.129138 secs 2023-03-09T15:05:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:05:52 smithi181 ceph-mon[94189]: pgmap v762: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 5.7 KiB/s wr, 5 op/s 2023-03-09T15:05:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:05:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 463.129138 secs 2023-03-09T15:05:53.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 925.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:05:54.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:05:54 smithi122 ceph-mon[93869]: pgmap v763: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 5.7 KiB/s wr, 7 op/s 2023-03-09T15:05:54.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:05:54 smithi181 ceph-mon[94189]: pgmap v763: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 5.7 KiB/s wr, 7 op/s 2023-03-09T15:05:54.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:05:54 smithi038 ceph-mon[92422]: pgmap v763: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 5.7 KiB/s wr, 7 op/s 2023-03-09T15:05:55.474 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:05:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 465.944920 secs 2023-03-09T15:05:55.474 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:05:55 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:05:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:05:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 465.944920 secs 2023-03-09T15:05:55.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:05:55 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:05:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:05:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 465.944920 secs 2023-03-09T15:05:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:05:55 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:05:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:05:56 smithi122 ceph-mon[93869]: pgmap v764: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T15:05:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:05:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:05:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:05:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:05:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:05:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:05:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:05:56 smithi181 ceph-mon[94189]: pgmap v764: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T15:05:56.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:05:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:05:56.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:05:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:05:56.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:05:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:05:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:05:56 smithi038 ceph-mon[92422]: pgmap v764: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T15:05:56.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:05:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:05:56.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:05:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:05:56.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:05:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:05:57.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:05:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 468.129261 secs 2023-03-09T15:05:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:05:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 468.129261 secs 2023-03-09T15:05:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:05:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 468.129261 secs 2023-03-09T15:05:58.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:05:58 smithi122 ceph-mon[93869]: pgmap v765: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 2.3 KiB/s wr, 7 op/s 2023-03-09T15:05:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:05:58 smithi181 ceph-mon[94189]: pgmap v765: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 2.3 KiB/s wr, 7 op/s 2023-03-09T15:05:58.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:05:58 smithi038 ceph-mon[92422]: pgmap v765: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 2.3 KiB/s wr, 7 op/s 2023-03-09T15:05:58.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 930.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:06:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:06:00 smithi122 ceph-mon[93869]: pgmap v766: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 2.5 KiB/s wr, 5 op/s 2023-03-09T15:06:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:06:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 470.944981 secs 2023-03-09T15:06:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:06:00 smithi181 ceph-mon[94189]: pgmap v766: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 2.5 KiB/s wr, 5 op/s 2023-03-09T15:06:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:06:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 470.944981 secs 2023-03-09T15:06:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:06:00 smithi038 ceph-mon[92422]: pgmap v766: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 2.5 KiB/s wr, 5 op/s 2023-03-09T15:06:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:06:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 470.944981 secs 2023-03-09T15:06:02.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:06:02 smithi122 ceph-mon[93869]: pgmap v767: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 3.6 KiB/s wr, 6 op/s 2023-03-09T15:06:02.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:06:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 473.129378 secs 2023-03-09T15:06:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:06:02 smithi181 ceph-mon[94189]: pgmap v767: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 3.6 KiB/s wr, 6 op/s 2023-03-09T15:06:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:06:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 473.129378 secs 2023-03-09T15:06:02.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:06:02 smithi038 ceph-mon[92422]: pgmap v767: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 3.6 KiB/s wr, 6 op/s 2023-03-09T15:06:02.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:06:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 473.129378 secs 2023-03-09T15:06:03.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 935.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:06:04.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:06:04 smithi122 ceph-mon[93869]: pgmap v768: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 37 KiB/s wr, 11 op/s 2023-03-09T15:06:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:06:04 smithi181 ceph-mon[94189]: pgmap v768: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 37 KiB/s wr, 11 op/s 2023-03-09T15:06:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:06:04 smithi038 ceph-mon[92422]: pgmap v768: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 37 KiB/s wr, 11 op/s 2023-03-09T15:06:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:06:05 smithi122 ceph-mon[93869]: osdmap e262: 12 total, 12 up, 12 in 2023-03-09T15:06:05.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:06:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 475.945044 secs 2023-03-09T15:06:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:06:05 smithi181 ceph-mon[94189]: osdmap e262: 12 total, 12 up, 12 in 2023-03-09T15:06:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:06:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 475.945044 secs 2023-03-09T15:06:05.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:06:05 smithi038 ceph-mon[92422]: osdmap e262: 12 total, 12 up, 12 in 2023-03-09T15:06:05.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:06:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 475.945044 secs 2023-03-09T15:06:06.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:06:06 smithi122 ceph-mon[93869]: pgmap v770: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 69 KiB/s wr, 25 op/s 2023-03-09T15:06:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:06:06 smithi181 ceph-mon[94189]: pgmap v770: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 69 KiB/s wr, 25 op/s 2023-03-09T15:06:06.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:06:06 smithi038 ceph-mon[92422]: pgmap v770: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 69 KiB/s wr, 25 op/s 2023-03-09T15:06:07.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:06:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 478.129497 secs 2023-03-09T15:06:07.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:06:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 478.129497 secs 2023-03-09T15:06:07.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:06:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 478.129497 secs 2023-03-09T15:06:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:06:08 smithi122 ceph-mon[93869]: pgmap v771: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 67 KiB/s wr, 25 op/s 2023-03-09T15:06:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:06:08 smithi181 ceph-mon[94189]: pgmap v771: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 67 KiB/s wr, 25 op/s 2023-03-09T15:06:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:06:08 smithi038 ceph-mon[92422]: pgmap v771: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 67 KiB/s wr, 25 op/s 2023-03-09T15:06:08.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 940.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:06:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:06:10 smithi122 ceph-mon[93869]: pgmap v772: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 67 KiB/s wr, 25 op/s 2023-03-09T15:06:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:06:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 480.945126 secs 2023-03-09T15:06:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:06:10 smithi181 ceph-mon[94189]: pgmap v772: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 67 KiB/s wr, 25 op/s 2023-03-09T15:06:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:06:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 480.945126 secs 2023-03-09T15:06:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:06:10 smithi038 ceph-mon[92422]: pgmap v772: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 67 KiB/s wr, 25 op/s 2023-03-09T15:06:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:06:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 480.945126 secs 2023-03-09T15:06:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:06:12 smithi181 ceph-mon[94189]: pgmap v773: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 65 KiB/s wr, 25 op/s 2023-03-09T15:06:12.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:06:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 483.129604 secs 2023-03-09T15:06:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:06:12 smithi038 ceph-mon[92422]: pgmap v773: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 65 KiB/s wr, 25 op/s 2023-03-09T15:06:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:06:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 483.129604 secs 2023-03-09T15:06:12.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:06:12 smithi122 ceph-mon[93869]: pgmap v773: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 65 KiB/s wr, 25 op/s 2023-03-09T15:06:12.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:06:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 483.129604 secs 2023-03-09T15:06:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 945.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:06:14.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:06:14 smithi122 ceph-mon[93869]: osdmap e263: 12 total, 12 up, 12 in 2023-03-09T15:06:14.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:06:14 smithi122 ceph-mon[93869]: pgmap v775: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 448 B/s wr, 7 op/s 2023-03-09T15:06:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:06:14 smithi181 ceph-mon[94189]: osdmap e263: 12 total, 12 up, 12 in 2023-03-09T15:06:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:06:14 smithi181 ceph-mon[94189]: pgmap v775: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 448 B/s wr, 7 op/s 2023-03-09T15:06:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:06:14 smithi038 ceph-mon[92422]: osdmap e263: 12 total, 12 up, 12 in 2023-03-09T15:06:14.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:06:14 smithi038 ceph-mon[92422]: pgmap v775: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 448 B/s wr, 7 op/s 2023-03-09T15:06:15.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:06:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 485.945187 secs 2023-03-09T15:06:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:06:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 485.945187 secs 2023-03-09T15:06:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:06:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 485.945187 secs 2023-03-09T15:06:16.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:06:16 smithi122 ceph-mon[93869]: pgmap v776: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 409 B/s wr, 7 op/s 2023-03-09T15:06:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:06:16 smithi181 ceph-mon[94189]: pgmap v776: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 409 B/s wr, 7 op/s 2023-03-09T15:06:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:06:16 smithi038 ceph-mon[92422]: pgmap v776: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 409 B/s wr, 7 op/s 2023-03-09T15:06:17.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:06:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 488.129723 secs 2023-03-09T15:06:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:06:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 488.129723 secs 2023-03-09T15:06:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:06:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 488.129723 secs 2023-03-09T15:06:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:06:18 smithi122 ceph-mon[93869]: pgmap v777: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:06:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:06:18 smithi181 ceph-mon[94189]: pgmap v777: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:06:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:06:18 smithi038 ceph-mon[92422]: pgmap v777: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:06:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 950.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:06:20.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:06:20 smithi122 ceph-mon[93869]: pgmap v778: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:06:20.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:06:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 490.945246 secs 2023-03-09T15:06:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:06:20 smithi181 ceph-mon[94189]: pgmap v778: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:06:20.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:06:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 490.945246 secs 2023-03-09T15:06:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:06:20 smithi038 ceph-mon[92422]: pgmap v778: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:06:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:06:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 490.945246 secs 2023-03-09T15:06:22.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:06:22 smithi038 ceph-mon[92422]: pgmap v779: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:06:22.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:06:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 493.129837 secs 2023-03-09T15:06:22.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:06:22 smithi122 ceph-mon[93869]: pgmap v779: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:06:22.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:06:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 493.129837 secs 2023-03-09T15:06:22.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:06:22 smithi181 ceph-mon[94189]: pgmap v779: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:06:22.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:06:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 493.129837 secs 2023-03-09T15:06:23.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 955.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:06:24.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:06:24 smithi122 ceph-mon[93869]: pgmap v780: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T15:06:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:06:24 smithi181 ceph-mon[94189]: pgmap v780: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T15:06:24.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:06:24 smithi038 ceph-mon[92422]: pgmap v780: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T15:06:25.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:06:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 495.945314 secs 2023-03-09T15:06:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:06:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 495.945314 secs 2023-03-09T15:06:25.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:06:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 495.945314 secs 2023-03-09T15:06:26.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:06:26 smithi122 ceph-mon[93869]: pgmap v781: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T15:06:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:06:26 smithi181 ceph-mon[94189]: pgmap v781: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T15:06:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:06:26 smithi038 ceph-mon[92422]: pgmap v781: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T15:06:27.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:06:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 498.129989 secs 2023-03-09T15:06:27.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:06:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 498.129989 secs 2023-03-09T15:06:27.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:06:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 498.129989 secs 2023-03-09T15:06:28.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:06:28 smithi122 ceph-mon[93869]: pgmap v782: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 5.0 KiB/s wr, 7 op/s 2023-03-09T15:06:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:06:28 smithi181 ceph-mon[94189]: pgmap v782: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 5.0 KiB/s wr, 7 op/s 2023-03-09T15:06:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:06:28 smithi038 ceph-mon[92422]: pgmap v782: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 5.0 KiB/s wr, 7 op/s 2023-03-09T15:06:28.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 960.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:06:30.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:06:30 smithi122 ceph-mon[93869]: pgmap v783: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2023-03-09T15:06:30.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:06:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 500.945373 secs 2023-03-09T15:06:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:06:30 smithi181 ceph-mon[94189]: pgmap v783: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2023-03-09T15:06:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:06:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 500.945373 secs 2023-03-09T15:06:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:06:30 smithi038 ceph-mon[92422]: pgmap v783: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2023-03-09T15:06:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:06:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 500.945373 secs 2023-03-09T15:06:32.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:06:32 smithi122 ceph-mon[93869]: pgmap v784: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2023-03-09T15:06:32.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:06:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 503.130139 secs 2023-03-09T15:06:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:06:32 smithi181 ceph-mon[94189]: pgmap v784: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2023-03-09T15:06:32.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:06:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 503.130139 secs 2023-03-09T15:06:32.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:06:32 smithi038 ceph-mon[92422]: pgmap v784: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 5.0 KiB/s wr, 6 op/s 2023-03-09T15:06:32.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:06:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 503.130139 secs 2023-03-09T15:06:33.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 965.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:06:34.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:06:34 smithi122 ceph-mon[93869]: pgmap v785: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 19 KiB/s wr, 9 op/s 2023-03-09T15:06:34.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:06:34 smithi181 ceph-mon[94189]: pgmap v785: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 19 KiB/s wr, 9 op/s 2023-03-09T15:06:34.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:06:34 smithi038 ceph-mon[92422]: pgmap v785: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 19 KiB/s wr, 9 op/s 2023-03-09T15:06:35.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:06:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 505.945433 secs 2023-03-09T15:06:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:06:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 505.945433 secs 2023-03-09T15:06:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:06:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 505.945433 secs 2023-03-09T15:06:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:06:36 smithi122 ceph-mon[93869]: pgmap v786: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T15:06:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:06:36 smithi181 ceph-mon[94189]: pgmap v786: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T15:06:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:06:36 smithi038 ceph-mon[92422]: pgmap v786: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T15:06:37.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:06:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 508.130257 secs 2023-03-09T15:06:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:06:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 508.130257 secs 2023-03-09T15:06:37.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:06:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 508.130257 secs 2023-03-09T15:06:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:06:38 smithi181 ceph-mon[94189]: pgmap v787: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 19 KiB/s wr, 10 op/s 2023-03-09T15:06:38.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:06:38 smithi038 ceph-mon[92422]: pgmap v787: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 19 KiB/s wr, 10 op/s 2023-03-09T15:06:38.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:06:38 smithi122 ceph-mon[93869]: pgmap v787: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 19 KiB/s wr, 10 op/s 2023-03-09T15:06:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 970.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:06:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:06:40 smithi038 ceph-mon[92422]: pgmap v788: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T15:06:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:06:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 510.945494 secs 2023-03-09T15:06:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:06:40 smithi122 ceph-mon[93869]: pgmap v788: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T15:06:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:06:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 510.945494 secs 2023-03-09T15:06:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:06:40 smithi181 ceph-mon[94189]: pgmap v788: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T15:06:40.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:06:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 510.945494 secs 2023-03-09T15:06:42.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:06:42 smithi038 ceph-mon[92422]: pgmap v789: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T15:06:42.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:06:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 513.130379 secs 2023-03-09T15:06:42.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:06:42 smithi122 ceph-mon[93869]: pgmap v789: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T15:06:42.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:06:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 513.130379 secs 2023-03-09T15:06:42.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:06:42 smithi181 ceph-mon[94189]: pgmap v789: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T15:06:42.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:06:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 513.130379 secs 2023-03-09T15:06:43.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 975.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:06:44.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:06:44 smithi122 ceph-mon[93869]: pgmap v790: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 9 op/s 2023-03-09T15:06:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:06:44 smithi181 ceph-mon[94189]: pgmap v790: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 9 op/s 2023-03-09T15:06:44.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:06:44 smithi038 ceph-mon[92422]: pgmap v790: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 9 op/s 2023-03-09T15:06:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:06:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 515.945566 secs 2023-03-09T15:06:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:06:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 515.945566 secs 2023-03-09T15:06:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:06:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 515.945566 secs 2023-03-09T15:06:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:06:46 smithi122 ceph-mon[93869]: pgmap v791: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.2 KiB/s wr, 5 op/s 2023-03-09T15:06:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:06:46 smithi181 ceph-mon[94189]: pgmap v791: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.2 KiB/s wr, 5 op/s 2023-03-09T15:06:46.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:06:46 smithi038 ceph-mon[92422]: pgmap v791: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.2 KiB/s wr, 5 op/s 2023-03-09T15:06:47.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:06:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 518.130499 secs 2023-03-09T15:06:47.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:06:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 518.130499 secs 2023-03-09T15:06:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:06:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 518.130499 secs 2023-03-09T15:06:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:06:48 smithi122 ceph-mon[93869]: pgmap v792: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 6.1 KiB/s wr, 7 op/s 2023-03-09T15:06:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:06:48 smithi181 ceph-mon[94189]: pgmap v792: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 6.1 KiB/s wr, 7 op/s 2023-03-09T15:06:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:06:48 smithi038 ceph-mon[92422]: pgmap v792: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 6.1 KiB/s wr, 7 op/s 2023-03-09T15:06:48.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 980.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:06:50.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:06:50 smithi122 ceph-mon[93869]: pgmap v793: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 5.9 KiB/s wr, 5 op/s 2023-03-09T15:06:50.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:06:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 520.945625 secs 2023-03-09T15:06:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:06:50 smithi181 ceph-mon[94189]: pgmap v793: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 5.9 KiB/s wr, 5 op/s 2023-03-09T15:06:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:06:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 520.945625 secs 2023-03-09T15:06:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:06:50 smithi038 ceph-mon[92422]: pgmap v793: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 KiB/s rd, 5.9 KiB/s wr, 5 op/s 2023-03-09T15:06:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:06:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 520.945625 secs 2023-03-09T15:06:52.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:06:52 smithi122 ceph-mon[93869]: pgmap v794: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 5.9 KiB/s wr, 5 op/s 2023-03-09T15:06:52.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:06:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 523.130619 secs 2023-03-09T15:06:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:06:52 smithi181 ceph-mon[94189]: pgmap v794: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 5.9 KiB/s wr, 5 op/s 2023-03-09T15:06:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:06:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 523.130619 secs 2023-03-09T15:06:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:06:52 smithi038 ceph-mon[92422]: pgmap v794: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 5.9 KiB/s wr, 5 op/s 2023-03-09T15:06:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:06:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 523.130619 secs 2023-03-09T15:06:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 985.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:06:54.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:06:54 smithi122 ceph-mon[93869]: pgmap v795: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 5.9 KiB/s wr, 7 op/s 2023-03-09T15:06:54.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:06:54 smithi181 ceph-mon[94189]: pgmap v795: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 5.9 KiB/s wr, 7 op/s 2023-03-09T15:06:54.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:06:54 smithi038 ceph-mon[92422]: pgmap v795: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 5.9 KiB/s wr, 7 op/s 2023-03-09T15:06:55.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:06:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 525.945708 secs 2023-03-09T15:06:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:06:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 525.945708 secs 2023-03-09T15:06:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:06:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 525.945708 secs 2023-03-09T15:06:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:06:56 smithi122 ceph-mon[93869]: pgmap v796: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T15:06:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:06:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:06:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:06:56 smithi181 ceph-mon[94189]: pgmap v796: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T15:06:56.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:06:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:06:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:06:56 smithi038 ceph-mon[92422]: pgmap v796: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T15:06:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:06:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:06:57.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:06:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 528.130760 secs 2023-03-09T15:06:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:06:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 528.130760 secs 2023-03-09T15:06:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:06:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 528.130760 secs 2023-03-09T15:06:58.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:06:58 smithi122 ceph-mon[93869]: pgmap v797: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 2.9 KiB/s wr, 8 op/s 2023-03-09T15:06:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:06:58 smithi181 ceph-mon[94189]: pgmap v797: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 2.9 KiB/s wr, 8 op/s 2023-03-09T15:06:58.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:06:58 smithi038 ceph-mon[92422]: pgmap v797: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 2.9 KiB/s wr, 8 op/s 2023-03-09T15:06:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 990.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:06:59.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:06:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:06:59.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:06:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:06:59.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:06:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:06:59.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:06:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:06:59.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:06:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:06:59.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:06:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:06:59.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:06:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:06:59.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:06:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:06:59.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:06:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:06:59.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:06:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:06:59.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:06:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:06:59.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:06:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:06:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:06:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:06:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:06:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:06:59.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:06:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:06:59.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:06:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:06:59.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:06:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:06:59.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:06:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:06:59.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:06:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:06:59.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:06:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:06:59.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:06:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:06:59.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:06:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:06:59.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:06:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:06:59.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:06:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:06:59.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:06:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:06:59.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:06:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:06:59.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:06:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:06:59.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:06:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:06:59.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:06:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:06:59.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:06:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:06:59.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:06:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:06:59.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:06:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:06:59.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:06:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:06:59.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:06:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:06:59.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:06:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:06:59.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:06:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:07:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:07:00 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T15:07:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:07:00 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T15:07:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:07:00 smithi122 ceph-mon[93869]: pgmap v798: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 3.1 KiB/s wr, 6 op/s 2023-03-09T15:07:00.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:07:00 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi038 to 197.2M 2023-03-09T15:07:00.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:07:00 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi038 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-03-09T15:07:00.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:07:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 530.945771 secs 2023-03-09T15:07:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:07:00 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T15:07:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:07:00 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T15:07:00.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:07:00 smithi181 ceph-mon[94189]: pgmap v798: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 3.1 KiB/s wr, 6 op/s 2023-03-09T15:07:00.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:07:00 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi038 to 197.2M 2023-03-09T15:07:00.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:07:00 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi038 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-03-09T15:07:00.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:07:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 530.945771 secs 2023-03-09T15:07:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:07:00 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T15:07:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:07:00 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T15:07:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:07:00 smithi038 ceph-mon[92422]: pgmap v798: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 3.1 KiB/s wr, 6 op/s 2023-03-09T15:07:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:07:00 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi038 to 197.2M 2023-03-09T15:07:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:07:00 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi038 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-03-09T15:07:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:07:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 530.945771 secs 2023-03-09T15:07:02.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:07:02 smithi122 ceph-mon[93869]: pgmap v799: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 3.2 KiB/s wr, 6 op/s 2023-03-09T15:07:02.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:07:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 533.130884 secs 2023-03-09T15:07:02.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:07:02 smithi181 ceph-mon[94189]: pgmap v799: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 3.2 KiB/s wr, 6 op/s 2023-03-09T15:07:02.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:07:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 533.130884 secs 2023-03-09T15:07:03.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:07:02 smithi038 ceph-mon[92422]: pgmap v799: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 3.2 KiB/s wr, 6 op/s 2023-03-09T15:07:03.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:07:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 533.130884 secs 2023-03-09T15:07:03.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 995.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:07:04.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:07:04 smithi181 ceph-mon[94189]: pgmap v800: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 37 KiB/s wr, 11 op/s 2023-03-09T15:07:04.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:07:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:07:04.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:07:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:07:04.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:07:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:07:04.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:07:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:07:04.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:07:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:07:04.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:07:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:07:04.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:07:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:07:04.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:07:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:07:04.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:07:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:07:04.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:07:04 smithi038 ceph-mon[92422]: pgmap v800: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 37 KiB/s wr, 11 op/s 2023-03-09T15:07:04.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:07:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:07:04.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:07:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:07:04.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:07:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:07:04.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:07:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:07:04.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:07:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:07:04.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:07:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:07:04.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:07:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:07:04.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:07:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:07:04.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:07:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:07:04.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:07:04 smithi122 ceph-mon[93869]: pgmap v800: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 37 KiB/s wr, 11 op/s 2023-03-09T15:07:04.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:07:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:07:04.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:07:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:07:04.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:07:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:07:04.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:07:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:07:04.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:07:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:07:04.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:07:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:07:04.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:07:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:07:04.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:07:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:07:04.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:07:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:07:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:07:05 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T15:07:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:07:05 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T15:07:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:07:05 smithi038 ceph-mon[92422]: osdmap e264: 12 total, 12 up, 12 in 2023-03-09T15:07:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:07:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 535.945874 secs 2023-03-09T15:07:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:07:05 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T15:07:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:07:05 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T15:07:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:07:05 smithi122 ceph-mon[93869]: osdmap e264: 12 total, 12 up, 12 in 2023-03-09T15:07:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:07:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 535.945874 secs 2023-03-09T15:07:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:07:05 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T15:07:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:07:05 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T15:07:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:07:05 smithi181 ceph-mon[94189]: osdmap e264: 12 total, 12 up, 12 in 2023-03-09T15:07:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:07:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 535.945874 secs 2023-03-09T15:07:06.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:07:06 smithi122 ceph-mon[93869]: pgmap v802: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 70 KiB/s wr, 25 op/s 2023-03-09T15:07:06.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:07:06 smithi181 ceph-mon[94189]: pgmap v802: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 70 KiB/s wr, 25 op/s 2023-03-09T15:07:07.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:07:06 smithi038 ceph-mon[92422]: pgmap v802: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 70 KiB/s wr, 25 op/s 2023-03-09T15:07:07.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:07:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 538.131001 secs 2023-03-09T15:07:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:07:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 538.131001 secs 2023-03-09T15:07:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:07:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 538.131001 secs 2023-03-09T15:07:08.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1000.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:07:08.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:07:08 smithi122 ceph-mon[93869]: pgmap v803: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 67 KiB/s wr, 25 op/s 2023-03-09T15:07:08.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:07:08 smithi181 ceph-mon[94189]: pgmap v803: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 67 KiB/s wr, 25 op/s 2023-03-09T15:07:09.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:07:08 smithi038 ceph-mon[92422]: pgmap v803: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 67 KiB/s wr, 25 op/s 2023-03-09T15:07:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:07:10 smithi122 ceph-mon[93869]: pgmap v804: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 67 KiB/s wr, 25 op/s 2023-03-09T15:07:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:07:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 540.945937 secs 2023-03-09T15:07:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:07:10 smithi181 ceph-mon[94189]: pgmap v804: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 67 KiB/s wr, 25 op/s 2023-03-09T15:07:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:07:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 540.945937 secs 2023-03-09T15:07:11.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:07:10 smithi038 ceph-mon[92422]: pgmap v804: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 67 KiB/s wr, 25 op/s 2023-03-09T15:07:11.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:07:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 540.945937 secs 2023-03-09T15:07:12.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:07:12 smithi122 ceph-mon[93869]: pgmap v805: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 66 KiB/s wr, 25 op/s 2023-03-09T15:07:12.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:07:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 543.131125 secs 2023-03-09T15:07:12.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:07:12 smithi181 ceph-mon[94189]: pgmap v805: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 66 KiB/s wr, 25 op/s 2023-03-09T15:07:12.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:07:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 543.131125 secs 2023-03-09T15:07:13.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:07:12 smithi038 ceph-mon[92422]: pgmap v805: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 66 KiB/s wr, 25 op/s 2023-03-09T15:07:13.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:07:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 543.131125 secs 2023-03-09T15:07:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1005.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:07:14.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:07:14 smithi122 ceph-mon[93869]: osdmap e265: 12 total, 12 up, 12 in 2023-03-09T15:07:14.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:07:14 smithi122 ceph-mon[93869]: pgmap v807: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 470 B/s wr, 8 op/s 2023-03-09T15:07:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:07:14 smithi181 ceph-mon[94189]: osdmap e265: 12 total, 12 up, 12 in 2023-03-09T15:07:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:07:14 smithi181 ceph-mon[94189]: pgmap v807: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 470 B/s wr, 8 op/s 2023-03-09T15:07:14.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:07:14 smithi038 ceph-mon[92422]: osdmap e265: 12 total, 12 up, 12 in 2023-03-09T15:07:14.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:07:14 smithi038 ceph-mon[92422]: pgmap v807: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 470 B/s wr, 8 op/s 2023-03-09T15:07:15.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:07:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 545.945985 secs 2023-03-09T15:07:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:07:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 545.945985 secs 2023-03-09T15:07:15.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:07:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 545.945985 secs 2023-03-09T15:07:16.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:07:16 smithi122 ceph-mon[93869]: pgmap v808: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 409 B/s wr, 7 op/s 2023-03-09T15:07:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:07:16 smithi181 ceph-mon[94189]: pgmap v808: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 409 B/s wr, 7 op/s 2023-03-09T15:07:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:07:16 smithi038 ceph-mon[92422]: pgmap v808: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 409 B/s wr, 7 op/s 2023-03-09T15:07:17.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:07:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 548.131246 secs 2023-03-09T15:07:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:07:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 548.131246 secs 2023-03-09T15:07:17.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:07:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 548.131246 secs 2023-03-09T15:07:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:07:18 smithi122 ceph-mon[93869]: pgmap v809: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:07:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:07:18 smithi181 ceph-mon[94189]: pgmap v809: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:07:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:07:18 smithi038 ceph-mon[92422]: pgmap v809: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:07:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1010.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:07:20.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:07:20 smithi122 ceph-mon[93869]: pgmap v810: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:07:20.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:07:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 550.946069 secs 2023-03-09T15:07:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:07:20 smithi181 ceph-mon[94189]: pgmap v810: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:07:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:07:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 550.946069 secs 2023-03-09T15:07:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:07:20 smithi038 ceph-mon[92422]: pgmap v810: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:07:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:07:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 550.946069 secs 2023-03-09T15:07:22.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:07:22 smithi122 ceph-mon[93869]: pgmap v811: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:07:22.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:07:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 553.131370 secs 2023-03-09T15:07:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:07:22 smithi181 ceph-mon[94189]: pgmap v811: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:07:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:07:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 553.131370 secs 2023-03-09T15:07:22.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:07:22 smithi038 ceph-mon[92422]: pgmap v811: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:07:22.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:07:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 553.131370 secs 2023-03-09T15:07:23.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1015.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:07:24.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:07:24 smithi122 ceph-mon[93869]: pgmap v812: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T15:07:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:07:24 smithi181 ceph-mon[94189]: pgmap v812: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T15:07:24.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:07:24 smithi038 ceph-mon[92422]: pgmap v812: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T15:07:25.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:07:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 555.946129 secs 2023-03-09T15:07:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:07:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 555.946129 secs 2023-03-09T15:07:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:07:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 555.946129 secs 2023-03-09T15:07:26.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:07:26 smithi122 ceph-mon[93869]: pgmap v813: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T15:07:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:07:26 smithi181 ceph-mon[94189]: pgmap v813: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T15:07:26.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:07:26 smithi038 ceph-mon[92422]: pgmap v813: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T15:07:27.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:07:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 558.131478 secs 2023-03-09T15:07:27.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:07:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 558.131478 secs 2023-03-09T15:07:27.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:07:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 558.131478 secs 2023-03-09T15:07:28.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:07:28 smithi122 ceph-mon[93869]: pgmap v814: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 5.2 KiB/s wr, 7 op/s 2023-03-09T15:07:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:07:28 smithi181 ceph-mon[94189]: pgmap v814: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 5.2 KiB/s wr, 7 op/s 2023-03-09T15:07:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:07:28 smithi038 ceph-mon[92422]: pgmap v814: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 5.2 KiB/s wr, 7 op/s 2023-03-09T15:07:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1020.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:07:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:07:30 smithi038 ceph-mon[92422]: pgmap v815: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 5.2 KiB/s wr, 6 op/s 2023-03-09T15:07:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:07:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 560.946188 secs 2023-03-09T15:07:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:07:30 smithi122 ceph-mon[93869]: pgmap v815: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 5.2 KiB/s wr, 6 op/s 2023-03-09T15:07:30.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:07:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 560.946188 secs 2023-03-09T15:07:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:07:30 smithi181 ceph-mon[94189]: pgmap v815: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 5.2 KiB/s wr, 6 op/s 2023-03-09T15:07:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:07:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 560.946188 secs 2023-03-09T15:07:32.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:07:32 smithi122 ceph-mon[93869]: pgmap v816: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 5.2 KiB/s wr, 6 op/s 2023-03-09T15:07:32.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:07:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 563.131595 secs 2023-03-09T15:07:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:07:32 smithi181 ceph-mon[94189]: pgmap v816: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 5.2 KiB/s wr, 6 op/s 2023-03-09T15:07:32.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:07:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 563.131595 secs 2023-03-09T15:07:32.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:07:32 smithi038 ceph-mon[92422]: pgmap v816: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 5.2 KiB/s wr, 6 op/s 2023-03-09T15:07:32.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:07:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 563.131595 secs 2023-03-09T15:07:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1025.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:07:34.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:07:34 smithi122 ceph-mon[93869]: pgmap v817: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 20 KiB/s wr, 9 op/s 2023-03-09T15:07:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:07:34 smithi181 ceph-mon[94189]: pgmap v817: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 20 KiB/s wr, 9 op/s 2023-03-09T15:07:34.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:07:34 smithi038 ceph-mon[92422]: pgmap v817: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 20 KiB/s wr, 9 op/s 2023-03-09T15:07:35.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:07:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 565.946249 secs 2023-03-09T15:07:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:07:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 565.946249 secs 2023-03-09T15:07:35.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:07:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 565.946249 secs 2023-03-09T15:07:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:07:36 smithi122 ceph-mon[93869]: pgmap v818: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T15:07:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:07:36 smithi181 ceph-mon[94189]: pgmap v818: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T15:07:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:07:36 smithi038 ceph-mon[92422]: pgmap v818: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T15:07:37.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:07:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 568.131714 secs 2023-03-09T15:07:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:07:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 568.131714 secs 2023-03-09T15:07:37.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:07:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 568.131714 secs 2023-03-09T15:07:38.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:07:38 smithi122 ceph-mon[93869]: pgmap v819: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 20 KiB/s wr, 10 op/s 2023-03-09T15:07:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:07:38 smithi181 ceph-mon[94189]: pgmap v819: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 20 KiB/s wr, 10 op/s 2023-03-09T15:07:38.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:07:38 smithi038 ceph-mon[92422]: pgmap v819: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 20 KiB/s wr, 10 op/s 2023-03-09T15:07:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1030.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:07:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:07:40 smithi122 ceph-mon[93869]: pgmap v820: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 KiB/s wr, 7 op/s 2023-03-09T15:07:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:07:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 570.946309 secs 2023-03-09T15:07:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:07:40 smithi181 ceph-mon[94189]: pgmap v820: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 KiB/s wr, 7 op/s 2023-03-09T15:07:40.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:07:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 570.946309 secs 2023-03-09T15:07:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:07:40 smithi038 ceph-mon[92422]: pgmap v820: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 KiB/s wr, 7 op/s 2023-03-09T15:07:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:07:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 570.946309 secs 2023-03-09T15:07:42.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:07:42 smithi122 ceph-mon[93869]: pgmap v821: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 KiB/s wr, 7 op/s 2023-03-09T15:07:42.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:07:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 573.131832 secs 2023-03-09T15:07:42.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:07:42 smithi181 ceph-mon[94189]: pgmap v821: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 KiB/s wr, 7 op/s 2023-03-09T15:07:42.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:07:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 573.131832 secs 2023-03-09T15:07:42.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:07:42 smithi038 ceph-mon[92422]: pgmap v821: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 KiB/s wr, 7 op/s 2023-03-09T15:07:42.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:07:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 573.131832 secs 2023-03-09T15:07:43.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1035.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:07:44.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:07:44 smithi122 ceph-mon[93869]: pgmap v822: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 21 KiB/s wr, 9 op/s 2023-03-09T15:07:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:07:44 smithi181 ceph-mon[94189]: pgmap v822: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 21 KiB/s wr, 9 op/s 2023-03-09T15:07:44.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:07:44 smithi038 ceph-mon[92422]: pgmap v822: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 21 KiB/s wr, 9 op/s 2023-03-09T15:07:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:07:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 575.946369 secs 2023-03-09T15:07:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:07:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 575.946369 secs 2023-03-09T15:07:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:07:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 575.946369 secs 2023-03-09T15:07:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:07:46 smithi122 ceph-mon[93869]: pgmap v823: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.4 KiB/s wr, 5 op/s 2023-03-09T15:07:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:07:46 smithi181 ceph-mon[94189]: pgmap v823: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.4 KiB/s wr, 5 op/s 2023-03-09T15:07:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:07:46 smithi038 ceph-mon[92422]: pgmap v823: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.4 KiB/s wr, 5 op/s 2023-03-09T15:07:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:07:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 578.131945 secs 2023-03-09T15:07:47.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:07:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 578.131945 secs 2023-03-09T15:07:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:07:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 578.131945 secs 2023-03-09T15:07:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:07:48 smithi122 ceph-mon[93869]: pgmap v824: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 6.3 KiB/s wr, 7 op/s 2023-03-09T15:07:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:07:48 smithi181 ceph-mon[94189]: pgmap v824: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 6.3 KiB/s wr, 7 op/s 2023-03-09T15:07:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:07:48 smithi038 ceph-mon[92422]: pgmap v824: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 6.3 KiB/s wr, 7 op/s 2023-03-09T15:07:48.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1040.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:07:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:07:50 smithi122 ceph-mon[93869]: pgmap v825: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 6.2 KiB/s wr, 5 op/s 2023-03-09T15:07:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:07:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 580.946428 secs 2023-03-09T15:07:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:07:50 smithi181 ceph-mon[94189]: pgmap v825: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 6.2 KiB/s wr, 5 op/s 2023-03-09T15:07:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:07:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 580.946428 secs 2023-03-09T15:07:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:07:50 smithi038 ceph-mon[92422]: pgmap v825: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 6.2 KiB/s wr, 5 op/s 2023-03-09T15:07:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:07:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 580.946428 secs 2023-03-09T15:07:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:07:52 smithi122 ceph-mon[93869]: pgmap v826: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 6.2 KiB/s wr, 5 op/s 2023-03-09T15:07:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:07:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 583.132066 secs 2023-03-09T15:07:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:07:52 smithi181 ceph-mon[94189]: pgmap v826: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 6.2 KiB/s wr, 5 op/s 2023-03-09T15:07:52.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:07:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 583.132066 secs 2023-03-09T15:07:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:07:52 smithi038 ceph-mon[92422]: pgmap v826: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 6.2 KiB/s wr, 5 op/s 2023-03-09T15:07:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:07:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 583.132066 secs 2023-03-09T15:07:53.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1045.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:07:54.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:07:54 smithi122 ceph-mon[93869]: pgmap v827: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 6.2 KiB/s wr, 7 op/s 2023-03-09T15:07:54.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:07:54 smithi181 ceph-mon[94189]: pgmap v827: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 6.2 KiB/s wr, 7 op/s 2023-03-09T15:07:54.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:07:54 smithi038 ceph-mon[92422]: pgmap v827: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 6.2 KiB/s wr, 7 op/s 2023-03-09T15:07:55.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:07:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 585.946488 secs 2023-03-09T15:07:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:07:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 585.946488 secs 2023-03-09T15:07:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:07:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 585.946488 secs 2023-03-09T15:07:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:07:56 smithi122 ceph-mon[93869]: pgmap v828: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T15:07:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:07:56 smithi181 ceph-mon[94189]: pgmap v828: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T15:07:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:07:56 smithi038 ceph-mon[92422]: pgmap v828: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T15:07:57.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:07:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 588.132220 secs 2023-03-09T15:07:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:07:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 588.132220 secs 2023-03-09T15:07:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:07:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 588.132220 secs 2023-03-09T15:07:58.450 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:07:58 smithi038 ceph-mon[92422]: pgmap v829: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 2.4 KiB/s wr, 7 op/s 2023-03-09T15:07:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:07:58 smithi122 ceph-mon[93869]: pgmap v829: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 2.4 KiB/s wr, 7 op/s 2023-03-09T15:07:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:07:58 smithi181 ceph-mon[94189]: pgmap v829: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 2.4 KiB/s wr, 7 op/s 2023-03-09T15:07:58.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1050.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:08:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:08:00 smithi122 ceph-mon[93869]: pgmap v830: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 3.7 KiB/s wr, 5 op/s 2023-03-09T15:08:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:08:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 590.946572 secs 2023-03-09T15:08:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:08:00 smithi181 ceph-mon[94189]: pgmap v830: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 3.7 KiB/s wr, 5 op/s 2023-03-09T15:08:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:08:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 590.946572 secs 2023-03-09T15:08:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:08:00 smithi038 ceph-mon[92422]: pgmap v830: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 3.7 KiB/s wr, 5 op/s 2023-03-09T15:08:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:08:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 590.946572 secs 2023-03-09T15:08:02.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:08:02 smithi122 ceph-mon[93869]: pgmap v831: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 3.8 KiB/s wr, 6 op/s 2023-03-09T15:08:02.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:08:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 593.132351 secs 2023-03-09T15:08:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:08:02 smithi181 ceph-mon[94189]: pgmap v831: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 3.8 KiB/s wr, 6 op/s 2023-03-09T15:08:02.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:08:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 593.132351 secs 2023-03-09T15:08:02.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:08:02 smithi038 ceph-mon[92422]: pgmap v831: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 3.8 KiB/s wr, 6 op/s 2023-03-09T15:08:02.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:08:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 593.132351 secs 2023-03-09T15:08:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1055.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:08:04.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:08:04 smithi122 ceph-mon[93869]: pgmap v832: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 49 KiB/s wr, 23 op/s 2023-03-09T15:08:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:08:04 smithi181 ceph-mon[94189]: pgmap v832: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 49 KiB/s wr, 23 op/s 2023-03-09T15:08:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:08:04 smithi038 ceph-mon[92422]: pgmap v832: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 49 KiB/s wr, 23 op/s 2023-03-09T15:08:05.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:08:05 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:08:05.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:08:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 595.946635 secs 2023-03-09T15:08:05.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:08:05 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:08:05.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:08:05 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:08:05.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:08:05 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:08:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:08:05 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:08:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:08:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 595.946635 secs 2023-03-09T15:08:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:08:05 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:08:05.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:08:05 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:08:05.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:08:05 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:08:05.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:08:05 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:08:05.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:08:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 595.946635 secs 2023-03-09T15:08:05.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:08:05 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:08:05.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:08:05 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:08:05.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:08:05 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:08:06.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:08:06 smithi122 ceph-mon[93869]: pgmap v833: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 49 KiB/s wr, 21 op/s 2023-03-09T15:08:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:08:06 smithi181 ceph-mon[94189]: pgmap v833: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 49 KiB/s wr, 21 op/s 2023-03-09T15:08:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:08:06 smithi038 ceph-mon[92422]: pgmap v833: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 49 KiB/s wr, 21 op/s 2023-03-09T15:08:07.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:08:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 598.132487 secs 2023-03-09T15:08:07.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:08:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 598.132487 secs 2023-03-09T15:08:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:08:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 598.132487 secs 2023-03-09T15:08:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:08:08 smithi122 ceph-mon[93869]: pgmap v834: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 50 KiB/s wr, 23 op/s 2023-03-09T15:08:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:08:08 smithi181 ceph-mon[94189]: pgmap v834: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 50 KiB/s wr, 23 op/s 2023-03-09T15:08:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:08:08 smithi038 ceph-mon[92422]: pgmap v834: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 50 KiB/s wr, 23 op/s 2023-03-09T15:08:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1060.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:08:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:08:10 smithi122 ceph-mon[93869]: pgmap v836: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 66 KiB/s wr, 26 op/s 2023-03-09T15:08:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:08:10 smithi122 ceph-mon[93869]: osdmap e266: 12 total, 12 up, 12 in 2023-03-09T15:08:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:08:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 600.946696 secs 2023-03-09T15:08:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:08:10 smithi181 ceph-mon[94189]: pgmap v836: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 66 KiB/s wr, 26 op/s 2023-03-09T15:08:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:08:10 smithi181 ceph-mon[94189]: osdmap e266: 12 total, 12 up, 12 in 2023-03-09T15:08:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:08:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 600.946696 secs 2023-03-09T15:08:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:08:10 smithi038 ceph-mon[92422]: pgmap v836: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 66 KiB/s wr, 26 op/s 2023-03-09T15:08:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:08:10 smithi038 ceph-mon[92422]: osdmap e266: 12 total, 12 up, 12 in 2023-03-09T15:08:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:08:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 600.946696 secs 2023-03-09T15:08:12.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:08:12 smithi122 ceph-mon[93869]: pgmap v837: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 66 KiB/s wr, 25 op/s 2023-03-09T15:08:12.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:08:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 603.132604 secs 2023-03-09T15:08:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:08:12 smithi181 ceph-mon[94189]: pgmap v837: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 66 KiB/s wr, 25 op/s 2023-03-09T15:08:12.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:08:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 603.132604 secs 2023-03-09T15:08:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:08:12 smithi038 ceph-mon[92422]: pgmap v837: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 66 KiB/s wr, 25 op/s 2023-03-09T15:08:12.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:08:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 603.132604 secs 2023-03-09T15:08:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1065.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:08:14.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:08:14 smithi122 ceph-mon[93869]: pgmap v838: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 7 op/s 2023-03-09T15:08:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:08:14 smithi181 ceph-mon[94189]: pgmap v838: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 7 op/s 2023-03-09T15:08:14.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:08:14 smithi038 ceph-mon[92422]: pgmap v838: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 7 op/s 2023-03-09T15:08:15.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:08:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 605.946755 secs 2023-03-09T15:08:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:08:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 605.946755 secs 2023-03-09T15:08:15.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:08:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 605.946755 secs 2023-03-09T15:08:16.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:08:16 smithi122 ceph-mon[93869]: pgmap v839: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 7 op/s 2023-03-09T15:08:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:08:16 smithi181 ceph-mon[94189]: pgmap v839: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 7 op/s 2023-03-09T15:08:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:08:16 smithi038 ceph-mon[92422]: pgmap v839: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 7 op/s 2023-03-09T15:08:17.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:08:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 608.132716 secs 2023-03-09T15:08:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:08:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 608.132716 secs 2023-03-09T15:08:17.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:08:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 608.132716 secs 2023-03-09T15:08:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:08:18 smithi122 ceph-mon[93869]: pgmap v840: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 11 KiB/s wr, 7 op/s 2023-03-09T15:08:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:08:18 smithi122 ceph-mon[93869]: osdmap e267: 12 total, 12 up, 12 in 2023-03-09T15:08:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:08:18 smithi181 ceph-mon[94189]: pgmap v840: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 11 KiB/s wr, 7 op/s 2023-03-09T15:08:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:08:18 smithi181 ceph-mon[94189]: osdmap e267: 12 total, 12 up, 12 in 2023-03-09T15:08:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:08:18 smithi038 ceph-mon[92422]: pgmap v840: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 11 KiB/s wr, 7 op/s 2023-03-09T15:08:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:08:18 smithi038 ceph-mon[92422]: osdmap e267: 12 total, 12 up, 12 in 2023-03-09T15:08:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1070.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:08:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:08:20 smithi122 ceph-mon[93869]: pgmap v842: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:08:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:08:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 610.946814 secs 2023-03-09T15:08:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:08:20 smithi181 ceph-mon[94189]: pgmap v842: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:08:20.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:08:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 610.946814 secs 2023-03-09T15:08:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:08:20 smithi038 ceph-mon[92422]: pgmap v842: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:08:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:08:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 610.946814 secs 2023-03-09T15:08:22.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:08:22 smithi122 ceph-mon[93869]: pgmap v843: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:08:22.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:08:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 613.132864 secs 2023-03-09T15:08:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:08:22 smithi181 ceph-mon[94189]: pgmap v843: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:08:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:08:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 613.132864 secs 2023-03-09T15:08:22.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:08:22 smithi038 ceph-mon[92422]: pgmap v843: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:08:22.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:08:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 613.132864 secs 2023-03-09T15:08:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1075.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:08:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:08:24 smithi122 ceph-mon[93869]: pgmap v844: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:08:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:08:24 smithi181 ceph-mon[94189]: pgmap v844: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:08:24.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:08:24 smithi038 ceph-mon[92422]: pgmap v844: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:08:25.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:08:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 615.946876 secs 2023-03-09T15:08:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:08:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 615.946876 secs 2023-03-09T15:08:25.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:08:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 615.946876 secs 2023-03-09T15:08:26.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:08:26 smithi122 ceph-mon[93869]: pgmap v845: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:08:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:08:26 smithi181 ceph-mon[94189]: pgmap v845: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:08:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:08:26 smithi038 ceph-mon[92422]: pgmap v845: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:08:27.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:08:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 618.132983 secs 2023-03-09T15:08:27.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:08:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 618.132983 secs 2023-03-09T15:08:27.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:08:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 618.132983 secs 2023-03-09T15:08:28.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:08:28 smithi122 ceph-mon[93869]: pgmap v846: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 6.5 KiB/s wr, 7 op/s 2023-03-09T15:08:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:08:28 smithi181 ceph-mon[94189]: pgmap v846: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 6.5 KiB/s wr, 7 op/s 2023-03-09T15:08:28.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:08:28 smithi038 ceph-mon[92422]: pgmap v846: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 6.5 KiB/s wr, 7 op/s 2023-03-09T15:08:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1080.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:08:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:08:30 smithi122 ceph-mon[93869]: pgmap v847: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 5.7 KiB/s wr, 6 op/s 2023-03-09T15:08:30.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:08:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 620.946939 secs 2023-03-09T15:08:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:08:30 smithi181 ceph-mon[94189]: pgmap v847: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 5.7 KiB/s wr, 6 op/s 2023-03-09T15:08:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:08:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 620.946939 secs 2023-03-09T15:08:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:08:30 smithi038 ceph-mon[92422]: pgmap v847: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 5.7 KiB/s wr, 6 op/s 2023-03-09T15:08:30.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:08:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 620.946939 secs 2023-03-09T15:08:32.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:08:32 smithi122 ceph-mon[93869]: pgmap v848: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 5.3 KiB/s wr, 6 op/s 2023-03-09T15:08:32.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:08:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 623.133099 secs 2023-03-09T15:08:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:08:32 smithi181 ceph-mon[94189]: pgmap v848: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 5.3 KiB/s wr, 6 op/s 2023-03-09T15:08:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:08:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 623.133099 secs 2023-03-09T15:08:32.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:08:32 smithi038 ceph-mon[92422]: pgmap v848: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 5.3 KiB/s wr, 6 op/s 2023-03-09T15:08:32.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:08:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 623.133099 secs 2023-03-09T15:08:33.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1085.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:08:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:08:34 smithi122 ceph-mon[93869]: pgmap v849: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 20 KiB/s wr, 9 op/s 2023-03-09T15:08:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:08:34 smithi181 ceph-mon[94189]: pgmap v849: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 20 KiB/s wr, 9 op/s 2023-03-09T15:08:34.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:08:34 smithi038 ceph-mon[92422]: pgmap v849: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 20 KiB/s wr, 9 op/s 2023-03-09T15:08:35.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:08:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 625.946985 secs 2023-03-09T15:08:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:08:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 625.946985 secs 2023-03-09T15:08:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:08:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 625.946985 secs 2023-03-09T15:08:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:08:36 smithi122 ceph-mon[93869]: pgmap v850: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T15:08:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:08:36 smithi181 ceph-mon[94189]: pgmap v850: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T15:08:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:08:36 smithi038 ceph-mon[92422]: pgmap v850: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T15:08:37.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:08:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 628.133235 secs 2023-03-09T15:08:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:08:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 628.133235 secs 2023-03-09T15:08:37.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:08:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 628.133235 secs 2023-03-09T15:08:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:08:38 smithi122 ceph-mon[93869]: pgmap v851: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 21 KiB/s wr, 10 op/s 2023-03-09T15:08:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:08:38 smithi181 ceph-mon[94189]: pgmap v851: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 21 KiB/s wr, 10 op/s 2023-03-09T15:08:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:08:38 smithi038 ceph-mon[92422]: pgmap v851: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 21 KiB/s wr, 10 op/s 2023-03-09T15:08:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1090.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:08:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:08:40 smithi122 ceph-mon[93869]: pgmap v852: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 KiB/s wr, 7 op/s 2023-03-09T15:08:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:08:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 630.947044 secs 2023-03-09T15:08:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:08:40 smithi181 ceph-mon[94189]: pgmap v852: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 KiB/s wr, 7 op/s 2023-03-09T15:08:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:08:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 630.947044 secs 2023-03-09T15:08:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:08:40 smithi038 ceph-mon[92422]: pgmap v852: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 KiB/s wr, 7 op/s 2023-03-09T15:08:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:08:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 630.947044 secs 2023-03-09T15:08:42.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:08:42 smithi122 ceph-mon[93869]: pgmap v853: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 KiB/s wr, 7 op/s 2023-03-09T15:08:42.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:08:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 633.133349 secs 2023-03-09T15:08:42.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:08:42 smithi181 ceph-mon[94189]: pgmap v853: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 KiB/s wr, 7 op/s 2023-03-09T15:08:42.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:08:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 633.133349 secs 2023-03-09T15:08:42.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:08:42 smithi038 ceph-mon[92422]: pgmap v853: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 KiB/s wr, 7 op/s 2023-03-09T15:08:42.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:08:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 633.133349 secs 2023-03-09T15:08:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1095.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:08:44.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:08:44 smithi122 ceph-mon[93869]: pgmap v854: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 9 op/s 2023-03-09T15:08:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:08:44 smithi181 ceph-mon[94189]: pgmap v854: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 9 op/s 2023-03-09T15:08:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:08:44 smithi038 ceph-mon[92422]: pgmap v854: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 9 op/s 2023-03-09T15:08:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:08:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 635.947105 secs 2023-03-09T15:08:45.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:08:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 635.947105 secs 2023-03-09T15:08:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:08:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 635.947105 secs 2023-03-09T15:08:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:08:46 smithi122 ceph-mon[93869]: pgmap v855: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.5 KiB/s wr, 5 op/s 2023-03-09T15:08:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:08:46 smithi181 ceph-mon[94189]: pgmap v855: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.5 KiB/s wr, 5 op/s 2023-03-09T15:08:46.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:08:46 smithi038 ceph-mon[92422]: pgmap v855: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.5 KiB/s wr, 5 op/s 2023-03-09T15:08:47.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:08:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 638.133466 secs 2023-03-09T15:08:47.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:08:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 638.133466 secs 2023-03-09T15:08:47.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:08:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 638.133466 secs 2023-03-09T15:08:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:08:48 smithi122 ceph-mon[93869]: pgmap v856: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 6.5 KiB/s wr, 7 op/s 2023-03-09T15:08:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:08:48 smithi181 ceph-mon[94189]: pgmap v856: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 6.5 KiB/s wr, 7 op/s 2023-03-09T15:08:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:08:48 smithi038 ceph-mon[92422]: pgmap v856: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 6.5 KiB/s wr, 7 op/s 2023-03-09T15:08:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1100.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:08:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:08:50 smithi122 ceph-mon[93869]: pgmap v857: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 6.3 KiB/s wr, 5 op/s 2023-03-09T15:08:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:08:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 640.947168 secs 2023-03-09T15:08:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:08:50 smithi181 ceph-mon[94189]: pgmap v857: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 6.3 KiB/s wr, 5 op/s 2023-03-09T15:08:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:08:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 640.947168 secs 2023-03-09T15:08:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:08:50 smithi038 ceph-mon[92422]: pgmap v857: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 6.3 KiB/s wr, 5 op/s 2023-03-09T15:08:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:08:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 640.947168 secs 2023-03-09T15:08:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:08:52 smithi122 ceph-mon[93869]: pgmap v858: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 6.3 KiB/s wr, 5 op/s 2023-03-09T15:08:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:08:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 643.133584 secs 2023-03-09T15:08:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:08:52 smithi181 ceph-mon[94189]: pgmap v858: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 6.3 KiB/s wr, 5 op/s 2023-03-09T15:08:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:08:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 643.133584 secs 2023-03-09T15:08:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:08:52 smithi038 ceph-mon[92422]: pgmap v858: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 6.3 KiB/s wr, 5 op/s 2023-03-09T15:08:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:08:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 643.133584 secs 2023-03-09T15:08:53.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1105.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:08:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:08:54 smithi122 ceph-mon[93869]: pgmap v859: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 6.3 KiB/s wr, 7 op/s 2023-03-09T15:08:54.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:08:54 smithi181 ceph-mon[94189]: pgmap v859: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 6.3 KiB/s wr, 7 op/s 2023-03-09T15:08:54.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:08:54 smithi038 ceph-mon[92422]: pgmap v859: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 6.3 KiB/s wr, 7 op/s 2023-03-09T15:08:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:08:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 645.947227 secs 2023-03-09T15:08:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:08:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 645.947227 secs 2023-03-09T15:08:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:08:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 645.947227 secs 2023-03-09T15:08:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:08:56 smithi122 ceph-mon[93869]: pgmap v860: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T15:08:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:08:56 smithi181 ceph-mon[94189]: pgmap v860: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T15:08:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:08:56 smithi038 ceph-mon[92422]: pgmap v860: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T15:08:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:08:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 648.133723 secs 2023-03-09T15:08:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:08:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 648.133723 secs 2023-03-09T15:08:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:08:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 648.133723 secs 2023-03-09T15:08:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:08:58 smithi122 ceph-mon[93869]: pgmap v861: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 511 B/s wr, 7 op/s 2023-03-09T15:08:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:08:58 smithi181 ceph-mon[94189]: pgmap v861: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 511 B/s wr, 7 op/s 2023-03-09T15:08:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:08:58 smithi038 ceph-mon[92422]: pgmap v861: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 511 B/s wr, 7 op/s 2023-03-09T15:08:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1110.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:09:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:09:00 smithi122 ceph-mon[93869]: pgmap v862: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 1.4 KiB/s wr, 5 op/s 2023-03-09T15:09:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:09:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 650.947286 secs 2023-03-09T15:09:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:09:00 smithi181 ceph-mon[94189]: pgmap v862: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 1.4 KiB/s wr, 5 op/s 2023-03-09T15:09:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:09:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 650.947286 secs 2023-03-09T15:09:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:09:00 smithi038 ceph-mon[92422]: pgmap v862: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 1.4 KiB/s wr, 5 op/s 2023-03-09T15:09:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:09:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 650.947286 secs 2023-03-09T15:09:02.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:09:02 smithi122 ceph-mon[93869]: pgmap v863: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2023-03-09T15:09:02.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:09:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 653.133843 secs 2023-03-09T15:09:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:09:02 smithi181 ceph-mon[94189]: pgmap v863: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2023-03-09T15:09:02.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:09:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 653.133843 secs 2023-03-09T15:09:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:09:02 smithi038 ceph-mon[92422]: pgmap v863: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 1.7 KiB/s wr, 5 op/s 2023-03-09T15:09:02.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:09:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 653.133843 secs 2023-03-09T15:09:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1115.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:09:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:09:04 smithi122 ceph-mon[93869]: pgmap v864: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 49 KiB/s wr, 22 op/s 2023-03-09T15:09:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:09:04 smithi181 ceph-mon[94189]: pgmap v864: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 49 KiB/s wr, 22 op/s 2023-03-09T15:09:04.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:09:04 smithi038 ceph-mon[92422]: pgmap v864: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 49 KiB/s wr, 22 op/s 2023-03-09T15:09:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:09:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 655.947346 secs 2023-03-09T15:09:05.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:09:05 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:09:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:09:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 655.947346 secs 2023-03-09T15:09:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:09:05 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:09:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:09:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 655.947346 secs 2023-03-09T15:09:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:09:05 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:09:06.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:09:06 smithi122 ceph-mon[93869]: pgmap v865: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 49 KiB/s wr, 20 op/s 2023-03-09T15:09:06.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:09:06 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:09:06.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:09:06 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:09:06.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:09:06 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:09:06.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:09:06 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:09:06.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:09:06 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:09:06.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:09:06 smithi181 ceph-mon[94189]: pgmap v865: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 49 KiB/s wr, 20 op/s 2023-03-09T15:09:06.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:09:06 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:09:06.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:09:06 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:09:06.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:09:06 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:09:06.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:09:06 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:09:06.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:09:06 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:09:07.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:09:06 smithi038 ceph-mon[92422]: pgmap v865: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 49 KiB/s wr, 20 op/s 2023-03-09T15:09:07.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:09:06 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:09:07.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:09:06 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:09:07.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:09:06 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:09:07.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:09:06 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:09:07.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:09:06 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:09:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:09:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 658.133966 secs 2023-03-09T15:09:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:09:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 658.133966 secs 2023-03-09T15:09:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:09:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 658.133966 secs 2023-03-09T15:09:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1120.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:09:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:09:08 smithi122 ceph-mon[93869]: pgmap v866: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 50 KiB/s wr, 23 op/s 2023-03-09T15:09:08.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:09:08 smithi181 ceph-mon[94189]: pgmap v866: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 50 KiB/s wr, 23 op/s 2023-03-09T15:09:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:09:08 smithi038 ceph-mon[92422]: pgmap v866: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 50 KiB/s wr, 23 op/s 2023-03-09T15:09:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:09:10 smithi122 ceph-mon[93869]: pgmap v867: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 59 KiB/s wr, 22 op/s 2023-03-09T15:09:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:09:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 660.947405 secs 2023-03-09T15:09:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:09:10 smithi122 ceph-mon[93869]: osdmap e268: 12 total, 12 up, 12 in 2023-03-09T15:09:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:09:10 smithi181 ceph-mon[94189]: pgmap v867: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 59 KiB/s wr, 22 op/s 2023-03-09T15:09:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:09:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 660.947405 secs 2023-03-09T15:09:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:09:10 smithi181 ceph-mon[94189]: osdmap e268: 12 total, 12 up, 12 in 2023-03-09T15:09:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:09:10 smithi038 ceph-mon[92422]: pgmap v867: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 59 KiB/s wr, 22 op/s 2023-03-09T15:09:11.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:09:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 660.947405 secs 2023-03-09T15:09:11.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:09:10 smithi038 ceph-mon[92422]: osdmap e268: 12 total, 12 up, 12 in 2023-03-09T15:09:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:09:12 smithi122 ceph-mon[93869]: pgmap v869: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 69 KiB/s wr, 26 op/s 2023-03-09T15:09:12.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:09:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 663.134064 secs 2023-03-09T15:09:12.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:09:12 smithi181 ceph-mon[94189]: pgmap v869: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 69 KiB/s wr, 26 op/s 2023-03-09T15:09:12.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:09:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 663.134064 secs 2023-03-09T15:09:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:09:12 smithi038 ceph-mon[92422]: pgmap v869: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 69 KiB/s wr, 26 op/s 2023-03-09T15:09:13.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:09:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 663.134064 secs 2023-03-09T15:09:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1125.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:09:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:09:14 smithi122 ceph-mon[93869]: pgmap v870: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 13 KiB/s wr, 8 op/s 2023-03-09T15:09:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:09:14 smithi181 ceph-mon[94189]: pgmap v870: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 13 KiB/s wr, 8 op/s 2023-03-09T15:09:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:09:14 smithi038 ceph-mon[92422]: pgmap v870: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 13 KiB/s wr, 8 op/s 2023-03-09T15:09:15.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:09:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 665.947467 secs 2023-03-09T15:09:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:09:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 665.947467 secs 2023-03-09T15:09:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:09:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 665.947467 secs 2023-03-09T15:09:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:09:16 smithi122 ceph-mon[93869]: pgmap v871: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 13 KiB/s wr, 8 op/s 2023-03-09T15:09:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:09:16 smithi181 ceph-mon[94189]: pgmap v871: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 13 KiB/s wr, 8 op/s 2023-03-09T15:09:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:09:16 smithi038 ceph-mon[92422]: pgmap v871: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 13 KiB/s wr, 8 op/s 2023-03-09T15:09:17.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:09:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 668.134202 secs 2023-03-09T15:09:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:09:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 668.134202 secs 2023-03-09T15:09:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:09:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 668.134202 secs 2023-03-09T15:09:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:09:18 smithi122 ceph-mon[93869]: pgmap v872: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 11 KiB/s wr, 7 op/s 2023-03-09T15:09:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:09:18 smithi122 ceph-mon[93869]: osdmap e269: 12 total, 12 up, 12 in 2023-03-09T15:09:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:09:18 smithi181 ceph-mon[94189]: pgmap v872: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 11 KiB/s wr, 7 op/s 2023-03-09T15:09:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:09:18 smithi181 ceph-mon[94189]: osdmap e269: 12 total, 12 up, 12 in 2023-03-09T15:09:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:09:18 smithi038 ceph-mon[92422]: pgmap v872: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 11 KiB/s wr, 7 op/s 2023-03-09T15:09:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:09:18 smithi038 ceph-mon[92422]: osdmap e269: 12 total, 12 up, 12 in 2023-03-09T15:09:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1130.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:09:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:09:20 smithi122 ceph-mon[93869]: pgmap v874: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 211 B/s wr, 6 op/s 2023-03-09T15:09:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:09:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 670.947527 secs 2023-03-09T15:09:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:09:20 smithi181 ceph-mon[94189]: pgmap v874: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 211 B/s wr, 6 op/s 2023-03-09T15:09:20.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:09:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 670.947527 secs 2023-03-09T15:09:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:09:20 smithi038 ceph-mon[92422]: pgmap v874: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 211 B/s wr, 6 op/s 2023-03-09T15:09:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:09:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 670.947527 secs 2023-03-09T15:09:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:09:22 smithi122 ceph-mon[93869]: pgmap v875: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:09:22.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:09:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 673.134339 secs 2023-03-09T15:09:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:09:22 smithi181 ceph-mon[94189]: pgmap v875: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:09:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:09:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 673.134339 secs 2023-03-09T15:09:22.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:09:22 smithi038 ceph-mon[92422]: pgmap v875: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:09:22.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:09:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 673.134339 secs 2023-03-09T15:09:23.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1135.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:09:24.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:09:24 smithi122 ceph-mon[93869]: pgmap v876: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:09:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:09:24 smithi181 ceph-mon[94189]: pgmap v876: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:09:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:09:24 smithi038 ceph-mon[92422]: pgmap v876: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:09:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:09:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 675.947589 secs 2023-03-09T15:09:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:09:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 675.947589 secs 2023-03-09T15:09:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:09:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 675.947589 secs 2023-03-09T15:09:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:09:26 smithi122 ceph-mon[93869]: pgmap v877: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:09:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:09:26 smithi181 ceph-mon[94189]: pgmap v877: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:09:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:09:26 smithi038 ceph-mon[92422]: pgmap v877: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:09:27.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:09:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 678.134455 secs 2023-03-09T15:09:27.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:09:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 678.134455 secs 2023-03-09T15:09:27.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:09:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 678.134455 secs 2023-03-09T15:09:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:09:28 smithi122 ceph-mon[93869]: pgmap v878: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 6.9 KiB/s wr, 7 op/s 2023-03-09T15:09:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:09:28 smithi181 ceph-mon[94189]: pgmap v878: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 6.9 KiB/s wr, 7 op/s 2023-03-09T15:09:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:09:28 smithi038 ceph-mon[92422]: pgmap v878: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 6.9 KiB/s wr, 7 op/s 2023-03-09T15:09:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1140.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:09:30.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:09:30 smithi122 ceph-mon[93869]: pgmap v879: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 6.1 KiB/s wr, 6 op/s 2023-03-09T15:09:30.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:09:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 680.947650 secs 2023-03-09T15:09:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:09:30 smithi181 ceph-mon[94189]: pgmap v879: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 6.1 KiB/s wr, 6 op/s 2023-03-09T15:09:30.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:09:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 680.947650 secs 2023-03-09T15:09:30.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:09:30 smithi038 ceph-mon[92422]: pgmap v879: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 6.1 KiB/s wr, 6 op/s 2023-03-09T15:09:30.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:09:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 680.947650 secs 2023-03-09T15:09:32.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:09:32 smithi122 ceph-mon[93869]: pgmap v880: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 5.7 KiB/s wr, 6 op/s 2023-03-09T15:09:32.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:09:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 683.134580 secs 2023-03-09T15:09:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:09:32 smithi181 ceph-mon[94189]: pgmap v880: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 5.7 KiB/s wr, 6 op/s 2023-03-09T15:09:32.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:09:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 683.134580 secs 2023-03-09T15:09:32.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:09:32 smithi038 ceph-mon[92422]: pgmap v880: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 5.7 KiB/s wr, 6 op/s 2023-03-09T15:09:32.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:09:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 683.134580 secs 2023-03-09T15:09:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1145.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:09:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:09:34 smithi122 ceph-mon[93869]: pgmap v881: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 21 KiB/s wr, 9 op/s 2023-03-09T15:09:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:09:34 smithi181 ceph-mon[94189]: pgmap v881: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 21 KiB/s wr, 9 op/s 2023-03-09T15:09:34.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:09:34 smithi038 ceph-mon[92422]: pgmap v881: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 21 KiB/s wr, 9 op/s 2023-03-09T15:09:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:09:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 685.947711 secs 2023-03-09T15:09:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:09:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 685.947711 secs 2023-03-09T15:09:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:09:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 685.947711 secs 2023-03-09T15:09:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:09:36 smithi122 ceph-mon[93869]: pgmap v882: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 21 KiB/s wr, 7 op/s 2023-03-09T15:09:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:09:36 smithi181 ceph-mon[94189]: pgmap v882: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 21 KiB/s wr, 7 op/s 2023-03-09T15:09:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:09:36 smithi038 ceph-mon[92422]: pgmap v882: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 21 KiB/s wr, 7 op/s 2023-03-09T15:09:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:09:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 688.134728 secs 2023-03-09T15:09:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:09:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 688.134728 secs 2023-03-09T15:09:37.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:09:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 688.134728 secs 2023-03-09T15:09:38.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:09:38 smithi122 ceph-mon[93869]: pgmap v883: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 21 KiB/s wr, 10 op/s 2023-03-09T15:09:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:09:38 smithi181 ceph-mon[94189]: pgmap v883: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 21 KiB/s wr, 10 op/s 2023-03-09T15:09:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:09:38 smithi038 ceph-mon[92422]: pgmap v883: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 21 KiB/s wr, 10 op/s 2023-03-09T15:09:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1150.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:09:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:09:40 smithi122 ceph-mon[93869]: pgmap v884: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T15:09:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:09:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 690.947771 secs 2023-03-09T15:09:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:09:40 smithi181 ceph-mon[94189]: pgmap v884: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T15:09:40.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:09:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 690.947771 secs 2023-03-09T15:09:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:09:40 smithi038 ceph-mon[92422]: pgmap v884: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T15:09:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:09:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 690.947771 secs 2023-03-09T15:09:42.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:09:42 smithi122 ceph-mon[93869]: pgmap v885: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T15:09:42.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:09:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 693.134863 secs 2023-03-09T15:09:42.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:09:42 smithi181 ceph-mon[94189]: pgmap v885: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T15:09:42.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:09:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 693.134863 secs 2023-03-09T15:09:42.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:09:42 smithi038 ceph-mon[92422]: pgmap v885: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T15:09:42.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:09:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 693.134863 secs 2023-03-09T15:09:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1155.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:09:44.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:09:44 smithi122 ceph-mon[93869]: pgmap v886: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 9 op/s 2023-03-09T15:09:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:09:44 smithi181 ceph-mon[94189]: pgmap v886: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 9 op/s 2023-03-09T15:09:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:09:44 smithi038 ceph-mon[92422]: pgmap v886: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 9 op/s 2023-03-09T15:09:45.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:09:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 695.947836 secs 2023-03-09T15:09:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:09:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 695.947836 secs 2023-03-09T15:09:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:09:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 695.947836 secs 2023-03-09T15:09:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:09:46 smithi122 ceph-mon[93869]: pgmap v887: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 6.7 KiB/s wr, 5 op/s 2023-03-09T15:09:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:09:46 smithi181 ceph-mon[94189]: pgmap v887: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 6.7 KiB/s wr, 5 op/s 2023-03-09T15:09:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:09:46 smithi038 ceph-mon[92422]: pgmap v887: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 6.7 KiB/s wr, 5 op/s 2023-03-09T15:09:47.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:09:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 698.134956 secs 2023-03-09T15:09:47.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:09:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 698.134956 secs 2023-03-09T15:09:47.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:09:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 698.134956 secs 2023-03-09T15:09:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:09:48 smithi122 ceph-mon[93869]: pgmap v888: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 6.7 KiB/s wr, 7 op/s 2023-03-09T15:09:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:09:48 smithi181 ceph-mon[94189]: pgmap v888: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 6.7 KiB/s wr, 7 op/s 2023-03-09T15:09:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:09:48 smithi038 ceph-mon[92422]: pgmap v888: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 6.7 KiB/s wr, 7 op/s 2023-03-09T15:09:48.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1160.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:09:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:09:50 smithi122 ceph-mon[93869]: pgmap v889: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 6.5 KiB/s wr, 5 op/s 2023-03-09T15:09:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:09:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 700.947896 secs 2023-03-09T15:09:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:09:50 smithi181 ceph-mon[94189]: pgmap v889: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 6.5 KiB/s wr, 5 op/s 2023-03-09T15:09:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:09:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 700.947896 secs 2023-03-09T15:09:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:09:50 smithi038 ceph-mon[92422]: pgmap v889: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 6.5 KiB/s wr, 5 op/s 2023-03-09T15:09:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:09:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 700.947896 secs 2023-03-09T15:09:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:09:52 smithi122 ceph-mon[93869]: pgmap v890: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 6.5 KiB/s wr, 5 op/s 2023-03-09T15:09:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:09:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 703.135065 secs 2023-03-09T15:09:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:09:52 smithi181 ceph-mon[94189]: pgmap v890: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 6.5 KiB/s wr, 5 op/s 2023-03-09T15:09:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:09:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 703.135065 secs 2023-03-09T15:09:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:09:52 smithi038 ceph-mon[92422]: pgmap v890: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 6.5 KiB/s wr, 5 op/s 2023-03-09T15:09:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:09:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 703.135065 secs 2023-03-09T15:09:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1165.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:09:54.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:09:54 smithi122 ceph-mon[93869]: pgmap v891: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 6.5 KiB/s wr, 7 op/s 2023-03-09T15:09:54.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:09:54 smithi181 ceph-mon[94189]: pgmap v891: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 6.5 KiB/s wr, 7 op/s 2023-03-09T15:09:54.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:09:54 smithi038 ceph-mon[92422]: pgmap v891: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 6.5 KiB/s wr, 7 op/s 2023-03-09T15:09:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:09:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 705.947957 secs 2023-03-09T15:09:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:09:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 705.947957 secs 2023-03-09T15:09:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:09:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 705.947957 secs 2023-03-09T15:09:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:09:56 smithi122 ceph-mon[93869]: pgmap v892: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T15:09:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:09:56 smithi181 ceph-mon[94189]: pgmap v892: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T15:09:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:09:56 smithi038 ceph-mon[92422]: pgmap v892: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T15:09:57.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:09:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 708.135181 secs 2023-03-09T15:09:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:09:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 708.135181 secs 2023-03-09T15:09:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:09:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 708.135181 secs 2023-03-09T15:09:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:09:58 smithi122 ceph-mon[93869]: pgmap v893: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 2.7 KiB/s wr, 7 op/s 2023-03-09T15:09:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:09:58 smithi181 ceph-mon[94189]: pgmap v893: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 2.7 KiB/s wr, 7 op/s 2023-03-09T15:09:58.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:09:58 smithi038 ceph-mon[92422]: pgmap v893: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 2.7 KiB/s wr, 7 op/s 2023-03-09T15:09:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1170.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:10:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:10:00 smithi122 ceph-mon[93869]: pgmap v894: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 4.0 KiB/s wr, 6 op/s 2023-03-09T15:10:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:10:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 710.947983 secs 2023-03-09T15:10:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:10:00 smithi122 ceph-mon[93869]: Health detail: HEALTH_WARN 2 MDSs report slow requests 2023-03-09T15:10:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:10:00 smithi122 ceph-mon[93869]: [WRN] MDS_SLOW_REQUEST: 2 MDSs report slow requests 2023-03-09T15:10:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:10:00 smithi122 ceph-mon[93869]: mds.i(mds.2): 2 slow requests are blocked > 180 secs 2023-03-09T15:10:00.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:10:00 smithi122 ceph-mon[93869]: mds.h(mds.1): 2 slow requests are blocked > 180 secs 2023-03-09T15:10:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:10:00 smithi181 ceph-mon[94189]: pgmap v894: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 4.0 KiB/s wr, 6 op/s 2023-03-09T15:10:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:10:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 710.947983 secs 2023-03-09T15:10:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:10:00 smithi181 ceph-mon[94189]: Health detail: HEALTH_WARN 2 MDSs report slow requests 2023-03-09T15:10:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:10:00 smithi181 ceph-mon[94189]: [WRN] MDS_SLOW_REQUEST: 2 MDSs report slow requests 2023-03-09T15:10:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:10:00 smithi181 ceph-mon[94189]: mds.i(mds.2): 2 slow requests are blocked > 180 secs 2023-03-09T15:10:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:10:00 smithi181 ceph-mon[94189]: mds.h(mds.1): 2 slow requests are blocked > 180 secs 2023-03-09T15:10:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:10:00 smithi038 ceph-mon[92422]: pgmap v894: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 4.0 KiB/s wr, 6 op/s 2023-03-09T15:10:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:10:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 710.947983 secs 2023-03-09T15:10:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:10:00 smithi038 ceph-mon[92422]: Health detail: HEALTH_WARN 2 MDSs report slow requests 2023-03-09T15:10:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:10:00 smithi038 ceph-mon[92422]: [WRN] MDS_SLOW_REQUEST: 2 MDSs report slow requests 2023-03-09T15:10:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:10:00 smithi038 ceph-mon[92422]: mds.i(mds.2): 2 slow requests are blocked > 180 secs 2023-03-09T15:10:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:10:00 smithi038 ceph-mon[92422]: mds.h(mds.1): 2 slow requests are blocked > 180 secs 2023-03-09T15:10:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:10:02 smithi122 ceph-mon[93869]: pgmap v895: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 4.1 KiB/s wr, 6 op/s 2023-03-09T15:10:02.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:10:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 713.135296 secs 2023-03-09T15:10:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:10:02 smithi181 ceph-mon[94189]: pgmap v895: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 4.1 KiB/s wr, 6 op/s 2023-03-09T15:10:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:10:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 713.135296 secs 2023-03-09T15:10:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:10:02 smithi038 ceph-mon[92422]: pgmap v895: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 4.1 KiB/s wr, 6 op/s 2023-03-09T15:10:02.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:10:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 713.135296 secs 2023-03-09T15:10:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1175.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:10:04.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:10:04 smithi122 ceph-mon[93869]: pgmap v896: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 60 KiB/s wr, 23 op/s 2023-03-09T15:10:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:10:04 smithi181 ceph-mon[94189]: pgmap v896: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 60 KiB/s wr, 23 op/s 2023-03-09T15:10:04.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:10:04 smithi038 ceph-mon[92422]: pgmap v896: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 60 KiB/s wr, 23 op/s 2023-03-09T15:10:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:10:05 smithi122 ceph-mon[93869]: osdmap e270: 12 total, 12 up, 12 in 2023-03-09T15:10:05.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:10:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 715.948057 secs 2023-03-09T15:10:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:10:05 smithi181 ceph-mon[94189]: osdmap e270: 12 total, 12 up, 12 in 2023-03-09T15:10:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:10:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 715.948057 secs 2023-03-09T15:10:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:10:05 smithi038 ceph-mon[92422]: osdmap e270: 12 total, 12 up, 12 in 2023-03-09T15:10:05.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:10:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 715.948057 secs 2023-03-09T15:10:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:10:06 smithi122 ceph-mon[93869]: pgmap v898: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 72 KiB/s wr, 25 op/s 2023-03-09T15:10:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:10:06 smithi181 ceph-mon[94189]: pgmap v898: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 72 KiB/s wr, 25 op/s 2023-03-09T15:10:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:10:06 smithi038 ceph-mon[92422]: pgmap v898: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 72 KiB/s wr, 25 op/s 2023-03-09T15:10:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:10:07 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:10:07.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:10:07 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:10:07.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:10:07 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:10:07.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:10:07 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:10:07.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:10:07 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:10:07.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:10:07 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:10:07.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:10:07 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:10:07.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:10:07 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:10:07.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:10:07 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:10:07.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:10:07 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:10:07.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:10:07 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:10:07.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:10:07 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:10:07.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:10:07 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:10:07.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:10:07 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:10:07.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:10:07 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:10:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:10:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 718.135364 secs 2023-03-09T15:10:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:10:08 smithi122 ceph-mon[93869]: pgmap v899: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 69 KiB/s wr, 25 op/s 2023-03-09T15:10:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:10:08 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:10:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:10:08 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:10:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:10:08 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:10:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:10:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 718.135364 secs 2023-03-09T15:10:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:10:08 smithi181 ceph-mon[94189]: pgmap v899: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 69 KiB/s wr, 25 op/s 2023-03-09T15:10:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:10:08 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:10:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:10:08 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:10:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:10:08 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:10:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:10:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 718.135364 secs 2023-03-09T15:10:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:10:08 smithi038 ceph-mon[92422]: pgmap v899: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 69 KiB/s wr, 25 op/s 2023-03-09T15:10:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:10:08 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:10:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:10:08 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:10:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:10:08 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:10:08.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1180.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:10:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:10:10 smithi122 ceph-mon[93869]: pgmap v900: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 67 KiB/s wr, 26 op/s 2023-03-09T15:10:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:10:10 smithi122 ceph-mon[93869]: 2 slow requests, 2 included below; oldest blocked for > 720.948116 secs 2023-03-09T15:10:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:10:10 smithi122 ceph-mon[93869]: slow request 720.948115 seconds old, received at 2023-03-09T14:58:08.564667+0000: peer_request:mds.2:599 currently acquired locks 2023-03-09T15:10:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:10:10 smithi122 ceph-mon[93869]: slow request 720.937415 seconds old, received at 2023-03-09T14:58:08.575367+0000: peer_request(client.15375:6773.0 authpin) currently dispatched 2023-03-09T15:10:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:10:10 smithi181 ceph-mon[94189]: pgmap v900: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 67 KiB/s wr, 26 op/s 2023-03-09T15:10:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:10:10 smithi181 ceph-mon[94189]: 2 slow requests, 2 included below; oldest blocked for > 720.948116 secs 2023-03-09T15:10:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:10:10 smithi181 ceph-mon[94189]: slow request 720.948115 seconds old, received at 2023-03-09T14:58:08.564667+0000: peer_request:mds.2:599 currently acquired locks 2023-03-09T15:10:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:10:10 smithi181 ceph-mon[94189]: slow request 720.937415 seconds old, received at 2023-03-09T14:58:08.575367+0000: peer_request(client.15375:6773.0 authpin) currently dispatched 2023-03-09T15:10:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:10:10 smithi038 ceph-mon[92422]: pgmap v900: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 67 KiB/s wr, 26 op/s 2023-03-09T15:10:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:10:10 smithi038 ceph-mon[92422]: 2 slow requests, 2 included below; oldest blocked for > 720.948116 secs 2023-03-09T15:10:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:10:10 smithi038 ceph-mon[92422]: slow request 720.948115 seconds old, received at 2023-03-09T14:58:08.564667+0000: peer_request:mds.2:599 currently acquired locks 2023-03-09T15:10:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:10:10 smithi038 ceph-mon[92422]: slow request 720.937415 seconds old, received at 2023-03-09T14:58:08.575367+0000: peer_request(client.15375:6773.0 authpin) currently dispatched 2023-03-09T15:10:12.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:10:12 smithi122 ceph-mon[93869]: pgmap v901: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 67 KiB/s wr, 26 op/s 2023-03-09T15:10:12.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:10:12 smithi122 ceph-mon[93869]: 2 slow requests, 2 included below; oldest blocked for > 723.135503 secs 2023-03-09T15:10:12.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:10:12 smithi122 ceph-mon[93869]: slow request 723.135502 seconds old, received at 2023-03-09T14:58:08.560137+0000: client_request(mds.2:599 rename #0x10000000007/0000000100000001000000EB #0x60f/1000000062a caller_uid=0, caller_gid=0{}) currently acquired locks 2023-03-09T15:10:12.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:10:12 smithi122 ceph-mon[93869]: slow request 723.121122 seconds old, received at 2023-03-09T14:58:08.574517+0000: client_request(client.15375:6773 unlink #0x10000000007/0000000100000001000000EB 2023-03-09T14:58:08.569481+0000 caller_uid=26, caller_gid=26{26,}) currently requesting remote authpins 2023-03-09T15:10:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:10:12 smithi181 ceph-mon[94189]: pgmap v901: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 67 KiB/s wr, 26 op/s 2023-03-09T15:10:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:10:12 smithi181 ceph-mon[94189]: 2 slow requests, 2 included below; oldest blocked for > 723.135503 secs 2023-03-09T15:10:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:10:12 smithi181 ceph-mon[94189]: slow request 723.135502 seconds old, received at 2023-03-09T14:58:08.560137+0000: client_request(mds.2:599 rename #0x10000000007/0000000100000001000000EB #0x60f/1000000062a caller_uid=0, caller_gid=0{}) currently acquired locks 2023-03-09T15:10:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:10:12 smithi181 ceph-mon[94189]: slow request 723.121122 seconds old, received at 2023-03-09T14:58:08.574517+0000: client_request(client.15375:6773 unlink #0x10000000007/0000000100000001000000EB 2023-03-09T14:58:08.569481+0000 caller_uid=26, caller_gid=26{26,}) currently requesting remote authpins 2023-03-09T15:10:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:10:12 smithi038 ceph-mon[92422]: pgmap v901: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 67 KiB/s wr, 26 op/s 2023-03-09T15:10:12.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:10:12 smithi038 ceph-mon[92422]: 2 slow requests, 2 included below; oldest blocked for > 723.135503 secs 2023-03-09T15:10:12.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:10:12 smithi038 ceph-mon[92422]: slow request 723.135502 seconds old, received at 2023-03-09T14:58:08.560137+0000: client_request(mds.2:599 rename #0x10000000007/0000000100000001000000EB #0x60f/1000000062a caller_uid=0, caller_gid=0{}) currently acquired locks 2023-03-09T15:10:12.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:10:12 smithi038 ceph-mon[92422]: slow request 723.121122 seconds old, received at 2023-03-09T14:58:08.574517+0000: client_request(client.15375:6773 unlink #0x10000000007/0000000100000001000000EB 2023-03-09T14:58:08.569481+0000 caller_uid=26, caller_gid=26{26,}) currently requesting remote authpins 2023-03-09T15:10:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1185.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:10:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:10:14 smithi122 ceph-mon[93869]: osdmap e271: 12 total, 12 up, 12 in 2023-03-09T15:10:14.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:10:14 smithi122 ceph-mon[93869]: pgmap v903: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 448 B/s wr, 7 op/s 2023-03-09T15:10:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:10:14 smithi181 ceph-mon[94189]: osdmap e271: 12 total, 12 up, 12 in 2023-03-09T15:10:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:10:14 smithi181 ceph-mon[94189]: pgmap v903: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 448 B/s wr, 7 op/s 2023-03-09T15:10:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:10:14 smithi038 ceph-mon[92422]: osdmap e271: 12 total, 12 up, 12 in 2023-03-09T15:10:14.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:10:14 smithi038 ceph-mon[92422]: pgmap v903: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 448 B/s wr, 7 op/s 2023-03-09T15:10:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:10:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 725.948178 secs 2023-03-09T15:10:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:10:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 725.948178 secs 2023-03-09T15:10:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:10:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 725.948178 secs 2023-03-09T15:10:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:10:16 smithi122 ceph-mon[93869]: pgmap v904: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 409 B/s wr, 7 op/s 2023-03-09T15:10:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:10:16 smithi181 ceph-mon[94189]: pgmap v904: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 409 B/s wr, 7 op/s 2023-03-09T15:10:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:10:16 smithi038 ceph-mon[92422]: pgmap v904: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 409 B/s wr, 7 op/s 2023-03-09T15:10:17.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:10:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 728.135641 secs 2023-03-09T15:10:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:10:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 728.135641 secs 2023-03-09T15:10:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:10:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 728.135641 secs 2023-03-09T15:10:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:10:18 smithi122 ceph-mon[93869]: pgmap v905: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:10:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:10:18 smithi181 ceph-mon[94189]: pgmap v905: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:10:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:10:18 smithi038 ceph-mon[92422]: pgmap v905: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:10:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1190.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:10:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:10:20 smithi122 ceph-mon[93869]: pgmap v906: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:10:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:10:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 730.948239 secs 2023-03-09T15:10:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:10:20 smithi181 ceph-mon[94189]: pgmap v906: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:10:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:10:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 730.948239 secs 2023-03-09T15:10:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:10:20 smithi038 ceph-mon[92422]: pgmap v906: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:10:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:10:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 730.948239 secs 2023-03-09T15:10:22.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:10:22 smithi122 ceph-mon[93869]: pgmap v907: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:10:22.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:10:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 733.135794 secs 2023-03-09T15:10:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:10:22 smithi181 ceph-mon[94189]: pgmap v907: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:10:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:10:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 733.135794 secs 2023-03-09T15:10:22.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:10:22 smithi038 ceph-mon[92422]: pgmap v907: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:10:22.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:10:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 733.135794 secs 2023-03-09T15:10:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1195.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:10:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:10:24 smithi122 ceph-mon[93869]: pgmap v908: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-09T15:10:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:10:24 smithi181 ceph-mon[94189]: pgmap v908: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-09T15:10:24.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:10:24 smithi038 ceph-mon[92422]: pgmap v908: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-09T15:10:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:10:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 735.948321 secs 2023-03-09T15:10:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:10:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 735.948321 secs 2023-03-09T15:10:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:10:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 735.948321 secs 2023-03-09T15:10:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:10:26 smithi122 ceph-mon[93869]: pgmap v909: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T15:10:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:10:26 smithi181 ceph-mon[94189]: pgmap v909: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T15:10:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:10:26 smithi038 ceph-mon[92422]: pgmap v909: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T15:10:27.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:10:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 738.135931 secs 2023-03-09T15:10:27.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:10:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 738.135931 secs 2023-03-09T15:10:27.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:10:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 738.135931 secs 2023-03-09T15:10:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:10:28 smithi122 ceph-mon[93869]: pgmap v910: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 5.8 KiB/s wr, 7 op/s 2023-03-09T15:10:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:10:28 smithi181 ceph-mon[94189]: pgmap v910: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 5.8 KiB/s wr, 7 op/s 2023-03-09T15:10:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:10:28 smithi038 ceph-mon[92422]: pgmap v910: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 5.8 KiB/s wr, 7 op/s 2023-03-09T15:10:28.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1200.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:10:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:10:30 smithi122 ceph-mon[93869]: pgmap v911: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 5.8 KiB/s wr, 6 op/s 2023-03-09T15:10:30.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:10:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 740.948407 secs 2023-03-09T15:10:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:10:30 smithi181 ceph-mon[94189]: pgmap v911: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 5.8 KiB/s wr, 6 op/s 2023-03-09T15:10:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:10:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 740.948407 secs 2023-03-09T15:10:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:10:30 smithi038 ceph-mon[92422]: pgmap v911: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 5.8 KiB/s wr, 6 op/s 2023-03-09T15:10:30.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:10:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 740.948407 secs 2023-03-09T15:10:32.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:10:32 smithi122 ceph-mon[93869]: pgmap v912: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 5.8 KiB/s wr, 6 op/s 2023-03-09T15:10:32.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:10:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 743.136053 secs 2023-03-09T15:10:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:10:32 smithi181 ceph-mon[94189]: pgmap v912: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 5.8 KiB/s wr, 6 op/s 2023-03-09T15:10:32.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:10:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 743.136053 secs 2023-03-09T15:10:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:10:32 smithi038 ceph-mon[92422]: pgmap v912: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 5.8 KiB/s wr, 6 op/s 2023-03-09T15:10:32.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:10:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 743.136053 secs 2023-03-09T15:10:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1205.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:10:34.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:10:34 smithi122 ceph-mon[93869]: pgmap v913: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 22 KiB/s wr, 9 op/s 2023-03-09T15:10:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:10:34 smithi181 ceph-mon[94189]: pgmap v913: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 22 KiB/s wr, 9 op/s 2023-03-09T15:10:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:10:34 smithi038 ceph-mon[92422]: pgmap v913: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 22 KiB/s wr, 9 op/s 2023-03-09T15:10:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:10:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 745.948466 secs 2023-03-09T15:10:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:10:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 745.948466 secs 2023-03-09T15:10:35.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:10:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 745.948466 secs 2023-03-09T15:10:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:10:36 smithi122 ceph-mon[93869]: pgmap v914: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 22 KiB/s wr, 7 op/s 2023-03-09T15:10:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:10:36 smithi181 ceph-mon[94189]: pgmap v914: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 22 KiB/s wr, 7 op/s 2023-03-09T15:10:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:10:36 smithi038 ceph-mon[92422]: pgmap v914: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 22 KiB/s wr, 7 op/s 2023-03-09T15:10:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:10:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 748.136173 secs 2023-03-09T15:10:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:10:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 748.136173 secs 2023-03-09T15:10:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:10:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 748.136173 secs 2023-03-09T15:10:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:10:38 smithi122 ceph-mon[93869]: pgmap v915: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 22 KiB/s wr, 10 op/s 2023-03-09T15:10:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:10:38 smithi181 ceph-mon[94189]: pgmap v915: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 22 KiB/s wr, 10 op/s 2023-03-09T15:10:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:10:38 smithi038 ceph-mon[92422]: pgmap v915: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 22 KiB/s wr, 10 op/s 2023-03-09T15:10:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1210.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:10:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:10:40 smithi122 ceph-mon[93869]: pgmap v916: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T15:10:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:10:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 750.948526 secs 2023-03-09T15:10:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:10:40 smithi181 ceph-mon[94189]: pgmap v916: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T15:10:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:10:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 750.948526 secs 2023-03-09T15:10:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:10:40 smithi038 ceph-mon[92422]: pgmap v916: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T15:10:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:10:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 750.948526 secs 2023-03-09T15:10:42.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:10:42 smithi122 ceph-mon[93869]: pgmap v917: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T15:10:42.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:10:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 753.136297 secs 2023-03-09T15:10:42.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:10:42 smithi181 ceph-mon[94189]: pgmap v917: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T15:10:42.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:10:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 753.136297 secs 2023-03-09T15:10:42.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:10:42 smithi038 ceph-mon[92422]: pgmap v917: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T15:10:42.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:10:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 753.136297 secs 2023-03-09T15:10:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1215.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:10:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:10:44 smithi122 ceph-mon[93869]: pgmap v918: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 31 KiB/s wr, 9 op/s 2023-03-09T15:10:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:10:44 smithi181 ceph-mon[94189]: pgmap v918: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 31 KiB/s wr, 9 op/s 2023-03-09T15:10:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:10:44 smithi038 ceph-mon[92422]: pgmap v918: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 31 KiB/s wr, 9 op/s 2023-03-09T15:10:45.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:10:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 755.948588 secs 2023-03-09T15:10:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:10:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 755.948588 secs 2023-03-09T15:10:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:10:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 755.948588 secs 2023-03-09T15:10:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:10:46 smithi122 ceph-mon[93869]: pgmap v919: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 15 KiB/s wr, 5 op/s 2023-03-09T15:10:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:10:46 smithi181 ceph-mon[94189]: pgmap v919: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 15 KiB/s wr, 5 op/s 2023-03-09T15:10:46.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:10:46 smithi038 ceph-mon[92422]: pgmap v919: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 15 KiB/s wr, 5 op/s 2023-03-09T15:10:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:10:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 758.136368 secs 2023-03-09T15:10:47.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:10:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 758.136368 secs 2023-03-09T15:10:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:10:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 758.136368 secs 2023-03-09T15:10:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:10:48 smithi122 ceph-mon[93869]: pgmap v920: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 15 KiB/s wr, 7 op/s 2023-03-09T15:10:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:10:48 smithi181 ceph-mon[94189]: pgmap v920: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 15 KiB/s wr, 7 op/s 2023-03-09T15:10:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:10:48 smithi038 ceph-mon[92422]: pgmap v920: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 15 KiB/s wr, 7 op/s 2023-03-09T15:10:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1220.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:10:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:10:50 smithi122 ceph-mon[93869]: pgmap v921: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 KiB/s wr, 5 op/s 2023-03-09T15:10:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:10:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 760.948649 secs 2023-03-09T15:10:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:10:50 smithi181 ceph-mon[94189]: pgmap v921: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 KiB/s wr, 5 op/s 2023-03-09T15:10:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:10:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 760.948649 secs 2023-03-09T15:10:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:10:50 smithi038 ceph-mon[92422]: pgmap v921: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 KiB/s wr, 5 op/s 2023-03-09T15:10:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:10:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 760.948649 secs 2023-03-09T15:10:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:10:52 smithi122 ceph-mon[93869]: pgmap v922: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 15 KiB/s wr, 5 op/s 2023-03-09T15:10:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:10:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 763.136485 secs 2023-03-09T15:10:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:10:52 smithi181 ceph-mon[94189]: pgmap v922: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 15 KiB/s wr, 5 op/s 2023-03-09T15:10:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:10:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 763.136485 secs 2023-03-09T15:10:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:10:52 smithi038 ceph-mon[92422]: pgmap v922: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 15 KiB/s wr, 5 op/s 2023-03-09T15:10:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:10:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 763.136485 secs 2023-03-09T15:10:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1225.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:10:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:10:54 smithi122 ceph-mon[93869]: pgmap v923: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 15 KiB/s wr, 7 op/s 2023-03-09T15:10:54.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:10:54 smithi181 ceph-mon[94189]: pgmap v923: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 15 KiB/s wr, 7 op/s 2023-03-09T15:10:54.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:10:54 smithi038 ceph-mon[92422]: pgmap v923: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 15 KiB/s wr, 7 op/s 2023-03-09T15:10:55.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:10:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 765.948719 secs 2023-03-09T15:10:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:10:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 765.948719 secs 2023-03-09T15:10:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:10:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 765.948719 secs 2023-03-09T15:10:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:10:56 smithi122 ceph-mon[93869]: pgmap v924: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T15:10:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:10:56 smithi181 ceph-mon[94189]: pgmap v924: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T15:10:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:10:56 smithi038 ceph-mon[92422]: pgmap v924: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T15:10:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:10:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 768.136582 secs 2023-03-09T15:10:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:10:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 768.136582 secs 2023-03-09T15:10:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:10:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 768.136582 secs 2023-03-09T15:10:58.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:10:58 smithi122 ceph-mon[93869]: pgmap v925: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 3.5 KiB/s wr, 8 op/s 2023-03-09T15:10:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:10:58 smithi181 ceph-mon[94189]: pgmap v925: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 3.5 KiB/s wr, 8 op/s 2023-03-09T15:10:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:10:58 smithi038 ceph-mon[92422]: pgmap v925: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 3.5 KiB/s wr, 8 op/s 2023-03-09T15:10:58.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1230.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:11:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:11:00 smithi122 ceph-mon[93869]: pgmap v926: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 3.6 KiB/s wr, 6 op/s 2023-03-09T15:11:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:11:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 770.948778 secs 2023-03-09T15:11:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:11:00 smithi181 ceph-mon[94189]: pgmap v926: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 3.6 KiB/s wr, 6 op/s 2023-03-09T15:11:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:11:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 770.948778 secs 2023-03-09T15:11:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:11:00 smithi038 ceph-mon[92422]: pgmap v926: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 3.6 KiB/s wr, 6 op/s 2023-03-09T15:11:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:11:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 770.948778 secs 2023-03-09T15:11:02.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:11:02 smithi122 ceph-mon[93869]: pgmap v927: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 4.7 KiB/s wr, 6 op/s 2023-03-09T15:11:02.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:11:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 773.136682 secs 2023-03-09T15:11:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:11:02 smithi181 ceph-mon[94189]: pgmap v927: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 4.7 KiB/s wr, 6 op/s 2023-03-09T15:11:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:11:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 773.136682 secs 2023-03-09T15:11:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:11:02 smithi038 ceph-mon[92422]: pgmap v927: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 4.7 KiB/s wr, 6 op/s 2023-03-09T15:11:02.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:11:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 773.136682 secs 2023-03-09T15:11:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1235.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:11:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:11:04 smithi122 ceph-mon[93869]: pgmap v928: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 60 KiB/s wr, 23 op/s 2023-03-09T15:11:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:11:04 smithi181 ceph-mon[94189]: pgmap v928: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 60 KiB/s wr, 23 op/s 2023-03-09T15:11:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:11:04 smithi038 ceph-mon[92422]: pgmap v928: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 60 KiB/s wr, 23 op/s 2023-03-09T15:11:05.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:11:05 smithi122 ceph-mon[93869]: osdmap e272: 12 total, 12 up, 12 in 2023-03-09T15:11:05.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:11:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 775.948871 secs 2023-03-09T15:11:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:11:05 smithi181 ceph-mon[94189]: osdmap e272: 12 total, 12 up, 12 in 2023-03-09T15:11:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:11:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 775.948871 secs 2023-03-09T15:11:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:11:05 smithi038 ceph-mon[92422]: osdmap e272: 12 total, 12 up, 12 in 2023-03-09T15:11:05.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:11:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 775.948871 secs 2023-03-09T15:11:06.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:11:06 smithi122 ceph-mon[93869]: pgmap v930: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 72 KiB/s wr, 25 op/s 2023-03-09T15:11:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:11:06 smithi181 ceph-mon[94189]: pgmap v930: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 72 KiB/s wr, 25 op/s 2023-03-09T15:11:06.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:11:06 smithi038 ceph-mon[92422]: pgmap v930: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 72 KiB/s wr, 25 op/s 2023-03-09T15:11:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:11:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 778.136803 secs 2023-03-09T15:11:07.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:11:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 778.136803 secs 2023-03-09T15:11:07.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:11:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 778.136803 secs 2023-03-09T15:11:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:11:08 smithi122 ceph-mon[93869]: pgmap v931: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 68 KiB/s wr, 25 op/s 2023-03-09T15:11:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:11:08 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:11:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:11:08 smithi181 ceph-mon[94189]: pgmap v931: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 68 KiB/s wr, 25 op/s 2023-03-09T15:11:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:11:08 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:11:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:11:08 smithi038 ceph-mon[92422]: pgmap v931: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 68 KiB/s wr, 25 op/s 2023-03-09T15:11:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:11:08 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:11:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1240.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:11:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:11:09 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:11:09.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:11:09 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:11:09.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:11:09 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:11:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:11:09 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:11:09.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:11:09 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:11:09.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:11:09 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:11:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:11:09 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:11:09.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:11:09 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:11:09.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:11:09 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:11:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:11:10 smithi038 ceph-mon[92422]: pgmap v932: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 68 KiB/s wr, 25 op/s 2023-03-09T15:11:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:11:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 780.948932 secs 2023-03-09T15:11:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:11:10 smithi122 ceph-mon[93869]: pgmap v932: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 68 KiB/s wr, 25 op/s 2023-03-09T15:11:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:11:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 780.948932 secs 2023-03-09T15:11:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:11:10 smithi181 ceph-mon[94189]: pgmap v932: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 68 KiB/s wr, 25 op/s 2023-03-09T15:11:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:11:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 780.948932 secs 2023-03-09T15:11:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:11:12 smithi122 ceph-mon[93869]: pgmap v933: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 67 KiB/s wr, 24 op/s 2023-03-09T15:11:12.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:11:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 783.136949 secs 2023-03-09T15:11:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:11:12 smithi181 ceph-mon[94189]: pgmap v933: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 67 KiB/s wr, 24 op/s 2023-03-09T15:11:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:11:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 783.136949 secs 2023-03-09T15:11:12.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:11:12 smithi038 ceph-mon[92422]: pgmap v933: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 67 KiB/s wr, 24 op/s 2023-03-09T15:11:12.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:11:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 783.136949 secs 2023-03-09T15:11:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1245.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:11:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:11:14 smithi122 ceph-mon[93869]: osdmap e273: 12 total, 12 up, 12 in 2023-03-09T15:11:14.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:11:14 smithi122 ceph-mon[93869]: pgmap v935: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 448 B/s wr, 7 op/s 2023-03-09T15:11:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:11:14 smithi181 ceph-mon[94189]: osdmap e273: 12 total, 12 up, 12 in 2023-03-09T15:11:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:11:14 smithi181 ceph-mon[94189]: pgmap v935: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 448 B/s wr, 7 op/s 2023-03-09T15:11:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:11:14 smithi038 ceph-mon[92422]: osdmap e273: 12 total, 12 up, 12 in 2023-03-09T15:11:14.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:11:14 smithi038 ceph-mon[92422]: pgmap v935: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 448 B/s wr, 7 op/s 2023-03-09T15:11:15.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:11:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 785.948984 secs 2023-03-09T15:11:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:11:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 785.948984 secs 2023-03-09T15:11:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:11:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 785.948984 secs 2023-03-09T15:11:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:11:16 smithi122 ceph-mon[93869]: pgmap v936: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 409 B/s wr, 7 op/s 2023-03-09T15:11:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:11:16 smithi181 ceph-mon[94189]: pgmap v936: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 409 B/s wr, 7 op/s 2023-03-09T15:11:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:11:16 smithi038 ceph-mon[92422]: pgmap v936: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 409 B/s wr, 7 op/s 2023-03-09T15:11:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:11:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 788.137072 secs 2023-03-09T15:11:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:11:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 788.137072 secs 2023-03-09T15:11:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:11:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 788.137072 secs 2023-03-09T15:11:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:11:18 smithi122 ceph-mon[93869]: pgmap v937: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:11:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:11:18 smithi181 ceph-mon[94189]: pgmap v937: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:11:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:11:18 smithi038 ceph-mon[92422]: pgmap v937: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:11:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1250.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:11:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:11:20 smithi122 ceph-mon[93869]: pgmap v938: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:11:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:11:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 790.949044 secs 2023-03-09T15:11:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:11:20 smithi181 ceph-mon[94189]: pgmap v938: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:11:20.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:11:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 790.949044 secs 2023-03-09T15:11:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:11:20 smithi038 ceph-mon[92422]: pgmap v938: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:11:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:11:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 790.949044 secs 2023-03-09T15:11:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:11:22 smithi122 ceph-mon[93869]: pgmap v939: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:11:22.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:11:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 793.137162 secs 2023-03-09T15:11:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:11:22 smithi181 ceph-mon[94189]: pgmap v939: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:11:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:11:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 793.137162 secs 2023-03-09T15:11:22.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:11:22 smithi038 ceph-mon[92422]: pgmap v939: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:11:22.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:11:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 793.137162 secs 2023-03-09T15:11:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1255.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:11:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:11:24 smithi122 ceph-mon[93869]: pgmap v940: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-09T15:11:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:11:24 smithi181 ceph-mon[94189]: pgmap v940: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-09T15:11:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:11:24 smithi038 ceph-mon[92422]: pgmap v940: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-09T15:11:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:11:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 795.949103 secs 2023-03-09T15:11:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:11:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 795.949103 secs 2023-03-09T15:11:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:11:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 795.949103 secs 2023-03-09T15:11:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:11:26 smithi122 ceph-mon[93869]: pgmap v941: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T15:11:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:11:26 smithi181 ceph-mon[94189]: pgmap v941: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T15:11:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:11:26 smithi038 ceph-mon[92422]: pgmap v941: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T15:11:27.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:11:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 798.137273 secs 2023-03-09T15:11:27.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:11:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 798.137273 secs 2023-03-09T15:11:27.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:11:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 798.137273 secs 2023-03-09T15:11:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:11:28 smithi122 ceph-mon[93869]: pgmap v942: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 6.2 KiB/s wr, 7 op/s 2023-03-09T15:11:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:11:28 smithi181 ceph-mon[94189]: pgmap v942: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 6.2 KiB/s wr, 7 op/s 2023-03-09T15:11:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:11:28 smithi038 ceph-mon[92422]: pgmap v942: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 6.2 KiB/s wr, 7 op/s 2023-03-09T15:11:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1260.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:11:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:11:30 smithi122 ceph-mon[93869]: pgmap v943: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 15 KiB/s wr, 6 op/s 2023-03-09T15:11:30.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:11:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 800.949164 secs 2023-03-09T15:11:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:11:30 smithi181 ceph-mon[94189]: pgmap v943: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 15 KiB/s wr, 6 op/s 2023-03-09T15:11:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:11:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 800.949164 secs 2023-03-09T15:11:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:11:30 smithi038 ceph-mon[92422]: pgmap v943: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 15 KiB/s wr, 6 op/s 2023-03-09T15:11:30.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:11:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 800.949164 secs 2023-03-09T15:11:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:11:32 smithi122 ceph-mon[93869]: pgmap v944: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 15 KiB/s wr, 6 op/s 2023-03-09T15:11:32.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:11:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 803.137394 secs 2023-03-09T15:11:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:11:32 smithi181 ceph-mon[94189]: pgmap v944: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 15 KiB/s wr, 6 op/s 2023-03-09T15:11:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:11:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 803.137394 secs 2023-03-09T15:11:32.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:11:32 smithi038 ceph-mon[92422]: pgmap v944: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 15 KiB/s wr, 6 op/s 2023-03-09T15:11:32.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:11:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 803.137394 secs 2023-03-09T15:11:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1265.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:11:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:11:34 smithi122 ceph-mon[93869]: pgmap v945: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 31 KiB/s wr, 10 op/s 2023-03-09T15:11:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:11:34 smithi181 ceph-mon[94189]: pgmap v945: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 31 KiB/s wr, 10 op/s 2023-03-09T15:11:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:11:34 smithi038 ceph-mon[92422]: pgmap v945: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 31 KiB/s wr, 10 op/s 2023-03-09T15:11:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:11:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 805.949226 secs 2023-03-09T15:11:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:11:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 805.949226 secs 2023-03-09T15:11:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:11:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 805.949226 secs 2023-03-09T15:11:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:11:36 smithi122 ceph-mon[93869]: pgmap v946: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 31 KiB/s wr, 7 op/s 2023-03-09T15:11:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:11:36 smithi181 ceph-mon[94189]: pgmap v946: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 31 KiB/s wr, 7 op/s 2023-03-09T15:11:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:11:36 smithi038 ceph-mon[92422]: pgmap v946: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 31 KiB/s wr, 7 op/s 2023-03-09T15:11:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:11:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 808.137501 secs 2023-03-09T15:11:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:11:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 808.137501 secs 2023-03-09T15:11:37.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:11:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 808.137501 secs 2023-03-09T15:11:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:11:38 smithi181 ceph-mon[94189]: pgmap v947: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 31 KiB/s wr, 10 op/s 2023-03-09T15:11:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:11:38 smithi038 ceph-mon[92422]: pgmap v947: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 31 KiB/s wr, 10 op/s 2023-03-09T15:11:38.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:11:38 smithi122 ceph-mon[93869]: pgmap v947: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 31 KiB/s wr, 10 op/s 2023-03-09T15:11:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1270.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:11:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:11:40 smithi122 ceph-mon[93869]: pgmap v948: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 7 op/s 2023-03-09T15:11:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:11:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 810.949287 secs 2023-03-09T15:11:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:11:40 smithi181 ceph-mon[94189]: pgmap v948: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 7 op/s 2023-03-09T15:11:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:11:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 810.949287 secs 2023-03-09T15:11:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:11:40 smithi038 ceph-mon[92422]: pgmap v948: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 7 op/s 2023-03-09T15:11:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:11:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 810.949287 secs 2023-03-09T15:11:42.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:11:42 smithi122 ceph-mon[93869]: pgmap v949: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 7 op/s 2023-03-09T15:11:42.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:11:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 813.137622 secs 2023-03-09T15:11:42.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:11:42 smithi181 ceph-mon[94189]: pgmap v949: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 7 op/s 2023-03-09T15:11:42.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:11:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 813.137622 secs 2023-03-09T15:11:42.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:11:42 smithi038 ceph-mon[92422]: pgmap v949: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 7 op/s 2023-03-09T15:11:42.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:11:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 813.137622 secs 2023-03-09T15:11:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1275.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:11:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:11:44 smithi122 ceph-mon[93869]: pgmap v950: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 24 KiB/s wr, 9 op/s 2023-03-09T15:11:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:11:44 smithi181 ceph-mon[94189]: pgmap v950: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 24 KiB/s wr, 9 op/s 2023-03-09T15:11:44.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:11:44 smithi038 ceph-mon[92422]: pgmap v950: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 24 KiB/s wr, 9 op/s 2023-03-09T15:11:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:11:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 815.949346 secs 2023-03-09T15:11:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:11:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 815.949346 secs 2023-03-09T15:11:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:11:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 815.949346 secs 2023-03-09T15:11:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:11:46 smithi122 ceph-mon[93869]: pgmap v951: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 7.1 KiB/s wr, 5 op/s 2023-03-09T15:11:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:11:46 smithi181 ceph-mon[94189]: pgmap v951: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 7.1 KiB/s wr, 5 op/s 2023-03-09T15:11:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:11:46 smithi038 ceph-mon[92422]: pgmap v951: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 7.1 KiB/s wr, 5 op/s 2023-03-09T15:11:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:11:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 818.137723 secs 2023-03-09T15:11:47.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:11:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 818.137723 secs 2023-03-09T15:11:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:11:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 818.137723 secs 2023-03-09T15:11:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:11:48 smithi038 ceph-mon[92422]: pgmap v952: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.1 KiB/s wr, 7 op/s 2023-03-09T15:11:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:11:48 smithi122 ceph-mon[93869]: pgmap v952: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.1 KiB/s wr, 7 op/s 2023-03-09T15:11:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:11:48 smithi181 ceph-mon[94189]: pgmap v952: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.1 KiB/s wr, 7 op/s 2023-03-09T15:11:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1280.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:11:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:11:50 smithi122 ceph-mon[93869]: pgmap v953: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 6.9 KiB/s wr, 5 op/s 2023-03-09T15:11:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:11:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 820.949451 secs 2023-03-09T15:11:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:11:50 smithi181 ceph-mon[94189]: pgmap v953: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 6.9 KiB/s wr, 5 op/s 2023-03-09T15:11:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:11:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 820.949451 secs 2023-03-09T15:11:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:11:50 smithi038 ceph-mon[92422]: pgmap v953: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 6.9 KiB/s wr, 5 op/s 2023-03-09T15:11:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:11:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 820.949451 secs 2023-03-09T15:11:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:11:52 smithi122 ceph-mon[93869]: pgmap v954: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 6.9 KiB/s wr, 5 op/s 2023-03-09T15:11:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:11:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 823.137846 secs 2023-03-09T15:11:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:11:52 smithi181 ceph-mon[94189]: pgmap v954: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 6.9 KiB/s wr, 5 op/s 2023-03-09T15:11:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:11:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 823.137846 secs 2023-03-09T15:11:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:11:52 smithi038 ceph-mon[92422]: pgmap v954: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 6.9 KiB/s wr, 5 op/s 2023-03-09T15:11:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:11:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 823.137846 secs 2023-03-09T15:11:53.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1285.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:11:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:11:54 smithi122 ceph-mon[93869]: pgmap v955: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 6.9 KiB/s wr, 7 op/s 2023-03-09T15:11:54.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:11:54 smithi181 ceph-mon[94189]: pgmap v955: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 6.9 KiB/s wr, 7 op/s 2023-03-09T15:11:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:11:54 smithi038 ceph-mon[92422]: pgmap v955: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 6.9 KiB/s wr, 7 op/s 2023-03-09T15:11:55.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:11:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 825.949512 secs 2023-03-09T15:11:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:11:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 825.949512 secs 2023-03-09T15:11:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:11:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 825.949512 secs 2023-03-09T15:11:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:11:56 smithi122 ceph-mon[93869]: pgmap v956: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T15:11:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:11:56 smithi181 ceph-mon[94189]: pgmap v956: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T15:11:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:11:56 smithi038 ceph-mon[92422]: pgmap v956: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T15:11:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:11:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 828.137970 secs 2023-03-09T15:11:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:11:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 828.137970 secs 2023-03-09T15:11:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:11:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 828.137970 secs 2023-03-09T15:11:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:11:58 smithi181 ceph-mon[94189]: pgmap v957: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 4.1 KiB/s wr, 7 op/s 2023-03-09T15:11:58.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:11:58 smithi038 ceph-mon[92422]: pgmap v957: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 4.1 KiB/s wr, 7 op/s 2023-03-09T15:11:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:11:58 smithi122 ceph-mon[93869]: pgmap v957: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 4.1 KiB/s wr, 7 op/s 2023-03-09T15:11:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1290.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:12:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:12:00 smithi122 ceph-mon[93869]: pgmap v958: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4.2 KiB/s wr, 5 op/s 2023-03-09T15:12:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:12:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 830.949615 secs 2023-03-09T15:12:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:12:00 smithi181 ceph-mon[94189]: pgmap v958: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4.2 KiB/s wr, 5 op/s 2023-03-09T15:12:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:12:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 830.949615 secs 2023-03-09T15:12:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:12:00 smithi038 ceph-mon[92422]: pgmap v958: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4.2 KiB/s wr, 5 op/s 2023-03-09T15:12:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:12:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 830.949615 secs 2023-03-09T15:12:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:12:02 smithi122 ceph-mon[93869]: pgmap v959: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4.2 KiB/s wr, 5 op/s 2023-03-09T15:12:02.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:12:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 833.138069 secs 2023-03-09T15:12:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:12:02 smithi181 ceph-mon[94189]: pgmap v959: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4.2 KiB/s wr, 5 op/s 2023-03-09T15:12:02.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:12:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 833.138069 secs 2023-03-09T15:12:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:12:02 smithi038 ceph-mon[92422]: pgmap v959: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4.2 KiB/s wr, 5 op/s 2023-03-09T15:12:02.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:12:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 833.138069 secs 2023-03-09T15:12:03.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1295.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:12:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:12:04 smithi122 ceph-mon[93869]: pgmap v960: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 49 KiB/s wr, 23 op/s 2023-03-09T15:12:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:12:04 smithi181 ceph-mon[94189]: pgmap v960: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 49 KiB/s wr, 23 op/s 2023-03-09T15:12:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:12:04 smithi038 ceph-mon[92422]: pgmap v960: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 49 KiB/s wr, 23 op/s 2023-03-09T15:12:05.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:12:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 835.949674 secs 2023-03-09T15:12:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:12:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 835.949674 secs 2023-03-09T15:12:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:12:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 835.949674 secs 2023-03-09T15:12:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:12:06 smithi122 ceph-mon[93869]: pgmap v961: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 49 KiB/s wr, 21 op/s 2023-03-09T15:12:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:12:06 smithi181 ceph-mon[94189]: pgmap v961: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 49 KiB/s wr, 21 op/s 2023-03-09T15:12:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:12:06 smithi038 ceph-mon[92422]: pgmap v961: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 49 KiB/s wr, 21 op/s 2023-03-09T15:12:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:12:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 838.138191 secs 2023-03-09T15:12:07.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:12:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 838.138191 secs 2023-03-09T15:12:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:12:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 838.138191 secs 2023-03-09T15:12:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:12:08 smithi038 ceph-mon[92422]: pgmap v962: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 51 KiB/s wr, 23 op/s 2023-03-09T15:12:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:12:08 smithi122 ceph-mon[93869]: pgmap v962: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 51 KiB/s wr, 23 op/s 2023-03-09T15:12:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:12:08 smithi181 ceph-mon[94189]: pgmap v962: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 51 KiB/s wr, 23 op/s 2023-03-09T15:12:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1300.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:12:09.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:12:09 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:12:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:12:09 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:12:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:12:09 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:12:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:12:10 smithi038 ceph-mon[92422]: osdmap e274: 12 total, 12 up, 12 in 2023-03-09T15:12:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:12:10 smithi038 ceph-mon[92422]: pgmap v964: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 68 KiB/s wr, 25 op/s 2023-03-09T15:12:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:12:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 840.949736 secs 2023-03-09T15:12:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:12:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:12:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:12:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:12:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:12:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:12:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:12:10 smithi122 ceph-mon[93869]: osdmap e274: 12 total, 12 up, 12 in 2023-03-09T15:12:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:12:10 smithi122 ceph-mon[93869]: pgmap v964: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 68 KiB/s wr, 25 op/s 2023-03-09T15:12:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:12:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 840.949736 secs 2023-03-09T15:12:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:12:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:12:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:12:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:12:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:12:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:12:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:12:10 smithi181 ceph-mon[94189]: osdmap e274: 12 total, 12 up, 12 in 2023-03-09T15:12:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:12:10 smithi181 ceph-mon[94189]: pgmap v964: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 68 KiB/s wr, 25 op/s 2023-03-09T15:12:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:12:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 840.949736 secs 2023-03-09T15:12:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:12:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:12:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:12:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:12:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:12:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:12:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:12:12 smithi122 ceph-mon[93869]: pgmap v965: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 68 KiB/s wr, 25 op/s 2023-03-09T15:12:12.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:12:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 843.138272 secs 2023-03-09T15:12:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:12:12 smithi181 ceph-mon[94189]: pgmap v965: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 68 KiB/s wr, 25 op/s 2023-03-09T15:12:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:12:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 843.138272 secs 2023-03-09T15:12:12.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:12:12 smithi038 ceph-mon[92422]: pgmap v965: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 68 KiB/s wr, 25 op/s 2023-03-09T15:12:12.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:12:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 843.138272 secs 2023-03-09T15:12:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1305.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:12:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:12:14 smithi122 ceph-mon[93869]: pgmap v966: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T15:12:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:12:14 smithi181 ceph-mon[94189]: pgmap v966: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T15:12:14.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:12:14 smithi038 ceph-mon[92422]: pgmap v966: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T15:12:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:12:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 845.949795 secs 2023-03-09T15:12:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:12:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 845.949795 secs 2023-03-09T15:12:15.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:12:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 845.949795 secs 2023-03-09T15:12:16.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:12:16 smithi122 ceph-mon[93869]: pgmap v967: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T15:12:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:12:16 smithi181 ceph-mon[94189]: pgmap v967: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T15:12:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:12:16 smithi038 ceph-mon[92422]: pgmap v967: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T15:12:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:12:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 848.138361 secs 2023-03-09T15:12:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:12:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 848.138361 secs 2023-03-09T15:12:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:12:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 848.138361 secs 2023-03-09T15:12:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:12:18 smithi038 ceph-mon[92422]: pgmap v968: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 7 op/s 2023-03-09T15:12:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:12:18 smithi038 ceph-mon[92422]: osdmap e275: 12 total, 12 up, 12 in 2023-03-09T15:12:18.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:12:18 smithi122 ceph-mon[93869]: pgmap v968: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 7 op/s 2023-03-09T15:12:18.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:12:18 smithi122 ceph-mon[93869]: osdmap e275: 12 total, 12 up, 12 in 2023-03-09T15:12:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:12:18 smithi181 ceph-mon[94189]: pgmap v968: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 7 op/s 2023-03-09T15:12:18.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:12:18 smithi181 ceph-mon[94189]: osdmap e275: 12 total, 12 up, 12 in 2023-03-09T15:12:18.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1310.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:12:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:12:20 smithi122 ceph-mon[93869]: pgmap v970: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:12:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:12:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 850.949857 secs 2023-03-09T15:12:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:12:20 smithi181 ceph-mon[94189]: pgmap v970: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:12:20.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:12:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 850.949857 secs 2023-03-09T15:12:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:12:20 smithi038 ceph-mon[92422]: pgmap v970: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:12:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:12:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 850.949857 secs 2023-03-09T15:12:22.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:12:22 smithi122 ceph-mon[93869]: pgmap v971: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:12:22.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:12:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 853.138464 secs 2023-03-09T15:12:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:12:22 smithi181 ceph-mon[94189]: pgmap v971: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:12:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:12:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 853.138464 secs 2023-03-09T15:12:22.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:12:22 smithi038 ceph-mon[92422]: pgmap v971: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:12:22.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:12:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 853.138464 secs 2023-03-09T15:12:23.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1315.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:12:24.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:12:24 smithi122 ceph-mon[93869]: pgmap v972: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:12:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:12:24 smithi181 ceph-mon[94189]: pgmap v972: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:12:24.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:12:24 smithi038 ceph-mon[92422]: pgmap v972: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:12:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:12:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 855.949952 secs 2023-03-09T15:12:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:12:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 855.949952 secs 2023-03-09T15:12:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:12:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 855.949952 secs 2023-03-09T15:12:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:12:26 smithi122 ceph-mon[93869]: pgmap v973: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:12:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:12:26 smithi181 ceph-mon[94189]: pgmap v973: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:12:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:12:26 smithi038 ceph-mon[92422]: pgmap v973: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:12:27.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:12:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 858.138592 secs 2023-03-09T15:12:27.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:12:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 858.138592 secs 2023-03-09T15:12:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:12:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 858.138592 secs 2023-03-09T15:12:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:12:28 smithi038 ceph-mon[92422]: pgmap v974: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 10 KiB/s wr, 7 op/s 2023-03-09T15:12:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:12:28 smithi122 ceph-mon[93869]: pgmap v974: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 10 KiB/s wr, 7 op/s 2023-03-09T15:12:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:12:28 smithi181 ceph-mon[94189]: pgmap v974: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 10 KiB/s wr, 7 op/s 2023-03-09T15:12:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1320.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:12:30.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:12:30 smithi122 ceph-mon[93869]: pgmap v975: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 9.0 KiB/s wr, 7 op/s 2023-03-09T15:12:30.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:12:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 860.949984 secs 2023-03-09T15:12:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:12:30 smithi181 ceph-mon[94189]: pgmap v975: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 9.0 KiB/s wr, 7 op/s 2023-03-09T15:12:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:12:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 860.949984 secs 2023-03-09T15:12:30.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:12:30 smithi038 ceph-mon[92422]: pgmap v975: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 9.0 KiB/s wr, 7 op/s 2023-03-09T15:12:30.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:12:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 860.949984 secs 2023-03-09T15:12:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:12:32 smithi122 ceph-mon[93869]: pgmap v976: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 8.4 KiB/s wr, 7 op/s 2023-03-09T15:12:32.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:12:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 863.138730 secs 2023-03-09T15:12:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:12:32 smithi181 ceph-mon[94189]: pgmap v976: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 8.4 KiB/s wr, 7 op/s 2023-03-09T15:12:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:12:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 863.138730 secs 2023-03-09T15:12:32.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:12:32 smithi038 ceph-mon[92422]: pgmap v976: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 8.4 KiB/s wr, 7 op/s 2023-03-09T15:12:32.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:12:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 863.138730 secs 2023-03-09T15:12:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1325.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:12:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:12:34 smithi122 ceph-mon[93869]: pgmap v977: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 22 KiB/s wr, 9 op/s 2023-03-09T15:12:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:12:34 smithi181 ceph-mon[94189]: pgmap v977: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 22 KiB/s wr, 9 op/s 2023-03-09T15:12:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:12:34 smithi038 ceph-mon[92422]: pgmap v977: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 22 KiB/s wr, 9 op/s 2023-03-09T15:12:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:12:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 865.950061 secs 2023-03-09T15:12:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:12:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 865.950061 secs 2023-03-09T15:12:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:12:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 865.950061 secs 2023-03-09T15:12:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:12:36 smithi122 ceph-mon[93869]: pgmap v978: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 22 KiB/s wr, 7 op/s 2023-03-09T15:12:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:12:36 smithi181 ceph-mon[94189]: pgmap v978: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 22 KiB/s wr, 7 op/s 2023-03-09T15:12:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:12:36 smithi038 ceph-mon[92422]: pgmap v978: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 22 KiB/s wr, 7 op/s 2023-03-09T15:12:37.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:12:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 868.138825 secs 2023-03-09T15:12:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:12:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 868.138825 secs 2023-03-09T15:12:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:12:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 868.138825 secs 2023-03-09T15:12:38.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:12:38 smithi122 ceph-mon[93869]: pgmap v979: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 23 KiB/s wr, 9 op/s 2023-03-09T15:12:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:12:38 smithi181 ceph-mon[94189]: pgmap v979: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 23 KiB/s wr, 9 op/s 2023-03-09T15:12:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:12:38 smithi038 ceph-mon[92422]: pgmap v979: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 23 KiB/s wr, 9 op/s 2023-03-09T15:12:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1330.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:12:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:12:40 smithi122 ceph-mon[93869]: pgmap v980: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 14 KiB/s wr, 6 op/s 2023-03-09T15:12:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:12:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 870.950122 secs 2023-03-09T15:12:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:12:40 smithi181 ceph-mon[94189]: pgmap v980: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 14 KiB/s wr, 6 op/s 2023-03-09T15:12:40.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:12:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 870.950122 secs 2023-03-09T15:12:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:12:40 smithi038 ceph-mon[92422]: pgmap v980: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 14 KiB/s wr, 6 op/s 2023-03-09T15:12:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:12:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 870.950122 secs 2023-03-09T15:12:42.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:12:42 smithi122 ceph-mon[93869]: pgmap v981: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 14 KiB/s wr, 6 op/s 2023-03-09T15:12:42.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:12:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 873.138944 secs 2023-03-09T15:12:42.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:12:42 smithi181 ceph-mon[94189]: pgmap v981: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 14 KiB/s wr, 6 op/s 2023-03-09T15:12:42.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:12:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 873.138944 secs 2023-03-09T15:12:42.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:12:42 smithi038 ceph-mon[92422]: pgmap v981: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 14 KiB/s wr, 6 op/s 2023-03-09T15:12:42.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:12:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 873.138944 secs 2023-03-09T15:12:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1335.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:12:44.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:12:44 smithi122 ceph-mon[93869]: pgmap v982: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 21 KiB/s wr, 8 op/s 2023-03-09T15:12:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:12:44 smithi181 ceph-mon[94189]: pgmap v982: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 21 KiB/s wr, 8 op/s 2023-03-09T15:12:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:12:44 smithi038 ceph-mon[92422]: pgmap v982: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 21 KiB/s wr, 8 op/s 2023-03-09T15:12:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:12:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 875.950181 secs 2023-03-09T15:12:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:12:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 875.950181 secs 2023-03-09T15:12:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:12:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 875.950181 secs 2023-03-09T15:12:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:12:46 smithi122 ceph-mon[93869]: pgmap v983: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.3 KiB/s wr, 5 op/s 2023-03-09T15:12:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:12:46 smithi181 ceph-mon[94189]: pgmap v983: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.3 KiB/s wr, 5 op/s 2023-03-09T15:12:46.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:12:46 smithi038 ceph-mon[92422]: pgmap v983: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.3 KiB/s wr, 5 op/s 2023-03-09T15:12:47.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:12:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 878.139058 secs 2023-03-09T15:12:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:12:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 878.139058 secs 2023-03-09T15:12:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:12:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 878.139058 secs 2023-03-09T15:12:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:12:48 smithi038 ceph-mon[92422]: pgmap v984: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.3 KiB/s wr, 7 op/s 2023-03-09T15:12:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:12:48 smithi122 ceph-mon[93869]: pgmap v984: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.3 KiB/s wr, 7 op/s 2023-03-09T15:12:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:12:48 smithi181 ceph-mon[94189]: pgmap v984: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.3 KiB/s wr, 7 op/s 2023-03-09T15:12:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1340.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:12:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:12:50 smithi122 ceph-mon[93869]: pgmap v985: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 7.2 KiB/s wr, 5 op/s 2023-03-09T15:12:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:12:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 880.950244 secs 2023-03-09T15:12:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:12:50 smithi181 ceph-mon[94189]: pgmap v985: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 7.2 KiB/s wr, 5 op/s 2023-03-09T15:12:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:12:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 880.950244 secs 2023-03-09T15:12:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:12:50 smithi038 ceph-mon[92422]: pgmap v985: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 7.2 KiB/s wr, 5 op/s 2023-03-09T15:12:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:12:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 880.950244 secs 2023-03-09T15:12:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:12:52 smithi122 ceph-mon[93869]: pgmap v986: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.2 KiB/s wr, 5 op/s 2023-03-09T15:12:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:12:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 883.139171 secs 2023-03-09T15:12:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:12:52 smithi181 ceph-mon[94189]: pgmap v986: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.2 KiB/s wr, 5 op/s 2023-03-09T15:12:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:12:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 883.139171 secs 2023-03-09T15:12:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:12:52 smithi038 ceph-mon[92422]: pgmap v986: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.2 KiB/s wr, 5 op/s 2023-03-09T15:12:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:12:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 883.139171 secs 2023-03-09T15:12:53.801 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1345.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:12:54.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:12:54 smithi122 ceph-mon[93869]: pgmap v987: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.2 KiB/s wr, 7 op/s 2023-03-09T15:12:54.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:12:54 smithi181 ceph-mon[94189]: pgmap v987: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.2 KiB/s wr, 7 op/s 2023-03-09T15:12:54.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:12:54 smithi038 ceph-mon[92422]: pgmap v987: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.2 KiB/s wr, 7 op/s 2023-03-09T15:12:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:12:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 885.950304 secs 2023-03-09T15:12:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:12:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 885.950304 secs 2023-03-09T15:12:55.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:12:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 885.950304 secs 2023-03-09T15:12:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:12:56 smithi122 ceph-mon[93869]: pgmap v988: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T15:12:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:12:56 smithi181 ceph-mon[94189]: pgmap v988: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T15:12:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:12:56 smithi038 ceph-mon[92422]: pgmap v988: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T15:12:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:12:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 888.139284 secs 2023-03-09T15:12:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:12:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 888.139284 secs 2023-03-09T15:12:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:12:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 888.139284 secs 2023-03-09T15:12:58.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:12:58 smithi038 ceph-mon[92422]: pgmap v989: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 3.2 KiB/s wr, 7 op/s 2023-03-09T15:12:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:12:58 smithi122 ceph-mon[93869]: pgmap v989: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 3.2 KiB/s wr, 7 op/s 2023-03-09T15:12:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:12:58 smithi181 ceph-mon[94189]: pgmap v989: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 3.2 KiB/s wr, 7 op/s 2023-03-09T15:12:58.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1350.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:13:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:13:00 smithi122 ceph-mon[93869]: pgmap v990: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4.3 KiB/s wr, 5 op/s 2023-03-09T15:13:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:13:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 890.950364 secs 2023-03-09T15:13:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:13:00 smithi181 ceph-mon[94189]: pgmap v990: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4.3 KiB/s wr, 5 op/s 2023-03-09T15:13:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:13:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 890.950364 secs 2023-03-09T15:13:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:13:00 smithi038 ceph-mon[92422]: pgmap v990: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4.3 KiB/s wr, 5 op/s 2023-03-09T15:13:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:13:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 890.950364 secs 2023-03-09T15:13:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:13:02 smithi122 ceph-mon[93869]: pgmap v991: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 4.4 KiB/s wr, 6 op/s 2023-03-09T15:13:02.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:13:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 893.139362 secs 2023-03-09T15:13:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:13:02 smithi181 ceph-mon[94189]: pgmap v991: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 4.4 KiB/s wr, 6 op/s 2023-03-09T15:13:02.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:13:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 893.139362 secs 2023-03-09T15:13:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:13:02 smithi038 ceph-mon[92422]: pgmap v991: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 4.4 KiB/s wr, 6 op/s 2023-03-09T15:13:02.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:13:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 893.139362 secs 2023-03-09T15:13:03.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1355.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:13:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:13:04 smithi122 ceph-mon[93869]: pgmap v992: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 50 KiB/s wr, 23 op/s 2023-03-09T15:13:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:13:04 smithi181 ceph-mon[94189]: pgmap v992: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 50 KiB/s wr, 23 op/s 2023-03-09T15:13:04.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:13:04 smithi038 ceph-mon[92422]: pgmap v992: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 50 KiB/s wr, 23 op/s 2023-03-09T15:13:05.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:13:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 895.950425 secs 2023-03-09T15:13:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:13:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 895.950425 secs 2023-03-09T15:13:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:13:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 895.950425 secs 2023-03-09T15:13:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:13:06 smithi122 ceph-mon[93869]: pgmap v993: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 50 KiB/s wr, 21 op/s 2023-03-09T15:13:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:13:06 smithi181 ceph-mon[94189]: pgmap v993: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 50 KiB/s wr, 21 op/s 2023-03-09T15:13:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:13:06 smithi038 ceph-mon[92422]: pgmap v993: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 50 KiB/s wr, 21 op/s 2023-03-09T15:13:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:13:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 898.139421 secs 2023-03-09T15:13:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:13:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 898.139421 secs 2023-03-09T15:13:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:13:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 898.139421 secs 2023-03-09T15:13:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:13:08 smithi038 ceph-mon[92422]: pgmap v994: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 51 KiB/s wr, 23 op/s 2023-03-09T15:13:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:13:08 smithi122 ceph-mon[93869]: pgmap v994: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 51 KiB/s wr, 23 op/s 2023-03-09T15:13:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:13:08 smithi181 ceph-mon[94189]: pgmap v994: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 51 KiB/s wr, 23 op/s 2023-03-09T15:13:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1360.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:13:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:13:10 smithi122 ceph-mon[93869]: osdmap e276: 12 total, 12 up, 12 in 2023-03-09T15:13:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:13:10 smithi122 ceph-mon[93869]: pgmap v996: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 68 KiB/s wr, 25 op/s 2023-03-09T15:13:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:13:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 900.950486 secs 2023-03-09T15:13:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:13:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:13:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:13:10 smithi181 ceph-mon[94189]: osdmap e276: 12 total, 12 up, 12 in 2023-03-09T15:13:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:13:10 smithi181 ceph-mon[94189]: pgmap v996: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 68 KiB/s wr, 25 op/s 2023-03-09T15:13:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:13:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 900.950486 secs 2023-03-09T15:13:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:13:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:13:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:13:10 smithi038 ceph-mon[92422]: osdmap e276: 12 total, 12 up, 12 in 2023-03-09T15:13:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:13:10 smithi038 ceph-mon[92422]: pgmap v996: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 68 KiB/s wr, 25 op/s 2023-03-09T15:13:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:13:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 900.950486 secs 2023-03-09T15:13:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:13:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:13:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:13:12 smithi122 ceph-mon[93869]: pgmap v997: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 68 KiB/s wr, 25 op/s 2023-03-09T15:13:12.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:13:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 903.139495 secs 2023-03-09T15:13:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:13:12 smithi181 ceph-mon[94189]: pgmap v997: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 68 KiB/s wr, 25 op/s 2023-03-09T15:13:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:13:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 903.139495 secs 2023-03-09T15:13:12.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:13:12 smithi038 ceph-mon[92422]: pgmap v997: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 68 KiB/s wr, 25 op/s 2023-03-09T15:13:12.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:13:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 903.139495 secs 2023-03-09T15:13:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1365.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:13:14.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:13:14 smithi038 ceph-mon[92422]: pgmap v998: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T15:13:14.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:13:14 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:13:14.753 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:13:14 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:13:14.754 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:13:14 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:13:14.754 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:13:14 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:13:14.755 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:13:14 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:13:14.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:13:14 smithi122 ceph-mon[93869]: pgmap v998: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T15:13:14.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:13:14 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:13:14.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:13:14 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:13:14.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:13:14 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:13:14.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:13:14 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:13:14.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:13:14 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:13:14.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:13:14 smithi181 ceph-mon[94189]: pgmap v998: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T15:13:14.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:13:14 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:13:14.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:13:14 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:13:14.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:13:14 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:13:14.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:13:14 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:13:14.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:13:14 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:13:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:13:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 905.950545 secs 2023-03-09T15:13:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:13:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 905.950545 secs 2023-03-09T15:13:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:13:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 905.950545 secs 2023-03-09T15:13:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:13:16 smithi122 ceph-mon[93869]: pgmap v999: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T15:13:16.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:13:16 smithi181 ceph-mon[94189]: pgmap v999: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T15:13:17.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:13:16 smithi038 ceph-mon[92422]: pgmap v999: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T15:13:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:13:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 908.139573 secs 2023-03-09T15:13:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:13:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 908.139573 secs 2023-03-09T15:13:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:13:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 908.139573 secs 2023-03-09T15:13:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1370.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:13:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:13:18 smithi122 ceph-mon[93869]: pgmap v1000: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 12 KiB/s wr, 7 op/s 2023-03-09T15:13:18.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:13:18 smithi122 ceph-mon[93869]: osdmap e277: 12 total, 12 up, 12 in 2023-03-09T15:13:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:13:18 smithi181 ceph-mon[94189]: pgmap v1000: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 12 KiB/s wr, 7 op/s 2023-03-09T15:13:18.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:13:18 smithi181 ceph-mon[94189]: osdmap e277: 12 total, 12 up, 12 in 2023-03-09T15:13:19.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:13:18 smithi038 ceph-mon[92422]: pgmap v1000: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 12 KiB/s wr, 7 op/s 2023-03-09T15:13:19.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:13:18 smithi038 ceph-mon[92422]: osdmap e277: 12 total, 12 up, 12 in 2023-03-09T15:13:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:13:20 smithi122 ceph-mon[93869]: pgmap v1002: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:13:20.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:13:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 910.950606 secs 2023-03-09T15:13:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:13:20 smithi181 ceph-mon[94189]: pgmap v1002: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:13:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:13:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 910.950606 secs 2023-03-09T15:13:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:13:20 smithi038 ceph-mon[92422]: pgmap v1002: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:13:21.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:13:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 910.950606 secs 2023-03-09T15:13:22.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:13:22 smithi122 ceph-mon[93869]: pgmap v1003: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:13:22.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:13:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 913.139712 secs 2023-03-09T15:13:22.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:13:22 smithi181 ceph-mon[94189]: pgmap v1003: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:13:22.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:13:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 913.139712 secs 2023-03-09T15:13:23.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:13:22 smithi038 ceph-mon[92422]: pgmap v1003: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:13:23.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:13:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 913.139712 secs 2023-03-09T15:13:23.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1375.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:13:24.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:13:24 smithi122 ceph-mon[93869]: pgmap v1004: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:13:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:13:24 smithi181 ceph-mon[94189]: pgmap v1004: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:13:24.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:13:24 smithi038 ceph-mon[92422]: pgmap v1004: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:13:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:13:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 915.950691 secs 2023-03-09T15:13:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:13:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 915.950691 secs 2023-03-09T15:13:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:13:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 915.950691 secs 2023-03-09T15:13:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:13:26 smithi122 ceph-mon[93869]: pgmap v1005: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:13:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:13:26 smithi181 ceph-mon[94189]: pgmap v1005: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:13:26.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:13:26 smithi038 ceph-mon[92422]: pgmap v1005: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:13:27.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:13:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 918.139790 secs 2023-03-09T15:13:27.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:13:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 918.139790 secs 2023-03-09T15:13:27.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:13:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 918.139790 secs 2023-03-09T15:13:28.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:13:28 smithi038 ceph-mon[92422]: pgmap v1006: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 7.8 KiB/s wr, 7 op/s 2023-03-09T15:13:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:13:28 smithi122 ceph-mon[93869]: pgmap v1006: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 7.8 KiB/s wr, 7 op/s 2023-03-09T15:13:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:13:28 smithi181 ceph-mon[94189]: pgmap v1006: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 7.8 KiB/s wr, 7 op/s 2023-03-09T15:13:28.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1380.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:13:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:13:30 smithi122 ceph-mon[93869]: pgmap v1007: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 6.9 KiB/s wr, 6 op/s 2023-03-09T15:13:30.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:13:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 920.950752 secs 2023-03-09T15:13:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:13:30 smithi181 ceph-mon[94189]: pgmap v1007: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 6.9 KiB/s wr, 6 op/s 2023-03-09T15:13:30.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:13:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 920.950752 secs 2023-03-09T15:13:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:13:30 smithi038 ceph-mon[92422]: pgmap v1007: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 6.9 KiB/s wr, 6 op/s 2023-03-09T15:13:30.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:13:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 920.950752 secs 2023-03-09T15:13:32.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:13:32 smithi122 ceph-mon[93869]: pgmap v1008: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 6.5 KiB/s wr, 6 op/s 2023-03-09T15:13:32.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:13:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 923.139908 secs 2023-03-09T15:13:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:13:32 smithi181 ceph-mon[94189]: pgmap v1008: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 6.5 KiB/s wr, 6 op/s 2023-03-09T15:13:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:13:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 923.139908 secs 2023-03-09T15:13:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:13:32 smithi038 ceph-mon[92422]: pgmap v1008: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 6.5 KiB/s wr, 6 op/s 2023-03-09T15:13:32.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:13:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 923.139908 secs 2023-03-09T15:13:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1385.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:13:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:13:34 smithi122 ceph-mon[93869]: pgmap v1009: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 29 KiB/s wr, 10 op/s 2023-03-09T15:13:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:13:34 smithi181 ceph-mon[94189]: pgmap v1009: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 29 KiB/s wr, 10 op/s 2023-03-09T15:13:34.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:13:34 smithi038 ceph-mon[92422]: pgmap v1009: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 29 KiB/s wr, 10 op/s 2023-03-09T15:13:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:13:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 925.950812 secs 2023-03-09T15:13:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:13:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 925.950812 secs 2023-03-09T15:13:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:13:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 925.950812 secs 2023-03-09T15:13:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:13:36 smithi122 ceph-mon[93869]: pgmap v1010: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 29 KiB/s wr, 8 op/s 2023-03-09T15:13:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:13:36 smithi181 ceph-mon[94189]: pgmap v1010: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 29 KiB/s wr, 8 op/s 2023-03-09T15:13:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:13:36 smithi038 ceph-mon[92422]: pgmap v1010: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 29 KiB/s wr, 8 op/s 2023-03-09T15:13:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:13:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 928.140001 secs 2023-03-09T15:13:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:13:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 928.140001 secs 2023-03-09T15:13:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:13:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 928.140001 secs 2023-03-09T15:13:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:13:38 smithi122 ceph-mon[93869]: pgmap v1011: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 29 KiB/s wr, 11 op/s 2023-03-09T15:13:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:13:38 smithi181 ceph-mon[94189]: pgmap v1011: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 29 KiB/s wr, 11 op/s 2023-03-09T15:13:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:13:38 smithi038 ceph-mon[92422]: pgmap v1011: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 29 KiB/s wr, 11 op/s 2023-03-09T15:13:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1390.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:13:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:13:40 smithi122 ceph-mon[93869]: pgmap v1012: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 23 KiB/s wr, 8 op/s 2023-03-09T15:13:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:13:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 930.950871 secs 2023-03-09T15:13:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:13:40 smithi181 ceph-mon[94189]: pgmap v1012: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 23 KiB/s wr, 8 op/s 2023-03-09T15:13:40.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:13:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 930.950871 secs 2023-03-09T15:13:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:13:40 smithi038 ceph-mon[92422]: pgmap v1012: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 23 KiB/s wr, 8 op/s 2023-03-09T15:13:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:13:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 930.950871 secs 2023-03-09T15:13:42.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:13:42 smithi122 ceph-mon[93869]: pgmap v1013: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 23 KiB/s wr, 8 op/s 2023-03-09T15:13:42.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:13:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 933.140132 secs 2023-03-09T15:13:42.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:13:42 smithi181 ceph-mon[94189]: pgmap v1013: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 23 KiB/s wr, 8 op/s 2023-03-09T15:13:42.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:13:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 933.140132 secs 2023-03-09T15:13:42.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:13:42 smithi038 ceph-mon[92422]: pgmap v1013: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 23 KiB/s wr, 8 op/s 2023-03-09T15:13:42.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:13:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 933.140132 secs 2023-03-09T15:13:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1395.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:13:44.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:13:44 smithi122 ceph-mon[93869]: pgmap v1014: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 30 KiB/s wr, 10 op/s 2023-03-09T15:13:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:13:44 smithi181 ceph-mon[94189]: pgmap v1014: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 30 KiB/s wr, 10 op/s 2023-03-09T15:13:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:13:44 smithi038 ceph-mon[92422]: pgmap v1014: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 30 KiB/s wr, 10 op/s 2023-03-09T15:13:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:13:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 935.950934 secs 2023-03-09T15:13:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:13:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 935.950934 secs 2023-03-09T15:13:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:13:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 935.950934 secs 2023-03-09T15:13:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:13:46 smithi122 ceph-mon[93869]: pgmap v1015: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 7.5 KiB/s wr, 5 op/s 2023-03-09T15:13:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:13:46 smithi181 ceph-mon[94189]: pgmap v1015: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 7.5 KiB/s wr, 5 op/s 2023-03-09T15:13:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:13:46 smithi038 ceph-mon[92422]: pgmap v1015: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 7.5 KiB/s wr, 5 op/s 2023-03-09T15:13:47.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:13:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 938.140238 secs 2023-03-09T15:13:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:13:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 938.140238 secs 2023-03-09T15:13:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:13:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 938.140238 secs 2023-03-09T15:13:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:13:48 smithi038 ceph-mon[92422]: pgmap v1016: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 7.5 KiB/s wr, 7 op/s 2023-03-09T15:13:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:13:48 smithi122 ceph-mon[93869]: pgmap v1016: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 7.5 KiB/s wr, 7 op/s 2023-03-09T15:13:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:13:48 smithi181 ceph-mon[94189]: pgmap v1016: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 7.5 KiB/s wr, 7 op/s 2023-03-09T15:13:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1400.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:13:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:13:50 smithi122 ceph-mon[93869]: pgmap v1017: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.3 KiB/s wr, 5 op/s 2023-03-09T15:13:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:13:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 940.950984 secs 2023-03-09T15:13:50.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:13:50 smithi181 ceph-mon[94189]: pgmap v1017: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.3 KiB/s wr, 5 op/s 2023-03-09T15:13:50.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:13:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 940.950984 secs 2023-03-09T15:13:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:13:50 smithi038 ceph-mon[92422]: pgmap v1017: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.3 KiB/s wr, 5 op/s 2023-03-09T15:13:50.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:13:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 940.950984 secs 2023-03-09T15:13:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:13:52 smithi122 ceph-mon[93869]: pgmap v1018: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.3 KiB/s wr, 5 op/s 2023-03-09T15:13:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:13:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 943.140315 secs 2023-03-09T15:13:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:13:52 smithi181 ceph-mon[94189]: pgmap v1018: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.3 KiB/s wr, 5 op/s 2023-03-09T15:13:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:13:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 943.140315 secs 2023-03-09T15:13:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:13:52 smithi038 ceph-mon[92422]: pgmap v1018: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.3 KiB/s wr, 5 op/s 2023-03-09T15:13:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:13:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 943.140315 secs 2023-03-09T15:13:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1405.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:13:54.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:13:54 smithi122 ceph-mon[93869]: pgmap v1019: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.3 KiB/s wr, 7 op/s 2023-03-09T15:13:54.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:13:54 smithi181 ceph-mon[94189]: pgmap v1019: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.3 KiB/s wr, 7 op/s 2023-03-09T15:13:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:13:54 smithi038 ceph-mon[92422]: pgmap v1019: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.3 KiB/s wr, 7 op/s 2023-03-09T15:13:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:13:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 945.951046 secs 2023-03-09T15:13:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:13:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 945.951046 secs 2023-03-09T15:13:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:13:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 945.951046 secs 2023-03-09T15:13:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:13:56 smithi122 ceph-mon[93869]: pgmap v1020: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T15:13:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:13:56 smithi181 ceph-mon[94189]: pgmap v1020: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T15:13:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:13:56 smithi038 ceph-mon[92422]: pgmap v1020: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T15:13:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:13:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 948.140426 secs 2023-03-09T15:13:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:13:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 948.140426 secs 2023-03-09T15:13:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:13:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 948.140426 secs 2023-03-09T15:13:58.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:13:58 smithi122 ceph-mon[93869]: pgmap v1021: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-03-09T15:13:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:13:58 smithi181 ceph-mon[94189]: pgmap v1021: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-03-09T15:13:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:13:58 smithi038 ceph-mon[92422]: pgmap v1021: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-03-09T15:13:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1410.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:14:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:14:00 smithi122 ceph-mon[93869]: pgmap v1022: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T15:14:00.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:14:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 950.951122 secs 2023-03-09T15:14:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:14:00 smithi181 ceph-mon[94189]: pgmap v1022: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T15:14:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:14:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 950.951122 secs 2023-03-09T15:14:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:14:00 smithi038 ceph-mon[92422]: pgmap v1022: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T15:14:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:14:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 950.951122 secs 2023-03-09T15:14:02.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:14:02 smithi122 ceph-mon[93869]: pgmap v1023: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-03-09T15:14:02.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:14:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 953.140540 secs 2023-03-09T15:14:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:14:02 smithi181 ceph-mon[94189]: pgmap v1023: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-03-09T15:14:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:14:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 953.140540 secs 2023-03-09T15:14:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:14:02 smithi038 ceph-mon[92422]: pgmap v1023: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-03-09T15:14:02.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:14:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 953.140540 secs 2023-03-09T15:14:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1415.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:14:04.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:14:04 smithi122 ceph-mon[93869]: pgmap v1024: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 21 op/s 2023-03-09T15:14:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:14:04 smithi181 ceph-mon[94189]: pgmap v1024: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 21 op/s 2023-03-09T15:14:04.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:14:04 smithi038 ceph-mon[92422]: pgmap v1024: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 21 op/s 2023-03-09T15:14:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:14:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 955.951183 secs 2023-03-09T15:14:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:14:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 955.951183 secs 2023-03-09T15:14:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:14:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 955.951183 secs 2023-03-09T15:14:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:14:06 smithi122 ceph-mon[93869]: pgmap v1025: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-03-09T15:14:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:14:06 smithi181 ceph-mon[94189]: pgmap v1025: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-03-09T15:14:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:14:06 smithi038 ceph-mon[92422]: pgmap v1025: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-03-09T15:14:07.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:14:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 958.140674 secs 2023-03-09T15:14:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:14:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 958.140674 secs 2023-03-09T15:14:07.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:14:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 958.140674 secs 2023-03-09T15:14:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:14:08 smithi122 ceph-mon[93869]: pgmap v1026: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 21 op/s 2023-03-09T15:14:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:14:08 smithi181 ceph-mon[94189]: pgmap v1026: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 21 op/s 2023-03-09T15:14:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:14:08 smithi038 ceph-mon[92422]: pgmap v1026: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 21 op/s 2023-03-09T15:14:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1420.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:14:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:14:10 smithi122 ceph-mon[93869]: osdmap e278: 12 total, 12 up, 12 in 2023-03-09T15:14:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:14:10 smithi122 ceph-mon[93869]: pgmap v1028: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 24 op/s 2023-03-09T15:14:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:14:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 960.951242 secs 2023-03-09T15:14:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:14:10 smithi181 ceph-mon[94189]: osdmap e278: 12 total, 12 up, 12 in 2023-03-09T15:14:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:14:10 smithi181 ceph-mon[94189]: pgmap v1028: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 24 op/s 2023-03-09T15:14:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:14:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 960.951242 secs 2023-03-09T15:14:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:14:10 smithi038 ceph-mon[92422]: osdmap e278: 12 total, 12 up, 12 in 2023-03-09T15:14:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:14:10 smithi038 ceph-mon[92422]: pgmap v1028: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 24 op/s 2023-03-09T15:14:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:14:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 960.951242 secs 2023-03-09T15:14:12.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:14:12 smithi122 ceph-mon[93869]: pgmap v1029: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 24 op/s 2023-03-09T15:14:12.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:14:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 963.140800 secs 2023-03-09T15:14:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:14:12 smithi181 ceph-mon[94189]: pgmap v1029: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 24 op/s 2023-03-09T15:14:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:14:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 963.140800 secs 2023-03-09T15:14:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:14:12 smithi038 ceph-mon[92422]: pgmap v1029: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 24 op/s 2023-03-09T15:14:12.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:14:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 963.140800 secs 2023-03-09T15:14:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1425.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:14:14.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:14:14 smithi122 ceph-mon[93869]: pgmap v1030: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 7 op/s 2023-03-09T15:14:14.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:14:14 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:14:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:14:14 smithi181 ceph-mon[94189]: pgmap v1030: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 7 op/s 2023-03-09T15:14:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:14:14 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:14:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:14:14 smithi038 ceph-mon[92422]: pgmap v1030: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 7 op/s 2023-03-09T15:14:14.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:14:14 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:14:15.453 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:14:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 965.951315 secs 2023-03-09T15:14:15.457 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:14:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 965.951315 secs 2023-03-09T15:14:15.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:14:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 965.951315 secs 2023-03-09T15:14:16.429 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:14:16 smithi122 ceph-mon[93869]: pgmap v1031: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 7 op/s 2023-03-09T15:14:16.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:14:16 smithi181 ceph-mon[94189]: pgmap v1031: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 7 op/s 2023-03-09T15:14:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:14:16 smithi038 ceph-mon[92422]: pgmap v1031: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 7 op/s 2023-03-09T15:14:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:14:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 968.140887 secs 2023-03-09T15:14:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:14:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 968.140887 secs 2023-03-09T15:14:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:14:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 968.140887 secs 2023-03-09T15:14:18.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:14:18 smithi122 ceph-mon[93869]: pgmap v1032: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:14:18.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:14:18 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:14:18.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:14:18 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:14:18.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:14:18 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:14:18.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:14:18 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:14:18.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:14:18 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:14:18.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:14:18 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:14:18.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:14:18 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:14:18.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:14:18 smithi122 ceph-mon[93869]: osdmap e279: 12 total, 12 up, 12 in 2023-03-09T15:14:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:14:18 smithi181 ceph-mon[94189]: pgmap v1032: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:14:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:14:18 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:14:18.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:14:18 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:14:18.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:14:18 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:14:18.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:14:18 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:14:18.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:14:18 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:14:18.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:14:18 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:14:18.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:14:18 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:14:18.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:14:18 smithi181 ceph-mon[94189]: osdmap e279: 12 total, 12 up, 12 in 2023-03-09T15:14:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:14:18 smithi038 ceph-mon[92422]: pgmap v1032: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:14:18.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:14:18 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:14:18.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:14:18 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:14:18.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:14:18 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:14:18.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:14:18 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:14:18.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:14:18 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:14:18.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:14:18 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:14:18.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:14:18 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:14:18.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:14:18 smithi038 ceph-mon[92422]: osdmap e279: 12 total, 12 up, 12 in 2023-03-09T15:14:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1430.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:14:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:14:20 smithi122 ceph-mon[93869]: pgmap v1034: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:14:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:14:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 970.951374 secs 2023-03-09T15:14:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:14:20 smithi181 ceph-mon[94189]: pgmap v1034: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:14:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:14:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 970.951374 secs 2023-03-09T15:14:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:14:20 smithi038 ceph-mon[92422]: pgmap v1034: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:14:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:14:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 970.951374 secs 2023-03-09T15:14:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:14:22 smithi122 ceph-mon[93869]: pgmap v1035: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:14:22.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:14:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 973.141041 secs 2023-03-09T15:14:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:14:22 smithi181 ceph-mon[94189]: pgmap v1035: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:14:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:14:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 973.141041 secs 2023-03-09T15:14:22.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:14:22 smithi038 ceph-mon[92422]: pgmap v1035: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:14:22.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:14:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 973.141041 secs 2023-03-09T15:14:23.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1435.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:14:24.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:14:24 smithi122 ceph-mon[93869]: pgmap v1036: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:14:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:14:24 smithi181 ceph-mon[94189]: pgmap v1036: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:14:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:14:24 smithi038 ceph-mon[92422]: pgmap v1036: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:14:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:14:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 975.951435 secs 2023-03-09T15:14:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:14:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 975.951435 secs 2023-03-09T15:14:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:14:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 975.951435 secs 2023-03-09T15:14:26.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:14:26 smithi122 ceph-mon[93869]: pgmap v1037: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:14:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:14:26 smithi181 ceph-mon[94189]: pgmap v1037: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:14:26.902 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:14:26 smithi038 ceph-mon[92422]: pgmap v1037: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:14:27.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:14:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 978.141165 secs 2023-03-09T15:14:27.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:14:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 978.141165 secs 2023-03-09T15:14:27.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:14:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 978.141165 secs 2023-03-09T15:14:28.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:14:28 smithi122 ceph-mon[93869]: pgmap v1038: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 8.2 KiB/s wr, 7 op/s 2023-03-09T15:14:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:14:28 smithi181 ceph-mon[94189]: pgmap v1038: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 8.2 KiB/s wr, 7 op/s 2023-03-09T15:14:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:14:28 smithi038 ceph-mon[92422]: pgmap v1038: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 8.2 KiB/s wr, 7 op/s 2023-03-09T15:14:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1440.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:14:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:14:30 smithi122 ceph-mon[93869]: pgmap v1039: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 7.3 KiB/s wr, 6 op/s 2023-03-09T15:14:30.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:14:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 980.951494 secs 2023-03-09T15:14:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:14:30 smithi181 ceph-mon[94189]: pgmap v1039: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 7.3 KiB/s wr, 6 op/s 2023-03-09T15:14:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:14:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 980.951494 secs 2023-03-09T15:14:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:14:30 smithi038 ceph-mon[92422]: pgmap v1039: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 7.3 KiB/s wr, 6 op/s 2023-03-09T15:14:30.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:14:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 980.951494 secs 2023-03-09T15:14:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:14:32 smithi122 ceph-mon[93869]: pgmap v1040: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 6.8 KiB/s wr, 6 op/s 2023-03-09T15:14:32.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:14:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 983.141232 secs 2023-03-09T15:14:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:14:32 smithi181 ceph-mon[94189]: pgmap v1040: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 6.8 KiB/s wr, 6 op/s 2023-03-09T15:14:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:14:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 983.141232 secs 2023-03-09T15:14:32.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:14:32 smithi038 ceph-mon[92422]: pgmap v1040: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 6.8 KiB/s wr, 6 op/s 2023-03-09T15:14:32.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:14:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 983.141232 secs 2023-03-09T15:14:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1445.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:14:34.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:14:34 smithi122 ceph-mon[93869]: pgmap v1041: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 29 KiB/s wr, 10 op/s 2023-03-09T15:14:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:14:34 smithi181 ceph-mon[94189]: pgmap v1041: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 29 KiB/s wr, 10 op/s 2023-03-09T15:14:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:14:34 smithi038 ceph-mon[92422]: pgmap v1041: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 29 KiB/s wr, 10 op/s 2023-03-09T15:14:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:14:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 985.951555 secs 2023-03-09T15:14:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:14:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 985.951555 secs 2023-03-09T15:14:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:14:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 985.951555 secs 2023-03-09T15:14:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:14:36 smithi122 ceph-mon[93869]: pgmap v1042: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 30 KiB/s wr, 8 op/s 2023-03-09T15:14:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:14:36 smithi181 ceph-mon[94189]: pgmap v1042: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 30 KiB/s wr, 8 op/s 2023-03-09T15:14:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:14:36 smithi038 ceph-mon[92422]: pgmap v1042: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 30 KiB/s wr, 8 op/s 2023-03-09T15:14:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:14:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 988.141324 secs 2023-03-09T15:14:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:14:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 988.141324 secs 2023-03-09T15:14:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:14:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 988.141324 secs 2023-03-09T15:14:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:14:38 smithi122 ceph-mon[93869]: pgmap v1043: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 30 KiB/s wr, 11 op/s 2023-03-09T15:14:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:14:38 smithi181 ceph-mon[94189]: pgmap v1043: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 30 KiB/s wr, 11 op/s 2023-03-09T15:14:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:14:38 smithi038 ceph-mon[92422]: pgmap v1043: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 30 KiB/s wr, 11 op/s 2023-03-09T15:14:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1450.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:14:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:14:40 smithi122 ceph-mon[93869]: pgmap v1044: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 23 KiB/s wr, 8 op/s 2023-03-09T15:14:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:14:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 990.951614 secs 2023-03-09T15:14:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:14:40 smithi181 ceph-mon[94189]: pgmap v1044: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 23 KiB/s wr, 8 op/s 2023-03-09T15:14:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:14:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 990.951614 secs 2023-03-09T15:14:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:14:40 smithi038 ceph-mon[92422]: pgmap v1044: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 23 KiB/s wr, 8 op/s 2023-03-09T15:14:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:14:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 990.951614 secs 2023-03-09T15:14:42.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:14:42 smithi122 ceph-mon[93869]: pgmap v1045: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 23 KiB/s wr, 8 op/s 2023-03-09T15:14:42.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:14:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 993.141436 secs 2023-03-09T15:14:42.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:14:42 smithi181 ceph-mon[94189]: pgmap v1045: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 23 KiB/s wr, 8 op/s 2023-03-09T15:14:42.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:14:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 993.141436 secs 2023-03-09T15:14:42.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:14:42 smithi038 ceph-mon[92422]: pgmap v1045: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 23 KiB/s wr, 8 op/s 2023-03-09T15:14:42.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:14:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 993.141436 secs 2023-03-09T15:14:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1455.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:14:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:14:44 smithi122 ceph-mon[93869]: pgmap v1046: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 31 KiB/s wr, 10 op/s 2023-03-09T15:14:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:14:44 smithi181 ceph-mon[94189]: pgmap v1046: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 31 KiB/s wr, 10 op/s 2023-03-09T15:14:44.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:14:44 smithi038 ceph-mon[92422]: pgmap v1046: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 31 KiB/s wr, 10 op/s 2023-03-09T15:14:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:14:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 995.951675 secs 2023-03-09T15:14:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:14:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 995.951675 secs 2023-03-09T15:14:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:14:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 995.951675 secs 2023-03-09T15:14:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:14:46 smithi122 ceph-mon[93869]: pgmap v1047: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 8.2 KiB/s wr, 6 op/s 2023-03-09T15:14:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:14:46 smithi181 ceph-mon[94189]: pgmap v1047: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 8.2 KiB/s wr, 6 op/s 2023-03-09T15:14:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:14:46 smithi038 ceph-mon[92422]: pgmap v1047: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 8.2 KiB/s wr, 6 op/s 2023-03-09T15:14:47.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:14:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 998.141575 secs 2023-03-09T15:14:47.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:14:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 998.141575 secs 2023-03-09T15:14:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:14:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 998.141575 secs 2023-03-09T15:14:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:14:48 smithi122 ceph-mon[93869]: pgmap v1048: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 8.1 KiB/s wr, 8 op/s 2023-03-09T15:14:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:14:48 smithi181 ceph-mon[94189]: pgmap v1048: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 8.1 KiB/s wr, 8 op/s 2023-03-09T15:14:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:14:48 smithi038 ceph-mon[92422]: pgmap v1048: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 8.1 KiB/s wr, 8 op/s 2023-03-09T15:14:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1460.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:14:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:14:50 smithi122 ceph-mon[93869]: pgmap v1049: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.5 KiB/s wr, 5 op/s 2023-03-09T15:14:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:14:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1000.951735 secs 2023-03-09T15:14:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:14:50 smithi181 ceph-mon[94189]: pgmap v1049: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.5 KiB/s wr, 5 op/s 2023-03-09T15:14:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:14:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1000.951735 secs 2023-03-09T15:14:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:14:50 smithi038 ceph-mon[92422]: pgmap v1049: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.5 KiB/s wr, 5 op/s 2023-03-09T15:14:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:14:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1000.951735 secs 2023-03-09T15:14:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:14:52 smithi122 ceph-mon[93869]: pgmap v1050: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.5 KiB/s wr, 5 op/s 2023-03-09T15:14:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:14:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1003.141693 secs 2023-03-09T15:14:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:14:52 smithi181 ceph-mon[94189]: pgmap v1050: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.5 KiB/s wr, 5 op/s 2023-03-09T15:14:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:14:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1003.141693 secs 2023-03-09T15:14:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:14:52 smithi038 ceph-mon[92422]: pgmap v1050: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.5 KiB/s wr, 5 op/s 2023-03-09T15:14:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:14:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1003.141693 secs 2023-03-09T15:14:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1465.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:14:54.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:14:54 smithi122 ceph-mon[93869]: pgmap v1051: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.5 KiB/s wr, 7 op/s 2023-03-09T15:14:54.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:14:54 smithi181 ceph-mon[94189]: pgmap v1051: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.5 KiB/s wr, 7 op/s 2023-03-09T15:14:54.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:14:54 smithi038 ceph-mon[92422]: pgmap v1051: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.5 KiB/s wr, 7 op/s 2023-03-09T15:14:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:14:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1005.951794 secs 2023-03-09T15:14:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:14:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1005.951794 secs 2023-03-09T15:14:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:14:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1005.951794 secs 2023-03-09T15:14:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:14:56 smithi122 ceph-mon[93869]: pgmap v1052: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T15:14:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:14:56 smithi181 ceph-mon[94189]: pgmap v1052: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T15:14:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:14:56 smithi038 ceph-mon[92422]: pgmap v1052: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T15:14:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:14:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1008.141761 secs 2023-03-09T15:14:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:14:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1008.141761 secs 2023-03-09T15:14:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:14:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1008.141761 secs 2023-03-09T15:14:58.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:14:58 smithi122 ceph-mon[93869]: pgmap v1053: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T15:14:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:14:58 smithi181 ceph-mon[94189]: pgmap v1053: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T15:14:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:14:58 smithi038 ceph-mon[92422]: pgmap v1053: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T15:14:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1470.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:15:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:15:00 smithi122 ceph-mon[93869]: pgmap v1054: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T15:15:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:15:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1010.951880 secs 2023-03-09T15:15:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:15:00 smithi181 ceph-mon[94189]: pgmap v1054: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T15:15:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:15:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1010.951880 secs 2023-03-09T15:15:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:15:00 smithi038 ceph-mon[92422]: pgmap v1054: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T15:15:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:15:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1010.951880 secs 2023-03-09T15:15:02.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:15:02 smithi122 ceph-mon[93869]: pgmap v1055: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-03-09T15:15:02.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:15:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1013.141884 secs 2023-03-09T15:15:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:15:02 smithi181 ceph-mon[94189]: pgmap v1055: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-03-09T15:15:02.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:15:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1013.141884 secs 2023-03-09T15:15:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:15:02 smithi038 ceph-mon[92422]: pgmap v1055: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-03-09T15:15:02.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:15:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1013.141884 secs 2023-03-09T15:15:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1475.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:15:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:15:04 smithi122 ceph-mon[93869]: pgmap v1056: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 21 op/s 2023-03-09T15:15:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:15:04 smithi181 ceph-mon[94189]: pgmap v1056: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 21 op/s 2023-03-09T15:15:04.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:15:04 smithi038 ceph-mon[92422]: pgmap v1056: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 21 op/s 2023-03-09T15:15:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:15:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1015.951955 secs 2023-03-09T15:15:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:15:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1015.951955 secs 2023-03-09T15:15:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:15:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1015.951955 secs 2023-03-09T15:15:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:15:06 smithi122 ceph-mon[93869]: pgmap v1057: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-03-09T15:15:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:15:06 smithi181 ceph-mon[94189]: pgmap v1057: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-03-09T15:15:06.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:15:06 smithi038 ceph-mon[92422]: pgmap v1057: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-03-09T15:15:07.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:15:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1018.142018 secs 2023-03-09T15:15:07.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:15:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1018.142018 secs 2023-03-09T15:15:07.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:15:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1018.142018 secs 2023-03-09T15:15:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:15:08 smithi181 ceph-mon[94189]: pgmap v1058: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 21 op/s 2023-03-09T15:15:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:15:08 smithi038 ceph-mon[92422]: pgmap v1058: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 21 op/s 2023-03-09T15:15:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:15:08 smithi122 ceph-mon[93869]: pgmap v1058: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 21 op/s 2023-03-09T15:15:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1480.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:15:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:15:10 smithi038 ceph-mon[92422]: osdmap e280: 12 total, 12 up, 12 in 2023-03-09T15:15:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:15:10 smithi038 ceph-mon[92422]: pgmap v1060: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 24 op/s 2023-03-09T15:15:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:15:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1020.951983 secs 2023-03-09T15:15:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:15:10 smithi122 ceph-mon[93869]: osdmap e280: 12 total, 12 up, 12 in 2023-03-09T15:15:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:15:10 smithi122 ceph-mon[93869]: pgmap v1060: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 24 op/s 2023-03-09T15:15:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:15:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1020.951983 secs 2023-03-09T15:15:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:15:10 smithi181 ceph-mon[94189]: osdmap e280: 12 total, 12 up, 12 in 2023-03-09T15:15:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:15:10 smithi181 ceph-mon[94189]: pgmap v1060: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 24 op/s 2023-03-09T15:15:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:15:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1020.951983 secs 2023-03-09T15:15:12.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:15:12 smithi122 ceph-mon[93869]: pgmap v1061: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 24 op/s 2023-03-09T15:15:12.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:15:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1023.142159 secs 2023-03-09T15:15:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:15:12 smithi181 ceph-mon[94189]: pgmap v1061: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 24 op/s 2023-03-09T15:15:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:15:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1023.142159 secs 2023-03-09T15:15:12.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:15:12 smithi038 ceph-mon[92422]: pgmap v1061: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 24 op/s 2023-03-09T15:15:12.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:15:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1023.142159 secs 2023-03-09T15:15:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1485.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:15:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:15:14 smithi122 ceph-mon[93869]: pgmap v1062: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 7 op/s 2023-03-09T15:15:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:15:14 smithi181 ceph-mon[94189]: pgmap v1062: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 7 op/s 2023-03-09T15:15:14.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:15:14 smithi038 ceph-mon[92422]: pgmap v1062: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 7 op/s 2023-03-09T15:15:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:15:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1025.952043 secs 2023-03-09T15:15:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:15:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1025.952043 secs 2023-03-09T15:15:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:15:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1025.952043 secs 2023-03-09T15:15:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:15:16 smithi122 ceph-mon[93869]: pgmap v1063: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 7 op/s 2023-03-09T15:15:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:15:16 smithi181 ceph-mon[94189]: pgmap v1063: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 7 op/s 2023-03-09T15:15:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:15:16 smithi038 ceph-mon[92422]: pgmap v1063: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 7 op/s 2023-03-09T15:15:17.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:15:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1028.142290 secs 2023-03-09T15:15:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:15:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1028.142290 secs 2023-03-09T15:15:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:15:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1028.142290 secs 2023-03-09T15:15:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:15:18 smithi122 ceph-mon[93869]: pgmap v1064: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:15:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:15:18 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:15:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:15:18 smithi122 ceph-mon[93869]: osdmap e281: 12 total, 12 up, 12 in 2023-03-09T15:15:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:15:18 smithi181 ceph-mon[94189]: pgmap v1064: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:15:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:15:18 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:15:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:15:18 smithi181 ceph-mon[94189]: osdmap e281: 12 total, 12 up, 12 in 2023-03-09T15:15:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:15:18 smithi038 ceph-mon[92422]: pgmap v1064: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:15:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:15:18 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:15:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:15:18 smithi038 ceph-mon[92422]: osdmap e281: 12 total, 12 up, 12 in 2023-03-09T15:15:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1490.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:15:19.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:15:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:15:19.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:15:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:15:19.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:15:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:15:19.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:15:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:15:19.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:15:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:15:19.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:15:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:15:19.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:15:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:15:19.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:15:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:15:19.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:15:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:15:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:15:20 smithi181 ceph-mon[94189]: pgmap v1066: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:15:20.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:15:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1030.952103 secs 2023-03-09T15:15:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:15:20 smithi038 ceph-mon[92422]: pgmap v1066: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:15:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:15:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1030.952103 secs 2023-03-09T15:15:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:15:20 smithi122 ceph-mon[93869]: pgmap v1066: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:15:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:15:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1030.952103 secs 2023-03-09T15:15:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:15:22 smithi122 ceph-mon[93869]: pgmap v1067: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:15:22.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:15:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1033.142409 secs 2023-03-09T15:15:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:15:22 smithi181 ceph-mon[94189]: pgmap v1067: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:15:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:15:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1033.142409 secs 2023-03-09T15:15:22.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:15:22 smithi038 ceph-mon[92422]: pgmap v1067: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:15:22.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:15:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1033.142409 secs 2023-03-09T15:15:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1495.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:15:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:15:24 smithi122 ceph-mon[93869]: pgmap v1068: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:15:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:15:24 smithi181 ceph-mon[94189]: pgmap v1068: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:15:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:15:24 smithi038 ceph-mon[92422]: pgmap v1068: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:15:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:15:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1035.952163 secs 2023-03-09T15:15:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:15:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1035.952163 secs 2023-03-09T15:15:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:15:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1035.952163 secs 2023-03-09T15:15:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:15:26 smithi038 ceph-mon[92422]: pgmap v1069: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:15:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:15:26 smithi122 ceph-mon[93869]: pgmap v1069: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:15:26.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:15:26 smithi181 ceph-mon[94189]: pgmap v1069: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:15:27.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:15:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1038.142522 secs 2023-03-09T15:15:27.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:15:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1038.142522 secs 2023-03-09T15:15:27.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:15:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1038.142522 secs 2023-03-09T15:15:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:15:28 smithi038 ceph-mon[92422]: pgmap v1070: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 8.4 KiB/s wr, 7 op/s 2023-03-09T15:15:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:15:28 smithi122 ceph-mon[93869]: pgmap v1070: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 8.4 KiB/s wr, 7 op/s 2023-03-09T15:15:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:15:28 smithi181 ceph-mon[94189]: pgmap v1070: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 8.4 KiB/s wr, 7 op/s 2023-03-09T15:15:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1500.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:15:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:15:30 smithi122 ceph-mon[93869]: pgmap v1071: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 7.4 KiB/s wr, 6 op/s 2023-03-09T15:15:30.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:15:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1040.952222 secs 2023-03-09T15:15:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:15:30 smithi181 ceph-mon[94189]: pgmap v1071: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 7.4 KiB/s wr, 6 op/s 2023-03-09T15:15:30.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:15:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1040.952222 secs 2023-03-09T15:15:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:15:30 smithi038 ceph-mon[92422]: pgmap v1071: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 7.4 KiB/s wr, 6 op/s 2023-03-09T15:15:30.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:15:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1040.952222 secs 2023-03-09T15:15:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:15:32 smithi122 ceph-mon[93869]: pgmap v1072: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 7.0 KiB/s wr, 6 op/s 2023-03-09T15:15:32.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:15:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1043.142642 secs 2023-03-09T15:15:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:15:32 smithi181 ceph-mon[94189]: pgmap v1072: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 7.0 KiB/s wr, 6 op/s 2023-03-09T15:15:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:15:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1043.142642 secs 2023-03-09T15:15:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:15:32 smithi038 ceph-mon[92422]: pgmap v1072: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 7.0 KiB/s wr, 6 op/s 2023-03-09T15:15:32.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:15:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1043.142642 secs 2023-03-09T15:15:33.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1505.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:15:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:15:34 smithi122 ceph-mon[93869]: pgmap v1073: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 31 KiB/s wr, 10 op/s 2023-03-09T15:15:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:15:34 smithi181 ceph-mon[94189]: pgmap v1073: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 31 KiB/s wr, 10 op/s 2023-03-09T15:15:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:15:34 smithi038 ceph-mon[92422]: pgmap v1073: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 31 KiB/s wr, 10 op/s 2023-03-09T15:15:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:15:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1045.952281 secs 2023-03-09T15:15:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:15:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1045.952281 secs 2023-03-09T15:15:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:15:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1045.952281 secs 2023-03-09T15:15:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:15:36 smithi122 ceph-mon[93869]: pgmap v1074: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 31 KiB/s wr, 8 op/s 2023-03-09T15:15:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:15:36 smithi181 ceph-mon[94189]: pgmap v1074: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 31 KiB/s wr, 8 op/s 2023-03-09T15:15:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:15:36 smithi038 ceph-mon[92422]: pgmap v1074: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 31 KiB/s wr, 8 op/s 2023-03-09T15:15:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:15:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1048.142749 secs 2023-03-09T15:15:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:15:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1048.142749 secs 2023-03-09T15:15:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:15:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1048.142749 secs 2023-03-09T15:15:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:15:38 smithi122 ceph-mon[93869]: pgmap v1075: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 31 KiB/s wr, 11 op/s 2023-03-09T15:15:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:15:38 smithi181 ceph-mon[94189]: pgmap v1075: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 31 KiB/s wr, 11 op/s 2023-03-09T15:15:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:15:38 smithi038 ceph-mon[92422]: pgmap v1075: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 31 KiB/s wr, 11 op/s 2023-03-09T15:15:38.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1510.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:15:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:15:40 smithi122 ceph-mon[93869]: pgmap v1076: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 24 KiB/s wr, 8 op/s 2023-03-09T15:15:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:15:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1050.952341 secs 2023-03-09T15:15:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:15:40 smithi181 ceph-mon[94189]: pgmap v1076: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 24 KiB/s wr, 8 op/s 2023-03-09T15:15:40.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:15:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1050.952341 secs 2023-03-09T15:15:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:15:40 smithi038 ceph-mon[92422]: pgmap v1076: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 24 KiB/s wr, 8 op/s 2023-03-09T15:15:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:15:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1050.952341 secs 2023-03-09T15:15:42.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:15:42 smithi122 ceph-mon[93869]: pgmap v1077: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 24 KiB/s wr, 8 op/s 2023-03-09T15:15:42.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:15:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1053.142871 secs 2023-03-09T15:15:42.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:15:42 smithi181 ceph-mon[94189]: pgmap v1077: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 24 KiB/s wr, 8 op/s 2023-03-09T15:15:42.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:15:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1053.142871 secs 2023-03-09T15:15:42.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:15:42 smithi038 ceph-mon[92422]: pgmap v1077: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 24 KiB/s wr, 8 op/s 2023-03-09T15:15:42.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:15:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1053.142871 secs 2023-03-09T15:15:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1515.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:15:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:15:44 smithi122 ceph-mon[93869]: pgmap v1078: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 32 KiB/s wr, 10 op/s 2023-03-09T15:15:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:15:44 smithi181 ceph-mon[94189]: pgmap v1078: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 32 KiB/s wr, 10 op/s 2023-03-09T15:15:44.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:15:44 smithi038 ceph-mon[92422]: pgmap v1078: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 32 KiB/s wr, 10 op/s 2023-03-09T15:15:45.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:15:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1055.952401 secs 2023-03-09T15:15:45.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:15:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1055.952401 secs 2023-03-09T15:15:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:15:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1055.952401 secs 2023-03-09T15:15:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:15:46 smithi122 ceph-mon[93869]: pgmap v1079: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 8.0 KiB/s wr, 5 op/s 2023-03-09T15:15:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:15:46 smithi181 ceph-mon[94189]: pgmap v1079: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 8.0 KiB/s wr, 5 op/s 2023-03-09T15:15:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:15:46 smithi038 ceph-mon[92422]: pgmap v1079: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 8.0 KiB/s wr, 5 op/s 2023-03-09T15:15:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:15:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1058.143017 secs 2023-03-09T15:15:47.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:15:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1058.143017 secs 2023-03-09T15:15:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:15:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1058.143017 secs 2023-03-09T15:15:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:15:48 smithi181 ceph-mon[94189]: pgmap v1080: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 8.0 KiB/s wr, 7 op/s 2023-03-09T15:15:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:15:48 smithi038 ceph-mon[92422]: pgmap v1080: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 8.0 KiB/s wr, 7 op/s 2023-03-09T15:15:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:15:48 smithi122 ceph-mon[93869]: pgmap v1080: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 8.0 KiB/s wr, 7 op/s 2023-03-09T15:15:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1520.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:15:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:15:50 smithi122 ceph-mon[93869]: pgmap v1081: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.7 KiB/s wr, 5 op/s 2023-03-09T15:15:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:15:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1060.952459 secs 2023-03-09T15:15:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:15:50 smithi181 ceph-mon[94189]: pgmap v1081: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.7 KiB/s wr, 5 op/s 2023-03-09T15:15:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:15:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1060.952459 secs 2023-03-09T15:15:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:15:50 smithi038 ceph-mon[92422]: pgmap v1081: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 7.7 KiB/s wr, 5 op/s 2023-03-09T15:15:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:15:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1060.952459 secs 2023-03-09T15:15:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:15:52 smithi122 ceph-mon[93869]: pgmap v1082: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.7 KiB/s wr, 5 op/s 2023-03-09T15:15:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:15:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1063.143156 secs 2023-03-09T15:15:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:15:52 smithi181 ceph-mon[94189]: pgmap v1082: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.7 KiB/s wr, 5 op/s 2023-03-09T15:15:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:15:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1063.143156 secs 2023-03-09T15:15:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:15:52 smithi038 ceph-mon[92422]: pgmap v1082: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.7 KiB/s wr, 5 op/s 2023-03-09T15:15:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:15:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1063.143156 secs 2023-03-09T15:15:53.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1525.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:15:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:15:54 smithi122 ceph-mon[93869]: pgmap v1083: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.7 KiB/s wr, 7 op/s 2023-03-09T15:15:54.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:15:54 smithi181 ceph-mon[94189]: pgmap v1083: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.7 KiB/s wr, 7 op/s 2023-03-09T15:15:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:15:54 smithi038 ceph-mon[92422]: pgmap v1083: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.7 KiB/s wr, 7 op/s 2023-03-09T15:15:55.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:15:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1065.952521 secs 2023-03-09T15:15:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:15:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1065.952521 secs 2023-03-09T15:15:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:15:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1065.952521 secs 2023-03-09T15:15:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:15:56 smithi122 ceph-mon[93869]: pgmap v1084: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T15:15:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:15:56 smithi181 ceph-mon[94189]: pgmap v1084: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T15:15:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:15:56 smithi038 ceph-mon[92422]: pgmap v1084: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T15:15:57.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:15:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1068.143298 secs 2023-03-09T15:15:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:15:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1068.143298 secs 2023-03-09T15:15:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:15:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1068.143298 secs 2023-03-09T15:15:58.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:15:58 smithi038 ceph-mon[92422]: pgmap v1085: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T15:15:58.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:15:58 smithi122 ceph-mon[93869]: pgmap v1085: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T15:15:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:15:58 smithi181 ceph-mon[94189]: pgmap v1085: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T15:15:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1530.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:16:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:16:00 smithi122 ceph-mon[93869]: pgmap v1086: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T15:16:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:16:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1070.952587 secs 2023-03-09T15:16:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:16:00 smithi181 ceph-mon[94189]: pgmap v1086: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T15:16:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:16:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1070.952587 secs 2023-03-09T15:16:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:16:00 smithi038 ceph-mon[92422]: pgmap v1086: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T15:16:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:16:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1070.952587 secs 2023-03-09T15:16:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:16:02 smithi122 ceph-mon[93869]: pgmap v1087: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-03-09T15:16:02.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:16:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1073.143439 secs 2023-03-09T15:16:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:16:02 smithi181 ceph-mon[94189]: pgmap v1087: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-03-09T15:16:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:16:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1073.143439 secs 2023-03-09T15:16:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:16:02 smithi038 ceph-mon[92422]: pgmap v1087: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-03-09T15:16:02.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:16:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1073.143439 secs 2023-03-09T15:16:03.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1535.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:16:04.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:16:04 smithi038 ceph-mon[92422]: pgmap v1088: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 21 op/s 2023-03-09T15:16:04.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:16:04 smithi122 ceph-mon[93869]: pgmap v1088: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 21 op/s 2023-03-09T15:16:04.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:16:04 smithi181 ceph-mon[94189]: pgmap v1088: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 21 op/s 2023-03-09T15:16:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:16:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1075.952648 secs 2023-03-09T15:16:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:16:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1075.952648 secs 2023-03-09T15:16:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:16:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1075.952648 secs 2023-03-09T15:16:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:16:06 smithi038 ceph-mon[92422]: pgmap v1089: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-03-09T15:16:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:16:06 smithi122 ceph-mon[93869]: pgmap v1089: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-03-09T15:16:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:16:06 smithi181 ceph-mon[94189]: pgmap v1089: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 19 op/s 2023-03-09T15:16:07.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:16:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1078.143576 secs 2023-03-09T15:16:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:16:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1078.143576 secs 2023-03-09T15:16:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:16:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1078.143576 secs 2023-03-09T15:16:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:16:08 smithi038 ceph-mon[92422]: pgmap v1090: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 21 op/s 2023-03-09T15:16:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:16:08 smithi122 ceph-mon[93869]: pgmap v1090: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 21 op/s 2023-03-09T15:16:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:16:08 smithi181 ceph-mon[94189]: pgmap v1090: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 21 op/s 2023-03-09T15:16:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1540.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:16:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:16:10 smithi122 ceph-mon[93869]: osdmap e282: 12 total, 12 up, 12 in 2023-03-09T15:16:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:16:10 smithi122 ceph-mon[93869]: pgmap v1092: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 55 KiB/s wr, 23 op/s 2023-03-09T15:16:10.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:16:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1080.952708 secs 2023-03-09T15:16:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:16:10 smithi181 ceph-mon[94189]: osdmap e282: 12 total, 12 up, 12 in 2023-03-09T15:16:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:16:10 smithi181 ceph-mon[94189]: pgmap v1092: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 55 KiB/s wr, 23 op/s 2023-03-09T15:16:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:16:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1080.952708 secs 2023-03-09T15:16:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:16:10 smithi038 ceph-mon[92422]: osdmap e282: 12 total, 12 up, 12 in 2023-03-09T15:16:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:16:10 smithi038 ceph-mon[92422]: pgmap v1092: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 55 KiB/s wr, 23 op/s 2023-03-09T15:16:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:16:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1080.952708 secs 2023-03-09T15:16:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:16:12 smithi122 ceph-mon[93869]: pgmap v1093: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 55 KiB/s wr, 23 op/s 2023-03-09T15:16:12.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:16:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1083.143710 secs 2023-03-09T15:16:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:16:12 smithi181 ceph-mon[94189]: pgmap v1093: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 55 KiB/s wr, 23 op/s 2023-03-09T15:16:12.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:16:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1083.143710 secs 2023-03-09T15:16:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:16:12 smithi038 ceph-mon[92422]: pgmap v1093: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 55 KiB/s wr, 23 op/s 2023-03-09T15:16:12.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:16:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1083.143710 secs 2023-03-09T15:16:13.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1545.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:16:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:16:14 smithi122 ceph-mon[93869]: pgmap v1094: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 7 op/s 2023-03-09T15:16:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:16:14 smithi181 ceph-mon[94189]: pgmap v1094: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 7 op/s 2023-03-09T15:16:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:16:14 smithi038 ceph-mon[92422]: pgmap v1094: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 7 op/s 2023-03-09T15:16:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:16:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1085.952770 secs 2023-03-09T15:16:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:16:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1085.952770 secs 2023-03-09T15:16:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:16:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1085.952770 secs 2023-03-09T15:16:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:16:16 smithi038 ceph-mon[92422]: pgmap v1095: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 7 op/s 2023-03-09T15:16:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:16:16 smithi122 ceph-mon[93869]: pgmap v1095: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 7 op/s 2023-03-09T15:16:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:16:16 smithi181 ceph-mon[94189]: pgmap v1095: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 7 op/s 2023-03-09T15:16:17.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:16:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1088.143835 secs 2023-03-09T15:16:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:16:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1088.143835 secs 2023-03-09T15:16:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:16:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1088.143835 secs 2023-03-09T15:16:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:16:18 smithi038 ceph-mon[92422]: pgmap v1096: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 7 op/s 2023-03-09T15:16:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:16:18 smithi038 ceph-mon[92422]: osdmap e283: 12 total, 12 up, 12 in 2023-03-09T15:16:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:16:18 smithi122 ceph-mon[93869]: pgmap v1096: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 7 op/s 2023-03-09T15:16:18.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:16:18 smithi122 ceph-mon[93869]: osdmap e283: 12 total, 12 up, 12 in 2023-03-09T15:16:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:16:18 smithi181 ceph-mon[94189]: pgmap v1096: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 12 KiB/s wr, 7 op/s 2023-03-09T15:16:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:16:18 smithi181 ceph-mon[94189]: osdmap e283: 12 total, 12 up, 12 in 2023-03-09T15:16:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1550.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:16:19.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:16:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:16:19.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:16:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:16:19.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:16:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:16:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:16:20 smithi038 ceph-mon[92422]: pgmap v1098: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 11 KiB/s wr, 7 op/s 2023-03-09T15:16:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:16:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1090.952845 secs 2023-03-09T15:16:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:16:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:16:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:16:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:16:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:16:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:16:20.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:16:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:16:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:16:20 smithi122 ceph-mon[93869]: pgmap v1098: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 11 KiB/s wr, 7 op/s 2023-03-09T15:16:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:16:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1090.952845 secs 2023-03-09T15:16:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:16:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:16:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:16:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:16:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:16:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:16:20.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:16:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:16:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:16:20 smithi181 ceph-mon[94189]: pgmap v1098: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 11 KiB/s wr, 7 op/s 2023-03-09T15:16:20.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:16:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1090.952845 secs 2023-03-09T15:16:20.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:16:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:16:20.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:16:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:16:20.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:16:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:16:20.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:16:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:16:22.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:16:22 smithi122 ceph-mon[93869]: pgmap v1099: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 11 KiB/s wr, 10 op/s 2023-03-09T15:16:22.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:16:22 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:16:22.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:16:22 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:16:22.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:16:22 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:16:22.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:16:22 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:16:22.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:16:22 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:16:22.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:16:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1093.143931 secs 2023-03-09T15:16:22.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:16:22 smithi181 ceph-mon[94189]: pgmap v1099: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 11 KiB/s wr, 10 op/s 2023-03-09T15:16:22.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:16:22 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:16:22.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:16:22 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:16:22.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:16:22 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:16:22.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:16:22 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:16:22.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:16:22 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:16:22.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:16:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1093.143931 secs 2023-03-09T15:16:22.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:16:22 smithi038 ceph-mon[92422]: pgmap v1099: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 11 KiB/s wr, 10 op/s 2023-03-09T15:16:23.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:16:22 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:16:23.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:16:22 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:16:23.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:16:22 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:16:23.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:16:22 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:16:23.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:16:22 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:16:23.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:16:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1093.143931 secs 2023-03-09T15:16:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1555.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:16:24.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:16:24 smithi122 ceph-mon[93869]: pgmap v1100: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:16:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:16:24 smithi181 ceph-mon[94189]: pgmap v1100: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:16:24.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:16:24 smithi038 ceph-mon[92422]: pgmap v1100: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:16:25.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:16:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1095.952930 secs 2023-03-09T15:16:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:16:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1095.952930 secs 2023-03-09T15:16:25.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:16:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1095.952930 secs 2023-03-09T15:16:26.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:16:26 smithi038 ceph-mon[92422]: pgmap v1101: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:16:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:16:26 smithi122 ceph-mon[93869]: pgmap v1101: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:16:26.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:16:26 smithi181 ceph-mon[94189]: pgmap v1101: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:16:27.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:16:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1098.144046 secs 2023-03-09T15:16:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:16:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1098.144046 secs 2023-03-09T15:16:27.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:16:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1098.144046 secs 2023-03-09T15:16:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:16:28 smithi038 ceph-mon[92422]: pgmap v1102: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 102 B/s wr, 7 op/s 2023-03-09T15:16:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:16:28 smithi122 ceph-mon[93869]: pgmap v1102: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 102 B/s wr, 7 op/s 2023-03-09T15:16:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:16:28 smithi181 ceph-mon[94189]: pgmap v1102: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 102 B/s wr, 7 op/s 2023-03-09T15:16:28.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1560.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:16:30.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:16:30 smithi122 ceph-mon[93869]: pgmap v1103: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 91 B/s wr, 6 op/s 2023-03-09T15:16:30.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:16:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1100.952986 secs 2023-03-09T15:16:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:16:30 smithi181 ceph-mon[94189]: pgmap v1103: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 91 B/s wr, 6 op/s 2023-03-09T15:16:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:16:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1100.952986 secs 2023-03-09T15:16:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:16:30 smithi038 ceph-mon[92422]: pgmap v1103: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 91 B/s wr, 6 op/s 2023-03-09T15:16:30.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:16:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1100.952986 secs 2023-03-09T15:16:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:16:32 smithi122 ceph-mon[93869]: pgmap v1104: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 17 KiB/s wr, 9 op/s 2023-03-09T15:16:32.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:16:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1103.144187 secs 2023-03-09T15:16:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:16:32 smithi181 ceph-mon[94189]: pgmap v1104: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 17 KiB/s wr, 9 op/s 2023-03-09T15:16:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:16:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1103.144187 secs 2023-03-09T15:16:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:16:32 smithi038 ceph-mon[92422]: pgmap v1104: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 17 KiB/s wr, 9 op/s 2023-03-09T15:16:32.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:16:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1103.144187 secs 2023-03-09T15:16:33.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1565.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:16:34.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:16:34 smithi038 ceph-mon[92422]: pgmap v1105: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 18 KiB/s wr, 8 op/s 2023-03-09T15:16:34.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:16:34 smithi122 ceph-mon[93869]: pgmap v1105: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 18 KiB/s wr, 8 op/s 2023-03-09T15:16:34.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:16:34 smithi181 ceph-mon[94189]: pgmap v1105: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 18 KiB/s wr, 8 op/s 2023-03-09T15:16:35.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:16:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1105.953046 secs 2023-03-09T15:16:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:16:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1105.953046 secs 2023-03-09T15:16:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:16:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1105.953046 secs 2023-03-09T15:16:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:16:36 smithi038 ceph-mon[92422]: pgmap v1106: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 18 KiB/s wr, 8 op/s 2023-03-09T15:16:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:16:36 smithi122 ceph-mon[93869]: pgmap v1106: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 18 KiB/s wr, 8 op/s 2023-03-09T15:16:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:16:36 smithi181 ceph-mon[94189]: pgmap v1106: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 18 KiB/s wr, 8 op/s 2023-03-09T15:16:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:16:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1108.144314 secs 2023-03-09T15:16:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:16:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1108.144314 secs 2023-03-09T15:16:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:16:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1108.144314 secs 2023-03-09T15:16:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:16:38 smithi122 ceph-mon[93869]: pgmap v1107: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 41 KiB/s wr, 12 op/s 2023-03-09T15:16:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:16:38 smithi181 ceph-mon[94189]: pgmap v1107: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 41 KiB/s wr, 12 op/s 2023-03-09T15:16:38.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:16:38 smithi038 ceph-mon[92422]: pgmap v1107: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 41 KiB/s wr, 12 op/s 2023-03-09T15:16:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1570.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:16:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:16:40 smithi122 ceph-mon[93869]: pgmap v1108: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 41 KiB/s wr, 9 op/s 2023-03-09T15:16:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:16:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1110.953123 secs 2023-03-09T15:16:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:16:40 smithi181 ceph-mon[94189]: pgmap v1108: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 41 KiB/s wr, 9 op/s 2023-03-09T15:16:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:16:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1110.953123 secs 2023-03-09T15:16:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:16:40 smithi038 ceph-mon[92422]: pgmap v1108: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 41 KiB/s wr, 9 op/s 2023-03-09T15:16:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:16:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1110.953123 secs 2023-03-09T15:16:42.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:16:42 smithi122 ceph-mon[93869]: pgmap v1109: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 49 KiB/s wr, 11 op/s 2023-03-09T15:16:42.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:16:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1113.144435 secs 2023-03-09T15:16:42.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:16:42 smithi181 ceph-mon[94189]: pgmap v1109: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 49 KiB/s wr, 11 op/s 2023-03-09T15:16:42.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:16:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1113.144435 secs 2023-03-09T15:16:42.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:16:42 smithi038 ceph-mon[92422]: pgmap v1109: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 49 KiB/s wr, 11 op/s 2023-03-09T15:16:42.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:16:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1113.144435 secs 2023-03-09T15:16:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1575.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:16:44.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:16:44 smithi038 ceph-mon[92422]: pgmap v1110: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T15:16:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:16:44 smithi122 ceph-mon[93869]: pgmap v1110: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T15:16:44.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:16:44 smithi181 ceph-mon[94189]: pgmap v1110: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T15:16:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:16:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1115.953209 secs 2023-03-09T15:16:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:16:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1115.953209 secs 2023-03-09T15:16:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:16:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1115.953209 secs 2023-03-09T15:16:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:16:46 smithi122 ceph-mon[93869]: pgmap v1111: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 31 KiB/s wr, 6 op/s 2023-03-09T15:16:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:16:46 smithi181 ceph-mon[94189]: pgmap v1111: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 31 KiB/s wr, 6 op/s 2023-03-09T15:16:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:16:46 smithi038 ceph-mon[92422]: pgmap v1111: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 31 KiB/s wr, 6 op/s 2023-03-09T15:16:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:16:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1118.144536 secs 2023-03-09T15:16:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:16:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1118.144536 secs 2023-03-09T15:16:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:16:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1118.144536 secs 2023-03-09T15:16:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:16:48 smithi122 ceph-mon[93869]: pgmap v1112: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 31 KiB/s wr, 8 op/s 2023-03-09T15:16:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:16:48 smithi181 ceph-mon[94189]: pgmap v1112: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 31 KiB/s wr, 8 op/s 2023-03-09T15:16:48.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:16:48 smithi038 ceph-mon[92422]: pgmap v1112: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 31 KiB/s wr, 8 op/s 2023-03-09T15:16:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1580.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:16:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:16:50 smithi122 ceph-mon[93869]: pgmap v1113: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.2 KiB/s wr, 5 op/s 2023-03-09T15:16:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:16:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1120.953270 secs 2023-03-09T15:16:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:16:50 smithi181 ceph-mon[94189]: pgmap v1113: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.2 KiB/s wr, 5 op/s 2023-03-09T15:16:50.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:16:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1120.953270 secs 2023-03-09T15:16:50.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:16:50 smithi038 ceph-mon[92422]: pgmap v1113: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.2 KiB/s wr, 5 op/s 2023-03-09T15:16:50.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:16:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1120.953270 secs 2023-03-09T15:16:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:16:52 smithi122 ceph-mon[93869]: pgmap v1114: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.2 KiB/s wr, 7 op/s 2023-03-09T15:16:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:16:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1123.144675 secs 2023-03-09T15:16:52.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:16:52 smithi181 ceph-mon[94189]: pgmap v1114: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.2 KiB/s wr, 7 op/s 2023-03-09T15:16:52.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:16:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1123.144675 secs 2023-03-09T15:16:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:16:52 smithi038 ceph-mon[92422]: pgmap v1114: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.2 KiB/s wr, 7 op/s 2023-03-09T15:16:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:16:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1123.144675 secs 2023-03-09T15:16:53.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1585.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:16:54.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:16:54 smithi038 ceph-mon[92422]: pgmap v1115: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T15:16:54.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:16:54 smithi122 ceph-mon[93869]: pgmap v1115: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T15:16:54.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:16:54 smithi181 ceph-mon[94189]: pgmap v1115: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T15:16:55.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:16:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1125.953347 secs 2023-03-09T15:16:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:16:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1125.953347 secs 2023-03-09T15:16:55.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:16:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1125.953347 secs 2023-03-09T15:16:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:16:56 smithi038 ceph-mon[92422]: pgmap v1116: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T15:16:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:16:56 smithi122 ceph-mon[93869]: pgmap v1116: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T15:16:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:16:56 smithi181 ceph-mon[94189]: pgmap v1116: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T15:16:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:16:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1128.144774 secs 2023-03-09T15:16:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:16:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1128.144774 secs 2023-03-09T15:16:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:16:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1128.144774 secs 2023-03-09T15:16:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:16:58 smithi038 ceph-mon[92422]: pgmap v1117: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T15:16:58.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:16:58 smithi122 ceph-mon[93869]: pgmap v1117: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T15:16:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:16:58 smithi181 ceph-mon[94189]: pgmap v1117: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T15:16:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1590.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:17:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:00 smithi122 ceph-mon[93869]: pgmap v1118: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T15:17:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1130.953409 secs 2023-03-09T15:17:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:00 smithi181 ceph-mon[94189]: pgmap v1118: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T15:17:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1130.953409 secs 2023-03-09T15:17:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:00 smithi038 ceph-mon[92422]: pgmap v1118: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T15:17:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1130.953409 secs 2023-03-09T15:17:02.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:02 smithi122 ceph-mon[93869]: pgmap v1119: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T15:17:02.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1133.144867 secs 2023-03-09T15:17:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:02 smithi181 ceph-mon[94189]: pgmap v1119: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T15:17:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1133.144867 secs 2023-03-09T15:17:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:02 smithi038 ceph-mon[92422]: pgmap v1119: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T15:17:02.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1133.144867 secs 2023-03-09T15:17:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1595.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:17:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:04 smithi038 ceph-mon[92422]: pgmap v1120: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 21 KiB/s wr, 16 op/s 2023-03-09T15:17:04.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:04 smithi122 ceph-mon[93869]: pgmap v1120: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 21 KiB/s wr, 16 op/s 2023-03-09T15:17:04.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:04 smithi181 ceph-mon[94189]: pgmap v1120: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 21 KiB/s wr, 16 op/s 2023-03-09T15:17:05.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1135.953469 secs 2023-03-09T15:17:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1135.953469 secs 2023-03-09T15:17:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1135.953469 secs 2023-03-09T15:17:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:06 smithi122 ceph-mon[93869]: pgmap v1121: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 21 KiB/s wr, 16 op/s 2023-03-09T15:17:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:06 smithi181 ceph-mon[94189]: pgmap v1121: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 21 KiB/s wr, 16 op/s 2023-03-09T15:17:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:06 smithi038 ceph-mon[92422]: pgmap v1121: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 21 KiB/s wr, 16 op/s 2023-03-09T15:17:07.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1138.145016 secs 2023-03-09T15:17:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1138.145016 secs 2023-03-09T15:17:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1138.145016 secs 2023-03-09T15:17:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:08 smithi122 ceph-mon[93869]: pgmap v1122: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2023-03-09T15:17:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:08 smithi181 ceph-mon[94189]: pgmap v1122: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2023-03-09T15:17:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:08 smithi038 ceph-mon[92422]: pgmap v1122: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2023-03-09T15:17:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1600.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:17:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:10 smithi122 ceph-mon[93869]: osdmap e284: 12 total, 12 up, 12 in 2023-03-09T15:17:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:10 smithi122 ceph-mon[93869]: pgmap v1124: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 24 op/s 2023-03-09T15:17:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1140.953529 secs 2023-03-09T15:17:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:10 smithi181 ceph-mon[94189]: osdmap e284: 12 total, 12 up, 12 in 2023-03-09T15:17:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:10 smithi181 ceph-mon[94189]: pgmap v1124: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 24 op/s 2023-03-09T15:17:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1140.953529 secs 2023-03-09T15:17:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:10 smithi038 ceph-mon[92422]: osdmap e284: 12 total, 12 up, 12 in 2023-03-09T15:17:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:10 smithi038 ceph-mon[92422]: pgmap v1124: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 24 op/s 2023-03-09T15:17:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1140.953529 secs 2023-03-09T15:17:12.718 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:12 smithi122 ceph-mon[93869]: pgmap v1125: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 24 op/s 2023-03-09T15:17:12.719 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1143.145158 secs 2023-03-09T15:17:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:12 smithi181 ceph-mon[94189]: pgmap v1125: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 24 op/s 2023-03-09T15:17:12.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1143.145158 secs 2023-03-09T15:17:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:12 smithi038 ceph-mon[92422]: pgmap v1125: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 24 op/s 2023-03-09T15:17:12.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1143.145158 secs 2023-03-09T15:17:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1605.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:17:14.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:14 smithi038 ceph-mon[92422]: pgmap v1126: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 28 KiB/s wr, 10 op/s 2023-03-09T15:17:14.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:14 smithi122 ceph-mon[93869]: pgmap v1126: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 28 KiB/s wr, 10 op/s 2023-03-09T15:17:14.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:14 smithi181 ceph-mon[94189]: pgmap v1126: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 28 KiB/s wr, 10 op/s 2023-03-09T15:17:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1145.953590 secs 2023-03-09T15:17:15.663 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1145.953590 secs 2023-03-09T15:17:15.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1145.953590 secs 2023-03-09T15:17:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:16 smithi122 ceph-mon[93869]: pgmap v1127: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 28 KiB/s wr, 10 op/s 2023-03-09T15:17:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:16 smithi181 ceph-mon[94189]: pgmap v1127: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 28 KiB/s wr, 10 op/s 2023-03-09T15:17:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:16 smithi038 ceph-mon[92422]: pgmap v1127: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 28 KiB/s wr, 10 op/s 2023-03-09T15:17:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1148.145262 secs 2023-03-09T15:17:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1148.145262 secs 2023-03-09T15:17:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1148.145262 secs 2023-03-09T15:17:18.665 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:18 smithi122 ceph-mon[93869]: pgmap v1128: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:17:18.665 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:18 smithi122 ceph-mon[93869]: osdmap e285: 12 total, 12 up, 12 in 2023-03-09T15:17:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:18 smithi181 ceph-mon[94189]: pgmap v1128: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:17:18.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:18 smithi181 ceph-mon[94189]: osdmap e285: 12 total, 12 up, 12 in 2023-03-09T15:17:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:18 smithi038 ceph-mon[92422]: pgmap v1128: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:17:18.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:18 smithi038 ceph-mon[92422]: osdmap e285: 12 total, 12 up, 12 in 2023-03-09T15:17:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1610.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:17:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:20 smithi122 ceph-mon[93869]: pgmap v1130: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:17:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1150.953649 secs 2023-03-09T15:17:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:20 smithi181 ceph-mon[94189]: pgmap v1130: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:17:20.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1150.953649 secs 2023-03-09T15:17:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:20 smithi038 ceph-mon[92422]: pgmap v1130: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:17:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1150.953649 secs 2023-03-09T15:17:22.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:22 smithi122 ceph-mon[93869]: pgmap v1131: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:17:22.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1153.145376 secs 2023-03-09T15:17:22.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:22 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:17:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:22 smithi181 ceph-mon[94189]: pgmap v1131: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:17:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1153.145376 secs 2023-03-09T15:17:22.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:22 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:17:22.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:22 smithi038 ceph-mon[92422]: pgmap v1131: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:17:22.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1153.145376 secs 2023-03-09T15:17:22.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:22 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:17:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1615.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:17:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:24 smithi038 ceph-mon[92422]: pgmap v1132: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:17:24.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:24 smithi122 ceph-mon[93869]: pgmap v1132: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:17:24.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:24 smithi181 ceph-mon[94189]: pgmap v1132: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:17:25.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1155.953711 secs 2023-03-09T15:17:25.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:17:25.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:17:25.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:17:25.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:17:25.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:17:25.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:17:25.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:17:25.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:17:25.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:17:25.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:17:25.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:17:25.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:17:25.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:17:25.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:17:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1155.953711 secs 2023-03-09T15:17:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:17:25.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:17:25.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:17:25.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:17:25.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:17:25.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:17:25.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:17:25.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:17:25.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:17:25.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:17:25.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:17:25.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:17:25.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:17:25.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:17:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1155.953711 secs 2023-03-09T15:17:26.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:17:26.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:17:26.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:17:26.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:17:26.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:17:26.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:17:26.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:17:26.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:17:26.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:17:26.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:17:26.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:17:26.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:17:26.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:17:26.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:17:26.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:26 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T15:17:26.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:26 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi038 to 197.2M 2023-03-09T15:17:26.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:26 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T15:17:26.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:26 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi038 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-03-09T15:17:26.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:26 smithi122 ceph-mon[93869]: pgmap v1133: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:17:26.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:26 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:17:26.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:26 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:17:26.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:26 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:17:26.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:26 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:17:26.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:26 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:17:26.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:26 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:17:26.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:26 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:17:26.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:26 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T15:17:26.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:26 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi038 to 197.2M 2023-03-09T15:17:26.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:26 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T15:17:26.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:26 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi038 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-03-09T15:17:26.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:26 smithi181 ceph-mon[94189]: pgmap v1133: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:17:26.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:26 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:17:26.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:26 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:17:26.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:26 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:17:26.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:26 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:17:26.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:26 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:17:26.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:26 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:17:26.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:26 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:17:27.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:26 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T15:17:27.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:26 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi038 to 197.2M 2023-03-09T15:17:27.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:26 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T15:17:27.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:26 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi038 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-03-09T15:17:27.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:26 smithi038 ceph-mon[92422]: pgmap v1133: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:17:27.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:26 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:17:27.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:26 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:17:27.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:26 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:17:27.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:26 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:17:27.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:26 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:17:27.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:26 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:17:27.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:26 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:17:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:27 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T15:17:27.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:27 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T15:17:27.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1158.145508 secs 2023-03-09T15:17:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:27 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T15:17:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:27 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T15:17:27.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1158.145508 secs 2023-03-09T15:17:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:27 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T15:17:28.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:27 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T15:17:28.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1158.145508 secs 2023-03-09T15:17:28.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1620.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:17:28.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:28 smithi122 ceph-mon[93869]: pgmap v1134: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:17:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:28 smithi181 ceph-mon[94189]: pgmap v1134: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:17:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:28 smithi038 ceph-mon[92422]: pgmap v1134: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:17:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:30 smithi122 ceph-mon[93869]: pgmap v1135: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 91 B/s wr, 5 op/s 2023-03-09T15:17:30.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1160.953772 secs 2023-03-09T15:17:30.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:30 smithi181 ceph-mon[94189]: pgmap v1135: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 91 B/s wr, 5 op/s 2023-03-09T15:17:30.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1160.953772 secs 2023-03-09T15:17:30.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:30 smithi038 ceph-mon[92422]: pgmap v1135: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 91 B/s wr, 5 op/s 2023-03-09T15:17:31.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1160.953772 secs 2023-03-09T15:17:32.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:32 smithi122 ceph-mon[93869]: pgmap v1136: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 7.7 KiB/s wr, 8 op/s 2023-03-09T15:17:32.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1163.145648 secs 2023-03-09T15:17:32.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:32 smithi181 ceph-mon[94189]: pgmap v1136: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 7.7 KiB/s wr, 8 op/s 2023-03-09T15:17:32.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1163.145648 secs 2023-03-09T15:17:32.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:32 smithi038 ceph-mon[92422]: pgmap v1136: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 7.7 KiB/s wr, 8 op/s 2023-03-09T15:17:33.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1163.145648 secs 2023-03-09T15:17:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1625.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:17:34.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:34 smithi122 ceph-mon[93869]: pgmap v1137: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 8.0 KiB/s wr, 7 op/s 2023-03-09T15:17:34.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:34 smithi181 ceph-mon[94189]: pgmap v1137: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 8.0 KiB/s wr, 7 op/s 2023-03-09T15:17:34.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:34 smithi038 ceph-mon[92422]: pgmap v1137: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 8.0 KiB/s wr, 7 op/s 2023-03-09T15:17:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1165.953834 secs 2023-03-09T15:17:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1165.953834 secs 2023-03-09T15:17:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1165.953834 secs 2023-03-09T15:17:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:36 smithi122 ceph-mon[93869]: pgmap v1138: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 8.0 KiB/s wr, 7 op/s 2023-03-09T15:17:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:36 smithi181 ceph-mon[94189]: pgmap v1138: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 8.0 KiB/s wr, 7 op/s 2023-03-09T15:17:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:36 smithi038 ceph-mon[92422]: pgmap v1138: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 8.0 KiB/s wr, 7 op/s 2023-03-09T15:17:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1168.145765 secs 2023-03-09T15:17:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1168.145765 secs 2023-03-09T15:17:37.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1168.145765 secs 2023-03-09T15:17:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:38 smithi122 ceph-mon[93869]: pgmap v1139: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 27 KiB/s wr, 10 op/s 2023-03-09T15:17:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:38 smithi181 ceph-mon[94189]: pgmap v1139: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 27 KiB/s wr, 10 op/s 2023-03-09T15:17:38.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:38 smithi038 ceph-mon[92422]: pgmap v1139: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 27 KiB/s wr, 10 op/s 2023-03-09T15:17:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1630.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:17:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:40 smithi122 ceph-mon[93869]: pgmap v1140: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T15:17:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1170.953894 secs 2023-03-09T15:17:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:40 smithi181 ceph-mon[94189]: pgmap v1140: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T15:17:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1170.953894 secs 2023-03-09T15:17:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:40 smithi038 ceph-mon[92422]: pgmap v1140: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T15:17:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1170.953894 secs 2023-03-09T15:17:42.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:42 smithi122 ceph-mon[93869]: pgmap v1141: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 35 KiB/s wr, 10 op/s 2023-03-09T15:17:42.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1173.145885 secs 2023-03-09T15:17:42.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:42 smithi181 ceph-mon[94189]: pgmap v1141: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 35 KiB/s wr, 10 op/s 2023-03-09T15:17:42.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1173.145885 secs 2023-03-09T15:17:42.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:42 smithi038 ceph-mon[92422]: pgmap v1141: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 35 KiB/s wr, 10 op/s 2023-03-09T15:17:42.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1173.145885 secs 2023-03-09T15:17:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1635.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:17:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:44 smithi122 ceph-mon[93869]: pgmap v1142: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 28 KiB/s wr, 6 op/s 2023-03-09T15:17:44.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:44 smithi181 ceph-mon[94189]: pgmap v1142: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 28 KiB/s wr, 6 op/s 2023-03-09T15:17:44.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:44 smithi038 ceph-mon[92422]: pgmap v1142: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 28 KiB/s wr, 6 op/s 2023-03-09T15:17:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1175.953956 secs 2023-03-09T15:17:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1175.953956 secs 2023-03-09T15:17:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1175.953956 secs 2023-03-09T15:17:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:46 smithi122 ceph-mon[93869]: pgmap v1143: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 5 op/s 2023-03-09T15:17:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:46 smithi181 ceph-mon[94189]: pgmap v1143: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 5 op/s 2023-03-09T15:17:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:46 smithi038 ceph-mon[92422]: pgmap v1143: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 5 op/s 2023-03-09T15:17:47.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1178.145994 secs 2023-03-09T15:17:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1178.145994 secs 2023-03-09T15:17:47.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1178.145994 secs 2023-03-09T15:17:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:48 smithi122 ceph-mon[93869]: pgmap v1144: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 27 KiB/s wr, 8 op/s 2023-03-09T15:17:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:48 smithi181 ceph-mon[94189]: pgmap v1144: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 27 KiB/s wr, 8 op/s 2023-03-09T15:17:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:48 smithi038 ceph-mon[92422]: pgmap v1144: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 27 KiB/s wr, 8 op/s 2023-03-09T15:17:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1640.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:17:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:50 smithi122 ceph-mon[93869]: pgmap v1145: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.2 KiB/s wr, 5 op/s 2023-03-09T15:17:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1180.954010 secs 2023-03-09T15:17:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:50 smithi181 ceph-mon[94189]: pgmap v1145: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.2 KiB/s wr, 5 op/s 2023-03-09T15:17:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1180.954010 secs 2023-03-09T15:17:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:50 smithi038 ceph-mon[92422]: pgmap v1145: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.2 KiB/s wr, 5 op/s 2023-03-09T15:17:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1180.954010 secs 2023-03-09T15:17:52.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:52 smithi181 ceph-mon[94189]: pgmap v1146: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.2 KiB/s wr, 7 op/s 2023-03-09T15:17:52.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1183.146138 secs 2023-03-09T15:17:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:52 smithi038 ceph-mon[92422]: pgmap v1146: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.2 KiB/s wr, 7 op/s 2023-03-09T15:17:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1183.146138 secs 2023-03-09T15:17:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:52 smithi122 ceph-mon[93869]: pgmap v1146: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.2 KiB/s wr, 7 op/s 2023-03-09T15:17:52.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1183.146138 secs 2023-03-09T15:17:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1645.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:17:54.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:54 smithi122 ceph-mon[93869]: pgmap v1147: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T15:17:54.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:54 smithi181 ceph-mon[94189]: pgmap v1147: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T15:17:54.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:54 smithi038 ceph-mon[92422]: pgmap v1147: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T15:17:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1185.954070 secs 2023-03-09T15:17:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1185.954070 secs 2023-03-09T15:17:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1185.954070 secs 2023-03-09T15:17:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:56 smithi122 ceph-mon[93869]: pgmap v1148: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T15:17:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:56 smithi181 ceph-mon[94189]: pgmap v1148: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T15:17:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:56 smithi038 ceph-mon[92422]: pgmap v1148: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T15:17:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1188.146172 secs 2023-03-09T15:17:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1188.146172 secs 2023-03-09T15:17:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1188.146172 secs 2023-03-09T15:17:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:17:58 smithi122 ceph-mon[93869]: pgmap v1149: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T15:17:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:17:58 smithi181 ceph-mon[94189]: pgmap v1149: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T15:17:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:17:58 smithi038 ceph-mon[92422]: pgmap v1149: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T15:17:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1650.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:18:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:18:00 smithi181 ceph-mon[94189]: pgmap v1150: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 426 B/s wr, 5 op/s 2023-03-09T15:18:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:18:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1190.954129 secs 2023-03-09T15:18:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:18:00 smithi038 ceph-mon[92422]: pgmap v1150: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 426 B/s wr, 5 op/s 2023-03-09T15:18:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:18:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1190.954129 secs 2023-03-09T15:18:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:18:00 smithi122 ceph-mon[93869]: pgmap v1150: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 426 B/s wr, 5 op/s 2023-03-09T15:18:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:18:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1190.954129 secs 2023-03-09T15:18:02.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:18:02 smithi038 ceph-mon[92422]: pgmap v1151: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 29 KiB/s wr, 11 op/s 2023-03-09T15:18:02.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:18:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1193.146267 secs 2023-03-09T15:18:02.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:18:02 smithi122 ceph-mon[93869]: pgmap v1151: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 29 KiB/s wr, 11 op/s 2023-03-09T15:18:02.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:18:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1193.146267 secs 2023-03-09T15:18:02.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:18:02 smithi181 ceph-mon[94189]: pgmap v1151: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 29 KiB/s wr, 11 op/s 2023-03-09T15:18:02.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:18:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1193.146267 secs 2023-03-09T15:18:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1655.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:18:04.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:18:04 smithi122 ceph-mon[93869]: pgmap v1152: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 50 KiB/s wr, 21 op/s 2023-03-09T15:18:04.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:18:04 smithi181 ceph-mon[94189]: pgmap v1152: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 50 KiB/s wr, 21 op/s 2023-03-09T15:18:04.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:18:04 smithi038 ceph-mon[92422]: pgmap v1152: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 50 KiB/s wr, 21 op/s 2023-03-09T15:18:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:18:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1195.954190 secs 2023-03-09T15:18:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:18:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1195.954190 secs 2023-03-09T15:18:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:18:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1195.954190 secs 2023-03-09T15:18:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:18:06 smithi122 ceph-mon[93869]: pgmap v1153: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 50 KiB/s wr, 21 op/s 2023-03-09T15:18:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:18:06 smithi181 ceph-mon[94189]: pgmap v1153: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 50 KiB/s wr, 21 op/s 2023-03-09T15:18:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:18:06 smithi038 ceph-mon[92422]: pgmap v1153: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 50 KiB/s wr, 21 op/s 2023-03-09T15:18:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:18:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1198.146407 secs 2023-03-09T15:18:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:18:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1198.146407 secs 2023-03-09T15:18:07.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:18:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1198.146407 secs 2023-03-09T15:18:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:18:08 smithi122 ceph-mon[93869]: pgmap v1154: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 50 KiB/s wr, 24 op/s 2023-03-09T15:18:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:18:08 smithi181 ceph-mon[94189]: pgmap v1154: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 50 KiB/s wr, 24 op/s 2023-03-09T15:18:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:18:08 smithi038 ceph-mon[92422]: pgmap v1154: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 50 KiB/s wr, 24 op/s 2023-03-09T15:18:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1660.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:18:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:18:10 smithi122 ceph-mon[93869]: osdmap e286: 12 total, 12 up, 12 in 2023-03-09T15:18:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:18:10 smithi122 ceph-mon[93869]: pgmap v1156: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 60 KiB/s wr, 26 op/s 2023-03-09T15:18:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:18:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1200.954311 secs 2023-03-09T15:18:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:18:10 smithi181 ceph-mon[94189]: osdmap e286: 12 total, 12 up, 12 in 2023-03-09T15:18:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:18:10 smithi181 ceph-mon[94189]: pgmap v1156: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 60 KiB/s wr, 26 op/s 2023-03-09T15:18:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:18:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1200.954311 secs 2023-03-09T15:18:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:18:10 smithi038 ceph-mon[92422]: osdmap e286: 12 total, 12 up, 12 in 2023-03-09T15:18:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:18:10 smithi038 ceph-mon[92422]: pgmap v1156: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 60 KiB/s wr, 26 op/s 2023-03-09T15:18:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:18:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1200.954311 secs 2023-03-09T15:18:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:18:12 smithi038 ceph-mon[92422]: pgmap v1157: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 42 KiB/s wr, 22 op/s 2023-03-09T15:18:12.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:18:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1203.146521 secs 2023-03-09T15:18:12.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:18:12 smithi122 ceph-mon[93869]: pgmap v1157: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 42 KiB/s wr, 22 op/s 2023-03-09T15:18:12.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:18:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1203.146521 secs 2023-03-09T15:18:12.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:18:12 smithi181 ceph-mon[94189]: pgmap v1157: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 42 KiB/s wr, 22 op/s 2023-03-09T15:18:12.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:18:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1203.146521 secs 2023-03-09T15:18:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1665.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:18:14.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:18:14 smithi122 ceph-mon[93869]: pgmap v1158: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T15:18:14.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:18:14 smithi181 ceph-mon[94189]: pgmap v1158: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T15:18:14.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:18:14 smithi038 ceph-mon[92422]: pgmap v1158: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T15:18:15.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:18:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1205.954372 secs 2023-03-09T15:18:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:18:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1205.954372 secs 2023-03-09T15:18:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:18:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1205.954372 secs 2023-03-09T15:18:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:18:16 smithi122 ceph-mon[93869]: pgmap v1159: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T15:18:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:18:16 smithi181 ceph-mon[94189]: pgmap v1159: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T15:18:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:18:16 smithi038 ceph-mon[92422]: pgmap v1159: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T15:18:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:18:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1208.146646 secs 2023-03-09T15:18:17.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:18:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1208.146646 secs 2023-03-09T15:18:17.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:18:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1208.146646 secs 2023-03-09T15:18:18.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:18:18 smithi122 ceph-mon[93869]: pgmap v1160: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T15:18:18.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:18:18 smithi122 ceph-mon[93869]: osdmap e287: 12 total, 12 up, 12 in 2023-03-09T15:18:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:18:18 smithi181 ceph-mon[94189]: pgmap v1160: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T15:18:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:18:18 smithi181 ceph-mon[94189]: osdmap e287: 12 total, 12 up, 12 in 2023-03-09T15:18:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:18:18 smithi038 ceph-mon[92422]: pgmap v1160: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T15:18:18.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:18:18 smithi038 ceph-mon[92422]: osdmap e287: 12 total, 12 up, 12 in 2023-03-09T15:18:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1670.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:18:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:18:20 smithi038 ceph-mon[92422]: pgmap v1162: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 16 KiB/s wr, 6 op/s 2023-03-09T15:18:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:18:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1210.954454 secs 2023-03-09T15:18:20.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:18:20 smithi122 ceph-mon[93869]: pgmap v1162: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 16 KiB/s wr, 6 op/s 2023-03-09T15:18:20.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:18:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1210.954454 secs 2023-03-09T15:18:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:18:20 smithi181 ceph-mon[94189]: pgmap v1162: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 16 KiB/s wr, 6 op/s 2023-03-09T15:18:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:18:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1210.954454 secs 2023-03-09T15:18:22.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:18:22 smithi038 ceph-mon[92422]: pgmap v1163: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:18:22.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:18:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1213.146776 secs 2023-03-09T15:18:22.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:18:22 smithi122 ceph-mon[93869]: pgmap v1163: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:18:22.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:18:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1213.146776 secs 2023-03-09T15:18:22.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:18:22 smithi181 ceph-mon[94189]: pgmap v1163: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:18:22.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:18:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1213.146776 secs 2023-03-09T15:18:23.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1675.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:18:24.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:18:24 smithi122 ceph-mon[93869]: pgmap v1164: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:18:24.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:18:24 smithi181 ceph-mon[94189]: pgmap v1164: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:18:24.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:18:24 smithi038 ceph-mon[92422]: pgmap v1164: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:18:25.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:18:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1215.954516 secs 2023-03-09T15:18:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:18:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1215.954516 secs 2023-03-09T15:18:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:18:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1215.954516 secs 2023-03-09T15:18:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:18:26 smithi122 ceph-mon[93869]: pgmap v1165: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:18:26.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:18:26 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:18:26.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:18:26 smithi181 ceph-mon[94189]: pgmap v1165: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:18:26.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:18:26 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:18:26.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:18:26 smithi038 ceph-mon[92422]: pgmap v1165: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:18:26.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:18:26 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:18:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:18:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1218.146893 secs 2023-03-09T15:18:27.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:18:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1218.146893 secs 2023-03-09T15:18:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:18:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1218.146893 secs 2023-03-09T15:18:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:18:28 smithi122 ceph-mon[93869]: pgmap v1166: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:18:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:18:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:18:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:18:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:18:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:18:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:18:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:18:28 smithi181 ceph-mon[94189]: pgmap v1166: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:18:28.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:18:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:18:28.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:18:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:18:28.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:18:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:18:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:18:28 smithi038 ceph-mon[92422]: pgmap v1166: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:18:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:18:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:18:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:18:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:18:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:18:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:18:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1680.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:18:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:18:30 smithi038 ceph-mon[92422]: pgmap v1167: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 91 B/s wr, 5 op/s 2023-03-09T15:18:30.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:18:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1220.954578 secs 2023-03-09T15:18:30.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:18:30 smithi122 ceph-mon[93869]: pgmap v1167: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 91 B/s wr, 5 op/s 2023-03-09T15:18:30.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:18:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1220.954578 secs 2023-03-09T15:18:30.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:18:30 smithi181 ceph-mon[94189]: pgmap v1167: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 91 B/s wr, 5 op/s 2023-03-09T15:18:30.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:18:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1220.954578 secs 2023-03-09T15:18:32.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:18:32 smithi038 ceph-mon[92422]: pgmap v1168: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 7.7 KiB/s wr, 8 op/s 2023-03-09T15:18:32.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:18:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1223.147050 secs 2023-03-09T15:18:32.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:18:32 smithi122 ceph-mon[93869]: pgmap v1168: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 7.7 KiB/s wr, 8 op/s 2023-03-09T15:18:32.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:18:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1223.147050 secs 2023-03-09T15:18:32.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:18:32 smithi181 ceph-mon[94189]: pgmap v1168: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 7.7 KiB/s wr, 8 op/s 2023-03-09T15:18:32.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:18:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1223.147050 secs 2023-03-09T15:18:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1685.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:18:34.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:18:34 smithi122 ceph-mon[93869]: pgmap v1169: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 8.0 KiB/s wr, 6 op/s 2023-03-09T15:18:34.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:18:34 smithi181 ceph-mon[94189]: pgmap v1169: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 8.0 KiB/s wr, 6 op/s 2023-03-09T15:18:34.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:18:34 smithi038 ceph-mon[92422]: pgmap v1169: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 8.0 KiB/s wr, 6 op/s 2023-03-09T15:18:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:18:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1225.954636 secs 2023-03-09T15:18:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:18:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1225.954636 secs 2023-03-09T15:18:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:18:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1225.954636 secs 2023-03-09T15:18:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:18:36 smithi122 ceph-mon[93869]: pgmap v1170: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 8.0 KiB/s wr, 6 op/s 2023-03-09T15:18:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:18:36 smithi181 ceph-mon[94189]: pgmap v1170: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 8.0 KiB/s wr, 6 op/s 2023-03-09T15:18:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:18:36 smithi038 ceph-mon[92422]: pgmap v1170: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 8.0 KiB/s wr, 6 op/s 2023-03-09T15:18:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:18:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1228.147158 secs 2023-03-09T15:18:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:18:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1228.147158 secs 2023-03-09T15:18:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:18:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1228.147158 secs 2023-03-09T15:18:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:18:38 smithi122 ceph-mon[93869]: pgmap v1171: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 28 KiB/s wr, 10 op/s 2023-03-09T15:18:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:18:38 smithi181 ceph-mon[94189]: pgmap v1171: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 28 KiB/s wr, 10 op/s 2023-03-09T15:18:38.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:18:38 smithi038 ceph-mon[92422]: pgmap v1171: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 28 KiB/s wr, 10 op/s 2023-03-09T15:18:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1690.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:18:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:18:40 smithi038 ceph-mon[92422]: pgmap v1172: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T15:18:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:18:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1230.954696 secs 2023-03-09T15:18:40.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:18:40 smithi122 ceph-mon[93869]: pgmap v1172: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T15:18:40.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:18:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1230.954696 secs 2023-03-09T15:18:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:18:40 smithi181 ceph-mon[94189]: pgmap v1172: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T15:18:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:18:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1230.954696 secs 2023-03-09T15:18:42.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:18:42 smithi038 ceph-mon[92422]: pgmap v1173: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 36 KiB/s wr, 10 op/s 2023-03-09T15:18:42.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:18:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1233.147276 secs 2023-03-09T15:18:42.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:18:42 smithi122 ceph-mon[93869]: pgmap v1173: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 36 KiB/s wr, 10 op/s 2023-03-09T15:18:42.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:18:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1233.147276 secs 2023-03-09T15:18:42.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:18:42 smithi181 ceph-mon[94189]: pgmap v1173: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 36 KiB/s wr, 10 op/s 2023-03-09T15:18:42.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:18:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1233.147276 secs 2023-03-09T15:18:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1695.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:18:44.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:18:44 smithi122 ceph-mon[93869]: pgmap v1174: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 29 KiB/s wr, 7 op/s 2023-03-09T15:18:44.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:18:44 smithi181 ceph-mon[94189]: pgmap v1174: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 29 KiB/s wr, 7 op/s 2023-03-09T15:18:44.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:18:44 smithi038 ceph-mon[92422]: pgmap v1174: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 29 KiB/s wr, 7 op/s 2023-03-09T15:18:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:18:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1235.954754 secs 2023-03-09T15:18:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:18:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1235.954754 secs 2023-03-09T15:18:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:18:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1235.954754 secs 2023-03-09T15:18:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:18:46 smithi122 ceph-mon[93869]: pgmap v1175: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 29 KiB/s wr, 6 op/s 2023-03-09T15:18:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:18:46 smithi181 ceph-mon[94189]: pgmap v1175: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 29 KiB/s wr, 6 op/s 2023-03-09T15:18:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:18:46 smithi038 ceph-mon[92422]: pgmap v1175: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 29 KiB/s wr, 6 op/s 2023-03-09T15:18:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:18:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1238.147415 secs 2023-03-09T15:18:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:18:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1238.147415 secs 2023-03-09T15:18:47.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:18:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1238.147415 secs 2023-03-09T15:18:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:18:48 smithi122 ceph-mon[93869]: pgmap v1176: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 29 KiB/s wr, 8 op/s 2023-03-09T15:18:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:18:48 smithi181 ceph-mon[94189]: pgmap v1176: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 29 KiB/s wr, 8 op/s 2023-03-09T15:18:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:18:48 smithi038 ceph-mon[92422]: pgmap v1176: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 29 KiB/s wr, 8 op/s 2023-03-09T15:18:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1700.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:18:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:18:50 smithi038 ceph-mon[92422]: pgmap v1177: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.5 KiB/s wr, 5 op/s 2023-03-09T15:18:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:18:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1240.954837 secs 2023-03-09T15:18:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:18:50 smithi122 ceph-mon[93869]: pgmap v1177: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.5 KiB/s wr, 5 op/s 2023-03-09T15:18:50.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:18:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1240.954837 secs 2023-03-09T15:18:50.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:18:50 smithi181 ceph-mon[94189]: pgmap v1177: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.5 KiB/s wr, 5 op/s 2023-03-09T15:18:50.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:18:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1240.954837 secs 2023-03-09T15:18:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:18:52 smithi038 ceph-mon[92422]: pgmap v1178: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.5 KiB/s wr, 7 op/s 2023-03-09T15:18:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:18:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1243.147534 secs 2023-03-09T15:18:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:18:52 smithi122 ceph-mon[93869]: pgmap v1178: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.5 KiB/s wr, 7 op/s 2023-03-09T15:18:52.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:18:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1243.147534 secs 2023-03-09T15:18:52.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:18:52 smithi181 ceph-mon[94189]: pgmap v1178: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.5 KiB/s wr, 7 op/s 2023-03-09T15:18:52.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:18:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1243.147534 secs 2023-03-09T15:18:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1705.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:18:54.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:18:54 smithi122 ceph-mon[93869]: pgmap v1179: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T15:18:54.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:18:54 smithi181 ceph-mon[94189]: pgmap v1179: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T15:18:54.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:18:54 smithi038 ceph-mon[92422]: pgmap v1179: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T15:18:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:18:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1245.954897 secs 2023-03-09T15:18:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:18:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1245.954897 secs 2023-03-09T15:18:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:18:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1245.954897 secs 2023-03-09T15:18:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:18:56 smithi122 ceph-mon[93869]: pgmap v1180: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T15:18:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:18:56 smithi181 ceph-mon[94189]: pgmap v1180: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T15:18:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:18:56 smithi038 ceph-mon[92422]: pgmap v1180: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T15:18:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:18:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1248.147674 secs 2023-03-09T15:18:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:18:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1248.147674 secs 2023-03-09T15:18:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:18:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1248.147674 secs 2023-03-09T15:18:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:18:58 smithi122 ceph-mon[93869]: pgmap v1181: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T15:18:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:18:58 smithi181 ceph-mon[94189]: pgmap v1181: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T15:18:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:18:58 smithi038 ceph-mon[92422]: pgmap v1181: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T15:18:58.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1710.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:19:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:19:00 smithi038 ceph-mon[92422]: pgmap v1182: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T15:19:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:19:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1250.954959 secs 2023-03-09T15:19:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:19:00 smithi122 ceph-mon[93869]: pgmap v1182: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T15:19:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:19:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1250.954959 secs 2023-03-09T15:19:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:19:00 smithi181 ceph-mon[94189]: pgmap v1182: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T15:19:00.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:19:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1250.954959 secs 2023-03-09T15:19:02.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:19:02 smithi122 ceph-mon[93869]: pgmap v1183: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 29 KiB/s wr, 11 op/s 2023-03-09T15:19:02.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:19:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1253.147786 secs 2023-03-09T15:19:02.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:19:02 smithi181 ceph-mon[94189]: pgmap v1183: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 29 KiB/s wr, 11 op/s 2023-03-09T15:19:02.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:19:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1253.147786 secs 2023-03-09T15:19:02.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:19:02 smithi038 ceph-mon[92422]: pgmap v1183: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 29 KiB/s wr, 11 op/s 2023-03-09T15:19:03.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:19:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1253.147786 secs 2023-03-09T15:19:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1715.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:19:04.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:19:04 smithi122 ceph-mon[93869]: pgmap v1184: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 50 KiB/s wr, 21 op/s 2023-03-09T15:19:04.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:19:04 smithi181 ceph-mon[94189]: pgmap v1184: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 50 KiB/s wr, 21 op/s 2023-03-09T15:19:04.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:19:04 smithi038 ceph-mon[92422]: pgmap v1184: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 50 KiB/s wr, 21 op/s 2023-03-09T15:19:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:19:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1255.954983 secs 2023-03-09T15:19:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:19:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1255.954983 secs 2023-03-09T15:19:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:19:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1255.954983 secs 2023-03-09T15:19:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:19:06 smithi122 ceph-mon[93869]: pgmap v1185: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 50 KiB/s wr, 21 op/s 2023-03-09T15:19:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:19:06 smithi181 ceph-mon[94189]: pgmap v1185: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 50 KiB/s wr, 21 op/s 2023-03-09T15:19:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:19:06 smithi038 ceph-mon[92422]: pgmap v1185: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 50 KiB/s wr, 21 op/s 2023-03-09T15:19:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:19:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1258.147907 secs 2023-03-09T15:19:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:19:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1258.147907 secs 2023-03-09T15:19:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:19:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1258.147907 secs 2023-03-09T15:19:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:19:08 smithi122 ceph-mon[93869]: pgmap v1186: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 50 KiB/s wr, 24 op/s 2023-03-09T15:19:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:19:08 smithi181 ceph-mon[94189]: pgmap v1186: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 50 KiB/s wr, 24 op/s 2023-03-09T15:19:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:19:08 smithi038 ceph-mon[92422]: pgmap v1186: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 50 KiB/s wr, 24 op/s 2023-03-09T15:19:08.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1720.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:19:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:19:10 smithi122 ceph-mon[93869]: osdmap e288: 12 total, 12 up, 12 in 2023-03-09T15:19:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:19:10 smithi122 ceph-mon[93869]: pgmap v1188: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 61 KiB/s wr, 26 op/s 2023-03-09T15:19:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:19:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1260.955041 secs 2023-03-09T15:19:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:19:10 smithi181 ceph-mon[94189]: osdmap e288: 12 total, 12 up, 12 in 2023-03-09T15:19:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:19:10 smithi181 ceph-mon[94189]: pgmap v1188: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 61 KiB/s wr, 26 op/s 2023-03-09T15:19:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:19:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1260.955041 secs 2023-03-09T15:19:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:19:10 smithi038 ceph-mon[92422]: osdmap e288: 12 total, 12 up, 12 in 2023-03-09T15:19:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:19:10 smithi038 ceph-mon[92422]: pgmap v1188: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 61 KiB/s wr, 26 op/s 2023-03-09T15:19:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:19:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1260.955041 secs 2023-03-09T15:19:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:19:12 smithi122 ceph-mon[93869]: pgmap v1189: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 42 KiB/s wr, 22 op/s 2023-03-09T15:19:12.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:19:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1263.147988 secs 2023-03-09T15:19:12.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:19:12 smithi181 ceph-mon[94189]: pgmap v1189: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 42 KiB/s wr, 22 op/s 2023-03-09T15:19:12.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:19:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1263.147988 secs 2023-03-09T15:19:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:19:12 smithi038 ceph-mon[92422]: pgmap v1189: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 42 KiB/s wr, 22 op/s 2023-03-09T15:19:13.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:19:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1263.147988 secs 2023-03-09T15:19:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1725.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:19:14.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:19:14 smithi122 ceph-mon[93869]: pgmap v1190: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T15:19:14.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:19:14 smithi181 ceph-mon[94189]: pgmap v1190: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T15:19:14.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:19:14 smithi038 ceph-mon[92422]: pgmap v1190: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T15:19:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:19:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1265.955100 secs 2023-03-09T15:19:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:19:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1265.955100 secs 2023-03-09T15:19:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:19:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1265.955100 secs 2023-03-09T15:19:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:19:16 smithi122 ceph-mon[93869]: pgmap v1191: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T15:19:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:19:16 smithi181 ceph-mon[94189]: pgmap v1191: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T15:19:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:19:16 smithi038 ceph-mon[92422]: pgmap v1191: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T15:19:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:19:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1268.148136 secs 2023-03-09T15:19:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:19:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1268.148136 secs 2023-03-09T15:19:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:19:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1268.148136 secs 2023-03-09T15:19:18.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:19:18 smithi122 ceph-mon[93869]: pgmap v1192: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T15:19:18.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:19:18 smithi122 ceph-mon[93869]: osdmap e289: 12 total, 12 up, 12 in 2023-03-09T15:19:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:19:18 smithi181 ceph-mon[94189]: pgmap v1192: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T15:19:18.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:19:18 smithi181 ceph-mon[94189]: osdmap e289: 12 total, 12 up, 12 in 2023-03-09T15:19:18.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:19:18 smithi038 ceph-mon[92422]: pgmap v1192: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T15:19:18.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:19:18 smithi038 ceph-mon[92422]: osdmap e289: 12 total, 12 up, 12 in 2023-03-09T15:19:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1730.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:19:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:19:20 smithi122 ceph-mon[93869]: pgmap v1194: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 16 KiB/s wr, 6 op/s 2023-03-09T15:19:20.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:19:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1270.955159 secs 2023-03-09T15:19:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:19:20 smithi181 ceph-mon[94189]: pgmap v1194: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 16 KiB/s wr, 6 op/s 2023-03-09T15:19:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:19:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1270.955159 secs 2023-03-09T15:19:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:19:20 smithi038 ceph-mon[92422]: pgmap v1194: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 16 KiB/s wr, 6 op/s 2023-03-09T15:19:21.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:19:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1270.955159 secs 2023-03-09T15:19:22.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:19:22 smithi122 ceph-mon[93869]: pgmap v1195: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:19:22.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:19:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1273.148261 secs 2023-03-09T15:19:22.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:19:22 smithi181 ceph-mon[94189]: pgmap v1195: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:19:22.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:19:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1273.148261 secs 2023-03-09T15:19:22.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:19:22 smithi038 ceph-mon[92422]: pgmap v1195: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:19:23.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:19:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1273.148261 secs 2023-03-09T15:19:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1735.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:19:24.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:19:24 smithi122 ceph-mon[93869]: pgmap v1196: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:19:24.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:19:24 smithi181 ceph-mon[94189]: pgmap v1196: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:19:24.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:19:24 smithi038 ceph-mon[92422]: pgmap v1196: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:19:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:19:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1275.955219 secs 2023-03-09T15:19:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:19:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1275.955219 secs 2023-03-09T15:19:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:19:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1275.955219 secs 2023-03-09T15:19:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:19:26 smithi122 ceph-mon[93869]: pgmap v1197: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:19:26.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:19:26 smithi181 ceph-mon[94189]: pgmap v1197: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:19:26.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:19:26 smithi038 ceph-mon[92422]: pgmap v1197: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:19:27.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:19:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1278.148380 secs 2023-03-09T15:19:27.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:19:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1278.148380 secs 2023-03-09T15:19:27.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:19:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1278.148380 secs 2023-03-09T15:19:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:19:28 smithi122 ceph-mon[93869]: pgmap v1198: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:19:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:19:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:19:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:19:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:19:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:19:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:19:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:19:28 smithi181 ceph-mon[94189]: pgmap v1198: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:19:28.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:19:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:19:28.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:19:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:19:28.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:19:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:19:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:19:28 smithi038 ceph-mon[92422]: pgmap v1198: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:19:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:19:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:19:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:19:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:19:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:19:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:19:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1740.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:19:29.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:19:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:19:29.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:19:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:19:29.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:19:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:19:29.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:19:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:19:29.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:19:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:19:29.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:19:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:19:29.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:19:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:19:29.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:19:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:19:29.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:19:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:19:30.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:19:30 smithi122 ceph-mon[93869]: pgmap v1199: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 91 B/s wr, 5 op/s 2023-03-09T15:19:30.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:19:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1280.955282 secs 2023-03-09T15:19:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:19:30 smithi181 ceph-mon[94189]: pgmap v1199: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 91 B/s wr, 5 op/s 2023-03-09T15:19:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:19:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1280.955282 secs 2023-03-09T15:19:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:19:30 smithi038 ceph-mon[92422]: pgmap v1199: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 91 B/s wr, 5 op/s 2023-03-09T15:19:30.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:19:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1280.955282 secs 2023-03-09T15:19:32.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:19:32 smithi122 ceph-mon[93869]: pgmap v1200: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 8.0 KiB/s wr, 8 op/s 2023-03-09T15:19:32.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:19:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1283.148503 secs 2023-03-09T15:19:32.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:19:32 smithi181 ceph-mon[94189]: pgmap v1200: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 8.0 KiB/s wr, 8 op/s 2023-03-09T15:19:32.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:19:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1283.148503 secs 2023-03-09T15:19:33.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:19:32 smithi038 ceph-mon[92422]: pgmap v1200: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 8.0 KiB/s wr, 8 op/s 2023-03-09T15:19:33.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:19:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1283.148503 secs 2023-03-09T15:19:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1745.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:19:34.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:19:34 smithi122 ceph-mon[93869]: pgmap v1201: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 10 KiB/s wr, 7 op/s 2023-03-09T15:19:34.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:19:34 smithi181 ceph-mon[94189]: pgmap v1201: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 10 KiB/s wr, 7 op/s 2023-03-09T15:19:34.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:19:34 smithi038 ceph-mon[92422]: pgmap v1201: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 10 KiB/s wr, 7 op/s 2023-03-09T15:19:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:19:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1285.955343 secs 2023-03-09T15:19:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:19:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1285.955343 secs 2023-03-09T15:19:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:19:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1285.955343 secs 2023-03-09T15:19:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:19:36 smithi122 ceph-mon[93869]: pgmap v1202: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 13 KiB/s wr, 7 op/s 2023-03-09T15:19:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:19:36 smithi181 ceph-mon[94189]: pgmap v1202: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 13 KiB/s wr, 7 op/s 2023-03-09T15:19:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:19:36 smithi038 ceph-mon[92422]: pgmap v1202: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 13 KiB/s wr, 7 op/s 2023-03-09T15:19:38.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:19:38 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:19:38.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:19:38 smithi122 ceph-mon[93869]: pgmap v1203: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 41 KiB/s wr, 11 op/s 2023-03-09T15:19:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:19:38 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:19:38.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:19:38 smithi181 ceph-mon[94189]: pgmap v1203: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 41 KiB/s wr, 11 op/s 2023-03-09T15:19:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:19:38 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:19:38.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:19:38 smithi038 ceph-mon[92422]: pgmap v1203: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 41 KiB/s wr, 11 op/s 2023-03-09T15:19:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1750.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:19:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:19:39 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:19:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:19:39 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:19:39.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:19:39 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:19:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:19:40 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:19:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:19:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:19:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:19:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:19:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:19:40 smithi122 ceph-mon[93869]: pgmap v1204: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 41 KiB/s wr, 9 op/s 2023-03-09T15:19:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:19:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1290.955403 secs 2023-03-09T15:19:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:19:40 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:19:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:19:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:19:40.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:19:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:19:40.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:19:40 smithi181 ceph-mon[94189]: pgmap v1204: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 41 KiB/s wr, 9 op/s 2023-03-09T15:19:40.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:19:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1290.955403 secs 2023-03-09T15:19:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:19:40 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:19:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:19:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:19:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:19:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:19:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:19:40 smithi038 ceph-mon[92422]: pgmap v1204: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 41 KiB/s wr, 9 op/s 2023-03-09T15:19:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:19:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1290.955403 secs 2023-03-09T15:19:42.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:19:42 smithi122 ceph-mon[93869]: pgmap v1205: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 72 KiB/s wr, 13 op/s 2023-03-09T15:19:42.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:19:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1293.148744 secs 2023-03-09T15:19:42.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:19:42 smithi181 ceph-mon[94189]: pgmap v1205: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 72 KiB/s wr, 13 op/s 2023-03-09T15:19:42.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:19:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1293.148744 secs 2023-03-09T15:19:42.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:19:42 smithi038 ceph-mon[92422]: pgmap v1205: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 72 KiB/s wr, 13 op/s 2023-03-09T15:19:42.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:19:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1293.148744 secs 2023-03-09T15:19:43.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1755.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:19:44.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:19:44 smithi122 ceph-mon[93869]: pgmap v1206: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 64 KiB/s wr, 10 op/s 2023-03-09T15:19:44.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:19:44 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:19:44.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:19:44 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:19:44.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:19:44 smithi181 ceph-mon[94189]: pgmap v1206: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 64 KiB/s wr, 10 op/s 2023-03-09T15:19:44.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:19:44 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:19:44.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:19:44 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:19:45.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:19:44 smithi038 ceph-mon[92422]: pgmap v1206: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 64 KiB/s wr, 10 op/s 2023-03-09T15:19:45.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:19:44 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:19:45.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:19:44 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:19:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:19:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1295.955466 secs 2023-03-09T15:19:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:19:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1295.955466 secs 2023-03-09T15:19:46.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:19:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1295.955466 secs 2023-03-09T15:19:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:19:46 smithi122 ceph-mon[93869]: pgmap v1207: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 62 KiB/s wr, 9 op/s 2023-03-09T15:19:46.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:19:46 smithi181 ceph-mon[94189]: pgmap v1207: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 62 KiB/s wr, 9 op/s 2023-03-09T15:19:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:19:46 smithi038 ceph-mon[92422]: pgmap v1207: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 62 KiB/s wr, 9 op/s 2023-03-09T15:19:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:19:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1298.148895 secs 2023-03-09T15:19:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:19:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1298.148895 secs 2023-03-09T15:19:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:19:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1298.148895 secs 2023-03-09T15:19:48.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1760.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:19:48.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:19:48 smithi122 ceph-mon[93869]: pgmap v1208: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 59 KiB/s wr, 11 op/s 2023-03-09T15:19:48.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:19:48 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:19:48.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:19:48 smithi181 ceph-mon[94189]: pgmap v1208: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 59 KiB/s wr, 11 op/s 2023-03-09T15:19:48.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:19:48 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:19:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:19:48 smithi038 ceph-mon[92422]: pgmap v1208: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 59 KiB/s wr, 11 op/s 2023-03-09T15:19:49.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:19:48 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:19:50.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:19:50 smithi122 ceph-mon[93869]: pgmap v1209: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 31 KiB/s wr, 7 op/s 2023-03-09T15:19:50.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:19:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1300.955525 secs 2023-03-09T15:19:50.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:19:50 smithi181 ceph-mon[94189]: pgmap v1209: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 31 KiB/s wr, 7 op/s 2023-03-09T15:19:50.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:19:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1300.955525 secs 2023-03-09T15:19:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:19:50 smithi038 ceph-mon[92422]: pgmap v1209: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 31 KiB/s wr, 7 op/s 2023-03-09T15:19:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:19:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1300.955525 secs 2023-03-09T15:19:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:19:52 smithi122 ceph-mon[93869]: pgmap v1210: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 36 KiB/s wr, 10 op/s 2023-03-09T15:19:52.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:19:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1303.149019 secs 2023-03-09T15:19:52.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:19:52 smithi181 ceph-mon[94189]: pgmap v1210: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 36 KiB/s wr, 10 op/s 2023-03-09T15:19:52.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:19:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1303.149019 secs 2023-03-09T15:19:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:19:52 smithi038 ceph-mon[92422]: pgmap v1210: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 36 KiB/s wr, 10 op/s 2023-03-09T15:19:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:19:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1303.149019 secs 2023-03-09T15:19:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1765.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:19:54.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:19:54 smithi122 ceph-mon[93869]: pgmap v1211: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 5.2 KiB/s wr, 5 op/s 2023-03-09T15:19:54.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:19:54 smithi181 ceph-mon[94189]: pgmap v1211: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 5.2 KiB/s wr, 5 op/s 2023-03-09T15:19:54.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:19:54 smithi038 ceph-mon[92422]: pgmap v1211: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 5.2 KiB/s wr, 5 op/s 2023-03-09T15:19:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:19:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1305.955583 secs 2023-03-09T15:19:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:19:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1305.955583 secs 2023-03-09T15:19:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:19:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1305.955583 secs 2023-03-09T15:19:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:19:56 smithi122 ceph-mon[93869]: pgmap v1212: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 5.2 KiB/s wr, 5 op/s 2023-03-09T15:19:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:19:56 smithi181 ceph-mon[94189]: pgmap v1212: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 5.2 KiB/s wr, 5 op/s 2023-03-09T15:19:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:19:56 smithi038 ceph-mon[92422]: pgmap v1212: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 5.2 KiB/s wr, 5 op/s 2023-03-09T15:19:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:19:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1308.149138 secs 2023-03-09T15:19:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:19:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1308.149138 secs 2023-03-09T15:19:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:19:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1308.149138 secs 2023-03-09T15:19:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:19:58 smithi122 ceph-mon[93869]: pgmap v1213: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 5.2 KiB/s wr, 7 op/s 2023-03-09T15:19:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:19:58 smithi181 ceph-mon[94189]: pgmap v1213: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 5.2 KiB/s wr, 7 op/s 2023-03-09T15:19:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:19:58 smithi038 ceph-mon[92422]: pgmap v1213: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 5.2 KiB/s wr, 7 op/s 2023-03-09T15:19:58.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1770.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:20:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:20:00 smithi122 ceph-mon[93869]: pgmap v1214: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 5.6 KiB/s wr, 5 op/s 2023-03-09T15:20:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:20:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1310.955646 secs 2023-03-09T15:20:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:20:00 smithi122 ceph-mon[93869]: overall HEALTH_WARN 2 MDSs report slow requests 2023-03-09T15:20:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:20:00 smithi181 ceph-mon[94189]: pgmap v1214: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 5.6 KiB/s wr, 5 op/s 2023-03-09T15:20:00.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:20:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1310.955646 secs 2023-03-09T15:20:00.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:20:00 smithi181 ceph-mon[94189]: overall HEALTH_WARN 2 MDSs report slow requests 2023-03-09T15:20:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:20:00 smithi038 ceph-mon[92422]: pgmap v1214: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 5.6 KiB/s wr, 5 op/s 2023-03-09T15:20:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:20:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1310.955646 secs 2023-03-09T15:20:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:20:00 smithi038 ceph-mon[92422]: overall HEALTH_WARN 2 MDSs report slow requests 2023-03-09T15:20:02.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:20:02 smithi122 ceph-mon[93869]: pgmap v1215: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 34 KiB/s wr, 11 op/s 2023-03-09T15:20:02.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:20:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1313.149291 secs 2023-03-09T15:20:02.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:20:02 smithi181 ceph-mon[94189]: pgmap v1215: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 34 KiB/s wr, 11 op/s 2023-03-09T15:20:02.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:20:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1313.149291 secs 2023-03-09T15:20:02.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:20:02 smithi038 ceph-mon[92422]: pgmap v1215: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 34 KiB/s wr, 11 op/s 2023-03-09T15:20:03.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:20:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1313.149291 secs 2023-03-09T15:20:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1775.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:20:04.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:20:04 smithi122 ceph-mon[93869]: pgmap v1216: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 50 KiB/s wr, 21 op/s 2023-03-09T15:20:04.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:20:04 smithi181 ceph-mon[94189]: pgmap v1216: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 50 KiB/s wr, 21 op/s 2023-03-09T15:20:04.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:20:04 smithi038 ceph-mon[92422]: pgmap v1216: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 50 KiB/s wr, 21 op/s 2023-03-09T15:20:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:20:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1315.955708 secs 2023-03-09T15:20:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:20:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1315.955708 secs 2023-03-09T15:20:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:20:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1315.955708 secs 2023-03-09T15:20:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:20:06 smithi122 ceph-mon[93869]: pgmap v1217: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 50 KiB/s wr, 21 op/s 2023-03-09T15:20:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:20:06 smithi181 ceph-mon[94189]: pgmap v1217: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 50 KiB/s wr, 21 op/s 2023-03-09T15:20:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:20:06 smithi038 ceph-mon[92422]: pgmap v1217: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 50 KiB/s wr, 21 op/s 2023-03-09T15:20:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:20:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1318.149431 secs 2023-03-09T15:20:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:20:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1318.149431 secs 2023-03-09T15:20:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:20:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1318.149431 secs 2023-03-09T15:20:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:20:08 smithi122 ceph-mon[93869]: pgmap v1218: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 50 KiB/s wr, 24 op/s 2023-03-09T15:20:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:20:08 smithi181 ceph-mon[94189]: pgmap v1218: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 50 KiB/s wr, 24 op/s 2023-03-09T15:20:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:20:08 smithi038 ceph-mon[92422]: pgmap v1218: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 50 KiB/s wr, 24 op/s 2023-03-09T15:20:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1780.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:20:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:20:10 smithi122 ceph-mon[93869]: osdmap e290: 12 total, 12 up, 12 in 2023-03-09T15:20:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:20:10 smithi122 ceph-mon[93869]: pgmap v1220: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 60 KiB/s wr, 25 op/s 2023-03-09T15:20:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:20:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1320.955770 secs 2023-03-09T15:20:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:20:10 smithi181 ceph-mon[94189]: osdmap e290: 12 total, 12 up, 12 in 2023-03-09T15:20:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:20:10 smithi181 ceph-mon[94189]: pgmap v1220: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 60 KiB/s wr, 25 op/s 2023-03-09T15:20:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:20:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1320.955770 secs 2023-03-09T15:20:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:20:10 smithi038 ceph-mon[92422]: osdmap e290: 12 total, 12 up, 12 in 2023-03-09T15:20:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:20:10 smithi038 ceph-mon[92422]: pgmap v1220: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 60 KiB/s wr, 25 op/s 2023-03-09T15:20:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:20:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1320.955770 secs 2023-03-09T15:20:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:20:12 smithi122 ceph-mon[93869]: pgmap v1221: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 43 KiB/s wr, 22 op/s 2023-03-09T15:20:12.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:20:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1323.149556 secs 2023-03-09T15:20:12.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:20:12 smithi181 ceph-mon[94189]: pgmap v1221: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 43 KiB/s wr, 22 op/s 2023-03-09T15:20:12.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:20:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1323.149556 secs 2023-03-09T15:20:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:20:12 smithi038 ceph-mon[92422]: pgmap v1221: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 43 KiB/s wr, 22 op/s 2023-03-09T15:20:13.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:20:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1323.149556 secs 2023-03-09T15:20:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1785.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:20:14.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:20:14 smithi122 ceph-mon[93869]: pgmap v1222: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 KiB/s wr, 7 op/s 2023-03-09T15:20:14.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:20:14 smithi181 ceph-mon[94189]: pgmap v1222: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 KiB/s wr, 7 op/s 2023-03-09T15:20:14.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:20:14 smithi038 ceph-mon[92422]: pgmap v1222: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 KiB/s wr, 7 op/s 2023-03-09T15:20:15.634 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:20:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1325.955832 secs 2023-03-09T15:20:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:20:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1325.955832 secs 2023-03-09T15:20:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:20:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1325.955832 secs 2023-03-09T15:20:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:20:16 smithi122 ceph-mon[93869]: pgmap v1223: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 KiB/s wr, 7 op/s 2023-03-09T15:20:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:20:16 smithi181 ceph-mon[94189]: pgmap v1223: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 KiB/s wr, 7 op/s 2023-03-09T15:20:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:20:16 smithi038 ceph-mon[92422]: pgmap v1223: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 KiB/s wr, 7 op/s 2023-03-09T15:20:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:20:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1328.149671 secs 2023-03-09T15:20:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:20:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1328.149671 secs 2023-03-09T15:20:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:20:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1328.149671 secs 2023-03-09T15:20:18.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:20:18 smithi122 ceph-mon[93869]: pgmap v1224: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 KiB/s wr, 7 op/s 2023-03-09T15:20:18.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:20:18 smithi122 ceph-mon[93869]: osdmap e291: 12 total, 12 up, 12 in 2023-03-09T15:20:18.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:20:18 smithi181 ceph-mon[94189]: pgmap v1224: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 KiB/s wr, 7 op/s 2023-03-09T15:20:18.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:20:18 smithi181 ceph-mon[94189]: osdmap e291: 12 total, 12 up, 12 in 2023-03-09T15:20:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:20:18 smithi038 ceph-mon[92422]: pgmap v1224: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 KiB/s wr, 7 op/s 2023-03-09T15:20:18.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:20:18 smithi038 ceph-mon[92422]: osdmap e291: 12 total, 12 up, 12 in 2023-03-09T15:20:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1790.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:20:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:20:20 smithi122 ceph-mon[93869]: pgmap v1226: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 KiB/s wr, 7 op/s 2023-03-09T15:20:20.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:20:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1330.955893 secs 2023-03-09T15:20:20.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:20:20 smithi181 ceph-mon[94189]: pgmap v1226: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 KiB/s wr, 7 op/s 2023-03-09T15:20:20.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:20:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1330.955893 secs 2023-03-09T15:20:21.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:20:20 smithi038 ceph-mon[92422]: pgmap v1226: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 KiB/s wr, 7 op/s 2023-03-09T15:20:21.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:20:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1330.955893 secs 2023-03-09T15:20:22.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:20:22 smithi122 ceph-mon[93869]: pgmap v1227: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 307 B/s wr, 6 op/s 2023-03-09T15:20:22.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:20:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1333.149811 secs 2023-03-09T15:20:22.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:20:22 smithi181 ceph-mon[94189]: pgmap v1227: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 307 B/s wr, 6 op/s 2023-03-09T15:20:22.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:20:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1333.149811 secs 2023-03-09T15:20:22.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:20:22 smithi038 ceph-mon[92422]: pgmap v1227: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 307 B/s wr, 6 op/s 2023-03-09T15:20:23.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:20:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1333.149811 secs 2023-03-09T15:20:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1795.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:20:24.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:20:24 smithi122 ceph-mon[93869]: pgmap v1228: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 307 B/s wr, 6 op/s 2023-03-09T15:20:24.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:20:24 smithi181 ceph-mon[94189]: pgmap v1228: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 307 B/s wr, 6 op/s 2023-03-09T15:20:24.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:20:24 smithi038 ceph-mon[92422]: pgmap v1228: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 307 B/s wr, 6 op/s 2023-03-09T15:20:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:20:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1335.955954 secs 2023-03-09T15:20:25.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:20:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1335.955954 secs 2023-03-09T15:20:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:20:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1335.955954 secs 2023-03-09T15:20:26.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:20:26 smithi122 ceph-mon[93869]: pgmap v1229: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 307 B/s wr, 6 op/s 2023-03-09T15:20:26.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:20:26 smithi181 ceph-mon[94189]: pgmap v1229: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 307 B/s wr, 6 op/s 2023-03-09T15:20:26.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:20:26 smithi038 ceph-mon[92422]: pgmap v1229: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 307 B/s wr, 6 op/s 2023-03-09T15:20:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:20:27 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 1338.149925 secs 2023-03-09T15:20:27.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:20:27 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 1338.149925 secs 2023-03-09T15:20:27.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:20:27 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 1338.149925 secs 2023-03-09T15:20:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:20:28 smithi122 ceph-mon[93869]: pgmap v1230: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 102 B/s wr, 5 op/s 2023-03-09T15:20:28.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:20:28 smithi181 ceph-mon[94189]: pgmap v1230: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 102 B/s wr, 5 op/s 2023-03-09T15:20:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:20:28 smithi038 ceph-mon[92422]: pgmap v1230: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 102 B/s wr, 5 op/s 2023-03-09T15:20:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1800.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:20:29.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:20:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:20:29.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:20:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:20:29.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:20:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:20:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:20:30 smithi122 ceph-mon[93869]: pgmap v1231: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 91 B/s wr, 5 op/s 2023-03-09T15:20:30.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:20:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1340.955984 secs 2023-03-09T15:20:30.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:20:30 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:20:30.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:20:30 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:20:30.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:20:30 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:20:30.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:20:30 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:20:30.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:20:30 smithi181 ceph-mon[94189]: pgmap v1231: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 91 B/s wr, 5 op/s 2023-03-09T15:20:30.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:20:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1340.955984 secs 2023-03-09T15:20:30.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:20:30 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:20:30.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:20:30 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:20:30.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:20:30 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:20:30.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:20:30 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:20:30.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:20:30 smithi038 ceph-mon[92422]: pgmap v1231: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 91 B/s wr, 5 op/s 2023-03-09T15:20:30.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:20:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1340.955984 secs 2023-03-09T15:20:31.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:20:30 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:20:31.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:20:30 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:20:31.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:20:30 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:20:31.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:20:30 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:20:31.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:20:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:20:31.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:20:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:20:31.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:20:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:20:31.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:20:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:20:31.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:20:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:20:31.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:20:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:20:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:20:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:20:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:20:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:20:32.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:20:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:20:32.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:20:32 smithi122 ceph-mon[93869]: pgmap v1232: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 8 op/s 2023-03-09T15:20:32.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:20:32 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:20:32.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:20:32 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:20:32.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:20:32 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 1343.150015 secs 2023-03-09T15:20:32.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:20:32 smithi181 ceph-mon[94189]: pgmap v1232: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 8 op/s 2023-03-09T15:20:32.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:20:32 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:20:32.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:20:32 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:20:32.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:20:32 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 1343.150015 secs 2023-03-09T15:20:32.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:20:32 smithi038 ceph-mon[92422]: pgmap v1232: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 8 op/s 2023-03-09T15:20:33.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:20:32 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:20:33.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:20:32 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:20:33.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:20:32 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 1343.150015 secs 2023-03-09T15:20:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1805.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:20:34.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:20:34 smithi122 ceph-mon[93869]: pgmap v1233: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 6 op/s 2023-03-09T15:20:34.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:20:34 smithi181 ceph-mon[94189]: pgmap v1233: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 6 op/s 2023-03-09T15:20:34.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:20:34 smithi038 ceph-mon[92422]: pgmap v1233: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 6 op/s 2023-03-09T15:20:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:20:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1345.956046 secs 2023-03-09T15:20:35.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:20:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1345.956046 secs 2023-03-09T15:20:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:20:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1345.956046 secs 2023-03-09T15:20:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:20:36 smithi122 ceph-mon[93869]: pgmap v1234: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 6 op/s 2023-03-09T15:20:36.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:20:36 smithi181 ceph-mon[94189]: pgmap v1234: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 6 op/s 2023-03-09T15:20:36.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:20:36 smithi038 ceph-mon[92422]: pgmap v1234: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 6 op/s 2023-03-09T15:20:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:20:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1348.150177 secs 2023-03-09T15:20:37.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:20:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1348.150177 secs 2023-03-09T15:20:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:20:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1348.150177 secs 2023-03-09T15:20:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:20:38 smithi122 ceph-mon[93869]: pgmap v1235: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 52 KiB/s wr, 10 op/s 2023-03-09T15:20:38.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:20:38 smithi181 ceph-mon[94189]: pgmap v1235: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 52 KiB/s wr, 10 op/s 2023-03-09T15:20:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:20:38 smithi038 ceph-mon[92422]: pgmap v1235: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 52 KiB/s wr, 10 op/s 2023-03-09T15:20:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1810.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:20:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:20:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:20:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:20:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:20:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:20:40 smithi122 ceph-mon[93869]: pgmap v1236: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 52 KiB/s wr, 8 op/s 2023-03-09T15:20:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:20:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1350.956104 secs 2023-03-09T15:20:40.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:20:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:20:40.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:20:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:20:40.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:20:40 smithi181 ceph-mon[94189]: pgmap v1236: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 52 KiB/s wr, 8 op/s 2023-03-09T15:20:40.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:20:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1350.956104 secs 2023-03-09T15:20:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:20:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:20:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:20:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:20:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:20:40 smithi038 ceph-mon[92422]: pgmap v1236: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 52 KiB/s wr, 8 op/s 2023-03-09T15:20:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:20:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1350.956104 secs 2023-03-09T15:20:42.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:20:42 smithi122 ceph-mon[93869]: pgmap v1237: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 96 KiB/s wr, 12 op/s 2023-03-09T15:20:42.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:20:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1353.150302 secs 2023-03-09T15:20:42.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:20:42 smithi181 ceph-mon[94189]: pgmap v1237: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 96 KiB/s wr, 12 op/s 2023-03-09T15:20:42.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:20:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1353.150302 secs 2023-03-09T15:20:42.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:20:42 smithi038 ceph-mon[92422]: pgmap v1237: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 96 KiB/s wr, 12 op/s 2023-03-09T15:20:42.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:20:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1353.150302 secs 2023-03-09T15:20:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1815.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:20:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:20:44 smithi122 ceph-mon[93869]: pgmap v1238: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 74 KiB/s wr, 9 op/s 2023-03-09T15:20:44.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:20:44 smithi181 ceph-mon[94189]: pgmap v1238: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 74 KiB/s wr, 9 op/s 2023-03-09T15:20:44.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:20:44 smithi038 ceph-mon[92422]: pgmap v1238: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 74 KiB/s wr, 9 op/s 2023-03-09T15:20:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:20:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1355.956166 secs 2023-03-09T15:20:45.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:20:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1355.956166 secs 2023-03-09T15:20:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:20:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1355.956166 secs 2023-03-09T15:20:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:20:46 smithi122 ceph-mon[93869]: pgmap v1239: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 74 KiB/s wr, 9 op/s 2023-03-09T15:20:46.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:20:46 smithi181 ceph-mon[94189]: pgmap v1239: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 74 KiB/s wr, 9 op/s 2023-03-09T15:20:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:20:46 smithi038 ceph-mon[92422]: pgmap v1239: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 74 KiB/s wr, 9 op/s 2023-03-09T15:20:47.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:20:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1358.150419 secs 2023-03-09T15:20:47.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:20:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1358.150419 secs 2023-03-09T15:20:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:20:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1358.150419 secs 2023-03-09T15:20:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:20:48 smithi122 ceph-mon[93869]: pgmap v1240: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 134 KiB/s wr, 14 op/s 2023-03-09T15:20:48.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:20:48 smithi181 ceph-mon[94189]: pgmap v1240: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 134 KiB/s wr, 14 op/s 2023-03-09T15:20:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:20:48 smithi038 ceph-mon[92422]: pgmap v1240: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 134 KiB/s wr, 14 op/s 2023-03-09T15:20:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1820.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:20:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:20:50 smithi122 ceph-mon[93869]: pgmap v1241: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 104 KiB/s wr, 10 op/s 2023-03-09T15:20:50.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:20:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1360.956227 secs 2023-03-09T15:20:50.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:20:50 smithi181 ceph-mon[94189]: pgmap v1241: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 104 KiB/s wr, 10 op/s 2023-03-09T15:20:50.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:20:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1360.956227 secs 2023-03-09T15:20:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:20:50 smithi038 ceph-mon[92422]: pgmap v1241: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 104 KiB/s wr, 10 op/s 2023-03-09T15:20:51.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:20:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1360.956227 secs 2023-03-09T15:20:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:20:52 smithi122 ceph-mon[93869]: pgmap v1242: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 132 KiB/s wr, 12 op/s 2023-03-09T15:20:52.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:20:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1363.150537 secs 2023-03-09T15:20:52.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:20:52 smithi181 ceph-mon[94189]: pgmap v1242: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 132 KiB/s wr, 12 op/s 2023-03-09T15:20:52.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:20:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1363.150537 secs 2023-03-09T15:20:53.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:20:52 smithi038 ceph-mon[92422]: pgmap v1242: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 132 KiB/s wr, 12 op/s 2023-03-09T15:20:53.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:20:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1363.150537 secs 2023-03-09T15:20:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1825.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:20:54.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:20:54 smithi122 ceph-mon[93869]: pgmap v1243: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 89 KiB/s wr, 9 op/s 2023-03-09T15:20:54.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:20:54 smithi181 ceph-mon[94189]: pgmap v1243: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 89 KiB/s wr, 9 op/s 2023-03-09T15:20:54.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:20:54 smithi038 ceph-mon[92422]: pgmap v1243: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 89 KiB/s wr, 9 op/s 2023-03-09T15:20:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:20:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1365.956288 secs 2023-03-09T15:20:55.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:20:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1365.956288 secs 2023-03-09T15:20:55.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:20:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1365.956288 secs 2023-03-09T15:20:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:20:56 smithi122 ceph-mon[93869]: pgmap v1244: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 110 KiB/s wr, 9 op/s 2023-03-09T15:20:56.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:20:56 smithi181 ceph-mon[94189]: pgmap v1244: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 110 KiB/s wr, 9 op/s 2023-03-09T15:20:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:20:56 smithi038 ceph-mon[92422]: pgmap v1244: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 110 KiB/s wr, 9 op/s 2023-03-09T15:20:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:20:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1368.150674 secs 2023-03-09T15:20:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:20:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1368.150674 secs 2023-03-09T15:20:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:20:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1368.150674 secs 2023-03-09T15:20:58.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:20:58 smithi122 ceph-mon[93869]: pgmap v1245: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 114 KiB/s wr, 12 op/s 2023-03-09T15:20:58.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:20:58 smithi181 ceph-mon[94189]: pgmap v1245: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 114 KiB/s wr, 12 op/s 2023-03-09T15:20:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:20:58 smithi038 ceph-mon[92422]: pgmap v1245: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 114 KiB/s wr, 12 op/s 2023-03-09T15:20:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1830.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:21:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:21:00 smithi122 ceph-mon[93869]: pgmap v1246: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 54 KiB/s wr, 7 op/s 2023-03-09T15:21:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:21:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1370.956345 secs 2023-03-09T15:21:00.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:21:00 smithi181 ceph-mon[94189]: pgmap v1246: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 54 KiB/s wr, 7 op/s 2023-03-09T15:21:00.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:21:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1370.956345 secs 2023-03-09T15:21:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:21:00 smithi038 ceph-mon[92422]: pgmap v1246: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 54 KiB/s wr, 7 op/s 2023-03-09T15:21:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:21:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1370.956345 secs 2023-03-09T15:21:02.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:21:02 smithi122 ceph-mon[93869]: pgmap v1247: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 77 KiB/s wr, 12 op/s 2023-03-09T15:21:02.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:21:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1373.150789 secs 2023-03-09T15:21:02.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:21:02 smithi181 ceph-mon[94189]: pgmap v1247: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 77 KiB/s wr, 12 op/s 2023-03-09T15:21:02.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:21:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1373.150789 secs 2023-03-09T15:21:02.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:21:02 smithi038 ceph-mon[92422]: pgmap v1247: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 77 KiB/s wr, 12 op/s 2023-03-09T15:21:03.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:21:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1373.150789 secs 2023-03-09T15:21:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1835.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:21:04.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:21:04 smithi122 ceph-mon[93869]: pgmap v1248: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 70 KiB/s wr, 21 op/s 2023-03-09T15:21:04.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:21:04 smithi181 ceph-mon[94189]: pgmap v1248: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 70 KiB/s wr, 21 op/s 2023-03-09T15:21:04.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:21:04 smithi038 ceph-mon[92422]: pgmap v1248: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 70 KiB/s wr, 21 op/s 2023-03-09T15:21:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:21:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1375.956405 secs 2023-03-09T15:21:05.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:21:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1375.956405 secs 2023-03-09T15:21:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:21:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1375.956405 secs 2023-03-09T15:21:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:21:06 smithi122 ceph-mon[93869]: pgmap v1249: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 80 KiB/s wr, 22 op/s 2023-03-09T15:21:06.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:21:06 smithi181 ceph-mon[94189]: pgmap v1249: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 80 KiB/s wr, 22 op/s 2023-03-09T15:21:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:21:06 smithi038 ceph-mon[92422]: pgmap v1249: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 80 KiB/s wr, 22 op/s 2023-03-09T15:21:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:21:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1378.150902 secs 2023-03-09T15:21:07.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:21:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1378.150902 secs 2023-03-09T15:21:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:21:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1378.150902 secs 2023-03-09T15:21:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:21:08 smithi122 ceph-mon[93869]: pgmap v1250: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 61 KiB/s wr, 24 op/s 2023-03-09T15:21:08.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:21:08 smithi181 ceph-mon[94189]: pgmap v1250: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 61 KiB/s wr, 24 op/s 2023-03-09T15:21:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:21:08 smithi038 ceph-mon[92422]: pgmap v1250: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 61 KiB/s wr, 24 op/s 2023-03-09T15:21:08.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1840.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:21:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:21:10 smithi122 ceph-mon[93869]: osdmap e292: 12 total, 12 up, 12 in 2023-03-09T15:21:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:21:10 smithi122 ceph-mon[93869]: pgmap v1252: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 69 KiB/s wr, 26 op/s 2023-03-09T15:21:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:21:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1380.956469 secs 2023-03-09T15:21:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:21:10 smithi181 ceph-mon[94189]: osdmap e292: 12 total, 12 up, 12 in 2023-03-09T15:21:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:21:10 smithi181 ceph-mon[94189]: pgmap v1252: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 69 KiB/s wr, 26 op/s 2023-03-09T15:21:10.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:21:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1380.956469 secs 2023-03-09T15:21:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:21:10 smithi038 ceph-mon[92422]: osdmap e292: 12 total, 12 up, 12 in 2023-03-09T15:21:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:21:10 smithi038 ceph-mon[92422]: pgmap v1252: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 69 KiB/s wr, 26 op/s 2023-03-09T15:21:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:21:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1380.956469 secs 2023-03-09T15:21:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:21:12 smithi122 ceph-mon[93869]: pgmap v1253: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 45 KiB/s wr, 23 op/s 2023-03-09T15:21:12.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:21:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1383.151024 secs 2023-03-09T15:21:12.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:21:12 smithi181 ceph-mon[94189]: pgmap v1253: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 45 KiB/s wr, 23 op/s 2023-03-09T15:21:12.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:21:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1383.151024 secs 2023-03-09T15:21:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:21:12 smithi038 ceph-mon[92422]: pgmap v1253: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 45 KiB/s wr, 23 op/s 2023-03-09T15:21:13.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:21:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1383.151024 secs 2023-03-09T15:21:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1845.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:21:14.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:21:14 smithi122 ceph-mon[93869]: pgmap v1254: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 8 op/s 2023-03-09T15:21:14.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:21:14 smithi181 ceph-mon[94189]: pgmap v1254: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 8 op/s 2023-03-09T15:21:14.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:21:14 smithi038 ceph-mon[92422]: pgmap v1254: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 8 op/s 2023-03-09T15:21:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:21:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1385.956529 secs 2023-03-09T15:21:15.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:21:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1385.956529 secs 2023-03-09T15:21:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:21:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1385.956529 secs 2023-03-09T15:21:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:21:16 smithi122 ceph-mon[93869]: pgmap v1255: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 8.2 KiB/s wr, 8 op/s 2023-03-09T15:21:16.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:21:16 smithi181 ceph-mon[94189]: pgmap v1255: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 8.2 KiB/s wr, 8 op/s 2023-03-09T15:21:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:21:16 smithi038 ceph-mon[92422]: pgmap v1255: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 8.2 KiB/s wr, 8 op/s 2023-03-09T15:21:17.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:21:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1388.151171 secs 2023-03-09T15:21:17.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:21:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1388.151171 secs 2023-03-09T15:21:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:21:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1388.151171 secs 2023-03-09T15:21:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:21:18 smithi122 ceph-mon[93869]: pgmap v1256: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 5.2 KiB/s wr, 7 op/s 2023-03-09T15:21:18.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:21:18 smithi122 ceph-mon[93869]: osdmap e293: 12 total, 12 up, 12 in 2023-03-09T15:21:18.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:21:18 smithi181 ceph-mon[94189]: pgmap v1256: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 5.2 KiB/s wr, 7 op/s 2023-03-09T15:21:18.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:21:18 smithi181 ceph-mon[94189]: osdmap e293: 12 total, 12 up, 12 in 2023-03-09T15:21:18.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:21:18 smithi038 ceph-mon[92422]: pgmap v1256: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 5.2 KiB/s wr, 7 op/s 2023-03-09T15:21:18.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:21:18 smithi038 ceph-mon[92422]: osdmap e293: 12 total, 12 up, 12 in 2023-03-09T15:21:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1850.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:21:20.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:21:20 smithi122 ceph-mon[93869]: pgmap v1258: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 8 op/s 2023-03-09T15:21:20.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:21:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1390.956603 secs 2023-03-09T15:21:20.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:21:20 smithi181 ceph-mon[94189]: pgmap v1258: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 8 op/s 2023-03-09T15:21:20.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:21:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1390.956603 secs 2023-03-09T15:21:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:21:20 smithi038 ceph-mon[92422]: pgmap v1258: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 8 op/s 2023-03-09T15:21:21.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:21:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1390.956603 secs 2023-03-09T15:21:22.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:21:22 smithi122 ceph-mon[93869]: pgmap v1259: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 16 KiB/s wr, 8 op/s 2023-03-09T15:21:22.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:21:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1393.151285 secs 2023-03-09T15:21:22.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:21:22 smithi181 ceph-mon[94189]: pgmap v1259: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 16 KiB/s wr, 8 op/s 2023-03-09T15:21:22.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:21:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1393.151285 secs 2023-03-09T15:21:22.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:21:22 smithi038 ceph-mon[92422]: pgmap v1259: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 16 KiB/s wr, 8 op/s 2023-03-09T15:21:23.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:21:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1393.151285 secs 2023-03-09T15:21:23.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1855.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:21:24.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:21:24 smithi122 ceph-mon[93869]: pgmap v1260: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T15:21:24.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:21:24 smithi181 ceph-mon[94189]: pgmap v1260: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T15:21:24.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:21:24 smithi038 ceph-mon[92422]: pgmap v1260: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T15:21:25.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:21:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1395.956662 secs 2023-03-09T15:21:25.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:21:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1395.956662 secs 2023-03-09T15:21:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:21:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1395.956662 secs 2023-03-09T15:21:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:21:26 smithi122 ceph-mon[93869]: pgmap v1261: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T15:21:26.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:21:26 smithi181 ceph-mon[94189]: pgmap v1261: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T15:21:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:21:26 smithi038 ceph-mon[92422]: pgmap v1261: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T15:21:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:21:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1398.151405 secs 2023-03-09T15:21:27.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:21:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1398.151405 secs 2023-03-09T15:21:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:21:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1398.151405 secs 2023-03-09T15:21:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:21:28 smithi122 ceph-mon[93869]: pgmap v1262: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T15:21:28.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:21:28 smithi181 ceph-mon[94189]: pgmap v1262: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T15:21:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:21:28 smithi038 ceph-mon[92422]: pgmap v1262: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T15:21:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1860.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:21:30.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:21:30 smithi122 ceph-mon[93869]: pgmap v1263: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T15:21:30.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:21:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1400.956736 secs 2023-03-09T15:21:30.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:21:30 smithi181 ceph-mon[94189]: pgmap v1263: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T15:21:30.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:21:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1400.956736 secs 2023-03-09T15:21:31.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:21:30 smithi038 ceph-mon[92422]: pgmap v1263: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T15:21:31.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:21:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1400.956736 secs 2023-03-09T15:21:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:21:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:21:31.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:21:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:21:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:21:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:21:32.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:21:32 smithi122 ceph-mon[93869]: pgmap v1264: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 KiB/s wr, 8 op/s 2023-03-09T15:21:32.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:21:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:21:32.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:21:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:21:32.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:21:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:21:32.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:21:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1403.151483 secs 2023-03-09T15:21:32.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:21:32 smithi181 ceph-mon[94189]: pgmap v1264: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 KiB/s wr, 8 op/s 2023-03-09T15:21:32.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:21:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:21:32.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:21:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:21:32.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:21:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:21:32.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:21:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1403.151483 secs 2023-03-09T15:21:33.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:21:32 smithi038 ceph-mon[92422]: pgmap v1264: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 KiB/s wr, 8 op/s 2023-03-09T15:21:33.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:21:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:21:33.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:21:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:21:33.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:21:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:21:33.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:21:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1403.151483 secs 2023-03-09T15:21:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1865.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:21:34.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:21:34 smithi122 ceph-mon[93869]: pgmap v1265: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 5 op/s 2023-03-09T15:21:34.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:21:34 smithi181 ceph-mon[94189]: pgmap v1265: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 5 op/s 2023-03-09T15:21:34.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:21:34 smithi038 ceph-mon[92422]: pgmap v1265: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 5 op/s 2023-03-09T15:21:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:21:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1405.956817 secs 2023-03-09T15:21:35.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:21:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1405.956817 secs 2023-03-09T15:21:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:21:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1405.956817 secs 2023-03-09T15:21:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:21:36 smithi122 ceph-mon[93869]: pgmap v1266: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 43 KiB/s wr, 6 op/s 2023-03-09T15:21:36.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:21:36 smithi181 ceph-mon[94189]: pgmap v1266: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 43 KiB/s wr, 6 op/s 2023-03-09T15:21:36.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:21:36 smithi038 ceph-mon[92422]: pgmap v1266: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 43 KiB/s wr, 6 op/s 2023-03-09T15:21:37.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:21:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1408.151625 secs 2023-03-09T15:21:37.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:21:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1408.151625 secs 2023-03-09T15:21:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:21:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1408.151625 secs 2023-03-09T15:21:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:21:38 smithi122 ceph-mon[93869]: pgmap v1267: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 46 KiB/s wr, 8 op/s 2023-03-09T15:21:38.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:21:38 smithi181 ceph-mon[94189]: pgmap v1267: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 46 KiB/s wr, 8 op/s 2023-03-09T15:21:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:21:38 smithi038 ceph-mon[92422]: pgmap v1267: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 46 KiB/s wr, 8 op/s 2023-03-09T15:21:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1870.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:21:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:21:40 smithi122 ceph-mon[93869]: pgmap v1268: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 64 KiB/s wr, 7 op/s 2023-03-09T15:21:40.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:21:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1410.956878 secs 2023-03-09T15:21:40.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:21:40 smithi181 ceph-mon[94189]: pgmap v1268: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 64 KiB/s wr, 7 op/s 2023-03-09T15:21:40.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:21:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1410.956878 secs 2023-03-09T15:21:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:21:40 smithi038 ceph-mon[92422]: pgmap v1268: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 64 KiB/s wr, 7 op/s 2023-03-09T15:21:41.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:21:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1410.956878 secs 2023-03-09T15:21:42.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:21:42 smithi122 ceph-mon[93869]: pgmap v1269: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 52 KiB/s wr, 10 op/s 2023-03-09T15:21:42.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:21:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1413.151718 secs 2023-03-09T15:21:42.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:21:42 smithi181 ceph-mon[94189]: pgmap v1269: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 52 KiB/s wr, 10 op/s 2023-03-09T15:21:42.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:21:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1413.151718 secs 2023-03-09T15:21:42.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:21:42 smithi038 ceph-mon[92422]: pgmap v1269: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 52 KiB/s wr, 10 op/s 2023-03-09T15:21:43.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:21:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1413.151718 secs 2023-03-09T15:21:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1875.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:21:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:21:44 smithi122 ceph-mon[93869]: pgmap v1270: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 52 KiB/s wr, 8 op/s 2023-03-09T15:21:44.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:21:44 smithi181 ceph-mon[94189]: pgmap v1270: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 52 KiB/s wr, 8 op/s 2023-03-09T15:21:44.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:21:44 smithi038 ceph-mon[92422]: pgmap v1270: 129 pgs: 129 active+clean; 8.4 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 52 KiB/s wr, 8 op/s 2023-03-09T15:21:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:21:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1415.956939 secs 2023-03-09T15:21:45.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:21:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1415.956939 secs 2023-03-09T15:21:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:21:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1415.956939 secs 2023-03-09T15:21:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:21:46 smithi122 ceph-mon[93869]: pgmap v1271: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 88 KiB/s wr, 9 op/s 2023-03-09T15:21:46.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:21:46 smithi181 ceph-mon[94189]: pgmap v1271: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 88 KiB/s wr, 9 op/s 2023-03-09T15:21:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:21:46 smithi038 ceph-mon[92422]: pgmap v1271: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 88 KiB/s wr, 9 op/s 2023-03-09T15:21:47.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:21:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1418.151813 secs 2023-03-09T15:21:47.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:21:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1418.151813 secs 2023-03-09T15:21:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:21:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1418.151813 secs 2023-03-09T15:21:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:21:48 smithi122 ceph-mon[93869]: pgmap v1272: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 65 KiB/s wr, 12 op/s 2023-03-09T15:21:48.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:21:48 smithi181 ceph-mon[94189]: pgmap v1272: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 65 KiB/s wr, 12 op/s 2023-03-09T15:21:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:21:48 smithi038 ceph-mon[92422]: pgmap v1272: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 65 KiB/s wr, 12 op/s 2023-03-09T15:21:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1880.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:21:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:21:50 smithi122 ceph-mon[93869]: pgmap v1273: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 80 KiB/s wr, 11 op/s 2023-03-09T15:21:50.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:21:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1420.956986 secs 2023-03-09T15:21:50.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:21:50 smithi181 ceph-mon[94189]: pgmap v1273: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 80 KiB/s wr, 11 op/s 2023-03-09T15:21:50.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:21:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1420.956986 secs 2023-03-09T15:21:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:21:50 smithi038 ceph-mon[92422]: pgmap v1273: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 80 KiB/s wr, 11 op/s 2023-03-09T15:21:51.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:21:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1420.956986 secs 2023-03-09T15:21:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:21:52 smithi122 ceph-mon[93869]: pgmap v1274: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 65 KiB/s wr, 12 op/s 2023-03-09T15:21:52.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:21:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1423.151953 secs 2023-03-09T15:21:52.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:21:52 smithi181 ceph-mon[94189]: pgmap v1274: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 65 KiB/s wr, 12 op/s 2023-03-09T15:21:52.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:21:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1423.151953 secs 2023-03-09T15:21:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:21:52 smithi038 ceph-mon[92422]: pgmap v1274: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 65 KiB/s wr, 12 op/s 2023-03-09T15:21:53.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:21:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1423.151953 secs 2023-03-09T15:21:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1885.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:21:54.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:21:54 smithi122 ceph-mon[93869]: pgmap v1275: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 66 KiB/s wr, 10 op/s 2023-03-09T15:21:54.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:21:54 smithi181 ceph-mon[94189]: pgmap v1275: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 66 KiB/s wr, 10 op/s 2023-03-09T15:21:54.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:21:54 smithi038 ceph-mon[92422]: pgmap v1275: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 66 KiB/s wr, 10 op/s 2023-03-09T15:21:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:21:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1425.957076 secs 2023-03-09T15:21:55.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:21:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1425.957076 secs 2023-03-09T15:21:55.753 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:21:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1425.957076 secs 2023-03-09T15:21:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:21:56 smithi122 ceph-mon[93869]: pgmap v1276: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 122 KiB/s wr, 11 op/s 2023-03-09T15:21:56.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:21:56 smithi181 ceph-mon[94189]: pgmap v1276: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 122 KiB/s wr, 11 op/s 2023-03-09T15:21:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:21:56 smithi038 ceph-mon[92422]: pgmap v1276: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 122 KiB/s wr, 11 op/s 2023-03-09T15:21:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:21:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1428.152073 secs 2023-03-09T15:21:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:21:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1428.152073 secs 2023-03-09T15:21:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:21:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1428.152073 secs 2023-03-09T15:21:58.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:21:58 smithi122 ceph-mon[93869]: pgmap v1277: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 87 KiB/s wr, 13 op/s 2023-03-09T15:21:58.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:21:58 smithi181 ceph-mon[94189]: pgmap v1277: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 87 KiB/s wr, 13 op/s 2023-03-09T15:21:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:21:58 smithi038 ceph-mon[92422]: pgmap v1277: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 87 KiB/s wr, 13 op/s 2023-03-09T15:21:58.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1890.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:22:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:22:00 smithi122 ceph-mon[93869]: pgmap v1278: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 92 KiB/s wr, 10 op/s 2023-03-09T15:22:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:22:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1430.957134 secs 2023-03-09T15:22:00.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:22:00 smithi181 ceph-mon[94189]: pgmap v1278: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 92 KiB/s wr, 10 op/s 2023-03-09T15:22:00.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:22:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1430.957134 secs 2023-03-09T15:22:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:22:00 smithi038 ceph-mon[92422]: pgmap v1278: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 92 KiB/s wr, 10 op/s 2023-03-09T15:22:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:22:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1430.957134 secs 2023-03-09T15:22:02.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:22:02 smithi122 ceph-mon[93869]: pgmap v1279: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 96 KiB/s wr, 13 op/s 2023-03-09T15:22:02.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:22:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1433.152132 secs 2023-03-09T15:22:02.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:22:02 smithi181 ceph-mon[94189]: pgmap v1279: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 96 KiB/s wr, 13 op/s 2023-03-09T15:22:02.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:22:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1433.152132 secs 2023-03-09T15:22:02.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:22:02 smithi038 ceph-mon[92422]: pgmap v1279: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 96 KiB/s wr, 13 op/s 2023-03-09T15:22:03.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:22:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1433.152132 secs 2023-03-09T15:22:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1895.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:22:04.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:22:04 smithi122 ceph-mon[93869]: pgmap v1280: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 117 KiB/s wr, 23 op/s 2023-03-09T15:22:04.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:22:04 smithi181 ceph-mon[94189]: pgmap v1280: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 117 KiB/s wr, 23 op/s 2023-03-09T15:22:04.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:22:04 smithi038 ceph-mon[92422]: pgmap v1280: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 117 KiB/s wr, 23 op/s 2023-03-09T15:22:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:22:05 smithi122 ceph-mon[93869]: osdmap e294: 12 total, 12 up, 12 in 2023-03-09T15:22:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:22:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1435.957196 secs 2023-03-09T15:22:05.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:22:05 smithi181 ceph-mon[94189]: osdmap e294: 12 total, 12 up, 12 in 2023-03-09T15:22:05.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:22:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1435.957196 secs 2023-03-09T15:22:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:22:05 smithi038 ceph-mon[92422]: osdmap e294: 12 total, 12 up, 12 in 2023-03-09T15:22:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:22:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1435.957196 secs 2023-03-09T15:22:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:22:06 smithi122 ceph-mon[93869]: pgmap v1282: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 92 KiB/s wr, 27 op/s 2023-03-09T15:22:06.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:22:06 smithi181 ceph-mon[94189]: pgmap v1282: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 92 KiB/s wr, 27 op/s 2023-03-09T15:22:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:22:06 smithi038 ceph-mon[92422]: pgmap v1282: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 92 KiB/s wr, 27 op/s 2023-03-09T15:22:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:22:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1438.152247 secs 2023-03-09T15:22:07.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:22:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1438.152247 secs 2023-03-09T15:22:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:22:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1438.152247 secs 2023-03-09T15:22:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:22:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:22:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:22:08 smithi122 ceph-mon[93869]: pgmap v1283: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 94 KiB/s wr, 26 op/s 2023-03-09T15:22:08.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:22:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:22:08.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:22:08 smithi181 ceph-mon[94189]: pgmap v1283: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 94 KiB/s wr, 26 op/s 2023-03-09T15:22:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:22:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:22:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:22:08 smithi038 ceph-mon[92422]: pgmap v1283: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 94 KiB/s wr, 26 op/s 2023-03-09T15:22:08.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1900.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:22:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:22:10 smithi122 ceph-mon[93869]: pgmap v1284: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 94 KiB/s wr, 26 op/s 2023-03-09T15:22:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:22:10 smithi122 ceph-mon[93869]: 2 slow requests, 2 included below; oldest blocked for > 1440.957254 secs 2023-03-09T15:22:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:22:10 smithi122 ceph-mon[93869]: slow request 1440.957254 seconds old, received at 2023-03-09T14:58:08.564667+0000: peer_request:mds.2:599 currently acquired locks 2023-03-09T15:22:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:22:10 smithi122 ceph-mon[93869]: slow request 1440.946553 seconds old, received at 2023-03-09T14:58:08.575367+0000: peer_request(client.15375:6773.0 authpin) currently dispatched 2023-03-09T15:22:10.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:22:10 smithi181 ceph-mon[94189]: pgmap v1284: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 94 KiB/s wr, 26 op/s 2023-03-09T15:22:10.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:22:10 smithi181 ceph-mon[94189]: 2 slow requests, 2 included below; oldest blocked for > 1440.957254 secs 2023-03-09T15:22:10.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:22:10 smithi181 ceph-mon[94189]: slow request 1440.957254 seconds old, received at 2023-03-09T14:58:08.564667+0000: peer_request:mds.2:599 currently acquired locks 2023-03-09T15:22:10.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:22:10 smithi181 ceph-mon[94189]: slow request 1440.946553 seconds old, received at 2023-03-09T14:58:08.575367+0000: peer_request(client.15375:6773.0 authpin) currently dispatched 2023-03-09T15:22:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:22:10 smithi038 ceph-mon[92422]: pgmap v1284: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 94 KiB/s wr, 26 op/s 2023-03-09T15:22:11.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:22:10 smithi038 ceph-mon[92422]: 2 slow requests, 2 included below; oldest blocked for > 1440.957254 secs 2023-03-09T15:22:11.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:22:10 smithi038 ceph-mon[92422]: slow request 1440.957254 seconds old, received at 2023-03-09T14:58:08.564667+0000: peer_request:mds.2:599 currently acquired locks 2023-03-09T15:22:11.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:22:10 smithi038 ceph-mon[92422]: slow request 1440.946553 seconds old, received at 2023-03-09T14:58:08.575367+0000: peer_request(client.15375:6773.0 authpin) currently dispatched 2023-03-09T15:22:12.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:22:12 smithi122 ceph-mon[93869]: pgmap v1285: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 67 KiB/s wr, 23 op/s 2023-03-09T15:22:12.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:22:12 smithi122 ceph-mon[93869]: 2 slow requests, 2 included below; oldest blocked for > 1443.152343 secs 2023-03-09T15:22:12.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:22:12 smithi122 ceph-mon[93869]: slow request 1443.152342 seconds old, received at 2023-03-09T14:58:08.560137+0000: client_request(mds.2:599 rename #0x10000000007/0000000100000001000000EB #0x60f/1000000062a caller_uid=0, caller_gid=0{}) currently acquired locks 2023-03-09T15:22:12.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:22:12 smithi122 ceph-mon[93869]: slow request 1443.137962 seconds old, received at 2023-03-09T14:58:08.574517+0000: client_request(client.15375:6773 unlink #0x10000000007/0000000100000001000000EB 2023-03-09T14:58:08.569481+0000 caller_uid=26, caller_gid=26{26,}) currently requesting remote authpins 2023-03-09T15:22:12.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:22:12 smithi181 ceph-mon[94189]: pgmap v1285: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 67 KiB/s wr, 23 op/s 2023-03-09T15:22:12.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:22:12 smithi181 ceph-mon[94189]: 2 slow requests, 2 included below; oldest blocked for > 1443.152343 secs 2023-03-09T15:22:12.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:22:12 smithi181 ceph-mon[94189]: slow request 1443.152342 seconds old, received at 2023-03-09T14:58:08.560137+0000: client_request(mds.2:599 rename #0x10000000007/0000000100000001000000EB #0x60f/1000000062a caller_uid=0, caller_gid=0{}) currently acquired locks 2023-03-09T15:22:12.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:22:12 smithi181 ceph-mon[94189]: slow request 1443.137962 seconds old, received at 2023-03-09T14:58:08.574517+0000: client_request(client.15375:6773 unlink #0x10000000007/0000000100000001000000EB 2023-03-09T14:58:08.569481+0000 caller_uid=26, caller_gid=26{26,}) currently requesting remote authpins 2023-03-09T15:22:13.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:22:12 smithi038 ceph-mon[92422]: pgmap v1285: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 67 KiB/s wr, 23 op/s 2023-03-09T15:22:13.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:22:12 smithi038 ceph-mon[92422]: 2 slow requests, 2 included below; oldest blocked for > 1443.152343 secs 2023-03-09T15:22:13.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:22:12 smithi038 ceph-mon[92422]: slow request 1443.152342 seconds old, received at 2023-03-09T14:58:08.560137+0000: client_request(mds.2:599 rename #0x10000000007/0000000100000001000000EB #0x60f/1000000062a caller_uid=0, caller_gid=0{}) currently acquired locks 2023-03-09T15:22:13.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:22:12 smithi038 ceph-mon[92422]: slow request 1443.137962 seconds old, received at 2023-03-09T14:58:08.574517+0000: client_request(client.15375:6773 unlink #0x10000000007/0000000100000001000000EB 2023-03-09T14:58:08.569481+0000 caller_uid=26, caller_gid=26{26,}) currently requesting remote authpins 2023-03-09T15:22:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1905.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:22:14.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:22:14 smithi122 ceph-mon[93869]: osdmap e295: 12 total, 12 up, 12 in 2023-03-09T15:22:14.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:22:14 smithi122 ceph-mon[93869]: pgmap v1287: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 KiB/s wr, 7 op/s 2023-03-09T15:22:14.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:22:14 smithi181 ceph-mon[94189]: osdmap e295: 12 total, 12 up, 12 in 2023-03-09T15:22:14.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:22:14 smithi181 ceph-mon[94189]: pgmap v1287: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 KiB/s wr, 7 op/s 2023-03-09T15:22:14.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:22:14 smithi038 ceph-mon[92422]: osdmap e295: 12 total, 12 up, 12 in 2023-03-09T15:22:14.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:22:14 smithi038 ceph-mon[92422]: pgmap v1287: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 KiB/s wr, 7 op/s 2023-03-09T15:22:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:22:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1445.957313 secs 2023-03-09T15:22:15.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:22:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1445.957313 secs 2023-03-09T15:22:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:22:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1445.957313 secs 2023-03-09T15:22:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:22:16 smithi122 ceph-mon[93869]: pgmap v1288: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 7 op/s 2023-03-09T15:22:16.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:22:16 smithi181 ceph-mon[94189]: pgmap v1288: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 7 op/s 2023-03-09T15:22:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:22:16 smithi038 ceph-mon[92422]: pgmap v1288: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 7 op/s 2023-03-09T15:22:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:22:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1448.152412 secs 2023-03-09T15:22:17.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:22:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1448.152412 secs 2023-03-09T15:22:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:22:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1448.152412 secs 2023-03-09T15:22:18.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:22:18 smithi122 ceph-mon[93869]: pgmap v1289: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 6 op/s 2023-03-09T15:22:18.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:22:18 smithi181 ceph-mon[94189]: pgmap v1289: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 6 op/s 2023-03-09T15:22:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:22:18 smithi038 ceph-mon[92422]: pgmap v1289: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 6 op/s 2023-03-09T15:22:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1910.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:22:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:22:20 smithi122 ceph-mon[93869]: pgmap v1290: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2023-03-09T15:22:20.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:22:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1450.957372 secs 2023-03-09T15:22:20.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:22:20 smithi181 ceph-mon[94189]: pgmap v1290: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2023-03-09T15:22:20.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:22:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1450.957372 secs 2023-03-09T15:22:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:22:20 smithi038 ceph-mon[92422]: pgmap v1290: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2023-03-09T15:22:21.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:22:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1450.957372 secs 2023-03-09T15:22:22.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:22:22 smithi122 ceph-mon[93869]: pgmap v1291: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2023-03-09T15:22:22.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:22:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1453.152477 secs 2023-03-09T15:22:22.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:22:22 smithi181 ceph-mon[94189]: pgmap v1291: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2023-03-09T15:22:22.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:22:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1453.152477 secs 2023-03-09T15:22:22.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:22:22 smithi038 ceph-mon[92422]: pgmap v1291: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2023-03-09T15:22:22.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:22:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1453.152477 secs 2023-03-09T15:22:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1915.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:22:24.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:22:24 smithi122 ceph-mon[93869]: pgmap v1292: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2023-03-09T15:22:24.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:22:24 smithi181 ceph-mon[94189]: pgmap v1292: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2023-03-09T15:22:24.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:22:24 smithi038 ceph-mon[92422]: pgmap v1292: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2023-03-09T15:22:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:22:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1455.957434 secs 2023-03-09T15:22:25.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:22:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1455.957434 secs 2023-03-09T15:22:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:22:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1455.957434 secs 2023-03-09T15:22:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:22:26 smithi122 ceph-mon[93869]: pgmap v1293: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 6 op/s 2023-03-09T15:22:26.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:22:26 smithi181 ceph-mon[94189]: pgmap v1293: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 6 op/s 2023-03-09T15:22:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:22:26 smithi038 ceph-mon[92422]: pgmap v1293: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 6 op/s 2023-03-09T15:22:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1920.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:22:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:22:28 smithi122 ceph-mon[93869]: pgmap v1294: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 KiB/s wr, 8 op/s 2023-03-09T15:22:28.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:22:28 smithi181 ceph-mon[94189]: pgmap v1294: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 KiB/s wr, 8 op/s 2023-03-09T15:22:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:22:28 smithi038 ceph-mon[92422]: pgmap v1294: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 KiB/s wr, 8 op/s 2023-03-09T15:22:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:22:30 smithi122 ceph-mon[93869]: pgmap v1295: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 28 KiB/s wr, 6 op/s 2023-03-09T15:22:30.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:22:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1460.957497 secs 2023-03-09T15:22:30.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:22:30 smithi181 ceph-mon[94189]: pgmap v1295: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 28 KiB/s wr, 6 op/s 2023-03-09T15:22:30.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:22:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1460.957497 secs 2023-03-09T15:22:30.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:22:30 smithi038 ceph-mon[92422]: pgmap v1295: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 28 KiB/s wr, 6 op/s 2023-03-09T15:22:31.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:22:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1460.957497 secs 2023-03-09T15:22:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:22:31 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:22:31.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:22:31 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:22:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:22:31 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:22:32.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:22:32 smithi122 ceph-mon[93869]: pgmap v1296: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 28 KiB/s wr, 8 op/s 2023-03-09T15:22:32.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:22:32 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:22:32.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:22:32 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:22:32.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:22:32 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:22:32.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:22:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:22:32.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:22:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:22:32.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:22:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:22:32.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:22:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:22:32.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:22:32 smithi181 ceph-mon[94189]: pgmap v1296: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 28 KiB/s wr, 8 op/s 2023-03-09T15:22:32.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:22:32 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:22:32.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:22:32 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:22:32.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:22:32 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:22:32.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:22:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:22:32.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:22:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:22:32.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:22:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:22:32.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:22:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:22:33.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:22:32 smithi038 ceph-mon[92422]: pgmap v1296: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 28 KiB/s wr, 8 op/s 2023-03-09T15:22:33.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:22:32 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:22:33.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:22:32 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:22:33.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:22:32 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:22:33.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:22:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:22:33.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:22:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:22:33.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:22:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:22:33.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:22:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:22:33.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1925.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:22:34.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:22:34 smithi122 ceph-mon[93869]: pgmap v1297: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 30 KiB/s wr, 7 op/s 2023-03-09T15:22:34.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:22:34 smithi181 ceph-mon[94189]: pgmap v1297: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 30 KiB/s wr, 7 op/s 2023-03-09T15:22:34.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:22:34 smithi038 ceph-mon[92422]: pgmap v1297: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 30 KiB/s wr, 7 op/s 2023-03-09T15:22:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:22:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1465.957555 secs 2023-03-09T15:22:35.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:22:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1465.957555 secs 2023-03-09T15:22:35.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:22:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1465.957555 secs 2023-03-09T15:22:36.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:22:36 smithi122 ceph-mon[93869]: pgmap v1298: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 72 KiB/s wr, 7 op/s 2023-03-09T15:22:36.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:22:36 smithi181 ceph-mon[94189]: pgmap v1298: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 72 KiB/s wr, 7 op/s 2023-03-09T15:22:36.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:22:36 smithi038 ceph-mon[92422]: pgmap v1298: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 72 KiB/s wr, 7 op/s 2023-03-09T15:22:38.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1930.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:22:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:22:38 smithi122 ceph-mon[93869]: pgmap v1299: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 62 KiB/s wr, 10 op/s 2023-03-09T15:22:38.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:22:38 smithi181 ceph-mon[94189]: pgmap v1299: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 62 KiB/s wr, 10 op/s 2023-03-09T15:22:39.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:22:38 smithi038 ceph-mon[92422]: pgmap v1299: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 62 KiB/s wr, 10 op/s 2023-03-09T15:22:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:22:40 smithi122 ceph-mon[93869]: pgmap v1300: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 67 KiB/s wr, 8 op/s 2023-03-09T15:22:40.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:22:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1470.957618 secs 2023-03-09T15:22:40.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:22:40 smithi181 ceph-mon[94189]: pgmap v1300: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 67 KiB/s wr, 8 op/s 2023-03-09T15:22:40.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:22:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1470.957618 secs 2023-03-09T15:22:41.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:22:40 smithi038 ceph-mon[92422]: pgmap v1300: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 67 KiB/s wr, 8 op/s 2023-03-09T15:22:41.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:22:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1470.957618 secs 2023-03-09T15:22:42.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:22:42 smithi122 ceph-mon[93869]: pgmap v1301: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 53 KiB/s wr, 10 op/s 2023-03-09T15:22:42.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:22:42 smithi181 ceph-mon[94189]: pgmap v1301: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 53 KiB/s wr, 10 op/s 2023-03-09T15:22:43.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:22:42 smithi038 ceph-mon[92422]: pgmap v1301: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 53 KiB/s wr, 10 op/s 2023-03-09T15:22:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1935.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:22:44.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:22:44 smithi122 ceph-mon[93869]: pgmap v1302: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 75 KiB/s wr, 10 op/s 2023-03-09T15:22:44.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:22:44 smithi181 ceph-mon[94189]: pgmap v1302: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 75 KiB/s wr, 10 op/s 2023-03-09T15:22:44.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:22:44 smithi038 ceph-mon[92422]: pgmap v1302: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 75 KiB/s wr, 10 op/s 2023-03-09T15:22:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:22:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1475.957695 secs 2023-03-09T15:22:45.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:22:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1475.957695 secs 2023-03-09T15:22:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:22:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1475.957695 secs 2023-03-09T15:22:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:22:46 smithi122 ceph-mon[93869]: pgmap v1303: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 88 KiB/s wr, 10 op/s 2023-03-09T15:22:46.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:22:46 smithi181 ceph-mon[94189]: pgmap v1303: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 88 KiB/s wr, 10 op/s 2023-03-09T15:22:46.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:22:46 smithi038 ceph-mon[92422]: pgmap v1303: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 88 KiB/s wr, 10 op/s 2023-03-09T15:22:48.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1940.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:22:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:22:48 smithi122 ceph-mon[93869]: pgmap v1304: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 47 KiB/s wr, 11 op/s 2023-03-09T15:22:48.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:22:48 smithi181 ceph-mon[94189]: pgmap v1304: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 47 KiB/s wr, 11 op/s 2023-03-09T15:22:49.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:22:48 smithi038 ceph-mon[92422]: pgmap v1304: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 47 KiB/s wr, 11 op/s 2023-03-09T15:22:50.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:22:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1480.957776 secs 2023-03-09T15:22:50.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:22:50 smithi122 ceph-mon[93869]: pgmap v1305: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 10 op/s 2023-03-09T15:22:50.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:22:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1480.957776 secs 2023-03-09T15:22:50.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:22:50 smithi181 ceph-mon[94189]: pgmap v1305: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 10 op/s 2023-03-09T15:22:51.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:22:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1480.957776 secs 2023-03-09T15:22:51.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:22:50 smithi038 ceph-mon[92422]: pgmap v1305: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 10 op/s 2023-03-09T15:22:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:22:52 smithi122 ceph-mon[93869]: pgmap v1306: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 53 KiB/s wr, 10 op/s 2023-03-09T15:22:52.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:22:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1483.153218 secs 2023-03-09T15:22:52.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:22:52 smithi181 ceph-mon[94189]: pgmap v1306: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 53 KiB/s wr, 10 op/s 2023-03-09T15:22:52.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:22:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1483.153218 secs 2023-03-09T15:22:53.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:22:52 smithi038 ceph-mon[92422]: pgmap v1306: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 53 KiB/s wr, 10 op/s 2023-03-09T15:22:53.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:22:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1483.153218 secs 2023-03-09T15:22:53.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1945.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:22:54.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:22:54 smithi122 ceph-mon[93869]: pgmap v1307: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 83 KiB/s wr, 11 op/s 2023-03-09T15:22:54.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:22:54 smithi181 ceph-mon[94189]: pgmap v1307: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 83 KiB/s wr, 11 op/s 2023-03-09T15:22:54.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:22:54 smithi038 ceph-mon[92422]: pgmap v1307: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 83 KiB/s wr, 11 op/s 2023-03-09T15:22:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:22:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1485.957834 secs 2023-03-09T15:22:55.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:22:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1485.957834 secs 2023-03-09T15:22:55.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:22:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1485.957834 secs 2023-03-09T15:22:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:22:56 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:22:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:22:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:22:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:22:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:22:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:22:56 smithi122 ceph-mon[93869]: pgmap v1308: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 60 KiB/s wr, 9 op/s 2023-03-09T15:22:56.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:22:56 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:22:56.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:22:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:22:56.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:22:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:22:56.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:22:56 smithi181 ceph-mon[94189]: pgmap v1308: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 60 KiB/s wr, 9 op/s 2023-03-09T15:22:56.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:22:56 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:22:56.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:22:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:22:56.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:22:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:22:56.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:22:56 smithi038 ceph-mon[92422]: pgmap v1308: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 60 KiB/s wr, 9 op/s 2023-03-09T15:22:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:22:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1488.153334 secs 2023-03-09T15:22:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:22:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1488.153334 secs 2023-03-09T15:22:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:22:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1488.153334 secs 2023-03-09T15:22:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:22:58 smithi122 ceph-mon[93869]: pgmap v1309: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 47 KiB/s wr, 11 op/s 2023-03-09T15:22:58.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:22:58 smithi181 ceph-mon[94189]: pgmap v1309: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 47 KiB/s wr, 11 op/s 2023-03-09T15:22:58.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:22:58 smithi038 ceph-mon[92422]: pgmap v1309: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 47 KiB/s wr, 11 op/s 2023-03-09T15:22:58.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1950.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:23:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:23:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1490.957893 secs 2023-03-09T15:23:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:23:00 smithi122 ceph-mon[93869]: pgmap v1310: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 70 KiB/s wr, 9 op/s 2023-03-09T15:23:00.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:23:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1490.957893 secs 2023-03-09T15:23:00.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:23:00 smithi181 ceph-mon[94189]: pgmap v1310: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 70 KiB/s wr, 9 op/s 2023-03-09T15:23:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:23:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1490.957893 secs 2023-03-09T15:23:01.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:23:00 smithi038 ceph-mon[92422]: pgmap v1310: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 70 KiB/s wr, 9 op/s 2023-03-09T15:23:02.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:23:02 smithi122 ceph-mon[93869]: pgmap v1311: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 77 KiB/s wr, 12 op/s 2023-03-09T15:23:02.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:23:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1493.153455 secs 2023-03-09T15:23:02.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:23:02 smithi181 ceph-mon[94189]: pgmap v1311: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 77 KiB/s wr, 12 op/s 2023-03-09T15:23:02.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:23:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1493.153455 secs 2023-03-09T15:23:03.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:23:02 smithi038 ceph-mon[92422]: pgmap v1311: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 77 KiB/s wr, 12 op/s 2023-03-09T15:23:03.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:23:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1493.153455 secs 2023-03-09T15:23:03.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1955.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:23:04.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:23:04 smithi122 ceph-mon[93869]: pgmap v1312: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 115 KiB/s wr, 24 op/s 2023-03-09T15:23:04.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:23:04 smithi181 ceph-mon[94189]: pgmap v1312: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 115 KiB/s wr, 24 op/s 2023-03-09T15:23:04.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:23:04 smithi038 ceph-mon[92422]: pgmap v1312: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 115 KiB/s wr, 24 op/s 2023-03-09T15:23:05.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:23:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1495.957952 secs 2023-03-09T15:23:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:23:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1495.957952 secs 2023-03-09T15:23:05.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:23:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1495.957952 secs 2023-03-09T15:23:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:23:06 smithi122 ceph-mon[93869]: pgmap v1313: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 86 KiB/s wr, 21 op/s 2023-03-09T15:23:06.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:23:06 smithi181 ceph-mon[94189]: pgmap v1313: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 86 KiB/s wr, 21 op/s 2023-03-09T15:23:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:23:06 smithi038 ceph-mon[92422]: pgmap v1313: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 86 KiB/s wr, 21 op/s 2023-03-09T15:23:07.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:23:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1498.153566 secs 2023-03-09T15:23:07.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:23:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1498.153566 secs 2023-03-09T15:23:07.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:23:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1498.153566 secs 2023-03-09T15:23:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:23:08 smithi122 ceph-mon[93869]: pgmap v1314: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 88 KiB/s wr, 24 op/s 2023-03-09T15:23:08.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:23:08 smithi181 ceph-mon[94189]: pgmap v1314: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 88 KiB/s wr, 24 op/s 2023-03-09T15:23:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:23:08 smithi038 ceph-mon[92422]: pgmap v1314: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 88 KiB/s wr, 24 op/s 2023-03-09T15:23:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1960.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:23:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:23:10 smithi122 ceph-mon[93869]: osdmap e296: 12 total, 12 up, 12 in 2023-03-09T15:23:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:23:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1500.957992 secs 2023-03-09T15:23:10.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:23:10 smithi122 ceph-mon[93869]: pgmap v1316: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 76 KiB/s wr, 25 op/s 2023-03-09T15:23:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:23:10 smithi181 ceph-mon[94189]: osdmap e296: 12 total, 12 up, 12 in 2023-03-09T15:23:10.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:23:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1500.957992 secs 2023-03-09T15:23:10.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:23:10 smithi181 ceph-mon[94189]: pgmap v1316: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 76 KiB/s wr, 25 op/s 2023-03-09T15:23:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:23:10 smithi038 ceph-mon[92422]: osdmap e296: 12 total, 12 up, 12 in 2023-03-09T15:23:10.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:23:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1500.957992 secs 2023-03-09T15:23:10.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:23:10 smithi038 ceph-mon[92422]: pgmap v1316: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 76 KiB/s wr, 25 op/s 2023-03-09T15:23:12.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:23:12 smithi122 ceph-mon[93869]: pgmap v1317: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 49 KiB/s wr, 23 op/s 2023-03-09T15:23:12.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:23:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1503.153685 secs 2023-03-09T15:23:12.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:23:12 smithi181 ceph-mon[94189]: pgmap v1317: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 49 KiB/s wr, 23 op/s 2023-03-09T15:23:12.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:23:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1503.153685 secs 2023-03-09T15:23:13.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:23:12 smithi038 ceph-mon[92422]: pgmap v1317: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 49 KiB/s wr, 23 op/s 2023-03-09T15:23:13.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:23:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1503.153685 secs 2023-03-09T15:23:13.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1965.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:23:14.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:23:14 smithi122 ceph-mon[93869]: pgmap v1318: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 4.5 KiB/s wr, 8 op/s 2023-03-09T15:23:14.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:23:14 smithi181 ceph-mon[94189]: pgmap v1318: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 4.5 KiB/s wr, 8 op/s 2023-03-09T15:23:14.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:23:14 smithi038 ceph-mon[92422]: pgmap v1318: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 4.5 KiB/s wr, 8 op/s 2023-03-09T15:23:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:23:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1505.958048 secs 2023-03-09T15:23:15.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:23:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1505.958048 secs 2023-03-09T15:23:15.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:23:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1505.958048 secs 2023-03-09T15:23:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:23:16 smithi122 ceph-mon[93869]: pgmap v1319: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 4.5 KiB/s wr, 8 op/s 2023-03-09T15:23:16.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:23:16 smithi181 ceph-mon[94189]: pgmap v1319: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 4.5 KiB/s wr, 8 op/s 2023-03-09T15:23:16.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:23:16 smithi038 ceph-mon[92422]: pgmap v1319: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 4.5 KiB/s wr, 8 op/s 2023-03-09T15:23:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:23:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1508.153767 secs 2023-03-09T15:23:17.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:23:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1508.153767 secs 2023-03-09T15:23:17.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:23:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1508.153767 secs 2023-03-09T15:23:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:23:18 smithi122 ceph-mon[93869]: pgmap v1320: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2023-03-09T15:23:18.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:23:18 smithi122 ceph-mon[93869]: osdmap e297: 12 total, 12 up, 12 in 2023-03-09T15:23:18.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:23:18 smithi181 ceph-mon[94189]: pgmap v1320: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2023-03-09T15:23:18.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:23:18 smithi181 ceph-mon[94189]: osdmap e297: 12 total, 12 up, 12 in 2023-03-09T15:23:18.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:23:18 smithi038 ceph-mon[92422]: pgmap v1320: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2023-03-09T15:23:18.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:23:18 smithi038 ceph-mon[92422]: osdmap e297: 12 total, 12 up, 12 in 2023-03-09T15:23:18.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1970.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:23:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:23:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1510.958110 secs 2023-03-09T15:23:20.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:23:20 smithi122 ceph-mon[93869]: pgmap v1322: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 17 KiB/s wr, 8 op/s 2023-03-09T15:23:20.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:23:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1510.958110 secs 2023-03-09T15:23:20.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:23:20 smithi181 ceph-mon[94189]: pgmap v1322: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 17 KiB/s wr, 8 op/s 2023-03-09T15:23:21.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:23:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1510.958110 secs 2023-03-09T15:23:21.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:23:20 smithi038 ceph-mon[92422]: pgmap v1322: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 17 KiB/s wr, 8 op/s 2023-03-09T15:23:22.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:23:22 smithi122 ceph-mon[93869]: pgmap v1323: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 17 KiB/s wr, 8 op/s 2023-03-09T15:23:22.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:23:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1513.153902 secs 2023-03-09T15:23:22.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:23:22 smithi181 ceph-mon[94189]: pgmap v1323: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 17 KiB/s wr, 8 op/s 2023-03-09T15:23:22.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:23:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1513.153902 secs 2023-03-09T15:23:23.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:23:22 smithi038 ceph-mon[92422]: pgmap v1323: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 17 KiB/s wr, 8 op/s 2023-03-09T15:23:23.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:23:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1513.153902 secs 2023-03-09T15:23:23.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1975.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:23:24.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:23:24 smithi122 ceph-mon[93869]: pgmap v1324: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 KiB/s wr, 8 op/s 2023-03-09T15:23:24.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:23:24 smithi181 ceph-mon[94189]: pgmap v1324: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 KiB/s wr, 8 op/s 2023-03-09T15:23:24.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:23:24 smithi038 ceph-mon[92422]: pgmap v1324: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 KiB/s wr, 8 op/s 2023-03-09T15:23:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:23:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1515.958173 secs 2023-03-09T15:23:25.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:23:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1515.958173 secs 2023-03-09T15:23:25.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:23:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1515.958173 secs 2023-03-09T15:23:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:23:26 smithi122 ceph-mon[93869]: pgmap v1325: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 KiB/s wr, 8 op/s 2023-03-09T15:23:26.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:23:26 smithi181 ceph-mon[94189]: pgmap v1325: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 KiB/s wr, 8 op/s 2023-03-09T15:23:26.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:23:26 smithi038 ceph-mon[92422]: pgmap v1325: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 KiB/s wr, 8 op/s 2023-03-09T15:23:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:23:27 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 1518.153992 secs 2023-03-09T15:23:27.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:23:27 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 1518.153992 secs 2023-03-09T15:23:27.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:23:27 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 1518.153992 secs 2023-03-09T15:23:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:23:28 smithi122 ceph-mon[93869]: pgmap v1326: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 KiB/s wr, 8 op/s 2023-03-09T15:23:28.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:23:28 smithi181 ceph-mon[94189]: pgmap v1326: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 KiB/s wr, 8 op/s 2023-03-09T15:23:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:23:28 smithi038 ceph-mon[92422]: pgmap v1326: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 KiB/s wr, 8 op/s 2023-03-09T15:23:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1980.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:23:30.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:23:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1520.958236 secs 2023-03-09T15:23:30.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:23:30 smithi122 ceph-mon[93869]: pgmap v1327: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 6 op/s 2023-03-09T15:23:30.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:23:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1520.958236 secs 2023-03-09T15:23:30.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:23:30 smithi181 ceph-mon[94189]: pgmap v1327: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 6 op/s 2023-03-09T15:23:31.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:23:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1520.958236 secs 2023-03-09T15:23:31.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:23:30 smithi038 ceph-mon[92422]: pgmap v1327: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 6 op/s 2023-03-09T15:23:32.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:23:32 smithi122 ceph-mon[93869]: pgmap v1328: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 8 op/s 2023-03-09T15:23:32.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:23:32 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:23:32.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:23:32 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:23:32.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:23:32 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 1523.154147 secs 2023-03-09T15:23:32.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:23:32 smithi181 ceph-mon[94189]: pgmap v1328: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 8 op/s 2023-03-09T15:23:32.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:23:32 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:23:32.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:23:32 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:23:32.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:23:32 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 1523.154147 secs 2023-03-09T15:23:33.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:23:32 smithi038 ceph-mon[92422]: pgmap v1328: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 8 op/s 2023-03-09T15:23:33.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:23:32 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:23:33.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:23:32 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:23:33.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:23:32 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 1523.154147 secs 2023-03-09T15:23:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1985.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:23:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:23:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:23:33.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:23:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:23:33.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:23:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:23:33.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:23:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:23:33.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:23:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:23:33.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:23:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:23:33.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:23:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:23:33.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:23:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:23:34.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:23:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:23:34.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:23:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:23:34.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:23:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:23:34.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:23:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:23:34.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:23:34 smithi122 ceph-mon[93869]: pgmap v1329: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 58 KiB/s wr, 8 op/s 2023-03-09T15:23:34.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:23:34 smithi181 ceph-mon[94189]: pgmap v1329: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 58 KiB/s wr, 8 op/s 2023-03-09T15:23:35.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:23:34 smithi038 ceph-mon[92422]: pgmap v1329: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 58 KiB/s wr, 8 op/s 2023-03-09T15:23:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:23:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1525.958299 secs 2023-03-09T15:23:35.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:23:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1525.958299 secs 2023-03-09T15:23:36.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:23:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1525.958299 secs 2023-03-09T15:23:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:23:36 smithi122 ceph-mon[93869]: pgmap v1330: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 57 KiB/s wr, 7 op/s 2023-03-09T15:23:36.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:23:36 smithi181 ceph-mon[94189]: pgmap v1330: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 57 KiB/s wr, 7 op/s 2023-03-09T15:23:37.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:23:36 smithi038 ceph-mon[92422]: pgmap v1330: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 57 KiB/s wr, 7 op/s 2023-03-09T15:23:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:23:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1528.154212 secs 2023-03-09T15:23:37.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:23:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1528.154212 secs 2023-03-09T15:23:38.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:23:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1528.154212 secs 2023-03-09T15:23:38.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1990.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:23:38.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:23:38 smithi122 ceph-mon[93869]: pgmap v1331: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 60 KiB/s wr, 10 op/s 2023-03-09T15:23:38.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:23:38 smithi181 ceph-mon[94189]: pgmap v1331: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 60 KiB/s wr, 10 op/s 2023-03-09T15:23:39.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:23:38 smithi038 ceph-mon[92422]: pgmap v1331: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 60 KiB/s wr, 10 op/s 2023-03-09T15:23:40.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:23:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1530.958358 secs 2023-03-09T15:23:40.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:23:40 smithi122 ceph-mon[93869]: pgmap v1332: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 75 KiB/s wr, 9 op/s 2023-03-09T15:23:40.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:23:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:23:40.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:23:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:23:40.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:23:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1530.958358 secs 2023-03-09T15:23:40.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:23:40 smithi181 ceph-mon[94189]: pgmap v1332: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 75 KiB/s wr, 9 op/s 2023-03-09T15:23:40.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:23:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:23:40.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:23:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:23:41.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:23:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1530.958358 secs 2023-03-09T15:23:41.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:23:40 smithi038 ceph-mon[92422]: pgmap v1332: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 75 KiB/s wr, 9 op/s 2023-03-09T15:23:41.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:23:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:23:41.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:23:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:23:42.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:23:42 smithi122 ceph-mon[93869]: pgmap v1333: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 64 KiB/s wr, 10 op/s 2023-03-09T15:23:42.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:23:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1533.154326 secs 2023-03-09T15:23:42.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:23:42 smithi181 ceph-mon[94189]: pgmap v1333: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 64 KiB/s wr, 10 op/s 2023-03-09T15:23:42.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:23:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1533.154326 secs 2023-03-09T15:23:43.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:23:42 smithi038 ceph-mon[92422]: pgmap v1333: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 64 KiB/s wr, 10 op/s 2023-03-09T15:23:43.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:23:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1533.154326 secs 2023-03-09T15:23:43.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 1995.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:23:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:23:44 smithi122 ceph-mon[93869]: pgmap v1334: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 110 KiB/s wr, 11 op/s 2023-03-09T15:23:44.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:23:44 smithi181 ceph-mon[94189]: pgmap v1334: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 110 KiB/s wr, 11 op/s 2023-03-09T15:23:44.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:23:44 smithi038 ceph-mon[92422]: pgmap v1334: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 110 KiB/s wr, 11 op/s 2023-03-09T15:23:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:23:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1535.958440 secs 2023-03-09T15:23:45.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:23:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1535.958440 secs 2023-03-09T15:23:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:23:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1535.958440 secs 2023-03-09T15:23:46.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:23:46 smithi122 ceph-mon[93869]: pgmap v1335: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 65 KiB/s wr, 9 op/s 2023-03-09T15:23:46.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:23:46 smithi181 ceph-mon[94189]: pgmap v1335: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 65 KiB/s wr, 9 op/s 2023-03-09T15:23:46.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:23:46 smithi038 ceph-mon[92422]: pgmap v1335: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 65 KiB/s wr, 9 op/s 2023-03-09T15:23:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:23:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1538.154451 secs 2023-03-09T15:23:47.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:23:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1538.154451 secs 2023-03-09T15:23:47.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:23:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1538.154451 secs 2023-03-09T15:23:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:23:48 smithi122 ceph-mon[93869]: pgmap v1336: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 67 KiB/s wr, 12 op/s 2023-03-09T15:23:48.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:23:48 smithi181 ceph-mon[94189]: pgmap v1336: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 67 KiB/s wr, 12 op/s 2023-03-09T15:23:48.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:23:48 smithi038 ceph-mon[92422]: pgmap v1336: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 67 KiB/s wr, 12 op/s 2023-03-09T15:23:48.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2000.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:23:50.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:23:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1540.958503 secs 2023-03-09T15:23:50.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:23:50 smithi122 ceph-mon[93869]: pgmap v1337: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 81 KiB/s wr, 9 op/s 2023-03-09T15:23:50.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:23:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1540.958503 secs 2023-03-09T15:23:50.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:23:50 smithi181 ceph-mon[94189]: pgmap v1337: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 81 KiB/s wr, 9 op/s 2023-03-09T15:23:51.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:23:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1540.958503 secs 2023-03-09T15:23:51.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:23:50 smithi038 ceph-mon[92422]: pgmap v1337: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 81 KiB/s wr, 9 op/s 2023-03-09T15:23:52.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:23:52 smithi122 ceph-mon[93869]: pgmap v1338: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 67 KiB/s wr, 10 op/s 2023-03-09T15:23:52.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:23:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1543.154549 secs 2023-03-09T15:23:52.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:23:52 smithi181 ceph-mon[94189]: pgmap v1338: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 67 KiB/s wr, 10 op/s 2023-03-09T15:23:52.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:23:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1543.154549 secs 2023-03-09T15:23:53.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:23:52 smithi038 ceph-mon[92422]: pgmap v1338: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 67 KiB/s wr, 10 op/s 2023-03-09T15:23:53.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:23:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1543.154549 secs 2023-03-09T15:23:53.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2005.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:23:54.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:23:54 smithi122 ceph-mon[93869]: pgmap v1339: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 93 KiB/s wr, 11 op/s 2023-03-09T15:23:54.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:23:54 smithi181 ceph-mon[94189]: pgmap v1339: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 93 KiB/s wr, 11 op/s 2023-03-09T15:23:54.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:23:54 smithi038 ceph-mon[92422]: pgmap v1339: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 93 KiB/s wr, 11 op/s 2023-03-09T15:23:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:23:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1545.958563 secs 2023-03-09T15:23:55.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:23:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1545.958563 secs 2023-03-09T15:23:55.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:23:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1545.958563 secs 2023-03-09T15:23:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:23:56 smithi122 ceph-mon[93869]: pgmap v1340: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 47 KiB/s wr, 8 op/s 2023-03-09T15:23:56.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:23:56 smithi181 ceph-mon[94189]: pgmap v1340: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 47 KiB/s wr, 8 op/s 2023-03-09T15:23:56.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:23:56 smithi038 ceph-mon[92422]: pgmap v1340: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 47 KiB/s wr, 8 op/s 2023-03-09T15:23:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:23:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1548.154667 secs 2023-03-09T15:23:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:23:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1548.154667 secs 2023-03-09T15:23:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:23:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1548.154667 secs 2023-03-09T15:23:58.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:23:58 smithi122 ceph-mon[93869]: pgmap v1341: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 47 KiB/s wr, 10 op/s 2023-03-09T15:23:58.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:23:58 smithi181 ceph-mon[94189]: pgmap v1341: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 47 KiB/s wr, 10 op/s 2023-03-09T15:23:58.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:23:58 smithi038 ceph-mon[92422]: pgmap v1341: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 47 KiB/s wr, 10 op/s 2023-03-09T15:23:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2010.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:24:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:24:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1550.958623 secs 2023-03-09T15:24:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:24:00 smithi122 ceph-mon[93869]: pgmap v1342: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 60 KiB/s wr, 8 op/s 2023-03-09T15:24:00.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:24:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1550.958623 secs 2023-03-09T15:24:00.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:24:00 smithi181 ceph-mon[94189]: pgmap v1342: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 60 KiB/s wr, 8 op/s 2023-03-09T15:24:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:24:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1550.958623 secs 2023-03-09T15:24:01.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:24:00 smithi038 ceph-mon[92422]: pgmap v1342: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 60 KiB/s wr, 8 op/s 2023-03-09T15:24:02.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:24:02 smithi122 ceph-mon[93869]: pgmap v1343: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 66 KiB/s wr, 11 op/s 2023-03-09T15:24:02.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:24:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1553.154786 secs 2023-03-09T15:24:02.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:24:02 smithi181 ceph-mon[94189]: pgmap v1343: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 66 KiB/s wr, 11 op/s 2023-03-09T15:24:02.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:24:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1553.154786 secs 2023-03-09T15:24:03.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:24:02 smithi038 ceph-mon[92422]: pgmap v1343: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 66 KiB/s wr, 11 op/s 2023-03-09T15:24:03.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:24:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1553.154786 secs 2023-03-09T15:24:03.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2015.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:24:04.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:24:04 smithi122 ceph-mon[93869]: pgmap v1344: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 99 KiB/s wr, 23 op/s 2023-03-09T15:24:04.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:24:04 smithi181 ceph-mon[94189]: pgmap v1344: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 99 KiB/s wr, 23 op/s 2023-03-09T15:24:04.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:24:04 smithi038 ceph-mon[92422]: pgmap v1344: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 99 KiB/s wr, 23 op/s 2023-03-09T15:24:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:24:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1555.958685 secs 2023-03-09T15:24:05.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:24:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1555.958685 secs 2023-03-09T15:24:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:24:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1555.958685 secs 2023-03-09T15:24:06.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:24:06 smithi122 ceph-mon[93869]: pgmap v1345: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 72 KiB/s wr, 21 op/s 2023-03-09T15:24:06.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:24:06 smithi181 ceph-mon[94189]: pgmap v1345: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 72 KiB/s wr, 21 op/s 2023-03-09T15:24:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:24:06 smithi038 ceph-mon[92422]: pgmap v1345: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 72 KiB/s wr, 21 op/s 2023-03-09T15:24:07.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:24:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1558.154922 secs 2023-03-09T15:24:07.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:24:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1558.154922 secs 2023-03-09T15:24:07.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:24:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1558.154922 secs 2023-03-09T15:24:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:24:08 smithi122 ceph-mon[93869]: pgmap v1346: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 75 KiB/s wr, 24 op/s 2023-03-09T15:24:08.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:24:08 smithi181 ceph-mon[94189]: pgmap v1346: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 75 KiB/s wr, 24 op/s 2023-03-09T15:24:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:24:08 smithi038 ceph-mon[92422]: pgmap v1346: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 75 KiB/s wr, 24 op/s 2023-03-09T15:24:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2020.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:24:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:24:10 smithi122 ceph-mon[93869]: osdmap e298: 12 total, 12 up, 12 in 2023-03-09T15:24:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:24:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1560.958745 secs 2023-03-09T15:24:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:24:10 smithi122 ceph-mon[93869]: pgmap v1348: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 70 KiB/s wr, 25 op/s 2023-03-09T15:24:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:24:10 smithi181 ceph-mon[94189]: osdmap e298: 12 total, 12 up, 12 in 2023-03-09T15:24:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:24:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1560.958745 secs 2023-03-09T15:24:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:24:10 smithi181 ceph-mon[94189]: pgmap v1348: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 70 KiB/s wr, 25 op/s 2023-03-09T15:24:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:24:10 smithi038 ceph-mon[92422]: osdmap e298: 12 total, 12 up, 12 in 2023-03-09T15:24:10.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:24:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1560.958745 secs 2023-03-09T15:24:10.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:24:10 smithi038 ceph-mon[92422]: pgmap v1348: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 70 KiB/s wr, 25 op/s 2023-03-09T15:24:12.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:24:12 smithi122 ceph-mon[93869]: pgmap v1349: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 44 KiB/s wr, 23 op/s 2023-03-09T15:24:12.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:24:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1563.155072 secs 2023-03-09T15:24:12.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:24:12 smithi181 ceph-mon[94189]: pgmap v1349: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 44 KiB/s wr, 23 op/s 2023-03-09T15:24:12.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:24:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1563.155072 secs 2023-03-09T15:24:13.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:24:12 smithi038 ceph-mon[92422]: pgmap v1349: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 44 KiB/s wr, 23 op/s 2023-03-09T15:24:13.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:24:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1563.155072 secs 2023-03-09T15:24:13.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2025.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:24:14.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:24:14 smithi122 ceph-mon[93869]: pgmap v1350: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 4.5 KiB/s wr, 8 op/s 2023-03-09T15:24:14.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:24:14 smithi181 ceph-mon[94189]: pgmap v1350: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 4.5 KiB/s wr, 8 op/s 2023-03-09T15:24:14.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:24:14 smithi038 ceph-mon[92422]: pgmap v1350: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 4.5 KiB/s wr, 8 op/s 2023-03-09T15:24:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:24:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1565.958808 secs 2023-03-09T15:24:15.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:24:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1565.958808 secs 2023-03-09T15:24:15.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:24:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1565.958808 secs 2023-03-09T15:24:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:24:16 smithi122 ceph-mon[93869]: pgmap v1351: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 4.6 KiB/s wr, 8 op/s 2023-03-09T15:24:16.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:24:16 smithi181 ceph-mon[94189]: pgmap v1351: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 4.6 KiB/s wr, 8 op/s 2023-03-09T15:24:16.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:24:16 smithi038 ceph-mon[92422]: pgmap v1351: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 4.6 KiB/s wr, 8 op/s 2023-03-09T15:24:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:24:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1568.155191 secs 2023-03-09T15:24:17.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:24:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1568.155191 secs 2023-03-09T15:24:17.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:24:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1568.155191 secs 2023-03-09T15:24:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:24:18 smithi122 ceph-mon[93869]: pgmap v1352: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2023-03-09T15:24:18.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:24:18 smithi122 ceph-mon[93869]: osdmap e299: 12 total, 12 up, 12 in 2023-03-09T15:24:18.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:24:18 smithi181 ceph-mon[94189]: pgmap v1352: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2023-03-09T15:24:18.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:24:18 smithi181 ceph-mon[94189]: osdmap e299: 12 total, 12 up, 12 in 2023-03-09T15:24:18.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:24:18 smithi038 ceph-mon[92422]: pgmap v1352: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2023-03-09T15:24:18.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:24:18 smithi038 ceph-mon[92422]: osdmap e299: 12 total, 12 up, 12 in 2023-03-09T15:24:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2030.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:24:20.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:24:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1570.958867 secs 2023-03-09T15:24:20.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:24:20 smithi122 ceph-mon[93869]: pgmap v1354: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 18 KiB/s wr, 8 op/s 2023-03-09T15:24:20.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:24:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1570.958867 secs 2023-03-09T15:24:20.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:24:20 smithi181 ceph-mon[94189]: pgmap v1354: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 18 KiB/s wr, 8 op/s 2023-03-09T15:24:21.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:24:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1570.958867 secs 2023-03-09T15:24:21.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:24:20 smithi038 ceph-mon[92422]: pgmap v1354: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 18 KiB/s wr, 8 op/s 2023-03-09T15:24:22.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:24:22 smithi122 ceph-mon[93869]: pgmap v1355: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 17 KiB/s wr, 8 op/s 2023-03-09T15:24:22.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:24:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1573.155330 secs 2023-03-09T15:24:22.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:24:22 smithi181 ceph-mon[94189]: pgmap v1355: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 17 KiB/s wr, 8 op/s 2023-03-09T15:24:22.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:24:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1573.155330 secs 2023-03-09T15:24:23.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:24:22 smithi038 ceph-mon[92422]: pgmap v1355: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 17 KiB/s wr, 8 op/s 2023-03-09T15:24:23.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:24:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1573.155330 secs 2023-03-09T15:24:23.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2035.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:24:24.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:24:24 smithi122 ceph-mon[93869]: pgmap v1356: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 8 op/s 2023-03-09T15:24:24.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:24:24 smithi181 ceph-mon[94189]: pgmap v1356: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 8 op/s 2023-03-09T15:24:24.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:24:24 smithi038 ceph-mon[92422]: pgmap v1356: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 8 op/s 2023-03-09T15:24:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:24:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1575.958951 secs 2023-03-09T15:24:25.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:24:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1575.958951 secs 2023-03-09T15:24:25.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:24:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1575.958951 secs 2023-03-09T15:24:26.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:24:26 smithi122 ceph-mon[93869]: pgmap v1357: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 7 op/s 2023-03-09T15:24:26.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:24:26 smithi181 ceph-mon[94189]: pgmap v1357: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 7 op/s 2023-03-09T15:24:26.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:24:26 smithi038 ceph-mon[92422]: pgmap v1357: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 7 op/s 2023-03-09T15:24:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:24:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1578.155442 secs 2023-03-09T15:24:27.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:24:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1578.155442 secs 2023-03-09T15:24:27.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:24:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1578.155442 secs 2023-03-09T15:24:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:24:28 smithi122 ceph-mon[93869]: pgmap v1358: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 KiB/s wr, 7 op/s 2023-03-09T15:24:28.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:24:28 smithi181 ceph-mon[94189]: pgmap v1358: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 KiB/s wr, 7 op/s 2023-03-09T15:24:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:24:28 smithi038 ceph-mon[92422]: pgmap v1358: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 KiB/s wr, 7 op/s 2023-03-09T15:24:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2040.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:24:30.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:24:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1580.958985 secs 2023-03-09T15:24:30.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:24:30 smithi122 ceph-mon[93869]: pgmap v1359: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 21 KiB/s wr, 6 op/s 2023-03-09T15:24:30.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:24:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1580.958985 secs 2023-03-09T15:24:30.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:24:30 smithi181 ceph-mon[94189]: pgmap v1359: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 21 KiB/s wr, 6 op/s 2023-03-09T15:24:31.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:24:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1580.958985 secs 2023-03-09T15:24:31.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:24:30 smithi038 ceph-mon[92422]: pgmap v1359: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 21 KiB/s wr, 6 op/s 2023-03-09T15:24:32.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:24:32 smithi122 ceph-mon[93869]: pgmap v1360: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 8 op/s 2023-03-09T15:24:32.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:24:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1583.155537 secs 2023-03-09T15:24:32.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:24:32 smithi181 ceph-mon[94189]: pgmap v1360: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 8 op/s 2023-03-09T15:24:32.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:24:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1583.155537 secs 2023-03-09T15:24:33.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:24:32 smithi038 ceph-mon[92422]: pgmap v1360: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 8 op/s 2023-03-09T15:24:33.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:24:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1583.155537 secs 2023-03-09T15:24:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2045.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:24:34.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:24:34 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:24:34.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:24:34 smithi122 ceph-mon[93869]: pgmap v1361: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 26 KiB/s wr, 8 op/s 2023-03-09T15:24:34.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:24:34 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:24:34.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:24:34 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:24:34.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:24:34 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:24:34.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:24:34 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:24:34.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:24:34 smithi181 ceph-mon[94189]: pgmap v1361: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 26 KiB/s wr, 8 op/s 2023-03-09T15:24:34.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:24:34 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:24:34.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:24:34 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:24:34.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:24:34 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:24:34.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:24:34 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:24:34.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:24:34 smithi038 ceph-mon[92422]: pgmap v1361: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 26 KiB/s wr, 8 op/s 2023-03-09T15:24:34.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:24:34 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:24:34.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:24:34 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:24:34.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:24:34 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:24:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:24:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1585.959045 secs 2023-03-09T15:24:35.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:24:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1585.959045 secs 2023-03-09T15:24:35.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:24:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1585.959045 secs 2023-03-09T15:24:36.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:24:36 smithi122 ceph-mon[93869]: pgmap v1362: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 26 KiB/s wr, 7 op/s 2023-03-09T15:24:36.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:24:36 smithi181 ceph-mon[94189]: pgmap v1362: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 26 KiB/s wr, 7 op/s 2023-03-09T15:24:36.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:24:36 smithi038 ceph-mon[92422]: pgmap v1362: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 26 KiB/s wr, 7 op/s 2023-03-09T15:24:38.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2050.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:24:38.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:24:38 smithi122 ceph-mon[93869]: pgmap v1363: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 43 KiB/s wr, 10 op/s 2023-03-09T15:24:38.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:24:38 smithi181 ceph-mon[94189]: pgmap v1363: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 43 KiB/s wr, 10 op/s 2023-03-09T15:24:39.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:24:38 smithi038 ceph-mon[92422]: pgmap v1363: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 43 KiB/s wr, 10 op/s 2023-03-09T15:24:39.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:24:39 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:24:39.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:24:39 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:24:40.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:24:39 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:24:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:24:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1590.959130 secs 2023-03-09T15:24:40.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:24:40 smithi122 ceph-mon[93869]: pgmap v1364: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 57 KiB/s wr, 9 op/s 2023-03-09T15:24:40.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:24:40 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:24:40.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:24:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:24:40.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:24:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:24:40.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:24:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1590.959130 secs 2023-03-09T15:24:40.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:24:40 smithi181 ceph-mon[94189]: pgmap v1364: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 57 KiB/s wr, 9 op/s 2023-03-09T15:24:40.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:24:40 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:24:40.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:24:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:24:40.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:24:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:24:41.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:24:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1590.959130 secs 2023-03-09T15:24:41.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:24:40 smithi038 ceph-mon[92422]: pgmap v1364: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 57 KiB/s wr, 9 op/s 2023-03-09T15:24:41.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:24:40 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:24:41.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:24:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:24:41.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:24:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:24:42.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:24:42 smithi122 ceph-mon[93869]: pgmap v1365: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 67 KiB/s wr, 12 op/s 2023-03-09T15:24:42.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:24:42 smithi181 ceph-mon[94189]: pgmap v1365: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 67 KiB/s wr, 12 op/s 2023-03-09T15:24:43.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:24:42 smithi038 ceph-mon[92422]: pgmap v1365: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 67 KiB/s wr, 12 op/s 2023-03-09T15:24:43.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2055.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:24:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:24:44 smithi122 ceph-mon[93869]: pgmap v1366: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 70 KiB/s wr, 11 op/s 2023-03-09T15:24:44.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:24:44 smithi181 ceph-mon[94189]: pgmap v1366: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 70 KiB/s wr, 11 op/s 2023-03-09T15:24:44.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:24:44 smithi038 ceph-mon[92422]: pgmap v1366: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 70 KiB/s wr, 11 op/s 2023-03-09T15:24:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:24:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1595.959189 secs 2023-03-09T15:24:45.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:24:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1595.959189 secs 2023-03-09T15:24:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:24:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1595.959189 secs 2023-03-09T15:24:46.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:24:46 smithi122 ceph-mon[93869]: pgmap v1367: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 63 KiB/s wr, 10 op/s 2023-03-09T15:24:46.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:24:46 smithi181 ceph-mon[94189]: pgmap v1367: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 63 KiB/s wr, 10 op/s 2023-03-09T15:24:46.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:24:46 smithi038 ceph-mon[92422]: pgmap v1367: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 63 KiB/s wr, 10 op/s 2023-03-09T15:24:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2060.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:24:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:24:48 smithi122 ceph-mon[93869]: pgmap v1368: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 93 KiB/s wr, 13 op/s 2023-03-09T15:24:48.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:24:48 smithi181 ceph-mon[94189]: pgmap v1368: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 93 KiB/s wr, 13 op/s 2023-03-09T15:24:49.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:24:48 smithi038 ceph-mon[92422]: pgmap v1368: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 93 KiB/s wr, 13 op/s 2023-03-09T15:24:50.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:24:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1600.959249 secs 2023-03-09T15:24:50.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:24:50 smithi122 ceph-mon[93869]: pgmap v1369: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 76 KiB/s wr, 10 op/s 2023-03-09T15:24:50.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:24:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1600.959249 secs 2023-03-09T15:24:50.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:24:50 smithi181 ceph-mon[94189]: pgmap v1369: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 76 KiB/s wr, 10 op/s 2023-03-09T15:24:51.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:24:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1600.959249 secs 2023-03-09T15:24:51.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:24:50 smithi038 ceph-mon[92422]: pgmap v1369: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 76 KiB/s wr, 10 op/s 2023-03-09T15:24:52.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:24:52 smithi122 ceph-mon[93869]: pgmap v1370: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 104 KiB/s wr, 12 op/s 2023-03-09T15:24:52.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:24:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1603.155984 secs 2023-03-09T15:24:52.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:24:52 smithi181 ceph-mon[94189]: pgmap v1370: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 104 KiB/s wr, 12 op/s 2023-03-09T15:24:52.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:24:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1603.155984 secs 2023-03-09T15:24:53.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:24:52 smithi038 ceph-mon[92422]: pgmap v1370: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 104 KiB/s wr, 12 op/s 2023-03-09T15:24:53.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:24:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1603.155984 secs 2023-03-09T15:24:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2065.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:24:54.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:24:54 smithi122 ceph-mon[93869]: pgmap v1371: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 78 KiB/s wr, 10 op/s 2023-03-09T15:24:54.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:24:54 smithi181 ceph-mon[94189]: pgmap v1371: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 78 KiB/s wr, 10 op/s 2023-03-09T15:24:54.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:24:54 smithi038 ceph-mon[92422]: pgmap v1371: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 78 KiB/s wr, 10 op/s 2023-03-09T15:24:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:24:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1605.959308 secs 2023-03-09T15:24:55.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:24:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1605.959308 secs 2023-03-09T15:24:55.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:24:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1605.959308 secs 2023-03-09T15:24:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:24:56 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:24:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:24:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:24:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:24:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:24:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:24:56 smithi122 ceph-mon[93869]: pgmap v1372: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 75 KiB/s wr, 8 op/s 2023-03-09T15:24:56.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:24:56 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:24:56.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:24:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:24:56.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:24:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:24:56.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:24:56 smithi181 ceph-mon[94189]: pgmap v1372: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 75 KiB/s wr, 8 op/s 2023-03-09T15:24:56.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:24:56 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:24:56.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:24:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:24:56.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:24:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:24:56.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:24:56 smithi038 ceph-mon[92422]: pgmap v1372: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 75 KiB/s wr, 8 op/s 2023-03-09T15:24:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:24:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1608.156085 secs 2023-03-09T15:24:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:24:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1608.156085 secs 2023-03-09T15:24:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:24:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1608.156085 secs 2023-03-09T15:24:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:24:58 smithi122 ceph-mon[93869]: pgmap v1373: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 104 KiB/s wr, 12 op/s 2023-03-09T15:24:58.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:24:58 smithi181 ceph-mon[94189]: pgmap v1373: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 104 KiB/s wr, 12 op/s 2023-03-09T15:24:58.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:24:58 smithi038 ceph-mon[92422]: pgmap v1373: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 104 KiB/s wr, 12 op/s 2023-03-09T15:24:58.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2070.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:25:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:25:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1610.959388 secs 2023-03-09T15:25:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:25:00 smithi122 ceph-mon[93869]: pgmap v1374: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 75 KiB/s wr, 9 op/s 2023-03-09T15:25:00.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:25:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1610.959388 secs 2023-03-09T15:25:00.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:25:00 smithi181 ceph-mon[94189]: pgmap v1374: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 75 KiB/s wr, 9 op/s 2023-03-09T15:25:01.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:25:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1610.959388 secs 2023-03-09T15:25:01.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:25:00 smithi038 ceph-mon[92422]: pgmap v1374: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 75 KiB/s wr, 9 op/s 2023-03-09T15:25:02.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:25:02 smithi122 ceph-mon[93869]: pgmap v1375: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 104 KiB/s wr, 12 op/s 2023-03-09T15:25:02.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:25:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1613.156234 secs 2023-03-09T15:25:02.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:25:02 smithi181 ceph-mon[94189]: pgmap v1375: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 104 KiB/s wr, 12 op/s 2023-03-09T15:25:02.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:25:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1613.156234 secs 2023-03-09T15:25:03.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:25:02 smithi038 ceph-mon[92422]: pgmap v1375: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 104 KiB/s wr, 12 op/s 2023-03-09T15:25:03.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:25:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1613.156234 secs 2023-03-09T15:25:03.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2075.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:25:04.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:25:04 smithi122 ceph-mon[93869]: pgmap v1376: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 85 KiB/s wr, 22 op/s 2023-03-09T15:25:04.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:25:04 smithi181 ceph-mon[94189]: pgmap v1376: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 85 KiB/s wr, 22 op/s 2023-03-09T15:25:04.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:25:04 smithi038 ceph-mon[92422]: pgmap v1376: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 85 KiB/s wr, 22 op/s 2023-03-09T15:25:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:25:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1615.959475 secs 2023-03-09T15:25:05.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:25:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1615.959475 secs 2023-03-09T15:25:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:25:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1615.959475 secs 2023-03-09T15:25:06.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:25:06 smithi122 ceph-mon[93869]: pgmap v1377: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 82 KiB/s wr, 20 op/s 2023-03-09T15:25:06.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:25:06 smithi181 ceph-mon[94189]: pgmap v1377: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 82 KiB/s wr, 20 op/s 2023-03-09T15:25:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:25:06 smithi038 ceph-mon[92422]: pgmap v1377: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 82 KiB/s wr, 20 op/s 2023-03-09T15:25:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:25:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1618.156352 secs 2023-03-09T15:25:07.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:25:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1618.156352 secs 2023-03-09T15:25:07.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:25:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1618.156352 secs 2023-03-09T15:25:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:25:08 smithi122 ceph-mon[93869]: pgmap v1378: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 104 KiB/s wr, 25 op/s 2023-03-09T15:25:08.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:25:08 smithi181 ceph-mon[94189]: pgmap v1378: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 104 KiB/s wr, 25 op/s 2023-03-09T15:25:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:25:08 smithi038 ceph-mon[92422]: pgmap v1378: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 104 KiB/s wr, 25 op/s 2023-03-09T15:25:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2080.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:25:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:25:10 smithi122 ceph-mon[93869]: osdmap e300: 12 total, 12 up, 12 in 2023-03-09T15:25:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:25:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1620.959535 secs 2023-03-09T15:25:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:25:10 smithi122 ceph-mon[93869]: pgmap v1380: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 89 KiB/s wr, 25 op/s 2023-03-09T15:25:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:25:10 smithi181 ceph-mon[94189]: osdmap e300: 12 total, 12 up, 12 in 2023-03-09T15:25:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:25:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1620.959535 secs 2023-03-09T15:25:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:25:10 smithi181 ceph-mon[94189]: pgmap v1380: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 89 KiB/s wr, 25 op/s 2023-03-09T15:25:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:25:10 smithi038 ceph-mon[92422]: osdmap e300: 12 total, 12 up, 12 in 2023-03-09T15:25:10.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:25:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1620.959535 secs 2023-03-09T15:25:10.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:25:10 smithi038 ceph-mon[92422]: pgmap v1380: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 89 KiB/s wr, 25 op/s 2023-03-09T15:25:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:25:12 smithi122 ceph-mon[93869]: pgmap v1381: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 55 KiB/s wr, 23 op/s 2023-03-09T15:25:12.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:25:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1623.156476 secs 2023-03-09T15:25:12.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:25:12 smithi181 ceph-mon[94189]: pgmap v1381: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 55 KiB/s wr, 23 op/s 2023-03-09T15:25:12.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:25:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1623.156476 secs 2023-03-09T15:25:13.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:25:12 smithi038 ceph-mon[92422]: pgmap v1381: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 55 KiB/s wr, 23 op/s 2023-03-09T15:25:13.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:25:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1623.156476 secs 2023-03-09T15:25:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2085.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:25:14.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:25:14 smithi122 ceph-mon[93869]: pgmap v1382: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 28 KiB/s wr, 9 op/s 2023-03-09T15:25:14.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:25:14 smithi181 ceph-mon[94189]: pgmap v1382: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 28 KiB/s wr, 9 op/s 2023-03-09T15:25:14.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:25:14 smithi038 ceph-mon[92422]: pgmap v1382: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 28 KiB/s wr, 9 op/s 2023-03-09T15:25:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:25:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1625.959598 secs 2023-03-09T15:25:15.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:25:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1625.959598 secs 2023-03-09T15:25:15.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:25:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1625.959598 secs 2023-03-09T15:25:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:25:16 smithi122 ceph-mon[93869]: pgmap v1383: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 28 KiB/s wr, 9 op/s 2023-03-09T15:25:16.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:25:16 smithi181 ceph-mon[94189]: pgmap v1383: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 28 KiB/s wr, 9 op/s 2023-03-09T15:25:16.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:25:16 smithi038 ceph-mon[92422]: pgmap v1383: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 28 KiB/s wr, 9 op/s 2023-03-09T15:25:17.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:25:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1628.156593 secs 2023-03-09T15:25:17.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:25:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1628.156593 secs 2023-03-09T15:25:17.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:25:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1628.156593 secs 2023-03-09T15:25:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:25:18 smithi122 ceph-mon[93869]: pgmap v1384: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 7 op/s 2023-03-09T15:25:18.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:25:18 smithi122 ceph-mon[93869]: osdmap e301: 12 total, 12 up, 12 in 2023-03-09T15:25:18.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:25:18 smithi181 ceph-mon[94189]: pgmap v1384: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 7 op/s 2023-03-09T15:25:18.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:25:18 smithi181 ceph-mon[94189]: osdmap e301: 12 total, 12 up, 12 in 2023-03-09T15:25:18.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:25:18 smithi038 ceph-mon[92422]: pgmap v1384: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 7 op/s 2023-03-09T15:25:18.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:25:18 smithi038 ceph-mon[92422]: osdmap e301: 12 total, 12 up, 12 in 2023-03-09T15:25:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2090.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:25:20.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:25:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1630.959682 secs 2023-03-09T15:25:20.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:25:20 smithi122 ceph-mon[93869]: pgmap v1386: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 8 op/s 2023-03-09T15:25:20.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:25:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1630.959682 secs 2023-03-09T15:25:20.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:25:20 smithi181 ceph-mon[94189]: pgmap v1386: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 8 op/s 2023-03-09T15:25:21.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:25:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1630.959682 secs 2023-03-09T15:25:21.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:25:20 smithi038 ceph-mon[92422]: pgmap v1386: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 8 op/s 2023-03-09T15:25:22.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:25:22 smithi122 ceph-mon[93869]: pgmap v1387: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 28 KiB/s wr, 8 op/s 2023-03-09T15:25:22.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:25:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1633.156723 secs 2023-03-09T15:25:22.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:25:22 smithi181 ceph-mon[94189]: pgmap v1387: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 28 KiB/s wr, 8 op/s 2023-03-09T15:25:22.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:25:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1633.156723 secs 2023-03-09T15:25:23.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:25:22 smithi038 ceph-mon[92422]: pgmap v1387: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 28 KiB/s wr, 8 op/s 2023-03-09T15:25:23.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:25:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1633.156723 secs 2023-03-09T15:25:23.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2095.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:25:24.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:25:24 smithi122 ceph-mon[93869]: pgmap v1388: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 8 op/s 2023-03-09T15:25:24.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:25:24 smithi181 ceph-mon[94189]: pgmap v1388: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 8 op/s 2023-03-09T15:25:24.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:25:24 smithi038 ceph-mon[92422]: pgmap v1388: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 8 op/s 2023-03-09T15:25:25.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:25:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1635.959745 secs 2023-03-09T15:25:25.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:25:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1635.959745 secs 2023-03-09T15:25:25.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:25:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1635.959745 secs 2023-03-09T15:25:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:25:26 smithi122 ceph-mon[93869]: pgmap v1389: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 8 op/s 2023-03-09T15:25:26.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:25:26 smithi181 ceph-mon[94189]: pgmap v1389: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 8 op/s 2023-03-09T15:25:26.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:25:26 smithi038 ceph-mon[92422]: pgmap v1389: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 8 op/s 2023-03-09T15:25:27.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:25:27 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:25:27.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:25:27 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:25:27.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:25:27 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:25:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:25:28 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:25:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:25:28 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:25:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:25:28 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:25:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:25:28 smithi122 ceph-mon[93869]: pgmap v1390: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 32 KiB/s wr, 8 op/s 2023-03-09T15:25:28.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:25:28 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:25:28.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:25:28 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:25:28.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:25:28 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:25:28.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:25:28 smithi181 ceph-mon[94189]: pgmap v1390: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 32 KiB/s wr, 8 op/s 2023-03-09T15:25:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:25:28 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:25:28.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:25:28 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:25:28.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:25:28 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:25:28.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:25:28 smithi038 ceph-mon[92422]: pgmap v1390: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 32 KiB/s wr, 8 op/s 2023-03-09T15:25:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2100.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:25:30.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:25:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1640.959804 secs 2023-03-09T15:25:30.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:25:30 smithi122 ceph-mon[93869]: pgmap v1391: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 29 KiB/s wr, 7 op/s 2023-03-09T15:25:30.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:25:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1640.959804 secs 2023-03-09T15:25:30.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:25:30 smithi181 ceph-mon[94189]: pgmap v1391: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 29 KiB/s wr, 7 op/s 2023-03-09T15:25:31.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:25:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1640.959804 secs 2023-03-09T15:25:31.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:25:30 smithi038 ceph-mon[92422]: pgmap v1391: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 29 KiB/s wr, 7 op/s 2023-03-09T15:25:32.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:25:32 smithi122 ceph-mon[93869]: pgmap v1392: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 50 KiB/s wr, 9 op/s 2023-03-09T15:25:32.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:25:32 smithi181 ceph-mon[94189]: pgmap v1392: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 50 KiB/s wr, 9 op/s 2023-03-09T15:25:33.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:25:32 smithi038 ceph-mon[92422]: pgmap v1392: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 50 KiB/s wr, 9 op/s 2023-03-09T15:25:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2105.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:25:34.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:25:34 smithi122 ceph-mon[93869]: pgmap v1393: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 39 KiB/s wr, 8 op/s 2023-03-09T15:25:34.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:25:34 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:25:34.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:25:34 smithi181 ceph-mon[94189]: pgmap v1393: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 39 KiB/s wr, 8 op/s 2023-03-09T15:25:34.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:25:34 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:25:34.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:25:34 smithi038 ceph-mon[92422]: pgmap v1393: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 39 KiB/s wr, 8 op/s 2023-03-09T15:25:34.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:25:34 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:25:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:25:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1645.959867 secs 2023-03-09T15:25:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:25:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:25:35.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:25:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:25:35.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:25:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:25:35.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:25:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1645.959867 secs 2023-03-09T15:25:35.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:25:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:25:35.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:25:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:25:35.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:25:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:25:35.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:25:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1645.959867 secs 2023-03-09T15:25:35.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:25:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:25:35.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:25:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:25:35.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:25:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:25:36.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:25:36 smithi122 ceph-mon[93869]: pgmap v1394: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 39 KiB/s wr, 7 op/s 2023-03-09T15:25:36.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:25:36 smithi181 ceph-mon[94189]: pgmap v1394: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 39 KiB/s wr, 7 op/s 2023-03-09T15:25:36.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:25:36 smithi038 ceph-mon[92422]: pgmap v1394: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 39 KiB/s wr, 7 op/s 2023-03-09T15:25:37.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:25:37 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 1648.157068 secs 2023-03-09T15:25:37.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:25:37 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 1648.157068 secs 2023-03-09T15:25:37.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:25:37 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 1648.157068 secs 2023-03-09T15:25:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:25:38 smithi122 ceph-mon[93869]: pgmap v1395: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 75 KiB/s wr, 11 op/s 2023-03-09T15:25:38.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:25:38 smithi181 ceph-mon[94189]: pgmap v1395: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 75 KiB/s wr, 11 op/s 2023-03-09T15:25:38.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:25:38 smithi038 ceph-mon[92422]: pgmap v1395: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 75 KiB/s wr, 11 op/s 2023-03-09T15:25:38.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2110.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:25:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:25:39 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:25:39.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:25:39 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:25:39.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:25:39 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:25:39.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:25:39 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:25:39.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:25:39 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:25:39.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:25:39 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:25:39.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:25:39 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:25:39.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:25:39 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:25:39.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:25:39 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:25:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:25:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1650.959929 secs 2023-03-09T15:25:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:25:40 smithi122 ceph-mon[93869]: pgmap v1396: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 63 KiB/s wr, 9 op/s 2023-03-09T15:25:40.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:25:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1650.959929 secs 2023-03-09T15:25:40.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:25:40 smithi181 ceph-mon[94189]: pgmap v1396: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 63 KiB/s wr, 9 op/s 2023-03-09T15:25:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:25:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1650.959929 secs 2023-03-09T15:25:40.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:25:40 smithi038 ceph-mon[92422]: pgmap v1396: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 63 KiB/s wr, 9 op/s 2023-03-09T15:25:42.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:25:42 smithi122 ceph-mon[93869]: pgmap v1397: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 105 KiB/s wr, 12 op/s 2023-03-09T15:25:42.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:25:42 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 1653.157176 secs 2023-03-09T15:25:42.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:25:42 smithi181 ceph-mon[94189]: pgmap v1397: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 105 KiB/s wr, 12 op/s 2023-03-09T15:25:42.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:25:42 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 1653.157176 secs 2023-03-09T15:25:43.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:25:42 smithi038 ceph-mon[92422]: pgmap v1397: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 105 KiB/s wr, 12 op/s 2023-03-09T15:25:43.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:25:42 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 1653.157176 secs 2023-03-09T15:25:43.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2115.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:25:44.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:25:44 smithi122 ceph-mon[93869]: pgmap v1398: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 84 KiB/s wr, 10 op/s 2023-03-09T15:25:44.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:25:44 smithi181 ceph-mon[94189]: pgmap v1398: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 84 KiB/s wr, 10 op/s 2023-03-09T15:25:44.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:25:44 smithi038 ceph-mon[92422]: pgmap v1398: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 84 KiB/s wr, 10 op/s 2023-03-09T15:25:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:25:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1655.959982 secs 2023-03-09T15:25:45.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:25:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1655.959982 secs 2023-03-09T15:25:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:25:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1655.959982 secs 2023-03-09T15:25:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:25:46 smithi122 ceph-mon[93869]: pgmap v1399: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 81 KiB/s wr, 9 op/s 2023-03-09T15:25:46.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:25:46 smithi181 ceph-mon[94189]: pgmap v1399: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 81 KiB/s wr, 9 op/s 2023-03-09T15:25:46.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:25:46 smithi038 ceph-mon[92422]: pgmap v1399: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 81 KiB/s wr, 9 op/s 2023-03-09T15:25:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:25:47 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 1658.157272 secs 2023-03-09T15:25:47.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:25:47 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 1658.157272 secs 2023-03-09T15:25:47.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:25:47 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 1658.157272 secs 2023-03-09T15:25:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:25:48 smithi122 ceph-mon[93869]: pgmap v1400: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 104 KiB/s wr, 13 op/s 2023-03-09T15:25:48.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:25:48 smithi181 ceph-mon[94189]: pgmap v1400: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 104 KiB/s wr, 13 op/s 2023-03-09T15:25:48.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:25:48 smithi038 ceph-mon[92422]: pgmap v1400: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 104 KiB/s wr, 13 op/s 2023-03-09T15:25:48.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2120.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:25:50.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:25:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1660.960042 secs 2023-03-09T15:25:50.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:25:50 smithi122 ceph-mon[93869]: pgmap v1401: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 71 KiB/s wr, 9 op/s 2023-03-09T15:25:50.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:25:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1660.960042 secs 2023-03-09T15:25:50.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:25:50 smithi181 ceph-mon[94189]: pgmap v1401: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 71 KiB/s wr, 9 op/s 2023-03-09T15:25:51.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:25:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1660.960042 secs 2023-03-09T15:25:51.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:25:50 smithi038 ceph-mon[92422]: pgmap v1401: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 71 KiB/s wr, 9 op/s 2023-03-09T15:25:52.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:25:52 smithi122 ceph-mon[93869]: pgmap v1402: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 95 KiB/s wr, 12 op/s 2023-03-09T15:25:52.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:25:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1663.157366 secs 2023-03-09T15:25:52.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:25:52 smithi181 ceph-mon[94189]: pgmap v1402: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 95 KiB/s wr, 12 op/s 2023-03-09T15:25:52.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:25:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1663.157366 secs 2023-03-09T15:25:53.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:25:52 smithi038 ceph-mon[92422]: pgmap v1402: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 95 KiB/s wr, 12 op/s 2023-03-09T15:25:53.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:25:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1663.157366 secs 2023-03-09T15:25:53.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2125.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:25:54.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:25:54 smithi122 ceph-mon[93869]: pgmap v1403: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 54 KiB/s wr, 10 op/s 2023-03-09T15:25:54.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:25:54 smithi181 ceph-mon[94189]: pgmap v1403: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 54 KiB/s wr, 10 op/s 2023-03-09T15:25:54.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:25:54 smithi038 ceph-mon[92422]: pgmap v1403: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 54 KiB/s wr, 10 op/s 2023-03-09T15:25:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:25:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1665.960104 secs 2023-03-09T15:25:55.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:25:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1665.960104 secs 2023-03-09T15:25:55.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:25:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1665.960104 secs 2023-03-09T15:25:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:25:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:25:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:25:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:25:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:25:56 smithi122 ceph-mon[93869]: pgmap v1404: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 67 KiB/s wr, 9 op/s 2023-03-09T15:25:56.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:25:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:25:56.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:25:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:25:56.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:25:56 smithi181 ceph-mon[94189]: pgmap v1404: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 67 KiB/s wr, 9 op/s 2023-03-09T15:25:56.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:25:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:25:56.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:25:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:25:56.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:25:56 smithi038 ceph-mon[92422]: pgmap v1404: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 67 KiB/s wr, 9 op/s 2023-03-09T15:25:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:25:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1668.157435 secs 2023-03-09T15:25:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:25:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1668.157435 secs 2023-03-09T15:25:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:25:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1668.157435 secs 2023-03-09T15:25:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:25:58 smithi122 ceph-mon[93869]: pgmap v1405: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 67 KiB/s wr, 12 op/s 2023-03-09T15:25:58.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:25:58 smithi181 ceph-mon[94189]: pgmap v1405: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 67 KiB/s wr, 12 op/s 2023-03-09T15:25:58.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:25:58 smithi038 ceph-mon[92422]: pgmap v1405: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 67 KiB/s wr, 12 op/s 2023-03-09T15:25:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2130.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:26:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:26:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1670.960163 secs 2023-03-09T15:26:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:26:00 smithi122 ceph-mon[93869]: pgmap v1406: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 53 KiB/s wr, 9 op/s 2023-03-09T15:26:00.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:26:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1670.960163 secs 2023-03-09T15:26:00.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:26:00 smithi181 ceph-mon[94189]: pgmap v1406: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 53 KiB/s wr, 9 op/s 2023-03-09T15:26:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:26:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1670.960163 secs 2023-03-09T15:26:01.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:26:00 smithi038 ceph-mon[92422]: pgmap v1406: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 53 KiB/s wr, 9 op/s 2023-03-09T15:26:02.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:26:02 smithi122 ceph-mon[93869]: pgmap v1407: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 50 KiB/s wr, 11 op/s 2023-03-09T15:26:02.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:26:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1673.157572 secs 2023-03-09T15:26:02.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:26:02 smithi181 ceph-mon[94189]: pgmap v1407: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 50 KiB/s wr, 11 op/s 2023-03-09T15:26:02.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:26:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1673.157572 secs 2023-03-09T15:26:03.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:26:02 smithi038 ceph-mon[92422]: pgmap v1407: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 50 KiB/s wr, 11 op/s 2023-03-09T15:26:03.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:26:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1673.157572 secs 2023-03-09T15:26:03.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2135.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:26:04.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:26:04 smithi122 ceph-mon[93869]: pgmap v1408: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 49 KiB/s wr, 21 op/s 2023-03-09T15:26:04.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:26:04 smithi181 ceph-mon[94189]: pgmap v1408: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 49 KiB/s wr, 21 op/s 2023-03-09T15:26:04.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:26:04 smithi038 ceph-mon[92422]: pgmap v1408: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 49 KiB/s wr, 21 op/s 2023-03-09T15:26:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:26:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1675.960225 secs 2023-03-09T15:26:05.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:26:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1675.960225 secs 2023-03-09T15:26:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:26:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1675.960225 secs 2023-03-09T15:26:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:26:06 smithi122 ceph-mon[93869]: pgmap v1409: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 59 KiB/s wr, 21 op/s 2023-03-09T15:26:06.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:26:06 smithi181 ceph-mon[94189]: pgmap v1409: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 59 KiB/s wr, 21 op/s 2023-03-09T15:26:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:26:06 smithi038 ceph-mon[92422]: pgmap v1409: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 59 KiB/s wr, 21 op/s 2023-03-09T15:26:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:26:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1678.157717 secs 2023-03-09T15:26:07.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:26:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1678.157717 secs 2023-03-09T15:26:07.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:26:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1678.157717 secs 2023-03-09T15:26:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:26:08 smithi122 ceph-mon[93869]: pgmap v1410: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 66 KiB/s wr, 25 op/s 2023-03-09T15:26:08.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:26:08 smithi181 ceph-mon[94189]: pgmap v1410: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 66 KiB/s wr, 25 op/s 2023-03-09T15:26:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:26:08 smithi038 ceph-mon[92422]: pgmap v1410: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 66 KiB/s wr, 25 op/s 2023-03-09T15:26:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2140.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:26:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:26:10 smithi122 ceph-mon[93869]: osdmap e302: 12 total, 12 up, 12 in 2023-03-09T15:26:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:26:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1680.960314 secs 2023-03-09T15:26:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:26:10 smithi122 ceph-mon[93869]: pgmap v1412: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 69 KiB/s wr, 26 op/s 2023-03-09T15:26:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:26:10 smithi181 ceph-mon[94189]: osdmap e302: 12 total, 12 up, 12 in 2023-03-09T15:26:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:26:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1680.960314 secs 2023-03-09T15:26:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:26:10 smithi181 ceph-mon[94189]: pgmap v1412: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 69 KiB/s wr, 26 op/s 2023-03-09T15:26:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:26:10 smithi038 ceph-mon[92422]: osdmap e302: 12 total, 12 up, 12 in 2023-03-09T15:26:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:26:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1680.960314 secs 2023-03-09T15:26:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:26:10 smithi038 ceph-mon[92422]: pgmap v1412: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 69 KiB/s wr, 26 op/s 2023-03-09T15:26:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:26:12 smithi122 ceph-mon[93869]: pgmap v1413: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 69 KiB/s wr, 25 op/s 2023-03-09T15:26:12.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:26:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1683.157839 secs 2023-03-09T15:26:12.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:26:12 smithi181 ceph-mon[94189]: pgmap v1413: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 69 KiB/s wr, 25 op/s 2023-03-09T15:26:12.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:26:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1683.157839 secs 2023-03-09T15:26:13.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:26:12 smithi038 ceph-mon[92422]: pgmap v1413: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 69 KiB/s wr, 25 op/s 2023-03-09T15:26:13.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:26:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1683.157839 secs 2023-03-09T15:26:13.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2145.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:26:14.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:26:14 smithi122 ceph-mon[93869]: pgmap v1414: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 41 KiB/s wr, 11 op/s 2023-03-09T15:26:14.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:26:14 smithi181 ceph-mon[94189]: pgmap v1414: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 41 KiB/s wr, 11 op/s 2023-03-09T15:26:14.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:26:14 smithi038 ceph-mon[92422]: pgmap v1414: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 41 KiB/s wr, 11 op/s 2023-03-09T15:26:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:26:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1685.960372 secs 2023-03-09T15:26:15.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:26:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1685.960372 secs 2023-03-09T15:26:15.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:26:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1685.960372 secs 2023-03-09T15:26:16.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:26:16 smithi122 ceph-mon[93869]: pgmap v1415: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 29 KiB/s wr, 10 op/s 2023-03-09T15:26:16.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:26:16 smithi181 ceph-mon[94189]: pgmap v1415: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 29 KiB/s wr, 10 op/s 2023-03-09T15:26:16.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:26:16 smithi038 ceph-mon[92422]: pgmap v1415: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 29 KiB/s wr, 10 op/s 2023-03-09T15:26:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:26:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1688.157949 secs 2023-03-09T15:26:17.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:26:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1688.157949 secs 2023-03-09T15:26:17.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:26:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1688.157949 secs 2023-03-09T15:26:18.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:26:18 smithi122 ceph-mon[93869]: pgmap v1416: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2023-03-09T15:26:18.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:26:18 smithi122 ceph-mon[93869]: osdmap e303: 12 total, 12 up, 12 in 2023-03-09T15:26:18.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:26:18 smithi181 ceph-mon[94189]: pgmap v1416: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2023-03-09T15:26:18.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:26:18 smithi181 ceph-mon[94189]: osdmap e303: 12 total, 12 up, 12 in 2023-03-09T15:26:18.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:26:18 smithi038 ceph-mon[92422]: pgmap v1416: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2023-03-09T15:26:18.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:26:18 smithi038 ceph-mon[92422]: osdmap e303: 12 total, 12 up, 12 in 2023-03-09T15:26:18.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2150.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:26:20.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:26:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1690.960433 secs 2023-03-09T15:26:20.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:26:20 smithi122 ceph-mon[93869]: pgmap v1418: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T15:26:20.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:26:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1690.960433 secs 2023-03-09T15:26:20.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:26:20 smithi181 ceph-mon[94189]: pgmap v1418: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T15:26:21.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:26:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1690.960433 secs 2023-03-09T15:26:21.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:26:20 smithi038 ceph-mon[92422]: pgmap v1418: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T15:26:22.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:26:22 smithi122 ceph-mon[93869]: pgmap v1419: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 7 op/s 2023-03-09T15:26:22.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:26:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1693.158084 secs 2023-03-09T15:26:22.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:26:22 smithi181 ceph-mon[94189]: pgmap v1419: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 7 op/s 2023-03-09T15:26:22.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:26:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1693.158084 secs 2023-03-09T15:26:23.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:26:22 smithi038 ceph-mon[92422]: pgmap v1419: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 7 op/s 2023-03-09T15:26:23.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:26:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1693.158084 secs 2023-03-09T15:26:23.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2155.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:26:24.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:26:24 smithi122 ceph-mon[93869]: pgmap v1420: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 8 op/s 2023-03-09T15:26:24.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:26:24 smithi181 ceph-mon[94189]: pgmap v1420: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 8 op/s 2023-03-09T15:26:24.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:26:24 smithi038 ceph-mon[92422]: pgmap v1420: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 8 op/s 2023-03-09T15:26:25.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:26:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1695.960494 secs 2023-03-09T15:26:25.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:26:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1695.960494 secs 2023-03-09T15:26:25.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:26:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1695.960494 secs 2023-03-09T15:26:26.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:26:26 smithi122 ceph-mon[93869]: pgmap v1421: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 7 op/s 2023-03-09T15:26:26.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:26:26 smithi181 ceph-mon[94189]: pgmap v1421: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 7 op/s 2023-03-09T15:26:26.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:26:26 smithi038 ceph-mon[92422]: pgmap v1421: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 7 op/s 2023-03-09T15:26:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:26:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1698.158168 secs 2023-03-09T15:26:27.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:26:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1698.158168 secs 2023-03-09T15:26:27.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:26:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1698.158168 secs 2023-03-09T15:26:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:26:28 smithi122 ceph-mon[93869]: pgmap v1422: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 KiB/s wr, 7 op/s 2023-03-09T15:26:28.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:26:28 smithi181 ceph-mon[94189]: pgmap v1422: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 KiB/s wr, 7 op/s 2023-03-09T15:26:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:26:28 smithi038 ceph-mon[92422]: pgmap v1422: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 KiB/s wr, 7 op/s 2023-03-09T15:26:28.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2160.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:26:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:26:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1700.960554 secs 2023-03-09T15:26:30.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:26:30 smithi122 ceph-mon[93869]: pgmap v1423: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 41 KiB/s wr, 8 op/s 2023-03-09T15:26:30.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:26:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1700.960554 secs 2023-03-09T15:26:30.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:26:30 smithi181 ceph-mon[94189]: pgmap v1423: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 41 KiB/s wr, 8 op/s 2023-03-09T15:26:31.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:26:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1700.960554 secs 2023-03-09T15:26:31.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:26:30 smithi038 ceph-mon[92422]: pgmap v1423: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 41 KiB/s wr, 8 op/s 2023-03-09T15:26:32.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:26:32 smithi122 ceph-mon[93869]: pgmap v1424: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 25 KiB/s wr, 8 op/s 2023-03-09T15:26:32.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:26:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1703.158307 secs 2023-03-09T15:26:32.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:26:32 smithi181 ceph-mon[94189]: pgmap v1424: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 25 KiB/s wr, 8 op/s 2023-03-09T15:26:32.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:26:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1703.158307 secs 2023-03-09T15:26:33.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:26:32 smithi038 ceph-mon[92422]: pgmap v1424: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 25 KiB/s wr, 8 op/s 2023-03-09T15:26:33.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:26:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1703.158307 secs 2023-03-09T15:26:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2165.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:26:34.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:26:34 smithi122 ceph-mon[93869]: pgmap v1425: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 27 KiB/s wr, 8 op/s 2023-03-09T15:26:34.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:26:34 smithi181 ceph-mon[94189]: pgmap v1425: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 27 KiB/s wr, 8 op/s 2023-03-09T15:26:34.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:26:34 smithi038 ceph-mon[92422]: pgmap v1425: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 27 KiB/s wr, 8 op/s 2023-03-09T15:26:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:26:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1705.960616 secs 2023-03-09T15:26:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:26:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:26:35.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:26:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1705.960616 secs 2023-03-09T15:26:35.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:26:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:26:35.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:26:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1705.960616 secs 2023-03-09T15:26:35.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:26:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:26:36.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:26:36 smithi122 ceph-mon[93869]: pgmap v1426: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 57 KiB/s wr, 8 op/s 2023-03-09T15:26:36.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:26:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:26:36.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:26:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:26:36.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:26:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:26:36.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:26:36 smithi181 ceph-mon[94189]: pgmap v1426: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 57 KiB/s wr, 8 op/s 2023-03-09T15:26:36.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:26:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:26:36.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:26:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:26:36.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:26:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:26:36.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:26:36 smithi038 ceph-mon[92422]: pgmap v1426: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 57 KiB/s wr, 8 op/s 2023-03-09T15:26:36.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:26:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:26:36.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:26:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:26:36.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:26:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:26:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:26:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1708.158411 secs 2023-03-09T15:26:37.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:26:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1708.158411 secs 2023-03-09T15:26:37.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:26:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1708.158411 secs 2023-03-09T15:26:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:26:38 smithi122 ceph-mon[93869]: pgmap v1427: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 60 KiB/s wr, 10 op/s 2023-03-09T15:26:38.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:26:38 smithi181 ceph-mon[94189]: pgmap v1427: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 60 KiB/s wr, 10 op/s 2023-03-09T15:26:38.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:26:38 smithi038 ceph-mon[92422]: pgmap v1427: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 60 KiB/s wr, 10 op/s 2023-03-09T15:26:38.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2170.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:26:40.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:26:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1710.960677 secs 2023-03-09T15:26:40.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:26:40 smithi122 ceph-mon[93869]: pgmap v1428: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 90 KiB/s wr, 10 op/s 2023-03-09T15:26:40.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:26:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1710.960677 secs 2023-03-09T15:26:40.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:26:40 smithi181 ceph-mon[94189]: pgmap v1428: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 90 KiB/s wr, 10 op/s 2023-03-09T15:26:41.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:26:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1710.960677 secs 2023-03-09T15:26:41.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:26:40 smithi038 ceph-mon[92422]: pgmap v1428: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 90 KiB/s wr, 10 op/s 2023-03-09T15:26:42.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:26:42 smithi122 ceph-mon[93869]: pgmap v1429: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 66 KiB/s wr, 10 op/s 2023-03-09T15:26:42.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:26:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1713.158531 secs 2023-03-09T15:26:42.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:26:42 smithi181 ceph-mon[94189]: pgmap v1429: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 66 KiB/s wr, 10 op/s 2023-03-09T15:26:42.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:26:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1713.158531 secs 2023-03-09T15:26:43.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:26:42 smithi038 ceph-mon[92422]: pgmap v1429: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 66 KiB/s wr, 10 op/s 2023-03-09T15:26:43.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:26:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1713.158531 secs 2023-03-09T15:26:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2175.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:26:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:26:44 smithi122 ceph-mon[93869]: pgmap v1430: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 69 KiB/s wr, 10 op/s 2023-03-09T15:26:44.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:26:44 smithi181 ceph-mon[94189]: pgmap v1430: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 69 KiB/s wr, 10 op/s 2023-03-09T15:26:44.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:26:44 smithi038 ceph-mon[92422]: pgmap v1430: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 69 KiB/s wr, 10 op/s 2023-03-09T15:26:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:26:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1715.960736 secs 2023-03-09T15:26:45.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:26:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1715.960736 secs 2023-03-09T15:26:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:26:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1715.960736 secs 2023-03-09T15:26:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:26:46 smithi122 ceph-mon[93869]: pgmap v1431: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 94 KiB/s wr, 9 op/s 2023-03-09T15:26:46.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:26:46 smithi181 ceph-mon[94189]: pgmap v1431: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 94 KiB/s wr, 9 op/s 2023-03-09T15:26:46.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:26:46 smithi038 ceph-mon[92422]: pgmap v1431: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 94 KiB/s wr, 9 op/s 2023-03-09T15:26:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:26:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1718.158659 secs 2023-03-09T15:26:47.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:26:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1718.158659 secs 2023-03-09T15:26:47.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:26:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1718.158659 secs 2023-03-09T15:26:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:26:48 smithi122 ceph-mon[93869]: pgmap v1432: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 67 KiB/s wr, 11 op/s 2023-03-09T15:26:48.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:26:48 smithi181 ceph-mon[94189]: pgmap v1432: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 67 KiB/s wr, 11 op/s 2023-03-09T15:26:48.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:26:48 smithi038 ceph-mon[92422]: pgmap v1432: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 67 KiB/s wr, 11 op/s 2023-03-09T15:26:48.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2180.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:26:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:26:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1720.960796 secs 2023-03-09T15:26:50.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:26:50 smithi122 ceph-mon[93869]: pgmap v1433: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 96 KiB/s wr, 10 op/s 2023-03-09T15:26:50.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:26:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1720.960796 secs 2023-03-09T15:26:50.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:26:50 smithi181 ceph-mon[94189]: pgmap v1433: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 96 KiB/s wr, 10 op/s 2023-03-09T15:26:51.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:26:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1720.960796 secs 2023-03-09T15:26:51.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:26:50 smithi038 ceph-mon[92422]: pgmap v1433: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 96 KiB/s wr, 10 op/s 2023-03-09T15:26:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:26:52 smithi122 ceph-mon[93869]: pgmap v1434: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 66 KiB/s wr, 10 op/s 2023-03-09T15:26:52.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:26:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1723.158778 secs 2023-03-09T15:26:52.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:26:52 smithi181 ceph-mon[94189]: pgmap v1434: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 66 KiB/s wr, 10 op/s 2023-03-09T15:26:52.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:26:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1723.158778 secs 2023-03-09T15:26:53.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:26:52 smithi038 ceph-mon[92422]: pgmap v1434: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 66 KiB/s wr, 10 op/s 2023-03-09T15:26:53.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:26:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1723.158778 secs 2023-03-09T15:26:53.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2185.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:26:54.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:26:54 smithi122 ceph-mon[93869]: pgmap v1435: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 69 KiB/s wr, 9 op/s 2023-03-09T15:26:54.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:26:54 smithi181 ceph-mon[94189]: pgmap v1435: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 69 KiB/s wr, 9 op/s 2023-03-09T15:26:54.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:26:54 smithi038 ceph-mon[92422]: pgmap v1435: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 69 KiB/s wr, 9 op/s 2023-03-09T15:26:55.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:26:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1725.960856 secs 2023-03-09T15:26:55.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:26:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1725.960856 secs 2023-03-09T15:26:55.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:26:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1725.960856 secs 2023-03-09T15:26:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:26:56 smithi122 ceph-mon[93869]: pgmap v1436: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 94 KiB/s wr, 9 op/s 2023-03-09T15:26:56.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:26:56 smithi181 ceph-mon[94189]: pgmap v1436: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 94 KiB/s wr, 9 op/s 2023-03-09T15:26:56.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:26:56 smithi038 ceph-mon[92422]: pgmap v1436: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 94 KiB/s wr, 9 op/s 2023-03-09T15:26:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:26:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1728.158894 secs 2023-03-09T15:26:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:26:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1728.158894 secs 2023-03-09T15:26:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:26:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1728.158894 secs 2023-03-09T15:26:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:26:58 smithi122 ceph-mon[93869]: pgmap v1437: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 67 KiB/s wr, 10 op/s 2023-03-09T15:26:58.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:26:58 smithi181 ceph-mon[94189]: pgmap v1437: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 67 KiB/s wr, 10 op/s 2023-03-09T15:26:58.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:26:58 smithi038 ceph-mon[92422]: pgmap v1437: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 67 KiB/s wr, 10 op/s 2023-03-09T15:26:58.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2190.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:27:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1730.960919 secs 2023-03-09T15:27:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:00 smithi122 ceph-mon[93869]: pgmap v1438: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 82 KiB/s wr, 9 op/s 2023-03-09T15:27:00.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1730.960919 secs 2023-03-09T15:27:00.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:00 smithi181 ceph-mon[94189]: pgmap v1438: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 82 KiB/s wr, 9 op/s 2023-03-09T15:27:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1730.960919 secs 2023-03-09T15:27:01.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:00 smithi038 ceph-mon[92422]: pgmap v1438: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 82 KiB/s wr, 9 op/s 2023-03-09T15:27:02.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:02 smithi122 ceph-mon[93869]: pgmap v1439: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 50 KiB/s wr, 9 op/s 2023-03-09T15:27:02.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1733.159013 secs 2023-03-09T15:27:02.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:02 smithi181 ceph-mon[94189]: pgmap v1439: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 50 KiB/s wr, 9 op/s 2023-03-09T15:27:02.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1733.159013 secs 2023-03-09T15:27:03.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:02 smithi038 ceph-mon[92422]: pgmap v1439: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 50 KiB/s wr, 9 op/s 2023-03-09T15:27:03.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1733.159013 secs 2023-03-09T15:27:03.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2195.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:27:04.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:04 smithi122 ceph-mon[93869]: pgmap v1440: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 73 KiB/s wr, 19 op/s 2023-03-09T15:27:04.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:04 smithi181 ceph-mon[94189]: pgmap v1440: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 73 KiB/s wr, 19 op/s 2023-03-09T15:27:04.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:04 smithi038 ceph-mon[92422]: pgmap v1440: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 73 KiB/s wr, 19 op/s 2023-03-09T15:27:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1735.961003 secs 2023-03-09T15:27:05.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1735.961003 secs 2023-03-09T15:27:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1735.961003 secs 2023-03-09T15:27:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:06 smithi122 ceph-mon[93869]: pgmap v1441: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 82 KiB/s wr, 19 op/s 2023-03-09T15:27:06.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:06 smithi181 ceph-mon[94189]: pgmap v1441: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 82 KiB/s wr, 19 op/s 2023-03-09T15:27:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:06 smithi038 ceph-mon[92422]: pgmap v1441: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 82 KiB/s wr, 19 op/s 2023-03-09T15:27:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1738.159161 secs 2023-03-09T15:27:07.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1738.159161 secs 2023-03-09T15:27:07.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1738.159161 secs 2023-03-09T15:27:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:08 smithi122 ceph-mon[93869]: pgmap v1442: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 78 KiB/s wr, 23 op/s 2023-03-09T15:27:08.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:08 smithi181 ceph-mon[94189]: pgmap v1442: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 78 KiB/s wr, 23 op/s 2023-03-09T15:27:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:08 smithi038 ceph-mon[92422]: pgmap v1442: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 78 KiB/s wr, 23 op/s 2023-03-09T15:27:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2200.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:27:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:10 smithi122 ceph-mon[93869]: osdmap e304: 12 total, 12 up, 12 in 2023-03-09T15:27:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1740.963286 secs 2023-03-09T15:27:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:10 smithi122 ceph-mon[93869]: pgmap v1444: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 73 KiB/s wr, 25 op/s 2023-03-09T15:27:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:10 smithi181 ceph-mon[94189]: osdmap e304: 12 total, 12 up, 12 in 2023-03-09T15:27:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1740.963286 secs 2023-03-09T15:27:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:10 smithi181 ceph-mon[94189]: pgmap v1444: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 73 KiB/s wr, 25 op/s 2023-03-09T15:27:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:10 smithi038 ceph-mon[92422]: osdmap e304: 12 total, 12 up, 12 in 2023-03-09T15:27:10.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1740.963286 secs 2023-03-09T15:27:10.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:10 smithi038 ceph-mon[92422]: pgmap v1444: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 73 KiB/s wr, 25 op/s 2023-03-09T15:27:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:12 smithi122 ceph-mon[93869]: pgmap v1445: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 73 KiB/s wr, 25 op/s 2023-03-09T15:27:12.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1743.159275 secs 2023-03-09T15:27:12.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:12 smithi181 ceph-mon[94189]: pgmap v1445: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 73 KiB/s wr, 25 op/s 2023-03-09T15:27:12.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1743.159275 secs 2023-03-09T15:27:13.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:12 smithi038 ceph-mon[92422]: pgmap v1445: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 73 KiB/s wr, 25 op/s 2023-03-09T15:27:13.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1743.159275 secs 2023-03-09T15:27:13.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2205.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:27:14.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:14 smithi122 ceph-mon[93869]: pgmap v1446: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 45 KiB/s wr, 11 op/s 2023-03-09T15:27:14.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:14 smithi181 ceph-mon[94189]: pgmap v1446: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 45 KiB/s wr, 11 op/s 2023-03-09T15:27:14.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:14 smithi038 ceph-mon[92422]: pgmap v1446: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 45 KiB/s wr, 11 op/s 2023-03-09T15:27:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1745.963341 secs 2023-03-09T15:27:15.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1745.963341 secs 2023-03-09T15:27:15.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1745.963341 secs 2023-03-09T15:27:16.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:16 smithi122 ceph-mon[93869]: pgmap v1447: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 44 KiB/s wr, 11 op/s 2023-03-09T15:27:16.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:16 smithi181 ceph-mon[94189]: pgmap v1447: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 44 KiB/s wr, 11 op/s 2023-03-09T15:27:16.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:16 smithi038 ceph-mon[92422]: pgmap v1447: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 44 KiB/s wr, 11 op/s 2023-03-09T15:27:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1748.159418 secs 2023-03-09T15:27:17.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1748.159418 secs 2023-03-09T15:27:17.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1748.159418 secs 2023-03-09T15:27:18.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:18 smithi122 ceph-mon[93869]: pgmap v1448: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 14 KiB/s wr, 8 op/s 2023-03-09T15:27:18.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:18 smithi122 ceph-mon[93869]: osdmap e305: 12 total, 12 up, 12 in 2023-03-09T15:27:18.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:18 smithi181 ceph-mon[94189]: pgmap v1448: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 14 KiB/s wr, 8 op/s 2023-03-09T15:27:18.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:18 smithi181 ceph-mon[94189]: osdmap e305: 12 total, 12 up, 12 in 2023-03-09T15:27:18.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:18 smithi038 ceph-mon[92422]: pgmap v1448: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 14 KiB/s wr, 8 op/s 2023-03-09T15:27:18.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:18 smithi038 ceph-mon[92422]: osdmap e305: 12 total, 12 up, 12 in 2023-03-09T15:27:18.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2210.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:27:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1750.963401 secs 2023-03-09T15:27:20.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:20 smithi122 ceph-mon[93869]: pgmap v1450: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 30 KiB/s wr, 8 op/s 2023-03-09T15:27:20.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1750.963401 secs 2023-03-09T15:27:20.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:20 smithi181 ceph-mon[94189]: pgmap v1450: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 30 KiB/s wr, 8 op/s 2023-03-09T15:27:21.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1750.963401 secs 2023-03-09T15:27:21.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:20 smithi038 ceph-mon[92422]: pgmap v1450: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 30 KiB/s wr, 8 op/s 2023-03-09T15:27:22.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:22 smithi122 ceph-mon[93869]: pgmap v1451: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 31 KiB/s wr, 8 op/s 2023-03-09T15:27:22.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1753.159538 secs 2023-03-09T15:27:22.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:22 smithi181 ceph-mon[94189]: pgmap v1451: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 31 KiB/s wr, 8 op/s 2023-03-09T15:27:22.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1753.159538 secs 2023-03-09T15:27:23.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:22 smithi038 ceph-mon[92422]: pgmap v1451: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 31 KiB/s wr, 8 op/s 2023-03-09T15:27:23.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1753.159538 secs 2023-03-09T15:27:23.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2215.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:27:24.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:24 smithi122 ceph-mon[93869]: pgmap v1452: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 31 KiB/s wr, 8 op/s 2023-03-09T15:27:24.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:24 smithi181 ceph-mon[94189]: pgmap v1452: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 31 KiB/s wr, 8 op/s 2023-03-09T15:27:24.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:24 smithi038 ceph-mon[92422]: pgmap v1452: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 31 KiB/s wr, 8 op/s 2023-03-09T15:27:25.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1755.963461 secs 2023-03-09T15:27:25.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1755.963461 secs 2023-03-09T15:27:25.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1755.963461 secs 2023-03-09T15:27:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:26 smithi122 ceph-mon[93869]: pgmap v1453: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 18 KiB/s wr, 8 op/s 2023-03-09T15:27:26.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:26 smithi181 ceph-mon[94189]: pgmap v1453: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 18 KiB/s wr, 8 op/s 2023-03-09T15:27:26.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:26 smithi038 ceph-mon[92422]: pgmap v1453: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 18 KiB/s wr, 8 op/s 2023-03-09T15:27:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:27 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:27:27.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:27 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:27:27.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:27 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:27:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:28 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:27:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:28 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:27:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:28 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:27:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:28 smithi122 ceph-mon[93869]: pgmap v1454: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 18 KiB/s wr, 7 op/s 2023-03-09T15:27:28.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:28 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:27:28.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:28 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:27:28.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:28 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:27:28.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:28 smithi181 ceph-mon[94189]: pgmap v1454: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 18 KiB/s wr, 7 op/s 2023-03-09T15:27:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:28 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:27:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:28 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:27:28.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:28 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:27:28.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:28 smithi038 ceph-mon[92422]: pgmap v1454: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 18 KiB/s wr, 7 op/s 2023-03-09T15:27:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2220.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:27:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1760.963521 secs 2023-03-09T15:27:30.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:30 smithi122 ceph-mon[93869]: pgmap v1455: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 56 KiB/s wr, 8 op/s 2023-03-09T15:27:30.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1760.963521 secs 2023-03-09T15:27:30.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:30 smithi181 ceph-mon[94189]: pgmap v1455: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 56 KiB/s wr, 8 op/s 2023-03-09T15:27:31.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1760.963521 secs 2023-03-09T15:27:31.004 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:30 smithi038 ceph-mon[92422]: pgmap v1455: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 56 KiB/s wr, 8 op/s 2023-03-09T15:27:32.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:32 smithi122 ceph-mon[93869]: pgmap v1456: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 38 KiB/s wr, 8 op/s 2023-03-09T15:27:32.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:32 smithi181 ceph-mon[94189]: pgmap v1456: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 38 KiB/s wr, 8 op/s 2023-03-09T15:27:33.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:32 smithi038 ceph-mon[92422]: pgmap v1456: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 38 KiB/s wr, 8 op/s 2023-03-09T15:27:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2225.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:27:34.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:34 smithi122 ceph-mon[93869]: pgmap v1457: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 41 KiB/s wr, 8 op/s 2023-03-09T15:27:34.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:34 smithi181 ceph-mon[94189]: pgmap v1457: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 41 KiB/s wr, 8 op/s 2023-03-09T15:27:34.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:34 smithi038 ceph-mon[92422]: pgmap v1457: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 41 KiB/s wr, 8 op/s 2023-03-09T15:27:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1765.963580 secs 2023-03-09T15:27:35.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1765.963580 secs 2023-03-09T15:27:35.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1765.963580 secs 2023-03-09T15:27:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:36 smithi122 ceph-mon[93869]: pgmap v1458: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 69 KiB/s wr, 9 op/s 2023-03-09T15:27:36.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:27:36.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:36 smithi181 ceph-mon[94189]: pgmap v1458: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 69 KiB/s wr, 9 op/s 2023-03-09T15:27:36.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:27:36.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:36 smithi038 ceph-mon[92422]: pgmap v1458: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 69 KiB/s wr, 9 op/s 2023-03-09T15:27:36.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:27:38.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2230.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:27:38.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:38 smithi122 ceph-mon[93869]: pgmap v1459: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 74 KiB/s wr, 11 op/s 2023-03-09T15:27:38.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:38 smithi181 ceph-mon[94189]: pgmap v1459: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 74 KiB/s wr, 11 op/s 2023-03-09T15:27:39.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:38 smithi038 ceph-mon[92422]: pgmap v1459: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 74 KiB/s wr, 11 op/s 2023-03-09T15:27:40.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:27:40.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:27:40.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:27:40.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:27:40.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:27:40.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:27:40.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:27:40.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:27:40.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:27:40.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:27:40.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:27:40.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:27:40.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:27:40.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:27:40.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:27:40.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:27:40.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:27:40.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:27:40.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:27:40.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:27:40.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:27:40.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:27:40.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:27:40.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:27:40.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:27:40.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:27:40.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:27:40.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:27:40.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:27:40.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:27:40.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:27:40.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:27:40.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:27:40.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:27:40.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:27:40.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:27:40.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:27:40.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:27:40.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:27:40.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:27:40.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:27:40.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:27:40.896 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1770.963641 secs 2023-03-09T15:27:40.896 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:40 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T15:27:40.896 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:40 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi038 to 197.2M 2023-03-09T15:27:40.897 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:40 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T15:27:40.897 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:40 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi038 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-03-09T15:27:40.897 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:40 smithi181 ceph-mon[94189]: pgmap v1460: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 74 KiB/s wr, 9 op/s 2023-03-09T15:27:40.897 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:27:40.897 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:27:40.897 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:27:40.898 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:27:40.898 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:40 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T15:27:40.898 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:40 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T15:27:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1770.963641 secs 2023-03-09T15:27:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:40 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T15:27:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:40 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi038 to 197.2M 2023-03-09T15:27:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:40 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T15:27:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:40 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi038 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-03-09T15:27:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:40 smithi122 ceph-mon[93869]: pgmap v1460: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 74 KiB/s wr, 9 op/s 2023-03-09T15:27:41.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:27:41.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:27:41.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:27:41.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:27:41.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:40 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T15:27:41.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:40 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T15:27:41.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:27:41.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:27:41.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:27:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:27:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:27:41.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:27:41.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1770.963641 secs 2023-03-09T15:27:41.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:40 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T15:27:41.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:40 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi038 to 197.2M 2023-03-09T15:27:41.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:40 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T15:27:41.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:40 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi038 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-03-09T15:27:41.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:40 smithi038 ceph-mon[92422]: pgmap v1460: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 74 KiB/s wr, 9 op/s 2023-03-09T15:27:41.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:27:41.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:27:41.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:27:41.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:27:41.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:40 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T15:27:41.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:40 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T15:27:41.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:27:41.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:27:41.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:27:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:41 smithi122 ceph-mon[93869]: pgmap v1461: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 85 KiB/s wr, 12 op/s 2023-03-09T15:27:42.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:41 smithi181 ceph-mon[94189]: pgmap v1461: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 85 KiB/s wr, 12 op/s 2023-03-09T15:27:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:41 smithi038 ceph-mon[92422]: pgmap v1461: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 85 KiB/s wr, 12 op/s 2023-03-09T15:27:43.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2235.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:27:44.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:44 smithi122 ceph-mon[93869]: pgmap v1462: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 88 KiB/s wr, 11 op/s 2023-03-09T15:27:44.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:44 smithi181 ceph-mon[94189]: pgmap v1462: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 88 KiB/s wr, 11 op/s 2023-03-09T15:27:44.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:44 smithi038 ceph-mon[92422]: pgmap v1462: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 88 KiB/s wr, 11 op/s 2023-03-09T15:27:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1775.963697 secs 2023-03-09T15:27:45.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1775.963697 secs 2023-03-09T15:27:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1775.963697 secs 2023-03-09T15:27:46.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:46 smithi122 ceph-mon[93869]: pgmap v1463: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 85 KiB/s wr, 11 op/s 2023-03-09T15:27:46.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:46 smithi181 ceph-mon[94189]: pgmap v1463: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 85 KiB/s wr, 11 op/s 2023-03-09T15:27:46.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:46 smithi038 ceph-mon[92422]: pgmap v1463: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 85 KiB/s wr, 11 op/s 2023-03-09T15:27:48.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2240.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:27:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:48 smithi122 ceph-mon[93869]: pgmap v1464: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 80 KiB/s wr, 12 op/s 2023-03-09T15:27:48.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:48 smithi181 ceph-mon[94189]: pgmap v1464: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 80 KiB/s wr, 12 op/s 2023-03-09T15:27:49.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:48 smithi038 ceph-mon[92422]: pgmap v1464: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 80 KiB/s wr, 12 op/s 2023-03-09T15:27:50.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1780.963758 secs 2023-03-09T15:27:50.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:50 smithi122 ceph-mon[93869]: pgmap v1465: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 75 KiB/s wr, 10 op/s 2023-03-09T15:27:50.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1780.963758 secs 2023-03-09T15:27:50.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:50 smithi181 ceph-mon[94189]: pgmap v1465: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 75 KiB/s wr, 10 op/s 2023-03-09T15:27:51.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1780.963758 secs 2023-03-09T15:27:51.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:50 smithi038 ceph-mon[92422]: pgmap v1465: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 75 KiB/s wr, 10 op/s 2023-03-09T15:27:52.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:52 smithi122 ceph-mon[93869]: pgmap v1466: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 106 KiB/s wr, 12 op/s 2023-03-09T15:27:52.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1783.160253 secs 2023-03-09T15:27:52.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:52 smithi181 ceph-mon[94189]: pgmap v1466: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 106 KiB/s wr, 12 op/s 2023-03-09T15:27:52.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1783.160253 secs 2023-03-09T15:27:53.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:52 smithi038 ceph-mon[92422]: pgmap v1466: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 106 KiB/s wr, 12 op/s 2023-03-09T15:27:53.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1783.160253 secs 2023-03-09T15:27:53.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2245.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:27:54.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:54 smithi122 ceph-mon[93869]: pgmap v1467: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 59 KiB/s wr, 10 op/s 2023-03-09T15:27:54.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:54 smithi181 ceph-mon[94189]: pgmap v1467: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 59 KiB/s wr, 10 op/s 2023-03-09T15:27:54.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:54 smithi038 ceph-mon[92422]: pgmap v1467: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 59 KiB/s wr, 10 op/s 2023-03-09T15:27:55.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1785.963841 secs 2023-03-09T15:27:55.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1785.963841 secs 2023-03-09T15:27:55.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1785.963841 secs 2023-03-09T15:27:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:56 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:27:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:27:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:27:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:56 smithi122 ceph-mon[93869]: pgmap v1468: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 56 KiB/s wr, 9 op/s 2023-03-09T15:27:56.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:56 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:27:56.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:27:56.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:27:56.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:56 smithi181 ceph-mon[94189]: pgmap v1468: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 56 KiB/s wr, 9 op/s 2023-03-09T15:27:56.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:56 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:27:56.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:27:56.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:27:56.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:56 smithi038 ceph-mon[92422]: pgmap v1468: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 56 KiB/s wr, 9 op/s 2023-03-09T15:27:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1788.160368 secs 2023-03-09T15:27:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1788.160368 secs 2023-03-09T15:27:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1788.160368 secs 2023-03-09T15:27:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:27:58 smithi122 ceph-mon[93869]: pgmap v1469: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 81 KiB/s wr, 10 op/s 2023-03-09T15:27:58.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:27:58 smithi181 ceph-mon[94189]: pgmap v1469: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 81 KiB/s wr, 10 op/s 2023-03-09T15:27:58.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:27:58 smithi038 ceph-mon[92422]: pgmap v1469: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 81 KiB/s wr, 10 op/s 2023-03-09T15:27:58.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2250.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:28:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:28:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1790.963899 secs 2023-03-09T15:28:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:28:00 smithi122 ceph-mon[93869]: pgmap v1470: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 57 KiB/s wr, 7 op/s 2023-03-09T15:28:00.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:28:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1790.963899 secs 2023-03-09T15:28:00.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:28:00 smithi181 ceph-mon[94189]: pgmap v1470: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 57 KiB/s wr, 7 op/s 2023-03-09T15:28:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:28:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1790.963899 secs 2023-03-09T15:28:01.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:28:00 smithi038 ceph-mon[92422]: pgmap v1470: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 57 KiB/s wr, 7 op/s 2023-03-09T15:28:02.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:28:02 smithi122 ceph-mon[93869]: pgmap v1471: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 57 KiB/s wr, 9 op/s 2023-03-09T15:28:02.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:28:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1793.160464 secs 2023-03-09T15:28:02.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:28:02 smithi181 ceph-mon[94189]: pgmap v1471: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 57 KiB/s wr, 9 op/s 2023-03-09T15:28:02.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:28:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1793.160464 secs 2023-03-09T15:28:03.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:28:02 smithi038 ceph-mon[92422]: pgmap v1471: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 57 KiB/s wr, 9 op/s 2023-03-09T15:28:03.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:28:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1793.160464 secs 2023-03-09T15:28:03.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2255.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:28:04.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:28:04 smithi122 ceph-mon[93869]: pgmap v1472: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 49 KiB/s wr, 19 op/s 2023-03-09T15:28:04.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:28:04 smithi181 ceph-mon[94189]: pgmap v1472: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 49 KiB/s wr, 19 op/s 2023-03-09T15:28:04.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:28:04 smithi038 ceph-mon[92422]: pgmap v1472: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 49 KiB/s wr, 19 op/s 2023-03-09T15:28:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:28:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1795.963962 secs 2023-03-09T15:28:05.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:28:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1795.963962 secs 2023-03-09T15:28:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:28:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1795.963962 secs 2023-03-09T15:28:06.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:28:06 smithi122 ceph-mon[93869]: pgmap v1473: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 49 KiB/s wr, 19 op/s 2023-03-09T15:28:06.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:28:06 smithi181 ceph-mon[94189]: pgmap v1473: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 49 KiB/s wr, 19 op/s 2023-03-09T15:28:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:28:06 smithi038 ceph-mon[92422]: pgmap v1473: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 49 KiB/s wr, 19 op/s 2023-03-09T15:28:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:28:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1798.160589 secs 2023-03-09T15:28:07.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:28:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1798.160589 secs 2023-03-09T15:28:07.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:28:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1798.160589 secs 2023-03-09T15:28:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:28:08 smithi122 ceph-mon[93869]: pgmap v1474: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 85 KiB/s wr, 23 op/s 2023-03-09T15:28:08.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:28:08 smithi181 ceph-mon[94189]: pgmap v1474: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 85 KiB/s wr, 23 op/s 2023-03-09T15:28:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:28:08 smithi038 ceph-mon[92422]: pgmap v1474: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 85 KiB/s wr, 23 op/s 2023-03-09T15:28:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2260.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:28:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:28:10 smithi122 ceph-mon[93869]: osdmap e306: 12 total, 12 up, 12 in 2023-03-09T15:28:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:28:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1800.963988 secs 2023-03-09T15:28:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:28:10 smithi122 ceph-mon[93869]: pgmap v1476: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 73 KiB/s wr, 25 op/s 2023-03-09T15:28:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:28:10 smithi181 ceph-mon[94189]: osdmap e306: 12 total, 12 up, 12 in 2023-03-09T15:28:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:28:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1800.963988 secs 2023-03-09T15:28:10.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:28:10 smithi181 ceph-mon[94189]: pgmap v1476: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 73 KiB/s wr, 25 op/s 2023-03-09T15:28:10.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:28:10 smithi038 ceph-mon[92422]: osdmap e306: 12 total, 12 up, 12 in 2023-03-09T15:28:10.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:28:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1800.963988 secs 2023-03-09T15:28:10.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:28:10 smithi038 ceph-mon[92422]: pgmap v1476: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 73 KiB/s wr, 25 op/s 2023-03-09T15:28:12.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:28:12 smithi122 ceph-mon[93869]: pgmap v1477: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 73 KiB/s wr, 25 op/s 2023-03-09T15:28:12.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:28:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1803.160710 secs 2023-03-09T15:28:12.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:28:12 smithi181 ceph-mon[94189]: pgmap v1477: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 73 KiB/s wr, 25 op/s 2023-03-09T15:28:12.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:28:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1803.160710 secs 2023-03-09T15:28:13.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:28:12 smithi038 ceph-mon[92422]: pgmap v1477: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 73 KiB/s wr, 25 op/s 2023-03-09T15:28:13.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:28:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1803.160710 secs 2023-03-09T15:28:13.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2265.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:28:14.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:28:14 smithi122 ceph-mon[93869]: pgmap v1478: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 45 KiB/s wr, 11 op/s 2023-03-09T15:28:14.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:28:14 smithi181 ceph-mon[94189]: pgmap v1478: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 45 KiB/s wr, 11 op/s 2023-03-09T15:28:14.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:28:14 smithi038 ceph-mon[92422]: pgmap v1478: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 45 KiB/s wr, 11 op/s 2023-03-09T15:28:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:28:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1805.964049 secs 2023-03-09T15:28:15.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:28:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1805.964049 secs 2023-03-09T15:28:15.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:28:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1805.964049 secs 2023-03-09T15:28:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:28:16 smithi122 ceph-mon[93869]: pgmap v1479: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 45 KiB/s wr, 11 op/s 2023-03-09T15:28:16.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:28:16 smithi181 ceph-mon[94189]: pgmap v1479: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 45 KiB/s wr, 11 op/s 2023-03-09T15:28:16.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:28:16 smithi038 ceph-mon[92422]: pgmap v1479: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 45 KiB/s wr, 11 op/s 2023-03-09T15:28:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:28:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1808.160830 secs 2023-03-09T15:28:17.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:28:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1808.160830 secs 2023-03-09T15:28:17.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:28:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1808.160830 secs 2023-03-09T15:28:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:28:18 smithi122 ceph-mon[93869]: pgmap v1480: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 2.3 KiB/s wr, 8 op/s 2023-03-09T15:28:18.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:28:18 smithi122 ceph-mon[93869]: osdmap e307: 12 total, 12 up, 12 in 2023-03-09T15:28:18.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:28:18 smithi181 ceph-mon[94189]: pgmap v1480: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 2.3 KiB/s wr, 8 op/s 2023-03-09T15:28:18.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:28:18 smithi181 ceph-mon[94189]: osdmap e307: 12 total, 12 up, 12 in 2023-03-09T15:28:18.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:28:18 smithi038 ceph-mon[92422]: pgmap v1480: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 2.3 KiB/s wr, 8 op/s 2023-03-09T15:28:18.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:28:18 smithi038 ceph-mon[92422]: osdmap e307: 12 total, 12 up, 12 in 2023-03-09T15:28:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2270.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:28:20.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:28:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1810.964139 secs 2023-03-09T15:28:20.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:28:20 smithi122 ceph-mon[93869]: pgmap v1482: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 1.4 KiB/s wr, 8 op/s 2023-03-09T15:28:20.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:28:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1810.964139 secs 2023-03-09T15:28:20.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:28:20 smithi181 ceph-mon[94189]: pgmap v1482: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 1.4 KiB/s wr, 8 op/s 2023-03-09T15:28:21.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:28:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1810.964139 secs 2023-03-09T15:28:21.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:28:20 smithi038 ceph-mon[92422]: pgmap v1482: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 1.4 KiB/s wr, 8 op/s 2023-03-09T15:28:22.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:28:22 smithi122 ceph-mon[93869]: pgmap v1483: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 19 KiB/s wr, 8 op/s 2023-03-09T15:28:22.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:28:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1813.160954 secs 2023-03-09T15:28:22.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:28:22 smithi181 ceph-mon[94189]: pgmap v1483: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 19 KiB/s wr, 8 op/s 2023-03-09T15:28:22.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:28:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1813.160954 secs 2023-03-09T15:28:23.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:28:22 smithi038 ceph-mon[92422]: pgmap v1483: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 19 KiB/s wr, 8 op/s 2023-03-09T15:28:23.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:28:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1813.160954 secs 2023-03-09T15:28:23.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2275.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:28:24.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:28:24 smithi122 ceph-mon[93869]: pgmap v1484: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 8 op/s 2023-03-09T15:28:24.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:28:24 smithi181 ceph-mon[94189]: pgmap v1484: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 8 op/s 2023-03-09T15:28:24.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:28:24 smithi038 ceph-mon[92422]: pgmap v1484: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 8 op/s 2023-03-09T15:28:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:28:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1815.964201 secs 2023-03-09T15:28:25.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:28:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1815.964201 secs 2023-03-09T15:28:25.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:28:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1815.964201 secs 2023-03-09T15:28:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:28:26 smithi122 ceph-mon[93869]: pgmap v1485: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 8 op/s 2023-03-09T15:28:26.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:28:26 smithi181 ceph-mon[94189]: pgmap v1485: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 8 op/s 2023-03-09T15:28:26.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:28:26 smithi038 ceph-mon[92422]: pgmap v1485: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 8 op/s 2023-03-09T15:28:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:28:27 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 1818.161071 secs 2023-03-09T15:28:27.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:28:27 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 1818.161071 secs 2023-03-09T15:28:27.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:28:27 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 1818.161071 secs 2023-03-09T15:28:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:28:28 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:28:28.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:28:28 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:28:28.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:28:28 smithi038 ceph-mon[92422]: pgmap v1486: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 18 KiB/s wr, 7 op/s 2023-03-09T15:28:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2280.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:28:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:28:28 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:28:28.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:28:28 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:28:28.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:28:28 smithi122 ceph-mon[93869]: pgmap v1486: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 18 KiB/s wr, 7 op/s 2023-03-09T15:28:28.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:28:28 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:28:28.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:28:28 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:28:28.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:28:28 smithi181 ceph-mon[94189]: pgmap v1486: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 18 KiB/s wr, 7 op/s 2023-03-09T15:28:30.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:28:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1820.964269 secs 2023-03-09T15:28:30.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:28:30 smithi122 ceph-mon[93869]: pgmap v1487: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 16 KiB/s wr, 6 op/s 2023-03-09T15:28:30.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:28:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1820.964269 secs 2023-03-09T15:28:30.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:28:30 smithi181 ceph-mon[94189]: pgmap v1487: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 16 KiB/s wr, 6 op/s 2023-03-09T15:28:31.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:28:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1820.964269 secs 2023-03-09T15:28:31.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:28:30 smithi038 ceph-mon[92422]: pgmap v1487: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 16 KiB/s wr, 6 op/s 2023-03-09T15:28:32.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:28:32 smithi122 ceph-mon[93869]: pgmap v1488: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 40 KiB/s wr, 8 op/s 2023-03-09T15:28:32.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:28:32 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 1823.161150 secs 2023-03-09T15:28:32.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:28:32 smithi181 ceph-mon[94189]: pgmap v1488: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 40 KiB/s wr, 8 op/s 2023-03-09T15:28:32.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:28:32 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 1823.161150 secs 2023-03-09T15:28:33.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:28:32 smithi038 ceph-mon[92422]: pgmap v1488: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 40 KiB/s wr, 8 op/s 2023-03-09T15:28:33.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:28:32 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 1823.161150 secs 2023-03-09T15:28:33.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2285.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:28:34.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:28:34 smithi122 ceph-mon[93869]: pgmap v1489: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T15:28:34.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:28:34 smithi181 ceph-mon[94189]: pgmap v1489: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T15:28:34.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:28:34 smithi038 ceph-mon[92422]: pgmap v1489: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T15:28:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:28:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1825.964326 secs 2023-03-09T15:28:35.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:28:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1825.964326 secs 2023-03-09T15:28:35.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:28:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1825.964326 secs 2023-03-09T15:28:36.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:28:36 smithi122 ceph-mon[93869]: pgmap v1490: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T15:28:36.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:28:36 smithi181 ceph-mon[94189]: pgmap v1490: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T15:28:36.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:28:36 smithi038 ceph-mon[92422]: pgmap v1490: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T15:28:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:28:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1828.161275 secs 2023-03-09T15:28:37.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:28:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1828.161275 secs 2023-03-09T15:28:37.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:28:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1828.161275 secs 2023-03-09T15:28:38.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:28:38 smithi181 ceph-mon[94189]: pgmap v1491: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 67 KiB/s wr, 10 op/s 2023-03-09T15:28:38.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:28:38 smithi038 ceph-mon[92422]: pgmap v1491: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 67 KiB/s wr, 10 op/s 2023-03-09T15:28:38.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2290.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:28:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:28:38 smithi122 ceph-mon[93869]: pgmap v1491: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 67 KiB/s wr, 10 op/s 2023-03-09T15:28:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:28:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:28:40.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:28:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:28:40.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:28:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1830.964386 secs 2023-03-09T15:28:40.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:28:40 smithi038 ceph-mon[92422]: pgmap v1492: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 67 KiB/s wr, 8 op/s 2023-03-09T15:28:40.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:28:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:28:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:28:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:28:40.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:28:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:28:40.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:28:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1830.964386 secs 2023-03-09T15:28:40.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:28:40 smithi122 ceph-mon[93869]: pgmap v1492: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 67 KiB/s wr, 8 op/s 2023-03-09T15:28:40.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:28:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:28:40.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:28:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:28:40.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:28:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:28:40.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:28:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1830.964386 secs 2023-03-09T15:28:40.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:28:40 smithi181 ceph-mon[94189]: pgmap v1492: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 67 KiB/s wr, 8 op/s 2023-03-09T15:28:40.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:28:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:28:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:28:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:28:41.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:28:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:28:41.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:28:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:28:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:28:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:28:41.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:28:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:28:41.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:28:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:28:41.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:28:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:28:41.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:28:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:28:41.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:28:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:28:42.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:28:42 smithi122 ceph-mon[93869]: pgmap v1493: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 94 KiB/s wr, 11 op/s 2023-03-09T15:28:42.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:28:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1833.161391 secs 2023-03-09T15:28:42.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:28:42 smithi181 ceph-mon[94189]: pgmap v1493: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 94 KiB/s wr, 11 op/s 2023-03-09T15:28:42.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:28:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1833.161391 secs 2023-03-09T15:28:43.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:28:42 smithi038 ceph-mon[92422]: pgmap v1493: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 94 KiB/s wr, 11 op/s 2023-03-09T15:28:43.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:28:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1833.161391 secs 2023-03-09T15:28:43.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2295.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:28:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:28:44 smithi122 ceph-mon[93869]: pgmap v1494: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 72 KiB/s wr, 10 op/s 2023-03-09T15:28:44.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:28:44 smithi181 ceph-mon[94189]: pgmap v1494: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 72 KiB/s wr, 10 op/s 2023-03-09T15:28:44.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:28:44 smithi038 ceph-mon[92422]: pgmap v1494: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 72 KiB/s wr, 10 op/s 2023-03-09T15:28:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:28:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1835.964446 secs 2023-03-09T15:28:45.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:28:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1835.964446 secs 2023-03-09T15:28:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:28:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1835.964446 secs 2023-03-09T15:28:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:28:46 smithi122 ceph-mon[93869]: pgmap v1495: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 69 KiB/s wr, 10 op/s 2023-03-09T15:28:46.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:28:46 smithi181 ceph-mon[94189]: pgmap v1495: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 69 KiB/s wr, 10 op/s 2023-03-09T15:28:46.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:28:46 smithi038 ceph-mon[92422]: pgmap v1495: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 69 KiB/s wr, 10 op/s 2023-03-09T15:28:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:28:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1838.161496 secs 2023-03-09T15:28:47.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:28:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1838.161496 secs 2023-03-09T15:28:47.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:28:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1838.161496 secs 2023-03-09T15:28:48.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:28:48 smithi038 ceph-mon[92422]: pgmap v1496: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 101 KiB/s wr, 11 op/s 2023-03-09T15:28:48.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2300.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:28:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:28:48 smithi122 ceph-mon[93869]: pgmap v1496: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 101 KiB/s wr, 11 op/s 2023-03-09T15:28:48.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:28:48 smithi181 ceph-mon[94189]: pgmap v1496: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 101 KiB/s wr, 11 op/s 2023-03-09T15:28:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:28:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1840.964508 secs 2023-03-09T15:28:50.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:28:50 smithi122 ceph-mon[93869]: pgmap v1497: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 63 KiB/s wr, 9 op/s 2023-03-09T15:28:50.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:28:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1840.964508 secs 2023-03-09T15:28:50.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:28:50 smithi181 ceph-mon[94189]: pgmap v1497: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 63 KiB/s wr, 9 op/s 2023-03-09T15:28:51.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:28:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1840.964508 secs 2023-03-09T15:28:51.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:28:50 smithi038 ceph-mon[92422]: pgmap v1497: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 63 KiB/s wr, 9 op/s 2023-03-09T15:28:52.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:28:52 smithi181 ceph-mon[94189]: pgmap v1498: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 109 KiB/s wr, 11 op/s 2023-03-09T15:28:52.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:28:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1843.161614 secs 2023-03-09T15:28:53.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:28:52 smithi038 ceph-mon[92422]: pgmap v1498: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 109 KiB/s wr, 11 op/s 2023-03-09T15:28:53.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:28:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1843.161614 secs 2023-03-09T15:28:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:28:52 smithi122 ceph-mon[93869]: pgmap v1498: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 109 KiB/s wr, 11 op/s 2023-03-09T15:28:53.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:28:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1843.161614 secs 2023-03-09T15:28:53.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2305.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:28:54.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:28:54 smithi122 ceph-mon[93869]: pgmap v1499: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 85 KiB/s wr, 9 op/s 2023-03-09T15:28:54.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:28:54 smithi181 ceph-mon[94189]: pgmap v1499: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 85 KiB/s wr, 9 op/s 2023-03-09T15:28:54.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:28:54 smithi038 ceph-mon[92422]: pgmap v1499: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 85 KiB/s wr, 9 op/s 2023-03-09T15:28:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:28:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1845.964567 secs 2023-03-09T15:28:55.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:28:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1845.964567 secs 2023-03-09T15:28:55.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:28:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1845.964567 secs 2023-03-09T15:28:56.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:28:56 smithi038 ceph-mon[92422]: pgmap v1500: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 82 KiB/s wr, 9 op/s 2023-03-09T15:28:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:28:56 smithi122 ceph-mon[93869]: pgmap v1500: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 82 KiB/s wr, 9 op/s 2023-03-09T15:28:56.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:28:56 smithi181 ceph-mon[94189]: pgmap v1500: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 82 KiB/s wr, 9 op/s 2023-03-09T15:28:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:28:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1848.161755 secs 2023-03-09T15:28:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:28:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1848.161755 secs 2023-03-09T15:28:57.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:28:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1848.161755 secs 2023-03-09T15:28:58.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:28:58 smithi038 ceph-mon[92422]: pgmap v1501: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 98 KiB/s wr, 10 op/s 2023-03-09T15:28:58.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2310.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:28:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:28:58 smithi122 ceph-mon[93869]: pgmap v1501: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 98 KiB/s wr, 10 op/s 2023-03-09T15:28:58.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:28:58 smithi181 ceph-mon[94189]: pgmap v1501: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 98 KiB/s wr, 10 op/s 2023-03-09T15:29:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:29:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1850.964629 secs 2023-03-09T15:29:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:29:00 smithi122 ceph-mon[93869]: pgmap v1502: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 67 KiB/s wr, 8 op/s 2023-03-09T15:29:00.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:29:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1850.964629 secs 2023-03-09T15:29:00.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:29:00 smithi181 ceph-mon[94189]: pgmap v1502: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 67 KiB/s wr, 8 op/s 2023-03-09T15:29:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:29:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1850.964629 secs 2023-03-09T15:29:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:29:00 smithi038 ceph-mon[92422]: pgmap v1502: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 67 KiB/s wr, 8 op/s 2023-03-09T15:29:02.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:29:02 smithi122 ceph-mon[93869]: pgmap v1503: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 85 KiB/s wr, 10 op/s 2023-03-09T15:29:02.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:29:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1853.161823 secs 2023-03-09T15:29:02.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:29:02 smithi181 ceph-mon[94189]: pgmap v1503: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 85 KiB/s wr, 10 op/s 2023-03-09T15:29:02.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:29:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1853.161823 secs 2023-03-09T15:29:03.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:29:02 smithi038 ceph-mon[92422]: pgmap v1503: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 85 KiB/s wr, 10 op/s 2023-03-09T15:29:03.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:29:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1853.161823 secs 2023-03-09T15:29:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2315.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:29:04.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:29:04 smithi122 ceph-mon[93869]: pgmap v1504: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 63 KiB/s wr, 20 op/s 2023-03-09T15:29:04.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:29:04 smithi181 ceph-mon[94189]: pgmap v1504: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 63 KiB/s wr, 20 op/s 2023-03-09T15:29:04.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:29:04 smithi038 ceph-mon[92422]: pgmap v1504: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 63 KiB/s wr, 20 op/s 2023-03-09T15:29:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:29:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1855.964687 secs 2023-03-09T15:29:05.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:29:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1855.964687 secs 2023-03-09T15:29:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:29:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1855.964687 secs 2023-03-09T15:29:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:29:06 smithi122 ceph-mon[93869]: pgmap v1505: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 60 KiB/s wr, 20 op/s 2023-03-09T15:29:06.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:29:06 smithi181 ceph-mon[94189]: pgmap v1505: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 60 KiB/s wr, 20 op/s 2023-03-09T15:29:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:29:06 smithi038 ceph-mon[92422]: pgmap v1505: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 60 KiB/s wr, 20 op/s 2023-03-09T15:29:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:29:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1858.161942 secs 2023-03-09T15:29:07.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:29:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1858.161942 secs 2023-03-09T15:29:07.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:29:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1858.161942 secs 2023-03-09T15:29:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:29:08 smithi038 ceph-mon[92422]: pgmap v1506: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 96 KiB/s wr, 24 op/s 2023-03-09T15:29:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2320.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:29:08.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:29:08 smithi122 ceph-mon[93869]: pgmap v1506: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 96 KiB/s wr, 24 op/s 2023-03-09T15:29:08.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:29:08 smithi181 ceph-mon[94189]: pgmap v1506: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 96 KiB/s wr, 24 op/s 2023-03-09T15:29:10.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:29:10 smithi038 ceph-mon[92422]: osdmap e308: 12 total, 12 up, 12 in 2023-03-09T15:29:10.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:29:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1860.964754 secs 2023-03-09T15:29:10.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:29:10 smithi038 ceph-mon[92422]: pgmap v1508: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 95 KiB/s wr, 26 op/s 2023-03-09T15:29:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:29:10 smithi122 ceph-mon[93869]: osdmap e308: 12 total, 12 up, 12 in 2023-03-09T15:29:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:29:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1860.964754 secs 2023-03-09T15:29:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:29:10 smithi122 ceph-mon[93869]: pgmap v1508: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 95 KiB/s wr, 26 op/s 2023-03-09T15:29:10.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:29:10 smithi181 ceph-mon[94189]: osdmap e308: 12 total, 12 up, 12 in 2023-03-09T15:29:10.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:29:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1860.964754 secs 2023-03-09T15:29:10.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:29:10 smithi181 ceph-mon[94189]: pgmap v1508: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 95 KiB/s wr, 26 op/s 2023-03-09T15:29:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:29:12 smithi122 ceph-mon[93869]: pgmap v1509: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 74 KiB/s wr, 26 op/s 2023-03-09T15:29:12.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:29:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1863.162057 secs 2023-03-09T15:29:12.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:29:12 smithi181 ceph-mon[94189]: pgmap v1509: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 74 KiB/s wr, 26 op/s 2023-03-09T15:29:12.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:29:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1863.162057 secs 2023-03-09T15:29:13.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:29:12 smithi038 ceph-mon[92422]: pgmap v1509: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 74 KiB/s wr, 26 op/s 2023-03-09T15:29:13.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:29:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1863.162057 secs 2023-03-09T15:29:13.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2325.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:29:14.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:29:14 smithi122 ceph-mon[93869]: pgmap v1510: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 46 KiB/s wr, 12 op/s 2023-03-09T15:29:14.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:29:14 smithi181 ceph-mon[94189]: pgmap v1510: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 46 KiB/s wr, 12 op/s 2023-03-09T15:29:14.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:29:14 smithi038 ceph-mon[92422]: pgmap v1510: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 46 KiB/s wr, 12 op/s 2023-03-09T15:29:15.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:29:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1865.964815 secs 2023-03-09T15:29:15.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:29:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1865.964815 secs 2023-03-09T15:29:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:29:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1865.964815 secs 2023-03-09T15:29:16.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:29:16 smithi038 ceph-mon[92422]: pgmap v1511: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 58 KiB/s wr, 12 op/s 2023-03-09T15:29:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:29:16 smithi122 ceph-mon[93869]: pgmap v1511: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 58 KiB/s wr, 12 op/s 2023-03-09T15:29:16.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:29:16 smithi181 ceph-mon[94189]: pgmap v1511: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 58 KiB/s wr, 12 op/s 2023-03-09T15:29:17.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:29:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1868.162201 secs 2023-03-09T15:29:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:29:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1868.162201 secs 2023-03-09T15:29:17.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:29:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1868.162201 secs 2023-03-09T15:29:18.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:29:18 smithi038 ceph-mon[92422]: pgmap v1512: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 15 KiB/s wr, 8 op/s 2023-03-09T15:29:18.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:29:18 smithi038 ceph-mon[92422]: osdmap e309: 12 total, 12 up, 12 in 2023-03-09T15:29:18.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2330.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:29:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:29:18 smithi122 ceph-mon[93869]: pgmap v1512: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 15 KiB/s wr, 8 op/s 2023-03-09T15:29:18.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:29:18 smithi122 ceph-mon[93869]: osdmap e309: 12 total, 12 up, 12 in 2023-03-09T15:29:18.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:29:18 smithi181 ceph-mon[94189]: pgmap v1512: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 15 KiB/s wr, 8 op/s 2023-03-09T15:29:18.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:29:18 smithi181 ceph-mon[94189]: osdmap e309: 12 total, 12 up, 12 in 2023-03-09T15:29:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:29:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1870.964875 secs 2023-03-09T15:29:20.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:29:20 smithi122 ceph-mon[93869]: pgmap v1514: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 15 KiB/s wr, 8 op/s 2023-03-09T15:29:20.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:29:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1870.964875 secs 2023-03-09T15:29:20.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:29:20 smithi181 ceph-mon[94189]: pgmap v1514: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 15 KiB/s wr, 8 op/s 2023-03-09T15:29:21.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:29:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1870.964875 secs 2023-03-09T15:29:21.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:29:20 smithi038 ceph-mon[92422]: pgmap v1514: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 15 KiB/s wr, 8 op/s 2023-03-09T15:29:22.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:29:22 smithi181 ceph-mon[94189]: pgmap v1515: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 32 KiB/s wr, 8 op/s 2023-03-09T15:29:22.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:29:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1873.162367 secs 2023-03-09T15:29:23.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:29:22 smithi038 ceph-mon[92422]: pgmap v1515: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 32 KiB/s wr, 8 op/s 2023-03-09T15:29:23.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:29:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1873.162367 secs 2023-03-09T15:29:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:29:22 smithi122 ceph-mon[93869]: pgmap v1515: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 32 KiB/s wr, 8 op/s 2023-03-09T15:29:23.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:29:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1873.162367 secs 2023-03-09T15:29:23.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2335.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:29:24.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:29:24 smithi122 ceph-mon[93869]: pgmap v1516: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 32 KiB/s wr, 8 op/s 2023-03-09T15:29:24.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:29:24 smithi181 ceph-mon[94189]: pgmap v1516: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 32 KiB/s wr, 8 op/s 2023-03-09T15:29:24.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:29:24 smithi038 ceph-mon[92422]: pgmap v1516: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 32 KiB/s wr, 8 op/s 2023-03-09T15:29:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:29:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1875.964934 secs 2023-03-09T15:29:25.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:29:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1875.964934 secs 2023-03-09T15:29:25.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:29:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1875.964934 secs 2023-03-09T15:29:26.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:29:26 smithi038 ceph-mon[92422]: pgmap v1517: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 35 KiB/s wr, 8 op/s 2023-03-09T15:29:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:29:26 smithi122 ceph-mon[93869]: pgmap v1517: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 35 KiB/s wr, 8 op/s 2023-03-09T15:29:26.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:29:26 smithi181 ceph-mon[94189]: pgmap v1517: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 35 KiB/s wr, 8 op/s 2023-03-09T15:29:27.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:29:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1878.162469 secs 2023-03-09T15:29:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:29:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1878.162469 secs 2023-03-09T15:29:27.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:29:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1878.162469 secs 2023-03-09T15:29:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:29:28 smithi038 ceph-mon[92422]: pgmap v1518: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 35 KiB/s wr, 8 op/s 2023-03-09T15:29:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2340.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:29:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:29:28 smithi122 ceph-mon[93869]: pgmap v1518: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 35 KiB/s wr, 8 op/s 2023-03-09T15:29:28.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:29:28 smithi181 ceph-mon[94189]: pgmap v1518: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 35 KiB/s wr, 8 op/s 2023-03-09T15:29:30.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:29:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1880.964983 secs 2023-03-09T15:29:30.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:29:30 smithi181 ceph-mon[94189]: pgmap v1519: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 31 KiB/s wr, 6 op/s 2023-03-09T15:29:31.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:29:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1880.964983 secs 2023-03-09T15:29:31.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:29:30 smithi038 ceph-mon[92422]: pgmap v1519: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 31 KiB/s wr, 6 op/s 2023-03-09T15:29:31.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:29:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1880.964983 secs 2023-03-09T15:29:31.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:29:30 smithi122 ceph-mon[93869]: pgmap v1519: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 31 KiB/s wr, 6 op/s 2023-03-09T15:29:33.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:29:32 smithi038 ceph-mon[92422]: pgmap v1520: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 55 KiB/s wr, 9 op/s 2023-03-09T15:29:33.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:29:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1883.162587 secs 2023-03-09T15:29:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:29:32 smithi122 ceph-mon[93869]: pgmap v1520: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 55 KiB/s wr, 9 op/s 2023-03-09T15:29:33.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:29:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1883.162587 secs 2023-03-09T15:29:33.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:29:32 smithi181 ceph-mon[94189]: pgmap v1520: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 55 KiB/s wr, 9 op/s 2023-03-09T15:29:33.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:29:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1883.162587 secs 2023-03-09T15:29:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2345.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:29:34.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:29:34 smithi181 ceph-mon[94189]: pgmap v1521: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 7 op/s 2023-03-09T15:29:34.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:29:34 smithi038 ceph-mon[92422]: pgmap v1521: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 7 op/s 2023-03-09T15:29:34.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:29:34 smithi122 ceph-mon[93869]: pgmap v1521: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 7 op/s 2023-03-09T15:29:35.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:29:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1885.965044 secs 2023-03-09T15:29:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:29:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1885.965044 secs 2023-03-09T15:29:35.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:29:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1885.965044 secs 2023-03-09T15:29:36.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:29:36 smithi038 ceph-mon[92422]: pgmap v1522: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 74 KiB/s wr, 9 op/s 2023-03-09T15:29:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:29:36 smithi122 ceph-mon[93869]: pgmap v1522: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 74 KiB/s wr, 9 op/s 2023-03-09T15:29:36.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:29:36 smithi181 ceph-mon[94189]: pgmap v1522: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 74 KiB/s wr, 9 op/s 2023-03-09T15:29:38.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2350.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:29:39.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:29:38 smithi038 ceph-mon[92422]: pgmap v1523: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 64 KiB/s wr, 9 op/s 2023-03-09T15:29:39.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:29:38 smithi122 ceph-mon[93869]: pgmap v1523: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 64 KiB/s wr, 9 op/s 2023-03-09T15:29:39.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:29:38 smithi181 ceph-mon[94189]: pgmap v1523: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 64 KiB/s wr, 9 op/s 2023-03-09T15:29:40.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:29:39 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:29:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:29:39 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:29:40.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:29:39 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:29:41.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:29:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1890.965105 secs 2023-03-09T15:29:41.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:29:40 smithi038 ceph-mon[92422]: pgmap v1524: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 64 KiB/s wr, 8 op/s 2023-03-09T15:29:41.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:29:40 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:29:41.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:29:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:29:41.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:29:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:29:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:29:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1890.965105 secs 2023-03-09T15:29:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:29:40 smithi122 ceph-mon[93869]: pgmap v1524: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 64 KiB/s wr, 8 op/s 2023-03-09T15:29:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:29:40 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:29:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:29:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:29:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:29:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:29:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:29:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1890.965105 secs 2023-03-09T15:29:41.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:29:40 smithi181 ceph-mon[94189]: pgmap v1524: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 64 KiB/s wr, 8 op/s 2023-03-09T15:29:41.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:29:40 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:29:41.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:29:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:29:41.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:29:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:29:42.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:29:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:29:42.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:29:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:29:42.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:29:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:29:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:29:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:29:42.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:29:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:29:42.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:29:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:29:42.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:29:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:29:42.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:29:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:29:42.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:29:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:29:43.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:29:42 smithi038 ceph-mon[92422]: pgmap v1525: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 114 KiB/s wr, 12 op/s 2023-03-09T15:29:43.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:29:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:29:43.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:29:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:29:43.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:29:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:29:43.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:29:42 smithi122 ceph-mon[93869]: pgmap v1525: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 114 KiB/s wr, 12 op/s 2023-03-09T15:29:43.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:29:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:29:43.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:29:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:29:43.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:29:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:29:43.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:29:42 smithi181 ceph-mon[94189]: pgmap v1525: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 114 KiB/s wr, 12 op/s 2023-03-09T15:29:43.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:29:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:29:43.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:29:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:29:43.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:29:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:29:43.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2355.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:29:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:29:44 smithi122 ceph-mon[93869]: pgmap v1526: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 91 KiB/s wr, 10 op/s 2023-03-09T15:29:44.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:29:44 smithi181 ceph-mon[94189]: pgmap v1526: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 91 KiB/s wr, 10 op/s 2023-03-09T15:29:44.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:29:44 smithi038 ceph-mon[92422]: pgmap v1526: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 91 KiB/s wr, 10 op/s 2023-03-09T15:29:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:29:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1895.965164 secs 2023-03-09T15:29:45.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:29:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1895.965164 secs 2023-03-09T15:29:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:29:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1895.965164 secs 2023-03-09T15:29:46.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:29:46 smithi038 ceph-mon[92422]: pgmap v1527: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 108 KiB/s wr, 12 op/s 2023-03-09T15:29:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:29:46 smithi122 ceph-mon[93869]: pgmap v1527: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 108 KiB/s wr, 12 op/s 2023-03-09T15:29:46.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:29:46 smithi181 ceph-mon[94189]: pgmap v1527: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 108 KiB/s wr, 12 op/s 2023-03-09T15:29:48.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2360.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:29:49.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:29:48 smithi038 ceph-mon[92422]: pgmap v1528: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 81 KiB/s wr, 10 op/s 2023-03-09T15:29:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:29:48 smithi122 ceph-mon[93869]: pgmap v1528: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 81 KiB/s wr, 10 op/s 2023-03-09T15:29:49.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:29:48 smithi181 ceph-mon[94189]: pgmap v1528: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 81 KiB/s wr, 10 op/s 2023-03-09T15:29:51.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:29:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1900.965224 secs 2023-03-09T15:29:51.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:29:50 smithi038 ceph-mon[92422]: pgmap v1529: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 78 KiB/s wr, 9 op/s 2023-03-09T15:29:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:29:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1900.965224 secs 2023-03-09T15:29:51.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:29:50 smithi122 ceph-mon[93869]: pgmap v1529: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 78 KiB/s wr, 9 op/s 2023-03-09T15:29:51.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:29:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1900.965224 secs 2023-03-09T15:29:51.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:29:50 smithi181 ceph-mon[94189]: pgmap v1529: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 78 KiB/s wr, 9 op/s 2023-03-09T15:29:53.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:29:52 smithi038 ceph-mon[92422]: pgmap v1530: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 108 KiB/s wr, 11 op/s 2023-03-09T15:29:53.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:29:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1903.163075 secs 2023-03-09T15:29:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:29:52 smithi122 ceph-mon[93869]: pgmap v1530: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 108 KiB/s wr, 11 op/s 2023-03-09T15:29:53.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:29:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1903.163075 secs 2023-03-09T15:29:53.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:29:52 smithi181 ceph-mon[94189]: pgmap v1530: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 108 KiB/s wr, 11 op/s 2023-03-09T15:29:53.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:29:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1903.163075 secs 2023-03-09T15:29:53.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2365.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:29:54.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:29:54 smithi181 ceph-mon[94189]: pgmap v1531: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 58 KiB/s wr, 9 op/s 2023-03-09T15:29:54.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:29:54 smithi038 ceph-mon[92422]: pgmap v1531: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 58 KiB/s wr, 9 op/s 2023-03-09T15:29:54.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:29:54 smithi122 ceph-mon[93869]: pgmap v1531: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 58 KiB/s wr, 9 op/s 2023-03-09T15:29:55.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:29:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1905.965284 secs 2023-03-09T15:29:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:29:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1905.965284 secs 2023-03-09T15:29:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:29:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1905.965284 secs 2023-03-09T15:29:56.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:29:56 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:29:56.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:29:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:29:56.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:29:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:29:56.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:29:56 smithi038 ceph-mon[92422]: pgmap v1532: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 94 KiB/s wr, 10 op/s 2023-03-09T15:29:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:29:56 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:29:56.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:29:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:29:56.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:29:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:29:56.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:29:56 smithi122 ceph-mon[93869]: pgmap v1532: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 94 KiB/s wr, 10 op/s 2023-03-09T15:29:56.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:29:56 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:29:56.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:29:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:29:56.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:29:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:29:56.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:29:56 smithi181 ceph-mon[94189]: pgmap v1532: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 94 KiB/s wr, 10 op/s 2023-03-09T15:29:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:29:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1908.163204 secs 2023-03-09T15:29:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:29:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1908.163204 secs 2023-03-09T15:29:57.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:29:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1908.163204 secs 2023-03-09T15:29:58.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:29:58 smithi038 ceph-mon[92422]: pgmap v1533: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 75 KiB/s wr, 8 op/s 2023-03-09T15:29:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2370.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:29:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:29:58 smithi122 ceph-mon[93869]: pgmap v1533: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 75 KiB/s wr, 8 op/s 2023-03-09T15:29:58.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:29:58 smithi181 ceph-mon[94189]: pgmap v1533: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 75 KiB/s wr, 8 op/s 2023-03-09T15:30:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:30:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1910.965343 secs 2023-03-09T15:30:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:30:00 smithi038 ceph-mon[92422]: pgmap v1534: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 70 KiB/s wr, 7 op/s 2023-03-09T15:30:01.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:30:00 smithi038 ceph-mon[92422]: overall HEALTH_WARN 2 MDSs report slow requests 2023-03-09T15:30:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:30:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1910.965343 secs 2023-03-09T15:30:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:30:00 smithi122 ceph-mon[93869]: pgmap v1534: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 70 KiB/s wr, 7 op/s 2023-03-09T15:30:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:30:00 smithi122 ceph-mon[93869]: overall HEALTH_WARN 2 MDSs report slow requests 2023-03-09T15:30:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:30:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1910.965343 secs 2023-03-09T15:30:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:30:00 smithi181 ceph-mon[94189]: pgmap v1534: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 70 KiB/s wr, 7 op/s 2023-03-09T15:30:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:30:00 smithi181 ceph-mon[94189]: overall HEALTH_WARN 2 MDSs report slow requests 2023-03-09T15:30:03.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:30:02 smithi038 ceph-mon[92422]: pgmap v1535: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 70 KiB/s wr, 9 op/s 2023-03-09T15:30:03.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:30:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1913.163324 secs 2023-03-09T15:30:03.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:30:02 smithi122 ceph-mon[93869]: pgmap v1535: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 70 KiB/s wr, 9 op/s 2023-03-09T15:30:03.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:30:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1913.163324 secs 2023-03-09T15:30:03.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:30:02 smithi181 ceph-mon[94189]: pgmap v1535: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 70 KiB/s wr, 9 op/s 2023-03-09T15:30:03.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:30:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1913.163324 secs 2023-03-09T15:30:03.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2375.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:30:04.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:30:04 smithi038 ceph-mon[92422]: pgmap v1536: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 63 KiB/s wr, 19 op/s 2023-03-09T15:30:04.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:30:04 smithi122 ceph-mon[93869]: pgmap v1536: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 63 KiB/s wr, 19 op/s 2023-03-09T15:30:04.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:30:04 smithi181 ceph-mon[94189]: pgmap v1536: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 63 KiB/s wr, 19 op/s 2023-03-09T15:30:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:30:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1915.965404 secs 2023-03-09T15:30:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:30:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1915.965404 secs 2023-03-09T15:30:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:30:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1915.965404 secs 2023-03-09T15:30:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:30:06 smithi038 ceph-mon[92422]: pgmap v1537: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 97 KiB/s wr, 23 op/s 2023-03-09T15:30:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:30:06 smithi122 ceph-mon[93869]: pgmap v1537: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 97 KiB/s wr, 23 op/s 2023-03-09T15:30:06.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:30:06 smithi181 ceph-mon[94189]: pgmap v1537: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 97 KiB/s wr, 23 op/s 2023-03-09T15:30:07.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:30:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1918.163430 secs 2023-03-09T15:30:07.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:30:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1918.163430 secs 2023-03-09T15:30:07.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:30:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1918.163430 secs 2023-03-09T15:30:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2380.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:30:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:30:08 smithi122 ceph-mon[93869]: pgmap v1538: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 61 KiB/s wr, 22 op/s 2023-03-09T15:30:08.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:30:08 smithi181 ceph-mon[94189]: pgmap v1538: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 61 KiB/s wr, 22 op/s 2023-03-09T15:30:09.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:30:08 smithi038 ceph-mon[92422]: pgmap v1538: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 61 KiB/s wr, 22 op/s 2023-03-09T15:30:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:30:10 smithi122 ceph-mon[93869]: osdmap e310: 12 total, 12 up, 12 in 2023-03-09T15:30:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:30:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1920.965468 secs 2023-03-09T15:30:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:30:10 smithi122 ceph-mon[93869]: pgmap v1540: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 73 KiB/s wr, 25 op/s 2023-03-09T15:30:10.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:30:10 smithi181 ceph-mon[94189]: osdmap e310: 12 total, 12 up, 12 in 2023-03-09T15:30:10.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:30:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1920.965468 secs 2023-03-09T15:30:10.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:30:10 smithi181 ceph-mon[94189]: pgmap v1540: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 73 KiB/s wr, 25 op/s 2023-03-09T15:30:11.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:30:10 smithi038 ceph-mon[92422]: osdmap e310: 12 total, 12 up, 12 in 2023-03-09T15:30:11.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:30:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1920.965468 secs 2023-03-09T15:30:11.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:30:10 smithi038 ceph-mon[92422]: pgmap v1540: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 73 KiB/s wr, 25 op/s 2023-03-09T15:30:13.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:30:12 smithi038 ceph-mon[92422]: pgmap v1541: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 74 KiB/s wr, 25 op/s 2023-03-09T15:30:13.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:30:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1923.163548 secs 2023-03-09T15:30:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:30:12 smithi122 ceph-mon[93869]: pgmap v1541: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 74 KiB/s wr, 25 op/s 2023-03-09T15:30:13.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:30:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1923.163548 secs 2023-03-09T15:30:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:30:12 smithi181 ceph-mon[94189]: pgmap v1541: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 74 KiB/s wr, 25 op/s 2023-03-09T15:30:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:30:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1923.163548 secs 2023-03-09T15:30:13.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2385.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:30:14.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:30:14 smithi038 ceph-mon[92422]: pgmap v1542: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 46 KiB/s wr, 11 op/s 2023-03-09T15:30:14.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:30:14 smithi122 ceph-mon[93869]: pgmap v1542: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 46 KiB/s wr, 11 op/s 2023-03-09T15:30:14.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:30:14 smithi181 ceph-mon[94189]: pgmap v1542: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 46 KiB/s wr, 11 op/s 2023-03-09T15:30:15.754 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:30:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1925.965527 secs 2023-03-09T15:30:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:30:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1925.965527 secs 2023-03-09T15:30:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:30:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1925.965527 secs 2023-03-09T15:30:16.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:30:16 smithi038 ceph-mon[92422]: pgmap v1543: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 5.1 KiB/s wr, 8 op/s 2023-03-09T15:30:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:30:16 smithi122 ceph-mon[93869]: pgmap v1543: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 5.1 KiB/s wr, 8 op/s 2023-03-09T15:30:16.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:30:16 smithi181 ceph-mon[94189]: pgmap v1543: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 5.1 KiB/s wr, 8 op/s 2023-03-09T15:30:17.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:30:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1928.163664 secs 2023-03-09T15:30:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:30:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1928.163664 secs 2023-03-09T15:30:17.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:30:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1928.163664 secs 2023-03-09T15:30:18.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:30:18 smithi038 ceph-mon[92422]: pgmap v1544: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 2.3 KiB/s wr, 8 op/s 2023-03-09T15:30:18.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:30:18 smithi038 ceph-mon[92422]: osdmap e311: 12 total, 12 up, 12 in 2023-03-09T15:30:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2390.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:30:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:30:18 smithi122 ceph-mon[93869]: pgmap v1544: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 2.3 KiB/s wr, 8 op/s 2023-03-09T15:30:18.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:30:18 smithi122 ceph-mon[93869]: osdmap e311: 12 total, 12 up, 12 in 2023-03-09T15:30:18.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:30:18 smithi181 ceph-mon[94189]: pgmap v1544: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 2.3 KiB/s wr, 8 op/s 2023-03-09T15:30:18.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:30:18 smithi181 ceph-mon[94189]: osdmap e311: 12 total, 12 up, 12 in 2023-03-09T15:30:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:30:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1930.965601 secs 2023-03-09T15:30:21.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:30:20 smithi122 ceph-mon[93869]: pgmap v1546: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 2.4 KiB/s wr, 8 op/s 2023-03-09T15:30:21.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:30:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1930.965601 secs 2023-03-09T15:30:21.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:30:20 smithi181 ceph-mon[94189]: pgmap v1546: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 2.4 KiB/s wr, 8 op/s 2023-03-09T15:30:21.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:30:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1930.965601 secs 2023-03-09T15:30:21.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:30:20 smithi038 ceph-mon[92422]: pgmap v1546: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 2.4 KiB/s wr, 8 op/s 2023-03-09T15:30:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:30:22 smithi122 ceph-mon[93869]: pgmap v1547: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 19 KiB/s wr, 8 op/s 2023-03-09T15:30:23.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:30:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1933.163780 secs 2023-03-09T15:30:23.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:30:22 smithi181 ceph-mon[94189]: pgmap v1547: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 19 KiB/s wr, 8 op/s 2023-03-09T15:30:23.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:30:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1933.163780 secs 2023-03-09T15:30:23.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:30:22 smithi038 ceph-mon[92422]: pgmap v1547: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 19 KiB/s wr, 8 op/s 2023-03-09T15:30:23.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:30:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1933.163780 secs 2023-03-09T15:30:23.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2395.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:30:24.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:30:24 smithi038 ceph-mon[92422]: pgmap v1548: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T15:30:24.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:30:24 smithi122 ceph-mon[93869]: pgmap v1548: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T15:30:24.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:30:24 smithi181 ceph-mon[94189]: pgmap v1548: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T15:30:25.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:30:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1935.965683 secs 2023-03-09T15:30:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:30:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1935.965683 secs 2023-03-09T15:30:25.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:30:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1935.965683 secs 2023-03-09T15:30:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:30:26 smithi122 ceph-mon[93869]: pgmap v1549: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T15:30:26.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:30:26 smithi181 ceph-mon[94189]: pgmap v1549: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T15:30:27.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:30:26 smithi038 ceph-mon[92422]: pgmap v1549: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T15:30:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:30:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1938.163893 secs 2023-03-09T15:30:27.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:30:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1938.163893 secs 2023-03-09T15:30:28.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:30:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1938.163893 secs 2023-03-09T15:30:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2400.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:30:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:30:28 smithi122 ceph-mon[93869]: pgmap v1550: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T15:30:28.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:30:28 smithi181 ceph-mon[94189]: pgmap v1550: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T15:30:29.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:30:28 smithi038 ceph-mon[92422]: pgmap v1550: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T15:30:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:30:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1940.965746 secs 2023-03-09T15:30:31.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:30:30 smithi122 ceph-mon[93869]: pgmap v1551: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 6 op/s 2023-03-09T15:30:31.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:30:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1940.965746 secs 2023-03-09T15:30:31.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:30:30 smithi181 ceph-mon[94189]: pgmap v1551: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 6 op/s 2023-03-09T15:30:31.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:30:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1940.965746 secs 2023-03-09T15:30:31.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:30:30 smithi038 ceph-mon[92422]: pgmap v1551: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 6 op/s 2023-03-09T15:30:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:30:32 smithi122 ceph-mon[93869]: pgmap v1552: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 35 KiB/s wr, 8 op/s 2023-03-09T15:30:33.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:30:32 smithi181 ceph-mon[94189]: pgmap v1552: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 35 KiB/s wr, 8 op/s 2023-03-09T15:30:33.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:30:32 smithi038 ceph-mon[92422]: pgmap v1552: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 35 KiB/s wr, 8 op/s 2023-03-09T15:30:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2405.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:30:34.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:30:34 smithi038 ceph-mon[92422]: pgmap v1553: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 6 op/s 2023-03-09T15:30:34.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:30:34 smithi122 ceph-mon[93869]: pgmap v1553: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 6 op/s 2023-03-09T15:30:34.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:30:34 smithi181 ceph-mon[94189]: pgmap v1553: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 6 op/s 2023-03-09T15:30:35.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:30:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1945.965806 secs 2023-03-09T15:30:35.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:30:35 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:30:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:30:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1945.965806 secs 2023-03-09T15:30:35.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:30:35 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:30:35.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:30:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1945.965806 secs 2023-03-09T15:30:35.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:30:35 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:30:36.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:30:36 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:30:36.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:30:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:30:36.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:30:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:30:36.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:30:36 smithi038 ceph-mon[92422]: pgmap v1554: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 50 KiB/s wr, 8 op/s 2023-03-09T15:30:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:30:36 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:30:36.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:30:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:30:36.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:30:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:30:36.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:30:36 smithi122 ceph-mon[93869]: pgmap v1554: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 50 KiB/s wr, 8 op/s 2023-03-09T15:30:36.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:30:36 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:30:36.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:30:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:30:36.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:30:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:30:36.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:30:36 smithi181 ceph-mon[94189]: pgmap v1554: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 50 KiB/s wr, 8 op/s 2023-03-09T15:30:38.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2410.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:30:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:30:38 smithi122 ceph-mon[93869]: pgmap v1555: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 7 op/s 2023-03-09T15:30:39.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:30:38 smithi181 ceph-mon[94189]: pgmap v1555: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 7 op/s 2023-03-09T15:30:39.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:30:38 smithi038 ceph-mon[92422]: pgmap v1555: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 7 op/s 2023-03-09T15:30:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:30:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1950.965867 secs 2023-03-09T15:30:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:30:40 smithi122 ceph-mon[93869]: pgmap v1556: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 53 KiB/s wr, 7 op/s 2023-03-09T15:30:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:30:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1950.965867 secs 2023-03-09T15:30:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:30:40 smithi181 ceph-mon[94189]: pgmap v1556: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 53 KiB/s wr, 7 op/s 2023-03-09T15:30:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:30:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1950.965867 secs 2023-03-09T15:30:41.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:30:40 smithi038 ceph-mon[92422]: pgmap v1556: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 53 KiB/s wr, 7 op/s 2023-03-09T15:30:42.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:30:42 smithi122 ceph-mon[93869]: pgmap v1557: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 73 KiB/s wr, 10 op/s 2023-03-09T15:30:42.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:30:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:30:43.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:30:42 smithi181 ceph-mon[94189]: pgmap v1557: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 73 KiB/s wr, 10 op/s 2023-03-09T15:30:43.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:30:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:30:43.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:30:42 smithi038 ceph-mon[92422]: pgmap v1557: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 73 KiB/s wr, 10 op/s 2023-03-09T15:30:43.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:30:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:30:43.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2415.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:30:44.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:30:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:30:44.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:30:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:30:44.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:30:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:30:44.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:30:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:30:44.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:30:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:30:44.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:30:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:30:44.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:30:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:30:44.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:30:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:30:44.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:30:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:30:44.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:30:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:30:44.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:30:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:30:44.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:30:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:30:44.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:30:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:30:44.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:30:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:30:44.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:30:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:30:44.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:30:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:30:44.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:30:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:30:44.254 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:30:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:30:44.254 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:30:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:30:44.254 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:30:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:30:44.255 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:30:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:30:45.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:30:44 smithi122 ceph-mon[93869]: pgmap v1558: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 58 KiB/s wr, 10 op/s 2023-03-09T15:30:45.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:30:44 smithi181 ceph-mon[94189]: pgmap v1558: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 58 KiB/s wr, 10 op/s 2023-03-09T15:30:45.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:30:44 smithi038 ceph-mon[92422]: pgmap v1558: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 58 KiB/s wr, 10 op/s 2023-03-09T15:30:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:30:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1955.965926 secs 2023-03-09T15:30:46.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:30:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1955.965926 secs 2023-03-09T15:30:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:30:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1955.965926 secs 2023-03-09T15:30:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:30:46 smithi122 ceph-mon[93869]: pgmap v1559: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 101 KiB/s wr, 12 op/s 2023-03-09T15:30:47.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:30:46 smithi181 ceph-mon[94189]: pgmap v1559: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 101 KiB/s wr, 12 op/s 2023-03-09T15:30:47.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:30:46 smithi038 ceph-mon[92422]: pgmap v1559: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 101 KiB/s wr, 12 op/s 2023-03-09T15:30:48.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2420.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:30:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:30:48 smithi122 ceph-mon[93869]: pgmap v1560: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 74 KiB/s wr, 11 op/s 2023-03-09T15:30:49.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:30:48 smithi181 ceph-mon[94189]: pgmap v1560: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 74 KiB/s wr, 11 op/s 2023-03-09T15:30:49.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:30:48 smithi038 ceph-mon[92422]: pgmap v1560: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 74 KiB/s wr, 11 op/s 2023-03-09T15:30:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:30:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1960.965983 secs 2023-03-09T15:30:51.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:30:50 smithi122 ceph-mon[93869]: pgmap v1561: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 71 KiB/s wr, 10 op/s 2023-03-09T15:30:51.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:30:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1960.965983 secs 2023-03-09T15:30:51.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:30:50 smithi181 ceph-mon[94189]: pgmap v1561: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 71 KiB/s wr, 10 op/s 2023-03-09T15:30:51.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:30:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1960.965983 secs 2023-03-09T15:30:51.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:30:50 smithi038 ceph-mon[92422]: pgmap v1561: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 71 KiB/s wr, 10 op/s 2023-03-09T15:30:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:30:52 smithi122 ceph-mon[93869]: pgmap v1562: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 78 KiB/s wr, 13 op/s 2023-03-09T15:30:53.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:30:52 smithi181 ceph-mon[94189]: pgmap v1562: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 78 KiB/s wr, 13 op/s 2023-03-09T15:30:53.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:30:52 smithi038 ceph-mon[92422]: pgmap v1562: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 78 KiB/s wr, 13 op/s 2023-03-09T15:30:53.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2425.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:30:54.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:30:54 smithi038 ceph-mon[92422]: pgmap v1563: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 61 KiB/s wr, 10 op/s 2023-03-09T15:30:54.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:30:54 smithi122 ceph-mon[93869]: pgmap v1563: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 61 KiB/s wr, 10 op/s 2023-03-09T15:30:54.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:30:54 smithi181 ceph-mon[94189]: pgmap v1563: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 61 KiB/s wr, 10 op/s 2023-03-09T15:30:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:30:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1965.966051 secs 2023-03-09T15:30:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:30:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1965.966051 secs 2023-03-09T15:30:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:30:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1965.966051 secs 2023-03-09T15:30:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:30:56 smithi122 ceph-mon[93869]: pgmap v1564: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 117 KiB/s wr, 12 op/s 2023-03-09T15:30:56.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:30:56 smithi181 ceph-mon[94189]: pgmap v1564: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 117 KiB/s wr, 12 op/s 2023-03-09T15:30:57.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:30:56 smithi038 ceph-mon[92422]: pgmap v1564: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 117 KiB/s wr, 12 op/s 2023-03-09T15:30:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:30:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1968.164566 secs 2023-03-09T15:30:57.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:30:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1968.164566 secs 2023-03-09T15:30:58.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:30:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1968.164566 secs 2023-03-09T15:30:58.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2430.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:30:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:30:58 smithi122 ceph-mon[93869]: pgmap v1565: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 77 KiB/s wr, 10 op/s 2023-03-09T15:30:58.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:30:58 smithi181 ceph-mon[94189]: pgmap v1565: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 77 KiB/s wr, 10 op/s 2023-03-09T15:30:59.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:30:58 smithi038 ceph-mon[92422]: pgmap v1565: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 77 KiB/s wr, 10 op/s 2023-03-09T15:30:59.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:30:59 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:30:59.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:30:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:30:59.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:30:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:30:59.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:30:59 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:30:59.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:30:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:30:59.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:30:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:31:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:30:59 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:31:00.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:30:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:31:00.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:30:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:31:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:31:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1970.966110 secs 2023-03-09T15:31:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:31:00 smithi122 ceph-mon[93869]: pgmap v1566: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 73 KiB/s wr, 9 op/s 2023-03-09T15:31:00.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:31:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1970.966110 secs 2023-03-09T15:31:00.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:31:00 smithi181 ceph-mon[94189]: pgmap v1566: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 73 KiB/s wr, 9 op/s 2023-03-09T15:31:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:31:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1970.966110 secs 2023-03-09T15:31:01.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:31:00 smithi038 ceph-mon[92422]: pgmap v1566: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 73 KiB/s wr, 9 op/s 2023-03-09T15:31:03.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:31:02 smithi122 ceph-mon[93869]: pgmap v1567: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 128 KiB/s wr, 14 op/s 2023-03-09T15:31:03.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:31:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1973.164679 secs 2023-03-09T15:31:03.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:31:02 smithi181 ceph-mon[94189]: pgmap v1567: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 128 KiB/s wr, 14 op/s 2023-03-09T15:31:03.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:31:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1973.164679 secs 2023-03-09T15:31:03.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:31:02 smithi038 ceph-mon[92422]: pgmap v1567: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 128 KiB/s wr, 14 op/s 2023-03-09T15:31:03.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:31:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1973.164679 secs 2023-03-09T15:31:03.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2435.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:31:04.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:31:04 smithi038 ceph-mon[92422]: pgmap v1568: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 143 KiB/s wr, 24 op/s 2023-03-09T15:31:04.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:31:04 smithi122 ceph-mon[93869]: pgmap v1568: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 143 KiB/s wr, 24 op/s 2023-03-09T15:31:04.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:31:04 smithi181 ceph-mon[94189]: pgmap v1568: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 143 KiB/s wr, 24 op/s 2023-03-09T15:31:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:31:05 smithi122 ceph-mon[93869]: osdmap e312: 12 total, 12 up, 12 in 2023-03-09T15:31:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:31:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1975.966174 secs 2023-03-09T15:31:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:31:05 smithi181 ceph-mon[94189]: osdmap e312: 12 total, 12 up, 12 in 2023-03-09T15:31:05.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:31:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1975.966174 secs 2023-03-09T15:31:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:31:05 smithi038 ceph-mon[92422]: osdmap e312: 12 total, 12 up, 12 in 2023-03-09T15:31:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:31:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1975.966174 secs 2023-03-09T15:31:06.831 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:31:06 smithi181 ceph-mon[94189]: pgmap v1570: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 138 KiB/s wr, 27 op/s 2023-03-09T15:31:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:31:06 smithi122 ceph-mon[93869]: pgmap v1570: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 138 KiB/s wr, 27 op/s 2023-03-09T15:31:07.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:31:06 smithi038 ceph-mon[92422]: pgmap v1570: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 138 KiB/s wr, 27 op/s 2023-03-09T15:31:07.825 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:31:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1978.164770 secs 2023-03-09T15:31:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:31:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1978.164770 secs 2023-03-09T15:31:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:31:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1978.164770 secs 2023-03-09T15:31:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2440.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:31:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:31:08 smithi122 ceph-mon[93869]: pgmap v1571: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 134 KiB/s wr, 27 op/s 2023-03-09T15:31:08.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:31:08 smithi181 ceph-mon[94189]: pgmap v1571: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 134 KiB/s wr, 27 op/s 2023-03-09T15:31:09.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:31:08 smithi038 ceph-mon[92422]: pgmap v1571: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 134 KiB/s wr, 27 op/s 2023-03-09T15:31:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:31:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1980.966232 secs 2023-03-09T15:31:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:31:10 smithi122 ceph-mon[93869]: pgmap v1572: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 134 KiB/s wr, 26 op/s 2023-03-09T15:31:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:31:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1980.966232 secs 2023-03-09T15:31:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:31:10 smithi181 ceph-mon[94189]: pgmap v1572: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 134 KiB/s wr, 26 op/s 2023-03-09T15:31:11.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:31:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1980.966232 secs 2023-03-09T15:31:11.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:31:10 smithi038 ceph-mon[92422]: pgmap v1572: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 134 KiB/s wr, 26 op/s 2023-03-09T15:31:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:31:12 smithi122 ceph-mon[93869]: pgmap v1573: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 82 KiB/s wr, 22 op/s 2023-03-09T15:31:13.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:31:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1983.164886 secs 2023-03-09T15:31:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:31:12 smithi181 ceph-mon[94189]: pgmap v1573: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 82 KiB/s wr, 22 op/s 2023-03-09T15:31:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:31:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1983.164886 secs 2023-03-09T15:31:13.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:31:12 smithi038 ceph-mon[92422]: pgmap v1573: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 82 KiB/s wr, 22 op/s 2023-03-09T15:31:13.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:31:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1983.164886 secs 2023-03-09T15:31:13.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2445.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:31:14.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:31:14 smithi038 ceph-mon[92422]: osdmap e313: 12 total, 12 up, 12 in 2023-03-09T15:31:14.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:31:14 smithi038 ceph-mon[92422]: pgmap v1575: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 60 KiB/s wr, 8 op/s 2023-03-09T15:31:14.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:31:14 smithi122 ceph-mon[93869]: osdmap e313: 12 total, 12 up, 12 in 2023-03-09T15:31:14.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:31:14 smithi122 ceph-mon[93869]: pgmap v1575: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 60 KiB/s wr, 8 op/s 2023-03-09T15:31:14.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:31:14 smithi181 ceph-mon[94189]: osdmap e313: 12 total, 12 up, 12 in 2023-03-09T15:31:14.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:31:14 smithi181 ceph-mon[94189]: pgmap v1575: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 60 KiB/s wr, 8 op/s 2023-03-09T15:31:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:31:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1985.966298 secs 2023-03-09T15:31:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:31:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1985.966298 secs 2023-03-09T15:31:16.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:31:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1985.966298 secs 2023-03-09T15:31:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:31:16 smithi122 ceph-mon[93869]: pgmap v1576: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T15:31:16.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:31:16 smithi181 ceph-mon[94189]: pgmap v1576: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T15:31:17.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:31:16 smithi038 ceph-mon[92422]: pgmap v1576: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T15:31:17.834 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:31:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1988.165007 secs 2023-03-09T15:31:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:31:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1988.165007 secs 2023-03-09T15:31:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:31:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1988.165007 secs 2023-03-09T15:31:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2450.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:31:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:31:18 smithi122 ceph-mon[93869]: pgmap v1577: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 6 op/s 2023-03-09T15:31:18.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:31:18 smithi181 ceph-mon[94189]: pgmap v1577: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 6 op/s 2023-03-09T15:31:19.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:31:18 smithi038 ceph-mon[92422]: pgmap v1577: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 6 op/s 2023-03-09T15:31:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:31:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1990.966357 secs 2023-03-09T15:31:21.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:31:20 smithi122 ceph-mon[93869]: pgmap v1578: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 6 op/s 2023-03-09T15:31:21.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:31:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1990.966357 secs 2023-03-09T15:31:21.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:31:20 smithi181 ceph-mon[94189]: pgmap v1578: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 6 op/s 2023-03-09T15:31:21.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:31:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1990.966357 secs 2023-03-09T15:31:21.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:31:20 smithi038 ceph-mon[92422]: pgmap v1578: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 6 op/s 2023-03-09T15:31:23.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:31:22 smithi122 ceph-mon[93869]: pgmap v1579: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T15:31:23.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:31:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1993.165156 secs 2023-03-09T15:31:23.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:31:22 smithi181 ceph-mon[94189]: pgmap v1579: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T15:31:23.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:31:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1993.165156 secs 2023-03-09T15:31:23.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:31:22 smithi038 ceph-mon[92422]: pgmap v1579: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T15:31:23.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:31:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1993.165156 secs 2023-03-09T15:31:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2455.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:31:24.849 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:31:24 smithi181 ceph-mon[94189]: pgmap v1580: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T15:31:24.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:31:24 smithi122 ceph-mon[93869]: pgmap v1580: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T15:31:25.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:31:24 smithi038 ceph-mon[92422]: pgmap v1580: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T15:31:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:31:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1995.966421 secs 2023-03-09T15:31:25.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:31:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1995.966421 secs 2023-03-09T15:31:26.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:31:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1995.966421 secs 2023-03-09T15:31:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:31:26 smithi122 ceph-mon[93869]: pgmap v1581: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 8 op/s 2023-03-09T15:31:26.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:31:26 smithi181 ceph-mon[94189]: pgmap v1581: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 8 op/s 2023-03-09T15:31:27.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:31:26 smithi038 ceph-mon[92422]: pgmap v1581: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 8 op/s 2023-03-09T15:31:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:31:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 1998.165280 secs 2023-03-09T15:31:27.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:31:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 1998.165280 secs 2023-03-09T15:31:28.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:31:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 1998.165280 secs 2023-03-09T15:31:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2460.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:31:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:31:28 smithi122 ceph-mon[93869]: pgmap v1582: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T15:31:28.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:31:28 smithi181 ceph-mon[94189]: pgmap v1582: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T15:31:29.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:31:28 smithi038 ceph-mon[92422]: pgmap v1582: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T15:31:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:31:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2000.966481 secs 2023-03-09T15:31:31.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:31:30 smithi122 ceph-mon[93869]: pgmap v1583: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 6 op/s 2023-03-09T15:31:31.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:31:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2000.966481 secs 2023-03-09T15:31:31.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:31:30 smithi181 ceph-mon[94189]: pgmap v1583: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 6 op/s 2023-03-09T15:31:31.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:31:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2000.966481 secs 2023-03-09T15:31:31.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:31:30 smithi038 ceph-mon[92422]: pgmap v1583: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 6 op/s 2023-03-09T15:31:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:31:32 smithi122 ceph-mon[93869]: pgmap v1584: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 68 KiB/s wr, 9 op/s 2023-03-09T15:31:33.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:31:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2003.165387 secs 2023-03-09T15:31:33.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:31:32 smithi181 ceph-mon[94189]: pgmap v1584: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 68 KiB/s wr, 9 op/s 2023-03-09T15:31:33.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:31:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2003.165387 secs 2023-03-09T15:31:33.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:31:32 smithi038 ceph-mon[92422]: pgmap v1584: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 68 KiB/s wr, 9 op/s 2023-03-09T15:31:33.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:31:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2003.165387 secs 2023-03-09T15:31:33.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2465.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:31:34.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:31:34 smithi122 ceph-mon[93869]: pgmap v1585: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 62 KiB/s wr, 8 op/s 2023-03-09T15:31:34.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:31:34 smithi181 ceph-mon[94189]: pgmap v1585: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 62 KiB/s wr, 8 op/s 2023-03-09T15:31:35.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:31:34 smithi038 ceph-mon[92422]: pgmap v1585: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 62 KiB/s wr, 8 op/s 2023-03-09T15:31:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:31:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2005.966543 secs 2023-03-09T15:31:35.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:31:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2005.966543 secs 2023-03-09T15:31:36.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:31:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2005.966543 secs 2023-03-09T15:31:36.862 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:31:36 smithi181 ceph-mon[94189]: pgmap v1586: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 100 KiB/s wr, 34 op/s 2023-03-09T15:31:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:31:36 smithi122 ceph-mon[93869]: pgmap v1586: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 100 KiB/s wr, 34 op/s 2023-03-09T15:31:37.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:31:36 smithi038 ceph-mon[92422]: pgmap v1586: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 100 KiB/s wr, 34 op/s 2023-03-09T15:31:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:31:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2008.165508 secs 2023-03-09T15:31:37.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:31:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2008.165508 secs 2023-03-09T15:31:38.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:31:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2008.165508 secs 2023-03-09T15:31:38.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2470.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:31:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:31:38 smithi122 ceph-mon[93869]: pgmap v1587: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 88 KiB/s wr, 33 op/s 2023-03-09T15:31:38.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:31:38 smithi181 ceph-mon[94189]: pgmap v1587: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 88 KiB/s wr, 33 op/s 2023-03-09T15:31:39.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:31:38 smithi038 ceph-mon[92422]: pgmap v1587: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 88 KiB/s wr, 33 op/s 2023-03-09T15:31:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:31:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2010.966603 secs 2023-03-09T15:31:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:31:40 smithi122 ceph-mon[93869]: pgmap v1588: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 88 KiB/s wr, 32 op/s 2023-03-09T15:31:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:31:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2010.966603 secs 2023-03-09T15:31:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:31:40 smithi181 ceph-mon[94189]: pgmap v1588: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 88 KiB/s wr, 32 op/s 2023-03-09T15:31:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:31:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2010.966603 secs 2023-03-09T15:31:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:31:40 smithi038 ceph-mon[92422]: pgmap v1588: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 88 KiB/s wr, 32 op/s 2023-03-09T15:31:43.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:31:42 smithi122 ceph-mon[93869]: pgmap v1589: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 123 KiB/s wr, 35 op/s 2023-03-09T15:31:43.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:31:42 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 2013.165644 secs 2023-03-09T15:31:43.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:31:42 smithi181 ceph-mon[94189]: pgmap v1589: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 123 KiB/s wr, 35 op/s 2023-03-09T15:31:43.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:31:42 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 2013.165644 secs 2023-03-09T15:31:43.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:31:42 smithi038 ceph-mon[92422]: pgmap v1589: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 123 KiB/s wr, 35 op/s 2023-03-09T15:31:43.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:31:42 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 2013.165644 secs 2023-03-09T15:31:43.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2475.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:31:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:31:43 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:31:44.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:31:43 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:31:44.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:31:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:31:44.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:31:43 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:31:44.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:31:43 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:31:44.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:31:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:31:44.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:31:43 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:31:44.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:31:43 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:31:44.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:31:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:31:45.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:31:44 smithi122 ceph-mon[93869]: pgmap v1590: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 86 KiB/s wr, 34 op/s 2023-03-09T15:31:45.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:31:44 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:31:45.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:31:44 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:31:45.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:31:44 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:31:45.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:31:44 smithi181 ceph-mon[94189]: pgmap v1590: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 86 KiB/s wr, 34 op/s 2023-03-09T15:31:45.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:31:44 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:31:45.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:31:44 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:31:45.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:31:44 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:31:45.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:31:44 smithi038 ceph-mon[92422]: pgmap v1590: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 86 KiB/s wr, 34 op/s 2023-03-09T15:31:45.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:31:44 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:31:45.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:31:44 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:31:45.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:31:44 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:31:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:31:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2015.966670 secs 2023-03-09T15:31:46.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:31:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2015.966670 secs 2023-03-09T15:31:46.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:31:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2015.966670 secs 2023-03-09T15:31:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:31:46 smithi122 ceph-mon[93869]: pgmap v1591: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 125 KiB/s wr, 36 op/s 2023-03-09T15:31:47.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:31:46 smithi181 ceph-mon[94189]: pgmap v1591: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 125 KiB/s wr, 36 op/s 2023-03-09T15:31:47.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:31:46 smithi038 ceph-mon[92422]: pgmap v1591: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 125 KiB/s wr, 36 op/s 2023-03-09T15:31:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:31:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2018.165741 secs 2023-03-09T15:31:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:31:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:31:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:31:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:31:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:31:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2018.165741 secs 2023-03-09T15:31:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:31:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:31:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:31:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:31:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:31:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2018.165741 secs 2023-03-09T15:31:48.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:31:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:31:48.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:31:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:31:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2480.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:31:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:31:48 smithi122 ceph-mon[93869]: pgmap v1592: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 90 KiB/s wr, 10 op/s 2023-03-09T15:31:49.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:31:48 smithi181 ceph-mon[94189]: pgmap v1592: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 90 KiB/s wr, 10 op/s 2023-03-09T15:31:49.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:31:48 smithi038 ceph-mon[92422]: pgmap v1592: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 90 KiB/s wr, 10 op/s 2023-03-09T15:31:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:31:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2020.966742 secs 2023-03-09T15:31:51.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:31:50 smithi122 ceph-mon[93869]: pgmap v1593: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 87 KiB/s wr, 9 op/s 2023-03-09T15:31:51.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:31:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2020.966742 secs 2023-03-09T15:31:51.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:31:50 smithi181 ceph-mon[94189]: pgmap v1593: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 87 KiB/s wr, 9 op/s 2023-03-09T15:31:51.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:31:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2020.966742 secs 2023-03-09T15:31:51.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:31:50 smithi038 ceph-mon[92422]: pgmap v1593: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 87 KiB/s wr, 9 op/s 2023-03-09T15:31:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:31:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2023.165844 secs 2023-03-09T15:31:53.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:31:52 smithi122 ceph-mon[93869]: pgmap v1594: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 96 KiB/s wr, 12 op/s 2023-03-09T15:31:53.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:31:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2023.165844 secs 2023-03-09T15:31:53.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:31:52 smithi181 ceph-mon[94189]: pgmap v1594: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 96 KiB/s wr, 12 op/s 2023-03-09T15:31:53.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:31:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2023.165844 secs 2023-03-09T15:31:53.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:31:52 smithi038 ceph-mon[92422]: pgmap v1594: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 96 KiB/s wr, 12 op/s 2023-03-09T15:31:53.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2485.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:31:54.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:31:54 smithi122 ceph-mon[93869]: pgmap v1595: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 60 KiB/s wr, 10 op/s 2023-03-09T15:31:54.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:31:54 smithi181 ceph-mon[94189]: pgmap v1595: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 60 KiB/s wr, 10 op/s 2023-03-09T15:31:55.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:31:54 smithi038 ceph-mon[92422]: pgmap v1595: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 60 KiB/s wr, 10 op/s 2023-03-09T15:31:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:31:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2025.966801 secs 2023-03-09T15:31:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:31:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2025.966801 secs 2023-03-09T15:31:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:31:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2025.966801 secs 2023-03-09T15:31:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:31:56 smithi122 ceph-mon[93869]: pgmap v1596: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 106 KiB/s wr, 11 op/s 2023-03-09T15:31:56.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:31:56 smithi181 ceph-mon[94189]: pgmap v1596: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 106 KiB/s wr, 11 op/s 2023-03-09T15:31:57.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:31:56 smithi038 ceph-mon[92422]: pgmap v1596: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 106 KiB/s wr, 11 op/s 2023-03-09T15:31:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:31:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2028.165963 secs 2023-03-09T15:31:57.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:31:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2028.165963 secs 2023-03-09T15:31:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:31:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2028.165963 secs 2023-03-09T15:31:58.801 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2490.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:31:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:31:58 smithi122 ceph-mon[93869]: pgmap v1597: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 61 KiB/s wr, 8 op/s 2023-03-09T15:31:58.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:31:58 smithi181 ceph-mon[94189]: pgmap v1597: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 61 KiB/s wr, 8 op/s 2023-03-09T15:31:59.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:31:58 smithi038 ceph-mon[92422]: pgmap v1597: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 61 KiB/s wr, 8 op/s 2023-03-09T15:32:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:32:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2030.966862 secs 2023-03-09T15:32:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:32:00 smithi122 ceph-mon[93869]: pgmap v1598: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 58 KiB/s wr, 8 op/s 2023-03-09T15:32:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:32:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2030.966862 secs 2023-03-09T15:32:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:32:00 smithi181 ceph-mon[94189]: pgmap v1598: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 58 KiB/s wr, 8 op/s 2023-03-09T15:32:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:32:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2030.966862 secs 2023-03-09T15:32:01.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:32:00 smithi038 ceph-mon[92422]: pgmap v1598: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 58 KiB/s wr, 8 op/s 2023-03-09T15:32:03.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:32:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2033.166079 secs 2023-03-09T15:32:03.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:32:02 smithi122 ceph-mon[93869]: pgmap v1599: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 96 KiB/s wr, 12 op/s 2023-03-09T15:32:03.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:32:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2033.166079 secs 2023-03-09T15:32:03.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:32:02 smithi181 ceph-mon[94189]: pgmap v1599: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 96 KiB/s wr, 12 op/s 2023-03-09T15:32:03.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:32:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2033.166079 secs 2023-03-09T15:32:03.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:32:02 smithi038 ceph-mon[92422]: pgmap v1599: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 96 KiB/s wr, 12 op/s 2023-03-09T15:32:03.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2495.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:32:04.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:32:04 smithi122 ceph-mon[93869]: pgmap v1600: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 111 KiB/s wr, 22 op/s 2023-03-09T15:32:04.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:32:04 smithi181 ceph-mon[94189]: pgmap v1600: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 111 KiB/s wr, 22 op/s 2023-03-09T15:32:05.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:32:04 smithi038 ceph-mon[92422]: pgmap v1600: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 111 KiB/s wr, 22 op/s 2023-03-09T15:32:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:32:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2035.966921 secs 2023-03-09T15:32:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:32:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2035.966921 secs 2023-03-09T15:32:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:32:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2035.966921 secs 2023-03-09T15:32:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:32:06 smithi122 ceph-mon[93869]: pgmap v1601: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 112 KiB/s wr, 24 op/s 2023-03-09T15:32:06.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:32:06 smithi181 ceph-mon[94189]: pgmap v1601: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 112 KiB/s wr, 24 op/s 2023-03-09T15:32:07.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:32:06 smithi038 ceph-mon[92422]: pgmap v1601: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 112 KiB/s wr, 24 op/s 2023-03-09T15:32:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:32:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2038.166218 secs 2023-03-09T15:32:07.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:32:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2038.166218 secs 2023-03-09T15:32:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:32:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2038.166218 secs 2023-03-09T15:32:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2500.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:32:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:32:08 smithi122 ceph-mon[93869]: pgmap v1602: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 63 KiB/s wr, 22 op/s 2023-03-09T15:32:08.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:32:08 smithi181 ceph-mon[94189]: pgmap v1602: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 63 KiB/s wr, 22 op/s 2023-03-09T15:32:09.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:32:08 smithi038 ceph-mon[92422]: pgmap v1602: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 63 KiB/s wr, 22 op/s 2023-03-09T15:32:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:32:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2040.966982 secs 2023-03-09T15:32:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:32:10 smithi122 ceph-mon[93869]: osdmap e314: 12 total, 12 up, 12 in 2023-03-09T15:32:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:32:10 smithi122 ceph-mon[93869]: pgmap v1604: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 75 KiB/s wr, 25 op/s 2023-03-09T15:32:10.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:32:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2040.966982 secs 2023-03-09T15:32:10.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:32:10 smithi181 ceph-mon[94189]: osdmap e314: 12 total, 12 up, 12 in 2023-03-09T15:32:10.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:32:10 smithi181 ceph-mon[94189]: pgmap v1604: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 75 KiB/s wr, 25 op/s 2023-03-09T15:32:11.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:32:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2040.966982 secs 2023-03-09T15:32:11.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:32:10 smithi038 ceph-mon[92422]: osdmap e314: 12 total, 12 up, 12 in 2023-03-09T15:32:11.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:32:10 smithi038 ceph-mon[92422]: pgmap v1604: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 75 KiB/s wr, 25 op/s 2023-03-09T15:32:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:32:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2043.166357 secs 2023-03-09T15:32:13.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:32:12 smithi122 ceph-mon[93869]: pgmap v1605: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 31 KiB/s wr, 22 op/s 2023-03-09T15:32:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:32:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2043.166357 secs 2023-03-09T15:32:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:32:12 smithi181 ceph-mon[94189]: pgmap v1605: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 31 KiB/s wr, 22 op/s 2023-03-09T15:32:13.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:32:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2043.166357 secs 2023-03-09T15:32:13.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:32:12 smithi038 ceph-mon[92422]: pgmap v1605: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 31 KiB/s wr, 22 op/s 2023-03-09T15:32:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2505.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:32:14.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:32:14 smithi122 ceph-mon[93869]: pgmap v1606: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 2.2 KiB/s wr, 8 op/s 2023-03-09T15:32:14.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:32:14 smithi181 ceph-mon[94189]: pgmap v1606: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 2.2 KiB/s wr, 8 op/s 2023-03-09T15:32:15.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:32:14 smithi038 ceph-mon[92422]: pgmap v1606: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 2.2 KiB/s wr, 8 op/s 2023-03-09T15:32:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:32:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2045.967045 secs 2023-03-09T15:32:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:32:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2045.967045 secs 2023-03-09T15:32:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:32:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2045.967045 secs 2023-03-09T15:32:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:32:16 smithi122 ceph-mon[93869]: pgmap v1607: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T15:32:16.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:32:16 smithi181 ceph-mon[94189]: pgmap v1607: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T15:32:17.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:32:16 smithi038 ceph-mon[92422]: pgmap v1607: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T15:32:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:32:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2048.166466 secs 2023-03-09T15:32:17.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:32:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2048.166466 secs 2023-03-09T15:32:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:32:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2048.166466 secs 2023-03-09T15:32:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2510.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:32:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:32:18 smithi122 ceph-mon[93869]: pgmap v1608: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T15:32:18.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:32:18 smithi122 ceph-mon[93869]: osdmap e315: 12 total, 12 up, 12 in 2023-03-09T15:32:18.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:32:18 smithi181 ceph-mon[94189]: pgmap v1608: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T15:32:18.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:32:18 smithi181 ceph-mon[94189]: osdmap e315: 12 total, 12 up, 12 in 2023-03-09T15:32:19.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:32:18 smithi038 ceph-mon[92422]: pgmap v1608: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T15:32:19.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:32:18 smithi038 ceph-mon[92422]: osdmap e315: 12 total, 12 up, 12 in 2023-03-09T15:32:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:32:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2050.967103 secs 2023-03-09T15:32:21.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:32:20 smithi122 ceph-mon[93869]: pgmap v1610: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 8 op/s 2023-03-09T15:32:21.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:32:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2050.967103 secs 2023-03-09T15:32:21.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:32:20 smithi181 ceph-mon[94189]: pgmap v1610: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 8 op/s 2023-03-09T15:32:21.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:32:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2050.967103 secs 2023-03-09T15:32:21.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:32:20 smithi038 ceph-mon[92422]: pgmap v1610: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 8 op/s 2023-03-09T15:32:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:32:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2053.166594 secs 2023-03-09T15:32:23.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:32:22 smithi122 ceph-mon[93869]: pgmap v1611: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T15:32:23.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:32:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2053.166594 secs 2023-03-09T15:32:23.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:32:22 smithi181 ceph-mon[94189]: pgmap v1611: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T15:32:23.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:32:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2053.166594 secs 2023-03-09T15:32:23.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:32:22 smithi038 ceph-mon[92422]: pgmap v1611: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T15:32:23.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2515.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:32:24.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:32:24 smithi122 ceph-mon[93869]: pgmap v1612: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T15:32:24.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:32:24 smithi181 ceph-mon[94189]: pgmap v1612: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T15:32:25.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:32:24 smithi038 ceph-mon[92422]: pgmap v1612: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T15:32:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:32:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2055.967173 secs 2023-03-09T15:32:25.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:32:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2055.967173 secs 2023-03-09T15:32:26.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:32:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2055.967173 secs 2023-03-09T15:32:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:32:26 smithi122 ceph-mon[93869]: pgmap v1613: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T15:32:26.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:32:26 smithi181 ceph-mon[94189]: pgmap v1613: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T15:32:27.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:32:26 smithi038 ceph-mon[92422]: pgmap v1613: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T15:32:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:32:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2058.166740 secs 2023-03-09T15:32:27.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:32:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2058.166740 secs 2023-03-09T15:32:28.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:32:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2058.166740 secs 2023-03-09T15:32:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2520.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:32:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:32:28 smithi122 ceph-mon[93869]: pgmap v1614: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 6 op/s 2023-03-09T15:32:28.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:32:28 smithi181 ceph-mon[94189]: pgmap v1614: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 6 op/s 2023-03-09T15:32:29.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:32:28 smithi038 ceph-mon[92422]: pgmap v1614: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 6 op/s 2023-03-09T15:32:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:32:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2060.967233 secs 2023-03-09T15:32:31.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:32:30 smithi122 ceph-mon[93869]: pgmap v1615: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 KiB/s wr, 5 op/s 2023-03-09T15:32:31.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:32:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2060.967233 secs 2023-03-09T15:32:31.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:32:30 smithi181 ceph-mon[94189]: pgmap v1615: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 KiB/s wr, 5 op/s 2023-03-09T15:32:31.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:32:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2060.967233 secs 2023-03-09T15:32:31.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:32:30 smithi038 ceph-mon[92422]: pgmap v1615: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 KiB/s wr, 5 op/s 2023-03-09T15:32:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:32:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2063.166838 secs 2023-03-09T15:32:33.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:32:32 smithi122 ceph-mon[93869]: pgmap v1616: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 34 KiB/s wr, 7 op/s 2023-03-09T15:32:33.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:32:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2063.166838 secs 2023-03-09T15:32:33.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:32:32 smithi181 ceph-mon[94189]: pgmap v1616: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 34 KiB/s wr, 7 op/s 2023-03-09T15:32:33.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:32:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2063.166838 secs 2023-03-09T15:32:33.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:32:32 smithi038 ceph-mon[92422]: pgmap v1616: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 34 KiB/s wr, 7 op/s 2023-03-09T15:32:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2525.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:32:34.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:32:34 smithi122 ceph-mon[93869]: pgmap v1617: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 34 KiB/s wr, 6 op/s 2023-03-09T15:32:34.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:32:34 smithi181 ceph-mon[94189]: pgmap v1617: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 34 KiB/s wr, 6 op/s 2023-03-09T15:32:35.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:32:34 smithi038 ceph-mon[92422]: pgmap v1617: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 34 KiB/s wr, 6 op/s 2023-03-09T15:32:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:32:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2065.967292 secs 2023-03-09T15:32:35.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:32:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2065.967292 secs 2023-03-09T15:32:36.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:32:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2065.967292 secs 2023-03-09T15:32:36.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:32:36 smithi122 ceph-mon[93869]: pgmap v1618: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 65 KiB/s wr, 8 op/s 2023-03-09T15:32:36.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:32:36 smithi181 ceph-mon[94189]: pgmap v1618: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 65 KiB/s wr, 8 op/s 2023-03-09T15:32:37.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:32:36 smithi038 ceph-mon[92422]: pgmap v1618: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 65 KiB/s wr, 8 op/s 2023-03-09T15:32:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:32:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2068.166952 secs 2023-03-09T15:32:37.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:32:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2068.166952 secs 2023-03-09T15:32:38.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:32:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2068.166952 secs 2023-03-09T15:32:38.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2530.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:32:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:32:38 smithi122 ceph-mon[93869]: pgmap v1619: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 7 op/s 2023-03-09T15:32:38.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:32:38 smithi181 ceph-mon[94189]: pgmap v1619: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 7 op/s 2023-03-09T15:32:39.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:32:38 smithi038 ceph-mon[92422]: pgmap v1619: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 7 op/s 2023-03-09T15:32:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:32:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2070.967368 secs 2023-03-09T15:32:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:32:40 smithi122 ceph-mon[93869]: pgmap v1620: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 53 KiB/s wr, 7 op/s 2023-03-09T15:32:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:32:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2070.967368 secs 2023-03-09T15:32:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:32:40 smithi181 ceph-mon[94189]: pgmap v1620: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 53 KiB/s wr, 7 op/s 2023-03-09T15:32:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:32:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2070.967368 secs 2023-03-09T15:32:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:32:40 smithi038 ceph-mon[92422]: pgmap v1620: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 53 KiB/s wr, 7 op/s 2023-03-09T15:32:43.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:32:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2073.167066 secs 2023-03-09T15:32:43.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:32:42 smithi122 ceph-mon[93869]: pgmap v1621: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 60 KiB/s wr, 10 op/s 2023-03-09T15:32:43.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:32:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2073.167066 secs 2023-03-09T15:32:43.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:32:42 smithi181 ceph-mon[94189]: pgmap v1621: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 60 KiB/s wr, 10 op/s 2023-03-09T15:32:43.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:32:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2073.167066 secs 2023-03-09T15:32:43.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:32:42 smithi038 ceph-mon[92422]: pgmap v1621: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 60 KiB/s wr, 10 op/s 2023-03-09T15:32:43.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2535.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:32:44.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:32:44 smithi122 ceph-mon[93869]: pgmap v1622: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 45 KiB/s wr, 9 op/s 2023-03-09T15:32:44.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:32:44 smithi181 ceph-mon[94189]: pgmap v1622: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 45 KiB/s wr, 9 op/s 2023-03-09T15:32:45.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:32:44 smithi038 ceph-mon[92422]: pgmap v1622: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 45 KiB/s wr, 9 op/s 2023-03-09T15:32:45.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:32:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2075.967427 secs 2023-03-09T15:32:45.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:32:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:32:45.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:32:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:32:45.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:32:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:32:45.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:32:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:32:45.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:32:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2075.967427 secs 2023-03-09T15:32:45.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:32:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:32:45.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:32:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:32:45.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:32:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:32:45.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:32:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:32:46.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:32:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2075.967427 secs 2023-03-09T15:32:46.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:32:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:32:46.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:32:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:32:46.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:32:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:32:46.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:32:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:32:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:32:46 smithi122 ceph-mon[93869]: pgmap v1623: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 104 KiB/s wr, 11 op/s 2023-03-09T15:32:46.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:32:46 smithi181 ceph-mon[94189]: pgmap v1623: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 104 KiB/s wr, 11 op/s 2023-03-09T15:32:47.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:32:46 smithi038 ceph-mon[92422]: pgmap v1623: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 104 KiB/s wr, 11 op/s 2023-03-09T15:32:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:32:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2078.167183 secs 2023-03-09T15:32:47.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:32:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2078.167183 secs 2023-03-09T15:32:48.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:32:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2078.167183 secs 2023-03-09T15:32:48.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2540.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:32:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:32:48 smithi122 ceph-mon[93869]: pgmap v1624: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 75 KiB/s wr, 9 op/s 2023-03-09T15:32:48.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:32:48 smithi181 ceph-mon[94189]: pgmap v1624: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 75 KiB/s wr, 9 op/s 2023-03-09T15:32:49.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:32:48 smithi038 ceph-mon[92422]: pgmap v1624: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 75 KiB/s wr, 9 op/s 2023-03-09T15:32:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:32:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2080.967506 secs 2023-03-09T15:32:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:32:50 smithi122 ceph-mon[93869]: pgmap v1625: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 72 KiB/s wr, 9 op/s 2023-03-09T15:32:51.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:32:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2080.967506 secs 2023-03-09T15:32:51.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:32:50 smithi181 ceph-mon[94189]: pgmap v1625: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 72 KiB/s wr, 9 op/s 2023-03-09T15:32:51.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:32:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2080.967506 secs 2023-03-09T15:32:51.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:32:50 smithi038 ceph-mon[92422]: pgmap v1625: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 72 KiB/s wr, 9 op/s 2023-03-09T15:32:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:32:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2083.167322 secs 2023-03-09T15:32:53.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:32:52 smithi122 ceph-mon[93869]: pgmap v1626: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 80 KiB/s wr, 11 op/s 2023-03-09T15:32:53.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:32:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2083.167322 secs 2023-03-09T15:32:53.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:32:52 smithi181 ceph-mon[94189]: pgmap v1626: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 80 KiB/s wr, 11 op/s 2023-03-09T15:32:53.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:32:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2083.167322 secs 2023-03-09T15:32:53.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:32:52 smithi038 ceph-mon[92422]: pgmap v1626: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 80 KiB/s wr, 11 op/s 2023-03-09T15:32:53.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2545.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:32:54.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:32:54 smithi122 ceph-mon[93869]: pgmap v1627: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 76 KiB/s wr, 9 op/s 2023-03-09T15:32:54.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:32:54 smithi181 ceph-mon[94189]: pgmap v1627: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 76 KiB/s wr, 9 op/s 2023-03-09T15:32:55.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:32:54 smithi038 ceph-mon[92422]: pgmap v1627: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 76 KiB/s wr, 9 op/s 2023-03-09T15:32:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:32:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2085.967565 secs 2023-03-09T15:32:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:32:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2085.967565 secs 2023-03-09T15:32:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:32:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2085.967565 secs 2023-03-09T15:32:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:32:56 smithi122 ceph-mon[93869]: pgmap v1628: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 137 KiB/s wr, 11 op/s 2023-03-09T15:32:56.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:32:56 smithi181 ceph-mon[94189]: pgmap v1628: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 137 KiB/s wr, 11 op/s 2023-03-09T15:32:57.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:32:56 smithi038 ceph-mon[92422]: pgmap v1628: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 137 KiB/s wr, 11 op/s 2023-03-09T15:32:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:32:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2088.167444 secs 2023-03-09T15:32:57.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:32:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2088.167444 secs 2023-03-09T15:32:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:32:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2088.167444 secs 2023-03-09T15:32:58.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2550.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:32:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:32:58 smithi122 ceph-mon[93869]: pgmap v1629: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 81 KiB/s wr, 9 op/s 2023-03-09T15:32:58.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:32:58 smithi181 ceph-mon[94189]: pgmap v1629: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 81 KiB/s wr, 9 op/s 2023-03-09T15:32:59.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:32:58 smithi038 ceph-mon[92422]: pgmap v1629: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 81 KiB/s wr, 9 op/s 2023-03-09T15:33:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:33:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2090.967650 secs 2023-03-09T15:33:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:33:00 smithi122 ceph-mon[93869]: pgmap v1630: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 78 KiB/s wr, 8 op/s 2023-03-09T15:33:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:33:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2090.967650 secs 2023-03-09T15:33:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:33:00 smithi181 ceph-mon[94189]: pgmap v1630: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 78 KiB/s wr, 8 op/s 2023-03-09T15:33:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:33:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2090.967650 secs 2023-03-09T15:33:01.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:33:00 smithi038 ceph-mon[92422]: pgmap v1630: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 78 KiB/s wr, 8 op/s 2023-03-09T15:33:03.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:33:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2093.167561 secs 2023-03-09T15:33:03.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:33:02 smithi122 ceph-mon[93869]: pgmap v1631: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 112 KiB/s wr, 13 op/s 2023-03-09T15:33:03.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:33:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2093.167561 secs 2023-03-09T15:33:03.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:33:02 smithi181 ceph-mon[94189]: pgmap v1631: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 112 KiB/s wr, 13 op/s 2023-03-09T15:33:03.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:33:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2093.167561 secs 2023-03-09T15:33:03.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:33:02 smithi038 ceph-mon[92422]: pgmap v1631: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 112 KiB/s wr, 13 op/s 2023-03-09T15:33:03.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2555.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:33:04.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:33:04 smithi122 ceph-mon[93869]: pgmap v1632: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 136 KiB/s wr, 23 op/s 2023-03-09T15:33:04.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:33:04 smithi181 ceph-mon[94189]: pgmap v1632: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 136 KiB/s wr, 23 op/s 2023-03-09T15:33:05.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:33:04 smithi038 ceph-mon[92422]: pgmap v1632: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 136 KiB/s wr, 23 op/s 2023-03-09T15:33:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:33:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2095.967707 secs 2023-03-09T15:33:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:33:05 smithi122 ceph-mon[93869]: osdmap e316: 12 total, 12 up, 12 in 2023-03-09T15:33:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:33:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2095.967707 secs 2023-03-09T15:33:05.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:33:05 smithi181 ceph-mon[94189]: osdmap e316: 12 total, 12 up, 12 in 2023-03-09T15:33:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:33:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2095.967707 secs 2023-03-09T15:33:06.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:33:05 smithi038 ceph-mon[92422]: osdmap e316: 12 total, 12 up, 12 in 2023-03-09T15:33:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:33:06 smithi122 ceph-mon[93869]: pgmap v1634: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 99 KiB/s wr, 26 op/s 2023-03-09T15:33:06.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:33:06 smithi181 ceph-mon[94189]: pgmap v1634: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 99 KiB/s wr, 26 op/s 2023-03-09T15:33:07.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:33:06 smithi038 ceph-mon[92422]: pgmap v1634: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 99 KiB/s wr, 26 op/s 2023-03-09T15:33:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:33:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2098.167699 secs 2023-03-09T15:33:07.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:33:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2098.167699 secs 2023-03-09T15:33:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:33:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2098.167699 secs 2023-03-09T15:33:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2560.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:33:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:33:08 smithi122 ceph-mon[93869]: pgmap v1635: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 113 KiB/s wr, 27 op/s 2023-03-09T15:33:08.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:33:08 smithi181 ceph-mon[94189]: pgmap v1635: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 113 KiB/s wr, 27 op/s 2023-03-09T15:33:09.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:33:08 smithi038 ceph-mon[92422]: pgmap v1635: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 113 KiB/s wr, 27 op/s 2023-03-09T15:33:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:33:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2100.967766 secs 2023-03-09T15:33:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:33:10 smithi122 ceph-mon[93869]: pgmap v1636: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 112 KiB/s wr, 26 op/s 2023-03-09T15:33:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:33:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2100.967766 secs 2023-03-09T15:33:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:33:10 smithi181 ceph-mon[94189]: pgmap v1636: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 112 KiB/s wr, 26 op/s 2023-03-09T15:33:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:33:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2100.967766 secs 2023-03-09T15:33:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:33:10 smithi038 ceph-mon[92422]: pgmap v1636: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 112 KiB/s wr, 26 op/s 2023-03-09T15:33:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:33:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2103.167832 secs 2023-03-09T15:33:13.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:33:12 smithi122 ceph-mon[93869]: pgmap v1637: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 71 KiB/s wr, 23 op/s 2023-03-09T15:33:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:33:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2103.167832 secs 2023-03-09T15:33:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:33:12 smithi181 ceph-mon[94189]: pgmap v1637: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 71 KiB/s wr, 23 op/s 2023-03-09T15:33:13.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:33:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2103.167832 secs 2023-03-09T15:33:13.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:33:12 smithi038 ceph-mon[92422]: pgmap v1637: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 71 KiB/s wr, 23 op/s 2023-03-09T15:33:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2565.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:33:14.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:33:14 smithi122 ceph-mon[93869]: osdmap e317: 12 total, 12 up, 12 in 2023-03-09T15:33:14.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:33:14 smithi122 ceph-mon[93869]: pgmap v1639: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 55 KiB/s wr, 10 op/s 2023-03-09T15:33:14.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:33:14 smithi181 ceph-mon[94189]: osdmap e317: 12 total, 12 up, 12 in 2023-03-09T15:33:14.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:33:14 smithi181 ceph-mon[94189]: pgmap v1639: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 55 KiB/s wr, 10 op/s 2023-03-09T15:33:15.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:33:14 smithi038 ceph-mon[92422]: osdmap e317: 12 total, 12 up, 12 in 2023-03-09T15:33:15.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:33:14 smithi038 ceph-mon[92422]: pgmap v1639: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 55 KiB/s wr, 10 op/s 2023-03-09T15:33:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:33:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2105.967825 secs 2023-03-09T15:33:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:33:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2105.967825 secs 2023-03-09T15:33:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:33:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2105.967825 secs 2023-03-09T15:33:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:33:16 smithi122 ceph-mon[93869]: pgmap v1640: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 35 KiB/s wr, 9 op/s 2023-03-09T15:33:16.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:33:16 smithi181 ceph-mon[94189]: pgmap v1640: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 35 KiB/s wr, 9 op/s 2023-03-09T15:33:17.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:33:16 smithi038 ceph-mon[92422]: pgmap v1640: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 35 KiB/s wr, 9 op/s 2023-03-09T15:33:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:33:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2108.167969 secs 2023-03-09T15:33:17.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:33:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2108.167969 secs 2023-03-09T15:33:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:33:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2108.167969 secs 2023-03-09T15:33:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2570.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:33:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:33:18 smithi122 ceph-mon[93869]: pgmap v1641: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 8 op/s 2023-03-09T15:33:18.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:33:18 smithi181 ceph-mon[94189]: pgmap v1641: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 8 op/s 2023-03-09T15:33:19.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:33:18 smithi038 ceph-mon[92422]: pgmap v1641: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 8 op/s 2023-03-09T15:33:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:33:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2110.967892 secs 2023-03-09T15:33:21.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:33:20 smithi122 ceph-mon[93869]: pgmap v1642: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 8 op/s 2023-03-09T15:33:21.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:33:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2110.967892 secs 2023-03-09T15:33:21.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:33:20 smithi181 ceph-mon[94189]: pgmap v1642: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 8 op/s 2023-03-09T15:33:21.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:33:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2110.967892 secs 2023-03-09T15:33:21.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:33:20 smithi038 ceph-mon[92422]: pgmap v1642: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 8 op/s 2023-03-09T15:33:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:33:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2113.168070 secs 2023-03-09T15:33:23.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:33:22 smithi122 ceph-mon[93869]: pgmap v1643: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 8 op/s 2023-03-09T15:33:23.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:33:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2113.168070 secs 2023-03-09T15:33:23.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:33:22 smithi181 ceph-mon[94189]: pgmap v1643: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 8 op/s 2023-03-09T15:33:23.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:33:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2113.168070 secs 2023-03-09T15:33:23.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:33:22 smithi038 ceph-mon[92422]: pgmap v1643: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 8 op/s 2023-03-09T15:33:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2575.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:33:24.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:33:24 smithi122 ceph-mon[93869]: pgmap v1644: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 21 KiB/s wr, 8 op/s 2023-03-09T15:33:24.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:33:24 smithi181 ceph-mon[94189]: pgmap v1644: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 21 KiB/s wr, 8 op/s 2023-03-09T15:33:25.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:33:24 smithi038 ceph-mon[92422]: pgmap v1644: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 21 KiB/s wr, 8 op/s 2023-03-09T15:33:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:33:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2115.967951 secs 2023-03-09T15:33:25.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:33:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2115.967951 secs 2023-03-09T15:33:26.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:33:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2115.967951 secs 2023-03-09T15:33:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:33:26 smithi122 ceph-mon[93869]: pgmap v1645: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 18 KiB/s wr, 9 op/s 2023-03-09T15:33:26.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:33:26 smithi181 ceph-mon[94189]: pgmap v1645: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 18 KiB/s wr, 9 op/s 2023-03-09T15:33:27.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:33:26 smithi038 ceph-mon[92422]: pgmap v1645: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 18 KiB/s wr, 9 op/s 2023-03-09T15:33:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:33:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2118.168190 secs 2023-03-09T15:33:27.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:33:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2118.168190 secs 2023-03-09T15:33:28.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:33:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2118.168190 secs 2023-03-09T15:33:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2580.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:33:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:33:28 smithi122 ceph-mon[93869]: pgmap v1646: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 18 KiB/s wr, 8 op/s 2023-03-09T15:33:28.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:33:28 smithi181 ceph-mon[94189]: pgmap v1646: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 18 KiB/s wr, 8 op/s 2023-03-09T15:33:29.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:33:28 smithi038 ceph-mon[92422]: pgmap v1646: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 18 KiB/s wr, 8 op/s 2023-03-09T15:33:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:33:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2120.967985 secs 2023-03-09T15:33:31.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:33:30 smithi122 ceph-mon[93869]: pgmap v1647: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 7 op/s 2023-03-09T15:33:31.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:33:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2120.967985 secs 2023-03-09T15:33:31.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:33:30 smithi181 ceph-mon[94189]: pgmap v1647: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 7 op/s 2023-03-09T15:33:31.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:33:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2120.967985 secs 2023-03-09T15:33:31.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:33:30 smithi038 ceph-mon[92422]: pgmap v1647: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 7 op/s 2023-03-09T15:33:33.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:33:32 smithi122 ceph-mon[93869]: pgmap v1648: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 9 op/s 2023-03-09T15:33:33.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:33:32 smithi181 ceph-mon[94189]: pgmap v1648: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 9 op/s 2023-03-09T15:33:33.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:33:32 smithi038 ceph-mon[92422]: pgmap v1648: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 9 op/s 2023-03-09T15:33:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2585.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:33:34.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:33:34 smithi122 ceph-mon[93869]: pgmap v1649: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 86 KiB/s wr, 20 op/s 2023-03-09T15:33:34.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:33:34 smithi181 ceph-mon[94189]: pgmap v1649: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 86 KiB/s wr, 20 op/s 2023-03-09T15:33:35.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:33:34 smithi038 ceph-mon[92422]: pgmap v1649: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 86 KiB/s wr, 20 op/s 2023-03-09T15:33:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:33:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2125.968044 secs 2023-03-09T15:33:35.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:33:35 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:33:35.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:33:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2125.968044 secs 2023-03-09T15:33:35.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:33:35 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:33:36.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:33:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2125.968044 secs 2023-03-09T15:33:36.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:33:35 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:33:36.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:33:36 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:33:36.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:33:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:33:36.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:33:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:33:36.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:33:36 smithi122 ceph-mon[93869]: pgmap v1650: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 70 KiB/s wr, 21 op/s 2023-03-09T15:33:36.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:33:36 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:33:36.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:33:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:33:36.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:33:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:33:36.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:33:36 smithi181 ceph-mon[94189]: pgmap v1650: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 70 KiB/s wr, 21 op/s 2023-03-09T15:33:37.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:33:36 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:33:37.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:33:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:33:37.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:33:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:33:37.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:33:36 smithi038 ceph-mon[92422]: pgmap v1650: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 70 KiB/s wr, 21 op/s 2023-03-09T15:33:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2590.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:33:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:33:38 smithi122 ceph-mon[93869]: pgmap v1651: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 114 KiB/s wr, 21 op/s 2023-03-09T15:33:39.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:33:38 smithi181 ceph-mon[94189]: pgmap v1651: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 114 KiB/s wr, 21 op/s 2023-03-09T15:33:39.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:33:38 smithi038 ceph-mon[92422]: pgmap v1651: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 114 KiB/s wr, 21 op/s 2023-03-09T15:33:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:33:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2130.968105 secs 2023-03-09T15:33:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:33:40 smithi122 ceph-mon[93869]: pgmap v1652: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 114 KiB/s wr, 34 op/s 2023-03-09T15:33:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:33:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2130.968105 secs 2023-03-09T15:33:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:33:40 smithi181 ceph-mon[94189]: pgmap v1652: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 114 KiB/s wr, 34 op/s 2023-03-09T15:33:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:33:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2130.968105 secs 2023-03-09T15:33:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:33:40 smithi038 ceph-mon[92422]: pgmap v1652: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 114 KiB/s wr, 34 op/s 2023-03-09T15:33:43.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:33:42 smithi122 ceph-mon[93869]: pgmap v1653: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 114 KiB/s wr, 36 op/s 2023-03-09T15:33:43.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:33:42 smithi181 ceph-mon[94189]: pgmap v1653: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 114 KiB/s wr, 36 op/s 2023-03-09T15:33:43.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:33:42 smithi038 ceph-mon[92422]: pgmap v1653: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 114 KiB/s wr, 36 op/s 2023-03-09T15:33:43.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2595.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:33:44.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:33:44 smithi122 ceph-mon[93869]: pgmap v1654: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 156 KiB/s wr, 36 op/s 2023-03-09T15:33:44.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:33:44 smithi181 ceph-mon[94189]: pgmap v1654: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 156 KiB/s wr, 36 op/s 2023-03-09T15:33:45.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:33:44 smithi038 ceph-mon[92422]: pgmap v1654: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 156 KiB/s wr, 36 op/s 2023-03-09T15:33:45.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:33:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2135.968164 secs 2023-03-09T15:33:45.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:33:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:33:45.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:33:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2135.968164 secs 2023-03-09T15:33:45.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:33:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:33:46.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:33:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2135.968164 secs 2023-03-09T15:33:46.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:33:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:33:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:33:46 smithi122 ceph-mon[93869]: pgmap v1655: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 88 KiB/s wr, 25 op/s 2023-03-09T15:33:46.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:33:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:33:46.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:33:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:33:46.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:33:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:33:46.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:33:46 smithi181 ceph-mon[94189]: pgmap v1655: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 88 KiB/s wr, 25 op/s 2023-03-09T15:33:46.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:33:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:33:46.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:33:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:33:46.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:33:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:33:47.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:33:46 smithi038 ceph-mon[92422]: pgmap v1655: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 88 KiB/s wr, 25 op/s 2023-03-09T15:33:47.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:33:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:33:47.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:33:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:33:47.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:33:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:33:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:33:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2138.168638 secs 2023-03-09T15:33:47.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:33:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2138.168638 secs 2023-03-09T15:33:48.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:33:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2138.168638 secs 2023-03-09T15:33:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2600.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:33:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:33:48 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:33:48.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:33:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:33:48.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:33:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:33:48.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:33:48 smithi122 ceph-mon[93869]: pgmap v1656: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 112 KiB/s wr, 25 op/s 2023-03-09T15:33:48.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:33:48 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:33:48.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:33:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:33:48.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:33:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:33:48.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:33:48 smithi181 ceph-mon[94189]: pgmap v1656: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 112 KiB/s wr, 25 op/s 2023-03-09T15:33:49.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:33:48 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:33:49.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:33:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:33:49.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:33:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:33:49.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:33:48 smithi038 ceph-mon[92422]: pgmap v1656: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 112 KiB/s wr, 25 op/s 2023-03-09T15:33:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:33:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2140.968226 secs 2023-03-09T15:33:51.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:33:50 smithi122 ceph-mon[93869]: pgmap v1657: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 68 KiB/s wr, 23 op/s 2023-03-09T15:33:51.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:33:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2140.968226 secs 2023-03-09T15:33:51.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:33:50 smithi181 ceph-mon[94189]: pgmap v1657: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 68 KiB/s wr, 23 op/s 2023-03-09T15:33:51.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:33:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2140.968226 secs 2023-03-09T15:33:51.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:33:50 smithi038 ceph-mon[92422]: pgmap v1657: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 68 KiB/s wr, 23 op/s 2023-03-09T15:33:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:33:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2143.168773 secs 2023-03-09T15:33:53.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:33:52 smithi122 ceph-mon[93869]: pgmap v1658: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 68 KiB/s wr, 10 op/s 2023-03-09T15:33:53.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:33:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2143.168773 secs 2023-03-09T15:33:53.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:33:52 smithi181 ceph-mon[94189]: pgmap v1658: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 68 KiB/s wr, 10 op/s 2023-03-09T15:33:53.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:33:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2143.168773 secs 2023-03-09T15:33:53.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:33:52 smithi038 ceph-mon[92422]: pgmap v1658: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 68 KiB/s wr, 10 op/s 2023-03-09T15:33:53.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2605.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:33:54.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:33:54 smithi122 ceph-mon[93869]: pgmap v1659: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 97 KiB/s wr, 10 op/s 2023-03-09T15:33:54.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:33:54 smithi181 ceph-mon[94189]: pgmap v1659: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 97 KiB/s wr, 10 op/s 2023-03-09T15:33:55.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:33:54 smithi038 ceph-mon[92422]: pgmap v1659: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 97 KiB/s wr, 10 op/s 2023-03-09T15:33:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:33:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2145.968287 secs 2023-03-09T15:33:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:33:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2145.968287 secs 2023-03-09T15:33:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:33:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2145.968287 secs 2023-03-09T15:33:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:33:56 smithi122 ceph-mon[93869]: pgmap v1660: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 55 KiB/s wr, 10 op/s 2023-03-09T15:33:56.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:33:56 smithi181 ceph-mon[94189]: pgmap v1660: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 55 KiB/s wr, 10 op/s 2023-03-09T15:33:57.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:33:56 smithi038 ceph-mon[92422]: pgmap v1660: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 55 KiB/s wr, 10 op/s 2023-03-09T15:33:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:33:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2148.168902 secs 2023-03-09T15:33:57.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:33:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2148.168902 secs 2023-03-09T15:33:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:33:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2148.168902 secs 2023-03-09T15:33:58.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2610.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:33:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:33:58 smithi122 ceph-mon[93869]: pgmap v1661: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 74 KiB/s wr, 9 op/s 2023-03-09T15:33:58.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:33:58 smithi181 ceph-mon[94189]: pgmap v1661: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 74 KiB/s wr, 9 op/s 2023-03-09T15:33:59.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:33:58 smithi038 ceph-mon[92422]: pgmap v1661: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 74 KiB/s wr, 9 op/s 2023-03-09T15:34:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:34:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2150.968346 secs 2023-03-09T15:34:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:34:00 smithi122 ceph-mon[93869]: pgmap v1662: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 49 KiB/s wr, 7 op/s 2023-03-09T15:34:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:34:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2150.968346 secs 2023-03-09T15:34:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:34:00 smithi181 ceph-mon[94189]: pgmap v1662: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 49 KiB/s wr, 7 op/s 2023-03-09T15:34:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:34:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2150.968346 secs 2023-03-09T15:34:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:34:00 smithi038 ceph-mon[92422]: pgmap v1662: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 49 KiB/s wr, 7 op/s 2023-03-09T15:34:03.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:34:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2153.168987 secs 2023-03-09T15:34:03.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:34:02 smithi122 ceph-mon[93869]: pgmap v1663: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 72 KiB/s wr, 11 op/s 2023-03-09T15:34:03.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:34:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2153.168987 secs 2023-03-09T15:34:03.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:34:02 smithi181 ceph-mon[94189]: pgmap v1663: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 72 KiB/s wr, 11 op/s 2023-03-09T15:34:03.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:34:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2153.168987 secs 2023-03-09T15:34:03.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:34:02 smithi038 ceph-mon[92422]: pgmap v1663: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 72 KiB/s wr, 11 op/s 2023-03-09T15:34:03.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2615.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:34:04.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:34:04 smithi122 ceph-mon[93869]: pgmap v1664: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 111 KiB/s wr, 23 op/s 2023-03-09T15:34:04.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:34:04 smithi181 ceph-mon[94189]: pgmap v1664: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 111 KiB/s wr, 23 op/s 2023-03-09T15:34:05.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:34:04 smithi038 ceph-mon[92422]: pgmap v1664: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 111 KiB/s wr, 23 op/s 2023-03-09T15:34:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:34:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2155.968407 secs 2023-03-09T15:34:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:34:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2155.968407 secs 2023-03-09T15:34:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:34:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2155.968407 secs 2023-03-09T15:34:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:34:06 smithi122 ceph-mon[93869]: pgmap v1665: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 82 KiB/s wr, 23 op/s 2023-03-09T15:34:06.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:34:06 smithi181 ceph-mon[94189]: pgmap v1665: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 82 KiB/s wr, 23 op/s 2023-03-09T15:34:07.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:34:06 smithi038 ceph-mon[92422]: pgmap v1665: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 82 KiB/s wr, 23 op/s 2023-03-09T15:34:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:34:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2158.169089 secs 2023-03-09T15:34:07.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:34:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2158.169089 secs 2023-03-09T15:34:08.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:34:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2158.169089 secs 2023-03-09T15:34:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2620.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:34:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:34:08 smithi122 ceph-mon[93869]: pgmap v1666: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 82 KiB/s wr, 23 op/s 2023-03-09T15:34:08.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:34:08 smithi181 ceph-mon[94189]: pgmap v1666: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 82 KiB/s wr, 23 op/s 2023-03-09T15:34:09.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:34:08 smithi038 ceph-mon[92422]: pgmap v1666: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 82 KiB/s wr, 23 op/s 2023-03-09T15:34:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:34:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2160.968467 secs 2023-03-09T15:34:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:34:10 smithi122 ceph-mon[93869]: osdmap e318: 12 total, 12 up, 12 in 2023-03-09T15:34:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:34:10 smithi122 ceph-mon[93869]: pgmap v1668: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 76 KiB/s wr, 25 op/s 2023-03-09T15:34:10.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:34:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2160.968467 secs 2023-03-09T15:34:10.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:34:10 smithi181 ceph-mon[94189]: osdmap e318: 12 total, 12 up, 12 in 2023-03-09T15:34:10.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:34:10 smithi181 ceph-mon[94189]: pgmap v1668: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 76 KiB/s wr, 25 op/s 2023-03-09T15:34:11.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:34:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2160.968467 secs 2023-03-09T15:34:11.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:34:10 smithi038 ceph-mon[92422]: osdmap e318: 12 total, 12 up, 12 in 2023-03-09T15:34:11.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:34:10 smithi038 ceph-mon[92422]: pgmap v1668: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 76 KiB/s wr, 25 op/s 2023-03-09T15:34:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:34:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2163.169189 secs 2023-03-09T15:34:13.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:34:12 smithi122 ceph-mon[93869]: pgmap v1669: 129 pgs: 1 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 121 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 49 KiB/s wr, 22 op/s 2023-03-09T15:34:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:34:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2163.169189 secs 2023-03-09T15:34:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:34:12 smithi181 ceph-mon[94189]: pgmap v1669: 129 pgs: 1 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 121 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 49 KiB/s wr, 22 op/s 2023-03-09T15:34:13.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:34:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2163.169189 secs 2023-03-09T15:34:13.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:34:12 smithi038 ceph-mon[92422]: pgmap v1669: 129 pgs: 1 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 121 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 49 KiB/s wr, 22 op/s 2023-03-09T15:34:13.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2625.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:34:14.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:34:14 smithi122 ceph-mon[93869]: pgmap v1670: 129 pgs: 1 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 121 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 22 KiB/s wr, 8 op/s 2023-03-09T15:34:14.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:34:14 smithi181 ceph-mon[94189]: pgmap v1670: 129 pgs: 1 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 121 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 22 KiB/s wr, 8 op/s 2023-03-09T15:34:15.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:34:14 smithi038 ceph-mon[92422]: pgmap v1670: 129 pgs: 1 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 121 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 22 KiB/s wr, 8 op/s 2023-03-09T15:34:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:34:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2165.968528 secs 2023-03-09T15:34:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:34:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2165.968528 secs 2023-03-09T15:34:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:34:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2165.968528 secs 2023-03-09T15:34:16.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:34:16 smithi122 ceph-mon[93869]: pgmap v1671: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 21 KiB/s wr, 8 op/s 2023-03-09T15:34:16.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:34:16 smithi181 ceph-mon[94189]: pgmap v1671: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 21 KiB/s wr, 8 op/s 2023-03-09T15:34:17.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:34:16 smithi038 ceph-mon[92422]: pgmap v1671: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 21 KiB/s wr, 8 op/s 2023-03-09T15:34:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:34:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2168.169295 secs 2023-03-09T15:34:17.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:34:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2168.169295 secs 2023-03-09T15:34:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:34:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2168.169295 secs 2023-03-09T15:34:18.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2630.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:34:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:34:18 smithi122 ceph-mon[93869]: pgmap v1672: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 21 KiB/s wr, 8 op/s 2023-03-09T15:34:18.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:34:18 smithi122 ceph-mon[93869]: osdmap e319: 12 total, 12 up, 12 in 2023-03-09T15:34:18.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:34:18 smithi181 ceph-mon[94189]: pgmap v1672: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 21 KiB/s wr, 8 op/s 2023-03-09T15:34:18.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:34:18 smithi181 ceph-mon[94189]: osdmap e319: 12 total, 12 up, 12 in 2023-03-09T15:34:19.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:34:18 smithi038 ceph-mon[92422]: pgmap v1672: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 21 KiB/s wr, 8 op/s 2023-03-09T15:34:19.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:34:18 smithi038 ceph-mon[92422]: osdmap e319: 12 total, 12 up, 12 in 2023-03-09T15:34:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:34:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2170.968609 secs 2023-03-09T15:34:21.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:34:20 smithi122 ceph-mon[93869]: pgmap v1674: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 21 KiB/s wr, 8 op/s 2023-03-09T15:34:21.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:34:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2170.968609 secs 2023-03-09T15:34:21.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:34:20 smithi181 ceph-mon[94189]: pgmap v1674: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 21 KiB/s wr, 8 op/s 2023-03-09T15:34:21.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:34:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2170.968609 secs 2023-03-09T15:34:21.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:34:20 smithi038 ceph-mon[92422]: pgmap v1674: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 21 KiB/s wr, 8 op/s 2023-03-09T15:34:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:34:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2173.169410 secs 2023-03-09T15:34:23.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:34:22 smithi122 ceph-mon[93869]: pgmap v1675: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 21 KiB/s wr, 8 op/s 2023-03-09T15:34:23.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:34:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2173.169410 secs 2023-03-09T15:34:23.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:34:22 smithi181 ceph-mon[94189]: pgmap v1675: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 21 KiB/s wr, 8 op/s 2023-03-09T15:34:23.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:34:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2173.169410 secs 2023-03-09T15:34:23.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:34:22 smithi038 ceph-mon[92422]: pgmap v1675: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 21 KiB/s wr, 8 op/s 2023-03-09T15:34:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2635.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:34:24.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:34:24 smithi122 ceph-mon[93869]: pgmap v1676: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T15:34:24.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:34:24 smithi181 ceph-mon[94189]: pgmap v1676: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T15:34:25.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:34:24 smithi038 ceph-mon[92422]: pgmap v1676: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T15:34:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:34:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2175.968687 secs 2023-03-09T15:34:25.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:34:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2175.968687 secs 2023-03-09T15:34:26.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:34:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2175.968687 secs 2023-03-09T15:34:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:34:26 smithi122 ceph-mon[93869]: pgmap v1677: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T15:34:26.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:34:26 smithi181 ceph-mon[94189]: pgmap v1677: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T15:34:27.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:34:26 smithi038 ceph-mon[92422]: pgmap v1677: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T15:34:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:34:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2178.169559 secs 2023-03-09T15:34:27.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:34:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2178.169559 secs 2023-03-09T15:34:28.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:34:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2178.169559 secs 2023-03-09T15:34:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2640.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:34:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:34:28 smithi122 ceph-mon[93869]: pgmap v1678: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 6 op/s 2023-03-09T15:34:28.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:34:28 smithi181 ceph-mon[94189]: pgmap v1678: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 6 op/s 2023-03-09T15:34:29.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:34:28 smithi038 ceph-mon[92422]: pgmap v1678: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 6 op/s 2023-03-09T15:34:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:34:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2180.968747 secs 2023-03-09T15:34:31.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:34:30 smithi122 ceph-mon[93869]: pgmap v1679: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 5 op/s 2023-03-09T15:34:31.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:34:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2180.968747 secs 2023-03-09T15:34:31.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:34:30 smithi181 ceph-mon[94189]: pgmap v1679: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 5 op/s 2023-03-09T15:34:31.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:34:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2180.968747 secs 2023-03-09T15:34:31.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:34:30 smithi038 ceph-mon[92422]: pgmap v1679: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 5 op/s 2023-03-09T15:34:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:34:32 smithi122 ceph-mon[93869]: pgmap v1680: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 KiB/s wr, 7 op/s 2023-03-09T15:34:33.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:34:32 smithi181 ceph-mon[94189]: pgmap v1680: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 KiB/s wr, 7 op/s 2023-03-09T15:34:33.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:34:32 smithi038 ceph-mon[92422]: pgmap v1680: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 KiB/s wr, 7 op/s 2023-03-09T15:34:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2645.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:34:34.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:34:34 smithi122 ceph-mon[93869]: pgmap v1681: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 35 KiB/s wr, 7 op/s 2023-03-09T15:34:34.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:34:34 smithi181 ceph-mon[94189]: pgmap v1681: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 35 KiB/s wr, 7 op/s 2023-03-09T15:34:35.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:34:34 smithi038 ceph-mon[92422]: pgmap v1681: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 35 KiB/s wr, 7 op/s 2023-03-09T15:34:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:34:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2185.968806 secs 2023-03-09T15:34:35.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:34:35 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:34:35.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:34:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2185.968806 secs 2023-03-09T15:34:35.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:34:35 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:34:36.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:34:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2185.968806 secs 2023-03-09T15:34:36.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:34:35 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:34:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:34:36 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:34:36.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:34:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:34:36.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:34:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:34:36.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:34:36 smithi122 ceph-mon[93869]: pgmap v1682: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T15:34:36.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:34:36 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:34:36.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:34:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:34:36.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:34:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:34:36.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:34:36 smithi181 ceph-mon[94189]: pgmap v1682: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T15:34:37.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:34:36 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:34:37.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:34:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:34:37.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:34:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:34:37.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:34:36 smithi038 ceph-mon[92422]: pgmap v1682: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T15:34:38.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2650.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:34:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:34:38 smithi122 ceph-mon[93869]: pgmap v1683: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 55 KiB/s wr, 8 op/s 2023-03-09T15:34:39.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:34:38 smithi181 ceph-mon[94189]: pgmap v1683: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 55 KiB/s wr, 8 op/s 2023-03-09T15:34:39.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:34:38 smithi038 ceph-mon[92422]: pgmap v1683: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 55 KiB/s wr, 8 op/s 2023-03-09T15:34:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:34:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2190.968890 secs 2023-03-09T15:34:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:34:40 smithi122 ceph-mon[93869]: pgmap v1684: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 55 KiB/s wr, 7 op/s 2023-03-09T15:34:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:34:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2190.968890 secs 2023-03-09T15:34:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:34:40 smithi181 ceph-mon[94189]: pgmap v1684: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 55 KiB/s wr, 7 op/s 2023-03-09T15:34:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:34:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2190.968890 secs 2023-03-09T15:34:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:34:40 smithi038 ceph-mon[92422]: pgmap v1684: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 55 KiB/s wr, 7 op/s 2023-03-09T15:34:43.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:34:42 smithi122 ceph-mon[93869]: pgmap v1685: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 55 KiB/s wr, 9 op/s 2023-03-09T15:34:43.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:34:42 smithi181 ceph-mon[94189]: pgmap v1685: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 55 KiB/s wr, 9 op/s 2023-03-09T15:34:43.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:34:42 smithi038 ceph-mon[92422]: pgmap v1685: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 55 KiB/s wr, 9 op/s 2023-03-09T15:34:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2655.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:34:44.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:34:44 smithi122 ceph-mon[93869]: pgmap v1686: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 105 KiB/s wr, 11 op/s 2023-03-09T15:34:44.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:34:44 smithi181 ceph-mon[94189]: pgmap v1686: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 105 KiB/s wr, 11 op/s 2023-03-09T15:34:45.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:34:44 smithi038 ceph-mon[92422]: pgmap v1686: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 105 KiB/s wr, 11 op/s 2023-03-09T15:34:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:34:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2195.968950 secs 2023-03-09T15:34:45.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:34:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2195.968950 secs 2023-03-09T15:34:46.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:34:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2195.968950 secs 2023-03-09T15:34:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:34:46 smithi122 ceph-mon[93869]: pgmap v1687: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 87 KiB/s wr, 11 op/s 2023-03-09T15:34:46.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:34:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:34:46.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:34:46 smithi181 ceph-mon[94189]: pgmap v1687: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 87 KiB/s wr, 11 op/s 2023-03-09T15:34:46.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:34:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:34:47.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:34:46 smithi038 ceph-mon[92422]: pgmap v1687: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 87 KiB/s wr, 11 op/s 2023-03-09T15:34:47.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:34:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:34:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:34:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:34:47.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:34:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:34:47.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:34:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:34:47.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:34:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:34:47.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:34:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:34:47.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:34:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:34:48.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:34:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:34:48.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:34:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:34:48.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:34:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:34:48.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2660.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:34:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:34:48 smithi122 ceph-mon[93869]: pgmap v1688: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 111 KiB/s wr, 11 op/s 2023-03-09T15:34:48.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:34:48 smithi181 ceph-mon[94189]: pgmap v1688: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 111 KiB/s wr, 11 op/s 2023-03-09T15:34:49.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:34:48 smithi038 ceph-mon[92422]: pgmap v1688: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 111 KiB/s wr, 11 op/s 2023-03-09T15:34:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:34:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2200.968980 secs 2023-03-09T15:34:51.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:34:50 smithi122 ceph-mon[93869]: pgmap v1689: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 75 KiB/s wr, 9 op/s 2023-03-09T15:34:51.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:34:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2200.968980 secs 2023-03-09T15:34:51.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:34:50 smithi181 ceph-mon[94189]: pgmap v1689: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 75 KiB/s wr, 9 op/s 2023-03-09T15:34:51.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:34:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2200.968980 secs 2023-03-09T15:34:51.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:34:50 smithi038 ceph-mon[92422]: pgmap v1689: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 75 KiB/s wr, 9 op/s 2023-03-09T15:34:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:34:52 smithi122 ceph-mon[93869]: pgmap v1690: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 76 KiB/s wr, 11 op/s 2023-03-09T15:34:53.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:34:52 smithi181 ceph-mon[94189]: pgmap v1690: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 76 KiB/s wr, 11 op/s 2023-03-09T15:34:53.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:34:52 smithi038 ceph-mon[92422]: pgmap v1690: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 76 KiB/s wr, 11 op/s 2023-03-09T15:34:53.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2665.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:34:54.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:34:54 smithi122 ceph-mon[93869]: pgmap v1691: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 126 KiB/s wr, 11 op/s 2023-03-09T15:34:54.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:34:54 smithi181 ceph-mon[94189]: pgmap v1691: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 126 KiB/s wr, 11 op/s 2023-03-09T15:34:55.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:34:54 smithi038 ceph-mon[92422]: pgmap v1691: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 126 KiB/s wr, 11 op/s 2023-03-09T15:34:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:34:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2205.969042 secs 2023-03-09T15:34:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:34:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2205.969042 secs 2023-03-09T15:34:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:34:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2205.969042 secs 2023-03-09T15:34:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:34:56 smithi122 ceph-mon[93869]: pgmap v1692: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 75 KiB/s wr, 10 op/s 2023-03-09T15:34:56.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:34:56 smithi181 ceph-mon[94189]: pgmap v1692: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 75 KiB/s wr, 10 op/s 2023-03-09T15:34:57.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:34:56 smithi038 ceph-mon[92422]: pgmap v1692: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 75 KiB/s wr, 10 op/s 2023-03-09T15:34:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:34:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2208.170198 secs 2023-03-09T15:34:57.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:34:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2208.170198 secs 2023-03-09T15:34:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:34:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2208.170198 secs 2023-03-09T15:34:58.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2670.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:34:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:34:58 smithi122 ceph-mon[93869]: pgmap v1693: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 103 KiB/s wr, 10 op/s 2023-03-09T15:34:58.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:34:58 smithi181 ceph-mon[94189]: pgmap v1693: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 103 KiB/s wr, 10 op/s 2023-03-09T15:34:59.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:34:58 smithi038 ceph-mon[92422]: pgmap v1693: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 103 KiB/s wr, 10 op/s 2023-03-09T15:34:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:34:59 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:34:59.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:34:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:34:59.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:34:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:34:59.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:34:59 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:34:59.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:34:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:34:59.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:34:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:35:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:34:59 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:35:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:34:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:35:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:34:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:35:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:35:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2210.969101 secs 2023-03-09T15:35:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:35:00 smithi122 ceph-mon[93869]: pgmap v1694: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 78 KiB/s wr, 8 op/s 2023-03-09T15:35:00.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:35:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2210.969101 secs 2023-03-09T15:35:00.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:35:00 smithi181 ceph-mon[94189]: pgmap v1694: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 78 KiB/s wr, 8 op/s 2023-03-09T15:35:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:35:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2210.969101 secs 2023-03-09T15:35:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:35:00 smithi038 ceph-mon[92422]: pgmap v1694: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 78 KiB/s wr, 8 op/s 2023-03-09T15:35:03.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:35:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2213.170309 secs 2023-03-09T15:35:03.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:35:02 smithi122 ceph-mon[93869]: pgmap v1695: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 101 KiB/s wr, 12 op/s 2023-03-09T15:35:03.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:35:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2213.170309 secs 2023-03-09T15:35:03.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:35:02 smithi181 ceph-mon[94189]: pgmap v1695: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 101 KiB/s wr, 12 op/s 2023-03-09T15:35:03.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:35:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2213.170309 secs 2023-03-09T15:35:03.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:35:02 smithi038 ceph-mon[92422]: pgmap v1695: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 101 KiB/s wr, 12 op/s 2023-03-09T15:35:03.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2675.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:35:04.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:35:04 smithi122 ceph-mon[93869]: pgmap v1696: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 159 KiB/s wr, 24 op/s 2023-03-09T15:35:04.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:35:04 smithi181 ceph-mon[94189]: pgmap v1696: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 159 KiB/s wr, 24 op/s 2023-03-09T15:35:05.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:35:04 smithi038 ceph-mon[92422]: pgmap v1696: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 159 KiB/s wr, 24 op/s 2023-03-09T15:35:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:35:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2215.969160 secs 2023-03-09T15:35:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:35:05 smithi122 ceph-mon[93869]: osdmap e320: 12 total, 12 up, 12 in 2023-03-09T15:35:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:35:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2215.969160 secs 2023-03-09T15:35:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:35:05 smithi181 ceph-mon[94189]: osdmap e320: 12 total, 12 up, 12 in 2023-03-09T15:35:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:35:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2215.969160 secs 2023-03-09T15:35:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:35:05 smithi038 ceph-mon[92422]: osdmap e320: 12 total, 12 up, 12 in 2023-03-09T15:35:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:35:06 smithi122 ceph-mon[93869]: pgmap v1698: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 131 KiB/s wr, 27 op/s 2023-03-09T15:35:06.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:35:06 smithi181 ceph-mon[94189]: pgmap v1698: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 131 KiB/s wr, 27 op/s 2023-03-09T15:35:07.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:35:06 smithi038 ceph-mon[92422]: pgmap v1698: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 131 KiB/s wr, 27 op/s 2023-03-09T15:35:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:35:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2218.170438 secs 2023-03-09T15:35:07.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:35:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2218.170438 secs 2023-03-09T15:35:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:35:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2218.170438 secs 2023-03-09T15:35:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2680.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:35:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:35:08 smithi122 ceph-mon[93869]: pgmap v1699: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 115 KiB/s wr, 26 op/s 2023-03-09T15:35:08.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:35:08 smithi181 ceph-mon[94189]: pgmap v1699: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 115 KiB/s wr, 26 op/s 2023-03-09T15:35:09.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:35:08 smithi038 ceph-mon[92422]: pgmap v1699: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 115 KiB/s wr, 26 op/s 2023-03-09T15:35:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:35:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2220.969218 secs 2023-03-09T15:35:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:35:10 smithi122 ceph-mon[93869]: pgmap v1700: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 115 KiB/s wr, 26 op/s 2023-03-09T15:35:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:35:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2220.969218 secs 2023-03-09T15:35:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:35:10 smithi181 ceph-mon[94189]: pgmap v1700: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 115 KiB/s wr, 26 op/s 2023-03-09T15:35:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:35:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2220.969218 secs 2023-03-09T15:35:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:35:10 smithi038 ceph-mon[92422]: pgmap v1700: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 115 KiB/s wr, 26 op/s 2023-03-09T15:35:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:35:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2223.170562 secs 2023-03-09T15:35:13.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:35:12 smithi122 ceph-mon[93869]: pgmap v1701: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 88 KiB/s wr, 23 op/s 2023-03-09T15:35:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:35:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2223.170562 secs 2023-03-09T15:35:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:35:12 smithi181 ceph-mon[94189]: pgmap v1701: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 88 KiB/s wr, 23 op/s 2023-03-09T15:35:13.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:35:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2223.170562 secs 2023-03-09T15:35:13.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:35:12 smithi038 ceph-mon[92422]: pgmap v1701: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 88 KiB/s wr, 23 op/s 2023-03-09T15:35:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2685.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:35:14.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:35:14 smithi122 ceph-mon[93869]: osdmap e321: 12 total, 12 up, 12 in 2023-03-09T15:35:14.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:35:14 smithi122 ceph-mon[93869]: pgmap v1703: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 38 KiB/s wr, 8 op/s 2023-03-09T15:35:14.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:35:14 smithi181 ceph-mon[94189]: osdmap e321: 12 total, 12 up, 12 in 2023-03-09T15:35:14.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:35:14 smithi181 ceph-mon[94189]: pgmap v1703: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 38 KiB/s wr, 8 op/s 2023-03-09T15:35:15.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:35:14 smithi038 ceph-mon[92422]: osdmap e321: 12 total, 12 up, 12 in 2023-03-09T15:35:15.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:35:14 smithi038 ceph-mon[92422]: pgmap v1703: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 38 KiB/s wr, 8 op/s 2023-03-09T15:35:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:35:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2225.969276 secs 2023-03-09T15:35:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:35:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2225.969276 secs 2023-03-09T15:35:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:35:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2225.969276 secs 2023-03-09T15:35:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:35:16 smithi122 ceph-mon[93869]: pgmap v1704: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 35 KiB/s wr, 7 op/s 2023-03-09T15:35:16.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:35:16 smithi181 ceph-mon[94189]: pgmap v1704: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 35 KiB/s wr, 7 op/s 2023-03-09T15:35:17.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:35:16 smithi038 ceph-mon[92422]: pgmap v1704: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 35 KiB/s wr, 7 op/s 2023-03-09T15:35:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:35:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2228.170688 secs 2023-03-09T15:35:17.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:35:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2228.170688 secs 2023-03-09T15:35:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:35:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2228.170688 secs 2023-03-09T15:35:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2690.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:35:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:35:18 smithi122 ceph-mon[93869]: pgmap v1705: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T15:35:18.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:35:18 smithi181 ceph-mon[94189]: pgmap v1705: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T15:35:19.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:35:18 smithi038 ceph-mon[92422]: pgmap v1705: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T15:35:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:35:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2230.969335 secs 2023-03-09T15:35:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:35:20 smithi122 ceph-mon[93869]: pgmap v1706: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T15:35:21.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:35:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2230.969335 secs 2023-03-09T15:35:21.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:35:20 smithi181 ceph-mon[94189]: pgmap v1706: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T15:35:21.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:35:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2230.969335 secs 2023-03-09T15:35:21.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:35:20 smithi038 ceph-mon[92422]: pgmap v1706: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T15:35:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:35:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2233.170821 secs 2023-03-09T15:35:23.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:35:22 smithi122 ceph-mon[93869]: pgmap v1707: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T15:35:23.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:35:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2233.170821 secs 2023-03-09T15:35:23.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:35:22 smithi181 ceph-mon[94189]: pgmap v1707: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T15:35:23.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:35:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2233.170821 secs 2023-03-09T15:35:23.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:35:22 smithi038 ceph-mon[92422]: pgmap v1707: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T15:35:23.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2695.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:35:24.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:35:24 smithi122 ceph-mon[93869]: pgmap v1708: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 21 KiB/s wr, 8 op/s 2023-03-09T15:35:24.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:35:24 smithi181 ceph-mon[94189]: pgmap v1708: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 21 KiB/s wr, 8 op/s 2023-03-09T15:35:25.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:35:24 smithi038 ceph-mon[92422]: pgmap v1708: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 21 KiB/s wr, 8 op/s 2023-03-09T15:35:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:35:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2235.969396 secs 2023-03-09T15:35:25.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:35:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2235.969396 secs 2023-03-09T15:35:26.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:35:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2235.969396 secs 2023-03-09T15:35:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:35:26 smithi122 ceph-mon[93869]: pgmap v1709: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 19 KiB/s wr, 8 op/s 2023-03-09T15:35:26.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:35:26 smithi181 ceph-mon[94189]: pgmap v1709: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 19 KiB/s wr, 8 op/s 2023-03-09T15:35:27.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:35:26 smithi038 ceph-mon[92422]: pgmap v1709: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 19 KiB/s wr, 8 op/s 2023-03-09T15:35:27.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:35:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2238.170928 secs 2023-03-09T15:35:27.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:35:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2238.170928 secs 2023-03-09T15:35:28.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:35:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2238.170928 secs 2023-03-09T15:35:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2700.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:35:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:35:28 smithi122 ceph-mon[93869]: pgmap v1710: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 8 op/s 2023-03-09T15:35:28.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:35:28 smithi181 ceph-mon[94189]: pgmap v1710: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 8 op/s 2023-03-09T15:35:29.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:35:28 smithi038 ceph-mon[92422]: pgmap v1710: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 8 op/s 2023-03-09T15:35:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:35:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2240.969457 secs 2023-03-09T15:35:31.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:35:30 smithi122 ceph-mon[93869]: pgmap v1711: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 6 op/s 2023-03-09T15:35:31.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:35:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2240.969457 secs 2023-03-09T15:35:31.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:35:30 smithi181 ceph-mon[94189]: pgmap v1711: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 6 op/s 2023-03-09T15:35:31.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:35:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2240.969457 secs 2023-03-09T15:35:31.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:35:30 smithi038 ceph-mon[92422]: pgmap v1711: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 6 op/s 2023-03-09T15:35:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:35:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2243.170991 secs 2023-03-09T15:35:33.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:35:32 smithi122 ceph-mon[93869]: pgmap v1712: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 18 KiB/s wr, 8 op/s 2023-03-09T15:35:33.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:35:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2243.170991 secs 2023-03-09T15:35:33.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:35:32 smithi181 ceph-mon[94189]: pgmap v1712: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 18 KiB/s wr, 8 op/s 2023-03-09T15:35:33.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:35:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2243.170991 secs 2023-03-09T15:35:33.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:35:32 smithi038 ceph-mon[92422]: pgmap v1712: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 18 KiB/s wr, 8 op/s 2023-03-09T15:35:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2705.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:35:34.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:35:34 smithi122 ceph-mon[93869]: pgmap v1713: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 89 KiB/s wr, 9 op/s 2023-03-09T15:35:34.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:35:34 smithi181 ceph-mon[94189]: pgmap v1713: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 89 KiB/s wr, 9 op/s 2023-03-09T15:35:35.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:35:34 smithi038 ceph-mon[92422]: pgmap v1713: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 89 KiB/s wr, 9 op/s 2023-03-09T15:35:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:35:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2245.969517 secs 2023-03-09T15:35:35.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:35:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2245.969517 secs 2023-03-09T15:35:36.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:35:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2245.969517 secs 2023-03-09T15:35:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:35:36 smithi122 ceph-mon[93869]: pgmap v1714: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 72 KiB/s wr, 10 op/s 2023-03-09T15:35:36.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:35:36 smithi181 ceph-mon[94189]: pgmap v1714: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 72 KiB/s wr, 10 op/s 2023-03-09T15:35:37.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:35:36 smithi038 ceph-mon[92422]: pgmap v1714: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 72 KiB/s wr, 10 op/s 2023-03-09T15:35:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:35:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2248.171131 secs 2023-03-09T15:35:37.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:35:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2248.171131 secs 2023-03-09T15:35:38.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:35:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2248.171131 secs 2023-03-09T15:35:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2710.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:35:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:35:38 smithi122 ceph-mon[93869]: pgmap v1715: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 118 KiB/s wr, 10 op/s 2023-03-09T15:35:38.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:35:38 smithi181 ceph-mon[94189]: pgmap v1715: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 118 KiB/s wr, 10 op/s 2023-03-09T15:35:39.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:35:38 smithi038 ceph-mon[92422]: pgmap v1715: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 118 KiB/s wr, 10 op/s 2023-03-09T15:35:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:35:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2250.969578 secs 2023-03-09T15:35:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:35:40 smithi122 ceph-mon[93869]: pgmap v1716: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 117 KiB/s wr, 9 op/s 2023-03-09T15:35:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:35:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2250.969578 secs 2023-03-09T15:35:41.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:35:40 smithi181 ceph-mon[94189]: pgmap v1716: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 117 KiB/s wr, 9 op/s 2023-03-09T15:35:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:35:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2250.969578 secs 2023-03-09T15:35:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:35:40 smithi038 ceph-mon[92422]: pgmap v1716: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 117 KiB/s wr, 9 op/s 2023-03-09T15:35:43.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:35:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2253.171250 secs 2023-03-09T15:35:43.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:35:42 smithi122 ceph-mon[93869]: pgmap v1717: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 118 KiB/s wr, 11 op/s 2023-03-09T15:35:43.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:35:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2253.171250 secs 2023-03-09T15:35:43.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:35:42 smithi181 ceph-mon[94189]: pgmap v1717: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 118 KiB/s wr, 11 op/s 2023-03-09T15:35:43.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:35:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2253.171250 secs 2023-03-09T15:35:43.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:35:42 smithi038 ceph-mon[92422]: pgmap v1717: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 118 KiB/s wr, 11 op/s 2023-03-09T15:35:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2715.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:35:44.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:35:44 smithi122 ceph-mon[93869]: pgmap v1718: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 132 KiB/s wr, 12 op/s 2023-03-09T15:35:44.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:35:44 smithi181 ceph-mon[94189]: pgmap v1718: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 132 KiB/s wr, 12 op/s 2023-03-09T15:35:45.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:35:44 smithi038 ceph-mon[92422]: pgmap v1718: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 132 KiB/s wr, 12 op/s 2023-03-09T15:35:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:35:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2255.969638 secs 2023-03-09T15:35:45.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:35:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2255.969638 secs 2023-03-09T15:35:46.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:35:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2255.969638 secs 2023-03-09T15:35:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:35:46 smithi122 ceph-mon[93869]: pgmap v1719: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 61 KiB/s wr, 11 op/s 2023-03-09T15:35:46.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:35:46 smithi181 ceph-mon[94189]: pgmap v1719: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 61 KiB/s wr, 11 op/s 2023-03-09T15:35:47.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:35:46 smithi038 ceph-mon[92422]: pgmap v1719: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 61 KiB/s wr, 11 op/s 2023-03-09T15:35:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:35:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2258.171334 secs 2023-03-09T15:35:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:35:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:35:47.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:35:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2258.171334 secs 2023-03-09T15:35:47.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:35:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:35:48.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:35:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2258.171334 secs 2023-03-09T15:35:48.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:35:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:35:48.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2720.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:35:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:35:48 smithi122 ceph-mon[93869]: pgmap v1720: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 119 KiB/s wr, 11 op/s 2023-03-09T15:35:48.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:35:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:35:48.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:35:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:35:48.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:35:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:35:48.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:35:48 smithi181 ceph-mon[94189]: pgmap v1720: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 119 KiB/s wr, 11 op/s 2023-03-09T15:35:48.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:35:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:35:48.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:35:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:35:48.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:35:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:35:49.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:35:48 smithi038 ceph-mon[92422]: pgmap v1720: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 119 KiB/s wr, 11 op/s 2023-03-09T15:35:49.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:35:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:35:49.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:35:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:35:49.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:35:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:35:51.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:35:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2260.969699 secs 2023-03-09T15:35:51.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:35:50 smithi122 ceph-mon[93869]: pgmap v1721: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 73 KiB/s wr, 9 op/s 2023-03-09T15:35:51.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:35:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2260.969699 secs 2023-03-09T15:35:51.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:35:50 smithi181 ceph-mon[94189]: pgmap v1721: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 73 KiB/s wr, 9 op/s 2023-03-09T15:35:51.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:35:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2260.969699 secs 2023-03-09T15:35:51.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:35:50 smithi038 ceph-mon[92422]: pgmap v1721: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 73 KiB/s wr, 9 op/s 2023-03-09T15:35:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:35:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2263.171447 secs 2023-03-09T15:35:53.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:35:52 smithi122 ceph-mon[93869]: pgmap v1722: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 73 KiB/s wr, 11 op/s 2023-03-09T15:35:53.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:35:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2263.171447 secs 2023-03-09T15:35:53.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:35:52 smithi181 ceph-mon[94189]: pgmap v1722: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 73 KiB/s wr, 11 op/s 2023-03-09T15:35:53.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:35:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2263.171447 secs 2023-03-09T15:35:53.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:35:52 smithi038 ceph-mon[92422]: pgmap v1722: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 73 KiB/s wr, 11 op/s 2023-03-09T15:35:53.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2725.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:35:54.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:35:54 smithi122 ceph-mon[93869]: pgmap v1723: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 101 KiB/s wr, 11 op/s 2023-03-09T15:35:54.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:35:54 smithi181 ceph-mon[94189]: pgmap v1723: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 101 KiB/s wr, 11 op/s 2023-03-09T15:35:55.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:35:54 smithi038 ceph-mon[92422]: pgmap v1723: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 101 KiB/s wr, 11 op/s 2023-03-09T15:35:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:35:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2265.969760 secs 2023-03-09T15:35:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:35:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2265.969760 secs 2023-03-09T15:35:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:35:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2265.969760 secs 2023-03-09T15:35:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:35:56 smithi122 ceph-mon[93869]: pgmap v1724: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 86 KiB/s wr, 10 op/s 2023-03-09T15:35:56.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:35:56 smithi181 ceph-mon[94189]: pgmap v1724: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 86 KiB/s wr, 10 op/s 2023-03-09T15:35:57.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:35:56 smithi038 ceph-mon[92422]: pgmap v1724: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 86 KiB/s wr, 10 op/s 2023-03-09T15:35:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:35:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2268.171562 secs 2023-03-09T15:35:57.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:35:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2268.171562 secs 2023-03-09T15:35:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:35:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2268.171562 secs 2023-03-09T15:35:58.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2730.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:35:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:35:58 smithi122 ceph-mon[93869]: pgmap v1725: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 103 KiB/s wr, 10 op/s 2023-03-09T15:35:58.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:35:58 smithi181 ceph-mon[94189]: pgmap v1725: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 103 KiB/s wr, 10 op/s 2023-03-09T15:35:59.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:35:58 smithi038 ceph-mon[92422]: pgmap v1725: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 103 KiB/s wr, 10 op/s 2023-03-09T15:36:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:36:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2270.969822 secs 2023-03-09T15:36:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:36:00 smithi122 ceph-mon[93869]: pgmap v1726: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 45 KiB/s wr, 7 op/s 2023-03-09T15:36:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:36:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2270.969822 secs 2023-03-09T15:36:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:36:00 smithi181 ceph-mon[94189]: pgmap v1726: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 45 KiB/s wr, 7 op/s 2023-03-09T15:36:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:36:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2270.969822 secs 2023-03-09T15:36:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:36:00 smithi038 ceph-mon[92422]: pgmap v1726: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 45 KiB/s wr, 7 op/s 2023-03-09T15:36:03.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:36:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2273.171706 secs 2023-03-09T15:36:03.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:36:02 smithi122 ceph-mon[93869]: pgmap v1727: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 68 KiB/s wr, 11 op/s 2023-03-09T15:36:03.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:36:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2273.171706 secs 2023-03-09T15:36:03.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:36:02 smithi181 ceph-mon[94189]: pgmap v1727: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 68 KiB/s wr, 11 op/s 2023-03-09T15:36:03.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:36:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2273.171706 secs 2023-03-09T15:36:03.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:36:02 smithi038 ceph-mon[92422]: pgmap v1727: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 68 KiB/s wr, 11 op/s 2023-03-09T15:36:03.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2735.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:36:04.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:36:04 smithi122 ceph-mon[93869]: pgmap v1728: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 94 KiB/s wr, 23 op/s 2023-03-09T15:36:04.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:36:04 smithi181 ceph-mon[94189]: pgmap v1728: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 94 KiB/s wr, 23 op/s 2023-03-09T15:36:05.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:36:04 smithi038 ceph-mon[92422]: pgmap v1728: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 94 KiB/s wr, 23 op/s 2023-03-09T15:36:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:36:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2275.969880 secs 2023-03-09T15:36:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:36:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2275.969880 secs 2023-03-09T15:36:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:36:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2275.969880 secs 2023-03-09T15:36:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:36:06 smithi122 ceph-mon[93869]: pgmap v1729: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 66 KiB/s wr, 22 op/s 2023-03-09T15:36:06.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:36:06 smithi181 ceph-mon[94189]: pgmap v1729: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 66 KiB/s wr, 22 op/s 2023-03-09T15:36:07.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:36:06 smithi038 ceph-mon[92422]: pgmap v1729: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 66 KiB/s wr, 22 op/s 2023-03-09T15:36:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:36:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2278.171826 secs 2023-03-09T15:36:07.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:36:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2278.171826 secs 2023-03-09T15:36:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:36:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2278.171826 secs 2023-03-09T15:36:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2740.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:36:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:36:08 smithi122 ceph-mon[93869]: pgmap v1730: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 69 KiB/s wr, 22 op/s 2023-03-09T15:36:08.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:36:08 smithi181 ceph-mon[94189]: pgmap v1730: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 69 KiB/s wr, 22 op/s 2023-03-09T15:36:09.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:36:08 smithi038 ceph-mon[92422]: pgmap v1730: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 69 KiB/s wr, 22 op/s 2023-03-09T15:36:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:36:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2280.969941 secs 2023-03-09T15:36:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:36:10 smithi122 ceph-mon[93869]: osdmap e322: 12 total, 12 up, 12 in 2023-03-09T15:36:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:36:10 smithi122 ceph-mon[93869]: pgmap v1732: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 62 KiB/s wr, 25 op/s 2023-03-09T15:36:10.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:36:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2280.969941 secs 2023-03-09T15:36:10.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:36:10 smithi181 ceph-mon[94189]: osdmap e322: 12 total, 12 up, 12 in 2023-03-09T15:36:10.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:36:10 smithi181 ceph-mon[94189]: pgmap v1732: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 62 KiB/s wr, 25 op/s 2023-03-09T15:36:11.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:36:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2280.969941 secs 2023-03-09T15:36:11.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:36:10 smithi038 ceph-mon[92422]: osdmap e322: 12 total, 12 up, 12 in 2023-03-09T15:36:11.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:36:10 smithi038 ceph-mon[92422]: pgmap v1732: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 62 KiB/s wr, 25 op/s 2023-03-09T15:36:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:36:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2283.171942 secs 2023-03-09T15:36:13.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:36:12 smithi122 ceph-mon[93869]: pgmap v1733: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 35 KiB/s wr, 22 op/s 2023-03-09T15:36:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:36:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2283.171942 secs 2023-03-09T15:36:13.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:36:12 smithi181 ceph-mon[94189]: pgmap v1733: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 35 KiB/s wr, 22 op/s 2023-03-09T15:36:13.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:36:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2283.171942 secs 2023-03-09T15:36:13.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:36:12 smithi038 ceph-mon[92422]: pgmap v1733: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 35 KiB/s wr, 22 op/s 2023-03-09T15:36:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2745.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:36:14.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:36:14 smithi122 ceph-mon[93869]: pgmap v1734: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 25 KiB/s wr, 8 op/s 2023-03-09T15:36:14.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:36:14 smithi181 ceph-mon[94189]: pgmap v1734: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 25 KiB/s wr, 8 op/s 2023-03-09T15:36:15.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:36:14 smithi038 ceph-mon[92422]: pgmap v1734: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 25 KiB/s wr, 8 op/s 2023-03-09T15:36:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:36:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2285.969987 secs 2023-03-09T15:36:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:36:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2285.969987 secs 2023-03-09T15:36:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:36:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2285.969987 secs 2023-03-09T15:36:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:36:16 smithi122 ceph-mon[93869]: pgmap v1735: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 26 KiB/s wr, 8 op/s 2023-03-09T15:36:16.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:36:16 smithi181 ceph-mon[94189]: pgmap v1735: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 26 KiB/s wr, 8 op/s 2023-03-09T15:36:17.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:36:16 smithi038 ceph-mon[92422]: pgmap v1735: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 26 KiB/s wr, 8 op/s 2023-03-09T15:36:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:36:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2288.172065 secs 2023-03-09T15:36:17.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:36:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2288.172065 secs 2023-03-09T15:36:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:36:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2288.172065 secs 2023-03-09T15:36:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2750.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:36:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:36:18 smithi122 ceph-mon[93869]: pgmap v1736: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 8 op/s 2023-03-09T15:36:18.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:36:18 smithi122 ceph-mon[93869]: osdmap e323: 12 total, 12 up, 12 in 2023-03-09T15:36:18.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:36:18 smithi181 ceph-mon[94189]: pgmap v1736: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 8 op/s 2023-03-09T15:36:18.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:36:18 smithi181 ceph-mon[94189]: osdmap e323: 12 total, 12 up, 12 in 2023-03-09T15:36:19.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:36:18 smithi038 ceph-mon[92422]: pgmap v1736: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 8 op/s 2023-03-09T15:36:19.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:36:18 smithi038 ceph-mon[92422]: osdmap e323: 12 total, 12 up, 12 in 2023-03-09T15:36:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:36:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2290.970048 secs 2023-03-09T15:36:21.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:36:20 smithi122 ceph-mon[93869]: pgmap v1738: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 8 op/s 2023-03-09T15:36:21.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:36:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2290.970048 secs 2023-03-09T15:36:21.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:36:20 smithi181 ceph-mon[94189]: pgmap v1738: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 8 op/s 2023-03-09T15:36:21.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:36:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2290.970048 secs 2023-03-09T15:36:21.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:36:20 smithi038 ceph-mon[92422]: pgmap v1738: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 8 op/s 2023-03-09T15:36:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:36:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2293.172163 secs 2023-03-09T15:36:23.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:36:22 smithi122 ceph-mon[93869]: pgmap v1739: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 8 op/s 2023-03-09T15:36:23.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:36:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2293.172163 secs 2023-03-09T15:36:23.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:36:22 smithi181 ceph-mon[94189]: pgmap v1739: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 8 op/s 2023-03-09T15:36:23.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:36:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2293.172163 secs 2023-03-09T15:36:23.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:36:22 smithi038 ceph-mon[92422]: pgmap v1739: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 8 op/s 2023-03-09T15:36:23.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2755.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:36:24.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:36:24 smithi122 ceph-mon[93869]: pgmap v1740: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 20 KiB/s wr, 8 op/s 2023-03-09T15:36:24.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:36:24 smithi181 ceph-mon[94189]: pgmap v1740: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 20 KiB/s wr, 8 op/s 2023-03-09T15:36:25.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:36:24 smithi038 ceph-mon[92422]: pgmap v1740: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 20 KiB/s wr, 8 op/s 2023-03-09T15:36:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:36:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2295.970108 secs 2023-03-09T15:36:25.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:36:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2295.970108 secs 2023-03-09T15:36:26.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:36:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2295.970108 secs 2023-03-09T15:36:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:36:26 smithi122 ceph-mon[93869]: pgmap v1741: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T15:36:26.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:36:26 smithi181 ceph-mon[94189]: pgmap v1741: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T15:36:27.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:36:26 smithi038 ceph-mon[92422]: pgmap v1741: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T15:36:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:36:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2298.172269 secs 2023-03-09T15:36:27.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:36:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2298.172269 secs 2023-03-09T15:36:28.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:36:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2298.172269 secs 2023-03-09T15:36:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2760.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:36:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:36:28 smithi122 ceph-mon[93869]: pgmap v1742: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 39 KiB/s wr, 7 op/s 2023-03-09T15:36:28.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:36:28 smithi181 ceph-mon[94189]: pgmap v1742: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 39 KiB/s wr, 7 op/s 2023-03-09T15:36:29.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:36:28 smithi038 ceph-mon[92422]: pgmap v1742: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 39 KiB/s wr, 7 op/s 2023-03-09T15:36:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:36:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2300.970167 secs 2023-03-09T15:36:31.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:36:30 smithi122 ceph-mon[93869]: pgmap v1743: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 35 KiB/s wr, 6 op/s 2023-03-09T15:36:31.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:36:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2300.970167 secs 2023-03-09T15:36:31.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:36:30 smithi181 ceph-mon[94189]: pgmap v1743: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 35 KiB/s wr, 6 op/s 2023-03-09T15:36:31.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:36:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2300.970167 secs 2023-03-09T15:36:31.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:36:30 smithi038 ceph-mon[92422]: pgmap v1743: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 35 KiB/s wr, 6 op/s 2023-03-09T15:36:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:36:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2303.172388 secs 2023-03-09T15:36:33.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:36:32 smithi122 ceph-mon[93869]: pgmap v1744: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 33 KiB/s wr, 7 op/s 2023-03-09T15:36:33.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:36:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2303.172388 secs 2023-03-09T15:36:33.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:36:32 smithi181 ceph-mon[94189]: pgmap v1744: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 33 KiB/s wr, 7 op/s 2023-03-09T15:36:33.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:36:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2303.172388 secs 2023-03-09T15:36:33.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:36:32 smithi038 ceph-mon[92422]: pgmap v1744: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 33 KiB/s wr, 7 op/s 2023-03-09T15:36:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2765.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:36:34.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:36:34 smithi122 ceph-mon[93869]: pgmap v1745: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 38 KiB/s wr, 7 op/s 2023-03-09T15:36:34.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:36:34 smithi181 ceph-mon[94189]: pgmap v1745: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 38 KiB/s wr, 7 op/s 2023-03-09T15:36:35.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:36:34 smithi038 ceph-mon[92422]: pgmap v1745: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 38 KiB/s wr, 7 op/s 2023-03-09T15:36:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:36:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2305.970230 secs 2023-03-09T15:36:35.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:36:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2305.970230 secs 2023-03-09T15:36:36.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:36:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2305.970230 secs 2023-03-09T15:36:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:36:36 smithi122 ceph-mon[93869]: pgmap v1746: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 22 KiB/s wr, 19 op/s 2023-03-09T15:36:36.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:36:36 smithi181 ceph-mon[94189]: pgmap v1746: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 22 KiB/s wr, 19 op/s 2023-03-09T15:36:37.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:36:36 smithi038 ceph-mon[92422]: pgmap v1746: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 22 KiB/s wr, 19 op/s 2023-03-09T15:36:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2770.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:36:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:36:38 smithi122 ceph-mon[93869]: pgmap v1747: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 74 KiB/s wr, 20 op/s 2023-03-09T15:36:39.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:36:38 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:36:39.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:36:38 smithi181 ceph-mon[94189]: pgmap v1747: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 74 KiB/s wr, 20 op/s 2023-03-09T15:36:39.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:36:38 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:36:39.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:36:38 smithi038 ceph-mon[92422]: pgmap v1747: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 74 KiB/s wr, 20 op/s 2023-03-09T15:36:39.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:36:38 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:36:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:36:39 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:36:40.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:36:39 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:36:40.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:36:39 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:36:40.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:36:39 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:36:40.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:36:39 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:36:40.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:36:39 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:36:40.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:36:39 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:36:40.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:36:39 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:36:40.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:36:39 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:36:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:36:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2310.970311 secs 2023-03-09T15:36:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:36:40 smithi122 ceph-mon[93869]: pgmap v1748: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 57 KiB/s wr, 19 op/s 2023-03-09T15:36:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:36:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2310.970311 secs 2023-03-09T15:36:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:36:40 smithi181 ceph-mon[94189]: pgmap v1748: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 57 KiB/s wr, 19 op/s 2023-03-09T15:36:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:36:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2310.970311 secs 2023-03-09T15:36:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:36:40 smithi038 ceph-mon[92422]: pgmap v1748: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 57 KiB/s wr, 19 op/s 2023-03-09T15:36:43.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:36:42 smithi122 ceph-mon[93869]: pgmap v1749: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 57 KiB/s wr, 46 op/s 2023-03-09T15:36:43.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:36:42 smithi181 ceph-mon[94189]: pgmap v1749: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 57 KiB/s wr, 46 op/s 2023-03-09T15:36:43.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:36:42 smithi038 ceph-mon[92422]: pgmap v1749: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 57 KiB/s wr, 46 op/s 2023-03-09T15:36:43.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2775.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:36:44.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:36:44 smithi122 ceph-mon[93869]: pgmap v1750: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 109 KiB/s wr, 47 op/s 2023-03-09T15:36:44.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:36:44 smithi181 ceph-mon[94189]: pgmap v1750: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 109 KiB/s wr, 47 op/s 2023-03-09T15:36:45.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:36:44 smithi038 ceph-mon[92422]: pgmap v1750: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 109 KiB/s wr, 47 op/s 2023-03-09T15:36:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:36:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2315.970371 secs 2023-03-09T15:36:45.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:36:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2315.970371 secs 2023-03-09T15:36:46.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:36:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2315.970371 secs 2023-03-09T15:36:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:36:46 smithi122 ceph-mon[93869]: pgmap v1751: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 104 KiB/s wr, 47 op/s 2023-03-09T15:36:46.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:36:46 smithi181 ceph-mon[94189]: pgmap v1751: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 104 KiB/s wr, 47 op/s 2023-03-09T15:36:47.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:36:46 smithi038 ceph-mon[92422]: pgmap v1751: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 104 KiB/s wr, 47 op/s 2023-03-09T15:36:48.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2780.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:36:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:36:48 smithi122 ceph-mon[93869]: pgmap v1752: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 124 KiB/s wr, 36 op/s 2023-03-09T15:36:49.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:36:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:36:49.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:36:48 smithi181 ceph-mon[94189]: pgmap v1752: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 124 KiB/s wr, 36 op/s 2023-03-09T15:36:49.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:36:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:36:49.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:36:48 smithi038 ceph-mon[92422]: pgmap v1752: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 124 KiB/s wr, 36 op/s 2023-03-09T15:36:49.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:36:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:36:50.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:36:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:36:50.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:36:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:36:50.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:36:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:36:50.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:36:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:36:50.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:36:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:36:50.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:36:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:36:50.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:36:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:36:50.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:36:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:36:50.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:36:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:36:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:36:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2320.970431 secs 2023-03-09T15:36:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:36:50 smithi122 ceph-mon[93869]: pgmap v1753: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 73 KiB/s wr, 34 op/s 2023-03-09T15:36:51.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:36:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2320.970431 secs 2023-03-09T15:36:51.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:36:50 smithi181 ceph-mon[94189]: pgmap v1753: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 73 KiB/s wr, 34 op/s 2023-03-09T15:36:51.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:36:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2320.970431 secs 2023-03-09T15:36:51.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:36:50 smithi038 ceph-mon[92422]: pgmap v1753: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 73 KiB/s wr, 34 op/s 2023-03-09T15:36:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:36:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2323.172840 secs 2023-03-09T15:36:53.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:36:52 smithi122 ceph-mon[93869]: pgmap v1754: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 73 KiB/s wr, 35 op/s 2023-03-09T15:36:53.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:36:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2323.172840 secs 2023-03-09T15:36:53.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:36:52 smithi181 ceph-mon[94189]: pgmap v1754: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 73 KiB/s wr, 35 op/s 2023-03-09T15:36:53.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:36:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2323.172840 secs 2023-03-09T15:36:53.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:36:52 smithi038 ceph-mon[92422]: pgmap v1754: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 73 KiB/s wr, 35 op/s 2023-03-09T15:36:53.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2785.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:36:54.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:36:54 smithi122 ceph-mon[93869]: pgmap v1755: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 112 KiB/s wr, 11 op/s 2023-03-09T15:36:54.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:36:54 smithi181 ceph-mon[94189]: pgmap v1755: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 112 KiB/s wr, 11 op/s 2023-03-09T15:36:55.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:36:54 smithi038 ceph-mon[92422]: pgmap v1755: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 112 KiB/s wr, 11 op/s 2023-03-09T15:36:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:36:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2325.970492 secs 2023-03-09T15:36:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:36:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2325.970492 secs 2023-03-09T15:36:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:36:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2325.970492 secs 2023-03-09T15:36:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:36:56 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:36:56.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:36:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:36:56.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:36:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:36:56.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:36:56 smithi122 ceph-mon[93869]: pgmap v1756: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 61 KiB/s wr, 10 op/s 2023-03-09T15:36:56.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:36:56 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:36:56.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:36:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:36:56.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:36:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:36:56.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:36:56 smithi181 ceph-mon[94189]: pgmap v1756: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 61 KiB/s wr, 10 op/s 2023-03-09T15:36:57.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:36:56 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:36:57.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:36:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:36:57.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:36:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:36:57.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:36:56 smithi038 ceph-mon[92422]: pgmap v1756: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 61 KiB/s wr, 10 op/s 2023-03-09T15:36:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:36:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2328.172952 secs 2023-03-09T15:36:57.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:36:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2328.172952 secs 2023-03-09T15:36:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:36:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2328.172952 secs 2023-03-09T15:36:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2790.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:36:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:36:58 smithi122 ceph-mon[93869]: pgmap v1757: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 101 KiB/s wr, 10 op/s 2023-03-09T15:36:58.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:36:58 smithi181 ceph-mon[94189]: pgmap v1757: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 101 KiB/s wr, 10 op/s 2023-03-09T15:36:59.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:36:58 smithi038 ceph-mon[92422]: pgmap v1757: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 101 KiB/s wr, 10 op/s 2023-03-09T15:37:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2330.970550 secs 2023-03-09T15:37:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:00 smithi122 ceph-mon[93869]: pgmap v1758: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 81 KiB/s wr, 8 op/s 2023-03-09T15:37:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2330.970550 secs 2023-03-09T15:37:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:00 smithi181 ceph-mon[94189]: pgmap v1758: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 81 KiB/s wr, 8 op/s 2023-03-09T15:37:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2330.970550 secs 2023-03-09T15:37:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:00 smithi038 ceph-mon[92422]: pgmap v1758: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 81 KiB/s wr, 8 op/s 2023-03-09T15:37:03.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2333.173068 secs 2023-03-09T15:37:03.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:02 smithi122 ceph-mon[93869]: pgmap v1759: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 107 KiB/s wr, 13 op/s 2023-03-09T15:37:03.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2333.173068 secs 2023-03-09T15:37:03.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:02 smithi181 ceph-mon[94189]: pgmap v1759: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 107 KiB/s wr, 13 op/s 2023-03-09T15:37:03.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2333.173068 secs 2023-03-09T15:37:03.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:02 smithi038 ceph-mon[92422]: pgmap v1759: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 107 KiB/s wr, 13 op/s 2023-03-09T15:37:03.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2795.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:37:04.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:04 smithi122 ceph-mon[93869]: pgmap v1760: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 146 KiB/s wr, 25 op/s 2023-03-09T15:37:04.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:04 smithi181 ceph-mon[94189]: pgmap v1760: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 146 KiB/s wr, 25 op/s 2023-03-09T15:37:05.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:04 smithi038 ceph-mon[92422]: pgmap v1760: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 146 KiB/s wr, 25 op/s 2023-03-09T15:37:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2335.970610 secs 2023-03-09T15:37:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2335.970610 secs 2023-03-09T15:37:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2335.970610 secs 2023-03-09T15:37:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:06 smithi122 ceph-mon[93869]: pgmap v1761: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 106 KiB/s wr, 24 op/s 2023-03-09T15:37:06.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:06 smithi181 ceph-mon[94189]: pgmap v1761: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 106 KiB/s wr, 24 op/s 2023-03-09T15:37:07.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:06 smithi038 ceph-mon[92422]: pgmap v1761: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 106 KiB/s wr, 24 op/s 2023-03-09T15:37:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2338.173195 secs 2023-03-09T15:37:07.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2338.173195 secs 2023-03-09T15:37:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2338.173195 secs 2023-03-09T15:37:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2800.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:37:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:08 smithi122 ceph-mon[93869]: pgmap v1762: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 114 KiB/s wr, 24 op/s 2023-03-09T15:37:08.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:08 smithi181 ceph-mon[94189]: pgmap v1762: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 114 KiB/s wr, 24 op/s 2023-03-09T15:37:09.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:08 smithi038 ceph-mon[92422]: pgmap v1762: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 114 KiB/s wr, 24 op/s 2023-03-09T15:37:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2340.970670 secs 2023-03-09T15:37:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:10 smithi122 ceph-mon[93869]: osdmap e324: 12 total, 12 up, 12 in 2023-03-09T15:37:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:10 smithi122 ceph-mon[93869]: pgmap v1764: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 87 KiB/s wr, 26 op/s 2023-03-09T15:37:10.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2340.970670 secs 2023-03-09T15:37:10.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:10 smithi181 ceph-mon[94189]: osdmap e324: 12 total, 12 up, 12 in 2023-03-09T15:37:10.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:10 smithi181 ceph-mon[94189]: pgmap v1764: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 87 KiB/s wr, 26 op/s 2023-03-09T15:37:11.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2340.970670 secs 2023-03-09T15:37:11.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:10 smithi038 ceph-mon[92422]: osdmap e324: 12 total, 12 up, 12 in 2023-03-09T15:37:11.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:10 smithi038 ceph-mon[92422]: pgmap v1764: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 87 KiB/s wr, 26 op/s 2023-03-09T15:37:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2343.173318 secs 2023-03-09T15:37:13.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:12 smithi122 ceph-mon[93869]: pgmap v1765: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 57 KiB/s wr, 23 op/s 2023-03-09T15:37:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2343.173318 secs 2023-03-09T15:37:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:12 smithi181 ceph-mon[94189]: pgmap v1765: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 57 KiB/s wr, 23 op/s 2023-03-09T15:37:13.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2343.173318 secs 2023-03-09T15:37:13.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:12 smithi038 ceph-mon[92422]: pgmap v1765: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 57 KiB/s wr, 23 op/s 2023-03-09T15:37:13.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2805.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:37:14.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:14 smithi122 ceph-mon[93869]: pgmap v1766: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 11 KiB/s wr, 8 op/s 2023-03-09T15:37:14.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:14 smithi181 ceph-mon[94189]: pgmap v1766: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 11 KiB/s wr, 8 op/s 2023-03-09T15:37:15.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:14 smithi038 ceph-mon[92422]: pgmap v1766: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 11 KiB/s wr, 8 op/s 2023-03-09T15:37:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2345.970736 secs 2023-03-09T15:37:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2345.970736 secs 2023-03-09T15:37:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2345.970736 secs 2023-03-09T15:37:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:16 smithi122 ceph-mon[93869]: pgmap v1767: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 11 KiB/s wr, 8 op/s 2023-03-09T15:37:16.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:16 smithi181 ceph-mon[94189]: pgmap v1767: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 11 KiB/s wr, 8 op/s 2023-03-09T15:37:17.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:16 smithi038 ceph-mon[92422]: pgmap v1767: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 11 KiB/s wr, 8 op/s 2023-03-09T15:37:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2348.173434 secs 2023-03-09T15:37:17.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2348.173434 secs 2023-03-09T15:37:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2348.173434 secs 2023-03-09T15:37:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2810.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:37:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:18 smithi122 ceph-mon[93869]: pgmap v1768: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 8 op/s 2023-03-09T15:37:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:18 smithi122 ceph-mon[93869]: osdmap e325: 12 total, 12 up, 12 in 2023-03-09T15:37:18.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:18 smithi181 ceph-mon[94189]: pgmap v1768: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 8 op/s 2023-03-09T15:37:18.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:18 smithi181 ceph-mon[94189]: osdmap e325: 12 total, 12 up, 12 in 2023-03-09T15:37:19.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:18 smithi038 ceph-mon[92422]: pgmap v1768: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 8 op/s 2023-03-09T15:37:19.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:18 smithi038 ceph-mon[92422]: osdmap e325: 12 total, 12 up, 12 in 2023-03-09T15:37:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2350.970797 secs 2023-03-09T15:37:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:20 smithi122 ceph-mon[93869]: pgmap v1770: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 8 op/s 2023-03-09T15:37:21.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2350.970797 secs 2023-03-09T15:37:21.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:20 smithi181 ceph-mon[94189]: pgmap v1770: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 8 op/s 2023-03-09T15:37:21.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2350.970797 secs 2023-03-09T15:37:21.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:20 smithi038 ceph-mon[92422]: pgmap v1770: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 8 op/s 2023-03-09T15:37:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2353.173546 secs 2023-03-09T15:37:23.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:22 smithi122 ceph-mon[93869]: pgmap v1771: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 23 KiB/s wr, 8 op/s 2023-03-09T15:37:23.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2353.173546 secs 2023-03-09T15:37:23.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:22 smithi181 ceph-mon[94189]: pgmap v1771: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 23 KiB/s wr, 8 op/s 2023-03-09T15:37:23.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2353.173546 secs 2023-03-09T15:37:23.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:22 smithi038 ceph-mon[92422]: pgmap v1771: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 23 KiB/s wr, 8 op/s 2023-03-09T15:37:23.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2815.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:37:24.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:24 smithi122 ceph-mon[93869]: pgmap v1772: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 8 op/s 2023-03-09T15:37:24.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:24 smithi181 ceph-mon[94189]: pgmap v1772: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 8 op/s 2023-03-09T15:37:25.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:24 smithi038 ceph-mon[92422]: pgmap v1772: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 8 op/s 2023-03-09T15:37:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2355.970857 secs 2023-03-09T15:37:25.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2355.970857 secs 2023-03-09T15:37:26.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2355.970857 secs 2023-03-09T15:37:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:26 smithi122 ceph-mon[93869]: pgmap v1773: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 8 op/s 2023-03-09T15:37:26.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:26 smithi181 ceph-mon[94189]: pgmap v1773: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 8 op/s 2023-03-09T15:37:27.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:26 smithi038 ceph-mon[92422]: pgmap v1773: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 8 op/s 2023-03-09T15:37:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2358.173687 secs 2023-03-09T15:37:27.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2358.173687 secs 2023-03-09T15:37:28.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2358.173687 secs 2023-03-09T15:37:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2820.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:37:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:28 smithi122 ceph-mon[93869]: pgmap v1774: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T15:37:28.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:28 smithi181 ceph-mon[94189]: pgmap v1774: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T15:37:29.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:28 smithi038 ceph-mon[92422]: pgmap v1774: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T15:37:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2360.970915 secs 2023-03-09T15:37:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:30 smithi122 ceph-mon[93869]: pgmap v1775: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 6 op/s 2023-03-09T15:37:31.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2360.970915 secs 2023-03-09T15:37:31.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:30 smithi181 ceph-mon[94189]: pgmap v1775: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 6 op/s 2023-03-09T15:37:31.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2360.970915 secs 2023-03-09T15:37:31.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:30 smithi038 ceph-mon[92422]: pgmap v1775: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 6 op/s 2023-03-09T15:37:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:32 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 2363.173793 secs 2023-03-09T15:37:33.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:32 smithi122 ceph-mon[93869]: pgmap v1776: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 9 op/s 2023-03-09T15:37:33.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:32 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 2363.173793 secs 2023-03-09T15:37:33.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:32 smithi181 ceph-mon[94189]: pgmap v1776: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 9 op/s 2023-03-09T15:37:33.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:32 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 2363.173793 secs 2023-03-09T15:37:33.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:32 smithi038 ceph-mon[92422]: pgmap v1776: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 20 KiB/s wr, 9 op/s 2023-03-09T15:37:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2825.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:37:34.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:34 smithi122 ceph-mon[93869]: pgmap v1777: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 76 KiB/s wr, 7 op/s 2023-03-09T15:37:34.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:34 smithi181 ceph-mon[94189]: pgmap v1777: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 76 KiB/s wr, 7 op/s 2023-03-09T15:37:35.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:34 smithi038 ceph-mon[92422]: pgmap v1777: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 76 KiB/s wr, 7 op/s 2023-03-09T15:37:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2365.970979 secs 2023-03-09T15:37:35.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2365.970979 secs 2023-03-09T15:37:36.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2365.970979 secs 2023-03-09T15:37:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:37:36.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:37:36.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:36 smithi122 ceph-mon[93869]: pgmap v1778: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 76 KiB/s wr, 9 op/s 2023-03-09T15:37:36.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:37:36.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:37:36.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:36 smithi181 ceph-mon[94189]: pgmap v1778: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 76 KiB/s wr, 9 op/s 2023-03-09T15:37:37.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:37:37.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:37:37.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:36 smithi038 ceph-mon[92422]: pgmap v1778: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 76 KiB/s wr, 9 op/s 2023-03-09T15:37:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2368.173915 secs 2023-03-09T15:37:37.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2368.173915 secs 2023-03-09T15:37:38.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2368.173915 secs 2023-03-09T15:37:38.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2830.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:37:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:38 smithi122 ceph-mon[93869]: pgmap v1779: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 99 KiB/s wr, 10 op/s 2023-03-09T15:37:38.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:38 smithi181 ceph-mon[94189]: pgmap v1779: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 99 KiB/s wr, 10 op/s 2023-03-09T15:37:39.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:38 smithi038 ceph-mon[92422]: pgmap v1779: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 99 KiB/s wr, 10 op/s 2023-03-09T15:37:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2370.971038 secs 2023-03-09T15:37:40.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:37:40.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:37:40.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:40 smithi122 ceph-mon[93869]: pgmap v1780: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 86 KiB/s wr, 9 op/s 2023-03-09T15:37:40.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2370.971038 secs 2023-03-09T15:37:40.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:37:40.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:37:40.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:40 smithi181 ceph-mon[94189]: pgmap v1780: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 86 KiB/s wr, 9 op/s 2023-03-09T15:37:41.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2370.971038 secs 2023-03-09T15:37:41.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:37:41.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:37:41.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:40 smithi038 ceph-mon[92422]: pgmap v1780: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 86 KiB/s wr, 9 op/s 2023-03-09T15:37:43.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2373.174036 secs 2023-03-09T15:37:43.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:42 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:37:43.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:42 smithi122 ceph-mon[93869]: pgmap v1781: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 99 KiB/s wr, 13 op/s 2023-03-09T15:37:43.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2373.174036 secs 2023-03-09T15:37:43.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:42 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:37:43.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:42 smithi181 ceph-mon[94189]: pgmap v1781: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 99 KiB/s wr, 13 op/s 2023-03-09T15:37:43.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2373.174036 secs 2023-03-09T15:37:43.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:42 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:37:43.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:42 smithi038 ceph-mon[92422]: pgmap v1781: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 99 KiB/s wr, 13 op/s 2023-03-09T15:37:43.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2835.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:37:44.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:44 smithi122 ceph-mon[93869]: pgmap v1782: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 169 KiB/s wr, 13 op/s 2023-03-09T15:37:44.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:44 smithi181 ceph-mon[94189]: pgmap v1782: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 169 KiB/s wr, 13 op/s 2023-03-09T15:37:45.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:44 smithi038 ceph-mon[92422]: pgmap v1782: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 169 KiB/s wr, 13 op/s 2023-03-09T15:37:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2375.971101 secs 2023-03-09T15:37:45.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2375.971101 secs 2023-03-09T15:37:46.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2375.971101 secs 2023-03-09T15:37:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:46 smithi122 ceph-mon[93869]: pgmap v1783: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 113 KiB/s wr, 14 op/s 2023-03-09T15:37:46.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:46 smithi181 ceph-mon[94189]: pgmap v1783: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 113 KiB/s wr, 14 op/s 2023-03-09T15:37:47.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:46 smithi038 ceph-mon[92422]: pgmap v1783: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 113 KiB/s wr, 14 op/s 2023-03-09T15:37:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2378.174159 secs 2023-03-09T15:37:47.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2378.174159 secs 2023-03-09T15:37:48.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2378.174159 secs 2023-03-09T15:37:48.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2840.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:37:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:48 smithi122 ceph-mon[93869]: pgmap v1784: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 131 KiB/s wr, 13 op/s 2023-03-09T15:37:48.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:48 smithi181 ceph-mon[94189]: pgmap v1784: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 131 KiB/s wr, 13 op/s 2023-03-09T15:37:49.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:48 smithi038 ceph-mon[92422]: pgmap v1784: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 131 KiB/s wr, 13 op/s 2023-03-09T15:37:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:37:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:37:50.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:37:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2380.971163 secs 2023-03-09T15:37:50.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:50 smithi122 ceph-mon[93869]: pgmap v1785: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 107 KiB/s wr, 11 op/s 2023-03-09T15:37:50.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2380.971163 secs 2023-03-09T15:37:50.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:50 smithi181 ceph-mon[94189]: pgmap v1785: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 107 KiB/s wr, 11 op/s 2023-03-09T15:37:51.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2380.971163 secs 2023-03-09T15:37:51.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:50 smithi038 ceph-mon[92422]: pgmap v1785: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 107 KiB/s wr, 11 op/s 2023-03-09T15:37:53.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2383.174271 secs 2023-03-09T15:37:53.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:52 smithi038 ceph-mon[92422]: pgmap v1786: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 113 KiB/s wr, 14 op/s 2023-03-09T15:37:53.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:37:53.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:37:53.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:37:53.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:37:53.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:37:53.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:37:53.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:37:53.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:37:53.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:37:53.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:37:53.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:37:53.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:37:53.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:37:53.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:37:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2383.174271 secs 2023-03-09T15:37:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:52 smithi122 ceph-mon[93869]: pgmap v1786: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 113 KiB/s wr, 14 op/s 2023-03-09T15:37:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:37:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:37:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:37:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:37:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:37:53.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:37:53.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:37:53.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:37:53.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:37:53.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:37:53.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:37:53.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:37:53.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:37:53.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:37:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2383.174271 secs 2023-03-09T15:37:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:52 smithi181 ceph-mon[94189]: pgmap v1786: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 113 KiB/s wr, 14 op/s 2023-03-09T15:37:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:37:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:37:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:37:53.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:37:53.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:37:53.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:37:53.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:37:53.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:37:53.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:37:53.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:37:53.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:37:53.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:37:53.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:37:53.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:37:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2845.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:37:54.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:53 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T15:37:54.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:53 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T15:37:54.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:53 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi038 to 197.2M 2023-03-09T15:37:54.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:53 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi038 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-03-09T15:37:54.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:37:54.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:37:54.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:37:54.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:37:54.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:37:54.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:37:54.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:37:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:53 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T15:37:54.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:53 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T15:37:54.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:53 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi038 to 197.2M 2023-03-09T15:37:54.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:53 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi038 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-03-09T15:37:54.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:37:54.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:37:54.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:37:54.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:37:54.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:37:54.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:37:54.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:37:54.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:53 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T15:37:54.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:53 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T15:37:54.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:53 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi038 to 197.2M 2023-03-09T15:37:54.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:53 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi038 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-03-09T15:37:54.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:37:54.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:37:54.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:37:54.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:37:54.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:37:54.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:37:54.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:37:55.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:54 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T15:37:55.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:54 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T15:37:55.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:54 smithi038 ceph-mon[92422]: pgmap v1787: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 121 KiB/s wr, 11 op/s 2023-03-09T15:37:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:54 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T15:37:55.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:54 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T15:37:55.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:54 smithi122 ceph-mon[93869]: pgmap v1787: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 121 KiB/s wr, 11 op/s 2023-03-09T15:37:55.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:54 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T15:37:55.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:54 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T15:37:55.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:54 smithi181 ceph-mon[94189]: pgmap v1787: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 121 KiB/s wr, 11 op/s 2023-03-09T15:37:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2385.971246 secs 2023-03-09T15:37:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:56 smithi122 ceph-mon[93869]: pgmap v1788: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 49 KiB/s wr, 11 op/s 2023-03-09T15:37:56.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2385.971246 secs 2023-03-09T15:37:56.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:56 smithi181 ceph-mon[94189]: pgmap v1788: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 49 KiB/s wr, 11 op/s 2023-03-09T15:37:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2385.971246 secs 2023-03-09T15:37:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:56 smithi038 ceph-mon[92422]: pgmap v1788: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 49 KiB/s wr, 11 op/s 2023-03-09T15:37:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2388.174418 secs 2023-03-09T15:37:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2388.174418 secs 2023-03-09T15:37:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2388.174418 secs 2023-03-09T15:37:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:37:58 smithi122 ceph-mon[93869]: pgmap v1789: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 64 KiB/s wr, 9 op/s 2023-03-09T15:37:58.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:37:58 smithi181 ceph-mon[94189]: pgmap v1789: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 64 KiB/s wr, 9 op/s 2023-03-09T15:37:58.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:37:58 smithi038 ceph-mon[92422]: pgmap v1789: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 64 KiB/s wr, 9 op/s 2023-03-09T15:37:58.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2850.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:38:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:38:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2390.971339 secs 2023-03-09T15:38:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:38:00 smithi122 ceph-mon[93869]: pgmap v1790: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 47 KiB/s wr, 8 op/s 2023-03-09T15:38:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:38:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2390.971339 secs 2023-03-09T15:38:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:38:00 smithi181 ceph-mon[94189]: pgmap v1790: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 47 KiB/s wr, 8 op/s 2023-03-09T15:38:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:38:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2390.971339 secs 2023-03-09T15:38:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:38:00 smithi038 ceph-mon[92422]: pgmap v1790: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 47 KiB/s wr, 8 op/s 2023-03-09T15:38:03.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:38:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2393.174571 secs 2023-03-09T15:38:03.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:38:02 smithi122 ceph-mon[93869]: pgmap v1791: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 69 KiB/s wr, 12 op/s 2023-03-09T15:38:03.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:38:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2393.174571 secs 2023-03-09T15:38:03.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:38:02 smithi181 ceph-mon[94189]: pgmap v1791: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 69 KiB/s wr, 12 op/s 2023-03-09T15:38:03.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:38:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2393.174571 secs 2023-03-09T15:38:03.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:38:02 smithi038 ceph-mon[92422]: pgmap v1791: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 69 KiB/s wr, 12 op/s 2023-03-09T15:38:03.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2855.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:38:04.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:38:04 smithi122 ceph-mon[93869]: pgmap v1792: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 96 KiB/s wr, 21 op/s 2023-03-09T15:38:04.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:38:04 smithi181 ceph-mon[94189]: pgmap v1792: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 96 KiB/s wr, 21 op/s 2023-03-09T15:38:05.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:38:04 smithi038 ceph-mon[92422]: pgmap v1792: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 96 KiB/s wr, 21 op/s 2023-03-09T15:38:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:38:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2395.971403 secs 2023-03-09T15:38:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:38:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2395.971403 secs 2023-03-09T15:38:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:38:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2395.971403 secs 2023-03-09T15:38:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:38:06 smithi122 ceph-mon[93869]: pgmap v1793: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 73 KiB/s wr, 22 op/s 2023-03-09T15:38:06.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:38:06 smithi181 ceph-mon[94189]: pgmap v1793: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 73 KiB/s wr, 22 op/s 2023-03-09T15:38:07.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:38:06 smithi038 ceph-mon[92422]: pgmap v1793: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 73 KiB/s wr, 22 op/s 2023-03-09T15:38:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:38:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2398.174717 secs 2023-03-09T15:38:07.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:38:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2398.174717 secs 2023-03-09T15:38:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:38:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2398.174717 secs 2023-03-09T15:38:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2860.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:38:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:38:08 smithi122 ceph-mon[93869]: pgmap v1794: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 73 KiB/s wr, 21 op/s 2023-03-09T15:38:08.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:38:08 smithi181 ceph-mon[94189]: pgmap v1794: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 73 KiB/s wr, 21 op/s 2023-03-09T15:38:09.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:38:08 smithi038 ceph-mon[92422]: pgmap v1794: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 73 KiB/s wr, 21 op/s 2023-03-09T15:38:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:38:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2400.971482 secs 2023-03-09T15:38:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:38:10 smithi122 ceph-mon[93869]: osdmap e326: 12 total, 12 up, 12 in 2023-03-09T15:38:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:38:10 smithi122 ceph-mon[93869]: pgmap v1796: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 68 KiB/s wr, 24 op/s 2023-03-09T15:38:10.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:38:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2400.971482 secs 2023-03-09T15:38:10.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:38:10 smithi181 ceph-mon[94189]: osdmap e326: 12 total, 12 up, 12 in 2023-03-09T15:38:10.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:38:10 smithi181 ceph-mon[94189]: pgmap v1796: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 68 KiB/s wr, 24 op/s 2023-03-09T15:38:11.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:38:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2400.971482 secs 2023-03-09T15:38:11.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:38:10 smithi038 ceph-mon[92422]: osdmap e326: 12 total, 12 up, 12 in 2023-03-09T15:38:11.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:38:10 smithi038 ceph-mon[92422]: pgmap v1796: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 68 KiB/s wr, 24 op/s 2023-03-09T15:38:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:38:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2403.174841 secs 2023-03-09T15:38:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:38:12 smithi122 ceph-mon[93869]: pgmap v1797: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 41 KiB/s wr, 21 op/s 2023-03-09T15:38:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:38:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2403.174841 secs 2023-03-09T15:38:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:38:12 smithi181 ceph-mon[94189]: pgmap v1797: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 41 KiB/s wr, 21 op/s 2023-03-09T15:38:13.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:38:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2403.174841 secs 2023-03-09T15:38:13.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:38:12 smithi038 ceph-mon[92422]: pgmap v1797: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 41 KiB/s wr, 21 op/s 2023-03-09T15:38:13.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2865.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:38:14.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:38:14 smithi122 ceph-mon[93869]: pgmap v1798: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2023-03-09T15:38:14.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:38:14 smithi181 ceph-mon[94189]: pgmap v1798: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2023-03-09T15:38:15.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:38:14 smithi038 ceph-mon[92422]: pgmap v1798: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2023-03-09T15:38:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:38:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2405.971542 secs 2023-03-09T15:38:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:38:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2405.971542 secs 2023-03-09T15:38:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:38:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2405.971542 secs 2023-03-09T15:38:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:38:16 smithi122 ceph-mon[93869]: pgmap v1799: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2023-03-09T15:38:16.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:38:16 smithi181 ceph-mon[94189]: pgmap v1799: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2023-03-09T15:38:17.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:38:16 smithi038 ceph-mon[92422]: pgmap v1799: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2023-03-09T15:38:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:38:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2408.174968 secs 2023-03-09T15:38:17.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:38:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2408.174968 secs 2023-03-09T15:38:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:38:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2408.174968 secs 2023-03-09T15:38:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2870.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:38:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:38:18 smithi122 ceph-mon[93869]: pgmap v1800: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2023-03-09T15:38:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:38:18 smithi122 ceph-mon[93869]: osdmap e327: 12 total, 12 up, 12 in 2023-03-09T15:38:18.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:38:18 smithi181 ceph-mon[94189]: pgmap v1800: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2023-03-09T15:38:18.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:38:18 smithi181 ceph-mon[94189]: osdmap e327: 12 total, 12 up, 12 in 2023-03-09T15:38:19.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:38:18 smithi038 ceph-mon[92422]: pgmap v1800: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2023-03-09T15:38:19.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:38:18 smithi038 ceph-mon[92422]: osdmap e327: 12 total, 12 up, 12 in 2023-03-09T15:38:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:38:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2410.971621 secs 2023-03-09T15:38:21.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:38:20 smithi122 ceph-mon[93869]: pgmap v1802: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 7 op/s 2023-03-09T15:38:21.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:38:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2410.971621 secs 2023-03-09T15:38:21.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:38:20 smithi181 ceph-mon[94189]: pgmap v1802: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 7 op/s 2023-03-09T15:38:21.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:38:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2410.971621 secs 2023-03-09T15:38:21.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:38:20 smithi038 ceph-mon[92422]: pgmap v1802: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 7 op/s 2023-03-09T15:38:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:38:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2413.175102 secs 2023-03-09T15:38:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:38:22 smithi122 ceph-mon[93869]: pgmap v1803: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2023-03-09T15:38:23.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:38:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2413.175102 secs 2023-03-09T15:38:23.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:38:22 smithi181 ceph-mon[94189]: pgmap v1803: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2023-03-09T15:38:23.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:38:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2413.175102 secs 2023-03-09T15:38:23.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:38:22 smithi038 ceph-mon[92422]: pgmap v1803: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2023-03-09T15:38:23.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2875.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:38:24.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:38:24 smithi122 ceph-mon[93869]: pgmap v1804: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:38:24.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:38:24 smithi181 ceph-mon[94189]: pgmap v1804: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:38:25.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:38:24 smithi038 ceph-mon[92422]: pgmap v1804: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:38:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:38:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2415.971681 secs 2023-03-09T15:38:25.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:38:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2415.971681 secs 2023-03-09T15:38:26.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:38:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2415.971681 secs 2023-03-09T15:38:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:38:26 smithi122 ceph-mon[93869]: pgmap v1805: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:38:26.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:38:26 smithi181 ceph-mon[94189]: pgmap v1805: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:38:27.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:38:26 smithi038 ceph-mon[92422]: pgmap v1805: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:38:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:38:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2418.175205 secs 2023-03-09T15:38:27.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:38:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2418.175205 secs 2023-03-09T15:38:28.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:38:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2418.175205 secs 2023-03-09T15:38:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2880.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:38:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:38:28 smithi122 ceph-mon[93869]: pgmap v1806: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:38:28.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:38:28 smithi181 ceph-mon[94189]: pgmap v1806: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:38:29.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:38:28 smithi038 ceph-mon[92422]: pgmap v1806: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:38:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:38:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2420.971742 secs 2023-03-09T15:38:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:38:30 smithi122 ceph-mon[93869]: pgmap v1807: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 182 B/s wr, 5 op/s 2023-03-09T15:38:31.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:38:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2420.971742 secs 2023-03-09T15:38:31.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:38:30 smithi181 ceph-mon[94189]: pgmap v1807: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 182 B/s wr, 5 op/s 2023-03-09T15:38:31.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:38:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2420.971742 secs 2023-03-09T15:38:31.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:38:30 smithi038 ceph-mon[92422]: pgmap v1807: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 182 B/s wr, 5 op/s 2023-03-09T15:38:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:38:32 smithi122 ceph-mon[93869]: pgmap v1808: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 341 B/s wr, 7 op/s 2023-03-09T15:38:33.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:38:32 smithi181 ceph-mon[94189]: pgmap v1808: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 341 B/s wr, 7 op/s 2023-03-09T15:38:33.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:38:32 smithi038 ceph-mon[92422]: pgmap v1808: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 341 B/s wr, 7 op/s 2023-03-09T15:38:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2885.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:38:34.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:38:34 smithi122 ceph-mon[93869]: pgmap v1809: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 6 op/s 2023-03-09T15:38:34.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:38:34 smithi181 ceph-mon[94189]: pgmap v1809: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 6 op/s 2023-03-09T15:38:35.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:38:34 smithi038 ceph-mon[92422]: pgmap v1809: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 6 op/s 2023-03-09T15:38:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:38:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2425.971801 secs 2023-03-09T15:38:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:38:35 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:38:35.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:38:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2425.971801 secs 2023-03-09T15:38:35.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:38:35 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:38:36.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:38:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2425.971801 secs 2023-03-09T15:38:36.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:38:35 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:38:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:38:36 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:38:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:38:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:38:36.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:38:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:38:36.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:38:36 smithi122 ceph-mon[93869]: pgmap v1810: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T15:38:36.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:38:36 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:38:36.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:38:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:38:36.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:38:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:38:36.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:38:36 smithi181 ceph-mon[94189]: pgmap v1810: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T15:38:37.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:38:36 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:38:37.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:38:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:38:37.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:38:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:38:37.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:38:36 smithi038 ceph-mon[92422]: pgmap v1810: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T15:38:38.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2890.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:38:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:38:38 smithi122 ceph-mon[93869]: pgmap v1811: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 58 KiB/s wr, 7 op/s 2023-03-09T15:38:39.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:38:38 smithi181 ceph-mon[94189]: pgmap v1811: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 58 KiB/s wr, 7 op/s 2023-03-09T15:38:39.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:38:38 smithi038 ceph-mon[92422]: pgmap v1811: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 58 KiB/s wr, 7 op/s 2023-03-09T15:38:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:38:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2430.971879 secs 2023-03-09T15:38:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:38:40 smithi122 ceph-mon[93869]: pgmap v1812: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 58 KiB/s wr, 6 op/s 2023-03-09T15:38:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:38:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2430.971879 secs 2023-03-09T15:38:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:38:40 smithi181 ceph-mon[94189]: pgmap v1812: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 58 KiB/s wr, 6 op/s 2023-03-09T15:38:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:38:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2430.971879 secs 2023-03-09T15:38:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:38:40 smithi038 ceph-mon[92422]: pgmap v1812: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 58 KiB/s wr, 6 op/s 2023-03-09T15:38:43.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:38:42 smithi122 ceph-mon[93869]: pgmap v1813: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 73 KiB/s wr, 11 op/s 2023-03-09T15:38:43.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:38:42 smithi181 ceph-mon[94189]: pgmap v1813: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 73 KiB/s wr, 11 op/s 2023-03-09T15:38:43.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:38:42 smithi038 ceph-mon[92422]: pgmap v1813: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 73 KiB/s wr, 11 op/s 2023-03-09T15:38:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2895.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:38:44.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:38:44 smithi122 ceph-mon[93869]: pgmap v1814: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 85 KiB/s wr, 9 op/s 2023-03-09T15:38:44.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:38:44 smithi181 ceph-mon[94189]: pgmap v1814: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 85 KiB/s wr, 9 op/s 2023-03-09T15:38:45.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:38:44 smithi038 ceph-mon[92422]: pgmap v1814: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 85 KiB/s wr, 9 op/s 2023-03-09T15:38:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:38:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2435.971939 secs 2023-03-09T15:38:45.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:38:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2435.971939 secs 2023-03-09T15:38:46.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:38:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2435.971939 secs 2023-03-09T15:38:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:38:46 smithi122 ceph-mon[93869]: pgmap v1815: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 116 KiB/s wr, 12 op/s 2023-03-09T15:38:46.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:38:46 smithi181 ceph-mon[94189]: pgmap v1815: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 116 KiB/s wr, 12 op/s 2023-03-09T15:38:47.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:38:46 smithi038 ceph-mon[92422]: pgmap v1815: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 116 KiB/s wr, 12 op/s 2023-03-09T15:38:47.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:38:47 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 2438.175585 secs 2023-03-09T15:38:47.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:38:47 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 2438.175585 secs 2023-03-09T15:38:48.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:38:47 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 2438.175585 secs 2023-03-09T15:38:48.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2900.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:38:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:38:48 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:38:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:38:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:38:48.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:38:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:38:48.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:38:48 smithi122 ceph-mon[93869]: pgmap v1816: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 119 KiB/s wr, 11 op/s 2023-03-09T15:38:48.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:38:48 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:38:48.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:38:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:38:48.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:38:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:38:48.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:38:48 smithi181 ceph-mon[94189]: pgmap v1816: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 119 KiB/s wr, 11 op/s 2023-03-09T15:38:49.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:38:48 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:38:49.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:38:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:38:49.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:38:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:38:49.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:38:48 smithi038 ceph-mon[92422]: pgmap v1816: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 119 KiB/s wr, 11 op/s 2023-03-09T15:38:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:38:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2440.971999 secs 2023-03-09T15:38:51.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:38:50 smithi122 ceph-mon[93869]: pgmap v1817: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 81 KiB/s wr, 10 op/s 2023-03-09T15:38:51.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:38:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2440.971999 secs 2023-03-09T15:38:51.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:38:50 smithi181 ceph-mon[94189]: pgmap v1817: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 81 KiB/s wr, 10 op/s 2023-03-09T15:38:51.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:38:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2440.971999 secs 2023-03-09T15:38:51.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:38:50 smithi038 ceph-mon[92422]: pgmap v1817: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 81 KiB/s wr, 10 op/s 2023-03-09T15:38:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:38:52 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 2443.175696 secs 2023-03-09T15:38:53.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:38:52 smithi122 ceph-mon[93869]: pgmap v1818: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 89 KiB/s wr, 13 op/s 2023-03-09T15:38:53.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:38:52 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 2443.175696 secs 2023-03-09T15:38:53.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:38:52 smithi181 ceph-mon[94189]: pgmap v1818: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 89 KiB/s wr, 13 op/s 2023-03-09T15:38:53.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:38:52 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 2443.175696 secs 2023-03-09T15:38:53.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:38:52 smithi038 ceph-mon[92422]: pgmap v1818: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 89 KiB/s wr, 13 op/s 2023-03-09T15:38:53.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2905.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:38:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:38:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:38:54.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:38:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:38:54.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:38:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:38:55.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:38:54 smithi122 ceph-mon[93869]: pgmap v1819: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 74 KiB/s wr, 9 op/s 2023-03-09T15:38:55.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:38:54 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:38:55.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:38:54 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:38:55.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:38:54 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:38:55.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:38:54 smithi181 ceph-mon[94189]: pgmap v1819: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 74 KiB/s wr, 9 op/s 2023-03-09T15:38:55.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:38:54 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:38:55.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:38:54 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:38:55.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:38:54 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:38:55.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:38:54 smithi038 ceph-mon[92422]: pgmap v1819: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 74 KiB/s wr, 9 op/s 2023-03-09T15:38:55.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:38:54 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:38:55.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:38:54 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:38:55.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:38:54 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:38:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:38:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2445.972058 secs 2023-03-09T15:38:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:38:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2445.972058 secs 2023-03-09T15:38:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:38:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2445.972058 secs 2023-03-09T15:38:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:38:56 smithi122 ceph-mon[93869]: pgmap v1820: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 132 KiB/s wr, 11 op/s 2023-03-09T15:38:57.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:38:56 smithi181 ceph-mon[94189]: pgmap v1820: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 132 KiB/s wr, 11 op/s 2023-03-09T15:38:57.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:38:56 smithi038 ceph-mon[92422]: pgmap v1820: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 132 KiB/s wr, 11 op/s 2023-03-09T15:38:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:38:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2448.175815 secs 2023-03-09T15:38:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:38:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2448.175815 secs 2023-03-09T15:38:58.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:38:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2448.175815 secs 2023-03-09T15:38:58.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2910.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:38:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:38:58 smithi122 ceph-mon[93869]: pgmap v1821: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 89 KiB/s wr, 10 op/s 2023-03-09T15:38:59.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:38:58 smithi181 ceph-mon[94189]: pgmap v1821: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 89 KiB/s wr, 10 op/s 2023-03-09T15:38:59.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:38:58 smithi038 ceph-mon[92422]: pgmap v1821: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 89 KiB/s wr, 10 op/s 2023-03-09T15:39:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:39:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2450.972117 secs 2023-03-09T15:39:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:39:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:39:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:39:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:39:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:39:00 smithi122 ceph-mon[93869]: pgmap v1822: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 85 KiB/s wr, 9 op/s 2023-03-09T15:39:00.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:39:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2450.972117 secs 2023-03-09T15:39:00.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:39:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:39:00.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:39:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:39:00.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:39:00 smithi181 ceph-mon[94189]: pgmap v1822: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 85 KiB/s wr, 9 op/s 2023-03-09T15:39:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:39:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2450.972117 secs 2023-03-09T15:39:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:39:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:39:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:39:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:39:01.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:39:00 smithi038 ceph-mon[92422]: pgmap v1822: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 85 KiB/s wr, 9 op/s 2023-03-09T15:39:03.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:39:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2453.175923 secs 2023-03-09T15:39:03.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:39:02 smithi122 ceph-mon[93869]: pgmap v1823: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 141 KiB/s wr, 14 op/s 2023-03-09T15:39:03.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:39:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2453.175923 secs 2023-03-09T15:39:03.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:39:02 smithi181 ceph-mon[94189]: pgmap v1823: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 141 KiB/s wr, 14 op/s 2023-03-09T15:39:03.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:39:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2453.175923 secs 2023-03-09T15:39:03.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:39:02 smithi038 ceph-mon[92422]: pgmap v1823: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 141 KiB/s wr, 14 op/s 2023-03-09T15:39:03.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2915.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:39:04.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:39:04 smithi122 ceph-mon[93869]: pgmap v1824: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 154 KiB/s wr, 23 op/s 2023-03-09T15:39:04.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:39:04 smithi181 ceph-mon[94189]: pgmap v1824: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 154 KiB/s wr, 23 op/s 2023-03-09T15:39:05.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:39:04 smithi038 ceph-mon[92422]: pgmap v1824: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 154 KiB/s wr, 23 op/s 2023-03-09T15:39:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:39:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2455.972180 secs 2023-03-09T15:39:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:39:05 smithi122 ceph-mon[93869]: osdmap e328: 12 total, 12 up, 12 in 2023-03-09T15:39:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:39:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2455.972180 secs 2023-03-09T15:39:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:39:05 smithi181 ceph-mon[94189]: osdmap e328: 12 total, 12 up, 12 in 2023-03-09T15:39:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:39:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2455.972180 secs 2023-03-09T15:39:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:39:05 smithi038 ceph-mon[92422]: osdmap e328: 12 total, 12 up, 12 in 2023-03-09T15:39:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:39:06 smithi122 ceph-mon[93869]: pgmap v1826: 129 pgs: 7 active+clean+snaptrim_wait, 122 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 158 KiB/s wr, 27 op/s 2023-03-09T15:39:06.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:39:06 smithi181 ceph-mon[94189]: pgmap v1826: 129 pgs: 7 active+clean+snaptrim_wait, 122 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 158 KiB/s wr, 27 op/s 2023-03-09T15:39:07.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:39:06 smithi038 ceph-mon[92422]: pgmap v1826: 129 pgs: 7 active+clean+snaptrim_wait, 122 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 158 KiB/s wr, 27 op/s 2023-03-09T15:39:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:39:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2458.176047 secs 2023-03-09T15:39:07.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:39:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2458.176047 secs 2023-03-09T15:39:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:39:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2458.176047 secs 2023-03-09T15:39:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2920.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:39:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:39:08 smithi122 ceph-mon[93869]: pgmap v1827: 129 pgs: 7 active+clean+snaptrim_wait, 122 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 149 KiB/s wr, 27 op/s 2023-03-09T15:39:08.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:39:08 smithi181 ceph-mon[94189]: pgmap v1827: 129 pgs: 7 active+clean+snaptrim_wait, 122 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 149 KiB/s wr, 27 op/s 2023-03-09T15:39:09.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:39:08 smithi038 ceph-mon[92422]: pgmap v1827: 129 pgs: 7 active+clean+snaptrim_wait, 122 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 149 KiB/s wr, 27 op/s 2023-03-09T15:39:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:39:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2460.972240 secs 2023-03-09T15:39:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:39:10 smithi122 ceph-mon[93869]: pgmap v1828: 129 pgs: 7 active+clean+snaptrim_wait, 122 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 149 KiB/s wr, 26 op/s 2023-03-09T15:39:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:39:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2460.972240 secs 2023-03-09T15:39:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:39:10 smithi181 ceph-mon[94189]: pgmap v1828: 129 pgs: 7 active+clean+snaptrim_wait, 122 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 149 KiB/s wr, 26 op/s 2023-03-09T15:39:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:39:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2460.972240 secs 2023-03-09T15:39:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:39:10 smithi038 ceph-mon[92422]: pgmap v1828: 129 pgs: 7 active+clean+snaptrim_wait, 122 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 149 KiB/s wr, 26 op/s 2023-03-09T15:39:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:39:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2463.176167 secs 2023-03-09T15:39:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:39:12 smithi122 ceph-mon[93869]: pgmap v1829: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 97 KiB/s wr, 23 op/s 2023-03-09T15:39:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:39:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2463.176167 secs 2023-03-09T15:39:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:39:12 smithi181 ceph-mon[94189]: pgmap v1829: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 97 KiB/s wr, 23 op/s 2023-03-09T15:39:13.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:39:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2463.176167 secs 2023-03-09T15:39:13.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:39:12 smithi038 ceph-mon[92422]: pgmap v1829: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 97 KiB/s wr, 23 op/s 2023-03-09T15:39:13.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2925.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:39:14.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:39:14 smithi122 ceph-mon[93869]: osdmap e329: 12 total, 12 up, 12 in 2023-03-09T15:39:14.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:39:14 smithi122 ceph-mon[93869]: pgmap v1831: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 77 KiB/s wr, 9 op/s 2023-03-09T15:39:14.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:39:14 smithi181 ceph-mon[94189]: osdmap e329: 12 total, 12 up, 12 in 2023-03-09T15:39:14.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:39:14 smithi181 ceph-mon[94189]: pgmap v1831: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 77 KiB/s wr, 9 op/s 2023-03-09T15:39:15.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:39:14 smithi038 ceph-mon[92422]: osdmap e329: 12 total, 12 up, 12 in 2023-03-09T15:39:15.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:39:14 smithi038 ceph-mon[92422]: pgmap v1831: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 77 KiB/s wr, 9 op/s 2023-03-09T15:39:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:39:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2465.972302 secs 2023-03-09T15:39:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:39:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2465.972302 secs 2023-03-09T15:39:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:39:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2465.972302 secs 2023-03-09T15:39:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:39:16 smithi122 ceph-mon[93869]: pgmap v1832: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 17 KiB/s wr, 6 op/s 2023-03-09T15:39:16.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:39:16 smithi181 ceph-mon[94189]: pgmap v1832: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 17 KiB/s wr, 6 op/s 2023-03-09T15:39:17.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:39:16 smithi038 ceph-mon[92422]: pgmap v1832: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 17 KiB/s wr, 6 op/s 2023-03-09T15:39:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:39:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2468.176288 secs 2023-03-09T15:39:17.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:39:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2468.176288 secs 2023-03-09T15:39:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:39:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2468.176288 secs 2023-03-09T15:39:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2930.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:39:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:39:18 smithi122 ceph-mon[93869]: pgmap v1833: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 6 op/s 2023-03-09T15:39:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:39:18 smithi181 ceph-mon[94189]: pgmap v1833: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 6 op/s 2023-03-09T15:39:19.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:39:18 smithi038 ceph-mon[92422]: pgmap v1833: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 6 op/s 2023-03-09T15:39:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:39:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2470.972374 secs 2023-03-09T15:39:21.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:39:20 smithi122 ceph-mon[93869]: pgmap v1834: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 6 op/s 2023-03-09T15:39:21.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:39:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2470.972374 secs 2023-03-09T15:39:21.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:39:20 smithi181 ceph-mon[94189]: pgmap v1834: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 6 op/s 2023-03-09T15:39:21.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:39:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2470.972374 secs 2023-03-09T15:39:21.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:39:20 smithi038 ceph-mon[92422]: pgmap v1834: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 6 op/s 2023-03-09T15:39:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:39:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2473.176409 secs 2023-03-09T15:39:23.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:39:22 smithi122 ceph-mon[93869]: pgmap v1835: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T15:39:23.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:39:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2473.176409 secs 2023-03-09T15:39:23.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:39:22 smithi181 ceph-mon[94189]: pgmap v1835: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T15:39:23.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:39:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2473.176409 secs 2023-03-09T15:39:23.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:39:22 smithi038 ceph-mon[92422]: pgmap v1835: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T15:39:23.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2935.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:39:24.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:39:24 smithi122 ceph-mon[93869]: pgmap v1836: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T15:39:24.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:39:24 smithi181 ceph-mon[94189]: pgmap v1836: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T15:39:25.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:39:24 smithi038 ceph-mon[92422]: pgmap v1836: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T15:39:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:39:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2475.972435 secs 2023-03-09T15:39:25.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:39:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2475.972435 secs 2023-03-09T15:39:26.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:39:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2475.972435 secs 2023-03-09T15:39:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:39:26 smithi122 ceph-mon[93869]: pgmap v1837: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 32 KiB/s wr, 9 op/s 2023-03-09T15:39:26.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:39:26 smithi181 ceph-mon[94189]: pgmap v1837: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 32 KiB/s wr, 9 op/s 2023-03-09T15:39:27.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:39:26 smithi038 ceph-mon[92422]: pgmap v1837: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 32 KiB/s wr, 9 op/s 2023-03-09T15:39:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:39:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2478.176529 secs 2023-03-09T15:39:27.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:39:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2478.176529 secs 2023-03-09T15:39:28.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:39:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2478.176529 secs 2023-03-09T15:39:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2940.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:39:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:39:28 smithi122 ceph-mon[93869]: pgmap v1838: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T15:39:28.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:39:28 smithi181 ceph-mon[94189]: pgmap v1838: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T15:39:29.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:39:28 smithi038 ceph-mon[92422]: pgmap v1838: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T15:39:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:39:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2480.972496 secs 2023-03-09T15:39:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:39:30 smithi122 ceph-mon[93869]: pgmap v1839: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T15:39:31.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:39:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2480.972496 secs 2023-03-09T15:39:31.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:39:30 smithi181 ceph-mon[94189]: pgmap v1839: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T15:39:31.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:39:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2480.972496 secs 2023-03-09T15:39:31.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:39:30 smithi038 ceph-mon[92422]: pgmap v1839: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T15:39:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:39:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2483.176633 secs 2023-03-09T15:39:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:39:32 smithi122 ceph-mon[93869]: pgmap v1840: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 85 KiB/s wr, 10 op/s 2023-03-09T15:39:33.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:39:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2483.176633 secs 2023-03-09T15:39:33.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:39:32 smithi181 ceph-mon[94189]: pgmap v1840: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 85 KiB/s wr, 10 op/s 2023-03-09T15:39:33.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:39:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2483.176633 secs 2023-03-09T15:39:33.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:39:32 smithi038 ceph-mon[92422]: pgmap v1840: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 85 KiB/s wr, 10 op/s 2023-03-09T15:39:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2945.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:39:34.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:39:34 smithi122 ceph-mon[93869]: pgmap v1841: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 73 KiB/s wr, 18 op/s 2023-03-09T15:39:34.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:39:34 smithi181 ceph-mon[94189]: pgmap v1841: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 73 KiB/s wr, 18 op/s 2023-03-09T15:39:35.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:39:34 smithi038 ceph-mon[92422]: pgmap v1841: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 73 KiB/s wr, 18 op/s 2023-03-09T15:39:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:39:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2485.972556 secs 2023-03-09T15:39:35.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:39:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2485.972556 secs 2023-03-09T15:39:36.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:39:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2485.972556 secs 2023-03-09T15:39:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:39:36 smithi122 ceph-mon[93869]: pgmap v1842: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 114 KiB/s wr, 21 op/s 2023-03-09T15:39:36.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:39:36 smithi181 ceph-mon[94189]: pgmap v1842: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 114 KiB/s wr, 21 op/s 2023-03-09T15:39:37.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:39:36 smithi038 ceph-mon[92422]: pgmap v1842: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 114 KiB/s wr, 21 op/s 2023-03-09T15:39:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:39:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2488.176733 secs 2023-03-09T15:39:37.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:39:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2488.176733 secs 2023-03-09T15:39:38.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:39:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2488.176733 secs 2023-03-09T15:39:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2950.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:39:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:39:38 smithi122 ceph-mon[93869]: pgmap v1843: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 109 KiB/s wr, 36 op/s 2023-03-09T15:39:38.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:39:38 smithi181 ceph-mon[94189]: pgmap v1843: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 109 KiB/s wr, 36 op/s 2023-03-09T15:39:39.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:39:38 smithi038 ceph-mon[92422]: pgmap v1843: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 109 KiB/s wr, 36 op/s 2023-03-09T15:39:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:39:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2490.972618 secs 2023-03-09T15:39:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:39:40 smithi122 ceph-mon[93869]: pgmap v1844: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 109 KiB/s wr, 36 op/s 2023-03-09T15:39:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:39:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2490.972618 secs 2023-03-09T15:39:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:39:40 smithi181 ceph-mon[94189]: pgmap v1844: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 109 KiB/s wr, 36 op/s 2023-03-09T15:39:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:39:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2490.972618 secs 2023-03-09T15:39:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:39:40 smithi038 ceph-mon[92422]: pgmap v1844: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 109 KiB/s wr, 36 op/s 2023-03-09T15:39:43.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:39:42 smithi122 ceph-mon[93869]: pgmap v1845: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 156 KiB/s wr, 41 op/s 2023-03-09T15:39:43.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:39:42 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:39:43.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:39:42 smithi181 ceph-mon[94189]: pgmap v1845: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 156 KiB/s wr, 41 op/s 2023-03-09T15:39:43.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:39:42 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:39:43.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:39:42 smithi038 ceph-mon[92422]: pgmap v1845: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 156 KiB/s wr, 41 op/s 2023-03-09T15:39:43.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:39:42 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:39:43.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2955.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:39:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:39:43 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:39:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:39:43 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:39:44.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:39:43 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:39:44.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:39:43 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:39:44.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:39:43 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:39:44.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:39:43 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:39:44.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:39:43 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:39:44.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:39:43 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:39:44.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:39:43 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:39:45.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:39:44 smithi122 ceph-mon[93869]: pgmap v1846: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 103 KiB/s wr, 37 op/s 2023-03-09T15:39:45.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:39:44 smithi181 ceph-mon[94189]: pgmap v1846: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 103 KiB/s wr, 37 op/s 2023-03-09T15:39:45.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:39:44 smithi038 ceph-mon[92422]: pgmap v1846: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 103 KiB/s wr, 37 op/s 2023-03-09T15:39:46.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:39:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2495.972682 secs 2023-03-09T15:39:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:39:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2495.972682 secs 2023-03-09T15:39:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:39:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2495.972682 secs 2023-03-09T15:39:47.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:39:46 smithi038 ceph-mon[92422]: pgmap v1847: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 103 KiB/s wr, 29 op/s 2023-03-09T15:39:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:39:46 smithi122 ceph-mon[93869]: pgmap v1847: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 103 KiB/s wr, 29 op/s 2023-03-09T15:39:47.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:39:46 smithi181 ceph-mon[94189]: pgmap v1847: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 103 KiB/s wr, 29 op/s 2023-03-09T15:39:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:39:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2498.176967 secs 2023-03-09T15:39:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:39:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:39:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:39:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:39:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:39:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2498.176967 secs 2023-03-09T15:39:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:39:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:39:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:39:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:39:48.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:39:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2498.176967 secs 2023-03-09T15:39:48.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:39:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:39:48.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:39:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:39:48.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2960.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:39:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:39:49 smithi122 ceph-mon[93869]: pgmap v1848: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 107 KiB/s wr, 27 op/s 2023-03-09T15:39:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:39:49 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:39:49.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:39:49 smithi181 ceph-mon[94189]: pgmap v1848: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 107 KiB/s wr, 27 op/s 2023-03-09T15:39:49.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:39:49 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:39:49.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:39:49 smithi038 ceph-mon[92422]: pgmap v1848: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 107 KiB/s wr, 27 op/s 2023-03-09T15:39:49.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:39:49 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:39:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:39:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2500.972742 secs 2023-03-09T15:39:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:39:51 smithi122 ceph-mon[93869]: pgmap v1849: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 93 KiB/s wr, 9 op/s 2023-03-09T15:39:51.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:39:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2500.972742 secs 2023-03-09T15:39:51.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:39:51 smithi181 ceph-mon[94189]: pgmap v1849: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 93 KiB/s wr, 9 op/s 2023-03-09T15:39:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:39:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2500.972742 secs 2023-03-09T15:39:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:39:51 smithi038 ceph-mon[92422]: pgmap v1849: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 93 KiB/s wr, 9 op/s 2023-03-09T15:39:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:39:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2503.177083 secs 2023-03-09T15:39:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:39:53 smithi122 ceph-mon[93869]: pgmap v1850: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 141 KiB/s wr, 13 op/s 2023-03-09T15:39:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:39:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2503.177083 secs 2023-03-09T15:39:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:39:53 smithi181 ceph-mon[94189]: pgmap v1850: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 141 KiB/s wr, 13 op/s 2023-03-09T15:39:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:39:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2503.177083 secs 2023-03-09T15:39:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:39:53 smithi038 ceph-mon[92422]: pgmap v1850: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 141 KiB/s wr, 13 op/s 2023-03-09T15:39:53.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2965.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:39:54.665 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:39:54 smithi122 ceph-mon[93869]: pgmap v1851: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 94 KiB/s wr, 8 op/s 2023-03-09T15:39:54.665 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:39:54 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:39:54.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:39:54 smithi181 ceph-mon[94189]: pgmap v1851: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 94 KiB/s wr, 8 op/s 2023-03-09T15:39:54.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:39:54 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:39:55.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:39:54 smithi038 ceph-mon[92422]: pgmap v1851: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 94 KiB/s wr, 8 op/s 2023-03-09T15:39:55.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:39:54 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:39:56.087 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:39:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2505.972804 secs 2023-03-09T15:39:56.087 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:39:55 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:39:56.088 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:39:55 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:39:56.088 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:39:55 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:39:56.088 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:39:55 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:39:56.088 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:39:55 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:39:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:39:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2505.972804 secs 2023-03-09T15:39:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:39:55 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:39:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:39:55 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:39:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:39:55 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:39:56.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:39:55 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:39:56.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:39:55 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:39:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:39:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2505.972804 secs 2023-03-09T15:39:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:39:55 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:39:56.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:39:55 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:39:56.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:39:55 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:39:56.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:39:55 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:39:56.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:39:55 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:39:57.133 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:39:56 smithi122 ceph-mon[93869]: pgmap v1852: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 94 KiB/s wr, 10 op/s 2023-03-09T15:39:57.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:39:56 smithi181 ceph-mon[94189]: pgmap v1852: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 94 KiB/s wr, 10 op/s 2023-03-09T15:39:57.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:39:56 smithi038 ceph-mon[92422]: pgmap v1852: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 94 KiB/s wr, 10 op/s 2023-03-09T15:39:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:39:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2508.177197 secs 2023-03-09T15:39:58.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:39:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2508.177197 secs 2023-03-09T15:39:58.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:39:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2508.177197 secs 2023-03-09T15:39:58.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2970.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:39:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:39:58 smithi122 ceph-mon[93869]: pgmap v1853: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 112 KiB/s wr, 9 op/s 2023-03-09T15:39:59.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:39:58 smithi181 ceph-mon[94189]: pgmap v1853: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 112 KiB/s wr, 9 op/s 2023-03-09T15:39:59.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:39:58 smithi038 ceph-mon[92422]: pgmap v1853: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 112 KiB/s wr, 9 op/s 2023-03-09T15:40:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:40:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2510.972863 secs 2023-03-09T15:40:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:40:00 smithi122 ceph-mon[93869]: pgmap v1854: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 67 KiB/s wr, 7 op/s 2023-03-09T15:40:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:40:00 smithi122 ceph-mon[93869]: overall HEALTH_WARN 2 MDSs report slow requests 2023-03-09T15:40:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:40:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2510.972863 secs 2023-03-09T15:40:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:40:00 smithi181 ceph-mon[94189]: pgmap v1854: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 67 KiB/s wr, 7 op/s 2023-03-09T15:40:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:40:00 smithi181 ceph-mon[94189]: overall HEALTH_WARN 2 MDSs report slow requests 2023-03-09T15:40:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:40:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2510.972863 secs 2023-03-09T15:40:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:40:00 smithi038 ceph-mon[92422]: pgmap v1854: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 67 KiB/s wr, 7 op/s 2023-03-09T15:40:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:40:00 smithi038 ceph-mon[92422]: overall HEALTH_WARN 2 MDSs report slow requests 2023-03-09T15:40:03.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:40:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2513.177305 secs 2023-03-09T15:40:03.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:40:02 smithi122 ceph-mon[93869]: pgmap v1855: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 98 KiB/s wr, 11 op/s 2023-03-09T15:40:03.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:40:02 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:40:03.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:40:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2513.177305 secs 2023-03-09T15:40:03.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:40:02 smithi181 ceph-mon[94189]: pgmap v1855: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 98 KiB/s wr, 11 op/s 2023-03-09T15:40:03.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:40:02 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:40:03.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:40:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2513.177305 secs 2023-03-09T15:40:03.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:40:02 smithi038 ceph-mon[92422]: pgmap v1855: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 98 KiB/s wr, 11 op/s 2023-03-09T15:40:03.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:40:02 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:40:03.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2975.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:40:04.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:40:04 smithi122 ceph-mon[93869]: pgmap v1856: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 72 KiB/s wr, 20 op/s 2023-03-09T15:40:04.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:40:04 smithi181 ceph-mon[94189]: pgmap v1856: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 72 KiB/s wr, 20 op/s 2023-03-09T15:40:05.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:40:04 smithi038 ceph-mon[92422]: pgmap v1856: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 72 KiB/s wr, 20 op/s 2023-03-09T15:40:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:40:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2515.972927 secs 2023-03-09T15:40:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:40:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2515.972927 secs 2023-03-09T15:40:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:40:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2515.972927 secs 2023-03-09T15:40:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:40:06 smithi122 ceph-mon[93869]: pgmap v1857: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 83 KiB/s wr, 23 op/s 2023-03-09T15:40:06.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:40:06 smithi181 ceph-mon[94189]: pgmap v1857: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 83 KiB/s wr, 23 op/s 2023-03-09T15:40:07.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:40:06 smithi038 ceph-mon[92422]: pgmap v1857: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 83 KiB/s wr, 23 op/s 2023-03-09T15:40:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:40:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2518.177391 secs 2023-03-09T15:40:07.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:40:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2518.177391 secs 2023-03-09T15:40:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:40:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2518.177391 secs 2023-03-09T15:40:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2980.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:40:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:40:08 smithi122 ceph-mon[93869]: pgmap v1858: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 84 KiB/s wr, 22 op/s 2023-03-09T15:40:08.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:40:08 smithi181 ceph-mon[94189]: pgmap v1858: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 84 KiB/s wr, 22 op/s 2023-03-09T15:40:09.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:40:08 smithi038 ceph-mon[92422]: pgmap v1858: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 84 KiB/s wr, 22 op/s 2023-03-09T15:40:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:40:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2520.972985 secs 2023-03-09T15:40:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:40:10 smithi122 ceph-mon[93869]: osdmap e330: 12 total, 12 up, 12 in 2023-03-09T15:40:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:40:10 smithi122 ceph-mon[93869]: pgmap v1860: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 79 KiB/s wr, 25 op/s 2023-03-09T15:40:10.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:40:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2520.972985 secs 2023-03-09T15:40:10.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:40:10 smithi181 ceph-mon[94189]: osdmap e330: 12 total, 12 up, 12 in 2023-03-09T15:40:10.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:40:10 smithi181 ceph-mon[94189]: pgmap v1860: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 79 KiB/s wr, 25 op/s 2023-03-09T15:40:11.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:40:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2520.972985 secs 2023-03-09T15:40:11.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:40:10 smithi038 ceph-mon[92422]: osdmap e330: 12 total, 12 up, 12 in 2023-03-09T15:40:11.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:40:10 smithi038 ceph-mon[92422]: pgmap v1860: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 79 KiB/s wr, 25 op/s 2023-03-09T15:40:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:40:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2523.177479 secs 2023-03-09T15:40:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:40:12 smithi122 ceph-mon[93869]: pgmap v1861: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 63 KiB/s wr, 24 op/s 2023-03-09T15:40:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:40:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2523.177479 secs 2023-03-09T15:40:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:40:12 smithi181 ceph-mon[94189]: pgmap v1861: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 63 KiB/s wr, 24 op/s 2023-03-09T15:40:13.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:40:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2523.177479 secs 2023-03-09T15:40:13.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:40:12 smithi038 ceph-mon[92422]: pgmap v1861: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 63 KiB/s wr, 24 op/s 2023-03-09T15:40:13.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2985.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:40:14.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:40:14 smithi122 ceph-mon[93869]: pgmap v1862: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 37 KiB/s wr, 9 op/s 2023-03-09T15:40:14.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:40:14 smithi181 ceph-mon[94189]: pgmap v1862: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 37 KiB/s wr, 9 op/s 2023-03-09T15:40:15.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:40:14 smithi038 ceph-mon[92422]: pgmap v1862: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 37 KiB/s wr, 9 op/s 2023-03-09T15:40:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:40:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2525.973044 secs 2023-03-09T15:40:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:40:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2525.973044 secs 2023-03-09T15:40:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:40:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2525.973044 secs 2023-03-09T15:40:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:40:16 smithi122 ceph-mon[93869]: pgmap v1863: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T15:40:16.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:40:16 smithi181 ceph-mon[94189]: pgmap v1863: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T15:40:17.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:40:16 smithi038 ceph-mon[92422]: pgmap v1863: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T15:40:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:40:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2528.177548 secs 2023-03-09T15:40:17.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:40:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2528.177548 secs 2023-03-09T15:40:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:40:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2528.177548 secs 2023-03-09T15:40:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2990.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:40:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:40:18 smithi122 ceph-mon[93869]: pgmap v1864: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T15:40:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:40:18 smithi122 ceph-mon[93869]: osdmap e331: 12 total, 12 up, 12 in 2023-03-09T15:40:18.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:40:18 smithi181 ceph-mon[94189]: pgmap v1864: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T15:40:18.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:40:18 smithi181 ceph-mon[94189]: osdmap e331: 12 total, 12 up, 12 in 2023-03-09T15:40:19.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:40:18 smithi038 ceph-mon[92422]: pgmap v1864: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T15:40:19.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:40:18 smithi038 ceph-mon[92422]: osdmap e331: 12 total, 12 up, 12 in 2023-03-09T15:40:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:40:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2530.973105 secs 2023-03-09T15:40:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:40:20 smithi122 ceph-mon[93869]: pgmap v1866: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 8 op/s 2023-03-09T15:40:21.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:40:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2530.973105 secs 2023-03-09T15:40:21.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:40:20 smithi181 ceph-mon[94189]: pgmap v1866: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 8 op/s 2023-03-09T15:40:21.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:40:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2530.973105 secs 2023-03-09T15:40:21.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:40:20 smithi038 ceph-mon[92422]: pgmap v1866: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 8 op/s 2023-03-09T15:40:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:40:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2533.177667 secs 2023-03-09T15:40:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:40:22 smithi122 ceph-mon[93869]: pgmap v1867: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 7 op/s 2023-03-09T15:40:23.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:40:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2533.177667 secs 2023-03-09T15:40:23.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:40:22 smithi181 ceph-mon[94189]: pgmap v1867: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 7 op/s 2023-03-09T15:40:23.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:40:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2533.177667 secs 2023-03-09T15:40:23.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:40:22 smithi038 ceph-mon[92422]: pgmap v1867: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 7 op/s 2023-03-09T15:40:23.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 2995.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:40:24.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:40:24 smithi122 ceph-mon[93869]: pgmap v1868: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 7 op/s 2023-03-09T15:40:24.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:40:24 smithi181 ceph-mon[94189]: pgmap v1868: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 7 op/s 2023-03-09T15:40:25.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:40:24 smithi038 ceph-mon[92422]: pgmap v1868: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 7 op/s 2023-03-09T15:40:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:40:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2535.973165 secs 2023-03-09T15:40:25.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:40:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2535.973165 secs 2023-03-09T15:40:26.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:40:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2535.973165 secs 2023-03-09T15:40:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:40:26 smithi122 ceph-mon[93869]: pgmap v1869: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 7 op/s 2023-03-09T15:40:26.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:40:26 smithi181 ceph-mon[94189]: pgmap v1869: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 7 op/s 2023-03-09T15:40:27.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:40:26 smithi038 ceph-mon[92422]: pgmap v1869: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 7 op/s 2023-03-09T15:40:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:40:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2538.177796 secs 2023-03-09T15:40:27.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:40:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2538.177796 secs 2023-03-09T15:40:28.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:40:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2538.177796 secs 2023-03-09T15:40:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3000.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:40:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:40:28 smithi122 ceph-mon[93869]: pgmap v1870: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 6 op/s 2023-03-09T15:40:28.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:40:28 smithi181 ceph-mon[94189]: pgmap v1870: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 6 op/s 2023-03-09T15:40:29.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:40:28 smithi038 ceph-mon[92422]: pgmap v1870: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 6 op/s 2023-03-09T15:40:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:40:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2540.973226 secs 2023-03-09T15:40:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:40:30 smithi122 ceph-mon[93869]: pgmap v1871: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 5 op/s 2023-03-09T15:40:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:40:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2540.973226 secs 2023-03-09T15:40:31.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:40:30 smithi181 ceph-mon[94189]: pgmap v1871: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 5 op/s 2023-03-09T15:40:31.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:40:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2540.973226 secs 2023-03-09T15:40:31.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:40:30 smithi038 ceph-mon[92422]: pgmap v1871: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 5 op/s 2023-03-09T15:40:33.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:40:32 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 2543.177904 secs 2023-03-09T15:40:33.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:40:32 smithi181 ceph-mon[94189]: pgmap v1872: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 39 KiB/s wr, 8 op/s 2023-03-09T15:40:33.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:40:32 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 2543.177904 secs 2023-03-09T15:40:33.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:40:32 smithi038 ceph-mon[92422]: pgmap v1872: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 39 KiB/s wr, 8 op/s 2023-03-09T15:40:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:40:32 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 2543.177904 secs 2023-03-09T15:40:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:40:32 smithi122 ceph-mon[93869]: pgmap v1872: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 39 KiB/s wr, 8 op/s 2023-03-09T15:40:33.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3005.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:40:34.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:40:34 smithi122 ceph-mon[93869]: pgmap v1873: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 21 KiB/s wr, 5 op/s 2023-03-09T15:40:34.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:40:34 smithi181 ceph-mon[94189]: pgmap v1873: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 21 KiB/s wr, 5 op/s 2023-03-09T15:40:35.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:40:34 smithi038 ceph-mon[92422]: pgmap v1873: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 21 KiB/s wr, 5 op/s 2023-03-09T15:40:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:40:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2545.973289 secs 2023-03-09T15:40:35.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:40:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2545.973289 secs 2023-03-09T15:40:36.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:40:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2545.973289 secs 2023-03-09T15:40:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:40:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:40:36.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:40:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:40:36.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:40:36 smithi122 ceph-mon[93869]: pgmap v1874: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 21 KiB/s wr, 7 op/s 2023-03-09T15:40:36.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:40:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:40:36.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:40:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:40:36.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:40:36 smithi181 ceph-mon[94189]: pgmap v1874: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 21 KiB/s wr, 7 op/s 2023-03-09T15:40:37.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:40:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:40:37.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:40:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:40:37.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:40:36 smithi038 ceph-mon[92422]: pgmap v1874: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 21 KiB/s wr, 7 op/s 2023-03-09T15:40:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:40:37 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 2548.178011 secs 2023-03-09T15:40:37.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:40:37 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 2548.178011 secs 2023-03-09T15:40:38.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:40:37 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 2548.178011 secs 2023-03-09T15:40:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3010.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:40:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:40:38 smithi122 ceph-mon[93869]: pgmap v1875: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 61 KiB/s wr, 7 op/s 2023-03-09T15:40:38.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:40:38 smithi181 ceph-mon[94189]: pgmap v1875: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 61 KiB/s wr, 7 op/s 2023-03-09T15:40:39.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:40:38 smithi038 ceph-mon[92422]: pgmap v1875: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 61 KiB/s wr, 7 op/s 2023-03-09T15:40:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:40:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2550.973349 secs 2023-03-09T15:40:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:40:40 smithi122 ceph-mon[93869]: pgmap v1876: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 60 KiB/s wr, 7 op/s 2023-03-09T15:40:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:40:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2550.973349 secs 2023-03-09T15:40:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:40:40 smithi181 ceph-mon[94189]: pgmap v1876: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 60 KiB/s wr, 7 op/s 2023-03-09T15:40:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:40:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2550.973349 secs 2023-03-09T15:40:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:40:40 smithi038 ceph-mon[92422]: pgmap v1876: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 60 KiB/s wr, 7 op/s 2023-03-09T15:40:43.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:40:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2553.178130 secs 2023-03-09T15:40:43.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:40:42 smithi038 ceph-mon[92422]: pgmap v1877: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 115 KiB/s wr, 11 op/s 2023-03-09T15:40:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:40:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2553.178130 secs 2023-03-09T15:40:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:40:42 smithi122 ceph-mon[93869]: pgmap v1877: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 115 KiB/s wr, 11 op/s 2023-03-09T15:40:43.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:40:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2553.178130 secs 2023-03-09T15:40:43.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:40:42 smithi181 ceph-mon[94189]: pgmap v1877: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 115 KiB/s wr, 11 op/s 2023-03-09T15:40:43.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3015.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:40:44.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:40:43 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:40:44.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:40:43 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:40:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:40:43 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:40:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:40:43 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:40:44.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:40:43 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:40:44.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:40:43 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:40:45.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:40:44 smithi038 ceph-mon[92422]: pgmap v1878: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 96 KiB/s wr, 9 op/s 2023-03-09T15:40:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:40:44 smithi122 ceph-mon[93869]: pgmap v1878: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 96 KiB/s wr, 9 op/s 2023-03-09T15:40:45.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:40:44 smithi181 ceph-mon[94189]: pgmap v1878: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 96 KiB/s wr, 9 op/s 2023-03-09T15:40:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:40:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2555.973408 secs 2023-03-09T15:40:46.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:40:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2555.973408 secs 2023-03-09T15:40:46.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:40:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2555.973408 secs 2023-03-09T15:40:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:40:47 smithi122 ceph-mon[93869]: pgmap v1879: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 98 KiB/s wr, 11 op/s 2023-03-09T15:40:47.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:40:47 smithi181 ceph-mon[94189]: pgmap v1879: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 98 KiB/s wr, 11 op/s 2023-03-09T15:40:47.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:40:47 smithi038 ceph-mon[92422]: pgmap v1879: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 98 KiB/s wr, 11 op/s 2023-03-09T15:40:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:40:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2558.178248 secs 2023-03-09T15:40:48.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:40:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2558.178248 secs 2023-03-09T15:40:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:40:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2558.178248 secs 2023-03-09T15:40:48.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3020.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:40:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:40:49 smithi122 ceph-mon[93869]: pgmap v1880: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 120 KiB/s wr, 11 op/s 2023-03-09T15:40:49.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:40:49 smithi181 ceph-mon[94189]: pgmap v1880: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 120 KiB/s wr, 11 op/s 2023-03-09T15:40:49.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:40:49 smithi038 ceph-mon[92422]: pgmap v1880: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 120 KiB/s wr, 11 op/s 2023-03-09T15:40:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:40:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2560.973469 secs 2023-03-09T15:40:51.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:40:51 smithi122 ceph-mon[93869]: pgmap v1881: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 83 KiB/s wr, 10 op/s 2023-03-09T15:40:51.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:40:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2560.973469 secs 2023-03-09T15:40:51.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:40:51 smithi181 ceph-mon[94189]: pgmap v1881: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 83 KiB/s wr, 10 op/s 2023-03-09T15:40:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:40:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2560.973469 secs 2023-03-09T15:40:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:40:51 smithi038 ceph-mon[92422]: pgmap v1881: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 83 KiB/s wr, 10 op/s 2023-03-09T15:40:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:40:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2563.178358 secs 2023-03-09T15:40:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:40:53 smithi122 ceph-mon[93869]: pgmap v1882: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 91 KiB/s wr, 13 op/s 2023-03-09T15:40:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:40:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2563.178358 secs 2023-03-09T15:40:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:40:53 smithi181 ceph-mon[94189]: pgmap v1882: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 91 KiB/s wr, 13 op/s 2023-03-09T15:40:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:40:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2563.178358 secs 2023-03-09T15:40:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:40:53 smithi038 ceph-mon[92422]: pgmap v1882: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 91 KiB/s wr, 13 op/s 2023-03-09T15:40:53.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3025.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:40:54.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:40:54 smithi122 ceph-mon[93869]: pgmap v1883: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 40 KiB/s wr, 9 op/s 2023-03-09T15:40:54.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:40:54 smithi181 ceph-mon[94189]: pgmap v1883: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 40 KiB/s wr, 9 op/s 2023-03-09T15:40:55.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:40:54 smithi038 ceph-mon[92422]: pgmap v1883: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 40 KiB/s wr, 9 op/s 2023-03-09T15:40:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:40:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2565.973553 secs 2023-03-09T15:40:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:40:55 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:40:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:40:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2565.973553 secs 2023-03-09T15:40:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:40:55 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:40:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:40:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2565.973553 secs 2023-03-09T15:40:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:40:55 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:40:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:40:56 smithi122 ceph-mon[93869]: pgmap v1884: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 112 KiB/s wr, 11 op/s 2023-03-09T15:40:56.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:40:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:40:56.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:40:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:40:56.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:40:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:40:56.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:40:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:40:56.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:40:56 smithi181 ceph-mon[94189]: pgmap v1884: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 112 KiB/s wr, 11 op/s 2023-03-09T15:40:56.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:40:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:40:56.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:40:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:40:56.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:40:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:40:56.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:40:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:40:57.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:40:56 smithi038 ceph-mon[92422]: pgmap v1884: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 112 KiB/s wr, 11 op/s 2023-03-09T15:40:57.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:40:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:40:57.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:40:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:40:57.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:40:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:40:57.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:40:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:40:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:40:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2568.178469 secs 2023-03-09T15:40:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:40:57 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:40:57.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:40:57 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:40:57.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:40:57 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:40:57.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:40:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2568.178469 secs 2023-03-09T15:40:57.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:40:57 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:40:57.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:40:57 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:40:57.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:40:57 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:40:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:40:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2568.178469 secs 2023-03-09T15:40:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:40:57 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:40:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:40:57 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:40:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:40:57 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:40:58.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3030.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:40:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:40:58 smithi122 ceph-mon[93869]: pgmap v1885: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 110 KiB/s wr, 10 op/s 2023-03-09T15:40:58.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:40:58 smithi181 ceph-mon[94189]: pgmap v1885: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 110 KiB/s wr, 10 op/s 2023-03-09T15:40:59.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:40:58 smithi038 ceph-mon[92422]: pgmap v1885: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 110 KiB/s wr, 10 op/s 2023-03-09T15:41:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:41:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2570.973614 secs 2023-03-09T15:41:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:41:00 smithi038 ceph-mon[92422]: pgmap v1886: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 91 KiB/s wr, 10 op/s 2023-03-09T15:41:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:41:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2570.973614 secs 2023-03-09T15:41:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:41:00 smithi122 ceph-mon[93869]: pgmap v1886: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 91 KiB/s wr, 10 op/s 2023-03-09T15:41:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:41:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2570.973614 secs 2023-03-09T15:41:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:41:00 smithi181 ceph-mon[94189]: pgmap v1886: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 91 KiB/s wr, 10 op/s 2023-03-09T15:41:03.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:41:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2573.178615 secs 2023-03-09T15:41:03.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:41:02 smithi038 ceph-mon[92422]: pgmap v1887: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 101 KiB/s wr, 12 op/s 2023-03-09T15:41:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:41:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2573.178615 secs 2023-03-09T15:41:03.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:41:02 smithi122 ceph-mon[93869]: pgmap v1887: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 101 KiB/s wr, 12 op/s 2023-03-09T15:41:03.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:41:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2573.178615 secs 2023-03-09T15:41:03.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:41:02 smithi181 ceph-mon[94189]: pgmap v1887: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 101 KiB/s wr, 12 op/s 2023-03-09T15:41:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3035.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:41:04.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:41:04 smithi122 ceph-mon[93869]: pgmap v1888: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 117 KiB/s wr, 22 op/s 2023-03-09T15:41:04.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:41:04 smithi181 ceph-mon[94189]: pgmap v1888: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 117 KiB/s wr, 22 op/s 2023-03-09T15:41:05.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:41:04 smithi038 ceph-mon[92422]: pgmap v1888: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 117 KiB/s wr, 22 op/s 2023-03-09T15:41:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:41:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2575.973684 secs 2023-03-09T15:41:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:41:05 smithi122 ceph-mon[93869]: osdmap e332: 12 total, 12 up, 12 in 2023-03-09T15:41:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:41:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2575.973684 secs 2023-03-09T15:41:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:41:05 smithi181 ceph-mon[94189]: osdmap e332: 12 total, 12 up, 12 in 2023-03-09T15:41:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:41:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2575.973684 secs 2023-03-09T15:41:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:41:05 smithi038 ceph-mon[92422]: osdmap e332: 12 total, 12 up, 12 in 2023-03-09T15:41:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:41:06 smithi122 ceph-mon[93869]: pgmap v1890: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 124 KiB/s wr, 27 op/s 2023-03-09T15:41:06.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:41:06 smithi181 ceph-mon[94189]: pgmap v1890: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 124 KiB/s wr, 27 op/s 2023-03-09T15:41:07.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:41:06 smithi038 ceph-mon[92422]: pgmap v1890: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 124 KiB/s wr, 27 op/s 2023-03-09T15:41:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:41:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2578.178735 secs 2023-03-09T15:41:07.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:41:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2578.178735 secs 2023-03-09T15:41:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:41:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2578.178735 secs 2023-03-09T15:41:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3040.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:41:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:41:08 smithi122 ceph-mon[93869]: pgmap v1891: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 124 KiB/s wr, 27 op/s 2023-03-09T15:41:08.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:41:08 smithi181 ceph-mon[94189]: pgmap v1891: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 124 KiB/s wr, 27 op/s 2023-03-09T15:41:09.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:41:08 smithi038 ceph-mon[92422]: pgmap v1891: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 124 KiB/s wr, 27 op/s 2023-03-09T15:41:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:41:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2580.973743 secs 2023-03-09T15:41:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:41:10 smithi038 ceph-mon[92422]: pgmap v1892: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 121 KiB/s wr, 26 op/s 2023-03-09T15:41:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:41:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2580.973743 secs 2023-03-09T15:41:11.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:41:10 smithi122 ceph-mon[93869]: pgmap v1892: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 121 KiB/s wr, 26 op/s 2023-03-09T15:41:11.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:41:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2580.973743 secs 2023-03-09T15:41:11.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:41:10 smithi181 ceph-mon[94189]: pgmap v1892: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 121 KiB/s wr, 26 op/s 2023-03-09T15:41:13.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:41:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2583.178814 secs 2023-03-09T15:41:13.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:41:12 smithi038 ceph-mon[92422]: pgmap v1893: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 120 KiB/s wr, 25 op/s 2023-03-09T15:41:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:41:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2583.178814 secs 2023-03-09T15:41:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:41:12 smithi122 ceph-mon[93869]: pgmap v1893: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 120 KiB/s wr, 25 op/s 2023-03-09T15:41:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:41:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2583.178814 secs 2023-03-09T15:41:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:41:12 smithi181 ceph-mon[94189]: pgmap v1893: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 120 KiB/s wr, 25 op/s 2023-03-09T15:41:13.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3045.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:41:14.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:41:14 smithi122 ceph-mon[93869]: osdmap e333: 12 total, 12 up, 12 in 2023-03-09T15:41:14.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:41:14 smithi122 ceph-mon[93869]: pgmap v1895: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 99 KiB/s wr, 11 op/s 2023-03-09T15:41:14.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:41:14 smithi181 ceph-mon[94189]: osdmap e333: 12 total, 12 up, 12 in 2023-03-09T15:41:14.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:41:14 smithi181 ceph-mon[94189]: pgmap v1895: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 99 KiB/s wr, 11 op/s 2023-03-09T15:41:15.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:41:14 smithi038 ceph-mon[92422]: osdmap e333: 12 total, 12 up, 12 in 2023-03-09T15:41:15.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:41:14 smithi038 ceph-mon[92422]: pgmap v1895: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 99 KiB/s wr, 11 op/s 2023-03-09T15:41:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:41:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2585.973800 secs 2023-03-09T15:41:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:41:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2585.973800 secs 2023-03-09T15:41:16.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:41:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2585.973800 secs 2023-03-09T15:41:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:41:16 smithi122 ceph-mon[93869]: pgmap v1896: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 6 op/s 2023-03-09T15:41:16.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:41:16 smithi181 ceph-mon[94189]: pgmap v1896: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 6 op/s 2023-03-09T15:41:17.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:41:16 smithi038 ceph-mon[92422]: pgmap v1896: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 6 op/s 2023-03-09T15:41:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:41:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2588.178935 secs 2023-03-09T15:41:17.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:41:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2588.178935 secs 2023-03-09T15:41:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:41:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2588.178935 secs 2023-03-09T15:41:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3050.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:41:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:41:18 smithi122 ceph-mon[93869]: pgmap v1897: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 6 op/s 2023-03-09T15:41:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:41:18 smithi181 ceph-mon[94189]: pgmap v1897: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 6 op/s 2023-03-09T15:41:19.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:41:18 smithi038 ceph-mon[92422]: pgmap v1897: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 6 op/s 2023-03-09T15:41:21.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:41:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2590.973885 secs 2023-03-09T15:41:21.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:41:20 smithi038 ceph-mon[92422]: pgmap v1898: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 6 op/s 2023-03-09T15:41:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:41:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2590.973885 secs 2023-03-09T15:41:21.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:41:20 smithi122 ceph-mon[93869]: pgmap v1898: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 6 op/s 2023-03-09T15:41:21.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:41:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2590.973885 secs 2023-03-09T15:41:21.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:41:20 smithi181 ceph-mon[94189]: pgmap v1898: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 6 op/s 2023-03-09T15:41:23.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:41:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2593.179031 secs 2023-03-09T15:41:23.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:41:22 smithi038 ceph-mon[92422]: pgmap v1899: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.6 KiB/s wr, 6 op/s 2023-03-09T15:41:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:41:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2593.179031 secs 2023-03-09T15:41:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:41:22 smithi122 ceph-mon[93869]: pgmap v1899: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.6 KiB/s wr, 6 op/s 2023-03-09T15:41:23.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:41:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2593.179031 secs 2023-03-09T15:41:23.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:41:22 smithi181 ceph-mon[94189]: pgmap v1899: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.6 KiB/s wr, 6 op/s 2023-03-09T15:41:23.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3055.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:41:24.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:41:24 smithi122 ceph-mon[93869]: pgmap v1900: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.8 KiB/s wr, 7 op/s 2023-03-09T15:41:24.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:41:24 smithi181 ceph-mon[94189]: pgmap v1900: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.8 KiB/s wr, 7 op/s 2023-03-09T15:41:25.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:41:24 smithi038 ceph-mon[92422]: pgmap v1900: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.8 KiB/s wr, 7 op/s 2023-03-09T15:41:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:41:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2595.973969 secs 2023-03-09T15:41:25.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:41:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2595.973969 secs 2023-03-09T15:41:26.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:41:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2595.973969 secs 2023-03-09T15:41:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:41:26 smithi122 ceph-mon[93869]: pgmap v1901: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 8 op/s 2023-03-09T15:41:26.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:41:26 smithi181 ceph-mon[94189]: pgmap v1901: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 8 op/s 2023-03-09T15:41:27.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:41:26 smithi038 ceph-mon[92422]: pgmap v1901: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 8 op/s 2023-03-09T15:41:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:41:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2598.179155 secs 2023-03-09T15:41:27.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:41:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2598.179155 secs 2023-03-09T15:41:28.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:41:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2598.179155 secs 2023-03-09T15:41:28.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3060.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:41:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:41:28 smithi122 ceph-mon[93869]: pgmap v1902: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 6 op/s 2023-03-09T15:41:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:41:28 smithi181 ceph-mon[94189]: pgmap v1902: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 6 op/s 2023-03-09T15:41:29.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:41:28 smithi038 ceph-mon[92422]: pgmap v1902: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 6 op/s 2023-03-09T15:41:31.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:41:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2600.973986 secs 2023-03-09T15:41:31.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:41:30 smithi038 ceph-mon[92422]: pgmap v1903: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T15:41:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:41:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2600.973986 secs 2023-03-09T15:41:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:41:30 smithi122 ceph-mon[93869]: pgmap v1903: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T15:41:31.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:41:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2600.973986 secs 2023-03-09T15:41:31.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:41:30 smithi181 ceph-mon[94189]: pgmap v1903: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T15:41:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:41:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2603.179297 secs 2023-03-09T15:41:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:41:33 smithi122 ceph-mon[93869]: pgmap v1904: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 68 KiB/s wr, 10 op/s 2023-03-09T15:41:33.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:41:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2603.179297 secs 2023-03-09T15:41:33.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:41:33 smithi181 ceph-mon[94189]: pgmap v1904: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 68 KiB/s wr, 10 op/s 2023-03-09T15:41:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:41:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2603.179297 secs 2023-03-09T15:41:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:41:33 smithi038 ceph-mon[92422]: pgmap v1904: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 68 KiB/s wr, 10 op/s 2023-03-09T15:41:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3065.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:41:34.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:41:34 smithi122 ceph-mon[93869]: pgmap v1905: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 74 KiB/s wr, 8 op/s 2023-03-09T15:41:34.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:41:34 smithi181 ceph-mon[94189]: pgmap v1905: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 74 KiB/s wr, 8 op/s 2023-03-09T15:41:35.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:41:34 smithi038 ceph-mon[92422]: pgmap v1905: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 74 KiB/s wr, 8 op/s 2023-03-09T15:41:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:41:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2605.974054 secs 2023-03-09T15:41:35.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:41:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2605.974054 secs 2023-03-09T15:41:36.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:41:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2605.974054 secs 2023-03-09T15:41:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:41:36 smithi122 ceph-mon[93869]: pgmap v1906: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 118 KiB/s wr, 36 op/s 2023-03-09T15:41:36.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:41:36 smithi181 ceph-mon[94189]: pgmap v1906: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 118 KiB/s wr, 36 op/s 2023-03-09T15:41:37.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:41:36 smithi038 ceph-mon[92422]: pgmap v1906: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 118 KiB/s wr, 36 op/s 2023-03-09T15:41:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:41:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2608.179392 secs 2023-03-09T15:41:37.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:41:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2608.179392 secs 2023-03-09T15:41:38.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:41:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2608.179392 secs 2023-03-09T15:41:38.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3070.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:41:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:41:38 smithi122 ceph-mon[93869]: pgmap v1907: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 99 KiB/s wr, 33 op/s 2023-03-09T15:41:38.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:41:38 smithi181 ceph-mon[94189]: pgmap v1907: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 99 KiB/s wr, 33 op/s 2023-03-09T15:41:39.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:41:38 smithi038 ceph-mon[92422]: pgmap v1907: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 99 KiB/s wr, 33 op/s 2023-03-09T15:41:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:41:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2610.974115 secs 2023-03-09T15:41:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:41:41 smithi122 ceph-mon[93869]: pgmap v1908: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 103 KiB/s wr, 34 op/s 2023-03-09T15:41:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:41:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2610.974115 secs 2023-03-09T15:41:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:41:41 smithi181 ceph-mon[94189]: pgmap v1908: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 103 KiB/s wr, 34 op/s 2023-03-09T15:41:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:41:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2610.974115 secs 2023-03-09T15:41:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:41:41 smithi038 ceph-mon[92422]: pgmap v1908: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 103 KiB/s wr, 34 op/s 2023-03-09T15:41:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:41:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2613.179512 secs 2023-03-09T15:41:43.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:41:43 smithi122 ceph-mon[93869]: pgmap v1909: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 145 KiB/s wr, 37 op/s 2023-03-09T15:41:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:41:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2613.179512 secs 2023-03-09T15:41:43.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:41:43 smithi181 ceph-mon[94189]: pgmap v1909: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 145 KiB/s wr, 37 op/s 2023-03-09T15:41:43.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:41:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2613.179512 secs 2023-03-09T15:41:43.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:41:43 smithi038 ceph-mon[92422]: pgmap v1909: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 145 KiB/s wr, 37 op/s 2023-03-09T15:41:43.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3075.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:41:44.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:41:44 smithi122 ceph-mon[93869]: pgmap v1910: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 99 KiB/s wr, 35 op/s 2023-03-09T15:41:44.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:41:44 smithi181 ceph-mon[94189]: pgmap v1910: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 99 KiB/s wr, 35 op/s 2023-03-09T15:41:45.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:41:44 smithi038 ceph-mon[92422]: pgmap v1910: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 99 KiB/s wr, 35 op/s 2023-03-09T15:41:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:41:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2615.974174 secs 2023-03-09T15:41:45.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:41:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2615.974174 secs 2023-03-09T15:41:46.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:41:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2615.974174 secs 2023-03-09T15:41:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:41:46 smithi122 ceph-mon[93869]: pgmap v1911: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 148 KiB/s wr, 37 op/s 2023-03-09T15:41:46.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:41:46 smithi181 ceph-mon[94189]: pgmap v1911: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 148 KiB/s wr, 37 op/s 2023-03-09T15:41:47.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:41:46 smithi038 ceph-mon[92422]: pgmap v1911: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 148 KiB/s wr, 37 op/s 2023-03-09T15:41:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:41:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2618.179630 secs 2023-03-09T15:41:47.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:41:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2618.179630 secs 2023-03-09T15:41:48.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:41:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2618.179630 secs 2023-03-09T15:41:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3080.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:41:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:41:48 smithi122 ceph-mon[93869]: pgmap v1912: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 103 KiB/s wr, 10 op/s 2023-03-09T15:41:48.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:41:48 smithi181 ceph-mon[94189]: pgmap v1912: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 103 KiB/s wr, 10 op/s 2023-03-09T15:41:49.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:41:48 smithi038 ceph-mon[92422]: pgmap v1912: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 103 KiB/s wr, 10 op/s 2023-03-09T15:41:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:41:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2620.974245 secs 2023-03-09T15:41:51.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:41:51 smithi122 ceph-mon[93869]: pgmap v1913: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 106 KiB/s wr, 10 op/s 2023-03-09T15:41:51.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:41:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2620.974245 secs 2023-03-09T15:41:51.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:41:51 smithi181 ceph-mon[94189]: pgmap v1913: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 106 KiB/s wr, 10 op/s 2023-03-09T15:41:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:41:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2620.974245 secs 2023-03-09T15:41:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:41:51 smithi038 ceph-mon[92422]: pgmap v1913: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 106 KiB/s wr, 10 op/s 2023-03-09T15:41:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:41:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2623.179726 secs 2023-03-09T15:41:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:41:53 smithi122 ceph-mon[93869]: pgmap v1914: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 113 KiB/s wr, 12 op/s 2023-03-09T15:41:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:41:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2623.179726 secs 2023-03-09T15:41:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:41:53 smithi181 ceph-mon[94189]: pgmap v1914: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 113 KiB/s wr, 12 op/s 2023-03-09T15:41:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:41:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2623.179726 secs 2023-03-09T15:41:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:41:53 smithi038 ceph-mon[92422]: pgmap v1914: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 113 KiB/s wr, 12 op/s 2023-03-09T15:41:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3085.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:41:54.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:41:54 smithi122 ceph-mon[93869]: pgmap v1915: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 71 KiB/s wr, 9 op/s 2023-03-09T15:41:54.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:41:54 smithi181 ceph-mon[94189]: pgmap v1915: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 71 KiB/s wr, 9 op/s 2023-03-09T15:41:55.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:41:54 smithi038 ceph-mon[92422]: pgmap v1915: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 71 KiB/s wr, 9 op/s 2023-03-09T15:41:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:41:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2625.974302 secs 2023-03-09T15:41:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:41:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2625.974302 secs 2023-03-09T15:41:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:41:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2625.974302 secs 2023-03-09T15:41:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:41:56 smithi122 ceph-mon[93869]: pgmap v1916: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 113 KiB/s wr, 12 op/s 2023-03-09T15:41:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:41:56 smithi181 ceph-mon[94189]: pgmap v1916: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 113 KiB/s wr, 12 op/s 2023-03-09T15:41:57.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:41:56 smithi038 ceph-mon[92422]: pgmap v1916: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 113 KiB/s wr, 12 op/s 2023-03-09T15:41:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:41:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2628.179831 secs 2023-03-09T15:41:57.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:41:57 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:41:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:41:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2628.179831 secs 2023-03-09T15:41:57.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:41:57 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:41:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:41:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2628.179831 secs 2023-03-09T15:41:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:41:57 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:41:58.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3090.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:41:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:41:58 smithi122 ceph-mon[93869]: pgmap v1917: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 8 op/s 2023-03-09T15:41:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:41:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:41:58.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:41:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:41:58.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:41:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:41:58.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:41:58 smithi181 ceph-mon[94189]: pgmap v1917: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 8 op/s 2023-03-09T15:41:58.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:41:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:41:58.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:41:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:41:58.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:41:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:41:59.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:41:58 smithi038 ceph-mon[92422]: pgmap v1917: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 8 op/s 2023-03-09T15:41:59.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:41:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:41:59.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:41:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:41:59.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:41:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:42:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:42:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2630.974363 secs 2023-03-09T15:42:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:42:01 smithi122 ceph-mon[93869]: pgmap v1918: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 8 op/s 2023-03-09T15:42:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:42:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2630.974363 secs 2023-03-09T15:42:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:42:01 smithi181 ceph-mon[94189]: pgmap v1918: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 8 op/s 2023-03-09T15:42:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:42:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2630.974363 secs 2023-03-09T15:42:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:42:01 smithi038 ceph-mon[92422]: pgmap v1918: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 8 op/s 2023-03-09T15:42:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:42:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2633.179949 secs 2023-03-09T15:42:03.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:42:03 smithi122 ceph-mon[93869]: pgmap v1919: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 73 KiB/s wr, 10 op/s 2023-03-09T15:42:03.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:42:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2633.179949 secs 2023-03-09T15:42:03.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:42:03 smithi181 ceph-mon[94189]: pgmap v1919: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 73 KiB/s wr, 10 op/s 2023-03-09T15:42:03.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:42:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2633.179949 secs 2023-03-09T15:42:03.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:42:03 smithi038 ceph-mon[92422]: pgmap v1919: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 73 KiB/s wr, 10 op/s 2023-03-09T15:42:03.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3095.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:42:04.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:42:04 smithi122 ceph-mon[93869]: pgmap v1920: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 87 KiB/s wr, 19 op/s 2023-03-09T15:42:04.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:42:04 smithi181 ceph-mon[94189]: pgmap v1920: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 87 KiB/s wr, 19 op/s 2023-03-09T15:42:05.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:42:04 smithi038 ceph-mon[92422]: pgmap v1920: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 87 KiB/s wr, 19 op/s 2023-03-09T15:42:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:42:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2635.974424 secs 2023-03-09T15:42:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:42:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2635.974424 secs 2023-03-09T15:42:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:42:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2635.974424 secs 2023-03-09T15:42:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:42:06 smithi122 ceph-mon[93869]: pgmap v1921: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 111 KiB/s wr, 24 op/s 2023-03-09T15:42:06.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:42:06 smithi181 ceph-mon[94189]: pgmap v1921: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 111 KiB/s wr, 24 op/s 2023-03-09T15:42:07.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:42:06 smithi038 ceph-mon[92422]: pgmap v1921: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 111 KiB/s wr, 24 op/s 2023-03-09T15:42:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:42:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2638.180067 secs 2023-03-09T15:42:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:42:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2638.180067 secs 2023-03-09T15:42:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:42:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2638.180067 secs 2023-03-09T15:42:08.801 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3100.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:42:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:42:08 smithi122 ceph-mon[93869]: pgmap v1922: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 66 KiB/s wr, 21 op/s 2023-03-09T15:42:08.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:42:08 smithi181 ceph-mon[94189]: pgmap v1922: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 66 KiB/s wr, 21 op/s 2023-03-09T15:42:09.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:42:08 smithi038 ceph-mon[92422]: pgmap v1922: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 66 KiB/s wr, 21 op/s 2023-03-09T15:42:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:42:09 smithi122 ceph-mon[93869]: osdmap e334: 12 total, 12 up, 12 in 2023-03-09T15:42:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:42:09 smithi181 ceph-mon[94189]: osdmap e334: 12 total, 12 up, 12 in 2023-03-09T15:42:10.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:42:09 smithi038 ceph-mon[92422]: osdmap e334: 12 total, 12 up, 12 in 2023-03-09T15:42:11.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:42:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2640.974487 secs 2023-03-09T15:42:11.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:42:10 smithi038 ceph-mon[92422]: pgmap v1924: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 79 KiB/s wr, 25 op/s 2023-03-09T15:42:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:42:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2640.974487 secs 2023-03-09T15:42:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:42:10 smithi122 ceph-mon[93869]: pgmap v1924: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 79 KiB/s wr, 25 op/s 2023-03-09T15:42:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:42:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2640.974487 secs 2023-03-09T15:42:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:42:10 smithi181 ceph-mon[94189]: pgmap v1924: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 79 KiB/s wr, 25 op/s 2023-03-09T15:42:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:42:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2643.180184 secs 2023-03-09T15:42:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:42:13 smithi122 ceph-mon[93869]: pgmap v1925: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 59 KiB/s wr, 25 op/s 2023-03-09T15:42:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:42:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2643.180184 secs 2023-03-09T15:42:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:42:13 smithi181 ceph-mon[94189]: pgmap v1925: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 59 KiB/s wr, 25 op/s 2023-03-09T15:42:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:42:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2643.180184 secs 2023-03-09T15:42:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:42:13 smithi038 ceph-mon[92422]: pgmap v1925: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 59 KiB/s wr, 25 op/s 2023-03-09T15:42:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3105.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:42:14.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:42:14 smithi122 ceph-mon[93869]: pgmap v1926: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 30 KiB/s wr, 11 op/s 2023-03-09T15:42:14.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:42:14 smithi181 ceph-mon[94189]: pgmap v1926: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 30 KiB/s wr, 11 op/s 2023-03-09T15:42:15.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:42:14 smithi038 ceph-mon[92422]: pgmap v1926: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 30 KiB/s wr, 11 op/s 2023-03-09T15:42:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:42:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2645.974546 secs 2023-03-09T15:42:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:42:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2645.974546 secs 2023-03-09T15:42:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:42:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2645.974546 secs 2023-03-09T15:42:16.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:42:16 smithi181 ceph-mon[94189]: pgmap v1927: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 8 op/s 2023-03-09T15:42:17.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:42:16 smithi038 ceph-mon[92422]: pgmap v1927: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 8 op/s 2023-03-09T15:42:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:42:16 smithi122 ceph-mon[93869]: pgmap v1927: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 8 op/s 2023-03-09T15:42:17.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:42:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2648.180325 secs 2023-03-09T15:42:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:42:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2648.180325 secs 2023-03-09T15:42:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:42:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2648.180325 secs 2023-03-09T15:42:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3110.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:42:19.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:42:18 smithi038 ceph-mon[92422]: pgmap v1928: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 8 op/s 2023-03-09T15:42:19.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:42:18 smithi038 ceph-mon[92422]: osdmap e335: 12 total, 12 up, 12 in 2023-03-09T15:42:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:42:18 smithi122 ceph-mon[93869]: pgmap v1928: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 8 op/s 2023-03-09T15:42:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:42:18 smithi122 ceph-mon[93869]: osdmap e335: 12 total, 12 up, 12 in 2023-03-09T15:42:19.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:42:18 smithi181 ceph-mon[94189]: pgmap v1928: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 8 op/s 2023-03-09T15:42:19.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:42:18 smithi181 ceph-mon[94189]: osdmap e335: 12 total, 12 up, 12 in 2023-03-09T15:42:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:42:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2650.974609 secs 2023-03-09T15:42:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:42:21 smithi122 ceph-mon[93869]: pgmap v1930: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 8 op/s 2023-03-09T15:42:21.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:42:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2650.974609 secs 2023-03-09T15:42:21.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:42:21 smithi181 ceph-mon[94189]: pgmap v1930: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 8 op/s 2023-03-09T15:42:21.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:42:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2650.974609 secs 2023-03-09T15:42:21.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:42:21 smithi038 ceph-mon[92422]: pgmap v1930: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 8 op/s 2023-03-09T15:42:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:42:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2653.180441 secs 2023-03-09T15:42:23.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:42:23 smithi122 ceph-mon[93869]: pgmap v1931: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T15:42:23.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:42:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2653.180441 secs 2023-03-09T15:42:23.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:42:23 smithi181 ceph-mon[94189]: pgmap v1931: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T15:42:23.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:42:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2653.180441 secs 2023-03-09T15:42:23.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:42:23 smithi038 ceph-mon[92422]: pgmap v1931: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T15:42:23.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3115.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:42:24.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:42:24 smithi122 ceph-mon[93869]: pgmap v1932: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T15:42:24.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:42:24 smithi181 ceph-mon[94189]: pgmap v1932: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T15:42:25.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:42:24 smithi038 ceph-mon[92422]: pgmap v1932: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T15:42:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:42:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2655.974672 secs 2023-03-09T15:42:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:42:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2655.974672 secs 2023-03-09T15:42:26.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:42:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2655.974672 secs 2023-03-09T15:42:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:42:26 smithi122 ceph-mon[93869]: pgmap v1933: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T15:42:26.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:42:26 smithi181 ceph-mon[94189]: pgmap v1933: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T15:42:27.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:42:26 smithi038 ceph-mon[92422]: pgmap v1933: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T15:42:27.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:42:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2658.180572 secs 2023-03-09T15:42:28.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:42:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2658.180572 secs 2023-03-09T15:42:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:42:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2658.180572 secs 2023-03-09T15:42:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3120.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:42:29.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:42:28 smithi038 ceph-mon[92422]: pgmap v1934: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 6 op/s 2023-03-09T15:42:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:42:28 smithi122 ceph-mon[93869]: pgmap v1934: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 6 op/s 2023-03-09T15:42:29.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:42:28 smithi181 ceph-mon[94189]: pgmap v1934: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 6 op/s 2023-03-09T15:42:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:42:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2660.974733 secs 2023-03-09T15:42:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:42:31 smithi122 ceph-mon[93869]: pgmap v1935: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 6 op/s 2023-03-09T15:42:31.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:42:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2660.974733 secs 2023-03-09T15:42:31.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:42:31 smithi181 ceph-mon[94189]: pgmap v1935: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 6 op/s 2023-03-09T15:42:31.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:42:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2660.974733 secs 2023-03-09T15:42:31.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:42:31 smithi038 ceph-mon[92422]: pgmap v1935: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 6 op/s 2023-03-09T15:42:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:42:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2663.180684 secs 2023-03-09T15:42:33.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:42:33 smithi122 ceph-mon[93869]: pgmap v1936: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 40 KiB/s wr, 8 op/s 2023-03-09T15:42:33.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:42:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2663.180684 secs 2023-03-09T15:42:33.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:42:33 smithi181 ceph-mon[94189]: pgmap v1936: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 40 KiB/s wr, 8 op/s 2023-03-09T15:42:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:42:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2663.180684 secs 2023-03-09T15:42:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:42:33 smithi038 ceph-mon[92422]: pgmap v1936: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 40 KiB/s wr, 8 op/s 2023-03-09T15:42:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3125.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:42:34.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:42:34 smithi122 ceph-mon[93869]: pgmap v1937: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 40 KiB/s wr, 6 op/s 2023-03-09T15:42:34.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:42:34 smithi181 ceph-mon[94189]: pgmap v1937: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 40 KiB/s wr, 6 op/s 2023-03-09T15:42:35.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:42:34 smithi038 ceph-mon[92422]: pgmap v1937: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 40 KiB/s wr, 6 op/s 2023-03-09T15:42:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:42:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2665.974794 secs 2023-03-09T15:42:35.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:42:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2665.974794 secs 2023-03-09T15:42:36.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:42:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2665.974794 secs 2023-03-09T15:42:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:42:36 smithi122 ceph-mon[93869]: pgmap v1938: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 77 KiB/s wr, 8 op/s 2023-03-09T15:42:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:42:36 smithi181 ceph-mon[94189]: pgmap v1938: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 77 KiB/s wr, 8 op/s 2023-03-09T15:42:37.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:42:36 smithi038 ceph-mon[92422]: pgmap v1938: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 77 KiB/s wr, 8 op/s 2023-03-09T15:42:37.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:42:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2668.180816 secs 2023-03-09T15:42:38.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:42:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2668.180816 secs 2023-03-09T15:42:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:42:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2668.180816 secs 2023-03-09T15:42:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3130.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:42:39.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:42:38 smithi038 ceph-mon[92422]: pgmap v1939: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 6 op/s 2023-03-09T15:42:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:42:38 smithi122 ceph-mon[93869]: pgmap v1939: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 6 op/s 2023-03-09T15:42:39.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:42:38 smithi181 ceph-mon[94189]: pgmap v1939: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 6 op/s 2023-03-09T15:42:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:42:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2670.974851 secs 2023-03-09T15:42:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:42:41 smithi122 ceph-mon[93869]: pgmap v1940: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 62 KiB/s wr, 7 op/s 2023-03-09T15:42:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:42:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2670.974851 secs 2023-03-09T15:42:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:42:41 smithi181 ceph-mon[94189]: pgmap v1940: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 62 KiB/s wr, 7 op/s 2023-03-09T15:42:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:42:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2670.974851 secs 2023-03-09T15:42:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:42:41 smithi038 ceph-mon[92422]: pgmap v1940: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 62 KiB/s wr, 7 op/s 2023-03-09T15:42:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:42:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2673.180912 secs 2023-03-09T15:42:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:42:43 smithi122 ceph-mon[93869]: pgmap v1941: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 86 KiB/s wr, 10 op/s 2023-03-09T15:42:43.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:42:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2673.180912 secs 2023-03-09T15:42:43.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:42:43 smithi181 ceph-mon[94189]: pgmap v1941: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 86 KiB/s wr, 10 op/s 2023-03-09T15:42:43.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:42:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2673.180912 secs 2023-03-09T15:42:43.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:42:43 smithi038 ceph-mon[92422]: pgmap v1941: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 86 KiB/s wr, 10 op/s 2023-03-09T15:42:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3135.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:42:44.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:42:44 smithi122 ceph-mon[93869]: pgmap v1942: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 68 KiB/s wr, 9 op/s 2023-03-09T15:42:44.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:42:44 smithi181 ceph-mon[94189]: pgmap v1942: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 68 KiB/s wr, 9 op/s 2023-03-09T15:42:45.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:42:44 smithi038 ceph-mon[92422]: pgmap v1942: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 68 KiB/s wr, 9 op/s 2023-03-09T15:42:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:42:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2675.974915 secs 2023-03-09T15:42:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:42:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2675.974915 secs 2023-03-09T15:42:46.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:42:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2675.974915 secs 2023-03-09T15:42:46.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:42:46 smithi181 ceph-mon[94189]: pgmap v1943: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 121 KiB/s wr, 11 op/s 2023-03-09T15:42:47.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:42:46 smithi038 ceph-mon[92422]: pgmap v1943: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 121 KiB/s wr, 11 op/s 2023-03-09T15:42:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:42:46 smithi122 ceph-mon[93869]: pgmap v1943: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 121 KiB/s wr, 11 op/s 2023-03-09T15:42:48.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:42:47 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:42:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:42:47 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:42:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:42:47 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:42:48.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3140.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:42:49.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:42:48 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:42:49.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:42:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:42:49.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:42:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:42:49.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:42:48 smithi038 ceph-mon[92422]: pgmap v1944: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 84 KiB/s wr, 9 op/s 2023-03-09T15:42:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:42:48 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:42:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:42:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:42:49.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:42:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:42:49.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:42:48 smithi122 ceph-mon[93869]: pgmap v1944: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 84 KiB/s wr, 9 op/s 2023-03-09T15:42:49.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:42:48 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:42:49.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:42:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:42:49.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:42:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:42:49.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:42:48 smithi181 ceph-mon[94189]: pgmap v1944: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 84 KiB/s wr, 9 op/s 2023-03-09T15:42:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:42:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2680.974999 secs 2023-03-09T15:42:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:42:51 smithi122 ceph-mon[93869]: pgmap v1945: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 87 KiB/s wr, 10 op/s 2023-03-09T15:42:51.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:42:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2680.974999 secs 2023-03-09T15:42:51.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:42:51 smithi181 ceph-mon[94189]: pgmap v1945: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 87 KiB/s wr, 10 op/s 2023-03-09T15:42:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:42:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2680.974999 secs 2023-03-09T15:42:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:42:51 smithi038 ceph-mon[92422]: pgmap v1945: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 87 KiB/s wr, 10 op/s 2023-03-09T15:42:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:42:53 smithi122 ceph-mon[93869]: pgmap v1946: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 93 KiB/s wr, 11 op/s 2023-03-09T15:42:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:42:53 smithi181 ceph-mon[94189]: pgmap v1946: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 93 KiB/s wr, 11 op/s 2023-03-09T15:42:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:42:53 smithi038 ceph-mon[92422]: pgmap v1946: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 93 KiB/s wr, 11 op/s 2023-03-09T15:42:53.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3145.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:42:54.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:42:54 smithi122 ceph-mon[93869]: pgmap v1947: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 72 KiB/s wr, 9 op/s 2023-03-09T15:42:54.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:42:54 smithi181 ceph-mon[94189]: pgmap v1947: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 72 KiB/s wr, 9 op/s 2023-03-09T15:42:55.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:42:54 smithi038 ceph-mon[92422]: pgmap v1947: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 72 KiB/s wr, 9 op/s 2023-03-09T15:42:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:42:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2685.975060 secs 2023-03-09T15:42:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:42:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2685.975060 secs 2023-03-09T15:42:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:42:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2685.975060 secs 2023-03-09T15:42:56.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:42:56 smithi181 ceph-mon[94189]: pgmap v1948: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 142 KiB/s wr, 11 op/s 2023-03-09T15:42:57.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:42:56 smithi038 ceph-mon[92422]: pgmap v1948: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 142 KiB/s wr, 11 op/s 2023-03-09T15:42:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:42:56 smithi122 ceph-mon[93869]: pgmap v1948: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 142 KiB/s wr, 11 op/s 2023-03-09T15:42:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:42:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2688.181336 secs 2023-03-09T15:42:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:42:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2688.181336 secs 2023-03-09T15:42:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:42:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2688.181336 secs 2023-03-09T15:42:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3150.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:42:59.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:42:58 smithi038 ceph-mon[92422]: pgmap v1949: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 89 KiB/s wr, 8 op/s 2023-03-09T15:42:59.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:42:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:42:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:42:58 smithi122 ceph-mon[93869]: pgmap v1949: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 89 KiB/s wr, 8 op/s 2023-03-09T15:42:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:42:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:42:59.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:42:58 smithi181 ceph-mon[94189]: pgmap v1949: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 89 KiB/s wr, 8 op/s 2023-03-09T15:42:59.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:42:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:43:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:42:59 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:43:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:42:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:43:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:42:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:43:00.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:42:59 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:43:00.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:42:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:43:00.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:42:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:43:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:42:59 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:43:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:42:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:43:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:42:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:43:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:43:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2690.975120 secs 2023-03-09T15:43:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:43:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:43:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:43:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:43:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:43:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:43:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:43:00 smithi122 ceph-mon[93869]: pgmap v1950: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 102 KiB/s wr, 9 op/s 2023-03-09T15:43:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:43:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2690.975120 secs 2023-03-09T15:43:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:43:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:43:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:43:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:43:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:43:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:43:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:43:00 smithi181 ceph-mon[94189]: pgmap v1950: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 102 KiB/s wr, 9 op/s 2023-03-09T15:43:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:43:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2690.975120 secs 2023-03-09T15:43:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:43:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:43:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:43:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:43:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:43:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:43:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:43:00 smithi038 ceph-mon[92422]: pgmap v1950: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 102 KiB/s wr, 9 op/s 2023-03-09T15:43:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:43:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2693.181449 secs 2023-03-09T15:43:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:43:03 smithi122 ceph-mon[93869]: pgmap v1951: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 102 KiB/s wr, 11 op/s 2023-03-09T15:43:03.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:43:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2693.181449 secs 2023-03-09T15:43:03.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:43:03 smithi181 ceph-mon[94189]: pgmap v1951: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 102 KiB/s wr, 11 op/s 2023-03-09T15:43:03.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:43:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2693.181449 secs 2023-03-09T15:43:03.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:43:03 smithi038 ceph-mon[92422]: pgmap v1951: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 102 KiB/s wr, 11 op/s 2023-03-09T15:43:03.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3155.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:43:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:43:05 smithi122 ceph-mon[93869]: pgmap v1952: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 159 KiB/s wr, 22 op/s 2023-03-09T15:43:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:43:05 smithi122 ceph-mon[93869]: osdmap e336: 12 total, 12 up, 12 in 2023-03-09T15:43:05.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:43:05 smithi181 ceph-mon[94189]: pgmap v1952: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 159 KiB/s wr, 22 op/s 2023-03-09T15:43:05.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:43:05 smithi181 ceph-mon[94189]: osdmap e336: 12 total, 12 up, 12 in 2023-03-09T15:43:05.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:43:05 smithi038 ceph-mon[92422]: pgmap v1952: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 159 KiB/s wr, 22 op/s 2023-03-09T15:43:05.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:43:05 smithi038 ceph-mon[92422]: osdmap e336: 12 total, 12 up, 12 in 2023-03-09T15:43:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:43:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2695.975180 secs 2023-03-09T15:43:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:43:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2695.975180 secs 2023-03-09T15:43:06.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:43:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2695.975180 secs 2023-03-09T15:43:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:43:07 smithi122 ceph-mon[93869]: pgmap v1954: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 126 KiB/s wr, 27 op/s 2023-03-09T15:43:07.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:43:07 smithi181 ceph-mon[94189]: pgmap v1954: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 126 KiB/s wr, 27 op/s 2023-03-09T15:43:07.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:43:07 smithi038 ceph-mon[92422]: pgmap v1954: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 126 KiB/s wr, 27 op/s 2023-03-09T15:43:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:43:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2698.181563 secs 2023-03-09T15:43:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:43:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2698.181563 secs 2023-03-09T15:43:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:43:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2698.181563 secs 2023-03-09T15:43:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3160.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:43:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:43:09 smithi122 ceph-mon[93869]: pgmap v1955: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 126 KiB/s wr, 27 op/s 2023-03-09T15:43:09.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:43:09 smithi181 ceph-mon[94189]: pgmap v1955: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 126 KiB/s wr, 27 op/s 2023-03-09T15:43:09.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:43:09 smithi038 ceph-mon[92422]: pgmap v1955: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 126 KiB/s wr, 27 op/s 2023-03-09T15:43:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:43:10 smithi122 ceph-mon[93869]: pgmap v1956: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 125 KiB/s wr, 27 op/s 2023-03-09T15:43:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:43:10 smithi181 ceph-mon[94189]: pgmap v1956: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 125 KiB/s wr, 27 op/s 2023-03-09T15:43:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:43:10 smithi038 ceph-mon[92422]: pgmap v1956: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 125 KiB/s wr, 27 op/s 2023-03-09T15:43:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:43:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2700.975261 secs 2023-03-09T15:43:11.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:43:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2700.975261 secs 2023-03-09T15:43:11.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:43:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2700.975261 secs 2023-03-09T15:43:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:43:12 smithi122 ceph-mon[93869]: pgmap v1957: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 122 KiB/s wr, 26 op/s 2023-03-09T15:43:12.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:43:12 smithi181 ceph-mon[94189]: pgmap v1957: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 122 KiB/s wr, 26 op/s 2023-03-09T15:43:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:43:12 smithi038 ceph-mon[92422]: pgmap v1957: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 122 KiB/s wr, 26 op/s 2023-03-09T15:43:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:43:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2703.181685 secs 2023-03-09T15:43:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:43:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2703.181685 secs 2023-03-09T15:43:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:43:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2703.181685 secs 2023-03-09T15:43:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3165.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:43:14.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:43:14 smithi122 ceph-mon[93869]: osdmap e337: 12 total, 12 up, 12 in 2023-03-09T15:43:14.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:43:14 smithi122 ceph-mon[93869]: pgmap v1959: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 47 KiB/s wr, 11 op/s 2023-03-09T15:43:14.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:43:14 smithi181 ceph-mon[94189]: osdmap e337: 12 total, 12 up, 12 in 2023-03-09T15:43:14.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:43:14 smithi181 ceph-mon[94189]: pgmap v1959: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 47 KiB/s wr, 11 op/s 2023-03-09T15:43:15.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:43:14 smithi038 ceph-mon[92422]: osdmap e337: 12 total, 12 up, 12 in 2023-03-09T15:43:15.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:43:14 smithi038 ceph-mon[92422]: pgmap v1959: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 47 KiB/s wr, 11 op/s 2023-03-09T15:43:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:43:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2705.975322 secs 2023-03-09T15:43:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:43:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2705.975322 secs 2023-03-09T15:43:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:43:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2705.975322 secs 2023-03-09T15:43:17.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:43:16 smithi038 ceph-mon[92422]: pgmap v1960: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 21 KiB/s wr, 8 op/s 2023-03-09T15:43:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:43:16 smithi122 ceph-mon[93869]: pgmap v1960: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 21 KiB/s wr, 8 op/s 2023-03-09T15:43:17.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:43:16 smithi181 ceph-mon[94189]: pgmap v1960: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 21 KiB/s wr, 8 op/s 2023-03-09T15:43:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:43:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2708.181808 secs 2023-03-09T15:43:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:43:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2708.181808 secs 2023-03-09T15:43:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:43:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2708.181808 secs 2023-03-09T15:43:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3170.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:43:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:43:18 smithi122 ceph-mon[93869]: pgmap v1961: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 21 KiB/s wr, 8 op/s 2023-03-09T15:43:19.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:43:18 smithi181 ceph-mon[94189]: pgmap v1961: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 21 KiB/s wr, 8 op/s 2023-03-09T15:43:19.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:43:18 smithi038 ceph-mon[92422]: pgmap v1961: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 21 KiB/s wr, 8 op/s 2023-03-09T15:43:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:43:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2710.975382 secs 2023-03-09T15:43:21.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:43:21 smithi122 ceph-mon[93869]: pgmap v1962: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 4.9 KiB/s wr, 8 op/s 2023-03-09T15:43:21.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:43:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2710.975382 secs 2023-03-09T15:43:21.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:43:21 smithi181 ceph-mon[94189]: pgmap v1962: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 4.9 KiB/s wr, 8 op/s 2023-03-09T15:43:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:43:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2710.975382 secs 2023-03-09T15:43:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:43:21 smithi038 ceph-mon[92422]: pgmap v1962: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 4.9 KiB/s wr, 8 op/s 2023-03-09T15:43:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:43:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2713.181943 secs 2023-03-09T15:43:23.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:43:23 smithi122 ceph-mon[93869]: pgmap v1963: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 5.0 KiB/s wr, 8 op/s 2023-03-09T15:43:23.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:43:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2713.181943 secs 2023-03-09T15:43:23.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:43:23 smithi181 ceph-mon[94189]: pgmap v1963: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 5.0 KiB/s wr, 8 op/s 2023-03-09T15:43:23.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:43:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2713.181943 secs 2023-03-09T15:43:23.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:43:23 smithi038 ceph-mon[92422]: pgmap v1963: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 5.0 KiB/s wr, 8 op/s 2023-03-09T15:43:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3175.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:43:25.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:43:24 smithi038 ceph-mon[92422]: pgmap v1964: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 24 KiB/s wr, 9 op/s 2023-03-09T15:43:25.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:43:24 smithi122 ceph-mon[93869]: pgmap v1964: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 24 KiB/s wr, 9 op/s 2023-03-09T15:43:25.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:43:24 smithi181 ceph-mon[94189]: pgmap v1964: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 24 KiB/s wr, 9 op/s 2023-03-09T15:43:26.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:43:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2715.975443 secs 2023-03-09T15:43:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:43:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2715.975443 secs 2023-03-09T15:43:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:43:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2715.975443 secs 2023-03-09T15:43:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:43:26 smithi122 ceph-mon[93869]: pgmap v1965: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 21 KiB/s wr, 9 op/s 2023-03-09T15:43:27.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:43:26 smithi181 ceph-mon[94189]: pgmap v1965: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 21 KiB/s wr, 9 op/s 2023-03-09T15:43:27.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:43:26 smithi038 ceph-mon[92422]: pgmap v1965: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 21 KiB/s wr, 9 op/s 2023-03-09T15:43:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:43:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2718.182055 secs 2023-03-09T15:43:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:43:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2718.182055 secs 2023-03-09T15:43:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:43:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2718.182055 secs 2023-03-09T15:43:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3180.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:43:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:43:28 smithi122 ceph-mon[93869]: pgmap v1966: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 21 KiB/s wr, 8 op/s 2023-03-09T15:43:29.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:43:28 smithi181 ceph-mon[94189]: pgmap v1966: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 21 KiB/s wr, 8 op/s 2023-03-09T15:43:29.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:43:28 smithi038 ceph-mon[92422]: pgmap v1966: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 21 KiB/s wr, 8 op/s 2023-03-09T15:43:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:43:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2720.975506 secs 2023-03-09T15:43:31.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:43:31 smithi122 ceph-mon[93869]: pgmap v1967: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 42 KiB/s wr, 7 op/s 2023-03-09T15:43:31.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:43:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2720.975506 secs 2023-03-09T15:43:31.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:43:31 smithi181 ceph-mon[94189]: pgmap v1967: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 42 KiB/s wr, 7 op/s 2023-03-09T15:43:31.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:43:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2720.975506 secs 2023-03-09T15:43:31.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:43:31 smithi038 ceph-mon[92422]: pgmap v1967: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 42 KiB/s wr, 7 op/s 2023-03-09T15:43:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:43:33 smithi122 ceph-mon[93869]: pgmap v1968: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 42 KiB/s wr, 9 op/s 2023-03-09T15:43:33.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:43:33 smithi181 ceph-mon[94189]: pgmap v1968: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 42 KiB/s wr, 9 op/s 2023-03-09T15:43:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:43:33 smithi038 ceph-mon[92422]: pgmap v1968: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 42 KiB/s wr, 9 op/s 2023-03-09T15:43:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3185.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:43:35.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:43:34 smithi038 ceph-mon[92422]: pgmap v1969: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 122 KiB/s wr, 9 op/s 2023-03-09T15:43:35.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:43:34 smithi122 ceph-mon[93869]: pgmap v1969: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 122 KiB/s wr, 9 op/s 2023-03-09T15:43:35.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:43:34 smithi181 ceph-mon[94189]: pgmap v1969: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 122 KiB/s wr, 9 op/s 2023-03-09T15:43:36.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:43:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2725.975565 secs 2023-03-09T15:43:36.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:43:35 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:43:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:43:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2725.975565 secs 2023-03-09T15:43:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:43:35 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:43:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:43:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2725.975565 secs 2023-03-09T15:43:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:43:35 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:43:37.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:43:36 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:43:37.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:43:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:43:37.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:43:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:43:37.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:43:36 smithi038 ceph-mon[92422]: pgmap v1970: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 103 KiB/s wr, 10 op/s 2023-03-09T15:43:37.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:43:36 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:43:37.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:43:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:43:37.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:43:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:43:37.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:43:36 smithi122 ceph-mon[93869]: pgmap v1970: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 103 KiB/s wr, 10 op/s 2023-03-09T15:43:37.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:43:36 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:43:37.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:43:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:43:37.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:43:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:43:37.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:43:36 smithi181 ceph-mon[94189]: pgmap v1970: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 103 KiB/s wr, 10 op/s 2023-03-09T15:43:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:43:37 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 2728.182223 secs 2023-03-09T15:43:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:43:37 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 2728.182223 secs 2023-03-09T15:43:38.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:43:37 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 2728.182223 secs 2023-03-09T15:43:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3190.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:43:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:43:38 smithi122 ceph-mon[93869]: pgmap v1971: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 105 KiB/s wr, 9 op/s 2023-03-09T15:43:39.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:43:38 smithi181 ceph-mon[94189]: pgmap v1971: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 105 KiB/s wr, 9 op/s 2023-03-09T15:43:39.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:43:38 smithi038 ceph-mon[92422]: pgmap v1971: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 105 KiB/s wr, 9 op/s 2023-03-09T15:43:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:43:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2730.975650 secs 2023-03-09T15:43:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:43:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:43:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:43:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:43:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:43:40 smithi122 ceph-mon[93869]: pgmap v1972: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 148 KiB/s wr, 9 op/s 2023-03-09T15:43:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:43:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2730.975650 secs 2023-03-09T15:43:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:43:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:43:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:43:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:43:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:43:40 smithi181 ceph-mon[94189]: pgmap v1972: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 148 KiB/s wr, 9 op/s 2023-03-09T15:43:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:43:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2730.975650 secs 2023-03-09T15:43:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:43:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:43:41.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:43:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:43:41.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:43:40 smithi038 ceph-mon[92422]: pgmap v1972: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 148 KiB/s wr, 9 op/s 2023-03-09T15:43:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:43:43 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 2733.182304 secs 2023-03-09T15:43:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:43:43 smithi122 ceph-mon[93869]: pgmap v1973: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 128 KiB/s wr, 12 op/s 2023-03-09T15:43:43.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:43:43 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 2733.182304 secs 2023-03-09T15:43:43.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:43:43 smithi181 ceph-mon[94189]: pgmap v1973: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 128 KiB/s wr, 12 op/s 2023-03-09T15:43:43.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:43:43 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 2733.182304 secs 2023-03-09T15:43:43.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:43:43 smithi038 ceph-mon[92422]: pgmap v1973: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 128 KiB/s wr, 12 op/s 2023-03-09T15:43:43.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3195.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:43:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:43:44 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:43:44.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:43:44 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:43:44.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:43:44 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:43:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:43:45 smithi122 ceph-mon[93869]: pgmap v1974: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 174 KiB/s wr, 12 op/s 2023-03-09T15:43:45.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:43:45 smithi181 ceph-mon[94189]: pgmap v1974: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 174 KiB/s wr, 12 op/s 2023-03-09T15:43:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:43:45 smithi038 ceph-mon[92422]: pgmap v1974: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 174 KiB/s wr, 12 op/s 2023-03-09T15:43:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:43:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2735.975710 secs 2023-03-09T15:43:46.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:43:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2735.975710 secs 2023-03-09T15:43:46.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:43:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2735.975710 secs 2023-03-09T15:43:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:43:47 smithi122 ceph-mon[93869]: pgmap v1975: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 94 KiB/s wr, 11 op/s 2023-03-09T15:43:47.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:43:47 smithi181 ceph-mon[94189]: pgmap v1975: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 94 KiB/s wr, 11 op/s 2023-03-09T15:43:47.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:43:47 smithi038 ceph-mon[92422]: pgmap v1975: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 94 KiB/s wr, 11 op/s 2023-03-09T15:43:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:43:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2738.182435 secs 2023-03-09T15:43:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:43:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:43:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:43:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:43:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:43:48 smithi122 ceph-mon[93869]: pgmap v1976: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 97 KiB/s wr, 9 op/s 2023-03-09T15:43:48.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:43:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2738.182435 secs 2023-03-09T15:43:48.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:43:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:43:48.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:43:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:43:48.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:43:48 smithi181 ceph-mon[94189]: pgmap v1976: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 97 KiB/s wr, 9 op/s 2023-03-09T15:43:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:43:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2738.182435 secs 2023-03-09T15:43:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:43:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:43:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:43:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:43:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:43:48 smithi038 ceph-mon[92422]: pgmap v1976: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 97 KiB/s wr, 9 op/s 2023-03-09T15:43:48.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3200.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:43:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:43:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2740.975770 secs 2023-03-09T15:43:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:43:51 smithi122 ceph-mon[93869]: pgmap v1977: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 136 KiB/s wr, 9 op/s 2023-03-09T15:43:51.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:43:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2740.975770 secs 2023-03-09T15:43:51.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:43:51 smithi181 ceph-mon[94189]: pgmap v1977: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 136 KiB/s wr, 9 op/s 2023-03-09T15:43:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:43:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2740.975770 secs 2023-03-09T15:43:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:43:51 smithi038 ceph-mon[92422]: pgmap v1977: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 136 KiB/s wr, 9 op/s 2023-03-09T15:43:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:43:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2743.182530 secs 2023-03-09T15:43:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:43:53 smithi122 ceph-mon[93869]: pgmap v1978: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 93 KiB/s wr, 10 op/s 2023-03-09T15:43:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:43:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2743.182530 secs 2023-03-09T15:43:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:43:53 smithi181 ceph-mon[94189]: pgmap v1978: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 93 KiB/s wr, 10 op/s 2023-03-09T15:43:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:43:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2743.182530 secs 2023-03-09T15:43:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:43:53 smithi038 ceph-mon[92422]: pgmap v1978: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 93 KiB/s wr, 10 op/s 2023-03-09T15:43:53.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3205.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:43:55.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:43:54 smithi038 ceph-mon[92422]: pgmap v1979: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 126 KiB/s wr, 9 op/s 2023-03-09T15:43:55.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:43:54 smithi122 ceph-mon[93869]: pgmap v1979: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 126 KiB/s wr, 9 op/s 2023-03-09T15:43:55.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:43:54 smithi181 ceph-mon[94189]: pgmap v1979: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 126 KiB/s wr, 9 op/s 2023-03-09T15:43:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:43:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2745.975844 secs 2023-03-09T15:43:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:43:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2745.975844 secs 2023-03-09T15:43:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:43:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2745.975844 secs 2023-03-09T15:43:57.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:43:56 smithi038 ceph-mon[92422]: pgmap v1980: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 80 KiB/s wr, 9 op/s 2023-03-09T15:43:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:43:56 smithi122 ceph-mon[93869]: pgmap v1980: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 80 KiB/s wr, 9 op/s 2023-03-09T15:43:57.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:43:56 smithi181 ceph-mon[94189]: pgmap v1980: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 80 KiB/s wr, 9 op/s 2023-03-09T15:43:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:43:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2748.182625 secs 2023-03-09T15:43:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:43:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2748.182625 secs 2023-03-09T15:43:58.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:43:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2748.182625 secs 2023-03-09T15:43:58.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3210.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:43:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:43:58 smithi122 ceph-mon[93869]: pgmap v1981: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 86 KiB/s wr, 8 op/s 2023-03-09T15:43:59.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:43:58 smithi181 ceph-mon[94189]: pgmap v1981: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 86 KiB/s wr, 8 op/s 2023-03-09T15:43:59.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:43:58 smithi038 ceph-mon[92422]: pgmap v1981: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 86 KiB/s wr, 8 op/s 2023-03-09T15:44:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:43:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:44:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:43:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:44:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:43:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:44:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:44:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2750.975905 secs 2023-03-09T15:44:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:44:01 smithi122 ceph-mon[93869]: pgmap v1982: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 83 KiB/s wr, 8 op/s 2023-03-09T15:44:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:44:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2750.975905 secs 2023-03-09T15:44:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:44:01 smithi181 ceph-mon[94189]: pgmap v1982: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 83 KiB/s wr, 8 op/s 2023-03-09T15:44:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:44:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2750.975905 secs 2023-03-09T15:44:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:44:01 smithi038 ceph-mon[92422]: pgmap v1982: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 83 KiB/s wr, 8 op/s 2023-03-09T15:44:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:44:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2753.182771 secs 2023-03-09T15:44:03.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:44:03 smithi122 ceph-mon[93869]: pgmap v1983: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 64 KiB/s wr, 11 op/s 2023-03-09T15:44:03.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:44:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2753.182771 secs 2023-03-09T15:44:03.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:44:03 smithi181 ceph-mon[94189]: pgmap v1983: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 64 KiB/s wr, 11 op/s 2023-03-09T15:44:03.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:44:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2753.182771 secs 2023-03-09T15:44:03.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:44:03 smithi038 ceph-mon[92422]: pgmap v1983: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 64 KiB/s wr, 11 op/s 2023-03-09T15:44:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3215.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:44:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:44:05 smithi122 ceph-mon[93869]: pgmap v1984: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 88 KiB/s wr, 23 op/s 2023-03-09T15:44:05.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:44:05 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:44:05.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:44:05 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:44:05.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:44:05 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:44:05.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:44:05 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:44:05.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:44:05 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:44:05.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:44:05 smithi181 ceph-mon[94189]: pgmap v1984: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 88 KiB/s wr, 23 op/s 2023-03-09T15:44:05.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:44:05 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:44:05.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:44:05 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:44:05.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:44:05 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:44:05.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:44:05 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:44:05.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:44:05 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:44:05.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:44:05 smithi038 ceph-mon[92422]: pgmap v1984: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 88 KiB/s wr, 23 op/s 2023-03-09T15:44:05.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:44:05 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:44:05.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:44:05 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:44:05.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:44:05 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:44:05.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:44:05 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:44:05.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:44:05 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:44:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:44:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2755.975964 secs 2023-03-09T15:44:06.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:44:06 smithi122 ceph-mon[93869]: pgmap v1985: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 56 KiB/s wr, 23 op/s 2023-03-09T15:44:06.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:44:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2755.975964 secs 2023-03-09T15:44:06.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:44:06 smithi181 ceph-mon[94189]: pgmap v1985: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 56 KiB/s wr, 23 op/s 2023-03-09T15:44:06.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:44:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2755.975964 secs 2023-03-09T15:44:06.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:44:06 smithi038 ceph-mon[92422]: pgmap v1985: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 56 KiB/s wr, 23 op/s 2023-03-09T15:44:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:44:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2758.182865 secs 2023-03-09T15:44:07.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:44:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2758.182865 secs 2023-03-09T15:44:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:44:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2758.182865 secs 2023-03-09T15:44:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3220.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:44:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:44:08 smithi122 ceph-mon[93869]: pgmap v1986: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 59 KiB/s wr, 23 op/s 2023-03-09T15:44:08.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:44:08 smithi181 ceph-mon[94189]: pgmap v1986: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 59 KiB/s wr, 23 op/s 2023-03-09T15:44:09.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:44:08 smithi038 ceph-mon[92422]: pgmap v1986: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 59 KiB/s wr, 23 op/s 2023-03-09T15:44:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:44:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2760.975986 secs 2023-03-09T15:44:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:44:10 smithi122 ceph-mon[93869]: osdmap e338: 12 total, 12 up, 12 in 2023-03-09T15:44:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:44:10 smithi122 ceph-mon[93869]: pgmap v1988: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 63 KiB/s wr, 26 op/s 2023-03-09T15:44:10.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:44:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2760.975986 secs 2023-03-09T15:44:10.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:44:10 smithi181 ceph-mon[94189]: osdmap e338: 12 total, 12 up, 12 in 2023-03-09T15:44:10.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:44:10 smithi181 ceph-mon[94189]: pgmap v1988: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 63 KiB/s wr, 26 op/s 2023-03-09T15:44:11.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:44:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2760.975986 secs 2023-03-09T15:44:11.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:44:10 smithi038 ceph-mon[92422]: osdmap e338: 12 total, 12 up, 12 in 2023-03-09T15:44:11.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:44:10 smithi038 ceph-mon[92422]: pgmap v1988: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 63 KiB/s wr, 26 op/s 2023-03-09T15:44:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:44:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2763.182956 secs 2023-03-09T15:44:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:44:13 smithi122 ceph-mon[93869]: pgmap v1989: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 80 KiB/s wr, 24 op/s 2023-03-09T15:44:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:44:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2763.182956 secs 2023-03-09T15:44:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:44:13 smithi181 ceph-mon[94189]: pgmap v1989: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 80 KiB/s wr, 24 op/s 2023-03-09T15:44:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:44:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2763.182956 secs 2023-03-09T15:44:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:44:13 smithi038 ceph-mon[92422]: pgmap v1989: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 80 KiB/s wr, 24 op/s 2023-03-09T15:44:13.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3225.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:44:15.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:44:14 smithi038 ceph-mon[92422]: pgmap v1990: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 51 KiB/s wr, 10 op/s 2023-03-09T15:44:15.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:44:14 smithi122 ceph-mon[93869]: pgmap v1990: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 51 KiB/s wr, 10 op/s 2023-03-09T15:44:15.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:44:14 smithi181 ceph-mon[94189]: pgmap v1990: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 51 KiB/s wr, 10 op/s 2023-03-09T15:44:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:44:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2765.976045 secs 2023-03-09T15:44:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:44:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2765.976045 secs 2023-03-09T15:44:16.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:44:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2765.976045 secs 2023-03-09T15:44:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:44:16 smithi122 ceph-mon[93869]: pgmap v1991: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 49 KiB/s wr, 9 op/s 2023-03-09T15:44:17.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:44:16 smithi181 ceph-mon[94189]: pgmap v1991: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 49 KiB/s wr, 9 op/s 2023-03-09T15:44:17.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:44:16 smithi038 ceph-mon[92422]: pgmap v1991: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 49 KiB/s wr, 9 op/s 2023-03-09T15:44:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:44:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2768.183092 secs 2023-03-09T15:44:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:44:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2768.183092 secs 2023-03-09T15:44:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:44:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2768.183092 secs 2023-03-09T15:44:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3230.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:44:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:44:18 smithi122 ceph-mon[93869]: pgmap v1992: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 45 KiB/s wr, 9 op/s 2023-03-09T15:44:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:44:18 smithi122 ceph-mon[93869]: osdmap e339: 12 total, 12 up, 12 in 2023-03-09T15:44:19.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:44:18 smithi181 ceph-mon[94189]: pgmap v1992: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 45 KiB/s wr, 9 op/s 2023-03-09T15:44:19.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:44:18 smithi181 ceph-mon[94189]: osdmap e339: 12 total, 12 up, 12 in 2023-03-09T15:44:19.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:44:18 smithi038 ceph-mon[92422]: pgmap v1992: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 45 KiB/s wr, 9 op/s 2023-03-09T15:44:19.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:44:18 smithi038 ceph-mon[92422]: osdmap e339: 12 total, 12 up, 12 in 2023-03-09T15:44:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:44:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2770.976106 secs 2023-03-09T15:44:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:44:21 smithi122 ceph-mon[93869]: pgmap v1994: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 45 KiB/s wr, 8 op/s 2023-03-09T15:44:21.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:44:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2770.976106 secs 2023-03-09T15:44:21.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:44:21 smithi181 ceph-mon[94189]: pgmap v1994: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 45 KiB/s wr, 8 op/s 2023-03-09T15:44:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:44:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2770.976106 secs 2023-03-09T15:44:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:44:21 smithi038 ceph-mon[92422]: pgmap v1994: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 45 KiB/s wr, 8 op/s 2023-03-09T15:44:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:44:22 smithi122 ceph-mon[93869]: pgmap v1995: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 921 B/s wr, 7 op/s 2023-03-09T15:44:22.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:44:22 smithi181 ceph-mon[94189]: pgmap v1995: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 921 B/s wr, 7 op/s 2023-03-09T15:44:22.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:44:22 smithi038 ceph-mon[92422]: pgmap v1995: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 921 B/s wr, 7 op/s 2023-03-09T15:44:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:44:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2773.183194 secs 2023-03-09T15:44:23.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:44:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2773.183194 secs 2023-03-09T15:44:23.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:44:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2773.183194 secs 2023-03-09T15:44:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3235.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:44:25.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:44:24 smithi038 ceph-mon[92422]: pgmap v1996: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 818 B/s wr, 7 op/s 2023-03-09T15:44:25.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:44:24 smithi122 ceph-mon[93869]: pgmap v1996: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 818 B/s wr, 7 op/s 2023-03-09T15:44:25.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:44:24 smithi181 ceph-mon[94189]: pgmap v1996: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 818 B/s wr, 7 op/s 2023-03-09T15:44:26.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:44:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2775.976171 secs 2023-03-09T15:44:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:44:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2775.976171 secs 2023-03-09T15:44:26.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:44:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2775.976171 secs 2023-03-09T15:44:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:44:26 smithi122 ceph-mon[93869]: pgmap v1997: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 716 B/s wr, 7 op/s 2023-03-09T15:44:27.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:44:26 smithi181 ceph-mon[94189]: pgmap v1997: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 716 B/s wr, 7 op/s 2023-03-09T15:44:27.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:44:26 smithi038 ceph-mon[92422]: pgmap v1997: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 716 B/s wr, 7 op/s 2023-03-09T15:44:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:44:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2778.183292 secs 2023-03-09T15:44:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:44:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2778.183292 secs 2023-03-09T15:44:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:44:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2778.183292 secs 2023-03-09T15:44:28.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3240.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:44:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:44:28 smithi122 ceph-mon[93869]: pgmap v1998: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 6 op/s 2023-03-09T15:44:29.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:44:28 smithi181 ceph-mon[94189]: pgmap v1998: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 6 op/s 2023-03-09T15:44:29.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:44:28 smithi038 ceph-mon[92422]: pgmap v1998: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 6 op/s 2023-03-09T15:44:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:44:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2780.976231 secs 2023-03-09T15:44:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:44:31 smithi122 ceph-mon[93869]: pgmap v1999: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 6 op/s 2023-03-09T15:44:31.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:44:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2780.976231 secs 2023-03-09T15:44:31.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:44:31 smithi181 ceph-mon[94189]: pgmap v1999: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 6 op/s 2023-03-09T15:44:31.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:44:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2780.976231 secs 2023-03-09T15:44:31.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:44:31 smithi038 ceph-mon[92422]: pgmap v1999: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 6 op/s 2023-03-09T15:44:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:44:32 smithi122 ceph-mon[93869]: pgmap v2000: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 40 KiB/s wr, 8 op/s 2023-03-09T15:44:32.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:44:32 smithi181 ceph-mon[94189]: pgmap v2000: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 40 KiB/s wr, 8 op/s 2023-03-09T15:44:32.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:44:32 smithi038 ceph-mon[92422]: pgmap v2000: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 40 KiB/s wr, 8 op/s 2023-03-09T15:44:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:44:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2783.183385 secs 2023-03-09T15:44:33.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:44:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2783.183385 secs 2023-03-09T15:44:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:44:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2783.183385 secs 2023-03-09T15:44:33.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3245.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:44:35.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:44:34 smithi122 ceph-mon[93869]: pgmap v2001: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 40 KiB/s wr, 6 op/s 2023-03-09T15:44:35.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:44:34 smithi181 ceph-mon[94189]: pgmap v2001: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 40 KiB/s wr, 6 op/s 2023-03-09T15:44:35.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:44:34 smithi038 ceph-mon[92422]: pgmap v2001: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 40 KiB/s wr, 6 op/s 2023-03-09T15:44:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:44:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2785.976290 secs 2023-03-09T15:44:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:44:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2785.976290 secs 2023-03-09T15:44:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:44:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2785.976290 secs 2023-03-09T15:44:37.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:44:36 smithi122 ceph-mon[93869]: pgmap v2002: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 40 KiB/s wr, 7 op/s 2023-03-09T15:44:37.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:44:36 smithi181 ceph-mon[94189]: pgmap v2002: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 40 KiB/s wr, 7 op/s 2023-03-09T15:44:37.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:44:36 smithi038 ceph-mon[92422]: pgmap v2002: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 40 KiB/s wr, 7 op/s 2023-03-09T15:44:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:44:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2788.183491 secs 2023-03-09T15:44:38.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:44:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2788.183491 secs 2023-03-09T15:44:38.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:44:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2788.183491 secs 2023-03-09T15:44:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3250.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:44:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:44:38 smithi122 ceph-mon[93869]: pgmap v2003: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 81 KiB/s wr, 8 op/s 2023-03-09T15:44:39.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:44:38 smithi181 ceph-mon[94189]: pgmap v2003: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 81 KiB/s wr, 8 op/s 2023-03-09T15:44:39.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:44:38 smithi038 ceph-mon[92422]: pgmap v2003: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 81 KiB/s wr, 8 op/s 2023-03-09T15:44:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:44:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2790.976373 secs 2023-03-09T15:44:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:44:41 smithi122 ceph-mon[93869]: pgmap v2004: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 63 KiB/s wr, 7 op/s 2023-03-09T15:44:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:44:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2790.976373 secs 2023-03-09T15:44:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:44:41 smithi181 ceph-mon[94189]: pgmap v2004: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 63 KiB/s wr, 7 op/s 2023-03-09T15:44:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:44:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2790.976373 secs 2023-03-09T15:44:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:44:41 smithi038 ceph-mon[92422]: pgmap v2004: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 63 KiB/s wr, 7 op/s 2023-03-09T15:44:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:44:42 smithi122 ceph-mon[93869]: pgmap v2005: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 119 KiB/s wr, 10 op/s 2023-03-09T15:44:42.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:44:42 smithi181 ceph-mon[94189]: pgmap v2005: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 119 KiB/s wr, 10 op/s 2023-03-09T15:44:42.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:44:42 smithi038 ceph-mon[92422]: pgmap v2005: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 119 KiB/s wr, 10 op/s 2023-03-09T15:44:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:44:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2793.183607 secs 2023-03-09T15:44:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:44:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2793.183607 secs 2023-03-09T15:44:43.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:44:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2793.183607 secs 2023-03-09T15:44:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3255.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:44:45.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:44:44 smithi122 ceph-mon[93869]: pgmap v2006: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 100 KiB/s wr, 9 op/s 2023-03-09T15:44:45.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:44:44 smithi181 ceph-mon[94189]: pgmap v2006: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 100 KiB/s wr, 9 op/s 2023-03-09T15:44:45.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:44:44 smithi038 ceph-mon[92422]: pgmap v2006: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 100 KiB/s wr, 9 op/s 2023-03-09T15:44:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:44:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2795.976433 secs 2023-03-09T15:44:46.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:44:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2795.976433 secs 2023-03-09T15:44:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:44:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2795.976433 secs 2023-03-09T15:44:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:44:46 smithi122 ceph-mon[93869]: pgmap v2007: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 100 KiB/s wr, 10 op/s 2023-03-09T15:44:47.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:44:46 smithi181 ceph-mon[94189]: pgmap v2007: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 100 KiB/s wr, 10 op/s 2023-03-09T15:44:47.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:44:46 smithi038 ceph-mon[92422]: pgmap v2007: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 100 KiB/s wr, 10 op/s 2023-03-09T15:44:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:44:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2798.183700 secs 2023-03-09T15:44:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:44:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2798.183700 secs 2023-03-09T15:44:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:44:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2798.183700 secs 2023-03-09T15:44:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3260.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:44:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:44:48 smithi122 ceph-mon[93869]: pgmap v2008: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 129 KiB/s wr, 11 op/s 2023-03-09T15:44:49.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:44:48 smithi181 ceph-mon[94189]: pgmap v2008: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 129 KiB/s wr, 11 op/s 2023-03-09T15:44:49.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:44:48 smithi038 ceph-mon[92422]: pgmap v2008: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 129 KiB/s wr, 11 op/s 2023-03-09T15:44:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:44:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2800.976492 secs 2023-03-09T15:44:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:44:51 smithi122 ceph-mon[93869]: pgmap v2009: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 88 KiB/s wr, 9 op/s 2023-03-09T15:44:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:44:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2800.976492 secs 2023-03-09T15:44:51.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:44:51 smithi181 ceph-mon[94189]: pgmap v2009: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 88 KiB/s wr, 9 op/s 2023-03-09T15:44:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:44:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2800.976492 secs 2023-03-09T15:44:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:44:51 smithi038 ceph-mon[92422]: pgmap v2009: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 88 KiB/s wr, 9 op/s 2023-03-09T15:44:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:44:52 smithi122 ceph-mon[93869]: pgmap v2010: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 159 KiB/s wr, 13 op/s 2023-03-09T15:44:52.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:44:52 smithi181 ceph-mon[94189]: pgmap v2010: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 159 KiB/s wr, 13 op/s 2023-03-09T15:44:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:44:52 smithi038 ceph-mon[92422]: pgmap v2010: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 159 KiB/s wr, 13 op/s 2023-03-09T15:44:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:44:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2803.183837 secs 2023-03-09T15:44:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:44:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2803.183837 secs 2023-03-09T15:44:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:44:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2803.183837 secs 2023-03-09T15:44:53.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3265.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:44:55.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:44:54 smithi122 ceph-mon[93869]: pgmap v2011: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 110 KiB/s wr, 11 op/s 2023-03-09T15:44:55.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:44:54 smithi181 ceph-mon[94189]: pgmap v2011: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 110 KiB/s wr, 11 op/s 2023-03-09T15:44:55.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:44:54 smithi038 ceph-mon[92422]: pgmap v2011: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 110 KiB/s wr, 11 op/s 2023-03-09T15:44:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:44:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2805.976563 secs 2023-03-09T15:44:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:44:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2805.976563 secs 2023-03-09T15:44:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:44:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2805.976563 secs 2023-03-09T15:44:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:44:56 smithi122 ceph-mon[93869]: pgmap v2012: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 106 KiB/s wr, 11 op/s 2023-03-09T15:44:57.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:44:56 smithi181 ceph-mon[94189]: pgmap v2012: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 106 KiB/s wr, 11 op/s 2023-03-09T15:44:57.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:44:56 smithi038 ceph-mon[92422]: pgmap v2012: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 106 KiB/s wr, 11 op/s 2023-03-09T15:44:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:44:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2808.183938 secs 2023-03-09T15:44:58.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:44:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2808.183938 secs 2023-03-09T15:44:58.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:44:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2808.183938 secs 2023-03-09T15:44:58.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3270.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:44:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:44:58 smithi122 ceph-mon[93869]: pgmap v2013: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 126 KiB/s wr, 11 op/s 2023-03-09T15:44:59.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:44:58 smithi181 ceph-mon[94189]: pgmap v2013: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 126 KiB/s wr, 11 op/s 2023-03-09T15:44:59.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:44:58 smithi038 ceph-mon[92422]: pgmap v2013: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 126 KiB/s wr, 11 op/s 2023-03-09T15:45:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:45:00 smithi122 ceph-mon[93869]: pgmap v2014: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 98 KiB/s wr, 9 op/s 2023-03-09T15:45:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:45:00 smithi181 ceph-mon[94189]: pgmap v2014: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 98 KiB/s wr, 9 op/s 2023-03-09T15:45:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:45:00 smithi038 ceph-mon[92422]: pgmap v2014: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 98 KiB/s wr, 9 op/s 2023-03-09T15:45:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:45:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2810.976623 secs 2023-03-09T15:45:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:45:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2810.976623 secs 2023-03-09T15:45:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:45:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2810.976623 secs 2023-03-09T15:45:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:45:02 smithi122 ceph-mon[93869]: pgmap v2015: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 134 KiB/s wr, 14 op/s 2023-03-09T15:45:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:45:02 smithi181 ceph-mon[94189]: pgmap v2015: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 134 KiB/s wr, 14 op/s 2023-03-09T15:45:02.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:45:02 smithi038 ceph-mon[92422]: pgmap v2015: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 134 KiB/s wr, 14 op/s 2023-03-09T15:45:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:45:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2813.184071 secs 2023-03-09T15:45:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:45:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2813.184071 secs 2023-03-09T15:45:03.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:45:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2813.184071 secs 2023-03-09T15:45:03.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3275.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:45:05.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:45:04 smithi122 ceph-mon[93869]: pgmap v2016: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 87 KiB/s wr, 23 op/s 2023-03-09T15:45:05.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:45:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:45:05.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:45:04 smithi181 ceph-mon[94189]: pgmap v2016: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 87 KiB/s wr, 23 op/s 2023-03-09T15:45:05.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:45:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:45:05.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:45:04 smithi038 ceph-mon[92422]: pgmap v2016: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 87 KiB/s wr, 23 op/s 2023-03-09T15:45:05.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:45:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:45:06.017 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:45:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2815.976682 secs 2023-03-09T15:45:06.147 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:45:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2815.976682 secs 2023-03-09T15:45:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:45:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2815.976682 secs 2023-03-09T15:45:06.816 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:45:06 smithi181 ceph-mon[94189]: pgmap v2017: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 81 KiB/s wr, 23 op/s 2023-03-09T15:45:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:45:06 smithi122 ceph-mon[93869]: pgmap v2017: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 81 KiB/s wr, 23 op/s 2023-03-09T15:45:07.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:45:06 smithi038 ceph-mon[92422]: pgmap v2017: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 81 KiB/s wr, 23 op/s 2023-03-09T15:45:07.901 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:45:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2818.184147 secs 2023-03-09T15:45:08.147 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:45:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2818.184147 secs 2023-03-09T15:45:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:45:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2818.184147 secs 2023-03-09T15:45:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3280.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:45:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:45:08 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:45:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:45:08 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:45:09.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:45:08 smithi122 ceph-mon[93869]: pgmap v2018: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 81 KiB/s wr, 23 op/s 2023-03-09T15:45:09.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:45:08 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:45:09.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:45:08 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:45:09.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:45:08 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:45:09.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:45:08 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:45:09.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:45:08 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:45:09.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:45:08 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:45:09.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:45:08 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:45:09.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:45:08 smithi181 ceph-mon[94189]: pgmap v2018: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 81 KiB/s wr, 23 op/s 2023-03-09T15:45:09.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:45:08 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:45:09.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:45:08 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:45:09.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:45:08 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:45:09.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:45:08 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:45:09.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:45:08 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:45:09.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:45:08 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:45:09.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:45:08 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:45:09.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:45:08 smithi038 ceph-mon[92422]: pgmap v2018: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 81 KiB/s wr, 23 op/s 2023-03-09T15:45:09.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:45:08 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:45:09.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:45:08 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:45:09.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:45:08 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:45:09.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:45:08 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:45:09.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:45:08 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:45:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:45:10 smithi122 ceph-mon[93869]: osdmap e340: 12 total, 12 up, 12 in 2023-03-09T15:45:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:45:10 smithi122 ceph-mon[93869]: pgmap v2020: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 74 KiB/s wr, 25 op/s 2023-03-09T15:45:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:45:10 smithi181 ceph-mon[94189]: osdmap e340: 12 total, 12 up, 12 in 2023-03-09T15:45:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:45:10 smithi181 ceph-mon[94189]: pgmap v2020: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 74 KiB/s wr, 25 op/s 2023-03-09T15:45:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:45:10 smithi038 ceph-mon[92422]: osdmap e340: 12 total, 12 up, 12 in 2023-03-09T15:45:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:45:10 smithi038 ceph-mon[92422]: pgmap v2020: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 74 KiB/s wr, 25 op/s 2023-03-09T15:45:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:45:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2820.976757 secs 2023-03-09T15:45:11.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:45:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2820.976757 secs 2023-03-09T15:45:11.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:45:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2820.976757 secs 2023-03-09T15:45:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:45:12 smithi122 ceph-mon[93869]: pgmap v2021: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 30 KiB/s wr, 21 op/s 2023-03-09T15:45:12.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:45:12 smithi181 ceph-mon[94189]: pgmap v2021: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 30 KiB/s wr, 21 op/s 2023-03-09T15:45:12.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:45:12 smithi038 ceph-mon[92422]: pgmap v2021: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 30 KiB/s wr, 21 op/s 2023-03-09T15:45:13.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:45:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2823.184236 secs 2023-03-09T15:45:13.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:45:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2823.184236 secs 2023-03-09T15:45:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3285.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:45:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:45:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2823.184236 secs 2023-03-09T15:45:15.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:45:14 smithi122 ceph-mon[93869]: pgmap v2022: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 614 B/s wr, 6 op/s 2023-03-09T15:45:15.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:45:14 smithi181 ceph-mon[94189]: pgmap v2022: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 614 B/s wr, 6 op/s 2023-03-09T15:45:15.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:45:14 smithi038 ceph-mon[92422]: pgmap v2022: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 614 B/s wr, 6 op/s 2023-03-09T15:45:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:45:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2825.976814 secs 2023-03-09T15:45:16.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:45:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2825.976814 secs 2023-03-09T15:45:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:45:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2825.976814 secs 2023-03-09T15:45:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:45:16 smithi122 ceph-mon[93869]: pgmap v2023: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2023-03-09T15:45:17.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:45:16 smithi181 ceph-mon[94189]: pgmap v2023: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2023-03-09T15:45:17.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:45:16 smithi038 ceph-mon[92422]: pgmap v2023: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2023-03-09T15:45:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:45:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2828.184363 secs 2023-03-09T15:45:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:45:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2828.184363 secs 2023-03-09T15:45:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:45:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2828.184363 secs 2023-03-09T15:45:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3290.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:45:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:45:18 smithi122 ceph-mon[93869]: pgmap v2024: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2023-03-09T15:45:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:45:18 smithi122 ceph-mon[93869]: osdmap e341: 12 total, 12 up, 12 in 2023-03-09T15:45:19.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:45:18 smithi181 ceph-mon[94189]: pgmap v2024: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2023-03-09T15:45:19.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:45:18 smithi181 ceph-mon[94189]: osdmap e341: 12 total, 12 up, 12 in 2023-03-09T15:45:19.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:45:18 smithi038 ceph-mon[92422]: pgmap v2024: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2023-03-09T15:45:19.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:45:18 smithi038 ceph-mon[92422]: osdmap e341: 12 total, 12 up, 12 in 2023-03-09T15:45:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:45:20 smithi122 ceph-mon[93869]: pgmap v2026: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2023-03-09T15:45:20.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:45:20 smithi181 ceph-mon[94189]: pgmap v2026: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2023-03-09T15:45:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:45:20 smithi038 ceph-mon[92422]: pgmap v2026: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2023-03-09T15:45:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:45:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2830.976872 secs 2023-03-09T15:45:21.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:45:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2830.976872 secs 2023-03-09T15:45:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:45:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2830.976872 secs 2023-03-09T15:45:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:45:22 smithi122 ceph-mon[93869]: pgmap v2027: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 25 KiB/s wr, 7 op/s 2023-03-09T15:45:22.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:45:22 smithi181 ceph-mon[94189]: pgmap v2027: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 25 KiB/s wr, 7 op/s 2023-03-09T15:45:22.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:45:22 smithi038 ceph-mon[92422]: pgmap v2027: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 25 KiB/s wr, 7 op/s 2023-03-09T15:45:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:45:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2833.184479 secs 2023-03-09T15:45:23.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:45:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2833.184479 secs 2023-03-09T15:45:23.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:45:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2833.184479 secs 2023-03-09T15:45:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3295.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:45:25.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:45:24 smithi122 ceph-mon[93869]: pgmap v2028: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 25 KiB/s wr, 7 op/s 2023-03-09T15:45:25.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:45:24 smithi181 ceph-mon[94189]: pgmap v2028: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 25 KiB/s wr, 7 op/s 2023-03-09T15:45:25.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:45:24 smithi038 ceph-mon[92422]: pgmap v2028: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 25 KiB/s wr, 7 op/s 2023-03-09T15:45:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:45:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2835.976934 secs 2023-03-09T15:45:26.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:45:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2835.976934 secs 2023-03-09T15:45:26.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:45:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2835.976934 secs 2023-03-09T15:45:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:45:26 smithi122 ceph-mon[93869]: pgmap v2029: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 25 KiB/s wr, 8 op/s 2023-03-09T15:45:27.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:45:26 smithi181 ceph-mon[94189]: pgmap v2029: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 25 KiB/s wr, 8 op/s 2023-03-09T15:45:27.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:45:26 smithi038 ceph-mon[92422]: pgmap v2029: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 25 KiB/s wr, 8 op/s 2023-03-09T15:45:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:45:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2838.184583 secs 2023-03-09T15:45:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:45:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2838.184583 secs 2023-03-09T15:45:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:45:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2838.184583 secs 2023-03-09T15:45:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3300.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:45:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:45:28 smithi122 ceph-mon[93869]: pgmap v2030: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 8 op/s 2023-03-09T15:45:29.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:45:28 smithi181 ceph-mon[94189]: pgmap v2030: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 8 op/s 2023-03-09T15:45:29.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:45:28 smithi038 ceph-mon[92422]: pgmap v2030: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 8 op/s 2023-03-09T15:45:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:45:30 smithi122 ceph-mon[93869]: pgmap v2031: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 7 op/s 2023-03-09T15:45:30.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:45:30 smithi181 ceph-mon[94189]: pgmap v2031: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 7 op/s 2023-03-09T15:45:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:45:30 smithi038 ceph-mon[92422]: pgmap v2031: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 7 op/s 2023-03-09T15:45:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:45:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2840.977004 secs 2023-03-09T15:45:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:45:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2840.977004 secs 2023-03-09T15:45:31.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:45:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2840.977004 secs 2023-03-09T15:45:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:45:32 smithi122 ceph-mon[93869]: pgmap v2032: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 72 KiB/s wr, 10 op/s 2023-03-09T15:45:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:45:32 smithi181 ceph-mon[94189]: pgmap v2032: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 72 KiB/s wr, 10 op/s 2023-03-09T15:45:32.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:45:32 smithi038 ceph-mon[92422]: pgmap v2032: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 72 KiB/s wr, 10 op/s 2023-03-09T15:45:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3305.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:45:35.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:45:34 smithi122 ceph-mon[93869]: pgmap v2033: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 59 KiB/s wr, 8 op/s 2023-03-09T15:45:35.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:45:34 smithi181 ceph-mon[94189]: pgmap v2033: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 59 KiB/s wr, 8 op/s 2023-03-09T15:45:35.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:45:34 smithi038 ceph-mon[92422]: pgmap v2033: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 59 KiB/s wr, 8 op/s 2023-03-09T15:45:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:45:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2845.977064 secs 2023-03-09T15:45:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:45:35 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:45:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:45:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2845.977064 secs 2023-03-09T15:45:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:45:35 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:45:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:45:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2845.977064 secs 2023-03-09T15:45:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:45:35 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:45:37.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:45:36 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:45:37.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:45:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:45:37.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:45:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:45:37.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:45:36 smithi122 ceph-mon[93869]: pgmap v2034: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 106 KiB/s wr, 35 op/s 2023-03-09T15:45:37.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:45:36 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:45:37.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:45:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:45:37.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:45:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:45:37.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:45:36 smithi181 ceph-mon[94189]: pgmap v2034: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 106 KiB/s wr, 35 op/s 2023-03-09T15:45:37.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:45:36 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:45:37.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:45:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:45:37.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:45:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:45:37.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:45:36 smithi038 ceph-mon[92422]: pgmap v2034: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 106 KiB/s wr, 35 op/s 2023-03-09T15:45:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:45:38 smithi122 ceph-mon[93869]: pgmap v2035: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 109 KiB/s wr, 34 op/s 2023-03-09T15:45:38.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:45:38 smithi181 ceph-mon[94189]: pgmap v2035: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 109 KiB/s wr, 34 op/s 2023-03-09T15:45:38.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:45:38 smithi038 ceph-mon[92422]: pgmap v2035: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 109 KiB/s wr, 34 op/s 2023-03-09T15:45:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3310.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:45:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:45:40 smithi122 ceph-mon[93869]: pgmap v2036: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 109 KiB/s wr, 33 op/s 2023-03-09T15:45:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:45:40 smithi181 ceph-mon[94189]: pgmap v2036: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 109 KiB/s wr, 33 op/s 2023-03-09T15:45:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:45:40 smithi038 ceph-mon[92422]: pgmap v2036: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 109 KiB/s wr, 33 op/s 2023-03-09T15:45:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:45:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2850.977134 secs 2023-03-09T15:45:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:45:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2850.977134 secs 2023-03-09T15:45:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:45:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2850.977134 secs 2023-03-09T15:45:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:45:42 smithi122 ceph-mon[93869]: pgmap v2037: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 154 KiB/s wr, 37 op/s 2023-03-09T15:45:42.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:45:42 smithi181 ceph-mon[94189]: pgmap v2037: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 154 KiB/s wr, 37 op/s 2023-03-09T15:45:42.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:45:42 smithi038 ceph-mon[92422]: pgmap v2037: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 154 KiB/s wr, 37 op/s 2023-03-09T15:45:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3315.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:45:45.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:45:44 smithi122 ceph-mon[93869]: pgmap v2038: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 106 KiB/s wr, 34 op/s 2023-03-09T15:45:45.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:45:44 smithi181 ceph-mon[94189]: pgmap v2038: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 106 KiB/s wr, 34 op/s 2023-03-09T15:45:45.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:45:44 smithi038 ceph-mon[92422]: pgmap v2038: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 106 KiB/s wr, 34 op/s 2023-03-09T15:45:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:45:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2855.977195 secs 2023-03-09T15:45:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:45:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2855.977195 secs 2023-03-09T15:45:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:45:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2855.977195 secs 2023-03-09T15:45:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:45:46 smithi122 ceph-mon[93869]: pgmap v2039: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 157 KiB/s wr, 36 op/s 2023-03-09T15:45:47.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:45:46 smithi181 ceph-mon[94189]: pgmap v2039: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 157 KiB/s wr, 36 op/s 2023-03-09T15:45:47.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:45:46 smithi038 ceph-mon[92422]: pgmap v2039: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 157 KiB/s wr, 36 op/s 2023-03-09T15:45:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:45:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2858.185091 secs 2023-03-09T15:45:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:45:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2858.185091 secs 2023-03-09T15:45:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:45:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2858.185091 secs 2023-03-09T15:45:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3320.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:45:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:45:48 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:45:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:45:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:45:49.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:45:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:45:49.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:45:48 smithi122 ceph-mon[93869]: pgmap v2040: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 113 KiB/s wr, 10 op/s 2023-03-09T15:45:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:45:48 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:45:49.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:45:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:45:49.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:45:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:45:49.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:45:48 smithi181 ceph-mon[94189]: pgmap v2040: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 113 KiB/s wr, 10 op/s 2023-03-09T15:45:49.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:45:48 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:45:49.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:45:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:45:49.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:45:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:45:49.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:45:48 smithi038 ceph-mon[92422]: pgmap v2040: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 113 KiB/s wr, 10 op/s 2023-03-09T15:45:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:45:50 smithi122 ceph-mon[93869]: pgmap v2041: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 109 KiB/s wr, 8 op/s 2023-03-09T15:45:50.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:45:50 smithi181 ceph-mon[94189]: pgmap v2041: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 109 KiB/s wr, 8 op/s 2023-03-09T15:45:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:45:50 smithi038 ceph-mon[92422]: pgmap v2041: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 109 KiB/s wr, 8 op/s 2023-03-09T15:45:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:45:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2860.977255 secs 2023-03-09T15:45:51.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:45:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2860.977255 secs 2023-03-09T15:45:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:45:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2860.977255 secs 2023-03-09T15:45:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:45:52 smithi122 ceph-mon[93869]: pgmap v2042: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 120 KiB/s wr, 11 op/s 2023-03-09T15:45:52.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:45:52 smithi181 ceph-mon[94189]: pgmap v2042: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 120 KiB/s wr, 11 op/s 2023-03-09T15:45:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:45:52 smithi038 ceph-mon[92422]: pgmap v2042: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 120 KiB/s wr, 11 op/s 2023-03-09T15:45:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:45:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2863.185208 secs 2023-03-09T15:45:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:45:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2863.185208 secs 2023-03-09T15:45:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:45:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2863.185208 secs 2023-03-09T15:45:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3325.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:45:55.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:45:54 smithi122 ceph-mon[93869]: pgmap v2043: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 75 KiB/s wr, 8 op/s 2023-03-09T15:45:55.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:45:54 smithi181 ceph-mon[94189]: pgmap v2043: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 75 KiB/s wr, 8 op/s 2023-03-09T15:45:55.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:45:54 smithi038 ceph-mon[92422]: pgmap v2043: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 75 KiB/s wr, 8 op/s 2023-03-09T15:45:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:45:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2865.977339 secs 2023-03-09T15:45:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:45:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2865.977339 secs 2023-03-09T15:45:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:45:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2865.977339 secs 2023-03-09T15:45:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:45:56 smithi122 ceph-mon[93869]: pgmap v2044: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 117 KiB/s wr, 10 op/s 2023-03-09T15:45:57.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:45:56 smithi181 ceph-mon[94189]: pgmap v2044: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 117 KiB/s wr, 10 op/s 2023-03-09T15:45:57.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:45:56 smithi038 ceph-mon[92422]: pgmap v2044: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 117 KiB/s wr, 10 op/s 2023-03-09T15:45:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:45:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2868.185312 secs 2023-03-09T15:45:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:45:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2868.185312 secs 2023-03-09T15:45:58.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:45:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2868.185312 secs 2023-03-09T15:45:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3330.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:45:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:45:58 smithi122 ceph-mon[93869]: pgmap v2045: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 60 KiB/s wr, 8 op/s 2023-03-09T15:45:59.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:45:58 smithi181 ceph-mon[94189]: pgmap v2045: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 60 KiB/s wr, 8 op/s 2023-03-09T15:45:59.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:45:58 smithi038 ceph-mon[92422]: pgmap v2045: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 60 KiB/s wr, 8 op/s 2023-03-09T15:46:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:46:00 smithi122 ceph-mon[93869]: pgmap v2046: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 56 KiB/s wr, 7 op/s 2023-03-09T15:46:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:46:00 smithi181 ceph-mon[94189]: pgmap v2046: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 56 KiB/s wr, 7 op/s 2023-03-09T15:46:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:46:00 smithi038 ceph-mon[92422]: pgmap v2046: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 56 KiB/s wr, 7 op/s 2023-03-09T15:46:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:46:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2870.977398 secs 2023-03-09T15:46:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:46:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2870.977398 secs 2023-03-09T15:46:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:46:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2870.977398 secs 2023-03-09T15:46:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:46:02 smithi122 ceph-mon[93869]: pgmap v2047: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 97 KiB/s wr, 12 op/s 2023-03-09T15:46:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:46:02 smithi181 ceph-mon[94189]: pgmap v2047: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 97 KiB/s wr, 12 op/s 2023-03-09T15:46:02.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:46:02 smithi038 ceph-mon[92422]: pgmap v2047: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 97 KiB/s wr, 12 op/s 2023-03-09T15:46:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:46:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2873.185415 secs 2023-03-09T15:46:03.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:46:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2873.185415 secs 2023-03-09T15:46:03.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:46:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2873.185415 secs 2023-03-09T15:46:03.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3335.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:46:05.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:46:04 smithi122 ceph-mon[93869]: pgmap v2048: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 110 KiB/s wr, 22 op/s 2023-03-09T15:46:05.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:46:04 smithi181 ceph-mon[94189]: pgmap v2048: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 110 KiB/s wr, 22 op/s 2023-03-09T15:46:05.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:46:04 smithi038 ceph-mon[92422]: pgmap v2048: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 110 KiB/s wr, 22 op/s 2023-03-09T15:46:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:46:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2875.977460 secs 2023-03-09T15:46:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:46:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2875.977460 secs 2023-03-09T15:46:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:46:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2875.977460 secs 2023-03-09T15:46:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:46:06 smithi122 ceph-mon[93869]: pgmap v2049: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 129 KiB/s wr, 24 op/s 2023-03-09T15:46:07.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:46:06 smithi181 ceph-mon[94189]: pgmap v2049: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 129 KiB/s wr, 24 op/s 2023-03-09T15:46:07.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:46:06 smithi038 ceph-mon[92422]: pgmap v2049: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 129 KiB/s wr, 24 op/s 2023-03-09T15:46:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:46:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2878.185533 secs 2023-03-09T15:46:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:46:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2878.185533 secs 2023-03-09T15:46:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:46:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2878.185533 secs 2023-03-09T15:46:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3340.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:46:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:46:08 smithi122 ceph-mon[93869]: pgmap v2050: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 85 KiB/s wr, 22 op/s 2023-03-09T15:46:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:46:08 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:46:09.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:46:08 smithi181 ceph-mon[94189]: pgmap v2050: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 85 KiB/s wr, 22 op/s 2023-03-09T15:46:09.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:46:08 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:46:09.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:46:08 smithi038 ceph-mon[92422]: pgmap v2050: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 85 KiB/s wr, 22 op/s 2023-03-09T15:46:09.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:46:08 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:46:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:46:09 smithi122 ceph-mon[93869]: osdmap e342: 12 total, 12 up, 12 in 2023-03-09T15:46:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:46:09 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:46:10.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:46:09 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:46:10.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:46:09 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:46:10.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:46:09 smithi181 ceph-mon[94189]: osdmap e342: 12 total, 12 up, 12 in 2023-03-09T15:46:10.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:46:09 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:46:10.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:46:09 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:46:10.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:46:09 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:46:10.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:46:09 smithi038 ceph-mon[92422]: osdmap e342: 12 total, 12 up, 12 in 2023-03-09T15:46:10.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:46:09 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:46:10.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:46:09 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:46:10.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:46:09 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:46:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:46:10 smithi122 ceph-mon[93869]: 2 slow requests, 2 included below; oldest blocked for > 2880.977519 secs 2023-03-09T15:46:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:46:10 smithi122 ceph-mon[93869]: slow request 2880.977519 seconds old, received at 2023-03-09T14:58:08.564667+0000: peer_request:mds.2:599 currently acquired locks 2023-03-09T15:46:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:46:10 smithi122 ceph-mon[93869]: slow request 2880.966818 seconds old, received at 2023-03-09T14:58:08.575367+0000: peer_request(client.15375:6773.0 authpin) currently dispatched 2023-03-09T15:46:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:46:10 smithi122 ceph-mon[93869]: pgmap v2052: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 101 KiB/s wr, 26 op/s 2023-03-09T15:46:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:46:10 smithi181 ceph-mon[94189]: 2 slow requests, 2 included below; oldest blocked for > 2880.977519 secs 2023-03-09T15:46:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:46:10 smithi181 ceph-mon[94189]: slow request 2880.977519 seconds old, received at 2023-03-09T14:58:08.564667+0000: peer_request:mds.2:599 currently acquired locks 2023-03-09T15:46:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:46:10 smithi181 ceph-mon[94189]: slow request 2880.966818 seconds old, received at 2023-03-09T14:58:08.575367+0000: peer_request(client.15375:6773.0 authpin) currently dispatched 2023-03-09T15:46:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:46:10 smithi181 ceph-mon[94189]: pgmap v2052: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 101 KiB/s wr, 26 op/s 2023-03-09T15:46:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:46:10 smithi038 ceph-mon[92422]: 2 slow requests, 2 included below; oldest blocked for > 2880.977519 secs 2023-03-09T15:46:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:46:10 smithi038 ceph-mon[92422]: slow request 2880.977519 seconds old, received at 2023-03-09T14:58:08.564667+0000: peer_request:mds.2:599 currently acquired locks 2023-03-09T15:46:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:46:10 smithi038 ceph-mon[92422]: slow request 2880.966818 seconds old, received at 2023-03-09T14:58:08.575367+0000: peer_request(client.15375:6773.0 authpin) currently dispatched 2023-03-09T15:46:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:46:10 smithi038 ceph-mon[92422]: pgmap v2052: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 101 KiB/s wr, 26 op/s 2023-03-09T15:46:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:46:12 smithi122 ceph-mon[93869]: pgmap v2053: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 53 KiB/s wr, 22 op/s 2023-03-09T15:46:12.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:46:12 smithi181 ceph-mon[94189]: pgmap v2053: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 53 KiB/s wr, 22 op/s 2023-03-09T15:46:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:46:12 smithi038 ceph-mon[92422]: pgmap v2053: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 53 KiB/s wr, 22 op/s 2023-03-09T15:46:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:46:13 smithi122 ceph-mon[93869]: 2 slow requests, 2 included below; oldest blocked for > 2883.185631 secs 2023-03-09T15:46:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:46:13 smithi122 ceph-mon[93869]: slow request 2883.185631 seconds old, received at 2023-03-09T14:58:08.560137+0000: client_request(mds.2:599 rename #0x10000000007/0000000100000001000000EB #0x60f/1000000062a caller_uid=0, caller_gid=0{}) currently acquired locks 2023-03-09T15:46:13.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:46:13 smithi122 ceph-mon[93869]: slow request 2883.171251 seconds old, received at 2023-03-09T14:58:08.574517+0000: client_request(client.15375:6773 unlink #0x10000000007/0000000100000001000000EB 2023-03-09T14:58:08.569481+0000 caller_uid=26, caller_gid=26{26,}) currently requesting remote authpins 2023-03-09T15:46:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:46:13 smithi181 ceph-mon[94189]: 2 slow requests, 2 included below; oldest blocked for > 2883.185631 secs 2023-03-09T15:46:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:46:13 smithi181 ceph-mon[94189]: slow request 2883.185631 seconds old, received at 2023-03-09T14:58:08.560137+0000: client_request(mds.2:599 rename #0x10000000007/0000000100000001000000EB #0x60f/1000000062a caller_uid=0, caller_gid=0{}) currently acquired locks 2023-03-09T15:46:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:46:13 smithi181 ceph-mon[94189]: slow request 2883.171251 seconds old, received at 2023-03-09T14:58:08.574517+0000: client_request(client.15375:6773 unlink #0x10000000007/0000000100000001000000EB 2023-03-09T14:58:08.569481+0000 caller_uid=26, caller_gid=26{26,}) currently requesting remote authpins 2023-03-09T15:46:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:46:13 smithi038 ceph-mon[92422]: 2 slow requests, 2 included below; oldest blocked for > 2883.185631 secs 2023-03-09T15:46:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:46:13 smithi038 ceph-mon[92422]: slow request 2883.185631 seconds old, received at 2023-03-09T14:58:08.560137+0000: client_request(mds.2:599 rename #0x10000000007/0000000100000001000000EB #0x60f/1000000062a caller_uid=0, caller_gid=0{}) currently acquired locks 2023-03-09T15:46:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:46:13 smithi038 ceph-mon[92422]: slow request 2883.171251 seconds old, received at 2023-03-09T14:58:08.574517+0000: client_request(client.15375:6773 unlink #0x10000000007/0000000100000001000000EB 2023-03-09T14:58:08.569481+0000 caller_uid=26, caller_gid=26{26,}) currently requesting remote authpins 2023-03-09T15:46:13.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3345.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:46:15.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:46:14 smithi122 ceph-mon[93869]: pgmap v2054: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 8 op/s 2023-03-09T15:46:15.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:46:14 smithi181 ceph-mon[94189]: pgmap v2054: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 8 op/s 2023-03-09T15:46:15.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:46:14 smithi038 ceph-mon[92422]: pgmap v2054: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 8 op/s 2023-03-09T15:46:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:46:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2885.977580 secs 2023-03-09T15:46:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:46:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2885.977580 secs 2023-03-09T15:46:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:46:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2885.977580 secs 2023-03-09T15:46:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:46:16 smithi122 ceph-mon[93869]: pgmap v2055: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 7 op/s 2023-03-09T15:46:17.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:46:16 smithi181 ceph-mon[94189]: pgmap v2055: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 7 op/s 2023-03-09T15:46:17.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:46:16 smithi038 ceph-mon[92422]: pgmap v2055: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 7 op/s 2023-03-09T15:46:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:46:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2888.185743 secs 2023-03-09T15:46:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:46:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2888.185743 secs 2023-03-09T15:46:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:46:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2888.185743 secs 2023-03-09T15:46:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3350.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:46:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:46:18 smithi122 ceph-mon[93869]: pgmap v2056: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 8 op/s 2023-03-09T15:46:19.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:46:18 smithi122 ceph-mon[93869]: osdmap e343: 12 total, 12 up, 12 in 2023-03-09T15:46:19.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:46:18 smithi181 ceph-mon[94189]: pgmap v2056: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 8 op/s 2023-03-09T15:46:19.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:46:18 smithi181 ceph-mon[94189]: osdmap e343: 12 total, 12 up, 12 in 2023-03-09T15:46:19.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:46:18 smithi038 ceph-mon[92422]: pgmap v2056: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 8 op/s 2023-03-09T15:46:19.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:46:18 smithi038 ceph-mon[92422]: osdmap e343: 12 total, 12 up, 12 in 2023-03-09T15:46:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:46:20 smithi122 ceph-mon[93869]: pgmap v2058: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 8 op/s 2023-03-09T15:46:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:46:20 smithi181 ceph-mon[94189]: pgmap v2058: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 8 op/s 2023-03-09T15:46:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:46:20 smithi038 ceph-mon[92422]: pgmap v2058: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 8 op/s 2023-03-09T15:46:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:46:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2890.977640 secs 2023-03-09T15:46:21.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:46:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2890.977640 secs 2023-03-09T15:46:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:46:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2890.977640 secs 2023-03-09T15:46:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:46:22 smithi122 ceph-mon[93869]: pgmap v2059: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 7 op/s 2023-03-09T15:46:22.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:46:22 smithi181 ceph-mon[94189]: pgmap v2059: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 7 op/s 2023-03-09T15:46:22.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:46:22 smithi038 ceph-mon[92422]: pgmap v2059: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 7 op/s 2023-03-09T15:46:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:46:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2893.185886 secs 2023-03-09T15:46:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:46:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2893.185886 secs 2023-03-09T15:46:23.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:46:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2893.185886 secs 2023-03-09T15:46:23.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3355.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:46:25.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:46:24 smithi122 ceph-mon[93869]: pgmap v2060: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T15:46:25.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:46:24 smithi181 ceph-mon[94189]: pgmap v2060: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T15:46:25.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:46:24 smithi038 ceph-mon[92422]: pgmap v2060: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T15:46:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:46:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2895.977698 secs 2023-03-09T15:46:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:46:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2895.977698 secs 2023-03-09T15:46:26.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:46:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2895.977698 secs 2023-03-09T15:46:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:46:26 smithi122 ceph-mon[93869]: pgmap v2061: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2023-03-09T15:46:27.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:46:26 smithi181 ceph-mon[94189]: pgmap v2061: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2023-03-09T15:46:27.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:46:26 smithi038 ceph-mon[92422]: pgmap v2061: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2023-03-09T15:46:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:46:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2898.186009 secs 2023-03-09T15:46:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:46:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2898.186009 secs 2023-03-09T15:46:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:46:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2898.186009 secs 2023-03-09T15:46:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3360.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:46:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:46:28 smithi122 ceph-mon[93869]: pgmap v2062: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:46:29.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:46:28 smithi181 ceph-mon[94189]: pgmap v2062: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:46:29.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:46:28 smithi038 ceph-mon[92422]: pgmap v2062: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T15:46:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:46:30 smithi122 ceph-mon[93869]: pgmap v2063: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 90 B/s wr, 5 op/s 2023-03-09T15:46:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:46:30 smithi181 ceph-mon[94189]: pgmap v2063: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 90 B/s wr, 5 op/s 2023-03-09T15:46:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:46:30 smithi038 ceph-mon[92422]: pgmap v2063: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 90 B/s wr, 5 op/s 2023-03-09T15:46:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:46:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2900.977760 secs 2023-03-09T15:46:31.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:46:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2900.977760 secs 2023-03-09T15:46:31.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:46:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2900.977760 secs 2023-03-09T15:46:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:46:32 smithi122 ceph-mon[93869]: pgmap v2064: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T15:46:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:46:32 smithi181 ceph-mon[94189]: pgmap v2064: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T15:46:32.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:46:32 smithi038 ceph-mon[92422]: pgmap v2064: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T15:46:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3365.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:46:35.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:46:34 smithi122 ceph-mon[93869]: pgmap v2065: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 6 op/s 2023-03-09T15:46:35.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:46:34 smithi181 ceph-mon[94189]: pgmap v2065: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 6 op/s 2023-03-09T15:46:35.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:46:34 smithi038 ceph-mon[92422]: pgmap v2065: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 6 op/s 2023-03-09T15:46:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:46:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2905.977822 secs 2023-03-09T15:46:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:46:35 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:46:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:46:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2905.977822 secs 2023-03-09T15:46:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:46:35 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:46:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:46:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2905.977822 secs 2023-03-09T15:46:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:46:35 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:46:37.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:46:36 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:46:37.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:46:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:46:37.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:46:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:46:37.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:46:36 smithi122 ceph-mon[93869]: pgmap v2066: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 62 KiB/s wr, 8 op/s 2023-03-09T15:46:37.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:46:36 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:46:37.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:46:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:46:37.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:46:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:46:37.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:46:36 smithi181 ceph-mon[94189]: pgmap v2066: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 62 KiB/s wr, 8 op/s 2023-03-09T15:46:37.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:46:36 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:46:37.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:46:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:46:37.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:46:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:46:37.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:46:36 smithi038 ceph-mon[92422]: pgmap v2066: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 62 KiB/s wr, 8 op/s 2023-03-09T15:46:38.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:46:38 smithi122 ceph-mon[93869]: pgmap v2067: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 65 KiB/s wr, 7 op/s 2023-03-09T15:46:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:46:38 smithi181 ceph-mon[94189]: pgmap v2067: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 65 KiB/s wr, 7 op/s 2023-03-09T15:46:38.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:46:38 smithi038 ceph-mon[92422]: pgmap v2067: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 65 KiB/s wr, 7 op/s 2023-03-09T15:46:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3370.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:46:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:46:40 smithi122 ceph-mon[93869]: pgmap v2068: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 65 KiB/s wr, 7 op/s 2023-03-09T15:46:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:46:40 smithi181 ceph-mon[94189]: pgmap v2068: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 65 KiB/s wr, 7 op/s 2023-03-09T15:46:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:46:40 smithi038 ceph-mon[92422]: pgmap v2068: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 65 KiB/s wr, 7 op/s 2023-03-09T15:46:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:46:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2910.977879 secs 2023-03-09T15:46:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:46:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2910.977879 secs 2023-03-09T15:46:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:46:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2910.977879 secs 2023-03-09T15:46:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:46:42 smithi122 ceph-mon[93869]: pgmap v2069: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 91 KiB/s wr, 11 op/s 2023-03-09T15:46:42.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:46:42 smithi181 ceph-mon[94189]: pgmap v2069: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 91 KiB/s wr, 11 op/s 2023-03-09T15:46:42.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:46:42 smithi038 ceph-mon[92422]: pgmap v2069: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 91 KiB/s wr, 11 op/s 2023-03-09T15:46:43.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3375.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:46:45.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:46:44 smithi122 ceph-mon[93869]: pgmap v2070: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 73 KiB/s wr, 10 op/s 2023-03-09T15:46:45.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:46:44 smithi181 ceph-mon[94189]: pgmap v2070: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 73 KiB/s wr, 10 op/s 2023-03-09T15:46:45.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:46:44 smithi038 ceph-mon[92422]: pgmap v2070: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 73 KiB/s wr, 10 op/s 2023-03-09T15:46:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:46:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2915.977939 secs 2023-03-09T15:46:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:46:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2915.977939 secs 2023-03-09T15:46:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:46:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2915.977939 secs 2023-03-09T15:46:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:46:46 smithi122 ceph-mon[93869]: pgmap v2071: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 130 KiB/s wr, 13 op/s 2023-03-09T15:46:47.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:46:46 smithi181 ceph-mon[94189]: pgmap v2071: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 130 KiB/s wr, 13 op/s 2023-03-09T15:46:47.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:46:46 smithi038 ceph-mon[92422]: pgmap v2071: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 130 KiB/s wr, 13 op/s 2023-03-09T15:46:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:46:47 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 2918.186423 secs 2023-03-09T15:46:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:46:47 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 2918.186423 secs 2023-03-09T15:46:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:46:47 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 2918.186423 secs 2023-03-09T15:46:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3380.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:46:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:46:48 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:46:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:46:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:46:49.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:46:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:46:49.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:46:48 smithi122 ceph-mon[93869]: pgmap v2072: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 94 KiB/s wr, 11 op/s 2023-03-09T15:46:49.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:46:48 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:46:49.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:46:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:46:49.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:46:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:46:49.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:46:48 smithi181 ceph-mon[94189]: pgmap v2072: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 94 KiB/s wr, 11 op/s 2023-03-09T15:46:49.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:46:48 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:46:49.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:46:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:46:49.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:46:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:46:49.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:46:48 smithi038 ceph-mon[92422]: pgmap v2072: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 94 KiB/s wr, 11 op/s 2023-03-09T15:46:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:46:50 smithi122 ceph-mon[93869]: pgmap v2073: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 91 KiB/s wr, 10 op/s 2023-03-09T15:46:50.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:46:50 smithi181 ceph-mon[94189]: pgmap v2073: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 91 KiB/s wr, 10 op/s 2023-03-09T15:46:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:46:50 smithi038 ceph-mon[92422]: pgmap v2073: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 91 KiB/s wr, 10 op/s 2023-03-09T15:46:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:46:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2920.977982 secs 2023-03-09T15:46:51.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:46:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2920.977982 secs 2023-03-09T15:46:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:46:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2920.977982 secs 2023-03-09T15:46:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:46:52 smithi122 ceph-mon[93869]: pgmap v2074: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 100 KiB/s wr, 12 op/s 2023-03-09T15:46:52.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:46:52 smithi181 ceph-mon[94189]: pgmap v2074: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 100 KiB/s wr, 12 op/s 2023-03-09T15:46:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:46:52 smithi038 ceph-mon[92422]: pgmap v2074: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 100 KiB/s wr, 12 op/s 2023-03-09T15:46:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:46:53 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 2923.186498 secs 2023-03-09T15:46:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:46:53 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 2923.186498 secs 2023-03-09T15:46:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:46:53 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 2923.186498 secs 2023-03-09T15:46:53.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3385.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:46:55.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:46:54 smithi122 ceph-mon[93869]: pgmap v2075: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 77 KiB/s wr, 9 op/s 2023-03-09T15:46:55.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:46:54 smithi181 ceph-mon[94189]: pgmap v2075: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 77 KiB/s wr, 9 op/s 2023-03-09T15:46:55.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:46:54 smithi038 ceph-mon[92422]: pgmap v2075: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 77 KiB/s wr, 9 op/s 2023-03-09T15:46:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:46:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2925.978044 secs 2023-03-09T15:46:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:46:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2925.978044 secs 2023-03-09T15:46:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:46:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2925.978044 secs 2023-03-09T15:46:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:46:56 smithi122 ceph-mon[93869]: pgmap v2076: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 151 KiB/s wr, 11 op/s 2023-03-09T15:46:57.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:46:56 smithi181 ceph-mon[94189]: pgmap v2076: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 151 KiB/s wr, 11 op/s 2023-03-09T15:46:57.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:46:56 smithi038 ceph-mon[92422]: pgmap v2076: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 151 KiB/s wr, 11 op/s 2023-03-09T15:46:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:46:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2928.186609 secs 2023-03-09T15:46:58.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:46:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2928.186609 secs 2023-03-09T15:46:58.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:46:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2928.186609 secs 2023-03-09T15:46:58.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3390.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:46:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:46:58 smithi122 ceph-mon[93869]: pgmap v2077: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 98 KiB/s wr, 9 op/s 2023-03-09T15:46:59.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:46:58 smithi181 ceph-mon[94189]: pgmap v2077: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 98 KiB/s wr, 9 op/s 2023-03-09T15:46:59.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:46:58 smithi038 ceph-mon[92422]: pgmap v2077: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 98 KiB/s wr, 9 op/s 2023-03-09T15:47:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:47:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2930.978104 secs 2023-03-09T15:47:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:47:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:47:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:47:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:47:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:47:00 smithi122 ceph-mon[93869]: pgmap v2078: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 94 KiB/s wr, 8 op/s 2023-03-09T15:47:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:47:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2930.978104 secs 2023-03-09T15:47:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:47:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:47:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:47:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:47:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:47:00 smithi181 ceph-mon[94189]: pgmap v2078: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 94 KiB/s wr, 8 op/s 2023-03-09T15:47:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:47:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2930.978104 secs 2023-03-09T15:47:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:47:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:47:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:47:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:47:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:47:00 smithi038 ceph-mon[92422]: pgmap v2078: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 94 KiB/s wr, 8 op/s 2023-03-09T15:47:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:47:02 smithi122 ceph-mon[93869]: pgmap v2079: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 132 KiB/s wr, 13 op/s 2023-03-09T15:47:02.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:47:02 smithi181 ceph-mon[94189]: pgmap v2079: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 132 KiB/s wr, 13 op/s 2023-03-09T15:47:02.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:47:02 smithi038 ceph-mon[92422]: pgmap v2079: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 132 KiB/s wr, 13 op/s 2023-03-09T15:47:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:47:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2933.186705 secs 2023-03-09T15:47:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:47:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2933.186705 secs 2023-03-09T15:47:03.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:47:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2933.186705 secs 2023-03-09T15:47:03.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3395.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:47:05.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:47:04 smithi122 ceph-mon[93869]: pgmap v2080: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 147 KiB/s wr, 24 op/s 2023-03-09T15:47:05.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:47:04 smithi122 ceph-mon[93869]: osdmap e344: 12 total, 12 up, 12 in 2023-03-09T15:47:05.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:47:04 smithi181 ceph-mon[94189]: pgmap v2080: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 147 KiB/s wr, 24 op/s 2023-03-09T15:47:05.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:47:04 smithi181 ceph-mon[94189]: osdmap e344: 12 total, 12 up, 12 in 2023-03-09T15:47:05.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:47:04 smithi038 ceph-mon[92422]: pgmap v2080: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 147 KiB/s wr, 24 op/s 2023-03-09T15:47:05.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:47:04 smithi038 ceph-mon[92422]: osdmap e344: 12 total, 12 up, 12 in 2023-03-09T15:47:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:47:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2935.978164 secs 2023-03-09T15:47:06.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:47:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2935.978164 secs 2023-03-09T15:47:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:47:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2935.978164 secs 2023-03-09T15:47:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:47:06 smithi122 ceph-mon[93869]: pgmap v2082: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 130 KiB/s wr, 27 op/s 2023-03-09T15:47:07.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:47:06 smithi181 ceph-mon[94189]: pgmap v2082: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 130 KiB/s wr, 27 op/s 2023-03-09T15:47:07.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:47:06 smithi038 ceph-mon[92422]: pgmap v2082: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 130 KiB/s wr, 27 op/s 2023-03-09T15:47:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:47:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2938.186858 secs 2023-03-09T15:47:08.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:47:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2938.186858 secs 2023-03-09T15:47:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:47:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2938.186858 secs 2023-03-09T15:47:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3400.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:47:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:47:08 smithi122 ceph-mon[93869]: pgmap v2083: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 127 KiB/s wr, 26 op/s 2023-03-09T15:47:09.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:47:08 smithi181 ceph-mon[94189]: pgmap v2083: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 127 KiB/s wr, 26 op/s 2023-03-09T15:47:09.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:47:08 smithi038 ceph-mon[92422]: pgmap v2083: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 127 KiB/s wr, 26 op/s 2023-03-09T15:47:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:47:09 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:47:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:47:09 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:47:10.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:47:09 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:47:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:47:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2940.978226 secs 2023-03-09T15:47:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:47:10 smithi122 ceph-mon[93869]: pgmap v2084: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 127 KiB/s wr, 26 op/s 2023-03-09T15:47:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:47:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:47:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:47:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:47:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:47:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:47:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:47:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:47:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:47:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2940.978226 secs 2023-03-09T15:47:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:47:10 smithi181 ceph-mon[94189]: pgmap v2084: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 127 KiB/s wr, 26 op/s 2023-03-09T15:47:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:47:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:47:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:47:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:47:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:47:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:47:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:47:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:47:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:47:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2940.978226 secs 2023-03-09T15:47:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:47:10 smithi038 ceph-mon[92422]: pgmap v2084: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 127 KiB/s wr, 26 op/s 2023-03-09T15:47:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:47:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:47:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:47:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:47:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:47:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:47:11.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:47:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:47:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:47:12 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:47:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:47:12 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:47:12.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:47:12 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:47:12.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:47:12 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:47:12.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:47:12 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:47:12.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:47:12 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:47:12.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:47:12 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:47:12.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:47:12 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:47:12.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:47:12 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:47:12.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:47:12 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:47:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:47:12 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:47:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:47:12 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:47:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:47:12 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:47:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:47:12 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:47:12.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:47:12 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:47:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:47:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2943.186951 secs 2023-03-09T15:47:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:47:13 smithi122 ceph-mon[93869]: pgmap v2085: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 98 KiB/s wr, 22 op/s 2023-03-09T15:47:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:47:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2943.186951 secs 2023-03-09T15:47:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:47:13 smithi181 ceph-mon[94189]: pgmap v2085: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 98 KiB/s wr, 22 op/s 2023-03-09T15:47:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:47:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2943.186951 secs 2023-03-09T15:47:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:47:13 smithi038 ceph-mon[92422]: pgmap v2085: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 98 KiB/s wr, 22 op/s 2023-03-09T15:47:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3405.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:47:15.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:47:14 smithi122 ceph-mon[93869]: osdmap e345: 12 total, 12 up, 12 in 2023-03-09T15:47:15.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:47:14 smithi122 ceph-mon[93869]: pgmap v2087: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 70 KiB/s wr, 7 op/s 2023-03-09T15:47:15.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:47:14 smithi181 ceph-mon[94189]: osdmap e345: 12 total, 12 up, 12 in 2023-03-09T15:47:15.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:47:14 smithi181 ceph-mon[94189]: pgmap v2087: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 70 KiB/s wr, 7 op/s 2023-03-09T15:47:15.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:47:14 smithi038 ceph-mon[92422]: osdmap e345: 12 total, 12 up, 12 in 2023-03-09T15:47:15.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:47:14 smithi038 ceph-mon[92422]: pgmap v2087: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 70 KiB/s wr, 7 op/s 2023-03-09T15:47:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:47:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2945.978286 secs 2023-03-09T15:47:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:47:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2945.978286 secs 2023-03-09T15:47:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:47:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2945.978286 secs 2023-03-09T15:47:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:47:16 smithi122 ceph-mon[93869]: pgmap v2088: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 6 op/s 2023-03-09T15:47:17.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:47:16 smithi181 ceph-mon[94189]: pgmap v2088: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 6 op/s 2023-03-09T15:47:17.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:47:16 smithi038 ceph-mon[92422]: pgmap v2088: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 6 op/s 2023-03-09T15:47:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:47:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2948.187017 secs 2023-03-09T15:47:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:47:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2948.187017 secs 2023-03-09T15:47:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:47:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2948.187017 secs 2023-03-09T15:47:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3410.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:47:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:47:18 smithi122 ceph-mon[93869]: pgmap v2089: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 6 op/s 2023-03-09T15:47:19.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:47:18 smithi181 ceph-mon[94189]: pgmap v2089: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 6 op/s 2023-03-09T15:47:19.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:47:18 smithi038 ceph-mon[92422]: pgmap v2089: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 6 op/s 2023-03-09T15:47:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:47:20 smithi122 ceph-mon[93869]: pgmap v2090: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 6 op/s 2023-03-09T15:47:20.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:47:20 smithi181 ceph-mon[94189]: pgmap v2090: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 6 op/s 2023-03-09T15:47:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:47:20 smithi038 ceph-mon[92422]: pgmap v2090: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 6 op/s 2023-03-09T15:47:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:47:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2950.978345 secs 2023-03-09T15:47:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:47:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2950.978345 secs 2023-03-09T15:47:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:47:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2950.978345 secs 2023-03-09T15:47:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:47:22 smithi122 ceph-mon[93869]: pgmap v2091: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.6 KiB/s wr, 6 op/s 2023-03-09T15:47:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:47:22 smithi181 ceph-mon[94189]: pgmap v2091: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.6 KiB/s wr, 6 op/s 2023-03-09T15:47:22.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:47:22 smithi038 ceph-mon[92422]: pgmap v2091: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.6 KiB/s wr, 6 op/s 2023-03-09T15:47:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:47:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2953.187133 secs 2023-03-09T15:47:23.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:47:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2953.187133 secs 2023-03-09T15:47:23.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:47:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2953.187133 secs 2023-03-09T15:47:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3415.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:47:25.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:47:24 smithi122 ceph-mon[93869]: pgmap v2092: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.8 KiB/s wr, 7 op/s 2023-03-09T15:47:25.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:47:24 smithi181 ceph-mon[94189]: pgmap v2092: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.8 KiB/s wr, 7 op/s 2023-03-09T15:47:25.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:47:24 smithi038 ceph-mon[92422]: pgmap v2092: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.8 KiB/s wr, 7 op/s 2023-03-09T15:47:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:47:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2955.978406 secs 2023-03-09T15:47:26.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:47:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2955.978406 secs 2023-03-09T15:47:26.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:47:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2955.978406 secs 2023-03-09T15:47:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:47:26 smithi122 ceph-mon[93869]: pgmap v2093: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 9 op/s 2023-03-09T15:47:27.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:47:26 smithi181 ceph-mon[94189]: pgmap v2093: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 9 op/s 2023-03-09T15:47:27.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:47:26 smithi038 ceph-mon[92422]: pgmap v2093: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 9 op/s 2023-03-09T15:47:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:47:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2958.187245 secs 2023-03-09T15:47:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:47:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2958.187245 secs 2023-03-09T15:47:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:47:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2958.187245 secs 2023-03-09T15:47:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3420.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:47:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:47:28 smithi122 ceph-mon[93869]: pgmap v2094: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 22 KiB/s wr, 7 op/s 2023-03-09T15:47:29.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:47:28 smithi181 ceph-mon[94189]: pgmap v2094: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 22 KiB/s wr, 7 op/s 2023-03-09T15:47:29.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:47:28 smithi038 ceph-mon[92422]: pgmap v2094: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 22 KiB/s wr, 7 op/s 2023-03-09T15:47:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:47:30 smithi122 ceph-mon[93869]: pgmap v2095: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 22 KiB/s wr, 6 op/s 2023-03-09T15:47:30.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:47:30 smithi181 ceph-mon[94189]: pgmap v2095: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 22 KiB/s wr, 6 op/s 2023-03-09T15:47:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:47:30 smithi038 ceph-mon[92422]: pgmap v2095: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 22 KiB/s wr, 6 op/s 2023-03-09T15:47:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:47:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2960.978497 secs 2023-03-09T15:47:31.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:47:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2960.978497 secs 2023-03-09T15:47:31.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:47:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2960.978497 secs 2023-03-09T15:47:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:47:32 smithi122 ceph-mon[93869]: pgmap v2096: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 85 KiB/s wr, 9 op/s 2023-03-09T15:47:32.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:47:32 smithi181 ceph-mon[94189]: pgmap v2096: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 85 KiB/s wr, 9 op/s 2023-03-09T15:47:32.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:47:32 smithi038 ceph-mon[92422]: pgmap v2096: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 85 KiB/s wr, 9 op/s 2023-03-09T15:47:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:47:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2963.187379 secs 2023-03-09T15:47:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:47:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2963.187379 secs 2023-03-09T15:47:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:47:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2963.187379 secs 2023-03-09T15:47:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3425.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:47:35.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:47:34 smithi122 ceph-mon[93869]: pgmap v2097: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 91 KiB/s wr, 28 op/s 2023-03-09T15:47:35.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:47:34 smithi181 ceph-mon[94189]: pgmap v2097: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 91 KiB/s wr, 28 op/s 2023-03-09T15:47:35.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:47:34 smithi038 ceph-mon[92422]: pgmap v2097: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 91 KiB/s wr, 28 op/s 2023-03-09T15:47:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:47:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2965.978557 secs 2023-03-09T15:47:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:47:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2965.978557 secs 2023-03-09T15:47:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:47:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2965.978557 secs 2023-03-09T15:47:37.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:47:36 smithi122 ceph-mon[93869]: pgmap v2098: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 107 KiB/s wr, 30 op/s 2023-03-09T15:47:37.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:47:36 smithi181 ceph-mon[94189]: pgmap v2098: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 107 KiB/s wr, 30 op/s 2023-03-09T15:47:37.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:47:36 smithi038 ceph-mon[92422]: pgmap v2098: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 107 KiB/s wr, 30 op/s 2023-03-09T15:47:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:47:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2968.187508 secs 2023-03-09T15:47:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:47:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2968.187508 secs 2023-03-09T15:47:38.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:47:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2968.187508 secs 2023-03-09T15:47:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3430.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:47:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:47:38 smithi122 ceph-mon[93869]: pgmap v2099: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 142 KiB/s wr, 30 op/s 2023-03-09T15:47:39.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:47:38 smithi181 ceph-mon[94189]: pgmap v2099: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 142 KiB/s wr, 30 op/s 2023-03-09T15:47:39.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:47:38 smithi038 ceph-mon[92422]: pgmap v2099: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 142 KiB/s wr, 30 op/s 2023-03-09T15:47:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:47:40 smithi122 ceph-mon[93869]: pgmap v2100: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 141 KiB/s wr, 33 op/s 2023-03-09T15:47:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:47:40 smithi181 ceph-mon[94189]: pgmap v2100: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 141 KiB/s wr, 33 op/s 2023-03-09T15:47:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:47:40 smithi038 ceph-mon[92422]: pgmap v2100: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 141 KiB/s wr, 33 op/s 2023-03-09T15:47:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:47:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2970.978640 secs 2023-03-09T15:47:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:47:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2970.978640 secs 2023-03-09T15:47:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:47:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2970.978640 secs 2023-03-09T15:47:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:47:42 smithi122 ceph-mon[93869]: pgmap v2101: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 190 KiB/s wr, 37 op/s 2023-03-09T15:47:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:47:42 smithi181 ceph-mon[94189]: pgmap v2101: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 190 KiB/s wr, 37 op/s 2023-03-09T15:47:42.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:47:42 smithi038 ceph-mon[92422]: pgmap v2101: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 190 KiB/s wr, 37 op/s 2023-03-09T15:47:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:47:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2973.187601 secs 2023-03-09T15:47:43.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:47:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2973.187601 secs 2023-03-09T15:47:43.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:47:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2973.187601 secs 2023-03-09T15:47:43.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3435.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:47:45.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:47:44 smithi122 ceph-mon[93869]: pgmap v2102: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 131 KiB/s wr, 35 op/s 2023-03-09T15:47:45.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:47:44 smithi181 ceph-mon[94189]: pgmap v2102: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 131 KiB/s wr, 35 op/s 2023-03-09T15:47:45.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:47:44 smithi038 ceph-mon[92422]: pgmap v2102: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 131 KiB/s wr, 35 op/s 2023-03-09T15:47:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:47:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2975.978701 secs 2023-03-09T15:47:46.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:47:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2975.978701 secs 2023-03-09T15:47:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:47:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2975.978701 secs 2023-03-09T15:47:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:47:46 smithi122 ceph-mon[93869]: pgmap v2103: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 124 KiB/s wr, 16 op/s 2023-03-09T15:47:47.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:47:46 smithi181 ceph-mon[94189]: pgmap v2103: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 124 KiB/s wr, 16 op/s 2023-03-09T15:47:47.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:47:46 smithi038 ceph-mon[92422]: pgmap v2103: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 124 KiB/s wr, 16 op/s 2023-03-09T15:47:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:47:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2978.187730 secs 2023-03-09T15:47:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:47:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2978.187730 secs 2023-03-09T15:47:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:47:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2978.187730 secs 2023-03-09T15:47:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3440.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:47:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:47:48 smithi122 ceph-mon[93869]: pgmap v2104: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 139 KiB/s wr, 15 op/s 2023-03-09T15:47:49.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:47:48 smithi181 ceph-mon[94189]: pgmap v2104: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 139 KiB/s wr, 15 op/s 2023-03-09T15:47:49.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:47:48 smithi038 ceph-mon[92422]: pgmap v2104: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 139 KiB/s wr, 15 op/s 2023-03-09T15:47:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:47:50 smithi122 ceph-mon[93869]: pgmap v2105: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 84 KiB/s wr, 12 op/s 2023-03-09T15:47:50.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:47:50 smithi181 ceph-mon[94189]: pgmap v2105: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 84 KiB/s wr, 12 op/s 2023-03-09T15:47:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:47:50 smithi038 ceph-mon[92422]: pgmap v2105: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 84 KiB/s wr, 12 op/s 2023-03-09T15:47:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:47:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2980.978759 secs 2023-03-09T15:47:51.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:47:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2980.978759 secs 2023-03-09T15:47:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:47:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2980.978759 secs 2023-03-09T15:47:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:47:52 smithi122 ceph-mon[93869]: pgmap v2106: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 138 KiB/s wr, 11 op/s 2023-03-09T15:47:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:47:52 smithi181 ceph-mon[94189]: pgmap v2106: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 138 KiB/s wr, 11 op/s 2023-03-09T15:47:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:47:52 smithi038 ceph-mon[92422]: pgmap v2106: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 138 KiB/s wr, 11 op/s 2023-03-09T15:47:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:47:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2983.187851 secs 2023-03-09T15:47:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:47:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2983.187851 secs 2023-03-09T15:47:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:47:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2983.187851 secs 2023-03-09T15:47:53.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3445.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:47:55.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:47:54 smithi122 ceph-mon[93869]: pgmap v2107: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 89 KiB/s wr, 8 op/s 2023-03-09T15:47:55.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:47:54 smithi181 ceph-mon[94189]: pgmap v2107: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 89 KiB/s wr, 8 op/s 2023-03-09T15:47:55.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:47:54 smithi038 ceph-mon[92422]: pgmap v2107: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 89 KiB/s wr, 8 op/s 2023-03-09T15:47:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:47:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2985.978821 secs 2023-03-09T15:47:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:47:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2985.978821 secs 2023-03-09T15:47:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:47:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2985.978821 secs 2023-03-09T15:47:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:47:56 smithi122 ceph-mon[93869]: pgmap v2108: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 86 KiB/s wr, 9 op/s 2023-03-09T15:47:57.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:47:56 smithi181 ceph-mon[94189]: pgmap v2108: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 86 KiB/s wr, 9 op/s 2023-03-09T15:47:57.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:47:56 smithi038 ceph-mon[92422]: pgmap v2108: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 86 KiB/s wr, 9 op/s 2023-03-09T15:47:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:47:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2988.187976 secs 2023-03-09T15:47:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:47:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2988.187976 secs 2023-03-09T15:47:58.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:47:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2988.187976 secs 2023-03-09T15:47:58.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3450.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:47:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:47:58 smithi122 ceph-mon[93869]: pgmap v2109: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 106 KiB/s wr, 9 op/s 2023-03-09T15:47:59.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:47:58 smithi181 ceph-mon[94189]: pgmap v2109: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 106 KiB/s wr, 9 op/s 2023-03-09T15:47:59.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:47:58 smithi038 ceph-mon[92422]: pgmap v2109: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 106 KiB/s wr, 9 op/s 2023-03-09T15:48:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:00 smithi122 ceph-mon[93869]: pgmap v2110: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 75 KiB/s wr, 7 op/s 2023-03-09T15:48:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:00 smithi181 ceph-mon[94189]: pgmap v2110: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 75 KiB/s wr, 7 op/s 2023-03-09T15:48:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:00 smithi038 ceph-mon[92422]: pgmap v2110: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 75 KiB/s wr, 7 op/s 2023-03-09T15:48:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2990.978881 secs 2023-03-09T15:48:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2990.978881 secs 2023-03-09T15:48:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2990.978881 secs 2023-03-09T15:48:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:02 smithi122 ceph-mon[93869]: pgmap v2111: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 117 KiB/s wr, 12 op/s 2023-03-09T15:48:02.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:02 smithi181 ceph-mon[94189]: pgmap v2111: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 117 KiB/s wr, 12 op/s 2023-03-09T15:48:02.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:02 smithi038 ceph-mon[92422]: pgmap v2111: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 117 KiB/s wr, 12 op/s 2023-03-09T15:48:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2993.188126 secs 2023-03-09T15:48:03.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2993.188126 secs 2023-03-09T15:48:03.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2993.188126 secs 2023-03-09T15:48:03.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3455.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:48:05.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:04 smithi122 ceph-mon[93869]: pgmap v2112: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 87 KiB/s wr, 21 op/s 2023-03-09T15:48:05.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:04 smithi181 ceph-mon[94189]: pgmap v2112: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 87 KiB/s wr, 21 op/s 2023-03-09T15:48:05.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:04 smithi038 ceph-mon[92422]: pgmap v2112: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 87 KiB/s wr, 21 op/s 2023-03-09T15:48:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2995.978964 secs 2023-03-09T15:48:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2995.978964 secs 2023-03-09T15:48:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2995.978964 secs 2023-03-09T15:48:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:06 smithi122 ceph-mon[93869]: pgmap v2113: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 88 KiB/s wr, 23 op/s 2023-03-09T15:48:07.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:06 smithi181 ceph-mon[94189]: pgmap v2113: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 88 KiB/s wr, 23 op/s 2023-03-09T15:48:07.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:06 smithi038 ceph-mon[92422]: pgmap v2113: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 88 KiB/s wr, 23 op/s 2023-03-09T15:48:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 2998.188244 secs 2023-03-09T15:48:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 2998.188244 secs 2023-03-09T15:48:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 2998.188244 secs 2023-03-09T15:48:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3460.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:48:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:08 smithi122 ceph-mon[93869]: pgmap v2114: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 88 KiB/s wr, 22 op/s 2023-03-09T15:48:09.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:08 smithi181 ceph-mon[94189]: pgmap v2114: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 88 KiB/s wr, 22 op/s 2023-03-09T15:48:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:08 smithi038 ceph-mon[92422]: pgmap v2114: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 88 KiB/s wr, 22 op/s 2023-03-09T15:48:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:09 smithi122 ceph-mon[93869]: osdmap e346: 12 total, 12 up, 12 in 2023-03-09T15:48:10.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:09 smithi181 ceph-mon[94189]: osdmap e346: 12 total, 12 up, 12 in 2023-03-09T15:48:10.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:09 smithi038 ceph-mon[92422]: osdmap e346: 12 total, 12 up, 12 in 2023-03-09T15:48:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3000.978983 secs 2023-03-09T15:48:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:10 smithi122 ceph-mon[93869]: pgmap v2116: 129 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+snaptrim, 127 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 81 KiB/s wr, 25 op/s 2023-03-09T15:48:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3000.978983 secs 2023-03-09T15:48:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:10 smithi181 ceph-mon[94189]: pgmap v2116: 129 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+snaptrim, 127 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 81 KiB/s wr, 25 op/s 2023-03-09T15:48:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3000.978983 secs 2023-03-09T15:48:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:10 smithi038 ceph-mon[92422]: pgmap v2116: 129 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+snaptrim, 127 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 81 KiB/s wr, 25 op/s 2023-03-09T15:48:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:11 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:48:12.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:11 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:48:12.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:11 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:48:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3003.188388 secs 2023-03-09T15:48:13.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:12 smithi122 ceph-mon[93869]: pgmap v2117: 129 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+snaptrim, 127 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 55 KiB/s wr, 22 op/s 2023-03-09T15:48:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3003.188388 secs 2023-03-09T15:48:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:12 smithi181 ceph-mon[94189]: pgmap v2117: 129 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+snaptrim, 127 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 55 KiB/s wr, 22 op/s 2023-03-09T15:48:13.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3003.188388 secs 2023-03-09T15:48:13.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:12 smithi038 ceph-mon[92422]: pgmap v2117: 129 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+snaptrim, 127 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 55 KiB/s wr, 22 op/s 2023-03-09T15:48:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3465.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:48:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:48:15.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:48:15.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:15 smithi122 ceph-mon[93869]: pgmap v2118: 129 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+snaptrim, 127 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 27 KiB/s wr, 8 op/s 2023-03-09T15:48:15.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:48:15.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:48:15.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:48:15.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:48:15.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:48:15.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:48:15.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:48:15.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:48:15.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:48:15.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:48:15.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:48:15.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:48:15.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:48:15.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:48:15.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:15 smithi181 ceph-mon[94189]: pgmap v2118: 129 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+snaptrim, 127 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 27 KiB/s wr, 8 op/s 2023-03-09T15:48:15.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:48:15.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:48:15.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:48:15.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:48:15.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:48:15.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:48:15.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:48:15.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:48:15.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:48:15.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:48:15.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:48:15.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:48:15.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:48:15.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:48:15.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:15 smithi038 ceph-mon[92422]: pgmap v2118: 129 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+snaptrim, 127 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 27 KiB/s wr, 8 op/s 2023-03-09T15:48:15.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:48:15.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:48:15.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:48:15.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:48:15.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:48:15.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:48:15.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:48:15.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:48:15.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:48:15.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:48:15.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:48:15.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:48:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3005.979054 secs 2023-03-09T15:48:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:16 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T15:48:16.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:16 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T15:48:16.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:16 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi038 to 197.2M 2023-03-09T15:48:16.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:16 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi038 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-03-09T15:48:16.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:16 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:48:16.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:16 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:48:16.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:16 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:48:16.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:16 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:48:16.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:16 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:48:16.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:16 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:48:16.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:16 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:48:16.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3005.979054 secs 2023-03-09T15:48:16.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:16 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T15:48:16.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:16 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T15:48:16.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:16 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi038 to 197.2M 2023-03-09T15:48:16.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:16 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi038 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-03-09T15:48:16.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:16 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:48:16.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:16 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:48:16.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:16 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:48:16.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:16 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:48:16.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:16 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:48:16.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:16 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:48:16.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:16 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:48:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3005.979054 secs 2023-03-09T15:48:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:16 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T15:48:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:16 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T15:48:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:16 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi038 to 197.2M 2023-03-09T15:48:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:16 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi038 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-03-09T15:48:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:16 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:48:16.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:16 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:48:16.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:16 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:48:16.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:16 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:48:16.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:16 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:48:16.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:16 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:48:16.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:16 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:48:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:17 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T15:48:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:17 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T15:48:17.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:17 smithi122 ceph-mon[93869]: pgmap v2119: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 25 KiB/s wr, 7 op/s 2023-03-09T15:48:17.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:17 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T15:48:17.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:17 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T15:48:17.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:17 smithi181 ceph-mon[94189]: pgmap v2119: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 25 KiB/s wr, 7 op/s 2023-03-09T15:48:17.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:17 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T15:48:17.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:17 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T15:48:17.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:17 smithi038 ceph-mon[92422]: pgmap v2119: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 25 KiB/s wr, 7 op/s 2023-03-09T15:48:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3008.188488 secs 2023-03-09T15:48:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3008.188488 secs 2023-03-09T15:48:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3008.188488 secs 2023-03-09T15:48:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3470.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:48:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:19 smithi122 ceph-mon[93869]: pgmap v2120: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 25 KiB/s wr, 8 op/s 2023-03-09T15:48:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:19 smithi122 ceph-mon[93869]: osdmap e347: 12 total, 12 up, 12 in 2023-03-09T15:48:19.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:19 smithi181 ceph-mon[94189]: pgmap v2120: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 25 KiB/s wr, 8 op/s 2023-03-09T15:48:19.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:19 smithi181 ceph-mon[94189]: osdmap e347: 12 total, 12 up, 12 in 2023-03-09T15:48:19.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:19 smithi038 ceph-mon[92422]: pgmap v2120: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 25 KiB/s wr, 8 op/s 2023-03-09T15:48:19.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:19 smithi038 ceph-mon[92422]: osdmap e347: 12 total, 12 up, 12 in 2023-03-09T15:48:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:20 smithi122 ceph-mon[93869]: pgmap v2122: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 25 KiB/s wr, 8 op/s 2023-03-09T15:48:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:20 smithi181 ceph-mon[94189]: pgmap v2122: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 25 KiB/s wr, 8 op/s 2023-03-09T15:48:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:20 smithi038 ceph-mon[92422]: pgmap v2122: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 25 KiB/s wr, 8 op/s 2023-03-09T15:48:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3010.979112 secs 2023-03-09T15:48:21.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3010.979112 secs 2023-03-09T15:48:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3010.979112 secs 2023-03-09T15:48:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:22 smithi122 ceph-mon[93869]: pgmap v2123: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 25 KiB/s wr, 7 op/s 2023-03-09T15:48:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:22 smithi181 ceph-mon[94189]: pgmap v2123: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 25 KiB/s wr, 7 op/s 2023-03-09T15:48:22.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:22 smithi038 ceph-mon[92422]: pgmap v2123: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 25 KiB/s wr, 7 op/s 2023-03-09T15:48:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3013.188609 secs 2023-03-09T15:48:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3013.188609 secs 2023-03-09T15:48:23.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3013.188609 secs 2023-03-09T15:48:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3475.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:48:25.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:24 smithi122 ceph-mon[93869]: pgmap v2124: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 25 KiB/s wr, 7 op/s 2023-03-09T15:48:25.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:24 smithi181 ceph-mon[94189]: pgmap v2124: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 25 KiB/s wr, 7 op/s 2023-03-09T15:48:25.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:24 smithi038 ceph-mon[92422]: pgmap v2124: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 25 KiB/s wr, 7 op/s 2023-03-09T15:48:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3015.979198 secs 2023-03-09T15:48:26.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3015.979198 secs 2023-03-09T15:48:26.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3015.979198 secs 2023-03-09T15:48:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:26 smithi122 ceph-mon[93869]: pgmap v2125: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 25 KiB/s wr, 7 op/s 2023-03-09T15:48:27.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:26 smithi181 ceph-mon[94189]: pgmap v2125: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 25 KiB/s wr, 7 op/s 2023-03-09T15:48:27.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:26 smithi038 ceph-mon[92422]: pgmap v2125: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 25 KiB/s wr, 7 op/s 2023-03-09T15:48:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3018.188731 secs 2023-03-09T15:48:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3018.188731 secs 2023-03-09T15:48:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3018.188731 secs 2023-03-09T15:48:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3480.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:48:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:28 smithi122 ceph-mon[93869]: pgmap v2126: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 6 op/s 2023-03-09T15:48:29.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:28 smithi181 ceph-mon[94189]: pgmap v2126: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 6 op/s 2023-03-09T15:48:29.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:28 smithi038 ceph-mon[92422]: pgmap v2126: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 6 op/s 2023-03-09T15:48:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:30 smithi122 ceph-mon[93869]: pgmap v2127: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 5 op/s 2023-03-09T15:48:30.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:30 smithi181 ceph-mon[94189]: pgmap v2127: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 5 op/s 2023-03-09T15:48:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:30 smithi038 ceph-mon[92422]: pgmap v2127: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 5 op/s 2023-03-09T15:48:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3020.979258 secs 2023-03-09T15:48:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3020.979258 secs 2023-03-09T15:48:31.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3020.979258 secs 2023-03-09T15:48:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:32 smithi122 ceph-mon[93869]: pgmap v2128: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 44 KiB/s wr, 8 op/s 2023-03-09T15:48:32.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:32 smithi181 ceph-mon[94189]: pgmap v2128: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 44 KiB/s wr, 8 op/s 2023-03-09T15:48:32.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:32 smithi038 ceph-mon[92422]: pgmap v2128: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 44 KiB/s wr, 8 op/s 2023-03-09T15:48:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3023.188851 secs 2023-03-09T15:48:33.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3023.188851 secs 2023-03-09T15:48:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3023.188851 secs 2023-03-09T15:48:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3485.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:48:35.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:34 smithi122 ceph-mon[93869]: pgmap v2129: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 23 KiB/s wr, 6 op/s 2023-03-09T15:48:35.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:34 smithi181 ceph-mon[94189]: pgmap v2129: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 23 KiB/s wr, 6 op/s 2023-03-09T15:48:35.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:34 smithi038 ceph-mon[92422]: pgmap v2129: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 23 KiB/s wr, 6 op/s 2023-03-09T15:48:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3025.979318 secs 2023-03-09T15:48:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3025.979318 secs 2023-03-09T15:48:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3025.979318 secs 2023-03-09T15:48:37.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:36 smithi122 ceph-mon[93869]: pgmap v2130: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T15:48:37.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:36 smithi181 ceph-mon[94189]: pgmap v2130: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T15:48:37.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:36 smithi038 ceph-mon[92422]: pgmap v2130: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T15:48:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3028.188963 secs 2023-03-09T15:48:38.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3028.188963 secs 2023-03-09T15:48:38.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3028.188963 secs 2023-03-09T15:48:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3490.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:48:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:38 smithi122 ceph-mon[93869]: pgmap v2131: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 68 KiB/s wr, 8 op/s 2023-03-09T15:48:39.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:38 smithi181 ceph-mon[94189]: pgmap v2131: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 68 KiB/s wr, 8 op/s 2023-03-09T15:48:39.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:38 smithi038 ceph-mon[92422]: pgmap v2131: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 68 KiB/s wr, 8 op/s 2023-03-09T15:48:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:40 smithi122 ceph-mon[93869]: pgmap v2132: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 67 KiB/s wr, 7 op/s 2023-03-09T15:48:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:40 smithi181 ceph-mon[94189]: pgmap v2132: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 67 KiB/s wr, 7 op/s 2023-03-09T15:48:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:40 smithi038 ceph-mon[92422]: pgmap v2132: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 67 KiB/s wr, 7 op/s 2023-03-09T15:48:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3030.979376 secs 2023-03-09T15:48:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3030.979376 secs 2023-03-09T15:48:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3030.979376 secs 2023-03-09T15:48:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:42 smithi122 ceph-mon[93869]: pgmap v2133: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 77 KiB/s wr, 10 op/s 2023-03-09T15:48:42.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:42 smithi181 ceph-mon[94189]: pgmap v2133: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 77 KiB/s wr, 10 op/s 2023-03-09T15:48:42.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:42 smithi038 ceph-mon[92422]: pgmap v2133: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 77 KiB/s wr, 10 op/s 2023-03-09T15:48:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3033.189094 secs 2023-03-09T15:48:43.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3033.189094 secs 2023-03-09T15:48:43.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3033.189094 secs 2023-03-09T15:48:43.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3495.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:48:45.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:44 smithi122 ceph-mon[93869]: pgmap v2134: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 57 KiB/s wr, 9 op/s 2023-03-09T15:48:45.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:44 smithi181 ceph-mon[94189]: pgmap v2134: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 57 KiB/s wr, 9 op/s 2023-03-09T15:48:45.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:44 smithi038 ceph-mon[92422]: pgmap v2134: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 57 KiB/s wr, 9 op/s 2023-03-09T15:48:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3035.979436 secs 2023-03-09T15:48:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3035.979436 secs 2023-03-09T15:48:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3035.979436 secs 2023-03-09T15:48:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:46 smithi122 ceph-mon[93869]: pgmap v2135: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 133 KiB/s wr, 12 op/s 2023-03-09T15:48:47.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:46 smithi181 ceph-mon[94189]: pgmap v2135: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 133 KiB/s wr, 12 op/s 2023-03-09T15:48:47.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:46 smithi038 ceph-mon[92422]: pgmap v2135: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 133 KiB/s wr, 12 op/s 2023-03-09T15:48:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3038.189200 secs 2023-03-09T15:48:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3038.189200 secs 2023-03-09T15:48:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3038.189200 secs 2023-03-09T15:48:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3500.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:48:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:48 smithi122 ceph-mon[93869]: pgmap v2136: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 137 KiB/s wr, 11 op/s 2023-03-09T15:48:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:48 smithi181 ceph-mon[94189]: pgmap v2136: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 137 KiB/s wr, 11 op/s 2023-03-09T15:48:49.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:48 smithi038 ceph-mon[92422]: pgmap v2136: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 137 KiB/s wr, 11 op/s 2023-03-09T15:48:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:50 smithi122 ceph-mon[93869]: pgmap v2137: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 93 KiB/s wr, 9 op/s 2023-03-09T15:48:50.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:50 smithi181 ceph-mon[94189]: pgmap v2137: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 93 KiB/s wr, 9 op/s 2023-03-09T15:48:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:50 smithi038 ceph-mon[92422]: pgmap v2137: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 93 KiB/s wr, 9 op/s 2023-03-09T15:48:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3040.979496 secs 2023-03-09T15:48:51.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3040.979496 secs 2023-03-09T15:48:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3040.979496 secs 2023-03-09T15:48:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:52 smithi122 ceph-mon[93869]: pgmap v2138: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 102 KiB/s wr, 12 op/s 2023-03-09T15:48:52.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:52 smithi181 ceph-mon[94189]: pgmap v2138: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 102 KiB/s wr, 12 op/s 2023-03-09T15:48:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:52 smithi038 ceph-mon[92422]: pgmap v2138: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 102 KiB/s wr, 12 op/s 2023-03-09T15:48:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3043.189349 secs 2023-03-09T15:48:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3043.189349 secs 2023-03-09T15:48:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3043.189349 secs 2023-03-09T15:48:53.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3505.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:48:55.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:54 smithi122 ceph-mon[93869]: pgmap v2139: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 96 KiB/s wr, 9 op/s 2023-03-09T15:48:55.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:54 smithi181 ceph-mon[94189]: pgmap v2139: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 96 KiB/s wr, 9 op/s 2023-03-09T15:48:55.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:54 smithi038 ceph-mon[92422]: pgmap v2139: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 96 KiB/s wr, 9 op/s 2023-03-09T15:48:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3045.979581 secs 2023-03-09T15:48:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3045.979581 secs 2023-03-09T15:48:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3045.979581 secs 2023-03-09T15:48:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:56 smithi122 ceph-mon[93869]: pgmap v2140: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 176 KiB/s wr, 11 op/s 2023-03-09T15:48:57.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:56 smithi181 ceph-mon[94189]: pgmap v2140: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 176 KiB/s wr, 11 op/s 2023-03-09T15:48:57.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:56 smithi038 ceph-mon[92422]: pgmap v2140: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 176 KiB/s wr, 11 op/s 2023-03-09T15:48:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3048.189469 secs 2023-03-09T15:48:58.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3048.189469 secs 2023-03-09T15:48:58.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3048.189469 secs 2023-03-09T15:48:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3510.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:48:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:48:58 smithi122 ceph-mon[93869]: pgmap v2141: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 104 KiB/s wr, 9 op/s 2023-03-09T15:48:59.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:48:58 smithi181 ceph-mon[94189]: pgmap v2141: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 104 KiB/s wr, 9 op/s 2023-03-09T15:48:59.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:48:58 smithi038 ceph-mon[92422]: pgmap v2141: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 104 KiB/s wr, 9 op/s 2023-03-09T15:49:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:49:00 smithi122 ceph-mon[93869]: pgmap v2142: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 100 KiB/s wr, 8 op/s 2023-03-09T15:49:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:49:00 smithi181 ceph-mon[94189]: pgmap v2142: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 100 KiB/s wr, 8 op/s 2023-03-09T15:49:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:49:00 smithi038 ceph-mon[92422]: pgmap v2142: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 100 KiB/s wr, 8 op/s 2023-03-09T15:49:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:49:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3050.979641 secs 2023-03-09T15:49:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:49:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3050.979641 secs 2023-03-09T15:49:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:49:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3050.979641 secs 2023-03-09T15:49:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:49:02 smithi122 ceph-mon[93869]: pgmap v2143: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 138 KiB/s wr, 13 op/s 2023-03-09T15:49:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:49:02 smithi181 ceph-mon[94189]: pgmap v2143: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 138 KiB/s wr, 13 op/s 2023-03-09T15:49:02.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:49:02 smithi038 ceph-mon[92422]: pgmap v2143: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 138 KiB/s wr, 13 op/s 2023-03-09T15:49:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:49:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3053.189606 secs 2023-03-09T15:49:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:49:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3053.189606 secs 2023-03-09T15:49:03.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:49:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3053.189606 secs 2023-03-09T15:49:03.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3515.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:49:05.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:49:04 smithi122 ceph-mon[93869]: pgmap v2144: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 153 KiB/s wr, 23 op/s 2023-03-09T15:49:05.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:49:04 smithi122 ceph-mon[93869]: osdmap e348: 12 total, 12 up, 12 in 2023-03-09T15:49:05.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:49:04 smithi181 ceph-mon[94189]: pgmap v2144: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 153 KiB/s wr, 23 op/s 2023-03-09T15:49:05.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:49:04 smithi181 ceph-mon[94189]: osdmap e348: 12 total, 12 up, 12 in 2023-03-09T15:49:05.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:49:04 smithi038 ceph-mon[92422]: pgmap v2144: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 153 KiB/s wr, 23 op/s 2023-03-09T15:49:05.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:49:04 smithi038 ceph-mon[92422]: osdmap e348: 12 total, 12 up, 12 in 2023-03-09T15:49:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:49:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3055.979704 secs 2023-03-09T15:49:06.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:49:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3055.979704 secs 2023-03-09T15:49:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:49:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3055.979704 secs 2023-03-09T15:49:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:49:06 smithi122 ceph-mon[93869]: pgmap v2146: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 133 KiB/s wr, 27 op/s 2023-03-09T15:49:07.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:49:06 smithi181 ceph-mon[94189]: pgmap v2146: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 133 KiB/s wr, 27 op/s 2023-03-09T15:49:07.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:49:06 smithi038 ceph-mon[92422]: pgmap v2146: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 133 KiB/s wr, 27 op/s 2023-03-09T15:49:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:49:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3058.189734 secs 2023-03-09T15:49:08.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:49:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3058.189734 secs 2023-03-09T15:49:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:49:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3058.189734 secs 2023-03-09T15:49:08.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3520.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:49:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:49:08 smithi122 ceph-mon[93869]: pgmap v2147: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 129 KiB/s wr, 26 op/s 2023-03-09T15:49:09.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:49:08 smithi181 ceph-mon[94189]: pgmap v2147: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 129 KiB/s wr, 26 op/s 2023-03-09T15:49:09.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:49:08 smithi038 ceph-mon[92422]: pgmap v2147: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 129 KiB/s wr, 26 op/s 2023-03-09T15:49:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:49:10 smithi122 ceph-mon[93869]: pgmap v2148: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 129 KiB/s wr, 26 op/s 2023-03-09T15:49:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:49:10 smithi181 ceph-mon[94189]: pgmap v2148: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 129 KiB/s wr, 26 op/s 2023-03-09T15:49:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:49:10 smithi038 ceph-mon[92422]: pgmap v2148: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 129 KiB/s wr, 26 op/s 2023-03-09T15:49:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:49:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3060.979761 secs 2023-03-09T15:49:11.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:49:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3060.979761 secs 2023-03-09T15:49:11.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:49:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3060.979761 secs 2023-03-09T15:49:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:49:12 smithi122 ceph-mon[93869]: pgmap v2149: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 101 KiB/s wr, 22 op/s 2023-03-09T15:49:12.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:49:12 smithi181 ceph-mon[94189]: pgmap v2149: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 101 KiB/s wr, 22 op/s 2023-03-09T15:49:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:49:12 smithi038 ceph-mon[92422]: pgmap v2149: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 101 KiB/s wr, 22 op/s 2023-03-09T15:49:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:49:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3063.189844 secs 2023-03-09T15:49:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:49:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3063.189844 secs 2023-03-09T15:49:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:49:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3063.189844 secs 2023-03-09T15:49:13.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3525.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:49:15.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:49:14 smithi122 ceph-mon[93869]: osdmap e349: 12 total, 12 up, 12 in 2023-03-09T15:49:15.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:49:14 smithi122 ceph-mon[93869]: pgmap v2151: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 7 op/s 2023-03-09T15:49:15.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:49:14 smithi181 ceph-mon[94189]: osdmap e349: 12 total, 12 up, 12 in 2023-03-09T15:49:15.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:49:14 smithi181 ceph-mon[94189]: pgmap v2151: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 7 op/s 2023-03-09T15:49:15.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:49:14 smithi038 ceph-mon[92422]: osdmap e349: 12 total, 12 up, 12 in 2023-03-09T15:49:15.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:49:14 smithi038 ceph-mon[92422]: pgmap v2151: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 7 op/s 2023-03-09T15:49:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:49:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3065.979820 secs 2023-03-09T15:49:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:49:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:49:16.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:49:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3065.979820 secs 2023-03-09T15:49:16.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:49:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:49:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:49:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3065.979820 secs 2023-03-09T15:49:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:49:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:49:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:49:16 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:49:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:49:16 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:49:17.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:49:16 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:49:17.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:49:16 smithi122 ceph-mon[93869]: pgmap v2152: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 6 op/s 2023-03-09T15:49:17.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:49:16 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:49:17.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:49:16 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:49:17.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:49:16 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:49:17.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:49:16 smithi181 ceph-mon[94189]: pgmap v2152: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 6 op/s 2023-03-09T15:49:17.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:49:16 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:49:17.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:49:16 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:49:17.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:49:16 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:49:17.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:49:16 smithi038 ceph-mon[92422]: pgmap v2152: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 6 op/s 2023-03-09T15:49:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:49:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3068.189949 secs 2023-03-09T15:49:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:49:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3068.189949 secs 2023-03-09T15:49:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:49:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3068.189949 secs 2023-03-09T15:49:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3530.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:49:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:49:19 smithi122 ceph-mon[93869]: pgmap v2153: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 6 op/s 2023-03-09T15:49:19.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:49:19 smithi181 ceph-mon[94189]: pgmap v2153: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 6 op/s 2023-03-09T15:49:19.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:49:19 smithi038 ceph-mon[92422]: pgmap v2153: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 6 op/s 2023-03-09T15:49:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:49:20 smithi122 ceph-mon[93869]: pgmap v2154: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 6 op/s 2023-03-09T15:49:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:49:20 smithi181 ceph-mon[94189]: pgmap v2154: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 6 op/s 2023-03-09T15:49:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:49:20 smithi038 ceph-mon[92422]: pgmap v2154: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 6 op/s 2023-03-09T15:49:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:49:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3070.979878 secs 2023-03-09T15:49:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:49:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3070.979878 secs 2023-03-09T15:49:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:49:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3070.979878 secs 2023-03-09T15:49:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:49:22 smithi122 ceph-mon[93869]: pgmap v2155: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.6 KiB/s wr, 6 op/s 2023-03-09T15:49:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:49:22 smithi181 ceph-mon[94189]: pgmap v2155: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.6 KiB/s wr, 6 op/s 2023-03-09T15:49:22.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:49:22 smithi038 ceph-mon[92422]: pgmap v2155: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.6 KiB/s wr, 6 op/s 2023-03-09T15:49:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:49:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3073.190097 secs 2023-03-09T15:49:23.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:49:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3073.190097 secs 2023-03-09T15:49:23.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:49:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3073.190097 secs 2023-03-09T15:49:23.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3535.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:49:25.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:49:24 smithi122 ceph-mon[93869]: pgmap v2156: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.8 KiB/s wr, 7 op/s 2023-03-09T15:49:25.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:49:24 smithi181 ceph-mon[94189]: pgmap v2156: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.8 KiB/s wr, 7 op/s 2023-03-09T15:49:25.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:49:24 smithi038 ceph-mon[92422]: pgmap v2156: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.8 KiB/s wr, 7 op/s 2023-03-09T15:49:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:49:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3075.979937 secs 2023-03-09T15:49:26.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:49:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3075.979937 secs 2023-03-09T15:49:26.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:49:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3075.979937 secs 2023-03-09T15:49:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:49:26 smithi122 ceph-mon[93869]: pgmap v2157: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 8 op/s 2023-03-09T15:49:27.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:49:26 smithi181 ceph-mon[94189]: pgmap v2157: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 8 op/s 2023-03-09T15:49:27.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:49:26 smithi038 ceph-mon[92422]: pgmap v2157: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 8 op/s 2023-03-09T15:49:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:49:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3078.190193 secs 2023-03-09T15:49:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:49:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3078.190193 secs 2023-03-09T15:49:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:49:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3078.190193 secs 2023-03-09T15:49:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3540.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:49:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:49:28 smithi122 ceph-mon[93869]: pgmap v2158: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 7 op/s 2023-03-09T15:49:29.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:49:28 smithi181 ceph-mon[94189]: pgmap v2158: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 7 op/s 2023-03-09T15:49:29.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:49:28 smithi038 ceph-mon[92422]: pgmap v2158: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 7 op/s 2023-03-09T15:49:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:49:30 smithi122 ceph-mon[93869]: pgmap v2159: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 6 op/s 2023-03-09T15:49:30.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:49:30 smithi181 ceph-mon[94189]: pgmap v2159: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 6 op/s 2023-03-09T15:49:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:49:30 smithi038 ceph-mon[92422]: pgmap v2159: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 6 op/s 2023-03-09T15:49:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:49:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3080.979988 secs 2023-03-09T15:49:31.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:49:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3080.979988 secs 2023-03-09T15:49:31.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:49:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3080.979988 secs 2023-03-09T15:49:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:49:32 smithi122 ceph-mon[93869]: pgmap v2160: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 80 KiB/s wr, 10 op/s 2023-03-09T15:49:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:49:32 smithi181 ceph-mon[94189]: pgmap v2160: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 80 KiB/s wr, 10 op/s 2023-03-09T15:49:32.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:49:32 smithi038 ceph-mon[92422]: pgmap v2160: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 80 KiB/s wr, 10 op/s 2023-03-09T15:49:33.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3545.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:49:35.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:49:34 smithi122 ceph-mon[93869]: pgmap v2161: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 86 KiB/s wr, 8 op/s 2023-03-09T15:49:35.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:49:34 smithi181 ceph-mon[94189]: pgmap v2161: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 86 KiB/s wr, 8 op/s 2023-03-09T15:49:35.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:49:34 smithi038 ceph-mon[92422]: pgmap v2161: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 86 KiB/s wr, 8 op/s 2023-03-09T15:49:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:49:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3085.980046 secs 2023-03-09T15:49:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:49:35 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:49:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:49:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3085.980046 secs 2023-03-09T15:49:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:49:35 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:49:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:49:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3085.980046 secs 2023-03-09T15:49:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:49:35 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:49:37.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:49:36 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:49:37.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:49:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:49:37.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:49:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:49:37.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:49:36 smithi122 ceph-mon[93869]: pgmap v2162: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 110 KiB/s wr, 11 op/s 2023-03-09T15:49:37.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:49:36 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:49:37.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:49:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:49:37.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:49:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:49:37.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:49:36 smithi181 ceph-mon[94189]: pgmap v2162: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 110 KiB/s wr, 11 op/s 2023-03-09T15:49:37.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:49:36 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:49:37.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:49:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:49:37.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:49:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:49:37.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:49:36 smithi038 ceph-mon[92422]: pgmap v2162: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 110 KiB/s wr, 11 op/s 2023-03-09T15:49:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3550.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:49:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:49:39 smithi122 ceph-mon[93869]: pgmap v2163: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 146 KiB/s wr, 11 op/s 2023-03-09T15:49:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:49:39 smithi181 ceph-mon[94189]: pgmap v2163: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 146 KiB/s wr, 11 op/s 2023-03-09T15:49:40.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:49:39 smithi038 ceph-mon[92422]: pgmap v2163: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 146 KiB/s wr, 11 op/s 2023-03-09T15:49:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:49:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3090.980107 secs 2023-03-09T15:49:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:49:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3090.980107 secs 2023-03-09T15:49:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:49:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3090.980107 secs 2023-03-09T15:49:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:49:41 smithi122 ceph-mon[93869]: pgmap v2164: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 145 KiB/s wr, 10 op/s 2023-03-09T15:49:42.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:49:41 smithi181 ceph-mon[94189]: pgmap v2164: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 145 KiB/s wr, 10 op/s 2023-03-09T15:49:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:49:41 smithi038 ceph-mon[92422]: pgmap v2164: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 145 KiB/s wr, 10 op/s 2023-03-09T15:49:43.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3555.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:49:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:49:43 smithi122 ceph-mon[93869]: pgmap v2165: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 163 KiB/s wr, 13 op/s 2023-03-09T15:49:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:49:43 smithi181 ceph-mon[94189]: pgmap v2165: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 163 KiB/s wr, 13 op/s 2023-03-09T15:49:44.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:49:43 smithi038 ceph-mon[92422]: pgmap v2165: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 163 KiB/s wr, 13 op/s 2023-03-09T15:49:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:49:45 smithi122 ceph-mon[93869]: pgmap v2166: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 106 KiB/s wr, 11 op/s 2023-03-09T15:49:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:49:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3095.980165 secs 2023-03-09T15:49:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:49:45 smithi181 ceph-mon[94189]: pgmap v2166: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 106 KiB/s wr, 11 op/s 2023-03-09T15:49:46.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:49:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3095.980165 secs 2023-03-09T15:49:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:49:45 smithi038 ceph-mon[92422]: pgmap v2166: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 106 KiB/s wr, 11 op/s 2023-03-09T15:49:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:49:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3095.980165 secs 2023-03-09T15:49:47.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:49:46 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:49:47.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:49:46 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:49:47.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:49:46 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:49:47.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:49:46 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:49:47.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:49:46 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:49:47.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:49:46 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:49:47.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:49:46 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:49:47.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:49:46 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:49:47.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:49:46 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:49:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:49:47 smithi122 ceph-mon[93869]: pgmap v2167: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 99 KiB/s wr, 11 op/s 2023-03-09T15:49:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:49:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3098.190633 secs 2023-03-09T15:49:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:49:47 smithi181 ceph-mon[94189]: pgmap v2167: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 99 KiB/s wr, 11 op/s 2023-03-09T15:49:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:49:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3098.190633 secs 2023-03-09T15:49:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:49:47 smithi038 ceph-mon[92422]: pgmap v2167: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 99 KiB/s wr, 11 op/s 2023-03-09T15:49:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:49:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3098.190633 secs 2023-03-09T15:49:48.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3560.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:49:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:49:49 smithi122 ceph-mon[93869]: pgmap v2168: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 147 KiB/s wr, 11 op/s 2023-03-09T15:49:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:49:49 smithi181 ceph-mon[94189]: pgmap v2168: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 147 KiB/s wr, 11 op/s 2023-03-09T15:49:50.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:49:49 smithi038 ceph-mon[92422]: pgmap v2168: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 147 KiB/s wr, 11 op/s 2023-03-09T15:49:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:49:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3100.980227 secs 2023-03-09T15:49:51.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:49:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3100.980227 secs 2023-03-09T15:49:51.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:49:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3100.980227 secs 2023-03-09T15:49:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:49:51 smithi122 ceph-mon[93869]: pgmap v2169: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 90 KiB/s wr, 8 op/s 2023-03-09T15:49:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:49:51 smithi181 ceph-mon[94189]: pgmap v2169: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 90 KiB/s wr, 8 op/s 2023-03-09T15:49:52.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:49:51 smithi038 ceph-mon[92422]: pgmap v2169: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 90 KiB/s wr, 8 op/s 2023-03-09T15:49:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:49:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3103.190768 secs 2023-03-09T15:49:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:49:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3103.190768 secs 2023-03-09T15:49:53.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:49:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3103.190768 secs 2023-03-09T15:49:53.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3565.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:49:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:49:53 smithi122 ceph-mon[93869]: pgmap v2170: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 125 KiB/s wr, 11 op/s 2023-03-09T15:49:54.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:49:53 smithi181 ceph-mon[94189]: pgmap v2170: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 125 KiB/s wr, 11 op/s 2023-03-09T15:49:54.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:49:53 smithi038 ceph-mon[92422]: pgmap v2170: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 125 KiB/s wr, 11 op/s 2023-03-09T15:49:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:49:55 smithi122 ceph-mon[93869]: pgmap v2171: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 110 KiB/s wr, 8 op/s 2023-03-09T15:49:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:49:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3105.980286 secs 2023-03-09T15:49:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:49:55 smithi181 ceph-mon[94189]: pgmap v2171: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 110 KiB/s wr, 8 op/s 2023-03-09T15:49:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:49:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3105.980286 secs 2023-03-09T15:49:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:49:55 smithi038 ceph-mon[92422]: pgmap v2171: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 110 KiB/s wr, 8 op/s 2023-03-09T15:49:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:49:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3105.980286 secs 2023-03-09T15:49:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:49:57 smithi122 ceph-mon[93869]: pgmap v2172: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 110 KiB/s wr, 9 op/s 2023-03-09T15:49:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:49:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3108.190901 secs 2023-03-09T15:49:58.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:49:57 smithi181 ceph-mon[94189]: pgmap v2172: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 110 KiB/s wr, 9 op/s 2023-03-09T15:49:58.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:49:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3108.190901 secs 2023-03-09T15:49:58.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:49:57 smithi038 ceph-mon[92422]: pgmap v2172: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 110 KiB/s wr, 9 op/s 2023-03-09T15:49:58.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:49:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3108.190901 secs 2023-03-09T15:49:58.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3570.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:50:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:49:59 smithi122 ceph-mon[93869]: pgmap v2173: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 130 KiB/s wr, 9 op/s 2023-03-09T15:50:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:49:59 smithi181 ceph-mon[94189]: pgmap v2173: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 130 KiB/s wr, 9 op/s 2023-03-09T15:50:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:49:59 smithi038 ceph-mon[92422]: pgmap v2173: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 130 KiB/s wr, 9 op/s 2023-03-09T15:50:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:50:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3110.980344 secs 2023-03-09T15:50:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:50:00 smithi038 ceph-mon[92422]: overall HEALTH_WARN 2 MDSs report slow requests 2023-03-09T15:50:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:50:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3110.980344 secs 2023-03-09T15:50:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:50:00 smithi122 ceph-mon[93869]: overall HEALTH_WARN 2 MDSs report slow requests 2023-03-09T15:50:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:50:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3110.980344 secs 2023-03-09T15:50:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:50:00 smithi181 ceph-mon[94189]: overall HEALTH_WARN 2 MDSs report slow requests 2023-03-09T15:50:02.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:50:01 smithi038 ceph-mon[92422]: pgmap v2174: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 59 KiB/s wr, 6 op/s 2023-03-09T15:50:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:50:01 smithi122 ceph-mon[93869]: pgmap v2174: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 59 KiB/s wr, 6 op/s 2023-03-09T15:50:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:50:01 smithi181 ceph-mon[94189]: pgmap v2174: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 59 KiB/s wr, 6 op/s 2023-03-09T15:50:03.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:50:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3113.190971 secs 2023-03-09T15:50:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:50:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3113.190971 secs 2023-03-09T15:50:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:50:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3113.190971 secs 2023-03-09T15:50:03.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3575.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:50:04.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:50:03 smithi038 ceph-mon[92422]: pgmap v2175: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 71 KiB/s wr, 10 op/s 2023-03-09T15:50:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:50:03 smithi122 ceph-mon[93869]: pgmap v2175: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 71 KiB/s wr, 10 op/s 2023-03-09T15:50:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:50:03 smithi181 ceph-mon[94189]: pgmap v2175: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 71 KiB/s wr, 10 op/s 2023-03-09T15:50:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:50:06 smithi122 ceph-mon[93869]: pgmap v2176: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 60 KiB/s wr, 20 op/s 2023-03-09T15:50:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:50:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3115.980414 secs 2023-03-09T15:50:06.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:50:06 smithi181 ceph-mon[94189]: pgmap v2176: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 60 KiB/s wr, 20 op/s 2023-03-09T15:50:06.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:50:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3115.980414 secs 2023-03-09T15:50:06.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:50:06 smithi038 ceph-mon[92422]: pgmap v2176: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 60 KiB/s wr, 20 op/s 2023-03-09T15:50:06.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:50:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3115.980414 secs 2023-03-09T15:50:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:50:08 smithi122 ceph-mon[93869]: pgmap v2177: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 70 KiB/s wr, 22 op/s 2023-03-09T15:50:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:50:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3118.191088 secs 2023-03-09T15:50:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:50:08 smithi181 ceph-mon[94189]: pgmap v2177: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 70 KiB/s wr, 22 op/s 2023-03-09T15:50:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:50:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3118.191088 secs 2023-03-09T15:50:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:50:08 smithi038 ceph-mon[92422]: pgmap v2177: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 70 KiB/s wr, 22 op/s 2023-03-09T15:50:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:50:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3118.191088 secs 2023-03-09T15:50:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3580.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:50:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:50:10 smithi122 ceph-mon[93869]: pgmap v2178: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 73 KiB/s wr, 22 op/s 2023-03-09T15:50:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:50:10 smithi122 ceph-mon[93869]: osdmap e350: 12 total, 12 up, 12 in 2023-03-09T15:50:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:50:10 smithi181 ceph-mon[94189]: pgmap v2178: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 73 KiB/s wr, 22 op/s 2023-03-09T15:50:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:50:10 smithi181 ceph-mon[94189]: osdmap e350: 12 total, 12 up, 12 in 2023-03-09T15:50:10.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:50:10 smithi038 ceph-mon[92422]: pgmap v2178: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 73 KiB/s wr, 22 op/s 2023-03-09T15:50:10.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:50:10 smithi038 ceph-mon[92422]: osdmap e350: 12 total, 12 up, 12 in 2023-03-09T15:50:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:50:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3120.980485 secs 2023-03-09T15:50:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:50:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3120.980485 secs 2023-03-09T15:50:11.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:50:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3120.980485 secs 2023-03-09T15:50:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:50:12 smithi122 ceph-mon[93869]: pgmap v2180: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 63 KiB/s wr, 25 op/s 2023-03-09T15:50:12.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:50:12 smithi181 ceph-mon[94189]: pgmap v2180: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 63 KiB/s wr, 25 op/s 2023-03-09T15:50:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:50:12 smithi038 ceph-mon[92422]: pgmap v2180: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 63 KiB/s wr, 25 op/s 2023-03-09T15:50:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:50:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3123.191212 secs 2023-03-09T15:50:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:50:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3123.191212 secs 2023-03-09T15:50:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:50:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3123.191212 secs 2023-03-09T15:50:13.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3585.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:50:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:50:14 smithi122 ceph-mon[93869]: pgmap v2181: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 75 KiB/s wr, 24 op/s 2023-03-09T15:50:14.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:50:14 smithi181 ceph-mon[94189]: pgmap v2181: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 75 KiB/s wr, 24 op/s 2023-03-09T15:50:14.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:50:14 smithi038 ceph-mon[92422]: pgmap v2181: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 75 KiB/s wr, 24 op/s 2023-03-09T15:50:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:50:16 smithi122 ceph-mon[93869]: pgmap v2182: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 46 KiB/s wr, 10 op/s 2023-03-09T15:50:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:50:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3125.980544 secs 2023-03-09T15:50:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:50:16 smithi181 ceph-mon[94189]: pgmap v2182: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 46 KiB/s wr, 10 op/s 2023-03-09T15:50:16.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:50:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3125.980544 secs 2023-03-09T15:50:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:50:16 smithi038 ceph-mon[92422]: pgmap v2182: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 46 KiB/s wr, 10 op/s 2023-03-09T15:50:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:50:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3125.980544 secs 2023-03-09T15:50:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:50:17 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:50:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:50:17 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:50:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:50:17 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:50:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:50:17 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:50:17.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:50:17 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:50:17.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:50:17 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:50:17.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:50:17 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:50:17.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:50:17 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:50:17.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:50:17 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:50:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:50:18 smithi122 ceph-mon[93869]: pgmap v2183: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 31 KiB/s wr, 9 op/s 2023-03-09T15:50:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:50:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3128.191295 secs 2023-03-09T15:50:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:50:18 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:50:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:50:18 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:50:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:50:18 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:50:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:50:18 smithi181 ceph-mon[94189]: pgmap v2183: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 31 KiB/s wr, 9 op/s 2023-03-09T15:50:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:50:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3128.191295 secs 2023-03-09T15:50:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:50:18 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:50:18.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:50:18 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:50:18.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:50:18 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:50:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:50:18 smithi038 ceph-mon[92422]: pgmap v2183: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 31 KiB/s wr, 9 op/s 2023-03-09T15:50:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:50:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3128.191295 secs 2023-03-09T15:50:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:50:18 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:50:18.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:50:18 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:50:18.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:50:18 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:50:18.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3590.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:50:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:50:19 smithi122 ceph-mon[93869]: pgmap v2184: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 8 op/s 2023-03-09T15:50:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:50:19 smithi122 ceph-mon[93869]: osdmap e351: 12 total, 12 up, 12 in 2023-03-09T15:50:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:50:19 smithi181 ceph-mon[94189]: pgmap v2184: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 8 op/s 2023-03-09T15:50:20.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:50:19 smithi181 ceph-mon[94189]: osdmap e351: 12 total, 12 up, 12 in 2023-03-09T15:50:20.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:50:19 smithi038 ceph-mon[92422]: pgmap v2184: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 8 op/s 2023-03-09T15:50:20.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:50:19 smithi038 ceph-mon[92422]: osdmap e351: 12 total, 12 up, 12 in 2023-03-09T15:50:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:50:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3130.980602 secs 2023-03-09T15:50:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:50:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3130.980602 secs 2023-03-09T15:50:21.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:50:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3130.980602 secs 2023-03-09T15:50:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:50:21 smithi122 ceph-mon[93869]: pgmap v2186: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 8 op/s 2023-03-09T15:50:22.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:50:21 smithi181 ceph-mon[94189]: pgmap v2186: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 8 op/s 2023-03-09T15:50:22.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:50:21 smithi038 ceph-mon[92422]: pgmap v2186: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 8 op/s 2023-03-09T15:50:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:50:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3133.191406 secs 2023-03-09T15:50:23.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:50:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3133.191406 secs 2023-03-09T15:50:23.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:50:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3133.191406 secs 2023-03-09T15:50:23.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3595.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:50:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:50:23 smithi122 ceph-mon[93869]: pgmap v2187: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T15:50:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:50:23 smithi181 ceph-mon[94189]: pgmap v2187: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T15:50:24.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:50:23 smithi038 ceph-mon[92422]: pgmap v2187: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T15:50:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:50:25 smithi122 ceph-mon[93869]: pgmap v2188: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T15:50:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:50:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3135.980664 secs 2023-03-09T15:50:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:50:25 smithi181 ceph-mon[94189]: pgmap v2188: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T15:50:26.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:50:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3135.980664 secs 2023-03-09T15:50:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:50:25 smithi038 ceph-mon[92422]: pgmap v2188: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T15:50:26.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:50:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3135.980664 secs 2023-03-09T15:50:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:50:27 smithi122 ceph-mon[93869]: pgmap v2189: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T15:50:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:50:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3138.191501 secs 2023-03-09T15:50:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:50:27 smithi181 ceph-mon[94189]: pgmap v2189: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T15:50:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:50:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3138.191501 secs 2023-03-09T15:50:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:50:27 smithi038 ceph-mon[92422]: pgmap v2189: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T15:50:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:50:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3138.191501 secs 2023-03-09T15:50:28.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3600.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:50:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:50:29 smithi122 ceph-mon[93869]: pgmap v2190: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 6 op/s 2023-03-09T15:50:30.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:50:29 smithi181 ceph-mon[94189]: pgmap v2190: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 6 op/s 2023-03-09T15:50:30.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:50:29 smithi038 ceph-mon[92422]: pgmap v2190: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 6 op/s 2023-03-09T15:50:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:50:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3140.980722 secs 2023-03-09T15:50:31.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:50:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3140.980722 secs 2023-03-09T15:50:31.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:50:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3140.980722 secs 2023-03-09T15:50:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:50:31 smithi122 ceph-mon[93869]: pgmap v2191: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 21 KiB/s wr, 6 op/s 2023-03-09T15:50:32.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:50:31 smithi181 ceph-mon[94189]: pgmap v2191: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 21 KiB/s wr, 6 op/s 2023-03-09T15:50:32.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:50:31 smithi038 ceph-mon[92422]: pgmap v2191: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 21 KiB/s wr, 6 op/s 2023-03-09T15:50:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3605.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:50:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:50:33 smithi122 ceph-mon[93869]: pgmap v2192: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 44 KiB/s wr, 8 op/s 2023-03-09T15:50:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:50:33 smithi181 ceph-mon[94189]: pgmap v2192: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 44 KiB/s wr, 8 op/s 2023-03-09T15:50:34.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:50:33 smithi038 ceph-mon[92422]: pgmap v2192: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 44 KiB/s wr, 8 op/s 2023-03-09T15:50:35.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:50:34 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:50:35.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:50:34 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:50:35.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:50:34 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:50:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:50:35 smithi122 ceph-mon[93869]: pgmap v2193: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 6 op/s 2023-03-09T15:50:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:50:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3145.980782 secs 2023-03-09T15:50:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:50:35 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:50:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:50:35 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:50:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:50:35 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:50:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:50:35 smithi181 ceph-mon[94189]: pgmap v2193: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 6 op/s 2023-03-09T15:50:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:50:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3145.980782 secs 2023-03-09T15:50:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:50:35 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:50:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:50:35 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:50:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:50:35 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:50:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:50:35 smithi038 ceph-mon[92422]: pgmap v2193: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 6 op/s 2023-03-09T15:50:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:50:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3145.980782 secs 2023-03-09T15:50:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:50:35 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:50:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:50:35 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:50:36.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:50:35 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:50:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:50:37 smithi122 ceph-mon[93869]: pgmap v2194: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 24 KiB/s wr, 8 op/s 2023-03-09T15:50:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:50:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3148.191737 secs 2023-03-09T15:50:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:50:37 smithi181 ceph-mon[94189]: pgmap v2194: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 24 KiB/s wr, 8 op/s 2023-03-09T15:50:38.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:50:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3148.191737 secs 2023-03-09T15:50:38.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:50:37 smithi038 ceph-mon[92422]: pgmap v2194: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 24 KiB/s wr, 8 op/s 2023-03-09T15:50:38.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:50:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3148.191737 secs 2023-03-09T15:50:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3610.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:50:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:50:39 smithi122 ceph-mon[93869]: pgmap v2195: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 91 KiB/s wr, 37 op/s 2023-03-09T15:50:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:50:39 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:50:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:50:39 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:50:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:50:39 smithi181 ceph-mon[94189]: pgmap v2195: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 91 KiB/s wr, 37 op/s 2023-03-09T15:50:40.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:50:39 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:50:40.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:50:39 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:50:40.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:50:39 smithi038 ceph-mon[92422]: pgmap v2195: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 91 KiB/s wr, 37 op/s 2023-03-09T15:50:40.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:50:39 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:50:40.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:50:39 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:50:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:50:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3150.980856 secs 2023-03-09T15:50:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:50:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3150.980856 secs 2023-03-09T15:50:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:50:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3150.980856 secs 2023-03-09T15:50:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:50:41 smithi122 ceph-mon[93869]: pgmap v2196: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 91 KiB/s wr, 36 op/s 2023-03-09T15:50:42.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:50:41 smithi181 ceph-mon[94189]: pgmap v2196: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 91 KiB/s wr, 36 op/s 2023-03-09T15:50:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:50:41 smithi038 ceph-mon[92422]: pgmap v2196: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 91 KiB/s wr, 36 op/s 2023-03-09T15:50:43.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:50:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3153.191856 secs 2023-03-09T15:50:43.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:50:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3153.191856 secs 2023-03-09T15:50:43.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:50:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3153.191856 secs 2023-03-09T15:50:43.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3615.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:50:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:50:43 smithi122 ceph-mon[93869]: pgmap v2197: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 112 KiB/s wr, 40 op/s 2023-03-09T15:50:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:50:43 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:50:44.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:50:43 smithi181 ceph-mon[94189]: pgmap v2197: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 112 KiB/s wr, 40 op/s 2023-03-09T15:50:44.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:50:43 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:50:44.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:50:43 smithi038 ceph-mon[92422]: pgmap v2197: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 112 KiB/s wr, 40 op/s 2023-03-09T15:50:44.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:50:43 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:50:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:50:45 smithi122 ceph-mon[93869]: pgmap v2198: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 94 KiB/s wr, 48 op/s 2023-03-09T15:50:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:50:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3155.980917 secs 2023-03-09T15:50:46.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:50:45 smithi181 ceph-mon[94189]: pgmap v2198: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 94 KiB/s wr, 48 op/s 2023-03-09T15:50:46.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:50:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3155.980917 secs 2023-03-09T15:50:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:50:45 smithi038 ceph-mon[92422]: pgmap v2198: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 94 KiB/s wr, 48 op/s 2023-03-09T15:50:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:50:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3155.980917 secs 2023-03-09T15:50:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:50:47 smithi122 ceph-mon[93869]: pgmap v2199: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 155 KiB/s wr, 50 op/s 2023-03-09T15:50:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:50:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3158.191971 secs 2023-03-09T15:50:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:50:47 smithi181 ceph-mon[94189]: pgmap v2199: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 155 KiB/s wr, 50 op/s 2023-03-09T15:50:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:50:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3158.191971 secs 2023-03-09T15:50:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:50:47 smithi038 ceph-mon[92422]: pgmap v2199: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 155 KiB/s wr, 50 op/s 2023-03-09T15:50:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:50:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3158.191971 secs 2023-03-09T15:50:48.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3620.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:50:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:50:49 smithi122 ceph-mon[93869]: pgmap v2200: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 155 KiB/s wr, 50 op/s 2023-03-09T15:50:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:50:49 smithi181 ceph-mon[94189]: pgmap v2200: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 155 KiB/s wr, 50 op/s 2023-03-09T15:50:50.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:50:49 smithi038 ceph-mon[92422]: pgmap v2200: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 155 KiB/s wr, 50 op/s 2023-03-09T15:50:51.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:50:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3160.980984 secs 2023-03-09T15:50:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:50:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3160.980984 secs 2023-03-09T15:50:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:50:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3160.980984 secs 2023-03-09T15:50:52.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:50:51 smithi038 ceph-mon[92422]: pgmap v2201: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 101 KiB/s wr, 19 op/s 2023-03-09T15:50:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:50:51 smithi122 ceph-mon[93869]: pgmap v2201: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 101 KiB/s wr, 19 op/s 2023-03-09T15:50:52.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:50:51 smithi181 ceph-mon[94189]: pgmap v2201: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 101 KiB/s wr, 19 op/s 2023-03-09T15:50:53.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:50:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3163.192087 secs 2023-03-09T15:50:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:50:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3163.192087 secs 2023-03-09T15:50:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:50:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3163.192087 secs 2023-03-09T15:50:53.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3625.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:50:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:50:54 smithi122 ceph-mon[93869]: pgmap v2202: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 101 KiB/s wr, 21 op/s 2023-03-09T15:50:54.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:50:53 smithi181 ceph-mon[94189]: pgmap v2202: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 101 KiB/s wr, 21 op/s 2023-03-09T15:50:54.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:50:53 smithi038 ceph-mon[92422]: pgmap v2202: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 101 KiB/s wr, 21 op/s 2023-03-09T15:50:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:50:56 smithi122 ceph-mon[93869]: pgmap v2203: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 84 KiB/s wr, 19 op/s 2023-03-09T15:50:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:50:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3165.981044 secs 2023-03-09T15:50:56.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:50:56 smithi181 ceph-mon[94189]: pgmap v2203: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 84 KiB/s wr, 19 op/s 2023-03-09T15:50:56.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:50:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3165.981044 secs 2023-03-09T15:50:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:50:56 smithi038 ceph-mon[92422]: pgmap v2203: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 84 KiB/s wr, 19 op/s 2023-03-09T15:50:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:50:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3165.981044 secs 2023-03-09T15:50:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:50:58 smithi122 ceph-mon[93869]: pgmap v2204: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 113 KiB/s wr, 10 op/s 2023-03-09T15:50:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:50:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3168.192153 secs 2023-03-09T15:50:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:50:58 smithi181 ceph-mon[94189]: pgmap v2204: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 113 KiB/s wr, 10 op/s 2023-03-09T15:50:58.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:50:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3168.192153 secs 2023-03-09T15:50:58.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:50:58 smithi038 ceph-mon[92422]: pgmap v2204: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 113 KiB/s wr, 10 op/s 2023-03-09T15:50:58.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:50:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3168.192153 secs 2023-03-09T15:50:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3630.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:51:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:51:00 smithi122 ceph-mon[93869]: pgmap v2205: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 55 KiB/s wr, 8 op/s 2023-03-09T15:51:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:51:00 smithi181 ceph-mon[94189]: pgmap v2205: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 55 KiB/s wr, 8 op/s 2023-03-09T15:51:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:51:00 smithi038 ceph-mon[92422]: pgmap v2205: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 55 KiB/s wr, 8 op/s 2023-03-09T15:51:01.358 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:51:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3170.981103 secs 2023-03-09T15:51:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:51:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3170.981103 secs 2023-03-09T15:51:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:51:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3170.981103 secs 2023-03-09T15:51:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:51:02 smithi122 ceph-mon[93869]: pgmap v2206: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 100 KiB/s wr, 8 op/s 2023-03-09T15:51:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:51:02 smithi181 ceph-mon[94189]: pgmap v2206: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 100 KiB/s wr, 8 op/s 2023-03-09T15:51:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:51:02 smithi038 ceph-mon[92422]: pgmap v2206: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 100 KiB/s wr, 8 op/s 2023-03-09T15:51:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:51:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3173.192263 secs 2023-03-09T15:51:03.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:51:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3173.192263 secs 2023-03-09T15:51:03.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:51:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3173.192263 secs 2023-03-09T15:51:03.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3635.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:51:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:51:04 smithi122 ceph-mon[93869]: pgmap v2207: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 88 KiB/s wr, 10 op/s 2023-03-09T15:51:04.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:51:04 smithi181 ceph-mon[94189]: pgmap v2207: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 88 KiB/s wr, 10 op/s 2023-03-09T15:51:04.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:51:04 smithi038 ceph-mon[92422]: pgmap v2207: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 88 KiB/s wr, 10 op/s 2023-03-09T15:51:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:51:06 smithi122 ceph-mon[93869]: pgmap v2208: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 113 KiB/s wr, 20 op/s 2023-03-09T15:51:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:51:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3175.981166 secs 2023-03-09T15:51:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:51:06 smithi181 ceph-mon[94189]: pgmap v2208: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 113 KiB/s wr, 20 op/s 2023-03-09T15:51:06.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:51:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3175.981166 secs 2023-03-09T15:51:06.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:51:06 smithi038 ceph-mon[92422]: pgmap v2208: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 113 KiB/s wr, 20 op/s 2023-03-09T15:51:06.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:51:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3175.981166 secs 2023-03-09T15:51:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:51:08 smithi122 ceph-mon[93869]: pgmap v2209: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 147 KiB/s wr, 24 op/s 2023-03-09T15:51:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:51:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3178.192403 secs 2023-03-09T15:51:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:51:08 smithi181 ceph-mon[94189]: pgmap v2209: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 147 KiB/s wr, 24 op/s 2023-03-09T15:51:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:51:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3178.192403 secs 2023-03-09T15:51:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:51:08 smithi038 ceph-mon[92422]: pgmap v2209: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 147 KiB/s wr, 24 op/s 2023-03-09T15:51:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:51:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3178.192403 secs 2023-03-09T15:51:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3640.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:51:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:51:10 smithi122 ceph-mon[93869]: pgmap v2210: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 112 KiB/s wr, 23 op/s 2023-03-09T15:51:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:51:10 smithi122 ceph-mon[93869]: osdmap e352: 12 total, 12 up, 12 in 2023-03-09T15:51:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:51:10 smithi181 ceph-mon[94189]: pgmap v2210: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 112 KiB/s wr, 23 op/s 2023-03-09T15:51:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:51:10 smithi181 ceph-mon[94189]: osdmap e352: 12 total, 12 up, 12 in 2023-03-09T15:51:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:51:10 smithi038 ceph-mon[92422]: pgmap v2210: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 112 KiB/s wr, 23 op/s 2023-03-09T15:51:10.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:51:10 smithi038 ceph-mon[92422]: osdmap e352: 12 total, 12 up, 12 in 2023-03-09T15:51:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:51:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3180.981223 secs 2023-03-09T15:51:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:51:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3180.981223 secs 2023-03-09T15:51:11.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:51:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3180.981223 secs 2023-03-09T15:51:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:51:12 smithi122 ceph-mon[93869]: pgmap v2212: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 77 KiB/s wr, 26 op/s 2023-03-09T15:51:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:51:12 smithi181 ceph-mon[94189]: pgmap v2212: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 77 KiB/s wr, 26 op/s 2023-03-09T15:51:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:51:12 smithi038 ceph-mon[92422]: pgmap v2212: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 77 KiB/s wr, 26 op/s 2023-03-09T15:51:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:51:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3183.192520 secs 2023-03-09T15:51:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:51:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3183.192520 secs 2023-03-09T15:51:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:51:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3183.192520 secs 2023-03-09T15:51:13.801 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3645.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:51:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:51:14 smithi122 ceph-mon[93869]: pgmap v2213: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 77 KiB/s wr, 25 op/s 2023-03-09T15:51:14.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:51:14 smithi181 ceph-mon[94189]: pgmap v2213: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 77 KiB/s wr, 25 op/s 2023-03-09T15:51:14.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:51:14 smithi038 ceph-mon[92422]: pgmap v2213: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 77 KiB/s wr, 25 op/s 2023-03-09T15:51:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:51:16 smithi122 ceph-mon[93869]: pgmap v2214: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 47 KiB/s wr, 10 op/s 2023-03-09T15:51:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:51:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3185.981283 secs 2023-03-09T15:51:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:51:16 smithi181 ceph-mon[94189]: pgmap v2214: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 47 KiB/s wr, 10 op/s 2023-03-09T15:51:16.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:51:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3185.981283 secs 2023-03-09T15:51:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:51:16 smithi038 ceph-mon[92422]: pgmap v2214: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 47 KiB/s wr, 10 op/s 2023-03-09T15:51:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:51:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3185.981283 secs 2023-03-09T15:51:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:51:18 smithi122 ceph-mon[93869]: pgmap v2215: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.8 KiB/s wr, 7 op/s 2023-03-09T15:51:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:51:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3188.192657 secs 2023-03-09T15:51:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:51:18 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:51:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:51:18 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:51:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:51:18 smithi181 ceph-mon[94189]: pgmap v2215: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.8 KiB/s wr, 7 op/s 2023-03-09T15:51:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:51:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3188.192657 secs 2023-03-09T15:51:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:51:18 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:51:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:51:18 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:51:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:51:18 smithi038 ceph-mon[92422]: pgmap v2215: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.8 KiB/s wr, 7 op/s 2023-03-09T15:51:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:51:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3188.192657 secs 2023-03-09T15:51:18.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:51:18 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:51:18.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:51:18 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:51:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3650.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:51:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:51:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:51:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:51:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:51:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:51:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:51:19.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:51:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:51:19.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:51:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:51:19.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:51:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:51:19.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:51:19 smithi122 ceph-mon[93869]: osdmap e353: 12 total, 12 up, 12 in 2023-03-09T15:51:19.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:51:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:51:19.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:51:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:51:19.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:51:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:51:19.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:51:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:51:19.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:51:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:51:19.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:51:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:51:19.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:51:19 smithi181 ceph-mon[94189]: osdmap e353: 12 total, 12 up, 12 in 2023-03-09T15:51:19.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:51:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:51:19.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:51:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:51:19.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:51:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:51:19.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:51:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:51:19.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:51:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:51:19.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:51:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:51:19.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:51:19 smithi038 ceph-mon[92422]: osdmap e353: 12 total, 12 up, 12 in 2023-03-09T15:51:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:51:20 smithi122 ceph-mon[93869]: pgmap v2216: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2023-03-09T15:51:20.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:51:20 smithi181 ceph-mon[94189]: pgmap v2216: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2023-03-09T15:51:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:51:20 smithi038 ceph-mon[92422]: pgmap v2216: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2023-03-09T15:51:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:51:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3190.981342 secs 2023-03-09T15:51:21.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:51:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3190.981342 secs 2023-03-09T15:51:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:51:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3190.981342 secs 2023-03-09T15:51:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:51:22 smithi122 ceph-mon[93869]: pgmap v2218: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T15:51:22.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:51:22 smithi181 ceph-mon[94189]: pgmap v2218: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T15:51:22.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:51:22 smithi038 ceph-mon[92422]: pgmap v2218: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T15:51:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:51:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3193.192755 secs 2023-03-09T15:51:23.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:51:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3193.192755 secs 2023-03-09T15:51:23.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:51:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3193.192755 secs 2023-03-09T15:51:23.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3655.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:51:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:51:24 smithi122 ceph-mon[93869]: pgmap v2219: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 8 op/s 2023-03-09T15:51:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:51:24 smithi181 ceph-mon[94189]: pgmap v2219: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 8 op/s 2023-03-09T15:51:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:51:24 smithi038 ceph-mon[92422]: pgmap v2219: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 8 op/s 2023-03-09T15:51:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:51:25 smithi122 ceph-mon[93869]: pgmap v2220: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 8 op/s 2023-03-09T15:51:25.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:51:25 smithi181 ceph-mon[94189]: pgmap v2220: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 8 op/s 2023-03-09T15:51:25.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:51:25 smithi038 ceph-mon[92422]: pgmap v2220: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 8 op/s 2023-03-09T15:51:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:51:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3195.981402 secs 2023-03-09T15:51:26.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:51:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3195.981402 secs 2023-03-09T15:51:26.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:51:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3195.981402 secs 2023-03-09T15:51:27.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:51:27 smithi122 ceph-mon[93869]: pgmap v2221: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 8 op/s 2023-03-09T15:51:27.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:51:27 smithi181 ceph-mon[94189]: pgmap v2221: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 8 op/s 2023-03-09T15:51:27.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:51:27 smithi038 ceph-mon[92422]: pgmap v2221: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 8 op/s 2023-03-09T15:51:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:51:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3198.192872 secs 2023-03-09T15:51:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:51:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3198.192872 secs 2023-03-09T15:51:28.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:51:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3198.192872 secs 2023-03-09T15:51:28.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3660.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:51:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:51:29 smithi122 ceph-mon[93869]: pgmap v2222: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T15:51:30.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:51:29 smithi181 ceph-mon[94189]: pgmap v2222: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T15:51:30.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:51:29 smithi038 ceph-mon[92422]: pgmap v2222: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T15:51:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:51:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3200.981462 secs 2023-03-09T15:51:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:51:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3200.981462 secs 2023-03-09T15:51:31.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:51:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3200.981462 secs 2023-03-09T15:51:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:51:31 smithi122 ceph-mon[93869]: pgmap v2223: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 73 KiB/s wr, 8 op/s 2023-03-09T15:51:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:51:31 smithi181 ceph-mon[94189]: pgmap v2223: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 73 KiB/s wr, 8 op/s 2023-03-09T15:51:32.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:51:31 smithi038 ceph-mon[92422]: pgmap v2223: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 73 KiB/s wr, 8 op/s 2023-03-09T15:51:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:51:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3203.192986 secs 2023-03-09T15:51:33.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:51:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3203.192986 secs 2023-03-09T15:51:33.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:51:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3203.192986 secs 2023-03-09T15:51:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3665.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:51:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:51:33 smithi122 ceph-mon[93869]: pgmap v2224: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 48 KiB/s wr, 9 op/s 2023-03-09T15:51:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:51:33 smithi181 ceph-mon[94189]: pgmap v2224: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 48 KiB/s wr, 9 op/s 2023-03-09T15:51:34.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:51:33 smithi038 ceph-mon[92422]: pgmap v2224: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 48 KiB/s wr, 9 op/s 2023-03-09T15:51:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:51:35 smithi122 ceph-mon[93869]: pgmap v2225: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 48 KiB/s wr, 7 op/s 2023-03-09T15:51:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:51:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3205.981523 secs 2023-03-09T15:51:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:51:35 smithi181 ceph-mon[94189]: pgmap v2225: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 48 KiB/s wr, 7 op/s 2023-03-09T15:51:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:51:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3205.981523 secs 2023-03-09T15:51:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:51:35 smithi038 ceph-mon[92422]: pgmap v2225: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 48 KiB/s wr, 7 op/s 2023-03-09T15:51:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:51:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3205.981523 secs 2023-03-09T15:51:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:51:37 smithi122 ceph-mon[93869]: pgmap v2226: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 90 KiB/s wr, 9 op/s 2023-03-09T15:51:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:51:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3208.193121 secs 2023-03-09T15:51:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:51:37 smithi181 ceph-mon[94189]: pgmap v2226: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 90 KiB/s wr, 9 op/s 2023-03-09T15:51:38.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:51:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3208.193121 secs 2023-03-09T15:51:38.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:51:37 smithi038 ceph-mon[92422]: pgmap v2226: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 90 KiB/s wr, 9 op/s 2023-03-09T15:51:38.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:51:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3208.193121 secs 2023-03-09T15:51:38.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3670.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:51:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:51:39 smithi122 ceph-mon[93869]: pgmap v2227: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 93 KiB/s wr, 8 op/s 2023-03-09T15:51:40.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:51:39 smithi181 ceph-mon[94189]: pgmap v2227: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 93 KiB/s wr, 8 op/s 2023-03-09T15:51:40.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:51:39 smithi038 ceph-mon[92422]: pgmap v2227: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 93 KiB/s wr, 8 op/s 2023-03-09T15:51:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:51:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3210.981593 secs 2023-03-09T15:51:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:51:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3210.981593 secs 2023-03-09T15:51:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:51:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3210.981593 secs 2023-03-09T15:51:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:51:41 smithi122 ceph-mon[93869]: pgmap v2228: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 103 KiB/s wr, 8 op/s 2023-03-09T15:51:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:51:41 smithi181 ceph-mon[94189]: pgmap v2228: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 103 KiB/s wr, 8 op/s 2023-03-09T15:51:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:51:41 smithi038 ceph-mon[92422]: pgmap v2228: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 103 KiB/s wr, 8 op/s 2023-03-09T15:51:43.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:51:42 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:51:43.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:51:42 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:51:43.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:51:42 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:51:43.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3675.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:51:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:51:43 smithi122 ceph-mon[93869]: pgmap v2229: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 56 KiB/s wr, 10 op/s 2023-03-09T15:51:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:51:43 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:51:44.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:51:43 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:51:44.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:51:43 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:51:44.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:51:43 smithi181 ceph-mon[94189]: pgmap v2229: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 56 KiB/s wr, 10 op/s 2023-03-09T15:51:44.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:51:43 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:51:44.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:51:43 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:51:44.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:51:43 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:51:44.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:51:43 smithi038 ceph-mon[92422]: pgmap v2229: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 56 KiB/s wr, 10 op/s 2023-03-09T15:51:44.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:51:43 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:51:44.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:51:43 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:51:44.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:51:43 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:51:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:51:45 smithi122 ceph-mon[93869]: pgmap v2230: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 63 KiB/s wr, 9 op/s 2023-03-09T15:51:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:51:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3215.981654 secs 2023-03-09T15:51:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:51:45 smithi181 ceph-mon[94189]: pgmap v2230: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 63 KiB/s wr, 9 op/s 2023-03-09T15:51:46.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:51:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3215.981654 secs 2023-03-09T15:51:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:51:45 smithi038 ceph-mon[92422]: pgmap v2230: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 63 KiB/s wr, 9 op/s 2023-03-09T15:51:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:51:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3215.981654 secs 2023-03-09T15:51:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:51:47 smithi122 ceph-mon[93869]: pgmap v2231: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 150 KiB/s wr, 12 op/s 2023-03-09T15:51:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:51:47 smithi181 ceph-mon[94189]: pgmap v2231: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 150 KiB/s wr, 12 op/s 2023-03-09T15:51:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:51:47 smithi038 ceph-mon[92422]: pgmap v2231: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 150 KiB/s wr, 12 op/s 2023-03-09T15:51:48.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3680.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:51:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:51:49 smithi122 ceph-mon[93869]: pgmap v2232: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 109 KiB/s wr, 11 op/s 2023-03-09T15:51:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:51:49 smithi181 ceph-mon[94189]: pgmap v2232: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 109 KiB/s wr, 11 op/s 2023-03-09T15:51:50.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:51:49 smithi038 ceph-mon[92422]: pgmap v2232: 129 pgs: 129 active+clean; 8.5 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 109 KiB/s wr, 11 op/s 2023-03-09T15:51:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:51:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3220.981714 secs 2023-03-09T15:51:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:51:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3220.981714 secs 2023-03-09T15:51:51.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:51:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3220.981714 secs 2023-03-09T15:51:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:51:51 smithi122 ceph-mon[93869]: pgmap v2233: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 153 KiB/s wr, 11 op/s 2023-03-09T15:51:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:51:51 smithi181 ceph-mon[94189]: pgmap v2233: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 153 KiB/s wr, 11 op/s 2023-03-09T15:51:52.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:51:51 smithi038 ceph-mon[92422]: pgmap v2233: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 153 KiB/s wr, 11 op/s 2023-03-09T15:51:53.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3685.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:51:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:51:53 smithi122 ceph-mon[93869]: pgmap v2234: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 144 KiB/s wr, 12 op/s 2023-03-09T15:51:54.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:51:53 smithi181 ceph-mon[94189]: pgmap v2234: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 144 KiB/s wr, 12 op/s 2023-03-09T15:51:54.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:51:53 smithi038 ceph-mon[92422]: pgmap v2234: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 144 KiB/s wr, 12 op/s 2023-03-09T15:51:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:51:55 smithi122 ceph-mon[93869]: pgmap v2235: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 148 KiB/s wr, 11 op/s 2023-03-09T15:51:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:51:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3225.981773 secs 2023-03-09T15:51:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:51:55 smithi181 ceph-mon[94189]: pgmap v2235: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 148 KiB/s wr, 11 op/s 2023-03-09T15:51:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:51:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3225.981773 secs 2023-03-09T15:51:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:51:55 smithi038 ceph-mon[92422]: pgmap v2235: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 148 KiB/s wr, 11 op/s 2023-03-09T15:51:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:51:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3225.981773 secs 2023-03-09T15:51:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:51:57 smithi122 ceph-mon[93869]: pgmap v2236: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 187 KiB/s wr, 12 op/s 2023-03-09T15:51:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:51:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3228.193566 secs 2023-03-09T15:51:58.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:51:57 smithi181 ceph-mon[94189]: pgmap v2236: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 187 KiB/s wr, 12 op/s 2023-03-09T15:51:58.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:51:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3228.193566 secs 2023-03-09T15:51:58.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:51:57 smithi038 ceph-mon[92422]: pgmap v2236: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 187 KiB/s wr, 12 op/s 2023-03-09T15:51:58.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:51:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3228.193566 secs 2023-03-09T15:51:58.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3690.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:52:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:51:59 smithi122 ceph-mon[93869]: pgmap v2237: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 104 KiB/s wr, 9 op/s 2023-03-09T15:52:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:51:59 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:52:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:51:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:52:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:51:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:52:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:51:59 smithi181 ceph-mon[94189]: pgmap v2237: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 104 KiB/s wr, 9 op/s 2023-03-09T15:52:00.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:51:59 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:52:00.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:51:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:52:00.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:51:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:52:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:51:59 smithi038 ceph-mon[92422]: pgmap v2237: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 104 KiB/s wr, 9 op/s 2023-03-09T15:52:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:51:59 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:52:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:51:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:52:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:51:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:52:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:52:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3230.981835 secs 2023-03-09T15:52:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:52:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3230.981835 secs 2023-03-09T15:52:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:52:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3230.981835 secs 2023-03-09T15:52:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:52:01 smithi122 ceph-mon[93869]: pgmap v2238: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 151 KiB/s wr, 9 op/s 2023-03-09T15:52:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:52:01 smithi181 ceph-mon[94189]: pgmap v2238: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 151 KiB/s wr, 9 op/s 2023-03-09T15:52:02.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:52:01 smithi038 ceph-mon[92422]: pgmap v2238: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 151 KiB/s wr, 9 op/s 2023-03-09T15:52:03.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:52:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3233.193705 secs 2023-03-09T15:52:03.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:52:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3233.193705 secs 2023-03-09T15:52:03.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:52:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3233.193705 secs 2023-03-09T15:52:03.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3695.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:52:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:52:03 smithi122 ceph-mon[93869]: pgmap v2239: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 103 KiB/s wr, 10 op/s 2023-03-09T15:52:04.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:52:03 smithi181 ceph-mon[94189]: pgmap v2239: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 103 KiB/s wr, 10 op/s 2023-03-09T15:52:04.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:52:03 smithi038 ceph-mon[92422]: pgmap v2239: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 103 KiB/s wr, 10 op/s 2023-03-09T15:52:05.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:52:04 smithi122 ceph-mon[93869]: osdmap e354: 12 total, 12 up, 12 in 2023-03-09T15:52:05.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:52:04 smithi181 ceph-mon[94189]: osdmap e354: 12 total, 12 up, 12 in 2023-03-09T15:52:05.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:52:04 smithi038 ceph-mon[92422]: osdmap e354: 12 total, 12 up, 12 in 2023-03-09T15:52:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:52:05 smithi122 ceph-mon[93869]: pgmap v2241: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 149 KiB/s wr, 23 op/s 2023-03-09T15:52:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:52:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3235.981919 secs 2023-03-09T15:52:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:52:05 smithi181 ceph-mon[94189]: pgmap v2241: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 149 KiB/s wr, 23 op/s 2023-03-09T15:52:06.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:52:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3235.981919 secs 2023-03-09T15:52:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:52:05 smithi038 ceph-mon[92422]: pgmap v2241: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 149 KiB/s wr, 23 op/s 2023-03-09T15:52:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:52:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3235.981919 secs 2023-03-09T15:52:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:52:07 smithi122 ceph-mon[93869]: pgmap v2242: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 171 KiB/s wr, 27 op/s 2023-03-09T15:52:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:52:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3238.193823 secs 2023-03-09T15:52:08.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:52:07 smithi181 ceph-mon[94189]: pgmap v2242: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 171 KiB/s wr, 27 op/s 2023-03-09T15:52:08.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:52:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3238.193823 secs 2023-03-09T15:52:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:52:07 smithi038 ceph-mon[92422]: pgmap v2242: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 171 KiB/s wr, 27 op/s 2023-03-09T15:52:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:52:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3238.193823 secs 2023-03-09T15:52:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3700.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:52:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:52:09 smithi122 ceph-mon[93869]: pgmap v2243: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 167 KiB/s wr, 26 op/s 2023-03-09T15:52:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:52:09 smithi181 ceph-mon[94189]: pgmap v2243: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 167 KiB/s wr, 26 op/s 2023-03-09T15:52:10.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:52:09 smithi038 ceph-mon[92422]: pgmap v2243: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 167 KiB/s wr, 26 op/s 2023-03-09T15:52:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:52:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3240.982003 secs 2023-03-09T15:52:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:52:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3240.982003 secs 2023-03-09T15:52:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:52:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3240.982003 secs 2023-03-09T15:52:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:52:11 smithi122 ceph-mon[93869]: pgmap v2244: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 110 KiB/s wr, 26 op/s 2023-03-09T15:52:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:52:11 smithi181 ceph-mon[94189]: pgmap v2244: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 110 KiB/s wr, 26 op/s 2023-03-09T15:52:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:52:11 smithi038 ceph-mon[92422]: pgmap v2244: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 110 KiB/s wr, 26 op/s 2023-03-09T15:52:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:52:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3243.193945 secs 2023-03-09T15:52:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:52:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3243.193945 secs 2023-03-09T15:52:13.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:52:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3243.193945 secs 2023-03-09T15:52:13.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3705.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:52:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:52:13 smithi122 ceph-mon[93869]: pgmap v2245: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 110 KiB/s wr, 26 op/s 2023-03-09T15:52:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:52:13 smithi122 ceph-mon[93869]: osdmap e355: 12 total, 12 up, 12 in 2023-03-09T15:52:14.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:52:13 smithi181 ceph-mon[94189]: pgmap v2245: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 110 KiB/s wr, 26 op/s 2023-03-09T15:52:14.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:52:13 smithi181 ceph-mon[94189]: osdmap e355: 12 total, 12 up, 12 in 2023-03-09T15:52:14.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:52:13 smithi038 ceph-mon[92422]: pgmap v2245: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 110 KiB/s wr, 26 op/s 2023-03-09T15:52:14.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:52:13 smithi038 ceph-mon[92422]: osdmap e355: 12 total, 12 up, 12 in 2023-03-09T15:52:15.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:52:14 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3245.982090 secs 2023-03-09T15:52:15.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:52:14 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3245.982090 secs 2023-03-09T15:52:15.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:52:14 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3245.982090 secs 2023-03-09T15:52:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:52:15 smithi122 ceph-mon[93869]: pgmap v2247: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 84 KiB/s wr, 11 op/s 2023-03-09T15:52:16.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:52:15 smithi181 ceph-mon[94189]: pgmap v2247: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 84 KiB/s wr, 11 op/s 2023-03-09T15:52:16.255 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:52:15 smithi038 ceph-mon[92422]: pgmap v2247: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 84 KiB/s wr, 11 op/s 2023-03-09T15:52:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:52:17 smithi122 ceph-mon[93869]: pgmap v2248: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T15:52:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:52:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3248.194067 secs 2023-03-09T15:52:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:52:17 smithi181 ceph-mon[94189]: pgmap v2248: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T15:52:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:52:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3248.194067 secs 2023-03-09T15:52:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:52:17 smithi038 ceph-mon[92422]: pgmap v2248: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T15:52:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:52:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3248.194067 secs 2023-03-09T15:52:18.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3710.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:52:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:52:19 smithi122 ceph-mon[93869]: pgmap v2249: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 8 op/s 2023-03-09T15:52:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:52:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:52:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:52:19 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3250.982152 secs 2023-03-09T15:52:20.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:52:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:52:20.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:52:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:52:20.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:52:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:52:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:52:19 smithi181 ceph-mon[94189]: pgmap v2249: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 8 op/s 2023-03-09T15:52:20.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:52:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:52:20.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:52:19 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3250.982152 secs 2023-03-09T15:52:20.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:52:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:52:20.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:52:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:52:20.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:52:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:52:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:52:19 smithi038 ceph-mon[92422]: pgmap v2249: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 8 op/s 2023-03-09T15:52:20.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:52:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:52:20.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:52:19 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3250.982152 secs 2023-03-09T15:52:20.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:52:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:52:20.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:52:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:52:20.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:52:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:52:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:52:21 smithi122 ceph-mon[93869]: pgmap v2250: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T15:52:22.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:52:21 smithi181 ceph-mon[94189]: pgmap v2250: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T15:52:22.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:52:21 smithi038 ceph-mon[92422]: pgmap v2250: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T15:52:23.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:52:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3253.194167 secs 2023-03-09T15:52:23.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:52:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3253.194167 secs 2023-03-09T15:52:23.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:52:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3253.194167 secs 2023-03-09T15:52:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3715.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:52:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:52:23 smithi122 ceph-mon[93869]: pgmap v2251: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T15:52:24.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:52:23 smithi181 ceph-mon[94189]: pgmap v2251: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T15:52:24.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:52:23 smithi038 ceph-mon[92422]: pgmap v2251: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T15:52:25.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:52:24 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3255.982212 secs 2023-03-09T15:52:25.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:52:24 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3255.982212 secs 2023-03-09T15:52:25.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:52:24 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3255.982212 secs 2023-03-09T15:52:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:52:25 smithi122 ceph-mon[93869]: pgmap v2252: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T15:52:26.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:52:25 smithi181 ceph-mon[94189]: pgmap v2252: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T15:52:26.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:52:25 smithi038 ceph-mon[92422]: pgmap v2252: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T15:52:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:52:27 smithi122 ceph-mon[93869]: pgmap v2253: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 35 KiB/s wr, 8 op/s 2023-03-09T15:52:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:52:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3258.194290 secs 2023-03-09T15:52:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:52:27 smithi181 ceph-mon[94189]: pgmap v2253: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 35 KiB/s wr, 8 op/s 2023-03-09T15:52:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:52:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3258.194290 secs 2023-03-09T15:52:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:52:27 smithi038 ceph-mon[92422]: pgmap v2253: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 35 KiB/s wr, 8 op/s 2023-03-09T15:52:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:52:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3258.194290 secs 2023-03-09T15:52:28.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3720.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:52:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:52:29 smithi122 ceph-mon[93869]: pgmap v2254: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T15:52:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:52:29 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3260.982296 secs 2023-03-09T15:52:30.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:52:29 smithi181 ceph-mon[94189]: pgmap v2254: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T15:52:30.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:52:29 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3260.982296 secs 2023-03-09T15:52:30.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:52:29 smithi038 ceph-mon[92422]: pgmap v2254: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T15:52:30.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:52:29 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3260.982296 secs 2023-03-09T15:52:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:52:31 smithi122 ceph-mon[93869]: pgmap v2255: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 46 KiB/s wr, 6 op/s 2023-03-09T15:52:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:52:31 smithi181 ceph-mon[94189]: pgmap v2255: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 46 KiB/s wr, 6 op/s 2023-03-09T15:52:32.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:52:31 smithi038 ceph-mon[92422]: pgmap v2255: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 46 KiB/s wr, 6 op/s 2023-03-09T15:52:33.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3725.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:52:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:52:33 smithi122 ceph-mon[93869]: pgmap v2256: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 46 KiB/s wr, 8 op/s 2023-03-09T15:52:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:52:33 smithi181 ceph-mon[94189]: pgmap v2256: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 46 KiB/s wr, 8 op/s 2023-03-09T15:52:34.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:52:33 smithi038 ceph-mon[92422]: pgmap v2256: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 46 KiB/s wr, 8 op/s 2023-03-09T15:52:35.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:52:34 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3265.982356 secs 2023-03-09T15:52:35.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:52:34 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:52:35.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:52:34 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3265.982356 secs 2023-03-09T15:52:35.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:52:34 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:52:35.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:52:34 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3265.982356 secs 2023-03-09T15:52:35.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:52:34 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:52:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:52:35 smithi122 ceph-mon[93869]: pgmap v2257: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 7 op/s 2023-03-09T15:52:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:52:35 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:52:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:52:35 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:52:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:52:35 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:52:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:52:35 smithi181 ceph-mon[94189]: pgmap v2257: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 7 op/s 2023-03-09T15:52:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:52:35 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:52:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:52:35 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:52:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:52:35 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:52:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:52:35 smithi038 ceph-mon[92422]: pgmap v2257: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 7 op/s 2023-03-09T15:52:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:52:35 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:52:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:52:35 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:52:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:52:35 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:52:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:52:37 smithi122 ceph-mon[93869]: pgmap v2258: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 120 KiB/s wr, 10 op/s 2023-03-09T15:52:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:52:37 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 3268.194507 secs 2023-03-09T15:52:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:52:37 smithi181 ceph-mon[94189]: pgmap v2258: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 120 KiB/s wr, 10 op/s 2023-03-09T15:52:38.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:52:37 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 3268.194507 secs 2023-03-09T15:52:38.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:52:37 smithi038 ceph-mon[92422]: pgmap v2258: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 120 KiB/s wr, 10 op/s 2023-03-09T15:52:38.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:52:37 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 3268.194507 secs 2023-03-09T15:52:38.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3730.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:52:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:52:39 smithi122 ceph-mon[93869]: pgmap v2259: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 103 KiB/s wr, 9 op/s 2023-03-09T15:52:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:52:39 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:52:40.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:52:39 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:52:40.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:52:39 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3270.982416 secs 2023-03-09T15:52:40.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:52:39 smithi181 ceph-mon[94189]: pgmap v2259: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 103 KiB/s wr, 9 op/s 2023-03-09T15:52:40.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:52:39 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:52:40.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:52:39 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:52:40.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:52:39 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3270.982416 secs 2023-03-09T15:52:40.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:52:39 smithi038 ceph-mon[92422]: pgmap v2259: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 103 KiB/s wr, 9 op/s 2023-03-09T15:52:40.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:52:39 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:52:40.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:52:39 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:52:40.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:52:39 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3270.982416 secs 2023-03-09T15:52:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:52:41 smithi122 ceph-mon[93869]: pgmap v2260: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 112 KiB/s wr, 9 op/s 2023-03-09T15:52:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:52:41 smithi181 ceph-mon[94189]: pgmap v2260: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 112 KiB/s wr, 9 op/s 2023-03-09T15:52:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:52:41 smithi038 ceph-mon[92422]: pgmap v2260: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 112 KiB/s wr, 9 op/s 2023-03-09T15:52:43.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:52:42 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 3273.194645 secs 2023-03-09T15:52:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:52:42 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 3273.194645 secs 2023-03-09T15:52:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:52:42 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 3273.194645 secs 2023-03-09T15:52:43.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3735.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:52:44.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:52:43 smithi038 ceph-mon[92422]: pgmap v2261: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 88 KiB/s wr, 11 op/s 2023-03-09T15:52:44.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:52:43 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:52:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:52:43 smithi122 ceph-mon[93869]: pgmap v2261: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 88 KiB/s wr, 11 op/s 2023-03-09T15:52:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:52:43 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:52:44.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:52:43 smithi181 ceph-mon[94189]: pgmap v2261: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 88 KiB/s wr, 11 op/s 2023-03-09T15:52:44.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:52:43 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:52:45.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:52:44 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3275.982493 secs 2023-03-09T15:52:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:52:44 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3275.982493 secs 2023-03-09T15:52:45.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:52:44 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3275.982493 secs 2023-03-09T15:52:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:52:45 smithi038 ceph-mon[92422]: pgmap v2262: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 169 KiB/s wr, 11 op/s 2023-03-09T15:52:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:52:45 smithi122 ceph-mon[93869]: pgmap v2262: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 169 KiB/s wr, 11 op/s 2023-03-09T15:52:46.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:52:45 smithi181 ceph-mon[94189]: pgmap v2262: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 169 KiB/s wr, 11 op/s 2023-03-09T15:52:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:52:48 smithi122 ceph-mon[93869]: pgmap v2263: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 163 KiB/s wr, 12 op/s 2023-03-09T15:52:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:52:48 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 3278.194773 secs 2023-03-09T15:52:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:52:48 smithi181 ceph-mon[94189]: pgmap v2263: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 163 KiB/s wr, 12 op/s 2023-03-09T15:52:48.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:52:48 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 3278.194773 secs 2023-03-09T15:52:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:52:48 smithi038 ceph-mon[92422]: pgmap v2263: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 163 KiB/s wr, 12 op/s 2023-03-09T15:52:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:52:48 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 3278.194773 secs 2023-03-09T15:52:48.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3740.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:52:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:52:50 smithi122 ceph-mon[93869]: pgmap v2264: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 96 KiB/s wr, 10 op/s 2023-03-09T15:52:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:52:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3280.982552 secs 2023-03-09T15:52:50.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:52:50 smithi181 ceph-mon[94189]: pgmap v2264: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 96 KiB/s wr, 10 op/s 2023-03-09T15:52:50.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:52:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3280.982552 secs 2023-03-09T15:52:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:52:50 smithi038 ceph-mon[92422]: pgmap v2264: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 96 KiB/s wr, 10 op/s 2023-03-09T15:52:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:52:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3280.982552 secs 2023-03-09T15:52:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:52:52 smithi122 ceph-mon[93869]: pgmap v2265: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 123 KiB/s wr, 9 op/s 2023-03-09T15:52:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:52:52 smithi181 ceph-mon[94189]: pgmap v2265: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 123 KiB/s wr, 9 op/s 2023-03-09T15:52:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:52:52 smithi038 ceph-mon[92422]: pgmap v2265: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 123 KiB/s wr, 9 op/s 2023-03-09T15:52:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:52:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3283.194890 secs 2023-03-09T15:52:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:52:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3283.194890 secs 2023-03-09T15:52:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:52:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3283.194890 secs 2023-03-09T15:52:53.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3745.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:52:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:52:54 smithi122 ceph-mon[93869]: pgmap v2266: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 113 KiB/s wr, 10 op/s 2023-03-09T15:52:54.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:52:54 smithi181 ceph-mon[94189]: pgmap v2266: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 113 KiB/s wr, 10 op/s 2023-03-09T15:52:54.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:52:54 smithi038 ceph-mon[92422]: pgmap v2266: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 113 KiB/s wr, 10 op/s 2023-03-09T15:52:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:52:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3285.982612 secs 2023-03-09T15:52:55.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:52:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3285.982612 secs 2023-03-09T15:52:55.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:52:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3285.982612 secs 2023-03-09T15:52:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:52:56 smithi122 ceph-mon[93869]: pgmap v2267: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 144 KiB/s wr, 10 op/s 2023-03-09T15:52:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:52:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:52:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:52:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:52:56.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:52:56 smithi181 ceph-mon[94189]: pgmap v2267: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 144 KiB/s wr, 10 op/s 2023-03-09T15:52:56.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:52:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:52:56.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:52:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:52:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:52:56 smithi038 ceph-mon[92422]: pgmap v2267: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 144 KiB/s wr, 10 op/s 2023-03-09T15:52:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:52:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:52:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:52:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:52:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:52:58 smithi122 ceph-mon[93869]: pgmap v2268: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 63 KiB/s wr, 10 op/s 2023-03-09T15:52:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:52:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3288.194955 secs 2023-03-09T15:52:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:52:58 smithi181 ceph-mon[94189]: pgmap v2268: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 63 KiB/s wr, 10 op/s 2023-03-09T15:52:58.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:52:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3288.194955 secs 2023-03-09T15:52:58.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:52:58 smithi038 ceph-mon[92422]: pgmap v2268: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 63 KiB/s wr, 10 op/s 2023-03-09T15:52:58.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:52:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3288.194955 secs 2023-03-09T15:52:58.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3750.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:53:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:53:00 smithi122 ceph-mon[93869]: pgmap v2269: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 66 KiB/s wr, 8 op/s 2023-03-09T15:53:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:53:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3290.982707 secs 2023-03-09T15:53:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:53:00 smithi181 ceph-mon[94189]: pgmap v2269: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 66 KiB/s wr, 8 op/s 2023-03-09T15:53:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:53:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3290.982707 secs 2023-03-09T15:53:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:53:00 smithi038 ceph-mon[92422]: pgmap v2269: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 66 KiB/s wr, 8 op/s 2023-03-09T15:53:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:53:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3290.982707 secs 2023-03-09T15:53:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:53:02 smithi122 ceph-mon[93869]: pgmap v2270: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 114 KiB/s wr, 8 op/s 2023-03-09T15:53:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:53:02 smithi181 ceph-mon[94189]: pgmap v2270: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 114 KiB/s wr, 8 op/s 2023-03-09T15:53:02.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:53:02 smithi038 ceph-mon[92422]: pgmap v2270: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 114 KiB/s wr, 8 op/s 2023-03-09T15:53:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:53:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3293.195090 secs 2023-03-09T15:53:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:53:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3293.195090 secs 2023-03-09T15:53:03.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:53:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3293.195090 secs 2023-03-09T15:53:03.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3755.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:53:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:53:04 smithi122 ceph-mon[93869]: pgmap v2271: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 84 KiB/s wr, 9 op/s 2023-03-09T15:53:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:53:04 smithi181 ceph-mon[94189]: pgmap v2271: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 84 KiB/s wr, 9 op/s 2023-03-09T15:53:04.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:53:04 smithi038 ceph-mon[92422]: pgmap v2271: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 84 KiB/s wr, 9 op/s 2023-03-09T15:53:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:53:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3295.982765 secs 2023-03-09T15:53:05.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:53:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3295.982765 secs 2023-03-09T15:53:05.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:53:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3295.982765 secs 2023-03-09T15:53:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:53:06 smithi122 ceph-mon[93869]: pgmap v2272: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 109 KiB/s wr, 21 op/s 2023-03-09T15:53:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:53:06 smithi181 ceph-mon[94189]: pgmap v2272: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 109 KiB/s wr, 21 op/s 2023-03-09T15:53:06.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:53:06 smithi038 ceph-mon[92422]: pgmap v2272: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 109 KiB/s wr, 21 op/s 2023-03-09T15:53:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:53:08 smithi122 ceph-mon[93869]: pgmap v2273: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 101 KiB/s wr, 23 op/s 2023-03-09T15:53:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:53:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3298.195214 secs 2023-03-09T15:53:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:53:08 smithi181 ceph-mon[94189]: pgmap v2273: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 101 KiB/s wr, 23 op/s 2023-03-09T15:53:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:53:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3298.195214 secs 2023-03-09T15:53:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:53:08 smithi038 ceph-mon[92422]: pgmap v2273: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 101 KiB/s wr, 23 op/s 2023-03-09T15:53:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:53:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3298.195214 secs 2023-03-09T15:53:08.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3760.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:53:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:53:10 smithi122 ceph-mon[93869]: pgmap v2274: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 105 KiB/s wr, 22 op/s 2023-03-09T15:53:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:53:10 smithi122 ceph-mon[93869]: osdmap e356: 12 total, 12 up, 12 in 2023-03-09T15:53:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:53:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3300.982842 secs 2023-03-09T15:53:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:53:10 smithi181 ceph-mon[94189]: pgmap v2274: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 105 KiB/s wr, 22 op/s 2023-03-09T15:53:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:53:10 smithi181 ceph-mon[94189]: osdmap e356: 12 total, 12 up, 12 in 2023-03-09T15:53:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:53:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3300.982842 secs 2023-03-09T15:53:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:53:10 smithi038 ceph-mon[92422]: pgmap v2274: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 105 KiB/s wr, 22 op/s 2023-03-09T15:53:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:53:10 smithi038 ceph-mon[92422]: osdmap e356: 12 total, 12 up, 12 in 2023-03-09T15:53:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:53:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3300.982842 secs 2023-03-09T15:53:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:53:12 smithi122 ceph-mon[93869]: pgmap v2276: 129 pgs: 9 active+clean+snaptrim_wait, 120 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 64 KiB/s wr, 25 op/s 2023-03-09T15:53:12.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:53:12 smithi181 ceph-mon[94189]: pgmap v2276: 129 pgs: 9 active+clean+snaptrim_wait, 120 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 64 KiB/s wr, 25 op/s 2023-03-09T15:53:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:53:12 smithi038 ceph-mon[92422]: pgmap v2276: 129 pgs: 9 active+clean+snaptrim_wait, 120 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 64 KiB/s wr, 25 op/s 2023-03-09T15:53:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:53:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3303.195350 secs 2023-03-09T15:53:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:53:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3303.195350 secs 2023-03-09T15:53:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:53:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3303.195350 secs 2023-03-09T15:53:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3765.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:53:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:53:14 smithi122 ceph-mon[93869]: pgmap v2277: 129 pgs: 9 active+clean+snaptrim_wait, 120 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 64 KiB/s wr, 25 op/s 2023-03-09T15:53:14.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:53:14 smithi181 ceph-mon[94189]: pgmap v2277: 129 pgs: 9 active+clean+snaptrim_wait, 120 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 64 KiB/s wr, 25 op/s 2023-03-09T15:53:14.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:53:14 smithi038 ceph-mon[92422]: pgmap v2277: 129 pgs: 9 active+clean+snaptrim_wait, 120 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 64 KiB/s wr, 25 op/s 2023-03-09T15:53:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:53:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3305.982904 secs 2023-03-09T15:53:15.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:53:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3305.982904 secs 2023-03-09T15:53:15.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:53:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3305.982904 secs 2023-03-09T15:53:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:53:16 smithi122 ceph-mon[93869]: pgmap v2278: 129 pgs: 9 active+clean+snaptrim_wait, 120 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 33 KiB/s wr, 10 op/s 2023-03-09T15:53:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:53:16 smithi181 ceph-mon[94189]: pgmap v2278: 129 pgs: 9 active+clean+snaptrim_wait, 120 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 33 KiB/s wr, 10 op/s 2023-03-09T15:53:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:53:16 smithi038 ceph-mon[92422]: pgmap v2278: 129 pgs: 9 active+clean+snaptrim_wait, 120 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 33 KiB/s wr, 10 op/s 2023-03-09T15:53:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:53:18 smithi122 ceph-mon[93869]: pgmap v2279: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 6.2 KiB/s wr, 7 op/s 2023-03-09T15:53:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:53:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3308.195468 secs 2023-03-09T15:53:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:53:18 smithi181 ceph-mon[94189]: pgmap v2279: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 6.2 KiB/s wr, 7 op/s 2023-03-09T15:53:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:53:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3308.195468 secs 2023-03-09T15:53:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:53:18 smithi038 ceph-mon[92422]: pgmap v2279: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 6.2 KiB/s wr, 7 op/s 2023-03-09T15:53:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:53:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3308.195468 secs 2023-03-09T15:53:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3770.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:53:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:53:19 smithi122 ceph-mon[93869]: pgmap v2280: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2023-03-09T15:53:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:53:19 smithi122 ceph-mon[93869]: osdmap e357: 12 total, 12 up, 12 in 2023-03-09T15:53:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:53:19 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3310.982961 secs 2023-03-09T15:53:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:53:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:53:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:53:19 smithi181 ceph-mon[94189]: pgmap v2280: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2023-03-09T15:53:20.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:53:19 smithi181 ceph-mon[94189]: osdmap e357: 12 total, 12 up, 12 in 2023-03-09T15:53:20.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:53:19 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3310.982961 secs 2023-03-09T15:53:20.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:53:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:53:20.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:53:19 smithi038 ceph-mon[92422]: pgmap v2280: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2023-03-09T15:53:20.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:53:19 smithi038 ceph-mon[92422]: osdmap e357: 12 total, 12 up, 12 in 2023-03-09T15:53:20.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:53:19 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3310.982961 secs 2023-03-09T15:53:20.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:53:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:53:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:53:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:53:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:53:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:53:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:53:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:53:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:53:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:53:21.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:53:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:53:21.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:53:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:53:21.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:53:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:53:21.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:53:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:53:21.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:53:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:53:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:53:21 smithi122 ceph-mon[93869]: pgmap v2282: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 8 op/s 2023-03-09T15:53:22.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:53:21 smithi181 ceph-mon[94189]: pgmap v2282: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 8 op/s 2023-03-09T15:53:22.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:53:21 smithi038 ceph-mon[92422]: pgmap v2282: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 8 op/s 2023-03-09T15:53:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:53:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3313.195607 secs 2023-03-09T15:53:23.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:53:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3313.195607 secs 2023-03-09T15:53:23.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:53:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3313.195607 secs 2023-03-09T15:53:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3775.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:53:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:53:23 smithi122 ceph-mon[93869]: pgmap v2283: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 28 KiB/s wr, 8 op/s 2023-03-09T15:53:24.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:53:23 smithi181 ceph-mon[94189]: pgmap v2283: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 28 KiB/s wr, 8 op/s 2023-03-09T15:53:24.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:53:23 smithi038 ceph-mon[92422]: pgmap v2283: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 28 KiB/s wr, 8 op/s 2023-03-09T15:53:25.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:53:24 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3315.982982 secs 2023-03-09T15:53:25.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:53:24 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3315.982982 secs 2023-03-09T15:53:25.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:53:24 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3315.982982 secs 2023-03-09T15:53:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:53:25 smithi122 ceph-mon[93869]: pgmap v2284: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T15:53:26.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:53:25 smithi181 ceph-mon[94189]: pgmap v2284: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T15:53:26.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:53:25 smithi038 ceph-mon[92422]: pgmap v2284: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T15:53:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:53:27 smithi122 ceph-mon[93869]: pgmap v2285: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T15:53:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:53:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3318.195708 secs 2023-03-09T15:53:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:53:27 smithi181 ceph-mon[94189]: pgmap v2285: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T15:53:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:53:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3318.195708 secs 2023-03-09T15:53:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:53:27 smithi038 ceph-mon[92422]: pgmap v2285: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T15:53:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:53:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3318.195708 secs 2023-03-09T15:53:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3780.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:53:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:53:29 smithi122 ceph-mon[93869]: pgmap v2286: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T15:53:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:53:29 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3320.983041 secs 2023-03-09T15:53:30.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:53:29 smithi181 ceph-mon[94189]: pgmap v2286: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T15:53:30.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:53:29 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3320.983041 secs 2023-03-09T15:53:30.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:53:29 smithi038 ceph-mon[92422]: pgmap v2286: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T15:53:30.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:53:29 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3320.983041 secs 2023-03-09T15:53:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:53:31 smithi122 ceph-mon[93869]: pgmap v2287: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 72 KiB/s wr, 7 op/s 2023-03-09T15:53:32.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:53:31 smithi181 ceph-mon[94189]: pgmap v2287: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 72 KiB/s wr, 7 op/s 2023-03-09T15:53:32.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:53:31 smithi038 ceph-mon[92422]: pgmap v2287: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 72 KiB/s wr, 7 op/s 2023-03-09T15:53:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:53:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3323.195836 secs 2023-03-09T15:53:33.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:53:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3323.195836 secs 2023-03-09T15:53:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:53:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3323.195836 secs 2023-03-09T15:53:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3785.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:53:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:53:33 smithi122 ceph-mon[93869]: pgmap v2288: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 46 KiB/s wr, 8 op/s 2023-03-09T15:53:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:53:33 smithi181 ceph-mon[94189]: pgmap v2288: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 46 KiB/s wr, 8 op/s 2023-03-09T15:53:34.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:53:33 smithi038 ceph-mon[92422]: pgmap v2288: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 46 KiB/s wr, 8 op/s 2023-03-09T15:53:35.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:53:34 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3325.983100 secs 2023-03-09T15:53:35.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:53:34 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3325.983100 secs 2023-03-09T15:53:35.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:53:34 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3325.983100 secs 2023-03-09T15:53:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:53:35 smithi122 ceph-mon[93869]: pgmap v2289: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 129 KiB/s wr, 8 op/s 2023-03-09T15:53:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:53:35 smithi181 ceph-mon[94189]: pgmap v2289: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 129 KiB/s wr, 8 op/s 2023-03-09T15:53:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:53:35 smithi038 ceph-mon[92422]: pgmap v2289: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 129 KiB/s wr, 8 op/s 2023-03-09T15:53:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:53:37 smithi122 ceph-mon[93869]: pgmap v2290: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 129 KiB/s wr, 10 op/s 2023-03-09T15:53:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:53:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3328.195929 secs 2023-03-09T15:53:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:53:37 smithi181 ceph-mon[94189]: pgmap v2290: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 129 KiB/s wr, 10 op/s 2023-03-09T15:53:38.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:53:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3328.195929 secs 2023-03-09T15:53:38.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:53:37 smithi038 ceph-mon[92422]: pgmap v2290: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 129 KiB/s wr, 10 op/s 2023-03-09T15:53:38.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:53:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3328.195929 secs 2023-03-09T15:53:38.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3790.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:53:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:53:39 smithi122 ceph-mon[93869]: pgmap v2291: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 133 KiB/s wr, 9 op/s 2023-03-09T15:53:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:53:39 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3330.983162 secs 2023-03-09T15:53:40.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:53:39 smithi181 ceph-mon[94189]: pgmap v2291: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 133 KiB/s wr, 9 op/s 2023-03-09T15:53:40.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:53:39 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3330.983162 secs 2023-03-09T15:53:40.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:53:39 smithi038 ceph-mon[92422]: pgmap v2291: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 133 KiB/s wr, 9 op/s 2023-03-09T15:53:40.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:53:39 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3330.983162 secs 2023-03-09T15:53:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:53:41 smithi122 ceph-mon[93869]: pgmap v2292: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 133 KiB/s wr, 9 op/s 2023-03-09T15:53:42.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:53:41 smithi181 ceph-mon[94189]: pgmap v2292: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 133 KiB/s wr, 9 op/s 2023-03-09T15:53:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:53:41 smithi038 ceph-mon[92422]: pgmap v2292: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 133 KiB/s wr, 9 op/s 2023-03-09T15:53:43.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:53:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3333.196058 secs 2023-03-09T15:53:43.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:53:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3333.196058 secs 2023-03-09T15:53:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:53:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3333.196058 secs 2023-03-09T15:53:43.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3795.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:53:44.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:53:43 smithi181 ceph-mon[94189]: pgmap v2293: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 88 KiB/s wr, 11 op/s 2023-03-09T15:53:44.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:53:43 smithi038 ceph-mon[92422]: pgmap v2293: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 88 KiB/s wr, 11 op/s 2023-03-09T15:53:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:53:43 smithi122 ceph-mon[93869]: pgmap v2293: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 88 KiB/s wr, 11 op/s 2023-03-09T15:53:45.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:53:44 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3335.983224 secs 2023-03-09T15:53:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:53:44 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3335.983224 secs 2023-03-09T15:53:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:53:44 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3335.983224 secs 2023-03-09T15:53:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:53:45 smithi038 ceph-mon[92422]: pgmap v2294: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 158 KiB/s wr, 11 op/s 2023-03-09T15:53:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:53:45 smithi122 ceph-mon[93869]: pgmap v2294: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 158 KiB/s wr, 11 op/s 2023-03-09T15:53:46.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:53:45 smithi181 ceph-mon[94189]: pgmap v2294: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 158 KiB/s wr, 11 op/s 2023-03-09T15:53:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:53:48 smithi122 ceph-mon[93869]: pgmap v2295: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 74 KiB/s wr, 11 op/s 2023-03-09T15:53:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:53:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3338.196193 secs 2023-03-09T15:53:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:53:48 smithi181 ceph-mon[94189]: pgmap v2295: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 74 KiB/s wr, 11 op/s 2023-03-09T15:53:48.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:53:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3338.196193 secs 2023-03-09T15:53:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:53:47 smithi038 ceph-mon[92422]: pgmap v2295: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 74 KiB/s wr, 11 op/s 2023-03-09T15:53:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:53:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3338.196193 secs 2023-03-09T15:53:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3800.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:53:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:53:50 smithi122 ceph-mon[93869]: pgmap v2296: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 107 KiB/s wr, 11 op/s 2023-03-09T15:53:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:53:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3340.983283 secs 2023-03-09T15:53:50.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:53:50 smithi181 ceph-mon[94189]: pgmap v2296: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 107 KiB/s wr, 11 op/s 2023-03-09T15:53:50.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:53:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3340.983283 secs 2023-03-09T15:53:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:53:50 smithi038 ceph-mon[92422]: pgmap v2296: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 107 KiB/s wr, 11 op/s 2023-03-09T15:53:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:53:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3340.983283 secs 2023-03-09T15:53:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:53:52 smithi122 ceph-mon[93869]: pgmap v2297: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 102 KiB/s wr, 10 op/s 2023-03-09T15:53:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:53:52 smithi181 ceph-mon[94189]: pgmap v2297: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 102 KiB/s wr, 10 op/s 2023-03-09T15:53:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:53:52 smithi038 ceph-mon[92422]: pgmap v2297: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 102 KiB/s wr, 10 op/s 2023-03-09T15:53:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:53:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3343.196308 secs 2023-03-09T15:53:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:53:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3343.196308 secs 2023-03-09T15:53:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:53:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3343.196308 secs 2023-03-09T15:53:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3805.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:53:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:53:54 smithi122 ceph-mon[93869]: pgmap v2298: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 103 KiB/s wr, 11 op/s 2023-03-09T15:53:54.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:53:54 smithi181 ceph-mon[94189]: pgmap v2298: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 103 KiB/s wr, 11 op/s 2023-03-09T15:53:54.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:53:54 smithi038 ceph-mon[92422]: pgmap v2298: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 103 KiB/s wr, 11 op/s 2023-03-09T15:53:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:53:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3345.983343 secs 2023-03-09T15:53:55.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:53:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3345.983343 secs 2023-03-09T15:53:55.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:53:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3345.983343 secs 2023-03-09T15:53:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:53:56 smithi122 ceph-mon[93869]: pgmap v2299: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 147 KiB/s wr, 11 op/s 2023-03-09T15:53:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:53:56 smithi181 ceph-mon[94189]: pgmap v2299: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 147 KiB/s wr, 11 op/s 2023-03-09T15:53:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:53:56 smithi038 ceph-mon[92422]: pgmap v2299: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 147 KiB/s wr, 11 op/s 2023-03-09T15:53:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:53:58 smithi122 ceph-mon[93869]: pgmap v2300: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 78 KiB/s wr, 10 op/s 2023-03-09T15:53:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:53:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3348.196403 secs 2023-03-09T15:53:58.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:53:58 smithi181 ceph-mon[94189]: pgmap v2300: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 78 KiB/s wr, 10 op/s 2023-03-09T15:53:58.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:53:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3348.196403 secs 2023-03-09T15:53:58.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:53:58 smithi038 ceph-mon[92422]: pgmap v2300: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 78 KiB/s wr, 10 op/s 2023-03-09T15:53:58.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:53:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3348.196403 secs 2023-03-09T15:53:58.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3810.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:54:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:54:00 smithi122 ceph-mon[93869]: pgmap v2301: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 105 KiB/s wr, 10 op/s 2023-03-09T15:54:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:54:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3350.983402 secs 2023-03-09T15:54:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:54:00 smithi181 ceph-mon[94189]: pgmap v2301: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 105 KiB/s wr, 10 op/s 2023-03-09T15:54:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:54:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3350.983402 secs 2023-03-09T15:54:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:54:00 smithi038 ceph-mon[92422]: pgmap v2301: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 105 KiB/s wr, 10 op/s 2023-03-09T15:54:00.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:54:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3350.983402 secs 2023-03-09T15:54:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:54:02 smithi122 ceph-mon[93869]: pgmap v2302: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 7 op/s 2023-03-09T15:54:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:54:02 smithi181 ceph-mon[94189]: pgmap v2302: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 7 op/s 2023-03-09T15:54:02.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:54:02 smithi038 ceph-mon[92422]: pgmap v2302: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 7 op/s 2023-03-09T15:54:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:54:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3353.196520 secs 2023-03-09T15:54:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:54:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3353.196520 secs 2023-03-09T15:54:03.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:54:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3353.196520 secs 2023-03-09T15:54:03.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3815.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:54:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:54:04 smithi122 ceph-mon[93869]: pgmap v2303: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 73 KiB/s wr, 9 op/s 2023-03-09T15:54:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:54:04 smithi181 ceph-mon[94189]: pgmap v2303: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 73 KiB/s wr, 9 op/s 2023-03-09T15:54:04.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:54:04 smithi038 ceph-mon[92422]: pgmap v2303: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 73 KiB/s wr, 9 op/s 2023-03-09T15:54:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:54:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3355.983462 secs 2023-03-09T15:54:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:54:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3355.983462 secs 2023-03-09T15:54:05.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:54:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3355.983462 secs 2023-03-09T15:54:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:54:06 smithi122 ceph-mon[93869]: pgmap v2304: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 113 KiB/s wr, 21 op/s 2023-03-09T15:54:06.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:54:06 smithi181 ceph-mon[94189]: pgmap v2304: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 113 KiB/s wr, 21 op/s 2023-03-09T15:54:06.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:54:06 smithi038 ceph-mon[92422]: pgmap v2304: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 113 KiB/s wr, 21 op/s 2023-03-09T15:54:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:54:08 smithi122 ceph-mon[93869]: pgmap v2305: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 91 KiB/s wr, 23 op/s 2023-03-09T15:54:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:54:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3358.196621 secs 2023-03-09T15:54:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:54:08 smithi181 ceph-mon[94189]: pgmap v2305: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 91 KiB/s wr, 23 op/s 2023-03-09T15:54:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:54:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3358.196621 secs 2023-03-09T15:54:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:54:08 smithi038 ceph-mon[92422]: pgmap v2305: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 91 KiB/s wr, 23 op/s 2023-03-09T15:54:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:54:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3358.196621 secs 2023-03-09T15:54:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3820.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:54:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:54:10 smithi122 ceph-mon[93869]: pgmap v2306: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 95 KiB/s wr, 23 op/s 2023-03-09T15:54:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:54:10 smithi122 ceph-mon[93869]: osdmap e358: 12 total, 12 up, 12 in 2023-03-09T15:54:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:54:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3360.983523 secs 2023-03-09T15:54:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:54:10 smithi181 ceph-mon[94189]: pgmap v2306: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 95 KiB/s wr, 23 op/s 2023-03-09T15:54:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:54:10 smithi181 ceph-mon[94189]: osdmap e358: 12 total, 12 up, 12 in 2023-03-09T15:54:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:54:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3360.983523 secs 2023-03-09T15:54:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:54:10 smithi038 ceph-mon[92422]: pgmap v2306: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 95 KiB/s wr, 23 op/s 2023-03-09T15:54:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:54:10 smithi038 ceph-mon[92422]: osdmap e358: 12 total, 12 up, 12 in 2023-03-09T15:54:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:54:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3360.983523 secs 2023-03-09T15:54:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:54:12 smithi122 ceph-mon[93869]: pgmap v2308: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 81 KiB/s wr, 25 op/s 2023-03-09T15:54:12.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:54:12 smithi181 ceph-mon[94189]: pgmap v2308: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 81 KiB/s wr, 25 op/s 2023-03-09T15:54:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:54:12 smithi038 ceph-mon[92422]: pgmap v2308: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 81 KiB/s wr, 25 op/s 2023-03-09T15:54:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:54:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3363.196747 secs 2023-03-09T15:54:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:54:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3363.196747 secs 2023-03-09T15:54:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:54:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3363.196747 secs 2023-03-09T15:54:13.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3825.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:54:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:54:14 smithi122 ceph-mon[93869]: pgmap v2309: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 82 KiB/s wr, 25 op/s 2023-03-09T15:54:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:54:14 smithi181 ceph-mon[94189]: pgmap v2309: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 82 KiB/s wr, 25 op/s 2023-03-09T15:54:14.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:54:14 smithi038 ceph-mon[92422]: pgmap v2309: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 82 KiB/s wr, 25 op/s 2023-03-09T15:54:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:54:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3365.983581 secs 2023-03-09T15:54:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:54:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3365.983581 secs 2023-03-09T15:54:15.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:54:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3365.983581 secs 2023-03-09T15:54:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:54:16 smithi122 ceph-mon[93869]: pgmap v2310: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 33 KiB/s wr, 10 op/s 2023-03-09T15:54:16.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:54:16 smithi181 ceph-mon[94189]: pgmap v2310: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 33 KiB/s wr, 10 op/s 2023-03-09T15:54:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:54:16 smithi038 ceph-mon[92422]: pgmap v2310: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 33 KiB/s wr, 10 op/s 2023-03-09T15:54:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:54:18 smithi122 ceph-mon[93869]: pgmap v2311: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 6.3 KiB/s wr, 8 op/s 2023-03-09T15:54:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:54:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3368.196811 secs 2023-03-09T15:54:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:54:18 smithi181 ceph-mon[94189]: pgmap v2311: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 6.3 KiB/s wr, 8 op/s 2023-03-09T15:54:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:54:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3368.196811 secs 2023-03-09T15:54:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:54:18 smithi038 ceph-mon[92422]: pgmap v2311: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 6.3 KiB/s wr, 8 op/s 2023-03-09T15:54:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:54:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3368.196811 secs 2023-03-09T15:54:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3830.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:54:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:54:19 smithi122 ceph-mon[93869]: pgmap v2312: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 29 KiB/s wr, 8 op/s 2023-03-09T15:54:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:54:19 smithi122 ceph-mon[93869]: osdmap e359: 12 total, 12 up, 12 in 2023-03-09T15:54:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:54:19 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3370.983638 secs 2023-03-09T15:54:20.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:54:19 smithi181 ceph-mon[94189]: pgmap v2312: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 29 KiB/s wr, 8 op/s 2023-03-09T15:54:20.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:54:19 smithi181 ceph-mon[94189]: osdmap e359: 12 total, 12 up, 12 in 2023-03-09T15:54:20.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:54:19 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3370.983638 secs 2023-03-09T15:54:20.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:54:19 smithi038 ceph-mon[92422]: pgmap v2312: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 29 KiB/s wr, 8 op/s 2023-03-09T15:54:20.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:54:19 smithi038 ceph-mon[92422]: osdmap e359: 12 total, 12 up, 12 in 2023-03-09T15:54:20.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:54:19 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3370.983638 secs 2023-03-09T15:54:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:54:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:54:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:54:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:54:21.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:54:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:54:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:54:21 smithi122 ceph-mon[93869]: pgmap v2314: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 29 KiB/s wr, 8 op/s 2023-03-09T15:54:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:54:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:54:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:54:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:54:22.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:54:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:54:22.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:54:21 smithi181 ceph-mon[94189]: pgmap v2314: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 29 KiB/s wr, 8 op/s 2023-03-09T15:54:22.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:54:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:54:22.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:54:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:54:22.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:54:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:54:22.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:54:21 smithi038 ceph-mon[92422]: pgmap v2314: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 29 KiB/s wr, 8 op/s 2023-03-09T15:54:22.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:54:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:54:22.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:54:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:54:22.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:54:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:54:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:54:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3373.196949 secs 2023-03-09T15:54:23.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:54:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3373.196949 secs 2023-03-09T15:54:23.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:54:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3373.196949 secs 2023-03-09T15:54:23.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3835.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:54:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:54:23 smithi122 ceph-mon[93869]: pgmap v2315: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 8 op/s 2023-03-09T15:54:24.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:54:23 smithi181 ceph-mon[94189]: pgmap v2315: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 8 op/s 2023-03-09T15:54:24.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:54:23 smithi038 ceph-mon[92422]: pgmap v2315: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 8 op/s 2023-03-09T15:54:25.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:54:24 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3375.983721 secs 2023-03-09T15:54:25.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:54:24 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3375.983721 secs 2023-03-09T15:54:25.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:54:24 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3375.983721 secs 2023-03-09T15:54:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:54:25 smithi122 ceph-mon[93869]: pgmap v2316: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T15:54:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:54:25 smithi181 ceph-mon[94189]: pgmap v2316: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T15:54:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:54:25 smithi038 ceph-mon[92422]: pgmap v2316: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T15:54:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:54:27 smithi122 ceph-mon[93869]: pgmap v2317: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T15:54:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:54:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3378.197098 secs 2023-03-09T15:54:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:54:27 smithi181 ceph-mon[94189]: pgmap v2317: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T15:54:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:54:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3378.197098 secs 2023-03-09T15:54:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:54:27 smithi038 ceph-mon[92422]: pgmap v2317: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T15:54:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:54:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3378.197098 secs 2023-03-09T15:54:28.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3840.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:54:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:54:29 smithi122 ceph-mon[93869]: pgmap v2318: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T15:54:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:54:29 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3380.983804 secs 2023-03-09T15:54:30.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:54:29 smithi181 ceph-mon[94189]: pgmap v2318: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T15:54:30.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:54:29 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3380.983804 secs 2023-03-09T15:54:30.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:54:29 smithi038 ceph-mon[92422]: pgmap v2318: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T15:54:30.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:54:29 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3380.983804 secs 2023-03-09T15:54:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:54:31 smithi122 ceph-mon[93869]: pgmap v2319: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 452 B/s wr, 6 op/s 2023-03-09T15:54:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:54:31 smithi181 ceph-mon[94189]: pgmap v2319: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 452 B/s wr, 6 op/s 2023-03-09T15:54:32.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:54:31 smithi038 ceph-mon[92422]: pgmap v2319: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 452 B/s wr, 6 op/s 2023-03-09T15:54:33.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:54:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3383.197223 secs 2023-03-09T15:54:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:54:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3383.197223 secs 2023-03-09T15:54:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:54:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3383.197223 secs 2023-03-09T15:54:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3845.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:54:34.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:54:33 smithi038 ceph-mon[92422]: pgmap v2320: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 511 B/s wr, 7 op/s 2023-03-09T15:54:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:54:33 smithi122 ceph-mon[93869]: pgmap v2320: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 511 B/s wr, 7 op/s 2023-03-09T15:54:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:54:33 smithi181 ceph-mon[94189]: pgmap v2320: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 511 B/s wr, 7 op/s 2023-03-09T15:54:35.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:54:34 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3385.983864 secs 2023-03-09T15:54:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:54:34 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3385.983864 secs 2023-03-09T15:54:35.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:54:34 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3385.983864 secs 2023-03-09T15:54:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:54:35 smithi038 ceph-mon[92422]: pgmap v2321: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 25 KiB/s wr, 6 op/s 2023-03-09T15:54:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:54:35 smithi122 ceph-mon[93869]: pgmap v2321: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 25 KiB/s wr, 6 op/s 2023-03-09T15:54:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:54:35 smithi181 ceph-mon[94189]: pgmap v2321: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 25 KiB/s wr, 6 op/s 2023-03-09T15:54:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:54:38 smithi122 ceph-mon[93869]: pgmap v2322: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 26 KiB/s wr, 7 op/s 2023-03-09T15:54:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:54:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3388.197306 secs 2023-03-09T15:54:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:54:38 smithi181 ceph-mon[94189]: pgmap v2322: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 26 KiB/s wr, 7 op/s 2023-03-09T15:54:38.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:54:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3388.197306 secs 2023-03-09T15:54:38.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:54:38 smithi038 ceph-mon[92422]: pgmap v2322: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 26 KiB/s wr, 7 op/s 2023-03-09T15:54:38.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:54:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3388.197306 secs 2023-03-09T15:54:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3850.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:54:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:54:40 smithi122 ceph-mon[93869]: pgmap v2323: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 102 KiB/s wr, 7 op/s 2023-03-09T15:54:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:54:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3390.983937 secs 2023-03-09T15:54:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:54:40 smithi181 ceph-mon[94189]: pgmap v2323: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 102 KiB/s wr, 7 op/s 2023-03-09T15:54:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:54:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3390.983937 secs 2023-03-09T15:54:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:54:40 smithi038 ceph-mon[92422]: pgmap v2323: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 102 KiB/s wr, 7 op/s 2023-03-09T15:54:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:54:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3390.983937 secs 2023-03-09T15:54:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:54:42 smithi122 ceph-mon[93869]: pgmap v2324: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 113 KiB/s wr, 7 op/s 2023-03-09T15:54:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:54:42 smithi181 ceph-mon[94189]: pgmap v2324: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 113 KiB/s wr, 7 op/s 2023-03-09T15:54:42.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:54:42 smithi038 ceph-mon[92422]: pgmap v2324: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 113 KiB/s wr, 7 op/s 2023-03-09T15:54:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:54:43 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:54:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:54:43 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:54:43.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:54:43 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:54:43.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3855.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:54:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:54:44 smithi122 ceph-mon[93869]: pgmap v2325: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 113 KiB/s wr, 10 op/s 2023-03-09T15:54:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:54:44 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:54:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:54:44 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:54:44.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:54:44 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:54:44.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:54:44 smithi181 ceph-mon[94189]: pgmap v2325: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 113 KiB/s wr, 10 op/s 2023-03-09T15:54:44.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:54:44 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:54:44.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:54:44 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:54:44.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:54:44 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:54:44.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:54:44 smithi038 ceph-mon[92422]: pgmap v2325: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 113 KiB/s wr, 10 op/s 2023-03-09T15:54:44.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:54:44 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:54:44.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:54:44 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:54:44.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:54:44 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:54:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:54:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3395.983996 secs 2023-03-09T15:54:45.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:54:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3395.983996 secs 2023-03-09T15:54:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:54:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3395.983996 secs 2023-03-09T15:54:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:54:46 smithi122 ceph-mon[93869]: pgmap v2326: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 165 KiB/s wr, 10 op/s 2023-03-09T15:54:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:54:46 smithi181 ceph-mon[94189]: pgmap v2326: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 165 KiB/s wr, 10 op/s 2023-03-09T15:54:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:54:46 smithi038 ceph-mon[92422]: pgmap v2326: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 165 KiB/s wr, 10 op/s 2023-03-09T15:54:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:54:48 smithi122 ceph-mon[93869]: pgmap v2327: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 190 KiB/s wr, 13 op/s 2023-03-09T15:54:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:54:48 smithi181 ceph-mon[94189]: pgmap v2327: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 190 KiB/s wr, 13 op/s 2023-03-09T15:54:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:54:48 smithi038 ceph-mon[92422]: pgmap v2327: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 190 KiB/s wr, 13 op/s 2023-03-09T15:54:48.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3860.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:54:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:54:50 smithi122 ceph-mon[93869]: pgmap v2328: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 197 KiB/s wr, 13 op/s 2023-03-09T15:54:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:54:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3400.984058 secs 2023-03-09T15:54:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:54:50 smithi181 ceph-mon[94189]: pgmap v2328: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 197 KiB/s wr, 13 op/s 2023-03-09T15:54:50.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:54:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3400.984058 secs 2023-03-09T15:54:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:54:50 smithi038 ceph-mon[92422]: pgmap v2328: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 197 KiB/s wr, 13 op/s 2023-03-09T15:54:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:54:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3400.984058 secs 2023-03-09T15:54:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:54:52 smithi122 ceph-mon[93869]: pgmap v2329: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 159 KiB/s wr, 12 op/s 2023-03-09T15:54:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:54:52 smithi181 ceph-mon[94189]: pgmap v2329: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 159 KiB/s wr, 12 op/s 2023-03-09T15:54:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:54:52 smithi038 ceph-mon[92422]: pgmap v2329: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 159 KiB/s wr, 12 op/s 2023-03-09T15:54:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:54:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3403.197639 secs 2023-03-09T15:54:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:54:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3403.197639 secs 2023-03-09T15:54:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:54:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3403.197639 secs 2023-03-09T15:54:53.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3865.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:54:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:54:54 smithi122 ceph-mon[93869]: pgmap v2330: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 149 KiB/s wr, 13 op/s 2023-03-09T15:54:54.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:54:54 smithi181 ceph-mon[94189]: pgmap v2330: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 149 KiB/s wr, 13 op/s 2023-03-09T15:54:54.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:54:54 smithi038 ceph-mon[92422]: pgmap v2330: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 149 KiB/s wr, 13 op/s 2023-03-09T15:54:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:54:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3405.984119 secs 2023-03-09T15:54:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:54:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3405.984119 secs 2023-03-09T15:54:55.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:54:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3405.984119 secs 2023-03-09T15:54:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:54:56 smithi122 ceph-mon[93869]: pgmap v2331: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 152 KiB/s wr, 12 op/s 2023-03-09T15:54:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:54:56 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:54:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:54:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:54:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:54:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:54:56.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:54:56 smithi181 ceph-mon[94189]: pgmap v2331: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 152 KiB/s wr, 12 op/s 2023-03-09T15:54:56.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:54:56 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:54:56.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:54:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:54:56.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:54:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:54:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:54:56 smithi038 ceph-mon[92422]: pgmap v2331: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 152 KiB/s wr, 12 op/s 2023-03-09T15:54:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:54:56 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:54:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:54:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:54:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:54:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:54:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:54:58 smithi122 ceph-mon[93869]: pgmap v2332: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 138 KiB/s wr, 12 op/s 2023-03-09T15:54:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:54:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3408.197777 secs 2023-03-09T15:54:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:54:58 smithi181 ceph-mon[94189]: pgmap v2332: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 138 KiB/s wr, 12 op/s 2023-03-09T15:54:58.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:54:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3408.197777 secs 2023-03-09T15:54:58.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:54:58 smithi038 ceph-mon[92422]: pgmap v2332: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 138 KiB/s wr, 12 op/s 2023-03-09T15:54:58.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:54:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3408.197777 secs 2023-03-09T15:54:58.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3870.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:55:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:55:00 smithi122 ceph-mon[93869]: pgmap v2333: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 88 KiB/s wr, 9 op/s 2023-03-09T15:55:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:55:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3410.984213 secs 2023-03-09T15:55:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:55:00 smithi181 ceph-mon[94189]: pgmap v2333: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 88 KiB/s wr, 9 op/s 2023-03-09T15:55:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:55:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3410.984213 secs 2023-03-09T15:55:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:55:00 smithi038 ceph-mon[92422]: pgmap v2333: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 88 KiB/s wr, 9 op/s 2023-03-09T15:55:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:55:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3410.984213 secs 2023-03-09T15:55:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:55:02 smithi122 ceph-mon[93869]: pgmap v2334: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 103 KiB/s wr, 8 op/s 2023-03-09T15:55:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:55:02 smithi181 ceph-mon[94189]: pgmap v2334: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 103 KiB/s wr, 8 op/s 2023-03-09T15:55:02.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:55:02 smithi038 ceph-mon[92422]: pgmap v2334: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 103 KiB/s wr, 8 op/s 2023-03-09T15:55:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:55:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3413.197911 secs 2023-03-09T15:55:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:55:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3413.197911 secs 2023-03-09T15:55:03.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:55:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3413.197911 secs 2023-03-09T15:55:03.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3875.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:55:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:55:04 smithi122 ceph-mon[93869]: pgmap v2335: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 64 KiB/s wr, 9 op/s 2023-03-09T15:55:04.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:55:04 smithi181 ceph-mon[94189]: pgmap v2335: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 64 KiB/s wr, 9 op/s 2023-03-09T15:55:04.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:55:04 smithi038 ceph-mon[92422]: pgmap v2335: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 64 KiB/s wr, 9 op/s 2023-03-09T15:55:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:55:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3415.984273 secs 2023-03-09T15:55:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:55:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3415.984273 secs 2023-03-09T15:55:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:55:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3415.984273 secs 2023-03-09T15:55:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:55:06 smithi122 ceph-mon[93869]: pgmap v2336: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 85 KiB/s wr, 19 op/s 2023-03-09T15:55:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:55:06 smithi181 ceph-mon[94189]: pgmap v2336: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 85 KiB/s wr, 19 op/s 2023-03-09T15:55:06.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:55:06 smithi038 ceph-mon[92422]: pgmap v2336: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 85 KiB/s wr, 19 op/s 2023-03-09T15:55:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:55:08 smithi122 ceph-mon[93869]: pgmap v2337: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 105 KiB/s wr, 23 op/s 2023-03-09T15:55:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:55:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3418.198054 secs 2023-03-09T15:55:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:55:08 smithi181 ceph-mon[94189]: pgmap v2337: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 105 KiB/s wr, 23 op/s 2023-03-09T15:55:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:55:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3418.198054 secs 2023-03-09T15:55:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:55:08 smithi038 ceph-mon[92422]: pgmap v2337: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 105 KiB/s wr, 23 op/s 2023-03-09T15:55:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:55:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3418.198054 secs 2023-03-09T15:55:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3880.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:55:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:55:10 smithi122 ceph-mon[93869]: pgmap v2338: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 67 KiB/s wr, 21 op/s 2023-03-09T15:55:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:55:10 smithi122 ceph-mon[93869]: osdmap e360: 12 total, 12 up, 12 in 2023-03-09T15:55:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:55:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3420.984334 secs 2023-03-09T15:55:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:55:10 smithi181 ceph-mon[94189]: pgmap v2338: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 67 KiB/s wr, 21 op/s 2023-03-09T15:55:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:55:10 smithi181 ceph-mon[94189]: osdmap e360: 12 total, 12 up, 12 in 2023-03-09T15:55:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:55:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3420.984334 secs 2023-03-09T15:55:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:55:10 smithi038 ceph-mon[92422]: pgmap v2338: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 67 KiB/s wr, 21 op/s 2023-03-09T15:55:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:55:10 smithi038 ceph-mon[92422]: osdmap e360: 12 total, 12 up, 12 in 2023-03-09T15:55:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:55:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3420.984334 secs 2023-03-09T15:55:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:55:12 smithi122 ceph-mon[93869]: pgmap v2340: 129 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+snaptrim, 127 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 23 op/s 2023-03-09T15:55:12.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:55:12 smithi181 ceph-mon[94189]: pgmap v2340: 129 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+snaptrim, 127 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 23 op/s 2023-03-09T15:55:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:55:12 smithi038 ceph-mon[92422]: pgmap v2340: 129 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+snaptrim, 127 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 23 op/s 2023-03-09T15:55:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:55:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3423.198158 secs 2023-03-09T15:55:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:55:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3423.198158 secs 2023-03-09T15:55:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:55:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3423.198158 secs 2023-03-09T15:55:13.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3885.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:55:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:55:14 smithi122 ceph-mon[93869]: pgmap v2341: 129 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+snaptrim, 127 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 23 op/s 2023-03-09T15:55:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:55:14 smithi181 ceph-mon[94189]: pgmap v2341: 129 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+snaptrim, 127 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 23 op/s 2023-03-09T15:55:14.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:55:14 smithi038 ceph-mon[92422]: pgmap v2341: 129 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+snaptrim, 127 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 23 op/s 2023-03-09T15:55:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:55:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3425.984395 secs 2023-03-09T15:55:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:55:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3425.984395 secs 2023-03-09T15:55:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:55:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3425.984395 secs 2023-03-09T15:55:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:55:16 smithi122 ceph-mon[93869]: pgmap v2342: 129 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+snaptrim, 127 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 28 KiB/s wr, 9 op/s 2023-03-09T15:55:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:55:16 smithi181 ceph-mon[94189]: pgmap v2342: 129 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+snaptrim, 127 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 28 KiB/s wr, 9 op/s 2023-03-09T15:55:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:55:16 smithi038 ceph-mon[92422]: pgmap v2342: 129 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+snaptrim, 127 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 28 KiB/s wr, 9 op/s 2023-03-09T15:55:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:55:18 smithi122 ceph-mon[93869]: pgmap v2343: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:55:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:55:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3428.198252 secs 2023-03-09T15:55:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:55:18 smithi181 ceph-mon[94189]: pgmap v2343: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:55:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:55:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3428.198252 secs 2023-03-09T15:55:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:55:18 smithi038 ceph-mon[92422]: pgmap v2343: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:55:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:55:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3428.198252 secs 2023-03-09T15:55:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3890.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:55:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:55:19 smithi122 ceph-mon[93869]: pgmap v2344: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:55:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:55:19 smithi122 ceph-mon[93869]: osdmap e361: 12 total, 12 up, 12 in 2023-03-09T15:55:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:55:19 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3430.984482 secs 2023-03-09T15:55:20.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:55:19 smithi181 ceph-mon[94189]: pgmap v2344: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:55:20.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:55:19 smithi181 ceph-mon[94189]: osdmap e361: 12 total, 12 up, 12 in 2023-03-09T15:55:20.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:55:19 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3430.984482 secs 2023-03-09T15:55:20.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:55:19 smithi038 ceph-mon[92422]: pgmap v2344: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:55:20.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:55:19 smithi038 ceph-mon[92422]: osdmap e361: 12 total, 12 up, 12 in 2023-03-09T15:55:20.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:55:19 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3430.984482 secs 2023-03-09T15:55:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:55:21 smithi122 ceph-mon[93869]: pgmap v2346: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 6 op/s 2023-03-09T15:55:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:55:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:55:22.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:55:21 smithi181 ceph-mon[94189]: pgmap v2346: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 6 op/s 2023-03-09T15:55:22.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:55:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:55:22.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:55:21 smithi038 ceph-mon[92422]: pgmap v2346: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 6 op/s 2023-03-09T15:55:22.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:55:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:55:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:55:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3433.198334 secs 2023-03-09T15:55:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:55:22 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:55:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:55:22 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:55:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:55:22 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:55:23.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:55:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3433.198334 secs 2023-03-09T15:55:23.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:55:22 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:55:23.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:55:22 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:55:23.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:55:22 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:55:23.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:55:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3433.198334 secs 2023-03-09T15:55:23.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:55:22 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:55:23.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:55:22 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:55:23.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:55:22 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:55:23.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3895.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:55:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:55:24 smithi122 ceph-mon[93869]: pgmap v2347: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 6 op/s 2023-03-09T15:55:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:55:24 smithi181 ceph-mon[94189]: pgmap v2347: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 6 op/s 2023-03-09T15:55:24.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:55:24 smithi038 ceph-mon[92422]: pgmap v2347: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 6 op/s 2023-03-09T15:55:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:55:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3435.984551 secs 2023-03-09T15:55:25.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:55:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3435.984551 secs 2023-03-09T15:55:25.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:55:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3435.984551 secs 2023-03-09T15:55:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:55:26 smithi122 ceph-mon[93869]: pgmap v2348: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T15:55:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:55:26 smithi181 ceph-mon[94189]: pgmap v2348: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T15:55:26.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:55:26 smithi038 ceph-mon[92422]: pgmap v2348: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T15:55:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:55:28 smithi122 ceph-mon[93869]: pgmap v2349: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 6 op/s 2023-03-09T15:55:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:55:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3438.198413 secs 2023-03-09T15:55:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:55:28 smithi181 ceph-mon[94189]: pgmap v2349: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 6 op/s 2023-03-09T15:55:28.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:55:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3438.198413 secs 2023-03-09T15:55:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:55:28 smithi038 ceph-mon[92422]: pgmap v2349: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 6 op/s 2023-03-09T15:55:28.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:55:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3438.198413 secs 2023-03-09T15:55:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3900.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:55:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:55:30 smithi122 ceph-mon[93869]: pgmap v2350: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T15:55:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:55:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3440.984619 secs 2023-03-09T15:55:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:55:30 smithi181 ceph-mon[94189]: pgmap v2350: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T15:55:30.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:55:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3440.984619 secs 2023-03-09T15:55:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:55:30 smithi038 ceph-mon[92422]: pgmap v2350: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T15:55:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:55:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3440.984619 secs 2023-03-09T15:55:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:55:32 smithi122 ceph-mon[93869]: pgmap v2351: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 51 KiB/s wr, 7 op/s 2023-03-09T15:55:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:55:32 smithi181 ceph-mon[94189]: pgmap v2351: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 51 KiB/s wr, 7 op/s 2023-03-09T15:55:32.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:55:32 smithi038 ceph-mon[92422]: pgmap v2351: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 51 KiB/s wr, 7 op/s 2023-03-09T15:55:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3905.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:55:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:55:34 smithi122 ceph-mon[93869]: pgmap v2352: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 29 KiB/s wr, 8 op/s 2023-03-09T15:55:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:55:34 smithi181 ceph-mon[94189]: pgmap v2352: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 29 KiB/s wr, 8 op/s 2023-03-09T15:55:34.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:55:34 smithi038 ceph-mon[92422]: pgmap v2352: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 29 KiB/s wr, 8 op/s 2023-03-09T15:55:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:55:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3445.984679 secs 2023-03-09T15:55:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:55:35 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:55:35.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:55:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3445.984679 secs 2023-03-09T15:55:35.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:55:35 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:55:35.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:55:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3445.984679 secs 2023-03-09T15:55:35.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:55:35 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:55:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:55:36 smithi122 ceph-mon[93869]: pgmap v2353: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 7 op/s 2023-03-09T15:55:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:55:36 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:55:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:55:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:55:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:55:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:55:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:55:36 smithi181 ceph-mon[94189]: pgmap v2353: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 7 op/s 2023-03-09T15:55:36.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:55:36 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:55:36.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:55:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:55:36.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:55:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:55:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:55:36 smithi038 ceph-mon[92422]: pgmap v2353: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 7 op/s 2023-03-09T15:55:36.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:55:36 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:55:36.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:55:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:55:36.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:55:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:55:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:55:38 smithi122 ceph-mon[93869]: pgmap v2354: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 103 KiB/s wr, 9 op/s 2023-03-09T15:55:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:55:38 smithi181 ceph-mon[94189]: pgmap v2354: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 103 KiB/s wr, 9 op/s 2023-03-09T15:55:38.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:55:38 smithi038 ceph-mon[92422]: pgmap v2354: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 103 KiB/s wr, 9 op/s 2023-03-09T15:55:38.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3910.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:55:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:55:40 smithi122 ceph-mon[93869]: pgmap v2355: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 107 KiB/s wr, 8 op/s 2023-03-09T15:55:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:55:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3450.984739 secs 2023-03-09T15:55:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:55:40 smithi181 ceph-mon[94189]: pgmap v2355: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 107 KiB/s wr, 8 op/s 2023-03-09T15:55:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:55:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3450.984739 secs 2023-03-09T15:55:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:55:40 smithi038 ceph-mon[92422]: pgmap v2355: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 107 KiB/s wr, 8 op/s 2023-03-09T15:55:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:55:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3450.984739 secs 2023-03-09T15:55:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:55:41 smithi038 ceph-mon[92422]: pgmap v2356: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 164 KiB/s wr, 8 op/s 2023-03-09T15:55:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:55:41 smithi122 ceph-mon[93869]: pgmap v2356: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 164 KiB/s wr, 8 op/s 2023-03-09T15:55:41.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:55:41 smithi181 ceph-mon[94189]: pgmap v2356: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 164 KiB/s wr, 8 op/s 2023-03-09T15:55:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:55:43 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 3453.198723 secs 2023-03-09T15:55:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:55:43 smithi122 ceph-mon[93869]: pgmap v2357: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 137 KiB/s wr, 10 op/s 2023-03-09T15:55:43.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:55:43 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 3453.198723 secs 2023-03-09T15:55:43.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:55:43 smithi181 ceph-mon[94189]: pgmap v2357: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 137 KiB/s wr, 10 op/s 2023-03-09T15:55:43.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:55:43 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 3453.198723 secs 2023-03-09T15:55:43.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:55:43 smithi038 ceph-mon[92422]: pgmap v2357: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 137 KiB/s wr, 10 op/s 2023-03-09T15:55:43.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3915.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:55:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:55:44 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:55:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:55:44 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:55:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:55:44 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:55:44.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:55:44 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:55:44.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:55:44 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:55:44.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:55:44 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:55:44.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:55:44 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:55:44.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:55:44 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:55:44.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:55:44 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:55:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:55:45 smithi122 ceph-mon[93869]: pgmap v2358: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 169 KiB/s wr, 10 op/s 2023-03-09T15:55:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:55:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3455.984800 secs 2023-03-09T15:55:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:55:45 smithi181 ceph-mon[94189]: pgmap v2358: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 169 KiB/s wr, 10 op/s 2023-03-09T15:55:45.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:55:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3455.984800 secs 2023-03-09T15:55:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:55:45 smithi038 ceph-mon[92422]: pgmap v2358: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 169 KiB/s wr, 10 op/s 2023-03-09T15:55:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:55:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3455.984800 secs 2023-03-09T15:55:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:55:47 smithi122 ceph-mon[93869]: pgmap v2359: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 174 KiB/s wr, 12 op/s 2023-03-09T15:55:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:55:47 smithi181 ceph-mon[94189]: pgmap v2359: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 174 KiB/s wr, 12 op/s 2023-03-09T15:55:47.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:55:47 smithi038 ceph-mon[92422]: pgmap v2359: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 174 KiB/s wr, 12 op/s 2023-03-09T15:55:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:55:48 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 3458.198787 secs 2023-03-09T15:55:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:55:48 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 3458.198787 secs 2023-03-09T15:55:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:55:48 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 3458.198787 secs 2023-03-09T15:55:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3920.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:55:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:55:49 smithi122 ceph-mon[93869]: pgmap v2360: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 126 KiB/s wr, 11 op/s 2023-03-09T15:55:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:55:49 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3460.984862 secs 2023-03-09T15:55:50.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:55:49 smithi181 ceph-mon[94189]: pgmap v2360: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 126 KiB/s wr, 11 op/s 2023-03-09T15:55:50.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:55:49 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3460.984862 secs 2023-03-09T15:55:50.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:55:49 smithi038 ceph-mon[92422]: pgmap v2360: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 126 KiB/s wr, 11 op/s 2023-03-09T15:55:50.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:55:49 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3460.984862 secs 2023-03-09T15:55:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:55:51 smithi122 ceph-mon[93869]: pgmap v2361: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 122 KiB/s wr, 10 op/s 2023-03-09T15:55:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:55:51 smithi181 ceph-mon[94189]: pgmap v2361: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 122 KiB/s wr, 10 op/s 2023-03-09T15:55:52.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:55:51 smithi038 ceph-mon[92422]: pgmap v2361: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 122 KiB/s wr, 10 op/s 2023-03-09T15:55:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:55:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3463.198900 secs 2023-03-09T15:55:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:55:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3463.198900 secs 2023-03-09T15:55:53.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:55:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3463.198900 secs 2023-03-09T15:55:53.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3925.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:55:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:55:53 smithi122 ceph-mon[93869]: pgmap v2362: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 65 KiB/s wr, 11 op/s 2023-03-09T15:55:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:55:53 smithi181 ceph-mon[94189]: pgmap v2362: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 65 KiB/s wr, 11 op/s 2023-03-09T15:55:54.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:55:53 smithi038 ceph-mon[92422]: pgmap v2362: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 65 KiB/s wr, 11 op/s 2023-03-09T15:55:55.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:55:54 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3465.984923 secs 2023-03-09T15:55:55.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:55:54 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3465.984923 secs 2023-03-09T15:55:55.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:55:54 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3465.984923 secs 2023-03-09T15:55:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:55:55 smithi122 ceph-mon[93869]: pgmap v2363: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 142 KiB/s wr, 11 op/s 2023-03-09T15:55:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:55:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:55:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:55:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:55:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:55:55 smithi181 ceph-mon[94189]: pgmap v2363: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 142 KiB/s wr, 11 op/s 2023-03-09T15:55:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:55:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:55:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:55:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:55:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:55:55 smithi038 ceph-mon[92422]: pgmap v2363: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 142 KiB/s wr, 11 op/s 2023-03-09T15:55:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:55:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:55:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:55:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:55:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:55:57 smithi122 ceph-mon[93869]: pgmap v2364: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 110 KiB/s wr, 11 op/s 2023-03-09T15:55:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:55:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3468.199019 secs 2023-03-09T15:55:58.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:55:57 smithi181 ceph-mon[94189]: pgmap v2364: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 110 KiB/s wr, 11 op/s 2023-03-09T15:55:58.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:55:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3468.199019 secs 2023-03-09T15:55:58.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:55:57 smithi038 ceph-mon[92422]: pgmap v2364: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 110 KiB/s wr, 11 op/s 2023-03-09T15:55:58.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:55:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3468.199019 secs 2023-03-09T15:55:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3930.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:56:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:55:59 smithi122 ceph-mon[93869]: pgmap v2365: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 115 KiB/s wr, 10 op/s 2023-03-09T15:56:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:55:59 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3470.984983 secs 2023-03-09T15:56:00.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:55:59 smithi181 ceph-mon[94189]: pgmap v2365: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 115 KiB/s wr, 10 op/s 2023-03-09T15:56:00.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:55:59 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3470.984983 secs 2023-03-09T15:56:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:55:59 smithi038 ceph-mon[92422]: pgmap v2365: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 115 KiB/s wr, 10 op/s 2023-03-09T15:56:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:55:59 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3470.984983 secs 2023-03-09T15:56:02.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:56:01 smithi038 ceph-mon[92422]: pgmap v2366: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 89 KiB/s wr, 8 op/s 2023-03-09T15:56:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:56:01 smithi122 ceph-mon[93869]: pgmap v2366: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 89 KiB/s wr, 8 op/s 2023-03-09T15:56:02.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:56:01 smithi181 ceph-mon[94189]: pgmap v2366: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 89 KiB/s wr, 8 op/s 2023-03-09T15:56:03.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:56:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3473.199137 secs 2023-03-09T15:56:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:56:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3473.199137 secs 2023-03-09T15:56:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:56:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3473.199137 secs 2023-03-09T15:56:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3935.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:56:04.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:56:03 smithi038 ceph-mon[92422]: pgmap v2367: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 111 KiB/s wr, 11 op/s 2023-03-09T15:56:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:56:03 smithi122 ceph-mon[93869]: pgmap v2367: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 111 KiB/s wr, 11 op/s 2023-03-09T15:56:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:56:03 smithi181 ceph-mon[94189]: pgmap v2367: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 111 KiB/s wr, 11 op/s 2023-03-09T15:56:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:56:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3475.985068 secs 2023-03-09T15:56:05.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:56:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3475.985068 secs 2023-03-09T15:56:05.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:56:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3475.985068 secs 2023-03-09T15:56:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:56:06 smithi122 ceph-mon[93869]: pgmap v2368: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 156 KiB/s wr, 23 op/s 2023-03-09T15:56:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:56:06 smithi181 ceph-mon[94189]: pgmap v2368: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 156 KiB/s wr, 23 op/s 2023-03-09T15:56:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:56:06 smithi038 ceph-mon[92422]: pgmap v2368: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 156 KiB/s wr, 23 op/s 2023-03-09T15:56:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:56:08 smithi122 ceph-mon[93869]: pgmap v2369: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 77 KiB/s wr, 22 op/s 2023-03-09T15:56:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:56:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3478.199202 secs 2023-03-09T15:56:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:56:08 smithi181 ceph-mon[94189]: pgmap v2369: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 77 KiB/s wr, 22 op/s 2023-03-09T15:56:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:56:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3478.199202 secs 2023-03-09T15:56:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:56:08 smithi038 ceph-mon[92422]: pgmap v2369: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 77 KiB/s wr, 22 op/s 2023-03-09T15:56:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:56:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3478.199202 secs 2023-03-09T15:56:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3940.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:56:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:56:10 smithi122 ceph-mon[93869]: pgmap v2370: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 77 KiB/s wr, 21 op/s 2023-03-09T15:56:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:56:10 smithi122 ceph-mon[93869]: osdmap e362: 12 total, 12 up, 12 in 2023-03-09T15:56:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:56:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3480.985128 secs 2023-03-09T15:56:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:56:10 smithi181 ceph-mon[94189]: pgmap v2370: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 77 KiB/s wr, 21 op/s 2023-03-09T15:56:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:56:10 smithi181 ceph-mon[94189]: osdmap e362: 12 total, 12 up, 12 in 2023-03-09T15:56:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:56:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3480.985128 secs 2023-03-09T15:56:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:56:10 smithi038 ceph-mon[92422]: pgmap v2370: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 77 KiB/s wr, 21 op/s 2023-03-09T15:56:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:56:10 smithi038 ceph-mon[92422]: osdmap e362: 12 total, 12 up, 12 in 2023-03-09T15:56:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:56:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3480.985128 secs 2023-03-09T15:56:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:56:12 smithi122 ceph-mon[93869]: pgmap v2372: 129 pgs: 1 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 122 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 81 KiB/s wr, 24 op/s 2023-03-09T15:56:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:56:12 smithi181 ceph-mon[94189]: pgmap v2372: 129 pgs: 1 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 122 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 81 KiB/s wr, 24 op/s 2023-03-09T15:56:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:56:12 smithi038 ceph-mon[92422]: pgmap v2372: 129 pgs: 1 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 122 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 81 KiB/s wr, 24 op/s 2023-03-09T15:56:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:56:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3483.199320 secs 2023-03-09T15:56:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:56:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3483.199320 secs 2023-03-09T15:56:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:56:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3483.199320 secs 2023-03-09T15:56:13.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3945.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:56:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:56:14 smithi122 ceph-mon[93869]: pgmap v2373: 129 pgs: 1 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 122 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 54 KiB/s wr, 21 op/s 2023-03-09T15:56:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:56:14 smithi181 ceph-mon[94189]: pgmap v2373: 129 pgs: 1 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 122 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 54 KiB/s wr, 21 op/s 2023-03-09T15:56:14.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:56:14 smithi038 ceph-mon[92422]: pgmap v2373: 129 pgs: 1 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 122 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 54 KiB/s wr, 21 op/s 2023-03-09T15:56:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:56:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3485.985201 secs 2023-03-09T15:56:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:56:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3485.985201 secs 2023-03-09T15:56:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:56:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3485.985201 secs 2023-03-09T15:56:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:56:16 smithi122 ceph-mon[93869]: pgmap v2374: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 6 op/s 2023-03-09T15:56:16.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:56:16 smithi181 ceph-mon[94189]: pgmap v2374: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 6 op/s 2023-03-09T15:56:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:56:16 smithi038 ceph-mon[92422]: pgmap v2374: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 6 op/s 2023-03-09T15:56:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:56:18 smithi122 ceph-mon[93869]: pgmap v2375: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 6 op/s 2023-03-09T15:56:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:56:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3488.199436 secs 2023-03-09T15:56:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:56:18 smithi181 ceph-mon[94189]: pgmap v2375: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 6 op/s 2023-03-09T15:56:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:56:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3488.199436 secs 2023-03-09T15:56:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:56:18 smithi038 ceph-mon[92422]: pgmap v2375: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 6 op/s 2023-03-09T15:56:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:56:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3488.199436 secs 2023-03-09T15:56:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3950.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:56:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:56:19 smithi122 ceph-mon[93869]: pgmap v2376: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 6 op/s 2023-03-09T15:56:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:56:19 smithi122 ceph-mon[93869]: osdmap e363: 12 total, 12 up, 12 in 2023-03-09T15:56:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:56:19 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3490.985263 secs 2023-03-09T15:56:20.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:56:19 smithi181 ceph-mon[94189]: pgmap v2376: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 6 op/s 2023-03-09T15:56:20.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:56:19 smithi181 ceph-mon[94189]: osdmap e363: 12 total, 12 up, 12 in 2023-03-09T15:56:20.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:56:19 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3490.985263 secs 2023-03-09T15:56:20.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:56:19 smithi038 ceph-mon[92422]: pgmap v2376: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 6 op/s 2023-03-09T15:56:20.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:56:19 smithi038 ceph-mon[92422]: osdmap e363: 12 total, 12 up, 12 in 2023-03-09T15:56:20.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:56:19 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3490.985263 secs 2023-03-09T15:56:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:56:21 smithi122 ceph-mon[93869]: pgmap v2378: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 6 op/s 2023-03-09T15:56:22.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:56:21 smithi181 ceph-mon[94189]: pgmap v2378: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 6 op/s 2023-03-09T15:56:22.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:56:21 smithi038 ceph-mon[92422]: pgmap v2378: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 6 op/s 2023-03-09T15:56:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:56:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3493.199564 secs 2023-03-09T15:56:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:56:22 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:56:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:56:22 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:56:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:56:22 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:56:23.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:56:22 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:56:23.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:56:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3493.199564 secs 2023-03-09T15:56:23.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:56:22 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:56:23.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:56:22 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:56:23.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:56:22 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:56:23.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:56:22 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:56:23.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:56:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3493.199564 secs 2023-03-09T15:56:23.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:56:22 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:56:23.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:56:22 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:56:23.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:56:22 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:56:23.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:56:22 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:56:23.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3955.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:56:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:56:23 smithi122 ceph-mon[93869]: pgmap v2379: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 6 op/s 2023-03-09T15:56:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:56:23 smithi181 ceph-mon[94189]: pgmap v2379: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 6 op/s 2023-03-09T15:56:24.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:56:23 smithi038 ceph-mon[92422]: pgmap v2379: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 6 op/s 2023-03-09T15:56:25.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:56:24 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3495.985323 secs 2023-03-09T15:56:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:56:24 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3495.985323 secs 2023-03-09T15:56:25.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:56:24 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3495.985323 secs 2023-03-09T15:56:26.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:56:25 smithi038 ceph-mon[92422]: pgmap v2380: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 7 op/s 2023-03-09T15:56:26.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:56:25 smithi122 ceph-mon[93869]: pgmap v2380: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 7 op/s 2023-03-09T15:56:26.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:56:25 smithi181 ceph-mon[94189]: pgmap v2380: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 7 op/s 2023-03-09T15:56:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:56:27 smithi038 ceph-mon[92422]: pgmap v2381: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 614 B/s wr, 7 op/s 2023-03-09T15:56:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:56:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3498.199684 secs 2023-03-09T15:56:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:56:27 smithi122 ceph-mon[93869]: pgmap v2381: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 614 B/s wr, 7 op/s 2023-03-09T15:56:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:56:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3498.199684 secs 2023-03-09T15:56:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:56:27 smithi181 ceph-mon[94189]: pgmap v2381: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 614 B/s wr, 7 op/s 2023-03-09T15:56:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:56:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3498.199684 secs 2023-03-09T15:56:28.801 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3960.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:56:30.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:56:29 smithi038 ceph-mon[92422]: pgmap v2382: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T15:56:30.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:56:29 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3500.985382 secs 2023-03-09T15:56:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:56:29 smithi122 ceph-mon[93869]: pgmap v2382: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T15:56:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:56:29 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3500.985382 secs 2023-03-09T15:56:30.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:56:29 smithi181 ceph-mon[94189]: pgmap v2382: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T15:56:30.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:56:29 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3500.985382 secs 2023-03-09T15:56:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:56:32 smithi122 ceph-mon[93869]: pgmap v2383: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 6 op/s 2023-03-09T15:56:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:56:32 smithi181 ceph-mon[94189]: pgmap v2383: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 6 op/s 2023-03-09T15:56:32.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:56:32 smithi038 ceph-mon[92422]: pgmap v2383: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 6 op/s 2023-03-09T15:56:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:56:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3503.199803 secs 2023-03-09T15:56:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:56:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3503.199803 secs 2023-03-09T15:56:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:56:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3503.199803 secs 2023-03-09T15:56:33.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3965.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:56:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:56:34 smithi122 ceph-mon[93869]: pgmap v2384: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 23 KiB/s wr, 8 op/s 2023-03-09T15:56:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:56:34 smithi181 ceph-mon[94189]: pgmap v2384: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 23 KiB/s wr, 8 op/s 2023-03-09T15:56:34.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:56:34 smithi038 ceph-mon[92422]: pgmap v2384: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 23 KiB/s wr, 8 op/s 2023-03-09T15:56:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:56:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3505.985445 secs 2023-03-09T15:56:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:56:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3505.985445 secs 2023-03-09T15:56:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:56:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3505.985445 secs 2023-03-09T15:56:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:56:36 smithi122 ceph-mon[93869]: pgmap v2385: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 7 op/s 2023-03-09T15:56:36.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:56:36 smithi181 ceph-mon[94189]: pgmap v2385: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 7 op/s 2023-03-09T15:56:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:56:36 smithi038 ceph-mon[92422]: pgmap v2385: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 7 op/s 2023-03-09T15:56:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:56:38 smithi122 ceph-mon[93869]: pgmap v2386: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 30 KiB/s wr, 8 op/s 2023-03-09T15:56:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:56:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3508.199949 secs 2023-03-09T15:56:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:56:38 smithi181 ceph-mon[94189]: pgmap v2386: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 30 KiB/s wr, 8 op/s 2023-03-09T15:56:38.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:56:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3508.199949 secs 2023-03-09T15:56:38.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:56:38 smithi038 ceph-mon[92422]: pgmap v2386: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 30 KiB/s wr, 8 op/s 2023-03-09T15:56:38.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:56:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3508.199949 secs 2023-03-09T15:56:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3970.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:56:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:56:40 smithi122 ceph-mon[93869]: pgmap v2387: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 121 KiB/s wr, 8 op/s 2023-03-09T15:56:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:56:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3510.985531 secs 2023-03-09T15:56:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:56:40 smithi181 ceph-mon[94189]: pgmap v2387: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 121 KiB/s wr, 8 op/s 2023-03-09T15:56:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:56:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3510.985531 secs 2023-03-09T15:56:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:56:40 smithi038 ceph-mon[92422]: pgmap v2387: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 121 KiB/s wr, 8 op/s 2023-03-09T15:56:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:56:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3510.985531 secs 2023-03-09T15:56:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:56:42 smithi122 ceph-mon[93869]: pgmap v2388: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 98 KiB/s wr, 7 op/s 2023-03-09T15:56:42.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:56:42 smithi181 ceph-mon[94189]: pgmap v2388: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 98 KiB/s wr, 7 op/s 2023-03-09T15:56:42.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:56:42 smithi038 ceph-mon[92422]: pgmap v2388: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 98 KiB/s wr, 7 op/s 2023-03-09T15:56:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:56:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3513.200070 secs 2023-03-09T15:56:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:56:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3513.200070 secs 2023-03-09T15:56:43.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:56:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3513.200070 secs 2023-03-09T15:56:43.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3975.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:56:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:56:44 smithi122 ceph-mon[93869]: pgmap v2389: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 103 KiB/s wr, 11 op/s 2023-03-09T15:56:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:56:44 smithi181 ceph-mon[94189]: pgmap v2389: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 103 KiB/s wr, 11 op/s 2023-03-09T15:56:44.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:56:44 smithi038 ceph-mon[92422]: pgmap v2389: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 103 KiB/s wr, 11 op/s 2023-03-09T15:56:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:56:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3515.985599 secs 2023-03-09T15:56:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:56:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3515.985599 secs 2023-03-09T15:56:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:56:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3515.985599 secs 2023-03-09T15:56:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:56:46 smithi122 ceph-mon[93869]: pgmap v2390: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 166 KiB/s wr, 11 op/s 2023-03-09T15:56:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:56:46 smithi181 ceph-mon[94189]: pgmap v2390: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 166 KiB/s wr, 11 op/s 2023-03-09T15:56:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:56:46 smithi038 ceph-mon[92422]: pgmap v2390: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 166 KiB/s wr, 11 op/s 2023-03-09T15:56:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:56:48 smithi122 ceph-mon[93869]: pgmap v2391: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 159 KiB/s wr, 11 op/s 2023-03-09T15:56:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:56:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3518.200188 secs 2023-03-09T15:56:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:56:48 smithi181 ceph-mon[94189]: pgmap v2391: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 159 KiB/s wr, 11 op/s 2023-03-09T15:56:48.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:56:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3518.200188 secs 2023-03-09T15:56:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:56:48 smithi038 ceph-mon[92422]: pgmap v2391: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 159 KiB/s wr, 11 op/s 2023-03-09T15:56:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:56:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3518.200188 secs 2023-03-09T15:56:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3980.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:56:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:56:50 smithi122 ceph-mon[93869]: pgmap v2392: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 202 KiB/s wr, 12 op/s 2023-03-09T15:56:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:56:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3520.985665 secs 2023-03-09T15:56:50.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:56:50 smithi181 ceph-mon[94189]: pgmap v2392: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 202 KiB/s wr, 12 op/s 2023-03-09T15:56:50.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:56:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3520.985665 secs 2023-03-09T15:56:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:56:50 smithi038 ceph-mon[92422]: pgmap v2392: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 202 KiB/s wr, 12 op/s 2023-03-09T15:56:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:56:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3520.985665 secs 2023-03-09T15:56:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:56:52 smithi122 ceph-mon[93869]: pgmap v2393: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 111 KiB/s wr, 10 op/s 2023-03-09T15:56:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:56:52 smithi181 ceph-mon[94189]: pgmap v2393: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 111 KiB/s wr, 10 op/s 2023-03-09T15:56:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:56:52 smithi038 ceph-mon[92422]: pgmap v2393: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 111 KiB/s wr, 10 op/s 2023-03-09T15:56:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:56:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3523.200273 secs 2023-03-09T15:56:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:56:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3523.200273 secs 2023-03-09T15:56:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:56:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3523.200273 secs 2023-03-09T15:56:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3985.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:56:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:56:54 smithi122 ceph-mon[93869]: pgmap v2394: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 164 KiB/s wr, 13 op/s 2023-03-09T15:56:54.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:56:54 smithi181 ceph-mon[94189]: pgmap v2394: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 164 KiB/s wr, 13 op/s 2023-03-09T15:56:54.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:56:54 smithi038 ceph-mon[92422]: pgmap v2394: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 164 KiB/s wr, 13 op/s 2023-03-09T15:56:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:56:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3525.985724 secs 2023-03-09T15:56:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:56:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3525.985724 secs 2023-03-09T15:56:55.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:56:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3525.985724 secs 2023-03-09T15:56:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:56:56 smithi122 ceph-mon[93869]: pgmap v2395: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 159 KiB/s wr, 10 op/s 2023-03-09T15:56:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:56:56 smithi181 ceph-mon[94189]: pgmap v2395: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 159 KiB/s wr, 10 op/s 2023-03-09T15:56:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:56:56 smithi038 ceph-mon[92422]: pgmap v2395: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 159 KiB/s wr, 10 op/s 2023-03-09T15:56:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:56:58 smithi122 ceph-mon[93869]: pgmap v2396: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 96 KiB/s wr, 10 op/s 2023-03-09T15:56:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:56:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3528.200371 secs 2023-03-09T15:56:58.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:56:58 smithi181 ceph-mon[94189]: pgmap v2396: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 96 KiB/s wr, 10 op/s 2023-03-09T15:56:58.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:56:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3528.200371 secs 2023-03-09T15:56:58.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:56:58 smithi038 ceph-mon[92422]: pgmap v2396: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 96 KiB/s wr, 10 op/s 2023-03-09T15:56:58.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:56:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3528.200371 secs 2023-03-09T15:56:58.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3990.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:57:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:57:00 smithi122 ceph-mon[93869]: pgmap v2397: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 126 KiB/s wr, 10 op/s 2023-03-09T15:57:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:57:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3530.985784 secs 2023-03-09T15:57:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:57:00 smithi181 ceph-mon[94189]: pgmap v2397: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 126 KiB/s wr, 10 op/s 2023-03-09T15:57:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:57:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3530.985784 secs 2023-03-09T15:57:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:57:00 smithi038 ceph-mon[92422]: pgmap v2397: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 126 KiB/s wr, 10 op/s 2023-03-09T15:57:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:57:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3530.985784 secs 2023-03-09T15:57:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:57:02 smithi122 ceph-mon[93869]: pgmap v2398: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 83 KiB/s wr, 7 op/s 2023-03-09T15:57:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:57:02 smithi181 ceph-mon[94189]: pgmap v2398: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 83 KiB/s wr, 7 op/s 2023-03-09T15:57:02.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:57:02 smithi038 ceph-mon[92422]: pgmap v2398: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 83 KiB/s wr, 7 op/s 2023-03-09T15:57:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:57:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3533.200489 secs 2023-03-09T15:57:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:57:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3533.200489 secs 2023-03-09T15:57:03.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:57:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3533.200489 secs 2023-03-09T15:57:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 3995.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:57:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:57:04 smithi122 ceph-mon[93869]: pgmap v2399: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 129 KiB/s wr, 13 op/s 2023-03-09T15:57:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:57:04 smithi181 ceph-mon[94189]: pgmap v2399: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 129 KiB/s wr, 13 op/s 2023-03-09T15:57:04.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:57:04 smithi038 ceph-mon[92422]: pgmap v2399: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 129 KiB/s wr, 13 op/s 2023-03-09T15:57:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:57:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3535.985845 secs 2023-03-09T15:57:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:57:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3535.985845 secs 2023-03-09T15:57:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:57:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3535.985845 secs 2023-03-09T15:57:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:57:06 smithi122 ceph-mon[93869]: pgmap v2400: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 97 KiB/s wr, 21 op/s 2023-03-09T15:57:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:57:06 smithi181 ceph-mon[94189]: pgmap v2400: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 97 KiB/s wr, 21 op/s 2023-03-09T15:57:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:57:06 smithi038 ceph-mon[92422]: pgmap v2400: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 97 KiB/s wr, 21 op/s 2023-03-09T15:57:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:57:08 smithi122 ceph-mon[93869]: pgmap v2401: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 97 KiB/s wr, 22 op/s 2023-03-09T15:57:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:57:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3538.200572 secs 2023-03-09T15:57:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:57:08 smithi181 ceph-mon[94189]: pgmap v2401: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 97 KiB/s wr, 22 op/s 2023-03-09T15:57:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:57:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3538.200572 secs 2023-03-09T15:57:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:57:08 smithi038 ceph-mon[92422]: pgmap v2401: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 97 KiB/s wr, 22 op/s 2023-03-09T15:57:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:57:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3538.200572 secs 2023-03-09T15:57:08.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4000.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:57:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:57:10 smithi122 ceph-mon[93869]: pgmap v2402: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 97 KiB/s wr, 21 op/s 2023-03-09T15:57:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:57:10 smithi122 ceph-mon[93869]: osdmap e364: 12 total, 12 up, 12 in 2023-03-09T15:57:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:57:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3540.985910 secs 2023-03-09T15:57:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:57:10 smithi181 ceph-mon[94189]: pgmap v2402: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 97 KiB/s wr, 21 op/s 2023-03-09T15:57:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:57:10 smithi181 ceph-mon[94189]: osdmap e364: 12 total, 12 up, 12 in 2023-03-09T15:57:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:57:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3540.985910 secs 2023-03-09T15:57:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:57:10 smithi038 ceph-mon[92422]: pgmap v2402: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 97 KiB/s wr, 21 op/s 2023-03-09T15:57:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:57:10 smithi038 ceph-mon[92422]: osdmap e364: 12 total, 12 up, 12 in 2023-03-09T15:57:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:57:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3540.985910 secs 2023-03-09T15:57:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:57:12 smithi122 ceph-mon[93869]: pgmap v2404: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 81 KiB/s wr, 24 op/s 2023-03-09T15:57:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:57:12 smithi181 ceph-mon[94189]: pgmap v2404: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 81 KiB/s wr, 24 op/s 2023-03-09T15:57:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:57:12 smithi038 ceph-mon[92422]: pgmap v2404: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 81 KiB/s wr, 24 op/s 2023-03-09T15:57:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:57:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3543.200716 secs 2023-03-09T15:57:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:57:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3543.200716 secs 2023-03-09T15:57:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:57:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3543.200716 secs 2023-03-09T15:57:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4005.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:57:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:57:14 smithi122 ceph-mon[93869]: pgmap v2405: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 26 KiB/s wr, 20 op/s 2023-03-09T15:57:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:57:14 smithi181 ceph-mon[94189]: pgmap v2405: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 26 KiB/s wr, 20 op/s 2023-03-09T15:57:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:57:14 smithi038 ceph-mon[92422]: pgmap v2405: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 26 KiB/s wr, 20 op/s 2023-03-09T15:57:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:57:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3545.985969 secs 2023-03-09T15:57:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:57:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3545.985969 secs 2023-03-09T15:57:15.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:57:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3545.985969 secs 2023-03-09T15:57:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:57:16 smithi122 ceph-mon[93869]: pgmap v2406: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:57:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:57:16 smithi181 ceph-mon[94189]: pgmap v2406: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:57:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:57:16 smithi038 ceph-mon[92422]: pgmap v2406: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:57:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:57:18 smithi122 ceph-mon[93869]: pgmap v2407: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:57:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:57:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3548.200832 secs 2023-03-09T15:57:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:57:18 smithi181 ceph-mon[94189]: pgmap v2407: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:57:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:57:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3548.200832 secs 2023-03-09T15:57:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:57:18 smithi038 ceph-mon[92422]: pgmap v2407: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:57:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:57:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3548.200832 secs 2023-03-09T15:57:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4010.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:57:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:57:19 smithi122 ceph-mon[93869]: pgmap v2408: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:57:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:57:19 smithi122 ceph-mon[93869]: osdmap e365: 12 total, 12 up, 12 in 2023-03-09T15:57:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:57:19 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3550.986032 secs 2023-03-09T15:57:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:57:19 smithi181 ceph-mon[94189]: pgmap v2408: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:57:20.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:57:19 smithi181 ceph-mon[94189]: osdmap e365: 12 total, 12 up, 12 in 2023-03-09T15:57:20.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:57:19 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3550.986032 secs 2023-03-09T15:57:20.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:57:19 smithi038 ceph-mon[92422]: pgmap v2408: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:57:20.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:57:19 smithi038 ceph-mon[92422]: osdmap e365: 12 total, 12 up, 12 in 2023-03-09T15:57:20.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:57:19 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3550.986032 secs 2023-03-09T15:57:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:57:21 smithi122 ceph-mon[93869]: pgmap v2410: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:57:22.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:57:21 smithi181 ceph-mon[94189]: pgmap v2410: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:57:22.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:57:21 smithi038 ceph-mon[92422]: pgmap v2410: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:57:23.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:57:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3553.200971 secs 2023-03-09T15:57:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:57:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3553.200971 secs 2023-03-09T15:57:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:57:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3553.200971 secs 2023-03-09T15:57:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4015.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:57:24.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:57:23 smithi038 ceph-mon[92422]: pgmap v2411: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 614 B/s wr, 6 op/s 2023-03-09T15:57:24.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:57:23 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:57:24.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:57:23 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:57:24.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:57:23 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:57:24.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:57:23 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:57:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:57:23 smithi122 ceph-mon[93869]: pgmap v2411: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 614 B/s wr, 6 op/s 2023-03-09T15:57:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:57:23 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:57:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:57:23 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:57:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:57:23 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:57:24.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:57:23 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:57:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:57:23 smithi181 ceph-mon[94189]: pgmap v2411: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 614 B/s wr, 6 op/s 2023-03-09T15:57:24.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:57:23 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:57:24.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:57:23 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:57:24.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:57:23 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:57:24.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:57:23 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:57:25.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:57:24 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3555.986094 secs 2023-03-09T15:57:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:57:24 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3555.986094 secs 2023-03-09T15:57:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:57:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3555.986094 secs 2023-03-09T15:57:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:57:26 smithi122 ceph-mon[93869]: pgmap v2412: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 614 B/s wr, 6 op/s 2023-03-09T15:57:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:57:26 smithi181 ceph-mon[94189]: pgmap v2412: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 614 B/s wr, 6 op/s 2023-03-09T15:57:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:57:26 smithi038 ceph-mon[92422]: pgmap v2412: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 614 B/s wr, 6 op/s 2023-03-09T15:57:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:57:28 smithi122 ceph-mon[93869]: pgmap v2413: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 818 B/s wr, 6 op/s 2023-03-09T15:57:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:57:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3558.201125 secs 2023-03-09T15:57:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:57:28 smithi181 ceph-mon[94189]: pgmap v2413: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 818 B/s wr, 6 op/s 2023-03-09T15:57:28.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:57:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3558.201125 secs 2023-03-09T15:57:28.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:57:28 smithi038 ceph-mon[92422]: pgmap v2413: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 818 B/s wr, 6 op/s 2023-03-09T15:57:28.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:57:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3558.201125 secs 2023-03-09T15:57:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4020.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:57:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:57:30 smithi122 ceph-mon[93869]: pgmap v2414: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T15:57:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:57:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3560.986153 secs 2023-03-09T15:57:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:57:30 smithi181 ceph-mon[94189]: pgmap v2414: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T15:57:30.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:57:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3560.986153 secs 2023-03-09T15:57:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:57:30 smithi038 ceph-mon[92422]: pgmap v2414: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T15:57:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:57:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3560.986153 secs 2023-03-09T15:57:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:57:32 smithi122 ceph-mon[93869]: pgmap v2415: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 21 KiB/s wr, 6 op/s 2023-03-09T15:57:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:57:32 smithi181 ceph-mon[94189]: pgmap v2415: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 21 KiB/s wr, 6 op/s 2023-03-09T15:57:32.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:57:32 smithi038 ceph-mon[92422]: pgmap v2415: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 21 KiB/s wr, 6 op/s 2023-03-09T15:57:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:57:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3563.201241 secs 2023-03-09T15:57:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:57:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3563.201241 secs 2023-03-09T15:57:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:57:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3563.201241 secs 2023-03-09T15:57:33.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4025.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:57:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:57:34 smithi122 ceph-mon[93869]: pgmap v2416: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 50 KiB/s wr, 9 op/s 2023-03-09T15:57:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:57:34 smithi181 ceph-mon[94189]: pgmap v2416: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 50 KiB/s wr, 9 op/s 2023-03-09T15:57:34.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:57:34 smithi038 ceph-mon[92422]: pgmap v2416: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 50 KiB/s wr, 9 op/s 2023-03-09T15:57:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:57:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3565.986215 secs 2023-03-09T15:57:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:57:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3565.986215 secs 2023-03-09T15:57:35.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:57:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3565.986215 secs 2023-03-09T15:57:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:57:36 smithi122 ceph-mon[93869]: pgmap v2417: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 49 KiB/s wr, 6 op/s 2023-03-09T15:57:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:57:36 smithi181 ceph-mon[94189]: pgmap v2417: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 49 KiB/s wr, 6 op/s 2023-03-09T15:57:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:57:36 smithi038 ceph-mon[92422]: pgmap v2417: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 49 KiB/s wr, 6 op/s 2023-03-09T15:57:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:57:38 smithi122 ceph-mon[93869]: pgmap v2418: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 50 KiB/s wr, 8 op/s 2023-03-09T15:57:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:57:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3568.201353 secs 2023-03-09T15:57:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:57:38 smithi181 ceph-mon[94189]: pgmap v2418: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 50 KiB/s wr, 8 op/s 2023-03-09T15:57:38.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:57:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3568.201353 secs 2023-03-09T15:57:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:57:38 smithi038 ceph-mon[92422]: pgmap v2418: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 50 KiB/s wr, 8 op/s 2023-03-09T15:57:38.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:57:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3568.201353 secs 2023-03-09T15:57:38.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4030.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:57:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:57:40 smithi122 ceph-mon[93869]: pgmap v2419: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 128 KiB/s wr, 8 op/s 2023-03-09T15:57:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:57:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3570.986294 secs 2023-03-09T15:57:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:57:40 smithi181 ceph-mon[94189]: pgmap v2419: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 128 KiB/s wr, 8 op/s 2023-03-09T15:57:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:57:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3570.986294 secs 2023-03-09T15:57:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:57:40 smithi038 ceph-mon[92422]: pgmap v2419: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 128 KiB/s wr, 8 op/s 2023-03-09T15:57:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:57:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3570.986294 secs 2023-03-09T15:57:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:57:42 smithi122 ceph-mon[93869]: pgmap v2420: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 109 KiB/s wr, 7 op/s 2023-03-09T15:57:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:57:42 smithi181 ceph-mon[94189]: pgmap v2420: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 109 KiB/s wr, 7 op/s 2023-03-09T15:57:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:57:42 smithi038 ceph-mon[92422]: pgmap v2420: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 109 KiB/s wr, 7 op/s 2023-03-09T15:57:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:57:43 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:57:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:57:43 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:57:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:57:43 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:57:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4035.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:57:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:57:44 smithi122 ceph-mon[93869]: pgmap v2421: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 192 KiB/s wr, 12 op/s 2023-03-09T15:57:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:57:44 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:57:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:57:44 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:57:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:57:44 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:57:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:57:44 smithi181 ceph-mon[94189]: pgmap v2421: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 192 KiB/s wr, 12 op/s 2023-03-09T15:57:44.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:57:44 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:57:44.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:57:44 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:57:44.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:57:44 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:57:44.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:57:44 smithi038 ceph-mon[92422]: pgmap v2421: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 192 KiB/s wr, 12 op/s 2023-03-09T15:57:44.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:57:44 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:57:44.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:57:44 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:57:44.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:57:44 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:57:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:57:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3575.986364 secs 2023-03-09T15:57:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:57:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3575.986364 secs 2023-03-09T15:57:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:57:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3575.986364 secs 2023-03-09T15:57:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:57:46 smithi122 ceph-mon[93869]: pgmap v2422: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 163 KiB/s wr, 9 op/s 2023-03-09T15:57:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:57:46 smithi181 ceph-mon[94189]: pgmap v2422: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 163 KiB/s wr, 9 op/s 2023-03-09T15:57:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:57:46 smithi038 ceph-mon[92422]: pgmap v2422: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 163 KiB/s wr, 9 op/s 2023-03-09T15:57:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:57:48 smithi122 ceph-mon[93869]: pgmap v2423: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 196 KiB/s wr, 12 op/s 2023-03-09T15:57:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:57:48 smithi181 ceph-mon[94189]: pgmap v2423: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 196 KiB/s wr, 12 op/s 2023-03-09T15:57:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:57:48 smithi038 ceph-mon[92422]: pgmap v2423: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 196 KiB/s wr, 12 op/s 2023-03-09T15:57:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4040.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:57:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:57:50 smithi122 ceph-mon[93869]: pgmap v2424: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 203 KiB/s wr, 12 op/s 2023-03-09T15:57:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:57:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3580.986425 secs 2023-03-09T15:57:50.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:57:50 smithi181 ceph-mon[94189]: pgmap v2424: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 203 KiB/s wr, 12 op/s 2023-03-09T15:57:50.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:57:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3580.986425 secs 2023-03-09T15:57:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:57:50 smithi038 ceph-mon[92422]: pgmap v2424: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 203 KiB/s wr, 12 op/s 2023-03-09T15:57:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:57:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3580.986425 secs 2023-03-09T15:57:52.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:57:52 smithi122 ceph-mon[93869]: pgmap v2425: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 124 KiB/s wr, 10 op/s 2023-03-09T15:57:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:57:52 smithi181 ceph-mon[94189]: pgmap v2425: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 124 KiB/s wr, 10 op/s 2023-03-09T15:57:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:57:52 smithi038 ceph-mon[92422]: pgmap v2425: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 124 KiB/s wr, 10 op/s 2023-03-09T15:57:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:57:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3583.201684 secs 2023-03-09T15:57:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:57:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3583.201684 secs 2023-03-09T15:57:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:57:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3583.201684 secs 2023-03-09T15:57:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4045.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:57:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:57:54 smithi122 ceph-mon[93869]: pgmap v2426: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 175 KiB/s wr, 14 op/s 2023-03-09T15:57:54.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:57:54 smithi181 ceph-mon[94189]: pgmap v2426: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 175 KiB/s wr, 14 op/s 2023-03-09T15:57:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:57:54 smithi038 ceph-mon[92422]: pgmap v2426: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 175 KiB/s wr, 14 op/s 2023-03-09T15:57:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:57:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3585.986487 secs 2023-03-09T15:57:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:57:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3585.986487 secs 2023-03-09T15:57:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:57:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3585.986487 secs 2023-03-09T15:57:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:57:56 smithi122 ceph-mon[93869]: pgmap v2427: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 91 KiB/s wr, 10 op/s 2023-03-09T15:57:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:57:56 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:57:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:57:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:57:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:57:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:57:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:57:56 smithi181 ceph-mon[94189]: pgmap v2427: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 91 KiB/s wr, 10 op/s 2023-03-09T15:57:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:57:56 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:57:56.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:57:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:57:56.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:57:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:57:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:57:56 smithi038 ceph-mon[92422]: pgmap v2427: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 91 KiB/s wr, 10 op/s 2023-03-09T15:57:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:57:56 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:57:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:57:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:57:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:57:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:57:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:57:58 smithi122 ceph-mon[93869]: pgmap v2428: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 131 KiB/s wr, 12 op/s 2023-03-09T15:57:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:57:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3588.201747 secs 2023-03-09T15:57:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:57:58 smithi181 ceph-mon[94189]: pgmap v2428: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 131 KiB/s wr, 12 op/s 2023-03-09T15:57:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:57:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3588.201747 secs 2023-03-09T15:57:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:57:58 smithi038 ceph-mon[92422]: pgmap v2428: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 131 KiB/s wr, 12 op/s 2023-03-09T15:57:58.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:57:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3588.201747 secs 2023-03-09T15:57:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4050.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:58:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:00 smithi122 ceph-mon[93869]: pgmap v2429: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 97 KiB/s wr, 10 op/s 2023-03-09T15:58:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3590.986545 secs 2023-03-09T15:58:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:00 smithi181 ceph-mon[94189]: pgmap v2429: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 97 KiB/s wr, 10 op/s 2023-03-09T15:58:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3590.986545 secs 2023-03-09T15:58:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:00 smithi038 ceph-mon[92422]: pgmap v2429: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 97 KiB/s wr, 10 op/s 2023-03-09T15:58:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3590.986545 secs 2023-03-09T15:58:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:02 smithi122 ceph-mon[93869]: pgmap v2430: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 91 KiB/s wr, 8 op/s 2023-03-09T15:58:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:02 smithi181 ceph-mon[94189]: pgmap v2430: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 91 KiB/s wr, 8 op/s 2023-03-09T15:58:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:02 smithi038 ceph-mon[92422]: pgmap v2430: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 91 KiB/s wr, 8 op/s 2023-03-09T15:58:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3593.201891 secs 2023-03-09T15:58:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3593.201891 secs 2023-03-09T15:58:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3593.201891 secs 2023-03-09T15:58:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4055.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:58:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:04 smithi122 ceph-mon[93869]: pgmap v2431: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 137 KiB/s wr, 14 op/s 2023-03-09T15:58:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:04 smithi181 ceph-mon[94189]: pgmap v2431: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 137 KiB/s wr, 14 op/s 2023-03-09T15:58:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:04 smithi038 ceph-mon[92422]: pgmap v2431: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 137 KiB/s wr, 14 op/s 2023-03-09T15:58:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3595.986658 secs 2023-03-09T15:58:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3595.986658 secs 2023-03-09T15:58:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3595.986658 secs 2023-03-09T15:58:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:06 smithi122 ceph-mon[93869]: pgmap v2432: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 108 KiB/s wr, 21 op/s 2023-03-09T15:58:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:06 smithi181 ceph-mon[94189]: pgmap v2432: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 108 KiB/s wr, 21 op/s 2023-03-09T15:58:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:06 smithi038 ceph-mon[92422]: pgmap v2432: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 108 KiB/s wr, 21 op/s 2023-03-09T15:58:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:08 smithi122 ceph-mon[93869]: pgmap v2433: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 108 KiB/s wr, 23 op/s 2023-03-09T15:58:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3598.202005 secs 2023-03-09T15:58:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:08 smithi181 ceph-mon[94189]: pgmap v2433: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 108 KiB/s wr, 23 op/s 2023-03-09T15:58:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3598.202005 secs 2023-03-09T15:58:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:08 smithi038 ceph-mon[92422]: pgmap v2433: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 108 KiB/s wr, 23 op/s 2023-03-09T15:58:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3598.202005 secs 2023-03-09T15:58:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4060.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:58:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:10 smithi122 ceph-mon[93869]: pgmap v2434: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 68 KiB/s wr, 21 op/s 2023-03-09T15:58:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:10 smithi122 ceph-mon[93869]: osdmap e366: 12 total, 12 up, 12 in 2023-03-09T15:58:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3600.986744 secs 2023-03-09T15:58:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:10 smithi181 ceph-mon[94189]: pgmap v2434: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 68 KiB/s wr, 21 op/s 2023-03-09T15:58:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:10 smithi181 ceph-mon[94189]: osdmap e366: 12 total, 12 up, 12 in 2023-03-09T15:58:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3600.986744 secs 2023-03-09T15:58:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:10 smithi038 ceph-mon[92422]: pgmap v2434: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 68 KiB/s wr, 21 op/s 2023-03-09T15:58:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:10 smithi038 ceph-mon[92422]: osdmap e366: 12 total, 12 up, 12 in 2023-03-09T15:58:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3600.986744 secs 2023-03-09T15:58:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:12 smithi122 ceph-mon[93869]: pgmap v2436: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 81 KiB/s wr, 24 op/s 2023-03-09T15:58:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:12 smithi181 ceph-mon[94189]: pgmap v2436: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 81 KiB/s wr, 24 op/s 2023-03-09T15:58:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:12 smithi038 ceph-mon[92422]: pgmap v2436: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 81 KiB/s wr, 24 op/s 2023-03-09T15:58:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3603.202141 secs 2023-03-09T15:58:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3603.202141 secs 2023-03-09T15:58:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3603.202141 secs 2023-03-09T15:58:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4065.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:58:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:14 smithi122 ceph-mon[93869]: pgmap v2437: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 45 KiB/s wr, 20 op/s 2023-03-09T15:58:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:14 smithi181 ceph-mon[94189]: pgmap v2437: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 45 KiB/s wr, 20 op/s 2023-03-09T15:58:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:14 smithi038 ceph-mon[92422]: pgmap v2437: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 45 KiB/s wr, 20 op/s 2023-03-09T15:58:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3605.986801 secs 2023-03-09T15:58:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3605.986801 secs 2023-03-09T15:58:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3605.986801 secs 2023-03-09T15:58:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:16 smithi122 ceph-mon[93869]: pgmap v2438: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 6 op/s 2023-03-09T15:58:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:16 smithi181 ceph-mon[94189]: pgmap v2438: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 6 op/s 2023-03-09T15:58:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:16 smithi038 ceph-mon[92422]: pgmap v2438: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 6 op/s 2023-03-09T15:58:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:18 smithi122 ceph-mon[93869]: pgmap v2439: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 6 op/s 2023-03-09T15:58:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3608.202204 secs 2023-03-09T15:58:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:18 smithi181 ceph-mon[94189]: pgmap v2439: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 6 op/s 2023-03-09T15:58:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3608.202204 secs 2023-03-09T15:58:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:18 smithi038 ceph-mon[92422]: pgmap v2439: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 6 op/s 2023-03-09T15:58:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3608.202204 secs 2023-03-09T15:58:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4070.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:58:20.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:19 smithi038 ceph-mon[92422]: pgmap v2440: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 6 op/s 2023-03-09T15:58:20.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:19 smithi038 ceph-mon[92422]: osdmap e367: 12 total, 12 up, 12 in 2023-03-09T15:58:20.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:19 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3610.986879 secs 2023-03-09T15:58:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:19 smithi122 ceph-mon[93869]: pgmap v2440: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 6 op/s 2023-03-09T15:58:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:19 smithi122 ceph-mon[93869]: osdmap e367: 12 total, 12 up, 12 in 2023-03-09T15:58:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:19 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3610.986879 secs 2023-03-09T15:58:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:19 smithi181 ceph-mon[94189]: pgmap v2440: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 6 op/s 2023-03-09T15:58:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:19 smithi181 ceph-mon[94189]: osdmap e367: 12 total, 12 up, 12 in 2023-03-09T15:58:20.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:19 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3610.986879 secs 2023-03-09T15:58:22.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:21 smithi038 ceph-mon[92422]: pgmap v2442: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 6 op/s 2023-03-09T15:58:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:22 smithi122 ceph-mon[93869]: pgmap v2442: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 6 op/s 2023-03-09T15:58:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:21 smithi181 ceph-mon[94189]: pgmap v2442: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 6 op/s 2023-03-09T15:58:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3613.202271 secs 2023-03-09T15:58:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3613.202271 secs 2023-03-09T15:58:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3613.202271 secs 2023-03-09T15:58:23.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4075.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:58:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:24 smithi122 ceph-mon[93869]: pgmap v2443: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T15:58:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:58:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:24 smithi181 ceph-mon[94189]: pgmap v2443: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T15:58:24.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:58:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:24 smithi038 ceph-mon[92422]: pgmap v2443: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T15:58:24.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:58:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3615.986960 secs 2023-03-09T15:58:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3615.986960 secs 2023-03-09T15:58:25.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3615.986960 secs 2023-03-09T15:58:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:26 smithi122 ceph-mon[93869]: pgmap v2444: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T15:58:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:26 smithi181 ceph-mon[94189]: pgmap v2444: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T15:58:26.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:26 smithi038 ceph-mon[92422]: pgmap v2444: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T15:58:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:27 smithi122 ceph-mon[93869]: pgmap v2445: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T15:58:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3618.202371 secs 2023-03-09T15:58:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:58:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:58:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:58:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:58:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:58:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:58:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:58:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:58:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:58:28.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:58:28.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:58:28.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:58:28.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:58:28.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:58:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:27 smithi181 ceph-mon[94189]: pgmap v2445: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T15:58:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3618.202371 secs 2023-03-09T15:58:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:58:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:58:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:58:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:58:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:58:28.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:58:28.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:58:28.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:58:28.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:58:28.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:58:28.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:58:28.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:58:28.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:58:28.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:58:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:27 smithi038 ceph-mon[92422]: pgmap v2445: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T15:58:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3618.202371 secs 2023-03-09T15:58:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:58:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:58:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:58:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:58:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:58:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:58:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:58:28.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:58:28.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:58:28.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:58:28.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:58:28.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:58:28.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:58:28.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:58:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4080.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:58:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:28 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T15:58:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:28 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T15:58:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:28 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi038 to 197.2M 2023-03-09T15:58:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:28 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi038 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-03-09T15:58:29.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:58:29.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:58:29.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:58:29.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:58:29.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:28 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T15:58:29.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:28 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T15:58:29.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:58:29.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:58:29.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:58:29.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:28 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T15:58:29.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:28 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T15:58:29.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:28 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi038 to 197.2M 2023-03-09T15:58:29.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:28 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi038 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-03-09T15:58:29.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:58:29.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:58:29.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:58:29.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:58:29.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:28 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T15:58:29.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:28 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T15:58:29.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:58:29.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:58:29.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:58:29.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:28 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T15:58:29.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:28 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T15:58:29.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:28 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi038 to 197.2M 2023-03-09T15:58:29.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:28 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi038 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-03-09T15:58:29.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:58:29.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:58:29.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:58:29.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T15:58:29.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:28 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T15:58:29.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:28 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T15:58:29.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:58:29.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:58:29.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:58:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:29 smithi038 ceph-mon[92422]: pgmap v2446: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 614 B/s wr, 7 op/s 2023-03-09T15:58:30.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:29 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3620.987020 secs 2023-03-09T15:58:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:29 smithi122 ceph-mon[93869]: pgmap v2446: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 614 B/s wr, 7 op/s 2023-03-09T15:58:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:29 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3620.987020 secs 2023-03-09T15:58:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:29 smithi181 ceph-mon[94189]: pgmap v2446: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 614 B/s wr, 7 op/s 2023-03-09T15:58:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:29 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3620.987020 secs 2023-03-09T15:58:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:31 smithi038 ceph-mon[92422]: pgmap v2447: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 542 B/s wr, 6 op/s 2023-03-09T15:58:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:31 smithi122 ceph-mon[93869]: pgmap v2447: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 542 B/s wr, 6 op/s 2023-03-09T15:58:32.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:31 smithi181 ceph-mon[94189]: pgmap v2447: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 542 B/s wr, 6 op/s 2023-03-09T15:58:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4085.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:58:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:34 smithi122 ceph-mon[93869]: pgmap v2448: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 31 KiB/s wr, 9 op/s 2023-03-09T15:58:34.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:34 smithi181 ceph-mon[94189]: pgmap v2448: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 31 KiB/s wr, 9 op/s 2023-03-09T15:58:34.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:34 smithi038 ceph-mon[92422]: pgmap v2448: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 31 KiB/s wr, 9 op/s 2023-03-09T15:58:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3625.987078 secs 2023-03-09T15:58:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:35 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:58:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3625.987078 secs 2023-03-09T15:58:35.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:35 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:58:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3625.987078 secs 2023-03-09T15:58:35.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:35 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:58:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:36 smithi122 ceph-mon[93869]: pgmap v2449: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 30 KiB/s wr, 6 op/s 2023-03-09T15:58:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:36 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:58:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:58:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:58:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:36 smithi181 ceph-mon[94189]: pgmap v2449: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 30 KiB/s wr, 6 op/s 2023-03-09T15:58:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:36 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:58:36.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:58:36.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:58:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:36 smithi038 ceph-mon[92422]: pgmap v2449: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 30 KiB/s wr, 6 op/s 2023-03-09T15:58:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:36 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:58:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:58:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:58:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:38 smithi122 ceph-mon[93869]: pgmap v2450: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 107 KiB/s wr, 9 op/s 2023-03-09T15:58:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:38 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 3628.202642 secs 2023-03-09T15:58:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:38 smithi181 ceph-mon[94189]: pgmap v2450: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 107 KiB/s wr, 9 op/s 2023-03-09T15:58:38.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:38 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 3628.202642 secs 2023-03-09T15:58:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:38 smithi038 ceph-mon[92422]: pgmap v2450: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 107 KiB/s wr, 9 op/s 2023-03-09T15:58:38.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:38 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 3628.202642 secs 2023-03-09T15:58:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4090.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:58:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:40 smithi122 ceph-mon[93869]: pgmap v2451: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 110 KiB/s wr, 8 op/s 2023-03-09T15:58:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:58:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:58:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3630.987139 secs 2023-03-09T15:58:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:40 smithi181 ceph-mon[94189]: pgmap v2451: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 110 KiB/s wr, 8 op/s 2023-03-09T15:58:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:58:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:58:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3630.987139 secs 2023-03-09T15:58:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:40 smithi038 ceph-mon[92422]: pgmap v2451: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 110 KiB/s wr, 8 op/s 2023-03-09T15:58:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:58:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:58:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3630.987139 secs 2023-03-09T15:58:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:42 smithi122 ceph-mon[93869]: pgmap v2452: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 110 KiB/s wr, 7 op/s 2023-03-09T15:58:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:42 smithi181 ceph-mon[94189]: pgmap v2452: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 110 KiB/s wr, 7 op/s 2023-03-09T15:58:42.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:42 smithi038 ceph-mon[92422]: pgmap v2452: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 110 KiB/s wr, 7 op/s 2023-03-09T15:58:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3633.202759 secs 2023-03-09T15:58:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3633.202759 secs 2023-03-09T15:58:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3633.202759 secs 2023-03-09T15:58:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4095.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:58:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:44 smithi122 ceph-mon[93869]: pgmap v2453: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 174 KiB/s wr, 11 op/s 2023-03-09T15:58:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:44 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:58:44.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:44 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:58:44.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:44 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:58:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:44 smithi181 ceph-mon[94189]: pgmap v2453: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 174 KiB/s wr, 11 op/s 2023-03-09T15:58:44.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:44 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:58:44.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:44 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:58:44.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:44 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:58:44.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:44 smithi038 ceph-mon[92422]: pgmap v2453: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 174 KiB/s wr, 11 op/s 2023-03-09T15:58:44.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:44 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:58:44.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:44 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:58:44.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:44 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:58:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3635.987201 secs 2023-03-09T15:58:45.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3635.987201 secs 2023-03-09T15:58:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3635.987201 secs 2023-03-09T15:58:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:46 smithi122 ceph-mon[93869]: pgmap v2454: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 145 KiB/s wr, 9 op/s 2023-03-09T15:58:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:46 smithi181 ceph-mon[94189]: pgmap v2454: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 145 KiB/s wr, 9 op/s 2023-03-09T15:58:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:46 smithi038 ceph-mon[92422]: pgmap v2454: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 145 KiB/s wr, 9 op/s 2023-03-09T15:58:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:48 smithi122 ceph-mon[93869]: pgmap v2455: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 198 KiB/s wr, 12 op/s 2023-03-09T15:58:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3638.202872 secs 2023-03-09T15:58:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:48 smithi181 ceph-mon[94189]: pgmap v2455: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 198 KiB/s wr, 12 op/s 2023-03-09T15:58:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3638.202872 secs 2023-03-09T15:58:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:48 smithi038 ceph-mon[92422]: pgmap v2455: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 198 KiB/s wr, 12 op/s 2023-03-09T15:58:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3638.202872 secs 2023-03-09T15:58:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4100.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:58:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:50 smithi122 ceph-mon[93869]: pgmap v2456: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 129 KiB/s wr, 11 op/s 2023-03-09T15:58:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3640.987261 secs 2023-03-09T15:58:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:50 smithi181 ceph-mon[94189]: pgmap v2456: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 129 KiB/s wr, 11 op/s 2023-03-09T15:58:50.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3640.987261 secs 2023-03-09T15:58:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:50 smithi038 ceph-mon[92422]: pgmap v2456: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 129 KiB/s wr, 11 op/s 2023-03-09T15:58:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3640.987261 secs 2023-03-09T15:58:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:52 smithi122 ceph-mon[93869]: pgmap v2457: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 165 KiB/s wr, 11 op/s 2023-03-09T15:58:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:52 smithi181 ceph-mon[94189]: pgmap v2457: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 165 KiB/s wr, 11 op/s 2023-03-09T15:58:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:52 smithi038 ceph-mon[92422]: pgmap v2457: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 165 KiB/s wr, 11 op/s 2023-03-09T15:58:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3643.203023 secs 2023-03-09T15:58:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3643.203023 secs 2023-03-09T15:58:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3643.203023 secs 2023-03-09T15:58:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4105.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:58:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:54 smithi122 ceph-mon[93869]: pgmap v2458: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 169 KiB/s wr, 14 op/s 2023-03-09T15:58:54.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:54 smithi181 ceph-mon[94189]: pgmap v2458: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 169 KiB/s wr, 14 op/s 2023-03-09T15:58:54.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:54 smithi038 ceph-mon[92422]: pgmap v2458: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 169 KiB/s wr, 14 op/s 2023-03-09T15:58:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3645.987320 secs 2023-03-09T15:58:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3645.987320 secs 2023-03-09T15:58:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3645.987320 secs 2023-03-09T15:58:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:56 smithi122 ceph-mon[93869]: pgmap v2459: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 105 KiB/s wr, 10 op/s 2023-03-09T15:58:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:56 smithi181 ceph-mon[94189]: pgmap v2459: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 105 KiB/s wr, 10 op/s 2023-03-09T15:58:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:56 smithi038 ceph-mon[92422]: pgmap v2459: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 105 KiB/s wr, 10 op/s 2023-03-09T15:58:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:58 smithi122 ceph-mon[93869]: pgmap v2460: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 144 KiB/s wr, 13 op/s 2023-03-09T15:58:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:58:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3648.203140 secs 2023-03-09T15:58:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:58 smithi181 ceph-mon[94189]: pgmap v2460: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 144 KiB/s wr, 13 op/s 2023-03-09T15:58:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:58:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3648.203140 secs 2023-03-09T15:58:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:58 smithi038 ceph-mon[92422]: pgmap v2460: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 144 KiB/s wr, 13 op/s 2023-03-09T15:58:58.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:58:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3648.203140 secs 2023-03-09T15:58:58.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4110.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:59:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:59:00 smithi122 ceph-mon[93869]: pgmap v2461: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 91 KiB/s wr, 10 op/s 2023-03-09T15:59:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:59:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3650.987378 secs 2023-03-09T15:59:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:59:00 smithi181 ceph-mon[94189]: pgmap v2461: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 91 KiB/s wr, 10 op/s 2023-03-09T15:59:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:59:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3650.987378 secs 2023-03-09T15:59:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:59:00 smithi038 ceph-mon[92422]: pgmap v2461: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 91 KiB/s wr, 10 op/s 2023-03-09T15:59:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:59:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3650.987378 secs 2023-03-09T15:59:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:59:02 smithi122 ceph-mon[93869]: pgmap v2462: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 108 KiB/s wr, 9 op/s 2023-03-09T15:59:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:59:02 smithi181 ceph-mon[94189]: pgmap v2462: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 108 KiB/s wr, 9 op/s 2023-03-09T15:59:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:59:02 smithi038 ceph-mon[92422]: pgmap v2462: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 108 KiB/s wr, 9 op/s 2023-03-09T15:59:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:59:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3653.203284 secs 2023-03-09T15:59:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:59:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3653.203284 secs 2023-03-09T15:59:03.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:59:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3653.203284 secs 2023-03-09T15:59:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4115.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:59:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:59:04 smithi122 ceph-mon[93869]: pgmap v2463: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 90 KiB/s wr, 13 op/s 2023-03-09T15:59:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:59:04 smithi181 ceph-mon[94189]: pgmap v2463: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 90 KiB/s wr, 13 op/s 2023-03-09T15:59:04.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:59:04 smithi038 ceph-mon[92422]: pgmap v2463: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 90 KiB/s wr, 13 op/s 2023-03-09T15:59:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:59:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3655.987437 secs 2023-03-09T15:59:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:59:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3655.987437 secs 2023-03-09T15:59:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:59:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3655.987437 secs 2023-03-09T15:59:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:59:06 smithi122 ceph-mon[93869]: pgmap v2464: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 108 KiB/s wr, 22 op/s 2023-03-09T15:59:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:59:06 smithi181 ceph-mon[94189]: pgmap v2464: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 108 KiB/s wr, 22 op/s 2023-03-09T15:59:06.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:59:06 smithi038 ceph-mon[92422]: pgmap v2464: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 108 KiB/s wr, 22 op/s 2023-03-09T15:59:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:59:08 smithi122 ceph-mon[93869]: pgmap v2465: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 109 KiB/s wr, 24 op/s 2023-03-09T15:59:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:59:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3658.203397 secs 2023-03-09T15:59:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:59:08 smithi181 ceph-mon[94189]: pgmap v2465: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 109 KiB/s wr, 24 op/s 2023-03-09T15:59:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:59:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3658.203397 secs 2023-03-09T15:59:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:59:08 smithi038 ceph-mon[92422]: pgmap v2465: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 109 KiB/s wr, 24 op/s 2023-03-09T15:59:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:59:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3658.203397 secs 2023-03-09T15:59:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4120.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:59:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:59:10 smithi122 ceph-mon[93869]: pgmap v2466: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 69 KiB/s wr, 21 op/s 2023-03-09T15:59:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:59:10 smithi122 ceph-mon[93869]: osdmap e368: 12 total, 12 up, 12 in 2023-03-09T15:59:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:59:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3660.987523 secs 2023-03-09T15:59:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:59:10 smithi181 ceph-mon[94189]: pgmap v2466: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 69 KiB/s wr, 21 op/s 2023-03-09T15:59:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:59:10 smithi181 ceph-mon[94189]: osdmap e368: 12 total, 12 up, 12 in 2023-03-09T15:59:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:59:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3660.987523 secs 2023-03-09T15:59:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:59:10 smithi038 ceph-mon[92422]: pgmap v2466: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 69 KiB/s wr, 21 op/s 2023-03-09T15:59:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:59:10 smithi038 ceph-mon[92422]: osdmap e368: 12 total, 12 up, 12 in 2023-03-09T15:59:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:59:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3660.987523 secs 2023-03-09T15:59:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:59:12 smithi122 ceph-mon[93869]: pgmap v2468: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 54 KiB/s wr, 24 op/s 2023-03-09T15:59:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:59:12 smithi181 ceph-mon[94189]: pgmap v2468: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 54 KiB/s wr, 24 op/s 2023-03-09T15:59:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:59:12 smithi038 ceph-mon[92422]: pgmap v2468: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 54 KiB/s wr, 24 op/s 2023-03-09T15:59:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:59:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3663.203476 secs 2023-03-09T15:59:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:59:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3663.203476 secs 2023-03-09T15:59:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:59:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3663.203476 secs 2023-03-09T15:59:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4125.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:59:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:59:14 smithi122 ceph-mon[93869]: pgmap v2469: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 27 KiB/s wr, 21 op/s 2023-03-09T15:59:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:59:14 smithi181 ceph-mon[94189]: pgmap v2469: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 27 KiB/s wr, 21 op/s 2023-03-09T15:59:14.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:59:14 smithi038 ceph-mon[92422]: pgmap v2469: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 27 KiB/s wr, 21 op/s 2023-03-09T15:59:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:59:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3665.987600 secs 2023-03-09T15:59:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:59:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3665.987600 secs 2023-03-09T15:59:15.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:59:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3665.987600 secs 2023-03-09T15:59:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:59:16 smithi122 ceph-mon[93869]: pgmap v2470: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2023-03-09T15:59:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:59:16 smithi181 ceph-mon[94189]: pgmap v2470: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2023-03-09T15:59:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:59:16 smithi038 ceph-mon[92422]: pgmap v2470: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2023-03-09T15:59:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:59:18 smithi122 ceph-mon[93869]: pgmap v2471: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:59:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:59:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3668.203596 secs 2023-03-09T15:59:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:59:18 smithi181 ceph-mon[94189]: pgmap v2471: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:59:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:59:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3668.203596 secs 2023-03-09T15:59:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:59:18 smithi038 ceph-mon[92422]: pgmap v2471: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T15:59:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:59:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3668.203596 secs 2023-03-09T15:59:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4130.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:59:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:59:20 smithi122 ceph-mon[93869]: pgmap v2472: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 614 B/s wr, 6 op/s 2023-03-09T15:59:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:59:20 smithi122 ceph-mon[93869]: osdmap e369: 12 total, 12 up, 12 in 2023-03-09T15:59:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:59:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3670.987661 secs 2023-03-09T15:59:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:59:20 smithi181 ceph-mon[94189]: pgmap v2472: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 614 B/s wr, 6 op/s 2023-03-09T15:59:20.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:59:20 smithi181 ceph-mon[94189]: osdmap e369: 12 total, 12 up, 12 in 2023-03-09T15:59:20.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:59:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3670.987661 secs 2023-03-09T15:59:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:59:20 smithi038 ceph-mon[92422]: pgmap v2472: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 614 B/s wr, 6 op/s 2023-03-09T15:59:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:59:20 smithi038 ceph-mon[92422]: osdmap e369: 12 total, 12 up, 12 in 2023-03-09T15:59:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:59:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3670.987661 secs 2023-03-09T15:59:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:59:22 smithi122 ceph-mon[93869]: pgmap v2474: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 24 KiB/s wr, 6 op/s 2023-03-09T15:59:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:59:22 smithi181 ceph-mon[94189]: pgmap v2474: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 24 KiB/s wr, 6 op/s 2023-03-09T15:59:22.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:59:22 smithi038 ceph-mon[92422]: pgmap v2474: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 24 KiB/s wr, 6 op/s 2023-03-09T15:59:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:59:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3673.203730 secs 2023-03-09T15:59:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:59:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3673.203730 secs 2023-03-09T15:59:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:59:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3673.203730 secs 2023-03-09T15:59:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4135.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:59:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:59:24 smithi122 ceph-mon[93869]: pgmap v2475: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 7 op/s 2023-03-09T15:59:24.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:59:24 smithi181 ceph-mon[94189]: pgmap v2475: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 7 op/s 2023-03-09T15:59:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:59:24 smithi038 ceph-mon[92422]: pgmap v2475: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 7 op/s 2023-03-09T15:59:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:59:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3675.987721 secs 2023-03-09T15:59:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:59:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3675.987721 secs 2023-03-09T15:59:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:59:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3675.987721 secs 2023-03-09T15:59:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:59:26 smithi122 ceph-mon[93869]: pgmap v2476: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 7 op/s 2023-03-09T15:59:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:59:26 smithi181 ceph-mon[94189]: pgmap v2476: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 7 op/s 2023-03-09T15:59:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:59:26 smithi038 ceph-mon[92422]: pgmap v2476: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 7 op/s 2023-03-09T15:59:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:59:28 smithi122 ceph-mon[93869]: pgmap v2477: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 7 op/s 2023-03-09T15:59:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:59:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3678.203852 secs 2023-03-09T15:59:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:59:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:59:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:59:28 smithi181 ceph-mon[94189]: pgmap v2477: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 7 op/s 2023-03-09T15:59:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:59:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3678.203852 secs 2023-03-09T15:59:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:59:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:59:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:59:28 smithi038 ceph-mon[92422]: pgmap v2477: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 7 op/s 2023-03-09T15:59:28.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:59:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3678.203852 secs 2023-03-09T15:59:28.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:59:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T15:59:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4140.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:59:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:59:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:59:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:59:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:59:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:59:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:59:29.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:59:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:59:29.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:59:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:59:29.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:59:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:59:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:59:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T15:59:29.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:59:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T15:59:29.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:59:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T15:59:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:59:30 smithi122 ceph-mon[93869]: pgmap v2478: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 7 op/s 2023-03-09T15:59:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:59:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3680.987779 secs 2023-03-09T15:59:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:59:30 smithi181 ceph-mon[94189]: pgmap v2478: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 7 op/s 2023-03-09T15:59:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:59:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3680.987779 secs 2023-03-09T15:59:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:59:30 smithi038 ceph-mon[92422]: pgmap v2478: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 7 op/s 2023-03-09T15:59:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:59:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3680.987779 secs 2023-03-09T15:59:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:59:32 smithi122 ceph-mon[93869]: pgmap v2479: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 75 KiB/s wr, 7 op/s 2023-03-09T15:59:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:59:32 smithi181 ceph-mon[94189]: pgmap v2479: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 75 KiB/s wr, 7 op/s 2023-03-09T15:59:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:59:32 smithi038 ceph-mon[92422]: pgmap v2479: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 75 KiB/s wr, 7 op/s 2023-03-09T15:59:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:59:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3683.203967 secs 2023-03-09T15:59:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:59:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3683.203967 secs 2023-03-09T15:59:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:59:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3683.203967 secs 2023-03-09T15:59:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4145.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:59:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:59:34 smithi122 ceph-mon[93869]: pgmap v2480: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 51 KiB/s wr, 8 op/s 2023-03-09T15:59:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:59:34 smithi181 ceph-mon[94189]: pgmap v2480: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 51 KiB/s wr, 8 op/s 2023-03-09T15:59:34.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:59:34 smithi038 ceph-mon[92422]: pgmap v2480: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 51 KiB/s wr, 8 op/s 2023-03-09T15:59:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:59:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3685.987838 secs 2023-03-09T15:59:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:59:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3685.987838 secs 2023-03-09T15:59:35.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:59:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3685.987838 secs 2023-03-09T15:59:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:59:36 smithi122 ceph-mon[93869]: pgmap v2481: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 50 KiB/s wr, 6 op/s 2023-03-09T15:59:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:59:36 smithi181 ceph-mon[94189]: pgmap v2481: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 50 KiB/s wr, 6 op/s 2023-03-09T15:59:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:59:36 smithi038 ceph-mon[92422]: pgmap v2481: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 50 KiB/s wr, 6 op/s 2023-03-09T15:59:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:59:38 smithi122 ceph-mon[93869]: pgmap v2482: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 128 KiB/s wr, 8 op/s 2023-03-09T15:59:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:59:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3688.204086 secs 2023-03-09T15:59:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:59:38 smithi181 ceph-mon[94189]: pgmap v2482: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 128 KiB/s wr, 8 op/s 2023-03-09T15:59:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:59:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3688.204086 secs 2023-03-09T15:59:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:59:38 smithi038 ceph-mon[92422]: pgmap v2482: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 128 KiB/s wr, 8 op/s 2023-03-09T15:59:38.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:59:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3688.204086 secs 2023-03-09T15:59:38.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4150.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:59:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:59:40 smithi122 ceph-mon[93869]: pgmap v2483: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 108 KiB/s wr, 7 op/s 2023-03-09T15:59:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:59:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3690.987902 secs 2023-03-09T15:59:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:59:40 smithi181 ceph-mon[94189]: pgmap v2483: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 108 KiB/s wr, 7 op/s 2023-03-09T15:59:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:59:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3690.987902 secs 2023-03-09T15:59:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:59:40 smithi038 ceph-mon[92422]: pgmap v2483: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 108 KiB/s wr, 7 op/s 2023-03-09T15:59:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:59:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3690.987902 secs 2023-03-09T15:59:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:59:42 smithi122 ceph-mon[93869]: pgmap v2484: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 165 KiB/s wr, 8 op/s 2023-03-09T15:59:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:59:42 smithi181 ceph-mon[94189]: pgmap v2484: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 165 KiB/s wr, 8 op/s 2023-03-09T15:59:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:59:42 smithi038 ceph-mon[92422]: pgmap v2484: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 165 KiB/s wr, 8 op/s 2023-03-09T15:59:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:59:43 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:59:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:59:43 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:59:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:59:43 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T15:59:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4155.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:59:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:59:44 smithi122 ceph-mon[93869]: pgmap v2485: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 180 KiB/s wr, 11 op/s 2023-03-09T15:59:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:59:44 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:59:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:59:44 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:59:44.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:59:44 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:59:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:59:44 smithi181 ceph-mon[94189]: pgmap v2485: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 180 KiB/s wr, 11 op/s 2023-03-09T15:59:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:59:44 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:59:44.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:59:44 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:59:44.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:59:44 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:59:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:59:44 smithi038 ceph-mon[92422]: pgmap v2485: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 180 KiB/s wr, 11 op/s 2023-03-09T15:59:44.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:59:44 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:59:44.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:59:44 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:59:44.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:59:44 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:59:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:59:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3695.987963 secs 2023-03-09T15:59:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:59:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3695.987963 secs 2023-03-09T15:59:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:59:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3695.987963 secs 2023-03-09T15:59:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:59:46 smithi122 ceph-mon[93869]: pgmap v2486: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 181 KiB/s wr, 9 op/s 2023-03-09T15:59:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:59:46 smithi181 ceph-mon[94189]: pgmap v2486: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 181 KiB/s wr, 9 op/s 2023-03-09T15:59:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:59:46 smithi038 ceph-mon[92422]: pgmap v2486: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 181 KiB/s wr, 9 op/s 2023-03-09T15:59:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:59:48 smithi122 ceph-mon[93869]: pgmap v2487: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 181 KiB/s wr, 12 op/s 2023-03-09T15:59:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:59:48 smithi181 ceph-mon[94189]: pgmap v2487: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 181 KiB/s wr, 12 op/s 2023-03-09T15:59:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:59:48 smithi038 ceph-mon[92422]: pgmap v2487: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 181 KiB/s wr, 12 op/s 2023-03-09T15:59:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4160.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:59:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:59:50 smithi122 ceph-mon[93869]: pgmap v2488: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 131 KiB/s wr, 11 op/s 2023-03-09T15:59:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:59:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3700.987986 secs 2023-03-09T15:59:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:59:50 smithi181 ceph-mon[94189]: pgmap v2488: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 131 KiB/s wr, 11 op/s 2023-03-09T15:59:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:59:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3700.987986 secs 2023-03-09T15:59:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:59:50 smithi038 ceph-mon[92422]: pgmap v2488: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 131 KiB/s wr, 11 op/s 2023-03-09T15:59:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:59:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3700.987986 secs 2023-03-09T15:59:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:59:52 smithi122 ceph-mon[93869]: pgmap v2489: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 127 KiB/s wr, 11 op/s 2023-03-09T15:59:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:59:52 smithi181 ceph-mon[94189]: pgmap v2489: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 127 KiB/s wr, 11 op/s 2023-03-09T15:59:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:59:52 smithi038 ceph-mon[92422]: pgmap v2489: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 127 KiB/s wr, 11 op/s 2023-03-09T15:59:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:59:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3703.204409 secs 2023-03-09T15:59:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:59:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3703.204409 secs 2023-03-09T15:59:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:59:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3703.204409 secs 2023-03-09T15:59:53.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4165.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T15:59:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:59:54 smithi122 ceph-mon[93869]: pgmap v2490: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 151 KiB/s wr, 13 op/s 2023-03-09T15:59:54.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:59:54 smithi181 ceph-mon[94189]: pgmap v2490: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 151 KiB/s wr, 13 op/s 2023-03-09T15:59:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:59:54 smithi038 ceph-mon[92422]: pgmap v2490: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 151 KiB/s wr, 13 op/s 2023-03-09T15:59:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:59:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3705.988048 secs 2023-03-09T15:59:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:59:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3705.988048 secs 2023-03-09T15:59:55.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:59:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3705.988048 secs 2023-03-09T15:59:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:59:56 smithi122 ceph-mon[93869]: pgmap v2491: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 110 KiB/s wr, 10 op/s 2023-03-09T15:59:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:59:56 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:59:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:59:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:59:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:59:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:59:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:59:56 smithi181 ceph-mon[94189]: pgmap v2491: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 110 KiB/s wr, 10 op/s 2023-03-09T15:59:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:59:56 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:59:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:59:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:59:56.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:59:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:59:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:59:56 smithi038 ceph-mon[92422]: pgmap v2491: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 110 KiB/s wr, 10 op/s 2023-03-09T15:59:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:59:56 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T15:59:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:59:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T15:59:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:59:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T15:59:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:59:58 smithi122 ceph-mon[93869]: pgmap v2492: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 110 KiB/s wr, 11 op/s 2023-03-09T15:59:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 15:59:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3708.204523 secs 2023-03-09T15:59:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:59:58 smithi181 ceph-mon[94189]: pgmap v2492: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 110 KiB/s wr, 11 op/s 2023-03-09T15:59:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 15:59:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3708.204523 secs 2023-03-09T15:59:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:59:58 smithi038 ceph-mon[92422]: pgmap v2492: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 110 KiB/s wr, 11 op/s 2023-03-09T15:59:58.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 15:59:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3708.204523 secs 2023-03-09T15:59:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4170.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:00:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:00:00 smithi122 ceph-mon[93869]: pgmap v2493: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 120 KiB/s wr, 10 op/s 2023-03-09T16:00:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:00:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3710.988109 secs 2023-03-09T16:00:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:00:00 smithi122 ceph-mon[93869]: overall HEALTH_WARN 2 MDSs report slow requests 2023-03-09T16:00:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:00:00 smithi181 ceph-mon[94189]: pgmap v2493: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 120 KiB/s wr, 10 op/s 2023-03-09T16:00:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:00:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3710.988109 secs 2023-03-09T16:00:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:00:00 smithi181 ceph-mon[94189]: overall HEALTH_WARN 2 MDSs report slow requests 2023-03-09T16:00:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:00:00 smithi038 ceph-mon[92422]: pgmap v2493: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 120 KiB/s wr, 10 op/s 2023-03-09T16:00:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:00:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3710.988109 secs 2023-03-09T16:00:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:00:00 smithi038 ceph-mon[92422]: overall HEALTH_WARN 2 MDSs report slow requests 2023-03-09T16:00:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:00:02 smithi122 ceph-mon[93869]: pgmap v2494: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 93 KiB/s wr, 8 op/s 2023-03-09T16:00:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:00:02 smithi181 ceph-mon[94189]: pgmap v2494: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 93 KiB/s wr, 8 op/s 2023-03-09T16:00:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:00:02 smithi038 ceph-mon[92422]: pgmap v2494: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 93 KiB/s wr, 8 op/s 2023-03-09T16:00:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:00:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3713.204639 secs 2023-03-09T16:00:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:00:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3713.204639 secs 2023-03-09T16:00:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:00:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3713.204639 secs 2023-03-09T16:00:03.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4175.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:00:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:00:04 smithi122 ceph-mon[93869]: pgmap v2495: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 140 KiB/s wr, 13 op/s 2023-03-09T16:00:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:00:04 smithi181 ceph-mon[94189]: pgmap v2495: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 140 KiB/s wr, 13 op/s 2023-03-09T16:00:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:00:04 smithi038 ceph-mon[92422]: pgmap v2495: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 140 KiB/s wr, 13 op/s 2023-03-09T16:00:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:00:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3715.988169 secs 2023-03-09T16:00:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:00:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3715.988169 secs 2023-03-09T16:00:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:00:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3715.988169 secs 2023-03-09T16:00:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:00:06 smithi122 ceph-mon[93869]: pgmap v2496: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 79 KiB/s wr, 21 op/s 2023-03-09T16:00:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:00:06 smithi181 ceph-mon[94189]: pgmap v2496: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 79 KiB/s wr, 21 op/s 2023-03-09T16:00:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:00:06 smithi038 ceph-mon[92422]: pgmap v2496: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 79 KiB/s wr, 21 op/s 2023-03-09T16:00:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:00:08 smithi122 ceph-mon[93869]: pgmap v2497: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 79 KiB/s wr, 22 op/s 2023-03-09T16:00:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:00:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3718.204763 secs 2023-03-09T16:00:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:00:08 smithi181 ceph-mon[94189]: pgmap v2497: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 79 KiB/s wr, 22 op/s 2023-03-09T16:00:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:00:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3718.204763 secs 2023-03-09T16:00:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:00:08 smithi038 ceph-mon[92422]: pgmap v2497: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 79 KiB/s wr, 22 op/s 2023-03-09T16:00:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:00:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3718.204763 secs 2023-03-09T16:00:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4180.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:00:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:00:10 smithi181 ceph-mon[94189]: pgmap v2498: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 79 KiB/s wr, 21 op/s 2023-03-09T16:00:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:00:10 smithi181 ceph-mon[94189]: osdmap e370: 12 total, 12 up, 12 in 2023-03-09T16:00:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:00:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3720.988229 secs 2023-03-09T16:00:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:00:10 smithi038 ceph-mon[92422]: pgmap v2498: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 79 KiB/s wr, 21 op/s 2023-03-09T16:00:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:00:10 smithi038 ceph-mon[92422]: osdmap e370: 12 total, 12 up, 12 in 2023-03-09T16:00:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:00:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3720.988229 secs 2023-03-09T16:00:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:00:10 smithi122 ceph-mon[93869]: pgmap v2498: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 79 KiB/s wr, 21 op/s 2023-03-09T16:00:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:00:10 smithi122 ceph-mon[93869]: osdmap e370: 12 total, 12 up, 12 in 2023-03-09T16:00:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:00:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3720.988229 secs 2023-03-09T16:00:11.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:00:11 smithi122 ceph-mon[93869]: pgmap v2500: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 82 KiB/s wr, 24 op/s 2023-03-09T16:00:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:00:11 smithi181 ceph-mon[94189]: pgmap v2500: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 82 KiB/s wr, 24 op/s 2023-03-09T16:00:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:00:11 smithi038 ceph-mon[92422]: pgmap v2500: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 82 KiB/s wr, 24 op/s 2023-03-09T16:00:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:00:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3723.204886 secs 2023-03-09T16:00:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:00:13 smithi122 ceph-mon[93869]: pgmap v2501: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 46 KiB/s wr, 20 op/s 2023-03-09T16:00:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:00:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3723.204886 secs 2023-03-09T16:00:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:00:13 smithi181 ceph-mon[94189]: pgmap v2501: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 46 KiB/s wr, 20 op/s 2023-03-09T16:00:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:00:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3723.204886 secs 2023-03-09T16:00:13.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:00:13 smithi038 ceph-mon[92422]: pgmap v2501: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 46 KiB/s wr, 20 op/s 2023-03-09T16:00:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4185.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:00:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:00:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3725.988291 secs 2023-03-09T16:00:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:00:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3725.988291 secs 2023-03-09T16:00:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:00:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3725.988291 secs 2023-03-09T16:00:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:00:16 smithi122 ceph-mon[93869]: pgmap v2502: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 6 op/s 2023-03-09T16:00:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:00:16 smithi181 ceph-mon[94189]: pgmap v2502: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 6 op/s 2023-03-09T16:00:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:00:16 smithi038 ceph-mon[92422]: pgmap v2502: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 6 op/s 2023-03-09T16:00:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:00:18 smithi122 ceph-mon[93869]: pgmap v2503: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 6 op/s 2023-03-09T16:00:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:00:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3728.204981 secs 2023-03-09T16:00:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:00:18 smithi181 ceph-mon[94189]: pgmap v2503: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 6 op/s 2023-03-09T16:00:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:00:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3728.204981 secs 2023-03-09T16:00:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:00:18 smithi038 ceph-mon[92422]: pgmap v2503: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 6 op/s 2023-03-09T16:00:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:00:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3728.204981 secs 2023-03-09T16:00:18.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4190.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:00:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:00:20 smithi122 ceph-mon[93869]: pgmap v2504: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 6 op/s 2023-03-09T16:00:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:00:20 smithi122 ceph-mon[93869]: osdmap e371: 12 total, 12 up, 12 in 2023-03-09T16:00:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:00:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3730.988353 secs 2023-03-09T16:00:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:00:20 smithi181 ceph-mon[94189]: pgmap v2504: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 6 op/s 2023-03-09T16:00:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:00:20 smithi181 ceph-mon[94189]: osdmap e371: 12 total, 12 up, 12 in 2023-03-09T16:00:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:00:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3730.988353 secs 2023-03-09T16:00:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:00:20 smithi038 ceph-mon[92422]: pgmap v2504: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 6 op/s 2023-03-09T16:00:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:00:20 smithi038 ceph-mon[92422]: osdmap e371: 12 total, 12 up, 12 in 2023-03-09T16:00:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:00:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3730.988353 secs 2023-03-09T16:00:22.398 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:00:22 smithi181 ceph-mon[94189]: pgmap v2506: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 6 op/s 2023-03-09T16:00:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:00:22 smithi122 ceph-mon[93869]: pgmap v2506: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 6 op/s 2023-03-09T16:00:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:00:22 smithi038 ceph-mon[92422]: pgmap v2506: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 6 op/s 2023-03-09T16:00:23.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:00:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3733.205101 secs 2023-03-09T16:00:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:00:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3733.205101 secs 2023-03-09T16:00:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:00:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3733.205101 secs 2023-03-09T16:00:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4195.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:00:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:00:24 smithi122 ceph-mon[93869]: pgmap v2507: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 29 KiB/s wr, 7 op/s 2023-03-09T16:00:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:00:24 smithi181 ceph-mon[94189]: pgmap v2507: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 29 KiB/s wr, 7 op/s 2023-03-09T16:00:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:00:24 smithi038 ceph-mon[92422]: pgmap v2507: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 29 KiB/s wr, 7 op/s 2023-03-09T16:00:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:00:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3735.988411 secs 2023-03-09T16:00:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:00:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3735.988411 secs 2023-03-09T16:00:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:00:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3735.988411 secs 2023-03-09T16:00:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:00:26 smithi122 ceph-mon[93869]: pgmap v2508: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 29 KiB/s wr, 7 op/s 2023-03-09T16:00:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:00:26 smithi181 ceph-mon[94189]: pgmap v2508: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 29 KiB/s wr, 7 op/s 2023-03-09T16:00:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:00:26 smithi038 ceph-mon[92422]: pgmap v2508: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 29 KiB/s wr, 7 op/s 2023-03-09T16:00:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:00:28 smithi122 ceph-mon[93869]: pgmap v2509: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 29 KiB/s wr, 7 op/s 2023-03-09T16:00:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:00:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3738.205170 secs 2023-03-09T16:00:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:00:28 smithi181 ceph-mon[94189]: pgmap v2509: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 29 KiB/s wr, 7 op/s 2023-03-09T16:00:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:00:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3738.205170 secs 2023-03-09T16:00:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:00:28 smithi038 ceph-mon[92422]: pgmap v2509: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 29 KiB/s wr, 7 op/s 2023-03-09T16:00:28.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:00:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3738.205170 secs 2023-03-09T16:00:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4200.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:00:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:00:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:00:29.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:00:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:00:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:00:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:00:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:00:30 smithi122 ceph-mon[93869]: pgmap v2510: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 29 KiB/s wr, 7 op/s 2023-03-09T16:00:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:00:30 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:00:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:00:30 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:00:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:00:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3740.988484 secs 2023-03-09T16:00:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:00:30 smithi181 ceph-mon[94189]: pgmap v2510: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 29 KiB/s wr, 7 op/s 2023-03-09T16:00:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:00:30 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:00:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:00:30 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:00:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:00:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3740.988484 secs 2023-03-09T16:00:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:00:30 smithi038 ceph-mon[92422]: pgmap v2510: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 29 KiB/s wr, 7 op/s 2023-03-09T16:00:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:00:30 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:00:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:00:30 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:00:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:00:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3740.988484 secs 2023-03-09T16:00:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:00:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:00:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:00:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:00:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:00:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:00:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:00:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:00:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:00:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:00:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:00:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:00:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:00:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:00:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:00:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:00:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:00:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:00:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:00:32 smithi122 ceph-mon[93869]: pgmap v2511: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 26 KiB/s wr, 6 op/s 2023-03-09T16:00:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:00:32 smithi181 ceph-mon[94189]: pgmap v2511: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 26 KiB/s wr, 6 op/s 2023-03-09T16:00:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:00:32 smithi038 ceph-mon[92422]: pgmap v2511: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 26 KiB/s wr, 6 op/s 2023-03-09T16:00:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4205.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:00:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:00:34 smithi122 ceph-mon[93869]: pgmap v2512: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 51 KiB/s wr, 9 op/s 2023-03-09T16:00:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:00:34 smithi181 ceph-mon[94189]: pgmap v2512: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 51 KiB/s wr, 9 op/s 2023-03-09T16:00:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:00:34 smithi038 ceph-mon[92422]: pgmap v2512: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 51 KiB/s wr, 9 op/s 2023-03-09T16:00:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:00:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3745.988545 secs 2023-03-09T16:00:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:00:35 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:00:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:00:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3745.988545 secs 2023-03-09T16:00:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:00:35 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:00:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:00:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3745.988545 secs 2023-03-09T16:00:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:00:35 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:00:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:00:36 smithi122 ceph-mon[93869]: pgmap v2513: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 27 KiB/s wr, 6 op/s 2023-03-09T16:00:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:00:36 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:00:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:00:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:00:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:00:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:00:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:00:36 smithi181 ceph-mon[94189]: pgmap v2513: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 27 KiB/s wr, 6 op/s 2023-03-09T16:00:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:00:36 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:00:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:00:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:00:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:00:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:00:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:00:36 smithi038 ceph-mon[92422]: pgmap v2513: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 27 KiB/s wr, 6 op/s 2023-03-09T16:00:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:00:36 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:00:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:00:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:00:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:00:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:00:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:00:38 smithi122 ceph-mon[93869]: pgmap v2514: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T16:00:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:00:38 smithi181 ceph-mon[94189]: pgmap v2514: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T16:00:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:00:38 smithi038 ceph-mon[92422]: pgmap v2514: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T16:00:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4210.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:00:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:00:40 smithi122 ceph-mon[93869]: pgmap v2515: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 109 KiB/s wr, 8 op/s 2023-03-09T16:00:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:00:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3750.988629 secs 2023-03-09T16:00:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:00:40 smithi181 ceph-mon[94189]: pgmap v2515: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 109 KiB/s wr, 8 op/s 2023-03-09T16:00:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:00:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3750.988629 secs 2023-03-09T16:00:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:00:40 smithi038 ceph-mon[92422]: pgmap v2515: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 109 KiB/s wr, 8 op/s 2023-03-09T16:00:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:00:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3750.988629 secs 2023-03-09T16:00:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:00:42 smithi122 ceph-mon[93869]: pgmap v2516: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 110 KiB/s wr, 7 op/s 2023-03-09T16:00:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:00:42 smithi181 ceph-mon[94189]: pgmap v2516: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 110 KiB/s wr, 7 op/s 2023-03-09T16:00:42.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:00:42 smithi038 ceph-mon[92422]: pgmap v2516: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 110 KiB/s wr, 7 op/s 2023-03-09T16:00:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4215.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:00:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:00:44 smithi122 ceph-mon[93869]: pgmap v2517: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 142 KiB/s wr, 11 op/s 2023-03-09T16:00:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:00:44 smithi181 ceph-mon[94189]: pgmap v2517: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 142 KiB/s wr, 11 op/s 2023-03-09T16:00:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:00:44 smithi038 ceph-mon[92422]: pgmap v2517: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 142 KiB/s wr, 11 op/s 2023-03-09T16:00:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:00:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3755.988687 secs 2023-03-09T16:00:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:00:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3755.988687 secs 2023-03-09T16:00:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:00:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3755.988687 secs 2023-03-09T16:00:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:00:46 smithi122 ceph-mon[93869]: pgmap v2518: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 183 KiB/s wr, 9 op/s 2023-03-09T16:00:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:00:46 smithi181 ceph-mon[94189]: pgmap v2518: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 183 KiB/s wr, 9 op/s 2023-03-09T16:00:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:00:46 smithi038 ceph-mon[92422]: pgmap v2518: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 183 KiB/s wr, 9 op/s 2023-03-09T16:00:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:00:48 smithi122 ceph-mon[93869]: pgmap v2519: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 183 KiB/s wr, 11 op/s 2023-03-09T16:00:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:00:48 smithi181 ceph-mon[94189]: pgmap v2519: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 183 KiB/s wr, 11 op/s 2023-03-09T16:00:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:00:48 smithi038 ceph-mon[92422]: pgmap v2519: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 183 KiB/s wr, 11 op/s 2023-03-09T16:00:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4220.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:00:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:00:50 smithi122 ceph-mon[93869]: pgmap v2520: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 210 KiB/s wr, 12 op/s 2023-03-09T16:00:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:00:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3760.988749 secs 2023-03-09T16:00:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:00:50 smithi181 ceph-mon[94189]: pgmap v2520: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 210 KiB/s wr, 12 op/s 2023-03-09T16:00:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:00:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3760.988749 secs 2023-03-09T16:00:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:00:50 smithi038 ceph-mon[92422]: pgmap v2520: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 210 KiB/s wr, 12 op/s 2023-03-09T16:00:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:00:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3760.988749 secs 2023-03-09T16:00:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:00:52 smithi122 ceph-mon[93869]: pgmap v2521: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 128 KiB/s wr, 10 op/s 2023-03-09T16:00:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:00:52 smithi181 ceph-mon[94189]: pgmap v2521: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 128 KiB/s wr, 10 op/s 2023-03-09T16:00:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:00:52 smithi038 ceph-mon[92422]: pgmap v2521: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 128 KiB/s wr, 10 op/s 2023-03-09T16:00:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:00:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3763.205725 secs 2023-03-09T16:00:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:00:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3763.205725 secs 2023-03-09T16:00:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:00:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3763.205725 secs 2023-03-09T16:00:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4225.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:00:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:00:54 smithi122 ceph-mon[93869]: pgmap v2522: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 131 KiB/s wr, 13 op/s 2023-03-09T16:00:54.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:00:54 smithi181 ceph-mon[94189]: pgmap v2522: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 131 KiB/s wr, 13 op/s 2023-03-09T16:00:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:00:54 smithi038 ceph-mon[92422]: pgmap v2522: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 131 KiB/s wr, 13 op/s 2023-03-09T16:00:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:00:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3765.988810 secs 2023-03-09T16:00:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:00:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3765.988810 secs 2023-03-09T16:00:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:00:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3765.988810 secs 2023-03-09T16:00:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:00:56 smithi122 ceph-mon[93869]: pgmap v2523: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 171 KiB/s wr, 10 op/s 2023-03-09T16:00:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:00:56 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:00:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:00:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:00:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:00:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:00:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:00:56 smithi181 ceph-mon[94189]: pgmap v2523: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 171 KiB/s wr, 10 op/s 2023-03-09T16:00:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:00:56 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:00:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:00:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:00:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:00:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:00:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:00:56 smithi038 ceph-mon[92422]: pgmap v2523: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 171 KiB/s wr, 10 op/s 2023-03-09T16:00:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:00:56 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:00:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:00:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:00:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:00:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:00:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:00:58 smithi122 ceph-mon[93869]: pgmap v2524: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 104 KiB/s wr, 11 op/s 2023-03-09T16:00:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:00:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3768.205844 secs 2023-03-09T16:00:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:00:58 smithi181 ceph-mon[94189]: pgmap v2524: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 104 KiB/s wr, 11 op/s 2023-03-09T16:00:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:00:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3768.205844 secs 2023-03-09T16:00:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:00:58 smithi038 ceph-mon[92422]: pgmap v2524: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 104 KiB/s wr, 11 op/s 2023-03-09T16:00:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:00:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3768.205844 secs 2023-03-09T16:00:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4230.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:01:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:01:00 smithi122 ceph-mon[93869]: pgmap v2525: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 114 KiB/s wr, 11 op/s 2023-03-09T16:01:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:01:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3770.988869 secs 2023-03-09T16:01:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:01:00 smithi181 ceph-mon[94189]: pgmap v2525: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 114 KiB/s wr, 11 op/s 2023-03-09T16:01:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:01:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3770.988869 secs 2023-03-09T16:01:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:01:00 smithi038 ceph-mon[92422]: pgmap v2525: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 114 KiB/s wr, 11 op/s 2023-03-09T16:01:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:01:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3770.988869 secs 2023-03-09T16:01:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:01:02 smithi122 ceph-mon[93869]: pgmap v2526: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 86 KiB/s wr, 8 op/s 2023-03-09T16:01:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:01:02 smithi181 ceph-mon[94189]: pgmap v2526: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 86 KiB/s wr, 8 op/s 2023-03-09T16:01:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:01:02 smithi038 ceph-mon[92422]: pgmap v2526: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 86 KiB/s wr, 8 op/s 2023-03-09T16:01:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:01:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3773.205961 secs 2023-03-09T16:01:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:01:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3773.205961 secs 2023-03-09T16:01:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:01:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3773.205961 secs 2023-03-09T16:01:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4235.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:01:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:01:04 smithi122 ceph-mon[93869]: pgmap v2527: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 109 KiB/s wr, 12 op/s 2023-03-09T16:01:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:01:04 smithi181 ceph-mon[94189]: pgmap v2527: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 109 KiB/s wr, 12 op/s 2023-03-09T16:01:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:01:04 smithi038 ceph-mon[92422]: pgmap v2527: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 109 KiB/s wr, 12 op/s 2023-03-09T16:01:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:01:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3775.988932 secs 2023-03-09T16:01:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:01:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3775.988932 secs 2023-03-09T16:01:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:01:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3775.988932 secs 2023-03-09T16:01:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:01:06 smithi122 ceph-mon[93869]: pgmap v2528: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 151 KiB/s wr, 22 op/s 2023-03-09T16:01:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:01:06 smithi181 ceph-mon[94189]: pgmap v2528: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 151 KiB/s wr, 22 op/s 2023-03-09T16:01:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:01:06 smithi038 ceph-mon[92422]: pgmap v2528: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 151 KiB/s wr, 22 op/s 2023-03-09T16:01:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:01:08 smithi181 ceph-mon[94189]: pgmap v2529: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 79 KiB/s wr, 22 op/s 2023-03-09T16:01:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:01:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3778.206122 secs 2023-03-09T16:01:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:01:08 smithi038 ceph-mon[92422]: pgmap v2529: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 79 KiB/s wr, 22 op/s 2023-03-09T16:01:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:01:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3778.206122 secs 2023-03-09T16:01:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:01:08 smithi122 ceph-mon[93869]: pgmap v2529: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 79 KiB/s wr, 22 op/s 2023-03-09T16:01:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:01:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3778.206122 secs 2023-03-09T16:01:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4240.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:01:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:01:10 smithi038 ceph-mon[92422]: pgmap v2530: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 79 KiB/s wr, 21 op/s 2023-03-09T16:01:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:01:10 smithi038 ceph-mon[92422]: osdmap e372: 12 total, 12 up, 12 in 2023-03-09T16:01:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:01:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3780.988981 secs 2023-03-09T16:01:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:01:10 smithi122 ceph-mon[93869]: pgmap v2530: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 79 KiB/s wr, 21 op/s 2023-03-09T16:01:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:01:10 smithi122 ceph-mon[93869]: osdmap e372: 12 total, 12 up, 12 in 2023-03-09T16:01:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:01:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3780.988981 secs 2023-03-09T16:01:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:01:10 smithi181 ceph-mon[94189]: pgmap v2530: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 79 KiB/s wr, 21 op/s 2023-03-09T16:01:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:01:10 smithi181 ceph-mon[94189]: osdmap e372: 12 total, 12 up, 12 in 2023-03-09T16:01:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:01:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3780.988981 secs 2023-03-09T16:01:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:01:11 smithi038 ceph-mon[92422]: pgmap v2532: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 82 KiB/s wr, 24 op/s 2023-03-09T16:01:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:01:11 smithi122 ceph-mon[93869]: pgmap v2532: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 82 KiB/s wr, 24 op/s 2023-03-09T16:01:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:01:11 smithi181 ceph-mon[94189]: pgmap v2532: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 82 KiB/s wr, 24 op/s 2023-03-09T16:01:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:01:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3783.206236 secs 2023-03-09T16:01:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:01:13 smithi122 ceph-mon[93869]: pgmap v2533: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 55 KiB/s wr, 21 op/s 2023-03-09T16:01:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:01:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3783.206236 secs 2023-03-09T16:01:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:01:13 smithi181 ceph-mon[94189]: pgmap v2533: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 55 KiB/s wr, 21 op/s 2023-03-09T16:01:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:01:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3783.206236 secs 2023-03-09T16:01:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:01:13 smithi038 ceph-mon[92422]: pgmap v2533: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 55 KiB/s wr, 21 op/s 2023-03-09T16:01:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4245.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:01:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:01:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3785.989042 secs 2023-03-09T16:01:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:01:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3785.989042 secs 2023-03-09T16:01:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:01:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3785.989042 secs 2023-03-09T16:01:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:01:16 smithi122 ceph-mon[93869]: pgmap v2534: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T16:01:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:01:16 smithi181 ceph-mon[94189]: pgmap v2534: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T16:01:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:01:16 smithi038 ceph-mon[92422]: pgmap v2534: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T16:01:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:01:18 smithi122 ceph-mon[93869]: pgmap v2535: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2023-03-09T16:01:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:01:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3788.206361 secs 2023-03-09T16:01:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:01:18 smithi181 ceph-mon[94189]: pgmap v2535: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2023-03-09T16:01:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:01:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3788.206361 secs 2023-03-09T16:01:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:01:18 smithi038 ceph-mon[92422]: pgmap v2535: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2023-03-09T16:01:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:01:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3788.206361 secs 2023-03-09T16:01:18.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4250.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:01:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:01:20 smithi122 ceph-mon[93869]: pgmap v2536: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 716 B/s wr, 6 op/s 2023-03-09T16:01:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:01:20 smithi122 ceph-mon[93869]: osdmap e373: 12 total, 12 up, 12 in 2023-03-09T16:01:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:01:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3790.989104 secs 2023-03-09T16:01:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:01:20 smithi181 ceph-mon[94189]: pgmap v2536: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 716 B/s wr, 6 op/s 2023-03-09T16:01:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:01:20 smithi181 ceph-mon[94189]: osdmap e373: 12 total, 12 up, 12 in 2023-03-09T16:01:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:01:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3790.989104 secs 2023-03-09T16:01:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:01:20 smithi038 ceph-mon[92422]: pgmap v2536: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 716 B/s wr, 6 op/s 2023-03-09T16:01:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:01:20 smithi038 ceph-mon[92422]: osdmap e373: 12 total, 12 up, 12 in 2023-03-09T16:01:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:01:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3790.989104 secs 2023-03-09T16:01:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:01:22 smithi122 ceph-mon[93869]: pgmap v2538: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 716 B/s wr, 6 op/s 2023-03-09T16:01:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:01:22 smithi181 ceph-mon[94189]: pgmap v2538: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 716 B/s wr, 6 op/s 2023-03-09T16:01:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:01:22 smithi038 ceph-mon[92422]: pgmap v2538: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 716 B/s wr, 6 op/s 2023-03-09T16:01:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:01:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3793.206496 secs 2023-03-09T16:01:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:01:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3793.206496 secs 2023-03-09T16:01:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:01:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3793.206496 secs 2023-03-09T16:01:23.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4255.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:01:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:01:24 smithi122 ceph-mon[93869]: pgmap v2539: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2023-03-09T16:01:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:01:24 smithi181 ceph-mon[94189]: pgmap v2539: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2023-03-09T16:01:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:01:24 smithi038 ceph-mon[92422]: pgmap v2539: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2023-03-09T16:01:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:01:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3795.989175 secs 2023-03-09T16:01:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:01:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3795.989175 secs 2023-03-09T16:01:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:01:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3795.989175 secs 2023-03-09T16:01:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:01:26 smithi122 ceph-mon[93869]: pgmap v2540: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 54 KiB/s wr, 8 op/s 2023-03-09T16:01:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:01:26 smithi181 ceph-mon[94189]: pgmap v2540: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 54 KiB/s wr, 8 op/s 2023-03-09T16:01:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:01:26 smithi038 ceph-mon[92422]: pgmap v2540: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 54 KiB/s wr, 8 op/s 2023-03-09T16:01:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:01:28 smithi122 ceph-mon[93869]: pgmap v2541: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 54 KiB/s wr, 7 op/s 2023-03-09T16:01:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:01:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3798.206609 secs 2023-03-09T16:01:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:01:28 smithi181 ceph-mon[94189]: pgmap v2541: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 54 KiB/s wr, 7 op/s 2023-03-09T16:01:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:01:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3798.206609 secs 2023-03-09T16:01:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:01:28 smithi038 ceph-mon[92422]: pgmap v2541: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 54 KiB/s wr, 7 op/s 2023-03-09T16:01:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:01:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3798.206609 secs 2023-03-09T16:01:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4260.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:01:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:01:30 smithi122 ceph-mon[93869]: pgmap v2542: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 54 KiB/s wr, 7 op/s 2023-03-09T16:01:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:01:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3800.989237 secs 2023-03-09T16:01:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:01:30 smithi181 ceph-mon[94189]: pgmap v2542: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 54 KiB/s wr, 7 op/s 2023-03-09T16:01:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:01:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3800.989237 secs 2023-03-09T16:01:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:01:30 smithi038 ceph-mon[92422]: pgmap v2542: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 54 KiB/s wr, 7 op/s 2023-03-09T16:01:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:01:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3800.989237 secs 2023-03-09T16:01:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:01:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:01:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:01:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:01:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:01:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:01:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:01:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:01:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:01:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:01:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:01:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:01:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:01:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:01:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:01:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:01:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:01:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:01:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:01:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:01:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:01:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:01:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:01:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:01:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:01:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:01:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:01:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:01:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:01:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:01:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:01:32 smithi122 ceph-mon[93869]: pgmap v2543: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 48 KiB/s wr, 6 op/s 2023-03-09T16:01:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:01:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:01:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:01:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:01:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:01:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:01:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:01:32 smithi181 ceph-mon[94189]: pgmap v2543: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 48 KiB/s wr, 6 op/s 2023-03-09T16:01:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:01:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:01:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:01:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:01:32.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:01:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:01:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:01:32 smithi038 ceph-mon[92422]: pgmap v2543: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 48 KiB/s wr, 6 op/s 2023-03-09T16:01:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:01:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:01:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:01:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:01:32.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:01:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:01:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:01:33 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 3803.206720 secs 2023-03-09T16:01:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:01:33 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 3803.206720 secs 2023-03-09T16:01:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:01:33 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 3803.206720 secs 2023-03-09T16:01:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4265.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:01:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:01:34 smithi122 ceph-mon[93869]: pgmap v2544: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 45 KiB/s wr, 8 op/s 2023-03-09T16:01:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:01:34 smithi181 ceph-mon[94189]: pgmap v2544: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 45 KiB/s wr, 8 op/s 2023-03-09T16:01:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:01:34 smithi038 ceph-mon[92422]: pgmap v2544: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 45 KiB/s wr, 8 op/s 2023-03-09T16:01:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:01:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3805.989296 secs 2023-03-09T16:01:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:01:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3805.989296 secs 2023-03-09T16:01:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:01:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3805.989296 secs 2023-03-09T16:01:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:01:36 smithi122 ceph-mon[93869]: pgmap v2545: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 72 KiB/s wr, 6 op/s 2023-03-09T16:01:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:01:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:01:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:01:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:01:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:01:36 smithi181 ceph-mon[94189]: pgmap v2545: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 72 KiB/s wr, 6 op/s 2023-03-09T16:01:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:01:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:01:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:01:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:01:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:01:36 smithi038 ceph-mon[92422]: pgmap v2545: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 72 KiB/s wr, 6 op/s 2023-03-09T16:01:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:01:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:01:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:01:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:01:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:01:38 smithi122 ceph-mon[93869]: pgmap v2546: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 30 KiB/s wr, 7 op/s 2023-03-09T16:01:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:01:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3808.206831 secs 2023-03-09T16:01:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:01:38 smithi181 ceph-mon[94189]: pgmap v2546: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 30 KiB/s wr, 7 op/s 2023-03-09T16:01:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:01:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3808.206831 secs 2023-03-09T16:01:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:01:38 smithi038 ceph-mon[92422]: pgmap v2546: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 30 KiB/s wr, 7 op/s 2023-03-09T16:01:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:01:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3808.206831 secs 2023-03-09T16:01:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4270.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:01:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:01:40 smithi122 ceph-mon[93869]: pgmap v2547: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 110 KiB/s wr, 8 op/s 2023-03-09T16:01:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:01:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:01:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:01:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:01:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:01:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3810.989357 secs 2023-03-09T16:01:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:01:40 smithi181 ceph-mon[94189]: pgmap v2547: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 110 KiB/s wr, 8 op/s 2023-03-09T16:01:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:01:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:01:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:01:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:01:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:01:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3810.989357 secs 2023-03-09T16:01:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:01:40 smithi038 ceph-mon[92422]: pgmap v2547: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 110 KiB/s wr, 8 op/s 2023-03-09T16:01:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:01:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:01:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:01:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:01:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:01:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3810.989357 secs 2023-03-09T16:01:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:01:42 smithi122 ceph-mon[93869]: pgmap v2548: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 172 KiB/s wr, 8 op/s 2023-03-09T16:01:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:01:42 smithi181 ceph-mon[94189]: pgmap v2548: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 172 KiB/s wr, 8 op/s 2023-03-09T16:01:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:01:42 smithi038 ceph-mon[92422]: pgmap v2548: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 172 KiB/s wr, 8 op/s 2023-03-09T16:01:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:01:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3813.206966 secs 2023-03-09T16:01:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:01:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3813.206966 secs 2023-03-09T16:01:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:01:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3813.206966 secs 2023-03-09T16:01:43.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4275.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:01:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:01:44 smithi122 ceph-mon[93869]: pgmap v2549: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 177 KiB/s wr, 11 op/s 2023-03-09T16:01:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:01:44 smithi181 ceph-mon[94189]: pgmap v2549: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 177 KiB/s wr, 11 op/s 2023-03-09T16:01:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:01:44 smithi038 ceph-mon[92422]: pgmap v2549: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 177 KiB/s wr, 11 op/s 2023-03-09T16:01:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:01:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3815.989415 secs 2023-03-09T16:01:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:01:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3815.989415 secs 2023-03-09T16:01:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:01:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3815.989415 secs 2023-03-09T16:01:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:01:46 smithi122 ceph-mon[93869]: pgmap v2550: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 232 KiB/s wr, 11 op/s 2023-03-09T16:01:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:01:46 smithi181 ceph-mon[94189]: pgmap v2550: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 232 KiB/s wr, 11 op/s 2023-03-09T16:01:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:01:46 smithi038 ceph-mon[92422]: pgmap v2550: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 232 KiB/s wr, 11 op/s 2023-03-09T16:01:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:01:48 smithi122 ceph-mon[93869]: pgmap v2551: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 205 KiB/s wr, 12 op/s 2023-03-09T16:01:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:01:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3818.207111 secs 2023-03-09T16:01:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:01:48 smithi181 ceph-mon[94189]: pgmap v2551: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 205 KiB/s wr, 12 op/s 2023-03-09T16:01:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:01:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3818.207111 secs 2023-03-09T16:01:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:01:48 smithi038 ceph-mon[92422]: pgmap v2551: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 205 KiB/s wr, 12 op/s 2023-03-09T16:01:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:01:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3818.207111 secs 2023-03-09T16:01:48.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4280.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:01:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:01:50 smithi122 ceph-mon[93869]: pgmap v2552: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 211 KiB/s wr, 12 op/s 2023-03-09T16:01:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:01:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3820.989500 secs 2023-03-09T16:01:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:01:50 smithi181 ceph-mon[94189]: pgmap v2552: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 211 KiB/s wr, 12 op/s 2023-03-09T16:01:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:01:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3820.989500 secs 2023-03-09T16:01:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:01:50 smithi038 ceph-mon[92422]: pgmap v2552: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 211 KiB/s wr, 12 op/s 2023-03-09T16:01:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:01:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3820.989500 secs 2023-03-09T16:01:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:01:52 smithi122 ceph-mon[93869]: pgmap v2553: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 179 KiB/s wr, 11 op/s 2023-03-09T16:01:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:01:52 smithi181 ceph-mon[94189]: pgmap v2553: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 179 KiB/s wr, 11 op/s 2023-03-09T16:01:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:01:52 smithi038 ceph-mon[92422]: pgmap v2553: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 179 KiB/s wr, 11 op/s 2023-03-09T16:01:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:01:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3823.207210 secs 2023-03-09T16:01:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:01:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3823.207210 secs 2023-03-09T16:01:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:01:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3823.207210 secs 2023-03-09T16:01:53.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4285.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:01:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:01:54 smithi122 ceph-mon[93869]: pgmap v2554: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 122 KiB/s wr, 13 op/s 2023-03-09T16:01:54.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:01:54 smithi181 ceph-mon[94189]: pgmap v2554: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 122 KiB/s wr, 13 op/s 2023-03-09T16:01:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:01:54 smithi038 ceph-mon[92422]: pgmap v2554: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 122 KiB/s wr, 13 op/s 2023-03-09T16:01:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:01:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3825.989559 secs 2023-03-09T16:01:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:01:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3825.989559 secs 2023-03-09T16:01:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:01:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3825.989559 secs 2023-03-09T16:01:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:01:56 smithi122 ceph-mon[93869]: pgmap v2555: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 158 KiB/s wr, 11 op/s 2023-03-09T16:01:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:01:56 smithi181 ceph-mon[94189]: pgmap v2555: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 158 KiB/s wr, 11 op/s 2023-03-09T16:01:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:01:56 smithi038 ceph-mon[92422]: pgmap v2555: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 158 KiB/s wr, 11 op/s 2023-03-09T16:01:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:01:58 smithi122 ceph-mon[93869]: pgmap v2556: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 103 KiB/s wr, 10 op/s 2023-03-09T16:01:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:01:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3828.207317 secs 2023-03-09T16:01:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:01:58 smithi181 ceph-mon[94189]: pgmap v2556: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 103 KiB/s wr, 10 op/s 2023-03-09T16:01:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:01:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3828.207317 secs 2023-03-09T16:01:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:01:58 smithi038 ceph-mon[92422]: pgmap v2556: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 103 KiB/s wr, 10 op/s 2023-03-09T16:01:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:01:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3828.207317 secs 2023-03-09T16:01:58.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4290.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:02:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:02:00 smithi122 ceph-mon[93869]: pgmap v2557: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 103 KiB/s wr, 9 op/s 2023-03-09T16:02:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:02:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3830.989620 secs 2023-03-09T16:02:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:02:00 smithi181 ceph-mon[94189]: pgmap v2557: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 103 KiB/s wr, 9 op/s 2023-03-09T16:02:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:02:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3830.989620 secs 2023-03-09T16:02:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:02:00 smithi038 ceph-mon[92422]: pgmap v2557: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 103 KiB/s wr, 9 op/s 2023-03-09T16:02:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:02:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3830.989620 secs 2023-03-09T16:02:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:02:02 smithi122 ceph-mon[93869]: pgmap v2558: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 119 KiB/s wr, 8 op/s 2023-03-09T16:02:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:02:02 smithi181 ceph-mon[94189]: pgmap v2558: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 119 KiB/s wr, 8 op/s 2023-03-09T16:02:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:02:02 smithi038 ceph-mon[92422]: pgmap v2558: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 119 KiB/s wr, 8 op/s 2023-03-09T16:02:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:02:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3833.207434 secs 2023-03-09T16:02:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:02:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3833.207434 secs 2023-03-09T16:02:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:02:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3833.207434 secs 2023-03-09T16:02:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4295.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:02:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:02:04 smithi122 ceph-mon[93869]: pgmap v2559: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 81 KiB/s wr, 11 op/s 2023-03-09T16:02:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:02:04 smithi181 ceph-mon[94189]: pgmap v2559: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 81 KiB/s wr, 11 op/s 2023-03-09T16:02:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:02:04 smithi038 ceph-mon[92422]: pgmap v2559: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 81 KiB/s wr, 11 op/s 2023-03-09T16:02:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:02:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3835.989681 secs 2023-03-09T16:02:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:02:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3835.989681 secs 2023-03-09T16:02:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:02:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3835.989681 secs 2023-03-09T16:02:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:02:06 smithi122 ceph-mon[93869]: pgmap v2560: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 100 KiB/s wr, 21 op/s 2023-03-09T16:02:06.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:02:06 smithi181 ceph-mon[94189]: pgmap v2560: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 100 KiB/s wr, 21 op/s 2023-03-09T16:02:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:02:06 smithi038 ceph-mon[92422]: pgmap v2560: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 100 KiB/s wr, 21 op/s 2023-03-09T16:02:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:02:08 smithi038 ceph-mon[92422]: pgmap v2561: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 69 KiB/s wr, 22 op/s 2023-03-09T16:02:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:02:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3838.207532 secs 2023-03-09T16:02:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:02:08 smithi122 ceph-mon[93869]: pgmap v2561: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 69 KiB/s wr, 22 op/s 2023-03-09T16:02:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:02:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3838.207532 secs 2023-03-09T16:02:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:02:08 smithi181 ceph-mon[94189]: pgmap v2561: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 69 KiB/s wr, 22 op/s 2023-03-09T16:02:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:02:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3838.207532 secs 2023-03-09T16:02:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4300.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:02:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:02:10 smithi038 ceph-mon[92422]: pgmap v2562: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 69 KiB/s wr, 21 op/s 2023-03-09T16:02:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:02:10 smithi038 ceph-mon[92422]: osdmap e374: 12 total, 12 up, 12 in 2023-03-09T16:02:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:02:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3840.989742 secs 2023-03-09T16:02:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:02:10 smithi122 ceph-mon[93869]: pgmap v2562: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 69 KiB/s wr, 21 op/s 2023-03-09T16:02:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:02:10 smithi122 ceph-mon[93869]: osdmap e374: 12 total, 12 up, 12 in 2023-03-09T16:02:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:02:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3840.989742 secs 2023-03-09T16:02:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:02:10 smithi181 ceph-mon[94189]: pgmap v2562: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 69 KiB/s wr, 21 op/s 2023-03-09T16:02:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:02:10 smithi181 ceph-mon[94189]: osdmap e374: 12 total, 12 up, 12 in 2023-03-09T16:02:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:02:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3840.989742 secs 2023-03-09T16:02:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:02:12 smithi038 ceph-mon[92422]: pgmap v2564: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 74 KiB/s wr, 24 op/s 2023-03-09T16:02:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:02:12 smithi122 ceph-mon[93869]: pgmap v2564: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 74 KiB/s wr, 24 op/s 2023-03-09T16:02:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:02:12 smithi181 ceph-mon[94189]: pgmap v2564: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 74 KiB/s wr, 24 op/s 2023-03-09T16:02:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:02:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3843.207651 secs 2023-03-09T16:02:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:02:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3843.207651 secs 2023-03-09T16:02:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:02:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3843.207651 secs 2023-03-09T16:02:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4305.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:02:14.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:02:14 smithi122 ceph-mon[93869]: pgmap v2565: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 61 KiB/s wr, 22 op/s 2023-03-09T16:02:14.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:02:14 smithi181 ceph-mon[94189]: pgmap v2565: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 61 KiB/s wr, 22 op/s 2023-03-09T16:02:14.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:02:14 smithi038 ceph-mon[92422]: pgmap v2565: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 61 KiB/s wr, 22 op/s 2023-03-09T16:02:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:02:15 smithi122 ceph-mon[93869]: pgmap v2566: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 34 KiB/s wr, 7 op/s 2023-03-09T16:02:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:02:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3845.989805 secs 2023-03-09T16:02:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:02:15 smithi181 ceph-mon[94189]: pgmap v2566: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 34 KiB/s wr, 7 op/s 2023-03-09T16:02:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:02:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3845.989805 secs 2023-03-09T16:02:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:02:15 smithi038 ceph-mon[92422]: pgmap v2566: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 34 KiB/s wr, 7 op/s 2023-03-09T16:02:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:02:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3845.989805 secs 2023-03-09T16:02:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:02:17 smithi122 ceph-mon[93869]: pgmap v2567: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 6 op/s 2023-03-09T16:02:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:02:17 smithi181 ceph-mon[94189]: pgmap v2567: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 6 op/s 2023-03-09T16:02:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:02:17 smithi038 ceph-mon[92422]: pgmap v2567: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 6 op/s 2023-03-09T16:02:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:02:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3848.207763 secs 2023-03-09T16:02:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:02:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3848.207763 secs 2023-03-09T16:02:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:02:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3848.207763 secs 2023-03-09T16:02:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4310.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:02:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:02:20 smithi122 ceph-mon[93869]: pgmap v2568: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 21 KiB/s wr, 6 op/s 2023-03-09T16:02:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:02:20 smithi122 ceph-mon[93869]: osdmap e375: 12 total, 12 up, 12 in 2023-03-09T16:02:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:02:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3850.989869 secs 2023-03-09T16:02:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:02:20 smithi181 ceph-mon[94189]: pgmap v2568: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 21 KiB/s wr, 6 op/s 2023-03-09T16:02:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:02:20 smithi181 ceph-mon[94189]: osdmap e375: 12 total, 12 up, 12 in 2023-03-09T16:02:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:02:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3850.989869 secs 2023-03-09T16:02:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:02:20 smithi038 ceph-mon[92422]: pgmap v2568: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 21 KiB/s wr, 6 op/s 2023-03-09T16:02:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:02:20 smithi038 ceph-mon[92422]: osdmap e375: 12 total, 12 up, 12 in 2023-03-09T16:02:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:02:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3850.989869 secs 2023-03-09T16:02:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:02:22 smithi122 ceph-mon[93869]: pgmap v2570: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T16:02:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:02:22 smithi181 ceph-mon[94189]: pgmap v2570: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T16:02:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:02:22 smithi038 ceph-mon[92422]: pgmap v2570: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T16:02:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:02:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3853.207905 secs 2023-03-09T16:02:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:02:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3853.207905 secs 2023-03-09T16:02:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:02:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3853.207905 secs 2023-03-09T16:02:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4315.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:02:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:02:24 smithi122 ceph-mon[93869]: pgmap v2571: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 614 B/s wr, 6 op/s 2023-03-09T16:02:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:02:24 smithi181 ceph-mon[94189]: pgmap v2571: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 614 B/s wr, 6 op/s 2023-03-09T16:02:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:02:24 smithi038 ceph-mon[92422]: pgmap v2571: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 614 B/s wr, 6 op/s 2023-03-09T16:02:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:02:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3855.989931 secs 2023-03-09T16:02:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:02:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3855.989931 secs 2023-03-09T16:02:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:02:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3855.989931 secs 2023-03-09T16:02:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:02:26 smithi122 ceph-mon[93869]: pgmap v2572: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 7 op/s 2023-03-09T16:02:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:02:26 smithi181 ceph-mon[94189]: pgmap v2572: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 7 op/s 2023-03-09T16:02:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:02:26 smithi038 ceph-mon[92422]: pgmap v2572: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 7 op/s 2023-03-09T16:02:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:02:28 smithi122 ceph-mon[93869]: pgmap v2573: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 7 op/s 2023-03-09T16:02:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:02:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3858.208025 secs 2023-03-09T16:02:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:02:28 smithi181 ceph-mon[94189]: pgmap v2573: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 7 op/s 2023-03-09T16:02:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:02:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3858.208025 secs 2023-03-09T16:02:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:02:28 smithi038 ceph-mon[92422]: pgmap v2573: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 7 op/s 2023-03-09T16:02:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:02:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3858.208025 secs 2023-03-09T16:02:28.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4320.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:02:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:02:30 smithi122 ceph-mon[93869]: pgmap v2574: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 7 op/s 2023-03-09T16:02:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:02:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3860.989984 secs 2023-03-09T16:02:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:02:30 smithi181 ceph-mon[94189]: pgmap v2574: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 7 op/s 2023-03-09T16:02:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:02:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3860.989984 secs 2023-03-09T16:02:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:02:30 smithi038 ceph-mon[92422]: pgmap v2574: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 7 op/s 2023-03-09T16:02:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:02:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3860.989984 secs 2023-03-09T16:02:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:02:32 smithi122 ceph-mon[93869]: pgmap v2575: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 55 KiB/s wr, 7 op/s 2023-03-09T16:02:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:02:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:02:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:02:32 smithi181 ceph-mon[94189]: pgmap v2575: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 55 KiB/s wr, 7 op/s 2023-03-09T16:02:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:02:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:02:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:02:32 smithi038 ceph-mon[92422]: pgmap v2575: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 55 KiB/s wr, 7 op/s 2023-03-09T16:02:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:02:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:02:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:02:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3863.208121 secs 2023-03-09T16:02:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:02:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:02:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:02:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:02:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:02:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:02:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:02:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3863.208121 secs 2023-03-09T16:02:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:02:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:02:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:02:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:02:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:02:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:02:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:02:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3863.208121 secs 2023-03-09T16:02:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:02:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:02:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:02:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:02:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:02:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:02:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4325.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:02:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:02:34 smithi122 ceph-mon[93869]: pgmap v2576: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 56 KiB/s wr, 8 op/s 2023-03-09T16:02:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:02:34 smithi181 ceph-mon[94189]: pgmap v2576: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 56 KiB/s wr, 8 op/s 2023-03-09T16:02:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:02:34 smithi038 ceph-mon[92422]: pgmap v2576: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 56 KiB/s wr, 8 op/s 2023-03-09T16:02:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:02:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3865.990047 secs 2023-03-09T16:02:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:02:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3865.990047 secs 2023-03-09T16:02:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:02:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3865.990047 secs 2023-03-09T16:02:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:02:36 smithi122 ceph-mon[93869]: pgmap v2577: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 96 KiB/s wr, 7 op/s 2023-03-09T16:02:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:02:36 smithi181 ceph-mon[94189]: pgmap v2577: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 96 KiB/s wr, 7 op/s 2023-03-09T16:02:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:02:36 smithi038 ceph-mon[92422]: pgmap v2577: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 96 KiB/s wr, 7 op/s 2023-03-09T16:02:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:02:38 smithi122 ceph-mon[93869]: pgmap v2578: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 91 KiB/s wr, 9 op/s 2023-03-09T16:02:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:02:38 smithi181 ceph-mon[94189]: pgmap v2578: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 91 KiB/s wr, 9 op/s 2023-03-09T16:02:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:02:38 smithi038 ceph-mon[92422]: pgmap v2578: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 91 KiB/s wr, 9 op/s 2023-03-09T16:02:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4330.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:02:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:02:39 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:02:39.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:02:39 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:02:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:02:39 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:02:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:02:40 smithi122 ceph-mon[93869]: pgmap v2579: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 96 KiB/s wr, 9 op/s 2023-03-09T16:02:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:02:40 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:02:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:02:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:02:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:02:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:02:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:02:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3870.990105 secs 2023-03-09T16:02:40.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:02:40 smithi181 ceph-mon[94189]: pgmap v2579: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 96 KiB/s wr, 9 op/s 2023-03-09T16:02:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:02:40 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:02:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:02:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:02:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:02:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:02:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:02:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3870.990105 secs 2023-03-09T16:02:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:02:40 smithi038 ceph-mon[92422]: pgmap v2579: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 96 KiB/s wr, 9 op/s 2023-03-09T16:02:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:02:40 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:02:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:02:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:02:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:02:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:02:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:02:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3870.990105 secs 2023-03-09T16:02:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:02:42 smithi122 ceph-mon[93869]: pgmap v2580: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 95 KiB/s wr, 9 op/s 2023-03-09T16:02:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:02:42 smithi181 ceph-mon[94189]: pgmap v2580: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 95 KiB/s wr, 9 op/s 2023-03-09T16:02:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:02:42 smithi038 ceph-mon[92422]: pgmap v2580: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 95 KiB/s wr, 9 op/s 2023-03-09T16:02:43.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4335.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:02:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:02:44 smithi122 ceph-mon[93869]: pgmap v2581: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 82 KiB/s wr, 11 op/s 2023-03-09T16:02:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:02:44 smithi181 ceph-mon[94189]: pgmap v2581: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 82 KiB/s wr, 11 op/s 2023-03-09T16:02:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:02:44 smithi038 ceph-mon[92422]: pgmap v2581: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 82 KiB/s wr, 11 op/s 2023-03-09T16:02:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:02:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3875.990165 secs 2023-03-09T16:02:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:02:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3875.990165 secs 2023-03-09T16:02:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:02:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3875.990165 secs 2023-03-09T16:02:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:02:46 smithi122 ceph-mon[93869]: pgmap v2582: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 79 KiB/s wr, 9 op/s 2023-03-09T16:02:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:02:46 smithi181 ceph-mon[94189]: pgmap v2582: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 79 KiB/s wr, 9 op/s 2023-03-09T16:02:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:02:46 smithi038 ceph-mon[92422]: pgmap v2582: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 79 KiB/s wr, 9 op/s 2023-03-09T16:02:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:02:48 smithi122 ceph-mon[93869]: pgmap v2583: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 127 KiB/s wr, 11 op/s 2023-03-09T16:02:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:02:48 smithi181 ceph-mon[94189]: pgmap v2583: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 127 KiB/s wr, 11 op/s 2023-03-09T16:02:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:02:48 smithi038 ceph-mon[92422]: pgmap v2583: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 127 KiB/s wr, 11 op/s 2023-03-09T16:02:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4340.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:02:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:02:50 smithi122 ceph-mon[93869]: pgmap v2584: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 108 KiB/s wr, 10 op/s 2023-03-09T16:02:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:02:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3880.990225 secs 2023-03-09T16:02:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:02:50 smithi181 ceph-mon[94189]: pgmap v2584: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 108 KiB/s wr, 10 op/s 2023-03-09T16:02:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:02:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3880.990225 secs 2023-03-09T16:02:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:02:50 smithi038 ceph-mon[92422]: pgmap v2584: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 108 KiB/s wr, 10 op/s 2023-03-09T16:02:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:02:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3880.990225 secs 2023-03-09T16:02:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:02:52 smithi122 ceph-mon[93869]: pgmap v2585: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 105 KiB/s wr, 9 op/s 2023-03-09T16:02:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:02:52 smithi181 ceph-mon[94189]: pgmap v2585: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 105 KiB/s wr, 9 op/s 2023-03-09T16:02:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:02:52 smithi038 ceph-mon[92422]: pgmap v2585: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 105 KiB/s wr, 9 op/s 2023-03-09T16:02:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:02:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3883.208616 secs 2023-03-09T16:02:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:02:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3883.208616 secs 2023-03-09T16:02:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:02:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3883.208616 secs 2023-03-09T16:02:53.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4345.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:02:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:02:54 smithi122 ceph-mon[93869]: pgmap v2586: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 206 KiB/s wr, 13 op/s 2023-03-09T16:02:54.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:02:54 smithi181 ceph-mon[94189]: pgmap v2586: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 206 KiB/s wr, 13 op/s 2023-03-09T16:02:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:02:54 smithi038 ceph-mon[92422]: pgmap v2586: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 206 KiB/s wr, 13 op/s 2023-03-09T16:02:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:02:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3885.990283 secs 2023-03-09T16:02:55.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:02:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3885.990283 secs 2023-03-09T16:02:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:02:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3885.990283 secs 2023-03-09T16:02:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:02:56 smithi122 ceph-mon[93869]: pgmap v2587: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 191 KiB/s wr, 10 op/s 2023-03-09T16:02:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:02:56 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:02:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:02:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:02:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:02:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:02:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:02:56 smithi181 ceph-mon[94189]: pgmap v2587: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 191 KiB/s wr, 10 op/s 2023-03-09T16:02:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:02:56 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:02:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:02:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:02:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:02:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:02:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:02:56 smithi038 ceph-mon[92422]: pgmap v2587: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 191 KiB/s wr, 10 op/s 2023-03-09T16:02:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:02:56 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:02:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:02:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:02:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:02:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:02:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:02:58 smithi122 ceph-mon[93869]: pgmap v2588: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 228 KiB/s wr, 12 op/s 2023-03-09T16:02:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:02:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3888.208725 secs 2023-03-09T16:02:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:02:58 smithi181 ceph-mon[94189]: pgmap v2588: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 228 KiB/s wr, 12 op/s 2023-03-09T16:02:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:02:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3888.208725 secs 2023-03-09T16:02:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:02:58 smithi038 ceph-mon[92422]: pgmap v2588: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 228 KiB/s wr, 12 op/s 2023-03-09T16:02:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:02:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3888.208725 secs 2023-03-09T16:02:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4350.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:03:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:03:00 smithi122 ceph-mon[93869]: pgmap v2589: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 144 KiB/s wr, 10 op/s 2023-03-09T16:03:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:03:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3890.990345 secs 2023-03-09T16:03:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:03:00 smithi181 ceph-mon[94189]: pgmap v2589: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 144 KiB/s wr, 10 op/s 2023-03-09T16:03:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:03:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3890.990345 secs 2023-03-09T16:03:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:03:00 smithi038 ceph-mon[92422]: pgmap v2589: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 144 KiB/s wr, 10 op/s 2023-03-09T16:03:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:03:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3890.990345 secs 2023-03-09T16:03:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:03:02 smithi122 ceph-mon[93869]: pgmap v2590: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 143 KiB/s wr, 8 op/s 2023-03-09T16:03:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:03:02 smithi181 ceph-mon[94189]: pgmap v2590: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 143 KiB/s wr, 8 op/s 2023-03-09T16:03:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:03:02 smithi038 ceph-mon[92422]: pgmap v2590: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 143 KiB/s wr, 8 op/s 2023-03-09T16:03:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:03:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3893.208819 secs 2023-03-09T16:03:03.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:03:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3893.208819 secs 2023-03-09T16:03:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:03:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3893.208819 secs 2023-03-09T16:03:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4355.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:03:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:03:04 smithi122 ceph-mon[93869]: pgmap v2591: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 184 KiB/s wr, 11 op/s 2023-03-09T16:03:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:03:04 smithi181 ceph-mon[94189]: pgmap v2591: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 184 KiB/s wr, 11 op/s 2023-03-09T16:03:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:03:04 smithi038 ceph-mon[92422]: pgmap v2591: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 184 KiB/s wr, 11 op/s 2023-03-09T16:03:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:03:05 smithi122 ceph-mon[93869]: osdmap e376: 12 total, 12 up, 12 in 2023-03-09T16:03:05.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:03:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3895.990406 secs 2023-03-09T16:03:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:03:05 smithi181 ceph-mon[94189]: osdmap e376: 12 total, 12 up, 12 in 2023-03-09T16:03:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:03:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3895.990406 secs 2023-03-09T16:03:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:03:05 smithi038 ceph-mon[92422]: osdmap e376: 12 total, 12 up, 12 in 2023-03-09T16:03:05.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:03:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3895.990406 secs 2023-03-09T16:03:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:03:06 smithi122 ceph-mon[93869]: pgmap v2593: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 125 KiB/s wr, 22 op/s 2023-03-09T16:03:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:03:06 smithi181 ceph-mon[94189]: pgmap v2593: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 125 KiB/s wr, 22 op/s 2023-03-09T16:03:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:03:06 smithi038 ceph-mon[92422]: pgmap v2593: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 125 KiB/s wr, 22 op/s 2023-03-09T16:03:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:03:08 smithi122 ceph-mon[93869]: pgmap v2594: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 138 KiB/s wr, 25 op/s 2023-03-09T16:03:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:03:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3898.208963 secs 2023-03-09T16:03:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:03:08 smithi181 ceph-mon[94189]: pgmap v2594: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 138 KiB/s wr, 25 op/s 2023-03-09T16:03:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:03:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3898.208963 secs 2023-03-09T16:03:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:03:08 smithi038 ceph-mon[92422]: pgmap v2594: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 138 KiB/s wr, 25 op/s 2023-03-09T16:03:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:03:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3898.208963 secs 2023-03-09T16:03:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4360.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:03:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:03:10 smithi038 ceph-mon[92422]: pgmap v2595: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 134 KiB/s wr, 25 op/s 2023-03-09T16:03:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:03:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3900.990468 secs 2023-03-09T16:03:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:03:10 smithi122 ceph-mon[93869]: pgmap v2595: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 134 KiB/s wr, 25 op/s 2023-03-09T16:03:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:03:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3900.990468 secs 2023-03-09T16:03:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:03:10 smithi181 ceph-mon[94189]: pgmap v2595: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 134 KiB/s wr, 25 op/s 2023-03-09T16:03:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:03:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3900.990468 secs 2023-03-09T16:03:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:03:12 smithi038 ceph-mon[92422]: pgmap v2596: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 134 KiB/s wr, 25 op/s 2023-03-09T16:03:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:03:12 smithi122 ceph-mon[93869]: pgmap v2596: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 134 KiB/s wr, 25 op/s 2023-03-09T16:03:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:03:12 smithi181 ceph-mon[94189]: pgmap v2596: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 134 KiB/s wr, 25 op/s 2023-03-09T16:03:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:03:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3903.209055 secs 2023-03-09T16:03:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:03:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3903.209055 secs 2023-03-09T16:03:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:03:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3903.209055 secs 2023-03-09T16:03:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4365.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:03:14.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:03:14 smithi122 ceph-mon[93869]: pgmap v2597: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 112 KiB/s wr, 25 op/s 2023-03-09T16:03:14.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:03:14 smithi122 ceph-mon[93869]: osdmap e377: 12 total, 12 up, 12 in 2023-03-09T16:03:14.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:03:14 smithi181 ceph-mon[94189]: pgmap v2597: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 112 KiB/s wr, 25 op/s 2023-03-09T16:03:14.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:03:14 smithi181 ceph-mon[94189]: osdmap e377: 12 total, 12 up, 12 in 2023-03-09T16:03:14.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:03:14 smithi038 ceph-mon[92422]: pgmap v2597: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 112 KiB/s wr, 25 op/s 2023-03-09T16:03:14.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:03:14 smithi038 ceph-mon[92422]: osdmap e377: 12 total, 12 up, 12 in 2023-03-09T16:03:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:03:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3905.990530 secs 2023-03-09T16:03:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:03:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3905.990530 secs 2023-03-09T16:03:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:03:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3905.990530 secs 2023-03-09T16:03:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:03:16 smithi122 ceph-mon[93869]: pgmap v2599: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 86 KiB/s wr, 10 op/s 2023-03-09T16:03:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:03:16 smithi181 ceph-mon[94189]: pgmap v2599: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 86 KiB/s wr, 10 op/s 2023-03-09T16:03:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:03:16 smithi038 ceph-mon[92422]: pgmap v2599: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 86 KiB/s wr, 10 op/s 2023-03-09T16:03:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:03:17 smithi122 ceph-mon[93869]: pgmap v2600: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T16:03:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:03:17 smithi181 ceph-mon[94189]: pgmap v2600: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T16:03:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:03:17 smithi038 ceph-mon[92422]: pgmap v2600: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T16:03:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:03:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3908.209174 secs 2023-03-09T16:03:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:03:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3908.209174 secs 2023-03-09T16:03:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:03:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3908.209174 secs 2023-03-09T16:03:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4370.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:03:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:03:20 smithi122 ceph-mon[93869]: pgmap v2601: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T16:03:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:03:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3910.990611 secs 2023-03-09T16:03:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:03:20 smithi181 ceph-mon[94189]: pgmap v2601: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T16:03:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:03:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3910.990611 secs 2023-03-09T16:03:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:03:20 smithi038 ceph-mon[92422]: pgmap v2601: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T16:03:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:03:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3910.990611 secs 2023-03-09T16:03:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:03:22 smithi122 ceph-mon[93869]: pgmap v2602: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T16:03:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:03:22 smithi181 ceph-mon[94189]: pgmap v2602: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T16:03:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:03:22 smithi038 ceph-mon[92422]: pgmap v2602: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T16:03:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:03:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3913.209283 secs 2023-03-09T16:03:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:03:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3913.209283 secs 2023-03-09T16:03:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:03:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3913.209283 secs 2023-03-09T16:03:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4375.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:03:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:03:24 smithi122 ceph-mon[93869]: pgmap v2603: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 921 B/s wr, 7 op/s 2023-03-09T16:03:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:03:24 smithi181 ceph-mon[94189]: pgmap v2603: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 921 B/s wr, 7 op/s 2023-03-09T16:03:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:03:24 smithi038 ceph-mon[92422]: pgmap v2603: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 921 B/s wr, 7 op/s 2023-03-09T16:03:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:03:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3915.990671 secs 2023-03-09T16:03:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:03:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3915.990671 secs 2023-03-09T16:03:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:03:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3915.990671 secs 2023-03-09T16:03:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:03:26 smithi122 ceph-mon[93869]: pgmap v2604: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 26 KiB/s wr, 7 op/s 2023-03-09T16:03:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:03:26 smithi181 ceph-mon[94189]: pgmap v2604: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 26 KiB/s wr, 7 op/s 2023-03-09T16:03:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:03:26 smithi038 ceph-mon[92422]: pgmap v2604: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 26 KiB/s wr, 7 op/s 2023-03-09T16:03:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:03:28 smithi122 ceph-mon[93869]: pgmap v2605: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 25 KiB/s wr, 8 op/s 2023-03-09T16:03:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:03:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3918.209421 secs 2023-03-09T16:03:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:03:28 smithi181 ceph-mon[94189]: pgmap v2605: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 25 KiB/s wr, 8 op/s 2023-03-09T16:03:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:03:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3918.209421 secs 2023-03-09T16:03:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:03:28 smithi038 ceph-mon[92422]: pgmap v2605: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 25 KiB/s wr, 8 op/s 2023-03-09T16:03:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:03:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3918.209421 secs 2023-03-09T16:03:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4380.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:03:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:03:30 smithi122 ceph-mon[93869]: pgmap v2606: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 25 KiB/s wr, 7 op/s 2023-03-09T16:03:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:03:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3920.990732 secs 2023-03-09T16:03:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:03:30 smithi181 ceph-mon[94189]: pgmap v2606: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 25 KiB/s wr, 7 op/s 2023-03-09T16:03:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:03:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3920.990732 secs 2023-03-09T16:03:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:03:30 smithi038 ceph-mon[92422]: pgmap v2606: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 25 KiB/s wr, 7 op/s 2023-03-09T16:03:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:03:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3920.990732 secs 2023-03-09T16:03:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:03:32 smithi122 ceph-mon[93869]: pgmap v2607: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 6 op/s 2023-03-09T16:03:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:03:32 smithi181 ceph-mon[94189]: pgmap v2607: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 6 op/s 2023-03-09T16:03:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:03:32 smithi038 ceph-mon[92422]: pgmap v2607: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 6 op/s 2023-03-09T16:03:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:03:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:03:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:03:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:03:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:03:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:03:33.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:03:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:03:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:03:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:03:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:03:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:03:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:03:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:03:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:03:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:03:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:03:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:03:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:03:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:03:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:03:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:03:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:03:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:03:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4385.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:03:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:03:34 smithi122 ceph-mon[93869]: pgmap v2608: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 53 KiB/s wr, 9 op/s 2023-03-09T16:03:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:03:34 smithi181 ceph-mon[94189]: pgmap v2608: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 53 KiB/s wr, 9 op/s 2023-03-09T16:03:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:03:34 smithi038 ceph-mon[92422]: pgmap v2608: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 53 KiB/s wr, 9 op/s 2023-03-09T16:03:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:03:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3925.990790 secs 2023-03-09T16:03:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:03:35 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:03:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:03:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3925.990790 secs 2023-03-09T16:03:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:03:35 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:03:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:03:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3925.990790 secs 2023-03-09T16:03:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:03:35 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:03:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:03:36 smithi122 ceph-mon[93869]: pgmap v2609: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 113 KiB/s wr, 7 op/s 2023-03-09T16:03:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:03:36 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:03:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:03:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:03:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:03:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:03:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:03:36 smithi181 ceph-mon[94189]: pgmap v2609: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 113 KiB/s wr, 7 op/s 2023-03-09T16:03:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:03:36 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:03:36.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:03:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:03:36.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:03:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:03:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:03:36 smithi038 ceph-mon[92422]: pgmap v2609: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 113 KiB/s wr, 7 op/s 2023-03-09T16:03:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:03:36 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:03:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:03:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:03:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:03:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:03:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:03:38 smithi122 ceph-mon[93869]: pgmap v2610: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 91 KiB/s wr, 9 op/s 2023-03-09T16:03:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:03:38 smithi181 ceph-mon[94189]: pgmap v2610: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 91 KiB/s wr, 9 op/s 2023-03-09T16:03:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:03:38 smithi038 ceph-mon[92422]: pgmap v2610: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 91 KiB/s wr, 9 op/s 2023-03-09T16:03:38.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4390.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:03:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:03:40 smithi122 ceph-mon[93869]: pgmap v2611: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 93 KiB/s wr, 8 op/s 2023-03-09T16:03:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:03:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3930.990850 secs 2023-03-09T16:03:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:03:40 smithi181 ceph-mon[94189]: pgmap v2611: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 93 KiB/s wr, 8 op/s 2023-03-09T16:03:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:03:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3930.990850 secs 2023-03-09T16:03:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:03:40 smithi038 ceph-mon[92422]: pgmap v2611: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 93 KiB/s wr, 8 op/s 2023-03-09T16:03:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:03:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3930.990850 secs 2023-03-09T16:03:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:03:42 smithi122 ceph-mon[93869]: pgmap v2612: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 156 KiB/s wr, 8 op/s 2023-03-09T16:03:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:03:42 smithi181 ceph-mon[94189]: pgmap v2612: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 156 KiB/s wr, 8 op/s 2023-03-09T16:03:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:03:42 smithi038 ceph-mon[92422]: pgmap v2612: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 156 KiB/s wr, 8 op/s 2023-03-09T16:03:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:03:43 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 3933.209783 secs 2023-03-09T16:03:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:03:43 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 3933.209783 secs 2023-03-09T16:03:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:03:43 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 3933.209783 secs 2023-03-09T16:03:43.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4395.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:03:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:03:44 smithi122 ceph-mon[93869]: pgmap v2613: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 133 KiB/s wr, 11 op/s 2023-03-09T16:03:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:03:44 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:03:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:03:44 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:03:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:03:44 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:03:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:03:44 smithi181 ceph-mon[94189]: pgmap v2613: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 133 KiB/s wr, 11 op/s 2023-03-09T16:03:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:03:44 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:03:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:03:44 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:03:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:03:44 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:03:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:03:44 smithi038 ceph-mon[92422]: pgmap v2613: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 133 KiB/s wr, 11 op/s 2023-03-09T16:03:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:03:44 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:03:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:03:44 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:03:44.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:03:44 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:03:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:03:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3935.990910 secs 2023-03-09T16:03:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:03:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3935.990910 secs 2023-03-09T16:03:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:03:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3935.990910 secs 2023-03-09T16:03:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:03:46 smithi122 ceph-mon[93869]: pgmap v2614: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 190 KiB/s wr, 10 op/s 2023-03-09T16:03:46.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:03:46 smithi181 ceph-mon[94189]: pgmap v2614: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 190 KiB/s wr, 10 op/s 2023-03-09T16:03:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:03:46 smithi038 ceph-mon[92422]: pgmap v2614: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 190 KiB/s wr, 10 op/s 2023-03-09T16:03:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:03:48 smithi122 ceph-mon[93869]: pgmap v2615: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 130 KiB/s wr, 11 op/s 2023-03-09T16:03:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:03:48 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 3938.209860 secs 2023-03-09T16:03:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:03:48 smithi181 ceph-mon[94189]: pgmap v2615: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 130 KiB/s wr, 11 op/s 2023-03-09T16:03:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:03:48 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 3938.209860 secs 2023-03-09T16:03:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:03:48 smithi038 ceph-mon[92422]: pgmap v2615: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 130 KiB/s wr, 11 op/s 2023-03-09T16:03:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:03:48 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 3938.209860 secs 2023-03-09T16:03:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4400.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:03:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:03:50 smithi122 ceph-mon[93869]: pgmap v2616: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 138 KiB/s wr, 11 op/s 2023-03-09T16:03:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:03:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3940.990997 secs 2023-03-09T16:03:50.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:03:50 smithi181 ceph-mon[94189]: pgmap v2616: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 138 KiB/s wr, 11 op/s 2023-03-09T16:03:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:03:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3940.990997 secs 2023-03-09T16:03:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:03:50 smithi038 ceph-mon[92422]: pgmap v2616: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 138 KiB/s wr, 11 op/s 2023-03-09T16:03:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:03:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3940.990997 secs 2023-03-09T16:03:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:03:52 smithi122 ceph-mon[93869]: pgmap v2617: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 176 KiB/s wr, 11 op/s 2023-03-09T16:03:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:03:52 smithi181 ceph-mon[94189]: pgmap v2617: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 176 KiB/s wr, 11 op/s 2023-03-09T16:03:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:03:52 smithi038 ceph-mon[92422]: pgmap v2617: 129 pgs: 129 active+clean; 8.6 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 176 KiB/s wr, 11 op/s 2023-03-09T16:03:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:03:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3943.209955 secs 2023-03-09T16:03:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:03:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3943.209955 secs 2023-03-09T16:03:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:03:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3943.209955 secs 2023-03-09T16:03:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4405.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:03:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:03:54 smithi038 ceph-mon[92422]: pgmap v2618: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 117 KiB/s wr, 12 op/s 2023-03-09T16:03:54.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:03:54 smithi122 ceph-mon[93869]: pgmap v2618: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 117 KiB/s wr, 12 op/s 2023-03-09T16:03:54.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:03:54 smithi181 ceph-mon[94189]: pgmap v2618: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 117 KiB/s wr, 12 op/s 2023-03-09T16:03:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:03:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3945.991057 secs 2023-03-09T16:03:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:03:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3945.991057 secs 2023-03-09T16:03:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:03:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3945.991057 secs 2023-03-09T16:03:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:03:56 smithi122 ceph-mon[93869]: pgmap v2619: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 154 KiB/s wr, 10 op/s 2023-03-09T16:03:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:03:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:03:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:03:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:03:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:03:56 smithi181 ceph-mon[94189]: pgmap v2619: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 154 KiB/s wr, 10 op/s 2023-03-09T16:03:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:03:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:03:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:03:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:03:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:03:56 smithi038 ceph-mon[92422]: pgmap v2619: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 154 KiB/s wr, 10 op/s 2023-03-09T16:03:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:03:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:03:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:03:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:03:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:03:58 smithi122 ceph-mon[93869]: pgmap v2620: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 97 KiB/s wr, 10 op/s 2023-03-09T16:03:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:03:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3948.210076 secs 2023-03-09T16:03:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:03:58 smithi181 ceph-mon[94189]: pgmap v2620: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 97 KiB/s wr, 10 op/s 2023-03-09T16:03:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:03:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3948.210076 secs 2023-03-09T16:03:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:03:58 smithi038 ceph-mon[92422]: pgmap v2620: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 97 KiB/s wr, 10 op/s 2023-03-09T16:03:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:03:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3948.210076 secs 2023-03-09T16:03:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4410.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:04:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:04:00 smithi122 ceph-mon[93869]: pgmap v2621: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 97 KiB/s wr, 9 op/s 2023-03-09T16:04:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:04:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3950.991116 secs 2023-03-09T16:04:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:04:00 smithi181 ceph-mon[94189]: pgmap v2621: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 97 KiB/s wr, 9 op/s 2023-03-09T16:04:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:04:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3950.991116 secs 2023-03-09T16:04:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:04:00 smithi038 ceph-mon[92422]: pgmap v2621: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 97 KiB/s wr, 9 op/s 2023-03-09T16:04:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:04:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3950.991116 secs 2023-03-09T16:04:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:04:02 smithi122 ceph-mon[93869]: pgmap v2622: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 114 KiB/s wr, 8 op/s 2023-03-09T16:04:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:04:02 smithi181 ceph-mon[94189]: pgmap v2622: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 114 KiB/s wr, 8 op/s 2023-03-09T16:04:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:04:02 smithi038 ceph-mon[92422]: pgmap v2622: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 114 KiB/s wr, 8 op/s 2023-03-09T16:04:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:04:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3953.210179 secs 2023-03-09T16:04:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:04:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3953.210179 secs 2023-03-09T16:04:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:04:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3953.210179 secs 2023-03-09T16:04:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4415.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:04:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:04:04 smithi122 ceph-mon[93869]: pgmap v2623: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 72 KiB/s wr, 9 op/s 2023-03-09T16:04:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:04:04 smithi181 ceph-mon[94189]: pgmap v2623: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 72 KiB/s wr, 9 op/s 2023-03-09T16:04:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:04:04 smithi038 ceph-mon[92422]: pgmap v2623: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 72 KiB/s wr, 9 op/s 2023-03-09T16:04:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:04:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3955.991179 secs 2023-03-09T16:04:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:04:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3955.991179 secs 2023-03-09T16:04:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:04:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3955.991179 secs 2023-03-09T16:04:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:04:06 smithi122 ceph-mon[93869]: pgmap v2624: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 90 KiB/s wr, 19 op/s 2023-03-09T16:04:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:04:06 smithi181 ceph-mon[94189]: pgmap v2624: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 90 KiB/s wr, 19 op/s 2023-03-09T16:04:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:04:06 smithi038 ceph-mon[92422]: pgmap v2624: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 90 KiB/s wr, 19 op/s 2023-03-09T16:04:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:04:08 smithi122 ceph-mon[93869]: pgmap v2625: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 70 KiB/s wr, 22 op/s 2023-03-09T16:04:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:04:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3958.210260 secs 2023-03-09T16:04:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:04:08 smithi181 ceph-mon[94189]: pgmap v2625: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 70 KiB/s wr, 22 op/s 2023-03-09T16:04:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:04:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3958.210260 secs 2023-03-09T16:04:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:04:08 smithi038 ceph-mon[92422]: pgmap v2625: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 70 KiB/s wr, 22 op/s 2023-03-09T16:04:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:04:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3958.210260 secs 2023-03-09T16:04:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4420.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:04:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:04:10 smithi122 ceph-mon[93869]: pgmap v2626: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 70 KiB/s wr, 21 op/s 2023-03-09T16:04:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:04:10 smithi122 ceph-mon[93869]: osdmap e378: 12 total, 12 up, 12 in 2023-03-09T16:04:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:04:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3960.991241 secs 2023-03-09T16:04:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:04:10 smithi181 ceph-mon[94189]: pgmap v2626: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 70 KiB/s wr, 21 op/s 2023-03-09T16:04:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:04:10 smithi181 ceph-mon[94189]: osdmap e378: 12 total, 12 up, 12 in 2023-03-09T16:04:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:04:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3960.991241 secs 2023-03-09T16:04:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:04:10 smithi038 ceph-mon[92422]: pgmap v2626: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 70 KiB/s wr, 21 op/s 2023-03-09T16:04:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:04:10 smithi038 ceph-mon[92422]: osdmap e378: 12 total, 12 up, 12 in 2023-03-09T16:04:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:04:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3960.991241 secs 2023-03-09T16:04:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:04:12 smithi122 ceph-mon[93869]: pgmap v2628: 129 pgs: 9 active+clean+snaptrim_wait, 120 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 54 KiB/s wr, 24 op/s 2023-03-09T16:04:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:04:12 smithi181 ceph-mon[94189]: pgmap v2628: 129 pgs: 9 active+clean+snaptrim_wait, 120 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 54 KiB/s wr, 24 op/s 2023-03-09T16:04:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:04:12 smithi038 ceph-mon[92422]: pgmap v2628: 129 pgs: 9 active+clean+snaptrim_wait, 120 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 54 KiB/s wr, 24 op/s 2023-03-09T16:04:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:04:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3963.210344 secs 2023-03-09T16:04:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:04:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3963.210344 secs 2023-03-09T16:04:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:04:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3963.210344 secs 2023-03-09T16:04:13.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4425.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:04:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:04:14 smithi122 ceph-mon[93869]: pgmap v2629: 129 pgs: 9 active+clean+snaptrim_wait, 120 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 54 KiB/s wr, 23 op/s 2023-03-09T16:04:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:04:14 smithi181 ceph-mon[94189]: pgmap v2629: 129 pgs: 9 active+clean+snaptrim_wait, 120 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 54 KiB/s wr, 23 op/s 2023-03-09T16:04:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:04:14 smithi038 ceph-mon[92422]: pgmap v2629: 129 pgs: 9 active+clean+snaptrim_wait, 120 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 54 KiB/s wr, 23 op/s 2023-03-09T16:04:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:04:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3965.991302 secs 2023-03-09T16:04:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:04:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3965.991302 secs 2023-03-09T16:04:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:04:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3965.991302 secs 2023-03-09T16:04:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:04:16 smithi122 ceph-mon[93869]: pgmap v2630: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 27 KiB/s wr, 8 op/s 2023-03-09T16:04:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:04:16 smithi181 ceph-mon[94189]: pgmap v2630: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 27 KiB/s wr, 8 op/s 2023-03-09T16:04:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:04:16 smithi038 ceph-mon[92422]: pgmap v2630: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 27 KiB/s wr, 8 op/s 2023-03-09T16:04:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:04:18 smithi122 ceph-mon[93869]: pgmap v2631: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T16:04:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:04:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3968.210435 secs 2023-03-09T16:04:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:04:18 smithi181 ceph-mon[94189]: pgmap v2631: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T16:04:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:04:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3968.210435 secs 2023-03-09T16:04:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:04:18 smithi038 ceph-mon[92422]: pgmap v2631: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T16:04:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:04:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3968.210435 secs 2023-03-09T16:04:18.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4430.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:04:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:04:20 smithi122 ceph-mon[93869]: pgmap v2632: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T16:04:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:04:20 smithi122 ceph-mon[93869]: osdmap e379: 12 total, 12 up, 12 in 2023-03-09T16:04:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:04:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3970.991359 secs 2023-03-09T16:04:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:04:20 smithi181 ceph-mon[94189]: pgmap v2632: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T16:04:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:04:20 smithi181 ceph-mon[94189]: osdmap e379: 12 total, 12 up, 12 in 2023-03-09T16:04:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:04:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3970.991359 secs 2023-03-09T16:04:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:04:20 smithi038 ceph-mon[92422]: pgmap v2632: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T16:04:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:04:20 smithi038 ceph-mon[92422]: osdmap e379: 12 total, 12 up, 12 in 2023-03-09T16:04:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:04:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3970.991359 secs 2023-03-09T16:04:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:04:22 smithi122 ceph-mon[93869]: pgmap v2634: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 3.9 KiB/s wr, 6 op/s 2023-03-09T16:04:22.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:04:22 smithi181 ceph-mon[94189]: pgmap v2634: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 3.9 KiB/s wr, 6 op/s 2023-03-09T16:04:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:04:22 smithi038 ceph-mon[92422]: pgmap v2634: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 3.9 KiB/s wr, 6 op/s 2023-03-09T16:04:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:04:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3973.210557 secs 2023-03-09T16:04:23.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:04:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3973.210557 secs 2023-03-09T16:04:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:04:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3973.210557 secs 2023-03-09T16:04:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4435.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:04:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:04:24 smithi122 ceph-mon[93869]: pgmap v2635: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 26 KiB/s wr, 7 op/s 2023-03-09T16:04:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:04:24 smithi181 ceph-mon[94189]: pgmap v2635: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 26 KiB/s wr, 7 op/s 2023-03-09T16:04:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:04:24 smithi038 ceph-mon[92422]: pgmap v2635: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 26 KiB/s wr, 7 op/s 2023-03-09T16:04:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:04:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3975.991420 secs 2023-03-09T16:04:25.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:04:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3975.991420 secs 2023-03-09T16:04:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:04:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3975.991420 secs 2023-03-09T16:04:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:04:26 smithi122 ceph-mon[93869]: pgmap v2636: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 26 KiB/s wr, 8 op/s 2023-03-09T16:04:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:04:26 smithi181 ceph-mon[94189]: pgmap v2636: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 26 KiB/s wr, 8 op/s 2023-03-09T16:04:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:04:26 smithi038 ceph-mon[92422]: pgmap v2636: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 26 KiB/s wr, 8 op/s 2023-03-09T16:04:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:04:28 smithi122 ceph-mon[93869]: pgmap v2637: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 26 KiB/s wr, 8 op/s 2023-03-09T16:04:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:04:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3978.210679 secs 2023-03-09T16:04:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:04:28 smithi181 ceph-mon[94189]: pgmap v2637: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 26 KiB/s wr, 8 op/s 2023-03-09T16:04:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:04:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3978.210679 secs 2023-03-09T16:04:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:04:28 smithi038 ceph-mon[92422]: pgmap v2637: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 26 KiB/s wr, 8 op/s 2023-03-09T16:04:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:04:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3978.210679 secs 2023-03-09T16:04:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4440.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:04:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:04:30 smithi122 ceph-mon[93869]: pgmap v2638: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 84 KiB/s wr, 9 op/s 2023-03-09T16:04:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:04:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3980.991480 secs 2023-03-09T16:04:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:04:30 smithi181 ceph-mon[94189]: pgmap v2638: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 84 KiB/s wr, 9 op/s 2023-03-09T16:04:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:04:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3980.991480 secs 2023-03-09T16:04:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:04:30 smithi038 ceph-mon[92422]: pgmap v2638: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 84 KiB/s wr, 9 op/s 2023-03-09T16:04:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:04:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3980.991480 secs 2023-03-09T16:04:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:04:32 smithi122 ceph-mon[93869]: pgmap v2639: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 74 KiB/s wr, 8 op/s 2023-03-09T16:04:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:04:32 smithi181 ceph-mon[94189]: pgmap v2639: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 74 KiB/s wr, 8 op/s 2023-03-09T16:04:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:04:32 smithi038 ceph-mon[92422]: pgmap v2639: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 74 KiB/s wr, 8 op/s 2023-03-09T16:04:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:04:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3983.210797 secs 2023-03-09T16:04:33.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:04:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3983.210797 secs 2023-03-09T16:04:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:04:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3983.210797 secs 2023-03-09T16:04:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4445.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:04:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:04:34 smithi122 ceph-mon[93869]: pgmap v2640: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 74 KiB/s wr, 10 op/s 2023-03-09T16:04:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:04:34 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:04:34.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:04:34 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:04:34.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:04:34 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:04:34.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:04:34 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:04:34.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:04:34 smithi181 ceph-mon[94189]: pgmap v2640: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 74 KiB/s wr, 10 op/s 2023-03-09T16:04:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:04:34 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:04:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:04:34 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:04:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:04:34 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:04:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:04:34 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:04:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:04:34 smithi038 ceph-mon[92422]: pgmap v2640: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 74 KiB/s wr, 10 op/s 2023-03-09T16:04:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:04:34 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:04:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:04:34 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:04:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:04:34 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:04:34.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:04:34 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:04:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:04:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3985.991539 secs 2023-03-09T16:04:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:04:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3985.991539 secs 2023-03-09T16:04:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:04:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3985.991539 secs 2023-03-09T16:04:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:04:36 smithi122 ceph-mon[93869]: pgmap v2641: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 56 KiB/s wr, 8 op/s 2023-03-09T16:04:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:04:36 smithi181 ceph-mon[94189]: pgmap v2641: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 56 KiB/s wr, 8 op/s 2023-03-09T16:04:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:04:36 smithi038 ceph-mon[92422]: pgmap v2641: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 56 KiB/s wr, 8 op/s 2023-03-09T16:04:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:04:38 smithi122 ceph-mon[93869]: pgmap v2642: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 56 KiB/s wr, 9 op/s 2023-03-09T16:04:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:04:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3988.210912 secs 2023-03-09T16:04:38.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:04:38 smithi181 ceph-mon[94189]: pgmap v2642: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 56 KiB/s wr, 9 op/s 2023-03-09T16:04:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:04:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3988.210912 secs 2023-03-09T16:04:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:04:38 smithi038 ceph-mon[92422]: pgmap v2642: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 56 KiB/s wr, 9 op/s 2023-03-09T16:04:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:04:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3988.210912 secs 2023-03-09T16:04:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4450.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:04:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:04:40 smithi122 ceph-mon[93869]: pgmap v2643: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 142 KiB/s wr, 9 op/s 2023-03-09T16:04:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:04:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3990.991608 secs 2023-03-09T16:04:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:04:40 smithi181 ceph-mon[94189]: pgmap v2643: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 142 KiB/s wr, 9 op/s 2023-03-09T16:04:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:04:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3990.991608 secs 2023-03-09T16:04:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:04:40 smithi038 ceph-mon[92422]: pgmap v2643: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 142 KiB/s wr, 9 op/s 2023-03-09T16:04:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:04:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3990.991608 secs 2023-03-09T16:04:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:04:42 smithi122 ceph-mon[93869]: pgmap v2644: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 94 KiB/s wr, 8 op/s 2023-03-09T16:04:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:04:42 smithi181 ceph-mon[94189]: pgmap v2644: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 94 KiB/s wr, 8 op/s 2023-03-09T16:04:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:04:42 smithi038 ceph-mon[92422]: pgmap v2644: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 94 KiB/s wr, 8 op/s 2023-03-09T16:04:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:04:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3993.211016 secs 2023-03-09T16:04:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:04:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3993.211016 secs 2023-03-09T16:04:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:04:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3993.211016 secs 2023-03-09T16:04:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4455.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:04:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:04:44 smithi122 ceph-mon[93869]: pgmap v2645: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 200 KiB/s wr, 12 op/s 2023-03-09T16:04:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:04:44 smithi181 ceph-mon[94189]: pgmap v2645: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 200 KiB/s wr, 12 op/s 2023-03-09T16:04:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:04:44 smithi038 ceph-mon[92422]: pgmap v2645: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 200 KiB/s wr, 12 op/s 2023-03-09T16:04:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:04:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3995.991665 secs 2023-03-09T16:04:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:04:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3995.991665 secs 2023-03-09T16:04:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:04:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3995.991665 secs 2023-03-09T16:04:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:04:46 smithi122 ceph-mon[93869]: pgmap v2646: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 192 KiB/s wr, 10 op/s 2023-03-09T16:04:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:04:46 smithi181 ceph-mon[94189]: pgmap v2646: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 192 KiB/s wr, 10 op/s 2023-03-09T16:04:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:04:46 smithi038 ceph-mon[92422]: pgmap v2646: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 192 KiB/s wr, 10 op/s 2023-03-09T16:04:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:04:48 smithi122 ceph-mon[93869]: pgmap v2647: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 193 KiB/s wr, 11 op/s 2023-03-09T16:04:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:04:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 3998.211135 secs 2023-03-09T16:04:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:04:48 smithi181 ceph-mon[94189]: pgmap v2647: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 193 KiB/s wr, 11 op/s 2023-03-09T16:04:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:04:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 3998.211135 secs 2023-03-09T16:04:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:04:48 smithi038 ceph-mon[92422]: pgmap v2647: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 193 KiB/s wr, 11 op/s 2023-03-09T16:04:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:04:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 3998.211135 secs 2023-03-09T16:04:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4460.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:04:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:04:50 smithi122 ceph-mon[93869]: pgmap v2648: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 221 KiB/s wr, 12 op/s 2023-03-09T16:04:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:04:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4000.991739 secs 2023-03-09T16:04:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:04:50 smithi181 ceph-mon[94189]: pgmap v2648: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 221 KiB/s wr, 12 op/s 2023-03-09T16:04:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:04:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4000.991739 secs 2023-03-09T16:04:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:04:50 smithi038 ceph-mon[92422]: pgmap v2648: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 221 KiB/s wr, 12 op/s 2023-03-09T16:04:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:04:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4000.991739 secs 2023-03-09T16:04:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:04:52 smithi122 ceph-mon[93869]: pgmap v2649: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 135 KiB/s wr, 9 op/s 2023-03-09T16:04:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:04:52 smithi181 ceph-mon[94189]: pgmap v2649: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 135 KiB/s wr, 9 op/s 2023-03-09T16:04:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:04:52 smithi038 ceph-mon[92422]: pgmap v2649: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 135 KiB/s wr, 9 op/s 2023-03-09T16:04:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:04:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4003.211235 secs 2023-03-09T16:04:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:04:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4003.211235 secs 2023-03-09T16:04:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:04:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4003.211235 secs 2023-03-09T16:04:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4465.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:04:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:04:54 smithi038 ceph-mon[92422]: pgmap v2650: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 207 KiB/s wr, 13 op/s 2023-03-09T16:04:54.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:04:54 smithi122 ceph-mon[93869]: pgmap v2650: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 207 KiB/s wr, 13 op/s 2023-03-09T16:04:54.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:04:54 smithi181 ceph-mon[94189]: pgmap v2650: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 207 KiB/s wr, 13 op/s 2023-03-09T16:04:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:04:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4005.991800 secs 2023-03-09T16:04:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:04:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4005.991800 secs 2023-03-09T16:04:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:04:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4005.991800 secs 2023-03-09T16:04:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:04:56 smithi038 ceph-mon[92422]: pgmap v2651: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 101 KiB/s wr, 9 op/s 2023-03-09T16:04:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:04:56 smithi122 ceph-mon[93869]: pgmap v2651: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 101 KiB/s wr, 9 op/s 2023-03-09T16:04:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:04:56 smithi181 ceph-mon[94189]: pgmap v2651: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 101 KiB/s wr, 9 op/s 2023-03-09T16:04:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:04:58 smithi122 ceph-mon[93869]: pgmap v2652: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 127 KiB/s wr, 11 op/s 2023-03-09T16:04:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:04:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4008.211367 secs 2023-03-09T16:04:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:04:58 smithi181 ceph-mon[94189]: pgmap v2652: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 127 KiB/s wr, 11 op/s 2023-03-09T16:04:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:04:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4008.211367 secs 2023-03-09T16:04:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:04:58 smithi038 ceph-mon[92422]: pgmap v2652: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 127 KiB/s wr, 11 op/s 2023-03-09T16:04:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:04:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4008.211367 secs 2023-03-09T16:04:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4470.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:05:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:05:00 smithi122 ceph-mon[93869]: pgmap v2653: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 127 KiB/s wr, 10 op/s 2023-03-09T16:05:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:05:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4010.991879 secs 2023-03-09T16:05:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:05:00 smithi181 ceph-mon[94189]: pgmap v2653: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 127 KiB/s wr, 10 op/s 2023-03-09T16:05:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:05:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4010.991879 secs 2023-03-09T16:05:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:05:00 smithi038 ceph-mon[92422]: pgmap v2653: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 127 KiB/s wr, 10 op/s 2023-03-09T16:05:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:05:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4010.991879 secs 2023-03-09T16:05:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:05:02 smithi122 ceph-mon[93869]: pgmap v2654: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 99 KiB/s wr, 8 op/s 2023-03-09T16:05:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:05:02 smithi181 ceph-mon[94189]: pgmap v2654: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 99 KiB/s wr, 8 op/s 2023-03-09T16:05:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:05:02 smithi038 ceph-mon[92422]: pgmap v2654: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 99 KiB/s wr, 8 op/s 2023-03-09T16:05:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:05:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4013.211519 secs 2023-03-09T16:05:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:05:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4013.211519 secs 2023-03-09T16:05:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:05:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4013.211519 secs 2023-03-09T16:05:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4475.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:05:04.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:05:04 smithi122 ceph-mon[93869]: pgmap v2655: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 124 KiB/s wr, 11 op/s 2023-03-09T16:05:04.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:05:04 smithi181 ceph-mon[94189]: pgmap v2655: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 124 KiB/s wr, 11 op/s 2023-03-09T16:05:04.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:05:04 smithi038 ceph-mon[92422]: pgmap v2655: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 124 KiB/s wr, 11 op/s 2023-03-09T16:05:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:05:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4015.991984 secs 2023-03-09T16:05:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:05:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4015.991984 secs 2023-03-09T16:05:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:05:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4015.991984 secs 2023-03-09T16:05:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:05:06 smithi122 ceph-mon[93869]: pgmap v2656: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 63 KiB/s wr, 18 op/s 2023-03-09T16:05:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:05:06 smithi181 ceph-mon[94189]: pgmap v2656: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 63 KiB/s wr, 18 op/s 2023-03-09T16:05:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:05:06 smithi038 ceph-mon[92422]: pgmap v2656: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 63 KiB/s wr, 18 op/s 2023-03-09T16:05:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:05:07 smithi122 ceph-mon[93869]: pgmap v2657: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 75 KiB/s wr, 21 op/s 2023-03-09T16:05:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:05:07 smithi181 ceph-mon[94189]: pgmap v2657: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 75 KiB/s wr, 21 op/s 2023-03-09T16:05:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:05:07 smithi038 ceph-mon[92422]: pgmap v2657: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 75 KiB/s wr, 21 op/s 2023-03-09T16:05:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:05:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4018.211637 secs 2023-03-09T16:05:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:05:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4018.211637 secs 2023-03-09T16:05:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:05:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4018.211637 secs 2023-03-09T16:05:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4480.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:05:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:05:10 smithi122 ceph-mon[93869]: pgmap v2658: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 48 KiB/s wr, 19 op/s 2023-03-09T16:05:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:05:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4020.992046 secs 2023-03-09T16:05:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:05:10 smithi181 ceph-mon[94189]: pgmap v2658: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 48 KiB/s wr, 19 op/s 2023-03-09T16:05:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:05:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4020.992046 secs 2023-03-09T16:05:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:05:10 smithi038 ceph-mon[92422]: pgmap v2658: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 48 KiB/s wr, 19 op/s 2023-03-09T16:05:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:05:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4020.992046 secs 2023-03-09T16:05:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:05:12 smithi122 ceph-mon[93869]: pgmap v2659: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 48 KiB/s wr, 18 op/s 2023-03-09T16:05:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:05:12 smithi181 ceph-mon[94189]: pgmap v2659: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 48 KiB/s wr, 18 op/s 2023-03-09T16:05:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:05:12 smithi038 ceph-mon[92422]: pgmap v2659: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 48 KiB/s wr, 18 op/s 2023-03-09T16:05:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:05:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4023.211698 secs 2023-03-09T16:05:13.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:05:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4023.211698 secs 2023-03-09T16:05:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:05:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4023.211698 secs 2023-03-09T16:05:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4485.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:05:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:05:14 smithi122 ceph-mon[93869]: pgmap v2660: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 65 KiB/s wr, 21 op/s 2023-03-09T16:05:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:05:14 smithi181 ceph-mon[94189]: pgmap v2660: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 65 KiB/s wr, 21 op/s 2023-03-09T16:05:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:05:14 smithi038 ceph-mon[92422]: pgmap v2660: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 65 KiB/s wr, 21 op/s 2023-03-09T16:05:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:05:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4025.992107 secs 2023-03-09T16:05:15.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:05:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4025.992107 secs 2023-03-09T16:05:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:05:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4025.992107 secs 2023-03-09T16:05:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:05:16 smithi122 ceph-mon[93869]: pgmap v2661: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 40 KiB/s wr, 17 op/s 2023-03-09T16:05:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:05:16 smithi181 ceph-mon[94189]: pgmap v2661: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 40 KiB/s wr, 17 op/s 2023-03-09T16:05:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:05:16 smithi038 ceph-mon[92422]: pgmap v2661: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 40 KiB/s wr, 17 op/s 2023-03-09T16:05:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:05:18 smithi122 ceph-mon[93869]: pgmap v2662: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 29 KiB/s wr, 9 op/s 2023-03-09T16:05:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:05:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4028.211820 secs 2023-03-09T16:05:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:05:18 smithi181 ceph-mon[94189]: pgmap v2662: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 29 KiB/s wr, 9 op/s 2023-03-09T16:05:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:05:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4028.211820 secs 2023-03-09T16:05:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:05:18 smithi038 ceph-mon[92422]: pgmap v2662: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 29 KiB/s wr, 9 op/s 2023-03-09T16:05:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:05:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4028.211820 secs 2023-03-09T16:05:18.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4490.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:05:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:05:20 smithi122 ceph-mon[93869]: pgmap v2663: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 5 op/s 2023-03-09T16:05:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:05:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4030.992167 secs 2023-03-09T16:05:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:05:20 smithi181 ceph-mon[94189]: pgmap v2663: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 5 op/s 2023-03-09T16:05:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:05:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4030.992167 secs 2023-03-09T16:05:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:05:20 smithi038 ceph-mon[92422]: pgmap v2663: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 5 op/s 2023-03-09T16:05:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:05:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4030.992167 secs 2023-03-09T16:05:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:05:22 smithi122 ceph-mon[93869]: pgmap v2664: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 5 op/s 2023-03-09T16:05:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:05:22 smithi181 ceph-mon[94189]: pgmap v2664: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 5 op/s 2023-03-09T16:05:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:05:22 smithi038 ceph-mon[92422]: pgmap v2664: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 5 op/s 2023-03-09T16:05:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:05:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4033.211918 secs 2023-03-09T16:05:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:05:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4033.211918 secs 2023-03-09T16:05:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:05:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4033.211918 secs 2023-03-09T16:05:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4495.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:05:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:05:24 smithi122 ceph-mon[93869]: pgmap v2665: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 18 KiB/s wr, 8 op/s 2023-03-09T16:05:24.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:05:24 smithi181 ceph-mon[94189]: pgmap v2665: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 18 KiB/s wr, 8 op/s 2023-03-09T16:05:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:05:24 smithi038 ceph-mon[92422]: pgmap v2665: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 18 KiB/s wr, 8 op/s 2023-03-09T16:05:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:05:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4035.992227 secs 2023-03-09T16:05:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:05:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4035.992227 secs 2023-03-09T16:05:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:05:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4035.992227 secs 2023-03-09T16:05:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:05:26 smithi122 ceph-mon[93869]: pgmap v2666: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 341 B/s wr, 5 op/s 2023-03-09T16:05:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:05:26 smithi181 ceph-mon[94189]: pgmap v2666: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 341 B/s wr, 5 op/s 2023-03-09T16:05:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:05:26 smithi038 ceph-mon[92422]: pgmap v2666: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 341 B/s wr, 5 op/s 2023-03-09T16:05:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:05:28 smithi122 ceph-mon[93869]: pgmap v2667: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 7 op/s 2023-03-09T16:05:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:05:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4038.212050 secs 2023-03-09T16:05:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:05:28 smithi181 ceph-mon[94189]: pgmap v2667: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 7 op/s 2023-03-09T16:05:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:05:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4038.212050 secs 2023-03-09T16:05:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:05:28 smithi038 ceph-mon[92422]: pgmap v2667: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 7 op/s 2023-03-09T16:05:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:05:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4038.212050 secs 2023-03-09T16:05:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4500.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:05:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:05:30 smithi122 ceph-mon[93869]: pgmap v2668: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T16:05:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:05:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4040.992287 secs 2023-03-09T16:05:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:05:30 smithi181 ceph-mon[94189]: pgmap v2668: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T16:05:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:05:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4040.992287 secs 2023-03-09T16:05:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:05:30 smithi038 ceph-mon[92422]: pgmap v2668: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T16:05:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:05:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4040.992287 secs 2023-03-09T16:05:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:05:32 smithi122 ceph-mon[93869]: pgmap v2669: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T16:05:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:05:32 smithi181 ceph-mon[94189]: pgmap v2669: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T16:05:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:05:32 smithi038 ceph-mon[92422]: pgmap v2669: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T16:05:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:05:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4043.212182 secs 2023-03-09T16:05:33.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:05:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4043.212182 secs 2023-03-09T16:05:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:05:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4043.212182 secs 2023-03-09T16:05:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4505.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:05:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:05:34 smithi122 ceph-mon[93869]: pgmap v2670: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 54 KiB/s wr, 9 op/s 2023-03-09T16:05:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:05:34 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:05:34.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:05:34 smithi181 ceph-mon[94189]: pgmap v2670: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 54 KiB/s wr, 9 op/s 2023-03-09T16:05:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:05:34 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:05:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:05:34 smithi038 ceph-mon[92422]: pgmap v2670: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 54 KiB/s wr, 9 op/s 2023-03-09T16:05:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:05:34 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:05:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:05:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4045.992351 secs 2023-03-09T16:05:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:05:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:05:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:05:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:05:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:05:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:05:35.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:05:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4045.992351 secs 2023-03-09T16:05:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:05:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:05:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:05:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:05:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:05:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:05:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:05:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4045.992351 secs 2023-03-09T16:05:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:05:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:05:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:05:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:05:35.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:05:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:05:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:05:36 smithi122 ceph-mon[93869]: pgmap v2671: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 54 KiB/s wr, 6 op/s 2023-03-09T16:05:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:05:36 smithi181 ceph-mon[94189]: pgmap v2671: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 54 KiB/s wr, 6 op/s 2023-03-09T16:05:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:05:36 smithi038 ceph-mon[92422]: pgmap v2671: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 54 KiB/s wr, 6 op/s 2023-03-09T16:05:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:05:38 smithi038 ceph-mon[92422]: pgmap v2672: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 116 KiB/s wr, 9 op/s 2023-03-09T16:05:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:05:38 smithi122 ceph-mon[93869]: pgmap v2672: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 116 KiB/s wr, 9 op/s 2023-03-09T16:05:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:05:38 smithi181 ceph-mon[94189]: pgmap v2672: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 116 KiB/s wr, 9 op/s 2023-03-09T16:05:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4510.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:05:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:05:39 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:05:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:05:39 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:05:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:05:39 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:05:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:05:40 smithi038 ceph-mon[92422]: pgmap v2673: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 120 KiB/s wr, 8 op/s 2023-03-09T16:05:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:05:40 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:05:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:05:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:05:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:05:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:05:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:05:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4050.992411 secs 2023-03-09T16:05:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:05:40 smithi122 ceph-mon[93869]: pgmap v2673: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 120 KiB/s wr, 8 op/s 2023-03-09T16:05:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:05:40 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:05:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:05:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:05:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:05:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:05:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:05:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4050.992411 secs 2023-03-09T16:05:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:05:40 smithi181 ceph-mon[94189]: pgmap v2673: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 120 KiB/s wr, 8 op/s 2023-03-09T16:05:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:05:40 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:05:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:05:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:05:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:05:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:05:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:05:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4050.992411 secs 2023-03-09T16:05:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:05:42 smithi038 ceph-mon[92422]: pgmap v2674: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 121 KiB/s wr, 7 op/s 2023-03-09T16:05:42.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:05:42 smithi122 ceph-mon[93869]: pgmap v2674: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 121 KiB/s wr, 7 op/s 2023-03-09T16:05:42.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:05:42 smithi181 ceph-mon[94189]: pgmap v2674: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 121 KiB/s wr, 7 op/s 2023-03-09T16:05:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4515.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:05:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:05:44 smithi038 ceph-mon[92422]: pgmap v2675: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 191 KiB/s wr, 12 op/s 2023-03-09T16:05:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:05:44 smithi122 ceph-mon[93869]: pgmap v2675: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 191 KiB/s wr, 12 op/s 2023-03-09T16:05:44.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:05:44 smithi181 ceph-mon[94189]: pgmap v2675: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 191 KiB/s wr, 12 op/s 2023-03-09T16:05:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:05:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4055.992471 secs 2023-03-09T16:05:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:05:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4055.992471 secs 2023-03-09T16:05:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:05:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4055.992471 secs 2023-03-09T16:05:46.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:05:46 smithi122 ceph-mon[93869]: pgmap v2676: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 137 KiB/s wr, 9 op/s 2023-03-09T16:05:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:05:46 smithi181 ceph-mon[94189]: pgmap v2676: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 137 KiB/s wr, 9 op/s 2023-03-09T16:05:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:05:46 smithi038 ceph-mon[92422]: pgmap v2676: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 137 KiB/s wr, 9 op/s 2023-03-09T16:05:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:05:48 smithi122 ceph-mon[93869]: pgmap v2677: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 195 KiB/s wr, 12 op/s 2023-03-09T16:05:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:05:48 smithi181 ceph-mon[94189]: pgmap v2677: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 195 KiB/s wr, 12 op/s 2023-03-09T16:05:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:05:48 smithi038 ceph-mon[92422]: pgmap v2677: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 195 KiB/s wr, 12 op/s 2023-03-09T16:05:48.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4520.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:05:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:05:50 smithi122 ceph-mon[93869]: pgmap v2678: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 141 KiB/s wr, 11 op/s 2023-03-09T16:05:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:05:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4060.992532 secs 2023-03-09T16:05:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:05:50 smithi181 ceph-mon[94189]: pgmap v2678: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 141 KiB/s wr, 11 op/s 2023-03-09T16:05:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:05:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4060.992532 secs 2023-03-09T16:05:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:05:50 smithi038 ceph-mon[92422]: pgmap v2678: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 141 KiB/s wr, 11 op/s 2023-03-09T16:05:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:05:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4060.992532 secs 2023-03-09T16:05:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:05:52 smithi122 ceph-mon[93869]: pgmap v2679: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 145 KiB/s wr, 10 op/s 2023-03-09T16:05:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:05:52 smithi181 ceph-mon[94189]: pgmap v2679: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 145 KiB/s wr, 10 op/s 2023-03-09T16:05:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:05:52 smithi038 ceph-mon[92422]: pgmap v2679: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 145 KiB/s wr, 10 op/s 2023-03-09T16:05:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:05:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4063.212606 secs 2023-03-09T16:05:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:05:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4063.212606 secs 2023-03-09T16:05:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:05:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4063.212606 secs 2023-03-09T16:05:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4525.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:05:54.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:05:54 smithi122 ceph-mon[93869]: pgmap v2680: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 184 KiB/s wr, 13 op/s 2023-03-09T16:05:54.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:05:54 smithi181 ceph-mon[94189]: pgmap v2680: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 184 KiB/s wr, 13 op/s 2023-03-09T16:05:54.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:05:54 smithi038 ceph-mon[92422]: pgmap v2680: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 184 KiB/s wr, 13 op/s 2023-03-09T16:05:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:05:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4065.992589 secs 2023-03-09T16:05:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:05:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4065.992589 secs 2023-03-09T16:05:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:05:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4065.992589 secs 2023-03-09T16:05:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:05:56 smithi122 ceph-mon[93869]: pgmap v2681: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 157 KiB/s wr, 10 op/s 2023-03-09T16:05:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:05:56 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:05:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:05:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:05:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:05:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:05:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:05:56 smithi181 ceph-mon[94189]: pgmap v2681: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 157 KiB/s wr, 10 op/s 2023-03-09T16:05:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:05:56 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:05:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:05:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:05:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:05:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:05:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:05:56 smithi038 ceph-mon[92422]: pgmap v2681: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 157 KiB/s wr, 10 op/s 2023-03-09T16:05:56.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:05:56 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:05:56.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:05:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:05:56.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:05:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:05:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:05:57 smithi122 ceph-mon[93869]: pgmap v2682: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 157 KiB/s wr, 12 op/s 2023-03-09T16:05:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:05:57 smithi181 ceph-mon[94189]: pgmap v2682: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 157 KiB/s wr, 12 op/s 2023-03-09T16:05:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:05:57 smithi038 ceph-mon[92422]: pgmap v2682: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 157 KiB/s wr, 12 op/s 2023-03-09T16:05:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:05:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4068.212745 secs 2023-03-09T16:05:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:05:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4068.212745 secs 2023-03-09T16:05:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:05:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4068.212745 secs 2023-03-09T16:05:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4530.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:06:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:06:00 smithi122 ceph-mon[93869]: pgmap v2683: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 100 KiB/s wr, 9 op/s 2023-03-09T16:06:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:06:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4070.992650 secs 2023-03-09T16:06:00.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:06:00 smithi181 ceph-mon[94189]: pgmap v2683: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 100 KiB/s wr, 9 op/s 2023-03-09T16:06:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:06:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4070.992650 secs 2023-03-09T16:06:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:06:00 smithi038 ceph-mon[92422]: pgmap v2683: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 100 KiB/s wr, 9 op/s 2023-03-09T16:06:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:06:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4070.992650 secs 2023-03-09T16:06:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:06:02 smithi122 ceph-mon[93869]: pgmap v2684: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 117 KiB/s wr, 8 op/s 2023-03-09T16:06:02.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:06:02 smithi181 ceph-mon[94189]: pgmap v2684: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 117 KiB/s wr, 8 op/s 2023-03-09T16:06:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:06:02 smithi038 ceph-mon[92422]: pgmap v2684: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 117 KiB/s wr, 8 op/s 2023-03-09T16:06:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:06:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4073.212863 secs 2023-03-09T16:06:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:06:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4073.212863 secs 2023-03-09T16:06:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:06:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4073.212863 secs 2023-03-09T16:06:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4535.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:06:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:06:04 smithi122 ceph-mon[93869]: pgmap v2685: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 109 KiB/s wr, 11 op/s 2023-03-09T16:06:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:06:04 smithi181 ceph-mon[94189]: pgmap v2685: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 109 KiB/s wr, 11 op/s 2023-03-09T16:06:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:06:04 smithi038 ceph-mon[92422]: pgmap v2685: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 109 KiB/s wr, 11 op/s 2023-03-09T16:06:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:06:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4075.992712 secs 2023-03-09T16:06:05.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:06:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4075.992712 secs 2023-03-09T16:06:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:06:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4075.992712 secs 2023-03-09T16:06:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:06:06 smithi122 ceph-mon[93869]: pgmap v2686: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 92 KiB/s wr, 20 op/s 2023-03-09T16:06:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:06:06 smithi181 ceph-mon[94189]: pgmap v2686: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 92 KiB/s wr, 20 op/s 2023-03-09T16:06:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:06:06 smithi038 ceph-mon[92422]: pgmap v2686: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 92 KiB/s wr, 20 op/s 2023-03-09T16:06:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:06:08 smithi122 ceph-mon[93869]: pgmap v2687: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 71 KiB/s wr, 23 op/s 2023-03-09T16:06:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:06:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4078.212929 secs 2023-03-09T16:06:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:06:08 smithi181 ceph-mon[94189]: pgmap v2687: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 71 KiB/s wr, 23 op/s 2023-03-09T16:06:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:06:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4078.212929 secs 2023-03-09T16:06:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:06:08 smithi038 ceph-mon[92422]: pgmap v2687: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 71 KiB/s wr, 23 op/s 2023-03-09T16:06:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:06:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4078.212929 secs 2023-03-09T16:06:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4540.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:06:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:06:10 smithi122 ceph-mon[93869]: pgmap v2688: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 71 KiB/s wr, 21 op/s 2023-03-09T16:06:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:06:10 smithi122 ceph-mon[93869]: osdmap e380: 12 total, 12 up, 12 in 2023-03-09T16:06:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:06:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4080.992772 secs 2023-03-09T16:06:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:06:10 smithi181 ceph-mon[94189]: pgmap v2688: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 71 KiB/s wr, 21 op/s 2023-03-09T16:06:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:06:10 smithi181 ceph-mon[94189]: osdmap e380: 12 total, 12 up, 12 in 2023-03-09T16:06:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:06:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4080.992772 secs 2023-03-09T16:06:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:06:10 smithi038 ceph-mon[92422]: pgmap v2688: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 71 KiB/s wr, 21 op/s 2023-03-09T16:06:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:06:10 smithi038 ceph-mon[92422]: osdmap e380: 12 total, 12 up, 12 in 2023-03-09T16:06:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:06:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4080.992772 secs 2023-03-09T16:06:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:06:12 smithi122 ceph-mon[93869]: pgmap v2690: 129 pgs: 12 active+clean+snaptrim_wait, 117 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 54 KiB/s wr, 24 op/s 2023-03-09T16:06:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:06:12 smithi181 ceph-mon[94189]: pgmap v2690: 129 pgs: 12 active+clean+snaptrim_wait, 117 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 54 KiB/s wr, 24 op/s 2023-03-09T16:06:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:06:12 smithi038 ceph-mon[92422]: pgmap v2690: 129 pgs: 12 active+clean+snaptrim_wait, 117 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 54 KiB/s wr, 24 op/s 2023-03-09T16:06:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:06:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4083.213076 secs 2023-03-09T16:06:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:06:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4083.213076 secs 2023-03-09T16:06:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:06:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4083.213076 secs 2023-03-09T16:06:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4545.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:06:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:06:14 smithi122 ceph-mon[93869]: pgmap v2691: 129 pgs: 12 active+clean+snaptrim_wait, 117 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 54 KiB/s wr, 23 op/s 2023-03-09T16:06:14.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:06:14 smithi181 ceph-mon[94189]: pgmap v2691: 129 pgs: 12 active+clean+snaptrim_wait, 117 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 54 KiB/s wr, 23 op/s 2023-03-09T16:06:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:06:14 smithi038 ceph-mon[92422]: pgmap v2691: 129 pgs: 12 active+clean+snaptrim_wait, 117 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 54 KiB/s wr, 23 op/s 2023-03-09T16:06:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:06:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4085.992833 secs 2023-03-09T16:06:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:06:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4085.992833 secs 2023-03-09T16:06:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:06:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4085.992833 secs 2023-03-09T16:06:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:06:16 smithi122 ceph-mon[93869]: pgmap v2692: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 27 KiB/s wr, 8 op/s 2023-03-09T16:06:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:06:16 smithi181 ceph-mon[94189]: pgmap v2692: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 27 KiB/s wr, 8 op/s 2023-03-09T16:06:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:06:16 smithi038 ceph-mon[92422]: pgmap v2692: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 27 KiB/s wr, 8 op/s 2023-03-09T16:06:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:06:18 smithi122 ceph-mon[93869]: pgmap v2693: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T16:06:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:06:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4088.213194 secs 2023-03-09T16:06:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:06:18 smithi181 ceph-mon[94189]: pgmap v2693: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T16:06:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:06:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4088.213194 secs 2023-03-09T16:06:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:06:18 smithi038 ceph-mon[92422]: pgmap v2693: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T16:06:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:06:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4088.213194 secs 2023-03-09T16:06:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4550.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:06:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:06:20 smithi122 ceph-mon[93869]: pgmap v2694: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T16:06:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:06:20 smithi122 ceph-mon[93869]: osdmap e381: 12 total, 12 up, 12 in 2023-03-09T16:06:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:06:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4090.992892 secs 2023-03-09T16:06:20.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:06:20 smithi181 ceph-mon[94189]: pgmap v2694: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T16:06:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:06:20 smithi181 ceph-mon[94189]: osdmap e381: 12 total, 12 up, 12 in 2023-03-09T16:06:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:06:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4090.992892 secs 2023-03-09T16:06:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:06:20 smithi038 ceph-mon[92422]: pgmap v2694: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T16:06:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:06:20 smithi038 ceph-mon[92422]: osdmap e381: 12 total, 12 up, 12 in 2023-03-09T16:06:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:06:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4090.992892 secs 2023-03-09T16:06:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:06:22 smithi122 ceph-mon[93869]: pgmap v2696: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T16:06:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:06:22 smithi181 ceph-mon[94189]: pgmap v2696: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T16:06:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:06:22 smithi038 ceph-mon[92422]: pgmap v2696: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T16:06:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:06:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4093.213276 secs 2023-03-09T16:06:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:06:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4093.213276 secs 2023-03-09T16:06:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:06:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4093.213276 secs 2023-03-09T16:06:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4555.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:06:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:06:24 smithi122 ceph-mon[93869]: pgmap v2697: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 716 B/s wr, 6 op/s 2023-03-09T16:06:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:06:24 smithi181 ceph-mon[94189]: pgmap v2697: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 716 B/s wr, 6 op/s 2023-03-09T16:06:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:06:24 smithi038 ceph-mon[92422]: pgmap v2697: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 716 B/s wr, 6 op/s 2023-03-09T16:06:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:06:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4095.992950 secs 2023-03-09T16:06:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:06:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4095.992950 secs 2023-03-09T16:06:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:06:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4095.992950 secs 2023-03-09T16:06:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:06:26 smithi122 ceph-mon[93869]: pgmap v2698: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 27 KiB/s wr, 6 op/s 2023-03-09T16:06:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:06:26 smithi181 ceph-mon[94189]: pgmap v2698: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 27 KiB/s wr, 6 op/s 2023-03-09T16:06:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:06:26 smithi038 ceph-mon[92422]: pgmap v2698: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 27 KiB/s wr, 6 op/s 2023-03-09T16:06:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:06:28 smithi122 ceph-mon[93869]: pgmap v2699: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 27 KiB/s wr, 6 op/s 2023-03-09T16:06:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:06:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4098.213393 secs 2023-03-09T16:06:28.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:06:28 smithi181 ceph-mon[94189]: pgmap v2699: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 27 KiB/s wr, 6 op/s 2023-03-09T16:06:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:06:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4098.213393 secs 2023-03-09T16:06:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:06:28 smithi038 ceph-mon[92422]: pgmap v2699: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 27 KiB/s wr, 6 op/s 2023-03-09T16:06:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:06:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4098.213393 secs 2023-03-09T16:06:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4560.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:06:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:06:30 smithi122 ceph-mon[93869]: pgmap v2700: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T16:06:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:06:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4100.992985 secs 2023-03-09T16:06:30.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:06:30 smithi181 ceph-mon[94189]: pgmap v2700: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T16:06:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:06:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4100.992985 secs 2023-03-09T16:06:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:06:30 smithi038 ceph-mon[92422]: pgmap v2700: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T16:06:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:06:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4100.992985 secs 2023-03-09T16:06:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:06:32 smithi122 ceph-mon[93869]: pgmap v2701: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 81 KiB/s wr, 7 op/s 2023-03-09T16:06:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:06:32 smithi181 ceph-mon[94189]: pgmap v2701: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 81 KiB/s wr, 7 op/s 2023-03-09T16:06:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:06:32 smithi038 ceph-mon[92422]: pgmap v2701: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 81 KiB/s wr, 7 op/s 2023-03-09T16:06:33.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4565.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:06:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:06:34 smithi122 ceph-mon[93869]: pgmap v2702: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 77 KiB/s wr, 8 op/s 2023-03-09T16:06:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:06:34 smithi181 ceph-mon[94189]: pgmap v2702: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 77 KiB/s wr, 8 op/s 2023-03-09T16:06:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:06:34 smithi038 ceph-mon[92422]: pgmap v2702: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 77 KiB/s wr, 8 op/s 2023-03-09T16:06:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:06:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4105.993011 secs 2023-03-09T16:06:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:06:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:06:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:06:35 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:06:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:06:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4105.993011 secs 2023-03-09T16:06:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:06:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:06:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:06:35 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:06:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:06:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4105.993011 secs 2023-03-09T16:06:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:06:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:06:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:06:35 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:06:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:06:36 smithi122 ceph-mon[93869]: pgmap v2703: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 117 KiB/s wr, 7 op/s 2023-03-09T16:06:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:06:36 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:06:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:06:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:06:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:06:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:06:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:06:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:06:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:06:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:06:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:06:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:06:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:06:36 smithi181 ceph-mon[94189]: pgmap v2703: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 117 KiB/s wr, 7 op/s 2023-03-09T16:06:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:06:36 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:06:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:06:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:06:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:06:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:06:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:06:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:06:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:06:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:06:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:06:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:06:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:06:36 smithi038 ceph-mon[92422]: pgmap v2703: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 117 KiB/s wr, 7 op/s 2023-03-09T16:06:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:06:36 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:06:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:06:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:06:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:06:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:06:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:06:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:06:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:06:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:06:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:06:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:06:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:06:38 smithi122 ceph-mon[93869]: pgmap v2704: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 95 KiB/s wr, 8 op/s 2023-03-09T16:06:38.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:06:38 smithi181 ceph-mon[94189]: pgmap v2704: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 95 KiB/s wr, 8 op/s 2023-03-09T16:06:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:06:38 smithi038 ceph-mon[92422]: pgmap v2704: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 95 KiB/s wr, 8 op/s 2023-03-09T16:06:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4570.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:06:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:06:40 smithi122 ceph-mon[93869]: pgmap v2705: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 100 KiB/s wr, 7 op/s 2023-03-09T16:06:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:06:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4110.993072 secs 2023-03-09T16:06:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:06:40 smithi181 ceph-mon[94189]: pgmap v2705: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 100 KiB/s wr, 7 op/s 2023-03-09T16:06:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:06:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4110.993072 secs 2023-03-09T16:06:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:06:40 smithi038 ceph-mon[92422]: pgmap v2705: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 100 KiB/s wr, 7 op/s 2023-03-09T16:06:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:06:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4110.993072 secs 2023-03-09T16:06:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:06:42 smithi122 ceph-mon[93869]: pgmap v2706: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 144 KiB/s wr, 8 op/s 2023-03-09T16:06:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:06:42 smithi181 ceph-mon[94189]: pgmap v2706: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 144 KiB/s wr, 8 op/s 2023-03-09T16:06:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:06:42 smithi038 ceph-mon[92422]: pgmap v2706: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 144 KiB/s wr, 8 op/s 2023-03-09T16:06:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:06:43 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 4113.213706 secs 2023-03-09T16:06:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:06:43 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 4113.213706 secs 2023-03-09T16:06:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:06:43 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 4113.213706 secs 2023-03-09T16:06:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4575.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:06:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:06:44 smithi038 ceph-mon[92422]: pgmap v2707: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 95 KiB/s wr, 11 op/s 2023-03-09T16:06:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:06:44 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:06:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:06:44 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:06:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:06:44 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:06:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:06:44 smithi122 ceph-mon[93869]: pgmap v2707: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 95 KiB/s wr, 11 op/s 2023-03-09T16:06:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:06:44 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:06:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:06:44 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:06:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:06:44 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:06:44.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:06:44 smithi181 ceph-mon[94189]: pgmap v2707: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 95 KiB/s wr, 11 op/s 2023-03-09T16:06:44.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:06:44 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:06:44.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:06:44 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:06:44.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:06:44 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:06:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:06:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4115.993134 secs 2023-03-09T16:06:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:06:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4115.993134 secs 2023-03-09T16:06:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:06:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4115.993134 secs 2023-03-09T16:06:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:06:46 smithi122 ceph-mon[93869]: pgmap v2708: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 161 KiB/s wr, 10 op/s 2023-03-09T16:06:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:06:46 smithi181 ceph-mon[94189]: pgmap v2708: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 161 KiB/s wr, 10 op/s 2023-03-09T16:06:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:06:46 smithi038 ceph-mon[92422]: pgmap v2708: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 161 KiB/s wr, 10 op/s 2023-03-09T16:06:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:06:48 smithi122 ceph-mon[93869]: pgmap v2709: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 121 KiB/s wr, 11 op/s 2023-03-09T16:06:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:06:48 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 4118.213824 secs 2023-03-09T16:06:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:06:48 smithi181 ceph-mon[94189]: pgmap v2709: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 121 KiB/s wr, 11 op/s 2023-03-09T16:06:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:06:48 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 4118.213824 secs 2023-03-09T16:06:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:06:48 smithi038 ceph-mon[92422]: pgmap v2709: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 121 KiB/s wr, 11 op/s 2023-03-09T16:06:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:06:48 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 4118.213824 secs 2023-03-09T16:06:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4580.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:06:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:06:50 smithi122 ceph-mon[93869]: pgmap v2710: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 129 KiB/s wr, 11 op/s 2023-03-09T16:06:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:06:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4120.993191 secs 2023-03-09T16:06:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:06:50 smithi181 ceph-mon[94189]: pgmap v2710: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 129 KiB/s wr, 11 op/s 2023-03-09T16:06:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:06:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4120.993191 secs 2023-03-09T16:06:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:06:50 smithi038 ceph-mon[92422]: pgmap v2710: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 129 KiB/s wr, 11 op/s 2023-03-09T16:06:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:06:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4120.993191 secs 2023-03-09T16:06:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:06:52 smithi122 ceph-mon[93869]: pgmap v2711: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 162 KiB/s wr, 11 op/s 2023-03-09T16:06:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:06:52 smithi181 ceph-mon[94189]: pgmap v2711: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 162 KiB/s wr, 11 op/s 2023-03-09T16:06:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:06:52 smithi038 ceph-mon[92422]: pgmap v2711: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 162 KiB/s wr, 11 op/s 2023-03-09T16:06:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:06:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4123.213953 secs 2023-03-09T16:06:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:06:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4123.213953 secs 2023-03-09T16:06:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:06:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4123.213953 secs 2023-03-09T16:06:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4585.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:06:54.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:06:54 smithi122 ceph-mon[93869]: pgmap v2712: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 122 KiB/s wr, 12 op/s 2023-03-09T16:06:54.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:06:54 smithi181 ceph-mon[94189]: pgmap v2712: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 122 KiB/s wr, 12 op/s 2023-03-09T16:06:54.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:06:54 smithi038 ceph-mon[92422]: pgmap v2712: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 122 KiB/s wr, 12 op/s 2023-03-09T16:06:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:06:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4125.993250 secs 2023-03-09T16:06:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:06:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4125.993250 secs 2023-03-09T16:06:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:06:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4125.993250 secs 2023-03-09T16:06:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:06:56 smithi122 ceph-mon[93869]: pgmap v2713: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 171 KiB/s wr, 10 op/s 2023-03-09T16:06:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:06:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:06:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:06:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:06:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:06:56 smithi181 ceph-mon[94189]: pgmap v2713: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 171 KiB/s wr, 10 op/s 2023-03-09T16:06:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:06:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:06:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:06:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:06:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:06:56 smithi038 ceph-mon[92422]: pgmap v2713: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 171 KiB/s wr, 10 op/s 2023-03-09T16:06:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:06:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:06:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:06:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:06:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:06:57 smithi122 ceph-mon[93869]: pgmap v2714: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 105 KiB/s wr, 11 op/s 2023-03-09T16:06:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:06:57 smithi181 ceph-mon[94189]: pgmap v2714: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 105 KiB/s wr, 11 op/s 2023-03-09T16:06:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:06:57 smithi038 ceph-mon[92422]: pgmap v2714: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 105 KiB/s wr, 11 op/s 2023-03-09T16:06:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:06:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4128.214082 secs 2023-03-09T16:06:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:06:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4128.214082 secs 2023-03-09T16:06:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:06:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4128.214082 secs 2023-03-09T16:06:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4590.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:07:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:07:00 smithi122 ceph-mon[93869]: pgmap v2715: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 105 KiB/s wr, 9 op/s 2023-03-09T16:07:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:07:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4130.993313 secs 2023-03-09T16:07:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:07:00 smithi181 ceph-mon[94189]: pgmap v2715: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 105 KiB/s wr, 9 op/s 2023-03-09T16:07:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:07:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4130.993313 secs 2023-03-09T16:07:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:07:00 smithi038 ceph-mon[92422]: pgmap v2715: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 105 KiB/s wr, 9 op/s 2023-03-09T16:07:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:07:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4130.993313 secs 2023-03-09T16:07:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:07:02 smithi122 ceph-mon[93869]: pgmap v2716: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 115 KiB/s wr, 9 op/s 2023-03-09T16:07:02.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:07:02 smithi181 ceph-mon[94189]: pgmap v2716: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 115 KiB/s wr, 9 op/s 2023-03-09T16:07:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:07:02 smithi038 ceph-mon[92422]: pgmap v2716: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 115 KiB/s wr, 9 op/s 2023-03-09T16:07:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:07:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4133.214173 secs 2023-03-09T16:07:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:07:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4133.214173 secs 2023-03-09T16:07:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:07:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4133.214173 secs 2023-03-09T16:07:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4595.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:07:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:07:04 smithi122 ceph-mon[93869]: pgmap v2717: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 78 KiB/s wr, 10 op/s 2023-03-09T16:07:04.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:07:04 smithi181 ceph-mon[94189]: pgmap v2717: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 78 KiB/s wr, 10 op/s 2023-03-09T16:07:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:07:04 smithi038 ceph-mon[92422]: pgmap v2717: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 78 KiB/s wr, 10 op/s 2023-03-09T16:07:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:07:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4135.993369 secs 2023-03-09T16:07:05.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:07:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4135.993369 secs 2023-03-09T16:07:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:07:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4135.993369 secs 2023-03-09T16:07:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:07:06 smithi122 ceph-mon[93869]: pgmap v2718: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 110 KiB/s wr, 20 op/s 2023-03-09T16:07:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:07:06 smithi181 ceph-mon[94189]: pgmap v2718: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 110 KiB/s wr, 20 op/s 2023-03-09T16:07:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:07:06 smithi038 ceph-mon[92422]: pgmap v2718: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 110 KiB/s wr, 20 op/s 2023-03-09T16:07:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:07:08 smithi122 ceph-mon[93869]: pgmap v2719: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 78 KiB/s wr, 23 op/s 2023-03-09T16:07:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:07:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4138.214295 secs 2023-03-09T16:07:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:07:08 smithi181 ceph-mon[94189]: pgmap v2719: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 78 KiB/s wr, 23 op/s 2023-03-09T16:07:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:07:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4138.214295 secs 2023-03-09T16:07:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:07:08 smithi038 ceph-mon[92422]: pgmap v2719: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 78 KiB/s wr, 23 op/s 2023-03-09T16:07:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:07:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4138.214295 secs 2023-03-09T16:07:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4600.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:07:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:07:10 smithi122 ceph-mon[93869]: pgmap v2720: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 77 KiB/s wr, 21 op/s 2023-03-09T16:07:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:07:10 smithi122 ceph-mon[93869]: osdmap e382: 12 total, 12 up, 12 in 2023-03-09T16:07:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:07:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4140.993429 secs 2023-03-09T16:07:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:07:10 smithi181 ceph-mon[94189]: pgmap v2720: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 77 KiB/s wr, 21 op/s 2023-03-09T16:07:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:07:10 smithi181 ceph-mon[94189]: osdmap e382: 12 total, 12 up, 12 in 2023-03-09T16:07:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:07:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4140.993429 secs 2023-03-09T16:07:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:07:10 smithi038 ceph-mon[92422]: pgmap v2720: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 77 KiB/s wr, 21 op/s 2023-03-09T16:07:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:07:10 smithi038 ceph-mon[92422]: osdmap e382: 12 total, 12 up, 12 in 2023-03-09T16:07:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:07:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4140.993429 secs 2023-03-09T16:07:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:07:12 smithi122 ceph-mon[93869]: pgmap v2722: 129 pgs: 1 active+clean+snaptrim_wait, 128 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 71 KiB/s wr, 24 op/s 2023-03-09T16:07:12.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:07:12 smithi181 ceph-mon[94189]: pgmap v2722: 129 pgs: 1 active+clean+snaptrim_wait, 128 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 71 KiB/s wr, 24 op/s 2023-03-09T16:07:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:07:12 smithi038 ceph-mon[92422]: pgmap v2722: 129 pgs: 1 active+clean+snaptrim_wait, 128 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 71 KiB/s wr, 24 op/s 2023-03-09T16:07:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:07:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4143.214408 secs 2023-03-09T16:07:13.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:07:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4143.214408 secs 2023-03-09T16:07:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:07:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4143.214408 secs 2023-03-09T16:07:13.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4605.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:07:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:07:14 smithi122 ceph-mon[93869]: pgmap v2723: 129 pgs: 1 active+clean+snaptrim_wait, 128 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 71 KiB/s wr, 24 op/s 2023-03-09T16:07:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:07:14 smithi181 ceph-mon[94189]: pgmap v2723: 129 pgs: 1 active+clean+snaptrim_wait, 128 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 71 KiB/s wr, 24 op/s 2023-03-09T16:07:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:07:14 smithi038 ceph-mon[92422]: pgmap v2723: 129 pgs: 1 active+clean+snaptrim_wait, 128 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 71 KiB/s wr, 24 op/s 2023-03-09T16:07:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:07:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4145.993490 secs 2023-03-09T16:07:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:07:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4145.993490 secs 2023-03-09T16:07:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:07:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4145.993490 secs 2023-03-09T16:07:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:07:16 smithi122 ceph-mon[93869]: pgmap v2724: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 27 KiB/s wr, 8 op/s 2023-03-09T16:07:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:07:16 smithi181 ceph-mon[94189]: pgmap v2724: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 27 KiB/s wr, 8 op/s 2023-03-09T16:07:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:07:16 smithi038 ceph-mon[92422]: pgmap v2724: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 27 KiB/s wr, 8 op/s 2023-03-09T16:07:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:07:18 smithi122 ceph-mon[93869]: pgmap v2725: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 307 B/s wr, 6 op/s 2023-03-09T16:07:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:07:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4148.214534 secs 2023-03-09T16:07:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:07:18 smithi181 ceph-mon[94189]: pgmap v2725: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 307 B/s wr, 6 op/s 2023-03-09T16:07:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:07:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4148.214534 secs 2023-03-09T16:07:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:07:18 smithi038 ceph-mon[92422]: pgmap v2725: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 307 B/s wr, 6 op/s 2023-03-09T16:07:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:07:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4148.214534 secs 2023-03-09T16:07:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4610.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:07:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:07:20 smithi122 ceph-mon[93869]: pgmap v2726: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 716 B/s wr, 6 op/s 2023-03-09T16:07:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:07:20 smithi122 ceph-mon[93869]: osdmap e383: 12 total, 12 up, 12 in 2023-03-09T16:07:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:07:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4150.993575 secs 2023-03-09T16:07:20.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:07:20 smithi181 ceph-mon[94189]: pgmap v2726: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 716 B/s wr, 6 op/s 2023-03-09T16:07:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:07:20 smithi181 ceph-mon[94189]: osdmap e383: 12 total, 12 up, 12 in 2023-03-09T16:07:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:07:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4150.993575 secs 2023-03-09T16:07:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:07:20 smithi038 ceph-mon[92422]: pgmap v2726: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 716 B/s wr, 6 op/s 2023-03-09T16:07:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:07:20 smithi038 ceph-mon[92422]: osdmap e383: 12 total, 12 up, 12 in 2023-03-09T16:07:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:07:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4150.993575 secs 2023-03-09T16:07:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:07:22 smithi122 ceph-mon[93869]: pgmap v2728: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 614 B/s wr, 6 op/s 2023-03-09T16:07:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:07:22 smithi181 ceph-mon[94189]: pgmap v2728: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 614 B/s wr, 6 op/s 2023-03-09T16:07:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:07:22 smithi038 ceph-mon[92422]: pgmap v2728: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 614 B/s wr, 6 op/s 2023-03-09T16:07:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:07:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4153.214668 secs 2023-03-09T16:07:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:07:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4153.214668 secs 2023-03-09T16:07:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:07:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4153.214668 secs 2023-03-09T16:07:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4615.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:07:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:07:24 smithi122 ceph-mon[93869]: pgmap v2729: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.1 KiB/s wr, 7 op/s 2023-03-09T16:07:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:07:24 smithi181 ceph-mon[94189]: pgmap v2729: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.1 KiB/s wr, 7 op/s 2023-03-09T16:07:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:07:24 smithi038 ceph-mon[92422]: pgmap v2729: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.1 KiB/s wr, 7 op/s 2023-03-09T16:07:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:07:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4155.993634 secs 2023-03-09T16:07:25.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:07:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4155.993634 secs 2023-03-09T16:07:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:07:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4155.993634 secs 2023-03-09T16:07:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:07:26 smithi122 ceph-mon[93869]: pgmap v2730: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T16:07:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:07:26 smithi181 ceph-mon[94189]: pgmap v2730: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T16:07:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:07:26 smithi038 ceph-mon[92422]: pgmap v2730: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T16:07:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:07:28 smithi122 ceph-mon[93869]: pgmap v2731: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T16:07:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:07:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4158.214803 secs 2023-03-09T16:07:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:07:28 smithi181 ceph-mon[94189]: pgmap v2731: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T16:07:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:07:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4158.214803 secs 2023-03-09T16:07:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:07:28 smithi038 ceph-mon[92422]: pgmap v2731: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T16:07:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:07:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4158.214803 secs 2023-03-09T16:07:28.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4620.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:07:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:07:30 smithi122 ceph-mon[93869]: pgmap v2732: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 31 KiB/s wr, 7 op/s 2023-03-09T16:07:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:07:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4160.993712 secs 2023-03-09T16:07:30.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:07:30 smithi181 ceph-mon[94189]: pgmap v2732: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 31 KiB/s wr, 7 op/s 2023-03-09T16:07:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:07:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4160.993712 secs 2023-03-09T16:07:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:07:30 smithi038 ceph-mon[92422]: pgmap v2732: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 31 KiB/s wr, 7 op/s 2023-03-09T16:07:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:07:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4160.993712 secs 2023-03-09T16:07:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:07:32 smithi122 ceph-mon[93869]: pgmap v2733: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 59 KiB/s wr, 7 op/s 2023-03-09T16:07:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:07:32 smithi181 ceph-mon[94189]: pgmap v2733: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 59 KiB/s wr, 7 op/s 2023-03-09T16:07:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:07:32 smithi038 ceph-mon[92422]: pgmap v2733: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 59 KiB/s wr, 7 op/s 2023-03-09T16:07:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:07:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4163.214933 secs 2023-03-09T16:07:33.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:07:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4163.214933 secs 2023-03-09T16:07:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:07:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4163.214933 secs 2023-03-09T16:07:33.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4625.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:07:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:07:34 smithi122 ceph-mon[93869]: pgmap v2734: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 56 KiB/s wr, 8 op/s 2023-03-09T16:07:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:07:34 smithi181 ceph-mon[94189]: pgmap v2734: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 56 KiB/s wr, 8 op/s 2023-03-09T16:07:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:07:34 smithi038 ceph-mon[92422]: pgmap v2734: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 56 KiB/s wr, 8 op/s 2023-03-09T16:07:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:07:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4165.993793 secs 2023-03-09T16:07:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:07:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4165.993793 secs 2023-03-09T16:07:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:07:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4165.993793 secs 2023-03-09T16:07:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:07:36 smithi038 ceph-mon[92422]: pgmap v2735: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 118 KiB/s wr, 7 op/s 2023-03-09T16:07:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:07:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:07:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:07:36 smithi122 ceph-mon[93869]: pgmap v2735: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 118 KiB/s wr, 7 op/s 2023-03-09T16:07:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:07:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:07:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:07:36 smithi181 ceph-mon[94189]: pgmap v2735: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 118 KiB/s wr, 7 op/s 2023-03-09T16:07:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:07:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:07:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:07:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:07:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:07:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:07:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:07:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:07:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:07:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:07:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:07:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:07:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:07:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:07:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:07:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:07:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:07:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:07:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:07:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:07:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:07:38 smithi038 ceph-mon[92422]: pgmap v2736: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 93 KiB/s wr, 8 op/s 2023-03-09T16:07:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:07:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4168.215040 secs 2023-03-09T16:07:38.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:07:38 smithi122 ceph-mon[93869]: pgmap v2736: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 93 KiB/s wr, 8 op/s 2023-03-09T16:07:38.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:07:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4168.215040 secs 2023-03-09T16:07:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:07:38 smithi181 ceph-mon[94189]: pgmap v2736: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 93 KiB/s wr, 8 op/s 2023-03-09T16:07:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:07:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4168.215040 secs 2023-03-09T16:07:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4630.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:07:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:07:40 smithi122 ceph-mon[93869]: pgmap v2737: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 97 KiB/s wr, 7 op/s 2023-03-09T16:07:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:07:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4170.993855 secs 2023-03-09T16:07:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:07:40 smithi181 ceph-mon[94189]: pgmap v2737: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 97 KiB/s wr, 7 op/s 2023-03-09T16:07:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:07:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4170.993855 secs 2023-03-09T16:07:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:07:40 smithi038 ceph-mon[92422]: pgmap v2737: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 97 KiB/s wr, 7 op/s 2023-03-09T16:07:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:07:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4170.993855 secs 2023-03-09T16:07:42.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:07:42 smithi122 ceph-mon[93869]: pgmap v2738: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 163 KiB/s wr, 8 op/s 2023-03-09T16:07:42.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:07:42 smithi181 ceph-mon[94189]: pgmap v2738: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 163 KiB/s wr, 8 op/s 2023-03-09T16:07:42.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:07:42 smithi038 ceph-mon[92422]: pgmap v2738: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 163 KiB/s wr, 8 op/s 2023-03-09T16:07:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:07:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4173.215185 secs 2023-03-09T16:07:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:07:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4173.215185 secs 2023-03-09T16:07:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:07:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4173.215185 secs 2023-03-09T16:07:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4635.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:07:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:07:44 smithi122 ceph-mon[93869]: pgmap v2739: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 139 KiB/s wr, 10 op/s 2023-03-09T16:07:44.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:07:44 smithi181 ceph-mon[94189]: pgmap v2739: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 139 KiB/s wr, 10 op/s 2023-03-09T16:07:44.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:07:44 smithi038 ceph-mon[92422]: pgmap v2739: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 139 KiB/s wr, 10 op/s 2023-03-09T16:07:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:07:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4175.993917 secs 2023-03-09T16:07:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:07:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4175.993917 secs 2023-03-09T16:07:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:07:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4175.993917 secs 2023-03-09T16:07:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:07:46 smithi122 ceph-mon[93869]: pgmap v2740: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 212 KiB/s wr, 10 op/s 2023-03-09T16:07:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:07:46 smithi181 ceph-mon[94189]: pgmap v2740: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 212 KiB/s wr, 10 op/s 2023-03-09T16:07:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:07:46 smithi038 ceph-mon[92422]: pgmap v2740: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 212 KiB/s wr, 10 op/s 2023-03-09T16:07:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:07:48 smithi122 ceph-mon[93869]: pgmap v2741: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 150 KiB/s wr, 11 op/s 2023-03-09T16:07:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:07:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4178.215254 secs 2023-03-09T16:07:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:07:48 smithi181 ceph-mon[94189]: pgmap v2741: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 150 KiB/s wr, 11 op/s 2023-03-09T16:07:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:07:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4178.215254 secs 2023-03-09T16:07:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:07:48 smithi038 ceph-mon[92422]: pgmap v2741: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 150 KiB/s wr, 11 op/s 2023-03-09T16:07:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:07:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4178.215254 secs 2023-03-09T16:07:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4640.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:07:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:07:50 smithi122 ceph-mon[93869]: pgmap v2742: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 158 KiB/s wr, 11 op/s 2023-03-09T16:07:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:07:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4180.993985 secs 2023-03-09T16:07:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:07:50 smithi181 ceph-mon[94189]: pgmap v2742: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 158 KiB/s wr, 11 op/s 2023-03-09T16:07:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:07:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4180.993985 secs 2023-03-09T16:07:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:07:50 smithi038 ceph-mon[92422]: pgmap v2742: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 158 KiB/s wr, 11 op/s 2023-03-09T16:07:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:07:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4180.993985 secs 2023-03-09T16:07:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:07:52 smithi122 ceph-mon[93869]: pgmap v2743: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 185 KiB/s wr, 11 op/s 2023-03-09T16:07:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:07:52 smithi181 ceph-mon[94189]: pgmap v2743: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 185 KiB/s wr, 11 op/s 2023-03-09T16:07:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:07:52 smithi038 ceph-mon[92422]: pgmap v2743: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 185 KiB/s wr, 11 op/s 2023-03-09T16:07:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:07:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4183.215381 secs 2023-03-09T16:07:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:07:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4183.215381 secs 2023-03-09T16:07:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:07:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4183.215381 secs 2023-03-09T16:07:53.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4645.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:07:54.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:07:54 smithi122 ceph-mon[93869]: pgmap v2744: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 123 KiB/s wr, 12 op/s 2023-03-09T16:07:54.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:07:54 smithi181 ceph-mon[94189]: pgmap v2744: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 123 KiB/s wr, 12 op/s 2023-03-09T16:07:54.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:07:54 smithi038 ceph-mon[92422]: pgmap v2744: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 123 KiB/s wr, 12 op/s 2023-03-09T16:07:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:07:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4185.994046 secs 2023-03-09T16:07:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:07:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4185.994046 secs 2023-03-09T16:07:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:07:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4185.994046 secs 2023-03-09T16:07:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:07:56 smithi122 ceph-mon[93869]: pgmap v2745: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 283 KiB/s rd, 162 KiB/s wr, 11 op/s 2023-03-09T16:07:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:07:56 smithi181 ceph-mon[94189]: pgmap v2745: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 283 KiB/s rd, 162 KiB/s wr, 11 op/s 2023-03-09T16:07:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:07:56 smithi038 ceph-mon[92422]: pgmap v2745: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 283 KiB/s rd, 162 KiB/s wr, 11 op/s 2023-03-09T16:07:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:07:57 smithi122 ceph-mon[93869]: pgmap v2746: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 89 KiB/s wr, 11 op/s 2023-03-09T16:07:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:07:57 smithi181 ceph-mon[94189]: pgmap v2746: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 89 KiB/s wr, 11 op/s 2023-03-09T16:07:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:07:57 smithi038 ceph-mon[92422]: pgmap v2746: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 89 KiB/s wr, 11 op/s 2023-03-09T16:07:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:07:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4188.215443 secs 2023-03-09T16:07:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:07:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4188.215443 secs 2023-03-09T16:07:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:07:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4188.215443 secs 2023-03-09T16:07:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4650.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:08:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:00 smithi122 ceph-mon[93869]: pgmap v2747: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 89 KiB/s wr, 10 op/s 2023-03-09T16:08:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4190.994107 secs 2023-03-09T16:08:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:00 smithi181 ceph-mon[94189]: pgmap v2747: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 89 KiB/s wr, 10 op/s 2023-03-09T16:08:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4190.994107 secs 2023-03-09T16:08:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:00 smithi038 ceph-mon[92422]: pgmap v2747: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 89 KiB/s wr, 10 op/s 2023-03-09T16:08:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4190.994107 secs 2023-03-09T16:08:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:02 smithi122 ceph-mon[93869]: pgmap v2748: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 107 KiB/s wr, 9 op/s 2023-03-09T16:08:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:02 smithi181 ceph-mon[94189]: pgmap v2748: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 107 KiB/s wr, 9 op/s 2023-03-09T16:08:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:02 smithi038 ceph-mon[92422]: pgmap v2748: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 107 KiB/s wr, 9 op/s 2023-03-09T16:08:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4193.215555 secs 2023-03-09T16:08:03.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4193.215555 secs 2023-03-09T16:08:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4193.215555 secs 2023-03-09T16:08:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4655.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:08:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:04 smithi122 ceph-mon[93869]: pgmap v2749: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 75 KiB/s wr, 10 op/s 2023-03-09T16:08:04.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:04 smithi181 ceph-mon[94189]: pgmap v2749: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 75 KiB/s wr, 10 op/s 2023-03-09T16:08:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:04 smithi038 ceph-mon[92422]: pgmap v2749: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 75 KiB/s wr, 10 op/s 2023-03-09T16:08:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4195.994169 secs 2023-03-09T16:08:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4195.994169 secs 2023-03-09T16:08:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4195.994169 secs 2023-03-09T16:08:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:06 smithi122 ceph-mon[93869]: pgmap v2750: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 93 KiB/s wr, 20 op/s 2023-03-09T16:08:06.487 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:06 smithi181 ceph-mon[94189]: pgmap v2750: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 93 KiB/s wr, 20 op/s 2023-03-09T16:08:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:06 smithi038 ceph-mon[92422]: pgmap v2750: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 93 KiB/s wr, 20 op/s 2023-03-09T16:08:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:08 smithi122 ceph-mon[93869]: pgmap v2751: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 72 KiB/s wr, 23 op/s 2023-03-09T16:08:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4198.215673 secs 2023-03-09T16:08:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:08 smithi181 ceph-mon[94189]: pgmap v2751: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 72 KiB/s wr, 23 op/s 2023-03-09T16:08:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4198.215673 secs 2023-03-09T16:08:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:08 smithi038 ceph-mon[92422]: pgmap v2751: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 72 KiB/s wr, 23 op/s 2023-03-09T16:08:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4198.215673 secs 2023-03-09T16:08:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4660.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:08:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:10 smithi122 ceph-mon[93869]: pgmap v2752: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 71 KiB/s wr, 21 op/s 2023-03-09T16:08:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:10 smithi122 ceph-mon[93869]: osdmap e384: 12 total, 12 up, 12 in 2023-03-09T16:08:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4200.994259 secs 2023-03-09T16:08:10.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:10 smithi181 ceph-mon[94189]: pgmap v2752: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 71 KiB/s wr, 21 op/s 2023-03-09T16:08:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:10 smithi181 ceph-mon[94189]: osdmap e384: 12 total, 12 up, 12 in 2023-03-09T16:08:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4200.994259 secs 2023-03-09T16:08:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:10 smithi038 ceph-mon[92422]: pgmap v2752: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 71 KiB/s wr, 21 op/s 2023-03-09T16:08:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:10 smithi038 ceph-mon[92422]: osdmap e384: 12 total, 12 up, 12 in 2023-03-09T16:08:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4200.994259 secs 2023-03-09T16:08:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:12 smithi038 ceph-mon[92422]: pgmap v2754: 129 pgs: 9 active+clean+snaptrim_wait, 120 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 54 KiB/s wr, 24 op/s 2023-03-09T16:08:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:12 smithi122 ceph-mon[93869]: pgmap v2754: 129 pgs: 9 active+clean+snaptrim_wait, 120 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 54 KiB/s wr, 24 op/s 2023-03-09T16:08:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:12 smithi181 ceph-mon[94189]: pgmap v2754: 129 pgs: 9 active+clean+snaptrim_wait, 120 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 54 KiB/s wr, 24 op/s 2023-03-09T16:08:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4203.215809 secs 2023-03-09T16:08:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4203.215809 secs 2023-03-09T16:08:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4203.215809 secs 2023-03-09T16:08:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4665.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:08:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:14 smithi038 ceph-mon[92422]: pgmap v2755: 129 pgs: 9 active+clean+snaptrim_wait, 120 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 54 KiB/s wr, 23 op/s 2023-03-09T16:08:14.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:14 smithi122 ceph-mon[93869]: pgmap v2755: 129 pgs: 9 active+clean+snaptrim_wait, 120 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 54 KiB/s wr, 23 op/s 2023-03-09T16:08:14.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:14 smithi181 ceph-mon[94189]: pgmap v2755: 129 pgs: 9 active+clean+snaptrim_wait, 120 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 54 KiB/s wr, 23 op/s 2023-03-09T16:08:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4205.994319 secs 2023-03-09T16:08:15.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4205.994319 secs 2023-03-09T16:08:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4205.994319 secs 2023-03-09T16:08:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:16 smithi122 ceph-mon[93869]: pgmap v2756: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 27 KiB/s wr, 8 op/s 2023-03-09T16:08:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:16 smithi181 ceph-mon[94189]: pgmap v2756: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 27 KiB/s wr, 8 op/s 2023-03-09T16:08:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:16 smithi038 ceph-mon[92422]: pgmap v2756: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 27 KiB/s wr, 8 op/s 2023-03-09T16:08:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:18 smithi122 ceph-mon[93869]: pgmap v2757: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T16:08:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4208.215938 secs 2023-03-09T16:08:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:18 smithi181 ceph-mon[94189]: pgmap v2757: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T16:08:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4208.215938 secs 2023-03-09T16:08:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:18 smithi038 ceph-mon[92422]: pgmap v2757: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T16:08:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4208.215938 secs 2023-03-09T16:08:18.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4670.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:08:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:20 smithi122 ceph-mon[93869]: pgmap v2758: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T16:08:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:20 smithi122 ceph-mon[93869]: osdmap e385: 12 total, 12 up, 12 in 2023-03-09T16:08:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4210.994398 secs 2023-03-09T16:08:20.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:20 smithi181 ceph-mon[94189]: pgmap v2758: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T16:08:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:20 smithi181 ceph-mon[94189]: osdmap e385: 12 total, 12 up, 12 in 2023-03-09T16:08:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4210.994398 secs 2023-03-09T16:08:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:20 smithi038 ceph-mon[92422]: pgmap v2758: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T16:08:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:20 smithi038 ceph-mon[92422]: osdmap e385: 12 total, 12 up, 12 in 2023-03-09T16:08:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4210.994398 secs 2023-03-09T16:08:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:22 smithi122 ceph-mon[93869]: pgmap v2760: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 27 KiB/s wr, 6 op/s 2023-03-09T16:08:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:22 smithi181 ceph-mon[94189]: pgmap v2760: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 27 KiB/s wr, 6 op/s 2023-03-09T16:08:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:22 smithi038 ceph-mon[92422]: pgmap v2760: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 27 KiB/s wr, 6 op/s 2023-03-09T16:08:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4213.216075 secs 2023-03-09T16:08:23.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4213.216075 secs 2023-03-09T16:08:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4213.216075 secs 2023-03-09T16:08:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4675.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:08:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:24 smithi122 ceph-mon[93869]: pgmap v2761: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T16:08:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:24 smithi181 ceph-mon[94189]: pgmap v2761: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T16:08:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:24 smithi038 ceph-mon[92422]: pgmap v2761: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T16:08:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4215.994457 secs 2023-03-09T16:08:25.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4215.994457 secs 2023-03-09T16:08:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4215.994457 secs 2023-03-09T16:08:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:26 smithi122 ceph-mon[93869]: pgmap v2762: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 7 op/s 2023-03-09T16:08:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:26 smithi181 ceph-mon[94189]: pgmap v2762: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 7 op/s 2023-03-09T16:08:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:26 smithi038 ceph-mon[92422]: pgmap v2762: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 7 op/s 2023-03-09T16:08:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:28 smithi122 ceph-mon[93869]: pgmap v2763: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 7 op/s 2023-03-09T16:08:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4218.216187 secs 2023-03-09T16:08:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:28 smithi181 ceph-mon[94189]: pgmap v2763: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 7 op/s 2023-03-09T16:08:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4218.216187 secs 2023-03-09T16:08:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:28 smithi038 ceph-mon[92422]: pgmap v2763: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 7 op/s 2023-03-09T16:08:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4218.216187 secs 2023-03-09T16:08:28.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4680.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:08:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:30 smithi122 ceph-mon[93869]: pgmap v2764: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 59 KiB/s wr, 8 op/s 2023-03-09T16:08:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4220.994529 secs 2023-03-09T16:08:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:30 smithi181 ceph-mon[94189]: pgmap v2764: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 59 KiB/s wr, 8 op/s 2023-03-09T16:08:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4220.994529 secs 2023-03-09T16:08:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:30 smithi038 ceph-mon[92422]: pgmap v2764: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 59 KiB/s wr, 8 op/s 2023-03-09T16:08:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4220.994529 secs 2023-03-09T16:08:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:32 smithi122 ceph-mon[93869]: pgmap v2765: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 60 KiB/s wr, 7 op/s 2023-03-09T16:08:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:32 smithi181 ceph-mon[94189]: pgmap v2765: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 60 KiB/s wr, 7 op/s 2023-03-09T16:08:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:32 smithi038 ceph-mon[92422]: pgmap v2765: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 60 KiB/s wr, 7 op/s 2023-03-09T16:08:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4223.216283 secs 2023-03-09T16:08:33.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4223.216283 secs 2023-03-09T16:08:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4223.216283 secs 2023-03-09T16:08:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4685.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:08:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:34 smithi122 ceph-mon[93869]: pgmap v2766: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 57 KiB/s wr, 8 op/s 2023-03-09T16:08:34.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:34 smithi181 ceph-mon[94189]: pgmap v2766: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 57 KiB/s wr, 8 op/s 2023-03-09T16:08:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:34 smithi038 ceph-mon[92422]: pgmap v2766: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 57 KiB/s wr, 8 op/s 2023-03-09T16:08:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4225.994591 secs 2023-03-09T16:08:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4225.994591 secs 2023-03-09T16:08:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4225.994591 secs 2023-03-09T16:08:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:36 smithi181 ceph-mon[94189]: pgmap v2767: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 120 KiB/s wr, 7 op/s 2023-03-09T16:08:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:36 smithi038 ceph-mon[92422]: pgmap v2767: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 120 KiB/s wr, 7 op/s 2023-03-09T16:08:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:36 smithi122 ceph-mon[93869]: pgmap v2767: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 120 KiB/s wr, 7 op/s 2023-03-09T16:08:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:08:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:08:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:08:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:38 smithi038 ceph-mon[92422]: pgmap v2768: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 94 KiB/s wr, 8 op/s 2023-03-09T16:08:38.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4228.216357 secs 2023-03-09T16:08:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:38 smithi122 ceph-mon[93869]: pgmap v2768: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 94 KiB/s wr, 8 op/s 2023-03-09T16:08:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4228.216357 secs 2023-03-09T16:08:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:38 smithi181 ceph-mon[94189]: pgmap v2768: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 94 KiB/s wr, 8 op/s 2023-03-09T16:08:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4228.216357 secs 2023-03-09T16:08:38.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4690.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:08:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:40 smithi122 ceph-mon[93869]: pgmap v2769: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 98 KiB/s wr, 7 op/s 2023-03-09T16:08:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:08:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:08:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:08:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:08:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4230.994654 secs 2023-03-09T16:08:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:08:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:08:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:08:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:08:40.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:08:40.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:08:40.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:08:40.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:08:40.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:08:40.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:08:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:40 smithi181 ceph-mon[94189]: pgmap v2769: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 98 KiB/s wr, 7 op/s 2023-03-09T16:08:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:08:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:08:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:08:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:08:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4230.994654 secs 2023-03-09T16:08:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:08:40.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:08:40.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:08:40.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:08:40.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:08:40.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:08:40.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:08:40.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:08:40.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:08:40.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:08:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:40 smithi038 ceph-mon[92422]: pgmap v2769: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 98 KiB/s wr, 7 op/s 2023-03-09T16:08:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:08:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:08:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:08:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:08:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4230.994654 secs 2023-03-09T16:08:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:08:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:08:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:08:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:08:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:08:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:08:40.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:08:40.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:08:40.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:08:40.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:08:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:41 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T16:08:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:41 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T16:08:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:41 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi038 to 197.2M 2023-03-09T16:08:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:41 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi038 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-03-09T16:08:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:08:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:08:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:08:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:08:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:08:41.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:08:41.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:08:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:41 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T16:08:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:41 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T16:08:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:41 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi038 to 197.2M 2023-03-09T16:08:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:41 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi038 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-03-09T16:08:41.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:08:41.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:08:41.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:08:41.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:08:41.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:08:41.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:08:41.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:08:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:41 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T16:08:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:41 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T16:08:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:41 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi038 to 197.2M 2023-03-09T16:08:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:41 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi038 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-03-09T16:08:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:08:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:08:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:08:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:08:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:08:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:08:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:08:42.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:42 smithi122 ceph-mon[93869]: pgmap v2770: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 128 KiB/s wr, 7 op/s 2023-03-09T16:08:42.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:42 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T16:08:42.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:42 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T16:08:42.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:42 smithi181 ceph-mon[94189]: pgmap v2770: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 128 KiB/s wr, 7 op/s 2023-03-09T16:08:42.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:42 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T16:08:42.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:42 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T16:08:42.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:42 smithi038 ceph-mon[92422]: pgmap v2770: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 128 KiB/s wr, 7 op/s 2023-03-09T16:08:42.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:42 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T16:08:42.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:42 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T16:08:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:43 smithi122 ceph-mon[93869]: pgmap v2771: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 104 KiB/s wr, 10 op/s 2023-03-09T16:08:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:43 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:08:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:43 smithi181 ceph-mon[94189]: pgmap v2771: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 104 KiB/s wr, 10 op/s 2023-03-09T16:08:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:43 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:08:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:43 smithi038 ceph-mon[92422]: pgmap v2771: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 104 KiB/s wr, 10 op/s 2023-03-09T16:08:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:43 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:08:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4695.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:08:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:44 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:08:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:44 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:08:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:44 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:08:44.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:44 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:08:44.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:44 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:08:44.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:44 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:08:44.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:44 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:08:44.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:44 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:08:44.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:44 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:08:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:45 smithi122 ceph-mon[93869]: pgmap v2772: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 173 KiB/s wr, 9 op/s 2023-03-09T16:08:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4235.994715 secs 2023-03-09T16:08:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:45 smithi181 ceph-mon[94189]: pgmap v2772: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 173 KiB/s wr, 9 op/s 2023-03-09T16:08:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4235.994715 secs 2023-03-09T16:08:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:45 smithi038 ceph-mon[92422]: pgmap v2772: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 173 KiB/s wr, 9 op/s 2023-03-09T16:08:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4235.994715 secs 2023-03-09T16:08:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:47 smithi122 ceph-mon[93869]: pgmap v2773: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 138 KiB/s wr, 12 op/s 2023-03-09T16:08:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:47 smithi181 ceph-mon[94189]: pgmap v2773: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 138 KiB/s wr, 12 op/s 2023-03-09T16:08:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:47 smithi038 ceph-mon[92422]: pgmap v2773: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 138 KiB/s wr, 12 op/s 2023-03-09T16:08:48.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4700.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:08:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:50 smithi122 ceph-mon[93869]: pgmap v2774: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 143 KiB/s wr, 11 op/s 2023-03-09T16:08:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4240.994800 secs 2023-03-09T16:08:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:50 smithi181 ceph-mon[94189]: pgmap v2774: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 143 KiB/s wr, 11 op/s 2023-03-09T16:08:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4240.994800 secs 2023-03-09T16:08:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:50 smithi038 ceph-mon[92422]: pgmap v2774: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 143 KiB/s wr, 11 op/s 2023-03-09T16:08:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4240.994800 secs 2023-03-09T16:08:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:52 smithi122 ceph-mon[93869]: pgmap v2775: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 138 KiB/s wr, 10 op/s 2023-03-09T16:08:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:52 smithi181 ceph-mon[94189]: pgmap v2775: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 138 KiB/s wr, 10 op/s 2023-03-09T16:08:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:52 smithi038 ceph-mon[92422]: pgmap v2775: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 138 KiB/s wr, 10 op/s 2023-03-09T16:08:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4243.216662 secs 2023-03-09T16:08:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4243.216662 secs 2023-03-09T16:08:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4243.216662 secs 2023-03-09T16:08:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4705.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:08:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:54 smithi122 ceph-mon[93869]: pgmap v2776: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 169 KiB/s wr, 13 op/s 2023-03-09T16:08:54.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:54 smithi181 ceph-mon[94189]: pgmap v2776: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 169 KiB/s wr, 13 op/s 2023-03-09T16:08:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:54 smithi038 ceph-mon[92422]: pgmap v2776: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 169 KiB/s wr, 13 op/s 2023-03-09T16:08:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4245.994858 secs 2023-03-09T16:08:55.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4245.994858 secs 2023-03-09T16:08:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4245.994858 secs 2023-03-09T16:08:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:56 smithi122 ceph-mon[93869]: pgmap v2777: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 164 KiB/s wr, 10 op/s 2023-03-09T16:08:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:56 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:08:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:08:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:08:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:56 smithi181 ceph-mon[94189]: pgmap v2777: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 164 KiB/s wr, 10 op/s 2023-03-09T16:08:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:56 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:08:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:08:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:08:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:56 smithi038 ceph-mon[92422]: pgmap v2777: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 164 KiB/s wr, 10 op/s 2023-03-09T16:08:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:56 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:08:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:08:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:08:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:58 smithi122 ceph-mon[93869]: pgmap v2778: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 140 KiB/s wr, 12 op/s 2023-03-09T16:08:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:08:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4248.216787 secs 2023-03-09T16:08:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:58 smithi181 ceph-mon[94189]: pgmap v2778: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 140 KiB/s wr, 12 op/s 2023-03-09T16:08:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:08:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4248.216787 secs 2023-03-09T16:08:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:58 smithi038 ceph-mon[92422]: pgmap v2778: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 140 KiB/s wr, 12 op/s 2023-03-09T16:08:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:08:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4248.216787 secs 2023-03-09T16:08:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4710.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:09:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:09:00 smithi122 ceph-mon[93869]: pgmap v2779: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 112 KiB/s wr, 10 op/s 2023-03-09T16:09:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:09:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4250.994920 secs 2023-03-09T16:09:00.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:09:00 smithi181 ceph-mon[94189]: pgmap v2779: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 112 KiB/s wr, 10 op/s 2023-03-09T16:09:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:09:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4250.994920 secs 2023-03-09T16:09:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:09:00 smithi038 ceph-mon[92422]: pgmap v2779: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 112 KiB/s wr, 10 op/s 2023-03-09T16:09:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:09:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4250.994920 secs 2023-03-09T16:09:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:09:02 smithi122 ceph-mon[93869]: pgmap v2780: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 107 KiB/s wr, 8 op/s 2023-03-09T16:09:02.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:09:02 smithi181 ceph-mon[94189]: pgmap v2780: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 107 KiB/s wr, 8 op/s 2023-03-09T16:09:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:09:02 smithi038 ceph-mon[92422]: pgmap v2780: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 107 KiB/s wr, 8 op/s 2023-03-09T16:09:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:09:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4253.216902 secs 2023-03-09T16:09:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:09:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4253.216902 secs 2023-03-09T16:09:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:09:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4253.216902 secs 2023-03-09T16:09:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4715.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:09:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:09:04 smithi038 ceph-mon[92422]: pgmap v2781: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 157 KiB/s wr, 13 op/s 2023-03-09T16:09:04.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:09:04 smithi122 ceph-mon[93869]: pgmap v2781: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 157 KiB/s wr, 13 op/s 2023-03-09T16:09:04.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:09:04 smithi181 ceph-mon[94189]: pgmap v2781: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 157 KiB/s wr, 13 op/s 2023-03-09T16:09:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:09:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4255.995027 secs 2023-03-09T16:09:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:09:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4255.995027 secs 2023-03-09T16:09:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:09:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4255.995027 secs 2023-03-09T16:09:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:09:06 smithi038 ceph-mon[92422]: pgmap v2782: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 117 KiB/s wr, 21 op/s 2023-03-09T16:09:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:09:06 smithi122 ceph-mon[93869]: pgmap v2782: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 117 KiB/s wr, 21 op/s 2023-03-09T16:09:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:09:06 smithi181 ceph-mon[94189]: pgmap v2782: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 117 KiB/s wr, 21 op/s 2023-03-09T16:09:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:09:08 smithi038 ceph-mon[92422]: pgmap v2783: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 117 KiB/s wr, 23 op/s 2023-03-09T16:09:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:09:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4258.217027 secs 2023-03-09T16:09:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:09:08 smithi122 ceph-mon[93869]: pgmap v2783: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 117 KiB/s wr, 23 op/s 2023-03-09T16:09:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:09:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4258.217027 secs 2023-03-09T16:09:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:09:08 smithi181 ceph-mon[94189]: pgmap v2783: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 117 KiB/s wr, 23 op/s 2023-03-09T16:09:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:09:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4258.217027 secs 2023-03-09T16:09:08.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4720.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:09:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:09:10 smithi122 ceph-mon[93869]: pgmap v2784: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 72 KiB/s wr, 21 op/s 2023-03-09T16:09:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:09:10 smithi122 ceph-mon[93869]: osdmap e386: 12 total, 12 up, 12 in 2023-03-09T16:09:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:09:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4260.995117 secs 2023-03-09T16:09:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:09:10 smithi181 ceph-mon[94189]: pgmap v2784: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 72 KiB/s wr, 21 op/s 2023-03-09T16:09:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:09:10 smithi181 ceph-mon[94189]: osdmap e386: 12 total, 12 up, 12 in 2023-03-09T16:09:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:09:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4260.995117 secs 2023-03-09T16:09:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:09:10 smithi038 ceph-mon[92422]: pgmap v2784: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 72 KiB/s wr, 21 op/s 2023-03-09T16:09:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:09:10 smithi038 ceph-mon[92422]: osdmap e386: 12 total, 12 up, 12 in 2023-03-09T16:09:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:09:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4260.995117 secs 2023-03-09T16:09:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:09:12 smithi122 ceph-mon[93869]: pgmap v2786: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 86 KiB/s wr, 24 op/s 2023-03-09T16:09:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:09:12 smithi181 ceph-mon[94189]: pgmap v2786: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 86 KiB/s wr, 24 op/s 2023-03-09T16:09:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:09:12 smithi038 ceph-mon[92422]: pgmap v2786: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 86 KiB/s wr, 24 op/s 2023-03-09T16:09:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:09:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4263.217087 secs 2023-03-09T16:09:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:09:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4263.217087 secs 2023-03-09T16:09:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:09:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4263.217087 secs 2023-03-09T16:09:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4725.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:09:14.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:09:14 smithi122 ceph-mon[93869]: pgmap v2787: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 48 KiB/s wr, 21 op/s 2023-03-09T16:09:14.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:09:14 smithi181 ceph-mon[94189]: pgmap v2787: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 48 KiB/s wr, 21 op/s 2023-03-09T16:09:14.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:09:14 smithi038 ceph-mon[92422]: pgmap v2787: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 48 KiB/s wr, 21 op/s 2023-03-09T16:09:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:09:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4265.995179 secs 2023-03-09T16:09:15.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:09:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4265.995179 secs 2023-03-09T16:09:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:09:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4265.995179 secs 2023-03-09T16:09:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:09:16 smithi122 ceph-mon[93869]: pgmap v2788: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 22 KiB/s wr, 6 op/s 2023-03-09T16:09:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:09:16 smithi181 ceph-mon[94189]: pgmap v2788: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 22 KiB/s wr, 6 op/s 2023-03-09T16:09:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:09:16 smithi038 ceph-mon[92422]: pgmap v2788: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 22 KiB/s wr, 6 op/s 2023-03-09T16:09:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:09:18 smithi122 ceph-mon[93869]: pgmap v2789: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 6 op/s 2023-03-09T16:09:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:09:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4268.217205 secs 2023-03-09T16:09:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:09:18 smithi181 ceph-mon[94189]: pgmap v2789: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 6 op/s 2023-03-09T16:09:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:09:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4268.217205 secs 2023-03-09T16:09:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:09:18 smithi038 ceph-mon[92422]: pgmap v2789: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 6 op/s 2023-03-09T16:09:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:09:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4268.217205 secs 2023-03-09T16:09:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4730.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:09:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:09:20 smithi122 ceph-mon[93869]: pgmap v2790: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 6 op/s 2023-03-09T16:09:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:09:20 smithi122 ceph-mon[93869]: osdmap e387: 12 total, 12 up, 12 in 2023-03-09T16:09:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:09:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4270.995241 secs 2023-03-09T16:09:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:09:20 smithi181 ceph-mon[94189]: pgmap v2790: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 6 op/s 2023-03-09T16:09:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:09:20 smithi181 ceph-mon[94189]: osdmap e387: 12 total, 12 up, 12 in 2023-03-09T16:09:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:09:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4270.995241 secs 2023-03-09T16:09:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:09:20 smithi038 ceph-mon[92422]: pgmap v2790: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 6 op/s 2023-03-09T16:09:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:09:20 smithi038 ceph-mon[92422]: osdmap e387: 12 total, 12 up, 12 in 2023-03-09T16:09:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:09:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4270.995241 secs 2023-03-09T16:09:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:09:22 smithi122 ceph-mon[93869]: pgmap v2792: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 6 op/s 2023-03-09T16:09:22.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:09:22 smithi181 ceph-mon[94189]: pgmap v2792: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 6 op/s 2023-03-09T16:09:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:09:22 smithi038 ceph-mon[92422]: pgmap v2792: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 6 op/s 2023-03-09T16:09:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:09:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4273.217294 secs 2023-03-09T16:09:23.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:09:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4273.217294 secs 2023-03-09T16:09:23.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:09:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4273.217294 secs 2023-03-09T16:09:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4735.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:09:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:09:24 smithi122 ceph-mon[93869]: pgmap v2793: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 614 B/s wr, 6 op/s 2023-03-09T16:09:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:09:24 smithi181 ceph-mon[94189]: pgmap v2793: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 614 B/s wr, 6 op/s 2023-03-09T16:09:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:09:24 smithi038 ceph-mon[92422]: pgmap v2793: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 614 B/s wr, 6 op/s 2023-03-09T16:09:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:09:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4275.995301 secs 2023-03-09T16:09:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:09:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4275.995301 secs 2023-03-09T16:09:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:09:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4275.995301 secs 2023-03-09T16:09:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:09:26 smithi038 ceph-mon[92422]: pgmap v2794: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 614 B/s wr, 7 op/s 2023-03-09T16:09:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:09:26 smithi122 ceph-mon[93869]: pgmap v2794: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 614 B/s wr, 7 op/s 2023-03-09T16:09:26.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:09:26 smithi181 ceph-mon[94189]: pgmap v2794: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 614 B/s wr, 7 op/s 2023-03-09T16:09:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:09:28 smithi038 ceph-mon[92422]: pgmap v2795: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T16:09:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:09:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4278.217411 secs 2023-03-09T16:09:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:09:28 smithi122 ceph-mon[93869]: pgmap v2795: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T16:09:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:09:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4278.217411 secs 2023-03-09T16:09:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:09:28 smithi181 ceph-mon[94189]: pgmap v2795: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T16:09:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:09:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4278.217411 secs 2023-03-09T16:09:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4740.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:09:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:09:30 smithi038 ceph-mon[92422]: pgmap v2796: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T16:09:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:09:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4280.995363 secs 2023-03-09T16:09:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:09:30 smithi122 ceph-mon[93869]: pgmap v2796: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T16:09:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:09:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4280.995363 secs 2023-03-09T16:09:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:09:30 smithi181 ceph-mon[94189]: pgmap v2796: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T16:09:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:09:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4280.995363 secs 2023-03-09T16:09:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:09:32 smithi038 ceph-mon[92422]: pgmap v2797: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T16:09:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:09:32 smithi122 ceph-mon[93869]: pgmap v2797: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T16:09:32.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:09:32 smithi181 ceph-mon[94189]: pgmap v2797: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T16:09:33.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4745.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:09:34.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:09:34 smithi122 ceph-mon[93869]: pgmap v2798: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 9 op/s 2023-03-09T16:09:34.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:09:34 smithi181 ceph-mon[94189]: pgmap v2798: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 9 op/s 2023-03-09T16:09:34.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:09:34 smithi038 ceph-mon[92422]: pgmap v2798: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 9 op/s 2023-03-09T16:09:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:09:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4285.995444 secs 2023-03-09T16:09:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:09:35 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:09:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:09:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4285.995444 secs 2023-03-09T16:09:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:09:35 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:09:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:09:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4285.995444 secs 2023-03-09T16:09:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:09:35 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:09:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:09:36 smithi122 ceph-mon[93869]: pgmap v2799: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 57 KiB/s wr, 6 op/s 2023-03-09T16:09:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:09:36 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:09:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:09:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:09:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:09:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:09:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:09:36 smithi181 ceph-mon[94189]: pgmap v2799: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 57 KiB/s wr, 6 op/s 2023-03-09T16:09:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:09:36 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:09:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:09:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:09:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:09:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:09:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:09:36 smithi038 ceph-mon[92422]: pgmap v2799: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 57 KiB/s wr, 6 op/s 2023-03-09T16:09:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:09:36 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:09:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:09:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:09:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:09:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:09:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:09:38 smithi122 ceph-mon[93869]: pgmap v2800: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 121 KiB/s wr, 8 op/s 2023-03-09T16:09:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:09:38 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 4288.217700 secs 2023-03-09T16:09:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:09:38 smithi181 ceph-mon[94189]: pgmap v2800: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 121 KiB/s wr, 8 op/s 2023-03-09T16:09:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:09:38 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 4288.217700 secs 2023-03-09T16:09:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:09:38 smithi038 ceph-mon[92422]: pgmap v2800: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 121 KiB/s wr, 8 op/s 2023-03-09T16:09:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:09:38 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 4288.217700 secs 2023-03-09T16:09:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4750.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:09:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:09:40 smithi122 ceph-mon[93869]: pgmap v2801: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 99 KiB/s wr, 7 op/s 2023-03-09T16:09:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:09:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:09:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:09:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:09:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:09:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4290.995504 secs 2023-03-09T16:09:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:09:40 smithi181 ceph-mon[94189]: pgmap v2801: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 99 KiB/s wr, 7 op/s 2023-03-09T16:09:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:09:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:09:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:09:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:09:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:09:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4290.995504 secs 2023-03-09T16:09:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:09:40 smithi038 ceph-mon[92422]: pgmap v2801: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 99 KiB/s wr, 7 op/s 2023-03-09T16:09:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:09:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:09:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:09:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:09:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:09:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4290.995504 secs 2023-03-09T16:09:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:09:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:09:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:09:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:09:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:09:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:09:42.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:09:42 smithi122 ceph-mon[93869]: pgmap v2802: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 99 KiB/s wr, 7 op/s 2023-03-09T16:09:42.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:09:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:09:42.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:09:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:09:42.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:09:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:09:42.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:09:42 smithi181 ceph-mon[94189]: pgmap v2802: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 99 KiB/s wr, 7 op/s 2023-03-09T16:09:42.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:09:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:09:42.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:09:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:09:42.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:09:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:09:42.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:09:42 smithi038 ceph-mon[92422]: pgmap v2802: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 99 KiB/s wr, 7 op/s 2023-03-09T16:09:42.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:09:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:09:42.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:09:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:09:42.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:09:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:09:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:09:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4293.217807 secs 2023-03-09T16:09:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:09:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4293.217807 secs 2023-03-09T16:09:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:09:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4293.217807 secs 2023-03-09T16:09:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4755.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:09:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:09:44 smithi122 ceph-mon[93869]: pgmap v2803: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 155 KiB/s wr, 10 op/s 2023-03-09T16:09:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:09:44 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:09:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:09:44 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:09:44.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:09:44 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:09:44.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:09:44 smithi181 ceph-mon[94189]: pgmap v2803: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 155 KiB/s wr, 10 op/s 2023-03-09T16:09:44.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:09:44 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:09:44.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:09:44 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:09:44.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:09:44 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:09:44.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:09:44 smithi038 ceph-mon[92422]: pgmap v2803: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 155 KiB/s wr, 10 op/s 2023-03-09T16:09:44.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:09:44 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:09:44.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:09:44 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:09:44.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:09:44 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:09:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:09:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4295.995564 secs 2023-03-09T16:09:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:09:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4295.995564 secs 2023-03-09T16:09:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:09:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4295.995564 secs 2023-03-09T16:09:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:09:46 smithi122 ceph-mon[93869]: pgmap v2804: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 181 KiB/s wr, 9 op/s 2023-03-09T16:09:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:09:46 smithi181 ceph-mon[94189]: pgmap v2804: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 181 KiB/s wr, 9 op/s 2023-03-09T16:09:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:09:46 smithi038 ceph-mon[92422]: pgmap v2804: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 181 KiB/s wr, 9 op/s 2023-03-09T16:09:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:09:47 smithi122 ceph-mon[93869]: pgmap v2805: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 182 KiB/s wr, 12 op/s 2023-03-09T16:09:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:09:47 smithi181 ceph-mon[94189]: pgmap v2805: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 182 KiB/s wr, 12 op/s 2023-03-09T16:09:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:09:47 smithi038 ceph-mon[92422]: pgmap v2805: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 182 KiB/s wr, 12 op/s 2023-03-09T16:09:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:09:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4298.217940 secs 2023-03-09T16:09:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:09:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4298.217940 secs 2023-03-09T16:09:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:09:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4298.217940 secs 2023-03-09T16:09:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4760.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:09:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:09:50 smithi122 ceph-mon[93869]: pgmap v2806: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 144 KiB/s wr, 11 op/s 2023-03-09T16:09:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:09:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4300.995625 secs 2023-03-09T16:09:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:09:50 smithi181 ceph-mon[94189]: pgmap v2806: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 144 KiB/s wr, 11 op/s 2023-03-09T16:09:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:09:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4300.995625 secs 2023-03-09T16:09:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:09:50 smithi038 ceph-mon[92422]: pgmap v2806: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 144 KiB/s wr, 11 op/s 2023-03-09T16:09:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:09:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4300.995625 secs 2023-03-09T16:09:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:09:52 smithi122 ceph-mon[93869]: pgmap v2807: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 140 KiB/s wr, 10 op/s 2023-03-09T16:09:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:09:52 smithi181 ceph-mon[94189]: pgmap v2807: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 140 KiB/s wr, 10 op/s 2023-03-09T16:09:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:09:52 smithi038 ceph-mon[92422]: pgmap v2807: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 140 KiB/s wr, 10 op/s 2023-03-09T16:09:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:09:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4303.218053 secs 2023-03-09T16:09:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:09:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4303.218053 secs 2023-03-09T16:09:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:09:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4303.218053 secs 2023-03-09T16:09:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4765.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:09:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:09:54 smithi122 ceph-mon[93869]: pgmap v2808: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 148 KiB/s wr, 14 op/s 2023-03-09T16:09:54.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:09:54 smithi181 ceph-mon[94189]: pgmap v2808: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 148 KiB/s wr, 14 op/s 2023-03-09T16:09:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:09:54 smithi038 ceph-mon[92422]: pgmap v2808: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 148 KiB/s wr, 14 op/s 2023-03-09T16:09:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:09:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4305.995686 secs 2023-03-09T16:09:55.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:09:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4305.995686 secs 2023-03-09T16:09:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:09:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4305.995686 secs 2023-03-09T16:09:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:09:56 smithi181 ceph-mon[94189]: pgmap v2809: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 172 KiB/s wr, 11 op/s 2023-03-09T16:09:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:09:56 smithi038 ceph-mon[92422]: pgmap v2809: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 172 KiB/s wr, 11 op/s 2023-03-09T16:09:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:09:56 smithi122 ceph-mon[93869]: pgmap v2809: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 172 KiB/s wr, 11 op/s 2023-03-09T16:09:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:09:58 smithi038 ceph-mon[92422]: pgmap v2810: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 116 KiB/s wr, 12 op/s 2023-03-09T16:09:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:09:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4308.218194 secs 2023-03-09T16:09:58.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:09:58 smithi122 ceph-mon[93869]: pgmap v2810: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 116 KiB/s wr, 12 op/s 2023-03-09T16:09:58.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:09:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4308.218194 secs 2023-03-09T16:09:58.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:09:58 smithi181 ceph-mon[94189]: pgmap v2810: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 116 KiB/s wr, 12 op/s 2023-03-09T16:09:58.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:09:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4308.218194 secs 2023-03-09T16:09:58.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4770.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:10:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:10:00 smithi038 ceph-mon[92422]: pgmap v2811: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 127 KiB/s wr, 11 op/s 2023-03-09T16:10:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:10:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4310.995745 secs 2023-03-09T16:10:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:10:00 smithi038 ceph-mon[92422]: overall HEALTH_WARN 2 MDSs report slow requests 2023-03-09T16:10:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:10:00 smithi122 ceph-mon[93869]: pgmap v2811: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 127 KiB/s wr, 11 op/s 2023-03-09T16:10:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:10:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4310.995745 secs 2023-03-09T16:10:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:10:00 smithi122 ceph-mon[93869]: overall HEALTH_WARN 2 MDSs report slow requests 2023-03-09T16:10:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:10:00 smithi181 ceph-mon[94189]: pgmap v2811: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 127 KiB/s wr, 11 op/s 2023-03-09T16:10:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:10:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4310.995745 secs 2023-03-09T16:10:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:10:00 smithi181 ceph-mon[94189]: overall HEALTH_WARN 2 MDSs report slow requests 2023-03-09T16:10:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:10:02 smithi038 ceph-mon[92422]: pgmap v2812: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 99 KiB/s wr, 8 op/s 2023-03-09T16:10:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:10:02 smithi122 ceph-mon[93869]: pgmap v2812: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 99 KiB/s wr, 8 op/s 2023-03-09T16:10:02.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:10:02 smithi181 ceph-mon[94189]: pgmap v2812: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 99 KiB/s wr, 8 op/s 2023-03-09T16:10:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:10:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4313.218314 secs 2023-03-09T16:10:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:10:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4313.218314 secs 2023-03-09T16:10:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:10:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4313.218314 secs 2023-03-09T16:10:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4775.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:10:04.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:10:04 smithi122 ceph-mon[93869]: pgmap v2813: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 122 KiB/s wr, 13 op/s 2023-03-09T16:10:04.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:10:04 smithi181 ceph-mon[94189]: pgmap v2813: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 122 KiB/s wr, 13 op/s 2023-03-09T16:10:04.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:10:04 smithi038 ceph-mon[92422]: pgmap v2813: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 122 KiB/s wr, 13 op/s 2023-03-09T16:10:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:10:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4315.995806 secs 2023-03-09T16:10:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:10:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4315.995806 secs 2023-03-09T16:10:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:10:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4315.995806 secs 2023-03-09T16:10:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:10:06 smithi122 ceph-mon[93869]: pgmap v2814: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 163 KiB/s wr, 22 op/s 2023-03-09T16:10:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:10:06 smithi181 ceph-mon[94189]: pgmap v2814: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 163 KiB/s wr, 22 op/s 2023-03-09T16:10:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:10:06 smithi038 ceph-mon[92422]: pgmap v2814: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 163 KiB/s wr, 22 op/s 2023-03-09T16:10:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:10:08 smithi122 ceph-mon[93869]: pgmap v2815: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 83 KiB/s wr, 22 op/s 2023-03-09T16:10:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:10:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4318.218408 secs 2023-03-09T16:10:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:10:08 smithi181 ceph-mon[94189]: pgmap v2815: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 83 KiB/s wr, 22 op/s 2023-03-09T16:10:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:10:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4318.218408 secs 2023-03-09T16:10:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:10:08 smithi038 ceph-mon[92422]: pgmap v2815: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 83 KiB/s wr, 22 op/s 2023-03-09T16:10:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:10:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4318.218408 secs 2023-03-09T16:10:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4780.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:10:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:10:10 smithi122 ceph-mon[93869]: pgmap v2816: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 83 KiB/s wr, 21 op/s 2023-03-09T16:10:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:10:10 smithi122 ceph-mon[93869]: osdmap e388: 12 total, 12 up, 12 in 2023-03-09T16:10:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:10:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4320.995867 secs 2023-03-09T16:10:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:10:10 smithi181 ceph-mon[94189]: pgmap v2816: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 83 KiB/s wr, 21 op/s 2023-03-09T16:10:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:10:10 smithi181 ceph-mon[94189]: osdmap e388: 12 total, 12 up, 12 in 2023-03-09T16:10:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:10:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4320.995867 secs 2023-03-09T16:10:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:10:10 smithi038 ceph-mon[92422]: pgmap v2816: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 83 KiB/s wr, 21 op/s 2023-03-09T16:10:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:10:10 smithi038 ceph-mon[92422]: osdmap e388: 12 total, 12 up, 12 in 2023-03-09T16:10:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:10:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4320.995867 secs 2023-03-09T16:10:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:10:12 smithi122 ceph-mon[93869]: pgmap v2818: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 86 KiB/s wr, 24 op/s 2023-03-09T16:10:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:10:12 smithi181 ceph-mon[94189]: pgmap v2818: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 86 KiB/s wr, 24 op/s 2023-03-09T16:10:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:10:12 smithi038 ceph-mon[92422]: pgmap v2818: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 86 KiB/s wr, 24 op/s 2023-03-09T16:10:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:10:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4323.218533 secs 2023-03-09T16:10:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:10:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4323.218533 secs 2023-03-09T16:10:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:10:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4323.218533 secs 2023-03-09T16:10:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4785.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:10:14.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:10:14 smithi122 ceph-mon[93869]: pgmap v2819: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 59 KiB/s wr, 21 op/s 2023-03-09T16:10:14.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:10:14 smithi181 ceph-mon[94189]: pgmap v2819: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 59 KiB/s wr, 21 op/s 2023-03-09T16:10:14.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:10:14 smithi038 ceph-mon[92422]: pgmap v2819: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 59 KiB/s wr, 21 op/s 2023-03-09T16:10:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:10:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4325.995928 secs 2023-03-09T16:10:15.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:10:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4325.995928 secs 2023-03-09T16:10:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:10:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4325.995928 secs 2023-03-09T16:10:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:10:16 smithi122 ceph-mon[93869]: pgmap v2820: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T16:10:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:10:16 smithi181 ceph-mon[94189]: pgmap v2820: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T16:10:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:10:16 smithi038 ceph-mon[92422]: pgmap v2820: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T16:10:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:10:18 smithi122 ceph-mon[93869]: pgmap v2821: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2023-03-09T16:10:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:10:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4328.218672 secs 2023-03-09T16:10:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:10:18 smithi181 ceph-mon[94189]: pgmap v2821: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2023-03-09T16:10:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:10:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4328.218672 secs 2023-03-09T16:10:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:10:18 smithi038 ceph-mon[92422]: pgmap v2821: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2023-03-09T16:10:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:10:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4328.218672 secs 2023-03-09T16:10:18.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4790.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:10:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:10:20 smithi122 ceph-mon[93869]: pgmap v2822: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2023-03-09T16:10:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:10:20 smithi122 ceph-mon[93869]: osdmap e389: 12 total, 12 up, 12 in 2023-03-09T16:10:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:10:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4330.995985 secs 2023-03-09T16:10:20.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:10:20 smithi181 ceph-mon[94189]: pgmap v2822: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2023-03-09T16:10:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:10:20 smithi181 ceph-mon[94189]: osdmap e389: 12 total, 12 up, 12 in 2023-03-09T16:10:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:10:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4330.995985 secs 2023-03-09T16:10:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:10:20 smithi038 ceph-mon[92422]: pgmap v2822: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2023-03-09T16:10:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:10:20 smithi038 ceph-mon[92422]: osdmap e389: 12 total, 12 up, 12 in 2023-03-09T16:10:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:10:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4330.995985 secs 2023-03-09T16:10:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:10:22 smithi122 ceph-mon[93869]: pgmap v2824: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2023-03-09T16:10:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:10:22 smithi181 ceph-mon[94189]: pgmap v2824: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2023-03-09T16:10:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:10:22 smithi038 ceph-mon[92422]: pgmap v2824: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2023-03-09T16:10:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:10:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4333.218821 secs 2023-03-09T16:10:23.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:10:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4333.218821 secs 2023-03-09T16:10:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:10:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4333.218821 secs 2023-03-09T16:10:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4795.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:10:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:10:24 smithi122 ceph-mon[93869]: pgmap v2825: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2023-03-09T16:10:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:10:24 smithi181 ceph-mon[94189]: pgmap v2825: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2023-03-09T16:10:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:10:24 smithi038 ceph-mon[92422]: pgmap v2825: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2023-03-09T16:10:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:10:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4335.996069 secs 2023-03-09T16:10:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:10:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4335.996069 secs 2023-03-09T16:10:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:10:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4335.996069 secs 2023-03-09T16:10:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:10:26 smithi038 ceph-mon[92422]: pgmap v2826: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 6 op/s 2023-03-09T16:10:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:10:26 smithi122 ceph-mon[93869]: pgmap v2826: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 6 op/s 2023-03-09T16:10:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:10:26 smithi181 ceph-mon[94189]: pgmap v2826: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 6 op/s 2023-03-09T16:10:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:10:28 smithi038 ceph-mon[92422]: pgmap v2827: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 6 op/s 2023-03-09T16:10:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:10:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4338.218939 secs 2023-03-09T16:10:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:10:28 smithi122 ceph-mon[93869]: pgmap v2827: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 6 op/s 2023-03-09T16:10:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:10:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4338.218939 secs 2023-03-09T16:10:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:10:28 smithi181 ceph-mon[94189]: pgmap v2827: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 6 op/s 2023-03-09T16:10:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:10:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4338.218939 secs 2023-03-09T16:10:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4800.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:10:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:10:30 smithi038 ceph-mon[92422]: pgmap v2828: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 60 KiB/s wr, 7 op/s 2023-03-09T16:10:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:10:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4340.996129 secs 2023-03-09T16:10:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:10:30 smithi122 ceph-mon[93869]: pgmap v2828: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 60 KiB/s wr, 7 op/s 2023-03-09T16:10:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:10:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4340.996129 secs 2023-03-09T16:10:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:10:30 smithi181 ceph-mon[94189]: pgmap v2828: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 60 KiB/s wr, 7 op/s 2023-03-09T16:10:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:10:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4340.996129 secs 2023-03-09T16:10:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:10:32 smithi038 ceph-mon[92422]: pgmap v2829: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 52 KiB/s wr, 6 op/s 2023-03-09T16:10:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:10:32 smithi122 ceph-mon[93869]: pgmap v2829: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 52 KiB/s wr, 6 op/s 2023-03-09T16:10:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:10:32 smithi181 ceph-mon[94189]: pgmap v2829: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 52 KiB/s wr, 6 op/s 2023-03-09T16:10:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:10:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4343.219021 secs 2023-03-09T16:10:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:10:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4343.219021 secs 2023-03-09T16:10:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:10:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4343.219021 secs 2023-03-09T16:10:33.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4805.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:10:34.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:10:34 smithi122 ceph-mon[93869]: pgmap v2830: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 50 KiB/s wr, 8 op/s 2023-03-09T16:10:34.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:10:34 smithi181 ceph-mon[94189]: pgmap v2830: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 50 KiB/s wr, 8 op/s 2023-03-09T16:10:34.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:10:34 smithi038 ceph-mon[92422]: pgmap v2830: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 50 KiB/s wr, 8 op/s 2023-03-09T16:10:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:10:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4345.996189 secs 2023-03-09T16:10:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:10:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4345.996189 secs 2023-03-09T16:10:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:10:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4345.996189 secs 2023-03-09T16:10:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:10:36 smithi122 ceph-mon[93869]: pgmap v2831: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 80 KiB/s wr, 7 op/s 2023-03-09T16:10:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:10:36 smithi181 ceph-mon[94189]: pgmap v2831: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 80 KiB/s wr, 7 op/s 2023-03-09T16:10:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:10:36 smithi038 ceph-mon[92422]: pgmap v2831: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 80 KiB/s wr, 7 op/s 2023-03-09T16:10:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:10:38 smithi122 ceph-mon[93869]: pgmap v2832: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 8 op/s 2023-03-09T16:10:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:10:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4348.219140 secs 2023-03-09T16:10:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:10:38 smithi181 ceph-mon[94189]: pgmap v2832: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 8 op/s 2023-03-09T16:10:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:10:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4348.219140 secs 2023-03-09T16:10:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:10:38 smithi038 ceph-mon[92422]: pgmap v2832: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 8 op/s 2023-03-09T16:10:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:10:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4348.219140 secs 2023-03-09T16:10:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4810.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:10:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:10:40 smithi122 ceph-mon[93869]: pgmap v2833: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 130 KiB/s wr, 8 op/s 2023-03-09T16:10:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:10:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4350.996251 secs 2023-03-09T16:10:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:10:40 smithi181 ceph-mon[94189]: pgmap v2833: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 130 KiB/s wr, 8 op/s 2023-03-09T16:10:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:10:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4350.996251 secs 2023-03-09T16:10:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:10:40 smithi038 ceph-mon[92422]: pgmap v2833: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 130 KiB/s wr, 8 op/s 2023-03-09T16:10:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:10:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4350.996251 secs 2023-03-09T16:10:42.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:10:42 smithi122 ceph-mon[93869]: pgmap v2834: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 103 KiB/s wr, 7 op/s 2023-03-09T16:10:42.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:10:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:10:42.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:10:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:10:42.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:10:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:10:42.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:10:42 smithi181 ceph-mon[94189]: pgmap v2834: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 103 KiB/s wr, 7 op/s 2023-03-09T16:10:42.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:10:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:10:42.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:10:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:10:42.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:10:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:10:42.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:10:42 smithi038 ceph-mon[92422]: pgmap v2834: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 103 KiB/s wr, 7 op/s 2023-03-09T16:10:42.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:10:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:10:42.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:10:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:10:42.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:10:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:10:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:10:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4353.219209 secs 2023-03-09T16:10:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:10:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:10:43.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:10:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:10:43.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:10:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:10:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:10:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4353.219209 secs 2023-03-09T16:10:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:10:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:10:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:10:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:10:43.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:10:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:10:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:10:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4353.219209 secs 2023-03-09T16:10:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:10:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:10:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:10:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:10:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:10:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:10:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4815.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:10:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:10:44 smithi122 ceph-mon[93869]: pgmap v2835: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 176 KiB/s wr, 12 op/s 2023-03-09T16:10:44.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:10:44 smithi181 ceph-mon[94189]: pgmap v2835: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 176 KiB/s wr, 12 op/s 2023-03-09T16:10:44.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:10:44 smithi038 ceph-mon[92422]: pgmap v2835: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 176 KiB/s wr, 12 op/s 2023-03-09T16:10:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:10:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4355.996312 secs 2023-03-09T16:10:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:10:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4355.996312 secs 2023-03-09T16:10:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:10:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4355.996312 secs 2023-03-09T16:10:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:10:46 smithi122 ceph-mon[93869]: pgmap v2836: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 175 KiB/s wr, 10 op/s 2023-03-09T16:10:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:10:46 smithi181 ceph-mon[94189]: pgmap v2836: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 175 KiB/s wr, 10 op/s 2023-03-09T16:10:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:10:46 smithi038 ceph-mon[92422]: pgmap v2836: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 175 KiB/s wr, 10 op/s 2023-03-09T16:10:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:10:48 smithi122 ceph-mon[93869]: pgmap v2837: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 206 KiB/s wr, 13 op/s 2023-03-09T16:10:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:10:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4358.219277 secs 2023-03-09T16:10:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:10:48 smithi181 ceph-mon[94189]: pgmap v2837: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 206 KiB/s wr, 13 op/s 2023-03-09T16:10:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:10:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4358.219277 secs 2023-03-09T16:10:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:10:48 smithi038 ceph-mon[92422]: pgmap v2837: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 206 KiB/s wr, 13 op/s 2023-03-09T16:10:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:10:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4358.219277 secs 2023-03-09T16:10:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4820.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:10:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:10:50 smithi122 ceph-mon[93869]: pgmap v2838: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 211 KiB/s wr, 13 op/s 2023-03-09T16:10:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:10:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4360.996371 secs 2023-03-09T16:10:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:10:50 smithi181 ceph-mon[94189]: pgmap v2838: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 211 KiB/s wr, 13 op/s 2023-03-09T16:10:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:10:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4360.996371 secs 2023-03-09T16:10:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:10:50 smithi038 ceph-mon[92422]: pgmap v2838: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 211 KiB/s wr, 13 op/s 2023-03-09T16:10:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:10:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4360.996371 secs 2023-03-09T16:10:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:10:52 smithi122 ceph-mon[93869]: pgmap v2839: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 139 KiB/s wr, 10 op/s 2023-03-09T16:10:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:10:52 smithi181 ceph-mon[94189]: pgmap v2839: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 139 KiB/s wr, 10 op/s 2023-03-09T16:10:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:10:52 smithi038 ceph-mon[92422]: pgmap v2839: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 139 KiB/s wr, 10 op/s 2023-03-09T16:10:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:10:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4363.219393 secs 2023-03-09T16:10:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:10:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4363.219393 secs 2023-03-09T16:10:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:10:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4363.219393 secs 2023-03-09T16:10:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4825.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:10:54.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:10:54 smithi122 ceph-mon[93869]: pgmap v2840: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 202 KiB/s wr, 14 op/s 2023-03-09T16:10:54.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:10:54 smithi181 ceph-mon[94189]: pgmap v2840: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 202 KiB/s wr, 14 op/s 2023-03-09T16:10:54.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:10:54 smithi038 ceph-mon[92422]: pgmap v2840: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 202 KiB/s wr, 14 op/s 2023-03-09T16:10:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:10:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4365.996435 secs 2023-03-09T16:10:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:10:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4365.996435 secs 2023-03-09T16:10:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:10:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4365.996435 secs 2023-03-09T16:10:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:10:56 smithi122 ceph-mon[93869]: pgmap v2841: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 129 KiB/s wr, 10 op/s 2023-03-09T16:10:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:10:56 smithi181 ceph-mon[94189]: pgmap v2841: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 129 KiB/s wr, 10 op/s 2023-03-09T16:10:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:10:56 smithi038 ceph-mon[92422]: pgmap v2841: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 129 KiB/s wr, 10 op/s 2023-03-09T16:10:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:10:57 smithi122 ceph-mon[93869]: pgmap v2842: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 175 KiB/s wr, 11 op/s 2023-03-09T16:10:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:10:57 smithi181 ceph-mon[94189]: pgmap v2842: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 175 KiB/s wr, 11 op/s 2023-03-09T16:10:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:10:57 smithi038 ceph-mon[92422]: pgmap v2842: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 175 KiB/s wr, 11 op/s 2023-03-09T16:10:58.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:10:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4368.219507 secs 2023-03-09T16:10:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:10:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4368.219507 secs 2023-03-09T16:10:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:10:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4368.219507 secs 2023-03-09T16:10:58.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4830.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:11:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:11:00 smithi181 ceph-mon[94189]: pgmap v2843: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 114 KiB/s wr, 9 op/s 2023-03-09T16:11:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:11:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4370.996496 secs 2023-03-09T16:11:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:11:00 smithi038 ceph-mon[92422]: pgmap v2843: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 114 KiB/s wr, 9 op/s 2023-03-09T16:11:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:11:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4370.996496 secs 2023-03-09T16:11:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:11:00 smithi122 ceph-mon[93869]: pgmap v2843: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 114 KiB/s wr, 9 op/s 2023-03-09T16:11:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:11:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4370.996496 secs 2023-03-09T16:11:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:11:02 smithi038 ceph-mon[92422]: pgmap v2844: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 109 KiB/s wr, 7 op/s 2023-03-09T16:11:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:11:02 smithi122 ceph-mon[93869]: pgmap v2844: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 109 KiB/s wr, 7 op/s 2023-03-09T16:11:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:11:02 smithi181 ceph-mon[94189]: pgmap v2844: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 109 KiB/s wr, 7 op/s 2023-03-09T16:11:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:11:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4373.219630 secs 2023-03-09T16:11:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:11:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4373.219630 secs 2023-03-09T16:11:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:11:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4373.219630 secs 2023-03-09T16:11:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4835.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:11:04.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:11:04 smithi122 ceph-mon[93869]: pgmap v2845: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 160 KiB/s wr, 13 op/s 2023-03-09T16:11:04.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:11:04 smithi181 ceph-mon[94189]: pgmap v2845: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 160 KiB/s wr, 13 op/s 2023-03-09T16:11:04.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:11:04 smithi038 ceph-mon[92422]: pgmap v2845: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 160 KiB/s wr, 13 op/s 2023-03-09T16:11:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:11:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4375.996555 secs 2023-03-09T16:11:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:11:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4375.996555 secs 2023-03-09T16:11:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:11:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4375.996555 secs 2023-03-09T16:11:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:11:06 smithi122 ceph-mon[93869]: pgmap v2846: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 118 KiB/s wr, 21 op/s 2023-03-09T16:11:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:11:06 smithi181 ceph-mon[94189]: pgmap v2846: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 118 KiB/s wr, 21 op/s 2023-03-09T16:11:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:11:06 smithi038 ceph-mon[92422]: pgmap v2846: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 118 KiB/s wr, 21 op/s 2023-03-09T16:11:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:11:08 smithi122 ceph-mon[93869]: pgmap v2847: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 118 KiB/s wr, 23 op/s 2023-03-09T16:11:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:11:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4378.219743 secs 2023-03-09T16:11:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:11:08 smithi181 ceph-mon[94189]: pgmap v2847: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 118 KiB/s wr, 23 op/s 2023-03-09T16:11:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:11:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4378.219743 secs 2023-03-09T16:11:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:11:08 smithi038 ceph-mon[92422]: pgmap v2847: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 118 KiB/s wr, 23 op/s 2023-03-09T16:11:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:11:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4378.219743 secs 2023-03-09T16:11:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4840.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:11:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:11:10 smithi122 ceph-mon[93869]: pgmap v2848: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 72 KiB/s wr, 21 op/s 2023-03-09T16:11:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:11:10 smithi122 ceph-mon[93869]: osdmap e390: 12 total, 12 up, 12 in 2023-03-09T16:11:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:11:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4380.996615 secs 2023-03-09T16:11:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:11:10 smithi181 ceph-mon[94189]: pgmap v2848: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 72 KiB/s wr, 21 op/s 2023-03-09T16:11:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:11:10 smithi181 ceph-mon[94189]: osdmap e390: 12 total, 12 up, 12 in 2023-03-09T16:11:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:11:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4380.996615 secs 2023-03-09T16:11:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:11:10 smithi038 ceph-mon[92422]: pgmap v2848: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 72 KiB/s wr, 21 op/s 2023-03-09T16:11:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:11:10 smithi038 ceph-mon[92422]: osdmap e390: 12 total, 12 up, 12 in 2023-03-09T16:11:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:11:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4380.996615 secs 2023-03-09T16:11:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:11:12 smithi122 ceph-mon[93869]: pgmap v2850: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 87 KiB/s wr, 24 op/s 2023-03-09T16:11:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:11:12 smithi181 ceph-mon[94189]: pgmap v2850: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 87 KiB/s wr, 24 op/s 2023-03-09T16:11:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:11:12 smithi038 ceph-mon[92422]: pgmap v2850: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 87 KiB/s wr, 24 op/s 2023-03-09T16:11:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:11:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4383.219893 secs 2023-03-09T16:11:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:11:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4383.219893 secs 2023-03-09T16:11:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:11:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4383.219893 secs 2023-03-09T16:11:13.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4845.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:11:14.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:11:14 smithi122 ceph-mon[93869]: pgmap v2851: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 48 KiB/s wr, 20 op/s 2023-03-09T16:11:14.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:11:14 smithi181 ceph-mon[94189]: pgmap v2851: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 48 KiB/s wr, 20 op/s 2023-03-09T16:11:14.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:11:14 smithi038 ceph-mon[92422]: pgmap v2851: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 48 KiB/s wr, 20 op/s 2023-03-09T16:11:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:11:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4385.996675 secs 2023-03-09T16:11:15.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:11:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4385.996675 secs 2023-03-09T16:11:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:11:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4385.996675 secs 2023-03-09T16:11:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:11:16 smithi122 ceph-mon[93869]: pgmap v2852: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 23 KiB/s wr, 6 op/s 2023-03-09T16:11:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:11:16 smithi181 ceph-mon[94189]: pgmap v2852: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 23 KiB/s wr, 6 op/s 2023-03-09T16:11:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:11:16 smithi038 ceph-mon[92422]: pgmap v2852: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 23 KiB/s wr, 6 op/s 2023-03-09T16:11:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:11:18 smithi122 ceph-mon[93869]: pgmap v2853: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 6 op/s 2023-03-09T16:11:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:11:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4388.220014 secs 2023-03-09T16:11:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:11:18 smithi181 ceph-mon[94189]: pgmap v2853: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 6 op/s 2023-03-09T16:11:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:11:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4388.220014 secs 2023-03-09T16:11:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:11:18 smithi038 ceph-mon[92422]: pgmap v2853: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 6 op/s 2023-03-09T16:11:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:11:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4388.220014 secs 2023-03-09T16:11:18.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4850.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:11:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:11:20 smithi038 ceph-mon[92422]: pgmap v2854: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 6 op/s 2023-03-09T16:11:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:11:20 smithi038 ceph-mon[92422]: osdmap e391: 12 total, 12 up, 12 in 2023-03-09T16:11:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:11:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4390.996745 secs 2023-03-09T16:11:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:11:20 smithi122 ceph-mon[93869]: pgmap v2854: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 6 op/s 2023-03-09T16:11:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:11:20 smithi122 ceph-mon[93869]: osdmap e391: 12 total, 12 up, 12 in 2023-03-09T16:11:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:11:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4390.996745 secs 2023-03-09T16:11:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:11:20 smithi181 ceph-mon[94189]: pgmap v2854: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 6 op/s 2023-03-09T16:11:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:11:20 smithi181 ceph-mon[94189]: osdmap e391: 12 total, 12 up, 12 in 2023-03-09T16:11:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:11:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4390.996745 secs 2023-03-09T16:11:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:11:22 smithi038 ceph-mon[92422]: pgmap v2856: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 6 op/s 2023-03-09T16:11:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:11:22 smithi122 ceph-mon[93869]: pgmap v2856: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 6 op/s 2023-03-09T16:11:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:11:22 smithi181 ceph-mon[94189]: pgmap v2856: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 6 op/s 2023-03-09T16:11:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:11:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4393.220141 secs 2023-03-09T16:11:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:11:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4393.220141 secs 2023-03-09T16:11:23.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:11:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4393.220141 secs 2023-03-09T16:11:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4855.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:11:24.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:11:24 smithi122 ceph-mon[93869]: pgmap v2857: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T16:11:24.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:11:24 smithi181 ceph-mon[94189]: pgmap v2857: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T16:11:24.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:11:24 smithi038 ceph-mon[92422]: pgmap v2857: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T16:11:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:11:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4395.996808 secs 2023-03-09T16:11:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:11:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4395.996808 secs 2023-03-09T16:11:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:11:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4395.996808 secs 2023-03-09T16:11:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:11:26 smithi122 ceph-mon[93869]: pgmap v2858: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T16:11:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:11:26 smithi181 ceph-mon[94189]: pgmap v2858: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T16:11:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:11:26 smithi038 ceph-mon[92422]: pgmap v2858: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T16:11:28.732 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:11:28 smithi122 ceph-mon[93869]: pgmap v2859: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T16:11:28.732 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:11:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4398.220273 secs 2023-03-09T16:11:28.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:11:28 smithi181 ceph-mon[94189]: pgmap v2859: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T16:11:28.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:11:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4398.220273 secs 2023-03-09T16:11:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:11:28 smithi038 ceph-mon[92422]: pgmap v2859: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T16:11:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:11:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4398.220273 secs 2023-03-09T16:11:28.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4860.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:11:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:11:30 smithi122 ceph-mon[93869]: pgmap v2860: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 614 B/s wr, 7 op/s 2023-03-09T16:11:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:11:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4400.996889 secs 2023-03-09T16:11:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:11:30 smithi181 ceph-mon[94189]: pgmap v2860: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 614 B/s wr, 7 op/s 2023-03-09T16:11:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:11:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4400.996889 secs 2023-03-09T16:11:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:11:30 smithi038 ceph-mon[92422]: pgmap v2860: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 614 B/s wr, 7 op/s 2023-03-09T16:11:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:11:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4400.996889 secs 2023-03-09T16:11:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:11:32 smithi122 ceph-mon[93869]: pgmap v2861: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 540 B/s wr, 6 op/s 2023-03-09T16:11:32.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:11:32 smithi181 ceph-mon[94189]: pgmap v2861: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 540 B/s wr, 6 op/s 2023-03-09T16:11:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:11:32 smithi038 ceph-mon[92422]: pgmap v2861: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 540 B/s wr, 6 op/s 2023-03-09T16:11:33.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4865.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:11:34.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:11:34 smithi122 ceph-mon[93869]: pgmap v2862: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 9 op/s 2023-03-09T16:11:34.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:11:34 smithi181 ceph-mon[94189]: pgmap v2862: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 9 op/s 2023-03-09T16:11:34.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:11:34 smithi038 ceph-mon[92422]: pgmap v2862: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 9 op/s 2023-03-09T16:11:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:11:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4405.996948 secs 2023-03-09T16:11:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:11:35 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:11:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:11:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4405.996948 secs 2023-03-09T16:11:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:11:35 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:11:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:11:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4405.996948 secs 2023-03-09T16:11:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:11:35 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:11:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:11:36 smithi122 ceph-mon[93869]: pgmap v2863: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 58 KiB/s wr, 6 op/s 2023-03-09T16:11:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:11:36 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:11:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:11:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:11:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:11:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:11:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:11:36 smithi181 ceph-mon[94189]: pgmap v2863: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 58 KiB/s wr, 6 op/s 2023-03-09T16:11:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:11:36 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:11:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:11:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:11:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:11:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:11:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:11:36 smithi038 ceph-mon[92422]: pgmap v2863: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 58 KiB/s wr, 6 op/s 2023-03-09T16:11:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:11:36 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:11:36.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:11:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:11:36.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:11:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:11:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:11:38 smithi122 ceph-mon[93869]: pgmap v2864: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 124 KiB/s wr, 9 op/s 2023-03-09T16:11:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:11:38 smithi181 ceph-mon[94189]: pgmap v2864: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 124 KiB/s wr, 9 op/s 2023-03-09T16:11:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:11:38 smithi038 ceph-mon[92422]: pgmap v2864: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 124 KiB/s wr, 9 op/s 2023-03-09T16:11:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4870.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:11:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:11:40 smithi122 ceph-mon[93869]: pgmap v2865: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 129 KiB/s wr, 8 op/s 2023-03-09T16:11:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:11:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4410.996986 secs 2023-03-09T16:11:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:11:40 smithi181 ceph-mon[94189]: pgmap v2865: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 129 KiB/s wr, 8 op/s 2023-03-09T16:11:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:11:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4410.996986 secs 2023-03-09T16:11:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:11:40 smithi038 ceph-mon[92422]: pgmap v2865: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 129 KiB/s wr, 8 op/s 2023-03-09T16:11:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:11:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4410.996986 secs 2023-03-09T16:11:42.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:11:42 smithi122 ceph-mon[93869]: pgmap v2866: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 129 KiB/s wr, 8 op/s 2023-03-09T16:11:42.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:11:42 smithi181 ceph-mon[94189]: pgmap v2866: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 129 KiB/s wr, 8 op/s 2023-03-09T16:11:42.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:11:42 smithi038 ceph-mon[92422]: pgmap v2866: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 129 KiB/s wr, 8 op/s 2023-03-09T16:11:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:11:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:11:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:11:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:11:43.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:11:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:11:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4875.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:11:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:11:44 smithi122 ceph-mon[93869]: pgmap v2867: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 204 KiB/s wr, 12 op/s 2023-03-09T16:11:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:11:44 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:11:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:11:44 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:11:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:11:44 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:11:44.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:11:44 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:11:44.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:11:44 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:11:44.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:11:44 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:11:44.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:11:44 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:11:44.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:11:44 smithi181 ceph-mon[94189]: pgmap v2867: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 204 KiB/s wr, 12 op/s 2023-03-09T16:11:44.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:11:44 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:11:44.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:11:44 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:11:44.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:11:44 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:11:44.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:11:44 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:11:44.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:11:44 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:11:44.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:11:44 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:11:44.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:11:44 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:11:44.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:11:44 smithi038 ceph-mon[92422]: pgmap v2867: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 204 KiB/s wr, 12 op/s 2023-03-09T16:11:44.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:11:44 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:11:44.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:11:44 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:11:44.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:11:44 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:11:44.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:11:44 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:11:44.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:11:44 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:11:44.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:11:44 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:11:44.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:11:44 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:11:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:11:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4415.997045 secs 2023-03-09T16:11:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:11:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4415.997045 secs 2023-03-09T16:11:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:11:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4415.997045 secs 2023-03-09T16:11:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:11:46 smithi122 ceph-mon[93869]: pgmap v2868: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 147 KiB/s wr, 10 op/s 2023-03-09T16:11:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:11:46 smithi181 ceph-mon[94189]: pgmap v2868: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 147 KiB/s wr, 10 op/s 2023-03-09T16:11:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:11:46 smithi038 ceph-mon[92422]: pgmap v2868: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 147 KiB/s wr, 10 op/s 2023-03-09T16:11:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:11:48 smithi122 ceph-mon[93869]: pgmap v2869: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 209 KiB/s wr, 13 op/s 2023-03-09T16:11:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:11:48 smithi181 ceph-mon[94189]: pgmap v2869: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 209 KiB/s wr, 13 op/s 2023-03-09T16:11:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:11:48 smithi038 ceph-mon[92422]: pgmap v2869: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 209 KiB/s wr, 13 op/s 2023-03-09T16:11:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4880.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:11:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:11:50 smithi122 ceph-mon[93869]: pgmap v2870: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 148 KiB/s wr, 11 op/s 2023-03-09T16:11:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:11:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4420.997105 secs 2023-03-09T16:11:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:11:50 smithi181 ceph-mon[94189]: pgmap v2870: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 148 KiB/s wr, 11 op/s 2023-03-09T16:11:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:11:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4420.997105 secs 2023-03-09T16:11:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:11:50 smithi038 ceph-mon[92422]: pgmap v2870: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 148 KiB/s wr, 11 op/s 2023-03-09T16:11:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:11:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4420.997105 secs 2023-03-09T16:11:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:11:51 smithi122 ceph-mon[93869]: pgmap v2871: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 143 KiB/s wr, 10 op/s 2023-03-09T16:11:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:11:51 smithi181 ceph-mon[94189]: pgmap v2871: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 143 KiB/s wr, 10 op/s 2023-03-09T16:11:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:11:51 smithi038 ceph-mon[92422]: pgmap v2871: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 143 KiB/s wr, 10 op/s 2023-03-09T16:11:53.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4885.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:11:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:11:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4423.220890 secs 2023-03-09T16:11:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:11:53 smithi122 ceph-mon[93869]: pgmap v2872: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 197 KiB/s wr, 13 op/s 2023-03-09T16:11:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:11:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4423.220890 secs 2023-03-09T16:11:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:11:53 smithi181 ceph-mon[94189]: pgmap v2872: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 197 KiB/s wr, 13 op/s 2023-03-09T16:11:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:11:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4423.220890 secs 2023-03-09T16:11:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:11:53 smithi038 ceph-mon[92422]: pgmap v2872: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 197 KiB/s wr, 13 op/s 2023-03-09T16:11:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:11:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4425.997167 secs 2023-03-09T16:11:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:11:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4425.997167 secs 2023-03-09T16:11:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:11:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4425.997167 secs 2023-03-09T16:11:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:11:56 smithi122 ceph-mon[93869]: pgmap v2873: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 122 KiB/s wr, 9 op/s 2023-03-09T16:11:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:11:56 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:11:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:11:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:11:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:11:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:11:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:11:56 smithi181 ceph-mon[94189]: pgmap v2873: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 122 KiB/s wr, 9 op/s 2023-03-09T16:11:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:11:56 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:11:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:11:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:11:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:11:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:11:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:11:56 smithi038 ceph-mon[92422]: pgmap v2873: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 122 KiB/s wr, 9 op/s 2023-03-09T16:11:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:11:56 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:11:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:11:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:11:56.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:11:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:11:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:11:58 smithi122 ceph-mon[93869]: pgmap v2874: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 168 KiB/s wr, 11 op/s 2023-03-09T16:11:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:11:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4428.220999 secs 2023-03-09T16:11:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:11:58 smithi181 ceph-mon[94189]: pgmap v2874: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 168 KiB/s wr, 11 op/s 2023-03-09T16:11:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:11:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4428.220999 secs 2023-03-09T16:11:58.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:11:58 smithi038 ceph-mon[92422]: pgmap v2874: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 168 KiB/s wr, 11 op/s 2023-03-09T16:11:58.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:11:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4428.220999 secs 2023-03-09T16:11:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4890.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:12:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:12:00 smithi122 ceph-mon[93869]: pgmap v2875: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 106 KiB/s wr, 9 op/s 2023-03-09T16:12:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:12:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4430.997228 secs 2023-03-09T16:12:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:12:00 smithi181 ceph-mon[94189]: pgmap v2875: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 106 KiB/s wr, 9 op/s 2023-03-09T16:12:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:12:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4430.997228 secs 2023-03-09T16:12:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:12:00 smithi038 ceph-mon[92422]: pgmap v2875: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 106 KiB/s wr, 9 op/s 2023-03-09T16:12:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:12:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4430.997228 secs 2023-03-09T16:12:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:12:02 smithi122 ceph-mon[93869]: pgmap v2876: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 101 KiB/s wr, 8 op/s 2023-03-09T16:12:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:12:02 smithi181 ceph-mon[94189]: pgmap v2876: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 101 KiB/s wr, 8 op/s 2023-03-09T16:12:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:12:02 smithi038 ceph-mon[92422]: pgmap v2876: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 101 KiB/s wr, 8 op/s 2023-03-09T16:12:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:12:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4433.221134 secs 2023-03-09T16:12:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:12:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4433.221134 secs 2023-03-09T16:12:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:12:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4433.221134 secs 2023-03-09T16:12:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4895.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:12:04.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:12:04 smithi122 ceph-mon[93869]: pgmap v2877: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 152 KiB/s wr, 13 op/s 2023-03-09T16:12:04.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:12:04 smithi181 ceph-mon[94189]: pgmap v2877: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 152 KiB/s wr, 13 op/s 2023-03-09T16:12:04.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:12:04 smithi038 ceph-mon[92422]: pgmap v2877: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 152 KiB/s wr, 13 op/s 2023-03-09T16:12:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:12:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4435.997313 secs 2023-03-09T16:12:05.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:12:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4435.997313 secs 2023-03-09T16:12:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:12:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4435.997313 secs 2023-03-09T16:12:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:12:06 smithi122 ceph-mon[93869]: pgmap v2878: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 119 KiB/s wr, 21 op/s 2023-03-09T16:12:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:12:06 smithi181 ceph-mon[94189]: pgmap v2878: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 119 KiB/s wr, 21 op/s 2023-03-09T16:12:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:12:06 smithi038 ceph-mon[92422]: pgmap v2878: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 119 KiB/s wr, 21 op/s 2023-03-09T16:12:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:12:08 smithi122 ceph-mon[93869]: pgmap v2879: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 119 KiB/s wr, 23 op/s 2023-03-09T16:12:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:12:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4438.221259 secs 2023-03-09T16:12:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:12:08 smithi181 ceph-mon[94189]: pgmap v2879: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 119 KiB/s wr, 23 op/s 2023-03-09T16:12:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:12:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4438.221259 secs 2023-03-09T16:12:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:12:08 smithi038 ceph-mon[92422]: pgmap v2879: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 119 KiB/s wr, 23 op/s 2023-03-09T16:12:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:12:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4438.221259 secs 2023-03-09T16:12:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4900.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:12:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:12:10 smithi122 ceph-mon[93869]: pgmap v2880: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 73 KiB/s wr, 21 op/s 2023-03-09T16:12:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:12:10 smithi122 ceph-mon[93869]: osdmap e392: 12 total, 12 up, 12 in 2023-03-09T16:12:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:12:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4440.997376 secs 2023-03-09T16:12:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:12:10 smithi181 ceph-mon[94189]: pgmap v2880: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 73 KiB/s wr, 21 op/s 2023-03-09T16:12:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:12:10 smithi181 ceph-mon[94189]: osdmap e392: 12 total, 12 up, 12 in 2023-03-09T16:12:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:12:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4440.997376 secs 2023-03-09T16:12:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:12:10 smithi038 ceph-mon[92422]: pgmap v2880: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 73 KiB/s wr, 21 op/s 2023-03-09T16:12:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:12:10 smithi038 ceph-mon[92422]: osdmap e392: 12 total, 12 up, 12 in 2023-03-09T16:12:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:12:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4440.997376 secs 2023-03-09T16:12:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:12:12 smithi122 ceph-mon[93869]: pgmap v2882: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 87 KiB/s wr, 24 op/s 2023-03-09T16:12:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:12:12 smithi181 ceph-mon[94189]: pgmap v2882: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 87 KiB/s wr, 24 op/s 2023-03-09T16:12:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:12:12 smithi038 ceph-mon[92422]: pgmap v2882: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 87 KiB/s wr, 24 op/s 2023-03-09T16:12:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:12:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4443.221405 secs 2023-03-09T16:12:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:12:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4443.221405 secs 2023-03-09T16:12:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:12:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4443.221405 secs 2023-03-09T16:12:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4905.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:12:14.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:12:14 smithi122 ceph-mon[93869]: pgmap v2883: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 KiB/s wr, 20 op/s 2023-03-09T16:12:14.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:12:14 smithi181 ceph-mon[94189]: pgmap v2883: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 KiB/s wr, 20 op/s 2023-03-09T16:12:14.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:12:14 smithi038 ceph-mon[92422]: pgmap v2883: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 26 KiB/s wr, 20 op/s 2023-03-09T16:12:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:12:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4445.997437 secs 2023-03-09T16:12:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:12:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4445.997437 secs 2023-03-09T16:12:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:12:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4445.997437 secs 2023-03-09T16:12:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:12:16 smithi122 ceph-mon[93869]: pgmap v2884: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T16:12:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:12:16 smithi181 ceph-mon[94189]: pgmap v2884: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T16:12:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:12:16 smithi038 ceph-mon[92422]: pgmap v2884: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T16:12:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:12:18 smithi122 ceph-mon[93869]: pgmap v2885: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T16:12:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:12:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4448.221527 secs 2023-03-09T16:12:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:12:18 smithi181 ceph-mon[94189]: pgmap v2885: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T16:12:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:12:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4448.221527 secs 2023-03-09T16:12:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:12:18 smithi038 ceph-mon[92422]: pgmap v2885: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T16:12:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:12:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4448.221527 secs 2023-03-09T16:12:18.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4910.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:12:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:12:20 smithi038 ceph-mon[92422]: pgmap v2886: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T16:12:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:12:20 smithi038 ceph-mon[92422]: osdmap e393: 12 total, 12 up, 12 in 2023-03-09T16:12:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:12:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4450.997505 secs 2023-03-09T16:12:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:12:20 smithi122 ceph-mon[93869]: pgmap v2886: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T16:12:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:12:20 smithi122 ceph-mon[93869]: osdmap e393: 12 total, 12 up, 12 in 2023-03-09T16:12:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:12:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4450.997505 secs 2023-03-09T16:12:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:12:20 smithi181 ceph-mon[94189]: pgmap v2886: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T16:12:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:12:20 smithi181 ceph-mon[94189]: osdmap e393: 12 total, 12 up, 12 in 2023-03-09T16:12:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:12:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4450.997505 secs 2023-03-09T16:12:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:12:22 smithi122 ceph-mon[93869]: pgmap v2888: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 28 KiB/s wr, 8 op/s 2023-03-09T16:12:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:12:22 smithi181 ceph-mon[94189]: pgmap v2888: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 28 KiB/s wr, 8 op/s 2023-03-09T16:12:22.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:12:22 smithi038 ceph-mon[92422]: pgmap v2888: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 28 KiB/s wr, 8 op/s 2023-03-09T16:12:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:12:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4453.221682 secs 2023-03-09T16:12:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:12:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4453.221682 secs 2023-03-09T16:12:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:12:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4453.221682 secs 2023-03-09T16:12:23.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4915.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:12:24.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:12:24 smithi122 ceph-mon[93869]: pgmap v2889: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 28 KiB/s wr, 6 op/s 2023-03-09T16:12:24.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:12:24 smithi181 ceph-mon[94189]: pgmap v2889: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 28 KiB/s wr, 6 op/s 2023-03-09T16:12:24.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:12:24 smithi038 ceph-mon[92422]: pgmap v2889: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 28 KiB/s wr, 6 op/s 2023-03-09T16:12:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:12:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4455.997566 secs 2023-03-09T16:12:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:12:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4455.997566 secs 2023-03-09T16:12:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:12:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4455.997566 secs 2023-03-09T16:12:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:12:26 smithi122 ceph-mon[93869]: pgmap v2890: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 28 KiB/s wr, 6 op/s 2023-03-09T16:12:26.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:12:26 smithi181 ceph-mon[94189]: pgmap v2890: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 28 KiB/s wr, 6 op/s 2023-03-09T16:12:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:12:26 smithi038 ceph-mon[92422]: pgmap v2890: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 28 KiB/s wr, 6 op/s 2023-03-09T16:12:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:12:28 smithi122 ceph-mon[93869]: pgmap v2891: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 28 KiB/s wr, 6 op/s 2023-03-09T16:12:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:12:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4458.221792 secs 2023-03-09T16:12:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:12:28 smithi181 ceph-mon[94189]: pgmap v2891: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 28 KiB/s wr, 6 op/s 2023-03-09T16:12:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:12:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4458.221792 secs 2023-03-09T16:12:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:12:28 smithi038 ceph-mon[92422]: pgmap v2891: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 28 KiB/s wr, 6 op/s 2023-03-09T16:12:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:12:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4458.221792 secs 2023-03-09T16:12:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4920.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:12:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:12:30 smithi122 ceph-mon[93869]: pgmap v2892: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T16:12:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:12:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4460.997656 secs 2023-03-09T16:12:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:12:30 smithi181 ceph-mon[94189]: pgmap v2892: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T16:12:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:12:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4460.997656 secs 2023-03-09T16:12:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:12:30 smithi038 ceph-mon[92422]: pgmap v2892: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T16:12:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:12:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4460.997656 secs 2023-03-09T16:12:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:12:32 smithi122 ceph-mon[93869]: pgmap v2893: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 86 KiB/s wr, 8 op/s 2023-03-09T16:12:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:12:32 smithi181 ceph-mon[94189]: pgmap v2893: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 86 KiB/s wr, 8 op/s 2023-03-09T16:12:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:12:32 smithi038 ceph-mon[92422]: pgmap v2893: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 86 KiB/s wr, 8 op/s 2023-03-09T16:12:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:12:33 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 4463.221931 secs 2023-03-09T16:12:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:12:33 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 4463.221931 secs 2023-03-09T16:12:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:12:33 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 4463.221931 secs 2023-03-09T16:12:33.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4925.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:12:34.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:12:34 smithi122 ceph-mon[93869]: pgmap v2894: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 59 KiB/s wr, 8 op/s 2023-03-09T16:12:34.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:12:34 smithi181 ceph-mon[94189]: pgmap v2894: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 59 KiB/s wr, 8 op/s 2023-03-09T16:12:34.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:12:34 smithi038 ceph-mon[92422]: pgmap v2894: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 59 KiB/s wr, 8 op/s 2023-03-09T16:12:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:12:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4465.997732 secs 2023-03-09T16:12:35.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:12:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4465.997732 secs 2023-03-09T16:12:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:12:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4465.997732 secs 2023-03-09T16:12:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:12:36 smithi122 ceph-mon[93869]: pgmap v2895: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 7 op/s 2023-03-09T16:12:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:12:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:12:36.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:12:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:12:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:12:36 smithi181 ceph-mon[94189]: pgmap v2895: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 7 op/s 2023-03-09T16:12:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:12:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:12:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:12:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:12:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:12:36 smithi038 ceph-mon[92422]: pgmap v2895: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 7 op/s 2023-03-09T16:12:36.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:12:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:12:36.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:12:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:12:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:12:38 smithi122 ceph-mon[93869]: pgmap v2896: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 125 KiB/s wr, 10 op/s 2023-03-09T16:12:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:12:38 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 4468.222068 secs 2023-03-09T16:12:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:12:38 smithi181 ceph-mon[94189]: pgmap v2896: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 125 KiB/s wr, 10 op/s 2023-03-09T16:12:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:12:38 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 4468.222068 secs 2023-03-09T16:12:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:12:38 smithi038 ceph-mon[92422]: pgmap v2896: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 125 KiB/s wr, 10 op/s 2023-03-09T16:12:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:12:38 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 4468.222068 secs 2023-03-09T16:12:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4930.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:12:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:12:40 smithi122 ceph-mon[93869]: pgmap v2897: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 130 KiB/s wr, 9 op/s 2023-03-09T16:12:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:12:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4470.997792 secs 2023-03-09T16:12:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:12:40 smithi181 ceph-mon[94189]: pgmap v2897: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 130 KiB/s wr, 9 op/s 2023-03-09T16:12:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:12:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4470.997792 secs 2023-03-09T16:12:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:12:40 smithi038 ceph-mon[92422]: pgmap v2897: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 130 KiB/s wr, 9 op/s 2023-03-09T16:12:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:12:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4470.997792 secs 2023-03-09T16:12:42.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:12:42 smithi122 ceph-mon[93869]: pgmap v2898: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 200 KiB/s wr, 10 op/s 2023-03-09T16:12:42.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:12:42 smithi181 ceph-mon[94189]: pgmap v2898: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 200 KiB/s wr, 10 op/s 2023-03-09T16:12:42.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:12:42 smithi038 ceph-mon[92422]: pgmap v2898: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 200 KiB/s wr, 10 op/s 2023-03-09T16:12:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:12:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4473.222172 secs 2023-03-09T16:12:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:12:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4473.222172 secs 2023-03-09T16:12:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:12:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4473.222172 secs 2023-03-09T16:12:43.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4935.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:12:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:12:44 smithi122 ceph-mon[93869]: pgmap v2899: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 149 KiB/s wr, 10 op/s 2023-03-09T16:12:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:12:44 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:12:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:12:44 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:12:44.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:12:44 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:12:44.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:12:44 smithi181 ceph-mon[94189]: pgmap v2899: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 149 KiB/s wr, 10 op/s 2023-03-09T16:12:44.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:12:44 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:12:44.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:12:44 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:12:44.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:12:44 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:12:44.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:12:44 smithi038 ceph-mon[92422]: pgmap v2899: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 149 KiB/s wr, 10 op/s 2023-03-09T16:12:44.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:12:44 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:12:44.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:12:44 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:12:44.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:12:44 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:12:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:12:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4475.997887 secs 2023-03-09T16:12:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:12:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:12:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:12:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:12:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:12:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:12:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:12:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4475.997887 secs 2023-03-09T16:12:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:12:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:12:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:12:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:12:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:12:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:12:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:12:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4475.997887 secs 2023-03-09T16:12:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:12:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:12:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:12:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:12:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:12:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:12:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:12:46 smithi122 ceph-mon[93869]: pgmap v2900: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 148 KiB/s wr, 9 op/s 2023-03-09T16:12:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:12:46 smithi181 ceph-mon[94189]: pgmap v2900: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 148 KiB/s wr, 9 op/s 2023-03-09T16:12:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:12:46 smithi038 ceph-mon[92422]: pgmap v2900: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 148 KiB/s wr, 9 op/s 2023-03-09T16:12:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:12:48 smithi122 ceph-mon[93869]: pgmap v2901: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 210 KiB/s wr, 12 op/s 2023-03-09T16:12:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:12:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4478.222294 secs 2023-03-09T16:12:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:12:48 smithi181 ceph-mon[94189]: pgmap v2901: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 210 KiB/s wr, 12 op/s 2023-03-09T16:12:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:12:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4478.222294 secs 2023-03-09T16:12:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:12:48 smithi038 ceph-mon[92422]: pgmap v2901: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 210 KiB/s wr, 12 op/s 2023-03-09T16:12:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:12:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4478.222294 secs 2023-03-09T16:12:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4940.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:12:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:12:50 smithi122 ceph-mon[93869]: pgmap v2902: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 149 KiB/s wr, 10 op/s 2023-03-09T16:12:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:12:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4480.997948 secs 2023-03-09T16:12:50.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:12:50 smithi181 ceph-mon[94189]: pgmap v2902: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 149 KiB/s wr, 10 op/s 2023-03-09T16:12:50.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:12:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4480.997948 secs 2023-03-09T16:12:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:12:50 smithi038 ceph-mon[92422]: pgmap v2902: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 149 KiB/s wr, 10 op/s 2023-03-09T16:12:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:12:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4480.997948 secs 2023-03-09T16:12:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:12:51 smithi122 ceph-mon[93869]: pgmap v2903: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 200 KiB/s wr, 11 op/s 2023-03-09T16:12:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:12:51 smithi181 ceph-mon[94189]: pgmap v2903: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 200 KiB/s wr, 11 op/s 2023-03-09T16:12:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:12:51 smithi038 ceph-mon[92422]: pgmap v2903: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 200 KiB/s wr, 11 op/s 2023-03-09T16:12:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4945.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:12:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:12:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4483.222410 secs 2023-03-09T16:12:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:12:53 smithi122 ceph-mon[93869]: pgmap v2904: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 138 KiB/s wr, 11 op/s 2023-03-09T16:12:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:12:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4483.222410 secs 2023-03-09T16:12:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:12:53 smithi181 ceph-mon[94189]: pgmap v2904: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 138 KiB/s wr, 11 op/s 2023-03-09T16:12:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:12:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4483.222410 secs 2023-03-09T16:12:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:12:53 smithi038 ceph-mon[92422]: pgmap v2904: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 138 KiB/s wr, 11 op/s 2023-03-09T16:12:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:12:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4485.998007 secs 2023-03-09T16:12:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:12:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4485.998007 secs 2023-03-09T16:12:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:12:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4485.998007 secs 2023-03-09T16:12:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:12:56 smithi122 ceph-mon[93869]: pgmap v2905: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 131 KiB/s wr, 9 op/s 2023-03-09T16:12:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:12:56 smithi181 ceph-mon[94189]: pgmap v2905: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 131 KiB/s wr, 9 op/s 2023-03-09T16:12:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:12:56 smithi038 ceph-mon[92422]: pgmap v2905: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 131 KiB/s wr, 9 op/s 2023-03-09T16:12:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:12:58 smithi122 ceph-mon[93869]: pgmap v2906: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 178 KiB/s wr, 12 op/s 2023-03-09T16:12:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:12:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4488.222508 secs 2023-03-09T16:12:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:12:58 smithi181 ceph-mon[94189]: pgmap v2906: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 178 KiB/s wr, 12 op/s 2023-03-09T16:12:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:12:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4488.222508 secs 2023-03-09T16:12:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:12:58 smithi038 ceph-mon[92422]: pgmap v2906: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 178 KiB/s wr, 12 op/s 2023-03-09T16:12:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:12:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4488.222508 secs 2023-03-09T16:12:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4950.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:13:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:13:00 smithi122 ceph-mon[93869]: pgmap v2907: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 116 KiB/s wr, 9 op/s 2023-03-09T16:13:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:13:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4490.998079 secs 2023-03-09T16:13:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:13:00 smithi181 ceph-mon[94189]: pgmap v2907: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 116 KiB/s wr, 9 op/s 2023-03-09T16:13:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:13:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4490.998079 secs 2023-03-09T16:13:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:13:00 smithi038 ceph-mon[92422]: pgmap v2907: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 116 KiB/s wr, 9 op/s 2023-03-09T16:13:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:13:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4490.998079 secs 2023-03-09T16:13:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:13:02 smithi122 ceph-mon[93869]: pgmap v2908: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 111 KiB/s wr, 9 op/s 2023-03-09T16:13:02.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:13:02 smithi181 ceph-mon[94189]: pgmap v2908: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 111 KiB/s wr, 9 op/s 2023-03-09T16:13:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:13:02 smithi038 ceph-mon[92422]: pgmap v2908: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 111 KiB/s wr, 9 op/s 2023-03-09T16:13:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:13:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4493.222639 secs 2023-03-09T16:13:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:13:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4493.222639 secs 2023-03-09T16:13:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:13:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4493.222639 secs 2023-03-09T16:13:03.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4955.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:13:04.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:13:04 smithi122 ceph-mon[93869]: pgmap v2909: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 106 KiB/s wr, 12 op/s 2023-03-09T16:13:04.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:13:04 smithi181 ceph-mon[94189]: pgmap v2909: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 106 KiB/s wr, 12 op/s 2023-03-09T16:13:04.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:13:04 smithi038 ceph-mon[92422]: pgmap v2909: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 106 KiB/s wr, 12 op/s 2023-03-09T16:13:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:13:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4495.998155 secs 2023-03-09T16:13:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:13:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4495.998155 secs 2023-03-09T16:13:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:13:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4495.998155 secs 2023-03-09T16:13:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:13:06 smithi122 ceph-mon[93869]: pgmap v2910: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 120 KiB/s wr, 22 op/s 2023-03-09T16:13:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:13:06 smithi181 ceph-mon[94189]: pgmap v2910: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 120 KiB/s wr, 22 op/s 2023-03-09T16:13:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:13:06 smithi038 ceph-mon[92422]: pgmap v2910: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 120 KiB/s wr, 22 op/s 2023-03-09T16:13:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:13:08 smithi122 ceph-mon[93869]: pgmap v2911: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 120 KiB/s wr, 24 op/s 2023-03-09T16:13:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:13:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4498.222779 secs 2023-03-09T16:13:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:13:08 smithi181 ceph-mon[94189]: pgmap v2911: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 120 KiB/s wr, 24 op/s 2023-03-09T16:13:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:13:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4498.222779 secs 2023-03-09T16:13:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:13:08 smithi038 ceph-mon[92422]: pgmap v2911: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 120 KiB/s wr, 24 op/s 2023-03-09T16:13:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:13:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4498.222779 secs 2023-03-09T16:13:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4960.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:13:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:13:10 smithi122 ceph-mon[93869]: pgmap v2912: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 73 KiB/s wr, 22 op/s 2023-03-09T16:13:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:13:10 smithi122 ceph-mon[93869]: osdmap e394: 12 total, 12 up, 12 in 2023-03-09T16:13:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:13:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4500.998217 secs 2023-03-09T16:13:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:13:10 smithi181 ceph-mon[94189]: pgmap v2912: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 73 KiB/s wr, 22 op/s 2023-03-09T16:13:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:13:10 smithi181 ceph-mon[94189]: osdmap e394: 12 total, 12 up, 12 in 2023-03-09T16:13:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:13:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4500.998217 secs 2023-03-09T16:13:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:13:10 smithi038 ceph-mon[92422]: pgmap v2912: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 73 KiB/s wr, 22 op/s 2023-03-09T16:13:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:13:10 smithi038 ceph-mon[92422]: osdmap e394: 12 total, 12 up, 12 in 2023-03-09T16:13:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:13:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4500.998217 secs 2023-03-09T16:13:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:13:12 smithi122 ceph-mon[93869]: pgmap v2914: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 87 KiB/s wr, 24 op/s 2023-03-09T16:13:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:13:12 smithi181 ceph-mon[94189]: pgmap v2914: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 87 KiB/s wr, 24 op/s 2023-03-09T16:13:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:13:12 smithi038 ceph-mon[92422]: pgmap v2914: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 87 KiB/s wr, 24 op/s 2023-03-09T16:13:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:13:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4503.222888 secs 2023-03-09T16:13:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:13:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4503.222888 secs 2023-03-09T16:13:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:13:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4503.222888 secs 2023-03-09T16:13:13.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4965.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:13:14.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:13:14 smithi122 ceph-mon[93869]: pgmap v2915: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 49 KiB/s wr, 21 op/s 2023-03-09T16:13:14.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:13:14 smithi181 ceph-mon[94189]: pgmap v2915: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 49 KiB/s wr, 21 op/s 2023-03-09T16:13:14.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:13:14 smithi038 ceph-mon[92422]: pgmap v2915: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 49 KiB/s wr, 21 op/s 2023-03-09T16:13:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:13:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4505.998276 secs 2023-03-09T16:13:15.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:13:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4505.998276 secs 2023-03-09T16:13:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:13:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4505.998276 secs 2023-03-09T16:13:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:13:16 smithi122 ceph-mon[93869]: pgmap v2916: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T16:13:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:13:16 smithi181 ceph-mon[94189]: pgmap v2916: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T16:13:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:13:16 smithi038 ceph-mon[92422]: pgmap v2916: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T16:13:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:13:18 smithi122 ceph-mon[93869]: pgmap v2917: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 6 op/s 2023-03-09T16:13:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:13:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4508.222976 secs 2023-03-09T16:13:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:13:18 smithi181 ceph-mon[94189]: pgmap v2917: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 6 op/s 2023-03-09T16:13:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:13:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4508.222976 secs 2023-03-09T16:13:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:13:18 smithi038 ceph-mon[92422]: pgmap v2917: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 6 op/s 2023-03-09T16:13:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:13:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4508.222976 secs 2023-03-09T16:13:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4970.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:13:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:13:20 smithi122 ceph-mon[93869]: pgmap v2918: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T16:13:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:13:20 smithi122 ceph-mon[93869]: osdmap e395: 12 total, 12 up, 12 in 2023-03-09T16:13:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:13:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4510.998335 secs 2023-03-09T16:13:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:13:20 smithi181 ceph-mon[94189]: pgmap v2918: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T16:13:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:13:20 smithi181 ceph-mon[94189]: osdmap e395: 12 total, 12 up, 12 in 2023-03-09T16:13:20.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:13:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4510.998335 secs 2023-03-09T16:13:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:13:20 smithi038 ceph-mon[92422]: pgmap v2918: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T16:13:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:13:20 smithi038 ceph-mon[92422]: osdmap e395: 12 total, 12 up, 12 in 2023-03-09T16:13:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:13:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4510.998335 secs 2023-03-09T16:13:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:13:22 smithi122 ceph-mon[93869]: pgmap v2920: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T16:13:22.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:13:22 smithi181 ceph-mon[94189]: pgmap v2920: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T16:13:22.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:13:22 smithi038 ceph-mon[92422]: pgmap v2920: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T16:13:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:13:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4513.223073 secs 2023-03-09T16:13:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:13:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4513.223073 secs 2023-03-09T16:13:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:13:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4513.223073 secs 2023-03-09T16:13:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4975.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:13:24.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:13:24 smithi122 ceph-mon[93869]: pgmap v2921: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 33 KiB/s wr, 6 op/s 2023-03-09T16:13:24.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:13:24 smithi181 ceph-mon[94189]: pgmap v2921: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 33 KiB/s wr, 6 op/s 2023-03-09T16:13:24.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:13:24 smithi038 ceph-mon[92422]: pgmap v2921: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 33 KiB/s wr, 6 op/s 2023-03-09T16:13:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:13:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4515.998395 secs 2023-03-09T16:13:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:13:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4515.998395 secs 2023-03-09T16:13:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:13:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4515.998395 secs 2023-03-09T16:13:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:13:26 smithi122 ceph-mon[93869]: pgmap v2922: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 33 KiB/s wr, 7 op/s 2023-03-09T16:13:26.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:13:26 smithi181 ceph-mon[94189]: pgmap v2922: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 33 KiB/s wr, 7 op/s 2023-03-09T16:13:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:13:26 smithi038 ceph-mon[92422]: pgmap v2922: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 33 KiB/s wr, 7 op/s 2023-03-09T16:13:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:13:28 smithi122 ceph-mon[93869]: pgmap v2923: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 33 KiB/s wr, 6 op/s 2023-03-09T16:13:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:13:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4518.223192 secs 2023-03-09T16:13:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:13:28 smithi181 ceph-mon[94189]: pgmap v2923: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 33 KiB/s wr, 6 op/s 2023-03-09T16:13:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:13:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4518.223192 secs 2023-03-09T16:13:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:13:28 smithi038 ceph-mon[92422]: pgmap v2923: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 33 KiB/s wr, 6 op/s 2023-03-09T16:13:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:13:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4518.223192 secs 2023-03-09T16:13:28.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4980.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:13:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:13:30 smithi122 ceph-mon[93869]: pgmap v2924: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 33 KiB/s wr, 7 op/s 2023-03-09T16:13:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:13:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4520.998455 secs 2023-03-09T16:13:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:13:30 smithi181 ceph-mon[94189]: pgmap v2924: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 33 KiB/s wr, 7 op/s 2023-03-09T16:13:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:13:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4520.998455 secs 2023-03-09T16:13:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:13:30 smithi038 ceph-mon[92422]: pgmap v2924: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 33 KiB/s wr, 7 op/s 2023-03-09T16:13:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:13:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4520.998455 secs 2023-03-09T16:13:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:13:32 smithi122 ceph-mon[93869]: pgmap v2925: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 29 KiB/s wr, 7 op/s 2023-03-09T16:13:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:13:32 smithi181 ceph-mon[94189]: pgmap v2925: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 29 KiB/s wr, 7 op/s 2023-03-09T16:13:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:13:32 smithi038 ceph-mon[92422]: pgmap v2925: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 29 KiB/s wr, 7 op/s 2023-03-09T16:13:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:13:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4523.223281 secs 2023-03-09T16:13:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:13:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4523.223281 secs 2023-03-09T16:13:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:13:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4523.223281 secs 2023-03-09T16:13:33.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4985.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:13:34.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:13:34 smithi122 ceph-mon[93869]: pgmap v2926: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 87 KiB/s wr, 8 op/s 2023-03-09T16:13:34.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:13:34 smithi181 ceph-mon[94189]: pgmap v2926: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 87 KiB/s wr, 8 op/s 2023-03-09T16:13:34.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:13:34 smithi038 ceph-mon[92422]: pgmap v2926: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 87 KiB/s wr, 8 op/s 2023-03-09T16:13:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:13:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4525.998514 secs 2023-03-09T16:13:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:13:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4525.998514 secs 2023-03-09T16:13:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:13:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4525.998514 secs 2023-03-09T16:13:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:13:36 smithi122 ceph-mon[93869]: pgmap v2927: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 59 KiB/s wr, 6 op/s 2023-03-09T16:13:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:13:36 smithi181 ceph-mon[94189]: pgmap v2927: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 59 KiB/s wr, 6 op/s 2023-03-09T16:13:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:13:36 smithi038 ceph-mon[92422]: pgmap v2927: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 59 KiB/s wr, 6 op/s 2023-03-09T16:13:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:13:38 smithi122 ceph-mon[93869]: pgmap v2928: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 91 KiB/s wr, 9 op/s 2023-03-09T16:13:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:13:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4528.223397 secs 2023-03-09T16:13:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:13:38 smithi181 ceph-mon[94189]: pgmap v2928: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 91 KiB/s wr, 9 op/s 2023-03-09T16:13:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:13:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4528.223397 secs 2023-03-09T16:13:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:13:38 smithi038 ceph-mon[92422]: pgmap v2928: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 91 KiB/s wr, 9 op/s 2023-03-09T16:13:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:13:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4528.223397 secs 2023-03-09T16:13:38.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4990.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:13:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:13:40 smithi122 ceph-mon[93869]: pgmap v2929: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 96 KiB/s wr, 8 op/s 2023-03-09T16:13:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:13:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4530.998574 secs 2023-03-09T16:13:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:13:40 smithi181 ceph-mon[94189]: pgmap v2929: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 96 KiB/s wr, 8 op/s 2023-03-09T16:13:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:13:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4530.998574 secs 2023-03-09T16:13:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:13:40 smithi038 ceph-mon[92422]: pgmap v2929: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 96 KiB/s wr, 8 op/s 2023-03-09T16:13:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:13:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4530.998574 secs 2023-03-09T16:13:42.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:13:42 smithi122 ceph-mon[93869]: pgmap v2930: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 95 KiB/s wr, 8 op/s 2023-03-09T16:13:42.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:13:42 smithi181 ceph-mon[94189]: pgmap v2930: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 95 KiB/s wr, 8 op/s 2023-03-09T16:13:42.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:13:42 smithi038 ceph-mon[92422]: pgmap v2930: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 95 KiB/s wr, 8 op/s 2023-03-09T16:13:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:13:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4533.223529 secs 2023-03-09T16:13:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:13:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4533.223529 secs 2023-03-09T16:13:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:13:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4533.223529 secs 2023-03-09T16:13:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 4995.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:13:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:13:44 smithi122 ceph-mon[93869]: pgmap v2931: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 112 KiB/s wr, 10 op/s 2023-03-09T16:13:44.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:13:44 smithi181 ceph-mon[94189]: pgmap v2931: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 112 KiB/s wr, 10 op/s 2023-03-09T16:13:44.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:13:44 smithi038 ceph-mon[92422]: pgmap v2931: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 112 KiB/s wr, 10 op/s 2023-03-09T16:13:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:13:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4535.998637 secs 2023-03-09T16:13:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:13:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:13:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:13:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4535.998637 secs 2023-03-09T16:13:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:13:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:13:45.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:13:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4535.998637 secs 2023-03-09T16:13:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:13:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:13:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:13:46 smithi122 ceph-mon[93869]: pgmap v2932: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 54 KiB/s wr, 9 op/s 2023-03-09T16:13:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:13:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:13:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:13:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:13:46.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:13:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:13:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:13:46 smithi181 ceph-mon[94189]: pgmap v2932: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 54 KiB/s wr, 9 op/s 2023-03-09T16:13:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:13:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:13:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:13:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:13:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:13:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:13:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:13:46 smithi038 ceph-mon[92422]: pgmap v2932: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 54 KiB/s wr, 9 op/s 2023-03-09T16:13:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:13:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:13:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:13:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:13:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:13:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:13:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:13:47 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:13:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:13:47 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:13:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:13:47 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:13:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:13:48 smithi122 ceph-mon[93869]: pgmap v2933: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 160 KiB/s wr, 12 op/s 2023-03-09T16:13:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:13:48 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:13:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:13:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:13:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:13:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:13:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:13:48 smithi181 ceph-mon[94189]: pgmap v2933: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 160 KiB/s wr, 12 op/s 2023-03-09T16:13:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:13:48 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:13:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:13:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:13:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:13:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:13:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:13:48 smithi038 ceph-mon[92422]: pgmap v2933: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 160 KiB/s wr, 12 op/s 2023-03-09T16:13:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:13:48 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:13:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:13:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:13:48.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:13:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:13:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5000.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:13:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:13:50 smithi122 ceph-mon[93869]: pgmap v2934: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 130 KiB/s wr, 10 op/s 2023-03-09T16:13:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:13:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4540.998720 secs 2023-03-09T16:13:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:13:50 smithi181 ceph-mon[94189]: pgmap v2934: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 130 KiB/s wr, 10 op/s 2023-03-09T16:13:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:13:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4540.998720 secs 2023-03-09T16:13:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:13:50 smithi038 ceph-mon[92422]: pgmap v2934: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 130 KiB/s wr, 10 op/s 2023-03-09T16:13:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:13:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4540.998720 secs 2023-03-09T16:13:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:13:52 smithi122 ceph-mon[93869]: pgmap v2935: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 125 KiB/s wr, 11 op/s 2023-03-09T16:13:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:13:52 smithi181 ceph-mon[94189]: pgmap v2935: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 125 KiB/s wr, 11 op/s 2023-03-09T16:13:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:13:52 smithi038 ceph-mon[92422]: pgmap v2935: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 125 KiB/s wr, 11 op/s 2023-03-09T16:13:53.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5005.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:13:54.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:13:54 smithi122 ceph-mon[93869]: pgmap v2936: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 138 KiB/s wr, 11 op/s 2023-03-09T16:13:54.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:13:54 smithi181 ceph-mon[94189]: pgmap v2936: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 138 KiB/s wr, 11 op/s 2023-03-09T16:13:54.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:13:54 smithi038 ceph-mon[92422]: pgmap v2936: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 138 KiB/s wr, 11 op/s 2023-03-09T16:13:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:13:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4545.998780 secs 2023-03-09T16:13:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:13:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4545.998780 secs 2023-03-09T16:13:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:13:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4545.998780 secs 2023-03-09T16:13:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:13:56 smithi122 ceph-mon[93869]: pgmap v2937: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 121 KiB/s wr, 9 op/s 2023-03-09T16:13:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:13:56 smithi181 ceph-mon[94189]: pgmap v2937: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 121 KiB/s wr, 9 op/s 2023-03-09T16:13:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:13:56 smithi038 ceph-mon[92422]: pgmap v2937: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 121 KiB/s wr, 9 op/s 2023-03-09T16:13:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:13:58 smithi122 ceph-mon[93869]: pgmap v2938: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 237 KiB/s wr, 13 op/s 2023-03-09T16:13:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:13:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4548.223873 secs 2023-03-09T16:13:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:13:58 smithi181 ceph-mon[94189]: pgmap v2938: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 237 KiB/s wr, 13 op/s 2023-03-09T16:13:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:13:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4548.223873 secs 2023-03-09T16:13:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:13:58 smithi038 ceph-mon[92422]: pgmap v2938: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 237 KiB/s wr, 13 op/s 2023-03-09T16:13:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:13:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4548.223873 secs 2023-03-09T16:13:58.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5010.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:14:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:14:00 smithi122 ceph-mon[93869]: pgmap v2939: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 132 KiB/s wr, 10 op/s 2023-03-09T16:14:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:14:00 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:14:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:14:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:14:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:14:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:14:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:14:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4550.998862 secs 2023-03-09T16:14:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:14:00 smithi181 ceph-mon[94189]: pgmap v2939: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 132 KiB/s wr, 10 op/s 2023-03-09T16:14:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:14:00 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:14:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:14:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:14:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:14:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:14:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:14:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4550.998862 secs 2023-03-09T16:14:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:14:00 smithi038 ceph-mon[92422]: pgmap v2939: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 132 KiB/s wr, 10 op/s 2023-03-09T16:14:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:14:00 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:14:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:14:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:14:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:14:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:14:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:14:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4550.998862 secs 2023-03-09T16:14:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:14:02 smithi122 ceph-mon[93869]: pgmap v2940: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 130 KiB/s wr, 10 op/s 2023-03-09T16:14:02.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:14:02 smithi181 ceph-mon[94189]: pgmap v2940: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 130 KiB/s wr, 10 op/s 2023-03-09T16:14:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:14:02 smithi038 ceph-mon[92422]: pgmap v2940: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 130 KiB/s wr, 10 op/s 2023-03-09T16:14:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:14:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4553.224007 secs 2023-03-09T16:14:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:14:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4553.224007 secs 2023-03-09T16:14:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:14:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4553.224007 secs 2023-03-09T16:14:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5015.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:14:04.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:14:04 smithi122 ceph-mon[93869]: pgmap v2941: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 173 KiB/s wr, 13 op/s 2023-03-09T16:14:04.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:14:04 smithi181 ceph-mon[94189]: pgmap v2941: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 173 KiB/s wr, 13 op/s 2023-03-09T16:14:04.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:14:04 smithi038 ceph-mon[92422]: pgmap v2941: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 173 KiB/s wr, 13 op/s 2023-03-09T16:14:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:14:05 smithi122 ceph-mon[93869]: osdmap e396: 12 total, 12 up, 12 in 2023-03-09T16:14:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:14:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4555.998949 secs 2023-03-09T16:14:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:14:05 smithi181 ceph-mon[94189]: osdmap e396: 12 total, 12 up, 12 in 2023-03-09T16:14:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:14:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4555.998949 secs 2023-03-09T16:14:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:14:05 smithi038 ceph-mon[92422]: osdmap e396: 12 total, 12 up, 12 in 2023-03-09T16:14:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:14:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4555.998949 secs 2023-03-09T16:14:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:14:06 smithi122 ceph-mon[93869]: pgmap v2943: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 218 KiB/s wr, 27 op/s 2023-03-09T16:14:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:14:06 smithi181 ceph-mon[94189]: pgmap v2943: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 218 KiB/s wr, 27 op/s 2023-03-09T16:14:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:14:06 smithi038 ceph-mon[92422]: pgmap v2943: 129 pgs: 129 active+clean; 8.6 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 218 KiB/s wr, 27 op/s 2023-03-09T16:14:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:14:08 smithi122 ceph-mon[93869]: pgmap v2944: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 154 KiB/s wr, 26 op/s 2023-03-09T16:14:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:14:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4558.224142 secs 2023-03-09T16:14:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:14:08 smithi181 ceph-mon[94189]: pgmap v2944: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 154 KiB/s wr, 26 op/s 2023-03-09T16:14:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:14:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4558.224142 secs 2023-03-09T16:14:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:14:08 smithi038 ceph-mon[92422]: pgmap v2944: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 154 KiB/s wr, 26 op/s 2023-03-09T16:14:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:14:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4558.224142 secs 2023-03-09T16:14:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5020.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:14:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:14:10 smithi122 ceph-mon[93869]: pgmap v2945: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 154 KiB/s wr, 26 op/s 2023-03-09T16:14:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:14:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4560.998988 secs 2023-03-09T16:14:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:14:10 smithi181 ceph-mon[94189]: pgmap v2945: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 154 KiB/s wr, 26 op/s 2023-03-09T16:14:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:14:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4560.998988 secs 2023-03-09T16:14:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:14:10 smithi038 ceph-mon[92422]: pgmap v2945: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 154 KiB/s wr, 26 op/s 2023-03-09T16:14:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:14:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4560.998988 secs 2023-03-09T16:14:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:14:12 smithi122 ceph-mon[93869]: pgmap v2946: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 159 KiB/s wr, 26 op/s 2023-03-09T16:14:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:14:12 smithi181 ceph-mon[94189]: pgmap v2946: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 159 KiB/s wr, 26 op/s 2023-03-09T16:14:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:14:12 smithi038 ceph-mon[92422]: pgmap v2946: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 159 KiB/s wr, 26 op/s 2023-03-09T16:14:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:14:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4563.224264 secs 2023-03-09T16:14:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:14:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4563.224264 secs 2023-03-09T16:14:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:14:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4563.224264 secs 2023-03-09T16:14:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5025.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:14:14.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:14:14 smithi122 ceph-mon[93869]: pgmap v2947: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 132 KiB/s wr, 23 op/s 2023-03-09T16:14:14.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:14:14 smithi122 ceph-mon[93869]: osdmap e397: 12 total, 12 up, 12 in 2023-03-09T16:14:14.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:14:14 smithi181 ceph-mon[94189]: pgmap v2947: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 132 KiB/s wr, 23 op/s 2023-03-09T16:14:14.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:14:14 smithi181 ceph-mon[94189]: osdmap e397: 12 total, 12 up, 12 in 2023-03-09T16:14:14.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:14:14 smithi038 ceph-mon[92422]: pgmap v2947: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 132 KiB/s wr, 23 op/s 2023-03-09T16:14:14.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:14:14 smithi038 ceph-mon[92422]: osdmap e397: 12 total, 12 up, 12 in 2023-03-09T16:14:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:14:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4565.999050 secs 2023-03-09T16:14:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:14:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4565.999050 secs 2023-03-09T16:14:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:14:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4565.999050 secs 2023-03-09T16:14:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:14:16 smithi122 ceph-mon[93869]: pgmap v2949: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 106 KiB/s wr, 8 op/s 2023-03-09T16:14:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:14:16 smithi181 ceph-mon[94189]: pgmap v2949: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 106 KiB/s wr, 8 op/s 2023-03-09T16:14:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:14:16 smithi038 ceph-mon[92422]: pgmap v2949: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 106 KiB/s wr, 8 op/s 2023-03-09T16:14:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:14:18 smithi122 ceph-mon[93869]: pgmap v2950: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 30 KiB/s wr, 7 op/s 2023-03-09T16:14:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:14:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4568.224326 secs 2023-03-09T16:14:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:14:18 smithi181 ceph-mon[94189]: pgmap v2950: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 30 KiB/s wr, 7 op/s 2023-03-09T16:14:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:14:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4568.224326 secs 2023-03-09T16:14:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:14:18 smithi038 ceph-mon[92422]: pgmap v2950: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 30 KiB/s wr, 7 op/s 2023-03-09T16:14:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:14:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4568.224326 secs 2023-03-09T16:14:18.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5030.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:14:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:14:20 smithi122 ceph-mon[93869]: pgmap v2951: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 30 KiB/s wr, 7 op/s 2023-03-09T16:14:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:14:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4570.999108 secs 2023-03-09T16:14:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:14:20 smithi181 ceph-mon[94189]: pgmap v2951: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 30 KiB/s wr, 7 op/s 2023-03-09T16:14:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:14:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4570.999108 secs 2023-03-09T16:14:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:14:20 smithi038 ceph-mon[92422]: pgmap v2951: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 30 KiB/s wr, 7 op/s 2023-03-09T16:14:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:14:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4570.999108 secs 2023-03-09T16:14:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:14:22 smithi122 ceph-mon[93869]: pgmap v2952: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 26 KiB/s wr, 7 op/s 2023-03-09T16:14:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:14:22 smithi181 ceph-mon[94189]: pgmap v2952: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 26 KiB/s wr, 7 op/s 2023-03-09T16:14:22.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:14:22 smithi038 ceph-mon[92422]: pgmap v2952: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 26 KiB/s wr, 7 op/s 2023-03-09T16:14:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:14:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4573.224436 secs 2023-03-09T16:14:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:14:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4573.224436 secs 2023-03-09T16:14:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:14:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4573.224436 secs 2023-03-09T16:14:23.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5035.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:14:24.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:14:24 smithi122 ceph-mon[93869]: pgmap v2953: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 35 KiB/s wr, 7 op/s 2023-03-09T16:14:24.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:14:24 smithi181 ceph-mon[94189]: pgmap v2953: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 35 KiB/s wr, 7 op/s 2023-03-09T16:14:24.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:14:24 smithi038 ceph-mon[92422]: pgmap v2953: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 35 KiB/s wr, 7 op/s 2023-03-09T16:14:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:14:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4575.999169 secs 2023-03-09T16:14:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:14:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4575.999169 secs 2023-03-09T16:14:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:14:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4575.999169 secs 2023-03-09T16:14:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:14:26 smithi122 ceph-mon[93869]: pgmap v2954: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 34 KiB/s wr, 7 op/s 2023-03-09T16:14:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:14:26 smithi181 ceph-mon[94189]: pgmap v2954: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 34 KiB/s wr, 7 op/s 2023-03-09T16:14:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:14:26 smithi038 ceph-mon[92422]: pgmap v2954: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 34 KiB/s wr, 7 op/s 2023-03-09T16:14:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:14:28 smithi122 ceph-mon[93869]: pgmap v2955: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 55 KiB/s wr, 9 op/s 2023-03-09T16:14:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:14:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4578.224541 secs 2023-03-09T16:14:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:14:28 smithi181 ceph-mon[94189]: pgmap v2955: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 55 KiB/s wr, 9 op/s 2023-03-09T16:14:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:14:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4578.224541 secs 2023-03-09T16:14:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:14:28 smithi038 ceph-mon[92422]: pgmap v2955: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 55 KiB/s wr, 9 op/s 2023-03-09T16:14:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:14:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4578.224541 secs 2023-03-09T16:14:28.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5040.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:14:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:14:30 smithi122 ceph-mon[93869]: pgmap v2956: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 55 KiB/s wr, 7 op/s 2023-03-09T16:14:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:14:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4580.999237 secs 2023-03-09T16:14:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:14:30 smithi181 ceph-mon[94189]: pgmap v2956: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 55 KiB/s wr, 7 op/s 2023-03-09T16:14:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:14:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4580.999237 secs 2023-03-09T16:14:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:14:30 smithi038 ceph-mon[92422]: pgmap v2956: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 55 KiB/s wr, 7 op/s 2023-03-09T16:14:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:14:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4580.999237 secs 2023-03-09T16:14:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:14:32 smithi122 ceph-mon[93869]: pgmap v2957: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 55 KiB/s wr, 8 op/s 2023-03-09T16:14:32.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:14:32 smithi181 ceph-mon[94189]: pgmap v2957: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 55 KiB/s wr, 8 op/s 2023-03-09T16:14:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:14:32 smithi038 ceph-mon[92422]: pgmap v2957: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 55 KiB/s wr, 8 op/s 2023-03-09T16:14:33.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5045.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:14:34.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:14:34 smithi122 ceph-mon[93869]: pgmap v2958: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 86 KiB/s wr, 8 op/s 2023-03-09T16:14:34.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:14:34 smithi181 ceph-mon[94189]: pgmap v2958: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 86 KiB/s wr, 8 op/s 2023-03-09T16:14:34.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:14:34 smithi038 ceph-mon[92422]: pgmap v2958: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 86 KiB/s wr, 8 op/s 2023-03-09T16:14:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:14:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4585.999319 secs 2023-03-09T16:14:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:14:35 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:14:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:14:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4585.999319 secs 2023-03-09T16:14:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:14:35 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:14:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:14:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4585.999319 secs 2023-03-09T16:14:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:14:35 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:14:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:14:36 smithi122 ceph-mon[93869]: pgmap v2959: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 58 KiB/s wr, 6 op/s 2023-03-09T16:14:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:14:36 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:14:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:14:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:14:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:14:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:14:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:14:36 smithi181 ceph-mon[94189]: pgmap v2959: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 58 KiB/s wr, 6 op/s 2023-03-09T16:14:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:14:36 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:14:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:14:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:14:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:14:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:14:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:14:36 smithi038 ceph-mon[92422]: pgmap v2959: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 58 KiB/s wr, 6 op/s 2023-03-09T16:14:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:14:36 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:14:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:14:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:14:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:14:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:14:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:14:38 smithi122 ceph-mon[93869]: pgmap v2960: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 134 KiB/s wr, 11 op/s 2023-03-09T16:14:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:14:38 smithi181 ceph-mon[94189]: pgmap v2960: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 134 KiB/s wr, 11 op/s 2023-03-09T16:14:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:14:38 smithi038 ceph-mon[92422]: pgmap v2960: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 134 KiB/s wr, 11 op/s 2023-03-09T16:14:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5050.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:14:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:14:40 smithi122 ceph-mon[93869]: pgmap v2961: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 108 KiB/s wr, 9 op/s 2023-03-09T16:14:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:14:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4590.999380 secs 2023-03-09T16:14:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:14:40 smithi181 ceph-mon[94189]: pgmap v2961: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 108 KiB/s wr, 9 op/s 2023-03-09T16:14:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:14:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4590.999380 secs 2023-03-09T16:14:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:14:40 smithi038 ceph-mon[92422]: pgmap v2961: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 108 KiB/s wr, 9 op/s 2023-03-09T16:14:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:14:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4590.999380 secs 2023-03-09T16:14:42.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:14:42 smithi122 ceph-mon[93869]: pgmap v2962: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 113 KiB/s wr, 10 op/s 2023-03-09T16:14:42.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:14:42 smithi181 ceph-mon[94189]: pgmap v2962: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 113 KiB/s wr, 10 op/s 2023-03-09T16:14:42.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:14:42 smithi038 ceph-mon[92422]: pgmap v2962: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 113 KiB/s wr, 10 op/s 2023-03-09T16:14:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5055.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:14:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:14:44 smithi122 ceph-mon[93869]: pgmap v2963: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 171 KiB/s wr, 11 op/s 2023-03-09T16:14:44.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:14:44 smithi181 ceph-mon[94189]: pgmap v2963: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 171 KiB/s wr, 11 op/s 2023-03-09T16:14:44.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:14:44 smithi038 ceph-mon[92422]: pgmap v2963: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 171 KiB/s wr, 11 op/s 2023-03-09T16:14:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:14:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4595.999440 secs 2023-03-09T16:14:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:14:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4595.999440 secs 2023-03-09T16:14:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:14:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4595.999440 secs 2023-03-09T16:14:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:14:46 smithi122 ceph-mon[93869]: pgmap v2964: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 140 KiB/s wr, 10 op/s 2023-03-09T16:14:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:14:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:14:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:14:46 smithi181 ceph-mon[94189]: pgmap v2964: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 140 KiB/s wr, 10 op/s 2023-03-09T16:14:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:14:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:14:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:14:46 smithi038 ceph-mon[92422]: pgmap v2964: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 140 KiB/s wr, 10 op/s 2023-03-09T16:14:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:14:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:14:48.482 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:14:48 smithi038 ceph-mon[92422]: pgmap v2965: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 144 KiB/s wr, 13 op/s 2023-03-09T16:14:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:14:48 smithi122 ceph-mon[93869]: pgmap v2965: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 144 KiB/s wr, 13 op/s 2023-03-09T16:14:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:14:48 smithi181 ceph-mon[94189]: pgmap v2965: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 144 KiB/s wr, 13 op/s 2023-03-09T16:14:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5060.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:14:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:14:50 smithi122 ceph-mon[93869]: pgmap v2966: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 140 KiB/s wr, 10 op/s 2023-03-09T16:14:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:14:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4600.999500 secs 2023-03-09T16:14:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:14:50 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:14:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:14:50 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:14:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:14:50 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:14:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:14:50 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:14:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:14:50 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:14:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:14:50 smithi181 ceph-mon[94189]: pgmap v2966: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 140 KiB/s wr, 10 op/s 2023-03-09T16:14:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:14:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4600.999500 secs 2023-03-09T16:14:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:14:50 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:14:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:14:50 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:14:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:14:50 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:14:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:14:50 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:14:50.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:14:50 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:14:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:14:50 smithi038 ceph-mon[92422]: pgmap v2966: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 140 KiB/s wr, 10 op/s 2023-03-09T16:14:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:14:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4600.999500 secs 2023-03-09T16:14:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:14:50 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:14:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:14:50 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:14:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:14:50 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:14:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:14:50 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:14:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:14:50 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:14:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:14:52 smithi122 ceph-mon[93869]: pgmap v2967: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 145 KiB/s wr, 12 op/s 2023-03-09T16:14:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:14:52 smithi181 ceph-mon[94189]: pgmap v2967: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 145 KiB/s wr, 12 op/s 2023-03-09T16:14:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:14:52 smithi038 ceph-mon[92422]: pgmap v2967: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 145 KiB/s wr, 12 op/s 2023-03-09T16:14:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:14:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4603.225182 secs 2023-03-09T16:14:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:14:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4603.225182 secs 2023-03-09T16:14:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:14:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4603.225182 secs 2023-03-09T16:14:53.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5065.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:14:54.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:14:54 smithi122 ceph-mon[93869]: pgmap v2968: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 158 KiB/s wr, 11 op/s 2023-03-09T16:14:54.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:14:54 smithi181 ceph-mon[94189]: pgmap v2968: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 158 KiB/s wr, 11 op/s 2023-03-09T16:14:54.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:14:54 smithi038 ceph-mon[92422]: pgmap v2968: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 158 KiB/s wr, 11 op/s 2023-03-09T16:14:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:14:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4605.999573 secs 2023-03-09T16:14:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:14:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4605.999573 secs 2023-03-09T16:14:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:14:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4605.999573 secs 2023-03-09T16:14:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:14:56 smithi122 ceph-mon[93869]: pgmap v2969: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 100 KiB/s wr, 10 op/s 2023-03-09T16:14:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:14:56 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:14:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:14:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:14:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:14:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:14:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:14:56 smithi181 ceph-mon[94189]: pgmap v2969: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 100 KiB/s wr, 10 op/s 2023-03-09T16:14:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:14:56 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:14:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:14:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:14:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:14:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:14:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:14:56 smithi038 ceph-mon[92422]: pgmap v2969: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 100 KiB/s wr, 10 op/s 2023-03-09T16:14:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:14:56 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:14:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:14:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:14:56.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:14:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:14:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:14:58 smithi122 ceph-mon[93869]: pgmap v2970: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 100 KiB/s wr, 12 op/s 2023-03-09T16:14:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:14:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4608.225291 secs 2023-03-09T16:14:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:14:58 smithi181 ceph-mon[94189]: pgmap v2970: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 100 KiB/s wr, 12 op/s 2023-03-09T16:14:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:14:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4608.225291 secs 2023-03-09T16:14:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:14:58 smithi038 ceph-mon[92422]: pgmap v2970: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 100 KiB/s wr, 12 op/s 2023-03-09T16:14:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:14:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4608.225291 secs 2023-03-09T16:14:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5070.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:15:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:15:00 smithi122 ceph-mon[93869]: pgmap v2971: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 159 KiB/s wr, 10 op/s 2023-03-09T16:15:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:15:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4610.999635 secs 2023-03-09T16:15:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:15:00 smithi181 ceph-mon[94189]: pgmap v2971: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 159 KiB/s wr, 10 op/s 2023-03-09T16:15:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:15:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4610.999635 secs 2023-03-09T16:15:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:15:00 smithi038 ceph-mon[92422]: pgmap v2971: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 159 KiB/s wr, 10 op/s 2023-03-09T16:15:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:15:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4610.999635 secs 2023-03-09T16:15:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:15:01 smithi122 ceph-mon[93869]: pgmap v2972: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 91 KiB/s wr, 10 op/s 2023-03-09T16:15:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:15:01 smithi181 ceph-mon[94189]: pgmap v2972: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 91 KiB/s wr, 10 op/s 2023-03-09T16:15:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:15:01 smithi038 ceph-mon[92422]: pgmap v2972: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 91 KiB/s wr, 10 op/s 2023-03-09T16:15:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5075.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:15:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:15:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4613.225413 secs 2023-03-09T16:15:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:15:03 smithi038 ceph-mon[92422]: pgmap v2973: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 141 KiB/s wr, 11 op/s 2023-03-09T16:15:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:15:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4613.225413 secs 2023-03-09T16:15:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:15:03 smithi122 ceph-mon[93869]: pgmap v2973: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 141 KiB/s wr, 11 op/s 2023-03-09T16:15:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:15:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4613.225413 secs 2023-03-09T16:15:04.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:15:03 smithi181 ceph-mon[94189]: pgmap v2973: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 141 KiB/s wr, 11 op/s 2023-03-09T16:15:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:15:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4615.999698 secs 2023-03-09T16:15:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:15:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4615.999698 secs 2023-03-09T16:15:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:15:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4615.999698 secs 2023-03-09T16:15:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:15:06 smithi122 ceph-mon[93869]: pgmap v2974: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 144 KiB/s wr, 21 op/s 2023-03-09T16:15:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:15:06 smithi181 ceph-mon[94189]: pgmap v2974: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 144 KiB/s wr, 21 op/s 2023-03-09T16:15:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:15:06 smithi038 ceph-mon[92422]: pgmap v2974: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 144 KiB/s wr, 21 op/s 2023-03-09T16:15:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:15:08 smithi122 ceph-mon[93869]: pgmap v2975: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 155 KiB/s wr, 24 op/s 2023-03-09T16:15:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:15:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4618.225530 secs 2023-03-09T16:15:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:15:08 smithi181 ceph-mon[94189]: pgmap v2975: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 155 KiB/s wr, 24 op/s 2023-03-09T16:15:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:15:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4618.225530 secs 2023-03-09T16:15:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:15:08 smithi038 ceph-mon[92422]: pgmap v2975: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 155 KiB/s wr, 24 op/s 2023-03-09T16:15:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:15:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4618.225530 secs 2023-03-09T16:15:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5080.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:15:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:15:10 smithi122 ceph-mon[93869]: pgmap v2976: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 155 KiB/s wr, 22 op/s 2023-03-09T16:15:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:15:10 smithi122 ceph-mon[93869]: osdmap e398: 12 total, 12 up, 12 in 2023-03-09T16:15:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:15:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4620.999760 secs 2023-03-09T16:15:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:15:10 smithi181 ceph-mon[94189]: pgmap v2976: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 155 KiB/s wr, 22 op/s 2023-03-09T16:15:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:15:10 smithi181 ceph-mon[94189]: osdmap e398: 12 total, 12 up, 12 in 2023-03-09T16:15:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:15:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4620.999760 secs 2023-03-09T16:15:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:15:10 smithi038 ceph-mon[92422]: pgmap v2976: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 155 KiB/s wr, 22 op/s 2023-03-09T16:15:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:15:10 smithi038 ceph-mon[92422]: osdmap e398: 12 total, 12 up, 12 in 2023-03-09T16:15:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:15:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4620.999760 secs 2023-03-09T16:15:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:15:12 smithi122 ceph-mon[93869]: pgmap v2978: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 104 KiB/s wr, 24 op/s 2023-03-09T16:15:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:15:12 smithi181 ceph-mon[94189]: pgmap v2978: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 104 KiB/s wr, 24 op/s 2023-03-09T16:15:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:15:12 smithi038 ceph-mon[92422]: pgmap v2978: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 104 KiB/s wr, 24 op/s 2023-03-09T16:15:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:15:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4623.225654 secs 2023-03-09T16:15:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:15:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4623.225654 secs 2023-03-09T16:15:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:15:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4623.225654 secs 2023-03-09T16:15:13.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5085.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:15:14.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:15:14 smithi122 ceph-mon[93869]: pgmap v2979: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 58 KiB/s wr, 22 op/s 2023-03-09T16:15:14.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:15:14 smithi181 ceph-mon[94189]: pgmap v2979: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 58 KiB/s wr, 22 op/s 2023-03-09T16:15:14.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:15:14 smithi038 ceph-mon[92422]: pgmap v2979: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 58 KiB/s wr, 22 op/s 2023-03-09T16:15:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:15:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4625.999824 secs 2023-03-09T16:15:15.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:15:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4625.999824 secs 2023-03-09T16:15:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:15:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4625.999824 secs 2023-03-09T16:15:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:15:16 smithi122 ceph-mon[93869]: pgmap v2980: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T16:15:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:15:16 smithi181 ceph-mon[94189]: pgmap v2980: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T16:15:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:15:16 smithi038 ceph-mon[92422]: pgmap v2980: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T16:15:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:15:18 smithi122 ceph-mon[93869]: pgmap v2981: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 6 op/s 2023-03-09T16:15:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:15:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4628.225736 secs 2023-03-09T16:15:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:15:18 smithi181 ceph-mon[94189]: pgmap v2981: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 6 op/s 2023-03-09T16:15:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:15:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4628.225736 secs 2023-03-09T16:15:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:15:18 smithi038 ceph-mon[92422]: pgmap v2981: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 6 op/s 2023-03-09T16:15:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:15:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4628.225736 secs 2023-03-09T16:15:18.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5090.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:15:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:15:20 smithi122 ceph-mon[93869]: pgmap v2982: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 6 op/s 2023-03-09T16:15:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:15:20 smithi122 ceph-mon[93869]: osdmap e399: 12 total, 12 up, 12 in 2023-03-09T16:15:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:15:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4630.999884 secs 2023-03-09T16:15:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:15:20 smithi181 ceph-mon[94189]: pgmap v2982: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 6 op/s 2023-03-09T16:15:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:15:20 smithi181 ceph-mon[94189]: osdmap e399: 12 total, 12 up, 12 in 2023-03-09T16:15:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:15:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4630.999884 secs 2023-03-09T16:15:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:15:20 smithi038 ceph-mon[92422]: pgmap v2982: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 6 op/s 2023-03-09T16:15:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:15:20 smithi038 ceph-mon[92422]: osdmap e399: 12 total, 12 up, 12 in 2023-03-09T16:15:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:15:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4630.999884 secs 2023-03-09T16:15:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:15:22 smithi122 ceph-mon[93869]: pgmap v2984: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 6 op/s 2023-03-09T16:15:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:15:22 smithi181 ceph-mon[94189]: pgmap v2984: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 6 op/s 2023-03-09T16:15:22.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:15:22 smithi038 ceph-mon[92422]: pgmap v2984: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 6 op/s 2023-03-09T16:15:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:15:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4633.225839 secs 2023-03-09T16:15:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:15:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4633.225839 secs 2023-03-09T16:15:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:15:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4633.225839 secs 2023-03-09T16:15:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5095.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:15:24.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:15:24 smithi122 ceph-mon[93869]: pgmap v2985: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T16:15:24.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:15:24 smithi181 ceph-mon[94189]: pgmap v2985: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T16:15:24.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:15:24 smithi038 ceph-mon[92422]: pgmap v2985: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T16:15:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:15:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4635.999943 secs 2023-03-09T16:15:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:15:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4635.999943 secs 2023-03-09T16:15:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:15:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4635.999943 secs 2023-03-09T16:15:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:15:26 smithi122 ceph-mon[93869]: pgmap v2986: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T16:15:26.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:15:26 smithi181 ceph-mon[94189]: pgmap v2986: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T16:15:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:15:26 smithi038 ceph-mon[92422]: pgmap v2986: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T16:15:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:15:28 smithi122 ceph-mon[93869]: pgmap v2987: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T16:15:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:15:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4638.225971 secs 2023-03-09T16:15:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:15:28 smithi181 ceph-mon[94189]: pgmap v2987: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T16:15:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:15:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4638.225971 secs 2023-03-09T16:15:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:15:28 smithi038 ceph-mon[92422]: pgmap v2987: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T16:15:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:15:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4638.225971 secs 2023-03-09T16:15:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5100.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:15:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:15:30 smithi122 ceph-mon[93869]: pgmap v2988: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 53 KiB/s wr, 7 op/s 2023-03-09T16:15:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:15:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4640.999983 secs 2023-03-09T16:15:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:15:30 smithi181 ceph-mon[94189]: pgmap v2988: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 53 KiB/s wr, 7 op/s 2023-03-09T16:15:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:15:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4640.999983 secs 2023-03-09T16:15:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:15:30 smithi038 ceph-mon[92422]: pgmap v2988: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 53 KiB/s wr, 7 op/s 2023-03-09T16:15:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:15:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4640.999983 secs 2023-03-09T16:15:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:15:32 smithi122 ceph-mon[93869]: pgmap v2989: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 46 KiB/s wr, 8 op/s 2023-03-09T16:15:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:15:32 smithi181 ceph-mon[94189]: pgmap v2989: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 46 KiB/s wr, 8 op/s 2023-03-09T16:15:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:15:32 smithi038 ceph-mon[92422]: pgmap v2989: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 46 KiB/s wr, 8 op/s 2023-03-09T16:15:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:15:33 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 4643.226090 secs 2023-03-09T16:15:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:15:33 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 4643.226090 secs 2023-03-09T16:15:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:15:33 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 4643.226090 secs 2023-03-09T16:15:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5105.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:15:34.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:15:34 smithi122 ceph-mon[93869]: pgmap v2990: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 76 KiB/s wr, 7 op/s 2023-03-09T16:15:34.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:15:34 smithi181 ceph-mon[94189]: pgmap v2990: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 76 KiB/s wr, 7 op/s 2023-03-09T16:15:34.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:15:34 smithi038 ceph-mon[92422]: pgmap v2990: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 76 KiB/s wr, 7 op/s 2023-03-09T16:15:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:15:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4646.000069 secs 2023-03-09T16:15:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:15:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4646.000069 secs 2023-03-09T16:15:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:15:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4646.000069 secs 2023-03-09T16:15:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:15:36 smithi122 ceph-mon[93869]: pgmap v2991: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 60 KiB/s wr, 6 op/s 2023-03-09T16:15:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:15:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:15:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:15:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:15:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:15:36 smithi181 ceph-mon[94189]: pgmap v2991: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 60 KiB/s wr, 6 op/s 2023-03-09T16:15:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:15:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:15:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:15:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:15:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:15:36 smithi038 ceph-mon[92422]: pgmap v2991: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 60 KiB/s wr, 6 op/s 2023-03-09T16:15:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:15:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:15:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:15:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:15:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:15:38 smithi122 ceph-mon[93869]: pgmap v2992: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 69 KiB/s wr, 8 op/s 2023-03-09T16:15:38.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:15:38 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 4648.226227 secs 2023-03-09T16:15:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:15:38 smithi181 ceph-mon[94189]: pgmap v2992: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 69 KiB/s wr, 8 op/s 2023-03-09T16:15:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:15:38 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 4648.226227 secs 2023-03-09T16:15:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:15:38 smithi038 ceph-mon[92422]: pgmap v2992: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 69 KiB/s wr, 8 op/s 2023-03-09T16:15:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:15:38 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 4648.226227 secs 2023-03-09T16:15:38.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5110.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:15:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:15:40 smithi122 ceph-mon[93869]: pgmap v2993: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 97 KiB/s wr, 7 op/s 2023-03-09T16:15:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:15:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4651.000128 secs 2023-03-09T16:15:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:15:40 smithi181 ceph-mon[94189]: pgmap v2993: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 97 KiB/s wr, 7 op/s 2023-03-09T16:15:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:15:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4651.000128 secs 2023-03-09T16:15:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:15:40 smithi038 ceph-mon[92422]: pgmap v2993: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 97 KiB/s wr, 7 op/s 2023-03-09T16:15:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:15:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4651.000128 secs 2023-03-09T16:15:42.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:15:42 smithi122 ceph-mon[93869]: pgmap v2994: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 73 KiB/s wr, 9 op/s 2023-03-09T16:15:42.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:15:42 smithi181 ceph-mon[94189]: pgmap v2994: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 73 KiB/s wr, 9 op/s 2023-03-09T16:15:42.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:15:42 smithi038 ceph-mon[92422]: pgmap v2994: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 73 KiB/s wr, 9 op/s 2023-03-09T16:15:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:15:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4653.226341 secs 2023-03-09T16:15:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:15:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4653.226341 secs 2023-03-09T16:15:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:15:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4653.226341 secs 2023-03-09T16:15:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5115.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:15:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:15:44 smithi122 ceph-mon[93869]: pgmap v2995: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 110 KiB/s wr, 9 op/s 2023-03-09T16:15:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:15:44 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:15:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:15:44 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:15:44.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:15:44 smithi181 ceph-mon[94189]: pgmap v2995: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 110 KiB/s wr, 9 op/s 2023-03-09T16:15:44.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:15:44 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:15:44.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:15:44 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:15:44.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:15:44 smithi038 ceph-mon[92422]: pgmap v2995: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 110 KiB/s wr, 9 op/s 2023-03-09T16:15:44.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:15:44 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:15:44.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:15:44 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:15:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:15:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4656.000191 secs 2023-03-09T16:15:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:15:45 smithi122 ceph-mon[93869]: pgmap v2996: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 78 KiB/s wr, 9 op/s 2023-03-09T16:15:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:15:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4656.000191 secs 2023-03-09T16:15:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:15:45 smithi181 ceph-mon[94189]: pgmap v2996: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 78 KiB/s wr, 9 op/s 2023-03-09T16:15:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:15:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4656.000191 secs 2023-03-09T16:15:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:15:45 smithi038 ceph-mon[92422]: pgmap v2996: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 78 KiB/s wr, 9 op/s 2023-03-09T16:15:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:15:47 smithi122 ceph-mon[93869]: pgmap v2997: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 163 KiB/s wr, 12 op/s 2023-03-09T16:15:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:15:47 smithi181 ceph-mon[94189]: pgmap v2997: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 163 KiB/s wr, 12 op/s 2023-03-09T16:15:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:15:47 smithi038 ceph-mon[92422]: pgmap v2997: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 163 KiB/s wr, 12 op/s 2023-03-09T16:15:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5120.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:15:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:15:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4658.226458 secs 2023-03-09T16:15:49.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:15:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4658.226458 secs 2023-03-09T16:15:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:15:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4658.226458 secs 2023-03-09T16:15:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:15:50 smithi122 ceph-mon[93869]: pgmap v2998: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 155 KiB/s wr, 10 op/s 2023-03-09T16:15:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:15:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4661.000251 secs 2023-03-09T16:15:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:15:50 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:15:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:15:50 smithi181 ceph-mon[94189]: pgmap v2998: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 155 KiB/s wr, 10 op/s 2023-03-09T16:15:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:15:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4661.000251 secs 2023-03-09T16:15:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:15:50 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:15:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:15:50 smithi038 ceph-mon[92422]: pgmap v2998: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 155 KiB/s wr, 10 op/s 2023-03-09T16:15:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:15:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4661.000251 secs 2023-03-09T16:15:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:15:50 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:15:52.510 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:15:52 smithi181 ceph-mon[94189]: pgmap v2999: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 127 KiB/s wr, 12 op/s 2023-03-09T16:15:52.621 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:15:52 smithi122 ceph-mon[93869]: pgmap v2999: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 127 KiB/s wr, 12 op/s 2023-03-09T16:15:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:15:52 smithi038 ceph-mon[92422]: pgmap v2999: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 127 KiB/s wr, 12 op/s 2023-03-09T16:15:53.595 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:15:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4663.226543 secs 2023-03-09T16:15:53.672 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:15:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4663.226543 secs 2023-03-09T16:15:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:15:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4663.226543 secs 2023-03-09T16:15:53.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5125.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:15:54.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:15:54 smithi122 ceph-mon[93869]: pgmap v3000: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 136 KiB/s wr, 11 op/s 2023-03-09T16:15:54.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:15:54 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:15:54.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:15:54 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:15:54.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:15:54 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:15:54.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:15:54 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:15:54.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:15:54 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:15:54.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:15:54 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:15:54.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:15:54 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:15:54.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:15:54 smithi181 ceph-mon[94189]: pgmap v3000: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 136 KiB/s wr, 11 op/s 2023-03-09T16:15:54.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:15:54 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:15:54.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:15:54 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:15:54.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:15:54 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:15:54.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:15:54 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:15:54.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:15:54 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:15:54.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:15:54 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:15:54.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:15:54 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:15:54.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:15:54 smithi038 ceph-mon[92422]: pgmap v3000: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 136 KiB/s wr, 11 op/s 2023-03-09T16:15:54.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:15:54 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:15:54.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:15:54 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:15:54.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:15:54 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:15:54.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:15:54 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:15:54.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:15:54 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:15:54.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:15:54 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:15:54.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:15:54 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:15:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:15:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4666.000312 secs 2023-03-09T16:15:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:15:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4666.000312 secs 2023-03-09T16:15:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:15:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4666.000312 secs 2023-03-09T16:15:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:15:56 smithi122 ceph-mon[93869]: pgmap v3001: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 99 KiB/s wr, 9 op/s 2023-03-09T16:15:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:15:56 smithi181 ceph-mon[94189]: pgmap v3001: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 99 KiB/s wr, 9 op/s 2023-03-09T16:15:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:15:56 smithi038 ceph-mon[92422]: pgmap v3001: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 99 KiB/s wr, 9 op/s 2023-03-09T16:15:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:15:58 smithi122 ceph-mon[93869]: pgmap v3002: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 218 KiB/s wr, 13 op/s 2023-03-09T16:15:58.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:15:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4668.226688 secs 2023-03-09T16:15:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:15:58 smithi181 ceph-mon[94189]: pgmap v3002: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 218 KiB/s wr, 13 op/s 2023-03-09T16:15:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:15:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4668.226688 secs 2023-03-09T16:15:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:15:58 smithi038 ceph-mon[92422]: pgmap v3002: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 218 KiB/s wr, 13 op/s 2023-03-09T16:15:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:15:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4668.226688 secs 2023-03-09T16:15:58.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5130.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:16:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:16:00 smithi122 ceph-mon[93869]: pgmap v3003: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 133 KiB/s wr, 9 op/s 2023-03-09T16:16:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:16:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4671.000375 secs 2023-03-09T16:16:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:16:00 smithi181 ceph-mon[94189]: pgmap v3003: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 133 KiB/s wr, 9 op/s 2023-03-09T16:16:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:16:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4671.000375 secs 2023-03-09T16:16:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:16:00 smithi038 ceph-mon[92422]: pgmap v3003: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 133 KiB/s wr, 9 op/s 2023-03-09T16:16:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:16:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4671.000375 secs 2023-03-09T16:16:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:16:02 smithi122 ceph-mon[93869]: pgmap v3004: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 153 KiB/s wr, 12 op/s 2023-03-09T16:16:02.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:16:02 smithi181 ceph-mon[94189]: pgmap v3004: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 153 KiB/s wr, 12 op/s 2023-03-09T16:16:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:16:02 smithi038 ceph-mon[92422]: pgmap v3004: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 153 KiB/s wr, 12 op/s 2023-03-09T16:16:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:16:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4673.226796 secs 2023-03-09T16:16:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:16:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4673.226796 secs 2023-03-09T16:16:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:16:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4673.226796 secs 2023-03-09T16:16:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5135.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:16:04.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:16:04 smithi122 ceph-mon[93869]: pgmap v3005: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 153 KiB/s wr, 10 op/s 2023-03-09T16:16:04.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:16:04 smithi181 ceph-mon[94189]: pgmap v3005: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 153 KiB/s wr, 10 op/s 2023-03-09T16:16:04.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:16:04 smithi038 ceph-mon[92422]: pgmap v3005: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 153 KiB/s wr, 10 op/s 2023-03-09T16:16:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:16:05 smithi122 ceph-mon[93869]: osdmap e400: 12 total, 12 up, 12 in 2023-03-09T16:16:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:16:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4676.000434 secs 2023-03-09T16:16:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:16:05 smithi181 ceph-mon[94189]: osdmap e400: 12 total, 12 up, 12 in 2023-03-09T16:16:05.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:16:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4676.000434 secs 2023-03-09T16:16:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:16:05 smithi038 ceph-mon[92422]: osdmap e400: 12 total, 12 up, 12 in 2023-03-09T16:16:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:16:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4676.000434 secs 2023-03-09T16:16:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:16:06 smithi122 ceph-mon[93869]: pgmap v3007: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 194 KiB/s wr, 24 op/s 2023-03-09T16:16:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:16:06 smithi181 ceph-mon[94189]: pgmap v3007: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 194 KiB/s wr, 24 op/s 2023-03-09T16:16:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:16:06 smithi038 ceph-mon[92422]: pgmap v3007: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 194 KiB/s wr, 24 op/s 2023-03-09T16:16:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:16:08 smithi122 ceph-mon[93869]: pgmap v3008: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 156 KiB/s wr, 26 op/s 2023-03-09T16:16:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:16:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4678.226892 secs 2023-03-09T16:16:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:16:08 smithi181 ceph-mon[94189]: pgmap v3008: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 156 KiB/s wr, 26 op/s 2023-03-09T16:16:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:16:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4678.226892 secs 2023-03-09T16:16:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:16:08 smithi038 ceph-mon[92422]: pgmap v3008: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 156 KiB/s wr, 26 op/s 2023-03-09T16:16:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:16:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4678.226892 secs 2023-03-09T16:16:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5140.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:16:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:16:10 smithi122 ceph-mon[93869]: pgmap v3009: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 156 KiB/s wr, 26 op/s 2023-03-09T16:16:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:16:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4681.000495 secs 2023-03-09T16:16:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:16:10 smithi181 ceph-mon[94189]: pgmap v3009: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 156 KiB/s wr, 26 op/s 2023-03-09T16:16:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:16:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4681.000495 secs 2023-03-09T16:16:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:16:10 smithi038 ceph-mon[92422]: pgmap v3009: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 156 KiB/s wr, 26 op/s 2023-03-09T16:16:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:16:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4681.000495 secs 2023-03-09T16:16:12.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:16:12 smithi122 ceph-mon[93869]: pgmap v3010: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 162 KiB/s wr, 26 op/s 2023-03-09T16:16:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:16:12 smithi181 ceph-mon[94189]: pgmap v3010: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 162 KiB/s wr, 26 op/s 2023-03-09T16:16:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:16:12 smithi038 ceph-mon[92422]: pgmap v3010: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 162 KiB/s wr, 26 op/s 2023-03-09T16:16:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:16:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4683.227014 secs 2023-03-09T16:16:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:16:13 smithi122 ceph-mon[93869]: pgmap v3011: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 162 KiB/s wr, 26 op/s 2023-03-09T16:16:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:16:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4683.227014 secs 2023-03-09T16:16:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:16:13 smithi181 ceph-mon[94189]: pgmap v3011: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 162 KiB/s wr, 26 op/s 2023-03-09T16:16:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:16:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4683.227014 secs 2023-03-09T16:16:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:16:13 smithi038 ceph-mon[92422]: pgmap v3011: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 162 KiB/s wr, 26 op/s 2023-03-09T16:16:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5145.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:16:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:16:15 smithi122 ceph-mon[93869]: osdmap e401: 12 total, 12 up, 12 in 2023-03-09T16:16:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:16:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4686.000603 secs 2023-03-09T16:16:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:16:15 smithi181 ceph-mon[94189]: osdmap e401: 12 total, 12 up, 12 in 2023-03-09T16:16:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:16:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4686.000603 secs 2023-03-09T16:16:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:16:15 smithi038 ceph-mon[92422]: osdmap e401: 12 total, 12 up, 12 in 2023-03-09T16:16:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:16:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4686.000603 secs 2023-03-09T16:16:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:16:16 smithi122 ceph-mon[93869]: pgmap v3013: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 136 KiB/s wr, 12 op/s 2023-03-09T16:16:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:16:16 smithi181 ceph-mon[94189]: pgmap v3013: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 136 KiB/s wr, 12 op/s 2023-03-09T16:16:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:16:16 smithi038 ceph-mon[92422]: pgmap v3013: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 136 KiB/s wr, 12 op/s 2023-03-09T16:16:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:16:18 smithi122 ceph-mon[93869]: pgmap v3014: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 31 KiB/s wr, 8 op/s 2023-03-09T16:16:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:16:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4688.227139 secs 2023-03-09T16:16:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:16:18 smithi181 ceph-mon[94189]: pgmap v3014: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 31 KiB/s wr, 8 op/s 2023-03-09T16:16:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:16:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4688.227139 secs 2023-03-09T16:16:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:16:18 smithi038 ceph-mon[92422]: pgmap v3014: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 31 KiB/s wr, 8 op/s 2023-03-09T16:16:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:16:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4688.227139 secs 2023-03-09T16:16:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5150.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:16:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:16:20 smithi122 ceph-mon[93869]: pgmap v3015: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 31 KiB/s wr, 8 op/s 2023-03-09T16:16:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:16:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4691.000662 secs 2023-03-09T16:16:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:16:20 smithi181 ceph-mon[94189]: pgmap v3015: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 31 KiB/s wr, 8 op/s 2023-03-09T16:16:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:16:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4691.000662 secs 2023-03-09T16:16:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:16:20 smithi038 ceph-mon[92422]: pgmap v3015: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 31 KiB/s wr, 8 op/s 2023-03-09T16:16:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:16:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4691.000662 secs 2023-03-09T16:16:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:16:22 smithi122 ceph-mon[93869]: pgmap v3016: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 KiB/s wr, 8 op/s 2023-03-09T16:16:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:16:22 smithi181 ceph-mon[94189]: pgmap v3016: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 KiB/s wr, 8 op/s 2023-03-09T16:16:22.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:16:22 smithi038 ceph-mon[92422]: pgmap v3016: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 KiB/s wr, 8 op/s 2023-03-09T16:16:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:16:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4693.227215 secs 2023-03-09T16:16:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:16:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4693.227215 secs 2023-03-09T16:16:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:16:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4693.227215 secs 2023-03-09T16:16:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5155.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:16:24.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:16:24 smithi122 ceph-mon[93869]: pgmap v3017: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 KiB/s wr, 8 op/s 2023-03-09T16:16:24.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:16:24 smithi181 ceph-mon[94189]: pgmap v3017: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 KiB/s wr, 8 op/s 2023-03-09T16:16:24.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:16:24 smithi038 ceph-mon[92422]: pgmap v3017: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 KiB/s wr, 8 op/s 2023-03-09T16:16:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:16:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4696.000725 secs 2023-03-09T16:16:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:16:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4696.000725 secs 2023-03-09T16:16:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:16:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4696.000725 secs 2023-03-09T16:16:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:16:26 smithi122 ceph-mon[93869]: pgmap v3018: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 34 KiB/s wr, 8 op/s 2023-03-09T16:16:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:16:26 smithi181 ceph-mon[94189]: pgmap v3018: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 34 KiB/s wr, 8 op/s 2023-03-09T16:16:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:16:26 smithi038 ceph-mon[92422]: pgmap v3018: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 34 KiB/s wr, 8 op/s 2023-03-09T16:16:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:16:28 smithi122 ceph-mon[93869]: pgmap v3019: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 56 KiB/s wr, 10 op/s 2023-03-09T16:16:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:16:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4698.227351 secs 2023-03-09T16:16:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:16:28 smithi181 ceph-mon[94189]: pgmap v3019: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 56 KiB/s wr, 10 op/s 2023-03-09T16:16:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:16:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4698.227351 secs 2023-03-09T16:16:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:16:28 smithi038 ceph-mon[92422]: pgmap v3019: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 56 KiB/s wr, 10 op/s 2023-03-09T16:16:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:16:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4698.227351 secs 2023-03-09T16:16:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5160.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:16:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:16:30 smithi122 ceph-mon[93869]: pgmap v3020: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 56 KiB/s wr, 7 op/s 2023-03-09T16:16:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:16:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4701.000784 secs 2023-03-09T16:16:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:16:30 smithi181 ceph-mon[94189]: pgmap v3020: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 56 KiB/s wr, 7 op/s 2023-03-09T16:16:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:16:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4701.000784 secs 2023-03-09T16:16:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:16:30 smithi038 ceph-mon[92422]: pgmap v3020: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 56 KiB/s wr, 7 op/s 2023-03-09T16:16:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:16:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4701.000784 secs 2023-03-09T16:16:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:16:32 smithi122 ceph-mon[93869]: pgmap v3021: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 88 KiB/s wr, 10 op/s 2023-03-09T16:16:32.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:16:32 smithi181 ceph-mon[94189]: pgmap v3021: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 88 KiB/s wr, 10 op/s 2023-03-09T16:16:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:16:32 smithi038 ceph-mon[92422]: pgmap v3021: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 88 KiB/s wr, 10 op/s 2023-03-09T16:16:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:16:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4703.227431 secs 2023-03-09T16:16:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:16:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4703.227431 secs 2023-03-09T16:16:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:16:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4703.227431 secs 2023-03-09T16:16:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5165.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:16:34.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:16:34 smithi122 ceph-mon[93869]: pgmap v3022: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 7 op/s 2023-03-09T16:16:34.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:16:34 smithi181 ceph-mon[94189]: pgmap v3022: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 7 op/s 2023-03-09T16:16:34.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:16:34 smithi038 ceph-mon[92422]: pgmap v3022: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 7 op/s 2023-03-09T16:16:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:16:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4706.000844 secs 2023-03-09T16:16:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:16:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4706.000844 secs 2023-03-09T16:16:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:16:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4706.000844 secs 2023-03-09T16:16:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:16:36 smithi122 ceph-mon[93869]: pgmap v3023: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 59 KiB/s wr, 7 op/s 2023-03-09T16:16:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:16:36 smithi181 ceph-mon[94189]: pgmap v3023: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 59 KiB/s wr, 7 op/s 2023-03-09T16:16:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:16:36 smithi038 ceph-mon[92422]: pgmap v3023: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 59 KiB/s wr, 7 op/s 2023-03-09T16:16:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:16:38 smithi122 ceph-mon[93869]: pgmap v3024: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 137 KiB/s wr, 11 op/s 2023-03-09T16:16:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:16:38 smithi181 ceph-mon[94189]: pgmap v3024: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 137 KiB/s wr, 11 op/s 2023-03-09T16:16:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:16:38 smithi038 ceph-mon[92422]: pgmap v3024: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 137 KiB/s wr, 11 op/s 2023-03-09T16:16:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5170.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:16:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:16:39 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:16:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:16:39 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:16:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:16:39 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:16:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:16:40 smithi122 ceph-mon[93869]: pgmap v3025: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 111 KiB/s wr, 8 op/s 2023-03-09T16:16:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:16:40 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:16:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:16:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:16:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:16:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:16:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:16:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4711.000903 secs 2023-03-09T16:16:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:16:40 smithi181 ceph-mon[94189]: pgmap v3025: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 111 KiB/s wr, 8 op/s 2023-03-09T16:16:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:16:40 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:16:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:16:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:16:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:16:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:16:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:16:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4711.000903 secs 2023-03-09T16:16:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:16:40 smithi038 ceph-mon[92422]: pgmap v3025: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 111 KiB/s wr, 8 op/s 2023-03-09T16:16:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:16:40 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:16:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:16:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:16:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:16:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:16:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:16:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4711.000903 secs 2023-03-09T16:16:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:16:41 smithi122 ceph-mon[93869]: pgmap v3026: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 171 KiB/s wr, 12 op/s 2023-03-09T16:16:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:16:41 smithi181 ceph-mon[94189]: pgmap v3026: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 171 KiB/s wr, 12 op/s 2023-03-09T16:16:41.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:16:41 smithi038 ceph-mon[92422]: pgmap v3026: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 171 KiB/s wr, 12 op/s 2023-03-09T16:16:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5175.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:16:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:16:43 smithi122 ceph-mon[93869]: pgmap v3027: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 139 KiB/s wr, 9 op/s 2023-03-09T16:16:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:16:43 smithi181 ceph-mon[94189]: pgmap v3027: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 139 KiB/s wr, 9 op/s 2023-03-09T16:16:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:16:43 smithi038 ceph-mon[92422]: pgmap v3027: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 139 KiB/s wr, 9 op/s 2023-03-09T16:16:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:16:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4716.000964 secs 2023-03-09T16:16:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:16:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4716.000964 secs 2023-03-09T16:16:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:16:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4716.000964 secs 2023-03-09T16:16:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:16:46 smithi122 ceph-mon[93869]: pgmap v3028: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 139 KiB/s wr, 9 op/s 2023-03-09T16:16:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:16:46 smithi181 ceph-mon[94189]: pgmap v3028: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 139 KiB/s wr, 9 op/s 2023-03-09T16:16:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:16:46 smithi038 ceph-mon[92422]: pgmap v3028: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 139 KiB/s wr, 9 op/s 2023-03-09T16:16:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:16:48 smithi122 ceph-mon[93869]: pgmap v3029: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 221 KiB/s wr, 13 op/s 2023-03-09T16:16:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:16:48 smithi181 ceph-mon[94189]: pgmap v3029: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 221 KiB/s wr, 13 op/s 2023-03-09T16:16:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:16:48 smithi038 ceph-mon[92422]: pgmap v3029: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 221 KiB/s wr, 13 op/s 2023-03-09T16:16:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5180.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:16:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:16:50 smithi122 ceph-mon[93869]: pgmap v3030: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 144 KiB/s wr, 9 op/s 2023-03-09T16:16:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:16:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4721.000980 secs 2023-03-09T16:16:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:16:50 smithi181 ceph-mon[94189]: pgmap v3030: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 144 KiB/s wr, 9 op/s 2023-03-09T16:16:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:16:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4721.000980 secs 2023-03-09T16:16:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:16:50 smithi038 ceph-mon[92422]: pgmap v3030: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 144 KiB/s wr, 9 op/s 2023-03-09T16:16:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:16:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4721.000980 secs 2023-03-09T16:16:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:16:52 smithi122 ceph-mon[93869]: pgmap v3031: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 164 KiB/s wr, 12 op/s 2023-03-09T16:16:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:16:52 smithi181 ceph-mon[94189]: pgmap v3031: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 164 KiB/s wr, 12 op/s 2023-03-09T16:16:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:16:52 smithi038 ceph-mon[92422]: pgmap v3031: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 164 KiB/s wr, 12 op/s 2023-03-09T16:16:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:16:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4723.227971 secs 2023-03-09T16:16:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:16:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4723.227971 secs 2023-03-09T16:16:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:16:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4723.227971 secs 2023-03-09T16:16:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5185.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:16:54.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:16:54 smithi122 ceph-mon[93869]: pgmap v3032: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 104 KiB/s wr, 8 op/s 2023-03-09T16:16:54.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:16:54 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:16:54.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:16:54 smithi181 ceph-mon[94189]: pgmap v3032: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 104 KiB/s wr, 8 op/s 2023-03-09T16:16:54.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:16:54 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:16:54.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:16:54 smithi038 ceph-mon[92422]: pgmap v3032: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 104 KiB/s wr, 8 op/s 2023-03-09T16:16:54.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:16:54 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:16:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:16:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4726.001007 secs 2023-03-09T16:16:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:16:55 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:16:55.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:16:55 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:16:55.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:16:55 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:16:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:16:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4726.001007 secs 2023-03-09T16:16:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:16:55 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:16:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:16:55 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:16:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:16:55 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:16:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:16:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4726.001007 secs 2023-03-09T16:16:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:16:55 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:16:55.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:16:55 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:16:55.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:16:55 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:16:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:16:56 smithi122 ceph-mon[93869]: pgmap v3033: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 104 KiB/s wr, 8 op/s 2023-03-09T16:16:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:16:56 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:16:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:16:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:16:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:16:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:16:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:16:56 smithi181 ceph-mon[94189]: pgmap v3033: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 104 KiB/s wr, 8 op/s 2023-03-09T16:16:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:16:56 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:16:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:16:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:16:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:16:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:16:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:16:56 smithi038 ceph-mon[92422]: pgmap v3033: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 104 KiB/s wr, 8 op/s 2023-03-09T16:16:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:16:56 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:16:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:16:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:16:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:16:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:16:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:16:58 smithi122 ceph-mon[93869]: pgmap v3034: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 170 KiB/s wr, 12 op/s 2023-03-09T16:16:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:16:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4728.228087 secs 2023-03-09T16:16:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:16:58 smithi181 ceph-mon[94189]: pgmap v3034: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 170 KiB/s wr, 12 op/s 2023-03-09T16:16:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:16:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4728.228087 secs 2023-03-09T16:16:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:16:58 smithi038 ceph-mon[92422]: pgmap v3034: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 170 KiB/s wr, 12 op/s 2023-03-09T16:16:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:16:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4728.228087 secs 2023-03-09T16:16:58.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5190.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:17:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:17:00 smithi122 ceph-mon[93869]: pgmap v3035: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 88 KiB/s wr, 8 op/s 2023-03-09T16:17:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:17:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4731.001066 secs 2023-03-09T16:17:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:17:00 smithi181 ceph-mon[94189]: pgmap v3035: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 88 KiB/s wr, 8 op/s 2023-03-09T16:17:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:17:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4731.001066 secs 2023-03-09T16:17:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:17:00 smithi038 ceph-mon[92422]: pgmap v3035: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 88 KiB/s wr, 8 op/s 2023-03-09T16:17:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:17:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4731.001066 secs 2023-03-09T16:17:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:17:02 smithi122 ceph-mon[93869]: pgmap v3036: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 128 KiB/s wr, 12 op/s 2023-03-09T16:17:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:17:02 smithi181 ceph-mon[94189]: pgmap v3036: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 128 KiB/s wr, 12 op/s 2023-03-09T16:17:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:17:02 smithi038 ceph-mon[92422]: pgmap v3036: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 128 KiB/s wr, 12 op/s 2023-03-09T16:17:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:17:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4733.228175 secs 2023-03-09T16:17:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:17:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4733.228175 secs 2023-03-09T16:17:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:17:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4733.228175 secs 2023-03-09T16:17:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5195.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:17:04.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:17:04 smithi122 ceph-mon[93869]: pgmap v3037: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 107 KiB/s wr, 9 op/s 2023-03-09T16:17:04.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:17:04 smithi181 ceph-mon[94189]: pgmap v3037: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 107 KiB/s wr, 9 op/s 2023-03-09T16:17:04.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:17:04 smithi038 ceph-mon[92422]: pgmap v3037: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 107 KiB/s wr, 9 op/s 2023-03-09T16:17:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:17:05 smithi122 ceph-mon[93869]: osdmap e402: 12 total, 12 up, 12 in 2023-03-09T16:17:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:17:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4736.001128 secs 2023-03-09T16:17:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:17:05 smithi181 ceph-mon[94189]: osdmap e402: 12 total, 12 up, 12 in 2023-03-09T16:17:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:17:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4736.001128 secs 2023-03-09T16:17:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:17:05 smithi038 ceph-mon[92422]: osdmap e402: 12 total, 12 up, 12 in 2023-03-09T16:17:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:17:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4736.001128 secs 2023-03-09T16:17:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:17:06 smithi122 ceph-mon[93869]: pgmap v3039: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 154 KiB/s wr, 25 op/s 2023-03-09T16:17:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:17:06 smithi181 ceph-mon[94189]: pgmap v3039: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 154 KiB/s wr, 25 op/s 2023-03-09T16:17:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:17:06 smithi038 ceph-mon[92422]: pgmap v3039: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 154 KiB/s wr, 25 op/s 2023-03-09T16:17:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:17:08 smithi122 ceph-mon[93869]: pgmap v3040: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 143 KiB/s wr, 27 op/s 2023-03-09T16:17:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:17:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4738.228253 secs 2023-03-09T16:17:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:17:08 smithi181 ceph-mon[94189]: pgmap v3040: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 143 KiB/s wr, 27 op/s 2023-03-09T16:17:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:17:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4738.228253 secs 2023-03-09T16:17:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:17:08 smithi038 ceph-mon[92422]: pgmap v3040: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 143 KiB/s wr, 27 op/s 2023-03-09T16:17:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:17:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4738.228253 secs 2023-03-09T16:17:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5200.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:17:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:17:10 smithi122 ceph-mon[93869]: pgmap v3041: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 141 KiB/s wr, 26 op/s 2023-03-09T16:17:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:17:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4741.001207 secs 2023-03-09T16:17:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:17:10 smithi181 ceph-mon[94189]: pgmap v3041: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 141 KiB/s wr, 26 op/s 2023-03-09T16:17:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:17:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4741.001207 secs 2023-03-09T16:17:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:17:10 smithi038 ceph-mon[92422]: pgmap v3041: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 141 KiB/s wr, 26 op/s 2023-03-09T16:17:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:17:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4741.001207 secs 2023-03-09T16:17:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:17:12 smithi122 ceph-mon[93869]: pgmap v3042: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 112 KiB/s wr, 25 op/s 2023-03-09T16:17:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:17:12 smithi181 ceph-mon[94189]: pgmap v3042: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 112 KiB/s wr, 25 op/s 2023-03-09T16:17:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:17:12 smithi038 ceph-mon[92422]: pgmap v3042: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 112 KiB/s wr, 25 op/s 2023-03-09T16:17:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:17:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4743.228400 secs 2023-03-09T16:17:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:17:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4743.228400 secs 2023-03-09T16:17:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:17:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4743.228400 secs 2023-03-09T16:17:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5205.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:17:14.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:17:14 smithi122 ceph-mon[93869]: pgmap v3043: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 112 KiB/s wr, 25 op/s 2023-03-09T16:17:14.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:17:14 smithi122 ceph-mon[93869]: osdmap e403: 12 total, 12 up, 12 in 2023-03-09T16:17:14.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:17:14 smithi181 ceph-mon[94189]: pgmap v3043: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 112 KiB/s wr, 25 op/s 2023-03-09T16:17:14.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:17:14 smithi181 ceph-mon[94189]: osdmap e403: 12 total, 12 up, 12 in 2023-03-09T16:17:14.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:17:14 smithi038 ceph-mon[92422]: pgmap v3043: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 112 KiB/s wr, 25 op/s 2023-03-09T16:17:14.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:17:14 smithi038 ceph-mon[92422]: osdmap e403: 12 total, 12 up, 12 in 2023-03-09T16:17:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:17:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4746.001308 secs 2023-03-09T16:17:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:17:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4746.001308 secs 2023-03-09T16:17:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:17:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4746.001308 secs 2023-03-09T16:17:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:17:16 smithi122 ceph-mon[93869]: pgmap v3045: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 87 KiB/s wr, 10 op/s 2023-03-09T16:17:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:17:16 smithi181 ceph-mon[94189]: pgmap v3045: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 87 KiB/s wr, 10 op/s 2023-03-09T16:17:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:17:16 smithi038 ceph-mon[92422]: pgmap v3045: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 87 KiB/s wr, 10 op/s 2023-03-09T16:17:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:17:18 smithi122 ceph-mon[93869]: pgmap v3046: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 6 op/s 2023-03-09T16:17:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:17:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4748.228517 secs 2023-03-09T16:17:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:17:18 smithi181 ceph-mon[94189]: pgmap v3046: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 6 op/s 2023-03-09T16:17:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:17:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4748.228517 secs 2023-03-09T16:17:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:17:18 smithi038 ceph-mon[92422]: pgmap v3046: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 6 op/s 2023-03-09T16:17:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:17:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4748.228517 secs 2023-03-09T16:17:18.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5210.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:17:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:17:20 smithi122 ceph-mon[93869]: pgmap v3047: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T16:17:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:17:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4751.001369 secs 2023-03-09T16:17:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:17:20 smithi181 ceph-mon[94189]: pgmap v3047: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T16:17:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:17:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4751.001369 secs 2023-03-09T16:17:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:17:20 smithi038 ceph-mon[92422]: pgmap v3047: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T16:17:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:17:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4751.001369 secs 2023-03-09T16:17:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:17:22 smithi122 ceph-mon[93869]: pgmap v3048: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T16:17:22.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:17:22 smithi181 ceph-mon[94189]: pgmap v3048: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T16:17:22.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:17:22 smithi038 ceph-mon[92422]: pgmap v3048: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T16:17:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:17:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4753.228654 secs 2023-03-09T16:17:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:17:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4753.228654 secs 2023-03-09T16:17:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:17:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4753.228654 secs 2023-03-09T16:17:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5215.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:17:24.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:17:24 smithi122 ceph-mon[93869]: pgmap v3049: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T16:17:24.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:17:24 smithi181 ceph-mon[94189]: pgmap v3049: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T16:17:24.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:17:24 smithi038 ceph-mon[92422]: pgmap v3049: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T16:17:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:17:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4756.001431 secs 2023-03-09T16:17:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:17:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4756.001431 secs 2023-03-09T16:17:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:17:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4756.001431 secs 2023-03-09T16:17:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:17:26 smithi122 ceph-mon[93869]: pgmap v3050: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 196 B/s wr, 6 op/s 2023-03-09T16:17:26.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:17:26 smithi181 ceph-mon[94189]: pgmap v3050: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 196 B/s wr, 6 op/s 2023-03-09T16:17:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:17:26 smithi038 ceph-mon[92422]: pgmap v3050: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 196 B/s wr, 6 op/s 2023-03-09T16:17:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:17:27 smithi122 ceph-mon[93869]: pgmap v3051: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 8 op/s 2023-03-09T16:17:27.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:17:27 smithi181 ceph-mon[94189]: pgmap v3051: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 8 op/s 2023-03-09T16:17:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:17:27 smithi038 ceph-mon[92422]: pgmap v3051: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 8 op/s 2023-03-09T16:17:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:17:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4758.228774 secs 2023-03-09T16:17:28.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:17:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4758.228774 secs 2023-03-09T16:17:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:17:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4758.228774 secs 2023-03-09T16:17:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5220.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:17:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:17:30 smithi122 ceph-mon[93869]: pgmap v3052: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 170 B/s wr, 5 op/s 2023-03-09T16:17:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:17:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4761.001490 secs 2023-03-09T16:17:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:17:30 smithi181 ceph-mon[94189]: pgmap v3052: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 170 B/s wr, 5 op/s 2023-03-09T16:17:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:17:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4761.001490 secs 2023-03-09T16:17:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:17:30 smithi038 ceph-mon[92422]: pgmap v3052: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 170 B/s wr, 5 op/s 2023-03-09T16:17:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:17:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4761.001490 secs 2023-03-09T16:17:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:17:32 smithi122 ceph-mon[93869]: pgmap v3053: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 33 KiB/s wr, 7 op/s 2023-03-09T16:17:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:17:32 smithi181 ceph-mon[94189]: pgmap v3053: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 33 KiB/s wr, 7 op/s 2023-03-09T16:17:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:17:32 smithi038 ceph-mon[92422]: pgmap v3053: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 33 KiB/s wr, 7 op/s 2023-03-09T16:17:33.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5225.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:17:34.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:17:34 smithi122 ceph-mon[93869]: pgmap v3054: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 33 KiB/s wr, 5 op/s 2023-03-09T16:17:34.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:17:34 smithi181 ceph-mon[94189]: pgmap v3054: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 33 KiB/s wr, 5 op/s 2023-03-09T16:17:34.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:17:34 smithi038 ceph-mon[92422]: pgmap v3054: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 33 KiB/s wr, 5 op/s 2023-03-09T16:17:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:17:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4766.001551 secs 2023-03-09T16:17:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:17:35 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:17:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:17:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4766.001551 secs 2023-03-09T16:17:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:17:35 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:17:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:17:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4766.001551 secs 2023-03-09T16:17:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:17:35 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:17:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:17:36 smithi122 ceph-mon[93869]: pgmap v3055: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 33 KiB/s wr, 5 op/s 2023-03-09T16:17:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:17:36 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:17:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:17:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:17:36.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:17:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:17:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:17:36 smithi181 ceph-mon[94189]: pgmap v3055: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 33 KiB/s wr, 5 op/s 2023-03-09T16:17:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:17:36 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:17:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:17:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:17:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:17:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:17:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:17:36 smithi038 ceph-mon[92422]: pgmap v3055: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 33 KiB/s wr, 5 op/s 2023-03-09T16:17:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:17:36 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:17:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:17:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:17:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:17:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:17:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:17:38 smithi122 ceph-mon[93869]: pgmap v3056: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 70 KiB/s wr, 8 op/s 2023-03-09T16:17:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:17:38 smithi181 ceph-mon[94189]: pgmap v3056: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 70 KiB/s wr, 8 op/s 2023-03-09T16:17:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:17:38 smithi038 ceph-mon[92422]: pgmap v3056: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 70 KiB/s wr, 8 op/s 2023-03-09T16:17:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5230.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:17:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:17:40 smithi122 ceph-mon[93869]: pgmap v3057: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 70 KiB/s wr, 6 op/s 2023-03-09T16:17:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:17:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4771.001612 secs 2023-03-09T16:17:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:17:40 smithi181 ceph-mon[94189]: pgmap v3057: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 70 KiB/s wr, 6 op/s 2023-03-09T16:17:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:17:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4771.001612 secs 2023-03-09T16:17:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:17:40 smithi038 ceph-mon[92422]: pgmap v3057: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 70 KiB/s wr, 6 op/s 2023-03-09T16:17:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:17:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4771.001612 secs 2023-03-09T16:17:42.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:17:42 smithi122 ceph-mon[93869]: pgmap v3058: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 87 KiB/s wr, 10 op/s 2023-03-09T16:17:42.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:17:42 smithi181 ceph-mon[94189]: pgmap v3058: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 87 KiB/s wr, 10 op/s 2023-03-09T16:17:42.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:17:42 smithi038 ceph-mon[92422]: pgmap v3058: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 87 KiB/s wr, 10 op/s 2023-03-09T16:17:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:17:43 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 4773.229137 secs 2023-03-09T16:17:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:17:43 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 4773.229137 secs 2023-03-09T16:17:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:17:43 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 4773.229137 secs 2023-03-09T16:17:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5235.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:17:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:17:44 smithi122 ceph-mon[93869]: pgmap v3059: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 55 KiB/s wr, 7 op/s 2023-03-09T16:17:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:17:44 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:17:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:17:44 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:17:44.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:17:44 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:17:44.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:17:44 smithi181 ceph-mon[94189]: pgmap v3059: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 55 KiB/s wr, 7 op/s 2023-03-09T16:17:44.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:17:44 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:17:44.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:17:44 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:17:44.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:17:44 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:17:44.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:17:44 smithi038 ceph-mon[92422]: pgmap v3059: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 55 KiB/s wr, 7 op/s 2023-03-09T16:17:44.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:17:44 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:17:44.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:17:44 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:17:44.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:17:44 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:17:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:17:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4776.001673 secs 2023-03-09T16:17:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:17:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4776.001673 secs 2023-03-09T16:17:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:17:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4776.001673 secs 2023-03-09T16:17:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:17:46 smithi122 ceph-mon[93869]: pgmap v3060: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 55 KiB/s wr, 8 op/s 2023-03-09T16:17:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:17:46 smithi181 ceph-mon[94189]: pgmap v3060: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 55 KiB/s wr, 8 op/s 2023-03-09T16:17:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:17:46 smithi038 ceph-mon[92422]: pgmap v3060: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 55 KiB/s wr, 8 op/s 2023-03-09T16:17:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:17:48 smithi122 ceph-mon[93869]: pgmap v3061: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 166 KiB/s wr, 11 op/s 2023-03-09T16:17:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:17:48 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 4778.229267 secs 2023-03-09T16:17:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:17:48 smithi181 ceph-mon[94189]: pgmap v3061: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 166 KiB/s wr, 11 op/s 2023-03-09T16:17:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:17:48 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 4778.229267 secs 2023-03-09T16:17:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:17:48 smithi038 ceph-mon[92422]: pgmap v3061: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 166 KiB/s wr, 11 op/s 2023-03-09T16:17:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:17:48 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 4778.229267 secs 2023-03-09T16:17:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5240.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:17:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:17:50 smithi122 ceph-mon[93869]: pgmap v3062: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 129 KiB/s wr, 9 op/s 2023-03-09T16:17:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:17:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4781.001732 secs 2023-03-09T16:17:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:17:50 smithi181 ceph-mon[94189]: pgmap v3062: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 129 KiB/s wr, 9 op/s 2023-03-09T16:17:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:17:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4781.001732 secs 2023-03-09T16:17:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:17:50 smithi038 ceph-mon[92422]: pgmap v3062: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 129 KiB/s wr, 9 op/s 2023-03-09T16:17:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:17:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4781.001732 secs 2023-03-09T16:17:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:17:52 smithi122 ceph-mon[93869]: pgmap v3063: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 143 KiB/s wr, 12 op/s 2023-03-09T16:17:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:17:52 smithi181 ceph-mon[94189]: pgmap v3063: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 143 KiB/s wr, 12 op/s 2023-03-09T16:17:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:17:52 smithi038 ceph-mon[92422]: pgmap v3063: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 143 KiB/s wr, 12 op/s 2023-03-09T16:17:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:17:53 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 4783.229404 secs 2023-03-09T16:17:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:17:53 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 4783.229404 secs 2023-03-09T16:17:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:17:53 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 4783.229404 secs 2023-03-09T16:17:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5245.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:17:54.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:17:54 smithi122 ceph-mon[93869]: pgmap v3064: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 168 KiB/s wr, 9 op/s 2023-03-09T16:17:54.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:17:54 smithi181 ceph-mon[94189]: pgmap v3064: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 168 KiB/s wr, 9 op/s 2023-03-09T16:17:54.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:17:54 smithi038 ceph-mon[92422]: pgmap v3064: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 168 KiB/s wr, 9 op/s 2023-03-09T16:17:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:17:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4786.001799 secs 2023-03-09T16:17:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:17:55 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:17:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:17:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4786.001799 secs 2023-03-09T16:17:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:17:55 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:17:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:17:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4786.001799 secs 2023-03-09T16:17:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:17:55 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:17:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:17:56 smithi122 ceph-mon[93869]: pgmap v3065: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 168 KiB/s wr, 9 op/s 2023-03-09T16:17:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:17:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:17:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:17:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:17:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:17:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:17:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:17:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:17:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:17:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:17:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:17:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:17:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:17:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:17:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:17:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:17:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:17:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:17:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:17:56 smithi181 ceph-mon[94189]: pgmap v3065: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 168 KiB/s wr, 9 op/s 2023-03-09T16:17:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:17:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:17:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:17:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:17:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:17:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:17:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:17:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:17:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:17:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:17:56.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:17:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:17:56.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:17:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:17:56.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:17:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:17:56.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:17:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:17:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:17:56 smithi038 ceph-mon[92422]: pgmap v3065: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 168 KiB/s wr, 9 op/s 2023-03-09T16:17:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:17:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:17:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:17:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:17:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:17:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:17:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:17:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:17:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:17:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:17:56.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:17:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:17:56.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:17:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:17:56.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:17:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:17:56.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:17:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:17:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:17:57 smithi122 ceph-mon[93869]: pgmap v3066: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 208 KiB/s wr, 12 op/s 2023-03-09T16:17:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:17:57 smithi181 ceph-mon[94189]: pgmap v3066: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 208 KiB/s wr, 12 op/s 2023-03-09T16:17:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:17:57 smithi038 ceph-mon[92422]: pgmap v3066: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 208 KiB/s wr, 12 op/s 2023-03-09T16:17:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:17:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4788.229513 secs 2023-03-09T16:17:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:17:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4788.229513 secs 2023-03-09T16:17:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:17:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4788.229513 secs 2023-03-09T16:17:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5250.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:18:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:18:00 smithi122 ceph-mon[93869]: pgmap v3067: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 141 KiB/s wr, 9 op/s 2023-03-09T16:18:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:18:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:18:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:18:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:18:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:18:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4791.001858 secs 2023-03-09T16:18:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:18:00 smithi181 ceph-mon[94189]: pgmap v3067: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 141 KiB/s wr, 9 op/s 2023-03-09T16:18:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:18:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:18:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:18:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:18:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:18:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4791.001858 secs 2023-03-09T16:18:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:18:00 smithi038 ceph-mon[92422]: pgmap v3067: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 141 KiB/s wr, 9 op/s 2023-03-09T16:18:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:18:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:18:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:18:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:18:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:18:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4791.001858 secs 2023-03-09T16:18:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:18:02 smithi122 ceph-mon[93869]: pgmap v3068: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 141 KiB/s wr, 11 op/s 2023-03-09T16:18:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:18:02 smithi181 ceph-mon[94189]: pgmap v3068: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 141 KiB/s wr, 11 op/s 2023-03-09T16:18:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:18:02 smithi038 ceph-mon[92422]: pgmap v3068: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 141 KiB/s wr, 11 op/s 2023-03-09T16:18:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:18:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4793.229612 secs 2023-03-09T16:18:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:18:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4793.229612 secs 2023-03-09T16:18:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:18:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4793.229612 secs 2023-03-09T16:18:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5255.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:18:04.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:18:04 smithi122 ceph-mon[93869]: pgmap v3069: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 186 KiB/s wr, 10 op/s 2023-03-09T16:18:04.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:18:04 smithi181 ceph-mon[94189]: pgmap v3069: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 186 KiB/s wr, 10 op/s 2023-03-09T16:18:04.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:18:04 smithi038 ceph-mon[92422]: pgmap v3069: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 186 KiB/s wr, 10 op/s 2023-03-09T16:18:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:18:05 smithi122 ceph-mon[93869]: osdmap e404: 12 total, 12 up, 12 in 2023-03-09T16:18:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:18:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4796.002178 secs 2023-03-09T16:18:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:18:05 smithi181 ceph-mon[94189]: osdmap e404: 12 total, 12 up, 12 in 2023-03-09T16:18:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:18:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4796.002178 secs 2023-03-09T16:18:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:18:05 smithi038 ceph-mon[92422]: osdmap e404: 12 total, 12 up, 12 in 2023-03-09T16:18:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:18:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4796.002178 secs 2023-03-09T16:18:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:18:06 smithi122 ceph-mon[93869]: pgmap v3071: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 198 KiB/s wr, 25 op/s 2023-03-09T16:18:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:18:06 smithi181 ceph-mon[94189]: pgmap v3071: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 198 KiB/s wr, 25 op/s 2023-03-09T16:18:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:18:06 smithi038 ceph-mon[92422]: pgmap v3071: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 198 KiB/s wr, 25 op/s 2023-03-09T16:18:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:18:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4798.229700 secs 2023-03-09T16:18:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:18:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4798.229700 secs 2023-03-09T16:18:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:18:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4798.229700 secs 2023-03-09T16:18:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:18:08 smithi122 ceph-mon[93869]: pgmap v3072: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 181 KiB/s wr, 27 op/s 2023-03-09T16:18:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:18:08 smithi181 ceph-mon[94189]: pgmap v3072: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 181 KiB/s wr, 27 op/s 2023-03-09T16:18:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:18:08 smithi038 ceph-mon[92422]: pgmap v3072: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 181 KiB/s wr, 27 op/s 2023-03-09T16:18:08.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5260.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:18:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:18:10 smithi122 ceph-mon[93869]: pgmap v3073: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 158 KiB/s wr, 26 op/s 2023-03-09T16:18:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:18:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4801.002238 secs 2023-03-09T16:18:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:18:10 smithi181 ceph-mon[94189]: pgmap v3073: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 158 KiB/s wr, 26 op/s 2023-03-09T16:18:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:18:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4801.002238 secs 2023-03-09T16:18:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:18:10 smithi038 ceph-mon[92422]: pgmap v3073: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 158 KiB/s wr, 26 op/s 2023-03-09T16:18:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:18:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4801.002238 secs 2023-03-09T16:18:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:18:12 smithi122 ceph-mon[93869]: pgmap v3074: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 163 KiB/s wr, 26 op/s 2023-03-09T16:18:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:18:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4803.229830 secs 2023-03-09T16:18:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:18:12 smithi181 ceph-mon[94189]: pgmap v3074: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 163 KiB/s wr, 26 op/s 2023-03-09T16:18:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:18:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4803.229830 secs 2023-03-09T16:18:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:18:12 smithi038 ceph-mon[92422]: pgmap v3074: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 163 KiB/s wr, 26 op/s 2023-03-09T16:18:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:18:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4803.229830 secs 2023-03-09T16:18:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5265.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:18:14.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:18:14 smithi122 ceph-mon[93869]: pgmap v3075: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 117 KiB/s wr, 25 op/s 2023-03-09T16:18:14.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:18:14 smithi122 ceph-mon[93869]: osdmap e405: 12 total, 12 up, 12 in 2023-03-09T16:18:14.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:18:14 smithi181 ceph-mon[94189]: pgmap v3075: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 117 KiB/s wr, 25 op/s 2023-03-09T16:18:14.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:18:14 smithi181 ceph-mon[94189]: osdmap e405: 12 total, 12 up, 12 in 2023-03-09T16:18:14.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:18:14 smithi038 ceph-mon[92422]: pgmap v3075: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 117 KiB/s wr, 25 op/s 2023-03-09T16:18:14.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:18:14 smithi038 ceph-mon[92422]: osdmap e405: 12 total, 12 up, 12 in 2023-03-09T16:18:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:18:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4806.002297 secs 2023-03-09T16:18:15.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:18:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4806.002297 secs 2023-03-09T16:18:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:18:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4806.002297 secs 2023-03-09T16:18:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:18:16 smithi122 ceph-mon[93869]: pgmap v3077: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 92 KiB/s wr, 11 op/s 2023-03-09T16:18:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:18:16 smithi181 ceph-mon[94189]: pgmap v3077: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 92 KiB/s wr, 11 op/s 2023-03-09T16:18:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:18:16 smithi038 ceph-mon[92422]: pgmap v3077: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 92 KiB/s wr, 11 op/s 2023-03-09T16:18:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:18:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4808.229923 secs 2023-03-09T16:18:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:18:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4808.229923 secs 2023-03-09T16:18:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:18:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4808.229923 secs 2023-03-09T16:18:18.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:18:18 smithi122 ceph-mon[93869]: pgmap v3078: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 61 KiB/s wr, 8 op/s 2023-03-09T16:18:18.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:18:18 smithi181 ceph-mon[94189]: pgmap v3078: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 61 KiB/s wr, 8 op/s 2023-03-09T16:18:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:18:18 smithi038 ceph-mon[92422]: pgmap v3078: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 61 KiB/s wr, 8 op/s 2023-03-09T16:18:18.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5270.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:18:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:18:20 smithi122 ceph-mon[93869]: pgmap v3079: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 66 KiB/s wr, 7 op/s 2023-03-09T16:18:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:18:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4811.002356 secs 2023-03-09T16:18:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:18:20 smithi181 ceph-mon[94189]: pgmap v3079: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 66 KiB/s wr, 7 op/s 2023-03-09T16:18:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:18:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4811.002356 secs 2023-03-09T16:18:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:18:20 smithi038 ceph-mon[92422]: pgmap v3079: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 66 KiB/s wr, 7 op/s 2023-03-09T16:18:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:18:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4811.002356 secs 2023-03-09T16:18:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:18:21 smithi122 ceph-mon[93869]: pgmap v3080: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 62 KiB/s wr, 7 op/s 2023-03-09T16:18:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:18:21 smithi181 ceph-mon[94189]: pgmap v3080: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 62 KiB/s wr, 7 op/s 2023-03-09T16:18:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:18:21 smithi038 ceph-mon[92422]: pgmap v3080: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 62 KiB/s wr, 7 op/s 2023-03-09T16:18:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:18:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4813.229991 secs 2023-03-09T16:18:22.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:18:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4813.229991 secs 2023-03-09T16:18:22.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:18:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4813.229991 secs 2023-03-09T16:18:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:18:23 smithi038 ceph-mon[92422]: pgmap v3081: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 37 KiB/s wr, 7 op/s 2023-03-09T16:18:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5275.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:18:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:18:23 smithi122 ceph-mon[93869]: pgmap v3081: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 37 KiB/s wr, 7 op/s 2023-03-09T16:18:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:18:23 smithi181 ceph-mon[94189]: pgmap v3081: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 37 KiB/s wr, 7 op/s 2023-03-09T16:18:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:18:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4816.002416 secs 2023-03-09T16:18:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:18:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4816.002416 secs 2023-03-09T16:18:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:18:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4816.002416 secs 2023-03-09T16:18:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:18:26 smithi122 ceph-mon[93869]: pgmap v3082: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 35 KiB/s wr, 7 op/s 2023-03-09T16:18:26.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:18:26 smithi181 ceph-mon[94189]: pgmap v3082: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 35 KiB/s wr, 7 op/s 2023-03-09T16:18:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:18:26 smithi038 ceph-mon[92422]: pgmap v3082: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 35 KiB/s wr, 7 op/s 2023-03-09T16:18:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:18:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4818.230131 secs 2023-03-09T16:18:27.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:18:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4818.230131 secs 2023-03-09T16:18:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:18:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4818.230131 secs 2023-03-09T16:18:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:18:28 smithi122 ceph-mon[93869]: pgmap v3083: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 31 KiB/s wr, 8 op/s 2023-03-09T16:18:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:18:28 smithi181 ceph-mon[94189]: pgmap v3083: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 31 KiB/s wr, 8 op/s 2023-03-09T16:18:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:18:28 smithi038 ceph-mon[92422]: pgmap v3083: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 31 KiB/s wr, 8 op/s 2023-03-09T16:18:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5280.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:18:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:18:30 smithi122 ceph-mon[93869]: pgmap v3084: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 57 KiB/s wr, 7 op/s 2023-03-09T16:18:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:18:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4821.002474 secs 2023-03-09T16:18:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:18:30 smithi181 ceph-mon[94189]: pgmap v3084: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 57 KiB/s wr, 7 op/s 2023-03-09T16:18:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:18:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4821.002474 secs 2023-03-09T16:18:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:18:30 smithi038 ceph-mon[92422]: pgmap v3084: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 57 KiB/s wr, 7 op/s 2023-03-09T16:18:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:18:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4821.002474 secs 2023-03-09T16:18:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:18:32 smithi122 ceph-mon[93869]: pgmap v3085: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 28 KiB/s wr, 8 op/s 2023-03-09T16:18:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:18:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4823.230178 secs 2023-03-09T16:18:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:18:32 smithi181 ceph-mon[94189]: pgmap v3085: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 28 KiB/s wr, 8 op/s 2023-03-09T16:18:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:18:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4823.230178 secs 2023-03-09T16:18:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:18:32 smithi038 ceph-mon[92422]: pgmap v3085: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 28 KiB/s wr, 8 op/s 2023-03-09T16:18:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:18:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4823.230178 secs 2023-03-09T16:18:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5285.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:18:34.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:18:34 smithi122 ceph-mon[93869]: pgmap v3086: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 98 KiB/s wr, 7 op/s 2023-03-09T16:18:34.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:18:34 smithi181 ceph-mon[94189]: pgmap v3086: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 98 KiB/s wr, 7 op/s 2023-03-09T16:18:34.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:18:34 smithi038 ceph-mon[92422]: pgmap v3086: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 98 KiB/s wr, 7 op/s 2023-03-09T16:18:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:18:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4826.002532 secs 2023-03-09T16:18:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:18:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4826.002532 secs 2023-03-09T16:18:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:18:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4826.002532 secs 2023-03-09T16:18:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:18:36 smithi122 ceph-mon[93869]: pgmap v3087: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 98 KiB/s wr, 7 op/s 2023-03-09T16:18:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:18:36 smithi181 ceph-mon[94189]: pgmap v3087: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 98 KiB/s wr, 7 op/s 2023-03-09T16:18:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:18:36 smithi038 ceph-mon[92422]: pgmap v3087: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 98 KiB/s wr, 7 op/s 2023-03-09T16:18:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:18:37 smithi122 ceph-mon[93869]: pgmap v3088: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 107 KiB/s wr, 10 op/s 2023-03-09T16:18:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:18:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4828.230294 secs 2023-03-09T16:18:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:18:37 smithi181 ceph-mon[94189]: pgmap v3088: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 107 KiB/s wr, 10 op/s 2023-03-09T16:18:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:18:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4828.230294 secs 2023-03-09T16:18:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:18:37 smithi038 ceph-mon[92422]: pgmap v3088: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 107 KiB/s wr, 10 op/s 2023-03-09T16:18:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:18:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4828.230294 secs 2023-03-09T16:18:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5290.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:18:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:18:40 smithi122 ceph-mon[93869]: pgmap v3089: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 140 KiB/s wr, 10 op/s 2023-03-09T16:18:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:18:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4831.002592 secs 2023-03-09T16:18:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:18:40 smithi181 ceph-mon[94189]: pgmap v3089: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 140 KiB/s wr, 10 op/s 2023-03-09T16:18:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:18:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4831.002592 secs 2023-03-09T16:18:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:18:40 smithi038 ceph-mon[92422]: pgmap v3089: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 140 KiB/s wr, 10 op/s 2023-03-09T16:18:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:18:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4831.002592 secs 2023-03-09T16:18:42.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:18:42 smithi122 ceph-mon[93869]: pgmap v3090: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 118 KiB/s wr, 12 op/s 2023-03-09T16:18:42.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:18:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4833.230373 secs 2023-03-09T16:18:42.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:18:42 smithi181 ceph-mon[94189]: pgmap v3090: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 118 KiB/s wr, 12 op/s 2023-03-09T16:18:42.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:18:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4833.230373 secs 2023-03-09T16:18:42.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:18:42 smithi038 ceph-mon[92422]: pgmap v3090: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 118 KiB/s wr, 12 op/s 2023-03-09T16:18:42.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:18:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4833.230373 secs 2023-03-09T16:18:43.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5295.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:18:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:18:44 smithi122 ceph-mon[93869]: pgmap v3091: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 216 KiB/s wr, 11 op/s 2023-03-09T16:18:44.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:18:44 smithi181 ceph-mon[94189]: pgmap v3091: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 216 KiB/s wr, 11 op/s 2023-03-09T16:18:44.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:18:44 smithi038 ceph-mon[92422]: pgmap v3091: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 216 KiB/s wr, 11 op/s 2023-03-09T16:18:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:18:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4836.002677 secs 2023-03-09T16:18:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:18:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4836.002677 secs 2023-03-09T16:18:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:18:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4836.002677 secs 2023-03-09T16:18:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:18:46 smithi122 ceph-mon[93869]: pgmap v3092: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 146 KiB/s wr, 10 op/s 2023-03-09T16:18:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:18:46 smithi181 ceph-mon[94189]: pgmap v3092: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 146 KiB/s wr, 10 op/s 2023-03-09T16:18:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:18:46 smithi038 ceph-mon[92422]: pgmap v3092: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 146 KiB/s wr, 10 op/s 2023-03-09T16:18:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:18:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4838.230491 secs 2023-03-09T16:18:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:18:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4838.230491 secs 2023-03-09T16:18:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:18:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4838.230491 secs 2023-03-09T16:18:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:18:48 smithi122 ceph-mon[93869]: pgmap v3093: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 151 KiB/s wr, 13 op/s 2023-03-09T16:18:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:18:48 smithi181 ceph-mon[94189]: pgmap v3093: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 151 KiB/s wr, 13 op/s 2023-03-09T16:18:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:18:48 smithi038 ceph-mon[92422]: pgmap v3093: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 151 KiB/s wr, 13 op/s 2023-03-09T16:18:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5300.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:18:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:18:50 smithi122 ceph-mon[93869]: pgmap v3094: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 179 KiB/s wr, 11 op/s 2023-03-09T16:18:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:18:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4841.002739 secs 2023-03-09T16:18:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:18:50 smithi181 ceph-mon[94189]: pgmap v3094: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 179 KiB/s wr, 11 op/s 2023-03-09T16:18:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:18:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4841.002739 secs 2023-03-09T16:18:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:18:50 smithi038 ceph-mon[92422]: pgmap v3094: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 179 KiB/s wr, 11 op/s 2023-03-09T16:18:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:18:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4841.002739 secs 2023-03-09T16:18:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:18:52 smithi122 ceph-mon[93869]: pgmap v3095: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 150 KiB/s wr, 12 op/s 2023-03-09T16:18:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:18:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4843.230618 secs 2023-03-09T16:18:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:18:52 smithi181 ceph-mon[94189]: pgmap v3095: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 150 KiB/s wr, 12 op/s 2023-03-09T16:18:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:18:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4843.230618 secs 2023-03-09T16:18:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:18:52 smithi038 ceph-mon[92422]: pgmap v3095: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 150 KiB/s wr, 12 op/s 2023-03-09T16:18:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:18:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4843.230618 secs 2023-03-09T16:18:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5305.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:18:54.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:18:54 smithi122 ceph-mon[93869]: pgmap v3096: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 201 KiB/s wr, 10 op/s 2023-03-09T16:18:54.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:18:54 smithi181 ceph-mon[94189]: pgmap v3096: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 201 KiB/s wr, 10 op/s 2023-03-09T16:18:54.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:18:54 smithi038 ceph-mon[92422]: pgmap v3096: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 201 KiB/s wr, 10 op/s 2023-03-09T16:18:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:18:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4846.002800 secs 2023-03-09T16:18:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:18:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4846.002800 secs 2023-03-09T16:18:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:18:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4846.002800 secs 2023-03-09T16:18:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:18:56 smithi122 ceph-mon[93869]: pgmap v3097: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 102 KiB/s wr, 9 op/s 2023-03-09T16:18:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:18:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:18:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:18:56 smithi181 ceph-mon[94189]: pgmap v3097: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 102 KiB/s wr, 9 op/s 2023-03-09T16:18:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:18:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:18:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:18:56 smithi038 ceph-mon[92422]: pgmap v3097: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 102 KiB/s wr, 9 op/s 2023-03-09T16:18:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:18:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:18:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:18:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4848.230736 secs 2023-03-09T16:18:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:18:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4848.230736 secs 2023-03-09T16:18:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:18:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4848.230736 secs 2023-03-09T16:18:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:18:58 smithi122 ceph-mon[93869]: pgmap v3098: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 102 KiB/s wr, 11 op/s 2023-03-09T16:18:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:18:58 smithi181 ceph-mon[94189]: pgmap v3098: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 102 KiB/s wr, 11 op/s 2023-03-09T16:18:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:18:58 smithi038 ceph-mon[92422]: pgmap v3098: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 102 KiB/s wr, 11 op/s 2023-03-09T16:18:58.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5310.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:19:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:00 smithi122 ceph-mon[93869]: pgmap v3099: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 128 KiB/s wr, 10 op/s 2023-03-09T16:19:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:19:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:19:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:19:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:19:00.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4851.002860 secs 2023-03-09T16:19:00.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:19:00.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:19:00.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:19:00.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:19:00.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:19:00.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:19:00.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:19:00.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:19:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:00 smithi181 ceph-mon[94189]: pgmap v3099: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 128 KiB/s wr, 10 op/s 2023-03-09T16:19:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:19:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:19:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:19:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:19:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4851.002860 secs 2023-03-09T16:19:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:19:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:19:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:19:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:19:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:19:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:19:00.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:19:00.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:19:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:00 smithi038 ceph-mon[92422]: pgmap v3099: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 128 KiB/s wr, 10 op/s 2023-03-09T16:19:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:19:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:19:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:19:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:19:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4851.002860 secs 2023-03-09T16:19:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:19:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:19:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:19:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:19:00.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:19:00.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:19:00.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:19:00.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:19:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:01 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi038 to 197.2M 2023-03-09T16:19:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:01 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T16:19:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:01 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi038 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-03-09T16:19:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:01 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T16:19:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:01 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:19:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:01 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:19:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:01 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi038 to 197.2M 2023-03-09T16:19:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:01 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T16:19:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:01 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi038 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-03-09T16:19:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:01 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T16:19:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:01 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:19:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:01 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:19:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:01 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi038 to 197.2M 2023-03-09T16:19:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:01 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T16:19:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:01 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi038 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-03-09T16:19:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:01 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T16:19:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:01 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:19:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:01 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:19:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:02 smithi122 ceph-mon[93869]: pgmap v3100: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 95 KiB/s wr, 11 op/s 2023-03-09T16:19:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:19:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:19:02.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:19:02.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:19:02.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:02 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T16:19:02.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:02 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T16:19:02.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:19:02.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:19:02.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:19:02.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4853.230813 secs 2023-03-09T16:19:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:02 smithi181 ceph-mon[94189]: pgmap v3100: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 95 KiB/s wr, 11 op/s 2023-03-09T16:19:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:19:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:19:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:19:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:19:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:02 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T16:19:02.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:02 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T16:19:02.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:19:02.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:19:02.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:19:02.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4853.230813 secs 2023-03-09T16:19:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:02 smithi038 ceph-mon[92422]: pgmap v3100: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 95 KiB/s wr, 11 op/s 2023-03-09T16:19:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:19:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:19:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:19:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:19:02.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:02 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T16:19:02.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:02 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T16:19:02.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:19:02.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:19:02.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:19:02.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4853.230813 secs 2023-03-09T16:19:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5315.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:19:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:04 smithi122 ceph-mon[93869]: pgmap v3101: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 177 KiB/s wr, 10 op/s 2023-03-09T16:19:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:04 smithi181 ceph-mon[94189]: pgmap v3101: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 177 KiB/s wr, 10 op/s 2023-03-09T16:19:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:04 smithi038 ceph-mon[92422]: pgmap v3101: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 177 KiB/s wr, 10 op/s 2023-03-09T16:19:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:05 smithi122 ceph-mon[93869]: osdmap e406: 12 total, 12 up, 12 in 2023-03-09T16:19:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4856.002924 secs 2023-03-09T16:19:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:05 smithi181 ceph-mon[94189]: osdmap e406: 12 total, 12 up, 12 in 2023-03-09T16:19:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4856.002924 secs 2023-03-09T16:19:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:05 smithi038 ceph-mon[92422]: osdmap e406: 12 total, 12 up, 12 in 2023-03-09T16:19:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4856.002924 secs 2023-03-09T16:19:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:06 smithi122 ceph-mon[93869]: pgmap v3103: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 172 KiB/s wr, 24 op/s 2023-03-09T16:19:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:06 smithi181 ceph-mon[94189]: pgmap v3103: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 172 KiB/s wr, 24 op/s 2023-03-09T16:19:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:06 smithi038 ceph-mon[92422]: pgmap v3103: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 172 KiB/s wr, 24 op/s 2023-03-09T16:19:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4858.230913 secs 2023-03-09T16:19:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4858.230913 secs 2023-03-09T16:19:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4858.230913 secs 2023-03-09T16:19:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:08 smithi122 ceph-mon[93869]: pgmap v3104: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 203 KiB/s wr, 27 op/s 2023-03-09T16:19:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:08 smithi181 ceph-mon[94189]: pgmap v3104: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 203 KiB/s wr, 27 op/s 2023-03-09T16:19:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:08 smithi038 ceph-mon[92422]: pgmap v3104: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 203 KiB/s wr, 27 op/s 2023-03-09T16:19:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5320.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:19:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:10 smithi122 ceph-mon[93869]: pgmap v3105: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 167 KiB/s wr, 26 op/s 2023-03-09T16:19:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4861.003003 secs 2023-03-09T16:19:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:10 smithi181 ceph-mon[94189]: pgmap v3105: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 167 KiB/s wr, 26 op/s 2023-03-09T16:19:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4861.003003 secs 2023-03-09T16:19:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:10 smithi038 ceph-mon[92422]: pgmap v3105: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 167 KiB/s wr, 26 op/s 2023-03-09T16:19:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4861.003003 secs 2023-03-09T16:19:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:12 smithi122 ceph-mon[93869]: pgmap v3106: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 161 KiB/s wr, 25 op/s 2023-03-09T16:19:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4863.231053 secs 2023-03-09T16:19:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:12 smithi181 ceph-mon[94189]: pgmap v3106: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 161 KiB/s wr, 25 op/s 2023-03-09T16:19:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4863.231053 secs 2023-03-09T16:19:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:12 smithi038 ceph-mon[92422]: pgmap v3106: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 161 KiB/s wr, 25 op/s 2023-03-09T16:19:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4863.231053 secs 2023-03-09T16:19:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5325.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:19:14.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:14 smithi122 ceph-mon[93869]: pgmap v3107: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 77 KiB/s wr, 24 op/s 2023-03-09T16:19:14.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:14 smithi122 ceph-mon[93869]: osdmap e407: 12 total, 12 up, 12 in 2023-03-09T16:19:14.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:14 smithi181 ceph-mon[94189]: pgmap v3107: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 77 KiB/s wr, 24 op/s 2023-03-09T16:19:14.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:14 smithi181 ceph-mon[94189]: osdmap e407: 12 total, 12 up, 12 in 2023-03-09T16:19:14.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:14 smithi038 ceph-mon[92422]: pgmap v3107: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 77 KiB/s wr, 24 op/s 2023-03-09T16:19:14.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:14 smithi038 ceph-mon[92422]: osdmap e407: 12 total, 12 up, 12 in 2023-03-09T16:19:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4866.003065 secs 2023-03-09T16:19:15.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4866.003065 secs 2023-03-09T16:19:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4866.003065 secs 2023-03-09T16:19:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:16 smithi122 ceph-mon[93869]: pgmap v3109: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 51 KiB/s wr, 9 op/s 2023-03-09T16:19:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:16 smithi181 ceph-mon[94189]: pgmap v3109: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 51 KiB/s wr, 9 op/s 2023-03-09T16:19:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:16 smithi038 ceph-mon[92422]: pgmap v3109: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 51 KiB/s wr, 9 op/s 2023-03-09T16:19:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4868.231174 secs 2023-03-09T16:19:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4868.231174 secs 2023-03-09T16:19:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4868.231174 secs 2023-03-09T16:19:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:18 smithi122 ceph-mon[93869]: pgmap v3110: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 20 KiB/s wr, 6 op/s 2023-03-09T16:19:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:18 smithi181 ceph-mon[94189]: pgmap v3110: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 20 KiB/s wr, 6 op/s 2023-03-09T16:19:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:18 smithi038 ceph-mon[92422]: pgmap v3110: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 20 KiB/s wr, 6 op/s 2023-03-09T16:19:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5330.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:19:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:20 smithi122 ceph-mon[93869]: pgmap v3111: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T16:19:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4871.003126 secs 2023-03-09T16:19:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:20 smithi181 ceph-mon[94189]: pgmap v3111: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T16:19:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4871.003126 secs 2023-03-09T16:19:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:20 smithi038 ceph-mon[92422]: pgmap v3111: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T16:19:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4871.003126 secs 2023-03-09T16:19:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:22 smithi122 ceph-mon[93869]: pgmap v3112: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 6 op/s 2023-03-09T16:19:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4873.231278 secs 2023-03-09T16:19:22.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:22 smithi181 ceph-mon[94189]: pgmap v3112: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 6 op/s 2023-03-09T16:19:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4873.231278 secs 2023-03-09T16:19:22.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:22 smithi038 ceph-mon[92422]: pgmap v3112: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 6 op/s 2023-03-09T16:19:22.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4873.231278 secs 2023-03-09T16:19:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5335.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:19:24.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:24 smithi122 ceph-mon[93869]: pgmap v3113: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T16:19:24.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:24 smithi181 ceph-mon[94189]: pgmap v3113: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T16:19:24.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:24 smithi038 ceph-mon[92422]: pgmap v3113: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T16:19:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4876.003188 secs 2023-03-09T16:19:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4876.003188 secs 2023-03-09T16:19:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4876.003188 secs 2023-03-09T16:19:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:26 smithi122 ceph-mon[93869]: pgmap v3114: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 196 B/s wr, 6 op/s 2023-03-09T16:19:26.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:26 smithi181 ceph-mon[94189]: pgmap v3114: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 196 B/s wr, 6 op/s 2023-03-09T16:19:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:26 smithi038 ceph-mon[92422]: pgmap v3114: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 196 B/s wr, 6 op/s 2023-03-09T16:19:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4878.231359 secs 2023-03-09T16:19:27.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4878.231359 secs 2023-03-09T16:19:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4878.231359 secs 2023-03-09T16:19:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:28 smithi122 ceph-mon[93869]: pgmap v3115: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 7 op/s 2023-03-09T16:19:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:28 smithi181 ceph-mon[94189]: pgmap v3115: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 7 op/s 2023-03-09T16:19:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:28 smithi038 ceph-mon[92422]: pgmap v3115: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 7 op/s 2023-03-09T16:19:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5340.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:19:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:30 smithi122 ceph-mon[93869]: pgmap v3116: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 6 op/s 2023-03-09T16:19:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4881.003250 secs 2023-03-09T16:19:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:30 smithi181 ceph-mon[94189]: pgmap v3116: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 6 op/s 2023-03-09T16:19:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4881.003250 secs 2023-03-09T16:19:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:30 smithi038 ceph-mon[92422]: pgmap v3116: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 6 op/s 2023-03-09T16:19:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4881.003250 secs 2023-03-09T16:19:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:32 smithi122 ceph-mon[93869]: pgmap v3117: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 18 KiB/s wr, 7 op/s 2023-03-09T16:19:32.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:32 smithi181 ceph-mon[94189]: pgmap v3117: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 18 KiB/s wr, 7 op/s 2023-03-09T16:19:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:32 smithi038 ceph-mon[92422]: pgmap v3117: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 18 KiB/s wr, 7 op/s 2023-03-09T16:19:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5345.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:19:34.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:34 smithi122 ceph-mon[93869]: pgmap v3118: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 34 KiB/s wr, 6 op/s 2023-03-09T16:19:34.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:34 smithi181 ceph-mon[94189]: pgmap v3118: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 34 KiB/s wr, 6 op/s 2023-03-09T16:19:34.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:34 smithi038 ceph-mon[92422]: pgmap v3118: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 34 KiB/s wr, 6 op/s 2023-03-09T16:19:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4886.003311 secs 2023-03-09T16:19:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:35 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:19:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4886.003311 secs 2023-03-09T16:19:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:35 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:19:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4886.003311 secs 2023-03-09T16:19:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:35 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:19:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:36 smithi122 ceph-mon[93869]: pgmap v3119: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 34 KiB/s wr, 6 op/s 2023-03-09T16:19:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:36 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:19:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:19:36.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:19:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:36 smithi181 ceph-mon[94189]: pgmap v3119: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 34 KiB/s wr, 6 op/s 2023-03-09T16:19:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:36 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:19:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:19:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:19:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:36 smithi038 ceph-mon[92422]: pgmap v3119: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 34 KiB/s wr, 6 op/s 2023-03-09T16:19:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:36 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:19:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:19:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:19:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:38 smithi122 ceph-mon[93869]: pgmap v3120: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 71 KiB/s wr, 9 op/s 2023-03-09T16:19:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:38 smithi181 ceph-mon[94189]: pgmap v3120: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 71 KiB/s wr, 9 op/s 2023-03-09T16:19:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:38 smithi038 ceph-mon[92422]: pgmap v3120: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 71 KiB/s wr, 9 op/s 2023-03-09T16:19:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5350.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:19:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:40 smithi122 ceph-mon[93869]: pgmap v3121: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 72 KiB/s wr, 8 op/s 2023-03-09T16:19:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4891.003395 secs 2023-03-09T16:19:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:40 smithi181 ceph-mon[94189]: pgmap v3121: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 72 KiB/s wr, 8 op/s 2023-03-09T16:19:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4891.003395 secs 2023-03-09T16:19:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:40 smithi038 ceph-mon[92422]: pgmap v3121: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 72 KiB/s wr, 8 op/s 2023-03-09T16:19:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4891.003395 secs 2023-03-09T16:19:42.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:42 smithi122 ceph-mon[93869]: pgmap v3122: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 114 KiB/s wr, 11 op/s 2023-03-09T16:19:42.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:42 smithi181 ceph-mon[94189]: pgmap v3122: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 114 KiB/s wr, 11 op/s 2023-03-09T16:19:42.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:42 smithi038 ceph-mon[92422]: pgmap v3122: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 114 KiB/s wr, 11 op/s 2023-03-09T16:19:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:43 smithi038 ceph-mon[92422]: pgmap v3123: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 96 KiB/s wr, 9 op/s 2023-03-09T16:19:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5355.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:19:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:43 smithi122 ceph-mon[93869]: pgmap v3123: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 96 KiB/s wr, 9 op/s 2023-03-09T16:19:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:43 smithi181 ceph-mon[94189]: pgmap v3123: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 96 KiB/s wr, 9 op/s 2023-03-09T16:19:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4896.003479 secs 2023-03-09T16:19:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4896.003479 secs 2023-03-09T16:19:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4896.003479 secs 2023-03-09T16:19:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:46 smithi122 ceph-mon[93869]: pgmap v3124: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 81 KiB/s wr, 9 op/s 2023-03-09T16:19:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:46 smithi181 ceph-mon[94189]: pgmap v3124: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 81 KiB/s wr, 9 op/s 2023-03-09T16:19:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:46 smithi038 ceph-mon[92422]: pgmap v3124: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 81 KiB/s wr, 9 op/s 2023-03-09T16:19:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:48 smithi122 ceph-mon[93869]: pgmap v3125: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 170 KiB/s wr, 13 op/s 2023-03-09T16:19:48.741 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:48 smithi181 ceph-mon[94189]: pgmap v3125: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 170 KiB/s wr, 13 op/s 2023-03-09T16:19:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:48 smithi038 ceph-mon[92422]: pgmap v3125: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 170 KiB/s wr, 13 op/s 2023-03-09T16:19:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5360.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:19:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:50 smithi122 ceph-mon[93869]: pgmap v3126: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 132 KiB/s wr, 10 op/s 2023-03-09T16:19:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4901.003539 secs 2023-03-09T16:19:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:50 smithi181 ceph-mon[94189]: pgmap v3126: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 132 KiB/s wr, 10 op/s 2023-03-09T16:19:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4901.003539 secs 2023-03-09T16:19:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:50 smithi038 ceph-mon[92422]: pgmap v3126: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 132 KiB/s wr, 10 op/s 2023-03-09T16:19:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4901.003539 secs 2023-03-09T16:19:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:52 smithi122 ceph-mon[93869]: pgmap v3127: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 146 KiB/s wr, 12 op/s 2023-03-09T16:19:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:52 smithi181 ceph-mon[94189]: pgmap v3127: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 146 KiB/s wr, 12 op/s 2023-03-09T16:19:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:52 smithi038 ceph-mon[92422]: pgmap v3127: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 146 KiB/s wr, 12 op/s 2023-03-09T16:19:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5365.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:19:54.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:54 smithi122 ceph-mon[93869]: pgmap v3128: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 103 KiB/s wr, 9 op/s 2023-03-09T16:19:54.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:54 smithi181 ceph-mon[94189]: pgmap v3128: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 103 KiB/s wr, 9 op/s 2023-03-09T16:19:54.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:54 smithi038 ceph-mon[92422]: pgmap v3128: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 103 KiB/s wr, 9 op/s 2023-03-09T16:19:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4906.003600 secs 2023-03-09T16:19:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4906.003600 secs 2023-03-09T16:19:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4906.003600 secs 2023-03-09T16:19:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:56 smithi122 ceph-mon[93869]: pgmap v3129: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 103 KiB/s wr, 8 op/s 2023-03-09T16:19:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:56 smithi181 ceph-mon[94189]: pgmap v3129: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 103 KiB/s wr, 8 op/s 2023-03-09T16:19:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:56 smithi038 ceph-mon[92422]: pgmap v3129: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 103 KiB/s wr, 8 op/s 2023-03-09T16:19:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4908.231988 secs 2023-03-09T16:19:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4908.231988 secs 2023-03-09T16:19:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4908.231988 secs 2023-03-09T16:19:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:19:58 smithi122 ceph-mon[93869]: pgmap v3130: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 227 KiB/s wr, 12 op/s 2023-03-09T16:19:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:19:58 smithi181 ceph-mon[94189]: pgmap v3130: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 227 KiB/s wr, 12 op/s 2023-03-09T16:19:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:19:58 smithi038 ceph-mon[92422]: pgmap v3130: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 227 KiB/s wr, 12 op/s 2023-03-09T16:19:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5370.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:20:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:20:00 smithi122 ceph-mon[93869]: pgmap v3131: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 138 KiB/s wr, 8 op/s 2023-03-09T16:20:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:20:00 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:20:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:20:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:20:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:20:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:20:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:20:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4911.003720 secs 2023-03-09T16:20:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:20:00 smithi122 ceph-mon[93869]: overall HEALTH_WARN 2 MDSs report slow requests 2023-03-09T16:20:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:20:00 smithi181 ceph-mon[94189]: pgmap v3131: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 138 KiB/s wr, 8 op/s 2023-03-09T16:20:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:20:00 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:20:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:20:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:20:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:20:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:20:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:20:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4911.003720 secs 2023-03-09T16:20:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:20:00 smithi181 ceph-mon[94189]: overall HEALTH_WARN 2 MDSs report slow requests 2023-03-09T16:20:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:20:00 smithi038 ceph-mon[92422]: pgmap v3131: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 138 KiB/s wr, 8 op/s 2023-03-09T16:20:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:20:00 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:20:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:20:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:20:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:20:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:20:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:20:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4911.003720 secs 2023-03-09T16:20:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:20:00 smithi038 ceph-mon[92422]: overall HEALTH_WARN 2 MDSs report slow requests 2023-03-09T16:20:01.686 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:20:01 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:20:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:20:01 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:20:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:20:01 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:20:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:20:02 smithi122 ceph-mon[93869]: pgmap v3132: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 182 KiB/s wr, 13 op/s 2023-03-09T16:20:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:20:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4913.232063 secs 2023-03-09T16:20:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:20:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:20:02.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:20:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:20:02.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:20:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:20:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:20:02 smithi181 ceph-mon[94189]: pgmap v3132: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 182 KiB/s wr, 13 op/s 2023-03-09T16:20:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:20:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4913.232063 secs 2023-03-09T16:20:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:20:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:20:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:20:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:20:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:20:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:20:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:20:02 smithi038 ceph-mon[92422]: pgmap v3132: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 182 KiB/s wr, 13 op/s 2023-03-09T16:20:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:20:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4913.232063 secs 2023-03-09T16:20:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:20:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:20:02.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:20:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:20:02.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:20:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:20:03.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5375.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:20:04.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:20:04 smithi122 ceph-mon[93869]: pgmap v3133: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 188 KiB/s wr, 10 op/s 2023-03-09T16:20:04.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:20:04 smithi181 ceph-mon[94189]: pgmap v3133: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 188 KiB/s wr, 10 op/s 2023-03-09T16:20:04.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:20:04 smithi038 ceph-mon[92422]: pgmap v3133: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 188 KiB/s wr, 10 op/s 2023-03-09T16:20:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:20:05 smithi122 ceph-mon[93869]: osdmap e408: 12 total, 12 up, 12 in 2023-03-09T16:20:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:20:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4916.003781 secs 2023-03-09T16:20:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:20:05 smithi181 ceph-mon[94189]: osdmap e408: 12 total, 12 up, 12 in 2023-03-09T16:20:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:20:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4916.003781 secs 2023-03-09T16:20:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:20:05 smithi038 ceph-mon[92422]: osdmap e408: 12 total, 12 up, 12 in 2023-03-09T16:20:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:20:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4916.003781 secs 2023-03-09T16:20:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:20:06 smithi122 ceph-mon[93869]: pgmap v3135: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 252 KiB/s wr, 27 op/s 2023-03-09T16:20:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:20:06 smithi181 ceph-mon[94189]: pgmap v3135: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 252 KiB/s wr, 27 op/s 2023-03-09T16:20:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:20:06 smithi038 ceph-mon[92422]: pgmap v3135: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 252 KiB/s wr, 27 op/s 2023-03-09T16:20:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:20:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4918.232199 secs 2023-03-09T16:20:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:20:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4918.232199 secs 2023-03-09T16:20:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:20:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4918.232199 secs 2023-03-09T16:20:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:20:08 smithi122 ceph-mon[93869]: pgmap v3136: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 132 KiB/s wr, 26 op/s 2023-03-09T16:20:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:20:08 smithi181 ceph-mon[94189]: pgmap v3136: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 132 KiB/s wr, 26 op/s 2023-03-09T16:20:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:20:08 smithi038 ceph-mon[92422]: pgmap v3136: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 132 KiB/s wr, 26 op/s 2023-03-09T16:20:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5380.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:20:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:20:10 smithi122 ceph-mon[93869]: pgmap v3137: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 163 KiB/s wr, 27 op/s 2023-03-09T16:20:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:20:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4921.003843 secs 2023-03-09T16:20:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:20:10 smithi181 ceph-mon[94189]: pgmap v3137: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 163 KiB/s wr, 27 op/s 2023-03-09T16:20:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:20:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4921.003843 secs 2023-03-09T16:20:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:20:10 smithi038 ceph-mon[92422]: pgmap v3137: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 163 KiB/s wr, 27 op/s 2023-03-09T16:20:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:20:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4921.003843 secs 2023-03-09T16:20:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:20:12 smithi122 ceph-mon[93869]: pgmap v3138: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 115 KiB/s wr, 24 op/s 2023-03-09T16:20:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:20:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4923.232316 secs 2023-03-09T16:20:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:20:12 smithi181 ceph-mon[94189]: pgmap v3138: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 115 KiB/s wr, 24 op/s 2023-03-09T16:20:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:20:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4923.232316 secs 2023-03-09T16:20:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:20:12 smithi038 ceph-mon[92422]: pgmap v3138: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 115 KiB/s wr, 24 op/s 2023-03-09T16:20:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:20:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4923.232316 secs 2023-03-09T16:20:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5385.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:20:14.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:20:14 smithi122 ceph-mon[93869]: pgmap v3139: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 166 KiB/s wr, 25 op/s 2023-03-09T16:20:14.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:20:14 smithi122 ceph-mon[93869]: osdmap e409: 12 total, 12 up, 12 in 2023-03-09T16:20:14.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:20:14 smithi181 ceph-mon[94189]: pgmap v3139: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 166 KiB/s wr, 25 op/s 2023-03-09T16:20:14.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:20:14 smithi181 ceph-mon[94189]: osdmap e409: 12 total, 12 up, 12 in 2023-03-09T16:20:14.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:20:14 smithi038 ceph-mon[92422]: pgmap v3139: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 166 KiB/s wr, 25 op/s 2023-03-09T16:20:14.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:20:14 smithi038 ceph-mon[92422]: osdmap e409: 12 total, 12 up, 12 in 2023-03-09T16:20:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:20:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4926.003906 secs 2023-03-09T16:20:15.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:20:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4926.003906 secs 2023-03-09T16:20:15.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:20:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4926.003906 secs 2023-03-09T16:20:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:20:16 smithi122 ceph-mon[93869]: pgmap v3141: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 140 KiB/s wr, 10 op/s 2023-03-09T16:20:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:20:16 smithi181 ceph-mon[94189]: pgmap v3141: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 140 KiB/s wr, 10 op/s 2023-03-09T16:20:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:20:16 smithi038 ceph-mon[92422]: pgmap v3141: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 140 KiB/s wr, 10 op/s 2023-03-09T16:20:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:20:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4928.232470 secs 2023-03-09T16:20:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:20:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4928.232470 secs 2023-03-09T16:20:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:20:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4928.232470 secs 2023-03-09T16:20:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:20:18 smithi122 ceph-mon[93869]: pgmap v3142: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 110 KiB/s wr, 9 op/s 2023-03-09T16:20:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:20:18 smithi181 ceph-mon[94189]: pgmap v3142: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 110 KiB/s wr, 9 op/s 2023-03-09T16:20:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:20:18 smithi038 ceph-mon[92422]: pgmap v3142: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 110 KiB/s wr, 9 op/s 2023-03-09T16:20:18.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5390.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:20:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:20:20 smithi122 ceph-mon[93869]: pgmap v3143: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 80 KiB/s wr, 8 op/s 2023-03-09T16:20:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:20:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4931.003988 secs 2023-03-09T16:20:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:20:20 smithi181 ceph-mon[94189]: pgmap v3143: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 80 KiB/s wr, 8 op/s 2023-03-09T16:20:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:20:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4931.003988 secs 2023-03-09T16:20:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:20:20 smithi038 ceph-mon[92422]: pgmap v3143: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 80 KiB/s wr, 8 op/s 2023-03-09T16:20:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:20:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4931.003988 secs 2023-03-09T16:20:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:20:22 smithi122 ceph-mon[93869]: pgmap v3144: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 74 KiB/s wr, 8 op/s 2023-03-09T16:20:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:20:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4933.232586 secs 2023-03-09T16:20:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:20:22 smithi181 ceph-mon[94189]: pgmap v3144: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 74 KiB/s wr, 8 op/s 2023-03-09T16:20:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:20:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4933.232586 secs 2023-03-09T16:20:22.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:20:22 smithi038 ceph-mon[92422]: pgmap v3144: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 74 KiB/s wr, 8 op/s 2023-03-09T16:20:22.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:20:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4933.232586 secs 2023-03-09T16:20:23.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5395.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:20:24.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:20:24 smithi122 ceph-mon[93869]: pgmap v3145: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 20 KiB/s wr, 8 op/s 2023-03-09T16:20:24.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:20:24 smithi181 ceph-mon[94189]: pgmap v3145: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 20 KiB/s wr, 8 op/s 2023-03-09T16:20:24.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:20:24 smithi038 ceph-mon[92422]: pgmap v3145: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 20 KiB/s wr, 8 op/s 2023-03-09T16:20:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:20:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4936.004048 secs 2023-03-09T16:20:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:20:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4936.004048 secs 2023-03-09T16:20:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:20:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4936.004048 secs 2023-03-09T16:20:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:20:26 smithi122 ceph-mon[93869]: pgmap v3146: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 20 KiB/s wr, 8 op/s 2023-03-09T16:20:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:20:26 smithi181 ceph-mon[94189]: pgmap v3146: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 20 KiB/s wr, 8 op/s 2023-03-09T16:20:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:20:26 smithi038 ceph-mon[92422]: pgmap v3146: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 20 KiB/s wr, 8 op/s 2023-03-09T16:20:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:20:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4938.232723 secs 2023-03-09T16:20:27.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:20:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4938.232723 secs 2023-03-09T16:20:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:20:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4938.232723 secs 2023-03-09T16:20:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:20:28 smithi122 ceph-mon[93869]: pgmap v3147: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 17 KiB/s wr, 9 op/s 2023-03-09T16:20:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:20:28 smithi181 ceph-mon[94189]: pgmap v3147: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 17 KiB/s wr, 9 op/s 2023-03-09T16:20:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:20:28 smithi038 ceph-mon[92422]: pgmap v3147: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 17 KiB/s wr, 9 op/s 2023-03-09T16:20:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5400.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:20:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:20:30 smithi122 ceph-mon[93869]: pgmap v3148: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T16:20:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:20:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4941.004108 secs 2023-03-09T16:20:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:20:30 smithi181 ceph-mon[94189]: pgmap v3148: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T16:20:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:20:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4941.004108 secs 2023-03-09T16:20:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:20:30 smithi038 ceph-mon[92422]: pgmap v3148: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T16:20:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:20:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4941.004108 secs 2023-03-09T16:20:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:20:32 smithi122 ceph-mon[93869]: pgmap v3149: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 44 KiB/s wr, 9 op/s 2023-03-09T16:20:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:20:32 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 4943.232852 secs 2023-03-09T16:20:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:20:32 smithi181 ceph-mon[94189]: pgmap v3149: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 44 KiB/s wr, 9 op/s 2023-03-09T16:20:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:20:32 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 4943.232852 secs 2023-03-09T16:20:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:20:32 smithi038 ceph-mon[92422]: pgmap v3149: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 44 KiB/s wr, 9 op/s 2023-03-09T16:20:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:20:32 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 4943.232852 secs 2023-03-09T16:20:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:20:33 smithi122 ceph-mon[93869]: pgmap v3150: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 77 KiB/s wr, 7 op/s 2023-03-09T16:20:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:20:33 smithi181 ceph-mon[94189]: pgmap v3150: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 77 KiB/s wr, 7 op/s 2023-03-09T16:20:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:20:33 smithi038 ceph-mon[92422]: pgmap v3150: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 77 KiB/s wr, 7 op/s 2023-03-09T16:20:33.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5405.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:20:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:20:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4946.004180 secs 2023-03-09T16:20:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:20:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4946.004180 secs 2023-03-09T16:20:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:20:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4946.004180 secs 2023-03-09T16:20:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:20:36 smithi122 ceph-mon[93869]: pgmap v3151: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 61 KiB/s wr, 6 op/s 2023-03-09T16:20:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:20:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:20:36.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:20:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:20:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:20:36 smithi181 ceph-mon[94189]: pgmap v3151: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 61 KiB/s wr, 6 op/s 2023-03-09T16:20:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:20:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:20:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:20:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:20:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:20:36 smithi038 ceph-mon[92422]: pgmap v3151: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 61 KiB/s wr, 6 op/s 2023-03-09T16:20:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:20:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:20:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:20:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:20:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:20:37 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 4948.232996 secs 2023-03-09T16:20:37.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:20:37 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 4948.232996 secs 2023-03-09T16:20:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:20:37 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 4948.232996 secs 2023-03-09T16:20:38.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:20:38 smithi122 ceph-mon[93869]: pgmap v3152: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 66 KiB/s wr, 8 op/s 2023-03-09T16:20:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:20:38 smithi181 ceph-mon[94189]: pgmap v3152: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 66 KiB/s wr, 8 op/s 2023-03-09T16:20:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:20:38 smithi038 ceph-mon[92422]: pgmap v3152: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 66 KiB/s wr, 8 op/s 2023-03-09T16:20:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5410.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:20:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:20:40 smithi122 ceph-mon[93869]: pgmap v3153: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 100 KiB/s wr, 7 op/s 2023-03-09T16:20:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:20:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4951.004239 secs 2023-03-09T16:20:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:20:40 smithi181 ceph-mon[94189]: pgmap v3153: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 100 KiB/s wr, 7 op/s 2023-03-09T16:20:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:20:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4951.004239 secs 2023-03-09T16:20:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:20:40 smithi038 ceph-mon[92422]: pgmap v3153: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 100 KiB/s wr, 7 op/s 2023-03-09T16:20:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:20:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4951.004239 secs 2023-03-09T16:20:42.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:20:42 smithi122 ceph-mon[93869]: pgmap v3154: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 77 KiB/s wr, 10 op/s 2023-03-09T16:20:42.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:20:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4953.233129 secs 2023-03-09T16:20:42.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:20:42 smithi181 ceph-mon[94189]: pgmap v3154: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 77 KiB/s wr, 10 op/s 2023-03-09T16:20:42.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:20:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4953.233129 secs 2023-03-09T16:20:42.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:20:42 smithi038 ceph-mon[92422]: pgmap v3154: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 77 KiB/s wr, 10 op/s 2023-03-09T16:20:42.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:20:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4953.233129 secs 2023-03-09T16:20:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5415.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:20:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:20:44 smithi122 ceph-mon[93869]: pgmap v3155: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 163 KiB/s wr, 9 op/s 2023-03-09T16:20:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:20:44 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:20:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:20:44 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:20:44.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:20:44 smithi181 ceph-mon[94189]: pgmap v3155: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 163 KiB/s wr, 9 op/s 2023-03-09T16:20:44.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:20:44 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:20:44.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:20:44 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:20:44.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:20:44 smithi038 ceph-mon[92422]: pgmap v3155: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 163 KiB/s wr, 9 op/s 2023-03-09T16:20:44.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:20:44 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:20:44.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:20:44 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:20:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:20:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4956.004298 secs 2023-03-09T16:20:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:20:45 smithi122 ceph-mon[93869]: pgmap v3156: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 129 KiB/s wr, 8 op/s 2023-03-09T16:20:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:20:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4956.004298 secs 2023-03-09T16:20:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:20:45 smithi181 ceph-mon[94189]: pgmap v3156: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 129 KiB/s wr, 8 op/s 2023-03-09T16:20:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:20:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4956.004298 secs 2023-03-09T16:20:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:20:45 smithi038 ceph-mon[92422]: pgmap v3156: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 129 KiB/s wr, 8 op/s 2023-03-09T16:20:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:20:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4958.233244 secs 2023-03-09T16:20:47.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:20:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4958.233244 secs 2023-03-09T16:20:47.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:20:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4958.233244 secs 2023-03-09T16:20:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:20:47 smithi122 ceph-mon[93869]: pgmap v3157: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 134 KiB/s wr, 11 op/s 2023-03-09T16:20:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:20:47 smithi181 ceph-mon[94189]: pgmap v3157: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 134 KiB/s wr, 11 op/s 2023-03-09T16:20:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:20:47 smithi038 ceph-mon[92422]: pgmap v3157: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 134 KiB/s wr, 11 op/s 2023-03-09T16:20:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5420.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:20:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:20:50 smithi122 ceph-mon[93869]: pgmap v3158: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 171 KiB/s wr, 10 op/s 2023-03-09T16:20:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:20:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4961.004358 secs 2023-03-09T16:20:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:20:50 smithi181 ceph-mon[94189]: pgmap v3158: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 171 KiB/s wr, 10 op/s 2023-03-09T16:20:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:20:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4961.004358 secs 2023-03-09T16:20:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:20:50 smithi038 ceph-mon[92422]: pgmap v3158: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 171 KiB/s wr, 10 op/s 2023-03-09T16:20:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:20:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4961.004358 secs 2023-03-09T16:20:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:20:52 smithi122 ceph-mon[93869]: pgmap v3159: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 137 KiB/s wr, 11 op/s 2023-03-09T16:20:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:20:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4963.233318 secs 2023-03-09T16:20:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:20:52 smithi181 ceph-mon[94189]: pgmap v3159: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 137 KiB/s wr, 11 op/s 2023-03-09T16:20:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:20:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4963.233318 secs 2023-03-09T16:20:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:20:52 smithi038 ceph-mon[92422]: pgmap v3159: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 137 KiB/s wr, 11 op/s 2023-03-09T16:20:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:20:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4963.233318 secs 2023-03-09T16:20:53.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5425.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:20:54.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:20:54 smithi122 ceph-mon[93869]: pgmap v3160: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 264 KiB/s wr, 10 op/s 2023-03-09T16:20:54.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:20:54 smithi181 ceph-mon[94189]: pgmap v3160: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 264 KiB/s wr, 10 op/s 2023-03-09T16:20:54.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:20:54 smithi038 ceph-mon[92422]: pgmap v3160: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 264 KiB/s wr, 10 op/s 2023-03-09T16:20:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:20:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4966.004420 secs 2023-03-09T16:20:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:20:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4966.004420 secs 2023-03-09T16:20:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:20:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4966.004420 secs 2023-03-09T16:20:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:20:56 smithi122 ceph-mon[93869]: pgmap v3161: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 179 KiB/s wr, 8 op/s 2023-03-09T16:20:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:20:56 smithi181 ceph-mon[94189]: pgmap v3161: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 179 KiB/s wr, 8 op/s 2023-03-09T16:20:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:20:56 smithi038 ceph-mon[92422]: pgmap v3161: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 179 KiB/s wr, 8 op/s 2023-03-09T16:20:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:20:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4968.233455 secs 2023-03-09T16:20:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:20:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4968.233455 secs 2023-03-09T16:20:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:20:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4968.233455 secs 2023-03-09T16:20:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:20:58 smithi122 ceph-mon[93869]: pgmap v3162: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 188 KiB/s wr, 11 op/s 2023-03-09T16:20:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:20:58 smithi181 ceph-mon[94189]: pgmap v3162: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 188 KiB/s wr, 11 op/s 2023-03-09T16:20:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:20:58 smithi038 ceph-mon[92422]: pgmap v3162: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 188 KiB/s wr, 11 op/s 2023-03-09T16:20:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5430.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:21:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:21:00 smithi122 ceph-mon[93869]: pgmap v3163: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 222 KiB/s wr, 9 op/s 2023-03-09T16:21:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:21:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4971.004479 secs 2023-03-09T16:21:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:21:00 smithi181 ceph-mon[94189]: pgmap v3163: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 222 KiB/s wr, 9 op/s 2023-03-09T16:21:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:21:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4971.004479 secs 2023-03-09T16:21:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:21:00 smithi038 ceph-mon[92422]: pgmap v3163: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 222 KiB/s wr, 9 op/s 2023-03-09T16:21:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:21:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4971.004479 secs 2023-03-09T16:21:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:21:02 smithi122 ceph-mon[93869]: pgmap v3164: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 203 KiB/s wr, 12 op/s 2023-03-09T16:21:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:21:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4973.233591 secs 2023-03-09T16:21:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:21:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:21:02.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:21:02 smithi181 ceph-mon[94189]: pgmap v3164: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 203 KiB/s wr, 12 op/s 2023-03-09T16:21:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:21:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4973.233591 secs 2023-03-09T16:21:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:21:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:21:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:21:02 smithi038 ceph-mon[92422]: pgmap v3164: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 203 KiB/s wr, 12 op/s 2023-03-09T16:21:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:21:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4973.233591 secs 2023-03-09T16:21:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:21:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:21:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5435.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:21:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:21:03 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:21:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:21:03 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:21:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:21:03 smithi122 ceph-mon[93869]: pgmap v3165: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 263 KiB/s wr, 11 op/s 2023-03-09T16:21:03.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:21:03 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:21:03.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:21:03 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:21:03.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:21:03 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:21:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:21:03 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:21:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:21:03 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:21:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:21:03 smithi181 ceph-mon[94189]: pgmap v3165: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 263 KiB/s wr, 11 op/s 2023-03-09T16:21:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:21:03 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:21:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:21:03 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:21:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:21:03 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:21:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:21:03 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:21:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:21:03 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:21:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:21:03 smithi038 ceph-mon[92422]: pgmap v3165: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 263 KiB/s wr, 11 op/s 2023-03-09T16:21:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:21:03 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:21:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:21:03 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:21:04.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:21:03 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:21:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:21:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4976.004568 secs 2023-03-09T16:21:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:21:05 smithi122 ceph-mon[93869]: osdmap e410: 12 total, 12 up, 12 in 2023-03-09T16:21:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:21:05 smithi122 ceph-mon[93869]: pgmap v3167: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 183 KiB/s wr, 26 op/s 2023-03-09T16:21:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:21:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4976.004568 secs 2023-03-09T16:21:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:21:05 smithi181 ceph-mon[94189]: osdmap e410: 12 total, 12 up, 12 in 2023-03-09T16:21:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:21:05 smithi181 ceph-mon[94189]: pgmap v3167: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 183 KiB/s wr, 26 op/s 2023-03-09T16:21:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:21:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4976.004568 secs 2023-03-09T16:21:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:21:05 smithi038 ceph-mon[92422]: osdmap e410: 12 total, 12 up, 12 in 2023-03-09T16:21:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:21:05 smithi038 ceph-mon[92422]: pgmap v3167: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 183 KiB/s wr, 26 op/s 2023-03-09T16:21:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:21:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4978.233707 secs 2023-03-09T16:21:07.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:21:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4978.233707 secs 2023-03-09T16:21:07.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:21:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4978.233707 secs 2023-03-09T16:21:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:21:07 smithi122 ceph-mon[93869]: pgmap v3168: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 177 KiB/s wr, 26 op/s 2023-03-09T16:21:08.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:21:07 smithi181 ceph-mon[94189]: pgmap v3168: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 177 KiB/s wr, 26 op/s 2023-03-09T16:21:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:21:07 smithi038 ceph-mon[92422]: pgmap v3168: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 177 KiB/s wr, 26 op/s 2023-03-09T16:21:08.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5440.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:21:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:21:10 smithi122 ceph-mon[93869]: pgmap v3169: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 161 KiB/s wr, 26 op/s 2023-03-09T16:21:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:21:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4981.004627 secs 2023-03-09T16:21:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:21:10 smithi181 ceph-mon[94189]: pgmap v3169: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 161 KiB/s wr, 26 op/s 2023-03-09T16:21:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:21:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4981.004627 secs 2023-03-09T16:21:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:21:10 smithi038 ceph-mon[92422]: pgmap v3169: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 161 KiB/s wr, 26 op/s 2023-03-09T16:21:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:21:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4981.004627 secs 2023-03-09T16:21:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:21:12 smithi122 ceph-mon[93869]: pgmap v3170: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 135 KiB/s wr, 23 op/s 2023-03-09T16:21:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:21:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4983.233822 secs 2023-03-09T16:21:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:21:12 smithi181 ceph-mon[94189]: pgmap v3170: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 135 KiB/s wr, 23 op/s 2023-03-09T16:21:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:21:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4983.233822 secs 2023-03-09T16:21:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:21:12 smithi038 ceph-mon[92422]: pgmap v3170: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 135 KiB/s wr, 23 op/s 2023-03-09T16:21:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:21:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4983.233822 secs 2023-03-09T16:21:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5445.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:21:14.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:21:14 smithi122 ceph-mon[93869]: pgmap v3171: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 87 KiB/s wr, 22 op/s 2023-03-09T16:21:14.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:21:14 smithi122 ceph-mon[93869]: osdmap e411: 12 total, 12 up, 12 in 2023-03-09T16:21:14.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:21:14 smithi181 ceph-mon[94189]: pgmap v3171: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 87 KiB/s wr, 22 op/s 2023-03-09T16:21:14.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:21:14 smithi181 ceph-mon[94189]: osdmap e411: 12 total, 12 up, 12 in 2023-03-09T16:21:14.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:21:14 smithi038 ceph-mon[92422]: pgmap v3171: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 87 KiB/s wr, 22 op/s 2023-03-09T16:21:14.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:21:14 smithi038 ceph-mon[92422]: osdmap e411: 12 total, 12 up, 12 in 2023-03-09T16:21:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:21:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4986.004713 secs 2023-03-09T16:21:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:21:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4986.004713 secs 2023-03-09T16:21:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:21:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4986.004713 secs 2023-03-09T16:21:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:21:16 smithi122 ceph-mon[93869]: pgmap v3173: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 62 KiB/s wr, 8 op/s 2023-03-09T16:21:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:21:16 smithi181 ceph-mon[94189]: pgmap v3173: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 62 KiB/s wr, 8 op/s 2023-03-09T16:21:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:21:16 smithi038 ceph-mon[92422]: pgmap v3173: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 62 KiB/s wr, 8 op/s 2023-03-09T16:21:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:21:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4988.233942 secs 2023-03-09T16:21:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:21:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4988.233942 secs 2023-03-09T16:21:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:21:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4988.233942 secs 2023-03-09T16:21:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:21:18 smithi122 ceph-mon[93869]: pgmap v3174: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 58 KiB/s wr, 7 op/s 2023-03-09T16:21:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:21:18 smithi181 ceph-mon[94189]: pgmap v3174: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 58 KiB/s wr, 7 op/s 2023-03-09T16:21:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:21:18 smithi038 ceph-mon[92422]: pgmap v3174: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 58 KiB/s wr, 7 op/s 2023-03-09T16:21:18.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5450.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:21:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:21:20 smithi122 ceph-mon[93869]: pgmap v3175: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 63 KiB/s wr, 7 op/s 2023-03-09T16:21:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:21:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4991.004773 secs 2023-03-09T16:21:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:21:20 smithi181 ceph-mon[94189]: pgmap v3175: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 63 KiB/s wr, 7 op/s 2023-03-09T16:21:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:21:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4991.004773 secs 2023-03-09T16:21:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:21:20 smithi038 ceph-mon[92422]: pgmap v3175: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 63 KiB/s wr, 7 op/s 2023-03-09T16:21:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:21:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4991.004773 secs 2023-03-09T16:21:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:21:22 smithi122 ceph-mon[93869]: pgmap v3176: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 63 KiB/s wr, 7 op/s 2023-03-09T16:21:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:21:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4993.234036 secs 2023-03-09T16:21:22.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:21:22 smithi181 ceph-mon[94189]: pgmap v3176: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 63 KiB/s wr, 7 op/s 2023-03-09T16:21:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:21:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4993.234036 secs 2023-03-09T16:21:22.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:21:22 smithi038 ceph-mon[92422]: pgmap v3176: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 63 KiB/s wr, 7 op/s 2023-03-09T16:21:22.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:21:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4993.234036 secs 2023-03-09T16:21:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5455.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:21:24.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:21:24 smithi122 ceph-mon[93869]: pgmap v3177: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 38 KiB/s wr, 7 op/s 2023-03-09T16:21:24.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:21:24 smithi181 ceph-mon[94189]: pgmap v3177: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 38 KiB/s wr, 7 op/s 2023-03-09T16:21:24.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:21:24 smithi038 ceph-mon[92422]: pgmap v3177: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 38 KiB/s wr, 7 op/s 2023-03-09T16:21:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:21:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4996.004835 secs 2023-03-09T16:21:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:21:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4996.004835 secs 2023-03-09T16:21:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:21:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4996.004835 secs 2023-03-09T16:21:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:21:26 smithi122 ceph-mon[93869]: pgmap v3178: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 KiB/s wr, 7 op/s 2023-03-09T16:21:26.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:21:26 smithi181 ceph-mon[94189]: pgmap v3178: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 KiB/s wr, 7 op/s 2023-03-09T16:21:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:21:26 smithi038 ceph-mon[92422]: pgmap v3178: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 KiB/s wr, 7 op/s 2023-03-09T16:21:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:21:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 4998.234142 secs 2023-03-09T16:21:27.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:21:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 4998.234142 secs 2023-03-09T16:21:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:21:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 4998.234142 secs 2023-03-09T16:21:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:21:28 smithi122 ceph-mon[93869]: pgmap v3179: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 31 KiB/s wr, 8 op/s 2023-03-09T16:21:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:21:28 smithi181 ceph-mon[94189]: pgmap v3179: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 31 KiB/s wr, 8 op/s 2023-03-09T16:21:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:21:28 smithi038 ceph-mon[92422]: pgmap v3179: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 31 KiB/s wr, 8 op/s 2023-03-09T16:21:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5460.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:21:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:21:30 smithi122 ceph-mon[93869]: pgmap v3180: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 58 KiB/s wr, 7 op/s 2023-03-09T16:21:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:21:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5001.004894 secs 2023-03-09T16:21:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:21:30 smithi181 ceph-mon[94189]: pgmap v3180: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 58 KiB/s wr, 7 op/s 2023-03-09T16:21:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:21:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5001.004894 secs 2023-03-09T16:21:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:21:30 smithi038 ceph-mon[92422]: pgmap v3180: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 58 KiB/s wr, 7 op/s 2023-03-09T16:21:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:21:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5001.004894 secs 2023-03-09T16:21:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:21:32 smithi122 ceph-mon[93869]: pgmap v3181: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 55 KiB/s wr, 9 op/s 2023-03-09T16:21:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:21:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5003.234280 secs 2023-03-09T16:21:32.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:21:32 smithi181 ceph-mon[94189]: pgmap v3181: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 55 KiB/s wr, 9 op/s 2023-03-09T16:21:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:21:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5003.234280 secs 2023-03-09T16:21:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:21:32 smithi038 ceph-mon[92422]: pgmap v3181: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 55 KiB/s wr, 9 op/s 2023-03-09T16:21:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:21:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5003.234280 secs 2023-03-09T16:21:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5465.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:21:34.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:21:34 smithi122 ceph-mon[93869]: pgmap v3182: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 62 KiB/s wr, 7 op/s 2023-03-09T16:21:34.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:21:34 smithi181 ceph-mon[94189]: pgmap v3182: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 62 KiB/s wr, 7 op/s 2023-03-09T16:21:34.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:21:34 smithi038 ceph-mon[92422]: pgmap v3182: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 62 KiB/s wr, 7 op/s 2023-03-09T16:21:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:21:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5006.004955 secs 2023-03-09T16:21:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:21:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5006.004955 secs 2023-03-09T16:21:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:21:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5006.004955 secs 2023-03-09T16:21:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:21:36 smithi122 ceph-mon[93869]: pgmap v3183: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 135 KiB/s wr, 8 op/s 2023-03-09T16:21:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:21:36 smithi181 ceph-mon[94189]: pgmap v3183: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 135 KiB/s wr, 8 op/s 2023-03-09T16:21:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:21:36 smithi038 ceph-mon[92422]: pgmap v3183: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 135 KiB/s wr, 8 op/s 2023-03-09T16:21:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:21:37 smithi038 ceph-mon[92422]: pgmap v3184: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 144 KiB/s wr, 11 op/s 2023-03-09T16:21:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:21:37 smithi122 ceph-mon[93869]: pgmap v3184: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 144 KiB/s wr, 11 op/s 2023-03-09T16:21:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:21:37 smithi181 ceph-mon[94189]: pgmap v3184: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 144 KiB/s wr, 11 op/s 2023-03-09T16:21:38.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5470.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:21:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:21:39 smithi122 ceph-mon[93869]: pgmap v3185: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 144 KiB/s wr, 10 op/s 2023-03-09T16:21:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:21:39 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:21:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:21:39 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5011.004991 secs 2023-03-09T16:21:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:21:39 smithi181 ceph-mon[94189]: pgmap v3185: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 144 KiB/s wr, 10 op/s 2023-03-09T16:21:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:21:39 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:21:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:21:39 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5011.004991 secs 2023-03-09T16:21:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:21:39 smithi038 ceph-mon[92422]: pgmap v3185: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 144 KiB/s wr, 10 op/s 2023-03-09T16:21:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:21:39 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:21:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:21:39 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5011.004991 secs 2023-03-09T16:21:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:21:40 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:21:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:21:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:21:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:21:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:21:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:21:40 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:21:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:21:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:21:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:21:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:21:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:21:40 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:21:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:21:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:21:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:21:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:21:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:21:41 smithi122 ceph-mon[93869]: pgmap v3186: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 135 KiB/s wr, 12 op/s 2023-03-09T16:21:42.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:21:41 smithi181 ceph-mon[94189]: pgmap v3186: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 135 KiB/s wr, 12 op/s 2023-03-09T16:21:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:21:41 smithi038 ceph-mon[92422]: pgmap v3186: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 135 KiB/s wr, 12 op/s 2023-03-09T16:21:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5475.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:21:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:21:43 smithi122 ceph-mon[93869]: pgmap v3187: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 109 KiB/s wr, 10 op/s 2023-03-09T16:21:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:21:43 smithi181 ceph-mon[94189]: pgmap v3187: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 109 KiB/s wr, 10 op/s 2023-03-09T16:21:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:21:43 smithi038 ceph-mon[92422]: pgmap v3187: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 109 KiB/s wr, 10 op/s 2023-03-09T16:21:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:21:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5016.005051 secs 2023-03-09T16:21:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:21:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5016.005051 secs 2023-03-09T16:21:45.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:21:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5016.005051 secs 2023-03-09T16:21:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:21:46 smithi122 ceph-mon[93869]: pgmap v3188: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 228 KiB/s wr, 11 op/s 2023-03-09T16:21:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:21:46 smithi181 ceph-mon[94189]: pgmap v3188: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 228 KiB/s wr, 11 op/s 2023-03-09T16:21:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:21:46 smithi038 ceph-mon[92422]: pgmap v3188: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 228 KiB/s wr, 11 op/s 2023-03-09T16:21:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:21:48 smithi122 ceph-mon[93869]: pgmap v3189: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 160 KiB/s wr, 13 op/s 2023-03-09T16:21:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:21:48 smithi181 ceph-mon[94189]: pgmap v3189: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 160 KiB/s wr, 13 op/s 2023-03-09T16:21:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:21:48 smithi038 ceph-mon[92422]: pgmap v3189: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 160 KiB/s wr, 13 op/s 2023-03-09T16:21:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5480.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:21:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:21:50 smithi122 ceph-mon[93869]: pgmap v3190: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 151 KiB/s wr, 10 op/s 2023-03-09T16:21:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:21:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5021.005110 secs 2023-03-09T16:21:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:21:50 smithi181 ceph-mon[94189]: pgmap v3190: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 151 KiB/s wr, 10 op/s 2023-03-09T16:21:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:21:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5021.005110 secs 2023-03-09T16:21:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:21:50 smithi038 ceph-mon[92422]: pgmap v3190: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 151 KiB/s wr, 10 op/s 2023-03-09T16:21:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:21:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5021.005110 secs 2023-03-09T16:21:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:21:51 smithi122 ceph-mon[93869]: pgmap v3191: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 198 KiB/s wr, 12 op/s 2023-03-09T16:21:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:21:51 smithi181 ceph-mon[94189]: pgmap v3191: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 198 KiB/s wr, 12 op/s 2023-03-09T16:21:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:21:51 smithi038 ceph-mon[92422]: pgmap v3191: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 198 KiB/s wr, 12 op/s 2023-03-09T16:21:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:21:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5023.234754 secs 2023-03-09T16:21:52.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:21:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5023.234754 secs 2023-03-09T16:21:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:21:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5023.234754 secs 2023-03-09T16:21:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5485.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:21:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:21:53 smithi122 ceph-mon[93869]: pgmap v3192: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 180 KiB/s wr, 9 op/s 2023-03-09T16:21:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:21:53 smithi181 ceph-mon[94189]: pgmap v3192: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 180 KiB/s wr, 9 op/s 2023-03-09T16:21:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:21:53 smithi038 ceph-mon[92422]: pgmap v3192: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 180 KiB/s wr, 9 op/s 2023-03-09T16:21:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:21:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5026.005192 secs 2023-03-09T16:21:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:21:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5026.005192 secs 2023-03-09T16:21:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:21:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5026.005192 secs 2023-03-09T16:21:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:21:56 smithi122 ceph-mon[93869]: pgmap v3193: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 249 KiB/s wr, 9 op/s 2023-03-09T16:21:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:21:56 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:21:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:21:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:21:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:21:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:21:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:21:56 smithi181 ceph-mon[94189]: pgmap v3193: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 249 KiB/s wr, 9 op/s 2023-03-09T16:21:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:21:56 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:21:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:21:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:21:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:21:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:21:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:21:56 smithi038 ceph-mon[92422]: pgmap v3193: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 249 KiB/s wr, 9 op/s 2023-03-09T16:21:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:21:56 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:21:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:21:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:21:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:21:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:21:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:21:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5028.234872 secs 2023-03-09T16:21:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:21:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5028.234872 secs 2023-03-09T16:21:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:21:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5028.234872 secs 2023-03-09T16:21:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:21:58 smithi122 ceph-mon[93869]: pgmap v3194: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 122 KiB/s wr, 10 op/s 2023-03-09T16:21:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:21:58 smithi181 ceph-mon[94189]: pgmap v3194: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 122 KiB/s wr, 10 op/s 2023-03-09T16:21:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:21:58 smithi038 ceph-mon[92422]: pgmap v3194: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 122 KiB/s wr, 10 op/s 2023-03-09T16:21:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5490.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:22:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:22:00 smithi122 ceph-mon[93869]: pgmap v3195: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 117 KiB/s wr, 7 op/s 2023-03-09T16:22:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:22:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5031.005255 secs 2023-03-09T16:22:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:22:00 smithi181 ceph-mon[94189]: pgmap v3195: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 117 KiB/s wr, 7 op/s 2023-03-09T16:22:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:22:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5031.005255 secs 2023-03-09T16:22:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:22:00 smithi038 ceph-mon[92422]: pgmap v3195: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 117 KiB/s wr, 7 op/s 2023-03-09T16:22:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:22:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5031.005255 secs 2023-03-09T16:22:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:22:02 smithi122 ceph-mon[93869]: pgmap v3196: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 161 KiB/s wr, 13 op/s 2023-03-09T16:22:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:22:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5033.234990 secs 2023-03-09T16:22:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:22:02 smithi181 ceph-mon[94189]: pgmap v3196: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 161 KiB/s wr, 13 op/s 2023-03-09T16:22:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:22:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5033.234990 secs 2023-03-09T16:22:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:22:02 smithi038 ceph-mon[92422]: pgmap v3196: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 161 KiB/s wr, 13 op/s 2023-03-09T16:22:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:22:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5033.234990 secs 2023-03-09T16:22:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:22:03 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:22:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:22:03 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:22:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:22:03 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:22:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5495.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:22:04.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:22:04 smithi122 ceph-mon[93869]: pgmap v3197: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 113 KiB/s wr, 10 op/s 2023-03-09T16:22:04.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:22:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:22:04.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:22:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:22:04.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:22:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:22:04.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:22:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:22:04.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:22:04 smithi181 ceph-mon[94189]: pgmap v3197: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 113 KiB/s wr, 10 op/s 2023-03-09T16:22:04.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:22:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:22:04.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:22:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:22:04.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:22:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:22:04.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:22:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:22:04.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:22:04 smithi038 ceph-mon[92422]: pgmap v3197: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 113 KiB/s wr, 10 op/s 2023-03-09T16:22:04.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:22:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:22:04.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:22:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:22:04.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:22:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:22:04.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:22:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:22:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:22:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5036.005322 secs 2023-03-09T16:22:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:22:05 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:22:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:22:05 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:22:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:22:05 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:22:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:22:05 smithi122 ceph-mon[93869]: pgmap v3198: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 148 KiB/s wr, 23 op/s 2023-03-09T16:22:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:22:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5036.005322 secs 2023-03-09T16:22:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:22:05 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:22:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:22:05 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:22:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:22:05 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:22:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:22:05 smithi181 ceph-mon[94189]: pgmap v3198: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 148 KiB/s wr, 23 op/s 2023-03-09T16:22:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:22:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5036.005322 secs 2023-03-09T16:22:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:22:05 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:22:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:22:05 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:22:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:22:05 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:22:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:22:05 smithi038 ceph-mon[92422]: pgmap v3198: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 148 KiB/s wr, 23 op/s 2023-03-09T16:22:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:22:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5038.235115 secs 2023-03-09T16:22:07.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:22:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5038.235115 secs 2023-03-09T16:22:07.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:22:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5038.235115 secs 2023-03-09T16:22:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:22:07 smithi122 ceph-mon[93869]: pgmap v3199: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 84 KiB/s wr, 24 op/s 2023-03-09T16:22:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:22:07 smithi181 ceph-mon[94189]: pgmap v3199: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 84 KiB/s wr, 24 op/s 2023-03-09T16:22:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:22:07 smithi038 ceph-mon[92422]: pgmap v3199: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 84 KiB/s wr, 24 op/s 2023-03-09T16:22:08.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5500.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:22:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:22:09 smithi122 ceph-mon[93869]: pgmap v3200: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 83 KiB/s wr, 22 op/s 2023-03-09T16:22:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:22:09 smithi122 ceph-mon[93869]: osdmap e412: 12 total, 12 up, 12 in 2023-03-09T16:22:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:22:09 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5041.005384 secs 2023-03-09T16:22:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:22:09 smithi181 ceph-mon[94189]: pgmap v3200: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 83 KiB/s wr, 22 op/s 2023-03-09T16:22:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:22:09 smithi181 ceph-mon[94189]: osdmap e412: 12 total, 12 up, 12 in 2023-03-09T16:22:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:22:09 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5041.005384 secs 2023-03-09T16:22:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:22:09 smithi038 ceph-mon[92422]: pgmap v3200: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 83 KiB/s wr, 22 op/s 2023-03-09T16:22:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:22:09 smithi038 ceph-mon[92422]: osdmap e412: 12 total, 12 up, 12 in 2023-03-09T16:22:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:22:09 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5041.005384 secs 2023-03-09T16:22:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:22:11 smithi122 ceph-mon[93869]: pgmap v3202: 129 pgs: 1 active+clean+snaptrim, 18 active+clean+snaptrim_wait, 110 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 73 KiB/s wr, 23 op/s 2023-03-09T16:22:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:22:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5043.235231 secs 2023-03-09T16:22:12.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:22:11 smithi181 ceph-mon[94189]: pgmap v3202: 129 pgs: 1 active+clean+snaptrim, 18 active+clean+snaptrim_wait, 110 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 73 KiB/s wr, 23 op/s 2023-03-09T16:22:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:22:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5043.235231 secs 2023-03-09T16:22:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:22:11 smithi038 ceph-mon[92422]: pgmap v3202: 129 pgs: 1 active+clean+snaptrim, 18 active+clean+snaptrim_wait, 110 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 73 KiB/s wr, 23 op/s 2023-03-09T16:22:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:22:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5043.235231 secs 2023-03-09T16:22:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5505.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:22:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:22:13 smithi122 ceph-mon[93869]: pgmap v3203: 129 pgs: 1 active+clean+snaptrim, 18 active+clean+snaptrim_wait, 110 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 73 KiB/s wr, 23 op/s 2023-03-09T16:22:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:22:13 smithi181 ceph-mon[94189]: pgmap v3203: 129 pgs: 1 active+clean+snaptrim, 18 active+clean+snaptrim_wait, 110 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 73 KiB/s wr, 23 op/s 2023-03-09T16:22:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:22:13 smithi038 ceph-mon[92422]: pgmap v3203: 129 pgs: 1 active+clean+snaptrim, 18 active+clean+snaptrim_wait, 110 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 73 KiB/s wr, 23 op/s 2023-03-09T16:22:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:22:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5046.005442 secs 2023-03-09T16:22:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:22:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5046.005442 secs 2023-03-09T16:22:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:22:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5046.005442 secs 2023-03-09T16:22:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:22:16 smithi122 ceph-mon[93869]: pgmap v3204: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 68 KiB/s wr, 8 op/s 2023-03-09T16:22:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:22:16 smithi181 ceph-mon[94189]: pgmap v3204: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 68 KiB/s wr, 8 op/s 2023-03-09T16:22:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:22:16 smithi038 ceph-mon[92422]: pgmap v3204: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 68 KiB/s wr, 8 op/s 2023-03-09T16:22:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:22:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5048.235347 secs 2023-03-09T16:22:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:22:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5048.235347 secs 2023-03-09T16:22:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:22:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5048.235347 secs 2023-03-09T16:22:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:22:18 smithi122 ceph-mon[93869]: pgmap v3205: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 62 KiB/s wr, 8 op/s 2023-03-09T16:22:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:22:18 smithi181 ceph-mon[94189]: pgmap v3205: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 62 KiB/s wr, 8 op/s 2023-03-09T16:22:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:22:18 smithi038 ceph-mon[92422]: pgmap v3205: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 62 KiB/s wr, 8 op/s 2023-03-09T16:22:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5510.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:22:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:22:20 smithi122 ceph-mon[93869]: pgmap v3206: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 62 KiB/s wr, 8 op/s 2023-03-09T16:22:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:22:20 smithi122 ceph-mon[93869]: osdmap e413: 12 total, 12 up, 12 in 2023-03-09T16:22:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:22:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5051.005504 secs 2023-03-09T16:22:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:22:20 smithi181 ceph-mon[94189]: pgmap v3206: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 62 KiB/s wr, 8 op/s 2023-03-09T16:22:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:22:20 smithi181 ceph-mon[94189]: osdmap e413: 12 total, 12 up, 12 in 2023-03-09T16:22:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:22:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5051.005504 secs 2023-03-09T16:22:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:22:20 smithi038 ceph-mon[92422]: pgmap v3206: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 62 KiB/s wr, 8 op/s 2023-03-09T16:22:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:22:20 smithi038 ceph-mon[92422]: osdmap e413: 12 total, 12 up, 12 in 2023-03-09T16:22:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:22:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5051.005504 secs 2023-03-09T16:22:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:22:22 smithi122 ceph-mon[93869]: pgmap v3208: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 37 KiB/s wr, 7 op/s 2023-03-09T16:22:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:22:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5053.235455 secs 2023-03-09T16:22:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:22:22 smithi181 ceph-mon[94189]: pgmap v3208: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 37 KiB/s wr, 7 op/s 2023-03-09T16:22:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:22:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5053.235455 secs 2023-03-09T16:22:22.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:22:22 smithi038 ceph-mon[92422]: pgmap v3208: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 37 KiB/s wr, 7 op/s 2023-03-09T16:22:22.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:22:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5053.235455 secs 2023-03-09T16:22:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5515.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:22:24.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:22:24 smithi122 ceph-mon[93869]: pgmap v3209: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 37 KiB/s wr, 7 op/s 2023-03-09T16:22:24.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:22:24 smithi181 ceph-mon[94189]: pgmap v3209: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 37 KiB/s wr, 7 op/s 2023-03-09T16:22:24.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:22:24 smithi038 ceph-mon[92422]: pgmap v3209: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 37 KiB/s wr, 7 op/s 2023-03-09T16:22:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:22:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5056.005567 secs 2023-03-09T16:22:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:22:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5056.005567 secs 2023-03-09T16:22:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:22:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5056.005567 secs 2023-03-09T16:22:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:22:26 smithi122 ceph-mon[93869]: pgmap v3210: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 614 B/s wr, 7 op/s 2023-03-09T16:22:26.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:22:26 smithi181 ceph-mon[94189]: pgmap v3210: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 614 B/s wr, 7 op/s 2023-03-09T16:22:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:22:26 smithi038 ceph-mon[92422]: pgmap v3210: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 614 B/s wr, 7 op/s 2023-03-09T16:22:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:22:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5058.235567 secs 2023-03-09T16:22:27.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:22:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5058.235567 secs 2023-03-09T16:22:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:22:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5058.235567 secs 2023-03-09T16:22:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:22:28 smithi122 ceph-mon[93869]: pgmap v3211: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 409 B/s wr, 7 op/s 2023-03-09T16:22:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:22:28 smithi181 ceph-mon[94189]: pgmap v3211: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 409 B/s wr, 7 op/s 2023-03-09T16:22:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:22:28 smithi038 ceph-mon[92422]: pgmap v3211: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 409 B/s wr, 7 op/s 2023-03-09T16:22:28.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5520.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:22:30.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:22:30 smithi122 ceph-mon[93869]: pgmap v3212: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2023-03-09T16:22:30.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:22:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5061.005645 secs 2023-03-09T16:22:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:22:30 smithi181 ceph-mon[94189]: pgmap v3212: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2023-03-09T16:22:30.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:22:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5061.005645 secs 2023-03-09T16:22:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:22:30 smithi038 ceph-mon[92422]: pgmap v3212: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2023-03-09T16:22:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:22:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5061.005645 secs 2023-03-09T16:22:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:22:32 smithi122 ceph-mon[93869]: pgmap v3213: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 36 KiB/s wr, 8 op/s 2023-03-09T16:22:32.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:22:32 smithi181 ceph-mon[94189]: pgmap v3213: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 36 KiB/s wr, 8 op/s 2023-03-09T16:22:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:22:32 smithi038 ceph-mon[92422]: pgmap v3213: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 36 KiB/s wr, 8 op/s 2023-03-09T16:22:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5525.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:22:34.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:22:34 smithi122 ceph-mon[93869]: pgmap v3214: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 34 KiB/s wr, 5 op/s 2023-03-09T16:22:34.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:22:34 smithi181 ceph-mon[94189]: pgmap v3214: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 34 KiB/s wr, 5 op/s 2023-03-09T16:22:34.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:22:34 smithi038 ceph-mon[92422]: pgmap v3214: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 34 KiB/s wr, 5 op/s 2023-03-09T16:22:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:22:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5066.005703 secs 2023-03-09T16:22:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:22:35 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:22:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:22:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5066.005703 secs 2023-03-09T16:22:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:22:35 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:22:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:22:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5066.005703 secs 2023-03-09T16:22:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:22:35 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:22:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:22:36 smithi122 ceph-mon[93869]: pgmap v3215: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 69 KiB/s wr, 6 op/s 2023-03-09T16:22:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:22:36 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:22:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:22:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:22:36.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:22:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:22:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:22:36 smithi181 ceph-mon[94189]: pgmap v3215: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 69 KiB/s wr, 6 op/s 2023-03-09T16:22:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:22:36 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:22:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:22:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:22:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:22:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:22:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:22:36 smithi038 ceph-mon[92422]: pgmap v3215: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 69 KiB/s wr, 6 op/s 2023-03-09T16:22:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:22:36 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:22:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:22:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:22:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:22:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:22:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:22:37 smithi122 ceph-mon[93869]: pgmap v3216: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 74 KiB/s wr, 8 op/s 2023-03-09T16:22:37.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:22:37 smithi181 ceph-mon[94189]: pgmap v3216: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 74 KiB/s wr, 8 op/s 2023-03-09T16:22:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:22:37 smithi038 ceph-mon[92422]: pgmap v3216: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 74 KiB/s wr, 8 op/s 2023-03-09T16:22:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5530.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:22:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:22:40 smithi122 ceph-mon[93869]: pgmap v3217: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 74 KiB/s wr, 6 op/s 2023-03-09T16:22:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:22:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5071.005762 secs 2023-03-09T16:22:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:22:40 smithi181 ceph-mon[94189]: pgmap v3217: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 74 KiB/s wr, 6 op/s 2023-03-09T16:22:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:22:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5071.005762 secs 2023-03-09T16:22:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:22:40 smithi038 ceph-mon[92422]: pgmap v3217: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 74 KiB/s wr, 6 op/s 2023-03-09T16:22:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:22:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5071.005762 secs 2023-03-09T16:22:42.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:22:42 smithi122 ceph-mon[93869]: pgmap v3218: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 KiB/s wr, 10 op/s 2023-03-09T16:22:42.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:22:42 smithi181 ceph-mon[94189]: pgmap v3218: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 KiB/s wr, 10 op/s 2023-03-09T16:22:42.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:22:42 smithi038 ceph-mon[92422]: pgmap v3218: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 KiB/s wr, 10 op/s 2023-03-09T16:22:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5535.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:22:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:22:44 smithi122 ceph-mon[93869]: pgmap v3219: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 58 KiB/s wr, 8 op/s 2023-03-09T16:22:44.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:22:44 smithi181 ceph-mon[94189]: pgmap v3219: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 58 KiB/s wr, 8 op/s 2023-03-09T16:22:44.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:22:44 smithi038 ceph-mon[92422]: pgmap v3219: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 58 KiB/s wr, 8 op/s 2023-03-09T16:22:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:22:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5076.005831 secs 2023-03-09T16:22:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:22:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5076.005831 secs 2023-03-09T16:22:45.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:22:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5076.005831 secs 2023-03-09T16:22:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:22:46 smithi122 ceph-mon[93869]: pgmap v3220: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 170 KiB/s wr, 9 op/s 2023-03-09T16:22:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:22:46 smithi181 ceph-mon[94189]: pgmap v3220: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 170 KiB/s wr, 9 op/s 2023-03-09T16:22:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:22:46 smithi038 ceph-mon[92422]: pgmap v3220: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 170 KiB/s wr, 9 op/s 2023-03-09T16:22:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:22:48 smithi122 ceph-mon[93869]: pgmap v3221: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 141 KiB/s wr, 11 op/s 2023-03-09T16:22:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:22:48 smithi181 ceph-mon[94189]: pgmap v3221: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 141 KiB/s wr, 11 op/s 2023-03-09T16:22:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:22:48 smithi038 ceph-mon[92422]: pgmap v3221: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 141 KiB/s wr, 11 op/s 2023-03-09T16:22:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5540.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:22:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:22:50 smithi122 ceph-mon[93869]: pgmap v3222: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 136 KiB/s wr, 9 op/s 2023-03-09T16:22:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:22:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5081.005893 secs 2023-03-09T16:22:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:22:50 smithi181 ceph-mon[94189]: pgmap v3222: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 136 KiB/s wr, 9 op/s 2023-03-09T16:22:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:22:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5081.005893 secs 2023-03-09T16:22:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:22:50 smithi038 ceph-mon[92422]: pgmap v3222: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 136 KiB/s wr, 9 op/s 2023-03-09T16:22:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:22:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5081.005893 secs 2023-03-09T16:22:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:22:52 smithi122 ceph-mon[93869]: pgmap v3223: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 150 KiB/s wr, 12 op/s 2023-03-09T16:22:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:22:52 smithi181 ceph-mon[94189]: pgmap v3223: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 150 KiB/s wr, 12 op/s 2023-03-09T16:22:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:22:52 smithi038 ceph-mon[92422]: pgmap v3223: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 150 KiB/s wr, 12 op/s 2023-03-09T16:22:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5545.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:22:54.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:22:54 smithi122 ceph-mon[93869]: pgmap v3224: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 132 KiB/s wr, 9 op/s 2023-03-09T16:22:54.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:22:54 smithi181 ceph-mon[94189]: pgmap v3224: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 132 KiB/s wr, 9 op/s 2023-03-09T16:22:54.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:22:54 smithi038 ceph-mon[92422]: pgmap v3224: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 132 KiB/s wr, 9 op/s 2023-03-09T16:22:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:22:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5086.005953 secs 2023-03-09T16:22:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:22:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5086.005953 secs 2023-03-09T16:22:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:22:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5086.005953 secs 2023-03-09T16:22:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:22:56 smithi122 ceph-mon[93869]: pgmap v3225: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 180 KiB/s wr, 9 op/s 2023-03-09T16:22:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:22:56 smithi181 ceph-mon[94189]: pgmap v3225: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 180 KiB/s wr, 9 op/s 2023-03-09T16:22:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:22:56 smithi038 ceph-mon[92422]: pgmap v3225: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 180 KiB/s wr, 9 op/s 2023-03-09T16:22:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:22:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5088.236284 secs 2023-03-09T16:22:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:22:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5088.236284 secs 2023-03-09T16:22:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:22:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5088.236284 secs 2023-03-09T16:22:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:22:58 smithi122 ceph-mon[93869]: pgmap v3226: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 148 KiB/s wr, 11 op/s 2023-03-09T16:22:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:22:58 smithi181 ceph-mon[94189]: pgmap v3226: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 148 KiB/s wr, 11 op/s 2023-03-09T16:22:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:22:58 smithi038 ceph-mon[92422]: pgmap v3226: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 148 KiB/s wr, 11 op/s 2023-03-09T16:22:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5550.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:23:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:23:00 smithi122 ceph-mon[93869]: pgmap v3227: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 143 KiB/s wr, 9 op/s 2023-03-09T16:23:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:23:00 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:23:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:23:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:23:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:23:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:23:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:23:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5091.006040 secs 2023-03-09T16:23:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:23:00 smithi181 ceph-mon[94189]: pgmap v3227: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 143 KiB/s wr, 9 op/s 2023-03-09T16:23:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:23:00 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:23:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:23:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:23:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:23:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:23:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:23:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5091.006040 secs 2023-03-09T16:23:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:23:00 smithi038 ceph-mon[92422]: pgmap v3227: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 143 KiB/s wr, 9 op/s 2023-03-09T16:23:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:23:00 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:23:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:23:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:23:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:23:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:23:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:23:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5091.006040 secs 2023-03-09T16:23:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:23:02 smithi122 ceph-mon[93869]: pgmap v3228: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 188 KiB/s wr, 15 op/s 2023-03-09T16:23:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:23:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5093.236420 secs 2023-03-09T16:23:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:23:02 smithi181 ceph-mon[94189]: pgmap v3228: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 188 KiB/s wr, 15 op/s 2023-03-09T16:23:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:23:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5093.236420 secs 2023-03-09T16:23:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:23:02 smithi038 ceph-mon[92422]: pgmap v3228: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 188 KiB/s wr, 15 op/s 2023-03-09T16:23:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:23:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5093.236420 secs 2023-03-09T16:23:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5555.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:23:04.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:23:04 smithi122 ceph-mon[93869]: pgmap v3229: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 196 KiB/s wr, 24 op/s 2023-03-09T16:23:04.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:23:04 smithi181 ceph-mon[94189]: pgmap v3229: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 196 KiB/s wr, 24 op/s 2023-03-09T16:23:04.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:23:04 smithi038 ceph-mon[92422]: pgmap v3229: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 196 KiB/s wr, 24 op/s 2023-03-09T16:23:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:23:05 smithi122 ceph-mon[93869]: osdmap e414: 12 total, 12 up, 12 in 2023-03-09T16:23:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:23:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5096.006101 secs 2023-03-09T16:23:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:23:05 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:23:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:23:05 smithi181 ceph-mon[94189]: osdmap e414: 12 total, 12 up, 12 in 2023-03-09T16:23:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:23:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5096.006101 secs 2023-03-09T16:23:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:23:05 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:23:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:23:05 smithi038 ceph-mon[92422]: osdmap e414: 12 total, 12 up, 12 in 2023-03-09T16:23:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:23:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5096.006101 secs 2023-03-09T16:23:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:23:05 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:23:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:23:06 smithi122 ceph-mon[93869]: pgmap v3231: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 177 KiB/s wr, 27 op/s 2023-03-09T16:23:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:23:06 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:23:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:23:06 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:23:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:23:06 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:23:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:23:06 smithi181 ceph-mon[94189]: pgmap v3231: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 177 KiB/s wr, 27 op/s 2023-03-09T16:23:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:23:06 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:23:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:23:06 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:23:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:23:06 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:23:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:23:06 smithi038 ceph-mon[92422]: pgmap v3231: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 177 KiB/s wr, 27 op/s 2023-03-09T16:23:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:23:06 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:23:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:23:06 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:23:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:23:06 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:23:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:23:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5098.236494 secs 2023-03-09T16:23:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:23:07 smithi122 ceph-mon[93869]: pgmap v3232: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 164 KiB/s wr, 27 op/s 2023-03-09T16:23:07.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:23:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5098.236494 secs 2023-03-09T16:23:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:23:07 smithi181 ceph-mon[94189]: pgmap v3232: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 164 KiB/s wr, 27 op/s 2023-03-09T16:23:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:23:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5098.236494 secs 2023-03-09T16:23:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:23:07 smithi038 ceph-mon[92422]: pgmap v3232: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 164 KiB/s wr, 27 op/s 2023-03-09T16:23:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5560.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:23:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:23:10 smithi122 ceph-mon[93869]: pgmap v3233: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 164 KiB/s wr, 27 op/s 2023-03-09T16:23:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:23:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5101.006161 secs 2023-03-09T16:23:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:23:10 smithi181 ceph-mon[94189]: pgmap v3233: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 164 KiB/s wr, 27 op/s 2023-03-09T16:23:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:23:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5101.006161 secs 2023-03-09T16:23:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:23:10 smithi038 ceph-mon[92422]: pgmap v3233: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 164 KiB/s wr, 27 op/s 2023-03-09T16:23:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:23:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5101.006161 secs 2023-03-09T16:23:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:23:12 smithi122 ceph-mon[93869]: pgmap v3234: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 137 KiB/s wr, 23 op/s 2023-03-09T16:23:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:23:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5103.236591 secs 2023-03-09T16:23:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:23:12 smithi181 ceph-mon[94189]: pgmap v3234: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 137 KiB/s wr, 23 op/s 2023-03-09T16:23:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:23:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5103.236591 secs 2023-03-09T16:23:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:23:12 smithi038 ceph-mon[92422]: pgmap v3234: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 137 KiB/s wr, 23 op/s 2023-03-09T16:23:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:23:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5103.236591 secs 2023-03-09T16:23:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5565.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:23:14.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:23:14 smithi122 ceph-mon[93869]: pgmap v3235: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 111 KiB/s wr, 8 op/s 2023-03-09T16:23:14.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:23:14 smithi181 ceph-mon[94189]: pgmap v3235: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 111 KiB/s wr, 8 op/s 2023-03-09T16:23:14.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:23:14 smithi038 ceph-mon[92422]: pgmap v3235: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 111 KiB/s wr, 8 op/s 2023-03-09T16:23:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:23:15 smithi122 ceph-mon[93869]: osdmap e415: 12 total, 12 up, 12 in 2023-03-09T16:23:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:23:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5106.006223 secs 2023-03-09T16:23:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:23:15 smithi122 ceph-mon[93869]: pgmap v3237: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 111 KiB/s wr, 8 op/s 2023-03-09T16:23:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:23:15 smithi181 ceph-mon[94189]: osdmap e415: 12 total, 12 up, 12 in 2023-03-09T16:23:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:23:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5106.006223 secs 2023-03-09T16:23:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:23:15 smithi181 ceph-mon[94189]: pgmap v3237: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 111 KiB/s wr, 8 op/s 2023-03-09T16:23:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:23:15 smithi038 ceph-mon[92422]: osdmap e415: 12 total, 12 up, 12 in 2023-03-09T16:23:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:23:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5106.006223 secs 2023-03-09T16:23:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:23:15 smithi038 ceph-mon[92422]: pgmap v3237: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 111 KiB/s wr, 8 op/s 2023-03-09T16:23:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:23:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5108.236715 secs 2023-03-09T16:23:17.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:23:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5108.236715 secs 2023-03-09T16:23:17.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:23:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5108.236715 secs 2023-03-09T16:23:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:23:17 smithi122 ceph-mon[93869]: pgmap v3238: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T16:23:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:23:17 smithi181 ceph-mon[94189]: pgmap v3238: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T16:23:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:23:17 smithi038 ceph-mon[92422]: pgmap v3238: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T16:23:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5570.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:23:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:23:20 smithi122 ceph-mon[93869]: pgmap v3239: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T16:23:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:23:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5111.006285 secs 2023-03-09T16:23:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:23:20 smithi181 ceph-mon[94189]: pgmap v3239: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T16:23:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:23:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5111.006285 secs 2023-03-09T16:23:20.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:23:20 smithi038 ceph-mon[92422]: pgmap v3239: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T16:23:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:23:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5111.006285 secs 2023-03-09T16:23:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:23:22 smithi122 ceph-mon[93869]: pgmap v3240: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 38 KiB/s wr, 7 op/s 2023-03-09T16:23:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:23:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5113.236788 secs 2023-03-09T16:23:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:23:22 smithi181 ceph-mon[94189]: pgmap v3240: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 38 KiB/s wr, 7 op/s 2023-03-09T16:23:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:23:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5113.236788 secs 2023-03-09T16:23:22.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:23:22 smithi038 ceph-mon[92422]: pgmap v3240: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 38 KiB/s wr, 7 op/s 2023-03-09T16:23:22.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:23:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5113.236788 secs 2023-03-09T16:23:23.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5575.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:23:24.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:23:24 smithi122 ceph-mon[93869]: pgmap v3241: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 38 KiB/s wr, 7 op/s 2023-03-09T16:23:24.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:23:24 smithi181 ceph-mon[94189]: pgmap v3241: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 38 KiB/s wr, 7 op/s 2023-03-09T16:23:24.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:23:24 smithi038 ceph-mon[92422]: pgmap v3241: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 38 KiB/s wr, 7 op/s 2023-03-09T16:23:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:23:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5116.006345 secs 2023-03-09T16:23:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:23:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5116.006345 secs 2023-03-09T16:23:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:23:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5116.006345 secs 2023-03-09T16:23:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:23:26 smithi122 ceph-mon[93869]: pgmap v3242: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 KiB/s wr, 7 op/s 2023-03-09T16:23:26.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:23:26 smithi181 ceph-mon[94189]: pgmap v3242: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 KiB/s wr, 7 op/s 2023-03-09T16:23:26.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:23:26 smithi038 ceph-mon[92422]: pgmap v3242: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 KiB/s wr, 7 op/s 2023-03-09T16:23:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:23:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5118.236920 secs 2023-03-09T16:23:27.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:23:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5118.236920 secs 2023-03-09T16:23:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:23:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5118.236920 secs 2023-03-09T16:23:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:23:28 smithi122 ceph-mon[93869]: pgmap v3243: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 32 KiB/s wr, 9 op/s 2023-03-09T16:23:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:23:28 smithi181 ceph-mon[94189]: pgmap v3243: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 32 KiB/s wr, 9 op/s 2023-03-09T16:23:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:23:28 smithi038 ceph-mon[92422]: pgmap v3243: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 32 KiB/s wr, 9 op/s 2023-03-09T16:23:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5580.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:23:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:23:30 smithi122 ceph-mon[93869]: pgmap v3244: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 32 KiB/s wr, 6 op/s 2023-03-09T16:23:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:23:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5121.006406 secs 2023-03-09T16:23:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:23:30 smithi181 ceph-mon[94189]: pgmap v3244: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 32 KiB/s wr, 6 op/s 2023-03-09T16:23:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:23:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5121.006406 secs 2023-03-09T16:23:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:23:30 smithi038 ceph-mon[92422]: pgmap v3244: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 32 KiB/s wr, 6 op/s 2023-03-09T16:23:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:23:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5121.006406 secs 2023-03-09T16:23:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:23:32 smithi122 ceph-mon[93869]: pgmap v3245: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 134 KiB/s wr, 10 op/s 2023-03-09T16:23:32.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:23:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5123.237056 secs 2023-03-09T16:23:32.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:23:32 smithi181 ceph-mon[94189]: pgmap v3245: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 134 KiB/s wr, 10 op/s 2023-03-09T16:23:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:23:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5123.237056 secs 2023-03-09T16:23:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:23:32 smithi038 ceph-mon[92422]: pgmap v3245: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 134 KiB/s wr, 10 op/s 2023-03-09T16:23:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:23:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5123.237056 secs 2023-03-09T16:23:33.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5585.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:23:34.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:23:34 smithi122 ceph-mon[93869]: pgmap v3246: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 103 KiB/s wr, 7 op/s 2023-03-09T16:23:34.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:23:34 smithi181 ceph-mon[94189]: pgmap v3246: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 103 KiB/s wr, 7 op/s 2023-03-09T16:23:34.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:23:34 smithi038 ceph-mon[92422]: pgmap v3246: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 103 KiB/s wr, 7 op/s 2023-03-09T16:23:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:23:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5126.006490 secs 2023-03-09T16:23:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:23:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5126.006490 secs 2023-03-09T16:23:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:23:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5126.006490 secs 2023-03-09T16:23:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:23:36 smithi122 ceph-mon[93869]: pgmap v3247: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 103 KiB/s wr, 7 op/s 2023-03-09T16:23:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:23:36 smithi181 ceph-mon[94189]: pgmap v3247: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 103 KiB/s wr, 7 op/s 2023-03-09T16:23:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:23:36 smithi038 ceph-mon[92422]: pgmap v3247: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 103 KiB/s wr, 7 op/s 2023-03-09T16:23:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:23:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5128.237166 secs 2023-03-09T16:23:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:23:37 smithi181 ceph-mon[94189]: pgmap v3248: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 182 KiB/s wr, 38 op/s 2023-03-09T16:23:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:23:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5128.237166 secs 2023-03-09T16:23:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:23:37 smithi038 ceph-mon[92422]: pgmap v3248: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 182 KiB/s wr, 38 op/s 2023-03-09T16:23:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:23:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5128.237166 secs 2023-03-09T16:23:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:23:37 smithi122 ceph-mon[93869]: pgmap v3248: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 182 KiB/s wr, 38 op/s 2023-03-09T16:23:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5590.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:23:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:23:40 smithi122 ceph-mon[93869]: pgmap v3249: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 182 KiB/s wr, 35 op/s 2023-03-09T16:23:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:23:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5131.006550 secs 2023-03-09T16:23:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:23:40 smithi181 ceph-mon[94189]: pgmap v3249: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 182 KiB/s wr, 35 op/s 2023-03-09T16:23:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:23:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5131.006550 secs 2023-03-09T16:23:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:23:40 smithi038 ceph-mon[92422]: pgmap v3249: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 182 KiB/s wr, 35 op/s 2023-03-09T16:23:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:23:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5131.006550 secs 2023-03-09T16:23:42.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:23:42 smithi122 ceph-mon[93869]: pgmap v3250: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 235 KiB/s wr, 40 op/s 2023-03-09T16:23:42.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:23:42 smithi181 ceph-mon[94189]: pgmap v3250: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 235 KiB/s wr, 40 op/s 2023-03-09T16:23:42.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:23:42 smithi038 ceph-mon[92422]: pgmap v3250: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 235 KiB/s wr, 40 op/s 2023-03-09T16:23:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:23:43 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:23:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:23:43 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:23:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:23:43 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:23:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5595.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:23:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:23:44 smithi122 ceph-mon[93869]: pgmap v3251: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 133 KiB/s wr, 37 op/s 2023-03-09T16:23:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:23:44 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:23:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:23:44 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:23:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:23:44 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:23:44.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:23:44 smithi181 ceph-mon[94189]: pgmap v3251: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 133 KiB/s wr, 37 op/s 2023-03-09T16:23:44.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:23:44 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:23:44.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:23:44 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:23:44.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:23:44 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:23:44.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:23:44 smithi038 ceph-mon[92422]: pgmap v3251: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 133 KiB/s wr, 37 op/s 2023-03-09T16:23:44.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:23:44 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:23:44.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:23:44 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:23:44.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:23:44 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:23:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:23:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5136.006611 secs 2023-03-09T16:23:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:23:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5136.006611 secs 2023-03-09T16:23:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:23:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5136.006611 secs 2023-03-09T16:23:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:23:46 smithi122 ceph-mon[93869]: pgmap v3252: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 133 KiB/s wr, 36 op/s 2023-03-09T16:23:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:23:46 smithi181 ceph-mon[94189]: pgmap v3252: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 133 KiB/s wr, 36 op/s 2023-03-09T16:23:46.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:23:46 smithi038 ceph-mon[92422]: pgmap v3252: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 133 KiB/s wr, 36 op/s 2023-03-09T16:23:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:23:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5138.237369 secs 2023-03-09T16:23:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:23:47 smithi038 ceph-mon[92422]: pgmap v3253: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 254 KiB/s wr, 41 op/s 2023-03-09T16:23:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:23:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5138.237369 secs 2023-03-09T16:23:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:23:47 smithi122 ceph-mon[93869]: pgmap v3253: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 254 KiB/s wr, 41 op/s 2023-03-09T16:23:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:23:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5138.237369 secs 2023-03-09T16:23:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:23:47 smithi181 ceph-mon[94189]: pgmap v3253: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 254 KiB/s wr, 41 op/s 2023-03-09T16:23:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:23:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:23:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:23:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:23:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5600.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:23:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:23:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:23:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:23:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:23:48.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:23:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:23:48.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:23:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:23:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:23:49 smithi038 ceph-mon[92422]: pgmap v3254: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 175 KiB/s wr, 10 op/s 2023-03-09T16:23:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:23:49 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:23:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:23:49 smithi122 ceph-mon[93869]: pgmap v3254: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 175 KiB/s wr, 10 op/s 2023-03-09T16:23:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:23:49 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:23:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:23:49 smithi181 ceph-mon[94189]: pgmap v3254: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 175 KiB/s wr, 10 op/s 2023-03-09T16:23:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:23:49 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:23:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:23:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5141.006672 secs 2023-03-09T16:23:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:23:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5141.006672 secs 2023-03-09T16:23:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:23:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5141.006672 secs 2023-03-09T16:23:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:23:51 smithi122 ceph-mon[93869]: pgmap v3255: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 175 KiB/s wr, 13 op/s 2023-03-09T16:23:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:23:51 smithi181 ceph-mon[94189]: pgmap v3255: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 175 KiB/s wr, 13 op/s 2023-03-09T16:23:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:23:51 smithi038 ceph-mon[92422]: pgmap v3255: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 175 KiB/s wr, 13 op/s 2023-03-09T16:23:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:23:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5143.237509 secs 2023-03-09T16:23:52.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:23:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5143.237509 secs 2023-03-09T16:23:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:23:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5143.237509 secs 2023-03-09T16:23:53.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5605.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:23:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:23:53 smithi122 ceph-mon[93869]: pgmap v3256: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 122 KiB/s wr, 9 op/s 2023-03-09T16:23:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:23:53 smithi181 ceph-mon[94189]: pgmap v3256: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 122 KiB/s wr, 9 op/s 2023-03-09T16:23:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:23:53 smithi038 ceph-mon[92422]: pgmap v3256: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 122 KiB/s wr, 9 op/s 2023-03-09T16:23:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:23:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5146.006732 secs 2023-03-09T16:23:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:23:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5146.006732 secs 2023-03-09T16:23:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:23:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5146.006732 secs 2023-03-09T16:23:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:23:56 smithi122 ceph-mon[93869]: pgmap v3257: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 122 KiB/s wr, 9 op/s 2023-03-09T16:23:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:23:56 smithi181 ceph-mon[94189]: pgmap v3257: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 122 KiB/s wr, 9 op/s 2023-03-09T16:23:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:23:56 smithi038 ceph-mon[92422]: pgmap v3257: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 122 KiB/s wr, 9 op/s 2023-03-09T16:23:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:23:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5148.237633 secs 2023-03-09T16:23:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:23:57 smithi038 ceph-mon[92422]: pgmap v3258: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 209 KiB/s wr, 13 op/s 2023-03-09T16:23:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:23:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5148.237633 secs 2023-03-09T16:23:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:23:57 smithi122 ceph-mon[93869]: pgmap v3258: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 209 KiB/s wr, 13 op/s 2023-03-09T16:23:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:23:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5148.237633 secs 2023-03-09T16:23:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:23:57 smithi181 ceph-mon[94189]: pgmap v3258: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 209 KiB/s wr, 13 op/s 2023-03-09T16:23:58.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5610.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:24:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:24:00 smithi122 ceph-mon[93869]: pgmap v3259: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 88 KiB/s wr, 8 op/s 2023-03-09T16:24:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:24:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5151.006790 secs 2023-03-09T16:24:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:24:00 smithi181 ceph-mon[94189]: pgmap v3259: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 88 KiB/s wr, 8 op/s 2023-03-09T16:24:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:24:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5151.006790 secs 2023-03-09T16:24:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:24:00 smithi038 ceph-mon[92422]: pgmap v3259: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 88 KiB/s wr, 8 op/s 2023-03-09T16:24:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:24:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5151.006790 secs 2023-03-09T16:24:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:24:02 smithi122 ceph-mon[93869]: pgmap v3260: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 140 KiB/s wr, 13 op/s 2023-03-09T16:24:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:24:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5153.237751 secs 2023-03-09T16:24:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:24:02 smithi181 ceph-mon[94189]: pgmap v3260: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 140 KiB/s wr, 13 op/s 2023-03-09T16:24:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:24:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5153.237751 secs 2023-03-09T16:24:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:24:02 smithi038 ceph-mon[92422]: pgmap v3260: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 140 KiB/s wr, 13 op/s 2023-03-09T16:24:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:24:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5153.237751 secs 2023-03-09T16:24:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5615.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:24:04.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:24:04 smithi122 ceph-mon[93869]: pgmap v3261: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 161 KiB/s wr, 22 op/s 2023-03-09T16:24:04.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:24:04 smithi181 ceph-mon[94189]: pgmap v3261: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 161 KiB/s wr, 22 op/s 2023-03-09T16:24:04.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:24:04 smithi038 ceph-mon[92422]: pgmap v3261: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 161 KiB/s wr, 22 op/s 2023-03-09T16:24:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:24:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5156.006851 secs 2023-03-09T16:24:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:24:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5156.006851 secs 2023-03-09T16:24:05.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:24:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5156.006851 secs 2023-03-09T16:24:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:24:06 smithi181 ceph-mon[94189]: pgmap v3262: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 161 KiB/s wr, 21 op/s 2023-03-09T16:24:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:24:06 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:24:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:24:06 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:24:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:24:06 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:24:06.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:24:06 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:24:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:24:06 smithi038 ceph-mon[92422]: pgmap v3262: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 161 KiB/s wr, 21 op/s 2023-03-09T16:24:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:24:06 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:24:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:24:06 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:24:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:24:06 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:24:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:24:06 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:24:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:24:06 smithi122 ceph-mon[93869]: pgmap v3262: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 161 KiB/s wr, 21 op/s 2023-03-09T16:24:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:24:06 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:24:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:24:06 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:24:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:24:06 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:24:06.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:24:06 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:24:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:24:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5158.237887 secs 2023-03-09T16:24:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:24:07 smithi038 ceph-mon[92422]: pgmap v3263: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 192 KiB/s wr, 25 op/s 2023-03-09T16:24:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:24:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5158.237887 secs 2023-03-09T16:24:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:24:07 smithi122 ceph-mon[93869]: pgmap v3263: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 192 KiB/s wr, 25 op/s 2023-03-09T16:24:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:24:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5158.237887 secs 2023-03-09T16:24:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:24:07 smithi181 ceph-mon[94189]: pgmap v3263: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 192 KiB/s wr, 25 op/s 2023-03-09T16:24:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5620.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:24:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:24:09 smithi122 ceph-mon[93869]: osdmap e416: 12 total, 12 up, 12 in 2023-03-09T16:24:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:24:09 smithi122 ceph-mon[93869]: pgmap v3265: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 125 KiB/s wr, 25 op/s 2023-03-09T16:24:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:24:09 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5161.006910 secs 2023-03-09T16:24:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:24:09 smithi181 ceph-mon[94189]: osdmap e416: 12 total, 12 up, 12 in 2023-03-09T16:24:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:24:09 smithi181 ceph-mon[94189]: pgmap v3265: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 125 KiB/s wr, 25 op/s 2023-03-09T16:24:10.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:24:09 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5161.006910 secs 2023-03-09T16:24:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:24:09 smithi038 ceph-mon[92422]: osdmap e416: 12 total, 12 up, 12 in 2023-03-09T16:24:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:24:09 smithi038 ceph-mon[92422]: pgmap v3265: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 125 KiB/s wr, 25 op/s 2023-03-09T16:24:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:24:09 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5161.006910 secs 2023-03-09T16:24:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:24:11 smithi122 ceph-mon[93869]: pgmap v3266: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 63 KiB/s wr, 22 op/s 2023-03-09T16:24:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:24:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5163.238009 secs 2023-03-09T16:24:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:24:11 smithi181 ceph-mon[94189]: pgmap v3266: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 63 KiB/s wr, 22 op/s 2023-03-09T16:24:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:24:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5163.238009 secs 2023-03-09T16:24:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:24:11 smithi038 ceph-mon[92422]: pgmap v3266: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 63 KiB/s wr, 22 op/s 2023-03-09T16:24:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:24:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5163.238009 secs 2023-03-09T16:24:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5625.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:24:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:24:13 smithi122 ceph-mon[93869]: pgmap v3267: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 38 KiB/s wr, 8 op/s 2023-03-09T16:24:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:24:13 smithi181 ceph-mon[94189]: pgmap v3267: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 38 KiB/s wr, 8 op/s 2023-03-09T16:24:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:24:13 smithi038 ceph-mon[92422]: pgmap v3267: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 38 KiB/s wr, 8 op/s 2023-03-09T16:24:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:24:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5166.006969 secs 2023-03-09T16:24:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:24:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5166.006969 secs 2023-03-09T16:24:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:24:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5166.006969 secs 2023-03-09T16:24:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:24:16 smithi122 ceph-mon[93869]: pgmap v3268: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 38 KiB/s wr, 8 op/s 2023-03-09T16:24:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:24:16 smithi181 ceph-mon[94189]: pgmap v3268: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 38 KiB/s wr, 8 op/s 2023-03-09T16:24:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:24:16 smithi038 ceph-mon[92422]: pgmap v3268: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 38 KiB/s wr, 8 op/s 2023-03-09T16:24:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:24:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5168.238123 secs 2023-03-09T16:24:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:24:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5168.238123 secs 2023-03-09T16:24:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:24:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5168.238123 secs 2023-03-09T16:24:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:24:18 smithi181 ceph-mon[94189]: pgmap v3269: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 38 KiB/s wr, 8 op/s 2023-03-09T16:24:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:24:18 smithi038 ceph-mon[92422]: pgmap v3269: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 38 KiB/s wr, 8 op/s 2023-03-09T16:24:18.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5630.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:24:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:24:18 smithi122 ceph-mon[93869]: pgmap v3269: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 38 KiB/s wr, 8 op/s 2023-03-09T16:24:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:24:20 smithi122 ceph-mon[93869]: pgmap v3270: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 37 KiB/s wr, 8 op/s 2023-03-09T16:24:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:24:20 smithi122 ceph-mon[93869]: osdmap e417: 12 total, 12 up, 12 in 2023-03-09T16:24:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:24:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5171.006983 secs 2023-03-09T16:24:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:24:20 smithi181 ceph-mon[94189]: pgmap v3270: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 37 KiB/s wr, 8 op/s 2023-03-09T16:24:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:24:20 smithi181 ceph-mon[94189]: osdmap e417: 12 total, 12 up, 12 in 2023-03-09T16:24:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:24:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5171.006983 secs 2023-03-09T16:24:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:24:20 smithi038 ceph-mon[92422]: pgmap v3270: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 37 KiB/s wr, 8 op/s 2023-03-09T16:24:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:24:20 smithi038 ceph-mon[92422]: osdmap e417: 12 total, 12 up, 12 in 2023-03-09T16:24:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:24:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5171.006983 secs 2023-03-09T16:24:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:24:22 smithi122 ceph-mon[93869]: pgmap v3272: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 38 KiB/s wr, 8 op/s 2023-03-09T16:24:22.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:24:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5173.238243 secs 2023-03-09T16:24:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:24:22 smithi181 ceph-mon[94189]: pgmap v3272: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 38 KiB/s wr, 8 op/s 2023-03-09T16:24:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:24:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5173.238243 secs 2023-03-09T16:24:22.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:24:22 smithi038 ceph-mon[92422]: pgmap v3272: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 38 KiB/s wr, 8 op/s 2023-03-09T16:24:22.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:24:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5173.238243 secs 2023-03-09T16:24:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:24:23 smithi038 ceph-mon[92422]: pgmap v3273: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 38 KiB/s wr, 8 op/s 2023-03-09T16:24:23.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5635.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:24:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:24:23 smithi122 ceph-mon[93869]: pgmap v3273: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 38 KiB/s wr, 8 op/s 2023-03-09T16:24:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:24:23 smithi181 ceph-mon[94189]: pgmap v3273: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 38 KiB/s wr, 8 op/s 2023-03-09T16:24:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:24:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5176.007041 secs 2023-03-09T16:24:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:24:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5176.007041 secs 2023-03-09T16:24:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:24:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5176.007041 secs 2023-03-09T16:24:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:24:26 smithi038 ceph-mon[92422]: pgmap v3274: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 38 KiB/s wr, 8 op/s 2023-03-09T16:24:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:24:26 smithi122 ceph-mon[93869]: pgmap v3274: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 38 KiB/s wr, 8 op/s 2023-03-09T16:24:26.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:24:26 smithi181 ceph-mon[94189]: pgmap v3274: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 38 KiB/s wr, 8 op/s 2023-03-09T16:24:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:24:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5178.238324 secs 2023-03-09T16:24:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:24:27 smithi038 ceph-mon[92422]: pgmap v3275: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 37 KiB/s wr, 8 op/s 2023-03-09T16:24:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:24:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5178.238324 secs 2023-03-09T16:24:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:24:27 smithi122 ceph-mon[93869]: pgmap v3275: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 37 KiB/s wr, 8 op/s 2023-03-09T16:24:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:24:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5178.238324 secs 2023-03-09T16:24:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:24:27 smithi181 ceph-mon[94189]: pgmap v3275: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 37 KiB/s wr, 8 op/s 2023-03-09T16:24:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5640.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:24:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:24:30 smithi122 ceph-mon[93869]: pgmap v3276: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 37 KiB/s wr, 7 op/s 2023-03-09T16:24:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:24:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5181.007100 secs 2023-03-09T16:24:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:24:30 smithi181 ceph-mon[94189]: pgmap v3276: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 37 KiB/s wr, 7 op/s 2023-03-09T16:24:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:24:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5181.007100 secs 2023-03-09T16:24:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:24:30 smithi038 ceph-mon[92422]: pgmap v3276: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 37 KiB/s wr, 7 op/s 2023-03-09T16:24:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:24:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5181.007100 secs 2023-03-09T16:24:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:24:32 smithi122 ceph-mon[93869]: pgmap v3277: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 74 KiB/s wr, 10 op/s 2023-03-09T16:24:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:24:32 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 5183.238431 secs 2023-03-09T16:24:32.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:24:32 smithi181 ceph-mon[94189]: pgmap v3277: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 74 KiB/s wr, 10 op/s 2023-03-09T16:24:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:24:32 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 5183.238431 secs 2023-03-09T16:24:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:24:32 smithi038 ceph-mon[92422]: pgmap v3277: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 74 KiB/s wr, 10 op/s 2023-03-09T16:24:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:24:32 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 5183.238431 secs 2023-03-09T16:24:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5645.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:24:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:24:33 smithi122 ceph-mon[93869]: pgmap v3278: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 70 KiB/s wr, 7 op/s 2023-03-09T16:24:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:24:33 smithi181 ceph-mon[94189]: pgmap v3278: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 70 KiB/s wr, 7 op/s 2023-03-09T16:24:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:24:33 smithi038 ceph-mon[92422]: pgmap v3278: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 70 KiB/s wr, 7 op/s 2023-03-09T16:24:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:24:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5186.007160 secs 2023-03-09T16:24:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:24:35 smithi038 ceph-mon[92422]: pgmap v3279: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 70 KiB/s wr, 6 op/s 2023-03-09T16:24:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:24:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5186.007160 secs 2023-03-09T16:24:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:24:35 smithi122 ceph-mon[93869]: pgmap v3279: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 70 KiB/s wr, 6 op/s 2023-03-09T16:24:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:24:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5186.007160 secs 2023-03-09T16:24:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:24:35 smithi181 ceph-mon[94189]: pgmap v3279: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 70 KiB/s wr, 6 op/s 2023-03-09T16:24:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:24:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:24:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:24:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:24:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:24:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:24:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:24:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:24:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:24:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:24:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:24:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:24:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:24:37 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 5188.238578 secs 2023-03-09T16:24:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:24:37 smithi038 ceph-mon[92422]: pgmap v3280: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 110 KiB/s wr, 10 op/s 2023-03-09T16:24:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:24:37 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 5188.238578 secs 2023-03-09T16:24:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:24:37 smithi122 ceph-mon[93869]: pgmap v3280: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 110 KiB/s wr, 10 op/s 2023-03-09T16:24:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:24:37 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 5188.238578 secs 2023-03-09T16:24:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:24:37 smithi181 ceph-mon[94189]: pgmap v3280: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 110 KiB/s wr, 10 op/s 2023-03-09T16:24:38.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5650.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:24:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:24:40 smithi122 ceph-mon[93869]: pgmap v3281: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 79 KiB/s wr, 6 op/s 2023-03-09T16:24:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:24:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5191.007221 secs 2023-03-09T16:24:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:24:40 smithi181 ceph-mon[94189]: pgmap v3281: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 79 KiB/s wr, 6 op/s 2023-03-09T16:24:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:24:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5191.007221 secs 2023-03-09T16:24:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:24:40 smithi038 ceph-mon[92422]: pgmap v3281: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 79 KiB/s wr, 6 op/s 2023-03-09T16:24:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:24:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5191.007221 secs 2023-03-09T16:24:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:24:41 smithi122 ceph-mon[93869]: pgmap v3282: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 124 KiB/s wr, 11 op/s 2023-03-09T16:24:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:24:41 smithi181 ceph-mon[94189]: pgmap v3282: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 124 KiB/s wr, 11 op/s 2023-03-09T16:24:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:24:41 smithi038 ceph-mon[92422]: pgmap v3282: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 124 KiB/s wr, 11 op/s 2023-03-09T16:24:42.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:24:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5193.238698 secs 2023-03-09T16:24:42.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:24:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5193.238698 secs 2023-03-09T16:24:42.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:24:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5193.238698 secs 2023-03-09T16:24:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5655.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:24:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:24:43 smithi122 ceph-mon[93869]: pgmap v3283: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 85 KiB/s wr, 8 op/s 2023-03-09T16:24:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:24:43 smithi181 ceph-mon[94189]: pgmap v3283: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 85 KiB/s wr, 8 op/s 2023-03-09T16:24:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:24:43 smithi038 ceph-mon[92422]: pgmap v3283: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 85 KiB/s wr, 8 op/s 2023-03-09T16:24:44.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:24:44 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:24:44.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:24:44 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:24:44.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:24:44 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:24:44.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:24:44 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:24:44.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:24:44 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:24:44.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:24:44 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:24:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:24:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5196.007283 secs 2023-03-09T16:24:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:24:45 smithi122 ceph-mon[93869]: pgmap v3284: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 85 KiB/s wr, 8 op/s 2023-03-09T16:24:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:24:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5196.007283 secs 2023-03-09T16:24:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:24:45 smithi181 ceph-mon[94189]: pgmap v3284: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 85 KiB/s wr, 8 op/s 2023-03-09T16:24:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:24:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5196.007283 secs 2023-03-09T16:24:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:24:45 smithi038 ceph-mon[92422]: pgmap v3284: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 85 KiB/s wr, 8 op/s 2023-03-09T16:24:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:24:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5198.238795 secs 2023-03-09T16:24:47.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:24:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5198.238795 secs 2023-03-09T16:24:47.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:24:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5198.238795 secs 2023-03-09T16:24:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:24:47 smithi122 ceph-mon[93869]: pgmap v3285: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 179 KiB/s wr, 12 op/s 2023-03-09T16:24:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:24:47 smithi181 ceph-mon[94189]: pgmap v3285: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 179 KiB/s wr, 12 op/s 2023-03-09T16:24:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:24:47 smithi038 ceph-mon[92422]: pgmap v3285: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 179 KiB/s wr, 12 op/s 2023-03-09T16:24:48.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5660.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:24:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:24:50 smithi181 ceph-mon[94189]: pgmap v3286: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 140 KiB/s wr, 9 op/s 2023-03-09T16:24:50.747 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:24:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5201.007343 secs 2023-03-09T16:24:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:24:50 smithi038 ceph-mon[92422]: pgmap v3286: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 140 KiB/s wr, 9 op/s 2023-03-09T16:24:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:24:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5201.007343 secs 2023-03-09T16:24:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:24:50 smithi122 ceph-mon[93869]: pgmap v3286: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 140 KiB/s wr, 9 op/s 2023-03-09T16:24:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:24:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5201.007343 secs 2023-03-09T16:24:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:24:51 smithi038 ceph-mon[92422]: pgmap v3287: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 154 KiB/s wr, 12 op/s 2023-03-09T16:24:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:24:51 smithi122 ceph-mon[93869]: pgmap v3287: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 154 KiB/s wr, 12 op/s 2023-03-09T16:24:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:24:51 smithi181 ceph-mon[94189]: pgmap v3287: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 154 KiB/s wr, 12 op/s 2023-03-09T16:24:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:24:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5203.238933 secs 2023-03-09T16:24:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:24:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5203.238933 secs 2023-03-09T16:24:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:24:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5203.238933 secs 2023-03-09T16:24:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5665.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:24:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:24:53 smithi122 ceph-mon[93869]: pgmap v3288: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 109 KiB/s wr, 9 op/s 2023-03-09T16:24:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:24:53 smithi181 ceph-mon[94189]: pgmap v3288: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 109 KiB/s wr, 9 op/s 2023-03-09T16:24:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:24:53 smithi038 ceph-mon[92422]: pgmap v3288: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 109 KiB/s wr, 9 op/s 2023-03-09T16:24:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:24:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5206.007406 secs 2023-03-09T16:24:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:24:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5206.007406 secs 2023-03-09T16:24:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:24:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5206.007406 secs 2023-03-09T16:24:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:24:56 smithi038 ceph-mon[92422]: pgmap v3289: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 109 KiB/s wr, 8 op/s 2023-03-09T16:24:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:24:56 smithi122 ceph-mon[93869]: pgmap v3289: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 109 KiB/s wr, 8 op/s 2023-03-09T16:24:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:24:56 smithi181 ceph-mon[94189]: pgmap v3289: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 109 KiB/s wr, 8 op/s 2023-03-09T16:24:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:24:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5208.239062 secs 2023-03-09T16:24:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:24:57 smithi038 ceph-mon[92422]: pgmap v3290: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 240 KiB/s wr, 13 op/s 2023-03-09T16:24:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:24:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5208.239062 secs 2023-03-09T16:24:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:24:57 smithi122 ceph-mon[93869]: pgmap v3290: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 240 KiB/s wr, 13 op/s 2023-03-09T16:24:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:24:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5208.239062 secs 2023-03-09T16:24:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:24:57 smithi181 ceph-mon[94189]: pgmap v3290: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 240 KiB/s wr, 13 op/s 2023-03-09T16:24:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5670.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:25:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:25:00 smithi181 ceph-mon[94189]: pgmap v3291: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 146 KiB/s wr, 9 op/s 2023-03-09T16:25:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:25:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5211.007467 secs 2023-03-09T16:25:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:25:00 smithi038 ceph-mon[92422]: pgmap v3291: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 146 KiB/s wr, 9 op/s 2023-03-09T16:25:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:25:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5211.007467 secs 2023-03-09T16:25:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:25:00 smithi122 ceph-mon[93869]: pgmap v3291: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 146 KiB/s wr, 9 op/s 2023-03-09T16:25:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:25:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5211.007467 secs 2023-03-09T16:25:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:25:01 smithi038 ceph-mon[92422]: pgmap v3292: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 191 KiB/s wr, 15 op/s 2023-03-09T16:25:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:25:01 smithi122 ceph-mon[93869]: pgmap v3292: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 191 KiB/s wr, 15 op/s 2023-03-09T16:25:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:25:01 smithi181 ceph-mon[94189]: pgmap v3292: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 191 KiB/s wr, 15 op/s 2023-03-09T16:25:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:25:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5213.239200 secs 2023-03-09T16:25:02.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:25:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5213.239200 secs 2023-03-09T16:25:02.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:25:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5213.239200 secs 2023-03-09T16:25:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5675.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:25:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:25:03 smithi122 ceph-mon[93869]: pgmap v3293: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 199 KiB/s wr, 24 op/s 2023-03-09T16:25:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:25:03 smithi181 ceph-mon[94189]: pgmap v3293: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 199 KiB/s wr, 24 op/s 2023-03-09T16:25:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:25:03 smithi038 ceph-mon[92422]: pgmap v3293: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 199 KiB/s wr, 24 op/s 2023-03-09T16:25:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:25:05 smithi122 ceph-mon[93869]: osdmap e418: 12 total, 12 up, 12 in 2023-03-09T16:25:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:25:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5216.007530 secs 2023-03-09T16:25:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:25:05 smithi122 ceph-mon[93869]: pgmap v3295: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 238 KiB/s wr, 28 op/s 2023-03-09T16:25:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:25:05 smithi181 ceph-mon[94189]: osdmap e418: 12 total, 12 up, 12 in 2023-03-09T16:25:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:25:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5216.007530 secs 2023-03-09T16:25:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:25:05 smithi181 ceph-mon[94189]: pgmap v3295: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 238 KiB/s wr, 28 op/s 2023-03-09T16:25:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:25:05 smithi038 ceph-mon[92422]: osdmap e418: 12 total, 12 up, 12 in 2023-03-09T16:25:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:25:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5216.007530 secs 2023-03-09T16:25:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:25:05 smithi038 ceph-mon[92422]: pgmap v3295: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 238 KiB/s wr, 28 op/s 2023-03-09T16:25:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:25:06 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:25:06.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:25:06 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:25:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:25:06 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:25:07.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:25:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5218.239313 secs 2023-03-09T16:25:07.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:25:07 smithi122 ceph-mon[93869]: pgmap v3296: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 166 KiB/s wr, 27 op/s 2023-03-09T16:25:07.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:25:07 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:25:07.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:25:07 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:25:07.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:25:07 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:25:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:25:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5218.239313 secs 2023-03-09T16:25:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:25:07 smithi181 ceph-mon[94189]: pgmap v3296: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 166 KiB/s wr, 27 op/s 2023-03-09T16:25:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:25:07 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:25:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:25:07 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:25:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:25:07 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:25:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:25:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5218.239313 secs 2023-03-09T16:25:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:25:07 smithi038 ceph-mon[92422]: pgmap v3296: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 166 KiB/s wr, 27 op/s 2023-03-09T16:25:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:25:07 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:25:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:25:07 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:25:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:25:07 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:25:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5680.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:25:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:25:09 smithi122 ceph-mon[93869]: pgmap v3297: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 166 KiB/s wr, 26 op/s 2023-03-09T16:25:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:25:09 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5221.007590 secs 2023-03-09T16:25:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:25:09 smithi181 ceph-mon[94189]: pgmap v3297: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 166 KiB/s wr, 26 op/s 2023-03-09T16:25:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:25:09 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5221.007590 secs 2023-03-09T16:25:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:25:09 smithi038 ceph-mon[92422]: pgmap v3297: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 166 KiB/s wr, 26 op/s 2023-03-09T16:25:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:25:09 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5221.007590 secs 2023-03-09T16:25:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:25:11 smithi038 ceph-mon[92422]: pgmap v3298: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 139 KiB/s wr, 23 op/s 2023-03-09T16:25:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:25:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5223.239377 secs 2023-03-09T16:25:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:25:11 smithi122 ceph-mon[93869]: pgmap v3298: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 139 KiB/s wr, 23 op/s 2023-03-09T16:25:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:25:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5223.239377 secs 2023-03-09T16:25:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:25:11 smithi181 ceph-mon[94189]: pgmap v3298: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 139 KiB/s wr, 23 op/s 2023-03-09T16:25:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:25:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5223.239377 secs 2023-03-09T16:25:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5685.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:25:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:25:13 smithi038 ceph-mon[92422]: pgmap v3299: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 113 KiB/s wr, 8 op/s 2023-03-09T16:25:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:25:13 smithi122 ceph-mon[93869]: pgmap v3299: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 113 KiB/s wr, 8 op/s 2023-03-09T16:25:14.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:25:13 smithi181 ceph-mon[94189]: pgmap v3299: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 113 KiB/s wr, 8 op/s 2023-03-09T16:25:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:25:15 smithi038 ceph-mon[92422]: osdmap e419: 12 total, 12 up, 12 in 2023-03-09T16:25:15.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:25:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5226.007666 secs 2023-03-09T16:25:15.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:25:15 smithi038 ceph-mon[92422]: pgmap v3301: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 113 KiB/s wr, 8 op/s 2023-03-09T16:25:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:25:15 smithi122 ceph-mon[93869]: osdmap e419: 12 total, 12 up, 12 in 2023-03-09T16:25:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:25:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5226.007666 secs 2023-03-09T16:25:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:25:15 smithi122 ceph-mon[93869]: pgmap v3301: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 113 KiB/s wr, 8 op/s 2023-03-09T16:25:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:25:15 smithi181 ceph-mon[94189]: osdmap e419: 12 total, 12 up, 12 in 2023-03-09T16:25:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:25:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5226.007666 secs 2023-03-09T16:25:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:25:15 smithi181 ceph-mon[94189]: pgmap v3301: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 113 KiB/s wr, 8 op/s 2023-03-09T16:25:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:25:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5228.239489 secs 2023-03-09T16:25:17.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:25:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5228.239489 secs 2023-03-09T16:25:17.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:25:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5228.239489 secs 2023-03-09T16:25:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:25:17 smithi122 ceph-mon[93869]: pgmap v3302: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T16:25:18.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:25:17 smithi181 ceph-mon[94189]: pgmap v3302: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T16:25:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:25:17 smithi038 ceph-mon[92422]: pgmap v3302: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T16:25:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5690.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:25:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:25:19 smithi122 ceph-mon[93869]: pgmap v3303: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T16:25:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:25:19 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5231.007725 secs 2023-03-09T16:25:20.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:25:19 smithi181 ceph-mon[94189]: pgmap v3303: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T16:25:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:25:19 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5231.007725 secs 2023-03-09T16:25:20.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:25:19 smithi038 ceph-mon[92422]: pgmap v3303: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T16:25:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:25:19 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5231.007725 secs 2023-03-09T16:25:22.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:25:21 smithi181 ceph-mon[94189]: pgmap v3304: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 39 KiB/s wr, 7 op/s 2023-03-09T16:25:22.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:25:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5233.239616 secs 2023-03-09T16:25:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:25:21 smithi038 ceph-mon[92422]: pgmap v3304: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 39 KiB/s wr, 7 op/s 2023-03-09T16:25:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:25:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5233.239616 secs 2023-03-09T16:25:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:25:21 smithi122 ceph-mon[93869]: pgmap v3304: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 39 KiB/s wr, 7 op/s 2023-03-09T16:25:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:25:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5233.239616 secs 2023-03-09T16:25:23.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5695.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:25:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:25:23 smithi038 ceph-mon[92422]: pgmap v3305: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 39 KiB/s wr, 8 op/s 2023-03-09T16:25:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:25:23 smithi122 ceph-mon[93869]: pgmap v3305: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 39 KiB/s wr, 8 op/s 2023-03-09T16:25:24.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:25:23 smithi181 ceph-mon[94189]: pgmap v3305: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 39 KiB/s wr, 8 op/s 2023-03-09T16:25:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:25:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5236.007785 secs 2023-03-09T16:25:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:25:25 smithi038 ceph-mon[92422]: pgmap v3306: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 37 KiB/s wr, 7 op/s 2023-03-09T16:25:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:25:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5236.007785 secs 2023-03-09T16:25:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:25:25 smithi122 ceph-mon[93869]: pgmap v3306: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 37 KiB/s wr, 7 op/s 2023-03-09T16:25:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:25:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5236.007785 secs 2023-03-09T16:25:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:25:25 smithi181 ceph-mon[94189]: pgmap v3306: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 37 KiB/s wr, 7 op/s 2023-03-09T16:25:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:25:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5238.239735 secs 2023-03-09T16:25:27.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:25:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5238.239735 secs 2023-03-09T16:25:27.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:25:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5238.239735 secs 2023-03-09T16:25:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:25:27 smithi122 ceph-mon[93869]: pgmap v3307: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 33 KiB/s wr, 9 op/s 2023-03-09T16:25:28.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:25:27 smithi181 ceph-mon[94189]: pgmap v3307: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 33 KiB/s wr, 9 op/s 2023-03-09T16:25:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:25:27 smithi038 ceph-mon[92422]: pgmap v3307: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 33 KiB/s wr, 9 op/s 2023-03-09T16:25:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5700.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:25:30.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:25:29 smithi181 ceph-mon[94189]: pgmap v3308: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 32 KiB/s wr, 6 op/s 2023-03-09T16:25:30.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:25:29 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5241.007846 secs 2023-03-09T16:25:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:25:29 smithi038 ceph-mon[92422]: pgmap v3308: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 32 KiB/s wr, 6 op/s 2023-03-09T16:25:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:25:29 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5241.007846 secs 2023-03-09T16:25:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:25:29 smithi122 ceph-mon[93869]: pgmap v3308: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 32 KiB/s wr, 6 op/s 2023-03-09T16:25:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:25:29 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5241.007846 secs 2023-03-09T16:25:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:25:31 smithi038 ceph-mon[92422]: pgmap v3309: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 137 KiB/s wr, 10 op/s 2023-03-09T16:25:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:25:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5243.239849 secs 2023-03-09T16:25:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:25:31 smithi122 ceph-mon[93869]: pgmap v3309: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 137 KiB/s wr, 10 op/s 2023-03-09T16:25:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:25:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5243.239849 secs 2023-03-09T16:25:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:25:31 smithi181 ceph-mon[94189]: pgmap v3309: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 137 KiB/s wr, 10 op/s 2023-03-09T16:25:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:25:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5243.239849 secs 2023-03-09T16:25:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5705.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:25:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:25:33 smithi038 ceph-mon[92422]: pgmap v3310: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 105 KiB/s wr, 7 op/s 2023-03-09T16:25:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:25:33 smithi122 ceph-mon[93869]: pgmap v3310: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 105 KiB/s wr, 7 op/s 2023-03-09T16:25:34.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:25:33 smithi181 ceph-mon[94189]: pgmap v3310: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 105 KiB/s wr, 7 op/s 2023-03-09T16:25:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:25:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5246.007904 secs 2023-03-09T16:25:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:25:35 smithi038 ceph-mon[92422]: pgmap v3311: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 105 KiB/s wr, 7 op/s 2023-03-09T16:25:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:25:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5246.007904 secs 2023-03-09T16:25:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:25:35 smithi122 ceph-mon[93869]: pgmap v3311: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 105 KiB/s wr, 7 op/s 2023-03-09T16:25:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:25:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5246.007904 secs 2023-03-09T16:25:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:25:35 smithi181 ceph-mon[94189]: pgmap v3311: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 105 KiB/s wr, 7 op/s 2023-03-09T16:25:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:25:37 smithi181 ceph-mon[94189]: pgmap v3312: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 185 KiB/s wr, 12 op/s 2023-03-09T16:25:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:25:37 smithi038 ceph-mon[92422]: pgmap v3312: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 185 KiB/s wr, 12 op/s 2023-03-09T16:25:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:25:37 smithi122 ceph-mon[93869]: pgmap v3312: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 185 KiB/s wr, 12 op/s 2023-03-09T16:25:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5710.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:25:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:25:39 smithi038 ceph-mon[92422]: pgmap v3313: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 185 KiB/s wr, 10 op/s 2023-03-09T16:25:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:25:39 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:25:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:25:39 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5251.007966 secs 2023-03-09T16:25:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:25:39 smithi122 ceph-mon[93869]: pgmap v3313: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 185 KiB/s wr, 10 op/s 2023-03-09T16:25:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:25:39 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:25:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:25:39 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5251.007966 secs 2023-03-09T16:25:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:25:39 smithi181 ceph-mon[94189]: pgmap v3313: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 185 KiB/s wr, 10 op/s 2023-03-09T16:25:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:25:39 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:25:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:25:39 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5251.007966 secs 2023-03-09T16:25:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:25:40 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:25:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:25:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:25:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:25:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:25:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:25:40 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:25:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:25:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:25:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:25:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:25:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:25:40 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:25:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:25:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:25:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:25:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:25:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:25:41 smithi038 ceph-mon[92422]: pgmap v3314: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 239 KiB/s wr, 14 op/s 2023-03-09T16:25:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:25:41 smithi122 ceph-mon[93869]: pgmap v3314: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 239 KiB/s wr, 14 op/s 2023-03-09T16:25:42.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:25:41 smithi181 ceph-mon[94189]: pgmap v3314: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 239 KiB/s wr, 14 op/s 2023-03-09T16:25:43.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5715.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:25:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:25:43 smithi038 ceph-mon[92422]: pgmap v3315: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 135 KiB/s wr, 11 op/s 2023-03-09T16:25:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:25:43 smithi122 ceph-mon[93869]: pgmap v3315: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 135 KiB/s wr, 11 op/s 2023-03-09T16:25:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:25:43 smithi181 ceph-mon[94189]: pgmap v3315: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 135 KiB/s wr, 11 op/s 2023-03-09T16:25:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:25:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5256.008001 secs 2023-03-09T16:25:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:25:45 smithi122 ceph-mon[93869]: pgmap v3316: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 135 KiB/s wr, 10 op/s 2023-03-09T16:25:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:25:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5256.008001 secs 2023-03-09T16:25:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:25:45 smithi181 ceph-mon[94189]: pgmap v3316: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 135 KiB/s wr, 10 op/s 2023-03-09T16:25:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:25:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5256.008001 secs 2023-03-09T16:25:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:25:45 smithi038 ceph-mon[92422]: pgmap v3316: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 135 KiB/s wr, 10 op/s 2023-03-09T16:25:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:25:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5258.240179 secs 2023-03-09T16:25:47.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:25:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5258.240179 secs 2023-03-09T16:25:47.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:25:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5258.240179 secs 2023-03-09T16:25:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:25:47 smithi122 ceph-mon[93869]: pgmap v3317: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 241 KiB/s wr, 14 op/s 2023-03-09T16:25:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:25:47 smithi181 ceph-mon[94189]: pgmap v3317: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 241 KiB/s wr, 14 op/s 2023-03-09T16:25:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:25:47 smithi038 ceph-mon[92422]: pgmap v3317: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 241 KiB/s wr, 14 op/s 2023-03-09T16:25:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5720.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:25:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:25:48 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:25:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:25:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:25:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:25:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:25:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:25:48 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:25:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:25:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:25:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:25:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:25:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:25:48 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:25:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:25:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:25:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:25:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:25:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:25:49 smithi122 ceph-mon[93869]: pgmap v3318: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 161 KiB/s wr, 9 op/s 2023-03-09T16:25:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:25:49 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5261.008065 secs 2023-03-09T16:25:50.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:25:49 smithi181 ceph-mon[94189]: pgmap v3318: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 161 KiB/s wr, 9 op/s 2023-03-09T16:25:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:25:49 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5261.008065 secs 2023-03-09T16:25:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:25:49 smithi038 ceph-mon[92422]: pgmap v3318: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 161 KiB/s wr, 9 op/s 2023-03-09T16:25:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:25:49 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5261.008065 secs 2023-03-09T16:25:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:25:51 smithi122 ceph-mon[93869]: pgmap v3319: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 192 KiB/s wr, 12 op/s 2023-03-09T16:25:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:25:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5263.240299 secs 2023-03-09T16:25:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:25:51 smithi181 ceph-mon[94189]: pgmap v3319: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 192 KiB/s wr, 12 op/s 2023-03-09T16:25:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:25:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5263.240299 secs 2023-03-09T16:25:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:25:51 smithi038 ceph-mon[92422]: pgmap v3319: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 192 KiB/s wr, 12 op/s 2023-03-09T16:25:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:25:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5263.240299 secs 2023-03-09T16:25:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5725.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:25:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:25:53 smithi122 ceph-mon[93869]: pgmap v3320: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 137 KiB/s wr, 8 op/s 2023-03-09T16:25:54.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:25:53 smithi181 ceph-mon[94189]: pgmap v3320: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 137 KiB/s wr, 8 op/s 2023-03-09T16:25:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:25:53 smithi038 ceph-mon[92422]: pgmap v3320: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 137 KiB/s wr, 8 op/s 2023-03-09T16:25:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:25:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5266.008141 secs 2023-03-09T16:25:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:25:55 smithi122 ceph-mon[93869]: pgmap v3321: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 137 KiB/s wr, 8 op/s 2023-03-09T16:25:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:25:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5266.008141 secs 2023-03-09T16:25:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:25:55 smithi181 ceph-mon[94189]: pgmap v3321: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 137 KiB/s wr, 8 op/s 2023-03-09T16:25:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:25:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5266.008141 secs 2023-03-09T16:25:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:25:55 smithi038 ceph-mon[92422]: pgmap v3321: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 137 KiB/s wr, 8 op/s 2023-03-09T16:25:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:25:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5268.240424 secs 2023-03-09T16:25:57.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:25:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5268.240424 secs 2023-03-09T16:25:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:25:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5268.240424 secs 2023-03-09T16:25:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:25:57 smithi122 ceph-mon[93869]: pgmap v3322: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 199 KiB/s wr, 11 op/s 2023-03-09T16:25:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:25:57 smithi181 ceph-mon[94189]: pgmap v3322: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 199 KiB/s wr, 11 op/s 2023-03-09T16:25:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:25:57 smithi038 ceph-mon[92422]: pgmap v3322: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 199 KiB/s wr, 11 op/s 2023-03-09T16:25:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5730.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:26:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:25:59 smithi038 ceph-mon[92422]: pgmap v3323: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 93 KiB/s wr, 7 op/s 2023-03-09T16:26:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:25:59 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5271.008203 secs 2023-03-09T16:26:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:25:59 smithi122 ceph-mon[93869]: pgmap v3323: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 93 KiB/s wr, 7 op/s 2023-03-09T16:26:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:25:59 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5271.008203 secs 2023-03-09T16:26:00.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:25:59 smithi181 ceph-mon[94189]: pgmap v3323: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 93 KiB/s wr, 7 op/s 2023-03-09T16:26:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:25:59 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5271.008203 secs 2023-03-09T16:26:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:26:01 smithi038 ceph-mon[92422]: pgmap v3324: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 105 KiB/s wr, 11 op/s 2023-03-09T16:26:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:26:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5273.240557 secs 2023-03-09T16:26:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:26:01 smithi122 ceph-mon[93869]: pgmap v3324: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 105 KiB/s wr, 11 op/s 2023-03-09T16:26:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:26:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5273.240557 secs 2023-03-09T16:26:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:26:01 smithi181 ceph-mon[94189]: pgmap v3324: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 105 KiB/s wr, 11 op/s 2023-03-09T16:26:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:26:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5273.240557 secs 2023-03-09T16:26:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5735.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:26:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:26:04 smithi122 ceph-mon[93869]: pgmap v3325: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 96 KiB/s wr, 20 op/s 2023-03-09T16:26:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:26:04 smithi181 ceph-mon[94189]: pgmap v3325: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 96 KiB/s wr, 20 op/s 2023-03-09T16:26:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:26:04 smithi038 ceph-mon[92422]: pgmap v3325: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 96 KiB/s wr, 20 op/s 2023-03-09T16:26:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:26:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5276.008263 secs 2023-03-09T16:26:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:26:05 smithi122 ceph-mon[93869]: pgmap v3326: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 96 KiB/s wr, 20 op/s 2023-03-09T16:26:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:26:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5276.008263 secs 2023-03-09T16:26:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:26:05 smithi181 ceph-mon[94189]: pgmap v3326: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 96 KiB/s wr, 20 op/s 2023-03-09T16:26:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:26:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5276.008263 secs 2023-03-09T16:26:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:26:05 smithi038 ceph-mon[92422]: pgmap v3326: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 96 KiB/s wr, 20 op/s 2023-03-09T16:26:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:26:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5278.240633 secs 2023-03-09T16:26:07.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:26:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5278.240633 secs 2023-03-09T16:26:07.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:26:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5278.240633 secs 2023-03-09T16:26:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:26:07 smithi122 ceph-mon[93869]: pgmap v3327: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 142 KiB/s wr, 24 op/s 2023-03-09T16:26:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:26:07 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:26:08.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:26:07 smithi181 ceph-mon[94189]: pgmap v3327: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 142 KiB/s wr, 24 op/s 2023-03-09T16:26:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:26:07 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:26:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:26:07 smithi038 ceph-mon[92422]: pgmap v3327: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 142 KiB/s wr, 24 op/s 2023-03-09T16:26:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:26:07 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:26:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5740.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:26:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:26:08 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:26:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:26:08 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:26:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:26:08 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:26:09.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:26:08 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:26:09.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:26:08 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:26:09.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:26:08 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:26:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:26:08 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:26:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:26:08 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:26:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:26:08 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:26:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:26:09 smithi122 ceph-mon[93869]: pgmap v3328: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 81 KiB/s wr, 20 op/s 2023-03-09T16:26:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:26:09 smithi122 ceph-mon[93869]: osdmap e420: 12 total, 12 up, 12 in 2023-03-09T16:26:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:26:09 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5281.008320 secs 2023-03-09T16:26:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:26:09 smithi181 ceph-mon[94189]: pgmap v3328: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 81 KiB/s wr, 20 op/s 2023-03-09T16:26:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:26:09 smithi181 ceph-mon[94189]: osdmap e420: 12 total, 12 up, 12 in 2023-03-09T16:26:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:26:09 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5281.008320 secs 2023-03-09T16:26:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:26:09 smithi038 ceph-mon[92422]: pgmap v3328: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 81 KiB/s wr, 20 op/s 2023-03-09T16:26:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:26:09 smithi038 ceph-mon[92422]: osdmap e420: 12 total, 12 up, 12 in 2023-03-09T16:26:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:26:09 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5281.008320 secs 2023-03-09T16:26:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:26:11 smithi122 ceph-mon[93869]: pgmap v3330: 129 pgs: 5 active+clean+snaptrim_wait, 124 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 88 KiB/s wr, 23 op/s 2023-03-09T16:26:12.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:26:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5283.240762 secs 2023-03-09T16:26:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:26:11 smithi181 ceph-mon[94189]: pgmap v3330: 129 pgs: 5 active+clean+snaptrim_wait, 124 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 88 KiB/s wr, 23 op/s 2023-03-09T16:26:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:26:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5283.240762 secs 2023-03-09T16:26:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:26:11 smithi038 ceph-mon[92422]: pgmap v3330: 129 pgs: 5 active+clean+snaptrim_wait, 124 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 88 KiB/s wr, 23 op/s 2023-03-09T16:26:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:26:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5283.240762 secs 2023-03-09T16:26:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5745.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:26:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:26:13 smithi181 ceph-mon[94189]: pgmap v3331: 129 pgs: 5 active+clean+snaptrim_wait, 124 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 62 KiB/s wr, 9 op/s 2023-03-09T16:26:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:26:13 smithi038 ceph-mon[92422]: pgmap v3331: 129 pgs: 5 active+clean+snaptrim_wait, 124 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 62 KiB/s wr, 9 op/s 2023-03-09T16:26:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:26:13 smithi122 ceph-mon[93869]: pgmap v3331: 129 pgs: 5 active+clean+snaptrim_wait, 124 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 62 KiB/s wr, 9 op/s 2023-03-09T16:26:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:26:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5286.008390 secs 2023-03-09T16:26:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:26:15 smithi122 ceph-mon[93869]: pgmap v3332: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 62 KiB/s wr, 9 op/s 2023-03-09T16:26:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:26:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5286.008390 secs 2023-03-09T16:26:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:26:15 smithi181 ceph-mon[94189]: pgmap v3332: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 62 KiB/s wr, 9 op/s 2023-03-09T16:26:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:26:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5286.008390 secs 2023-03-09T16:26:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:26:15 smithi038 ceph-mon[92422]: pgmap v3332: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 62 KiB/s wr, 9 op/s 2023-03-09T16:26:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:26:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5288.240915 secs 2023-03-09T16:26:17.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:26:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5288.240915 secs 2023-03-09T16:26:17.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:26:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5288.240915 secs 2023-03-09T16:26:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:26:17 smithi122 ceph-mon[93869]: pgmap v3333: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 61 KiB/s wr, 8 op/s 2023-03-09T16:26:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:26:17 smithi181 ceph-mon[94189]: pgmap v3333: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 61 KiB/s wr, 8 op/s 2023-03-09T16:26:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:26:17 smithi038 ceph-mon[92422]: pgmap v3333: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 61 KiB/s wr, 8 op/s 2023-03-09T16:26:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5750.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:26:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:26:19 smithi038 ceph-mon[92422]: pgmap v3334: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 60 KiB/s wr, 9 op/s 2023-03-09T16:26:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:26:19 smithi038 ceph-mon[92422]: osdmap e421: 12 total, 12 up, 12 in 2023-03-09T16:26:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:26:19 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5291.008449 secs 2023-03-09T16:26:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:26:19 smithi122 ceph-mon[93869]: pgmap v3334: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 60 KiB/s wr, 9 op/s 2023-03-09T16:26:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:26:19 smithi122 ceph-mon[93869]: osdmap e421: 12 total, 12 up, 12 in 2023-03-09T16:26:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:26:19 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5291.008449 secs 2023-03-09T16:26:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:26:19 smithi181 ceph-mon[94189]: pgmap v3334: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 60 KiB/s wr, 9 op/s 2023-03-09T16:26:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:26:19 smithi181 ceph-mon[94189]: osdmap e421: 12 total, 12 up, 12 in 2023-03-09T16:26:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:26:19 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5291.008449 secs 2023-03-09T16:26:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:26:22 smithi122 ceph-mon[93869]: pgmap v3336: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 56 KiB/s wr, 8 op/s 2023-03-09T16:26:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:26:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5293.241043 secs 2023-03-09T16:26:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:26:22 smithi181 ceph-mon[94189]: pgmap v3336: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 56 KiB/s wr, 8 op/s 2023-03-09T16:26:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:26:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5293.241043 secs 2023-03-09T16:26:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:26:22 smithi038 ceph-mon[92422]: pgmap v3336: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 56 KiB/s wr, 8 op/s 2023-03-09T16:26:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:26:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5293.241043 secs 2023-03-09T16:26:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5755.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:26:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:26:24 smithi122 ceph-mon[93869]: pgmap v3337: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 126 KiB/s wr, 9 op/s 2023-03-09T16:26:24.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:26:24 smithi181 ceph-mon[94189]: pgmap v3337: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 126 KiB/s wr, 9 op/s 2023-03-09T16:26:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:26:24 smithi038 ceph-mon[92422]: pgmap v3337: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 126 KiB/s wr, 9 op/s 2023-03-09T16:26:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:26:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5296.008510 secs 2023-03-09T16:26:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:26:25 smithi122 ceph-mon[93869]: pgmap v3338: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 126 KiB/s wr, 9 op/s 2023-03-09T16:26:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:26:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5296.008510 secs 2023-03-09T16:26:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:26:25 smithi181 ceph-mon[94189]: pgmap v3338: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 126 KiB/s wr, 9 op/s 2023-03-09T16:26:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:26:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5296.008510 secs 2023-03-09T16:26:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:26:25 smithi038 ceph-mon[92422]: pgmap v3338: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 126 KiB/s wr, 9 op/s 2023-03-09T16:26:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:26:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5298.241136 secs 2023-03-09T16:26:27.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:26:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5298.241136 secs 2023-03-09T16:26:27.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:26:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5298.241136 secs 2023-03-09T16:26:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:26:27 smithi122 ceph-mon[93869]: pgmap v3339: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 73 KiB/s wr, 8 op/s 2023-03-09T16:26:28.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:26:27 smithi181 ceph-mon[94189]: pgmap v3339: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 73 KiB/s wr, 8 op/s 2023-03-09T16:26:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:26:27 smithi038 ceph-mon[92422]: pgmap v3339: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 73 KiB/s wr, 8 op/s 2023-03-09T16:26:28.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5760.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:26:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:26:30 smithi122 ceph-mon[93869]: pgmap v3340: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 73 KiB/s wr, 8 op/s 2023-03-09T16:26:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:26:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5301.008570 secs 2023-03-09T16:26:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:26:30 smithi181 ceph-mon[94189]: pgmap v3340: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 73 KiB/s wr, 8 op/s 2023-03-09T16:26:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:26:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5301.008570 secs 2023-03-09T16:26:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:26:30 smithi038 ceph-mon[92422]: pgmap v3340: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 73 KiB/s wr, 8 op/s 2023-03-09T16:26:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:26:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5301.008570 secs 2023-03-09T16:26:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:26:32 smithi122 ceph-mon[93869]: pgmap v3341: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 64 KiB/s wr, 10 op/s 2023-03-09T16:26:32.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:26:32 smithi181 ceph-mon[94189]: pgmap v3341: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 64 KiB/s wr, 10 op/s 2023-03-09T16:26:32.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:26:32 smithi038 ceph-mon[92422]: pgmap v3341: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 64 KiB/s wr, 10 op/s 2023-03-09T16:26:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5765.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:26:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:26:34 smithi122 ceph-mon[93869]: pgmap v3342: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 95 KiB/s wr, 8 op/s 2023-03-09T16:26:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:26:34 smithi181 ceph-mon[94189]: pgmap v3342: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 95 KiB/s wr, 8 op/s 2023-03-09T16:26:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:26:34 smithi038 ceph-mon[92422]: pgmap v3342: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 95 KiB/s wr, 8 op/s 2023-03-09T16:26:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:26:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5306.008629 secs 2023-03-09T16:26:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:26:35 smithi122 ceph-mon[93869]: pgmap v3343: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 KiB/s wr, 7 op/s 2023-03-09T16:26:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:26:35 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:26:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:26:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5306.008629 secs 2023-03-09T16:26:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:26:35 smithi181 ceph-mon[94189]: pgmap v3343: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 KiB/s wr, 7 op/s 2023-03-09T16:26:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:26:35 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:26:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:26:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5306.008629 secs 2023-03-09T16:26:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:26:35 smithi038 ceph-mon[92422]: pgmap v3343: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 KiB/s wr, 7 op/s 2023-03-09T16:26:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:26:35 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:26:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:26:36 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:26:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:26:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:26:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:26:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:26:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:26:36 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:26:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:26:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:26:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:26:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:26:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:26:36 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:26:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:26:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:26:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:26:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:26:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:26:37 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 5308.241379 secs 2023-03-09T16:26:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:26:37 smithi122 ceph-mon[93869]: pgmap v3344: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 41 KiB/s wr, 9 op/s 2023-03-09T16:26:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:26:37 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 5308.241379 secs 2023-03-09T16:26:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:26:37 smithi181 ceph-mon[94189]: pgmap v3344: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 41 KiB/s wr, 9 op/s 2023-03-09T16:26:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:26:37 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 5308.241379 secs 2023-03-09T16:26:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:26:37 smithi038 ceph-mon[92422]: pgmap v3344: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 41 KiB/s wr, 9 op/s 2023-03-09T16:26:38.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5770.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:26:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:26:40 smithi122 ceph-mon[93869]: pgmap v3345: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 103 KiB/s wr, 8 op/s 2023-03-09T16:26:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:26:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5311.008691 secs 2023-03-09T16:26:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:26:40 smithi181 ceph-mon[94189]: pgmap v3345: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 103 KiB/s wr, 8 op/s 2023-03-09T16:26:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:26:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5311.008691 secs 2023-03-09T16:26:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:26:40 smithi038 ceph-mon[92422]: pgmap v3345: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 103 KiB/s wr, 8 op/s 2023-03-09T16:26:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:26:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5311.008691 secs 2023-03-09T16:26:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:26:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:26:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:26:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:26:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:26:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:26:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:26:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:26:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:26:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:26:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:26:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:26:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:26:42 smithi122 ceph-mon[93869]: pgmap v3346: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 103 KiB/s wr, 11 op/s 2023-03-09T16:26:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:26:42 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 5313.241499 secs 2023-03-09T16:26:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:26:42 smithi181 ceph-mon[94189]: pgmap v3346: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 103 KiB/s wr, 11 op/s 2023-03-09T16:26:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:26:42 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 5313.241499 secs 2023-03-09T16:26:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:26:42 smithi038 ceph-mon[92422]: pgmap v3346: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 103 KiB/s wr, 11 op/s 2023-03-09T16:26:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:26:42 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 5313.241499 secs 2023-03-09T16:26:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5775.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:26:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:26:44 smithi122 ceph-mon[93869]: pgmap v3347: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 188 KiB/s wr, 10 op/s 2023-03-09T16:26:44.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:26:44 smithi181 ceph-mon[94189]: pgmap v3347: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 188 KiB/s wr, 10 op/s 2023-03-09T16:26:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:26:44 smithi038 ceph-mon[92422]: pgmap v3347: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 188 KiB/s wr, 10 op/s 2023-03-09T16:26:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:26:45 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:26:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:26:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5316.008752 secs 2023-03-09T16:26:45.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:26:45 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:26:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:26:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5316.008752 secs 2023-03-09T16:26:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:26:45 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:26:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:26:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5316.008752 secs 2023-03-09T16:26:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:26:46 smithi122 ceph-mon[93869]: pgmap v3348: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 153 KiB/s wr, 9 op/s 2023-03-09T16:26:46.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:26:46 smithi181 ceph-mon[94189]: pgmap v3348: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 153 KiB/s wr, 9 op/s 2023-03-09T16:26:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:26:46 smithi038 ceph-mon[92422]: pgmap v3348: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 153 KiB/s wr, 9 op/s 2023-03-09T16:26:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:26:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5318.241635 secs 2023-03-09T16:26:47.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:26:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5318.241635 secs 2023-03-09T16:26:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:26:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5318.241635 secs 2023-03-09T16:26:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:26:48 smithi122 ceph-mon[93869]: pgmap v3349: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 162 KiB/s wr, 12 op/s 2023-03-09T16:26:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:26:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:26:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:26:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:26:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:26:48 smithi181 ceph-mon[94189]: pgmap v3349: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 162 KiB/s wr, 12 op/s 2023-03-09T16:26:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:26:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:26:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:26:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:26:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:26:48 smithi038 ceph-mon[92422]: pgmap v3349: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 162 KiB/s wr, 12 op/s 2023-03-09T16:26:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:26:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:26:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:26:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:26:48.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5780.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:26:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:26:50 smithi122 ceph-mon[93869]: pgmap v3350: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 193 KiB/s wr, 10 op/s 2023-03-09T16:26:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:26:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5321.008813 secs 2023-03-09T16:26:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:26:50 smithi181 ceph-mon[94189]: pgmap v3350: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 193 KiB/s wr, 10 op/s 2023-03-09T16:26:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:26:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5321.008813 secs 2023-03-09T16:26:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:26:50 smithi038 ceph-mon[92422]: pgmap v3350: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 193 KiB/s wr, 10 op/s 2023-03-09T16:26:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:26:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5321.008813 secs 2023-03-09T16:26:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:26:52 smithi122 ceph-mon[93869]: pgmap v3351: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 136 KiB/s wr, 11 op/s 2023-03-09T16:26:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:26:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5323.241763 secs 2023-03-09T16:26:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:26:52 smithi181 ceph-mon[94189]: pgmap v3351: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 136 KiB/s wr, 11 op/s 2023-03-09T16:26:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:26:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5323.241763 secs 2023-03-09T16:26:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:26:52 smithi038 ceph-mon[92422]: pgmap v3351: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 136 KiB/s wr, 11 op/s 2023-03-09T16:26:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:26:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5323.241763 secs 2023-03-09T16:26:53.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5785.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:26:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:26:54 smithi122 ceph-mon[93869]: pgmap v3352: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 191 KiB/s wr, 10 op/s 2023-03-09T16:26:54.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:26:54 smithi181 ceph-mon[94189]: pgmap v3352: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 191 KiB/s wr, 10 op/s 2023-03-09T16:26:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:26:54 smithi038 ceph-mon[92422]: pgmap v3352: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 191 KiB/s wr, 10 op/s 2023-03-09T16:26:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:26:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5326.008876 secs 2023-03-09T16:26:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:26:55 smithi122 ceph-mon[93869]: pgmap v3353: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 105 KiB/s wr, 8 op/s 2023-03-09T16:26:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:26:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5326.008876 secs 2023-03-09T16:26:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:26:55 smithi181 ceph-mon[94189]: pgmap v3353: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 105 KiB/s wr, 8 op/s 2023-03-09T16:26:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:26:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5326.008876 secs 2023-03-09T16:26:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:26:55 smithi038 ceph-mon[92422]: pgmap v3353: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 105 KiB/s wr, 8 op/s 2023-03-09T16:26:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:26:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5328.241844 secs 2023-03-09T16:26:57.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:26:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5328.241844 secs 2023-03-09T16:26:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:26:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5328.241844 secs 2023-03-09T16:26:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:26:57 smithi122 ceph-mon[93869]: pgmap v3354: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 146 KiB/s wr, 11 op/s 2023-03-09T16:26:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:26:57 smithi181 ceph-mon[94189]: pgmap v3354: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 146 KiB/s wr, 11 op/s 2023-03-09T16:26:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:26:57 smithi038 ceph-mon[92422]: pgmap v3354: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 146 KiB/s wr, 11 op/s 2023-03-09T16:26:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5790.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:27:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:27:00 smithi122 ceph-mon[93869]: pgmap v3355: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 147 KiB/s wr, 9 op/s 2023-03-09T16:27:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:27:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5331.008934 secs 2023-03-09T16:27:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:27:00 smithi181 ceph-mon[94189]: pgmap v3355: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 147 KiB/s wr, 9 op/s 2023-03-09T16:27:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:27:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5331.008934 secs 2023-03-09T16:27:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:27:00 smithi038 ceph-mon[92422]: pgmap v3355: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 147 KiB/s wr, 9 op/s 2023-03-09T16:27:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:27:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5331.008934 secs 2023-03-09T16:27:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:27:02 smithi122 ceph-mon[93869]: pgmap v3356: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 179 KiB/s wr, 11 op/s 2023-03-09T16:27:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:27:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5333.241957 secs 2023-03-09T16:27:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:27:02 smithi181 ceph-mon[94189]: pgmap v3356: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 179 KiB/s wr, 11 op/s 2023-03-09T16:27:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:27:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5333.241957 secs 2023-03-09T16:27:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:27:02 smithi038 ceph-mon[92422]: pgmap v3356: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 179 KiB/s wr, 11 op/s 2023-03-09T16:27:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:27:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5333.241957 secs 2023-03-09T16:27:03.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5795.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:27:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:27:04 smithi122 ceph-mon[93869]: pgmap v3357: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 196 KiB/s wr, 21 op/s 2023-03-09T16:27:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:27:04 smithi181 ceph-mon[94189]: pgmap v3357: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 196 KiB/s wr, 21 op/s 2023-03-09T16:27:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:27:04 smithi038 ceph-mon[92422]: pgmap v3357: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 196 KiB/s wr, 21 op/s 2023-03-09T16:27:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:27:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5336.008983 secs 2023-03-09T16:27:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:27:05 smithi122 ceph-mon[93869]: pgmap v3358: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 141 KiB/s wr, 20 op/s 2023-03-09T16:27:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:27:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5336.008983 secs 2023-03-09T16:27:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:27:05 smithi181 ceph-mon[94189]: pgmap v3358: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 141 KiB/s wr, 20 op/s 2023-03-09T16:27:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:27:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5336.008983 secs 2023-03-09T16:27:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:27:05 smithi038 ceph-mon[92422]: pgmap v3358: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 141 KiB/s wr, 20 op/s 2023-03-09T16:27:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:27:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5338.242073 secs 2023-03-09T16:27:07.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:27:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5338.242073 secs 2023-03-09T16:27:07.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:27:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5338.242073 secs 2023-03-09T16:27:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:27:07 smithi122 ceph-mon[93869]: pgmap v3359: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 195 KiB/s wr, 26 op/s 2023-03-09T16:27:08.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:27:07 smithi181 ceph-mon[94189]: pgmap v3359: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 195 KiB/s wr, 26 op/s 2023-03-09T16:27:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:27:07 smithi038 ceph-mon[92422]: pgmap v3359: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 195 KiB/s wr, 26 op/s 2023-03-09T16:27:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5800.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:27:09.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:27:08 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:27:09.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:27:08 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:27:09.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:27:08 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:27:09.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:27:08 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:27:09.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:27:08 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:27:09.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:27:08 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:27:09.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:27:08 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:27:09.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:27:08 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:27:09.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:27:08 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:27:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:27:08 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:27:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:27:08 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:27:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:27:08 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:27:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:27:09 smithi122 ceph-mon[93869]: osdmap e422: 12 total, 12 up, 12 in 2023-03-09T16:27:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:27:09 smithi122 ceph-mon[93869]: pgmap v3361: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 172 KiB/s wr, 26 op/s 2023-03-09T16:27:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:27:09 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5341.009043 secs 2023-03-09T16:27:10.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:27:09 smithi181 ceph-mon[94189]: osdmap e422: 12 total, 12 up, 12 in 2023-03-09T16:27:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:27:09 smithi181 ceph-mon[94189]: pgmap v3361: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 172 KiB/s wr, 26 op/s 2023-03-09T16:27:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:27:09 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5341.009043 secs 2023-03-09T16:27:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:27:09 smithi038 ceph-mon[92422]: osdmap e422: 12 total, 12 up, 12 in 2023-03-09T16:27:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:27:09 smithi038 ceph-mon[92422]: pgmap v3361: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 172 KiB/s wr, 26 op/s 2023-03-09T16:27:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:27:09 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5341.009043 secs 2023-03-09T16:27:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:27:12 smithi122 ceph-mon[93869]: pgmap v3362: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 96 KiB/s wr, 25 op/s 2023-03-09T16:27:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:27:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5343.242202 secs 2023-03-09T16:27:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:27:12 smithi181 ceph-mon[94189]: pgmap v3362: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 96 KiB/s wr, 25 op/s 2023-03-09T16:27:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:27:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5343.242202 secs 2023-03-09T16:27:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:27:12 smithi038 ceph-mon[92422]: pgmap v3362: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 96 KiB/s wr, 25 op/s 2023-03-09T16:27:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:27:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5343.242202 secs 2023-03-09T16:27:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5805.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:27:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:27:14 smithi122 ceph-mon[93869]: pgmap v3363: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 70 KiB/s wr, 11 op/s 2023-03-09T16:27:14.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:27:14 smithi181 ceph-mon[94189]: pgmap v3363: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 70 KiB/s wr, 11 op/s 2023-03-09T16:27:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:27:14 smithi038 ceph-mon[92422]: pgmap v3363: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 70 KiB/s wr, 11 op/s 2023-03-09T16:27:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:27:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5346.009101 secs 2023-03-09T16:27:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:27:15 smithi122 ceph-mon[93869]: pgmap v3364: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 71 KiB/s wr, 11 op/s 2023-03-09T16:27:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:27:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5346.009101 secs 2023-03-09T16:27:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:27:15 smithi181 ceph-mon[94189]: pgmap v3364: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 71 KiB/s wr, 11 op/s 2023-03-09T16:27:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:27:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5346.009101 secs 2023-03-09T16:27:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:27:15 smithi038 ceph-mon[92422]: pgmap v3364: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 71 KiB/s wr, 11 op/s 2023-03-09T16:27:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:27:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5348.242353 secs 2023-03-09T16:27:17.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:27:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5348.242353 secs 2023-03-09T16:27:17.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:27:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5348.242353 secs 2023-03-09T16:27:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:27:17 smithi122 ceph-mon[93869]: pgmap v3365: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 34 KiB/s wr, 8 op/s 2023-03-09T16:27:18.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:27:17 smithi181 ceph-mon[94189]: pgmap v3365: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 34 KiB/s wr, 8 op/s 2023-03-09T16:27:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:27:17 smithi038 ceph-mon[92422]: pgmap v3365: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 34 KiB/s wr, 8 op/s 2023-03-09T16:27:18.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5810.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:27:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:27:20 smithi122 ceph-mon[93869]: pgmap v3366: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 34 KiB/s wr, 8 op/s 2023-03-09T16:27:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:27:20 smithi122 ceph-mon[93869]: osdmap e423: 12 total, 12 up, 12 in 2023-03-09T16:27:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:27:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5351.009163 secs 2023-03-09T16:27:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:27:20 smithi181 ceph-mon[94189]: pgmap v3366: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 34 KiB/s wr, 8 op/s 2023-03-09T16:27:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:27:20 smithi181 ceph-mon[94189]: osdmap e423: 12 total, 12 up, 12 in 2023-03-09T16:27:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:27:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5351.009163 secs 2023-03-09T16:27:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:27:20 smithi038 ceph-mon[92422]: pgmap v3366: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 34 KiB/s wr, 8 op/s 2023-03-09T16:27:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:27:20 smithi038 ceph-mon[92422]: osdmap e423: 12 total, 12 up, 12 in 2023-03-09T16:27:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:27:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5351.009163 secs 2023-03-09T16:27:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:27:22 smithi122 ceph-mon[93869]: pgmap v3368: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 67 KiB/s wr, 8 op/s 2023-03-09T16:27:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:27:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5353.242434 secs 2023-03-09T16:27:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:27:22 smithi181 ceph-mon[94189]: pgmap v3368: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 67 KiB/s wr, 8 op/s 2023-03-09T16:27:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:27:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5353.242434 secs 2023-03-09T16:27:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:27:22 smithi038 ceph-mon[92422]: pgmap v3368: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 67 KiB/s wr, 8 op/s 2023-03-09T16:27:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:27:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5353.242434 secs 2023-03-09T16:27:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5815.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:27:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:27:24 smithi122 ceph-mon[93869]: pgmap v3369: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 67 KiB/s wr, 8 op/s 2023-03-09T16:27:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:27:24 smithi181 ceph-mon[94189]: pgmap v3369: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 67 KiB/s wr, 8 op/s 2023-03-09T16:27:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:27:24 smithi038 ceph-mon[92422]: pgmap v3369: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 67 KiB/s wr, 8 op/s 2023-03-09T16:27:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:27:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5356.009224 secs 2023-03-09T16:27:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:27:25 smithi122 ceph-mon[93869]: pgmap v3370: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 66 KiB/s wr, 8 op/s 2023-03-09T16:27:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:27:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5356.009224 secs 2023-03-09T16:27:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:27:25 smithi181 ceph-mon[94189]: pgmap v3370: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 66 KiB/s wr, 8 op/s 2023-03-09T16:27:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:27:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5356.009224 secs 2023-03-09T16:27:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:27:25 smithi038 ceph-mon[92422]: pgmap v3370: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 66 KiB/s wr, 8 op/s 2023-03-09T16:27:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:27:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5358.242551 secs 2023-03-09T16:27:27.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:27:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5358.242551 secs 2023-03-09T16:27:27.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:27:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5358.242551 secs 2023-03-09T16:27:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:27:27 smithi122 ceph-mon[93869]: pgmap v3371: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 74 KiB/s wr, 7 op/s 2023-03-09T16:27:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:27:27 smithi181 ceph-mon[94189]: pgmap v3371: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 74 KiB/s wr, 7 op/s 2023-03-09T16:27:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:27:27 smithi038 ceph-mon[92422]: pgmap v3371: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 74 KiB/s wr, 7 op/s 2023-03-09T16:27:28.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5820.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:27:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:27:30 smithi122 ceph-mon[93869]: pgmap v3372: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 73 KiB/s wr, 7 op/s 2023-03-09T16:27:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:27:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5361.009285 secs 2023-03-09T16:27:30.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:27:30 smithi181 ceph-mon[94189]: pgmap v3372: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 73 KiB/s wr, 7 op/s 2023-03-09T16:27:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:27:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5361.009285 secs 2023-03-09T16:27:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:27:30 smithi038 ceph-mon[92422]: pgmap v3372: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 73 KiB/s wr, 7 op/s 2023-03-09T16:27:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:27:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5361.009285 secs 2023-03-09T16:27:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:27:32 smithi122 ceph-mon[93869]: pgmap v3373: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 135 KiB/s wr, 9 op/s 2023-03-09T16:27:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:27:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5363.242665 secs 2023-03-09T16:27:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:27:32 smithi181 ceph-mon[94189]: pgmap v3373: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 135 KiB/s wr, 9 op/s 2023-03-09T16:27:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:27:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5363.242665 secs 2023-03-09T16:27:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:27:32 smithi038 ceph-mon[92422]: pgmap v3373: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 135 KiB/s wr, 9 op/s 2023-03-09T16:27:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:27:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5363.242665 secs 2023-03-09T16:27:33.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5825.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:27:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:27:34 smithi122 ceph-mon[93869]: pgmap v3374: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 97 KiB/s wr, 6 op/s 2023-03-09T16:27:34.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:27:34 smithi181 ceph-mon[94189]: pgmap v3374: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 97 KiB/s wr, 6 op/s 2023-03-09T16:27:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:27:34 smithi038 ceph-mon[92422]: pgmap v3374: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 97 KiB/s wr, 6 op/s 2023-03-09T16:27:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:27:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5366.009347 secs 2023-03-09T16:27:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:27:35 smithi122 ceph-mon[93869]: pgmap v3375: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 97 KiB/s wr, 6 op/s 2023-03-09T16:27:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:27:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5366.009347 secs 2023-03-09T16:27:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:27:35 smithi181 ceph-mon[94189]: pgmap v3375: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 97 KiB/s wr, 6 op/s 2023-03-09T16:27:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:27:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5366.009347 secs 2023-03-09T16:27:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:27:35 smithi038 ceph-mon[92422]: pgmap v3375: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 97 KiB/s wr, 6 op/s 2023-03-09T16:27:37.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:27:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5368.242809 secs 2023-03-09T16:27:37.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:27:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5368.242809 secs 2023-03-09T16:27:37.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:27:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5368.242809 secs 2023-03-09T16:27:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:27:37 smithi122 ceph-mon[93869]: pgmap v3376: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 138 KiB/s wr, 10 op/s 2023-03-09T16:27:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:27:37 smithi181 ceph-mon[94189]: pgmap v3376: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 138 KiB/s wr, 10 op/s 2023-03-09T16:27:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:27:37 smithi038 ceph-mon[92422]: pgmap v3376: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 138 KiB/s wr, 10 op/s 2023-03-09T16:27:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5830.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:27:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:27:40 smithi122 ceph-mon[93869]: pgmap v3377: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 109 KiB/s wr, 7 op/s 2023-03-09T16:27:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:27:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5371.009410 secs 2023-03-09T16:27:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:27:40 smithi181 ceph-mon[94189]: pgmap v3377: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 109 KiB/s wr, 7 op/s 2023-03-09T16:27:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:27:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5371.009410 secs 2023-03-09T16:27:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:27:40 smithi038 ceph-mon[92422]: pgmap v3377: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 109 KiB/s wr, 7 op/s 2023-03-09T16:27:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:27:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5371.009410 secs 2023-03-09T16:27:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:27:42 smithi122 ceph-mon[93869]: pgmap v3378: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 171 KiB/s wr, 12 op/s 2023-03-09T16:27:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:27:42 smithi181 ceph-mon[94189]: pgmap v3378: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 171 KiB/s wr, 12 op/s 2023-03-09T16:27:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:27:42 smithi038 ceph-mon[92422]: pgmap v3378: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 171 KiB/s wr, 12 op/s 2023-03-09T16:27:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:27:43 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:27:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:27:43 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:27:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:27:43 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:27:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:27:43 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:27:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:27:43 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:27:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:27:43 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:27:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5835.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:27:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:27:44 smithi122 ceph-mon[93869]: pgmap v3379: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 104 KiB/s wr, 10 op/s 2023-03-09T16:27:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:27:44 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:27:44.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:27:44 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:27:44.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:27:44 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:27:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:27:44 smithi181 ceph-mon[94189]: pgmap v3379: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 104 KiB/s wr, 10 op/s 2023-03-09T16:27:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:27:44 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:27:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:27:44 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:27:44.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:27:44 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:27:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:27:44 smithi038 ceph-mon[92422]: pgmap v3379: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 104 KiB/s wr, 10 op/s 2023-03-09T16:27:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:27:44 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:27:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:27:44 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:27:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:27:44 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:27:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:27:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5376.009472 secs 2023-03-09T16:27:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:27:45 smithi122 ceph-mon[93869]: pgmap v3380: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 109 KiB/s wr, 10 op/s 2023-03-09T16:27:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:27:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5376.009472 secs 2023-03-09T16:27:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:27:45 smithi181 ceph-mon[94189]: pgmap v3380: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 109 KiB/s wr, 10 op/s 2023-03-09T16:27:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:27:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5376.009472 secs 2023-03-09T16:27:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:27:45 smithi038 ceph-mon[92422]: pgmap v3380: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 109 KiB/s wr, 10 op/s 2023-03-09T16:27:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:27:48 smithi122 ceph-mon[93869]: pgmap v3381: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 254 KiB/s wr, 18 op/s 2023-03-09T16:27:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:27:48 smithi181 ceph-mon[94189]: pgmap v3381: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 254 KiB/s wr, 18 op/s 2023-03-09T16:27:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:27:48 smithi038 ceph-mon[92422]: pgmap v3381: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 254 KiB/s wr, 18 op/s 2023-03-09T16:27:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5840.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:27:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:27:50 smithi122 ceph-mon[93869]: pgmap v3382: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 217 KiB/s wr, 15 op/s 2023-03-09T16:27:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:27:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5381.009530 secs 2023-03-09T16:27:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:27:50 smithi181 ceph-mon[94189]: pgmap v3382: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 217 KiB/s wr, 15 op/s 2023-03-09T16:27:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:27:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5381.009530 secs 2023-03-09T16:27:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:27:50 smithi038 ceph-mon[92422]: pgmap v3382: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 217 KiB/s wr, 15 op/s 2023-03-09T16:27:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:27:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5381.009530 secs 2023-03-09T16:27:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:27:52 smithi122 ceph-mon[93869]: pgmap v3383: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 320 KiB/s wr, 26 op/s 2023-03-09T16:27:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:27:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5383.243185 secs 2023-03-09T16:27:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:27:52 smithi181 ceph-mon[94189]: pgmap v3383: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 320 KiB/s wr, 26 op/s 2023-03-09T16:27:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:27:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5383.243185 secs 2023-03-09T16:27:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:27:52 smithi038 ceph-mon[92422]: pgmap v3383: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 320 KiB/s wr, 26 op/s 2023-03-09T16:27:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:27:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5383.243185 secs 2023-03-09T16:27:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5845.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:27:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:27:54 smithi122 ceph-mon[93869]: pgmap v3384: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 258 KiB/s wr, 24 op/s 2023-03-09T16:27:54.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:27:54 smithi181 ceph-mon[94189]: pgmap v3384: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 258 KiB/s wr, 24 op/s 2023-03-09T16:27:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:27:54 smithi038 ceph-mon[92422]: pgmap v3384: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 258 KiB/s wr, 24 op/s 2023-03-09T16:27:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:27:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5386.009616 secs 2023-03-09T16:27:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:27:55 smithi122 ceph-mon[93869]: pgmap v3385: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 262 KiB/s wr, 23 op/s 2023-03-09T16:27:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:27:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5386.009616 secs 2023-03-09T16:27:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:27:55 smithi181 ceph-mon[94189]: pgmap v3385: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 262 KiB/s wr, 23 op/s 2023-03-09T16:27:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:27:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5386.009616 secs 2023-03-09T16:27:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:27:55 smithi038 ceph-mon[92422]: pgmap v3385: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 262 KiB/s wr, 23 op/s 2023-03-09T16:27:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:27:56 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:27:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:27:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:27:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:27:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:27:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:27:56 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:27:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:27:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:27:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:27:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:27:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:27:56 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:27:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:27:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:27:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:27:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:27:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:27:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5388.243309 secs 2023-03-09T16:27:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:27:57 smithi122 ceph-mon[93869]: pgmap v3386: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 403 KiB/s wr, 32 op/s 2023-03-09T16:27:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:27:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5388.243309 secs 2023-03-09T16:27:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:27:57 smithi181 ceph-mon[94189]: pgmap v3386: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 403 KiB/s wr, 32 op/s 2023-03-09T16:27:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:27:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5388.243309 secs 2023-03-09T16:27:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:27:57 smithi038 ceph-mon[92422]: pgmap v3386: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 403 KiB/s wr, 32 op/s 2023-03-09T16:27:58.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5850.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:28:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:28:00 smithi122 ceph-mon[93869]: pgmap v3387: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 264 KiB/s wr, 24 op/s 2023-03-09T16:28:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:28:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5391.009678 secs 2023-03-09T16:28:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:28:00 smithi181 ceph-mon[94189]: pgmap v3387: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 264 KiB/s wr, 24 op/s 2023-03-09T16:28:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:28:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5391.009678 secs 2023-03-09T16:28:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:28:00 smithi038 ceph-mon[92422]: pgmap v3387: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 264 KiB/s wr, 24 op/s 2023-03-09T16:28:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:28:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5391.009678 secs 2023-03-09T16:28:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:28:02 smithi122 ceph-mon[93869]: pgmap v3388: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 362 KiB/s wr, 33 op/s 2023-03-09T16:28:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:28:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5393.243430 secs 2023-03-09T16:28:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:28:02 smithi181 ceph-mon[94189]: pgmap v3388: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 362 KiB/s wr, 33 op/s 2023-03-09T16:28:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:28:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5393.243430 secs 2023-03-09T16:28:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:28:02 smithi038 ceph-mon[92422]: pgmap v3388: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 362 KiB/s wr, 33 op/s 2023-03-09T16:28:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:28:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5393.243430 secs 2023-03-09T16:28:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5855.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:28:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:28:04 smithi122 ceph-mon[93869]: pgmap v3389: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 281 KiB/s wr, 34 op/s 2023-03-09T16:28:04.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:28:04 smithi181 ceph-mon[94189]: pgmap v3389: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 281 KiB/s wr, 34 op/s 2023-03-09T16:28:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:28:04 smithi038 ceph-mon[92422]: pgmap v3389: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 281 KiB/s wr, 34 op/s 2023-03-09T16:28:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:28:05 smithi122 ceph-mon[93869]: osdmap e424: 12 total, 12 up, 12 in 2023-03-09T16:28:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:28:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5396.009739 secs 2023-03-09T16:28:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:28:05 smithi181 ceph-mon[94189]: osdmap e424: 12 total, 12 up, 12 in 2023-03-09T16:28:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:28:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5396.009739 secs 2023-03-09T16:28:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:28:05 smithi038 ceph-mon[92422]: osdmap e424: 12 total, 12 up, 12 in 2023-03-09T16:28:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:28:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5396.009739 secs 2023-03-09T16:28:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:28:06 smithi122 ceph-mon[93869]: pgmap v3391: 129 pgs: 2 active+clean+snaptrim, 17 active+clean+snaptrim_wait, 110 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 431 KiB/s wr, 43 op/s 2023-03-09T16:28:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:28:06 smithi181 ceph-mon[94189]: pgmap v3391: 129 pgs: 2 active+clean+snaptrim, 17 active+clean+snaptrim_wait, 110 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 431 KiB/s wr, 43 op/s 2023-03-09T16:28:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:28:06 smithi038 ceph-mon[92422]: pgmap v3391: 129 pgs: 2 active+clean+snaptrim, 17 active+clean+snaptrim_wait, 110 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 431 KiB/s wr, 43 op/s 2023-03-09T16:28:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:28:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5398.243556 secs 2023-03-09T16:28:07.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:28:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5398.243556 secs 2023-03-09T16:28:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:28:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5398.243556 secs 2023-03-09T16:28:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:28:08 smithi122 ceph-mon[93869]: pgmap v3392: 129 pgs: 2 active+clean+snaptrim, 17 active+clean+snaptrim_wait, 110 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 310 KiB/s wr, 43 op/s 2023-03-09T16:28:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:28:08 smithi181 ceph-mon[94189]: pgmap v3392: 129 pgs: 2 active+clean+snaptrim, 17 active+clean+snaptrim_wait, 110 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 310 KiB/s wr, 43 op/s 2023-03-09T16:28:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:28:08 smithi038 ceph-mon[92422]: pgmap v3392: 129 pgs: 2 active+clean+snaptrim, 17 active+clean+snaptrim_wait, 110 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 310 KiB/s wr, 43 op/s 2023-03-09T16:28:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5860.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:28:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:28:09 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:28:09.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:28:09 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:28:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:28:09 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:28:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:28:10 smithi122 ceph-mon[93869]: pgmap v3393: 129 pgs: 2 active+clean+snaptrim, 17 active+clean+snaptrim_wait, 110 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 317 KiB/s wr, 43 op/s 2023-03-09T16:28:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:28:10 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:28:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:28:10 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:28:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:28:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:28:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:28:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:28:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:28:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:28:10.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:28:10 smithi181 ceph-mon[94189]: pgmap v3393: 129 pgs: 2 active+clean+snaptrim, 17 active+clean+snaptrim_wait, 110 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 317 KiB/s wr, 43 op/s 2023-03-09T16:28:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:28:10 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:28:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:28:10 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:28:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:28:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:28:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:28:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:28:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:28:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:28:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:28:10 smithi038 ceph-mon[92422]: pgmap v3393: 129 pgs: 2 active+clean+snaptrim, 17 active+clean+snaptrim_wait, 110 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 317 KiB/s wr, 43 op/s 2023-03-09T16:28:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:28:10 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:28:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:28:10 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:28:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:28:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:28:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:28:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:28:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:28:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:28:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:28:11 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:28:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:28:11 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] up:active 2023-03-09T16:28:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:28:11 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:28:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:28:11 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:28:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:28:11 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] up:active 2023-03-09T16:28:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:28:11 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:28:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:28:11 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:28:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:28:11 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] up:active 2023-03-09T16:28:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:28:11 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:28:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:28:12 smithi122 ceph-mon[93869]: pgmap v3394: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 682 KiB/s rd, 916 KiB/s wr, 47 op/s 2023-03-09T16:28:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:28:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5403.243670 secs 2023-03-09T16:28:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:28:12 smithi181 ceph-mon[94189]: pgmap v3394: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 682 KiB/s rd, 916 KiB/s wr, 47 op/s 2023-03-09T16:28:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:28:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5403.243670 secs 2023-03-09T16:28:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:28:12 smithi038 ceph-mon[92422]: pgmap v3394: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 682 KiB/s rd, 916 KiB/s wr, 47 op/s 2023-03-09T16:28:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:28:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5403.243670 secs 2023-03-09T16:28:13.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5865.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:28:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:28:14 smithi122 ceph-mon[93869]: pgmap v3395: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 673 KiB/s rd, 896 KiB/s wr, 32 op/s 2023-03-09T16:28:14.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:28:14 smithi181 ceph-mon[94189]: pgmap v3395: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 673 KiB/s rd, 896 KiB/s wr, 32 op/s 2023-03-09T16:28:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:28:14 smithi038 ceph-mon[92422]: pgmap v3395: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 673 KiB/s rd, 896 KiB/s wr, 32 op/s 2023-03-09T16:28:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:28:15 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:28:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:28:15 smithi122 ceph-mon[93869]: osdmap e425: 12 total, 12 up, 12 in 2023-03-09T16:28:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:28:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5406.009911 secs 2023-03-09T16:28:15.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:28:15 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:28:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:28:15 smithi181 ceph-mon[94189]: osdmap e425: 12 total, 12 up, 12 in 2023-03-09T16:28:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:28:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5406.009911 secs 2023-03-09T16:28:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:28:15 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:28:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:28:15 smithi038 ceph-mon[92422]: osdmap e425: 12 total, 12 up, 12 in 2023-03-09T16:28:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:28:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5406.009911 secs 2023-03-09T16:28:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:28:16 smithi122 ceph-mon[93869]: pgmap v3397: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 728 KiB/s rd, 871 KiB/s wr, 29 op/s 2023-03-09T16:28:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:28:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:28:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:28:16 smithi181 ceph-mon[94189]: pgmap v3397: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 728 KiB/s rd, 871 KiB/s wr, 29 op/s 2023-03-09T16:28:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:28:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:28:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:28:16 smithi038 ceph-mon[92422]: pgmap v3397: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 728 KiB/s rd, 871 KiB/s wr, 29 op/s 2023-03-09T16:28:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:28:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:28:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:28:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5408.243734 secs 2023-03-09T16:28:17.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:28:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5408.243734 secs 2023-03-09T16:28:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:28:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5408.243734 secs 2023-03-09T16:28:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:28:18 smithi122 ceph-mon[93869]: pgmap v3398: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 34 op/s 2023-03-09T16:28:18.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:28:18 smithi181 ceph-mon[94189]: pgmap v3398: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 34 op/s 2023-03-09T16:28:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:28:18 smithi038 ceph-mon[92422]: pgmap v3398: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 34 op/s 2023-03-09T16:28:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5870.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:28:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:28:19 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:28:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:28:19 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] up:active 2023-03-09T16:28:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:28:19 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:28:19.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:28:19 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:28:19.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:28:19 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] up:active 2023-03-09T16:28:19.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:28:19 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:28:19.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:28:19 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:28:19.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:28:19 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] up:active 2023-03-09T16:28:19.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:28:19 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:28:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:28:20 smithi122 ceph-mon[93869]: pgmap v3399: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 34 op/s 2023-03-09T16:28:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:28:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5411.009968 secs 2023-03-09T16:28:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:28:20 smithi181 ceph-mon[94189]: pgmap v3399: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 34 op/s 2023-03-09T16:28:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:28:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5411.009968 secs 2023-03-09T16:28:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:28:20 smithi038 ceph-mon[92422]: pgmap v3399: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 34 op/s 2023-03-09T16:28:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:28:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5411.009968 secs 2023-03-09T16:28:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:28:22 smithi122 ceph-mon[93869]: pgmap v3400: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 882 KiB/s wr, 25 op/s 2023-03-09T16:28:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:28:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5413.243850 secs 2023-03-09T16:28:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:28:22 smithi181 ceph-mon[94189]: pgmap v3400: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 882 KiB/s wr, 25 op/s 2023-03-09T16:28:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:28:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5413.243850 secs 2023-03-09T16:28:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:28:22 smithi038 ceph-mon[92422]: pgmap v3400: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 882 KiB/s wr, 25 op/s 2023-03-09T16:28:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:28:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5413.243850 secs 2023-03-09T16:28:23.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5875.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:28:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:28:24 smithi122 ceph-mon[93869]: pgmap v3401: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 876 KiB/s wr, 24 op/s 2023-03-09T16:28:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:28:24 smithi181 ceph-mon[94189]: pgmap v3401: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 876 KiB/s wr, 24 op/s 2023-03-09T16:28:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:28:24 smithi038 ceph-mon[92422]: pgmap v3401: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 876 KiB/s wr, 24 op/s 2023-03-09T16:28:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:28:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5416.010028 secs 2023-03-09T16:28:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:28:25 smithi122 ceph-mon[93869]: pgmap v3402: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 843 KiB/s rd, 768 KiB/s wr, 22 op/s 2023-03-09T16:28:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:28:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5416.010028 secs 2023-03-09T16:28:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:28:25 smithi181 ceph-mon[94189]: pgmap v3402: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 843 KiB/s rd, 768 KiB/s wr, 22 op/s 2023-03-09T16:28:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:28:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5416.010028 secs 2023-03-09T16:28:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:28:25 smithi038 ceph-mon[92422]: pgmap v3402: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 843 KiB/s rd, 768 KiB/s wr, 22 op/s 2023-03-09T16:28:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:28:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5418.243946 secs 2023-03-09T16:28:27.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:28:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5418.243946 secs 2023-03-09T16:28:27.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:28:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5418.243946 secs 2023-03-09T16:28:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:28:27 smithi122 ceph-mon[93869]: pgmap v3403: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 753 KiB/s rd, 686 KiB/s wr, 24 op/s 2023-03-09T16:28:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:28:27 smithi181 ceph-mon[94189]: pgmap v3403: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 753 KiB/s rd, 686 KiB/s wr, 24 op/s 2023-03-09T16:28:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:28:27 smithi038 ceph-mon[92422]: pgmap v3403: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 753 KiB/s rd, 686 KiB/s wr, 24 op/s 2023-03-09T16:28:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5880.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:28:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:28:30 smithi122 ceph-mon[93869]: pgmap v3404: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 72 KiB/s wr, 10 op/s 2023-03-09T16:28:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:28:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5421.010088 secs 2023-03-09T16:28:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:28:30 smithi181 ceph-mon[94189]: pgmap v3404: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 72 KiB/s wr, 10 op/s 2023-03-09T16:28:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:28:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5421.010088 secs 2023-03-09T16:28:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:28:30 smithi038 ceph-mon[92422]: pgmap v3404: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 72 KiB/s wr, 10 op/s 2023-03-09T16:28:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:28:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5421.010088 secs 2023-03-09T16:28:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:28:32 smithi122 ceph-mon[93869]: pgmap v3405: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 103 KiB/s wr, 12 op/s 2023-03-09T16:28:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:28:32 smithi181 ceph-mon[94189]: pgmap v3405: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 103 KiB/s wr, 12 op/s 2023-03-09T16:28:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:28:32 smithi038 ceph-mon[92422]: pgmap v3405: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 103 KiB/s wr, 12 op/s 2023-03-09T16:28:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5885.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:28:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:28:34 smithi122 ceph-mon[93869]: pgmap v3406: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 54 KiB/s wr, 8 op/s 2023-03-09T16:28:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:28:34 smithi181 ceph-mon[94189]: pgmap v3406: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 54 KiB/s wr, 8 op/s 2023-03-09T16:28:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:28:34 smithi038 ceph-mon[92422]: pgmap v3406: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 54 KiB/s wr, 8 op/s 2023-03-09T16:28:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:28:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5426.010150 secs 2023-03-09T16:28:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:28:35 smithi122 ceph-mon[93869]: pgmap v3407: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 54 KiB/s wr, 7 op/s 2023-03-09T16:28:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:28:35 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:28:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:28:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5426.010150 secs 2023-03-09T16:28:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:28:35 smithi181 ceph-mon[94189]: pgmap v3407: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 54 KiB/s wr, 7 op/s 2023-03-09T16:28:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:28:35 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:28:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:28:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5426.010150 secs 2023-03-09T16:28:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:28:35 smithi038 ceph-mon[92422]: pgmap v3407: 129 pgs: 129 active+clean; 8.7 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 54 KiB/s wr, 7 op/s 2023-03-09T16:28:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:28:35 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:28:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:28:36 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:28:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:28:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:28:36.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:28:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:28:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:28:36 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:28:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:28:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:28:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:28:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:28:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:28:36 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:28:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:28:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:28:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:28:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:28:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:28:37 smithi122 ceph-mon[93869]: pgmap v3408: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 95 KiB/s wr, 11 op/s 2023-03-09T16:28:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:28:37 smithi181 ceph-mon[94189]: pgmap v3408: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 95 KiB/s wr, 11 op/s 2023-03-09T16:28:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:28:37 smithi038 ceph-mon[92422]: pgmap v3408: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 95 KiB/s wr, 11 op/s 2023-03-09T16:28:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5890.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:28:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:28:40 smithi122 ceph-mon[93869]: pgmap v3409: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 78 KiB/s wr, 7 op/s 2023-03-09T16:28:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:28:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5431.010211 secs 2023-03-09T16:28:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:28:40 smithi181 ceph-mon[94189]: pgmap v3409: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 78 KiB/s wr, 7 op/s 2023-03-09T16:28:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:28:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5431.010211 secs 2023-03-09T16:28:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:28:40 smithi038 ceph-mon[92422]: pgmap v3409: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 78 KiB/s wr, 7 op/s 2023-03-09T16:28:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:28:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5431.010211 secs 2023-03-09T16:28:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:28:42 smithi122 ceph-mon[93869]: pgmap v3410: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 124 KiB/s wr, 12 op/s 2023-03-09T16:28:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:28:42 smithi181 ceph-mon[94189]: pgmap v3410: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 124 KiB/s wr, 12 op/s 2023-03-09T16:28:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:28:42 smithi038 ceph-mon[92422]: pgmap v3410: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 124 KiB/s wr, 12 op/s 2023-03-09T16:28:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5895.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:28:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:28:44 smithi122 ceph-mon[93869]: pgmap v3411: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 88 KiB/s wr, 9 op/s 2023-03-09T16:28:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:28:44 smithi181 ceph-mon[94189]: pgmap v3411: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 88 KiB/s wr, 9 op/s 2023-03-09T16:28:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:28:44 smithi038 ceph-mon[92422]: pgmap v3411: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 88 KiB/s wr, 9 op/s 2023-03-09T16:28:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:28:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5436.010273 secs 2023-03-09T16:28:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:28:45 smithi122 ceph-mon[93869]: pgmap v3412: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 128 KiB/s wr, 9 op/s 2023-03-09T16:28:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:28:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5436.010273 secs 2023-03-09T16:28:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:28:45 smithi181 ceph-mon[94189]: pgmap v3412: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 128 KiB/s wr, 9 op/s 2023-03-09T16:28:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:28:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5436.010273 secs 2023-03-09T16:28:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:28:45 smithi038 ceph-mon[92422]: pgmap v3412: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 128 KiB/s wr, 9 op/s 2023-03-09T16:28:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:28:48 smithi122 ceph-mon[93869]: pgmap v3413: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 193 KiB/s wr, 13 op/s 2023-03-09T16:28:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:28:48 smithi181 ceph-mon[94189]: pgmap v3413: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 193 KiB/s wr, 13 op/s 2023-03-09T16:28:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:28:48 smithi038 ceph-mon[92422]: pgmap v3413: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 193 KiB/s wr, 13 op/s 2023-03-09T16:28:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5900.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:28:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:28:50 smithi122 ceph-mon[93869]: pgmap v3414: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 151 KiB/s wr, 9 op/s 2023-03-09T16:28:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:28:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5441.010355 secs 2023-03-09T16:28:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:28:50 smithi181 ceph-mon[94189]: pgmap v3414: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 151 KiB/s wr, 9 op/s 2023-03-09T16:28:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:28:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5441.010355 secs 2023-03-09T16:28:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:28:50 smithi038 ceph-mon[92422]: pgmap v3414: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 151 KiB/s wr, 9 op/s 2023-03-09T16:28:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:28:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5441.010355 secs 2023-03-09T16:28:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:28:52 smithi122 ceph-mon[93869]: pgmap v3415: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 166 KiB/s wr, 13 op/s 2023-03-09T16:28:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:28:52 smithi181 ceph-mon[94189]: pgmap v3415: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 166 KiB/s wr, 13 op/s 2023-03-09T16:28:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:28:52 smithi038 ceph-mon[92422]: pgmap v3415: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 166 KiB/s wr, 13 op/s 2023-03-09T16:28:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5905.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:28:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:28:54 smithi122 ceph-mon[93869]: pgmap v3416: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 120 KiB/s wr, 9 op/s 2023-03-09T16:28:54.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:28:54 smithi181 ceph-mon[94189]: pgmap v3416: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 120 KiB/s wr, 9 op/s 2023-03-09T16:28:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:28:54 smithi038 ceph-mon[92422]: pgmap v3416: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 120 KiB/s wr, 9 op/s 2023-03-09T16:28:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:28:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5446.010437 secs 2023-03-09T16:28:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:28:55 smithi122 ceph-mon[93869]: pgmap v3417: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 246 KiB/s wr, 10 op/s 2023-03-09T16:28:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:28:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5446.010437 secs 2023-03-09T16:28:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:28:55 smithi181 ceph-mon[94189]: pgmap v3417: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 246 KiB/s wr, 10 op/s 2023-03-09T16:28:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:28:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5446.010437 secs 2023-03-09T16:28:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:28:55 smithi038 ceph-mon[92422]: pgmap v3417: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 246 KiB/s wr, 10 op/s 2023-03-09T16:28:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:28:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5448.244603 secs 2023-03-09T16:28:57.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:28:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5448.244603 secs 2023-03-09T16:28:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:28:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5448.244603 secs 2023-03-09T16:28:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:28:57 smithi122 ceph-mon[93869]: pgmap v3418: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 215 KiB/s wr, 12 op/s 2023-03-09T16:28:58.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:28:57 smithi181 ceph-mon[94189]: pgmap v3418: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 215 KiB/s wr, 12 op/s 2023-03-09T16:28:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:28:57 smithi038 ceph-mon[92422]: pgmap v3418: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 215 KiB/s wr, 12 op/s 2023-03-09T16:28:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5910.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:29:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:00 smithi122 ceph-mon[93869]: pgmap v3419: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 151 KiB/s wr, 9 op/s 2023-03-09T16:29:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:00 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:29:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:29:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:29:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5451.010496 secs 2023-03-09T16:29:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:00 smithi181 ceph-mon[94189]: pgmap v3419: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 151 KiB/s wr, 9 op/s 2023-03-09T16:29:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:00 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:29:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:29:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:29:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5451.010496 secs 2023-03-09T16:29:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:00 smithi038 ceph-mon[92422]: pgmap v3419: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 151 KiB/s wr, 9 op/s 2023-03-09T16:29:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:00 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:29:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:29:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:29:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5451.010496 secs 2023-03-09T16:29:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:02 smithi122 ceph-mon[93869]: pgmap v3420: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 209 KiB/s wr, 12 op/s 2023-03-09T16:29:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5453.244733 secs 2023-03-09T16:29:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:02 smithi181 ceph-mon[94189]: pgmap v3420: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 209 KiB/s wr, 12 op/s 2023-03-09T16:29:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5453.244733 secs 2023-03-09T16:29:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:02 smithi038 ceph-mon[92422]: pgmap v3420: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 209 KiB/s wr, 12 op/s 2023-03-09T16:29:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5453.244733 secs 2023-03-09T16:29:03.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5915.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:29:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:04 smithi122 ceph-mon[93869]: pgmap v3421: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 216 KiB/s wr, 21 op/s 2023-03-09T16:29:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:04 smithi181 ceph-mon[94189]: pgmap v3421: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 216 KiB/s wr, 21 op/s 2023-03-09T16:29:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:04 smithi038 ceph-mon[92422]: pgmap v3421: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 216 KiB/s wr, 21 op/s 2023-03-09T16:29:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:05 smithi122 ceph-mon[93869]: osdmap e426: 12 total, 12 up, 12 in 2023-03-09T16:29:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5456.010557 secs 2023-03-09T16:29:05.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:05 smithi181 ceph-mon[94189]: osdmap e426: 12 total, 12 up, 12 in 2023-03-09T16:29:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5456.010557 secs 2023-03-09T16:29:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:05 smithi038 ceph-mon[92422]: osdmap e426: 12 total, 12 up, 12 in 2023-03-09T16:29:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5456.010557 secs 2023-03-09T16:29:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:06 smithi122 ceph-mon[93869]: pgmap v3423: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 215 KiB/s wr, 25 op/s 2023-03-09T16:29:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:06 smithi181 ceph-mon[94189]: pgmap v3423: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 215 KiB/s wr, 25 op/s 2023-03-09T16:29:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:06 smithi038 ceph-mon[92422]: pgmap v3423: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 215 KiB/s wr, 25 op/s 2023-03-09T16:29:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5458.244851 secs 2023-03-09T16:29:07.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5458.244851 secs 2023-03-09T16:29:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5458.244851 secs 2023-03-09T16:29:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:08 smithi122 ceph-mon[93869]: pgmap v3424: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 237 KiB/s wr, 27 op/s 2023-03-09T16:29:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:08 smithi181 ceph-mon[94189]: pgmap v3424: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 237 KiB/s wr, 27 op/s 2023-03-09T16:29:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:08 smithi038 ceph-mon[92422]: pgmap v3424: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 237 KiB/s wr, 27 op/s 2023-03-09T16:29:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5920.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:29:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:10 smithi122 ceph-mon[93869]: pgmap v3425: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 237 KiB/s wr, 27 op/s 2023-03-09T16:29:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5461.010615 secs 2023-03-09T16:29:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:29:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:10 smithi181 ceph-mon[94189]: pgmap v3425: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 237 KiB/s wr, 27 op/s 2023-03-09T16:29:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5461.010615 secs 2023-03-09T16:29:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:29:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:10 smithi038 ceph-mon[92422]: pgmap v3425: 129 pgs: 2 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 121 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 237 KiB/s wr, 27 op/s 2023-03-09T16:29:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5461.010615 secs 2023-03-09T16:29:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:29:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:12 smithi122 ceph-mon[93869]: pgmap v3426: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 193 KiB/s wr, 27 op/s 2023-03-09T16:29:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5463.244971 secs 2023-03-09T16:29:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:12 smithi181 ceph-mon[94189]: pgmap v3426: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 193 KiB/s wr, 27 op/s 2023-03-09T16:29:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5463.244971 secs 2023-03-09T16:29:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:12 smithi038 ceph-mon[92422]: pgmap v3426: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 193 KiB/s wr, 27 op/s 2023-03-09T16:29:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5463.244971 secs 2023-03-09T16:29:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5925.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:29:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:13 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:29:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:13 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:29:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:13 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:29:13.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:13 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:29:13.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:13 smithi181 ceph-mon[94189]: pgmap v3427: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 176 KiB/s wr, 12 op/s 2023-03-09T16:29:13.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:13 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:29:13.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:13 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:29:13.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:13 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:29:13.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:13 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:29:13.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:13 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:29:13.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:13 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:29:13.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:13 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:29:13.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:13 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:29:13.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:13 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi038 to 197.2M 2023-03-09T16:29:13.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:13 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T16:29:13.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:13 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi038 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-03-09T16:29:13.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:13 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T16:29:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:13 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:29:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:13 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:29:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:13 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:29:14.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:13 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:29:14.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:13 smithi038 ceph-mon[92422]: pgmap v3427: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 176 KiB/s wr, 12 op/s 2023-03-09T16:29:14.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:13 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:29:14.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:13 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:29:14.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:13 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:29:14.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:13 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:29:14.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:13 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:29:14.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:13 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:29:14.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:13 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:29:14.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:13 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:29:14.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:13 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi038 to 197.2M 2023-03-09T16:29:14.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:13 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T16:29:14.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:13 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi038 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-03-09T16:29:14.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:13 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T16:29:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:13 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:29:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:13 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:29:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:13 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:29:14.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:13 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:29:14.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:13 smithi122 ceph-mon[93869]: pgmap v3427: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 176 KiB/s wr, 12 op/s 2023-03-09T16:29:14.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:13 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:29:14.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:13 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:29:14.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:13 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:29:14.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:13 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:29:14.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:13 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:29:14.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:13 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:29:14.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:13 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:29:14.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:13 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:29:14.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:13 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi038 to 197.2M 2023-03-09T16:29:14.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:13 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T16:29:14.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:13 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi038 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-03-09T16:29:14.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:13 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T16:29:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:29:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:29:15.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:15 smithi122 ceph-mon[93869]: osdmap e427: 12 total, 12 up, 12 in 2023-03-09T16:29:15.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5466.010678 secs 2023-03-09T16:29:15.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:29:15.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:29:15.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:29:15.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:29:15.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:29:15.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:29:15.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:29:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:29:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:29:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:15 smithi181 ceph-mon[94189]: osdmap e427: 12 total, 12 up, 12 in 2023-03-09T16:29:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5466.010678 secs 2023-03-09T16:29:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:29:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:29:15.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:29:15.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:29:15.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:29:15.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:29:15.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:29:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:29:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:29:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:15 smithi038 ceph-mon[92422]: osdmap e427: 12 total, 12 up, 12 in 2023-03-09T16:29:15.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5466.010678 secs 2023-03-09T16:29:15.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:29:15.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:29:15.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:29:15.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:29:15.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:29:15.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:29:15.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:29:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:16 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T16:29:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:16 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T16:29:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:16 smithi122 ceph-mon[93869]: pgmap v3429: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 68 KiB/s wr, 11 op/s 2023-03-09T16:29:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:16 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T16:29:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:16 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T16:29:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:16 smithi181 ceph-mon[94189]: pgmap v3429: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 68 KiB/s wr, 11 op/s 2023-03-09T16:29:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:16 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T16:29:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:16 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T16:29:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:16 smithi038 ceph-mon[92422]: pgmap v3429: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 68 KiB/s wr, 11 op/s 2023-03-09T16:29:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5468.245091 secs 2023-03-09T16:29:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5468.245091 secs 2023-03-09T16:29:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5468.245091 secs 2023-03-09T16:29:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:18 smithi122 ceph-mon[93869]: pgmap v3430: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 34 KiB/s wr, 8 op/s 2023-03-09T16:29:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:18 smithi181 ceph-mon[94189]: pgmap v3430: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 34 KiB/s wr, 8 op/s 2023-03-09T16:29:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:18 smithi038 ceph-mon[92422]: pgmap v3430: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 34 KiB/s wr, 8 op/s 2023-03-09T16:29:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5930.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:29:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:20 smithi122 ceph-mon[93869]: pgmap v3431: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 34 KiB/s wr, 9 op/s 2023-03-09T16:29:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5471.010739 secs 2023-03-09T16:29:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:20 smithi181 ceph-mon[94189]: pgmap v3431: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 34 KiB/s wr, 9 op/s 2023-03-09T16:29:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5471.010739 secs 2023-03-09T16:29:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:20 smithi038 ceph-mon[92422]: pgmap v3431: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 34 KiB/s wr, 9 op/s 2023-03-09T16:29:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5471.010739 secs 2023-03-09T16:29:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:22 smithi122 ceph-mon[93869]: pgmap v3432: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 10 KiB/s wr, 8 op/s 2023-03-09T16:29:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5473.245173 secs 2023-03-09T16:29:22.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:22 smithi181 ceph-mon[94189]: pgmap v3432: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 10 KiB/s wr, 8 op/s 2023-03-09T16:29:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5473.245173 secs 2023-03-09T16:29:22.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:22 smithi038 ceph-mon[92422]: pgmap v3432: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 10 KiB/s wr, 8 op/s 2023-03-09T16:29:22.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5473.245173 secs 2023-03-09T16:29:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5935.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:29:24.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:24 smithi122 ceph-mon[93869]: pgmap v3433: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 65 KiB/s wr, 9 op/s 2023-03-09T16:29:24.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:24 smithi181 ceph-mon[94189]: pgmap v3433: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 65 KiB/s wr, 9 op/s 2023-03-09T16:29:24.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:24 smithi038 ceph-mon[92422]: pgmap v3433: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 65 KiB/s wr, 9 op/s 2023-03-09T16:29:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5476.010799 secs 2023-03-09T16:29:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:25 smithi122 ceph-mon[93869]: pgmap v3434: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 62 KiB/s wr, 8 op/s 2023-03-09T16:29:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5476.010799 secs 2023-03-09T16:29:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:25 smithi181 ceph-mon[94189]: pgmap v3434: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 62 KiB/s wr, 8 op/s 2023-03-09T16:29:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5476.010799 secs 2023-03-09T16:29:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:25 smithi038 ceph-mon[92422]: pgmap v3434: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 62 KiB/s wr, 8 op/s 2023-03-09T16:29:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5478.245279 secs 2023-03-09T16:29:27.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5478.245279 secs 2023-03-09T16:29:27.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5478.245279 secs 2023-03-09T16:29:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:27 smithi122 ceph-mon[93869]: pgmap v3435: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 55 KiB/s wr, 10 op/s 2023-03-09T16:29:28.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:27 smithi181 ceph-mon[94189]: pgmap v3435: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 55 KiB/s wr, 10 op/s 2023-03-09T16:29:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:27 smithi038 ceph-mon[92422]: pgmap v3435: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 55 KiB/s wr, 10 op/s 2023-03-09T16:29:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5940.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:29:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:30 smithi122 ceph-mon[93869]: pgmap v3436: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 54 KiB/s wr, 7 op/s 2023-03-09T16:29:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5481.010860 secs 2023-03-09T16:29:30.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:30 smithi181 ceph-mon[94189]: pgmap v3436: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 54 KiB/s wr, 7 op/s 2023-03-09T16:29:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5481.010860 secs 2023-03-09T16:29:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:30 smithi038 ceph-mon[92422]: pgmap v3436: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 54 KiB/s wr, 7 op/s 2023-03-09T16:29:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5481.010860 secs 2023-03-09T16:29:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:32 smithi122 ceph-mon[93869]: pgmap v3437: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 59 KiB/s wr, 9 op/s 2023-03-09T16:29:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:32 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 5483.245416 secs 2023-03-09T16:29:32.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:32 smithi181 ceph-mon[94189]: pgmap v3437: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 59 KiB/s wr, 9 op/s 2023-03-09T16:29:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:32 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 5483.245416 secs 2023-03-09T16:29:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:32 smithi038 ceph-mon[92422]: pgmap v3437: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 59 KiB/s wr, 9 op/s 2023-03-09T16:29:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:32 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 5483.245416 secs 2023-03-09T16:29:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5945.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:29:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:34 smithi122 ceph-mon[93869]: pgmap v3438: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 162 KiB/s wr, 8 op/s 2023-03-09T16:29:34.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:34 smithi181 ceph-mon[94189]: pgmap v3438: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 162 KiB/s wr, 8 op/s 2023-03-09T16:29:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:34 smithi038 ceph-mon[92422]: pgmap v3438: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 162 KiB/s wr, 8 op/s 2023-03-09T16:29:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5486.010922 secs 2023-03-09T16:29:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:35 smithi122 ceph-mon[93869]: pgmap v3439: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 108 KiB/s wr, 7 op/s 2023-03-09T16:29:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5486.010922 secs 2023-03-09T16:29:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:35 smithi181 ceph-mon[94189]: pgmap v3439: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 108 KiB/s wr, 7 op/s 2023-03-09T16:29:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5486.010922 secs 2023-03-09T16:29:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:35 smithi038 ceph-mon[92422]: pgmap v3439: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 108 KiB/s wr, 7 op/s 2023-03-09T16:29:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:29:36.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:29:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:29:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:29:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:29:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:29:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5488.245481 secs 2023-03-09T16:29:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:37 smithi122 ceph-mon[93869]: pgmap v3440: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 196 KiB/s wr, 37 op/s 2023-03-09T16:29:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5488.245481 secs 2023-03-09T16:29:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:37 smithi181 ceph-mon[94189]: pgmap v3440: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 196 KiB/s wr, 37 op/s 2023-03-09T16:29:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5488.245481 secs 2023-03-09T16:29:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:37 smithi038 ceph-mon[92422]: pgmap v3440: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 196 KiB/s wr, 37 op/s 2023-03-09T16:29:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5950.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:29:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:40 smithi122 ceph-mon[93869]: pgmap v3441: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 196 KiB/s wr, 34 op/s 2023-03-09T16:29:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:29:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:29:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5491.010985 secs 2023-03-09T16:29:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:40 smithi181 ceph-mon[94189]: pgmap v3441: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 196 KiB/s wr, 34 op/s 2023-03-09T16:29:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:29:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:29:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5491.010985 secs 2023-03-09T16:29:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:40 smithi038 ceph-mon[92422]: pgmap v3441: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 196 KiB/s wr, 34 op/s 2023-03-09T16:29:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:29:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:29:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5491.010985 secs 2023-03-09T16:29:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:42 smithi122 ceph-mon[93869]: pgmap v3442: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 201 KiB/s wr, 37 op/s 2023-03-09T16:29:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:42 smithi181 ceph-mon[94189]: pgmap v3442: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 201 KiB/s wr, 37 op/s 2023-03-09T16:29:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:42 smithi038 ceph-mon[92422]: pgmap v3442: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 201 KiB/s wr, 37 op/s 2023-03-09T16:29:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:43 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:29:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:43 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:29:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:43 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:29:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5955.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:29:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:44 smithi122 ceph-mon[93869]: pgmap v3443: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 251 KiB/s wr, 36 op/s 2023-03-09T16:29:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:44 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:29:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:44 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:29:44.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:44 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:29:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:44 smithi181 ceph-mon[94189]: pgmap v3443: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 251 KiB/s wr, 36 op/s 2023-03-09T16:29:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:44 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:29:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:44 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:29:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:44 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:29:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:44 smithi038 ceph-mon[92422]: pgmap v3443: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 251 KiB/s wr, 36 op/s 2023-03-09T16:29:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:44 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:29:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:44 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:29:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:44 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:29:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5496.011045 secs 2023-03-09T16:29:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:45 smithi122 ceph-mon[93869]: pgmap v3444: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 214 KiB/s wr, 35 op/s 2023-03-09T16:29:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5496.011045 secs 2023-03-09T16:29:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:45 smithi181 ceph-mon[94189]: pgmap v3444: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 214 KiB/s wr, 35 op/s 2023-03-09T16:29:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5496.011045 secs 2023-03-09T16:29:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:45 smithi038 ceph-mon[92422]: pgmap v3444: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 214 KiB/s wr, 35 op/s 2023-03-09T16:29:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5498.245744 secs 2023-03-09T16:29:47.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5498.245744 secs 2023-03-09T16:29:47.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5498.245744 secs 2023-03-09T16:29:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:47 smithi122 ceph-mon[93869]: pgmap v3445: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 266 KiB/s wr, 39 op/s 2023-03-09T16:29:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:47 smithi181 ceph-mon[94189]: pgmap v3445: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 266 KiB/s wr, 39 op/s 2023-03-09T16:29:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:47 smithi038 ceph-mon[92422]: pgmap v3445: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 266 KiB/s wr, 39 op/s 2023-03-09T16:29:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5960.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:29:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:29:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:29:49.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:29:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:29:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:29:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:29:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:49 smithi038 ceph-mon[92422]: pgmap v3446: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 179 KiB/s wr, 9 op/s 2023-03-09T16:29:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:49 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:29:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:49 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5501.011107 secs 2023-03-09T16:29:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:49 smithi122 ceph-mon[93869]: pgmap v3446: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 179 KiB/s wr, 9 op/s 2023-03-09T16:29:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:49 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:29:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:49 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5501.011107 secs 2023-03-09T16:29:50.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:49 smithi181 ceph-mon[94189]: pgmap v3446: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 179 KiB/s wr, 9 op/s 2023-03-09T16:29:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:49 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:29:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:49 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5501.011107 secs 2023-03-09T16:29:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:52 smithi122 ceph-mon[93869]: pgmap v3447: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 192 KiB/s wr, 12 op/s 2023-03-09T16:29:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5503.245867 secs 2023-03-09T16:29:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:52 smithi181 ceph-mon[94189]: pgmap v3447: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 192 KiB/s wr, 12 op/s 2023-03-09T16:29:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5503.245867 secs 2023-03-09T16:29:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:52 smithi038 ceph-mon[92422]: pgmap v3447: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 192 KiB/s wr, 12 op/s 2023-03-09T16:29:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5503.245867 secs 2023-03-09T16:29:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5965.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:29:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:54 smithi122 ceph-mon[93869]: pgmap v3448: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 187 KiB/s wr, 10 op/s 2023-03-09T16:29:54.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:54 smithi181 ceph-mon[94189]: pgmap v3448: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 187 KiB/s wr, 10 op/s 2023-03-09T16:29:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:54 smithi038 ceph-mon[92422]: pgmap v3448: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 187 KiB/s wr, 10 op/s 2023-03-09T16:29:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5506.011170 secs 2023-03-09T16:29:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:55 smithi122 ceph-mon[93869]: pgmap v3449: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 218 KiB/s wr, 10 op/s 2023-03-09T16:29:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5506.011170 secs 2023-03-09T16:29:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:55 smithi181 ceph-mon[94189]: pgmap v3449: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 218 KiB/s wr, 10 op/s 2023-03-09T16:29:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5506.011170 secs 2023-03-09T16:29:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:55 smithi038 ceph-mon[92422]: pgmap v3449: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 218 KiB/s wr, 10 op/s 2023-03-09T16:29:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5508.245987 secs 2023-03-09T16:29:57.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5508.245987 secs 2023-03-09T16:29:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5508.245987 secs 2023-03-09T16:29:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:29:57 smithi122 ceph-mon[93869]: pgmap v3450: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 152 KiB/s wr, 11 op/s 2023-03-09T16:29:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:29:57 smithi181 ceph-mon[94189]: pgmap v3450: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 152 KiB/s wr, 11 op/s 2023-03-09T16:29:58.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:29:57 smithi038 ceph-mon[92422]: pgmap v3450: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 152 KiB/s wr, 11 op/s 2023-03-09T16:29:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5970.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:30:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:30:00 smithi122 ceph-mon[93869]: pgmap v3451: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 99 KiB/s wr, 8 op/s 2023-03-09T16:30:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:30:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5511.011233 secs 2023-03-09T16:30:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:30:00 smithi122 ceph-mon[93869]: overall HEALTH_WARN 2 MDSs report slow requests 2023-03-09T16:30:00.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:30:00 smithi181 ceph-mon[94189]: pgmap v3451: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 99 KiB/s wr, 8 op/s 2023-03-09T16:30:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:30:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5511.011233 secs 2023-03-09T16:30:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:30:00 smithi181 ceph-mon[94189]: overall HEALTH_WARN 2 MDSs report slow requests 2023-03-09T16:30:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:30:00 smithi038 ceph-mon[92422]: pgmap v3451: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 99 KiB/s wr, 8 op/s 2023-03-09T16:30:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:30:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5511.011233 secs 2023-03-09T16:30:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:30:00 smithi038 ceph-mon[92422]: overall HEALTH_WARN 2 MDSs report slow requests 2023-03-09T16:30:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:30:02 smithi122 ceph-mon[93869]: pgmap v3452: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 100 KiB/s wr, 10 op/s 2023-03-09T16:30:02.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:30:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5513.246082 secs 2023-03-09T16:30:02.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:30:02 smithi181 ceph-mon[94189]: pgmap v3452: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 100 KiB/s wr, 10 op/s 2023-03-09T16:30:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:30:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5513.246082 secs 2023-03-09T16:30:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:30:02 smithi038 ceph-mon[92422]: pgmap v3452: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 100 KiB/s wr, 10 op/s 2023-03-09T16:30:02.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:30:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5513.246082 secs 2023-03-09T16:30:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5975.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:30:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:30:04 smithi122 ceph-mon[93869]: pgmap v3453: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 108 KiB/s wr, 19 op/s 2023-03-09T16:30:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:30:04 smithi181 ceph-mon[94189]: pgmap v3453: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 108 KiB/s wr, 19 op/s 2023-03-09T16:30:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:30:04 smithi038 ceph-mon[92422]: pgmap v3453: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 108 KiB/s wr, 19 op/s 2023-03-09T16:30:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:30:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5516.011294 secs 2023-03-09T16:30:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:30:05 smithi122 ceph-mon[93869]: pgmap v3454: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 162 KiB/s wr, 19 op/s 2023-03-09T16:30:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:30:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5516.011294 secs 2023-03-09T16:30:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:30:05 smithi181 ceph-mon[94189]: pgmap v3454: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 162 KiB/s wr, 19 op/s 2023-03-09T16:30:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:30:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5516.011294 secs 2023-03-09T16:30:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:30:05 smithi038 ceph-mon[92422]: pgmap v3454: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 162 KiB/s wr, 19 op/s 2023-03-09T16:30:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:30:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5518.246188 secs 2023-03-09T16:30:07.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:30:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5518.246188 secs 2023-03-09T16:30:07.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:30:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5518.246188 secs 2023-03-09T16:30:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:30:07 smithi122 ceph-mon[93869]: pgmap v3455: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 104 KiB/s wr, 23 op/s 2023-03-09T16:30:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:30:07 smithi181 ceph-mon[94189]: pgmap v3455: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 104 KiB/s wr, 23 op/s 2023-03-09T16:30:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:30:07 smithi038 ceph-mon[92422]: pgmap v3455: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 104 KiB/s wr, 23 op/s 2023-03-09T16:30:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5980.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:30:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:30:09 smithi122 ceph-mon[93869]: osdmap e428: 12 total, 12 up, 12 in 2023-03-09T16:30:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:30:09 smithi122 ceph-mon[93869]: pgmap v3457: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 124 KiB/s wr, 25 op/s 2023-03-09T16:30:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:30:09 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5521.011356 secs 2023-03-09T16:30:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:30:09 smithi181 ceph-mon[94189]: osdmap e428: 12 total, 12 up, 12 in 2023-03-09T16:30:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:30:09 smithi181 ceph-mon[94189]: pgmap v3457: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 124 KiB/s wr, 25 op/s 2023-03-09T16:30:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:30:09 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5521.011356 secs 2023-03-09T16:30:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:30:09 smithi038 ceph-mon[92422]: osdmap e428: 12 total, 12 up, 12 in 2023-03-09T16:30:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:30:09 smithi038 ceph-mon[92422]: pgmap v3457: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 124 KiB/s wr, 25 op/s 2023-03-09T16:30:10.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:30:09 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5521.011356 secs 2023-03-09T16:30:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:30:12 smithi122 ceph-mon[93869]: pgmap v3458: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 124 KiB/s wr, 25 op/s 2023-03-09T16:30:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:30:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5523.246311 secs 2023-03-09T16:30:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:30:12 smithi181 ceph-mon[94189]: pgmap v3458: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 124 KiB/s wr, 25 op/s 2023-03-09T16:30:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:30:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5523.246311 secs 2023-03-09T16:30:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:30:12 smithi038 ceph-mon[92422]: pgmap v3458: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 124 KiB/s wr, 25 op/s 2023-03-09T16:30:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:30:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5523.246311 secs 2023-03-09T16:30:13.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5985.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:30:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:30:14 smithi122 ceph-mon[93869]: pgmap v3459: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 98 KiB/s wr, 11 op/s 2023-03-09T16:30:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:30:14 smithi181 ceph-mon[94189]: pgmap v3459: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 98 KiB/s wr, 11 op/s 2023-03-09T16:30:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:30:14 smithi038 ceph-mon[92422]: pgmap v3459: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 98 KiB/s wr, 11 op/s 2023-03-09T16:30:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:30:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5526.011415 secs 2023-03-09T16:30:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:30:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:30:15.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:30:15 smithi122 ceph-mon[93869]: pgmap v3460: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 77 KiB/s wr, 11 op/s 2023-03-09T16:30:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:30:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5526.011415 secs 2023-03-09T16:30:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:30:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:30:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:30:15 smithi181 ceph-mon[94189]: pgmap v3460: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 77 KiB/s wr, 11 op/s 2023-03-09T16:30:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:30:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5526.011415 secs 2023-03-09T16:30:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:30:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:30:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:30:15 smithi038 ceph-mon[92422]: pgmap v3460: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 77 KiB/s wr, 11 op/s 2023-03-09T16:30:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:30:16 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:30:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:30:16 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:30:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:30:16 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:30:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:30:16 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:30:16.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:30:16 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:30:16.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:30:16 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:30:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:30:16 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:30:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:30:16 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:30:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:30:16 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:30:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:30:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5528.246434 secs 2023-03-09T16:30:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:30:17 smithi122 ceph-mon[93869]: pgmap v3461: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 44 KiB/s wr, 8 op/s 2023-03-09T16:30:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:30:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5528.246434 secs 2023-03-09T16:30:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:30:17 smithi181 ceph-mon[94189]: pgmap v3461: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 44 KiB/s wr, 8 op/s 2023-03-09T16:30:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:30:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5528.246434 secs 2023-03-09T16:30:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:30:17 smithi038 ceph-mon[92422]: pgmap v3461: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 44 KiB/s wr, 8 op/s 2023-03-09T16:30:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5990.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:30:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:30:20 smithi122 ceph-mon[93869]: pgmap v3462: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 44 KiB/s wr, 8 op/s 2023-03-09T16:30:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:30:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5531.011476 secs 2023-03-09T16:30:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:30:20 smithi122 ceph-mon[93869]: osdmap e429: 12 total, 12 up, 12 in 2023-03-09T16:30:20.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:30:20 smithi181 ceph-mon[94189]: pgmap v3462: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 44 KiB/s wr, 8 op/s 2023-03-09T16:30:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:30:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5531.011476 secs 2023-03-09T16:30:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:30:20 smithi181 ceph-mon[94189]: osdmap e429: 12 total, 12 up, 12 in 2023-03-09T16:30:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:30:20 smithi038 ceph-mon[92422]: pgmap v3462: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 44 KiB/s wr, 8 op/s 2023-03-09T16:30:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:30:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5531.011476 secs 2023-03-09T16:30:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:30:20 smithi038 ceph-mon[92422]: osdmap e429: 12 total, 12 up, 12 in 2023-03-09T16:30:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:30:22 smithi122 ceph-mon[93869]: pgmap v3464: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T16:30:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:30:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5533.246555 secs 2023-03-09T16:30:22.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:30:22 smithi181 ceph-mon[94189]: pgmap v3464: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T16:30:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:30:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5533.246555 secs 2023-03-09T16:30:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:30:22 smithi038 ceph-mon[92422]: pgmap v3464: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T16:30:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:30:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5533.246555 secs 2023-03-09T16:30:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 5995.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:30:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:30:24 smithi122 ceph-mon[93869]: pgmap v3465: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T16:30:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:30:24 smithi181 ceph-mon[94189]: pgmap v3465: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T16:30:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:30:24 smithi038 ceph-mon[92422]: pgmap v3465: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T16:30:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:30:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5536.011540 secs 2023-03-09T16:30:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:30:25 smithi122 ceph-mon[93869]: pgmap v3466: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T16:30:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:30:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5536.011540 secs 2023-03-09T16:30:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:30:25 smithi181 ceph-mon[94189]: pgmap v3466: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T16:30:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:30:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5536.011540 secs 2023-03-09T16:30:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:30:25 smithi038 ceph-mon[92422]: pgmap v3466: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T16:30:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:30:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5538.246649 secs 2023-03-09T16:30:27.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:30:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5538.246649 secs 2023-03-09T16:30:27.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:30:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5538.246649 secs 2023-03-09T16:30:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:30:27 smithi122 ceph-mon[93869]: pgmap v3467: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 6 op/s 2023-03-09T16:30:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:30:27 smithi181 ceph-mon[94189]: pgmap v3467: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 6 op/s 2023-03-09T16:30:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:30:27 smithi038 ceph-mon[92422]: pgmap v3467: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 6 op/s 2023-03-09T16:30:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6000.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:30:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:30:30 smithi122 ceph-mon[93869]: pgmap v3468: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 6 op/s 2023-03-09T16:30:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:30:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5541.011598 secs 2023-03-09T16:30:30.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:30:30 smithi181 ceph-mon[94189]: pgmap v3468: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 6 op/s 2023-03-09T16:30:30.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:30:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5541.011598 secs 2023-03-09T16:30:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:30:30 smithi038 ceph-mon[92422]: pgmap v3468: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 6 op/s 2023-03-09T16:30:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:30:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5541.011598 secs 2023-03-09T16:30:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:30:32 smithi122 ceph-mon[93869]: pgmap v3469: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 8 op/s 2023-03-09T16:30:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:30:32 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 5543.246790 secs 2023-03-09T16:30:32.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:30:32 smithi181 ceph-mon[94189]: pgmap v3469: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 8 op/s 2023-03-09T16:30:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:30:32 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 5543.246790 secs 2023-03-09T16:30:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:30:32 smithi038 ceph-mon[92422]: pgmap v3469: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 8 op/s 2023-03-09T16:30:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:30:32 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 5543.246790 secs 2023-03-09T16:30:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6005.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:30:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:30:34 smithi122 ceph-mon[93869]: pgmap v3470: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 73 KiB/s wr, 6 op/s 2023-03-09T16:30:34.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:30:34 smithi181 ceph-mon[94189]: pgmap v3470: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 73 KiB/s wr, 6 op/s 2023-03-09T16:30:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:30:34 smithi038 ceph-mon[92422]: pgmap v3470: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 73 KiB/s wr, 6 op/s 2023-03-09T16:30:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:30:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5546.011671 secs 2023-03-09T16:30:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:30:35 smithi122 ceph-mon[93869]: pgmap v3471: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 110 KiB/s wr, 6 op/s 2023-03-09T16:30:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:30:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5546.011671 secs 2023-03-09T16:30:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:30:35 smithi181 ceph-mon[94189]: pgmap v3471: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 110 KiB/s wr, 6 op/s 2023-03-09T16:30:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:30:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5546.011671 secs 2023-03-09T16:30:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:30:35 smithi038 ceph-mon[92422]: pgmap v3471: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 110 KiB/s wr, 6 op/s 2023-03-09T16:30:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:30:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:30:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:30:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:30:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:30:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:30:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:30:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:30:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:30:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:30:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:30:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:30:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:30:37 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 5548.246932 secs 2023-03-09T16:30:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:30:37 smithi122 ceph-mon[93869]: pgmap v3472: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 79 KiB/s wr, 8 op/s 2023-03-09T16:30:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:30:37 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 5548.246932 secs 2023-03-09T16:30:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:30:37 smithi181 ceph-mon[94189]: pgmap v3472: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 79 KiB/s wr, 8 op/s 2023-03-09T16:30:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:30:37 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 5548.246932 secs 2023-03-09T16:30:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:30:37 smithi038 ceph-mon[92422]: pgmap v3472: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 79 KiB/s wr, 8 op/s 2023-03-09T16:30:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6010.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:30:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:30:40 smithi122 ceph-mon[93869]: pgmap v3473: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 79 KiB/s wr, 6 op/s 2023-03-09T16:30:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:30:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5551.011729 secs 2023-03-09T16:30:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:30:40 smithi181 ceph-mon[94189]: pgmap v3473: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 79 KiB/s wr, 6 op/s 2023-03-09T16:30:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:30:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5551.011729 secs 2023-03-09T16:30:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:30:40 smithi038 ceph-mon[92422]: pgmap v3473: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 79 KiB/s wr, 6 op/s 2023-03-09T16:30:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:30:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5551.011729 secs 2023-03-09T16:30:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:30:42 smithi122 ceph-mon[93869]: pgmap v3474: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 99 KiB/s wr, 11 op/s 2023-03-09T16:30:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:30:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5553.247061 secs 2023-03-09T16:30:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:30:42 smithi181 ceph-mon[94189]: pgmap v3474: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 99 KiB/s wr, 11 op/s 2023-03-09T16:30:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:30:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5553.247061 secs 2023-03-09T16:30:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:30:42 smithi038 ceph-mon[92422]: pgmap v3474: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 99 KiB/s wr, 11 op/s 2023-03-09T16:30:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:30:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5553.247061 secs 2023-03-09T16:30:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6015.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:30:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:30:44 smithi122 ceph-mon[93869]: pgmap v3475: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 62 KiB/s wr, 9 op/s 2023-03-09T16:30:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:30:44 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:30:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:30:44 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:30:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:30:44 smithi181 ceph-mon[94189]: pgmap v3475: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 62 KiB/s wr, 9 op/s 2023-03-09T16:30:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:30:44 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:30:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:30:44 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:30:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:30:44 smithi038 ceph-mon[92422]: pgmap v3475: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 62 KiB/s wr, 9 op/s 2023-03-09T16:30:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:30:44 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:30:44.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:30:44 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:30:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:30:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5556.011791 secs 2023-03-09T16:30:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:30:45 smithi122 ceph-mon[93869]: pgmap v3476: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 190 KiB/s wr, 10 op/s 2023-03-09T16:30:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:30:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5556.011791 secs 2023-03-09T16:30:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:30:45 smithi181 ceph-mon[94189]: pgmap v3476: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 190 KiB/s wr, 10 op/s 2023-03-09T16:30:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:30:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5556.011791 secs 2023-03-09T16:30:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:30:45 smithi038 ceph-mon[92422]: pgmap v3476: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 190 KiB/s wr, 10 op/s 2023-03-09T16:30:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:30:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5558.247203 secs 2023-03-09T16:30:47.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:30:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5558.247203 secs 2023-03-09T16:30:47.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:30:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5558.247203 secs 2023-03-09T16:30:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:30:47 smithi122 ceph-mon[93869]: pgmap v3477: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 158 KiB/s wr, 12 op/s 2023-03-09T16:30:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:30:47 smithi181 ceph-mon[94189]: pgmap v3477: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 158 KiB/s wr, 12 op/s 2023-03-09T16:30:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:30:47 smithi038 ceph-mon[92422]: pgmap v3477: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 158 KiB/s wr, 12 op/s 2023-03-09T16:30:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6020.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:30:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:30:50 smithi122 ceph-mon[93869]: pgmap v3478: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 154 KiB/s wr, 10 op/s 2023-03-09T16:30:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:30:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5561.011851 secs 2023-03-09T16:30:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:30:50 smithi181 ceph-mon[94189]: pgmap v3478: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 154 KiB/s wr, 10 op/s 2023-03-09T16:30:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:30:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5561.011851 secs 2023-03-09T16:30:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:30:50 smithi038 ceph-mon[92422]: pgmap v3478: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 154 KiB/s wr, 10 op/s 2023-03-09T16:30:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:30:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5561.011851 secs 2023-03-09T16:30:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:30:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:30:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:30:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:30:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:30:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:30:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:30:52 smithi122 ceph-mon[93869]: pgmap v3479: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 169 KiB/s wr, 13 op/s 2023-03-09T16:30:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:30:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5563.247388 secs 2023-03-09T16:30:52.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:30:52 smithi181 ceph-mon[94189]: pgmap v3479: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 169 KiB/s wr, 13 op/s 2023-03-09T16:30:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:30:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5563.247388 secs 2023-03-09T16:30:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:30:52 smithi038 ceph-mon[92422]: pgmap v3479: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 169 KiB/s wr, 13 op/s 2023-03-09T16:30:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:30:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5563.247388 secs 2023-03-09T16:30:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6025.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:30:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:30:54 smithi122 ceph-mon[93869]: pgmap v3480: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 149 KiB/s wr, 9 op/s 2023-03-09T16:30:54.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:30:54 smithi181 ceph-mon[94189]: pgmap v3480: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 149 KiB/s wr, 9 op/s 2023-03-09T16:30:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:30:54 smithi038 ceph-mon[92422]: pgmap v3480: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 149 KiB/s wr, 9 op/s 2023-03-09T16:30:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:30:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:30:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:30:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5566.011911 secs 2023-03-09T16:30:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:30:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:30:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:30:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5566.011911 secs 2023-03-09T16:30:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:30:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:30:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:30:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5566.011911 secs 2023-03-09T16:30:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:30:56 smithi122 ceph-mon[93869]: pgmap v3481: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 282 KiB/s wr, 10 op/s 2023-03-09T16:30:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:30:56 smithi181 ceph-mon[94189]: pgmap v3481: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 282 KiB/s wr, 10 op/s 2023-03-09T16:30:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:30:56 smithi038 ceph-mon[92422]: pgmap v3481: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 282 KiB/s wr, 10 op/s 2023-03-09T16:30:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:30:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5568.247491 secs 2023-03-09T16:30:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:30:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5568.247491 secs 2023-03-09T16:30:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:30:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5568.247491 secs 2023-03-09T16:30:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:30:58 smithi122 ceph-mon[93869]: pgmap v3482: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 164 KiB/s wr, 11 op/s 2023-03-09T16:30:58.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:30:58 smithi181 ceph-mon[94189]: pgmap v3482: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 164 KiB/s wr, 11 op/s 2023-03-09T16:30:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:30:58 smithi038 ceph-mon[92422]: pgmap v3482: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 164 KiB/s wr, 11 op/s 2023-03-09T16:30:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6030.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:31:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:31:00 smithi122 ceph-mon[93869]: pgmap v3483: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 158 KiB/s wr, 8 op/s 2023-03-09T16:31:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:31:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5571.011980 secs 2023-03-09T16:31:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:31:00 smithi181 ceph-mon[94189]: pgmap v3483: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 158 KiB/s wr, 8 op/s 2023-03-09T16:31:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:31:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5571.011980 secs 2023-03-09T16:31:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:31:00 smithi038 ceph-mon[92422]: pgmap v3483: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 158 KiB/s wr, 8 op/s 2023-03-09T16:31:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:31:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5571.011980 secs 2023-03-09T16:31:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:31:02 smithi122 ceph-mon[93869]: pgmap v3484: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 211 KiB/s wr, 13 op/s 2023-03-09T16:31:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:31:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5573.247591 secs 2023-03-09T16:31:02.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:31:02 smithi181 ceph-mon[94189]: pgmap v3484: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 211 KiB/s wr, 13 op/s 2023-03-09T16:31:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:31:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5573.247591 secs 2023-03-09T16:31:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:31:02 smithi038 ceph-mon[92422]: pgmap v3484: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 211 KiB/s wr, 13 op/s 2023-03-09T16:31:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:31:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5573.247591 secs 2023-03-09T16:31:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6035.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:31:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:31:04 smithi122 ceph-mon[93869]: pgmap v3485: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 219 KiB/s wr, 23 op/s 2023-03-09T16:31:04.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:31:04 smithi181 ceph-mon[94189]: pgmap v3485: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 219 KiB/s wr, 23 op/s 2023-03-09T16:31:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:31:04 smithi038 ceph-mon[92422]: pgmap v3485: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 219 KiB/s wr, 23 op/s 2023-03-09T16:31:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:31:05 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:31:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:31:05 smithi122 ceph-mon[93869]: osdmap e430: 12 total, 12 up, 12 in 2023-03-09T16:31:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:31:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5576.012042 secs 2023-03-09T16:31:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:31:05 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:31:05.522 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:31:05 smithi181 ceph-mon[94189]: osdmap e430: 12 total, 12 up, 12 in 2023-03-09T16:31:05.523 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:31:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5576.012042 secs 2023-03-09T16:31:05.523 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:31:05 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:31:05.523 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:31:05 smithi038 ceph-mon[92422]: osdmap e430: 12 total, 12 up, 12 in 2023-03-09T16:31:05.524 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:31:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5576.012042 secs 2023-03-09T16:31:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:31:06 smithi122 ceph-mon[93869]: pgmap v3487: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 103 KiB/s wr, 26 op/s 2023-03-09T16:31:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:31:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:31:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:31:06 smithi181 ceph-mon[94189]: pgmap v3487: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 103 KiB/s wr, 26 op/s 2023-03-09T16:31:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:31:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:31:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:31:06 smithi038 ceph-mon[92422]: pgmap v3487: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 103 KiB/s wr, 26 op/s 2023-03-09T16:31:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:31:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:31:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:31:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5578.247720 secs 2023-03-09T16:31:07.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:31:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5578.247720 secs 2023-03-09T16:31:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:31:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5578.247720 secs 2023-03-09T16:31:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:31:08 smithi122 ceph-mon[93869]: pgmap v3488: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 190 KiB/s wr, 28 op/s 2023-03-09T16:31:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:31:08 smithi181 ceph-mon[94189]: pgmap v3488: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 190 KiB/s wr, 28 op/s 2023-03-09T16:31:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:31:08 smithi038 ceph-mon[92422]: pgmap v3488: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 190 KiB/s wr, 28 op/s 2023-03-09T16:31:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6040.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:31:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:31:10 smithi122 ceph-mon[93869]: pgmap v3489: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 190 KiB/s wr, 28 op/s 2023-03-09T16:31:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:31:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5581.012105 secs 2023-03-09T16:31:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:31:10 smithi181 ceph-mon[94189]: pgmap v3489: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 190 KiB/s wr, 28 op/s 2023-03-09T16:31:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:31:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5581.012105 secs 2023-03-09T16:31:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:31:10 smithi038 ceph-mon[92422]: pgmap v3489: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 190 KiB/s wr, 28 op/s 2023-03-09T16:31:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:31:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5581.012105 secs 2023-03-09T16:31:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:31:12 smithi122 ceph-mon[93869]: pgmap v3490: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 127 KiB/s wr, 24 op/s 2023-03-09T16:31:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:31:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5583.247812 secs 2023-03-09T16:31:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:31:12 smithi181 ceph-mon[94189]: pgmap v3490: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 127 KiB/s wr, 24 op/s 2023-03-09T16:31:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:31:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5583.247812 secs 2023-03-09T16:31:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:31:12 smithi038 ceph-mon[92422]: pgmap v3490: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 127 KiB/s wr, 24 op/s 2023-03-09T16:31:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:31:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5583.247812 secs 2023-03-09T16:31:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6045.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:31:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:31:14 smithi122 ceph-mon[93869]: pgmap v3491: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 127 KiB/s wr, 9 op/s 2023-03-09T16:31:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:31:14 smithi181 ceph-mon[94189]: pgmap v3491: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 127 KiB/s wr, 9 op/s 2023-03-09T16:31:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:31:14 smithi038 ceph-mon[92422]: pgmap v3491: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 127 KiB/s wr, 9 op/s 2023-03-09T16:31:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:31:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5586.012185 secs 2023-03-09T16:31:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:31:15 smithi122 ceph-mon[93869]: osdmap e431: 12 total, 12 up, 12 in 2023-03-09T16:31:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:31:15 smithi122 ceph-mon[93869]: pgmap v3493: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 128 KiB/s wr, 9 op/s 2023-03-09T16:31:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:31:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5586.012185 secs 2023-03-09T16:31:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:31:15 smithi181 ceph-mon[94189]: osdmap e431: 12 total, 12 up, 12 in 2023-03-09T16:31:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:31:15 smithi181 ceph-mon[94189]: pgmap v3493: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 128 KiB/s wr, 9 op/s 2023-03-09T16:31:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:31:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5586.012185 secs 2023-03-09T16:31:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:31:15 smithi038 ceph-mon[92422]: osdmap e431: 12 total, 12 up, 12 in 2023-03-09T16:31:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:31:15 smithi038 ceph-mon[92422]: pgmap v3493: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 128 KiB/s wr, 9 op/s 2023-03-09T16:31:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:31:16 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:31:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:31:16 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:31:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:31:16 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:31:16.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:31:16 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:31:16.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:31:16 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:31:16.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:31:16 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:31:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:31:16 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:31:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:31:16 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:31:17.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:31:16 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:31:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:31:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5588.247904 secs 2023-03-09T16:31:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:31:17 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:31:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:31:17 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:31:17.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:31:17 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:31:17.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:31:17 smithi122 ceph-mon[93869]: pgmap v3494: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 29 KiB/s wr, 8 op/s 2023-03-09T16:31:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:31:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5588.247904 secs 2023-03-09T16:31:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:31:17 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:31:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:31:17 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:31:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:31:17 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:31:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:31:17 smithi181 ceph-mon[94189]: pgmap v3494: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 29 KiB/s wr, 8 op/s 2023-03-09T16:31:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:31:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5588.247904 secs 2023-03-09T16:31:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:31:17 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:31:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:31:17 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:31:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:31:17 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:31:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:31:17 smithi038 ceph-mon[92422]: pgmap v3494: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 29 KiB/s wr, 8 op/s 2023-03-09T16:31:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6050.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:31:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:31:18 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:31:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:31:18 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:31:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:31:18 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:31:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:31:19 smithi122 ceph-mon[93869]: pgmap v3495: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 29 KiB/s wr, 8 op/s 2023-03-09T16:31:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:31:19 smithi181 ceph-mon[94189]: pgmap v3495: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 29 KiB/s wr, 8 op/s 2023-03-09T16:31:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:31:19 smithi038 ceph-mon[92422]: pgmap v3495: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 29 KiB/s wr, 8 op/s 2023-03-09T16:31:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:31:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5591.012244 secs 2023-03-09T16:31:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:31:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5591.012244 secs 2023-03-09T16:31:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:31:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5591.012244 secs 2023-03-09T16:31:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:31:21 smithi122 ceph-mon[93869]: pgmap v3496: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 30 KiB/s wr, 8 op/s 2023-03-09T16:31:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:31:21 smithi181 ceph-mon[94189]: pgmap v3496: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 30 KiB/s wr, 8 op/s 2023-03-09T16:31:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:31:21 smithi038 ceph-mon[92422]: pgmap v3496: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 30 KiB/s wr, 8 op/s 2023-03-09T16:31:22.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:31:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5593.247974 secs 2023-03-09T16:31:22.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:31:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5593.247974 secs 2023-03-09T16:31:22.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:31:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5593.247974 secs 2023-03-09T16:31:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6055.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:31:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:31:23 smithi122 ceph-mon[93869]: pgmap v3497: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 46 KiB/s wr, 9 op/s 2023-03-09T16:31:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:31:23 smithi181 ceph-mon[94189]: pgmap v3497: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 46 KiB/s wr, 9 op/s 2023-03-09T16:31:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:31:23 smithi038 ceph-mon[92422]: pgmap v3497: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 46 KiB/s wr, 9 op/s 2023-03-09T16:31:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:31:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5596.012305 secs 2023-03-09T16:31:26.042 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:31:25 smithi122 ceph-mon[93869]: pgmap v3498: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 43 KiB/s wr, 8 op/s 2023-03-09T16:31:26.043 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:31:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5596.012305 secs 2023-03-09T16:31:26.043 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:31:25 smithi038 ceph-mon[92422]: pgmap v3498: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 43 KiB/s wr, 8 op/s 2023-03-09T16:31:26.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:31:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5596.012305 secs 2023-03-09T16:31:26.044 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:31:25 smithi181 ceph-mon[94189]: pgmap v3498: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 43 KiB/s wr, 8 op/s 2023-03-09T16:31:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:31:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5598.248057 secs 2023-03-09T16:31:27.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:31:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5598.248057 secs 2023-03-09T16:31:27.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:31:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5598.248057 secs 2023-03-09T16:31:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:31:27 smithi122 ceph-mon[93869]: pgmap v3499: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 68 KiB/s wr, 10 op/s 2023-03-09T16:31:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:31:27 smithi181 ceph-mon[94189]: pgmap v3499: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 68 KiB/s wr, 10 op/s 2023-03-09T16:31:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:31:27 smithi038 ceph-mon[92422]: pgmap v3499: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 68 KiB/s wr, 10 op/s 2023-03-09T16:31:28.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6060.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:31:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:31:30 smithi122 ceph-mon[93869]: pgmap v3500: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 68 KiB/s wr, 7 op/s 2023-03-09T16:31:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:31:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5601.012368 secs 2023-03-09T16:31:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:31:30 smithi181 ceph-mon[94189]: pgmap v3500: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 68 KiB/s wr, 7 op/s 2023-03-09T16:31:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:31:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5601.012368 secs 2023-03-09T16:31:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:31:30 smithi038 ceph-mon[92422]: pgmap v3500: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 68 KiB/s wr, 7 op/s 2023-03-09T16:31:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:31:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5601.012368 secs 2023-03-09T16:31:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:31:32 smithi122 ceph-mon[93869]: pgmap v3501: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 68 KiB/s wr, 9 op/s 2023-03-09T16:31:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:31:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5603.248146 secs 2023-03-09T16:31:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:31:32 smithi181 ceph-mon[94189]: pgmap v3501: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 68 KiB/s wr, 9 op/s 2023-03-09T16:31:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:31:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5603.248146 secs 2023-03-09T16:31:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:31:32 smithi038 ceph-mon[92422]: pgmap v3501: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 68 KiB/s wr, 9 op/s 2023-03-09T16:31:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:31:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5603.248146 secs 2023-03-09T16:31:33.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6065.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:31:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:31:34 smithi122 ceph-mon[93869]: pgmap v3502: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 104 KiB/s wr, 7 op/s 2023-03-09T16:31:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:31:34 smithi181 ceph-mon[94189]: pgmap v3502: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 104 KiB/s wr, 7 op/s 2023-03-09T16:31:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:31:34 smithi038 ceph-mon[92422]: pgmap v3502: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 104 KiB/s wr, 7 op/s 2023-03-09T16:31:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:31:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5606.012427 secs 2023-03-09T16:31:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:31:35 smithi122 ceph-mon[93869]: pgmap v3503: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 68 KiB/s wr, 6 op/s 2023-03-09T16:31:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:31:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5606.012427 secs 2023-03-09T16:31:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:31:35 smithi181 ceph-mon[94189]: pgmap v3503: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 68 KiB/s wr, 6 op/s 2023-03-09T16:31:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:31:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5606.012427 secs 2023-03-09T16:31:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:31:35 smithi038 ceph-mon[92422]: pgmap v3503: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 68 KiB/s wr, 6 op/s 2023-03-09T16:31:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:31:38 smithi122 ceph-mon[93869]: pgmap v3504: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 162 KiB/s wr, 11 op/s 2023-03-09T16:31:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:31:38 smithi181 ceph-mon[94189]: pgmap v3504: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 162 KiB/s wr, 11 op/s 2023-03-09T16:31:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:31:38 smithi038 ceph-mon[92422]: pgmap v3504: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 162 KiB/s wr, 11 op/s 2023-03-09T16:31:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6070.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:31:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:31:39 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:31:39.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:31:39 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:31:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:31:39 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:31:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:31:40 smithi122 ceph-mon[93869]: pgmap v3505: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 131 KiB/s wr, 8 op/s 2023-03-09T16:31:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:31:40 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:31:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:31:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:31:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:31:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:31:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:31:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5611.012489 secs 2023-03-09T16:31:40.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:31:40 smithi181 ceph-mon[94189]: pgmap v3505: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 131 KiB/s wr, 8 op/s 2023-03-09T16:31:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:31:40 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:31:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:31:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:31:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:31:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:31:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:31:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5611.012489 secs 2023-03-09T16:31:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:31:40 smithi038 ceph-mon[92422]: pgmap v3505: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 131 KiB/s wr, 8 op/s 2023-03-09T16:31:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:31:40 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:31:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:31:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:31:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:31:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:31:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:31:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5611.012489 secs 2023-03-09T16:31:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:31:42 smithi122 ceph-mon[93869]: pgmap v3506: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 136 KiB/s wr, 11 op/s 2023-03-09T16:31:42.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:31:42 smithi181 ceph-mon[94189]: pgmap v3506: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 136 KiB/s wr, 11 op/s 2023-03-09T16:31:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:31:42 smithi038 ceph-mon[92422]: pgmap v3506: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 136 KiB/s wr, 11 op/s 2023-03-09T16:31:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6075.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:31:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:31:44 smithi122 ceph-mon[93869]: pgmap v3507: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 255 KiB/s wr, 11 op/s 2023-03-09T16:31:44.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:31:44 smithi181 ceph-mon[94189]: pgmap v3507: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 255 KiB/s wr, 11 op/s 2023-03-09T16:31:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:31:44 smithi038 ceph-mon[92422]: pgmap v3507: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 255 KiB/s wr, 11 op/s 2023-03-09T16:31:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:31:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5616.012549 secs 2023-03-09T16:31:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:31:45 smithi122 ceph-mon[93869]: pgmap v3508: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 218 KiB/s wr, 10 op/s 2023-03-09T16:31:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:31:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5616.012549 secs 2023-03-09T16:31:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:31:45 smithi181 ceph-mon[94189]: pgmap v3508: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 218 KiB/s wr, 10 op/s 2023-03-09T16:31:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:31:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5616.012549 secs 2023-03-09T16:31:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:31:45 smithi038 ceph-mon[92422]: pgmap v3508: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 218 KiB/s wr, 10 op/s 2023-03-09T16:31:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:31:48 smithi122 ceph-mon[93869]: pgmap v3509: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 224 KiB/s wr, 13 op/s 2023-03-09T16:31:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:31:48 smithi181 ceph-mon[94189]: pgmap v3509: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 224 KiB/s wr, 13 op/s 2023-03-09T16:31:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:31:48 smithi038 ceph-mon[92422]: pgmap v3509: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 224 KiB/s wr, 13 op/s 2023-03-09T16:31:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6080.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:31:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:31:50 smithi122 ceph-mon[93869]: pgmap v3510: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 173 KiB/s wr, 10 op/s 2023-03-09T16:31:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:31:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5621.012634 secs 2023-03-09T16:31:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:31:50 smithi181 ceph-mon[94189]: pgmap v3510: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 173 KiB/s wr, 10 op/s 2023-03-09T16:31:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:31:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5621.012634 secs 2023-03-09T16:31:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:31:50 smithi038 ceph-mon[92422]: pgmap v3510: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 173 KiB/s wr, 10 op/s 2023-03-09T16:31:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:31:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5621.012634 secs 2023-03-09T16:31:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:31:52 smithi122 ceph-mon[93869]: pgmap v3511: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 177 KiB/s wr, 12 op/s 2023-03-09T16:31:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:31:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5623.248635 secs 2023-03-09T16:31:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:31:52 smithi181 ceph-mon[94189]: pgmap v3511: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 177 KiB/s wr, 12 op/s 2023-03-09T16:31:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:31:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5623.248635 secs 2023-03-09T16:31:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:31:52 smithi038 ceph-mon[92422]: pgmap v3511: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 177 KiB/s wr, 12 op/s 2023-03-09T16:31:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:31:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5623.248635 secs 2023-03-09T16:31:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6085.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:31:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:31:54 smithi122 ceph-mon[93869]: pgmap v3512: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 242 KiB/s wr, 11 op/s 2023-03-09T16:31:54.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:31:54 smithi181 ceph-mon[94189]: pgmap v3512: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 242 KiB/s wr, 11 op/s 2023-03-09T16:31:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:31:54 smithi038 ceph-mon[92422]: pgmap v3512: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 242 KiB/s wr, 11 op/s 2023-03-09T16:31:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:31:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5626.012709 secs 2023-03-09T16:31:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:31:55 smithi122 ceph-mon[93869]: pgmap v3513: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 122 KiB/s wr, 8 op/s 2023-03-09T16:31:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:31:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5626.012709 secs 2023-03-09T16:31:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:31:55 smithi181 ceph-mon[94189]: pgmap v3513: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 122 KiB/s wr, 8 op/s 2023-03-09T16:31:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:31:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5626.012709 secs 2023-03-09T16:31:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:31:55 smithi038 ceph-mon[92422]: pgmap v3513: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 122 KiB/s wr, 8 op/s 2023-03-09T16:31:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:31:56 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:31:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:31:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:31:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:31:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:31:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:31:56 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:31:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:31:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:31:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:31:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:31:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:31:56 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:31:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:31:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:31:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:31:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:31:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:31:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5628.248759 secs 2023-03-09T16:31:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:31:57 smithi122 ceph-mon[93869]: pgmap v3514: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 123 KiB/s wr, 11 op/s 2023-03-09T16:31:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:31:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5628.248759 secs 2023-03-09T16:31:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:31:57 smithi181 ceph-mon[94189]: pgmap v3514: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 123 KiB/s wr, 11 op/s 2023-03-09T16:31:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:31:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5628.248759 secs 2023-03-09T16:31:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:31:57 smithi038 ceph-mon[92422]: pgmap v3514: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 123 KiB/s wr, 11 op/s 2023-03-09T16:31:58.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6090.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:32:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:32:00 smithi122 ceph-mon[93869]: pgmap v3515: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 155 KiB/s wr, 8 op/s 2023-03-09T16:32:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:32:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5631.012767 secs 2023-03-09T16:32:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:32:00 smithi181 ceph-mon[94189]: pgmap v3515: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 155 KiB/s wr, 8 op/s 2023-03-09T16:32:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:32:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5631.012767 secs 2023-03-09T16:32:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:32:00 smithi038 ceph-mon[92422]: pgmap v3515: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 155 KiB/s wr, 8 op/s 2023-03-09T16:32:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:32:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5631.012767 secs 2023-03-09T16:32:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:32:02 smithi122 ceph-mon[93869]: pgmap v3516: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 135 KiB/s wr, 12 op/s 2023-03-09T16:32:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:32:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5633.248873 secs 2023-03-09T16:32:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:32:02 smithi181 ceph-mon[94189]: pgmap v3516: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 135 KiB/s wr, 12 op/s 2023-03-09T16:32:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:32:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5633.248873 secs 2023-03-09T16:32:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:32:02 smithi038 ceph-mon[92422]: pgmap v3516: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 135 KiB/s wr, 12 op/s 2023-03-09T16:32:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:32:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5633.248873 secs 2023-03-09T16:32:03.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6095.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:32:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:32:04 smithi122 ceph-mon[93869]: pgmap v3517: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 188 KiB/s wr, 22 op/s 2023-03-09T16:32:04.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:32:04 smithi181 ceph-mon[94189]: pgmap v3517: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 188 KiB/s wr, 22 op/s 2023-03-09T16:32:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:32:04 smithi038 ceph-mon[92422]: pgmap v3517: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 188 KiB/s wr, 22 op/s 2023-03-09T16:32:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:32:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5636.012826 secs 2023-03-09T16:32:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:32:05 smithi122 ceph-mon[93869]: pgmap v3518: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 119 KiB/s wr, 21 op/s 2023-03-09T16:32:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:32:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5636.012826 secs 2023-03-09T16:32:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:32:05 smithi181 ceph-mon[94189]: pgmap v3518: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 119 KiB/s wr, 21 op/s 2023-03-09T16:32:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:32:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5636.012826 secs 2023-03-09T16:32:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:32:05 smithi038 ceph-mon[92422]: pgmap v3518: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 119 KiB/s wr, 21 op/s 2023-03-09T16:32:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:32:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5638.249005 secs 2023-03-09T16:32:07.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:32:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5638.249005 secs 2023-03-09T16:32:07.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:32:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5638.249005 secs 2023-03-09T16:32:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:32:07 smithi038 ceph-mon[92422]: pgmap v3519: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 125 KiB/s wr, 24 op/s 2023-03-09T16:32:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:32:07 smithi122 ceph-mon[93869]: pgmap v3519: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 125 KiB/s wr, 24 op/s 2023-03-09T16:32:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:32:07 smithi181 ceph-mon[94189]: pgmap v3519: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 125 KiB/s wr, 24 op/s 2023-03-09T16:32:08.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6100.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:32:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:32:09 smithi038 ceph-mon[92422]: osdmap e432: 12 total, 12 up, 12 in 2023-03-09T16:32:10.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:32:09 smithi038 ceph-mon[92422]: pgmap v3521: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 104 KiB/s wr, 25 op/s 2023-03-09T16:32:10.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:32:09 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5641.012909 secs 2023-03-09T16:32:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:32:09 smithi122 ceph-mon[93869]: osdmap e432: 12 total, 12 up, 12 in 2023-03-09T16:32:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:32:09 smithi122 ceph-mon[93869]: pgmap v3521: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 104 KiB/s wr, 25 op/s 2023-03-09T16:32:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:32:09 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5641.012909 secs 2023-03-09T16:32:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:32:09 smithi181 ceph-mon[94189]: osdmap e432: 12 total, 12 up, 12 in 2023-03-09T16:32:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:32:09 smithi181 ceph-mon[94189]: pgmap v3521: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 104 KiB/s wr, 25 op/s 2023-03-09T16:32:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:32:09 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5641.012909 secs 2023-03-09T16:32:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:32:12 smithi122 ceph-mon[93869]: pgmap v3522: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 83 KiB/s wr, 23 op/s 2023-03-09T16:32:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:32:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5643.249130 secs 2023-03-09T16:32:12.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:32:12 smithi181 ceph-mon[94189]: pgmap v3522: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 83 KiB/s wr, 23 op/s 2023-03-09T16:32:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:32:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5643.249130 secs 2023-03-09T16:32:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:32:12 smithi038 ceph-mon[92422]: pgmap v3522: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 83 KiB/s wr, 23 op/s 2023-03-09T16:32:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:32:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5643.249130 secs 2023-03-09T16:32:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6105.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:32:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:32:14 smithi122 ceph-mon[93869]: pgmap v3523: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 79 KiB/s wr, 9 op/s 2023-03-09T16:32:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:32:14 smithi181 ceph-mon[94189]: pgmap v3523: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 79 KiB/s wr, 9 op/s 2023-03-09T16:32:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:32:14 smithi038 ceph-mon[92422]: pgmap v3523: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 79 KiB/s wr, 9 op/s 2023-03-09T16:32:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:32:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5646.012975 secs 2023-03-09T16:32:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:32:15 smithi122 ceph-mon[93869]: pgmap v3524: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 79 KiB/s wr, 9 op/s 2023-03-09T16:32:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:32:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5646.012975 secs 2023-03-09T16:32:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:32:15 smithi181 ceph-mon[94189]: pgmap v3524: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 79 KiB/s wr, 9 op/s 2023-03-09T16:32:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:32:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5646.012975 secs 2023-03-09T16:32:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:32:15 smithi038 ceph-mon[92422]: pgmap v3524: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 79 KiB/s wr, 9 op/s 2023-03-09T16:32:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:32:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5648.249251 secs 2023-03-09T16:32:17.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:32:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5648.249251 secs 2023-03-09T16:32:17.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:32:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5648.249251 secs 2023-03-09T16:32:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:32:17 smithi038 ceph-mon[92422]: pgmap v3525: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 8 op/s 2023-03-09T16:32:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:32:17 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:32:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:32:17 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:32:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:32:17 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:32:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:32:17 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:32:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:32:17 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:32:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:32:17 smithi122 ceph-mon[93869]: pgmap v3525: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 8 op/s 2023-03-09T16:32:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:32:17 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:32:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:32:17 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:32:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:32:17 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:32:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:32:17 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:32:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:32:17 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:32:18.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:32:17 smithi181 ceph-mon[94189]: pgmap v3525: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 8 op/s 2023-03-09T16:32:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:32:17 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:32:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:32:17 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:32:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:32:17 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:32:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:32:17 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:32:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:32:17 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:32:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6110.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:32:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:32:18 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:32:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:32:18 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:32:19.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:32:18 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:32:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:32:18 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:32:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:32:18 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:32:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:32:18 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:32:19.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:32:18 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:32:19.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:32:18 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:32:19.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:32:18 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:32:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:32:20 smithi122 ceph-mon[93869]: pgmap v3526: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 8 op/s 2023-03-09T16:32:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:32:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5651.012981 secs 2023-03-09T16:32:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:32:20 smithi122 ceph-mon[93869]: osdmap e433: 12 total, 12 up, 12 in 2023-03-09T16:32:20.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:32:20 smithi181 ceph-mon[94189]: pgmap v3526: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 8 op/s 2023-03-09T16:32:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:32:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5651.012981 secs 2023-03-09T16:32:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:32:20 smithi181 ceph-mon[94189]: osdmap e433: 12 total, 12 up, 12 in 2023-03-09T16:32:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:32:20 smithi038 ceph-mon[92422]: pgmap v3526: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 8 op/s 2023-03-09T16:32:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:32:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5651.012981 secs 2023-03-09T16:32:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:32:20 smithi038 ceph-mon[92422]: osdmap e433: 12 total, 12 up, 12 in 2023-03-09T16:32:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:32:22 smithi122 ceph-mon[93869]: pgmap v3528: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 66 KiB/s wr, 8 op/s 2023-03-09T16:32:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:32:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5653.249374 secs 2023-03-09T16:32:22.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:32:22 smithi181 ceph-mon[94189]: pgmap v3528: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 66 KiB/s wr, 8 op/s 2023-03-09T16:32:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:32:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5653.249374 secs 2023-03-09T16:32:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:32:22 smithi038 ceph-mon[92422]: pgmap v3528: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 66 KiB/s wr, 8 op/s 2023-03-09T16:32:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:32:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5653.249374 secs 2023-03-09T16:32:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6115.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:32:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:32:24 smithi122 ceph-mon[93869]: pgmap v3529: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T16:32:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:32:24 smithi181 ceph-mon[94189]: pgmap v3529: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T16:32:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:32:24 smithi038 ceph-mon[92422]: pgmap v3529: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T16:32:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:32:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5656.013043 secs 2023-03-09T16:32:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:32:25 smithi122 ceph-mon[93869]: pgmap v3530: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T16:32:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:32:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5656.013043 secs 2023-03-09T16:32:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:32:25 smithi181 ceph-mon[94189]: pgmap v3530: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T16:32:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:32:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5656.013043 secs 2023-03-09T16:32:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:32:25 smithi038 ceph-mon[92422]: pgmap v3530: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T16:32:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:32:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5658.249455 secs 2023-03-09T16:32:27.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:32:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5658.249455 secs 2023-03-09T16:32:27.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:32:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5658.249455 secs 2023-03-09T16:32:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:32:27 smithi038 ceph-mon[92422]: pgmap v3531: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T16:32:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:32:27 smithi122 ceph-mon[93869]: pgmap v3531: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T16:32:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:32:27 smithi181 ceph-mon[94189]: pgmap v3531: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T16:32:28.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6120.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:32:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:32:30 smithi122 ceph-mon[93869]: pgmap v3532: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 44 KiB/s wr, 6 op/s 2023-03-09T16:32:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:32:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5661.013103 secs 2023-03-09T16:32:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:32:30 smithi181 ceph-mon[94189]: pgmap v3532: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 44 KiB/s wr, 6 op/s 2023-03-09T16:32:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:32:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5661.013103 secs 2023-03-09T16:32:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:32:30 smithi038 ceph-mon[92422]: pgmap v3532: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 44 KiB/s wr, 6 op/s 2023-03-09T16:32:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:32:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5661.013103 secs 2023-03-09T16:32:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:32:32 smithi122 ceph-mon[93869]: pgmap v3533: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 77 KiB/s wr, 8 op/s 2023-03-09T16:32:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:32:32 smithi181 ceph-mon[94189]: pgmap v3533: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 77 KiB/s wr, 8 op/s 2023-03-09T16:32:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:32:32 smithi038 ceph-mon[92422]: pgmap v3533: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 77 KiB/s wr, 8 op/s 2023-03-09T16:32:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6125.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:32:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:32:34 smithi122 ceph-mon[93869]: pgmap v3534: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 74 KiB/s wr, 6 op/s 2023-03-09T16:32:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:32:34 smithi181 ceph-mon[94189]: pgmap v3534: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 74 KiB/s wr, 6 op/s 2023-03-09T16:32:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:32:34 smithi038 ceph-mon[92422]: pgmap v3534: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 74 KiB/s wr, 6 op/s 2023-03-09T16:32:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:32:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5666.013163 secs 2023-03-09T16:32:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:32:35 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:32:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:32:35 smithi122 ceph-mon[93869]: pgmap v3535: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 38 KiB/s wr, 6 op/s 2023-03-09T16:32:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:32:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5666.013163 secs 2023-03-09T16:32:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:32:35 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:32:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:32:35 smithi181 ceph-mon[94189]: pgmap v3535: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 38 KiB/s wr, 6 op/s 2023-03-09T16:32:36.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:32:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5666.013163 secs 2023-03-09T16:32:36.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:32:35 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:32:36.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:32:35 smithi038 ceph-mon[92422]: pgmap v3535: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 38 KiB/s wr, 6 op/s 2023-03-09T16:32:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:32:36 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:32:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:32:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:32:36.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:32:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:32:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:32:36 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:32:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:32:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:32:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:32:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:32:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:32:36 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:32:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:32:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:32:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:32:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:32:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:32:37 smithi122 ceph-mon[93869]: pgmap v3536: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 81 KiB/s wr, 9 op/s 2023-03-09T16:32:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:32:37 smithi181 ceph-mon[94189]: pgmap v3536: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 81 KiB/s wr, 9 op/s 2023-03-09T16:32:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:32:37 smithi038 ceph-mon[92422]: pgmap v3536: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 81 KiB/s wr, 9 op/s 2023-03-09T16:32:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6130.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:32:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:32:40 smithi122 ceph-mon[93869]: pgmap v3537: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 81 KiB/s wr, 7 op/s 2023-03-09T16:32:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:32:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5671.013243 secs 2023-03-09T16:32:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:32:40 smithi181 ceph-mon[94189]: pgmap v3537: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 81 KiB/s wr, 7 op/s 2023-03-09T16:32:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:32:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5671.013243 secs 2023-03-09T16:32:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:32:40 smithi038 ceph-mon[92422]: pgmap v3537: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 81 KiB/s wr, 7 op/s 2023-03-09T16:32:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:32:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5671.013243 secs 2023-03-09T16:32:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:32:42 smithi122 ceph-mon[93869]: pgmap v3538: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 129 KiB/s wr, 11 op/s 2023-03-09T16:32:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:32:42 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 5673.249815 secs 2023-03-09T16:32:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:32:42 smithi181 ceph-mon[94189]: pgmap v3538: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 129 KiB/s wr, 11 op/s 2023-03-09T16:32:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:32:42 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 5673.249815 secs 2023-03-09T16:32:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:32:42 smithi038 ceph-mon[92422]: pgmap v3538: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 129 KiB/s wr, 11 op/s 2023-03-09T16:32:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:32:42 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 5673.249815 secs 2023-03-09T16:32:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:32:43 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:32:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:32:43 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:32:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:32:43 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:32:43.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:32:43 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:32:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:32:43 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:32:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:32:43 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:32:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:32:43 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:32:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:32:43 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:32:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:32:43 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:32:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6135.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:32:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:32:44 smithi122 ceph-mon[93869]: pgmap v3539: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 92 KiB/s wr, 9 op/s 2023-03-09T16:32:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:32:44 smithi181 ceph-mon[94189]: pgmap v3539: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 92 KiB/s wr, 9 op/s 2023-03-09T16:32:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:32:44 smithi038 ceph-mon[92422]: pgmap v3539: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 92 KiB/s wr, 9 op/s 2023-03-09T16:32:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:32:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5676.013302 secs 2023-03-09T16:32:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:32:45 smithi122 ceph-mon[93869]: pgmap v3540: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 91 KiB/s wr, 8 op/s 2023-03-09T16:32:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:32:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5676.013302 secs 2023-03-09T16:32:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:32:45 smithi181 ceph-mon[94189]: pgmap v3540: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 91 KiB/s wr, 8 op/s 2023-03-09T16:32:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:32:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5676.013302 secs 2023-03-09T16:32:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:32:45 smithi038 ceph-mon[92422]: pgmap v3540: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 91 KiB/s wr, 8 op/s 2023-03-09T16:32:47.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:32:46 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 5678.249932 secs 2023-03-09T16:32:47.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:32:46 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 5678.249932 secs 2023-03-09T16:32:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:32:46 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 5678.249932 secs 2023-03-09T16:32:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:32:47 smithi038 ceph-mon[92422]: pgmap v3541: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 199 KiB/s wr, 13 op/s 2023-03-09T16:32:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:32:47 smithi122 ceph-mon[93869]: pgmap v3541: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 199 KiB/s wr, 13 op/s 2023-03-09T16:32:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:32:47 smithi181 ceph-mon[94189]: pgmap v3541: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 199 KiB/s wr, 13 op/s 2023-03-09T16:32:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6140.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:32:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:32:50 smithi122 ceph-mon[93869]: pgmap v3542: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 157 KiB/s wr, 10 op/s 2023-03-09T16:32:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:32:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5681.013389 secs 2023-03-09T16:32:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:32:50 smithi181 ceph-mon[94189]: pgmap v3542: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 157 KiB/s wr, 10 op/s 2023-03-09T16:32:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:32:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5681.013389 secs 2023-03-09T16:32:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:32:50 smithi038 ceph-mon[92422]: pgmap v3542: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 157 KiB/s wr, 10 op/s 2023-03-09T16:32:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:32:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5681.013389 secs 2023-03-09T16:32:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:32:52 smithi122 ceph-mon[93869]: pgmap v3543: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 172 KiB/s wr, 12 op/s 2023-03-09T16:32:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:32:52 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 5683.250016 secs 2023-03-09T16:32:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:32:52 smithi181 ceph-mon[94189]: pgmap v3543: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 172 KiB/s wr, 12 op/s 2023-03-09T16:32:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:32:52 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 5683.250016 secs 2023-03-09T16:32:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:32:52 smithi038 ceph-mon[92422]: pgmap v3543: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 172 KiB/s wr, 12 op/s 2023-03-09T16:32:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:32:52 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 5683.250016 secs 2023-03-09T16:32:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6145.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:32:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:32:54 smithi122 ceph-mon[93869]: pgmap v3544: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 124 KiB/s wr, 9 op/s 2023-03-09T16:32:54.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:32:54 smithi181 ceph-mon[94189]: pgmap v3544: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 124 KiB/s wr, 9 op/s 2023-03-09T16:32:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:32:54 smithi038 ceph-mon[92422]: pgmap v3544: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 124 KiB/s wr, 9 op/s 2023-03-09T16:32:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:32:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5686.013452 secs 2023-03-09T16:32:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:32:55 smithi122 ceph-mon[93869]: pgmap v3545: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 124 KiB/s wr, 8 op/s 2023-03-09T16:32:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:32:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5686.013452 secs 2023-03-09T16:32:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:32:55 smithi181 ceph-mon[94189]: pgmap v3545: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 124 KiB/s wr, 8 op/s 2023-03-09T16:32:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:32:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5686.013452 secs 2023-03-09T16:32:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:32:55 smithi038 ceph-mon[92422]: pgmap v3545: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 124 KiB/s wr, 8 op/s 2023-03-09T16:32:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:32:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5688.250123 secs 2023-03-09T16:32:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:32:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5688.250123 secs 2023-03-09T16:32:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:32:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5688.250123 secs 2023-03-09T16:32:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:32:57 smithi038 ceph-mon[92422]: pgmap v3546: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 272 KiB/s wr, 13 op/s 2023-03-09T16:32:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:32:57 smithi122 ceph-mon[93869]: pgmap v3546: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 272 KiB/s wr, 13 op/s 2023-03-09T16:32:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:32:57 smithi181 ceph-mon[94189]: pgmap v3546: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 272 KiB/s wr, 13 op/s 2023-03-09T16:32:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6150.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:33:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:33:00 smithi122 ceph-mon[93869]: pgmap v3547: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 164 KiB/s wr, 9 op/s 2023-03-09T16:33:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:33:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:33:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:33:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:33:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:33:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5691.013513 secs 2023-03-09T16:33:00.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:33:00 smithi181 ceph-mon[94189]: pgmap v3547: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 164 KiB/s wr, 9 op/s 2023-03-09T16:33:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:33:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:33:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:33:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:33:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:33:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5691.013513 secs 2023-03-09T16:33:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:33:00 smithi038 ceph-mon[92422]: pgmap v3547: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 164 KiB/s wr, 9 op/s 2023-03-09T16:33:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:33:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:33:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:33:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:33:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:33:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5691.013513 secs 2023-03-09T16:33:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:33:02 smithi122 ceph-mon[93869]: pgmap v3548: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 210 KiB/s wr, 14 op/s 2023-03-09T16:33:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:33:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5693.250248 secs 2023-03-09T16:33:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:33:02 smithi181 ceph-mon[94189]: pgmap v3548: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 210 KiB/s wr, 14 op/s 2023-03-09T16:33:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:33:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5693.250248 secs 2023-03-09T16:33:02.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:33:02 smithi038 ceph-mon[92422]: pgmap v3548: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 210 KiB/s wr, 14 op/s 2023-03-09T16:33:02.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:33:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5693.250248 secs 2023-03-09T16:33:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6155.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:33:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:33:04 smithi122 ceph-mon[93869]: pgmap v3549: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 217 KiB/s wr, 23 op/s 2023-03-09T16:33:04.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:33:04 smithi181 ceph-mon[94189]: pgmap v3549: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 217 KiB/s wr, 23 op/s 2023-03-09T16:33:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:33:04 smithi038 ceph-mon[92422]: pgmap v3549: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 217 KiB/s wr, 23 op/s 2023-03-09T16:33:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:33:05 smithi122 ceph-mon[93869]: osdmap e434: 12 total, 12 up, 12 in 2023-03-09T16:33:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:33:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5696.013575 secs 2023-03-09T16:33:05.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:33:05 smithi181 ceph-mon[94189]: osdmap e434: 12 total, 12 up, 12 in 2023-03-09T16:33:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:33:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5696.013575 secs 2023-03-09T16:33:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:33:05 smithi038 ceph-mon[92422]: osdmap e434: 12 total, 12 up, 12 in 2023-03-09T16:33:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:33:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5696.013575 secs 2023-03-09T16:33:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:33:06 smithi122 ceph-mon[93869]: pgmap v3551: 129 pgs: 2 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 115 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 260 KiB/s wr, 28 op/s 2023-03-09T16:33:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:33:06 smithi181 ceph-mon[94189]: pgmap v3551: 129 pgs: 2 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 115 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 260 KiB/s wr, 28 op/s 2023-03-09T16:33:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:33:06 smithi038 ceph-mon[92422]: pgmap v3551: 129 pgs: 2 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 115 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 260 KiB/s wr, 28 op/s 2023-03-09T16:33:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:33:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5698.250345 secs 2023-03-09T16:33:07.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:33:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5698.250345 secs 2023-03-09T16:33:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:33:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5698.250345 secs 2023-03-09T16:33:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:33:08 smithi122 ceph-mon[93869]: pgmap v3552: 129 pgs: 2 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 115 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 182 KiB/s wr, 27 op/s 2023-03-09T16:33:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:33:08 smithi181 ceph-mon[94189]: pgmap v3552: 129 pgs: 2 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 115 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 182 KiB/s wr, 27 op/s 2023-03-09T16:33:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:33:08 smithi038 ceph-mon[92422]: pgmap v3552: 129 pgs: 2 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 115 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 182 KiB/s wr, 27 op/s 2023-03-09T16:33:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6160.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:33:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:33:10 smithi122 ceph-mon[93869]: pgmap v3553: 129 pgs: 2 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 115 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 182 KiB/s wr, 27 op/s 2023-03-09T16:33:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:33:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5701.013635 secs 2023-03-09T16:33:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:33:10 smithi122 ceph-mon[93869]: osdmap e435: 12 total, 12 up, 12 in 2023-03-09T16:33:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:33:10 smithi181 ceph-mon[94189]: pgmap v3553: 129 pgs: 2 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 115 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 182 KiB/s wr, 27 op/s 2023-03-09T16:33:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:33:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5701.013635 secs 2023-03-09T16:33:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:33:10 smithi181 ceph-mon[94189]: osdmap e435: 12 total, 12 up, 12 in 2023-03-09T16:33:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:33:10 smithi038 ceph-mon[92422]: pgmap v3553: 129 pgs: 2 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 115 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 182 KiB/s wr, 27 op/s 2023-03-09T16:33:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:33:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5701.013635 secs 2023-03-09T16:33:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:33:10 smithi038 ceph-mon[92422]: osdmap e435: 12 total, 12 up, 12 in 2023-03-09T16:33:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:33:12 smithi122 ceph-mon[93869]: pgmap v3555: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 162 KiB/s wr, 11 op/s 2023-03-09T16:33:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:33:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5703.250462 secs 2023-03-09T16:33:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:33:12 smithi181 ceph-mon[94189]: pgmap v3555: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 162 KiB/s wr, 11 op/s 2023-03-09T16:33:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:33:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5703.250462 secs 2023-03-09T16:33:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:33:12 smithi038 ceph-mon[92422]: pgmap v3555: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 162 KiB/s wr, 11 op/s 2023-03-09T16:33:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:33:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5703.250462 secs 2023-03-09T16:33:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6165.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:33:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:33:14 smithi122 ceph-mon[93869]: pgmap v3556: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 145 KiB/s wr, 10 op/s 2023-03-09T16:33:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:33:14 smithi181 ceph-mon[94189]: pgmap v3556: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 145 KiB/s wr, 10 op/s 2023-03-09T16:33:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:33:14 smithi038 ceph-mon[92422]: pgmap v3556: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 145 KiB/s wr, 10 op/s 2023-03-09T16:33:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:33:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5706.013695 secs 2023-03-09T16:33:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:33:15 smithi122 ceph-mon[93869]: osdmap e436: 12 total, 12 up, 12 in 2023-03-09T16:33:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:33:15 smithi122 ceph-mon[93869]: pgmap v3558: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 37 KiB/s wr, 5 op/s 2023-03-09T16:33:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:33:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5706.013695 secs 2023-03-09T16:33:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:33:15 smithi181 ceph-mon[94189]: osdmap e436: 12 total, 12 up, 12 in 2023-03-09T16:33:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:33:15 smithi181 ceph-mon[94189]: pgmap v3558: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 37 KiB/s wr, 5 op/s 2023-03-09T16:33:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:33:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5706.013695 secs 2023-03-09T16:33:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:33:15 smithi038 ceph-mon[92422]: osdmap e436: 12 total, 12 up, 12 in 2023-03-09T16:33:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:33:15 smithi038 ceph-mon[92422]: pgmap v3558: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 37 KiB/s wr, 5 op/s 2023-03-09T16:33:17.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:33:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5708.250603 secs 2023-03-09T16:33:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:33:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5708.250603 secs 2023-03-09T16:33:17.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:33:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5708.250603 secs 2023-03-09T16:33:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:33:17 smithi038 ceph-mon[92422]: pgmap v3559: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 37 KiB/s wr, 9 op/s 2023-03-09T16:33:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:33:17 smithi122 ceph-mon[93869]: pgmap v3559: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 37 KiB/s wr, 9 op/s 2023-03-09T16:33:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:33:17 smithi181 ceph-mon[94189]: pgmap v3559: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 37 KiB/s wr, 9 op/s 2023-03-09T16:33:18.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6170.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:33:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:33:18 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:33:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:33:18 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:33:19.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:33:18 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:33:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:33:20 smithi122 ceph-mon[93869]: pgmap v3560: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 32 KiB/s wr, 8 op/s 2023-03-09T16:33:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:33:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:33:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:33:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:33:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:33:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:33:20.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:33:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5711.013753 secs 2023-03-09T16:33:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:33:20 smithi181 ceph-mon[94189]: pgmap v3560: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 32 KiB/s wr, 8 op/s 2023-03-09T16:33:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:33:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:33:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:33:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:33:20.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:33:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:33:20.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:33:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5711.013753 secs 2023-03-09T16:33:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:33:20 smithi038 ceph-mon[92422]: pgmap v3560: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 32 KiB/s wr, 8 op/s 2023-03-09T16:33:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:33:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:33:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:33:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:33:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:33:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:33:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:33:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5711.013753 secs 2023-03-09T16:33:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:33:22 smithi122 ceph-mon[93869]: pgmap v3561: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 45 KiB/s wr, 7 op/s 2023-03-09T16:33:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:33:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5713.250751 secs 2023-03-09T16:33:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:33:22 smithi181 ceph-mon[94189]: pgmap v3561: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 45 KiB/s wr, 7 op/s 2023-03-09T16:33:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:33:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5713.250751 secs 2023-03-09T16:33:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:33:22 smithi038 ceph-mon[92422]: pgmap v3561: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 45 KiB/s wr, 7 op/s 2023-03-09T16:33:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:33:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5713.250751 secs 2023-03-09T16:33:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6175.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:33:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:33:24 smithi122 ceph-mon[93869]: pgmap v3562: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 46 KiB/s wr, 8 op/s 2023-03-09T16:33:24.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:33:24 smithi181 ceph-mon[94189]: pgmap v3562: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 46 KiB/s wr, 8 op/s 2023-03-09T16:33:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:33:24 smithi038 ceph-mon[92422]: pgmap v3562: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 46 KiB/s wr, 8 op/s 2023-03-09T16:33:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:33:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5716.013837 secs 2023-03-09T16:33:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:33:25 smithi122 ceph-mon[93869]: pgmap v3563: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T16:33:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:33:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5716.013837 secs 2023-03-09T16:33:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:33:25 smithi181 ceph-mon[94189]: pgmap v3563: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T16:33:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:33:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5716.013837 secs 2023-03-09T16:33:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:33:25 smithi038 ceph-mon[92422]: pgmap v3563: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T16:33:27.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:33:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5718.250866 secs 2023-03-09T16:33:27.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:33:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5718.250866 secs 2023-03-09T16:33:27.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:33:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5718.250866 secs 2023-03-09T16:33:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:33:27 smithi038 ceph-mon[92422]: pgmap v3564: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 38 KiB/s wr, 9 op/s 2023-03-09T16:33:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:33:27 smithi122 ceph-mon[93869]: pgmap v3564: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 38 KiB/s wr, 9 op/s 2023-03-09T16:33:28.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:33:27 smithi181 ceph-mon[94189]: pgmap v3564: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 38 KiB/s wr, 9 op/s 2023-03-09T16:33:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6180.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:33:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:33:30 smithi122 ceph-mon[93869]: pgmap v3565: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 38 KiB/s wr, 6 op/s 2023-03-09T16:33:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:33:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5721.013897 secs 2023-03-09T16:33:30.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:33:30 smithi181 ceph-mon[94189]: pgmap v3565: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 38 KiB/s wr, 6 op/s 2023-03-09T16:33:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:33:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5721.013897 secs 2023-03-09T16:33:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:33:30 smithi038 ceph-mon[92422]: pgmap v3565: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 38 KiB/s wr, 6 op/s 2023-03-09T16:33:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:33:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5721.013897 secs 2023-03-09T16:33:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:33:32 smithi122 ceph-mon[93869]: pgmap v3566: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 150 KiB/s wr, 10 op/s 2023-03-09T16:33:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:33:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5723.250988 secs 2023-03-09T16:33:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:33:32 smithi181 ceph-mon[94189]: pgmap v3566: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 150 KiB/s wr, 10 op/s 2023-03-09T16:33:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:33:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5723.250988 secs 2023-03-09T16:33:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:33:32 smithi038 ceph-mon[92422]: pgmap v3566: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 150 KiB/s wr, 10 op/s 2023-03-09T16:33:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:33:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5723.250988 secs 2023-03-09T16:33:33.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6185.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:33:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:33:34 smithi122 ceph-mon[93869]: pgmap v3567: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 112 KiB/s wr, 7 op/s 2023-03-09T16:33:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:33:34 smithi181 ceph-mon[94189]: pgmap v3567: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 112 KiB/s wr, 7 op/s 2023-03-09T16:33:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:33:34 smithi038 ceph-mon[92422]: pgmap v3567: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 112 KiB/s wr, 7 op/s 2023-03-09T16:33:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:33:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5726.013958 secs 2023-03-09T16:33:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:33:35 smithi122 ceph-mon[93869]: pgmap v3568: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 112 KiB/s wr, 7 op/s 2023-03-09T16:33:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:33:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5726.013958 secs 2023-03-09T16:33:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:33:35 smithi181 ceph-mon[94189]: pgmap v3568: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 112 KiB/s wr, 7 op/s 2023-03-09T16:33:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:33:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5726.013958 secs 2023-03-09T16:33:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:33:35 smithi038 ceph-mon[92422]: pgmap v3568: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 112 KiB/s wr, 7 op/s 2023-03-09T16:33:37.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:33:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5728.251139 secs 2023-03-09T16:33:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:33:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5728.251139 secs 2023-03-09T16:33:37.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:33:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5728.251139 secs 2023-03-09T16:33:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:33:37 smithi038 ceph-mon[92422]: pgmap v3569: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 205 KiB/s wr, 12 op/s 2023-03-09T16:33:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:33:37 smithi122 ceph-mon[93869]: pgmap v3569: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 205 KiB/s wr, 12 op/s 2023-03-09T16:33:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:33:37 smithi181 ceph-mon[94189]: pgmap v3569: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 205 KiB/s wr, 12 op/s 2023-03-09T16:33:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6190.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:33:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:33:40 smithi122 ceph-mon[93869]: pgmap v3570: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 205 KiB/s wr, 10 op/s 2023-03-09T16:33:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:33:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5731.014020 secs 2023-03-09T16:33:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:33:40 smithi181 ceph-mon[94189]: pgmap v3570: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 205 KiB/s wr, 10 op/s 2023-03-09T16:33:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:33:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5731.014020 secs 2023-03-09T16:33:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:33:40 smithi038 ceph-mon[92422]: pgmap v3570: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 205 KiB/s wr, 10 op/s 2023-03-09T16:33:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:33:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5731.014020 secs 2023-03-09T16:33:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:33:42 smithi122 ceph-mon[93869]: pgmap v3571: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 265 KiB/s wr, 14 op/s 2023-03-09T16:33:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:33:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5733.251192 secs 2023-03-09T16:33:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:33:42 smithi181 ceph-mon[94189]: pgmap v3571: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 265 KiB/s wr, 14 op/s 2023-03-09T16:33:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:33:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5733.251192 secs 2023-03-09T16:33:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:33:42 smithi038 ceph-mon[92422]: pgmap v3571: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 265 KiB/s wr, 14 op/s 2023-03-09T16:33:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:33:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5733.251192 secs 2023-03-09T16:33:43.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6195.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:33:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:33:44 smithi122 ceph-mon[93869]: pgmap v3572: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 153 KiB/s wr, 11 op/s 2023-03-09T16:33:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:33:44 smithi181 ceph-mon[94189]: pgmap v3572: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 153 KiB/s wr, 11 op/s 2023-03-09T16:33:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:33:44 smithi038 ceph-mon[92422]: pgmap v3572: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 153 KiB/s wr, 11 op/s 2023-03-09T16:33:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:33:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5736.014080 secs 2023-03-09T16:33:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:33:45 smithi122 ceph-mon[93869]: pgmap v3573: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 153 KiB/s wr, 11 op/s 2023-03-09T16:33:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:33:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5736.014080 secs 2023-03-09T16:33:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:33:45 smithi181 ceph-mon[94189]: pgmap v3573: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 153 KiB/s wr, 11 op/s 2023-03-09T16:33:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:33:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5736.014080 secs 2023-03-09T16:33:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:33:45 smithi038 ceph-mon[92422]: pgmap v3573: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 153 KiB/s wr, 11 op/s 2023-03-09T16:33:47.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:33:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5738.251308 secs 2023-03-09T16:33:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:33:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5738.251308 secs 2023-03-09T16:33:47.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:33:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5738.251308 secs 2023-03-09T16:33:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:33:47 smithi038 ceph-mon[92422]: pgmap v3574: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 277 KiB/s wr, 15 op/s 2023-03-09T16:33:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:33:47 smithi122 ceph-mon[93869]: pgmap v3574: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 277 KiB/s wr, 15 op/s 2023-03-09T16:33:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:33:47 smithi181 ceph-mon[94189]: pgmap v3574: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 277 KiB/s wr, 15 op/s 2023-03-09T16:33:48.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6200.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:33:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:33:50 smithi122 ceph-mon[93869]: pgmap v3575: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 184 KiB/s wr, 10 op/s 2023-03-09T16:33:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:33:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5741.014142 secs 2023-03-09T16:33:50.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:33:50 smithi181 ceph-mon[94189]: pgmap v3575: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 184 KiB/s wr, 10 op/s 2023-03-09T16:33:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:33:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5741.014142 secs 2023-03-09T16:33:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:33:50 smithi038 ceph-mon[92422]: pgmap v3575: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 184 KiB/s wr, 10 op/s 2023-03-09T16:33:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:33:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5741.014142 secs 2023-03-09T16:33:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:33:52 smithi122 ceph-mon[93869]: pgmap v3576: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 221 KiB/s wr, 13 op/s 2023-03-09T16:33:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:33:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5743.251376 secs 2023-03-09T16:33:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:33:52 smithi181 ceph-mon[94189]: pgmap v3576: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 221 KiB/s wr, 13 op/s 2023-03-09T16:33:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:33:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5743.251376 secs 2023-03-09T16:33:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:33:52 smithi038 ceph-mon[92422]: pgmap v3576: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 221 KiB/s wr, 13 op/s 2023-03-09T16:33:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:33:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5743.251376 secs 2023-03-09T16:33:53.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6205.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:33:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:33:54 smithi122 ceph-mon[93869]: pgmap v3577: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 166 KiB/s wr, 9 op/s 2023-03-09T16:33:54.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:33:54 smithi181 ceph-mon[94189]: pgmap v3577: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 166 KiB/s wr, 9 op/s 2023-03-09T16:33:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:33:54 smithi038 ceph-mon[92422]: pgmap v3577: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 166 KiB/s wr, 9 op/s 2023-03-09T16:33:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:33:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5746.014211 secs 2023-03-09T16:33:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:33:55 smithi122 ceph-mon[93869]: pgmap v3578: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 166 KiB/s wr, 8 op/s 2023-03-09T16:33:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:33:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5746.014211 secs 2023-03-09T16:33:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:33:55 smithi181 ceph-mon[94189]: pgmap v3578: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 166 KiB/s wr, 8 op/s 2023-03-09T16:33:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:33:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5746.014211 secs 2023-03-09T16:33:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:33:55 smithi038 ceph-mon[92422]: pgmap v3578: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 166 KiB/s wr, 8 op/s 2023-03-09T16:33:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:33:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5748.251520 secs 2023-03-09T16:33:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:33:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5748.251520 secs 2023-03-09T16:33:57.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:33:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5748.251520 secs 2023-03-09T16:33:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:33:57 smithi038 ceph-mon[92422]: pgmap v3579: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 236 KiB/s wr, 11 op/s 2023-03-09T16:33:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:33:57 smithi122 ceph-mon[93869]: pgmap v3579: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 236 KiB/s wr, 11 op/s 2023-03-09T16:33:58.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:33:57 smithi181 ceph-mon[94189]: pgmap v3579: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 236 KiB/s wr, 11 op/s 2023-03-09T16:33:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6210.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:34:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:34:00 smithi122 ceph-mon[93869]: pgmap v3580: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 112 KiB/s wr, 6 op/s 2023-03-09T16:34:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:34:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5751.014272 secs 2023-03-09T16:34:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:34:00 smithi181 ceph-mon[94189]: pgmap v3580: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 112 KiB/s wr, 6 op/s 2023-03-09T16:34:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:34:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5751.014272 secs 2023-03-09T16:34:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:34:00 smithi038 ceph-mon[92422]: pgmap v3580: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 112 KiB/s wr, 6 op/s 2023-03-09T16:34:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:34:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5751.014272 secs 2023-03-09T16:34:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:34:02 smithi122 ceph-mon[93869]: pgmap v3581: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 134 KiB/s wr, 10 op/s 2023-03-09T16:34:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:34:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5753.251643 secs 2023-03-09T16:34:02.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:34:02 smithi181 ceph-mon[94189]: pgmap v3581: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 134 KiB/s wr, 10 op/s 2023-03-09T16:34:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:34:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5753.251643 secs 2023-03-09T16:34:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:34:02 smithi038 ceph-mon[92422]: pgmap v3581: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 134 KiB/s wr, 10 op/s 2023-03-09T16:34:02.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:34:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5753.251643 secs 2023-03-09T16:34:03.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6215.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:34:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:34:04 smithi122 ceph-mon[93869]: pgmap v3582: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 123 KiB/s wr, 21 op/s 2023-03-09T16:34:04.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:34:04 smithi181 ceph-mon[94189]: pgmap v3582: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 123 KiB/s wr, 21 op/s 2023-03-09T16:34:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:34:04 smithi038 ceph-mon[92422]: pgmap v3582: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 123 KiB/s wr, 21 op/s 2023-03-09T16:34:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:34:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5756.014329 secs 2023-03-09T16:34:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:34:05 smithi122 ceph-mon[93869]: pgmap v3583: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 119 KiB/s wr, 20 op/s 2023-03-09T16:34:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:34:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5756.014329 secs 2023-03-09T16:34:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:34:05 smithi181 ceph-mon[94189]: pgmap v3583: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 119 KiB/s wr, 20 op/s 2023-03-09T16:34:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:34:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5756.014329 secs 2023-03-09T16:34:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:34:05 smithi038 ceph-mon[92422]: pgmap v3583: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 119 KiB/s wr, 20 op/s 2023-03-09T16:34:07.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:34:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5758.251758 secs 2023-03-09T16:34:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:34:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5758.251758 secs 2023-03-09T16:34:07.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:34:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5758.251758 secs 2023-03-09T16:34:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:34:07 smithi038 ceph-mon[92422]: pgmap v3584: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 159 KiB/s wr, 24 op/s 2023-03-09T16:34:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:34:07 smithi122 ceph-mon[93869]: pgmap v3584: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 159 KiB/s wr, 24 op/s 2023-03-09T16:34:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:34:07 smithi181 ceph-mon[94189]: pgmap v3584: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 159 KiB/s wr, 24 op/s 2023-03-09T16:34:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6220.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:34:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:34:10 smithi122 ceph-mon[93869]: osdmap e437: 12 total, 12 up, 12 in 2023-03-09T16:34:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:34:10 smithi122 ceph-mon[93869]: pgmap v3586: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 106 KiB/s wr, 25 op/s 2023-03-09T16:34:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:34:10 smithi122 ceph-mon[93869]: 2 slow requests, 2 included below; oldest blocked for > 5761.014391 secs 2023-03-09T16:34:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:34:10 smithi122 ceph-mon[93869]: slow request 5761.014391 seconds old, received at 2023-03-09T14:58:08.564667+0000: peer_request:mds.2:599 currently acquired locks 2023-03-09T16:34:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:34:10 smithi122 ceph-mon[93869]: slow request 5761.003690 seconds old, received at 2023-03-09T14:58:08.575367+0000: peer_request(client.15375:6773.0 authpin) currently dispatched 2023-03-09T16:34:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:34:10 smithi181 ceph-mon[94189]: osdmap e437: 12 total, 12 up, 12 in 2023-03-09T16:34:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:34:10 smithi181 ceph-mon[94189]: pgmap v3586: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 106 KiB/s wr, 25 op/s 2023-03-09T16:34:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:34:10 smithi181 ceph-mon[94189]: 2 slow requests, 2 included below; oldest blocked for > 5761.014391 secs 2023-03-09T16:34:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:34:10 smithi181 ceph-mon[94189]: slow request 5761.014391 seconds old, received at 2023-03-09T14:58:08.564667+0000: peer_request:mds.2:599 currently acquired locks 2023-03-09T16:34:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:34:10 smithi181 ceph-mon[94189]: slow request 5761.003690 seconds old, received at 2023-03-09T14:58:08.575367+0000: peer_request(client.15375:6773.0 authpin) currently dispatched 2023-03-09T16:34:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:34:10 smithi038 ceph-mon[92422]: osdmap e437: 12 total, 12 up, 12 in 2023-03-09T16:34:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:34:10 smithi038 ceph-mon[92422]: pgmap v3586: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 106 KiB/s wr, 25 op/s 2023-03-09T16:34:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:34:10 smithi038 ceph-mon[92422]: 2 slow requests, 2 included below; oldest blocked for > 5761.014391 secs 2023-03-09T16:34:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:34:10 smithi038 ceph-mon[92422]: slow request 5761.014391 seconds old, received at 2023-03-09T14:58:08.564667+0000: peer_request:mds.2:599 currently acquired locks 2023-03-09T16:34:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:34:10 smithi038 ceph-mon[92422]: slow request 5761.003690 seconds old, received at 2023-03-09T14:58:08.575367+0000: peer_request(client.15375:6773.0 authpin) currently dispatched 2023-03-09T16:34:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:34:12 smithi122 ceph-mon[93869]: pgmap v3587: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 80 KiB/s wr, 24 op/s 2023-03-09T16:34:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:34:12 smithi122 ceph-mon[93869]: 2 slow requests, 2 included below; oldest blocked for > 5763.251885 secs 2023-03-09T16:34:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:34:12 smithi122 ceph-mon[93869]: slow request 5763.251884 seconds old, received at 2023-03-09T14:58:08.560137+0000: client_request(mds.2:599 rename #0x10000000007/0000000100000001000000EB #0x60f/1000000062a caller_uid=0, caller_gid=0{}) currently acquired locks 2023-03-09T16:34:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:34:12 smithi122 ceph-mon[93869]: slow request 5763.237504 seconds old, received at 2023-03-09T14:58:08.574517+0000: client_request(client.15375:6773 unlink #0x10000000007/0000000100000001000000EB 2023-03-09T14:58:08.569481+0000 caller_uid=26, caller_gid=26{26,}) currently requesting remote authpins 2023-03-09T16:34:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:34:12 smithi181 ceph-mon[94189]: pgmap v3587: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 80 KiB/s wr, 24 op/s 2023-03-09T16:34:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:34:12 smithi181 ceph-mon[94189]: 2 slow requests, 2 included below; oldest blocked for > 5763.251885 secs 2023-03-09T16:34:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:34:12 smithi181 ceph-mon[94189]: slow request 5763.251884 seconds old, received at 2023-03-09T14:58:08.560137+0000: client_request(mds.2:599 rename #0x10000000007/0000000100000001000000EB #0x60f/1000000062a caller_uid=0, caller_gid=0{}) currently acquired locks 2023-03-09T16:34:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:34:12 smithi181 ceph-mon[94189]: slow request 5763.237504 seconds old, received at 2023-03-09T14:58:08.574517+0000: client_request(client.15375:6773 unlink #0x10000000007/0000000100000001000000EB 2023-03-09T14:58:08.569481+0000 caller_uid=26, caller_gid=26{26,}) currently requesting remote authpins 2023-03-09T16:34:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:34:12 smithi038 ceph-mon[92422]: pgmap v3587: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 80 KiB/s wr, 24 op/s 2023-03-09T16:34:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:34:12 smithi038 ceph-mon[92422]: 2 slow requests, 2 included below; oldest blocked for > 5763.251885 secs 2023-03-09T16:34:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:34:12 smithi038 ceph-mon[92422]: slow request 5763.251884 seconds old, received at 2023-03-09T14:58:08.560137+0000: client_request(mds.2:599 rename #0x10000000007/0000000100000001000000EB #0x60f/1000000062a caller_uid=0, caller_gid=0{}) currently acquired locks 2023-03-09T16:34:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:34:12 smithi038 ceph-mon[92422]: slow request 5763.237504 seconds old, received at 2023-03-09T14:58:08.574517+0000: client_request(client.15375:6773 unlink #0x10000000007/0000000100000001000000EB 2023-03-09T14:58:08.569481+0000 caller_uid=26, caller_gid=26{26,}) currently requesting remote authpins 2023-03-09T16:34:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6225.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:34:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:34:14 smithi122 ceph-mon[93869]: pgmap v3588: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 49 KiB/s wr, 9 op/s 2023-03-09T16:34:14.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:34:14 smithi181 ceph-mon[94189]: pgmap v3588: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 49 KiB/s wr, 9 op/s 2023-03-09T16:34:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:34:14 smithi038 ceph-mon[92422]: pgmap v3588: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 49 KiB/s wr, 9 op/s 2023-03-09T16:34:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:34:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5766.014452 secs 2023-03-09T16:34:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:34:15 smithi122 ceph-mon[93869]: pgmap v3589: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 49 KiB/s wr, 9 op/s 2023-03-09T16:34:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:34:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5766.014452 secs 2023-03-09T16:34:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:34:15 smithi181 ceph-mon[94189]: pgmap v3589: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 49 KiB/s wr, 9 op/s 2023-03-09T16:34:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:34:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5766.014452 secs 2023-03-09T16:34:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:34:15 smithi038 ceph-mon[92422]: pgmap v3589: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 49 KiB/s wr, 9 op/s 2023-03-09T16:34:17.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:34:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5768.252004 secs 2023-03-09T16:34:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:34:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5768.252004 secs 2023-03-09T16:34:17.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:34:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5768.252004 secs 2023-03-09T16:34:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:34:18 smithi122 ceph-mon[93869]: pgmap v3590: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 46 KiB/s wr, 9 op/s 2023-03-09T16:34:18.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:34:18 smithi181 ceph-mon[94189]: pgmap v3590: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 46 KiB/s wr, 9 op/s 2023-03-09T16:34:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:34:18 smithi038 ceph-mon[92422]: pgmap v3590: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 46 KiB/s wr, 9 op/s 2023-03-09T16:34:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6230.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:34:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:34:20 smithi122 ceph-mon[93869]: pgmap v3591: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 46 KiB/s wr, 9 op/s 2023-03-09T16:34:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:34:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:34:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:34:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5771.014532 secs 2023-03-09T16:34:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:34:20 smithi122 ceph-mon[93869]: osdmap e438: 12 total, 12 up, 12 in 2023-03-09T16:34:20.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:34:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:34:20.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:34:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:34:20.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:34:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:34:20.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:34:20 smithi181 ceph-mon[94189]: pgmap v3591: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 46 KiB/s wr, 9 op/s 2023-03-09T16:34:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:34:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:34:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:34:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5771.014532 secs 2023-03-09T16:34:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:34:20 smithi181 ceph-mon[94189]: osdmap e438: 12 total, 12 up, 12 in 2023-03-09T16:34:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:34:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:34:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:34:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:34:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:34:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:34:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:34:20 smithi038 ceph-mon[92422]: pgmap v3591: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 46 KiB/s wr, 9 op/s 2023-03-09T16:34:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:34:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:34:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:34:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5771.014532 secs 2023-03-09T16:34:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:34:20 smithi038 ceph-mon[92422]: osdmap e438: 12 total, 12 up, 12 in 2023-03-09T16:34:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:34:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:34:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:34:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:34:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:34:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:34:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:34:22 smithi122 ceph-mon[93869]: pgmap v3593: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 46 KiB/s wr, 9 op/s 2023-03-09T16:34:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:34:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5773.252143 secs 2023-03-09T16:34:22.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:34:22 smithi181 ceph-mon[94189]: pgmap v3593: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 46 KiB/s wr, 9 op/s 2023-03-09T16:34:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:34:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5773.252143 secs 2023-03-09T16:34:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:34:22 smithi038 ceph-mon[92422]: pgmap v3593: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 46 KiB/s wr, 9 op/s 2023-03-09T16:34:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:34:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5773.252143 secs 2023-03-09T16:34:23.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6235.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:34:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:34:24 smithi122 ceph-mon[93869]: pgmap v3594: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 46 KiB/s wr, 8 op/s 2023-03-09T16:34:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:34:24 smithi181 ceph-mon[94189]: pgmap v3594: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 46 KiB/s wr, 8 op/s 2023-03-09T16:34:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:34:24 smithi038 ceph-mon[92422]: pgmap v3594: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 46 KiB/s wr, 8 op/s 2023-03-09T16:34:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:34:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5776.014601 secs 2023-03-09T16:34:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:34:25 smithi122 ceph-mon[93869]: pgmap v3595: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 46 KiB/s wr, 8 op/s 2023-03-09T16:34:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:34:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5776.014601 secs 2023-03-09T16:34:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:34:25 smithi181 ceph-mon[94189]: pgmap v3595: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 46 KiB/s wr, 8 op/s 2023-03-09T16:34:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:34:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5776.014601 secs 2023-03-09T16:34:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:34:25 smithi038 ceph-mon[92422]: pgmap v3595: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 46 KiB/s wr, 8 op/s 2023-03-09T16:34:27.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:34:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5778.252263 secs 2023-03-09T16:34:27.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:34:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5778.252263 secs 2023-03-09T16:34:27.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:34:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5778.252263 secs 2023-03-09T16:34:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:34:28 smithi122 ceph-mon[93869]: pgmap v3596: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 82 KiB/s wr, 7 op/s 2023-03-09T16:34:28.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:34:28 smithi181 ceph-mon[94189]: pgmap v3596: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 82 KiB/s wr, 7 op/s 2023-03-09T16:34:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:34:28 smithi038 ceph-mon[92422]: pgmap v3596: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 82 KiB/s wr, 7 op/s 2023-03-09T16:34:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6240.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:34:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:34:30 smithi122 ceph-mon[93869]: pgmap v3597: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 82 KiB/s wr, 7 op/s 2023-03-09T16:34:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:34:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5781.014661 secs 2023-03-09T16:34:30.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:34:30 smithi181 ceph-mon[94189]: pgmap v3597: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 82 KiB/s wr, 7 op/s 2023-03-09T16:34:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:34:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5781.014661 secs 2023-03-09T16:34:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:34:30 smithi038 ceph-mon[92422]: pgmap v3597: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 82 KiB/s wr, 7 op/s 2023-03-09T16:34:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:34:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5781.014661 secs 2023-03-09T16:34:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:34:32 smithi122 ceph-mon[93869]: pgmap v3598: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 112 KiB/s wr, 9 op/s 2023-03-09T16:34:32.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:34:32 smithi181 ceph-mon[94189]: pgmap v3598: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 112 KiB/s wr, 9 op/s 2023-03-09T16:34:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:34:32 smithi038 ceph-mon[92422]: pgmap v3598: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 112 KiB/s wr, 9 op/s 2023-03-09T16:34:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6245.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:34:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:34:34 smithi122 ceph-mon[93869]: pgmap v3599: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 107 KiB/s wr, 7 op/s 2023-03-09T16:34:34.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:34:34 smithi181 ceph-mon[94189]: pgmap v3599: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 107 KiB/s wr, 7 op/s 2023-03-09T16:34:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:34:34 smithi038 ceph-mon[92422]: pgmap v3599: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 107 KiB/s wr, 7 op/s 2023-03-09T16:34:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:34:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5786.014721 secs 2023-03-09T16:34:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:34:35 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:34:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:34:35 smithi122 ceph-mon[93869]: pgmap v3600: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 107 KiB/s wr, 6 op/s 2023-03-09T16:34:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:34:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5786.014721 secs 2023-03-09T16:34:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:34:35 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:34:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:34:35 smithi181 ceph-mon[94189]: pgmap v3600: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 107 KiB/s wr, 6 op/s 2023-03-09T16:34:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:34:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5786.014721 secs 2023-03-09T16:34:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:34:35 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:34:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:34:35 smithi038 ceph-mon[92422]: pgmap v3600: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 107 KiB/s wr, 6 op/s 2023-03-09T16:34:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:34:36 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:34:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:34:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:34:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:34:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:34:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:34:36 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:34:36.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:34:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:34:36.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:34:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:34:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:34:36 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:34:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:34:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:34:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:34:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:34:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:34:37 smithi122 ceph-mon[93869]: pgmap v3601: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 151 KiB/s wr, 10 op/s 2023-03-09T16:34:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:34:37 smithi181 ceph-mon[94189]: pgmap v3601: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 151 KiB/s wr, 10 op/s 2023-03-09T16:34:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:34:37 smithi038 ceph-mon[92422]: pgmap v3601: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 151 KiB/s wr, 10 op/s 2023-03-09T16:34:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6250.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:34:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:34:40 smithi122 ceph-mon[93869]: pgmap v3602: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 82 KiB/s wr, 7 op/s 2023-03-09T16:34:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:34:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5791.014786 secs 2023-03-09T16:34:40.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:34:40 smithi181 ceph-mon[94189]: pgmap v3602: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 82 KiB/s wr, 7 op/s 2023-03-09T16:34:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:34:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5791.014786 secs 2023-03-09T16:34:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:34:40 smithi038 ceph-mon[92422]: pgmap v3602: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 82 KiB/s wr, 7 op/s 2023-03-09T16:34:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:34:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5791.014786 secs 2023-03-09T16:34:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:34:42 smithi122 ceph-mon[93869]: pgmap v3603: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 144 KiB/s wr, 10 op/s 2023-03-09T16:34:42.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:34:42 smithi181 ceph-mon[94189]: pgmap v3603: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 144 KiB/s wr, 10 op/s 2023-03-09T16:34:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:34:42 smithi038 ceph-mon[92422]: pgmap v3603: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 144 KiB/s wr, 10 op/s 2023-03-09T16:34:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6255.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:34:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:34:44 smithi122 ceph-mon[93869]: pgmap v3604: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 116 KiB/s wr, 9 op/s 2023-03-09T16:34:44.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:34:44 smithi181 ceph-mon[94189]: pgmap v3604: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 116 KiB/s wr, 9 op/s 2023-03-09T16:34:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:34:44 smithi038 ceph-mon[92422]: pgmap v3604: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 116 KiB/s wr, 9 op/s 2023-03-09T16:34:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:34:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5796.014847 secs 2023-03-09T16:34:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:34:45 smithi122 ceph-mon[93869]: pgmap v3605: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 115 KiB/s wr, 8 op/s 2023-03-09T16:34:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:34:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5796.014847 secs 2023-03-09T16:34:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:34:45 smithi181 ceph-mon[94189]: pgmap v3605: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 115 KiB/s wr, 8 op/s 2023-03-09T16:34:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:34:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5796.014847 secs 2023-03-09T16:34:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:34:45 smithi038 ceph-mon[92422]: pgmap v3605: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 115 KiB/s wr, 8 op/s 2023-03-09T16:34:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:34:48 smithi122 ceph-mon[93869]: pgmap v3606: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 216 KiB/s wr, 13 op/s 2023-03-09T16:34:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:34:48 smithi181 ceph-mon[94189]: pgmap v3606: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 216 KiB/s wr, 13 op/s 2023-03-09T16:34:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:34:48 smithi038 ceph-mon[92422]: pgmap v3606: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 216 KiB/s wr, 13 op/s 2023-03-09T16:34:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6260.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:34:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:34:50 smithi122 ceph-mon[93869]: pgmap v3607: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 172 KiB/s wr, 10 op/s 2023-03-09T16:34:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:34:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5801.014932 secs 2023-03-09T16:34:50.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:34:50 smithi181 ceph-mon[94189]: pgmap v3607: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 172 KiB/s wr, 10 op/s 2023-03-09T16:34:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:34:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5801.014932 secs 2023-03-09T16:34:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:34:50 smithi038 ceph-mon[92422]: pgmap v3607: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 172 KiB/s wr, 10 op/s 2023-03-09T16:34:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:34:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5801.014932 secs 2023-03-09T16:34:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:34:52 smithi122 ceph-mon[93869]: pgmap v3608: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 259 KiB/s wr, 13 op/s 2023-03-09T16:34:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:34:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5803.252829 secs 2023-03-09T16:34:52.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:34:52 smithi181 ceph-mon[94189]: pgmap v3608: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 259 KiB/s wr, 13 op/s 2023-03-09T16:34:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:34:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5803.252829 secs 2023-03-09T16:34:52.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:34:52 smithi038 ceph-mon[92422]: pgmap v3608: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 259 KiB/s wr, 13 op/s 2023-03-09T16:34:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:34:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5803.252829 secs 2023-03-09T16:34:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6265.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:34:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:34:54 smithi122 ceph-mon[93869]: pgmap v3609: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 203 KiB/s wr, 11 op/s 2023-03-09T16:34:54.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:34:54 smithi181 ceph-mon[94189]: pgmap v3609: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 203 KiB/s wr, 11 op/s 2023-03-09T16:34:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:34:54 smithi038 ceph-mon[92422]: pgmap v3609: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 203 KiB/s wr, 11 op/s 2023-03-09T16:34:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:34:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5806.014984 secs 2023-03-09T16:34:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:34:55 smithi122 ceph-mon[93869]: pgmap v3610: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 193 KiB/s wr, 9 op/s 2023-03-09T16:34:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:34:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5806.014984 secs 2023-03-09T16:34:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:34:55 smithi181 ceph-mon[94189]: pgmap v3610: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 193 KiB/s wr, 9 op/s 2023-03-09T16:34:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:34:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5806.014984 secs 2023-03-09T16:34:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:34:55 smithi038 ceph-mon[92422]: pgmap v3610: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 193 KiB/s wr, 9 op/s 2023-03-09T16:34:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:34:56 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:34:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:34:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:34:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:34:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:34:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:34:56 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:34:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:34:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:34:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:34:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:34:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:34:56 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:34:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:34:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:34:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:34:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:34:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:34:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5808.252948 secs 2023-03-09T16:34:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:34:57 smithi122 ceph-mon[93869]: pgmap v3611: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 221 KiB/s wr, 13 op/s 2023-03-09T16:34:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:34:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5808.252948 secs 2023-03-09T16:34:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:34:57 smithi181 ceph-mon[94189]: pgmap v3611: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 221 KiB/s wr, 13 op/s 2023-03-09T16:34:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:34:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5808.252948 secs 2023-03-09T16:34:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:34:57 smithi038 ceph-mon[92422]: pgmap v3611: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 221 KiB/s wr, 13 op/s 2023-03-09T16:34:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6270.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:35:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:35:00 smithi122 ceph-mon[93869]: pgmap v3612: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 121 KiB/s wr, 8 op/s 2023-03-09T16:35:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:35:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5811.015045 secs 2023-03-09T16:35:00.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:35:00 smithi181 ceph-mon[94189]: pgmap v3612: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 121 KiB/s wr, 8 op/s 2023-03-09T16:35:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:35:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5811.015045 secs 2023-03-09T16:35:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:35:00 smithi038 ceph-mon[92422]: pgmap v3612: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 121 KiB/s wr, 8 op/s 2023-03-09T16:35:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:35:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5811.015045 secs 2023-03-09T16:35:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:35:02 smithi122 ceph-mon[93869]: pgmap v3613: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 194 KiB/s wr, 13 op/s 2023-03-09T16:35:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:35:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5813.253059 secs 2023-03-09T16:35:02.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:35:02 smithi181 ceph-mon[94189]: pgmap v3613: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 194 KiB/s wr, 13 op/s 2023-03-09T16:35:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:35:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5813.253059 secs 2023-03-09T16:35:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:35:02 smithi038 ceph-mon[92422]: pgmap v3613: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 194 KiB/s wr, 13 op/s 2023-03-09T16:35:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:35:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5813.253059 secs 2023-03-09T16:35:03.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6275.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:35:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:35:04 smithi122 ceph-mon[93869]: pgmap v3614: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 133 KiB/s wr, 22 op/s 2023-03-09T16:35:04.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:35:04 smithi181 ceph-mon[94189]: pgmap v3614: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 133 KiB/s wr, 22 op/s 2023-03-09T16:35:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:35:04 smithi038 ceph-mon[92422]: pgmap v3614: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 133 KiB/s wr, 22 op/s 2023-03-09T16:35:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:35:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5816.015103 secs 2023-03-09T16:35:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:35:05 smithi122 ceph-mon[93869]: pgmap v3615: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 128 KiB/s wr, 21 op/s 2023-03-09T16:35:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:35:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5816.015103 secs 2023-03-09T16:35:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:35:05 smithi181 ceph-mon[94189]: pgmap v3615: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 128 KiB/s wr, 21 op/s 2023-03-09T16:35:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:35:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5816.015103 secs 2023-03-09T16:35:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:35:05 smithi038 ceph-mon[92422]: pgmap v3615: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 128 KiB/s wr, 21 op/s 2023-03-09T16:35:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:35:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5818.253180 secs 2023-03-09T16:35:07.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:35:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5818.253180 secs 2023-03-09T16:35:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:35:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5818.253180 secs 2023-03-09T16:35:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:35:08 smithi122 ceph-mon[93869]: pgmap v3616: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 158 KiB/s wr, 24 op/s 2023-03-09T16:35:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:35:08 smithi181 ceph-mon[94189]: pgmap v3616: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 158 KiB/s wr, 24 op/s 2023-03-09T16:35:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:35:08 smithi038 ceph-mon[92422]: pgmap v3616: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 158 KiB/s wr, 24 op/s 2023-03-09T16:35:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6280.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:35:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:35:10 smithi122 ceph-mon[93869]: osdmap e439: 12 total, 12 up, 12 in 2023-03-09T16:35:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:35:10 smithi122 ceph-mon[93869]: pgmap v3618: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 155 KiB/s wr, 25 op/s 2023-03-09T16:35:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:35:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5821.015178 secs 2023-03-09T16:35:10.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:35:10 smithi181 ceph-mon[94189]: osdmap e439: 12 total, 12 up, 12 in 2023-03-09T16:35:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:35:10 smithi181 ceph-mon[94189]: pgmap v3618: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 155 KiB/s wr, 25 op/s 2023-03-09T16:35:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:35:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5821.015178 secs 2023-03-09T16:35:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:35:10 smithi038 ceph-mon[92422]: osdmap e439: 12 total, 12 up, 12 in 2023-03-09T16:35:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:35:10 smithi038 ceph-mon[92422]: pgmap v3618: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 155 KiB/s wr, 25 op/s 2023-03-09T16:35:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:35:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5821.015178 secs 2023-03-09T16:35:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:35:12 smithi122 ceph-mon[93869]: pgmap v3619: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 69 KiB/s wr, 23 op/s 2023-03-09T16:35:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:35:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5823.253322 secs 2023-03-09T16:35:12.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:35:12 smithi181 ceph-mon[94189]: pgmap v3619: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 69 KiB/s wr, 23 op/s 2023-03-09T16:35:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:35:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5823.253322 secs 2023-03-09T16:35:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:35:12 smithi038 ceph-mon[92422]: pgmap v3619: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 69 KiB/s wr, 23 op/s 2023-03-09T16:35:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:35:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5823.253322 secs 2023-03-09T16:35:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6285.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:35:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:35:14 smithi122 ceph-mon[93869]: pgmap v3620: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 38 KiB/s wr, 8 op/s 2023-03-09T16:35:14.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:35:14 smithi181 ceph-mon[94189]: pgmap v3620: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 38 KiB/s wr, 8 op/s 2023-03-09T16:35:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:35:14 smithi038 ceph-mon[92422]: pgmap v3620: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 38 KiB/s wr, 8 op/s 2023-03-09T16:35:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:35:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5826.015237 secs 2023-03-09T16:35:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:35:15 smithi122 ceph-mon[93869]: pgmap v3621: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 38 KiB/s wr, 8 op/s 2023-03-09T16:35:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:35:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5826.015237 secs 2023-03-09T16:35:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:35:15 smithi181 ceph-mon[94189]: pgmap v3621: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 38 KiB/s wr, 8 op/s 2023-03-09T16:35:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:35:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5826.015237 secs 2023-03-09T16:35:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:35:15 smithi038 ceph-mon[92422]: pgmap v3621: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 38 KiB/s wr, 8 op/s 2023-03-09T16:35:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:35:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5828.253442 secs 2023-03-09T16:35:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:35:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5828.253442 secs 2023-03-09T16:35:17.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:35:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5828.253442 secs 2023-03-09T16:35:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:35:18 smithi122 ceph-mon[93869]: pgmap v3622: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 30 KiB/s wr, 8 op/s 2023-03-09T16:35:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:35:18 smithi181 ceph-mon[94189]: pgmap v3622: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 30 KiB/s wr, 8 op/s 2023-03-09T16:35:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:35:18 smithi038 ceph-mon[92422]: pgmap v3622: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 30 KiB/s wr, 8 op/s 2023-03-09T16:35:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6290.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:35:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:35:20 smithi122 ceph-mon[93869]: pgmap v3623: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 30 KiB/s wr, 7 op/s 2023-03-09T16:35:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:35:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5831.015298 secs 2023-03-09T16:35:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:35:20 smithi122 ceph-mon[93869]: osdmap e440: 12 total, 12 up, 12 in 2023-03-09T16:35:20.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:35:20 smithi181 ceph-mon[94189]: pgmap v3623: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 30 KiB/s wr, 7 op/s 2023-03-09T16:35:20.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:35:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5831.015298 secs 2023-03-09T16:35:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:35:20 smithi181 ceph-mon[94189]: osdmap e440: 12 total, 12 up, 12 in 2023-03-09T16:35:20.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:35:20 smithi038 ceph-mon[92422]: pgmap v3623: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 30 KiB/s wr, 7 op/s 2023-03-09T16:35:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:35:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5831.015298 secs 2023-03-09T16:35:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:35:20 smithi038 ceph-mon[92422]: osdmap e440: 12 total, 12 up, 12 in 2023-03-09T16:35:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:35:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:35:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:35:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:35:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:35:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:35:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:35:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:35:21.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:35:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:35:21.533 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:35:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:35:21.533 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:35:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:35:21.534 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:35:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:35:21.534 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:35:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:35:21.534 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:35:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:35:21.535 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:35:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:35:21.535 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:35:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:35:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:35:22 smithi122 ceph-mon[93869]: pgmap v3625: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 74 KiB/s wr, 7 op/s 2023-03-09T16:35:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:35:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5833.253575 secs 2023-03-09T16:35:22.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:35:22 smithi181 ceph-mon[94189]: pgmap v3625: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 74 KiB/s wr, 7 op/s 2023-03-09T16:35:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:35:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5833.253575 secs 2023-03-09T16:35:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:35:22 smithi038 ceph-mon[92422]: pgmap v3625: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 74 KiB/s wr, 7 op/s 2023-03-09T16:35:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:35:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5833.253575 secs 2023-03-09T16:35:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6295.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:35:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:35:24 smithi122 ceph-mon[93869]: pgmap v3626: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 74 KiB/s wr, 8 op/s 2023-03-09T16:35:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:35:24 smithi181 ceph-mon[94189]: pgmap v3626: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 74 KiB/s wr, 8 op/s 2023-03-09T16:35:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:35:24 smithi038 ceph-mon[92422]: pgmap v3626: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 74 KiB/s wr, 8 op/s 2023-03-09T16:35:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:35:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5836.015359 secs 2023-03-09T16:35:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:35:25 smithi122 ceph-mon[93869]: pgmap v3627: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 74 KiB/s wr, 8 op/s 2023-03-09T16:35:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:35:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5836.015359 secs 2023-03-09T16:35:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:35:25 smithi181 ceph-mon[94189]: pgmap v3627: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 74 KiB/s wr, 8 op/s 2023-03-09T16:35:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:35:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5836.015359 secs 2023-03-09T16:35:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:35:25 smithi038 ceph-mon[92422]: pgmap v3627: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 74 KiB/s wr, 8 op/s 2023-03-09T16:35:27.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:35:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5838.253699 secs 2023-03-09T16:35:27.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:35:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5838.253699 secs 2023-03-09T16:35:27.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:35:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5838.253699 secs 2023-03-09T16:35:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:35:28 smithi122 ceph-mon[93869]: pgmap v3628: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 46 KiB/s wr, 7 op/s 2023-03-09T16:35:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:35:28 smithi181 ceph-mon[94189]: pgmap v3628: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 46 KiB/s wr, 7 op/s 2023-03-09T16:35:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:35:28 smithi038 ceph-mon[92422]: pgmap v3628: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 46 KiB/s wr, 7 op/s 2023-03-09T16:35:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6300.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:35:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:35:30 smithi122 ceph-mon[93869]: pgmap v3629: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 45 KiB/s wr, 7 op/s 2023-03-09T16:35:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:35:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5841.015440 secs 2023-03-09T16:35:30.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:35:30 smithi181 ceph-mon[94189]: pgmap v3629: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 45 KiB/s wr, 7 op/s 2023-03-09T16:35:30.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:35:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5841.015440 secs 2023-03-09T16:35:30.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:35:30 smithi038 ceph-mon[92422]: pgmap v3629: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 45 KiB/s wr, 7 op/s 2023-03-09T16:35:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:35:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5841.015440 secs 2023-03-09T16:35:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:35:32 smithi122 ceph-mon[93869]: pgmap v3630: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 119 KiB/s wr, 9 op/s 2023-03-09T16:35:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:35:32 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 5843.253797 secs 2023-03-09T16:35:32.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:35:32 smithi181 ceph-mon[94189]: pgmap v3630: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 119 KiB/s wr, 9 op/s 2023-03-09T16:35:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:35:32 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 5843.253797 secs 2023-03-09T16:35:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:35:32 smithi038 ceph-mon[92422]: pgmap v3630: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 119 KiB/s wr, 9 op/s 2023-03-09T16:35:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:35:32 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 5843.253797 secs 2023-03-09T16:35:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6305.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:35:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:35:34 smithi122 ceph-mon[93869]: pgmap v3631: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 77 KiB/s wr, 7 op/s 2023-03-09T16:35:34.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:35:34 smithi181 ceph-mon[94189]: pgmap v3631: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 77 KiB/s wr, 7 op/s 2023-03-09T16:35:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:35:34 smithi038 ceph-mon[92422]: pgmap v3631: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 77 KiB/s wr, 7 op/s 2023-03-09T16:35:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:35:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5846.015500 secs 2023-03-09T16:35:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:35:35 smithi122 ceph-mon[93869]: pgmap v3632: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 76 KiB/s wr, 6 op/s 2023-03-09T16:35:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:35:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5846.015500 secs 2023-03-09T16:35:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:35:35 smithi181 ceph-mon[94189]: pgmap v3632: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 76 KiB/s wr, 6 op/s 2023-03-09T16:35:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:35:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5846.015500 secs 2023-03-09T16:35:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:35:35 smithi038 ceph-mon[92422]: pgmap v3632: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 76 KiB/s wr, 6 op/s 2023-03-09T16:35:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:35:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:35:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:35:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:35:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:35:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:35:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:35:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:35:36.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:35:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:35:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:35:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:35:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:35:37 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 5848.253922 secs 2023-03-09T16:35:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:35:37 smithi122 ceph-mon[93869]: pgmap v3633: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 120 KiB/s wr, 10 op/s 2023-03-09T16:35:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:35:37 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 5848.253922 secs 2023-03-09T16:35:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:35:37 smithi181 ceph-mon[94189]: pgmap v3633: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 120 KiB/s wr, 10 op/s 2023-03-09T16:35:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:35:37 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 5848.253922 secs 2023-03-09T16:35:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:35:37 smithi038 ceph-mon[92422]: pgmap v3633: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 120 KiB/s wr, 10 op/s 2023-03-09T16:35:38.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6310.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:35:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:35:40 smithi122 ceph-mon[93869]: pgmap v3634: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 120 KiB/s wr, 7 op/s 2023-03-09T16:35:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:35:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5851.015559 secs 2023-03-09T16:35:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:35:40 smithi181 ceph-mon[94189]: pgmap v3634: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 120 KiB/s wr, 7 op/s 2023-03-09T16:35:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:35:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5851.015559 secs 2023-03-09T16:35:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:35:40 smithi038 ceph-mon[92422]: pgmap v3634: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 120 KiB/s wr, 7 op/s 2023-03-09T16:35:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:35:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5851.015559 secs 2023-03-09T16:35:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:35:42 smithi122 ceph-mon[93869]: pgmap v3635: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 136 KiB/s wr, 11 op/s 2023-03-09T16:35:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:35:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5853.254012 secs 2023-03-09T16:35:42.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:35:42 smithi181 ceph-mon[94189]: pgmap v3635: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 136 KiB/s wr, 11 op/s 2023-03-09T16:35:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:35:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5853.254012 secs 2023-03-09T16:35:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:35:42 smithi038 ceph-mon[92422]: pgmap v3635: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 136 KiB/s wr, 11 op/s 2023-03-09T16:35:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:35:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5853.254012 secs 2023-03-09T16:35:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6315.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:35:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:35:44 smithi122 ceph-mon[93869]: pgmap v3636: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 65 KiB/s wr, 10 op/s 2023-03-09T16:35:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:35:44 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:35:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:35:44 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:35:44.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:35:44 smithi181 ceph-mon[94189]: pgmap v3636: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 65 KiB/s wr, 10 op/s 2023-03-09T16:35:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:35:44 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:35:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:35:44 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:35:44.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:35:44 smithi038 ceph-mon[92422]: pgmap v3636: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 65 KiB/s wr, 10 op/s 2023-03-09T16:35:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:35:44 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:35:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:35:44 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:35:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:35:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5856.015621 secs 2023-03-09T16:35:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:35:45 smithi122 ceph-mon[93869]: pgmap v3637: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 65 KiB/s wr, 9 op/s 2023-03-09T16:35:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:35:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5856.015621 secs 2023-03-09T16:35:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:35:45 smithi181 ceph-mon[94189]: pgmap v3637: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 65 KiB/s wr, 9 op/s 2023-03-09T16:35:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:35:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5856.015621 secs 2023-03-09T16:35:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:35:45 smithi038 ceph-mon[92422]: pgmap v3637: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 65 KiB/s wr, 9 op/s 2023-03-09T16:35:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:35:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5858.254145 secs 2023-03-09T16:35:47.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:35:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5858.254145 secs 2023-03-09T16:35:47.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:35:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5858.254145 secs 2023-03-09T16:35:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:35:48 smithi122 ceph-mon[93869]: pgmap v3638: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 204 KiB/s wr, 14 op/s 2023-03-09T16:35:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:35:48 smithi181 ceph-mon[94189]: pgmap v3638: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 204 KiB/s wr, 14 op/s 2023-03-09T16:35:48.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:35:48 smithi038 ceph-mon[92422]: pgmap v3638: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 204 KiB/s wr, 14 op/s 2023-03-09T16:35:48.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6320.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:35:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:35:50 smithi122 ceph-mon[93869]: pgmap v3639: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 160 KiB/s wr, 10 op/s 2023-03-09T16:35:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:35:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5861.015683 secs 2023-03-09T16:35:50.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:35:50 smithi181 ceph-mon[94189]: pgmap v3639: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 160 KiB/s wr, 10 op/s 2023-03-09T16:35:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:35:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5861.015683 secs 2023-03-09T16:35:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:35:50 smithi038 ceph-mon[92422]: pgmap v3639: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 160 KiB/s wr, 10 op/s 2023-03-09T16:35:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:35:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5861.015683 secs 2023-03-09T16:35:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:35:52 smithi122 ceph-mon[93869]: pgmap v3640: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 176 KiB/s wr, 12 op/s 2023-03-09T16:35:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:35:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5863.254227 secs 2023-03-09T16:35:52.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:35:52 smithi181 ceph-mon[94189]: pgmap v3640: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 176 KiB/s wr, 12 op/s 2023-03-09T16:35:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:35:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5863.254227 secs 2023-03-09T16:35:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:35:52 smithi038 ceph-mon[92422]: pgmap v3640: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 176 KiB/s wr, 12 op/s 2023-03-09T16:35:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:35:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5863.254227 secs 2023-03-09T16:35:53.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6325.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:35:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:35:54 smithi122 ceph-mon[93869]: pgmap v3641: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 166 KiB/s wr, 10 op/s 2023-03-09T16:35:54.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:35:54 smithi181 ceph-mon[94189]: pgmap v3641: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 166 KiB/s wr, 10 op/s 2023-03-09T16:35:54.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:35:54 smithi038 ceph-mon[92422]: pgmap v3641: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 166 KiB/s wr, 10 op/s 2023-03-09T16:35:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:35:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5866.015745 secs 2023-03-09T16:35:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:35:55 smithi122 ceph-mon[93869]: pgmap v3642: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 160 KiB/s wr, 8 op/s 2023-03-09T16:35:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:35:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5866.015745 secs 2023-03-09T16:35:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:35:55 smithi181 ceph-mon[94189]: pgmap v3642: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 160 KiB/s wr, 8 op/s 2023-03-09T16:35:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:35:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5866.015745 secs 2023-03-09T16:35:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:35:55 smithi038 ceph-mon[92422]: pgmap v3642: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 160 KiB/s wr, 8 op/s 2023-03-09T16:35:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:35:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5868.254340 secs 2023-03-09T16:35:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:35:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5868.254340 secs 2023-03-09T16:35:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:35:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5868.254340 secs 2023-03-09T16:35:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:35:58 smithi122 ceph-mon[93869]: pgmap v3643: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 400 KiB/s wr, 13 op/s 2023-03-09T16:35:58.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:35:58 smithi181 ceph-mon[94189]: pgmap v3643: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 400 KiB/s wr, 13 op/s 2023-03-09T16:35:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:35:58 smithi038 ceph-mon[92422]: pgmap v3643: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 400 KiB/s wr, 13 op/s 2023-03-09T16:35:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6330.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:36:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:36:00 smithi122 ceph-mon[93869]: pgmap v3644: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 261 KiB/s wr, 10 op/s 2023-03-09T16:36:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:36:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5871.015807 secs 2023-03-09T16:36:00.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:36:00 smithi181 ceph-mon[94189]: pgmap v3644: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 261 KiB/s wr, 10 op/s 2023-03-09T16:36:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:36:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5871.015807 secs 2023-03-09T16:36:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:36:00 smithi038 ceph-mon[92422]: pgmap v3644: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 261 KiB/s wr, 10 op/s 2023-03-09T16:36:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:36:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5871.015807 secs 2023-03-09T16:36:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:36:02 smithi122 ceph-mon[93869]: pgmap v3645: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 272 KiB/s wr, 13 op/s 2023-03-09T16:36:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:36:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5873.254461 secs 2023-03-09T16:36:02.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:36:02 smithi181 ceph-mon[94189]: pgmap v3645: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 272 KiB/s wr, 13 op/s 2023-03-09T16:36:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:36:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5873.254461 secs 2023-03-09T16:36:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:36:02 smithi038 ceph-mon[92422]: pgmap v3645: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 272 KiB/s wr, 13 op/s 2023-03-09T16:36:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:36:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5873.254461 secs 2023-03-09T16:36:03.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6335.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:36:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:36:04 smithi038 ceph-mon[92422]: pgmap v3646: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 321 KiB/s wr, 25 op/s 2023-03-09T16:36:04.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:36:04 smithi122 ceph-mon[93869]: pgmap v3646: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 321 KiB/s wr, 25 op/s 2023-03-09T16:36:04.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:36:04 smithi181 ceph-mon[94189]: pgmap v3646: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 321 KiB/s wr, 25 op/s 2023-03-09T16:36:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:36:05 smithi038 ceph-mon[92422]: osdmap e441: 12 total, 12 up, 12 in 2023-03-09T16:36:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:36:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5876.015869 secs 2023-03-09T16:36:05.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:36:05 smithi122 ceph-mon[93869]: osdmap e441: 12 total, 12 up, 12 in 2023-03-09T16:36:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:36:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5876.015869 secs 2023-03-09T16:36:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:36:05 smithi181 ceph-mon[94189]: osdmap e441: 12 total, 12 up, 12 in 2023-03-09T16:36:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:36:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5876.015869 secs 2023-03-09T16:36:06.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:36:06 smithi122 ceph-mon[93869]: pgmap v3648: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 357 KiB/s rd, 379 KiB/s wr, 28 op/s 2023-03-09T16:36:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:36:06 smithi181 ceph-mon[94189]: pgmap v3648: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 357 KiB/s rd, 379 KiB/s wr, 28 op/s 2023-03-09T16:36:06.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:36:06 smithi038 ceph-mon[92422]: pgmap v3648: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 357 KiB/s rd, 379 KiB/s wr, 28 op/s 2023-03-09T16:36:07.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:36:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5878.254601 secs 2023-03-09T16:36:07.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:36:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5878.254601 secs 2023-03-09T16:36:07.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:36:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5878.254601 secs 2023-03-09T16:36:08.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:36:08 smithi122 ceph-mon[93869]: pgmap v3649: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 204 KiB/s wr, 27 op/s 2023-03-09T16:36:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:36:08 smithi181 ceph-mon[94189]: pgmap v3649: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 204 KiB/s wr, 27 op/s 2023-03-09T16:36:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:36:08 smithi038 ceph-mon[92422]: pgmap v3649: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 204 KiB/s wr, 27 op/s 2023-03-09T16:36:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6340.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:36:10.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:36:10 smithi122 ceph-mon[93869]: pgmap v3650: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 203 KiB/s wr, 27 op/s 2023-03-09T16:36:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:36:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5881.015939 secs 2023-03-09T16:36:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:36:10 smithi181 ceph-mon[94189]: pgmap v3650: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 203 KiB/s wr, 27 op/s 2023-03-09T16:36:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:36:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5881.015939 secs 2023-03-09T16:36:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:36:10 smithi038 ceph-mon[92422]: pgmap v3650: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 203 KiB/s wr, 27 op/s 2023-03-09T16:36:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:36:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5881.015939 secs 2023-03-09T16:36:12.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:36:12 smithi122 ceph-mon[93869]: pgmap v3651: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 190 KiB/s wr, 24 op/s 2023-03-09T16:36:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:36:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5883.254740 secs 2023-03-09T16:36:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:36:12 smithi181 ceph-mon[94189]: pgmap v3651: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 190 KiB/s wr, 24 op/s 2023-03-09T16:36:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:36:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5883.254740 secs 2023-03-09T16:36:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:36:12 smithi038 ceph-mon[92422]: pgmap v3651: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 190 KiB/s wr, 24 op/s 2023-03-09T16:36:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:36:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5883.254740 secs 2023-03-09T16:36:13.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6345.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:36:14.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:36:14 smithi122 ceph-mon[93869]: pgmap v3652: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 142 KiB/s wr, 9 op/s 2023-03-09T16:36:14.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:36:14 smithi181 ceph-mon[94189]: pgmap v3652: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 142 KiB/s wr, 9 op/s 2023-03-09T16:36:14.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:36:14 smithi038 ceph-mon[92422]: pgmap v3652: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 142 KiB/s wr, 9 op/s 2023-03-09T16:36:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:36:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5886.015984 secs 2023-03-09T16:36:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:36:15 smithi122 ceph-mon[93869]: osdmap e442: 12 total, 12 up, 12 in 2023-03-09T16:36:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:36:15 smithi122 ceph-mon[93869]: pgmap v3654: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 142 KiB/s wr, 9 op/s 2023-03-09T16:36:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:36:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5886.015984 secs 2023-03-09T16:36:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:36:15 smithi181 ceph-mon[94189]: osdmap e442: 12 total, 12 up, 12 in 2023-03-09T16:36:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:36:15 smithi181 ceph-mon[94189]: pgmap v3654: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 142 KiB/s wr, 9 op/s 2023-03-09T16:36:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:36:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5886.015984 secs 2023-03-09T16:36:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:36:15 smithi038 ceph-mon[92422]: osdmap e442: 12 total, 12 up, 12 in 2023-03-09T16:36:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:36:15 smithi038 ceph-mon[92422]: pgmap v3654: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 142 KiB/s wr, 9 op/s 2023-03-09T16:36:17.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:36:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5888.254857 secs 2023-03-09T16:36:17.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:36:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5888.254857 secs 2023-03-09T16:36:17.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:36:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5888.254857 secs 2023-03-09T16:36:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:36:18 smithi122 ceph-mon[93869]: pgmap v3655: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 7 op/s 2023-03-09T16:36:18.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:36:18 smithi181 ceph-mon[94189]: pgmap v3655: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 7 op/s 2023-03-09T16:36:18.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:36:18 smithi038 ceph-mon[92422]: pgmap v3655: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 7 op/s 2023-03-09T16:36:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6350.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:36:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:36:20 smithi122 ceph-mon[93869]: pgmap v3656: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 7 op/s 2023-03-09T16:36:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:36:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5891.016046 secs 2023-03-09T16:36:20.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:36:20 smithi181 ceph-mon[94189]: pgmap v3656: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 7 op/s 2023-03-09T16:36:20.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:36:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5891.016046 secs 2023-03-09T16:36:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:36:20 smithi038 ceph-mon[92422]: pgmap v3656: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 7 op/s 2023-03-09T16:36:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:36:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5891.016046 secs 2023-03-09T16:36:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:36:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:36:21.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:36:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:36:21.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:36:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:36:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:36:22 smithi038 ceph-mon[92422]: pgmap v3657: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 31 KiB/s wr, 7 op/s 2023-03-09T16:36:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:36:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5893.254945 secs 2023-03-09T16:36:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:36:22 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:36:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:36:22 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:36:22.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:36:22 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:36:22.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:36:22 smithi122 ceph-mon[93869]: pgmap v3657: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 31 KiB/s wr, 7 op/s 2023-03-09T16:36:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:36:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5893.254945 secs 2023-03-09T16:36:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:36:22 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:36:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:36:22 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:36:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:36:22 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:36:22.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:36:22 smithi181 ceph-mon[94189]: pgmap v3657: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 31 KiB/s wr, 7 op/s 2023-03-09T16:36:22.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:36:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5893.254945 secs 2023-03-09T16:36:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:36:22 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:36:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:36:22 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:36:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:36:22 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:36:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6355.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:36:24.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:36:24 smithi038 ceph-mon[92422]: pgmap v3658: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 69 KiB/s wr, 8 op/s 2023-03-09T16:36:24.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:36:24 smithi122 ceph-mon[93869]: pgmap v3658: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 69 KiB/s wr, 8 op/s 2023-03-09T16:36:24.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:36:24 smithi181 ceph-mon[94189]: pgmap v3658: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 69 KiB/s wr, 8 op/s 2023-03-09T16:36:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:36:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5896.016106 secs 2023-03-09T16:36:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:36:25 smithi122 ceph-mon[93869]: pgmap v3659: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 66 KiB/s wr, 7 op/s 2023-03-09T16:36:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:36:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5896.016106 secs 2023-03-09T16:36:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:36:25 smithi181 ceph-mon[94189]: pgmap v3659: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 66 KiB/s wr, 7 op/s 2023-03-09T16:36:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:36:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5896.016106 secs 2023-03-09T16:36:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:36:25 smithi038 ceph-mon[92422]: pgmap v3659: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 66 KiB/s wr, 7 op/s 2023-03-09T16:36:27.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:36:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5898.255069 secs 2023-03-09T16:36:27.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:36:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5898.255069 secs 2023-03-09T16:36:27.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:36:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5898.255069 secs 2023-03-09T16:36:28.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:36:28 smithi122 ceph-mon[93869]: pgmap v3660: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 9 op/s 2023-03-09T16:36:28.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:36:28 smithi181 ceph-mon[94189]: pgmap v3660: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 9 op/s 2023-03-09T16:36:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:36:28 smithi038 ceph-mon[92422]: pgmap v3660: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 9 op/s 2023-03-09T16:36:28.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6360.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:36:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:36:30 smithi038 ceph-mon[92422]: pgmap v3661: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 58 KiB/s wr, 7 op/s 2023-03-09T16:36:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:36:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5901.016167 secs 2023-03-09T16:36:30.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:36:30 smithi122 ceph-mon[93869]: pgmap v3661: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 58 KiB/s wr, 7 op/s 2023-03-09T16:36:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:36:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5901.016167 secs 2023-03-09T16:36:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:36:30 smithi181 ceph-mon[94189]: pgmap v3661: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 58 KiB/s wr, 7 op/s 2023-03-09T16:36:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:36:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5901.016167 secs 2023-03-09T16:36:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:36:32 smithi038 ceph-mon[92422]: pgmap v3662: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 57 KiB/s wr, 8 op/s 2023-03-09T16:36:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:36:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5903.255194 secs 2023-03-09T16:36:32.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:36:32 smithi122 ceph-mon[93869]: pgmap v3662: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 57 KiB/s wr, 8 op/s 2023-03-09T16:36:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:36:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5903.255194 secs 2023-03-09T16:36:32.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:36:32 smithi181 ceph-mon[94189]: pgmap v3662: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 57 KiB/s wr, 8 op/s 2023-03-09T16:36:32.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:36:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5903.255194 secs 2023-03-09T16:36:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6365.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:36:34.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:36:34 smithi038 ceph-mon[92422]: pgmap v3663: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 177 KiB/s wr, 9 op/s 2023-03-09T16:36:34.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:36:34 smithi122 ceph-mon[93869]: pgmap v3663: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 177 KiB/s wr, 9 op/s 2023-03-09T16:36:34.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:36:34 smithi181 ceph-mon[94189]: pgmap v3663: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 177 KiB/s wr, 9 op/s 2023-03-09T16:36:35.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:36:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5906.016228 secs 2023-03-09T16:36:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:36:35 smithi122 ceph-mon[93869]: pgmap v3664: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 121 KiB/s wr, 8 op/s 2023-03-09T16:36:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:36:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5906.016228 secs 2023-03-09T16:36:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:36:35 smithi181 ceph-mon[94189]: pgmap v3664: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 121 KiB/s wr, 8 op/s 2023-03-09T16:36:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:36:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5906.016228 secs 2023-03-09T16:36:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:36:35 smithi038 ceph-mon[92422]: pgmap v3664: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 121 KiB/s wr, 8 op/s 2023-03-09T16:36:38.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:36:38 smithi181 ceph-mon[94189]: pgmap v3665: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 126 KiB/s wr, 11 op/s 2023-03-09T16:36:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:36:38 smithi038 ceph-mon[92422]: pgmap v3665: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 126 KiB/s wr, 11 op/s 2023-03-09T16:36:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:36:38 smithi122 ceph-mon[93869]: pgmap v3665: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 126 KiB/s wr, 11 op/s 2023-03-09T16:36:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6370.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:36:39.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:36:39 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:36:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:36:39 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:36:39.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:36:39 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:36:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:36:40 smithi038 ceph-mon[92422]: pgmap v3666: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 218 KiB/s wr, 10 op/s 2023-03-09T16:36:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:36:40 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:36:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:36:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:36:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:36:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:36:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:36:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5911.016288 secs 2023-03-09T16:36:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:36:40 smithi122 ceph-mon[93869]: pgmap v3666: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 218 KiB/s wr, 10 op/s 2023-03-09T16:36:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:36:40 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:36:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:36:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:36:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:36:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:36:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:36:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5911.016288 secs 2023-03-09T16:36:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:36:40 smithi181 ceph-mon[94189]: pgmap v3666: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 218 KiB/s wr, 10 op/s 2023-03-09T16:36:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:36:40 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:36:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:36:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:36:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:36:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:36:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:36:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5911.016288 secs 2023-03-09T16:36:42.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:36:42 smithi038 ceph-mon[92422]: pgmap v3667: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 218 KiB/s wr, 12 op/s 2023-03-09T16:36:42.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:36:42 smithi122 ceph-mon[93869]: pgmap v3667: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 218 KiB/s wr, 12 op/s 2023-03-09T16:36:42.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:36:42 smithi181 ceph-mon[94189]: pgmap v3667: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 218 KiB/s wr, 12 op/s 2023-03-09T16:36:43.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6375.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:36:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:36:44 smithi038 ceph-mon[92422]: pgmap v3668: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 248 KiB/s wr, 12 op/s 2023-03-09T16:36:44.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:36:44 smithi122 ceph-mon[93869]: pgmap v3668: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 248 KiB/s wr, 12 op/s 2023-03-09T16:36:44.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:36:44 smithi181 ceph-mon[94189]: pgmap v3668: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 248 KiB/s wr, 12 op/s 2023-03-09T16:36:45.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:36:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5916.016375 secs 2023-03-09T16:36:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:36:45 smithi122 ceph-mon[93869]: pgmap v3669: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 127 KiB/s wr, 9 op/s 2023-03-09T16:36:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:36:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5916.016375 secs 2023-03-09T16:36:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:36:45 smithi181 ceph-mon[94189]: pgmap v3669: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 127 KiB/s wr, 9 op/s 2023-03-09T16:36:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:36:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5916.016375 secs 2023-03-09T16:36:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:36:45 smithi038 ceph-mon[92422]: pgmap v3669: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 127 KiB/s wr, 9 op/s 2023-03-09T16:36:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:36:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5918.255519 secs 2023-03-09T16:36:47.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:36:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5918.255519 secs 2023-03-09T16:36:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:36:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5918.255519 secs 2023-03-09T16:36:48.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:36:48 smithi122 ceph-mon[93869]: pgmap v3670: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 133 KiB/s wr, 11 op/s 2023-03-09T16:36:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:36:48 smithi181 ceph-mon[94189]: pgmap v3670: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 133 KiB/s wr, 11 op/s 2023-03-09T16:36:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:36:48 smithi038 ceph-mon[92422]: pgmap v3670: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 133 KiB/s wr, 11 op/s 2023-03-09T16:36:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6380.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:36:49.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:36:49 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:36:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:36:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:36:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:36:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:36:49.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:36:49 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:36:49.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:36:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:36:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:36:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:36:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:36:49 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:36:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:36:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:36:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:36:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:36:50.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:36:50 smithi122 ceph-mon[93869]: pgmap v3671: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 243 KiB/s wr, 9 op/s 2023-03-09T16:36:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:36:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5921.016436 secs 2023-03-09T16:36:50.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:36:50 smithi181 ceph-mon[94189]: pgmap v3671: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 243 KiB/s wr, 9 op/s 2023-03-09T16:36:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:36:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5921.016436 secs 2023-03-09T16:36:50.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:36:50 smithi038 ceph-mon[92422]: pgmap v3671: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 243 KiB/s wr, 9 op/s 2023-03-09T16:36:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:36:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5921.016436 secs 2023-03-09T16:36:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:36:52 smithi038 ceph-mon[92422]: pgmap v3672: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 151 KiB/s wr, 10 op/s 2023-03-09T16:36:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:36:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5923.255658 secs 2023-03-09T16:36:52.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:36:52 smithi122 ceph-mon[93869]: pgmap v3672: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 151 KiB/s wr, 10 op/s 2023-03-09T16:36:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:36:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5923.255658 secs 2023-03-09T16:36:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:36:52 smithi181 ceph-mon[94189]: pgmap v3672: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 151 KiB/s wr, 10 op/s 2023-03-09T16:36:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:36:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5923.255658 secs 2023-03-09T16:36:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6385.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:36:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:36:54 smithi038 ceph-mon[92422]: pgmap v3673: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 207 KiB/s wr, 9 op/s 2023-03-09T16:36:54.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:36:54 smithi122 ceph-mon[93869]: pgmap v3673: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 207 KiB/s wr, 9 op/s 2023-03-09T16:36:54.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:36:54 smithi181 ceph-mon[94189]: pgmap v3673: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 207 KiB/s wr, 9 op/s 2023-03-09T16:36:55.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:36:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5926.016496 secs 2023-03-09T16:36:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:36:55 smithi122 ceph-mon[93869]: pgmap v3674: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 178 KiB/s wr, 8 op/s 2023-03-09T16:36:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:36:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5926.016496 secs 2023-03-09T16:36:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:36:55 smithi181 ceph-mon[94189]: pgmap v3674: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 178 KiB/s wr, 8 op/s 2023-03-09T16:36:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:36:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5926.016496 secs 2023-03-09T16:36:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:36:55 smithi038 ceph-mon[92422]: pgmap v3674: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 178 KiB/s wr, 8 op/s 2023-03-09T16:36:57.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:36:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5928.255783 secs 2023-03-09T16:36:57.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:36:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5928.255783 secs 2023-03-09T16:36:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:36:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5928.255783 secs 2023-03-09T16:36:58.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:36:58 smithi122 ceph-mon[93869]: pgmap v3675: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 217 KiB/s wr, 11 op/s 2023-03-09T16:36:58.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:36:58 smithi181 ceph-mon[94189]: pgmap v3675: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 217 KiB/s wr, 11 op/s 2023-03-09T16:36:58.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:36:58 smithi038 ceph-mon[92422]: pgmap v3675: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 217 KiB/s wr, 11 op/s 2023-03-09T16:36:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6390.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:37:00.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:37:00 smithi038 ceph-mon[92422]: pgmap v3676: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 211 KiB/s wr, 8 op/s 2023-03-09T16:37:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:37:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5931.016562 secs 2023-03-09T16:37:00.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:37:00 smithi122 ceph-mon[93869]: pgmap v3676: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 211 KiB/s wr, 8 op/s 2023-03-09T16:37:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:37:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5931.016562 secs 2023-03-09T16:37:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:37:00 smithi181 ceph-mon[94189]: pgmap v3676: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 211 KiB/s wr, 8 op/s 2023-03-09T16:37:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:37:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5931.016562 secs 2023-03-09T16:37:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:37:02 smithi038 ceph-mon[92422]: pgmap v3677: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 125 KiB/s wr, 9 op/s 2023-03-09T16:37:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:37:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5933.255905 secs 2023-03-09T16:37:02.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:37:02 smithi122 ceph-mon[93869]: pgmap v3677: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 125 KiB/s wr, 9 op/s 2023-03-09T16:37:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:37:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5933.255905 secs 2023-03-09T16:37:02.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:37:02 smithi181 ceph-mon[94189]: pgmap v3677: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 125 KiB/s wr, 9 op/s 2023-03-09T16:37:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:37:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5933.255905 secs 2023-03-09T16:37:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6395.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:37:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:37:04 smithi038 ceph-mon[92422]: pgmap v3678: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 151 KiB/s wr, 20 op/s 2023-03-09T16:37:04.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:37:04 smithi122 ceph-mon[93869]: pgmap v3678: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 151 KiB/s wr, 20 op/s 2023-03-09T16:37:04.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:37:04 smithi181 ceph-mon[94189]: pgmap v3678: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 151 KiB/s wr, 20 op/s 2023-03-09T16:37:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:37:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5936.016633 secs 2023-03-09T16:37:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:37:05 smithi122 ceph-mon[93869]: pgmap v3679: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 94 KiB/s wr, 19 op/s 2023-03-09T16:37:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:37:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5936.016633 secs 2023-03-09T16:37:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:37:05 smithi181 ceph-mon[94189]: pgmap v3679: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 94 KiB/s wr, 19 op/s 2023-03-09T16:37:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:37:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5936.016633 secs 2023-03-09T16:37:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:37:05 smithi038 ceph-mon[92422]: pgmap v3679: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 94 KiB/s wr, 19 op/s 2023-03-09T16:37:07.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:37:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5938.256028 secs 2023-03-09T16:37:07.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:37:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5938.256028 secs 2023-03-09T16:37:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:37:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5938.256028 secs 2023-03-09T16:37:08.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:37:08 smithi122 ceph-mon[93869]: pgmap v3680: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 124 KiB/s wr, 25 op/s 2023-03-09T16:37:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:37:08 smithi181 ceph-mon[94189]: pgmap v3680: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 124 KiB/s wr, 25 op/s 2023-03-09T16:37:08.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:37:08 smithi038 ceph-mon[92422]: pgmap v3680: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 124 KiB/s wr, 25 op/s 2023-03-09T16:37:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6400.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:37:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:37:10 smithi122 ceph-mon[93869]: osdmap e443: 12 total, 12 up, 12 in 2023-03-09T16:37:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:37:10 smithi122 ceph-mon[93869]: pgmap v3682: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 102 KiB/s wr, 26 op/s 2023-03-09T16:37:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:37:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5941.016692 secs 2023-03-09T16:37:10.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:37:10 smithi181 ceph-mon[94189]: osdmap e443: 12 total, 12 up, 12 in 2023-03-09T16:37:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:37:10 smithi181 ceph-mon[94189]: pgmap v3682: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 102 KiB/s wr, 26 op/s 2023-03-09T16:37:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:37:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5941.016692 secs 2023-03-09T16:37:10.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:37:10 smithi038 ceph-mon[92422]: osdmap e443: 12 total, 12 up, 12 in 2023-03-09T16:37:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:37:10 smithi038 ceph-mon[92422]: pgmap v3682: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 102 KiB/s wr, 26 op/s 2023-03-09T16:37:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:37:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5941.016692 secs 2023-03-09T16:37:12.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:37:12 smithi038 ceph-mon[92422]: pgmap v3683: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 68 KiB/s wr, 26 op/s 2023-03-09T16:37:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:37:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5943.256148 secs 2023-03-09T16:37:12.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:37:12 smithi122 ceph-mon[93869]: pgmap v3683: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 68 KiB/s wr, 26 op/s 2023-03-09T16:37:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:37:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5943.256148 secs 2023-03-09T16:37:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:37:12 smithi181 ceph-mon[94189]: pgmap v3683: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 68 KiB/s wr, 26 op/s 2023-03-09T16:37:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:37:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5943.256148 secs 2023-03-09T16:37:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6405.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:37:14.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:37:14 smithi122 ceph-mon[93869]: pgmap v3684: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 37 KiB/s wr, 12 op/s 2023-03-09T16:37:14.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:37:14 smithi181 ceph-mon[94189]: pgmap v3684: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 37 KiB/s wr, 12 op/s 2023-03-09T16:37:14.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:37:14 smithi038 ceph-mon[92422]: pgmap v3684: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 37 KiB/s wr, 12 op/s 2023-03-09T16:37:15.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:37:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5946.016752 secs 2023-03-09T16:37:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:37:15 smithi122 ceph-mon[93869]: pgmap v3685: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 37 KiB/s wr, 12 op/s 2023-03-09T16:37:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:37:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5946.016752 secs 2023-03-09T16:37:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:37:15 smithi181 ceph-mon[94189]: pgmap v3685: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 37 KiB/s wr, 12 op/s 2023-03-09T16:37:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:37:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5946.016752 secs 2023-03-09T16:37:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:37:15 smithi038 ceph-mon[92422]: pgmap v3685: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 37 KiB/s wr, 12 op/s 2023-03-09T16:37:17.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:37:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5948.256266 secs 2023-03-09T16:37:17.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:37:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5948.256266 secs 2023-03-09T16:37:17.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:37:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5948.256266 secs 2023-03-09T16:37:18.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:37:18 smithi122 ceph-mon[93869]: pgmap v3686: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 48 KiB/s wr, 9 op/s 2023-03-09T16:37:18.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:37:18 smithi181 ceph-mon[94189]: pgmap v3686: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 48 KiB/s wr, 9 op/s 2023-03-09T16:37:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:37:18 smithi038 ceph-mon[92422]: pgmap v3686: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 48 KiB/s wr, 9 op/s 2023-03-09T16:37:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6410.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:37:20.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:37:20 smithi038 ceph-mon[92422]: pgmap v3687: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 47 KiB/s wr, 9 op/s 2023-03-09T16:37:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:37:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5951.016812 secs 2023-03-09T16:37:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:37:20 smithi038 ceph-mon[92422]: osdmap e444: 12 total, 12 up, 12 in 2023-03-09T16:37:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:37:20 smithi122 ceph-mon[93869]: pgmap v3687: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 47 KiB/s wr, 9 op/s 2023-03-09T16:37:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:37:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5951.016812 secs 2023-03-09T16:37:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:37:20 smithi122 ceph-mon[93869]: osdmap e444: 12 total, 12 up, 12 in 2023-03-09T16:37:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:37:20 smithi181 ceph-mon[94189]: pgmap v3687: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 47 KiB/s wr, 9 op/s 2023-03-09T16:37:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:37:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5951.016812 secs 2023-03-09T16:37:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:37:20 smithi181 ceph-mon[94189]: osdmap e444: 12 total, 12 up, 12 in 2023-03-09T16:37:22.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:37:22 smithi038 ceph-mon[92422]: pgmap v3689: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 47 KiB/s wr, 8 op/s 2023-03-09T16:37:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:37:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5953.256334 secs 2023-03-09T16:37:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:37:22 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:37:22.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:37:22 smithi122 ceph-mon[93869]: pgmap v3689: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 47 KiB/s wr, 8 op/s 2023-03-09T16:37:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:37:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5953.256334 secs 2023-03-09T16:37:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:37:22 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:37:22.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:37:22 smithi181 ceph-mon[94189]: pgmap v3689: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 47 KiB/s wr, 8 op/s 2023-03-09T16:37:22.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:37:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5953.256334 secs 2023-03-09T16:37:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:37:22 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:37:23.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:37:23 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:37:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:37:23 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:37:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:37:23 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:37:23.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:37:23 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:37:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:37:23 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:37:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:37:23 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:37:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:37:23 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:37:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:37:23 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:37:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:37:23 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:37:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6415.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:37:24.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:37:24 smithi122 ceph-mon[93869]: pgmap v3690: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 47 KiB/s wr, 7 op/s 2023-03-09T16:37:24.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:37:24 smithi181 ceph-mon[94189]: pgmap v3690: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 47 KiB/s wr, 7 op/s 2023-03-09T16:37:24.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:37:24 smithi038 ceph-mon[92422]: pgmap v3690: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 47 KiB/s wr, 7 op/s 2023-03-09T16:37:25.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:37:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5956.016874 secs 2023-03-09T16:37:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:37:25 smithi122 ceph-mon[93869]: pgmap v3691: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 47 KiB/s wr, 7 op/s 2023-03-09T16:37:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:37:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5956.016874 secs 2023-03-09T16:37:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:37:25 smithi181 ceph-mon[94189]: pgmap v3691: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 47 KiB/s wr, 7 op/s 2023-03-09T16:37:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:37:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5956.016874 secs 2023-03-09T16:37:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:37:25 smithi038 ceph-mon[92422]: pgmap v3691: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 47 KiB/s wr, 7 op/s 2023-03-09T16:37:27.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:37:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5958.256404 secs 2023-03-09T16:37:27.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:37:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5958.256404 secs 2023-03-09T16:37:27.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:37:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5958.256404 secs 2023-03-09T16:37:28.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:37:28 smithi122 ceph-mon[93869]: pgmap v3692: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 84 KiB/s wr, 6 op/s 2023-03-09T16:37:28.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:37:28 smithi181 ceph-mon[94189]: pgmap v3692: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 84 KiB/s wr, 6 op/s 2023-03-09T16:37:28.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:37:28 smithi038 ceph-mon[92422]: pgmap v3692: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 84 KiB/s wr, 6 op/s 2023-03-09T16:37:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6420.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:37:30.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:37:30 smithi038 ceph-mon[92422]: pgmap v3693: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 84 KiB/s wr, 6 op/s 2023-03-09T16:37:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:37:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5961.016943 secs 2023-03-09T16:37:30.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:37:30 smithi122 ceph-mon[93869]: pgmap v3693: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 84 KiB/s wr, 6 op/s 2023-03-09T16:37:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:37:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5961.016943 secs 2023-03-09T16:37:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:37:30 smithi181 ceph-mon[94189]: pgmap v3693: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 84 KiB/s wr, 6 op/s 2023-03-09T16:37:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:37:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5961.016943 secs 2023-03-09T16:37:32.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:37:32 smithi122 ceph-mon[93869]: pgmap v3694: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 114 KiB/s wr, 8 op/s 2023-03-09T16:37:32.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:37:32 smithi181 ceph-mon[94189]: pgmap v3694: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 114 KiB/s wr, 8 op/s 2023-03-09T16:37:32.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:37:32 smithi038 ceph-mon[92422]: pgmap v3694: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 114 KiB/s wr, 8 op/s 2023-03-09T16:37:33.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6425.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:37:34.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:37:34 smithi122 ceph-mon[93869]: pgmap v3695: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 110 KiB/s wr, 7 op/s 2023-03-09T16:37:34.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:37:34 smithi181 ceph-mon[94189]: pgmap v3695: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 110 KiB/s wr, 7 op/s 2023-03-09T16:37:34.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:37:34 smithi038 ceph-mon[92422]: pgmap v3695: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 110 KiB/s wr, 7 op/s 2023-03-09T16:37:35.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:37:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5966.017001 secs 2023-03-09T16:37:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:37:35 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:37:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:37:35 smithi122 ceph-mon[93869]: pgmap v3696: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 110 KiB/s wr, 6 op/s 2023-03-09T16:37:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:37:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5966.017001 secs 2023-03-09T16:37:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:37:35 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:37:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:37:35 smithi181 ceph-mon[94189]: pgmap v3696: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 110 KiB/s wr, 6 op/s 2023-03-09T16:37:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:37:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5966.017001 secs 2023-03-09T16:37:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:37:35 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:37:36.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:37:35 smithi038 ceph-mon[92422]: pgmap v3696: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 110 KiB/s wr, 6 op/s 2023-03-09T16:37:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:37:36 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:37:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:37:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:37:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:37:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:37:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:37:36 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:37:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:37:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:37:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:37:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:37:37.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:37:36 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:37:37.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:37:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:37:37.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:37:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:37:37.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:37:37 smithi038 ceph-mon[92422]: pgmap v3697: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 155 KiB/s wr, 10 op/s 2023-03-09T16:37:38.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:37:37 smithi122 ceph-mon[93869]: pgmap v3697: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 155 KiB/s wr, 10 op/s 2023-03-09T16:37:38.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:37:37 smithi181 ceph-mon[94189]: pgmap v3697: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 155 KiB/s wr, 10 op/s 2023-03-09T16:37:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6430.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:37:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:37:40 smithi038 ceph-mon[92422]: pgmap v3698: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 85 KiB/s wr, 7 op/s 2023-03-09T16:37:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:37:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5971.017061 secs 2023-03-09T16:37:40.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:37:40 smithi122 ceph-mon[93869]: pgmap v3698: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 85 KiB/s wr, 7 op/s 2023-03-09T16:37:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:37:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5971.017061 secs 2023-03-09T16:37:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:37:40 smithi181 ceph-mon[94189]: pgmap v3698: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 85 KiB/s wr, 7 op/s 2023-03-09T16:37:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:37:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5971.017061 secs 2023-03-09T16:37:42.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:37:42 smithi122 ceph-mon[93869]: pgmap v3699: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 148 KiB/s wr, 10 op/s 2023-03-09T16:37:42.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:37:42 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 5973.256769 secs 2023-03-09T16:37:42.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:37:42 smithi181 ceph-mon[94189]: pgmap v3699: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 148 KiB/s wr, 10 op/s 2023-03-09T16:37:42.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:37:42 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 5973.256769 secs 2023-03-09T16:37:42.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:37:42 smithi038 ceph-mon[92422]: pgmap v3699: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 148 KiB/s wr, 10 op/s 2023-03-09T16:37:42.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:37:42 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 5973.256769 secs 2023-03-09T16:37:43.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:37:43 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:37:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:37:43 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:37:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:37:43 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:37:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:37:43 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:37:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:37:43 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:37:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:37:43 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:37:43.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:37:43 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:37:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:37:43 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:37:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:37:43 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:37:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6435.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:37:44.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:37:44 smithi122 ceph-mon[93869]: pgmap v3700: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 119 KiB/s wr, 9 op/s 2023-03-09T16:37:44.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:37:44 smithi181 ceph-mon[94189]: pgmap v3700: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 119 KiB/s wr, 9 op/s 2023-03-09T16:37:44.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:37:44 smithi038 ceph-mon[92422]: pgmap v3700: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 119 KiB/s wr, 9 op/s 2023-03-09T16:37:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:37:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5976.017151 secs 2023-03-09T16:37:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:37:45 smithi038 ceph-mon[92422]: pgmap v3701: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 119 KiB/s wr, 9 op/s 2023-03-09T16:37:46.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:37:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5976.017151 secs 2023-03-09T16:37:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:37:45 smithi122 ceph-mon[93869]: pgmap v3701: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 119 KiB/s wr, 9 op/s 2023-03-09T16:37:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:37:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5976.017151 secs 2023-03-09T16:37:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:37:45 smithi181 ceph-mon[94189]: pgmap v3701: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 119 KiB/s wr, 9 op/s 2023-03-09T16:37:47.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:37:47 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 5978.256897 secs 2023-03-09T16:37:47.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:37:47 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 5978.256897 secs 2023-03-09T16:37:47.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:37:47 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 5978.256897 secs 2023-03-09T16:37:48.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:37:48 smithi122 ceph-mon[93869]: pgmap v3702: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 221 KiB/s wr, 13 op/s 2023-03-09T16:37:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:37:48 smithi181 ceph-mon[94189]: pgmap v3702: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 221 KiB/s wr, 13 op/s 2023-03-09T16:37:48.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:37:48 smithi038 ceph-mon[92422]: pgmap v3702: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 221 KiB/s wr, 13 op/s 2023-03-09T16:37:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6440.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:37:50.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:37:50 smithi122 ceph-mon[93869]: pgmap v3703: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 176 KiB/s wr, 10 op/s 2023-03-09T16:37:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:37:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5981.017209 secs 2023-03-09T16:37:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:37:50 smithi181 ceph-mon[94189]: pgmap v3703: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 176 KiB/s wr, 10 op/s 2023-03-09T16:37:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:37:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5981.017209 secs 2023-03-09T16:37:50.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:37:50 smithi038 ceph-mon[92422]: pgmap v3703: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 176 KiB/s wr, 10 op/s 2023-03-09T16:37:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:37:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5981.017209 secs 2023-03-09T16:37:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:37:52 smithi122 ceph-mon[93869]: pgmap v3704: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 197 KiB/s wr, 13 op/s 2023-03-09T16:37:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:37:52 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 5983.257017 secs 2023-03-09T16:37:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:37:52 smithi181 ceph-mon[94189]: pgmap v3704: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 197 KiB/s wr, 13 op/s 2023-03-09T16:37:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:37:52 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 5983.257017 secs 2023-03-09T16:37:52.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:37:52 smithi038 ceph-mon[92422]: pgmap v3704: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 197 KiB/s wr, 13 op/s 2023-03-09T16:37:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:37:52 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 5983.257017 secs 2023-03-09T16:37:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6445.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:37:54.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:37:54 smithi122 ceph-mon[93869]: pgmap v3705: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 140 KiB/s wr, 11 op/s 2023-03-09T16:37:54.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:37:54 smithi181 ceph-mon[94189]: pgmap v3705: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 140 KiB/s wr, 11 op/s 2023-03-09T16:37:54.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:37:54 smithi038 ceph-mon[92422]: pgmap v3705: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 140 KiB/s wr, 11 op/s 2023-03-09T16:37:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:37:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5986.017269 secs 2023-03-09T16:37:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:37:55 smithi181 ceph-mon[94189]: pgmap v3706: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 129 KiB/s wr, 9 op/s 2023-03-09T16:37:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:37:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5986.017269 secs 2023-03-09T16:37:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:37:55 smithi038 ceph-mon[92422]: pgmap v3706: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 129 KiB/s wr, 9 op/s 2023-03-09T16:37:56.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:37:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5986.017269 secs 2023-03-09T16:37:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:37:55 smithi122 ceph-mon[93869]: pgmap v3706: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 129 KiB/s wr, 9 op/s 2023-03-09T16:37:57.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:37:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5988.257149 secs 2023-03-09T16:37:57.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:37:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5988.257149 secs 2023-03-09T16:37:57.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:37:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5988.257149 secs 2023-03-09T16:37:58.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:37:58 smithi122 ceph-mon[93869]: pgmap v3707: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 210 KiB/s wr, 13 op/s 2023-03-09T16:37:58.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:37:58 smithi181 ceph-mon[94189]: pgmap v3707: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 210 KiB/s wr, 13 op/s 2023-03-09T16:37:58.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:37:58 smithi038 ceph-mon[92422]: pgmap v3707: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 210 KiB/s wr, 13 op/s 2023-03-09T16:37:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6450.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:38:00.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:38:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:38:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:38:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:38:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:38:00 smithi122 ceph-mon[93869]: pgmap v3708: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 108 KiB/s wr, 9 op/s 2023-03-09T16:38:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:38:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5991.017330 secs 2023-03-09T16:38:00.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:38:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:38:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:38:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:38:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:38:00 smithi181 ceph-mon[94189]: pgmap v3708: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 108 KiB/s wr, 9 op/s 2023-03-09T16:38:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:38:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5991.017330 secs 2023-03-09T16:38:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:38:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:38:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:38:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:38:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:38:00 smithi038 ceph-mon[92422]: pgmap v3708: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 108 KiB/s wr, 9 op/s 2023-03-09T16:38:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:38:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5991.017330 secs 2023-03-09T16:38:02.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:38:02 smithi122 ceph-mon[93869]: pgmap v3709: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 189 KiB/s wr, 12 op/s 2023-03-09T16:38:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:38:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5993.257278 secs 2023-03-09T16:38:02.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:38:02 smithi181 ceph-mon[94189]: pgmap v3709: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 189 KiB/s wr, 12 op/s 2023-03-09T16:38:02.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:38:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5993.257278 secs 2023-03-09T16:38:02.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:38:02 smithi038 ceph-mon[92422]: pgmap v3709: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 189 KiB/s wr, 12 op/s 2023-03-09T16:38:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:38:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5993.257278 secs 2023-03-09T16:38:03.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6455.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:38:04.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:38:04 smithi122 ceph-mon[93869]: pgmap v3710: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 195 KiB/s wr, 21 op/s 2023-03-09T16:38:04.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:38:04 smithi181 ceph-mon[94189]: pgmap v3710: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 195 KiB/s wr, 21 op/s 2023-03-09T16:38:04.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:38:04 smithi038 ceph-mon[92422]: pgmap v3710: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 195 KiB/s wr, 21 op/s 2023-03-09T16:38:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:38:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5996.017390 secs 2023-03-09T16:38:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:38:05 smithi181 ceph-mon[94189]: pgmap v3711: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 190 KiB/s wr, 20 op/s 2023-03-09T16:38:05.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:38:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5996.017390 secs 2023-03-09T16:38:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:38:05 smithi038 ceph-mon[92422]: pgmap v3711: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 190 KiB/s wr, 20 op/s 2023-03-09T16:38:06.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:38:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5996.017390 secs 2023-03-09T16:38:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:38:05 smithi122 ceph-mon[93869]: pgmap v3711: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 190 KiB/s wr, 20 op/s 2023-03-09T16:38:07.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:38:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 5998.257394 secs 2023-03-09T16:38:07.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:38:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 5998.257394 secs 2023-03-09T16:38:07.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:38:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 5998.257394 secs 2023-03-09T16:38:08.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:38:08 smithi122 ceph-mon[93869]: pgmap v3712: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 238 KiB/s wr, 26 op/s 2023-03-09T16:38:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:38:08 smithi181 ceph-mon[94189]: pgmap v3712: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 238 KiB/s wr, 26 op/s 2023-03-09T16:38:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:38:08 smithi038 ceph-mon[92422]: pgmap v3712: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 238 KiB/s wr, 26 op/s 2023-03-09T16:38:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6460.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:38:10.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:38:10 smithi122 ceph-mon[93869]: osdmap e445: 12 total, 12 up, 12 in 2023-03-09T16:38:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:38:10 smithi122 ceph-mon[93869]: pgmap v3714: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 188 KiB/s wr, 26 op/s 2023-03-09T16:38:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:38:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6001.017474 secs 2023-03-09T16:38:10.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:38:10 smithi181 ceph-mon[94189]: osdmap e445: 12 total, 12 up, 12 in 2023-03-09T16:38:10.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:38:10 smithi181 ceph-mon[94189]: pgmap v3714: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 188 KiB/s wr, 26 op/s 2023-03-09T16:38:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:38:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6001.017474 secs 2023-03-09T16:38:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:38:10 smithi038 ceph-mon[92422]: osdmap e445: 12 total, 12 up, 12 in 2023-03-09T16:38:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:38:10 smithi038 ceph-mon[92422]: pgmap v3714: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 188 KiB/s wr, 26 op/s 2023-03-09T16:38:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:38:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6001.017474 secs 2023-03-09T16:38:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:38:12 smithi122 ceph-mon[93869]: pgmap v3715: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 91 KiB/s wr, 25 op/s 2023-03-09T16:38:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:38:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6003.257512 secs 2023-03-09T16:38:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:38:12 smithi181 ceph-mon[94189]: pgmap v3715: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 91 KiB/s wr, 25 op/s 2023-03-09T16:38:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:38:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6003.257512 secs 2023-03-09T16:38:12.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:38:12 smithi038 ceph-mon[92422]: pgmap v3715: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 91 KiB/s wr, 25 op/s 2023-03-09T16:38:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:38:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6003.257512 secs 2023-03-09T16:38:13.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6465.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:38:14.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:38:14 smithi122 ceph-mon[93869]: pgmap v3716: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 59 KiB/s wr, 11 op/s 2023-03-09T16:38:14.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:38:14 smithi181 ceph-mon[94189]: pgmap v3716: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 59 KiB/s wr, 11 op/s 2023-03-09T16:38:14.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:38:14 smithi038 ceph-mon[92422]: pgmap v3716: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 59 KiB/s wr, 11 op/s 2023-03-09T16:38:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:38:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6006.017534 secs 2023-03-09T16:38:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:38:15 smithi181 ceph-mon[94189]: pgmap v3717: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 59 KiB/s wr, 10 op/s 2023-03-09T16:38:15.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:38:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6006.017534 secs 2023-03-09T16:38:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:38:15 smithi038 ceph-mon[92422]: pgmap v3717: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 59 KiB/s wr, 10 op/s 2023-03-09T16:38:16.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:38:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6006.017534 secs 2023-03-09T16:38:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:38:15 smithi122 ceph-mon[93869]: pgmap v3717: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 59 KiB/s wr, 10 op/s 2023-03-09T16:38:17.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:38:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6008.257643 secs 2023-03-09T16:38:17.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:38:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6008.257643 secs 2023-03-09T16:38:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:38:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6008.257643 secs 2023-03-09T16:38:18.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:38:18 smithi122 ceph-mon[93869]: pgmap v3718: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T16:38:18.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:38:18 smithi181 ceph-mon[94189]: pgmap v3718: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T16:38:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:38:18 smithi038 ceph-mon[92422]: pgmap v3718: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T16:38:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6470.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:38:20.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:38:20 smithi122 ceph-mon[93869]: pgmap v3719: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T16:38:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:38:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6011.017593 secs 2023-03-09T16:38:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:38:20 smithi122 ceph-mon[93869]: osdmap e446: 12 total, 12 up, 12 in 2023-03-09T16:38:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:38:20 smithi181 ceph-mon[94189]: pgmap v3719: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T16:38:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:38:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6011.017593 secs 2023-03-09T16:38:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:38:20 smithi181 ceph-mon[94189]: osdmap e446: 12 total, 12 up, 12 in 2023-03-09T16:38:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:38:20 smithi038 ceph-mon[92422]: pgmap v3719: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T16:38:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:38:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6011.017593 secs 2023-03-09T16:38:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:38:20 smithi038 ceph-mon[92422]: osdmap e446: 12 total, 12 up, 12 in 2023-03-09T16:38:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:38:22 smithi122 ceph-mon[93869]: pgmap v3721: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 69 KiB/s wr, 7 op/s 2023-03-09T16:38:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:38:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6013.257719 secs 2023-03-09T16:38:22.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:38:22 smithi181 ceph-mon[94189]: pgmap v3721: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 69 KiB/s wr, 7 op/s 2023-03-09T16:38:22.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:38:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6013.257719 secs 2023-03-09T16:38:22.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:38:22 smithi038 ceph-mon[92422]: pgmap v3721: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 69 KiB/s wr, 7 op/s 2023-03-09T16:38:22.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:38:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6013.257719 secs 2023-03-09T16:38:23.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:38:23 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:38:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:38:23 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:38:23.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:38:23 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:38:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6475.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:38:24.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:38:24 smithi122 ceph-mon[93869]: pgmap v3722: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 69 KiB/s wr, 7 op/s 2023-03-09T16:38:24.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:38:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:38:24.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:38:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:38:24.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:38:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:38:24.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:38:24 smithi181 ceph-mon[94189]: pgmap v3722: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 69 KiB/s wr, 7 op/s 2023-03-09T16:38:24.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:38:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:38:24.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:38:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:38:24.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:38:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:38:24.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:38:24 smithi038 ceph-mon[92422]: pgmap v3722: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 69 KiB/s wr, 7 op/s 2023-03-09T16:38:24.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:38:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:38:24.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:38:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:38:24.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:38:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:38:25.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:38:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6016.017690 secs 2023-03-09T16:38:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:38:25 smithi038 ceph-mon[92422]: pgmap v3723: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 69 KiB/s wr, 7 op/s 2023-03-09T16:38:26.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:38:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6016.017690 secs 2023-03-09T16:38:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:38:25 smithi122 ceph-mon[93869]: pgmap v3723: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 69 KiB/s wr, 7 op/s 2023-03-09T16:38:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:38:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6016.017690 secs 2023-03-09T16:38:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:38:25 smithi181 ceph-mon[94189]: pgmap v3723: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 69 KiB/s wr, 7 op/s 2023-03-09T16:38:27.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:38:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6018.257838 secs 2023-03-09T16:38:27.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:38:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6018.257838 secs 2023-03-09T16:38:27.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:38:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6018.257838 secs 2023-03-09T16:38:28.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:38:28 smithi122 ceph-mon[93869]: pgmap v3724: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 46 KiB/s wr, 7 op/s 2023-03-09T16:38:28.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:38:28 smithi181 ceph-mon[94189]: pgmap v3724: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 46 KiB/s wr, 7 op/s 2023-03-09T16:38:28.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:38:28 smithi038 ceph-mon[92422]: pgmap v3724: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 46 KiB/s wr, 7 op/s 2023-03-09T16:38:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6480.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:38:30.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:38:30 smithi122 ceph-mon[93869]: pgmap v3725: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 46 KiB/s wr, 7 op/s 2023-03-09T16:38:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:38:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6021.017750 secs 2023-03-09T16:38:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:38:30 smithi181 ceph-mon[94189]: pgmap v3725: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 46 KiB/s wr, 7 op/s 2023-03-09T16:38:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:38:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6021.017750 secs 2023-03-09T16:38:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:38:30 smithi038 ceph-mon[92422]: pgmap v3725: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 46 KiB/s wr, 7 op/s 2023-03-09T16:38:30.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:38:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6021.017750 secs 2023-03-09T16:38:32.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:38:32 smithi122 ceph-mon[93869]: pgmap v3726: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 46 KiB/s wr, 8 op/s 2023-03-09T16:38:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:38:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6023.257961 secs 2023-03-09T16:38:32.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:38:32 smithi181 ceph-mon[94189]: pgmap v3726: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 46 KiB/s wr, 8 op/s 2023-03-09T16:38:32.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:38:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6023.257961 secs 2023-03-09T16:38:32.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:38:32 smithi038 ceph-mon[92422]: pgmap v3726: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 46 KiB/s wr, 8 op/s 2023-03-09T16:38:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:38:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6023.257961 secs 2023-03-09T16:38:33.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6485.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:38:34.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:38:34 smithi122 ceph-mon[93869]: pgmap v3727: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 40 KiB/s wr, 7 op/s 2023-03-09T16:38:34.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:38:34 smithi181 ceph-mon[94189]: pgmap v3727: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 40 KiB/s wr, 7 op/s 2023-03-09T16:38:34.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:38:34 smithi038 ceph-mon[92422]: pgmap v3727: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 40 KiB/s wr, 7 op/s 2023-03-09T16:38:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:38:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6026.017811 secs 2023-03-09T16:38:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:38:35 smithi038 ceph-mon[92422]: pgmap v3728: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 40 KiB/s wr, 6 op/s 2023-03-09T16:38:36.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:38:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6026.017811 secs 2023-03-09T16:38:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:38:35 smithi122 ceph-mon[93869]: pgmap v3728: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 40 KiB/s wr, 6 op/s 2023-03-09T16:38:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:38:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6026.017811 secs 2023-03-09T16:38:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:38:35 smithi181 ceph-mon[94189]: pgmap v3728: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 40 KiB/s wr, 6 op/s 2023-03-09T16:38:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:38:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6028.258083 secs 2023-03-09T16:38:37.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:38:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6028.258083 secs 2023-03-09T16:38:37.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:38:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6028.258083 secs 2023-03-09T16:38:38.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:38:38 smithi122 ceph-mon[93869]: pgmap v3729: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 45 KiB/s wr, 9 op/s 2023-03-09T16:38:38.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:38:38 smithi181 ceph-mon[94189]: pgmap v3729: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 45 KiB/s wr, 9 op/s 2023-03-09T16:38:38.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:38:38 smithi038 ceph-mon[92422]: pgmap v3729: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 45 KiB/s wr, 9 op/s 2023-03-09T16:38:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6490.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:38:40.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:38:40 smithi122 ceph-mon[93869]: pgmap v3730: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 85 KiB/s wr, 7 op/s 2023-03-09T16:38:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:38:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6031.017870 secs 2023-03-09T16:38:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:38:40 smithi181 ceph-mon[94189]: pgmap v3730: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 85 KiB/s wr, 7 op/s 2023-03-09T16:38:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:38:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6031.017870 secs 2023-03-09T16:38:40.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:38:40 smithi038 ceph-mon[92422]: pgmap v3730: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 85 KiB/s wr, 7 op/s 2023-03-09T16:38:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:38:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6031.017870 secs 2023-03-09T16:38:42.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:38:42 smithi122 ceph-mon[93869]: pgmap v3731: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 85 KiB/s wr, 10 op/s 2023-03-09T16:38:42.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:38:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6033.258221 secs 2023-03-09T16:38:42.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:38:42 smithi181 ceph-mon[94189]: pgmap v3731: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 85 KiB/s wr, 10 op/s 2023-03-09T16:38:42.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:38:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6033.258221 secs 2023-03-09T16:38:42.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:38:42 smithi038 ceph-mon[92422]: pgmap v3731: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 85 KiB/s wr, 10 op/s 2023-03-09T16:38:42.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:38:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6033.258221 secs 2023-03-09T16:38:43.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6495.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:38:44.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:38:44 smithi122 ceph-mon[93869]: pgmap v3732: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 193 KiB/s wr, 10 op/s 2023-03-09T16:38:44.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:38:44 smithi181 ceph-mon[94189]: pgmap v3732: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 193 KiB/s wr, 10 op/s 2023-03-09T16:38:44.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:38:44 smithi038 ceph-mon[92422]: pgmap v3732: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 193 KiB/s wr, 10 op/s 2023-03-09T16:38:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:38:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6036.017933 secs 2023-03-09T16:38:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:38:45 smithi038 ceph-mon[92422]: pgmap v3733: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 158 KiB/s wr, 9 op/s 2023-03-09T16:38:46.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:38:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6036.017933 secs 2023-03-09T16:38:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:38:45 smithi122 ceph-mon[93869]: pgmap v3733: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 158 KiB/s wr, 9 op/s 2023-03-09T16:38:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:38:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6036.017933 secs 2023-03-09T16:38:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:38:45 smithi181 ceph-mon[94189]: pgmap v3733: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 158 KiB/s wr, 9 op/s 2023-03-09T16:38:47.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:38:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6038.258327 secs 2023-03-09T16:38:47.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:38:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6038.258327 secs 2023-03-09T16:38:47.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:38:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6038.258327 secs 2023-03-09T16:38:48.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:38:48 smithi122 ceph-mon[93869]: pgmap v3734: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 164 KiB/s wr, 12 op/s 2023-03-09T16:38:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:38:48 smithi181 ceph-mon[94189]: pgmap v3734: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 164 KiB/s wr, 12 op/s 2023-03-09T16:38:48.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:38:48 smithi038 ceph-mon[92422]: pgmap v3734: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 164 KiB/s wr, 12 op/s 2023-03-09T16:38:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6500.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:38:50.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:38:50 smithi122 ceph-mon[93869]: pgmap v3735: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 204 KiB/s wr, 10 op/s 2023-03-09T16:38:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:38:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6041.017983 secs 2023-03-09T16:38:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:38:50 smithi181 ceph-mon[94189]: pgmap v3735: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 204 KiB/s wr, 10 op/s 2023-03-09T16:38:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:38:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6041.017983 secs 2023-03-09T16:38:50.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:38:50 smithi038 ceph-mon[92422]: pgmap v3735: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 204 KiB/s wr, 10 op/s 2023-03-09T16:38:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:38:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6041.017983 secs 2023-03-09T16:38:52.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:38:52 smithi122 ceph-mon[93869]: pgmap v3736: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 164 KiB/s wr, 11 op/s 2023-03-09T16:38:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:38:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6043.258451 secs 2023-03-09T16:38:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:38:52 smithi181 ceph-mon[94189]: pgmap v3736: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 164 KiB/s wr, 11 op/s 2023-03-09T16:38:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:38:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6043.258451 secs 2023-03-09T16:38:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:38:52 smithi038 ceph-mon[92422]: pgmap v3736: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 164 KiB/s wr, 11 op/s 2023-03-09T16:38:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:38:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6043.258451 secs 2023-03-09T16:38:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6505.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:38:54.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:38:54 smithi122 ceph-mon[93869]: pgmap v3737: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 272 KiB/s wr, 11 op/s 2023-03-09T16:38:54.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:38:54 smithi181 ceph-mon[94189]: pgmap v3737: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 272 KiB/s wr, 11 op/s 2023-03-09T16:38:54.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:38:54 smithi038 ceph-mon[92422]: pgmap v3737: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 272 KiB/s wr, 11 op/s 2023-03-09T16:38:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:38:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6046.018007 secs 2023-03-09T16:38:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:38:55 smithi038 ceph-mon[92422]: pgmap v3738: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 196 KiB/s wr, 9 op/s 2023-03-09T16:38:56.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:38:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6046.018007 secs 2023-03-09T16:38:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:38:55 smithi122 ceph-mon[93869]: pgmap v3738: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 196 KiB/s wr, 9 op/s 2023-03-09T16:38:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:38:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6046.018007 secs 2023-03-09T16:38:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:38:55 smithi181 ceph-mon[94189]: pgmap v3738: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 196 KiB/s wr, 9 op/s 2023-03-09T16:38:57.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:38:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6048.258576 secs 2023-03-09T16:38:57.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:38:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6048.258576 secs 2023-03-09T16:38:57.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:38:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6048.258576 secs 2023-03-09T16:38:58.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:38:58 smithi122 ceph-mon[93869]: pgmap v3739: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 206 KiB/s wr, 11 op/s 2023-03-09T16:38:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:38:58 smithi181 ceph-mon[94189]: pgmap v3739: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 206 KiB/s wr, 11 op/s 2023-03-09T16:38:58.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:38:58 smithi038 ceph-mon[92422]: pgmap v3739: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 206 KiB/s wr, 11 op/s 2023-03-09T16:38:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6510.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:39:00.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:00 smithi122 ceph-mon[93869]: pgmap v3740: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 209 KiB/s wr, 9 op/s 2023-03-09T16:39:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6051.018067 secs 2023-03-09T16:39:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:00 smithi181 ceph-mon[94189]: pgmap v3740: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 209 KiB/s wr, 9 op/s 2023-03-09T16:39:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6051.018067 secs 2023-03-09T16:39:00.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:00 smithi038 ceph-mon[92422]: pgmap v3740: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 209 KiB/s wr, 9 op/s 2023-03-09T16:39:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6051.018067 secs 2023-03-09T16:39:02.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:02 smithi122 ceph-mon[93869]: pgmap v3741: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 196 KiB/s wr, 11 op/s 2023-03-09T16:39:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6053.258699 secs 2023-03-09T16:39:02.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:02 smithi181 ceph-mon[94189]: pgmap v3741: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 196 KiB/s wr, 11 op/s 2023-03-09T16:39:02.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6053.258699 secs 2023-03-09T16:39:02.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:02 smithi038 ceph-mon[92422]: pgmap v3741: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 196 KiB/s wr, 11 op/s 2023-03-09T16:39:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6053.258699 secs 2023-03-09T16:39:03.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6515.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:39:04.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:04 smithi122 ceph-mon[93869]: pgmap v3742: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 218 KiB/s wr, 22 op/s 2023-03-09T16:39:04.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:04 smithi181 ceph-mon[94189]: pgmap v3742: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 218 KiB/s wr, 22 op/s 2023-03-09T16:39:04.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:04 smithi038 ceph-mon[92422]: pgmap v3742: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 218 KiB/s wr, 22 op/s 2023-03-09T16:39:05.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:05 smithi122 ceph-mon[93869]: osdmap e447: 12 total, 12 up, 12 in 2023-03-09T16:39:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6056.018126 secs 2023-03-09T16:39:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:05 smithi181 ceph-mon[94189]: osdmap e447: 12 total, 12 up, 12 in 2023-03-09T16:39:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6056.018126 secs 2023-03-09T16:39:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:05 smithi038 ceph-mon[92422]: osdmap e447: 12 total, 12 up, 12 in 2023-03-09T16:39:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6056.018126 secs 2023-03-09T16:39:06.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:06 smithi122 ceph-mon[93869]: pgmap v3744: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 188 KiB/s wr, 24 op/s 2023-03-09T16:39:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:06 smithi181 ceph-mon[94189]: pgmap v3744: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 188 KiB/s wr, 24 op/s 2023-03-09T16:39:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:06 smithi038 ceph-mon[92422]: pgmap v3744: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 188 KiB/s wr, 24 op/s 2023-03-09T16:39:07.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6058.258819 secs 2023-03-09T16:39:07.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6058.258819 secs 2023-03-09T16:39:07.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6058.258819 secs 2023-03-09T16:39:08.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:08 smithi122 ceph-mon[93869]: pgmap v3745: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 209 KiB/s wr, 27 op/s 2023-03-09T16:39:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:08 smithi181 ceph-mon[94189]: pgmap v3745: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 209 KiB/s wr, 27 op/s 2023-03-09T16:39:08.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:08 smithi038 ceph-mon[92422]: pgmap v3745: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 209 KiB/s wr, 27 op/s 2023-03-09T16:39:08.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6520.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:39:10.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:10 smithi122 ceph-mon[93869]: pgmap v3746: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 199 KiB/s wr, 26 op/s 2023-03-09T16:39:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6061.018185 secs 2023-03-09T16:39:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:10 smithi181 ceph-mon[94189]: pgmap v3746: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 199 KiB/s wr, 26 op/s 2023-03-09T16:39:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6061.018185 secs 2023-03-09T16:39:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:10 smithi038 ceph-mon[92422]: pgmap v3746: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 199 KiB/s wr, 26 op/s 2023-03-09T16:39:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6061.018185 secs 2023-03-09T16:39:12.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:12 smithi122 ceph-mon[93869]: pgmap v3747: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 190 KiB/s wr, 26 op/s 2023-03-09T16:39:12.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6063.258943 secs 2023-03-09T16:39:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:12 smithi181 ceph-mon[94189]: pgmap v3747: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 190 KiB/s wr, 26 op/s 2023-03-09T16:39:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6063.258943 secs 2023-03-09T16:39:12.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:12 smithi038 ceph-mon[92422]: pgmap v3747: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 190 KiB/s wr, 26 op/s 2023-03-09T16:39:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6063.258943 secs 2023-03-09T16:39:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6525.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:39:14.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:14 smithi122 ceph-mon[93869]: pgmap v3748: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 170 KiB/s wr, 11 op/s 2023-03-09T16:39:14.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:14 smithi181 ceph-mon[94189]: pgmap v3748: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 170 KiB/s wr, 11 op/s 2023-03-09T16:39:14.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:14 smithi038 ceph-mon[92422]: pgmap v3748: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 170 KiB/s wr, 11 op/s 2023-03-09T16:39:15.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6066.018269 secs 2023-03-09T16:39:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:15 smithi038 ceph-mon[92422]: osdmap e448: 12 total, 12 up, 12 in 2023-03-09T16:39:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:15 smithi038 ceph-mon[92422]: pgmap v3750: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 70 KiB/s wr, 10 op/s 2023-03-09T16:39:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6066.018269 secs 2023-03-09T16:39:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:15 smithi122 ceph-mon[93869]: osdmap e448: 12 total, 12 up, 12 in 2023-03-09T16:39:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:15 smithi122 ceph-mon[93869]: pgmap v3750: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 70 KiB/s wr, 10 op/s 2023-03-09T16:39:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6066.018269 secs 2023-03-09T16:39:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:15 smithi181 ceph-mon[94189]: osdmap e448: 12 total, 12 up, 12 in 2023-03-09T16:39:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:15 smithi181 ceph-mon[94189]: pgmap v3750: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 70 KiB/s wr, 10 op/s 2023-03-09T16:39:17.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6068.259042 secs 2023-03-09T16:39:17.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6068.259042 secs 2023-03-09T16:39:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6068.259042 secs 2023-03-09T16:39:18.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:18 smithi122 ceph-mon[93869]: pgmap v3751: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 37 KiB/s wr, 7 op/s 2023-03-09T16:39:18.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:18 smithi181 ceph-mon[94189]: pgmap v3751: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 37 KiB/s wr, 7 op/s 2023-03-09T16:39:18.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:18 smithi038 ceph-mon[92422]: pgmap v3751: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 37 KiB/s wr, 7 op/s 2023-03-09T16:39:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6530.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:39:20.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:20 smithi122 ceph-mon[93869]: pgmap v3752: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 37 KiB/s wr, 7 op/s 2023-03-09T16:39:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6071.018328 secs 2023-03-09T16:39:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:20 smithi181 ceph-mon[94189]: pgmap v3752: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 37 KiB/s wr, 7 op/s 2023-03-09T16:39:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6071.018328 secs 2023-03-09T16:39:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:20 smithi038 ceph-mon[92422]: pgmap v3752: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 37 KiB/s wr, 7 op/s 2023-03-09T16:39:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6071.018328 secs 2023-03-09T16:39:22.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:22 smithi122 ceph-mon[93869]: pgmap v3753: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 54 KiB/s wr, 7 op/s 2023-03-09T16:39:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6073.259193 secs 2023-03-09T16:39:22.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:22 smithi181 ceph-mon[94189]: pgmap v3753: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 54 KiB/s wr, 7 op/s 2023-03-09T16:39:22.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6073.259193 secs 2023-03-09T16:39:22.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:22 smithi038 ceph-mon[92422]: pgmap v3753: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 54 KiB/s wr, 7 op/s 2023-03-09T16:39:22.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6073.259193 secs 2023-03-09T16:39:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6535.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:39:24.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:24 smithi122 ceph-mon[93869]: pgmap v3754: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 48 KiB/s wr, 8 op/s 2023-03-09T16:39:24.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:39:24.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:24 smithi181 ceph-mon[94189]: pgmap v3754: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 48 KiB/s wr, 8 op/s 2023-03-09T16:39:24.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:39:24.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:24 smithi038 ceph-mon[92422]: pgmap v3754: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 48 KiB/s wr, 8 op/s 2023-03-09T16:39:24.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:39:25.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6076.018392 secs 2023-03-09T16:39:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:25 smithi038 ceph-mon[92422]: pgmap v3755: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 45 KiB/s wr, 7 op/s 2023-03-09T16:39:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6076.018392 secs 2023-03-09T16:39:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:25 smithi122 ceph-mon[93869]: pgmap v3755: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 45 KiB/s wr, 7 op/s 2023-03-09T16:39:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6076.018392 secs 2023-03-09T16:39:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:25 smithi181 ceph-mon[94189]: pgmap v3755: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 45 KiB/s wr, 7 op/s 2023-03-09T16:39:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:39:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:39:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:39:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:39:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6078.259257 secs 2023-03-09T16:39:27.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:39:27.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:39:27.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:39:27.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:39:27.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:39:27.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:39:27.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:39:27.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:39:27.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:39:27.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:39:27.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:39:27.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:39:27.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:39:27.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:39:27.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6078.259257 secs 2023-03-09T16:39:27.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:39:27.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:39:27.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:39:27.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:39:27.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:39:27.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:39:27.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:39:27.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:39:27.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:39:27.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:39:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:39:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:39:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:39:27.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:39:27.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6078.259257 secs 2023-03-09T16:39:27.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:39:27.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:39:27.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:39:27.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:39:27.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:39:27.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:39:27.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:39:27.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:39:27.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:39:27.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:39:28.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:28 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi038 to 197.2M 2023-03-09T16:39:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:28 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi038 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-03-09T16:39:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:28 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T16:39:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:28 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T16:39:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:28 smithi122 ceph-mon[93869]: pgmap v3756: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 40 KiB/s wr, 9 op/s 2023-03-09T16:39:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:39:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:39:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:39:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:39:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:39:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:39:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:39:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:28 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi038 to 197.2M 2023-03-09T16:39:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:28 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi038 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-03-09T16:39:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:28 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T16:39:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:28 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T16:39:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:28 smithi181 ceph-mon[94189]: pgmap v3756: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 40 KiB/s wr, 9 op/s 2023-03-09T16:39:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:39:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:39:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:39:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:39:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:39:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:39:28.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:39:28.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:28 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi038 to 197.2M 2023-03-09T16:39:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:28 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi038 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-03-09T16:39:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:28 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T16:39:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:28 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T16:39:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:28 smithi038 ceph-mon[92422]: pgmap v3756: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 40 KiB/s wr, 9 op/s 2023-03-09T16:39:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:39:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:39:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:39:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:39:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:39:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:39:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:39:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6540.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:39:29.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:29 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T16:39:29.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:29 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T16:39:29.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:29 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T16:39:29.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:29 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T16:39:29.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:29 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T16:39:29.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:29 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T16:39:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:30 smithi122 ceph-mon[93869]: pgmap v3757: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 40 KiB/s wr, 6 op/s 2023-03-09T16:39:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6081.018471 secs 2023-03-09T16:39:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:30 smithi181 ceph-mon[94189]: pgmap v3757: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 40 KiB/s wr, 6 op/s 2023-03-09T16:39:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6081.018471 secs 2023-03-09T16:39:30.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:30 smithi038 ceph-mon[92422]: pgmap v3757: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 40 KiB/s wr, 6 op/s 2023-03-09T16:39:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6081.018471 secs 2023-03-09T16:39:32.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:32 smithi122 ceph-mon[93869]: pgmap v3758: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 79 KiB/s wr, 8 op/s 2023-03-09T16:39:32.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:32 smithi181 ceph-mon[94189]: pgmap v3758: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 79 KiB/s wr, 8 op/s 2023-03-09T16:39:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:32 smithi038 ceph-mon[92422]: pgmap v3758: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 79 KiB/s wr, 8 op/s 2023-03-09T16:39:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6545.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:39:34.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:34 smithi122 ceph-mon[93869]: pgmap v3759: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 46 KiB/s wr, 7 op/s 2023-03-09T16:39:34.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:34 smithi181 ceph-mon[94189]: pgmap v3759: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 46 KiB/s wr, 7 op/s 2023-03-09T16:39:34.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:34 smithi038 ceph-mon[92422]: pgmap v3759: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 46 KiB/s wr, 7 op/s 2023-03-09T16:39:36.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6086.018531 secs 2023-03-09T16:39:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:35 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:39:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:35 smithi122 ceph-mon[93869]: pgmap v3760: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 133 KiB/s wr, 8 op/s 2023-03-09T16:39:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6086.018531 secs 2023-03-09T16:39:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:35 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:39:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:35 smithi181 ceph-mon[94189]: pgmap v3760: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 133 KiB/s wr, 8 op/s 2023-03-09T16:39:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6086.018531 secs 2023-03-09T16:39:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:35 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:39:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:35 smithi038 ceph-mon[92422]: pgmap v3760: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 133 KiB/s wr, 8 op/s 2023-03-09T16:39:37.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:36 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:39:37.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:39:37.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:39:37.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:36 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:39:37.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:39:37.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:39:37.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:36 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:39:37.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:39:37.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:39:38.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:37 smithi122 ceph-mon[93869]: pgmap v3761: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 165 KiB/s wr, 35 op/s 2023-03-09T16:39:38.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:37 smithi181 ceph-mon[94189]: pgmap v3761: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 165 KiB/s wr, 35 op/s 2023-03-09T16:39:38.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:37 smithi038 ceph-mon[92422]: pgmap v3761: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 165 KiB/s wr, 35 op/s 2023-03-09T16:39:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6550.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:39:40.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:40 smithi122 ceph-mon[93869]: pgmap v3762: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 165 KiB/s wr, 32 op/s 2023-03-09T16:39:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6091.018593 secs 2023-03-09T16:39:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:40 smithi181 ceph-mon[94189]: pgmap v3762: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 165 KiB/s wr, 32 op/s 2023-03-09T16:39:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6091.018593 secs 2023-03-09T16:39:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:40 smithi038 ceph-mon[92422]: pgmap v3762: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 165 KiB/s wr, 32 op/s 2023-03-09T16:39:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6091.018593 secs 2023-03-09T16:39:42.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:42 smithi122 ceph-mon[93869]: pgmap v3763: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 248 KiB/s wr, 36 op/s 2023-03-09T16:39:42.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6093.259629 secs 2023-03-09T16:39:42.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:42 smithi181 ceph-mon[94189]: pgmap v3763: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 248 KiB/s wr, 36 op/s 2023-03-09T16:39:42.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6093.259629 secs 2023-03-09T16:39:42.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:42 smithi038 ceph-mon[92422]: pgmap v3763: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 248 KiB/s wr, 36 op/s 2023-03-09T16:39:42.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6093.259629 secs 2023-03-09T16:39:43.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:43 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:39:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:43 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:39:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:43 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:39:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:43 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:39:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:43 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:39:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:43 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:39:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:43 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:39:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:43 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:39:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:43 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:39:43.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6555.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:39:44.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:44 smithi122 ceph-mon[93869]: pgmap v3764: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 214 KiB/s wr, 35 op/s 2023-03-09T16:39:44.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:44 smithi181 ceph-mon[94189]: pgmap v3764: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 214 KiB/s wr, 35 op/s 2023-03-09T16:39:44.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:44 smithi038 ceph-mon[92422]: pgmap v3764: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 214 KiB/s wr, 35 op/s 2023-03-09T16:39:46.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6096.018654 secs 2023-03-09T16:39:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:45 smithi122 ceph-mon[93869]: pgmap v3765: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 209 KiB/s wr, 34 op/s 2023-03-09T16:39:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6096.018654 secs 2023-03-09T16:39:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:45 smithi181 ceph-mon[94189]: pgmap v3765: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 209 KiB/s wr, 34 op/s 2023-03-09T16:39:46.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6096.018654 secs 2023-03-09T16:39:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:45 smithi038 ceph-mon[92422]: pgmap v3765: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 209 KiB/s wr, 34 op/s 2023-03-09T16:39:47.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6098.259749 secs 2023-03-09T16:39:47.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6098.259749 secs 2023-03-09T16:39:47.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6098.259749 secs 2023-03-09T16:39:48.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:48 smithi122 ceph-mon[93869]: pgmap v3766: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 240 KiB/s wr, 37 op/s 2023-03-09T16:39:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:48 smithi181 ceph-mon[94189]: pgmap v3766: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 240 KiB/s wr, 37 op/s 2023-03-09T16:39:48.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:48 smithi038 ceph-mon[92422]: pgmap v3766: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 240 KiB/s wr, 37 op/s 2023-03-09T16:39:48.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6560.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:39:50.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:50 smithi122 ceph-mon[93869]: pgmap v3767: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 209 KiB/s wr, 10 op/s 2023-03-09T16:39:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6101.018714 secs 2023-03-09T16:39:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:50 smithi181 ceph-mon[94189]: pgmap v3767: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 209 KiB/s wr, 10 op/s 2023-03-09T16:39:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6101.018714 secs 2023-03-09T16:39:50.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:50 smithi038 ceph-mon[92422]: pgmap v3767: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 209 KiB/s wr, 10 op/s 2023-03-09T16:39:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6101.018714 secs 2023-03-09T16:39:52.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:52 smithi122 ceph-mon[93869]: pgmap v3768: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 228 KiB/s wr, 13 op/s 2023-03-09T16:39:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6103.259871 secs 2023-03-09T16:39:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:52 smithi181 ceph-mon[94189]: pgmap v3768: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 228 KiB/s wr, 13 op/s 2023-03-09T16:39:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6103.259871 secs 2023-03-09T16:39:52.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:52 smithi038 ceph-mon[92422]: pgmap v3768: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 228 KiB/s wr, 13 op/s 2023-03-09T16:39:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6103.259871 secs 2023-03-09T16:39:53.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6565.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:39:54.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:54 smithi122 ceph-mon[93869]: pgmap v3769: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 144 KiB/s wr, 10 op/s 2023-03-09T16:39:54.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:54 smithi181 ceph-mon[94189]: pgmap v3769: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 144 KiB/s wr, 10 op/s 2023-03-09T16:39:54.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:54 smithi038 ceph-mon[92422]: pgmap v3769: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 144 KiB/s wr, 10 op/s 2023-03-09T16:39:56.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6106.018774 secs 2023-03-09T16:39:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:55 smithi122 ceph-mon[93869]: pgmap v3770: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 139 KiB/s wr, 9 op/s 2023-03-09T16:39:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6106.018774 secs 2023-03-09T16:39:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:55 smithi181 ceph-mon[94189]: pgmap v3770: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 139 KiB/s wr, 9 op/s 2023-03-09T16:39:56.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6106.018774 secs 2023-03-09T16:39:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:55 smithi038 ceph-mon[92422]: pgmap v3770: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 139 KiB/s wr, 9 op/s 2023-03-09T16:39:57.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6108.259992 secs 2023-03-09T16:39:57.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6108.259992 secs 2023-03-09T16:39:57.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6108.259992 secs 2023-03-09T16:39:58.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:39:58 smithi122 ceph-mon[93869]: pgmap v3771: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 242 KiB/s wr, 12 op/s 2023-03-09T16:39:58.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:39:58 smithi181 ceph-mon[94189]: pgmap v3771: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 242 KiB/s wr, 12 op/s 2023-03-09T16:39:58.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:39:58 smithi038 ceph-mon[92422]: pgmap v3771: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 242 KiB/s wr, 12 op/s 2023-03-09T16:39:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6570.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:40:00.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:40:00 smithi122 ceph-mon[93869]: pgmap v3772: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 123 KiB/s wr, 8 op/s 2023-03-09T16:40:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:40:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6111.018836 secs 2023-03-09T16:40:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:40:00 smithi122 ceph-mon[93869]: overall HEALTH_WARN 2 MDSs report slow requests 2023-03-09T16:40:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:40:00 smithi181 ceph-mon[94189]: pgmap v3772: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 123 KiB/s wr, 8 op/s 2023-03-09T16:40:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:40:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6111.018836 secs 2023-03-09T16:40:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:40:00 smithi181 ceph-mon[94189]: overall HEALTH_WARN 2 MDSs report slow requests 2023-03-09T16:40:00.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:40:00 smithi038 ceph-mon[92422]: pgmap v3772: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 123 KiB/s wr, 8 op/s 2023-03-09T16:40:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:40:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6111.018836 secs 2023-03-09T16:40:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:40:00 smithi038 ceph-mon[92422]: overall HEALTH_WARN 2 MDSs report slow requests 2023-03-09T16:40:02.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:40:02 smithi122 ceph-mon[93869]: pgmap v3773: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 156 KiB/s wr, 10 op/s 2023-03-09T16:40:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:40:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6113.260136 secs 2023-03-09T16:40:02.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:40:02 smithi181 ceph-mon[94189]: pgmap v3773: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 156 KiB/s wr, 10 op/s 2023-03-09T16:40:02.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:40:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6113.260136 secs 2023-03-09T16:40:02.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:40:02 smithi038 ceph-mon[92422]: pgmap v3773: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 156 KiB/s wr, 10 op/s 2023-03-09T16:40:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:40:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6113.260136 secs 2023-03-09T16:40:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6575.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:40:04.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:40:04 smithi122 ceph-mon[93869]: pgmap v3774: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 158 KiB/s wr, 20 op/s 2023-03-09T16:40:04.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:40:04 smithi181 ceph-mon[94189]: pgmap v3774: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 158 KiB/s wr, 20 op/s 2023-03-09T16:40:04.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:40:04 smithi038 ceph-mon[92422]: pgmap v3774: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 158 KiB/s wr, 20 op/s 2023-03-09T16:40:06.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:40:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6116.018895 secs 2023-03-09T16:40:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:40:05 smithi122 ceph-mon[93869]: pgmap v3775: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 174 KiB/s wr, 20 op/s 2023-03-09T16:40:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:40:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6116.018895 secs 2023-03-09T16:40:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:40:05 smithi181 ceph-mon[94189]: pgmap v3775: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 174 KiB/s wr, 20 op/s 2023-03-09T16:40:06.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:40:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6116.018895 secs 2023-03-09T16:40:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:40:05 smithi038 ceph-mon[92422]: pgmap v3775: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 174 KiB/s wr, 20 op/s 2023-03-09T16:40:07.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:40:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6118.260252 secs 2023-03-09T16:40:07.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:40:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6118.260252 secs 2023-03-09T16:40:07.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:40:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6118.260252 secs 2023-03-09T16:40:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:40:08 smithi122 ceph-mon[93869]: pgmap v3776: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 202 KiB/s wr, 25 op/s 2023-03-09T16:40:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:40:08 smithi181 ceph-mon[94189]: pgmap v3776: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 202 KiB/s wr, 25 op/s 2023-03-09T16:40:08.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:40:08 smithi038 ceph-mon[92422]: pgmap v3776: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 202 KiB/s wr, 25 op/s 2023-03-09T16:40:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6580.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:40:10.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:40:10 smithi122 ceph-mon[93869]: osdmap e449: 12 total, 12 up, 12 in 2023-03-09T16:40:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:40:10 smithi122 ceph-mon[93869]: pgmap v3778: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 119 KiB/s wr, 26 op/s 2023-03-09T16:40:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:40:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6121.018986 secs 2023-03-09T16:40:10.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:40:10 smithi181 ceph-mon[94189]: osdmap e449: 12 total, 12 up, 12 in 2023-03-09T16:40:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:40:10 smithi181 ceph-mon[94189]: pgmap v3778: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 119 KiB/s wr, 26 op/s 2023-03-09T16:40:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:40:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6121.018986 secs 2023-03-09T16:40:10.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:40:10 smithi038 ceph-mon[92422]: osdmap e449: 12 total, 12 up, 12 in 2023-03-09T16:40:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:40:10 smithi038 ceph-mon[92422]: pgmap v3778: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 119 KiB/s wr, 26 op/s 2023-03-09T16:40:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:40:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6121.018986 secs 2023-03-09T16:40:12.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:40:12 smithi122 ceph-mon[93869]: pgmap v3779: 129 pgs: 1 active+clean+snaptrim_wait, 128 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 80 KiB/s wr, 26 op/s 2023-03-09T16:40:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:40:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6123.260382 secs 2023-03-09T16:40:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:40:12 smithi181 ceph-mon[94189]: pgmap v3779: 129 pgs: 1 active+clean+snaptrim_wait, 128 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 80 KiB/s wr, 26 op/s 2023-03-09T16:40:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:40:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6123.260382 secs 2023-03-09T16:40:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:40:12 smithi038 ceph-mon[92422]: pgmap v3779: 129 pgs: 1 active+clean+snaptrim_wait, 128 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 80 KiB/s wr, 26 op/s 2023-03-09T16:40:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:40:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6123.260382 secs 2023-03-09T16:40:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6585.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:40:14.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:40:14 smithi122 ceph-mon[93869]: pgmap v3780: 129 pgs: 1 active+clean+snaptrim_wait, 128 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 54 KiB/s wr, 12 op/s 2023-03-09T16:40:14.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:40:14 smithi181 ceph-mon[94189]: pgmap v3780: 129 pgs: 1 active+clean+snaptrim_wait, 128 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 54 KiB/s wr, 12 op/s 2023-03-09T16:40:14.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:40:14 smithi038 ceph-mon[92422]: pgmap v3780: 129 pgs: 1 active+clean+snaptrim_wait, 128 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 54 KiB/s wr, 12 op/s 2023-03-09T16:40:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:40:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6126.019048 secs 2023-03-09T16:40:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:40:15 smithi122 ceph-mon[93869]: pgmap v3781: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 81 KiB/s wr, 11 op/s 2023-03-09T16:40:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:40:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6126.019048 secs 2023-03-09T16:40:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:40:15 smithi181 ceph-mon[94189]: pgmap v3781: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 81 KiB/s wr, 11 op/s 2023-03-09T16:40:16.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:40:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6126.019048 secs 2023-03-09T16:40:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:40:15 smithi038 ceph-mon[92422]: pgmap v3781: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 81 KiB/s wr, 11 op/s 2023-03-09T16:40:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:40:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6128.260516 secs 2023-03-09T16:40:17.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:40:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6128.260516 secs 2023-03-09T16:40:17.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:40:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6128.260516 secs 2023-03-09T16:40:18.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:40:18 smithi122 ceph-mon[93869]: pgmap v3782: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 48 KiB/s wr, 8 op/s 2023-03-09T16:40:18.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:40:18 smithi181 ceph-mon[94189]: pgmap v3782: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 48 KiB/s wr, 8 op/s 2023-03-09T16:40:18.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:40:18 smithi038 ceph-mon[92422]: pgmap v3782: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 48 KiB/s wr, 8 op/s 2023-03-09T16:40:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6590.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:40:20.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:40:20 smithi122 ceph-mon[93869]: pgmap v3783: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 47 KiB/s wr, 8 op/s 2023-03-09T16:40:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:40:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6131.019109 secs 2023-03-09T16:40:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:40:20 smithi122 ceph-mon[93869]: osdmap e450: 12 total, 12 up, 12 in 2023-03-09T16:40:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:40:20 smithi181 ceph-mon[94189]: pgmap v3783: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 47 KiB/s wr, 8 op/s 2023-03-09T16:40:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:40:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6131.019109 secs 2023-03-09T16:40:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:40:20 smithi181 ceph-mon[94189]: osdmap e450: 12 total, 12 up, 12 in 2023-03-09T16:40:20.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:40:20 smithi038 ceph-mon[92422]: pgmap v3783: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 47 KiB/s wr, 8 op/s 2023-03-09T16:40:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:40:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6131.019109 secs 2023-03-09T16:40:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:40:20 smithi038 ceph-mon[92422]: osdmap e450: 12 total, 12 up, 12 in 2023-03-09T16:40:22.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:40:22 smithi122 ceph-mon[93869]: pgmap v3785: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 47 KiB/s wr, 7 op/s 2023-03-09T16:40:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:40:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6133.260668 secs 2023-03-09T16:40:22.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:40:22 smithi181 ceph-mon[94189]: pgmap v3785: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 47 KiB/s wr, 7 op/s 2023-03-09T16:40:22.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:40:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6133.260668 secs 2023-03-09T16:40:22.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:40:22 smithi038 ceph-mon[92422]: pgmap v3785: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 47 KiB/s wr, 7 op/s 2023-03-09T16:40:22.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:40:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6133.260668 secs 2023-03-09T16:40:23.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6595.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:40:24.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:40:24 smithi122 ceph-mon[93869]: pgmap v3786: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 47 KiB/s wr, 7 op/s 2023-03-09T16:40:24.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:40:24 smithi181 ceph-mon[94189]: pgmap v3786: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 47 KiB/s wr, 7 op/s 2023-03-09T16:40:24.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:40:24 smithi038 ceph-mon[92422]: pgmap v3786: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 47 KiB/s wr, 7 op/s 2023-03-09T16:40:26.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:40:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6136.019213 secs 2023-03-09T16:40:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:40:25 smithi122 ceph-mon[93869]: pgmap v3787: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 47 KiB/s wr, 7 op/s 2023-03-09T16:40:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:40:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6136.019213 secs 2023-03-09T16:40:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:40:25 smithi181 ceph-mon[94189]: pgmap v3787: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 47 KiB/s wr, 7 op/s 2023-03-09T16:40:26.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:40:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6136.019213 secs 2023-03-09T16:40:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:40:25 smithi038 ceph-mon[92422]: pgmap v3787: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 47 KiB/s wr, 7 op/s 2023-03-09T16:40:27.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:40:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6138.260786 secs 2023-03-09T16:40:27.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:40:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6138.260786 secs 2023-03-09T16:40:27.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:40:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6138.260786 secs 2023-03-09T16:40:28.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:40:28 smithi122 ceph-mon[93869]: pgmap v3788: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 47 KiB/s wr, 7 op/s 2023-03-09T16:40:28.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:40:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:40:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:40:28 smithi181 ceph-mon[94189]: pgmap v3788: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 47 KiB/s wr, 7 op/s 2023-03-09T16:40:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:40:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:40:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:40:28 smithi038 ceph-mon[92422]: pgmap v3788: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 47 KiB/s wr, 7 op/s 2023-03-09T16:40:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:40:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:40:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6600.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:40:29.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:40:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:40:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:40:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:40:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:40:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:40:29.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:40:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:40:29.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:40:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:40:29.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:40:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:40:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:40:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:40:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:40:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:40:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:40:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:40:30.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:40:30 smithi122 ceph-mon[93869]: pgmap v3789: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 47 KiB/s wr, 6 op/s 2023-03-09T16:40:30.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:40:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6141.019275 secs 2023-03-09T16:40:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:40:30 smithi181 ceph-mon[94189]: pgmap v3789: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 47 KiB/s wr, 6 op/s 2023-03-09T16:40:30.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:40:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6141.019275 secs 2023-03-09T16:40:30.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:40:30 smithi038 ceph-mon[92422]: pgmap v3789: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 47 KiB/s wr, 6 op/s 2023-03-09T16:40:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:40:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6141.019275 secs 2023-03-09T16:40:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:40:32 smithi122 ceph-mon[93869]: pgmap v3790: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 41 KiB/s wr, 8 op/s 2023-03-09T16:40:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:40:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6143.260880 secs 2023-03-09T16:40:32.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:40:32 smithi181 ceph-mon[94189]: pgmap v3790: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 41 KiB/s wr, 8 op/s 2023-03-09T16:40:32.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:40:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6143.260880 secs 2023-03-09T16:40:32.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:40:32 smithi038 ceph-mon[92422]: pgmap v3790: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 41 KiB/s wr, 8 op/s 2023-03-09T16:40:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:40:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6143.260880 secs 2023-03-09T16:40:33.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6605.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:40:34.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:40:34 smithi122 ceph-mon[93869]: pgmap v3791: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 39 KiB/s wr, 6 op/s 2023-03-09T16:40:34.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:40:34 smithi181 ceph-mon[94189]: pgmap v3791: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 39 KiB/s wr, 6 op/s 2023-03-09T16:40:34.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:40:34 smithi038 ceph-mon[92422]: pgmap v3791: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 39 KiB/s wr, 6 op/s 2023-03-09T16:40:36.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:40:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6146.019332 secs 2023-03-09T16:40:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:40:35 smithi122 ceph-mon[93869]: pgmap v3792: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 79 KiB/s wr, 6 op/s 2023-03-09T16:40:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:40:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6146.019332 secs 2023-03-09T16:40:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:40:35 smithi181 ceph-mon[94189]: pgmap v3792: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 79 KiB/s wr, 6 op/s 2023-03-09T16:40:36.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:40:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6146.019332 secs 2023-03-09T16:40:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:40:35 smithi038 ceph-mon[92422]: pgmap v3792: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 79 KiB/s wr, 6 op/s 2023-03-09T16:40:37.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:40:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6148.261024 secs 2023-03-09T16:40:37.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:40:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6148.261024 secs 2023-03-09T16:40:37.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:40:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6148.261024 secs 2023-03-09T16:40:38.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:40:38 smithi122 ceph-mon[93869]: pgmap v3793: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 41 KiB/s wr, 7 op/s 2023-03-09T16:40:38.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:40:38 smithi181 ceph-mon[94189]: pgmap v3793: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 41 KiB/s wr, 7 op/s 2023-03-09T16:40:38.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:40:38 smithi038 ceph-mon[92422]: pgmap v3793: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 41 KiB/s wr, 7 op/s 2023-03-09T16:40:38.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6610.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:40:40.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:40:40 smithi122 ceph-mon[93869]: pgmap v3794: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 41 KiB/s wr, 5 op/s 2023-03-09T16:40:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:40:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6151.019417 secs 2023-03-09T16:40:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:40:40 smithi181 ceph-mon[94189]: pgmap v3794: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 41 KiB/s wr, 5 op/s 2023-03-09T16:40:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:40:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6151.019417 secs 2023-03-09T16:40:40.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:40:40 smithi038 ceph-mon[92422]: pgmap v3794: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 41 KiB/s wr, 5 op/s 2023-03-09T16:40:40.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:40:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6151.019417 secs 2023-03-09T16:40:42.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:40:42 smithi122 ceph-mon[93869]: pgmap v3795: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 111 KiB/s wr, 8 op/s 2023-03-09T16:40:42.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:40:42 smithi181 ceph-mon[94189]: pgmap v3795: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 111 KiB/s wr, 8 op/s 2023-03-09T16:40:42.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:40:42 smithi038 ceph-mon[92422]: pgmap v3795: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 111 KiB/s wr, 8 op/s 2023-03-09T16:40:43.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:40:43 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:40:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:40:43 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:40:43.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:40:43 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:40:43.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6615.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:40:44.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:40:44 smithi122 ceph-mon[93869]: pgmap v3796: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 116 KiB/s wr, 6 op/s 2023-03-09T16:40:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:40:44 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:40:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:40:44 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:40:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:40:44 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:40:44.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:40:44 smithi181 ceph-mon[94189]: pgmap v3796: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 116 KiB/s wr, 6 op/s 2023-03-09T16:40:44.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:40:44 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:40:44.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:40:44 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:40:44.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:40:44 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:40:44.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:40:44 smithi038 ceph-mon[92422]: pgmap v3796: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 116 KiB/s wr, 6 op/s 2023-03-09T16:40:44.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:40:44 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:40:44.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:40:44 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:40:44.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:40:44 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:40:46.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:40:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6156.019482 secs 2023-03-09T16:40:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:40:45 smithi122 ceph-mon[93869]: pgmap v3797: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 162 KiB/s wr, 7 op/s 2023-03-09T16:40:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:40:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6156.019482 secs 2023-03-09T16:40:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:40:45 smithi181 ceph-mon[94189]: pgmap v3797: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 162 KiB/s wr, 7 op/s 2023-03-09T16:40:46.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:40:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6156.019482 secs 2023-03-09T16:40:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:40:45 smithi038 ceph-mon[92422]: pgmap v3797: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 162 KiB/s wr, 7 op/s 2023-03-09T16:40:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:40:48 smithi181 ceph-mon[94189]: pgmap v3798: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 128 KiB/s wr, 10 op/s 2023-03-09T16:40:48.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:40:48 smithi122 ceph-mon[93869]: pgmap v3798: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 128 KiB/s wr, 10 op/s 2023-03-09T16:40:48.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:40:48 smithi038 ceph-mon[92422]: pgmap v3798: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 128 KiB/s wr, 10 op/s 2023-03-09T16:40:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6620.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:40:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:40:50 smithi122 ceph-mon[93869]: pgmap v3799: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 128 KiB/s wr, 8 op/s 2023-03-09T16:40:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:40:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6161.019565 secs 2023-03-09T16:40:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:40:50 smithi181 ceph-mon[94189]: pgmap v3799: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 128 KiB/s wr, 8 op/s 2023-03-09T16:40:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:40:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6161.019565 secs 2023-03-09T16:40:50.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:40:50 smithi038 ceph-mon[92422]: pgmap v3799: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 128 KiB/s wr, 8 op/s 2023-03-09T16:40:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:40:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6161.019565 secs 2023-03-09T16:40:52.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:40:52 smithi122 ceph-mon[93869]: pgmap v3800: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 192 KiB/s wr, 11 op/s 2023-03-09T16:40:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:40:52 smithi181 ceph-mon[94189]: pgmap v3800: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 192 KiB/s wr, 11 op/s 2023-03-09T16:40:52.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:40:52 smithi038 ceph-mon[92422]: pgmap v3800: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 192 KiB/s wr, 11 op/s 2023-03-09T16:40:53.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6625.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:40:54.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:40:54 smithi122 ceph-mon[93869]: pgmap v3801: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 173 KiB/s wr, 10 op/s 2023-03-09T16:40:54.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:40:54 smithi181 ceph-mon[94189]: pgmap v3801: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 173 KiB/s wr, 10 op/s 2023-03-09T16:40:54.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:40:54 smithi038 ceph-mon[92422]: pgmap v3801: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 173 KiB/s wr, 10 op/s 2023-03-09T16:40:56.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:40:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6166.019650 secs 2023-03-09T16:40:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:40:55 smithi122 ceph-mon[93869]: pgmap v3802: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 167 KiB/s wr, 10 op/s 2023-03-09T16:40:56.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:40:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6166.019650 secs 2023-03-09T16:40:56.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:40:55 smithi181 ceph-mon[94189]: pgmap v3802: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 167 KiB/s wr, 10 op/s 2023-03-09T16:40:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:40:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6166.019650 secs 2023-03-09T16:40:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:40:55 smithi038 ceph-mon[92422]: pgmap v3802: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 167 KiB/s wr, 10 op/s 2023-03-09T16:40:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:40:58 smithi122 ceph-mon[93869]: pgmap v3803: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 122 KiB/s wr, 12 op/s 2023-03-09T16:40:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:40:58 smithi181 ceph-mon[94189]: pgmap v3803: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 122 KiB/s wr, 12 op/s 2023-03-09T16:40:58.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:40:58 smithi038 ceph-mon[92422]: pgmap v3803: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 122 KiB/s wr, 12 op/s 2023-03-09T16:40:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6630.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:41:00.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:00 smithi122 ceph-mon[93869]: pgmap v3804: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 225 KiB/s wr, 10 op/s 2023-03-09T16:41:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6171.019722 secs 2023-03-09T16:41:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:00 smithi181 ceph-mon[94189]: pgmap v3804: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 225 KiB/s wr, 10 op/s 2023-03-09T16:41:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6171.019722 secs 2023-03-09T16:41:00.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:00 smithi038 ceph-mon[92422]: pgmap v3804: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 225 KiB/s wr, 10 op/s 2023-03-09T16:41:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6171.019722 secs 2023-03-09T16:41:02.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:02 smithi122 ceph-mon[93869]: pgmap v3805: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 225 KiB/s wr, 12 op/s 2023-03-09T16:41:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6173.261616 secs 2023-03-09T16:41:02.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:02 smithi181 ceph-mon[94189]: pgmap v3805: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 225 KiB/s wr, 12 op/s 2023-03-09T16:41:02.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6173.261616 secs 2023-03-09T16:41:02.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:02 smithi038 ceph-mon[92422]: pgmap v3805: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 225 KiB/s wr, 12 op/s 2023-03-09T16:41:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6173.261616 secs 2023-03-09T16:41:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:03 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:41:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:03 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:41:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:03 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:41:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:03 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:41:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:03 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:41:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:03 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:41:03.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:03 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:41:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:03 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:41:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:03 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:41:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6635.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:41:04.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:04 smithi122 ceph-mon[93869]: pgmap v3806: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 249 KiB/s wr, 24 op/s 2023-03-09T16:41:04.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:04 smithi181 ceph-mon[94189]: pgmap v3806: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 249 KiB/s wr, 24 op/s 2023-03-09T16:41:04.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:04 smithi038 ceph-mon[92422]: pgmap v3806: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 249 KiB/s wr, 24 op/s 2023-03-09T16:41:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6176.019784 secs 2023-03-09T16:41:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:05 smithi122 ceph-mon[93869]: pgmap v3807: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 198 KiB/s wr, 22 op/s 2023-03-09T16:41:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6176.019784 secs 2023-03-09T16:41:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:05 smithi181 ceph-mon[94189]: pgmap v3807: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 198 KiB/s wr, 22 op/s 2023-03-09T16:41:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6176.019784 secs 2023-03-09T16:41:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:05 smithi038 ceph-mon[92422]: pgmap v3807: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 198 KiB/s wr, 22 op/s 2023-03-09T16:41:07.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6178.261758 secs 2023-03-09T16:41:07.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6178.261758 secs 2023-03-09T16:41:07.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6178.261758 secs 2023-03-09T16:41:08.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:08 smithi122 ceph-mon[93869]: pgmap v3808: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 221 KiB/s wr, 27 op/s 2023-03-09T16:41:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:08 smithi181 ceph-mon[94189]: pgmap v3808: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 221 KiB/s wr, 27 op/s 2023-03-09T16:41:08.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:08 smithi038 ceph-mon[92422]: pgmap v3808: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 221 KiB/s wr, 27 op/s 2023-03-09T16:41:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6640.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:41:10.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:10 smithi122 ceph-mon[93869]: osdmap e451: 12 total, 12 up, 12 in 2023-03-09T16:41:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:10 smithi122 ceph-mon[93869]: pgmap v3810: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 228 KiB/s wr, 28 op/s 2023-03-09T16:41:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6181.019843 secs 2023-03-09T16:41:10.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:10 smithi181 ceph-mon[94189]: osdmap e451: 12 total, 12 up, 12 in 2023-03-09T16:41:10.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:10 smithi181 ceph-mon[94189]: pgmap v3810: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 228 KiB/s wr, 28 op/s 2023-03-09T16:41:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6181.019843 secs 2023-03-09T16:41:10.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:10 smithi038 ceph-mon[92422]: osdmap e451: 12 total, 12 up, 12 in 2023-03-09T16:41:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:10 smithi038 ceph-mon[92422]: pgmap v3810: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 228 KiB/s wr, 28 op/s 2023-03-09T16:41:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6181.019843 secs 2023-03-09T16:41:12.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:12 smithi122 ceph-mon[93869]: pgmap v3811: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 227 KiB/s wr, 27 op/s 2023-03-09T16:41:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6183.261888 secs 2023-03-09T16:41:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:12 smithi181 ceph-mon[94189]: pgmap v3811: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 227 KiB/s wr, 27 op/s 2023-03-09T16:41:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6183.261888 secs 2023-03-09T16:41:12.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:12 smithi038 ceph-mon[92422]: pgmap v3811: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 227 KiB/s wr, 27 op/s 2023-03-09T16:41:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6183.261888 secs 2023-03-09T16:41:13.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6645.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:41:14.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:14 smithi122 ceph-mon[93869]: pgmap v3812: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 164 KiB/s wr, 11 op/s 2023-03-09T16:41:14.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:14 smithi181 ceph-mon[94189]: pgmap v3812: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 164 KiB/s wr, 11 op/s 2023-03-09T16:41:14.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:14 smithi038 ceph-mon[92422]: pgmap v3812: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 164 KiB/s wr, 11 op/s 2023-03-09T16:41:16.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6186.019906 secs 2023-03-09T16:41:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:15 smithi122 ceph-mon[93869]: pgmap v3813: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 165 KiB/s wr, 11 op/s 2023-03-09T16:41:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6186.019906 secs 2023-03-09T16:41:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:15 smithi181 ceph-mon[94189]: pgmap v3813: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 165 KiB/s wr, 11 op/s 2023-03-09T16:41:16.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6186.019906 secs 2023-03-09T16:41:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:15 smithi038 ceph-mon[92422]: pgmap v3813: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 165 KiB/s wr, 11 op/s 2023-03-09T16:41:17.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6188.262028 secs 2023-03-09T16:41:17.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6188.262028 secs 2023-03-09T16:41:17.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6188.262028 secs 2023-03-09T16:41:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:18 smithi122 ceph-mon[93869]: pgmap v3814: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 137 KiB/s wr, 8 op/s 2023-03-09T16:41:18.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:18 smithi181 ceph-mon[94189]: pgmap v3814: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 137 KiB/s wr, 8 op/s 2023-03-09T16:41:18.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:18 smithi038 ceph-mon[92422]: pgmap v3814: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 137 KiB/s wr, 8 op/s 2023-03-09T16:41:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6650.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:41:20.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:20 smithi122 ceph-mon[93869]: pgmap v3815: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 72 KiB/s wr, 7 op/s 2023-03-09T16:41:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6191.019968 secs 2023-03-09T16:41:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:20 smithi122 ceph-mon[93869]: osdmap e452: 12 total, 12 up, 12 in 2023-03-09T16:41:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:20 smithi181 ceph-mon[94189]: pgmap v3815: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 72 KiB/s wr, 7 op/s 2023-03-09T16:41:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6191.019968 secs 2023-03-09T16:41:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:20 smithi181 ceph-mon[94189]: osdmap e452: 12 total, 12 up, 12 in 2023-03-09T16:41:20.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:20 smithi038 ceph-mon[92422]: pgmap v3815: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 72 KiB/s wr, 7 op/s 2023-03-09T16:41:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6191.019968 secs 2023-03-09T16:41:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:20 smithi038 ceph-mon[92422]: osdmap e452: 12 total, 12 up, 12 in 2023-03-09T16:41:22.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:22 smithi122 ceph-mon[93869]: pgmap v3817: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 74 KiB/s wr, 7 op/s 2023-03-09T16:41:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6193.262155 secs 2023-03-09T16:41:22.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:22 smithi181 ceph-mon[94189]: pgmap v3817: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 74 KiB/s wr, 7 op/s 2023-03-09T16:41:22.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6193.262155 secs 2023-03-09T16:41:22.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:22 smithi038 ceph-mon[92422]: pgmap v3817: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 74 KiB/s wr, 7 op/s 2023-03-09T16:41:22.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6193.262155 secs 2023-03-09T16:41:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6655.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:41:24.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:24 smithi122 ceph-mon[93869]: pgmap v3818: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 33 KiB/s wr, 7 op/s 2023-03-09T16:41:24.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:24 smithi181 ceph-mon[94189]: pgmap v3818: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 33 KiB/s wr, 7 op/s 2023-03-09T16:41:24.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:24 smithi038 ceph-mon[92422]: pgmap v3818: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 33 KiB/s wr, 7 op/s 2023-03-09T16:41:26.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6196.019997 secs 2023-03-09T16:41:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:25 smithi122 ceph-mon[93869]: pgmap v3819: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T16:41:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6196.019997 secs 2023-03-09T16:41:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:25 smithi181 ceph-mon[94189]: pgmap v3819: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T16:41:26.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6196.019997 secs 2023-03-09T16:41:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:25 smithi038 ceph-mon[92422]: pgmap v3819: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T16:41:27.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6198.262268 secs 2023-03-09T16:41:27.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6198.262268 secs 2023-03-09T16:41:27.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6198.262268 secs 2023-03-09T16:41:28.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:28 smithi122 ceph-mon[93869]: pgmap v3820: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 78 KiB/s wr, 8 op/s 2023-03-09T16:41:28.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:28 smithi181 ceph-mon[94189]: pgmap v3820: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 78 KiB/s wr, 8 op/s 2023-03-09T16:41:28.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:28 smithi038 ceph-mon[92422]: pgmap v3820: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 78 KiB/s wr, 8 op/s 2023-03-09T16:41:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6660.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:41:29.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:41:29.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:41:29.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:41:30.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:30 smithi122 ceph-mon[93869]: pgmap v3821: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 48 KiB/s wr, 8 op/s 2023-03-09T16:41:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:30 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:41:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:30 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:41:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6201.020058 secs 2023-03-09T16:41:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:30 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:41:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:30 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:41:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:30 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:41:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:30 smithi181 ceph-mon[94189]: pgmap v3821: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 48 KiB/s wr, 8 op/s 2023-03-09T16:41:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:30 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:41:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:30 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:41:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6201.020058 secs 2023-03-09T16:41:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:30 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:41:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:30 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:41:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:30 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:41:30.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:30 smithi038 ceph-mon[92422]: pgmap v3821: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 48 KiB/s wr, 8 op/s 2023-03-09T16:41:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:30 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:41:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:30 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:41:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6201.020058 secs 2023-03-09T16:41:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:30 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:41:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:30 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:41:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:30 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:41:32.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:32 smithi122 ceph-mon[93869]: pgmap v3822: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 93 KiB/s wr, 10 op/s 2023-03-09T16:41:32.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:32 smithi181 ceph-mon[94189]: pgmap v3822: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 93 KiB/s wr, 10 op/s 2023-03-09T16:41:32.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:32 smithi038 ceph-mon[92422]: pgmap v3822: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 93 KiB/s wr, 10 op/s 2023-03-09T16:41:33.801 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6665.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:41:34.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:34 smithi122 ceph-mon[93869]: pgmap v3823: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 89 KiB/s wr, 8 op/s 2023-03-09T16:41:34.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:34 smithi181 ceph-mon[94189]: pgmap v3823: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 89 KiB/s wr, 8 op/s 2023-03-09T16:41:34.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:34 smithi038 ceph-mon[92422]: pgmap v3823: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 89 KiB/s wr, 8 op/s 2023-03-09T16:41:36.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6206.020118 secs 2023-03-09T16:41:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:35 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:41:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:35 smithi122 ceph-mon[93869]: pgmap v3824: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 89 KiB/s wr, 7 op/s 2023-03-09T16:41:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6206.020118 secs 2023-03-09T16:41:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:35 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:41:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:35 smithi181 ceph-mon[94189]: pgmap v3824: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 89 KiB/s wr, 7 op/s 2023-03-09T16:41:36.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6206.020118 secs 2023-03-09T16:41:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:35 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:41:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:35 smithi038 ceph-mon[92422]: pgmap v3824: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 89 KiB/s wr, 7 op/s 2023-03-09T16:41:37.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:36 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:41:37.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:41:37.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:41:37.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:36 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:41:37.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:41:37.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:41:37.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:36 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:41:37.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:41:37.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:41:38.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:37 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 6208.262499 secs 2023-03-09T16:41:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:37 smithi122 ceph-mon[93869]: pgmap v3825: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 105 KiB/s wr, 10 op/s 2023-03-09T16:41:38.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:37 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 6208.262499 secs 2023-03-09T16:41:38.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:37 smithi181 ceph-mon[94189]: pgmap v3825: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 105 KiB/s wr, 10 op/s 2023-03-09T16:41:38.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:37 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 6208.262499 secs 2023-03-09T16:41:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:37 smithi038 ceph-mon[92422]: pgmap v3825: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 105 KiB/s wr, 10 op/s 2023-03-09T16:41:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6670.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:41:40.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:41:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:41:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:40 smithi122 ceph-mon[93869]: pgmap v3826: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 66 KiB/s wr, 8 op/s 2023-03-09T16:41:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6211.020177 secs 2023-03-09T16:41:40.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:41:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:41:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:40 smithi181 ceph-mon[94189]: pgmap v3826: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 66 KiB/s wr, 8 op/s 2023-03-09T16:41:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6211.020177 secs 2023-03-09T16:41:40.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:41:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:41:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:40 smithi038 ceph-mon[92422]: pgmap v3826: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 66 KiB/s wr, 8 op/s 2023-03-09T16:41:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6211.020177 secs 2023-03-09T16:41:42.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:42 smithi122 ceph-mon[93869]: pgmap v3827: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 142 KiB/s wr, 10 op/s 2023-03-09T16:41:42.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:42 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 6213.262606 secs 2023-03-09T16:41:42.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:42 smithi181 ceph-mon[94189]: pgmap v3827: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 142 KiB/s wr, 10 op/s 2023-03-09T16:41:42.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:42 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 6213.262606 secs 2023-03-09T16:41:42.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:42 smithi038 ceph-mon[92422]: pgmap v3827: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 142 KiB/s wr, 10 op/s 2023-03-09T16:41:42.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:42 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 6213.262606 secs 2023-03-09T16:41:43.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6675.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:41:44.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:44 smithi122 ceph-mon[93869]: pgmap v3828: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 99 KiB/s wr, 10 op/s 2023-03-09T16:41:44.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:44 smithi181 ceph-mon[94189]: pgmap v3828: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 99 KiB/s wr, 10 op/s 2023-03-09T16:41:44.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:44 smithi038 ceph-mon[92422]: pgmap v3828: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 99 KiB/s wr, 10 op/s 2023-03-09T16:41:45.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6216.020237 secs 2023-03-09T16:41:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:45 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:41:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6216.020237 secs 2023-03-09T16:41:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:45 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:41:45.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6216.020237 secs 2023-03-09T16:41:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:45 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:41:46.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:46 smithi122 ceph-mon[93869]: pgmap v3829: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 99 KiB/s wr, 8 op/s 2023-03-09T16:41:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:46 smithi181 ceph-mon[94189]: pgmap v3829: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 99 KiB/s wr, 8 op/s 2023-03-09T16:41:46.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:46 smithi038 ceph-mon[92422]: pgmap v3829: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 99 KiB/s wr, 8 op/s 2023-03-09T16:41:47.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6218.262704 secs 2023-03-09T16:41:47.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6218.262704 secs 2023-03-09T16:41:47.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6218.262704 secs 2023-03-09T16:41:48.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:48 smithi122 ceph-mon[93869]: pgmap v3830: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 263 KiB/s wr, 14 op/s 2023-03-09T16:41:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:41:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:41:48.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:48 smithi181 ceph-mon[94189]: pgmap v3830: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 263 KiB/s wr, 14 op/s 2023-03-09T16:41:48.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:41:48.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:41:48.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:48 smithi038 ceph-mon[92422]: pgmap v3830: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 263 KiB/s wr, 14 op/s 2023-03-09T16:41:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:41:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:41:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6680.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:41:50.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:50 smithi122 ceph-mon[93869]: pgmap v3831: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 247 KiB/s wr, 11 op/s 2023-03-09T16:41:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6221.020298 secs 2023-03-09T16:41:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:50 smithi181 ceph-mon[94189]: pgmap v3831: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 247 KiB/s wr, 11 op/s 2023-03-09T16:41:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6221.020298 secs 2023-03-09T16:41:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:50 smithi038 ceph-mon[92422]: pgmap v3831: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 247 KiB/s wr, 11 op/s 2023-03-09T16:41:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6221.020298 secs 2023-03-09T16:41:52.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:52 smithi122 ceph-mon[93869]: pgmap v3832: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 260 KiB/s wr, 13 op/s 2023-03-09T16:41:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6223.262822 secs 2023-03-09T16:41:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:52 smithi181 ceph-mon[94189]: pgmap v3832: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 260 KiB/s wr, 13 op/s 2023-03-09T16:41:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6223.262822 secs 2023-03-09T16:41:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:52 smithi038 ceph-mon[92422]: pgmap v3832: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 260 KiB/s wr, 13 op/s 2023-03-09T16:41:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6223.262822 secs 2023-03-09T16:41:53.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6685.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:41:54.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:54 smithi122 ceph-mon[93869]: pgmap v3833: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 184 KiB/s wr, 11 op/s 2023-03-09T16:41:54.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:54 smithi181 ceph-mon[94189]: pgmap v3833: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 184 KiB/s wr, 11 op/s 2023-03-09T16:41:54.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:54 smithi038 ceph-mon[92422]: pgmap v3833: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 184 KiB/s wr, 11 op/s 2023-03-09T16:41:56.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6226.020360 secs 2023-03-09T16:41:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:55 smithi122 ceph-mon[93869]: pgmap v3834: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 254 KiB/s wr, 10 op/s 2023-03-09T16:41:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6226.020360 secs 2023-03-09T16:41:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:55 smithi181 ceph-mon[94189]: pgmap v3834: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 254 KiB/s wr, 10 op/s 2023-03-09T16:41:56.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6226.020360 secs 2023-03-09T16:41:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:55 smithi038 ceph-mon[92422]: pgmap v3834: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 254 KiB/s wr, 10 op/s 2023-03-09T16:41:57.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6228.262938 secs 2023-03-09T16:41:57.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6228.262938 secs 2023-03-09T16:41:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6228.262938 secs 2023-03-09T16:41:58.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:41:58 smithi122 ceph-mon[93869]: pgmap v3835: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 259 KiB/s wr, 13 op/s 2023-03-09T16:41:58.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:41:58 smithi181 ceph-mon[94189]: pgmap v3835: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 259 KiB/s wr, 13 op/s 2023-03-09T16:41:58.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:41:58 smithi038 ceph-mon[92422]: pgmap v3835: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 259 KiB/s wr, 13 op/s 2023-03-09T16:41:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6690.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:42:00.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:42:00 smithi122 ceph-mon[93869]: pgmap v3836: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 95 KiB/s wr, 9 op/s 2023-03-09T16:42:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:42:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6231.020422 secs 2023-03-09T16:42:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:42:00 smithi181 ceph-mon[94189]: pgmap v3836: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 95 KiB/s wr, 9 op/s 2023-03-09T16:42:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:42:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6231.020422 secs 2023-03-09T16:42:00.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:42:00 smithi038 ceph-mon[92422]: pgmap v3836: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 95 KiB/s wr, 9 op/s 2023-03-09T16:42:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:42:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6231.020422 secs 2023-03-09T16:42:02.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:42:02 smithi122 ceph-mon[93869]: pgmap v3837: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 117 KiB/s wr, 13 op/s 2023-03-09T16:42:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:42:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6233.263059 secs 2023-03-09T16:42:02.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:42:02 smithi181 ceph-mon[94189]: pgmap v3837: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 117 KiB/s wr, 13 op/s 2023-03-09T16:42:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:42:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6233.263059 secs 2023-03-09T16:42:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:42:02 smithi038 ceph-mon[92422]: pgmap v3837: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 117 KiB/s wr, 13 op/s 2023-03-09T16:42:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:42:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6233.263059 secs 2023-03-09T16:42:03.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6695.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:42:04.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:42:04 smithi122 ceph-mon[93869]: pgmap v3838: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 126 KiB/s wr, 22 op/s 2023-03-09T16:42:04.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:42:04 smithi181 ceph-mon[94189]: pgmap v3838: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 126 KiB/s wr, 22 op/s 2023-03-09T16:42:04.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:42:04 smithi038 ceph-mon[92422]: pgmap v3838: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 126 KiB/s wr, 22 op/s 2023-03-09T16:42:06.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:42:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6236.020482 secs 2023-03-09T16:42:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:42:05 smithi122 ceph-mon[93869]: pgmap v3839: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 152 KiB/s wr, 22 op/s 2023-03-09T16:42:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:42:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6236.020482 secs 2023-03-09T16:42:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:42:05 smithi181 ceph-mon[94189]: pgmap v3839: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 152 KiB/s wr, 22 op/s 2023-03-09T16:42:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:42:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6236.020482 secs 2023-03-09T16:42:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:42:05 smithi038 ceph-mon[92422]: pgmap v3839: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 152 KiB/s wr, 22 op/s 2023-03-09T16:42:07.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:42:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6238.263173 secs 2023-03-09T16:42:07.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:42:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6238.263173 secs 2023-03-09T16:42:07.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:42:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6238.263173 secs 2023-03-09T16:42:08.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:42:08 smithi122 ceph-mon[93869]: pgmap v3840: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 75 KiB/s wr, 23 op/s 2023-03-09T16:42:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:42:08 smithi181 ceph-mon[94189]: pgmap v3840: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 75 KiB/s wr, 23 op/s 2023-03-09T16:42:08.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:42:08 smithi038 ceph-mon[92422]: pgmap v3840: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 75 KiB/s wr, 23 op/s 2023-03-09T16:42:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6700.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:42:10.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:42:10 smithi122 ceph-mon[93869]: osdmap e453: 12 total, 12 up, 12 in 2023-03-09T16:42:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:42:10 smithi122 ceph-mon[93869]: pgmap v3842: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 84 KiB/s wr, 24 op/s 2023-03-09T16:42:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:42:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6241.020543 secs 2023-03-09T16:42:10.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:42:10 smithi181 ceph-mon[94189]: osdmap e453: 12 total, 12 up, 12 in 2023-03-09T16:42:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:42:10 smithi181 ceph-mon[94189]: pgmap v3842: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 84 KiB/s wr, 24 op/s 2023-03-09T16:42:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:42:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6241.020543 secs 2023-03-09T16:42:10.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:42:10 smithi038 ceph-mon[92422]: osdmap e453: 12 total, 12 up, 12 in 2023-03-09T16:42:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:42:10 smithi038 ceph-mon[92422]: pgmap v3842: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 84 KiB/s wr, 24 op/s 2023-03-09T16:42:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:42:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6241.020543 secs 2023-03-09T16:42:12.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:42:12 smithi122 ceph-mon[93869]: pgmap v3843: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 57 KiB/s wr, 21 op/s 2023-03-09T16:42:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:42:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6243.263282 secs 2023-03-09T16:42:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:42:12 smithi181 ceph-mon[94189]: pgmap v3843: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 57 KiB/s wr, 21 op/s 2023-03-09T16:42:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:42:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6243.263282 secs 2023-03-09T16:42:12.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:42:12 smithi038 ceph-mon[92422]: pgmap v3843: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 57 KiB/s wr, 21 op/s 2023-03-09T16:42:12.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:42:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6243.263282 secs 2023-03-09T16:42:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6705.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:42:14.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:42:14 smithi122 ceph-mon[93869]: pgmap v3844: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 31 KiB/s wr, 6 op/s 2023-03-09T16:42:14.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:42:14 smithi181 ceph-mon[94189]: pgmap v3844: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 31 KiB/s wr, 6 op/s 2023-03-09T16:42:14.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:42:14 smithi038 ceph-mon[92422]: pgmap v3844: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 31 KiB/s wr, 6 op/s 2023-03-09T16:42:16.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:42:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6246.020603 secs 2023-03-09T16:42:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:42:15 smithi122 ceph-mon[93869]: pgmap v3845: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 30 KiB/s wr, 6 op/s 2023-03-09T16:42:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:42:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6246.020603 secs 2023-03-09T16:42:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:42:15 smithi181 ceph-mon[94189]: pgmap v3845: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 30 KiB/s wr, 6 op/s 2023-03-09T16:42:16.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:42:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6246.020603 secs 2023-03-09T16:42:16.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:42:15 smithi038 ceph-mon[92422]: pgmap v3845: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 30 KiB/s wr, 6 op/s 2023-03-09T16:42:17.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:42:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6248.263377 secs 2023-03-09T16:42:17.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:42:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6248.263377 secs 2023-03-09T16:42:17.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:42:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6248.263377 secs 2023-03-09T16:42:18.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:42:18 smithi122 ceph-mon[93869]: pgmap v3846: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 30 KiB/s wr, 6 op/s 2023-03-09T16:42:18.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:42:18 smithi181 ceph-mon[94189]: pgmap v3846: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 30 KiB/s wr, 6 op/s 2023-03-09T16:42:18.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:42:18 smithi038 ceph-mon[92422]: pgmap v3846: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 30 KiB/s wr, 6 op/s 2023-03-09T16:42:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6710.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:42:20.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:42:20 smithi122 ceph-mon[93869]: pgmap v3847: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 29 KiB/s wr, 6 op/s 2023-03-09T16:42:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:42:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6251.020709 secs 2023-03-09T16:42:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:42:20 smithi122 ceph-mon[93869]: osdmap e454: 12 total, 12 up, 12 in 2023-03-09T16:42:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:42:20 smithi181 ceph-mon[94189]: pgmap v3847: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 29 KiB/s wr, 6 op/s 2023-03-09T16:42:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:42:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6251.020709 secs 2023-03-09T16:42:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:42:20 smithi181 ceph-mon[94189]: osdmap e454: 12 total, 12 up, 12 in 2023-03-09T16:42:20.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:42:20 smithi038 ceph-mon[92422]: pgmap v3847: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 29 KiB/s wr, 6 op/s 2023-03-09T16:42:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:42:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6251.020709 secs 2023-03-09T16:42:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:42:20 smithi038 ceph-mon[92422]: osdmap e454: 12 total, 12 up, 12 in 2023-03-09T16:42:22.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:42:22 smithi122 ceph-mon[93869]: pgmap v3849: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 30 KiB/s wr, 6 op/s 2023-03-09T16:42:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:42:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6253.263500 secs 2023-03-09T16:42:22.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:42:22 smithi181 ceph-mon[94189]: pgmap v3849: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 30 KiB/s wr, 6 op/s 2023-03-09T16:42:22.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:42:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6253.263500 secs 2023-03-09T16:42:22.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:42:22 smithi038 ceph-mon[92422]: pgmap v3849: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 30 KiB/s wr, 6 op/s 2023-03-09T16:42:22.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:42:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6253.263500 secs 2023-03-09T16:42:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6715.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:42:24.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:42:24 smithi122 ceph-mon[93869]: pgmap v3850: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 30 KiB/s wr, 7 op/s 2023-03-09T16:42:24.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:42:24 smithi181 ceph-mon[94189]: pgmap v3850: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 30 KiB/s wr, 7 op/s 2023-03-09T16:42:24.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:42:24 smithi038 ceph-mon[92422]: pgmap v3850: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 30 KiB/s wr, 7 op/s 2023-03-09T16:42:26.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:42:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6256.020770 secs 2023-03-09T16:42:26.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:42:25 smithi122 ceph-mon[93869]: pgmap v3851: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 47 KiB/s wr, 7 op/s 2023-03-09T16:42:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:42:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6256.020770 secs 2023-03-09T16:42:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:42:25 smithi181 ceph-mon[94189]: pgmap v3851: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 47 KiB/s wr, 7 op/s 2023-03-09T16:42:26.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:42:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6256.020770 secs 2023-03-09T16:42:26.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:42:25 smithi038 ceph-mon[92422]: pgmap v3851: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 47 KiB/s wr, 7 op/s 2023-03-09T16:42:27.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:42:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6258.263607 secs 2023-03-09T16:42:27.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:42:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6258.263607 secs 2023-03-09T16:42:27.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:42:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6258.263607 secs 2023-03-09T16:42:28.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:42:28 smithi122 ceph-mon[93869]: pgmap v3852: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 48 KiB/s wr, 7 op/s 2023-03-09T16:42:28.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:42:28 smithi181 ceph-mon[94189]: pgmap v3852: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 48 KiB/s wr, 7 op/s 2023-03-09T16:42:28.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:42:28 smithi038 ceph-mon[92422]: pgmap v3852: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 48 KiB/s wr, 7 op/s 2023-03-09T16:42:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6720.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:42:30.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:42:30 smithi122 ceph-mon[93869]: pgmap v3853: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 55 KiB/s wr, 8 op/s 2023-03-09T16:42:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:42:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6261.020832 secs 2023-03-09T16:42:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:42:30 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:42:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:42:30 smithi181 ceph-mon[94189]: pgmap v3853: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 55 KiB/s wr, 8 op/s 2023-03-09T16:42:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:42:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6261.020832 secs 2023-03-09T16:42:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:42:30 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:42:30.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:42:30 smithi038 ceph-mon[92422]: pgmap v3853: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 55 KiB/s wr, 8 op/s 2023-03-09T16:42:30.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:42:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6261.020832 secs 2023-03-09T16:42:30.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:42:30 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:42:31.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:42:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:42:31.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:42:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:42:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:42:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:42:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:42:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:42:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:42:31 smithi122 ceph-mon[93869]: pgmap v3854: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 47 KiB/s wr, 9 op/s 2023-03-09T16:42:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:42:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:42:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:42:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:42:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:42:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:42:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:42:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:42:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:42:31 smithi181 ceph-mon[94189]: pgmap v3854: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 47 KiB/s wr, 9 op/s 2023-03-09T16:42:31.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:42:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:42:31.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:42:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:42:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:42:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:42:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:42:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:42:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:42:31 smithi038 ceph-mon[92422]: pgmap v3854: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 47 KiB/s wr, 9 op/s 2023-03-09T16:42:33.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:42:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6263.263707 secs 2023-03-09T16:42:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:42:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:42:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:42:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:42:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:42:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:42:33.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:42:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6263.263707 secs 2023-03-09T16:42:33.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:42:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:42:33.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:42:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:42:33.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:42:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:42:33.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:42:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6263.263707 secs 2023-03-09T16:42:33.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:42:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:42:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:42:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:42:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:42:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:42:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6725.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:42:34.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:42:33 smithi122 ceph-mon[93869]: pgmap v3855: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 46 KiB/s wr, 7 op/s 2023-03-09T16:42:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:42:33 smithi181 ceph-mon[94189]: pgmap v3855: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 46 KiB/s wr, 7 op/s 2023-03-09T16:42:34.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:42:33 smithi038 ceph-mon[92422]: pgmap v3855: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 46 KiB/s wr, 7 op/s 2023-03-09T16:42:36.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:42:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6266.020917 secs 2023-03-09T16:42:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:42:35 smithi122 ceph-mon[93869]: pgmap v3856: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 150 KiB/s wr, 8 op/s 2023-03-09T16:42:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:42:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6266.020917 secs 2023-03-09T16:42:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:42:35 smithi181 ceph-mon[94189]: pgmap v3856: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 150 KiB/s wr, 8 op/s 2023-03-09T16:42:36.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:42:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6266.020917 secs 2023-03-09T16:42:36.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:42:35 smithi038 ceph-mon[92422]: pgmap v3856: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 150 KiB/s wr, 8 op/s 2023-03-09T16:42:37.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:42:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6268.263851 secs 2023-03-09T16:42:37.506 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:42:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6268.263851 secs 2023-03-09T16:42:37.506 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:42:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6268.263851 secs 2023-03-09T16:42:38.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:42:38 smithi122 ceph-mon[93869]: pgmap v3857: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 123 KiB/s wr, 10 op/s 2023-03-09T16:42:38.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:42:38 smithi181 ceph-mon[94189]: pgmap v3857: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 123 KiB/s wr, 10 op/s 2023-03-09T16:42:38.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:42:38 smithi038 ceph-mon[92422]: pgmap v3857: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 123 KiB/s wr, 10 op/s 2023-03-09T16:42:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6730.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:42:40.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:42:40 smithi122 ceph-mon[93869]: pgmap v3858: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 133 KiB/s wr, 9 op/s 2023-03-09T16:42:40.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:42:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6271.020981 secs 2023-03-09T16:42:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:42:40 smithi181 ceph-mon[94189]: pgmap v3858: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 133 KiB/s wr, 9 op/s 2023-03-09T16:42:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:42:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6271.020981 secs 2023-03-09T16:42:40.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:42:40 smithi038 ceph-mon[92422]: pgmap v3858: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 133 KiB/s wr, 9 op/s 2023-03-09T16:42:40.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:42:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6271.020981 secs 2023-03-09T16:42:42.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:42:42 smithi122 ceph-mon[93869]: pgmap v3859: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 128 KiB/s wr, 10 op/s 2023-03-09T16:42:42.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:42:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6273.263979 secs 2023-03-09T16:42:42.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:42:42 smithi181 ceph-mon[94189]: pgmap v3859: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 128 KiB/s wr, 10 op/s 2023-03-09T16:42:42.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:42:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6273.263979 secs 2023-03-09T16:42:42.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:42:42 smithi038 ceph-mon[92422]: pgmap v3859: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 128 KiB/s wr, 10 op/s 2023-03-09T16:42:42.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:42:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6273.263979 secs 2023-03-09T16:42:43.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6735.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:42:44.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:42:44 smithi122 ceph-mon[93869]: pgmap v3860: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 150 KiB/s wr, 10 op/s 2023-03-09T16:42:44.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:42:44 smithi181 ceph-mon[94189]: pgmap v3860: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 150 KiB/s wr, 10 op/s 2023-03-09T16:42:44.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:42:44 smithi038 ceph-mon[92422]: pgmap v3860: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 150 KiB/s wr, 10 op/s 2023-03-09T16:42:46.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:42:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6276.021042 secs 2023-03-09T16:42:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:42:45 smithi122 ceph-mon[93869]: pgmap v3861: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 204 KiB/s wr, 10 op/s 2023-03-09T16:42:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:42:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6276.021042 secs 2023-03-09T16:42:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:42:45 smithi181 ceph-mon[94189]: pgmap v3861: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 204 KiB/s wr, 10 op/s 2023-03-09T16:42:46.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:42:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6276.021042 secs 2023-03-09T16:42:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:42:45 smithi038 ceph-mon[92422]: pgmap v3861: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 204 KiB/s wr, 10 op/s 2023-03-09T16:42:47.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:42:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6278.264135 secs 2023-03-09T16:42:47.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:42:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6278.264135 secs 2023-03-09T16:42:47.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:42:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6278.264135 secs 2023-03-09T16:42:48.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:42:48 smithi122 ceph-mon[93869]: pgmap v3862: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 191 KiB/s wr, 13 op/s 2023-03-09T16:42:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:42:48 smithi181 ceph-mon[94189]: pgmap v3862: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 191 KiB/s wr, 13 op/s 2023-03-09T16:42:48.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:42:48 smithi038 ceph-mon[92422]: pgmap v3862: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 191 KiB/s wr, 13 op/s 2023-03-09T16:42:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6740.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:42:50.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:42:50 smithi122 ceph-mon[93869]: pgmap v3863: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 180 KiB/s wr, 10 op/s 2023-03-09T16:42:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:42:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6281.021099 secs 2023-03-09T16:42:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:42:50 smithi181 ceph-mon[94189]: pgmap v3863: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 180 KiB/s wr, 10 op/s 2023-03-09T16:42:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:42:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6281.021099 secs 2023-03-09T16:42:50.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:42:50 smithi038 ceph-mon[92422]: pgmap v3863: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 180 KiB/s wr, 10 op/s 2023-03-09T16:42:50.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:42:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6281.021099 secs 2023-03-09T16:42:52.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:42:52 smithi122 ceph-mon[93869]: pgmap v3864: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 169 KiB/s wr, 11 op/s 2023-03-09T16:42:52.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:42:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6283.264248 secs 2023-03-09T16:42:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:42:52 smithi181 ceph-mon[94189]: pgmap v3864: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 169 KiB/s wr, 11 op/s 2023-03-09T16:42:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:42:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6283.264248 secs 2023-03-09T16:42:52.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:42:52 smithi038 ceph-mon[92422]: pgmap v3864: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 169 KiB/s wr, 11 op/s 2023-03-09T16:42:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:42:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6283.264248 secs 2023-03-09T16:42:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6745.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:42:54.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:42:54 smithi122 ceph-mon[93869]: pgmap v3865: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 302 KiB/s wr, 12 op/s 2023-03-09T16:42:54.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:42:54 smithi181 ceph-mon[94189]: pgmap v3865: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 302 KiB/s wr, 12 op/s 2023-03-09T16:42:54.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:42:54 smithi038 ceph-mon[92422]: pgmap v3865: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 302 KiB/s wr, 12 op/s 2023-03-09T16:42:56.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:42:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6286.021163 secs 2023-03-09T16:42:56.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:42:55 smithi122 ceph-mon[93869]: pgmap v3866: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 280 KiB/s wr, 10 op/s 2023-03-09T16:42:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:42:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6286.021163 secs 2023-03-09T16:42:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:42:55 smithi181 ceph-mon[94189]: pgmap v3866: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 280 KiB/s wr, 10 op/s 2023-03-09T16:42:56.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:42:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6286.021163 secs 2023-03-09T16:42:56.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:42:55 smithi038 ceph-mon[92422]: pgmap v3866: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 280 KiB/s wr, 10 op/s 2023-03-09T16:42:57.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:42:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6288.264368 secs 2023-03-09T16:42:57.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:42:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6288.264368 secs 2023-03-09T16:42:57.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:42:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6288.264368 secs 2023-03-09T16:42:58.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:42:58 smithi122 ceph-mon[93869]: pgmap v3867: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 225 KiB/s wr, 13 op/s 2023-03-09T16:42:58.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:42:58 smithi181 ceph-mon[94189]: pgmap v3867: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 225 KiB/s wr, 13 op/s 2023-03-09T16:42:58.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:42:58 smithi038 ceph-mon[92422]: pgmap v3867: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 225 KiB/s wr, 13 op/s 2023-03-09T16:42:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6750.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:43:00.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:43:00 smithi122 ceph-mon[93869]: pgmap v3868: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 134 KiB/s wr, 9 op/s 2023-03-09T16:43:00.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:43:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6291.021224 secs 2023-03-09T16:43:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:43:00 smithi181 ceph-mon[94189]: pgmap v3868: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 134 KiB/s wr, 9 op/s 2023-03-09T16:43:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:43:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6291.021224 secs 2023-03-09T16:43:00.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:43:00 smithi038 ceph-mon[92422]: pgmap v3868: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 134 KiB/s wr, 9 op/s 2023-03-09T16:43:00.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:43:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6291.021224 secs 2023-03-09T16:43:02.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:43:02 smithi122 ceph-mon[93869]: pgmap v3869: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 156 KiB/s wr, 12 op/s 2023-03-09T16:43:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:43:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6293.264478 secs 2023-03-09T16:43:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:43:02 smithi181 ceph-mon[94189]: pgmap v3869: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 156 KiB/s wr, 12 op/s 2023-03-09T16:43:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:43:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6293.264478 secs 2023-03-09T16:43:02.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:43:02 smithi038 ceph-mon[92422]: pgmap v3869: 129 pgs: 129 active+clean; 8.8 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 156 KiB/s wr, 12 op/s 2023-03-09T16:43:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:43:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6293.264478 secs 2023-03-09T16:43:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6755.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:43:04.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:43:04 smithi122 ceph-mon[93869]: pgmap v3870: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 278 KiB/s wr, 24 op/s 2023-03-09T16:43:04.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:43:04 smithi181 ceph-mon[94189]: pgmap v3870: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 278 KiB/s wr, 24 op/s 2023-03-09T16:43:04.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:43:04 smithi038 ceph-mon[92422]: pgmap v3870: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 278 KiB/s wr, 24 op/s 2023-03-09T16:43:06.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:43:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6296.021287 secs 2023-03-09T16:43:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:43:05 smithi122 ceph-mon[93869]: pgmap v3871: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 145 KiB/s wr, 22 op/s 2023-03-09T16:43:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:43:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6296.021287 secs 2023-03-09T16:43:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:43:05 smithi181 ceph-mon[94189]: pgmap v3871: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 145 KiB/s wr, 22 op/s 2023-03-09T16:43:06.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:43:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6296.021287 secs 2023-03-09T16:43:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:43:05 smithi038 ceph-mon[92422]: pgmap v3871: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 145 KiB/s wr, 22 op/s 2023-03-09T16:43:07.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:43:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6298.264615 secs 2023-03-09T16:43:07.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:43:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6298.264615 secs 2023-03-09T16:43:07.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:43:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6298.264615 secs 2023-03-09T16:43:08.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:43:08 smithi122 ceph-mon[93869]: pgmap v3872: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 151 KiB/s wr, 25 op/s 2023-03-09T16:43:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:43:08 smithi181 ceph-mon[94189]: pgmap v3872: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 151 KiB/s wr, 25 op/s 2023-03-09T16:43:08.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:43:08 smithi038 ceph-mon[92422]: pgmap v3872: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 151 KiB/s wr, 25 op/s 2023-03-09T16:43:08.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6760.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:43:10.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:43:10 smithi122 ceph-mon[93869]: osdmap e455: 12 total, 12 up, 12 in 2023-03-09T16:43:10.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:43:10 smithi122 ceph-mon[93869]: pgmap v3874: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 181 KiB/s wr, 25 op/s 2023-03-09T16:43:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:43:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6301.021346 secs 2023-03-09T16:43:10.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:43:10 smithi181 ceph-mon[94189]: osdmap e455: 12 total, 12 up, 12 in 2023-03-09T16:43:10.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:43:10 smithi181 ceph-mon[94189]: pgmap v3874: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 181 KiB/s wr, 25 op/s 2023-03-09T16:43:10.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:43:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6301.021346 secs 2023-03-09T16:43:10.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:43:10 smithi038 ceph-mon[92422]: osdmap e455: 12 total, 12 up, 12 in 2023-03-09T16:43:10.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:43:10 smithi038 ceph-mon[92422]: pgmap v3874: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 181 KiB/s wr, 25 op/s 2023-03-09T16:43:10.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:43:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6301.021346 secs 2023-03-09T16:43:12.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:43:12 smithi122 ceph-mon[93869]: pgmap v3875: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 169 KiB/s wr, 23 op/s 2023-03-09T16:43:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:43:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6303.264733 secs 2023-03-09T16:43:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:43:12 smithi181 ceph-mon[94189]: pgmap v3875: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 169 KiB/s wr, 23 op/s 2023-03-09T16:43:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:43:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6303.264733 secs 2023-03-09T16:43:12.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:43:12 smithi038 ceph-mon[92422]: pgmap v3875: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 169 KiB/s wr, 23 op/s 2023-03-09T16:43:12.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:43:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6303.264733 secs 2023-03-09T16:43:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6765.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:43:14.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:43:14 smithi122 ceph-mon[93869]: pgmap v3876: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 45 KiB/s wr, 8 op/s 2023-03-09T16:43:14.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:43:14 smithi181 ceph-mon[94189]: pgmap v3876: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 45 KiB/s wr, 8 op/s 2023-03-09T16:43:14.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:43:14 smithi038 ceph-mon[92422]: pgmap v3876: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 45 KiB/s wr, 8 op/s 2023-03-09T16:43:16.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:43:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6306.021407 secs 2023-03-09T16:43:16.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:43:15 smithi122 ceph-mon[93869]: pgmap v3877: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 92 KiB/s wr, 9 op/s 2023-03-09T16:43:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:43:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6306.021407 secs 2023-03-09T16:43:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:43:15 smithi181 ceph-mon[94189]: pgmap v3877: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 92 KiB/s wr, 9 op/s 2023-03-09T16:43:16.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:43:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6306.021407 secs 2023-03-09T16:43:16.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:43:15 smithi038 ceph-mon[92422]: pgmap v3877: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 92 KiB/s wr, 9 op/s 2023-03-09T16:43:17.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:43:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6308.264869 secs 2023-03-09T16:43:17.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:43:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6308.264869 secs 2023-03-09T16:43:17.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:43:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6308.264869 secs 2023-03-09T16:43:18.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:43:18 smithi122 ceph-mon[93869]: pgmap v3878: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 86 KiB/s wr, 8 op/s 2023-03-09T16:43:18.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:43:18 smithi181 ceph-mon[94189]: pgmap v3878: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 86 KiB/s wr, 8 op/s 2023-03-09T16:43:18.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:43:18 smithi038 ceph-mon[92422]: pgmap v3878: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 86 KiB/s wr, 8 op/s 2023-03-09T16:43:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6770.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:43:20.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:43:20 smithi122 ceph-mon[93869]: pgmap v3879: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 84 KiB/s wr, 9 op/s 2023-03-09T16:43:20.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:43:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6311.021467 secs 2023-03-09T16:43:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:43:20 smithi122 ceph-mon[93869]: osdmap e456: 12 total, 12 up, 12 in 2023-03-09T16:43:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:43:20 smithi181 ceph-mon[94189]: pgmap v3879: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 84 KiB/s wr, 9 op/s 2023-03-09T16:43:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:43:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6311.021467 secs 2023-03-09T16:43:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:43:20 smithi181 ceph-mon[94189]: osdmap e456: 12 total, 12 up, 12 in 2023-03-09T16:43:20.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:43:20 smithi038 ceph-mon[92422]: pgmap v3879: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 84 KiB/s wr, 9 op/s 2023-03-09T16:43:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:43:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6311.021467 secs 2023-03-09T16:43:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:43:20 smithi038 ceph-mon[92422]: osdmap e456: 12 total, 12 up, 12 in 2023-03-09T16:43:22.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:43:22 smithi122 ceph-mon[93869]: pgmap v3881: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 71 KiB/s wr, 8 op/s 2023-03-09T16:43:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:43:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6313.264989 secs 2023-03-09T16:43:22.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:43:22 smithi181 ceph-mon[94189]: pgmap v3881: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 71 KiB/s wr, 8 op/s 2023-03-09T16:43:22.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:43:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6313.264989 secs 2023-03-09T16:43:22.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:43:22 smithi038 ceph-mon[92422]: pgmap v3881: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 71 KiB/s wr, 8 op/s 2023-03-09T16:43:22.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:43:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6313.264989 secs 2023-03-09T16:43:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6775.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:43:24.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:43:24 smithi122 ceph-mon[93869]: pgmap v3882: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 49 KiB/s wr, 8 op/s 2023-03-09T16:43:24.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:43:24 smithi181 ceph-mon[94189]: pgmap v3882: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 49 KiB/s wr, 8 op/s 2023-03-09T16:43:24.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:43:24 smithi038 ceph-mon[92422]: pgmap v3882: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 49 KiB/s wr, 8 op/s 2023-03-09T16:43:26.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:43:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6316.021526 secs 2023-03-09T16:43:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:43:25 smithi122 ceph-mon[93869]: pgmap v3883: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 89 KiB/s wr, 8 op/s 2023-03-09T16:43:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:43:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6316.021526 secs 2023-03-09T16:43:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:43:25 smithi181 ceph-mon[94189]: pgmap v3883: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 89 KiB/s wr, 8 op/s 2023-03-09T16:43:26.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:43:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6316.021526 secs 2023-03-09T16:43:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:43:25 smithi038 ceph-mon[92422]: pgmap v3883: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 89 KiB/s wr, 8 op/s 2023-03-09T16:43:27.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:43:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6318.265116 secs 2023-03-09T16:43:27.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:43:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6318.265116 secs 2023-03-09T16:43:27.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:43:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6318.265116 secs 2023-03-09T16:43:28.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:43:28 smithi122 ceph-mon[93869]: pgmap v3884: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 89 KiB/s wr, 7 op/s 2023-03-09T16:43:28.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:43:28 smithi181 ceph-mon[94189]: pgmap v3884: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 89 KiB/s wr, 7 op/s 2023-03-09T16:43:28.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:43:28 smithi038 ceph-mon[92422]: pgmap v3884: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 89 KiB/s wr, 7 op/s 2023-03-09T16:43:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6780.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:43:30.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:43:30 smithi122 ceph-mon[93869]: pgmap v3885: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 89 KiB/s wr, 7 op/s 2023-03-09T16:43:30.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:43:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6321.021612 secs 2023-03-09T16:43:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:43:30 smithi181 ceph-mon[94189]: pgmap v3885: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 89 KiB/s wr, 7 op/s 2023-03-09T16:43:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:43:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6321.021612 secs 2023-03-09T16:43:30.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:43:30 smithi038 ceph-mon[92422]: pgmap v3885: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 89 KiB/s wr, 7 op/s 2023-03-09T16:43:30.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:43:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6321.021612 secs 2023-03-09T16:43:32.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:43:32 smithi122 ceph-mon[93869]: pgmap v3886: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 77 KiB/s wr, 8 op/s 2023-03-09T16:43:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:43:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:43:32.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:43:32 smithi181 ceph-mon[94189]: pgmap v3886: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 77 KiB/s wr, 8 op/s 2023-03-09T16:43:32.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:43:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:43:32.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:43:32 smithi038 ceph-mon[92422]: pgmap v3886: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 77 KiB/s wr, 8 op/s 2023-03-09T16:43:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:43:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:43:33.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:43:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:43:33.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:43:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:43:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:43:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:43:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:43:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:43:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:43:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:43:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:43:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:43:33.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:43:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:43:33.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:43:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:43:33.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:43:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:43:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6785.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:43:34.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:43:34 smithi122 ceph-mon[93869]: pgmap v3887: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 74 KiB/s wr, 7 op/s 2023-03-09T16:43:34.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:43:34 smithi181 ceph-mon[94189]: pgmap v3887: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 74 KiB/s wr, 7 op/s 2023-03-09T16:43:34.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:43:34 smithi038 ceph-mon[92422]: pgmap v3887: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 74 KiB/s wr, 7 op/s 2023-03-09T16:43:36.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:43:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6326.021671 secs 2023-03-09T16:43:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:43:35 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:43:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:43:35 smithi122 ceph-mon[93869]: pgmap v3888: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 167 KiB/s wr, 7 op/s 2023-03-09T16:43:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:43:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6326.021671 secs 2023-03-09T16:43:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:43:35 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:43:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:43:35 smithi181 ceph-mon[94189]: pgmap v3888: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 167 KiB/s wr, 7 op/s 2023-03-09T16:43:36.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:43:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6326.021671 secs 2023-03-09T16:43:36.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:43:35 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:43:36.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:43:35 smithi038 ceph-mon[92422]: pgmap v3888: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 167 KiB/s wr, 7 op/s 2023-03-09T16:43:37.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:43:36 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:43:37.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:43:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:43:37.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:43:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:43:37.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:43:36 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:43:37.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:43:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:43:37.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:43:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:43:37.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:43:36 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:43:37.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:43:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:43:37.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:43:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:43:38.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:43:37 smithi122 ceph-mon[93869]: pgmap v3889: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 106 KiB/s wr, 10 op/s 2023-03-09T16:43:38.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:43:37 smithi181 ceph-mon[94189]: pgmap v3889: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 106 KiB/s wr, 10 op/s 2023-03-09T16:43:38.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:43:37 smithi038 ceph-mon[92422]: pgmap v3889: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 106 KiB/s wr, 10 op/s 2023-03-09T16:43:38.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6790.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:43:40.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:43:40 smithi122 ceph-mon[93869]: pgmap v3890: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 106 KiB/s wr, 41 op/s 2023-03-09T16:43:40.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:43:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6331.021733 secs 2023-03-09T16:43:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:43:40 smithi181 ceph-mon[94189]: pgmap v3890: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 106 KiB/s wr, 41 op/s 2023-03-09T16:43:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:43:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6331.021733 secs 2023-03-09T16:43:40.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:43:40 smithi038 ceph-mon[92422]: pgmap v3890: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 106 KiB/s wr, 41 op/s 2023-03-09T16:43:40.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:43:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6331.021733 secs 2023-03-09T16:43:42.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:43:42 smithi122 ceph-mon[93869]: pgmap v3891: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 208 KiB/s wr, 44 op/s 2023-03-09T16:43:42.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:43:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6333.265426 secs 2023-03-09T16:43:42.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:43:42 smithi181 ceph-mon[94189]: pgmap v3891: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 208 KiB/s wr, 44 op/s 2023-03-09T16:43:42.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:43:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6333.265426 secs 2023-03-09T16:43:42.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:43:42 smithi038 ceph-mon[92422]: pgmap v3891: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 208 KiB/s wr, 44 op/s 2023-03-09T16:43:42.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:43:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6333.265426 secs 2023-03-09T16:43:43.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:43:43 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:43:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:43:43 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:43:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:43:43 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:43:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:43:43 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:43:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:43:43 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:43:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:43:43 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:43:43.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:43:43 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:43:43.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:43:43 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:43:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:43:43 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:43:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6795.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:43:44.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:43:44 smithi122 ceph-mon[93869]: pgmap v3892: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 214 KiB/s wr, 43 op/s 2023-03-09T16:43:44.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:43:44 smithi181 ceph-mon[94189]: pgmap v3892: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 214 KiB/s wr, 43 op/s 2023-03-09T16:43:44.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:43:44 smithi038 ceph-mon[92422]: pgmap v3892: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 214 KiB/s wr, 43 op/s 2023-03-09T16:43:46.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:43:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6336.021818 secs 2023-03-09T16:43:46.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:43:45 smithi122 ceph-mon[93869]: pgmap v3893: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 283 KiB/s rd, 235 KiB/s wr, 43 op/s 2023-03-09T16:43:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:43:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6336.021818 secs 2023-03-09T16:43:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:43:45 smithi181 ceph-mon[94189]: pgmap v3893: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 283 KiB/s rd, 235 KiB/s wr, 43 op/s 2023-03-09T16:43:46.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:43:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6336.021818 secs 2023-03-09T16:43:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:43:45 smithi038 ceph-mon[92422]: pgmap v3893: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 283 KiB/s rd, 235 KiB/s wr, 43 op/s 2023-03-09T16:43:47.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:43:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6338.265515 secs 2023-03-09T16:43:47.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:43:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6338.265515 secs 2023-03-09T16:43:47.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:43:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6338.265515 secs 2023-03-09T16:43:48.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:43:48 smithi122 ceph-mon[93869]: pgmap v3894: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 197 KiB/s wr, 46 op/s 2023-03-09T16:43:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:43:48 smithi181 ceph-mon[94189]: pgmap v3894: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 197 KiB/s wr, 46 op/s 2023-03-09T16:43:48.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:43:48 smithi038 ceph-mon[92422]: pgmap v3894: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 197 KiB/s wr, 46 op/s 2023-03-09T16:43:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6800.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:43:50.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:43:50 smithi122 ceph-mon[93869]: pgmap v3895: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 185 KiB/s wr, 43 op/s 2023-03-09T16:43:50.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:43:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6341.021876 secs 2023-03-09T16:43:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:43:50 smithi181 ceph-mon[94189]: pgmap v3895: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 185 KiB/s wr, 43 op/s 2023-03-09T16:43:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:43:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6341.021876 secs 2023-03-09T16:43:50.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:43:50 smithi038 ceph-mon[92422]: pgmap v3895: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 185 KiB/s wr, 43 op/s 2023-03-09T16:43:50.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:43:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6341.021876 secs 2023-03-09T16:43:52.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:43:52 smithi122 ceph-mon[93869]: pgmap v3896: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 263 KiB/s wr, 13 op/s 2023-03-09T16:43:52.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:43:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6343.265634 secs 2023-03-09T16:43:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:43:52 smithi181 ceph-mon[94189]: pgmap v3896: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 263 KiB/s wr, 13 op/s 2023-03-09T16:43:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:43:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6343.265634 secs 2023-03-09T16:43:52.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:43:52 smithi038 ceph-mon[92422]: pgmap v3896: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 263 KiB/s wr, 13 op/s 2023-03-09T16:43:52.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:43:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6343.265634 secs 2023-03-09T16:43:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6805.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:43:54.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:43:54 smithi122 ceph-mon[93869]: pgmap v3897: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 161 KiB/s wr, 11 op/s 2023-03-09T16:43:54.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:43:54 smithi181 ceph-mon[94189]: pgmap v3897: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 161 KiB/s wr, 11 op/s 2023-03-09T16:43:54.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:43:54 smithi038 ceph-mon[92422]: pgmap v3897: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 161 KiB/s wr, 11 op/s 2023-03-09T16:43:56.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:43:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6346.021939 secs 2023-03-09T16:43:56.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:43:55 smithi122 ceph-mon[93869]: pgmap v3898: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 155 KiB/s wr, 10 op/s 2023-03-09T16:43:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:43:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6346.021939 secs 2023-03-09T16:43:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:43:55 smithi181 ceph-mon[94189]: pgmap v3898: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 155 KiB/s wr, 10 op/s 2023-03-09T16:43:56.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:43:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6346.021939 secs 2023-03-09T16:43:56.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:43:55 smithi038 ceph-mon[92422]: pgmap v3898: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 155 KiB/s wr, 10 op/s 2023-03-09T16:43:57.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:43:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6348.265753 secs 2023-03-09T16:43:57.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:43:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6348.265753 secs 2023-03-09T16:43:57.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:43:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6348.265753 secs 2023-03-09T16:43:58.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:43:58 smithi122 ceph-mon[93869]: pgmap v3899: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 221 KiB/s wr, 13 op/s 2023-03-09T16:43:58.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:43:58 smithi181 ceph-mon[94189]: pgmap v3899: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 221 KiB/s wr, 13 op/s 2023-03-09T16:43:58.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:43:58 smithi038 ceph-mon[92422]: pgmap v3899: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 221 KiB/s wr, 13 op/s 2023-03-09T16:43:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6810.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:44:00.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:44:00 smithi122 ceph-mon[93869]: pgmap v3900: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 166 KiB/s wr, 9 op/s 2023-03-09T16:44:00.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:44:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6351.021985 secs 2023-03-09T16:44:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:44:00 smithi181 ceph-mon[94189]: pgmap v3900: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 166 KiB/s wr, 9 op/s 2023-03-09T16:44:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:44:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6351.021985 secs 2023-03-09T16:44:00.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:44:00 smithi038 ceph-mon[92422]: pgmap v3900: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 166 KiB/s wr, 9 op/s 2023-03-09T16:44:00.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:44:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6351.021985 secs 2023-03-09T16:44:02.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:44:02 smithi122 ceph-mon[93869]: pgmap v3901: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 221 KiB/s wr, 14 op/s 2023-03-09T16:44:02.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:44:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6353.265888 secs 2023-03-09T16:44:02.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:44:02 smithi181 ceph-mon[94189]: pgmap v3901: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 221 KiB/s wr, 14 op/s 2023-03-09T16:44:02.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:44:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6353.265888 secs 2023-03-09T16:44:02.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:44:02 smithi038 ceph-mon[92422]: pgmap v3901: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 221 KiB/s wr, 14 op/s 2023-03-09T16:44:02.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:44:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6353.265888 secs 2023-03-09T16:44:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6815.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:44:04.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:44:04 smithi122 ceph-mon[93869]: pgmap v3902: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 165 KiB/s wr, 22 op/s 2023-03-09T16:44:04.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:44:04 smithi181 ceph-mon[94189]: pgmap v3902: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 165 KiB/s wr, 22 op/s 2023-03-09T16:44:04.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:44:04 smithi038 ceph-mon[92422]: pgmap v3902: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 165 KiB/s wr, 22 op/s 2023-03-09T16:44:06.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:44:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6356.022066 secs 2023-03-09T16:44:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:44:05 smithi122 ceph-mon[93869]: pgmap v3903: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 164 KiB/s wr, 21 op/s 2023-03-09T16:44:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:44:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6356.022066 secs 2023-03-09T16:44:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:44:05 smithi181 ceph-mon[94189]: pgmap v3903: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 164 KiB/s wr, 21 op/s 2023-03-09T16:44:06.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:44:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6356.022066 secs 2023-03-09T16:44:06.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:44:05 smithi038 ceph-mon[92422]: pgmap v3903: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 164 KiB/s wr, 21 op/s 2023-03-09T16:44:07.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:44:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6358.266023 secs 2023-03-09T16:44:07.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:44:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6358.266023 secs 2023-03-09T16:44:07.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:44:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6358.266023 secs 2023-03-09T16:44:08.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:44:08 smithi122 ceph-mon[93869]: pgmap v3904: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 164 KiB/s wr, 24 op/s 2023-03-09T16:44:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:44:08 smithi181 ceph-mon[94189]: pgmap v3904: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 164 KiB/s wr, 24 op/s 2023-03-09T16:44:08.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:44:08 smithi038 ceph-mon[92422]: pgmap v3904: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 164 KiB/s wr, 24 op/s 2023-03-09T16:44:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6820.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:44:10.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:44:10 smithi122 ceph-mon[93869]: osdmap e457: 12 total, 12 up, 12 in 2023-03-09T16:44:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:44:10 smithi122 ceph-mon[93869]: pgmap v3906: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 91 KiB/s wr, 24 op/s 2023-03-09T16:44:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:44:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6361.022129 secs 2023-03-09T16:44:10.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:44:10 smithi181 ceph-mon[94189]: osdmap e457: 12 total, 12 up, 12 in 2023-03-09T16:44:10.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:44:10 smithi181 ceph-mon[94189]: pgmap v3906: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 91 KiB/s wr, 24 op/s 2023-03-09T16:44:10.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:44:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6361.022129 secs 2023-03-09T16:44:10.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:44:10 smithi038 ceph-mon[92422]: osdmap e457: 12 total, 12 up, 12 in 2023-03-09T16:44:10.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:44:10 smithi038 ceph-mon[92422]: pgmap v3906: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 91 KiB/s wr, 24 op/s 2023-03-09T16:44:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:44:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6361.022129 secs 2023-03-09T16:44:12.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:44:12 smithi122 ceph-mon[93869]: pgmap v3907: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 64 KiB/s wr, 21 op/s 2023-03-09T16:44:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:44:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6363.266142 secs 2023-03-09T16:44:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:44:12 smithi181 ceph-mon[94189]: pgmap v3907: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 64 KiB/s wr, 21 op/s 2023-03-09T16:44:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:44:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6363.266142 secs 2023-03-09T16:44:12.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:44:12 smithi038 ceph-mon[92422]: pgmap v3907: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 64 KiB/s wr, 21 op/s 2023-03-09T16:44:12.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:44:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6363.266142 secs 2023-03-09T16:44:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6825.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:44:14.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:44:14 smithi122 ceph-mon[93869]: pgmap v3908: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 38 KiB/s wr, 6 op/s 2023-03-09T16:44:14.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:44:14 smithi181 ceph-mon[94189]: pgmap v3908: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 38 KiB/s wr, 6 op/s 2023-03-09T16:44:14.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:44:14 smithi038 ceph-mon[92422]: pgmap v3908: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 38 KiB/s wr, 6 op/s 2023-03-09T16:44:16.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:44:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6366.022215 secs 2023-03-09T16:44:16.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:44:15 smithi122 ceph-mon[93869]: pgmap v3909: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 38 KiB/s wr, 6 op/s 2023-03-09T16:44:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:44:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6366.022215 secs 2023-03-09T16:44:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:44:15 smithi181 ceph-mon[94189]: pgmap v3909: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 38 KiB/s wr, 6 op/s 2023-03-09T16:44:16.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:44:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6366.022215 secs 2023-03-09T16:44:16.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:44:15 smithi038 ceph-mon[92422]: pgmap v3909: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 38 KiB/s wr, 6 op/s 2023-03-09T16:44:17.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:44:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6368.266261 secs 2023-03-09T16:44:17.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:44:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6368.266261 secs 2023-03-09T16:44:17.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:44:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6368.266261 secs 2023-03-09T16:44:18.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:44:18 smithi122 ceph-mon[93869]: pgmap v3910: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 39 KiB/s wr, 7 op/s 2023-03-09T16:44:18.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:44:18 smithi181 ceph-mon[94189]: pgmap v3910: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 39 KiB/s wr, 7 op/s 2023-03-09T16:44:18.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:44:18 smithi038 ceph-mon[92422]: pgmap v3910: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 39 KiB/s wr, 7 op/s 2023-03-09T16:44:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6830.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:44:20.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:44:20 smithi122 ceph-mon[93869]: pgmap v3911: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 38 KiB/s wr, 7 op/s 2023-03-09T16:44:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:44:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6371.022276 secs 2023-03-09T16:44:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:44:20 smithi122 ceph-mon[93869]: osdmap e458: 12 total, 12 up, 12 in 2023-03-09T16:44:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:44:20 smithi181 ceph-mon[94189]: pgmap v3911: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 38 KiB/s wr, 7 op/s 2023-03-09T16:44:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:44:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6371.022276 secs 2023-03-09T16:44:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:44:20 smithi181 ceph-mon[94189]: osdmap e458: 12 total, 12 up, 12 in 2023-03-09T16:44:20.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:44:20 smithi038 ceph-mon[92422]: pgmap v3911: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 38 KiB/s wr, 7 op/s 2023-03-09T16:44:20.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:44:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6371.022276 secs 2023-03-09T16:44:20.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:44:20 smithi038 ceph-mon[92422]: osdmap e458: 12 total, 12 up, 12 in 2023-03-09T16:44:22.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:44:22 smithi122 ceph-mon[93869]: pgmap v3913: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 49 KiB/s wr, 7 op/s 2023-03-09T16:44:22.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:44:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6373.266323 secs 2023-03-09T16:44:22.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:44:22 smithi181 ceph-mon[94189]: pgmap v3913: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 49 KiB/s wr, 7 op/s 2023-03-09T16:44:22.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:44:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6373.266323 secs 2023-03-09T16:44:22.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:44:22 smithi038 ceph-mon[92422]: pgmap v3913: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 49 KiB/s wr, 7 op/s 2023-03-09T16:44:22.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:44:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6373.266323 secs 2023-03-09T16:44:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6835.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:44:24.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:44:24 smithi122 ceph-mon[93869]: pgmap v3914: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 49 KiB/s wr, 7 op/s 2023-03-09T16:44:24.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:44:24 smithi181 ceph-mon[94189]: pgmap v3914: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 49 KiB/s wr, 7 op/s 2023-03-09T16:44:24.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:44:24 smithi038 ceph-mon[92422]: pgmap v3914: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 49 KiB/s wr, 7 op/s 2023-03-09T16:44:26.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:44:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6376.022335 secs 2023-03-09T16:44:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:44:25 smithi122 ceph-mon[93869]: pgmap v3915: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 48 KiB/s wr, 7 op/s 2023-03-09T16:44:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:44:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6376.022335 secs 2023-03-09T16:44:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:44:25 smithi181 ceph-mon[94189]: pgmap v3915: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 48 KiB/s wr, 7 op/s 2023-03-09T16:44:26.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:44:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6376.022335 secs 2023-03-09T16:44:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:44:25 smithi038 ceph-mon[92422]: pgmap v3915: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 48 KiB/s wr, 7 op/s 2023-03-09T16:44:27.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:44:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6378.266442 secs 2023-03-09T16:44:27.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:44:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6378.266442 secs 2023-03-09T16:44:27.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:44:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6378.266442 secs 2023-03-09T16:44:28.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:44:28 smithi122 ceph-mon[93869]: pgmap v3916: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 48 KiB/s wr, 7 op/s 2023-03-09T16:44:28.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:44:28 smithi181 ceph-mon[94189]: pgmap v3916: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 48 KiB/s wr, 7 op/s 2023-03-09T16:44:28.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:44:28 smithi038 ceph-mon[92422]: pgmap v3916: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 48 KiB/s wr, 7 op/s 2023-03-09T16:44:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6840.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:44:30.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:44:30 smithi122 ceph-mon[93869]: pgmap v3917: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 48 KiB/s wr, 7 op/s 2023-03-09T16:44:30.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:44:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6381.022394 secs 2023-03-09T16:44:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:44:30 smithi181 ceph-mon[94189]: pgmap v3917: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 48 KiB/s wr, 7 op/s 2023-03-09T16:44:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:44:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6381.022394 secs 2023-03-09T16:44:30.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:44:30 smithi038 ceph-mon[92422]: pgmap v3917: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 48 KiB/s wr, 7 op/s 2023-03-09T16:44:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:44:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6381.022394 secs 2023-03-09T16:44:32.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:44:32 smithi122 ceph-mon[93869]: pgmap v3918: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 75 KiB/s wr, 9 op/s 2023-03-09T16:44:32.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:44:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6383.266594 secs 2023-03-09T16:44:32.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:44:32 smithi181 ceph-mon[94189]: pgmap v3918: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 75 KiB/s wr, 9 op/s 2023-03-09T16:44:32.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:44:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6383.266594 secs 2023-03-09T16:44:32.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:44:32 smithi038 ceph-mon[92422]: pgmap v3918: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 75 KiB/s wr, 9 op/s 2023-03-09T16:44:32.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:44:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6383.266594 secs 2023-03-09T16:44:33.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:44:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:44:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:44:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:44:33.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:44:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:44:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6845.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:44:34.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:44:34 smithi122 ceph-mon[93869]: pgmap v3919: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 32 KiB/s wr, 6 op/s 2023-03-09T16:44:34.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:44:34 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:44:34.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:44:34 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:44:34.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:44:34 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:44:34.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:44:34 smithi181 ceph-mon[94189]: pgmap v3919: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 32 KiB/s wr, 6 op/s 2023-03-09T16:44:34.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:44:34 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:44:34.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:44:34 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:44:34.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:44:34 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:44:34.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:44:34 smithi038 ceph-mon[92422]: pgmap v3919: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 32 KiB/s wr, 6 op/s 2023-03-09T16:44:34.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:44:34 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:44:34.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:44:34 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:44:34.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:44:34 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:44:36.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:44:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6386.022456 secs 2023-03-09T16:44:36.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:44:35 smithi122 ceph-mon[93869]: pgmap v3920: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 42 KiB/s wr, 6 op/s 2023-03-09T16:44:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:44:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6386.022456 secs 2023-03-09T16:44:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:44:35 smithi181 ceph-mon[94189]: pgmap v3920: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 42 KiB/s wr, 6 op/s 2023-03-09T16:44:36.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:44:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6386.022456 secs 2023-03-09T16:44:36.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:44:35 smithi038 ceph-mon[92422]: pgmap v3920: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 42 KiB/s wr, 6 op/s 2023-03-09T16:44:37.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:44:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6388.266711 secs 2023-03-09T16:44:37.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:44:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6388.266711 secs 2023-03-09T16:44:37.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:44:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6388.266711 secs 2023-03-09T16:44:38.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:44:38 smithi122 ceph-mon[93869]: pgmap v3921: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 42 KiB/s wr, 9 op/s 2023-03-09T16:44:38.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:44:38 smithi181 ceph-mon[94189]: pgmap v3921: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 42 KiB/s wr, 9 op/s 2023-03-09T16:44:38.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:44:38 smithi038 ceph-mon[92422]: pgmap v3921: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 42 KiB/s wr, 9 op/s 2023-03-09T16:44:38.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6850.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:44:40.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:44:40 smithi122 ceph-mon[93869]: pgmap v3922: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 114 KiB/s wr, 7 op/s 2023-03-09T16:44:40.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:44:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6391.022526 secs 2023-03-09T16:44:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:44:40 smithi181 ceph-mon[94189]: pgmap v3922: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 114 KiB/s wr, 7 op/s 2023-03-09T16:44:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:44:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6391.022526 secs 2023-03-09T16:44:40.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:44:40 smithi038 ceph-mon[92422]: pgmap v3922: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 114 KiB/s wr, 7 op/s 2023-03-09T16:44:40.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:44:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6391.022526 secs 2023-03-09T16:44:42.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:44:42 smithi122 ceph-mon[93869]: pgmap v3923: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 120 KiB/s wr, 10 op/s 2023-03-09T16:44:42.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:44:42 smithi181 ceph-mon[94189]: pgmap v3923: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 120 KiB/s wr, 10 op/s 2023-03-09T16:44:42.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:44:42 smithi038 ceph-mon[92422]: pgmap v3923: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 120 KiB/s wr, 10 op/s 2023-03-09T16:44:43.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:44:43 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:44:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:44:43 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:44:43.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:44:43 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:44:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6855.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:44:44.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:44:44 smithi122 ceph-mon[93869]: pgmap v3924: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 88 KiB/s wr, 7 op/s 2023-03-09T16:44:44.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:44:44 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:44:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:44:44 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:44:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:44:44 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:44:44.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:44:44 smithi181 ceph-mon[94189]: pgmap v3924: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 88 KiB/s wr, 7 op/s 2023-03-09T16:44:44.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:44:44 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:44:44.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:44:44 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:44:44.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:44:44 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:44:44.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:44:44 smithi038 ceph-mon[92422]: pgmap v3924: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 88 KiB/s wr, 7 op/s 2023-03-09T16:44:44.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:44:44 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:44:44.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:44:44 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:44:44.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:44:44 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:44:46.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:44:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6396.022586 secs 2023-03-09T16:44:46.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:44:45 smithi122 ceph-mon[93869]: pgmap v3925: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 104 KiB/s wr, 8 op/s 2023-03-09T16:44:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:44:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6396.022586 secs 2023-03-09T16:44:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:44:45 smithi181 ceph-mon[94189]: pgmap v3925: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 104 KiB/s wr, 8 op/s 2023-03-09T16:44:46.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:44:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6396.022586 secs 2023-03-09T16:44:46.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:44:45 smithi038 ceph-mon[92422]: pgmap v3925: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 104 KiB/s wr, 8 op/s 2023-03-09T16:44:48.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:44:48 smithi122 ceph-mon[93869]: pgmap v3926: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 100 KiB/s wr, 11 op/s 2023-03-09T16:44:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:44:48 smithi181 ceph-mon[94189]: pgmap v3926: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 100 KiB/s wr, 11 op/s 2023-03-09T16:44:48.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:44:48 smithi038 ceph-mon[92422]: pgmap v3926: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 100 KiB/s wr, 11 op/s 2023-03-09T16:44:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6860.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:44:50.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:44:50 smithi122 ceph-mon[93869]: pgmap v3927: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 243 KiB/s wr, 10 op/s 2023-03-09T16:44:50.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:44:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6401.022648 secs 2023-03-09T16:44:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:44:50 smithi181 ceph-mon[94189]: pgmap v3927: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 243 KiB/s wr, 10 op/s 2023-03-09T16:44:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:44:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6401.022648 secs 2023-03-09T16:44:50.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:44:50 smithi038 ceph-mon[92422]: pgmap v3927: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 243 KiB/s wr, 10 op/s 2023-03-09T16:44:50.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:44:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6401.022648 secs 2023-03-09T16:44:52.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:44:52 smithi122 ceph-mon[93869]: pgmap v3928: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 177 KiB/s wr, 12 op/s 2023-03-09T16:44:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:44:52 smithi181 ceph-mon[94189]: pgmap v3928: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 177 KiB/s wr, 12 op/s 2023-03-09T16:44:52.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:44:52 smithi038 ceph-mon[92422]: pgmap v3928: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 177 KiB/s wr, 12 op/s 2023-03-09T16:44:53.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6865.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:44:54.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:44:54 smithi122 ceph-mon[93869]: pgmap v3929: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 172 KiB/s wr, 9 op/s 2023-03-09T16:44:54.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:44:54 smithi181 ceph-mon[94189]: pgmap v3929: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 172 KiB/s wr, 9 op/s 2023-03-09T16:44:54.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:44:54 smithi038 ceph-mon[92422]: pgmap v3929: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 172 KiB/s wr, 9 op/s 2023-03-09T16:44:56.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:44:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6406.022729 secs 2023-03-09T16:44:56.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:44:55 smithi122 ceph-mon[93869]: pgmap v3930: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 219 KiB/s wr, 10 op/s 2023-03-09T16:44:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:44:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6406.022729 secs 2023-03-09T16:44:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:44:55 smithi181 ceph-mon[94189]: pgmap v3930: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 219 KiB/s wr, 10 op/s 2023-03-09T16:44:56.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:44:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6406.022729 secs 2023-03-09T16:44:56.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:44:55 smithi038 ceph-mon[92422]: pgmap v3930: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 219 KiB/s wr, 10 op/s 2023-03-09T16:44:58.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:44:58 smithi122 ceph-mon[93869]: pgmap v3931: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 203 KiB/s wr, 12 op/s 2023-03-09T16:44:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:44:58 smithi181 ceph-mon[94189]: pgmap v3931: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 203 KiB/s wr, 12 op/s 2023-03-09T16:44:58.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:44:58 smithi038 ceph-mon[92422]: pgmap v3931: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 203 KiB/s wr, 12 op/s 2023-03-09T16:44:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6870.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:45:00.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:45:00 smithi122 ceph-mon[93869]: pgmap v3932: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 251 KiB/s wr, 10 op/s 2023-03-09T16:45:00.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:45:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6411.022790 secs 2023-03-09T16:45:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:45:00 smithi181 ceph-mon[94189]: pgmap v3932: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 251 KiB/s wr, 10 op/s 2023-03-09T16:45:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:45:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6411.022790 secs 2023-03-09T16:45:00.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:45:00 smithi038 ceph-mon[92422]: pgmap v3932: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 251 KiB/s wr, 10 op/s 2023-03-09T16:45:00.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:45:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6411.022790 secs 2023-03-09T16:45:02.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:45:02 smithi122 ceph-mon[93869]: pgmap v3933: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 141 KiB/s wr, 13 op/s 2023-03-09T16:45:02.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:45:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6413.267185 secs 2023-03-09T16:45:02.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:45:02 smithi181 ceph-mon[94189]: pgmap v3933: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 141 KiB/s wr, 13 op/s 2023-03-09T16:45:02.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:45:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6413.267185 secs 2023-03-09T16:45:02.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:45:02 smithi038 ceph-mon[92422]: pgmap v3933: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 141 KiB/s wr, 13 op/s 2023-03-09T16:45:02.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:45:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6413.267185 secs 2023-03-09T16:45:03.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:45:03 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:45:03.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:45:03 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:45:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:45:03 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:45:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:45:03 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:45:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:45:03 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:45:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:45:03 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:45:03.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:45:03 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:45:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:45:03 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:45:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:45:03 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:45:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6875.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:45:04.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:45:04 smithi181 ceph-mon[94189]: pgmap v3934: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 334 KiB/s wr, 24 op/s 2023-03-09T16:45:04.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:45:04 smithi038 ceph-mon[92422]: pgmap v3934: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 334 KiB/s wr, 24 op/s 2023-03-09T16:45:04.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:45:04 smithi122 ceph-mon[93869]: pgmap v3934: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 334 KiB/s wr, 24 op/s 2023-03-09T16:45:06.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:45:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6416.022852 secs 2023-03-09T16:45:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:45:05 smithi122 ceph-mon[93869]: pgmap v3935: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 333 KiB/s wr, 24 op/s 2023-03-09T16:45:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:45:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6416.022852 secs 2023-03-09T16:45:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:45:05 smithi181 ceph-mon[94189]: pgmap v3935: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 333 KiB/s wr, 24 op/s 2023-03-09T16:45:06.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:45:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6416.022852 secs 2023-03-09T16:45:06.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:45:05 smithi038 ceph-mon[92422]: pgmap v3935: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 333 KiB/s wr, 24 op/s 2023-03-09T16:45:07.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:45:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6418.267301 secs 2023-03-09T16:45:07.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:45:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6418.267301 secs 2023-03-09T16:45:07.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:45:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6418.267301 secs 2023-03-09T16:45:08.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:45:08 smithi122 ceph-mon[93869]: pgmap v3936: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 319 KiB/s wr, 27 op/s 2023-03-09T16:45:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:45:08 smithi181 ceph-mon[94189]: pgmap v3936: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 319 KiB/s wr, 27 op/s 2023-03-09T16:45:08.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:45:08 smithi038 ceph-mon[92422]: pgmap v3936: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 319 KiB/s wr, 27 op/s 2023-03-09T16:45:08.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6880.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:45:10.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:45:10 smithi122 ceph-mon[93869]: osdmap e459: 12 total, 12 up, 12 in 2023-03-09T16:45:10.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:45:10 smithi122 ceph-mon[93869]: pgmap v3938: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 318 KiB/s wr, 28 op/s 2023-03-09T16:45:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:45:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6421.022916 secs 2023-03-09T16:45:10.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:45:10 smithi181 ceph-mon[94189]: osdmap e459: 12 total, 12 up, 12 in 2023-03-09T16:45:10.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:45:10 smithi181 ceph-mon[94189]: pgmap v3938: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 318 KiB/s wr, 28 op/s 2023-03-09T16:45:10.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:45:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6421.022916 secs 2023-03-09T16:45:10.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:45:10 smithi038 ceph-mon[92422]: osdmap e459: 12 total, 12 up, 12 in 2023-03-09T16:45:10.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:45:10 smithi038 ceph-mon[92422]: pgmap v3938: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 318 KiB/s wr, 28 op/s 2023-03-09T16:45:10.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:45:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6421.022916 secs 2023-03-09T16:45:12.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:45:12 smithi122 ceph-mon[93869]: pgmap v3939: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 277 KiB/s wr, 25 op/s 2023-03-09T16:45:12.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:45:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6423.267423 secs 2023-03-09T16:45:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:45:12 smithi181 ceph-mon[94189]: pgmap v3939: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 277 KiB/s wr, 25 op/s 2023-03-09T16:45:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:45:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6423.267423 secs 2023-03-09T16:45:12.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:45:12 smithi038 ceph-mon[92422]: pgmap v3939: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 277 KiB/s wr, 25 op/s 2023-03-09T16:45:12.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:45:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6423.267423 secs 2023-03-09T16:45:13.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6885.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:45:14.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:45:14 smithi122 ceph-mon[93869]: pgmap v3940: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 151 KiB/s wr, 9 op/s 2023-03-09T16:45:14.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:45:14 smithi181 ceph-mon[94189]: pgmap v3940: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 151 KiB/s wr, 9 op/s 2023-03-09T16:45:14.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:45:14 smithi038 ceph-mon[92422]: pgmap v3940: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 151 KiB/s wr, 9 op/s 2023-03-09T16:45:16.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:45:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6426.022978 secs 2023-03-09T16:45:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:45:15 smithi122 ceph-mon[93869]: pgmap v3941: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 150 KiB/s wr, 8 op/s 2023-03-09T16:45:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:45:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6426.022978 secs 2023-03-09T16:45:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:45:15 smithi181 ceph-mon[94189]: pgmap v3941: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 150 KiB/s wr, 8 op/s 2023-03-09T16:45:16.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:45:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6426.022978 secs 2023-03-09T16:45:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:45:15 smithi038 ceph-mon[92422]: pgmap v3941: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 150 KiB/s wr, 8 op/s 2023-03-09T16:45:17.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:45:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6428.267539 secs 2023-03-09T16:45:17.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:45:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6428.267539 secs 2023-03-09T16:45:17.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:45:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6428.267539 secs 2023-03-09T16:45:18.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:45:18 smithi122 ceph-mon[93869]: pgmap v3942: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 149 KiB/s wr, 8 op/s 2023-03-09T16:45:18.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:45:18 smithi181 ceph-mon[94189]: pgmap v3942: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 149 KiB/s wr, 8 op/s 2023-03-09T16:45:18.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:45:18 smithi038 ceph-mon[92422]: pgmap v3942: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 149 KiB/s wr, 8 op/s 2023-03-09T16:45:18.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6890.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:45:20.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:45:20 smithi122 ceph-mon[93869]: pgmap v3943: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 147 KiB/s wr, 7 op/s 2023-03-09T16:45:20.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:45:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6431.023039 secs 2023-03-09T16:45:20.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:45:20 smithi122 ceph-mon[93869]: osdmap e460: 12 total, 12 up, 12 in 2023-03-09T16:45:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:45:20 smithi181 ceph-mon[94189]: pgmap v3943: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 147 KiB/s wr, 7 op/s 2023-03-09T16:45:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:45:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6431.023039 secs 2023-03-09T16:45:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:45:20 smithi181 ceph-mon[94189]: osdmap e460: 12 total, 12 up, 12 in 2023-03-09T16:45:20.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:45:20 smithi038 ceph-mon[92422]: pgmap v3943: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 147 KiB/s wr, 7 op/s 2023-03-09T16:45:20.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:45:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6431.023039 secs 2023-03-09T16:45:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:45:20 smithi038 ceph-mon[92422]: osdmap e460: 12 total, 12 up, 12 in 2023-03-09T16:45:22.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:45:22 smithi122 ceph-mon[93869]: pgmap v3945: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 150 KiB/s wr, 8 op/s 2023-03-09T16:45:22.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:45:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6433.267646 secs 2023-03-09T16:45:22.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:45:22 smithi181 ceph-mon[94189]: pgmap v3945: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 150 KiB/s wr, 8 op/s 2023-03-09T16:45:22.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:45:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6433.267646 secs 2023-03-09T16:45:22.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:45:22 smithi038 ceph-mon[92422]: pgmap v3945: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 150 KiB/s wr, 8 op/s 2023-03-09T16:45:22.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:45:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6433.267646 secs 2023-03-09T16:45:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6895.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:45:24.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:45:24 smithi122 ceph-mon[93869]: pgmap v3946: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 61 KiB/s wr, 7 op/s 2023-03-09T16:45:24.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:45:24 smithi181 ceph-mon[94189]: pgmap v3946: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 61 KiB/s wr, 7 op/s 2023-03-09T16:45:24.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:45:24 smithi038 ceph-mon[92422]: pgmap v3946: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 61 KiB/s wr, 7 op/s 2023-03-09T16:45:26.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:45:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6436.023097 secs 2023-03-09T16:45:26.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:45:25 smithi122 ceph-mon[93869]: pgmap v3947: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 61 KiB/s wr, 8 op/s 2023-03-09T16:45:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:45:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6436.023097 secs 2023-03-09T16:45:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:45:25 smithi181 ceph-mon[94189]: pgmap v3947: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 61 KiB/s wr, 8 op/s 2023-03-09T16:45:26.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:45:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6436.023097 secs 2023-03-09T16:45:26.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:45:25 smithi038 ceph-mon[92422]: pgmap v3947: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 61 KiB/s wr, 8 op/s 2023-03-09T16:45:27.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:45:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6438.267765 secs 2023-03-09T16:45:27.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:45:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6438.267765 secs 2023-03-09T16:45:27.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:45:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6438.267765 secs 2023-03-09T16:45:28.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:45:28 smithi122 ceph-mon[93869]: pgmap v3948: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 51 KiB/s wr, 7 op/s 2023-03-09T16:45:28.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:45:28 smithi181 ceph-mon[94189]: pgmap v3948: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 51 KiB/s wr, 7 op/s 2023-03-09T16:45:28.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:45:28 smithi038 ceph-mon[92422]: pgmap v3948: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 51 KiB/s wr, 7 op/s 2023-03-09T16:45:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6900.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:45:30.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:45:30 smithi122 ceph-mon[93869]: pgmap v3949: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 8 op/s 2023-03-09T16:45:30.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:45:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6441.023159 secs 2023-03-09T16:45:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:45:30 smithi181 ceph-mon[94189]: pgmap v3949: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 8 op/s 2023-03-09T16:45:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:45:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6441.023159 secs 2023-03-09T16:45:30.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:45:30 smithi038 ceph-mon[92422]: pgmap v3949: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 8 op/s 2023-03-09T16:45:30.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:45:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6441.023159 secs 2023-03-09T16:45:32.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:45:32 smithi122 ceph-mon[93869]: pgmap v3950: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 43 KiB/s wr, 9 op/s 2023-03-09T16:45:32.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:45:32 smithi181 ceph-mon[94189]: pgmap v3950: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 43 KiB/s wr, 9 op/s 2023-03-09T16:45:32.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:45:32 smithi038 ceph-mon[92422]: pgmap v3950: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 43 KiB/s wr, 9 op/s 2023-03-09T16:45:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6905.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:45:34.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:45:34 smithi122 ceph-mon[93869]: pgmap v3951: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 42 KiB/s wr, 8 op/s 2023-03-09T16:45:34.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:45:34 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:45:34.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:45:34 smithi181 ceph-mon[94189]: pgmap v3951: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 42 KiB/s wr, 8 op/s 2023-03-09T16:45:34.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:45:34 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:45:34.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:45:34 smithi038 ceph-mon[92422]: pgmap v3951: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 42 KiB/s wr, 8 op/s 2023-03-09T16:45:34.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:45:34 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:45:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:45:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6446.023222 secs 2023-03-09T16:45:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:45:35 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:45:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:45:35 smithi038 ceph-mon[92422]: pgmap v3952: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T16:45:36.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:45:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6446.023222 secs 2023-03-09T16:45:36.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:45:35 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:45:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:45:35 smithi122 ceph-mon[93869]: pgmap v3952: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T16:45:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:45:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6446.023222 secs 2023-03-09T16:45:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:45:35 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:45:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:45:35 smithi181 ceph-mon[94189]: pgmap v3952: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T16:45:37.174 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:45:36 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:45:37.175 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:45:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:45:37.175 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:45:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:45:37.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:45:36 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:45:37.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:45:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:45:37.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:45:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:45:37.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:45:36 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:45:37.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:45:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:45:37.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:45:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:45:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6910.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:45:38.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:45:38 smithi122 ceph-mon[93869]: pgmap v3953: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 29 KiB/s wr, 10 op/s 2023-03-09T16:45:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:45:38 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:45:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:45:38 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:45:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:45:38 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:45:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:45:38 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:45:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:45:38 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:45:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:45:38 smithi181 ceph-mon[94189]: pgmap v3953: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 29 KiB/s wr, 10 op/s 2023-03-09T16:45:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:45:38 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:45:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:45:38 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:45:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:45:38 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:45:38.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:45:38 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:45:38.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:45:38 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:45:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:45:38 smithi038 ceph-mon[92422]: pgmap v3953: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 29 KiB/s wr, 10 op/s 2023-03-09T16:45:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:45:38 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:45:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:45:38 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:45:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:45:38 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:45:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:45:38 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:45:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:45:38 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:45:40.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:45:40 smithi122 ceph-mon[93869]: pgmap v3954: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 104 KiB/s wr, 8 op/s 2023-03-09T16:45:40.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:45:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6451.023283 secs 2023-03-09T16:45:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:45:40 smithi181 ceph-mon[94189]: pgmap v3954: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 104 KiB/s wr, 8 op/s 2023-03-09T16:45:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:45:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6451.023283 secs 2023-03-09T16:45:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:45:40 smithi038 ceph-mon[92422]: pgmap v3954: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 104 KiB/s wr, 8 op/s 2023-03-09T16:45:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:45:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6451.023283 secs 2023-03-09T16:45:42.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:45:42 smithi122 ceph-mon[93869]: pgmap v3955: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 111 KiB/s wr, 11 op/s 2023-03-09T16:45:42.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:45:42 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 6453.268159 secs 2023-03-09T16:45:42.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:45:42 smithi181 ceph-mon[94189]: pgmap v3955: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 111 KiB/s wr, 11 op/s 2023-03-09T16:45:42.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:45:42 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 6453.268159 secs 2023-03-09T16:45:42.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:45:42 smithi038 ceph-mon[92422]: pgmap v3955: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 111 KiB/s wr, 11 op/s 2023-03-09T16:45:42.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:45:42 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 6453.268159 secs 2023-03-09T16:45:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6915.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:45:43.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:45:43 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:45:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:45:43 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:45:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:45:43 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:45:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:45:43 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:45:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:45:43 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:45:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:45:43 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:45:43.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:45:43 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:45:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:45:43 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:45:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:45:43 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:45:44.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:45:44 smithi122 ceph-mon[93869]: pgmap v3956: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 224 KiB/s wr, 11 op/s 2023-03-09T16:45:44.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:45:44 smithi181 ceph-mon[94189]: pgmap v3956: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 224 KiB/s wr, 11 op/s 2023-03-09T16:45:44.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:45:44 smithi038 ceph-mon[92422]: pgmap v3956: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 224 KiB/s wr, 11 op/s 2023-03-09T16:45:46.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:45:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6456.023342 secs 2023-03-09T16:45:46.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:45:45 smithi122 ceph-mon[93869]: pgmap v3957: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 224 KiB/s wr, 9 op/s 2023-03-09T16:45:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:45:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6456.023342 secs 2023-03-09T16:45:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:45:45 smithi181 ceph-mon[94189]: pgmap v3957: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 224 KiB/s wr, 9 op/s 2023-03-09T16:45:46.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:45:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6456.023342 secs 2023-03-09T16:45:46.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:45:45 smithi038 ceph-mon[92422]: pgmap v3957: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 224 KiB/s wr, 9 op/s 2023-03-09T16:45:47.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:45:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6458.268287 secs 2023-03-09T16:45:47.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:45:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6458.268287 secs 2023-03-09T16:45:47.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:45:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6458.268287 secs 2023-03-09T16:45:48.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:45:48 smithi122 ceph-mon[93869]: pgmap v3958: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 230 KiB/s wr, 12 op/s 2023-03-09T16:45:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:45:48 smithi181 ceph-mon[94189]: pgmap v3958: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 230 KiB/s wr, 12 op/s 2023-03-09T16:45:48.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:45:48 smithi038 ceph-mon[92422]: pgmap v3958: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 230 KiB/s wr, 12 op/s 2023-03-09T16:45:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6920.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:45:49.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:45:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:45:49.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:45:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:45:49.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:45:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:45:49.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:45:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:45:49.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:45:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:45:49.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:45:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:45:50.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:45:50 smithi122 ceph-mon[93869]: pgmap v3959: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 261 KiB/s wr, 11 op/s 2023-03-09T16:45:50.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:45:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6461.023418 secs 2023-03-09T16:45:50.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:45:50 smithi181 ceph-mon[94189]: pgmap v3959: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 261 KiB/s wr, 11 op/s 2023-03-09T16:45:50.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:45:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6461.023418 secs 2023-03-09T16:45:50.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:45:50 smithi038 ceph-mon[92422]: pgmap v3959: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 261 KiB/s wr, 11 op/s 2023-03-09T16:45:50.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:45:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6461.023418 secs 2023-03-09T16:45:52.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:45:52 smithi122 ceph-mon[93869]: pgmap v3960: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 170 KiB/s wr, 12 op/s 2023-03-09T16:45:52.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:45:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6463.268428 secs 2023-03-09T16:45:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:45:52 smithi181 ceph-mon[94189]: pgmap v3960: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 170 KiB/s wr, 12 op/s 2023-03-09T16:45:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:45:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6463.268428 secs 2023-03-09T16:45:52.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:45:52 smithi038 ceph-mon[92422]: pgmap v3960: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 170 KiB/s wr, 12 op/s 2023-03-09T16:45:52.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:45:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6463.268428 secs 2023-03-09T16:45:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6925.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:45:54.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:45:54 smithi122 ceph-mon[93869]: pgmap v3961: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 240 KiB/s wr, 10 op/s 2023-03-09T16:45:54.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:45:54 smithi181 ceph-mon[94189]: pgmap v3961: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 240 KiB/s wr, 10 op/s 2023-03-09T16:45:54.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:45:54 smithi038 ceph-mon[92422]: pgmap v3961: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 240 KiB/s wr, 10 op/s 2023-03-09T16:45:56.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:45:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6466.023478 secs 2023-03-09T16:45:56.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:45:55 smithi122 ceph-mon[93869]: pgmap v3962: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 127 KiB/s wr, 8 op/s 2023-03-09T16:45:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:45:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6466.023478 secs 2023-03-09T16:45:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:45:55 smithi181 ceph-mon[94189]: pgmap v3962: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 127 KiB/s wr, 8 op/s 2023-03-09T16:45:56.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:45:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6466.023478 secs 2023-03-09T16:45:56.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:45:55 smithi038 ceph-mon[92422]: pgmap v3962: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 127 KiB/s wr, 8 op/s 2023-03-09T16:45:57.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:45:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6468.268562 secs 2023-03-09T16:45:57.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:45:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6468.268562 secs 2023-03-09T16:45:57.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:45:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6468.268562 secs 2023-03-09T16:45:58.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:45:58 smithi122 ceph-mon[93869]: pgmap v3963: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 219 KiB/s wr, 13 op/s 2023-03-09T16:45:58.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:45:58 smithi181 ceph-mon[94189]: pgmap v3963: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 219 KiB/s wr, 13 op/s 2023-03-09T16:45:58.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:45:58 smithi038 ceph-mon[92422]: pgmap v3963: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 219 KiB/s wr, 13 op/s 2023-03-09T16:45:58.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6930.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:46:00.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:46:00 smithi122 ceph-mon[93869]: pgmap v3964: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 213 KiB/s wr, 10 op/s 2023-03-09T16:46:00.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:46:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6471.023561 secs 2023-03-09T16:46:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:46:00 smithi181 ceph-mon[94189]: pgmap v3964: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 213 KiB/s wr, 10 op/s 2023-03-09T16:46:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:46:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6471.023561 secs 2023-03-09T16:46:00.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:46:00 smithi038 ceph-mon[92422]: pgmap v3964: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 213 KiB/s wr, 10 op/s 2023-03-09T16:46:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:46:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6471.023561 secs 2023-03-09T16:46:02.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:46:02 smithi122 ceph-mon[93869]: pgmap v3965: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 262 KiB/s wr, 14 op/s 2023-03-09T16:46:02.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:46:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6473.268668 secs 2023-03-09T16:46:02.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:46:02 smithi181 ceph-mon[94189]: pgmap v3965: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 262 KiB/s wr, 14 op/s 2023-03-09T16:46:02.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:46:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6473.268668 secs 2023-03-09T16:46:02.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:46:02 smithi038 ceph-mon[92422]: pgmap v3965: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 262 KiB/s wr, 14 op/s 2023-03-09T16:46:02.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:46:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6473.268668 secs 2023-03-09T16:46:03.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6935.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:46:04.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:46:04 smithi122 ceph-mon[93869]: pgmap v3966: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 276 KiB/s wr, 24 op/s 2023-03-09T16:46:04.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:46:04 smithi181 ceph-mon[94189]: pgmap v3966: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 276 KiB/s wr, 24 op/s 2023-03-09T16:46:04.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:46:04 smithi038 ceph-mon[92422]: pgmap v3966: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 276 KiB/s wr, 24 op/s 2023-03-09T16:46:06.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:46:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6476.023622 secs 2023-03-09T16:46:06.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:46:05 smithi122 ceph-mon[93869]: pgmap v3967: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 200 KiB/s wr, 22 op/s 2023-03-09T16:46:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:46:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6476.023622 secs 2023-03-09T16:46:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:46:05 smithi181 ceph-mon[94189]: pgmap v3967: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 200 KiB/s wr, 22 op/s 2023-03-09T16:46:06.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:46:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6476.023622 secs 2023-03-09T16:46:06.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:46:05 smithi038 ceph-mon[92422]: pgmap v3967: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 200 KiB/s wr, 22 op/s 2023-03-09T16:46:07.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:46:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6478.268764 secs 2023-03-09T16:46:07.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:46:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6478.268764 secs 2023-03-09T16:46:07.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:46:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6478.268764 secs 2023-03-09T16:46:08.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:46:08 smithi122 ceph-mon[93869]: pgmap v3968: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 200 KiB/s wr, 25 op/s 2023-03-09T16:46:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:46:08 smithi181 ceph-mon[94189]: pgmap v3968: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 200 KiB/s wr, 25 op/s 2023-03-09T16:46:08.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:46:08 smithi038 ceph-mon[92422]: pgmap v3968: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 200 KiB/s wr, 25 op/s 2023-03-09T16:46:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6940.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:46:10.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:46:10 smithi122 ceph-mon[93869]: osdmap e461: 12 total, 12 up, 12 in 2023-03-09T16:46:10.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:46:10 smithi122 ceph-mon[93869]: pgmap v3970: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 129 KiB/s wr, 24 op/s 2023-03-09T16:46:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:46:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6481.023712 secs 2023-03-09T16:46:10.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:46:10 smithi181 ceph-mon[94189]: osdmap e461: 12 total, 12 up, 12 in 2023-03-09T16:46:10.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:46:10 smithi181 ceph-mon[94189]: pgmap v3970: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 129 KiB/s wr, 24 op/s 2023-03-09T16:46:10.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:46:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6481.023712 secs 2023-03-09T16:46:10.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:46:10 smithi038 ceph-mon[92422]: osdmap e461: 12 total, 12 up, 12 in 2023-03-09T16:46:10.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:46:10 smithi038 ceph-mon[92422]: pgmap v3970: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 129 KiB/s wr, 24 op/s 2023-03-09T16:46:10.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:46:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6481.023712 secs 2023-03-09T16:46:12.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:46:12 smithi122 ceph-mon[93869]: pgmap v3971: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 64 KiB/s wr, 21 op/s 2023-03-09T16:46:12.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:46:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6483.268900 secs 2023-03-09T16:46:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:46:12 smithi181 ceph-mon[94189]: pgmap v3971: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 64 KiB/s wr, 21 op/s 2023-03-09T16:46:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:46:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6483.268900 secs 2023-03-09T16:46:12.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:46:12 smithi038 ceph-mon[92422]: pgmap v3971: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 64 KiB/s wr, 21 op/s 2023-03-09T16:46:12.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:46:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6483.268900 secs 2023-03-09T16:46:13.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6945.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:46:14.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:46:14 smithi122 ceph-mon[93869]: pgmap v3972: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 39 KiB/s wr, 6 op/s 2023-03-09T16:46:14.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:46:14 smithi181 ceph-mon[94189]: pgmap v3972: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 39 KiB/s wr, 6 op/s 2023-03-09T16:46:14.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:46:14 smithi038 ceph-mon[92422]: pgmap v3972: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 39 KiB/s wr, 6 op/s 2023-03-09T16:46:16.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:46:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6486.023803 secs 2023-03-09T16:46:16.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:46:15 smithi122 ceph-mon[93869]: pgmap v3973: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 39 KiB/s wr, 6 op/s 2023-03-09T16:46:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:46:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6486.023803 secs 2023-03-09T16:46:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:46:15 smithi181 ceph-mon[94189]: pgmap v3973: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 39 KiB/s wr, 6 op/s 2023-03-09T16:46:16.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:46:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6486.023803 secs 2023-03-09T16:46:16.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:46:15 smithi038 ceph-mon[92422]: pgmap v3973: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 39 KiB/s wr, 6 op/s 2023-03-09T16:46:17.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:46:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6488.269042 secs 2023-03-09T16:46:17.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:46:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6488.269042 secs 2023-03-09T16:46:17.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:46:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6488.269042 secs 2023-03-09T16:46:18.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:46:18 smithi122 ceph-mon[93869]: pgmap v3974: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 39 KiB/s wr, 7 op/s 2023-03-09T16:46:18.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:46:18 smithi181 ceph-mon[94189]: pgmap v3974: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 39 KiB/s wr, 7 op/s 2023-03-09T16:46:18.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:46:18 smithi038 ceph-mon[92422]: pgmap v3974: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 39 KiB/s wr, 7 op/s 2023-03-09T16:46:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6950.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:46:20.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:46:20 smithi122 ceph-mon[93869]: pgmap v3975: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 38 KiB/s wr, 7 op/s 2023-03-09T16:46:20.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:46:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6491.023858 secs 2023-03-09T16:46:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:46:20 smithi122 ceph-mon[93869]: osdmap e462: 12 total, 12 up, 12 in 2023-03-09T16:46:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:46:20 smithi181 ceph-mon[94189]: pgmap v3975: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 38 KiB/s wr, 7 op/s 2023-03-09T16:46:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:46:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6491.023858 secs 2023-03-09T16:46:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:46:20 smithi181 ceph-mon[94189]: osdmap e462: 12 total, 12 up, 12 in 2023-03-09T16:46:20.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:46:20 smithi038 ceph-mon[92422]: pgmap v3975: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 38 KiB/s wr, 7 op/s 2023-03-09T16:46:20.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:46:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6491.023858 secs 2023-03-09T16:46:20.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:46:20 smithi038 ceph-mon[92422]: osdmap e462: 12 total, 12 up, 12 in 2023-03-09T16:46:22.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:46:22 smithi122 ceph-mon[93869]: pgmap v3977: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 7 op/s 2023-03-09T16:46:22.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:46:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6493.269190 secs 2023-03-09T16:46:22.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:46:22 smithi181 ceph-mon[94189]: pgmap v3977: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 7 op/s 2023-03-09T16:46:22.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:46:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6493.269190 secs 2023-03-09T16:46:22.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:46:22 smithi038 ceph-mon[92422]: pgmap v3977: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 7 op/s 2023-03-09T16:46:22.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:46:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6493.269190 secs 2023-03-09T16:46:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6955.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:46:24.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:46:24 smithi122 ceph-mon[93869]: pgmap v3978: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 7 op/s 2023-03-09T16:46:24.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:46:24 smithi181 ceph-mon[94189]: pgmap v3978: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 7 op/s 2023-03-09T16:46:24.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:46:24 smithi038 ceph-mon[92422]: pgmap v3978: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 7 op/s 2023-03-09T16:46:26.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:46:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6496.023933 secs 2023-03-09T16:46:26.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:46:25 smithi122 ceph-mon[93869]: pgmap v3979: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 7 op/s 2023-03-09T16:46:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:46:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6496.023933 secs 2023-03-09T16:46:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:46:25 smithi181 ceph-mon[94189]: pgmap v3979: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 7 op/s 2023-03-09T16:46:26.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:46:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6496.023933 secs 2023-03-09T16:46:26.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:46:25 smithi038 ceph-mon[92422]: pgmap v3979: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 7 op/s 2023-03-09T16:46:27.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:46:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6498.269279 secs 2023-03-09T16:46:27.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:46:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6498.269279 secs 2023-03-09T16:46:27.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:46:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6498.269279 secs 2023-03-09T16:46:28.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:46:28 smithi122 ceph-mon[93869]: pgmap v3980: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 7 op/s 2023-03-09T16:46:28.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:46:28 smithi181 ceph-mon[94189]: pgmap v3980: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 7 op/s 2023-03-09T16:46:28.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:46:28 smithi038 ceph-mon[92422]: pgmap v3980: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 7 op/s 2023-03-09T16:46:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6960.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:46:30.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:46:30 smithi122 ceph-mon[93869]: pgmap v3981: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 7 op/s 2023-03-09T16:46:30.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:46:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6501.023981 secs 2023-03-09T16:46:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:46:30 smithi181 ceph-mon[94189]: pgmap v3981: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 7 op/s 2023-03-09T16:46:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:46:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6501.023981 secs 2023-03-09T16:46:30.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:46:30 smithi038 ceph-mon[92422]: pgmap v3981: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 7 op/s 2023-03-09T16:46:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:46:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6501.023981 secs 2023-03-09T16:46:32.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:46:32 smithi122 ceph-mon[93869]: pgmap v3982: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 139 KiB/s wr, 10 op/s 2023-03-09T16:46:32.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:46:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6503.269398 secs 2023-03-09T16:46:32.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:46:32 smithi181 ceph-mon[94189]: pgmap v3982: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 139 KiB/s wr, 10 op/s 2023-03-09T16:46:32.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:46:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6503.269398 secs 2023-03-09T16:46:32.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:46:32 smithi038 ceph-mon[92422]: pgmap v3982: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 139 KiB/s wr, 10 op/s 2023-03-09T16:46:32.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:46:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6503.269398 secs 2023-03-09T16:46:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6965.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:46:34.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:46:34 smithi122 ceph-mon[93869]: pgmap v3983: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 92 KiB/s wr, 6 op/s 2023-03-09T16:46:34.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:46:34 smithi181 ceph-mon[94189]: pgmap v3983: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 92 KiB/s wr, 6 op/s 2023-03-09T16:46:34.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:46:34 smithi038 ceph-mon[92422]: pgmap v3983: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 92 KiB/s wr, 6 op/s 2023-03-09T16:46:36.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:46:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6506.024042 secs 2023-03-09T16:46:36.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:46:35 smithi122 ceph-mon[93869]: pgmap v3984: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 92 KiB/s wr, 6 op/s 2023-03-09T16:46:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:46:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6506.024042 secs 2023-03-09T16:46:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:46:35 smithi181 ceph-mon[94189]: pgmap v3984: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 92 KiB/s wr, 6 op/s 2023-03-09T16:46:36.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:46:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6506.024042 secs 2023-03-09T16:46:36.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:46:35 smithi038 ceph-mon[92422]: pgmap v3984: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 92 KiB/s wr, 6 op/s 2023-03-09T16:46:37.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:46:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6508.269519 secs 2023-03-09T16:46:37.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:46:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6508.269519 secs 2023-03-09T16:46:37.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:46:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6508.269519 secs 2023-03-09T16:46:38.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:46:38 smithi122 ceph-mon[93869]: pgmap v3985: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 126 KiB/s wr, 10 op/s 2023-03-09T16:46:38.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:46:38 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:46:38.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:46:38 smithi181 ceph-mon[94189]: pgmap v3985: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 126 KiB/s wr, 10 op/s 2023-03-09T16:46:38.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:46:38 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:46:38.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:46:38 smithi038 ceph-mon[92422]: pgmap v3985: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 126 KiB/s wr, 10 op/s 2023-03-09T16:46:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:46:38 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:46:38.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6970.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:46:40.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:46:40 smithi122 ceph-mon[93869]: pgmap v3986: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 126 KiB/s wr, 7 op/s 2023-03-09T16:46:40.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:46:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6511.024103 secs 2023-03-09T16:46:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:46:40 smithi181 ceph-mon[94189]: pgmap v3986: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 126 KiB/s wr, 7 op/s 2023-03-09T16:46:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:46:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6511.024103 secs 2023-03-09T16:46:40.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:46:40 smithi038 ceph-mon[92422]: pgmap v3986: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 126 KiB/s wr, 7 op/s 2023-03-09T16:46:40.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:46:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6511.024103 secs 2023-03-09T16:46:42.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:46:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:46:42.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:46:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:46:42.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:46:42 smithi122 ceph-mon[93869]: pgmap v3987: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 205 KiB/s wr, 12 op/s 2023-03-09T16:46:42.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:46:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6513.269635 secs 2023-03-09T16:46:42.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:46:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:46:42.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:46:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:46:42.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:46:42 smithi181 ceph-mon[94189]: pgmap v3987: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 205 KiB/s wr, 12 op/s 2023-03-09T16:46:42.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:46:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6513.269635 secs 2023-03-09T16:46:42.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:46:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:46:42.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:46:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:46:42.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:46:42 smithi038 ceph-mon[92422]: pgmap v3987: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 205 KiB/s wr, 12 op/s 2023-03-09T16:46:42.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:46:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6513.269635 secs 2023-03-09T16:46:43.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6975.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:46:44.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:46:44 smithi122 ceph-mon[93869]: pgmap v3988: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 114 KiB/s wr, 9 op/s 2023-03-09T16:46:44.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:46:44 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:46:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:46:44 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:46:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:46:44 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:46:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:46:44 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:46:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:46:44 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:46:44.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:46:44 smithi181 ceph-mon[94189]: pgmap v3988: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 114 KiB/s wr, 9 op/s 2023-03-09T16:46:44.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:46:44 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:46:44.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:46:44 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:46:44.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:46:44 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:46:44.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:46:44 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:46:44.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:46:44 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:46:44.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:46:44 smithi038 ceph-mon[92422]: pgmap v3988: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 114 KiB/s wr, 9 op/s 2023-03-09T16:46:44.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:46:44 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:46:44.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:46:44 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:46:44.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:46:44 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:46:44.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:46:44 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:46:44.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:46:44 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:46:46.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:46:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6516.024164 secs 2023-03-09T16:46:46.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:46:45 smithi122 ceph-mon[93869]: pgmap v3989: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 114 KiB/s wr, 9 op/s 2023-03-09T16:46:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:46:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6516.024164 secs 2023-03-09T16:46:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:46:45 smithi181 ceph-mon[94189]: pgmap v3989: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 114 KiB/s wr, 9 op/s 2023-03-09T16:46:46.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:46:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6516.024164 secs 2023-03-09T16:46:46.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:46:45 smithi038 ceph-mon[92422]: pgmap v3989: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 114 KiB/s wr, 9 op/s 2023-03-09T16:46:47.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:46:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6518.269794 secs 2023-03-09T16:46:47.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:46:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6518.269794 secs 2023-03-09T16:46:47.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:46:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6518.269794 secs 2023-03-09T16:46:48.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:46:48 smithi122 ceph-mon[93869]: pgmap v3990: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 243 KiB/s wr, 14 op/s 2023-03-09T16:46:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:46:48 smithi181 ceph-mon[94189]: pgmap v3990: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 243 KiB/s wr, 14 op/s 2023-03-09T16:46:48.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:46:48 smithi038 ceph-mon[92422]: pgmap v3990: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 243 KiB/s wr, 14 op/s 2023-03-09T16:46:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6980.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:46:50.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:46:50 smithi122 ceph-mon[93869]: pgmap v3991: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 209 KiB/s wr, 10 op/s 2023-03-09T16:46:50.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:46:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6521.024224 secs 2023-03-09T16:46:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:46:50 smithi181 ceph-mon[94189]: pgmap v3991: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 209 KiB/s wr, 10 op/s 2023-03-09T16:46:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:46:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6521.024224 secs 2023-03-09T16:46:50.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:46:50 smithi038 ceph-mon[92422]: pgmap v3991: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 209 KiB/s wr, 10 op/s 2023-03-09T16:46:50.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:46:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6521.024224 secs 2023-03-09T16:46:52.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:46:52 smithi122 ceph-mon[93869]: pgmap v3992: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 335 KiB/s rd, 263 KiB/s wr, 14 op/s 2023-03-09T16:46:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:46:52 smithi181 ceph-mon[94189]: pgmap v3992: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 335 KiB/s rd, 263 KiB/s wr, 14 op/s 2023-03-09T16:46:52.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:46:52 smithi038 ceph-mon[92422]: pgmap v3992: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 335 KiB/s rd, 263 KiB/s wr, 14 op/s 2023-03-09T16:46:53.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6985.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:46:54.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:46:54 smithi122 ceph-mon[93869]: pgmap v3993: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 184 KiB/s wr, 9 op/s 2023-03-09T16:46:54.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:46:54 smithi181 ceph-mon[94189]: pgmap v3993: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 184 KiB/s wr, 9 op/s 2023-03-09T16:46:54.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:46:54 smithi038 ceph-mon[92422]: pgmap v3993: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 184 KiB/s wr, 9 op/s 2023-03-09T16:46:56.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:46:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6526.024285 secs 2023-03-09T16:46:56.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:46:55 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:46:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:46:55 smithi122 ceph-mon[93869]: pgmap v3994: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 183 KiB/s wr, 9 op/s 2023-03-09T16:46:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:46:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6526.024285 secs 2023-03-09T16:46:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:46:55 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:46:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:46:55 smithi181 ceph-mon[94189]: pgmap v3994: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 183 KiB/s wr, 9 op/s 2023-03-09T16:46:56.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:46:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6526.024285 secs 2023-03-09T16:46:56.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:46:55 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:46:56.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:46:55 smithi038 ceph-mon[92422]: pgmap v3994: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 183 KiB/s wr, 9 op/s 2023-03-09T16:46:57.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:46:56 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:46:57.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:46:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:46:57.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:46:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:46:57.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:46:56 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:46:57.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:46:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:46:57.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:46:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:46:57.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:46:56 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:46:57.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:46:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:46:57.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:46:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:46:58.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:46:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6528.270033 secs 2023-03-09T16:46:58.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:46:57 smithi122 ceph-mon[93869]: pgmap v3995: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 271 KiB/s wr, 12 op/s 2023-03-09T16:46:58.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:46:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6528.270033 secs 2023-03-09T16:46:58.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:46:57 smithi181 ceph-mon[94189]: pgmap v3995: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 271 KiB/s wr, 12 op/s 2023-03-09T16:46:58.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:46:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6528.270033 secs 2023-03-09T16:46:58.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:46:57 smithi038 ceph-mon[92422]: pgmap v3995: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 271 KiB/s wr, 12 op/s 2023-03-09T16:46:58.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6990.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:47:00.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:47:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:47:00.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:47:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:47:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:47:00 smithi122 ceph-mon[93869]: pgmap v3996: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 142 KiB/s wr, 8 op/s 2023-03-09T16:47:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:47:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6531.024348 secs 2023-03-09T16:47:00.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:47:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:47:00.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:47:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:47:00.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:47:00 smithi181 ceph-mon[94189]: pgmap v3996: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 142 KiB/s wr, 8 op/s 2023-03-09T16:47:00.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:47:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6531.024348 secs 2023-03-09T16:47:00.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:47:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:47:00.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:47:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:47:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:47:00 smithi038 ceph-mon[92422]: pgmap v3996: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 142 KiB/s wr, 8 op/s 2023-03-09T16:47:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:47:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6531.024348 secs 2023-03-09T16:47:02.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:47:02 smithi122 ceph-mon[93869]: pgmap v3997: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 170 KiB/s wr, 12 op/s 2023-03-09T16:47:02.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:47:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6533.270150 secs 2023-03-09T16:47:02.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:47:02 smithi181 ceph-mon[94189]: pgmap v3997: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 170 KiB/s wr, 12 op/s 2023-03-09T16:47:02.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:47:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6533.270150 secs 2023-03-09T16:47:02.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:47:02 smithi038 ceph-mon[92422]: pgmap v3997: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 170 KiB/s wr, 12 op/s 2023-03-09T16:47:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:47:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6533.270150 secs 2023-03-09T16:47:03.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 6995.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:47:04.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:47:04 smithi122 ceph-mon[93869]: pgmap v3998: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 189 KiB/s wr, 21 op/s 2023-03-09T16:47:04.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:47:04 smithi181 ceph-mon[94189]: pgmap v3998: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 189 KiB/s wr, 21 op/s 2023-03-09T16:47:04.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:47:04 smithi038 ceph-mon[92422]: pgmap v3998: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 189 KiB/s wr, 21 op/s 2023-03-09T16:47:05.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:47:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6536.024411 secs 2023-03-09T16:47:05.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:47:05 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:47:05.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:47:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6536.024411 secs 2023-03-09T16:47:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:47:05 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:47:05.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:47:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6536.024411 secs 2023-03-09T16:47:05.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:47:05 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:47:06.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:47:06 smithi122 ceph-mon[93869]: pgmap v3999: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 189 KiB/s wr, 21 op/s 2023-03-09T16:47:06.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:47:06 smithi181 ceph-mon[94189]: pgmap v3999: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 189 KiB/s wr, 21 op/s 2023-03-09T16:47:06.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:47:06 smithi038 ceph-mon[92422]: pgmap v3999: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 189 KiB/s wr, 21 op/s 2023-03-09T16:47:07.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:47:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6538.270284 secs 2023-03-09T16:47:07.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:47:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6538.270284 secs 2023-03-09T16:47:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:47:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6538.270284 secs 2023-03-09T16:47:08.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:47:08 smithi038 ceph-mon[92422]: pgmap v4000: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 206 KiB/s wr, 26 op/s 2023-03-09T16:47:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7000.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:47:08.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:47:08 smithi122 ceph-mon[93869]: pgmap v4000: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 206 KiB/s wr, 26 op/s 2023-03-09T16:47:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:47:08 smithi181 ceph-mon[94189]: pgmap v4000: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 206 KiB/s wr, 26 op/s 2023-03-09T16:47:10.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:47:10 smithi038 ceph-mon[92422]: pgmap v4001: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 118 KiB/s wr, 23 op/s 2023-03-09T16:47:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:47:10 smithi038 ceph-mon[92422]: osdmap e463: 12 total, 12 up, 12 in 2023-03-09T16:47:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:47:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6541.024472 secs 2023-03-09T16:47:10.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:47:10 smithi122 ceph-mon[93869]: pgmap v4001: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 118 KiB/s wr, 23 op/s 2023-03-09T16:47:10.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:47:10 smithi122 ceph-mon[93869]: osdmap e463: 12 total, 12 up, 12 in 2023-03-09T16:47:10.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:47:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6541.024472 secs 2023-03-09T16:47:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:47:10 smithi181 ceph-mon[94189]: pgmap v4001: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 118 KiB/s wr, 23 op/s 2023-03-09T16:47:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:47:10 smithi181 ceph-mon[94189]: osdmap e463: 12 total, 12 up, 12 in 2023-03-09T16:47:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:47:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6541.024472 secs 2023-03-09T16:47:12.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:47:12 smithi122 ceph-mon[93869]: pgmap v4003: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 115 KiB/s wr, 25 op/s 2023-03-09T16:47:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:47:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6543.270387 secs 2023-03-09T16:47:12.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:47:12 smithi038 ceph-mon[92422]: pgmap v4003: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 115 KiB/s wr, 25 op/s 2023-03-09T16:47:12.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:47:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6543.270387 secs 2023-03-09T16:47:12.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:47:12 smithi181 ceph-mon[94189]: pgmap v4003: 129 pgs: 129 active+clean; 8.9 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 115 KiB/s wr, 25 op/s 2023-03-09T16:47:12.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:47:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6543.270387 secs 2023-03-09T16:47:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7005.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:47:14.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:47:14 smithi122 ceph-mon[93869]: pgmap v4004: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 58 KiB/s wr, 11 op/s 2023-03-09T16:47:14.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:47:14 smithi181 ceph-mon[94189]: pgmap v4004: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 58 KiB/s wr, 11 op/s 2023-03-09T16:47:14.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:47:14 smithi038 ceph-mon[92422]: pgmap v4004: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 58 KiB/s wr, 11 op/s 2023-03-09T16:47:16.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:47:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6546.024535 secs 2023-03-09T16:47:16.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:47:15 smithi122 ceph-mon[93869]: pgmap v4005: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 59 KiB/s wr, 11 op/s 2023-03-09T16:47:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:47:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6546.024535 secs 2023-03-09T16:47:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:47:15 smithi181 ceph-mon[94189]: pgmap v4005: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 59 KiB/s wr, 11 op/s 2023-03-09T16:47:16.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:47:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6546.024535 secs 2023-03-09T16:47:16.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:47:15 smithi038 ceph-mon[92422]: pgmap v4005: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 59 KiB/s wr, 11 op/s 2023-03-09T16:47:17.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:47:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6548.270503 secs 2023-03-09T16:47:17.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:47:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6548.270503 secs 2023-03-09T16:47:17.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:47:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6548.270503 secs 2023-03-09T16:47:18.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:47:18 smithi122 ceph-mon[93869]: pgmap v4006: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 39 KiB/s wr, 9 op/s 2023-03-09T16:47:18.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:47:18 smithi181 ceph-mon[94189]: pgmap v4006: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 39 KiB/s wr, 9 op/s 2023-03-09T16:47:18.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:47:18 smithi038 ceph-mon[92422]: pgmap v4006: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 39 KiB/s wr, 9 op/s 2023-03-09T16:47:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7010.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:47:20.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:47:20 smithi122 ceph-mon[93869]: pgmap v4007: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 39 KiB/s wr, 8 op/s 2023-03-09T16:47:20.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:47:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6551.024622 secs 2023-03-09T16:47:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:47:20 smithi122 ceph-mon[93869]: osdmap e464: 12 total, 12 up, 12 in 2023-03-09T16:47:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:47:20 smithi181 ceph-mon[94189]: pgmap v4007: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 39 KiB/s wr, 8 op/s 2023-03-09T16:47:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:47:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6551.024622 secs 2023-03-09T16:47:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:47:20 smithi181 ceph-mon[94189]: osdmap e464: 12 total, 12 up, 12 in 2023-03-09T16:47:20.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:47:20 smithi038 ceph-mon[92422]: pgmap v4007: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 39 KiB/s wr, 8 op/s 2023-03-09T16:47:20.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:47:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6551.024622 secs 2023-03-09T16:47:20.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:47:20 smithi038 ceph-mon[92422]: osdmap e464: 12 total, 12 up, 12 in 2023-03-09T16:47:22.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:47:22 smithi122 ceph-mon[93869]: pgmap v4009: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 33 KiB/s wr, 8 op/s 2023-03-09T16:47:22.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:47:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6553.270618 secs 2023-03-09T16:47:22.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:47:22 smithi181 ceph-mon[94189]: pgmap v4009: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 33 KiB/s wr, 8 op/s 2023-03-09T16:47:22.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:47:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6553.270618 secs 2023-03-09T16:47:22.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:47:22 smithi038 ceph-mon[92422]: pgmap v4009: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 33 KiB/s wr, 8 op/s 2023-03-09T16:47:22.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:47:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6553.270618 secs 2023-03-09T16:47:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7015.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:47:24.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:47:24 smithi122 ceph-mon[93869]: pgmap v4010: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 51 KiB/s wr, 8 op/s 2023-03-09T16:47:24.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:47:24 smithi181 ceph-mon[94189]: pgmap v4010: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 51 KiB/s wr, 8 op/s 2023-03-09T16:47:24.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:47:24 smithi038 ceph-mon[92422]: pgmap v4010: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 51 KiB/s wr, 8 op/s 2023-03-09T16:47:26.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:47:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6556.024685 secs 2023-03-09T16:47:26.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:47:25 smithi122 ceph-mon[93869]: pgmap v4011: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 8 op/s 2023-03-09T16:47:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:47:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6556.024685 secs 2023-03-09T16:47:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:47:25 smithi181 ceph-mon[94189]: pgmap v4011: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 8 op/s 2023-03-09T16:47:26.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:47:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6556.024685 secs 2023-03-09T16:47:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:47:25 smithi038 ceph-mon[92422]: pgmap v4011: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 8 op/s 2023-03-09T16:47:27.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:47:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6558.270740 secs 2023-03-09T16:47:27.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:47:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6558.270740 secs 2023-03-09T16:47:27.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:47:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6558.270740 secs 2023-03-09T16:47:28.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:47:28 smithi122 ceph-mon[93869]: pgmap v4012: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 7 op/s 2023-03-09T16:47:28.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:47:28 smithi181 ceph-mon[94189]: pgmap v4012: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 7 op/s 2023-03-09T16:47:28.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:47:28 smithi038 ceph-mon[92422]: pgmap v4012: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 7 op/s 2023-03-09T16:47:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7020.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:47:30.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:47:30 smithi122 ceph-mon[93869]: pgmap v4013: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 7 op/s 2023-03-09T16:47:30.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:47:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6561.024747 secs 2023-03-09T16:47:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:47:30 smithi181 ceph-mon[94189]: pgmap v4013: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 7 op/s 2023-03-09T16:47:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:47:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6561.024747 secs 2023-03-09T16:47:30.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:47:30 smithi038 ceph-mon[92422]: pgmap v4013: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 7 op/s 2023-03-09T16:47:30.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:47:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6561.024747 secs 2023-03-09T16:47:32.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:47:32 smithi181 ceph-mon[94189]: pgmap v4014: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 43 KiB/s wr, 9 op/s 2023-03-09T16:47:32.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:47:32 smithi038 ceph-mon[92422]: pgmap v4014: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 43 KiB/s wr, 9 op/s 2023-03-09T16:47:32.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:47:32 smithi122 ceph-mon[93869]: pgmap v4014: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 43 KiB/s wr, 9 op/s 2023-03-09T16:47:33.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7025.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:47:34.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:47:34 smithi181 ceph-mon[94189]: pgmap v4015: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 82 KiB/s wr, 6 op/s 2023-03-09T16:47:34.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:47:34 smithi038 ceph-mon[92422]: pgmap v4015: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 82 KiB/s wr, 6 op/s 2023-03-09T16:47:34.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:47:34 smithi122 ceph-mon[93869]: pgmap v4015: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 82 KiB/s wr, 6 op/s 2023-03-09T16:47:36.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:47:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6566.024835 secs 2023-03-09T16:47:36.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:47:35 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:47:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:47:35 smithi122 ceph-mon[93869]: pgmap v4016: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 41 KiB/s wr, 5 op/s 2023-03-09T16:47:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:47:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6566.024835 secs 2023-03-09T16:47:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:47:35 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:47:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:47:35 smithi181 ceph-mon[94189]: pgmap v4016: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 41 KiB/s wr, 5 op/s 2023-03-09T16:47:36.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:47:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6566.024835 secs 2023-03-09T16:47:36.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:47:35 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:47:36.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:47:35 smithi038 ceph-mon[92422]: pgmap v4016: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 41 KiB/s wr, 5 op/s 2023-03-09T16:47:37.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:47:36 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:47:37.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:47:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:47:37.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:47:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:47:37.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:47:36 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:47:37.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:47:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:47:37.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:47:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:47:37.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:47:36 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:47:37.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:47:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:47:37.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:47:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:47:38.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:47:37 smithi122 ceph-mon[93869]: pgmap v4017: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 84 KiB/s wr, 8 op/s 2023-03-09T16:47:38.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:47:37 smithi181 ceph-mon[94189]: pgmap v4017: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 84 KiB/s wr, 8 op/s 2023-03-09T16:47:38.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:47:37 smithi038 ceph-mon[92422]: pgmap v4017: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 84 KiB/s wr, 8 op/s 2023-03-09T16:47:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7030.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:47:40.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:47:40 smithi122 ceph-mon[93869]: pgmap v4018: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 84 KiB/s wr, 6 op/s 2023-03-09T16:47:40.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:47:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6571.024907 secs 2023-03-09T16:47:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:47:40 smithi181 ceph-mon[94189]: pgmap v4018: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 84 KiB/s wr, 6 op/s 2023-03-09T16:47:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:47:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6571.024907 secs 2023-03-09T16:47:40.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:47:40 smithi038 ceph-mon[92422]: pgmap v4018: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 84 KiB/s wr, 6 op/s 2023-03-09T16:47:40.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:47:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6571.024907 secs 2023-03-09T16:47:42.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:47:42 smithi038 ceph-mon[92422]: pgmap v4019: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 89 KiB/s wr, 8 op/s 2023-03-09T16:47:42.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:47:42 smithi122 ceph-mon[93869]: pgmap v4019: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 89 KiB/s wr, 8 op/s 2023-03-09T16:47:42.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:47:42 smithi181 ceph-mon[94189]: pgmap v4019: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 89 KiB/s wr, 8 op/s 2023-03-09T16:47:43.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7035.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:47:44.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:47:44 smithi038 ceph-mon[92422]: pgmap v4020: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 164 KiB/s wr, 7 op/s 2023-03-09T16:47:44.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:47:44 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:47:44.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:47:44 smithi122 ceph-mon[93869]: pgmap v4020: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 164 KiB/s wr, 7 op/s 2023-03-09T16:47:44.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:47:44 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:47:44.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:47:44 smithi181 ceph-mon[94189]: pgmap v4020: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 164 KiB/s wr, 7 op/s 2023-03-09T16:47:44.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:47:44 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:47:45.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:47:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6576.024967 secs 2023-03-09T16:47:45.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:47:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:47:45.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:47:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:47:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:47:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:47:45.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:47:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6576.024967 secs 2023-03-09T16:47:45.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:47:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:47:45.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:47:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:47:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:47:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:47:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:47:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6576.024967 secs 2023-03-09T16:47:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:47:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:47:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:47:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:47:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:47:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:47:46.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:47:46 smithi038 ceph-mon[92422]: pgmap v4021: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 123 KiB/s wr, 6 op/s 2023-03-09T16:47:46.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:47:46 smithi122 ceph-mon[93869]: pgmap v4021: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 123 KiB/s wr, 6 op/s 2023-03-09T16:47:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:47:46 smithi181 ceph-mon[94189]: pgmap v4021: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 123 KiB/s wr, 6 op/s 2023-03-09T16:47:48.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:47:48 smithi038 ceph-mon[92422]: pgmap v4022: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 240 KiB/s wr, 10 op/s 2023-03-09T16:47:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7040.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:47:48.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:47:48 smithi122 ceph-mon[93869]: pgmap v4022: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 240 KiB/s wr, 10 op/s 2023-03-09T16:47:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:47:48 smithi181 ceph-mon[94189]: pgmap v4022: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 240 KiB/s wr, 10 op/s 2023-03-09T16:47:50.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:47:50 smithi038 ceph-mon[92422]: pgmap v4023: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 197 KiB/s wr, 8 op/s 2023-03-09T16:47:50.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:47:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6581.025029 secs 2023-03-09T16:47:50.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:47:50 smithi122 ceph-mon[93869]: pgmap v4023: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 197 KiB/s wr, 8 op/s 2023-03-09T16:47:50.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:47:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6581.025029 secs 2023-03-09T16:47:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:47:50 smithi181 ceph-mon[94189]: pgmap v4023: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 197 KiB/s wr, 8 op/s 2023-03-09T16:47:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:47:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6581.025029 secs 2023-03-09T16:47:52.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:47:52 smithi122 ceph-mon[93869]: pgmap v4024: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 319 KiB/s wr, 12 op/s 2023-03-09T16:47:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:47:52 smithi181 ceph-mon[94189]: pgmap v4024: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 319 KiB/s wr, 12 op/s 2023-03-09T16:47:52.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:47:52 smithi038 ceph-mon[92422]: pgmap v4024: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 319 KiB/s wr, 12 op/s 2023-03-09T16:47:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7045.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:47:54.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:47:54 smithi122 ceph-mon[93869]: pgmap v4025: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 328 KiB/s rd, 313 KiB/s wr, 10 op/s 2023-03-09T16:47:54.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:47:54 smithi181 ceph-mon[94189]: pgmap v4025: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 328 KiB/s rd, 313 KiB/s wr, 10 op/s 2023-03-09T16:47:54.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:47:54 smithi038 ceph-mon[92422]: pgmap v4025: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 328 KiB/s rd, 313 KiB/s wr, 10 op/s 2023-03-09T16:47:56.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:47:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6586.025089 secs 2023-03-09T16:47:56.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:47:55 smithi122 ceph-mon[93869]: pgmap v4026: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 239 KiB/s wr, 9 op/s 2023-03-09T16:47:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:47:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6586.025089 secs 2023-03-09T16:47:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:47:55 smithi181 ceph-mon[94189]: pgmap v4026: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 239 KiB/s wr, 9 op/s 2023-03-09T16:47:56.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:47:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6586.025089 secs 2023-03-09T16:47:56.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:47:55 smithi038 ceph-mon[92422]: pgmap v4026: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 239 KiB/s wr, 9 op/s 2023-03-09T16:47:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:47:58 smithi181 ceph-mon[94189]: pgmap v4027: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 257 KiB/s wr, 13 op/s 2023-03-09T16:47:58.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:47:58 smithi038 ceph-mon[92422]: pgmap v4027: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 257 KiB/s wr, 13 op/s 2023-03-09T16:47:58.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7050.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:47:58.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:47:58 smithi122 ceph-mon[93869]: pgmap v4027: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 257 KiB/s wr, 13 op/s 2023-03-09T16:48:00.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:00 smithi038 ceph-mon[92422]: pgmap v4028: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 140 KiB/s wr, 9 op/s 2023-03-09T16:48:00.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6591.027429 secs 2023-03-09T16:48:00.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:00 smithi122 ceph-mon[93869]: pgmap v4028: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 140 KiB/s wr, 9 op/s 2023-03-09T16:48:00.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6591.027429 secs 2023-03-09T16:48:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:00 smithi181 ceph-mon[94189]: pgmap v4028: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 140 KiB/s wr, 9 op/s 2023-03-09T16:48:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6591.027429 secs 2023-03-09T16:48:02.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:02 smithi038 ceph-mon[92422]: pgmap v4029: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 315 KiB/s wr, 13 op/s 2023-03-09T16:48:02.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6593.271407 secs 2023-03-09T16:48:02.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:02 smithi122 ceph-mon[93869]: pgmap v4029: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 315 KiB/s wr, 13 op/s 2023-03-09T16:48:02.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6593.271407 secs 2023-03-09T16:48:02.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:02 smithi181 ceph-mon[94189]: pgmap v4029: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 315 KiB/s wr, 13 op/s 2023-03-09T16:48:02.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6593.271407 secs 2023-03-09T16:48:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7055.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:48:03.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:03 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:48:03.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:03 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:48:03.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:03 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:48:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:03 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:48:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:03 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:48:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:03 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:48:03.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:03 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:48:03.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:03 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:48:03.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:03 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:48:04.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:04 smithi122 ceph-mon[93869]: pgmap v4030: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 215 KiB/s wr, 22 op/s 2023-03-09T16:48:04.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:04 smithi181 ceph-mon[94189]: pgmap v4030: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 215 KiB/s wr, 22 op/s 2023-03-09T16:48:04.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:04 smithi038 ceph-mon[92422]: pgmap v4030: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 215 KiB/s wr, 22 op/s 2023-03-09T16:48:05.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:05 smithi122 ceph-mon[93869]: osdmap e465: 12 total, 12 up, 12 in 2023-03-09T16:48:05.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6596.027491 secs 2023-03-09T16:48:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:05 smithi181 ceph-mon[94189]: osdmap e465: 12 total, 12 up, 12 in 2023-03-09T16:48:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6596.027491 secs 2023-03-09T16:48:05.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:05 smithi038 ceph-mon[92422]: osdmap e465: 12 total, 12 up, 12 in 2023-03-09T16:48:05.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6596.027491 secs 2023-03-09T16:48:06.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:06 smithi122 ceph-mon[93869]: pgmap v4032: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 327 KiB/s wr, 26 op/s 2023-03-09T16:48:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:06 smithi181 ceph-mon[94189]: pgmap v4032: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 327 KiB/s wr, 26 op/s 2023-03-09T16:48:06.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:06 smithi038 ceph-mon[92422]: pgmap v4032: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 327 KiB/s wr, 26 op/s 2023-03-09T16:48:07.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6598.271550 secs 2023-03-09T16:48:07.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6598.271550 secs 2023-03-09T16:48:07.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6598.271550 secs 2023-03-09T16:48:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7060.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:48:08.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:08 smithi122 ceph-mon[93869]: pgmap v4033: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 345 KiB/s wr, 29 op/s 2023-03-09T16:48:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:08 smithi181 ceph-mon[94189]: pgmap v4033: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 345 KiB/s wr, 29 op/s 2023-03-09T16:48:08.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:08 smithi038 ceph-mon[92422]: pgmap v4033: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 345 KiB/s wr, 29 op/s 2023-03-09T16:48:10.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:10 smithi122 ceph-mon[93869]: pgmap v4034: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 451 KiB/s wr, 30 op/s 2023-03-09T16:48:10.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6601.027571 secs 2023-03-09T16:48:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:10 smithi122 ceph-mon[93869]: osdmap e466: 12 total, 12 up, 12 in 2023-03-09T16:48:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:10 smithi181 ceph-mon[94189]: pgmap v4034: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 451 KiB/s wr, 30 op/s 2023-03-09T16:48:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6601.027571 secs 2023-03-09T16:48:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:10 smithi181 ceph-mon[94189]: osdmap e466: 12 total, 12 up, 12 in 2023-03-09T16:48:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:10 smithi038 ceph-mon[92422]: pgmap v4034: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 451 KiB/s wr, 30 op/s 2023-03-09T16:48:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6601.027571 secs 2023-03-09T16:48:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:10 smithi038 ceph-mon[92422]: osdmap e466: 12 total, 12 up, 12 in 2023-03-09T16:48:12.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:12 smithi122 ceph-mon[93869]: pgmap v4036: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 268 KiB/s wr, 15 op/s 2023-03-09T16:48:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6603.271683 secs 2023-03-09T16:48:12.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:12 smithi181 ceph-mon[94189]: pgmap v4036: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 268 KiB/s wr, 15 op/s 2023-03-09T16:48:12.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6603.271683 secs 2023-03-09T16:48:12.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:12 smithi038 ceph-mon[92422]: pgmap v4036: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 268 KiB/s wr, 15 op/s 2023-03-09T16:48:12.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6603.271683 secs 2023-03-09T16:48:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7065.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:48:14.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:14 smithi122 ceph-mon[93869]: pgmap v4037: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 163 KiB/s wr, 13 op/s 2023-03-09T16:48:14.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:14 smithi181 ceph-mon[94189]: pgmap v4037: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 163 KiB/s wr, 13 op/s 2023-03-09T16:48:14.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:14 smithi038 ceph-mon[92422]: pgmap v4037: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 163 KiB/s wr, 13 op/s 2023-03-09T16:48:16.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6606.027631 secs 2023-03-09T16:48:16.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:15 smithi122 ceph-mon[93869]: pgmap v4038: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 176 KiB/s wr, 12 op/s 2023-03-09T16:48:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6606.027631 secs 2023-03-09T16:48:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:15 smithi181 ceph-mon[94189]: pgmap v4038: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 176 KiB/s wr, 12 op/s 2023-03-09T16:48:16.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6606.027631 secs 2023-03-09T16:48:16.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:15 smithi038 ceph-mon[92422]: pgmap v4038: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 176 KiB/s wr, 12 op/s 2023-03-09T16:48:17.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6608.271821 secs 2023-03-09T16:48:17.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6608.271821 secs 2023-03-09T16:48:17.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6608.271821 secs 2023-03-09T16:48:18.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:18 smithi122 ceph-mon[93869]: pgmap v4039: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 138 KiB/s wr, 7 op/s 2023-03-09T16:48:18.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:18 smithi181 ceph-mon[94189]: pgmap v4039: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 138 KiB/s wr, 7 op/s 2023-03-09T16:48:18.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:18 smithi038 ceph-mon[92422]: pgmap v4039: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 138 KiB/s wr, 7 op/s 2023-03-09T16:48:18.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7070.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:48:20.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:20 smithi038 ceph-mon[92422]: pgmap v4040: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 33 KiB/s wr, 7 op/s 2023-03-09T16:48:20.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6611.027691 secs 2023-03-09T16:48:20.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:20 smithi122 ceph-mon[93869]: pgmap v4040: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 33 KiB/s wr, 7 op/s 2023-03-09T16:48:20.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6611.027691 secs 2023-03-09T16:48:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:20 smithi181 ceph-mon[94189]: pgmap v4040: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 33 KiB/s wr, 7 op/s 2023-03-09T16:48:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6611.027691 secs 2023-03-09T16:48:22.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:22 smithi038 ceph-mon[92422]: pgmap v4041: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 29 KiB/s wr, 8 op/s 2023-03-09T16:48:22.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6613.271952 secs 2023-03-09T16:48:22.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:22 smithi122 ceph-mon[93869]: pgmap v4041: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 29 KiB/s wr, 8 op/s 2023-03-09T16:48:22.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6613.271952 secs 2023-03-09T16:48:22.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:22 smithi181 ceph-mon[94189]: pgmap v4041: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 29 KiB/s wr, 8 op/s 2023-03-09T16:48:22.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6613.271952 secs 2023-03-09T16:48:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7075.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:48:24.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:24 smithi038 ceph-mon[92422]: pgmap v4042: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 28 KiB/s wr, 6 op/s 2023-03-09T16:48:24.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:24 smithi122 ceph-mon[93869]: pgmap v4042: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 28 KiB/s wr, 6 op/s 2023-03-09T16:48:24.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:24 smithi181 ceph-mon[94189]: pgmap v4042: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 28 KiB/s wr, 6 op/s 2023-03-09T16:48:26.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6616.027751 secs 2023-03-09T16:48:26.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:25 smithi122 ceph-mon[93869]: pgmap v4043: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 69 KiB/s wr, 6 op/s 2023-03-09T16:48:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6616.027751 secs 2023-03-09T16:48:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:25 smithi181 ceph-mon[94189]: pgmap v4043: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 69 KiB/s wr, 6 op/s 2023-03-09T16:48:26.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6616.027751 secs 2023-03-09T16:48:26.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:25 smithi038 ceph-mon[92422]: pgmap v4043: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 69 KiB/s wr, 6 op/s 2023-03-09T16:48:27.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6618.272046 secs 2023-03-09T16:48:27.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6618.272046 secs 2023-03-09T16:48:27.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6618.272046 secs 2023-03-09T16:48:28.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:28 smithi122 ceph-mon[93869]: pgmap v4044: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 43 KiB/s wr, 8 op/s 2023-03-09T16:48:28.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:28 smithi181 ceph-mon[94189]: pgmap v4044: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 43 KiB/s wr, 8 op/s 2023-03-09T16:48:28.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:28 smithi038 ceph-mon[92422]: pgmap v4044: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 43 KiB/s wr, 8 op/s 2023-03-09T16:48:28.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7080.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:48:30.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:30 smithi038 ceph-mon[92422]: pgmap v4045: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 43 KiB/s wr, 7 op/s 2023-03-09T16:48:30.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6621.027810 secs 2023-03-09T16:48:30.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:30 smithi122 ceph-mon[93869]: pgmap v4045: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 43 KiB/s wr, 7 op/s 2023-03-09T16:48:30.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6621.027810 secs 2023-03-09T16:48:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:30 smithi181 ceph-mon[94189]: pgmap v4045: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 43 KiB/s wr, 7 op/s 2023-03-09T16:48:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6621.027810 secs 2023-03-09T16:48:32.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:32 smithi038 ceph-mon[92422]: pgmap v4046: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 42 KiB/s wr, 9 op/s 2023-03-09T16:48:32.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:32 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 6623.272163 secs 2023-03-09T16:48:32.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:32 smithi122 ceph-mon[93869]: pgmap v4046: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 42 KiB/s wr, 9 op/s 2023-03-09T16:48:32.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:32 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 6623.272163 secs 2023-03-09T16:48:32.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:32 smithi181 ceph-mon[94189]: pgmap v4046: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 42 KiB/s wr, 9 op/s 2023-03-09T16:48:32.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:32 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 6623.272163 secs 2023-03-09T16:48:33.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7085.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:48:34.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:34 smithi122 ceph-mon[93869]: pgmap v4047: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 6 op/s 2023-03-09T16:48:34.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:34 smithi181 ceph-mon[94189]: pgmap v4047: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 6 op/s 2023-03-09T16:48:34.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:34 smithi038 ceph-mon[92422]: pgmap v4047: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 6 op/s 2023-03-09T16:48:36.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6626.027872 secs 2023-03-09T16:48:36.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:35 smithi122 ceph-mon[93869]: pgmap v4048: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 85 KiB/s wr, 7 op/s 2023-03-09T16:48:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6626.027872 secs 2023-03-09T16:48:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:35 smithi181 ceph-mon[94189]: pgmap v4048: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 85 KiB/s wr, 7 op/s 2023-03-09T16:48:36.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6626.027872 secs 2023-03-09T16:48:36.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:35 smithi038 ceph-mon[92422]: pgmap v4048: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 85 KiB/s wr, 7 op/s 2023-03-09T16:48:37.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:48:37.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:48:37.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:48:37.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:48:37.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:48:37.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:48:38.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6628.272318 secs 2023-03-09T16:48:38.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:37 smithi122 ceph-mon[93869]: pgmap v4049: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 50 KiB/s wr, 9 op/s 2023-03-09T16:48:38.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6628.272318 secs 2023-03-09T16:48:38.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:37 smithi181 ceph-mon[94189]: pgmap v4049: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 50 KiB/s wr, 9 op/s 2023-03-09T16:48:38.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6628.272318 secs 2023-03-09T16:48:38.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:37 smithi038 ceph-mon[92422]: pgmap v4049: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 50 KiB/s wr, 9 op/s 2023-03-09T16:48:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7090.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:48:40.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:48:40.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:48:40.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:40 smithi122 ceph-mon[93869]: pgmap v4050: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 136 KiB/s wr, 8 op/s 2023-03-09T16:48:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6631.027933 secs 2023-03-09T16:48:40.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:48:40.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:48:40.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:40 smithi181 ceph-mon[94189]: pgmap v4050: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 136 KiB/s wr, 8 op/s 2023-03-09T16:48:40.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6631.027933 secs 2023-03-09T16:48:40.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:48:40.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:48:40.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:40 smithi038 ceph-mon[92422]: pgmap v4050: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 136 KiB/s wr, 8 op/s 2023-03-09T16:48:40.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6631.027933 secs 2023-03-09T16:48:42.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:42 smithi038 ceph-mon[92422]: pgmap v4051: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 156 KiB/s wr, 10 op/s 2023-03-09T16:48:42.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6633.272449 secs 2023-03-09T16:48:42.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:42 smithi122 ceph-mon[93869]: pgmap v4051: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 156 KiB/s wr, 10 op/s 2023-03-09T16:48:42.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6633.272449 secs 2023-03-09T16:48:42.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:42 smithi181 ceph-mon[94189]: pgmap v4051: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 156 KiB/s wr, 10 op/s 2023-03-09T16:48:42.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6633.272449 secs 2023-03-09T16:48:43.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7095.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:48:44.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:44 smithi122 ceph-mon[93869]: pgmap v4052: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 156 KiB/s wr, 9 op/s 2023-03-09T16:48:44.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:44 smithi181 ceph-mon[94189]: pgmap v4052: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 156 KiB/s wr, 9 op/s 2023-03-09T16:48:44.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:44 smithi038 ceph-mon[92422]: pgmap v4052: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 156 KiB/s wr, 9 op/s 2023-03-09T16:48:45.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6636.027986 secs 2023-03-09T16:48:45.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:48:45.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6636.027986 secs 2023-03-09T16:48:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:48:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6636.027986 secs 2023-03-09T16:48:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:48:46.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:46 smithi122 ceph-mon[93869]: pgmap v4053: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 156 KiB/s wr, 9 op/s 2023-03-09T16:48:46.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:48:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:48:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:48:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:48:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:48:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:48:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:48:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:48:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:48:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:46 smithi181 ceph-mon[94189]: pgmap v4053: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 156 KiB/s wr, 9 op/s 2023-03-09T16:48:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:48:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:48:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:48:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:48:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:48:46.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:48:46.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:48:46.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:48:46.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:48:46.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:46 smithi038 ceph-mon[92422]: pgmap v4053: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 156 KiB/s wr, 9 op/s 2023-03-09T16:48:46.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:48:46.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:48:46.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:48:46.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:48:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:48:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:48:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:48:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:48:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:48:47.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6638.272588 secs 2023-03-09T16:48:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6638.272588 secs 2023-03-09T16:48:47.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6638.272588 secs 2023-03-09T16:48:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7100.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:48:48.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:48 smithi122 ceph-mon[93869]: pgmap v4054: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 230 KiB/s wr, 12 op/s 2023-03-09T16:48:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:48 smithi181 ceph-mon[94189]: pgmap v4054: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 230 KiB/s wr, 12 op/s 2023-03-09T16:48:48.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:48 smithi038 ceph-mon[92422]: pgmap v4054: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 230 KiB/s wr, 12 op/s 2023-03-09T16:48:50.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:50 smithi122 ceph-mon[93869]: pgmap v4055: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 224 KiB/s wr, 11 op/s 2023-03-09T16:48:50.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6641.028045 secs 2023-03-09T16:48:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:50 smithi181 ceph-mon[94189]: pgmap v4055: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 224 KiB/s wr, 11 op/s 2023-03-09T16:48:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6641.028045 secs 2023-03-09T16:48:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:50 smithi038 ceph-mon[92422]: pgmap v4055: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 224 KiB/s wr, 11 op/s 2023-03-09T16:48:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6641.028045 secs 2023-03-09T16:48:52.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:52 smithi122 ceph-mon[93869]: pgmap v4056: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 223 KiB/s wr, 13 op/s 2023-03-09T16:48:52.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6643.272707 secs 2023-03-09T16:48:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:52 smithi181 ceph-mon[94189]: pgmap v4056: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 223 KiB/s wr, 13 op/s 2023-03-09T16:48:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6643.272707 secs 2023-03-09T16:48:52.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:52 smithi038 ceph-mon[92422]: pgmap v4056: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 223 KiB/s wr, 13 op/s 2023-03-09T16:48:52.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6643.272707 secs 2023-03-09T16:48:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7105.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:48:54.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:54 smithi122 ceph-mon[93869]: pgmap v4057: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 204 KiB/s wr, 10 op/s 2023-03-09T16:48:54.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:54 smithi181 ceph-mon[94189]: pgmap v4057: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 204 KiB/s wr, 10 op/s 2023-03-09T16:48:54.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:54 smithi038 ceph-mon[92422]: pgmap v4057: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 204 KiB/s wr, 10 op/s 2023-03-09T16:48:56.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6646.028105 secs 2023-03-09T16:48:56.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:55 smithi122 ceph-mon[93869]: pgmap v4058: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 203 KiB/s wr, 10 op/s 2023-03-09T16:48:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6646.028105 secs 2023-03-09T16:48:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:55 smithi181 ceph-mon[94189]: pgmap v4058: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 203 KiB/s wr, 10 op/s 2023-03-09T16:48:56.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6646.028105 secs 2023-03-09T16:48:56.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:55 smithi038 ceph-mon[92422]: pgmap v4058: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 203 KiB/s wr, 10 op/s 2023-03-09T16:48:57.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6648.272848 secs 2023-03-09T16:48:57.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6648.272848 secs 2023-03-09T16:48:57.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6648.272848 secs 2023-03-09T16:48:58.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:48:58 smithi122 ceph-mon[93869]: pgmap v4059: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 297 KiB/s wr, 13 op/s 2023-03-09T16:48:58.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:48:58 smithi181 ceph-mon[94189]: pgmap v4059: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 297 KiB/s wr, 13 op/s 2023-03-09T16:48:58.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:48:58 smithi038 ceph-mon[92422]: pgmap v4059: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 297 KiB/s wr, 13 op/s 2023-03-09T16:48:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7110.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:49:00.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:00 smithi122 ceph-mon[93869]: pgmap v4060: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 180 KiB/s wr, 9 op/s 2023-03-09T16:49:00.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6651.028166 secs 2023-03-09T16:49:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:00 smithi181 ceph-mon[94189]: pgmap v4060: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 180 KiB/s wr, 9 op/s 2023-03-09T16:49:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6651.028166 secs 2023-03-09T16:49:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:00 smithi038 ceph-mon[92422]: pgmap v4060: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 180 KiB/s wr, 9 op/s 2023-03-09T16:49:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6651.028166 secs 2023-03-09T16:49:02.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:02 smithi122 ceph-mon[93869]: pgmap v4061: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 212 KiB/s wr, 11 op/s 2023-03-09T16:49:02.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6653.272993 secs 2023-03-09T16:49:02.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:02 smithi181 ceph-mon[94189]: pgmap v4061: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 212 KiB/s wr, 11 op/s 2023-03-09T16:49:02.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6653.272993 secs 2023-03-09T16:49:02.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:02 smithi038 ceph-mon[92422]: pgmap v4061: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 212 KiB/s wr, 11 op/s 2023-03-09T16:49:02.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6653.272993 secs 2023-03-09T16:49:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7115.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:49:04.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:04 smithi122 ceph-mon[93869]: pgmap v4062: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 148 KiB/s wr, 20 op/s 2023-03-09T16:49:04.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:04 smithi181 ceph-mon[94189]: pgmap v4062: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 148 KiB/s wr, 20 op/s 2023-03-09T16:49:04.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:04 smithi038 ceph-mon[92422]: pgmap v4062: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 148 KiB/s wr, 20 op/s 2023-03-09T16:49:06.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6656.028226 secs 2023-03-09T16:49:06.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:05 smithi122 ceph-mon[93869]: pgmap v4063: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 147 KiB/s wr, 19 op/s 2023-03-09T16:49:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6656.028226 secs 2023-03-09T16:49:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:05 smithi181 ceph-mon[94189]: pgmap v4063: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 147 KiB/s wr, 19 op/s 2023-03-09T16:49:06.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6656.028226 secs 2023-03-09T16:49:06.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:05 smithi038 ceph-mon[92422]: pgmap v4063: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 147 KiB/s wr, 19 op/s 2023-03-09T16:49:07.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6658.273132 secs 2023-03-09T16:49:07.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6658.273132 secs 2023-03-09T16:49:07.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6658.273132 secs 2023-03-09T16:49:08.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:08 smithi122 ceph-mon[93869]: pgmap v4064: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 177 KiB/s wr, 25 op/s 2023-03-09T16:49:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:08 smithi181 ceph-mon[94189]: pgmap v4064: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 177 KiB/s wr, 25 op/s 2023-03-09T16:49:08.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:08 smithi038 ceph-mon[92422]: pgmap v4064: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 177 KiB/s wr, 25 op/s 2023-03-09T16:49:08.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7120.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:49:10.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:10 smithi122 ceph-mon[93869]: osdmap e467: 12 total, 12 up, 12 in 2023-03-09T16:49:10.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:10 smithi122 ceph-mon[93869]: pgmap v4066: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 100 KiB/s wr, 25 op/s 2023-03-09T16:49:10.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6661.028288 secs 2023-03-09T16:49:10.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:10 smithi181 ceph-mon[94189]: osdmap e467: 12 total, 12 up, 12 in 2023-03-09T16:49:10.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:10 smithi181 ceph-mon[94189]: pgmap v4066: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 100 KiB/s wr, 25 op/s 2023-03-09T16:49:10.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6661.028288 secs 2023-03-09T16:49:10.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:10 smithi038 ceph-mon[92422]: osdmap e467: 12 total, 12 up, 12 in 2023-03-09T16:49:10.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:10 smithi038 ceph-mon[92422]: pgmap v4066: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 100 KiB/s wr, 25 op/s 2023-03-09T16:49:10.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6661.028288 secs 2023-03-09T16:49:12.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:12 smithi122 ceph-mon[93869]: pgmap v4067: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 89 KiB/s wr, 25 op/s 2023-03-09T16:49:12.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6663.273251 secs 2023-03-09T16:49:12.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:12 smithi181 ceph-mon[94189]: pgmap v4067: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 89 KiB/s wr, 25 op/s 2023-03-09T16:49:12.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6663.273251 secs 2023-03-09T16:49:12.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:12 smithi038 ceph-mon[92422]: pgmap v4067: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 89 KiB/s wr, 25 op/s 2023-03-09T16:49:12.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6663.273251 secs 2023-03-09T16:49:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7125.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:49:14.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:14 smithi122 ceph-mon[93869]: pgmap v4068: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 81 KiB/s wr, 12 op/s 2023-03-09T16:49:14.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:14 smithi181 ceph-mon[94189]: pgmap v4068: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 81 KiB/s wr, 12 op/s 2023-03-09T16:49:14.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:14 smithi038 ceph-mon[92422]: pgmap v4068: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 81 KiB/s wr, 12 op/s 2023-03-09T16:49:16.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6666.028359 secs 2023-03-09T16:49:16.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:15 smithi122 ceph-mon[93869]: pgmap v4069: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 81 KiB/s wr, 12 op/s 2023-03-09T16:49:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6666.028359 secs 2023-03-09T16:49:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:15 smithi181 ceph-mon[94189]: pgmap v4069: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 81 KiB/s wr, 12 op/s 2023-03-09T16:49:16.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6666.028359 secs 2023-03-09T16:49:16.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:15 smithi038 ceph-mon[92422]: pgmap v4069: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 81 KiB/s wr, 12 op/s 2023-03-09T16:49:17.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6668.273341 secs 2023-03-09T16:49:17.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6668.273341 secs 2023-03-09T16:49:17.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6668.273341 secs 2023-03-09T16:49:18.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:18 smithi122 ceph-mon[93869]: pgmap v4070: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 47 KiB/s wr, 8 op/s 2023-03-09T16:49:18.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:18 smithi181 ceph-mon[94189]: pgmap v4070: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 47 KiB/s wr, 8 op/s 2023-03-09T16:49:18.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:18 smithi038 ceph-mon[92422]: pgmap v4070: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 47 KiB/s wr, 8 op/s 2023-03-09T16:49:18.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7130.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:49:20.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:20 smithi122 ceph-mon[93869]: pgmap v4071: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 94 KiB/s wr, 9 op/s 2023-03-09T16:49:20.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6671.028419 secs 2023-03-09T16:49:20.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:20 smithi122 ceph-mon[93869]: osdmap e468: 12 total, 12 up, 12 in 2023-03-09T16:49:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:20 smithi181 ceph-mon[94189]: pgmap v4071: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 94 KiB/s wr, 9 op/s 2023-03-09T16:49:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6671.028419 secs 2023-03-09T16:49:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:20 smithi181 ceph-mon[94189]: osdmap e468: 12 total, 12 up, 12 in 2023-03-09T16:49:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:20 smithi038 ceph-mon[92422]: pgmap v4071: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 94 KiB/s wr, 9 op/s 2023-03-09T16:49:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6671.028419 secs 2023-03-09T16:49:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:20 smithi038 ceph-mon[92422]: osdmap e468: 12 total, 12 up, 12 in 2023-03-09T16:49:22.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:22 smithi122 ceph-mon[93869]: pgmap v4073: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 69 KiB/s wr, 8 op/s 2023-03-09T16:49:22.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6673.273435 secs 2023-03-09T16:49:22.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:22 smithi181 ceph-mon[94189]: pgmap v4073: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 69 KiB/s wr, 8 op/s 2023-03-09T16:49:22.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6673.273435 secs 2023-03-09T16:49:22.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:22 smithi038 ceph-mon[92422]: pgmap v4073: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 69 KiB/s wr, 8 op/s 2023-03-09T16:49:22.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6673.273435 secs 2023-03-09T16:49:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7135.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:49:24.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:24 smithi122 ceph-mon[93869]: pgmap v4074: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 51 KiB/s wr, 8 op/s 2023-03-09T16:49:24.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:24 smithi181 ceph-mon[94189]: pgmap v4074: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 51 KiB/s wr, 8 op/s 2023-03-09T16:49:24.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:24 smithi038 ceph-mon[92422]: pgmap v4074: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 51 KiB/s wr, 8 op/s 2023-03-09T16:49:26.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6676.028498 secs 2023-03-09T16:49:26.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:25 smithi122 ceph-mon[93869]: pgmap v4075: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 52 KiB/s wr, 9 op/s 2023-03-09T16:49:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6676.028498 secs 2023-03-09T16:49:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:25 smithi181 ceph-mon[94189]: pgmap v4075: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 52 KiB/s wr, 9 op/s 2023-03-09T16:49:26.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6676.028498 secs 2023-03-09T16:49:26.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:25 smithi038 ceph-mon[92422]: pgmap v4075: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 52 KiB/s wr, 9 op/s 2023-03-09T16:49:27.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6678.273576 secs 2023-03-09T16:49:27.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6678.273576 secs 2023-03-09T16:49:27.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6678.273576 secs 2023-03-09T16:49:28.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:28 smithi122 ceph-mon[93869]: pgmap v4076: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 51 KiB/s wr, 9 op/s 2023-03-09T16:49:28.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:28 smithi181 ceph-mon[94189]: pgmap v4076: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 51 KiB/s wr, 9 op/s 2023-03-09T16:49:28.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:28 smithi038 ceph-mon[92422]: pgmap v4076: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 51 KiB/s wr, 9 op/s 2023-03-09T16:49:28.793 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7140.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:49:30.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:30 smithi122 ceph-mon[93869]: pgmap v4077: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.4 KiB/s wr, 8 op/s 2023-03-09T16:49:30.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6681.028558 secs 2023-03-09T16:49:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:30 smithi181 ceph-mon[94189]: pgmap v4077: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.4 KiB/s wr, 8 op/s 2023-03-09T16:49:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6681.028558 secs 2023-03-09T16:49:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:30 smithi038 ceph-mon[92422]: pgmap v4077: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.4 KiB/s wr, 8 op/s 2023-03-09T16:49:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6681.028558 secs 2023-03-09T16:49:32.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:32 smithi122 ceph-mon[93869]: pgmap v4078: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2023-03-09T16:49:32.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6683.273675 secs 2023-03-09T16:49:32.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:32 smithi181 ceph-mon[94189]: pgmap v4078: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2023-03-09T16:49:32.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6683.273675 secs 2023-03-09T16:49:32.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:32 smithi038 ceph-mon[92422]: pgmap v4078: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2023-03-09T16:49:32.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6683.273675 secs 2023-03-09T16:49:33.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7145.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:49:34.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:34 smithi122 ceph-mon[93869]: pgmap v4079: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 106 KiB/s wr, 8 op/s 2023-03-09T16:49:34.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:34 smithi181 ceph-mon[94189]: pgmap v4079: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 106 KiB/s wr, 8 op/s 2023-03-09T16:49:34.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:34 smithi038 ceph-mon[92422]: pgmap v4079: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 106 KiB/s wr, 8 op/s 2023-03-09T16:49:36.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6686.028616 secs 2023-03-09T16:49:36.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:35 smithi122 ceph-mon[93869]: pgmap v4080: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 106 KiB/s wr, 7 op/s 2023-03-09T16:49:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6686.028616 secs 2023-03-09T16:49:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:35 smithi181 ceph-mon[94189]: pgmap v4080: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 106 KiB/s wr, 7 op/s 2023-03-09T16:49:36.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6686.028616 secs 2023-03-09T16:49:36.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:35 smithi038 ceph-mon[92422]: pgmap v4080: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 106 KiB/s wr, 7 op/s 2023-03-09T16:49:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7150.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:49:38.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:38 smithi122 ceph-mon[93869]: pgmap v4081: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 118 KiB/s wr, 10 op/s 2023-03-09T16:49:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:38 smithi181 ceph-mon[94189]: pgmap v4081: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 118 KiB/s wr, 10 op/s 2023-03-09T16:49:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:38 smithi038 ceph-mon[92422]: pgmap v4081: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 118 KiB/s wr, 10 op/s 2023-03-09T16:49:39.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:39 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:49:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:39 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:49:39.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:39 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:49:40.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:40 smithi122 ceph-mon[93869]: pgmap v4082: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 128 KiB/s wr, 9 op/s 2023-03-09T16:49:40.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:40 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:49:40.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:49:40.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:49:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6691.028675 secs 2023-03-09T16:49:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:40 smithi181 ceph-mon[94189]: pgmap v4082: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 128 KiB/s wr, 9 op/s 2023-03-09T16:49:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:40 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:49:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:49:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:49:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6691.028675 secs 2023-03-09T16:49:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:40 smithi038 ceph-mon[92422]: pgmap v4082: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 128 KiB/s wr, 9 op/s 2023-03-09T16:49:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:40 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:49:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:49:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:49:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6691.028675 secs 2023-03-09T16:49:42.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:42 smithi122 ceph-mon[93869]: pgmap v4083: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 129 KiB/s wr, 11 op/s 2023-03-09T16:49:42.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:42 smithi181 ceph-mon[94189]: pgmap v4083: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 129 KiB/s wr, 11 op/s 2023-03-09T16:49:42.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:42 smithi038 ceph-mon[92422]: pgmap v4083: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 129 KiB/s wr, 11 op/s 2023-03-09T16:49:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7155.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:49:44.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:44 smithi122 ceph-mon[93869]: pgmap v4084: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 289 KiB/s wr, 10 op/s 2023-03-09T16:49:44.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:44 smithi181 ceph-mon[94189]: pgmap v4084: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 289 KiB/s wr, 10 op/s 2023-03-09T16:49:44.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:44 smithi038 ceph-mon[92422]: pgmap v4084: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 289 KiB/s wr, 10 op/s 2023-03-09T16:49:46.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6696.028752 secs 2023-03-09T16:49:46.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:45 smithi122 ceph-mon[93869]: pgmap v4085: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 184 KiB/s wr, 9 op/s 2023-03-09T16:49:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6696.028752 secs 2023-03-09T16:49:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:45 smithi181 ceph-mon[94189]: pgmap v4085: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 184 KiB/s wr, 9 op/s 2023-03-09T16:49:46.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6696.028752 secs 2023-03-09T16:49:46.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:45 smithi038 ceph-mon[92422]: pgmap v4085: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 184 KiB/s wr, 9 op/s 2023-03-09T16:49:47.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:49:47.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:49:47.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:49:48.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:47 smithi122 ceph-mon[93869]: pgmap v4086: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 250 KiB/s wr, 12 op/s 2023-03-09T16:49:48.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:47 smithi181 ceph-mon[94189]: pgmap v4086: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 250 KiB/s wr, 12 op/s 2023-03-09T16:49:48.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:47 smithi038 ceph-mon[92422]: pgmap v4086: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 250 KiB/s wr, 12 op/s 2023-03-09T16:49:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7160.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:49:50.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:49:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:49:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:49:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:49:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:49 smithi122 ceph-mon[93869]: pgmap v4087: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 249 KiB/s wr, 10 op/s 2023-03-09T16:49:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:49:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:49:50.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:49:50.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:49:50.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:49 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T16:49:50.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:49 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T16:49:50.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:49 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6701.028826 secs 2023-03-09T16:49:50.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:49:50.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:49:50.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:49:50.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:49:50.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:49 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi038 to 197.2M 2023-03-09T16:49:50.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:49 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi038 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-03-09T16:49:50.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:49:50.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:49:50.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:49:50.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:49:50.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:49 smithi181 ceph-mon[94189]: pgmap v4087: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 249 KiB/s wr, 10 op/s 2023-03-09T16:49:50.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:49:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:49:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:49:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:49:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:49 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T16:49:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:49 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T16:49:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:49 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6701.028826 secs 2023-03-09T16:49:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:49:50.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:49:50.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:49:50.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:49:50.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:49 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi038 to 197.2M 2023-03-09T16:49:50.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:49 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi038 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-03-09T16:49:50.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:49:50.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:49:50.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:49:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:49:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:49 smithi038 ceph-mon[92422]: pgmap v4087: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 249 KiB/s wr, 10 op/s 2023-03-09T16:49:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:49:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:49:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:49:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:49:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:49 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T16:49:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:49 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T16:49:50.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:49 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6701.028826 secs 2023-03-09T16:49:50.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:49:50.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:49:50.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:49:50.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:49:50.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:49 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi038 to 197.2M 2023-03-09T16:49:50.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:49 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi038 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-03-09T16:49:51.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:49:51.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:49:51.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:49:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:49:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:49:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:49:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:49:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:49:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:49:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:49:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:49:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:49:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:49:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:49:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:49:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:49:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:49:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:49:51.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:49:51.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:49:51.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:49:51.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:49:51.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:49:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T16:49:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:49:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:49:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:49:52.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:52 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T16:49:52.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:52 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T16:49:52.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:52 smithi122 ceph-mon[93869]: pgmap v4088: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 299 KiB/s wr, 13 op/s 2023-03-09T16:49:52.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6703.274052 secs 2023-03-09T16:49:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:52 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T16:49:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:52 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T16:49:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:52 smithi181 ceph-mon[94189]: pgmap v4088: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 299 KiB/s wr, 13 op/s 2023-03-09T16:49:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6703.274052 secs 2023-03-09T16:49:52.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:52 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T16:49:52.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:52 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T16:49:52.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:52 smithi038 ceph-mon[92422]: pgmap v4088: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 299 KiB/s wr, 13 op/s 2023-03-09T16:49:52.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6703.274052 secs 2023-03-09T16:49:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7165.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:49:54.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:54 smithi122 ceph-mon[93869]: pgmap v4089: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 299 KiB/s wr, 12 op/s 2023-03-09T16:49:54.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:54 smithi181 ceph-mon[94189]: pgmap v4089: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 299 KiB/s wr, 12 op/s 2023-03-09T16:49:54.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:54 smithi038 ceph-mon[92422]: pgmap v4089: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 299 KiB/s wr, 12 op/s 2023-03-09T16:49:56.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6706.028888 secs 2023-03-09T16:49:56.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:55 smithi122 ceph-mon[93869]: pgmap v4090: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 139 KiB/s wr, 10 op/s 2023-03-09T16:49:56.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6706.028888 secs 2023-03-09T16:49:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:55 smithi181 ceph-mon[94189]: pgmap v4090: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 139 KiB/s wr, 10 op/s 2023-03-09T16:49:56.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6706.028888 secs 2023-03-09T16:49:56.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:55 smithi038 ceph-mon[92422]: pgmap v4090: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 139 KiB/s wr, 10 op/s 2023-03-09T16:49:57.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:56 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:49:57.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:49:57.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:49:57.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:56 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:49:57.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:49:57.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:49:57.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:56 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:49:57.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:49:57.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:49:58.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6708.274167 secs 2023-03-09T16:49:58.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:49:57 smithi122 ceph-mon[93869]: pgmap v4091: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 231 KiB/s wr, 13 op/s 2023-03-09T16:49:58.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6708.274167 secs 2023-03-09T16:49:58.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:49:57 smithi181 ceph-mon[94189]: pgmap v4091: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 231 KiB/s wr, 13 op/s 2023-03-09T16:49:58.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6708.274167 secs 2023-03-09T16:49:58.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:49:57 smithi038 ceph-mon[92422]: pgmap v4091: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 231 KiB/s wr, 13 op/s 2023-03-09T16:49:58.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7170.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:50:00.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:50:00 smithi122 ceph-mon[93869]: pgmap v4092: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 165 KiB/s wr, 10 op/s 2023-03-09T16:50:00.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:50:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6711.028948 secs 2023-03-09T16:50:00.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:50:00 smithi122 ceph-mon[93869]: overall HEALTH_WARN 2 MDSs report slow requests 2023-03-09T16:50:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:50:00 smithi181 ceph-mon[94189]: pgmap v4092: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 165 KiB/s wr, 10 op/s 2023-03-09T16:50:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:50:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6711.028948 secs 2023-03-09T16:50:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:50:00 smithi181 ceph-mon[94189]: overall HEALTH_WARN 2 MDSs report slow requests 2023-03-09T16:50:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:50:00 smithi038 ceph-mon[92422]: pgmap v4092: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 165 KiB/s wr, 10 op/s 2023-03-09T16:50:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:50:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6711.028948 secs 2023-03-09T16:50:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:50:00 smithi038 ceph-mon[92422]: overall HEALTH_WARN 2 MDSs report slow requests 2023-03-09T16:50:02.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:50:02 smithi122 ceph-mon[93869]: pgmap v4093: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 193 KiB/s wr, 12 op/s 2023-03-09T16:50:02.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:50:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6713.274290 secs 2023-03-09T16:50:02.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:50:02 smithi181 ceph-mon[94189]: pgmap v4093: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 193 KiB/s wr, 12 op/s 2023-03-09T16:50:02.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:50:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6713.274290 secs 2023-03-09T16:50:02.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:50:02 smithi038 ceph-mon[92422]: pgmap v4093: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 193 KiB/s wr, 12 op/s 2023-03-09T16:50:02.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:50:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6713.274290 secs 2023-03-09T16:50:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7175.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:50:04.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:50:04 smithi122 ceph-mon[93869]: pgmap v4094: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 152 KiB/s wr, 20 op/s 2023-03-09T16:50:04.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:50:04 smithi181 ceph-mon[94189]: pgmap v4094: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 152 KiB/s wr, 20 op/s 2023-03-09T16:50:04.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:50:04 smithi038 ceph-mon[92422]: pgmap v4094: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 152 KiB/s wr, 20 op/s 2023-03-09T16:50:06.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:50:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6716.029008 secs 2023-03-09T16:50:06.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:50:05 smithi122 ceph-mon[93869]: pgmap v4095: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 152 KiB/s wr, 19 op/s 2023-03-09T16:50:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:50:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6716.029008 secs 2023-03-09T16:50:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:50:05 smithi181 ceph-mon[94189]: pgmap v4095: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 152 KiB/s wr, 19 op/s 2023-03-09T16:50:06.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:50:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6716.029008 secs 2023-03-09T16:50:06.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:50:05 smithi038 ceph-mon[92422]: pgmap v4095: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 152 KiB/s wr, 19 op/s 2023-03-09T16:50:07.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:50:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6718.274389 secs 2023-03-09T16:50:07.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:50:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6718.274389 secs 2023-03-09T16:50:07.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:50:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6718.274389 secs 2023-03-09T16:50:08.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:50:08 smithi122 ceph-mon[93869]: pgmap v4096: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 176 KiB/s wr, 24 op/s 2023-03-09T16:50:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:50:08 smithi181 ceph-mon[94189]: pgmap v4096: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 176 KiB/s wr, 24 op/s 2023-03-09T16:50:08.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:50:08 smithi038 ceph-mon[92422]: pgmap v4096: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 176 KiB/s wr, 24 op/s 2023-03-09T16:50:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7180.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:50:10.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:50:10 smithi122 ceph-mon[93869]: osdmap e469: 12 total, 12 up, 12 in 2023-03-09T16:50:10.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:50:10 smithi122 ceph-mon[93869]: pgmap v4098: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 100 KiB/s wr, 24 op/s 2023-03-09T16:50:10.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:50:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6721.029069 secs 2023-03-09T16:50:10.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:50:10 smithi181 ceph-mon[94189]: osdmap e469: 12 total, 12 up, 12 in 2023-03-09T16:50:10.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:50:10 smithi181 ceph-mon[94189]: pgmap v4098: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 100 KiB/s wr, 24 op/s 2023-03-09T16:50:10.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:50:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6721.029069 secs 2023-03-09T16:50:10.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:50:10 smithi038 ceph-mon[92422]: osdmap e469: 12 total, 12 up, 12 in 2023-03-09T16:50:10.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:50:10 smithi038 ceph-mon[92422]: pgmap v4098: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 100 KiB/s wr, 24 op/s 2023-03-09T16:50:10.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:50:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6721.029069 secs 2023-03-09T16:50:12.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:50:12 smithi122 ceph-mon[93869]: pgmap v4099: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 86 KiB/s wr, 24 op/s 2023-03-09T16:50:12.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:50:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6723.274503 secs 2023-03-09T16:50:12.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:50:12 smithi181 ceph-mon[94189]: pgmap v4099: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 86 KiB/s wr, 24 op/s 2023-03-09T16:50:12.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:50:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6723.274503 secs 2023-03-09T16:50:12.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:50:12 smithi038 ceph-mon[92422]: pgmap v4099: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 86 KiB/s wr, 24 op/s 2023-03-09T16:50:12.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:50:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6723.274503 secs 2023-03-09T16:50:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7185.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:50:14.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:50:14 smithi122 ceph-mon[93869]: pgmap v4100: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 60 KiB/s wr, 9 op/s 2023-03-09T16:50:14.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:50:14 smithi181 ceph-mon[94189]: pgmap v4100: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 60 KiB/s wr, 9 op/s 2023-03-09T16:50:14.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:50:14 smithi038 ceph-mon[92422]: pgmap v4100: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 60 KiB/s wr, 9 op/s 2023-03-09T16:50:16.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:50:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6726.029129 secs 2023-03-09T16:50:16.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:50:15 smithi122 ceph-mon[93869]: pgmap v4101: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 60 KiB/s wr, 9 op/s 2023-03-09T16:50:16.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:50:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6726.029129 secs 2023-03-09T16:50:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:50:15 smithi181 ceph-mon[94189]: pgmap v4101: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 60 KiB/s wr, 9 op/s 2023-03-09T16:50:16.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:50:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6726.029129 secs 2023-03-09T16:50:16.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:50:15 smithi038 ceph-mon[92422]: pgmap v4101: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 60 KiB/s wr, 9 op/s 2023-03-09T16:50:17.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:50:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6728.274632 secs 2023-03-09T16:50:17.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:50:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6728.274632 secs 2023-03-09T16:50:17.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:50:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6728.274632 secs 2023-03-09T16:50:18.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:50:18 smithi122 ceph-mon[93869]: pgmap v4102: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T16:50:18.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:50:18 smithi181 ceph-mon[94189]: pgmap v4102: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T16:50:18.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:50:18 smithi038 ceph-mon[92422]: pgmap v4102: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T16:50:18.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7190.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:50:20.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:50:20 smithi122 ceph-mon[93869]: pgmap v4103: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 31 KiB/s wr, 7 op/s 2023-03-09T16:50:20.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:50:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6731.029191 secs 2023-03-09T16:50:20.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:50:20 smithi122 ceph-mon[93869]: osdmap e470: 12 total, 12 up, 12 in 2023-03-09T16:50:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:50:20 smithi181 ceph-mon[94189]: pgmap v4103: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 31 KiB/s wr, 7 op/s 2023-03-09T16:50:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:50:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6731.029191 secs 2023-03-09T16:50:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:50:20 smithi181 ceph-mon[94189]: osdmap e470: 12 total, 12 up, 12 in 2023-03-09T16:50:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:50:20 smithi038 ceph-mon[92422]: pgmap v4103: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 31 KiB/s wr, 7 op/s 2023-03-09T16:50:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:50:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6731.029191 secs 2023-03-09T16:50:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:50:20 smithi038 ceph-mon[92422]: osdmap e470: 12 total, 12 up, 12 in 2023-03-09T16:50:22.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:50:22 smithi122 ceph-mon[93869]: pgmap v4105: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 51 KiB/s wr, 7 op/s 2023-03-09T16:50:22.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:50:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6733.274751 secs 2023-03-09T16:50:22.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:50:22 smithi181 ceph-mon[94189]: pgmap v4105: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 51 KiB/s wr, 7 op/s 2023-03-09T16:50:22.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:50:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6733.274751 secs 2023-03-09T16:50:22.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:50:22 smithi038 ceph-mon[92422]: pgmap v4105: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 51 KiB/s wr, 7 op/s 2023-03-09T16:50:22.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:50:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6733.274751 secs 2023-03-09T16:50:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7195.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:50:24.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:50:24 smithi122 ceph-mon[93869]: pgmap v4106: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 51 KiB/s wr, 7 op/s 2023-03-09T16:50:24.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:50:24 smithi181 ceph-mon[94189]: pgmap v4106: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 51 KiB/s wr, 7 op/s 2023-03-09T16:50:24.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:50:24 smithi038 ceph-mon[92422]: pgmap v4106: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 51 KiB/s wr, 7 op/s 2023-03-09T16:50:26.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:50:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6736.029253 secs 2023-03-09T16:50:26.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:50:25 smithi122 ceph-mon[93869]: pgmap v4107: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 51 KiB/s wr, 7 op/s 2023-03-09T16:50:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:50:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6736.029253 secs 2023-03-09T16:50:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:50:25 smithi181 ceph-mon[94189]: pgmap v4107: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 51 KiB/s wr, 7 op/s 2023-03-09T16:50:26.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:50:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6736.029253 secs 2023-03-09T16:50:26.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:50:25 smithi038 ceph-mon[92422]: pgmap v4107: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 51 KiB/s wr, 7 op/s 2023-03-09T16:50:27.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:50:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6738.274892 secs 2023-03-09T16:50:27.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:50:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6738.274892 secs 2023-03-09T16:50:27.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:50:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6738.274892 secs 2023-03-09T16:50:28.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:50:28 smithi122 ceph-mon[93869]: pgmap v4108: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 51 KiB/s wr, 7 op/s 2023-03-09T16:50:28.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:50:28 smithi181 ceph-mon[94189]: pgmap v4108: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 51 KiB/s wr, 7 op/s 2023-03-09T16:50:28.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:50:28 smithi038 ceph-mon[92422]: pgmap v4108: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 51 KiB/s wr, 7 op/s 2023-03-09T16:50:28.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7200.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:50:30.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:50:30 smithi122 ceph-mon[93869]: pgmap v4109: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 51 KiB/s wr, 6 op/s 2023-03-09T16:50:30.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:50:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6741.029313 secs 2023-03-09T16:50:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:50:30 smithi181 ceph-mon[94189]: pgmap v4109: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 51 KiB/s wr, 6 op/s 2023-03-09T16:50:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:50:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6741.029313 secs 2023-03-09T16:50:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:50:30 smithi038 ceph-mon[92422]: pgmap v4109: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 51 KiB/s wr, 6 op/s 2023-03-09T16:50:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:50:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6741.029313 secs 2023-03-09T16:50:32.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:50:32 smithi122 ceph-mon[93869]: pgmap v4110: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 87 KiB/s wr, 9 op/s 2023-03-09T16:50:32.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:50:32 smithi181 ceph-mon[94189]: pgmap v4110: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 87 KiB/s wr, 9 op/s 2023-03-09T16:50:32.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:50:32 smithi038 ceph-mon[92422]: pgmap v4110: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 87 KiB/s wr, 9 op/s 2023-03-09T16:50:33.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7205.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:50:34.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:50:34 smithi122 ceph-mon[93869]: pgmap v4111: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 42 KiB/s wr, 6 op/s 2023-03-09T16:50:34.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:50:34 smithi181 ceph-mon[94189]: pgmap v4111: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 42 KiB/s wr, 6 op/s 2023-03-09T16:50:34.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:50:34 smithi038 ceph-mon[92422]: pgmap v4111: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 42 KiB/s wr, 6 op/s 2023-03-09T16:50:36.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:50:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6746.029374 secs 2023-03-09T16:50:36.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:50:35 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:50:36.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:50:35 smithi122 ceph-mon[93869]: pgmap v4112: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 42 KiB/s wr, 6 op/s 2023-03-09T16:50:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:50:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6746.029374 secs 2023-03-09T16:50:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:50:35 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:50:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:50:35 smithi181 ceph-mon[94189]: pgmap v4112: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 42 KiB/s wr, 6 op/s 2023-03-09T16:50:36.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:50:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6746.029374 secs 2023-03-09T16:50:36.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:50:35 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:50:36.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:50:35 smithi038 ceph-mon[92422]: pgmap v4112: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 42 KiB/s wr, 6 op/s 2023-03-09T16:50:37.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:50:36 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:50:37.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:50:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:50:37.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:50:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:50:37.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:50:36 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:50:37.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:50:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:50:37.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:50:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:50:37.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:50:36 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:50:37.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:50:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:50:37.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:50:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:50:38.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:50:37 smithi122 ceph-mon[93869]: pgmap v4113: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 86 KiB/s wr, 9 op/s 2023-03-09T16:50:38.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:50:37 smithi181 ceph-mon[94189]: pgmap v4113: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 86 KiB/s wr, 9 op/s 2023-03-09T16:50:38.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:50:37 smithi038 ceph-mon[92422]: pgmap v4113: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 86 KiB/s wr, 9 op/s 2023-03-09T16:50:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7210.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:50:40.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:50:40 smithi122 ceph-mon[93869]: pgmap v4114: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 86 KiB/s wr, 6 op/s 2023-03-09T16:50:40.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:50:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6751.029436 secs 2023-03-09T16:50:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:50:40 smithi181 ceph-mon[94189]: pgmap v4114: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 86 KiB/s wr, 6 op/s 2023-03-09T16:50:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:50:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6751.029436 secs 2023-03-09T16:50:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:50:40 smithi038 ceph-mon[92422]: pgmap v4114: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 86 KiB/s wr, 6 op/s 2023-03-09T16:50:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:50:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6751.029436 secs 2023-03-09T16:50:42.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:50:42 smithi122 ceph-mon[93869]: pgmap v4115: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 173 KiB/s wr, 10 op/s 2023-03-09T16:50:42.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:50:42 smithi181 ceph-mon[94189]: pgmap v4115: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 173 KiB/s wr, 10 op/s 2023-03-09T16:50:42.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:50:42 smithi038 ceph-mon[92422]: pgmap v4115: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 173 KiB/s wr, 10 op/s 2023-03-09T16:50:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7215.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:50:44.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:50:44 smithi122 ceph-mon[93869]: pgmap v4116: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 131 KiB/s wr, 7 op/s 2023-03-09T16:50:44.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:50:44 smithi181 ceph-mon[94189]: pgmap v4116: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 131 KiB/s wr, 7 op/s 2023-03-09T16:50:44.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:50:44 smithi038 ceph-mon[92422]: pgmap v4116: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 131 KiB/s wr, 7 op/s 2023-03-09T16:50:46.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:50:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6756.029496 secs 2023-03-09T16:50:46.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:50:45 smithi122 ceph-mon[93869]: pgmap v4117: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 131 KiB/s wr, 7 op/s 2023-03-09T16:50:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:50:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6756.029496 secs 2023-03-09T16:50:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:50:45 smithi181 ceph-mon[94189]: pgmap v4117: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 131 KiB/s wr, 7 op/s 2023-03-09T16:50:46.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:50:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6756.029496 secs 2023-03-09T16:50:46.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:50:45 smithi038 ceph-mon[92422]: pgmap v4117: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 131 KiB/s wr, 7 op/s 2023-03-09T16:50:47.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:50:47 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 6758.275336 secs 2023-03-09T16:50:47.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:50:47 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 6758.275336 secs 2023-03-09T16:50:47.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:50:47 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 6758.275336 secs 2023-03-09T16:50:48.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:50:48 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:50:48.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:50:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:50:48.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:50:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:50:48.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:50:48 smithi122 ceph-mon[93869]: pgmap v4118: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 211 KiB/s wr, 11 op/s 2023-03-09T16:50:48.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:50:48 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:50:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:50:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:50:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:50:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:50:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:50:48 smithi181 ceph-mon[94189]: pgmap v4118: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 211 KiB/s wr, 11 op/s 2023-03-09T16:50:48.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:50:48 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:50:48.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:50:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:50:48.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:50:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:50:48.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:50:48 smithi038 ceph-mon[92422]: pgmap v4118: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 211 KiB/s wr, 11 op/s 2023-03-09T16:50:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7220.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:50:50.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:50:50 smithi122 ceph-mon[93869]: pgmap v4119: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 168 KiB/s wr, 8 op/s 2023-03-09T16:50:50.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:50:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6761.029558 secs 2023-03-09T16:50:50.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:50:50 smithi181 ceph-mon[94189]: pgmap v4119: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 168 KiB/s wr, 8 op/s 2023-03-09T16:50:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:50:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6761.029558 secs 2023-03-09T16:50:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:50:50 smithi038 ceph-mon[92422]: pgmap v4119: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 168 KiB/s wr, 8 op/s 2023-03-09T16:50:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:50:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6761.029558 secs 2023-03-09T16:50:51.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:50:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:50:51.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:50:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:50:51.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:50:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:50:52.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:50:52 smithi122 ceph-mon[93869]: pgmap v4120: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 286 KiB/s wr, 13 op/s 2023-03-09T16:50:52.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:50:52 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 6763.275425 secs 2023-03-09T16:50:52.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:50:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:50:52.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:50:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:50:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:50:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:50:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:50:52 smithi181 ceph-mon[94189]: pgmap v4120: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 286 KiB/s wr, 13 op/s 2023-03-09T16:50:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:50:52 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 6763.275425 secs 2023-03-09T16:50:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:50:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:50:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:50:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:50:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:50:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:50:52.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:50:52 smithi038 ceph-mon[92422]: pgmap v4120: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 286 KiB/s wr, 13 op/s 2023-03-09T16:50:52.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:50:52 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 6763.275425 secs 2023-03-09T16:50:52.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:50:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:50:52.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:50:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:50:52.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:50:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:50:53.793 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7225.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:50:54.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:50:54 smithi122 ceph-mon[93869]: pgmap v4121: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 199 KiB/s wr, 10 op/s 2023-03-09T16:50:54.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:50:54 smithi181 ceph-mon[94189]: pgmap v4121: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 199 KiB/s wr, 10 op/s 2023-03-09T16:50:54.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:50:54 smithi038 ceph-mon[92422]: pgmap v4121: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 199 KiB/s wr, 10 op/s 2023-03-09T16:50:56.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:50:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6766.029632 secs 2023-03-09T16:50:56.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:50:55 smithi122 ceph-mon[93869]: pgmap v4122: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 199 KiB/s wr, 10 op/s 2023-03-09T16:50:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:50:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6766.029632 secs 2023-03-09T16:50:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:50:55 smithi181 ceph-mon[94189]: pgmap v4122: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 199 KiB/s wr, 10 op/s 2023-03-09T16:50:56.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:50:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6766.029632 secs 2023-03-09T16:50:56.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:50:55 smithi038 ceph-mon[92422]: pgmap v4122: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 199 KiB/s wr, 10 op/s 2023-03-09T16:50:57.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:50:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6768.275549 secs 2023-03-09T16:50:57.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:50:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6768.275549 secs 2023-03-09T16:50:57.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:50:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6768.275549 secs 2023-03-09T16:50:58.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:50:58 smithi122 ceph-mon[93869]: pgmap v4123: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 233 KiB/s wr, 13 op/s 2023-03-09T16:50:58.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:50:58 smithi181 ceph-mon[94189]: pgmap v4123: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 233 KiB/s wr, 13 op/s 2023-03-09T16:50:58.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:50:58 smithi038 ceph-mon[92422]: pgmap v4123: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 233 KiB/s wr, 13 op/s 2023-03-09T16:50:58.793 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7230.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:51:00.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:51:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:51:00.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:51:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:51:00.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:51:00 smithi122 ceph-mon[93869]: pgmap v4124: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 200 KiB/s wr, 9 op/s 2023-03-09T16:51:00.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:51:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6771.029692 secs 2023-03-09T16:51:00.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:51:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:51:00.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:51:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:51:00.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:51:00 smithi181 ceph-mon[94189]: pgmap v4124: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 200 KiB/s wr, 9 op/s 2023-03-09T16:51:00.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:51:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6771.029692 secs 2023-03-09T16:51:00.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:51:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:51:00.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:51:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:51:00.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:51:00 smithi038 ceph-mon[92422]: pgmap v4124: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 200 KiB/s wr, 9 op/s 2023-03-09T16:51:00.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:51:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6771.029692 secs 2023-03-09T16:51:02.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:51:02 smithi122 ceph-mon[93869]: pgmap v4125: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 210 KiB/s wr, 12 op/s 2023-03-09T16:51:02.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:51:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6773.275639 secs 2023-03-09T16:51:02.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:51:02 smithi181 ceph-mon[94189]: pgmap v4125: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 210 KiB/s wr, 12 op/s 2023-03-09T16:51:02.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:51:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6773.275639 secs 2023-03-09T16:51:02.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:51:02 smithi038 ceph-mon[92422]: pgmap v4125: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 210 KiB/s wr, 12 op/s 2023-03-09T16:51:02.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:51:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6773.275639 secs 2023-03-09T16:51:03.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7235.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:51:04.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:51:04 smithi122 ceph-mon[93869]: pgmap v4126: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 152 KiB/s wr, 21 op/s 2023-03-09T16:51:04.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:51:04 smithi181 ceph-mon[94189]: pgmap v4126: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 152 KiB/s wr, 21 op/s 2023-03-09T16:51:04.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:51:04 smithi038 ceph-mon[92422]: pgmap v4126: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 152 KiB/s wr, 21 op/s 2023-03-09T16:51:06.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:51:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6776.029752 secs 2023-03-09T16:51:06.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:51:05 smithi122 ceph-mon[93869]: pgmap v4127: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 152 KiB/s wr, 20 op/s 2023-03-09T16:51:06.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:51:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6776.029752 secs 2023-03-09T16:51:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:51:05 smithi181 ceph-mon[94189]: pgmap v4127: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 152 KiB/s wr, 20 op/s 2023-03-09T16:51:06.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:51:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6776.029752 secs 2023-03-09T16:51:06.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:51:05 smithi038 ceph-mon[92422]: pgmap v4127: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 152 KiB/s wr, 20 op/s 2023-03-09T16:51:07.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:51:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6778.275764 secs 2023-03-09T16:51:07.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:51:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6778.275764 secs 2023-03-09T16:51:07.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:51:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6778.275764 secs 2023-03-09T16:51:08.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:51:08 smithi122 ceph-mon[93869]: pgmap v4128: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 174 KiB/s wr, 25 op/s 2023-03-09T16:51:08.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:51:08 smithi181 ceph-mon[94189]: pgmap v4128: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 174 KiB/s wr, 25 op/s 2023-03-09T16:51:08.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:51:08 smithi038 ceph-mon[92422]: pgmap v4128: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 174 KiB/s wr, 25 op/s 2023-03-09T16:51:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7240.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:51:10.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:51:10 smithi122 ceph-mon[93869]: osdmap e471: 12 total, 12 up, 12 in 2023-03-09T16:51:10.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:51:10 smithi122 ceph-mon[93869]: pgmap v4130: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 145 KiB/s wr, 26 op/s 2023-03-09T16:51:10.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:51:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6781.029813 secs 2023-03-09T16:51:10.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:51:10 smithi181 ceph-mon[94189]: osdmap e471: 12 total, 12 up, 12 in 2023-03-09T16:51:10.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:51:10 smithi181 ceph-mon[94189]: pgmap v4130: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 145 KiB/s wr, 26 op/s 2023-03-09T16:51:10.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:51:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6781.029813 secs 2023-03-09T16:51:10.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:51:10 smithi038 ceph-mon[92422]: osdmap e471: 12 total, 12 up, 12 in 2023-03-09T16:51:10.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:51:10 smithi038 ceph-mon[92422]: pgmap v4130: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 145 KiB/s wr, 26 op/s 2023-03-09T16:51:10.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:51:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6781.029813 secs 2023-03-09T16:51:12.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:51:12 smithi122 ceph-mon[93869]: pgmap v4131: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 132 KiB/s wr, 25 op/s 2023-03-09T16:51:12.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:51:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6783.275876 secs 2023-03-09T16:51:12.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:51:12 smithi181 ceph-mon[94189]: pgmap v4131: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 132 KiB/s wr, 25 op/s 2023-03-09T16:51:12.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:51:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6783.275876 secs 2023-03-09T16:51:12.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:51:12 smithi038 ceph-mon[92422]: pgmap v4131: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 132 KiB/s wr, 25 op/s 2023-03-09T16:51:12.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:51:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6783.275876 secs 2023-03-09T16:51:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7245.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:51:14.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:51:14 smithi122 ceph-mon[93869]: pgmap v4132: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 60 KiB/s wr, 9 op/s 2023-03-09T16:51:14.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:51:14 smithi181 ceph-mon[94189]: pgmap v4132: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 60 KiB/s wr, 9 op/s 2023-03-09T16:51:14.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:51:14 smithi038 ceph-mon[92422]: pgmap v4132: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 60 KiB/s wr, 9 op/s 2023-03-09T16:51:16.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:51:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6786.029873 secs 2023-03-09T16:51:16.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:51:15 smithi122 ceph-mon[93869]: pgmap v4133: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 60 KiB/s wr, 9 op/s 2023-03-09T16:51:16.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:51:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6786.029873 secs 2023-03-09T16:51:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:51:15 smithi181 ceph-mon[94189]: pgmap v4133: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 60 KiB/s wr, 9 op/s 2023-03-09T16:51:16.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:51:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6786.029873 secs 2023-03-09T16:51:16.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:51:15 smithi038 ceph-mon[92422]: pgmap v4133: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 60 KiB/s wr, 9 op/s 2023-03-09T16:51:17.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:51:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6788.275940 secs 2023-03-09T16:51:17.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:51:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6788.275940 secs 2023-03-09T16:51:17.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:51:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6788.275940 secs 2023-03-09T16:51:18.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:51:18 smithi122 ceph-mon[93869]: pgmap v4134: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 32 KiB/s wr, 6 op/s 2023-03-09T16:51:18.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:51:18 smithi181 ceph-mon[94189]: pgmap v4134: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 32 KiB/s wr, 6 op/s 2023-03-09T16:51:18.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:51:18 smithi038 ceph-mon[92422]: pgmap v4134: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 32 KiB/s wr, 6 op/s 2023-03-09T16:51:18.793 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7250.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:51:20.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:51:20 smithi122 ceph-mon[93869]: pgmap v4135: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 198 B/s wr, 6 op/s 2023-03-09T16:51:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:51:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6791.029964 secs 2023-03-09T16:51:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:51:20 smithi122 ceph-mon[93869]: osdmap e472: 12 total, 12 up, 12 in 2023-03-09T16:51:20.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:51:20 smithi181 ceph-mon[94189]: pgmap v4135: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 198 B/s wr, 6 op/s 2023-03-09T16:51:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:51:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6791.029964 secs 2023-03-09T16:51:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:51:20 smithi181 ceph-mon[94189]: osdmap e472: 12 total, 12 up, 12 in 2023-03-09T16:51:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:51:20 smithi038 ceph-mon[92422]: pgmap v4135: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 198 B/s wr, 6 op/s 2023-03-09T16:51:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:51:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6791.029964 secs 2023-03-09T16:51:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:51:20 smithi038 ceph-mon[92422]: osdmap e472: 12 total, 12 up, 12 in 2023-03-09T16:51:22.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:51:22 smithi122 ceph-mon[93869]: pgmap v4137: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T16:51:22.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:51:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6793.276057 secs 2023-03-09T16:51:22.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:51:22 smithi181 ceph-mon[94189]: pgmap v4137: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T16:51:22.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:51:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6793.276057 secs 2023-03-09T16:51:22.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:51:22 smithi038 ceph-mon[92422]: pgmap v4137: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T16:51:22.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:51:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6793.276057 secs 2023-03-09T16:51:23.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7255.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:51:24.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:51:24 smithi122 ceph-mon[93869]: pgmap v4138: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T16:51:24.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:51:24 smithi181 ceph-mon[94189]: pgmap v4138: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T16:51:24.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:51:24 smithi038 ceph-mon[92422]: pgmap v4138: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T16:51:26.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:51:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6796.029983 secs 2023-03-09T16:51:26.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:51:25 smithi122 ceph-mon[93869]: pgmap v4139: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T16:51:26.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:51:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6796.029983 secs 2023-03-09T16:51:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:51:25 smithi181 ceph-mon[94189]: pgmap v4139: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T16:51:26.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:51:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6796.029983 secs 2023-03-09T16:51:26.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:51:25 smithi038 ceph-mon[92422]: pgmap v4139: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T16:51:27.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:51:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6798.276171 secs 2023-03-09T16:51:27.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:51:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6798.276171 secs 2023-03-09T16:51:27.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:51:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6798.276171 secs 2023-03-09T16:51:28.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:51:28 smithi122 ceph-mon[93869]: pgmap v4140: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 614 B/s wr, 6 op/s 2023-03-09T16:51:28.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:51:28 smithi181 ceph-mon[94189]: pgmap v4140: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 614 B/s wr, 6 op/s 2023-03-09T16:51:28.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:51:28 smithi038 ceph-mon[92422]: pgmap v4140: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 614 B/s wr, 6 op/s 2023-03-09T16:51:28.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7260.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:51:30.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:51:30 smithi122 ceph-mon[93869]: pgmap v4141: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 40 KiB/s wr, 7 op/s 2023-03-09T16:51:30.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:51:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6801.030046 secs 2023-03-09T16:51:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:51:30 smithi181 ceph-mon[94189]: pgmap v4141: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 40 KiB/s wr, 7 op/s 2023-03-09T16:51:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:51:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6801.030046 secs 2023-03-09T16:51:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:51:30 smithi038 ceph-mon[92422]: pgmap v4141: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 40 KiB/s wr, 7 op/s 2023-03-09T16:51:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:51:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6801.030046 secs 2023-03-09T16:51:32.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:51:32 smithi122 ceph-mon[93869]: pgmap v4142: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 35 KiB/s wr, 8 op/s 2023-03-09T16:51:32.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:51:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6803.276285 secs 2023-03-09T16:51:32.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:51:32 smithi181 ceph-mon[94189]: pgmap v4142: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 35 KiB/s wr, 8 op/s 2023-03-09T16:51:32.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:51:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6803.276285 secs 2023-03-09T16:51:32.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:51:32 smithi038 ceph-mon[92422]: pgmap v4142: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 35 KiB/s wr, 8 op/s 2023-03-09T16:51:32.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:51:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6803.276285 secs 2023-03-09T16:51:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7265.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:51:34.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:51:34 smithi122 ceph-mon[93869]: pgmap v4143: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 146 KiB/s wr, 7 op/s 2023-03-09T16:51:34.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:51:34 smithi181 ceph-mon[94189]: pgmap v4143: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 146 KiB/s wr, 7 op/s 2023-03-09T16:51:34.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:51:34 smithi038 ceph-mon[92422]: pgmap v4143: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 146 KiB/s wr, 7 op/s 2023-03-09T16:51:36.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:51:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6806.030106 secs 2023-03-09T16:51:36.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:51:35 smithi122 ceph-mon[93869]: pgmap v4144: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 146 KiB/s wr, 7 op/s 2023-03-09T16:51:36.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:51:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6806.030106 secs 2023-03-09T16:51:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:51:35 smithi181 ceph-mon[94189]: pgmap v4144: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 146 KiB/s wr, 7 op/s 2023-03-09T16:51:36.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:51:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6806.030106 secs 2023-03-09T16:51:36.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:51:35 smithi038 ceph-mon[92422]: pgmap v4144: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 146 KiB/s wr, 7 op/s 2023-03-09T16:51:37.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:51:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6808.276363 secs 2023-03-09T16:51:37.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:51:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6808.276363 secs 2023-03-09T16:51:37.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:51:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6808.276363 secs 2023-03-09T16:51:38.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:51:38 smithi122 ceph-mon[93869]: pgmap v4145: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 160 KiB/s wr, 11 op/s 2023-03-09T16:51:38.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:51:38 smithi181 ceph-mon[94189]: pgmap v4145: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 160 KiB/s wr, 11 op/s 2023-03-09T16:51:38.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:51:38 smithi038 ceph-mon[92422]: pgmap v4145: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 160 KiB/s wr, 11 op/s 2023-03-09T16:51:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7270.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:51:40.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:51:40 smithi122 ceph-mon[93869]: pgmap v4146: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 226 KiB/s wr, 10 op/s 2023-03-09T16:51:40.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:51:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6811.030166 secs 2023-03-09T16:51:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:51:40 smithi181 ceph-mon[94189]: pgmap v4146: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 226 KiB/s wr, 10 op/s 2023-03-09T16:51:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:51:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6811.030166 secs 2023-03-09T16:51:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:51:40 smithi038 ceph-mon[92422]: pgmap v4146: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 226 KiB/s wr, 10 op/s 2023-03-09T16:51:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:51:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6811.030166 secs 2023-03-09T16:51:42.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:51:42 smithi122 ceph-mon[93869]: pgmap v4147: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 219 KiB/s wr, 12 op/s 2023-03-09T16:51:42.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:51:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6813.276440 secs 2023-03-09T16:51:42.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:51:42 smithi181 ceph-mon[94189]: pgmap v4147: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 219 KiB/s wr, 12 op/s 2023-03-09T16:51:42.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:51:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6813.276440 secs 2023-03-09T16:51:42.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:51:42 smithi038 ceph-mon[92422]: pgmap v4147: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 219 KiB/s wr, 12 op/s 2023-03-09T16:51:42.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:51:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6813.276440 secs 2023-03-09T16:51:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7275.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:51:44.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:51:44 smithi122 ceph-mon[93869]: pgmap v4148: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 238 KiB/s wr, 10 op/s 2023-03-09T16:51:44.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:51:44 smithi181 ceph-mon[94189]: pgmap v4148: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 238 KiB/s wr, 10 op/s 2023-03-09T16:51:44.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:51:44 smithi038 ceph-mon[92422]: pgmap v4148: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 238 KiB/s wr, 10 op/s 2023-03-09T16:51:46.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:51:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6816.030227 secs 2023-03-09T16:51:46.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:51:45 smithi122 ceph-mon[93869]: pgmap v4149: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 256 KiB/s wr, 11 op/s 2023-03-09T16:51:46.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:51:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6816.030227 secs 2023-03-09T16:51:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:51:45 smithi181 ceph-mon[94189]: pgmap v4149: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 256 KiB/s wr, 11 op/s 2023-03-09T16:51:46.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:51:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6816.030227 secs 2023-03-09T16:51:46.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:51:45 smithi038 ceph-mon[92422]: pgmap v4149: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 256 KiB/s wr, 11 op/s 2023-03-09T16:51:47.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:51:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6818.276597 secs 2023-03-09T16:51:47.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:51:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6818.276597 secs 2023-03-09T16:51:47.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:51:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6818.276597 secs 2023-03-09T16:51:48.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:51:48 smithi122 ceph-mon[93869]: pgmap v4150: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 273 KiB/s wr, 14 op/s 2023-03-09T16:51:48.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:51:48 smithi181 ceph-mon[94189]: pgmap v4150: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 273 KiB/s wr, 14 op/s 2023-03-09T16:51:48.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:51:48 smithi038 ceph-mon[92422]: pgmap v4150: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 273 KiB/s wr, 14 op/s 2023-03-09T16:51:48.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7280.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:51:50.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:51:50 smithi122 ceph-mon[93869]: pgmap v4151: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 259 KiB/s wr, 11 op/s 2023-03-09T16:51:50.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:51:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6821.030312 secs 2023-03-09T16:51:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:51:50 smithi181 ceph-mon[94189]: pgmap v4151: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 259 KiB/s wr, 11 op/s 2023-03-09T16:51:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:51:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6821.030312 secs 2023-03-09T16:51:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:51:50 smithi038 ceph-mon[92422]: pgmap v4151: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 259 KiB/s wr, 11 op/s 2023-03-09T16:51:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:51:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6821.030312 secs 2023-03-09T16:51:52.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:51:52 smithi122 ceph-mon[93869]: pgmap v4152: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 231 KiB/s wr, 12 op/s 2023-03-09T16:51:52.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:51:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6823.276715 secs 2023-03-09T16:51:52.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:51:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:51:52.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:51:52 smithi181 ceph-mon[94189]: pgmap v4152: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 231 KiB/s wr, 12 op/s 2023-03-09T16:51:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:51:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6823.276715 secs 2023-03-09T16:51:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:51:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:51:52.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:51:52 smithi038 ceph-mon[92422]: pgmap v4152: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 231 KiB/s wr, 12 op/s 2023-03-09T16:51:52.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:51:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6823.276715 secs 2023-03-09T16:51:52.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:51:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:51:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7285.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:51:53.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:51:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:51:53.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:51:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:51:53.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:51:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:51:53.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:51:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:51:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:51:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:51:53.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:51:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:51:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:51:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:51:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:51:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:51:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:51:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:51:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:51:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:51:53.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:51:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:51:53.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:51:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:51:53.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:51:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:51:53.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:51:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:51:53.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:51:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:51:54.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:51:54 smithi122 ceph-mon[93869]: pgmap v4153: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 206 KiB/s wr, 10 op/s 2023-03-09T16:51:54.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:51:54 smithi181 ceph-mon[94189]: pgmap v4153: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 206 KiB/s wr, 10 op/s 2023-03-09T16:51:54.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:51:54 smithi038 ceph-mon[92422]: pgmap v4153: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 206 KiB/s wr, 10 op/s 2023-03-09T16:51:56.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:51:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6826.030397 secs 2023-03-09T16:51:56.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:51:55 smithi122 ceph-mon[93869]: pgmap v4154: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 243 KiB/s wr, 9 op/s 2023-03-09T16:51:56.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:51:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6826.030397 secs 2023-03-09T16:51:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:51:55 smithi181 ceph-mon[94189]: pgmap v4154: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 243 KiB/s wr, 9 op/s 2023-03-09T16:51:56.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:51:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6826.030397 secs 2023-03-09T16:51:56.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:51:55 smithi038 ceph-mon[92422]: pgmap v4154: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 243 KiB/s wr, 9 op/s 2023-03-09T16:51:57.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:51:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6828.276833 secs 2023-03-09T16:51:57.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:51:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6828.276833 secs 2023-03-09T16:51:57.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:51:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6828.276833 secs 2023-03-09T16:51:58.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:51:58 smithi122 ceph-mon[93869]: pgmap v4155: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 114 KiB/s wr, 10 op/s 2023-03-09T16:51:58.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:51:58 smithi181 ceph-mon[94189]: pgmap v4155: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 114 KiB/s wr, 10 op/s 2023-03-09T16:51:58.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:51:58 smithi038 ceph-mon[92422]: pgmap v4155: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 114 KiB/s wr, 10 op/s 2023-03-09T16:51:58.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7290.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:52:00.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:52:00 smithi122 ceph-mon[93869]: pgmap v4156: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 96 KiB/s wr, 7 op/s 2023-03-09T16:52:00.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:52:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6831.030457 secs 2023-03-09T16:52:00.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:52:00 smithi181 ceph-mon[94189]: pgmap v4156: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 96 KiB/s wr, 7 op/s 2023-03-09T16:52:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:52:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6831.030457 secs 2023-03-09T16:52:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:52:00 smithi038 ceph-mon[92422]: pgmap v4156: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 96 KiB/s wr, 7 op/s 2023-03-09T16:52:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:52:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6831.030457 secs 2023-03-09T16:52:02.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:52:02 smithi122 ceph-mon[93869]: pgmap v4157: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 123 KiB/s wr, 9 op/s 2023-03-09T16:52:02.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:52:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6833.276958 secs 2023-03-09T16:52:02.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:52:02 smithi181 ceph-mon[94189]: pgmap v4157: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 123 KiB/s wr, 9 op/s 2023-03-09T16:52:02.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:52:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6833.276958 secs 2023-03-09T16:52:02.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:52:02 smithi038 ceph-mon[92422]: pgmap v4157: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 123 KiB/s wr, 9 op/s 2023-03-09T16:52:02.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:52:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6833.276958 secs 2023-03-09T16:52:03.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7295.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:52:04.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:52:04 smithi122 ceph-mon[93869]: pgmap v4158: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 106 KiB/s wr, 18 op/s 2023-03-09T16:52:04.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:52:04 smithi181 ceph-mon[94189]: pgmap v4158: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 106 KiB/s wr, 18 op/s 2023-03-09T16:52:04.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:52:04 smithi038 ceph-mon[92422]: pgmap v4158: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 106 KiB/s wr, 18 op/s 2023-03-09T16:52:06.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:52:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6836.030518 secs 2023-03-09T16:52:06.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:52:05 smithi122 ceph-mon[93869]: pgmap v4159: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 107 KiB/s wr, 19 op/s 2023-03-09T16:52:06.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:52:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6836.030518 secs 2023-03-09T16:52:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:52:05 smithi181 ceph-mon[94189]: pgmap v4159: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 107 KiB/s wr, 19 op/s 2023-03-09T16:52:06.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:52:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6836.030518 secs 2023-03-09T16:52:06.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:52:05 smithi038 ceph-mon[92422]: pgmap v4159: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 107 KiB/s wr, 19 op/s 2023-03-09T16:52:07.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:52:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6838.277079 secs 2023-03-09T16:52:07.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:52:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6838.277079 secs 2023-03-09T16:52:07.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:52:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6838.277079 secs 2023-03-09T16:52:08.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:52:08 smithi122 ceph-mon[93869]: pgmap v4160: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 72 KiB/s wr, 22 op/s 2023-03-09T16:52:08.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:52:08 smithi181 ceph-mon[94189]: pgmap v4160: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 72 KiB/s wr, 22 op/s 2023-03-09T16:52:08.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:52:08 smithi038 ceph-mon[92422]: pgmap v4160: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 72 KiB/s wr, 22 op/s 2023-03-09T16:52:08.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7300.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:52:10.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:52:10 smithi122 ceph-mon[93869]: osdmap e473: 12 total, 12 up, 12 in 2023-03-09T16:52:10.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:52:10 smithi122 ceph-mon[93869]: pgmap v4162: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 87 KiB/s wr, 24 op/s 2023-03-09T16:52:10.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:52:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6841.030578 secs 2023-03-09T16:52:10.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:52:10 smithi181 ceph-mon[94189]: osdmap e473: 12 total, 12 up, 12 in 2023-03-09T16:52:10.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:52:10 smithi181 ceph-mon[94189]: pgmap v4162: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 87 KiB/s wr, 24 op/s 2023-03-09T16:52:10.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:52:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6841.030578 secs 2023-03-09T16:52:10.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:52:10 smithi038 ceph-mon[92422]: osdmap e473: 12 total, 12 up, 12 in 2023-03-09T16:52:10.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:52:10 smithi038 ceph-mon[92422]: pgmap v4162: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 87 KiB/s wr, 24 op/s 2023-03-09T16:52:10.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:52:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6841.030578 secs 2023-03-09T16:52:12.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:52:12 smithi122 ceph-mon[93869]: pgmap v4163: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 86 KiB/s wr, 24 op/s 2023-03-09T16:52:12.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:52:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6843.277197 secs 2023-03-09T16:52:12.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:52:12 smithi181 ceph-mon[94189]: pgmap v4163: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 86 KiB/s wr, 24 op/s 2023-03-09T16:52:12.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:52:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6843.277197 secs 2023-03-09T16:52:12.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:52:12 smithi038 ceph-mon[92422]: pgmap v4163: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 86 KiB/s wr, 24 op/s 2023-03-09T16:52:12.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:52:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6843.277197 secs 2023-03-09T16:52:13.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7305.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:52:14.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:52:14 smithi122 ceph-mon[93869]: pgmap v4164: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 61 KiB/s wr, 9 op/s 2023-03-09T16:52:14.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:52:14 smithi181 ceph-mon[94189]: pgmap v4164: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 61 KiB/s wr, 9 op/s 2023-03-09T16:52:14.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:52:14 smithi038 ceph-mon[92422]: pgmap v4164: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 61 KiB/s wr, 9 op/s 2023-03-09T16:52:16.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:52:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6846.030637 secs 2023-03-09T16:52:16.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:52:15 smithi038 ceph-mon[92422]: pgmap v4165: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 59 KiB/s wr, 9 op/s 2023-03-09T16:52:16.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:52:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6846.030637 secs 2023-03-09T16:52:16.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:52:15 smithi122 ceph-mon[93869]: pgmap v4165: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 59 KiB/s wr, 9 op/s 2023-03-09T16:52:16.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:52:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6846.030637 secs 2023-03-09T16:52:16.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:52:15 smithi181 ceph-mon[94189]: pgmap v4165: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 59 KiB/s wr, 9 op/s 2023-03-09T16:52:17.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:52:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6848.277316 secs 2023-03-09T16:52:17.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:52:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6848.277316 secs 2023-03-09T16:52:17.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:52:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6848.277316 secs 2023-03-09T16:52:18.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:52:18 smithi122 ceph-mon[93869]: pgmap v4166: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 32 KiB/s wr, 6 op/s 2023-03-09T16:52:18.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:52:18 smithi181 ceph-mon[94189]: pgmap v4166: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 32 KiB/s wr, 6 op/s 2023-03-09T16:52:18.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:52:18 smithi038 ceph-mon[92422]: pgmap v4166: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 32 KiB/s wr, 6 op/s 2023-03-09T16:52:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7310.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:52:20.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:52:20 smithi122 ceph-mon[93869]: pgmap v4167: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 31 KiB/s wr, 6 op/s 2023-03-09T16:52:20.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:52:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6851.030697 secs 2023-03-09T16:52:20.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:52:20 smithi122 ceph-mon[93869]: osdmap e474: 12 total, 12 up, 12 in 2023-03-09T16:52:20.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:52:20 smithi181 ceph-mon[94189]: pgmap v4167: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 31 KiB/s wr, 6 op/s 2023-03-09T16:52:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:52:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6851.030697 secs 2023-03-09T16:52:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:52:20 smithi181 ceph-mon[94189]: osdmap e474: 12 total, 12 up, 12 in 2023-03-09T16:52:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:52:20 smithi038 ceph-mon[92422]: pgmap v4167: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 31 KiB/s wr, 6 op/s 2023-03-09T16:52:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:52:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6851.030697 secs 2023-03-09T16:52:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:52:20 smithi038 ceph-mon[92422]: osdmap e474: 12 total, 12 up, 12 in 2023-03-09T16:52:22.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:52:22 smithi122 ceph-mon[93869]: pgmap v4169: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 614 B/s wr, 7 op/s 2023-03-09T16:52:22.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:52:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6853.277439 secs 2023-03-09T16:52:22.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:52:22 smithi181 ceph-mon[94189]: pgmap v4169: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 614 B/s wr, 7 op/s 2023-03-09T16:52:22.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:52:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6853.277439 secs 2023-03-09T16:52:22.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:52:22 smithi038 ceph-mon[92422]: pgmap v4169: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 614 B/s wr, 7 op/s 2023-03-09T16:52:22.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:52:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6853.277439 secs 2023-03-09T16:52:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7315.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:52:24.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:52:24 smithi122 ceph-mon[93869]: pgmap v4170: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 716 B/s wr, 7 op/s 2023-03-09T16:52:24.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:52:24 smithi181 ceph-mon[94189]: pgmap v4170: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 716 B/s wr, 7 op/s 2023-03-09T16:52:24.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:52:24 smithi038 ceph-mon[92422]: pgmap v4170: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 716 B/s wr, 7 op/s 2023-03-09T16:52:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:52:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6856.030758 secs 2023-03-09T16:52:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:52:25 smithi181 ceph-mon[94189]: pgmap v4171: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 45 KiB/s wr, 7 op/s 2023-03-09T16:52:26.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:52:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6856.030758 secs 2023-03-09T16:52:26.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:52:25 smithi038 ceph-mon[92422]: pgmap v4171: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 45 KiB/s wr, 7 op/s 2023-03-09T16:52:26.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:52:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6856.030758 secs 2023-03-09T16:52:26.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:52:25 smithi122 ceph-mon[93869]: pgmap v4171: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 45 KiB/s wr, 7 op/s 2023-03-09T16:52:27.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:52:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6858.277555 secs 2023-03-09T16:52:27.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:52:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6858.277555 secs 2023-03-09T16:52:27.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:52:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6858.277555 secs 2023-03-09T16:52:28.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:52:28 smithi122 ceph-mon[93869]: pgmap v4172: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 51 KiB/s wr, 8 op/s 2023-03-09T16:52:28.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:52:28 smithi181 ceph-mon[94189]: pgmap v4172: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 51 KiB/s wr, 8 op/s 2023-03-09T16:52:28.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:52:28 smithi038 ceph-mon[92422]: pgmap v4172: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 51 KiB/s wr, 8 op/s 2023-03-09T16:52:28.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7320.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:52:30.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:52:30 smithi122 ceph-mon[93869]: pgmap v4173: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 51 KiB/s wr, 7 op/s 2023-03-09T16:52:30.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:52:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6861.030819 secs 2023-03-09T16:52:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:52:30 smithi181 ceph-mon[94189]: pgmap v4173: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 51 KiB/s wr, 7 op/s 2023-03-09T16:52:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:52:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6861.030819 secs 2023-03-09T16:52:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:52:30 smithi038 ceph-mon[92422]: pgmap v4173: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 51 KiB/s wr, 7 op/s 2023-03-09T16:52:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:52:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6861.030819 secs 2023-03-09T16:52:32.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:52:32 smithi122 ceph-mon[93869]: pgmap v4174: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 45 KiB/s wr, 9 op/s 2023-03-09T16:52:32.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:52:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6863.277687 secs 2023-03-09T16:52:32.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:52:32 smithi181 ceph-mon[94189]: pgmap v4174: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 45 KiB/s wr, 9 op/s 2023-03-09T16:52:32.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:52:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6863.277687 secs 2023-03-09T16:52:32.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:52:32 smithi038 ceph-mon[92422]: pgmap v4174: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 45 KiB/s wr, 9 op/s 2023-03-09T16:52:32.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:52:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6863.277687 secs 2023-03-09T16:52:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7325.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:52:34.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:52:34 smithi122 ceph-mon[93869]: pgmap v4175: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 43 KiB/s wr, 6 op/s 2023-03-09T16:52:34.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:52:34 smithi181 ceph-mon[94189]: pgmap v4175: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 43 KiB/s wr, 6 op/s 2023-03-09T16:52:34.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:52:34 smithi038 ceph-mon[92422]: pgmap v4175: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 43 KiB/s wr, 6 op/s 2023-03-09T16:52:35.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:52:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6866.030883 secs 2023-03-09T16:52:35.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:52:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6866.030883 secs 2023-03-09T16:52:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:52:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6866.030883 secs 2023-03-09T16:52:36.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:52:36 smithi122 ceph-mon[93869]: pgmap v4176: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 43 KiB/s wr, 7 op/s 2023-03-09T16:52:36.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:52:36 smithi181 ceph-mon[94189]: pgmap v4176: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 43 KiB/s wr, 7 op/s 2023-03-09T16:52:36.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:52:36 smithi038 ceph-mon[92422]: pgmap v4176: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 43 KiB/s wr, 7 op/s 2023-03-09T16:52:37.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:52:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6868.277762 secs 2023-03-09T16:52:37.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:52:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6868.277762 secs 2023-03-09T16:52:37.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:52:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6868.277762 secs 2023-03-09T16:52:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7330.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:52:38.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:52:38 smithi122 ceph-mon[93869]: pgmap v4177: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 50 KiB/s wr, 9 op/s 2023-03-09T16:52:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:52:38 smithi181 ceph-mon[94189]: pgmap v4177: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 50 KiB/s wr, 9 op/s 2023-03-09T16:52:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:52:38 smithi038 ceph-mon[92422]: pgmap v4177: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 50 KiB/s wr, 9 op/s 2023-03-09T16:52:40.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:52:40 smithi122 ceph-mon[93869]: pgmap v4178: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 45 KiB/s wr, 6 op/s 2023-03-09T16:52:40.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:52:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6871.030943 secs 2023-03-09T16:52:40.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:52:40 smithi181 ceph-mon[94189]: pgmap v4178: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 45 KiB/s wr, 6 op/s 2023-03-09T16:52:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:52:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6871.030943 secs 2023-03-09T16:52:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:52:40 smithi038 ceph-mon[92422]: pgmap v4178: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 45 KiB/s wr, 6 op/s 2023-03-09T16:52:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:52:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6871.030943 secs 2023-03-09T16:52:42.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:52:42 smithi122 ceph-mon[93869]: pgmap v4179: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 133 KiB/s wr, 10 op/s 2023-03-09T16:52:42.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:52:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6873.277907 secs 2023-03-09T16:52:42.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:52:42 smithi181 ceph-mon[94189]: pgmap v4179: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 133 KiB/s wr, 10 op/s 2023-03-09T16:52:42.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:52:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6873.277907 secs 2023-03-09T16:52:42.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:52:42 smithi038 ceph-mon[92422]: pgmap v4179: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 133 KiB/s wr, 10 op/s 2023-03-09T16:52:42.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:52:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6873.277907 secs 2023-03-09T16:52:43.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7335.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:52:44.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:52:44 smithi122 ceph-mon[93869]: pgmap v4180: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 133 KiB/s wr, 7 op/s 2023-03-09T16:52:44.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:52:44 smithi181 ceph-mon[94189]: pgmap v4180: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 133 KiB/s wr, 7 op/s 2023-03-09T16:52:44.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:52:44 smithi038 ceph-mon[92422]: pgmap v4180: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 133 KiB/s wr, 7 op/s 2023-03-09T16:52:45.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:52:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6876.031005 secs 2023-03-09T16:52:45.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:52:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6876.031005 secs 2023-03-09T16:52:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:52:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6876.031005 secs 2023-03-09T16:52:46.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:52:46 smithi122 ceph-mon[93869]: pgmap v4181: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 133 KiB/s wr, 7 op/s 2023-03-09T16:52:46.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:52:46 smithi181 ceph-mon[94189]: pgmap v4181: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 133 KiB/s wr, 7 op/s 2023-03-09T16:52:46.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:52:46 smithi038 ceph-mon[92422]: pgmap v4181: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 133 KiB/s wr, 7 op/s 2023-03-09T16:52:47.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:52:47 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:52:47.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:52:47 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:52:47.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:52:47 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:52:48.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7340.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:52:48.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:52:48 smithi122 ceph-mon[93869]: pgmap v4182: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 215 KiB/s wr, 11 op/s 2023-03-09T16:52:48.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:52:48 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:52:48.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:52:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:52:48.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:52:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:52:48.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:52:48 smithi181 ceph-mon[94189]: pgmap v4182: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 215 KiB/s wr, 11 op/s 2023-03-09T16:52:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:52:48 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:52:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:52:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:52:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:52:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:52:48.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:52:48 smithi038 ceph-mon[92422]: pgmap v4182: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 215 KiB/s wr, 11 op/s 2023-03-09T16:52:48.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:52:48 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:52:48.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:52:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:52:48.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:52:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:52:50.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:52:50 smithi122 ceph-mon[93869]: pgmap v4183: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 171 KiB/s wr, 9 op/s 2023-03-09T16:52:50.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:52:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6881.031067 secs 2023-03-09T16:52:50.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:52:50 smithi181 ceph-mon[94189]: pgmap v4183: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 171 KiB/s wr, 9 op/s 2023-03-09T16:52:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:52:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6881.031067 secs 2023-03-09T16:52:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:52:50 smithi038 ceph-mon[92422]: pgmap v4183: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 171 KiB/s wr, 9 op/s 2023-03-09T16:52:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:52:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6881.031067 secs 2023-03-09T16:52:52.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:52:52 smithi122 ceph-mon[93869]: pgmap v4184: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 290 KiB/s wr, 13 op/s 2023-03-09T16:52:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:52:52 smithi181 ceph-mon[94189]: pgmap v4184: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 290 KiB/s wr, 13 op/s 2023-03-09T16:52:52.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:52:52 smithi038 ceph-mon[92422]: pgmap v4184: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 290 KiB/s wr, 13 op/s 2023-03-09T16:52:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7345.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:52:53.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:52:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:52:53.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:52:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:52:53.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:52:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:52:54.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:52:54 smithi122 ceph-mon[93869]: pgmap v4185: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 202 KiB/s wr, 10 op/s 2023-03-09T16:52:54.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:52:54 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:52:54.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:52:54 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:52:54.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:52:54 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:52:54.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:52:54 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:52:54.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:52:54 smithi181 ceph-mon[94189]: pgmap v4185: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 202 KiB/s wr, 10 op/s 2023-03-09T16:52:54.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:52:54 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:52:54.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:52:54 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:52:54.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:52:54 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:52:54.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:52:54 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:52:54.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:52:54 smithi038 ceph-mon[92422]: pgmap v4185: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 202 KiB/s wr, 10 op/s 2023-03-09T16:52:54.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:52:54 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:52:54.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:52:54 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:52:54.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:52:54 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:52:54.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:52:54 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:52:55.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:52:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6886.031142 secs 2023-03-09T16:52:55.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:52:55 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:52:55.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:52:55 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:52:55.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:52:55 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:52:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:52:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6886.031142 secs 2023-03-09T16:52:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:52:55 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:52:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:52:55 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:52:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:52:55 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:52:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:52:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6886.031142 secs 2023-03-09T16:52:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:52:55 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:52:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:52:55 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:52:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:52:55 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:52:56.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:52:56 smithi122 ceph-mon[93869]: pgmap v4186: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 202 KiB/s wr, 10 op/s 2023-03-09T16:52:56.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:52:56 smithi181 ceph-mon[94189]: pgmap v4186: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 202 KiB/s wr, 10 op/s 2023-03-09T16:52:56.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:52:56 smithi038 ceph-mon[92422]: pgmap v4186: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 202 KiB/s wr, 10 op/s 2023-03-09T16:52:57.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:52:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6888.278289 secs 2023-03-09T16:52:57.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:52:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6888.278289 secs 2023-03-09T16:52:57.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:52:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6888.278289 secs 2023-03-09T16:52:58.793 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7350.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:52:58.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:52:58 smithi122 ceph-mon[93869]: pgmap v4187: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 236 KiB/s wr, 13 op/s 2023-03-09T16:52:58.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:52:58 smithi181 ceph-mon[94189]: pgmap v4187: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 236 KiB/s wr, 13 op/s 2023-03-09T16:52:58.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:52:58 smithi038 ceph-mon[92422]: pgmap v4187: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 236 KiB/s wr, 13 op/s 2023-03-09T16:52:59.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:52:59 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:52:59.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:52:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:52:59.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:52:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:53:00.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:52:59 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:53:00.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:52:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:53:00.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:52:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:53:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:52:59 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:53:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:52:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:53:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:52:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:53:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:53:00 smithi038 ceph-mon[92422]: pgmap v4188: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 250 KiB/s wr, 10 op/s 2023-03-09T16:53:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:53:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6891.031204 secs 2023-03-09T16:53:01.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:53:00 smithi122 ceph-mon[93869]: pgmap v4188: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 250 KiB/s wr, 10 op/s 2023-03-09T16:53:01.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:53:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6891.031204 secs 2023-03-09T16:53:01.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:53:00 smithi181 ceph-mon[94189]: pgmap v4188: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 250 KiB/s wr, 10 op/s 2023-03-09T16:53:01.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:53:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6891.031204 secs 2023-03-09T16:53:02.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:53:02 smithi038 ceph-mon[92422]: pgmap v4189: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 274 KiB/s wr, 14 op/s 2023-03-09T16:53:02.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:53:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6893.278423 secs 2023-03-09T16:53:03.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:53:02 smithi122 ceph-mon[93869]: pgmap v4189: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 274 KiB/s wr, 14 op/s 2023-03-09T16:53:03.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:53:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6893.278423 secs 2023-03-09T16:53:03.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:53:02 smithi181 ceph-mon[94189]: pgmap v4189: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 274 KiB/s wr, 14 op/s 2023-03-09T16:53:03.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:53:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6893.278423 secs 2023-03-09T16:53:03.793 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7355.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:53:05.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:53:04 smithi122 ceph-mon[93869]: pgmap v4190: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 175 KiB/s wr, 23 op/s 2023-03-09T16:53:05.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:53:04 smithi181 ceph-mon[94189]: pgmap v4190: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 175 KiB/s wr, 23 op/s 2023-03-09T16:53:05.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:53:04 smithi038 ceph-mon[92422]: pgmap v4190: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 175 KiB/s wr, 23 op/s 2023-03-09T16:53:06.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:53:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6896.031265 secs 2023-03-09T16:53:06.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:53:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6896.031265 secs 2023-03-09T16:53:06.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:53:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6896.031265 secs 2023-03-09T16:53:07.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:53:06 smithi122 ceph-mon[93869]: pgmap v4191: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 203 KiB/s wr, 23 op/s 2023-03-09T16:53:07.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:53:06 smithi181 ceph-mon[94189]: pgmap v4191: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 203 KiB/s wr, 23 op/s 2023-03-09T16:53:07.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:53:06 smithi038 ceph-mon[92422]: pgmap v4191: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 203 KiB/s wr, 23 op/s 2023-03-09T16:53:08.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:53:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6898.278524 secs 2023-03-09T16:53:08.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:53:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6898.278524 secs 2023-03-09T16:53:08.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:53:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6898.278524 secs 2023-03-09T16:53:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7360.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:53:09.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:53:08 smithi122 ceph-mon[93869]: pgmap v4192: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 204 KiB/s wr, 26 op/s 2023-03-09T16:53:09.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:53:08 smithi181 ceph-mon[94189]: pgmap v4192: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 204 KiB/s wr, 26 op/s 2023-03-09T16:53:09.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:53:08 smithi038 ceph-mon[92422]: pgmap v4192: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 204 KiB/s wr, 26 op/s 2023-03-09T16:53:10.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:53:09 smithi122 ceph-mon[93869]: osdmap e475: 12 total, 12 up, 12 in 2023-03-09T16:53:10.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:53:09 smithi122 ceph-mon[93869]: pgmap v4194: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 89 KiB/s wr, 26 op/s 2023-03-09T16:53:10.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:53:09 smithi181 ceph-mon[94189]: osdmap e475: 12 total, 12 up, 12 in 2023-03-09T16:53:10.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:53:09 smithi181 ceph-mon[94189]: pgmap v4194: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 89 KiB/s wr, 26 op/s 2023-03-09T16:53:10.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:53:09 smithi038 ceph-mon[92422]: osdmap e475: 12 total, 12 up, 12 in 2023-03-09T16:53:10.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:53:09 smithi038 ceph-mon[92422]: pgmap v4194: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 89 KiB/s wr, 26 op/s 2023-03-09T16:53:11.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:53:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6901.031326 secs 2023-03-09T16:53:11.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:53:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6901.031326 secs 2023-03-09T16:53:11.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:53:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6901.031326 secs 2023-03-09T16:53:12.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:53:11 smithi122 ceph-mon[93869]: pgmap v4195: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 60 KiB/s wr, 23 op/s 2023-03-09T16:53:12.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:53:11 smithi181 ceph-mon[94189]: pgmap v4195: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 60 KiB/s wr, 23 op/s 2023-03-09T16:53:12.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:53:11 smithi038 ceph-mon[92422]: pgmap v4195: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 60 KiB/s wr, 23 op/s 2023-03-09T16:53:13.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:53:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6903.278638 secs 2023-03-09T16:53:13.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:53:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6903.278638 secs 2023-03-09T16:53:13.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:53:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6903.278638 secs 2023-03-09T16:53:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7365.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:53:14.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:53:13 smithi122 ceph-mon[93869]: pgmap v4196: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 35 KiB/s wr, 8 op/s 2023-03-09T16:53:14.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:53:13 smithi181 ceph-mon[94189]: pgmap v4196: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 35 KiB/s wr, 8 op/s 2023-03-09T16:53:14.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:53:13 smithi038 ceph-mon[92422]: pgmap v4196: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 35 KiB/s wr, 8 op/s 2023-03-09T16:53:15.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:53:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6906.031387 secs 2023-03-09T16:53:15.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:53:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6906.031387 secs 2023-03-09T16:53:15.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:53:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6906.031387 secs 2023-03-09T16:53:16.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:53:16 smithi122 ceph-mon[93869]: pgmap v4197: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 716 B/s wr, 7 op/s 2023-03-09T16:53:16.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:53:16 smithi181 ceph-mon[94189]: pgmap v4197: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 716 B/s wr, 7 op/s 2023-03-09T16:53:16.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:53:16 smithi038 ceph-mon[92422]: pgmap v4197: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 716 B/s wr, 7 op/s 2023-03-09T16:53:17.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:53:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6908.278749 secs 2023-03-09T16:53:17.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:53:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6908.278749 secs 2023-03-09T16:53:17.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:53:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6908.278749 secs 2023-03-09T16:53:18.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:53:18 smithi122 ceph-mon[93869]: pgmap v4198: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 307 B/s wr, 6 op/s 2023-03-09T16:53:18.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:53:18 smithi181 ceph-mon[94189]: pgmap v4198: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 307 B/s wr, 6 op/s 2023-03-09T16:53:18.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:53:18 smithi038 ceph-mon[92422]: pgmap v4198: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 307 B/s wr, 6 op/s 2023-03-09T16:53:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7370.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:53:20.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:53:20 smithi122 ceph-mon[93869]: pgmap v4199: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 285 B/s wr, 6 op/s 2023-03-09T16:53:20.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:53:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6911.031475 secs 2023-03-09T16:53:20.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:53:20 smithi122 ceph-mon[93869]: osdmap e476: 12 total, 12 up, 12 in 2023-03-09T16:53:20.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:53:20 smithi181 ceph-mon[94189]: pgmap v4199: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 285 B/s wr, 6 op/s 2023-03-09T16:53:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:53:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6911.031475 secs 2023-03-09T16:53:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:53:20 smithi181 ceph-mon[94189]: osdmap e476: 12 total, 12 up, 12 in 2023-03-09T16:53:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:53:20 smithi038 ceph-mon[92422]: pgmap v4199: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 285 B/s wr, 6 op/s 2023-03-09T16:53:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:53:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6911.031475 secs 2023-03-09T16:53:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:53:20 smithi038 ceph-mon[92422]: osdmap e476: 12 total, 12 up, 12 in 2023-03-09T16:53:22.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:53:22 smithi122 ceph-mon[93869]: pgmap v4201: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T16:53:22.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:53:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6913.278866 secs 2023-03-09T16:53:22.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:53:22 smithi181 ceph-mon[94189]: pgmap v4201: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T16:53:22.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:53:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6913.278866 secs 2023-03-09T16:53:22.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:53:22 smithi038 ceph-mon[92422]: pgmap v4201: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T16:53:22.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:53:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6913.278866 secs 2023-03-09T16:53:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7375.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:53:24.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:53:24 smithi122 ceph-mon[93869]: pgmap v4202: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T16:53:24.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:53:24 smithi181 ceph-mon[94189]: pgmap v4202: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T16:53:24.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:53:24 smithi038 ceph-mon[92422]: pgmap v4202: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T16:53:25.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:53:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6916.031535 secs 2023-03-09T16:53:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:53:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6916.031535 secs 2023-03-09T16:53:25.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:53:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6916.031535 secs 2023-03-09T16:53:26.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:53:26 smithi122 ceph-mon[93869]: pgmap v4203: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2023-03-09T16:53:26.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:53:26 smithi181 ceph-mon[94189]: pgmap v4203: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2023-03-09T16:53:26.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:53:26 smithi038 ceph-mon[92422]: pgmap v4203: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2023-03-09T16:53:27.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:53:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6918.279004 secs 2023-03-09T16:53:27.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:53:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6918.279004 secs 2023-03-09T16:53:27.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:53:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6918.279004 secs 2023-03-09T16:53:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7380.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:53:28.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:53:28 smithi122 ceph-mon[93869]: pgmap v4204: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2023-03-09T16:53:28.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:53:28 smithi181 ceph-mon[94189]: pgmap v4204: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2023-03-09T16:53:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:53:28 smithi038 ceph-mon[92422]: pgmap v4204: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2023-03-09T16:53:30.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:53:30 smithi122 ceph-mon[93869]: pgmap v4205: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 40 KiB/s wr, 6 op/s 2023-03-09T16:53:30.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:53:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6921.031597 secs 2023-03-09T16:53:30.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:53:30 smithi181 ceph-mon[94189]: pgmap v4205: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 40 KiB/s wr, 6 op/s 2023-03-09T16:53:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:53:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6921.031597 secs 2023-03-09T16:53:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:53:30 smithi038 ceph-mon[92422]: pgmap v4205: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 40 KiB/s wr, 6 op/s 2023-03-09T16:53:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:53:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6921.031597 secs 2023-03-09T16:53:32.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:53:32 smithi122 ceph-mon[93869]: pgmap v4206: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 35 KiB/s wr, 7 op/s 2023-03-09T16:53:32.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:53:32 smithi181 ceph-mon[94189]: pgmap v4206: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 35 KiB/s wr, 7 op/s 2023-03-09T16:53:32.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:53:32 smithi038 ceph-mon[92422]: pgmap v4206: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 35 KiB/s wr, 7 op/s 2023-03-09T16:53:33.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7385.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:53:34.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:53:34 smithi122 ceph-mon[93869]: pgmap v4207: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 34 KiB/s wr, 6 op/s 2023-03-09T16:53:34.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:53:34 smithi181 ceph-mon[94189]: pgmap v4207: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 34 KiB/s wr, 6 op/s 2023-03-09T16:53:34.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:53:34 smithi038 ceph-mon[92422]: pgmap v4207: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 34 KiB/s wr, 6 op/s 2023-03-09T16:53:35.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:53:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6926.031652 secs 2023-03-09T16:53:35.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:53:35 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:53:35.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:53:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6926.031652 secs 2023-03-09T16:53:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:53:35 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:53:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:53:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6926.031652 secs 2023-03-09T16:53:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:53:35 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:53:36.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:53:36 smithi122 ceph-mon[93869]: pgmap v4208: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 161 KiB/s wr, 7 op/s 2023-03-09T16:53:36.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:53:36 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:53:36.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:53:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:53:36.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:53:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:53:36.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:53:36 smithi181 ceph-mon[94189]: pgmap v4208: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 161 KiB/s wr, 7 op/s 2023-03-09T16:53:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:53:36 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:53:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:53:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:53:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:53:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:53:36.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:53:36 smithi038 ceph-mon[92422]: pgmap v4208: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 161 KiB/s wr, 7 op/s 2023-03-09T16:53:36.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:53:36 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:53:36.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:53:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:53:36.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:53:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:53:37.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:53:37 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 6928.279245 secs 2023-03-09T16:53:37.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:53:37 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 6928.279245 secs 2023-03-09T16:53:37.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:53:37 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 6928.279245 secs 2023-03-09T16:53:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7390.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:53:38.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:53:38 smithi122 ceph-mon[93869]: pgmap v4209: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 169 KiB/s wr, 10 op/s 2023-03-09T16:53:38.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:53:38 smithi181 ceph-mon[94189]: pgmap v4209: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 169 KiB/s wr, 10 op/s 2023-03-09T16:53:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:53:38 smithi038 ceph-mon[92422]: pgmap v4209: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 169 KiB/s wr, 10 op/s 2023-03-09T16:53:40.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:53:40 smithi122 ceph-mon[93869]: pgmap v4210: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 273 KiB/s wr, 10 op/s 2023-03-09T16:53:40.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:53:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6931.031714 secs 2023-03-09T16:53:40.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:53:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:53:40.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:53:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:53:40.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:53:40 smithi181 ceph-mon[94189]: pgmap v4210: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 273 KiB/s wr, 10 op/s 2023-03-09T16:53:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:53:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6931.031714 secs 2023-03-09T16:53:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:53:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:53:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:53:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:53:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:53:40 smithi038 ceph-mon[92422]: pgmap v4210: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 273 KiB/s wr, 10 op/s 2023-03-09T16:53:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:53:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6931.031714 secs 2023-03-09T16:53:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:53:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:53:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:53:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:53:42.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:53:42 smithi122 ceph-mon[93869]: pgmap v4211: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 255 KiB/s wr, 12 op/s 2023-03-09T16:53:42.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:53:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6933.279343 secs 2023-03-09T16:53:42.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:53:42 smithi181 ceph-mon[94189]: pgmap v4211: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 255 KiB/s wr, 12 op/s 2023-03-09T16:53:42.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:53:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6933.279343 secs 2023-03-09T16:53:42.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:53:42 smithi038 ceph-mon[92422]: pgmap v4211: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 255 KiB/s wr, 12 op/s 2023-03-09T16:53:42.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:53:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6933.279343 secs 2023-03-09T16:53:43.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7395.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:53:44.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:53:44 smithi122 ceph-mon[93869]: pgmap v4212: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 305 KiB/s wr, 12 op/s 2023-03-09T16:53:44.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:53:44 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:53:44.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:53:44 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:53:44.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:53:44 smithi181 ceph-mon[94189]: pgmap v4212: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 305 KiB/s wr, 12 op/s 2023-03-09T16:53:44.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:53:44 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:53:44.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:53:44 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:53:44.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:53:44 smithi038 ceph-mon[92422]: pgmap v4212: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 305 KiB/s wr, 12 op/s 2023-03-09T16:53:44.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:53:44 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:53:44.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:53:44 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:53:45.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:53:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6936.031773 secs 2023-03-09T16:53:45.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:53:45 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:53:45.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:53:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6936.031773 secs 2023-03-09T16:53:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:53:45 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:53:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:53:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6936.031773 secs 2023-03-09T16:53:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:53:45 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:53:46.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:53:46 smithi122 ceph-mon[93869]: pgmap v4213: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 305 KiB/s wr, 11 op/s 2023-03-09T16:53:46.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:53:46 smithi181 ceph-mon[94189]: pgmap v4213: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 305 KiB/s wr, 11 op/s 2023-03-09T16:53:46.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:53:46 smithi038 ceph-mon[92422]: pgmap v4213: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 305 KiB/s wr, 11 op/s 2023-03-09T16:53:47.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:53:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6938.279420 secs 2023-03-09T16:53:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:53:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6938.279420 secs 2023-03-09T16:53:47.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:53:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6938.279420 secs 2023-03-09T16:53:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7400.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:53:48.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:53:48 smithi122 ceph-mon[93869]: pgmap v4214: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 183 KiB/s wr, 13 op/s 2023-03-09T16:53:48.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:53:48 smithi181 ceph-mon[94189]: pgmap v4214: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 183 KiB/s wr, 13 op/s 2023-03-09T16:53:48.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:53:48 smithi038 ceph-mon[92422]: pgmap v4214: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 183 KiB/s wr, 13 op/s 2023-03-09T16:53:50.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:53:50 smithi122 ceph-mon[93869]: pgmap v4215: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 234 KiB/s wr, 11 op/s 2023-03-09T16:53:50.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:53:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6941.031835 secs 2023-03-09T16:53:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:53:50 smithi181 ceph-mon[94189]: pgmap v4215: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 234 KiB/s wr, 11 op/s 2023-03-09T16:53:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:53:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6941.031835 secs 2023-03-09T16:53:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:53:50 smithi038 ceph-mon[92422]: pgmap v4215: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 234 KiB/s wr, 11 op/s 2023-03-09T16:53:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:53:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6941.031835 secs 2023-03-09T16:53:52.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:53:52 smithi122 ceph-mon[93869]: pgmap v4216: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 129 KiB/s wr, 12 op/s 2023-03-09T16:53:52.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:53:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6943.279542 secs 2023-03-09T16:53:52.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:53:52 smithi181 ceph-mon[94189]: pgmap v4216: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 129 KiB/s wr, 12 op/s 2023-03-09T16:53:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:53:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6943.279542 secs 2023-03-09T16:53:52.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:53:52 smithi038 ceph-mon[92422]: pgmap v4216: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 129 KiB/s wr, 12 op/s 2023-03-09T16:53:52.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:53:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6943.279542 secs 2023-03-09T16:53:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7405.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:53:54.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:53:54 smithi122 ceph-mon[93869]: pgmap v4217: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 148 KiB/s wr, 9 op/s 2023-03-09T16:53:54.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:53:54 smithi181 ceph-mon[94189]: pgmap v4217: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 148 KiB/s wr, 9 op/s 2023-03-09T16:53:54.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:53:54 smithi038 ceph-mon[92422]: pgmap v4217: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 148 KiB/s wr, 9 op/s 2023-03-09T16:53:55.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:53:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6946.031893 secs 2023-03-09T16:53:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:53:55 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:53:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:53:55 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:53:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:53:55 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:53:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:53:55 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:53:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:53:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6946.031893 secs 2023-03-09T16:53:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:53:55 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:53:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:53:55 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:53:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:53:55 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:53:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:53:55 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:53:56.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:53:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6946.031893 secs 2023-03-09T16:53:56.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:53:55 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:53:56.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:53:55 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:53:56.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:53:55 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:53:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:53:55 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:53:56.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:53:56 smithi038 ceph-mon[92422]: pgmap v4218: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 97 KiB/s wr, 8 op/s 2023-03-09T16:53:57.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:53:56 smithi122 ceph-mon[93869]: pgmap v4218: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 97 KiB/s wr, 8 op/s 2023-03-09T16:53:57.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:53:56 smithi181 ceph-mon[94189]: pgmap v4218: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 97 KiB/s wr, 8 op/s 2023-03-09T16:53:57.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:53:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6948.279666 secs 2023-03-09T16:53:58.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:53:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6948.279666 secs 2023-03-09T16:53:58.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:53:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6948.279666 secs 2023-03-09T16:53:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7410.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:53:58.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:53:58 smithi038 ceph-mon[92422]: pgmap v4219: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 97 KiB/s wr, 10 op/s 2023-03-09T16:53:59.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:53:58 smithi122 ceph-mon[93869]: pgmap v4219: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 97 KiB/s wr, 10 op/s 2023-03-09T16:53:59.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:53:58 smithi181 ceph-mon[94189]: pgmap v4219: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 97 KiB/s wr, 10 op/s 2023-03-09T16:54:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:54:00 smithi038 ceph-mon[92422]: pgmap v4220: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 119 KiB/s wr, 7 op/s 2023-03-09T16:54:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:54:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6951.031955 secs 2023-03-09T16:54:01.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:54:00 smithi122 ceph-mon[93869]: pgmap v4220: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 119 KiB/s wr, 7 op/s 2023-03-09T16:54:01.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:54:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6951.031955 secs 2023-03-09T16:54:01.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:54:00 smithi181 ceph-mon[94189]: pgmap v4220: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 119 KiB/s wr, 7 op/s 2023-03-09T16:54:01.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:54:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6951.031955 secs 2023-03-09T16:54:02.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:54:02 smithi038 ceph-mon[92422]: pgmap v4221: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 83 KiB/s wr, 11 op/s 2023-03-09T16:54:02.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:54:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6953.279783 secs 2023-03-09T16:54:03.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:54:02 smithi122 ceph-mon[93869]: pgmap v4221: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 83 KiB/s wr, 11 op/s 2023-03-09T16:54:03.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:54:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6953.279783 secs 2023-03-09T16:54:03.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:54:02 smithi181 ceph-mon[94189]: pgmap v4221: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 83 KiB/s wr, 11 op/s 2023-03-09T16:54:03.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:54:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6953.279783 secs 2023-03-09T16:54:03.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7415.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:54:05.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:54:04 smithi122 ceph-mon[93869]: pgmap v4222: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 148 KiB/s wr, 21 op/s 2023-03-09T16:54:05.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:54:04 smithi122 ceph-mon[93869]: osdmap e477: 12 total, 12 up, 12 in 2023-03-09T16:54:05.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:54:04 smithi181 ceph-mon[94189]: pgmap v4222: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 148 KiB/s wr, 21 op/s 2023-03-09T16:54:05.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:54:04 smithi181 ceph-mon[94189]: osdmap e477: 12 total, 12 up, 12 in 2023-03-09T16:54:05.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:54:04 smithi038 ceph-mon[92422]: pgmap v4222: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 148 KiB/s wr, 21 op/s 2023-03-09T16:54:05.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:54:04 smithi038 ceph-mon[92422]: osdmap e477: 12 total, 12 up, 12 in 2023-03-09T16:54:06.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:54:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6956.031983 secs 2023-03-09T16:54:06.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:54:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6956.031983 secs 2023-03-09T16:54:06.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:54:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6956.031983 secs 2023-03-09T16:54:07.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:54:06 smithi122 ceph-mon[93869]: pgmap v4224: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 138 KiB/s wr, 25 op/s 2023-03-09T16:54:07.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:54:06 smithi181 ceph-mon[94189]: pgmap v4224: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 138 KiB/s wr, 25 op/s 2023-03-09T16:54:07.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:54:06 smithi038 ceph-mon[92422]: pgmap v4224: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 138 KiB/s wr, 25 op/s 2023-03-09T16:54:08.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:54:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6958.279896 secs 2023-03-09T16:54:08.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:54:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6958.279896 secs 2023-03-09T16:54:08.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:54:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6958.279896 secs 2023-03-09T16:54:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7420.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:54:09.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:54:08 smithi122 ceph-mon[93869]: pgmap v4225: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 138 KiB/s wr, 24 op/s 2023-03-09T16:54:09.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:54:08 smithi181 ceph-mon[94189]: pgmap v4225: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 138 KiB/s wr, 24 op/s 2023-03-09T16:54:09.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:54:08 smithi038 ceph-mon[92422]: pgmap v4225: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 138 KiB/s wr, 24 op/s 2023-03-09T16:54:10.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:54:09 smithi122 ceph-mon[93869]: pgmap v4226: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 105 KiB/s wr, 24 op/s 2023-03-09T16:54:10.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:54:09 smithi181 ceph-mon[94189]: pgmap v4226: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 105 KiB/s wr, 24 op/s 2023-03-09T16:54:10.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:54:09 smithi038 ceph-mon[92422]: pgmap v4226: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 105 KiB/s wr, 24 op/s 2023-03-09T16:54:11.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:54:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6961.032044 secs 2023-03-09T16:54:11.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:54:10 smithi122 ceph-mon[93869]: osdmap e478: 12 total, 12 up, 12 in 2023-03-09T16:54:11.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:54:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6961.032044 secs 2023-03-09T16:54:11.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:54:10 smithi181 ceph-mon[94189]: osdmap e478: 12 total, 12 up, 12 in 2023-03-09T16:54:11.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:54:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6961.032044 secs 2023-03-09T16:54:11.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:54:10 smithi038 ceph-mon[92422]: osdmap e478: 12 total, 12 up, 12 in 2023-03-09T16:54:12.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:54:11 smithi122 ceph-mon[93869]: pgmap v4228: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 383 B/s wr, 7 op/s 2023-03-09T16:54:12.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:54:11 smithi181 ceph-mon[94189]: pgmap v4228: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 383 B/s wr, 7 op/s 2023-03-09T16:54:12.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:54:11 smithi038 ceph-mon[92422]: pgmap v4228: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 383 B/s wr, 7 op/s 2023-03-09T16:54:13.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:54:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6963.280033 secs 2023-03-09T16:54:13.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:54:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6963.280033 secs 2023-03-09T16:54:13.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:54:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6963.280033 secs 2023-03-09T16:54:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7425.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:54:14.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:54:13 smithi122 ceph-mon[93869]: pgmap v4229: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 419 B/s wr, 7 op/s 2023-03-09T16:54:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:54:13 smithi181 ceph-mon[94189]: pgmap v4229: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 419 B/s wr, 7 op/s 2023-03-09T16:54:14.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:54:13 smithi038 ceph-mon[92422]: pgmap v4229: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 419 B/s wr, 7 op/s 2023-03-09T16:54:15.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:54:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6966.032104 secs 2023-03-09T16:54:15.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:54:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6966.032104 secs 2023-03-09T16:54:15.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:54:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6966.032104 secs 2023-03-09T16:54:16.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:54:16 smithi122 ceph-mon[93869]: pgmap v4230: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 409 B/s wr, 6 op/s 2023-03-09T16:54:16.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:54:16 smithi181 ceph-mon[94189]: pgmap v4230: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 409 B/s wr, 6 op/s 2023-03-09T16:54:16.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:54:16 smithi038 ceph-mon[92422]: pgmap v4230: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 409 B/s wr, 6 op/s 2023-03-09T16:54:17.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:54:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6968.280160 secs 2023-03-09T16:54:17.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:54:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6968.280160 secs 2023-03-09T16:54:17.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:54:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6968.280160 secs 2023-03-09T16:54:18.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:54:18 smithi122 ceph-mon[93869]: pgmap v4231: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 921 B/s wr, 7 op/s 2023-03-09T16:54:18.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:54:18 smithi181 ceph-mon[94189]: pgmap v4231: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 921 B/s wr, 7 op/s 2023-03-09T16:54:18.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:54:18 smithi038 ceph-mon[92422]: pgmap v4231: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 921 B/s wr, 7 op/s 2023-03-09T16:54:18.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7430.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:54:20.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:54:20 smithi122 ceph-mon[93869]: pgmap v4232: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 52 KiB/s wr, 7 op/s 2023-03-09T16:54:20.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:54:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6971.032165 secs 2023-03-09T16:54:20.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:54:20 smithi181 ceph-mon[94189]: pgmap v4232: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 52 KiB/s wr, 7 op/s 2023-03-09T16:54:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:54:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6971.032165 secs 2023-03-09T16:54:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:54:20 smithi038 ceph-mon[92422]: pgmap v4232: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 52 KiB/s wr, 7 op/s 2023-03-09T16:54:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:54:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6971.032165 secs 2023-03-09T16:54:22.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:54:22 smithi122 ceph-mon[93869]: pgmap v4233: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 45 KiB/s wr, 8 op/s 2023-03-09T16:54:22.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:54:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6973.280273 secs 2023-03-09T16:54:22.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:54:22 smithi181 ceph-mon[94189]: pgmap v4233: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 45 KiB/s wr, 8 op/s 2023-03-09T16:54:22.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:54:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6973.280273 secs 2023-03-09T16:54:22.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:54:22 smithi038 ceph-mon[92422]: pgmap v4233: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 45 KiB/s wr, 8 op/s 2023-03-09T16:54:22.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:54:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6973.280273 secs 2023-03-09T16:54:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7435.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:54:24.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:54:24 smithi122 ceph-mon[93869]: pgmap v4234: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T16:54:24.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:54:24 smithi181 ceph-mon[94189]: pgmap v4234: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T16:54:24.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:54:24 smithi038 ceph-mon[92422]: pgmap v4234: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T16:54:25.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:54:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6976.032244 secs 2023-03-09T16:54:25.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:54:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6976.032244 secs 2023-03-09T16:54:25.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:54:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6976.032244 secs 2023-03-09T16:54:26.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:54:26 smithi122 ceph-mon[93869]: pgmap v4235: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 43 KiB/s wr, 6 op/s 2023-03-09T16:54:26.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:54:26 smithi181 ceph-mon[94189]: pgmap v4235: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 43 KiB/s wr, 6 op/s 2023-03-09T16:54:26.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:54:26 smithi038 ceph-mon[92422]: pgmap v4235: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 43 KiB/s wr, 6 op/s 2023-03-09T16:54:27.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:54:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6978.280372 secs 2023-03-09T16:54:27.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:54:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6978.280372 secs 2023-03-09T16:54:27.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:54:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6978.280372 secs 2023-03-09T16:54:28.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7440.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:54:28.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:54:28 smithi122 ceph-mon[93869]: pgmap v4236: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 44 KiB/s wr, 8 op/s 2023-03-09T16:54:28.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:54:28 smithi181 ceph-mon[94189]: pgmap v4236: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 44 KiB/s wr, 8 op/s 2023-03-09T16:54:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:54:28 smithi038 ceph-mon[92422]: pgmap v4236: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 44 KiB/s wr, 8 op/s 2023-03-09T16:54:30.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:54:30 smithi122 ceph-mon[93869]: pgmap v4237: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 43 KiB/s wr, 6 op/s 2023-03-09T16:54:30.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:54:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6981.032306 secs 2023-03-09T16:54:30.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:54:30 smithi181 ceph-mon[94189]: pgmap v4237: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 43 KiB/s wr, 6 op/s 2023-03-09T16:54:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:54:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6981.032306 secs 2023-03-09T16:54:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:54:30 smithi038 ceph-mon[92422]: pgmap v4237: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 43 KiB/s wr, 6 op/s 2023-03-09T16:54:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:54:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6981.032306 secs 2023-03-09T16:54:32.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:54:32 smithi122 ceph-mon[93869]: pgmap v4238: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 7 op/s 2023-03-09T16:54:32.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:54:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6983.280457 secs 2023-03-09T16:54:32.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:54:32 smithi181 ceph-mon[94189]: pgmap v4238: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 7 op/s 2023-03-09T16:54:32.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:54:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6983.280457 secs 2023-03-09T16:54:32.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:54:32 smithi038 ceph-mon[92422]: pgmap v4238: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 7 op/s 2023-03-09T16:54:32.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:54:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6983.280457 secs 2023-03-09T16:54:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7445.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:54:34.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:54:34 smithi122 ceph-mon[93869]: pgmap v4239: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 42 KiB/s wr, 6 op/s 2023-03-09T16:54:34.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:54:34 smithi181 ceph-mon[94189]: pgmap v4239: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 42 KiB/s wr, 6 op/s 2023-03-09T16:54:34.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:54:34 smithi038 ceph-mon[92422]: pgmap v4239: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 42 KiB/s wr, 6 op/s 2023-03-09T16:54:35.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:54:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6986.032366 secs 2023-03-09T16:54:35.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:54:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6986.032366 secs 2023-03-09T16:54:35.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:54:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6986.032366 secs 2023-03-09T16:54:36.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:54:36 smithi122 ceph-mon[93869]: pgmap v4240: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 42 KiB/s wr, 5 op/s 2023-03-09T16:54:36.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:54:36 smithi181 ceph-mon[94189]: pgmap v4240: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 42 KiB/s wr, 5 op/s 2023-03-09T16:54:36.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:54:36 smithi038 ceph-mon[92422]: pgmap v4240: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 42 KiB/s wr, 5 op/s 2023-03-09T16:54:37.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:54:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6988.280534 secs 2023-03-09T16:54:37.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:54:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6988.280534 secs 2023-03-09T16:54:37.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:54:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6988.280534 secs 2023-03-09T16:54:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7450.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:54:38.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:54:38 smithi122 ceph-mon[93869]: pgmap v4241: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 45 KiB/s wr, 8 op/s 2023-03-09T16:54:38.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:54:38 smithi181 ceph-mon[94189]: pgmap v4241: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 45 KiB/s wr, 8 op/s 2023-03-09T16:54:38.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:54:38 smithi038 ceph-mon[92422]: pgmap v4241: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 45 KiB/s wr, 8 op/s 2023-03-09T16:54:40.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:54:40 smithi122 ceph-mon[93869]: pgmap v4242: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 45 KiB/s wr, 8 op/s 2023-03-09T16:54:40.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:54:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6991.032427 secs 2023-03-09T16:54:40.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:54:40 smithi181 ceph-mon[94189]: pgmap v4242: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 45 KiB/s wr, 8 op/s 2023-03-09T16:54:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:54:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6991.032427 secs 2023-03-09T16:54:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:54:40 smithi038 ceph-mon[92422]: pgmap v4242: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 45 KiB/s wr, 8 op/s 2023-03-09T16:54:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:54:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6991.032427 secs 2023-03-09T16:54:42.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:54:42 smithi122 ceph-mon[93869]: pgmap v4243: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 103 KiB/s wr, 28 op/s 2023-03-09T16:54:42.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:54:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6993.280579 secs 2023-03-09T16:54:42.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:54:42 smithi181 ceph-mon[94189]: pgmap v4243: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 103 KiB/s wr, 28 op/s 2023-03-09T16:54:42.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:54:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6993.280579 secs 2023-03-09T16:54:42.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:54:42 smithi038 ceph-mon[92422]: pgmap v4243: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 103 KiB/s wr, 28 op/s 2023-03-09T16:54:42.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:54:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6993.280579 secs 2023-03-09T16:54:43.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7455.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:54:44.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:54:44 smithi122 ceph-mon[93869]: pgmap v4244: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 108 KiB/s wr, 31 op/s 2023-03-09T16:54:44.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:54:44 smithi181 ceph-mon[94189]: pgmap v4244: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 108 KiB/s wr, 31 op/s 2023-03-09T16:54:44.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:54:44 smithi038 ceph-mon[92422]: pgmap v4244: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 108 KiB/s wr, 31 op/s 2023-03-09T16:54:45.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:54:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6996.032487 secs 2023-03-09T16:54:45.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:54:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6996.032487 secs 2023-03-09T16:54:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:54:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6996.032487 secs 2023-03-09T16:54:46.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:54:46 smithi122 ceph-mon[93869]: pgmap v4245: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 66 KiB/s wr, 62 op/s 2023-03-09T16:54:46.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:54:46 smithi181 ceph-mon[94189]: pgmap v4245: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 66 KiB/s wr, 62 op/s 2023-03-09T16:54:46.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:54:46 smithi038 ceph-mon[92422]: pgmap v4245: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 66 KiB/s wr, 62 op/s 2023-03-09T16:54:47.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:54:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 6998.280696 secs 2023-03-09T16:54:47.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:54:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 6998.280696 secs 2023-03-09T16:54:47.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:54:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 6998.280696 secs 2023-03-09T16:54:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7460.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:54:48.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:54:48 smithi122 ceph-mon[93869]: pgmap v4246: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 68 KiB/s wr, 66 op/s 2023-03-09T16:54:48.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:54:48 smithi181 ceph-mon[94189]: pgmap v4246: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 68 KiB/s wr, 66 op/s 2023-03-09T16:54:48.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:54:48 smithi038 ceph-mon[92422]: pgmap v4246: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 68 KiB/s wr, 66 op/s 2023-03-09T16:54:50.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:54:50 smithi122 ceph-mon[93869]: pgmap v4247: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 65 KiB/s wr, 64 op/s 2023-03-09T16:54:50.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:54:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7001.032572 secs 2023-03-09T16:54:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:54:50 smithi181 ceph-mon[94189]: pgmap v4247: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 65 KiB/s wr, 64 op/s 2023-03-09T16:54:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:54:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7001.032572 secs 2023-03-09T16:54:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:54:50 smithi038 ceph-mon[92422]: pgmap v4247: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 65 KiB/s wr, 64 op/s 2023-03-09T16:54:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:54:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7001.032572 secs 2023-03-09T16:54:52.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:54:52 smithi122 ceph-mon[93869]: pgmap v4248: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 110 KiB/s wr, 65 op/s 2023-03-09T16:54:52.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:54:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7003.280835 secs 2023-03-09T16:54:52.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:54:52 smithi181 ceph-mon[94189]: pgmap v4248: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 110 KiB/s wr, 65 op/s 2023-03-09T16:54:52.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:54:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7003.280835 secs 2023-03-09T16:54:52.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:54:52 smithi038 ceph-mon[92422]: pgmap v4248: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 110 KiB/s wr, 65 op/s 2023-03-09T16:54:52.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:54:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7003.280835 secs 2023-03-09T16:54:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7465.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:54:54.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:54:54 smithi181 ceph-mon[94189]: pgmap v4249: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 53 KiB/s wr, 46 op/s 2023-03-09T16:54:54.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:54:54 smithi038 ceph-mon[92422]: pgmap v4249: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 53 KiB/s wr, 46 op/s 2023-03-09T16:54:55.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:54:54 smithi122 ceph-mon[93869]: pgmap v4249: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 53 KiB/s wr, 46 op/s 2023-03-09T16:54:55.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:54:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7006.032646 secs 2023-03-09T16:54:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:54:55 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:54:55.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:54:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7006.032646 secs 2023-03-09T16:54:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:54:55 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:54:56.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:54:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7006.032646 secs 2023-03-09T16:54:56.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:54:55 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:54:56.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:54:56 smithi038 ceph-mon[92422]: pgmap v4250: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 48 KiB/s wr, 41 op/s 2023-03-09T16:54:56.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:54:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:54:56.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:54:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:54:56.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:54:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:54:57.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:54:56 smithi122 ceph-mon[93869]: pgmap v4250: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 48 KiB/s wr, 41 op/s 2023-03-09T16:54:57.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:54:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:54:57.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:54:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:54:57.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:54:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:54:57.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:54:56 smithi181 ceph-mon[94189]: pgmap v4250: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 48 KiB/s wr, 41 op/s 2023-03-09T16:54:57.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:54:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:54:57.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:54:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:54:57.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:54:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:54:57.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:54:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7008.280951 secs 2023-03-09T16:54:58.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:54:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7008.280951 secs 2023-03-09T16:54:58.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:54:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7008.280951 secs 2023-03-09T16:54:58.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7470.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:54:58.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:54:58 smithi038 ceph-mon[92422]: pgmap v4251: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 56 KiB/s wr, 12 op/s 2023-03-09T16:54:59.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:54:58 smithi122 ceph-mon[93869]: pgmap v4251: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 56 KiB/s wr, 12 op/s 2023-03-09T16:54:59.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:54:58 smithi181 ceph-mon[94189]: pgmap v4251: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 56 KiB/s wr, 12 op/s 2023-03-09T16:55:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:55:00 smithi038 ceph-mon[92422]: pgmap v4252: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 54 KiB/s wr, 8 op/s 2023-03-09T16:55:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:55:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7011.032706 secs 2023-03-09T16:55:01.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:55:00 smithi122 ceph-mon[93869]: pgmap v4252: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 54 KiB/s wr, 8 op/s 2023-03-09T16:55:01.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:55:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7011.032706 secs 2023-03-09T16:55:01.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:55:00 smithi181 ceph-mon[94189]: pgmap v4252: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 54 KiB/s wr, 8 op/s 2023-03-09T16:55:01.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:55:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7011.032706 secs 2023-03-09T16:55:02.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:55:02 smithi038 ceph-mon[92422]: pgmap v4253: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 101 KiB/s wr, 14 op/s 2023-03-09T16:55:02.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:55:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7013.281063 secs 2023-03-09T16:55:03.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:55:02 smithi122 ceph-mon[93869]: pgmap v4253: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 101 KiB/s wr, 14 op/s 2023-03-09T16:55:03.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:55:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7013.281063 secs 2023-03-09T16:55:03.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:55:02 smithi181 ceph-mon[94189]: pgmap v4253: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 101 KiB/s wr, 14 op/s 2023-03-09T16:55:03.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:55:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7013.281063 secs 2023-03-09T16:55:03.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7475.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:55:04.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:55:04 smithi038 ceph-mon[92422]: pgmap v4254: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 76 KiB/s wr, 22 op/s 2023-03-09T16:55:05.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:55:04 smithi122 ceph-mon[93869]: pgmap v4254: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 76 KiB/s wr, 22 op/s 2023-03-09T16:55:05.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:55:04 smithi181 ceph-mon[94189]: pgmap v4254: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 76 KiB/s wr, 22 op/s 2023-03-09T16:55:06.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:55:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7016.032766 secs 2023-03-09T16:55:06.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:55:05 smithi122 ceph-mon[93869]: osdmap e479: 12 total, 12 up, 12 in 2023-03-09T16:55:06.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:55:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7016.032766 secs 2023-03-09T16:55:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:55:05 smithi181 ceph-mon[94189]: osdmap e479: 12 total, 12 up, 12 in 2023-03-09T16:55:06.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:55:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7016.032766 secs 2023-03-09T16:55:06.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:55:05 smithi038 ceph-mon[92422]: osdmap e479: 12 total, 12 up, 12 in 2023-03-09T16:55:07.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:55:06 smithi122 ceph-mon[93869]: pgmap v4256: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 91 KiB/s wr, 26 op/s 2023-03-09T16:55:07.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:55:06 smithi181 ceph-mon[94189]: pgmap v4256: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 91 KiB/s wr, 26 op/s 2023-03-09T16:55:07.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:55:06 smithi038 ceph-mon[92422]: pgmap v4256: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 91 KiB/s wr, 26 op/s 2023-03-09T16:55:08.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:55:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7018.281195 secs 2023-03-09T16:55:08.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:55:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7018.281195 secs 2023-03-09T16:55:08.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:55:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7018.281195 secs 2023-03-09T16:55:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7480.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:55:09.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:55:08 smithi122 ceph-mon[93869]: pgmap v4257: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 120 KiB/s wr, 27 op/s 2023-03-09T16:55:09.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:55:08 smithi181 ceph-mon[94189]: pgmap v4257: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 120 KiB/s wr, 27 op/s 2023-03-09T16:55:09.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:55:08 smithi038 ceph-mon[92422]: pgmap v4257: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 120 KiB/s wr, 27 op/s 2023-03-09T16:55:11.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:55:10 smithi122 ceph-mon[93869]: pgmap v4258: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 120 KiB/s wr, 27 op/s 2023-03-09T16:55:11.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:55:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7021.032827 secs 2023-03-09T16:55:11.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:55:10 smithi122 ceph-mon[93869]: osdmap e480: 12 total, 12 up, 12 in 2023-03-09T16:55:11.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:55:10 smithi181 ceph-mon[94189]: pgmap v4258: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 120 KiB/s wr, 27 op/s 2023-03-09T16:55:11.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:55:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7021.032827 secs 2023-03-09T16:55:11.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:55:10 smithi181 ceph-mon[94189]: osdmap e480: 12 total, 12 up, 12 in 2023-03-09T16:55:11.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:55:10 smithi038 ceph-mon[92422]: pgmap v4258: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 120 KiB/s wr, 27 op/s 2023-03-09T16:55:11.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:55:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7021.032827 secs 2023-03-09T16:55:11.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:55:10 smithi038 ceph-mon[92422]: osdmap e480: 12 total, 12 up, 12 in 2023-03-09T16:55:12.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:55:11 smithi122 ceph-mon[93869]: pgmap v4260: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 60 KiB/s wr, 11 op/s 2023-03-09T16:55:12.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:55:11 smithi181 ceph-mon[94189]: pgmap v4260: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 60 KiB/s wr, 11 op/s 2023-03-09T16:55:12.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:55:11 smithi038 ceph-mon[92422]: pgmap v4260: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 60 KiB/s wr, 11 op/s 2023-03-09T16:55:13.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:55:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7023.281319 secs 2023-03-09T16:55:13.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:55:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7023.281319 secs 2023-03-09T16:55:13.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:55:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7023.281319 secs 2023-03-09T16:55:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7485.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:55:14.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:55:13 smithi122 ceph-mon[93869]: pgmap v4261: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 55 KiB/s wr, 10 op/s 2023-03-09T16:55:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:55:13 smithi181 ceph-mon[94189]: pgmap v4261: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 55 KiB/s wr, 10 op/s 2023-03-09T16:55:14.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:55:13 smithi038 ceph-mon[92422]: pgmap v4261: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 55 KiB/s wr, 10 op/s 2023-03-09T16:55:15.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:55:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7026.032886 secs 2023-03-09T16:55:15.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:55:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7026.032886 secs 2023-03-09T16:55:15.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:55:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7026.032886 secs 2023-03-09T16:55:16.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:55:16 smithi122 ceph-mon[93869]: pgmap v4262: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 48 KiB/s wr, 9 op/s 2023-03-09T16:55:16.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:55:16 smithi181 ceph-mon[94189]: pgmap v4262: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 48 KiB/s wr, 9 op/s 2023-03-09T16:55:16.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:55:16 smithi038 ceph-mon[92422]: pgmap v4262: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 48 KiB/s wr, 9 op/s 2023-03-09T16:55:17.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:55:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7028.281450 secs 2023-03-09T16:55:17.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:55:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7028.281450 secs 2023-03-09T16:55:17.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:55:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7028.281450 secs 2023-03-09T16:55:18.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:55:18 smithi122 ceph-mon[93869]: pgmap v4263: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 7 op/s 2023-03-09T16:55:18.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:55:18 smithi181 ceph-mon[94189]: pgmap v4263: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 7 op/s 2023-03-09T16:55:18.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:55:18 smithi038 ceph-mon[92422]: pgmap v4263: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 7 op/s 2023-03-09T16:55:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7490.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:55:20.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:55:20 smithi122 ceph-mon[93869]: pgmap v4264: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 7 op/s 2023-03-09T16:55:20.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:55:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7031.032948 secs 2023-03-09T16:55:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:55:20 smithi181 ceph-mon[94189]: pgmap v4264: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 7 op/s 2023-03-09T16:55:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:55:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7031.032948 secs 2023-03-09T16:55:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:55:20 smithi038 ceph-mon[92422]: pgmap v4264: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 7 op/s 2023-03-09T16:55:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:55:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7031.032948 secs 2023-03-09T16:55:22.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:55:22 smithi122 ceph-mon[93869]: pgmap v4265: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 9 op/s 2023-03-09T16:55:22.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:55:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7033.281587 secs 2023-03-09T16:55:22.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:55:22 smithi181 ceph-mon[94189]: pgmap v4265: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 9 op/s 2023-03-09T16:55:22.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:55:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7033.281587 secs 2023-03-09T16:55:22.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:55:22 smithi038 ceph-mon[92422]: pgmap v4265: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 9 op/s 2023-03-09T16:55:22.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:55:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7033.281587 secs 2023-03-09T16:55:23.793 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7495.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:55:24.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:55:24 smithi122 ceph-mon[93869]: pgmap v4266: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 6 op/s 2023-03-09T16:55:24.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:55:24 smithi181 ceph-mon[94189]: pgmap v4266: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 6 op/s 2023-03-09T16:55:24.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:55:24 smithi038 ceph-mon[92422]: pgmap v4266: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 6 op/s 2023-03-09T16:55:25.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:55:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7036.032981 secs 2023-03-09T16:55:25.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:55:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7036.032981 secs 2023-03-09T16:55:25.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:55:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7036.032981 secs 2023-03-09T16:55:26.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:55:26 smithi122 ceph-mon[93869]: pgmap v4267: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 6 op/s 2023-03-09T16:55:26.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:55:26 smithi181 ceph-mon[94189]: pgmap v4267: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 6 op/s 2023-03-09T16:55:26.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:55:26 smithi038 ceph-mon[92422]: pgmap v4267: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 6 op/s 2023-03-09T16:55:27.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:55:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7038.281707 secs 2023-03-09T16:55:27.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:55:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7038.281707 secs 2023-03-09T16:55:27.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:55:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7038.281707 secs 2023-03-09T16:55:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7500.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:55:28.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:55:28 smithi122 ceph-mon[93869]: pgmap v4268: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 9 op/s 2023-03-09T16:55:28.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:55:28 smithi181 ceph-mon[94189]: pgmap v4268: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 9 op/s 2023-03-09T16:55:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:55:28 smithi038 ceph-mon[92422]: pgmap v4268: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 9 op/s 2023-03-09T16:55:30.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:55:30 smithi122 ceph-mon[93869]: pgmap v4269: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 6 op/s 2023-03-09T16:55:30.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:55:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7041.033043 secs 2023-03-09T16:55:30.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:55:30 smithi181 ceph-mon[94189]: pgmap v4269: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 6 op/s 2023-03-09T16:55:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:55:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7041.033043 secs 2023-03-09T16:55:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:55:30 smithi038 ceph-mon[92422]: pgmap v4269: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 6 op/s 2023-03-09T16:55:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:55:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7041.033043 secs 2023-03-09T16:55:32.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:55:32 smithi122 ceph-mon[93869]: pgmap v4270: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 43 KiB/s wr, 10 op/s 2023-03-09T16:55:32.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:55:32 smithi181 ceph-mon[94189]: pgmap v4270: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 43 KiB/s wr, 10 op/s 2023-03-09T16:55:32.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:55:32 smithi038 ceph-mon[92422]: pgmap v4270: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 43 KiB/s wr, 10 op/s 2023-03-09T16:55:33.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7505.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:55:34.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:55:34 smithi122 ceph-mon[93869]: pgmap v4271: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T16:55:34.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:55:34 smithi181 ceph-mon[94189]: pgmap v4271: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T16:55:34.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:55:34 smithi038 ceph-mon[92422]: pgmap v4271: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T16:55:35.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:55:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7046.033105 secs 2023-03-09T16:55:35.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:55:35 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:55:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:55:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7046.033105 secs 2023-03-09T16:55:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:55:35 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:55:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:55:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7046.033105 secs 2023-03-09T16:55:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:55:35 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:55:36.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:55:36 smithi122 ceph-mon[93869]: pgmap v4272: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 6 op/s 2023-03-09T16:55:36.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:55:36 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:55:36.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:55:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:55:36.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:55:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:55:36.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:55:36 smithi181 ceph-mon[94189]: pgmap v4272: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 6 op/s 2023-03-09T16:55:36.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:55:36 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:55:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:55:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:55:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:55:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:55:36.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:55:36 smithi038 ceph-mon[92422]: pgmap v4272: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 6 op/s 2023-03-09T16:55:36.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:55:36 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:55:36.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:55:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:55:36.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:55:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:55:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7510.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:55:38.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:55:38 smithi122 ceph-mon[93869]: pgmap v4273: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 30 KiB/s wr, 9 op/s 2023-03-09T16:55:38.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:55:38 smithi181 ceph-mon[94189]: pgmap v4273: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 30 KiB/s wr, 9 op/s 2023-03-09T16:55:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:55:38 smithi038 ceph-mon[92422]: pgmap v4273: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 30 KiB/s wr, 9 op/s 2023-03-09T16:55:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:55:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7051.033164 secs 2023-03-09T16:55:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:55:40 smithi181 ceph-mon[94189]: pgmap v4274: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 30 KiB/s wr, 7 op/s 2023-03-09T16:55:40.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:55:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7051.033164 secs 2023-03-09T16:55:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:55:40 smithi038 ceph-mon[92422]: pgmap v4274: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 30 KiB/s wr, 7 op/s 2023-03-09T16:55:41.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:55:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7051.033164 secs 2023-03-09T16:55:41.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:55:40 smithi122 ceph-mon[93869]: pgmap v4274: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 30 KiB/s wr, 7 op/s 2023-03-09T16:55:42.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:55:42 smithi038 ceph-mon[92422]: pgmap v4275: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 54 KiB/s wr, 11 op/s 2023-03-09T16:55:43.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:55:42 smithi122 ceph-mon[93869]: pgmap v4275: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 54 KiB/s wr, 11 op/s 2023-03-09T16:55:43.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:55:42 smithi181 ceph-mon[94189]: pgmap v4275: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 54 KiB/s wr, 11 op/s 2023-03-09T16:55:43.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7515.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:55:44.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:55:44 smithi038 ceph-mon[92422]: pgmap v4276: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 32 KiB/s wr, 9 op/s 2023-03-09T16:55:45.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:55:44 smithi122 ceph-mon[93869]: pgmap v4276: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 32 KiB/s wr, 9 op/s 2023-03-09T16:55:45.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:55:44 smithi181 ceph-mon[94189]: pgmap v4276: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 32 KiB/s wr, 9 op/s 2023-03-09T16:55:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:55:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7056.033224 secs 2023-03-09T16:55:46.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:55:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7056.033224 secs 2023-03-09T16:55:46.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:55:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7056.033224 secs 2023-03-09T16:55:46.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:55:46 smithi038 ceph-mon[92422]: pgmap v4277: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 32 KiB/s wr, 8 op/s 2023-03-09T16:55:47.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:55:46 smithi122 ceph-mon[93869]: pgmap v4277: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 32 KiB/s wr, 8 op/s 2023-03-09T16:55:47.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:55:46 smithi181 ceph-mon[94189]: pgmap v4277: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 32 KiB/s wr, 8 op/s 2023-03-09T16:55:48.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7520.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:55:48.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:55:48 smithi038 ceph-mon[92422]: pgmap v4278: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 69 KiB/s wr, 13 op/s 2023-03-09T16:55:49.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:55:48 smithi122 ceph-mon[93869]: pgmap v4278: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 69 KiB/s wr, 13 op/s 2023-03-09T16:55:49.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:55:48 smithi181 ceph-mon[94189]: pgmap v4278: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 69 KiB/s wr, 13 op/s 2023-03-09T16:55:51.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:55:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7061.033285 secs 2023-03-09T16:55:51.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:55:50 smithi122 ceph-mon[93869]: pgmap v4279: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 61 KiB/s wr, 9 op/s 2023-03-09T16:55:51.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:55:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7061.033285 secs 2023-03-09T16:55:51.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:55:50 smithi181 ceph-mon[94189]: pgmap v4279: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 61 KiB/s wr, 9 op/s 2023-03-09T16:55:51.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:55:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7061.033285 secs 2023-03-09T16:55:51.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:55:50 smithi038 ceph-mon[92422]: pgmap v4279: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 61 KiB/s wr, 9 op/s 2023-03-09T16:55:53.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:55:52 smithi122 ceph-mon[93869]: pgmap v4280: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 77 KiB/s wr, 13 op/s 2023-03-09T16:55:53.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:55:52 smithi181 ceph-mon[94189]: pgmap v4280: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 77 KiB/s wr, 13 op/s 2023-03-09T16:55:53.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:55:52 smithi038 ceph-mon[92422]: pgmap v4280: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 77 KiB/s wr, 13 op/s 2023-03-09T16:55:53.793 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7525.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:55:55.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:55:54 smithi122 ceph-mon[93869]: pgmap v4281: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 53 KiB/s wr, 9 op/s 2023-03-09T16:55:55.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:55:54 smithi181 ceph-mon[94189]: pgmap v4281: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 53 KiB/s wr, 9 op/s 2023-03-09T16:55:55.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:55:54 smithi038 ceph-mon[92422]: pgmap v4281: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 53 KiB/s wr, 9 op/s 2023-03-09T16:55:56.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:55:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7066.033346 secs 2023-03-09T16:55:56.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:55:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7066.033346 secs 2023-03-09T16:55:56.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:55:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7066.033346 secs 2023-03-09T16:55:57.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:55:56 smithi122 ceph-mon[93869]: pgmap v4282: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 53 KiB/s wr, 8 op/s 2023-03-09T16:55:57.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:55:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:55:57.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:55:56 smithi181 ceph-mon[94189]: pgmap v4282: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 53 KiB/s wr, 8 op/s 2023-03-09T16:55:57.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:55:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:55:57.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:55:56 smithi038 ceph-mon[92422]: pgmap v4282: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 53 KiB/s wr, 8 op/s 2023-03-09T16:55:57.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:55:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:55:58.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:55:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7068.282441 secs 2023-03-09T16:55:58.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:55:57 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:55:58.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:55:57 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:55:58.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:55:57 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:55:58.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:55:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7068.282441 secs 2023-03-09T16:55:58.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:55:57 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:55:58.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:55:57 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:55:58.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:55:57 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:55:58.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:55:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7068.282441 secs 2023-03-09T16:55:58.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:55:57 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:55:58.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:55:57 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:55:58.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:55:57 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:55:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7530.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:55:59.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:55:58 smithi122 ceph-mon[93869]: pgmap v4283: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 87 KiB/s wr, 13 op/s 2023-03-09T16:55:59.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:55:58 smithi181 ceph-mon[94189]: pgmap v4283: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 87 KiB/s wr, 13 op/s 2023-03-09T16:55:59.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:55:58 smithi038 ceph-mon[92422]: pgmap v4283: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 87 KiB/s wr, 13 op/s 2023-03-09T16:56:00.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:55:59 smithi122 ceph-mon[93869]: pgmap v4284: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 51 KiB/s wr, 9 op/s 2023-03-09T16:56:00.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:55:59 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:56:00.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:55:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:56:00.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:55:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:56:00.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:55:59 smithi181 ceph-mon[94189]: pgmap v4284: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 51 KiB/s wr, 9 op/s 2023-03-09T16:56:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:55:59 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:56:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:55:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:56:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:55:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:56:00.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:55:59 smithi038 ceph-mon[92422]: pgmap v4284: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 51 KiB/s wr, 9 op/s 2023-03-09T16:56:00.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:55:59 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:56:00.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:55:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:56:00.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:55:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:56:01.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:56:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7071.033406 secs 2023-03-09T16:56:01.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:56:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7071.033406 secs 2023-03-09T16:56:01.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:56:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7071.033406 secs 2023-03-09T16:56:02.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:56:01 smithi122 ceph-mon[93869]: pgmap v4285: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 86 KiB/s wr, 14 op/s 2023-03-09T16:56:02.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:56:01 smithi181 ceph-mon[94189]: pgmap v4285: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 86 KiB/s wr, 14 op/s 2023-03-09T16:56:02.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:56:01 smithi038 ceph-mon[92422]: pgmap v4285: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 86 KiB/s wr, 14 op/s 2023-03-09T16:56:03.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:56:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7073.282556 secs 2023-03-09T16:56:03.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:56:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7073.282556 secs 2023-03-09T16:56:03.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:56:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7073.282556 secs 2023-03-09T16:56:03.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7535.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:56:04.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:56:03 smithi122 ceph-mon[93869]: pgmap v4286: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 91 KiB/s wr, 23 op/s 2023-03-09T16:56:04.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:56:03 smithi181 ceph-mon[94189]: pgmap v4286: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 91 KiB/s wr, 23 op/s 2023-03-09T16:56:04.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:56:03 smithi038 ceph-mon[92422]: pgmap v4286: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 91 KiB/s wr, 23 op/s 2023-03-09T16:56:05.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:56:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7076.033467 secs 2023-03-09T16:56:05.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:56:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7076.033467 secs 2023-03-09T16:56:05.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:56:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7076.033467 secs 2023-03-09T16:56:06.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:56:06 smithi122 ceph-mon[93869]: pgmap v4287: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 91 KiB/s wr, 22 op/s 2023-03-09T16:56:06.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:56:06 smithi181 ceph-mon[94189]: pgmap v4287: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 91 KiB/s wr, 22 op/s 2023-03-09T16:56:06.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:56:06 smithi038 ceph-mon[92422]: pgmap v4287: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 91 KiB/s wr, 22 op/s 2023-03-09T16:56:07.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:56:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7078.282654 secs 2023-03-09T16:56:07.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:56:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7078.282654 secs 2023-03-09T16:56:07.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:56:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7078.282654 secs 2023-03-09T16:56:08.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:56:08 smithi122 ceph-mon[93869]: pgmap v4288: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 97 KiB/s wr, 26 op/s 2023-03-09T16:56:08.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:56:08 smithi181 ceph-mon[94189]: pgmap v4288: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 97 KiB/s wr, 26 op/s 2023-03-09T16:56:08.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:56:08 smithi038 ceph-mon[92422]: pgmap v4288: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 97 KiB/s wr, 26 op/s 2023-03-09T16:56:08.793 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7540.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:56:10.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:56:10 smithi122 ceph-mon[93869]: osdmap e481: 12 total, 12 up, 12 in 2023-03-09T16:56:10.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:56:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7081.033527 secs 2023-03-09T16:56:10.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:56:10 smithi122 ceph-mon[93869]: pgmap v4290: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 75 KiB/s wr, 25 op/s 2023-03-09T16:56:10.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:56:10 smithi181 ceph-mon[94189]: osdmap e481: 12 total, 12 up, 12 in 2023-03-09T16:56:10.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:56:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7081.033527 secs 2023-03-09T16:56:10.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:56:10 smithi181 ceph-mon[94189]: pgmap v4290: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 75 KiB/s wr, 25 op/s 2023-03-09T16:56:10.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:56:10 smithi038 ceph-mon[92422]: osdmap e481: 12 total, 12 up, 12 in 2023-03-09T16:56:10.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:56:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7081.033527 secs 2023-03-09T16:56:10.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:56:10 smithi038 ceph-mon[92422]: pgmap v4290: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 75 KiB/s wr, 25 op/s 2023-03-09T16:56:12.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:56:12 smithi122 ceph-mon[93869]: pgmap v4291: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 43 KiB/s wr, 22 op/s 2023-03-09T16:56:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:56:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7083.282779 secs 2023-03-09T16:56:12.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:56:12 smithi181 ceph-mon[94189]: pgmap v4291: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 43 KiB/s wr, 22 op/s 2023-03-09T16:56:12.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:56:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7083.282779 secs 2023-03-09T16:56:12.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:56:12 smithi038 ceph-mon[92422]: pgmap v4291: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 43 KiB/s wr, 22 op/s 2023-03-09T16:56:12.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:56:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7083.282779 secs 2023-03-09T16:56:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7545.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:56:14.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:56:14 smithi122 ceph-mon[93869]: pgmap v4292: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 18 KiB/s wr, 8 op/s 2023-03-09T16:56:14.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:56:14 smithi181 ceph-mon[94189]: pgmap v4292: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 18 KiB/s wr, 8 op/s 2023-03-09T16:56:14.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:56:14 smithi038 ceph-mon[92422]: pgmap v4292: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 18 KiB/s wr, 8 op/s 2023-03-09T16:56:15.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:56:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7086.033606 secs 2023-03-09T16:56:15.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:56:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7086.033606 secs 2023-03-09T16:56:15.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:56:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7086.033606 secs 2023-03-09T16:56:16.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:56:16 smithi122 ceph-mon[93869]: pgmap v4293: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 18 KiB/s wr, 8 op/s 2023-03-09T16:56:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:56:16 smithi181 ceph-mon[94189]: pgmap v4293: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 18 KiB/s wr, 8 op/s 2023-03-09T16:56:16.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:56:16 smithi038 ceph-mon[92422]: pgmap v4293: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 18 KiB/s wr, 8 op/s 2023-03-09T16:56:17.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:56:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7088.282896 secs 2023-03-09T16:56:17.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:56:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7088.282896 secs 2023-03-09T16:56:17.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:56:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7088.282896 secs 2023-03-09T16:56:18.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7550.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:56:18.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:56:18 smithi122 ceph-mon[93869]: pgmap v4294: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 7 op/s 2023-03-09T16:56:18.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:56:18 smithi181 ceph-mon[94189]: pgmap v4294: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 7 op/s 2023-03-09T16:56:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:56:18 smithi038 ceph-mon[92422]: pgmap v4294: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 7 op/s 2023-03-09T16:56:20.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:56:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7091.033679 secs 2023-03-09T16:56:20.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:56:20 smithi122 ceph-mon[93869]: pgmap v4295: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 7 op/s 2023-03-09T16:56:20.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:56:20 smithi122 ceph-mon[93869]: osdmap e482: 12 total, 12 up, 12 in 2023-03-09T16:56:20.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:56:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7091.033679 secs 2023-03-09T16:56:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:56:20 smithi181 ceph-mon[94189]: pgmap v4295: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 7 op/s 2023-03-09T16:56:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:56:20 smithi181 ceph-mon[94189]: osdmap e482: 12 total, 12 up, 12 in 2023-03-09T16:56:20.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:56:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7091.033679 secs 2023-03-09T16:56:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:56:20 smithi038 ceph-mon[92422]: pgmap v4295: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 7 op/s 2023-03-09T16:56:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:56:20 smithi038 ceph-mon[92422]: osdmap e482: 12 total, 12 up, 12 in 2023-03-09T16:56:22.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:56:22 smithi122 ceph-mon[93869]: pgmap v4297: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T16:56:22.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:56:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7093.283037 secs 2023-03-09T16:56:22.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:56:22 smithi181 ceph-mon[94189]: pgmap v4297: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T16:56:22.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:56:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7093.283037 secs 2023-03-09T16:56:22.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:56:22 smithi038 ceph-mon[92422]: pgmap v4297: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T16:56:22.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:56:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7093.283037 secs 2023-03-09T16:56:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7555.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:56:24.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:56:24 smithi122 ceph-mon[93869]: pgmap v4298: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T16:56:24.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:56:24 smithi181 ceph-mon[94189]: pgmap v4298: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T16:56:24.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:56:24 smithi038 ceph-mon[92422]: pgmap v4298: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T16:56:25.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:56:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7096.033739 secs 2023-03-09T16:56:25.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:56:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7096.033739 secs 2023-03-09T16:56:25.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:56:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7096.033739 secs 2023-03-09T16:56:26.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:56:26 smithi122 ceph-mon[93869]: pgmap v4299: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T16:56:26.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:56:26 smithi181 ceph-mon[94189]: pgmap v4299: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T16:56:26.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:56:26 smithi038 ceph-mon[92422]: pgmap v4299: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T16:56:27.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:56:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7098.283158 secs 2023-03-09T16:56:27.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:56:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7098.283158 secs 2023-03-09T16:56:27.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:56:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7098.283158 secs 2023-03-09T16:56:28.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7560.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:56:28.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:56:28 smithi122 ceph-mon[93869]: pgmap v4300: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T16:56:28.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:56:28 smithi181 ceph-mon[94189]: pgmap v4300: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T16:56:28.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:56:28 smithi038 ceph-mon[92422]: pgmap v4300: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T16:56:30.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:56:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7101.033801 secs 2023-03-09T16:56:30.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:56:30 smithi122 ceph-mon[93869]: pgmap v4301: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T16:56:30.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:56:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7101.033801 secs 2023-03-09T16:56:30.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:56:30 smithi181 ceph-mon[94189]: pgmap v4301: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T16:56:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:56:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7101.033801 secs 2023-03-09T16:56:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:56:30 smithi038 ceph-mon[92422]: pgmap v4301: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T16:56:32.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:56:32 smithi122 ceph-mon[93869]: pgmap v4302: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 9 op/s 2023-03-09T16:56:32.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:56:32 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 7103.283301 secs 2023-03-09T16:56:32.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:56:32 smithi181 ceph-mon[94189]: pgmap v4302: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 9 op/s 2023-03-09T16:56:32.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:56:32 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 7103.283301 secs 2023-03-09T16:56:32.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:56:32 smithi038 ceph-mon[92422]: pgmap v4302: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 9 op/s 2023-03-09T16:56:32.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:56:32 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 7103.283301 secs 2023-03-09T16:56:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7565.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:56:34.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:56:34 smithi122 ceph-mon[93869]: pgmap v4303: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 6 op/s 2023-03-09T16:56:34.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:56:34 smithi181 ceph-mon[94189]: pgmap v4303: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 6 op/s 2023-03-09T16:56:34.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:56:34 smithi038 ceph-mon[92422]: pgmap v4303: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 6 op/s 2023-03-09T16:56:35.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:56:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7106.033864 secs 2023-03-09T16:56:35.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:56:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7106.033864 secs 2023-03-09T16:56:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:56:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7106.033864 secs 2023-03-09T16:56:36.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:56:36 smithi122 ceph-mon[93869]: pgmap v4304: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 5 op/s 2023-03-09T16:56:36.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:56:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:56:36.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:56:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:56:36.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:56:36 smithi181 ceph-mon[94189]: pgmap v4304: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 5 op/s 2023-03-09T16:56:36.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:56:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:56:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:56:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:56:36.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:56:36 smithi038 ceph-mon[92422]: pgmap v4304: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 5 op/s 2023-03-09T16:56:36.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:56:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:56:36.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:56:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:56:37.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:56:37 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 7108.283373 secs 2023-03-09T16:56:37.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:56:37 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 7108.283373 secs 2023-03-09T16:56:38.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:56:37 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 7108.283373 secs 2023-03-09T16:56:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7570.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:56:38.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:56:38 smithi181 ceph-mon[94189]: pgmap v4305: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 9 op/s 2023-03-09T16:56:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:56:38 smithi038 ceph-mon[92422]: pgmap v4305: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 9 op/s 2023-03-09T16:56:39.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:56:38 smithi122 ceph-mon[93869]: pgmap v4305: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 9 op/s 2023-03-09T16:56:41.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:56:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7111.033925 secs 2023-03-09T16:56:41.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:56:40 smithi038 ceph-mon[92422]: pgmap v4306: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 6 op/s 2023-03-09T16:56:41.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:56:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7111.033925 secs 2023-03-09T16:56:41.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:56:40 smithi122 ceph-mon[93869]: pgmap v4306: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 6 op/s 2023-03-09T16:56:41.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:56:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7111.033925 secs 2023-03-09T16:56:41.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:56:40 smithi181 ceph-mon[94189]: pgmap v4306: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 6 op/s 2023-03-09T16:56:42.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:56:42 smithi038 ceph-mon[92422]: pgmap v4307: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 46 KiB/s wr, 9 op/s 2023-03-09T16:56:42.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:56:42 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 7113.283453 secs 2023-03-09T16:56:43.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:56:42 smithi122 ceph-mon[93869]: pgmap v4307: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 46 KiB/s wr, 9 op/s 2023-03-09T16:56:43.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:56:42 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 7113.283453 secs 2023-03-09T16:56:43.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:56:42 smithi181 ceph-mon[94189]: pgmap v4307: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 46 KiB/s wr, 9 op/s 2023-03-09T16:56:43.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:56:42 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 7113.283453 secs 2023-03-09T16:56:43.793 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7575.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:56:44.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:56:44 smithi038 ceph-mon[92422]: pgmap v4308: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 34 KiB/s wr, 6 op/s 2023-03-09T16:56:45.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:56:44 smithi122 ceph-mon[93869]: pgmap v4308: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 34 KiB/s wr, 6 op/s 2023-03-09T16:56:45.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:56:44 smithi181 ceph-mon[94189]: pgmap v4308: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 34 KiB/s wr, 6 op/s 2023-03-09T16:56:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:56:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7116.033984 secs 2023-03-09T16:56:46.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:56:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7116.033984 secs 2023-03-09T16:56:46.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:56:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7116.033984 secs 2023-03-09T16:56:46.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:56:46 smithi038 ceph-mon[92422]: pgmap v4309: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 34 KiB/s wr, 6 op/s 2023-03-09T16:56:47.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:56:46 smithi122 ceph-mon[93869]: pgmap v4309: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 34 KiB/s wr, 6 op/s 2023-03-09T16:56:47.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:56:46 smithi181 ceph-mon[94189]: pgmap v4309: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 34 KiB/s wr, 6 op/s 2023-03-09T16:56:48.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:56:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7118.283576 secs 2023-03-09T16:56:48.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:56:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7118.283576 secs 2023-03-09T16:56:48.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:56:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7118.283576 secs 2023-03-09T16:56:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7580.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:56:49.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:56:48 smithi122 ceph-mon[93869]: pgmap v4310: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 50 KiB/s wr, 10 op/s 2023-03-09T16:56:49.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:56:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:56:49.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:56:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:56:49.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:56:48 smithi181 ceph-mon[94189]: pgmap v4310: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 50 KiB/s wr, 10 op/s 2023-03-09T16:56:49.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:56:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:56:49.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:56:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:56:49.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:56:48 smithi038 ceph-mon[92422]: pgmap v4310: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 50 KiB/s wr, 10 op/s 2023-03-09T16:56:49.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:56:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:56:49.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:56:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:56:51.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:56:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7121.034045 secs 2023-03-09T16:56:51.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:56:50 smithi122 ceph-mon[93869]: pgmap v4311: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 40 KiB/s wr, 8 op/s 2023-03-09T16:56:51.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:56:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7121.034045 secs 2023-03-09T16:56:51.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:56:50 smithi181 ceph-mon[94189]: pgmap v4311: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 40 KiB/s wr, 8 op/s 2023-03-09T16:56:51.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:56:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7121.034045 secs 2023-03-09T16:56:51.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:56:50 smithi038 ceph-mon[92422]: pgmap v4311: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 40 KiB/s wr, 8 op/s 2023-03-09T16:56:53.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:56:52 smithi122 ceph-mon[93869]: pgmap v4312: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 76 KiB/s wr, 14 op/s 2023-03-09T16:56:53.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:56:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7123.283723 secs 2023-03-09T16:56:53.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:56:52 smithi181 ceph-mon[94189]: pgmap v4312: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 76 KiB/s wr, 14 op/s 2023-03-09T16:56:53.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:56:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7123.283723 secs 2023-03-09T16:56:53.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:56:52 smithi038 ceph-mon[92422]: pgmap v4312: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 76 KiB/s wr, 14 op/s 2023-03-09T16:56:53.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:56:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7123.283723 secs 2023-03-09T16:56:53.793 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7585.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:56:55.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:56:54 smithi122 ceph-mon[93869]: pgmap v4313: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 53 KiB/s wr, 12 op/s 2023-03-09T16:56:55.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:56:54 smithi181 ceph-mon[94189]: pgmap v4313: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 53 KiB/s wr, 12 op/s 2023-03-09T16:56:55.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:56:54 smithi038 ceph-mon[92422]: pgmap v4313: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 53 KiB/s wr, 12 op/s 2023-03-09T16:56:56.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:56:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7126.034105 secs 2023-03-09T16:56:56.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:56:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7126.034105 secs 2023-03-09T16:56:56.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:56:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7126.034105 secs 2023-03-09T16:56:57.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:56:56 smithi122 ceph-mon[93869]: pgmap v4314: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 53 KiB/s wr, 12 op/s 2023-03-09T16:56:57.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:56:56 smithi181 ceph-mon[94189]: pgmap v4314: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 53 KiB/s wr, 12 op/s 2023-03-09T16:56:57.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:56:56 smithi038 ceph-mon[92422]: pgmap v4314: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 53 KiB/s wr, 12 op/s 2023-03-09T16:56:58.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:56:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7128.283838 secs 2023-03-09T16:56:58.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:56:57 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:56:58.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:56:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7128.283838 secs 2023-03-09T16:56:58.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:56:57 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:56:58.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:56:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7128.283838 secs 2023-03-09T16:56:58.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:56:57 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:56:58.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7590.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:56:59.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:56:58 smithi122 ceph-mon[93869]: pgmap v4315: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 55 KiB/s wr, 16 op/s 2023-03-09T16:56:59.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:56:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:56:59.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:56:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:56:59.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:56:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:56:59.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:56:58 smithi181 ceph-mon[94189]: pgmap v4315: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 55 KiB/s wr, 16 op/s 2023-03-09T16:56:59.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:56:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:56:59.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:56:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:56:59.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:56:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:56:59.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:56:58 smithi038 ceph-mon[92422]: pgmap v4315: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 55 KiB/s wr, 16 op/s 2023-03-09T16:56:59.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:56:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:56:59.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:56:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:56:59.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:56:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:57:01.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:57:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7131.034167 secs 2023-03-09T16:57:01.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:57:00 smithi122 ceph-mon[93869]: pgmap v4316: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 39 KiB/s wr, 12 op/s 2023-03-09T16:57:01.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:57:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7131.034167 secs 2023-03-09T16:57:01.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:57:00 smithi181 ceph-mon[94189]: pgmap v4316: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 39 KiB/s wr, 12 op/s 2023-03-09T16:57:01.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:57:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7131.034167 secs 2023-03-09T16:57:01.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:57:00 smithi038 ceph-mon[92422]: pgmap v4316: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 39 KiB/s wr, 12 op/s 2023-03-09T16:57:03.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:57:02 smithi122 ceph-mon[93869]: pgmap v4317: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 114 KiB/s wr, 19 op/s 2023-03-09T16:57:03.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:57:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7133.283982 secs 2023-03-09T16:57:03.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:57:02 smithi181 ceph-mon[94189]: pgmap v4317: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 114 KiB/s wr, 19 op/s 2023-03-09T16:57:03.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:57:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7133.283982 secs 2023-03-09T16:57:03.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:57:02 smithi038 ceph-mon[92422]: pgmap v4317: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 114 KiB/s wr, 19 op/s 2023-03-09T16:57:03.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:57:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7133.283982 secs 2023-03-09T16:57:03.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7595.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:57:05.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:57:04 smithi122 ceph-mon[93869]: pgmap v4318: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 100 KiB/s wr, 25 op/s 2023-03-09T16:57:05.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:57:04 smithi122 ceph-mon[93869]: osdmap e483: 12 total, 12 up, 12 in 2023-03-09T16:57:05.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:57:04 smithi181 ceph-mon[94189]: pgmap v4318: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 100 KiB/s wr, 25 op/s 2023-03-09T16:57:05.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:57:04 smithi181 ceph-mon[94189]: osdmap e483: 12 total, 12 up, 12 in 2023-03-09T16:57:05.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:57:04 smithi038 ceph-mon[92422]: pgmap v4318: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 100 KiB/s wr, 25 op/s 2023-03-09T16:57:05.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:57:04 smithi038 ceph-mon[92422]: osdmap e483: 12 total, 12 up, 12 in 2023-03-09T16:57:06.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:57:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7136.034230 secs 2023-03-09T16:57:06.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:57:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7136.034230 secs 2023-03-09T16:57:06.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:57:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7136.034230 secs 2023-03-09T16:57:07.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:57:06 smithi122 ceph-mon[93869]: pgmap v4320: 129 pgs: 2 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 122 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 119 KiB/s wr, 29 op/s 2023-03-09T16:57:07.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:57:06 smithi181 ceph-mon[94189]: pgmap v4320: 129 pgs: 2 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 122 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 119 KiB/s wr, 29 op/s 2023-03-09T16:57:07.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:57:06 smithi038 ceph-mon[92422]: pgmap v4320: 129 pgs: 2 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 122 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 119 KiB/s wr, 29 op/s 2023-03-09T16:57:08.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:57:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7138.284127 secs 2023-03-09T16:57:08.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:57:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7138.284127 secs 2023-03-09T16:57:08.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:57:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7138.284127 secs 2023-03-09T16:57:08.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7600.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:57:09.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:57:08 smithi122 ceph-mon[93869]: pgmap v4321: 129 pgs: 2 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 122 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 145 KiB/s wr, 29 op/s 2023-03-09T16:57:09.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:57:08 smithi181 ceph-mon[94189]: pgmap v4321: 129 pgs: 2 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 122 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 145 KiB/s wr, 29 op/s 2023-03-09T16:57:09.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:57:08 smithi038 ceph-mon[92422]: pgmap v4321: 129 pgs: 2 active+clean+snaptrim, 5 active+clean+snaptrim_wait, 122 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 145 KiB/s wr, 29 op/s 2023-03-09T16:57:10.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:57:09 smithi122 ceph-mon[93869]: pgmap v4322: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 145 KiB/s wr, 29 op/s 2023-03-09T16:57:10.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:57:09 smithi181 ceph-mon[94189]: pgmap v4322: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 145 KiB/s wr, 29 op/s 2023-03-09T16:57:10.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:57:09 smithi038 ceph-mon[92422]: pgmap v4322: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 145 KiB/s wr, 29 op/s 2023-03-09T16:57:11.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:57:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7141.034292 secs 2023-03-09T16:57:11.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:57:10 smithi122 ceph-mon[93869]: osdmap e484: 12 total, 12 up, 12 in 2023-03-09T16:57:11.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:57:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7141.034292 secs 2023-03-09T16:57:11.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:57:10 smithi181 ceph-mon[94189]: osdmap e484: 12 total, 12 up, 12 in 2023-03-09T16:57:11.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:57:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7141.034292 secs 2023-03-09T16:57:11.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:57:10 smithi038 ceph-mon[92422]: osdmap e484: 12 total, 12 up, 12 in 2023-03-09T16:57:12.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:57:11 smithi122 ceph-mon[93869]: pgmap v4324: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 76 KiB/s wr, 11 op/s 2023-03-09T16:57:12.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:57:11 smithi181 ceph-mon[94189]: pgmap v4324: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 76 KiB/s wr, 11 op/s 2023-03-09T16:57:12.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:57:11 smithi038 ceph-mon[92422]: pgmap v4324: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 76 KiB/s wr, 11 op/s 2023-03-09T16:57:13.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:57:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7143.284239 secs 2023-03-09T16:57:13.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:57:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7143.284239 secs 2023-03-09T16:57:13.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:57:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7143.284239 secs 2023-03-09T16:57:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7605.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:57:14.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:57:13 smithi122 ceph-mon[93869]: pgmap v4325: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 62 KiB/s wr, 9 op/s 2023-03-09T16:57:14.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:57:13 smithi181 ceph-mon[94189]: pgmap v4325: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 62 KiB/s wr, 9 op/s 2023-03-09T16:57:14.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:57:13 smithi038 ceph-mon[92422]: pgmap v4325: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 62 KiB/s wr, 9 op/s 2023-03-09T16:57:15.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:57:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7146.034353 secs 2023-03-09T16:57:15.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:57:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7146.034353 secs 2023-03-09T16:57:15.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:57:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7146.034353 secs 2023-03-09T16:57:16.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:57:16 smithi122 ceph-mon[93869]: pgmap v4326: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 61 KiB/s wr, 9 op/s 2023-03-09T16:57:16.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:57:16 smithi181 ceph-mon[94189]: pgmap v4326: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 61 KiB/s wr, 9 op/s 2023-03-09T16:57:16.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:57:16 smithi038 ceph-mon[92422]: pgmap v4326: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 61 KiB/s wr, 9 op/s 2023-03-09T16:57:17.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:57:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7148.284351 secs 2023-03-09T16:57:17.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:57:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7148.284351 secs 2023-03-09T16:57:17.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:57:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7148.284351 secs 2023-03-09T16:57:18.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:57:18 smithi122 ceph-mon[93869]: pgmap v4327: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 49 KiB/s wr, 8 op/s 2023-03-09T16:57:18.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:57:18 smithi181 ceph-mon[94189]: pgmap v4327: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 49 KiB/s wr, 8 op/s 2023-03-09T16:57:18.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:57:18 smithi038 ceph-mon[92422]: pgmap v4327: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 49 KiB/s wr, 8 op/s 2023-03-09T16:57:18.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7610.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:57:20.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:57:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7151.034414 secs 2023-03-09T16:57:20.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:57:20 smithi122 ceph-mon[93869]: pgmap v4328: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 8 op/s 2023-03-09T16:57:20.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:57:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7151.034414 secs 2023-03-09T16:57:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:57:20 smithi181 ceph-mon[94189]: pgmap v4328: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 8 op/s 2023-03-09T16:57:20.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:57:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7151.034414 secs 2023-03-09T16:57:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:57:20 smithi038 ceph-mon[92422]: pgmap v4328: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 8 op/s 2023-03-09T16:57:22.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:57:22 smithi122 ceph-mon[93869]: pgmap v4329: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 9 op/s 2023-03-09T16:57:22.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:57:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7153.284468 secs 2023-03-09T16:57:22.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:57:22 smithi181 ceph-mon[94189]: pgmap v4329: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 9 op/s 2023-03-09T16:57:22.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:57:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7153.284468 secs 2023-03-09T16:57:22.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:57:22 smithi038 ceph-mon[92422]: pgmap v4329: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 9 op/s 2023-03-09T16:57:22.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:57:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7153.284468 secs 2023-03-09T16:57:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7615.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:57:24.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:57:24 smithi122 ceph-mon[93869]: pgmap v4330: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 KiB/s wr, 6 op/s 2023-03-09T16:57:24.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:57:24 smithi181 ceph-mon[94189]: pgmap v4330: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 KiB/s wr, 6 op/s 2023-03-09T16:57:24.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:57:24 smithi038 ceph-mon[92422]: pgmap v4330: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 KiB/s wr, 6 op/s 2023-03-09T16:57:25.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:57:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7156.034476 secs 2023-03-09T16:57:25.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:57:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7156.034476 secs 2023-03-09T16:57:25.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:57:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7156.034476 secs 2023-03-09T16:57:26.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:57:26 smithi122 ceph-mon[93869]: pgmap v4331: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 KiB/s wr, 6 op/s 2023-03-09T16:57:26.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:57:26 smithi181 ceph-mon[94189]: pgmap v4331: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 KiB/s wr, 6 op/s 2023-03-09T16:57:26.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:57:26 smithi038 ceph-mon[92422]: pgmap v4331: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 KiB/s wr, 6 op/s 2023-03-09T16:57:27.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:57:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7158.284605 secs 2023-03-09T16:57:27.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:57:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7158.284605 secs 2023-03-09T16:57:27.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:57:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7158.284605 secs 2023-03-09T16:57:28.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7620.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:57:28.974 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:57:28 smithi122 ceph-mon[93869]: pgmap v4332: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 28 KiB/s wr, 9 op/s 2023-03-09T16:57:28.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:57:28 smithi181 ceph-mon[94189]: pgmap v4332: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 28 KiB/s wr, 9 op/s 2023-03-09T16:57:28.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:57:28 smithi038 ceph-mon[92422]: pgmap v4332: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 28 KiB/s wr, 9 op/s 2023-03-09T16:57:30.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:57:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7161.034537 secs 2023-03-09T16:57:30.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:57:30 smithi122 ceph-mon[93869]: pgmap v4333: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 6 op/s 2023-03-09T16:57:30.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:57:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7161.034537 secs 2023-03-09T16:57:30.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:57:30 smithi181 ceph-mon[94189]: pgmap v4333: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 6 op/s 2023-03-09T16:57:30.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:57:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7161.034537 secs 2023-03-09T16:57:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:57:30 smithi038 ceph-mon[92422]: pgmap v4333: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 6 op/s 2023-03-09T16:57:32.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:57:32 smithi122 ceph-mon[93869]: pgmap v4334: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 9 op/s 2023-03-09T16:57:32.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:57:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7163.284705 secs 2023-03-09T16:57:32.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:57:32 smithi181 ceph-mon[94189]: pgmap v4334: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 9 op/s 2023-03-09T16:57:32.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:57:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7163.284705 secs 2023-03-09T16:57:32.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:57:32 smithi038 ceph-mon[92422]: pgmap v4334: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 9 op/s 2023-03-09T16:57:32.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:57:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7163.284705 secs 2023-03-09T16:57:33.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7625.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:57:34.974 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:57:34 smithi122 ceph-mon[93869]: pgmap v4335: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 6 op/s 2023-03-09T16:57:34.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:57:34 smithi181 ceph-mon[94189]: pgmap v4335: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 6 op/s 2023-03-09T16:57:34.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:57:34 smithi038 ceph-mon[92422]: pgmap v4335: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 6 op/s 2023-03-09T16:57:35.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:57:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7166.034595 secs 2023-03-09T16:57:35.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:57:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7166.034595 secs 2023-03-09T16:57:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:57:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7166.034595 secs 2023-03-09T16:57:36.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:57:36 smithi038 ceph-mon[92422]: pgmap v4336: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 6 op/s 2023-03-09T16:57:37.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:57:36 smithi122 ceph-mon[93869]: pgmap v4336: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 6 op/s 2023-03-09T16:57:37.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:57:36 smithi181 ceph-mon[94189]: pgmap v4336: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 6 op/s 2023-03-09T16:57:37.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:57:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7168.284829 secs 2023-03-09T16:57:38.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:57:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7168.284829 secs 2023-03-09T16:57:38.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:57:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7168.284829 secs 2023-03-09T16:57:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7630.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:57:39.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:57:38 smithi122 ceph-mon[93869]: pgmap v4337: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 41 KiB/s wr, 10 op/s 2023-03-09T16:57:39.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:57:38 smithi181 ceph-mon[94189]: pgmap v4337: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 41 KiB/s wr, 10 op/s 2023-03-09T16:57:39.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:57:38 smithi038 ceph-mon[92422]: pgmap v4337: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 41 KiB/s wr, 10 op/s 2023-03-09T16:57:41.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:57:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7171.034656 secs 2023-03-09T16:57:41.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:57:40 smithi122 ceph-mon[93869]: pgmap v4338: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T16:57:41.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:57:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7171.034656 secs 2023-03-09T16:57:41.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:57:40 smithi181 ceph-mon[94189]: pgmap v4338: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T16:57:41.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:57:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7171.034656 secs 2023-03-09T16:57:41.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:57:40 smithi038 ceph-mon[92422]: pgmap v4338: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T16:57:43.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:57:42 smithi122 ceph-mon[93869]: pgmap v4339: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 45 KiB/s wr, 10 op/s 2023-03-09T16:57:43.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:57:42 smithi181 ceph-mon[94189]: pgmap v4339: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 45 KiB/s wr, 10 op/s 2023-03-09T16:57:43.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:57:42 smithi038 ceph-mon[92422]: pgmap v4339: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 45 KiB/s wr, 10 op/s 2023-03-09T16:57:43.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7635.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:57:44.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:57:43 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:57:44.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:57:43 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:57:44.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:57:43 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:57:45.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:57:44 smithi122 ceph-mon[93869]: pgmap v4340: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 44 KiB/s wr, 8 op/s 2023-03-09T16:57:45.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:57:44 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:57:45.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:57:44 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:57:45.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:57:44 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:57:45.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:57:44 smithi181 ceph-mon[94189]: pgmap v4340: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 44 KiB/s wr, 8 op/s 2023-03-09T16:57:45.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:57:44 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:57:45.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:57:44 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:57:45.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:57:44 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:57:45.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:57:44 smithi038 ceph-mon[92422]: pgmap v4340: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 44 KiB/s wr, 8 op/s 2023-03-09T16:57:45.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:57:44 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:57:45.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:57:44 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:57:45.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:57:44 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:57:46.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:57:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7176.034713 secs 2023-03-09T16:57:46.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:57:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7176.034713 secs 2023-03-09T16:57:46.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:57:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7176.034713 secs 2023-03-09T16:57:47.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:57:46 smithi122 ceph-mon[93869]: pgmap v4341: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 44 KiB/s wr, 8 op/s 2023-03-09T16:57:47.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:57:46 smithi181 ceph-mon[94189]: pgmap v4341: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 44 KiB/s wr, 8 op/s 2023-03-09T16:57:47.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:57:46 smithi038 ceph-mon[92422]: pgmap v4341: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 44 KiB/s wr, 8 op/s 2023-03-09T16:57:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7640.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:57:49.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:57:48 smithi122 ceph-mon[93869]: pgmap v4342: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 80 KiB/s wr, 12 op/s 2023-03-09T16:57:49.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:57:48 smithi181 ceph-mon[94189]: pgmap v4342: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 80 KiB/s wr, 12 op/s 2023-03-09T16:57:49.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:57:48 smithi038 ceph-mon[92422]: pgmap v4342: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 80 KiB/s wr, 12 op/s 2023-03-09T16:57:51.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:57:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7181.034774 secs 2023-03-09T16:57:51.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:57:50 smithi122 ceph-mon[93869]: pgmap v4343: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 53 KiB/s wr, 9 op/s 2023-03-09T16:57:51.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:57:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7181.034774 secs 2023-03-09T16:57:51.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:57:50 smithi181 ceph-mon[94189]: pgmap v4343: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 53 KiB/s wr, 9 op/s 2023-03-09T16:57:51.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:57:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7181.034774 secs 2023-03-09T16:57:51.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:57:50 smithi038 ceph-mon[92422]: pgmap v4343: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 53 KiB/s wr, 9 op/s 2023-03-09T16:57:53.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:57:52 smithi122 ceph-mon[93869]: pgmap v4344: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 73 KiB/s wr, 12 op/s 2023-03-09T16:57:53.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:57:52 smithi181 ceph-mon[94189]: pgmap v4344: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 73 KiB/s wr, 12 op/s 2023-03-09T16:57:53.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:57:52 smithi038 ceph-mon[92422]: pgmap v4344: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 73 KiB/s wr, 12 op/s 2023-03-09T16:57:53.793 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7645.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:57:55.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:57:54 smithi122 ceph-mon[93869]: pgmap v4345: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 57 KiB/s wr, 9 op/s 2023-03-09T16:57:55.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:57:54 smithi181 ceph-mon[94189]: pgmap v4345: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 57 KiB/s wr, 9 op/s 2023-03-09T16:57:55.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:57:54 smithi038 ceph-mon[92422]: pgmap v4345: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 57 KiB/s wr, 9 op/s 2023-03-09T16:57:56.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:57:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7186.034832 secs 2023-03-09T16:57:56.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:57:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7186.034832 secs 2023-03-09T16:57:56.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:57:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7186.034832 secs 2023-03-09T16:57:57.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:57:56 smithi122 ceph-mon[93869]: pgmap v4346: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 56 KiB/s wr, 9 op/s 2023-03-09T16:57:57.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:57:56 smithi181 ceph-mon[94189]: pgmap v4346: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 56 KiB/s wr, 9 op/s 2023-03-09T16:57:57.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:57:56 smithi038 ceph-mon[92422]: pgmap v4346: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 56 KiB/s wr, 9 op/s 2023-03-09T16:57:58.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:57:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7188.285269 secs 2023-03-09T16:57:58.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:57:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7188.285269 secs 2023-03-09T16:57:58.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:57:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7188.285269 secs 2023-03-09T16:57:58.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7650.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:57:59.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:57:58 smithi122 ceph-mon[93869]: pgmap v4347: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 89 KiB/s wr, 12 op/s 2023-03-09T16:57:59.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:57:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:57:59.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:57:58 smithi181 ceph-mon[94189]: pgmap v4347: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 89 KiB/s wr, 12 op/s 2023-03-09T16:57:59.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:57:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:57:59.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:57:58 smithi038 ceph-mon[92422]: pgmap v4347: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 89 KiB/s wr, 12 op/s 2023-03-09T16:57:59.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:57:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:58:00.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:57:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:58:00.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:57:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:58:00.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:57:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:58:00.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:57:59 smithi122 ceph-mon[93869]: pgmap v4348: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 8 op/s 2023-03-09T16:58:00.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:57:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:58:00.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:57:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:58:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:57:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:58:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:57:59 smithi181 ceph-mon[94189]: pgmap v4348: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 8 op/s 2023-03-09T16:58:00.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:57:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:58:00.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:57:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:58:00.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:57:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:58:00.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:57:59 smithi038 ceph-mon[92422]: pgmap v4348: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 8 op/s 2023-03-09T16:58:01.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:58:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7191.034893 secs 2023-03-09T16:58:01.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:58:00 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:58:01.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:58:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:58:01.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:58:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:58:01.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:58:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7191.034893 secs 2023-03-09T16:58:01.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:58:00 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:58:01.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:58:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:58:01.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:58:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:58:01.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:58:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7191.034893 secs 2023-03-09T16:58:01.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:58:00 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:58:01.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:58:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:58:01.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:58:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:58:02.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:58:01 smithi122 ceph-mon[93869]: pgmap v4349: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 83 KiB/s wr, 13 op/s 2023-03-09T16:58:02.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:58:01 smithi181 ceph-mon[94189]: pgmap v4349: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 83 KiB/s wr, 13 op/s 2023-03-09T16:58:02.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:58:01 smithi038 ceph-mon[92422]: pgmap v4349: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 83 KiB/s wr, 13 op/s 2023-03-09T16:58:03.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:58:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7193.285384 secs 2023-03-09T16:58:03.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:58:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7193.285384 secs 2023-03-09T16:58:03.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:58:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7193.285384 secs 2023-03-09T16:58:03.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7655.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:58:04.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:58:03 smithi122 ceph-mon[93869]: pgmap v4350: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 84 KiB/s wr, 21 op/s 2023-03-09T16:58:04.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:58:03 smithi181 ceph-mon[94189]: pgmap v4350: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 84 KiB/s wr, 21 op/s 2023-03-09T16:58:04.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:58:03 smithi038 ceph-mon[92422]: pgmap v4350: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 84 KiB/s wr, 21 op/s 2023-03-09T16:58:05.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:58:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7196.034954 secs 2023-03-09T16:58:05.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:58:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7196.034954 secs 2023-03-09T16:58:05.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:58:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7196.034954 secs 2023-03-09T16:58:06.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:58:06 smithi122 ceph-mon[93869]: pgmap v4351: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 84 KiB/s wr, 21 op/s 2023-03-09T16:58:06.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:58:06 smithi181 ceph-mon[94189]: pgmap v4351: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 84 KiB/s wr, 21 op/s 2023-03-09T16:58:06.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:58:06 smithi038 ceph-mon[92422]: pgmap v4351: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 84 KiB/s wr, 21 op/s 2023-03-09T16:58:07.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:58:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7198.285485 secs 2023-03-09T16:58:07.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:58:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7198.285485 secs 2023-03-09T16:58:07.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:58:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7198.285485 secs 2023-03-09T16:58:08.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:58:08 smithi122 ceph-mon[93869]: pgmap v4352: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 102 KiB/s wr, 25 op/s 2023-03-09T16:58:08.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:58:08 smithi181 ceph-mon[94189]: pgmap v4352: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 102 KiB/s wr, 25 op/s 2023-03-09T16:58:08.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:58:08 smithi038 ceph-mon[92422]: pgmap v4352: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 102 KiB/s wr, 25 op/s 2023-03-09T16:58:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7660.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:58:10.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:58:10 smithi038 ceph-mon[92422]: osdmap e485: 12 total, 12 up, 12 in 2023-03-09T16:58:10.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:58:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7201.035014 secs 2023-03-09T16:58:10.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:58:10 smithi038 ceph-mon[92422]: pgmap v4354: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 82 KiB/s wr, 25 op/s 2023-03-09T16:58:10.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:58:10 smithi122 ceph-mon[93869]: osdmap e485: 12 total, 12 up, 12 in 2023-03-09T16:58:10.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:58:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7201.035014 secs 2023-03-09T16:58:10.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:58:10 smithi122 ceph-mon[93869]: pgmap v4354: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 82 KiB/s wr, 25 op/s 2023-03-09T16:58:10.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:58:10 smithi181 ceph-mon[94189]: osdmap e485: 12 total, 12 up, 12 in 2023-03-09T16:58:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:58:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7201.035014 secs 2023-03-09T16:58:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:58:10 smithi181 ceph-mon[94189]: pgmap v4354: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 82 KiB/s wr, 25 op/s 2023-03-09T16:58:12.974 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:58:12 smithi122 ceph-mon[93869]: pgmap v4355: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 65 KiB/s wr, 23 op/s 2023-03-09T16:58:12.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:58:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7203.285609 secs 2023-03-09T16:58:12.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:58:12 smithi181 ceph-mon[94189]: pgmap v4355: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 65 KiB/s wr, 23 op/s 2023-03-09T16:58:12.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:58:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7203.285609 secs 2023-03-09T16:58:12.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:58:12 smithi038 ceph-mon[92422]: pgmap v4355: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 65 KiB/s wr, 23 op/s 2023-03-09T16:58:12.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:58:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7203.285609 secs 2023-03-09T16:58:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7665.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:58:14.974 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:58:14 smithi122 ceph-mon[93869]: pgmap v4356: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 39 KiB/s wr, 9 op/s 2023-03-09T16:58:14.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:58:14 smithi181 ceph-mon[94189]: pgmap v4356: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 39 KiB/s wr, 9 op/s 2023-03-09T16:58:14.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:58:14 smithi038 ceph-mon[92422]: pgmap v4356: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 39 KiB/s wr, 9 op/s 2023-03-09T16:58:15.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:58:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7206.035073 secs 2023-03-09T16:58:16.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:58:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7206.035073 secs 2023-03-09T16:58:16.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:58:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7206.035073 secs 2023-03-09T16:58:16.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:58:16 smithi038 ceph-mon[92422]: pgmap v4357: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 39 KiB/s wr, 9 op/s 2023-03-09T16:58:17.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:58:16 smithi122 ceph-mon[93869]: pgmap v4357: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 39 KiB/s wr, 9 op/s 2023-03-09T16:58:17.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:58:16 smithi181 ceph-mon[94189]: pgmap v4357: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 39 KiB/s wr, 9 op/s 2023-03-09T16:58:17.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:58:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7208.285747 secs 2023-03-09T16:58:18.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:58:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7208.285747 secs 2023-03-09T16:58:18.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:58:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7208.285747 secs 2023-03-09T16:58:18.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7670.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:58:18.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:58:18 smithi038 ceph-mon[92422]: pgmap v4358: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T16:58:19.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:58:18 smithi122 ceph-mon[93869]: pgmap v4358: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T16:58:19.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:58:18 smithi181 ceph-mon[94189]: pgmap v4358: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T16:58:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:58:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7211.035151 secs 2023-03-09T16:58:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:58:20 smithi038 ceph-mon[92422]: pgmap v4359: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 7 op/s 2023-03-09T16:58:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:58:20 smithi038 ceph-mon[92422]: osdmap e486: 12 total, 12 up, 12 in 2023-03-09T16:58:21.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:58:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7211.035151 secs 2023-03-09T16:58:21.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:58:20 smithi122 ceph-mon[93869]: pgmap v4359: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 7 op/s 2023-03-09T16:58:21.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:58:20 smithi122 ceph-mon[93869]: osdmap e486: 12 total, 12 up, 12 in 2023-03-09T16:58:21.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:58:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7211.035151 secs 2023-03-09T16:58:21.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:58:20 smithi181 ceph-mon[94189]: pgmap v4359: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 7 op/s 2023-03-09T16:58:21.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:58:20 smithi181 ceph-mon[94189]: osdmap e486: 12 total, 12 up, 12 in 2023-03-09T16:58:23.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:58:22 smithi122 ceph-mon[93869]: pgmap v4361: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2023-03-09T16:58:23.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:58:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7213.285868 secs 2023-03-09T16:58:23.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:58:22 smithi181 ceph-mon[94189]: pgmap v4361: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2023-03-09T16:58:23.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:58:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7213.285868 secs 2023-03-09T16:58:23.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:58:22 smithi038 ceph-mon[92422]: pgmap v4361: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2023-03-09T16:58:23.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:58:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7213.285868 secs 2023-03-09T16:58:23.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7675.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:58:25.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:58:24 smithi122 ceph-mon[93869]: pgmap v4362: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2023-03-09T16:58:25.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:58:24 smithi181 ceph-mon[94189]: pgmap v4362: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2023-03-09T16:58:25.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:58:24 smithi038 ceph-mon[92422]: pgmap v4362: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2023-03-09T16:58:26.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:58:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7216.035214 secs 2023-03-09T16:58:26.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:58:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7216.035214 secs 2023-03-09T16:58:26.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:58:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7216.035214 secs 2023-03-09T16:58:27.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:58:26 smithi122 ceph-mon[93869]: pgmap v4363: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2023-03-09T16:58:27.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:58:26 smithi181 ceph-mon[94189]: pgmap v4363: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2023-03-09T16:58:27.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:58:26 smithi038 ceph-mon[92422]: pgmap v4363: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 2.0 KiB/s wr, 7 op/s 2023-03-09T16:58:28.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:58:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7218.286006 secs 2023-03-09T16:58:28.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:58:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7218.286006 secs 2023-03-09T16:58:28.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:58:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7218.286006 secs 2023-03-09T16:58:28.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7680.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:58:29.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:58:28 smithi122 ceph-mon[93869]: pgmap v4364: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 8 op/s 2023-03-09T16:58:29.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:58:28 smithi181 ceph-mon[94189]: pgmap v4364: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 8 op/s 2023-03-09T16:58:29.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:58:28 smithi038 ceph-mon[92422]: pgmap v4364: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 8 op/s 2023-03-09T16:58:31.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:58:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7221.035281 secs 2023-03-09T16:58:31.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:58:30 smithi122 ceph-mon[93869]: pgmap v4365: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 8 op/s 2023-03-09T16:58:31.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:58:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7221.035281 secs 2023-03-09T16:58:31.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:58:30 smithi181 ceph-mon[94189]: pgmap v4365: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 8 op/s 2023-03-09T16:58:31.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:58:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7221.035281 secs 2023-03-09T16:58:31.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:58:30 smithi038 ceph-mon[92422]: pgmap v4365: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 8 op/s 2023-03-09T16:58:33.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:58:32 smithi122 ceph-mon[93869]: pgmap v4366: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 10 op/s 2023-03-09T16:58:33.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:58:32 smithi181 ceph-mon[94189]: pgmap v4366: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 10 op/s 2023-03-09T16:58:33.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:58:32 smithi038 ceph-mon[92422]: pgmap v4366: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 10 op/s 2023-03-09T16:58:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7685.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:58:35.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:58:34 smithi122 ceph-mon[93869]: pgmap v4367: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 26 KiB/s wr, 7 op/s 2023-03-09T16:58:35.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:58:34 smithi181 ceph-mon[94189]: pgmap v4367: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 26 KiB/s wr, 7 op/s 2023-03-09T16:58:35.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:58:34 smithi038 ceph-mon[92422]: pgmap v4367: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 26 KiB/s wr, 7 op/s 2023-03-09T16:58:36.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:58:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7226.035353 secs 2023-03-09T16:58:36.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:58:35 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:58:36.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:58:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7226.035353 secs 2023-03-09T16:58:36.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:58:35 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:58:36.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:58:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7226.035353 secs 2023-03-09T16:58:36.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:58:35 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T16:58:37.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:58:36 smithi122 ceph-mon[93869]: pgmap v4368: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 26 KiB/s wr, 7 op/s 2023-03-09T16:58:37.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:58:36 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:58:37.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:58:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:58:37.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:58:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:58:37.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:58:36 smithi181 ceph-mon[94189]: pgmap v4368: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 26 KiB/s wr, 7 op/s 2023-03-09T16:58:37.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:58:36 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:58:37.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:58:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:58:37.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:58:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:58:37.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:58:36 smithi038 ceph-mon[92422]: pgmap v4368: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 26 KiB/s wr, 7 op/s 2023-03-09T16:58:37.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:58:36 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:58:37.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:58:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:58:37.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:58:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:58:38.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:58:37 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 7228.286266 secs 2023-03-09T16:58:38.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:58:37 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 7228.286266 secs 2023-03-09T16:58:38.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:58:37 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 7228.286266 secs 2023-03-09T16:58:38.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7690.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:58:39.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:58:38 smithi122 ceph-mon[93869]: pgmap v4369: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 54 KiB/s wr, 11 op/s 2023-03-09T16:58:39.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:58:38 smithi181 ceph-mon[94189]: pgmap v4369: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 54 KiB/s wr, 11 op/s 2023-03-09T16:58:39.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:58:38 smithi038 ceph-mon[92422]: pgmap v4369: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 54 KiB/s wr, 11 op/s 2023-03-09T16:58:41.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:58:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7231.035415 secs 2023-03-09T16:58:41.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:58:40 smithi122 ceph-mon[93869]: pgmap v4370: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 40 KiB/s wr, 8 op/s 2023-03-09T16:58:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:58:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:58:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:58:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:58:41.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:58:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7231.035415 secs 2023-03-09T16:58:41.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:58:40 smithi181 ceph-mon[94189]: pgmap v4370: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 40 KiB/s wr, 8 op/s 2023-03-09T16:58:41.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:58:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:58:41.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:58:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:58:41.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:58:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7231.035415 secs 2023-03-09T16:58:41.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:58:40 smithi038 ceph-mon[92422]: pgmap v4370: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 40 KiB/s wr, 8 op/s 2023-03-09T16:58:41.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:58:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:58:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:58:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:58:43.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:58:42 smithi122 ceph-mon[93869]: pgmap v4371: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 60 KiB/s wr, 11 op/s 2023-03-09T16:58:43.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:58:42 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 7233.286467 secs 2023-03-09T16:58:43.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:58:42 smithi181 ceph-mon[94189]: pgmap v4371: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 60 KiB/s wr, 11 op/s 2023-03-09T16:58:43.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:58:42 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 7233.286467 secs 2023-03-09T16:58:43.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:58:42 smithi038 ceph-mon[92422]: pgmap v4371: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 60 KiB/s wr, 11 op/s 2023-03-09T16:58:43.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:58:42 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 7233.286467 secs 2023-03-09T16:58:43.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7695.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:58:44.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:58:43 smithi122 ceph-mon[93869]: pgmap v4372: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 49 KiB/s wr, 9 op/s 2023-03-09T16:58:44.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:58:43 smithi181 ceph-mon[94189]: pgmap v4372: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 49 KiB/s wr, 9 op/s 2023-03-09T16:58:44.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:58:43 smithi038 ceph-mon[92422]: pgmap v4372: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 49 KiB/s wr, 9 op/s 2023-03-09T16:58:45.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:58:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7236.035475 secs 2023-03-09T16:58:45.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:58:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7236.035475 secs 2023-03-09T16:58:45.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:58:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7236.035475 secs 2023-03-09T16:58:46.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:58:46 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:58:46.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:58:46 smithi122 ceph-mon[93869]: pgmap v4373: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 65 KiB/s wr, 10 op/s 2023-03-09T16:58:46.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:58:46 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:58:46.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:58:46 smithi181 ceph-mon[94189]: pgmap v4373: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 65 KiB/s wr, 10 op/s 2023-03-09T16:58:46.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:58:46 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T16:58:46.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:58:46 smithi038 ceph-mon[92422]: pgmap v4373: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 65 KiB/s wr, 10 op/s 2023-03-09T16:58:47.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:58:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7238.286573 secs 2023-03-09T16:58:47.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:58:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7238.286573 secs 2023-03-09T16:58:47.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:58:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7238.286573 secs 2023-03-09T16:58:48.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:58:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:58:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:58:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:58:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:58:48 smithi181 ceph-mon[94189]: pgmap v4374: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 95 KiB/s wr, 14 op/s 2023-03-09T16:58:48.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:58:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:58:48.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:58:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:58:48.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:58:48 smithi038 ceph-mon[92422]: pgmap v4374: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 95 KiB/s wr, 14 op/s 2023-03-09T16:58:48.724 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:58:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T16:58:48.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:58:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T16:58:48.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:58:48 smithi122 ceph-mon[93869]: pgmap v4374: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 95 KiB/s wr, 14 op/s 2023-03-09T16:58:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7700.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:58:50.974 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:58:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7241.035535 secs 2023-03-09T16:58:50.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:58:50 smithi122 ceph-mon[93869]: pgmap v4375: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 67 KiB/s wr, 10 op/s 2023-03-09T16:58:50.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:58:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7241.035535 secs 2023-03-09T16:58:50.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:58:50 smithi181 ceph-mon[94189]: pgmap v4375: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 67 KiB/s wr, 10 op/s 2023-03-09T16:58:50.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:58:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7241.035535 secs 2023-03-09T16:58:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:58:50 smithi038 ceph-mon[92422]: pgmap v4375: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 67 KiB/s wr, 10 op/s 2023-03-09T16:58:52.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:58:52 smithi181 ceph-mon[94189]: pgmap v4376: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 84 KiB/s wr, 14 op/s 2023-03-09T16:58:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:58:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7243.286708 secs 2023-03-09T16:58:52.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:58:52 smithi038 ceph-mon[92422]: pgmap v4376: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 84 KiB/s wr, 14 op/s 2023-03-09T16:58:52.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:58:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7243.286708 secs 2023-03-09T16:58:53.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:58:52 smithi122 ceph-mon[93869]: pgmap v4376: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 84 KiB/s wr, 14 op/s 2023-03-09T16:58:53.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:58:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7243.286708 secs 2023-03-09T16:58:53.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7705.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:58:54.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:58:54 smithi038 ceph-mon[92422]: pgmap v4377: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 64 KiB/s wr, 11 op/s 2023-03-09T16:58:55.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:58:54 smithi122 ceph-mon[93869]: pgmap v4377: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 64 KiB/s wr, 11 op/s 2023-03-09T16:58:55.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:58:54 smithi181 ceph-mon[94189]: pgmap v4377: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 64 KiB/s wr, 11 op/s 2023-03-09T16:58:55.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:58:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7246.035619 secs 2023-03-09T16:58:56.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:58:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7246.035619 secs 2023-03-09T16:58:56.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:58:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7246.035619 secs 2023-03-09T16:58:56.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:58:56 smithi038 ceph-mon[92422]: pgmap v4378: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 82 KiB/s wr, 11 op/s 2023-03-09T16:58:57.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:58:56 smithi122 ceph-mon[93869]: pgmap v4378: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 82 KiB/s wr, 11 op/s 2023-03-09T16:58:57.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:58:56 smithi181 ceph-mon[94189]: pgmap v4378: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 82 KiB/s wr, 11 op/s 2023-03-09T16:58:58.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:58:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7248.286851 secs 2023-03-09T16:58:58.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:58:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7248.286851 secs 2023-03-09T16:58:58.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:58:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7248.286851 secs 2023-03-09T16:58:58.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7710.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:58:59.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:58:58 smithi122 ceph-mon[93869]: pgmap v4379: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 66 KiB/s wr, 13 op/s 2023-03-09T16:58:59.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:58:58 smithi181 ceph-mon[94189]: pgmap v4379: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 66 KiB/s wr, 13 op/s 2023-03-09T16:58:59.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:58:58 smithi038 ceph-mon[92422]: pgmap v4379: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 66 KiB/s wr, 13 op/s 2023-03-09T16:59:00.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:58:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:59:00.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:58:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:59:00.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:58:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:59:00.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:58:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:59:00.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:58:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:59:00.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:58:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:59:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:58:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:59:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:58:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:59:00.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:58:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T16:59:00.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:58:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T16:59:00.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:58:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T16:59:00.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:58:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T16:59:01.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:59:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7251.035680 secs 2023-03-09T16:59:01.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:59:00 smithi122 ceph-mon[93869]: pgmap v4380: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 KiB/s wr, 8 op/s 2023-03-09T16:59:01.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:59:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7251.035680 secs 2023-03-09T16:59:01.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:59:00 smithi181 ceph-mon[94189]: pgmap v4380: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 KiB/s wr, 8 op/s 2023-03-09T16:59:01.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:59:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7251.035680 secs 2023-03-09T16:59:01.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:59:00 smithi038 ceph-mon[92422]: pgmap v4380: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 KiB/s wr, 8 op/s 2023-03-09T16:59:03.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:59:02 smithi122 ceph-mon[93869]: pgmap v4381: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 46 KiB/s wr, 11 op/s 2023-03-09T16:59:03.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:59:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7253.286982 secs 2023-03-09T16:59:03.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:59:02 smithi181 ceph-mon[94189]: pgmap v4381: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 46 KiB/s wr, 11 op/s 2023-03-09T16:59:03.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:59:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7253.286982 secs 2023-03-09T16:59:03.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:59:02 smithi038 ceph-mon[92422]: pgmap v4381: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 46 KiB/s wr, 11 op/s 2023-03-09T16:59:03.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:59:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7253.286982 secs 2023-03-09T16:59:03.793 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7715.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:59:05.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:59:04 smithi122 ceph-mon[93869]: pgmap v4382: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 50 KiB/s wr, 19 op/s 2023-03-09T16:59:05.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:59:04 smithi181 ceph-mon[94189]: pgmap v4382: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 50 KiB/s wr, 19 op/s 2023-03-09T16:59:05.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:59:04 smithi038 ceph-mon[92422]: pgmap v4382: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 50 KiB/s wr, 19 op/s 2023-03-09T16:59:06.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:59:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7256.035748 secs 2023-03-09T16:59:06.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:59:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7256.035748 secs 2023-03-09T16:59:06.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:59:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7256.035748 secs 2023-03-09T16:59:07.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:59:06 smithi122 ceph-mon[93869]: pgmap v4383: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 74 KiB/s wr, 22 op/s 2023-03-09T16:59:07.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:59:06 smithi181 ceph-mon[94189]: pgmap v4383: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 74 KiB/s wr, 22 op/s 2023-03-09T16:59:07.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:59:06 smithi038 ceph-mon[92422]: pgmap v4383: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 74 KiB/s wr, 22 op/s 2023-03-09T16:59:08.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:59:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7258.287134 secs 2023-03-09T16:59:08.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:59:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7258.287134 secs 2023-03-09T16:59:08.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:59:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7258.287134 secs 2023-03-09T16:59:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7720.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:59:09.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:59:08 smithi122 ceph-mon[93869]: pgmap v4384: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 62 KiB/s wr, 24 op/s 2023-03-09T16:59:09.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:59:08 smithi181 ceph-mon[94189]: pgmap v4384: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 62 KiB/s wr, 24 op/s 2023-03-09T16:59:09.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:59:08 smithi038 ceph-mon[92422]: pgmap v4384: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 62 KiB/s wr, 24 op/s 2023-03-09T16:59:10.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:59:09 smithi122 ceph-mon[93869]: osdmap e487: 12 total, 12 up, 12 in 2023-03-09T16:59:10.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:59:09 smithi181 ceph-mon[94189]: osdmap e487: 12 total, 12 up, 12 in 2023-03-09T16:59:10.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:59:09 smithi038 ceph-mon[92422]: osdmap e487: 12 total, 12 up, 12 in 2023-03-09T16:59:11.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:59:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7261.035808 secs 2023-03-09T16:59:11.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:59:10 smithi122 ceph-mon[93869]: pgmap v4386: 129 pgs: 2 active+clean+snaptrim, 127 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 74 KiB/s wr, 25 op/s 2023-03-09T16:59:11.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:59:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7261.035808 secs 2023-03-09T16:59:11.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:59:10 smithi181 ceph-mon[94189]: pgmap v4386: 129 pgs: 2 active+clean+snaptrim, 127 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 74 KiB/s wr, 25 op/s 2023-03-09T16:59:11.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:59:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7261.035808 secs 2023-03-09T16:59:11.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:59:10 smithi038 ceph-mon[92422]: pgmap v4386: 129 pgs: 2 active+clean+snaptrim, 127 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 74 KiB/s wr, 25 op/s 2023-03-09T16:59:13.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:59:12 smithi122 ceph-mon[93869]: pgmap v4387: 129 pgs: 2 active+clean+snaptrim, 127 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 80 KiB/s wr, 25 op/s 2023-03-09T16:59:13.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:59:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7263.287251 secs 2023-03-09T16:59:13.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:59:12 smithi181 ceph-mon[94189]: pgmap v4387: 129 pgs: 2 active+clean+snaptrim, 127 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 80 KiB/s wr, 25 op/s 2023-03-09T16:59:13.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:59:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7263.287251 secs 2023-03-09T16:59:13.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:59:12 smithi038 ceph-mon[92422]: pgmap v4387: 129 pgs: 2 active+clean+snaptrim, 127 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 80 KiB/s wr, 25 op/s 2023-03-09T16:59:13.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:59:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7263.287251 secs 2023-03-09T16:59:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7725.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:59:15.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:59:14 smithi122 ceph-mon[93869]: pgmap v4388: 129 pgs: 2 active+clean+snaptrim, 127 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 54 KiB/s wr, 11 op/s 2023-03-09T16:59:15.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:59:14 smithi181 ceph-mon[94189]: pgmap v4388: 129 pgs: 2 active+clean+snaptrim, 127 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 54 KiB/s wr, 11 op/s 2023-03-09T16:59:15.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:59:14 smithi038 ceph-mon[92422]: pgmap v4388: 129 pgs: 2 active+clean+snaptrim, 127 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 54 KiB/s wr, 11 op/s 2023-03-09T16:59:16.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:59:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7266.035868 secs 2023-03-09T16:59:16.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:59:15 smithi122 ceph-mon[93869]: osdmap e488: 12 total, 12 up, 12 in 2023-03-09T16:59:16.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:59:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7266.035868 secs 2023-03-09T16:59:16.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:59:15 smithi181 ceph-mon[94189]: osdmap e488: 12 total, 12 up, 12 in 2023-03-09T16:59:16.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:59:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7266.035868 secs 2023-03-09T16:59:16.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:59:15 smithi038 ceph-mon[92422]: osdmap e488: 12 total, 12 up, 12 in 2023-03-09T16:59:17.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:59:16 smithi122 ceph-mon[93869]: pgmap v4390: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 42 KiB/s wr, 7 op/s 2023-03-09T16:59:17.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:59:16 smithi181 ceph-mon[94189]: pgmap v4390: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 42 KiB/s wr, 7 op/s 2023-03-09T16:59:17.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:59:16 smithi038 ceph-mon[92422]: pgmap v4390: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 42 KiB/s wr, 7 op/s 2023-03-09T16:59:18.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:59:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7268.287355 secs 2023-03-09T16:59:18.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:59:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7268.287355 secs 2023-03-09T16:59:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:59:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7268.287355 secs 2023-03-09T16:59:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7730.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:59:19.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:59:18 smithi122 ceph-mon[93869]: pgmap v4391: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 39 KiB/s wr, 9 op/s 2023-03-09T16:59:19.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:59:18 smithi181 ceph-mon[94189]: pgmap v4391: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 39 KiB/s wr, 9 op/s 2023-03-09T16:59:19.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:59:18 smithi038 ceph-mon[92422]: pgmap v4391: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 39 KiB/s wr, 9 op/s 2023-03-09T16:59:21.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:59:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7271.035929 secs 2023-03-09T16:59:21.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:59:20 smithi122 ceph-mon[93869]: pgmap v4392: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 34 KiB/s wr, 8 op/s 2023-03-09T16:59:21.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:59:20 smithi122 ceph-mon[93869]: osdmap e489: 12 total, 12 up, 12 in 2023-03-09T16:59:21.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:59:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7271.035929 secs 2023-03-09T16:59:21.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:59:20 smithi181 ceph-mon[94189]: pgmap v4392: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 34 KiB/s wr, 8 op/s 2023-03-09T16:59:21.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:59:20 smithi181 ceph-mon[94189]: osdmap e489: 12 total, 12 up, 12 in 2023-03-09T16:59:21.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:59:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7271.035929 secs 2023-03-09T16:59:21.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:59:20 smithi038 ceph-mon[92422]: pgmap v4392: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 34 KiB/s wr, 8 op/s 2023-03-09T16:59:21.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:59:20 smithi038 ceph-mon[92422]: osdmap e489: 12 total, 12 up, 12 in 2023-03-09T16:59:23.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:59:22 smithi122 ceph-mon[93869]: pgmap v4394: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 9 op/s 2023-03-09T16:59:23.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:59:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7273.287474 secs 2023-03-09T16:59:23.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:59:22 smithi181 ceph-mon[94189]: pgmap v4394: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 9 op/s 2023-03-09T16:59:23.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:59:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7273.287474 secs 2023-03-09T16:59:23.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:59:22 smithi038 ceph-mon[92422]: pgmap v4394: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 9 op/s 2023-03-09T16:59:23.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:59:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7273.287474 secs 2023-03-09T16:59:23.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7735.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:59:25.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:59:24 smithi122 ceph-mon[93869]: pgmap v4395: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 21 KiB/s wr, 9 op/s 2023-03-09T16:59:25.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:59:24 smithi181 ceph-mon[94189]: pgmap v4395: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 21 KiB/s wr, 9 op/s 2023-03-09T16:59:25.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:59:24 smithi038 ceph-mon[92422]: pgmap v4395: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 21 KiB/s wr, 9 op/s 2023-03-09T16:59:26.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:59:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7276.035989 secs 2023-03-09T16:59:26.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:59:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7276.035989 secs 2023-03-09T16:59:26.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:59:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7276.035989 secs 2023-03-09T16:59:27.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:59:26 smithi122 ceph-mon[93869]: pgmap v4396: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 28 KiB/s wr, 8 op/s 2023-03-09T16:59:27.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:59:26 smithi181 ceph-mon[94189]: pgmap v4396: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 28 KiB/s wr, 8 op/s 2023-03-09T16:59:27.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:59:26 smithi038 ceph-mon[92422]: pgmap v4396: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 28 KiB/s wr, 8 op/s 2023-03-09T16:59:28.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:59:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7278.287604 secs 2023-03-09T16:59:28.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:59:27 smithi122 ceph-mon[93869]: pgmap v4397: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T16:59:28.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:59:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7278.287604 secs 2023-03-09T16:59:28.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:59:27 smithi181 ceph-mon[94189]: pgmap v4397: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T16:59:28.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:59:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7278.287604 secs 2023-03-09T16:59:28.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:59:27 smithi038 ceph-mon[92422]: pgmap v4397: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T16:59:28.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7740.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:59:30.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:59:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7281.036047 secs 2023-03-09T16:59:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:59:30 smithi038 ceph-mon[92422]: pgmap v4398: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T16:59:31.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:59:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7281.036047 secs 2023-03-09T16:59:31.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:59:30 smithi122 ceph-mon[93869]: pgmap v4398: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T16:59:31.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:59:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7281.036047 secs 2023-03-09T16:59:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:59:30 smithi181 ceph-mon[94189]: pgmap v4398: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T16:59:32.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:59:32 smithi038 ceph-mon[92422]: pgmap v4399: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 25 KiB/s wr, 9 op/s 2023-03-09T16:59:32.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:59:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7283.287755 secs 2023-03-09T16:59:33.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:59:32 smithi122 ceph-mon[93869]: pgmap v4399: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 25 KiB/s wr, 9 op/s 2023-03-09T16:59:33.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:59:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7283.287755 secs 2023-03-09T16:59:33.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:59:32 smithi181 ceph-mon[94189]: pgmap v4399: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 25 KiB/s wr, 9 op/s 2023-03-09T16:59:33.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:59:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7283.287755 secs 2023-03-09T16:59:33.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7745.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:59:35.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:59:34 smithi122 ceph-mon[93869]: pgmap v4400: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 6 op/s 2023-03-09T16:59:35.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:59:34 smithi181 ceph-mon[94189]: pgmap v4400: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 6 op/s 2023-03-09T16:59:35.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:59:34 smithi038 ceph-mon[92422]: pgmap v4400: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 6 op/s 2023-03-09T16:59:36.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:59:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7286.036123 secs 2023-03-09T16:59:36.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:59:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7286.036123 secs 2023-03-09T16:59:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:59:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7286.036123 secs 2023-03-09T16:59:37.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:59:36 smithi122 ceph-mon[93869]: pgmap v4401: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 34 KiB/s wr, 7 op/s 2023-03-09T16:59:37.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:59:36 smithi181 ceph-mon[94189]: pgmap v4401: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 34 KiB/s wr, 7 op/s 2023-03-09T16:59:37.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:59:36 smithi038 ceph-mon[92422]: pgmap v4401: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 34 KiB/s wr, 7 op/s 2023-03-09T16:59:38.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:59:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7288.287891 secs 2023-03-09T16:59:38.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:59:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7288.287891 secs 2023-03-09T16:59:38.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:59:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7288.287891 secs 2023-03-09T16:59:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7750.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:59:39.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:59:38 smithi122 ceph-mon[93869]: pgmap v4402: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 11 KiB/s wr, 7 op/s 2023-03-09T16:59:39.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:59:38 smithi181 ceph-mon[94189]: pgmap v4402: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 11 KiB/s wr, 7 op/s 2023-03-09T16:59:39.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:59:38 smithi038 ceph-mon[92422]: pgmap v4402: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 11 KiB/s wr, 7 op/s 2023-03-09T16:59:41.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:59:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7291.036184 secs 2023-03-09T16:59:41.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:59:40 smithi122 ceph-mon[93869]: pgmap v4403: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 6 op/s 2023-03-09T16:59:41.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:59:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7291.036184 secs 2023-03-09T16:59:41.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:59:40 smithi181 ceph-mon[94189]: pgmap v4403: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 6 op/s 2023-03-09T16:59:41.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:59:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7291.036184 secs 2023-03-09T16:59:41.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:59:40 smithi038 ceph-mon[92422]: pgmap v4403: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 6 op/s 2023-03-09T16:59:43.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:59:42 smithi122 ceph-mon[93869]: pgmap v4404: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 43 KiB/s wr, 10 op/s 2023-03-09T16:59:43.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:59:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7293.287966 secs 2023-03-09T16:59:43.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:59:42 smithi181 ceph-mon[94189]: pgmap v4404: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 43 KiB/s wr, 10 op/s 2023-03-09T16:59:43.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:59:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7293.287966 secs 2023-03-09T16:59:43.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:59:42 smithi038 ceph-mon[92422]: pgmap v4404: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 43 KiB/s wr, 10 op/s 2023-03-09T16:59:43.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:59:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7293.287966 secs 2023-03-09T16:59:43.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7755.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:59:45.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:59:44 smithi122 ceph-mon[93869]: pgmap v4405: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 42 KiB/s wr, 8 op/s 2023-03-09T16:59:45.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:59:44 smithi181 ceph-mon[94189]: pgmap v4405: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 42 KiB/s wr, 8 op/s 2023-03-09T16:59:45.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:59:44 smithi038 ceph-mon[92422]: pgmap v4405: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 42 KiB/s wr, 8 op/s 2023-03-09T16:59:46.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:59:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7296.036244 secs 2023-03-09T16:59:46.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:59:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7296.036244 secs 2023-03-09T16:59:46.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:59:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7296.036244 secs 2023-03-09T16:59:47.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:59:46 smithi122 ceph-mon[93869]: pgmap v4406: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 44 KiB/s wr, 9 op/s 2023-03-09T16:59:47.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:59:46 smithi181 ceph-mon[94189]: pgmap v4406: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 44 KiB/s wr, 9 op/s 2023-03-09T16:59:47.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:59:46 smithi038 ceph-mon[92422]: pgmap v4406: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 44 KiB/s wr, 9 op/s 2023-03-09T16:59:48.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:59:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7298.288072 secs 2023-03-09T16:59:48.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:59:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7298.288072 secs 2023-03-09T16:59:48.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:59:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7298.288072 secs 2023-03-09T16:59:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7760.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:59:49.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:59:48 smithi122 ceph-mon[93869]: pgmap v4407: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 34 KiB/s wr, 10 op/s 2023-03-09T16:59:49.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:59:48 smithi181 ceph-mon[94189]: pgmap v4407: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 34 KiB/s wr, 10 op/s 2023-03-09T16:59:49.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:59:48 smithi038 ceph-mon[92422]: pgmap v4407: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 34 KiB/s wr, 10 op/s 2023-03-09T16:59:51.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:59:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7301.036303 secs 2023-03-09T16:59:51.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:59:50 smithi122 ceph-mon[93869]: pgmap v4408: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 34 KiB/s wr, 8 op/s 2023-03-09T16:59:51.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:59:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7301.036303 secs 2023-03-09T16:59:51.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:59:50 smithi181 ceph-mon[94189]: pgmap v4408: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 34 KiB/s wr, 8 op/s 2023-03-09T16:59:51.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:59:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7301.036303 secs 2023-03-09T16:59:51.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:59:50 smithi038 ceph-mon[92422]: pgmap v4408: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 34 KiB/s wr, 8 op/s 2023-03-09T16:59:53.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:59:52 smithi122 ceph-mon[93869]: pgmap v4409: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 86 KiB/s wr, 12 op/s 2023-03-09T16:59:53.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:59:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7303.288197 secs 2023-03-09T16:59:53.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:59:52 smithi181 ceph-mon[94189]: pgmap v4409: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 86 KiB/s wr, 12 op/s 2023-03-09T16:59:53.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:59:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7303.288197 secs 2023-03-09T16:59:53.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:59:52 smithi038 ceph-mon[92422]: pgmap v4409: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 86 KiB/s wr, 12 op/s 2023-03-09T16:59:53.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:59:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7303.288197 secs 2023-03-09T16:59:53.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7765.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:59:55.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:59:54 smithi122 ceph-mon[93869]: pgmap v4410: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 54 KiB/s wr, 9 op/s 2023-03-09T16:59:55.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:59:54 smithi181 ceph-mon[94189]: pgmap v4410: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 54 KiB/s wr, 9 op/s 2023-03-09T16:59:55.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:59:54 smithi038 ceph-mon[92422]: pgmap v4410: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 54 KiB/s wr, 9 op/s 2023-03-09T16:59:56.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:59:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7306.036368 secs 2023-03-09T16:59:56.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:59:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7306.036368 secs 2023-03-09T16:59:56.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:59:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7306.036368 secs 2023-03-09T16:59:57.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:59:56 smithi122 ceph-mon[93869]: pgmap v4411: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 69 KiB/s wr, 10 op/s 2023-03-09T16:59:57.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:59:56 smithi181 ceph-mon[94189]: pgmap v4411: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 69 KiB/s wr, 10 op/s 2023-03-09T16:59:57.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:59:56 smithi038 ceph-mon[92422]: pgmap v4411: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 69 KiB/s wr, 10 op/s 2023-03-09T16:59:58.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:59:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7308.288314 secs 2023-03-09T16:59:58.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:59:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7308.288314 secs 2023-03-09T16:59:58.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:59:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7308.288314 secs 2023-03-09T16:59:58.793 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7770.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T16:59:59.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 16:59:58 smithi122 ceph-mon[93869]: pgmap v4412: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 75 KiB/s wr, 11 op/s 2023-03-09T16:59:59.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 16:59:58 smithi181 ceph-mon[94189]: pgmap v4412: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 75 KiB/s wr, 11 op/s 2023-03-09T16:59:59.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 16:59:58 smithi038 ceph-mon[92422]: pgmap v4412: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 75 KiB/s wr, 11 op/s 2023-03-09T17:00:00.810 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7311.036427 secs 2023-03-09T17:00:00.810 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:00 smithi122 ceph-mon[93869]: pgmap v4413: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 75 KiB/s wr, 9 op/s 2023-03-09T17:00:00.810 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:00:00.810 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:00 smithi122 ceph-mon[93869]: overall HEALTH_WARN 2 MDSs report slow requests 2023-03-09T17:00:01.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7311.036427 secs 2023-03-09T17:00:01.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:00 smithi181 ceph-mon[94189]: pgmap v4413: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 75 KiB/s wr, 9 op/s 2023-03-09T17:00:01.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:00:01.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:00 smithi181 ceph-mon[94189]: overall HEALTH_WARN 2 MDSs report slow requests 2023-03-09T17:00:01.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7311.036427 secs 2023-03-09T17:00:01.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:00 smithi038 ceph-mon[92422]: pgmap v4413: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 75 KiB/s wr, 9 op/s 2023-03-09T17:00:01.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:00:01.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:00 smithi038 ceph-mon[92422]: overall HEALTH_WARN 2 MDSs report slow requests 2023-03-09T17:00:03.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:02 smithi122 ceph-mon[93869]: pgmap v4414: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 83 KiB/s wr, 12 op/s 2023-03-09T17:00:03.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7313.288395 secs 2023-03-09T17:00:03.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:00:03.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:00:03.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:00:03.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:00:03.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:02 smithi181 ceph-mon[94189]: pgmap v4414: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 83 KiB/s wr, 12 op/s 2023-03-09T17:00:03.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7313.288395 secs 2023-03-09T17:00:03.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:00:03.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:00:03.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:00:03.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:00:03.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:02 smithi038 ceph-mon[92422]: pgmap v4414: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 83 KiB/s wr, 12 op/s 2023-03-09T17:00:03.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7313.288395 secs 2023-03-09T17:00:03.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:00:03.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:00:03.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:00:03.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:00:03.793 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7775.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:00:04.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:03 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:00:04.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:03 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:00:04.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:03 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:00:04.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:03 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:00:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:03 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:00:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:03 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:00:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:03 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:00:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:03 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:00:04.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:03 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:00:04.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:03 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:00:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:03 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:00:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:03 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:00:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:03 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:00:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:03 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:00:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:03 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:00:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:03 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:00:04.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:03 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:00:04.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:03 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:00:04.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:03 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:00:04.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:03 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:00:04.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:03 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:00:04.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:03 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:00:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:03 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:00:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:03 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:00:05.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:04 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi038 to 197.2M 2023-03-09T17:00:05.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:04 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi038 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-03-09T17:00:05.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:04 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T17:00:05.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:04 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T17:00:05.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:04 smithi122 ceph-mon[93869]: pgmap v4415: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 53 KiB/s wr, 20 op/s 2023-03-09T17:00:05.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:04 smithi122 ceph-mon[93869]: osdmap e490: 12 total, 12 up, 12 in 2023-03-09T17:00:05.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:04 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi038 to 197.2M 2023-03-09T17:00:05.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:04 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi038 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-03-09T17:00:05.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:04 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T17:00:05.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:04 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T17:00:05.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:04 smithi181 ceph-mon[94189]: pgmap v4415: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 53 KiB/s wr, 20 op/s 2023-03-09T17:00:05.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:04 smithi181 ceph-mon[94189]: osdmap e490: 12 total, 12 up, 12 in 2023-03-09T17:00:05.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:04 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi038 to 197.2M 2023-03-09T17:00:05.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:04 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi038 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-03-09T17:00:05.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:04 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T17:00:05.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:04 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T17:00:05.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:04 smithi038 ceph-mon[92422]: pgmap v4415: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 53 KiB/s wr, 20 op/s 2023-03-09T17:00:05.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:04 smithi038 ceph-mon[92422]: osdmap e490: 12 total, 12 up, 12 in 2023-03-09T17:00:06.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7316.036490 secs 2023-03-09T17:00:06.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:05 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:00:06.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:05 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:00:06.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7316.036490 secs 2023-03-09T17:00:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:05 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:00:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:05 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:00:06.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7316.036490 secs 2023-03-09T17:00:06.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:05 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:00:06.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:05 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:00:07.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:06 smithi122 ceph-mon[93869]: pgmap v4417: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 109 KiB/s wr, 27 op/s 2023-03-09T17:00:07.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:06 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:00:07.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:06 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:00:07.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:06 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:00:07.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:06 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:00:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:06 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:00:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:06 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:00:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:06 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:00:07.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:06 smithi181 ceph-mon[94189]: pgmap v4417: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 109 KiB/s wr, 27 op/s 2023-03-09T17:00:07.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:06 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:00:07.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:06 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:00:07.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:06 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:00:07.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:06 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:00:07.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:06 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:00:07.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:06 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:00:07.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:06 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:00:07.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:06 smithi038 ceph-mon[92422]: pgmap v4417: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 109 KiB/s wr, 27 op/s 2023-03-09T17:00:07.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:06 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:00:07.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:06 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:00:07.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:06 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:00:07.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:06 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:00:07.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:06 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:00:07.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:06 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:00:07.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:06 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:00:08.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:07 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T17:00:08.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:07 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T17:00:08.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7318.288534 secs 2023-03-09T17:00:08.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:07 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T17:00:08.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:07 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T17:00:08.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7318.288534 secs 2023-03-09T17:00:08.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:07 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T17:00:08.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:07 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T17:00:08.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7318.288534 secs 2023-03-09T17:00:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7780.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:00:09.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:08 smithi122 ceph-mon[93869]: pgmap v4418: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 116 KiB/s wr, 27 op/s 2023-03-09T17:00:09.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:08 smithi181 ceph-mon[94189]: pgmap v4418: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 116 KiB/s wr, 27 op/s 2023-03-09T17:00:09.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:08 smithi038 ceph-mon[92422]: pgmap v4418: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 116 KiB/s wr, 27 op/s 2023-03-09T17:00:10.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:09 smithi122 ceph-mon[93869]: pgmap v4419: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 115 KiB/s wr, 27 op/s 2023-03-09T17:00:10.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:09 smithi181 ceph-mon[94189]: pgmap v4419: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 115 KiB/s wr, 27 op/s 2023-03-09T17:00:10.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:09 smithi038 ceph-mon[92422]: pgmap v4419: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 115 KiB/s wr, 27 op/s 2023-03-09T17:00:11.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7321.036549 secs 2023-03-09T17:00:11.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:10 smithi122 ceph-mon[93869]: osdmap e491: 12 total, 12 up, 12 in 2023-03-09T17:00:11.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7321.036549 secs 2023-03-09T17:00:11.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:10 smithi181 ceph-mon[94189]: osdmap e491: 12 total, 12 up, 12 in 2023-03-09T17:00:11.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7321.036549 secs 2023-03-09T17:00:11.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:10 smithi038 ceph-mon[92422]: osdmap e491: 12 total, 12 up, 12 in 2023-03-09T17:00:12.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:11 smithi122 ceph-mon[93869]: pgmap v4421: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 114 KiB/s wr, 15 op/s 2023-03-09T17:00:12.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:11 smithi181 ceph-mon[94189]: pgmap v4421: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 114 KiB/s wr, 15 op/s 2023-03-09T17:00:12.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:11 smithi038 ceph-mon[92422]: pgmap v4421: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 114 KiB/s wr, 15 op/s 2023-03-09T17:00:13.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7323.288656 secs 2023-03-09T17:00:13.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7323.288656 secs 2023-03-09T17:00:13.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7323.288656 secs 2023-03-09T17:00:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7785.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:00:14.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:13 smithi122 ceph-mon[93869]: pgmap v4422: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 92 KiB/s wr, 12 op/s 2023-03-09T17:00:14.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:13 smithi181 ceph-mon[94189]: pgmap v4422: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 92 KiB/s wr, 12 op/s 2023-03-09T17:00:14.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:13 smithi038 ceph-mon[92422]: pgmap v4422: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 92 KiB/s wr, 12 op/s 2023-03-09T17:00:15.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7326.036632 secs 2023-03-09T17:00:15.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7326.036632 secs 2023-03-09T17:00:15.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7326.036632 secs 2023-03-09T17:00:16.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:16 smithi122 ceph-mon[93869]: pgmap v4423: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T17:00:16.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:16 smithi181 ceph-mon[94189]: pgmap v4423: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T17:00:16.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:16 smithi038 ceph-mon[92422]: pgmap v4423: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T17:00:17.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7328.288807 secs 2023-03-09T17:00:17.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7328.288807 secs 2023-03-09T17:00:17.724 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7328.288807 secs 2023-03-09T17:00:18.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:18 smithi038 ceph-mon[92422]: pgmap v4424: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 7 op/s 2023-03-09T17:00:18.724 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:18 smithi122 ceph-mon[93869]: pgmap v4424: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 7 op/s 2023-03-09T17:00:18.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:18 smithi181 ceph-mon[94189]: pgmap v4424: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 7 op/s 2023-03-09T17:00:18.793 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7790.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:00:21.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7331.036692 secs 2023-03-09T17:00:21.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:20 smithi122 ceph-mon[93869]: pgmap v4425: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 7 op/s 2023-03-09T17:00:21.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7331.036692 secs 2023-03-09T17:00:21.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:20 smithi181 ceph-mon[94189]: pgmap v4425: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 7 op/s 2023-03-09T17:00:21.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7331.036692 secs 2023-03-09T17:00:21.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:20 smithi038 ceph-mon[92422]: pgmap v4425: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 7 op/s 2023-03-09T17:00:23.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:22 smithi122 ceph-mon[93869]: pgmap v4426: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 11 KiB/s wr, 8 op/s 2023-03-09T17:00:23.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7333.288878 secs 2023-03-09T17:00:23.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:22 smithi181 ceph-mon[94189]: pgmap v4426: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 11 KiB/s wr, 8 op/s 2023-03-09T17:00:23.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7333.288878 secs 2023-03-09T17:00:23.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:22 smithi038 ceph-mon[92422]: pgmap v4426: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 11 KiB/s wr, 8 op/s 2023-03-09T17:00:23.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7333.288878 secs 2023-03-09T17:00:23.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7795.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:00:25.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:24 smithi122 ceph-mon[93869]: pgmap v4427: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2023-03-09T17:00:25.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:24 smithi181 ceph-mon[94189]: pgmap v4427: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2023-03-09T17:00:25.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:24 smithi038 ceph-mon[92422]: pgmap v4427: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2023-03-09T17:00:26.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7336.036751 secs 2023-03-09T17:00:26.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7336.036751 secs 2023-03-09T17:00:26.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7336.036751 secs 2023-03-09T17:00:27.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:26 smithi122 ceph-mon[93869]: pgmap v4428: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 10 KiB/s wr, 6 op/s 2023-03-09T17:00:27.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:26 smithi181 ceph-mon[94189]: pgmap v4428: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 10 KiB/s wr, 6 op/s 2023-03-09T17:00:27.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:26 smithi038 ceph-mon[92422]: pgmap v4428: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 10 KiB/s wr, 6 op/s 2023-03-09T17:00:28.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7338.288972 secs 2023-03-09T17:00:28.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7338.288972 secs 2023-03-09T17:00:28.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7338.288972 secs 2023-03-09T17:00:28.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7800.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:00:29.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:28 smithi122 ceph-mon[93869]: pgmap v4429: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 10 KiB/s wr, 7 op/s 2023-03-09T17:00:29.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:28 smithi181 ceph-mon[94189]: pgmap v4429: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 10 KiB/s wr, 7 op/s 2023-03-09T17:00:29.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:28 smithi038 ceph-mon[92422]: pgmap v4429: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 10 KiB/s wr, 7 op/s 2023-03-09T17:00:31.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7341.036813 secs 2023-03-09T17:00:31.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:30 smithi122 ceph-mon[93869]: pgmap v4430: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 10 KiB/s wr, 5 op/s 2023-03-09T17:00:31.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7341.036813 secs 2023-03-09T17:00:31.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:30 smithi181 ceph-mon[94189]: pgmap v4430: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 10 KiB/s wr, 5 op/s 2023-03-09T17:00:31.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7341.036813 secs 2023-03-09T17:00:31.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:30 smithi038 ceph-mon[92422]: pgmap v4430: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 10 KiB/s wr, 5 op/s 2023-03-09T17:00:33.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:32 smithi122 ceph-mon[93869]: pgmap v4431: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 10 KiB/s wr, 8 op/s 2023-03-09T17:00:33.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7343.289084 secs 2023-03-09T17:00:33.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:32 smithi181 ceph-mon[94189]: pgmap v4431: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 10 KiB/s wr, 8 op/s 2023-03-09T17:00:33.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7343.289084 secs 2023-03-09T17:00:33.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:32 smithi038 ceph-mon[92422]: pgmap v4431: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 10 KiB/s wr, 8 op/s 2023-03-09T17:00:33.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7343.289084 secs 2023-03-09T17:00:33.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7805.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:00:35.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:34 smithi122 ceph-mon[93869]: pgmap v4432: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 10 KiB/s wr, 6 op/s 2023-03-09T17:00:35.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:34 smithi181 ceph-mon[94189]: pgmap v4432: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 10 KiB/s wr, 6 op/s 2023-03-09T17:00:35.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:34 smithi038 ceph-mon[92422]: pgmap v4432: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 10 KiB/s wr, 6 op/s 2023-03-09T17:00:36.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7346.036874 secs 2023-03-09T17:00:36.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7346.036874 secs 2023-03-09T17:00:36.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7346.036874 secs 2023-03-09T17:00:37.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:36 smithi122 ceph-mon[93869]: pgmap v4433: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 21 KiB/s wr, 6 op/s 2023-03-09T17:00:37.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:36 smithi181 ceph-mon[94189]: pgmap v4433: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 21 KiB/s wr, 6 op/s 2023-03-09T17:00:37.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:36 smithi038 ceph-mon[92422]: pgmap v4433: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 21 KiB/s wr, 6 op/s 2023-03-09T17:00:38.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7348.289206 secs 2023-03-09T17:00:38.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7348.289206 secs 2023-03-09T17:00:38.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7348.289206 secs 2023-03-09T17:00:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7810.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:00:39.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:38 smithi122 ceph-mon[93869]: pgmap v4434: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 11 KiB/s wr, 7 op/s 2023-03-09T17:00:39.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:38 smithi181 ceph-mon[94189]: pgmap v4434: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 11 KiB/s wr, 7 op/s 2023-03-09T17:00:39.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:38 smithi038 ceph-mon[92422]: pgmap v4434: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 11 KiB/s wr, 7 op/s 2023-03-09T17:00:41.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7351.036933 secs 2023-03-09T17:00:41.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:40 smithi122 ceph-mon[93869]: pgmap v4435: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 11 KiB/s wr, 6 op/s 2023-03-09T17:00:41.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7351.036933 secs 2023-03-09T17:00:41.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:40 smithi181 ceph-mon[94189]: pgmap v4435: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 11 KiB/s wr, 6 op/s 2023-03-09T17:00:41.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7351.036933 secs 2023-03-09T17:00:41.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:40 smithi038 ceph-mon[92422]: pgmap v4435: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 11 KiB/s wr, 6 op/s 2023-03-09T17:00:43.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:42 smithi122 ceph-mon[93869]: pgmap v4436: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 37 KiB/s wr, 9 op/s 2023-03-09T17:00:43.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7353.289321 secs 2023-03-09T17:00:43.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:42 smithi181 ceph-mon[94189]: pgmap v4436: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 37 KiB/s wr, 9 op/s 2023-03-09T17:00:43.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7353.289321 secs 2023-03-09T17:00:43.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:42 smithi038 ceph-mon[92422]: pgmap v4436: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 37 KiB/s wr, 9 op/s 2023-03-09T17:00:43.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7353.289321 secs 2023-03-09T17:00:43.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7815.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:00:45.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:44 smithi122 ceph-mon[93869]: pgmap v4437: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 37 KiB/s wr, 6 op/s 2023-03-09T17:00:45.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:44 smithi181 ceph-mon[94189]: pgmap v4437: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 37 KiB/s wr, 6 op/s 2023-03-09T17:00:45.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:44 smithi038 ceph-mon[92422]: pgmap v4437: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 37 KiB/s wr, 6 op/s 2023-03-09T17:00:46.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7356.036987 secs 2023-03-09T17:00:46.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7356.036987 secs 2023-03-09T17:00:46.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7356.036987 secs 2023-03-09T17:00:47.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:46 smithi122 ceph-mon[93869]: pgmap v4438: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 40 KiB/s wr, 8 op/s 2023-03-09T17:00:47.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:46 smithi181 ceph-mon[94189]: pgmap v4438: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 40 KiB/s wr, 8 op/s 2023-03-09T17:00:47.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:46 smithi038 ceph-mon[92422]: pgmap v4438: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 40 KiB/s wr, 8 op/s 2023-03-09T17:00:48.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:47 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:00:48.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:47 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:00:48.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:47 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:00:48.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7820.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:00:49.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:48 smithi122 ceph-mon[93869]: pgmap v4439: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 36 KiB/s wr, 9 op/s 2023-03-09T17:00:49.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:48 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:00:49.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:00:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:00:49.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:48 smithi181 ceph-mon[94189]: pgmap v4439: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 36 KiB/s wr, 9 op/s 2023-03-09T17:00:49.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:48 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:00:49.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:00:49.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:00:49.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:48 smithi038 ceph-mon[92422]: pgmap v4439: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 36 KiB/s wr, 9 op/s 2023-03-09T17:00:49.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:48 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:00:49.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:00:49.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:00:51.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7361.037047 secs 2023-03-09T17:00:51.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:50 smithi122 ceph-mon[93869]: pgmap v4440: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 36 KiB/s wr, 8 op/s 2023-03-09T17:00:51.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7361.037047 secs 2023-03-09T17:00:51.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:50 smithi181 ceph-mon[94189]: pgmap v4440: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 36 KiB/s wr, 8 op/s 2023-03-09T17:00:51.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7361.037047 secs 2023-03-09T17:00:51.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:50 smithi038 ceph-mon[92422]: pgmap v4440: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 36 KiB/s wr, 8 op/s 2023-03-09T17:00:53.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:52 smithi122 ceph-mon[93869]: pgmap v4441: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 83 KiB/s wr, 12 op/s 2023-03-09T17:00:53.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:52 smithi181 ceph-mon[94189]: pgmap v4441: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 83 KiB/s wr, 12 op/s 2023-03-09T17:00:53.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:52 smithi038 ceph-mon[92422]: pgmap v4441: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 83 KiB/s wr, 12 op/s 2023-03-09T17:00:53.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7825.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:00:55.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:54 smithi122 ceph-mon[93869]: pgmap v4442: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 57 KiB/s wr, 9 op/s 2023-03-09T17:00:55.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:54 smithi181 ceph-mon[94189]: pgmap v4442: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 57 KiB/s wr, 9 op/s 2023-03-09T17:00:55.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:54 smithi038 ceph-mon[92422]: pgmap v4442: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 57 KiB/s wr, 9 op/s 2023-03-09T17:00:56.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7366.037108 secs 2023-03-09T17:00:56.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7366.037108 secs 2023-03-09T17:00:56.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7366.037108 secs 2023-03-09T17:00:57.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:56 smithi122 ceph-mon[93869]: pgmap v4443: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 69 KiB/s wr, 10 op/s 2023-03-09T17:00:57.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:56 smithi181 ceph-mon[94189]: pgmap v4443: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 69 KiB/s wr, 10 op/s 2023-03-09T17:00:57.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:56 smithi038 ceph-mon[92422]: pgmap v4443: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 69 KiB/s wr, 10 op/s 2023-03-09T17:00:58.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7830.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:00:59.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:58 smithi122 ceph-mon[93869]: pgmap v4444: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 67 KiB/s wr, 11 op/s 2023-03-09T17:00:59.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:58 smithi181 ceph-mon[94189]: pgmap v4444: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 67 KiB/s wr, 11 op/s 2023-03-09T17:00:59.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:58 smithi038 ceph-mon[92422]: pgmap v4444: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 67 KiB/s wr, 11 op/s 2023-03-09T17:01:00.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:00:59 smithi122 ceph-mon[93869]: pgmap v4445: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 91 KiB/s wr, 10 op/s 2023-03-09T17:01:00.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:00:59 smithi181 ceph-mon[94189]: pgmap v4445: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 91 KiB/s wr, 10 op/s 2023-03-09T17:01:00.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:00:59 smithi038 ceph-mon[92422]: pgmap v4445: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 91 KiB/s wr, 10 op/s 2023-03-09T17:01:01.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:01:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7371.037169 secs 2023-03-09T17:01:01.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:01:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7371.037169 secs 2023-03-09T17:01:01.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:01:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7371.037169 secs 2023-03-09T17:01:01.898 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:01:01 smithi122 ceph-mon[93869]: pgmap v4446: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 115 KiB/s wr, 13 op/s 2023-03-09T17:01:02.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:01:01 smithi181 ceph-mon[94189]: pgmap v4446: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 115 KiB/s wr, 13 op/s 2023-03-09T17:01:02.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:01:01 smithi038 ceph-mon[92422]: pgmap v4446: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 115 KiB/s wr, 13 op/s 2023-03-09T17:01:03.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:01:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7373.289793 secs 2023-03-09T17:01:03.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:01:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7373.289793 secs 2023-03-09T17:01:03.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:01:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7373.289793 secs 2023-03-09T17:01:03.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7835.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:01:04.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:01:03 smithi122 ceph-mon[93869]: pgmap v4447: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 89 KiB/s wr, 22 op/s 2023-03-09T17:01:04.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:01:03 smithi181 ceph-mon[94189]: pgmap v4447: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 89 KiB/s wr, 22 op/s 2023-03-09T17:01:04.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:01:03 smithi038 ceph-mon[92422]: pgmap v4447: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 89 KiB/s wr, 22 op/s 2023-03-09T17:01:05.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:01:04 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:01:05.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:01:04 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:01:05.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:01:04 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:01:05.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:01:04 smithi122 ceph-mon[93869]: osdmap e492: 12 total, 12 up, 12 in 2023-03-09T17:01:05.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:01:04 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:01:05.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:01:04 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:01:05.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:01:04 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:01:05.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:01:04 smithi181 ceph-mon[94189]: osdmap e492: 12 total, 12 up, 12 in 2023-03-09T17:01:05.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:01:04 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:01:05.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:01:04 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:01:05.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:01:04 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:01:05.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:01:04 smithi038 ceph-mon[92422]: osdmap e492: 12 total, 12 up, 12 in 2023-03-09T17:01:06.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:01:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7376.037228 secs 2023-03-09T17:01:06.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:01:05 smithi122 ceph-mon[93869]: pgmap v4449: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 145 KiB/s wr, 29 op/s 2023-03-09T17:01:06.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:01:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7376.037228 secs 2023-03-09T17:01:06.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:01:05 smithi181 ceph-mon[94189]: pgmap v4449: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 145 KiB/s wr, 29 op/s 2023-03-09T17:01:06.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:01:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7376.037228 secs 2023-03-09T17:01:06.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:01:05 smithi038 ceph-mon[92422]: pgmap v4449: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 145 KiB/s wr, 29 op/s 2023-03-09T17:01:07.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:01:07 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:01:07.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:01:07 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:01:07.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:01:07 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:01:08.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:01:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7378.289886 secs 2023-03-09T17:01:08.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:01:08 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:01:08.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:01:08 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:01:08.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:01:08 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:01:08.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:01:08 smithi122 ceph-mon[93869]: pgmap v4450: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 151 KiB/s wr, 29 op/s 2023-03-09T17:01:08.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:01:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7378.289886 secs 2023-03-09T17:01:08.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:01:08 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:01:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:01:08 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:01:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:01:08 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:01:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:01:08 smithi181 ceph-mon[94189]: pgmap v4450: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 151 KiB/s wr, 29 op/s 2023-03-09T17:01:08.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:01:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7378.289886 secs 2023-03-09T17:01:08.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:01:08 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:01:08.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:01:08 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:01:08.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:01:08 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:01:08.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:01:08 smithi038 ceph-mon[92422]: pgmap v4450: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 151 KiB/s wr, 29 op/s 2023-03-09T17:01:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7840.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:01:11.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:01:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7381.037312 secs 2023-03-09T17:01:11.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:01:10 smithi122 ceph-mon[93869]: pgmap v4451: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 152 KiB/s wr, 29 op/s 2023-03-09T17:01:11.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:01:10 smithi122 ceph-mon[93869]: osdmap e493: 12 total, 12 up, 12 in 2023-03-09T17:01:11.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:01:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7381.037312 secs 2023-03-09T17:01:11.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:01:10 smithi181 ceph-mon[94189]: pgmap v4451: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 152 KiB/s wr, 29 op/s 2023-03-09T17:01:11.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:01:10 smithi181 ceph-mon[94189]: osdmap e493: 12 total, 12 up, 12 in 2023-03-09T17:01:11.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:01:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7381.037312 secs 2023-03-09T17:01:11.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:01:10 smithi038 ceph-mon[92422]: pgmap v4451: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 152 KiB/s wr, 29 op/s 2023-03-09T17:01:11.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:01:10 smithi038 ceph-mon[92422]: osdmap e493: 12 total, 12 up, 12 in 2023-03-09T17:01:13.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:01:12 smithi122 ceph-mon[93869]: pgmap v4453: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 124 KiB/s wr, 15 op/s 2023-03-09T17:01:13.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:01:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7383.290015 secs 2023-03-09T17:01:13.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:01:12 smithi181 ceph-mon[94189]: pgmap v4453: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 124 KiB/s wr, 15 op/s 2023-03-09T17:01:13.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:01:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7383.290015 secs 2023-03-09T17:01:13.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:01:12 smithi038 ceph-mon[92422]: pgmap v4453: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 124 KiB/s wr, 15 op/s 2023-03-09T17:01:13.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:01:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7383.290015 secs 2023-03-09T17:01:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7845.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:01:15.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:01:14 smithi122 ceph-mon[93869]: pgmap v4454: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 101 KiB/s wr, 12 op/s 2023-03-09T17:01:15.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:01:14 smithi181 ceph-mon[94189]: pgmap v4454: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 101 KiB/s wr, 12 op/s 2023-03-09T17:01:15.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:01:14 smithi038 ceph-mon[92422]: pgmap v4454: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 101 KiB/s wr, 12 op/s 2023-03-09T17:01:16.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:01:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7386.037396 secs 2023-03-09T17:01:16.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:01:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7386.037396 secs 2023-03-09T17:01:16.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:01:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7386.037396 secs 2023-03-09T17:01:17.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:01:16 smithi122 ceph-mon[93869]: pgmap v4455: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 67 KiB/s wr, 11 op/s 2023-03-09T17:01:17.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:01:16 smithi181 ceph-mon[94189]: pgmap v4455: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 67 KiB/s wr, 11 op/s 2023-03-09T17:01:17.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:01:16 smithi038 ceph-mon[92422]: pgmap v4455: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 67 KiB/s wr, 11 op/s 2023-03-09T17:01:18.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:01:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7388.290079 secs 2023-03-09T17:01:18.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:01:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7388.290079 secs 2023-03-09T17:01:18.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:01:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7388.290079 secs 2023-03-09T17:01:18.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7850.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:01:19.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:01:18 smithi122 ceph-mon[93869]: pgmap v4456: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 8 op/s 2023-03-09T17:01:19.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:01:18 smithi181 ceph-mon[94189]: pgmap v4456: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 8 op/s 2023-03-09T17:01:19.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:01:18 smithi038 ceph-mon[92422]: pgmap v4456: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 8 op/s 2023-03-09T17:01:21.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:01:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7391.037456 secs 2023-03-09T17:01:21.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:01:20 smithi122 ceph-mon[93869]: pgmap v4457: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 22 KiB/s wr, 7 op/s 2023-03-09T17:01:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:01:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7391.037456 secs 2023-03-09T17:01:21.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:01:20 smithi181 ceph-mon[94189]: pgmap v4457: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 22 KiB/s wr, 7 op/s 2023-03-09T17:01:21.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:01:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7391.037456 secs 2023-03-09T17:01:21.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:01:20 smithi038 ceph-mon[92422]: pgmap v4457: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 22 KiB/s wr, 7 op/s 2023-03-09T17:01:23.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:01:22 smithi122 ceph-mon[93869]: pgmap v4458: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 19 KiB/s wr, 9 op/s 2023-03-09T17:01:23.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:01:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7393.290202 secs 2023-03-09T17:01:23.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:01:22 smithi181 ceph-mon[94189]: pgmap v4458: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 19 KiB/s wr, 9 op/s 2023-03-09T17:01:23.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:01:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7393.290202 secs 2023-03-09T17:01:23.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:01:22 smithi038 ceph-mon[92422]: pgmap v4458: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 19 KiB/s wr, 9 op/s 2023-03-09T17:01:23.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:01:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7393.290202 secs 2023-03-09T17:01:23.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7855.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:01:25.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:01:24 smithi122 ceph-mon[93869]: pgmap v4459: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 16 KiB/s wr, 6 op/s 2023-03-09T17:01:25.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:01:24 smithi181 ceph-mon[94189]: pgmap v4459: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 16 KiB/s wr, 6 op/s 2023-03-09T17:01:25.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:01:24 smithi038 ceph-mon[92422]: pgmap v4459: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 16 KiB/s wr, 6 op/s 2023-03-09T17:01:26.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:01:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7396.037526 secs 2023-03-09T17:01:26.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:01:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7396.037526 secs 2023-03-09T17:01:26.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:01:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7396.037526 secs 2023-03-09T17:01:27.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:01:26 smithi122 ceph-mon[93869]: pgmap v4460: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 39 KiB/s wr, 9 op/s 2023-03-09T17:01:27.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:01:26 smithi181 ceph-mon[94189]: pgmap v4460: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 39 KiB/s wr, 9 op/s 2023-03-09T17:01:27.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:01:26 smithi038 ceph-mon[92422]: pgmap v4460: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 39 KiB/s wr, 9 op/s 2023-03-09T17:01:28.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:01:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7398.290315 secs 2023-03-09T17:01:28.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:01:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7398.290315 secs 2023-03-09T17:01:28.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:01:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7398.290315 secs 2023-03-09T17:01:28.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7860.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:01:29.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:01:28 smithi122 ceph-mon[93869]: pgmap v4461: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 6 op/s 2023-03-09T17:01:29.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:01:28 smithi181 ceph-mon[94189]: pgmap v4461: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 6 op/s 2023-03-09T17:01:29.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:01:28 smithi038 ceph-mon[92422]: pgmap v4461: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 6 op/s 2023-03-09T17:01:31.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:01:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7401.037588 secs 2023-03-09T17:01:31.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:01:30 smithi122 ceph-mon[93869]: pgmap v4462: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 22 KiB/s wr, 6 op/s 2023-03-09T17:01:31.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:01:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7401.037588 secs 2023-03-09T17:01:31.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:01:30 smithi181 ceph-mon[94189]: pgmap v4462: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 22 KiB/s wr, 6 op/s 2023-03-09T17:01:31.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:01:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7401.037588 secs 2023-03-09T17:01:31.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:01:30 smithi038 ceph-mon[92422]: pgmap v4462: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 22 KiB/s wr, 6 op/s 2023-03-09T17:01:33.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:01:32 smithi122 ceph-mon[93869]: pgmap v4463: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 8 op/s 2023-03-09T17:01:33.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:01:32 smithi181 ceph-mon[94189]: pgmap v4463: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 8 op/s 2023-03-09T17:01:33.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:01:32 smithi038 ceph-mon[92422]: pgmap v4463: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 8 op/s 2023-03-09T17:01:33.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7865.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:01:35.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:01:34 smithi122 ceph-mon[93869]: pgmap v4464: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T17:01:35.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:01:34 smithi181 ceph-mon[94189]: pgmap v4464: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T17:01:35.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:01:34 smithi038 ceph-mon[92422]: pgmap v4464: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T17:01:36.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:01:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7406.037650 secs 2023-03-09T17:01:36.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:01:35 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:01:36.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:01:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7406.037650 secs 2023-03-09T17:01:36.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:01:35 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:01:36.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:01:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7406.037650 secs 2023-03-09T17:01:36.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:01:35 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:01:37.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:01:36 smithi122 ceph-mon[93869]: pgmap v4465: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 41 KiB/s wr, 9 op/s 2023-03-09T17:01:37.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:01:36 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:01:37.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:01:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:01:37.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:01:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:01:37.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:01:36 smithi181 ceph-mon[94189]: pgmap v4465: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 41 KiB/s wr, 9 op/s 2023-03-09T17:01:37.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:01:36 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:01:37.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:01:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:01:37.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:01:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:01:37.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:01:36 smithi038 ceph-mon[92422]: pgmap v4465: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 41 KiB/s wr, 9 op/s 2023-03-09T17:01:37.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:01:36 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:01:37.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:01:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:01:37.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:01:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:01:38.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:01:37 smithi122 ceph-mon[93869]: pgmap v4466: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 18 KiB/s wr, 6 op/s 2023-03-09T17:01:38.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:01:37 smithi181 ceph-mon[94189]: pgmap v4466: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 18 KiB/s wr, 6 op/s 2023-03-09T17:01:38.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:01:37 smithi038 ceph-mon[92422]: pgmap v4466: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 18 KiB/s wr, 6 op/s 2023-03-09T17:01:38.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7870.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:01:41.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:01:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7411.037710 secs 2023-03-09T17:01:41.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:01:40 smithi122 ceph-mon[93869]: pgmap v4467: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 43 KiB/s wr, 8 op/s 2023-03-09T17:01:41.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:01:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7411.037710 secs 2023-03-09T17:01:41.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:01:40 smithi181 ceph-mon[94189]: pgmap v4467: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 43 KiB/s wr, 8 op/s 2023-03-09T17:01:41.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:01:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7411.037710 secs 2023-03-09T17:01:41.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:01:40 smithi038 ceph-mon[92422]: pgmap v4467: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 43 KiB/s wr, 8 op/s 2023-03-09T17:01:43.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:01:42 smithi122 ceph-mon[93869]: pgmap v4468: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 50 KiB/s wr, 10 op/s 2023-03-09T17:01:43.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:01:42 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 7413.290695 secs 2023-03-09T17:01:43.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:01:42 smithi181 ceph-mon[94189]: pgmap v4468: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 50 KiB/s wr, 10 op/s 2023-03-09T17:01:43.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:01:42 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 7413.290695 secs 2023-03-09T17:01:43.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:01:42 smithi038 ceph-mon[92422]: pgmap v4468: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 50 KiB/s wr, 10 op/s 2023-03-09T17:01:43.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:01:42 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 7413.290695 secs 2023-03-09T17:01:43.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7875.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:01:44.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:01:43 smithi122 ceph-mon[93869]: pgmap v4469: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 49 KiB/s wr, 8 op/s 2023-03-09T17:01:44.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:01:43 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:01:44.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:01:43 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:01:44.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:01:43 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:01:44.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:01:43 smithi181 ceph-mon[94189]: pgmap v4469: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 49 KiB/s wr, 8 op/s 2023-03-09T17:01:44.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:01:43 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:01:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:01:43 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:01:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:01:43 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:01:44.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:01:43 smithi038 ceph-mon[92422]: pgmap v4469: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 49 KiB/s wr, 8 op/s 2023-03-09T17:01:44.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:01:43 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:01:44.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:01:43 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:01:44.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:01:43 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:01:45.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:01:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7416.037770 secs 2023-03-09T17:01:45.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:01:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7416.037770 secs 2023-03-09T17:01:45.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:01:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7416.037770 secs 2023-03-09T17:01:46.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:01:46 smithi122 ceph-mon[93869]: pgmap v4470: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 89 KiB/s wr, 12 op/s 2023-03-09T17:01:46.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:01:46 smithi181 ceph-mon[94189]: pgmap v4470: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 89 KiB/s wr, 12 op/s 2023-03-09T17:01:46.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:01:46 smithi038 ceph-mon[92422]: pgmap v4470: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 89 KiB/s wr, 12 op/s 2023-03-09T17:01:47.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:01:47 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 7418.290816 secs 2023-03-09T17:01:47.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:01:47 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 7418.290816 secs 2023-03-09T17:01:47.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:01:47 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 7418.290816 secs 2023-03-09T17:01:48.724 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:01:48 smithi122 ceph-mon[93869]: pgmap v4471: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 71 KiB/s wr, 9 op/s 2023-03-09T17:01:48.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:01:48 smithi181 ceph-mon[94189]: pgmap v4471: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 71 KiB/s wr, 9 op/s 2023-03-09T17:01:48.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:01:48 smithi038 ceph-mon[92422]: pgmap v4471: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 71 KiB/s wr, 9 op/s 2023-03-09T17:01:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7880.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:01:51.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:01:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7421.037831 secs 2023-03-09T17:01:51.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:01:50 smithi122 ceph-mon[93869]: pgmap v4472: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 83 KiB/s wr, 10 op/s 2023-03-09T17:01:51.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:01:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7421.037831 secs 2023-03-09T17:01:51.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:01:50 smithi181 ceph-mon[94189]: pgmap v4472: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 83 KiB/s wr, 10 op/s 2023-03-09T17:01:51.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:01:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7421.037831 secs 2023-03-09T17:01:51.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:01:50 smithi038 ceph-mon[92422]: pgmap v4472: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 83 KiB/s wr, 10 op/s 2023-03-09T17:01:53.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:01:52 smithi122 ceph-mon[93869]: pgmap v4473: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 65 KiB/s wr, 11 op/s 2023-03-09T17:01:53.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:01:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7423.290884 secs 2023-03-09T17:01:53.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:01:52 smithi181 ceph-mon[94189]: pgmap v4473: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 65 KiB/s wr, 11 op/s 2023-03-09T17:01:53.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:01:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7423.290884 secs 2023-03-09T17:01:53.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:01:52 smithi038 ceph-mon[92422]: pgmap v4473: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 65 KiB/s wr, 11 op/s 2023-03-09T17:01:53.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:01:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7423.290884 secs 2023-03-09T17:01:53.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7885.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:01:55.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:01:54 smithi122 ceph-mon[93869]: pgmap v4474: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 58 KiB/s wr, 9 op/s 2023-03-09T17:01:55.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:01:54 smithi181 ceph-mon[94189]: pgmap v4474: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 58 KiB/s wr, 9 op/s 2023-03-09T17:01:55.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:01:54 smithi038 ceph-mon[92422]: pgmap v4474: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 58 KiB/s wr, 9 op/s 2023-03-09T17:01:56.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:01:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7426.037916 secs 2023-03-09T17:01:56.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:01:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7426.037916 secs 2023-03-09T17:01:56.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:01:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7426.037916 secs 2023-03-09T17:01:57.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:01:56 smithi122 ceph-mon[93869]: pgmap v4475: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 99 KiB/s wr, 12 op/s 2023-03-09T17:01:57.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:01:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:01:57.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:01:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:01:57.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:01:56 smithi181 ceph-mon[94189]: pgmap v4475: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 99 KiB/s wr, 12 op/s 2023-03-09T17:01:57.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:01:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:01:57.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:01:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:01:57.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:01:56 smithi038 ceph-mon[92422]: pgmap v4475: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 99 KiB/s wr, 12 op/s 2023-03-09T17:01:57.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:01:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:01:57.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:01:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:01:58.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:01:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7428.291021 secs 2023-03-09T17:01:58.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:01:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7428.291021 secs 2023-03-09T17:01:58.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:01:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7428.291021 secs 2023-03-09T17:01:58.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7890.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:01:59.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:01:58 smithi122 ceph-mon[93869]: pgmap v4476: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 60 KiB/s wr, 8 op/s 2023-03-09T17:01:59.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:01:58 smithi181 ceph-mon[94189]: pgmap v4476: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 60 KiB/s wr, 8 op/s 2023-03-09T17:01:59.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:01:58 smithi038 ceph-mon[92422]: pgmap v4476: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 60 KiB/s wr, 8 op/s 2023-03-09T17:02:01.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:02:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7431.037979 secs 2023-03-09T17:02:01.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:02:00 smithi122 ceph-mon[93869]: pgmap v4477: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 69 KiB/s wr, 9 op/s 2023-03-09T17:02:01.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:02:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7431.037979 secs 2023-03-09T17:02:01.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:02:00 smithi181 ceph-mon[94189]: pgmap v4477: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 69 KiB/s wr, 9 op/s 2023-03-09T17:02:01.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:02:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7431.037979 secs 2023-03-09T17:02:01.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:02:00 smithi038 ceph-mon[92422]: pgmap v4477: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 69 KiB/s wr, 9 op/s 2023-03-09T17:02:03.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:02:02 smithi122 ceph-mon[93869]: pgmap v4478: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 58 KiB/s wr, 10 op/s 2023-03-09T17:02:03.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:02:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7433.291146 secs 2023-03-09T17:02:03.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:02:02 smithi181 ceph-mon[94189]: pgmap v4478: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 58 KiB/s wr, 10 op/s 2023-03-09T17:02:03.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:02:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7433.291146 secs 2023-03-09T17:02:03.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:02:02 smithi038 ceph-mon[92422]: pgmap v4478: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 58 KiB/s wr, 10 op/s 2023-03-09T17:02:03.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:02:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7433.291146 secs 2023-03-09T17:02:03.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7895.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:02:05.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:02:04 smithi122 ceph-mon[93869]: pgmap v4479: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 73 KiB/s wr, 20 op/s 2023-03-09T17:02:05.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:02:04 smithi181 ceph-mon[94189]: pgmap v4479: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 73 KiB/s wr, 20 op/s 2023-03-09T17:02:05.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:02:04 smithi038 ceph-mon[92422]: pgmap v4479: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 73 KiB/s wr, 20 op/s 2023-03-09T17:02:06.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:02:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7436.038039 secs 2023-03-09T17:02:06.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:02:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7436.038039 secs 2023-03-09T17:02:06.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:02:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7436.038039 secs 2023-03-09T17:02:07.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:02:06 smithi122 ceph-mon[93869]: pgmap v4480: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 103 KiB/s wr, 25 op/s 2023-03-09T17:02:07.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:02:06 smithi181 ceph-mon[94189]: pgmap v4480: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 103 KiB/s wr, 25 op/s 2023-03-09T17:02:07.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:02:06 smithi038 ceph-mon[92422]: pgmap v4480: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 103 KiB/s wr, 25 op/s 2023-03-09T17:02:08.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:02:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7438.291267 secs 2023-03-09T17:02:08.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:02:07 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:02:08.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:02:07 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:02:08.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:02:07 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:02:08.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:02:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7438.291267 secs 2023-03-09T17:02:08.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:02:07 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:02:08.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:02:07 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:02:08.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:02:07 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:02:08.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:02:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7438.291267 secs 2023-03-09T17:02:08.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:02:07 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:02:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:02:07 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:02:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:02:07 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:02:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7900.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:02:09.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:02:08 smithi122 ceph-mon[93869]: pgmap v4481: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 62 KiB/s wr, 21 op/s 2023-03-09T17:02:09.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:02:08 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:02:09.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:02:08 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:02:09.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:02:08 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:02:09.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:02:08 smithi181 ceph-mon[94189]: pgmap v4481: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 62 KiB/s wr, 21 op/s 2023-03-09T17:02:09.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:02:08 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:02:09.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:02:08 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:02:09.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:02:08 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:02:09.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:02:08 smithi038 ceph-mon[92422]: pgmap v4481: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 62 KiB/s wr, 21 op/s 2023-03-09T17:02:09.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:02:08 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:02:09.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:02:08 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:02:09.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:02:08 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:02:10.974 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:02:10 smithi122 ceph-mon[93869]: osdmap e494: 12 total, 12 up, 12 in 2023-03-09T17:02:10.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:02:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7441.038115 secs 2023-03-09T17:02:10.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:02:10 smithi122 ceph-mon[93869]: pgmap v4483: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 74 KiB/s wr, 25 op/s 2023-03-09T17:02:10.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:02:10 smithi181 ceph-mon[94189]: osdmap e494: 12 total, 12 up, 12 in 2023-03-09T17:02:10.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:02:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7441.038115 secs 2023-03-09T17:02:10.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:02:10 smithi181 ceph-mon[94189]: pgmap v4483: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 74 KiB/s wr, 25 op/s 2023-03-09T17:02:10.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:02:10 smithi038 ceph-mon[92422]: osdmap e494: 12 total, 12 up, 12 in 2023-03-09T17:02:10.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:02:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7441.038115 secs 2023-03-09T17:02:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:02:10 smithi038 ceph-mon[92422]: pgmap v4483: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 74 KiB/s wr, 25 op/s 2023-03-09T17:02:13.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:02:12 smithi122 ceph-mon[93869]: pgmap v4484: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 74 KiB/s wr, 25 op/s 2023-03-09T17:02:13.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:02:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7443.291396 secs 2023-03-09T17:02:13.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:02:12 smithi181 ceph-mon[94189]: pgmap v4484: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 74 KiB/s wr, 25 op/s 2023-03-09T17:02:13.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:02:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7443.291396 secs 2023-03-09T17:02:13.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:02:12 smithi038 ceph-mon[92422]: pgmap v4484: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 74 KiB/s wr, 25 op/s 2023-03-09T17:02:13.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:02:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7443.291396 secs 2023-03-09T17:02:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7905.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:02:15.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:02:14 smithi122 ceph-mon[93869]: pgmap v4485: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 48 KiB/s wr, 10 op/s 2023-03-09T17:02:15.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:02:14 smithi181 ceph-mon[94189]: pgmap v4485: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 48 KiB/s wr, 10 op/s 2023-03-09T17:02:15.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:02:14 smithi038 ceph-mon[92422]: pgmap v4485: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 48 KiB/s wr, 10 op/s 2023-03-09T17:02:16.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:02:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7446.038164 secs 2023-03-09T17:02:16.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:02:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7446.038164 secs 2023-03-09T17:02:16.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:02:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7446.038164 secs 2023-03-09T17:02:17.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:02:16 smithi122 ceph-mon[93869]: pgmap v4486: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 21 KiB/s wr, 7 op/s 2023-03-09T17:02:17.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:02:16 smithi181 ceph-mon[94189]: pgmap v4486: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 21 KiB/s wr, 7 op/s 2023-03-09T17:02:17.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:02:16 smithi038 ceph-mon[92422]: pgmap v4486: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 21 KiB/s wr, 7 op/s 2023-03-09T17:02:18.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:02:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7448.291518 secs 2023-03-09T17:02:18.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:02:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7448.291518 secs 2023-03-09T17:02:18.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:02:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7448.291518 secs 2023-03-09T17:02:18.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7910.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:02:19.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:02:18 smithi122 ceph-mon[93869]: pgmap v4487: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 21 KiB/s wr, 7 op/s 2023-03-09T17:02:19.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:02:18 smithi181 ceph-mon[94189]: pgmap v4487: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 21 KiB/s wr, 7 op/s 2023-03-09T17:02:19.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:02:18 smithi038 ceph-mon[92422]: pgmap v4487: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 21 KiB/s wr, 7 op/s 2023-03-09T17:02:21.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:02:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7451.038223 secs 2023-03-09T17:02:21.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:02:20 smithi122 ceph-mon[93869]: pgmap v4488: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 8.7 KiB/s wr, 7 op/s 2023-03-09T17:02:21.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:02:20 smithi122 ceph-mon[93869]: osdmap e495: 12 total, 12 up, 12 in 2023-03-09T17:02:21.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:02:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7451.038223 secs 2023-03-09T17:02:21.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:02:20 smithi181 ceph-mon[94189]: pgmap v4488: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 8.7 KiB/s wr, 7 op/s 2023-03-09T17:02:21.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:02:20 smithi181 ceph-mon[94189]: osdmap e495: 12 total, 12 up, 12 in 2023-03-09T17:02:21.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:02:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7451.038223 secs 2023-03-09T17:02:21.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:02:20 smithi038 ceph-mon[92422]: pgmap v4488: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 8.7 KiB/s wr, 7 op/s 2023-03-09T17:02:21.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:02:20 smithi038 ceph-mon[92422]: osdmap e495: 12 total, 12 up, 12 in 2023-03-09T17:02:23.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:02:22 smithi122 ceph-mon[93869]: pgmap v4490: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 9.2 KiB/s wr, 7 op/s 2023-03-09T17:02:23.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:02:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7453.291644 secs 2023-03-09T17:02:23.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:02:22 smithi181 ceph-mon[94189]: pgmap v4490: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 9.2 KiB/s wr, 7 op/s 2023-03-09T17:02:23.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:02:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7453.291644 secs 2023-03-09T17:02:23.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:02:22 smithi038 ceph-mon[92422]: pgmap v4490: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 9.2 KiB/s wr, 7 op/s 2023-03-09T17:02:23.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:02:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7453.291644 secs 2023-03-09T17:02:23.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7915.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:02:25.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:02:24 smithi122 ceph-mon[93869]: pgmap v4491: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 9.2 KiB/s wr, 7 op/s 2023-03-09T17:02:25.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:02:24 smithi181 ceph-mon[94189]: pgmap v4491: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 9.2 KiB/s wr, 7 op/s 2023-03-09T17:02:25.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:02:24 smithi038 ceph-mon[92422]: pgmap v4491: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 9.2 KiB/s wr, 7 op/s 2023-03-09T17:02:26.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:02:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7456.038287 secs 2023-03-09T17:02:26.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:02:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7456.038287 secs 2023-03-09T17:02:26.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:02:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7456.038287 secs 2023-03-09T17:02:27.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:02:26 smithi122 ceph-mon[93869]: pgmap v4492: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T17:02:27.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:02:26 smithi181 ceph-mon[94189]: pgmap v4492: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T17:02:27.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:02:26 smithi038 ceph-mon[92422]: pgmap v4492: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T17:02:27.846 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:02:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7458.291765 secs 2023-03-09T17:02:28.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:02:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7458.291765 secs 2023-03-09T17:02:28.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:02:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7458.291765 secs 2023-03-09T17:02:28.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7920.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:02:29.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:02:28 smithi122 ceph-mon[93869]: pgmap v4493: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T17:02:29.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:02:28 smithi181 ceph-mon[94189]: pgmap v4493: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T17:02:29.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:02:28 smithi038 ceph-mon[92422]: pgmap v4493: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T17:02:31.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:02:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7461.038348 secs 2023-03-09T17:02:31.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:02:30 smithi122 ceph-mon[93869]: pgmap v4494: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 8 op/s 2023-03-09T17:02:31.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:02:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7461.038348 secs 2023-03-09T17:02:31.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:02:30 smithi181 ceph-mon[94189]: pgmap v4494: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 8 op/s 2023-03-09T17:02:31.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:02:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7461.038348 secs 2023-03-09T17:02:31.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:02:30 smithi038 ceph-mon[92422]: pgmap v4494: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 8 op/s 2023-03-09T17:02:33.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:02:32 smithi122 ceph-mon[93869]: pgmap v4495: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 16 KiB/s wr, 9 op/s 2023-03-09T17:02:33.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:02:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7463.291883 secs 2023-03-09T17:02:33.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:02:32 smithi181 ceph-mon[94189]: pgmap v4495: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 16 KiB/s wr, 9 op/s 2023-03-09T17:02:33.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:02:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7463.291883 secs 2023-03-09T17:02:33.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:02:32 smithi038 ceph-mon[92422]: pgmap v4495: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 16 KiB/s wr, 9 op/s 2023-03-09T17:02:33.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:02:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7463.291883 secs 2023-03-09T17:02:33.801 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7925.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:02:34.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:02:33 smithi122 ceph-mon[93869]: pgmap v4496: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T17:02:34.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:02:33 smithi181 ceph-mon[94189]: pgmap v4496: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T17:02:34.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:02:33 smithi038 ceph-mon[92422]: pgmap v4496: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T17:02:35.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:02:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7466.038407 secs 2023-03-09T17:02:35.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:02:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7466.038407 secs 2023-03-09T17:02:35.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:02:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7466.038407 secs 2023-03-09T17:02:36.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:02:36 smithi122 ceph-mon[93869]: pgmap v4497: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 59 KiB/s wr, 11 op/s 2023-03-09T17:02:36.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:02:36 smithi181 ceph-mon[94189]: pgmap v4497: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 59 KiB/s wr, 11 op/s 2023-03-09T17:02:36.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:02:36 smithi038 ceph-mon[92422]: pgmap v4497: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 59 KiB/s wr, 11 op/s 2023-03-09T17:02:37.724 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:02:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7468.292001 secs 2023-03-09T17:02:37.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:02:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7468.292001 secs 2023-03-09T17:02:37.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:02:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7468.292001 secs 2023-03-09T17:02:38.724 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:02:38 smithi122 ceph-mon[93869]: pgmap v4498: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 45 KiB/s wr, 8 op/s 2023-03-09T17:02:38.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:02:38 smithi181 ceph-mon[94189]: pgmap v4498: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 45 KiB/s wr, 8 op/s 2023-03-09T17:02:38.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:02:38 smithi038 ceph-mon[92422]: pgmap v4498: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 45 KiB/s wr, 8 op/s 2023-03-09T17:02:38.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7930.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:02:41.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:02:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7471.038466 secs 2023-03-09T17:02:41.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:02:40 smithi122 ceph-mon[93869]: pgmap v4499: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 66 KiB/s wr, 9 op/s 2023-03-09T17:02:41.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:02:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7471.038466 secs 2023-03-09T17:02:41.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:02:40 smithi181 ceph-mon[94189]: pgmap v4499: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 66 KiB/s wr, 9 op/s 2023-03-09T17:02:41.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:02:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7471.038466 secs 2023-03-09T17:02:41.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:02:40 smithi038 ceph-mon[92422]: pgmap v4499: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 66 KiB/s wr, 9 op/s 2023-03-09T17:02:43.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:02:42 smithi122 ceph-mon[93869]: pgmap v4500: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 67 KiB/s wr, 11 op/s 2023-03-09T17:02:43.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:02:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7473.292085 secs 2023-03-09T17:02:43.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:02:42 smithi181 ceph-mon[94189]: pgmap v4500: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 67 KiB/s wr, 11 op/s 2023-03-09T17:02:43.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:02:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7473.292085 secs 2023-03-09T17:02:43.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:02:42 smithi038 ceph-mon[92422]: pgmap v4500: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 67 KiB/s wr, 11 op/s 2023-03-09T17:02:43.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:02:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7473.292085 secs 2023-03-09T17:02:43.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7935.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:02:45.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:02:44 smithi122 ceph-mon[93869]: pgmap v4501: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 67 KiB/s wr, 10 op/s 2023-03-09T17:02:45.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:02:44 smithi181 ceph-mon[94189]: pgmap v4501: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 67 KiB/s wr, 10 op/s 2023-03-09T17:02:45.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:02:44 smithi038 ceph-mon[92422]: pgmap v4501: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 67 KiB/s wr, 10 op/s 2023-03-09T17:02:46.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:02:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7476.038527 secs 2023-03-09T17:02:46.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:02:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7476.038527 secs 2023-03-09T17:02:46.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:02:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7476.038527 secs 2023-03-09T17:02:47.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:02:46 smithi122 ceph-mon[93869]: pgmap v4502: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 97 KiB/s wr, 13 op/s 2023-03-09T17:02:47.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:02:46 smithi181 ceph-mon[94189]: pgmap v4502: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 97 KiB/s wr, 13 op/s 2023-03-09T17:02:47.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:02:46 smithi038 ceph-mon[92422]: pgmap v4502: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 97 KiB/s wr, 13 op/s 2023-03-09T17:02:48.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:02:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7478.292204 secs 2023-03-09T17:02:48.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:02:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7478.292204 secs 2023-03-09T17:02:48.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:02:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7478.292204 secs 2023-03-09T17:02:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7940.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:02:49.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:02:48 smithi122 ceph-mon[93869]: pgmap v4503: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 73 KiB/s wr, 10 op/s 2023-03-09T17:02:49.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:02:48 smithi181 ceph-mon[94189]: pgmap v4503: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 73 KiB/s wr, 10 op/s 2023-03-09T17:02:49.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:02:48 smithi038 ceph-mon[92422]: pgmap v4503: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 73 KiB/s wr, 10 op/s 2023-03-09T17:02:51.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:02:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7481.038586 secs 2023-03-09T17:02:51.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:02:50 smithi122 ceph-mon[93869]: pgmap v4504: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 73 KiB/s wr, 10 op/s 2023-03-09T17:02:51.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:02:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7481.038586 secs 2023-03-09T17:02:51.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:02:50 smithi181 ceph-mon[94189]: pgmap v4504: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 73 KiB/s wr, 10 op/s 2023-03-09T17:02:51.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:02:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7481.038586 secs 2023-03-09T17:02:51.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:02:50 smithi038 ceph-mon[92422]: pgmap v4504: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 73 KiB/s wr, 10 op/s 2023-03-09T17:02:53.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:02:52 smithi122 ceph-mon[93869]: pgmap v4505: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 52 KiB/s wr, 12 op/s 2023-03-09T17:02:53.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:02:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7483.292275 secs 2023-03-09T17:02:53.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:02:52 smithi181 ceph-mon[94189]: pgmap v4505: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 52 KiB/s wr, 12 op/s 2023-03-09T17:02:53.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:02:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7483.292275 secs 2023-03-09T17:02:53.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:02:52 smithi038 ceph-mon[92422]: pgmap v4505: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 52 KiB/s wr, 12 op/s 2023-03-09T17:02:53.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:02:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7483.292275 secs 2023-03-09T17:02:53.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7945.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:02:55.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:02:54 smithi122 ceph-mon[93869]: pgmap v4506: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 70 KiB/s wr, 12 op/s 2023-03-09T17:02:55.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:02:54 smithi181 ceph-mon[94189]: pgmap v4506: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 70 KiB/s wr, 12 op/s 2023-03-09T17:02:55.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:02:54 smithi038 ceph-mon[92422]: pgmap v4506: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 70 KiB/s wr, 12 op/s 2023-03-09T17:02:56.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:02:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7486.038649 secs 2023-03-09T17:02:56.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:02:55 smithi122 ceph-mon[93869]: pgmap v4507: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 70 KiB/s wr, 13 op/s 2023-03-09T17:02:56.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:02:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7486.038649 secs 2023-03-09T17:02:56.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:02:55 smithi181 ceph-mon[94189]: pgmap v4507: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 70 KiB/s wr, 13 op/s 2023-03-09T17:02:56.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:02:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7486.038649 secs 2023-03-09T17:02:56.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:02:55 smithi038 ceph-mon[92422]: pgmap v4507: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 70 KiB/s wr, 13 op/s 2023-03-09T17:02:57.724 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:02:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7488.292388 secs 2023-03-09T17:02:57.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:02:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7488.292388 secs 2023-03-09T17:02:57.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:02:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7488.292388 secs 2023-03-09T17:02:58.724 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:02:58 smithi122 ceph-mon[93869]: pgmap v4508: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 60 KiB/s wr, 11 op/s 2023-03-09T17:02:58.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:02:58 smithi181 ceph-mon[94189]: pgmap v4508: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 60 KiB/s wr, 11 op/s 2023-03-09T17:02:58.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:02:58 smithi038 ceph-mon[92422]: pgmap v4508: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 60 KiB/s wr, 11 op/s 2023-03-09T17:02:58.793 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7950.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:03:01.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:03:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7491.038709 secs 2023-03-09T17:03:01.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:03:00 smithi122 ceph-mon[93869]: pgmap v4509: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 40 KiB/s wr, 10 op/s 2023-03-09T17:03:01.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:03:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7491.038709 secs 2023-03-09T17:03:01.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:03:00 smithi181 ceph-mon[94189]: pgmap v4509: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 40 KiB/s wr, 10 op/s 2023-03-09T17:03:01.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:03:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7491.038709 secs 2023-03-09T17:03:01.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:03:00 smithi038 ceph-mon[92422]: pgmap v4509: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 40 KiB/s wr, 10 op/s 2023-03-09T17:03:03.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:03:02 smithi122 ceph-mon[93869]: pgmap v4510: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 63 KiB/s wr, 13 op/s 2023-03-09T17:03:03.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:03:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7493.292527 secs 2023-03-09T17:03:03.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:03:02 smithi181 ceph-mon[94189]: pgmap v4510: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 63 KiB/s wr, 13 op/s 2023-03-09T17:03:03.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:03:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7493.292527 secs 2023-03-09T17:03:03.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:03:02 smithi038 ceph-mon[92422]: pgmap v4510: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 63 KiB/s wr, 13 op/s 2023-03-09T17:03:03.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:03:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7493.292527 secs 2023-03-09T17:03:03.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7955.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:03:05.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:03:04 smithi122 ceph-mon[93869]: pgmap v4511: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 96 KiB/s wr, 24 op/s 2023-03-09T17:03:05.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:03:04 smithi181 ceph-mon[94189]: pgmap v4511: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 96 KiB/s wr, 24 op/s 2023-03-09T17:03:05.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:03:04 smithi038 ceph-mon[92422]: pgmap v4511: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 96 KiB/s wr, 24 op/s 2023-03-09T17:03:06.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:03:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7496.038767 secs 2023-03-09T17:03:06.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:03:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7496.038767 secs 2023-03-09T17:03:06.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:03:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7496.038767 secs 2023-03-09T17:03:07.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:03:06 smithi122 ceph-mon[93869]: pgmap v4512: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 77 KiB/s wr, 23 op/s 2023-03-09T17:03:07.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:03:06 smithi181 ceph-mon[94189]: pgmap v4512: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 77 KiB/s wr, 23 op/s 2023-03-09T17:03:07.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:03:06 smithi038 ceph-mon[92422]: pgmap v4512: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 77 KiB/s wr, 23 op/s 2023-03-09T17:03:08.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:03:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7498.292641 secs 2023-03-09T17:03:08.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:03:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7498.292641 secs 2023-03-09T17:03:08.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:03:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7498.292641 secs 2023-03-09T17:03:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7960.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:03:09.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:03:08 smithi122 ceph-mon[93869]: pgmap v4513: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 84 KiB/s wr, 23 op/s 2023-03-09T17:03:09.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:03:08 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:03:09.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:03:08 smithi181 ceph-mon[94189]: pgmap v4513: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 84 KiB/s wr, 23 op/s 2023-03-09T17:03:09.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:03:08 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:03:09.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:03:08 smithi038 ceph-mon[92422]: pgmap v4513: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 84 KiB/s wr, 23 op/s 2023-03-09T17:03:09.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:03:08 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:03:10.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:03:09 smithi122 ceph-mon[93869]: osdmap e496: 12 total, 12 up, 12 in 2023-03-09T17:03:10.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:03:09 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:03:10.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:03:09 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:03:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:03:09 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:03:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:03:09 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:03:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:03:09 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:03:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:03:09 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:03:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:03:09 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:03:10.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:03:09 smithi181 ceph-mon[94189]: osdmap e496: 12 total, 12 up, 12 in 2023-03-09T17:03:10.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:03:09 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:03:10.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:03:09 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:03:10.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:03:09 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:03:10.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:03:09 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:03:10.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:03:09 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:03:10.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:03:09 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:03:10.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:03:09 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:03:10.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:03:09 smithi038 ceph-mon[92422]: osdmap e496: 12 total, 12 up, 12 in 2023-03-09T17:03:10.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:03:09 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:03:10.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:03:09 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:03:10.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:03:09 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:03:10.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:03:09 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:03:10.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:03:09 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:03:10.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:03:09 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:03:10.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:03:09 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:03:11.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:03:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7501.038828 secs 2023-03-09T17:03:11.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:03:10 smithi122 ceph-mon[93869]: pgmap v4515: 129 pgs: 7 active+clean+snaptrim_wait, 122 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 76 KiB/s wr, 25 op/s 2023-03-09T17:03:11.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:03:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7501.038828 secs 2023-03-09T17:03:11.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:03:10 smithi181 ceph-mon[94189]: pgmap v4515: 129 pgs: 7 active+clean+snaptrim_wait, 122 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 76 KiB/s wr, 25 op/s 2023-03-09T17:03:11.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:03:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7501.038828 secs 2023-03-09T17:03:11.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:03:10 smithi038 ceph-mon[92422]: pgmap v4515: 129 pgs: 7 active+clean+snaptrim_wait, 122 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 76 KiB/s wr, 25 op/s 2023-03-09T17:03:13.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:03:12 smithi122 ceph-mon[93869]: pgmap v4516: 129 pgs: 7 active+clean+snaptrim_wait, 122 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 50 KiB/s wr, 23 op/s 2023-03-09T17:03:13.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:03:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7503.292741 secs 2023-03-09T17:03:13.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:03:12 smithi181 ceph-mon[94189]: pgmap v4516: 129 pgs: 7 active+clean+snaptrim_wait, 122 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 50 KiB/s wr, 23 op/s 2023-03-09T17:03:13.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:03:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7503.292741 secs 2023-03-09T17:03:13.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:03:12 smithi038 ceph-mon[92422]: pgmap v4516: 129 pgs: 7 active+clean+snaptrim_wait, 122 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 50 KiB/s wr, 23 op/s 2023-03-09T17:03:13.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:03:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7503.292741 secs 2023-03-09T17:03:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7965.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:03:15.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:03:14 smithi122 ceph-mon[93869]: pgmap v4517: 129 pgs: 7 active+clean+snaptrim_wait, 122 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 28 KiB/s wr, 8 op/s 2023-03-09T17:03:15.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:03:14 smithi181 ceph-mon[94189]: pgmap v4517: 129 pgs: 7 active+clean+snaptrim_wait, 122 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 28 KiB/s wr, 8 op/s 2023-03-09T17:03:15.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:03:14 smithi038 ceph-mon[92422]: pgmap v4517: 129 pgs: 7 active+clean+snaptrim_wait, 122 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 28 KiB/s wr, 8 op/s 2023-03-09T17:03:16.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:03:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7506.038888 secs 2023-03-09T17:03:16.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:03:15 smithi122 ceph-mon[93869]: osdmap e497: 12 total, 12 up, 12 in 2023-03-09T17:03:16.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:03:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7506.038888 secs 2023-03-09T17:03:16.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:03:15 smithi181 ceph-mon[94189]: osdmap e497: 12 total, 12 up, 12 in 2023-03-09T17:03:16.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:03:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7506.038888 secs 2023-03-09T17:03:16.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:03:15 smithi038 ceph-mon[92422]: osdmap e497: 12 total, 12 up, 12 in 2023-03-09T17:03:17.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:03:16 smithi122 ceph-mon[93869]: pgmap v4519: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 8 op/s 2023-03-09T17:03:17.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:03:16 smithi181 ceph-mon[94189]: pgmap v4519: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 8 op/s 2023-03-09T17:03:17.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:03:16 smithi038 ceph-mon[92422]: pgmap v4519: 129 pgs: 129 active+clean; 8.9 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 8 op/s 2023-03-09T17:03:18.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:03:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7508.292839 secs 2023-03-09T17:03:18.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:03:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7508.292839 secs 2023-03-09T17:03:18.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:03:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7508.292839 secs 2023-03-09T17:03:18.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7970.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:03:19.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:03:18 smithi122 ceph-mon[93869]: pgmap v4520: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 56 KiB/s wr, 9 op/s 2023-03-09T17:03:19.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:03:18 smithi181 ceph-mon[94189]: pgmap v4520: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 56 KiB/s wr, 9 op/s 2023-03-09T17:03:19.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:03:18 smithi038 ceph-mon[92422]: pgmap v4520: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 56 KiB/s wr, 9 op/s 2023-03-09T17:03:20.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:03:19 smithi122 ceph-mon[93869]: pgmap v4521: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 8 op/s 2023-03-09T17:03:20.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:03:19 smithi181 ceph-mon[94189]: pgmap v4521: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 8 op/s 2023-03-09T17:03:20.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:03:19 smithi038 ceph-mon[92422]: pgmap v4521: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 8 op/s 2023-03-09T17:03:21.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:03:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7511.038947 secs 2023-03-09T17:03:21.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:03:20 smithi122 ceph-mon[93869]: osdmap e498: 12 total, 12 up, 12 in 2023-03-09T17:03:21.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:03:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7511.038947 secs 2023-03-09T17:03:21.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:03:20 smithi181 ceph-mon[94189]: osdmap e498: 12 total, 12 up, 12 in 2023-03-09T17:03:21.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:03:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7511.038947 secs 2023-03-09T17:03:21.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:03:20 smithi038 ceph-mon[92422]: osdmap e498: 12 total, 12 up, 12 in 2023-03-09T17:03:22.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:03:21 smithi122 ceph-mon[93869]: pgmap v4523: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 38 KiB/s wr, 7 op/s 2023-03-09T17:03:22.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:03:21 smithi181 ceph-mon[94189]: pgmap v4523: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 38 KiB/s wr, 7 op/s 2023-03-09T17:03:22.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:03:21 smithi038 ceph-mon[92422]: pgmap v4523: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 38 KiB/s wr, 7 op/s 2023-03-09T17:03:23.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:03:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7513.292965 secs 2023-03-09T17:03:23.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:03:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7513.292965 secs 2023-03-09T17:03:23.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:03:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7513.292965 secs 2023-03-09T17:03:23.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7975.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:03:24.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:03:23 smithi122 ceph-mon[93869]: pgmap v4524: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 36 KiB/s wr, 9 op/s 2023-03-09T17:03:24.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:03:23 smithi181 ceph-mon[94189]: pgmap v4524: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 36 KiB/s wr, 9 op/s 2023-03-09T17:03:24.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:03:23 smithi038 ceph-mon[92422]: pgmap v4524: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 36 KiB/s wr, 9 op/s 2023-03-09T17:03:25.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:03:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7516.038984 secs 2023-03-09T17:03:25.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:03:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7516.038984 secs 2023-03-09T17:03:25.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:03:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7516.038984 secs 2023-03-09T17:03:26.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:03:26 smithi122 ceph-mon[93869]: pgmap v4525: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 31 KiB/s wr, 8 op/s 2023-03-09T17:03:26.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:03:26 smithi181 ceph-mon[94189]: pgmap v4525: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 31 KiB/s wr, 8 op/s 2023-03-09T17:03:26.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:03:26 smithi038 ceph-mon[92422]: pgmap v4525: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 31 KiB/s wr, 8 op/s 2023-03-09T17:03:27.724 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:03:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7518.293029 secs 2023-03-09T17:03:27.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:03:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7518.293029 secs 2023-03-09T17:03:27.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:03:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7518.293029 secs 2023-03-09T17:03:28.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:03:28 smithi122 ceph-mon[93869]: pgmap v4526: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 818 B/s wr, 7 op/s 2023-03-09T17:03:28.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:03:28 smithi181 ceph-mon[94189]: pgmap v4526: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 818 B/s wr, 7 op/s 2023-03-09T17:03:28.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:03:28 smithi038 ceph-mon[92422]: pgmap v4526: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 818 B/s wr, 7 op/s 2023-03-09T17:03:28.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7980.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:03:31.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:03:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7521.039045 secs 2023-03-09T17:03:31.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:03:30 smithi122 ceph-mon[93869]: pgmap v4527: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 818 B/s wr, 7 op/s 2023-03-09T17:03:31.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:03:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7521.039045 secs 2023-03-09T17:03:31.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:03:30 smithi181 ceph-mon[94189]: pgmap v4527: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 818 B/s wr, 7 op/s 2023-03-09T17:03:31.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:03:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7521.039045 secs 2023-03-09T17:03:31.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:03:30 smithi038 ceph-mon[92422]: pgmap v4527: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 818 B/s wr, 7 op/s 2023-03-09T17:03:33.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:03:32 smithi122 ceph-mon[93869]: pgmap v4528: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 793 B/s wr, 8 op/s 2023-03-09T17:03:33.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:03:32 smithi181 ceph-mon[94189]: pgmap v4528: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 793 B/s wr, 8 op/s 2023-03-09T17:03:33.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:03:32 smithi038 ceph-mon[92422]: pgmap v4528: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 793 B/s wr, 8 op/s 2023-03-09T17:03:33.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7985.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:03:35.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:03:34 smithi122 ceph-mon[93869]: pgmap v4529: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T17:03:35.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:03:34 smithi181 ceph-mon[94189]: pgmap v4529: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T17:03:35.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:03:34 smithi038 ceph-mon[92422]: pgmap v4529: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T17:03:36.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:03:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7526.039103 secs 2023-03-09T17:03:36.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:03:35 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:03:36.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:03:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7526.039103 secs 2023-03-09T17:03:36.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:03:35 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:03:36.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:03:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7526.039103 secs 2023-03-09T17:03:36.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:03:35 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:03:37.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:03:36 smithi122 ceph-mon[93869]: pgmap v4530: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T17:03:37.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:03:36 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:03:37.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:03:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:03:37.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:03:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:03:37.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:03:36 smithi181 ceph-mon[94189]: pgmap v4530: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T17:03:37.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:03:36 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:03:37.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:03:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:03:37.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:03:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:03:37.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:03:36 smithi038 ceph-mon[92422]: pgmap v4530: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T17:03:37.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:03:36 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:03:37.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:03:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:03:37.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:03:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:03:38.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7990.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:03:39.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:03:38 smithi122 ceph-mon[93869]: pgmap v4531: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 7 op/s 2023-03-09T17:03:39.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:03:38 smithi181 ceph-mon[94189]: pgmap v4531: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 7 op/s 2023-03-09T17:03:39.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:03:38 smithi038 ceph-mon[92422]: pgmap v4531: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 7 op/s 2023-03-09T17:03:41.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:03:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7531.039162 secs 2023-03-09T17:03:41.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:03:40 smithi122 ceph-mon[93869]: pgmap v4532: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 22 KiB/s wr, 6 op/s 2023-03-09T17:03:41.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:03:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7531.039162 secs 2023-03-09T17:03:41.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:03:40 smithi181 ceph-mon[94189]: pgmap v4532: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 22 KiB/s wr, 6 op/s 2023-03-09T17:03:41.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:03:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7531.039162 secs 2023-03-09T17:03:41.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:03:40 smithi038 ceph-mon[92422]: pgmap v4532: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 22 KiB/s wr, 6 op/s 2023-03-09T17:03:43.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:03:42 smithi122 ceph-mon[93869]: pgmap v4533: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 22 KiB/s wr, 8 op/s 2023-03-09T17:03:43.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:03:42 smithi181 ceph-mon[94189]: pgmap v4533: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 22 KiB/s wr, 8 op/s 2023-03-09T17:03:43.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:03:42 smithi038 ceph-mon[92422]: pgmap v4533: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 22 KiB/s wr, 8 op/s 2023-03-09T17:03:43.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 7995.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:03:45.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:03:44 smithi122 ceph-mon[93869]: pgmap v4534: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 53 KiB/s wr, 9 op/s 2023-03-09T17:03:45.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:03:44 smithi181 ceph-mon[94189]: pgmap v4534: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 53 KiB/s wr, 9 op/s 2023-03-09T17:03:45.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:03:44 smithi038 ceph-mon[92422]: pgmap v4534: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 53 KiB/s wr, 9 op/s 2023-03-09T17:03:46.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:03:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7536.039224 secs 2023-03-09T17:03:46.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:03:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7536.039224 secs 2023-03-09T17:03:46.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:03:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7536.039224 secs 2023-03-09T17:03:47.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:03:46 smithi122 ceph-mon[93869]: pgmap v4535: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 40 KiB/s wr, 10 op/s 2023-03-09T17:03:47.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:03:46 smithi181 ceph-mon[94189]: pgmap v4535: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 40 KiB/s wr, 10 op/s 2023-03-09T17:03:47.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:03:46 smithi038 ceph-mon[92422]: pgmap v4535: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 40 KiB/s wr, 10 op/s 2023-03-09T17:03:48.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:03:47 smithi122 ceph-mon[93869]: pgmap v4536: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 70 KiB/s wr, 10 op/s 2023-03-09T17:03:48.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:03:47 smithi181 ceph-mon[94189]: pgmap v4536: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 70 KiB/s wr, 10 op/s 2023-03-09T17:03:48.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:03:47 smithi038 ceph-mon[92422]: pgmap v4536: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 70 KiB/s wr, 10 op/s 2023-03-09T17:03:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8000.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:03:51.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:03:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7541.039297 secs 2023-03-09T17:03:51.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:03:50 smithi122 ceph-mon[93869]: pgmap v4537: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 62 KiB/s wr, 9 op/s 2023-03-09T17:03:51.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:03:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7541.039297 secs 2023-03-09T17:03:51.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:03:50 smithi181 ceph-mon[94189]: pgmap v4537: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 62 KiB/s wr, 9 op/s 2023-03-09T17:03:51.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:03:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7541.039297 secs 2023-03-09T17:03:51.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:03:50 smithi038 ceph-mon[92422]: pgmap v4537: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 62 KiB/s wr, 9 op/s 2023-03-09T17:03:53.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:03:52 smithi122 ceph-mon[93869]: pgmap v4538: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 63 KiB/s wr, 11 op/s 2023-03-09T17:03:53.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:03:52 smithi181 ceph-mon[94189]: pgmap v4538: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 63 KiB/s wr, 11 op/s 2023-03-09T17:03:53.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:03:52 smithi038 ceph-mon[92422]: pgmap v4538: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 63 KiB/s wr, 11 op/s 2023-03-09T17:03:53.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8005.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:03:55.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:03:54 smithi122 ceph-mon[93869]: pgmap v4539: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 85 KiB/s wr, 11 op/s 2023-03-09T17:03:55.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:03:54 smithi181 ceph-mon[94189]: pgmap v4539: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 85 KiB/s wr, 11 op/s 2023-03-09T17:03:55.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:03:54 smithi038 ceph-mon[92422]: pgmap v4539: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 85 KiB/s wr, 11 op/s 2023-03-09T17:03:56.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:03:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7546.039359 secs 2023-03-09T17:03:56.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:03:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7546.039359 secs 2023-03-09T17:03:56.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:03:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7546.039359 secs 2023-03-09T17:03:57.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:03:56 smithi122 ceph-mon[93869]: pgmap v4540: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 54 KiB/s wr, 10 op/s 2023-03-09T17:03:57.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:03:56 smithi181 ceph-mon[94189]: pgmap v4540: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 54 KiB/s wr, 10 op/s 2023-03-09T17:03:57.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:03:56 smithi038 ceph-mon[92422]: pgmap v4540: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 54 KiB/s wr, 10 op/s 2023-03-09T17:03:58.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:03:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7548.293677 secs 2023-03-09T17:03:58.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:03:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7548.293677 secs 2023-03-09T17:03:58.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:03:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7548.293677 secs 2023-03-09T17:03:58.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8010.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:03:59.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:03:58 smithi122 ceph-mon[93869]: pgmap v4541: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 103 KiB/s wr, 10 op/s 2023-03-09T17:03:59.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:03:58 smithi181 ceph-mon[94189]: pgmap v4541: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 103 KiB/s wr, 10 op/s 2023-03-09T17:03:59.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:03:58 smithi038 ceph-mon[92422]: pgmap v4541: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 103 KiB/s wr, 10 op/s 2023-03-09T17:04:00.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:03:59 smithi122 ceph-mon[93869]: pgmap v4542: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 72 KiB/s wr, 8 op/s 2023-03-09T17:04:00.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:03:59 smithi181 ceph-mon[94189]: pgmap v4542: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 72 KiB/s wr, 8 op/s 2023-03-09T17:04:00.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:03:59 smithi038 ceph-mon[92422]: pgmap v4542: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 72 KiB/s wr, 8 op/s 2023-03-09T17:04:01.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:04:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7551.039420 secs 2023-03-09T17:04:01.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:04:00 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:04:01.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:04:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:04:01.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:04:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:04:01.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:04:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7551.039420 secs 2023-03-09T17:04:01.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:04:00 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:04:01.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:04:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:04:01.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:04:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:04:01.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:04:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7551.039420 secs 2023-03-09T17:04:01.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:04:00 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:04:01.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:04:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:04:01.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:04:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:04:02.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:04:01 smithi122 ceph-mon[93869]: pgmap v4543: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 95 KiB/s wr, 12 op/s 2023-03-09T17:04:02.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:04:01 smithi181 ceph-mon[94189]: pgmap v4543: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 95 KiB/s wr, 12 op/s 2023-03-09T17:04:02.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:04:01 smithi038 ceph-mon[92422]: pgmap v4543: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 95 KiB/s wr, 12 op/s 2023-03-09T17:04:03.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:04:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7553.293761 secs 2023-03-09T17:04:03.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:04:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7553.293761 secs 2023-03-09T17:04:03.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:04:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7553.293761 secs 2023-03-09T17:04:03.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8015.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:04:04.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:04:03 smithi122 ceph-mon[93869]: pgmap v4544: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 129 KiB/s wr, 24 op/s 2023-03-09T17:04:04.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:04:03 smithi181 ceph-mon[94189]: pgmap v4544: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 129 KiB/s wr, 24 op/s 2023-03-09T17:04:04.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:04:03 smithi038 ceph-mon[92422]: pgmap v4544: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 129 KiB/s wr, 24 op/s 2023-03-09T17:04:05.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:04:04 smithi122 ceph-mon[93869]: osdmap e499: 12 total, 12 up, 12 in 2023-03-09T17:04:05.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:04:04 smithi181 ceph-mon[94189]: osdmap e499: 12 total, 12 up, 12 in 2023-03-09T17:04:05.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:04:04 smithi038 ceph-mon[92422]: osdmap e499: 12 total, 12 up, 12 in 2023-03-09T17:04:06.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:04:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7556.039480 secs 2023-03-09T17:04:06.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:04:05 smithi122 ceph-mon[93869]: pgmap v4546: 129 pgs: 14 active+clean+snaptrim_wait, 115 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 128 KiB/s wr, 27 op/s 2023-03-09T17:04:06.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:04:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7556.039480 secs 2023-03-09T17:04:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:04:05 smithi181 ceph-mon[94189]: pgmap v4546: 129 pgs: 14 active+clean+snaptrim_wait, 115 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 128 KiB/s wr, 27 op/s 2023-03-09T17:04:06.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:04:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7556.039480 secs 2023-03-09T17:04:06.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:04:05 smithi038 ceph-mon[92422]: pgmap v4546: 129 pgs: 14 active+clean+snaptrim_wait, 115 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 128 KiB/s wr, 27 op/s 2023-03-09T17:04:07.724 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:04:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7558.293890 secs 2023-03-09T17:04:07.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:04:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7558.293890 secs 2023-03-09T17:04:07.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:04:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7558.293890 secs 2023-03-09T17:04:08.724 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:04:08 smithi122 ceph-mon[93869]: pgmap v4547: 129 pgs: 14 active+clean+snaptrim_wait, 115 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 101 KiB/s wr, 26 op/s 2023-03-09T17:04:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:04:08 smithi181 ceph-mon[94189]: pgmap v4547: 129 pgs: 14 active+clean+snaptrim_wait, 115 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 101 KiB/s wr, 26 op/s 2023-03-09T17:04:08.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:04:08 smithi038 ceph-mon[92422]: pgmap v4547: 129 pgs: 14 active+clean+snaptrim_wait, 115 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 101 KiB/s wr, 26 op/s 2023-03-09T17:04:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8020.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:04:11.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:04:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7561.039540 secs 2023-03-09T17:04:11.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:04:10 smithi122 ceph-mon[93869]: pgmap v4548: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 101 KiB/s wr, 26 op/s 2023-03-09T17:04:11.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:04:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:04:11.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:04:10 smithi122 ceph-mon[93869]: osdmap e500: 12 total, 12 up, 12 in 2023-03-09T17:04:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:04:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:04:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:04:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:04:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:04:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:04:11.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:04:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7561.039540 secs 2023-03-09T17:04:11.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:04:10 smithi181 ceph-mon[94189]: pgmap v4548: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 101 KiB/s wr, 26 op/s 2023-03-09T17:04:11.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:04:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:04:11.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:04:10 smithi181 ceph-mon[94189]: osdmap e500: 12 total, 12 up, 12 in 2023-03-09T17:04:11.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:04:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:04:11.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:04:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:04:11.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:04:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:04:11.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:04:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7561.039540 secs 2023-03-09T17:04:11.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:04:10 smithi038 ceph-mon[92422]: pgmap v4548: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 101 KiB/s wr, 26 op/s 2023-03-09T17:04:11.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:04:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:04:11.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:04:10 smithi038 ceph-mon[92422]: osdmap e500: 12 total, 12 up, 12 in 2023-03-09T17:04:11.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:04:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:04:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:04:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:04:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:04:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:04:13.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:04:12 smithi122 ceph-mon[93869]: pgmap v4550: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 8 op/s 2023-03-09T17:04:13.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:04:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7563.293991 secs 2023-03-09T17:04:13.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:04:12 smithi181 ceph-mon[94189]: pgmap v4550: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 8 op/s 2023-03-09T17:04:13.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:04:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7563.293991 secs 2023-03-09T17:04:13.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:04:12 smithi038 ceph-mon[92422]: pgmap v4550: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 8 op/s 2023-03-09T17:04:13.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:04:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7563.293991 secs 2023-03-09T17:04:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8025.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:04:15.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:04:14 smithi122 ceph-mon[93869]: pgmap v4551: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 53 KiB/s wr, 9 op/s 2023-03-09T17:04:15.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:04:14 smithi181 ceph-mon[94189]: pgmap v4551: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 53 KiB/s wr, 9 op/s 2023-03-09T17:04:15.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:04:14 smithi038 ceph-mon[92422]: pgmap v4551: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 53 KiB/s wr, 9 op/s 2023-03-09T17:04:16.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:04:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7566.039613 secs 2023-03-09T17:04:16.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:04:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7566.039613 secs 2023-03-09T17:04:16.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:04:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7566.039613 secs 2023-03-09T17:04:17.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:04:16 smithi122 ceph-mon[93869]: pgmap v4552: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 52 KiB/s wr, 8 op/s 2023-03-09T17:04:17.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:04:16 smithi181 ceph-mon[94189]: pgmap v4552: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 52 KiB/s wr, 8 op/s 2023-03-09T17:04:17.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:04:16 smithi038 ceph-mon[92422]: pgmap v4552: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 52 KiB/s wr, 8 op/s 2023-03-09T17:04:18.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:04:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7568.294131 secs 2023-03-09T17:04:18.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:04:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7568.294131 secs 2023-03-09T17:04:18.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:04:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7568.294131 secs 2023-03-09T17:04:18.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8030.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:04:19.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:04:18 smithi122 ceph-mon[93869]: pgmap v4553: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T17:04:19.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:04:18 smithi181 ceph-mon[94189]: pgmap v4553: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T17:04:19.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:04:18 smithi038 ceph-mon[92422]: pgmap v4553: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T17:04:20.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:04:19 smithi122 ceph-mon[93869]: pgmap v4554: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T17:04:20.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:04:19 smithi181 ceph-mon[94189]: pgmap v4554: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T17:04:20.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:04:19 smithi038 ceph-mon[92422]: pgmap v4554: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T17:04:21.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:04:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7571.039676 secs 2023-03-09T17:04:21.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:04:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7571.039676 secs 2023-03-09T17:04:21.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:04:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7571.039676 secs 2023-03-09T17:04:22.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:04:21 smithi122 ceph-mon[93869]: pgmap v4555: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 8 op/s 2023-03-09T17:04:22.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:04:21 smithi181 ceph-mon[94189]: pgmap v4555: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 8 op/s 2023-03-09T17:04:22.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:04:21 smithi038 ceph-mon[92422]: pgmap v4555: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 8 op/s 2023-03-09T17:04:23.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:04:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7573.294224 secs 2023-03-09T17:04:23.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:04:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7573.294224 secs 2023-03-09T17:04:23.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:04:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7573.294224 secs 2023-03-09T17:04:23.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8035.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:04:24.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:04:23 smithi122 ceph-mon[93869]: pgmap v4556: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 40 KiB/s wr, 8 op/s 2023-03-09T17:04:24.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:04:23 smithi181 ceph-mon[94189]: pgmap v4556: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 40 KiB/s wr, 8 op/s 2023-03-09T17:04:24.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:04:23 smithi038 ceph-mon[92422]: pgmap v4556: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 40 KiB/s wr, 8 op/s 2023-03-09T17:04:25.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:04:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7576.039734 secs 2023-03-09T17:04:25.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:04:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7576.039734 secs 2023-03-09T17:04:25.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:04:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7576.039734 secs 2023-03-09T17:04:26.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:04:26 smithi122 ceph-mon[93869]: pgmap v4557: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 8 op/s 2023-03-09T17:04:26.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:04:26 smithi181 ceph-mon[94189]: pgmap v4557: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 8 op/s 2023-03-09T17:04:26.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:04:26 smithi038 ceph-mon[92422]: pgmap v4557: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 8 op/s 2023-03-09T17:04:27.724 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:04:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7578.294341 secs 2023-03-09T17:04:27.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:04:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7578.294341 secs 2023-03-09T17:04:27.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:04:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7578.294341 secs 2023-03-09T17:04:28.724 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:04:28 smithi122 ceph-mon[93869]: pgmap v4558: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 25 KiB/s wr, 8 op/s 2023-03-09T17:04:28.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:04:28 smithi181 ceph-mon[94189]: pgmap v4558: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 25 KiB/s wr, 8 op/s 2023-03-09T17:04:28.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:04:28 smithi038 ceph-mon[92422]: pgmap v4558: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 25 KiB/s wr, 8 op/s 2023-03-09T17:04:28.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8040.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:04:31.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:04:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7581.039794 secs 2023-03-09T17:04:31.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:04:30 smithi122 ceph-mon[93869]: pgmap v4559: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 25 KiB/s wr, 7 op/s 2023-03-09T17:04:31.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:04:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7581.039794 secs 2023-03-09T17:04:31.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:04:30 smithi181 ceph-mon[94189]: pgmap v4559: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 25 KiB/s wr, 7 op/s 2023-03-09T17:04:31.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:04:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7581.039794 secs 2023-03-09T17:04:31.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:04:30 smithi038 ceph-mon[92422]: pgmap v4559: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 25 KiB/s wr, 7 op/s 2023-03-09T17:04:33.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:04:32 smithi122 ceph-mon[93869]: pgmap v4560: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 8 op/s 2023-03-09T17:04:33.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:04:32 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 7583.294464 secs 2023-03-09T17:04:33.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:04:32 smithi181 ceph-mon[94189]: pgmap v4560: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 8 op/s 2023-03-09T17:04:33.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:04:32 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 7583.294464 secs 2023-03-09T17:04:33.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:04:32 smithi038 ceph-mon[92422]: pgmap v4560: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 8 op/s 2023-03-09T17:04:33.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:04:32 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 7583.294464 secs 2023-03-09T17:04:33.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8045.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:04:35.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:04:34 smithi122 ceph-mon[93869]: pgmap v4561: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 38 KiB/s wr, 8 op/s 2023-03-09T17:04:35.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:04:34 smithi181 ceph-mon[94189]: pgmap v4561: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 38 KiB/s wr, 8 op/s 2023-03-09T17:04:35.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:04:34 smithi038 ceph-mon[92422]: pgmap v4561: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 38 KiB/s wr, 8 op/s 2023-03-09T17:04:36.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:04:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7586.039855 secs 2023-03-09T17:04:36.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:04:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7586.039855 secs 2023-03-09T17:04:36.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:04:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7586.039855 secs 2023-03-09T17:04:37.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:04:36 smithi122 ceph-mon[93869]: pgmap v4562: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 15 KiB/s wr, 8 op/s 2023-03-09T17:04:37.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:04:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:04:37.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:04:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:04:37.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:04:36 smithi181 ceph-mon[94189]: pgmap v4562: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 15 KiB/s wr, 8 op/s 2023-03-09T17:04:37.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:04:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:04:37.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:04:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:04:37.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:04:36 smithi038 ceph-mon[92422]: pgmap v4562: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 15 KiB/s wr, 8 op/s 2023-03-09T17:04:37.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:04:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:04:37.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:04:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:04:38.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:04:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7588.294582 secs 2023-03-09T17:04:38.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:04:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7588.294582 secs 2023-03-09T17:04:38.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:04:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7588.294582 secs 2023-03-09T17:04:38.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8050.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:04:39.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:04:38 smithi122 ceph-mon[93869]: pgmap v4563: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 46 KiB/s wr, 8 op/s 2023-03-09T17:04:39.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:04:38 smithi181 ceph-mon[94189]: pgmap v4563: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 46 KiB/s wr, 8 op/s 2023-03-09T17:04:39.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:04:38 smithi038 ceph-mon[92422]: pgmap v4563: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 46 KiB/s wr, 8 op/s 2023-03-09T17:04:40.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:04:39 smithi122 ceph-mon[93869]: pgmap v4564: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 46 KiB/s wr, 8 op/s 2023-03-09T17:04:40.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:04:39 smithi181 ceph-mon[94189]: pgmap v4564: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 46 KiB/s wr, 8 op/s 2023-03-09T17:04:40.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:04:39 smithi038 ceph-mon[92422]: pgmap v4564: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 46 KiB/s wr, 8 op/s 2023-03-09T17:04:41.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:04:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7591.039940 secs 2023-03-09T17:04:41.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:04:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:04:41.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:04:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:04:41.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:04:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7591.039940 secs 2023-03-09T17:04:41.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:04:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:04:41.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:04:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:04:41.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:04:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7591.039940 secs 2023-03-09T17:04:41.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:04:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:04:41.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:04:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:04:42.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:04:41 smithi122 ceph-mon[93869]: pgmap v4565: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 46 KiB/s wr, 10 op/s 2023-03-09T17:04:42.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:04:41 smithi181 ceph-mon[94189]: pgmap v4565: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 46 KiB/s wr, 10 op/s 2023-03-09T17:04:42.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:04:41 smithi038 ceph-mon[92422]: pgmap v4565: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 46 KiB/s wr, 10 op/s 2023-03-09T17:04:43.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:04:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7593.294702 secs 2023-03-09T17:04:43.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:04:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7593.294702 secs 2023-03-09T17:04:43.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:04:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7593.294702 secs 2023-03-09T17:04:43.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8055.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:04:44.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:04:43 smithi122 ceph-mon[93869]: pgmap v4566: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 70 KiB/s wr, 10 op/s 2023-03-09T17:04:44.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:04:43 smithi181 ceph-mon[94189]: pgmap v4566: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 70 KiB/s wr, 10 op/s 2023-03-09T17:04:44.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:04:43 smithi038 ceph-mon[92422]: pgmap v4566: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 70 KiB/s wr, 10 op/s 2023-03-09T17:04:45.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:04:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7596.039991 secs 2023-03-09T17:04:45.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:04:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7596.039991 secs 2023-03-09T17:04:45.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:04:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7596.039991 secs 2023-03-09T17:04:46.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:04:46 smithi122 ceph-mon[93869]: pgmap v4567: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 81 KiB/s wr, 12 op/s 2023-03-09T17:04:46.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:04:46 smithi181 ceph-mon[94189]: pgmap v4567: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 81 KiB/s wr, 12 op/s 2023-03-09T17:04:46.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:04:46 smithi038 ceph-mon[92422]: pgmap v4567: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 81 KiB/s wr, 12 op/s 2023-03-09T17:04:47.724 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:04:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7598.294829 secs 2023-03-09T17:04:47.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:04:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7598.294829 secs 2023-03-09T17:04:47.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:04:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7598.294829 secs 2023-03-09T17:04:48.724 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:04:48 smithi122 ceph-mon[93869]: pgmap v4568: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 109 KiB/s wr, 12 op/s 2023-03-09T17:04:48.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:04:48 smithi181 ceph-mon[94189]: pgmap v4568: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 109 KiB/s wr, 12 op/s 2023-03-09T17:04:48.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:04:48 smithi038 ceph-mon[92422]: pgmap v4568: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 109 KiB/s wr, 12 op/s 2023-03-09T17:04:48.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8060.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:04:51.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:04:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7601.040050 secs 2023-03-09T17:04:51.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:04:50 smithi122 ceph-mon[93869]: pgmap v4569: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 77 KiB/s wr, 10 op/s 2023-03-09T17:04:51.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:04:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7601.040050 secs 2023-03-09T17:04:51.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:04:50 smithi181 ceph-mon[94189]: pgmap v4569: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 77 KiB/s wr, 10 op/s 2023-03-09T17:04:51.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:04:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7601.040050 secs 2023-03-09T17:04:51.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:04:50 smithi038 ceph-mon[92422]: pgmap v4569: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 77 KiB/s wr, 10 op/s 2023-03-09T17:04:53.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:04:52 smithi122 ceph-mon[93869]: pgmap v4570: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 90 KiB/s wr, 12 op/s 2023-03-09T17:04:53.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:04:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7603.294951 secs 2023-03-09T17:04:53.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:04:52 smithi181 ceph-mon[94189]: pgmap v4570: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 90 KiB/s wr, 12 op/s 2023-03-09T17:04:53.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:04:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7603.294951 secs 2023-03-09T17:04:53.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:04:52 smithi038 ceph-mon[92422]: pgmap v4570: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 90 KiB/s wr, 12 op/s 2023-03-09T17:04:53.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:04:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7603.294951 secs 2023-03-09T17:04:53.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8065.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:04:54.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:04:53 smithi122 ceph-mon[93869]: pgmap v4571: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 96 KiB/s wr, 11 op/s 2023-03-09T17:04:54.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:04:53 smithi181 ceph-mon[94189]: pgmap v4571: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 96 KiB/s wr, 11 op/s 2023-03-09T17:04:54.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:04:53 smithi038 ceph-mon[92422]: pgmap v4571: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 96 KiB/s wr, 11 op/s 2023-03-09T17:04:55.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:04:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7606.040113 secs 2023-03-09T17:04:55.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:04:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7606.040113 secs 2023-03-09T17:04:55.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:04:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7606.040113 secs 2023-03-09T17:04:56.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:04:56 smithi122 ceph-mon[93869]: pgmap v4572: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 93 KiB/s wr, 12 op/s 2023-03-09T17:04:56.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:04:56 smithi181 ceph-mon[94189]: pgmap v4572: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 93 KiB/s wr, 12 op/s 2023-03-09T17:04:56.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:04:56 smithi038 ceph-mon[92422]: pgmap v4572: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 93 KiB/s wr, 12 op/s 2023-03-09T17:04:57.724 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:04:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7608.295071 secs 2023-03-09T17:04:57.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:04:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7608.295071 secs 2023-03-09T17:04:57.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:04:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7608.295071 secs 2023-03-09T17:04:58.724 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:04:58 smithi122 ceph-mon[93869]: pgmap v4573: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 68 KiB/s wr, 10 op/s 2023-03-09T17:04:58.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:04:58 smithi181 ceph-mon[94189]: pgmap v4573: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 68 KiB/s wr, 10 op/s 2023-03-09T17:04:58.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:04:58 smithi038 ceph-mon[92422]: pgmap v4573: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 68 KiB/s wr, 10 op/s 2023-03-09T17:04:58.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8070.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:05:01.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:05:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7611.040173 secs 2023-03-09T17:05:01.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:05:00 smithi122 ceph-mon[93869]: pgmap v4574: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 40 KiB/s wr, 7 op/s 2023-03-09T17:05:01.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:05:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7611.040173 secs 2023-03-09T17:05:01.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:05:00 smithi181 ceph-mon[94189]: pgmap v4574: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 40 KiB/s wr, 7 op/s 2023-03-09T17:05:01.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:05:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7611.040173 secs 2023-03-09T17:05:01.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:05:00 smithi038 ceph-mon[92422]: pgmap v4574: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 40 KiB/s wr, 7 op/s 2023-03-09T17:05:03.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:05:02 smithi122 ceph-mon[93869]: pgmap v4575: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 63 KiB/s wr, 11 op/s 2023-03-09T17:05:03.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:05:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7613.295197 secs 2023-03-09T17:05:03.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:05:02 smithi181 ceph-mon[94189]: pgmap v4575: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 63 KiB/s wr, 11 op/s 2023-03-09T17:05:03.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:05:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7613.295197 secs 2023-03-09T17:05:03.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:05:02 smithi038 ceph-mon[92422]: pgmap v4575: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 63 KiB/s wr, 11 op/s 2023-03-09T17:05:03.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:05:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7613.295197 secs 2023-03-09T17:05:03.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8075.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:05:05.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:05:04 smithi122 ceph-mon[93869]: pgmap v4576: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 60 KiB/s wr, 19 op/s 2023-03-09T17:05:05.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:05:04 smithi181 ceph-mon[94189]: pgmap v4576: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 60 KiB/s wr, 19 op/s 2023-03-09T17:05:05.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:05:04 smithi038 ceph-mon[92422]: pgmap v4576: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 60 KiB/s wr, 19 op/s 2023-03-09T17:05:06.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:05:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7616.040235 secs 2023-03-09T17:05:06.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:05:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7616.040235 secs 2023-03-09T17:05:06.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:05:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7616.040235 secs 2023-03-09T17:05:07.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:05:06 smithi122 ceph-mon[93869]: pgmap v4577: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 55 KiB/s wr, 21 op/s 2023-03-09T17:05:07.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:05:06 smithi181 ceph-mon[94189]: pgmap v4577: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 55 KiB/s wr, 21 op/s 2023-03-09T17:05:07.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:05:06 smithi038 ceph-mon[92422]: pgmap v4577: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 55 KiB/s wr, 21 op/s 2023-03-09T17:05:08.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:05:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7618.295269 secs 2023-03-09T17:05:08.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:05:07 smithi122 ceph-mon[93869]: pgmap v4578: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 41 KiB/s wr, 19 op/s 2023-03-09T17:05:08.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:05:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7618.295269 secs 2023-03-09T17:05:08.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:05:07 smithi181 ceph-mon[94189]: pgmap v4578: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 41 KiB/s wr, 19 op/s 2023-03-09T17:05:08.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:05:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7618.295269 secs 2023-03-09T17:05:08.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:05:07 smithi038 ceph-mon[92422]: pgmap v4578: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 41 KiB/s wr, 19 op/s 2023-03-09T17:05:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8080.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:05:11.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:05:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7621.040296 secs 2023-03-09T17:05:11.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:05:10 smithi122 ceph-mon[93869]: pgmap v4579: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 41 KiB/s wr, 19 op/s 2023-03-09T17:05:11.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:05:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:05:11.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:05:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7621.040296 secs 2023-03-09T17:05:11.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:05:10 smithi181 ceph-mon[94189]: pgmap v4579: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 41 KiB/s wr, 19 op/s 2023-03-09T17:05:11.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:05:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:05:11.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:05:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7621.040296 secs 2023-03-09T17:05:11.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:05:10 smithi038 ceph-mon[92422]: pgmap v4579: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 41 KiB/s wr, 19 op/s 2023-03-09T17:05:11.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:05:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:05:12.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:05:11 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:05:12.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:05:11 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:05:12.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:05:11 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:05:12.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:05:11 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:05:12.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:05:11 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:05:12.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:05:11 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:05:12.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:05:11 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:05:12.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:05:11 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:05:12.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:05:11 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:05:13.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:05:12 smithi122 ceph-mon[93869]: pgmap v4580: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 51 KiB/s wr, 21 op/s 2023-03-09T17:05:13.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:05:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7623.295400 secs 2023-03-09T17:05:13.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:05:12 smithi181 ceph-mon[94189]: pgmap v4580: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 51 KiB/s wr, 21 op/s 2023-03-09T17:05:13.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:05:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7623.295400 secs 2023-03-09T17:05:13.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:05:12 smithi038 ceph-mon[92422]: pgmap v4580: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 51 KiB/s wr, 21 op/s 2023-03-09T17:05:13.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:05:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7623.295400 secs 2023-03-09T17:05:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8085.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:05:15.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:05:14 smithi122 ceph-mon[93869]: pgmap v4581: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 35 KiB/s wr, 18 op/s 2023-03-09T17:05:15.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:05:14 smithi181 ceph-mon[94189]: pgmap v4581: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 35 KiB/s wr, 18 op/s 2023-03-09T17:05:15.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:05:14 smithi038 ceph-mon[92422]: pgmap v4581: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 35 KiB/s wr, 18 op/s 2023-03-09T17:05:16.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:05:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7626.040356 secs 2023-03-09T17:05:16.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:05:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7626.040356 secs 2023-03-09T17:05:16.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:05:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7626.040356 secs 2023-03-09T17:05:17.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:05:16 smithi122 ceph-mon[93869]: pgmap v4582: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 39 KiB/s wr, 9 op/s 2023-03-09T17:05:17.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:05:16 smithi181 ceph-mon[94189]: pgmap v4582: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 39 KiB/s wr, 9 op/s 2023-03-09T17:05:17.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:05:16 smithi038 ceph-mon[92422]: pgmap v4582: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 39 KiB/s wr, 9 op/s 2023-03-09T17:05:18.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:05:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7628.295541 secs 2023-03-09T17:05:18.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:05:17 smithi122 ceph-mon[93869]: pgmap v4583: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 38 KiB/s wr, 7 op/s 2023-03-09T17:05:18.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:05:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7628.295541 secs 2023-03-09T17:05:18.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:05:17 smithi181 ceph-mon[94189]: pgmap v4583: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 38 KiB/s wr, 7 op/s 2023-03-09T17:05:18.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:05:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7628.295541 secs 2023-03-09T17:05:18.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:05:17 smithi038 ceph-mon[92422]: pgmap v4583: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 38 KiB/s wr, 7 op/s 2023-03-09T17:05:18.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8090.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:05:21.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:05:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7631.040417 secs 2023-03-09T17:05:21.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:05:20 smithi122 ceph-mon[93869]: pgmap v4584: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T17:05:21.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:05:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7631.040417 secs 2023-03-09T17:05:21.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:05:20 smithi181 ceph-mon[94189]: pgmap v4584: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T17:05:21.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:05:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7631.040417 secs 2023-03-09T17:05:21.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:05:20 smithi038 ceph-mon[92422]: pgmap v4584: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T17:05:23.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:05:22 smithi122 ceph-mon[93869]: pgmap v4585: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 32 KiB/s wr, 9 op/s 2023-03-09T17:05:23.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:05:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7633.295682 secs 2023-03-09T17:05:23.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:05:22 smithi181 ceph-mon[94189]: pgmap v4585: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 32 KiB/s wr, 9 op/s 2023-03-09T17:05:23.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:05:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7633.295682 secs 2023-03-09T17:05:23.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:05:22 smithi038 ceph-mon[92422]: pgmap v4585: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 32 KiB/s wr, 9 op/s 2023-03-09T17:05:23.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:05:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7633.295682 secs 2023-03-09T17:05:23.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8095.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:05:25.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:05:24 smithi122 ceph-mon[93869]: pgmap v4586: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T17:05:25.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:05:24 smithi181 ceph-mon[94189]: pgmap v4586: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T17:05:25.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:05:24 smithi038 ceph-mon[92422]: pgmap v4586: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T17:05:26.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:05:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7636.040477 secs 2023-03-09T17:05:26.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:05:25 smithi122 ceph-mon[93869]: pgmap v4587: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 43 KiB/s wr, 9 op/s 2023-03-09T17:05:26.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:05:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7636.040477 secs 2023-03-09T17:05:26.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:05:25 smithi181 ceph-mon[94189]: pgmap v4587: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 43 KiB/s wr, 9 op/s 2023-03-09T17:05:26.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:05:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7636.040477 secs 2023-03-09T17:05:26.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:05:25 smithi038 ceph-mon[92422]: pgmap v4587: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 43 KiB/s wr, 9 op/s 2023-03-09T17:05:27.724 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:05:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7638.295809 secs 2023-03-09T17:05:27.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:05:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7638.295809 secs 2023-03-09T17:05:27.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:05:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7638.295809 secs 2023-03-09T17:05:28.724 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:05:28 smithi122 ceph-mon[93869]: pgmap v4588: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T17:05:28.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:05:28 smithi181 ceph-mon[94189]: pgmap v4588: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T17:05:28.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:05:28 smithi038 ceph-mon[92422]: pgmap v4588: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T17:05:28.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8100.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:05:31.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:05:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7641.040538 secs 2023-03-09T17:05:31.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:05:30 smithi122 ceph-mon[93869]: pgmap v4589: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 28 KiB/s wr, 6 op/s 2023-03-09T17:05:31.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:05:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7641.040538 secs 2023-03-09T17:05:31.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:05:30 smithi181 ceph-mon[94189]: pgmap v4589: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 28 KiB/s wr, 6 op/s 2023-03-09T17:05:31.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:05:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7641.040538 secs 2023-03-09T17:05:31.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:05:30 smithi038 ceph-mon[92422]: pgmap v4589: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 28 KiB/s wr, 6 op/s 2023-03-09T17:05:33.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:05:32 smithi122 ceph-mon[93869]: pgmap v4590: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 28 KiB/s wr, 8 op/s 2023-03-09T17:05:33.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:05:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7643.295953 secs 2023-03-09T17:05:33.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:05:32 smithi181 ceph-mon[94189]: pgmap v4590: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 28 KiB/s wr, 8 op/s 2023-03-09T17:05:33.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:05:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7643.295953 secs 2023-03-09T17:05:33.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:05:32 smithi038 ceph-mon[92422]: pgmap v4590: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 28 KiB/s wr, 8 op/s 2023-03-09T17:05:33.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:05:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7643.295953 secs 2023-03-09T17:05:33.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8105.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:05:35.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:05:34 smithi122 ceph-mon[93869]: pgmap v4591: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 6 op/s 2023-03-09T17:05:35.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:05:34 smithi181 ceph-mon[94189]: pgmap v4591: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 6 op/s 2023-03-09T17:05:35.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:05:34 smithi038 ceph-mon[92422]: pgmap v4591: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 6 op/s 2023-03-09T17:05:36.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:05:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7646.040599 secs 2023-03-09T17:05:36.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:05:35 smithi122 ceph-mon[93869]: pgmap v4592: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 39 KiB/s wr, 8 op/s 2023-03-09T17:05:36.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:05:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7646.040599 secs 2023-03-09T17:05:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:05:35 smithi181 ceph-mon[94189]: pgmap v4592: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 39 KiB/s wr, 8 op/s 2023-03-09T17:05:36.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:05:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7646.040599 secs 2023-03-09T17:05:36.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:05:35 smithi038 ceph-mon[92422]: pgmap v4592: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 39 KiB/s wr, 8 op/s 2023-03-09T17:05:37.724 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:05:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7648.296071 secs 2023-03-09T17:05:37.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:05:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7648.296071 secs 2023-03-09T17:05:37.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:05:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7648.296071 secs 2023-03-09T17:05:38.724 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:05:38 smithi122 ceph-mon[93869]: pgmap v4593: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 6 op/s 2023-03-09T17:05:38.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:05:38 smithi181 ceph-mon[94189]: pgmap v4593: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 6 op/s 2023-03-09T17:05:38.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:05:38 smithi038 ceph-mon[92422]: pgmap v4593: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 6 op/s 2023-03-09T17:05:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8110.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:05:41.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:05:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7651.040683 secs 2023-03-09T17:05:41.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:05:40 smithi122 ceph-mon[93869]: pgmap v4594: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 5 op/s 2023-03-09T17:05:41.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:05:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7651.040683 secs 2023-03-09T17:05:41.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:05:40 smithi181 ceph-mon[94189]: pgmap v4594: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 5 op/s 2023-03-09T17:05:41.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:05:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7651.040683 secs 2023-03-09T17:05:41.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:05:40 smithi038 ceph-mon[92422]: pgmap v4594: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 5 op/s 2023-03-09T17:05:43.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:05:42 smithi122 ceph-mon[93869]: pgmap v4595: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 36 KiB/s wr, 8 op/s 2023-03-09T17:05:43.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:05:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7653.296166 secs 2023-03-09T17:05:43.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:05:42 smithi181 ceph-mon[94189]: pgmap v4595: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 36 KiB/s wr, 8 op/s 2023-03-09T17:05:43.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:05:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7653.296166 secs 2023-03-09T17:05:43.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:05:42 smithi038 ceph-mon[92422]: pgmap v4595: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 36 KiB/s wr, 8 op/s 2023-03-09T17:05:43.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:05:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7653.296166 secs 2023-03-09T17:05:43.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8115.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:05:45.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:05:44 smithi122 ceph-mon[93869]: pgmap v4596: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 6 op/s 2023-03-09T17:05:45.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:05:44 smithi181 ceph-mon[94189]: pgmap v4596: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 6 op/s 2023-03-09T17:05:45.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:05:44 smithi038 ceph-mon[92422]: pgmap v4596: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 6 op/s 2023-03-09T17:05:46.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:05:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7656.040771 secs 2023-03-09T17:05:46.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:05:45 smithi122 ceph-mon[93869]: pgmap v4597: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 45 KiB/s wr, 9 op/s 2023-03-09T17:05:46.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:05:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7656.040771 secs 2023-03-09T17:05:46.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:05:45 smithi181 ceph-mon[94189]: pgmap v4597: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 45 KiB/s wr, 9 op/s 2023-03-09T17:05:46.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:05:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7656.040771 secs 2023-03-09T17:05:46.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:05:45 smithi038 ceph-mon[92422]: pgmap v4597: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 45 KiB/s wr, 9 op/s 2023-03-09T17:05:47.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:05:47 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:05:47.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:05:47 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:05:47.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:05:47 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:05:48.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:05:48 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:05:48.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:05:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:05:48.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:05:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:05:48.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:05:48 smithi122 ceph-mon[93869]: pgmap v4598: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 39 KiB/s wr, 8 op/s 2023-03-09T17:05:48.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:05:48 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:05:48.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:05:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:05:48.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:05:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:05:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:05:48 smithi181 ceph-mon[94189]: pgmap v4598: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 39 KiB/s wr, 8 op/s 2023-03-09T17:05:48.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:05:48 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:05:48.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:05:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:05:48.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:05:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:05:48.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:05:48 smithi038 ceph-mon[92422]: pgmap v4598: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 39 KiB/s wr, 8 op/s 2023-03-09T17:05:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8120.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:05:51.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:05:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7661.040833 secs 2023-03-09T17:05:51.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:05:50 smithi122 ceph-mon[93869]: pgmap v4599: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 39 KiB/s wr, 8 op/s 2023-03-09T17:05:51.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:05:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7661.040833 secs 2023-03-09T17:05:51.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:05:50 smithi181 ceph-mon[94189]: pgmap v4599: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 39 KiB/s wr, 8 op/s 2023-03-09T17:05:51.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:05:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7661.040833 secs 2023-03-09T17:05:51.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:05:50 smithi038 ceph-mon[92422]: pgmap v4599: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 39 KiB/s wr, 8 op/s 2023-03-09T17:05:53.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:05:52 smithi122 ceph-mon[93869]: pgmap v4600: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 87 KiB/s wr, 11 op/s 2023-03-09T17:05:53.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:05:52 smithi181 ceph-mon[94189]: pgmap v4600: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 87 KiB/s wr, 11 op/s 2023-03-09T17:05:53.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:05:52 smithi038 ceph-mon[92422]: pgmap v4600: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 87 KiB/s wr, 11 op/s 2023-03-09T17:05:53.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8125.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:05:54.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:05:53 smithi122 ceph-mon[93869]: pgmap v4601: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 71 KiB/s wr, 10 op/s 2023-03-09T17:05:54.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:05:53 smithi181 ceph-mon[94189]: pgmap v4601: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 71 KiB/s wr, 10 op/s 2023-03-09T17:05:54.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:05:53 smithi038 ceph-mon[92422]: pgmap v4601: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 71 KiB/s wr, 10 op/s 2023-03-09T17:05:55.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:05:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7666.040894 secs 2023-03-09T17:05:55.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:05:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7666.040894 secs 2023-03-09T17:05:55.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:05:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7666.040894 secs 2023-03-09T17:05:56.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:05:56 smithi122 ceph-mon[93869]: pgmap v4602: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 85 KiB/s wr, 12 op/s 2023-03-09T17:05:56.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:05:56 smithi181 ceph-mon[94189]: pgmap v4602: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 85 KiB/s wr, 12 op/s 2023-03-09T17:05:56.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:05:56 smithi038 ceph-mon[92422]: pgmap v4602: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 85 KiB/s wr, 12 op/s 2023-03-09T17:05:58.793 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8130.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:05:59.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:05:58 smithi122 ceph-mon[93869]: pgmap v4603: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 88 KiB/s wr, 10 op/s 2023-03-09T17:05:59.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:05:58 smithi181 ceph-mon[94189]: pgmap v4603: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 88 KiB/s wr, 10 op/s 2023-03-09T17:05:59.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:05:58 smithi038 ceph-mon[92422]: pgmap v4603: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 88 KiB/s wr, 10 op/s 2023-03-09T17:06:01.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:06:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7671.040956 secs 2023-03-09T17:06:01.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:06:00 smithi122 ceph-mon[93869]: pgmap v4604: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 81 KiB/s wr, 9 op/s 2023-03-09T17:06:01.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:06:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7671.040956 secs 2023-03-09T17:06:01.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:06:00 smithi181 ceph-mon[94189]: pgmap v4604: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 81 KiB/s wr, 9 op/s 2023-03-09T17:06:01.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:06:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7671.040956 secs 2023-03-09T17:06:01.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:06:00 smithi038 ceph-mon[92422]: pgmap v4604: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 81 KiB/s wr, 9 op/s 2023-03-09T17:06:02.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:06:01 smithi122 ceph-mon[93869]: pgmap v4605: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 138 KiB/s wr, 14 op/s 2023-03-09T17:06:02.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:06:01 smithi181 ceph-mon[94189]: pgmap v4605: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 138 KiB/s wr, 14 op/s 2023-03-09T17:06:02.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:06:01 smithi038 ceph-mon[92422]: pgmap v4605: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 138 KiB/s wr, 14 op/s 2023-03-09T17:06:03.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:06:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7673.296604 secs 2023-03-09T17:06:03.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:06:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7673.296604 secs 2023-03-09T17:06:03.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:06:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7673.296604 secs 2023-03-09T17:06:03.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8135.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:06:04.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:06:03 smithi122 ceph-mon[93869]: pgmap v4606: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 119 KiB/s wr, 24 op/s 2023-03-09T17:06:04.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:06:03 smithi181 ceph-mon[94189]: pgmap v4606: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 119 KiB/s wr, 24 op/s 2023-03-09T17:06:04.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:06:03 smithi038 ceph-mon[92422]: pgmap v4606: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 119 KiB/s wr, 24 op/s 2023-03-09T17:06:05.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:06:04 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:06:05.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:06:04 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:06:05.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:06:04 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:06:05.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:06:04 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:06:05.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:06:04 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:06:05.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:06:04 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:06:05.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:06:04 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:06:05.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:06:04 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:06:05.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:06:04 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:06:06.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:06:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7676.041016 secs 2023-03-09T17:06:06.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:06:05 smithi122 ceph-mon[93869]: pgmap v4607: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 113 KiB/s wr, 25 op/s 2023-03-09T17:06:06.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:06:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7676.041016 secs 2023-03-09T17:06:06.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:06:05 smithi181 ceph-mon[94189]: pgmap v4607: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 113 KiB/s wr, 25 op/s 2023-03-09T17:06:06.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:06:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7676.041016 secs 2023-03-09T17:06:06.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:06:05 smithi038 ceph-mon[92422]: pgmap v4607: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 113 KiB/s wr, 25 op/s 2023-03-09T17:06:07.724 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:06:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7678.296728 secs 2023-03-09T17:06:07.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:06:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7678.296728 secs 2023-03-09T17:06:07.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:06:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7678.296728 secs 2023-03-09T17:06:08.724 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:06:08 smithi122 ceph-mon[93869]: pgmap v4608: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 100 KiB/s wr, 23 op/s 2023-03-09T17:06:08.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:06:08 smithi181 ceph-mon[94189]: pgmap v4608: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 100 KiB/s wr, 23 op/s 2023-03-09T17:06:08.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:06:08 smithi038 ceph-mon[92422]: pgmap v4608: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 100 KiB/s wr, 23 op/s 2023-03-09T17:06:08.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8140.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:06:10.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:06:10 smithi122 ceph-mon[93869]: osdmap e501: 12 total, 12 up, 12 in 2023-03-09T17:06:10.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:06:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7681.041078 secs 2023-03-09T17:06:10.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:06:10 smithi122 ceph-mon[93869]: pgmap v4610: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 111 KiB/s wr, 26 op/s 2023-03-09T17:06:10.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:06:10 smithi181 ceph-mon[94189]: osdmap e501: 12 total, 12 up, 12 in 2023-03-09T17:06:10.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:06:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7681.041078 secs 2023-03-09T17:06:10.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:06:10 smithi181 ceph-mon[94189]: pgmap v4610: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 111 KiB/s wr, 26 op/s 2023-03-09T17:06:10.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:06:10 smithi038 ceph-mon[92422]: osdmap e501: 12 total, 12 up, 12 in 2023-03-09T17:06:10.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:06:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7681.041078 secs 2023-03-09T17:06:10.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:06:10 smithi038 ceph-mon[92422]: pgmap v4610: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 111 KiB/s wr, 26 op/s 2023-03-09T17:06:11.974 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:06:11 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:06:12.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:06:11 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:06:12.004 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:06:11 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:06:12.974 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:06:12 smithi122 ceph-mon[93869]: pgmap v4611: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 55 KiB/s wr, 23 op/s 2023-03-09T17:06:12.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:06:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7683.296814 secs 2023-03-09T17:06:12.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:06:12 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:06:12.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:06:12 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:06:12.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:06:12 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:06:12.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:06:12 smithi181 ceph-mon[94189]: pgmap v4611: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 55 KiB/s wr, 23 op/s 2023-03-09T17:06:12.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:06:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7683.296814 secs 2023-03-09T17:06:12.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:06:12 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:06:12.978 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:06:12 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:06:12.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:06:12 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:06:12.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:06:12 smithi038 ceph-mon[92422]: pgmap v4611: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 55 KiB/s wr, 23 op/s 2023-03-09T17:06:12.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:06:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7683.296814 secs 2023-03-09T17:06:12.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:06:12 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:06:12.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:06:12 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:06:12.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:06:12 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:06:13.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8145.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:06:15.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:06:14 smithi122 ceph-mon[93869]: pgmap v4612: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 21 KiB/s wr, 8 op/s 2023-03-09T17:06:15.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:06:14 smithi181 ceph-mon[94189]: pgmap v4612: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 21 KiB/s wr, 8 op/s 2023-03-09T17:06:15.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:06:14 smithi038 ceph-mon[92422]: pgmap v4612: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 21 KiB/s wr, 8 op/s 2023-03-09T17:06:16.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:06:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7686.041138 secs 2023-03-09T17:06:16.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:06:15 smithi122 ceph-mon[93869]: pgmap v4613: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T17:06:16.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:06:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7686.041138 secs 2023-03-09T17:06:16.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:06:15 smithi181 ceph-mon[94189]: pgmap v4613: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T17:06:16.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:06:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7686.041138 secs 2023-03-09T17:06:16.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:06:15 smithi038 ceph-mon[92422]: pgmap v4613: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T17:06:17.724 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:06:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7688.296932 secs 2023-03-09T17:06:17.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:06:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7688.296932 secs 2023-03-09T17:06:17.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:06:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7688.296932 secs 2023-03-09T17:06:18.724 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:06:18 smithi122 ceph-mon[93869]: pgmap v4614: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T17:06:18.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:06:18 smithi181 ceph-mon[94189]: pgmap v4614: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T17:06:18.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:06:18 smithi038 ceph-mon[92422]: pgmap v4614: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T17:06:18.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8150.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:06:21.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:06:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7691.041198 secs 2023-03-09T17:06:21.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:06:20 smithi122 ceph-mon[93869]: pgmap v4615: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T17:06:21.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:06:20 smithi122 ceph-mon[93869]: osdmap e502: 12 total, 12 up, 12 in 2023-03-09T17:06:21.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:06:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7691.041198 secs 2023-03-09T17:06:21.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:06:20 smithi181 ceph-mon[94189]: pgmap v4615: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T17:06:21.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:06:20 smithi181 ceph-mon[94189]: osdmap e502: 12 total, 12 up, 12 in 2023-03-09T17:06:21.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:06:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7691.041198 secs 2023-03-09T17:06:21.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:06:20 smithi038 ceph-mon[92422]: pgmap v4615: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T17:06:21.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:06:20 smithi038 ceph-mon[92422]: osdmap e502: 12 total, 12 up, 12 in 2023-03-09T17:06:22.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:06:21 smithi122 ceph-mon[93869]: pgmap v4617: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.9 KiB/s wr, 7 op/s 2023-03-09T17:06:22.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:06:21 smithi181 ceph-mon[94189]: pgmap v4617: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.9 KiB/s wr, 7 op/s 2023-03-09T17:06:22.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:06:21 smithi038 ceph-mon[92422]: pgmap v4617: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.9 KiB/s wr, 7 op/s 2023-03-09T17:06:23.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:06:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7693.297050 secs 2023-03-09T17:06:23.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:06:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7693.297050 secs 2023-03-09T17:06:23.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:06:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7693.297050 secs 2023-03-09T17:06:23.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8155.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:06:24.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:06:23 smithi122 ceph-mon[93869]: pgmap v4618: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.0 KiB/s wr, 7 op/s 2023-03-09T17:06:24.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:06:23 smithi181 ceph-mon[94189]: pgmap v4618: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.0 KiB/s wr, 7 op/s 2023-03-09T17:06:24.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:06:23 smithi038 ceph-mon[92422]: pgmap v4618: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.0 KiB/s wr, 7 op/s 2023-03-09T17:06:25.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:06:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7696.041258 secs 2023-03-09T17:06:25.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:06:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7696.041258 secs 2023-03-09T17:06:25.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:06:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7696.041258 secs 2023-03-09T17:06:26.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:06:26 smithi038 ceph-mon[92422]: pgmap v4619: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T17:06:26.724 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:06:26 smithi122 ceph-mon[93869]: pgmap v4619: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T17:06:26.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:06:26 smithi181 ceph-mon[94189]: pgmap v4619: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T17:06:27.724 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:06:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7698.297177 secs 2023-03-09T17:06:27.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:06:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7698.297177 secs 2023-03-09T17:06:27.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:06:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7698.297177 secs 2023-03-09T17:06:28.724 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:06:28 smithi122 ceph-mon[93869]: pgmap v4620: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T17:06:28.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:06:28 smithi181 ceph-mon[94189]: pgmap v4620: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T17:06:28.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:06:28 smithi038 ceph-mon[92422]: pgmap v4620: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T17:06:28.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8160.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:06:30.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:06:29 smithi122 ceph-mon[93869]: pgmap v4621: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T17:06:30.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:06:29 smithi181 ceph-mon[94189]: pgmap v4621: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T17:06:30.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:06:29 smithi038 ceph-mon[92422]: pgmap v4621: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T17:06:31.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:06:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7701.041319 secs 2023-03-09T17:06:31.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:06:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7701.041319 secs 2023-03-09T17:06:31.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:06:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7701.041319 secs 2023-03-09T17:06:32.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:06:31 smithi122 ceph-mon[93869]: pgmap v4622: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 KiB/s wr, 8 op/s 2023-03-09T17:06:32.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:06:31 smithi181 ceph-mon[94189]: pgmap v4622: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 KiB/s wr, 8 op/s 2023-03-09T17:06:32.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:06:31 smithi038 ceph-mon[92422]: pgmap v4622: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 KiB/s wr, 8 op/s 2023-03-09T17:06:33.793 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8165.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:06:34.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:06:33 smithi122 ceph-mon[93869]: pgmap v4623: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 7 op/s 2023-03-09T17:06:34.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:06:33 smithi181 ceph-mon[94189]: pgmap v4623: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 7 op/s 2023-03-09T17:06:34.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:06:33 smithi038 ceph-mon[92422]: pgmap v4623: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 7 op/s 2023-03-09T17:06:35.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:06:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7706.041381 secs 2023-03-09T17:06:35.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:06:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7706.041381 secs 2023-03-09T17:06:35.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:06:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7706.041381 secs 2023-03-09T17:06:36.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:06:36 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:06:36.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:06:36 smithi122 ceph-mon[93869]: pgmap v4624: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 63 KiB/s wr, 10 op/s 2023-03-09T17:06:36.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:06:36 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:06:36.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:06:36 smithi181 ceph-mon[94189]: pgmap v4624: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 63 KiB/s wr, 10 op/s 2023-03-09T17:06:36.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:06:36 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:06:36.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:06:36 smithi038 ceph-mon[92422]: pgmap v4624: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 63 KiB/s wr, 10 op/s 2023-03-09T17:06:37.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:06:37 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:06:37.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:06:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:06:37.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:06:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:06:37.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:06:37 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:06:37.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:06:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:06:37.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:06:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:06:37.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:06:37 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:06:37.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:06:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:06:37.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:06:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:06:38.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:06:38 smithi122 ceph-mon[93869]: pgmap v4625: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 55 KiB/s wr, 8 op/s 2023-03-09T17:06:38.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:06:38 smithi181 ceph-mon[94189]: pgmap v4625: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 55 KiB/s wr, 8 op/s 2023-03-09T17:06:38.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:06:38 smithi038 ceph-mon[92422]: pgmap v4625: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 55 KiB/s wr, 8 op/s 2023-03-09T17:06:38.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8170.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:06:40.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:06:39 smithi122 ceph-mon[93869]: pgmap v4626: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 55 KiB/s wr, 8 op/s 2023-03-09T17:06:40.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:06:39 smithi181 ceph-mon[94189]: pgmap v4626: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 55 KiB/s wr, 8 op/s 2023-03-09T17:06:40.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:06:39 smithi038 ceph-mon[92422]: pgmap v4626: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 55 KiB/s wr, 8 op/s 2023-03-09T17:06:41.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:06:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7711.041442 secs 2023-03-09T17:06:41.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:06:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7711.041442 secs 2023-03-09T17:06:41.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:06:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7711.041442 secs 2023-03-09T17:06:42.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:06:41 smithi122 ceph-mon[93869]: pgmap v4627: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 68 KiB/s wr, 12 op/s 2023-03-09T17:06:42.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:06:41 smithi181 ceph-mon[94189]: pgmap v4627: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 68 KiB/s wr, 12 op/s 2023-03-09T17:06:42.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:06:41 smithi038 ceph-mon[92422]: pgmap v4627: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 68 KiB/s wr, 12 op/s 2023-03-09T17:06:43.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8175.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:06:44.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:06:43 smithi038 ceph-mon[92422]: pgmap v4628: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 67 KiB/s wr, 10 op/s 2023-03-09T17:06:44.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:06:43 smithi122 ceph-mon[93869]: pgmap v4628: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 67 KiB/s wr, 10 op/s 2023-03-09T17:06:44.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:06:43 smithi181 ceph-mon[94189]: pgmap v4628: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 67 KiB/s wr, 10 op/s 2023-03-09T17:06:45.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:06:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7716.041501 secs 2023-03-09T17:06:45.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:06:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7716.041501 secs 2023-03-09T17:06:45.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:06:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7716.041501 secs 2023-03-09T17:06:46.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:06:46 smithi038 ceph-mon[92422]: pgmap v4629: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 74 KiB/s wr, 58 op/s 2023-03-09T17:06:46.724 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:06:46 smithi122 ceph-mon[93869]: pgmap v4629: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 74 KiB/s wr, 58 op/s 2023-03-09T17:06:46.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:06:46 smithi181 ceph-mon[94189]: pgmap v4629: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 74 KiB/s wr, 58 op/s 2023-03-09T17:06:48.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:06:47 smithi122 ceph-mon[93869]: pgmap v4630: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 85 KiB/s wr, 73 op/s 2023-03-09T17:06:48.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:06:47 smithi181 ceph-mon[94189]: pgmap v4630: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 85 KiB/s wr, 73 op/s 2023-03-09T17:06:48.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:06:47 smithi038 ceph-mon[92422]: pgmap v4630: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 85 KiB/s wr, 73 op/s 2023-03-09T17:06:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8180.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:06:50.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:06:49 smithi122 ceph-mon[93869]: pgmap v4631: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 85 KiB/s wr, 77 op/s 2023-03-09T17:06:50.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:06:49 smithi181 ceph-mon[94189]: pgmap v4631: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 85 KiB/s wr, 77 op/s 2023-03-09T17:06:50.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:06:49 smithi038 ceph-mon[92422]: pgmap v4631: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 85 KiB/s wr, 77 op/s 2023-03-09T17:06:51.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:06:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7721.041563 secs 2023-03-09T17:06:51.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:06:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7721.041563 secs 2023-03-09T17:06:51.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:06:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7721.041563 secs 2023-03-09T17:06:52.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:06:51 smithi122 ceph-mon[93869]: pgmap v4632: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 112 KiB/s wr, 81 op/s 2023-03-09T17:06:52.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:06:51 smithi181 ceph-mon[94189]: pgmap v4632: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 112 KiB/s wr, 81 op/s 2023-03-09T17:06:52.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:06:51 smithi038 ceph-mon[92422]: pgmap v4632: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 112 KiB/s wr, 81 op/s 2023-03-09T17:06:53.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:06:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7723.297717 secs 2023-03-09T17:06:53.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:06:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7723.297717 secs 2023-03-09T17:06:53.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:06:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7723.297717 secs 2023-03-09T17:06:53.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8185.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:06:54.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:06:53 smithi181 ceph-mon[94189]: pgmap v4633: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 99 KiB/s wr, 77 op/s 2023-03-09T17:06:54.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:06:53 smithi038 ceph-mon[92422]: pgmap v4633: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 99 KiB/s wr, 77 op/s 2023-03-09T17:06:54.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:06:53 smithi122 ceph-mon[93869]: pgmap v4633: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 99 KiB/s wr, 77 op/s 2023-03-09T17:06:55.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:06:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7726.041624 secs 2023-03-09T17:06:55.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:06:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7726.041624 secs 2023-03-09T17:06:55.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:06:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7726.041624 secs 2023-03-09T17:06:56.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:06:56 smithi038 ceph-mon[92422]: pgmap v4634: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 99 KiB/s wr, 79 op/s 2023-03-09T17:06:56.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:06:56 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:06:56.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:06:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:06:56.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:06:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:06:56.724 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:06:56 smithi122 ceph-mon[93869]: pgmap v4634: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 99 KiB/s wr, 79 op/s 2023-03-09T17:06:56.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:06:56 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:06:56.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:06:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:06:56.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:06:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:06:56.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:06:56 smithi181 ceph-mon[94189]: pgmap v4634: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 99 KiB/s wr, 79 op/s 2023-03-09T17:06:56.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:06:56 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:06:56.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:06:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:06:56.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:06:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:06:57.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:06:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7728.297837 secs 2023-03-09T17:06:57.724 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:06:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7728.297837 secs 2023-03-09T17:06:57.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:06:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7728.297837 secs 2023-03-09T17:06:58.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:06:58 smithi038 ceph-mon[92422]: pgmap v4635: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 102 KiB/s wr, 31 op/s 2023-03-09T17:06:58.724 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:06:58 smithi122 ceph-mon[93869]: pgmap v4635: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 102 KiB/s wr, 31 op/s 2023-03-09T17:06:58.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:06:58 smithi181 ceph-mon[94189]: pgmap v4635: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 102 KiB/s wr, 31 op/s 2023-03-09T17:06:58.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8190.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:07:00.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:06:59 smithi122 ceph-mon[93869]: pgmap v4636: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 44 KiB/s wr, 13 op/s 2023-03-09T17:07:00.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:06:59 smithi181 ceph-mon[94189]: pgmap v4636: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 44 KiB/s wr, 13 op/s 2023-03-09T17:07:00.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:06:59 smithi038 ceph-mon[92422]: pgmap v4636: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 44 KiB/s wr, 13 op/s 2023-03-09T17:07:01.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:07:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7731.041686 secs 2023-03-09T17:07:01.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:07:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7731.041686 secs 2023-03-09T17:07:01.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:07:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7731.041686 secs 2023-03-09T17:07:02.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:07:01 smithi122 ceph-mon[93869]: pgmap v4637: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 110 KiB/s wr, 14 op/s 2023-03-09T17:07:02.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:07:01 smithi181 ceph-mon[94189]: pgmap v4637: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 110 KiB/s wr, 14 op/s 2023-03-09T17:07:02.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:07:01 smithi038 ceph-mon[92422]: pgmap v4637: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 110 KiB/s wr, 14 op/s 2023-03-09T17:07:03.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:07:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7733.297959 secs 2023-03-09T17:07:03.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:07:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7733.297959 secs 2023-03-09T17:07:03.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:07:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7733.297959 secs 2023-03-09T17:07:03.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8195.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:07:04.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:07:03 smithi181 ceph-mon[94189]: pgmap v4638: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 103 KiB/s wr, 22 op/s 2023-03-09T17:07:04.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:07:03 smithi038 ceph-mon[92422]: pgmap v4638: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 103 KiB/s wr, 22 op/s 2023-03-09T17:07:04.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:07:03 smithi122 ceph-mon[93869]: pgmap v4638: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 103 KiB/s wr, 22 op/s 2023-03-09T17:07:05.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:07:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7736.041748 secs 2023-03-09T17:07:05.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:07:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7736.041748 secs 2023-03-09T17:07:05.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:07:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7736.041748 secs 2023-03-09T17:07:06.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:07:06 smithi038 ceph-mon[92422]: pgmap v4639: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 103 KiB/s wr, 23 op/s 2023-03-09T17:07:06.724 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:07:06 smithi122 ceph-mon[93869]: pgmap v4639: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 103 KiB/s wr, 23 op/s 2023-03-09T17:07:06.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:07:06 smithi181 ceph-mon[94189]: pgmap v4639: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 103 KiB/s wr, 23 op/s 2023-03-09T17:07:07.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:07:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7738.298097 secs 2023-03-09T17:07:07.724 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:07:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7738.298097 secs 2023-03-09T17:07:07.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:07:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7738.298097 secs 2023-03-09T17:07:08.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:07:08 smithi038 ceph-mon[92422]: pgmap v4640: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 110 KiB/s wr, 22 op/s 2023-03-09T17:07:08.724 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:07:08 smithi122 ceph-mon[93869]: pgmap v4640: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 110 KiB/s wr, 22 op/s 2023-03-09T17:07:08.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:07:08 smithi181 ceph-mon[94189]: pgmap v4640: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 110 KiB/s wr, 22 op/s 2023-03-09T17:07:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8200.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:07:10.724 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:07:10 smithi122 ceph-mon[93869]: osdmap e503: 12 total, 12 up, 12 in 2023-03-09T17:07:10.724 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:07:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7741.041806 secs 2023-03-09T17:07:10.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:07:10 smithi122 ceph-mon[93869]: pgmap v4642: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 121 KiB/s wr, 25 op/s 2023-03-09T17:07:10.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:07:10 smithi181 ceph-mon[94189]: osdmap e503: 12 total, 12 up, 12 in 2023-03-09T17:07:10.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:07:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7741.041806 secs 2023-03-09T17:07:10.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:07:10 smithi181 ceph-mon[94189]: pgmap v4642: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 121 KiB/s wr, 25 op/s 2023-03-09T17:07:10.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:07:10 smithi038 ceph-mon[92422]: osdmap e503: 12 total, 12 up, 12 in 2023-03-09T17:07:10.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:07:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7741.041806 secs 2023-03-09T17:07:10.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:07:10 smithi038 ceph-mon[92422]: pgmap v4642: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 121 KiB/s wr, 25 op/s 2023-03-09T17:07:12.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:07:11 smithi122 ceph-mon[93869]: pgmap v4643: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 43 KiB/s wr, 22 op/s 2023-03-09T17:07:12.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:07:11 smithi181 ceph-mon[94189]: pgmap v4643: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 43 KiB/s wr, 22 op/s 2023-03-09T17:07:12.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:07:11 smithi038 ceph-mon[92422]: pgmap v4643: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 43 KiB/s wr, 22 op/s 2023-03-09T17:07:13.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:07:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7743.298235 secs 2023-03-09T17:07:13.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:07:12 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:07:13.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:07:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7743.298235 secs 2023-03-09T17:07:13.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:07:12 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:07:13.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:07:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7743.298235 secs 2023-03-09T17:07:13.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:07:12 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:07:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8205.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:07:14.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:07:13 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:07:14.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:07:13 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:07:14.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:07:13 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:07:14.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:07:13 smithi122 ceph-mon[93869]: pgmap v4644: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 8 op/s 2023-03-09T17:07:14.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:07:13 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:07:14.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:07:13 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:07:14.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:07:13 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:07:14.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:07:13 smithi181 ceph-mon[94189]: pgmap v4644: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 8 op/s 2023-03-09T17:07:14.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:07:13 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:07:14.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:07:13 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:07:14.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:07:13 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:07:14.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:07:13 smithi038 ceph-mon[92422]: pgmap v4644: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 8 op/s 2023-03-09T17:07:15.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:07:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7746.041866 secs 2023-03-09T17:07:15.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:07:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7746.041866 secs 2023-03-09T17:07:15.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:07:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7746.041866 secs 2023-03-09T17:07:16.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:07:16 smithi038 ceph-mon[92422]: pgmap v4645: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 18 KiB/s wr, 8 op/s 2023-03-09T17:07:16.724 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:07:16 smithi122 ceph-mon[93869]: pgmap v4645: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 18 KiB/s wr, 8 op/s 2023-03-09T17:07:16.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:07:16 smithi181 ceph-mon[94189]: pgmap v4645: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 18 KiB/s wr, 8 op/s 2023-03-09T17:07:17.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:07:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7748.298361 secs 2023-03-09T17:07:17.724 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:07:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7748.298361 secs 2023-03-09T17:07:17.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:07:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7748.298361 secs 2023-03-09T17:07:18.724 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:07:18 smithi122 ceph-mon[93869]: pgmap v4646: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 29 KiB/s wr, 8 op/s 2023-03-09T17:07:18.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:07:18 smithi181 ceph-mon[94189]: pgmap v4646: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 29 KiB/s wr, 8 op/s 2023-03-09T17:07:18.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:07:18 smithi038 ceph-mon[92422]: pgmap v4646: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 29 KiB/s wr, 8 op/s 2023-03-09T17:07:18.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8210.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:07:20.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:07:19 smithi122 ceph-mon[93869]: pgmap v4647: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T17:07:20.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:07:19 smithi181 ceph-mon[94189]: pgmap v4647: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T17:07:20.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:07:19 smithi038 ceph-mon[92422]: pgmap v4647: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T17:07:21.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:07:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7751.041927 secs 2023-03-09T17:07:21.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:07:20 smithi122 ceph-mon[93869]: osdmap e504: 12 total, 12 up, 12 in 2023-03-09T17:07:21.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:07:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7751.041927 secs 2023-03-09T17:07:21.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:07:20 smithi181 ceph-mon[94189]: osdmap e504: 12 total, 12 up, 12 in 2023-03-09T17:07:21.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:07:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7751.041927 secs 2023-03-09T17:07:21.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:07:20 smithi038 ceph-mon[92422]: osdmap e504: 12 total, 12 up, 12 in 2023-03-09T17:07:22.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:07:21 smithi122 ceph-mon[93869]: pgmap v4649: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 21 KiB/s wr, 7 op/s 2023-03-09T17:07:22.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:07:21 smithi181 ceph-mon[94189]: pgmap v4649: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 21 KiB/s wr, 7 op/s 2023-03-09T17:07:22.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:07:21 smithi038 ceph-mon[92422]: pgmap v4649: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 21 KiB/s wr, 7 op/s 2023-03-09T17:07:23.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:07:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7753.298484 secs 2023-03-09T17:07:23.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:07:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7753.298484 secs 2023-03-09T17:07:23.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:07:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7753.298484 secs 2023-03-09T17:07:23.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8215.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:07:24.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:07:23 smithi038 ceph-mon[92422]: pgmap v4650: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 21 KiB/s wr, 7 op/s 2023-03-09T17:07:24.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:07:23 smithi122 ceph-mon[93869]: pgmap v4650: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 21 KiB/s wr, 7 op/s 2023-03-09T17:07:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:07:23 smithi181 ceph-mon[94189]: pgmap v4650: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 21 KiB/s wr, 7 op/s 2023-03-09T17:07:25.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:07:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7756.041988 secs 2023-03-09T17:07:25.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:07:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7756.041988 secs 2023-03-09T17:07:25.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:07:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7756.041988 secs 2023-03-09T17:07:26.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:07:26 smithi038 ceph-mon[92422]: pgmap v4651: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T17:07:26.724 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:07:26 smithi122 ceph-mon[93869]: pgmap v4651: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T17:07:26.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:07:26 smithi181 ceph-mon[94189]: pgmap v4651: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T17:07:27.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:07:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7758.298624 secs 2023-03-09T17:07:27.724 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:07:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7758.298624 secs 2023-03-09T17:07:27.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:07:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7758.298624 secs 2023-03-09T17:07:28.724 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:07:28 smithi122 ceph-mon[93869]: pgmap v4652: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 34 KiB/s wr, 7 op/s 2023-03-09T17:07:28.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:07:28 smithi181 ceph-mon[94189]: pgmap v4652: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 34 KiB/s wr, 7 op/s 2023-03-09T17:07:28.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:07:28 smithi038 ceph-mon[92422]: pgmap v4652: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 34 KiB/s wr, 7 op/s 2023-03-09T17:07:28.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8220.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:07:30.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:07:29 smithi122 ceph-mon[93869]: pgmap v4653: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 34 KiB/s wr, 7 op/s 2023-03-09T17:07:30.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:07:29 smithi181 ceph-mon[94189]: pgmap v4653: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 34 KiB/s wr, 7 op/s 2023-03-09T17:07:30.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:07:29 smithi038 ceph-mon[92422]: pgmap v4653: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 34 KiB/s wr, 7 op/s 2023-03-09T17:07:31.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:07:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7761.042046 secs 2023-03-09T17:07:31.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:07:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7761.042046 secs 2023-03-09T17:07:31.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:07:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7761.042046 secs 2023-03-09T17:07:32.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:07:31 smithi122 ceph-mon[93869]: pgmap v4654: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 29 KiB/s wr, 9 op/s 2023-03-09T17:07:32.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:07:31 smithi181 ceph-mon[94189]: pgmap v4654: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 29 KiB/s wr, 9 op/s 2023-03-09T17:07:32.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:07:31 smithi038 ceph-mon[92422]: pgmap v4654: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 29 KiB/s wr, 9 op/s 2023-03-09T17:07:33.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:07:32 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 7763.298764 secs 2023-03-09T17:07:33.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:07:32 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 7763.298764 secs 2023-03-09T17:07:33.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:07:32 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 7763.298764 secs 2023-03-09T17:07:33.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8225.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:07:34.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:07:33 smithi038 ceph-mon[92422]: pgmap v4655: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 28 KiB/s wr, 6 op/s 2023-03-09T17:07:34.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:07:33 smithi122 ceph-mon[93869]: pgmap v4655: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 28 KiB/s wr, 6 op/s 2023-03-09T17:07:34.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:07:33 smithi181 ceph-mon[94189]: pgmap v4655: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 28 KiB/s wr, 6 op/s 2023-03-09T17:07:35.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:07:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7766.042103 secs 2023-03-09T17:07:35.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:07:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7766.042103 secs 2023-03-09T17:07:35.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:07:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7766.042103 secs 2023-03-09T17:07:36.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:07:36 smithi038 ceph-mon[92422]: pgmap v4656: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 28 KiB/s wr, 8 op/s 2023-03-09T17:07:36.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:07:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:07:36.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:07:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:07:36.724 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:07:36 smithi122 ceph-mon[93869]: pgmap v4656: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 28 KiB/s wr, 8 op/s 2023-03-09T17:07:36.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:07:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:07:36.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:07:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:07:36.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:07:36 smithi181 ceph-mon[94189]: pgmap v4656: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 28 KiB/s wr, 8 op/s 2023-03-09T17:07:36.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:07:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:07:36.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:07:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:07:37.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:07:37 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 7768.298915 secs 2023-03-09T17:07:37.724 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:07:37 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 7768.298915 secs 2023-03-09T17:07:37.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:07:37 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 7768.298915 secs 2023-03-09T17:07:38.724 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:07:38 smithi122 ceph-mon[93869]: pgmap v4657: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 41 KiB/s wr, 7 op/s 2023-03-09T17:07:38.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:07:38 smithi181 ceph-mon[94189]: pgmap v4657: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 41 KiB/s wr, 7 op/s 2023-03-09T17:07:38.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:07:38 smithi038 ceph-mon[92422]: pgmap v4657: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 41 KiB/s wr, 7 op/s 2023-03-09T17:07:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8230.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:07:40.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:07:39 smithi122 ceph-mon[93869]: pgmap v4658: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 6 op/s 2023-03-09T17:07:40.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:07:39 smithi181 ceph-mon[94189]: pgmap v4658: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 6 op/s 2023-03-09T17:07:40.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:07:39 smithi038 ceph-mon[92422]: pgmap v4658: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 6 op/s 2023-03-09T17:07:41.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:07:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7771.042161 secs 2023-03-09T17:07:41.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:07:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7771.042161 secs 2023-03-09T17:07:41.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:07:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7771.042161 secs 2023-03-09T17:07:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:07:41 smithi038 ceph-mon[92422]: pgmap v4659: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 53 KiB/s wr, 20 op/s 2023-03-09T17:07:42.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:07:41 smithi122 ceph-mon[93869]: pgmap v4659: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 53 KiB/s wr, 20 op/s 2023-03-09T17:07:42.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:07:41 smithi181 ceph-mon[94189]: pgmap v4659: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 53 KiB/s wr, 20 op/s 2023-03-09T17:07:43.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:07:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7773.299057 secs 2023-03-09T17:07:43.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:07:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7773.299057 secs 2023-03-09T17:07:43.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:07:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7773.299057 secs 2023-03-09T17:07:43.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8235.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:07:44.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:07:43 smithi038 ceph-mon[92422]: pgmap v4660: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 53 KiB/s wr, 18 op/s 2023-03-09T17:07:44.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:07:43 smithi122 ceph-mon[93869]: pgmap v4660: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 53 KiB/s wr, 18 op/s 2023-03-09T17:07:44.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:07:43 smithi181 ceph-mon[94189]: pgmap v4660: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 53 KiB/s wr, 18 op/s 2023-03-09T17:07:45.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:07:44 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:07:45.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:07:44 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:07:45.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:07:44 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:07:45.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:07:44 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:07:45.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:07:44 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:07:45.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:07:44 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:07:46.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:07:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7776.042220 secs 2023-03-09T17:07:46.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:07:45 smithi038 ceph-mon[92422]: pgmap v4661: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 53 KiB/s wr, 46 op/s 2023-03-09T17:07:46.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:07:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7776.042220 secs 2023-03-09T17:07:46.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:07:45 smithi122 ceph-mon[93869]: pgmap v4661: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 53 KiB/s wr, 46 op/s 2023-03-09T17:07:46.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:07:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7776.042220 secs 2023-03-09T17:07:46.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:07:45 smithi181 ceph-mon[94189]: pgmap v4661: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 53 KiB/s wr, 46 op/s 2023-03-09T17:07:47.724 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:07:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7778.299124 secs 2023-03-09T17:07:47.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:07:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7778.299124 secs 2023-03-09T17:07:47.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:07:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7778.299124 secs 2023-03-09T17:07:48.724 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:07:48 smithi122 ceph-mon[93869]: pgmap v4662: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 56 KiB/s wr, 46 op/s 2023-03-09T17:07:48.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:07:48 smithi181 ceph-mon[94189]: pgmap v4662: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 56 KiB/s wr, 46 op/s 2023-03-09T17:07:48.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:07:48 smithi038 ceph-mon[92422]: pgmap v4662: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 56 KiB/s wr, 46 op/s 2023-03-09T17:07:48.793 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8240.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:07:50.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:07:49 smithi122 ceph-mon[93869]: pgmap v4663: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 42 KiB/s wr, 45 op/s 2023-03-09T17:07:50.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:07:49 smithi181 ceph-mon[94189]: pgmap v4663: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 42 KiB/s wr, 45 op/s 2023-03-09T17:07:50.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:07:49 smithi038 ceph-mon[92422]: pgmap v4663: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 42 KiB/s wr, 45 op/s 2023-03-09T17:07:51.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:07:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7781.042280 secs 2023-03-09T17:07:51.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:07:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7781.042280 secs 2023-03-09T17:07:51.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:07:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7781.042280 secs 2023-03-09T17:07:52.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:07:51 smithi122 ceph-mon[93869]: pgmap v4664: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 103 KiB/s wr, 49 op/s 2023-03-09T17:07:52.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:07:51 smithi181 ceph-mon[94189]: pgmap v4664: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 103 KiB/s wr, 49 op/s 2023-03-09T17:07:52.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:07:51 smithi038 ceph-mon[92422]: pgmap v4664: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 103 KiB/s wr, 49 op/s 2023-03-09T17:07:53.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:07:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7783.299251 secs 2023-03-09T17:07:53.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:07:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7783.299251 secs 2023-03-09T17:07:53.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:07:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7783.299251 secs 2023-03-09T17:07:53.793 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8245.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:07:54.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:07:53 smithi038 ceph-mon[92422]: pgmap v4665: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 64 KiB/s wr, 34 op/s 2023-03-09T17:07:54.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:07:53 smithi122 ceph-mon[93869]: pgmap v4665: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 64 KiB/s wr, 34 op/s 2023-03-09T17:07:54.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:07:53 smithi181 ceph-mon[94189]: pgmap v4665: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 64 KiB/s wr, 34 op/s 2023-03-09T17:07:55.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:07:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7786.042339 secs 2023-03-09T17:07:55.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:07:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7786.042339 secs 2023-03-09T17:07:55.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:07:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7786.042339 secs 2023-03-09T17:07:56.724 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:07:56 smithi122 ceph-mon[93869]: pgmap v4666: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 64 KiB/s wr, 36 op/s 2023-03-09T17:07:56.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:07:56 smithi181 ceph-mon[94189]: pgmap v4666: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 64 KiB/s wr, 36 op/s 2023-03-09T17:07:56.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:07:56 smithi038 ceph-mon[92422]: pgmap v4666: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 64 KiB/s wr, 36 op/s 2023-03-09T17:07:57.724 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:07:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7788.299346 secs 2023-03-09T17:07:57.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:07:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7788.299346 secs 2023-03-09T17:07:57.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:07:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7788.299346 secs 2023-03-09T17:07:58.724 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:07:58 smithi122 ceph-mon[93869]: pgmap v4667: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 75 KiB/s wr, 10 op/s 2023-03-09T17:07:58.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:07:58 smithi181 ceph-mon[94189]: pgmap v4667: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 75 KiB/s wr, 10 op/s 2023-03-09T17:07:58.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:07:58 smithi038 ceph-mon[92422]: pgmap v4667: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 75 KiB/s wr, 10 op/s 2023-03-09T17:07:58.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8250.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:08:00.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:07:59 smithi122 ceph-mon[93869]: pgmap v4668: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 72 KiB/s wr, 8 op/s 2023-03-09T17:08:00.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:07:59 smithi181 ceph-mon[94189]: pgmap v4668: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 72 KiB/s wr, 8 op/s 2023-03-09T17:08:00.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:07:59 smithi038 ceph-mon[92422]: pgmap v4668: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 72 KiB/s wr, 8 op/s 2023-03-09T17:08:01.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:08:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7791.042398 secs 2023-03-09T17:08:01.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:08:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7791.042398 secs 2023-03-09T17:08:01.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:08:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7791.042398 secs 2023-03-09T17:08:02.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:08:01 smithi122 ceph-mon[93869]: pgmap v4669: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 128 KiB/s wr, 14 op/s 2023-03-09T17:08:02.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:08:01 smithi181 ceph-mon[94189]: pgmap v4669: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 128 KiB/s wr, 14 op/s 2023-03-09T17:08:02.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:08:01 smithi038 ceph-mon[92422]: pgmap v4669: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 128 KiB/s wr, 14 op/s 2023-03-09T17:08:03.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:08:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7793.299438 secs 2023-03-09T17:08:03.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:08:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7793.299438 secs 2023-03-09T17:08:03.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:08:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7793.299438 secs 2023-03-09T17:08:03.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8255.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:08:04.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:08:03 smithi038 ceph-mon[92422]: pgmap v4670: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 89 KiB/s wr, 22 op/s 2023-03-09T17:08:04.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:08:03 smithi122 ceph-mon[93869]: pgmap v4670: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 89 KiB/s wr, 22 op/s 2023-03-09T17:08:04.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:08:03 smithi181 ceph-mon[94189]: pgmap v4670: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 89 KiB/s wr, 22 op/s 2023-03-09T17:08:05.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:08:04 smithi038 ceph-mon[92422]: osdmap e505: 12 total, 12 up, 12 in 2023-03-09T17:08:05.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:08:04 smithi122 ceph-mon[93869]: osdmap e505: 12 total, 12 up, 12 in 2023-03-09T17:08:05.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:08:04 smithi181 ceph-mon[94189]: osdmap e505: 12 total, 12 up, 12 in 2023-03-09T17:08:06.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:08:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7796.042456 secs 2023-03-09T17:08:06.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:08:05 smithi038 ceph-mon[92422]: pgmap v4672: 129 pgs: 1 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 116 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 106 KiB/s wr, 27 op/s 2023-03-09T17:08:06.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:08:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7796.042456 secs 2023-03-09T17:08:06.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:08:05 smithi122 ceph-mon[93869]: pgmap v4672: 129 pgs: 1 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 116 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 106 KiB/s wr, 27 op/s 2023-03-09T17:08:06.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:08:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7796.042456 secs 2023-03-09T17:08:06.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:08:05 smithi181 ceph-mon[94189]: pgmap v4672: 129 pgs: 1 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 116 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 106 KiB/s wr, 27 op/s 2023-03-09T17:08:07.724 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:08:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7798.299507 secs 2023-03-09T17:08:07.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:08:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7798.299507 secs 2023-03-09T17:08:07.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:08:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7798.299507 secs 2023-03-09T17:08:08.724 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:08:08 smithi122 ceph-mon[93869]: pgmap v4673: 129 pgs: 1 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 116 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 141 KiB/s wr, 27 op/s 2023-03-09T17:08:08.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:08:08 smithi181 ceph-mon[94189]: pgmap v4673: 129 pgs: 1 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 116 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 141 KiB/s wr, 27 op/s 2023-03-09T17:08:08.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:08:08 smithi038 ceph-mon[92422]: pgmap v4673: 129 pgs: 1 active+clean+snaptrim, 12 active+clean+snaptrim_wait, 116 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 141 KiB/s wr, 27 op/s 2023-03-09T17:08:08.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8260.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:08:10.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:08:09 smithi122 ceph-mon[93869]: pgmap v4674: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 141 KiB/s wr, 27 op/s 2023-03-09T17:08:10.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:08:09 smithi181 ceph-mon[94189]: pgmap v4674: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 141 KiB/s wr, 27 op/s 2023-03-09T17:08:10.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:08:09 smithi038 ceph-mon[92422]: pgmap v4674: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 141 KiB/s wr, 27 op/s 2023-03-09T17:08:11.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:08:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7801.042519 secs 2023-03-09T17:08:11.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:08:10 smithi122 ceph-mon[93869]: osdmap e506: 12 total, 12 up, 12 in 2023-03-09T17:08:11.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:08:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7801.042519 secs 2023-03-09T17:08:11.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:08:10 smithi181 ceph-mon[94189]: osdmap e506: 12 total, 12 up, 12 in 2023-03-09T17:08:11.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:08:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7801.042519 secs 2023-03-09T17:08:11.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:08:10 smithi038 ceph-mon[92422]: osdmap e506: 12 total, 12 up, 12 in 2023-03-09T17:08:12.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:08:11 smithi181 ceph-mon[94189]: pgmap v4676: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 77 KiB/s wr, 11 op/s 2023-03-09T17:08:12.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:08:11 smithi038 ceph-mon[92422]: pgmap v4676: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 77 KiB/s wr, 11 op/s 2023-03-09T17:08:12.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:08:11 smithi122 ceph-mon[93869]: pgmap v4676: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 77 KiB/s wr, 11 op/s 2023-03-09T17:08:13.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:08:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7803.299602 secs 2023-03-09T17:08:13.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:08:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7803.299602 secs 2023-03-09T17:08:13.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:08:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7803.299602 secs 2023-03-09T17:08:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8265.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:08:14.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:08:13 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:08:14.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:08:13 smithi038 ceph-mon[92422]: pgmap v4677: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 62 KiB/s wr, 9 op/s 2023-03-09T17:08:14.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:08:13 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:08:14.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:08:13 smithi122 ceph-mon[93869]: pgmap v4677: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 62 KiB/s wr, 9 op/s 2023-03-09T17:08:14.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:08:13 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:08:14.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:08:13 smithi181 ceph-mon[94189]: pgmap v4677: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 62 KiB/s wr, 9 op/s 2023-03-09T17:08:15.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:08:14 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:08:15.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:08:14 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:08:15.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:08:14 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:08:15.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:08:14 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:08:15.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:08:14 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:08:15.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:08:14 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:08:15.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:08:14 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:08:15.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:08:14 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:08:15.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:08:14 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:08:16.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:08:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7806.042582 secs 2023-03-09T17:08:16.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:08:15 smithi038 ceph-mon[92422]: pgmap v4678: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 62 KiB/s wr, 8 op/s 2023-03-09T17:08:16.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:08:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7806.042582 secs 2023-03-09T17:08:16.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:08:15 smithi122 ceph-mon[93869]: pgmap v4678: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 62 KiB/s wr, 8 op/s 2023-03-09T17:08:16.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:08:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7806.042582 secs 2023-03-09T17:08:16.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:08:15 smithi181 ceph-mon[94189]: pgmap v4678: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 62 KiB/s wr, 8 op/s 2023-03-09T17:08:17.724 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:08:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7808.299704 secs 2023-03-09T17:08:17.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:08:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7808.299704 secs 2023-03-09T17:08:17.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:08:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7808.299704 secs 2023-03-09T17:08:18.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:08:18 smithi122 ceph-mon[93869]: pgmap v4679: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T17:08:18.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:08:18 smithi181 ceph-mon[94189]: pgmap v4679: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T17:08:18.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:08:18 smithi038 ceph-mon[92422]: pgmap v4679: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T17:08:18.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8270.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:08:20.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:08:19 smithi122 ceph-mon[93869]: pgmap v4680: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T17:08:20.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:08:19 smithi181 ceph-mon[94189]: pgmap v4680: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T17:08:20.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:08:19 smithi038 ceph-mon[92422]: pgmap v4680: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T17:08:21.224 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:08:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7811.042643 secs 2023-03-09T17:08:21.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:08:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7811.042643 secs 2023-03-09T17:08:21.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:08:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7811.042643 secs 2023-03-09T17:08:22.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:08:21 smithi038 ceph-mon[92422]: pgmap v4681: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 37 KiB/s wr, 9 op/s 2023-03-09T17:08:22.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:08:21 smithi122 ceph-mon[93869]: pgmap v4681: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 37 KiB/s wr, 9 op/s 2023-03-09T17:08:22.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:08:21 smithi181 ceph-mon[94189]: pgmap v4681: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 37 KiB/s wr, 9 op/s 2023-03-09T17:08:23.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:08:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7813.299820 secs 2023-03-09T17:08:23.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:08:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7813.299820 secs 2023-03-09T17:08:23.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:08:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7813.299820 secs 2023-03-09T17:08:23.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8275.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:08:24.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:08:23 smithi038 ceph-mon[92422]: pgmap v4682: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 6 op/s 2023-03-09T17:08:24.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:08:23 smithi122 ceph-mon[93869]: pgmap v4682: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 6 op/s 2023-03-09T17:08:24.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:08:23 smithi181 ceph-mon[94189]: pgmap v4682: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 6 op/s 2023-03-09T17:08:25.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:08:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7816.042726 secs 2023-03-09T17:08:25.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:08:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7816.042726 secs 2023-03-09T17:08:25.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:08:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7816.042726 secs 2023-03-09T17:08:26.724 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:08:26 smithi122 ceph-mon[93869]: pgmap v4683: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 26 KiB/s wr, 8 op/s 2023-03-09T17:08:26.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:08:26 smithi181 ceph-mon[94189]: pgmap v4683: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 26 KiB/s wr, 8 op/s 2023-03-09T17:08:26.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:08:26 smithi038 ceph-mon[92422]: pgmap v4683: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 26 KiB/s wr, 8 op/s 2023-03-09T17:08:27.724 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:08:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7818.299915 secs 2023-03-09T17:08:27.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:08:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7818.299915 secs 2023-03-09T17:08:27.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:08:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7818.299915 secs 2023-03-09T17:08:28.724 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:08:28 smithi122 ceph-mon[93869]: pgmap v4684: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 26 KiB/s wr, 8 op/s 2023-03-09T17:08:28.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:08:28 smithi181 ceph-mon[94189]: pgmap v4684: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 26 KiB/s wr, 8 op/s 2023-03-09T17:08:28.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:08:28 smithi038 ceph-mon[92422]: pgmap v4684: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 26 KiB/s wr, 8 op/s 2023-03-09T17:08:28.808 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8280.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:08:30.228 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:08:29 smithi181 ceph-mon[94189]: pgmap v4685: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 40 KiB/s wr, 27 op/s 2023-03-09T17:08:30.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:08:29 smithi038 ceph-mon[92422]: pgmap v4685: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 40 KiB/s wr, 27 op/s 2023-03-09T17:08:30.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:08:29 smithi122 ceph-mon[93869]: pgmap v4685: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 40 KiB/s wr, 27 op/s 2023-03-09T17:08:31.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:08:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7821.042785 secs 2023-03-09T17:08:31.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:08:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7821.042785 secs 2023-03-09T17:08:31.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:08:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7821.042785 secs 2023-03-09T17:08:32.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:08:31 smithi038 ceph-mon[92422]: pgmap v4686: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 95 KiB/s wr, 108 op/s 2023-03-09T17:08:32.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:08:31 smithi122 ceph-mon[93869]: pgmap v4686: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 95 KiB/s wr, 108 op/s 2023-03-09T17:08:32.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:08:31 smithi181 ceph-mon[94189]: pgmap v4686: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 95 KiB/s wr, 108 op/s 2023-03-09T17:08:33.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8285.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:08:34.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:08:33 smithi038 ceph-mon[92422]: pgmap v4687: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 83 KiB/s wr, 132 op/s 2023-03-09T17:08:34.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:08:33 smithi122 ceph-mon[93869]: pgmap v4687: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 83 KiB/s wr, 132 op/s 2023-03-09T17:08:34.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:08:33 smithi181 ceph-mon[94189]: pgmap v4687: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 83 KiB/s wr, 132 op/s 2023-03-09T17:08:35.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:08:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7826.042846 secs 2023-03-09T17:08:35.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:08:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7826.042846 secs 2023-03-09T17:08:35.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:08:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7826.042846 secs 2023-03-09T17:08:36.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:08:36 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:08:36.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:08:36 smithi122 ceph-mon[93869]: pgmap v4688: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 85 KiB/s wr, 135 op/s 2023-03-09T17:08:36.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:08:36 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:08:36.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:08:36 smithi181 ceph-mon[94189]: pgmap v4688: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 85 KiB/s wr, 135 op/s 2023-03-09T17:08:36.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:08:36 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:08:36.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:08:36 smithi038 ceph-mon[92422]: pgmap v4688: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 85 KiB/s wr, 135 op/s 2023-03-09T17:08:37.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:08:37 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:08:37.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:08:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:08:37.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:08:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:08:37.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:08:37 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:08:37.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:08:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:08:37.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:08:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:08:37.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:08:37 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:08:37.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:08:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:08:37.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:08:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:08:38.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:08:38 smithi122 ceph-mon[93869]: pgmap v4689: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 92 KiB/s wr, 133 op/s 2023-03-09T17:08:38.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:08:38 smithi181 ceph-mon[94189]: pgmap v4689: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 92 KiB/s wr, 133 op/s 2023-03-09T17:08:38.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:08:38 smithi038 ceph-mon[92422]: pgmap v4689: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 92 KiB/s wr, 133 op/s 2023-03-09T17:08:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8290.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:08:40.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:08:39 smithi038 ceph-mon[92422]: pgmap v4690: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 91 KiB/s wr, 132 op/s 2023-03-09T17:08:40.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:08:39 smithi122 ceph-mon[93869]: pgmap v4690: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 91 KiB/s wr, 132 op/s 2023-03-09T17:08:40.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:08:39 smithi181 ceph-mon[94189]: pgmap v4690: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 91 KiB/s wr, 132 op/s 2023-03-09T17:08:41.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:08:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7831.042906 secs 2023-03-09T17:08:41.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:08:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7831.042906 secs 2023-03-09T17:08:41.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:08:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7831.042906 secs 2023-03-09T17:08:42.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:08:41 smithi038 ceph-mon[92422]: pgmap v4691: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 110 KiB/s wr, 117 op/s 2023-03-09T17:08:42.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:08:41 smithi122 ceph-mon[93869]: pgmap v4691: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 110 KiB/s wr, 117 op/s 2023-03-09T17:08:42.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:08:41 smithi181 ceph-mon[94189]: pgmap v4691: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 110 KiB/s wr, 117 op/s 2023-03-09T17:08:43.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8295.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:08:44.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:08:43 smithi038 ceph-mon[92422]: pgmap v4692: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 55 KiB/s wr, 36 op/s 2023-03-09T17:08:44.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:08:43 smithi122 ceph-mon[93869]: pgmap v4692: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 55 KiB/s wr, 36 op/s 2023-03-09T17:08:44.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:08:43 smithi181 ceph-mon[94189]: pgmap v4692: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 55 KiB/s wr, 36 op/s 2023-03-09T17:08:45.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:08:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7836.042976 secs 2023-03-09T17:08:45.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:08:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7836.042976 secs 2023-03-09T17:08:45.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:08:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7836.042976 secs 2023-03-09T17:08:46.724 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:08:46 smithi122 ceph-mon[93869]: pgmap v4693: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 86 KiB/s wr, 12 op/s 2023-03-09T17:08:46.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:08:46 smithi181 ceph-mon[94189]: pgmap v4693: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 86 KiB/s wr, 12 op/s 2023-03-09T17:08:46.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:08:46 smithi038 ceph-mon[92422]: pgmap v4693: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 86 KiB/s wr, 12 op/s 2023-03-09T17:08:48.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:08:47 smithi038 ceph-mon[92422]: pgmap v4694: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 91 KiB/s wr, 10 op/s 2023-03-09T17:08:48.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:08:47 smithi122 ceph-mon[93869]: pgmap v4694: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 91 KiB/s wr, 10 op/s 2023-03-09T17:08:48.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:08:47 smithi181 ceph-mon[94189]: pgmap v4694: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 91 KiB/s wr, 10 op/s 2023-03-09T17:08:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8300.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:08:50.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:08:49 smithi038 ceph-mon[92422]: pgmap v4695: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 84 KiB/s wr, 9 op/s 2023-03-09T17:08:50.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:08:49 smithi122 ceph-mon[93869]: pgmap v4695: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 84 KiB/s wr, 9 op/s 2023-03-09T17:08:50.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:08:49 smithi181 ceph-mon[94189]: pgmap v4695: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 84 KiB/s wr, 9 op/s 2023-03-09T17:08:51.247 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:08:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7841.043037 secs 2023-03-09T17:08:51.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:08:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7841.043037 secs 2023-03-09T17:08:51.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:08:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7841.043037 secs 2023-03-09T17:08:52.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:08:51 smithi038 ceph-mon[92422]: pgmap v4696: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 105 KiB/s wr, 13 op/s 2023-03-09T17:08:52.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:08:51 smithi122 ceph-mon[93869]: pgmap v4696: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 105 KiB/s wr, 13 op/s 2023-03-09T17:08:52.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:08:51 smithi181 ceph-mon[94189]: pgmap v4696: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 105 KiB/s wr, 13 op/s 2023-03-09T17:08:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8305.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:08:54.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:08:53 smithi038 ceph-mon[92422]: pgmap v4697: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 9 op/s 2023-03-09T17:08:54.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:08:53 smithi122 ceph-mon[93869]: pgmap v4697: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 9 op/s 2023-03-09T17:08:54.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:08:53 smithi181 ceph-mon[94189]: pgmap v4697: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 9 op/s 2023-03-09T17:08:55.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:08:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7846.043096 secs 2023-03-09T17:08:55.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:08:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7846.043096 secs 2023-03-09T17:08:55.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:08:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7846.043096 secs 2023-03-09T17:08:56.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:08:56 smithi122 ceph-mon[93869]: pgmap v4698: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 123 KiB/s wr, 11 op/s 2023-03-09T17:08:56.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:08:56 smithi181 ceph-mon[94189]: pgmap v4698: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 123 KiB/s wr, 11 op/s 2023-03-09T17:08:56.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:08:56 smithi038 ceph-mon[92422]: pgmap v4698: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 123 KiB/s wr, 11 op/s 2023-03-09T17:08:57.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:08:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7848.300616 secs 2023-03-09T17:08:57.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:08:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7848.300616 secs 2023-03-09T17:08:57.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:08:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7848.300616 secs 2023-03-09T17:08:58.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:08:58 smithi122 ceph-mon[93869]: pgmap v4699: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 87 KiB/s wr, 9 op/s 2023-03-09T17:08:58.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:08:58 smithi181 ceph-mon[94189]: pgmap v4699: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 87 KiB/s wr, 9 op/s 2023-03-09T17:08:58.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:08:58 smithi038 ceph-mon[92422]: pgmap v4699: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 87 KiB/s wr, 9 op/s 2023-03-09T17:08:58.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8310.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:08:59.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:08:59 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:08:59.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:08:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:08:59.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:08:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:08:59.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:08:59 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:08:59.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:08:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:08:59.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:08:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:08:59.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:08:59 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:08:59.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:08:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:08:59.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:08:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:09:00.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:09:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7851.043158 secs 2023-03-09T17:09:00.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:09:00 smithi122 ceph-mon[93869]: pgmap v4700: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 80 KiB/s wr, 8 op/s 2023-03-09T17:09:00.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:09:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7851.043158 secs 2023-03-09T17:09:00.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:09:00 smithi181 ceph-mon[94189]: pgmap v4700: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 80 KiB/s wr, 8 op/s 2023-03-09T17:09:00.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:09:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7851.043158 secs 2023-03-09T17:09:00.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:09:00 smithi038 ceph-mon[92422]: pgmap v4700: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 80 KiB/s wr, 8 op/s 2023-03-09T17:09:02.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:09:01 smithi038 ceph-mon[92422]: pgmap v4701: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 115 KiB/s wr, 13 op/s 2023-03-09T17:09:02.474 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:09:01 smithi122 ceph-mon[93869]: pgmap v4701: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 115 KiB/s wr, 13 op/s 2023-03-09T17:09:02.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:09:01 smithi181 ceph-mon[94189]: pgmap v4701: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 115 KiB/s wr, 13 op/s 2023-03-09T17:09:03.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:09:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7853.300715 secs 2023-03-09T17:09:03.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:09:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7853.300715 secs 2023-03-09T17:09:03.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:09:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7853.300715 secs 2023-03-09T17:09:03.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8315.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:09:04.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:09:03 smithi038 ceph-mon[92422]: pgmap v4702: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 115 KiB/s wr, 21 op/s 2023-03-09T17:09:04.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:09:03 smithi122 ceph-mon[93869]: pgmap v4702: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 115 KiB/s wr, 21 op/s 2023-03-09T17:09:04.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:09:03 smithi181 ceph-mon[94189]: pgmap v4702: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 115 KiB/s wr, 21 op/s 2023-03-09T17:09:05.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:09:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7856.043218 secs 2023-03-09T17:09:05.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:09:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7856.043218 secs 2023-03-09T17:09:05.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:09:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7856.043218 secs 2023-03-09T17:09:06.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:09:06 smithi122 ceph-mon[93869]: pgmap v4703: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 128 KiB/s wr, 24 op/s 2023-03-09T17:09:06.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:09:06 smithi181 ceph-mon[94189]: pgmap v4703: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 128 KiB/s wr, 24 op/s 2023-03-09T17:09:06.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:09:06 smithi038 ceph-mon[92422]: pgmap v4703: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 128 KiB/s wr, 24 op/s 2023-03-09T17:09:07.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:09:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7858.300852 secs 2023-03-09T17:09:07.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:09:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7858.300852 secs 2023-03-09T17:09:07.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:09:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7858.300852 secs 2023-03-09T17:09:08.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:09:08 smithi122 ceph-mon[93869]: pgmap v4704: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 83 KiB/s wr, 22 op/s 2023-03-09T17:09:08.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:09:08 smithi181 ceph-mon[94189]: pgmap v4704: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 83 KiB/s wr, 22 op/s 2023-03-09T17:09:08.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:09:08 smithi038 ceph-mon[92422]: pgmap v4704: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 83 KiB/s wr, 22 op/s 2023-03-09T17:09:08.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8320.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:09:10.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:09:10 smithi122 ceph-mon[93869]: osdmap e507: 12 total, 12 up, 12 in 2023-03-09T17:09:10.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:09:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7861.043279 secs 2023-03-09T17:09:10.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:09:10 smithi122 ceph-mon[93869]: pgmap v4706: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 92 KiB/s wr, 25 op/s 2023-03-09T17:09:10.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:09:10 smithi181 ceph-mon[94189]: osdmap e507: 12 total, 12 up, 12 in 2023-03-09T17:09:10.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:09:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7861.043279 secs 2023-03-09T17:09:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:09:10 smithi181 ceph-mon[94189]: pgmap v4706: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 92 KiB/s wr, 25 op/s 2023-03-09T17:09:10.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:09:10 smithi038 ceph-mon[92422]: osdmap e507: 12 total, 12 up, 12 in 2023-03-09T17:09:10.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:09:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7861.043279 secs 2023-03-09T17:09:10.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:09:10 smithi038 ceph-mon[92422]: pgmap v4706: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 92 KiB/s wr, 25 op/s 2023-03-09T17:09:12.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:09:11 smithi038 ceph-mon[92422]: pgmap v4707: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 62 KiB/s wr, 24 op/s 2023-03-09T17:09:12.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:09:11 smithi122 ceph-mon[93869]: pgmap v4707: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 62 KiB/s wr, 24 op/s 2023-03-09T17:09:12.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:09:11 smithi181 ceph-mon[94189]: pgmap v4707: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 62 KiB/s wr, 24 op/s 2023-03-09T17:09:13.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:09:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7863.300967 secs 2023-03-09T17:09:13.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:09:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7863.300967 secs 2023-03-09T17:09:13.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:09:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7863.300967 secs 2023-03-09T17:09:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8325.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:09:14.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:09:13 smithi038 ceph-mon[92422]: pgmap v4708: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 37 KiB/s wr, 10 op/s 2023-03-09T17:09:14.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:09:13 smithi122 ceph-mon[93869]: pgmap v4708: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 37 KiB/s wr, 10 op/s 2023-03-09T17:09:14.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:09:13 smithi181 ceph-mon[94189]: pgmap v4708: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 37 KiB/s wr, 10 op/s 2023-03-09T17:09:15.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:09:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:09:15.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:09:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:09:15.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:09:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:09:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:09:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:09:15.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:09:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:09:15.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:09:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:09:15.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:09:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:09:15.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:09:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:09:15.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:09:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:09:15.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:09:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:09:15.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:09:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:09:15.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:09:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:09:16.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:09:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7866.043355 secs 2023-03-09T17:09:16.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:09:16 smithi122 ceph-mon[93869]: pgmap v4709: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 22 KiB/s wr, 8 op/s 2023-03-09T17:09:16.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:09:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7866.043355 secs 2023-03-09T17:09:16.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:09:16 smithi181 ceph-mon[94189]: pgmap v4709: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 22 KiB/s wr, 8 op/s 2023-03-09T17:09:16.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:09:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7866.043355 secs 2023-03-09T17:09:16.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:09:16 smithi038 ceph-mon[92422]: pgmap v4709: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 22 KiB/s wr, 8 op/s 2023-03-09T17:09:17.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:09:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7868.301130 secs 2023-03-09T17:09:17.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:09:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7868.301130 secs 2023-03-09T17:09:17.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:09:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7868.301130 secs 2023-03-09T17:09:18.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:09:18 smithi122 ceph-mon[93869]: pgmap v4710: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 14 KiB/s wr, 8 op/s 2023-03-09T17:09:18.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:09:18 smithi181 ceph-mon[94189]: pgmap v4710: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 14 KiB/s wr, 8 op/s 2023-03-09T17:09:18.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:09:18 smithi038 ceph-mon[92422]: pgmap v4710: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 14 KiB/s wr, 8 op/s 2023-03-09T17:09:18.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8330.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:09:20.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:09:19 smithi038 ceph-mon[92422]: pgmap v4711: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T17:09:20.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:09:19 smithi122 ceph-mon[93869]: pgmap v4711: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T17:09:20.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:09:19 smithi181 ceph-mon[94189]: pgmap v4711: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T17:09:21.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:09:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7871.043413 secs 2023-03-09T17:09:21.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:09:20 smithi038 ceph-mon[92422]: osdmap e508: 12 total, 12 up, 12 in 2023-03-09T17:09:21.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:09:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7871.043413 secs 2023-03-09T17:09:21.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:09:20 smithi122 ceph-mon[93869]: osdmap e508: 12 total, 12 up, 12 in 2023-03-09T17:09:21.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:09:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7871.043413 secs 2023-03-09T17:09:21.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:09:20 smithi181 ceph-mon[94189]: osdmap e508: 12 total, 12 up, 12 in 2023-03-09T17:09:22.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:09:21 smithi038 ceph-mon[92422]: pgmap v4713: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 21 KiB/s wr, 7 op/s 2023-03-09T17:09:22.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:09:21 smithi122 ceph-mon[93869]: pgmap v4713: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 21 KiB/s wr, 7 op/s 2023-03-09T17:09:22.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:09:21 smithi181 ceph-mon[94189]: pgmap v4713: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 21 KiB/s wr, 7 op/s 2023-03-09T17:09:23.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:09:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7873.301258 secs 2023-03-09T17:09:23.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:09:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7873.301258 secs 2023-03-09T17:09:23.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:09:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7873.301258 secs 2023-03-09T17:09:23.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8335.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:09:24.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:09:24 smithi122 ceph-mon[93869]: pgmap v4714: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T17:09:24.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:09:24 smithi181 ceph-mon[94189]: pgmap v4714: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T17:09:24.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:09:24 smithi038 ceph-mon[92422]: pgmap v4714: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T17:09:25.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:09:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7876.043498 secs 2023-03-09T17:09:25.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:09:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7876.043498 secs 2023-03-09T17:09:25.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:09:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7876.043498 secs 2023-03-09T17:09:26.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:09:26 smithi122 ceph-mon[93869]: pgmap v4715: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T17:09:26.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:09:26 smithi181 ceph-mon[94189]: pgmap v4715: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T17:09:26.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:09:26 smithi038 ceph-mon[92422]: pgmap v4715: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T17:09:27.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:09:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7878.301328 secs 2023-03-09T17:09:27.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:09:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7878.301328 secs 2023-03-09T17:09:27.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:09:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7878.301328 secs 2023-03-09T17:09:28.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:09:28 smithi122 ceph-mon[93869]: pgmap v4716: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T17:09:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:09:28 smithi181 ceph-mon[94189]: pgmap v4716: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T17:09:28.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:09:28 smithi038 ceph-mon[92422]: pgmap v4716: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T17:09:28.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8340.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:09:30.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:09:29 smithi038 ceph-mon[92422]: pgmap v4717: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T17:09:30.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:09:29 smithi122 ceph-mon[93869]: pgmap v4717: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T17:09:30.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:09:29 smithi181 ceph-mon[94189]: pgmap v4717: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T17:09:31.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:09:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7881.043562 secs 2023-03-09T17:09:31.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:09:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7881.043562 secs 2023-03-09T17:09:31.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:09:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7881.043562 secs 2023-03-09T17:09:32.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:09:32 smithi122 ceph-mon[93869]: pgmap v4718: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 18 KiB/s wr, 9 op/s 2023-03-09T17:09:32.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:09:32 smithi181 ceph-mon[94189]: pgmap v4718: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 18 KiB/s wr, 9 op/s 2023-03-09T17:09:32.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:09:32 smithi038 ceph-mon[92422]: pgmap v4718: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 18 KiB/s wr, 9 op/s 2023-03-09T17:09:33.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:09:33 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 7883.301449 secs 2023-03-09T17:09:33.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:09:33 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 7883.301449 secs 2023-03-09T17:09:33.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:09:33 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 7883.301449 secs 2023-03-09T17:09:33.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8345.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:09:34.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:09:34 smithi122 ceph-mon[93869]: pgmap v4719: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 6 op/s 2023-03-09T17:09:34.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:09:34 smithi181 ceph-mon[94189]: pgmap v4719: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 6 op/s 2023-03-09T17:09:34.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:09:34 smithi038 ceph-mon[92422]: pgmap v4719: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 6 op/s 2023-03-09T17:09:35.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:09:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7886.043623 secs 2023-03-09T17:09:35.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:09:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7886.043623 secs 2023-03-09T17:09:35.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:09:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7886.043623 secs 2023-03-09T17:09:36.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:09:36 smithi122 ceph-mon[93869]: pgmap v4720: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 30 KiB/s wr, 8 op/s 2023-03-09T17:09:36.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:09:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:09:36.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:09:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:09:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:09:36 smithi181 ceph-mon[94189]: pgmap v4720: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 30 KiB/s wr, 8 op/s 2023-03-09T17:09:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:09:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:09:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:09:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:09:36.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:09:36 smithi038 ceph-mon[92422]: pgmap v4720: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 30 KiB/s wr, 8 op/s 2023-03-09T17:09:36.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:09:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:09:36.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:09:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:09:37.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:09:37 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 7888.301596 secs 2023-03-09T17:09:37.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:09:37 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 7888.301596 secs 2023-03-09T17:09:37.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:09:37 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 7888.301596 secs 2023-03-09T17:09:38.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:09:38 smithi122 ceph-mon[93869]: pgmap v4721: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 30 KiB/s wr, 6 op/s 2023-03-09T17:09:38.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:09:38 smithi181 ceph-mon[94189]: pgmap v4721: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 30 KiB/s wr, 6 op/s 2023-03-09T17:09:38.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:09:38 smithi038 ceph-mon[92422]: pgmap v4721: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 30 KiB/s wr, 6 op/s 2023-03-09T17:09:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8350.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:09:40.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:09:39 smithi038 ceph-mon[92422]: pgmap v4722: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 30 KiB/s wr, 6 op/s 2023-03-09T17:09:40.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:09:39 smithi122 ceph-mon[93869]: pgmap v4722: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 30 KiB/s wr, 6 op/s 2023-03-09T17:09:40.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:09:39 smithi181 ceph-mon[94189]: pgmap v4722: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 30 KiB/s wr, 6 op/s 2023-03-09T17:09:41.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:09:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7891.043679 secs 2023-03-09T17:09:41.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:09:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7891.043679 secs 2023-03-09T17:09:41.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:09:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7891.043679 secs 2023-03-09T17:09:42.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:09:41 smithi038 ceph-mon[92422]: pgmap v4723: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 62 KiB/s wr, 9 op/s 2023-03-09T17:09:42.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:09:41 smithi122 ceph-mon[93869]: pgmap v4723: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 62 KiB/s wr, 9 op/s 2023-03-09T17:09:42.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:09:41 smithi181 ceph-mon[94189]: pgmap v4723: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 62 KiB/s wr, 9 op/s 2023-03-09T17:09:43.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:09:43 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 7893.301754 secs 2023-03-09T17:09:43.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:09:43 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 7893.301754 secs 2023-03-09T17:09:43.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:09:43 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 7893.301754 secs 2023-03-09T17:09:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8355.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:09:44.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:09:44 smithi122 ceph-mon[93869]: pgmap v4724: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 46 KiB/s wr, 6 op/s 2023-03-09T17:09:44.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:09:44 smithi181 ceph-mon[94189]: pgmap v4724: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 46 KiB/s wr, 6 op/s 2023-03-09T17:09:44.497 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:09:44 smithi038 ceph-mon[92422]: pgmap v4724: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 46 KiB/s wr, 6 op/s 2023-03-09T17:09:45.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:09:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7896.043765 secs 2023-03-09T17:09:45.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:09:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7896.043765 secs 2023-03-09T17:09:45.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:09:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7896.043765 secs 2023-03-09T17:09:46.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:09:46 smithi122 ceph-mon[93869]: pgmap v4725: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 50 KiB/s wr, 9 op/s 2023-03-09T17:09:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:09:46 smithi181 ceph-mon[94189]: pgmap v4725: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 50 KiB/s wr, 9 op/s 2023-03-09T17:09:46.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:09:46 smithi038 ceph-mon[92422]: pgmap v4725: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 50 KiB/s wr, 9 op/s 2023-03-09T17:09:47.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:09:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7898.301895 secs 2023-03-09T17:09:47.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:09:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7898.301895 secs 2023-03-09T17:09:47.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:09:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7898.301895 secs 2023-03-09T17:09:48.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:09:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:09:48.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:09:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:09:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:09:48 smithi122 ceph-mon[93869]: pgmap v4726: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 43 KiB/s wr, 8 op/s 2023-03-09T17:09:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:09:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:09:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:09:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:09:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:09:48 smithi181 ceph-mon[94189]: pgmap v4726: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 43 KiB/s wr, 8 op/s 2023-03-09T17:09:48.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:09:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:09:48.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:09:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:09:48.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:09:48 smithi038 ceph-mon[92422]: pgmap v4726: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 43 KiB/s wr, 8 op/s 2023-03-09T17:09:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8360.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:09:50.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:09:49 smithi038 ceph-mon[92422]: pgmap v4727: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 43 KiB/s wr, 8 op/s 2023-03-09T17:09:50.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:09:49 smithi122 ceph-mon[93869]: pgmap v4727: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 43 KiB/s wr, 8 op/s 2023-03-09T17:09:50.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:09:49 smithi181 ceph-mon[94189]: pgmap v4727: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 43 KiB/s wr, 8 op/s 2023-03-09T17:09:51.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:09:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7901.043824 secs 2023-03-09T17:09:51.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:09:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7901.043824 secs 2023-03-09T17:09:51.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:09:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7901.043824 secs 2023-03-09T17:09:52.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:09:52 smithi122 ceph-mon[93869]: pgmap v4728: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 102 KiB/s wr, 12 op/s 2023-03-09T17:09:52.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:09:52 smithi181 ceph-mon[94189]: pgmap v4728: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 102 KiB/s wr, 12 op/s 2023-03-09T17:09:52.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:09:52 smithi038 ceph-mon[92422]: pgmap v4728: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 102 KiB/s wr, 12 op/s 2023-03-09T17:09:53.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:09:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7903.302018 secs 2023-03-09T17:09:53.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:09:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7903.302018 secs 2023-03-09T17:09:53.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:09:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7903.302018 secs 2023-03-09T17:09:53.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8365.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:09:54.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:09:54 smithi122 ceph-mon[93869]: pgmap v4729: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 70 KiB/s wr, 9 op/s 2023-03-09T17:09:54.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:09:54 smithi181 ceph-mon[94189]: pgmap v4729: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 70 KiB/s wr, 9 op/s 2023-03-09T17:09:54.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:09:54 smithi038 ceph-mon[92422]: pgmap v4729: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 70 KiB/s wr, 9 op/s 2023-03-09T17:09:55.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:09:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7906.043885 secs 2023-03-09T17:09:55.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:09:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7906.043885 secs 2023-03-09T17:09:55.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:09:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7906.043885 secs 2023-03-09T17:09:56.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:09:56 smithi122 ceph-mon[93869]: pgmap v4730: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 73 KiB/s wr, 11 op/s 2023-03-09T17:09:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:09:56 smithi181 ceph-mon[94189]: pgmap v4730: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 73 KiB/s wr, 11 op/s 2023-03-09T17:09:56.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:09:56 smithi038 ceph-mon[92422]: pgmap v4730: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 73 KiB/s wr, 11 op/s 2023-03-09T17:09:57.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:09:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7908.302139 secs 2023-03-09T17:09:57.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:09:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7908.302139 secs 2023-03-09T17:09:57.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:09:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7908.302139 secs 2023-03-09T17:09:58.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:09:58 smithi122 ceph-mon[93869]: pgmap v4731: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 70 KiB/s wr, 10 op/s 2023-03-09T17:09:58.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:09:58 smithi181 ceph-mon[94189]: pgmap v4731: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 70 KiB/s wr, 10 op/s 2023-03-09T17:09:58.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:09:58 smithi038 ceph-mon[92422]: pgmap v4731: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 70 KiB/s wr, 10 op/s 2023-03-09T17:09:58.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8370.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:10:00.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:09:59 smithi038 ceph-mon[92422]: pgmap v4732: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 63 KiB/s wr, 8 op/s 2023-03-09T17:10:00.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:09:59 smithi122 ceph-mon[93869]: pgmap v4732: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 63 KiB/s wr, 8 op/s 2023-03-09T17:10:00.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:09:59 smithi181 ceph-mon[94189]: pgmap v4732: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 63 KiB/s wr, 8 op/s 2023-03-09T17:10:01.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7911.043947 secs 2023-03-09T17:10:01.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:00 smithi038 ceph-mon[92422]: overall HEALTH_WARN 2 MDSs report slow requests 2023-03-09T17:10:01.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7911.043947 secs 2023-03-09T17:10:01.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:00 smithi122 ceph-mon[93869]: overall HEALTH_WARN 2 MDSs report slow requests 2023-03-09T17:10:01.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7911.043947 secs 2023-03-09T17:10:01.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:00 smithi181 ceph-mon[94189]: overall HEALTH_WARN 2 MDSs report slow requests 2023-03-09T17:10:02.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:02 smithi122 ceph-mon[93869]: pgmap v4733: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 140 KiB/s wr, 16 op/s 2023-03-09T17:10:02.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:02 smithi181 ceph-mon[94189]: pgmap v4733: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 140 KiB/s wr, 16 op/s 2023-03-09T17:10:02.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:02 smithi038 ceph-mon[92422]: pgmap v4733: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 140 KiB/s wr, 16 op/s 2023-03-09T17:10:03.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7913.302261 secs 2023-03-09T17:10:03.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7913.302261 secs 2023-03-09T17:10:03.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7913.302261 secs 2023-03-09T17:10:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8375.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:10:04.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:04 smithi122 ceph-mon[93869]: pgmap v4734: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 103 KiB/s wr, 25 op/s 2023-03-09T17:10:04.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:04 smithi181 ceph-mon[94189]: pgmap v4734: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 103 KiB/s wr, 25 op/s 2023-03-09T17:10:04.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:04 smithi038 ceph-mon[92422]: pgmap v4734: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 103 KiB/s wr, 25 op/s 2023-03-09T17:10:05.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7916.043995 secs 2023-03-09T17:10:05.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7916.043995 secs 2023-03-09T17:10:05.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7916.043995 secs 2023-03-09T17:10:06.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:06 smithi122 ceph-mon[93869]: pgmap v4735: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 126 KiB/s wr, 29 op/s 2023-03-09T17:10:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:06 smithi181 ceph-mon[94189]: pgmap v4735: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 126 KiB/s wr, 29 op/s 2023-03-09T17:10:06.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:06 smithi038 ceph-mon[92422]: pgmap v4735: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 126 KiB/s wr, 29 op/s 2023-03-09T17:10:07.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7918.302328 secs 2023-03-09T17:10:07.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7918.302328 secs 2023-03-09T17:10:07.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7918.302328 secs 2023-03-09T17:10:08.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:08 smithi122 ceph-mon[93869]: pgmap v4736: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 133 KiB/s wr, 28 op/s 2023-03-09T17:10:08.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:08 smithi181 ceph-mon[94189]: pgmap v4736: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 133 KiB/s wr, 28 op/s 2023-03-09T17:10:08.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:08 smithi038 ceph-mon[92422]: pgmap v4736: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 133 KiB/s wr, 28 op/s 2023-03-09T17:10:08.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8380.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:10:10.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:10 smithi122 ceph-mon[93869]: osdmap e509: 12 total, 12 up, 12 in 2023-03-09T17:10:10.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7921.044058 secs 2023-03-09T17:10:10.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:10 smithi122 ceph-mon[93869]: pgmap v4738: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 160 KiB/s wr, 32 op/s 2023-03-09T17:10:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:10 smithi181 ceph-mon[94189]: osdmap e509: 12 total, 12 up, 12 in 2023-03-09T17:10:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7921.044058 secs 2023-03-09T17:10:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:10 smithi181 ceph-mon[94189]: pgmap v4738: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 160 KiB/s wr, 32 op/s 2023-03-09T17:10:10.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:10 smithi038 ceph-mon[92422]: osdmap e509: 12 total, 12 up, 12 in 2023-03-09T17:10:10.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7921.044058 secs 2023-03-09T17:10:10.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:10 smithi038 ceph-mon[92422]: pgmap v4738: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 160 KiB/s wr, 32 op/s 2023-03-09T17:10:12.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:11 smithi038 ceph-mon[92422]: pgmap v4739: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 108 KiB/s wr, 27 op/s 2023-03-09T17:10:12.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:11 smithi122 ceph-mon[93869]: pgmap v4739: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 108 KiB/s wr, 27 op/s 2023-03-09T17:10:12.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:11 smithi181 ceph-mon[94189]: pgmap v4739: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 108 KiB/s wr, 27 op/s 2023-03-09T17:10:13.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7923.302442 secs 2023-03-09T17:10:13.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7923.302442 secs 2023-03-09T17:10:13.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7923.302442 secs 2023-03-09T17:10:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8385.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:10:14.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:14 smithi122 ceph-mon[93869]: pgmap v4740: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 82 KiB/s wr, 11 op/s 2023-03-09T17:10:14.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:14 smithi181 ceph-mon[94189]: pgmap v4740: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 82 KiB/s wr, 11 op/s 2023-03-09T17:10:14.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:14 smithi038 ceph-mon[92422]: pgmap v4740: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 82 KiB/s wr, 11 op/s 2023-03-09T17:10:15.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:10:15.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:10:15.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:10:16.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7926.044118 secs 2023-03-09T17:10:16.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:16 smithi122 ceph-mon[93869]: pgmap v4741: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 56 KiB/s wr, 8 op/s 2023-03-09T17:10:16.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7926.044118 secs 2023-03-09T17:10:16.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:16 smithi181 ceph-mon[94189]: pgmap v4741: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 56 KiB/s wr, 8 op/s 2023-03-09T17:10:16.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7926.044118 secs 2023-03-09T17:10:16.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:16 smithi038 ceph-mon[92422]: pgmap v4741: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 56 KiB/s wr, 8 op/s 2023-03-09T17:10:17.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7928.302552 secs 2023-03-09T17:10:17.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7928.302552 secs 2023-03-09T17:10:17.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7928.302552 secs 2023-03-09T17:10:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8390.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:10:19.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:18 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:10:19.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:18 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:10:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:18 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:10:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:18 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:10:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:18 smithi122 ceph-mon[93869]: pgmap v4742: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 56 KiB/s wr, 7 op/s 2023-03-09T17:10:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:18 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:10:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:18 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:10:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:18 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:10:19.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:18 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:10:19.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:18 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:10:19.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:18 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:10:19.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:18 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:10:19.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:18 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:10:19.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:18 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:10:19.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:18 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:10:19.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:18 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:10:19.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:18 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:10:19.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:18 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:10:19.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:18 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:10:19.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:18 smithi181 ceph-mon[94189]: pgmap v4742: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 56 KiB/s wr, 7 op/s 2023-03-09T17:10:19.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:18 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:10:19.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:18 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:10:19.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:18 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:10:19.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:18 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:10:19.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:18 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:10:19.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:18 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:10:19.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:18 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:10:19.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:18 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:10:19.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:18 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:10:19.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:18 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:10:19.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:18 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:10:19.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:18 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:10:19.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:18 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:10:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:18 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:10:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:18 smithi038 ceph-mon[92422]: pgmap v4742: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 56 KiB/s wr, 7 op/s 2023-03-09T17:10:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:18 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:10:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:18 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:10:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:18 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:10:19.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:18 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:10:19.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:18 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:10:19.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:18 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:10:19.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:18 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:10:19.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:18 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:10:19.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:18 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:10:19.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:18 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:10:20.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:19 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi038 to 197.2M 2023-03-09T17:10:20.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:19 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T17:10:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:19 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi038 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-03-09T17:10:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:19 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T17:10:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:10:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:10:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:10:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:10:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:10:20.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:10:20.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:10:20.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:19 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi038 to 197.2M 2023-03-09T17:10:20.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:19 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T17:10:20.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:19 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi038 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-03-09T17:10:20.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:19 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T17:10:20.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:10:20.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:10:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:10:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:10:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:10:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:10:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:10:20.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:19 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi038 to 197.2M 2023-03-09T17:10:20.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:19 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T17:10:20.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:19 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi038 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-03-09T17:10:20.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:19 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T17:10:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:10:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:10:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:10:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:10:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:10:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:10:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:10:21.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:20 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T17:10:21.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:20 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T17:10:21.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7931.044177 secs 2023-03-09T17:10:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:20 smithi122 ceph-mon[93869]: pgmap v4743: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 53 KiB/s wr, 7 op/s 2023-03-09T17:10:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:20 smithi122 ceph-mon[93869]: osdmap e510: 12 total, 12 up, 12 in 2023-03-09T17:10:21.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:20 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T17:10:21.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:20 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T17:10:21.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7931.044177 secs 2023-03-09T17:10:21.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:20 smithi181 ceph-mon[94189]: pgmap v4743: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 53 KiB/s wr, 7 op/s 2023-03-09T17:10:21.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:20 smithi181 ceph-mon[94189]: osdmap e510: 12 total, 12 up, 12 in 2023-03-09T17:10:21.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:20 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T17:10:21.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:20 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T17:10:21.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7931.044177 secs 2023-03-09T17:10:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:20 smithi038 ceph-mon[92422]: pgmap v4743: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 53 KiB/s wr, 7 op/s 2023-03-09T17:10:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:20 smithi038 ceph-mon[92422]: osdmap e510: 12 total, 12 up, 12 in 2023-03-09T17:10:22.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:21 smithi038 ceph-mon[92422]: pgmap v4745: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 7 op/s 2023-03-09T17:10:22.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:21 smithi122 ceph-mon[93869]: pgmap v4745: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 7 op/s 2023-03-09T17:10:22.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:21 smithi181 ceph-mon[94189]: pgmap v4745: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 7 op/s 2023-03-09T17:10:23.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7933.302681 secs 2023-03-09T17:10:23.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7933.302681 secs 2023-03-09T17:10:23.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7933.302681 secs 2023-03-09T17:10:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8395.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:10:24.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:24 smithi122 ceph-mon[93869]: pgmap v4746: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 7 op/s 2023-03-09T17:10:24.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:24 smithi181 ceph-mon[94189]: pgmap v4746: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 7 op/s 2023-03-09T17:10:24.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:24 smithi038 ceph-mon[92422]: pgmap v4746: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 7 op/s 2023-03-09T17:10:25.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7936.044237 secs 2023-03-09T17:10:25.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7936.044237 secs 2023-03-09T17:10:25.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7936.044237 secs 2023-03-09T17:10:26.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:26 smithi122 ceph-mon[93869]: pgmap v4747: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 6 op/s 2023-03-09T17:10:26.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:26 smithi181 ceph-mon[94189]: pgmap v4747: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 6 op/s 2023-03-09T17:10:26.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:26 smithi038 ceph-mon[92422]: pgmap v4747: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 6 op/s 2023-03-09T17:10:27.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7938.302791 secs 2023-03-09T17:10:27.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7938.302791 secs 2023-03-09T17:10:27.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7938.302791 secs 2023-03-09T17:10:28.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:28 smithi122 ceph-mon[93869]: pgmap v4748: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 36 KiB/s wr, 7 op/s 2023-03-09T17:10:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:28 smithi181 ceph-mon[94189]: pgmap v4748: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 36 KiB/s wr, 7 op/s 2023-03-09T17:10:28.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:28 smithi038 ceph-mon[92422]: pgmap v4748: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 36 KiB/s wr, 7 op/s 2023-03-09T17:10:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8400.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:10:30.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:30 smithi122 ceph-mon[93869]: pgmap v4749: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 36 KiB/s wr, 7 op/s 2023-03-09T17:10:30.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:30 smithi181 ceph-mon[94189]: pgmap v4749: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 36 KiB/s wr, 7 op/s 2023-03-09T17:10:30.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:30 smithi038 ceph-mon[92422]: pgmap v4749: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 36 KiB/s wr, 7 op/s 2023-03-09T17:10:31.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7941.044297 secs 2023-03-09T17:10:31.478 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7941.044297 secs 2023-03-09T17:10:31.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7941.044297 secs 2023-03-09T17:10:32.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:32 smithi122 ceph-mon[93869]: pgmap v4750: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 30 KiB/s wr, 9 op/s 2023-03-09T17:10:32.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:32 smithi181 ceph-mon[94189]: pgmap v4750: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 30 KiB/s wr, 9 op/s 2023-03-09T17:10:32.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:32 smithi038 ceph-mon[92422]: pgmap v4750: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 30 KiB/s wr, 9 op/s 2023-03-09T17:10:33.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7943.302907 secs 2023-03-09T17:10:33.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7943.302907 secs 2023-03-09T17:10:33.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7943.302907 secs 2023-03-09T17:10:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8405.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:10:34.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:34 smithi122 ceph-mon[93869]: pgmap v4751: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 29 KiB/s wr, 6 op/s 2023-03-09T17:10:34.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:34 smithi181 ceph-mon[94189]: pgmap v4751: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 29 KiB/s wr, 6 op/s 2023-03-09T17:10:34.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:34 smithi038 ceph-mon[92422]: pgmap v4751: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 29 KiB/s wr, 6 op/s 2023-03-09T17:10:35.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7946.044356 secs 2023-03-09T17:10:35.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7946.044356 secs 2023-03-09T17:10:35.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7946.044356 secs 2023-03-09T17:10:36.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:36 smithi122 ceph-mon[93869]: pgmap v4752: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 30 KiB/s wr, 8 op/s 2023-03-09T17:10:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:36 smithi181 ceph-mon[94189]: pgmap v4752: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 30 KiB/s wr, 8 op/s 2023-03-09T17:10:36.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:36 smithi038 ceph-mon[92422]: pgmap v4752: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 30 KiB/s wr, 8 op/s 2023-03-09T17:10:37.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7948.303034 secs 2023-03-09T17:10:37.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7948.303034 secs 2023-03-09T17:10:37.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7948.303034 secs 2023-03-09T17:10:38.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:38 smithi122 ceph-mon[93869]: pgmap v4753: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T17:10:38.728 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:38 smithi181 ceph-mon[94189]: pgmap v4753: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T17:10:38.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:38 smithi038 ceph-mon[92422]: pgmap v4753: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T17:10:38.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8410.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:10:40.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:40 smithi122 ceph-mon[93869]: pgmap v4754: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 KiB/s wr, 6 op/s 2023-03-09T17:10:40.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:40 smithi181 ceph-mon[94189]: pgmap v4754: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 KiB/s wr, 6 op/s 2023-03-09T17:10:40.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:40 smithi038 ceph-mon[92422]: pgmap v4754: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 KiB/s wr, 6 op/s 2023-03-09T17:10:41.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7951.044416 secs 2023-03-09T17:10:41.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7951.044416 secs 2023-03-09T17:10:41.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7951.044416 secs 2023-03-09T17:10:42.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:42 smithi122 ceph-mon[93869]: pgmap v4755: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 56 KiB/s wr, 11 op/s 2023-03-09T17:10:42.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:42 smithi181 ceph-mon[94189]: pgmap v4755: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 56 KiB/s wr, 11 op/s 2023-03-09T17:10:42.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:42 smithi038 ceph-mon[92422]: pgmap v4755: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 56 KiB/s wr, 11 op/s 2023-03-09T17:10:43.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:43 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:10:43.772 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:43 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:10:43.773 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:43 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:10:43.773 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:43 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:10:43.773 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:43 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:10:43.774 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:43 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:10:43.793 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8415.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:10:44.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:44 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:10:44.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:44 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:10:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:44 smithi122 ceph-mon[93869]: pgmap v4756: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 56 KiB/s wr, 8 op/s 2023-03-09T17:10:44.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:44 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:10:44.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:44 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:10:44.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:44 smithi181 ceph-mon[94189]: pgmap v4756: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 56 KiB/s wr, 8 op/s 2023-03-09T17:10:44.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:44 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:10:44.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:44 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:10:44.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:44 smithi038 ceph-mon[92422]: pgmap v4756: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 56 KiB/s wr, 8 op/s 2023-03-09T17:10:45.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7956.044476 secs 2023-03-09T17:10:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7956.044476 secs 2023-03-09T17:10:45.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7956.044476 secs 2023-03-09T17:10:46.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:46 smithi122 ceph-mon[93869]: pgmap v4757: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 56 KiB/s wr, 10 op/s 2023-03-09T17:10:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:46 smithi181 ceph-mon[94189]: pgmap v4757: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 56 KiB/s wr, 10 op/s 2023-03-09T17:10:46.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:46 smithi038 ceph-mon[92422]: pgmap v4757: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 56 KiB/s wr, 10 op/s 2023-03-09T17:10:48.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:48 smithi122 ceph-mon[93869]: pgmap v4758: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 86 KiB/s wr, 11 op/s 2023-03-09T17:10:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:48 smithi181 ceph-mon[94189]: pgmap v4758: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 86 KiB/s wr, 11 op/s 2023-03-09T17:10:48.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:48 smithi038 ceph-mon[92422]: pgmap v4758: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 86 KiB/s wr, 11 op/s 2023-03-09T17:10:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8420.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:10:50.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:50 smithi122 ceph-mon[93869]: pgmap v4759: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 72 KiB/s wr, 9 op/s 2023-03-09T17:10:50.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:50 smithi181 ceph-mon[94189]: pgmap v4759: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 72 KiB/s wr, 9 op/s 2023-03-09T17:10:50.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:50 smithi038 ceph-mon[92422]: pgmap v4759: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 72 KiB/s wr, 9 op/s 2023-03-09T17:10:51.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7961.044538 secs 2023-03-09T17:10:51.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7961.044538 secs 2023-03-09T17:10:51.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7961.044538 secs 2023-03-09T17:10:52.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:52 smithi122 ceph-mon[93869]: pgmap v4760: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 109 KiB/s wr, 14 op/s 2023-03-09T17:10:52.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:52 smithi181 ceph-mon[94189]: pgmap v4760: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 109 KiB/s wr, 14 op/s 2023-03-09T17:10:52.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:52 smithi038 ceph-mon[92422]: pgmap v4760: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 109 KiB/s wr, 14 op/s 2023-03-09T17:10:53.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8425.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:10:54.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:54 smithi122 ceph-mon[93869]: pgmap v4761: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 68 KiB/s wr, 9 op/s 2023-03-09T17:10:54.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:54 smithi181 ceph-mon[94189]: pgmap v4761: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 68 KiB/s wr, 9 op/s 2023-03-09T17:10:54.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:54 smithi038 ceph-mon[92422]: pgmap v4761: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 68 KiB/s wr, 9 op/s 2023-03-09T17:10:55.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7966.044622 secs 2023-03-09T17:10:55.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7966.044622 secs 2023-03-09T17:10:55.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7966.044622 secs 2023-03-09T17:10:56.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:56 smithi122 ceph-mon[93869]: pgmap v4762: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 68 KiB/s wr, 10 op/s 2023-03-09T17:10:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:56 smithi181 ceph-mon[94189]: pgmap v4762: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 68 KiB/s wr, 10 op/s 2023-03-09T17:10:56.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:56 smithi038 ceph-mon[92422]: pgmap v4762: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 68 KiB/s wr, 10 op/s 2023-03-09T17:10:58.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:10:58 smithi122 ceph-mon[93869]: pgmap v4763: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 80 KiB/s wr, 10 op/s 2023-03-09T17:10:58.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:10:58 smithi181 ceph-mon[94189]: pgmap v4763: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 80 KiB/s wr, 10 op/s 2023-03-09T17:10:58.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:10:58 smithi038 ceph-mon[92422]: pgmap v4763: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 80 KiB/s wr, 10 op/s 2023-03-09T17:10:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8430.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:11:00.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:11:00 smithi122 ceph-mon[93869]: pgmap v4764: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 50 KiB/s wr, 8 op/s 2023-03-09T17:11:00.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:11:00 smithi181 ceph-mon[94189]: pgmap v4764: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 50 KiB/s wr, 8 op/s 2023-03-09T17:11:00.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:11:00 smithi038 ceph-mon[92422]: pgmap v4764: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 50 KiB/s wr, 8 op/s 2023-03-09T17:11:01.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:11:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7971.044683 secs 2023-03-09T17:11:01.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:11:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7971.044683 secs 2023-03-09T17:11:01.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:11:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7971.044683 secs 2023-03-09T17:11:02.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:11:02 smithi122 ceph-mon[93869]: pgmap v4765: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 95 KiB/s wr, 12 op/s 2023-03-09T17:11:02.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:11:02 smithi181 ceph-mon[94189]: pgmap v4765: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 95 KiB/s wr, 12 op/s 2023-03-09T17:11:02.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:11:02 smithi038 ceph-mon[92422]: pgmap v4765: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 95 KiB/s wr, 12 op/s 2023-03-09T17:11:03.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:11:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7973.303602 secs 2023-03-09T17:11:03.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:11:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7973.303602 secs 2023-03-09T17:11:03.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:11:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7973.303602 secs 2023-03-09T17:11:03.793 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8435.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:11:04.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:11:04 smithi122 ceph-mon[93869]: pgmap v4766: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 79 KiB/s wr, 20 op/s 2023-03-09T17:11:04.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:11:04 smithi181 ceph-mon[94189]: pgmap v4766: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 79 KiB/s wr, 20 op/s 2023-03-09T17:11:04.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:11:04 smithi038 ceph-mon[92422]: pgmap v4766: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 79 KiB/s wr, 20 op/s 2023-03-09T17:11:05.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:11:05 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:11:05.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:11:05 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:11:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:11:05 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:11:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:11:05 smithi122 ceph-mon[93869]: osdmap e511: 12 total, 12 up, 12 in 2023-03-09T17:11:05.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:11:05 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:11:05.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:11:05 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:11:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:11:05 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:11:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:11:05 smithi181 ceph-mon[94189]: osdmap e511: 12 total, 12 up, 12 in 2023-03-09T17:11:05.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:11:05 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:11:05.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:11:05 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:11:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:11:05 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:11:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:11:05 smithi038 ceph-mon[92422]: osdmap e511: 12 total, 12 up, 12 in 2023-03-09T17:11:06.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:11:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7976.044741 secs 2023-03-09T17:11:06.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:11:06 smithi122 ceph-mon[93869]: pgmap v4768: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 122 KiB/s wr, 27 op/s 2023-03-09T17:11:06.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:11:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7976.044741 secs 2023-03-09T17:11:06.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:11:06 smithi181 ceph-mon[94189]: pgmap v4768: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 122 KiB/s wr, 27 op/s 2023-03-09T17:11:06.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:11:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7976.044741 secs 2023-03-09T17:11:06.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:11:06 smithi038 ceph-mon[92422]: pgmap v4768: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 122 KiB/s wr, 27 op/s 2023-03-09T17:11:07.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:11:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7978.303719 secs 2023-03-09T17:11:07.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:11:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7978.303719 secs 2023-03-09T17:11:07.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:11:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7978.303719 secs 2023-03-09T17:11:08.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:11:08 smithi122 ceph-mon[93869]: pgmap v4769: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 150 KiB/s wr, 26 op/s 2023-03-09T17:11:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:11:08 smithi181 ceph-mon[94189]: pgmap v4769: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 150 KiB/s wr, 26 op/s 2023-03-09T17:11:08.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:11:08 smithi038 ceph-mon[92422]: pgmap v4769: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 150 KiB/s wr, 26 op/s 2023-03-09T17:11:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8440.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:11:10.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:11:10 smithi122 ceph-mon[93869]: pgmap v4770: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 150 KiB/s wr, 26 op/s 2023-03-09T17:11:10.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:11:10 smithi181 ceph-mon[94189]: pgmap v4770: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 150 KiB/s wr, 26 op/s 2023-03-09T17:11:10.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:11:10 smithi038 ceph-mon[92422]: pgmap v4770: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 150 KiB/s wr, 26 op/s 2023-03-09T17:11:11.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:11:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7981.044801 secs 2023-03-09T17:11:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:11:11 smithi122 ceph-mon[93869]: osdmap e512: 12 total, 12 up, 12 in 2023-03-09T17:11:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:11:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7981.044801 secs 2023-03-09T17:11:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:11:11 smithi181 ceph-mon[94189]: osdmap e512: 12 total, 12 up, 12 in 2023-03-09T17:11:11.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:11:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7981.044801 secs 2023-03-09T17:11:11.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:11:11 smithi038 ceph-mon[92422]: osdmap e512: 12 total, 12 up, 12 in 2023-03-09T17:11:12.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:11:12 smithi122 ceph-mon[93869]: pgmap v4772: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 114 KiB/s wr, 13 op/s 2023-03-09T17:11:12.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:11:12 smithi181 ceph-mon[94189]: pgmap v4772: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 114 KiB/s wr, 13 op/s 2023-03-09T17:11:12.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:11:12 smithi038 ceph-mon[92422]: pgmap v4772: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 114 KiB/s wr, 13 op/s 2023-03-09T17:11:13.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:11:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7983.303804 secs 2023-03-09T17:11:13.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:11:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7983.303804 secs 2023-03-09T17:11:13.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:11:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7983.303804 secs 2023-03-09T17:11:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8445.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:11:15.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:11:15 smithi122 ceph-mon[93869]: pgmap v4773: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 92 KiB/s wr, 11 op/s 2023-03-09T17:11:15.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:11:15 smithi181 ceph-mon[94189]: pgmap v4773: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 92 KiB/s wr, 11 op/s 2023-03-09T17:11:15.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:11:15 smithi038 ceph-mon[92422]: pgmap v4773: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 92 KiB/s wr, 11 op/s 2023-03-09T17:11:16.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:11:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7986.044880 secs 2023-03-09T17:11:16.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:11:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7986.044880 secs 2023-03-09T17:11:16.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:11:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7986.044880 secs 2023-03-09T17:11:17.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:11:17 smithi122 ceph-mon[93869]: pgmap v4774: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 65 KiB/s wr, 8 op/s 2023-03-09T17:11:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:11:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7988.303946 secs 2023-03-09T17:11:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:11:17 smithi181 ceph-mon[94189]: pgmap v4774: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 65 KiB/s wr, 8 op/s 2023-03-09T17:11:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:11:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7988.303946 secs 2023-03-09T17:11:17.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:11:17 smithi038 ceph-mon[92422]: pgmap v4774: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 65 KiB/s wr, 8 op/s 2023-03-09T17:11:17.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:11:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7988.303946 secs 2023-03-09T17:11:18.793 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8450.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:11:19.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:11:19 smithi122 ceph-mon[93869]: pgmap v4775: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T17:11:19.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:11:19 smithi181 ceph-mon[94189]: pgmap v4775: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T17:11:19.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:11:19 smithi038 ceph-mon[92422]: pgmap v4775: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T17:11:20.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:11:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:11:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:11:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7991.044942 secs 2023-03-09T17:11:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:11:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:11:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:11:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:11:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:11:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:11:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:11:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:11:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:11:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7991.044942 secs 2023-03-09T17:11:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:11:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:11:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:11:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:11:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:11:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:11:20.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:11:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:11:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:11:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7991.044942 secs 2023-03-09T17:11:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:11:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:11:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:11:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:11:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:11:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:11:21.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:11:21 smithi122 ceph-mon[93869]: pgmap v4776: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T17:11:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:11:21 smithi181 ceph-mon[94189]: pgmap v4776: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T17:11:21.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:11:21 smithi038 ceph-mon[92422]: pgmap v4776: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T17:11:22.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:11:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7993.304065 secs 2023-03-09T17:11:22.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:11:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7993.304065 secs 2023-03-09T17:11:22.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:11:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7993.304065 secs 2023-03-09T17:11:23.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:11:23 smithi122 ceph-mon[93869]: pgmap v4777: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 47 KiB/s wr, 10 op/s 2023-03-09T17:11:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:11:23 smithi181 ceph-mon[94189]: pgmap v4777: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 47 KiB/s wr, 10 op/s 2023-03-09T17:11:23.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:11:23 smithi038 ceph-mon[92422]: pgmap v4777: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 47 KiB/s wr, 10 op/s 2023-03-09T17:11:23.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8455.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:11:25.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:11:25 smithi122 ceph-mon[93869]: pgmap v4778: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 28 KiB/s wr, 6 op/s 2023-03-09T17:11:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:11:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7996.045015 secs 2023-03-09T17:11:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:11:25 smithi181 ceph-mon[94189]: pgmap v4778: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 28 KiB/s wr, 6 op/s 2023-03-09T17:11:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:11:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7996.045015 secs 2023-03-09T17:11:25.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:11:25 smithi038 ceph-mon[92422]: pgmap v4778: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 28 KiB/s wr, 6 op/s 2023-03-09T17:11:25.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:11:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7996.045015 secs 2023-03-09T17:11:27.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:11:27 smithi122 ceph-mon[93869]: pgmap v4779: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 29 KiB/s wr, 8 op/s 2023-03-09T17:11:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:11:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 7998.304175 secs 2023-03-09T17:11:27.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:11:27 smithi181 ceph-mon[94189]: pgmap v4779: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 29 KiB/s wr, 8 op/s 2023-03-09T17:11:27.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:11:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 7998.304175 secs 2023-03-09T17:11:27.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:11:27 smithi038 ceph-mon[92422]: pgmap v4779: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 29 KiB/s wr, 8 op/s 2023-03-09T17:11:27.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:11:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 7998.304175 secs 2023-03-09T17:11:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8460.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:11:29.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:11:29 smithi122 ceph-mon[93869]: pgmap v4780: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 28 KiB/s wr, 8 op/s 2023-03-09T17:11:29.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:11:29 smithi181 ceph-mon[94189]: pgmap v4780: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 28 KiB/s wr, 8 op/s 2023-03-09T17:11:29.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:11:29 smithi038 ceph-mon[92422]: pgmap v4780: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 28 KiB/s wr, 8 op/s 2023-03-09T17:11:30.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:11:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8001.045075 secs 2023-03-09T17:11:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:11:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8001.045075 secs 2023-03-09T17:11:30.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:11:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8001.045075 secs 2023-03-09T17:11:31.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:11:31 smithi122 ceph-mon[93869]: pgmap v4781: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T17:11:31.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:11:31 smithi181 ceph-mon[94189]: pgmap v4781: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T17:11:31.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:11:31 smithi038 ceph-mon[92422]: pgmap v4781: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T17:11:33.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:11:33 smithi122 ceph-mon[93869]: pgmap v4782: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 53 KiB/s wr, 10 op/s 2023-03-09T17:11:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:11:33 smithi181 ceph-mon[94189]: pgmap v4782: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 53 KiB/s wr, 10 op/s 2023-03-09T17:11:33.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:11:33 smithi038 ceph-mon[92422]: pgmap v4782: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 53 KiB/s wr, 10 op/s 2023-03-09T17:11:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8465.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:11:35.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:11:35 smithi122 ceph-mon[93869]: pgmap v4783: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 35 KiB/s wr, 8 op/s 2023-03-09T17:11:35.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:11:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8006.045137 secs 2023-03-09T17:11:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:11:35 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:11:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:11:35 smithi181 ceph-mon[94189]: pgmap v4783: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 35 KiB/s wr, 8 op/s 2023-03-09T17:11:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:11:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8006.045137 secs 2023-03-09T17:11:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:11:35 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:11:35.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:11:35 smithi038 ceph-mon[92422]: pgmap v4783: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 35 KiB/s wr, 8 op/s 2023-03-09T17:11:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:11:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8006.045137 secs 2023-03-09T17:11:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:11:35 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:11:36.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:11:36 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:11:36.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:11:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:11:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:11:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:11:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:11:36 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:11:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:11:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:11:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:11:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:11:36.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:11:36 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:11:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:11:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:11:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:11:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:11:37.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:11:37 smithi122 ceph-mon[93869]: pgmap v4784: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 35 KiB/s wr, 11 op/s 2023-03-09T17:11:37.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:11:37 smithi181 ceph-mon[94189]: pgmap v4784: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 35 KiB/s wr, 11 op/s 2023-03-09T17:11:37.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:11:37 smithi038 ceph-mon[92422]: pgmap v4784: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 35 KiB/s wr, 11 op/s 2023-03-09T17:11:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8470.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:11:39.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:11:39 smithi122 ceph-mon[93869]: pgmap v4785: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 35 KiB/s wr, 11 op/s 2023-03-09T17:11:39.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:11:39 smithi181 ceph-mon[94189]: pgmap v4785: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 35 KiB/s wr, 11 op/s 2023-03-09T17:11:39.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:11:39 smithi038 ceph-mon[92422]: pgmap v4785: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 35 KiB/s wr, 11 op/s 2023-03-09T17:11:40.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:11:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8011.045195 secs 2023-03-09T17:11:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:11:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8011.045195 secs 2023-03-09T17:11:40.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:11:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8011.045195 secs 2023-03-09T17:11:41.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:11:41 smithi122 ceph-mon[93869]: pgmap v4786: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 61 KiB/s wr, 10 op/s 2023-03-09T17:11:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:11:41 smithi181 ceph-mon[94189]: pgmap v4786: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 61 KiB/s wr, 10 op/s 2023-03-09T17:11:41.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:11:41 smithi038 ceph-mon[92422]: pgmap v4786: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 61 KiB/s wr, 10 op/s 2023-03-09T17:11:43.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:11:43 smithi122 ceph-mon[93869]: pgmap v4787: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 52 KiB/s wr, 13 op/s 2023-03-09T17:11:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:11:43 smithi181 ceph-mon[94189]: pgmap v4787: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 52 KiB/s wr, 13 op/s 2023-03-09T17:11:43.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:11:43 smithi038 ceph-mon[92422]: pgmap v4787: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 52 KiB/s wr, 13 op/s 2023-03-09T17:11:43.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8475.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:11:45.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:11:45 smithi122 ceph-mon[93869]: pgmap v4788: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 60 KiB/s wr, 10 op/s 2023-03-09T17:11:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:11:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8016.045254 secs 2023-03-09T17:11:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:11:45 smithi181 ceph-mon[94189]: pgmap v4788: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 60 KiB/s wr, 10 op/s 2023-03-09T17:11:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:11:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8016.045254 secs 2023-03-09T17:11:45.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:11:45 smithi038 ceph-mon[92422]: pgmap v4788: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 60 KiB/s wr, 10 op/s 2023-03-09T17:11:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:11:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8016.045254 secs 2023-03-09T17:11:47.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:11:47 smithi122 ceph-mon[93869]: pgmap v4789: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 51 KiB/s wr, 12 op/s 2023-03-09T17:11:47.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:11:47 smithi181 ceph-mon[94189]: pgmap v4789: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 51 KiB/s wr, 12 op/s 2023-03-09T17:11:47.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:11:47 smithi038 ceph-mon[92422]: pgmap v4789: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 51 KiB/s wr, 12 op/s 2023-03-09T17:11:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8480.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:11:49.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:11:49 smithi122 ceph-mon[93869]: pgmap v4790: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 66 KiB/s wr, 11 op/s 2023-03-09T17:11:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:11:49 smithi181 ceph-mon[94189]: pgmap v4790: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 66 KiB/s wr, 11 op/s 2023-03-09T17:11:49.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:11:49 smithi038 ceph-mon[92422]: pgmap v4790: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 66 KiB/s wr, 11 op/s 2023-03-09T17:11:50.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:11:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8021.045313 secs 2023-03-09T17:11:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:11:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8021.045313 secs 2023-03-09T17:11:50.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:11:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8021.045313 secs 2023-03-09T17:11:51.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:11:51 smithi122 ceph-mon[93869]: pgmap v4791: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 111 KiB/s wr, 10 op/s 2023-03-09T17:11:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:11:51 smithi181 ceph-mon[94189]: pgmap v4791: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 111 KiB/s wr, 10 op/s 2023-03-09T17:11:51.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:11:51 smithi038 ceph-mon[92422]: pgmap v4791: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 111 KiB/s wr, 10 op/s 2023-03-09T17:11:52.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:11:52 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 8023.304721 secs 2023-03-09T17:11:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:11:52 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 8023.304721 secs 2023-03-09T17:11:52.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:11:52 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 8023.304721 secs 2023-03-09T17:11:53.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:11:53 smithi122 ceph-mon[93869]: pgmap v4792: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 85 KiB/s wr, 13 op/s 2023-03-09T17:11:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:11:53 smithi181 ceph-mon[94189]: pgmap v4792: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 85 KiB/s wr, 13 op/s 2023-03-09T17:11:53.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:11:53 smithi038 ceph-mon[92422]: pgmap v4792: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 85 KiB/s wr, 13 op/s 2023-03-09T17:11:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8485.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:11:55.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:11:55 smithi122 ceph-mon[93869]: pgmap v4793: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 83 KiB/s wr, 11 op/s 2023-03-09T17:11:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:11:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8026.045376 secs 2023-03-09T17:11:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:11:55 smithi181 ceph-mon[94189]: pgmap v4793: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 83 KiB/s wr, 11 op/s 2023-03-09T17:11:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:11:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8026.045376 secs 2023-03-09T17:11:55.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:11:55 smithi038 ceph-mon[92422]: pgmap v4793: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 83 KiB/s wr, 11 op/s 2023-03-09T17:11:55.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:11:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8026.045376 secs 2023-03-09T17:11:56.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:11:56 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:11:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:11:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:11:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:11:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:11:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:11:56 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:11:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:11:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:11:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:11:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:11:56.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:11:56 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:11:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:11:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:11:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:11:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:11:57.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:11:57 smithi122 ceph-mon[93869]: pgmap v4794: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 67 KiB/s wr, 12 op/s 2023-03-09T17:11:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:11:57 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 8028.304847 secs 2023-03-09T17:11:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:11:57 smithi181 ceph-mon[94189]: pgmap v4794: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 67 KiB/s wr, 12 op/s 2023-03-09T17:11:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:11:57 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 8028.304847 secs 2023-03-09T17:11:57.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:11:57 smithi038 ceph-mon[92422]: pgmap v4794: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 67 KiB/s wr, 12 op/s 2023-03-09T17:11:57.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:11:57 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 8028.304847 secs 2023-03-09T17:11:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8490.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:11:59.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:11:59 smithi122 ceph-mon[93869]: pgmap v4795: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 74 KiB/s wr, 10 op/s 2023-03-09T17:11:59.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:11:59 smithi181 ceph-mon[94189]: pgmap v4795: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 74 KiB/s wr, 10 op/s 2023-03-09T17:11:59.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:11:59 smithi038 ceph-mon[92422]: pgmap v4795: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 74 KiB/s wr, 10 op/s 2023-03-09T17:12:00.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:12:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8031.045453 secs 2023-03-09T17:12:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:12:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8031.045453 secs 2023-03-09T17:12:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:12:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8031.045453 secs 2023-03-09T17:12:01.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:12:01 smithi122 ceph-mon[93869]: pgmap v4796: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 83 KiB/s wr, 10 op/s 2023-03-09T17:12:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:12:01 smithi181 ceph-mon[94189]: pgmap v4796: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 83 KiB/s wr, 10 op/s 2023-03-09T17:12:01.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:12:01 smithi038 ceph-mon[92422]: pgmap v4796: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 83 KiB/s wr, 10 op/s 2023-03-09T17:12:02.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:12:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8033.304964 secs 2023-03-09T17:12:02.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:12:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8033.304964 secs 2023-03-09T17:12:02.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:12:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8033.304964 secs 2023-03-09T17:12:03.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:12:03 smithi038 ceph-mon[92422]: pgmap v4797: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 53 KiB/s wr, 11 op/s 2023-03-09T17:12:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8495.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:12:03.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:12:03 smithi122 ceph-mon[93869]: pgmap v4797: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 53 KiB/s wr, 11 op/s 2023-03-09T17:12:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:12:03 smithi181 ceph-mon[94189]: pgmap v4797: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 53 KiB/s wr, 11 op/s 2023-03-09T17:12:04.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:12:04 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:12:04.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:12:04 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:12:04.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:12:04 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:12:04.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:12:04 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:12:04.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:12:04 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:12:04.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:12:04 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:12:05.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:12:05 smithi038 ceph-mon[92422]: pgmap v4798: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 115 KiB/s wr, 22 op/s 2023-03-09T17:12:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:12:05 smithi038 ceph-mon[92422]: osdmap e513: 12 total, 12 up, 12 in 2023-03-09T17:12:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:12:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8036.045513 secs 2023-03-09T17:12:05.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:12:05 smithi122 ceph-mon[93869]: pgmap v4798: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 115 KiB/s wr, 22 op/s 2023-03-09T17:12:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:12:05 smithi122 ceph-mon[93869]: osdmap e513: 12 total, 12 up, 12 in 2023-03-09T17:12:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:12:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8036.045513 secs 2023-03-09T17:12:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:12:05 smithi181 ceph-mon[94189]: pgmap v4798: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 115 KiB/s wr, 22 op/s 2023-03-09T17:12:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:12:05 smithi181 ceph-mon[94189]: osdmap e513: 12 total, 12 up, 12 in 2023-03-09T17:12:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:12:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8036.045513 secs 2023-03-09T17:12:07.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:12:07 smithi122 ceph-mon[93869]: pgmap v4800: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 158 KiB/s wr, 28 op/s 2023-03-09T17:12:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:12:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8038.305097 secs 2023-03-09T17:12:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:12:07 smithi181 ceph-mon[94189]: pgmap v4800: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 158 KiB/s wr, 28 op/s 2023-03-09T17:12:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:12:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8038.305097 secs 2023-03-09T17:12:07.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:12:07 smithi038 ceph-mon[92422]: pgmap v4800: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 158 KiB/s wr, 28 op/s 2023-03-09T17:12:07.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:12:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8038.305097 secs 2023-03-09T17:12:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8500.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:12:09.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:12:09 smithi122 ceph-mon[93869]: pgmap v4801: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 150 KiB/s wr, 28 op/s 2023-03-09T17:12:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:12:09 smithi181 ceph-mon[94189]: pgmap v4801: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 150 KiB/s wr, 28 op/s 2023-03-09T17:12:09.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:12:09 smithi038 ceph-mon[92422]: pgmap v4801: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 150 KiB/s wr, 28 op/s 2023-03-09T17:12:10.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:12:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8041.045573 secs 2023-03-09T17:12:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:12:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8041.045573 secs 2023-03-09T17:12:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:12:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8041.045573 secs 2023-03-09T17:12:11.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:12:11 smithi122 ceph-mon[93869]: pgmap v4802: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 152 KiB/s wr, 28 op/s 2023-03-09T17:12:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:12:11 smithi181 ceph-mon[94189]: pgmap v4802: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 152 KiB/s wr, 28 op/s 2023-03-09T17:12:11.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:12:11 smithi038 ceph-mon[92422]: pgmap v4802: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 152 KiB/s wr, 28 op/s 2023-03-09T17:12:12.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:12:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8043.305215 secs 2023-03-09T17:12:12.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:12:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8043.305215 secs 2023-03-09T17:12:12.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:12:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8043.305215 secs 2023-03-09T17:12:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8505.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:12:13.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:12:13 smithi122 ceph-mon[93869]: pgmap v4803: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 136 KiB/s wr, 27 op/s 2023-03-09T17:12:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:12:13 smithi181 ceph-mon[94189]: pgmap v4803: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 136 KiB/s wr, 27 op/s 2023-03-09T17:12:13.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:12:13 smithi038 ceph-mon[92422]: pgmap v4803: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 136 KiB/s wr, 27 op/s 2023-03-09T17:12:15.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:12:15 smithi122 ceph-mon[93869]: pgmap v4804: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 61 KiB/s wr, 11 op/s 2023-03-09T17:12:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:12:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8046.045658 secs 2023-03-09T17:12:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:12:15 smithi122 ceph-mon[93869]: osdmap e514: 12 total, 12 up, 12 in 2023-03-09T17:12:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:12:15 smithi181 ceph-mon[94189]: pgmap v4804: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 61 KiB/s wr, 11 op/s 2023-03-09T17:12:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:12:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8046.045658 secs 2023-03-09T17:12:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:12:15 smithi181 ceph-mon[94189]: osdmap e514: 12 total, 12 up, 12 in 2023-03-09T17:12:15.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:12:15 smithi038 ceph-mon[92422]: pgmap v4804: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 61 KiB/s wr, 11 op/s 2023-03-09T17:12:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:12:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8046.045658 secs 2023-03-09T17:12:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:12:15 smithi038 ceph-mon[92422]: osdmap e514: 12 total, 12 up, 12 in 2023-03-09T17:12:17.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:12:17 smithi122 ceph-mon[93869]: pgmap v4806: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 34 KiB/s wr, 8 op/s 2023-03-09T17:12:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:12:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8048.305315 secs 2023-03-09T17:12:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:12:17 smithi181 ceph-mon[94189]: pgmap v4806: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 34 KiB/s wr, 8 op/s 2023-03-09T17:12:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:12:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8048.305315 secs 2023-03-09T17:12:17.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:12:17 smithi038 ceph-mon[92422]: pgmap v4806: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 34 KiB/s wr, 8 op/s 2023-03-09T17:12:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:12:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8048.305315 secs 2023-03-09T17:12:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8510.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:12:19.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:12:19 smithi122 ceph-mon[93869]: pgmap v4807: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 33 KiB/s wr, 8 op/s 2023-03-09T17:12:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:12:19 smithi181 ceph-mon[94189]: pgmap v4807: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 33 KiB/s wr, 8 op/s 2023-03-09T17:12:19.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:12:19 smithi038 ceph-mon[92422]: pgmap v4807: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 33 KiB/s wr, 8 op/s 2023-03-09T17:12:20.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:12:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8051.045720 secs 2023-03-09T17:12:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:12:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:12:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:12:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8051.045720 secs 2023-03-09T17:12:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:12:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:12:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:12:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8051.045720 secs 2023-03-09T17:12:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:12:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:12:21.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:12:21 smithi122 ceph-mon[93869]: pgmap v4808: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T17:12:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:12:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:12:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:12:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:12:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:12:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:12:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:12:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:12:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:12:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:12:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:12:21 smithi181 ceph-mon[94189]: pgmap v4808: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T17:12:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:12:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:12:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:12:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:12:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:12:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:12:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:12:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:12:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:12:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:12:21.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:12:21 smithi038 ceph-mon[92422]: pgmap v4808: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T17:12:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:12:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:12:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:12:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:12:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:12:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:12:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:12:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:12:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:12:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:12:22.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:12:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8053.305460 secs 2023-03-09T17:12:22.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:12:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8053.305460 secs 2023-03-09T17:12:22.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:12:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8053.305460 secs 2023-03-09T17:12:23.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8515.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:12:23.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:12:23 smithi122 ceph-mon[93869]: pgmap v4809: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T17:12:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:12:23 smithi181 ceph-mon[94189]: pgmap v4809: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T17:12:23.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:12:23 smithi038 ceph-mon[92422]: pgmap v4809: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T17:12:25.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:12:25 smithi122 ceph-mon[93869]: pgmap v4810: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 35 KiB/s wr, 7 op/s 2023-03-09T17:12:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:12:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8056.045789 secs 2023-03-09T17:12:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:12:25 smithi181 ceph-mon[94189]: pgmap v4810: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 35 KiB/s wr, 7 op/s 2023-03-09T17:12:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:12:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8056.045789 secs 2023-03-09T17:12:25.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:12:25 smithi038 ceph-mon[92422]: pgmap v4810: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 35 KiB/s wr, 7 op/s 2023-03-09T17:12:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:12:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8056.045789 secs 2023-03-09T17:12:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:12:27 smithi122 ceph-mon[93869]: pgmap v4811: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 33 KiB/s wr, 9 op/s 2023-03-09T17:12:27.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:12:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8058.305605 secs 2023-03-09T17:12:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:12:27 smithi181 ceph-mon[94189]: pgmap v4811: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 33 KiB/s wr, 9 op/s 2023-03-09T17:12:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:12:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8058.305605 secs 2023-03-09T17:12:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:12:27 smithi038 ceph-mon[92422]: pgmap v4811: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 33 KiB/s wr, 9 op/s 2023-03-09T17:12:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:12:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8058.305605 secs 2023-03-09T17:12:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8520.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:12:29.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:12:29 smithi122 ceph-mon[93869]: pgmap v4812: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 29 KiB/s wr, 7 op/s 2023-03-09T17:12:29.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:12:29 smithi181 ceph-mon[94189]: pgmap v4812: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 29 KiB/s wr, 7 op/s 2023-03-09T17:12:29.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:12:29 smithi038 ceph-mon[92422]: pgmap v4812: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 29 KiB/s wr, 7 op/s 2023-03-09T17:12:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:12:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8061.045850 secs 2023-03-09T17:12:31.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:12:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8061.045850 secs 2023-03-09T17:12:31.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:12:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8061.045850 secs 2023-03-09T17:12:31.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:12:31 smithi038 ceph-mon[92422]: pgmap v4813: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 36 KiB/s wr, 7 op/s 2023-03-09T17:12:32.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:12:31 smithi122 ceph-mon[93869]: pgmap v4813: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 36 KiB/s wr, 7 op/s 2023-03-09T17:12:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:12:31 smithi181 ceph-mon[94189]: pgmap v4813: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 36 KiB/s wr, 7 op/s 2023-03-09T17:12:32.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:12:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8063.305751 secs 2023-03-09T17:12:33.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:12:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8063.305751 secs 2023-03-09T17:12:33.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:12:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8063.305751 secs 2023-03-09T17:12:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8525.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:12:34.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:12:33 smithi122 ceph-mon[93869]: pgmap v4814: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 24 KiB/s wr, 9 op/s 2023-03-09T17:12:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:12:33 smithi181 ceph-mon[94189]: pgmap v4814: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 24 KiB/s wr, 9 op/s 2023-03-09T17:12:34.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:12:33 smithi038 ceph-mon[92422]: pgmap v4814: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 24 KiB/s wr, 9 op/s 2023-03-09T17:12:36.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:12:35 smithi122 ceph-mon[93869]: pgmap v4815: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 38 KiB/s wr, 7 op/s 2023-03-09T17:12:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:12:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8066.045912 secs 2023-03-09T17:12:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:12:35 smithi181 ceph-mon[94189]: pgmap v4815: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 38 KiB/s wr, 7 op/s 2023-03-09T17:12:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:12:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8066.045912 secs 2023-03-09T17:12:36.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:12:35 smithi038 ceph-mon[92422]: pgmap v4815: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 38 KiB/s wr, 7 op/s 2023-03-09T17:12:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:12:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8066.045912 secs 2023-03-09T17:12:38.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:12:37 smithi122 ceph-mon[93869]: pgmap v4816: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 21 KiB/s wr, 8 op/s 2023-03-09T17:12:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:12:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8068.305895 secs 2023-03-09T17:12:38.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:12:37 smithi181 ceph-mon[94189]: pgmap v4816: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 21 KiB/s wr, 8 op/s 2023-03-09T17:12:38.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:12:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8068.305895 secs 2023-03-09T17:12:38.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:12:37 smithi038 ceph-mon[92422]: pgmap v4816: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 21 KiB/s wr, 8 op/s 2023-03-09T17:12:38.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:12:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8068.305895 secs 2023-03-09T17:12:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8530.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:12:40.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:12:39 smithi122 ceph-mon[93869]: pgmap v4817: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 21 KiB/s wr, 7 op/s 2023-03-09T17:12:40.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:12:39 smithi181 ceph-mon[94189]: pgmap v4817: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 21 KiB/s wr, 7 op/s 2023-03-09T17:12:40.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:12:39 smithi038 ceph-mon[92422]: pgmap v4817: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 21 KiB/s wr, 7 op/s 2023-03-09T17:12:41.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:12:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8071.045974 secs 2023-03-09T17:12:41.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:12:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8071.045974 secs 2023-03-09T17:12:41.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:12:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8071.045974 secs 2023-03-09T17:12:42.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:12:41 smithi122 ceph-mon[93869]: pgmap v4818: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 48 KiB/s wr, 6 op/s 2023-03-09T17:12:42.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:12:41 smithi181 ceph-mon[94189]: pgmap v4818: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 48 KiB/s wr, 6 op/s 2023-03-09T17:12:42.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:12:41 smithi038 ceph-mon[92422]: pgmap v4818: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 48 KiB/s wr, 6 op/s 2023-03-09T17:12:43.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:12:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8073.306046 secs 2023-03-09T17:12:43.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:12:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8073.306046 secs 2023-03-09T17:12:43.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:12:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8073.306046 secs 2023-03-09T17:12:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8535.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:12:44.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:12:43 smithi122 ceph-mon[93869]: pgmap v4819: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 49 KiB/s wr, 8 op/s 2023-03-09T17:12:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:12:43 smithi181 ceph-mon[94189]: pgmap v4819: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 49 KiB/s wr, 8 op/s 2023-03-09T17:12:44.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:12:43 smithi038 ceph-mon[92422]: pgmap v4819: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 49 KiB/s wr, 8 op/s 2023-03-09T17:12:46.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:12:45 smithi122 ceph-mon[93869]: pgmap v4820: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 53 KiB/s wr, 7 op/s 2023-03-09T17:12:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:12:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8076.045984 secs 2023-03-09T17:12:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:12:45 smithi181 ceph-mon[94189]: pgmap v4820: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 53 KiB/s wr, 7 op/s 2023-03-09T17:12:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:12:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8076.045984 secs 2023-03-09T17:12:46.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:12:45 smithi038 ceph-mon[92422]: pgmap v4820: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 53 KiB/s wr, 7 op/s 2023-03-09T17:12:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:12:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8076.045984 secs 2023-03-09T17:12:48.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:12:47 smithi122 ceph-mon[93869]: pgmap v4821: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 39 KiB/s wr, 9 op/s 2023-03-09T17:12:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:12:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8078.306185 secs 2023-03-09T17:12:48.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:12:47 smithi181 ceph-mon[94189]: pgmap v4821: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 39 KiB/s wr, 9 op/s 2023-03-09T17:12:48.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:12:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8078.306185 secs 2023-03-09T17:12:48.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:12:47 smithi038 ceph-mon[92422]: pgmap v4821: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 39 KiB/s wr, 9 op/s 2023-03-09T17:12:48.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:12:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8078.306185 secs 2023-03-09T17:12:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8540.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:12:50.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:12:49 smithi122 ceph-mon[93869]: pgmap v4822: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 47 KiB/s wr, 8 op/s 2023-03-09T17:12:50.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:12:49 smithi181 ceph-mon[94189]: pgmap v4822: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 47 KiB/s wr, 8 op/s 2023-03-09T17:12:50.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:12:49 smithi038 ceph-mon[92422]: pgmap v4822: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 47 KiB/s wr, 8 op/s 2023-03-09T17:12:51.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:12:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8081.046044 secs 2023-03-09T17:12:51.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:12:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8081.046044 secs 2023-03-09T17:12:51.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:12:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8081.046044 secs 2023-03-09T17:12:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:12:51 smithi122 ceph-mon[93869]: pgmap v4823: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 102 KiB/s wr, 9 op/s 2023-03-09T17:12:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:12:51 smithi181 ceph-mon[94189]: pgmap v4823: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 102 KiB/s wr, 9 op/s 2023-03-09T17:12:52.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:12:51 smithi038 ceph-mon[92422]: pgmap v4823: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 102 KiB/s wr, 9 op/s 2023-03-09T17:12:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:12:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8083.306288 secs 2023-03-09T17:12:53.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:12:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8083.306288 secs 2023-03-09T17:12:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:12:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8083.306288 secs 2023-03-09T17:12:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8545.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:12:54.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:12:53 smithi122 ceph-mon[93869]: pgmap v4824: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 83 KiB/s wr, 11 op/s 2023-03-09T17:12:54.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:12:53 smithi181 ceph-mon[94189]: pgmap v4824: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 83 KiB/s wr, 11 op/s 2023-03-09T17:12:54.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:12:53 smithi038 ceph-mon[92422]: pgmap v4824: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 83 KiB/s wr, 11 op/s 2023-03-09T17:12:56.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:12:55 smithi122 ceph-mon[93869]: pgmap v4825: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 76 KiB/s wr, 10 op/s 2023-03-09T17:12:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:12:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8086.046104 secs 2023-03-09T17:12:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:12:55 smithi181 ceph-mon[94189]: pgmap v4825: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 76 KiB/s wr, 10 op/s 2023-03-09T17:12:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:12:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8086.046104 secs 2023-03-09T17:12:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:12:55 smithi038 ceph-mon[92422]: pgmap v4825: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 76 KiB/s wr, 10 op/s 2023-03-09T17:12:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:12:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8086.046104 secs 2023-03-09T17:12:58.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:12:57 smithi122 ceph-mon[93869]: pgmap v4826: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 74 KiB/s wr, 11 op/s 2023-03-09T17:12:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:12:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8088.306388 secs 2023-03-09T17:12:58.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:12:57 smithi181 ceph-mon[94189]: pgmap v4826: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 74 KiB/s wr, 11 op/s 2023-03-09T17:12:58.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:12:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8088.306388 secs 2023-03-09T17:12:58.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:12:57 smithi038 ceph-mon[92422]: pgmap v4826: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 74 KiB/s wr, 11 op/s 2023-03-09T17:12:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:12:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8088.306388 secs 2023-03-09T17:12:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8550.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:13:00.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:12:59 smithi122 ceph-mon[93869]: pgmap v4827: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 81 KiB/s wr, 10 op/s 2023-03-09T17:13:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:12:59 smithi181 ceph-mon[94189]: pgmap v4827: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 81 KiB/s wr, 10 op/s 2023-03-09T17:13:00.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:12:59 smithi038 ceph-mon[92422]: pgmap v4827: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 81 KiB/s wr, 10 op/s 2023-03-09T17:13:01.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:13:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8091.046165 secs 2023-03-09T17:13:01.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:13:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8091.046165 secs 2023-03-09T17:13:01.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:13:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8091.046165 secs 2023-03-09T17:13:02.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:13:01 smithi122 ceph-mon[93869]: pgmap v4828: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 74 KiB/s wr, 9 op/s 2023-03-09T17:13:02.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:13:01 smithi181 ceph-mon[94189]: pgmap v4828: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 74 KiB/s wr, 9 op/s 2023-03-09T17:13:02.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:13:01 smithi038 ceph-mon[92422]: pgmap v4828: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 74 KiB/s wr, 9 op/s 2023-03-09T17:13:03.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:13:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8093.306515 secs 2023-03-09T17:13:03.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:13:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8093.306515 secs 2023-03-09T17:13:03.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:13:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8093.306515 secs 2023-03-09T17:13:03.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8555.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:13:04.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:13:03 smithi122 ceph-mon[93869]: pgmap v4829: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 114 KiB/s wr, 15 op/s 2023-03-09T17:13:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:13:03 smithi181 ceph-mon[94189]: pgmap v4829: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 114 KiB/s wr, 15 op/s 2023-03-09T17:13:04.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:13:03 smithi038 ceph-mon[92422]: pgmap v4829: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 114 KiB/s wr, 15 op/s 2023-03-09T17:13:06.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:13:05 smithi122 ceph-mon[93869]: pgmap v4830: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 127 KiB/s wr, 24 op/s 2023-03-09T17:13:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:13:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8096.046227 secs 2023-03-09T17:13:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:13:05 smithi181 ceph-mon[94189]: pgmap v4830: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 127 KiB/s wr, 24 op/s 2023-03-09T17:13:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:13:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8096.046227 secs 2023-03-09T17:13:06.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:13:05 smithi038 ceph-mon[92422]: pgmap v4830: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 127 KiB/s wr, 24 op/s 2023-03-09T17:13:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:13:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8096.046227 secs 2023-03-09T17:13:08.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:13:07 smithi122 ceph-mon[93869]: pgmap v4831: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 139 KiB/s wr, 26 op/s 2023-03-09T17:13:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:13:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8098.306604 secs 2023-03-09T17:13:08.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:13:07 smithi181 ceph-mon[94189]: pgmap v4831: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 139 KiB/s wr, 26 op/s 2023-03-09T17:13:08.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:13:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8098.306604 secs 2023-03-09T17:13:08.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:13:07 smithi038 ceph-mon[92422]: pgmap v4831: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 139 KiB/s wr, 26 op/s 2023-03-09T17:13:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:13:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8098.306604 secs 2023-03-09T17:13:08.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8560.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:13:10.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:13:09 smithi122 ceph-mon[93869]: pgmap v4832: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 136 KiB/s wr, 25 op/s 2023-03-09T17:13:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:13:09 smithi122 ceph-mon[93869]: osdmap e515: 12 total, 12 up, 12 in 2023-03-09T17:13:10.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:13:09 smithi181 ceph-mon[94189]: pgmap v4832: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 136 KiB/s wr, 25 op/s 2023-03-09T17:13:10.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:13:09 smithi181 ceph-mon[94189]: osdmap e515: 12 total, 12 up, 12 in 2023-03-09T17:13:10.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:13:09 smithi038 ceph-mon[92422]: pgmap v4832: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 136 KiB/s wr, 25 op/s 2023-03-09T17:13:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:13:09 smithi038 ceph-mon[92422]: osdmap e515: 12 total, 12 up, 12 in 2023-03-09T17:13:11.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:13:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8101.046301 secs 2023-03-09T17:13:11.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:13:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8101.046301 secs 2023-03-09T17:13:11.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:13:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8101.046301 secs 2023-03-09T17:13:12.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:13:11 smithi122 ceph-mon[93869]: pgmap v4834: 129 pgs: 9 active+clean+snaptrim_wait, 120 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 154 KiB/s wr, 28 op/s 2023-03-09T17:13:12.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:13:11 smithi181 ceph-mon[94189]: pgmap v4834: 129 pgs: 9 active+clean+snaptrim_wait, 120 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 154 KiB/s wr, 28 op/s 2023-03-09T17:13:12.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:13:11 smithi038 ceph-mon[92422]: pgmap v4834: 129 pgs: 9 active+clean+snaptrim_wait, 120 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 154 KiB/s wr, 28 op/s 2023-03-09T17:13:13.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:13:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8103.306714 secs 2023-03-09T17:13:13.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:13:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8103.306714 secs 2023-03-09T17:13:13.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:13:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8103.306714 secs 2023-03-09T17:13:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8565.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:13:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:13:13 smithi122 ceph-mon[93869]: pgmap v4835: 129 pgs: 9 active+clean+snaptrim_wait, 120 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 92 KiB/s wr, 23 op/s 2023-03-09T17:13:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:13:13 smithi181 ceph-mon[94189]: pgmap v4835: 129 pgs: 9 active+clean+snaptrim_wait, 120 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 92 KiB/s wr, 23 op/s 2023-03-09T17:13:14.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:13:13 smithi038 ceph-mon[92422]: pgmap v4835: 129 pgs: 9 active+clean+snaptrim_wait, 120 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 92 KiB/s wr, 23 op/s 2023-03-09T17:13:16.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:13:15 smithi122 ceph-mon[93869]: pgmap v4836: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 66 KiB/s wr, 8 op/s 2023-03-09T17:13:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:13:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8106.046360 secs 2023-03-09T17:13:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:13:15 smithi122 ceph-mon[93869]: osdmap e516: 12 total, 12 up, 12 in 2023-03-09T17:13:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:13:15 smithi181 ceph-mon[94189]: pgmap v4836: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 66 KiB/s wr, 8 op/s 2023-03-09T17:13:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:13:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8106.046360 secs 2023-03-09T17:13:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:13:15 smithi181 ceph-mon[94189]: osdmap e516: 12 total, 12 up, 12 in 2023-03-09T17:13:16.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:13:15 smithi038 ceph-mon[92422]: pgmap v4836: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 66 KiB/s wr, 8 op/s 2023-03-09T17:13:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:13:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8106.046360 secs 2023-03-09T17:13:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:13:15 smithi038 ceph-mon[92422]: osdmap e516: 12 total, 12 up, 12 in 2023-03-09T17:13:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:13:17 smithi122 ceph-mon[93869]: pgmap v4838: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 83 KiB/s wr, 9 op/s 2023-03-09T17:13:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:13:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8108.306821 secs 2023-03-09T17:13:18.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:13:17 smithi181 ceph-mon[94189]: pgmap v4838: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 83 KiB/s wr, 9 op/s 2023-03-09T17:13:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:13:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8108.306821 secs 2023-03-09T17:13:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:13:17 smithi038 ceph-mon[92422]: pgmap v4838: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 83 KiB/s wr, 9 op/s 2023-03-09T17:13:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:13:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8108.306821 secs 2023-03-09T17:13:18.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8570.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:13:20.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:13:19 smithi122 ceph-mon[93869]: pgmap v4839: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 73 KiB/s wr, 9 op/s 2023-03-09T17:13:20.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:13:19 smithi181 ceph-mon[94189]: pgmap v4839: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 73 KiB/s wr, 9 op/s 2023-03-09T17:13:20.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:13:19 smithi038 ceph-mon[92422]: pgmap v4839: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 73 KiB/s wr, 9 op/s 2023-03-09T17:13:21.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:13:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8111.046420 secs 2023-03-09T17:13:21.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:13:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8111.046420 secs 2023-03-09T17:13:21.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:13:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8111.046420 secs 2023-03-09T17:13:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:13:21 smithi122 ceph-mon[93869]: pgmap v4840: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 66 KiB/s wr, 8 op/s 2023-03-09T17:13:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:13:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:13:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:13:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:13:22.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:13:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:13:22.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:13:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:13:22.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:13:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:13:22.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:13:21 smithi181 ceph-mon[94189]: pgmap v4840: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 66 KiB/s wr, 8 op/s 2023-03-09T17:13:22.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:13:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:13:22.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:13:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:13:22.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:13:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:13:22.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:13:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:13:22.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:13:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:13:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:13:21 smithi038 ceph-mon[92422]: pgmap v4840: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 66 KiB/s wr, 8 op/s 2023-03-09T17:13:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:13:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:13:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:13:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:13:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:13:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:13:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:13:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:13:22.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:13:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:13:23.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:13:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8113.306904 secs 2023-03-09T17:13:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:13:22 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:13:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:13:22 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:13:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:13:22 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:13:23.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:13:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8113.306904 secs 2023-03-09T17:13:23.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:13:22 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:13:23.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:13:22 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:13:23.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:13:22 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:13:23.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:13:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8113.306904 secs 2023-03-09T17:13:23.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:13:22 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:13:23.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:13:22 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:13:23.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:13:22 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:13:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8575.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:13:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:13:23 smithi122 ceph-mon[93869]: pgmap v4841: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 16 KiB/s wr, 8 op/s 2023-03-09T17:13:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:13:23 smithi181 ceph-mon[94189]: pgmap v4841: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 16 KiB/s wr, 8 op/s 2023-03-09T17:13:24.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:13:23 smithi038 ceph-mon[92422]: pgmap v4841: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 16 KiB/s wr, 8 op/s 2023-03-09T17:13:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:13:25 smithi122 ceph-mon[93869]: pgmap v4842: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 16 KiB/s wr, 8 op/s 2023-03-09T17:13:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:13:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8116.046481 secs 2023-03-09T17:13:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:13:25 smithi181 ceph-mon[94189]: pgmap v4842: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 16 KiB/s wr, 8 op/s 2023-03-09T17:13:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:13:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8116.046481 secs 2023-03-09T17:13:26.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:13:25 smithi038 ceph-mon[92422]: pgmap v4842: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 16 KiB/s wr, 8 op/s 2023-03-09T17:13:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:13:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8116.046481 secs 2023-03-09T17:13:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:13:27 smithi122 ceph-mon[93869]: pgmap v4843: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 35 KiB/s wr, 10 op/s 2023-03-09T17:13:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:13:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8118.307021 secs 2023-03-09T17:13:28.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:13:27 smithi181 ceph-mon[94189]: pgmap v4843: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 35 KiB/s wr, 10 op/s 2023-03-09T17:13:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:13:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8118.307021 secs 2023-03-09T17:13:28.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:13:27 smithi038 ceph-mon[92422]: pgmap v4843: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 35 KiB/s wr, 10 op/s 2023-03-09T17:13:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:13:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8118.307021 secs 2023-03-09T17:13:28.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8580.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:13:30.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:13:29 smithi181 ceph-mon[94189]: pgmap v4844: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T17:13:30.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:13:29 smithi038 ceph-mon[92422]: pgmap v4844: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T17:13:30.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:13:29 smithi122 ceph-mon[93869]: pgmap v4844: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T17:13:31.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:13:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8121.046546 secs 2023-03-09T17:13:31.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:13:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8121.046546 secs 2023-03-09T17:13:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:13:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8121.046546 secs 2023-03-09T17:13:32.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:13:31 smithi038 ceph-mon[92422]: pgmap v4845: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 6 op/s 2023-03-09T17:13:32.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:13:31 smithi122 ceph-mon[93869]: pgmap v4845: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 6 op/s 2023-03-09T17:13:32.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:13:31 smithi181 ceph-mon[94189]: pgmap v4845: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 6 op/s 2023-03-09T17:13:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:13:32 smithi038 ceph-mon[92422]: pgmap v4846: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 9 op/s 2023-03-09T17:13:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:13:32 smithi122 ceph-mon[93869]: pgmap v4846: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 9 op/s 2023-03-09T17:13:33.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:13:32 smithi181 ceph-mon[94189]: pgmap v4846: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 9 op/s 2023-03-09T17:13:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8585.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:13:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:13:35 smithi122 ceph-mon[93869]: pgmap v4847: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 18 KiB/s wr, 6 op/s 2023-03-09T17:13:35.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:13:35 smithi181 ceph-mon[94189]: pgmap v4847: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 18 KiB/s wr, 6 op/s 2023-03-09T17:13:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:13:35 smithi038 ceph-mon[92422]: pgmap v4847: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 18 KiB/s wr, 6 op/s 2023-03-09T17:13:36.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:13:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8126.046608 secs 2023-03-09T17:13:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:13:36 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:13:36.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:13:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8126.046608 secs 2023-03-09T17:13:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:13:36 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:13:36.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:13:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8126.046608 secs 2023-03-09T17:13:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:13:36 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:13:37.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:13:37 smithi122 ceph-mon[93869]: pgmap v4848: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 52 KiB/s wr, 9 op/s 2023-03-09T17:13:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:13:37 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:13:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:13:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:13:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:13:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:13:37.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:13:37 smithi181 ceph-mon[94189]: pgmap v4848: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 52 KiB/s wr, 9 op/s 2023-03-09T17:13:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:13:37 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:13:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:13:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:13:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:13:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:13:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:13:37 smithi038 ceph-mon[92422]: pgmap v4848: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 52 KiB/s wr, 9 op/s 2023-03-09T17:13:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:13:37 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:13:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:13:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:13:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:13:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:13:38.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8590.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:13:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:13:39 smithi122 ceph-mon[93869]: pgmap v4849: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 43 KiB/s wr, 7 op/s 2023-03-09T17:13:39.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:13:39 smithi181 ceph-mon[94189]: pgmap v4849: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 43 KiB/s wr, 7 op/s 2023-03-09T17:13:39.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:13:39 smithi038 ceph-mon[92422]: pgmap v4849: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 43 KiB/s wr, 7 op/s 2023-03-09T17:13:41.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:13:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8131.046668 secs 2023-03-09T17:13:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:13:41 smithi122 ceph-mon[93869]: pgmap v4850: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 42 KiB/s wr, 6 op/s 2023-03-09T17:13:41.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:13:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8131.046668 secs 2023-03-09T17:13:41.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:13:41 smithi181 ceph-mon[94189]: pgmap v4850: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 42 KiB/s wr, 6 op/s 2023-03-09T17:13:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:13:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8131.046668 secs 2023-03-09T17:13:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:13:41 smithi038 ceph-mon[92422]: pgmap v4850: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 42 KiB/s wr, 6 op/s 2023-03-09T17:13:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:13:43 smithi122 ceph-mon[93869]: pgmap v4851: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 65 KiB/s wr, 11 op/s 2023-03-09T17:13:43.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:13:43 smithi181 ceph-mon[94189]: pgmap v4851: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 65 KiB/s wr, 11 op/s 2023-03-09T17:13:43.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:13:43 smithi038 ceph-mon[92422]: pgmap v4851: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 65 KiB/s wr, 11 op/s 2023-03-09T17:13:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8595.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:13:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:13:45 smithi122 ceph-mon[93869]: pgmap v4852: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 65 KiB/s wr, 8 op/s 2023-03-09T17:13:45.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:13:45 smithi181 ceph-mon[94189]: pgmap v4852: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 65 KiB/s wr, 8 op/s 2023-03-09T17:13:45.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:13:45 smithi038 ceph-mon[92422]: pgmap v4852: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 65 KiB/s wr, 8 op/s 2023-03-09T17:13:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:13:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8136.046729 secs 2023-03-09T17:13:46.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:13:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8136.046729 secs 2023-03-09T17:13:46.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:13:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8136.046729 secs 2023-03-09T17:13:47.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:13:47 smithi122 ceph-mon[93869]: pgmap v4853: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 107 KiB/s wr, 12 op/s 2023-03-09T17:13:47.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:13:47 smithi181 ceph-mon[94189]: pgmap v4853: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 107 KiB/s wr, 12 op/s 2023-03-09T17:13:47.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:13:47 smithi038 ceph-mon[92422]: pgmap v4853: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 107 KiB/s wr, 12 op/s 2023-03-09T17:13:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8600.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:13:49.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:13:49 smithi122 ceph-mon[93869]: pgmap v4854: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 81 KiB/s wr, 10 op/s 2023-03-09T17:13:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:13:49 smithi181 ceph-mon[94189]: pgmap v4854: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 81 KiB/s wr, 10 op/s 2023-03-09T17:13:49.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:13:49 smithi038 ceph-mon[92422]: pgmap v4854: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 81 KiB/s wr, 10 op/s 2023-03-09T17:13:50.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:13:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8141.046788 secs 2023-03-09T17:13:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:13:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8141.046788 secs 2023-03-09T17:13:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:13:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8141.046788 secs 2023-03-09T17:13:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:13:51 smithi122 ceph-mon[93869]: pgmap v4855: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 73 KiB/s wr, 9 op/s 2023-03-09T17:13:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:13:51 smithi181 ceph-mon[94189]: pgmap v4855: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 73 KiB/s wr, 9 op/s 2023-03-09T17:13:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:13:51 smithi038 ceph-mon[92422]: pgmap v4855: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 73 KiB/s wr, 9 op/s 2023-03-09T17:13:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:13:53 smithi122 ceph-mon[93869]: pgmap v4856: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 113 KiB/s wr, 13 op/s 2023-03-09T17:13:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:13:53 smithi181 ceph-mon[94189]: pgmap v4856: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 113 KiB/s wr, 13 op/s 2023-03-09T17:13:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:13:53 smithi038 ceph-mon[92422]: pgmap v4856: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 113 KiB/s wr, 13 op/s 2023-03-09T17:13:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8605.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:13:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:13:55 smithi122 ceph-mon[93869]: pgmap v4857: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 90 KiB/s wr, 9 op/s 2023-03-09T17:13:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:13:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8146.046850 secs 2023-03-09T17:13:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:13:55 smithi181 ceph-mon[94189]: pgmap v4857: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 90 KiB/s wr, 9 op/s 2023-03-09T17:13:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:13:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8146.046850 secs 2023-03-09T17:13:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:13:55 smithi038 ceph-mon[92422]: pgmap v4857: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 90 KiB/s wr, 9 op/s 2023-03-09T17:13:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:13:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8146.046850 secs 2023-03-09T17:13:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:13:57 smithi122 ceph-mon[93869]: pgmap v4858: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 126 KiB/s wr, 12 op/s 2023-03-09T17:13:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:13:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8148.307750 secs 2023-03-09T17:13:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:13:57 smithi181 ceph-mon[94189]: pgmap v4858: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 126 KiB/s wr, 12 op/s 2023-03-09T17:13:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:13:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8148.307750 secs 2023-03-09T17:13:57.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:13:57 smithi038 ceph-mon[92422]: pgmap v4858: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 126 KiB/s wr, 12 op/s 2023-03-09T17:13:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:13:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8148.307750 secs 2023-03-09T17:13:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8610.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:13:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:13:59 smithi122 ceph-mon[93869]: pgmap v4859: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 84 KiB/s wr, 9 op/s 2023-03-09T17:13:59.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:13:59 smithi181 ceph-mon[94189]: pgmap v4859: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 84 KiB/s wr, 9 op/s 2023-03-09T17:13:59.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:13:59 smithi038 ceph-mon[92422]: pgmap v4859: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 84 KiB/s wr, 9 op/s 2023-03-09T17:14:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:14:00 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:14:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:14:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:14:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:14:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:14:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:14:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8151.046912 secs 2023-03-09T17:14:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:14:00 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:14:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:14:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:14:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:14:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:14:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:14:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8151.046912 secs 2023-03-09T17:14:00.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:14:00 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:14:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:14:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:14:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:14:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:14:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:14:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8151.046912 secs 2023-03-09T17:14:01.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:14:01 smithi122 ceph-mon[93869]: pgmap v4860: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 76 KiB/s wr, 8 op/s 2023-03-09T17:14:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:14:01 smithi181 ceph-mon[94189]: pgmap v4860: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 76 KiB/s wr, 8 op/s 2023-03-09T17:14:01.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:14:01 smithi038 ceph-mon[92422]: pgmap v4860: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 76 KiB/s wr, 8 op/s 2023-03-09T17:14:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:14:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8153.307882 secs 2023-03-09T17:14:02.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:14:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8153.307882 secs 2023-03-09T17:14:02.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:14:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8153.307882 secs 2023-03-09T17:14:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:14:03 smithi122 ceph-mon[93869]: pgmap v4861: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 125 KiB/s wr, 13 op/s 2023-03-09T17:14:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:14:03 smithi181 ceph-mon[94189]: pgmap v4861: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 125 KiB/s wr, 13 op/s 2023-03-09T17:14:03.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:14:03 smithi038 ceph-mon[92422]: pgmap v4861: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 125 KiB/s wr, 13 op/s 2023-03-09T17:14:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8615.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:14:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:14:05 smithi122 ceph-mon[93869]: pgmap v4862: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 107 KiB/s wr, 21 op/s 2023-03-09T17:14:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:14:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8156.046975 secs 2023-03-09T17:14:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:14:05 smithi181 ceph-mon[94189]: pgmap v4862: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 107 KiB/s wr, 21 op/s 2023-03-09T17:14:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:14:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8156.046975 secs 2023-03-09T17:14:05.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:14:05 smithi038 ceph-mon[92422]: pgmap v4862: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 107 KiB/s wr, 21 op/s 2023-03-09T17:14:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:14:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8156.046975 secs 2023-03-09T17:14:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:14:07 smithi122 ceph-mon[93869]: pgmap v4863: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 107 KiB/s wr, 23 op/s 2023-03-09T17:14:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:14:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8158.307947 secs 2023-03-09T17:14:07.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:14:07 smithi181 ceph-mon[94189]: pgmap v4863: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 107 KiB/s wr, 23 op/s 2023-03-09T17:14:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:14:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8158.307947 secs 2023-03-09T17:14:07.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:14:07 smithi038 ceph-mon[92422]: pgmap v4863: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 107 KiB/s wr, 23 op/s 2023-03-09T17:14:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:14:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8158.307947 secs 2023-03-09T17:14:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8620.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:14:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:14:09 smithi122 ceph-mon[93869]: pgmap v4864: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 78 KiB/s wr, 21 op/s 2023-03-09T17:14:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:14:09 smithi181 ceph-mon[94189]: pgmap v4864: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 78 KiB/s wr, 21 op/s 2023-03-09T17:14:09.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:14:09 smithi038 ceph-mon[92422]: pgmap v4864: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 78 KiB/s wr, 21 op/s 2023-03-09T17:14:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:14:10 smithi122 ceph-mon[93869]: osdmap e517: 12 total, 12 up, 12 in 2023-03-09T17:14:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:14:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8161.047039 secs 2023-03-09T17:14:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:14:10 smithi181 ceph-mon[94189]: osdmap e517: 12 total, 12 up, 12 in 2023-03-09T17:14:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:14:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8161.047039 secs 2023-03-09T17:14:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:14:10 smithi038 ceph-mon[92422]: osdmap e517: 12 total, 12 up, 12 in 2023-03-09T17:14:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:14:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8161.047039 secs 2023-03-09T17:14:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:14:11 smithi122 ceph-mon[93869]: pgmap v4866: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 93 KiB/s wr, 24 op/s 2023-03-09T17:14:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:14:11 smithi181 ceph-mon[94189]: pgmap v4866: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 93 KiB/s wr, 24 op/s 2023-03-09T17:14:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:14:11 smithi038 ceph-mon[92422]: pgmap v4866: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 93 KiB/s wr, 24 op/s 2023-03-09T17:14:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:14:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8163.308088 secs 2023-03-09T17:14:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:14:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8163.308088 secs 2023-03-09T17:14:12.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:14:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8163.308088 secs 2023-03-09T17:14:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:14:13 smithi122 ceph-mon[93869]: pgmap v4867: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 59 KiB/s wr, 22 op/s 2023-03-09T17:14:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:14:13 smithi181 ceph-mon[94189]: pgmap v4867: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 59 KiB/s wr, 22 op/s 2023-03-09T17:14:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:14:13 smithi038 ceph-mon[92422]: pgmap v4867: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 59 KiB/s wr, 22 op/s 2023-03-09T17:14:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8625.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:14:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:14:15 smithi122 ceph-mon[93869]: pgmap v4868: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 33 KiB/s wr, 7 op/s 2023-03-09T17:14:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:14:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8166.047099 secs 2023-03-09T17:14:15.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:14:15 smithi181 ceph-mon[94189]: pgmap v4868: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 33 KiB/s wr, 7 op/s 2023-03-09T17:14:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:14:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8166.047099 secs 2023-03-09T17:14:15.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:14:15 smithi038 ceph-mon[92422]: pgmap v4868: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 33 KiB/s wr, 7 op/s 2023-03-09T17:14:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:14:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8166.047099 secs 2023-03-09T17:14:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:14:17 smithi122 ceph-mon[93869]: pgmap v4869: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 34 KiB/s wr, 8 op/s 2023-03-09T17:14:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:14:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8168.308200 secs 2023-03-09T17:14:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:14:17 smithi181 ceph-mon[94189]: pgmap v4869: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 34 KiB/s wr, 8 op/s 2023-03-09T17:14:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:14:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8168.308200 secs 2023-03-09T17:14:17.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:14:17 smithi038 ceph-mon[92422]: pgmap v4869: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 34 KiB/s wr, 8 op/s 2023-03-09T17:14:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:14:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8168.308200 secs 2023-03-09T17:14:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8630.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:14:19.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:14:19 smithi038 ceph-mon[92422]: pgmap v4870: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 26 KiB/s wr, 8 op/s 2023-03-09T17:14:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:14:19 smithi122 ceph-mon[93869]: pgmap v4870: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 26 KiB/s wr, 8 op/s 2023-03-09T17:14:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:14:19 smithi181 ceph-mon[94189]: pgmap v4870: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 26 KiB/s wr, 8 op/s 2023-03-09T17:14:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:14:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8171.047159 secs 2023-03-09T17:14:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:14:20 smithi038 ceph-mon[92422]: osdmap e518: 12 total, 12 up, 12 in 2023-03-09T17:14:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:14:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8171.047159 secs 2023-03-09T17:14:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:14:20 smithi122 ceph-mon[93869]: osdmap e518: 12 total, 12 up, 12 in 2023-03-09T17:14:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:14:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8171.047159 secs 2023-03-09T17:14:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:14:20 smithi181 ceph-mon[94189]: osdmap e518: 12 total, 12 up, 12 in 2023-03-09T17:14:21.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:14:21 smithi038 ceph-mon[92422]: pgmap v4871: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 25 KiB/s wr, 8 op/s 2023-03-09T17:14:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:14:21 smithi122 ceph-mon[93869]: pgmap v4871: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 25 KiB/s wr, 8 op/s 2023-03-09T17:14:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:14:21 smithi181 ceph-mon[94189]: pgmap v4871: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 25 KiB/s wr, 8 op/s 2023-03-09T17:14:22.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:14:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8173.308320 secs 2023-03-09T17:14:22.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:14:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8173.308320 secs 2023-03-09T17:14:22.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:14:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8173.308320 secs 2023-03-09T17:14:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8635.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:14:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:14:23 smithi122 ceph-mon[93869]: pgmap v4873: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 2.1 KiB/s wr, 7 op/s 2023-03-09T17:14:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:14:23 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:14:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:14:23 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:14:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:14:23 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:14:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:14:23 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:14:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:14:23 smithi181 ceph-mon[94189]: pgmap v4873: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 2.1 KiB/s wr, 7 op/s 2023-03-09T17:14:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:14:23 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:14:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:14:23 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:14:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:14:23 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:14:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:14:23 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:14:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:14:23 smithi038 ceph-mon[92422]: pgmap v4873: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 2.1 KiB/s wr, 7 op/s 2023-03-09T17:14:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:14:23 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:14:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:14:23 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:14:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:14:23 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:14:24.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:14:23 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:14:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:14:25 smithi122 ceph-mon[93869]: pgmap v4874: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 2.7 KiB/s wr, 8 op/s 2023-03-09T17:14:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:14:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8176.047220 secs 2023-03-09T17:14:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:14:25 smithi181 ceph-mon[94189]: pgmap v4874: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 2.7 KiB/s wr, 8 op/s 2023-03-09T17:14:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:14:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8176.047220 secs 2023-03-09T17:14:25.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:14:25 smithi038 ceph-mon[92422]: pgmap v4874: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 2.7 KiB/s wr, 8 op/s 2023-03-09T17:14:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:14:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8176.047220 secs 2023-03-09T17:14:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:14:27 smithi122 ceph-mon[93869]: pgmap v4875: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 23 KiB/s wr, 8 op/s 2023-03-09T17:14:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:14:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8178.308464 secs 2023-03-09T17:14:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:14:27 smithi181 ceph-mon[94189]: pgmap v4875: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 23 KiB/s wr, 8 op/s 2023-03-09T17:14:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:14:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8178.308464 secs 2023-03-09T17:14:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:14:27 smithi038 ceph-mon[92422]: pgmap v4875: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 23 KiB/s wr, 8 op/s 2023-03-09T17:14:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:14:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8178.308464 secs 2023-03-09T17:14:28.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8640.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:14:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:14:29 smithi122 ceph-mon[93869]: pgmap v4876: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 8 op/s 2023-03-09T17:14:29.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:14:29 smithi181 ceph-mon[94189]: pgmap v4876: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 8 op/s 2023-03-09T17:14:29.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:14:29 smithi038 ceph-mon[92422]: pgmap v4876: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 8 op/s 2023-03-09T17:14:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:14:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8181.047279 secs 2023-03-09T17:14:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:14:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8181.047279 secs 2023-03-09T17:14:30.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:14:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8181.047279 secs 2023-03-09T17:14:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:14:31 smithi122 ceph-mon[93869]: pgmap v4877: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 8 op/s 2023-03-09T17:14:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:14:31 smithi181 ceph-mon[94189]: pgmap v4877: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 8 op/s 2023-03-09T17:14:31.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:14:31 smithi038 ceph-mon[92422]: pgmap v4877: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 8 op/s 2023-03-09T17:14:32.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:14:32 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 8183.308583 secs 2023-03-09T17:14:32.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:14:32 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 8183.308583 secs 2023-03-09T17:14:32.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:14:32 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 8183.308583 secs 2023-03-09T17:14:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8645.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:14:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:14:33 smithi122 ceph-mon[93869]: pgmap v4878: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 10 op/s 2023-03-09T17:14:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:14:33 smithi181 ceph-mon[94189]: pgmap v4878: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 10 op/s 2023-03-09T17:14:33.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:14:33 smithi038 ceph-mon[92422]: pgmap v4878: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 10 op/s 2023-03-09T17:14:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:14:35 smithi122 ceph-mon[93869]: pgmap v4879: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 8 op/s 2023-03-09T17:14:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:14:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8186.047338 secs 2023-03-09T17:14:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:14:35 smithi181 ceph-mon[94189]: pgmap v4879: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 8 op/s 2023-03-09T17:14:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:14:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8186.047338 secs 2023-03-09T17:14:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:14:35 smithi038 ceph-mon[92422]: pgmap v4879: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 8 op/s 2023-03-09T17:14:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:14:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8186.047338 secs 2023-03-09T17:14:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:14:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:14:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:14:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:14:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:14:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:14:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:14:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:14:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:14:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:14:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:14:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:14:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:14:37 smithi122 ceph-mon[93869]: pgmap v4880: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 53 KiB/s wr, 9 op/s 2023-03-09T17:14:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:14:37 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 8188.308695 secs 2023-03-09T17:14:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:14:37 smithi181 ceph-mon[94189]: pgmap v4880: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 53 KiB/s wr, 9 op/s 2023-03-09T17:14:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:14:37 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 8188.308695 secs 2023-03-09T17:14:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:14:37 smithi038 ceph-mon[92422]: pgmap v4880: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 53 KiB/s wr, 9 op/s 2023-03-09T17:14:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:14:37 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 8188.308695 secs 2023-03-09T17:14:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8650.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:14:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:14:39 smithi122 ceph-mon[93869]: pgmap v4881: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 46 KiB/s wr, 8 op/s 2023-03-09T17:14:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:14:39 smithi181 ceph-mon[94189]: pgmap v4881: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 46 KiB/s wr, 8 op/s 2023-03-09T17:14:39.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:14:39 smithi038 ceph-mon[92422]: pgmap v4881: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 46 KiB/s wr, 8 op/s 2023-03-09T17:14:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:14:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8191.047400 secs 2023-03-09T17:14:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:14:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8191.047400 secs 2023-03-09T17:14:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:14:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8191.047400 secs 2023-03-09T17:14:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:14:41 smithi122 ceph-mon[93869]: pgmap v4882: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 45 KiB/s wr, 7 op/s 2023-03-09T17:14:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:14:41 smithi181 ceph-mon[94189]: pgmap v4882: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 45 KiB/s wr, 7 op/s 2023-03-09T17:14:41.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:14:41 smithi038 ceph-mon[92422]: pgmap v4882: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 45 KiB/s wr, 7 op/s 2023-03-09T17:14:42.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:14:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8193.308805 secs 2023-03-09T17:14:42.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:14:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8193.308805 secs 2023-03-09T17:14:42.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:14:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8193.308805 secs 2023-03-09T17:14:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8655.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:14:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:14:43 smithi122 ceph-mon[93869]: pgmap v4883: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 54 KiB/s wr, 11 op/s 2023-03-09T17:14:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:14:43 smithi181 ceph-mon[94189]: pgmap v4883: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 54 KiB/s wr, 11 op/s 2023-03-09T17:14:43.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:14:43 smithi038 ceph-mon[92422]: pgmap v4883: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 54 KiB/s wr, 11 op/s 2023-03-09T17:14:44.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:14:44 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:14:44.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:14:44 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:14:44.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:14:44 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:14:44.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:14:44 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:14:44.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:14:44 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:14:44.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:14:44 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:14:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:14:45 smithi122 ceph-mon[93869]: pgmap v4884: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 110 KiB/s wr, 10 op/s 2023-03-09T17:14:45.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:14:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8196.047459 secs 2023-03-09T17:14:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:14:45 smithi181 ceph-mon[94189]: pgmap v4884: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 110 KiB/s wr, 10 op/s 2023-03-09T17:14:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:14:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8196.047459 secs 2023-03-09T17:14:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:14:45 smithi038 ceph-mon[92422]: pgmap v4884: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 110 KiB/s wr, 10 op/s 2023-03-09T17:14:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:14:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8196.047459 secs 2023-03-09T17:14:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:14:47 smithi122 ceph-mon[93869]: pgmap v4885: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 76 KiB/s wr, 11 op/s 2023-03-09T17:14:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:14:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8198.308942 secs 2023-03-09T17:14:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:14:47 smithi181 ceph-mon[94189]: pgmap v4885: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 76 KiB/s wr, 11 op/s 2023-03-09T17:14:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:14:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8198.308942 secs 2023-03-09T17:14:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:14:47 smithi038 ceph-mon[92422]: pgmap v4885: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 76 KiB/s wr, 11 op/s 2023-03-09T17:14:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:14:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8198.308942 secs 2023-03-09T17:14:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8660.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:14:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:14:49 smithi122 ceph-mon[93869]: pgmap v4886: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 104 KiB/s wr, 11 op/s 2023-03-09T17:14:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:14:49 smithi181 ceph-mon[94189]: pgmap v4886: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 104 KiB/s wr, 11 op/s 2023-03-09T17:14:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:14:49 smithi038 ceph-mon[92422]: pgmap v4886: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 104 KiB/s wr, 11 op/s 2023-03-09T17:14:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:14:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8201.047518 secs 2023-03-09T17:14:50.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:14:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8201.047518 secs 2023-03-09T17:14:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:14:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8201.047518 secs 2023-03-09T17:14:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:14:51 smithi122 ceph-mon[93869]: pgmap v4887: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 93 KiB/s wr, 9 op/s 2023-03-09T17:14:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:14:51 smithi181 ceph-mon[94189]: pgmap v4887: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 93 KiB/s wr, 9 op/s 2023-03-09T17:14:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:14:51 smithi038 ceph-mon[92422]: pgmap v4887: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 93 KiB/s wr, 9 op/s 2023-03-09T17:14:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:14:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8203.309079 secs 2023-03-09T17:14:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:14:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8203.309079 secs 2023-03-09T17:14:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:14:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8203.309079 secs 2023-03-09T17:14:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8665.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:14:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:14:53 smithi122 ceph-mon[93869]: pgmap v4888: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 108 KiB/s wr, 12 op/s 2023-03-09T17:14:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:14:53 smithi181 ceph-mon[94189]: pgmap v4888: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 108 KiB/s wr, 12 op/s 2023-03-09T17:14:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:14:53 smithi038 ceph-mon[92422]: pgmap v4888: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 108 KiB/s wr, 12 op/s 2023-03-09T17:14:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:14:55 smithi122 ceph-mon[93869]: pgmap v4889: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 148 KiB/s wr, 10 op/s 2023-03-09T17:14:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:14:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8206.047580 secs 2023-03-09T17:14:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:14:55 smithi181 ceph-mon[94189]: pgmap v4889: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 148 KiB/s wr, 10 op/s 2023-03-09T17:14:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:14:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8206.047580 secs 2023-03-09T17:14:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:14:55 smithi038 ceph-mon[92422]: pgmap v4889: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 148 KiB/s wr, 10 op/s 2023-03-09T17:14:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:14:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8206.047580 secs 2023-03-09T17:14:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:14:57 smithi122 ceph-mon[93869]: pgmap v4890: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 91 KiB/s wr, 11 op/s 2023-03-09T17:14:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:14:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8208.309175 secs 2023-03-09T17:14:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:14:57 smithi181 ceph-mon[94189]: pgmap v4890: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 91 KiB/s wr, 11 op/s 2023-03-09T17:14:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:14:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8208.309175 secs 2023-03-09T17:14:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:14:57 smithi038 ceph-mon[92422]: pgmap v4890: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 91 KiB/s wr, 11 op/s 2023-03-09T17:14:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:14:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8208.309175 secs 2023-03-09T17:14:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8670.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:14:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:14:59 smithi122 ceph-mon[93869]: pgmap v4891: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 92 KiB/s wr, 10 op/s 2023-03-09T17:14:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:14:59 smithi181 ceph-mon[94189]: pgmap v4891: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 92 KiB/s wr, 10 op/s 2023-03-09T17:14:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:14:59 smithi038 ceph-mon[92422]: pgmap v4891: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 92 KiB/s wr, 10 op/s 2023-03-09T17:15:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:15:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8211.047664 secs 2023-03-09T17:15:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:15:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8211.047664 secs 2023-03-09T17:15:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:15:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8211.047664 secs 2023-03-09T17:15:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:15:01 smithi122 ceph-mon[93869]: pgmap v4892: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 64 KiB/s wr, 8 op/s 2023-03-09T17:15:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:15:01 smithi181 ceph-mon[94189]: pgmap v4892: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 64 KiB/s wr, 8 op/s 2023-03-09T17:15:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:15:01 smithi038 ceph-mon[92422]: pgmap v4892: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 64 KiB/s wr, 8 op/s 2023-03-09T17:15:02.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:15:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8213.309289 secs 2023-03-09T17:15:02.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:15:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8213.309289 secs 2023-03-09T17:15:02.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:15:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8213.309289 secs 2023-03-09T17:15:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8675.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:15:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:15:03 smithi122 ceph-mon[93869]: pgmap v4893: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 94 KiB/s wr, 13 op/s 2023-03-09T17:15:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:15:03 smithi181 ceph-mon[94189]: pgmap v4893: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 94 KiB/s wr, 13 op/s 2023-03-09T17:15:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:15:03 smithi038 ceph-mon[92422]: pgmap v4893: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 94 KiB/s wr, 13 op/s 2023-03-09T17:15:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:15:05 smithi122 ceph-mon[93869]: pgmap v4894: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 121 KiB/s wr, 23 op/s 2023-03-09T17:15:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:15:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8216.047752 secs 2023-03-09T17:15:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:15:05 smithi181 ceph-mon[94189]: pgmap v4894: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 121 KiB/s wr, 23 op/s 2023-03-09T17:15:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:15:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8216.047752 secs 2023-03-09T17:15:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:15:05 smithi038 ceph-mon[92422]: pgmap v4894: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 121 KiB/s wr, 23 op/s 2023-03-09T17:15:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:15:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8216.047752 secs 2023-03-09T17:15:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:15:07 smithi122 ceph-mon[93869]: pgmap v4895: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 73 KiB/s wr, 24 op/s 2023-03-09T17:15:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:15:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8218.309407 secs 2023-03-09T17:15:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:15:07 smithi181 ceph-mon[94189]: pgmap v4895: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 73 KiB/s wr, 24 op/s 2023-03-09T17:15:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:15:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8218.309407 secs 2023-03-09T17:15:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:15:07 smithi038 ceph-mon[92422]: pgmap v4895: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 73 KiB/s wr, 24 op/s 2023-03-09T17:15:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:15:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8218.309407 secs 2023-03-09T17:15:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8680.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:15:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:15:09 smithi122 ceph-mon[93869]: pgmap v4896: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 73 KiB/s wr, 23 op/s 2023-03-09T17:15:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:15:09 smithi181 ceph-mon[94189]: pgmap v4896: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 73 KiB/s wr, 23 op/s 2023-03-09T17:15:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:15:09 smithi038 ceph-mon[92422]: pgmap v4896: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 73 KiB/s wr, 23 op/s 2023-03-09T17:15:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:15:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8221.047812 secs 2023-03-09T17:15:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:15:10 smithi122 ceph-mon[93869]: osdmap e519: 12 total, 12 up, 12 in 2023-03-09T17:15:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:15:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8221.047812 secs 2023-03-09T17:15:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:15:10 smithi181 ceph-mon[94189]: osdmap e519: 12 total, 12 up, 12 in 2023-03-09T17:15:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:15:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8221.047812 secs 2023-03-09T17:15:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:15:10 smithi038 ceph-mon[92422]: osdmap e519: 12 total, 12 up, 12 in 2023-03-09T17:15:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:15:11 smithi122 ceph-mon[93869]: pgmap v4898: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 86 KiB/s wr, 25 op/s 2023-03-09T17:15:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:15:11 smithi181 ceph-mon[94189]: pgmap v4898: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 86 KiB/s wr, 25 op/s 2023-03-09T17:15:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:15:11 smithi038 ceph-mon[92422]: pgmap v4898: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 86 KiB/s wr, 25 op/s 2023-03-09T17:15:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:15:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8223.309519 secs 2023-03-09T17:15:12.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:15:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8223.309519 secs 2023-03-09T17:15:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:15:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8223.309519 secs 2023-03-09T17:15:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8685.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:15:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:15:13 smithi122 ceph-mon[93869]: pgmap v4899: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 51 KiB/s wr, 23 op/s 2023-03-09T17:15:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:15:13 smithi181 ceph-mon[94189]: pgmap v4899: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 51 KiB/s wr, 23 op/s 2023-03-09T17:15:13.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:15:13 smithi038 ceph-mon[92422]: pgmap v4899: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 51 KiB/s wr, 23 op/s 2023-03-09T17:15:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:15:15 smithi122 ceph-mon[93869]: pgmap v4900: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 15 KiB/s wr, 7 op/s 2023-03-09T17:15:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:15:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8226.047898 secs 2023-03-09T17:15:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:15:15 smithi181 ceph-mon[94189]: pgmap v4900: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 15 KiB/s wr, 7 op/s 2023-03-09T17:15:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:15:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8226.047898 secs 2023-03-09T17:15:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:15:15 smithi038 ceph-mon[92422]: pgmap v4900: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 15 KiB/s wr, 7 op/s 2023-03-09T17:15:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:15:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8226.047898 secs 2023-03-09T17:15:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:15:17 smithi181 ceph-mon[94189]: pgmap v4901: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 15 KiB/s wr, 7 op/s 2023-03-09T17:15:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:15:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8228.309633 secs 2023-03-09T17:15:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:15:17 smithi038 ceph-mon[92422]: pgmap v4901: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 15 KiB/s wr, 7 op/s 2023-03-09T17:15:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:15:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8228.309633 secs 2023-03-09T17:15:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:15:17 smithi122 ceph-mon[93869]: pgmap v4901: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 15 KiB/s wr, 7 op/s 2023-03-09T17:15:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:15:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8228.309633 secs 2023-03-09T17:15:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8690.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:15:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:15:19 smithi038 ceph-mon[92422]: pgmap v4902: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 37 KiB/s wr, 7 op/s 2023-03-09T17:15:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:15:19 smithi122 ceph-mon[93869]: pgmap v4902: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 37 KiB/s wr, 7 op/s 2023-03-09T17:15:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:15:19 smithi181 ceph-mon[94189]: pgmap v4902: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 37 KiB/s wr, 7 op/s 2023-03-09T17:15:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:15:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8231.047985 secs 2023-03-09T17:15:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:15:20 smithi038 ceph-mon[92422]: osdmap e520: 12 total, 12 up, 12 in 2023-03-09T17:15:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:15:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8231.047985 secs 2023-03-09T17:15:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:15:20 smithi122 ceph-mon[93869]: osdmap e520: 12 total, 12 up, 12 in 2023-03-09T17:15:21.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:15:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8231.047985 secs 2023-03-09T17:15:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:15:20 smithi181 ceph-mon[94189]: osdmap e520: 12 total, 12 up, 12 in 2023-03-09T17:15:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:15:21 smithi038 ceph-mon[92422]: pgmap v4903: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 36 KiB/s wr, 7 op/s 2023-03-09T17:15:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:15:21 smithi122 ceph-mon[93869]: pgmap v4903: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 36 KiB/s wr, 7 op/s 2023-03-09T17:15:22.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:15:21 smithi181 ceph-mon[94189]: pgmap v4903: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 36 KiB/s wr, 7 op/s 2023-03-09T17:15:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:15:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8233.309746 secs 2023-03-09T17:15:23.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:15:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8233.309746 secs 2023-03-09T17:15:23.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:15:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8233.309746 secs 2023-03-09T17:15:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8695.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:15:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:15:23 smithi122 ceph-mon[93869]: pgmap v4905: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 37 KiB/s wr, 7 op/s 2023-03-09T17:15:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:15:23 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:15:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:15:23 smithi181 ceph-mon[94189]: pgmap v4905: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 37 KiB/s wr, 7 op/s 2023-03-09T17:15:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:15:23 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:15:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:15:23 smithi038 ceph-mon[92422]: pgmap v4905: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 37 KiB/s wr, 7 op/s 2023-03-09T17:15:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:15:23 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:15:25.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:15:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:15:25.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:15:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:15:25.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:15:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:15:25.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:15:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:15:25.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:15:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:15:25.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:15:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:15:25.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:15:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:15:25.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:15:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:15:25.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:15:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:15:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:15:25 smithi122 ceph-mon[93869]: pgmap v4906: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T17:15:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:15:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8236.048048 secs 2023-03-09T17:15:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:15:25 smithi181 ceph-mon[94189]: pgmap v4906: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T17:15:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:15:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8236.048048 secs 2023-03-09T17:15:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:15:25 smithi038 ceph-mon[92422]: pgmap v4906: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T17:15:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:15:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8236.048048 secs 2023-03-09T17:15:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:15:27 smithi122 ceph-mon[93869]: pgmap v4907: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 7 op/s 2023-03-09T17:15:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:15:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8238.309843 secs 2023-03-09T17:15:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:15:27 smithi181 ceph-mon[94189]: pgmap v4907: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 7 op/s 2023-03-09T17:15:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:15:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8238.309843 secs 2023-03-09T17:15:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:15:27 smithi038 ceph-mon[92422]: pgmap v4907: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 7 op/s 2023-03-09T17:15:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:15:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8238.309843 secs 2023-03-09T17:15:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8700.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:15:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:15:29 smithi122 ceph-mon[93869]: pgmap v4908: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T17:15:30.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:15:29 smithi181 ceph-mon[94189]: pgmap v4908: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T17:15:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:15:29 smithi038 ceph-mon[92422]: pgmap v4908: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T17:15:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:15:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8241.048134 secs 2023-03-09T17:15:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:15:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8241.048134 secs 2023-03-09T17:15:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:15:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8241.048134 secs 2023-03-09T17:15:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:15:31 smithi122 ceph-mon[93869]: pgmap v4909: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T17:15:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:15:31 smithi181 ceph-mon[94189]: pgmap v4909: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T17:15:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:15:31 smithi038 ceph-mon[92422]: pgmap v4909: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T17:15:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:15:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8243.309965 secs 2023-03-09T17:15:33.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:15:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8243.309965 secs 2023-03-09T17:15:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:15:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8243.309965 secs 2023-03-09T17:15:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8705.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:15:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:15:33 smithi122 ceph-mon[93869]: pgmap v4910: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 877 B/s wr, 8 op/s 2023-03-09T17:15:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:15:33 smithi181 ceph-mon[94189]: pgmap v4910: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 877 B/s wr, 8 op/s 2023-03-09T17:15:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:15:33 smithi038 ceph-mon[92422]: pgmap v4910: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 877 B/s wr, 8 op/s 2023-03-09T17:15:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:15:35 smithi122 ceph-mon[93869]: pgmap v4911: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 597 B/s wr, 6 op/s 2023-03-09T17:15:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:15:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8246.048197 secs 2023-03-09T17:15:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:15:35 smithi181 ceph-mon[94189]: pgmap v4911: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 597 B/s wr, 6 op/s 2023-03-09T17:15:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:15:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8246.048197 secs 2023-03-09T17:15:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:15:35 smithi038 ceph-mon[92422]: pgmap v4911: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 597 B/s wr, 6 op/s 2023-03-09T17:15:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:15:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8246.048197 secs 2023-03-09T17:15:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:15:37 smithi122 ceph-mon[93869]: pgmap v4912: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 7 op/s 2023-03-09T17:15:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:15:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8248.310086 secs 2023-03-09T17:15:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:15:37 smithi181 ceph-mon[94189]: pgmap v4912: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 7 op/s 2023-03-09T17:15:38.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:15:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8248.310086 secs 2023-03-09T17:15:38.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:15:37 smithi038 ceph-mon[92422]: pgmap v4912: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 7 op/s 2023-03-09T17:15:38.254 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:15:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8248.310086 secs 2023-03-09T17:15:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8710.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:15:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:15:39 smithi122 ceph-mon[93869]: pgmap v4913: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 58 KiB/s wr, 7 op/s 2023-03-09T17:15:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:15:39 smithi181 ceph-mon[94189]: pgmap v4913: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 58 KiB/s wr, 7 op/s 2023-03-09T17:15:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:15:39 smithi038 ceph-mon[92422]: pgmap v4913: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 58 KiB/s wr, 7 op/s 2023-03-09T17:15:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:15:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8251.048259 secs 2023-03-09T17:15:41.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:15:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8251.048259 secs 2023-03-09T17:15:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:15:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8251.048259 secs 2023-03-09T17:15:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:15:41 smithi122 ceph-mon[93869]: pgmap v4914: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 58 KiB/s wr, 6 op/s 2023-03-09T17:15:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:15:41 smithi181 ceph-mon[94189]: pgmap v4914: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 58 KiB/s wr, 6 op/s 2023-03-09T17:15:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:15:41 smithi038 ceph-mon[92422]: pgmap v4914: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 58 KiB/s wr, 6 op/s 2023-03-09T17:15:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8715.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:15:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:15:43 smithi122 ceph-mon[93869]: pgmap v4915: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 66 KiB/s wr, 10 op/s 2023-03-09T17:15:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:15:43 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:15:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:15:43 smithi181 ceph-mon[94189]: pgmap v4915: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 66 KiB/s wr, 10 op/s 2023-03-09T17:15:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:15:43 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:15:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:15:43 smithi038 ceph-mon[92422]: pgmap v4915: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 66 KiB/s wr, 10 op/s 2023-03-09T17:15:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:15:43 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:15:45.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:15:44 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:15:45.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:15:44 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:15:45.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:15:44 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:15:45.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:15:44 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:15:45.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:15:44 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:15:45.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:15:44 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:15:45.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:15:44 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:15:45.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:15:44 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:15:45.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:15:44 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:15:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:15:45 smithi122 ceph-mon[93869]: pgmap v4916: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 81 KiB/s wr, 9 op/s 2023-03-09T17:15:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:15:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8256.048320 secs 2023-03-09T17:15:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:15:45 smithi181 ceph-mon[94189]: pgmap v4916: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 81 KiB/s wr, 9 op/s 2023-03-09T17:15:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:15:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8256.048320 secs 2023-03-09T17:15:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:15:45 smithi038 ceph-mon[92422]: pgmap v4916: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 81 KiB/s wr, 9 op/s 2023-03-09T17:15:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:15:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8256.048320 secs 2023-03-09T17:15:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:15:47 smithi122 ceph-mon[93869]: pgmap v4917: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 81 KiB/s wr, 10 op/s 2023-03-09T17:15:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:15:47 smithi181 ceph-mon[94189]: pgmap v4917: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 81 KiB/s wr, 10 op/s 2023-03-09T17:15:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:15:47 smithi038 ceph-mon[92422]: pgmap v4917: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 81 KiB/s wr, 10 op/s 2023-03-09T17:15:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8720.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:15:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:15:49 smithi122 ceph-mon[93869]: pgmap v4918: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 133 KiB/s wr, 10 op/s 2023-03-09T17:15:50.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:15:49 smithi181 ceph-mon[94189]: pgmap v4918: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 133 KiB/s wr, 10 op/s 2023-03-09T17:15:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:15:49 smithi038 ceph-mon[92422]: pgmap v4918: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 133 KiB/s wr, 10 op/s 2023-03-09T17:15:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:15:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8261.048413 secs 2023-03-09T17:15:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:15:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8261.048413 secs 2023-03-09T17:15:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:15:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8261.048413 secs 2023-03-09T17:15:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:15:51 smithi122 ceph-mon[93869]: pgmap v4919: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 76 KiB/s wr, 8 op/s 2023-03-09T17:15:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:15:51 smithi181 ceph-mon[94189]: pgmap v4919: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 76 KiB/s wr, 8 op/s 2023-03-09T17:15:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:15:51 smithi038 ceph-mon[92422]: pgmap v4919: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 76 KiB/s wr, 8 op/s 2023-03-09T17:15:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8725.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:15:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:15:53 smithi122 ceph-mon[93869]: pgmap v4920: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 77 KiB/s wr, 11 op/s 2023-03-09T17:15:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:15:53 smithi181 ceph-mon[94189]: pgmap v4920: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 77 KiB/s wr, 11 op/s 2023-03-09T17:15:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:15:53 smithi038 ceph-mon[92422]: pgmap v4920: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 77 KiB/s wr, 11 op/s 2023-03-09T17:15:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:15:55 smithi122 ceph-mon[93869]: pgmap v4921: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 85 KiB/s wr, 9 op/s 2023-03-09T17:15:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:15:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8266.048482 secs 2023-03-09T17:15:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:15:55 smithi181 ceph-mon[94189]: pgmap v4921: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 85 KiB/s wr, 9 op/s 2023-03-09T17:15:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:15:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8266.048482 secs 2023-03-09T17:15:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:15:55 smithi038 ceph-mon[92422]: pgmap v4921: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 85 KiB/s wr, 9 op/s 2023-03-09T17:15:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:15:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8266.048482 secs 2023-03-09T17:15:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:15:57 smithi122 ceph-mon[93869]: pgmap v4922: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 87 KiB/s wr, 10 op/s 2023-03-09T17:15:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:15:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8268.310538 secs 2023-03-09T17:15:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:15:57 smithi181 ceph-mon[94189]: pgmap v4922: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 87 KiB/s wr, 10 op/s 2023-03-09T17:15:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:15:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8268.310538 secs 2023-03-09T17:15:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:15:57 smithi038 ceph-mon[92422]: pgmap v4922: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 87 KiB/s wr, 10 op/s 2023-03-09T17:15:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:15:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8268.310538 secs 2023-03-09T17:15:58.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8730.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:16:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:15:59 smithi122 ceph-mon[93869]: pgmap v4923: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 104 KiB/s wr, 10 op/s 2023-03-09T17:16:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:15:59 smithi181 ceph-mon[94189]: pgmap v4923: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 104 KiB/s wr, 10 op/s 2023-03-09T17:16:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:15:59 smithi038 ceph-mon[92422]: pgmap v4923: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 104 KiB/s wr, 10 op/s 2023-03-09T17:16:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:16:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8271.048542 secs 2023-03-09T17:16:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:16:00 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:16:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:16:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:16:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:16:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:16:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:16:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8271.048542 secs 2023-03-09T17:16:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:16:00 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:16:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:16:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:16:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:16:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:16:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:16:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8271.048542 secs 2023-03-09T17:16:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:16:00 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:16:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:16:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:16:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:16:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:16:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:16:01 smithi122 ceph-mon[93869]: pgmap v4924: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 52 KiB/s wr, 8 op/s 2023-03-09T17:16:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:16:01 smithi181 ceph-mon[94189]: pgmap v4924: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 52 KiB/s wr, 8 op/s 2023-03-09T17:16:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:16:01 smithi038 ceph-mon[92422]: pgmap v4924: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 52 KiB/s wr, 8 op/s 2023-03-09T17:16:03.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:16:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8273.310641 secs 2023-03-09T17:16:03.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:16:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8273.310641 secs 2023-03-09T17:16:03.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:16:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8273.310641 secs 2023-03-09T17:16:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8735.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:16:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:16:03 smithi122 ceph-mon[93869]: pgmap v4925: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 99 KiB/s wr, 14 op/s 2023-03-09T17:16:04.237 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:16:03 smithi181 ceph-mon[94189]: pgmap v4925: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 99 KiB/s wr, 14 op/s 2023-03-09T17:16:04.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:16:03 smithi038 ceph-mon[92422]: pgmap v4925: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 99 KiB/s wr, 14 op/s 2023-03-09T17:16:05.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:16:04 smithi122 ceph-mon[93869]: osdmap e521: 12 total, 12 up, 12 in 2023-03-09T17:16:05.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:16:04 smithi181 ceph-mon[94189]: osdmap e521: 12 total, 12 up, 12 in 2023-03-09T17:16:05.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:16:04 smithi038 ceph-mon[92422]: osdmap e521: 12 total, 12 up, 12 in 2023-03-09T17:16:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:16:05 smithi122 ceph-mon[93869]: pgmap v4927: 129 pgs: 9 active+clean+snaptrim_wait, 120 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 124 KiB/s wr, 27 op/s 2023-03-09T17:16:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:16:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8276.048603 secs 2023-03-09T17:16:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:16:05 smithi181 ceph-mon[94189]: pgmap v4927: 129 pgs: 9 active+clean+snaptrim_wait, 120 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 124 KiB/s wr, 27 op/s 2023-03-09T17:16:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:16:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8276.048603 secs 2023-03-09T17:16:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:16:05 smithi038 ceph-mon[92422]: pgmap v4927: 129 pgs: 9 active+clean+snaptrim_wait, 120 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 124 KiB/s wr, 27 op/s 2023-03-09T17:16:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:16:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8276.048603 secs 2023-03-09T17:16:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:16:07 smithi122 ceph-mon[93869]: pgmap v4928: 129 pgs: 9 active+clean+snaptrim_wait, 120 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 150 KiB/s wr, 28 op/s 2023-03-09T17:16:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:16:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8278.310760 secs 2023-03-09T17:16:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:16:07 smithi181 ceph-mon[94189]: pgmap v4928: 129 pgs: 9 active+clean+snaptrim_wait, 120 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 150 KiB/s wr, 28 op/s 2023-03-09T17:16:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:16:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8278.310760 secs 2023-03-09T17:16:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:16:07 smithi038 ceph-mon[92422]: pgmap v4928: 129 pgs: 9 active+clean+snaptrim_wait, 120 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 150 KiB/s wr, 28 op/s 2023-03-09T17:16:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:16:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8278.310760 secs 2023-03-09T17:16:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8740.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:16:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:16:09 smithi122 ceph-mon[93869]: pgmap v4929: 129 pgs: 9 active+clean+snaptrim_wait, 120 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 131 KiB/s wr, 26 op/s 2023-03-09T17:16:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:16:09 smithi181 ceph-mon[94189]: pgmap v4929: 129 pgs: 9 active+clean+snaptrim_wait, 120 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 131 KiB/s wr, 26 op/s 2023-03-09T17:16:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:16:09 smithi038 ceph-mon[92422]: pgmap v4929: 129 pgs: 9 active+clean+snaptrim_wait, 120 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 131 KiB/s wr, 26 op/s 2023-03-09T17:16:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:16:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8281.048665 secs 2023-03-09T17:16:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:16:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8281.048665 secs 2023-03-09T17:16:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:16:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8281.048665 secs 2023-03-09T17:16:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:16:11 smithi122 ceph-mon[93869]: pgmap v4930: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 131 KiB/s wr, 26 op/s 2023-03-09T17:16:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:16:11 smithi181 ceph-mon[94189]: pgmap v4930: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 131 KiB/s wr, 26 op/s 2023-03-09T17:16:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:16:11 smithi038 ceph-mon[92422]: pgmap v4930: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 131 KiB/s wr, 26 op/s 2023-03-09T17:16:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:16:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8283.310826 secs 2023-03-09T17:16:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:16:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8283.310826 secs 2023-03-09T17:16:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:16:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8283.310826 secs 2023-03-09T17:16:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8745.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:16:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:16:13 smithi122 ceph-mon[93869]: pgmap v4931: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 83 KiB/s wr, 23 op/s 2023-03-09T17:16:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:16:13 smithi181 ceph-mon[94189]: pgmap v4931: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 83 KiB/s wr, 23 op/s 2023-03-09T17:16:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:16:13 smithi038 ceph-mon[92422]: pgmap v4931: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 83 KiB/s wr, 23 op/s 2023-03-09T17:16:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:16:15 smithi122 ceph-mon[93869]: pgmap v4932: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 58 KiB/s wr, 8 op/s 2023-03-09T17:16:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:16:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8286.048725 secs 2023-03-09T17:16:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:16:15 smithi122 ceph-mon[93869]: osdmap e522: 12 total, 12 up, 12 in 2023-03-09T17:16:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:16:15 smithi181 ceph-mon[94189]: pgmap v4932: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 58 KiB/s wr, 8 op/s 2023-03-09T17:16:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:16:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8286.048725 secs 2023-03-09T17:16:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:16:15 smithi181 ceph-mon[94189]: osdmap e522: 12 total, 12 up, 12 in 2023-03-09T17:16:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:16:15 smithi038 ceph-mon[92422]: pgmap v4932: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 58 KiB/s wr, 8 op/s 2023-03-09T17:16:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:16:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8286.048725 secs 2023-03-09T17:16:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:16:15 smithi038 ceph-mon[92422]: osdmap e522: 12 total, 12 up, 12 in 2023-03-09T17:16:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:16:17 smithi122 ceph-mon[93869]: pgmap v4934: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 11 KiB/s wr, 7 op/s 2023-03-09T17:16:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:16:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8288.310940 secs 2023-03-09T17:16:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:16:17 smithi181 ceph-mon[94189]: pgmap v4934: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 11 KiB/s wr, 7 op/s 2023-03-09T17:16:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:16:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8288.310940 secs 2023-03-09T17:16:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:16:17 smithi038 ceph-mon[92422]: pgmap v4934: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 11 KiB/s wr, 7 op/s 2023-03-09T17:16:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:16:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8288.310940 secs 2023-03-09T17:16:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8750.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:16:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:16:19 smithi122 ceph-mon[93869]: pgmap v4935: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 7 op/s 2023-03-09T17:16:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:16:19 smithi181 ceph-mon[94189]: pgmap v4935: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 7 op/s 2023-03-09T17:16:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:16:19 smithi038 ceph-mon[92422]: pgmap v4935: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 7 op/s 2023-03-09T17:16:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:16:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8291.048788 secs 2023-03-09T17:16:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:16:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8291.048788 secs 2023-03-09T17:16:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:16:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8291.048788 secs 2023-03-09T17:16:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:16:21 smithi122 ceph-mon[93869]: pgmap v4936: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 7 op/s 2023-03-09T17:16:22.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:16:21 smithi181 ceph-mon[94189]: pgmap v4936: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 7 op/s 2023-03-09T17:16:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:16:21 smithi038 ceph-mon[92422]: pgmap v4936: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 7 op/s 2023-03-09T17:16:23.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:16:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8293.311001 secs 2023-03-09T17:16:23.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:16:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8293.311001 secs 2023-03-09T17:16:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:16:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8293.311001 secs 2023-03-09T17:16:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8755.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:16:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:16:23 smithi122 ceph-mon[93869]: pgmap v4937: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T17:16:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:16:23 smithi181 ceph-mon[94189]: pgmap v4937: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T17:16:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:16:23 smithi038 ceph-mon[92422]: pgmap v4937: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T17:16:25.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:16:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:16:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:16:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:16:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:16:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:16:26.231 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:16:25 smithi038 ceph-mon[92422]: pgmap v4938: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T17:16:26.232 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:16:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8296.048849 secs 2023-03-09T17:16:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:16:25 smithi122 ceph-mon[93869]: pgmap v4938: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T17:16:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:16:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8296.048849 secs 2023-03-09T17:16:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:16:25 smithi181 ceph-mon[94189]: pgmap v4938: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T17:16:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:16:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8296.048849 secs 2023-03-09T17:16:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:16:27 smithi038 ceph-mon[92422]: pgmap v4939: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 22 KiB/s wr, 10 op/s 2023-03-09T17:16:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:16:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8298.311136 secs 2023-03-09T17:16:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:16:27 smithi122 ceph-mon[93869]: pgmap v4939: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 22 KiB/s wr, 10 op/s 2023-03-09T17:16:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:16:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8298.311136 secs 2023-03-09T17:16:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:16:27 smithi181 ceph-mon[94189]: pgmap v4939: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 22 KiB/s wr, 10 op/s 2023-03-09T17:16:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:16:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8298.311136 secs 2023-03-09T17:16:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8760.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:16:29.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:16:29 smithi122 ceph-mon[93869]: pgmap v4940: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T17:16:29.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:16:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:16:29.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:16:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:16:29.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:16:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:16:29.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:16:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:16:29.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:16:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:16:29.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:16:29 smithi181 ceph-mon[94189]: pgmap v4940: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T17:16:29.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:16:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:16:29.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:16:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:16:29.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:16:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:16:29.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:16:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:16:29.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:16:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:16:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:16:29 smithi038 ceph-mon[92422]: pgmap v4940: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T17:16:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:16:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:16:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:16:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:16:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:16:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:16:29.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:16:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:16:29.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:16:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:16:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:16:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8301.048911 secs 2023-03-09T17:16:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:16:31 smithi122 ceph-mon[93869]: pgmap v4941: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 6 op/s 2023-03-09T17:16:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:16:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8301.048911 secs 2023-03-09T17:16:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:16:31 smithi181 ceph-mon[94189]: pgmap v4941: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 6 op/s 2023-03-09T17:16:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:16:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8301.048911 secs 2023-03-09T17:16:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:16:31 smithi038 ceph-mon[92422]: pgmap v4941: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 6 op/s 2023-03-09T17:16:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:16:33 smithi122 ceph-mon[93869]: pgmap v4942: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 52 KiB/s wr, 10 op/s 2023-03-09T17:16:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:16:33 smithi181 ceph-mon[94189]: pgmap v4942: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 52 KiB/s wr, 10 op/s 2023-03-09T17:16:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:16:33 smithi038 ceph-mon[92422]: pgmap v4942: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 52 KiB/s wr, 10 op/s 2023-03-09T17:16:33.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8765.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:16:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:16:35 smithi122 ceph-mon[93869]: pgmap v4943: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 34 KiB/s wr, 7 op/s 2023-03-09T17:16:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:16:35 smithi181 ceph-mon[94189]: pgmap v4943: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 34 KiB/s wr, 7 op/s 2023-03-09T17:16:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:16:35 smithi038 ceph-mon[92422]: pgmap v4943: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 34 KiB/s wr, 7 op/s 2023-03-09T17:16:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:16:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8306.048976 secs 2023-03-09T17:16:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:16:36 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:16:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:16:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8306.048976 secs 2023-03-09T17:16:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:16:36 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:16:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:16:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8306.048976 secs 2023-03-09T17:16:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:16:36 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:16:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:16:37 smithi122 ceph-mon[93869]: pgmap v4944: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 37 KiB/s wr, 9 op/s 2023-03-09T17:16:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:16:37 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:16:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:16:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:16:37.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:16:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:16:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:16:37 smithi181 ceph-mon[94189]: pgmap v4944: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 37 KiB/s wr, 9 op/s 2023-03-09T17:16:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:16:37 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:16:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:16:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:16:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:16:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:16:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:16:37 smithi038 ceph-mon[92422]: pgmap v4944: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 37 KiB/s wr, 9 op/s 2023-03-09T17:16:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:16:37 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:16:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:16:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:16:37.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:16:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:16:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8770.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:16:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:16:39 smithi122 ceph-mon[93869]: pgmap v4945: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 36 KiB/s wr, 7 op/s 2023-03-09T17:16:39.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:16:39 smithi181 ceph-mon[94189]: pgmap v4945: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 36 KiB/s wr, 7 op/s 2023-03-09T17:16:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:16:39 smithi038 ceph-mon[92422]: pgmap v4945: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 36 KiB/s wr, 7 op/s 2023-03-09T17:16:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:16:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8311.049038 secs 2023-03-09T17:16:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:16:41 smithi122 ceph-mon[93869]: pgmap v4946: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 KiB/s wr, 6 op/s 2023-03-09T17:16:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:16:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8311.049038 secs 2023-03-09T17:16:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:16:41 smithi181 ceph-mon[94189]: pgmap v4946: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 KiB/s wr, 6 op/s 2023-03-09T17:16:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:16:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8311.049038 secs 2023-03-09T17:16:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:16:41 smithi038 ceph-mon[92422]: pgmap v4946: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 KiB/s wr, 6 op/s 2023-03-09T17:16:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:16:43 smithi122 ceph-mon[93869]: pgmap v4947: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 74 KiB/s wr, 10 op/s 2023-03-09T17:16:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:16:43 smithi181 ceph-mon[94189]: pgmap v4947: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 74 KiB/s wr, 10 op/s 2023-03-09T17:16:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:16:43 smithi038 ceph-mon[92422]: pgmap v4947: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 74 KiB/s wr, 10 op/s 2023-03-09T17:16:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8775.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:16:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:16:45 smithi122 ceph-mon[93869]: pgmap v4948: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 41 KiB/s wr, 7 op/s 2023-03-09T17:16:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:16:45 smithi181 ceph-mon[94189]: pgmap v4948: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 41 KiB/s wr, 7 op/s 2023-03-09T17:16:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:16:45 smithi038 ceph-mon[92422]: pgmap v4948: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 41 KiB/s wr, 7 op/s 2023-03-09T17:16:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:16:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8316.049100 secs 2023-03-09T17:16:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:16:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8316.049100 secs 2023-03-09T17:16:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:16:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8316.049100 secs 2023-03-09T17:16:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:16:47 smithi122 ceph-mon[93869]: pgmap v4949: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 64 KiB/s wr, 10 op/s 2023-03-09T17:16:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:16:47 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 8318.311586 secs 2023-03-09T17:16:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:16:47 smithi181 ceph-mon[94189]: pgmap v4949: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 64 KiB/s wr, 10 op/s 2023-03-09T17:16:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:16:47 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 8318.311586 secs 2023-03-09T17:16:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:16:47 smithi038 ceph-mon[92422]: pgmap v4949: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 64 KiB/s wr, 10 op/s 2023-03-09T17:16:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:16:47 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 8318.311586 secs 2023-03-09T17:16:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:16:48 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:16:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:16:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:16:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:16:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:16:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:16:48 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:16:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:16:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:16:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:16:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:16:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:16:48 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:16:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:16:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:16:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:16:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:16:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8780.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:16:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:16:49 smithi122 ceph-mon[93869]: pgmap v4950: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 77 KiB/s wr, 9 op/s 2023-03-09T17:16:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:16:49 smithi181 ceph-mon[94189]: pgmap v4950: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 77 KiB/s wr, 9 op/s 2023-03-09T17:16:49.757 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:16:49 smithi038 ceph-mon[92422]: pgmap v4950: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 77 KiB/s wr, 9 op/s 2023-03-09T17:16:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:16:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8321.049159 secs 2023-03-09T17:16:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:16:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8321.049159 secs 2023-03-09T17:16:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:16:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8321.049159 secs 2023-03-09T17:16:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:16:51 smithi122 ceph-mon[93869]: pgmap v4951: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 77 KiB/s wr, 8 op/s 2023-03-09T17:16:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:16:51 smithi181 ceph-mon[94189]: pgmap v4951: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 77 KiB/s wr, 8 op/s 2023-03-09T17:16:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:16:51 smithi038 ceph-mon[92422]: pgmap v4951: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 77 KiB/s wr, 8 op/s 2023-03-09T17:16:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:16:52 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 8323.311699 secs 2023-03-09T17:16:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:16:52 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 8323.311699 secs 2023-03-09T17:16:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:16:52 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 8323.311699 secs 2023-03-09T17:16:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:16:53 smithi122 ceph-mon[93869]: pgmap v4952: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 121 KiB/s wr, 12 op/s 2023-03-09T17:16:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:16:53 smithi181 ceph-mon[94189]: pgmap v4952: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 121 KiB/s wr, 12 op/s 2023-03-09T17:16:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:16:53 smithi038 ceph-mon[92422]: pgmap v4952: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 121 KiB/s wr, 12 op/s 2023-03-09T17:16:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8785.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:16:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:16:55 smithi122 ceph-mon[93869]: pgmap v4953: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 84 KiB/s wr, 9 op/s 2023-03-09T17:16:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:16:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8326.049221 secs 2023-03-09T17:16:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:16:55 smithi181 ceph-mon[94189]: pgmap v4953: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 84 KiB/s wr, 9 op/s 2023-03-09T17:16:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:16:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8326.049221 secs 2023-03-09T17:16:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:16:55 smithi038 ceph-mon[92422]: pgmap v4953: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 84 KiB/s wr, 9 op/s 2023-03-09T17:16:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:16:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8326.049221 secs 2023-03-09T17:16:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:16:57 smithi122 ceph-mon[93869]: pgmap v4954: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 90 KiB/s wr, 12 op/s 2023-03-09T17:16:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:16:57 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 8328.311813 secs 2023-03-09T17:16:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:16:57 smithi181 ceph-mon[94189]: pgmap v4954: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 90 KiB/s wr, 12 op/s 2023-03-09T17:16:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:16:57 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 8328.311813 secs 2023-03-09T17:16:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:16:57 smithi038 ceph-mon[92422]: pgmap v4954: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 90 KiB/s wr, 12 op/s 2023-03-09T17:16:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:16:57 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 8328.311813 secs 2023-03-09T17:16:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8790.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:16:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:16:59 smithi122 ceph-mon[93869]: pgmap v4955: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 75 KiB/s wr, 10 op/s 2023-03-09T17:16:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:16:59 smithi181 ceph-mon[94189]: pgmap v4955: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 75 KiB/s wr, 10 op/s 2023-03-09T17:16:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:16:59 smithi038 ceph-mon[92422]: pgmap v4955: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 75 KiB/s wr, 10 op/s 2023-03-09T17:17:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:17:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8331.049281 secs 2023-03-09T17:17:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:17:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8331.049281 secs 2023-03-09T17:17:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:17:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8331.049281 secs 2023-03-09T17:17:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:17:01 smithi122 ceph-mon[93869]: pgmap v4956: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 59 KiB/s wr, 8 op/s 2023-03-09T17:17:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:17:01 smithi181 ceph-mon[94189]: pgmap v4956: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 59 KiB/s wr, 8 op/s 2023-03-09T17:17:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:17:01 smithi038 ceph-mon[92422]: pgmap v4956: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 59 KiB/s wr, 8 op/s 2023-03-09T17:17:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:17:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8333.311948 secs 2023-03-09T17:17:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:17:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8333.311948 secs 2023-03-09T17:17:02.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:17:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8333.311948 secs 2023-03-09T17:17:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:17:03 smithi038 ceph-mon[92422]: pgmap v4957: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 166 KiB/s wr, 14 op/s 2023-03-09T17:17:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8795.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:17:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:17:03 smithi122 ceph-mon[93869]: pgmap v4957: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 166 KiB/s wr, 14 op/s 2023-03-09T17:17:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:17:03 smithi181 ceph-mon[94189]: pgmap v4957: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 166 KiB/s wr, 14 op/s 2023-03-09T17:17:04.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:17:04 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:17:04.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:17:04 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:17:04.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:17:04 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:17:04.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:17:04 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:17:04.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:17:04 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:17:04.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:17:04 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:17:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:17:05 smithi122 ceph-mon[93869]: pgmap v4958: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 143 KiB/s wr, 23 op/s 2023-03-09T17:17:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:17:05 smithi122 ceph-mon[93869]: osdmap e523: 12 total, 12 up, 12 in 2023-03-09T17:17:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:17:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8336.049924 secs 2023-03-09T17:17:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:17:05 smithi181 ceph-mon[94189]: pgmap v4958: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 143 KiB/s wr, 23 op/s 2023-03-09T17:17:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:17:05 smithi181 ceph-mon[94189]: osdmap e523: 12 total, 12 up, 12 in 2023-03-09T17:17:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:17:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8336.049924 secs 2023-03-09T17:17:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:17:05 smithi038 ceph-mon[92422]: pgmap v4958: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 143 KiB/s wr, 23 op/s 2023-03-09T17:17:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:17:05 smithi038 ceph-mon[92422]: osdmap e523: 12 total, 12 up, 12 in 2023-03-09T17:17:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:17:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8336.049924 secs 2023-03-09T17:17:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:17:07 smithi122 ceph-mon[93869]: pgmap v4960: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 230 KiB/s wr, 29 op/s 2023-03-09T17:17:07.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:17:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8338.312072 secs 2023-03-09T17:17:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:17:07 smithi181 ceph-mon[94189]: pgmap v4960: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 230 KiB/s wr, 29 op/s 2023-03-09T17:17:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:17:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8338.312072 secs 2023-03-09T17:17:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:17:07 smithi038 ceph-mon[92422]: pgmap v4960: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 230 KiB/s wr, 29 op/s 2023-03-09T17:17:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:17:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8338.312072 secs 2023-03-09T17:17:08.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8800.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:17:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:17:09 smithi122 ceph-mon[93869]: pgmap v4961: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 221 KiB/s wr, 28 op/s 2023-03-09T17:17:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:17:09 smithi181 ceph-mon[94189]: pgmap v4961: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 221 KiB/s wr, 28 op/s 2023-03-09T17:17:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:17:09 smithi038 ceph-mon[92422]: pgmap v4961: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 221 KiB/s wr, 28 op/s 2023-03-09T17:17:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:17:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8341.049982 secs 2023-03-09T17:17:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:17:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8341.049982 secs 2023-03-09T17:17:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:17:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8341.049982 secs 2023-03-09T17:17:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:17:11 smithi122 ceph-mon[93869]: pgmap v4962: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 221 KiB/s wr, 28 op/s 2023-03-09T17:17:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:17:11 smithi181 ceph-mon[94189]: pgmap v4962: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 221 KiB/s wr, 28 op/s 2023-03-09T17:17:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:17:11 smithi038 ceph-mon[92422]: pgmap v4962: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 221 KiB/s wr, 28 op/s 2023-03-09T17:17:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:17:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8343.312169 secs 2023-03-09T17:17:12.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:17:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8343.312169 secs 2023-03-09T17:17:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:17:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8343.312169 secs 2023-03-09T17:17:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8805.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:17:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:17:13 smithi122 ceph-mon[93869]: pgmap v4963: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 114 KiB/s wr, 26 op/s 2023-03-09T17:17:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:17:13 smithi181 ceph-mon[94189]: pgmap v4963: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 114 KiB/s wr, 26 op/s 2023-03-09T17:17:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:17:13 smithi038 ceph-mon[92422]: pgmap v4963: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 114 KiB/s wr, 26 op/s 2023-03-09T17:17:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:17:15 smithi122 ceph-mon[93869]: pgmap v4964: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 93 KiB/s wr, 11 op/s 2023-03-09T17:17:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:17:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8346.050066 secs 2023-03-09T17:17:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:17:15 smithi122 ceph-mon[93869]: osdmap e524: 12 total, 12 up, 12 in 2023-03-09T17:17:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:17:15 smithi181 ceph-mon[94189]: pgmap v4964: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 93 KiB/s wr, 11 op/s 2023-03-09T17:17:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:17:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8346.050066 secs 2023-03-09T17:17:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:17:15 smithi181 ceph-mon[94189]: osdmap e524: 12 total, 12 up, 12 in 2023-03-09T17:17:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:17:15 smithi038 ceph-mon[92422]: pgmap v4964: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 93 KiB/s wr, 11 op/s 2023-03-09T17:17:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:17:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8346.050066 secs 2023-03-09T17:17:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:17:15 smithi038 ceph-mon[92422]: osdmap e524: 12 total, 12 up, 12 in 2023-03-09T17:17:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:17:17 smithi122 ceph-mon[93869]: pgmap v4966: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 28 KiB/s wr, 9 op/s 2023-03-09T17:17:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:17:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8348.312267 secs 2023-03-09T17:17:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:17:17 smithi181 ceph-mon[94189]: pgmap v4966: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 28 KiB/s wr, 9 op/s 2023-03-09T17:17:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:17:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8348.312267 secs 2023-03-09T17:17:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:17:17 smithi038 ceph-mon[92422]: pgmap v4966: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 28 KiB/s wr, 9 op/s 2023-03-09T17:17:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:17:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8348.312267 secs 2023-03-09T17:17:18.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8810.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:17:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:17:19 smithi122 ceph-mon[93869]: pgmap v4967: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 28 KiB/s wr, 10 op/s 2023-03-09T17:17:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:17:19 smithi181 ceph-mon[94189]: pgmap v4967: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 28 KiB/s wr, 10 op/s 2023-03-09T17:17:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:17:19 smithi038 ceph-mon[92422]: pgmap v4967: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 28 KiB/s wr, 10 op/s 2023-03-09T17:17:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:17:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8351.050124 secs 2023-03-09T17:17:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:17:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8351.050124 secs 2023-03-09T17:17:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:17:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8351.050124 secs 2023-03-09T17:17:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:17:21 smithi122 ceph-mon[93869]: pgmap v4968: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 28 KiB/s wr, 10 op/s 2023-03-09T17:17:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:17:21 smithi181 ceph-mon[94189]: pgmap v4968: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 28 KiB/s wr, 10 op/s 2023-03-09T17:17:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:17:21 smithi038 ceph-mon[92422]: pgmap v4968: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 28 KiB/s wr, 10 op/s 2023-03-09T17:17:22.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:17:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8353.312391 secs 2023-03-09T17:17:22.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:17:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8353.312391 secs 2023-03-09T17:17:22.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:17:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8353.312391 secs 2023-03-09T17:17:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8815.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:17:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:17:23 smithi122 ceph-mon[93869]: pgmap v4969: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 21 KiB/s wr, 9 op/s 2023-03-09T17:17:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:17:23 smithi181 ceph-mon[94189]: pgmap v4969: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 21 KiB/s wr, 9 op/s 2023-03-09T17:17:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:17:23 smithi038 ceph-mon[92422]: pgmap v4969: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 21 KiB/s wr, 9 op/s 2023-03-09T17:17:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:17:25 smithi122 ceph-mon[93869]: pgmap v4970: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 16 KiB/s wr, 9 op/s 2023-03-09T17:17:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:17:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8356.050202 secs 2023-03-09T17:17:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:17:25 smithi181 ceph-mon[94189]: pgmap v4970: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 16 KiB/s wr, 9 op/s 2023-03-09T17:17:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:17:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8356.050202 secs 2023-03-09T17:17:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:17:25 smithi038 ceph-mon[92422]: pgmap v4970: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 16 KiB/s wr, 9 op/s 2023-03-09T17:17:26.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:17:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8356.050202 secs 2023-03-09T17:17:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:17:27 smithi122 ceph-mon[93869]: pgmap v4971: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 36 KiB/s wr, 11 op/s 2023-03-09T17:17:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:17:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8358.312530 secs 2023-03-09T17:17:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:17:27 smithi181 ceph-mon[94189]: pgmap v4971: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 36 KiB/s wr, 11 op/s 2023-03-09T17:17:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:17:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8358.312530 secs 2023-03-09T17:17:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:17:27 smithi038 ceph-mon[92422]: pgmap v4971: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 36 KiB/s wr, 11 op/s 2023-03-09T17:17:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:17:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8358.312530 secs 2023-03-09T17:17:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8820.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:17:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:17:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:17:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:17:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:17:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:17:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:17:29.884 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:17:29 smithi181 ceph-mon[94189]: pgmap v4972: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 32 KiB/s wr, 8 op/s 2023-03-09T17:17:29.951 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:17:29 smithi122 ceph-mon[93869]: pgmap v4972: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 32 KiB/s wr, 8 op/s 2023-03-09T17:17:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:17:29 smithi038 ceph-mon[92422]: pgmap v4972: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 32 KiB/s wr, 8 op/s 2023-03-09T17:17:30.897 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:17:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8361.050262 secs 2023-03-09T17:17:30.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:17:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8361.050262 secs 2023-03-09T17:17:30.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:17:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8361.050262 secs 2023-03-09T17:17:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:17:31 smithi122 ceph-mon[93869]: pgmap v4973: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 31 KiB/s wr, 7 op/s 2023-03-09T17:17:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:17:31 smithi181 ceph-mon[94189]: pgmap v4973: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 31 KiB/s wr, 7 op/s 2023-03-09T17:17:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:17:31 smithi038 ceph-mon[92422]: pgmap v4973: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 31 KiB/s wr, 7 op/s 2023-03-09T17:17:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:17:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8363.312594 secs 2023-03-09T17:17:33.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:17:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:17:33.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:17:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:17:33.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:17:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:17:33.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:17:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:17:33.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:17:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:17:33.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:17:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:17:33.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:17:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:17:33.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:17:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8363.312594 secs 2023-03-09T17:17:33.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:17:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:17:33.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:17:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:17:33.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:17:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:17:33.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:17:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:17:33.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:17:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:17:33.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:17:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:17:33.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:17:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:17:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:17:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8363.312594 secs 2023-03-09T17:17:33.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:17:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:17:33.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:17:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:17:33.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:17:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:17:33.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:17:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:17:33.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:17:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:17:33.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:17:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:17:33.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:17:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:17:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8825.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:17:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:17:33 smithi038 ceph-mon[92422]: pgmap v4974: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 31 KiB/s wr, 9 op/s 2023-03-09T17:17:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:17:33 smithi122 ceph-mon[93869]: pgmap v4974: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 31 KiB/s wr, 9 op/s 2023-03-09T17:17:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:17:33 smithi181 ceph-mon[94189]: pgmap v4974: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 31 KiB/s wr, 9 op/s 2023-03-09T17:17:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:17:35 smithi038 ceph-mon[92422]: pgmap v4975: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T17:17:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:17:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8366.050318 secs 2023-03-09T17:17:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:17:35 smithi122 ceph-mon[93869]: pgmap v4975: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T17:17:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:17:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8366.050318 secs 2023-03-09T17:17:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:17:35 smithi181 ceph-mon[94189]: pgmap v4975: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T17:17:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:17:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8366.050318 secs 2023-03-09T17:17:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:17:37 smithi038 ceph-mon[92422]: pgmap v4976: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 9 op/s 2023-03-09T17:17:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:17:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8368.312729 secs 2023-03-09T17:17:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:17:37 smithi122 ceph-mon[93869]: pgmap v4976: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 9 op/s 2023-03-09T17:17:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:17:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8368.312729 secs 2023-03-09T17:17:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:17:37 smithi181 ceph-mon[94189]: pgmap v4976: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 9 op/s 2023-03-09T17:17:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:17:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8368.312729 secs 2023-03-09T17:17:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8830.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:17:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:17:39 smithi038 ceph-mon[92422]: pgmap v4977: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 46 KiB/s wr, 8 op/s 2023-03-09T17:17:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:17:39 smithi122 ceph-mon[93869]: pgmap v4977: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 46 KiB/s wr, 8 op/s 2023-03-09T17:17:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:17:39 smithi181 ceph-mon[94189]: pgmap v4977: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 46 KiB/s wr, 8 op/s 2023-03-09T17:17:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:17:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8371.050401 secs 2023-03-09T17:17:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:17:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8371.050401 secs 2023-03-09T17:17:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:17:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8371.050401 secs 2023-03-09T17:17:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:17:42 smithi122 ceph-mon[93869]: pgmap v4978: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 46 KiB/s wr, 7 op/s 2023-03-09T17:17:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:17:42 smithi181 ceph-mon[94189]: pgmap v4978: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 46 KiB/s wr, 7 op/s 2023-03-09T17:17:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:17:42 smithi038 ceph-mon[92422]: pgmap v4978: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 46 KiB/s wr, 7 op/s 2023-03-09T17:17:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:17:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8373.312851 secs 2023-03-09T17:17:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:17:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8373.312851 secs 2023-03-09T17:17:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:17:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8373.312851 secs 2023-03-09T17:17:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8835.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:17:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:17:44 smithi122 ceph-mon[93869]: pgmap v4979: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 76 KiB/s wr, 10 op/s 2023-03-09T17:17:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:17:44 smithi181 ceph-mon[94189]: pgmap v4979: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 76 KiB/s wr, 10 op/s 2023-03-09T17:17:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:17:44 smithi038 ceph-mon[92422]: pgmap v4979: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 76 KiB/s wr, 10 op/s 2023-03-09T17:17:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:17:46 smithi122 ceph-mon[93869]: pgmap v4980: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 84 KiB/s wr, 9 op/s 2023-03-09T17:17:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:17:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8376.050461 secs 2023-03-09T17:17:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:17:46 smithi181 ceph-mon[94189]: pgmap v4980: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 84 KiB/s wr, 9 op/s 2023-03-09T17:17:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:17:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8376.050461 secs 2023-03-09T17:17:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:17:46 smithi038 ceph-mon[92422]: pgmap v4980: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 84 KiB/s wr, 9 op/s 2023-03-09T17:17:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:17:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8376.050461 secs 2023-03-09T17:17:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:17:47 smithi122 ceph-mon[93869]: pgmap v4981: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 117 KiB/s wr, 11 op/s 2023-03-09T17:17:47.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:17:47 smithi181 ceph-mon[94189]: pgmap v4981: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 117 KiB/s wr, 11 op/s 2023-03-09T17:17:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:17:47 smithi038 ceph-mon[92422]: pgmap v4981: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 117 KiB/s wr, 11 op/s 2023-03-09T17:17:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:17:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8378.312957 secs 2023-03-09T17:17:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:17:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8378.312957 secs 2023-03-09T17:17:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:17:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8378.312957 secs 2023-03-09T17:17:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8840.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:17:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:17:49 smithi122 ceph-mon[93869]: pgmap v4982: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 87 KiB/s wr, 10 op/s 2023-03-09T17:17:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:17:49 smithi181 ceph-mon[94189]: pgmap v4982: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 87 KiB/s wr, 10 op/s 2023-03-09T17:17:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:17:49 smithi038 ceph-mon[92422]: pgmap v4982: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 87 KiB/s wr, 10 op/s 2023-03-09T17:17:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:17:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8381.050527 secs 2023-03-09T17:17:51.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:17:51 smithi122 ceph-mon[93869]: pgmap v4983: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 79 KiB/s wr, 9 op/s 2023-03-09T17:17:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:17:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8381.050527 secs 2023-03-09T17:17:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:17:51 smithi181 ceph-mon[94189]: pgmap v4983: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 79 KiB/s wr, 9 op/s 2023-03-09T17:17:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:17:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8381.050527 secs 2023-03-09T17:17:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:17:51 smithi038 ceph-mon[92422]: pgmap v4983: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 79 KiB/s wr, 9 op/s 2023-03-09T17:17:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:17:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8383.313042 secs 2023-03-09T17:17:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:17:53 smithi122 ceph-mon[93869]: pgmap v4984: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 109 KiB/s wr, 11 op/s 2023-03-09T17:17:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:17:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8383.313042 secs 2023-03-09T17:17:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:17:53 smithi181 ceph-mon[94189]: pgmap v4984: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 109 KiB/s wr, 11 op/s 2023-03-09T17:17:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:17:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8383.313042 secs 2023-03-09T17:17:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:17:53 smithi038 ceph-mon[92422]: pgmap v4984: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 109 KiB/s wr, 11 op/s 2023-03-09T17:17:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8845.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:17:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:17:55 smithi122 ceph-mon[93869]: pgmap v4985: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 86 KiB/s wr, 9 op/s 2023-03-09T17:17:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:17:55 smithi181 ceph-mon[94189]: pgmap v4985: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 86 KiB/s wr, 9 op/s 2023-03-09T17:17:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:17:55 smithi038 ceph-mon[92422]: pgmap v4985: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 86 KiB/s wr, 9 op/s 2023-03-09T17:17:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:17:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8386.050588 secs 2023-03-09T17:17:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:17:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8386.050588 secs 2023-03-09T17:17:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:17:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8386.050588 secs 2023-03-09T17:17:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:17:57 smithi122 ceph-mon[93869]: pgmap v4986: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 128 KiB/s wr, 11 op/s 2023-03-09T17:17:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:17:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8388.313161 secs 2023-03-09T17:17:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:17:57 smithi181 ceph-mon[94189]: pgmap v4986: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 128 KiB/s wr, 11 op/s 2023-03-09T17:17:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:17:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8388.313161 secs 2023-03-09T17:17:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:17:57 smithi038 ceph-mon[92422]: pgmap v4986: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 128 KiB/s wr, 11 op/s 2023-03-09T17:17:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:17:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8388.313161 secs 2023-03-09T17:17:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8850.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:17:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:17:59 smithi122 ceph-mon[93869]: pgmap v4987: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 102 KiB/s wr, 9 op/s 2023-03-09T17:17:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:17:59 smithi181 ceph-mon[94189]: pgmap v4987: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 102 KiB/s wr, 9 op/s 2023-03-09T17:17:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:17:59 smithi038 ceph-mon[92422]: pgmap v4987: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 102 KiB/s wr, 9 op/s 2023-03-09T17:18:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8391.050669 secs 2023-03-09T17:18:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8391.050669 secs 2023-03-09T17:18:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8391.050669 secs 2023-03-09T17:18:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:01 smithi122 ceph-mon[93869]: pgmap v4988: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 94 KiB/s wr, 7 op/s 2023-03-09T17:18:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:01 smithi181 ceph-mon[94189]: pgmap v4988: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 94 KiB/s wr, 7 op/s 2023-03-09T17:18:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:01 smithi038 ceph-mon[92422]: pgmap v4988: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 94 KiB/s wr, 7 op/s 2023-03-09T17:18:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8393.313286 secs 2023-03-09T17:18:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8393.313286 secs 2023-03-09T17:18:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8393.313286 secs 2023-03-09T17:18:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:03 smithi122 ceph-mon[93869]: pgmap v4989: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 109 KiB/s wr, 11 op/s 2023-03-09T17:18:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:03 smithi181 ceph-mon[94189]: pgmap v4989: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 109 KiB/s wr, 11 op/s 2023-03-09T17:18:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:03 smithi038 ceph-mon[92422]: pgmap v4989: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 109 KiB/s wr, 11 op/s 2023-03-09T17:18:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8855.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:18:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:05 smithi038 ceph-mon[92422]: pgmap v4990: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 101 KiB/s wr, 21 op/s 2023-03-09T17:18:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:05 smithi038 ceph-mon[92422]: osdmap e525: 12 total, 12 up, 12 in 2023-03-09T17:18:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8396.050733 secs 2023-03-09T17:18:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:05 smithi122 ceph-mon[93869]: pgmap v4990: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 101 KiB/s wr, 21 op/s 2023-03-09T17:18:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:05 smithi122 ceph-mon[93869]: osdmap e525: 12 total, 12 up, 12 in 2023-03-09T17:18:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8396.050733 secs 2023-03-09T17:18:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:05 smithi181 ceph-mon[94189]: pgmap v4990: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 101 KiB/s wr, 21 op/s 2023-03-09T17:18:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:05 smithi181 ceph-mon[94189]: osdmap e525: 12 total, 12 up, 12 in 2023-03-09T17:18:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8396.050733 secs 2023-03-09T17:18:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:07 smithi038 ceph-mon[92422]: pgmap v4992: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 114 KiB/s wr, 26 op/s 2023-03-09T17:18:07.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8398.313390 secs 2023-03-09T17:18:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:07 smithi122 ceph-mon[93869]: pgmap v4992: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 114 KiB/s wr, 26 op/s 2023-03-09T17:18:07.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8398.313390 secs 2023-03-09T17:18:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:07 smithi181 ceph-mon[94189]: pgmap v4992: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 114 KiB/s wr, 26 op/s 2023-03-09T17:18:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8398.313390 secs 2023-03-09T17:18:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8860.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:18:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:09 smithi038 ceph-mon[92422]: pgmap v4993: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 114 KiB/s wr, 26 op/s 2023-03-09T17:18:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:09 smithi122 ceph-mon[93869]: pgmap v4993: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 114 KiB/s wr, 26 op/s 2023-03-09T17:18:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:09 smithi181 ceph-mon[94189]: pgmap v4993: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 114 KiB/s wr, 26 op/s 2023-03-09T17:18:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8401.050809 secs 2023-03-09T17:18:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8401.050809 secs 2023-03-09T17:18:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8401.050809 secs 2023-03-09T17:18:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:11 smithi038 ceph-mon[92422]: pgmap v4994: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 114 KiB/s wr, 26 op/s 2023-03-09T17:18:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:11 smithi122 ceph-mon[93869]: pgmap v4994: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 114 KiB/s wr, 26 op/s 2023-03-09T17:18:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:11 smithi181 ceph-mon[94189]: pgmap v4994: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 114 KiB/s wr, 26 op/s 2023-03-09T17:18:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8403.313474 secs 2023-03-09T17:18:12.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8403.313474 secs 2023-03-09T17:18:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8403.313474 secs 2023-03-09T17:18:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8865.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:18:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:13 smithi122 ceph-mon[93869]: pgmap v4995: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 111 KiB/s wr, 26 op/s 2023-03-09T17:18:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:13 smithi181 ceph-mon[94189]: pgmap v4995: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 111 KiB/s wr, 26 op/s 2023-03-09T17:18:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:13 smithi038 ceph-mon[92422]: pgmap v4995: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 111 KiB/s wr, 26 op/s 2023-03-09T17:18:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:15 smithi122 ceph-mon[93869]: pgmap v4996: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 85 KiB/s wr, 11 op/s 2023-03-09T17:18:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8406.050871 secs 2023-03-09T17:18:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:15 smithi122 ceph-mon[93869]: osdmap e526: 12 total, 12 up, 12 in 2023-03-09T17:18:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:15 smithi181 ceph-mon[94189]: pgmap v4996: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 85 KiB/s wr, 11 op/s 2023-03-09T17:18:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8406.050871 secs 2023-03-09T17:18:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:15 smithi181 ceph-mon[94189]: osdmap e526: 12 total, 12 up, 12 in 2023-03-09T17:18:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:15 smithi038 ceph-mon[92422]: pgmap v4996: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 85 KiB/s wr, 11 op/s 2023-03-09T17:18:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8406.050871 secs 2023-03-09T17:18:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:15 smithi038 ceph-mon[92422]: osdmap e526: 12 total, 12 up, 12 in 2023-03-09T17:18:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:17 smithi122 ceph-mon[93869]: pgmap v4998: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T17:18:17.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8408.313598 secs 2023-03-09T17:18:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:17 smithi181 ceph-mon[94189]: pgmap v4998: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T17:18:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8408.313598 secs 2023-03-09T17:18:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:17 smithi038 ceph-mon[92422]: pgmap v4998: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T17:18:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8408.313598 secs 2023-03-09T17:18:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8870.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:18:19.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:19 smithi122 ceph-mon[93869]: pgmap v4999: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T17:18:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:19 smithi181 ceph-mon[94189]: pgmap v4999: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T17:18:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:19 smithi038 ceph-mon[92422]: pgmap v4999: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T17:18:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8411.050931 secs 2023-03-09T17:18:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8411.050931 secs 2023-03-09T17:18:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8411.050931 secs 2023-03-09T17:18:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:21 smithi122 ceph-mon[93869]: pgmap v5000: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T17:18:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:21 smithi181 ceph-mon[94189]: pgmap v5000: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T17:18:22.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:21 smithi038 ceph-mon[92422]: pgmap v5000: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T17:18:22.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8413.313717 secs 2023-03-09T17:18:22.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8413.313717 secs 2023-03-09T17:18:22.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8413.313717 secs 2023-03-09T17:18:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8875.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:18:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:23 smithi122 ceph-mon[93869]: pgmap v5001: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T17:18:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:23 smithi181 ceph-mon[94189]: pgmap v5001: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T17:18:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:23 smithi038 ceph-mon[92422]: pgmap v5001: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T17:18:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:25 smithi122 ceph-mon[93869]: pgmap v5002: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T17:18:25.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8416.051013 secs 2023-03-09T17:18:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:25 smithi181 ceph-mon[94189]: pgmap v5002: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T17:18:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8416.051013 secs 2023-03-09T17:18:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:25 smithi038 ceph-mon[92422]: pgmap v5002: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T17:18:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8416.051013 secs 2023-03-09T17:18:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:27 smithi122 ceph-mon[93869]: pgmap v5003: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 37 KiB/s wr, 9 op/s 2023-03-09T17:18:27.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8418.313841 secs 2023-03-09T17:18:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:27 smithi181 ceph-mon[94189]: pgmap v5003: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 37 KiB/s wr, 9 op/s 2023-03-09T17:18:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8418.313841 secs 2023-03-09T17:18:28.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:27 smithi038 ceph-mon[92422]: pgmap v5003: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 37 KiB/s wr, 9 op/s 2023-03-09T17:18:28.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8418.313841 secs 2023-03-09T17:18:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8880.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:18:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:29 smithi122 ceph-mon[93869]: pgmap v5004: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 33 KiB/s wr, 7 op/s 2023-03-09T17:18:29.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:29 smithi181 ceph-mon[94189]: pgmap v5004: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 33 KiB/s wr, 7 op/s 2023-03-09T17:18:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:29 smithi038 ceph-mon[92422]: pgmap v5004: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 33 KiB/s wr, 7 op/s 2023-03-09T17:18:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8421.051113 secs 2023-03-09T17:18:30.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8421.051113 secs 2023-03-09T17:18:30.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8421.051113 secs 2023-03-09T17:18:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:31 smithi122 ceph-mon[93869]: pgmap v5005: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 32 KiB/s wr, 6 op/s 2023-03-09T17:18:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:31 smithi181 ceph-mon[94189]: pgmap v5005: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 32 KiB/s wr, 6 op/s 2023-03-09T17:18:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:31 smithi038 ceph-mon[92422]: pgmap v5005: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 32 KiB/s wr, 6 op/s 2023-03-09T17:18:32.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8423.313964 secs 2023-03-09T17:18:32.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8423.313964 secs 2023-03-09T17:18:32.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8423.313964 secs 2023-03-09T17:18:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8885.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:18:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:33 smithi122 ceph-mon[93869]: pgmap v5006: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 52 KiB/s wr, 9 op/s 2023-03-09T17:18:33.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:18:33.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:18:33.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:18:33.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:18:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:33 smithi181 ceph-mon[94189]: pgmap v5006: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 52 KiB/s wr, 9 op/s 2023-03-09T17:18:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:18:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:18:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:18:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:18:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:33 smithi038 ceph-mon[92422]: pgmap v5006: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 52 KiB/s wr, 9 op/s 2023-03-09T17:18:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:18:34.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:18:34.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:18:34.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:18:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:35 smithi122 ceph-mon[93869]: pgmap v5007: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 40 KiB/s wr, 7 op/s 2023-03-09T17:18:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8426.051173 secs 2023-03-09T17:18:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:35 smithi181 ceph-mon[94189]: pgmap v5007: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 40 KiB/s wr, 7 op/s 2023-03-09T17:18:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8426.051173 secs 2023-03-09T17:18:36.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:35 smithi038 ceph-mon[92422]: pgmap v5007: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 40 KiB/s wr, 7 op/s 2023-03-09T17:18:36.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8426.051173 secs 2023-03-09T17:18:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:37 smithi122 ceph-mon[93869]: pgmap v5008: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 66 KiB/s wr, 10 op/s 2023-03-09T17:18:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:37 smithi181 ceph-mon[94189]: pgmap v5008: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 66 KiB/s wr, 10 op/s 2023-03-09T17:18:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:37 smithi038 ceph-mon[92422]: pgmap v5008: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 66 KiB/s wr, 10 op/s 2023-03-09T17:18:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8890.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:18:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:39 smithi122 ceph-mon[93869]: pgmap v5009: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 54 KiB/s wr, 8 op/s 2023-03-09T17:18:39.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:39 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:18:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:39 smithi181 ceph-mon[94189]: pgmap v5009: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 54 KiB/s wr, 8 op/s 2023-03-09T17:18:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:39 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:18:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:39 smithi038 ceph-mon[92422]: pgmap v5009: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 54 KiB/s wr, 8 op/s 2023-03-09T17:18:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:39 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:18:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:40 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:18:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:18:40.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:18:40.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8431.051232 secs 2023-03-09T17:18:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:40 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:18:40.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:18:40.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:18:40.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8431.051232 secs 2023-03-09T17:18:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:40 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:18:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:18:41.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:18:41.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8431.051232 secs 2023-03-09T17:18:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:41 smithi122 ceph-mon[93869]: pgmap v5010: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 54 KiB/s wr, 8 op/s 2023-03-09T17:18:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:41 smithi181 ceph-mon[94189]: pgmap v5010: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 54 KiB/s wr, 8 op/s 2023-03-09T17:18:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:41 smithi038 ceph-mon[92422]: pgmap v5010: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 54 KiB/s wr, 8 op/s 2023-03-09T17:18:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8895.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:18:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:43 smithi122 ceph-mon[93869]: pgmap v5011: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 87 KiB/s wr, 12 op/s 2023-03-09T17:18:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:43 smithi181 ceph-mon[94189]: pgmap v5011: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 87 KiB/s wr, 12 op/s 2023-03-09T17:18:44.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:43 smithi038 ceph-mon[92422]: pgmap v5011: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 87 KiB/s wr, 12 op/s 2023-03-09T17:18:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:45 smithi122 ceph-mon[93869]: pgmap v5012: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 74 KiB/s wr, 11 op/s 2023-03-09T17:18:45.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:45 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:18:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:45 smithi181 ceph-mon[94189]: pgmap v5012: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 74 KiB/s wr, 11 op/s 2023-03-09T17:18:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:45 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:18:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:45 smithi038 ceph-mon[92422]: pgmap v5012: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 74 KiB/s wr, 11 op/s 2023-03-09T17:18:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:45 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:18:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:46 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:18:46.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:46 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:18:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:46 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:18:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:47 smithi122 ceph-mon[93869]: pgmap v5013: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 151 KiB/s wr, 18 op/s 2023-03-09T17:18:47.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:47 smithi122 ceph-mon[93869]: Health check cleared: MDS_SLOW_REQUEST (was: 1 MDSs report slow requests) 2023-03-09T17:18:47.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:47 smithi122 ceph-mon[93869]: Cluster is now healthy 2023-03-09T17:18:47.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] up:active 2023-03-09T17:18:47.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:18:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:47 smithi181 ceph-mon[94189]: pgmap v5013: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 151 KiB/s wr, 18 op/s 2023-03-09T17:18:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:47 smithi181 ceph-mon[94189]: Health check cleared: MDS_SLOW_REQUEST (was: 1 MDSs report slow requests) 2023-03-09T17:18:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:47 smithi181 ceph-mon[94189]: Cluster is now healthy 2023-03-09T17:18:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] up:active 2023-03-09T17:18:47.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:18:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:47 smithi038 ceph-mon[92422]: pgmap v5013: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 151 KiB/s wr, 18 op/s 2023-03-09T17:18:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:47 smithi038 ceph-mon[92422]: Health check cleared: MDS_SLOW_REQUEST (was: 1 MDSs report slow requests) 2023-03-09T17:18:48.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:47 smithi038 ceph-mon[92422]: Cluster is now healthy 2023-03-09T17:18:48.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] up:active 2023-03-09T17:18:48.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:18:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8900.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:18:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:49 smithi122 ceph-mon[93869]: pgmap v5014: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 135 KiB/s wr, 16 op/s 2023-03-09T17:18:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:49 smithi181 ceph-mon[94189]: pgmap v5014: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 135 KiB/s wr, 16 op/s 2023-03-09T17:18:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:49 smithi038 ceph-mon[92422]: pgmap v5014: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 135 KiB/s wr, 16 op/s 2023-03-09T17:18:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8441.051384 secs 2023-03-09T17:18:50.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8441.051384 secs 2023-03-09T17:18:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8441.051384 secs 2023-03-09T17:18:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:51 smithi122 ceph-mon[93869]: pgmap v5015: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 127 KiB/s wr, 15 op/s 2023-03-09T17:18:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:51 smithi122 ceph-mon[93869]: Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:18:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] up:active 2023-03-09T17:18:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:18:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:51 smithi181 ceph-mon[94189]: pgmap v5015: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 127 KiB/s wr, 15 op/s 2023-03-09T17:18:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:51 smithi181 ceph-mon[94189]: Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:18:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] up:active 2023-03-09T17:18:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:18:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:51 smithi038 ceph-mon[92422]: pgmap v5015: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 127 KiB/s wr, 15 op/s 2023-03-09T17:18:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:51 smithi038 ceph-mon[92422]: Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:18:52.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] up:active 2023-03-09T17:18:52.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:18:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8443.314411 secs 2023-03-09T17:18:52.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8443.314411 secs 2023-03-09T17:18:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8443.314411 secs 2023-03-09T17:18:53.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8905.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:18:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:53 smithi122 ceph-mon[93869]: pgmap v5016: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 161 KiB/s wr, 19 op/s 2023-03-09T17:18:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:53 smithi181 ceph-mon[94189]: pgmap v5016: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 161 KiB/s wr, 19 op/s 2023-03-09T17:18:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:53 smithi038 ceph-mon[92422]: pgmap v5016: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 161 KiB/s wr, 19 op/s 2023-03-09T17:18:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:55 smithi122 ceph-mon[93869]: pgmap v5017: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 147 KiB/s wr, 16 op/s 2023-03-09T17:18:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:18:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:55 smithi181 ceph-mon[94189]: pgmap v5017: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 147 KiB/s wr, 16 op/s 2023-03-09T17:18:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:18:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:55 smithi038 ceph-mon[92422]: pgmap v5017: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 147 KiB/s wr, 16 op/s 2023-03-09T17:18:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:18:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:56 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:18:56.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:18:56.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:18:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:56 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:18:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:18:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:18:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:56 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:18:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:18:57.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:18:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:57 smithi122 ceph-mon[93869]: pgmap v5018: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 237 KiB/s wr, 21 op/s 2023-03-09T17:18:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:57 smithi181 ceph-mon[94189]: pgmap v5018: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 237 KiB/s wr, 21 op/s 2023-03-09T17:18:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:57 smithi038 ceph-mon[92422]: pgmap v5018: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 237 KiB/s wr, 21 op/s 2023-03-09T17:18:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8910.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:18:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:58 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:18:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:58 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:18:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:58 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:18:59.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:58 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:18:59.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:58 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:18:59.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:58 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:18:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:59 smithi038 ceph-mon[92422]: pgmap v5019: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 164 KiB/s wr, 15 op/s 2023-03-09T17:19:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] up:active 2023-03-09T17:19:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:19:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:18:59 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:19:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:59 smithi122 ceph-mon[93869]: pgmap v5019: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 164 KiB/s wr, 15 op/s 2023-03-09T17:19:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] up:active 2023-03-09T17:19:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:19:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:18:59 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:19:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:59 smithi181 ceph-mon[94189]: pgmap v5019: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 164 KiB/s wr, 15 op/s 2023-03-09T17:19:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] up:active 2023-03-09T17:19:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:19:00.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:18:59 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:19:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8451.051505 secs 2023-03-09T17:19:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:00 smithi038 ceph-mon[92422]: Health check cleared: MDS_SLOW_REQUEST (was: 1 MDSs report slow requests) 2023-03-09T17:19:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:00 smithi038 ceph-mon[92422]: Cluster is now healthy 2023-03-09T17:19:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:19:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:19:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8451.051505 secs 2023-03-09T17:19:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:00 smithi122 ceph-mon[93869]: Health check cleared: MDS_SLOW_REQUEST (was: 1 MDSs report slow requests) 2023-03-09T17:19:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:00 smithi122 ceph-mon[93869]: Cluster is now healthy 2023-03-09T17:19:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:19:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:19:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8451.051505 secs 2023-03-09T17:19:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:00 smithi181 ceph-mon[94189]: Health check cleared: MDS_SLOW_REQUEST (was: 1 MDSs report slow requests) 2023-03-09T17:19:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:00 smithi181 ceph-mon[94189]: Cluster is now healthy 2023-03-09T17:19:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:19:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:19:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:01 smithi038 ceph-mon[92422]: pgmap v5020: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 154 KiB/s wr, 14 op/s 2023-03-09T17:19:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:01 smithi122 ceph-mon[93869]: pgmap v5020: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 154 KiB/s wr, 14 op/s 2023-03-09T17:19:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:01 smithi181 ceph-mon[94189]: pgmap v5020: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 154 KiB/s wr, 14 op/s 2023-03-09T17:19:03.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8453.314649 secs 2023-03-09T17:19:03.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8453.314649 secs 2023-03-09T17:19:03.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8453.314649 secs 2023-03-09T17:19:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8915.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:19:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:03 smithi122 ceph-mon[93869]: pgmap v5021: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 178 KiB/s wr, 17 op/s 2023-03-09T17:19:04.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:03 smithi122 ceph-mon[93869]: Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:19:04.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:03 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] up:active 2023-03-09T17:19:04.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:03 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:19:04.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:03 smithi181 ceph-mon[94189]: pgmap v5021: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 178 KiB/s wr, 17 op/s 2023-03-09T17:19:04.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:03 smithi181 ceph-mon[94189]: Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:19:04.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:03 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] up:active 2023-03-09T17:19:04.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:03 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:19:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:03 smithi038 ceph-mon[92422]: pgmap v5021: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 178 KiB/s wr, 17 op/s 2023-03-09T17:19:04.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:03 smithi038 ceph-mon[92422]: Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:19:04.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:03 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] up:active 2023-03-09T17:19:04.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:03 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:19:05.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:04 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:19:05.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:04 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:19:05.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:04 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:19:05.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:04 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:19:05.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:04 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:19:05.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:04 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:19:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:05 smithi122 ceph-mon[93869]: pgmap v5022: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 178 KiB/s wr, 27 op/s 2023-03-09T17:19:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:05 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:19:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:05 smithi181 ceph-mon[94189]: pgmap v5022: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 178 KiB/s wr, 27 op/s 2023-03-09T17:19:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:05 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:19:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:05 smithi038 ceph-mon[92422]: pgmap v5022: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 178 KiB/s wr, 27 op/s 2023-03-09T17:19:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:05 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:19:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:07 smithi122 ceph-mon[93869]: pgmap v5023: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 203 KiB/s wr, 32 op/s 2023-03-09T17:19:07.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] up:active 2023-03-09T17:19:07.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:19:07.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8458.314772 secs 2023-03-09T17:19:07.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:07 smithi181 ceph-mon[94189]: pgmap v5023: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 203 KiB/s wr, 32 op/s 2023-03-09T17:19:07.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] up:active 2023-03-09T17:19:07.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:19:07.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8458.314772 secs 2023-03-09T17:19:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:07 smithi038 ceph-mon[92422]: pgmap v5023: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 203 KiB/s wr, 32 op/s 2023-03-09T17:19:07.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] up:active 2023-03-09T17:19:07.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:19:07.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8458.314772 secs 2023-03-09T17:19:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8920.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:19:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:09 smithi122 ceph-mon[93869]: pgmap v5024: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 118 KiB/s wr, 28 op/s 2023-03-09T17:19:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:09 smithi181 ceph-mon[94189]: pgmap v5024: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 118 KiB/s wr, 28 op/s 2023-03-09T17:19:09.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:09 smithi038 ceph-mon[92422]: pgmap v5024: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 118 KiB/s wr, 28 op/s 2023-03-09T17:19:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:10 smithi038 ceph-mon[92422]: osdmap e527: 12 total, 12 up, 12 in 2023-03-09T17:19:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:10 smithi122 ceph-mon[93869]: osdmap e527: 12 total, 12 up, 12 in 2023-03-09T17:19:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:10 smithi181 ceph-mon[94189]: osdmap e527: 12 total, 12 up, 12 in 2023-03-09T17:19:11.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:11 smithi038 ceph-mon[92422]: pgmap v5026: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 138 KiB/s wr, 31 op/s 2023-03-09T17:19:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:11 smithi122 ceph-mon[93869]: pgmap v5026: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 138 KiB/s wr, 31 op/s 2023-03-09T17:19:11.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:11 smithi181 ceph-mon[94189]: pgmap v5026: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 138 KiB/s wr, 31 op/s 2023-03-09T17:19:12.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8463.314835 secs 2023-03-09T17:19:12.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8463.314835 secs 2023-03-09T17:19:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8463.314835 secs 2023-03-09T17:19:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8925.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:19:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:13 smithi122 ceph-mon[93869]: pgmap v5027: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 156 KiB/s wr, 32 op/s 2023-03-09T17:19:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:13 smithi181 ceph-mon[94189]: pgmap v5027: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 156 KiB/s wr, 32 op/s 2023-03-09T17:19:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:13 smithi038 ceph-mon[92422]: pgmap v5027: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 156 KiB/s wr, 32 op/s 2023-03-09T17:19:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:15 smithi122 ceph-mon[93869]: pgmap v5028: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 116 KiB/s wr, 17 op/s 2023-03-09T17:19:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8466.051715 secs 2023-03-09T17:19:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:15 smithi181 ceph-mon[94189]: pgmap v5028: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 116 KiB/s wr, 17 op/s 2023-03-09T17:19:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8466.051715 secs 2023-03-09T17:19:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:15 smithi038 ceph-mon[92422]: pgmap v5028: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 116 KiB/s wr, 17 op/s 2023-03-09T17:19:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8466.051715 secs 2023-03-09T17:19:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:17 smithi122 ceph-mon[93869]: pgmap v5029: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 113 KiB/s wr, 14 op/s 2023-03-09T17:19:17.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8468.314959 secs 2023-03-09T17:19:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:17 smithi181 ceph-mon[94189]: pgmap v5029: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 113 KiB/s wr, 14 op/s 2023-03-09T17:19:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8468.314959 secs 2023-03-09T17:19:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:17 smithi038 ceph-mon[92422]: pgmap v5029: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 113 KiB/s wr, 14 op/s 2023-03-09T17:19:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8468.314959 secs 2023-03-09T17:19:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8930.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:19:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:18 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:19:18.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:18 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] up:active 2023-03-09T17:19:18.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:18 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:19:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:18 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:19:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:18 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] up:active 2023-03-09T17:19:18.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:18 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:19:19.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:18 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:19:19.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:18 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] up:active 2023-03-09T17:19:19.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:18 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:19:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:19 smithi122 ceph-mon[93869]: pgmap v5030: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 99 KiB/s wr, 14 op/s 2023-03-09T17:19:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:19 smithi181 ceph-mon[94189]: pgmap v5030: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 99 KiB/s wr, 14 op/s 2023-03-09T17:19:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:19 smithi038 ceph-mon[92422]: pgmap v5030: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 99 KiB/s wr, 14 op/s 2023-03-09T17:19:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:21 smithi122 ceph-mon[93869]: pgmap v5031: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 93 KiB/s wr, 13 op/s 2023-03-09T17:19:21.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:21 smithi122 ceph-mon[93869]: osdmap e528: 12 total, 12 up, 12 in 2023-03-09T17:19:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:21 smithi181 ceph-mon[94189]: pgmap v5031: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 93 KiB/s wr, 13 op/s 2023-03-09T17:19:21.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:21 smithi181 ceph-mon[94189]: osdmap e528: 12 total, 12 up, 12 in 2023-03-09T17:19:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:21 smithi038 ceph-mon[92422]: pgmap v5031: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 93 KiB/s wr, 13 op/s 2023-03-09T17:19:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:21 smithi038 ceph-mon[92422]: osdmap e528: 12 total, 12 up, 12 in 2023-03-09T17:19:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:22 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:19:22.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8473.315090 secs 2023-03-09T17:19:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:22 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:19:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8473.315090 secs 2023-03-09T17:19:22.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:22 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:19:22.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8473.315090 secs 2023-03-09T17:19:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:23 smithi122 ceph-mon[93869]: pgmap v5033: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 94 KiB/s wr, 13 op/s 2023-03-09T17:19:23.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:23 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] up:active 2023-03-09T17:19:23.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:23 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:19:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:23 smithi181 ceph-mon[94189]: pgmap v5033: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 94 KiB/s wr, 13 op/s 2023-03-09T17:19:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:23 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] up:active 2023-03-09T17:19:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:23 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:19:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:23 smithi038 ceph-mon[92422]: pgmap v5033: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 94 KiB/s wr, 13 op/s 2023-03-09T17:19:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:23 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] up:active 2023-03-09T17:19:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:23 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:19:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8935.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:19:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:25 smithi122 ceph-mon[93869]: pgmap v5034: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 94 KiB/s wr, 13 op/s 2023-03-09T17:19:25.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8476.051885 secs 2023-03-09T17:19:25.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:25 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:19:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:25 smithi181 ceph-mon[94189]: pgmap v5034: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 94 KiB/s wr, 13 op/s 2023-03-09T17:19:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8476.051885 secs 2023-03-09T17:19:25.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:25 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:19:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:25 smithi038 ceph-mon[92422]: pgmap v5034: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 94 KiB/s wr, 13 op/s 2023-03-09T17:19:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8476.051885 secs 2023-03-09T17:19:25.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:25 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:19:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:27 smithi038 ceph-mon[92422]: pgmap v5035: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 43 KiB/s wr, 11 op/s 2023-03-09T17:19:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:27 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] up:active 2023-03-09T17:19:27.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:27 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:19:27.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8478.315207 secs 2023-03-09T17:19:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:27 smithi122 ceph-mon[93869]: pgmap v5035: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 43 KiB/s wr, 11 op/s 2023-03-09T17:19:27.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:27 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] up:active 2023-03-09T17:19:27.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:27 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:19:27.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8478.315207 secs 2023-03-09T17:19:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:27 smithi181 ceph-mon[94189]: pgmap v5035: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 43 KiB/s wr, 11 op/s 2023-03-09T17:19:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:27 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] up:active 2023-03-09T17:19:27.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:27 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:19:27.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8478.315207 secs 2023-03-09T17:19:28.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8940.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:19:29.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:29 smithi038 ceph-mon[92422]: pgmap v5036: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 10 op/s 2023-03-09T17:19:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:29 smithi122 ceph-mon[93869]: pgmap v5036: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 10 op/s 2023-03-09T17:19:29.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:29 smithi181 ceph-mon[94189]: pgmap v5036: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 10 op/s 2023-03-09T17:19:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8481.051945 secs 2023-03-09T17:19:30.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:30 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:19:30.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8481.051945 secs 2023-03-09T17:19:30.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:30 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:19:30.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8481.051945 secs 2023-03-09T17:19:31.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:30 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:19:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:31 smithi122 ceph-mon[93869]: pgmap v5037: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 10 op/s 2023-03-09T17:19:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:31 smithi181 ceph-mon[94189]: pgmap v5037: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 10 op/s 2023-03-09T17:19:32.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:31 smithi038 ceph-mon[92422]: pgmap v5037: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 10 op/s 2023-03-09T17:19:32.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:32 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 8483.315319 secs 2023-03-09T17:19:32.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:32 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 8483.315319 secs 2023-03-09T17:19:32.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:32 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 8483.315319 secs 2023-03-09T17:19:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8945.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:19:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:33 smithi122 ceph-mon[93869]: pgmap v5038: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 47 KiB/s wr, 11 op/s 2023-03-09T17:19:33.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:19:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:33 smithi181 ceph-mon[94189]: pgmap v5038: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 47 KiB/s wr, 11 op/s 2023-03-09T17:19:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:19:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:33 smithi038 ceph-mon[92422]: pgmap v5038: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 47 KiB/s wr, 11 op/s 2023-03-09T17:19:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:19:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:35 smithi122 ceph-mon[93869]: pgmap v5039: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 KiB/s wr, 7 op/s 2023-03-09T17:19:35.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8486.052009 secs 2023-03-09T17:19:35.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:19:35.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:19:35.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:19:35.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:19:35.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:19:35.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:19:35.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:19:35.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:19:35.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:19:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:35 smithi181 ceph-mon[94189]: pgmap v5039: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 KiB/s wr, 7 op/s 2023-03-09T17:19:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8486.052009 secs 2023-03-09T17:19:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:19:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:19:35.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:19:35.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:19:35.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:19:35.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:19:35.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:19:35.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:19:35.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:19:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:35 smithi038 ceph-mon[92422]: pgmap v5039: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 KiB/s wr, 7 op/s 2023-03-09T17:19:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8486.052009 secs 2023-03-09T17:19:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:19:36.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:19:36.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:19:36.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:19:36.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:19:36.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:19:36.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:19:36.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:19:36.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:19:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:19:36.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:19:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:19:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:19:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:19:37.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:19:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:37 smithi122 ceph-mon[93869]: pgmap v5040: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 34 KiB/s wr, 9 op/s 2023-03-09T17:19:37.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:37 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 8488.315442 secs 2023-03-09T17:19:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:37 smithi181 ceph-mon[94189]: pgmap v5040: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 34 KiB/s wr, 9 op/s 2023-03-09T17:19:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:37 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 8488.315442 secs 2023-03-09T17:19:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:37 smithi038 ceph-mon[92422]: pgmap v5040: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 34 KiB/s wr, 9 op/s 2023-03-09T17:19:38.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:37 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 8488.315442 secs 2023-03-09T17:19:38.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8950.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:19:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:39 smithi122 ceph-mon[93869]: pgmap v5041: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 34 KiB/s wr, 7 op/s 2023-03-09T17:19:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:39 smithi181 ceph-mon[94189]: pgmap v5041: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 34 KiB/s wr, 7 op/s 2023-03-09T17:19:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:39 smithi038 ceph-mon[92422]: pgmap v5041: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 34 KiB/s wr, 7 op/s 2023-03-09T17:19:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8491.052068 secs 2023-03-09T17:19:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8491.052068 secs 2023-03-09T17:19:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8491.052068 secs 2023-03-09T17:19:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:41 smithi122 ceph-mon[93869]: pgmap v5042: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 34 KiB/s wr, 6 op/s 2023-03-09T17:19:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:41 smithi181 ceph-mon[94189]: pgmap v5042: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 34 KiB/s wr, 6 op/s 2023-03-09T17:19:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:41 smithi038 ceph-mon[92422]: pgmap v5042: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 34 KiB/s wr, 6 op/s 2023-03-09T17:19:42.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:42 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 8493.315564 secs 2023-03-09T17:19:43.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:42 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 8493.315564 secs 2023-03-09T17:19:43.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:42 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 8493.315564 secs 2023-03-09T17:19:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8955.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:19:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:43 smithi038 ceph-mon[92422]: pgmap v5043: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 65 KiB/s wr, 9 op/s 2023-03-09T17:19:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:43 smithi122 ceph-mon[93869]: pgmap v5043: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 65 KiB/s wr, 9 op/s 2023-03-09T17:19:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:43 smithi181 ceph-mon[94189]: pgmap v5043: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 65 KiB/s wr, 9 op/s 2023-03-09T17:19:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:45 smithi038 ceph-mon[92422]: pgmap v5044: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 56 KiB/s wr, 7 op/s 2023-03-09T17:19:46.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8496.052128 secs 2023-03-09T17:19:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:45 smithi122 ceph-mon[93869]: pgmap v5044: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 56 KiB/s wr, 7 op/s 2023-03-09T17:19:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8496.052128 secs 2023-03-09T17:19:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:45 smithi181 ceph-mon[94189]: pgmap v5044: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 56 KiB/s wr, 7 op/s 2023-03-09T17:19:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8496.052128 secs 2023-03-09T17:19:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:47 smithi122 ceph-mon[93869]: pgmap v5045: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 102 KiB/s wr, 9 op/s 2023-03-09T17:19:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8498.315705 secs 2023-03-09T17:19:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:47 smithi181 ceph-mon[94189]: pgmap v5045: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 102 KiB/s wr, 9 op/s 2023-03-09T17:19:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8498.315705 secs 2023-03-09T17:19:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:47 smithi038 ceph-mon[92422]: pgmap v5045: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 102 KiB/s wr, 9 op/s 2023-03-09T17:19:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8498.315705 secs 2023-03-09T17:19:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8960.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:19:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:19:49.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:19:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:19:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:19:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:19:49.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:19:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:49 smithi122 ceph-mon[93869]: pgmap v5046: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 93 KiB/s wr, 8 op/s 2023-03-09T17:19:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:49 smithi181 ceph-mon[94189]: pgmap v5046: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 93 KiB/s wr, 8 op/s 2023-03-09T17:19:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:49 smithi038 ceph-mon[92422]: pgmap v5046: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 93 KiB/s wr, 8 op/s 2023-03-09T17:19:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8501.052188 secs 2023-03-09T17:19:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8501.052188 secs 2023-03-09T17:19:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8501.052188 secs 2023-03-09T17:19:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:51 smithi122 ceph-mon[93869]: pgmap v5047: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 93 KiB/s wr, 8 op/s 2023-03-09T17:19:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:51 smithi181 ceph-mon[94189]: pgmap v5047: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 93 KiB/s wr, 8 op/s 2023-03-09T17:19:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:51 smithi038 ceph-mon[92422]: pgmap v5047: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 93 KiB/s wr, 8 op/s 2023-03-09T17:19:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8503.315822 secs 2023-03-09T17:19:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8503.315822 secs 2023-03-09T17:19:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8503.315822 secs 2023-03-09T17:19:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8965.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:19:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:53 smithi122 ceph-mon[93869]: pgmap v5048: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 141 KiB/s wr, 11 op/s 2023-03-09T17:19:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:53 smithi181 ceph-mon[94189]: pgmap v5048: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 141 KiB/s wr, 11 op/s 2023-03-09T17:19:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:53 smithi038 ceph-mon[92422]: pgmap v5048: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 141 KiB/s wr, 11 op/s 2023-03-09T17:19:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:55 smithi122 ceph-mon[93869]: pgmap v5049: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 118 KiB/s wr, 10 op/s 2023-03-09T17:19:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8506.052247 secs 2023-03-09T17:19:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:55 smithi181 ceph-mon[94189]: pgmap v5049: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 118 KiB/s wr, 10 op/s 2023-03-09T17:19:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8506.052247 secs 2023-03-09T17:19:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:55 smithi038 ceph-mon[92422]: pgmap v5049: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 118 KiB/s wr, 10 op/s 2023-03-09T17:19:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8506.052247 secs 2023-03-09T17:19:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:57 smithi122 ceph-mon[93869]: pgmap v5050: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 116 KiB/s wr, 12 op/s 2023-03-09T17:19:58.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8508.315929 secs 2023-03-09T17:19:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:57 smithi181 ceph-mon[94189]: pgmap v5050: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 116 KiB/s wr, 12 op/s 2023-03-09T17:19:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8508.315929 secs 2023-03-09T17:19:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:57 smithi038 ceph-mon[92422]: pgmap v5050: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 116 KiB/s wr, 12 op/s 2023-03-09T17:19:58.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8508.315929 secs 2023-03-09T17:19:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8970.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:20:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:19:59 smithi122 ceph-mon[93869]: pgmap v5051: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 77 KiB/s wr, 10 op/s 2023-03-09T17:20:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:19:59 smithi181 ceph-mon[94189]: pgmap v5051: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 77 KiB/s wr, 10 op/s 2023-03-09T17:20:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:19:59 smithi038 ceph-mon[92422]: pgmap v5051: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 77 KiB/s wr, 10 op/s 2023-03-09T17:20:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8511.052307 secs 2023-03-09T17:20:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:00 smithi122 ceph-mon[93869]: overall HEALTH_WARN 2 MDSs report slow requests 2023-03-09T17:20:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8511.052307 secs 2023-03-09T17:20:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:00 smithi181 ceph-mon[94189]: overall HEALTH_WARN 2 MDSs report slow requests 2023-03-09T17:20:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8511.052307 secs 2023-03-09T17:20:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:00 smithi038 ceph-mon[92422]: overall HEALTH_WARN 2 MDSs report slow requests 2023-03-09T17:20:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:01 smithi122 ceph-mon[93869]: pgmap v5052: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 69 KiB/s wr, 8 op/s 2023-03-09T17:20:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:01 smithi181 ceph-mon[94189]: pgmap v5052: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 69 KiB/s wr, 8 op/s 2023-03-09T17:20:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:01 smithi038 ceph-mon[92422]: pgmap v5052: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 69 KiB/s wr, 8 op/s 2023-03-09T17:20:03.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8513.316048 secs 2023-03-09T17:20:03.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8513.316048 secs 2023-03-09T17:20:03.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8513.316048 secs 2023-03-09T17:20:03.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8975.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:20:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:03 smithi122 ceph-mon[93869]: pgmap v5053: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 136 KiB/s wr, 12 op/s 2023-03-09T17:20:04.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:03 smithi181 ceph-mon[94189]: pgmap v5053: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 136 KiB/s wr, 12 op/s 2023-03-09T17:20:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:03 smithi038 ceph-mon[92422]: pgmap v5053: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 136 KiB/s wr, 12 op/s 2023-03-09T17:20:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:05 smithi122 ceph-mon[93869]: pgmap v5054: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 124 KiB/s wr, 21 op/s 2023-03-09T17:20:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8516.052368 secs 2023-03-09T17:20:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:05 smithi181 ceph-mon[94189]: pgmap v5054: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 124 KiB/s wr, 21 op/s 2023-03-09T17:20:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8516.052368 secs 2023-03-09T17:20:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:05 smithi038 ceph-mon[92422]: pgmap v5054: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 124 KiB/s wr, 21 op/s 2023-03-09T17:20:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8516.052368 secs 2023-03-09T17:20:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:07 smithi122 ceph-mon[93869]: pgmap v5055: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 152 KiB/s wr, 26 op/s 2023-03-09T17:20:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8518.316166 secs 2023-03-09T17:20:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:07 smithi181 ceph-mon[94189]: pgmap v5055: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 152 KiB/s wr, 26 op/s 2023-03-09T17:20:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8518.316166 secs 2023-03-09T17:20:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:07 smithi038 ceph-mon[92422]: pgmap v5055: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 152 KiB/s wr, 26 op/s 2023-03-09T17:20:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8518.316166 secs 2023-03-09T17:20:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8980.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:20:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:09 smithi122 ceph-mon[93869]: pgmap v5056: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 147 KiB/s wr, 24 op/s 2023-03-09T17:20:10.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:09 smithi122 ceph-mon[93869]: osdmap e529: 12 total, 12 up, 12 in 2023-03-09T17:20:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:09 smithi181 ceph-mon[94189]: pgmap v5056: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 147 KiB/s wr, 24 op/s 2023-03-09T17:20:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:09 smithi181 ceph-mon[94189]: osdmap e529: 12 total, 12 up, 12 in 2023-03-09T17:20:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:09 smithi038 ceph-mon[92422]: pgmap v5056: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 147 KiB/s wr, 24 op/s 2023-03-09T17:20:10.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:09 smithi038 ceph-mon[92422]: osdmap e529: 12 total, 12 up, 12 in 2023-03-09T17:20:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8521.052427 secs 2023-03-09T17:20:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8521.052427 secs 2023-03-09T17:20:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8521.052427 secs 2023-03-09T17:20:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:11 smithi122 ceph-mon[93869]: pgmap v5058: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 167 KiB/s wr, 28 op/s 2023-03-09T17:20:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:11 smithi181 ceph-mon[94189]: pgmap v5058: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 167 KiB/s wr, 28 op/s 2023-03-09T17:20:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:11 smithi038 ceph-mon[92422]: pgmap v5058: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 167 KiB/s wr, 28 op/s 2023-03-09T17:20:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8523.316284 secs 2023-03-09T17:20:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8523.316284 secs 2023-03-09T17:20:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8523.316284 secs 2023-03-09T17:20:13.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8985.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:20:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:13 smithi122 ceph-mon[93869]: pgmap v5059: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 136 KiB/s wr, 26 op/s 2023-03-09T17:20:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:13 smithi181 ceph-mon[94189]: pgmap v5059: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 136 KiB/s wr, 26 op/s 2023-03-09T17:20:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:13 smithi038 ceph-mon[92422]: pgmap v5059: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 136 KiB/s wr, 26 op/s 2023-03-09T17:20:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:15 smithi122 ceph-mon[93869]: pgmap v5060: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 93 KiB/s wr, 11 op/s 2023-03-09T17:20:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8526.052509 secs 2023-03-09T17:20:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:15 smithi122 ceph-mon[93869]: osdmap e530: 12 total, 12 up, 12 in 2023-03-09T17:20:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:15 smithi181 ceph-mon[94189]: pgmap v5060: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 93 KiB/s wr, 11 op/s 2023-03-09T17:20:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8526.052509 secs 2023-03-09T17:20:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:15 smithi181 ceph-mon[94189]: osdmap e530: 12 total, 12 up, 12 in 2023-03-09T17:20:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:15 smithi038 ceph-mon[92422]: pgmap v5060: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 93 KiB/s wr, 11 op/s 2023-03-09T17:20:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8526.052509 secs 2023-03-09T17:20:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:15 smithi038 ceph-mon[92422]: osdmap e530: 12 total, 12 up, 12 in 2023-03-09T17:20:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:17 smithi122 ceph-mon[93869]: pgmap v5062: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 80 KiB/s wr, 8 op/s 2023-03-09T17:20:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8528.316365 secs 2023-03-09T17:20:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:17 smithi181 ceph-mon[94189]: pgmap v5062: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 80 KiB/s wr, 8 op/s 2023-03-09T17:20:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8528.316365 secs 2023-03-09T17:20:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:17 smithi038 ceph-mon[92422]: pgmap v5062: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 80 KiB/s wr, 8 op/s 2023-03-09T17:20:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8528.316365 secs 2023-03-09T17:20:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8990.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:20:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:19 smithi122 ceph-mon[93869]: pgmap v5063: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 69 KiB/s wr, 8 op/s 2023-03-09T17:20:20.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:19 smithi181 ceph-mon[94189]: pgmap v5063: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 69 KiB/s wr, 8 op/s 2023-03-09T17:20:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:19 smithi038 ceph-mon[92422]: pgmap v5063: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 69 KiB/s wr, 8 op/s 2023-03-09T17:20:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8531.052570 secs 2023-03-09T17:20:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8531.052570 secs 2023-03-09T17:20:21.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8531.052570 secs 2023-03-09T17:20:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:21 smithi122 ceph-mon[93869]: pgmap v5064: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 65 KiB/s wr, 7 op/s 2023-03-09T17:20:22.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:21 smithi181 ceph-mon[94189]: pgmap v5064: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 65 KiB/s wr, 7 op/s 2023-03-09T17:20:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:21 smithi038 ceph-mon[92422]: pgmap v5064: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 65 KiB/s wr, 7 op/s 2023-03-09T17:20:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8533.316473 secs 2023-03-09T17:20:23.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8533.316473 secs 2023-03-09T17:20:23.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8533.316473 secs 2023-03-09T17:20:23.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 8995.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:20:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:23 smithi122 ceph-mon[93869]: pgmap v5065: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 7 op/s 2023-03-09T17:20:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:23 smithi181 ceph-mon[94189]: pgmap v5065: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 7 op/s 2023-03-09T17:20:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:23 smithi038 ceph-mon[92422]: pgmap v5065: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 7 op/s 2023-03-09T17:20:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:25 smithi122 ceph-mon[93869]: pgmap v5066: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 17 KiB/s wr, 7 op/s 2023-03-09T17:20:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8536.052631 secs 2023-03-09T17:20:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:25 smithi181 ceph-mon[94189]: pgmap v5066: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 17 KiB/s wr, 7 op/s 2023-03-09T17:20:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8536.052631 secs 2023-03-09T17:20:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:25 smithi038 ceph-mon[92422]: pgmap v5066: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 17 KiB/s wr, 7 op/s 2023-03-09T17:20:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8536.052631 secs 2023-03-09T17:20:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:27 smithi122 ceph-mon[93869]: pgmap v5067: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 8 op/s 2023-03-09T17:20:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8538.316622 secs 2023-03-09T17:20:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:27 smithi181 ceph-mon[94189]: pgmap v5067: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 8 op/s 2023-03-09T17:20:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8538.316622 secs 2023-03-09T17:20:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:27 smithi038 ceph-mon[92422]: pgmap v5067: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 8 op/s 2023-03-09T17:20:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8538.316622 secs 2023-03-09T17:20:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9000.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:20:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:29 smithi122 ceph-mon[93869]: pgmap v5068: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 21 KiB/s wr, 7 op/s 2023-03-09T17:20:30.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:29 smithi181 ceph-mon[94189]: pgmap v5068: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 21 KiB/s wr, 7 op/s 2023-03-09T17:20:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:29 smithi038 ceph-mon[92422]: pgmap v5068: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 21 KiB/s wr, 7 op/s 2023-03-09T17:20:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8541.052690 secs 2023-03-09T17:20:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8541.052690 secs 2023-03-09T17:20:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8541.052690 secs 2023-03-09T17:20:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:31 smithi122 ceph-mon[93869]: pgmap v5069: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 21 KiB/s wr, 6 op/s 2023-03-09T17:20:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:31 smithi181 ceph-mon[94189]: pgmap v5069: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 21 KiB/s wr, 6 op/s 2023-03-09T17:20:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:31 smithi038 ceph-mon[92422]: pgmap v5069: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 21 KiB/s wr, 6 op/s 2023-03-09T17:20:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8543.316752 secs 2023-03-09T17:20:33.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8543.316752 secs 2023-03-09T17:20:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8543.316752 secs 2023-03-09T17:20:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9005.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:20:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:33 smithi122 ceph-mon[93869]: pgmap v5070: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 21 KiB/s wr, 8 op/s 2023-03-09T17:20:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:33 smithi181 ceph-mon[94189]: pgmap v5070: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 21 KiB/s wr, 8 op/s 2023-03-09T17:20:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:33 smithi038 ceph-mon[92422]: pgmap v5070: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 21 KiB/s wr, 8 op/s 2023-03-09T17:20:35.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:34 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:20:35.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:34 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:20:35.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:34 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:20:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:35 smithi122 ceph-mon[93869]: pgmap v5071: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 21 KiB/s wr, 6 op/s 2023-03-09T17:20:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8546.052777 secs 2023-03-09T17:20:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:35 smithi181 ceph-mon[94189]: pgmap v5071: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 21 KiB/s wr, 6 op/s 2023-03-09T17:20:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8546.052777 secs 2023-03-09T17:20:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:35 smithi038 ceph-mon[92422]: pgmap v5071: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 21 KiB/s wr, 6 op/s 2023-03-09T17:20:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8546.052777 secs 2023-03-09T17:20:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:37 smithi122 ceph-mon[93869]: pgmap v5072: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 43 KiB/s wr, 8 op/s 2023-03-09T17:20:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8548.316843 secs 2023-03-09T17:20:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:20:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:20:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:37 smithi181 ceph-mon[94189]: pgmap v5072: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 43 KiB/s wr, 8 op/s 2023-03-09T17:20:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8548.316843 secs 2023-03-09T17:20:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:20:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:20:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:37 smithi038 ceph-mon[92422]: pgmap v5072: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 43 KiB/s wr, 8 op/s 2023-03-09T17:20:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8548.316843 secs 2023-03-09T17:20:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:20:38.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:20:38.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9010.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:20:39.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:38 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:20:39.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:38 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:20:39.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:38 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:20:39.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:38 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:20:39.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:38 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:20:39.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:38 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:20:39.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:38 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:20:39.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:38 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:20:39.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:38 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:20:39.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:38 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:20:39.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:38 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:20:39.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:38 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:20:39.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:38 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:20:39.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:38 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:20:39.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:38 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:20:39.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:38 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:20:39.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:38 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:20:39.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:38 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:20:39.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:38 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:20:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:38 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:20:39.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:38 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:20:39.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:38 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:20:39.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:38 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:20:39.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:38 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:20:39.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:38 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:20:39.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:38 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:20:39.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:38 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:20:39.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:38 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:20:39.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:38 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:20:39.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:38 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:20:39.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:38 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:20:39.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:38 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:20:39.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:38 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:20:39.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:38 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:20:39.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:38 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:20:39.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:38 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:20:39.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:38 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:20:39.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:38 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:20:39.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:38 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:20:39.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:38 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:20:39.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:38 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:20:39.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:38 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:20:39.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:38 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:20:39.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:38 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:20:39.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:38 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:20:39.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:38 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:20:39.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:38 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:20:39.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:38 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:20:39.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:38 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:20:39.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:38 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:20:39.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:38 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:20:39.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:38 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:20:39.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:38 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:20:39.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:38 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:20:39.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:38 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:20:39.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:38 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:20:39.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:38 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:20:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:40 smithi122 ceph-mon[93869]: pgmap v5073: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 50 KiB/s wr, 8 op/s 2023-03-09T17:20:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:40 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T17:20:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:40 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T17:20:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:40 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T17:20:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:40 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi038 to 197.2M 2023-03-09T17:20:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:40 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T17:20:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:40 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi038 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-03-09T17:20:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:40 smithi181 ceph-mon[94189]: pgmap v5073: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 50 KiB/s wr, 8 op/s 2023-03-09T17:20:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:40 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T17:20:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:40 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T17:20:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:40 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T17:20:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:40 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi038 to 197.2M 2023-03-09T17:20:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:40 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T17:20:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:40 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi038 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-03-09T17:20:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:40 smithi038 ceph-mon[92422]: pgmap v5073: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 50 KiB/s wr, 8 op/s 2023-03-09T17:20:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:40 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T17:20:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:40 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T17:20:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:40 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T17:20:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:40 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi038 to 197.2M 2023-03-09T17:20:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:40 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T17:20:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:40 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi038 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-03-09T17:20:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8551.052839 secs 2023-03-09T17:20:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8551.052839 secs 2023-03-09T17:20:41.525 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8551.052839 secs 2023-03-09T17:20:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:42 smithi122 ceph-mon[93869]: pgmap v5074: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 49 KiB/s wr, 7 op/s 2023-03-09T17:20:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:42 smithi181 ceph-mon[94189]: pgmap v5074: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 49 KiB/s wr, 7 op/s 2023-03-09T17:20:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:42 smithi038 ceph-mon[92422]: pgmap v5074: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 49 KiB/s wr, 7 op/s 2023-03-09T17:20:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8553.316966 secs 2023-03-09T17:20:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8553.316966 secs 2023-03-09T17:20:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8553.316966 secs 2023-03-09T17:20:43.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9015.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:20:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:44 smithi122 ceph-mon[93869]: pgmap v5075: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 99 KiB/s wr, 11 op/s 2023-03-09T17:20:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:44 smithi181 ceph-mon[94189]: pgmap v5075: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 99 KiB/s wr, 11 op/s 2023-03-09T17:20:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:44 smithi038 ceph-mon[92422]: pgmap v5075: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 99 KiB/s wr, 11 op/s 2023-03-09T17:20:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:45 smithi122 ceph-mon[93869]: pgmap v5076: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 106 KiB/s wr, 10 op/s 2023-03-09T17:20:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:45 smithi181 ceph-mon[94189]: pgmap v5076: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 106 KiB/s wr, 10 op/s 2023-03-09T17:20:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:45 smithi038 ceph-mon[92422]: pgmap v5076: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 106 KiB/s wr, 10 op/s 2023-03-09T17:20:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8556.052896 secs 2023-03-09T17:20:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8556.052896 secs 2023-03-09T17:20:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8556.052896 secs 2023-03-09T17:20:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:47 smithi122 ceph-mon[93869]: pgmap v5077: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 107 KiB/s wr, 12 op/s 2023-03-09T17:20:47.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:47 smithi181 ceph-mon[94189]: pgmap v5077: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 107 KiB/s wr, 12 op/s 2023-03-09T17:20:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:47 smithi038 ceph-mon[92422]: pgmap v5077: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 107 KiB/s wr, 12 op/s 2023-03-09T17:20:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:48 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:20:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:48 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:20:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:48 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:20:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9020.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:20:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:49 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:20:49.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:20:49.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:20:49.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:49 smithi122 ceph-mon[93869]: pgmap v5078: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 109 KiB/s wr, 11 op/s 2023-03-09T17:20:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:49 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:20:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:20:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:20:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:49 smithi181 ceph-mon[94189]: pgmap v5078: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 109 KiB/s wr, 11 op/s 2023-03-09T17:20:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:49 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:20:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:20:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:20:49.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:49 smithi038 ceph-mon[92422]: pgmap v5078: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 109 KiB/s wr, 11 op/s 2023-03-09T17:20:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8561.052957 secs 2023-03-09T17:20:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8561.052957 secs 2023-03-09T17:20:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8561.052957 secs 2023-03-09T17:20:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:51 smithi038 ceph-mon[92422]: pgmap v5079: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 82 KiB/s wr, 10 op/s 2023-03-09T17:20:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:51 smithi122 ceph-mon[93869]: pgmap v5079: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 82 KiB/s wr, 10 op/s 2023-03-09T17:20:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:51 smithi181 ceph-mon[94189]: pgmap v5079: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 82 KiB/s wr, 10 op/s 2023-03-09T17:20:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:53 smithi038 ceph-mon[92422]: pgmap v5080: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 142 KiB/s wr, 13 op/s 2023-03-09T17:20:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9025.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:20:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:53 smithi122 ceph-mon[93869]: pgmap v5080: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 142 KiB/s wr, 13 op/s 2023-03-09T17:20:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:53 smithi181 ceph-mon[94189]: pgmap v5080: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 142 KiB/s wr, 13 op/s 2023-03-09T17:20:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:55 smithi122 ceph-mon[93869]: pgmap v5081: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 107 KiB/s wr, 10 op/s 2023-03-09T17:20:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8566.052983 secs 2023-03-09T17:20:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:55 smithi181 ceph-mon[94189]: pgmap v5081: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 107 KiB/s wr, 10 op/s 2023-03-09T17:20:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8566.052983 secs 2023-03-09T17:20:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:55 smithi038 ceph-mon[92422]: pgmap v5081: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 107 KiB/s wr, 10 op/s 2023-03-09T17:20:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8566.052983 secs 2023-03-09T17:20:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:57 smithi122 ceph-mon[93869]: pgmap v5082: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 100 KiB/s wr, 11 op/s 2023-03-09T17:20:57.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8568.317314 secs 2023-03-09T17:20:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:57 smithi181 ceph-mon[94189]: pgmap v5082: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 100 KiB/s wr, 11 op/s 2023-03-09T17:20:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8568.317314 secs 2023-03-09T17:20:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:57 smithi038 ceph-mon[92422]: pgmap v5082: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 100 KiB/s wr, 11 op/s 2023-03-09T17:20:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8568.317314 secs 2023-03-09T17:20:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9030.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:20:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:20:59 smithi122 ceph-mon[93869]: pgmap v5083: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 111 KiB/s wr, 11 op/s 2023-03-09T17:20:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:20:59 smithi181 ceph-mon[94189]: pgmap v5083: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 111 KiB/s wr, 11 op/s 2023-03-09T17:20:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:20:59 smithi038 ceph-mon[92422]: pgmap v5083: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 111 KiB/s wr, 11 op/s 2023-03-09T17:21:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:21:00 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:21:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:21:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:21:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:21:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:21:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:21:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8571.053042 secs 2023-03-09T17:21:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:21:00 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:21:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:21:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:21:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:21:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:21:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:21:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8571.053042 secs 2023-03-09T17:21:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:21:00 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:21:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:21:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:21:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:21:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:21:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:21:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8571.053042 secs 2023-03-09T17:21:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:21:01 smithi122 ceph-mon[93869]: pgmap v5084: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 87 KiB/s wr, 8 op/s 2023-03-09T17:21:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:21:01 smithi181 ceph-mon[94189]: pgmap v5084: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 87 KiB/s wr, 8 op/s 2023-03-09T17:21:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:21:01 smithi038 ceph-mon[92422]: pgmap v5084: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 87 KiB/s wr, 8 op/s 2023-03-09T17:21:02.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:21:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8573.317436 secs 2023-03-09T17:21:02.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:21:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8573.317436 secs 2023-03-09T17:21:02.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:21:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8573.317436 secs 2023-03-09T17:21:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9035.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:21:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:21:03 smithi122 ceph-mon[93869]: pgmap v5085: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 102 KiB/s wr, 11 op/s 2023-03-09T17:21:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:21:03 smithi181 ceph-mon[94189]: pgmap v5085: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 102 KiB/s wr, 11 op/s 2023-03-09T17:21:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:21:03 smithi038 ceph-mon[92422]: pgmap v5085: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 102 KiB/s wr, 11 op/s 2023-03-09T17:21:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:21:05 smithi122 ceph-mon[93869]: pgmap v5086: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 64 KiB/s wr, 20 op/s 2023-03-09T17:21:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:21:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8576.053101 secs 2023-03-09T17:21:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:21:05 smithi181 ceph-mon[94189]: pgmap v5086: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 64 KiB/s wr, 20 op/s 2023-03-09T17:21:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:21:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8576.053101 secs 2023-03-09T17:21:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:21:05 smithi038 ceph-mon[92422]: pgmap v5086: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 64 KiB/s wr, 20 op/s 2023-03-09T17:21:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:21:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8576.053101 secs 2023-03-09T17:21:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:21:07 smithi122 ceph-mon[93869]: pgmap v5087: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 71 KiB/s wr, 23 op/s 2023-03-09T17:21:07.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:21:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8578.317572 secs 2023-03-09T17:21:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:21:07 smithi181 ceph-mon[94189]: pgmap v5087: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 71 KiB/s wr, 23 op/s 2023-03-09T17:21:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:21:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8578.317572 secs 2023-03-09T17:21:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:21:07 smithi038 ceph-mon[92422]: pgmap v5087: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 71 KiB/s wr, 23 op/s 2023-03-09T17:21:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:21:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8578.317572 secs 2023-03-09T17:21:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9040.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:21:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:21:09 smithi122 ceph-mon[93869]: pgmap v5088: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 71 KiB/s wr, 22 op/s 2023-03-09T17:21:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:21:09 smithi181 ceph-mon[94189]: pgmap v5088: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 71 KiB/s wr, 22 op/s 2023-03-09T17:21:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:21:09 smithi038 ceph-mon[92422]: pgmap v5088: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 71 KiB/s wr, 22 op/s 2023-03-09T17:21:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:21:10 smithi122 ceph-mon[93869]: osdmap e531: 12 total, 12 up, 12 in 2023-03-09T17:21:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:21:10 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:21:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:21:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8581.053228 secs 2023-03-09T17:21:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:21:10 smithi181 ceph-mon[94189]: osdmap e531: 12 total, 12 up, 12 in 2023-03-09T17:21:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:21:10 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:21:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:21:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8581.053228 secs 2023-03-09T17:21:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:21:10 smithi038 ceph-mon[92422]: osdmap e531: 12 total, 12 up, 12 in 2023-03-09T17:21:11.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:21:10 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:21:11.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:21:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8581.053228 secs 2023-03-09T17:21:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:21:11 smithi122 ceph-mon[93869]: pgmap v5090: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 71 KiB/s wr, 24 op/s 2023-03-09T17:21:11.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:21:11 smithi181 ceph-mon[94189]: pgmap v5090: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 71 KiB/s wr, 24 op/s 2023-03-09T17:21:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:21:11 smithi038 ceph-mon[92422]: pgmap v5090: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 71 KiB/s wr, 24 op/s 2023-03-09T17:21:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:21:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8583.317710 secs 2023-03-09T17:21:12.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:21:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8583.317710 secs 2023-03-09T17:21:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:21:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8583.317710 secs 2023-03-09T17:21:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9045.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:21:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:21:13 smithi122 ceph-mon[93869]: pgmap v5091: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 70 KiB/s wr, 23 op/s 2023-03-09T17:21:13.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:21:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:21:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:21:13 smithi181 ceph-mon[94189]: pgmap v5091: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 70 KiB/s wr, 23 op/s 2023-03-09T17:21:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:21:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:21:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:21:13 smithi038 ceph-mon[92422]: pgmap v5091: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 70 KiB/s wr, 23 op/s 2023-03-09T17:21:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:21:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:21:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:21:15 smithi122 ceph-mon[93869]: pgmap v5092: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 45 KiB/s wr, 9 op/s 2023-03-09T17:21:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:21:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8586.053328 secs 2023-03-09T17:21:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:21:15 smithi181 ceph-mon[94189]: pgmap v5092: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 45 KiB/s wr, 9 op/s 2023-03-09T17:21:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:21:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8586.053328 secs 2023-03-09T17:21:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:21:15 smithi038 ceph-mon[92422]: pgmap v5092: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 45 KiB/s wr, 9 op/s 2023-03-09T17:21:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:21:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8586.053328 secs 2023-03-09T17:21:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:21:17 smithi122 ceph-mon[93869]: pgmap v5093: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T17:21:17.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:21:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8588.317816 secs 2023-03-09T17:21:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:21:17 smithi181 ceph-mon[94189]: pgmap v5093: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T17:21:17.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:21:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8588.317816 secs 2023-03-09T17:21:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:21:17 smithi038 ceph-mon[92422]: pgmap v5093: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T17:21:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:21:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8588.317816 secs 2023-03-09T17:21:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9050.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:21:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:21:19 smithi122 ceph-mon[93869]: pgmap v5094: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 43 KiB/s wr, 7 op/s 2023-03-09T17:21:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:21:19 smithi181 ceph-mon[94189]: pgmap v5094: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 43 KiB/s wr, 7 op/s 2023-03-09T17:21:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:21:19 smithi038 ceph-mon[92422]: pgmap v5094: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 43 KiB/s wr, 7 op/s 2023-03-09T17:21:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:21:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8591.053389 secs 2023-03-09T17:21:20.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:21:20 smithi122 ceph-mon[93869]: osdmap e532: 12 total, 12 up, 12 in 2023-03-09T17:21:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:21:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8591.053389 secs 2023-03-09T17:21:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:21:20 smithi181 ceph-mon[94189]: osdmap e532: 12 total, 12 up, 12 in 2023-03-09T17:21:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:21:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8591.053389 secs 2023-03-09T17:21:21.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:21:20 smithi038 ceph-mon[92422]: osdmap e532: 12 total, 12 up, 12 in 2023-03-09T17:21:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:21:21 smithi122 ceph-mon[93869]: pgmap v5095: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 40 KiB/s wr, 7 op/s 2023-03-09T17:21:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:21:21 smithi181 ceph-mon[94189]: pgmap v5095: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 40 KiB/s wr, 7 op/s 2023-03-09T17:21:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:21:21 smithi038 ceph-mon[92422]: pgmap v5095: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 40 KiB/s wr, 7 op/s 2023-03-09T17:21:22.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:21:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8593.317926 secs 2023-03-09T17:21:22.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:21:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8593.317926 secs 2023-03-09T17:21:22.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:21:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8593.317926 secs 2023-03-09T17:21:23.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9055.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:21:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:21:23 smithi122 ceph-mon[93869]: pgmap v5097: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 26 KiB/s wr, 8 op/s 2023-03-09T17:21:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:21:23 smithi181 ceph-mon[94189]: pgmap v5097: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 26 KiB/s wr, 8 op/s 2023-03-09T17:21:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:21:23 smithi038 ceph-mon[92422]: pgmap v5097: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 26 KiB/s wr, 8 op/s 2023-03-09T17:21:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:21:25 smithi122 ceph-mon[93869]: pgmap v5098: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 8 op/s 2023-03-09T17:21:25.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:21:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8596.053452 secs 2023-03-09T17:21:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:21:25 smithi181 ceph-mon[94189]: pgmap v5098: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 8 op/s 2023-03-09T17:21:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:21:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8596.053452 secs 2023-03-09T17:21:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:21:25 smithi038 ceph-mon[92422]: pgmap v5098: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 8 op/s 2023-03-09T17:21:26.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:21:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8596.053452 secs 2023-03-09T17:21:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:21:27 smithi122 ceph-mon[93869]: pgmap v5099: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 8 op/s 2023-03-09T17:21:27.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:21:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8598.318046 secs 2023-03-09T17:21:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:21:27 smithi181 ceph-mon[94189]: pgmap v5099: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 8 op/s 2023-03-09T17:21:27.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:21:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8598.318046 secs 2023-03-09T17:21:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:21:27 smithi038 ceph-mon[92422]: pgmap v5099: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 8 op/s 2023-03-09T17:21:28.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:21:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8598.318046 secs 2023-03-09T17:21:28.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9060.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:21:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:21:29 smithi122 ceph-mon[93869]: pgmap v5100: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 7 op/s 2023-03-09T17:21:29.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:21:29 smithi181 ceph-mon[94189]: pgmap v5100: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 7 op/s 2023-03-09T17:21:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:21:29 smithi038 ceph-mon[92422]: pgmap v5100: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 7 op/s 2023-03-09T17:21:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:21:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8601.053512 secs 2023-03-09T17:21:30.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:21:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8601.053512 secs 2023-03-09T17:21:30.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:21:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8601.053512 secs 2023-03-09T17:21:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:21:31 smithi122 ceph-mon[93869]: pgmap v5101: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 7 op/s 2023-03-09T17:21:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:21:31 smithi181 ceph-mon[94189]: pgmap v5101: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 7 op/s 2023-03-09T17:21:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:21:31 smithi038 ceph-mon[92422]: pgmap v5101: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 7 op/s 2023-03-09T17:21:33.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9065.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:21:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:21:33 smithi122 ceph-mon[93869]: pgmap v5102: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 39 KiB/s wr, 9 op/s 2023-03-09T17:21:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:21:33 smithi181 ceph-mon[94189]: pgmap v5102: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 39 KiB/s wr, 9 op/s 2023-03-09T17:21:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:21:33 smithi038 ceph-mon[92422]: pgmap v5102: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 39 KiB/s wr, 9 op/s 2023-03-09T17:21:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:21:35 smithi122 ceph-mon[93869]: pgmap v5103: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 37 KiB/s wr, 6 op/s 2023-03-09T17:21:35.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:21:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8606.053594 secs 2023-03-09T17:21:35.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:21:35 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:21:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:21:35 smithi181 ceph-mon[94189]: pgmap v5103: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 37 KiB/s wr, 6 op/s 2023-03-09T17:21:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:21:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8606.053594 secs 2023-03-09T17:21:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:21:35 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:21:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:21:35 smithi038 ceph-mon[92422]: pgmap v5103: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 37 KiB/s wr, 6 op/s 2023-03-09T17:21:36.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:21:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8606.053594 secs 2023-03-09T17:21:36.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:21:35 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:21:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:21:36 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:21:36.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:21:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:21:36.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:21:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:21:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:21:36 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:21:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:21:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:21:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:21:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:21:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:21:36 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:21:37.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:21:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:21:37.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:21:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:21:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:21:37 smithi122 ceph-mon[93869]: pgmap v5104: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 37 KiB/s wr, 7 op/s 2023-03-09T17:21:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:21:37 smithi181 ceph-mon[94189]: pgmap v5104: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 37 KiB/s wr, 7 op/s 2023-03-09T17:21:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:21:37 smithi038 ceph-mon[92422]: pgmap v5104: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 37 KiB/s wr, 7 op/s 2023-03-09T17:21:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9070.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:21:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:21:39 smithi122 ceph-mon[93869]: pgmap v5105: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 41 KiB/s wr, 7 op/s 2023-03-09T17:21:39.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:21:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:21:39.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:21:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:21:39.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:21:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:21:39.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:21:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:21:39.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:21:39 smithi181 ceph-mon[94189]: pgmap v5105: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 41 KiB/s wr, 7 op/s 2023-03-09T17:21:39.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:21:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:21:39.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:21:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:21:39.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:21:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:21:39.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:21:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:21:40.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:21:39 smithi038 ceph-mon[92422]: pgmap v5105: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 41 KiB/s wr, 7 op/s 2023-03-09T17:21:40.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:21:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:21:40.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:21:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:21:40.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:21:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:21:40.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:21:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:21:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:21:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8611.053653 secs 2023-03-09T17:21:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:21:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8611.053653 secs 2023-03-09T17:21:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:21:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8611.053653 secs 2023-03-09T17:21:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:21:41 smithi122 ceph-mon[93869]: pgmap v5106: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 21 KiB/s wr, 6 op/s 2023-03-09T17:21:41.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:21:41 smithi181 ceph-mon[94189]: pgmap v5106: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 21 KiB/s wr, 6 op/s 2023-03-09T17:21:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:21:41 smithi038 ceph-mon[92422]: pgmap v5106: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 21 KiB/s wr, 6 op/s 2023-03-09T17:21:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9075.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:21:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:21:43 smithi122 ceph-mon[93869]: pgmap v5107: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 53 KiB/s wr, 9 op/s 2023-03-09T17:21:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:21:43 smithi181 ceph-mon[94189]: pgmap v5107: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 53 KiB/s wr, 9 op/s 2023-03-09T17:21:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:21:43 smithi038 ceph-mon[92422]: pgmap v5107: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 53 KiB/s wr, 9 op/s 2023-03-09T17:21:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:21:45 smithi122 ceph-mon[93869]: pgmap v5108: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T17:21:45.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:21:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8616.053713 secs 2023-03-09T17:21:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:21:45 smithi181 ceph-mon[94189]: pgmap v5108: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T17:21:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:21:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8616.053713 secs 2023-03-09T17:21:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:21:45 smithi038 ceph-mon[92422]: pgmap v5108: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T17:21:46.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:21:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8616.053713 secs 2023-03-09T17:21:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:21:47 smithi122 ceph-mon[93869]: pgmap v5109: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 44 KiB/s wr, 9 op/s 2023-03-09T17:21:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:21:47 smithi181 ceph-mon[94189]: pgmap v5109: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 44 KiB/s wr, 9 op/s 2023-03-09T17:21:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:21:47 smithi038 ceph-mon[92422]: pgmap v5109: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 44 KiB/s wr, 9 op/s 2023-03-09T17:21:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9080.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:21:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:21:49 smithi122 ceph-mon[93869]: pgmap v5110: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 117 KiB/s wr, 10 op/s 2023-03-09T17:21:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:21:49 smithi181 ceph-mon[94189]: pgmap v5110: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 117 KiB/s wr, 10 op/s 2023-03-09T17:21:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:21:49 smithi038 ceph-mon[92422]: pgmap v5110: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 117 KiB/s wr, 10 op/s 2023-03-09T17:21:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:21:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8621.053772 secs 2023-03-09T17:21:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:21:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8621.053772 secs 2023-03-09T17:21:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:21:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8621.053772 secs 2023-03-09T17:21:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:21:51 smithi038 ceph-mon[92422]: pgmap v5111: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 113 KiB/s wr, 9 op/s 2023-03-09T17:21:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:21:51 smithi122 ceph-mon[93869]: pgmap v5111: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 113 KiB/s wr, 9 op/s 2023-03-09T17:21:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:21:51 smithi181 ceph-mon[94189]: pgmap v5111: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 113 KiB/s wr, 9 op/s 2023-03-09T17:21:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9085.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:21:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:21:53 smithi038 ceph-mon[92422]: pgmap v5112: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 132 KiB/s wr, 12 op/s 2023-03-09T17:21:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:21:53 smithi122 ceph-mon[93869]: pgmap v5112: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 132 KiB/s wr, 12 op/s 2023-03-09T17:21:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:21:53 smithi181 ceph-mon[94189]: pgmap v5112: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 132 KiB/s wr, 12 op/s 2023-03-09T17:21:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:21:55 smithi038 ceph-mon[92422]: pgmap v5113: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 100 KiB/s wr, 10 op/s 2023-03-09T17:21:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:21:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8626.053834 secs 2023-03-09T17:21:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:21:55 smithi122 ceph-mon[93869]: pgmap v5113: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 100 KiB/s wr, 10 op/s 2023-03-09T17:21:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:21:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8626.053834 secs 2023-03-09T17:21:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:21:55 smithi181 ceph-mon[94189]: pgmap v5113: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 100 KiB/s wr, 10 op/s 2023-03-09T17:21:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:21:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8626.053834 secs 2023-03-09T17:21:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:21:57 smithi038 ceph-mon[92422]: pgmap v5114: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 99 KiB/s wr, 12 op/s 2023-03-09T17:21:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:21:57 smithi122 ceph-mon[93869]: pgmap v5114: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 99 KiB/s wr, 12 op/s 2023-03-09T17:21:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:21:57 smithi181 ceph-mon[94189]: pgmap v5114: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 99 KiB/s wr, 12 op/s 2023-03-09T17:21:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9090.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:22:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:21:59 smithi122 ceph-mon[93869]: pgmap v5115: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 107 KiB/s wr, 11 op/s 2023-03-09T17:22:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:21:59 smithi181 ceph-mon[94189]: pgmap v5115: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 107 KiB/s wr, 11 op/s 2023-03-09T17:22:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:21:59 smithi038 ceph-mon[92422]: pgmap v5115: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 107 KiB/s wr, 11 op/s 2023-03-09T17:22:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:22:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8631.053894 secs 2023-03-09T17:22:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:22:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8631.053894 secs 2023-03-09T17:22:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:22:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8631.053894 secs 2023-03-09T17:22:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:22:01 smithi122 ceph-mon[93869]: pgmap v5116: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 33 KiB/s wr, 8 op/s 2023-03-09T17:22:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:22:01 smithi181 ceph-mon[94189]: pgmap v5116: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 33 KiB/s wr, 8 op/s 2023-03-09T17:22:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:22:01 smithi038 ceph-mon[92422]: pgmap v5116: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 33 KiB/s wr, 8 op/s 2023-03-09T17:22:03.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:22:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8633.318869 secs 2023-03-09T17:22:03.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:22:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8633.318869 secs 2023-03-09T17:22:03.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:22:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8633.318869 secs 2023-03-09T17:22:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9095.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:22:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:22:03 smithi122 ceph-mon[93869]: pgmap v5117: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 71 KiB/s wr, 12 op/s 2023-03-09T17:22:04.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:22:03 smithi181 ceph-mon[94189]: pgmap v5117: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 71 KiB/s wr, 12 op/s 2023-03-09T17:22:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:22:03 smithi038 ceph-mon[92422]: pgmap v5117: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 71 KiB/s wr, 12 op/s 2023-03-09T17:22:05.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:22:04 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:22:05.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:22:04 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:22:05.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:22:04 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:22:05.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:22:04 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:22:05.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:22:04 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:22:05.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:22:04 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:22:05.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:22:04 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:22:05.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:22:04 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:22:05.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:22:04 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:22:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:22:05 smithi122 ceph-mon[93869]: pgmap v5118: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 89 KiB/s wr, 21 op/s 2023-03-09T17:22:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:22:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8636.053954 secs 2023-03-09T17:22:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:22:05 smithi122 ceph-mon[93869]: osdmap e533: 12 total, 12 up, 12 in 2023-03-09T17:22:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:22:05 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:22:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:22:05 smithi181 ceph-mon[94189]: pgmap v5118: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 89 KiB/s wr, 21 op/s 2023-03-09T17:22:06.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:22:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8636.053954 secs 2023-03-09T17:22:06.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:22:05 smithi181 ceph-mon[94189]: osdmap e533: 12 total, 12 up, 12 in 2023-03-09T17:22:06.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:22:05 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:22:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:22:05 smithi038 ceph-mon[92422]: pgmap v5118: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 89 KiB/s wr, 21 op/s 2023-03-09T17:22:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:22:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8636.053954 secs 2023-03-09T17:22:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:22:05 smithi038 ceph-mon[92422]: osdmap e533: 12 total, 12 up, 12 in 2023-03-09T17:22:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:22:05 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:22:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:22:07 smithi122 ceph-mon[93869]: pgmap v5120: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 186 KiB/s wr, 28 op/s 2023-03-09T17:22:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:22:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8638.318987 secs 2023-03-09T17:22:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:22:07 smithi181 ceph-mon[94189]: pgmap v5120: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 186 KiB/s wr, 28 op/s 2023-03-09T17:22:08.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:22:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8638.318987 secs 2023-03-09T17:22:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:22:07 smithi038 ceph-mon[92422]: pgmap v5120: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 186 KiB/s wr, 28 op/s 2023-03-09T17:22:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:22:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8638.318987 secs 2023-03-09T17:22:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9100.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:22:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:22:09 smithi122 ceph-mon[93869]: pgmap v5121: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 177 KiB/s wr, 27 op/s 2023-03-09T17:22:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:22:09 smithi181 ceph-mon[94189]: pgmap v5121: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 177 KiB/s wr, 27 op/s 2023-03-09T17:22:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:22:09 smithi038 ceph-mon[92422]: pgmap v5121: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 177 KiB/s wr, 27 op/s 2023-03-09T17:22:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:22:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8641.053983 secs 2023-03-09T17:22:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:22:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8641.053983 secs 2023-03-09T17:22:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:22:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8641.053983 secs 2023-03-09T17:22:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:22:11 smithi122 ceph-mon[93869]: pgmap v5122: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 177 KiB/s wr, 27 op/s 2023-03-09T17:22:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:22:11 smithi181 ceph-mon[94189]: pgmap v5122: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 177 KiB/s wr, 27 op/s 2023-03-09T17:22:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:22:11 smithi038 ceph-mon[92422]: pgmap v5122: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 177 KiB/s wr, 27 op/s 2023-03-09T17:22:13.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:22:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8643.319123 secs 2023-03-09T17:22:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:22:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8643.319123 secs 2023-03-09T17:22:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:22:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8643.319123 secs 2023-03-09T17:22:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9105.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:22:14.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:22:13 smithi122 ceph-mon[93869]: pgmap v5123: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 144 KiB/s wr, 27 op/s 2023-03-09T17:22:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:22:13 smithi181 ceph-mon[94189]: pgmap v5123: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 144 KiB/s wr, 27 op/s 2023-03-09T17:22:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:22:13 smithi038 ceph-mon[92422]: pgmap v5123: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 144 KiB/s wr, 27 op/s 2023-03-09T17:22:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:22:15 smithi122 ceph-mon[93869]: pgmap v5124: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 110 KiB/s wr, 12 op/s 2023-03-09T17:22:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:22:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8646.054046 secs 2023-03-09T17:22:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:22:15 smithi122 ceph-mon[93869]: osdmap e534: 12 total, 12 up, 12 in 2023-03-09T17:22:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:22:15 smithi181 ceph-mon[94189]: pgmap v5124: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 110 KiB/s wr, 12 op/s 2023-03-09T17:22:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:22:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8646.054046 secs 2023-03-09T17:22:16.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:22:15 smithi181 ceph-mon[94189]: osdmap e534: 12 total, 12 up, 12 in 2023-03-09T17:22:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:22:15 smithi038 ceph-mon[92422]: pgmap v5124: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 110 KiB/s wr, 12 op/s 2023-03-09T17:22:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:22:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8646.054046 secs 2023-03-09T17:22:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:22:15 smithi038 ceph-mon[92422]: osdmap e534: 12 total, 12 up, 12 in 2023-03-09T17:22:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:22:17 smithi122 ceph-mon[93869]: pgmap v5126: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 35 KiB/s wr, 8 op/s 2023-03-09T17:22:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:22:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8648.319239 secs 2023-03-09T17:22:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:22:17 smithi181 ceph-mon[94189]: pgmap v5126: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 35 KiB/s wr, 8 op/s 2023-03-09T17:22:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:22:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8648.319239 secs 2023-03-09T17:22:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:22:17 smithi038 ceph-mon[92422]: pgmap v5126: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 35 KiB/s wr, 8 op/s 2023-03-09T17:22:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:22:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8648.319239 secs 2023-03-09T17:22:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9110.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:22:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:22:19 smithi122 ceph-mon[93869]: pgmap v5127: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 KiB/s wr, 8 op/s 2023-03-09T17:22:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:22:19 smithi181 ceph-mon[94189]: pgmap v5127: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 KiB/s wr, 8 op/s 2023-03-09T17:22:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:22:19 smithi038 ceph-mon[92422]: pgmap v5127: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 KiB/s wr, 8 op/s 2023-03-09T17:22:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:22:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8651.054109 secs 2023-03-09T17:22:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:22:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8651.054109 secs 2023-03-09T17:22:21.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:22:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8651.054109 secs 2023-03-09T17:22:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:22:21 smithi122 ceph-mon[93869]: pgmap v5128: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 KiB/s wr, 8 op/s 2023-03-09T17:22:22.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:22:21 smithi181 ceph-mon[94189]: pgmap v5128: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 KiB/s wr, 8 op/s 2023-03-09T17:22:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:22:21 smithi038 ceph-mon[92422]: pgmap v5128: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 KiB/s wr, 8 op/s 2023-03-09T17:22:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:22:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8653.319357 secs 2023-03-09T17:22:23.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:22:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8653.319357 secs 2023-03-09T17:22:23.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:22:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8653.319357 secs 2023-03-09T17:22:23.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9115.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:22:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:22:23 smithi122 ceph-mon[93869]: pgmap v5129: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T17:22:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:22:23 smithi181 ceph-mon[94189]: pgmap v5129: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T17:22:24.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:22:23 smithi038 ceph-mon[92422]: pgmap v5129: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T17:22:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:22:25 smithi122 ceph-mon[93869]: pgmap v5130: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T17:22:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:22:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8656.054172 secs 2023-03-09T17:22:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:22:25 smithi181 ceph-mon[94189]: pgmap v5130: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T17:22:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:22:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8656.054172 secs 2023-03-09T17:22:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:22:25 smithi038 ceph-mon[92422]: pgmap v5130: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T17:22:26.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:22:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8656.054172 secs 2023-03-09T17:22:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:22:27 smithi122 ceph-mon[93869]: pgmap v5131: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 37 KiB/s wr, 9 op/s 2023-03-09T17:22:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:22:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8658.319435 secs 2023-03-09T17:22:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:22:27 smithi181 ceph-mon[94189]: pgmap v5131: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 37 KiB/s wr, 9 op/s 2023-03-09T17:22:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:22:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8658.319435 secs 2023-03-09T17:22:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:22:27 smithi038 ceph-mon[92422]: pgmap v5131: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 37 KiB/s wr, 9 op/s 2023-03-09T17:22:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:22:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8658.319435 secs 2023-03-09T17:22:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9120.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:22:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:22:29 smithi122 ceph-mon[93869]: pgmap v5132: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 22 KiB/s wr, 7 op/s 2023-03-09T17:22:30.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:22:29 smithi181 ceph-mon[94189]: pgmap v5132: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 22 KiB/s wr, 7 op/s 2023-03-09T17:22:30.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:22:29 smithi038 ceph-mon[92422]: pgmap v5132: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 22 KiB/s wr, 7 op/s 2023-03-09T17:22:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:22:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8661.054233 secs 2023-03-09T17:22:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:22:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8661.054233 secs 2023-03-09T17:22:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:22:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8661.054233 secs 2023-03-09T17:22:32.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:22:31 smithi122 ceph-mon[93869]: pgmap v5133: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 21 KiB/s wr, 6 op/s 2023-03-09T17:22:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:22:31 smithi181 ceph-mon[94189]: pgmap v5133: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 21 KiB/s wr, 6 op/s 2023-03-09T17:22:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:22:31 smithi038 ceph-mon[92422]: pgmap v5133: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 21 KiB/s wr, 6 op/s 2023-03-09T17:22:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:22:32 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 8663.319547 secs 2023-03-09T17:22:33.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:22:32 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 8663.319547 secs 2023-03-09T17:22:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:22:32 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 8663.319547 secs 2023-03-09T17:22:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9125.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:22:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:22:33 smithi122 ceph-mon[93869]: pgmap v5134: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 21 KiB/s wr, 8 op/s 2023-03-09T17:22:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:22:33 smithi181 ceph-mon[94189]: pgmap v5134: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 21 KiB/s wr, 8 op/s 2023-03-09T17:22:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:22:33 smithi038 ceph-mon[92422]: pgmap v5134: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 21 KiB/s wr, 8 op/s 2023-03-09T17:22:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:22:35 smithi122 ceph-mon[93869]: pgmap v5135: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 21 KiB/s wr, 6 op/s 2023-03-09T17:22:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:22:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8666.054316 secs 2023-03-09T17:22:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:22:35 smithi181 ceph-mon[94189]: pgmap v5135: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 21 KiB/s wr, 6 op/s 2023-03-09T17:22:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:22:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8666.054316 secs 2023-03-09T17:22:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:22:35 smithi038 ceph-mon[92422]: pgmap v5135: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 21 KiB/s wr, 6 op/s 2023-03-09T17:22:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:22:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8666.054316 secs 2023-03-09T17:22:37.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:22:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:22:37.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:22:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:22:37.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:22:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:22:37.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:22:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:22:37.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:22:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:22:37.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:22:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:22:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:22:37 smithi122 ceph-mon[93869]: pgmap v5136: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 62 KiB/s wr, 8 op/s 2023-03-09T17:22:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:22:37 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 8668.319676 secs 2023-03-09T17:22:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:22:37 smithi181 ceph-mon[94189]: pgmap v5136: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 62 KiB/s wr, 8 op/s 2023-03-09T17:22:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:22:37 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 8668.319676 secs 2023-03-09T17:22:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:22:37 smithi038 ceph-mon[92422]: pgmap v5136: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 62 KiB/s wr, 8 op/s 2023-03-09T17:22:38.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:22:37 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 8668.319676 secs 2023-03-09T17:22:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9130.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:22:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:22:39 smithi122 ceph-mon[93869]: pgmap v5137: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 7 op/s 2023-03-09T17:22:40.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:22:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:22:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:22:39 smithi181 ceph-mon[94189]: pgmap v5137: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 7 op/s 2023-03-09T17:22:40.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:22:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:22:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:22:39 smithi038 ceph-mon[92422]: pgmap v5137: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 7 op/s 2023-03-09T17:22:40.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:22:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:22:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:22:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8671.054377 secs 2023-03-09T17:22:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:22:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:22:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:22:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:22:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:22:41 smithi122 ceph-mon[93869]: pgmap v5138: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 50 KiB/s wr, 6 op/s 2023-03-09T17:22:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:22:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:22:41.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:22:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:22:41.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:22:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:22:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:22:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8671.054377 secs 2023-03-09T17:22:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:22:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:22:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:22:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:22:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:22:41 smithi181 ceph-mon[94189]: pgmap v5138: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 50 KiB/s wr, 6 op/s 2023-03-09T17:22:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:22:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:22:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:22:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:22:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:22:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:22:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:22:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8671.054377 secs 2023-03-09T17:22:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:22:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:22:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:22:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:22:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:22:41 smithi038 ceph-mon[92422]: pgmap v5138: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 50 KiB/s wr, 6 op/s 2023-03-09T17:22:41.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:22:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:22:41.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:22:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:22:41.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:22:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:22:43.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:22:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8673.319796 secs 2023-03-09T17:22:43.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:22:43 smithi122 ceph-mon[93869]: pgmap v5139: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 83 KiB/s wr, 10 op/s 2023-03-09T17:22:43.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:22:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8673.319796 secs 2023-03-09T17:22:43.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:22:43 smithi181 ceph-mon[94189]: pgmap v5139: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 83 KiB/s wr, 10 op/s 2023-03-09T17:22:43.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:22:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8673.319796 secs 2023-03-09T17:22:43.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:22:43 smithi038 ceph-mon[92422]: pgmap v5139: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 83 KiB/s wr, 10 op/s 2023-03-09T17:22:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9135.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:22:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:22:45 smithi122 ceph-mon[93869]: pgmap v5140: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 91 KiB/s wr, 9 op/s 2023-03-09T17:22:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:22:45 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:22:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:22:45 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:22:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:22:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8676.054449 secs 2023-03-09T17:22:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:22:45 smithi181 ceph-mon[94189]: pgmap v5140: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 91 KiB/s wr, 9 op/s 2023-03-09T17:22:45.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:22:45 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:22:45.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:22:45 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:22:45.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:22:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8676.054449 secs 2023-03-09T17:22:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:22:45 smithi038 ceph-mon[92422]: pgmap v5140: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 91 KiB/s wr, 9 op/s 2023-03-09T17:22:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:22:45 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:22:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:22:45 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:22:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:22:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8676.054449 secs 2023-03-09T17:22:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:22:47 smithi122 ceph-mon[93869]: pgmap v5141: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 127 KiB/s wr, 11 op/s 2023-03-09T17:22:47.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:22:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8678.319929 secs 2023-03-09T17:22:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:22:47 smithi181 ceph-mon[94189]: pgmap v5141: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 127 KiB/s wr, 11 op/s 2023-03-09T17:22:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:22:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8678.319929 secs 2023-03-09T17:22:48.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:22:47 smithi038 ceph-mon[92422]: pgmap v5141: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 127 KiB/s wr, 11 op/s 2023-03-09T17:22:48.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:22:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8678.319929 secs 2023-03-09T17:22:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9140.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:22:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:22:49 smithi122 ceph-mon[93869]: pgmap v5142: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 94 KiB/s wr, 10 op/s 2023-03-09T17:22:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:22:49 smithi181 ceph-mon[94189]: pgmap v5142: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 94 KiB/s wr, 10 op/s 2023-03-09T17:22:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:22:49 smithi038 ceph-mon[92422]: pgmap v5142: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 94 KiB/s wr, 10 op/s 2023-03-09T17:22:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:22:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8681.054509 secs 2023-03-09T17:22:50.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:22:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8681.054509 secs 2023-03-09T17:22:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:22:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8681.054509 secs 2023-03-09T17:22:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:22:51 smithi122 ceph-mon[93869]: pgmap v5143: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 86 KiB/s wr, 9 op/s 2023-03-09T17:22:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:22:51 smithi181 ceph-mon[94189]: pgmap v5143: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 86 KiB/s wr, 9 op/s 2023-03-09T17:22:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:22:51 smithi038 ceph-mon[92422]: pgmap v5143: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 86 KiB/s wr, 9 op/s 2023-03-09T17:22:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:22:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8683.320021 secs 2023-03-09T17:22:52.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:22:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8683.320021 secs 2023-03-09T17:22:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:22:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8683.320021 secs 2023-03-09T17:22:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9145.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:22:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:22:53 smithi122 ceph-mon[93869]: pgmap v5144: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 118 KiB/s wr, 11 op/s 2023-03-09T17:22:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:22:53 smithi181 ceph-mon[94189]: pgmap v5144: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 118 KiB/s wr, 11 op/s 2023-03-09T17:22:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:22:53 smithi038 ceph-mon[92422]: pgmap v5144: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 118 KiB/s wr, 11 op/s 2023-03-09T17:22:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:22:55 smithi122 ceph-mon[93869]: pgmap v5145: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 93 KiB/s wr, 9 op/s 2023-03-09T17:22:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:22:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8686.054564 secs 2023-03-09T17:22:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:22:55 smithi181 ceph-mon[94189]: pgmap v5145: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 93 KiB/s wr, 9 op/s 2023-03-09T17:22:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:22:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8686.054564 secs 2023-03-09T17:22:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:22:55 smithi038 ceph-mon[92422]: pgmap v5145: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 93 KiB/s wr, 9 op/s 2023-03-09T17:22:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:22:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8686.054564 secs 2023-03-09T17:22:57.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:22:57 smithi122 ceph-mon[93869]: pgmap v5146: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 141 KiB/s wr, 11 op/s 2023-03-09T17:22:57.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:22:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8688.320159 secs 2023-03-09T17:22:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:22:57 smithi181 ceph-mon[94189]: pgmap v5146: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 141 KiB/s wr, 11 op/s 2023-03-09T17:22:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:22:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8688.320159 secs 2023-03-09T17:22:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:22:57 smithi038 ceph-mon[92422]: pgmap v5146: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 141 KiB/s wr, 11 op/s 2023-03-09T17:22:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:22:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8688.320159 secs 2023-03-09T17:22:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9150.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:22:59.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:22:59 smithi122 ceph-mon[93869]: pgmap v5147: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 112 KiB/s wr, 9 op/s 2023-03-09T17:22:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:22:59 smithi181 ceph-mon[94189]: pgmap v5147: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 112 KiB/s wr, 9 op/s 2023-03-09T17:22:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:22:59 smithi038 ceph-mon[92422]: pgmap v5147: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 112 KiB/s wr, 9 op/s 2023-03-09T17:23:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:23:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8691.054626 secs 2023-03-09T17:23:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:23:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8691.054626 secs 2023-03-09T17:23:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:23:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8691.054626 secs 2023-03-09T17:23:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:23:01 smithi122 ceph-mon[93869]: pgmap v5148: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 104 KiB/s wr, 8 op/s 2023-03-09T17:23:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:23:01 smithi181 ceph-mon[94189]: pgmap v5148: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 104 KiB/s wr, 8 op/s 2023-03-09T17:23:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:23:01 smithi038 ceph-mon[92422]: pgmap v5148: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 104 KiB/s wr, 8 op/s 2023-03-09T17:23:02.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:23:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8693.320280 secs 2023-03-09T17:23:02.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:23:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8693.320280 secs 2023-03-09T17:23:02.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:23:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8693.320280 secs 2023-03-09T17:23:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9155.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:23:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:23:03 smithi122 ceph-mon[93869]: pgmap v5149: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 144 KiB/s wr, 13 op/s 2023-03-09T17:23:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:23:03 smithi181 ceph-mon[94189]: pgmap v5149: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 144 KiB/s wr, 13 op/s 2023-03-09T17:23:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:23:03 smithi038 ceph-mon[92422]: pgmap v5149: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 144 KiB/s wr, 13 op/s 2023-03-09T17:23:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:23:05 smithi122 ceph-mon[93869]: pgmap v5150: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 133 KiB/s wr, 23 op/s 2023-03-09T17:23:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:23:05 smithi122 ceph-mon[93869]: osdmap e535: 12 total, 12 up, 12 in 2023-03-09T17:23:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:23:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8696.054689 secs 2023-03-09T17:23:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:23:05 smithi181 ceph-mon[94189]: pgmap v5150: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 133 KiB/s wr, 23 op/s 2023-03-09T17:23:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:23:05 smithi181 ceph-mon[94189]: osdmap e535: 12 total, 12 up, 12 in 2023-03-09T17:23:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:23:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8696.054689 secs 2023-03-09T17:23:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:23:05 smithi038 ceph-mon[92422]: pgmap v5150: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 133 KiB/s wr, 23 op/s 2023-03-09T17:23:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:23:05 smithi038 ceph-mon[92422]: osdmap e535: 12 total, 12 up, 12 in 2023-03-09T17:23:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:23:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8696.054689 secs 2023-03-09T17:23:07.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:23:07 smithi122 ceph-mon[93869]: pgmap v5152: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 135 KiB/s wr, 27 op/s 2023-03-09T17:23:07.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:23:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8698.320415 secs 2023-03-09T17:23:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:23:07 smithi181 ceph-mon[94189]: pgmap v5152: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 135 KiB/s wr, 27 op/s 2023-03-09T17:23:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:23:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8698.320415 secs 2023-03-09T17:23:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:23:07 smithi038 ceph-mon[92422]: pgmap v5152: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 135 KiB/s wr, 27 op/s 2023-03-09T17:23:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:23:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8698.320415 secs 2023-03-09T17:23:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9160.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:23:09.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:23:09 smithi122 ceph-mon[93869]: pgmap v5153: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 134 KiB/s wr, 26 op/s 2023-03-09T17:23:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:23:09 smithi181 ceph-mon[94189]: pgmap v5153: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 134 KiB/s wr, 26 op/s 2023-03-09T17:23:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:23:09 smithi038 ceph-mon[92422]: pgmap v5153: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 134 KiB/s wr, 26 op/s 2023-03-09T17:23:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:23:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8701.054750 secs 2023-03-09T17:23:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:23:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8701.054750 secs 2023-03-09T17:23:11.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:23:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8701.054750 secs 2023-03-09T17:23:11.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:23:11 smithi122 ceph-mon[93869]: pgmap v5154: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 134 KiB/s wr, 26 op/s 2023-03-09T17:23:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:23:11 smithi181 ceph-mon[94189]: pgmap v5154: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 134 KiB/s wr, 26 op/s 2023-03-09T17:23:12.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:23:11 smithi038 ceph-mon[92422]: pgmap v5154: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 134 KiB/s wr, 26 op/s 2023-03-09T17:23:12.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:23:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8703.320511 secs 2023-03-09T17:23:12.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:23:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8703.320511 secs 2023-03-09T17:23:13.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:23:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8703.320511 secs 2023-03-09T17:23:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9165.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:23:13.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:23:13 smithi122 ceph-mon[93869]: pgmap v5155: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 102 KiB/s wr, 23 op/s 2023-03-09T17:23:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:23:13 smithi181 ceph-mon[94189]: pgmap v5155: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 102 KiB/s wr, 23 op/s 2023-03-09T17:23:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:23:13 smithi038 ceph-mon[92422]: pgmap v5155: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 102 KiB/s wr, 23 op/s 2023-03-09T17:23:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:23:15 smithi122 ceph-mon[93869]: pgmap v5156: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 78 KiB/s wr, 9 op/s 2023-03-09T17:23:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:23:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8706.054809 secs 2023-03-09T17:23:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:23:15 smithi122 ceph-mon[93869]: osdmap e536: 12 total, 12 up, 12 in 2023-03-09T17:23:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:23:15 smithi181 ceph-mon[94189]: pgmap v5156: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 78 KiB/s wr, 9 op/s 2023-03-09T17:23:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:23:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8706.054809 secs 2023-03-09T17:23:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:23:15 smithi181 ceph-mon[94189]: osdmap e536: 12 total, 12 up, 12 in 2023-03-09T17:23:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:23:15 smithi038 ceph-mon[92422]: pgmap v5156: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 78 KiB/s wr, 9 op/s 2023-03-09T17:23:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:23:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8706.054809 secs 2023-03-09T17:23:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:23:15 smithi038 ceph-mon[92422]: osdmap e536: 12 total, 12 up, 12 in 2023-03-09T17:23:17.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:23:17 smithi122 ceph-mon[93869]: pgmap v5158: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T17:23:17.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:23:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8708.320600 secs 2023-03-09T17:23:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:23:17 smithi181 ceph-mon[94189]: pgmap v5158: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T17:23:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:23:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8708.320600 secs 2023-03-09T17:23:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:23:17 smithi038 ceph-mon[92422]: pgmap v5158: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T17:23:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:23:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8708.320600 secs 2023-03-09T17:23:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9170.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:23:19.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:23:19 smithi122 ceph-mon[93869]: pgmap v5159: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 7 op/s 2023-03-09T17:23:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:23:19 smithi181 ceph-mon[94189]: pgmap v5159: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 7 op/s 2023-03-09T17:23:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:23:19 smithi038 ceph-mon[92422]: pgmap v5159: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 7 op/s 2023-03-09T17:23:20.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:23:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8711.054872 secs 2023-03-09T17:23:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:23:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8711.054872 secs 2023-03-09T17:23:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:23:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8711.054872 secs 2023-03-09T17:23:21.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:23:21 smithi122 ceph-mon[93869]: pgmap v5160: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 7 op/s 2023-03-09T17:23:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:23:21 smithi181 ceph-mon[94189]: pgmap v5160: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 7 op/s 2023-03-09T17:23:22.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:23:21 smithi038 ceph-mon[92422]: pgmap v5160: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 7 op/s 2023-03-09T17:23:22.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:23:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8713.320720 secs 2023-03-09T17:23:22.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:23:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8713.320720 secs 2023-03-09T17:23:22.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:23:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8713.320720 secs 2023-03-09T17:23:23.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9175.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:23:23.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:23:23 smithi122 ceph-mon[93869]: pgmap v5161: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 7 op/s 2023-03-09T17:23:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:23:23 smithi181 ceph-mon[94189]: pgmap v5161: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 7 op/s 2023-03-09T17:23:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:23:23 smithi038 ceph-mon[92422]: pgmap v5161: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 7 op/s 2023-03-09T17:23:25.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:23:25 smithi122 ceph-mon[93869]: pgmap v5162: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T17:23:25.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:23:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8716.054933 secs 2023-03-09T17:23:25.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:23:25 smithi181 ceph-mon[94189]: pgmap v5162: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T17:23:25.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:23:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8716.054933 secs 2023-03-09T17:23:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:23:25 smithi038 ceph-mon[92422]: pgmap v5162: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 16 KiB/s wr, 7 op/s 2023-03-09T17:23:26.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:23:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8716.054933 secs 2023-03-09T17:23:27.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:23:27 smithi122 ceph-mon[93869]: pgmap v5163: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 39 KiB/s wr, 10 op/s 2023-03-09T17:23:27.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:23:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8718.320837 secs 2023-03-09T17:23:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:23:27 smithi181 ceph-mon[94189]: pgmap v5163: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 39 KiB/s wr, 10 op/s 2023-03-09T17:23:27.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:23:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8718.320837 secs 2023-03-09T17:23:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:23:27 smithi038 ceph-mon[92422]: pgmap v5163: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 39 KiB/s wr, 10 op/s 2023-03-09T17:23:28.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:23:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8718.320837 secs 2023-03-09T17:23:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9180.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:23:29.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:23:29 smithi122 ceph-mon[93869]: pgmap v5164: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 35 KiB/s wr, 7 op/s 2023-03-09T17:23:29.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:23:29 smithi181 ceph-mon[94189]: pgmap v5164: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 35 KiB/s wr, 7 op/s 2023-03-09T17:23:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:23:29 smithi038 ceph-mon[92422]: pgmap v5164: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 35 KiB/s wr, 7 op/s 2023-03-09T17:23:30.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:23:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8721.054983 secs 2023-03-09T17:23:30.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:23:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8721.054983 secs 2023-03-09T17:23:30.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:23:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8721.054983 secs 2023-03-09T17:23:31.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:23:31 smithi122 ceph-mon[93869]: pgmap v5165: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 35 KiB/s wr, 7 op/s 2023-03-09T17:23:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:23:31 smithi181 ceph-mon[94189]: pgmap v5165: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 35 KiB/s wr, 7 op/s 2023-03-09T17:23:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:23:31 smithi038 ceph-mon[92422]: pgmap v5165: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 35 KiB/s wr, 7 op/s 2023-03-09T17:23:32.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:23:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8723.320976 secs 2023-03-09T17:23:33.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:23:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8723.320976 secs 2023-03-09T17:23:33.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:23:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8723.320976 secs 2023-03-09T17:23:33.802 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9185.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:23:34.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:23:33 smithi038 ceph-mon[92422]: pgmap v5166: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 52 KiB/s wr, 9 op/s 2023-03-09T17:23:34.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:23:33 smithi122 ceph-mon[93869]: pgmap v5166: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 52 KiB/s wr, 9 op/s 2023-03-09T17:23:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:23:33 smithi181 ceph-mon[94189]: pgmap v5166: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 52 KiB/s wr, 9 op/s 2023-03-09T17:23:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:23:35 smithi038 ceph-mon[92422]: pgmap v5167: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 39 KiB/s wr, 7 op/s 2023-03-09T17:23:36.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:23:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8726.055042 secs 2023-03-09T17:23:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:23:35 smithi122 ceph-mon[93869]: pgmap v5167: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 39 KiB/s wr, 7 op/s 2023-03-09T17:23:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:23:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8726.055042 secs 2023-03-09T17:23:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:23:35 smithi181 ceph-mon[94189]: pgmap v5167: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 39 KiB/s wr, 7 op/s 2023-03-09T17:23:36.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:23:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8726.055042 secs 2023-03-09T17:23:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:23:37 smithi122 ceph-mon[93869]: pgmap v5168: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 67 KiB/s wr, 9 op/s 2023-03-09T17:23:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:23:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8728.321120 secs 2023-03-09T17:23:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:23:37 smithi181 ceph-mon[94189]: pgmap v5168: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 67 KiB/s wr, 9 op/s 2023-03-09T17:23:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:23:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8728.321120 secs 2023-03-09T17:23:38.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:23:37 smithi038 ceph-mon[92422]: pgmap v5168: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 67 KiB/s wr, 9 op/s 2023-03-09T17:23:38.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:23:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8728.321120 secs 2023-03-09T17:23:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9190.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:23:40.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:23:39 smithi122 ceph-mon[93869]: pgmap v5169: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 54 KiB/s wr, 8 op/s 2023-03-09T17:23:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:23:39 smithi181 ceph-mon[94189]: pgmap v5169: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 54 KiB/s wr, 8 op/s 2023-03-09T17:23:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:23:39 smithi038 ceph-mon[92422]: pgmap v5169: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 54 KiB/s wr, 8 op/s 2023-03-09T17:23:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:23:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8731.055105 secs 2023-03-09T17:23:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:23:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8731.055105 secs 2023-03-09T17:23:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:23:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8731.055105 secs 2023-03-09T17:23:42.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:23:41 smithi122 ceph-mon[93869]: pgmap v5170: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 54 KiB/s wr, 7 op/s 2023-03-09T17:23:42.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:23:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:23:42.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:23:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:23:42.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:23:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:23:42.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:23:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:23:42.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:23:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:23:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:23:41 smithi181 ceph-mon[94189]: pgmap v5170: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 54 KiB/s wr, 7 op/s 2023-03-09T17:23:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:23:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:23:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:23:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:23:42.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:23:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:23:42.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:23:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:23:42.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:23:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:23:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:23:41 smithi038 ceph-mon[92422]: pgmap v5170: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 54 KiB/s wr, 7 op/s 2023-03-09T17:23:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:23:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:23:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:23:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:23:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:23:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:23:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:23:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:23:42.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:23:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:23:43.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:23:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8733.321186 secs 2023-03-09T17:23:43.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:23:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:23:43.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:23:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:23:43.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:23:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:23:43.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:23:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8733.321186 secs 2023-03-09T17:23:43.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:23:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:23:43.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:23:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:23:43.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:23:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:23:43.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:23:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8733.321186 secs 2023-03-09T17:23:43.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:23:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:23:43.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:23:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:23:43.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:23:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:23:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9195.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:23:44.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:23:43 smithi122 ceph-mon[93869]: pgmap v5171: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 89 KiB/s wr, 10 op/s 2023-03-09T17:23:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:23:43 smithi181 ceph-mon[94189]: pgmap v5171: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 89 KiB/s wr, 10 op/s 2023-03-09T17:23:44.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:23:43 smithi038 ceph-mon[92422]: pgmap v5171: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 89 KiB/s wr, 10 op/s 2023-03-09T17:23:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:23:45 smithi122 ceph-mon[93869]: pgmap v5172: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 80 KiB/s wr, 9 op/s 2023-03-09T17:23:46.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:23:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8736.055164 secs 2023-03-09T17:23:46.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:23:45 smithi181 ceph-mon[94189]: pgmap v5172: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 80 KiB/s wr, 9 op/s 2023-03-09T17:23:46.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:23:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8736.055164 secs 2023-03-09T17:23:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:23:45 smithi038 ceph-mon[92422]: pgmap v5172: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 80 KiB/s wr, 9 op/s 2023-03-09T17:23:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:23:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8736.055164 secs 2023-03-09T17:23:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:23:47 smithi122 ceph-mon[93869]: pgmap v5173: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 136 KiB/s wr, 12 op/s 2023-03-09T17:23:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:23:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8738.321280 secs 2023-03-09T17:23:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:23:47 smithi181 ceph-mon[94189]: pgmap v5173: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 136 KiB/s wr, 12 op/s 2023-03-09T17:23:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:23:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8738.321280 secs 2023-03-09T17:23:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:23:47 smithi038 ceph-mon[92422]: pgmap v5173: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 136 KiB/s wr, 12 op/s 2023-03-09T17:23:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:23:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8738.321280 secs 2023-03-09T17:23:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9200.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:23:50.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:23:49 smithi122 ceph-mon[93869]: pgmap v5174: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 117 KiB/s wr, 10 op/s 2023-03-09T17:23:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:23:49 smithi181 ceph-mon[94189]: pgmap v5174: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 117 KiB/s wr, 10 op/s 2023-03-09T17:23:50.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:23:49 smithi038 ceph-mon[92422]: pgmap v5174: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 117 KiB/s wr, 10 op/s 2023-03-09T17:23:51.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:23:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8741.055224 secs 2023-03-09T17:23:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:23:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8741.055224 secs 2023-03-09T17:23:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:23:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8741.055224 secs 2023-03-09T17:23:52.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:23:51 smithi122 ceph-mon[93869]: pgmap v5175: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 108 KiB/s wr, 9 op/s 2023-03-09T17:23:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:23:51 smithi181 ceph-mon[94189]: pgmap v5175: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 108 KiB/s wr, 9 op/s 2023-03-09T17:23:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:23:51 smithi038 ceph-mon[92422]: pgmap v5175: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 108 KiB/s wr, 9 op/s 2023-03-09T17:23:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9205.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:23:54.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:23:53 smithi122 ceph-mon[93869]: pgmap v5176: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 129 KiB/s wr, 12 op/s 2023-03-09T17:23:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:23:53 smithi181 ceph-mon[94189]: pgmap v5176: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 129 KiB/s wr, 12 op/s 2023-03-09T17:23:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:23:53 smithi038 ceph-mon[92422]: pgmap v5176: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 129 KiB/s wr, 12 op/s 2023-03-09T17:23:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:23:55 smithi122 ceph-mon[93869]: pgmap v5177: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 109 KiB/s wr, 10 op/s 2023-03-09T17:23:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:23:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8746.055284 secs 2023-03-09T17:23:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:23:55 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:23:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:23:55 smithi181 ceph-mon[94189]: pgmap v5177: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 109 KiB/s wr, 10 op/s 2023-03-09T17:23:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:23:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8746.055284 secs 2023-03-09T17:23:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:23:55 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:23:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:23:55 smithi038 ceph-mon[92422]: pgmap v5177: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 109 KiB/s wr, 10 op/s 2023-03-09T17:23:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:23:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8746.055284 secs 2023-03-09T17:23:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:23:55 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:23:57.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:23:56 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:23:57.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:23:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:23:57.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:23:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:23:57.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:23:56 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:23:57.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:23:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:23:57.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:23:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:23:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:23:56 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:23:57.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:23:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:23:57.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:23:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:23:58.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:23:57 smithi122 ceph-mon[93869]: pgmap v5178: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 136 KiB/s wr, 11 op/s 2023-03-09T17:23:58.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:23:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8748.321480 secs 2023-03-09T17:23:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:23:57 smithi181 ceph-mon[94189]: pgmap v5178: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 136 KiB/s wr, 11 op/s 2023-03-09T17:23:58.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:23:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8748.321480 secs 2023-03-09T17:23:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:23:57 smithi038 ceph-mon[92422]: pgmap v5178: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 136 KiB/s wr, 11 op/s 2023-03-09T17:23:58.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:23:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8748.321480 secs 2023-03-09T17:23:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9210.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:24:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:23:59 smithi122 ceph-mon[93869]: pgmap v5179: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 80 KiB/s wr, 9 op/s 2023-03-09T17:24:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:23:59 smithi181 ceph-mon[94189]: pgmap v5179: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 80 KiB/s wr, 9 op/s 2023-03-09T17:24:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:23:59 smithi038 ceph-mon[92422]: pgmap v5179: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 80 KiB/s wr, 9 op/s 2023-03-09T17:24:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:24:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8751.055363 secs 2023-03-09T17:24:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:24:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:24:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:24:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:24:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:24:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8751.055363 secs 2023-03-09T17:24:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:24:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:24:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:24:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:24:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:24:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8751.055363 secs 2023-03-09T17:24:01.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:24:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:24:01.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:24:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:24:02.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:24:01 smithi122 ceph-mon[93869]: pgmap v5180: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 72 KiB/s wr, 8 op/s 2023-03-09T17:24:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:24:01 smithi181 ceph-mon[94189]: pgmap v5180: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 72 KiB/s wr, 8 op/s 2023-03-09T17:24:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:24:01 smithi038 ceph-mon[92422]: pgmap v5180: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 72 KiB/s wr, 8 op/s 2023-03-09T17:24:03.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:24:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8753.321597 secs 2023-03-09T17:24:03.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:24:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8753.321597 secs 2023-03-09T17:24:03.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:24:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8753.321597 secs 2023-03-09T17:24:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9215.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:24:04.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:24:03 smithi122 ceph-mon[93869]: pgmap v5181: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 120 KiB/s wr, 14 op/s 2023-03-09T17:24:04.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:24:03 smithi181 ceph-mon[94189]: pgmap v5181: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 120 KiB/s wr, 14 op/s 2023-03-09T17:24:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:24:03 smithi038 ceph-mon[92422]: pgmap v5181: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 120 KiB/s wr, 14 op/s 2023-03-09T17:24:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:24:05 smithi122 ceph-mon[93869]: pgmap v5182: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 128 KiB/s wr, 23 op/s 2023-03-09T17:24:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:24:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8756.055424 secs 2023-03-09T17:24:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:24:05 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:24:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:24:05 smithi181 ceph-mon[94189]: pgmap v5182: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 128 KiB/s wr, 23 op/s 2023-03-09T17:24:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:24:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8756.055424 secs 2023-03-09T17:24:06.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:24:05 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:24:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:24:05 smithi038 ceph-mon[92422]: pgmap v5182: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 128 KiB/s wr, 23 op/s 2023-03-09T17:24:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:24:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8756.055424 secs 2023-03-09T17:24:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:24:05 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:24:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:24:07 smithi122 ceph-mon[93869]: pgmap v5183: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 113 KiB/s wr, 25 op/s 2023-03-09T17:24:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:24:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8758.321715 secs 2023-03-09T17:24:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:24:07 smithi181 ceph-mon[94189]: pgmap v5183: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 113 KiB/s wr, 25 op/s 2023-03-09T17:24:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:24:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8758.321715 secs 2023-03-09T17:24:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:24:07 smithi038 ceph-mon[92422]: pgmap v5183: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 113 KiB/s wr, 25 op/s 2023-03-09T17:24:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:24:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8758.321715 secs 2023-03-09T17:24:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9220.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:24:10.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:24:09 smithi122 ceph-mon[93869]: pgmap v5184: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 78 KiB/s wr, 23 op/s 2023-03-09T17:24:10.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:24:09 smithi122 ceph-mon[93869]: osdmap e537: 12 total, 12 up, 12 in 2023-03-09T17:24:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:24:09 smithi181 ceph-mon[94189]: pgmap v5184: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 78 KiB/s wr, 23 op/s 2023-03-09T17:24:10.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:24:09 smithi181 ceph-mon[94189]: osdmap e537: 12 total, 12 up, 12 in 2023-03-09T17:24:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:24:09 smithi038 ceph-mon[92422]: pgmap v5184: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 78 KiB/s wr, 23 op/s 2023-03-09T17:24:10.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:24:09 smithi038 ceph-mon[92422]: osdmap e537: 12 total, 12 up, 12 in 2023-03-09T17:24:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:24:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8761.055498 secs 2023-03-09T17:24:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:24:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8761.055498 secs 2023-03-09T17:24:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:24:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8761.055498 secs 2023-03-09T17:24:12.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:24:11 smithi122 ceph-mon[93869]: pgmap v5186: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 93 KiB/s wr, 26 op/s 2023-03-09T17:24:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:24:11 smithi181 ceph-mon[94189]: pgmap v5186: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 93 KiB/s wr, 26 op/s 2023-03-09T17:24:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:24:11 smithi038 ceph-mon[92422]: pgmap v5186: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 93 KiB/s wr, 26 op/s 2023-03-09T17:24:13.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:24:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8763.321839 secs 2023-03-09T17:24:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:24:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8763.321839 secs 2023-03-09T17:24:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:24:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8763.321839 secs 2023-03-09T17:24:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9225.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:24:14.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:24:13 smithi122 ceph-mon[93869]: pgmap v5187: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 51 KiB/s wr, 22 op/s 2023-03-09T17:24:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:24:13 smithi181 ceph-mon[94189]: pgmap v5187: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 51 KiB/s wr, 22 op/s 2023-03-09T17:24:14.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:24:13 smithi038 ceph-mon[92422]: pgmap v5187: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 51 KiB/s wr, 22 op/s 2023-03-09T17:24:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:24:15 smithi122 ceph-mon[93869]: pgmap v5188: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 7 op/s 2023-03-09T17:24:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:24:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8766.055559 secs 2023-03-09T17:24:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:24:15 smithi122 ceph-mon[93869]: osdmap e538: 12 total, 12 up, 12 in 2023-03-09T17:24:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:24:15 smithi181 ceph-mon[94189]: pgmap v5188: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 7 op/s 2023-03-09T17:24:16.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:24:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8766.055559 secs 2023-03-09T17:24:16.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:24:15 smithi181 ceph-mon[94189]: osdmap e538: 12 total, 12 up, 12 in 2023-03-09T17:24:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:24:15 smithi038 ceph-mon[92422]: pgmap v5188: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 7 op/s 2023-03-09T17:24:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:24:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8766.055559 secs 2023-03-09T17:24:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:24:15 smithi038 ceph-mon[92422]: osdmap e538: 12 total, 12 up, 12 in 2023-03-09T17:24:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:24:17 smithi122 ceph-mon[93869]: pgmap v5190: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 8 op/s 2023-03-09T17:24:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:24:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8768.321950 secs 2023-03-09T17:24:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:24:17 smithi181 ceph-mon[94189]: pgmap v5190: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 8 op/s 2023-03-09T17:24:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:24:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8768.321950 secs 2023-03-09T17:24:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:24:17 smithi038 ceph-mon[92422]: pgmap v5190: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 8 op/s 2023-03-09T17:24:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:24:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8768.321950 secs 2023-03-09T17:24:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9230.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:24:20.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:24:19 smithi122 ceph-mon[93869]: pgmap v5191: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 8 op/s 2023-03-09T17:24:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:24:19 smithi181 ceph-mon[94189]: pgmap v5191: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 8 op/s 2023-03-09T17:24:20.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:24:19 smithi038 ceph-mon[92422]: pgmap v5191: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 8 op/s 2023-03-09T17:24:21.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:24:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8771.055620 secs 2023-03-09T17:24:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:24:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8771.055620 secs 2023-03-09T17:24:21.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:24:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8771.055620 secs 2023-03-09T17:24:22.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:24:21 smithi122 ceph-mon[93869]: pgmap v5192: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 18 KiB/s wr, 7 op/s 2023-03-09T17:24:22.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:24:21 smithi181 ceph-mon[94189]: pgmap v5192: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 18 KiB/s wr, 7 op/s 2023-03-09T17:24:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:24:21 smithi038 ceph-mon[92422]: pgmap v5192: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 18 KiB/s wr, 7 op/s 2023-03-09T17:24:23.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:24:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8773.322067 secs 2023-03-09T17:24:23.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:24:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8773.322067 secs 2023-03-09T17:24:23.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:24:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8773.322067 secs 2023-03-09T17:24:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9235.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:24:24.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:24:23 smithi122 ceph-mon[93869]: pgmap v5193: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 26 KiB/s wr, 8 op/s 2023-03-09T17:24:24.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:24:23 smithi181 ceph-mon[94189]: pgmap v5193: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 26 KiB/s wr, 8 op/s 2023-03-09T17:24:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:24:23 smithi038 ceph-mon[92422]: pgmap v5193: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 26 KiB/s wr, 8 op/s 2023-03-09T17:24:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:24:25 smithi181 ceph-mon[94189]: pgmap v5194: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 25 KiB/s wr, 8 op/s 2023-03-09T17:24:26.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:24:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8776.055684 secs 2023-03-09T17:24:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:24:25 smithi038 ceph-mon[92422]: pgmap v5194: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 25 KiB/s wr, 8 op/s 2023-03-09T17:24:26.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:24:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8776.055684 secs 2023-03-09T17:24:26.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:24:25 smithi122 ceph-mon[93869]: pgmap v5194: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 25 KiB/s wr, 8 op/s 2023-03-09T17:24:26.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:24:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8776.055684 secs 2023-03-09T17:24:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:24:27 smithi038 ceph-mon[92422]: pgmap v5195: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 24 KiB/s wr, 10 op/s 2023-03-09T17:24:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:24:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8778.322204 secs 2023-03-09T17:24:28.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:24:27 smithi122 ceph-mon[93869]: pgmap v5195: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 24 KiB/s wr, 10 op/s 2023-03-09T17:24:28.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:24:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8778.322204 secs 2023-03-09T17:24:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:24:27 smithi181 ceph-mon[94189]: pgmap v5195: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 24 KiB/s wr, 10 op/s 2023-03-09T17:24:28.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:24:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8778.322204 secs 2023-03-09T17:24:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9240.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:24:30.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:24:29 smithi122 ceph-mon[93869]: pgmap v5196: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 21 KiB/s wr, 7 op/s 2023-03-09T17:24:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:24:30 smithi181 ceph-mon[94189]: pgmap v5196: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 21 KiB/s wr, 7 op/s 2023-03-09T17:24:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:24:29 smithi038 ceph-mon[92422]: pgmap v5196: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 21 KiB/s wr, 7 op/s 2023-03-09T17:24:31.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:24:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8781.055745 secs 2023-03-09T17:24:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:24:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8781.055745 secs 2023-03-09T17:24:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:24:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8781.055745 secs 2023-03-09T17:24:32.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:24:32 smithi122 ceph-mon[93869]: pgmap v5197: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 21 KiB/s wr, 6 op/s 2023-03-09T17:24:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:24:32 smithi181 ceph-mon[94189]: pgmap v5197: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 21 KiB/s wr, 6 op/s 2023-03-09T17:24:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:24:32 smithi038 ceph-mon[92422]: pgmap v5197: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 21 KiB/s wr, 6 op/s 2023-03-09T17:24:33.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9245.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:24:34.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:24:34 smithi122 ceph-mon[93869]: pgmap v5198: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 21 KiB/s wr, 9 op/s 2023-03-09T17:24:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:24:34 smithi181 ceph-mon[94189]: pgmap v5198: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 21 KiB/s wr, 9 op/s 2023-03-09T17:24:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:24:34 smithi038 ceph-mon[92422]: pgmap v5198: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 21 KiB/s wr, 9 op/s 2023-03-09T17:24:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:24:36 smithi122 ceph-mon[93869]: pgmap v5199: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 7 op/s 2023-03-09T17:24:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:24:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8786.055802 secs 2023-03-09T17:24:36.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:24:36 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:24:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:24:36 smithi181 ceph-mon[94189]: pgmap v5199: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 7 op/s 2023-03-09T17:24:36.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:24:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8786.055802 secs 2023-03-09T17:24:36.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:24:36 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:24:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:24:36 smithi038 ceph-mon[92422]: pgmap v5199: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 7 op/s 2023-03-09T17:24:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:24:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8786.055802 secs 2023-03-09T17:24:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:24:36 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:24:37.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:24:37 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:24:37.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:24:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:24:37.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:24:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:24:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:24:37 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:24:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:24:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:24:37.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:24:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:24:37.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:24:37 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:24:37.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:24:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:24:37.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:24:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:24:38.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:24:38 smithi122 ceph-mon[93869]: pgmap v5200: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 19 KiB/s wr, 9 op/s 2023-03-09T17:24:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:24:38 smithi181 ceph-mon[94189]: pgmap v5200: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 19 KiB/s wr, 9 op/s 2023-03-09T17:24:38.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:24:38 smithi038 ceph-mon[92422]: pgmap v5200: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 19 KiB/s wr, 9 op/s 2023-03-09T17:24:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9250.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:24:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:24:40 smithi122 ceph-mon[93869]: pgmap v5201: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T17:24:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:24:40 smithi181 ceph-mon[94189]: pgmap v5201: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T17:24:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:24:40 smithi038 ceph-mon[92422]: pgmap v5201: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T17:24:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:24:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8791.055887 secs 2023-03-09T17:24:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:24:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8791.055887 secs 2023-03-09T17:24:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:24:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8791.055887 secs 2023-03-09T17:24:42.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:24:42 smithi122 ceph-mon[93869]: pgmap v5202: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T17:24:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:24:42 smithi181 ceph-mon[94189]: pgmap v5202: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T17:24:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:24:42 smithi038 ceph-mon[92422]: pgmap v5202: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T17:24:43.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:24:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:24:43.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:24:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:24:43.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:24:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:24:43.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:24:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:24:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:24:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:24:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:24:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:24:43.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:24:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:24:43.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:24:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:24:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:24:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:24:43.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:24:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:24:43.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:24:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:24:43.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:24:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:24:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9255.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:24:44.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:24:44 smithi122 ceph-mon[93869]: pgmap v5203: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 85 KiB/s wr, 10 op/s 2023-03-09T17:24:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:24:44 smithi181 ceph-mon[94189]: pgmap v5203: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 85 KiB/s wr, 10 op/s 2023-03-09T17:24:44.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:24:44 smithi038 ceph-mon[92422]: pgmap v5203: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 85 KiB/s wr, 10 op/s 2023-03-09T17:24:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:24:46 smithi122 ceph-mon[93869]: pgmap v5204: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 101 KiB/s wr, 9 op/s 2023-03-09T17:24:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:24:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8796.055948 secs 2023-03-09T17:24:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:24:46 smithi181 ceph-mon[94189]: pgmap v5204: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 101 KiB/s wr, 9 op/s 2023-03-09T17:24:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:24:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8796.055948 secs 2023-03-09T17:24:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:24:46 smithi038 ceph-mon[92422]: pgmap v5204: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 101 KiB/s wr, 9 op/s 2023-03-09T17:24:46.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:24:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8796.055948 secs 2023-03-09T17:24:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:24:48 smithi122 ceph-mon[93869]: pgmap v5205: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 128 KiB/s wr, 12 op/s 2023-03-09T17:24:48.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:24:48 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 8798.322690 secs 2023-03-09T17:24:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:24:48 smithi181 ceph-mon[94189]: pgmap v5205: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 128 KiB/s wr, 12 op/s 2023-03-09T17:24:48.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:24:48 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 8798.322690 secs 2023-03-09T17:24:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:24:48 smithi038 ceph-mon[92422]: pgmap v5205: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 128 KiB/s wr, 12 op/s 2023-03-09T17:24:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:24:48 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 8798.322690 secs 2023-03-09T17:24:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9260.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:24:49.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:24:49 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:24:49.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:24:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:24:49.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:24:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:24:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:24:49 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:24:49.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:24:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:24:49.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:24:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:24:49.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:24:49 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:24:49.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:24:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:24:49.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:24:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:24:50.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:24:50 smithi122 ceph-mon[93869]: pgmap v5206: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 117 KiB/s wr, 11 op/s 2023-03-09T17:24:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:24:50 smithi181 ceph-mon[94189]: pgmap v5206: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 117 KiB/s wr, 11 op/s 2023-03-09T17:24:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:24:50 smithi038 ceph-mon[92422]: pgmap v5206: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 117 KiB/s wr, 11 op/s 2023-03-09T17:24:51.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:24:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8801.056008 secs 2023-03-09T17:24:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:24:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8801.056008 secs 2023-03-09T17:24:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:24:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8801.056008 secs 2023-03-09T17:24:52.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:24:52 smithi122 ceph-mon[93869]: pgmap v5207: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 109 KiB/s wr, 10 op/s 2023-03-09T17:24:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:24:52 smithi181 ceph-mon[94189]: pgmap v5207: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 109 KiB/s wr, 10 op/s 2023-03-09T17:24:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:24:52 smithi038 ceph-mon[92422]: pgmap v5207: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 109 KiB/s wr, 10 op/s 2023-03-09T17:24:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:24:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8803.322781 secs 2023-03-09T17:24:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:24:53 smithi122 ceph-mon[93869]: pgmap v5208: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 176 KiB/s wr, 13 op/s 2023-03-09T17:24:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:24:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8803.322781 secs 2023-03-09T17:24:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:24:53 smithi181 ceph-mon[94189]: pgmap v5208: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 176 KiB/s wr, 13 op/s 2023-03-09T17:24:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:24:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8803.322781 secs 2023-03-09T17:24:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:24:53 smithi038 ceph-mon[92422]: pgmap v5208: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 176 KiB/s wr, 13 op/s 2023-03-09T17:24:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9265.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:24:55.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:24:55 smithi122 ceph-mon[93869]: pgmap v5209: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 126 KiB/s wr, 11 op/s 2023-03-09T17:24:55.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:24:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8806.056070 secs 2023-03-09T17:24:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:24:55 smithi181 ceph-mon[94189]: pgmap v5209: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 126 KiB/s wr, 11 op/s 2023-03-09T17:24:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:24:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8806.056070 secs 2023-03-09T17:24:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:24:55 smithi038 ceph-mon[92422]: pgmap v5209: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 126 KiB/s wr, 11 op/s 2023-03-09T17:24:55.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:24:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8806.056070 secs 2023-03-09T17:24:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:24:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:24:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:24:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:24:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:24:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:24:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:24:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:24:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:24:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:24:56.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:24:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:24:57.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:24:57 smithi122 ceph-mon[93869]: pgmap v5210: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 111 KiB/s wr, 12 op/s 2023-03-09T17:24:57.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:24:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8808.322878 secs 2023-03-09T17:24:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:24:57 smithi181 ceph-mon[94189]: pgmap v5210: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 111 KiB/s wr, 12 op/s 2023-03-09T17:24:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:24:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8808.322878 secs 2023-03-09T17:24:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:24:57 smithi038 ceph-mon[92422]: pgmap v5210: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 111 KiB/s wr, 12 op/s 2023-03-09T17:24:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:24:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8808.322878 secs 2023-03-09T17:24:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9270.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:24:59.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:24:59 smithi122 ceph-mon[93869]: pgmap v5211: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 84 KiB/s wr, 9 op/s 2023-03-09T17:24:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:24:59 smithi181 ceph-mon[94189]: pgmap v5211: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 84 KiB/s wr, 9 op/s 2023-03-09T17:24:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:24:59 smithi038 ceph-mon[92422]: pgmap v5211: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 84 KiB/s wr, 9 op/s 2023-03-09T17:25:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:25:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8811.056132 secs 2023-03-09T17:25:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:25:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8811.056132 secs 2023-03-09T17:25:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:25:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8811.056132 secs 2023-03-09T17:25:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:25:01 smithi122 ceph-mon[93869]: pgmap v5212: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 76 KiB/s wr, 8 op/s 2023-03-09T17:25:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:25:01 smithi181 ceph-mon[94189]: pgmap v5212: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 76 KiB/s wr, 8 op/s 2023-03-09T17:25:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:25:01 smithi038 ceph-mon[92422]: pgmap v5212: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 76 KiB/s wr, 8 op/s 2023-03-09T17:25:02.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:25:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8813.322998 secs 2023-03-09T17:25:02.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:25:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8813.322998 secs 2023-03-09T17:25:03.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:25:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8813.322998 secs 2023-03-09T17:25:03.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9275.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:25:03.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:25:03 smithi122 ceph-mon[93869]: pgmap v5213: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 142 KiB/s wr, 13 op/s 2023-03-09T17:25:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:25:03 smithi181 ceph-mon[94189]: pgmap v5213: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 142 KiB/s wr, 13 op/s 2023-03-09T17:25:04.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:25:03 smithi038 ceph-mon[92422]: pgmap v5213: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 142 KiB/s wr, 13 op/s 2023-03-09T17:25:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:25:05 smithi122 ceph-mon[93869]: pgmap v5214: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 96 KiB/s wr, 22 op/s 2023-03-09T17:25:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:25:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8816.056193 secs 2023-03-09T17:25:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:25:05 smithi181 ceph-mon[94189]: pgmap v5214: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 96 KiB/s wr, 22 op/s 2023-03-09T17:25:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:25:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8816.056193 secs 2023-03-09T17:25:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:25:05 smithi038 ceph-mon[92422]: pgmap v5214: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 96 KiB/s wr, 22 op/s 2023-03-09T17:25:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:25:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8816.056193 secs 2023-03-09T17:25:07.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:25:07 smithi122 ceph-mon[93869]: pgmap v5215: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 102 KiB/s wr, 23 op/s 2023-03-09T17:25:07.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:25:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8818.323138 secs 2023-03-09T17:25:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:25:07 smithi181 ceph-mon[94189]: pgmap v5215: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 102 KiB/s wr, 23 op/s 2023-03-09T17:25:07.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:25:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8818.323138 secs 2023-03-09T17:25:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:25:07 smithi038 ceph-mon[92422]: pgmap v5215: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 102 KiB/s wr, 23 op/s 2023-03-09T17:25:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:25:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8818.323138 secs 2023-03-09T17:25:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9280.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:25:09.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:25:09 smithi122 ceph-mon[93869]: pgmap v5216: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 102 KiB/s wr, 21 op/s 2023-03-09T17:25:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:25:09 smithi181 ceph-mon[94189]: pgmap v5216: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 102 KiB/s wr, 21 op/s 2023-03-09T17:25:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:25:09 smithi038 ceph-mon[92422]: pgmap v5216: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 102 KiB/s wr, 21 op/s 2023-03-09T17:25:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:25:10 smithi122 ceph-mon[93869]: osdmap e539: 12 total, 12 up, 12 in 2023-03-09T17:25:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:25:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8821.056258 secs 2023-03-09T17:25:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:25:10 smithi181 ceph-mon[94189]: osdmap e539: 12 total, 12 up, 12 in 2023-03-09T17:25:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:25:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8821.056258 secs 2023-03-09T17:25:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:25:10 smithi038 ceph-mon[92422]: osdmap e539: 12 total, 12 up, 12 in 2023-03-09T17:25:11.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:25:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8821.056258 secs 2023-03-09T17:25:11.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:25:11 smithi122 ceph-mon[93869]: pgmap v5218: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 122 KiB/s wr, 24 op/s 2023-03-09T17:25:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:25:11 smithi181 ceph-mon[94189]: pgmap v5218: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 122 KiB/s wr, 24 op/s 2023-03-09T17:25:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:25:11 smithi038 ceph-mon[92422]: pgmap v5218: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 122 KiB/s wr, 24 op/s 2023-03-09T17:25:12.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:25:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8823.323238 secs 2023-03-09T17:25:12.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:25:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8823.323238 secs 2023-03-09T17:25:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:25:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8823.323238 secs 2023-03-09T17:25:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9285.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:25:13.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:25:13 smithi122 ceph-mon[93869]: pgmap v5219: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 42 KiB/s wr, 21 op/s 2023-03-09T17:25:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:25:13 smithi181 ceph-mon[94189]: pgmap v5219: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 42 KiB/s wr, 21 op/s 2023-03-09T17:25:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:25:13 smithi038 ceph-mon[92422]: pgmap v5219: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 42 KiB/s wr, 21 op/s 2023-03-09T17:25:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:25:15 smithi122 ceph-mon[93869]: pgmap v5220: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 16 KiB/s wr, 6 op/s 2023-03-09T17:25:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:25:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8826.056317 secs 2023-03-09T17:25:15.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:25:15 smithi122 ceph-mon[93869]: osdmap e540: 12 total, 12 up, 12 in 2023-03-09T17:25:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:25:15 smithi181 ceph-mon[94189]: pgmap v5220: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 16 KiB/s wr, 6 op/s 2023-03-09T17:25:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:25:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8826.056317 secs 2023-03-09T17:25:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:25:15 smithi181 ceph-mon[94189]: osdmap e540: 12 total, 12 up, 12 in 2023-03-09T17:25:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:25:15 smithi038 ceph-mon[92422]: pgmap v5220: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 16 KiB/s wr, 6 op/s 2023-03-09T17:25:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:25:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8826.056317 secs 2023-03-09T17:25:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:25:15 smithi038 ceph-mon[92422]: osdmap e540: 12 total, 12 up, 12 in 2023-03-09T17:25:17.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:25:17 smithi122 ceph-mon[93869]: pgmap v5222: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 511 B/s wr, 7 op/s 2023-03-09T17:25:17.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:25:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8828.323306 secs 2023-03-09T17:25:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:25:17 smithi181 ceph-mon[94189]: pgmap v5222: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 511 B/s wr, 7 op/s 2023-03-09T17:25:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:25:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8828.323306 secs 2023-03-09T17:25:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:25:17 smithi038 ceph-mon[92422]: pgmap v5222: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 511 B/s wr, 7 op/s 2023-03-09T17:25:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:25:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8828.323306 secs 2023-03-09T17:25:18.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9290.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:25:19.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:25:19 smithi122 ceph-mon[93869]: pgmap v5223: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 710 B/s wr, 7 op/s 2023-03-09T17:25:19.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:25:19 smithi181 ceph-mon[94189]: pgmap v5223: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 710 B/s wr, 7 op/s 2023-03-09T17:25:20.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:25:19 smithi038 ceph-mon[92422]: pgmap v5223: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 710 B/s wr, 7 op/s 2023-03-09T17:25:20.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:25:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8831.056379 secs 2023-03-09T17:25:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:25:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8831.056379 secs 2023-03-09T17:25:21.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:25:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8831.056379 secs 2023-03-09T17:25:21.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:25:21 smithi122 ceph-mon[93869]: pgmap v5224: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 614 B/s wr, 6 op/s 2023-03-09T17:25:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:25:21 smithi181 ceph-mon[94189]: pgmap v5224: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 614 B/s wr, 6 op/s 2023-03-09T17:25:22.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:25:21 smithi038 ceph-mon[92422]: pgmap v5224: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 614 B/s wr, 6 op/s 2023-03-09T17:25:22.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:25:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8833.323426 secs 2023-03-09T17:25:22.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:25:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8833.323426 secs 2023-03-09T17:25:22.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:25:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8833.323426 secs 2023-03-09T17:25:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9295.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:25:23.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:25:23 smithi122 ceph-mon[93869]: pgmap v5225: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 614 B/s wr, 6 op/s 2023-03-09T17:25:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:25:23 smithi181 ceph-mon[94189]: pgmap v5225: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 614 B/s wr, 6 op/s 2023-03-09T17:25:24.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:25:23 smithi038 ceph-mon[92422]: pgmap v5225: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 614 B/s wr, 6 op/s 2023-03-09T17:25:25.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:25:25 smithi122 ceph-mon[93869]: pgmap v5226: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.1 KiB/s wr, 7 op/s 2023-03-09T17:25:25.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:25:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8836.056438 secs 2023-03-09T17:25:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:25:25 smithi181 ceph-mon[94189]: pgmap v5226: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.1 KiB/s wr, 7 op/s 2023-03-09T17:25:25.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:25:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8836.056438 secs 2023-03-09T17:25:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:25:25 smithi038 ceph-mon[92422]: pgmap v5226: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.1 KiB/s wr, 7 op/s 2023-03-09T17:25:26.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:25:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8836.056438 secs 2023-03-09T17:25:27.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:25:27 smithi122 ceph-mon[93869]: pgmap v5227: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 24 KiB/s wr, 9 op/s 2023-03-09T17:25:27.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:25:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8838.323547 secs 2023-03-09T17:25:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:25:27 smithi181 ceph-mon[94189]: pgmap v5227: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 24 KiB/s wr, 9 op/s 2023-03-09T17:25:27.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:25:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8838.323547 secs 2023-03-09T17:25:28.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:25:27 smithi038 ceph-mon[92422]: pgmap v5227: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 24 KiB/s wr, 9 op/s 2023-03-09T17:25:28.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:25:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8838.323547 secs 2023-03-09T17:25:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9300.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:25:29.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:25:29 smithi122 ceph-mon[93869]: pgmap v5228: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 7 op/s 2023-03-09T17:25:29.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:25:29 smithi181 ceph-mon[94189]: pgmap v5228: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 7 op/s 2023-03-09T17:25:30.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:25:29 smithi038 ceph-mon[92422]: pgmap v5228: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 7 op/s 2023-03-09T17:25:30.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:25:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8841.056498 secs 2023-03-09T17:25:30.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:25:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8841.056498 secs 2023-03-09T17:25:30.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:25:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8841.056498 secs 2023-03-09T17:25:31.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:25:31 smithi122 ceph-mon[93869]: pgmap v5229: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 6 op/s 2023-03-09T17:25:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:25:31 smithi181 ceph-mon[94189]: pgmap v5229: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 6 op/s 2023-03-09T17:25:32.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:25:31 smithi038 ceph-mon[92422]: pgmap v5229: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 6 op/s 2023-03-09T17:25:32.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:25:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8843.323701 secs 2023-03-09T17:25:32.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:25:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8843.323701 secs 2023-03-09T17:25:33.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:25:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8843.323701 secs 2023-03-09T17:25:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9305.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:25:33.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:25:33 smithi181 ceph-mon[94189]: pgmap v5230: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 8 op/s 2023-03-09T17:25:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:25:33 smithi038 ceph-mon[92422]: pgmap v5230: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 8 op/s 2023-03-09T17:25:34.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:25:33 smithi122 ceph-mon[93869]: pgmap v5230: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 8 op/s 2023-03-09T17:25:36.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:25:35 smithi038 ceph-mon[92422]: pgmap v5231: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 6 op/s 2023-03-09T17:25:36.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:25:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8846.056559 secs 2023-03-09T17:25:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:25:35 smithi122 ceph-mon[93869]: pgmap v5231: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 6 op/s 2023-03-09T17:25:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:25:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8846.056559 secs 2023-03-09T17:25:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:25:35 smithi181 ceph-mon[94189]: pgmap v5231: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 6 op/s 2023-03-09T17:25:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:25:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8846.056559 secs 2023-03-09T17:25:38.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:25:37 smithi038 ceph-mon[92422]: pgmap v5232: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 40 KiB/s wr, 8 op/s 2023-03-09T17:25:38.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:25:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8848.323824 secs 2023-03-09T17:25:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:25:37 smithi122 ceph-mon[93869]: pgmap v5232: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 40 KiB/s wr, 8 op/s 2023-03-09T17:25:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:25:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8848.323824 secs 2023-03-09T17:25:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:25:37 smithi181 ceph-mon[94189]: pgmap v5232: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 40 KiB/s wr, 8 op/s 2023-03-09T17:25:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:25:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8848.323824 secs 2023-03-09T17:25:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9310.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:25:40.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:25:39 smithi038 ceph-mon[92422]: pgmap v5233: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 6 op/s 2023-03-09T17:25:40.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:25:39 smithi122 ceph-mon[93869]: pgmap v5233: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 6 op/s 2023-03-09T17:25:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:25:39 smithi181 ceph-mon[94189]: pgmap v5233: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 6 op/s 2023-03-09T17:25:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:25:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8851.056618 secs 2023-03-09T17:25:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:25:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8851.056618 secs 2023-03-09T17:25:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:25:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8851.056618 secs 2023-03-09T17:25:42.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:25:41 smithi122 ceph-mon[93869]: pgmap v5234: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 5 op/s 2023-03-09T17:25:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:25:41 smithi181 ceph-mon[94189]: pgmap v5234: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 5 op/s 2023-03-09T17:25:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:25:41 smithi038 ceph-mon[92422]: pgmap v5234: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 5 op/s 2023-03-09T17:25:43.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:25:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8853.323924 secs 2023-03-09T17:25:43.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:25:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8853.323924 secs 2023-03-09T17:25:43.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:25:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8853.323924 secs 2023-03-09T17:25:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9315.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:25:44.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:25:43 smithi122 ceph-mon[93869]: pgmap v5235: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 54 KiB/s wr, 8 op/s 2023-03-09T17:25:44.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:25:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:25:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:25:43 smithi181 ceph-mon[94189]: pgmap v5235: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 54 KiB/s wr, 8 op/s 2023-03-09T17:25:44.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:25:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:25:44.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:25:43 smithi038 ceph-mon[92422]: pgmap v5235: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 54 KiB/s wr, 8 op/s 2023-03-09T17:25:44.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:25:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:25:45.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:25:44 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:25:45.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:25:44 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:25:45.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:25:44 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:25:45.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:25:44 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:25:45.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:25:44 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:25:45.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:25:44 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:25:45.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:25:44 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:25:45.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:25:44 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:25:45.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:25:44 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:25:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:25:45 smithi122 ceph-mon[93869]: pgmap v5236: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 61 KiB/s wr, 7 op/s 2023-03-09T17:25:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:25:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8856.056680 secs 2023-03-09T17:25:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:25:45 smithi181 ceph-mon[94189]: pgmap v5236: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 61 KiB/s wr, 7 op/s 2023-03-09T17:25:46.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:25:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8856.056680 secs 2023-03-09T17:25:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:25:45 smithi038 ceph-mon[92422]: pgmap v5236: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 61 KiB/s wr, 7 op/s 2023-03-09T17:25:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:25:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8856.056680 secs 2023-03-09T17:25:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:25:47 smithi122 ceph-mon[93869]: pgmap v5237: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 113 KiB/s wr, 10 op/s 2023-03-09T17:25:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:25:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8858.324064 secs 2023-03-09T17:25:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:25:47 smithi181 ceph-mon[94189]: pgmap v5237: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 113 KiB/s wr, 10 op/s 2023-03-09T17:25:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:25:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8858.324064 secs 2023-03-09T17:25:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:25:47 smithi038 ceph-mon[92422]: pgmap v5237: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 113 KiB/s wr, 10 op/s 2023-03-09T17:25:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:25:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8858.324064 secs 2023-03-09T17:25:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9320.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:25:50.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:25:49 smithi122 ceph-mon[93869]: pgmap v5238: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 102 KiB/s wr, 9 op/s 2023-03-09T17:25:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:25:49 smithi181 ceph-mon[94189]: pgmap v5238: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 102 KiB/s wr, 9 op/s 2023-03-09T17:25:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:25:49 smithi038 ceph-mon[92422]: pgmap v5238: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 102 KiB/s wr, 9 op/s 2023-03-09T17:25:51.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:25:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8861.056745 secs 2023-03-09T17:25:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:25:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8861.056745 secs 2023-03-09T17:25:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:25:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8861.056745 secs 2023-03-09T17:25:52.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:25:51 smithi122 ceph-mon[93869]: pgmap v5239: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 102 KiB/s wr, 8 op/s 2023-03-09T17:25:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:25:51 smithi181 ceph-mon[94189]: pgmap v5239: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 102 KiB/s wr, 8 op/s 2023-03-09T17:25:52.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:25:51 smithi038 ceph-mon[92422]: pgmap v5239: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 102 KiB/s wr, 8 op/s 2023-03-09T17:25:53.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:25:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8863.324180 secs 2023-03-09T17:25:53.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:25:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8863.324180 secs 2023-03-09T17:25:53.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:25:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8863.324180 secs 2023-03-09T17:25:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9325.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:25:54.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:25:53 smithi122 ceph-mon[93869]: pgmap v5240: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 162 KiB/s wr, 13 op/s 2023-03-09T17:25:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:25:53 smithi181 ceph-mon[94189]: pgmap v5240: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 162 KiB/s wr, 13 op/s 2023-03-09T17:25:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:25:53 smithi038 ceph-mon[92422]: pgmap v5240: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 162 KiB/s wr, 13 op/s 2023-03-09T17:25:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:25:55 smithi122 ceph-mon[93869]: pgmap v5241: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 135 KiB/s wr, 11 op/s 2023-03-09T17:25:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:25:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8866.056804 secs 2023-03-09T17:25:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:25:55 smithi181 ceph-mon[94189]: pgmap v5241: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 135 KiB/s wr, 11 op/s 2023-03-09T17:25:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:25:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8866.056804 secs 2023-03-09T17:25:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:25:55 smithi038 ceph-mon[92422]: pgmap v5241: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 135 KiB/s wr, 11 op/s 2023-03-09T17:25:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:25:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8866.056804 secs 2023-03-09T17:25:58.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:25:57 smithi122 ceph-mon[93869]: pgmap v5242: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 133 KiB/s wr, 13 op/s 2023-03-09T17:25:58.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:25:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8868.324324 secs 2023-03-09T17:25:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:25:57 smithi181 ceph-mon[94189]: pgmap v5242: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 133 KiB/s wr, 13 op/s 2023-03-09T17:25:58.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:25:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8868.324324 secs 2023-03-09T17:25:58.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:25:57 smithi038 ceph-mon[92422]: pgmap v5242: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 133 KiB/s wr, 13 op/s 2023-03-09T17:25:58.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:25:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8868.324324 secs 2023-03-09T17:25:58.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9330.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:26:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:25:59 smithi122 ceph-mon[93869]: pgmap v5243: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 96 KiB/s wr, 11 op/s 2023-03-09T17:26:00.238 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:25:59 smithi181 ceph-mon[94189]: pgmap v5243: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 96 KiB/s wr, 11 op/s 2023-03-09T17:26:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:25:59 smithi038 ceph-mon[92422]: pgmap v5243: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 96 KiB/s wr, 11 op/s 2023-03-09T17:26:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:26:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8871.056863 secs 2023-03-09T17:26:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:26:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8871.056863 secs 2023-03-09T17:26:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:26:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8871.056863 secs 2023-03-09T17:26:02.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:26:01 smithi122 ceph-mon[93869]: pgmap v5244: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 88 KiB/s wr, 9 op/s 2023-03-09T17:26:02.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:26:01 smithi181 ceph-mon[94189]: pgmap v5244: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 88 KiB/s wr, 9 op/s 2023-03-09T17:26:02.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:26:01 smithi038 ceph-mon[92422]: pgmap v5244: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 88 KiB/s wr, 9 op/s 2023-03-09T17:26:03.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:26:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8873.324445 secs 2023-03-09T17:26:03.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:26:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8873.324445 secs 2023-03-09T17:26:03.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:26:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8873.324445 secs 2023-03-09T17:26:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9335.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:26:04.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:26:03 smithi122 ceph-mon[93869]: pgmap v5245: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 176 KiB/s wr, 15 op/s 2023-03-09T17:26:04.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:26:03 smithi181 ceph-mon[94189]: pgmap v5245: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 176 KiB/s wr, 15 op/s 2023-03-09T17:26:04.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:26:03 smithi038 ceph-mon[92422]: pgmap v5245: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 176 KiB/s wr, 15 op/s 2023-03-09T17:26:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:26:05 smithi122 ceph-mon[93869]: pgmap v5246: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 145 KiB/s wr, 23 op/s 2023-03-09T17:26:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:26:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8876.056922 secs 2023-03-09T17:26:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:26:05 smithi181 ceph-mon[94189]: pgmap v5246: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 145 KiB/s wr, 23 op/s 2023-03-09T17:26:06.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:26:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8876.056922 secs 2023-03-09T17:26:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:26:05 smithi038 ceph-mon[92422]: pgmap v5246: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 145 KiB/s wr, 23 op/s 2023-03-09T17:26:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:26:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8876.056922 secs 2023-03-09T17:26:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:26:07 smithi122 ceph-mon[93869]: pgmap v5247: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 152 KiB/s wr, 25 op/s 2023-03-09T17:26:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:26:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8878.324542 secs 2023-03-09T17:26:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:26:07 smithi181 ceph-mon[94189]: pgmap v5247: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 152 KiB/s wr, 25 op/s 2023-03-09T17:26:08.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:26:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8878.324542 secs 2023-03-09T17:26:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:26:07 smithi038 ceph-mon[92422]: pgmap v5247: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 152 KiB/s wr, 25 op/s 2023-03-09T17:26:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:26:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8878.324542 secs 2023-03-09T17:26:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9340.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:26:10.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:26:09 smithi122 ceph-mon[93869]: pgmap v5248: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 146 KiB/s wr, 23 op/s 2023-03-09T17:26:10.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:26:09 smithi122 ceph-mon[93869]: osdmap e541: 12 total, 12 up, 12 in 2023-03-09T17:26:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:26:09 smithi181 ceph-mon[94189]: pgmap v5248: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 146 KiB/s wr, 23 op/s 2023-03-09T17:26:10.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:26:09 smithi181 ceph-mon[94189]: osdmap e541: 12 total, 12 up, 12 in 2023-03-09T17:26:10.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:26:09 smithi038 ceph-mon[92422]: pgmap v5248: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 146 KiB/s wr, 23 op/s 2023-03-09T17:26:10.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:26:09 smithi038 ceph-mon[92422]: osdmap e541: 12 total, 12 up, 12 in 2023-03-09T17:26:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:26:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8881.056981 secs 2023-03-09T17:26:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:26:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8881.056981 secs 2023-03-09T17:26:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:26:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8881.056981 secs 2023-03-09T17:26:12.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:26:11 smithi122 ceph-mon[93869]: pgmap v5250: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 158 KiB/s wr, 26 op/s 2023-03-09T17:26:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:26:11 smithi181 ceph-mon[94189]: pgmap v5250: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 158 KiB/s wr, 26 op/s 2023-03-09T17:26:12.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:26:11 smithi038 ceph-mon[92422]: pgmap v5250: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 158 KiB/s wr, 26 op/s 2023-03-09T17:26:13.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:26:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8883.324656 secs 2023-03-09T17:26:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:26:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8883.324656 secs 2023-03-09T17:26:13.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:26:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8883.324656 secs 2023-03-09T17:26:13.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9345.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:26:14.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:26:13 smithi122 ceph-mon[93869]: pgmap v5251: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 52 KiB/s wr, 22 op/s 2023-03-09T17:26:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:26:13 smithi181 ceph-mon[94189]: pgmap v5251: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 52 KiB/s wr, 22 op/s 2023-03-09T17:26:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:26:13 smithi038 ceph-mon[92422]: pgmap v5251: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 52 KiB/s wr, 22 op/s 2023-03-09T17:26:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:26:15 smithi122 ceph-mon[93869]: pgmap v5252: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 26 KiB/s wr, 8 op/s 2023-03-09T17:26:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:26:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8886.057043 secs 2023-03-09T17:26:16.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:26:15 smithi122 ceph-mon[93869]: osdmap e542: 12 total, 12 up, 12 in 2023-03-09T17:26:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:26:15 smithi181 ceph-mon[94189]: pgmap v5252: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 26 KiB/s wr, 8 op/s 2023-03-09T17:26:16.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:26:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8886.057043 secs 2023-03-09T17:26:16.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:26:15 smithi181 ceph-mon[94189]: osdmap e542: 12 total, 12 up, 12 in 2023-03-09T17:26:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:26:15 smithi038 ceph-mon[92422]: pgmap v5252: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 26 KiB/s wr, 8 op/s 2023-03-09T17:26:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:26:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8886.057043 secs 2023-03-09T17:26:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:26:15 smithi038 ceph-mon[92422]: osdmap e542: 12 total, 12 up, 12 in 2023-03-09T17:26:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:26:17 smithi122 ceph-mon[93869]: pgmap v5254: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 8 op/s 2023-03-09T17:26:18.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:26:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8888.324740 secs 2023-03-09T17:26:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:26:17 smithi181 ceph-mon[94189]: pgmap v5254: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 8 op/s 2023-03-09T17:26:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:26:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8888.324740 secs 2023-03-09T17:26:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:26:17 smithi038 ceph-mon[92422]: pgmap v5254: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 8 op/s 2023-03-09T17:26:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:26:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8888.324740 secs 2023-03-09T17:26:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9350.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:26:20.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:26:19 smithi122 ceph-mon[93869]: pgmap v5255: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 10 KiB/s wr, 8 op/s 2023-03-09T17:26:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:26:19 smithi181 ceph-mon[94189]: pgmap v5255: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 10 KiB/s wr, 8 op/s 2023-03-09T17:26:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:26:19 smithi038 ceph-mon[92422]: pgmap v5255: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 10 KiB/s wr, 8 op/s 2023-03-09T17:26:21.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:26:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8891.057118 secs 2023-03-09T17:26:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:26:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8891.057118 secs 2023-03-09T17:26:21.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:26:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8891.057118 secs 2023-03-09T17:26:22.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:26:21 smithi122 ceph-mon[93869]: pgmap v5256: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.7 KiB/s wr, 7 op/s 2023-03-09T17:26:22.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:26:21 smithi181 ceph-mon[94189]: pgmap v5256: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.7 KiB/s wr, 7 op/s 2023-03-09T17:26:22.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:26:21 smithi038 ceph-mon[92422]: pgmap v5256: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.7 KiB/s wr, 7 op/s 2023-03-09T17:26:23.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:26:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8893.324856 secs 2023-03-09T17:26:23.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:26:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8893.324856 secs 2023-03-09T17:26:23.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:26:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8893.324856 secs 2023-03-09T17:26:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9355.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:26:24.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:26:23 smithi122 ceph-mon[93869]: pgmap v5257: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 10 KiB/s wr, 7 op/s 2023-03-09T17:26:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:26:23 smithi181 ceph-mon[94189]: pgmap v5257: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 10 KiB/s wr, 7 op/s 2023-03-09T17:26:24.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:26:23 smithi038 ceph-mon[92422]: pgmap v5257: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 10 KiB/s wr, 7 op/s 2023-03-09T17:26:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:26:25 smithi122 ceph-mon[93869]: pgmap v5258: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.8 KiB/s wr, 7 op/s 2023-03-09T17:26:26.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:26:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8896.057180 secs 2023-03-09T17:26:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:26:25 smithi181 ceph-mon[94189]: pgmap v5258: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.8 KiB/s wr, 7 op/s 2023-03-09T17:26:26.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:26:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8896.057180 secs 2023-03-09T17:26:26.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:26:25 smithi038 ceph-mon[92422]: pgmap v5258: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.8 KiB/s wr, 7 op/s 2023-03-09T17:26:26.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:26:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8896.057180 secs 2023-03-09T17:26:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:26:27 smithi122 ceph-mon[93869]: pgmap v5259: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 26 KiB/s wr, 9 op/s 2023-03-09T17:26:28.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:26:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8898.324933 secs 2023-03-09T17:26:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:26:27 smithi181 ceph-mon[94189]: pgmap v5259: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 26 KiB/s wr, 9 op/s 2023-03-09T17:26:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:26:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8898.324933 secs 2023-03-09T17:26:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:26:27 smithi038 ceph-mon[92422]: pgmap v5259: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 26 KiB/s wr, 9 op/s 2023-03-09T17:26:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:26:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8898.324933 secs 2023-03-09T17:26:28.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9360.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:26:30.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:26:29 smithi122 ceph-mon[93869]: pgmap v5260: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 23 KiB/s wr, 6 op/s 2023-03-09T17:26:30.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:26:29 smithi181 ceph-mon[94189]: pgmap v5260: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 23 KiB/s wr, 6 op/s 2023-03-09T17:26:30.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:26:29 smithi038 ceph-mon[92422]: pgmap v5260: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 23 KiB/s wr, 6 op/s 2023-03-09T17:26:31.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:26:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8901.057243 secs 2023-03-09T17:26:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:26:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8901.057243 secs 2023-03-09T17:26:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:26:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8901.057243 secs 2023-03-09T17:26:32.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:26:31 smithi122 ceph-mon[93869]: pgmap v5261: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 5 op/s 2023-03-09T17:26:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:26:31 smithi181 ceph-mon[94189]: pgmap v5261: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 5 op/s 2023-03-09T17:26:32.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:26:31 smithi038 ceph-mon[92422]: pgmap v5261: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 5 op/s 2023-03-09T17:26:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9365.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:26:34.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:26:33 smithi181 ceph-mon[94189]: pgmap v5262: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 8 op/s 2023-03-09T17:26:34.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:26:33 smithi038 ceph-mon[92422]: pgmap v5262: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 8 op/s 2023-03-09T17:26:34.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:26:33 smithi122 ceph-mon[93869]: pgmap v5262: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 8 op/s 2023-03-09T17:26:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:26:35 smithi038 ceph-mon[92422]: pgmap v5263: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 6 op/s 2023-03-09T17:26:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:26:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8906.057304 secs 2023-03-09T17:26:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:26:35 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:26:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:26:35 smithi122 ceph-mon[93869]: pgmap v5263: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 6 op/s 2023-03-09T17:26:36.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:26:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8906.057304 secs 2023-03-09T17:26:36.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:26:35 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:26:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:26:35 smithi181 ceph-mon[94189]: pgmap v5263: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 6 op/s 2023-03-09T17:26:36.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:26:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8906.057304 secs 2023-03-09T17:26:36.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:26:35 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:26:37.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:26:36 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:26:37.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:26:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:26:37.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:26:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:26:37.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:26:36 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:26:37.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:26:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:26:37.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:26:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:26:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:26:36 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:26:37.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:26:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:26:37.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:26:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:26:38.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:26:37 smithi122 ceph-mon[93869]: pgmap v5264: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 66 KiB/s wr, 9 op/s 2023-03-09T17:26:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:26:37 smithi181 ceph-mon[94189]: pgmap v5264: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 66 KiB/s wr, 9 op/s 2023-03-09T17:26:38.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:26:37 smithi038 ceph-mon[92422]: pgmap v5264: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 66 KiB/s wr, 9 op/s 2023-03-09T17:26:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9370.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:26:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:26:40 smithi122 ceph-mon[93869]: pgmap v5265: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 7 op/s 2023-03-09T17:26:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:26:40 smithi181 ceph-mon[94189]: pgmap v5265: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 7 op/s 2023-03-09T17:26:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:26:40 smithi038 ceph-mon[92422]: pgmap v5265: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 7 op/s 2023-03-09T17:26:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:26:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8911.057366 secs 2023-03-09T17:26:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:26:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8911.057366 secs 2023-03-09T17:26:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:26:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8911.057366 secs 2023-03-09T17:26:42.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:26:42 smithi122 ceph-mon[93869]: pgmap v5266: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 53 KiB/s wr, 7 op/s 2023-03-09T17:26:42.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:26:42 smithi181 ceph-mon[94189]: pgmap v5266: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 53 KiB/s wr, 7 op/s 2023-03-09T17:26:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:26:42 smithi038 ceph-mon[92422]: pgmap v5266: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 53 KiB/s wr, 7 op/s 2023-03-09T17:26:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9375.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:26:44.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:26:44 smithi122 ceph-mon[93869]: pgmap v5267: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 88 KiB/s wr, 10 op/s 2023-03-09T17:26:44.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:26:44 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:26:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:26:44 smithi181 ceph-mon[94189]: pgmap v5267: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 88 KiB/s wr, 10 op/s 2023-03-09T17:26:44.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:26:44 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:26:44.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:26:44 smithi038 ceph-mon[92422]: pgmap v5267: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 88 KiB/s wr, 10 op/s 2023-03-09T17:26:44.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:26:44 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:26:45.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:26:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:26:45.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:26:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:26:45.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:26:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:26:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:26:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:26:45.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:26:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:26:45.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:26:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:26:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:26:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:26:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:26:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:26:45.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:26:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:26:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:26:46 smithi122 ceph-mon[93869]: pgmap v5268: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 95 KiB/s wr, 9 op/s 2023-03-09T17:26:46.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:26:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8916.057427 secs 2023-03-09T17:26:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:26:46 smithi181 ceph-mon[94189]: pgmap v5268: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 95 KiB/s wr, 9 op/s 2023-03-09T17:26:46.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:26:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8916.057427 secs 2023-03-09T17:26:46.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:26:46 smithi038 ceph-mon[92422]: pgmap v5268: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 95 KiB/s wr, 9 op/s 2023-03-09T17:26:46.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:26:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8916.057427 secs 2023-03-09T17:26:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:26:48 smithi122 ceph-mon[93869]: pgmap v5269: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 134 KiB/s wr, 11 op/s 2023-03-09T17:26:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:26:48 smithi181 ceph-mon[94189]: pgmap v5269: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 134 KiB/s wr, 11 op/s 2023-03-09T17:26:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:26:48 smithi038 ceph-mon[92422]: pgmap v5269: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 134 KiB/s wr, 11 op/s 2023-03-09T17:26:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9380.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:26:50.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:26:50 smithi122 ceph-mon[93869]: pgmap v5270: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 98 KiB/s wr, 10 op/s 2023-03-09T17:26:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:26:50 smithi181 ceph-mon[94189]: pgmap v5270: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 98 KiB/s wr, 10 op/s 2023-03-09T17:26:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:26:50 smithi038 ceph-mon[92422]: pgmap v5270: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 98 KiB/s wr, 10 op/s 2023-03-09T17:26:51.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:26:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8921.057489 secs 2023-03-09T17:26:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:26:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8921.057489 secs 2023-03-09T17:26:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:26:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8921.057489 secs 2023-03-09T17:26:52.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:26:52 smithi122 ceph-mon[93869]: pgmap v5271: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 98 KiB/s wr, 9 op/s 2023-03-09T17:26:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:26:52 smithi181 ceph-mon[94189]: pgmap v5271: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 98 KiB/s wr, 9 op/s 2023-03-09T17:26:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:26:52 smithi038 ceph-mon[92422]: pgmap v5271: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 98 KiB/s wr, 9 op/s 2023-03-09T17:26:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9385.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:26:54.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:26:54 smithi122 ceph-mon[93869]: pgmap v5272: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 140 KiB/s wr, 12 op/s 2023-03-09T17:26:54.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:26:54 smithi181 ceph-mon[94189]: pgmap v5272: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 140 KiB/s wr, 12 op/s 2023-03-09T17:26:54.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:26:54 smithi038 ceph-mon[92422]: pgmap v5272: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 140 KiB/s wr, 12 op/s 2023-03-09T17:26:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:26:56 smithi122 ceph-mon[93869]: pgmap v5273: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 155 KiB/s wr, 11 op/s 2023-03-09T17:26:56.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:26:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8926.057549 secs 2023-03-09T17:26:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:26:56 smithi181 ceph-mon[94189]: pgmap v5273: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 155 KiB/s wr, 11 op/s 2023-03-09T17:26:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:26:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8926.057549 secs 2023-03-09T17:26:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:26:56 smithi038 ceph-mon[92422]: pgmap v5273: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 155 KiB/s wr, 11 op/s 2023-03-09T17:26:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:26:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8926.057549 secs 2023-03-09T17:26:58.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:26:58 smithi122 ceph-mon[93869]: pgmap v5274: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 147 KiB/s wr, 12 op/s 2023-03-09T17:26:58.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:26:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8928.325653 secs 2023-03-09T17:26:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:26:58 smithi181 ceph-mon[94189]: pgmap v5274: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 147 KiB/s wr, 12 op/s 2023-03-09T17:26:58.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:26:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8928.325653 secs 2023-03-09T17:26:58.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:26:58 smithi038 ceph-mon[92422]: pgmap v5274: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 147 KiB/s wr, 12 op/s 2023-03-09T17:26:58.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:26:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8928.325653 secs 2023-03-09T17:26:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9390.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:27:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:27:00 smithi122 ceph-mon[93869]: pgmap v5275: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 117 KiB/s wr, 10 op/s 2023-03-09T17:27:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:27:00 smithi181 ceph-mon[94189]: pgmap v5275: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 117 KiB/s wr, 10 op/s 2023-03-09T17:27:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:27:00 smithi038 ceph-mon[92422]: pgmap v5275: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 117 KiB/s wr, 10 op/s 2023-03-09T17:27:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:27:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8931.057627 secs 2023-03-09T17:27:01.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:27:01 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:27:01.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:27:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:27:01.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:27:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:27:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:27:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8931.057627 secs 2023-03-09T17:27:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:27:01 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:27:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:27:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:27:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:27:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:27:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:27:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8931.057627 secs 2023-03-09T17:27:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:27:01 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:27:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:27:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:27:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:27:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:27:02.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:27:02 smithi122 ceph-mon[93869]: pgmap v5276: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 144 KiB/s wr, 9 op/s 2023-03-09T17:27:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:27:02 smithi181 ceph-mon[94189]: pgmap v5276: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 144 KiB/s wr, 9 op/s 2023-03-09T17:27:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:27:02 smithi038 ceph-mon[92422]: pgmap v5276: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 144 KiB/s wr, 9 op/s 2023-03-09T17:27:03.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:27:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8933.325749 secs 2023-03-09T17:27:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:27:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8933.325749 secs 2023-03-09T17:27:03.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:27:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8933.325749 secs 2023-03-09T17:27:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9395.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:27:04.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:27:04 smithi122 ceph-mon[93869]: pgmap v5277: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 159 KiB/s wr, 14 op/s 2023-03-09T17:27:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:27:04 smithi181 ceph-mon[94189]: pgmap v5277: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 159 KiB/s wr, 14 op/s 2023-03-09T17:27:04.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:27:04 smithi038 ceph-mon[92422]: pgmap v5277: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 159 KiB/s wr, 14 op/s 2023-03-09T17:27:05.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:27:05 smithi122 ceph-mon[93869]: osdmap e543: 12 total, 12 up, 12 in 2023-03-09T17:27:05.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:27:05 smithi181 ceph-mon[94189]: osdmap e543: 12 total, 12 up, 12 in 2023-03-09T17:27:05.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:27:05 smithi038 ceph-mon[92422]: osdmap e543: 12 total, 12 up, 12 in 2023-03-09T17:27:06.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:27:06 smithi122 ceph-mon[93869]: pgmap v5279: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 150 KiB/s wr, 27 op/s 2023-03-09T17:27:06.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:27:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8936.057688 secs 2023-03-09T17:27:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:27:06 smithi181 ceph-mon[94189]: pgmap v5279: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 150 KiB/s wr, 27 op/s 2023-03-09T17:27:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:27:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8936.057688 secs 2023-03-09T17:27:06.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:27:06 smithi038 ceph-mon[92422]: pgmap v5279: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 150 KiB/s wr, 27 op/s 2023-03-09T17:27:06.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:27:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8936.057688 secs 2023-03-09T17:27:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:27:08 smithi122 ceph-mon[93869]: pgmap v5280: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 151 KiB/s wr, 27 op/s 2023-03-09T17:27:08.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:27:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8938.325830 secs 2023-03-09T17:27:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:27:08 smithi181 ceph-mon[94189]: pgmap v5280: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 151 KiB/s wr, 27 op/s 2023-03-09T17:27:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:27:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8938.325830 secs 2023-03-09T17:27:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:27:08 smithi038 ceph-mon[92422]: pgmap v5280: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 151 KiB/s wr, 27 op/s 2023-03-09T17:27:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:27:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8938.325830 secs 2023-03-09T17:27:08.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9400.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:27:10.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:27:10 smithi122 ceph-mon[93869]: pgmap v5281: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 151 KiB/s wr, 27 op/s 2023-03-09T17:27:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:27:10 smithi181 ceph-mon[94189]: pgmap v5281: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 151 KiB/s wr, 27 op/s 2023-03-09T17:27:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:27:10 smithi038 ceph-mon[92422]: pgmap v5281: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 151 KiB/s wr, 27 op/s 2023-03-09T17:27:11.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:27:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8941.057749 secs 2023-03-09T17:27:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:27:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8941.057749 secs 2023-03-09T17:27:11.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:27:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8941.057749 secs 2023-03-09T17:27:12.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:27:12 smithi122 ceph-mon[93869]: pgmap v5282: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 125 KiB/s wr, 27 op/s 2023-03-09T17:27:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:27:12 smithi181 ceph-mon[94189]: pgmap v5282: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 125 KiB/s wr, 27 op/s 2023-03-09T17:27:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:27:12 smithi038 ceph-mon[92422]: pgmap v5282: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 125 KiB/s wr, 27 op/s 2023-03-09T17:27:13.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:27:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8943.325927 secs 2023-03-09T17:27:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:27:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8943.325927 secs 2023-03-09T17:27:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:27:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8943.325927 secs 2023-03-09T17:27:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9405.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:27:14.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:27:14 smithi122 ceph-mon[93869]: pgmap v5283: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 98 KiB/s wr, 24 op/s 2023-03-09T17:27:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:27:14 smithi181 ceph-mon[94189]: pgmap v5283: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 98 KiB/s wr, 24 op/s 2023-03-09T17:27:14.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:27:14 smithi038 ceph-mon[92422]: pgmap v5283: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 98 KiB/s wr, 24 op/s 2023-03-09T17:27:15.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:27:15 smithi122 ceph-mon[93869]: pgmap v5284: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 8 op/s 2023-03-09T17:27:15.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:27:15 smithi181 ceph-mon[94189]: pgmap v5284: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 8 op/s 2023-03-09T17:27:15.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:27:15 smithi038 ceph-mon[92422]: pgmap v5284: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 8 op/s 2023-03-09T17:27:16.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:27:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8946.057810 secs 2023-03-09T17:27:16.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:27:16 smithi122 ceph-mon[93869]: osdmap e544: 12 total, 12 up, 12 in 2023-03-09T17:27:16.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:27:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8946.057810 secs 2023-03-09T17:27:16.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:27:16 smithi181 ceph-mon[94189]: osdmap e544: 12 total, 12 up, 12 in 2023-03-09T17:27:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:27:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8946.057810 secs 2023-03-09T17:27:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:27:16 smithi038 ceph-mon[92422]: osdmap e544: 12 total, 12 up, 12 in 2023-03-09T17:27:17.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:27:17 smithi122 ceph-mon[93869]: pgmap v5286: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 28 KiB/s wr, 8 op/s 2023-03-09T17:27:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:27:17 smithi181 ceph-mon[94189]: pgmap v5286: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 28 KiB/s wr, 8 op/s 2023-03-09T17:27:17.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:27:17 smithi038 ceph-mon[92422]: pgmap v5286: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 28 KiB/s wr, 8 op/s 2023-03-09T17:27:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:27:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8948.326031 secs 2023-03-09T17:27:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:27:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8948.326031 secs 2023-03-09T17:27:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:27:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8948.326031 secs 2023-03-09T17:27:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9410.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:27:19.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:27:19 smithi122 ceph-mon[93869]: pgmap v5287: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T17:27:19.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:27:19 smithi181 ceph-mon[94189]: pgmap v5287: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T17:27:19.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:27:19 smithi038 ceph-mon[92422]: pgmap v5287: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T17:27:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:27:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8951.057879 secs 2023-03-09T17:27:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:27:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8951.057879 secs 2023-03-09T17:27:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:27:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8951.057879 secs 2023-03-09T17:27:21.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:27:21 smithi122 ceph-mon[93869]: pgmap v5288: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 42 KiB/s wr, 7 op/s 2023-03-09T17:27:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:27:21 smithi181 ceph-mon[94189]: pgmap v5288: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 42 KiB/s wr, 7 op/s 2023-03-09T17:27:22.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:27:21 smithi038 ceph-mon[92422]: pgmap v5288: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 42 KiB/s wr, 7 op/s 2023-03-09T17:27:22.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:27:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8953.326122 secs 2023-03-09T17:27:22.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:27:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8953.326122 secs 2023-03-09T17:27:23.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:27:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8953.326122 secs 2023-03-09T17:27:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9415.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:27:23.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:27:23 smithi122 ceph-mon[93869]: pgmap v5289: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 42 KiB/s wr, 7 op/s 2023-03-09T17:27:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:27:23 smithi181 ceph-mon[94189]: pgmap v5289: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 42 KiB/s wr, 7 op/s 2023-03-09T17:27:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:27:23 smithi038 ceph-mon[92422]: pgmap v5289: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 42 KiB/s wr, 7 op/s 2023-03-09T17:27:25.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:27:25 smithi122 ceph-mon[93869]: pgmap v5290: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 42 KiB/s wr, 8 op/s 2023-03-09T17:27:25.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:27:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8956.057938 secs 2023-03-09T17:27:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:27:25 smithi181 ceph-mon[94189]: pgmap v5290: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 42 KiB/s wr, 8 op/s 2023-03-09T17:27:25.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:27:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8956.057938 secs 2023-03-09T17:27:26.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:27:25 smithi038 ceph-mon[92422]: pgmap v5290: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 42 KiB/s wr, 8 op/s 2023-03-09T17:27:26.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:27:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8956.057938 secs 2023-03-09T17:27:27.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:27:27 smithi122 ceph-mon[93869]: pgmap v5291: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 40 KiB/s wr, 9 op/s 2023-03-09T17:27:27.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:27:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8958.326232 secs 2023-03-09T17:27:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:27:27 smithi181 ceph-mon[94189]: pgmap v5291: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 40 KiB/s wr, 9 op/s 2023-03-09T17:27:27.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:27:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8958.326232 secs 2023-03-09T17:27:28.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:27:27 smithi038 ceph-mon[92422]: pgmap v5291: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 40 KiB/s wr, 9 op/s 2023-03-09T17:27:28.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:27:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8958.326232 secs 2023-03-09T17:27:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9420.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:27:29.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:27:29 smithi122 ceph-mon[93869]: pgmap v5292: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 35 KiB/s wr, 7 op/s 2023-03-09T17:27:29.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:27:29 smithi181 ceph-mon[94189]: pgmap v5292: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 35 KiB/s wr, 7 op/s 2023-03-09T17:27:30.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:27:29 smithi038 ceph-mon[92422]: pgmap v5292: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 35 KiB/s wr, 7 op/s 2023-03-09T17:27:30.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:27:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8961.057984 secs 2023-03-09T17:27:30.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:27:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8961.057984 secs 2023-03-09T17:27:30.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:27:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8961.057984 secs 2023-03-09T17:27:31.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:27:31 smithi122 ceph-mon[93869]: pgmap v5293: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 35 KiB/s wr, 7 op/s 2023-03-09T17:27:31.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:27:31 smithi181 ceph-mon[94189]: pgmap v5293: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 35 KiB/s wr, 7 op/s 2023-03-09T17:27:32.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:27:31 smithi038 ceph-mon[92422]: pgmap v5293: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 35 KiB/s wr, 7 op/s 2023-03-09T17:27:32.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:27:32 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 8963.326351 secs 2023-03-09T17:27:32.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:27:32 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 8963.326351 secs 2023-03-09T17:27:33.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:27:32 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 8963.326351 secs 2023-03-09T17:27:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9425.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:27:33.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:27:33 smithi122 ceph-mon[93869]: pgmap v5294: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 1.5 KiB/s wr, 8 op/s 2023-03-09T17:27:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:27:33 smithi181 ceph-mon[94189]: pgmap v5294: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 1.5 KiB/s wr, 8 op/s 2023-03-09T17:27:34.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:27:33 smithi038 ceph-mon[92422]: pgmap v5294: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 1.5 KiB/s wr, 8 op/s 2023-03-09T17:27:35.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:27:35 smithi122 ceph-mon[93869]: pgmap v5295: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 49 KiB/s wr, 8 op/s 2023-03-09T17:27:35.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:27:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8966.058045 secs 2023-03-09T17:27:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:27:35 smithi181 ceph-mon[94189]: pgmap v5295: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 49 KiB/s wr, 8 op/s 2023-03-09T17:27:35.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:27:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8966.058045 secs 2023-03-09T17:27:36.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:27:35 smithi038 ceph-mon[92422]: pgmap v5295: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 49 KiB/s wr, 8 op/s 2023-03-09T17:27:36.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:27:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8966.058045 secs 2023-03-09T17:27:36.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:27:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:27:36.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:27:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:27:36.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:27:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:27:36.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:27:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:27:37.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:27:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:27:37.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:27:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:27:37.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:27:37 smithi122 ceph-mon[93869]: pgmap v5296: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 49 KiB/s wr, 8 op/s 2023-03-09T17:27:37.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:27:37 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 8968.326460 secs 2023-03-09T17:27:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:27:37 smithi181 ceph-mon[94189]: pgmap v5296: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 49 KiB/s wr, 8 op/s 2023-03-09T17:27:37.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:27:37 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 8968.326460 secs 2023-03-09T17:27:38.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:27:37 smithi038 ceph-mon[92422]: pgmap v5296: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 49 KiB/s wr, 8 op/s 2023-03-09T17:27:38.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:27:37 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 8968.326460 secs 2023-03-09T17:27:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9430.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:27:39.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:27:39 smithi122 ceph-mon[93869]: pgmap v5297: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 57 KiB/s wr, 7 op/s 2023-03-09T17:27:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:27:39 smithi181 ceph-mon[94189]: pgmap v5297: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 57 KiB/s wr, 7 op/s 2023-03-09T17:27:40.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:27:39 smithi038 ceph-mon[92422]: pgmap v5297: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 57 KiB/s wr, 7 op/s 2023-03-09T17:27:40.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:27:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8971.058106 secs 2023-03-09T17:27:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:27:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8971.058106 secs 2023-03-09T17:27:41.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:27:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8971.058106 secs 2023-03-09T17:27:41.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:27:41 smithi122 ceph-mon[93869]: pgmap v5298: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 92 KiB/s wr, 8 op/s 2023-03-09T17:27:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:27:41 smithi181 ceph-mon[94189]: pgmap v5298: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 92 KiB/s wr, 8 op/s 2023-03-09T17:27:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:27:41 smithi038 ceph-mon[92422]: pgmap v5298: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 92 KiB/s wr, 8 op/s 2023-03-09T17:27:42.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:27:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8973.326604 secs 2023-03-09T17:27:42.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:27:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8973.326604 secs 2023-03-09T17:27:43.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:27:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8973.326604 secs 2023-03-09T17:27:43.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9435.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:27:43.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:27:43 smithi122 ceph-mon[93869]: pgmap v5299: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 92 KiB/s wr, 10 op/s 2023-03-09T17:27:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:27:43 smithi181 ceph-mon[94189]: pgmap v5299: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 92 KiB/s wr, 10 op/s 2023-03-09T17:27:44.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:27:43 smithi038 ceph-mon[92422]: pgmap v5299: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 92 KiB/s wr, 10 op/s 2023-03-09T17:27:44.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:27:44 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:27:44.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:27:44 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:27:44.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:27:44 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:27:44.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:27:44 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:27:45.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:27:44 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:27:45.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:27:44 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:27:45.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:27:45 smithi122 ceph-mon[93869]: pgmap v5300: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 139 KiB/s wr, 10 op/s 2023-03-09T17:27:45.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:27:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8976.058168 secs 2023-03-09T17:27:45.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:27:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:27:45.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:27:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:27:45.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:27:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:27:45.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:27:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:27:45.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:27:45 smithi181 ceph-mon[94189]: pgmap v5300: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 139 KiB/s wr, 10 op/s 2023-03-09T17:27:45.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:27:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8976.058168 secs 2023-03-09T17:27:45.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:27:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:27:45.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:27:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:27:45.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:27:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:27:45.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:27:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:27:46.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:27:45 smithi038 ceph-mon[92422]: pgmap v5300: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 139 KiB/s wr, 10 op/s 2023-03-09T17:27:46.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:27:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8976.058168 secs 2023-03-09T17:27:46.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:27:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:27:46.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:27:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:27:46.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:27:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:27:46.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:27:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:27:47.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:27:47 smithi122 ceph-mon[93869]: pgmap v5301: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 91 KiB/s wr, 10 op/s 2023-03-09T17:27:47.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:27:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8978.326680 secs 2023-03-09T17:27:47.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:27:47 smithi181 ceph-mon[94189]: pgmap v5301: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 91 KiB/s wr, 10 op/s 2023-03-09T17:27:47.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:27:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8978.326680 secs 2023-03-09T17:27:48.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:27:47 smithi038 ceph-mon[92422]: pgmap v5301: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 91 KiB/s wr, 10 op/s 2023-03-09T17:27:48.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:27:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8978.326680 secs 2023-03-09T17:27:48.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9440.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:27:49.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:27:49 smithi122 ceph-mon[93869]: pgmap v5302: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 99 KiB/s wr, 10 op/s 2023-03-09T17:27:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:27:49 smithi181 ceph-mon[94189]: pgmap v5302: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 99 KiB/s wr, 10 op/s 2023-03-09T17:27:50.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:27:49 smithi038 ceph-mon[92422]: pgmap v5302: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 99 KiB/s wr, 10 op/s 2023-03-09T17:27:50.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:27:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8981.058231 secs 2023-03-09T17:27:50.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:27:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8981.058231 secs 2023-03-09T17:27:51.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:27:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8981.058231 secs 2023-03-09T17:27:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:27:51 smithi122 ceph-mon[93869]: pgmap v5303: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 135 KiB/s wr, 10 op/s 2023-03-09T17:27:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:27:51 smithi181 ceph-mon[94189]: pgmap v5303: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 135 KiB/s wr, 10 op/s 2023-03-09T17:27:52.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:27:51 smithi038 ceph-mon[92422]: pgmap v5303: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 135 KiB/s wr, 10 op/s 2023-03-09T17:27:52.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:27:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8983.326737 secs 2023-03-09T17:27:52.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:27:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8983.326737 secs 2023-03-09T17:27:53.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:27:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8983.326737 secs 2023-03-09T17:27:53.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9445.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:27:53.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:27:53 smithi181 ceph-mon[94189]: pgmap v5304: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 100 KiB/s wr, 11 op/s 2023-03-09T17:27:54.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:27:53 smithi038 ceph-mon[92422]: pgmap v5304: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 100 KiB/s wr, 11 op/s 2023-03-09T17:27:54.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:27:53 smithi122 ceph-mon[93869]: pgmap v5304: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 100 KiB/s wr, 11 op/s 2023-03-09T17:27:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:27:55 smithi038 ceph-mon[92422]: pgmap v5305: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 128 KiB/s wr, 11 op/s 2023-03-09T17:27:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:27:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8986.058293 secs 2023-03-09T17:27:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:27:55 smithi122 ceph-mon[93869]: pgmap v5305: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 128 KiB/s wr, 11 op/s 2023-03-09T17:27:56.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:27:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8986.058293 secs 2023-03-09T17:27:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:27:55 smithi181 ceph-mon[94189]: pgmap v5305: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 128 KiB/s wr, 11 op/s 2023-03-09T17:27:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:27:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8986.058293 secs 2023-03-09T17:27:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:27:57 smithi038 ceph-mon[92422]: pgmap v5306: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 105 KiB/s wr, 11 op/s 2023-03-09T17:27:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:27:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8988.326848 secs 2023-03-09T17:27:58.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:27:57 smithi122 ceph-mon[93869]: pgmap v5306: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 105 KiB/s wr, 11 op/s 2023-03-09T17:27:58.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:27:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8988.326848 secs 2023-03-09T17:27:58.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:27:57 smithi181 ceph-mon[94189]: pgmap v5306: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 105 KiB/s wr, 11 op/s 2023-03-09T17:27:58.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:27:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8988.326848 secs 2023-03-09T17:27:58.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9450.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:28:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:27:59 smithi038 ceph-mon[92422]: pgmap v5307: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 106 KiB/s wr, 10 op/s 2023-03-09T17:28:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:27:59 smithi122 ceph-mon[93869]: pgmap v5307: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 106 KiB/s wr, 10 op/s 2023-03-09T17:28:00.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:27:59 smithi181 ceph-mon[94189]: pgmap v5307: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 106 KiB/s wr, 10 op/s 2023-03-09T17:28:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:28:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8991.058354 secs 2023-03-09T17:28:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:28:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8991.058354 secs 2023-03-09T17:28:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:28:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8991.058354 secs 2023-03-09T17:28:02.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:28:01 smithi122 ceph-mon[93869]: pgmap v5308: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 97 KiB/s wr, 9 op/s 2023-03-09T17:28:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:28:01 smithi181 ceph-mon[94189]: pgmap v5308: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 97 KiB/s wr, 9 op/s 2023-03-09T17:28:02.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:28:01 smithi038 ceph-mon[92422]: pgmap v5308: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 97 KiB/s wr, 9 op/s 2023-03-09T17:28:03.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:28:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8993.326986 secs 2023-03-09T17:28:03.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:28:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8993.326986 secs 2023-03-09T17:28:03.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:28:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8993.326986 secs 2023-03-09T17:28:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9455.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:28:04.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:28:03 smithi122 ceph-mon[93869]: pgmap v5309: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 91 KiB/s wr, 12 op/s 2023-03-09T17:28:04.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:28:03 smithi181 ceph-mon[94189]: pgmap v5309: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 91 KiB/s wr, 12 op/s 2023-03-09T17:28:04.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:28:03 smithi038 ceph-mon[92422]: pgmap v5309: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 91 KiB/s wr, 12 op/s 2023-03-09T17:28:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:28:05 smithi122 ceph-mon[93869]: pgmap v5310: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 120 KiB/s wr, 23 op/s 2023-03-09T17:28:06.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:28:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8996.058417 secs 2023-03-09T17:28:06.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:28:05 smithi181 ceph-mon[94189]: pgmap v5310: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 120 KiB/s wr, 23 op/s 2023-03-09T17:28:06.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:28:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8996.058417 secs 2023-03-09T17:28:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:28:05 smithi038 ceph-mon[92422]: pgmap v5310: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 120 KiB/s wr, 23 op/s 2023-03-09T17:28:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:28:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8996.058417 secs 2023-03-09T17:28:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:28:07 smithi122 ceph-mon[93869]: pgmap v5311: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 104 KiB/s wr, 25 op/s 2023-03-09T17:28:08.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:28:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 8998.327048 secs 2023-03-09T17:28:08.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:28:07 smithi181 ceph-mon[94189]: pgmap v5311: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 104 KiB/s wr, 25 op/s 2023-03-09T17:28:08.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:28:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 8998.327048 secs 2023-03-09T17:28:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:28:07 smithi038 ceph-mon[92422]: pgmap v5311: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 104 KiB/s wr, 25 op/s 2023-03-09T17:28:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:28:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 8998.327048 secs 2023-03-09T17:28:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9460.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:28:10.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:28:09 smithi122 ceph-mon[93869]: pgmap v5312: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 87 KiB/s wr, 23 op/s 2023-03-09T17:28:10.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:28:09 smithi181 ceph-mon[94189]: pgmap v5312: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 87 KiB/s wr, 23 op/s 2023-03-09T17:28:10.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:28:09 smithi038 ceph-mon[92422]: pgmap v5312: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 87 KiB/s wr, 23 op/s 2023-03-09T17:28:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:28:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9001.058501 secs 2023-03-09T17:28:11.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:28:10 smithi122 ceph-mon[93869]: osdmap e545: 12 total, 12 up, 12 in 2023-03-09T17:28:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:28:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9001.058501 secs 2023-03-09T17:28:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:28:10 smithi181 ceph-mon[94189]: osdmap e545: 12 total, 12 up, 12 in 2023-03-09T17:28:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:28:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9001.058501 secs 2023-03-09T17:28:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:28:10 smithi038 ceph-mon[92422]: osdmap e545: 12 total, 12 up, 12 in 2023-03-09T17:28:12.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:28:11 smithi122 ceph-mon[93869]: pgmap v5314: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 104 KiB/s wr, 26 op/s 2023-03-09T17:28:12.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:28:11 smithi181 ceph-mon[94189]: pgmap v5314: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 104 KiB/s wr, 26 op/s 2023-03-09T17:28:12.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:28:11 smithi038 ceph-mon[92422]: pgmap v5314: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 104 KiB/s wr, 26 op/s 2023-03-09T17:28:13.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:28:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9003.327169 secs 2023-03-09T17:28:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:28:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9003.327169 secs 2023-03-09T17:28:13.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:28:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9003.327169 secs 2023-03-09T17:28:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9465.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:28:14.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:28:13 smithi122 ceph-mon[93869]: pgmap v5315: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 75 KiB/s wr, 25 op/s 2023-03-09T17:28:14.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:28:13 smithi181 ceph-mon[94189]: pgmap v5315: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 75 KiB/s wr, 25 op/s 2023-03-09T17:28:14.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:28:13 smithi038 ceph-mon[92422]: pgmap v5315: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 75 KiB/s wr, 25 op/s 2023-03-09T17:28:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:28:15 smithi122 ceph-mon[93869]: pgmap v5316: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 42 KiB/s wr, 10 op/s 2023-03-09T17:28:16.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:28:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9006.058560 secs 2023-03-09T17:28:16.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:28:15 smithi122 ceph-mon[93869]: osdmap e546: 12 total, 12 up, 12 in 2023-03-09T17:28:16.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:28:15 smithi181 ceph-mon[94189]: pgmap v5316: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 42 KiB/s wr, 10 op/s 2023-03-09T17:28:16.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:28:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9006.058560 secs 2023-03-09T17:28:16.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:28:15 smithi181 ceph-mon[94189]: osdmap e546: 12 total, 12 up, 12 in 2023-03-09T17:28:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:28:15 smithi038 ceph-mon[92422]: pgmap v5316: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 42 KiB/s wr, 10 op/s 2023-03-09T17:28:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:28:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9006.058560 secs 2023-03-09T17:28:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:28:15 smithi038 ceph-mon[92422]: osdmap e546: 12 total, 12 up, 12 in 2023-03-09T17:28:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:28:17 smithi122 ceph-mon[93869]: pgmap v5318: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 9 op/s 2023-03-09T17:28:18.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:28:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9008.327231 secs 2023-03-09T17:28:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:28:17 smithi181 ceph-mon[94189]: pgmap v5318: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 9 op/s 2023-03-09T17:28:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:28:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9008.327231 secs 2023-03-09T17:28:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:28:17 smithi038 ceph-mon[92422]: pgmap v5318: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 9 op/s 2023-03-09T17:28:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:28:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9008.327231 secs 2023-03-09T17:28:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9470.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:28:20.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:28:19 smithi122 ceph-mon[93869]: pgmap v5319: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 9 op/s 2023-03-09T17:28:20.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:28:19 smithi181 ceph-mon[94189]: pgmap v5319: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 9 op/s 2023-03-09T17:28:20.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:28:19 smithi038 ceph-mon[92422]: pgmap v5319: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 9 op/s 2023-03-09T17:28:21.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:28:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9011.058645 secs 2023-03-09T17:28:21.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:28:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9011.058645 secs 2023-03-09T17:28:21.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:28:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9011.058645 secs 2023-03-09T17:28:22.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:28:21 smithi122 ceph-mon[93869]: pgmap v5320: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 8 op/s 2023-03-09T17:28:22.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:28:21 smithi181 ceph-mon[94189]: pgmap v5320: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 8 op/s 2023-03-09T17:28:22.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:28:21 smithi038 ceph-mon[92422]: pgmap v5320: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 8 op/s 2023-03-09T17:28:23.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:28:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9013.327349 secs 2023-03-09T17:28:23.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:28:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9013.327349 secs 2023-03-09T17:28:23.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:28:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9013.327349 secs 2023-03-09T17:28:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9475.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:28:24.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:28:23 smithi122 ceph-mon[93869]: pgmap v5321: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T17:28:24.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:28:23 smithi181 ceph-mon[94189]: pgmap v5321: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T17:28:24.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:28:23 smithi038 ceph-mon[92422]: pgmap v5321: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T17:28:26.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:28:25 smithi122 ceph-mon[93869]: pgmap v5322: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T17:28:26.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:28:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9016.058707 secs 2023-03-09T17:28:26.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:28:25 smithi181 ceph-mon[94189]: pgmap v5322: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T17:28:26.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:28:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9016.058707 secs 2023-03-09T17:28:26.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:28:25 smithi038 ceph-mon[92422]: pgmap v5322: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T17:28:26.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:28:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9016.058707 secs 2023-03-09T17:28:28.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:28:27 smithi122 ceph-mon[93869]: pgmap v5323: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 25 KiB/s wr, 9 op/s 2023-03-09T17:28:28.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:28:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9018.327482 secs 2023-03-09T17:28:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:28:27 smithi181 ceph-mon[94189]: pgmap v5323: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 25 KiB/s wr, 9 op/s 2023-03-09T17:28:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:28:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9018.327482 secs 2023-03-09T17:28:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:28:27 smithi038 ceph-mon[92422]: pgmap v5323: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 25 KiB/s wr, 9 op/s 2023-03-09T17:28:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:28:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9018.327482 secs 2023-03-09T17:28:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9480.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:28:30.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:28:29 smithi122 ceph-mon[93869]: pgmap v5324: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 6 op/s 2023-03-09T17:28:30.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:28:29 smithi181 ceph-mon[94189]: pgmap v5324: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 6 op/s 2023-03-09T17:28:30.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:28:29 smithi038 ceph-mon[92422]: pgmap v5324: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 6 op/s 2023-03-09T17:28:31.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:28:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9021.058768 secs 2023-03-09T17:28:31.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:28:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9021.058768 secs 2023-03-09T17:28:31.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:28:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9021.058768 secs 2023-03-09T17:28:32.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:28:31 smithi122 ceph-mon[93869]: pgmap v5325: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 6 op/s 2023-03-09T17:28:32.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:28:31 smithi181 ceph-mon[94189]: pgmap v5325: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 6 op/s 2023-03-09T17:28:32.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:28:31 smithi038 ceph-mon[92422]: pgmap v5325: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 6 op/s 2023-03-09T17:28:33.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:28:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9023.327599 secs 2023-03-09T17:28:33.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:28:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9023.327599 secs 2023-03-09T17:28:33.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:28:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9023.327599 secs 2023-03-09T17:28:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9485.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:28:34.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:28:33 smithi122 ceph-mon[93869]: pgmap v5326: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 44 KiB/s wr, 8 op/s 2023-03-09T17:28:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:28:33 smithi181 ceph-mon[94189]: pgmap v5326: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 44 KiB/s wr, 8 op/s 2023-03-09T17:28:34.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:28:33 smithi038 ceph-mon[92422]: pgmap v5326: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 44 KiB/s wr, 8 op/s 2023-03-09T17:28:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:28:35 smithi122 ceph-mon[93869]: pgmap v5327: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 6 op/s 2023-03-09T17:28:36.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:28:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9026.058827 secs 2023-03-09T17:28:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:28:35 smithi181 ceph-mon[94189]: pgmap v5327: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 6 op/s 2023-03-09T17:28:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:28:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9026.058827 secs 2023-03-09T17:28:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:28:35 smithi038 ceph-mon[92422]: pgmap v5327: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 6 op/s 2023-03-09T17:28:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:28:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9026.058827 secs 2023-03-09T17:28:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:28:37 smithi122 ceph-mon[93869]: pgmap v5328: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 42 KiB/s wr, 8 op/s 2023-03-09T17:28:38.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:28:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9028.327713 secs 2023-03-09T17:28:38.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:28:37 smithi181 ceph-mon[94189]: pgmap v5328: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 42 KiB/s wr, 8 op/s 2023-03-09T17:28:38.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:28:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9028.327713 secs 2023-03-09T17:28:38.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:28:37 smithi038 ceph-mon[92422]: pgmap v5328: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 42 KiB/s wr, 8 op/s 2023-03-09T17:28:38.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:28:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9028.327713 secs 2023-03-09T17:28:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9490.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:28:40.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:28:39 smithi122 ceph-mon[93869]: pgmap v5329: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 6 op/s 2023-03-09T17:28:40.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:28:39 smithi181 ceph-mon[94189]: pgmap v5329: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 6 op/s 2023-03-09T17:28:40.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:28:39 smithi038 ceph-mon[92422]: pgmap v5329: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 6 op/s 2023-03-09T17:28:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:28:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9031.058888 secs 2023-03-09T17:28:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:28:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9031.058888 secs 2023-03-09T17:28:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:28:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9031.058888 secs 2023-03-09T17:28:42.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:28:41 smithi122 ceph-mon[93869]: pgmap v5330: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 6 op/s 2023-03-09T17:28:42.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:28:41 smithi181 ceph-mon[94189]: pgmap v5330: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 6 op/s 2023-03-09T17:28:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:28:41 smithi038 ceph-mon[92422]: pgmap v5330: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 6 op/s 2023-03-09T17:28:43.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:28:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9033.327849 secs 2023-03-09T17:28:43.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:28:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9033.327849 secs 2023-03-09T17:28:43.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:28:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9033.327849 secs 2023-03-09T17:28:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9495.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:28:44.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:28:43 smithi122 ceph-mon[93869]: pgmap v5331: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 78 KiB/s wr, 8 op/s 2023-03-09T17:28:44.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:28:43 smithi181 ceph-mon[94189]: pgmap v5331: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 78 KiB/s wr, 8 op/s 2023-03-09T17:28:44.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:28:43 smithi038 ceph-mon[92422]: pgmap v5331: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 78 KiB/s wr, 8 op/s 2023-03-09T17:28:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:28:45 smithi122 ceph-mon[93869]: pgmap v5332: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 64 KiB/s wr, 7 op/s 2023-03-09T17:28:46.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:28:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9036.058949 secs 2023-03-09T17:28:46.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:28:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:28:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:28:45 smithi181 ceph-mon[94189]: pgmap v5332: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 64 KiB/s wr, 7 op/s 2023-03-09T17:28:46.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:28:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9036.058949 secs 2023-03-09T17:28:46.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:28:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:28:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:28:45 smithi038 ceph-mon[92422]: pgmap v5332: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 64 KiB/s wr, 7 op/s 2023-03-09T17:28:46.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:28:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9036.058949 secs 2023-03-09T17:28:46.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:28:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:28:47.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:28:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:28:47.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:28:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:28:47.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:28:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:28:47.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:28:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:28:47.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:28:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:28:47.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:28:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:28:47.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:28:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:28:47.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:28:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:28:47.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:28:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:28:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:28:47 smithi122 ceph-mon[93869]: pgmap v5333: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 71 KiB/s wr, 9 op/s 2023-03-09T17:28:48.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:28:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9038.327996 secs 2023-03-09T17:28:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:28:47 smithi181 ceph-mon[94189]: pgmap v5333: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 71 KiB/s wr, 9 op/s 2023-03-09T17:28:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:28:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9038.327996 secs 2023-03-09T17:28:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:28:47 smithi038 ceph-mon[92422]: pgmap v5333: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 71 KiB/s wr, 9 op/s 2023-03-09T17:28:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:28:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9038.327996 secs 2023-03-09T17:28:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9500.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:28:50.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:28:49 smithi122 ceph-mon[93869]: pgmap v5334: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 59 KiB/s wr, 8 op/s 2023-03-09T17:28:50.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:28:49 smithi181 ceph-mon[94189]: pgmap v5334: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 59 KiB/s wr, 8 op/s 2023-03-09T17:28:50.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:28:49 smithi038 ceph-mon[92422]: pgmap v5334: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 59 KiB/s wr, 8 op/s 2023-03-09T17:28:51.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:28:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9041.059035 secs 2023-03-09T17:28:51.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:28:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9041.059035 secs 2023-03-09T17:28:51.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:28:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9041.059035 secs 2023-03-09T17:28:52.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:28:51 smithi122 ceph-mon[93869]: pgmap v5335: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 58 KiB/s wr, 8 op/s 2023-03-09T17:28:52.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:28:51 smithi181 ceph-mon[94189]: pgmap v5335: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 58 KiB/s wr, 8 op/s 2023-03-09T17:28:52.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:28:51 smithi038 ceph-mon[92422]: pgmap v5335: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 58 KiB/s wr, 8 op/s 2023-03-09T17:28:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9505.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:28:54.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:28:53 smithi038 ceph-mon[92422]: pgmap v5336: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 138 KiB/s wr, 11 op/s 2023-03-09T17:28:54.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:28:53 smithi122 ceph-mon[93869]: pgmap v5336: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 138 KiB/s wr, 11 op/s 2023-03-09T17:28:54.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:28:53 smithi181 ceph-mon[94189]: pgmap v5336: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 138 KiB/s wr, 11 op/s 2023-03-09T17:28:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:28:55 smithi038 ceph-mon[92422]: pgmap v5337: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 110 KiB/s wr, 10 op/s 2023-03-09T17:28:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:28:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9046.059092 secs 2023-03-09T17:28:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:28:55 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:28:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:28:55 smithi122 ceph-mon[93869]: pgmap v5337: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 110 KiB/s wr, 10 op/s 2023-03-09T17:28:56.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:28:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9046.059092 secs 2023-03-09T17:28:56.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:28:55 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:28:56.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:28:55 smithi181 ceph-mon[94189]: pgmap v5337: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 110 KiB/s wr, 10 op/s 2023-03-09T17:28:56.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:28:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9046.059092 secs 2023-03-09T17:28:56.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:28:55 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:28:57.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:28:56 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:28:57.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:28:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:28:57.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:28:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:28:57.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:28:56 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:28:57.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:28:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:28:57.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:28:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:28:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:28:56 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:28:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:28:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:28:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:28:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:28:58.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:28:58 smithi122 ceph-mon[93869]: pgmap v5338: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 108 KiB/s wr, 12 op/s 2023-03-09T17:28:58.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:28:58 smithi181 ceph-mon[94189]: pgmap v5338: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 108 KiB/s wr, 12 op/s 2023-03-09T17:28:58.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:28:58 smithi038 ceph-mon[92422]: pgmap v5338: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 108 KiB/s wr, 12 op/s 2023-03-09T17:28:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9510.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:29:00.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:29:00 smithi122 ceph-mon[93869]: pgmap v5339: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 117 KiB/s wr, 10 op/s 2023-03-09T17:29:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:29:00 smithi181 ceph-mon[94189]: pgmap v5339: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 117 KiB/s wr, 10 op/s 2023-03-09T17:29:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:29:00 smithi038 ceph-mon[92422]: pgmap v5339: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 117 KiB/s wr, 10 op/s 2023-03-09T17:29:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:29:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9051.059155 secs 2023-03-09T17:29:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:29:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9051.059155 secs 2023-03-09T17:29:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:29:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9051.059155 secs 2023-03-09T17:29:02.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:29:02 smithi122 ceph-mon[93869]: pgmap v5340: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 109 KiB/s wr, 9 op/s 2023-03-09T17:29:02.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:29:02 smithi181 ceph-mon[94189]: pgmap v5340: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 109 KiB/s wr, 9 op/s 2023-03-09T17:29:02.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:29:02 smithi038 ceph-mon[92422]: pgmap v5340: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 109 KiB/s wr, 9 op/s 2023-03-09T17:29:03.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:29:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9053.328320 secs 2023-03-09T17:29:03.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:29:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9053.328320 secs 2023-03-09T17:29:03.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:29:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9053.328320 secs 2023-03-09T17:29:03.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9515.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:29:04.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:29:04 smithi122 ceph-mon[93869]: pgmap v5341: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 186 KiB/s wr, 12 op/s 2023-03-09T17:29:04.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:29:04 smithi181 ceph-mon[94189]: pgmap v5341: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 186 KiB/s wr, 12 op/s 2023-03-09T17:29:04.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:29:04 smithi038 ceph-mon[92422]: pgmap v5341: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 186 KiB/s wr, 12 op/s 2023-03-09T17:29:05.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:29:05 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:29:05.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:29:05 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:29:05.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:29:05 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:29:05.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:29:05 smithi122 ceph-mon[93869]: osdmap e547: 12 total, 12 up, 12 in 2023-03-09T17:29:05.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:29:05 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:29:05.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:29:05 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:29:05.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:29:05 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:29:05.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:29:05 smithi181 ceph-mon[94189]: osdmap e547: 12 total, 12 up, 12 in 2023-03-09T17:29:05.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:29:05 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:29:05.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:29:05 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:29:05.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:29:05 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:29:05.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:29:05 smithi038 ceph-mon[92422]: osdmap e547: 12 total, 12 up, 12 in 2023-03-09T17:29:06.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:29:06 smithi122 ceph-mon[93869]: pgmap v5343: 129 pgs: 8 active+clean+snaptrim_wait, 121 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 153 KiB/s wr, 25 op/s 2023-03-09T17:29:06.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:29:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9056.059494 secs 2023-03-09T17:29:06.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:29:06 smithi181 ceph-mon[94189]: pgmap v5343: 129 pgs: 8 active+clean+snaptrim_wait, 121 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 153 KiB/s wr, 25 op/s 2023-03-09T17:29:06.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:29:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9056.059494 secs 2023-03-09T17:29:06.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:29:06 smithi038 ceph-mon[92422]: pgmap v5343: 129 pgs: 8 active+clean+snaptrim_wait, 121 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 153 KiB/s wr, 25 op/s 2023-03-09T17:29:06.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:29:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9056.059494 secs 2023-03-09T17:29:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:29:08 smithi122 ceph-mon[93869]: pgmap v5344: 129 pgs: 8 active+clean+snaptrim_wait, 121 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 214 KiB/s wr, 29 op/s 2023-03-09T17:29:08.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:29:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9058.328457 secs 2023-03-09T17:29:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:29:08 smithi181 ceph-mon[94189]: pgmap v5344: 129 pgs: 8 active+clean+snaptrim_wait, 121 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 214 KiB/s wr, 29 op/s 2023-03-09T17:29:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:29:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9058.328457 secs 2023-03-09T17:29:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:29:08 smithi038 ceph-mon[92422]: pgmap v5344: 129 pgs: 8 active+clean+snaptrim_wait, 121 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 214 KiB/s wr, 29 op/s 2023-03-09T17:29:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:29:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9058.328457 secs 2023-03-09T17:29:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9520.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:29:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:29:10 smithi122 ceph-mon[93869]: pgmap v5345: 129 pgs: 8 active+clean+snaptrim_wait, 121 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 196 KiB/s wr, 28 op/s 2023-03-09T17:29:10.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:29:10 smithi181 ceph-mon[94189]: pgmap v5345: 129 pgs: 8 active+clean+snaptrim_wait, 121 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 196 KiB/s wr, 28 op/s 2023-03-09T17:29:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:29:10 smithi038 ceph-mon[92422]: pgmap v5345: 129 pgs: 8 active+clean+snaptrim_wait, 121 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 196 KiB/s wr, 28 op/s 2023-03-09T17:29:11.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:29:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9061.059553 secs 2023-03-09T17:29:11.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:29:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9061.059553 secs 2023-03-09T17:29:11.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:29:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9061.059553 secs 2023-03-09T17:29:12.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:29:12 smithi122 ceph-mon[93869]: pgmap v5346: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 196 KiB/s wr, 28 op/s 2023-03-09T17:29:12.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:29:12 smithi181 ceph-mon[94189]: pgmap v5346: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 196 KiB/s wr, 28 op/s 2023-03-09T17:29:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:29:12 smithi038 ceph-mon[92422]: pgmap v5346: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 196 KiB/s wr, 28 op/s 2023-03-09T17:29:13.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:29:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9063.328576 secs 2023-03-09T17:29:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:29:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9063.328576 secs 2023-03-09T17:29:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:29:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9063.328576 secs 2023-03-09T17:29:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9525.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:29:14.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:29:14 smithi122 ceph-mon[93869]: pgmap v5347: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 104 KiB/s wr, 27 op/s 2023-03-09T17:29:14.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:29:14 smithi181 ceph-mon[94189]: pgmap v5347: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 104 KiB/s wr, 27 op/s 2023-03-09T17:29:14.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:29:14 smithi038 ceph-mon[92422]: pgmap v5347: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 104 KiB/s wr, 27 op/s 2023-03-09T17:29:16.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:29:16 smithi122 ceph-mon[93869]: pgmap v5348: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 75 KiB/s wr, 12 op/s 2023-03-09T17:29:16.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:29:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9066.059638 secs 2023-03-09T17:29:16.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:29:16 smithi122 ceph-mon[93869]: osdmap e548: 12 total, 12 up, 12 in 2023-03-09T17:29:16.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:29:16 smithi181 ceph-mon[94189]: pgmap v5348: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 75 KiB/s wr, 12 op/s 2023-03-09T17:29:16.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:29:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9066.059638 secs 2023-03-09T17:29:16.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:29:16 smithi181 ceph-mon[94189]: osdmap e548: 12 total, 12 up, 12 in 2023-03-09T17:29:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:29:16 smithi038 ceph-mon[92422]: pgmap v5348: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 75 KiB/s wr, 12 op/s 2023-03-09T17:29:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:29:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9066.059638 secs 2023-03-09T17:29:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:29:16 smithi038 ceph-mon[92422]: osdmap e548: 12 total, 12 up, 12 in 2023-03-09T17:29:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:29:18 smithi122 ceph-mon[93869]: pgmap v5350: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 10 KiB/s wr, 8 op/s 2023-03-09T17:29:18.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:29:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9068.328716 secs 2023-03-09T17:29:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:29:18 smithi181 ceph-mon[94189]: pgmap v5350: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 10 KiB/s wr, 8 op/s 2023-03-09T17:29:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:29:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9068.328716 secs 2023-03-09T17:29:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:29:18 smithi038 ceph-mon[92422]: pgmap v5350: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 10 KiB/s wr, 8 op/s 2023-03-09T17:29:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:29:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9068.328716 secs 2023-03-09T17:29:18.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9530.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:29:20.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:29:20 smithi122 ceph-mon[93869]: pgmap v5351: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.8 KiB/s wr, 8 op/s 2023-03-09T17:29:20.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:29:20 smithi181 ceph-mon[94189]: pgmap v5351: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.8 KiB/s wr, 8 op/s 2023-03-09T17:29:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:29:20 smithi038 ceph-mon[92422]: pgmap v5351: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.8 KiB/s wr, 8 op/s 2023-03-09T17:29:21.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:29:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9071.059702 secs 2023-03-09T17:29:21.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:29:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9071.059702 secs 2023-03-09T17:29:21.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:29:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9071.059702 secs 2023-03-09T17:29:22.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:29:22 smithi122 ceph-mon[93869]: pgmap v5352: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.8 KiB/s wr, 8 op/s 2023-03-09T17:29:22.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:29:22 smithi181 ceph-mon[94189]: pgmap v5352: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.8 KiB/s wr, 8 op/s 2023-03-09T17:29:22.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:29:22 smithi038 ceph-mon[92422]: pgmap v5352: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.8 KiB/s wr, 8 op/s 2023-03-09T17:29:23.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:29:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9073.328836 secs 2023-03-09T17:29:23.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:29:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9073.328836 secs 2023-03-09T17:29:23.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:29:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9073.328836 secs 2023-03-09T17:29:23.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9535.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:29:24.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:29:24 smithi122 ceph-mon[93869]: pgmap v5353: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 8 op/s 2023-03-09T17:29:24.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:29:24 smithi181 ceph-mon[94189]: pgmap v5353: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 8 op/s 2023-03-09T17:29:24.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:29:24 smithi038 ceph-mon[92422]: pgmap v5353: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 11 KiB/s wr, 8 op/s 2023-03-09T17:29:26.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:29:26 smithi122 ceph-mon[93869]: pgmap v5354: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.8 KiB/s wr, 8 op/s 2023-03-09T17:29:26.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:29:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9076.059762 secs 2023-03-09T17:29:26.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:29:26 smithi181 ceph-mon[94189]: pgmap v5354: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.8 KiB/s wr, 8 op/s 2023-03-09T17:29:26.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:29:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9076.059762 secs 2023-03-09T17:29:26.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:29:26 smithi038 ceph-mon[92422]: pgmap v5354: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.8 KiB/s wr, 8 op/s 2023-03-09T17:29:26.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:29:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9076.059762 secs 2023-03-09T17:29:28.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:29:28 smithi122 ceph-mon[93869]: pgmap v5355: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 26 KiB/s wr, 10 op/s 2023-03-09T17:29:28.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:29:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9078.328969 secs 2023-03-09T17:29:28.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:29:28 smithi181 ceph-mon[94189]: pgmap v5355: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 26 KiB/s wr, 10 op/s 2023-03-09T17:29:28.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:29:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9078.328969 secs 2023-03-09T17:29:28.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:29:28 smithi038 ceph-mon[92422]: pgmap v5355: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 26 KiB/s wr, 10 op/s 2023-03-09T17:29:28.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:29:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9078.328969 secs 2023-03-09T17:29:28.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9540.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:29:30.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:29:30 smithi122 ceph-mon[93869]: pgmap v5356: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T17:29:30.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:29:30 smithi181 ceph-mon[94189]: pgmap v5356: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T17:29:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:29:30 smithi038 ceph-mon[92422]: pgmap v5356: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T17:29:31.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:29:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9081.059822 secs 2023-03-09T17:29:31.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:29:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9081.059822 secs 2023-03-09T17:29:31.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:29:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9081.059822 secs 2023-03-09T17:29:32.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:29:32 smithi122 ceph-mon[93869]: pgmap v5357: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T17:29:32.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:29:32 smithi181 ceph-mon[94189]: pgmap v5357: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T17:29:32.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:29:32 smithi038 ceph-mon[92422]: pgmap v5357: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T17:29:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9545.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:29:34.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:29:34 smithi122 ceph-mon[93869]: pgmap v5358: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 9 op/s 2023-03-09T17:29:34.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:29:34 smithi181 ceph-mon[94189]: pgmap v5358: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 9 op/s 2023-03-09T17:29:34.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:29:34 smithi038 ceph-mon[92422]: pgmap v5358: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 9 op/s 2023-03-09T17:29:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:29:36 smithi122 ceph-mon[93869]: pgmap v5359: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T17:29:36.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:29:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9086.059884 secs 2023-03-09T17:29:36.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:29:36 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:29:36.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:29:36 smithi181 ceph-mon[94189]: pgmap v5359: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T17:29:36.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:29:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9086.059884 secs 2023-03-09T17:29:36.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:29:36 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:29:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:29:36 smithi038 ceph-mon[92422]: pgmap v5359: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T17:29:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:29:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9086.059884 secs 2023-03-09T17:29:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:29:36 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:29:37.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:29:37 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:29:37.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:29:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:29:37.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:29:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:29:37.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:29:37 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:29:37.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:29:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:29:37.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:29:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:29:37.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:29:37 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:29:37.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:29:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:29:37.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:29:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:29:38.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:29:38 smithi122 ceph-mon[93869]: pgmap v5360: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 69 KiB/s wr, 9 op/s 2023-03-09T17:29:38.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:29:38 smithi181 ceph-mon[94189]: pgmap v5360: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 69 KiB/s wr, 9 op/s 2023-03-09T17:29:38.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:29:38 smithi038 ceph-mon[92422]: pgmap v5360: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 69 KiB/s wr, 9 op/s 2023-03-09T17:29:38.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9550.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:29:39.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:29:39 smithi122 ceph-mon[93869]: pgmap v5361: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 55 KiB/s wr, 7 op/s 2023-03-09T17:29:39.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:29:39 smithi181 ceph-mon[94189]: pgmap v5361: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 55 KiB/s wr, 7 op/s 2023-03-09T17:29:39.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:29:39 smithi038 ceph-mon[92422]: pgmap v5361: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 55 KiB/s wr, 7 op/s 2023-03-09T17:29:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:29:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9091.059945 secs 2023-03-09T17:29:40.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:29:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9091.059945 secs 2023-03-09T17:29:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:29:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9091.059945 secs 2023-03-09T17:29:41.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:29:41 smithi122 ceph-mon[93869]: pgmap v5362: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 55 KiB/s wr, 6 op/s 2023-03-09T17:29:41.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:29:41 smithi181 ceph-mon[94189]: pgmap v5362: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 55 KiB/s wr, 6 op/s 2023-03-09T17:29:42.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:29:41 smithi038 ceph-mon[92422]: pgmap v5362: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 55 KiB/s wr, 6 op/s 2023-03-09T17:29:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9555.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:29:43.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:29:43 smithi122 ceph-mon[93869]: pgmap v5363: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 92 KiB/s wr, 10 op/s 2023-03-09T17:29:43.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:29:43 smithi181 ceph-mon[94189]: pgmap v5363: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 92 KiB/s wr, 10 op/s 2023-03-09T17:29:44.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:29:43 smithi038 ceph-mon[92422]: pgmap v5363: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 92 KiB/s wr, 10 op/s 2023-03-09T17:29:45.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:29:45 smithi122 ceph-mon[93869]: pgmap v5364: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 100 KiB/s wr, 9 op/s 2023-03-09T17:29:45.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:29:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9096.060010 secs 2023-03-09T17:29:45.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:29:45 smithi181 ceph-mon[94189]: pgmap v5364: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 100 KiB/s wr, 9 op/s 2023-03-09T17:29:45.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:29:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9096.060010 secs 2023-03-09T17:29:46.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:29:45 smithi038 ceph-mon[92422]: pgmap v5364: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 100 KiB/s wr, 9 op/s 2023-03-09T17:29:46.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:29:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9096.060010 secs 2023-03-09T17:29:46.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:29:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:29:46.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:29:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:29:47.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:29:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:29:47.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:29:47 smithi122 ceph-mon[93869]: pgmap v5365: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 140 KiB/s wr, 11 op/s 2023-03-09T17:29:47.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:29:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:29:47.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:29:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:29:47.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:29:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:29:47.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:29:47 smithi181 ceph-mon[94189]: pgmap v5365: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 140 KiB/s wr, 11 op/s 2023-03-09T17:29:47.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:29:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:29:47.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:29:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:29:47.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:29:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:29:48.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:29:47 smithi038 ceph-mon[92422]: pgmap v5365: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 140 KiB/s wr, 11 op/s 2023-03-09T17:29:48.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:29:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:29:48.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:29:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:29:48.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:29:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:29:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9560.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:29:49.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:29:49 smithi122 ceph-mon[93869]: pgmap v5366: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 102 KiB/s wr, 10 op/s 2023-03-09T17:29:49.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:29:49 smithi181 ceph-mon[94189]: pgmap v5366: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 102 KiB/s wr, 10 op/s 2023-03-09T17:29:50.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:29:49 smithi038 ceph-mon[92422]: pgmap v5366: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 102 KiB/s wr, 10 op/s 2023-03-09T17:29:50.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:29:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9101.060069 secs 2023-03-09T17:29:50.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:29:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9101.060069 secs 2023-03-09T17:29:51.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:29:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9101.060069 secs 2023-03-09T17:29:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:29:51 smithi122 ceph-mon[93869]: pgmap v5367: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 94 KiB/s wr, 9 op/s 2023-03-09T17:29:51.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:29:51 smithi181 ceph-mon[94189]: pgmap v5367: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 94 KiB/s wr, 9 op/s 2023-03-09T17:29:52.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:29:51 smithi038 ceph-mon[92422]: pgmap v5367: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 94 KiB/s wr, 9 op/s 2023-03-09T17:29:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9565.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:29:53.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:29:53 smithi122 ceph-mon[93869]: pgmap v5368: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 130 KiB/s wr, 12 op/s 2023-03-09T17:29:53.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:29:53 smithi181 ceph-mon[94189]: pgmap v5368: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 130 KiB/s wr, 12 op/s 2023-03-09T17:29:54.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:29:53 smithi038 ceph-mon[92422]: pgmap v5368: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 130 KiB/s wr, 12 op/s 2023-03-09T17:29:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:29:55 smithi122 ceph-mon[93869]: pgmap v5369: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 101 KiB/s wr, 9 op/s 2023-03-09T17:29:55.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:29:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9106.060133 secs 2023-03-09T17:29:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:29:55 smithi181 ceph-mon[94189]: pgmap v5369: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 101 KiB/s wr, 9 op/s 2023-03-09T17:29:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:29:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9106.060133 secs 2023-03-09T17:29:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:29:55 smithi038 ceph-mon[92422]: pgmap v5369: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 101 KiB/s wr, 9 op/s 2023-03-09T17:29:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:29:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9106.060133 secs 2023-03-09T17:29:57.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:29:57 smithi122 ceph-mon[93869]: pgmap v5370: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 155 KiB/s wr, 11 op/s 2023-03-09T17:29:57.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:29:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9108.329594 secs 2023-03-09T17:29:57.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:29:57 smithi181 ceph-mon[94189]: pgmap v5370: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 155 KiB/s wr, 11 op/s 2023-03-09T17:29:57.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:29:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9108.329594 secs 2023-03-09T17:29:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:29:57 smithi038 ceph-mon[92422]: pgmap v5370: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 155 KiB/s wr, 11 op/s 2023-03-09T17:29:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:29:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9108.329594 secs 2023-03-09T17:29:58.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9570.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:29:59.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:29:59 smithi122 ceph-mon[93869]: pgmap v5371: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 123 KiB/s wr, 9 op/s 2023-03-09T17:29:59.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:29:59 smithi181 ceph-mon[94189]: pgmap v5371: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 123 KiB/s wr, 9 op/s 2023-03-09T17:30:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:29:59 smithi038 ceph-mon[92422]: pgmap v5371: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 123 KiB/s wr, 9 op/s 2023-03-09T17:30:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9111.060196 secs 2023-03-09T17:30:00.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:00 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:30:00.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:30:00.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:30:00.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:00 smithi122 ceph-mon[93869]: overall HEALTH_WARN 2 MDSs report slow requests 2023-03-09T17:30:00.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9111.060196 secs 2023-03-09T17:30:00.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:00 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:30:00.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:30:00.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:30:00.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:00 smithi181 ceph-mon[94189]: overall HEALTH_WARN 2 MDSs report slow requests 2023-03-09T17:30:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9111.060196 secs 2023-03-09T17:30:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:00 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:30:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:30:01.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:30:01.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:00 smithi038 ceph-mon[92422]: overall HEALTH_WARN 2 MDSs report slow requests 2023-03-09T17:30:01.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:01 smithi122 ceph-mon[93869]: pgmap v5372: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 114 KiB/s wr, 8 op/s 2023-03-09T17:30:01.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:01 smithi181 ceph-mon[94189]: pgmap v5372: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 114 KiB/s wr, 8 op/s 2023-03-09T17:30:02.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:01 smithi038 ceph-mon[92422]: pgmap v5372: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 114 KiB/s wr, 8 op/s 2023-03-09T17:30:02.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9113.329715 secs 2023-03-09T17:30:02.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9113.329715 secs 2023-03-09T17:30:03.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9113.329715 secs 2023-03-09T17:30:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9575.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:30:04.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:03 smithi038 ceph-mon[92422]: pgmap v5373: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 133 KiB/s wr, 11 op/s 2023-03-09T17:30:04.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:03 smithi122 ceph-mon[93869]: pgmap v5373: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 133 KiB/s wr, 11 op/s 2023-03-09T17:30:04.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:03 smithi181 ceph-mon[94189]: pgmap v5373: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 133 KiB/s wr, 11 op/s 2023-03-09T17:30:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:05 smithi122 ceph-mon[93869]: pgmap v5374: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 117 KiB/s wr, 21 op/s 2023-03-09T17:30:06.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9116.060258 secs 2023-03-09T17:30:06.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:05 smithi122 ceph-mon[93869]: osdmap e549: 12 total, 12 up, 12 in 2023-03-09T17:30:06.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:05 smithi181 ceph-mon[94189]: pgmap v5374: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 117 KiB/s wr, 21 op/s 2023-03-09T17:30:06.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9116.060258 secs 2023-03-09T17:30:06.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:05 smithi181 ceph-mon[94189]: osdmap e549: 12 total, 12 up, 12 in 2023-03-09T17:30:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:05 smithi038 ceph-mon[92422]: pgmap v5374: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 117 KiB/s wr, 21 op/s 2023-03-09T17:30:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9116.060258 secs 2023-03-09T17:30:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:05 smithi038 ceph-mon[92422]: osdmap e549: 12 total, 12 up, 12 in 2023-03-09T17:30:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:07 smithi122 ceph-mon[93869]: pgmap v5376: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 142 KiB/s wr, 26 op/s 2023-03-09T17:30:08.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9118.329781 secs 2023-03-09T17:30:08.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:07 smithi181 ceph-mon[94189]: pgmap v5376: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 142 KiB/s wr, 26 op/s 2023-03-09T17:30:08.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9118.329781 secs 2023-03-09T17:30:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:07 smithi038 ceph-mon[92422]: pgmap v5376: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 142 KiB/s wr, 26 op/s 2023-03-09T17:30:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9118.329781 secs 2023-03-09T17:30:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9580.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:30:10.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:09 smithi122 ceph-mon[93869]: pgmap v5377: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 142 KiB/s wr, 26 op/s 2023-03-09T17:30:10.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:09 smithi181 ceph-mon[94189]: pgmap v5377: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 142 KiB/s wr, 26 op/s 2023-03-09T17:30:10.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:09 smithi038 ceph-mon[92422]: pgmap v5377: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 142 KiB/s wr, 26 op/s 2023-03-09T17:30:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9121.060372 secs 2023-03-09T17:30:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9121.060372 secs 2023-03-09T17:30:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9121.060372 secs 2023-03-09T17:30:12.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:11 smithi122 ceph-mon[93869]: pgmap v5378: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 142 KiB/s wr, 26 op/s 2023-03-09T17:30:12.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:11 smithi181 ceph-mon[94189]: pgmap v5378: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 142 KiB/s wr, 26 op/s 2023-03-09T17:30:12.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:11 smithi038 ceph-mon[92422]: pgmap v5378: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 142 KiB/s wr, 26 op/s 2023-03-09T17:30:13.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9123.329899 secs 2023-03-09T17:30:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9123.329899 secs 2023-03-09T17:30:13.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9123.329899 secs 2023-03-09T17:30:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9585.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:30:14.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:13 smithi122 ceph-mon[93869]: pgmap v5379: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 137 KiB/s wr, 26 op/s 2023-03-09T17:30:14.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:13 smithi181 ceph-mon[94189]: pgmap v5379: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 137 KiB/s wr, 26 op/s 2023-03-09T17:30:14.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:13 smithi038 ceph-mon[92422]: pgmap v5379: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 137 KiB/s wr, 26 op/s 2023-03-09T17:30:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:15 smithi122 ceph-mon[93869]: pgmap v5380: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 112 KiB/s wr, 11 op/s 2023-03-09T17:30:16.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9126.060458 secs 2023-03-09T17:30:16.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:15 smithi122 ceph-mon[93869]: osdmap e550: 12 total, 12 up, 12 in 2023-03-09T17:30:16.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:15 smithi181 ceph-mon[94189]: pgmap v5380: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 112 KiB/s wr, 11 op/s 2023-03-09T17:30:16.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9126.060458 secs 2023-03-09T17:30:16.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:15 smithi181 ceph-mon[94189]: osdmap e550: 12 total, 12 up, 12 in 2023-03-09T17:30:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:15 smithi038 ceph-mon[92422]: pgmap v5380: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 112 KiB/s wr, 11 op/s 2023-03-09T17:30:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9126.060458 secs 2023-03-09T17:30:16.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:15 smithi038 ceph-mon[92422]: osdmap e550: 12 total, 12 up, 12 in 2023-03-09T17:30:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:17 smithi122 ceph-mon[93869]: pgmap v5382: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T17:30:18.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9128.330030 secs 2023-03-09T17:30:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:17 smithi181 ceph-mon[94189]: pgmap v5382: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T17:30:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9128.330030 secs 2023-03-09T17:30:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:17 smithi038 ceph-mon[92422]: pgmap v5382: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T17:30:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9128.330030 secs 2023-03-09T17:30:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9590.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:30:20.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:19 smithi122 ceph-mon[93869]: pgmap v5383: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T17:30:20.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:19 smithi181 ceph-mon[94189]: pgmap v5383: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T17:30:20.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:19 smithi038 ceph-mon[92422]: pgmap v5383: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T17:30:21.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9131.060517 secs 2023-03-09T17:30:21.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9131.060517 secs 2023-03-09T17:30:21.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9131.060517 secs 2023-03-09T17:30:22.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:21 smithi122 ceph-mon[93869]: pgmap v5384: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T17:30:22.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:21 smithi181 ceph-mon[94189]: pgmap v5384: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T17:30:22.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:21 smithi038 ceph-mon[92422]: pgmap v5384: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T17:30:23.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9133.330127 secs 2023-03-09T17:30:23.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9133.330127 secs 2023-03-09T17:30:23.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9133.330127 secs 2023-03-09T17:30:23.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9595.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:30:24.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:23 smithi122 ceph-mon[93869]: pgmap v5385: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 3.6 KiB/s wr, 6 op/s 2023-03-09T17:30:24.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:23 smithi181 ceph-mon[94189]: pgmap v5385: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 3.6 KiB/s wr, 6 op/s 2023-03-09T17:30:24.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:23 smithi038 ceph-mon[92422]: pgmap v5385: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 3.6 KiB/s wr, 6 op/s 2023-03-09T17:30:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:25 smithi122 ceph-mon[93869]: pgmap v5386: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 45 KiB/s wr, 8 op/s 2023-03-09T17:30:26.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9136.060578 secs 2023-03-09T17:30:26.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:25 smithi181 ceph-mon[94189]: pgmap v5386: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 45 KiB/s wr, 8 op/s 2023-03-09T17:30:26.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9136.060578 secs 2023-03-09T17:30:26.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:25 smithi038 ceph-mon[92422]: pgmap v5386: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 45 KiB/s wr, 8 op/s 2023-03-09T17:30:26.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9136.060578 secs 2023-03-09T17:30:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:27 smithi122 ceph-mon[93869]: pgmap v5387: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 43 KiB/s wr, 9 op/s 2023-03-09T17:30:28.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9138.330207 secs 2023-03-09T17:30:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:27 smithi181 ceph-mon[94189]: pgmap v5387: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 43 KiB/s wr, 9 op/s 2023-03-09T17:30:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9138.330207 secs 2023-03-09T17:30:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:27 smithi038 ceph-mon[92422]: pgmap v5387: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 43 KiB/s wr, 9 op/s 2023-03-09T17:30:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9138.330207 secs 2023-03-09T17:30:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9600.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:30:30.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:29 smithi122 ceph-mon[93869]: pgmap v5388: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 38 KiB/s wr, 8 op/s 2023-03-09T17:30:30.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:29 smithi181 ceph-mon[94189]: pgmap v5388: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 38 KiB/s wr, 8 op/s 2023-03-09T17:30:30.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:29 smithi038 ceph-mon[92422]: pgmap v5388: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 38 KiB/s wr, 8 op/s 2023-03-09T17:30:31.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9141.060641 secs 2023-03-09T17:30:31.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9141.060641 secs 2023-03-09T17:30:31.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9141.060641 secs 2023-03-09T17:30:32.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:31 smithi122 ceph-mon[93869]: pgmap v5389: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 38 KiB/s wr, 8 op/s 2023-03-09T17:30:32.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:31 smithi181 ceph-mon[94189]: pgmap v5389: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 38 KiB/s wr, 8 op/s 2023-03-09T17:30:32.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:31 smithi038 ceph-mon[92422]: pgmap v5389: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 38 KiB/s wr, 8 op/s 2023-03-09T17:30:33.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:32 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 9143.330305 secs 2023-03-09T17:30:33.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:32 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 9143.330305 secs 2023-03-09T17:30:33.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:32 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 9143.330305 secs 2023-03-09T17:30:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9605.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:30:34.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:33 smithi122 ceph-mon[93869]: pgmap v5390: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 38 KiB/s wr, 10 op/s 2023-03-09T17:30:34.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:33 smithi181 ceph-mon[94189]: pgmap v5390: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 38 KiB/s wr, 10 op/s 2023-03-09T17:30:34.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:33 smithi038 ceph-mon[92422]: pgmap v5390: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 38 KiB/s wr, 10 op/s 2023-03-09T17:30:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:35 smithi122 ceph-mon[93869]: pgmap v5391: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 82 KiB/s wr, 9 op/s 2023-03-09T17:30:36.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9146.060702 secs 2023-03-09T17:30:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:35 smithi181 ceph-mon[94189]: pgmap v5391: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 82 KiB/s wr, 9 op/s 2023-03-09T17:30:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9146.060702 secs 2023-03-09T17:30:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:35 smithi038 ceph-mon[92422]: pgmap v5391: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 82 KiB/s wr, 9 op/s 2023-03-09T17:30:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9146.060702 secs 2023-03-09T17:30:37.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:30:37.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:30:37.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:30:37.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:30:37.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:30:37.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:30:38.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:37 smithi122 ceph-mon[93869]: pgmap v5392: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 47 KiB/s wr, 9 op/s 2023-03-09T17:30:38.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9148.330392 secs 2023-03-09T17:30:38.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:37 smithi181 ceph-mon[94189]: pgmap v5392: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 47 KiB/s wr, 9 op/s 2023-03-09T17:30:38.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9148.330392 secs 2023-03-09T17:30:38.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:37 smithi038 ceph-mon[92422]: pgmap v5392: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 47 KiB/s wr, 9 op/s 2023-03-09T17:30:38.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9148.330392 secs 2023-03-09T17:30:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9610.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:30:40.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:39 smithi122 ceph-mon[93869]: pgmap v5393: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 8 op/s 2023-03-09T17:30:40.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:39 smithi181 ceph-mon[94189]: pgmap v5393: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 8 op/s 2023-03-09T17:30:40.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:39 smithi038 ceph-mon[92422]: pgmap v5393: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 8 op/s 2023-03-09T17:30:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9151.060762 secs 2023-03-09T17:30:41.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:30:41.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:30:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9151.060762 secs 2023-03-09T17:30:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:30:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:30:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9151.060762 secs 2023-03-09T17:30:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:30:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:30:42.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:41 smithi122 ceph-mon[93869]: pgmap v5394: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 59 KiB/s wr, 7 op/s 2023-03-09T17:30:42.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:41 smithi181 ceph-mon[94189]: pgmap v5394: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 59 KiB/s wr, 7 op/s 2023-03-09T17:30:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:41 smithi038 ceph-mon[92422]: pgmap v5394: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 59 KiB/s wr, 7 op/s 2023-03-09T17:30:43.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9153.330543 secs 2023-03-09T17:30:43.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9153.330543 secs 2023-03-09T17:30:43.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9153.330543 secs 2023-03-09T17:30:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9615.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:30:44.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:43 smithi122 ceph-mon[93869]: pgmap v5395: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 59 KiB/s wr, 10 op/s 2023-03-09T17:30:44.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:43 smithi181 ceph-mon[94189]: pgmap v5395: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 59 KiB/s wr, 10 op/s 2023-03-09T17:30:44.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:43 smithi038 ceph-mon[92422]: pgmap v5395: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 59 KiB/s wr, 10 op/s 2023-03-09T17:30:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:45 smithi122 ceph-mon[93869]: pgmap v5396: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 141 KiB/s wr, 10 op/s 2023-03-09T17:30:46.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9156.060822 secs 2023-03-09T17:30:46.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:45 smithi181 ceph-mon[94189]: pgmap v5396: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 141 KiB/s wr, 10 op/s 2023-03-09T17:30:46.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9156.060822 secs 2023-03-09T17:30:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:45 smithi038 ceph-mon[92422]: pgmap v5396: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 141 KiB/s wr, 10 op/s 2023-03-09T17:30:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9156.060822 secs 2023-03-09T17:30:48.221 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:47 smithi122 ceph-mon[93869]: pgmap v5397: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 96 KiB/s wr, 11 op/s 2023-03-09T17:30:48.221 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9158.330673 secs 2023-03-09T17:30:48.221 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:30:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:47 smithi181 ceph-mon[94189]: pgmap v5397: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 96 KiB/s wr, 11 op/s 2023-03-09T17:30:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9158.330673 secs 2023-03-09T17:30:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:30:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:47 smithi038 ceph-mon[92422]: pgmap v5397: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 96 KiB/s wr, 11 op/s 2023-03-09T17:30:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9158.330673 secs 2023-03-09T17:30:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:30:48.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9620.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:30:50.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:49 smithi122 ceph-mon[93869]: pgmap v5398: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 132 KiB/s wr, 11 op/s 2023-03-09T17:30:50.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:30:50.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:30:50.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:49 smithi181 ceph-mon[94189]: pgmap v5398: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 132 KiB/s wr, 11 op/s 2023-03-09T17:30:50.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:30:50.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:30:50.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:49 smithi038 ceph-mon[92422]: pgmap v5398: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 132 KiB/s wr, 11 op/s 2023-03-09T17:30:50.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:30:50.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:30:51.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9161.060881 secs 2023-03-09T17:30:51.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:30:51.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:30:51.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:30:51.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:30:51.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:30:51.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:30:51.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:30:51.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:30:51.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:30:51.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:30:51.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9161.060881 secs 2023-03-09T17:30:51.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:30:51.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:30:51.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:30:51.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:30:51.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:30:51.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:30:51.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:30:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:30:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:30:51.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:30:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9161.060881 secs 2023-03-09T17:30:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:30:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:30:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:30:51.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:30:51.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:30:51.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:30:51.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:30:51.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:30:51.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:30:51.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:30:52.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:52 smithi122 ceph-mon[93869]: pgmap v5399: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 119 KiB/s wr, 10 op/s 2023-03-09T17:30:52.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:52 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T17:30:52.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:52 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T17:30:52.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:52 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi038 to 197.2M 2023-03-09T17:30:52.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:52 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi038 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-03-09T17:30:52.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:30:52.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:30:52.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:52 smithi181 ceph-mon[94189]: pgmap v5399: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 119 KiB/s wr, 10 op/s 2023-03-09T17:30:52.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:52 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T17:30:52.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:52 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T17:30:52.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:52 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi038 to 197.2M 2023-03-09T17:30:52.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:52 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi038 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-03-09T17:30:52.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:30:52.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:30:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:52 smithi038 ceph-mon[92422]: pgmap v5399: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 119 KiB/s wr, 10 op/s 2023-03-09T17:30:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:52 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T17:30:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:52 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T17:30:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:52 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi038 to 197.2M 2023-03-09T17:30:52.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:52 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi038 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-03-09T17:30:52.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:30:52.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:30:53.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9163.330790 secs 2023-03-09T17:30:53.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:30:53.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:30:53.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:30:53.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:30:53.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:30:53.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:30:53.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:30:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9163.330790 secs 2023-03-09T17:30:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:30:53.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:30:53.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:30:53.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:30:53.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:30:53.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:30:53.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:30:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9163.330790 secs 2023-03-09T17:30:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:30:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:30:53.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:30:53.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:30:53.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:30:53.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:30:53.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:30:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9625.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:30:54.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:54 smithi122 ceph-mon[93869]: pgmap v5400: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 120 KiB/s wr, 11 op/s 2023-03-09T17:30:54.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:54 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T17:30:54.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:54 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T17:30:54.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:54 smithi181 ceph-mon[94189]: pgmap v5400: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 120 KiB/s wr, 11 op/s 2023-03-09T17:30:54.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:54 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T17:30:54.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:54 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T17:30:54.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:54 smithi038 ceph-mon[92422]: pgmap v5400: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 120 KiB/s wr, 11 op/s 2023-03-09T17:30:54.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:54 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T17:30:54.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:54 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T17:30:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:56 smithi122 ceph-mon[93869]: pgmap v5401: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 196 KiB/s wr, 11 op/s 2023-03-09T17:30:56.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9166.060939 secs 2023-03-09T17:30:56.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:56 smithi181 ceph-mon[94189]: pgmap v5401: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 196 KiB/s wr, 11 op/s 2023-03-09T17:30:56.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9166.060939 secs 2023-03-09T17:30:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:56 smithi038 ceph-mon[92422]: pgmap v5401: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 196 KiB/s wr, 11 op/s 2023-03-09T17:30:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9166.060939 secs 2023-03-09T17:30:58.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:58 smithi122 ceph-mon[93869]: pgmap v5402: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 114 KiB/s wr, 10 op/s 2023-03-09T17:30:58.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:30:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9168.330886 secs 2023-03-09T17:30:58.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:58 smithi181 ceph-mon[94189]: pgmap v5402: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 114 KiB/s wr, 10 op/s 2023-03-09T17:30:58.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:30:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9168.330886 secs 2023-03-09T17:30:58.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:58 smithi038 ceph-mon[92422]: pgmap v5402: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 114 KiB/s wr, 10 op/s 2023-03-09T17:30:58.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:30:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9168.330886 secs 2023-03-09T17:30:58.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9630.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:31:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:31:00 smithi122 ceph-mon[93869]: pgmap v5403: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 114 KiB/s wr, 10 op/s 2023-03-09T17:31:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:31:00 smithi181 ceph-mon[94189]: pgmap v5403: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 114 KiB/s wr, 10 op/s 2023-03-09T17:31:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:31:00 smithi038 ceph-mon[92422]: pgmap v5403: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 114 KiB/s wr, 10 op/s 2023-03-09T17:31:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:31:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9171.060986 secs 2023-03-09T17:31:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:31:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9171.060986 secs 2023-03-09T17:31:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:31:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9171.060986 secs 2023-03-09T17:31:02.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:31:02 smithi122 ceph-mon[93869]: pgmap v5404: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 78 KiB/s wr, 7 op/s 2023-03-09T17:31:02.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:31:02 smithi181 ceph-mon[94189]: pgmap v5404: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 78 KiB/s wr, 7 op/s 2023-03-09T17:31:02.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:31:02 smithi038 ceph-mon[92422]: pgmap v5404: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 78 KiB/s wr, 7 op/s 2023-03-09T17:31:03.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:31:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9173.331002 secs 2023-03-09T17:31:03.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:31:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9173.331002 secs 2023-03-09T17:31:03.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:31:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9173.331002 secs 2023-03-09T17:31:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9635.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:31:04.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:31:04 smithi122 ceph-mon[93869]: pgmap v5405: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 78 KiB/s wr, 9 op/s 2023-03-09T17:31:04.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:31:04 smithi181 ceph-mon[94189]: pgmap v5405: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 78 KiB/s wr, 9 op/s 2023-03-09T17:31:04.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:31:04 smithi038 ceph-mon[92422]: pgmap v5405: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 78 KiB/s wr, 9 op/s 2023-03-09T17:31:06.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:31:06 smithi122 ceph-mon[93869]: pgmap v5406: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 122 KiB/s wr, 21 op/s 2023-03-09T17:31:06.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:31:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9176.061048 secs 2023-03-09T17:31:06.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:31:06 smithi181 ceph-mon[94189]: pgmap v5406: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 122 KiB/s wr, 21 op/s 2023-03-09T17:31:06.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:31:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9176.061048 secs 2023-03-09T17:31:06.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:31:06 smithi038 ceph-mon[92422]: pgmap v5406: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 122 KiB/s wr, 21 op/s 2023-03-09T17:31:06.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:31:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9176.061048 secs 2023-03-09T17:31:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:31:08 smithi122 ceph-mon[93869]: pgmap v5407: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 72 KiB/s wr, 23 op/s 2023-03-09T17:31:08.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:31:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9178.331074 secs 2023-03-09T17:31:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:31:08 smithi181 ceph-mon[94189]: pgmap v5407: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 72 KiB/s wr, 23 op/s 2023-03-09T17:31:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:31:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9178.331074 secs 2023-03-09T17:31:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:31:08 smithi038 ceph-mon[92422]: pgmap v5407: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 72 KiB/s wr, 23 op/s 2023-03-09T17:31:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:31:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9178.331074 secs 2023-03-09T17:31:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9640.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:31:10.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:31:10 smithi122 ceph-mon[93869]: pgmap v5408: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 72 KiB/s wr, 23 op/s 2023-03-09T17:31:10.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:31:10 smithi122 ceph-mon[93869]: osdmap e551: 12 total, 12 up, 12 in 2023-03-09T17:31:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:31:10 smithi181 ceph-mon[94189]: pgmap v5408: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 72 KiB/s wr, 23 op/s 2023-03-09T17:31:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:31:10 smithi181 ceph-mon[94189]: osdmap e551: 12 total, 12 up, 12 in 2023-03-09T17:31:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:31:10 smithi038 ceph-mon[92422]: pgmap v5408: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 72 KiB/s wr, 23 op/s 2023-03-09T17:31:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:31:10 smithi038 ceph-mon[92422]: osdmap e551: 12 total, 12 up, 12 in 2023-03-09T17:31:11.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:31:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9181.061109 secs 2023-03-09T17:31:11.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:31:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9181.061109 secs 2023-03-09T17:31:11.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:31:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9181.061109 secs 2023-03-09T17:31:12.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:31:12 smithi122 ceph-mon[93869]: pgmap v5410: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 86 KiB/s wr, 26 op/s 2023-03-09T17:31:12.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:31:12 smithi181 ceph-mon[94189]: pgmap v5410: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 86 KiB/s wr, 26 op/s 2023-03-09T17:31:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:31:12 smithi038 ceph-mon[92422]: pgmap v5410: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 86 KiB/s wr, 26 op/s 2023-03-09T17:31:13.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:31:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9183.331186 secs 2023-03-09T17:31:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:31:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9183.331186 secs 2023-03-09T17:31:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:31:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9183.331186 secs 2023-03-09T17:31:13.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9645.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:31:14.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:31:14 smithi122 ceph-mon[93869]: pgmap v5411: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 96 KiB/s wr, 27 op/s 2023-03-09T17:31:14.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:31:14 smithi181 ceph-mon[94189]: pgmap v5411: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 96 KiB/s wr, 27 op/s 2023-03-09T17:31:14.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:31:14 smithi038 ceph-mon[92422]: pgmap v5411: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 96 KiB/s wr, 27 op/s 2023-03-09T17:31:15.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:31:15 smithi122 ceph-mon[93869]: pgmap v5412: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 60 KiB/s wr, 11 op/s 2023-03-09T17:31:15.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:31:15 smithi181 ceph-mon[94189]: pgmap v5412: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 60 KiB/s wr, 11 op/s 2023-03-09T17:31:15.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:31:15 smithi038 ceph-mon[92422]: pgmap v5412: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 60 KiB/s wr, 11 op/s 2023-03-09T17:31:16.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:31:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9186.061169 secs 2023-03-09T17:31:16.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:31:16 smithi122 ceph-mon[93869]: osdmap e552: 12 total, 12 up, 12 in 2023-03-09T17:31:16.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:31:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9186.061169 secs 2023-03-09T17:31:16.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:31:16 smithi181 ceph-mon[94189]: osdmap e552: 12 total, 12 up, 12 in 2023-03-09T17:31:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:31:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9186.061169 secs 2023-03-09T17:31:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:31:16 smithi038 ceph-mon[92422]: osdmap e552: 12 total, 12 up, 12 in 2023-03-09T17:31:17.481 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:31:17 smithi122 ceph-mon[93869]: pgmap v5414: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 KiB/s wr, 9 op/s 2023-03-09T17:31:17.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:31:17 smithi181 ceph-mon[94189]: pgmap v5414: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 KiB/s wr, 9 op/s 2023-03-09T17:31:17.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:31:17 smithi038 ceph-mon[92422]: pgmap v5414: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 KiB/s wr, 9 op/s 2023-03-09T17:31:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:31:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9188.331265 secs 2023-03-09T17:31:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:31:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9188.331265 secs 2023-03-09T17:31:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:31:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9188.331265 secs 2023-03-09T17:31:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9650.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:31:19.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:31:19 smithi122 ceph-mon[93869]: pgmap v5415: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 60 KiB/s wr, 9 op/s 2023-03-09T17:31:19.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:31:19 smithi181 ceph-mon[94189]: pgmap v5415: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 60 KiB/s wr, 9 op/s 2023-03-09T17:31:19.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:31:19 smithi038 ceph-mon[92422]: pgmap v5415: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 60 KiB/s wr, 9 op/s 2023-03-09T17:31:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:31:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9191.061232 secs 2023-03-09T17:31:20.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:31:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9191.061232 secs 2023-03-09T17:31:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:31:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9191.061232 secs 2023-03-09T17:31:21.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:31:21 smithi122 ceph-mon[93869]: pgmap v5416: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 56 KiB/s wr, 8 op/s 2023-03-09T17:31:21.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:31:21 smithi181 ceph-mon[94189]: pgmap v5416: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 56 KiB/s wr, 8 op/s 2023-03-09T17:31:22.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:31:21 smithi038 ceph-mon[92422]: pgmap v5416: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 56 KiB/s wr, 8 op/s 2023-03-09T17:31:22.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:31:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9193.331390 secs 2023-03-09T17:31:22.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:31:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9193.331390 secs 2023-03-09T17:31:23.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:31:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9193.331390 secs 2023-03-09T17:31:23.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9655.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:31:23.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:31:23 smithi122 ceph-mon[93869]: pgmap v5417: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 46 KiB/s wr, 8 op/s 2023-03-09T17:31:23.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:31:23 smithi181 ceph-mon[94189]: pgmap v5417: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 46 KiB/s wr, 8 op/s 2023-03-09T17:31:24.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:31:23 smithi038 ceph-mon[92422]: pgmap v5417: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 46 KiB/s wr, 8 op/s 2023-03-09T17:31:25.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:31:25 smithi122 ceph-mon[93869]: pgmap v5418: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 29 KiB/s wr, 7 op/s 2023-03-09T17:31:25.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:31:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9196.061326 secs 2023-03-09T17:31:25.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:31:25 smithi181 ceph-mon[94189]: pgmap v5418: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 29 KiB/s wr, 7 op/s 2023-03-09T17:31:25.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:31:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9196.061326 secs 2023-03-09T17:31:26.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:31:25 smithi038 ceph-mon[92422]: pgmap v5418: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 29 KiB/s wr, 7 op/s 2023-03-09T17:31:26.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:31:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9196.061326 secs 2023-03-09T17:31:27.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:31:27 smithi122 ceph-mon[93869]: pgmap v5419: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T17:31:27.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:31:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9198.331513 secs 2023-03-09T17:31:27.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:31:27 smithi181 ceph-mon[94189]: pgmap v5419: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T17:31:27.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:31:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9198.331513 secs 2023-03-09T17:31:28.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:31:27 smithi038 ceph-mon[92422]: pgmap v5419: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T17:31:28.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:31:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9198.331513 secs 2023-03-09T17:31:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9660.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:31:29.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:31:29 smithi122 ceph-mon[93869]: pgmap v5420: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 46 KiB/s wr, 7 op/s 2023-03-09T17:31:29.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:31:29 smithi181 ceph-mon[94189]: pgmap v5420: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 46 KiB/s wr, 7 op/s 2023-03-09T17:31:30.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:31:29 smithi038 ceph-mon[92422]: pgmap v5420: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 46 KiB/s wr, 7 op/s 2023-03-09T17:31:30.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:31:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9201.061413 secs 2023-03-09T17:31:30.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:31:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9201.061413 secs 2023-03-09T17:31:31.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:31:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9201.061413 secs 2023-03-09T17:31:31.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:31:31 smithi122 ceph-mon[93869]: pgmap v5421: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 28 KiB/s wr, 6 op/s 2023-03-09T17:31:31.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:31:31 smithi181 ceph-mon[94189]: pgmap v5421: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 28 KiB/s wr, 6 op/s 2023-03-09T17:31:32.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:31:31 smithi038 ceph-mon[92422]: pgmap v5421: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 28 KiB/s wr, 6 op/s 2023-03-09T17:31:33.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:31:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9203.331649 secs 2023-03-09T17:31:33.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:31:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9203.331649 secs 2023-03-09T17:31:33.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:31:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9203.331649 secs 2023-03-09T17:31:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9665.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:31:34.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:31:33 smithi038 ceph-mon[92422]: pgmap v5422: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 28 KiB/s wr, 9 op/s 2023-03-09T17:31:34.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:31:33 smithi122 ceph-mon[93869]: pgmap v5422: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 28 KiB/s wr, 9 op/s 2023-03-09T17:31:34.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:31:33 smithi181 ceph-mon[94189]: pgmap v5422: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 28 KiB/s wr, 9 op/s 2023-03-09T17:31:36.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:31:35 smithi038 ceph-mon[92422]: pgmap v5423: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 43 KiB/s wr, 7 op/s 2023-03-09T17:31:36.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:31:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9206.061473 secs 2023-03-09T17:31:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:31:35 smithi122 ceph-mon[93869]: pgmap v5423: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 43 KiB/s wr, 7 op/s 2023-03-09T17:31:36.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:31:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9206.061473 secs 2023-03-09T17:31:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:31:35 smithi181 ceph-mon[94189]: pgmap v5423: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 43 KiB/s wr, 7 op/s 2023-03-09T17:31:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:31:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9206.061473 secs 2023-03-09T17:31:38.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:31:37 smithi038 ceph-mon[92422]: pgmap v5424: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 48 KiB/s wr, 9 op/s 2023-03-09T17:31:38.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:31:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9208.331770 secs 2023-03-09T17:31:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:31:37 smithi122 ceph-mon[93869]: pgmap v5424: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 48 KiB/s wr, 9 op/s 2023-03-09T17:31:38.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:31:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9208.331770 secs 2023-03-09T17:31:38.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:31:37 smithi181 ceph-mon[94189]: pgmap v5424: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 48 KiB/s wr, 9 op/s 2023-03-09T17:31:38.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:31:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9208.331770 secs 2023-03-09T17:31:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9670.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:31:40.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:31:39 smithi038 ceph-mon[92422]: pgmap v5425: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 47 KiB/s wr, 7 op/s 2023-03-09T17:31:40.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:31:39 smithi122 ceph-mon[93869]: pgmap v5425: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 47 KiB/s wr, 7 op/s 2023-03-09T17:31:40.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:31:39 smithi181 ceph-mon[94189]: pgmap v5425: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 47 KiB/s wr, 7 op/s 2023-03-09T17:31:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:31:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9211.061534 secs 2023-03-09T17:31:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:31:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9211.061534 secs 2023-03-09T17:31:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:31:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9211.061534 secs 2023-03-09T17:31:42.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:31:41 smithi122 ceph-mon[93869]: pgmap v5426: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 62 KiB/s wr, 7 op/s 2023-03-09T17:31:42.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:31:41 smithi181 ceph-mon[94189]: pgmap v5426: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 62 KiB/s wr, 7 op/s 2023-03-09T17:31:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:31:41 smithi038 ceph-mon[92422]: pgmap v5426: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 62 KiB/s wr, 7 op/s 2023-03-09T17:31:43.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:31:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9213.331882 secs 2023-03-09T17:31:43.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:31:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9213.331882 secs 2023-03-09T17:31:43.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:31:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9213.331882 secs 2023-03-09T17:31:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9675.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:31:44.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:31:43 smithi122 ceph-mon[93869]: pgmap v5427: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 66 KiB/s wr, 10 op/s 2023-03-09T17:31:44.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:31:43 smithi181 ceph-mon[94189]: pgmap v5427: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 66 KiB/s wr, 10 op/s 2023-03-09T17:31:44.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:31:43 smithi038 ceph-mon[92422]: pgmap v5427: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 66 KiB/s wr, 10 op/s 2023-03-09T17:31:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:31:45 smithi122 ceph-mon[93869]: pgmap v5428: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 66 KiB/s wr, 8 op/s 2023-03-09T17:31:46.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:31:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9216.061597 secs 2023-03-09T17:31:46.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:31:45 smithi181 ceph-mon[94189]: pgmap v5428: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 66 KiB/s wr, 8 op/s 2023-03-09T17:31:46.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:31:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9216.061597 secs 2023-03-09T17:31:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:31:45 smithi038 ceph-mon[92422]: pgmap v5428: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 66 KiB/s wr, 8 op/s 2023-03-09T17:31:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:31:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9216.061597 secs 2023-03-09T17:31:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:31:47 smithi122 ceph-mon[93869]: pgmap v5429: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 92 KiB/s wr, 12 op/s 2023-03-09T17:31:48.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:31:47 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:31:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:31:47 smithi181 ceph-mon[94189]: pgmap v5429: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 92 KiB/s wr, 12 op/s 2023-03-09T17:31:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:31:47 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:31:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:31:47 smithi038 ceph-mon[92422]: pgmap v5429: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 92 KiB/s wr, 12 op/s 2023-03-09T17:31:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:31:47 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:31:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9680.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:31:49.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:31:48 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:31:49.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:31:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:31:49.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:31:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:31:49.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:31:48 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:31:49.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:31:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:31:49.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:31:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:31:49.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:31:48 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:31:49.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:31:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:31:49.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:31:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:31:50.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:31:49 smithi122 ceph-mon[93869]: pgmap v5430: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 87 KiB/s wr, 10 op/s 2023-03-09T17:31:50.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:31:49 smithi181 ceph-mon[94189]: pgmap v5430: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 87 KiB/s wr, 10 op/s 2023-03-09T17:31:50.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:31:49 smithi038 ceph-mon[92422]: pgmap v5430: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 87 KiB/s wr, 10 op/s 2023-03-09T17:31:51.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:31:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9221.061656 secs 2023-03-09T17:31:51.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:31:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9221.061656 secs 2023-03-09T17:31:51.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:31:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9221.061656 secs 2023-03-09T17:31:52.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:31:51 smithi122 ceph-mon[93869]: pgmap v5431: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 143 KiB/s wr, 10 op/s 2023-03-09T17:31:52.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:31:51 smithi181 ceph-mon[94189]: pgmap v5431: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 143 KiB/s wr, 10 op/s 2023-03-09T17:31:52.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:31:51 smithi038 ceph-mon[92422]: pgmap v5431: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 143 KiB/s wr, 10 op/s 2023-03-09T17:31:53.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:31:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:31:53.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:31:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:31:53.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:31:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:31:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9685.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:31:54.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:31:53 smithi122 ceph-mon[93869]: pgmap v5432: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 113 KiB/s wr, 12 op/s 2023-03-09T17:31:54.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:31:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:31:54.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:31:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:31:54.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:31:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:31:54.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:31:53 smithi181 ceph-mon[94189]: pgmap v5432: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 113 KiB/s wr, 12 op/s 2023-03-09T17:31:54.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:31:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:31:54.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:31:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:31:54.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:31:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:31:54.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:31:53 smithi038 ceph-mon[92422]: pgmap v5432: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 113 KiB/s wr, 12 op/s 2023-03-09T17:31:54.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:31:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:31:54.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:31:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:31:54.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:31:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:31:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:31:55 smithi122 ceph-mon[93869]: pgmap v5433: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 105 KiB/s wr, 10 op/s 2023-03-09T17:31:56.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:31:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9226.061738 secs 2023-03-09T17:31:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:31:55 smithi181 ceph-mon[94189]: pgmap v5433: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 105 KiB/s wr, 10 op/s 2023-03-09T17:31:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:31:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9226.061738 secs 2023-03-09T17:31:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:31:55 smithi038 ceph-mon[92422]: pgmap v5433: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 105 KiB/s wr, 10 op/s 2023-03-09T17:31:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:31:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9226.061738 secs 2023-03-09T17:31:58.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:31:57 smithi122 ceph-mon[93869]: pgmap v5434: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 155 KiB/s wr, 13 op/s 2023-03-09T17:31:58.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:31:57 smithi181 ceph-mon[94189]: pgmap v5434: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 155 KiB/s wr, 13 op/s 2023-03-09T17:31:58.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:31:57 smithi038 ceph-mon[92422]: pgmap v5434: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 155 KiB/s wr, 13 op/s 2023-03-09T17:31:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9690.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:32:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:31:59 smithi122 ceph-mon[93869]: pgmap v5435: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 115 KiB/s wr, 9 op/s 2023-03-09T17:32:00.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:31:59 smithi181 ceph-mon[94189]: pgmap v5435: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 115 KiB/s wr, 9 op/s 2023-03-09T17:32:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:31:59 smithi038 ceph-mon[92422]: pgmap v5435: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 115 KiB/s wr, 9 op/s 2023-03-09T17:32:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:32:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9231.061800 secs 2023-03-09T17:32:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:32:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9231.061800 secs 2023-03-09T17:32:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:32:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9231.061800 secs 2023-03-09T17:32:02.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:32:01 smithi122 ceph-mon[93869]: pgmap v5436: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 157 KiB/s wr, 9 op/s 2023-03-09T17:32:02.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:32:01 smithi181 ceph-mon[94189]: pgmap v5436: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 157 KiB/s wr, 9 op/s 2023-03-09T17:32:02.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:32:01 smithi038 ceph-mon[92422]: pgmap v5436: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 157 KiB/s wr, 9 op/s 2023-03-09T17:32:03.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:32:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9233.332332 secs 2023-03-09T17:32:03.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:32:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9233.332332 secs 2023-03-09T17:32:03.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:32:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9233.332332 secs 2023-03-09T17:32:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9695.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:32:04.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:32:03 smithi122 ceph-mon[93869]: pgmap v5437: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 132 KiB/s wr, 13 op/s 2023-03-09T17:32:04.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:32:03 smithi181 ceph-mon[94189]: pgmap v5437: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 132 KiB/s wr, 13 op/s 2023-03-09T17:32:04.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:32:03 smithi038 ceph-mon[92422]: pgmap v5437: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 132 KiB/s wr, 13 op/s 2023-03-09T17:32:05.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:32:04 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:32:05.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:32:04 smithi122 ceph-mon[93869]: osdmap e553: 12 total, 12 up, 12 in 2023-03-09T17:32:05.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:32:04 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:32:05.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:32:04 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:32:05.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:32:04 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:32:05.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:32:04 smithi181 ceph-mon[94189]: osdmap e553: 12 total, 12 up, 12 in 2023-03-09T17:32:05.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:32:04 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:32:05.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:32:04 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:32:05.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:32:04 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:32:05.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:32:04 smithi038 ceph-mon[92422]: osdmap e553: 12 total, 12 up, 12 in 2023-03-09T17:32:05.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:32:04 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:32:05.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:32:04 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:32:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:32:05 smithi122 ceph-mon[93869]: pgmap v5439: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 174 KiB/s wr, 27 op/s 2023-03-09T17:32:06.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:32:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9236.061862 secs 2023-03-09T17:32:06.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:32:05 smithi181 ceph-mon[94189]: pgmap v5439: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 174 KiB/s wr, 27 op/s 2023-03-09T17:32:06.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:32:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9236.061862 secs 2023-03-09T17:32:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:32:05 smithi038 ceph-mon[92422]: pgmap v5439: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 174 KiB/s wr, 27 op/s 2023-03-09T17:32:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:32:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9236.061862 secs 2023-03-09T17:32:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:32:07 smithi122 ceph-mon[93869]: pgmap v5440: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 162 KiB/s wr, 27 op/s 2023-03-09T17:32:08.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:32:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9238.332399 secs 2023-03-09T17:32:08.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:32:07 smithi181 ceph-mon[94189]: pgmap v5440: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 162 KiB/s wr, 27 op/s 2023-03-09T17:32:08.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:32:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9238.332399 secs 2023-03-09T17:32:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:32:07 smithi038 ceph-mon[92422]: pgmap v5440: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 162 KiB/s wr, 27 op/s 2023-03-09T17:32:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:32:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9238.332399 secs 2023-03-09T17:32:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9700.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:32:10.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:32:09 smithi122 ceph-mon[93869]: pgmap v5441: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 162 KiB/s wr, 27 op/s 2023-03-09T17:32:10.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:32:09 smithi181 ceph-mon[94189]: pgmap v5441: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 162 KiB/s wr, 27 op/s 2023-03-09T17:32:10.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:32:09 smithi038 ceph-mon[92422]: pgmap v5441: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 162 KiB/s wr, 27 op/s 2023-03-09T17:32:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:32:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9241.061923 secs 2023-03-09T17:32:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:32:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9241.061923 secs 2023-03-09T17:32:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:32:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9241.061923 secs 2023-03-09T17:32:12.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:32:11 smithi122 ceph-mon[93869]: pgmap v5442: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 133 KiB/s wr, 27 op/s 2023-03-09T17:32:12.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:32:11 smithi181 ceph-mon[94189]: pgmap v5442: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 133 KiB/s wr, 27 op/s 2023-03-09T17:32:12.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:32:11 smithi038 ceph-mon[92422]: pgmap v5442: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 133 KiB/s wr, 27 op/s 2023-03-09T17:32:13.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:32:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9243.332466 secs 2023-03-09T17:32:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:32:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9243.332466 secs 2023-03-09T17:32:13.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:32:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9243.332466 secs 2023-03-09T17:32:13.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9705.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:32:14.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:32:13 smithi122 ceph-mon[93869]: pgmap v5443: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 97 KiB/s wr, 24 op/s 2023-03-09T17:32:14.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:32:13 smithi181 ceph-mon[94189]: pgmap v5443: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 97 KiB/s wr, 24 op/s 2023-03-09T17:32:14.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:32:13 smithi038 ceph-mon[92422]: pgmap v5443: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 97 KiB/s wr, 24 op/s 2023-03-09T17:32:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:32:15 smithi122 ceph-mon[93869]: pgmap v5444: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 68 KiB/s wr, 9 op/s 2023-03-09T17:32:16.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:32:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9246.061985 secs 2023-03-09T17:32:16.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:32:15 smithi122 ceph-mon[93869]: osdmap e554: 12 total, 12 up, 12 in 2023-03-09T17:32:16.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:32:15 smithi181 ceph-mon[94189]: pgmap v5444: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 68 KiB/s wr, 9 op/s 2023-03-09T17:32:16.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:32:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9246.061985 secs 2023-03-09T17:32:16.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:32:15 smithi181 ceph-mon[94189]: osdmap e554: 12 total, 12 up, 12 in 2023-03-09T17:32:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:32:15 smithi038 ceph-mon[92422]: pgmap v5444: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 68 KiB/s wr, 9 op/s 2023-03-09T17:32:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:32:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9246.061985 secs 2023-03-09T17:32:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:32:15 smithi038 ceph-mon[92422]: osdmap e554: 12 total, 12 up, 12 in 2023-03-09T17:32:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:32:17 smithi122 ceph-mon[93869]: pgmap v5446: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T17:32:18.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:32:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9248.332583 secs 2023-03-09T17:32:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:32:17 smithi181 ceph-mon[94189]: pgmap v5446: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T17:32:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:32:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9248.332583 secs 2023-03-09T17:32:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:32:17 smithi038 ceph-mon[92422]: pgmap v5446: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T17:32:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:32:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9248.332583 secs 2023-03-09T17:32:18.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9710.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:32:20.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:32:19 smithi122 ceph-mon[93869]: pgmap v5447: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 6 op/s 2023-03-09T17:32:20.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:32:19 smithi181 ceph-mon[94189]: pgmap v5447: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 6 op/s 2023-03-09T17:32:20.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:32:19 smithi038 ceph-mon[92422]: pgmap v5447: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 6 op/s 2023-03-09T17:32:21.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:32:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9251.062049 secs 2023-03-09T17:32:21.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:32:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9251.062049 secs 2023-03-09T17:32:21.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:32:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9251.062049 secs 2023-03-09T17:32:22.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:32:21 smithi122 ceph-mon[93869]: pgmap v5448: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 6 op/s 2023-03-09T17:32:22.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:32:21 smithi181 ceph-mon[94189]: pgmap v5448: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 6 op/s 2023-03-09T17:32:22.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:32:21 smithi038 ceph-mon[92422]: pgmap v5448: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 6 op/s 2023-03-09T17:32:23.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:32:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9253.332698 secs 2023-03-09T17:32:23.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:32:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9253.332698 secs 2023-03-09T17:32:23.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:32:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9253.332698 secs 2023-03-09T17:32:23.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9715.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:32:24.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:32:23 smithi122 ceph-mon[93869]: pgmap v5449: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 17 KiB/s wr, 6 op/s 2023-03-09T17:32:24.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:32:23 smithi181 ceph-mon[94189]: pgmap v5449: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 17 KiB/s wr, 6 op/s 2023-03-09T17:32:24.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:32:23 smithi038 ceph-mon[92422]: pgmap v5449: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 17 KiB/s wr, 6 op/s 2023-03-09T17:32:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:32:25 smithi122 ceph-mon[93869]: pgmap v5450: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 17 KiB/s wr, 6 op/s 2023-03-09T17:32:26.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:32:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9256.062111 secs 2023-03-09T17:32:26.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:32:25 smithi181 ceph-mon[94189]: pgmap v5450: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 17 KiB/s wr, 6 op/s 2023-03-09T17:32:26.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:32:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9256.062111 secs 2023-03-09T17:32:26.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:32:25 smithi038 ceph-mon[92422]: pgmap v5450: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 17 KiB/s wr, 6 op/s 2023-03-09T17:32:26.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:32:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9256.062111 secs 2023-03-09T17:32:28.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:32:27 smithi122 ceph-mon[93869]: pgmap v5451: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 41 KiB/s wr, 6 op/s 2023-03-09T17:32:28.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:32:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9258.332798 secs 2023-03-09T17:32:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:32:27 smithi181 ceph-mon[94189]: pgmap v5451: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 41 KiB/s wr, 6 op/s 2023-03-09T17:32:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:32:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9258.332798 secs 2023-03-09T17:32:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:32:27 smithi038 ceph-mon[92422]: pgmap v5451: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 41 KiB/s wr, 6 op/s 2023-03-09T17:32:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:32:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9258.332798 secs 2023-03-09T17:32:28.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9720.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:32:30.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:32:29 smithi122 ceph-mon[93869]: pgmap v5452: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 37 KiB/s wr, 6 op/s 2023-03-09T17:32:30.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:32:29 smithi181 ceph-mon[94189]: pgmap v5452: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 37 KiB/s wr, 6 op/s 2023-03-09T17:32:30.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:32:29 smithi038 ceph-mon[92422]: pgmap v5452: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 37 KiB/s wr, 6 op/s 2023-03-09T17:32:31.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:32:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9261.062172 secs 2023-03-09T17:32:31.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:32:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9261.062172 secs 2023-03-09T17:32:31.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:32:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9261.062172 secs 2023-03-09T17:32:32.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:32:31 smithi122 ceph-mon[93869]: pgmap v5453: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 37 KiB/s wr, 7 op/s 2023-03-09T17:32:32.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:32:31 smithi181 ceph-mon[94189]: pgmap v5453: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 37 KiB/s wr, 7 op/s 2023-03-09T17:32:32.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:32:31 smithi038 ceph-mon[92422]: pgmap v5453: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 37 KiB/s wr, 7 op/s 2023-03-09T17:32:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9725.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:32:34.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:32:33 smithi122 ceph-mon[93869]: pgmap v5454: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 24 KiB/s wr, 8 op/s 2023-03-09T17:32:34.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:32:33 smithi181 ceph-mon[94189]: pgmap v5454: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 24 KiB/s wr, 8 op/s 2023-03-09T17:32:34.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:32:33 smithi038 ceph-mon[92422]: pgmap v5454: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 24 KiB/s wr, 8 op/s 2023-03-09T17:32:36.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:32:35 smithi122 ceph-mon[93869]: pgmap v5455: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 6 op/s 2023-03-09T17:32:36.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:32:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9266.062233 secs 2023-03-09T17:32:36.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:32:35 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:32:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:32:35 smithi181 ceph-mon[94189]: pgmap v5455: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 6 op/s 2023-03-09T17:32:36.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:32:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9266.062233 secs 2023-03-09T17:32:36.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:32:35 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:32:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:32:35 smithi038 ceph-mon[92422]: pgmap v5455: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 6 op/s 2023-03-09T17:32:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:32:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9266.062233 secs 2023-03-09T17:32:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:32:35 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:32:37.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:32:36 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:32:37.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:32:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:32:37.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:32:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:32:37.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:32:36 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:32:37.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:32:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:32:37.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:32:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:32:37.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:32:36 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:32:37.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:32:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:32:37.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:32:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:32:38.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:32:37 smithi181 ceph-mon[94189]: pgmap v5456: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 60 KiB/s wr, 9 op/s 2023-03-09T17:32:38.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:32:37 smithi038 ceph-mon[92422]: pgmap v5456: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 60 KiB/s wr, 9 op/s 2023-03-09T17:32:38.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:32:37 smithi122 ceph-mon[93869]: pgmap v5456: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 60 KiB/s wr, 9 op/s 2023-03-09T17:32:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9730.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:32:40.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:32:39 smithi038 ceph-mon[92422]: pgmap v5457: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 38 KiB/s wr, 6 op/s 2023-03-09T17:32:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:32:39 smithi122 ceph-mon[93869]: pgmap v5457: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 38 KiB/s wr, 6 op/s 2023-03-09T17:32:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:32:39 smithi181 ceph-mon[94189]: pgmap v5457: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 38 KiB/s wr, 6 op/s 2023-03-09T17:32:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:32:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9271.062300 secs 2023-03-09T17:32:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:32:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9271.062300 secs 2023-03-09T17:32:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:32:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9271.062300 secs 2023-03-09T17:32:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:32:41 smithi038 ceph-mon[92422]: pgmap v5458: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 38 KiB/s wr, 6 op/s 2023-03-09T17:32:42.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:32:41 smithi122 ceph-mon[93869]: pgmap v5458: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 38 KiB/s wr, 6 op/s 2023-03-09T17:32:42.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:32:41 smithi181 ceph-mon[94189]: pgmap v5458: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 38 KiB/s wr, 6 op/s 2023-03-09T17:32:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9735.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:32:44.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:32:43 smithi038 ceph-mon[92422]: pgmap v5459: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 46 KiB/s wr, 9 op/s 2023-03-09T17:32:44.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:32:43 smithi122 ceph-mon[93869]: pgmap v5459: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 46 KiB/s wr, 9 op/s 2023-03-09T17:32:44.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:32:43 smithi181 ceph-mon[94189]: pgmap v5459: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 46 KiB/s wr, 9 op/s 2023-03-09T17:32:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:32:46 smithi122 ceph-mon[93869]: pgmap v5460: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 68 KiB/s wr, 7 op/s 2023-03-09T17:32:46.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:32:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9276.062360 secs 2023-03-09T17:32:46.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:32:46 smithi181 ceph-mon[94189]: pgmap v5460: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 68 KiB/s wr, 7 op/s 2023-03-09T17:32:46.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:32:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9276.062360 secs 2023-03-09T17:32:46.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:32:46 smithi038 ceph-mon[92422]: pgmap v5460: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 68 KiB/s wr, 7 op/s 2023-03-09T17:32:46.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:32:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9276.062360 secs 2023-03-09T17:32:48.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:32:48 smithi122 ceph-mon[93869]: pgmap v5461: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 76 KiB/s wr, 10 op/s 2023-03-09T17:32:48.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:32:48 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 9278.333230 secs 2023-03-09T17:32:48.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:32:48 smithi181 ceph-mon[94189]: pgmap v5461: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 76 KiB/s wr, 10 op/s 2023-03-09T17:32:48.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:32:48 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 9278.333230 secs 2023-03-09T17:32:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:32:48 smithi038 ceph-mon[92422]: pgmap v5461: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 76 KiB/s wr, 10 op/s 2023-03-09T17:32:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:32:48 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 9278.333230 secs 2023-03-09T17:32:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9740.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:32:49.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:32:49 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:32:49.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:32:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:32:49.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:32:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:32:49.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:32:49 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:32:49.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:32:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:32:49.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:32:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:32:49.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:32:49 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:32:49.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:32:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:32:49.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:32:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:32:50.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:32:50 smithi122 ceph-mon[93869]: pgmap v5462: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 98 KiB/s wr, 10 op/s 2023-03-09T17:32:50.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:32:50 smithi181 ceph-mon[94189]: pgmap v5462: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 98 KiB/s wr, 10 op/s 2023-03-09T17:32:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:32:50 smithi038 ceph-mon[92422]: pgmap v5462: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 98 KiB/s wr, 10 op/s 2023-03-09T17:32:51.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:32:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9281.062421 secs 2023-03-09T17:32:51.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:32:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9281.062421 secs 2023-03-09T17:32:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:32:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9281.062421 secs 2023-03-09T17:32:52.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:32:52 smithi122 ceph-mon[93869]: pgmap v5463: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 98 KiB/s wr, 10 op/s 2023-03-09T17:32:52.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:32:52 smithi181 ceph-mon[94189]: pgmap v5463: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 98 KiB/s wr, 10 op/s 2023-03-09T17:32:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:32:52 smithi038 ceph-mon[92422]: pgmap v5463: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 98 KiB/s wr, 10 op/s 2023-03-09T17:32:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:32:53 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 9283.333297 secs 2023-03-09T17:32:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:32:53 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 9283.333297 secs 2023-03-09T17:32:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:32:53 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 9283.333297 secs 2023-03-09T17:32:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9745.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:32:54.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:32:54 smithi122 ceph-mon[93869]: pgmap v5464: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 106 KiB/s wr, 12 op/s 2023-03-09T17:32:54.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:32:54 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:32:54.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:32:54 smithi181 ceph-mon[94189]: pgmap v5464: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 106 KiB/s wr, 12 op/s 2023-03-09T17:32:54.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:32:54 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:32:54.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:32:54 smithi038 ceph-mon[92422]: pgmap v5464: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 106 KiB/s wr, 12 op/s 2023-03-09T17:32:54.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:32:54 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:32:55.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:32:55 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:32:55.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:32:55 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:32:55.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:32:55 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:32:55.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:32:55 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:32:55.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:32:55 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:32:55.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:32:55 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:32:55.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:32:55 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:32:55.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:32:55 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:32:55.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:32:55 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:32:55.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:32:55 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:32:55.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:32:55 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:32:55.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:32:55 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:32:55.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:32:55 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:32:55.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:32:55 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:32:55.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:32:55 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:32:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:32:56 smithi122 ceph-mon[93869]: pgmap v5465: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 122 KiB/s wr, 10 op/s 2023-03-09T17:32:56.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:32:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9286.062480 secs 2023-03-09T17:32:56.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:32:56 smithi181 ceph-mon[94189]: pgmap v5465: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 122 KiB/s wr, 10 op/s 2023-03-09T17:32:56.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:32:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9286.062480 secs 2023-03-09T17:32:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:32:56 smithi038 ceph-mon[92422]: pgmap v5465: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 122 KiB/s wr, 10 op/s 2023-03-09T17:32:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:32:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9286.062480 secs 2023-03-09T17:32:58.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:32:58 smithi122 ceph-mon[93869]: pgmap v5466: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 100 KiB/s wr, 12 op/s 2023-03-09T17:32:58.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:32:58 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 9288.333418 secs 2023-03-09T17:32:58.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:32:58 smithi181 ceph-mon[94189]: pgmap v5466: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 100 KiB/s wr, 12 op/s 2023-03-09T17:32:58.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:32:58 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 9288.333418 secs 2023-03-09T17:32:58.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:32:58 smithi038 ceph-mon[92422]: pgmap v5466: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 100 KiB/s wr, 12 op/s 2023-03-09T17:32:58.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:32:58 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 9288.333418 secs 2023-03-09T17:32:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9750.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:33:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:33:00 smithi122 ceph-mon[93869]: pgmap v5467: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 151 KiB/s wr, 11 op/s 2023-03-09T17:33:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:33:00 smithi181 ceph-mon[94189]: pgmap v5467: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 151 KiB/s wr, 11 op/s 2023-03-09T17:33:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:33:00 smithi038 ceph-mon[92422]: pgmap v5467: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 151 KiB/s wr, 11 op/s 2023-03-09T17:33:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:33:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9291.062541 secs 2023-03-09T17:33:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:33:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9291.062541 secs 2023-03-09T17:33:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:33:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9291.062541 secs 2023-03-09T17:33:02.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:33:02 smithi122 ceph-mon[93869]: pgmap v5468: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 92 KiB/s wr, 9 op/s 2023-03-09T17:33:02.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:33:02 smithi181 ceph-mon[94189]: pgmap v5468: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 92 KiB/s wr, 9 op/s 2023-03-09T17:33:02.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:33:02 smithi038 ceph-mon[92422]: pgmap v5468: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 92 KiB/s wr, 9 op/s 2023-03-09T17:33:03.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:33:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9293.333554 secs 2023-03-09T17:33:03.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:33:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9293.333554 secs 2023-03-09T17:33:03.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:33:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9293.333554 secs 2023-03-09T17:33:03.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9755.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:33:04.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:33:04 smithi122 ceph-mon[93869]: pgmap v5469: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 130 KiB/s wr, 13 op/s 2023-03-09T17:33:04.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:33:04 smithi181 ceph-mon[94189]: pgmap v5469: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 130 KiB/s wr, 13 op/s 2023-03-09T17:33:04.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:33:04 smithi038 ceph-mon[92422]: pgmap v5469: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 130 KiB/s wr, 13 op/s 2023-03-09T17:33:05.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:33:05 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:33:05.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:33:05 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:33:05.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:33:05 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:33:05.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:33:05 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:33:05.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:33:05 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:33:05.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:33:05 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:33:06.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:33:06 smithi122 ceph-mon[93869]: pgmap v5470: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 173 KiB/s wr, 24 op/s 2023-03-09T17:33:06.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:33:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9296.062601 secs 2023-03-09T17:33:06.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:33:06 smithi181 ceph-mon[94189]: pgmap v5470: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 173 KiB/s wr, 24 op/s 2023-03-09T17:33:06.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:33:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9296.062601 secs 2023-03-09T17:33:06.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:33:06 smithi038 ceph-mon[92422]: pgmap v5470: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 173 KiB/s wr, 24 op/s 2023-03-09T17:33:06.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:33:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9296.062601 secs 2023-03-09T17:33:08.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:33:08 smithi122 ceph-mon[93869]: pgmap v5471: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 157 KiB/s wr, 25 op/s 2023-03-09T17:33:08.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:33:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9298.333692 secs 2023-03-09T17:33:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:33:08 smithi181 ceph-mon[94189]: pgmap v5471: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 157 KiB/s wr, 25 op/s 2023-03-09T17:33:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:33:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9298.333692 secs 2023-03-09T17:33:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:33:08 smithi038 ceph-mon[92422]: pgmap v5471: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 157 KiB/s wr, 25 op/s 2023-03-09T17:33:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:33:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9298.333692 secs 2023-03-09T17:33:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9760.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:33:10.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:33:10 smithi122 ceph-mon[93869]: pgmap v5472: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 203 KiB/s wr, 24 op/s 2023-03-09T17:33:10.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:33:10 smithi122 ceph-mon[93869]: osdmap e555: 12 total, 12 up, 12 in 2023-03-09T17:33:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:33:10 smithi181 ceph-mon[94189]: pgmap v5472: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 203 KiB/s wr, 24 op/s 2023-03-09T17:33:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:33:10 smithi181 ceph-mon[94189]: osdmap e555: 12 total, 12 up, 12 in 2023-03-09T17:33:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:33:10 smithi038 ceph-mon[92422]: pgmap v5472: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 203 KiB/s wr, 24 op/s 2023-03-09T17:33:10.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:33:10 smithi038 ceph-mon[92422]: osdmap e555: 12 total, 12 up, 12 in 2023-03-09T17:33:11.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:33:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9301.062662 secs 2023-03-09T17:33:11.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:33:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9301.062662 secs 2023-03-09T17:33:11.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:33:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9301.062662 secs 2023-03-09T17:33:12.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:33:12 smithi122 ceph-mon[93869]: pgmap v5474: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 173 KiB/s wr, 27 op/s 2023-03-09T17:33:12.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:33:12 smithi181 ceph-mon[94189]: pgmap v5474: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 173 KiB/s wr, 27 op/s 2023-03-09T17:33:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:33:12 smithi038 ceph-mon[92422]: pgmap v5474: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 173 KiB/s wr, 27 op/s 2023-03-09T17:33:13.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:33:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9303.333791 secs 2023-03-09T17:33:13.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:33:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9303.333791 secs 2023-03-09T17:33:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:33:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9303.333791 secs 2023-03-09T17:33:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9765.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:33:14.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:33:14 smithi122 ceph-mon[93869]: pgmap v5475: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 128 KiB/s wr, 24 op/s 2023-03-09T17:33:14.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:33:14 smithi181 ceph-mon[94189]: pgmap v5475: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 128 KiB/s wr, 24 op/s 2023-03-09T17:33:14.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:33:14 smithi038 ceph-mon[92422]: pgmap v5475: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 128 KiB/s wr, 24 op/s 2023-03-09T17:33:16.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:33:16 smithi122 ceph-mon[93869]: pgmap v5476: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 86 KiB/s wr, 8 op/s 2023-03-09T17:33:16.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:33:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9306.062758 secs 2023-03-09T17:33:16.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:33:16 smithi122 ceph-mon[93869]: osdmap e556: 12 total, 12 up, 12 in 2023-03-09T17:33:16.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:33:16 smithi181 ceph-mon[94189]: pgmap v5476: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 86 KiB/s wr, 8 op/s 2023-03-09T17:33:16.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:33:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9306.062758 secs 2023-03-09T17:33:16.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:33:16 smithi181 ceph-mon[94189]: osdmap e556: 12 total, 12 up, 12 in 2023-03-09T17:33:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:33:16 smithi038 ceph-mon[92422]: pgmap v5476: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 86 KiB/s wr, 8 op/s 2023-03-09T17:33:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:33:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9306.062758 secs 2023-03-09T17:33:16.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:33:16 smithi038 ceph-mon[92422]: osdmap e556: 12 total, 12 up, 12 in 2023-03-09T17:33:18.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:33:18 smithi122 ceph-mon[93869]: pgmap v5478: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 38 KiB/s wr, 8 op/s 2023-03-09T17:33:18.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:33:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9308.333887 secs 2023-03-09T17:33:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:33:18 smithi181 ceph-mon[94189]: pgmap v5478: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 38 KiB/s wr, 8 op/s 2023-03-09T17:33:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:33:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9308.333887 secs 2023-03-09T17:33:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:33:18 smithi038 ceph-mon[92422]: pgmap v5478: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 38 KiB/s wr, 8 op/s 2023-03-09T17:33:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:33:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9308.333887 secs 2023-03-09T17:33:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9770.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:33:20.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:33:20 smithi122 ceph-mon[93869]: pgmap v5479: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 52 KiB/s wr, 8 op/s 2023-03-09T17:33:20.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:33:20 smithi181 ceph-mon[94189]: pgmap v5479: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 52 KiB/s wr, 8 op/s 2023-03-09T17:33:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:33:20 smithi038 ceph-mon[92422]: pgmap v5479: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 52 KiB/s wr, 8 op/s 2023-03-09T17:33:21.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:33:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9311.062816 secs 2023-03-09T17:33:21.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:33:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9311.062816 secs 2023-03-09T17:33:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:33:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9311.062816 secs 2023-03-09T17:33:22.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:33:22 smithi122 ceph-mon[93869]: pgmap v5480: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 50 KiB/s wr, 7 op/s 2023-03-09T17:33:22.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:33:22 smithi181 ceph-mon[94189]: pgmap v5480: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 50 KiB/s wr, 7 op/s 2023-03-09T17:33:22.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:33:22 smithi038 ceph-mon[92422]: pgmap v5480: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 50 KiB/s wr, 7 op/s 2023-03-09T17:33:23.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:33:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9313.334004 secs 2023-03-09T17:33:23.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:33:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9313.334004 secs 2023-03-09T17:33:23.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:33:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9313.334004 secs 2023-03-09T17:33:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9775.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:33:24.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:33:24 smithi122 ceph-mon[93869]: pgmap v5481: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 50 KiB/s wr, 7 op/s 2023-03-09T17:33:24.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:33:24 smithi181 ceph-mon[94189]: pgmap v5481: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 50 KiB/s wr, 7 op/s 2023-03-09T17:33:24.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:33:24 smithi038 ceph-mon[92422]: pgmap v5481: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 50 KiB/s wr, 7 op/s 2023-03-09T17:33:25.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:33:25 smithi122 ceph-mon[93869]: pgmap v5482: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 57 KiB/s wr, 7 op/s 2023-03-09T17:33:25.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:33:25 smithi181 ceph-mon[94189]: pgmap v5482: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 57 KiB/s wr, 7 op/s 2023-03-09T17:33:25.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:33:25 smithi038 ceph-mon[92422]: pgmap v5482: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 57 KiB/s wr, 7 op/s 2023-03-09T17:33:26.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:33:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9316.062878 secs 2023-03-09T17:33:26.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:33:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9316.062878 secs 2023-03-09T17:33:26.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:33:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9316.062878 secs 2023-03-09T17:33:27.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:33:27 smithi122 ceph-mon[93869]: pgmap v5483: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T17:33:27.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:33:27 smithi181 ceph-mon[94189]: pgmap v5483: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T17:33:27.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:33:27 smithi038 ceph-mon[92422]: pgmap v5483: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T17:33:28.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:33:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9318.334136 secs 2023-03-09T17:33:28.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:33:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9318.334136 secs 2023-03-09T17:33:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:33:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9318.334136 secs 2023-03-09T17:33:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9780.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:33:29.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:33:29 smithi038 ceph-mon[92422]: pgmap v5484: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 40 KiB/s wr, 7 op/s 2023-03-09T17:33:29.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:33:29 smithi122 ceph-mon[93869]: pgmap v5484: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 40 KiB/s wr, 7 op/s 2023-03-09T17:33:29.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:33:29 smithi181 ceph-mon[94189]: pgmap v5484: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 40 KiB/s wr, 7 op/s 2023-03-09T17:33:30.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:33:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9321.062964 secs 2023-03-09T17:33:30.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:33:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9321.062964 secs 2023-03-09T17:33:30.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:33:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9321.062964 secs 2023-03-09T17:33:31.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:33:31 smithi122 ceph-mon[93869]: pgmap v5485: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 25 KiB/s wr, 6 op/s 2023-03-09T17:33:31.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:33:31 smithi181 ceph-mon[94189]: pgmap v5485: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 25 KiB/s wr, 6 op/s 2023-03-09T17:33:32.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:33:31 smithi038 ceph-mon[92422]: pgmap v5485: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 25 KiB/s wr, 6 op/s 2023-03-09T17:33:32.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:33:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9323.334266 secs 2023-03-09T17:33:33.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:33:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9323.334266 secs 2023-03-09T17:33:33.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:33:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9323.334266 secs 2023-03-09T17:33:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9785.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:33:34.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:33:33 smithi038 ceph-mon[92422]: pgmap v5486: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 25 KiB/s wr, 9 op/s 2023-03-09T17:33:34.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:33:33 smithi122 ceph-mon[93869]: pgmap v5486: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 25 KiB/s wr, 9 op/s 2023-03-09T17:33:34.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:33:33 smithi181 ceph-mon[94189]: pgmap v5486: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 25 KiB/s wr, 9 op/s 2023-03-09T17:33:36.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:33:35 smithi038 ceph-mon[92422]: pgmap v5487: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T17:33:36.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:33:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9326.062988 secs 2023-03-09T17:33:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:33:35 smithi122 ceph-mon[93869]: pgmap v5487: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T17:33:36.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:33:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9326.062988 secs 2023-03-09T17:33:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:33:35 smithi181 ceph-mon[94189]: pgmap v5487: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T17:33:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:33:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9326.062988 secs 2023-03-09T17:33:38.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:33:37 smithi038 ceph-mon[92422]: pgmap v5488: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 9 op/s 2023-03-09T17:33:38.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:33:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9328.334339 secs 2023-03-09T17:33:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:33:37 smithi122 ceph-mon[93869]: pgmap v5488: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 9 op/s 2023-03-09T17:33:38.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:33:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9328.334339 secs 2023-03-09T17:33:38.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:33:37 smithi181 ceph-mon[94189]: pgmap v5488: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 9 op/s 2023-03-09T17:33:38.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:33:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9328.334339 secs 2023-03-09T17:33:38.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9790.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:33:40.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:33:39 smithi038 ceph-mon[92422]: pgmap v5489: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 60 KiB/s wr, 16 op/s 2023-03-09T17:33:40.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:33:39 smithi122 ceph-mon[93869]: pgmap v5489: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 60 KiB/s wr, 16 op/s 2023-03-09T17:33:40.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:33:39 smithi181 ceph-mon[94189]: pgmap v5489: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 60 KiB/s wr, 16 op/s 2023-03-09T17:33:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:33:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9331.063047 secs 2023-03-09T17:33:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:33:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9331.063047 secs 2023-03-09T17:33:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:33:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9331.063047 secs 2023-03-09T17:33:42.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:33:41 smithi122 ceph-mon[93869]: pgmap v5490: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 60 KiB/s wr, 15 op/s 2023-03-09T17:33:42.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:33:41 smithi181 ceph-mon[94189]: pgmap v5490: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 60 KiB/s wr, 15 op/s 2023-03-09T17:33:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:33:41 smithi038 ceph-mon[92422]: pgmap v5490: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 60 KiB/s wr, 15 op/s 2023-03-09T17:33:43.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:33:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9333.334460 secs 2023-03-09T17:33:43.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:33:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9333.334460 secs 2023-03-09T17:33:43.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:33:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9333.334460 secs 2023-03-09T17:33:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9795.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:33:44.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:33:43 smithi122 ceph-mon[93869]: pgmap v5491: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 118 KiB/s wr, 20 op/s 2023-03-09T17:33:44.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:33:43 smithi181 ceph-mon[94189]: pgmap v5491: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 118 KiB/s wr, 20 op/s 2023-03-09T17:33:44.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:33:43 smithi038 ceph-mon[92422]: pgmap v5491: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 118 KiB/s wr, 20 op/s 2023-03-09T17:33:46.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:33:45 smithi122 ceph-mon[93869]: pgmap v5492: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 117 KiB/s wr, 31 op/s 2023-03-09T17:33:46.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:33:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9336.063107 secs 2023-03-09T17:33:46.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:33:45 smithi181 ceph-mon[94189]: pgmap v5492: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 117 KiB/s wr, 31 op/s 2023-03-09T17:33:46.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:33:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9336.063107 secs 2023-03-09T17:33:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:33:45 smithi038 ceph-mon[92422]: pgmap v5492: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 117 KiB/s wr, 31 op/s 2023-03-09T17:33:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:33:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9336.063107 secs 2023-03-09T17:33:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:33:47 smithi122 ceph-mon[93869]: pgmap v5493: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 173 KiB/s wr, 34 op/s 2023-03-09T17:33:48.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:33:47 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 9338.334585 secs 2023-03-09T17:33:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:33:47 smithi181 ceph-mon[94189]: pgmap v5493: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 173 KiB/s wr, 34 op/s 2023-03-09T17:33:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:33:47 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 9338.334585 secs 2023-03-09T17:33:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:33:47 smithi038 ceph-mon[92422]: pgmap v5493: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 173 KiB/s wr, 34 op/s 2023-03-09T17:33:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:33:47 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 9338.334585 secs 2023-03-09T17:33:48.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9800.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:33:49.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:33:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:33:49.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:33:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:33:49.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:33:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:33:49.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:33:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:33:49.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:33:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:33:49.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:33:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:33:50.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:33:49 smithi122 ceph-mon[93869]: pgmap v5494: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 162 KiB/s wr, 33 op/s 2023-03-09T17:33:50.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:33:49 smithi181 ceph-mon[94189]: pgmap v5494: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 162 KiB/s wr, 33 op/s 2023-03-09T17:33:50.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:33:49 smithi038 ceph-mon[92422]: pgmap v5494: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 162 KiB/s wr, 33 op/s 2023-03-09T17:33:51.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:33:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9341.063169 secs 2023-03-09T17:33:51.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:33:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9341.063169 secs 2023-03-09T17:33:51.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:33:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9341.063169 secs 2023-03-09T17:33:52.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:33:51 smithi122 ceph-mon[93869]: pgmap v5495: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 114 KiB/s wr, 24 op/s 2023-03-09T17:33:52.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:33:51 smithi181 ceph-mon[94189]: pgmap v5495: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 114 KiB/s wr, 24 op/s 2023-03-09T17:33:52.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:33:51 smithi038 ceph-mon[92422]: pgmap v5495: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 114 KiB/s wr, 24 op/s 2023-03-09T17:33:53.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:33:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9343.334718 secs 2023-03-09T17:33:53.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:33:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9343.334718 secs 2023-03-09T17:33:53.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:33:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9343.334718 secs 2023-03-09T17:33:53.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9805.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:33:54.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:33:53 smithi122 ceph-mon[93869]: pgmap v5496: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 189 KiB/s wr, 27 op/s 2023-03-09T17:33:54.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:33:53 smithi181 ceph-mon[94189]: pgmap v5496: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 189 KiB/s wr, 27 op/s 2023-03-09T17:33:54.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:33:53 smithi038 ceph-mon[92422]: pgmap v5496: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 189 KiB/s wr, 27 op/s 2023-03-09T17:33:56.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:33:55 smithi122 ceph-mon[93869]: pgmap v5497: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 131 KiB/s wr, 24 op/s 2023-03-09T17:33:56.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:33:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9346.063229 secs 2023-03-09T17:33:56.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:33:55 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:33:56.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:33:55 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:33:56.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:33:55 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:33:56.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:33:55 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:33:56.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:33:55 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:33:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:33:55 smithi181 ceph-mon[94189]: pgmap v5497: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 131 KiB/s wr, 24 op/s 2023-03-09T17:33:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:33:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9346.063229 secs 2023-03-09T17:33:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:33:55 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:33:56.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:33:55 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:33:56.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:33:55 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:33:56.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:33:55 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:33:56.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:33:55 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:33:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:33:55 smithi038 ceph-mon[92422]: pgmap v5497: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 131 KiB/s wr, 24 op/s 2023-03-09T17:33:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:33:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9346.063229 secs 2023-03-09T17:33:56.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:33:55 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:33:56.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:33:55 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:33:56.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:33:55 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:33:56.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:33:55 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:33:56.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:33:55 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:33:57.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:33:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:33:57.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:33:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:33:57.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:33:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:33:57.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:33:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:33:57.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:33:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:33:57.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:33:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:33:57.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:33:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:33:57.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:33:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:33:57.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:33:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:33:57.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:33:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:33:57.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:33:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:33:57.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:33:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:33:57.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:33:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:33:57.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:33:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:33:57.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:33:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:33:58.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:33:58 smithi122 ceph-mon[93869]: pgmap v5498: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 145 KiB/s wr, 12 op/s 2023-03-09T17:33:58.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:33:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9348.334854 secs 2023-03-09T17:33:58.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:33:58 smithi181 ceph-mon[94189]: pgmap v5498: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 145 KiB/s wr, 12 op/s 2023-03-09T17:33:58.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:33:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9348.334854 secs 2023-03-09T17:33:58.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:33:58 smithi038 ceph-mon[92422]: pgmap v5498: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 145 KiB/s wr, 12 op/s 2023-03-09T17:33:58.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:33:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9348.334854 secs 2023-03-09T17:33:58.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9810.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:34:00.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:34:00 smithi122 ceph-mon[93869]: pgmap v5499: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 90 KiB/s wr, 9 op/s 2023-03-09T17:34:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:34:00 smithi181 ceph-mon[94189]: pgmap v5499: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 90 KiB/s wr, 9 op/s 2023-03-09T17:34:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:34:00 smithi038 ceph-mon[92422]: pgmap v5499: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 90 KiB/s wr, 9 op/s 2023-03-09T17:34:01.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:34:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9351.063288 secs 2023-03-09T17:34:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:34:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9351.063288 secs 2023-03-09T17:34:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:34:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9351.063288 secs 2023-03-09T17:34:02.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:34:02 smithi122 ceph-mon[93869]: pgmap v5500: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 90 KiB/s wr, 8 op/s 2023-03-09T17:34:02.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:34:02 smithi181 ceph-mon[94189]: pgmap v5500: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 90 KiB/s wr, 8 op/s 2023-03-09T17:34:02.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:34:02 smithi038 ceph-mon[92422]: pgmap v5500: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 90 KiB/s wr, 8 op/s 2023-03-09T17:34:03.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:34:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9353.334936 secs 2023-03-09T17:34:03.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:34:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9353.334936 secs 2023-03-09T17:34:03.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:34:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9353.334936 secs 2023-03-09T17:34:03.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9815.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:34:04.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:34:04 smithi122 ceph-mon[93869]: pgmap v5501: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 175 KiB/s wr, 15 op/s 2023-03-09T17:34:04.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:34:04 smithi181 ceph-mon[94189]: pgmap v5501: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 175 KiB/s wr, 15 op/s 2023-03-09T17:34:04.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:34:04 smithi038 ceph-mon[92422]: pgmap v5501: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 175 KiB/s wr, 15 op/s 2023-03-09T17:34:06.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:34:06 smithi122 ceph-mon[93869]: pgmap v5502: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 121 KiB/s wr, 23 op/s 2023-03-09T17:34:06.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:34:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9356.063348 secs 2023-03-09T17:34:06.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:34:06 smithi181 ceph-mon[94189]: pgmap v5502: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 121 KiB/s wr, 23 op/s 2023-03-09T17:34:06.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:34:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9356.063348 secs 2023-03-09T17:34:06.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:34:06 smithi038 ceph-mon[92422]: pgmap v5502: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 121 KiB/s wr, 23 op/s 2023-03-09T17:34:06.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:34:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9356.063348 secs 2023-03-09T17:34:08.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:34:08 smithi122 ceph-mon[93869]: pgmap v5503: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 142 KiB/s wr, 26 op/s 2023-03-09T17:34:08.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:34:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9358.335059 secs 2023-03-09T17:34:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:34:08 smithi181 ceph-mon[94189]: pgmap v5503: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 142 KiB/s wr, 26 op/s 2023-03-09T17:34:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:34:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9358.335059 secs 2023-03-09T17:34:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:34:08 smithi038 ceph-mon[92422]: pgmap v5503: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 142 KiB/s wr, 26 op/s 2023-03-09T17:34:08.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:34:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9358.335059 secs 2023-03-09T17:34:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9820.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:34:10.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:34:10 smithi122 ceph-mon[93869]: pgmap v5504: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 128 KiB/s wr, 22 op/s 2023-03-09T17:34:10.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:34:10 smithi122 ceph-mon[93869]: osdmap e557: 12 total, 12 up, 12 in 2023-03-09T17:34:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:34:10 smithi181 ceph-mon[94189]: pgmap v5504: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 128 KiB/s wr, 22 op/s 2023-03-09T17:34:10.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:34:10 smithi181 ceph-mon[94189]: osdmap e557: 12 total, 12 up, 12 in 2023-03-09T17:34:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:34:10 smithi038 ceph-mon[92422]: pgmap v5504: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 128 KiB/s wr, 22 op/s 2023-03-09T17:34:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:34:10 smithi038 ceph-mon[92422]: osdmap e557: 12 total, 12 up, 12 in 2023-03-09T17:34:11.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:34:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9361.063411 secs 2023-03-09T17:34:11.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:34:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9361.063411 secs 2023-03-09T17:34:11.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:34:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9361.063411 secs 2023-03-09T17:34:12.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:34:12 smithi122 ceph-mon[93869]: pgmap v5506: 129 pgs: 1 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 124 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 153 KiB/s wr, 26 op/s 2023-03-09T17:34:12.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:34:12 smithi181 ceph-mon[94189]: pgmap v5506: 129 pgs: 1 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 124 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 153 KiB/s wr, 26 op/s 2023-03-09T17:34:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:34:12 smithi038 ceph-mon[92422]: pgmap v5506: 129 pgs: 1 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 124 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 153 KiB/s wr, 26 op/s 2023-03-09T17:34:13.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:34:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9363.335141 secs 2023-03-09T17:34:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:34:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9363.335141 secs 2023-03-09T17:34:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:34:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9363.335141 secs 2023-03-09T17:34:13.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9825.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:34:14.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:34:14 smithi122 ceph-mon[93869]: pgmap v5507: 129 pgs: 1 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 124 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 51 KiB/s wr, 22 op/s 2023-03-09T17:34:14.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:34:14 smithi181 ceph-mon[94189]: pgmap v5507: 129 pgs: 1 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 124 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 51 KiB/s wr, 22 op/s 2023-03-09T17:34:14.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:34:14 smithi038 ceph-mon[92422]: pgmap v5507: 129 pgs: 1 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 124 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 51 KiB/s wr, 22 op/s 2023-03-09T17:34:16.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:34:16 smithi122 ceph-mon[93869]: pgmap v5508: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 25 KiB/s wr, 7 op/s 2023-03-09T17:34:16.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:34:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9366.063471 secs 2023-03-09T17:34:16.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:34:16 smithi122 ceph-mon[93869]: osdmap e558: 12 total, 12 up, 12 in 2023-03-09T17:34:16.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:34:16 smithi181 ceph-mon[94189]: pgmap v5508: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 25 KiB/s wr, 7 op/s 2023-03-09T17:34:16.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:34:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9366.063471 secs 2023-03-09T17:34:16.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:34:16 smithi181 ceph-mon[94189]: osdmap e558: 12 total, 12 up, 12 in 2023-03-09T17:34:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:34:16 smithi038 ceph-mon[92422]: pgmap v5508: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 25 KiB/s wr, 7 op/s 2023-03-09T17:34:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:34:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9366.063471 secs 2023-03-09T17:34:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:34:16 smithi038 ceph-mon[92422]: osdmap e558: 12 total, 12 up, 12 in 2023-03-09T17:34:18.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:34:18 smithi122 ceph-mon[93869]: pgmap v5510: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2023-03-09T17:34:18.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:34:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9368.335244 secs 2023-03-09T17:34:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:34:18 smithi181 ceph-mon[94189]: pgmap v5510: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2023-03-09T17:34:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:34:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9368.335244 secs 2023-03-09T17:34:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:34:18 smithi038 ceph-mon[92422]: pgmap v5510: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 255 B/s wr, 7 op/s 2023-03-09T17:34:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:34:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9368.335244 secs 2023-03-09T17:34:18.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9830.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:34:20.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:34:20 smithi122 ceph-mon[93869]: pgmap v5511: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 219 B/s wr, 7 op/s 2023-03-09T17:34:20.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:34:20 smithi181 ceph-mon[94189]: pgmap v5511: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 219 B/s wr, 7 op/s 2023-03-09T17:34:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:34:20 smithi038 ceph-mon[92422]: pgmap v5511: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 219 B/s wr, 7 op/s 2023-03-09T17:34:21.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:34:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9371.063530 secs 2023-03-09T17:34:21.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:34:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9371.063530 secs 2023-03-09T17:34:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:34:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9371.063530 secs 2023-03-09T17:34:22.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:34:22 smithi122 ceph-mon[93869]: pgmap v5512: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T17:34:22.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:34:22 smithi181 ceph-mon[94189]: pgmap v5512: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T17:34:22.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:34:22 smithi038 ceph-mon[92422]: pgmap v5512: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T17:34:23.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:34:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9373.335342 secs 2023-03-09T17:34:23.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:34:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9373.335342 secs 2023-03-09T17:34:23.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:34:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9373.335342 secs 2023-03-09T17:34:23.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9835.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:34:24.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:34:24 smithi122 ceph-mon[93869]: pgmap v5513: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T17:34:24.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:34:24 smithi181 ceph-mon[94189]: pgmap v5513: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T17:34:24.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:34:24 smithi038 ceph-mon[92422]: pgmap v5513: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T17:34:26.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:34:26 smithi122 ceph-mon[93869]: pgmap v5514: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T17:34:26.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:34:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9376.063589 secs 2023-03-09T17:34:26.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:34:26 smithi181 ceph-mon[94189]: pgmap v5514: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T17:34:26.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:34:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9376.063589 secs 2023-03-09T17:34:26.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:34:26 smithi038 ceph-mon[92422]: pgmap v5514: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T17:34:26.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:34:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9376.063589 secs 2023-03-09T17:34:28.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:34:28 smithi122 ceph-mon[93869]: pgmap v5515: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 95 B/s wr, 6 op/s 2023-03-09T17:34:28.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:34:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9378.335457 secs 2023-03-09T17:34:28.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:34:28 smithi181 ceph-mon[94189]: pgmap v5515: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 95 B/s wr, 6 op/s 2023-03-09T17:34:28.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:34:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9378.335457 secs 2023-03-09T17:34:28.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:34:28 smithi038 ceph-mon[92422]: pgmap v5515: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 95 B/s wr, 6 op/s 2023-03-09T17:34:28.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:34:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9378.335457 secs 2023-03-09T17:34:28.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9840.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:34:30.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:34:30 smithi122 ceph-mon[93869]: pgmap v5516: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 170 B/s wr, 5 op/s 2023-03-09T17:34:30.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:34:30 smithi181 ceph-mon[94189]: pgmap v5516: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 170 B/s wr, 5 op/s 2023-03-09T17:34:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:34:30 smithi038 ceph-mon[92422]: pgmap v5516: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 170 B/s wr, 5 op/s 2023-03-09T17:34:31.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:34:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9381.063649 secs 2023-03-09T17:34:31.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:34:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9381.063649 secs 2023-03-09T17:34:31.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:34:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9381.063649 secs 2023-03-09T17:34:32.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:34:32 smithi122 ceph-mon[93869]: pgmap v5517: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 170 B/s wr, 5 op/s 2023-03-09T17:34:32.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:34:32 smithi181 ceph-mon[94189]: pgmap v5517: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 170 B/s wr, 5 op/s 2023-03-09T17:34:32.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:34:32 smithi038 ceph-mon[92422]: pgmap v5517: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 170 B/s wr, 5 op/s 2023-03-09T17:34:33.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:34:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9383.335575 secs 2023-03-09T17:34:33.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:34:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9383.335575 secs 2023-03-09T17:34:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:34:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9383.335575 secs 2023-03-09T17:34:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9845.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:34:34.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:34:34 smithi122 ceph-mon[93869]: pgmap v5518: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 8 op/s 2023-03-09T17:34:34.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:34:34 smithi181 ceph-mon[94189]: pgmap v5518: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 8 op/s 2023-03-09T17:34:34.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:34:34 smithi038 ceph-mon[92422]: pgmap v5518: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 255 B/s wr, 8 op/s 2023-03-09T17:34:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:34:36 smithi122 ceph-mon[93869]: pgmap v5519: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2023-03-09T17:34:36.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:34:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9386.063707 secs 2023-03-09T17:34:36.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:34:36 smithi181 ceph-mon[94189]: pgmap v5519: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2023-03-09T17:34:36.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:34:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9386.063707 secs 2023-03-09T17:34:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:34:36 smithi038 ceph-mon[92422]: pgmap v5519: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2023-03-09T17:34:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:34:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9386.063707 secs 2023-03-09T17:34:37.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:34:37 smithi122 ceph-mon[93869]: pgmap v5520: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 22 KiB/s wr, 8 op/s 2023-03-09T17:34:37.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:34:37 smithi181 ceph-mon[94189]: pgmap v5520: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 22 KiB/s wr, 8 op/s 2023-03-09T17:34:37.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:34:37 smithi038 ceph-mon[92422]: pgmap v5520: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 22 KiB/s wr, 8 op/s 2023-03-09T17:34:38.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:34:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9388.335697 secs 2023-03-09T17:34:38.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:34:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9388.335697 secs 2023-03-09T17:34:38.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:34:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9388.335697 secs 2023-03-09T17:34:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9850.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:34:39.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:34:39 smithi122 ceph-mon[93869]: pgmap v5521: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 5 op/s 2023-03-09T17:34:39.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:34:39 smithi181 ceph-mon[94189]: pgmap v5521: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 5 op/s 2023-03-09T17:34:39.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:34:39 smithi038 ceph-mon[92422]: pgmap v5521: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 5 op/s 2023-03-09T17:34:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:34:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9391.063766 secs 2023-03-09T17:34:40.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:34:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9391.063766 secs 2023-03-09T17:34:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:34:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9391.063766 secs 2023-03-09T17:34:42.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:34:41 smithi122 ceph-mon[93869]: pgmap v5522: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 5 op/s 2023-03-09T17:34:42.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:34:41 smithi181 ceph-mon[94189]: pgmap v5522: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 5 op/s 2023-03-09T17:34:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:34:41 smithi038 ceph-mon[92422]: pgmap v5522: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 5 op/s 2023-03-09T17:34:43.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:34:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9393.335821 secs 2023-03-09T17:34:43.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:34:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9393.335821 secs 2023-03-09T17:34:43.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:34:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9393.335821 secs 2023-03-09T17:34:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9855.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:34:44.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:34:43 smithi122 ceph-mon[93869]: pgmap v5523: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 70 KiB/s wr, 8 op/s 2023-03-09T17:34:44.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:34:43 smithi181 ceph-mon[94189]: pgmap v5523: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 70 KiB/s wr, 8 op/s 2023-03-09T17:34:44.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:34:43 smithi038 ceph-mon[92422]: pgmap v5523: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 70 KiB/s wr, 8 op/s 2023-03-09T17:34:46.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:34:45 smithi122 ceph-mon[93869]: pgmap v5524: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 70 KiB/s wr, 6 op/s 2023-03-09T17:34:46.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:34:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9396.063839 secs 2023-03-09T17:34:46.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:34:45 smithi181 ceph-mon[94189]: pgmap v5524: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 70 KiB/s wr, 6 op/s 2023-03-09T17:34:46.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:34:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9396.063839 secs 2023-03-09T17:34:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:34:45 smithi038 ceph-mon[92422]: pgmap v5524: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 70 KiB/s wr, 6 op/s 2023-03-09T17:34:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:34:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9396.063839 secs 2023-03-09T17:34:48.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:34:47 smithi122 ceph-mon[93869]: pgmap v5525: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 85 KiB/s wr, 10 op/s 2023-03-09T17:34:48.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:34:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9398.335974 secs 2023-03-09T17:34:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:34:47 smithi181 ceph-mon[94189]: pgmap v5525: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 85 KiB/s wr, 10 op/s 2023-03-09T17:34:48.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:34:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9398.335974 secs 2023-03-09T17:34:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:34:47 smithi038 ceph-mon[92422]: pgmap v5525: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 85 KiB/s wr, 10 op/s 2023-03-09T17:34:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:34:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9398.335974 secs 2023-03-09T17:34:48.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9860.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:34:50.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:34:49 smithi122 ceph-mon[93869]: pgmap v5526: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 88 KiB/s wr, 8 op/s 2023-03-09T17:34:50.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:34:49 smithi181 ceph-mon[94189]: pgmap v5526: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 88 KiB/s wr, 8 op/s 2023-03-09T17:34:50.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:34:49 smithi038 ceph-mon[92422]: pgmap v5526: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 88 KiB/s wr, 8 op/s 2023-03-09T17:34:51.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:34:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9401.063900 secs 2023-03-09T17:34:51.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:34:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9401.063900 secs 2023-03-09T17:34:51.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:34:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9401.063900 secs 2023-03-09T17:34:52.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:34:51 smithi122 ceph-mon[93869]: pgmap v5527: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 88 KiB/s wr, 8 op/s 2023-03-09T17:34:52.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:34:51 smithi181 ceph-mon[94189]: pgmap v5527: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 88 KiB/s wr, 8 op/s 2023-03-09T17:34:52.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:34:51 smithi038 ceph-mon[92422]: pgmap v5527: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 88 KiB/s wr, 8 op/s 2023-03-09T17:34:53.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:34:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9403.336133 secs 2023-03-09T17:34:53.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:34:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9403.336133 secs 2023-03-09T17:34:53.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:34:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9403.336133 secs 2023-03-09T17:34:53.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9865.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:34:54.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:34:53 smithi122 ceph-mon[93869]: pgmap v5528: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 125 KiB/s wr, 11 op/s 2023-03-09T17:34:54.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:34:53 smithi181 ceph-mon[94189]: pgmap v5528: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 125 KiB/s wr, 11 op/s 2023-03-09T17:34:54.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:34:53 smithi038 ceph-mon[92422]: pgmap v5528: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 125 KiB/s wr, 11 op/s 2023-03-09T17:34:56.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:34:55 smithi122 ceph-mon[93869]: pgmap v5529: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 102 KiB/s wr, 9 op/s 2023-03-09T17:34:56.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:34:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9406.063962 secs 2023-03-09T17:34:56.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:34:55 smithi181 ceph-mon[94189]: pgmap v5529: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 102 KiB/s wr, 9 op/s 2023-03-09T17:34:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:34:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9406.063962 secs 2023-03-09T17:34:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:34:55 smithi038 ceph-mon[92422]: pgmap v5529: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 102 KiB/s wr, 9 op/s 2023-03-09T17:34:56.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:34:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9406.063962 secs 2023-03-09T17:34:57.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:34:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:34:57.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:34:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:34:57.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:34:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:34:58.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:34:57 smithi122 ceph-mon[93869]: pgmap v5530: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 103 KiB/s wr, 12 op/s 2023-03-09T17:34:58.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:34:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9408.336229 secs 2023-03-09T17:34:58.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:34:57 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:34:58.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:34:57 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:34:58.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:34:57 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:34:58.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:34:57 smithi181 ceph-mon[94189]: pgmap v5530: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 103 KiB/s wr, 12 op/s 2023-03-09T17:34:58.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:34:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9408.336229 secs 2023-03-09T17:34:58.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:34:57 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:34:58.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:34:57 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:34:58.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:34:57 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:34:58.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:34:57 smithi038 ceph-mon[92422]: pgmap v5530: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 103 KiB/s wr, 12 op/s 2023-03-09T17:34:58.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:34:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9408.336229 secs 2023-03-09T17:34:58.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:34:57 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:34:58.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:34:57 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:34:58.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:34:57 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:34:58.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9870.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:35:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:34:59 smithi122 ceph-mon[93869]: pgmap v5531: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 151 KiB/s wr, 10 op/s 2023-03-09T17:35:00.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:34:59 smithi181 ceph-mon[94189]: pgmap v5531: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 151 KiB/s wr, 10 op/s 2023-03-09T17:35:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:34:59 smithi038 ceph-mon[92422]: pgmap v5531: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 151 KiB/s wr, 10 op/s 2023-03-09T17:35:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:35:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9411.063988 secs 2023-03-09T17:35:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:35:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9411.063988 secs 2023-03-09T17:35:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:35:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9411.063988 secs 2023-03-09T17:35:02.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:35:01 smithi122 ceph-mon[93869]: pgmap v5532: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 126 KiB/s wr, 9 op/s 2023-03-09T17:35:02.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:35:01 smithi181 ceph-mon[94189]: pgmap v5532: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 126 KiB/s wr, 9 op/s 2023-03-09T17:35:02.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:35:01 smithi038 ceph-mon[92422]: pgmap v5532: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 126 KiB/s wr, 9 op/s 2023-03-09T17:35:03.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:35:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9413.336381 secs 2023-03-09T17:35:03.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:35:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9413.336381 secs 2023-03-09T17:35:03.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:35:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9413.336381 secs 2023-03-09T17:35:03.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9875.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:35:04.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:35:03 smithi122 ceph-mon[93869]: pgmap v5533: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 164 KiB/s wr, 14 op/s 2023-03-09T17:35:04.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:35:03 smithi181 ceph-mon[94189]: pgmap v5533: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 164 KiB/s wr, 14 op/s 2023-03-09T17:35:04.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:35:03 smithi038 ceph-mon[92422]: pgmap v5533: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 164 KiB/s wr, 14 op/s 2023-03-09T17:35:06.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:35:05 smithi122 ceph-mon[93869]: pgmap v5534: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 179 KiB/s wr, 24 op/s 2023-03-09T17:35:06.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:35:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9416.064052 secs 2023-03-09T17:35:06.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:35:05 smithi181 ceph-mon[94189]: pgmap v5534: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 179 KiB/s wr, 24 op/s 2023-03-09T17:35:06.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:35:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9416.064052 secs 2023-03-09T17:35:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:35:05 smithi038 ceph-mon[92422]: pgmap v5534: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 179 KiB/s wr, 24 op/s 2023-03-09T17:35:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:35:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9416.064052 secs 2023-03-09T17:35:08.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:35:07 smithi122 ceph-mon[93869]: pgmap v5535: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 162 KiB/s wr, 26 op/s 2023-03-09T17:35:08.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:35:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9418.336491 secs 2023-03-09T17:35:08.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:35:07 smithi181 ceph-mon[94189]: pgmap v5535: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 162 KiB/s wr, 26 op/s 2023-03-09T17:35:08.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:35:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9418.336491 secs 2023-03-09T17:35:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:35:07 smithi038 ceph-mon[92422]: pgmap v5535: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 162 KiB/s wr, 26 op/s 2023-03-09T17:35:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:35:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9418.336491 secs 2023-03-09T17:35:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9880.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:35:10.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:35:09 smithi122 ceph-mon[93869]: pgmap v5536: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 210 KiB/s wr, 25 op/s 2023-03-09T17:35:10.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:35:09 smithi122 ceph-mon[93869]: osdmap e559: 12 total, 12 up, 12 in 2023-03-09T17:35:10.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:35:09 smithi181 ceph-mon[94189]: pgmap v5536: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 210 KiB/s wr, 25 op/s 2023-03-09T17:35:10.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:35:09 smithi181 ceph-mon[94189]: osdmap e559: 12 total, 12 up, 12 in 2023-03-09T17:35:10.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:35:09 smithi038 ceph-mon[92422]: pgmap v5536: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 210 KiB/s wr, 25 op/s 2023-03-09T17:35:10.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:35:09 smithi038 ceph-mon[92422]: osdmap e559: 12 total, 12 up, 12 in 2023-03-09T17:35:11.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:35:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9421.064112 secs 2023-03-09T17:35:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:35:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9421.064112 secs 2023-03-09T17:35:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:35:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9421.064112 secs 2023-03-09T17:35:12.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:35:11 smithi122 ceph-mon[93869]: pgmap v5538: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 176 KiB/s wr, 27 op/s 2023-03-09T17:35:12.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:35:11 smithi181 ceph-mon[94189]: pgmap v5538: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 176 KiB/s wr, 27 op/s 2023-03-09T17:35:12.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:35:11 smithi038 ceph-mon[92422]: pgmap v5538: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 176 KiB/s wr, 27 op/s 2023-03-09T17:35:13.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:35:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9423.336632 secs 2023-03-09T17:35:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:35:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9423.336632 secs 2023-03-09T17:35:13.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:35:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9423.336632 secs 2023-03-09T17:35:13.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9885.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:35:14.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:35:13 smithi122 ceph-mon[93869]: pgmap v5539: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 130 KiB/s wr, 25 op/s 2023-03-09T17:35:14.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:35:13 smithi181 ceph-mon[94189]: pgmap v5539: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 130 KiB/s wr, 25 op/s 2023-03-09T17:35:14.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:35:13 smithi038 ceph-mon[92422]: pgmap v5539: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 130 KiB/s wr, 25 op/s 2023-03-09T17:35:16.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:35:15 smithi122 ceph-mon[93869]: pgmap v5540: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 89 KiB/s wr, 9 op/s 2023-03-09T17:35:16.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:35:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9426.064175 secs 2023-03-09T17:35:16.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:35:15 smithi122 ceph-mon[93869]: osdmap e560: 12 total, 12 up, 12 in 2023-03-09T17:35:16.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:35:15 smithi181 ceph-mon[94189]: pgmap v5540: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 89 KiB/s wr, 9 op/s 2023-03-09T17:35:16.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:35:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9426.064175 secs 2023-03-09T17:35:16.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:35:15 smithi181 ceph-mon[94189]: osdmap e560: 12 total, 12 up, 12 in 2023-03-09T17:35:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:35:15 smithi038 ceph-mon[92422]: pgmap v5540: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 89 KiB/s wr, 9 op/s 2023-03-09T17:35:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:35:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9426.064175 secs 2023-03-09T17:35:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:35:15 smithi038 ceph-mon[92422]: osdmap e560: 12 total, 12 up, 12 in 2023-03-09T17:35:18.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:35:17 smithi122 ceph-mon[93869]: pgmap v5542: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T17:35:18.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:35:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9428.336777 secs 2023-03-09T17:35:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:35:17 smithi181 ceph-mon[94189]: pgmap v5542: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T17:35:18.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:35:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9428.336777 secs 2023-03-09T17:35:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:35:17 smithi038 ceph-mon[92422]: pgmap v5542: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T17:35:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:35:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9428.336777 secs 2023-03-09T17:35:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9890.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:35:20.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:35:19 smithi122 ceph-mon[93869]: pgmap v5543: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 42 KiB/s wr, 7 op/s 2023-03-09T17:35:20.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:35:19 smithi181 ceph-mon[94189]: pgmap v5543: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 42 KiB/s wr, 7 op/s 2023-03-09T17:35:20.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:35:19 smithi038 ceph-mon[92422]: pgmap v5543: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 42 KiB/s wr, 7 op/s 2023-03-09T17:35:21.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:35:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9431.064247 secs 2023-03-09T17:35:21.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:35:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9431.064247 secs 2023-03-09T17:35:21.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:35:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9431.064247 secs 2023-03-09T17:35:22.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:35:21 smithi122 ceph-mon[93869]: pgmap v5544: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 40 KiB/s wr, 7 op/s 2023-03-09T17:35:22.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:35:21 smithi181 ceph-mon[94189]: pgmap v5544: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 40 KiB/s wr, 7 op/s 2023-03-09T17:35:22.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:35:21 smithi038 ceph-mon[92422]: pgmap v5544: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 40 KiB/s wr, 7 op/s 2023-03-09T17:35:23.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:35:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9433.336873 secs 2023-03-09T17:35:23.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:35:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9433.336873 secs 2023-03-09T17:35:23.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:35:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9433.336873 secs 2023-03-09T17:35:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9895.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:35:24.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:35:23 smithi122 ceph-mon[93869]: pgmap v5545: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 41 KiB/s wr, 7 op/s 2023-03-09T17:35:24.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:35:23 smithi181 ceph-mon[94189]: pgmap v5545: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 41 KiB/s wr, 7 op/s 2023-03-09T17:35:24.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:35:23 smithi038 ceph-mon[92422]: pgmap v5545: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 41 KiB/s wr, 7 op/s 2023-03-09T17:35:26.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:35:25 smithi122 ceph-mon[93869]: pgmap v5546: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 49 KiB/s wr, 7 op/s 2023-03-09T17:35:26.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:35:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9436.064307 secs 2023-03-09T17:35:26.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:35:25 smithi181 ceph-mon[94189]: pgmap v5546: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 49 KiB/s wr, 7 op/s 2023-03-09T17:35:26.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:35:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9436.064307 secs 2023-03-09T17:35:26.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:35:25 smithi038 ceph-mon[92422]: pgmap v5546: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 49 KiB/s wr, 7 op/s 2023-03-09T17:35:26.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:35:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9436.064307 secs 2023-03-09T17:35:28.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:35:27 smithi122 ceph-mon[93869]: pgmap v5547: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 45 KiB/s wr, 6 op/s 2023-03-09T17:35:28.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:35:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9438.337016 secs 2023-03-09T17:35:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:35:27 smithi181 ceph-mon[94189]: pgmap v5547: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 45 KiB/s wr, 6 op/s 2023-03-09T17:35:28.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:35:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9438.337016 secs 2023-03-09T17:35:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:35:27 smithi038 ceph-mon[92422]: pgmap v5547: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 45 KiB/s wr, 6 op/s 2023-03-09T17:35:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:35:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9438.337016 secs 2023-03-09T17:35:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9900.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:35:30.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:35:29 smithi122 ceph-mon[93869]: pgmap v5548: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 41 KiB/s wr, 7 op/s 2023-03-09T17:35:30.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:35:29 smithi181 ceph-mon[94189]: pgmap v5548: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 41 KiB/s wr, 7 op/s 2023-03-09T17:35:30.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:35:29 smithi038 ceph-mon[92422]: pgmap v5548: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 41 KiB/s wr, 7 op/s 2023-03-09T17:35:31.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:35:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9441.064366 secs 2023-03-09T17:35:31.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:35:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9441.064366 secs 2023-03-09T17:35:31.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:35:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9441.064366 secs 2023-03-09T17:35:32.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:35:31 smithi122 ceph-mon[93869]: pgmap v5549: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 25 KiB/s wr, 6 op/s 2023-03-09T17:35:32.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:35:31 smithi181 ceph-mon[94189]: pgmap v5549: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 25 KiB/s wr, 6 op/s 2023-03-09T17:35:32.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:35:31 smithi038 ceph-mon[92422]: pgmap v5549: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 25 KiB/s wr, 6 op/s 2023-03-09T17:35:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9905.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:35:34.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:35:33 smithi122 ceph-mon[93869]: pgmap v5550: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 26 KiB/s wr, 9 op/s 2023-03-09T17:35:34.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:35:33 smithi181 ceph-mon[94189]: pgmap v5550: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 26 KiB/s wr, 9 op/s 2023-03-09T17:35:34.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:35:33 smithi038 ceph-mon[92422]: pgmap v5550: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 26 KiB/s wr, 9 op/s 2023-03-09T17:35:36.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:35:35 smithi122 ceph-mon[93869]: pgmap v5551: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 7 op/s 2023-03-09T17:35:36.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:35:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9446.064428 secs 2023-03-09T17:35:36.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:35:35 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:35:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:35:35 smithi181 ceph-mon[94189]: pgmap v5551: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 7 op/s 2023-03-09T17:35:36.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:35:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9446.064428 secs 2023-03-09T17:35:36.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:35:35 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:35:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:35:35 smithi038 ceph-mon[92422]: pgmap v5551: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 7 op/s 2023-03-09T17:35:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:35:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9446.064428 secs 2023-03-09T17:35:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:35:35 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:35:37.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:35:36 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:35:37.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:35:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:35:37.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:35:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:35:37.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:35:36 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:35:37.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:35:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:35:37.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:35:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:35:37.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:35:36 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:35:37.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:35:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:35:37.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:35:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:35:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:35:37 smithi122 ceph-mon[93869]: pgmap v5552: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.3 KiB/s wr, 8 op/s 2023-03-09T17:35:38.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:35:37 smithi181 ceph-mon[94189]: pgmap v5552: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.3 KiB/s wr, 8 op/s 2023-03-09T17:35:38.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:35:37 smithi038 ceph-mon[92422]: pgmap v5552: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.3 KiB/s wr, 8 op/s 2023-03-09T17:35:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9910.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:35:40.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:35:39 smithi122 ceph-mon[93869]: pgmap v5553: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 62 KiB/s wr, 8 op/s 2023-03-09T17:35:40.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:35:39 smithi181 ceph-mon[94189]: pgmap v5553: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 62 KiB/s wr, 8 op/s 2023-03-09T17:35:40.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:35:39 smithi038 ceph-mon[92422]: pgmap v5553: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 62 KiB/s wr, 8 op/s 2023-03-09T17:35:41.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:35:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9451.064488 secs 2023-03-09T17:35:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:35:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9451.064488 secs 2023-03-09T17:35:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:35:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9451.064488 secs 2023-03-09T17:35:42.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:35:41 smithi122 ceph-mon[93869]: pgmap v5554: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 61 KiB/s wr, 7 op/s 2023-03-09T17:35:42.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:35:41 smithi181 ceph-mon[94189]: pgmap v5554: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 61 KiB/s wr, 7 op/s 2023-03-09T17:35:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:35:41 smithi038 ceph-mon[92422]: pgmap v5554: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 61 KiB/s wr, 7 op/s 2023-03-09T17:35:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9915.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:35:44.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:35:43 smithi122 ceph-mon[93869]: pgmap v5555: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 70 KiB/s wr, 10 op/s 2023-03-09T17:35:44.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:35:43 smithi181 ceph-mon[94189]: pgmap v5555: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 70 KiB/s wr, 10 op/s 2023-03-09T17:35:44.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:35:43 smithi038 ceph-mon[92422]: pgmap v5555: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 70 KiB/s wr, 10 op/s 2023-03-09T17:35:46.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:35:45 smithi122 ceph-mon[93869]: pgmap v5556: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 114 KiB/s wr, 9 op/s 2023-03-09T17:35:46.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:35:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9456.064563 secs 2023-03-09T17:35:46.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:35:45 smithi181 ceph-mon[94189]: pgmap v5556: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 114 KiB/s wr, 9 op/s 2023-03-09T17:35:46.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:35:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9456.064563 secs 2023-03-09T17:35:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:35:45 smithi038 ceph-mon[92422]: pgmap v5556: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 114 KiB/s wr, 9 op/s 2023-03-09T17:35:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:35:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9456.064563 secs 2023-03-09T17:35:48.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:35:47 smithi122 ceph-mon[93869]: pgmap v5557: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 160 KiB/s wr, 12 op/s 2023-03-09T17:35:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:35:47 smithi181 ceph-mon[94189]: pgmap v5557: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 160 KiB/s wr, 12 op/s 2023-03-09T17:35:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:35:47 smithi038 ceph-mon[92422]: pgmap v5557: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 160 KiB/s wr, 12 op/s 2023-03-09T17:35:48.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9920.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:35:50.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:35:49 smithi122 ceph-mon[93869]: pgmap v5558: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 152 KiB/s wr, 11 op/s 2023-03-09T17:35:50.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:35:49 smithi181 ceph-mon[94189]: pgmap v5558: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 152 KiB/s wr, 11 op/s 2023-03-09T17:35:50.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:35:49 smithi038 ceph-mon[92422]: pgmap v5558: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 152 KiB/s wr, 11 op/s 2023-03-09T17:35:51.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:35:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9461.064624 secs 2023-03-09T17:35:51.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:35:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9461.064624 secs 2023-03-09T17:35:51.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:35:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9461.064624 secs 2023-03-09T17:35:52.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:35:51 smithi122 ceph-mon[93869]: pgmap v5559: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 99 KiB/s wr, 9 op/s 2023-03-09T17:35:52.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:35:51 smithi181 ceph-mon[94189]: pgmap v5559: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 99 KiB/s wr, 9 op/s 2023-03-09T17:35:52.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:35:51 smithi038 ceph-mon[92422]: pgmap v5559: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 99 KiB/s wr, 9 op/s 2023-03-09T17:35:53.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9925.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:35:54.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:35:53 smithi181 ceph-mon[94189]: pgmap v5560: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 147 KiB/s wr, 13 op/s 2023-03-09T17:35:54.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:35:53 smithi038 ceph-mon[92422]: pgmap v5560: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 147 KiB/s wr, 13 op/s 2023-03-09T17:35:54.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:35:53 smithi122 ceph-mon[93869]: pgmap v5560: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 147 KiB/s wr, 13 op/s 2023-03-09T17:35:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:35:55 smithi038 ceph-mon[92422]: pgmap v5561: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 139 KiB/s wr, 11 op/s 2023-03-09T17:35:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:35:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9466.064702 secs 2023-03-09T17:35:56.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:35:55 smithi122 ceph-mon[93869]: pgmap v5561: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 139 KiB/s wr, 11 op/s 2023-03-09T17:35:56.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:35:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9466.064702 secs 2023-03-09T17:35:56.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:35:55 smithi181 ceph-mon[94189]: pgmap v5561: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 139 KiB/s wr, 11 op/s 2023-03-09T17:35:56.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:35:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9466.064702 secs 2023-03-09T17:35:58.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:35:57 smithi038 ceph-mon[92422]: pgmap v5562: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 152 KiB/s wr, 13 op/s 2023-03-09T17:35:58.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:35:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9468.337717 secs 2023-03-09T17:35:58.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:35:57 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:35:58.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:35:57 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:35:58.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:35:57 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:35:58.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:35:57 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:35:58.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:35:57 smithi122 ceph-mon[93869]: pgmap v5562: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 152 KiB/s wr, 13 op/s 2023-03-09T17:35:58.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:35:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9468.337717 secs 2023-03-09T17:35:58.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:35:57 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:35:58.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:35:57 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:35:58.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:35:57 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:35:58.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:35:57 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:35:58.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:35:57 smithi181 ceph-mon[94189]: pgmap v5562: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 152 KiB/s wr, 13 op/s 2023-03-09T17:35:58.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:35:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9468.337717 secs 2023-03-09T17:35:58.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:35:57 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:35:58.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:35:57 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:35:58.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:35:57 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:35:58.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:35:57 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:35:58.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9930.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:36:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:35:59 smithi038 ceph-mon[92422]: pgmap v5563: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 106 KiB/s wr, 9 op/s 2023-03-09T17:36:00.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:35:59 smithi122 ceph-mon[93869]: pgmap v5563: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 106 KiB/s wr, 9 op/s 2023-03-09T17:36:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:35:59 smithi181 ceph-mon[94189]: pgmap v5563: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 106 KiB/s wr, 9 op/s 2023-03-09T17:36:01.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:36:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9471.064763 secs 2023-03-09T17:36:01.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:36:01 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:36:01.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:36:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:36:01.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:36:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:36:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:36:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9471.064763 secs 2023-03-09T17:36:01.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:36:01 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:36:01.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:36:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:36:01.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:36:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:36:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:36:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9471.064763 secs 2023-03-09T17:36:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:36:01 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:36:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:36:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:36:01.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:36:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:36:02.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:36:02 smithi122 ceph-mon[93869]: pgmap v5564: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 106 KiB/s wr, 8 op/s 2023-03-09T17:36:02.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:36:02 smithi181 ceph-mon[94189]: pgmap v5564: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 106 KiB/s wr, 8 op/s 2023-03-09T17:36:02.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:36:02 smithi038 ceph-mon[92422]: pgmap v5564: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 106 KiB/s wr, 8 op/s 2023-03-09T17:36:03.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:36:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9473.337835 secs 2023-03-09T17:36:03.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:36:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9473.337835 secs 2023-03-09T17:36:03.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:36:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9473.337835 secs 2023-03-09T17:36:03.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9935.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:36:04.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:36:04 smithi122 ceph-mon[93869]: pgmap v5565: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 152 KiB/s wr, 14 op/s 2023-03-09T17:36:04.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:36:04 smithi181 ceph-mon[94189]: pgmap v5565: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 152 KiB/s wr, 14 op/s 2023-03-09T17:36:04.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:36:04 smithi038 ceph-mon[92422]: pgmap v5565: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 152 KiB/s wr, 14 op/s 2023-03-09T17:36:06.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:36:06 smithi122 ceph-mon[93869]: pgmap v5566: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 126 KiB/s wr, 22 op/s 2023-03-09T17:36:06.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:36:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9476.064842 secs 2023-03-09T17:36:06.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:36:06 smithi181 ceph-mon[94189]: pgmap v5566: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 126 KiB/s wr, 22 op/s 2023-03-09T17:36:06.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:36:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9476.064842 secs 2023-03-09T17:36:06.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:36:06 smithi038 ceph-mon[92422]: pgmap v5566: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 126 KiB/s wr, 22 op/s 2023-03-09T17:36:06.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:36:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9476.064842 secs 2023-03-09T17:36:08.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:36:08 smithi122 ceph-mon[93869]: pgmap v5567: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 126 KiB/s wr, 24 op/s 2023-03-09T17:36:08.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:36:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9478.337973 secs 2023-03-09T17:36:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:36:08 smithi181 ceph-mon[94189]: pgmap v5567: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 126 KiB/s wr, 24 op/s 2023-03-09T17:36:08.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:36:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9478.337973 secs 2023-03-09T17:36:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:36:08 smithi038 ceph-mon[92422]: pgmap v5567: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 126 KiB/s wr, 24 op/s 2023-03-09T17:36:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:36:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9478.337973 secs 2023-03-09T17:36:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9940.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:36:10.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:36:10 smithi122 ceph-mon[93869]: pgmap v5568: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 68 KiB/s wr, 21 op/s 2023-03-09T17:36:10.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:36:10 smithi122 ceph-mon[93869]: osdmap e561: 12 total, 12 up, 12 in 2023-03-09T17:36:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:36:10 smithi181 ceph-mon[94189]: pgmap v5568: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 68 KiB/s wr, 21 op/s 2023-03-09T17:36:10.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:36:10 smithi181 ceph-mon[94189]: osdmap e561: 12 total, 12 up, 12 in 2023-03-09T17:36:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:36:10 smithi038 ceph-mon[92422]: pgmap v5568: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 68 KiB/s wr, 21 op/s 2023-03-09T17:36:10.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:36:10 smithi038 ceph-mon[92422]: osdmap e561: 12 total, 12 up, 12 in 2023-03-09T17:36:11.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:36:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9481.064901 secs 2023-03-09T17:36:11.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:36:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9481.064901 secs 2023-03-09T17:36:11.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:36:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9481.064901 secs 2023-03-09T17:36:12.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:36:12 smithi122 ceph-mon[93869]: pgmap v5570: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 81 KiB/s wr, 24 op/s 2023-03-09T17:36:12.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:36:12 smithi181 ceph-mon[94189]: pgmap v5570: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 81 KiB/s wr, 24 op/s 2023-03-09T17:36:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:36:12 smithi038 ceph-mon[92422]: pgmap v5570: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 81 KiB/s wr, 24 op/s 2023-03-09T17:36:13.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:36:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9483.338095 secs 2023-03-09T17:36:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:36:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9483.338095 secs 2023-03-09T17:36:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:36:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9483.338095 secs 2023-03-09T17:36:13.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9945.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:36:14.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:36:14 smithi122 ceph-mon[93869]: pgmap v5571: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 57 KiB/s wr, 22 op/s 2023-03-09T17:36:14.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:36:14 smithi181 ceph-mon[94189]: pgmap v5571: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 57 KiB/s wr, 22 op/s 2023-03-09T17:36:14.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:36:14 smithi038 ceph-mon[92422]: pgmap v5571: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 57 KiB/s wr, 22 op/s 2023-03-09T17:36:16.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:36:16 smithi122 ceph-mon[93869]: pgmap v5572: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T17:36:16.497 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:36:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9486.064962 secs 2023-03-09T17:36:16.497 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:36:16 smithi122 ceph-mon[93869]: osdmap e562: 12 total, 12 up, 12 in 2023-03-09T17:36:16.498 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:36:16 smithi181 ceph-mon[94189]: pgmap v5572: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T17:36:16.498 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:36:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9486.064962 secs 2023-03-09T17:36:16.498 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:36:16 smithi181 ceph-mon[94189]: osdmap e562: 12 total, 12 up, 12 in 2023-03-09T17:36:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:36:16 smithi038 ceph-mon[92422]: pgmap v5572: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T17:36:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:36:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9486.064962 secs 2023-03-09T17:36:16.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:36:16 smithi038 ceph-mon[92422]: osdmap e562: 12 total, 12 up, 12 in 2023-03-09T17:36:18.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:36:18 smithi122 ceph-mon[93869]: pgmap v5574: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 51 KiB/s wr, 9 op/s 2023-03-09T17:36:18.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:36:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9488.338218 secs 2023-03-09T17:36:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:36:18 smithi181 ceph-mon[94189]: pgmap v5574: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 51 KiB/s wr, 9 op/s 2023-03-09T17:36:18.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:36:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9488.338218 secs 2023-03-09T17:36:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:36:18 smithi038 ceph-mon[92422]: pgmap v5574: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 51 KiB/s wr, 9 op/s 2023-03-09T17:36:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:36:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9488.338218 secs 2023-03-09T17:36:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9950.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:36:20.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:36:20 smithi122 ceph-mon[93869]: pgmap v5575: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 45 KiB/s wr, 8 op/s 2023-03-09T17:36:20.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:36:20 smithi181 ceph-mon[94189]: pgmap v5575: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 45 KiB/s wr, 8 op/s 2023-03-09T17:36:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:36:20 smithi038 ceph-mon[92422]: pgmap v5575: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 45 KiB/s wr, 8 op/s 2023-03-09T17:36:21.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:36:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9491.064993 secs 2023-03-09T17:36:21.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:36:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9491.064993 secs 2023-03-09T17:36:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:36:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9491.064993 secs 2023-03-09T17:36:22.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:36:22 smithi122 ceph-mon[93869]: pgmap v5576: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 8 op/s 2023-03-09T17:36:22.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:36:22 smithi181 ceph-mon[94189]: pgmap v5576: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 8 op/s 2023-03-09T17:36:22.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:36:22 smithi038 ceph-mon[92422]: pgmap v5576: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 8 op/s 2023-03-09T17:36:23.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:36:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9493.338374 secs 2023-03-09T17:36:23.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:36:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9493.338374 secs 2023-03-09T17:36:23.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:36:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9493.338374 secs 2023-03-09T17:36:23.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9955.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:36:24.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:36:24 smithi122 ceph-mon[93869]: pgmap v5577: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 7 op/s 2023-03-09T17:36:24.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:36:24 smithi181 ceph-mon[94189]: pgmap v5577: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 7 op/s 2023-03-09T17:36:24.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:36:24 smithi038 ceph-mon[92422]: pgmap v5577: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 7 op/s 2023-03-09T17:36:26.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:36:26 smithi122 ceph-mon[93869]: pgmap v5578: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 8 op/s 2023-03-09T17:36:26.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:36:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9496.065050 secs 2023-03-09T17:36:26.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:36:26 smithi181 ceph-mon[94189]: pgmap v5578: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 8 op/s 2023-03-09T17:36:26.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:36:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9496.065050 secs 2023-03-09T17:36:26.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:36:26 smithi038 ceph-mon[92422]: pgmap v5578: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 8 op/s 2023-03-09T17:36:26.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:36:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9496.065050 secs 2023-03-09T17:36:28.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:36:28 smithi122 ceph-mon[93869]: pgmap v5579: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 29 KiB/s wr, 7 op/s 2023-03-09T17:36:28.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:36:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9498.338456 secs 2023-03-09T17:36:28.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:36:28 smithi181 ceph-mon[94189]: pgmap v5579: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 29 KiB/s wr, 7 op/s 2023-03-09T17:36:28.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:36:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9498.338456 secs 2023-03-09T17:36:28.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:36:28 smithi038 ceph-mon[92422]: pgmap v5579: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 29 KiB/s wr, 7 op/s 2023-03-09T17:36:28.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:36:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9498.338456 secs 2023-03-09T17:36:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9960.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:36:30.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:36:30 smithi122 ceph-mon[93869]: pgmap v5580: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 26 KiB/s wr, 7 op/s 2023-03-09T17:36:30.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:36:30 smithi181 ceph-mon[94189]: pgmap v5580: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 26 KiB/s wr, 7 op/s 2023-03-09T17:36:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:36:30 smithi038 ceph-mon[92422]: pgmap v5580: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 26 KiB/s wr, 7 op/s 2023-03-09T17:36:31.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:36:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9501.065143 secs 2023-03-09T17:36:31.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:36:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9501.065143 secs 2023-03-09T17:36:31.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:36:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9501.065143 secs 2023-03-09T17:36:32.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:36:32 smithi122 ceph-mon[93869]: pgmap v5581: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 6 op/s 2023-03-09T17:36:32.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:36:32 smithi181 ceph-mon[94189]: pgmap v5581: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 6 op/s 2023-03-09T17:36:32.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:36:32 smithi038 ceph-mon[92422]: pgmap v5581: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 6 op/s 2023-03-09T17:36:33.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:36:33 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 9503.338551 secs 2023-03-09T17:36:33.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:36:33 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 9503.338551 secs 2023-03-09T17:36:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:36:33 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 9503.338551 secs 2023-03-09T17:36:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9965.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:36:34.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:36:34 smithi122 ceph-mon[93869]: pgmap v5582: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 47 KiB/s wr, 10 op/s 2023-03-09T17:36:34.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:36:34 smithi181 ceph-mon[94189]: pgmap v5582: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 47 KiB/s wr, 10 op/s 2023-03-09T17:36:34.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:36:34 smithi038 ceph-mon[92422]: pgmap v5582: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 47 KiB/s wr, 10 op/s 2023-03-09T17:36:36.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:36:36 smithi122 ceph-mon[93869]: pgmap v5583: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 46 KiB/s wr, 7 op/s 2023-03-09T17:36:36.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:36:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9506.065206 secs 2023-03-09T17:36:36.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:36:36 smithi181 ceph-mon[94189]: pgmap v5583: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 46 KiB/s wr, 7 op/s 2023-03-09T17:36:36.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:36:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9506.065206 secs 2023-03-09T17:36:36.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:36:36 smithi038 ceph-mon[92422]: pgmap v5583: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 46 KiB/s wr, 7 op/s 2023-03-09T17:36:36.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:36:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9506.065206 secs 2023-03-09T17:36:37.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:36:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:36:37.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:36:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:36:37.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:36:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:36:37.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:36:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:36:37.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:36:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:36:37.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:36:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:36:38.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:36:38 smithi122 ceph-mon[93869]: pgmap v5584: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 88 KiB/s wr, 10 op/s 2023-03-09T17:36:38.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:36:38 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 9508.338669 secs 2023-03-09T17:36:38.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:36:38 smithi181 ceph-mon[94189]: pgmap v5584: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 88 KiB/s wr, 10 op/s 2023-03-09T17:36:38.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:36:38 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 9508.338669 secs 2023-03-09T17:36:38.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:36:38 smithi038 ceph-mon[92422]: pgmap v5584: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 88 KiB/s wr, 10 op/s 2023-03-09T17:36:38.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:36:38 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 9508.338669 secs 2023-03-09T17:36:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9970.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:36:40.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:36:40 smithi122 ceph-mon[93869]: pgmap v5585: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 64 KiB/s wr, 7 op/s 2023-03-09T17:36:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:36:40 smithi181 ceph-mon[94189]: pgmap v5585: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 64 KiB/s wr, 7 op/s 2023-03-09T17:36:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:36:40 smithi038 ceph-mon[92422]: pgmap v5585: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 64 KiB/s wr, 7 op/s 2023-03-09T17:36:41.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:36:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9511.065268 secs 2023-03-09T17:36:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:36:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9511.065268 secs 2023-03-09T17:36:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:36:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9511.065268 secs 2023-03-09T17:36:42.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:36:42 smithi122 ceph-mon[93869]: pgmap v5586: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 64 KiB/s wr, 7 op/s 2023-03-09T17:36:42.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:36:42 smithi181 ceph-mon[94189]: pgmap v5586: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 64 KiB/s wr, 7 op/s 2023-03-09T17:36:42.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:36:42 smithi038 ceph-mon[92422]: pgmap v5586: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 64 KiB/s wr, 7 op/s 2023-03-09T17:36:43.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:36:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9513.338734 secs 2023-03-09T17:36:43.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:36:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9513.338734 secs 2023-03-09T17:36:43.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:36:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9513.338734 secs 2023-03-09T17:36:43.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9975.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:36:44.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:36:44 smithi122 ceph-mon[93869]: pgmap v5587: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 115 KiB/s wr, 11 op/s 2023-03-09T17:36:44.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:36:44 smithi181 ceph-mon[94189]: pgmap v5587: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 115 KiB/s wr, 11 op/s 2023-03-09T17:36:44.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:36:44 smithi038 ceph-mon[92422]: pgmap v5587: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 115 KiB/s wr, 11 op/s 2023-03-09T17:36:45.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:36:45 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:36:45.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:36:45 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:36:45.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:36:45 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:36:45.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:36:45 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:36:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:36:45 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:36:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:36:45 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:36:46.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:36:46 smithi122 ceph-mon[93869]: pgmap v5588: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 94 KiB/s wr, 8 op/s 2023-03-09T17:36:46.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:36:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9516.065330 secs 2023-03-09T17:36:46.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:36:46 smithi181 ceph-mon[94189]: pgmap v5588: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 94 KiB/s wr, 8 op/s 2023-03-09T17:36:46.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:36:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9516.065330 secs 2023-03-09T17:36:46.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:36:46 smithi038 ceph-mon[92422]: pgmap v5588: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 94 KiB/s wr, 8 op/s 2023-03-09T17:36:46.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:36:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9516.065330 secs 2023-03-09T17:36:48.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:36:48 smithi122 ceph-mon[93869]: pgmap v5589: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 170 KiB/s wr, 12 op/s 2023-03-09T17:36:48.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:36:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9518.338858 secs 2023-03-09T17:36:48.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:36:48 smithi181 ceph-mon[94189]: pgmap v5589: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 170 KiB/s wr, 12 op/s 2023-03-09T17:36:48.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:36:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9518.338858 secs 2023-03-09T17:36:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:36:48 smithi038 ceph-mon[92422]: pgmap v5589: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 170 KiB/s wr, 12 op/s 2023-03-09T17:36:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:36:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9518.338858 secs 2023-03-09T17:36:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9980.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:36:50.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:36:50 smithi122 ceph-mon[93869]: pgmap v5590: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 129 KiB/s wr, 9 op/s 2023-03-09T17:36:50.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:36:50 smithi181 ceph-mon[94189]: pgmap v5590: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 129 KiB/s wr, 9 op/s 2023-03-09T17:36:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:36:50 smithi038 ceph-mon[92422]: pgmap v5590: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 129 KiB/s wr, 9 op/s 2023-03-09T17:36:51.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:36:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9521.065389 secs 2023-03-09T17:36:51.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:36:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9521.065389 secs 2023-03-09T17:36:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:36:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9521.065389 secs 2023-03-09T17:36:52.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:36:52 smithi122 ceph-mon[93869]: pgmap v5591: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 128 KiB/s wr, 9 op/s 2023-03-09T17:36:52.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:36:52 smithi181 ceph-mon[94189]: pgmap v5591: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 128 KiB/s wr, 9 op/s 2023-03-09T17:36:52.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:36:52 smithi038 ceph-mon[92422]: pgmap v5591: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 128 KiB/s wr, 9 op/s 2023-03-09T17:36:53.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:36:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9523.338968 secs 2023-03-09T17:36:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:36:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9523.338968 secs 2023-03-09T17:36:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:36:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9523.338968 secs 2023-03-09T17:36:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9985.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:36:54.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:36:54 smithi181 ceph-mon[94189]: pgmap v5592: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 162 KiB/s wr, 13 op/s 2023-03-09T17:36:54.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:36:54 smithi038 ceph-mon[92422]: pgmap v5592: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 162 KiB/s wr, 13 op/s 2023-03-09T17:36:54.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:36:54 smithi122 ceph-mon[93869]: pgmap v5592: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 162 KiB/s wr, 13 op/s 2023-03-09T17:36:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:36:56 smithi038 ceph-mon[92422]: pgmap v5593: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 111 KiB/s wr, 10 op/s 2023-03-09T17:36:56.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:36:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9526.065480 secs 2023-03-09T17:36:56.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:36:56 smithi122 ceph-mon[93869]: pgmap v5593: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 111 KiB/s wr, 10 op/s 2023-03-09T17:36:56.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:36:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9526.065480 secs 2023-03-09T17:36:56.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:36:56 smithi181 ceph-mon[94189]: pgmap v5593: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 111 KiB/s wr, 10 op/s 2023-03-09T17:36:56.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:36:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9526.065480 secs 2023-03-09T17:36:58.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:36:58 smithi038 ceph-mon[92422]: pgmap v5594: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 161 KiB/s wr, 13 op/s 2023-03-09T17:36:58.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:36:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9528.339101 secs 2023-03-09T17:36:58.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:36:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:36:58.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:36:58 smithi122 ceph-mon[93869]: pgmap v5594: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 161 KiB/s wr, 13 op/s 2023-03-09T17:36:58.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:36:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9528.339101 secs 2023-03-09T17:36:58.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:36:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:36:58.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:36:58 smithi181 ceph-mon[94189]: pgmap v5594: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 161 KiB/s wr, 13 op/s 2023-03-09T17:36:58.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:36:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9528.339101 secs 2023-03-09T17:36:58.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:36:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:36:58.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9990.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:36:59.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:36:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:36:59.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:36:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:36:59.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:36:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:36:59.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:36:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:36:59.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:36:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:36:59.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:36:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:36:59.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:36:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:36:59.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:36:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:36:59.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:36:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:37:00.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:37:00 smithi122 ceph-mon[93869]: pgmap v5595: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 84 KiB/s wr, 9 op/s 2023-03-09T17:37:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:37:00 smithi181 ceph-mon[94189]: pgmap v5595: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 84 KiB/s wr, 9 op/s 2023-03-09T17:37:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:37:00 smithi038 ceph-mon[92422]: pgmap v5595: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 84 KiB/s wr, 9 op/s 2023-03-09T17:37:01.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:37:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9531.065559 secs 2023-03-09T17:37:01.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:37:01 smithi122 ceph-mon[93869]: pgmap v5596: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 84 KiB/s wr, 8 op/s 2023-03-09T17:37:01.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:37:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9531.065559 secs 2023-03-09T17:37:01.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:37:01 smithi181 ceph-mon[94189]: pgmap v5596: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 84 KiB/s wr, 8 op/s 2023-03-09T17:37:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:37:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9531.065559 secs 2023-03-09T17:37:01.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:37:01 smithi038 ceph-mon[92422]: pgmap v5596: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 84 KiB/s wr, 8 op/s 2023-03-09T17:37:02.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:37:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9533.339203 secs 2023-03-09T17:37:02.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:37:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9533.339203 secs 2023-03-09T17:37:03.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:37:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9533.339203 secs 2023-03-09T17:37:03.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 9995.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:37:03.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:37:03 smithi122 ceph-mon[93869]: pgmap v5597: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 126 KiB/s wr, 13 op/s 2023-03-09T17:37:03.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:37:03 smithi181 ceph-mon[94189]: pgmap v5597: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 126 KiB/s wr, 13 op/s 2023-03-09T17:37:04.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:37:03 smithi038 ceph-mon[92422]: pgmap v5597: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 126 KiB/s wr, 13 op/s 2023-03-09T17:37:05.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:37:05 smithi122 ceph-mon[93869]: pgmap v5598: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 114 KiB/s wr, 20 op/s 2023-03-09T17:37:05.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:37:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9536.065619 secs 2023-03-09T17:37:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:37:05 smithi181 ceph-mon[94189]: pgmap v5598: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 114 KiB/s wr, 20 op/s 2023-03-09T17:37:05.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:37:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9536.065619 secs 2023-03-09T17:37:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:37:05 smithi038 ceph-mon[92422]: pgmap v5598: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 114 KiB/s wr, 20 op/s 2023-03-09T17:37:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:37:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9536.065619 secs 2023-03-09T17:37:08.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:37:07 smithi122 ceph-mon[93869]: pgmap v5599: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 134 KiB/s wr, 25 op/s 2023-03-09T17:37:08.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:37:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9538.339330 secs 2023-03-09T17:37:08.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:37:07 smithi181 ceph-mon[94189]: pgmap v5599: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 134 KiB/s wr, 25 op/s 2023-03-09T17:37:08.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:37:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9538.339330 secs 2023-03-09T17:37:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:37:07 smithi038 ceph-mon[92422]: pgmap v5599: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 134 KiB/s wr, 25 op/s 2023-03-09T17:37:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:37:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9538.339330 secs 2023-03-09T17:37:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10000.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:37:10.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:37:09 smithi122 ceph-mon[93869]: pgmap v5600: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 84 KiB/s wr, 22 op/s 2023-03-09T17:37:10.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:37:09 smithi122 ceph-mon[93869]: osdmap e563: 12 total, 12 up, 12 in 2023-03-09T17:37:10.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:37:09 smithi181 ceph-mon[94189]: pgmap v5600: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 84 KiB/s wr, 22 op/s 2023-03-09T17:37:10.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:37:09 smithi181 ceph-mon[94189]: osdmap e563: 12 total, 12 up, 12 in 2023-03-09T17:37:10.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:37:09 smithi038 ceph-mon[92422]: pgmap v5600: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 84 KiB/s wr, 22 op/s 2023-03-09T17:37:10.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:37:09 smithi038 ceph-mon[92422]: osdmap e563: 12 total, 12 up, 12 in 2023-03-09T17:37:11.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:37:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9541.065678 secs 2023-03-09T17:37:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:37:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9541.065678 secs 2023-03-09T17:37:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:37:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9541.065678 secs 2023-03-09T17:37:12.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:37:11 smithi122 ceph-mon[93869]: pgmap v5602: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 101 KiB/s wr, 26 op/s 2023-03-09T17:37:12.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:37:11 smithi181 ceph-mon[94189]: pgmap v5602: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 101 KiB/s wr, 26 op/s 2023-03-09T17:37:12.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:37:11 smithi038 ceph-mon[92422]: pgmap v5602: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 101 KiB/s wr, 26 op/s 2023-03-09T17:37:13.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:37:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9543.339451 secs 2023-03-09T17:37:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:37:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9543.339451 secs 2023-03-09T17:37:13.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:37:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9543.339451 secs 2023-03-09T17:37:13.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10005.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:37:14.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:37:13 smithi122 ceph-mon[93869]: pgmap v5603: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 78 KiB/s wr, 24 op/s 2023-03-09T17:37:14.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:37:13 smithi181 ceph-mon[94189]: pgmap v5603: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 78 KiB/s wr, 24 op/s 2023-03-09T17:37:14.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:37:13 smithi038 ceph-mon[92422]: pgmap v5603: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 78 KiB/s wr, 24 op/s 2023-03-09T17:37:16.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:37:15 smithi122 ceph-mon[93869]: pgmap v5604: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 53 KiB/s wr, 10 op/s 2023-03-09T17:37:16.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:37:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9546.065737 secs 2023-03-09T17:37:16.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:37:15 smithi122 ceph-mon[93869]: osdmap e564: 12 total, 12 up, 12 in 2023-03-09T17:37:16.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:37:15 smithi181 ceph-mon[94189]: pgmap v5604: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 53 KiB/s wr, 10 op/s 2023-03-09T17:37:16.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:37:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9546.065737 secs 2023-03-09T17:37:16.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:37:15 smithi181 ceph-mon[94189]: osdmap e564: 12 total, 12 up, 12 in 2023-03-09T17:37:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:37:15 smithi038 ceph-mon[92422]: pgmap v5604: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 53 KiB/s wr, 10 op/s 2023-03-09T17:37:16.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:37:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9546.065737 secs 2023-03-09T17:37:16.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:37:15 smithi038 ceph-mon[92422]: osdmap e564: 12 total, 12 up, 12 in 2023-03-09T17:37:18.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:37:17 smithi122 ceph-mon[93869]: pgmap v5606: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 38 KiB/s wr, 9 op/s 2023-03-09T17:37:18.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:37:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9548.339585 secs 2023-03-09T17:37:18.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:37:17 smithi181 ceph-mon[94189]: pgmap v5606: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 38 KiB/s wr, 9 op/s 2023-03-09T17:37:18.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:37:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9548.339585 secs 2023-03-09T17:37:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:37:17 smithi038 ceph-mon[92422]: pgmap v5606: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 38 KiB/s wr, 9 op/s 2023-03-09T17:37:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:37:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9548.339585 secs 2023-03-09T17:37:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10010.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:37:20.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:37:19 smithi122 ceph-mon[93869]: pgmap v5607: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 31 KiB/s wr, 8 op/s 2023-03-09T17:37:20.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:37:19 smithi181 ceph-mon[94189]: pgmap v5607: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 31 KiB/s wr, 8 op/s 2023-03-09T17:37:20.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:37:19 smithi038 ceph-mon[92422]: pgmap v5607: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 31 KiB/s wr, 8 op/s 2023-03-09T17:37:21.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:37:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9551.065796 secs 2023-03-09T17:37:21.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:37:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9551.065796 secs 2023-03-09T17:37:21.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:37:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9551.065796 secs 2023-03-09T17:37:22.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:37:21 smithi122 ceph-mon[93869]: pgmap v5608: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 7 op/s 2023-03-09T17:37:22.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:37:21 smithi181 ceph-mon[94189]: pgmap v5608: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 7 op/s 2023-03-09T17:37:22.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:37:21 smithi038 ceph-mon[92422]: pgmap v5608: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 7 op/s 2023-03-09T17:37:23.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:37:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9553.339722 secs 2023-03-09T17:37:23.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:37:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9553.339722 secs 2023-03-09T17:37:23.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:37:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9553.339722 secs 2023-03-09T17:37:23.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10015.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:37:24.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:37:23 smithi122 ceph-mon[93869]: pgmap v5609: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 31 KiB/s wr, 7 op/s 2023-03-09T17:37:24.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:37:23 smithi181 ceph-mon[94189]: pgmap v5609: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 31 KiB/s wr, 7 op/s 2023-03-09T17:37:24.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:37:23 smithi038 ceph-mon[92422]: pgmap v5609: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 31 KiB/s wr, 7 op/s 2023-03-09T17:37:26.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:37:25 smithi122 ceph-mon[93869]: pgmap v5610: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 31 KiB/s wr, 7 op/s 2023-03-09T17:37:26.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:37:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9556.065858 secs 2023-03-09T17:37:26.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:37:25 smithi181 ceph-mon[94189]: pgmap v5610: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 31 KiB/s wr, 7 op/s 2023-03-09T17:37:26.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:37:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9556.065858 secs 2023-03-09T17:37:26.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:37:25 smithi038 ceph-mon[92422]: pgmap v5610: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 31 KiB/s wr, 7 op/s 2023-03-09T17:37:26.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:37:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9556.065858 secs 2023-03-09T17:37:28.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:37:27 smithi122 ceph-mon[93869]: pgmap v5611: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T17:37:28.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:37:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9558.339846 secs 2023-03-09T17:37:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:37:27 smithi181 ceph-mon[94189]: pgmap v5611: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T17:37:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:37:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9558.339846 secs 2023-03-09T17:37:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:37:27 smithi038 ceph-mon[92422]: pgmap v5611: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T17:37:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:37:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9558.339846 secs 2023-03-09T17:37:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10020.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:37:30.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:37:29 smithi122 ceph-mon[93869]: pgmap v5612: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 6 op/s 2023-03-09T17:37:30.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:37:29 smithi181 ceph-mon[94189]: pgmap v5612: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 6 op/s 2023-03-09T17:37:30.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:37:29 smithi038 ceph-mon[92422]: pgmap v5612: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 6 op/s 2023-03-09T17:37:31.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:37:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9561.065918 secs 2023-03-09T17:37:31.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:37:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9561.065918 secs 2023-03-09T17:37:31.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:37:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9561.065918 secs 2023-03-09T17:37:32.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:37:31 smithi122 ceph-mon[93869]: pgmap v5613: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 6 op/s 2023-03-09T17:37:32.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:37:31 smithi181 ceph-mon[94189]: pgmap v5613: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 6 op/s 2023-03-09T17:37:32.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:37:31 smithi038 ceph-mon[92422]: pgmap v5613: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 6 op/s 2023-03-09T17:37:33.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:37:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9563.339965 secs 2023-03-09T17:37:33.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:37:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9563.339965 secs 2023-03-09T17:37:33.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:37:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9563.339965 secs 2023-03-09T17:37:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10025.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:37:34.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:37:33 smithi122 ceph-mon[93869]: pgmap v5614: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 49 KiB/s wr, 9 op/s 2023-03-09T17:37:34.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:37:33 smithi181 ceph-mon[94189]: pgmap v5614: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 49 KiB/s wr, 9 op/s 2023-03-09T17:37:34.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:37:33 smithi038 ceph-mon[92422]: pgmap v5614: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 49 KiB/s wr, 9 op/s 2023-03-09T17:37:36.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:37:35 smithi122 ceph-mon[93869]: pgmap v5615: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 6 op/s 2023-03-09T17:37:36.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:37:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9566.065983 secs 2023-03-09T17:37:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:37:35 smithi181 ceph-mon[94189]: pgmap v5615: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 6 op/s 2023-03-09T17:37:36.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:37:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9566.065983 secs 2023-03-09T17:37:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:37:35 smithi038 ceph-mon[92422]: pgmap v5615: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 6 op/s 2023-03-09T17:37:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:37:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9566.065983 secs 2023-03-09T17:37:38.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:37:37 smithi122 ceph-mon[93869]: pgmap v5616: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 48 KiB/s wr, 8 op/s 2023-03-09T17:37:38.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:37:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9568.340123 secs 2023-03-09T17:37:38.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:37:37 smithi181 ceph-mon[94189]: pgmap v5616: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 48 KiB/s wr, 8 op/s 2023-03-09T17:37:38.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:37:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9568.340123 secs 2023-03-09T17:37:38.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:37:37 smithi038 ceph-mon[92422]: pgmap v5616: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 48 KiB/s wr, 8 op/s 2023-03-09T17:37:38.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:37:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9568.340123 secs 2023-03-09T17:37:38.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10030.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:37:40.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:37:39 smithi122 ceph-mon[93869]: pgmap v5617: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 48 KiB/s wr, 6 op/s 2023-03-09T17:37:40.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:37:39 smithi181 ceph-mon[94189]: pgmap v5617: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 48 KiB/s wr, 6 op/s 2023-03-09T17:37:40.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:37:39 smithi038 ceph-mon[92422]: pgmap v5617: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 48 KiB/s wr, 6 op/s 2023-03-09T17:37:41.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:37:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9571.066051 secs 2023-03-09T17:37:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:37:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9571.066051 secs 2023-03-09T17:37:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:37:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9571.066051 secs 2023-03-09T17:37:42.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:37:41 smithi122 ceph-mon[93869]: pgmap v5618: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 48 KiB/s wr, 6 op/s 2023-03-09T17:37:42.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:37:41 smithi181 ceph-mon[94189]: pgmap v5618: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 48 KiB/s wr, 6 op/s 2023-03-09T17:37:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:37:41 smithi038 ceph-mon[92422]: pgmap v5618: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 48 KiB/s wr, 6 op/s 2023-03-09T17:37:43.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:37:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9573.340225 secs 2023-03-09T17:37:43.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:37:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9573.340225 secs 2023-03-09T17:37:43.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:37:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9573.340225 secs 2023-03-09T17:37:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10035.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:37:44.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:37:43 smithi122 ceph-mon[93869]: pgmap v5619: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 97 KiB/s wr, 9 op/s 2023-03-09T17:37:44.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:37:43 smithi181 ceph-mon[94189]: pgmap v5619: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 97 KiB/s wr, 9 op/s 2023-03-09T17:37:44.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:37:43 smithi038 ceph-mon[92422]: pgmap v5619: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 97 KiB/s wr, 9 op/s 2023-03-09T17:37:46.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:37:45 smithi122 ceph-mon[93869]: pgmap v5620: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 73 KiB/s wr, 6 op/s 2023-03-09T17:37:46.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:37:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9576.066111 secs 2023-03-09T17:37:46.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:37:45 smithi181 ceph-mon[94189]: pgmap v5620: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 73 KiB/s wr, 6 op/s 2023-03-09T17:37:46.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:37:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9576.066111 secs 2023-03-09T17:37:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:37:45 smithi038 ceph-mon[92422]: pgmap v5620: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 73 KiB/s wr, 6 op/s 2023-03-09T17:37:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:37:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9576.066111 secs 2023-03-09T17:37:48.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:37:47 smithi122 ceph-mon[93869]: pgmap v5621: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 143 KiB/s wr, 10 op/s 2023-03-09T17:37:48.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:37:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9578.340363 secs 2023-03-09T17:37:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:37:47 smithi181 ceph-mon[94189]: pgmap v5621: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 143 KiB/s wr, 10 op/s 2023-03-09T17:37:48.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:37:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9578.340363 secs 2023-03-09T17:37:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:37:47 smithi038 ceph-mon[92422]: pgmap v5621: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 143 KiB/s wr, 10 op/s 2023-03-09T17:37:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:37:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9578.340363 secs 2023-03-09T17:37:48.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10040.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:37:50.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:37:49 smithi122 ceph-mon[93869]: pgmap v5622: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 120 KiB/s wr, 8 op/s 2023-03-09T17:37:50.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:37:49 smithi181 ceph-mon[94189]: pgmap v5622: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 120 KiB/s wr, 8 op/s 2023-03-09T17:37:50.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:37:49 smithi038 ceph-mon[92422]: pgmap v5622: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 120 KiB/s wr, 8 op/s 2023-03-09T17:37:51.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:37:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9581.066175 secs 2023-03-09T17:37:51.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:37:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9581.066175 secs 2023-03-09T17:37:51.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:37:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9581.066175 secs 2023-03-09T17:37:52.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:37:51 smithi122 ceph-mon[93869]: pgmap v5623: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 120 KiB/s wr, 8 op/s 2023-03-09T17:37:52.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:37:51 smithi181 ceph-mon[94189]: pgmap v5623: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 120 KiB/s wr, 8 op/s 2023-03-09T17:37:52.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:37:51 smithi038 ceph-mon[92422]: pgmap v5623: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 120 KiB/s wr, 8 op/s 2023-03-09T17:37:53.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10045.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:37:54.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:37:53 smithi122 ceph-mon[93869]: pgmap v5624: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 170 KiB/s wr, 12 op/s 2023-03-09T17:37:54.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:37:53 smithi181 ceph-mon[94189]: pgmap v5624: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 170 KiB/s wr, 12 op/s 2023-03-09T17:37:54.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:37:53 smithi038 ceph-mon[92422]: pgmap v5624: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 170 KiB/s wr, 12 op/s 2023-03-09T17:37:56.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:37:55 smithi122 ceph-mon[93869]: pgmap v5625: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 151 KiB/s wr, 10 op/s 2023-03-09T17:37:56.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:37:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9586.066236 secs 2023-03-09T17:37:56.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:37:55 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:37:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:37:55 smithi181 ceph-mon[94189]: pgmap v5625: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 151 KiB/s wr, 10 op/s 2023-03-09T17:37:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:37:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9586.066236 secs 2023-03-09T17:37:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:37:55 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:37:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:37:55 smithi038 ceph-mon[92422]: pgmap v5625: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 151 KiB/s wr, 10 op/s 2023-03-09T17:37:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:37:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9586.066236 secs 2023-03-09T17:37:56.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:37:55 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:37:57.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:37:56 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:37:57.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:37:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:37:57.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:37:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:37:57.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:37:56 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:37:57.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:37:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:37:57.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:37:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:37:57.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:37:56 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:37:57.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:37:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:37:57.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:37:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:37:58.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:37:57 smithi122 ceph-mon[93869]: pgmap v5626: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 160 KiB/s wr, 13 op/s 2023-03-09T17:37:58.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:37:57 smithi181 ceph-mon[94189]: pgmap v5626: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 160 KiB/s wr, 13 op/s 2023-03-09T17:37:58.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:37:57 smithi038 ceph-mon[92422]: pgmap v5626: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 160 KiB/s wr, 13 op/s 2023-03-09T17:37:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10050.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:37:59.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:37:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:37:59.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:37:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:37:59.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:37:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:38:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:37:59 smithi038 ceph-mon[92422]: pgmap v5627: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 90 KiB/s wr, 10 op/s 2023-03-09T17:38:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:37:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:38:00.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:37:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:38:00.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:37:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:38:00.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:37:59 smithi122 ceph-mon[93869]: pgmap v5627: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 90 KiB/s wr, 10 op/s 2023-03-09T17:38:00.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:37:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:38:00.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:37:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:38:00.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:37:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:38:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:37:59 smithi181 ceph-mon[94189]: pgmap v5627: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 90 KiB/s wr, 10 op/s 2023-03-09T17:38:00.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:37:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:38:00.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:37:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:38:00.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:37:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:38:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:38:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9591.066297 secs 2023-03-09T17:38:01.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:38:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9591.066297 secs 2023-03-09T17:38:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:38:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9591.066297 secs 2023-03-09T17:38:02.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:38:01 smithi038 ceph-mon[92422]: pgmap v5628: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 151 KiB/s wr, 10 op/s 2023-03-09T17:38:02.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:38:01 smithi122 ceph-mon[93869]: pgmap v5628: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 151 KiB/s wr, 10 op/s 2023-03-09T17:38:02.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:38:01 smithi181 ceph-mon[94189]: pgmap v5628: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 151 KiB/s wr, 10 op/s 2023-03-09T17:38:03.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:38:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9593.340729 secs 2023-03-09T17:38:03.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:38:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9593.340729 secs 2023-03-09T17:38:03.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:38:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9593.340729 secs 2023-03-09T17:38:03.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10055.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:38:04.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:38:04 smithi122 ceph-mon[93869]: pgmap v5629: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 159 KiB/s wr, 14 op/s 2023-03-09T17:38:04.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:38:04 smithi181 ceph-mon[94189]: pgmap v5629: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 159 KiB/s wr, 14 op/s 2023-03-09T17:38:04.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:38:04 smithi038 ceph-mon[92422]: pgmap v5629: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 159 KiB/s wr, 14 op/s 2023-03-09T17:38:05.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:38:05 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:38:05.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:38:05 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:38:05.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:38:05 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:38:05.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:38:05 smithi122 ceph-mon[93869]: osdmap e565: 12 total, 12 up, 12 in 2023-03-09T17:38:05.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:38:05 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:38:05.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:38:05 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:38:05.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:38:05 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:38:05.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:38:05 smithi181 ceph-mon[94189]: osdmap e565: 12 total, 12 up, 12 in 2023-03-09T17:38:05.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:38:05 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:38:05.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:38:05 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:38:05.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:38:05 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:38:05.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:38:05 smithi038 ceph-mon[92422]: osdmap e565: 12 total, 12 up, 12 in 2023-03-09T17:38:06.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:38:06 smithi122 ceph-mon[93869]: pgmap v5631: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 169 KiB/s wr, 27 op/s 2023-03-09T17:38:06.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:38:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9596.066356 secs 2023-03-09T17:38:06.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:38:06 smithi181 ceph-mon[94189]: pgmap v5631: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 169 KiB/s wr, 27 op/s 2023-03-09T17:38:06.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:38:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9596.066356 secs 2023-03-09T17:38:06.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:38:06 smithi038 ceph-mon[92422]: pgmap v5631: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 169 KiB/s wr, 27 op/s 2023-03-09T17:38:06.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:38:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9596.066356 secs 2023-03-09T17:38:08.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:38:08 smithi122 ceph-mon[93869]: pgmap v5632: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 194 KiB/s wr, 29 op/s 2023-03-09T17:38:08.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:38:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9598.340839 secs 2023-03-09T17:38:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:38:08 smithi181 ceph-mon[94189]: pgmap v5632: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 194 KiB/s wr, 29 op/s 2023-03-09T17:38:08.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:38:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9598.340839 secs 2023-03-09T17:38:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:38:08 smithi038 ceph-mon[92422]: pgmap v5632: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 194 KiB/s wr, 29 op/s 2023-03-09T17:38:08.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:38:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9598.340839 secs 2023-03-09T17:38:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10060.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:38:10.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:38:10 smithi122 ceph-mon[93869]: pgmap v5633: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 194 KiB/s wr, 28 op/s 2023-03-09T17:38:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:38:10 smithi181 ceph-mon[94189]: pgmap v5633: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 194 KiB/s wr, 28 op/s 2023-03-09T17:38:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:38:10 smithi038 ceph-mon[92422]: pgmap v5633: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 194 KiB/s wr, 28 op/s 2023-03-09T17:38:11.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:38:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9601.066416 secs 2023-03-09T17:38:11.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:38:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9601.066416 secs 2023-03-09T17:38:11.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:38:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9601.066416 secs 2023-03-09T17:38:12.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:38:12 smithi122 ceph-mon[93869]: pgmap v5634: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 148 KiB/s wr, 28 op/s 2023-03-09T17:38:12.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:38:12 smithi181 ceph-mon[94189]: pgmap v5634: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 148 KiB/s wr, 28 op/s 2023-03-09T17:38:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:38:12 smithi038 ceph-mon[92422]: pgmap v5634: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 148 KiB/s wr, 28 op/s 2023-03-09T17:38:13.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:38:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9603.340974 secs 2023-03-09T17:38:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:38:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9603.340974 secs 2023-03-09T17:38:13.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:38:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9603.340974 secs 2023-03-09T17:38:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10065.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:38:14.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:38:14 smithi122 ceph-mon[93869]: pgmap v5635: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 140 KiB/s wr, 26 op/s 2023-03-09T17:38:14.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:38:14 smithi181 ceph-mon[94189]: pgmap v5635: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 140 KiB/s wr, 26 op/s 2023-03-09T17:38:14.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:38:14 smithi038 ceph-mon[92422]: pgmap v5635: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 140 KiB/s wr, 26 op/s 2023-03-09T17:38:16.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:38:16 smithi122 ceph-mon[93869]: pgmap v5636: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 62 KiB/s wr, 10 op/s 2023-03-09T17:38:16.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:38:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9606.066475 secs 2023-03-09T17:38:16.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:38:16 smithi122 ceph-mon[93869]: osdmap e566: 12 total, 12 up, 12 in 2023-03-09T17:38:16.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:38:16 smithi181 ceph-mon[94189]: pgmap v5636: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 62 KiB/s wr, 10 op/s 2023-03-09T17:38:16.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:38:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9606.066475 secs 2023-03-09T17:38:16.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:38:16 smithi181 ceph-mon[94189]: osdmap e566: 12 total, 12 up, 12 in 2023-03-09T17:38:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:38:16 smithi038 ceph-mon[92422]: pgmap v5636: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 62 KiB/s wr, 10 op/s 2023-03-09T17:38:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:38:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9606.066475 secs 2023-03-09T17:38:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:38:16 smithi038 ceph-mon[92422]: osdmap e566: 12 total, 12 up, 12 in 2023-03-09T17:38:18.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:38:18 smithi122 ceph-mon[93869]: pgmap v5638: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 39 KiB/s wr, 8 op/s 2023-03-09T17:38:18.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:38:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9608.341121 secs 2023-03-09T17:38:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:38:18 smithi181 ceph-mon[94189]: pgmap v5638: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 39 KiB/s wr, 8 op/s 2023-03-09T17:38:18.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:38:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9608.341121 secs 2023-03-09T17:38:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:38:18 smithi038 ceph-mon[92422]: pgmap v5638: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 39 KiB/s wr, 8 op/s 2023-03-09T17:38:18.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:38:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9608.341121 secs 2023-03-09T17:38:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10070.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:38:20.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:38:20 smithi122 ceph-mon[93869]: pgmap v5639: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 39 KiB/s wr, 7 op/s 2023-03-09T17:38:20.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:38:20 smithi181 ceph-mon[94189]: pgmap v5639: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 39 KiB/s wr, 7 op/s 2023-03-09T17:38:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:38:20 smithi038 ceph-mon[92422]: pgmap v5639: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 39 KiB/s wr, 7 op/s 2023-03-09T17:38:21.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:38:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9611.066555 secs 2023-03-09T17:38:21.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:38:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9611.066555 secs 2023-03-09T17:38:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:38:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9611.066555 secs 2023-03-09T17:38:22.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:38:22 smithi122 ceph-mon[93869]: pgmap v5640: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 29 KiB/s wr, 7 op/s 2023-03-09T17:38:22.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:38:22 smithi181 ceph-mon[94189]: pgmap v5640: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 29 KiB/s wr, 7 op/s 2023-03-09T17:38:22.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:38:22 smithi038 ceph-mon[92422]: pgmap v5640: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 29 KiB/s wr, 7 op/s 2023-03-09T17:38:23.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:38:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9613.341241 secs 2023-03-09T17:38:23.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:38:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9613.341241 secs 2023-03-09T17:38:23.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:38:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9613.341241 secs 2023-03-09T17:38:23.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10075.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:38:24.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:38:24 smithi122 ceph-mon[93869]: pgmap v5641: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T17:38:24.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:38:24 smithi181 ceph-mon[94189]: pgmap v5641: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T17:38:24.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:38:24 smithi038 ceph-mon[92422]: pgmap v5641: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T17:38:26.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:38:26 smithi122 ceph-mon[93869]: pgmap v5642: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 57 KiB/s wr, 8 op/s 2023-03-09T17:38:26.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:38:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9616.066615 secs 2023-03-09T17:38:26.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:38:26 smithi181 ceph-mon[94189]: pgmap v5642: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 57 KiB/s wr, 8 op/s 2023-03-09T17:38:26.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:38:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9616.066615 secs 2023-03-09T17:38:26.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:38:26 smithi038 ceph-mon[92422]: pgmap v5642: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 57 KiB/s wr, 8 op/s 2023-03-09T17:38:26.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:38:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9616.066615 secs 2023-03-09T17:38:28.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:38:28 smithi122 ceph-mon[93869]: pgmap v5643: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T17:38:28.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:38:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9618.341342 secs 2023-03-09T17:38:28.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:38:28 smithi181 ceph-mon[94189]: pgmap v5643: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T17:38:28.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:38:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9618.341342 secs 2023-03-09T17:38:28.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:38:28 smithi038 ceph-mon[92422]: pgmap v5643: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T17:38:28.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:38:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9618.341342 secs 2023-03-09T17:38:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10080.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:38:30.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:38:30 smithi122 ceph-mon[93869]: pgmap v5644: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 40 KiB/s wr, 6 op/s 2023-03-09T17:38:30.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:38:30 smithi181 ceph-mon[94189]: pgmap v5644: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 40 KiB/s wr, 6 op/s 2023-03-09T17:38:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:38:30 smithi038 ceph-mon[92422]: pgmap v5644: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 40 KiB/s wr, 6 op/s 2023-03-09T17:38:31.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:38:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9621.066675 secs 2023-03-09T17:38:31.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:38:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9621.066675 secs 2023-03-09T17:38:31.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:38:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9621.066675 secs 2023-03-09T17:38:32.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:38:32 smithi122 ceph-mon[93869]: pgmap v5645: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 40 KiB/s wr, 7 op/s 2023-03-09T17:38:32.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:38:32 smithi181 ceph-mon[94189]: pgmap v5645: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 40 KiB/s wr, 7 op/s 2023-03-09T17:38:32.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:38:32 smithi038 ceph-mon[92422]: pgmap v5645: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 40 KiB/s wr, 7 op/s 2023-03-09T17:38:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10085.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:38:34.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:38:34 smithi122 ceph-mon[93869]: pgmap v5646: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 26 KiB/s wr, 8 op/s 2023-03-09T17:38:34.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:38:34 smithi181 ceph-mon[94189]: pgmap v5646: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 26 KiB/s wr, 8 op/s 2023-03-09T17:38:34.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:38:34 smithi038 ceph-mon[92422]: pgmap v5646: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 26 KiB/s wr, 8 op/s 2023-03-09T17:38:36.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:38:36 smithi122 ceph-mon[93869]: pgmap v5647: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 6 op/s 2023-03-09T17:38:36.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:38:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9626.066734 secs 2023-03-09T17:38:36.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:38:36 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:38:36.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:38:36 smithi181 ceph-mon[94189]: pgmap v5647: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 6 op/s 2023-03-09T17:38:36.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:38:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9626.066734 secs 2023-03-09T17:38:36.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:38:36 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:38:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:38:36 smithi038 ceph-mon[92422]: pgmap v5647: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 6 op/s 2023-03-09T17:38:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:38:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9626.066734 secs 2023-03-09T17:38:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:38:36 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:38:37.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:38:37 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:38:37.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:38:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:38:37.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:38:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:38:37.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:38:37 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:38:37.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:38:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:38:37.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:38:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:38:37.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:38:37 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:38:37.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:38:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:38:37.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:38:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:38:38.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:38:38 smithi122 ceph-mon[93869]: pgmap v5648: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 853 B/s wr, 8 op/s 2023-03-09T17:38:38.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:38:38 smithi181 ceph-mon[94189]: pgmap v5648: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 853 B/s wr, 8 op/s 2023-03-09T17:38:38.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:38:38 smithi038 ceph-mon[92422]: pgmap v5648: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 853 B/s wr, 8 op/s 2023-03-09T17:38:38.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10090.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:38:40.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:38:40 smithi122 ceph-mon[93869]: pgmap v5649: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 853 B/s wr, 6 op/s 2023-03-09T17:38:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:38:40 smithi181 ceph-mon[94189]: pgmap v5649: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 853 B/s wr, 6 op/s 2023-03-09T17:38:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:38:40 smithi038 ceph-mon[92422]: pgmap v5649: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 853 B/s wr, 6 op/s 2023-03-09T17:38:41.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:38:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9631.066795 secs 2023-03-09T17:38:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:38:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9631.066795 secs 2023-03-09T17:38:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:38:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9631.066795 secs 2023-03-09T17:38:42.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:38:42 smithi122 ceph-mon[93869]: pgmap v5650: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 74 KiB/s wr, 7 op/s 2023-03-09T17:38:42.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:38:42 smithi181 ceph-mon[94189]: pgmap v5650: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 74 KiB/s wr, 7 op/s 2023-03-09T17:38:42.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:38:42 smithi038 ceph-mon[92422]: pgmap v5650: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 74 KiB/s wr, 7 op/s 2023-03-09T17:38:43.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10095.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:38:44.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:38:44 smithi122 ceph-mon[93869]: pgmap v5651: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 90 KiB/s wr, 9 op/s 2023-03-09T17:38:44.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:38:44 smithi181 ceph-mon[94189]: pgmap v5651: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 90 KiB/s wr, 9 op/s 2023-03-09T17:38:44.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:38:44 smithi038 ceph-mon[92422]: pgmap v5651: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 90 KiB/s wr, 9 op/s 2023-03-09T17:38:46.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:38:46 smithi122 ceph-mon[93869]: pgmap v5652: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 118 KiB/s wr, 8 op/s 2023-03-09T17:38:46.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:38:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9636.066856 secs 2023-03-09T17:38:46.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:38:46 smithi181 ceph-mon[94189]: pgmap v5652: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 118 KiB/s wr, 8 op/s 2023-03-09T17:38:46.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:38:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9636.066856 secs 2023-03-09T17:38:46.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:38:46 smithi038 ceph-mon[92422]: pgmap v5652: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 118 KiB/s wr, 8 op/s 2023-03-09T17:38:46.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:38:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9636.066856 secs 2023-03-09T17:38:48.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:38:48 smithi122 ceph-mon[93869]: pgmap v5653: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 119 KiB/s wr, 10 op/s 2023-03-09T17:38:48.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:38:48 smithi181 ceph-mon[94189]: pgmap v5653: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 119 KiB/s wr, 10 op/s 2023-03-09T17:38:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:38:48 smithi038 ceph-mon[92422]: pgmap v5653: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 119 KiB/s wr, 10 op/s 2023-03-09T17:38:48.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10100.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:38:50.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:38:50 smithi122 ceph-mon[93869]: pgmap v5654: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 119 KiB/s wr, 8 op/s 2023-03-09T17:38:50.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:38:50 smithi181 ceph-mon[94189]: pgmap v5654: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 119 KiB/s wr, 8 op/s 2023-03-09T17:38:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:38:50 smithi038 ceph-mon[92422]: pgmap v5654: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 119 KiB/s wr, 8 op/s 2023-03-09T17:38:51.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:38:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9641.066915 secs 2023-03-09T17:38:51.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:38:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9641.066915 secs 2023-03-09T17:38:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:38:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9641.066915 secs 2023-03-09T17:38:52.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:38:52 smithi122 ceph-mon[93869]: pgmap v5655: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 168 KiB/s wr, 9 op/s 2023-03-09T17:38:52.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:38:52 smithi181 ceph-mon[94189]: pgmap v5655: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 168 KiB/s wr, 9 op/s 2023-03-09T17:38:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:38:52 smithi038 ceph-mon[92422]: pgmap v5655: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 168 KiB/s wr, 9 op/s 2023-03-09T17:38:53.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10105.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:38:54.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:38:54 smithi122 ceph-mon[93869]: pgmap v5656: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 134 KiB/s wr, 12 op/s 2023-03-09T17:38:54.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:38:54 smithi181 ceph-mon[94189]: pgmap v5656: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 134 KiB/s wr, 12 op/s 2023-03-09T17:38:54.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:38:54 smithi038 ceph-mon[92422]: pgmap v5656: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 134 KiB/s wr, 12 op/s 2023-03-09T17:38:56.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:38:56 smithi122 ceph-mon[93869]: pgmap v5657: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 118 KiB/s wr, 9 op/s 2023-03-09T17:38:56.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:38:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9646.066999 secs 2023-03-09T17:38:56.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:38:56 smithi181 ceph-mon[94189]: pgmap v5657: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 118 KiB/s wr, 9 op/s 2023-03-09T17:38:56.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:38:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9646.066999 secs 2023-03-09T17:38:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:38:56 smithi038 ceph-mon[92422]: pgmap v5657: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 118 KiB/s wr, 9 op/s 2023-03-09T17:38:56.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:38:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9646.066999 secs 2023-03-09T17:38:58.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:38:58 smithi122 ceph-mon[93869]: pgmap v5658: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 176 KiB/s wr, 12 op/s 2023-03-09T17:38:58.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:38:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9648.342128 secs 2023-03-09T17:38:58.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:38:58 smithi181 ceph-mon[94189]: pgmap v5658: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 176 KiB/s wr, 12 op/s 2023-03-09T17:38:58.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:38:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9648.342128 secs 2023-03-09T17:38:58.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:38:58 smithi038 ceph-mon[92422]: pgmap v5658: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 176 KiB/s wr, 12 op/s 2023-03-09T17:38:58.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:38:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9648.342128 secs 2023-03-09T17:38:58.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10110.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:39:00.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:39:00 smithi122 ceph-mon[93869]: pgmap v5659: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 176 KiB/s wr, 10 op/s 2023-03-09T17:39:00.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:39:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:39:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:39:00 smithi181 ceph-mon[94189]: pgmap v5659: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 176 KiB/s wr, 10 op/s 2023-03-09T17:39:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:39:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:39:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:39:00 smithi038 ceph-mon[92422]: pgmap v5659: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 176 KiB/s wr, 10 op/s 2023-03-09T17:39:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:39:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:39:01.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:39:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9651.067064 secs 2023-03-09T17:39:01.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:39:01 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:39:01.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:39:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:39:01.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:39:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:39:01.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:39:01 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:39:01.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:39:01 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:39:01.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:39:01 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:39:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:39:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9651.067064 secs 2023-03-09T17:39:01.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:39:01 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:39:01.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:39:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:39:01.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:39:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:39:01.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:39:01 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:39:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:39:01 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:39:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:39:01 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:39:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:39:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9651.067064 secs 2023-03-09T17:39:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:39:01 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:39:01.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:39:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:39:01.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:39:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:39:01.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:39:01 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:39:01.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:39:01 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:39:01.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:39:01 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:39:02.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:39:02 smithi122 ceph-mon[93869]: pgmap v5660: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 176 KiB/s wr, 9 op/s 2023-03-09T17:39:02.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:39:02 smithi181 ceph-mon[94189]: pgmap v5660: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 176 KiB/s wr, 9 op/s 2023-03-09T17:39:02.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:39:02 smithi038 ceph-mon[92422]: pgmap v5660: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 176 KiB/s wr, 9 op/s 2023-03-09T17:39:03.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:39:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9653.342239 secs 2023-03-09T17:39:03.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:39:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9653.342239 secs 2023-03-09T17:39:03.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:39:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9653.342239 secs 2023-03-09T17:39:03.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10115.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:39:04.504 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:39:04 smithi038 ceph-mon[92422]: pgmap v5661: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 140 KiB/s wr, 12 op/s 2023-03-09T17:39:04.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:39:04 smithi122 ceph-mon[93869]: pgmap v5661: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 140 KiB/s wr, 12 op/s 2023-03-09T17:39:04.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:39:04 smithi181 ceph-mon[94189]: pgmap v5661: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 140 KiB/s wr, 12 op/s 2023-03-09T17:39:05.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:39:05 smithi122 ceph-mon[93869]: osdmap e567: 12 total, 12 up, 12 in 2023-03-09T17:39:05.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:39:05 smithi181 ceph-mon[94189]: osdmap e567: 12 total, 12 up, 12 in 2023-03-09T17:39:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:39:05 smithi038 ceph-mon[92422]: osdmap e567: 12 total, 12 up, 12 in 2023-03-09T17:39:06.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:39:06 smithi122 ceph-mon[93869]: pgmap v5663: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 147 KiB/s wr, 23 op/s 2023-03-09T17:39:06.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:39:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9656.069017 secs 2023-03-09T17:39:06.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:39:06 smithi181 ceph-mon[94189]: pgmap v5663: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 147 KiB/s wr, 23 op/s 2023-03-09T17:39:06.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:39:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9656.069017 secs 2023-03-09T17:39:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:39:06 smithi038 ceph-mon[92422]: pgmap v5663: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 147 KiB/s wr, 23 op/s 2023-03-09T17:39:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:39:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9656.069017 secs 2023-03-09T17:39:07.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:39:07 smithi122 ceph-mon[93869]: pgmap v5664: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 154 KiB/s wr, 26 op/s 2023-03-09T17:39:07.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:39:07 smithi181 ceph-mon[94189]: pgmap v5664: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 154 KiB/s wr, 26 op/s 2023-03-09T17:39:07.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:39:07 smithi038 ceph-mon[92422]: pgmap v5664: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 154 KiB/s wr, 26 op/s 2023-03-09T17:39:08.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:39:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9658.342355 secs 2023-03-09T17:39:08.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:39:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9658.342355 secs 2023-03-09T17:39:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:39:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9658.342355 secs 2023-03-09T17:39:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10120.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:39:09.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:39:09 smithi122 ceph-mon[93869]: pgmap v5665: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 154 KiB/s wr, 26 op/s 2023-03-09T17:39:09.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:39:09 smithi181 ceph-mon[94189]: pgmap v5665: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 154 KiB/s wr, 26 op/s 2023-03-09T17:39:09.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:39:09 smithi038 ceph-mon[92422]: pgmap v5665: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 154 KiB/s wr, 26 op/s 2023-03-09T17:39:10.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:39:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9661.069078 secs 2023-03-09T17:39:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:39:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9661.069078 secs 2023-03-09T17:39:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:39:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9661.069078 secs 2023-03-09T17:39:12.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:39:11 smithi122 ceph-mon[93869]: pgmap v5666: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 154 KiB/s wr, 26 op/s 2023-03-09T17:39:12.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:39:11 smithi181 ceph-mon[94189]: pgmap v5666: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 154 KiB/s wr, 26 op/s 2023-03-09T17:39:12.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:39:11 smithi038 ceph-mon[92422]: pgmap v5666: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 154 KiB/s wr, 26 op/s 2023-03-09T17:39:13.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:39:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9663.342481 secs 2023-03-09T17:39:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:39:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9663.342481 secs 2023-03-09T17:39:13.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:39:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9663.342481 secs 2023-03-09T17:39:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10125.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:39:14.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:39:13 smithi122 ceph-mon[93869]: pgmap v5667: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 138 KiB/s wr, 26 op/s 2023-03-09T17:39:14.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:39:13 smithi181 ceph-mon[94189]: pgmap v5667: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 138 KiB/s wr, 26 op/s 2023-03-09T17:39:14.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:39:13 smithi038 ceph-mon[92422]: pgmap v5667: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 138 KiB/s wr, 26 op/s 2023-03-09T17:39:16.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:39:15 smithi122 ceph-mon[93869]: pgmap v5668: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 109 KiB/s wr, 10 op/s 2023-03-09T17:39:16.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:39:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9666.069131 secs 2023-03-09T17:39:16.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:39:15 smithi122 ceph-mon[93869]: osdmap e568: 12 total, 12 up, 12 in 2023-03-09T17:39:16.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:39:15 smithi181 ceph-mon[94189]: pgmap v5668: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 109 KiB/s wr, 10 op/s 2023-03-09T17:39:16.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:39:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9666.069131 secs 2023-03-09T17:39:16.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:39:15 smithi181 ceph-mon[94189]: osdmap e568: 12 total, 12 up, 12 in 2023-03-09T17:39:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:39:15 smithi038 ceph-mon[92422]: pgmap v5668: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 109 KiB/s wr, 10 op/s 2023-03-09T17:39:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:39:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9666.069131 secs 2023-03-09T17:39:16.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:39:15 smithi038 ceph-mon[92422]: osdmap e568: 12 total, 12 up, 12 in 2023-03-09T17:39:18.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:39:17 smithi122 ceph-mon[93869]: pgmap v5670: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 7 op/s 2023-03-09T17:39:18.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:39:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9668.342623 secs 2023-03-09T17:39:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:39:17 smithi181 ceph-mon[94189]: pgmap v5670: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 7 op/s 2023-03-09T17:39:18.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:39:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9668.342623 secs 2023-03-09T17:39:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:39:17 smithi038 ceph-mon[92422]: pgmap v5670: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 7 op/s 2023-03-09T17:39:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:39:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9668.342623 secs 2023-03-09T17:39:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10130.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:39:20.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:39:19 smithi122 ceph-mon[93869]: pgmap v5671: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 7 op/s 2023-03-09T17:39:20.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:39:19 smithi181 ceph-mon[94189]: pgmap v5671: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 7 op/s 2023-03-09T17:39:20.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:39:19 smithi038 ceph-mon[92422]: pgmap v5671: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 7 op/s 2023-03-09T17:39:21.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:39:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9671.069192 secs 2023-03-09T17:39:21.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:39:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9671.069192 secs 2023-03-09T17:39:21.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:39:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9671.069192 secs 2023-03-09T17:39:22.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:39:21 smithi122 ceph-mon[93869]: pgmap v5672: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 7 op/s 2023-03-09T17:39:22.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:39:21 smithi181 ceph-mon[94189]: pgmap v5672: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 7 op/s 2023-03-09T17:39:22.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:39:21 smithi038 ceph-mon[92422]: pgmap v5672: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 7 op/s 2023-03-09T17:39:23.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:39:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9673.342748 secs 2023-03-09T17:39:23.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:39:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9673.342748 secs 2023-03-09T17:39:23.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:39:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9673.342748 secs 2023-03-09T17:39:23.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10135.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:39:24.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:39:23 smithi122 ceph-mon[93869]: pgmap v5673: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 41 KiB/s wr, 7 op/s 2023-03-09T17:39:24.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:39:23 smithi181 ceph-mon[94189]: pgmap v5673: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 41 KiB/s wr, 7 op/s 2023-03-09T17:39:24.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:39:23 smithi038 ceph-mon[92422]: pgmap v5673: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 41 KiB/s wr, 7 op/s 2023-03-09T17:39:26.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:39:25 smithi122 ceph-mon[93869]: pgmap v5674: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 41 KiB/s wr, 8 op/s 2023-03-09T17:39:26.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:39:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9676.069271 secs 2023-03-09T17:39:26.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:39:25 smithi181 ceph-mon[94189]: pgmap v5674: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 41 KiB/s wr, 8 op/s 2023-03-09T17:39:26.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:39:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9676.069271 secs 2023-03-09T17:39:26.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:39:25 smithi038 ceph-mon[92422]: pgmap v5674: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 41 KiB/s wr, 8 op/s 2023-03-09T17:39:26.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:39:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9676.069271 secs 2023-03-09T17:39:28.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:39:27 smithi122 ceph-mon[93869]: pgmap v5675: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 29 KiB/s wr, 7 op/s 2023-03-09T17:39:28.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:39:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9678.342853 secs 2023-03-09T17:39:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:39:27 smithi181 ceph-mon[94189]: pgmap v5675: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 29 KiB/s wr, 7 op/s 2023-03-09T17:39:28.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:39:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9678.342853 secs 2023-03-09T17:39:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:39:27 smithi038 ceph-mon[92422]: pgmap v5675: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 29 KiB/s wr, 7 op/s 2023-03-09T17:39:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:39:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9678.342853 secs 2023-03-09T17:39:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10140.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:39:30.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:39:29 smithi122 ceph-mon[93869]: pgmap v5676: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 26 KiB/s wr, 6 op/s 2023-03-09T17:39:30.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:39:29 smithi181 ceph-mon[94189]: pgmap v5676: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 26 KiB/s wr, 6 op/s 2023-03-09T17:39:30.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:39:29 smithi038 ceph-mon[92422]: pgmap v5676: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 26 KiB/s wr, 6 op/s 2023-03-09T17:39:31.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:39:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9681.069350 secs 2023-03-09T17:39:31.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:39:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9681.069350 secs 2023-03-09T17:39:31.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:39:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9681.069350 secs 2023-03-09T17:39:32.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:39:31 smithi122 ceph-mon[93869]: pgmap v5677: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 26 KiB/s wr, 6 op/s 2023-03-09T17:39:32.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:39:31 smithi181 ceph-mon[94189]: pgmap v5677: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 26 KiB/s wr, 6 op/s 2023-03-09T17:39:32.269 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:39:31 smithi038 ceph-mon[92422]: pgmap v5677: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 26 KiB/s wr, 6 op/s 2023-03-09T17:39:33.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:39:32 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 9683.342920 secs 2023-03-09T17:39:33.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:39:32 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 9683.342920 secs 2023-03-09T17:39:33.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:39:32 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 9683.342920 secs 2023-03-09T17:39:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10145.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:39:34.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:39:33 smithi122 ceph-mon[93869]: pgmap v5678: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 73 KiB/s wr, 10 op/s 2023-03-09T17:39:34.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:39:33 smithi181 ceph-mon[94189]: pgmap v5678: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 73 KiB/s wr, 10 op/s 2023-03-09T17:39:34.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:39:33 smithi038 ceph-mon[92422]: pgmap v5678: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 73 KiB/s wr, 10 op/s 2023-03-09T17:39:36.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:39:35 smithi122 ceph-mon[93869]: pgmap v5679: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 48 KiB/s wr, 7 op/s 2023-03-09T17:39:36.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:39:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9686.069435 secs 2023-03-09T17:39:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:39:35 smithi181 ceph-mon[94189]: pgmap v5679: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 48 KiB/s wr, 7 op/s 2023-03-09T17:39:36.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:39:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9686.069435 secs 2023-03-09T17:39:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:39:35 smithi038 ceph-mon[92422]: pgmap v5679: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 48 KiB/s wr, 7 op/s 2023-03-09T17:39:36.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:39:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9686.069435 secs 2023-03-09T17:39:37.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:39:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:39:37.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:39:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:39:37.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:39:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:39:37.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:39:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:39:37.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:39:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:39:37.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:39:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:39:38.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:39:37 smithi122 ceph-mon[93869]: pgmap v5680: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 53 KiB/s wr, 9 op/s 2023-03-09T17:39:38.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:39:37 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 9688.343026 secs 2023-03-09T17:39:38.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:39:37 smithi181 ceph-mon[94189]: pgmap v5680: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 53 KiB/s wr, 9 op/s 2023-03-09T17:39:38.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:39:37 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 9688.343026 secs 2023-03-09T17:39:38.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:39:37 smithi038 ceph-mon[92422]: pgmap v5680: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 53 KiB/s wr, 9 op/s 2023-03-09T17:39:38.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:39:37 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 9688.343026 secs 2023-03-09T17:39:38.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10150.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:39:40.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:39:39 smithi122 ceph-mon[93869]: pgmap v5681: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 7 op/s 2023-03-09T17:39:40.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:39:39 smithi181 ceph-mon[94189]: pgmap v5681: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 7 op/s 2023-03-09T17:39:40.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:39:39 smithi038 ceph-mon[92422]: pgmap v5681: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 7 op/s 2023-03-09T17:39:41.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:39:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9691.069496 secs 2023-03-09T17:39:41.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:39:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9691.069496 secs 2023-03-09T17:39:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:39:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9691.069496 secs 2023-03-09T17:39:42.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:39:41 smithi122 ceph-mon[93869]: pgmap v5682: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 6 op/s 2023-03-09T17:39:42.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:39:41 smithi181 ceph-mon[94189]: pgmap v5682: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 6 op/s 2023-03-09T17:39:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:39:41 smithi038 ceph-mon[92422]: pgmap v5682: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 6 op/s 2023-03-09T17:39:43.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:39:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9693.343120 secs 2023-03-09T17:39:43.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:39:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9693.343120 secs 2023-03-09T17:39:43.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:39:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9693.343120 secs 2023-03-09T17:39:43.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10155.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:39:44.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:39:43 smithi122 ceph-mon[93869]: pgmap v5683: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 103 KiB/s wr, 10 op/s 2023-03-09T17:39:44.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:39:43 smithi181 ceph-mon[94189]: pgmap v5683: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 103 KiB/s wr, 10 op/s 2023-03-09T17:39:44.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:39:43 smithi038 ceph-mon[92422]: pgmap v5683: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 103 KiB/s wr, 10 op/s 2023-03-09T17:39:45.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:39:44 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:39:45.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:39:44 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:39:45.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:39:44 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:39:45.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:39:44 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:39:45.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:39:44 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:39:45.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:39:44 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:39:46.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:39:45 smithi181 ceph-mon[94189]: pgmap v5684: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 56 KiB/s wr, 7 op/s 2023-03-09T17:39:46.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:39:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9696.069556 secs 2023-03-09T17:39:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:39:45 smithi038 ceph-mon[92422]: pgmap v5684: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 56 KiB/s wr, 7 op/s 2023-03-09T17:39:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:39:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9696.069556 secs 2023-03-09T17:39:46.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:39:45 smithi122 ceph-mon[93869]: pgmap v5684: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 56 KiB/s wr, 7 op/s 2023-03-09T17:39:46.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:39:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9696.069556 secs 2023-03-09T17:39:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:39:47 smithi038 ceph-mon[92422]: pgmap v5685: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 101 KiB/s wr, 11 op/s 2023-03-09T17:39:48.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:39:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9698.343237 secs 2023-03-09T17:39:48.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:39:47 smithi122 ceph-mon[93869]: pgmap v5685: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 101 KiB/s wr, 11 op/s 2023-03-09T17:39:48.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:39:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9698.343237 secs 2023-03-09T17:39:48.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:39:47 smithi181 ceph-mon[94189]: pgmap v5685: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 101 KiB/s wr, 11 op/s 2023-03-09T17:39:48.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:39:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9698.343237 secs 2023-03-09T17:39:48.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10160.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:39:50.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:39:49 smithi038 ceph-mon[92422]: pgmap v5686: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 97 KiB/s wr, 9 op/s 2023-03-09T17:39:50.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:39:49 smithi122 ceph-mon[93869]: pgmap v5686: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 97 KiB/s wr, 9 op/s 2023-03-09T17:39:50.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:39:49 smithi181 ceph-mon[94189]: pgmap v5686: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 97 KiB/s wr, 9 op/s 2023-03-09T17:39:51.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:39:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9701.069642 secs 2023-03-09T17:39:51.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:39:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9701.069642 secs 2023-03-09T17:39:51.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:39:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9701.069642 secs 2023-03-09T17:39:52.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:39:51 smithi038 ceph-mon[92422]: pgmap v5687: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 96 KiB/s wr, 8 op/s 2023-03-09T17:39:52.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:39:51 smithi122 ceph-mon[93869]: pgmap v5687: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 96 KiB/s wr, 8 op/s 2023-03-09T17:39:52.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:39:51 smithi181 ceph-mon[94189]: pgmap v5687: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 96 KiB/s wr, 8 op/s 2023-03-09T17:39:53.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:39:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9703.343353 secs 2023-03-09T17:39:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:39:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9703.343353 secs 2023-03-09T17:39:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:39:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9703.343353 secs 2023-03-09T17:39:53.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10165.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:39:54.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:39:54 smithi122 ceph-mon[93869]: pgmap v5688: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 167 KiB/s wr, 13 op/s 2023-03-09T17:39:54.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:39:54 smithi181 ceph-mon[94189]: pgmap v5688: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 167 KiB/s wr, 13 op/s 2023-03-09T17:39:54.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:39:54 smithi038 ceph-mon[92422]: pgmap v5688: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 167 KiB/s wr, 13 op/s 2023-03-09T17:39:56.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:39:56 smithi122 ceph-mon[93869]: pgmap v5689: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 116 KiB/s wr, 10 op/s 2023-03-09T17:39:56.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:39:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9706.069701 secs 2023-03-09T17:39:56.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:39:56 smithi181 ceph-mon[94189]: pgmap v5689: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 116 KiB/s wr, 10 op/s 2023-03-09T17:39:56.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:39:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9706.069701 secs 2023-03-09T17:39:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:39:56 smithi038 ceph-mon[92422]: pgmap v5689: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 116 KiB/s wr, 10 op/s 2023-03-09T17:39:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:39:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9706.069701 secs 2023-03-09T17:39:58.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:39:58 smithi122 ceph-mon[93869]: pgmap v5690: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 135 KiB/s wr, 13 op/s 2023-03-09T17:39:58.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:39:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9708.343440 secs 2023-03-09T17:39:58.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:39:58 smithi181 ceph-mon[94189]: pgmap v5690: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 135 KiB/s wr, 13 op/s 2023-03-09T17:39:58.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:39:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9708.343440 secs 2023-03-09T17:39:58.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:39:58 smithi038 ceph-mon[92422]: pgmap v5690: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 135 KiB/s wr, 13 op/s 2023-03-09T17:39:58.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:39:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9708.343440 secs 2023-03-09T17:39:58.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10170.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:40:00.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:40:00 smithi122 ceph-mon[93869]: pgmap v5691: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 90 KiB/s wr, 10 op/s 2023-03-09T17:40:00.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:40:00 smithi122 ceph-mon[93869]: overall HEALTH_WARN 2 MDSs report slow requests 2023-03-09T17:40:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:40:00 smithi181 ceph-mon[94189]: pgmap v5691: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 90 KiB/s wr, 10 op/s 2023-03-09T17:40:00.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:40:00 smithi181 ceph-mon[94189]: overall HEALTH_WARN 2 MDSs report slow requests 2023-03-09T17:40:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:40:00 smithi038 ceph-mon[92422]: pgmap v5691: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 90 KiB/s wr, 10 op/s 2023-03-09T17:40:00.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:40:00 smithi038 ceph-mon[92422]: overall HEALTH_WARN 2 MDSs report slow requests 2023-03-09T17:40:01.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:40:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9711.069761 secs 2023-03-09T17:40:01.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:40:01 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:40:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:40:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9711.069761 secs 2023-03-09T17:40:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:40:01 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:40:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:40:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9711.069761 secs 2023-03-09T17:40:01.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:40:01 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:40:02.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:40:02 smithi122 ceph-mon[93869]: pgmap v5692: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 89 KiB/s wr, 9 op/s 2023-03-09T17:40:02.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:40:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:40:02.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:40:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:40:02.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:40:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:40:02.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:40:02 smithi181 ceph-mon[94189]: pgmap v5692: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 89 KiB/s wr, 9 op/s 2023-03-09T17:40:02.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:40:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:40:02.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:40:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:40:02.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:40:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:40:02.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:40:02 smithi038 ceph-mon[92422]: pgmap v5692: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 89 KiB/s wr, 9 op/s 2023-03-09T17:40:02.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:40:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:40:02.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:40:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:40:02.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:40:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:40:03.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:40:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9713.343584 secs 2023-03-09T17:40:03.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:40:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9713.343584 secs 2023-03-09T17:40:03.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:40:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9713.343584 secs 2023-03-09T17:40:03.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10175.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:40:04.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:40:04 smithi122 ceph-mon[93869]: pgmap v5693: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 171 KiB/s wr, 13 op/s 2023-03-09T17:40:04.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:40:04 smithi181 ceph-mon[94189]: pgmap v5693: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 171 KiB/s wr, 13 op/s 2023-03-09T17:40:04.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:40:04 smithi038 ceph-mon[92422]: pgmap v5693: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 171 KiB/s wr, 13 op/s 2023-03-09T17:40:05.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:40:05 smithi122 ceph-mon[93869]: osdmap e569: 12 total, 12 up, 12 in 2023-03-09T17:40:05.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:40:05 smithi181 ceph-mon[94189]: osdmap e569: 12 total, 12 up, 12 in 2023-03-09T17:40:05.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:40:05 smithi038 ceph-mon[92422]: osdmap e569: 12 total, 12 up, 12 in 2023-03-09T17:40:06.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:40:06 smithi122 ceph-mon[93869]: pgmap v5695: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 146 KiB/s wr, 24 op/s 2023-03-09T17:40:06.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:40:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9716.069830 secs 2023-03-09T17:40:06.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:40:06 smithi181 ceph-mon[94189]: pgmap v5695: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 146 KiB/s wr, 24 op/s 2023-03-09T17:40:06.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:40:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9716.069830 secs 2023-03-09T17:40:06.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:40:06 smithi038 ceph-mon[92422]: pgmap v5695: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 146 KiB/s wr, 24 op/s 2023-03-09T17:40:06.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:40:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9716.069830 secs 2023-03-09T17:40:08.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:40:08 smithi122 ceph-mon[93869]: pgmap v5696: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 225 KiB/s wr, 27 op/s 2023-03-09T17:40:08.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:40:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9718.343718 secs 2023-03-09T17:40:08.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:40:08 smithi181 ceph-mon[94189]: pgmap v5696: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 225 KiB/s wr, 27 op/s 2023-03-09T17:40:08.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:40:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9718.343718 secs 2023-03-09T17:40:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:40:08 smithi038 ceph-mon[92422]: pgmap v5696: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 225 KiB/s wr, 27 op/s 2023-03-09T17:40:08.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:40:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9718.343718 secs 2023-03-09T17:40:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10180.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:40:10.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:40:10 smithi122 ceph-mon[93869]: pgmap v5697: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 226 KiB/s wr, 27 op/s 2023-03-09T17:40:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:40:10 smithi181 ceph-mon[94189]: pgmap v5697: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 226 KiB/s wr, 27 op/s 2023-03-09T17:40:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:40:10 smithi038 ceph-mon[92422]: pgmap v5697: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 226 KiB/s wr, 27 op/s 2023-03-09T17:40:11.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:40:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9721.069910 secs 2023-03-09T17:40:11.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:40:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9721.069910 secs 2023-03-09T17:40:11.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:40:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9721.069910 secs 2023-03-09T17:40:12.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:40:12 smithi122 ceph-mon[93869]: pgmap v5698: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 226 KiB/s wr, 27 op/s 2023-03-09T17:40:12.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:40:12 smithi181 ceph-mon[94189]: pgmap v5698: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 226 KiB/s wr, 27 op/s 2023-03-09T17:40:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:40:12 smithi038 ceph-mon[92422]: pgmap v5698: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 226 KiB/s wr, 27 op/s 2023-03-09T17:40:13.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:40:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9723.343811 secs 2023-03-09T17:40:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:40:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9723.343811 secs 2023-03-09T17:40:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:40:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9723.343811 secs 2023-03-09T17:40:13.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10185.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:40:14.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:40:14 smithi122 ceph-mon[93869]: pgmap v5699: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 150 KiB/s wr, 26 op/s 2023-03-09T17:40:14.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:40:14 smithi181 ceph-mon[94189]: pgmap v5699: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 150 KiB/s wr, 26 op/s 2023-03-09T17:40:14.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:40:14 smithi038 ceph-mon[92422]: pgmap v5699: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 150 KiB/s wr, 26 op/s 2023-03-09T17:40:16.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:40:16 smithi122 ceph-mon[93869]: pgmap v5700: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 118 KiB/s wr, 11 op/s 2023-03-09T17:40:16.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:40:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9726.069969 secs 2023-03-09T17:40:16.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:40:16 smithi122 ceph-mon[93869]: osdmap e570: 12 total, 12 up, 12 in 2023-03-09T17:40:16.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:40:16 smithi181 ceph-mon[94189]: pgmap v5700: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 118 KiB/s wr, 11 op/s 2023-03-09T17:40:16.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:40:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9726.069969 secs 2023-03-09T17:40:16.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:40:16 smithi181 ceph-mon[94189]: osdmap e570: 12 total, 12 up, 12 in 2023-03-09T17:40:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:40:16 smithi038 ceph-mon[92422]: pgmap v5700: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 118 KiB/s wr, 11 op/s 2023-03-09T17:40:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:40:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9726.069969 secs 2023-03-09T17:40:16.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:40:16 smithi038 ceph-mon[92422]: osdmap e570: 12 total, 12 up, 12 in 2023-03-09T17:40:18.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:40:18 smithi122 ceph-mon[93869]: pgmap v5702: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 33 KiB/s wr, 8 op/s 2023-03-09T17:40:18.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:40:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9728.343928 secs 2023-03-09T17:40:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:40:18 smithi181 ceph-mon[94189]: pgmap v5702: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 33 KiB/s wr, 8 op/s 2023-03-09T17:40:18.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:40:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9728.343928 secs 2023-03-09T17:40:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:40:18 smithi038 ceph-mon[92422]: pgmap v5702: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 33 KiB/s wr, 8 op/s 2023-03-09T17:40:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:40:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9728.343928 secs 2023-03-09T17:40:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10190.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:40:20.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:40:20 smithi122 ceph-mon[93869]: pgmap v5703: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 34 KiB/s wr, 7 op/s 2023-03-09T17:40:20.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:40:20 smithi181 ceph-mon[94189]: pgmap v5703: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 34 KiB/s wr, 7 op/s 2023-03-09T17:40:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:40:20 smithi038 ceph-mon[92422]: pgmap v5703: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 34 KiB/s wr, 7 op/s 2023-03-09T17:40:21.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:40:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9731.070032 secs 2023-03-09T17:40:21.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:40:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9731.070032 secs 2023-03-09T17:40:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:40:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9731.070032 secs 2023-03-09T17:40:22.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:40:22 smithi122 ceph-mon[93869]: pgmap v5704: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 34 KiB/s wr, 7 op/s 2023-03-09T17:40:22.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:40:22 smithi181 ceph-mon[94189]: pgmap v5704: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 34 KiB/s wr, 7 op/s 2023-03-09T17:40:22.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:40:22 smithi038 ceph-mon[92422]: pgmap v5704: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 34 KiB/s wr, 7 op/s 2023-03-09T17:40:23.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:40:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9733.344066 secs 2023-03-09T17:40:23.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:40:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9733.344066 secs 2023-03-09T17:40:23.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:40:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9733.344066 secs 2023-03-09T17:40:23.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10195.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:40:24.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:40:24 smithi122 ceph-mon[93869]: pgmap v5705: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 31 KiB/s wr, 7 op/s 2023-03-09T17:40:24.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:40:24 smithi181 ceph-mon[94189]: pgmap v5705: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 31 KiB/s wr, 7 op/s 2023-03-09T17:40:24.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:40:24 smithi038 ceph-mon[92422]: pgmap v5705: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 31 KiB/s wr, 7 op/s 2023-03-09T17:40:26.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:40:26 smithi122 ceph-mon[93869]: pgmap v5706: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 31 KiB/s wr, 7 op/s 2023-03-09T17:40:26.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:40:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9736.070091 secs 2023-03-09T17:40:26.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:40:26 smithi181 ceph-mon[94189]: pgmap v5706: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 31 KiB/s wr, 7 op/s 2023-03-09T17:40:26.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:40:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9736.070091 secs 2023-03-09T17:40:26.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:40:26 smithi038 ceph-mon[92422]: pgmap v5706: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 31 KiB/s wr, 7 op/s 2023-03-09T17:40:26.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:40:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9736.070091 secs 2023-03-09T17:40:28.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:40:28 smithi122 ceph-mon[93869]: pgmap v5707: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 48 KiB/s wr, 6 op/s 2023-03-09T17:40:28.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:40:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9738.344216 secs 2023-03-09T17:40:28.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:40:28 smithi181 ceph-mon[94189]: pgmap v5707: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 48 KiB/s wr, 6 op/s 2023-03-09T17:40:28.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:40:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9738.344216 secs 2023-03-09T17:40:28.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:40:28 smithi038 ceph-mon[92422]: pgmap v5707: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 48 KiB/s wr, 6 op/s 2023-03-09T17:40:28.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:40:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9738.344216 secs 2023-03-09T17:40:28.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10200.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:40:30.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:40:30 smithi122 ceph-mon[93869]: pgmap v5708: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 43 KiB/s wr, 6 op/s 2023-03-09T17:40:30.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:40:30 smithi181 ceph-mon[94189]: pgmap v5708: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 43 KiB/s wr, 6 op/s 2023-03-09T17:40:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:40:30 smithi038 ceph-mon[92422]: pgmap v5708: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 43 KiB/s wr, 6 op/s 2023-03-09T17:40:31.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:40:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9741.070154 secs 2023-03-09T17:40:31.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:40:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9741.070154 secs 2023-03-09T17:40:31.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:40:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9741.070154 secs 2023-03-09T17:40:32.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:40:32 smithi122 ceph-mon[93869]: pgmap v5709: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 6 op/s 2023-03-09T17:40:32.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:40:32 smithi181 ceph-mon[94189]: pgmap v5709: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 6 op/s 2023-03-09T17:40:32.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:40:32 smithi038 ceph-mon[92422]: pgmap v5709: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 6 op/s 2023-03-09T17:40:33.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:40:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9743.344294 secs 2023-03-09T17:40:33.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:40:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9743.344294 secs 2023-03-09T17:40:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:40:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9743.344294 secs 2023-03-09T17:40:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10205.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:40:34.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:40:34 smithi122 ceph-mon[93869]: pgmap v5710: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 42 KiB/s wr, 9 op/s 2023-03-09T17:40:34.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:40:34 smithi181 ceph-mon[94189]: pgmap v5710: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 42 KiB/s wr, 9 op/s 2023-03-09T17:40:34.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:40:34 smithi038 ceph-mon[92422]: pgmap v5710: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 42 KiB/s wr, 9 op/s 2023-03-09T17:40:36.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:40:36 smithi122 ceph-mon[93869]: pgmap v5711: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 25 KiB/s wr, 5 op/s 2023-03-09T17:40:36.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:40:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9746.070230 secs 2023-03-09T17:40:36.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:40:36 smithi181 ceph-mon[94189]: pgmap v5711: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 25 KiB/s wr, 5 op/s 2023-03-09T17:40:36.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:40:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9746.070230 secs 2023-03-09T17:40:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:40:36 smithi038 ceph-mon[92422]: pgmap v5711: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 25 KiB/s wr, 5 op/s 2023-03-09T17:40:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:40:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9746.070230 secs 2023-03-09T17:40:38.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:40:38 smithi122 ceph-mon[93869]: pgmap v5712: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 49 KiB/s wr, 8 op/s 2023-03-09T17:40:38.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:40:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9748.344389 secs 2023-03-09T17:40:38.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:40:38 smithi181 ceph-mon[94189]: pgmap v5712: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 49 KiB/s wr, 8 op/s 2023-03-09T17:40:38.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:40:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9748.344389 secs 2023-03-09T17:40:38.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:40:38 smithi038 ceph-mon[92422]: pgmap v5712: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 49 KiB/s wr, 8 op/s 2023-03-09T17:40:38.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:40:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9748.344389 secs 2023-03-09T17:40:38.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10210.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:40:40.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:40:40 smithi122 ceph-mon[93869]: pgmap v5713: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 24 KiB/s wr, 6 op/s 2023-03-09T17:40:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:40:40 smithi181 ceph-mon[94189]: pgmap v5713: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 24 KiB/s wr, 6 op/s 2023-03-09T17:40:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:40:40 smithi038 ceph-mon[92422]: pgmap v5713: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 24 KiB/s wr, 6 op/s 2023-03-09T17:40:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:40:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9751.070292 secs 2023-03-09T17:40:41.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:40:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9751.070292 secs 2023-03-09T17:40:41.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:40:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9751.070292 secs 2023-03-09T17:40:42.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:40:42 smithi038 ceph-mon[92422]: pgmap v5714: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 24 KiB/s wr, 5 op/s 2023-03-09T17:40:42.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:40:42 smithi122 ceph-mon[93869]: pgmap v5714: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 24 KiB/s wr, 5 op/s 2023-03-09T17:40:42.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:40:42 smithi181 ceph-mon[94189]: pgmap v5714: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 24 KiB/s wr, 5 op/s 2023-03-09T17:40:43.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10215.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:40:44.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:40:44 smithi038 ceph-mon[92422]: pgmap v5715: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 75 KiB/s wr, 9 op/s 2023-03-09T17:40:44.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:40:44 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:40:44.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:40:44 smithi122 ceph-mon[93869]: pgmap v5715: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 75 KiB/s wr, 9 op/s 2023-03-09T17:40:44.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:40:44 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:40:44.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:40:44 smithi181 ceph-mon[94189]: pgmap v5715: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 75 KiB/s wr, 9 op/s 2023-03-09T17:40:44.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:40:44 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:40:45.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:40:45 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:40:45.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:40:45 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:40:45.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:40:45 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:40:45.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:40:45 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:40:45.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:40:45 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:40:45.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:40:45 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:40:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:40:45 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:40:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:40:45 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:40:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:40:45 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:40:46.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:40:46 smithi122 ceph-mon[93869]: pgmap v5716: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 75 KiB/s wr, 6 op/s 2023-03-09T17:40:46.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:40:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9756.070352 secs 2023-03-09T17:40:46.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:40:46 smithi181 ceph-mon[94189]: pgmap v5716: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 75 KiB/s wr, 6 op/s 2023-03-09T17:40:46.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:40:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9756.070352 secs 2023-03-09T17:40:46.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:40:46 smithi038 ceph-mon[92422]: pgmap v5716: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 75 KiB/s wr, 6 op/s 2023-03-09T17:40:46.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:40:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9756.070352 secs 2023-03-09T17:40:48.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:40:48 smithi122 ceph-mon[93869]: pgmap v5717: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 109 KiB/s wr, 10 op/s 2023-03-09T17:40:48.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:40:48 smithi181 ceph-mon[94189]: pgmap v5717: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 109 KiB/s wr, 10 op/s 2023-03-09T17:40:48.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:40:48 smithi038 ceph-mon[92422]: pgmap v5717: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 109 KiB/s wr, 10 op/s 2023-03-09T17:40:48.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10220.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:40:50.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:40:50 smithi122 ceph-mon[93869]: pgmap v5718: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 86 KiB/s wr, 8 op/s 2023-03-09T17:40:50.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:40:50 smithi181 ceph-mon[94189]: pgmap v5718: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 86 KiB/s wr, 8 op/s 2023-03-09T17:40:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:40:50 smithi038 ceph-mon[92422]: pgmap v5718: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 86 KiB/s wr, 8 op/s 2023-03-09T17:40:51.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:40:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9761.070411 secs 2023-03-09T17:40:51.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:40:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9761.070411 secs 2023-03-09T17:40:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:40:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9761.070411 secs 2023-03-09T17:40:52.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:40:52 smithi122 ceph-mon[93869]: pgmap v5719: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 152 KiB/s wr, 9 op/s 2023-03-09T17:40:52.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:40:52 smithi181 ceph-mon[94189]: pgmap v5719: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 152 KiB/s wr, 9 op/s 2023-03-09T17:40:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:40:52 smithi038 ceph-mon[92422]: pgmap v5719: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 152 KiB/s wr, 9 op/s 2023-03-09T17:40:53.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10225.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:40:54.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:40:54 smithi122 ceph-mon[93869]: pgmap v5720: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 160 KiB/s wr, 13 op/s 2023-03-09T17:40:54.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:40:54 smithi181 ceph-mon[94189]: pgmap v5720: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 160 KiB/s wr, 13 op/s 2023-03-09T17:40:54.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:40:54 smithi038 ceph-mon[92422]: pgmap v5720: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 160 KiB/s wr, 13 op/s 2023-03-09T17:40:56.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:40:56 smithi122 ceph-mon[93869]: pgmap v5721: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 109 KiB/s wr, 10 op/s 2023-03-09T17:40:56.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:40:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9766.070474 secs 2023-03-09T17:40:56.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:40:56 smithi181 ceph-mon[94189]: pgmap v5721: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 109 KiB/s wr, 10 op/s 2023-03-09T17:40:56.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:40:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9766.070474 secs 2023-03-09T17:40:56.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:40:56 smithi038 ceph-mon[92422]: pgmap v5721: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 109 KiB/s wr, 10 op/s 2023-03-09T17:40:56.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:40:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9766.070474 secs 2023-03-09T17:40:57.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:40:57 smithi122 ceph-mon[93869]: pgmap v5722: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 117 KiB/s wr, 13 op/s 2023-03-09T17:40:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:40:57 smithi181 ceph-mon[94189]: pgmap v5722: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 117 KiB/s wr, 13 op/s 2023-03-09T17:40:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:40:57 smithi038 ceph-mon[92422]: pgmap v5722: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 117 KiB/s wr, 13 op/s 2023-03-09T17:40:58.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10230.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:40:59.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:40:59 smithi122 ceph-mon[93869]: pgmap v5723: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 83 KiB/s wr, 10 op/s 2023-03-09T17:40:59.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:40:59 smithi181 ceph-mon[94189]: pgmap v5723: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 83 KiB/s wr, 10 op/s 2023-03-09T17:41:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:40:59 smithi038 ceph-mon[92422]: pgmap v5723: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 83 KiB/s wr, 10 op/s 2023-03-09T17:41:00.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9771.070552 secs 2023-03-09T17:41:00.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9771.070552 secs 2023-03-09T17:41:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9771.070552 secs 2023-03-09T17:41:02.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:01 smithi122 ceph-mon[93869]: pgmap v5724: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 175 KiB/s wr, 11 op/s 2023-03-09T17:41:02.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:01 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:41:02.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:01 smithi181 ceph-mon[94189]: pgmap v5724: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 175 KiB/s wr, 11 op/s 2023-03-09T17:41:02.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:01 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:41:02.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:01 smithi038 ceph-mon[92422]: pgmap v5724: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 175 KiB/s wr, 11 op/s 2023-03-09T17:41:02.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:01 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:41:03.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9773.344958 secs 2023-03-09T17:41:03.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9773.344958 secs 2023-03-09T17:41:03.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9773.344958 secs 2023-03-09T17:41:03.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10235.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:41:04.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:03 smithi122 ceph-mon[93869]: pgmap v5725: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 147 KiB/s wr, 14 op/s 2023-03-09T17:41:04.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:03 smithi181 ceph-mon[94189]: pgmap v5725: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 147 KiB/s wr, 14 op/s 2023-03-09T17:41:04.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:03 smithi038 ceph-mon[92422]: pgmap v5725: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 147 KiB/s wr, 14 op/s 2023-03-09T17:41:05.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:04 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:41:05.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:04 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:41:05.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:04 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:41:05.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:41:05.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:41:05.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:41:05.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:41:05.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:41:05.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:41:05.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:41:05.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:41:05.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:41:05.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:41:05.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:41:05.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:41:05.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:41:05.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:41:05.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:04 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:41:05.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:04 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:41:05.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:04 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:41:05.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:41:05.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:41:05.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:41:05.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:41:05.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:41:05.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:41:05.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:41:05.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:41:05.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:41:05.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:41:05.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:41:05.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:41:05.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:41:05.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:41:05.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:04 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:41:05.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:04 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:41:05.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:04 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:41:05.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:41:05.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:41:05.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:41:05.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:41:05.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:41:05.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:41:05.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:41:05.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:41:05.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:41:05.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:41:05.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:41:05.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:41:05.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:41:05.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:41:06.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:06 smithi122 ceph-mon[93869]: pgmap v5726: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 160 KiB/s wr, 23 op/s 2023-03-09T17:41:06.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9776.070636 secs 2023-03-09T17:41:06.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:06 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T17:41:06.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:06 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T17:41:06.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:06 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi038 to 197.2M 2023-03-09T17:41:06.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:06 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi038 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-03-09T17:41:06.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:06 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:41:06.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:06 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:41:06.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:06 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:41:06.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:06 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:41:06.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:06 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:41:06.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:06 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:41:06.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:06 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:41:06.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:06 smithi181 ceph-mon[94189]: pgmap v5726: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 160 KiB/s wr, 23 op/s 2023-03-09T17:41:06.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9776.070636 secs 2023-03-09T17:41:06.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:06 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T17:41:06.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:06 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T17:41:06.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:06 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi038 to 197.2M 2023-03-09T17:41:06.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:06 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi038 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-03-09T17:41:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:06 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:41:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:06 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:41:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:06 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:41:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:06 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:41:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:06 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:41:06.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:06 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:41:06.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:06 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:41:06.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:06 smithi038 ceph-mon[92422]: pgmap v5726: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 160 KiB/s wr, 23 op/s 2023-03-09T17:41:06.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9776.070636 secs 2023-03-09T17:41:06.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:06 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T17:41:06.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:06 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T17:41:06.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:06 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi038 to 197.2M 2023-03-09T17:41:06.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:06 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi038 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-03-09T17:41:06.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:06 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:41:06.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:06 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:41:06.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:06 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:41:06.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:06 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:41:06.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:06 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:41:06.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:06 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:41:06.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:06 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:41:07.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:07 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T17:41:07.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:07 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T17:41:07.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:07 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T17:41:07.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:07 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T17:41:07.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:07 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T17:41:07.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:07 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T17:41:08.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:08 smithi122 ceph-mon[93869]: pgmap v5727: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 203 KiB/s wr, 26 op/s 2023-03-09T17:41:08.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9778.345082 secs 2023-03-09T17:41:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:08 smithi181 ceph-mon[94189]: pgmap v5727: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 203 KiB/s wr, 26 op/s 2023-03-09T17:41:08.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9778.345082 secs 2023-03-09T17:41:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:08 smithi038 ceph-mon[92422]: pgmap v5727: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 203 KiB/s wr, 26 op/s 2023-03-09T17:41:08.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9778.345082 secs 2023-03-09T17:41:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10240.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:41:10.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:10 smithi122 ceph-mon[93869]: pgmap v5728: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 196 KiB/s wr, 23 op/s 2023-03-09T17:41:10.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:10 smithi122 ceph-mon[93869]: osdmap e571: 12 total, 12 up, 12 in 2023-03-09T17:41:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:10 smithi181 ceph-mon[94189]: pgmap v5728: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 196 KiB/s wr, 23 op/s 2023-03-09T17:41:10.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:10 smithi181 ceph-mon[94189]: osdmap e571: 12 total, 12 up, 12 in 2023-03-09T17:41:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:10 smithi038 ceph-mon[92422]: pgmap v5728: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 196 KiB/s wr, 23 op/s 2023-03-09T17:41:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:10 smithi038 ceph-mon[92422]: osdmap e571: 12 total, 12 up, 12 in 2023-03-09T17:41:11.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9781.070698 secs 2023-03-09T17:41:11.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9781.070698 secs 2023-03-09T17:41:11.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9781.070698 secs 2023-03-09T17:41:12.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:12 smithi122 ceph-mon[93869]: pgmap v5730: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 189 KiB/s wr, 26 op/s 2023-03-09T17:41:12.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:12 smithi181 ceph-mon[94189]: pgmap v5730: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 189 KiB/s wr, 26 op/s 2023-03-09T17:41:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:12 smithi038 ceph-mon[92422]: pgmap v5730: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 189 KiB/s wr, 26 op/s 2023-03-09T17:41:13.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9783.345202 secs 2023-03-09T17:41:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9783.345202 secs 2023-03-09T17:41:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9783.345202 secs 2023-03-09T17:41:13.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10245.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:41:14.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:14 smithi122 ceph-mon[93869]: pgmap v5731: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 143 KiB/s wr, 23 op/s 2023-03-09T17:41:14.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:14 smithi181 ceph-mon[94189]: pgmap v5731: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 143 KiB/s wr, 23 op/s 2023-03-09T17:41:14.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:14 smithi038 ceph-mon[92422]: pgmap v5731: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 143 KiB/s wr, 23 op/s 2023-03-09T17:41:16.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:16 smithi122 ceph-mon[93869]: pgmap v5732: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 117 KiB/s wr, 8 op/s 2023-03-09T17:41:16.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9786.070758 secs 2023-03-09T17:41:16.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:16 smithi122 ceph-mon[93869]: osdmap e572: 12 total, 12 up, 12 in 2023-03-09T17:41:16.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:16 smithi181 ceph-mon[94189]: pgmap v5732: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 117 KiB/s wr, 8 op/s 2023-03-09T17:41:16.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9786.070758 secs 2023-03-09T17:41:16.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:16 smithi181 ceph-mon[94189]: osdmap e572: 12 total, 12 up, 12 in 2023-03-09T17:41:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:16 smithi038 ceph-mon[92422]: pgmap v5732: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 117 KiB/s wr, 8 op/s 2023-03-09T17:41:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9786.070758 secs 2023-03-09T17:41:16.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:16 smithi038 ceph-mon[92422]: osdmap e572: 12 total, 12 up, 12 in 2023-03-09T17:41:18.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:18 smithi122 ceph-mon[93869]: pgmap v5734: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 118 KiB/s wr, 9 op/s 2023-03-09T17:41:18.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9788.345317 secs 2023-03-09T17:41:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:18 smithi181 ceph-mon[94189]: pgmap v5734: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 118 KiB/s wr, 9 op/s 2023-03-09T17:41:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9788.345317 secs 2023-03-09T17:41:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:18 smithi038 ceph-mon[92422]: pgmap v5734: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 118 KiB/s wr, 9 op/s 2023-03-09T17:41:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9788.345317 secs 2023-03-09T17:41:18.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10250.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:41:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:20 smithi038 ceph-mon[92422]: pgmap v5735: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 100 KiB/s wr, 8 op/s 2023-03-09T17:41:20.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:20 smithi122 ceph-mon[93869]: pgmap v5735: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 100 KiB/s wr, 8 op/s 2023-03-09T17:41:20.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:20 smithi181 ceph-mon[94189]: pgmap v5735: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 100 KiB/s wr, 8 op/s 2023-03-09T17:41:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9791.070820 secs 2023-03-09T17:41:21.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9791.070820 secs 2023-03-09T17:41:21.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9791.070820 secs 2023-03-09T17:41:22.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:22 smithi038 ceph-mon[92422]: pgmap v5736: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 31 KiB/s wr, 7 op/s 2023-03-09T17:41:22.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:22 smithi122 ceph-mon[93869]: pgmap v5736: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 31 KiB/s wr, 7 op/s 2023-03-09T17:41:22.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:22 smithi181 ceph-mon[94189]: pgmap v5736: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 31 KiB/s wr, 7 op/s 2023-03-09T17:41:23.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9793.345402 secs 2023-03-09T17:41:23.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9793.345402 secs 2023-03-09T17:41:23.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9793.345402 secs 2023-03-09T17:41:23.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10255.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:41:24.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:24 smithi038 ceph-mon[92422]: pgmap v5737: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 31 KiB/s wr, 7 op/s 2023-03-09T17:41:24.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:24 smithi122 ceph-mon[93869]: pgmap v5737: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 31 KiB/s wr, 7 op/s 2023-03-09T17:41:24.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:24 smithi181 ceph-mon[94189]: pgmap v5737: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 31 KiB/s wr, 7 op/s 2023-03-09T17:41:26.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:26 smithi122 ceph-mon[93869]: pgmap v5738: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T17:41:26.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9796.070904 secs 2023-03-09T17:41:26.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:26 smithi181 ceph-mon[94189]: pgmap v5738: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T17:41:26.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9796.070904 secs 2023-03-09T17:41:26.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:26 smithi038 ceph-mon[92422]: pgmap v5738: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T17:41:26.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9796.070904 secs 2023-03-09T17:41:28.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:28 smithi122 ceph-mon[93869]: pgmap v5739: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 30 KiB/s wr, 6 op/s 2023-03-09T17:41:28.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9798.345541 secs 2023-03-09T17:41:28.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:28 smithi181 ceph-mon[94189]: pgmap v5739: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 30 KiB/s wr, 6 op/s 2023-03-09T17:41:28.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9798.345541 secs 2023-03-09T17:41:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:28 smithi038 ceph-mon[92422]: pgmap v5739: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 30 KiB/s wr, 6 op/s 2023-03-09T17:41:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9798.345541 secs 2023-03-09T17:41:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10260.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:41:30.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:30 smithi122 ceph-mon[93869]: pgmap v5740: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 6 op/s 2023-03-09T17:41:30.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:30 smithi181 ceph-mon[94189]: pgmap v5740: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 6 op/s 2023-03-09T17:41:30.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:30 smithi038 ceph-mon[92422]: pgmap v5740: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 6 op/s 2023-03-09T17:41:31.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9801.070963 secs 2023-03-09T17:41:31.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9801.070963 secs 2023-03-09T17:41:31.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9801.070963 secs 2023-03-09T17:41:32.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:32 smithi122 ceph-mon[93869]: pgmap v5741: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 6 op/s 2023-03-09T17:41:32.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:32 smithi181 ceph-mon[94189]: pgmap v5741: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 6 op/s 2023-03-09T17:41:32.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:32 smithi038 ceph-mon[92422]: pgmap v5741: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 6 op/s 2023-03-09T17:41:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10265.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:41:34.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:34 smithi122 ceph-mon[93869]: pgmap v5742: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 9 op/s 2023-03-09T17:41:34.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:34 smithi181 ceph-mon[94189]: pgmap v5742: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 9 op/s 2023-03-09T17:41:34.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:34 smithi038 ceph-mon[92422]: pgmap v5742: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 9 op/s 2023-03-09T17:41:36.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:36 smithi122 ceph-mon[93869]: pgmap v5743: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 6 op/s 2023-03-09T17:41:36.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9806.070985 secs 2023-03-09T17:41:36.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:36 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:41:36.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:36 smithi181 ceph-mon[94189]: pgmap v5743: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 6 op/s 2023-03-09T17:41:36.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9806.070985 secs 2023-03-09T17:41:36.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:36 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:41:36.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:36 smithi038 ceph-mon[92422]: pgmap v5743: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 6 op/s 2023-03-09T17:41:36.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9806.070985 secs 2023-03-09T17:41:36.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:36 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:41:37.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:37 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:41:37.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:41:37.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:41:37.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:37 smithi122 ceph-mon[93869]: pgmap v5744: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 KiB/s wr, 10 op/s 2023-03-09T17:41:37.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:37 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:41:37.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:41:37.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:41:37.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:37 smithi181 ceph-mon[94189]: pgmap v5744: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 KiB/s wr, 10 op/s 2023-03-09T17:41:37.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:37 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:41:37.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:41:37.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:41:37.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:37 smithi038 ceph-mon[92422]: pgmap v5744: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 KiB/s wr, 10 op/s 2023-03-09T17:41:38.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10270.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:41:39.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:39 smithi122 ceph-mon[93869]: pgmap v5745: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 66 KiB/s wr, 8 op/s 2023-03-09T17:41:39.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:39 smithi181 ceph-mon[94189]: pgmap v5745: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 66 KiB/s wr, 8 op/s 2023-03-09T17:41:40.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:39 smithi038 ceph-mon[92422]: pgmap v5745: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 66 KiB/s wr, 8 op/s 2023-03-09T17:41:40.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9811.071045 secs 2023-03-09T17:41:40.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9811.071045 secs 2023-03-09T17:41:41.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9811.071045 secs 2023-03-09T17:41:42.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:41 smithi122 ceph-mon[93869]: pgmap v5746: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 66 KiB/s wr, 8 op/s 2023-03-09T17:41:42.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:41 smithi181 ceph-mon[94189]: pgmap v5746: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 66 KiB/s wr, 8 op/s 2023-03-09T17:41:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:41 smithi038 ceph-mon[92422]: pgmap v5746: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 66 KiB/s wr, 8 op/s 2023-03-09T17:41:43.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:42 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 9813.345896 secs 2023-03-09T17:41:43.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:42 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 9813.345896 secs 2023-03-09T17:41:43.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:42 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 9813.345896 secs 2023-03-09T17:41:43.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10275.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:41:44.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:43 smithi122 ceph-mon[93869]: pgmap v5747: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 132 KiB/s wr, 12 op/s 2023-03-09T17:41:44.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:43 smithi181 ceph-mon[94189]: pgmap v5747: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 132 KiB/s wr, 12 op/s 2023-03-09T17:41:44.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:43 smithi038 ceph-mon[92422]: pgmap v5747: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 132 KiB/s wr, 12 op/s 2023-03-09T17:41:45.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:44 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:41:45.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:44 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:41:45.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:44 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:41:45.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:44 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:41:45.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:44 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:41:45.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:44 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:41:45.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:44 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:41:45.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:44 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:41:45.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:44 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:41:46.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:45 smithi122 ceph-mon[93869]: pgmap v5748: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 132 KiB/s wr, 9 op/s 2023-03-09T17:41:46.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9816.071122 secs 2023-03-09T17:41:46.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:45 smithi181 ceph-mon[94189]: pgmap v5748: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 132 KiB/s wr, 9 op/s 2023-03-09T17:41:46.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9816.071122 secs 2023-03-09T17:41:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:45 smithi038 ceph-mon[92422]: pgmap v5748: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 132 KiB/s wr, 9 op/s 2023-03-09T17:41:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9816.071122 secs 2023-03-09T17:41:48.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:47 smithi122 ceph-mon[93869]: pgmap v5749: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 192 KiB/s wr, 13 op/s 2023-03-09T17:41:48.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:47 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 9818.346020 secs 2023-03-09T17:41:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:47 smithi181 ceph-mon[94189]: pgmap v5749: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 192 KiB/s wr, 13 op/s 2023-03-09T17:41:48.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:47 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 9818.346020 secs 2023-03-09T17:41:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:47 smithi038 ceph-mon[92422]: pgmap v5749: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 192 KiB/s wr, 13 op/s 2023-03-09T17:41:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:47 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 9818.346020 secs 2023-03-09T17:41:48.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10280.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:41:50.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:49 smithi122 ceph-mon[93869]: pgmap v5750: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 127 KiB/s wr, 10 op/s 2023-03-09T17:41:50.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:49 smithi181 ceph-mon[94189]: pgmap v5750: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 127 KiB/s wr, 10 op/s 2023-03-09T17:41:50.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:49 smithi038 ceph-mon[92422]: pgmap v5750: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 127 KiB/s wr, 10 op/s 2023-03-09T17:41:51.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9821.071181 secs 2023-03-09T17:41:51.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9821.071181 secs 2023-03-09T17:41:51.541 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9821.071181 secs 2023-03-09T17:41:52.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:51 smithi122 ceph-mon[93869]: pgmap v5751: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 127 KiB/s wr, 10 op/s 2023-03-09T17:41:52.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:51 smithi181 ceph-mon[94189]: pgmap v5751: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 127 KiB/s wr, 10 op/s 2023-03-09T17:41:52.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:51 smithi038 ceph-mon[92422]: pgmap v5751: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 127 KiB/s wr, 10 op/s 2023-03-09T17:41:53.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9823.346138 secs 2023-03-09T17:41:53.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9823.346138 secs 2023-03-09T17:41:53.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9823.346138 secs 2023-03-09T17:41:53.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10285.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:41:54.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:53 smithi122 ceph-mon[93869]: pgmap v5752: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 182 KiB/s wr, 14 op/s 2023-03-09T17:41:54.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:53 smithi181 ceph-mon[94189]: pgmap v5752: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 182 KiB/s wr, 14 op/s 2023-03-09T17:41:54.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:53 smithi038 ceph-mon[92422]: pgmap v5752: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 182 KiB/s wr, 14 op/s 2023-03-09T17:41:56.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9826.071249 secs 2023-03-09T17:41:56.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:55 smithi122 ceph-mon[93869]: pgmap v5753: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 116 KiB/s wr, 10 op/s 2023-03-09T17:41:56.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9826.071249 secs 2023-03-09T17:41:56.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:55 smithi181 ceph-mon[94189]: pgmap v5753: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 116 KiB/s wr, 10 op/s 2023-03-09T17:41:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9826.071249 secs 2023-03-09T17:41:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:55 smithi038 ceph-mon[92422]: pgmap v5753: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 116 KiB/s wr, 10 op/s 2023-03-09T17:41:57.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:41:57.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:41:57.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:41:57.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:41:57.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:41:57.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:41:58.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:57 smithi122 ceph-mon[93869]: pgmap v5754: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 156 KiB/s wr, 12 op/s 2023-03-09T17:41:58.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9828.346272 secs 2023-03-09T17:41:58.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:57 smithi181 ceph-mon[94189]: pgmap v5754: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 156 KiB/s wr, 12 op/s 2023-03-09T17:41:58.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9828.346272 secs 2023-03-09T17:41:58.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:57 smithi038 ceph-mon[92422]: pgmap v5754: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 156 KiB/s wr, 12 op/s 2023-03-09T17:41:58.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9828.346272 secs 2023-03-09T17:41:58.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10290.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:42:00.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:41:59 smithi122 ceph-mon[93869]: pgmap v5755: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 95 KiB/s wr, 8 op/s 2023-03-09T17:42:00.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:41:59 smithi181 ceph-mon[94189]: pgmap v5755: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 95 KiB/s wr, 8 op/s 2023-03-09T17:42:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:41:59 smithi038 ceph-mon[92422]: pgmap v5755: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 95 KiB/s wr, 8 op/s 2023-03-09T17:42:01.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:42:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9831.071308 secs 2023-03-09T17:42:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:42:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9831.071308 secs 2023-03-09T17:42:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:42:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9831.071308 secs 2023-03-09T17:42:02.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:42:01 smithi122 ceph-mon[93869]: pgmap v5756: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 95 KiB/s wr, 8 op/s 2023-03-09T17:42:02.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:42:01 smithi181 ceph-mon[94189]: pgmap v5756: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 95 KiB/s wr, 8 op/s 2023-03-09T17:42:02.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:42:01 smithi038 ceph-mon[92422]: pgmap v5756: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 95 KiB/s wr, 8 op/s 2023-03-09T17:42:03.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:42:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9833.346353 secs 2023-03-09T17:42:03.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:42:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9833.346353 secs 2023-03-09T17:42:03.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:42:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9833.346353 secs 2023-03-09T17:42:03.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10295.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:42:04.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:42:03 smithi122 ceph-mon[93869]: pgmap v5757: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 173 KiB/s wr, 13 op/s 2023-03-09T17:42:04.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:42:03 smithi181 ceph-mon[94189]: pgmap v5757: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 173 KiB/s wr, 13 op/s 2023-03-09T17:42:04.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:42:03 smithi038 ceph-mon[92422]: pgmap v5757: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 173 KiB/s wr, 13 op/s 2023-03-09T17:42:06.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:42:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9836.071368 secs 2023-03-09T17:42:06.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:42:05 smithi122 ceph-mon[93869]: pgmap v5758: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 139 KiB/s wr, 21 op/s 2023-03-09T17:42:06.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:42:05 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:42:06.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:42:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9836.071368 secs 2023-03-09T17:42:06.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:42:05 smithi181 ceph-mon[94189]: pgmap v5758: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 139 KiB/s wr, 21 op/s 2023-03-09T17:42:06.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:42:05 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:42:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:42:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9836.071368 secs 2023-03-09T17:42:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:42:05 smithi038 ceph-mon[92422]: pgmap v5758: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 139 KiB/s wr, 21 op/s 2023-03-09T17:42:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:42:05 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:42:07.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:42:06 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:42:07.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:42:06 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:42:07.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:42:06 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:42:07.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:42:06 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:42:07.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:42:06 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:42:07.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:42:06 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:42:07.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:42:06 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:42:07.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:42:06 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:42:07.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:42:06 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:42:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:42:08 smithi038 ceph-mon[92422]: pgmap v5759: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 139 KiB/s wr, 24 op/s 2023-03-09T17:42:08.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:42:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9838.346475 secs 2023-03-09T17:42:08.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:42:08 smithi122 ceph-mon[93869]: pgmap v5759: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 139 KiB/s wr, 24 op/s 2023-03-09T17:42:08.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:42:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9838.346475 secs 2023-03-09T17:42:08.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:42:08 smithi181 ceph-mon[94189]: pgmap v5759: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 139 KiB/s wr, 24 op/s 2023-03-09T17:42:08.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:42:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9838.346475 secs 2023-03-09T17:42:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10300.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:42:10.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:42:10 smithi122 ceph-mon[93869]: pgmap v5760: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 100 KiB/s wr, 21 op/s 2023-03-09T17:42:10.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:42:10 smithi122 ceph-mon[93869]: osdmap e573: 12 total, 12 up, 12 in 2023-03-09T17:42:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:42:10 smithi181 ceph-mon[94189]: pgmap v5760: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 100 KiB/s wr, 21 op/s 2023-03-09T17:42:10.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:42:10 smithi181 ceph-mon[94189]: osdmap e573: 12 total, 12 up, 12 in 2023-03-09T17:42:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:42:10 smithi038 ceph-mon[92422]: pgmap v5760: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 100 KiB/s wr, 21 op/s 2023-03-09T17:42:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:42:10 smithi038 ceph-mon[92422]: osdmap e573: 12 total, 12 up, 12 in 2023-03-09T17:42:11.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:42:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9841.071453 secs 2023-03-09T17:42:11.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:42:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9841.071453 secs 2023-03-09T17:42:11.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:42:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9841.071453 secs 2023-03-09T17:42:12.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:42:12 smithi122 ceph-mon[93869]: pgmap v5762: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 121 KiB/s wr, 25 op/s 2023-03-09T17:42:12.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:42:12 smithi181 ceph-mon[94189]: pgmap v5762: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 121 KiB/s wr, 25 op/s 2023-03-09T17:42:12.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:42:12 smithi038 ceph-mon[92422]: pgmap v5762: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 121 KiB/s wr, 25 op/s 2023-03-09T17:42:13.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:42:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9843.346596 secs 2023-03-09T17:42:13.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:42:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9843.346596 secs 2023-03-09T17:42:13.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:42:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9843.346596 secs 2023-03-09T17:42:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10305.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:42:14.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:42:14 smithi122 ceph-mon[93869]: pgmap v5763: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 37 KiB/s wr, 22 op/s 2023-03-09T17:42:14.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:42:14 smithi181 ceph-mon[94189]: pgmap v5763: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 37 KiB/s wr, 22 op/s 2023-03-09T17:42:14.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:42:14 smithi038 ceph-mon[92422]: pgmap v5763: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 37 KiB/s wr, 22 op/s 2023-03-09T17:42:16.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:42:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9846.071513 secs 2023-03-09T17:42:16.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:42:16 smithi122 ceph-mon[93869]: pgmap v5764: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 8 op/s 2023-03-09T17:42:16.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:42:16 smithi122 ceph-mon[93869]: osdmap e574: 12 total, 12 up, 12 in 2023-03-09T17:42:16.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:42:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9846.071513 secs 2023-03-09T17:42:16.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:42:16 smithi181 ceph-mon[94189]: pgmap v5764: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 8 op/s 2023-03-09T17:42:16.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:42:16 smithi181 ceph-mon[94189]: osdmap e574: 12 total, 12 up, 12 in 2023-03-09T17:42:16.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:42:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9846.071513 secs 2023-03-09T17:42:16.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:42:16 smithi038 ceph-mon[92422]: pgmap v5764: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 8 op/s 2023-03-09T17:42:16.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:42:16 smithi038 ceph-mon[92422]: osdmap e574: 12 total, 12 up, 12 in 2023-03-09T17:42:18.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:42:18 smithi122 ceph-mon[93869]: pgmap v5766: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 26 KiB/s wr, 10 op/s 2023-03-09T17:42:18.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:42:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9848.346732 secs 2023-03-09T17:42:18.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:42:18 smithi181 ceph-mon[94189]: pgmap v5766: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 26 KiB/s wr, 10 op/s 2023-03-09T17:42:18.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:42:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9848.346732 secs 2023-03-09T17:42:18.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:42:18 smithi038 ceph-mon[92422]: pgmap v5766: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 26 KiB/s wr, 10 op/s 2023-03-09T17:42:18.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:42:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9848.346732 secs 2023-03-09T17:42:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10310.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:42:20.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:42:20 smithi122 ceph-mon[93869]: pgmap v5767: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 9 op/s 2023-03-09T17:42:20.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:42:20 smithi181 ceph-mon[94189]: pgmap v5767: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 9 op/s 2023-03-09T17:42:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:42:20 smithi038 ceph-mon[92422]: pgmap v5767: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 9 op/s 2023-03-09T17:42:21.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:42:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9851.071573 secs 2023-03-09T17:42:21.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:42:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9851.071573 secs 2023-03-09T17:42:21.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:42:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9851.071573 secs 2023-03-09T17:42:22.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:42:22 smithi122 ceph-mon[93869]: pgmap v5768: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 8 op/s 2023-03-09T17:42:22.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:42:22 smithi181 ceph-mon[94189]: pgmap v5768: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 8 op/s 2023-03-09T17:42:22.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:42:22 smithi038 ceph-mon[92422]: pgmap v5768: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 8 op/s 2023-03-09T17:42:23.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:42:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9853.346851 secs 2023-03-09T17:42:23.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:42:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9853.346851 secs 2023-03-09T17:42:23.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:42:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9853.346851 secs 2023-03-09T17:42:23.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10315.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:42:24.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:42:24 smithi122 ceph-mon[93869]: pgmap v5769: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 8 op/s 2023-03-09T17:42:24.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:42:24 smithi181 ceph-mon[94189]: pgmap v5769: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 8 op/s 2023-03-09T17:42:24.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:42:24 smithi038 ceph-mon[92422]: pgmap v5769: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 8 op/s 2023-03-09T17:42:25.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:42:25 smithi122 ceph-mon[93869]: pgmap v5770: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 8 op/s 2023-03-09T17:42:25.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:42:25 smithi181 ceph-mon[94189]: pgmap v5770: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 8 op/s 2023-03-09T17:42:25.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:42:25 smithi038 ceph-mon[92422]: pgmap v5770: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 8 op/s 2023-03-09T17:42:26.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:42:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9856.071652 secs 2023-03-09T17:42:26.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:42:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9856.071652 secs 2023-03-09T17:42:26.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:42:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9856.071652 secs 2023-03-09T17:42:27.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:42:27 smithi122 ceph-mon[93869]: pgmap v5771: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 55 KiB/s wr, 7 op/s 2023-03-09T17:42:27.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:42:27 smithi181 ceph-mon[94189]: pgmap v5771: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 55 KiB/s wr, 7 op/s 2023-03-09T17:42:27.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:42:27 smithi038 ceph-mon[92422]: pgmap v5771: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 55 KiB/s wr, 7 op/s 2023-03-09T17:42:28.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:42:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9858.346974 secs 2023-03-09T17:42:28.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:42:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9858.346974 secs 2023-03-09T17:42:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:42:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9858.346974 secs 2023-03-09T17:42:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10320.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:42:29.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:42:29 smithi122 ceph-mon[93869]: pgmap v5772: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 7 op/s 2023-03-09T17:42:29.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:42:29 smithi181 ceph-mon[94189]: pgmap v5772: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 7 op/s 2023-03-09T17:42:29.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:42:29 smithi038 ceph-mon[92422]: pgmap v5772: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 7 op/s 2023-03-09T17:42:30.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:42:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9861.071740 secs 2023-03-09T17:42:30.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:42:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9861.071740 secs 2023-03-09T17:42:30.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:42:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9861.071740 secs 2023-03-09T17:42:32.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:42:31 smithi122 ceph-mon[93869]: pgmap v5773: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 6 op/s 2023-03-09T17:42:32.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:42:31 smithi181 ceph-mon[94189]: pgmap v5773: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 6 op/s 2023-03-09T17:42:32.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:42:31 smithi038 ceph-mon[92422]: pgmap v5773: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 6 op/s 2023-03-09T17:42:33.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:42:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9863.347101 secs 2023-03-09T17:42:33.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:42:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9863.347101 secs 2023-03-09T17:42:33.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:42:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9863.347101 secs 2023-03-09T17:42:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10325.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:42:34.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:42:33 smithi122 ceph-mon[93869]: pgmap v5774: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 50 KiB/s wr, 9 op/s 2023-03-09T17:42:34.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:42:33 smithi181 ceph-mon[94189]: pgmap v5774: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 50 KiB/s wr, 9 op/s 2023-03-09T17:42:34.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:42:33 smithi038 ceph-mon[92422]: pgmap v5774: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 50 KiB/s wr, 9 op/s 2023-03-09T17:42:36.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:42:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9866.071825 secs 2023-03-09T17:42:36.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:42:35 smithi122 ceph-mon[93869]: pgmap v5775: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 6 op/s 2023-03-09T17:42:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:42:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9866.071825 secs 2023-03-09T17:42:36.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:42:35 smithi181 ceph-mon[94189]: pgmap v5775: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 6 op/s 2023-03-09T17:42:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:42:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9866.071825 secs 2023-03-09T17:42:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:42:35 smithi038 ceph-mon[92422]: pgmap v5775: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 6 op/s 2023-03-09T17:42:38.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:42:37 smithi122 ceph-mon[93869]: pgmap v5776: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 73 KiB/s wr, 9 op/s 2023-03-09T17:42:38.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:42:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9868.347192 secs 2023-03-09T17:42:38.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:42:37 smithi181 ceph-mon[94189]: pgmap v5776: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 73 KiB/s wr, 9 op/s 2023-03-09T17:42:38.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:42:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9868.347192 secs 2023-03-09T17:42:38.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:42:37 smithi038 ceph-mon[92422]: pgmap v5776: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 73 KiB/s wr, 9 op/s 2023-03-09T17:42:38.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:42:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9868.347192 secs 2023-03-09T17:42:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10330.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:42:40.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:42:39 smithi122 ceph-mon[93869]: pgmap v5777: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 65 KiB/s wr, 8 op/s 2023-03-09T17:42:40.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:42:39 smithi181 ceph-mon[94189]: pgmap v5777: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 65 KiB/s wr, 8 op/s 2023-03-09T17:42:40.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:42:39 smithi038 ceph-mon[92422]: pgmap v5777: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 65 KiB/s wr, 8 op/s 2023-03-09T17:42:41.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:42:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9871.071885 secs 2023-03-09T17:42:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:42:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9871.071885 secs 2023-03-09T17:42:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:42:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9871.071885 secs 2023-03-09T17:42:42.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:42:41 smithi122 ceph-mon[93869]: pgmap v5778: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 65 KiB/s wr, 8 op/s 2023-03-09T17:42:42.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:42:41 smithi181 ceph-mon[94189]: pgmap v5778: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 65 KiB/s wr, 8 op/s 2023-03-09T17:42:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:42:41 smithi038 ceph-mon[92422]: pgmap v5778: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 65 KiB/s wr, 8 op/s 2023-03-09T17:42:43.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:42:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9873.347309 secs 2023-03-09T17:42:43.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:42:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9873.347309 secs 2023-03-09T17:42:43.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:42:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9873.347309 secs 2023-03-09T17:42:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10335.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:42:44.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:42:43 smithi122 ceph-mon[93869]: pgmap v5779: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 131 KiB/s wr, 12 op/s 2023-03-09T17:42:44.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:42:43 smithi181 ceph-mon[94189]: pgmap v5779: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 131 KiB/s wr, 12 op/s 2023-03-09T17:42:44.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:42:43 smithi038 ceph-mon[92422]: pgmap v5779: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 131 KiB/s wr, 12 op/s 2023-03-09T17:42:46.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:42:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9876.071947 secs 2023-03-09T17:42:46.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:42:45 smithi122 ceph-mon[93869]: pgmap v5780: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 131 KiB/s wr, 10 op/s 2023-03-09T17:42:46.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:42:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9876.071947 secs 2023-03-09T17:42:46.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:42:45 smithi181 ceph-mon[94189]: pgmap v5780: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 131 KiB/s wr, 10 op/s 2023-03-09T17:42:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:42:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9876.071947 secs 2023-03-09T17:42:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:42:45 smithi038 ceph-mon[92422]: pgmap v5780: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 131 KiB/s wr, 10 op/s 2023-03-09T17:42:48.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:42:47 smithi122 ceph-mon[93869]: pgmap v5781: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 140 KiB/s wr, 12 op/s 2023-03-09T17:42:48.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:42:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9878.347460 secs 2023-03-09T17:42:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:42:47 smithi181 ceph-mon[94189]: pgmap v5781: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 140 KiB/s wr, 12 op/s 2023-03-09T17:42:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:42:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9878.347460 secs 2023-03-09T17:42:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:42:47 smithi038 ceph-mon[92422]: pgmap v5781: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 140 KiB/s wr, 12 op/s 2023-03-09T17:42:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:42:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9878.347460 secs 2023-03-09T17:42:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10340.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:42:50.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:42:49 smithi122 ceph-mon[93869]: pgmap v5782: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 133 KiB/s wr, 11 op/s 2023-03-09T17:42:50.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:42:49 smithi181 ceph-mon[94189]: pgmap v5782: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 133 KiB/s wr, 11 op/s 2023-03-09T17:42:50.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:42:49 smithi038 ceph-mon[92422]: pgmap v5782: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 133 KiB/s wr, 11 op/s 2023-03-09T17:42:51.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:42:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9881.072008 secs 2023-03-09T17:42:51.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:42:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9881.072008 secs 2023-03-09T17:42:51.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:42:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9881.072008 secs 2023-03-09T17:42:52.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:42:51 smithi122 ceph-mon[93869]: pgmap v5783: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 118 KiB/s wr, 9 op/s 2023-03-09T17:42:52.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:42:51 smithi181 ceph-mon[94189]: pgmap v5783: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 118 KiB/s wr, 9 op/s 2023-03-09T17:42:52.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:42:51 smithi038 ceph-mon[92422]: pgmap v5783: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 118 KiB/s wr, 9 op/s 2023-03-09T17:42:53.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:42:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9883.347596 secs 2023-03-09T17:42:53.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:42:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9883.347596 secs 2023-03-09T17:42:53.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:42:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9883.347596 secs 2023-03-09T17:42:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10345.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:42:54.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:42:53 smithi122 ceph-mon[93869]: pgmap v5784: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 173 KiB/s wr, 13 op/s 2023-03-09T17:42:54.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:42:53 smithi181 ceph-mon[94189]: pgmap v5784: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 173 KiB/s wr, 13 op/s 2023-03-09T17:42:54.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:42:53 smithi038 ceph-mon[92422]: pgmap v5784: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 173 KiB/s wr, 13 op/s 2023-03-09T17:42:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:42:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9886.072071 secs 2023-03-09T17:42:56.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:42:55 smithi122 ceph-mon[93869]: pgmap v5785: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 107 KiB/s wr, 9 op/s 2023-03-09T17:42:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:42:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9886.072071 secs 2023-03-09T17:42:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:42:55 smithi181 ceph-mon[94189]: pgmap v5785: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 107 KiB/s wr, 9 op/s 2023-03-09T17:42:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:42:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9886.072071 secs 2023-03-09T17:42:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:42:55 smithi038 ceph-mon[92422]: pgmap v5785: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 107 KiB/s wr, 9 op/s 2023-03-09T17:42:58.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:42:57 smithi038 ceph-mon[92422]: pgmap v5786: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 115 KiB/s wr, 11 op/s 2023-03-09T17:42:58.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:42:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9888.347705 secs 2023-03-09T17:42:58.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:42:57 smithi122 ceph-mon[93869]: pgmap v5786: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 115 KiB/s wr, 11 op/s 2023-03-09T17:42:58.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:42:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9888.347705 secs 2023-03-09T17:42:58.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:42:57 smithi181 ceph-mon[94189]: pgmap v5786: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 115 KiB/s wr, 11 op/s 2023-03-09T17:42:58.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:42:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9888.347705 secs 2023-03-09T17:42:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10350.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:43:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:42:59 smithi038 ceph-mon[92422]: pgmap v5787: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 156 KiB/s wr, 9 op/s 2023-03-09T17:43:00.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:42:59 smithi122 ceph-mon[93869]: pgmap v5787: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 156 KiB/s wr, 9 op/s 2023-03-09T17:43:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:42:59 smithi181 ceph-mon[94189]: pgmap v5787: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 156 KiB/s wr, 9 op/s 2023-03-09T17:43:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:43:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9891.072130 secs 2023-03-09T17:43:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:43:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9891.072130 secs 2023-03-09T17:43:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:43:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9891.072130 secs 2023-03-09T17:43:02.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:43:02 smithi122 ceph-mon[93869]: pgmap v5788: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 113 KiB/s wr, 8 op/s 2023-03-09T17:43:02.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:43:02 smithi181 ceph-mon[94189]: pgmap v5788: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 113 KiB/s wr, 8 op/s 2023-03-09T17:43:02.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:43:02 smithi038 ceph-mon[92422]: pgmap v5788: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 113 KiB/s wr, 8 op/s 2023-03-09T17:43:03.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:43:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9893.347820 secs 2023-03-09T17:43:03.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:43:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9893.347820 secs 2023-03-09T17:43:03.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:43:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9893.347820 secs 2023-03-09T17:43:03.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10355.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:43:04.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:43:04 smithi122 ceph-mon[93869]: pgmap v5789: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 199 KiB/s wr, 14 op/s 2023-03-09T17:43:04.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:43:04 smithi181 ceph-mon[94189]: pgmap v5789: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 199 KiB/s wr, 14 op/s 2023-03-09T17:43:04.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:43:04 smithi038 ceph-mon[92422]: pgmap v5789: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 199 KiB/s wr, 14 op/s 2023-03-09T17:43:06.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:43:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9896.072192 secs 2023-03-09T17:43:06.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:43:06 smithi122 ceph-mon[93869]: pgmap v5790: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 165 KiB/s wr, 22 op/s 2023-03-09T17:43:06.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:43:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9896.072192 secs 2023-03-09T17:43:06.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:43:06 smithi181 ceph-mon[94189]: pgmap v5790: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 165 KiB/s wr, 22 op/s 2023-03-09T17:43:06.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:43:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9896.072192 secs 2023-03-09T17:43:06.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:43:06 smithi038 ceph-mon[92422]: pgmap v5790: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 165 KiB/s wr, 22 op/s 2023-03-09T17:43:07.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:43:07 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:43:07.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:43:07 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:43:07.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:43:07 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:43:07.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:43:07 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:43:07.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:43:07 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:43:07.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:43:07 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:43:07.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:43:07 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:43:07.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:43:07 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:43:07.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:43:07 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:43:08.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:43:08 smithi122 ceph-mon[93869]: pgmap v5791: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 174 KiB/s wr, 25 op/s 2023-03-09T17:43:08.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:43:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9898.347937 secs 2023-03-09T17:43:08.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:43:08 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:43:08.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:43:08 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:43:08.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:43:08 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:43:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:43:08 smithi181 ceph-mon[94189]: pgmap v5791: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 174 KiB/s wr, 25 op/s 2023-03-09T17:43:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:43:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9898.347937 secs 2023-03-09T17:43:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:43:08 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:43:08.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:43:08 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:43:08.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:43:08 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:43:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:43:08 smithi038 ceph-mon[92422]: pgmap v5791: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 174 KiB/s wr, 25 op/s 2023-03-09T17:43:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:43:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9898.347937 secs 2023-03-09T17:43:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:43:08 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:43:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:43:08 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:43:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:43:08 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:43:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10360.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:43:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:43:10 smithi122 ceph-mon[93869]: pgmap v5792: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 183 KiB/s wr, 23 op/s 2023-03-09T17:43:10.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:43:10 smithi122 ceph-mon[93869]: osdmap e575: 12 total, 12 up, 12 in 2023-03-09T17:43:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:43:10 smithi181 ceph-mon[94189]: pgmap v5792: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 183 KiB/s wr, 23 op/s 2023-03-09T17:43:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:43:10 smithi181 ceph-mon[94189]: osdmap e575: 12 total, 12 up, 12 in 2023-03-09T17:43:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:43:10 smithi038 ceph-mon[92422]: pgmap v5792: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 183 KiB/s wr, 23 op/s 2023-03-09T17:43:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:43:10 smithi038 ceph-mon[92422]: osdmap e575: 12 total, 12 up, 12 in 2023-03-09T17:43:11.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:43:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9901.072252 secs 2023-03-09T17:43:11.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:43:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9901.072252 secs 2023-03-09T17:43:11.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:43:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9901.072252 secs 2023-03-09T17:43:12.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:43:12 smithi122 ceph-mon[93869]: pgmap v5794: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 160 KiB/s wr, 26 op/s 2023-03-09T17:43:12.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:43:12 smithi181 ceph-mon[94189]: pgmap v5794: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 160 KiB/s wr, 26 op/s 2023-03-09T17:43:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:43:12 smithi038 ceph-mon[92422]: pgmap v5794: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 160 KiB/s wr, 26 op/s 2023-03-09T17:43:13.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:43:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9903.348087 secs 2023-03-09T17:43:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:43:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9903.348087 secs 2023-03-09T17:43:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:43:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9903.348087 secs 2023-03-09T17:43:13.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10365.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:43:14.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:43:14 smithi122 ceph-mon[93869]: pgmap v5795: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 67 KiB/s wr, 23 op/s 2023-03-09T17:43:14.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:43:14 smithi181 ceph-mon[94189]: pgmap v5795: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 67 KiB/s wr, 23 op/s 2023-03-09T17:43:14.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:43:14 smithi038 ceph-mon[92422]: pgmap v5795: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 67 KiB/s wr, 23 op/s 2023-03-09T17:43:16.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:43:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9906.072314 secs 2023-03-09T17:43:16.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:43:16 smithi122 ceph-mon[93869]: pgmap v5796: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 41 KiB/s wr, 8 op/s 2023-03-09T17:43:16.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:43:16 smithi122 ceph-mon[93869]: osdmap e576: 12 total, 12 up, 12 in 2023-03-09T17:43:16.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:43:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9906.072314 secs 2023-03-09T17:43:16.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:43:16 smithi181 ceph-mon[94189]: pgmap v5796: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 41 KiB/s wr, 8 op/s 2023-03-09T17:43:16.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:43:16 smithi181 ceph-mon[94189]: osdmap e576: 12 total, 12 up, 12 in 2023-03-09T17:43:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:43:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9906.072314 secs 2023-03-09T17:43:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:43:16 smithi038 ceph-mon[92422]: pgmap v5796: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 41 KiB/s wr, 8 op/s 2023-03-09T17:43:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:43:16 smithi038 ceph-mon[92422]: osdmap e576: 12 total, 12 up, 12 in 2023-03-09T17:43:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:43:18 smithi122 ceph-mon[93869]: pgmap v5798: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 8 op/s 2023-03-09T17:43:18.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:43:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9908.348176 secs 2023-03-09T17:43:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:43:18 smithi181 ceph-mon[94189]: pgmap v5798: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 8 op/s 2023-03-09T17:43:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:43:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9908.348176 secs 2023-03-09T17:43:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:43:18 smithi038 ceph-mon[92422]: pgmap v5798: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 8 op/s 2023-03-09T17:43:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:43:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9908.348176 secs 2023-03-09T17:43:18.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10370.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:43:20.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:43:20 smithi122 ceph-mon[93869]: pgmap v5799: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 8 op/s 2023-03-09T17:43:20.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:43:20 smithi181 ceph-mon[94189]: pgmap v5799: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 8 op/s 2023-03-09T17:43:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:43:20 smithi038 ceph-mon[92422]: pgmap v5799: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 8 op/s 2023-03-09T17:43:21.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:43:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9911.072381 secs 2023-03-09T17:43:21.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:43:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9911.072381 secs 2023-03-09T17:43:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:43:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9911.072381 secs 2023-03-09T17:43:22.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:43:22 smithi122 ceph-mon[93869]: pgmap v5800: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 8 op/s 2023-03-09T17:43:22.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:43:22 smithi181 ceph-mon[94189]: pgmap v5800: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 8 op/s 2023-03-09T17:43:22.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:43:22 smithi038 ceph-mon[92422]: pgmap v5800: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 8 op/s 2023-03-09T17:43:23.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:43:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9913.348256 secs 2023-03-09T17:43:23.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:43:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9913.348256 secs 2023-03-09T17:43:23.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:43:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9913.348256 secs 2023-03-09T17:43:23.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10375.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:43:24.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:43:24 smithi122 ceph-mon[93869]: pgmap v5801: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 33 KiB/s wr, 7 op/s 2023-03-09T17:43:24.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:43:24 smithi181 ceph-mon[94189]: pgmap v5801: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 33 KiB/s wr, 7 op/s 2023-03-09T17:43:24.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:43:24 smithi038 ceph-mon[92422]: pgmap v5801: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 33 KiB/s wr, 7 op/s 2023-03-09T17:43:26.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:43:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9916.072441 secs 2023-03-09T17:43:26.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:43:26 smithi122 ceph-mon[93869]: pgmap v5802: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 33 KiB/s wr, 7 op/s 2023-03-09T17:43:26.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:43:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9916.072441 secs 2023-03-09T17:43:26.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:43:26 smithi181 ceph-mon[94189]: pgmap v5802: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 33 KiB/s wr, 7 op/s 2023-03-09T17:43:26.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:43:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9916.072441 secs 2023-03-09T17:43:26.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:43:26 smithi038 ceph-mon[92422]: pgmap v5802: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 33 KiB/s wr, 7 op/s 2023-03-09T17:43:28.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:43:28 smithi122 ceph-mon[93869]: pgmap v5803: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 7 op/s 2023-03-09T17:43:28.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:43:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9918.348318 secs 2023-03-09T17:43:28.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:43:28 smithi181 ceph-mon[94189]: pgmap v5803: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 7 op/s 2023-03-09T17:43:28.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:43:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9918.348318 secs 2023-03-09T17:43:28.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:43:28 smithi038 ceph-mon[92422]: pgmap v5803: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 7 op/s 2023-03-09T17:43:28.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:43:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9918.348318 secs 2023-03-09T17:43:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10380.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:43:30.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:43:30 smithi122 ceph-mon[93869]: pgmap v5804: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T17:43:30.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:43:30 smithi181 ceph-mon[94189]: pgmap v5804: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T17:43:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:43:30 smithi038 ceph-mon[92422]: pgmap v5804: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T17:43:31.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:43:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9921.072503 secs 2023-03-09T17:43:31.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:43:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9921.072503 secs 2023-03-09T17:43:31.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:43:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9921.072503 secs 2023-03-09T17:43:32.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:43:32 smithi122 ceph-mon[93869]: pgmap v5805: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 6 op/s 2023-03-09T17:43:32.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:43:32 smithi181 ceph-mon[94189]: pgmap v5805: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 6 op/s 2023-03-09T17:43:32.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:43:32 smithi038 ceph-mon[92422]: pgmap v5805: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 6 op/s 2023-03-09T17:43:33.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:43:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9923.348912 secs 2023-03-09T17:43:33.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:43:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9923.348912 secs 2023-03-09T17:43:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:43:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9923.348912 secs 2023-03-09T17:43:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10385.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:43:34.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:43:34 smithi122 ceph-mon[93869]: pgmap v5806: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 53 KiB/s wr, 9 op/s 2023-03-09T17:43:34.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:43:34 smithi181 ceph-mon[94189]: pgmap v5806: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 53 KiB/s wr, 9 op/s 2023-03-09T17:43:34.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:43:34 smithi038 ceph-mon[92422]: pgmap v5806: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 53 KiB/s wr, 9 op/s 2023-03-09T17:43:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:43:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9926.072563 secs 2023-03-09T17:43:36.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:43:36 smithi122 ceph-mon[93869]: pgmap v5807: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 26 KiB/s wr, 6 op/s 2023-03-09T17:43:36.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:43:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9926.072563 secs 2023-03-09T17:43:36.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:43:36 smithi181 ceph-mon[94189]: pgmap v5807: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 26 KiB/s wr, 6 op/s 2023-03-09T17:43:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:43:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9926.072563 secs 2023-03-09T17:43:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:43:36 smithi038 ceph-mon[92422]: pgmap v5807: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 26 KiB/s wr, 6 op/s 2023-03-09T17:43:38.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:43:38 smithi122 ceph-mon[93869]: pgmap v5808: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 8 op/s 2023-03-09T17:43:38.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:43:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9928.349058 secs 2023-03-09T17:43:38.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:43:38 smithi181 ceph-mon[94189]: pgmap v5808: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 8 op/s 2023-03-09T17:43:38.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:43:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9928.349058 secs 2023-03-09T17:43:38.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:43:38 smithi038 ceph-mon[92422]: pgmap v5808: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 8 op/s 2023-03-09T17:43:38.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:43:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9928.349058 secs 2023-03-09T17:43:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10390.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:43:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:43:40 smithi122 ceph-mon[93869]: pgmap v5809: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 51 KiB/s wr, 7 op/s 2023-03-09T17:43:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:43:40 smithi181 ceph-mon[94189]: pgmap v5809: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 51 KiB/s wr, 7 op/s 2023-03-09T17:43:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:43:40 smithi038 ceph-mon[92422]: pgmap v5809: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 51 KiB/s wr, 7 op/s 2023-03-09T17:43:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:43:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9931.072623 secs 2023-03-09T17:43:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:43:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9931.072623 secs 2023-03-09T17:43:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:43:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9931.072623 secs 2023-03-09T17:43:42.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:43:42 smithi122 ceph-mon[93869]: pgmap v5810: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 51 KiB/s wr, 7 op/s 2023-03-09T17:43:42.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:43:42 smithi181 ceph-mon[94189]: pgmap v5810: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 51 KiB/s wr, 7 op/s 2023-03-09T17:43:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:43:42 smithi038 ceph-mon[92422]: pgmap v5810: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 51 KiB/s wr, 7 op/s 2023-03-09T17:43:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10395.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:43:44.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:43:44 smithi122 ceph-mon[93869]: pgmap v5811: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 104 KiB/s wr, 9 op/s 2023-03-09T17:43:44.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:43:44 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:43:44.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:43:44 smithi181 ceph-mon[94189]: pgmap v5811: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 104 KiB/s wr, 9 op/s 2023-03-09T17:43:44.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:43:44 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:43:44.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:43:44 smithi038 ceph-mon[92422]: pgmap v5811: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 104 KiB/s wr, 9 op/s 2023-03-09T17:43:44.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:43:44 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:43:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:43:45 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:43:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:43:45 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:43:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:43:45 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:43:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:43:45 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:43:45.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:43:45 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:43:45.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:43:45 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:43:45.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:43:45 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:43:45.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:43:45 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:43:45.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:43:45 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:43:46.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:43:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9936.072683 secs 2023-03-09T17:43:46.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:43:46 smithi038 ceph-mon[92422]: pgmap v5812: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 78 KiB/s wr, 7 op/s 2023-03-09T17:43:46.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:43:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9936.072683 secs 2023-03-09T17:43:46.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:43:46 smithi122 ceph-mon[93869]: pgmap v5812: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 78 KiB/s wr, 7 op/s 2023-03-09T17:43:46.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:43:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9936.072683 secs 2023-03-09T17:43:46.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:43:46 smithi181 ceph-mon[94189]: pgmap v5812: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 78 KiB/s wr, 7 op/s 2023-03-09T17:43:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:43:48 smithi038 ceph-mon[92422]: pgmap v5813: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 87 KiB/s wr, 10 op/s 2023-03-09T17:43:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:43:48 smithi122 ceph-mon[93869]: pgmap v5813: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 87 KiB/s wr, 10 op/s 2023-03-09T17:43:48.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:43:48 smithi181 ceph-mon[94189]: pgmap v5813: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 87 KiB/s wr, 10 op/s 2023-03-09T17:43:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10400.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:43:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:43:50 smithi122 ceph-mon[93869]: pgmap v5814: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 115 KiB/s wr, 9 op/s 2023-03-09T17:43:50.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:43:50 smithi181 ceph-mon[94189]: pgmap v5814: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 115 KiB/s wr, 9 op/s 2023-03-09T17:43:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:43:50 smithi038 ceph-mon[92422]: pgmap v5814: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 115 KiB/s wr, 9 op/s 2023-03-09T17:43:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:43:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9941.072777 secs 2023-03-09T17:43:51.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:43:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9941.072777 secs 2023-03-09T17:43:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:43:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9941.072777 secs 2023-03-09T17:43:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:43:52 smithi122 ceph-mon[93869]: pgmap v5815: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 90 KiB/s wr, 8 op/s 2023-03-09T17:43:52.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:43:52 smithi181 ceph-mon[94189]: pgmap v5815: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 90 KiB/s wr, 8 op/s 2023-03-09T17:43:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:43:52 smithi038 ceph-mon[92422]: pgmap v5815: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 90 KiB/s wr, 8 op/s 2023-03-09T17:43:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10405.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:43:54.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:43:54 smithi122 ceph-mon[93869]: pgmap v5816: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 170 KiB/s wr, 12 op/s 2023-03-09T17:43:54.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:43:54 smithi181 ceph-mon[94189]: pgmap v5816: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 170 KiB/s wr, 12 op/s 2023-03-09T17:43:54.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:43:54 smithi038 ceph-mon[92422]: pgmap v5816: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 170 KiB/s wr, 12 op/s 2023-03-09T17:43:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:43:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9946.072865 secs 2023-03-09T17:43:56.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:43:56 smithi122 ceph-mon[93869]: pgmap v5817: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 124 KiB/s wr, 10 op/s 2023-03-09T17:43:56.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:43:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9946.072865 secs 2023-03-09T17:43:56.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:43:56 smithi181 ceph-mon[94189]: pgmap v5817: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 124 KiB/s wr, 10 op/s 2023-03-09T17:43:56.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:43:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9946.072865 secs 2023-03-09T17:43:56.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:43:56 smithi038 ceph-mon[92422]: pgmap v5817: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 124 KiB/s wr, 10 op/s 2023-03-09T17:43:58.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:43:58 smithi122 ceph-mon[93869]: pgmap v5818: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 125 KiB/s wr, 12 op/s 2023-03-09T17:43:58.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:43:58 smithi181 ceph-mon[94189]: pgmap v5818: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 125 KiB/s wr, 12 op/s 2023-03-09T17:43:58.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:43:58 smithi038 ceph-mon[92422]: pgmap v5818: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 125 KiB/s wr, 12 op/s 2023-03-09T17:43:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10410.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:44:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:00 smithi122 ceph-mon[93869]: pgmap v5819: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 138 KiB/s wr, 11 op/s 2023-03-09T17:44:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:00 smithi181 ceph-mon[94189]: pgmap v5819: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 138 KiB/s wr, 11 op/s 2023-03-09T17:44:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:00 smithi038 ceph-mon[92422]: pgmap v5819: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 138 KiB/s wr, 11 op/s 2023-03-09T17:44:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9951.072925 secs 2023-03-09T17:44:01.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9951.072925 secs 2023-03-09T17:44:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9951.072925 secs 2023-03-09T17:44:02.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:02 smithi122 ceph-mon[93869]: pgmap v5820: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 109 KiB/s wr, 10 op/s 2023-03-09T17:44:02.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:02 smithi181 ceph-mon[94189]: pgmap v5820: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 109 KiB/s wr, 10 op/s 2023-03-09T17:44:02.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:02 smithi038 ceph-mon[92422]: pgmap v5820: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 109 KiB/s wr, 10 op/s 2023-03-09T17:44:03.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9953.349492 secs 2023-03-09T17:44:03.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9953.349492 secs 2023-03-09T17:44:03.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9953.349492 secs 2023-03-09T17:44:03.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10415.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:44:04.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:04 smithi122 ceph-mon[93869]: pgmap v5821: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 215 KiB/s wr, 15 op/s 2023-03-09T17:44:04.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:04 smithi181 ceph-mon[94189]: pgmap v5821: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 215 KiB/s wr, 15 op/s 2023-03-09T17:44:04.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:04 smithi038 ceph-mon[92422]: pgmap v5821: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 215 KiB/s wr, 15 op/s 2023-03-09T17:44:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:05 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:44:05.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:05 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:44:05.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:05 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:44:05.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:05 smithi122 ceph-mon[93869]: pgmap v5822: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 165 KiB/s wr, 24 op/s 2023-03-09T17:44:05.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:05 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:44:05.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:05 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:44:05.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:05 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:44:05.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:05 smithi181 ceph-mon[94189]: pgmap v5822: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 165 KiB/s wr, 24 op/s 2023-03-09T17:44:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:05 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:44:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:05 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:44:05.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:05 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:44:05.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:05 smithi038 ceph-mon[92422]: pgmap v5822: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 165 KiB/s wr, 24 op/s 2023-03-09T17:44:06.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9956.072984 secs 2023-03-09T17:44:06.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9956.072984 secs 2023-03-09T17:44:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9956.072984 secs 2023-03-09T17:44:07.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:07 smithi122 ceph-mon[93869]: pgmap v5823: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 158 KiB/s wr, 26 op/s 2023-03-09T17:44:07.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:07 smithi181 ceph-mon[94189]: pgmap v5823: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 158 KiB/s wr, 26 op/s 2023-03-09T17:44:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:07 smithi038 ceph-mon[92422]: pgmap v5823: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 158 KiB/s wr, 26 op/s 2023-03-09T17:44:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9958.349610 secs 2023-03-09T17:44:08.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:08 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:44:08.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:08 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:44:08.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:08 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:44:08.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9958.349610 secs 2023-03-09T17:44:08.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:08 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:44:08.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:08 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:44:08.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:08 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:44:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9958.349610 secs 2023-03-09T17:44:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:08 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:44:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:08 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:44:08.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:08 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:44:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10420.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:44:09.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:09 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:44:09.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:09 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:44:09.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:09 smithi122 ceph-mon[93869]: pgmap v5824: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 174 KiB/s wr, 25 op/s 2023-03-09T17:44:09.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:09 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:44:09.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:09 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:44:09.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:09 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:44:09.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:09 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:44:09.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:09 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:44:09.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:09 smithi181 ceph-mon[94189]: pgmap v5824: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 174 KiB/s wr, 25 op/s 2023-03-09T17:44:09.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:09 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:44:09.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:09 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:44:09.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:09 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:44:09.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:09 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:44:09.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:09 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:44:09.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:09 smithi038 ceph-mon[92422]: pgmap v5824: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 174 KiB/s wr, 25 op/s 2023-03-09T17:44:09.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:09 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:44:09.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:09 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:44:09.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:09 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:44:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:10 smithi122 ceph-mon[93869]: osdmap e577: 12 total, 12 up, 12 in 2023-03-09T17:44:10.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9961.073045 secs 2023-03-09T17:44:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:10 smithi181 ceph-mon[94189]: osdmap e577: 12 total, 12 up, 12 in 2023-03-09T17:44:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9961.073045 secs 2023-03-09T17:44:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:10 smithi038 ceph-mon[92422]: osdmap e577: 12 total, 12 up, 12 in 2023-03-09T17:44:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9961.073045 secs 2023-03-09T17:44:12.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:11 smithi122 ceph-mon[93869]: pgmap v5826: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 185 KiB/s wr, 29 op/s 2023-03-09T17:44:12.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:11 smithi181 ceph-mon[94189]: pgmap v5826: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 185 KiB/s wr, 29 op/s 2023-03-09T17:44:12.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:11 smithi038 ceph-mon[92422]: pgmap v5826: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 185 KiB/s wr, 29 op/s 2023-03-09T17:44:13.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9963.349670 secs 2023-03-09T17:44:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9963.349670 secs 2023-03-09T17:44:13.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9963.349670 secs 2023-03-09T17:44:13.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10425.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:44:14.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:13 smithi122 ceph-mon[93869]: pgmap v5827: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 58 KiB/s wr, 23 op/s 2023-03-09T17:44:14.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:13 smithi181 ceph-mon[94189]: pgmap v5827: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 58 KiB/s wr, 23 op/s 2023-03-09T17:44:14.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:13 smithi038 ceph-mon[92422]: pgmap v5827: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 58 KiB/s wr, 23 op/s 2023-03-09T17:44:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9966.073131 secs 2023-03-09T17:44:16.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:15 smithi122 ceph-mon[93869]: pgmap v5828: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 23 KiB/s wr, 8 op/s 2023-03-09T17:44:16.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9966.073131 secs 2023-03-09T17:44:16.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:15 smithi181 ceph-mon[94189]: pgmap v5828: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 23 KiB/s wr, 8 op/s 2023-03-09T17:44:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9966.073131 secs 2023-03-09T17:44:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:15 smithi038 ceph-mon[92422]: pgmap v5828: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 23 KiB/s wr, 8 op/s 2023-03-09T17:44:17.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:16 smithi122 ceph-mon[93869]: osdmap e578: 12 total, 12 up, 12 in 2023-03-09T17:44:17.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:16 smithi181 ceph-mon[94189]: osdmap e578: 12 total, 12 up, 12 in 2023-03-09T17:44:17.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:16 smithi038 ceph-mon[92422]: osdmap e578: 12 total, 12 up, 12 in 2023-03-09T17:44:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:17 smithi122 ceph-mon[93869]: pgmap v5830: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 2.4 KiB/s wr, 7 op/s 2023-03-09T17:44:18.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9968.349780 secs 2023-03-09T17:44:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:17 smithi181 ceph-mon[94189]: pgmap v5830: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 2.4 KiB/s wr, 7 op/s 2023-03-09T17:44:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9968.349780 secs 2023-03-09T17:44:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:17 smithi038 ceph-mon[92422]: pgmap v5830: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 2.4 KiB/s wr, 7 op/s 2023-03-09T17:44:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9968.349780 secs 2023-03-09T17:44:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10430.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:44:20.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:19 smithi122 ceph-mon[93869]: pgmap v5831: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 2.1 KiB/s wr, 8 op/s 2023-03-09T17:44:20.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:19 smithi181 ceph-mon[94189]: pgmap v5831: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 2.1 KiB/s wr, 8 op/s 2023-03-09T17:44:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:19 smithi038 ceph-mon[92422]: pgmap v5831: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 2.1 KiB/s wr, 8 op/s 2023-03-09T17:44:21.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9971.073192 secs 2023-03-09T17:44:21.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9971.073192 secs 2023-03-09T17:44:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9971.073192 secs 2023-03-09T17:44:22.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:21 smithi122 ceph-mon[93869]: pgmap v5832: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 614 B/s wr, 6 op/s 2023-03-09T17:44:22.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:21 smithi181 ceph-mon[94189]: pgmap v5832: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 614 B/s wr, 6 op/s 2023-03-09T17:44:22.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:21 smithi038 ceph-mon[92422]: pgmap v5832: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 614 B/s wr, 6 op/s 2023-03-09T17:44:23.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9973.349844 secs 2023-03-09T17:44:23.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9973.349844 secs 2023-03-09T17:44:23.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9973.349844 secs 2023-03-09T17:44:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10435.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:44:24.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:23 smithi122 ceph-mon[93869]: pgmap v5833: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 32 KiB/s wr, 6 op/s 2023-03-09T17:44:24.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:23 smithi181 ceph-mon[94189]: pgmap v5833: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 32 KiB/s wr, 6 op/s 2023-03-09T17:44:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:23 smithi038 ceph-mon[92422]: pgmap v5833: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 32 KiB/s wr, 6 op/s 2023-03-09T17:44:26.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9976.073274 secs 2023-03-09T17:44:26.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:25 smithi181 ceph-mon[94189]: pgmap v5834: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 32 KiB/s wr, 6 op/s 2023-03-09T17:44:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9976.073274 secs 2023-03-09T17:44:26.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:25 smithi038 ceph-mon[92422]: pgmap v5834: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 32 KiB/s wr, 6 op/s 2023-03-09T17:44:26.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9976.073274 secs 2023-03-09T17:44:26.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:25 smithi122 ceph-mon[93869]: pgmap v5834: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 32 KiB/s wr, 6 op/s 2023-03-09T17:44:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:27 smithi038 ceph-mon[92422]: pgmap v5835: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 6 op/s 2023-03-09T17:44:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9978.349962 secs 2023-03-09T17:44:28.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:27 smithi122 ceph-mon[93869]: pgmap v5835: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 6 op/s 2023-03-09T17:44:28.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9978.349962 secs 2023-03-09T17:44:28.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:27 smithi181 ceph-mon[94189]: pgmap v5835: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 6 op/s 2023-03-09T17:44:28.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9978.349962 secs 2023-03-09T17:44:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10440.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:44:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:29 smithi038 ceph-mon[92422]: pgmap v5836: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T17:44:30.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:29 smithi122 ceph-mon[93869]: pgmap v5836: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T17:44:30.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:29 smithi181 ceph-mon[94189]: pgmap v5836: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T17:44:31.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9981.073334 secs 2023-03-09T17:44:31.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9981.073334 secs 2023-03-09T17:44:31.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9981.073334 secs 2023-03-09T17:44:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:31 smithi038 ceph-mon[92422]: pgmap v5837: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 26 KiB/s wr, 6 op/s 2023-03-09T17:44:32.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:31 smithi122 ceph-mon[93869]: pgmap v5837: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 26 KiB/s wr, 6 op/s 2023-03-09T17:44:32.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:31 smithi181 ceph-mon[94189]: pgmap v5837: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 26 KiB/s wr, 6 op/s 2023-03-09T17:44:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10445.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:44:34.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:33 smithi038 ceph-mon[92422]: pgmap v5838: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T17:44:34.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:33 smithi122 ceph-mon[93869]: pgmap v5838: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T17:44:34.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:33 smithi181 ceph-mon[94189]: pgmap v5838: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T17:44:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9986.073397 secs 2023-03-09T17:44:36.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:36 smithi122 ceph-mon[93869]: pgmap v5839: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T17:44:36.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:36 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:44:36.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9986.073397 secs 2023-03-09T17:44:36.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:36 smithi181 ceph-mon[94189]: pgmap v5839: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T17:44:36.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:36 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:44:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9986.073397 secs 2023-03-09T17:44:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:36 smithi038 ceph-mon[92422]: pgmap v5839: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T17:44:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:36 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:44:37.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:37 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:44:37.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:44:37.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:44:37.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:37 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:44:37.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:44:37.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:44:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:37 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:44:37.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:44:37.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:44:38.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:38 smithi122 ceph-mon[93869]: pgmap v5840: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 426 B/s wr, 7 op/s 2023-03-09T17:44:38.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:38 smithi181 ceph-mon[94189]: pgmap v5840: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 426 B/s wr, 7 op/s 2023-03-09T17:44:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:38 smithi038 ceph-mon[92422]: pgmap v5840: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 426 B/s wr, 7 op/s 2023-03-09T17:44:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10450.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:44:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:40 smithi122 ceph-mon[93869]: pgmap v5841: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 36 KiB/s wr, 7 op/s 2023-03-09T17:44:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:40 smithi181 ceph-mon[94189]: pgmap v5841: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 36 KiB/s wr, 7 op/s 2023-03-09T17:44:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:40 smithi038 ceph-mon[92422]: pgmap v5841: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 36 KiB/s wr, 7 op/s 2023-03-09T17:44:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9991.073457 secs 2023-03-09T17:44:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9991.073457 secs 2023-03-09T17:44:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9991.073457 secs 2023-03-09T17:44:42.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:42 smithi122 ceph-mon[93869]: pgmap v5842: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 71 KiB/s wr, 7 op/s 2023-03-09T17:44:42.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:42 smithi181 ceph-mon[94189]: pgmap v5842: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 71 KiB/s wr, 7 op/s 2023-03-09T17:44:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:42 smithi038 ceph-mon[92422]: pgmap v5842: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 71 KiB/s wr, 7 op/s 2023-03-09T17:44:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10455.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:44:44.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:44 smithi122 ceph-mon[93869]: pgmap v5843: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 71 KiB/s wr, 8 op/s 2023-03-09T17:44:44.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:44 smithi181 ceph-mon[94189]: pgmap v5843: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 71 KiB/s wr, 8 op/s 2023-03-09T17:44:44.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:44 smithi038 ceph-mon[92422]: pgmap v5843: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 71 KiB/s wr, 8 op/s 2023-03-09T17:44:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 9996.073519 secs 2023-03-09T17:44:46.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:46 smithi122 ceph-mon[93869]: pgmap v5844: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 79 KiB/s wr, 7 op/s 2023-03-09T17:44:46.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 9996.073519 secs 2023-03-09T17:44:46.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:46 smithi181 ceph-mon[94189]: pgmap v5844: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 79 KiB/s wr, 7 op/s 2023-03-09T17:44:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 9996.073519 secs 2023-03-09T17:44:46.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:46 smithi038 ceph-mon[92422]: pgmap v5844: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 79 KiB/s wr, 7 op/s 2023-03-09T17:44:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:48 smithi122 ceph-mon[93869]: pgmap v5845: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 87 KiB/s wr, 10 op/s 2023-03-09T17:44:48.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:48 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 9998.350365 secs 2023-03-09T17:44:48.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:48 smithi181 ceph-mon[94189]: pgmap v5845: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 87 KiB/s wr, 10 op/s 2023-03-09T17:44:48.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:48 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 9998.350365 secs 2023-03-09T17:44:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:48 smithi038 ceph-mon[92422]: pgmap v5845: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 87 KiB/s wr, 10 op/s 2023-03-09T17:44:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:48 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 9998.350365 secs 2023-03-09T17:44:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10460.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:44:49.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:49 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:44:49.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:44:49.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:44:49.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:49 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:44:49.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:44:49.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:44:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:49 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:44:49.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:44:49.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:44:50.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:50 smithi122 ceph-mon[93869]: pgmap v5846: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 95 KiB/s wr, 10 op/s 2023-03-09T17:44:50.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:50 smithi181 ceph-mon[94189]: pgmap v5846: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 95 KiB/s wr, 10 op/s 2023-03-09T17:44:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:50 smithi038 ceph-mon[92422]: pgmap v5846: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 95 KiB/s wr, 10 op/s 2023-03-09T17:44:51.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10001.073577 secs 2023-03-09T17:44:51.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10001.073577 secs 2023-03-09T17:44:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10001.073577 secs 2023-03-09T17:44:52.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:52 smithi122 ceph-mon[93869]: pgmap v5847: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 148 KiB/s wr, 11 op/s 2023-03-09T17:44:52.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:52 smithi181 ceph-mon[94189]: pgmap v5847: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 148 KiB/s wr, 11 op/s 2023-03-09T17:44:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:52 smithi038 ceph-mon[92422]: pgmap v5847: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 148 KiB/s wr, 11 op/s 2023-03-09T17:44:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10003.350477 secs 2023-03-09T17:44:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10003.350477 secs 2023-03-09T17:44:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10003.350477 secs 2023-03-09T17:44:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10465.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:44:54.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:54 smithi122 ceph-mon[93869]: pgmap v5848: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 114 KiB/s wr, 11 op/s 2023-03-09T17:44:54.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:54 smithi181 ceph-mon[94189]: pgmap v5848: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 114 KiB/s wr, 11 op/s 2023-03-09T17:44:54.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:54 smithi038 ceph-mon[92422]: pgmap v5848: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 114 KiB/s wr, 11 op/s 2023-03-09T17:44:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10006.073639 secs 2023-03-09T17:44:56.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:56 smithi122 ceph-mon[93869]: pgmap v5849: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 122 KiB/s wr, 10 op/s 2023-03-09T17:44:56.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10006.073639 secs 2023-03-09T17:44:56.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:56 smithi181 ceph-mon[94189]: pgmap v5849: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 122 KiB/s wr, 10 op/s 2023-03-09T17:44:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10006.073639 secs 2023-03-09T17:44:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:56 smithi038 ceph-mon[92422]: pgmap v5849: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 122 KiB/s wr, 10 op/s 2023-03-09T17:44:57.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:44:57.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:44:57.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:44:57.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:44:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:44:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:44:58.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:58 smithi122 ceph-mon[93869]: pgmap v5850: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 122 KiB/s wr, 12 op/s 2023-03-09T17:44:58.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:44:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10008.350609 secs 2023-03-09T17:44:58.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:58 smithi181 ceph-mon[94189]: pgmap v5850: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 122 KiB/s wr, 12 op/s 2023-03-09T17:44:58.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:44:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10008.350609 secs 2023-03-09T17:44:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:58 smithi038 ceph-mon[92422]: pgmap v5850: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 122 KiB/s wr, 12 op/s 2023-03-09T17:44:58.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:44:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10008.350609 secs 2023-03-09T17:44:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10470.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:45:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:45:00 smithi122 ceph-mon[93869]: pgmap v5851: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 122 KiB/s wr, 10 op/s 2023-03-09T17:45:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:45:00 smithi181 ceph-mon[94189]: pgmap v5851: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 122 KiB/s wr, 10 op/s 2023-03-09T17:45:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:45:00 smithi038 ceph-mon[92422]: pgmap v5851: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 122 KiB/s wr, 10 op/s 2023-03-09T17:45:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:45:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10011.073700 secs 2023-03-09T17:45:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:45:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10011.073700 secs 2023-03-09T17:45:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:45:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10011.073700 secs 2023-03-09T17:45:02.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:45:02 smithi122 ceph-mon[93869]: pgmap v5852: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 232 KiB/s wr, 13 op/s 2023-03-09T17:45:02.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:45:02 smithi181 ceph-mon[94189]: pgmap v5852: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 232 KiB/s wr, 13 op/s 2023-03-09T17:45:02.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:45:02 smithi038 ceph-mon[92422]: pgmap v5852: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 232 KiB/s wr, 13 op/s 2023-03-09T17:45:03.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:45:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10013.350725 secs 2023-03-09T17:45:03.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:45:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10013.350725 secs 2023-03-09T17:45:03.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:45:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10013.350725 secs 2023-03-09T17:45:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10475.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:45:04.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:45:04 smithi122 ceph-mon[93869]: pgmap v5853: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 144 KiB/s wr, 12 op/s 2023-03-09T17:45:04.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:45:04 smithi181 ceph-mon[94189]: pgmap v5853: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 144 KiB/s wr, 12 op/s 2023-03-09T17:45:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:45:04 smithi038 ceph-mon[92422]: pgmap v5853: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 144 KiB/s wr, 12 op/s 2023-03-09T17:45:06.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:45:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10016.073783 secs 2023-03-09T17:45:06.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:45:06 smithi122 ceph-mon[93869]: pgmap v5854: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 181 KiB/s wr, 23 op/s 2023-03-09T17:45:06.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:45:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10016.073783 secs 2023-03-09T17:45:06.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:45:06 smithi181 ceph-mon[94189]: pgmap v5854: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 181 KiB/s wr, 23 op/s 2023-03-09T17:45:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:45:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10016.073783 secs 2023-03-09T17:45:06.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:45:06 smithi038 ceph-mon[92422]: pgmap v5854: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 181 KiB/s wr, 23 op/s 2023-03-09T17:45:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:45:08 smithi122 ceph-mon[93869]: pgmap v5855: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 190 KiB/s wr, 25 op/s 2023-03-09T17:45:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:45:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10018.350813 secs 2023-03-09T17:45:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:45:08 smithi181 ceph-mon[94189]: pgmap v5855: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 190 KiB/s wr, 25 op/s 2023-03-09T17:45:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:45:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10018.350813 secs 2023-03-09T17:45:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:45:08 smithi038 ceph-mon[92422]: pgmap v5855: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 190 KiB/s wr, 25 op/s 2023-03-09T17:45:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:45:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10018.350813 secs 2023-03-09T17:45:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10480.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:45:09.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:45:09 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:45:09.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:45:09 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:45:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:45:09 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:45:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:45:10 smithi122 ceph-mon[93869]: pgmap v5856: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 182 KiB/s wr, 24 op/s 2023-03-09T17:45:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:45:10 smithi122 ceph-mon[93869]: osdmap e579: 12 total, 12 up, 12 in 2023-03-09T17:45:10.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:45:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:45:10.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:45:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:45:10.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:45:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:45:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:45:10 smithi181 ceph-mon[94189]: pgmap v5856: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 182 KiB/s wr, 24 op/s 2023-03-09T17:45:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:45:10 smithi181 ceph-mon[94189]: osdmap e579: 12 total, 12 up, 12 in 2023-03-09T17:45:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:45:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:45:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:45:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:45:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:45:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:45:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:45:10 smithi038 ceph-mon[92422]: pgmap v5856: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 182 KiB/s wr, 24 op/s 2023-03-09T17:45:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:45:10 smithi038 ceph-mon[92422]: osdmap e579: 12 total, 12 up, 12 in 2023-03-09T17:45:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:45:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:45:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:45:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:45:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:45:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:45:11.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:45:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10021.073852 secs 2023-03-09T17:45:11.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:45:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10021.073852 secs 2023-03-09T17:45:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:45:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10021.073852 secs 2023-03-09T17:45:12.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:45:12 smithi122 ceph-mon[93869]: pgmap v5858: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 134 KiB/s wr, 23 op/s 2023-03-09T17:45:12.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:45:12 smithi181 ceph-mon[94189]: pgmap v5858: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 134 KiB/s wr, 23 op/s 2023-03-09T17:45:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:45:12 smithi038 ceph-mon[92422]: pgmap v5858: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 134 KiB/s wr, 23 op/s 2023-03-09T17:45:13.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:45:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10023.350948 secs 2023-03-09T17:45:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:45:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10023.350948 secs 2023-03-09T17:45:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:45:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10023.350948 secs 2023-03-09T17:45:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10485.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:45:14.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:45:14 smithi122 ceph-mon[93869]: pgmap v5859: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 133 KiB/s wr, 23 op/s 2023-03-09T17:45:14.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:45:14 smithi181 ceph-mon[94189]: pgmap v5859: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 133 KiB/s wr, 23 op/s 2023-03-09T17:45:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:45:14 smithi038 ceph-mon[92422]: pgmap v5859: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 133 KiB/s wr, 23 op/s 2023-03-09T17:45:16.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:45:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10026.073909 secs 2023-03-09T17:45:16.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:45:16 smithi122 ceph-mon[93869]: pgmap v5860: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 98 KiB/s wr, 8 op/s 2023-03-09T17:45:16.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:45:16 smithi122 ceph-mon[93869]: osdmap e580: 12 total, 12 up, 12 in 2023-03-09T17:45:16.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:45:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10026.073909 secs 2023-03-09T17:45:16.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:45:16 smithi181 ceph-mon[94189]: pgmap v5860: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 98 KiB/s wr, 8 op/s 2023-03-09T17:45:16.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:45:16 smithi181 ceph-mon[94189]: osdmap e580: 12 total, 12 up, 12 in 2023-03-09T17:45:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:45:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10026.073909 secs 2023-03-09T17:45:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:45:16 smithi038 ceph-mon[92422]: pgmap v5860: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 98 KiB/s wr, 8 op/s 2023-03-09T17:45:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:45:16 smithi038 ceph-mon[92422]: osdmap e580: 12 total, 12 up, 12 in 2023-03-09T17:45:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:45:18 smithi122 ceph-mon[93869]: pgmap v5862: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 122 KiB/s wr, 9 op/s 2023-03-09T17:45:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:45:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10028.351087 secs 2023-03-09T17:45:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:45:18 smithi181 ceph-mon[94189]: pgmap v5862: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 122 KiB/s wr, 9 op/s 2023-03-09T17:45:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:45:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10028.351087 secs 2023-03-09T17:45:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:45:18 smithi038 ceph-mon[92422]: pgmap v5862: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 122 KiB/s wr, 9 op/s 2023-03-09T17:45:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:45:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10028.351087 secs 2023-03-09T17:45:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10490.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:45:20.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:45:20 smithi122 ceph-mon[93869]: pgmap v5863: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 103 KiB/s wr, 8 op/s 2023-03-09T17:45:20.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:45:20 smithi181 ceph-mon[94189]: pgmap v5863: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 103 KiB/s wr, 8 op/s 2023-03-09T17:45:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:45:20 smithi038 ceph-mon[92422]: pgmap v5863: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 103 KiB/s wr, 8 op/s 2023-03-09T17:45:21.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:45:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10031.073975 secs 2023-03-09T17:45:21.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:45:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10031.073975 secs 2023-03-09T17:45:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:45:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10031.073975 secs 2023-03-09T17:45:22.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:45:22 smithi122 ceph-mon[93869]: pgmap v5864: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T17:45:22.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:45:22 smithi181 ceph-mon[94189]: pgmap v5864: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T17:45:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:45:22 smithi038 ceph-mon[92422]: pgmap v5864: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T17:45:23.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:45:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10033.351201 secs 2023-03-09T17:45:23.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:45:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10033.351201 secs 2023-03-09T17:45:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:45:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10033.351201 secs 2023-03-09T17:45:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10495.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:45:24.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:45:24 smithi122 ceph-mon[93869]: pgmap v5865: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T17:45:24.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:45:24 smithi181 ceph-mon[94189]: pgmap v5865: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T17:45:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:45:24 smithi038 ceph-mon[92422]: pgmap v5865: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T17:45:26.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:45:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10036.073996 secs 2023-03-09T17:45:26.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:45:26 smithi122 ceph-mon[93869]: pgmap v5866: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 23 KiB/s wr, 8 op/s 2023-03-09T17:45:26.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:45:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10036.073996 secs 2023-03-09T17:45:26.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:45:26 smithi181 ceph-mon[94189]: pgmap v5866: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 23 KiB/s wr, 8 op/s 2023-03-09T17:45:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:45:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10036.073996 secs 2023-03-09T17:45:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:45:26 smithi038 ceph-mon[92422]: pgmap v5866: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 23 KiB/s wr, 8 op/s 2023-03-09T17:45:28.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:45:28 smithi122 ceph-mon[93869]: pgmap v5867: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 31 KiB/s wr, 7 op/s 2023-03-09T17:45:28.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:45:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10038.351297 secs 2023-03-09T17:45:28.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:45:28 smithi181 ceph-mon[94189]: pgmap v5867: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 31 KiB/s wr, 7 op/s 2023-03-09T17:45:28.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:45:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10038.351297 secs 2023-03-09T17:45:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:45:28 smithi038 ceph-mon[92422]: pgmap v5867: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 31 KiB/s wr, 7 op/s 2023-03-09T17:45:28.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:45:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10038.351297 secs 2023-03-09T17:45:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10500.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:45:30.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:45:30 smithi122 ceph-mon[93869]: pgmap v5868: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T17:45:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:45:30 smithi181 ceph-mon[94189]: pgmap v5868: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T17:45:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:45:30 smithi038 ceph-mon[92422]: pgmap v5868: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T17:45:31.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:45:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10041.074055 secs 2023-03-09T17:45:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:45:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10041.074055 secs 2023-03-09T17:45:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:45:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10041.074055 secs 2023-03-09T17:45:32.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:45:32 smithi122 ceph-mon[93869]: pgmap v5869: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 29 KiB/s wr, 9 op/s 2023-03-09T17:45:32.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:45:32 smithi181 ceph-mon[94189]: pgmap v5869: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 29 KiB/s wr, 9 op/s 2023-03-09T17:45:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:45:32 smithi038 ceph-mon[92422]: pgmap v5869: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 29 KiB/s wr, 9 op/s 2023-03-09T17:45:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:45:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10043.351368 secs 2023-03-09T17:45:33.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:45:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10043.351368 secs 2023-03-09T17:45:33.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:45:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10043.351368 secs 2023-03-09T17:45:33.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10505.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:45:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:45:34 smithi038 ceph-mon[92422]: pgmap v5870: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 28 KiB/s wr, 8 op/s 2023-03-09T17:45:34.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:45:34 smithi122 ceph-mon[93869]: pgmap v5870: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 28 KiB/s wr, 8 op/s 2023-03-09T17:45:34.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:45:34 smithi181 ceph-mon[94189]: pgmap v5870: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 28 KiB/s wr, 8 op/s 2023-03-09T17:45:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:45:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10046.074138 secs 2023-03-09T17:45:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:45:36 smithi038 ceph-mon[92422]: pgmap v5871: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 28 KiB/s wr, 8 op/s 2023-03-09T17:45:36.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:45:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10046.074138 secs 2023-03-09T17:45:36.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:45:36 smithi122 ceph-mon[93869]: pgmap v5871: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 28 KiB/s wr, 8 op/s 2023-03-09T17:45:36.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:45:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10046.074138 secs 2023-03-09T17:45:36.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:45:36 smithi181 ceph-mon[94189]: pgmap v5871: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 28 KiB/s wr, 8 op/s 2023-03-09T17:45:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:45:38 smithi038 ceph-mon[92422]: pgmap v5872: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 88 KiB/s wr, 10 op/s 2023-03-09T17:45:38.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:45:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10048.351457 secs 2023-03-09T17:45:38.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:45:38 smithi122 ceph-mon[93869]: pgmap v5872: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 88 KiB/s wr, 10 op/s 2023-03-09T17:45:38.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:45:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10048.351457 secs 2023-03-09T17:45:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:45:38 smithi181 ceph-mon[94189]: pgmap v5872: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 88 KiB/s wr, 10 op/s 2023-03-09T17:45:38.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:45:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10048.351457 secs 2023-03-09T17:45:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10510.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:45:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:45:40 smithi122 ceph-mon[93869]: pgmap v5873: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 69 KiB/s wr, 8 op/s 2023-03-09T17:45:40.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:45:40 smithi181 ceph-mon[94189]: pgmap v5873: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 69 KiB/s wr, 8 op/s 2023-03-09T17:45:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:45:40 smithi038 ceph-mon[92422]: pgmap v5873: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 69 KiB/s wr, 8 op/s 2023-03-09T17:45:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:45:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10051.074200 secs 2023-03-09T17:45:41.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:45:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10051.074200 secs 2023-03-09T17:45:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:45:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10051.074200 secs 2023-03-09T17:45:42.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:45:42 smithi122 ceph-mon[93869]: pgmap v5874: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 95 KiB/s wr, 9 op/s 2023-03-09T17:45:42.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:45:42 smithi181 ceph-mon[94189]: pgmap v5874: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 95 KiB/s wr, 9 op/s 2023-03-09T17:45:42.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:45:42 smithi038 ceph-mon[92422]: pgmap v5874: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 95 KiB/s wr, 9 op/s 2023-03-09T17:45:43.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:45:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10053.351583 secs 2023-03-09T17:45:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:45:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10053.351583 secs 2023-03-09T17:45:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:45:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10053.351583 secs 2023-03-09T17:45:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10515.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:45:44.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:45:44 smithi122 ceph-mon[93869]: pgmap v5875: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 95 KiB/s wr, 9 op/s 2023-03-09T17:45:44.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:45:44 smithi181 ceph-mon[94189]: pgmap v5875: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 95 KiB/s wr, 9 op/s 2023-03-09T17:45:44.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:45:44 smithi038 ceph-mon[92422]: pgmap v5875: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 95 KiB/s wr, 9 op/s 2023-03-09T17:45:46.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:45:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10056.074259 secs 2023-03-09T17:45:46.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:45:46 smithi122 ceph-mon[93869]: pgmap v5876: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 103 KiB/s wr, 9 op/s 2023-03-09T17:45:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:45:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10056.074259 secs 2023-03-09T17:45:46.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:45:46 smithi181 ceph-mon[94189]: pgmap v5876: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 103 KiB/s wr, 9 op/s 2023-03-09T17:45:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:45:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10056.074259 secs 2023-03-09T17:45:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:45:46 smithi038 ceph-mon[92422]: pgmap v5876: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 103 KiB/s wr, 9 op/s 2023-03-09T17:45:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:45:48 smithi122 ceph-mon[93869]: pgmap v5877: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 170 KiB/s wr, 12 op/s 2023-03-09T17:45:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:45:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10058.351701 secs 2023-03-09T17:45:48.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:45:48 smithi181 ceph-mon[94189]: pgmap v5877: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 170 KiB/s wr, 12 op/s 2023-03-09T17:45:48.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:45:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10058.351701 secs 2023-03-09T17:45:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:45:48 smithi038 ceph-mon[92422]: pgmap v5877: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 170 KiB/s wr, 12 op/s 2023-03-09T17:45:48.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:45:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10058.351701 secs 2023-03-09T17:45:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10520.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:45:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:45:50 smithi122 ceph-mon[93869]: pgmap v5878: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 119 KiB/s wr, 10 op/s 2023-03-09T17:45:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:45:50 smithi181 ceph-mon[94189]: pgmap v5878: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 119 KiB/s wr, 10 op/s 2023-03-09T17:45:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:45:50 smithi038 ceph-mon[92422]: pgmap v5878: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 119 KiB/s wr, 10 op/s 2023-03-09T17:45:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:45:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10061.074318 secs 2023-03-09T17:45:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:45:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10061.074318 secs 2023-03-09T17:45:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:45:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10061.074318 secs 2023-03-09T17:45:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:45:52 smithi122 ceph-mon[93869]: pgmap v5879: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 156 KiB/s wr, 11 op/s 2023-03-09T17:45:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:45:52 smithi181 ceph-mon[94189]: pgmap v5879: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 156 KiB/s wr, 11 op/s 2023-03-09T17:45:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:45:52 smithi038 ceph-mon[92422]: pgmap v5879: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 156 KiB/s wr, 11 op/s 2023-03-09T17:45:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10525.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:45:54.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:45:54 smithi122 ceph-mon[93869]: pgmap v5880: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 130 KiB/s wr, 10 op/s 2023-03-09T17:45:54.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:45:54 smithi181 ceph-mon[94189]: pgmap v5880: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 130 KiB/s wr, 10 op/s 2023-03-09T17:45:54.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:45:54 smithi038 ceph-mon[92422]: pgmap v5880: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 130 KiB/s wr, 10 op/s 2023-03-09T17:45:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:45:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10066.074375 secs 2023-03-09T17:45:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:45:56 smithi122 ceph-mon[93869]: pgmap v5881: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 146 KiB/s wr, 10 op/s 2023-03-09T17:45:56.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:45:56 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:45:56.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:45:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10066.074375 secs 2023-03-09T17:45:56.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:45:56 smithi181 ceph-mon[94189]: pgmap v5881: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 146 KiB/s wr, 10 op/s 2023-03-09T17:45:56.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:45:56 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:45:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:45:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10066.074375 secs 2023-03-09T17:45:56.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:45:56 smithi038 ceph-mon[92422]: pgmap v5881: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 146 KiB/s wr, 10 op/s 2023-03-09T17:45:56.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:45:56 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:45:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:45:57 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:45:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:45:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:45:57.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:45:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:45:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:45:57 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:45:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:45:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:45:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:45:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:45:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:45:57 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:45:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:45:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:45:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:45:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:45:58.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:45:58 smithi122 ceph-mon[93869]: pgmap v5882: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 194 KiB/s wr, 11 op/s 2023-03-09T17:45:58.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:45:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10068.351952 secs 2023-03-09T17:45:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:45:58 smithi181 ceph-mon[94189]: pgmap v5882: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 194 KiB/s wr, 11 op/s 2023-03-09T17:45:58.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:45:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10068.351952 secs 2023-03-09T17:45:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:45:58 smithi038 ceph-mon[92422]: pgmap v5882: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 194 KiB/s wr, 11 op/s 2023-03-09T17:45:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:45:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10068.351952 secs 2023-03-09T17:45:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10530.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:46:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:46:00 smithi122 ceph-mon[93869]: pgmap v5883: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 127 KiB/s wr, 9 op/s 2023-03-09T17:46:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:46:00 smithi181 ceph-mon[94189]: pgmap v5883: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 127 KiB/s wr, 9 op/s 2023-03-09T17:46:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:46:00 smithi038 ceph-mon[92422]: pgmap v5883: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 127 KiB/s wr, 9 op/s 2023-03-09T17:46:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:46:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10071.074458 secs 2023-03-09T17:46:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:46:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:46:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:46:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:46:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:46:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10071.074458 secs 2023-03-09T17:46:01.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:46:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:46:01.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:46:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:46:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:46:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10071.074458 secs 2023-03-09T17:46:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:46:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:46:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:46:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:46:02.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:46:02 smithi122 ceph-mon[93869]: pgmap v5884: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 170 KiB/s wr, 11 op/s 2023-03-09T17:46:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:46:02 smithi181 ceph-mon[94189]: pgmap v5884: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 170 KiB/s wr, 11 op/s 2023-03-09T17:46:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:46:02 smithi038 ceph-mon[92422]: pgmap v5884: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 170 KiB/s wr, 11 op/s 2023-03-09T17:46:03.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:46:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10073.352073 secs 2023-03-09T17:46:03.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:46:03 smithi122 ceph-mon[93869]: pgmap v5885: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 125 KiB/s wr, 10 op/s 2023-03-09T17:46:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:46:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10073.352073 secs 2023-03-09T17:46:03.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:46:03 smithi181 ceph-mon[94189]: pgmap v5885: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 125 KiB/s wr, 10 op/s 2023-03-09T17:46:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:46:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10073.352073 secs 2023-03-09T17:46:03.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:46:03 smithi038 ceph-mon[92422]: pgmap v5885: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 125 KiB/s wr, 10 op/s 2023-03-09T17:46:03.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10535.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:46:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:46:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10076.074519 secs 2023-03-09T17:46:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:46:05 smithi122 ceph-mon[93869]: pgmap v5886: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 146 KiB/s wr, 21 op/s 2023-03-09T17:46:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:46:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10076.074519 secs 2023-03-09T17:46:06.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:46:05 smithi181 ceph-mon[94189]: pgmap v5886: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 146 KiB/s wr, 21 op/s 2023-03-09T17:46:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:46:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10076.074519 secs 2023-03-09T17:46:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:46:05 smithi038 ceph-mon[92422]: pgmap v5886: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 146 KiB/s wr, 21 op/s 2023-03-09T17:46:07.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:46:06 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:46:07.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:46:06 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:46:07.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:46:06 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:46:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:46:07 smithi122 ceph-mon[93869]: pgmap v5887: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 142 KiB/s wr, 23 op/s 2023-03-09T17:46:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:46:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10078.352193 secs 2023-03-09T17:46:08.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:46:07 smithi181 ceph-mon[94189]: pgmap v5887: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 142 KiB/s wr, 23 op/s 2023-03-09T17:46:08.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:46:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10078.352193 secs 2023-03-09T17:46:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:46:07 smithi038 ceph-mon[92422]: pgmap v5887: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 142 KiB/s wr, 23 op/s 2023-03-09T17:46:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:46:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10078.352193 secs 2023-03-09T17:46:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10540.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:46:10.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:46:09 smithi122 ceph-mon[93869]: pgmap v5888: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 94 KiB/s wr, 21 op/s 2023-03-09T17:46:10.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:46:09 smithi122 ceph-mon[93869]: osdmap e581: 12 total, 12 up, 12 in 2023-03-09T17:46:10.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:46:09 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:46:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:46:09 smithi181 ceph-mon[94189]: pgmap v5888: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 94 KiB/s wr, 21 op/s 2023-03-09T17:46:10.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:46:09 smithi181 ceph-mon[94189]: osdmap e581: 12 total, 12 up, 12 in 2023-03-09T17:46:10.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:46:09 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:46:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:46:09 smithi038 ceph-mon[92422]: pgmap v5888: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 94 KiB/s wr, 21 op/s 2023-03-09T17:46:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:46:09 smithi038 ceph-mon[92422]: osdmap e581: 12 total, 12 up, 12 in 2023-03-09T17:46:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:46:09 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:46:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:46:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10081.074578 secs 2023-03-09T17:46:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:46:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:46:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:46:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:46:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:46:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:46:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:46:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10081.074578 secs 2023-03-09T17:46:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:46:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:46:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:46:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:46:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:46:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:46:11.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:46:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10081.074578 secs 2023-03-09T17:46:11.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:46:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:46:11.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:46:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:46:11.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:46:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:46:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:46:11 smithi181 ceph-mon[94189]: pgmap v5890: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 75 KiB/s wr, 25 op/s 2023-03-09T17:46:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:46:11 smithi038 ceph-mon[92422]: pgmap v5890: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 75 KiB/s wr, 25 op/s 2023-03-09T17:46:12.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:46:11 smithi122 ceph-mon[93869]: pgmap v5890: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 75 KiB/s wr, 25 op/s 2023-03-09T17:46:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:46:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10083.352312 secs 2023-03-09T17:46:13.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:46:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10083.352312 secs 2023-03-09T17:46:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:46:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10083.352312 secs 2023-03-09T17:46:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10545.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:46:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:46:13 smithi038 ceph-mon[92422]: pgmap v5891: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 75 KiB/s wr, 24 op/s 2023-03-09T17:46:14.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:46:13 smithi122 ceph-mon[93869]: pgmap v5891: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 75 KiB/s wr, 24 op/s 2023-03-09T17:46:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:46:13 smithi181 ceph-mon[94189]: pgmap v5891: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 75 KiB/s wr, 24 op/s 2023-03-09T17:46:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:46:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10086.074639 secs 2023-03-09T17:46:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:46:15 smithi038 ceph-mon[92422]: pgmap v5892: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 60 KiB/s wr, 10 op/s 2023-03-09T17:46:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:46:15 smithi038 ceph-mon[92422]: osdmap e582: 12 total, 12 up, 12 in 2023-03-09T17:46:16.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:46:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10086.074639 secs 2023-03-09T17:46:16.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:46:15 smithi122 ceph-mon[93869]: pgmap v5892: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 60 KiB/s wr, 10 op/s 2023-03-09T17:46:16.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:46:15 smithi122 ceph-mon[93869]: osdmap e582: 12 total, 12 up, 12 in 2023-03-09T17:46:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:46:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10086.074639 secs 2023-03-09T17:46:16.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:46:15 smithi181 ceph-mon[94189]: pgmap v5892: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 60 KiB/s wr, 10 op/s 2023-03-09T17:46:16.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:46:15 smithi181 ceph-mon[94189]: osdmap e582: 12 total, 12 up, 12 in 2023-03-09T17:46:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:46:17 smithi038 ceph-mon[92422]: pgmap v5894: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 46 KiB/s wr, 9 op/s 2023-03-09T17:46:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:46:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10088.352392 secs 2023-03-09T17:46:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:46:17 smithi122 ceph-mon[93869]: pgmap v5894: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 46 KiB/s wr, 9 op/s 2023-03-09T17:46:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:46:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10088.352392 secs 2023-03-09T17:46:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:46:17 smithi181 ceph-mon[94189]: pgmap v5894: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 46 KiB/s wr, 9 op/s 2023-03-09T17:46:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:46:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10088.352392 secs 2023-03-09T17:46:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10550.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:46:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:46:19 smithi038 ceph-mon[92422]: pgmap v5895: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 38 KiB/s wr, 8 op/s 2023-03-09T17:46:20.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:46:19 smithi122 ceph-mon[93869]: pgmap v5895: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 38 KiB/s wr, 8 op/s 2023-03-09T17:46:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:46:19 smithi181 ceph-mon[94189]: pgmap v5895: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 38 KiB/s wr, 8 op/s 2023-03-09T17:46:21.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:46:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10091.074698 secs 2023-03-09T17:46:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:46:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10091.074698 secs 2023-03-09T17:46:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:46:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10091.074698 secs 2023-03-09T17:46:22.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:46:22 smithi122 ceph-mon[93869]: pgmap v5896: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 73 KiB/s wr, 8 op/s 2023-03-09T17:46:22.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:46:22 smithi181 ceph-mon[94189]: pgmap v5896: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 73 KiB/s wr, 8 op/s 2023-03-09T17:46:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:46:22 smithi038 ceph-mon[92422]: pgmap v5896: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 73 KiB/s wr, 8 op/s 2023-03-09T17:46:23.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:46:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10093.352516 secs 2023-03-09T17:46:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:46:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10093.352516 secs 2023-03-09T17:46:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:46:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10093.352516 secs 2023-03-09T17:46:23.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10555.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:46:24.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:46:24 smithi122 ceph-mon[93869]: pgmap v5897: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 8 op/s 2023-03-09T17:46:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:46:24 smithi181 ceph-mon[94189]: pgmap v5897: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 8 op/s 2023-03-09T17:46:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:46:24 smithi038 ceph-mon[92422]: pgmap v5897: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 8 op/s 2023-03-09T17:46:26.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:46:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10096.074761 secs 2023-03-09T17:46:26.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:46:26 smithi122 ceph-mon[93869]: pgmap v5898: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 62 KiB/s wr, 8 op/s 2023-03-09T17:46:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:46:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10096.074761 secs 2023-03-09T17:46:26.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:46:26 smithi181 ceph-mon[94189]: pgmap v5898: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 62 KiB/s wr, 8 op/s 2023-03-09T17:46:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:46:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10096.074761 secs 2023-03-09T17:46:26.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:46:26 smithi038 ceph-mon[92422]: pgmap v5898: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 62 KiB/s wr, 8 op/s 2023-03-09T17:46:28.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:46:28 smithi122 ceph-mon[93869]: pgmap v5899: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 8 op/s 2023-03-09T17:46:28.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:46:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10098.352633 secs 2023-03-09T17:46:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:46:28 smithi181 ceph-mon[94189]: pgmap v5899: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 8 op/s 2023-03-09T17:46:28.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:46:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10098.352633 secs 2023-03-09T17:46:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:46:28 smithi038 ceph-mon[92422]: pgmap v5899: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 8 op/s 2023-03-09T17:46:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:46:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10098.352633 secs 2023-03-09T17:46:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10560.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:46:30.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:46:30 smithi122 ceph-mon[93869]: pgmap v5900: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 51 KiB/s wr, 8 op/s 2023-03-09T17:46:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:46:30 smithi181 ceph-mon[94189]: pgmap v5900: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 51 KiB/s wr, 8 op/s 2023-03-09T17:46:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:46:30 smithi038 ceph-mon[92422]: pgmap v5900: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 51 KiB/s wr, 8 op/s 2023-03-09T17:46:31.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:46:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10101.074844 secs 2023-03-09T17:46:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:46:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10101.074844 secs 2023-03-09T17:46:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:46:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10101.074844 secs 2023-03-09T17:46:32.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:46:32 smithi122 ceph-mon[93869]: pgmap v5901: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 51 KiB/s wr, 10 op/s 2023-03-09T17:46:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:46:32 smithi181 ceph-mon[94189]: pgmap v5901: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 51 KiB/s wr, 10 op/s 2023-03-09T17:46:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:46:32 smithi038 ceph-mon[92422]: pgmap v5901: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 51 KiB/s wr, 10 op/s 2023-03-09T17:46:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10565.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:46:34.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:46:34 smithi122 ceph-mon[93869]: pgmap v5902: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 853 B/s wr, 7 op/s 2023-03-09T17:46:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:46:34 smithi181 ceph-mon[94189]: pgmap v5902: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 853 B/s wr, 7 op/s 2023-03-09T17:46:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:46:34 smithi038 ceph-mon[92422]: pgmap v5902: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 853 B/s wr, 7 op/s 2023-03-09T17:46:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:46:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10106.074904 secs 2023-03-09T17:46:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:46:36 smithi122 ceph-mon[93869]: pgmap v5903: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 767 B/s wr, 7 op/s 2023-03-09T17:46:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:46:36 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:46:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:46:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10106.074904 secs 2023-03-09T17:46:36.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:46:36 smithi181 ceph-mon[94189]: pgmap v5903: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 767 B/s wr, 7 op/s 2023-03-09T17:46:36.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:46:36 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:46:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:46:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10106.074904 secs 2023-03-09T17:46:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:46:36 smithi038 ceph-mon[92422]: pgmap v5903: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 767 B/s wr, 7 op/s 2023-03-09T17:46:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:46:36 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:46:37.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:46:37 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:46:37.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:46:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:46:37.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:46:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:46:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:46:37 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:46:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:46:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:46:37.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:46:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:46:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:46:37 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:46:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:46:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:46:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:46:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:46:38.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:46:38 smithi122 ceph-mon[93869]: pgmap v5904: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 62 KiB/s wr, 9 op/s 2023-03-09T17:46:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:46:38 smithi181 ceph-mon[94189]: pgmap v5904: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 62 KiB/s wr, 9 op/s 2023-03-09T17:46:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:46:38 smithi038 ceph-mon[92422]: pgmap v5904: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 62 KiB/s wr, 9 op/s 2023-03-09T17:46:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10570.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:46:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:46:40 smithi122 ceph-mon[93869]: pgmap v5905: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 70 KiB/s wr, 8 op/s 2023-03-09T17:46:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:46:40 smithi181 ceph-mon[94189]: pgmap v5905: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 70 KiB/s wr, 8 op/s 2023-03-09T17:46:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:46:40 smithi038 ceph-mon[92422]: pgmap v5905: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 70 KiB/s wr, 8 op/s 2023-03-09T17:46:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:46:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10111.074967 secs 2023-03-09T17:46:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:46:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10111.074967 secs 2023-03-09T17:46:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:46:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10111.074967 secs 2023-03-09T17:46:42.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:46:42 smithi122 ceph-mon[93869]: pgmap v5906: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 96 KiB/s wr, 10 op/s 2023-03-09T17:46:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:46:42 smithi181 ceph-mon[94189]: pgmap v5906: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 96 KiB/s wr, 10 op/s 2023-03-09T17:46:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:46:42 smithi038 ceph-mon[92422]: pgmap v5906: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 96 KiB/s wr, 10 op/s 2023-03-09T17:46:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10575.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:46:44.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:46:44 smithi122 ceph-mon[93869]: pgmap v5907: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 96 KiB/s wr, 8 op/s 2023-03-09T17:46:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:46:44 smithi181 ceph-mon[94189]: pgmap v5907: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 96 KiB/s wr, 8 op/s 2023-03-09T17:46:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:46:44 smithi038 ceph-mon[92422]: pgmap v5907: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 96 KiB/s wr, 8 op/s 2023-03-09T17:46:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:46:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10116.074984 secs 2023-03-09T17:46:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:46:46 smithi122 ceph-mon[93869]: pgmap v5908: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 104 KiB/s wr, 9 op/s 2023-03-09T17:46:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:46:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10116.074984 secs 2023-03-09T17:46:46.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:46:46 smithi181 ceph-mon[94189]: pgmap v5908: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 104 KiB/s wr, 9 op/s 2023-03-09T17:46:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:46:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10116.074984 secs 2023-03-09T17:46:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:46:46 smithi038 ceph-mon[92422]: pgmap v5908: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 104 KiB/s wr, 9 op/s 2023-03-09T17:46:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:46:48 smithi122 ceph-mon[93869]: pgmap v5909: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 172 KiB/s wr, 12 op/s 2023-03-09T17:46:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:46:48 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 10118.353099 secs 2023-03-09T17:46:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:46:48 smithi181 ceph-mon[94189]: pgmap v5909: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 172 KiB/s wr, 12 op/s 2023-03-09T17:46:48.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:46:48 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 10118.353099 secs 2023-03-09T17:46:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:46:48 smithi038 ceph-mon[92422]: pgmap v5909: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 172 KiB/s wr, 12 op/s 2023-03-09T17:46:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:46:48 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 10118.353099 secs 2023-03-09T17:46:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10580.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:46:49.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:46:49 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:46:49.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:46:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:46:49.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:46:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:46:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:46:49 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:46:49.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:46:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:46:49.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:46:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:46:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:46:49 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:46:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:46:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:46:49.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:46:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:46:50.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:46:50 smithi122 ceph-mon[93869]: pgmap v5910: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 120 KiB/s wr, 10 op/s 2023-03-09T17:46:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:46:50 smithi181 ceph-mon[94189]: pgmap v5910: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 120 KiB/s wr, 10 op/s 2023-03-09T17:46:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:46:50 smithi038 ceph-mon[92422]: pgmap v5910: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 120 KiB/s wr, 10 op/s 2023-03-09T17:46:51.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:46:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10121.075046 secs 2023-03-09T17:46:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:46:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10121.075046 secs 2023-03-09T17:46:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:46:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10121.075046 secs 2023-03-09T17:46:52.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:46:52 smithi122 ceph-mon[93869]: pgmap v5911: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 140 KiB/s wr, 11 op/s 2023-03-09T17:46:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:46:52 smithi181 ceph-mon[94189]: pgmap v5911: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 140 KiB/s wr, 11 op/s 2023-03-09T17:46:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:46:52 smithi038 ceph-mon[92422]: pgmap v5911: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 140 KiB/s wr, 11 op/s 2023-03-09T17:46:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:46:53 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 10123.353215 secs 2023-03-09T17:46:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:46:53 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 10123.353215 secs 2023-03-09T17:46:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:46:53 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 10123.353215 secs 2023-03-09T17:46:53.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10585.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:46:54.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:46:54 smithi122 ceph-mon[93869]: pgmap v5912: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 114 KiB/s wr, 8 op/s 2023-03-09T17:46:54.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:46:54 smithi181 ceph-mon[94189]: pgmap v5912: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 114 KiB/s wr, 8 op/s 2023-03-09T17:46:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:46:54 smithi038 ceph-mon[92422]: pgmap v5912: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 114 KiB/s wr, 8 op/s 2023-03-09T17:46:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:46:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10126.075105 secs 2023-03-09T17:46:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:46:56 smithi122 ceph-mon[93869]: pgmap v5913: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 121 KiB/s wr, 8 op/s 2023-03-09T17:46:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:46:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10126.075105 secs 2023-03-09T17:46:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:46:56 smithi181 ceph-mon[94189]: pgmap v5913: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 121 KiB/s wr, 8 op/s 2023-03-09T17:46:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:46:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10126.075105 secs 2023-03-09T17:46:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:46:56 smithi038 ceph-mon[92422]: pgmap v5913: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 121 KiB/s wr, 8 op/s 2023-03-09T17:46:58.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:46:58 smithi122 ceph-mon[93869]: pgmap v5914: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 181 KiB/s wr, 10 op/s 2023-03-09T17:46:58.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:46:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10128.353358 secs 2023-03-09T17:46:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:46:58 smithi181 ceph-mon[94189]: pgmap v5914: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 181 KiB/s wr, 10 op/s 2023-03-09T17:46:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:46:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10128.353358 secs 2023-03-09T17:46:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:46:58 smithi038 ceph-mon[92422]: pgmap v5914: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 181 KiB/s wr, 10 op/s 2023-03-09T17:46:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:46:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10128.353358 secs 2023-03-09T17:46:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10590.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:47:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:47:00 smithi122 ceph-mon[93869]: pgmap v5915: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 122 KiB/s wr, 8 op/s 2023-03-09T17:47:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:47:00 smithi181 ceph-mon[94189]: pgmap v5915: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 122 KiB/s wr, 8 op/s 2023-03-09T17:47:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:47:00 smithi038 ceph-mon[92422]: pgmap v5915: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 122 KiB/s wr, 8 op/s 2023-03-09T17:47:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:47:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10131.075165 secs 2023-03-09T17:47:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:47:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:47:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:47:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:47:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:47:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10131.075165 secs 2023-03-09T17:47:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:47:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:47:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:47:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:47:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:47:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10131.075165 secs 2023-03-09T17:47:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:47:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:47:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:47:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:47:02.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:47:02 smithi122 ceph-mon[93869]: pgmap v5916: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 133 KiB/s wr, 9 op/s 2023-03-09T17:47:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:47:02 smithi181 ceph-mon[94189]: pgmap v5916: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 133 KiB/s wr, 9 op/s 2023-03-09T17:47:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:47:02 smithi038 ceph-mon[92422]: pgmap v5916: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 133 KiB/s wr, 9 op/s 2023-03-09T17:47:03.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:47:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10133.353477 secs 2023-03-09T17:47:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:47:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10133.353477 secs 2023-03-09T17:47:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:47:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10133.353477 secs 2023-03-09T17:47:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10595.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:47:04.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:47:04 smithi122 ceph-mon[93869]: pgmap v5917: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 105 KiB/s wr, 7 op/s 2023-03-09T17:47:04.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:47:04 smithi181 ceph-mon[94189]: pgmap v5917: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 105 KiB/s wr, 7 op/s 2023-03-09T17:47:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:47:04 smithi038 ceph-mon[92422]: pgmap v5917: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 105 KiB/s wr, 7 op/s 2023-03-09T17:47:05.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:47:05 smithi122 ceph-mon[93869]: osdmap e583: 12 total, 12 up, 12 in 2023-03-09T17:47:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:47:05 smithi181 ceph-mon[94189]: osdmap e583: 12 total, 12 up, 12 in 2023-03-09T17:47:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:47:05 smithi038 ceph-mon[92422]: osdmap e583: 12 total, 12 up, 12 in 2023-03-09T17:47:06.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:47:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10136.075224 secs 2023-03-09T17:47:06.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:47:06 smithi122 ceph-mon[93869]: pgmap v5919: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 152 KiB/s wr, 23 op/s 2023-03-09T17:47:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:47:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10136.075224 secs 2023-03-09T17:47:06.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:47:06 smithi181 ceph-mon[94189]: pgmap v5919: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 152 KiB/s wr, 23 op/s 2023-03-09T17:47:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:47:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10136.075224 secs 2023-03-09T17:47:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:47:06 smithi038 ceph-mon[92422]: pgmap v5919: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 152 KiB/s wr, 23 op/s 2023-03-09T17:47:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:47:08 smithi122 ceph-mon[93869]: pgmap v5920: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 144 KiB/s wr, 27 op/s 2023-03-09T17:47:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:47:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10138.353608 secs 2023-03-09T17:47:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:47:08 smithi181 ceph-mon[94189]: pgmap v5920: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 144 KiB/s wr, 27 op/s 2023-03-09T17:47:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:47:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10138.353608 secs 2023-03-09T17:47:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:47:08 smithi038 ceph-mon[92422]: pgmap v5920: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 144 KiB/s wr, 27 op/s 2023-03-09T17:47:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:47:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10138.353608 secs 2023-03-09T17:47:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10600.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:47:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:47:10 smithi122 ceph-mon[93869]: pgmap v5921: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 136 KiB/s wr, 26 op/s 2023-03-09T17:47:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:47:10 smithi181 ceph-mon[94189]: pgmap v5921: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 136 KiB/s wr, 26 op/s 2023-03-09T17:47:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:47:10 smithi038 ceph-mon[92422]: pgmap v5921: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 136 KiB/s wr, 26 op/s 2023-03-09T17:47:11.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:47:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10141.075281 secs 2023-03-09T17:47:11.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:47:11 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:47:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:47:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10141.075281 secs 2023-03-09T17:47:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:47:11 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:47:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:47:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10141.075281 secs 2023-03-09T17:47:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:47:11 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:47:12.464 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:47:12 smithi038 ceph-mon[92422]: pgmap v5922: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 112 KiB/s wr, 27 op/s 2023-03-09T17:47:12.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:47:12 smithi122 ceph-mon[93869]: pgmap v5922: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 112 KiB/s wr, 27 op/s 2023-03-09T17:47:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:47:12 smithi181 ceph-mon[94189]: pgmap v5922: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 112 KiB/s wr, 27 op/s 2023-03-09T17:47:13.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:47:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10143.353722 secs 2023-03-09T17:47:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:47:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10143.353722 secs 2023-03-09T17:47:13.491 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:47:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10143.353722 secs 2023-03-09T17:47:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10605.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:47:14.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:47:14 smithi122 ceph-mon[93869]: pgmap v5923: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 144 KiB/s wr, 27 op/s 2023-03-09T17:47:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:47:14 smithi181 ceph-mon[94189]: pgmap v5923: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 144 KiB/s wr, 27 op/s 2023-03-09T17:47:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:47:14 smithi038 ceph-mon[92422]: pgmap v5923: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 144 KiB/s wr, 27 op/s 2023-03-09T17:47:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:47:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:47:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:47:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:47:15.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:47:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:47:15.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:47:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:47:15.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:47:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:47:15.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:47:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10146.075343 secs 2023-03-09T17:47:15.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:47:15 smithi122 ceph-mon[93869]: pgmap v5924: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 111 KiB/s wr, 12 op/s 2023-03-09T17:47:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:47:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:47:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:47:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:47:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:47:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:47:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:47:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:47:15.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:47:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:47:15.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:47:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10146.075343 secs 2023-03-09T17:47:15.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:47:15 smithi181 ceph-mon[94189]: pgmap v5924: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 111 KiB/s wr, 12 op/s 2023-03-09T17:47:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:47:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:47:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:47:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:47:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:47:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:47:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:47:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:47:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:47:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:47:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:47:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10146.075343 secs 2023-03-09T17:47:16.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:47:15 smithi038 ceph-mon[92422]: pgmap v5924: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 111 KiB/s wr, 12 op/s 2023-03-09T17:47:17.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:47:16 smithi038 ceph-mon[92422]: osdmap e584: 12 total, 12 up, 12 in 2023-03-09T17:47:17.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:47:16 smithi122 ceph-mon[93869]: osdmap e584: 12 total, 12 up, 12 in 2023-03-09T17:47:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:47:16 smithi181 ceph-mon[94189]: osdmap e584: 12 total, 12 up, 12 in 2023-03-09T17:47:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:47:18 smithi122 ceph-mon[93869]: pgmap v5926: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 8 op/s 2023-03-09T17:47:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:47:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10148.353872 secs 2023-03-09T17:47:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:47:18 smithi181 ceph-mon[94189]: pgmap v5926: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 8 op/s 2023-03-09T17:47:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:47:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10148.353872 secs 2023-03-09T17:47:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:47:18 smithi038 ceph-mon[92422]: pgmap v5926: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 8 op/s 2023-03-09T17:47:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:47:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10148.353872 secs 2023-03-09T17:47:18.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10610.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:47:20.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:47:20 smithi122 ceph-mon[93869]: pgmap v5927: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 75 KiB/s wr, 9 op/s 2023-03-09T17:47:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:47:20 smithi181 ceph-mon[94189]: pgmap v5927: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 75 KiB/s wr, 9 op/s 2023-03-09T17:47:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:47:20 smithi038 ceph-mon[92422]: pgmap v5927: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 75 KiB/s wr, 9 op/s 2023-03-09T17:47:21.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:47:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10151.075402 secs 2023-03-09T17:47:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:47:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10151.075402 secs 2023-03-09T17:47:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:47:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10151.075402 secs 2023-03-09T17:47:22.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:47:22 smithi122 ceph-mon[93869]: pgmap v5928: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 74 KiB/s wr, 9 op/s 2023-03-09T17:47:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:47:22 smithi181 ceph-mon[94189]: pgmap v5928: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 74 KiB/s wr, 9 op/s 2023-03-09T17:47:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:47:22 smithi038 ceph-mon[92422]: pgmap v5928: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 74 KiB/s wr, 9 op/s 2023-03-09T17:47:23.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:47:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10153.353988 secs 2023-03-09T17:47:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:47:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10153.353988 secs 2023-03-09T17:47:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:47:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10153.353988 secs 2023-03-09T17:47:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10615.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:47:24.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:47:24 smithi122 ceph-mon[93869]: pgmap v5929: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 60 KiB/s wr, 8 op/s 2023-03-09T17:47:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:47:24 smithi181 ceph-mon[94189]: pgmap v5929: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 60 KiB/s wr, 8 op/s 2023-03-09T17:47:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:47:24 smithi038 ceph-mon[92422]: pgmap v5929: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 60 KiB/s wr, 8 op/s 2023-03-09T17:47:26.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:47:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10156.075463 secs 2023-03-09T17:47:26.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:47:26 smithi122 ceph-mon[93869]: pgmap v5930: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 52 KiB/s wr, 8 op/s 2023-03-09T17:47:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:47:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10156.075463 secs 2023-03-09T17:47:26.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:47:26 smithi181 ceph-mon[94189]: pgmap v5930: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 52 KiB/s wr, 8 op/s 2023-03-09T17:47:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:47:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10156.075463 secs 2023-03-09T17:47:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:47:26 smithi038 ceph-mon[92422]: pgmap v5930: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 52 KiB/s wr, 8 op/s 2023-03-09T17:47:28.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:47:28 smithi122 ceph-mon[93869]: pgmap v5931: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 48 KiB/s wr, 7 op/s 2023-03-09T17:47:28.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:47:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10158.354119 secs 2023-03-09T17:47:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:47:28 smithi181 ceph-mon[94189]: pgmap v5931: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 48 KiB/s wr, 7 op/s 2023-03-09T17:47:28.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:47:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10158.354119 secs 2023-03-09T17:47:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:47:28 smithi038 ceph-mon[92422]: pgmap v5931: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 48 KiB/s wr, 7 op/s 2023-03-09T17:47:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:47:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10158.354119 secs 2023-03-09T17:47:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10620.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:47:30.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:47:30 smithi122 ceph-mon[93869]: pgmap v5932: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 67 KiB/s wr, 7 op/s 2023-03-09T17:47:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:47:30 smithi181 ceph-mon[94189]: pgmap v5932: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 67 KiB/s wr, 7 op/s 2023-03-09T17:47:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:47:30 smithi038 ceph-mon[92422]: pgmap v5932: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 67 KiB/s wr, 7 op/s 2023-03-09T17:47:31.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:47:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10161.075523 secs 2023-03-09T17:47:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:47:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10161.075523 secs 2023-03-09T17:47:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:47:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10161.075523 secs 2023-03-09T17:47:32.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:47:32 smithi122 ceph-mon[93869]: pgmap v5933: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 40 KiB/s wr, 8 op/s 2023-03-09T17:47:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:47:32 smithi181 ceph-mon[94189]: pgmap v5933: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 40 KiB/s wr, 8 op/s 2023-03-09T17:47:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:47:32 smithi038 ceph-mon[92422]: pgmap v5933: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 40 KiB/s wr, 8 op/s 2023-03-09T17:47:33.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:47:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10163.354218 secs 2023-03-09T17:47:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:47:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10163.354218 secs 2023-03-09T17:47:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:47:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10163.354218 secs 2023-03-09T17:47:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10625.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:47:34.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:47:34 smithi122 ceph-mon[93869]: pgmap v5934: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 39 KiB/s wr, 6 op/s 2023-03-09T17:47:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:47:34 smithi181 ceph-mon[94189]: pgmap v5934: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 39 KiB/s wr, 6 op/s 2023-03-09T17:47:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:47:34 smithi038 ceph-mon[92422]: pgmap v5934: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 39 KiB/s wr, 6 op/s 2023-03-09T17:47:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:47:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10166.075607 secs 2023-03-09T17:47:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:47:36 smithi122 ceph-mon[93869]: pgmap v5935: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 24 KiB/s wr, 6 op/s 2023-03-09T17:47:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:47:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10166.075607 secs 2023-03-09T17:47:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:47:36 smithi181 ceph-mon[94189]: pgmap v5935: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 24 KiB/s wr, 6 op/s 2023-03-09T17:47:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:47:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10166.075607 secs 2023-03-09T17:47:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:47:36 smithi038 ceph-mon[92422]: pgmap v5935: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 24 KiB/s wr, 6 op/s 2023-03-09T17:47:38.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:47:38 smithi122 ceph-mon[93869]: pgmap v5936: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T17:47:38.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:47:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10168.354335 secs 2023-03-09T17:47:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:47:38 smithi181 ceph-mon[94189]: pgmap v5936: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T17:47:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:47:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10168.354335 secs 2023-03-09T17:47:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:47:38 smithi038 ceph-mon[92422]: pgmap v5936: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T17:47:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:47:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10168.354335 secs 2023-03-09T17:47:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10630.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:47:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:47:40 smithi122 ceph-mon[93869]: pgmap v5937: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 51 KiB/s wr, 6 op/s 2023-03-09T17:47:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:47:40 smithi181 ceph-mon[94189]: pgmap v5937: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 51 KiB/s wr, 6 op/s 2023-03-09T17:47:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:47:40 smithi038 ceph-mon[92422]: pgmap v5937: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 51 KiB/s wr, 6 op/s 2023-03-09T17:47:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:47:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10171.075668 secs 2023-03-09T17:47:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:47:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10171.075668 secs 2023-03-09T17:47:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:47:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10171.075668 secs 2023-03-09T17:47:42.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:47:42 smithi122 ceph-mon[93869]: pgmap v5938: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T17:47:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:47:42 smithi181 ceph-mon[94189]: pgmap v5938: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T17:47:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:47:42 smithi038 ceph-mon[92422]: pgmap v5938: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T17:47:43.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:47:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10173.354419 secs 2023-03-09T17:47:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:47:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10173.354419 secs 2023-03-09T17:47:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:47:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10173.354419 secs 2023-03-09T17:47:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10635.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:47:44.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:47:44 smithi122 ceph-mon[93869]: pgmap v5939: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 74 KiB/s wr, 6 op/s 2023-03-09T17:47:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:47:44 smithi181 ceph-mon[94189]: pgmap v5939: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 74 KiB/s wr, 6 op/s 2023-03-09T17:47:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:47:44 smithi038 ceph-mon[92422]: pgmap v5939: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 74 KiB/s wr, 6 op/s 2023-03-09T17:47:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:47:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10176.075725 secs 2023-03-09T17:47:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:47:46 smithi122 ceph-mon[93869]: pgmap v5940: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 82 KiB/s wr, 7 op/s 2023-03-09T17:47:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:47:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10176.075725 secs 2023-03-09T17:47:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:47:46 smithi181 ceph-mon[94189]: pgmap v5940: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 82 KiB/s wr, 7 op/s 2023-03-09T17:47:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:47:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10176.075725 secs 2023-03-09T17:47:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:47:46 smithi038 ceph-mon[92422]: pgmap v5940: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 82 KiB/s wr, 7 op/s 2023-03-09T17:47:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:47:48 smithi122 ceph-mon[93869]: pgmap v5941: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 82 KiB/s wr, 8 op/s 2023-03-09T17:47:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:47:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10178.354539 secs 2023-03-09T17:47:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:47:48 smithi181 ceph-mon[94189]: pgmap v5941: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 82 KiB/s wr, 8 op/s 2023-03-09T17:47:48.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:47:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10178.354539 secs 2023-03-09T17:47:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:47:48 smithi038 ceph-mon[92422]: pgmap v5941: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 82 KiB/s wr, 8 op/s 2023-03-09T17:47:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:47:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10178.354539 secs 2023-03-09T17:47:48.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10640.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:47:50.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:47:50 smithi122 ceph-mon[93869]: pgmap v5942: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 125 KiB/s wr, 9 op/s 2023-03-09T17:47:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:47:50 smithi181 ceph-mon[94189]: pgmap v5942: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 125 KiB/s wr, 9 op/s 2023-03-09T17:47:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:47:50 smithi038 ceph-mon[92422]: pgmap v5942: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 125 KiB/s wr, 9 op/s 2023-03-09T17:47:51.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:47:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10181.075785 secs 2023-03-09T17:47:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:47:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10181.075785 secs 2023-03-09T17:47:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:47:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10181.075785 secs 2023-03-09T17:47:52.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:47:52 smithi122 ceph-mon[93869]: pgmap v5943: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 99 KiB/s wr, 10 op/s 2023-03-09T17:47:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:47:52 smithi181 ceph-mon[94189]: pgmap v5943: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 99 KiB/s wr, 10 op/s 2023-03-09T17:47:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:47:52 smithi038 ceph-mon[92422]: pgmap v5943: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 99 KiB/s wr, 10 op/s 2023-03-09T17:47:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:47:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10183.354612 secs 2023-03-09T17:47:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:47:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10183.354612 secs 2023-03-09T17:47:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:47:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10183.354612 secs 2023-03-09T17:47:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10645.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:47:54.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:47:54 smithi122 ceph-mon[93869]: pgmap v5944: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 165 KiB/s wr, 9 op/s 2023-03-09T17:47:54.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:47:54 smithi181 ceph-mon[94189]: pgmap v5944: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 165 KiB/s wr, 9 op/s 2023-03-09T17:47:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:47:54 smithi038 ceph-mon[92422]: pgmap v5944: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 165 KiB/s wr, 9 op/s 2023-03-09T17:47:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:47:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10186.075846 secs 2023-03-09T17:47:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:47:56 smithi122 ceph-mon[93869]: pgmap v5945: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 126 KiB/s wr, 9 op/s 2023-03-09T17:47:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:47:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10186.075846 secs 2023-03-09T17:47:56.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:47:56 smithi181 ceph-mon[94189]: pgmap v5945: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 126 KiB/s wr, 9 op/s 2023-03-09T17:47:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:47:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10186.075846 secs 2023-03-09T17:47:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:47:56 smithi038 ceph-mon[92422]: pgmap v5945: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 126 KiB/s wr, 9 op/s 2023-03-09T17:47:58.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:47:58 smithi122 ceph-mon[93869]: pgmap v5946: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 118 KiB/s wr, 10 op/s 2023-03-09T17:47:58.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:47:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10188.354730 secs 2023-03-09T17:47:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:47:58 smithi181 ceph-mon[94189]: pgmap v5946: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 118 KiB/s wr, 10 op/s 2023-03-09T17:47:58.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:47:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10188.354730 secs 2023-03-09T17:47:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:47:58 smithi038 ceph-mon[92422]: pgmap v5946: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 118 KiB/s wr, 10 op/s 2023-03-09T17:47:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:47:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10188.354730 secs 2023-03-09T17:47:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10650.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:48:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:48:00 smithi122 ceph-mon[93869]: pgmap v5947: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 202 KiB/s wr, 11 op/s 2023-03-09T17:48:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:48:00 smithi181 ceph-mon[94189]: pgmap v5947: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 202 KiB/s wr, 11 op/s 2023-03-09T17:48:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:48:00 smithi038 ceph-mon[92422]: pgmap v5947: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 202 KiB/s wr, 11 op/s 2023-03-09T17:48:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:48:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10191.075907 secs 2023-03-09T17:48:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:48:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10191.075907 secs 2023-03-09T17:48:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:48:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10191.075907 secs 2023-03-09T17:48:02.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:48:02 smithi122 ceph-mon[93869]: pgmap v5948: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 159 KiB/s wr, 10 op/s 2023-03-09T17:48:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:48:02 smithi181 ceph-mon[94189]: pgmap v5948: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 159 KiB/s wr, 10 op/s 2023-03-09T17:48:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:48:02 smithi038 ceph-mon[92422]: pgmap v5948: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 159 KiB/s wr, 10 op/s 2023-03-09T17:48:03.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:48:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10193.354846 secs 2023-03-09T17:48:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:48:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10193.354846 secs 2023-03-09T17:48:03.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:48:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10193.354846 secs 2023-03-09T17:48:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10655.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:48:04.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:48:04 smithi122 ceph-mon[93869]: pgmap v5949: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 257 KiB/s wr, 11 op/s 2023-03-09T17:48:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:48:04 smithi181 ceph-mon[94189]: pgmap v5949: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 257 KiB/s wr, 11 op/s 2023-03-09T17:48:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:48:04 smithi038 ceph-mon[92422]: pgmap v5949: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 257 KiB/s wr, 11 op/s 2023-03-09T17:48:05.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:48:05 smithi122 ceph-mon[93869]: osdmap e585: 12 total, 12 up, 12 in 2023-03-09T17:48:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:48:05 smithi181 ceph-mon[94189]: osdmap e585: 12 total, 12 up, 12 in 2023-03-09T17:48:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:48:05 smithi038 ceph-mon[92422]: osdmap e585: 12 total, 12 up, 12 in 2023-03-09T17:48:06.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:48:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10196.075967 secs 2023-03-09T17:48:06.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:48:06 smithi122 ceph-mon[93869]: pgmap v5951: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 264 KiB/s wr, 26 op/s 2023-03-09T17:48:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:48:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10196.075967 secs 2023-03-09T17:48:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:48:06 smithi181 ceph-mon[94189]: pgmap v5951: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 264 KiB/s wr, 26 op/s 2023-03-09T17:48:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:48:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10196.075967 secs 2023-03-09T17:48:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:48:06 smithi038 ceph-mon[92422]: pgmap v5951: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 264 KiB/s wr, 26 op/s 2023-03-09T17:48:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:48:08 smithi122 ceph-mon[93869]: pgmap v5952: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 291 KiB/s wr, 29 op/s 2023-03-09T17:48:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:48:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10198.354963 secs 2023-03-09T17:48:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:48:08 smithi181 ceph-mon[94189]: pgmap v5952: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 291 KiB/s wr, 29 op/s 2023-03-09T17:48:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:48:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10198.354963 secs 2023-03-09T17:48:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:48:08 smithi038 ceph-mon[92422]: pgmap v5952: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 291 KiB/s wr, 29 op/s 2023-03-09T17:48:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:48:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10198.354963 secs 2023-03-09T17:48:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10660.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:48:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:48:10 smithi122 ceph-mon[93869]: pgmap v5953: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 221 KiB/s wr, 29 op/s 2023-03-09T17:48:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:48:10 smithi181 ceph-mon[94189]: pgmap v5953: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 221 KiB/s wr, 29 op/s 2023-03-09T17:48:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:48:10 smithi038 ceph-mon[92422]: pgmap v5953: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 221 KiB/s wr, 29 op/s 2023-03-09T17:48:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:48:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10201.075981 secs 2023-03-09T17:48:11.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:48:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10201.075981 secs 2023-03-09T17:48:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:48:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10201.075981 secs 2023-03-09T17:48:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:48:12 smithi038 ceph-mon[92422]: pgmap v5954: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 223 KiB/s wr, 29 op/s 2023-03-09T17:48:12.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:48:12 smithi122 ceph-mon[93869]: pgmap v5954: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 223 KiB/s wr, 29 op/s 2023-03-09T17:48:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:48:12 smithi181 ceph-mon[94189]: pgmap v5954: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 223 KiB/s wr, 29 op/s 2023-03-09T17:48:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:48:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10203.355088 secs 2023-03-09T17:48:13.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:48:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10203.355088 secs 2023-03-09T17:48:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:48:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10203.355088 secs 2023-03-09T17:48:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10665.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:48:14.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:48:14 smithi122 ceph-mon[93869]: pgmap v5955: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 175 KiB/s wr, 27 op/s 2023-03-09T17:48:14.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:48:14 smithi181 ceph-mon[94189]: pgmap v5955: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 175 KiB/s wr, 27 op/s 2023-03-09T17:48:14.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:48:14 smithi038 ceph-mon[92422]: pgmap v5955: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 175 KiB/s wr, 27 op/s 2023-03-09T17:48:15.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:48:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:48:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:48:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:48:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:48:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:48:16.601 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:48:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10206.076063 secs 2023-03-09T17:48:16.601 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:48:16 smithi122 ceph-mon[93869]: pgmap v5956: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 131 KiB/s wr, 11 op/s 2023-03-09T17:48:16.602 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:48:16 smithi122 ceph-mon[93869]: osdmap e586: 12 total, 12 up, 12 in 2023-03-09T17:48:16.644 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:48:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10206.076063 secs 2023-03-09T17:48:16.644 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:48:16 smithi181 ceph-mon[94189]: pgmap v5956: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 131 KiB/s wr, 11 op/s 2023-03-09T17:48:16.644 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:48:16 smithi181 ceph-mon[94189]: osdmap e586: 12 total, 12 up, 12 in 2023-03-09T17:48:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:48:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10206.076063 secs 2023-03-09T17:48:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:48:16 smithi038 ceph-mon[92422]: pgmap v5956: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 131 KiB/s wr, 11 op/s 2023-03-09T17:48:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:48:16 smithi038 ceph-mon[92422]: osdmap e586: 12 total, 12 up, 12 in 2023-03-09T17:48:18.364 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:48:18 smithi181 ceph-mon[94189]: pgmap v5958: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 112 KiB/s wr, 9 op/s 2023-03-09T17:48:18.364 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:48:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10208.355215 secs 2023-03-09T17:48:18.714 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:48:18 smithi122 ceph-mon[93869]: pgmap v5958: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 112 KiB/s wr, 9 op/s 2023-03-09T17:48:18.714 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:48:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10208.355215 secs 2023-03-09T17:48:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:48:18 smithi038 ceph-mon[92422]: pgmap v5958: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 112 KiB/s wr, 9 op/s 2023-03-09T17:48:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:48:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10208.355215 secs 2023-03-09T17:48:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10670.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:48:19.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:48:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:48:19.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:48:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:48:19.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:48:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:48:19.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:48:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:48:19.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:48:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:48:19.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:48:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:48:19.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:48:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:48:19.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:48:19 smithi122 ceph-mon[93869]: pgmap v5959: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 103 KiB/s wr, 8 op/s 2023-03-09T17:48:19.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:48:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:48:19.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:48:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:48:19.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:48:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:48:19.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:48:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:48:19.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:48:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:48:19.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:48:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:48:19.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:48:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:48:19.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:48:19 smithi181 ceph-mon[94189]: pgmap v5959: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 103 KiB/s wr, 8 op/s 2023-03-09T17:48:19.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:48:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:48:19.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:48:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:48:19.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:48:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:48:19.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:48:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:48:19.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:48:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:48:19.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:48:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:48:19.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:48:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:48:19.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:48:19 smithi038 ceph-mon[92422]: pgmap v5959: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 103 KiB/s wr, 8 op/s 2023-03-09T17:48:21.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:48:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10211.076124 secs 2023-03-09T17:48:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:48:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10211.076124 secs 2023-03-09T17:48:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:48:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10211.076124 secs 2023-03-09T17:48:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:48:21 smithi038 ceph-mon[92422]: pgmap v5960: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 102 KiB/s wr, 8 op/s 2023-03-09T17:48:22.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:48:21 smithi122 ceph-mon[93869]: pgmap v5960: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 102 KiB/s wr, 8 op/s 2023-03-09T17:48:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:48:21 smithi181 ceph-mon[94189]: pgmap v5960: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 102 KiB/s wr, 8 op/s 2023-03-09T17:48:23.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:48:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10213.355337 secs 2023-03-09T17:48:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:48:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10213.355337 secs 2023-03-09T17:48:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:48:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10213.355337 secs 2023-03-09T17:48:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10675.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:48:24.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:48:24 smithi122 ceph-mon[93869]: pgmap v5961: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 33 KiB/s wr, 7 op/s 2023-03-09T17:48:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:48:24 smithi181 ceph-mon[94189]: pgmap v5961: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 33 KiB/s wr, 7 op/s 2023-03-09T17:48:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:48:24 smithi038 ceph-mon[92422]: pgmap v5961: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 33 KiB/s wr, 7 op/s 2023-03-09T17:48:26.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:48:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10216.076184 secs 2023-03-09T17:48:26.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:48:26 smithi122 ceph-mon[93869]: pgmap v5962: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T17:48:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:48:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10216.076184 secs 2023-03-09T17:48:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:48:26 smithi181 ceph-mon[94189]: pgmap v5962: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T17:48:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:48:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10216.076184 secs 2023-03-09T17:48:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:48:26 smithi038 ceph-mon[92422]: pgmap v5962: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T17:48:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:48:28 smithi122 ceph-mon[93869]: pgmap v5963: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 22 KiB/s wr, 6 op/s 2023-03-09T17:48:28.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:48:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10218.355452 secs 2023-03-09T17:48:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:48:28 smithi181 ceph-mon[94189]: pgmap v5963: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 22 KiB/s wr, 6 op/s 2023-03-09T17:48:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:48:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10218.355452 secs 2023-03-09T17:48:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:48:28 smithi038 ceph-mon[92422]: pgmap v5963: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 22 KiB/s wr, 6 op/s 2023-03-09T17:48:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:48:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10218.355452 secs 2023-03-09T17:48:28.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10680.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:48:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:48:30 smithi122 ceph-mon[93869]: pgmap v5964: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 62 KiB/s wr, 8 op/s 2023-03-09T17:48:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:48:30 smithi181 ceph-mon[94189]: pgmap v5964: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 62 KiB/s wr, 8 op/s 2023-03-09T17:48:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:48:30 smithi038 ceph-mon[92422]: pgmap v5964: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 62 KiB/s wr, 8 op/s 2023-03-09T17:48:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:48:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10221.076256 secs 2023-03-09T17:48:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:48:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10221.076256 secs 2023-03-09T17:48:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:48:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10221.076256 secs 2023-03-09T17:48:32.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:48:32 smithi122 ceph-mon[93869]: pgmap v5965: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 45 KiB/s wr, 8 op/s 2023-03-09T17:48:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:48:32 smithi181 ceph-mon[94189]: pgmap v5965: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 45 KiB/s wr, 8 op/s 2023-03-09T17:48:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:48:32 smithi038 ceph-mon[92422]: pgmap v5965: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 45 KiB/s wr, 8 op/s 2023-03-09T17:48:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:48:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10223.355568 secs 2023-03-09T17:48:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:48:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10223.355568 secs 2023-03-09T17:48:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:48:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10223.355568 secs 2023-03-09T17:48:33.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10685.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:48:34.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:48:34 smithi122 ceph-mon[93869]: pgmap v5966: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T17:48:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:48:34 smithi181 ceph-mon[94189]: pgmap v5966: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T17:48:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:48:34 smithi038 ceph-mon[92422]: pgmap v5966: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T17:48:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:48:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10226.076342 secs 2023-03-09T17:48:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:48:36 smithi122 ceph-mon[93869]: pgmap v5967: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T17:48:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:48:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10226.076342 secs 2023-03-09T17:48:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:48:36 smithi181 ceph-mon[94189]: pgmap v5967: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T17:48:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:48:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10226.076342 secs 2023-03-09T17:48:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:48:36 smithi038 ceph-mon[92422]: pgmap v5967: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T17:48:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:48:38 smithi122 ceph-mon[93869]: pgmap v5968: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 44 KiB/s wr, 8 op/s 2023-03-09T17:48:38.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:48:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10228.355686 secs 2023-03-09T17:48:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:48:38 smithi181 ceph-mon[94189]: pgmap v5968: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 44 KiB/s wr, 8 op/s 2023-03-09T17:48:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:48:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10228.355686 secs 2023-03-09T17:48:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:48:38 smithi038 ceph-mon[92422]: pgmap v5968: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 44 KiB/s wr, 8 op/s 2023-03-09T17:48:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:48:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10228.355686 secs 2023-03-09T17:48:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10690.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:48:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:48:40 smithi122 ceph-mon[93869]: pgmap v5969: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 129 KiB/s wr, 9 op/s 2023-03-09T17:48:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:48:40 smithi181 ceph-mon[94189]: pgmap v5969: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 129 KiB/s wr, 9 op/s 2023-03-09T17:48:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:48:40 smithi038 ceph-mon[92422]: pgmap v5969: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 129 KiB/s wr, 9 op/s 2023-03-09T17:48:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:48:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10231.076402 secs 2023-03-09T17:48:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:48:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10231.076402 secs 2023-03-09T17:48:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:48:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10231.076402 secs 2023-03-09T17:48:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:48:42 smithi122 ceph-mon[93869]: pgmap v5970: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 86 KiB/s wr, 10 op/s 2023-03-09T17:48:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:48:42 smithi181 ceph-mon[94189]: pgmap v5970: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 86 KiB/s wr, 10 op/s 2023-03-09T17:48:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:48:42 smithi038 ceph-mon[92422]: pgmap v5970: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 86 KiB/s wr, 10 op/s 2023-03-09T17:48:43.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10695.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:48:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:48:44 smithi122 ceph-mon[93869]: pgmap v5971: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 176 KiB/s wr, 9 op/s 2023-03-09T17:48:44.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:48:44 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:48:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:48:44 smithi181 ceph-mon[94189]: pgmap v5971: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 176 KiB/s wr, 9 op/s 2023-03-09T17:48:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:48:44 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:48:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:48:44 smithi038 ceph-mon[92422]: pgmap v5971: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 176 KiB/s wr, 9 op/s 2023-03-09T17:48:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:48:44 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:48:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:48:45 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:48:45.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:48:45 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:48:45.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:48:45 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:48:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:48:45 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:48:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:48:45 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:48:45.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:48:45 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:48:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:48:45 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:48:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:48:45 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:48:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:48:45 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:48:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:48:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10236.076463 secs 2023-03-09T17:48:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:48:46 smithi122 ceph-mon[93869]: pgmap v5972: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 184 KiB/s wr, 10 op/s 2023-03-09T17:48:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:48:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10236.076463 secs 2023-03-09T17:48:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:48:46 smithi181 ceph-mon[94189]: pgmap v5972: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 184 KiB/s wr, 10 op/s 2023-03-09T17:48:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:48:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10236.076463 secs 2023-03-09T17:48:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:48:46 smithi038 ceph-mon[92422]: pgmap v5972: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 184 KiB/s wr, 10 op/s 2023-03-09T17:48:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:48:48 smithi122 ceph-mon[93869]: pgmap v5973: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 185 KiB/s wr, 12 op/s 2023-03-09T17:48:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:48:48 smithi181 ceph-mon[94189]: pgmap v5973: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 185 KiB/s wr, 12 op/s 2023-03-09T17:48:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:48:48 smithi038 ceph-mon[92422]: pgmap v5973: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 185 KiB/s wr, 12 op/s 2023-03-09T17:48:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10700.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:48:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:48:50 smithi122 ceph-mon[93869]: pgmap v5974: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 206 KiB/s wr, 12 op/s 2023-03-09T17:48:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:48:50 smithi181 ceph-mon[94189]: pgmap v5974: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 206 KiB/s wr, 12 op/s 2023-03-09T17:48:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:48:50 smithi038 ceph-mon[92422]: pgmap v5974: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 206 KiB/s wr, 12 op/s 2023-03-09T17:48:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:48:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10241.076524 secs 2023-03-09T17:48:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:48:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10241.076524 secs 2023-03-09T17:48:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:48:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10241.076524 secs 2023-03-09T17:48:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:48:52 smithi122 ceph-mon[93869]: pgmap v5975: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 121 KiB/s wr, 12 op/s 2023-03-09T17:48:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:48:52 smithi181 ceph-mon[94189]: pgmap v5975: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 121 KiB/s wr, 12 op/s 2023-03-09T17:48:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:48:52 smithi038 ceph-mon[92422]: pgmap v5975: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 121 KiB/s wr, 12 op/s 2023-03-09T17:48:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:48:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10243.356060 secs 2023-03-09T17:48:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:48:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10243.356060 secs 2023-03-09T17:48:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:48:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10243.356060 secs 2023-03-09T17:48:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10705.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:48:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:48:54 smithi122 ceph-mon[93869]: pgmap v5976: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 171 KiB/s wr, 11 op/s 2023-03-09T17:48:54.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:48:54 smithi181 ceph-mon[94189]: pgmap v5976: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 171 KiB/s wr, 11 op/s 2023-03-09T17:48:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:48:54 smithi038 ceph-mon[92422]: pgmap v5976: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 171 KiB/s wr, 11 op/s 2023-03-09T17:48:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:48:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10246.076586 secs 2023-03-09T17:48:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:48:56 smithi122 ceph-mon[93869]: pgmap v5977: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 90 KiB/s wr, 10 op/s 2023-03-09T17:48:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:48:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10246.076586 secs 2023-03-09T17:48:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:48:56 smithi181 ceph-mon[94189]: pgmap v5977: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 90 KiB/s wr, 10 op/s 2023-03-09T17:48:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:48:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10246.076586 secs 2023-03-09T17:48:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:48:56 smithi038 ceph-mon[92422]: pgmap v5977: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 90 KiB/s wr, 10 op/s 2023-03-09T17:48:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:48:57 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:48:57.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:48:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:48:57.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:48:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:48:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:48:57 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:48:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:48:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:48:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:48:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:48:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:48:57 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:48:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:48:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:48:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:48:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:48:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:48:58 smithi122 ceph-mon[93869]: pgmap v5978: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 82 KiB/s wr, 10 op/s 2023-03-09T17:48:58.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:48:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10248.356195 secs 2023-03-09T17:48:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:48:58 smithi181 ceph-mon[94189]: pgmap v5978: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 82 KiB/s wr, 10 op/s 2023-03-09T17:48:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:48:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10248.356195 secs 2023-03-09T17:48:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:48:58 smithi038 ceph-mon[92422]: pgmap v5978: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 82 KiB/s wr, 10 op/s 2023-03-09T17:48:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:48:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10248.356195 secs 2023-03-09T17:48:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10710.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:49:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:49:00 smithi122 ceph-mon[93869]: pgmap v5979: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 159 KiB/s wr, 10 op/s 2023-03-09T17:49:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:49:00 smithi181 ceph-mon[94189]: pgmap v5979: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 159 KiB/s wr, 10 op/s 2023-03-09T17:49:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:49:00 smithi038 ceph-mon[92422]: pgmap v5979: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 159 KiB/s wr, 10 op/s 2023-03-09T17:49:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:49:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10251.076647 secs 2023-03-09T17:49:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:49:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10251.076647 secs 2023-03-09T17:49:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:49:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10251.076647 secs 2023-03-09T17:49:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:49:02 smithi122 ceph-mon[93869]: pgmap v5980: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 137 KiB/s wr, 9 op/s 2023-03-09T17:49:02.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:49:02 smithi181 ceph-mon[94189]: pgmap v5980: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 137 KiB/s wr, 9 op/s 2023-03-09T17:49:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:49:02 smithi038 ceph-mon[92422]: pgmap v5980: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 137 KiB/s wr, 9 op/s 2023-03-09T17:49:03.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:49:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10253.356333 secs 2023-03-09T17:49:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:49:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10253.356333 secs 2023-03-09T17:49:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:49:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10253.356333 secs 2023-03-09T17:49:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10715.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:49:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:49:04 smithi038 ceph-mon[92422]: pgmap v5981: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 162 KiB/s wr, 9 op/s 2023-03-09T17:49:04.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:49:04 smithi122 ceph-mon[93869]: pgmap v5981: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 162 KiB/s wr, 9 op/s 2023-03-09T17:49:04.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:49:04 smithi181 ceph-mon[94189]: pgmap v5981: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 162 KiB/s wr, 9 op/s 2023-03-09T17:49:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:49:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10256.076706 secs 2023-03-09T17:49:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:49:06 smithi038 ceph-mon[92422]: pgmap v5982: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 141 KiB/s wr, 20 op/s 2023-03-09T17:49:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:49:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10256.076706 secs 2023-03-09T17:49:06.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:49:06 smithi122 ceph-mon[93869]: pgmap v5982: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 141 KiB/s wr, 20 op/s 2023-03-09T17:49:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:49:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10256.076706 secs 2023-03-09T17:49:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:49:06 smithi181 ceph-mon[94189]: pgmap v5982: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 141 KiB/s wr, 20 op/s 2023-03-09T17:49:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:49:08 smithi038 ceph-mon[92422]: pgmap v5983: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 155 KiB/s wr, 23 op/s 2023-03-09T17:49:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:49:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10258.356450 secs 2023-03-09T17:49:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:49:08 smithi122 ceph-mon[93869]: pgmap v5983: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 155 KiB/s wr, 23 op/s 2023-03-09T17:49:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:49:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10258.356450 secs 2023-03-09T17:49:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:49:08 smithi181 ceph-mon[94189]: pgmap v5983: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 155 KiB/s wr, 23 op/s 2023-03-09T17:49:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:49:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10258.356450 secs 2023-03-09T17:49:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10720.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:49:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:49:10 smithi122 ceph-mon[93869]: pgmap v5984: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 184 KiB/s wr, 23 op/s 2023-03-09T17:49:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:49:10 smithi122 ceph-mon[93869]: osdmap e587: 12 total, 12 up, 12 in 2023-03-09T17:49:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:49:10 smithi181 ceph-mon[94189]: pgmap v5984: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 184 KiB/s wr, 23 op/s 2023-03-09T17:49:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:49:10 smithi181 ceph-mon[94189]: osdmap e587: 12 total, 12 up, 12 in 2023-03-09T17:49:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:49:10 smithi038 ceph-mon[92422]: pgmap v5984: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 184 KiB/s wr, 23 op/s 2023-03-09T17:49:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:49:10 smithi038 ceph-mon[92422]: osdmap e587: 12 total, 12 up, 12 in 2023-03-09T17:49:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:49:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10261.076767 secs 2023-03-09T17:49:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:49:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10261.076767 secs 2023-03-09T17:49:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:49:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10261.076767 secs 2023-03-09T17:49:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:49:12 smithi122 ceph-mon[93869]: pgmap v5986: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 128 KiB/s wr, 26 op/s 2023-03-09T17:49:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:49:12 smithi181 ceph-mon[94189]: pgmap v5986: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 128 KiB/s wr, 26 op/s 2023-03-09T17:49:12.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:49:12 smithi038 ceph-mon[92422]: pgmap v5986: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 128 KiB/s wr, 26 op/s 2023-03-09T17:49:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:49:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10263.356567 secs 2023-03-09T17:49:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:49:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10263.356567 secs 2023-03-09T17:49:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:49:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10263.356567 secs 2023-03-09T17:49:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10725.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:49:14.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:49:14 smithi122 ceph-mon[93869]: pgmap v5987: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 98 KiB/s wr, 25 op/s 2023-03-09T17:49:14.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:49:14 smithi181 ceph-mon[94189]: pgmap v5987: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 98 KiB/s wr, 25 op/s 2023-03-09T17:49:14.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:49:14 smithi038 ceph-mon[92422]: pgmap v5987: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 98 KiB/s wr, 25 op/s 2023-03-09T17:49:16.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:49:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10266.076826 secs 2023-03-09T17:49:16.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:49:16 smithi122 ceph-mon[93869]: pgmap v5988: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 64 KiB/s wr, 11 op/s 2023-03-09T17:49:16.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:49:16 smithi122 ceph-mon[93869]: osdmap e588: 12 total, 12 up, 12 in 2023-03-09T17:49:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:49:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10266.076826 secs 2023-03-09T17:49:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:49:16 smithi181 ceph-mon[94189]: pgmap v5988: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 64 KiB/s wr, 11 op/s 2023-03-09T17:49:16.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:49:16 smithi181 ceph-mon[94189]: osdmap e588: 12 total, 12 up, 12 in 2023-03-09T17:49:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:49:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10266.076826 secs 2023-03-09T17:49:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:49:16 smithi038 ceph-mon[92422]: pgmap v5988: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 64 KiB/s wr, 11 op/s 2023-03-09T17:49:16.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:49:16 smithi038 ceph-mon[92422]: osdmap e588: 12 total, 12 up, 12 in 2023-03-09T17:49:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:49:18 smithi122 ceph-mon[93869]: pgmap v5990: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.1 KiB/s wr, 7 op/s 2023-03-09T17:49:18.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:49:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10268.356686 secs 2023-03-09T17:49:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:49:18 smithi181 ceph-mon[94189]: pgmap v5990: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.1 KiB/s wr, 7 op/s 2023-03-09T17:49:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:49:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10268.356686 secs 2023-03-09T17:49:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:49:18 smithi038 ceph-mon[92422]: pgmap v5990: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.1 KiB/s wr, 7 op/s 2023-03-09T17:49:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:49:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10268.356686 secs 2023-03-09T17:49:18.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10730.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:49:19.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:49:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:49:19.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:49:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:49:19.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:49:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:49:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:49:20 smithi122 ceph-mon[93869]: pgmap v5991: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 2.2 KiB/s wr, 8 op/s 2023-03-09T17:49:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:49:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:49:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:49:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:49:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:49:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:49:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:49:20 smithi181 ceph-mon[94189]: pgmap v5991: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 2.2 KiB/s wr, 8 op/s 2023-03-09T17:49:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:49:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:49:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:49:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:49:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:49:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:49:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:49:20 smithi038 ceph-mon[92422]: pgmap v5991: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 2.2 KiB/s wr, 8 op/s 2023-03-09T17:49:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:49:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:49:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:49:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:49:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:49:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:49:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:49:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10271.076929 secs 2023-03-09T17:49:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:49:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10271.076929 secs 2023-03-09T17:49:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:49:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10271.076929 secs 2023-03-09T17:49:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:49:22 smithi122 ceph-mon[93869]: pgmap v5992: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 2.1 KiB/s wr, 8 op/s 2023-03-09T17:49:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:49:22 smithi181 ceph-mon[94189]: pgmap v5992: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 2.1 KiB/s wr, 8 op/s 2023-03-09T17:49:22.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:49:22 smithi038 ceph-mon[92422]: pgmap v5992: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 2.1 KiB/s wr, 8 op/s 2023-03-09T17:49:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:49:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10273.356798 secs 2023-03-09T17:49:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:49:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10273.356798 secs 2023-03-09T17:49:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:49:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10273.356798 secs 2023-03-09T17:49:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10735.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:49:24.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:49:24 smithi122 ceph-mon[93869]: pgmap v5993: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 35 KiB/s wr, 8 op/s 2023-03-09T17:49:24.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:49:24 smithi181 ceph-mon[94189]: pgmap v5993: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 35 KiB/s wr, 8 op/s 2023-03-09T17:49:24.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:49:24 smithi038 ceph-mon[92422]: pgmap v5993: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 35 KiB/s wr, 8 op/s 2023-03-09T17:49:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:49:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10276.076992 secs 2023-03-09T17:49:26.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:49:26 smithi122 ceph-mon[93869]: pgmap v5994: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 34 KiB/s wr, 8 op/s 2023-03-09T17:49:26.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:49:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10276.076992 secs 2023-03-09T17:49:26.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:49:26 smithi181 ceph-mon[94189]: pgmap v5994: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 34 KiB/s wr, 8 op/s 2023-03-09T17:49:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:49:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10276.076992 secs 2023-03-09T17:49:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:49:26 smithi038 ceph-mon[92422]: pgmap v5994: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 34 KiB/s wr, 8 op/s 2023-03-09T17:49:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:49:28 smithi122 ceph-mon[93869]: pgmap v5995: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T17:49:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:49:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10278.356937 secs 2023-03-09T17:49:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:49:28 smithi181 ceph-mon[94189]: pgmap v5995: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T17:49:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:49:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10278.356937 secs 2023-03-09T17:49:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:49:28 smithi038 ceph-mon[92422]: pgmap v5995: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T17:49:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:49:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10278.356937 secs 2023-03-09T17:49:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10740.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:49:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:49:30 smithi122 ceph-mon[93869]: pgmap v5996: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 29 KiB/s wr, 7 op/s 2023-03-09T17:49:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:49:30 smithi181 ceph-mon[94189]: pgmap v5996: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 29 KiB/s wr, 7 op/s 2023-03-09T17:49:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:49:30 smithi038 ceph-mon[92422]: pgmap v5996: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 29 KiB/s wr, 7 op/s 2023-03-09T17:49:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:49:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10281.077054 secs 2023-03-09T17:49:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:49:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10281.077054 secs 2023-03-09T17:49:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:49:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10281.077054 secs 2023-03-09T17:49:32.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:49:32 smithi122 ceph-mon[93869]: pgmap v5997: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 29 KiB/s wr, 7 op/s 2023-03-09T17:49:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:49:32 smithi181 ceph-mon[94189]: pgmap v5997: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 29 KiB/s wr, 7 op/s 2023-03-09T17:49:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:49:32 smithi038 ceph-mon[92422]: pgmap v5997: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 29 KiB/s wr, 7 op/s 2023-03-09T17:49:33.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10745.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:49:34.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:49:34 smithi122 ceph-mon[93869]: pgmap v5998: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 101 KiB/s wr, 7 op/s 2023-03-09T17:49:34.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:49:34 smithi181 ceph-mon[94189]: pgmap v5998: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 101 KiB/s wr, 7 op/s 2023-03-09T17:49:34.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:49:34 smithi038 ceph-mon[92422]: pgmap v5998: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 101 KiB/s wr, 7 op/s 2023-03-09T17:49:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:49:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10286.077147 secs 2023-03-09T17:49:36.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:49:36 smithi122 ceph-mon[93869]: pgmap v5999: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 74 KiB/s wr, 6 op/s 2023-03-09T17:49:36.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:49:36 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:49:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:49:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10286.077147 secs 2023-03-09T17:49:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:49:36 smithi181 ceph-mon[94189]: pgmap v5999: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 74 KiB/s wr, 6 op/s 2023-03-09T17:49:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:49:36 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:49:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:49:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10286.077147 secs 2023-03-09T17:49:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:49:36 smithi038 ceph-mon[92422]: pgmap v5999: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 74 KiB/s wr, 6 op/s 2023-03-09T17:49:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:49:36 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:49:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:49:37 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:49:37.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:49:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:49:37.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:49:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:49:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:49:37 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:49:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:49:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:49:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:49:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:49:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:49:37 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:49:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:49:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:49:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:49:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:49:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:49:38 smithi122 ceph-mon[93869]: pgmap v6000: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 74 KiB/s wr, 8 op/s 2023-03-09T17:49:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:49:38 smithi181 ceph-mon[94189]: pgmap v6000: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 74 KiB/s wr, 8 op/s 2023-03-09T17:49:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:49:38 smithi038 ceph-mon[92422]: pgmap v6000: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 74 KiB/s wr, 8 op/s 2023-03-09T17:49:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10750.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:49:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:49:40 smithi122 ceph-mon[93869]: pgmap v6001: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 80 KiB/s wr, 8 op/s 2023-03-09T17:49:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:49:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10291.077242 secs 2023-03-09T17:49:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:49:40 smithi181 ceph-mon[94189]: pgmap v6001: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 80 KiB/s wr, 8 op/s 2023-03-09T17:49:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:49:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10291.077242 secs 2023-03-09T17:49:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:49:40 smithi038 ceph-mon[92422]: pgmap v6001: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 80 KiB/s wr, 8 op/s 2023-03-09T17:49:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:49:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10291.077242 secs 2023-03-09T17:49:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:49:42 smithi122 ceph-mon[93869]: pgmap v6002: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 80 KiB/s wr, 9 op/s 2023-03-09T17:49:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:49:42 smithi181 ceph-mon[94189]: pgmap v6002: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 80 KiB/s wr, 9 op/s 2023-03-09T17:49:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:49:42 smithi038 ceph-mon[92422]: pgmap v6002: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 80 KiB/s wr, 9 op/s 2023-03-09T17:49:43.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10755.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:49:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:49:44 smithi122 ceph-mon[93869]: pgmap v6003: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 80 KiB/s wr, 8 op/s 2023-03-09T17:49:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:49:44 smithi181 ceph-mon[94189]: pgmap v6003: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 80 KiB/s wr, 8 op/s 2023-03-09T17:49:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:49:44 smithi038 ceph-mon[92422]: pgmap v6003: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 80 KiB/s wr, 8 op/s 2023-03-09T17:49:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:49:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10296.077303 secs 2023-03-09T17:49:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:49:46 smithi122 ceph-mon[93869]: pgmap v6004: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 64 KiB/s wr, 9 op/s 2023-03-09T17:49:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:49:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10296.077303 secs 2023-03-09T17:49:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:49:46 smithi181 ceph-mon[94189]: pgmap v6004: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 64 KiB/s wr, 9 op/s 2023-03-09T17:49:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:49:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10296.077303 secs 2023-03-09T17:49:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:49:46 smithi038 ceph-mon[92422]: pgmap v6004: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 64 KiB/s wr, 9 op/s 2023-03-09T17:49:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:49:48 smithi122 ceph-mon[93869]: pgmap v6005: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 64 KiB/s wr, 10 op/s 2023-03-09T17:49:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:49:48 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 10298.357464 secs 2023-03-09T17:49:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:49:48 smithi181 ceph-mon[94189]: pgmap v6005: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 64 KiB/s wr, 10 op/s 2023-03-09T17:49:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:49:48 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 10298.357464 secs 2023-03-09T17:49:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:49:48 smithi038 ceph-mon[92422]: pgmap v6005: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 64 KiB/s wr, 10 op/s 2023-03-09T17:49:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:49:48 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 10298.357464 secs 2023-03-09T17:49:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10760.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:49:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:49:49 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:49:49.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:49:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:49:49.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:49:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:49:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:49:49 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:49:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:49:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:49:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:49:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:49:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:49:49 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:49:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:49:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:49:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:49:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:49:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:49:50 smithi122 ceph-mon[93869]: pgmap v6006: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 166 KiB/s wr, 12 op/s 2023-03-09T17:49:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:49:50 smithi181 ceph-mon[94189]: pgmap v6006: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 166 KiB/s wr, 12 op/s 2023-03-09T17:49:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:49:50 smithi038 ceph-mon[92422]: pgmap v6006: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 166 KiB/s wr, 12 op/s 2023-03-09T17:49:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:49:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10301.077386 secs 2023-03-09T17:49:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:49:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10301.077386 secs 2023-03-09T17:49:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:49:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10301.077386 secs 2023-03-09T17:49:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:49:52 smithi122 ceph-mon[93869]: pgmap v6007: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 160 KiB/s wr, 12 op/s 2023-03-09T17:49:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:49:52 smithi181 ceph-mon[94189]: pgmap v6007: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 160 KiB/s wr, 12 op/s 2023-03-09T17:49:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:49:52 smithi038 ceph-mon[92422]: pgmap v6007: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 160 KiB/s wr, 12 op/s 2023-03-09T17:49:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:49:53 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 10303.357582 secs 2023-03-09T17:49:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:49:53 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 10303.357582 secs 2023-03-09T17:49:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:49:53 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 10303.357582 secs 2023-03-09T17:49:53.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10765.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:49:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:49:54 smithi122 ceph-mon[93869]: pgmap v6008: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 160 KiB/s wr, 10 op/s 2023-03-09T17:49:54.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:49:54 smithi181 ceph-mon[94189]: pgmap v6008: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 160 KiB/s wr, 10 op/s 2023-03-09T17:49:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:49:54 smithi038 ceph-mon[92422]: pgmap v6008: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 160 KiB/s wr, 10 op/s 2023-03-09T17:49:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:49:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10306.077466 secs 2023-03-09T17:49:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:49:56 smithi122 ceph-mon[93869]: pgmap v6009: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 196 KiB/s wr, 12 op/s 2023-03-09T17:49:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:49:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10306.077466 secs 2023-03-09T17:49:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:49:56 smithi181 ceph-mon[94189]: pgmap v6009: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 196 KiB/s wr, 12 op/s 2023-03-09T17:49:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:49:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10306.077466 secs 2023-03-09T17:49:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:49:56 smithi038 ceph-mon[92422]: pgmap v6009: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 196 KiB/s wr, 12 op/s 2023-03-09T17:49:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:49:58 smithi122 ceph-mon[93869]: pgmap v6010: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 139 KiB/s wr, 12 op/s 2023-03-09T17:49:58.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:49:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10308.357696 secs 2023-03-09T17:49:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:49:58 smithi181 ceph-mon[94189]: pgmap v6010: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 139 KiB/s wr, 12 op/s 2023-03-09T17:49:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:49:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10308.357696 secs 2023-03-09T17:49:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:49:58 smithi038 ceph-mon[92422]: pgmap v6010: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 139 KiB/s wr, 12 op/s 2023-03-09T17:49:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:49:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10308.357696 secs 2023-03-09T17:49:58.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10770.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:50:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:50:00 smithi122 ceph-mon[93869]: pgmap v6011: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 151 KiB/s wr, 12 op/s 2023-03-09T17:50:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:50:00 smithi122 ceph-mon[93869]: overall HEALTH_WARN 2 MDSs report slow requests 2023-03-09T17:50:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:50:00 smithi181 ceph-mon[94189]: pgmap v6011: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 151 KiB/s wr, 12 op/s 2023-03-09T17:50:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:50:00 smithi181 ceph-mon[94189]: overall HEALTH_WARN 2 MDSs report slow requests 2023-03-09T17:50:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:50:00 smithi038 ceph-mon[92422]: pgmap v6011: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 151 KiB/s wr, 12 op/s 2023-03-09T17:50:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:50:00 smithi038 ceph-mon[92422]: overall HEALTH_WARN 2 MDSs report slow requests 2023-03-09T17:50:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:50:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10311.077526 secs 2023-03-09T17:50:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:50:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:50:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:50:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:50:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:50:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10311.077526 secs 2023-03-09T17:50:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:50:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:50:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:50:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:50:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:50:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10311.077526 secs 2023-03-09T17:50:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:50:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:50:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:50:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:50:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:50:02 smithi122 ceph-mon[93869]: pgmap v6012: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 80 KiB/s wr, 13 op/s 2023-03-09T17:50:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:50:02 smithi181 ceph-mon[94189]: pgmap v6012: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 80 KiB/s wr, 13 op/s 2023-03-09T17:50:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:50:02 smithi038 ceph-mon[92422]: pgmap v6012: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 80 KiB/s wr, 13 op/s 2023-03-09T17:50:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:50:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10313.357830 secs 2023-03-09T17:50:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:50:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10313.357830 secs 2023-03-09T17:50:03.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:50:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10313.357830 secs 2023-03-09T17:50:03.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10775.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:50:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:50:04 smithi122 ceph-mon[93869]: pgmap v6013: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 79 KiB/s wr, 11 op/s 2023-03-09T17:50:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:50:04 smithi181 ceph-mon[94189]: pgmap v6013: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 79 KiB/s wr, 11 op/s 2023-03-09T17:50:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:50:04 smithi038 ceph-mon[92422]: pgmap v6013: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 79 KiB/s wr, 11 op/s 2023-03-09T17:50:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:50:05 smithi122 ceph-mon[93869]: osdmap e589: 12 total, 12 up, 12 in 2023-03-09T17:50:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:50:05 smithi181 ceph-mon[94189]: osdmap e589: 12 total, 12 up, 12 in 2023-03-09T17:50:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:50:05 smithi038 ceph-mon[92422]: osdmap e589: 12 total, 12 up, 12 in 2023-03-09T17:50:06.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:50:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10316.077586 secs 2023-03-09T17:50:06.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:50:06 smithi122 ceph-mon[93869]: pgmap v6015: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 199 KiB/s wr, 28 op/s 2023-03-09T17:50:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:50:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10316.077586 secs 2023-03-09T17:50:06.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:50:06 smithi181 ceph-mon[94189]: pgmap v6015: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 199 KiB/s wr, 28 op/s 2023-03-09T17:50:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:50:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10316.077586 secs 2023-03-09T17:50:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:50:06 smithi038 ceph-mon[92422]: pgmap v6015: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 199 KiB/s wr, 28 op/s 2023-03-09T17:50:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:50:08 smithi122 ceph-mon[93869]: pgmap v6016: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 199 KiB/s wr, 28 op/s 2023-03-09T17:50:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:50:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10318.357945 secs 2023-03-09T17:50:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:50:08 smithi181 ceph-mon[94189]: pgmap v6016: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 199 KiB/s wr, 28 op/s 2023-03-09T17:50:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:50:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10318.357945 secs 2023-03-09T17:50:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:50:08 smithi038 ceph-mon[92422]: pgmap v6016: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 199 KiB/s wr, 28 op/s 2023-03-09T17:50:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:50:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10318.357945 secs 2023-03-09T17:50:08.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10780.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:50:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:50:10 smithi122 ceph-mon[93869]: pgmap v6017: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 210 KiB/s wr, 27 op/s 2023-03-09T17:50:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:50:10 smithi181 ceph-mon[94189]: pgmap v6017: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 210 KiB/s wr, 27 op/s 2023-03-09T17:50:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:50:10 smithi038 ceph-mon[92422]: pgmap v6017: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 210 KiB/s wr, 27 op/s 2023-03-09T17:50:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:50:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10321.077647 secs 2023-03-09T17:50:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:50:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10321.077647 secs 2023-03-09T17:50:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:50:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10321.077647 secs 2023-03-09T17:50:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:50:12 smithi122 ceph-mon[93869]: pgmap v6018: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 174 KiB/s wr, 24 op/s 2023-03-09T17:50:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:50:12 smithi181 ceph-mon[94189]: pgmap v6018: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 174 KiB/s wr, 24 op/s 2023-03-09T17:50:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:50:12 smithi038 ceph-mon[92422]: pgmap v6018: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 174 KiB/s wr, 24 op/s 2023-03-09T17:50:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:50:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10323.358080 secs 2023-03-09T17:50:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:50:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10323.358080 secs 2023-03-09T17:50:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:50:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10323.358080 secs 2023-03-09T17:50:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10785.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:50:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:50:14 smithi122 ceph-mon[93869]: pgmap v6019: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 174 KiB/s wr, 24 op/s 2023-03-09T17:50:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:50:14 smithi181 ceph-mon[94189]: pgmap v6019: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 174 KiB/s wr, 24 op/s 2023-03-09T17:50:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:50:14 smithi038 ceph-mon[92422]: pgmap v6019: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 174 KiB/s wr, 24 op/s 2023-03-09T17:50:16.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:50:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10326.077707 secs 2023-03-09T17:50:16.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:50:16 smithi122 ceph-mon[93869]: pgmap v6020: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 46 KiB/s wr, 7 op/s 2023-03-09T17:50:16.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:50:16 smithi122 ceph-mon[93869]: osdmap e590: 12 total, 12 up, 12 in 2023-03-09T17:50:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:50:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10326.077707 secs 2023-03-09T17:50:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:50:16 smithi181 ceph-mon[94189]: pgmap v6020: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 46 KiB/s wr, 7 op/s 2023-03-09T17:50:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:50:16 smithi181 ceph-mon[94189]: osdmap e590: 12 total, 12 up, 12 in 2023-03-09T17:50:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:50:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10326.077707 secs 2023-03-09T17:50:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:50:16 smithi038 ceph-mon[92422]: pgmap v6020: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 46 KiB/s wr, 7 op/s 2023-03-09T17:50:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:50:16 smithi038 ceph-mon[92422]: osdmap e590: 12 total, 12 up, 12 in 2023-03-09T17:50:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:50:18 smithi122 ceph-mon[93869]: pgmap v6022: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 49 KiB/s wr, 7 op/s 2023-03-09T17:50:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:50:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10328.358208 secs 2023-03-09T17:50:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:50:18 smithi181 ceph-mon[94189]: pgmap v6022: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 49 KiB/s wr, 7 op/s 2023-03-09T17:50:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:50:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10328.358208 secs 2023-03-09T17:50:18.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:50:18 smithi038 ceph-mon[92422]: pgmap v6022: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 49 KiB/s wr, 7 op/s 2023-03-09T17:50:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:50:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10328.358208 secs 2023-03-09T17:50:18.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10790.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:50:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:50:20 smithi122 ceph-mon[93869]: pgmap v6023: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 6 op/s 2023-03-09T17:50:20.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:50:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:50:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:50:20 smithi181 ceph-mon[94189]: pgmap v6023: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 6 op/s 2023-03-09T17:50:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:50:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:50:20.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:50:20 smithi038 ceph-mon[92422]: pgmap v6023: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 6 op/s 2023-03-09T17:50:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:50:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:50:21.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:50:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10331.077767 secs 2023-03-09T17:50:21.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:50:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:50:21.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:50:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:50:21.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:50:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:50:21.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:50:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:50:21.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:50:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:50:21.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:50:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:50:21.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:50:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:50:21.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:50:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:50:21.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:50:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:50:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:50:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10331.077767 secs 2023-03-09T17:50:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:50:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:50:21.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:50:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:50:21.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:50:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:50:21.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:50:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:50:21.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:50:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:50:21.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:50:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:50:21.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:50:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:50:21.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:50:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:50:21.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:50:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:50:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:50:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10331.077767 secs 2023-03-09T17:50:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:50:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:50:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:50:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:50:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:50:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:50:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:50:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:50:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:50:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:50:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:50:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:50:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:50:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:50:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:50:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:50:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:50:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:50:22.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:50:22 smithi122 ceph-mon[93869]: pgmap v6024: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 33 KiB/s wr, 7 op/s 2023-03-09T17:50:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:50:22 smithi181 ceph-mon[94189]: pgmap v6024: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 33 KiB/s wr, 7 op/s 2023-03-09T17:50:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:50:22 smithi038 ceph-mon[92422]: pgmap v6024: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 33 KiB/s wr, 7 op/s 2023-03-09T17:50:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:50:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10333.358323 secs 2023-03-09T17:50:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:50:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10333.358323 secs 2023-03-09T17:50:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:50:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10333.358323 secs 2023-03-09T17:50:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10795.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:50:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:50:24 smithi122 ceph-mon[93869]: pgmap v6025: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 34 KiB/s wr, 8 op/s 2023-03-09T17:50:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:50:24 smithi181 ceph-mon[94189]: pgmap v6025: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 34 KiB/s wr, 8 op/s 2023-03-09T17:50:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:50:24 smithi038 ceph-mon[92422]: pgmap v6025: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 34 KiB/s wr, 8 op/s 2023-03-09T17:50:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:50:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10336.077826 secs 2023-03-09T17:50:26.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:50:26 smithi122 ceph-mon[93869]: pgmap v6026: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 31 KiB/s wr, 8 op/s 2023-03-09T17:50:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:50:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10336.077826 secs 2023-03-09T17:50:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:50:26 smithi181 ceph-mon[94189]: pgmap v6026: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 31 KiB/s wr, 8 op/s 2023-03-09T17:50:26.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:50:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10336.077826 secs 2023-03-09T17:50:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:50:26 smithi038 ceph-mon[92422]: pgmap v6026: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 31 KiB/s wr, 8 op/s 2023-03-09T17:50:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:50:28 smithi038 ceph-mon[92422]: pgmap v6027: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T17:50:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:50:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10338.358437 secs 2023-03-09T17:50:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:50:28 smithi122 ceph-mon[93869]: pgmap v6027: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T17:50:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:50:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10338.358437 secs 2023-03-09T17:50:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:50:28 smithi181 ceph-mon[94189]: pgmap v6027: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T17:50:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:50:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10338.358437 secs 2023-03-09T17:50:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10800.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:50:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:50:30 smithi038 ceph-mon[92422]: pgmap v6028: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 53 KiB/s wr, 7 op/s 2023-03-09T17:50:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:50:30 smithi122 ceph-mon[93869]: pgmap v6028: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 53 KiB/s wr, 7 op/s 2023-03-09T17:50:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:50:30 smithi181 ceph-mon[94189]: pgmap v6028: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 53 KiB/s wr, 7 op/s 2023-03-09T17:50:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:50:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10341.077889 secs 2023-03-09T17:50:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:50:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10341.077889 secs 2023-03-09T17:50:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:50:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10341.077889 secs 2023-03-09T17:50:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:50:32 smithi122 ceph-mon[93869]: pgmap v6029: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 53 KiB/s wr, 9 op/s 2023-03-09T17:50:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:50:32 smithi181 ceph-mon[94189]: pgmap v6029: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 53 KiB/s wr, 9 op/s 2023-03-09T17:50:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:50:32 smithi038 ceph-mon[92422]: pgmap v6029: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 53 KiB/s wr, 9 op/s 2023-03-09T17:50:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:50:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10343.358553 secs 2023-03-09T17:50:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:50:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10343.358553 secs 2023-03-09T17:50:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:50:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10343.358553 secs 2023-03-09T17:50:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10805.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:50:34.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:50:34 smithi122 ceph-mon[93869]: pgmap v6030: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T17:50:34.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:50:34 smithi181 ceph-mon[94189]: pgmap v6030: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T17:50:34.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:50:34 smithi038 ceph-mon[92422]: pgmap v6030: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T17:50:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:50:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10346.077950 secs 2023-03-09T17:50:36.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:50:36 smithi122 ceph-mon[93869]: pgmap v6031: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T17:50:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:50:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10346.077950 secs 2023-03-09T17:50:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:50:36 smithi181 ceph-mon[94189]: pgmap v6031: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T17:50:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:50:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10346.077950 secs 2023-03-09T17:50:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:50:36 smithi038 ceph-mon[92422]: pgmap v6031: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T17:50:38.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:50:38 smithi122 ceph-mon[93869]: pgmap v6032: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 28 KiB/s wr, 8 op/s 2023-03-09T17:50:38.731 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:50:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10348.358651 secs 2023-03-09T17:50:38.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:50:38 smithi181 ceph-mon[94189]: pgmap v6032: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 28 KiB/s wr, 8 op/s 2023-03-09T17:50:38.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:50:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10348.358651 secs 2023-03-09T17:50:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:50:38 smithi038 ceph-mon[92422]: pgmap v6032: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 28 KiB/s wr, 8 op/s 2023-03-09T17:50:38.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:50:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10348.358651 secs 2023-03-09T17:50:38.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10810.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:50:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:50:40 smithi122 ceph-mon[93869]: pgmap v6033: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 56 KiB/s wr, 8 op/s 2023-03-09T17:50:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:50:40 smithi181 ceph-mon[94189]: pgmap v6033: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 56 KiB/s wr, 8 op/s 2023-03-09T17:50:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:50:40 smithi038 ceph-mon[92422]: pgmap v6033: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 56 KiB/s wr, 8 op/s 2023-03-09T17:50:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:50:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10351.078003 secs 2023-03-09T17:50:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:50:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10351.078003 secs 2023-03-09T17:50:41.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:50:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10351.078003 secs 2023-03-09T17:50:42.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:50:42 smithi122 ceph-mon[93869]: pgmap v6034: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 36 KiB/s wr, 9 op/s 2023-03-09T17:50:42.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:50:42 smithi181 ceph-mon[94189]: pgmap v6034: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 36 KiB/s wr, 9 op/s 2023-03-09T17:50:42.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:50:42 smithi038 ceph-mon[92422]: pgmap v6034: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 36 KiB/s wr, 9 op/s 2023-03-09T17:50:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:50:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10353.358758 secs 2023-03-09T17:50:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:50:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10353.358758 secs 2023-03-09T17:50:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:50:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10353.358758 secs 2023-03-09T17:50:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10815.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:50:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:50:44 smithi122 ceph-mon[93869]: pgmap v6035: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 85 KiB/s wr, 7 op/s 2023-03-09T17:50:44.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:50:44 smithi181 ceph-mon[94189]: pgmap v6035: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 85 KiB/s wr, 7 op/s 2023-03-09T17:50:44.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:50:44 smithi038 ceph-mon[92422]: pgmap v6035: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 85 KiB/s wr, 7 op/s 2023-03-09T17:50:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:50:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10356.078065 secs 2023-03-09T17:50:46.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:50:46 smithi122 ceph-mon[93869]: pgmap v6036: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 93 KiB/s wr, 8 op/s 2023-03-09T17:50:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:50:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10356.078065 secs 2023-03-09T17:50:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:50:46 smithi181 ceph-mon[94189]: pgmap v6036: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 93 KiB/s wr, 8 op/s 2023-03-09T17:50:46.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:50:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10356.078065 secs 2023-03-09T17:50:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:50:46 smithi038 ceph-mon[92422]: pgmap v6036: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 93 KiB/s wr, 8 op/s 2023-03-09T17:50:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:50:48 smithi122 ceph-mon[93869]: pgmap v6037: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 93 KiB/s wr, 9 op/s 2023-03-09T17:50:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:50:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10358.358858 secs 2023-03-09T17:50:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:50:48 smithi181 ceph-mon[94189]: pgmap v6037: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 93 KiB/s wr, 9 op/s 2023-03-09T17:50:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:50:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10358.358858 secs 2023-03-09T17:50:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:50:48 smithi038 ceph-mon[92422]: pgmap v6037: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 93 KiB/s wr, 9 op/s 2023-03-09T17:50:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:50:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10358.358858 secs 2023-03-09T17:50:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10820.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:50:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:50:50 smithi122 ceph-mon[93869]: pgmap v6038: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 167 KiB/s wr, 9 op/s 2023-03-09T17:50:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:50:50 smithi181 ceph-mon[94189]: pgmap v6038: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 167 KiB/s wr, 9 op/s 2023-03-09T17:50:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:50:50 smithi038 ceph-mon[92422]: pgmap v6038: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 167 KiB/s wr, 9 op/s 2023-03-09T17:50:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:50:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10361.078151 secs 2023-03-09T17:50:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:50:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10361.078151 secs 2023-03-09T17:50:51.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:50:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10361.078151 secs 2023-03-09T17:50:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:50:52 smithi122 ceph-mon[93869]: pgmap v6039: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 149 KiB/s wr, 11 op/s 2023-03-09T17:50:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:50:52 smithi181 ceph-mon[94189]: pgmap v6039: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 149 KiB/s wr, 11 op/s 2023-03-09T17:50:52.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:50:52 smithi038 ceph-mon[92422]: pgmap v6039: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 149 KiB/s wr, 11 op/s 2023-03-09T17:50:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:50:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10363.358973 secs 2023-03-09T17:50:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:50:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10363.358973 secs 2023-03-09T17:50:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:50:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10363.358973 secs 2023-03-09T17:50:53.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10825.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:50:54.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:50:54 smithi122 ceph-mon[93869]: pgmap v6040: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 171 KiB/s wr, 9 op/s 2023-03-09T17:50:54.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:50:54 smithi181 ceph-mon[94189]: pgmap v6040: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 171 KiB/s wr, 9 op/s 2023-03-09T17:50:54.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:50:54 smithi038 ceph-mon[92422]: pgmap v6040: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 171 KiB/s wr, 9 op/s 2023-03-09T17:50:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:50:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10366.078235 secs 2023-03-09T17:50:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:50:56 smithi122 ceph-mon[93869]: pgmap v6041: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 122 KiB/s wr, 9 op/s 2023-03-09T17:50:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:50:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10366.078235 secs 2023-03-09T17:50:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:50:56 smithi181 ceph-mon[94189]: pgmap v6041: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 122 KiB/s wr, 9 op/s 2023-03-09T17:50:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:50:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10366.078235 secs 2023-03-09T17:50:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:50:56 smithi038 ceph-mon[92422]: pgmap v6041: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 122 KiB/s wr, 9 op/s 2023-03-09T17:50:58.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:50:58 smithi122 ceph-mon[93869]: pgmap v6042: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 114 KiB/s wr, 11 op/s 2023-03-09T17:50:58.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:50:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10368.359131 secs 2023-03-09T17:50:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:50:58 smithi181 ceph-mon[94189]: pgmap v6042: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 114 KiB/s wr, 11 op/s 2023-03-09T17:50:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:50:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10368.359131 secs 2023-03-09T17:50:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:50:58 smithi038 ceph-mon[92422]: pgmap v6042: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 114 KiB/s wr, 11 op/s 2023-03-09T17:50:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:50:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10368.359131 secs 2023-03-09T17:50:58.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10830.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:51:00.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:00 smithi122 ceph-mon[93869]: pgmap v6043: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 188 KiB/s wr, 10 op/s 2023-03-09T17:51:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:00 smithi181 ceph-mon[94189]: pgmap v6043: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 188 KiB/s wr, 10 op/s 2023-03-09T17:51:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:00 smithi038 ceph-mon[92422]: pgmap v6043: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 188 KiB/s wr, 10 op/s 2023-03-09T17:51:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10371.078317 secs 2023-03-09T17:51:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10371.078317 secs 2023-03-09T17:51:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10371.078317 secs 2023-03-09T17:51:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:02 smithi122 ceph-mon[93869]: pgmap v6044: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 153 KiB/s wr, 13 op/s 2023-03-09T17:51:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:02 smithi181 ceph-mon[94189]: pgmap v6044: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 153 KiB/s wr, 13 op/s 2023-03-09T17:51:02.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:02 smithi038 ceph-mon[92422]: pgmap v6044: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 153 KiB/s wr, 13 op/s 2023-03-09T17:51:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10373.359232 secs 2023-03-09T17:51:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10373.359232 secs 2023-03-09T17:51:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10373.359232 secs 2023-03-09T17:51:03.793 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10835.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:51:04.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:04 smithi122 ceph-mon[93869]: pgmap v6045: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 182 KiB/s wr, 12 op/s 2023-03-09T17:51:04.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:04 smithi181 ceph-mon[94189]: pgmap v6045: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 182 KiB/s wr, 12 op/s 2023-03-09T17:51:04.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:04 smithi038 ceph-mon[92422]: pgmap v6045: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 182 KiB/s wr, 12 op/s 2023-03-09T17:51:05.656 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10376.078379 secs 2023-03-09T17:51:05.657 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:05 smithi181 ceph-mon[94189]: pgmap v6046: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 182 KiB/s wr, 24 op/s 2023-03-09T17:51:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10376.078379 secs 2023-03-09T17:51:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:05 smithi122 ceph-mon[93869]: pgmap v6046: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 182 KiB/s wr, 24 op/s 2023-03-09T17:51:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10376.078379 secs 2023-03-09T17:51:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:05 smithi038 ceph-mon[92422]: pgmap v6046: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 182 KiB/s wr, 24 op/s 2023-03-09T17:51:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:08 smithi122 ceph-mon[93869]: pgmap v6047: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 182 KiB/s wr, 25 op/s 2023-03-09T17:51:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10378.359320 secs 2023-03-09T17:51:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:08 smithi181 ceph-mon[94189]: pgmap v6047: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 182 KiB/s wr, 25 op/s 2023-03-09T17:51:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10378.359320 secs 2023-03-09T17:51:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:08 smithi038 ceph-mon[92422]: pgmap v6047: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 182 KiB/s wr, 25 op/s 2023-03-09T17:51:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10378.359320 secs 2023-03-09T17:51:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10840.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:51:10.407 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:10 smithi181 ceph-mon[94189]: pgmap v6048: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 221 KiB/s wr, 24 op/s 2023-03-09T17:51:10.407 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:10 smithi181 ceph-mon[94189]: osdmap e591: 12 total, 12 up, 12 in 2023-03-09T17:51:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:10 smithi122 ceph-mon[93869]: pgmap v6048: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 221 KiB/s wr, 24 op/s 2023-03-09T17:51:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:10 smithi122 ceph-mon[93869]: osdmap e591: 12 total, 12 up, 12 in 2023-03-09T17:51:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:10 smithi038 ceph-mon[92422]: pgmap v6048: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 221 KiB/s wr, 24 op/s 2023-03-09T17:51:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:10 smithi038 ceph-mon[92422]: osdmap e591: 12 total, 12 up, 12 in 2023-03-09T17:51:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10381.078438 secs 2023-03-09T17:51:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10381.078438 secs 2023-03-09T17:51:11.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10381.078438 secs 2023-03-09T17:51:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:12 smithi122 ceph-mon[93869]: pgmap v6050: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 130 KiB/s wr, 24 op/s 2023-03-09T17:51:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:12 smithi181 ceph-mon[94189]: pgmap v6050: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 130 KiB/s wr, 24 op/s 2023-03-09T17:51:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:12 smithi038 ceph-mon[92422]: pgmap v6050: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 130 KiB/s wr, 24 op/s 2023-03-09T17:51:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10383.359400 secs 2023-03-09T17:51:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10383.359400 secs 2023-03-09T17:51:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10383.359400 secs 2023-03-09T17:51:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10845.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:51:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:14 smithi122 ceph-mon[93869]: pgmap v6051: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 130 KiB/s wr, 24 op/s 2023-03-09T17:51:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:14 smithi181 ceph-mon[94189]: pgmap v6051: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 130 KiB/s wr, 24 op/s 2023-03-09T17:51:14.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:14 smithi038 ceph-mon[92422]: pgmap v6051: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 130 KiB/s wr, 24 op/s 2023-03-09T17:51:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10386.078497 secs 2023-03-09T17:51:16.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:16 smithi122 ceph-mon[93869]: pgmap v6052: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 105 KiB/s wr, 9 op/s 2023-03-09T17:51:16.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:16 smithi122 ceph-mon[93869]: osdmap e592: 12 total, 12 up, 12 in 2023-03-09T17:51:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10386.078497 secs 2023-03-09T17:51:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:16 smithi181 ceph-mon[94189]: pgmap v6052: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 105 KiB/s wr, 9 op/s 2023-03-09T17:51:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:16 smithi181 ceph-mon[94189]: osdmap e592: 12 total, 12 up, 12 in 2023-03-09T17:51:16.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10386.078497 secs 2023-03-09T17:51:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:16 smithi038 ceph-mon[92422]: pgmap v6052: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 105 KiB/s wr, 9 op/s 2023-03-09T17:51:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:16 smithi038 ceph-mon[92422]: osdmap e592: 12 total, 12 up, 12 in 2023-03-09T17:51:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:18 smithi122 ceph-mon[93869]: pgmap v6054: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 72 KiB/s wr, 9 op/s 2023-03-09T17:51:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10388.359493 secs 2023-03-09T17:51:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:18 smithi181 ceph-mon[94189]: pgmap v6054: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 72 KiB/s wr, 9 op/s 2023-03-09T17:51:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10388.359493 secs 2023-03-09T17:51:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:18 smithi038 ceph-mon[92422]: pgmap v6054: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 72 KiB/s wr, 9 op/s 2023-03-09T17:51:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10388.359493 secs 2023-03-09T17:51:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10850.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:51:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:20 smithi122 ceph-mon[93869]: pgmap v6055: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 83 KiB/s wr, 8 op/s 2023-03-09T17:51:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:20 smithi181 ceph-mon[94189]: pgmap v6055: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 83 KiB/s wr, 8 op/s 2023-03-09T17:51:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:20 smithi038 ceph-mon[92422]: pgmap v6055: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 83 KiB/s wr, 8 op/s 2023-03-09T17:51:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10391.078560 secs 2023-03-09T17:51:21.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:51:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10391.078560 secs 2023-03-09T17:51:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:51:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10391.078560 secs 2023-03-09T17:51:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:51:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:22 smithi122 ceph-mon[93869]: pgmap v6056: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 81 KiB/s wr, 8 op/s 2023-03-09T17:51:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:22 smithi181 ceph-mon[94189]: pgmap v6056: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 81 KiB/s wr, 8 op/s 2023-03-09T17:51:22.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:22 smithi038 ceph-mon[92422]: pgmap v6056: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 81 KiB/s wr, 8 op/s 2023-03-09T17:51:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10393.359606 secs 2023-03-09T17:51:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10393.359606 secs 2023-03-09T17:51:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10393.359606 secs 2023-03-09T17:51:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10855.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:51:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:24 smithi122 ceph-mon[93869]: pgmap v6057: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 34 KiB/s wr, 7 op/s 2023-03-09T17:51:24.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:51:24.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:51:24.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:51:24.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:51:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:24 smithi181 ceph-mon[94189]: pgmap v6057: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 34 KiB/s wr, 7 op/s 2023-03-09T17:51:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:51:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:51:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:51:24.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:51:24.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:24 smithi038 ceph-mon[92422]: pgmap v6057: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 34 KiB/s wr, 7 op/s 2023-03-09T17:51:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:51:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:51:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:51:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:51:25.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:51:25.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:51:25.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:51:25.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:51:25.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:51:25.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:51:25.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:51:25.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:51:25.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:51:25.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:25 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T17:51:25.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:51:25.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:25 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T17:51:25.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:25 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi038 to 197.2M 2023-03-09T17:51:25.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:25 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi038 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-03-09T17:51:25.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10396.078629 secs 2023-03-09T17:51:25.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:25 smithi122 ceph-mon[93869]: pgmap v6058: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T17:51:25.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:51:25.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:51:25.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:51:25.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:51:25.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:25 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T17:51:25.731 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:25 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T17:51:25.731 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:51:25.731 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:51:25.731 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:51:25.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:51:25.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:51:25.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:51:25.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:51:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:51:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:51:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:51:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:51:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:51:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:25 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T17:51:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:51:25.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:25 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T17:51:25.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:25 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi038 to 197.2M 2023-03-09T17:51:25.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:25 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi038 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-03-09T17:51:25.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10396.078629 secs 2023-03-09T17:51:25.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:25 smithi181 ceph-mon[94189]: pgmap v6058: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T17:51:25.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:51:25.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:51:25.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:51:25.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:51:25.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:25 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T17:51:25.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:25 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T17:51:25.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:51:25.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:51:25.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:51:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:51:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:51:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:51:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:51:25.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:51:25.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:51:25.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:51:25.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:51:25.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:51:25.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:25 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T17:51:25.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:51:25.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:25 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T17:51:25.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:25 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi038 to 197.2M 2023-03-09T17:51:25.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:25 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi038 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-03-09T17:51:25.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10396.078629 secs 2023-03-09T17:51:25.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:25 smithi038 ceph-mon[92422]: pgmap v6058: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T17:51:25.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:51:25.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:51:25.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:51:25.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T17:51:25.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:25 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T17:51:25.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:25 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T17:51:25.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:51:25.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:51:25.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:51:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:28 smithi122 ceph-mon[93869]: pgmap v6059: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 7 op/s 2023-03-09T17:51:28.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10398.359719 secs 2023-03-09T17:51:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:28 smithi181 ceph-mon[94189]: pgmap v6059: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 7 op/s 2023-03-09T17:51:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10398.359719 secs 2023-03-09T17:51:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:28 smithi038 ceph-mon[92422]: pgmap v6059: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 7 op/s 2023-03-09T17:51:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10398.359719 secs 2023-03-09T17:51:28.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10860.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:51:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:30 smithi122 ceph-mon[93869]: pgmap v6060: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T17:51:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:30 smithi181 ceph-mon[94189]: pgmap v6060: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T17:51:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:30 smithi038 ceph-mon[92422]: pgmap v6060: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T17:51:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10401.078689 secs 2023-03-09T17:51:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10401.078689 secs 2023-03-09T17:51:31.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10401.078689 secs 2023-03-09T17:51:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:32 smithi122 ceph-mon[93869]: pgmap v6061: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2023-03-09T17:51:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:32 smithi181 ceph-mon[94189]: pgmap v6061: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2023-03-09T17:51:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:32 smithi038 ceph-mon[92422]: pgmap v6061: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2023-03-09T17:51:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10865.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:51:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:34 smithi122 ceph-mon[93869]: pgmap v6062: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 30 KiB/s wr, 7 op/s 2023-03-09T17:51:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:34 smithi181 ceph-mon[94189]: pgmap v6062: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 30 KiB/s wr, 7 op/s 2023-03-09T17:51:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:34 smithi038 ceph-mon[92422]: pgmap v6062: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 30 KiB/s wr, 7 op/s 2023-03-09T17:51:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10406.078766 secs 2023-03-09T17:51:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:36 smithi122 ceph-mon[93869]: pgmap v6063: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 7 op/s 2023-03-09T17:51:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:36 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:51:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10406.078766 secs 2023-03-09T17:51:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:36 smithi181 ceph-mon[94189]: pgmap v6063: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 7 op/s 2023-03-09T17:51:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:36 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:51:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10406.078766 secs 2023-03-09T17:51:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:36 smithi038 ceph-mon[92422]: pgmap v6063: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 7 op/s 2023-03-09T17:51:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:36 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:51:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:37 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:51:37.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:51:37.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:51:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:37 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:51:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:51:37.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:51:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:37 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:51:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:51:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:51:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:38 smithi122 ceph-mon[93869]: pgmap v6064: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 30 KiB/s wr, 8 op/s 2023-03-09T17:51:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:38 smithi181 ceph-mon[94189]: pgmap v6064: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 30 KiB/s wr, 8 op/s 2023-03-09T17:51:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:38 smithi038 ceph-mon[92422]: pgmap v6064: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 30 KiB/s wr, 8 op/s 2023-03-09T17:51:38.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10870.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:51:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:40 smithi122 ceph-mon[93869]: pgmap v6065: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 101 KiB/s wr, 8 op/s 2023-03-09T17:51:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:40 smithi181 ceph-mon[94189]: pgmap v6065: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 101 KiB/s wr, 8 op/s 2023-03-09T17:51:40.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:40 smithi038 ceph-mon[92422]: pgmap v6065: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 101 KiB/s wr, 8 op/s 2023-03-09T17:51:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10411.078844 secs 2023-03-09T17:51:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10411.078844 secs 2023-03-09T17:51:41.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10411.078844 secs 2023-03-09T17:51:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:42 smithi122 ceph-mon[93869]: pgmap v6066: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 118 KiB/s wr, 11 op/s 2023-03-09T17:51:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:42 smithi181 ceph-mon[94189]: pgmap v6066: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 118 KiB/s wr, 11 op/s 2023-03-09T17:51:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:42 smithi038 ceph-mon[92422]: pgmap v6066: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 118 KiB/s wr, 11 op/s 2023-03-09T17:51:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10875.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:51:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:44 smithi122 ceph-mon[93869]: pgmap v6067: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 210 KiB/s wr, 9 op/s 2023-03-09T17:51:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:44 smithi181 ceph-mon[94189]: pgmap v6067: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 210 KiB/s wr, 9 op/s 2023-03-09T17:51:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:44 smithi038 ceph-mon[92422]: pgmap v6067: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 210 KiB/s wr, 9 op/s 2023-03-09T17:51:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10416.078904 secs 2023-03-09T17:51:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:46 smithi122 ceph-mon[93869]: pgmap v6068: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 191 KiB/s wr, 10 op/s 2023-03-09T17:51:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10416.078904 secs 2023-03-09T17:51:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:46 smithi181 ceph-mon[94189]: pgmap v6068: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 191 KiB/s wr, 10 op/s 2023-03-09T17:51:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10416.078904 secs 2023-03-09T17:51:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:46 smithi038 ceph-mon[92422]: pgmap v6068: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 191 KiB/s wr, 10 op/s 2023-03-09T17:51:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:48 smithi122 ceph-mon[93869]: pgmap v6069: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 191 KiB/s wr, 11 op/s 2023-03-09T17:51:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:48 smithi181 ceph-mon[94189]: pgmap v6069: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 191 KiB/s wr, 11 op/s 2023-03-09T17:51:48.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:48 smithi038 ceph-mon[92422]: pgmap v6069: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 191 KiB/s wr, 11 op/s 2023-03-09T17:51:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10880.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:51:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:50 smithi122 ceph-mon[93869]: pgmap v6070: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 213 KiB/s wr, 11 op/s 2023-03-09T17:51:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:50 smithi181 ceph-mon[94189]: pgmap v6070: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 213 KiB/s wr, 11 op/s 2023-03-09T17:51:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:50 smithi038 ceph-mon[92422]: pgmap v6070: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 213 KiB/s wr, 11 op/s 2023-03-09T17:51:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10421.078967 secs 2023-03-09T17:51:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10421.078967 secs 2023-03-09T17:51:51.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10421.078967 secs 2023-03-09T17:51:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:52 smithi122 ceph-mon[93869]: pgmap v6071: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 150 KiB/s wr, 11 op/s 2023-03-09T17:51:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:52 smithi181 ceph-mon[94189]: pgmap v6071: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 150 KiB/s wr, 11 op/s 2023-03-09T17:51:52.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:52 smithi038 ceph-mon[92422]: pgmap v6071: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 150 KiB/s wr, 11 op/s 2023-03-09T17:51:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10423.360300 secs 2023-03-09T17:51:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10423.360300 secs 2023-03-09T17:51:53.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10423.360300 secs 2023-03-09T17:51:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10885.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:51:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:54 smithi122 ceph-mon[93869]: pgmap v6072: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 204 KiB/s wr, 9 op/s 2023-03-09T17:51:54.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:54 smithi181 ceph-mon[94189]: pgmap v6072: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 204 KiB/s wr, 9 op/s 2023-03-09T17:51:54.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:54 smithi038 ceph-mon[92422]: pgmap v6072: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 204 KiB/s wr, 9 op/s 2023-03-09T17:51:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10426.078981 secs 2023-03-09T17:51:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:56 smithi122 ceph-mon[93869]: pgmap v6073: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 159 KiB/s wr, 10 op/s 2023-03-09T17:51:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10426.078981 secs 2023-03-09T17:51:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:56 smithi181 ceph-mon[94189]: pgmap v6073: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 159 KiB/s wr, 10 op/s 2023-03-09T17:51:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10426.078981 secs 2023-03-09T17:51:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:56 smithi038 ceph-mon[92422]: pgmap v6073: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 159 KiB/s wr, 10 op/s 2023-03-09T17:51:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:57 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:51:57.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:51:57.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:51:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:57 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:51:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:51:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:51:57.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:57 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:51:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:51:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:51:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:58 smithi122 ceph-mon[93869]: pgmap v6074: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 150 KiB/s wr, 11 op/s 2023-03-09T17:51:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:51:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10428.360417 secs 2023-03-09T17:51:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:58 smithi181 ceph-mon[94189]: pgmap v6074: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 150 KiB/s wr, 11 op/s 2023-03-09T17:51:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:51:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10428.360417 secs 2023-03-09T17:51:58.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:58 smithi038 ceph-mon[92422]: pgmap v6074: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 150 KiB/s wr, 11 op/s 2023-03-09T17:51:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:51:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10428.360417 secs 2023-03-09T17:51:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10890.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:52:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:52:00 smithi122 ceph-mon[93869]: pgmap v6075: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 150 KiB/s wr, 9 op/s 2023-03-09T17:52:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:52:00 smithi181 ceph-mon[94189]: pgmap v6075: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 150 KiB/s wr, 9 op/s 2023-03-09T17:52:00.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:52:00 smithi038 ceph-mon[92422]: pgmap v6075: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 150 KiB/s wr, 9 op/s 2023-03-09T17:52:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:52:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10431.079041 secs 2023-03-09T17:52:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:52:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10431.079041 secs 2023-03-09T17:52:01.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:52:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10431.079041 secs 2023-03-09T17:52:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:52:02 smithi122 ceph-mon[93869]: pgmap v6076: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 210 KiB/s wr, 13 op/s 2023-03-09T17:52:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:52:02 smithi181 ceph-mon[94189]: pgmap v6076: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 210 KiB/s wr, 13 op/s 2023-03-09T17:52:02.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:52:02 smithi038 ceph-mon[92422]: pgmap v6076: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 210 KiB/s wr, 13 op/s 2023-03-09T17:52:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:52:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10433.360539 secs 2023-03-09T17:52:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:52:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10433.360539 secs 2023-03-09T17:52:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:52:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10433.360539 secs 2023-03-09T17:52:03.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10895.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:52:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:52:04 smithi122 ceph-mon[93869]: pgmap v6077: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 202 KiB/s wr, 11 op/s 2023-03-09T17:52:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:52:04 smithi181 ceph-mon[94189]: pgmap v6077: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 202 KiB/s wr, 11 op/s 2023-03-09T17:52:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:52:04 smithi038 ceph-mon[92422]: pgmap v6077: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 202 KiB/s wr, 11 op/s 2023-03-09T17:52:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:52:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10436.079100 secs 2023-03-09T17:52:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:52:06 smithi122 ceph-mon[93869]: pgmap v6078: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 162 KiB/s wr, 23 op/s 2023-03-09T17:52:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:52:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10436.079100 secs 2023-03-09T17:52:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:52:06 smithi181 ceph-mon[94189]: pgmap v6078: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 162 KiB/s wr, 23 op/s 2023-03-09T17:52:06.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:52:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10436.079100 secs 2023-03-09T17:52:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:52:06 smithi038 ceph-mon[92422]: pgmap v6078: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 162 KiB/s wr, 23 op/s 2023-03-09T17:52:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:52:08 smithi122 ceph-mon[93869]: pgmap v6079: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 114 KiB/s wr, 23 op/s 2023-03-09T17:52:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:52:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10438.360661 secs 2023-03-09T17:52:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:52:08 smithi181 ceph-mon[94189]: pgmap v6079: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 114 KiB/s wr, 23 op/s 2023-03-09T17:52:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:52:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10438.360661 secs 2023-03-09T17:52:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:52:08 smithi038 ceph-mon[92422]: pgmap v6079: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 114 KiB/s wr, 23 op/s 2023-03-09T17:52:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:52:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10438.360661 secs 2023-03-09T17:52:08.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10900.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:52:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:52:10 smithi122 ceph-mon[93869]: pgmap v6080: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 114 KiB/s wr, 22 op/s 2023-03-09T17:52:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:52:10 smithi122 ceph-mon[93869]: osdmap e593: 12 total, 12 up, 12 in 2023-03-09T17:52:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:52:10 smithi181 ceph-mon[94189]: pgmap v6080: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 114 KiB/s wr, 22 op/s 2023-03-09T17:52:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:52:10 smithi181 ceph-mon[94189]: osdmap e593: 12 total, 12 up, 12 in 2023-03-09T17:52:10.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:52:10 smithi038 ceph-mon[92422]: pgmap v6080: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 114 KiB/s wr, 22 op/s 2023-03-09T17:52:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:52:10 smithi038 ceph-mon[92422]: osdmap e593: 12 total, 12 up, 12 in 2023-03-09T17:52:11.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:52:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10441.079161 secs 2023-03-09T17:52:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:52:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10441.079161 secs 2023-03-09T17:52:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:52:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10441.079161 secs 2023-03-09T17:52:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:52:12 smithi038 ceph-mon[92422]: pgmap v6082: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 48 KiB/s wr, 23 op/s 2023-03-09T17:52:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:52:12 smithi122 ceph-mon[93869]: pgmap v6082: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 48 KiB/s wr, 23 op/s 2023-03-09T17:52:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:52:12 smithi181 ceph-mon[94189]: pgmap v6082: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 48 KiB/s wr, 23 op/s 2023-03-09T17:52:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:52:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10443.360779 secs 2023-03-09T17:52:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:52:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10443.360779 secs 2023-03-09T17:52:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:52:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10443.360779 secs 2023-03-09T17:52:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10905.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:52:14.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:52:14 smithi122 ceph-mon[93869]: pgmap v6083: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 49 KiB/s wr, 23 op/s 2023-03-09T17:52:14.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:52:14 smithi181 ceph-mon[94189]: pgmap v6083: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 49 KiB/s wr, 23 op/s 2023-03-09T17:52:14.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:52:14 smithi038 ceph-mon[92422]: pgmap v6083: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 49 KiB/s wr, 23 op/s 2023-03-09T17:52:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:52:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10446.079224 secs 2023-03-09T17:52:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:52:16 smithi122 ceph-mon[93869]: pgmap v6084: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 8 op/s 2023-03-09T17:52:16.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:52:16 smithi122 ceph-mon[93869]: osdmap e594: 12 total, 12 up, 12 in 2023-03-09T17:52:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:52:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10446.079224 secs 2023-03-09T17:52:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:52:16 smithi181 ceph-mon[94189]: pgmap v6084: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 8 op/s 2023-03-09T17:52:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:52:16 smithi181 ceph-mon[94189]: osdmap e594: 12 total, 12 up, 12 in 2023-03-09T17:52:16.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:52:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10446.079224 secs 2023-03-09T17:52:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:52:16 smithi038 ceph-mon[92422]: pgmap v6084: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 8 op/s 2023-03-09T17:52:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:52:16 smithi038 ceph-mon[92422]: osdmap e594: 12 total, 12 up, 12 in 2023-03-09T17:52:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:52:18 smithi122 ceph-mon[93869]: pgmap v6086: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 KiB/s wr, 9 op/s 2023-03-09T17:52:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:52:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10448.360879 secs 2023-03-09T17:52:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:52:18 smithi181 ceph-mon[94189]: pgmap v6086: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 KiB/s wr, 9 op/s 2023-03-09T17:52:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:52:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10448.360879 secs 2023-03-09T17:52:18.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:52:18 smithi038 ceph-mon[92422]: pgmap v6086: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 KiB/s wr, 9 op/s 2023-03-09T17:52:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:52:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10448.360879 secs 2023-03-09T17:52:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10910.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:52:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:52:20 smithi122 ceph-mon[93869]: pgmap v6087: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.9 KiB/s wr, 6 op/s 2023-03-09T17:52:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:52:20 smithi181 ceph-mon[94189]: pgmap v6087: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.9 KiB/s wr, 6 op/s 2023-03-09T17:52:20.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:52:20 smithi038 ceph-mon[92422]: pgmap v6087: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.9 KiB/s wr, 6 op/s 2023-03-09T17:52:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:52:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10451.079284 secs 2023-03-09T17:52:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:52:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10451.079284 secs 2023-03-09T17:52:21.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:52:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10451.079284 secs 2023-03-09T17:52:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:52:22 smithi122 ceph-mon[93869]: pgmap v6088: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 KiB/s wr, 7 op/s 2023-03-09T17:52:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:52:22 smithi181 ceph-mon[94189]: pgmap v6088: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 KiB/s wr, 7 op/s 2023-03-09T17:52:22.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:52:22 smithi038 ceph-mon[92422]: pgmap v6088: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 KiB/s wr, 7 op/s 2023-03-09T17:52:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:52:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10453.360998 secs 2023-03-09T17:52:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:52:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10453.360998 secs 2023-03-09T17:52:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:52:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10453.360998 secs 2023-03-09T17:52:23.809 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10915.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:52:24.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:52:24 smithi122 ceph-mon[93869]: pgmap v6089: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 35 KiB/s wr, 7 op/s 2023-03-09T17:52:24.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:52:24 smithi181 ceph-mon[94189]: pgmap v6089: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 35 KiB/s wr, 7 op/s 2023-03-09T17:52:24.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:52:24 smithi038 ceph-mon[92422]: pgmap v6089: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 35 KiB/s wr, 7 op/s 2023-03-09T17:52:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:52:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10456.079362 secs 2023-03-09T17:52:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:52:26 smithi122 ceph-mon[93869]: pgmap v6090: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 35 KiB/s wr, 7 op/s 2023-03-09T17:52:26.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:52:26 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:52:26.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:52:26 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:52:26.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:52:26 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:52:26.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:52:26 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:52:26.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:52:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10456.079362 secs 2023-03-09T17:52:26.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:52:26 smithi181 ceph-mon[94189]: pgmap v6090: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 35 KiB/s wr, 7 op/s 2023-03-09T17:52:26.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:52:26 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:52:26.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:52:26 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:52:26.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:52:26 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:52:26.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:52:26 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:52:26.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:52:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10456.079362 secs 2023-03-09T17:52:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:52:26 smithi038 ceph-mon[92422]: pgmap v6090: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 35 KiB/s wr, 7 op/s 2023-03-09T17:52:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:52:26 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:52:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:52:26 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:52:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:52:26 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:52:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:52:26 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:52:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:52:28 smithi122 ceph-mon[93869]: pgmap v6091: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 33 KiB/s wr, 7 op/s 2023-03-09T17:52:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:52:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10458.361135 secs 2023-03-09T17:52:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:52:28 smithi181 ceph-mon[94189]: pgmap v6091: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 33 KiB/s wr, 7 op/s 2023-03-09T17:52:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:52:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10458.361135 secs 2023-03-09T17:52:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:52:28 smithi038 ceph-mon[92422]: pgmap v6091: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 33 KiB/s wr, 7 op/s 2023-03-09T17:52:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:52:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10458.361135 secs 2023-03-09T17:52:28.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10920.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:52:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:52:30 smithi122 ceph-mon[93869]: pgmap v6092: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 7 op/s 2023-03-09T17:52:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:52:30 smithi181 ceph-mon[94189]: pgmap v6092: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 7 op/s 2023-03-09T17:52:30.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:52:30 smithi038 ceph-mon[92422]: pgmap v6092: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 7 op/s 2023-03-09T17:52:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:52:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10461.079460 secs 2023-03-09T17:52:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:52:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10461.079460 secs 2023-03-09T17:52:31.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:52:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10461.079460 secs 2023-03-09T17:52:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:52:32 smithi122 ceph-mon[93869]: pgmap v6093: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 30 KiB/s wr, 9 op/s 2023-03-09T17:52:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:52:32 smithi181 ceph-mon[94189]: pgmap v6093: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 30 KiB/s wr, 9 op/s 2023-03-09T17:52:32.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:52:32 smithi038 ceph-mon[92422]: pgmap v6093: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 30 KiB/s wr, 9 op/s 2023-03-09T17:52:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:52:33 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 10463.361176 secs 2023-03-09T17:52:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:52:33 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 10463.361176 secs 2023-03-09T17:52:33.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:52:33 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 10463.361176 secs 2023-03-09T17:52:33.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10925.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:52:34.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:52:34 smithi122 ceph-mon[93869]: pgmap v6094: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2023-03-09T17:52:34.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:52:34 smithi181 ceph-mon[94189]: pgmap v6094: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2023-03-09T17:52:34.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:52:34 smithi038 ceph-mon[92422]: pgmap v6094: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2023-03-09T17:52:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:52:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10466.079519 secs 2023-03-09T17:52:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:52:36 smithi122 ceph-mon[93869]: pgmap v6095: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 98 KiB/s wr, 7 op/s 2023-03-09T17:52:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:52:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10466.079519 secs 2023-03-09T17:52:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:52:36 smithi181 ceph-mon[94189]: pgmap v6095: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 98 KiB/s wr, 7 op/s 2023-03-09T17:52:36.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:52:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10466.079519 secs 2023-03-09T17:52:36.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:52:36 smithi038 ceph-mon[92422]: pgmap v6095: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 98 KiB/s wr, 7 op/s 2023-03-09T17:52:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:52:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:52:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:52:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:52:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:52:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:52:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:52:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:52:37.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:52:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:52:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:52:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:52:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:52:38 smithi122 ceph-mon[93869]: pgmap v6096: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 98 KiB/s wr, 8 op/s 2023-03-09T17:52:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:52:38 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 10468.361237 secs 2023-03-09T17:52:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:52:38 smithi181 ceph-mon[94189]: pgmap v6096: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 98 KiB/s wr, 8 op/s 2023-03-09T17:52:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:52:38 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 10468.361237 secs 2023-03-09T17:52:38.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:52:38 smithi038 ceph-mon[92422]: pgmap v6096: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 98 KiB/s wr, 8 op/s 2023-03-09T17:52:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:52:38 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 10468.361237 secs 2023-03-09T17:52:38.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10930.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:52:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:52:40 smithi122 ceph-mon[93869]: pgmap v6097: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 97 KiB/s wr, 6 op/s 2023-03-09T17:52:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:52:40 smithi181 ceph-mon[94189]: pgmap v6097: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 97 KiB/s wr, 6 op/s 2023-03-09T17:52:40.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:52:40 smithi038 ceph-mon[92422]: pgmap v6097: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 97 KiB/s wr, 6 op/s 2023-03-09T17:52:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:52:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10471.079581 secs 2023-03-09T17:52:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:52:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10471.079581 secs 2023-03-09T17:52:41.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:52:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10471.079581 secs 2023-03-09T17:52:42.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:52:42 smithi122 ceph-mon[93869]: pgmap v6098: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 156 KiB/s wr, 11 op/s 2023-03-09T17:52:42.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:52:42 smithi181 ceph-mon[94189]: pgmap v6098: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 156 KiB/s wr, 11 op/s 2023-03-09T17:52:42.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:52:42 smithi038 ceph-mon[92422]: pgmap v6098: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 156 KiB/s wr, 11 op/s 2023-03-09T17:52:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:52:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10473.361379 secs 2023-03-09T17:52:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:52:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10473.361379 secs 2023-03-09T17:52:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:52:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10473.361379 secs 2023-03-09T17:52:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10935.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:52:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:52:44 smithi122 ceph-mon[93869]: pgmap v6099: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 156 KiB/s wr, 8 op/s 2023-03-09T17:52:44.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:52:44 smithi181 ceph-mon[94189]: pgmap v6099: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 156 KiB/s wr, 8 op/s 2023-03-09T17:52:44.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:52:44 smithi038 ceph-mon[92422]: pgmap v6099: 129 pgs: 129 active+clean; 9.1 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 156 KiB/s wr, 8 op/s 2023-03-09T17:52:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:52:45 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:52:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:52:45 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:52:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:52:45 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:52:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:52:45 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:52:45.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:52:45 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:52:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:52:45 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:52:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:52:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10476.079640 secs 2023-03-09T17:52:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:52:46 smithi122 ceph-mon[93869]: pgmap v6100: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 241 KiB/s wr, 10 op/s 2023-03-09T17:52:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:52:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10476.079640 secs 2023-03-09T17:52:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:52:46 smithi181 ceph-mon[94189]: pgmap v6100: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 241 KiB/s wr, 10 op/s 2023-03-09T17:52:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:52:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10476.079640 secs 2023-03-09T17:52:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:52:46 smithi038 ceph-mon[92422]: pgmap v6100: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 241 KiB/s wr, 10 op/s 2023-03-09T17:52:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:52:48 smithi122 ceph-mon[93869]: pgmap v6101: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 154 KiB/s wr, 11 op/s 2023-03-09T17:52:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:52:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10478.361517 secs 2023-03-09T17:52:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:52:48 smithi181 ceph-mon[94189]: pgmap v6101: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 154 KiB/s wr, 11 op/s 2023-03-09T17:52:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:52:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10478.361517 secs 2023-03-09T17:52:48.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:52:48 smithi038 ceph-mon[92422]: pgmap v6101: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 154 KiB/s wr, 11 op/s 2023-03-09T17:52:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:52:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10478.361517 secs 2023-03-09T17:52:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10940.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:52:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:52:50 smithi122 ceph-mon[93869]: pgmap v6102: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 154 KiB/s wr, 10 op/s 2023-03-09T17:52:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:52:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10481.079702 secs 2023-03-09T17:52:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:52:50 smithi181 ceph-mon[94189]: pgmap v6102: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 154 KiB/s wr, 10 op/s 2023-03-09T17:52:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:52:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10481.079702 secs 2023-03-09T17:52:50.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:52:50 smithi038 ceph-mon[92422]: pgmap v6102: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 154 KiB/s wr, 10 op/s 2023-03-09T17:52:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:52:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10481.079702 secs 2023-03-09T17:52:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:52:52 smithi122 ceph-mon[93869]: pgmap v6103: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 193 KiB/s wr, 14 op/s 2023-03-09T17:52:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:52:52 smithi181 ceph-mon[94189]: pgmap v6103: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 193 KiB/s wr, 14 op/s 2023-03-09T17:52:52.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:52:52 smithi038 ceph-mon[92422]: pgmap v6103: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 193 KiB/s wr, 14 op/s 2023-03-09T17:52:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:52:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10483.361556 secs 2023-03-09T17:52:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:52:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10483.361556 secs 2023-03-09T17:52:53.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:52:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10483.361556 secs 2023-03-09T17:52:53.801 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10945.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:52:54.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:52:54 smithi122 ceph-mon[93869]: pgmap v6104: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 135 KiB/s wr, 10 op/s 2023-03-09T17:52:54.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:52:54 smithi181 ceph-mon[94189]: pgmap v6104: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 135 KiB/s wr, 10 op/s 2023-03-09T17:52:54.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:52:54 smithi038 ceph-mon[92422]: pgmap v6104: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 135 KiB/s wr, 10 op/s 2023-03-09T17:52:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:52:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10486.079762 secs 2023-03-09T17:52:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:52:55 smithi122 ceph-mon[93869]: pgmap v6105: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 143 KiB/s wr, 10 op/s 2023-03-09T17:52:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:52:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10486.079762 secs 2023-03-09T17:52:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:52:55 smithi181 ceph-mon[94189]: pgmap v6105: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 143 KiB/s wr, 10 op/s 2023-03-09T17:52:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:52:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10486.079762 secs 2023-03-09T17:52:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:52:55 smithi038 ceph-mon[92422]: pgmap v6105: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 143 KiB/s wr, 10 op/s 2023-03-09T17:52:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:52:58 smithi122 ceph-mon[93869]: pgmap v6106: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 135 KiB/s wr, 11 op/s 2023-03-09T17:52:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:52:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10488.361655 secs 2023-03-09T17:52:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:52:58 smithi181 ceph-mon[94189]: pgmap v6106: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 135 KiB/s wr, 11 op/s 2023-03-09T17:52:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:52:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10488.361655 secs 2023-03-09T17:52:58.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:52:58 smithi038 ceph-mon[92422]: pgmap v6106: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 135 KiB/s wr, 11 op/s 2023-03-09T17:52:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:52:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10488.361655 secs 2023-03-09T17:52:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10950.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:53:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:53:00 smithi122 ceph-mon[93869]: pgmap v6107: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 125 KiB/s wr, 10 op/s 2023-03-09T17:53:00.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:53:00 smithi181 ceph-mon[94189]: pgmap v6107: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 125 KiB/s wr, 10 op/s 2023-03-09T17:53:00.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:53:00 smithi038 ceph-mon[92422]: pgmap v6107: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 125 KiB/s wr, 10 op/s 2023-03-09T17:53:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:53:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10491.079823 secs 2023-03-09T17:53:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:53:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10491.079823 secs 2023-03-09T17:53:01.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:53:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10491.079823 secs 2023-03-09T17:53:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:53:02 smithi122 ceph-mon[93869]: pgmap v6108: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 174 KiB/s wr, 15 op/s 2023-03-09T17:53:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:53:02 smithi181 ceph-mon[94189]: pgmap v6108: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 174 KiB/s wr, 15 op/s 2023-03-09T17:53:02.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:53:02 smithi038 ceph-mon[92422]: pgmap v6108: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 174 KiB/s wr, 15 op/s 2023-03-09T17:53:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:53:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10493.361773 secs 2023-03-09T17:53:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:53:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10493.361773 secs 2023-03-09T17:53:03.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:53:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10493.361773 secs 2023-03-09T17:53:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10955.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:53:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:53:04 smithi122 ceph-mon[93869]: pgmap v6109: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 155 KiB/s wr, 23 op/s 2023-03-09T17:53:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:53:04 smithi181 ceph-mon[94189]: pgmap v6109: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 155 KiB/s wr, 23 op/s 2023-03-09T17:53:04.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:53:04 smithi038 ceph-mon[92422]: pgmap v6109: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 155 KiB/s wr, 23 op/s 2023-03-09T17:53:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:53:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10496.079911 secs 2023-03-09T17:53:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:53:06 smithi122 ceph-mon[93869]: pgmap v6110: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 163 KiB/s wr, 22 op/s 2023-03-09T17:53:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:53:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10496.079911 secs 2023-03-09T17:53:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:53:06 smithi181 ceph-mon[94189]: pgmap v6110: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 163 KiB/s wr, 22 op/s 2023-03-09T17:53:06.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:53:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10496.079911 secs 2023-03-09T17:53:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:53:06 smithi038 ceph-mon[92422]: pgmap v6110: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 163 KiB/s wr, 22 op/s 2023-03-09T17:53:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:53:08 smithi122 ceph-mon[93869]: pgmap v6111: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 155 KiB/s wr, 24 op/s 2023-03-09T17:53:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:53:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10498.361897 secs 2023-03-09T17:53:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:53:08 smithi181 ceph-mon[94189]: pgmap v6111: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 155 KiB/s wr, 24 op/s 2023-03-09T17:53:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:53:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10498.361897 secs 2023-03-09T17:53:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:53:08 smithi038 ceph-mon[92422]: pgmap v6111: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 155 KiB/s wr, 24 op/s 2023-03-09T17:53:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:53:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10498.361897 secs 2023-03-09T17:53:08.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10960.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:53:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:53:10 smithi122 ceph-mon[93869]: pgmap v6112: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 79 KiB/s wr, 22 op/s 2023-03-09T17:53:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:53:10 smithi122 ceph-mon[93869]: osdmap e595: 12 total, 12 up, 12 in 2023-03-09T17:53:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:53:10 smithi181 ceph-mon[94189]: pgmap v6112: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 79 KiB/s wr, 22 op/s 2023-03-09T17:53:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:53:10 smithi181 ceph-mon[94189]: osdmap e595: 12 total, 12 up, 12 in 2023-03-09T17:53:10.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:53:10 smithi038 ceph-mon[92422]: pgmap v6112: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 79 KiB/s wr, 22 op/s 2023-03-09T17:53:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:53:10 smithi038 ceph-mon[92422]: osdmap e595: 12 total, 12 up, 12 in 2023-03-09T17:53:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:53:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10501.079986 secs 2023-03-09T17:53:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:53:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10501.079986 secs 2023-03-09T17:53:11.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:53:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10501.079986 secs 2023-03-09T17:53:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:53:12 smithi122 ceph-mon[93869]: pgmap v6114: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 59 KiB/s wr, 22 op/s 2023-03-09T17:53:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:53:12 smithi181 ceph-mon[94189]: pgmap v6114: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 59 KiB/s wr, 22 op/s 2023-03-09T17:53:12.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:53:12 smithi038 ceph-mon[92422]: pgmap v6114: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 59 KiB/s wr, 22 op/s 2023-03-09T17:53:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:53:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10503.362018 secs 2023-03-09T17:53:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:53:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10503.362018 secs 2023-03-09T17:53:13.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:53:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10503.362018 secs 2023-03-09T17:53:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10965.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:53:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:53:14 smithi122 ceph-mon[93869]: pgmap v6115: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 34 KiB/s wr, 8 op/s 2023-03-09T17:53:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:53:14 smithi181 ceph-mon[94189]: pgmap v6115: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 34 KiB/s wr, 8 op/s 2023-03-09T17:53:14.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:53:14 smithi038 ceph-mon[92422]: pgmap v6115: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 34 KiB/s wr, 8 op/s 2023-03-09T17:53:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:53:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10506.080046 secs 2023-03-09T17:53:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:53:16 smithi122 ceph-mon[93869]: pgmap v6116: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 24 KiB/s wr, 8 op/s 2023-03-09T17:53:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:53:16 smithi122 ceph-mon[93869]: osdmap e596: 12 total, 12 up, 12 in 2023-03-09T17:53:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:53:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10506.080046 secs 2023-03-09T17:53:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:53:16 smithi181 ceph-mon[94189]: pgmap v6116: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 24 KiB/s wr, 8 op/s 2023-03-09T17:53:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:53:16 smithi181 ceph-mon[94189]: osdmap e596: 12 total, 12 up, 12 in 2023-03-09T17:53:16.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:53:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10506.080046 secs 2023-03-09T17:53:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:53:16 smithi038 ceph-mon[92422]: pgmap v6116: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 24 KiB/s wr, 8 op/s 2023-03-09T17:53:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:53:16 smithi038 ceph-mon[92422]: osdmap e596: 12 total, 12 up, 12 in 2023-03-09T17:53:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:53:18 smithi122 ceph-mon[93869]: pgmap v6118: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 30 KiB/s wr, 9 op/s 2023-03-09T17:53:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:53:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10508.362131 secs 2023-03-09T17:53:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:53:18 smithi181 ceph-mon[94189]: pgmap v6118: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 30 KiB/s wr, 9 op/s 2023-03-09T17:53:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:53:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10508.362131 secs 2023-03-09T17:53:18.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:53:18 smithi038 ceph-mon[92422]: pgmap v6118: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 30 KiB/s wr, 9 op/s 2023-03-09T17:53:18.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:53:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10508.362131 secs 2023-03-09T17:53:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10970.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:53:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:53:20 smithi122 ceph-mon[93869]: pgmap v6119: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 6 op/s 2023-03-09T17:53:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:53:20 smithi181 ceph-mon[94189]: pgmap v6119: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 6 op/s 2023-03-09T17:53:20.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:53:20 smithi038 ceph-mon[92422]: pgmap v6119: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 6 op/s 2023-03-09T17:53:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:53:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10511.080102 secs 2023-03-09T17:53:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:53:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10511.080102 secs 2023-03-09T17:53:21.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:53:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10511.080102 secs 2023-03-09T17:53:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:53:22 smithi122 ceph-mon[93869]: pgmap v6120: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 KiB/s wr, 7 op/s 2023-03-09T17:53:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:53:22 smithi181 ceph-mon[94189]: pgmap v6120: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 KiB/s wr, 7 op/s 2023-03-09T17:53:22.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:53:22 smithi038 ceph-mon[92422]: pgmap v6120: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 KiB/s wr, 7 op/s 2023-03-09T17:53:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:53:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10513.362177 secs 2023-03-09T17:53:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:53:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10513.362177 secs 2023-03-09T17:53:23.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:53:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10513.362177 secs 2023-03-09T17:53:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10975.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:53:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:53:24 smithi122 ceph-mon[93869]: pgmap v6121: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 KiB/s wr, 7 op/s 2023-03-09T17:53:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:53:24 smithi181 ceph-mon[94189]: pgmap v6121: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 KiB/s wr, 7 op/s 2023-03-09T17:53:24.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:53:24 smithi038 ceph-mon[92422]: pgmap v6121: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 KiB/s wr, 7 op/s 2023-03-09T17:53:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:53:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10516.080166 secs 2023-03-09T17:53:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:53:26 smithi122 ceph-mon[93869]: pgmap v6122: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 KiB/s wr, 7 op/s 2023-03-09T17:53:26.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:53:26 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:53:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:53:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10516.080166 secs 2023-03-09T17:53:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:53:26 smithi181 ceph-mon[94189]: pgmap v6122: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 KiB/s wr, 7 op/s 2023-03-09T17:53:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:53:26 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:53:26.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:53:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10516.080166 secs 2023-03-09T17:53:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:53:26 smithi038 ceph-mon[92422]: pgmap v6122: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 KiB/s wr, 7 op/s 2023-03-09T17:53:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:53:26 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:53:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:53:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:53:27.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:53:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:53:27.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:53:27 smithi122 ceph-mon[93869]: pgmap v6123: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 33 KiB/s wr, 7 op/s 2023-03-09T17:53:27.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:53:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:53:27.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:53:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:53:27.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:53:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:53:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:53:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:53:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:53:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:53:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:53:27 smithi181 ceph-mon[94189]: pgmap v6123: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 33 KiB/s wr, 7 op/s 2023-03-09T17:53:27.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:53:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:53:27.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:53:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:53:27.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:53:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:53:27.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:53:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:53:27.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:53:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:53:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:53:27 smithi038 ceph-mon[92422]: pgmap v6123: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 33 KiB/s wr, 7 op/s 2023-03-09T17:53:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:53:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:53:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:53:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:53:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:53:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:53:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10980.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:53:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:53:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10518.362315 secs 2023-03-09T17:53:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:53:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10518.362315 secs 2023-03-09T17:53:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:53:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10518.362315 secs 2023-03-09T17:53:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:53:29 smithi122 ceph-mon[93869]: pgmap v6124: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 29 KiB/s wr, 6 op/s 2023-03-09T17:53:29.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:53:29 smithi181 ceph-mon[94189]: pgmap v6124: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 29 KiB/s wr, 6 op/s 2023-03-09T17:53:29.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:53:29 smithi038 ceph-mon[92422]: pgmap v6124: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 29 KiB/s wr, 6 op/s 2023-03-09T17:53:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:53:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10521.080224 secs 2023-03-09T17:53:30.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:53:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10521.080224 secs 2023-03-09T17:53:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:53:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10521.080224 secs 2023-03-09T17:53:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:53:32 smithi122 ceph-mon[93869]: pgmap v6125: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 29 KiB/s wr, 8 op/s 2023-03-09T17:53:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:53:32 smithi181 ceph-mon[94189]: pgmap v6125: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 29 KiB/s wr, 8 op/s 2023-03-09T17:53:32.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:53:32 smithi038 ceph-mon[92422]: pgmap v6125: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 29 KiB/s wr, 8 op/s 2023-03-09T17:53:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:53:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10523.362435 secs 2023-03-09T17:53:33.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:53:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10523.362435 secs 2023-03-09T17:53:33.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:53:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10523.362435 secs 2023-03-09T17:53:33.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10985.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:53:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:53:34 smithi122 ceph-mon[93869]: pgmap v6126: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T17:53:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:53:34 smithi181 ceph-mon[94189]: pgmap v6126: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T17:53:34.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:53:34 smithi038 ceph-mon[92422]: pgmap v6126: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T17:53:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:53:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10526.080311 secs 2023-03-09T17:53:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:53:36 smithi122 ceph-mon[93869]: pgmap v6127: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 6 op/s 2023-03-09T17:53:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:53:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10526.080311 secs 2023-03-09T17:53:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:53:36 smithi181 ceph-mon[94189]: pgmap v6127: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 6 op/s 2023-03-09T17:53:36.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:53:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10526.080311 secs 2023-03-09T17:53:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:53:36 smithi038 ceph-mon[92422]: pgmap v6127: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 6 op/s 2023-03-09T17:53:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:53:38 smithi122 ceph-mon[93869]: pgmap v6128: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 29 KiB/s wr, 8 op/s 2023-03-09T17:53:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:53:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10528.362558 secs 2023-03-09T17:53:38.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:53:38 smithi181 ceph-mon[94189]: pgmap v6128: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 29 KiB/s wr, 8 op/s 2023-03-09T17:53:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:53:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10528.362558 secs 2023-03-09T17:53:38.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:53:38 smithi038 ceph-mon[92422]: pgmap v6128: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 29 KiB/s wr, 8 op/s 2023-03-09T17:53:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:53:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10528.362558 secs 2023-03-09T17:53:38.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10990.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:53:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:53:40 smithi122 ceph-mon[93869]: pgmap v6129: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 29 KiB/s wr, 6 op/s 2023-03-09T17:53:40.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:53:40 smithi181 ceph-mon[94189]: pgmap v6129: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 29 KiB/s wr, 6 op/s 2023-03-09T17:53:40.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:53:40 smithi038 ceph-mon[92422]: pgmap v6129: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 29 KiB/s wr, 6 op/s 2023-03-09T17:53:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:53:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10531.080370 secs 2023-03-09T17:53:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:53:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10531.080370 secs 2023-03-09T17:53:41.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:53:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10531.080370 secs 2023-03-09T17:53:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:53:42 smithi122 ceph-mon[93869]: pgmap v6130: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 132 KiB/s wr, 10 op/s 2023-03-09T17:53:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:53:42 smithi181 ceph-mon[94189]: pgmap v6130: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 132 KiB/s wr, 10 op/s 2023-03-09T17:53:42.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:53:42 smithi038 ceph-mon[92422]: pgmap v6130: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 132 KiB/s wr, 10 op/s 2023-03-09T17:53:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 10995.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:53:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:53:44 smithi122 ceph-mon[93869]: pgmap v6131: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 132 KiB/s wr, 8 op/s 2023-03-09T17:53:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:53:44 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:53:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:53:44 smithi181 ceph-mon[94189]: pgmap v6131: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 132 KiB/s wr, 8 op/s 2023-03-09T17:53:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:53:44 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:53:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:53:44 smithi038 ceph-mon[92422]: pgmap v6131: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 132 KiB/s wr, 8 op/s 2023-03-09T17:53:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:53:44 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:53:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:53:45 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:53:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:53:45 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:53:45.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:53:45 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:53:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:53:45 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:53:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:53:45 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:53:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:53:45 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:53:45.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:53:45 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:53:45.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:53:45 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:53:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:53:45 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:53:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:53:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10536.080429 secs 2023-03-09T17:53:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:53:46 smithi122 ceph-mon[93869]: pgmap v6132: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 132 KiB/s wr, 8 op/s 2023-03-09T17:53:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:53:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10536.080429 secs 2023-03-09T17:53:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:53:46 smithi181 ceph-mon[94189]: pgmap v6132: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 132 KiB/s wr, 8 op/s 2023-03-09T17:53:46.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:53:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10536.080429 secs 2023-03-09T17:53:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:53:46 smithi038 ceph-mon[92422]: pgmap v6132: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 132 KiB/s wr, 8 op/s 2023-03-09T17:53:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:53:48 smithi122 ceph-mon[93869]: pgmap v6133: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 141 KiB/s wr, 10 op/s 2023-03-09T17:53:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:53:48 smithi181 ceph-mon[94189]: pgmap v6133: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 141 KiB/s wr, 10 op/s 2023-03-09T17:53:48.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:53:48 smithi038 ceph-mon[92422]: pgmap v6133: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 141 KiB/s wr, 10 op/s 2023-03-09T17:53:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11000.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:53:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:53:50 smithi122 ceph-mon[93869]: pgmap v6134: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 113 KiB/s wr, 8 op/s 2023-03-09T17:53:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:53:50 smithi181 ceph-mon[94189]: pgmap v6134: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 113 KiB/s wr, 8 op/s 2023-03-09T17:53:50.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:53:50 smithi038 ceph-mon[92422]: pgmap v6134: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 113 KiB/s wr, 8 op/s 2023-03-09T17:53:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:53:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10541.080489 secs 2023-03-09T17:53:51.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:53:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10541.080489 secs 2023-03-09T17:53:51.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:53:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10541.080489 secs 2023-03-09T17:53:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:53:52 smithi122 ceph-mon[93869]: pgmap v6135: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 217 KiB/s wr, 13 op/s 2023-03-09T17:53:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:53:52 smithi181 ceph-mon[94189]: pgmap v6135: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 217 KiB/s wr, 13 op/s 2023-03-09T17:53:52.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:53:52 smithi038 ceph-mon[92422]: pgmap v6135: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 217 KiB/s wr, 13 op/s 2023-03-09T17:53:53.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11005.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:53:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:53:54 smithi122 ceph-mon[93869]: pgmap v6136: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 114 KiB/s wr, 9 op/s 2023-03-09T17:53:54.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:53:54 smithi181 ceph-mon[94189]: pgmap v6136: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 114 KiB/s wr, 9 op/s 2023-03-09T17:53:54.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:53:54 smithi038 ceph-mon[92422]: pgmap v6136: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 114 KiB/s wr, 9 op/s 2023-03-09T17:53:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:53:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10546.080548 secs 2023-03-09T17:53:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:53:56 smithi122 ceph-mon[93869]: pgmap v6137: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 123 KiB/s wr, 9 op/s 2023-03-09T17:53:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:53:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10546.080548 secs 2023-03-09T17:53:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:53:56 smithi181 ceph-mon[94189]: pgmap v6137: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 123 KiB/s wr, 9 op/s 2023-03-09T17:53:56.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:53:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10546.080548 secs 2023-03-09T17:53:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:53:56 smithi038 ceph-mon[92422]: pgmap v6137: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 123 KiB/s wr, 9 op/s 2023-03-09T17:53:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:53:58 smithi122 ceph-mon[93869]: pgmap v6138: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 174 KiB/s wr, 11 op/s 2023-03-09T17:53:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:53:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10548.363044 secs 2023-03-09T17:53:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:53:58 smithi181 ceph-mon[94189]: pgmap v6138: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 174 KiB/s wr, 11 op/s 2023-03-09T17:53:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:53:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10548.363044 secs 2023-03-09T17:53:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:53:58 smithi038 ceph-mon[92422]: pgmap v6138: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 174 KiB/s wr, 11 op/s 2023-03-09T17:53:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:53:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10548.363044 secs 2023-03-09T17:53:58.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11010.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:54:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:54:00 smithi122 ceph-mon[93869]: pgmap v6139: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 165 KiB/s wr, 10 op/s 2023-03-09T17:54:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:54:00 smithi181 ceph-mon[94189]: pgmap v6139: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 165 KiB/s wr, 10 op/s 2023-03-09T17:54:00.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:54:00 smithi038 ceph-mon[92422]: pgmap v6139: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 165 KiB/s wr, 10 op/s 2023-03-09T17:54:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:54:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10551.080611 secs 2023-03-09T17:54:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:54:01 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:54:01.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:54:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:54:01.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:54:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:54:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:54:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10551.080611 secs 2023-03-09T17:54:01.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:54:01 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:54:01.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:54:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:54:01.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:54:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:54:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:54:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10551.080611 secs 2023-03-09T17:54:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:54:01 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:54:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:54:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:54:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:54:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:54:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:54:02 smithi122 ceph-mon[93869]: pgmap v6140: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 204 KiB/s wr, 14 op/s 2023-03-09T17:54:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:54:02 smithi181 ceph-mon[94189]: pgmap v6140: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 204 KiB/s wr, 14 op/s 2023-03-09T17:54:02.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:54:02 smithi038 ceph-mon[92422]: pgmap v6140: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 204 KiB/s wr, 14 op/s 2023-03-09T17:54:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:54:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10553.363110 secs 2023-03-09T17:54:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:54:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10553.363110 secs 2023-03-09T17:54:03.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:54:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10553.363110 secs 2023-03-09T17:54:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11015.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:54:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:54:04 smithi181 ceph-mon[94189]: pgmap v6141: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 121 KiB/s wr, 22 op/s 2023-03-09T17:54:04.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:54:04 smithi038 ceph-mon[92422]: pgmap v6141: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 121 KiB/s wr, 22 op/s 2023-03-09T17:54:04.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:54:04 smithi122 ceph-mon[93869]: pgmap v6141: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 121 KiB/s wr, 22 op/s 2023-03-09T17:54:05.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:54:05 smithi038 ceph-mon[92422]: osdmap e597: 12 total, 12 up, 12 in 2023-03-09T17:54:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:54:05 smithi122 ceph-mon[93869]: osdmap e597: 12 total, 12 up, 12 in 2023-03-09T17:54:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:54:05 smithi181 ceph-mon[94189]: osdmap e597: 12 total, 12 up, 12 in 2023-03-09T17:54:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:54:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10556.080667 secs 2023-03-09T17:54:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:54:06 smithi122 ceph-mon[93869]: pgmap v6143: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 154 KiB/s wr, 26 op/s 2023-03-09T17:54:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:54:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10556.080667 secs 2023-03-09T17:54:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:54:06 smithi181 ceph-mon[94189]: pgmap v6143: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 154 KiB/s wr, 26 op/s 2023-03-09T17:54:06.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:54:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10556.080667 secs 2023-03-09T17:54:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:54:06 smithi038 ceph-mon[92422]: pgmap v6143: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 154 KiB/s wr, 26 op/s 2023-03-09T17:54:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:54:08 smithi122 ceph-mon[93869]: pgmap v6144: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 172 KiB/s wr, 26 op/s 2023-03-09T17:54:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:54:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10558.363229 secs 2023-03-09T17:54:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:54:08 smithi181 ceph-mon[94189]: pgmap v6144: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 172 KiB/s wr, 26 op/s 2023-03-09T17:54:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:54:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10558.363229 secs 2023-03-09T17:54:08.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:54:08 smithi038 ceph-mon[92422]: pgmap v6144: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 172 KiB/s wr, 26 op/s 2023-03-09T17:54:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:54:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10558.363229 secs 2023-03-09T17:54:08.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11020.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:54:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:54:10 smithi122 ceph-mon[93869]: pgmap v6145: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 171 KiB/s wr, 26 op/s 2023-03-09T17:54:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:54:10 smithi181 ceph-mon[94189]: pgmap v6145: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 171 KiB/s wr, 26 op/s 2023-03-09T17:54:10.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:54:10 smithi038 ceph-mon[92422]: pgmap v6145: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 171 KiB/s wr, 26 op/s 2023-03-09T17:54:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:54:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10561.080740 secs 2023-03-09T17:54:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:54:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10561.080740 secs 2023-03-09T17:54:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:54:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10561.080740 secs 2023-03-09T17:54:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:54:12 smithi122 ceph-mon[93869]: pgmap v6146: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 125 KiB/s wr, 23 op/s 2023-03-09T17:54:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:54:12 smithi181 ceph-mon[94189]: pgmap v6146: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 125 KiB/s wr, 23 op/s 2023-03-09T17:54:12.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:54:12 smithi038 ceph-mon[92422]: pgmap v6146: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 125 KiB/s wr, 23 op/s 2023-03-09T17:54:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:54:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10563.363343 secs 2023-03-09T17:54:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:54:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10563.363343 secs 2023-03-09T17:54:13.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:54:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10563.363343 secs 2023-03-09T17:54:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11025.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:54:14.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:54:14 smithi122 ceph-mon[93869]: pgmap v6147: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 100 KiB/s wr, 8 op/s 2023-03-09T17:54:14.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:54:14 smithi181 ceph-mon[94189]: pgmap v6147: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 100 KiB/s wr, 8 op/s 2023-03-09T17:54:14.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:54:14 smithi038 ceph-mon[92422]: pgmap v6147: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 100 KiB/s wr, 8 op/s 2023-03-09T17:54:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:54:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10566.080800 secs 2023-03-09T17:54:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:54:16 smithi122 ceph-mon[93869]: pgmap v6148: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 85 KiB/s wr, 8 op/s 2023-03-09T17:54:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:54:16 smithi122 ceph-mon[93869]: osdmap e598: 12 total, 12 up, 12 in 2023-03-09T17:54:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:54:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10566.080800 secs 2023-03-09T17:54:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:54:16 smithi181 ceph-mon[94189]: pgmap v6148: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 85 KiB/s wr, 8 op/s 2023-03-09T17:54:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:54:16 smithi181 ceph-mon[94189]: osdmap e598: 12 total, 12 up, 12 in 2023-03-09T17:54:16.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:54:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10566.080800 secs 2023-03-09T17:54:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:54:16 smithi038 ceph-mon[92422]: pgmap v6148: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 85 KiB/s wr, 8 op/s 2023-03-09T17:54:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:54:16 smithi038 ceph-mon[92422]: osdmap e598: 12 total, 12 up, 12 in 2023-03-09T17:54:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:54:18 smithi122 ceph-mon[93869]: pgmap v6150: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 7 op/s 2023-03-09T17:54:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:54:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10568.363464 secs 2023-03-09T17:54:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:54:18 smithi181 ceph-mon[94189]: pgmap v6150: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 7 op/s 2023-03-09T17:54:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:54:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10568.363464 secs 2023-03-09T17:54:18.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:54:18 smithi038 ceph-mon[92422]: pgmap v6150: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 7 op/s 2023-03-09T17:54:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:54:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10568.363464 secs 2023-03-09T17:54:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11030.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:54:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:54:20 smithi122 ceph-mon[93869]: pgmap v6151: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 7 op/s 2023-03-09T17:54:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:54:20 smithi181 ceph-mon[94189]: pgmap v6151: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 7 op/s 2023-03-09T17:54:20.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:54:20 smithi038 ceph-mon[92422]: pgmap v6151: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 7 op/s 2023-03-09T17:54:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:54:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10571.080861 secs 2023-03-09T17:54:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:54:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10571.080861 secs 2023-03-09T17:54:21.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:54:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10571.080861 secs 2023-03-09T17:54:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:54:22 smithi122 ceph-mon[93869]: pgmap v6152: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 7 op/s 2023-03-09T17:54:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:54:22 smithi181 ceph-mon[94189]: pgmap v6152: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 7 op/s 2023-03-09T17:54:22.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:54:22 smithi038 ceph-mon[92422]: pgmap v6152: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 7 op/s 2023-03-09T17:54:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:54:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10573.363581 secs 2023-03-09T17:54:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:54:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10573.363581 secs 2023-03-09T17:54:23.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:54:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10573.363581 secs 2023-03-09T17:54:23.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11035.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:54:24.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:54:24 smithi122 ceph-mon[93869]: pgmap v6153: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 7 op/s 2023-03-09T17:54:24.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:54:24 smithi181 ceph-mon[94189]: pgmap v6153: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 7 op/s 2023-03-09T17:54:24.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:54:24 smithi038 ceph-mon[92422]: pgmap v6153: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 7 op/s 2023-03-09T17:54:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:54:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10576.080922 secs 2023-03-09T17:54:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:54:26 smithi122 ceph-mon[93869]: pgmap v6154: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1023 B/s wr, 7 op/s 2023-03-09T17:54:26.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:54:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10576.080922 secs 2023-03-09T17:54:26.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:54:26 smithi181 ceph-mon[94189]: pgmap v6154: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1023 B/s wr, 7 op/s 2023-03-09T17:54:26.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:54:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10576.080922 secs 2023-03-09T17:54:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:54:26 smithi038 ceph-mon[92422]: pgmap v6154: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1023 B/s wr, 7 op/s 2023-03-09T17:54:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:54:28 smithi122 ceph-mon[93869]: pgmap v6155: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 34 KiB/s wr, 7 op/s 2023-03-09T17:54:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:54:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10578.363714 secs 2023-03-09T17:54:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:54:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:54:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:54:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:54:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:54:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:54:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:54:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:54:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:54:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:54:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:54:28 smithi181 ceph-mon[94189]: pgmap v6155: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 34 KiB/s wr, 7 op/s 2023-03-09T17:54:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:54:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10578.363714 secs 2023-03-09T17:54:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:54:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:54:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:54:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:54:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:54:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:54:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:54:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:54:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:54:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:54:28.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:54:28 smithi038 ceph-mon[92422]: pgmap v6155: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 34 KiB/s wr, 7 op/s 2023-03-09T17:54:28.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:54:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10578.363714 secs 2023-03-09T17:54:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:54:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:54:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:54:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:54:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:54:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:54:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:54:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:54:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:54:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:54:28.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11040.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:54:29.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:54:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:54:29.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:54:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:54:29.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:54:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:54:29.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:54:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:54:29.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:54:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:54:29.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:54:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:54:29.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:54:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:54:29.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:54:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:54:29.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:54:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:54:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:54:30 smithi122 ceph-mon[93869]: pgmap v6156: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 31 KiB/s wr, 7 op/s 2023-03-09T17:54:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:54:30 smithi181 ceph-mon[94189]: pgmap v6156: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 31 KiB/s wr, 7 op/s 2023-03-09T17:54:30.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:54:30 smithi038 ceph-mon[92422]: pgmap v6156: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 31 KiB/s wr, 7 op/s 2023-03-09T17:54:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:54:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10581.080991 secs 2023-03-09T17:54:31.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:54:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10581.080991 secs 2023-03-09T17:54:31.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:54:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10581.080991 secs 2023-03-09T17:54:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:54:32 smithi122 ceph-mon[93869]: pgmap v6157: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 9 op/s 2023-03-09T17:54:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:54:32 smithi181 ceph-mon[94189]: pgmap v6157: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 9 op/s 2023-03-09T17:54:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:54:32 smithi038 ceph-mon[92422]: pgmap v6157: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 9 op/s 2023-03-09T17:54:33.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11045.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:54:34.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:54:34 smithi122 ceph-mon[93869]: pgmap v6158: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 57 KiB/s wr, 6 op/s 2023-03-09T17:54:34.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:54:34 smithi181 ceph-mon[94189]: pgmap v6158: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 57 KiB/s wr, 6 op/s 2023-03-09T17:54:34.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:54:34 smithi038 ceph-mon[92422]: pgmap v6158: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 57 KiB/s wr, 6 op/s 2023-03-09T17:54:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:54:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10586.081051 secs 2023-03-09T17:54:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:54:36 smithi122 ceph-mon[93869]: pgmap v6159: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 57 KiB/s wr, 6 op/s 2023-03-09T17:54:36.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:54:36 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:54:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:54:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10586.081051 secs 2023-03-09T17:54:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:54:36 smithi181 ceph-mon[94189]: pgmap v6159: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 57 KiB/s wr, 6 op/s 2023-03-09T17:54:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:54:36 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:54:36.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:54:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10586.081051 secs 2023-03-09T17:54:36.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:54:36 smithi038 ceph-mon[92422]: pgmap v6159: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 57 KiB/s wr, 6 op/s 2023-03-09T17:54:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:54:36 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:54:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:54:37 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:54:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:54:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:54:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:54:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:54:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:54:37 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:54:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:54:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:54:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:54:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:54:37.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:54:37 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:54:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:54:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:54:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:54:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:54:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:54:38 smithi122 ceph-mon[93869]: pgmap v6160: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 85 KiB/s wr, 9 op/s 2023-03-09T17:54:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:54:38 smithi181 ceph-mon[94189]: pgmap v6160: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 85 KiB/s wr, 9 op/s 2023-03-09T17:54:38.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:54:38 smithi038 ceph-mon[92422]: pgmap v6160: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 85 KiB/s wr, 9 op/s 2023-03-09T17:54:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11050.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:54:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:54:40 smithi122 ceph-mon[93869]: pgmap v6161: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 73 KiB/s wr, 7 op/s 2023-03-09T17:54:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:54:40 smithi181 ceph-mon[94189]: pgmap v6161: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 73 KiB/s wr, 7 op/s 2023-03-09T17:54:40.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:54:40 smithi038 ceph-mon[92422]: pgmap v6161: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 73 KiB/s wr, 7 op/s 2023-03-09T17:54:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:54:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10591.081112 secs 2023-03-09T17:54:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:54:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10591.081112 secs 2023-03-09T17:54:41.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:54:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10591.081112 secs 2023-03-09T17:54:42.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:54:42 smithi122 ceph-mon[93869]: pgmap v6162: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 159 KiB/s wr, 10 op/s 2023-03-09T17:54:42.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:54:42 smithi181 ceph-mon[94189]: pgmap v6162: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 159 KiB/s wr, 10 op/s 2023-03-09T17:54:42.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:54:42 smithi038 ceph-mon[92422]: pgmap v6162: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 159 KiB/s wr, 10 op/s 2023-03-09T17:54:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:54:43 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 10593.364019 secs 2023-03-09T17:54:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:54:43 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 10593.364019 secs 2023-03-09T17:54:43.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:54:43 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 10593.364019 secs 2023-03-09T17:54:43.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11055.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:54:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:54:44 smithi122 ceph-mon[93869]: pgmap v6163: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 133 KiB/s wr, 8 op/s 2023-03-09T17:54:44.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:54:44 smithi181 ceph-mon[94189]: pgmap v6163: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 133 KiB/s wr, 8 op/s 2023-03-09T17:54:44.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:54:44 smithi038 ceph-mon[92422]: pgmap v6163: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 133 KiB/s wr, 8 op/s 2023-03-09T17:54:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:54:45 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:54:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:54:45 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:54:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:54:45 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:54:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:54:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10596.081173 secs 2023-03-09T17:54:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:54:45 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:54:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:54:45 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:54:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:54:45 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:54:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:54:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10596.081173 secs 2023-03-09T17:54:45.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:54:45 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:54:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:54:45 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:54:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:54:45 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:54:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:54:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10596.081173 secs 2023-03-09T17:54:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:54:46 smithi122 ceph-mon[93869]: pgmap v6164: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 134 KiB/s wr, 9 op/s 2023-03-09T17:54:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:54:46 smithi181 ceph-mon[94189]: pgmap v6164: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 134 KiB/s wr, 9 op/s 2023-03-09T17:54:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:54:46 smithi038 ceph-mon[92422]: pgmap v6164: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 134 KiB/s wr, 9 op/s 2023-03-09T17:54:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:54:48 smithi122 ceph-mon[93869]: pgmap v6165: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 182 KiB/s wr, 12 op/s 2023-03-09T17:54:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:54:48 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 10598.364142 secs 2023-03-09T17:54:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:54:48 smithi181 ceph-mon[94189]: pgmap v6165: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 182 KiB/s wr, 12 op/s 2023-03-09T17:54:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:54:48 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 10598.364142 secs 2023-03-09T17:54:48.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:54:48 smithi038 ceph-mon[92422]: pgmap v6165: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 182 KiB/s wr, 12 op/s 2023-03-09T17:54:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:54:48 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 10598.364142 secs 2023-03-09T17:54:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11060.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:54:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:54:50 smithi122 ceph-mon[93869]: pgmap v6166: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 171 KiB/s wr, 11 op/s 2023-03-09T17:54:50.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:54:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10601.081245 secs 2023-03-09T17:54:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:54:50 smithi181 ceph-mon[94189]: pgmap v6166: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 171 KiB/s wr, 11 op/s 2023-03-09T17:54:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:54:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10601.081245 secs 2023-03-09T17:54:50.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:54:50 smithi038 ceph-mon[92422]: pgmap v6166: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 171 KiB/s wr, 11 op/s 2023-03-09T17:54:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:54:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10601.081245 secs 2023-03-09T17:54:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:54:52 smithi122 ceph-mon[93869]: pgmap v6167: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 228 KiB/s wr, 14 op/s 2023-03-09T17:54:52.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:54:52 smithi181 ceph-mon[94189]: pgmap v6167: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 228 KiB/s wr, 14 op/s 2023-03-09T17:54:52.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:54:52 smithi038 ceph-mon[92422]: pgmap v6167: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 228 KiB/s wr, 14 op/s 2023-03-09T17:54:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:54:53 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 10603.364256 secs 2023-03-09T17:54:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:54:53 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 10603.364256 secs 2023-03-09T17:54:53.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:54:53 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 10603.364256 secs 2023-03-09T17:54:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11065.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:54:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:54:54 smithi122 ceph-mon[93869]: pgmap v6168: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 141 KiB/s wr, 10 op/s 2023-03-09T17:54:54.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:54:54 smithi181 ceph-mon[94189]: pgmap v6168: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 141 KiB/s wr, 10 op/s 2023-03-09T17:54:54.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:54:54 smithi038 ceph-mon[92422]: pgmap v6168: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 141 KiB/s wr, 10 op/s 2023-03-09T17:54:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:54:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10606.081305 secs 2023-03-09T17:54:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:54:56 smithi122 ceph-mon[93869]: pgmap v6169: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 150 KiB/s wr, 10 op/s 2023-03-09T17:54:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:54:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10606.081305 secs 2023-03-09T17:54:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:54:56 smithi181 ceph-mon[94189]: pgmap v6169: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 150 KiB/s wr, 10 op/s 2023-03-09T17:54:56.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:54:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10606.081305 secs 2023-03-09T17:54:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:54:56 smithi038 ceph-mon[92422]: pgmap v6169: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 150 KiB/s wr, 10 op/s 2023-03-09T17:54:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:54:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10608.364376 secs 2023-03-09T17:54:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:54:58 smithi122 ceph-mon[93869]: pgmap v6170: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 162 KiB/s wr, 13 op/s 2023-03-09T17:54:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:54:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10608.364376 secs 2023-03-09T17:54:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:54:58 smithi181 ceph-mon[94189]: pgmap v6170: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 162 KiB/s wr, 13 op/s 2023-03-09T17:54:58.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:54:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10608.364376 secs 2023-03-09T17:54:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:54:58 smithi038 ceph-mon[92422]: pgmap v6170: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 162 KiB/s wr, 13 op/s 2023-03-09T17:54:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11070.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:55:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:55:00 smithi122 ceph-mon[93869]: pgmap v6171: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 114 KiB/s wr, 10 op/s 2023-03-09T17:55:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:55:00 smithi181 ceph-mon[94189]: pgmap v6171: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 114 KiB/s wr, 10 op/s 2023-03-09T17:55:00.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:55:00 smithi038 ceph-mon[92422]: pgmap v6171: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 114 KiB/s wr, 10 op/s 2023-03-09T17:55:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:55:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10611.081369 secs 2023-03-09T17:55:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:55:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:55:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:55:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:55:01.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:55:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10611.081369 secs 2023-03-09T17:55:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:55:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:55:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:55:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:55:01.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:55:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10611.081369 secs 2023-03-09T17:55:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:55:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:55:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:55:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:55:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:55:02 smithi122 ceph-mon[93869]: pgmap v6172: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 179 KiB/s wr, 13 op/s 2023-03-09T17:55:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:55:02 smithi181 ceph-mon[94189]: pgmap v6172: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 179 KiB/s wr, 13 op/s 2023-03-09T17:55:02.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:55:02 smithi038 ceph-mon[92422]: pgmap v6172: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 179 KiB/s wr, 13 op/s 2023-03-09T17:55:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:55:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10613.364494 secs 2023-03-09T17:55:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:55:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10613.364494 secs 2023-03-09T17:55:03.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:55:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10613.364494 secs 2023-03-09T17:55:03.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11075.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:55:04.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:55:04 smithi122 ceph-mon[93869]: pgmap v6173: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 126 KiB/s wr, 22 op/s 2023-03-09T17:55:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:55:04 smithi181 ceph-mon[94189]: pgmap v6173: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 126 KiB/s wr, 22 op/s 2023-03-09T17:55:04.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:55:04 smithi038 ceph-mon[92422]: pgmap v6173: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 126 KiB/s wr, 22 op/s 2023-03-09T17:55:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:55:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10616.081427 secs 2023-03-09T17:55:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:55:06 smithi122 ceph-mon[93869]: pgmap v6174: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 135 KiB/s wr, 22 op/s 2023-03-09T17:55:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:55:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10616.081427 secs 2023-03-09T17:55:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:55:06 smithi181 ceph-mon[94189]: pgmap v6174: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 135 KiB/s wr, 22 op/s 2023-03-09T17:55:06.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:55:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10616.081427 secs 2023-03-09T17:55:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:55:06 smithi038 ceph-mon[92422]: pgmap v6174: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 135 KiB/s wr, 22 op/s 2023-03-09T17:55:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:55:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10618.364561 secs 2023-03-09T17:55:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:55:08 smithi122 ceph-mon[93869]: pgmap v6175: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 158 KiB/s wr, 25 op/s 2023-03-09T17:55:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:55:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10618.364561 secs 2023-03-09T17:55:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:55:08 smithi181 ceph-mon[94189]: pgmap v6175: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 158 KiB/s wr, 25 op/s 2023-03-09T17:55:08.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:55:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10618.364561 secs 2023-03-09T17:55:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:55:08 smithi038 ceph-mon[92422]: pgmap v6175: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 158 KiB/s wr, 25 op/s 2023-03-09T17:55:08.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11080.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:55:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:55:10 smithi122 ceph-mon[93869]: pgmap v6176: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 145 KiB/s wr, 23 op/s 2023-03-09T17:55:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:55:10 smithi122 ceph-mon[93869]: osdmap e599: 12 total, 12 up, 12 in 2023-03-09T17:55:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:55:10 smithi181 ceph-mon[94189]: pgmap v6176: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 145 KiB/s wr, 23 op/s 2023-03-09T17:55:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:55:10 smithi181 ceph-mon[94189]: osdmap e599: 12 total, 12 up, 12 in 2023-03-09T17:55:10.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:55:10 smithi038 ceph-mon[92422]: pgmap v6176: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 145 KiB/s wr, 23 op/s 2023-03-09T17:55:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:55:10 smithi038 ceph-mon[92422]: osdmap e599: 12 total, 12 up, 12 in 2023-03-09T17:55:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:55:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10621.081487 secs 2023-03-09T17:55:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:55:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10621.081487 secs 2023-03-09T17:55:11.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:55:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10621.081487 secs 2023-03-09T17:55:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:55:12 smithi122 ceph-mon[93869]: pgmap v6178: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 75 KiB/s wr, 24 op/s 2023-03-09T17:55:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:55:12 smithi181 ceph-mon[94189]: pgmap v6178: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 75 KiB/s wr, 24 op/s 2023-03-09T17:55:12.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:55:12 smithi038 ceph-mon[92422]: pgmap v6178: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 75 KiB/s wr, 24 op/s 2023-03-09T17:55:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:55:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10623.364679 secs 2023-03-09T17:55:13.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:55:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10623.364679 secs 2023-03-09T17:55:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:55:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10623.364679 secs 2023-03-09T17:55:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11085.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:55:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:55:14 smithi122 ceph-mon[93869]: pgmap v6179: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 49 KiB/s wr, 9 op/s 2023-03-09T17:55:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:55:14 smithi181 ceph-mon[94189]: pgmap v6179: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 49 KiB/s wr, 9 op/s 2023-03-09T17:55:14.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:55:14 smithi038 ceph-mon[92422]: pgmap v6179: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 49 KiB/s wr, 9 op/s 2023-03-09T17:55:16.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:55:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10626.081547 secs 2023-03-09T17:55:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:55:16 smithi038 ceph-mon[92422]: pgmap v6180: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 40 KiB/s wr, 8 op/s 2023-03-09T17:55:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:55:16 smithi038 ceph-mon[92422]: osdmap e600: 12 total, 12 up, 12 in 2023-03-09T17:55:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:55:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10626.081547 secs 2023-03-09T17:55:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:55:16 smithi122 ceph-mon[93869]: pgmap v6180: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 40 KiB/s wr, 8 op/s 2023-03-09T17:55:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:55:16 smithi122 ceph-mon[93869]: osdmap e600: 12 total, 12 up, 12 in 2023-03-09T17:55:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:55:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10626.081547 secs 2023-03-09T17:55:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:55:16 smithi181 ceph-mon[94189]: pgmap v6180: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 40 KiB/s wr, 8 op/s 2023-03-09T17:55:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:55:16 smithi181 ceph-mon[94189]: osdmap e600: 12 total, 12 up, 12 in 2023-03-09T17:55:18.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:55:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10628.364803 secs 2023-03-09T17:55:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:55:18 smithi038 ceph-mon[92422]: pgmap v6182: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2023-03-09T17:55:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:55:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10628.364803 secs 2023-03-09T17:55:18.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:55:18 smithi122 ceph-mon[93869]: pgmap v6182: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2023-03-09T17:55:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:55:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10628.364803 secs 2023-03-09T17:55:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:55:18 smithi181 ceph-mon[94189]: pgmap v6182: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2023-03-09T17:55:18.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11090.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:55:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:55:20 smithi038 ceph-mon[92422]: pgmap v6183: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 7 op/s 2023-03-09T17:55:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:55:20 smithi122 ceph-mon[93869]: pgmap v6183: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 7 op/s 2023-03-09T17:55:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:55:20 smithi181 ceph-mon[94189]: pgmap v6183: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 7 op/s 2023-03-09T17:55:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:55:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10631.081607 secs 2023-03-09T17:55:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:55:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10631.081607 secs 2023-03-09T17:55:21.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:55:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10631.081607 secs 2023-03-09T17:55:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:55:22 smithi122 ceph-mon[93869]: pgmap v6184: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 6 op/s 2023-03-09T17:55:22.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:55:22 smithi181 ceph-mon[94189]: pgmap v6184: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 6 op/s 2023-03-09T17:55:22.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:55:22 smithi038 ceph-mon[92422]: pgmap v6184: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 6 op/s 2023-03-09T17:55:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:55:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10633.364919 secs 2023-03-09T17:55:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:55:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10633.364919 secs 2023-03-09T17:55:23.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:55:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10633.364919 secs 2023-03-09T17:55:23.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11095.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:55:24.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:55:24 smithi122 ceph-mon[93869]: pgmap v6185: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 7 op/s 2023-03-09T17:55:24.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:55:24 smithi181 ceph-mon[94189]: pgmap v6185: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 7 op/s 2023-03-09T17:55:24.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:55:24 smithi038 ceph-mon[92422]: pgmap v6185: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 7 op/s 2023-03-09T17:55:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:55:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10636.081667 secs 2023-03-09T17:55:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:55:26 smithi122 ceph-mon[93869]: pgmap v6186: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 11 KiB/s wr, 7 op/s 2023-03-09T17:55:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:55:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10636.081667 secs 2023-03-09T17:55:26.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:55:26 smithi181 ceph-mon[94189]: pgmap v6186: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 11 KiB/s wr, 7 op/s 2023-03-09T17:55:26.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:55:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10636.081667 secs 2023-03-09T17:55:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:55:26 smithi038 ceph-mon[92422]: pgmap v6186: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 11 KiB/s wr, 7 op/s 2023-03-09T17:55:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:55:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10638.365007 secs 2023-03-09T17:55:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:55:28 smithi122 ceph-mon[93869]: pgmap v6187: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 43 KiB/s wr, 7 op/s 2023-03-09T17:55:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:55:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10638.365007 secs 2023-03-09T17:55:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:55:28 smithi181 ceph-mon[94189]: pgmap v6187: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 43 KiB/s wr, 7 op/s 2023-03-09T17:55:28.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:55:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10638.365007 secs 2023-03-09T17:55:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:55:28 smithi038 ceph-mon[92422]: pgmap v6187: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 43 KiB/s wr, 7 op/s 2023-03-09T17:55:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11100.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:55:29.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:55:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:55:29.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:55:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:55:29.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:55:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:55:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:55:30 smithi122 ceph-mon[93869]: pgmap v6188: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 39 KiB/s wr, 7 op/s 2023-03-09T17:55:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:55:30 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:55:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:55:30 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:55:30.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:55:30 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:55:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:55:30 smithi181 ceph-mon[94189]: pgmap v6188: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 39 KiB/s wr, 7 op/s 2023-03-09T17:55:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:55:30 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:55:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:55:30 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:55:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:55:30 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:55:30.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:55:30 smithi038 ceph-mon[92422]: pgmap v6188: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 39 KiB/s wr, 7 op/s 2023-03-09T17:55:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:55:30 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:55:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:55:30 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:55:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:55:30 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:55:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:55:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10641.081747 secs 2023-03-09T17:55:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:55:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10641.081747 secs 2023-03-09T17:55:31.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:55:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10641.081747 secs 2023-03-09T17:55:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:55:32 smithi122 ceph-mon[93869]: pgmap v6189: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 30 KiB/s wr, 8 op/s 2023-03-09T17:55:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:55:32 smithi181 ceph-mon[94189]: pgmap v6189: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 30 KiB/s wr, 8 op/s 2023-03-09T17:55:32.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:55:32 smithi038 ceph-mon[92422]: pgmap v6189: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 30 KiB/s wr, 8 op/s 2023-03-09T17:55:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:55:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10643.365112 secs 2023-03-09T17:55:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:55:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10643.365112 secs 2023-03-09T17:55:33.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:55:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10643.365112 secs 2023-03-09T17:55:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11105.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:55:34.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:55:34 smithi122 ceph-mon[93869]: pgmap v6190: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 6 op/s 2023-03-09T17:55:34.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:55:34 smithi181 ceph-mon[94189]: pgmap v6190: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 6 op/s 2023-03-09T17:55:34.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:55:34 smithi038 ceph-mon[92422]: pgmap v6190: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 6 op/s 2023-03-09T17:55:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:55:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10646.081810 secs 2023-03-09T17:55:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:55:36 smithi122 ceph-mon[93869]: pgmap v6191: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 39 KiB/s wr, 6 op/s 2023-03-09T17:55:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:55:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10646.081810 secs 2023-03-09T17:55:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:55:36 smithi181 ceph-mon[94189]: pgmap v6191: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 39 KiB/s wr, 6 op/s 2023-03-09T17:55:36.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:55:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10646.081810 secs 2023-03-09T17:55:36.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:55:36 smithi038 ceph-mon[92422]: pgmap v6191: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 39 KiB/s wr, 6 op/s 2023-03-09T17:55:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:55:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10648.365230 secs 2023-03-09T17:55:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:55:38 smithi122 ceph-mon[93869]: pgmap v6192: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 104 KiB/s wr, 9 op/s 2023-03-09T17:55:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:55:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10648.365230 secs 2023-03-09T17:55:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:55:38 smithi181 ceph-mon[94189]: pgmap v6192: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 104 KiB/s wr, 9 op/s 2023-03-09T17:55:38.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:55:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10648.365230 secs 2023-03-09T17:55:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:55:38 smithi038 ceph-mon[92422]: pgmap v6192: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 104 KiB/s wr, 9 op/s 2023-03-09T17:55:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11110.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:55:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:55:40 smithi122 ceph-mon[93869]: pgmap v6193: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 79 KiB/s wr, 8 op/s 2023-03-09T17:55:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:55:40 smithi181 ceph-mon[94189]: pgmap v6193: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 79 KiB/s wr, 8 op/s 2023-03-09T17:55:40.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:55:40 smithi038 ceph-mon[92422]: pgmap v6193: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 79 KiB/s wr, 8 op/s 2023-03-09T17:55:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:55:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10651.081872 secs 2023-03-09T17:55:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:55:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10651.081872 secs 2023-03-09T17:55:41.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:55:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10651.081872 secs 2023-03-09T17:55:42.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:55:42 smithi122 ceph-mon[93869]: pgmap v6194: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 130 KiB/s wr, 11 op/s 2023-03-09T17:55:42.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:55:42 smithi181 ceph-mon[94189]: pgmap v6194: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 130 KiB/s wr, 11 op/s 2023-03-09T17:55:42.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:55:42 smithi038 ceph-mon[92422]: pgmap v6194: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 130 KiB/s wr, 11 op/s 2023-03-09T17:55:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:55:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10653.365347 secs 2023-03-09T17:55:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:55:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10653.365347 secs 2023-03-09T17:55:43.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:55:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10653.365347 secs 2023-03-09T17:55:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11115.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:55:44.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:55:44 smithi122 ceph-mon[93869]: pgmap v6195: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 135 KiB/s wr, 10 op/s 2023-03-09T17:55:44.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:55:44 smithi181 ceph-mon[94189]: pgmap v6195: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 135 KiB/s wr, 10 op/s 2023-03-09T17:55:44.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:55:44 smithi038 ceph-mon[92422]: pgmap v6195: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 135 KiB/s wr, 10 op/s 2023-03-09T17:55:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:55:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10656.081931 secs 2023-03-09T17:55:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:55:46 smithi122 ceph-mon[93869]: pgmap v6196: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 135 KiB/s wr, 10 op/s 2023-03-09T17:55:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:55:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10656.081931 secs 2023-03-09T17:55:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:55:46 smithi181 ceph-mon[94189]: pgmap v6196: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 135 KiB/s wr, 10 op/s 2023-03-09T17:55:46.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:55:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10656.081931 secs 2023-03-09T17:55:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:55:46 smithi038 ceph-mon[92422]: pgmap v6196: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 135 KiB/s wr, 10 op/s 2023-03-09T17:55:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:55:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10658.365472 secs 2023-03-09T17:55:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:55:48 smithi122 ceph-mon[93869]: pgmap v6197: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 191 KiB/s wr, 14 op/s 2023-03-09T17:55:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:55:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10658.365472 secs 2023-03-09T17:55:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:55:48 smithi181 ceph-mon[94189]: pgmap v6197: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 191 KiB/s wr, 14 op/s 2023-03-09T17:55:48.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:55:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10658.365472 secs 2023-03-09T17:55:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:55:48 smithi038 ceph-mon[92422]: pgmap v6197: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 191 KiB/s wr, 14 op/s 2023-03-09T17:55:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11120.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:55:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:55:50 smithi122 ceph-mon[93869]: pgmap v6198: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 126 KiB/s wr, 11 op/s 2023-03-09T17:55:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:55:50 smithi181 ceph-mon[94189]: pgmap v6198: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 126 KiB/s wr, 11 op/s 2023-03-09T17:55:50.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:55:50 smithi038 ceph-mon[92422]: pgmap v6198: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 126 KiB/s wr, 11 op/s 2023-03-09T17:55:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:55:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10661.081982 secs 2023-03-09T17:55:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:55:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10661.081982 secs 2023-03-09T17:55:51.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:55:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10661.081982 secs 2023-03-09T17:55:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:55:52 smithi122 ceph-mon[93869]: pgmap v6199: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 173 KiB/s wr, 13 op/s 2023-03-09T17:55:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:55:52 smithi181 ceph-mon[94189]: pgmap v6199: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 173 KiB/s wr, 13 op/s 2023-03-09T17:55:52.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:55:52 smithi038 ceph-mon[92422]: pgmap v6199: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 173 KiB/s wr, 13 op/s 2023-03-09T17:55:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:55:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10663.365589 secs 2023-03-09T17:55:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:55:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10663.365589 secs 2023-03-09T17:55:53.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:55:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10663.365589 secs 2023-03-09T17:55:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11125.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:55:54.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:55:54 smithi122 ceph-mon[93869]: pgmap v6200: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 122 KiB/s wr, 11 op/s 2023-03-09T17:55:54.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:55:54 smithi181 ceph-mon[94189]: pgmap v6200: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 122 KiB/s wr, 11 op/s 2023-03-09T17:55:54.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:55:54 smithi038 ceph-mon[92422]: pgmap v6200: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 122 KiB/s wr, 11 op/s 2023-03-09T17:55:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:55:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10666.082054 secs 2023-03-09T17:55:55.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:55:55 smithi122 ceph-mon[93869]: pgmap v6201: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 117 KiB/s wr, 10 op/s 2023-03-09T17:55:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:55:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10666.082054 secs 2023-03-09T17:55:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:55:55 smithi181 ceph-mon[94189]: pgmap v6201: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 117 KiB/s wr, 10 op/s 2023-03-09T17:55:55.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:55:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10666.082054 secs 2023-03-09T17:55:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:55:55 smithi038 ceph-mon[92422]: pgmap v6201: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 117 KiB/s wr, 10 op/s 2023-03-09T17:55:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:55:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10668.365722 secs 2023-03-09T17:55:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:55:58 smithi122 ceph-mon[93869]: pgmap v6202: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 219 KiB/s wr, 13 op/s 2023-03-09T17:55:58.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:55:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10668.365722 secs 2023-03-09T17:55:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:55:58 smithi181 ceph-mon[94189]: pgmap v6202: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 219 KiB/s wr, 13 op/s 2023-03-09T17:55:58.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:55:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10668.365722 secs 2023-03-09T17:55:58.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:55:58 smithi038 ceph-mon[92422]: pgmap v6202: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 219 KiB/s wr, 13 op/s 2023-03-09T17:55:58.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11130.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:56:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:56:00 smithi122 ceph-mon[93869]: pgmap v6203: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 155 KiB/s wr, 9 op/s 2023-03-09T17:56:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:56:00 smithi181 ceph-mon[94189]: pgmap v6203: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 155 KiB/s wr, 9 op/s 2023-03-09T17:56:00.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:56:00 smithi038 ceph-mon[92422]: pgmap v6203: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 155 KiB/s wr, 9 op/s 2023-03-09T17:56:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:56:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10671.082114 secs 2023-03-09T17:56:01.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:56:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10671.082114 secs 2023-03-09T17:56:01.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:56:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10671.082114 secs 2023-03-09T17:56:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:56:02 smithi122 ceph-mon[93869]: pgmap v6204: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 203 KiB/s wr, 11 op/s 2023-03-09T17:56:02.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:56:02 smithi181 ceph-mon[94189]: pgmap v6204: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 203 KiB/s wr, 11 op/s 2023-03-09T17:56:02.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:56:02 smithi038 ceph-mon[92422]: pgmap v6204: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 203 KiB/s wr, 11 op/s 2023-03-09T17:56:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:56:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10673.365861 secs 2023-03-09T17:56:03.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:56:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10673.365861 secs 2023-03-09T17:56:03.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:56:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10673.365861 secs 2023-03-09T17:56:03.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11135.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:56:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:56:04 smithi122 ceph-mon[93869]: pgmap v6205: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 173 KiB/s wr, 21 op/s 2023-03-09T17:56:04.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:56:04 smithi181 ceph-mon[94189]: pgmap v6205: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 173 KiB/s wr, 21 op/s 2023-03-09T17:56:04.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:56:04 smithi038 ceph-mon[92422]: pgmap v6205: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 173 KiB/s wr, 21 op/s 2023-03-09T17:56:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:56:05 smithi122 ceph-mon[93869]: osdmap e601: 12 total, 12 up, 12 in 2023-03-09T17:56:05.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:56:05 smithi181 ceph-mon[94189]: osdmap e601: 12 total, 12 up, 12 in 2023-03-09T17:56:05.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:56:05 smithi038 ceph-mon[92422]: osdmap e601: 12 total, 12 up, 12 in 2023-03-09T17:56:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:56:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10676.082178 secs 2023-03-09T17:56:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:56:06 smithi122 ceph-mon[93869]: pgmap v6207: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 207 KiB/s wr, 24 op/s 2023-03-09T17:56:06.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:56:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10676.082178 secs 2023-03-09T17:56:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:56:06 smithi181 ceph-mon[94189]: pgmap v6207: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 207 KiB/s wr, 24 op/s 2023-03-09T17:56:06.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:56:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10676.082178 secs 2023-03-09T17:56:06.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:56:06 smithi038 ceph-mon[92422]: pgmap v6207: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 207 KiB/s wr, 24 op/s 2023-03-09T17:56:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:56:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10678.365985 secs 2023-03-09T17:56:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:56:08 smithi122 ceph-mon[93869]: pgmap v6208: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 171 KiB/s wr, 26 op/s 2023-03-09T17:56:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:56:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10678.365985 secs 2023-03-09T17:56:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:56:08 smithi181 ceph-mon[94189]: pgmap v6208: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 171 KiB/s wr, 26 op/s 2023-03-09T17:56:08.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:56:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10678.365985 secs 2023-03-09T17:56:08.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:56:08 smithi038 ceph-mon[92422]: pgmap v6208: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 171 KiB/s wr, 26 op/s 2023-03-09T17:56:08.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11140.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:56:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:56:10 smithi122 ceph-mon[93869]: pgmap v6209: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 171 KiB/s wr, 26 op/s 2023-03-09T17:56:10.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:56:10 smithi181 ceph-mon[94189]: pgmap v6209: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 171 KiB/s wr, 26 op/s 2023-03-09T17:56:10.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:56:10 smithi038 ceph-mon[92422]: pgmap v6209: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 171 KiB/s wr, 26 op/s 2023-03-09T17:56:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:56:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10681.082238 secs 2023-03-09T17:56:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:56:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10681.082238 secs 2023-03-09T17:56:11.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:56:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10681.082238 secs 2023-03-09T17:56:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:56:12 smithi122 ceph-mon[93869]: pgmap v6210: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 137 KiB/s wr, 27 op/s 2023-03-09T17:56:12.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:56:12 smithi181 ceph-mon[94189]: pgmap v6210: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 137 KiB/s wr, 27 op/s 2023-03-09T17:56:12.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:56:12 smithi038 ceph-mon[92422]: pgmap v6210: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 137 KiB/s wr, 27 op/s 2023-03-09T17:56:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:56:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10683.366113 secs 2023-03-09T17:56:13.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:56:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10683.366113 secs 2023-03-09T17:56:13.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:56:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10683.366113 secs 2023-03-09T17:56:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11145.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:56:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:56:14 smithi122 ceph-mon[93869]: pgmap v6211: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 121 KiB/s wr, 12 op/s 2023-03-09T17:56:14.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:56:14 smithi181 ceph-mon[94189]: pgmap v6211: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 121 KiB/s wr, 12 op/s 2023-03-09T17:56:14.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:56:14 smithi038 ceph-mon[92422]: pgmap v6211: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 121 KiB/s wr, 12 op/s 2023-03-09T17:56:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:56:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10686.082299 secs 2023-03-09T17:56:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:56:16 smithi122 ceph-mon[93869]: pgmap v6212: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 113 KiB/s wr, 11 op/s 2023-03-09T17:56:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:56:16 smithi122 ceph-mon[93869]: osdmap e602: 12 total, 12 up, 12 in 2023-03-09T17:56:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:56:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10686.082299 secs 2023-03-09T17:56:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:56:16 smithi181 ceph-mon[94189]: pgmap v6212: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 113 KiB/s wr, 11 op/s 2023-03-09T17:56:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:56:16 smithi181 ceph-mon[94189]: osdmap e602: 12 total, 12 up, 12 in 2023-03-09T17:56:16.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:56:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10686.082299 secs 2023-03-09T17:56:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:56:16 smithi038 ceph-mon[92422]: pgmap v6212: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 113 KiB/s wr, 11 op/s 2023-03-09T17:56:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:56:16 smithi038 ceph-mon[92422]: osdmap e602: 12 total, 12 up, 12 in 2023-03-09T17:56:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:56:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10688.366220 secs 2023-03-09T17:56:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:56:18 smithi181 ceph-mon[94189]: pgmap v6214: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 36 KiB/s wr, 8 op/s 2023-03-09T17:56:18.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:56:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10688.366220 secs 2023-03-09T17:56:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:56:18 smithi038 ceph-mon[92422]: pgmap v6214: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 36 KiB/s wr, 8 op/s 2023-03-09T17:56:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:56:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10688.366220 secs 2023-03-09T17:56:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:56:18 smithi122 ceph-mon[93869]: pgmap v6214: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 36 KiB/s wr, 8 op/s 2023-03-09T17:56:18.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11150.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:56:20.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:56:20 smithi038 ceph-mon[92422]: pgmap v6215: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 36 KiB/s wr, 8 op/s 2023-03-09T17:56:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:56:20 smithi122 ceph-mon[93869]: pgmap v6215: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 36 KiB/s wr, 8 op/s 2023-03-09T17:56:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:56:20 smithi181 ceph-mon[94189]: pgmap v6215: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 36 KiB/s wr, 8 op/s 2023-03-09T17:56:21.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:56:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10691.082357 secs 2023-03-09T17:56:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:56:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10691.082357 secs 2023-03-09T17:56:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:56:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10691.082357 secs 2023-03-09T17:56:22.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:56:22 smithi038 ceph-mon[92422]: pgmap v6216: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 68 KiB/s wr, 8 op/s 2023-03-09T17:56:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:56:22 smithi122 ceph-mon[93869]: pgmap v6216: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 68 KiB/s wr, 8 op/s 2023-03-09T17:56:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:56:22 smithi181 ceph-mon[94189]: pgmap v6216: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 68 KiB/s wr, 8 op/s 2023-03-09T17:56:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:56:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10693.366325 secs 2023-03-09T17:56:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:56:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10693.366325 secs 2023-03-09T17:56:23.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:56:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10693.366325 secs 2023-03-09T17:56:23.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11155.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:56:24.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:56:24 smithi122 ceph-mon[93869]: pgmap v6217: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 57 KiB/s wr, 8 op/s 2023-03-09T17:56:24.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:56:24 smithi181 ceph-mon[94189]: pgmap v6217: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 57 KiB/s wr, 8 op/s 2023-03-09T17:56:24.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:56:24 smithi038 ceph-mon[92422]: pgmap v6217: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 57 KiB/s wr, 8 op/s 2023-03-09T17:56:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:56:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10696.082416 secs 2023-03-09T17:56:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:56:26 smithi122 ceph-mon[93869]: pgmap v6218: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 57 KiB/s wr, 8 op/s 2023-03-09T17:56:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:56:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10696.082416 secs 2023-03-09T17:56:26.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:56:26 smithi181 ceph-mon[94189]: pgmap v6218: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 57 KiB/s wr, 8 op/s 2023-03-09T17:56:26.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:56:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10696.082416 secs 2023-03-09T17:56:26.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:56:26 smithi038 ceph-mon[92422]: pgmap v6218: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 57 KiB/s wr, 8 op/s 2023-03-09T17:56:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:56:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10698.366422 secs 2023-03-09T17:56:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:56:28 smithi122 ceph-mon[93869]: pgmap v6219: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 8 op/s 2023-03-09T17:56:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:56:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10698.366422 secs 2023-03-09T17:56:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:56:28 smithi181 ceph-mon[94189]: pgmap v6219: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 8 op/s 2023-03-09T17:56:28.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:56:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10698.366422 secs 2023-03-09T17:56:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:56:28 smithi038 ceph-mon[92422]: pgmap v6219: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 8 op/s 2023-03-09T17:56:28.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11160.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:56:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:56:30 smithi122 ceph-mon[93869]: pgmap v6220: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 47 KiB/s wr, 6 op/s 2023-03-09T17:56:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:56:30 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:56:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:56:30 smithi181 ceph-mon[94189]: pgmap v6220: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 47 KiB/s wr, 6 op/s 2023-03-09T17:56:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:56:30 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:56:30.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:56:30 smithi038 ceph-mon[92422]: pgmap v6220: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 47 KiB/s wr, 6 op/s 2023-03-09T17:56:30.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:56:30 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:56:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:56:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10701.082476 secs 2023-03-09T17:56:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:56:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:56:31.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:56:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:56:31.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:56:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:56:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:56:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10701.082476 secs 2023-03-09T17:56:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:56:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:56:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:56:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:56:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:56:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:56:31.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:56:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10701.082476 secs 2023-03-09T17:56:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:56:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:56:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:56:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:56:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:56:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:56:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:56:32 smithi122 ceph-mon[93869]: pgmap v6221: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 104 KiB/s wr, 9 op/s 2023-03-09T17:56:32.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:56:32 smithi181 ceph-mon[94189]: pgmap v6221: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 104 KiB/s wr, 9 op/s 2023-03-09T17:56:32.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:56:32 smithi038 ceph-mon[92422]: pgmap v6221: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 104 KiB/s wr, 9 op/s 2023-03-09T17:56:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11165.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:56:34.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:56:34 smithi122 ceph-mon[93869]: pgmap v6222: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 58 KiB/s wr, 7 op/s 2023-03-09T17:56:34.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:56:34 smithi181 ceph-mon[94189]: pgmap v6222: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 58 KiB/s wr, 7 op/s 2023-03-09T17:56:34.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:56:34 smithi038 ceph-mon[92422]: pgmap v6222: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 58 KiB/s wr, 7 op/s 2023-03-09T17:56:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:56:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10706.082538 secs 2023-03-09T17:56:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:56:36 smithi122 ceph-mon[93869]: pgmap v6223: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 58 KiB/s wr, 6 op/s 2023-03-09T17:56:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:56:36 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:56:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:56:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10706.082538 secs 2023-03-09T17:56:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:56:36 smithi181 ceph-mon[94189]: pgmap v6223: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 58 KiB/s wr, 6 op/s 2023-03-09T17:56:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:56:36 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:56:36.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:56:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10706.082538 secs 2023-03-09T17:56:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:56:36 smithi038 ceph-mon[92422]: pgmap v6223: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 58 KiB/s wr, 6 op/s 2023-03-09T17:56:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:56:36 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:56:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:56:37 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:56:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:56:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:56:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:56:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:56:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:56:37 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:56:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:56:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:56:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:56:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:56:37.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:56:37 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:56:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:56:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:56:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:56:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:56:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:56:38 smithi122 ceph-mon[93869]: pgmap v6224: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 87 KiB/s wr, 9 op/s 2023-03-09T17:56:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:56:38 smithi181 ceph-mon[94189]: pgmap v6224: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 87 KiB/s wr, 9 op/s 2023-03-09T17:56:38.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:56:38 smithi038 ceph-mon[92422]: pgmap v6224: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 87 KiB/s wr, 9 op/s 2023-03-09T17:56:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11170.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:56:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:56:40 smithi122 ceph-mon[93869]: pgmap v6225: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 86 KiB/s wr, 6 op/s 2023-03-09T17:56:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:56:40 smithi181 ceph-mon[94189]: pgmap v6225: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 86 KiB/s wr, 6 op/s 2023-03-09T17:56:40.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:56:40 smithi038 ceph-mon[92422]: pgmap v6225: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 86 KiB/s wr, 6 op/s 2023-03-09T17:56:41.734 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:56:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10711.082596 secs 2023-03-09T17:56:41.736 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:56:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10711.082596 secs 2023-03-09T17:56:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:56:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10711.082596 secs 2023-03-09T17:56:42.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:56:42 smithi122 ceph-mon[93869]: pgmap v6226: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 138 KiB/s wr, 9 op/s 2023-03-09T17:56:42.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:56:42 smithi181 ceph-mon[94189]: pgmap v6226: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 138 KiB/s wr, 9 op/s 2023-03-09T17:56:42.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:56:42 smithi038 ceph-mon[92422]: pgmap v6226: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 138 KiB/s wr, 9 op/s 2023-03-09T17:56:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11175.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:56:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:56:44 smithi122 ceph-mon[93869]: pgmap v6227: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 91 KiB/s wr, 8 op/s 2023-03-09T17:56:44.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:56:44 smithi181 ceph-mon[94189]: pgmap v6227: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 91 KiB/s wr, 8 op/s 2023-03-09T17:56:44.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:56:44 smithi038 ceph-mon[92422]: pgmap v6227: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 91 KiB/s wr, 8 op/s 2023-03-09T17:56:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:56:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10716.082676 secs 2023-03-09T17:56:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:56:46 smithi122 ceph-mon[93869]: pgmap v6228: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 91 KiB/s wr, 7 op/s 2023-03-09T17:56:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:56:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10716.082676 secs 2023-03-09T17:56:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:56:46 smithi181 ceph-mon[94189]: pgmap v6228: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 91 KiB/s wr, 7 op/s 2023-03-09T17:56:46.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:56:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10716.082676 secs 2023-03-09T17:56:46.835 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:56:46 smithi038 ceph-mon[92422]: pgmap v6228: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 91 KiB/s wr, 7 op/s 2023-03-09T17:56:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:56:48 smithi122 ceph-mon[93869]: pgmap v6229: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 146 KiB/s wr, 11 op/s 2023-03-09T17:56:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:56:48 smithi181 ceph-mon[94189]: pgmap v6229: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 146 KiB/s wr, 11 op/s 2023-03-09T17:56:48.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:56:48 smithi038 ceph-mon[92422]: pgmap v6229: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 146 KiB/s wr, 11 op/s 2023-03-09T17:56:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11180.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:56:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:56:50 smithi122 ceph-mon[93869]: pgmap v6230: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 117 KiB/s wr, 8 op/s 2023-03-09T17:56:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:56:50 smithi181 ceph-mon[94189]: pgmap v6230: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 117 KiB/s wr, 8 op/s 2023-03-09T17:56:50.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:56:50 smithi038 ceph-mon[92422]: pgmap v6230: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 117 KiB/s wr, 8 op/s 2023-03-09T17:56:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:56:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10721.082735 secs 2023-03-09T17:56:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:56:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10721.082735 secs 2023-03-09T17:56:51.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:56:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10721.082735 secs 2023-03-09T17:56:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:56:52 smithi122 ceph-mon[93869]: pgmap v6231: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 193 KiB/s wr, 11 op/s 2023-03-09T17:56:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:56:52 smithi181 ceph-mon[94189]: pgmap v6231: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 193 KiB/s wr, 11 op/s 2023-03-09T17:56:52.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:56:52 smithi038 ceph-mon[92422]: pgmap v6231: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 193 KiB/s wr, 11 op/s 2023-03-09T17:56:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11185.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:56:54.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:56:54 smithi122 ceph-mon[93869]: pgmap v6232: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 149 KiB/s wr, 10 op/s 2023-03-09T17:56:54.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:56:54 smithi181 ceph-mon[94189]: pgmap v6232: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 149 KiB/s wr, 10 op/s 2023-03-09T17:56:54.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:56:54 smithi038 ceph-mon[92422]: pgmap v6232: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 149 KiB/s wr, 10 op/s 2023-03-09T17:56:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:56:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10726.082795 secs 2023-03-09T17:56:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:56:56 smithi122 ceph-mon[93869]: pgmap v6233: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 141 KiB/s wr, 9 op/s 2023-03-09T17:56:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:56:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10726.082795 secs 2023-03-09T17:56:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:56:56 smithi181 ceph-mon[94189]: pgmap v6233: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 141 KiB/s wr, 9 op/s 2023-03-09T17:56:56.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:56:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10726.082795 secs 2023-03-09T17:56:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:56:56 smithi038 ceph-mon[92422]: pgmap v6233: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 141 KiB/s wr, 9 op/s 2023-03-09T17:56:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:56:58 smithi122 ceph-mon[93869]: pgmap v6234: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 176 KiB/s wr, 13 op/s 2023-03-09T17:56:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:56:58 smithi181 ceph-mon[94189]: pgmap v6234: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 176 KiB/s wr, 13 op/s 2023-03-09T17:56:58.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:56:58 smithi038 ceph-mon[92422]: pgmap v6234: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 176 KiB/s wr, 13 op/s 2023-03-09T17:56:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11190.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:57:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:57:00 smithi122 ceph-mon[93869]: pgmap v6235: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 120 KiB/s wr, 9 op/s 2023-03-09T17:57:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:57:00 smithi181 ceph-mon[94189]: pgmap v6235: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 120 KiB/s wr, 9 op/s 2023-03-09T17:57:00.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:57:00 smithi038 ceph-mon[92422]: pgmap v6235: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 120 KiB/s wr, 9 op/s 2023-03-09T17:57:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:57:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10731.082880 secs 2023-03-09T17:57:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:57:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10731.082880 secs 2023-03-09T17:57:01.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:57:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10731.082880 secs 2023-03-09T17:57:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:57:02 smithi122 ceph-mon[93869]: pgmap v6236: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 120 KiB/s wr, 11 op/s 2023-03-09T17:57:02.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:57:02 smithi181 ceph-mon[94189]: pgmap v6236: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 120 KiB/s wr, 11 op/s 2023-03-09T17:57:02.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:57:02 smithi038 ceph-mon[92422]: pgmap v6236: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 120 KiB/s wr, 11 op/s 2023-03-09T17:57:03.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:57:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10733.367269 secs 2023-03-09T17:57:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:57:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10733.367269 secs 2023-03-09T17:57:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:57:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10733.367269 secs 2023-03-09T17:57:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11195.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:57:04.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:57:04 smithi122 ceph-mon[93869]: pgmap v6237: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 123 KiB/s wr, 22 op/s 2023-03-09T17:57:04.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:57:04 smithi181 ceph-mon[94189]: pgmap v6237: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 123 KiB/s wr, 22 op/s 2023-03-09T17:57:04.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:57:04 smithi038 ceph-mon[92422]: pgmap v6237: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 123 KiB/s wr, 22 op/s 2023-03-09T17:57:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:57:05 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:57:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:57:05 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:57:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:57:05 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:57:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:57:05 smithi122 ceph-mon[93869]: osdmap e603: 12 total, 12 up, 12 in 2023-03-09T17:57:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:57:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10736.082958 secs 2023-03-09T17:57:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:57:05 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:57:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:57:05 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:57:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:57:05 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:57:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:57:05 smithi181 ceph-mon[94189]: osdmap e603: 12 total, 12 up, 12 in 2023-03-09T17:57:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:57:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10736.082958 secs 2023-03-09T17:57:05.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:57:05 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:57:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:57:05 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:57:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:57:05 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:57:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:57:05 smithi038 ceph-mon[92422]: osdmap e603: 12 total, 12 up, 12 in 2023-03-09T17:57:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:57:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10736.082958 secs 2023-03-09T17:57:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:57:06 smithi122 ceph-mon[93869]: pgmap v6239: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 136 KiB/s wr, 25 op/s 2023-03-09T17:57:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:57:06 smithi181 ceph-mon[94189]: pgmap v6239: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 136 KiB/s wr, 25 op/s 2023-03-09T17:57:06.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:57:06 smithi038 ceph-mon[92422]: pgmap v6239: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 136 KiB/s wr, 25 op/s 2023-03-09T17:57:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:57:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10738.367364 secs 2023-03-09T17:57:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:57:08 smithi122 ceph-mon[93869]: pgmap v6240: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 214 KiB/s wr, 29 op/s 2023-03-09T17:57:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:57:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10738.367364 secs 2023-03-09T17:57:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:57:08 smithi181 ceph-mon[94189]: pgmap v6240: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 214 KiB/s wr, 29 op/s 2023-03-09T17:57:08.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:57:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10738.367364 secs 2023-03-09T17:57:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:57:08 smithi038 ceph-mon[92422]: pgmap v6240: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 214 KiB/s wr, 29 op/s 2023-03-09T17:57:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11200.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:57:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:57:10 smithi122 ceph-mon[93869]: pgmap v6241: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 214 KiB/s wr, 29 op/s 2023-03-09T17:57:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:57:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10741.083044 secs 2023-03-09T17:57:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:57:10 smithi181 ceph-mon[94189]: pgmap v6241: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 214 KiB/s wr, 29 op/s 2023-03-09T17:57:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:57:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10741.083044 secs 2023-03-09T17:57:10.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:57:10 smithi038 ceph-mon[92422]: pgmap v6241: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 214 KiB/s wr, 29 op/s 2023-03-09T17:57:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:57:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10741.083044 secs 2023-03-09T17:57:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:57:12 smithi122 ceph-mon[93869]: pgmap v6242: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 215 KiB/s wr, 28 op/s 2023-03-09T17:57:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:57:12 smithi181 ceph-mon[94189]: pgmap v6242: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 215 KiB/s wr, 28 op/s 2023-03-09T17:57:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:57:12 smithi038 ceph-mon[92422]: pgmap v6242: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 215 KiB/s wr, 28 op/s 2023-03-09T17:57:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:57:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10743.367475 secs 2023-03-09T17:57:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:57:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10743.367475 secs 2023-03-09T17:57:13.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:57:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10743.367475 secs 2023-03-09T17:57:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11205.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:57:14.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:57:14 smithi122 ceph-mon[93869]: pgmap v6243: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 174 KiB/s wr, 13 op/s 2023-03-09T17:57:14.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:57:14 smithi181 ceph-mon[94189]: pgmap v6243: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 174 KiB/s wr, 13 op/s 2023-03-09T17:57:14.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:57:14 smithi038 ceph-mon[92422]: pgmap v6243: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 174 KiB/s wr, 13 op/s 2023-03-09T17:57:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:57:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10746.083103 secs 2023-03-09T17:57:15.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:57:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10746.083103 secs 2023-03-09T17:57:15.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:57:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10746.083103 secs 2023-03-09T17:57:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:57:16 smithi122 ceph-mon[93869]: pgmap v6244: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 165 KiB/s wr, 12 op/s 2023-03-09T17:57:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:57:16 smithi122 ceph-mon[93869]: osdmap e604: 12 total, 12 up, 12 in 2023-03-09T17:57:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:57:16 smithi181 ceph-mon[94189]: pgmap v6244: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 165 KiB/s wr, 12 op/s 2023-03-09T17:57:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:57:16 smithi181 ceph-mon[94189]: osdmap e604: 12 total, 12 up, 12 in 2023-03-09T17:57:16.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:57:16 smithi038 ceph-mon[92422]: pgmap v6244: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 165 KiB/s wr, 12 op/s 2023-03-09T17:57:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:57:16 smithi038 ceph-mon[92422]: osdmap e604: 12 total, 12 up, 12 in 2023-03-09T17:57:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:57:17 smithi122 ceph-mon[93869]: pgmap v6246: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 65 KiB/s wr, 8 op/s 2023-03-09T17:57:17.785 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:57:17 smithi038 ceph-mon[92422]: pgmap v6246: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 65 KiB/s wr, 8 op/s 2023-03-09T17:57:17.785 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:57:17 smithi181 ceph-mon[94189]: pgmap v6246: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 65 KiB/s wr, 8 op/s 2023-03-09T17:57:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:57:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10748.367596 secs 2023-03-09T17:57:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:57:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10748.367596 secs 2023-03-09T17:57:18.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:57:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10748.367596 secs 2023-03-09T17:57:18.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11210.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:57:19.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:57:19 smithi038 ceph-mon[92422]: pgmap v6247: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 65 KiB/s wr, 8 op/s 2023-03-09T17:57:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:57:19 smithi122 ceph-mon[93869]: pgmap v6247: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 65 KiB/s wr, 8 op/s 2023-03-09T17:57:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:57:19 smithi181 ceph-mon[94189]: pgmap v6247: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 65 KiB/s wr, 8 op/s 2023-03-09T17:57:20.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:57:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10751.083159 secs 2023-03-09T17:57:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:57:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10751.083159 secs 2023-03-09T17:57:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:57:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10751.083159 secs 2023-03-09T17:57:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:57:22 smithi122 ceph-mon[93869]: pgmap v6248: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 65 KiB/s wr, 7 op/s 2023-03-09T17:57:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:57:22 smithi181 ceph-mon[94189]: pgmap v6248: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 65 KiB/s wr, 7 op/s 2023-03-09T17:57:22.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:57:22 smithi038 ceph-mon[92422]: pgmap v6248: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 65 KiB/s wr, 7 op/s 2023-03-09T17:57:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:57:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10753.367733 secs 2023-03-09T17:57:23.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:57:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10753.367733 secs 2023-03-09T17:57:23.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:57:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10753.367733 secs 2023-03-09T17:57:23.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11215.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:57:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:57:24 smithi122 ceph-mon[93869]: pgmap v6249: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T17:57:24.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:57:24 smithi181 ceph-mon[94189]: pgmap v6249: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T17:57:24.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:57:24 smithi038 ceph-mon[92422]: pgmap v6249: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T17:57:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:57:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10756.083217 secs 2023-03-09T17:57:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:57:26 smithi122 ceph-mon[93869]: pgmap v6250: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T17:57:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:57:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10756.083217 secs 2023-03-09T17:57:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:57:26 smithi181 ceph-mon[94189]: pgmap v6250: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T17:57:26.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:57:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10756.083217 secs 2023-03-09T17:57:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:57:26 smithi038 ceph-mon[92422]: pgmap v6250: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T17:57:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:57:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10758.367851 secs 2023-03-09T17:57:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:57:28 smithi122 ceph-mon[93869]: pgmap v6251: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 30 KiB/s wr, 8 op/s 2023-03-09T17:57:28.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:57:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10758.367851 secs 2023-03-09T17:57:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:57:28 smithi181 ceph-mon[94189]: pgmap v6251: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 30 KiB/s wr, 8 op/s 2023-03-09T17:57:28.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:57:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10758.367851 secs 2023-03-09T17:57:28.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:57:28 smithi038 ceph-mon[92422]: pgmap v6251: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 30 KiB/s wr, 8 op/s 2023-03-09T17:57:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11220.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:57:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:57:30 smithi122 ceph-mon[93869]: pgmap v6252: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 18 KiB/s wr, 6 op/s 2023-03-09T17:57:30.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:57:30 smithi181 ceph-mon[94189]: pgmap v6252: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 18 KiB/s wr, 6 op/s 2023-03-09T17:57:30.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:57:30 smithi038 ceph-mon[92422]: pgmap v6252: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 18 KiB/s wr, 6 op/s 2023-03-09T17:57:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:57:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10761.083278 secs 2023-03-09T17:57:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:57:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:57:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:57:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:57:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:57:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:57:31.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:57:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:57:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:57:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10761.083278 secs 2023-03-09T17:57:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:57:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:57:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:57:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:57:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:57:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:57:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:57:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:57:31.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:57:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10761.083278 secs 2023-03-09T17:57:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:57:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:57:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:57:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:57:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:57:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:57:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:57:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:57:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:57:32 smithi122 ceph-mon[93869]: pgmap v6253: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 18 KiB/s wr, 8 op/s 2023-03-09T17:57:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:57:32 smithi181 ceph-mon[94189]: pgmap v6253: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 18 KiB/s wr, 8 op/s 2023-03-09T17:57:32.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:57:32 smithi038 ceph-mon[92422]: pgmap v6253: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 18 KiB/s wr, 8 op/s 2023-03-09T17:57:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:57:33 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 10763.367974 secs 2023-03-09T17:57:33.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:57:33 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 10763.367974 secs 2023-03-09T17:57:33.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:57:33 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 10763.367974 secs 2023-03-09T17:57:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11225.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:57:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:57:34 smithi122 ceph-mon[93869]: pgmap v6254: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 48 KiB/s wr, 7 op/s 2023-03-09T17:57:34.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:57:34 smithi181 ceph-mon[94189]: pgmap v6254: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 48 KiB/s wr, 7 op/s 2023-03-09T17:57:34.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:57:34 smithi038 ceph-mon[92422]: pgmap v6254: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 48 KiB/s wr, 7 op/s 2023-03-09T17:57:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:57:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10766.083338 secs 2023-03-09T17:57:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:57:36 smithi122 ceph-mon[93869]: pgmap v6255: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 31 KiB/s wr, 6 op/s 2023-03-09T17:57:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:57:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10766.083338 secs 2023-03-09T17:57:36.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:57:36 smithi181 ceph-mon[94189]: pgmap v6255: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 31 KiB/s wr, 6 op/s 2023-03-09T17:57:36.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:57:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10766.083338 secs 2023-03-09T17:57:36.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:57:36 smithi038 ceph-mon[92422]: pgmap v6255: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 31 KiB/s wr, 6 op/s 2023-03-09T17:57:37.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:57:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:57:37.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:57:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:57:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:57:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:57:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:57:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:57:37.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:57:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:57:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:57:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:57:38.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:57:38 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 10768.368088 secs 2023-03-09T17:57:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:57:38 smithi038 ceph-mon[92422]: pgmap v6256: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 60 KiB/s wr, 9 op/s 2023-03-09T17:57:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:57:38 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 10768.368088 secs 2023-03-09T17:57:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:57:38 smithi122 ceph-mon[93869]: pgmap v6256: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 60 KiB/s wr, 9 op/s 2023-03-09T17:57:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:57:38 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 10768.368088 secs 2023-03-09T17:57:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:57:38 smithi181 ceph-mon[94189]: pgmap v6256: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 60 KiB/s wr, 9 op/s 2023-03-09T17:57:38.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11230.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:57:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:57:40 smithi038 ceph-mon[92422]: pgmap v6257: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 60 KiB/s wr, 6 op/s 2023-03-09T17:57:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:57:40 smithi122 ceph-mon[93869]: pgmap v6257: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 60 KiB/s wr, 6 op/s 2023-03-09T17:57:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:57:40 smithi181 ceph-mon[94189]: pgmap v6257: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 60 KiB/s wr, 6 op/s 2023-03-09T17:57:41.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:57:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10771.083397 secs 2023-03-09T17:57:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:57:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10771.083397 secs 2023-03-09T17:57:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:57:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10771.083397 secs 2023-03-09T17:57:42.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:57:42 smithi122 ceph-mon[93869]: pgmap v6258: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 61 KiB/s wr, 9 op/s 2023-03-09T17:57:42.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:57:42 smithi181 ceph-mon[94189]: pgmap v6258: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 61 KiB/s wr, 9 op/s 2023-03-09T17:57:42.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:57:42 smithi038 ceph-mon[92422]: pgmap v6258: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 61 KiB/s wr, 9 op/s 2023-03-09T17:57:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:57:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10773.368175 secs 2023-03-09T17:57:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:57:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10773.368175 secs 2023-03-09T17:57:43.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:57:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10773.368175 secs 2023-03-09T17:57:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11235.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:57:44.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:57:44 smithi122 ceph-mon[93869]: pgmap v6259: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 168 KiB/s wr, 9 op/s 2023-03-09T17:57:44.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:57:44 smithi181 ceph-mon[94189]: pgmap v6259: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 168 KiB/s wr, 9 op/s 2023-03-09T17:57:44.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:57:44 smithi038 ceph-mon[92422]: pgmap v6259: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 168 KiB/s wr, 9 op/s 2023-03-09T17:57:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:57:45 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:57:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:57:45 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:57:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:57:45 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:57:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:57:45 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:57:45.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:57:45 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:57:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:57:45 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:57:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:57:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10776.083456 secs 2023-03-09T17:57:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:57:46 smithi122 ceph-mon[93869]: pgmap v6260: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 137 KiB/s wr, 8 op/s 2023-03-09T17:57:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:57:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10776.083456 secs 2023-03-09T17:57:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:57:46 smithi181 ceph-mon[94189]: pgmap v6260: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 137 KiB/s wr, 8 op/s 2023-03-09T17:57:46.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:57:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10776.083456 secs 2023-03-09T17:57:46.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:57:46 smithi038 ceph-mon[92422]: pgmap v6260: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 137 KiB/s wr, 8 op/s 2023-03-09T17:57:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:57:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10778.368299 secs 2023-03-09T17:57:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:57:48 smithi122 ceph-mon[93869]: pgmap v6261: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 156 KiB/s wr, 12 op/s 2023-03-09T17:57:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:57:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10778.368299 secs 2023-03-09T17:57:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:57:48 smithi181 ceph-mon[94189]: pgmap v6261: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 156 KiB/s wr, 12 op/s 2023-03-09T17:57:48.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:57:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10778.368299 secs 2023-03-09T17:57:48.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:57:48 smithi038 ceph-mon[92422]: pgmap v6261: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 156 KiB/s wr, 12 op/s 2023-03-09T17:57:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11240.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:57:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:57:50 smithi122 ceph-mon[93869]: pgmap v6262: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 127 KiB/s wr, 9 op/s 2023-03-09T17:57:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:57:50 smithi181 ceph-mon[94189]: pgmap v6262: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 127 KiB/s wr, 9 op/s 2023-03-09T17:57:50.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:57:50 smithi038 ceph-mon[92422]: pgmap v6262: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 127 KiB/s wr, 9 op/s 2023-03-09T17:57:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:57:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10781.083515 secs 2023-03-09T17:57:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:57:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10781.083515 secs 2023-03-09T17:57:51.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:57:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10781.083515 secs 2023-03-09T17:57:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:57:52 smithi122 ceph-mon[93869]: pgmap v6263: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 263 KiB/s wr, 12 op/s 2023-03-09T17:57:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:57:52 smithi181 ceph-mon[94189]: pgmap v6263: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 263 KiB/s wr, 12 op/s 2023-03-09T17:57:52.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:57:52 smithi038 ceph-mon[92422]: pgmap v6263: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 263 KiB/s wr, 12 op/s 2023-03-09T17:57:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:57:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10783.368410 secs 2023-03-09T17:57:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:57:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10783.368410 secs 2023-03-09T17:57:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:57:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10783.368410 secs 2023-03-09T17:57:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11245.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:57:54.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:57:54 smithi122 ceph-mon[93869]: pgmap v6264: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 272 KiB/s wr, 11 op/s 2023-03-09T17:57:54.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:57:54 smithi181 ceph-mon[94189]: pgmap v6264: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 272 KiB/s wr, 11 op/s 2023-03-09T17:57:54.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:57:54 smithi038 ceph-mon[92422]: pgmap v6264: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 272 KiB/s wr, 11 op/s 2023-03-09T17:57:56.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:57:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10786.083575 secs 2023-03-09T17:57:56.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:57:56 smithi122 ceph-mon[93869]: pgmap v6265: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 165 KiB/s wr, 8 op/s 2023-03-09T17:57:56.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:57:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10786.083575 secs 2023-03-09T17:57:56.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:57:56 smithi181 ceph-mon[94189]: pgmap v6265: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 165 KiB/s wr, 8 op/s 2023-03-09T17:57:56.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:57:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10786.083575 secs 2023-03-09T17:57:56.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:57:56 smithi038 ceph-mon[92422]: pgmap v6265: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 165 KiB/s wr, 8 op/s 2023-03-09T17:57:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:57:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10788.368534 secs 2023-03-09T17:57:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:57:58 smithi122 ceph-mon[93869]: pgmap v6266: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 229 KiB/s wr, 13 op/s 2023-03-09T17:57:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:57:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10788.368534 secs 2023-03-09T17:57:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:57:58 smithi181 ceph-mon[94189]: pgmap v6266: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 229 KiB/s wr, 13 op/s 2023-03-09T17:57:58.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:57:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10788.368534 secs 2023-03-09T17:57:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:57:58 smithi038 ceph-mon[92422]: pgmap v6266: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 229 KiB/s wr, 13 op/s 2023-03-09T17:57:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11250.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:58:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:58:00 smithi122 ceph-mon[93869]: pgmap v6267: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 209 KiB/s wr, 9 op/s 2023-03-09T17:58:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:58:00 smithi181 ceph-mon[94189]: pgmap v6267: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 209 KiB/s wr, 9 op/s 2023-03-09T17:58:00.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:58:00 smithi038 ceph-mon[92422]: pgmap v6267: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 209 KiB/s wr, 9 op/s 2023-03-09T17:58:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:58:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10791.083637 secs 2023-03-09T17:58:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:58:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10791.083637 secs 2023-03-09T17:58:01.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:58:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10791.083637 secs 2023-03-09T17:58:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:58:02 smithi122 ceph-mon[93869]: pgmap v6268: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 250 KiB/s wr, 14 op/s 2023-03-09T17:58:02.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:58:02 smithi181 ceph-mon[94189]: pgmap v6268: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 250 KiB/s wr, 14 op/s 2023-03-09T17:58:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:58:02 smithi038 ceph-mon[92422]: pgmap v6268: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 250 KiB/s wr, 14 op/s 2023-03-09T17:58:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:58:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10793.368628 secs 2023-03-09T17:58:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:58:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10793.368628 secs 2023-03-09T17:58:03.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:58:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10793.368628 secs 2023-03-09T17:58:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11255.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:58:04.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:58:04 smithi122 ceph-mon[93869]: pgmap v6269: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 135 KiB/s wr, 24 op/s 2023-03-09T17:58:04.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:58:04 smithi181 ceph-mon[94189]: pgmap v6269: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 135 KiB/s wr, 24 op/s 2023-03-09T17:58:04.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:58:04 smithi038 ceph-mon[92422]: pgmap v6269: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 135 KiB/s wr, 24 op/s 2023-03-09T17:58:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:58:05 smithi122 ceph-mon[93869]: osdmap e605: 12 total, 12 up, 12 in 2023-03-09T17:58:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:58:05 smithi181 ceph-mon[94189]: osdmap e605: 12 total, 12 up, 12 in 2023-03-09T17:58:05.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:58:05 smithi038 ceph-mon[92422]: osdmap e605: 12 total, 12 up, 12 in 2023-03-09T17:58:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:58:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10796.083697 secs 2023-03-09T17:58:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:58:06 smithi122 ceph-mon[93869]: pgmap v6271: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 151 KiB/s wr, 27 op/s 2023-03-09T17:58:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:58:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10796.083697 secs 2023-03-09T17:58:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:58:06 smithi181 ceph-mon[94189]: pgmap v6271: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 151 KiB/s wr, 27 op/s 2023-03-09T17:58:06.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:58:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10796.083697 secs 2023-03-09T17:58:06.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:58:06 smithi038 ceph-mon[92422]: pgmap v6271: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 151 KiB/s wr, 27 op/s 2023-03-09T17:58:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:58:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10798.368767 secs 2023-03-09T17:58:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:58:08 smithi122 ceph-mon[93869]: pgmap v6272: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 164 KiB/s wr, 26 op/s 2023-03-09T17:58:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:58:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10798.368767 secs 2023-03-09T17:58:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:58:08 smithi181 ceph-mon[94189]: pgmap v6272: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 164 KiB/s wr, 26 op/s 2023-03-09T17:58:08.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:58:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10798.368767 secs 2023-03-09T17:58:08.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:58:08 smithi038 ceph-mon[92422]: pgmap v6272: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 164 KiB/s wr, 26 op/s 2023-03-09T17:58:08.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11260.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:58:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:58:10 smithi122 ceph-mon[93869]: pgmap v6273: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 164 KiB/s wr, 26 op/s 2023-03-09T17:58:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:58:10 smithi181 ceph-mon[94189]: pgmap v6273: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 164 KiB/s wr, 26 op/s 2023-03-09T17:58:10.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:58:10 smithi038 ceph-mon[92422]: pgmap v6273: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 164 KiB/s wr, 26 op/s 2023-03-09T17:58:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:58:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10801.083775 secs 2023-03-09T17:58:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:58:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10801.083775 secs 2023-03-09T17:58:11.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:58:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10801.083775 secs 2023-03-09T17:58:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:58:12 smithi122 ceph-mon[93869]: pgmap v6274: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 116 KiB/s wr, 23 op/s 2023-03-09T17:58:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:58:12 smithi181 ceph-mon[94189]: pgmap v6274: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 116 KiB/s wr, 23 op/s 2023-03-09T17:58:12.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:58:12 smithi038 ceph-mon[92422]: pgmap v6274: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 116 KiB/s wr, 23 op/s 2023-03-09T17:58:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:58:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10803.368890 secs 2023-03-09T17:58:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:58:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10803.368890 secs 2023-03-09T17:58:13.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:58:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10803.368890 secs 2023-03-09T17:58:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11265.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:58:14.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:58:14 smithi122 ceph-mon[93869]: pgmap v6275: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 101 KiB/s wr, 8 op/s 2023-03-09T17:58:14.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:58:14 smithi181 ceph-mon[94189]: pgmap v6275: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 101 KiB/s wr, 8 op/s 2023-03-09T17:58:14.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:58:14 smithi038 ceph-mon[92422]: pgmap v6275: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 101 KiB/s wr, 8 op/s 2023-03-09T17:58:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:58:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10806.083861 secs 2023-03-09T17:58:15.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:58:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10806.083861 secs 2023-03-09T17:58:15.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:58:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10806.083861 secs 2023-03-09T17:58:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:58:16 smithi122 ceph-mon[93869]: pgmap v6276: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 95 KiB/s wr, 7 op/s 2023-03-09T17:58:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:58:16 smithi122 ceph-mon[93869]: osdmap e606: 12 total, 12 up, 12 in 2023-03-09T17:58:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:58:16 smithi181 ceph-mon[94189]: pgmap v6276: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 95 KiB/s wr, 7 op/s 2023-03-09T17:58:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:58:16 smithi181 ceph-mon[94189]: osdmap e606: 12 total, 12 up, 12 in 2023-03-09T17:58:16.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:58:16 smithi038 ceph-mon[92422]: pgmap v6276: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 95 KiB/s wr, 7 op/s 2023-03-09T17:58:16.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:58:16 smithi038 ceph-mon[92422]: osdmap e606: 12 total, 12 up, 12 in 2023-03-09T17:58:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:58:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10808.369025 secs 2023-03-09T17:58:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:58:18 smithi122 ceph-mon[93869]: pgmap v6278: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 8 op/s 2023-03-09T17:58:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:58:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10808.369025 secs 2023-03-09T17:58:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:58:18 smithi181 ceph-mon[94189]: pgmap v6278: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 8 op/s 2023-03-09T17:58:18.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:58:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10808.369025 secs 2023-03-09T17:58:18.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:58:18 smithi038 ceph-mon[92422]: pgmap v6278: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 8 op/s 2023-03-09T17:58:18.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11270.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:58:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:58:20 smithi122 ceph-mon[93869]: pgmap v6279: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T17:58:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:58:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10811.083923 secs 2023-03-09T17:58:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:58:20 smithi181 ceph-mon[94189]: pgmap v6279: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T17:58:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:58:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10811.083923 secs 2023-03-09T17:58:20.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:58:20 smithi038 ceph-mon[92422]: pgmap v6279: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T17:58:20.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:58:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10811.083923 secs 2023-03-09T17:58:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:58:22 smithi122 ceph-mon[93869]: pgmap v6280: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T17:58:22.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:58:22 smithi181 ceph-mon[94189]: pgmap v6280: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T17:58:22.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:58:22 smithi038 ceph-mon[92422]: pgmap v6280: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T17:58:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:58:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10813.369153 secs 2023-03-09T17:58:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:58:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10813.369153 secs 2023-03-09T17:58:23.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:58:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10813.369153 secs 2023-03-09T17:58:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11275.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:58:24.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:58:24 smithi122 ceph-mon[93869]: pgmap v6281: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 7 op/s 2023-03-09T17:58:24.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:58:24 smithi181 ceph-mon[94189]: pgmap v6281: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 7 op/s 2023-03-09T17:58:24.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:58:24 smithi038 ceph-mon[92422]: pgmap v6281: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 7 op/s 2023-03-09T17:58:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:58:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10816.083981 secs 2023-03-09T17:58:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:58:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10816.083981 secs 2023-03-09T17:58:25.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:58:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10816.083981 secs 2023-03-09T17:58:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:58:26 smithi122 ceph-mon[93869]: pgmap v6282: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 7 op/s 2023-03-09T17:58:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:58:26 smithi181 ceph-mon[94189]: pgmap v6282: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 7 op/s 2023-03-09T17:58:26.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:58:26 smithi038 ceph-mon[92422]: pgmap v6282: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 7 op/s 2023-03-09T17:58:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:58:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10818.369271 secs 2023-03-09T17:58:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:58:28 smithi122 ceph-mon[93869]: pgmap v6283: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 46 KiB/s wr, 8 op/s 2023-03-09T17:58:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:58:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10818.369271 secs 2023-03-09T17:58:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:58:28 smithi181 ceph-mon[94189]: pgmap v6283: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 46 KiB/s wr, 8 op/s 2023-03-09T17:58:28.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:58:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10818.369271 secs 2023-03-09T17:58:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:58:28 smithi038 ceph-mon[92422]: pgmap v6283: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 46 KiB/s wr, 8 op/s 2023-03-09T17:58:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11280.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:58:29.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:58:29 smithi038 ceph-mon[92422]: pgmap v6284: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 32 KiB/s wr, 6 op/s 2023-03-09T17:58:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:58:29 smithi122 ceph-mon[93869]: pgmap v6284: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 32 KiB/s wr, 6 op/s 2023-03-09T17:58:29.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:58:29 smithi181 ceph-mon[94189]: pgmap v6284: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 32 KiB/s wr, 6 op/s 2023-03-09T17:58:30.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:58:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10821.084043 secs 2023-03-09T17:58:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:58:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10821.084043 secs 2023-03-09T17:58:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:58:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10821.084043 secs 2023-03-09T17:58:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:58:32 smithi122 ceph-mon[93869]: pgmap v6285: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 59 KiB/s wr, 9 op/s 2023-03-09T17:58:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:58:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:58:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:58:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:58:32.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:58:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:58:32.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:58:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:58:32.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:58:32 smithi181 ceph-mon[94189]: pgmap v6285: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 59 KiB/s wr, 9 op/s 2023-03-09T17:58:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:58:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:58:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:58:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:58:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:58:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:58:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:58:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:58:32.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:58:32 smithi038 ceph-mon[92422]: pgmap v6285: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 59 KiB/s wr, 9 op/s 2023-03-09T17:58:32.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:58:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:58:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:58:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:58:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:58:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:58:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:58:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:58:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:58:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10823.369378 secs 2023-03-09T17:58:33.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:58:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10823.369378 secs 2023-03-09T17:58:33.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:58:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10823.369378 secs 2023-03-09T17:58:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11285.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:58:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:58:34 smithi122 ceph-mon[93869]: pgmap v6286: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 59 KiB/s wr, 7 op/s 2023-03-09T17:58:34.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:58:34 smithi181 ceph-mon[94189]: pgmap v6286: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 59 KiB/s wr, 7 op/s 2023-03-09T17:58:34.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:58:34 smithi038 ceph-mon[92422]: pgmap v6286: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 59 KiB/s wr, 7 op/s 2023-03-09T17:58:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:58:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10826.084106 secs 2023-03-09T17:58:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:58:36 smithi122 ceph-mon[93869]: pgmap v6287: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 59 KiB/s wr, 6 op/s 2023-03-09T17:58:36.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:58:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10826.084106 secs 2023-03-09T17:58:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:58:36 smithi181 ceph-mon[94189]: pgmap v6287: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 59 KiB/s wr, 6 op/s 2023-03-09T17:58:36.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:58:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10826.084106 secs 2023-03-09T17:58:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:58:36 smithi038 ceph-mon[92422]: pgmap v6287: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 59 KiB/s wr, 6 op/s 2023-03-09T17:58:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:58:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10828.369496 secs 2023-03-09T17:58:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:58:38 smithi122 ceph-mon[93869]: pgmap v6288: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 89 KiB/s wr, 10 op/s 2023-03-09T17:58:38.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:58:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10828.369496 secs 2023-03-09T17:58:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:58:38 smithi181 ceph-mon[94189]: pgmap v6288: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 89 KiB/s wr, 10 op/s 2023-03-09T17:58:38.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:58:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10828.369496 secs 2023-03-09T17:58:38.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:58:38 smithi038 ceph-mon[92422]: pgmap v6288: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 89 KiB/s wr, 10 op/s 2023-03-09T17:58:38.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11290.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:58:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:58:40 smithi122 ceph-mon[93869]: pgmap v6289: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 58 KiB/s wr, 7 op/s 2023-03-09T17:58:40.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:58:40 smithi181 ceph-mon[94189]: pgmap v6289: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 58 KiB/s wr, 7 op/s 2023-03-09T17:58:40.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:58:40 smithi038 ceph-mon[92422]: pgmap v6289: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 58 KiB/s wr, 7 op/s 2023-03-09T17:58:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:58:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10831.084165 secs 2023-03-09T17:58:41.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:58:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10831.084165 secs 2023-03-09T17:58:41.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:58:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10831.084165 secs 2023-03-09T17:58:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:58:42 smithi122 ceph-mon[93869]: pgmap v6290: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 149 KiB/s wr, 11 op/s 2023-03-09T17:58:42.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:58:42 smithi181 ceph-mon[94189]: pgmap v6290: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 149 KiB/s wr, 11 op/s 2023-03-09T17:58:42.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:58:42 smithi038 ceph-mon[92422]: pgmap v6290: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 149 KiB/s wr, 11 op/s 2023-03-09T17:58:43.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11295.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:58:44.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:58:44 smithi038 ceph-mon[92422]: pgmap v6291: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 139 KiB/s wr, 10 op/s 2023-03-09T17:58:44.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:58:44 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:58:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:58:44 smithi122 ceph-mon[93869]: pgmap v6291: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 139 KiB/s wr, 10 op/s 2023-03-09T17:58:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:58:44 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:58:44.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:58:44 smithi181 ceph-mon[94189]: pgmap v6291: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 139 KiB/s wr, 10 op/s 2023-03-09T17:58:44.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:58:44 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:58:45.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:58:45 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:58:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:58:45 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:58:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:58:45 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:58:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:58:45 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:58:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:58:45 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:58:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:58:45 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:58:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:58:45 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:58:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:58:45 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:58:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:58:45 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:58:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:58:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10836.084224 secs 2023-03-09T17:58:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:58:46 smithi122 ceph-mon[93869]: pgmap v6292: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 139 KiB/s wr, 9 op/s 2023-03-09T17:58:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:58:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10836.084224 secs 2023-03-09T17:58:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:58:46 smithi181 ceph-mon[94189]: pgmap v6292: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 139 KiB/s wr, 9 op/s 2023-03-09T17:58:46.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:58:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10836.084224 secs 2023-03-09T17:58:46.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:58:46 smithi038 ceph-mon[92422]: pgmap v6292: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 139 KiB/s wr, 9 op/s 2023-03-09T17:58:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:58:48 smithi122 ceph-mon[93869]: pgmap v6293: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 199 KiB/s wr, 13 op/s 2023-03-09T17:58:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:58:48 smithi181 ceph-mon[94189]: pgmap v6293: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 199 KiB/s wr, 13 op/s 2023-03-09T17:58:48.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:58:48 smithi038 ceph-mon[92422]: pgmap v6293: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 199 KiB/s wr, 13 op/s 2023-03-09T17:58:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11300.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:58:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:58:50 smithi122 ceph-mon[93869]: pgmap v6294: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 169 KiB/s wr, 9 op/s 2023-03-09T17:58:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:58:50 smithi181 ceph-mon[94189]: pgmap v6294: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 169 KiB/s wr, 9 op/s 2023-03-09T17:58:50.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:58:50 smithi038 ceph-mon[92422]: pgmap v6294: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 169 KiB/s wr, 9 op/s 2023-03-09T17:58:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:58:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10841.084284 secs 2023-03-09T17:58:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:58:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10841.084284 secs 2023-03-09T17:58:51.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:58:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10841.084284 secs 2023-03-09T17:58:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:58:52 smithi122 ceph-mon[93869]: pgmap v6295: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 246 KiB/s wr, 12 op/s 2023-03-09T17:58:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:58:52 smithi181 ceph-mon[94189]: pgmap v6295: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 246 KiB/s wr, 12 op/s 2023-03-09T17:58:52.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:58:52 smithi038 ceph-mon[92422]: pgmap v6295: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 246 KiB/s wr, 12 op/s 2023-03-09T17:58:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11305.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:58:54.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:58:54 smithi122 ceph-mon[93869]: pgmap v6296: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 165 KiB/s wr, 10 op/s 2023-03-09T17:58:54.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:58:54 smithi181 ceph-mon[94189]: pgmap v6296: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 165 KiB/s wr, 10 op/s 2023-03-09T17:58:54.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:58:54 smithi038 ceph-mon[92422]: pgmap v6296: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 165 KiB/s wr, 10 op/s 2023-03-09T17:58:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:58:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10846.084345 secs 2023-03-09T17:58:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:58:56 smithi122 ceph-mon[93869]: pgmap v6297: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 147 KiB/s wr, 9 op/s 2023-03-09T17:58:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:58:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10846.084345 secs 2023-03-09T17:58:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:58:56 smithi181 ceph-mon[94189]: pgmap v6297: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 147 KiB/s wr, 9 op/s 2023-03-09T17:58:56.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:58:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10846.084345 secs 2023-03-09T17:58:56.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:58:56 smithi038 ceph-mon[92422]: pgmap v6297: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 147 KiB/s wr, 9 op/s 2023-03-09T17:58:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:58:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10848.369995 secs 2023-03-09T17:58:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:58:58 smithi122 ceph-mon[93869]: pgmap v6298: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 169 KiB/s wr, 12 op/s 2023-03-09T17:58:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:58:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10848.369995 secs 2023-03-09T17:58:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:58:58 smithi181 ceph-mon[94189]: pgmap v6298: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 169 KiB/s wr, 12 op/s 2023-03-09T17:58:58.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:58:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10848.369995 secs 2023-03-09T17:58:58.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:58:58 smithi038 ceph-mon[92422]: pgmap v6298: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 169 KiB/s wr, 12 op/s 2023-03-09T17:58:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11310.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:59:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:59:00 smithi122 ceph-mon[93869]: pgmap v6299: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 109 KiB/s wr, 8 op/s 2023-03-09T17:59:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:59:00 smithi181 ceph-mon[94189]: pgmap v6299: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 109 KiB/s wr, 8 op/s 2023-03-09T17:59:00.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:59:00 smithi038 ceph-mon[92422]: pgmap v6299: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 109 KiB/s wr, 8 op/s 2023-03-09T17:59:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:59:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10851.084429 secs 2023-03-09T17:59:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:59:01 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:59:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:59:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:59:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:59:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:59:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:59:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10851.084429 secs 2023-03-09T17:59:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:59:01 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:59:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:59:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:59:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:59:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:59:01.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:59:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10851.084429 secs 2023-03-09T17:59:01.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:59:01 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:59:01.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:59:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:59:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:59:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:59:02.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:59:02 smithi122 ceph-mon[93869]: pgmap v6300: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 207 KiB/s wr, 13 op/s 2023-03-09T17:59:02.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:59:02 smithi181 ceph-mon[94189]: pgmap v6300: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 207 KiB/s wr, 13 op/s 2023-03-09T17:59:02.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:59:02 smithi038 ceph-mon[92422]: pgmap v6300: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 207 KiB/s wr, 13 op/s 2023-03-09T17:59:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:59:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10853.370125 secs 2023-03-09T17:59:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:59:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10853.370125 secs 2023-03-09T17:59:03.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:59:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10853.370125 secs 2023-03-09T17:59:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11315.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:59:04.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:59:04 smithi122 ceph-mon[93869]: pgmap v6301: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 152 KiB/s wr, 23 op/s 2023-03-09T17:59:04.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:59:04 smithi181 ceph-mon[94189]: pgmap v6301: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 152 KiB/s wr, 23 op/s 2023-03-09T17:59:04.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:59:04 smithi038 ceph-mon[92422]: pgmap v6301: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 152 KiB/s wr, 23 op/s 2023-03-09T17:59:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:59:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10856.084491 secs 2023-03-09T17:59:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:59:06 smithi122 ceph-mon[93869]: pgmap v6302: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 142 KiB/s wr, 22 op/s 2023-03-09T17:59:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:59:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10856.084491 secs 2023-03-09T17:59:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:59:06 smithi181 ceph-mon[94189]: pgmap v6302: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 142 KiB/s wr, 22 op/s 2023-03-09T17:59:06.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:59:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10856.084491 secs 2023-03-09T17:59:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:59:06 smithi038 ceph-mon[92422]: pgmap v6302: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 142 KiB/s wr, 22 op/s 2023-03-09T17:59:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:59:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10858.370259 secs 2023-03-09T17:59:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:59:08 smithi122 ceph-mon[93869]: pgmap v6303: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 172 KiB/s wr, 26 op/s 2023-03-09T17:59:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:59:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10858.370259 secs 2023-03-09T17:59:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:59:08 smithi181 ceph-mon[94189]: pgmap v6303: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 172 KiB/s wr, 26 op/s 2023-03-09T17:59:08.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:59:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10858.370259 secs 2023-03-09T17:59:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:59:08 smithi038 ceph-mon[92422]: pgmap v6303: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 172 KiB/s wr, 26 op/s 2023-03-09T17:59:08.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11320.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:59:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:59:10 smithi122 ceph-mon[93869]: pgmap v6304: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 149 KiB/s wr, 22 op/s 2023-03-09T17:59:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:59:10 smithi122 ceph-mon[93869]: osdmap e607: 12 total, 12 up, 12 in 2023-03-09T17:59:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:59:10 smithi181 ceph-mon[94189]: pgmap v6304: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 149 KiB/s wr, 22 op/s 2023-03-09T17:59:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:59:10 smithi181 ceph-mon[94189]: osdmap e607: 12 total, 12 up, 12 in 2023-03-09T17:59:10.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:59:10 smithi038 ceph-mon[92422]: pgmap v6304: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 149 KiB/s wr, 22 op/s 2023-03-09T17:59:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:59:10 smithi038 ceph-mon[92422]: osdmap e607: 12 total, 12 up, 12 in 2023-03-09T17:59:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:59:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10861.084550 secs 2023-03-09T17:59:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:59:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10861.084550 secs 2023-03-09T17:59:11.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:59:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10861.084550 secs 2023-03-09T17:59:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:59:12 smithi122 ceph-mon[93869]: pgmap v6306: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 62 KiB/s wr, 23 op/s 2023-03-09T17:59:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:59:12 smithi181 ceph-mon[94189]: pgmap v6306: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 62 KiB/s wr, 23 op/s 2023-03-09T17:59:12.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:59:12 smithi038 ceph-mon[92422]: pgmap v6306: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 62 KiB/s wr, 23 op/s 2023-03-09T17:59:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:59:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10863.370353 secs 2023-03-09T17:59:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:59:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10863.370353 secs 2023-03-09T17:59:13.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:59:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10863.370353 secs 2023-03-09T17:59:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11325.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:59:14.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:59:14 smithi122 ceph-mon[93869]: pgmap v6307: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 36 KiB/s wr, 8 op/s 2023-03-09T17:59:14.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:59:14 smithi181 ceph-mon[94189]: pgmap v6307: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 36 KiB/s wr, 8 op/s 2023-03-09T17:59:14.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:59:14 smithi038 ceph-mon[92422]: pgmap v6307: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 36 KiB/s wr, 8 op/s 2023-03-09T17:59:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:59:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10866.084611 secs 2023-03-09T17:59:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:59:16 smithi122 ceph-mon[93869]: pgmap v6308: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 36 KiB/s wr, 8 op/s 2023-03-09T17:59:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:59:16 smithi122 ceph-mon[93869]: osdmap e608: 12 total, 12 up, 12 in 2023-03-09T17:59:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:59:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10866.084611 secs 2023-03-09T17:59:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:59:16 smithi181 ceph-mon[94189]: pgmap v6308: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 36 KiB/s wr, 8 op/s 2023-03-09T17:59:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:59:16 smithi181 ceph-mon[94189]: osdmap e608: 12 total, 12 up, 12 in 2023-03-09T17:59:16.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:59:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10866.084611 secs 2023-03-09T17:59:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:59:16 smithi038 ceph-mon[92422]: pgmap v6308: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 36 KiB/s wr, 8 op/s 2023-03-09T17:59:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:59:16 smithi038 ceph-mon[92422]: osdmap e608: 12 total, 12 up, 12 in 2023-03-09T17:59:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:59:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10868.370468 secs 2023-03-09T17:59:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:59:18 smithi122 ceph-mon[93869]: pgmap v6310: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 KiB/s wr, 10 op/s 2023-03-09T17:59:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:59:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10868.370468 secs 2023-03-09T17:59:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:59:18 smithi181 ceph-mon[94189]: pgmap v6310: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 KiB/s wr, 10 op/s 2023-03-09T17:59:18.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:59:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10868.370468 secs 2023-03-09T17:59:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:59:18 smithi038 ceph-mon[92422]: pgmap v6310: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 KiB/s wr, 10 op/s 2023-03-09T17:59:18.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11330.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:59:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:59:20 smithi122 ceph-mon[93869]: pgmap v6311: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 8 op/s 2023-03-09T17:59:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:59:20 smithi181 ceph-mon[94189]: pgmap v6311: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 8 op/s 2023-03-09T17:59:20.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:59:20 smithi038 ceph-mon[92422]: pgmap v6311: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 8 op/s 2023-03-09T17:59:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:59:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10871.084670 secs 2023-03-09T17:59:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:59:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10871.084670 secs 2023-03-09T17:59:21.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:59:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10871.084670 secs 2023-03-09T17:59:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:59:22 smithi122 ceph-mon[93869]: pgmap v6312: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 7 op/s 2023-03-09T17:59:22.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:59:22 smithi181 ceph-mon[94189]: pgmap v6312: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 7 op/s 2023-03-09T17:59:22.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:59:22 smithi038 ceph-mon[92422]: pgmap v6312: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 7 op/s 2023-03-09T17:59:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:59:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10873.370589 secs 2023-03-09T17:59:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:59:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10873.370589 secs 2023-03-09T17:59:23.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:59:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10873.370589 secs 2023-03-09T17:59:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11335.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:59:24.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:59:24 smithi122 ceph-mon[93869]: pgmap v6313: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 8 op/s 2023-03-09T17:59:24.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:59:24 smithi181 ceph-mon[94189]: pgmap v6313: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 8 op/s 2023-03-09T17:59:24.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:59:24 smithi038 ceph-mon[92422]: pgmap v6313: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 8 op/s 2023-03-09T17:59:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:59:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10876.084731 secs 2023-03-09T17:59:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:59:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10876.084731 secs 2023-03-09T17:59:25.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:59:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10876.084731 secs 2023-03-09T17:59:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:59:26 smithi122 ceph-mon[93869]: pgmap v6314: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 8 op/s 2023-03-09T17:59:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:59:26 smithi181 ceph-mon[94189]: pgmap v6314: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 8 op/s 2023-03-09T17:59:26.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:59:26 smithi038 ceph-mon[92422]: pgmap v6314: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 8 op/s 2023-03-09T17:59:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:59:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10878.370690 secs 2023-03-09T17:59:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:59:28 smithi122 ceph-mon[93869]: pgmap v6315: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 45 KiB/s wr, 8 op/s 2023-03-09T17:59:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:59:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10878.370690 secs 2023-03-09T17:59:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:59:28 smithi181 ceph-mon[94189]: pgmap v6315: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 45 KiB/s wr, 8 op/s 2023-03-09T17:59:28.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:59:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10878.370690 secs 2023-03-09T17:59:28.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:59:28 smithi038 ceph-mon[92422]: pgmap v6315: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 45 KiB/s wr, 8 op/s 2023-03-09T17:59:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11340.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:59:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:59:30 smithi122 ceph-mon[93869]: pgmap v6316: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 31 KiB/s wr, 6 op/s 2023-03-09T17:59:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:59:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10881.084808 secs 2023-03-09T17:59:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:59:30 smithi181 ceph-mon[94189]: pgmap v6316: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 31 KiB/s wr, 6 op/s 2023-03-09T17:59:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:59:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10881.084808 secs 2023-03-09T17:59:30.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:59:30 smithi038 ceph-mon[92422]: pgmap v6316: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 31 KiB/s wr, 6 op/s 2023-03-09T17:59:30.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:59:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10881.084808 secs 2023-03-09T17:59:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:59:32 smithi122 ceph-mon[93869]: pgmap v6317: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 31 KiB/s wr, 8 op/s 2023-03-09T17:59:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:59:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:59:32.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:59:32 smithi181 ceph-mon[94189]: pgmap v6317: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 31 KiB/s wr, 8 op/s 2023-03-09T17:59:32.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:59:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:59:32.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:59:32 smithi038 ceph-mon[92422]: pgmap v6317: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 31 KiB/s wr, 8 op/s 2023-03-09T17:59:32.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:59:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T17:59:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:59:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:59:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:59:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:59:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:59:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:59:33.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:59:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:59:33.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:59:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:59:33.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:59:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:59:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11345.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:59:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:59:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T17:59:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:59:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T17:59:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:59:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T17:59:34.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:59:34 smithi038 ceph-mon[92422]: pgmap v6318: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 31 KiB/s wr, 7 op/s 2023-03-09T17:59:34.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:59:34 smithi122 ceph-mon[93869]: pgmap v6318: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 31 KiB/s wr, 7 op/s 2023-03-09T17:59:34.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:59:34 smithi181 ceph-mon[94189]: pgmap v6318: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 31 KiB/s wr, 7 op/s 2023-03-09T17:59:35.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:59:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10886.084868 secs 2023-03-09T17:59:35.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:59:35 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:59:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:59:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10886.084868 secs 2023-03-09T17:59:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:59:35 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:59:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:59:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10886.084868 secs 2023-03-09T17:59:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:59:35 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T17:59:36.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:59:36 smithi038 ceph-mon[92422]: pgmap v6319: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 31 KiB/s wr, 6 op/s 2023-03-09T17:59:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:59:36 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:59:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:59:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:59:36.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:59:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:59:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:59:36 smithi122 ceph-mon[93869]: pgmap v6319: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 31 KiB/s wr, 6 op/s 2023-03-09T17:59:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:59:36 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:59:36.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:59:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:59:36.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:59:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:59:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:59:36 smithi181 ceph-mon[94189]: pgmap v6319: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 31 KiB/s wr, 6 op/s 2023-03-09T17:59:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:59:36 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:59:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:59:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:59:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:59:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:59:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:59:37 smithi122 ceph-mon[93869]: pgmap v6320: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 88 KiB/s wr, 9 op/s 2023-03-09T17:59:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:59:37 smithi181 ceph-mon[94189]: pgmap v6320: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 88 KiB/s wr, 9 op/s 2023-03-09T17:59:37.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:59:37 smithi038 ceph-mon[92422]: pgmap v6320: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 88 KiB/s wr, 9 op/s 2023-03-09T17:59:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11350.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:59:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:59:38 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 10888.370975 secs 2023-03-09T17:59:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:59:38 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 10888.370975 secs 2023-03-09T17:59:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:59:38 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 10888.370975 secs 2023-03-09T17:59:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:59:39 smithi122 ceph-mon[93869]: pgmap v6321: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 57 KiB/s wr, 6 op/s 2023-03-09T17:59:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:59:39 smithi181 ceph-mon[94189]: pgmap v6321: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 57 KiB/s wr, 6 op/s 2023-03-09T17:59:39.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:59:39 smithi038 ceph-mon[92422]: pgmap v6321: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 57 KiB/s wr, 6 op/s 2023-03-09T17:59:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:59:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:59:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:59:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:59:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:59:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10891.084952 secs 2023-03-09T17:59:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:59:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:59:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:59:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:59:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:59:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10891.084952 secs 2023-03-09T17:59:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:59:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:59:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:59:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:59:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:59:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10891.084952 secs 2023-03-09T17:59:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:59:41 smithi122 ceph-mon[93869]: pgmap v6322: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 114 KiB/s wr, 10 op/s 2023-03-09T17:59:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:59:41 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:59:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:59:41 smithi181 ceph-mon[94189]: pgmap v6322: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 114 KiB/s wr, 10 op/s 2023-03-09T17:59:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:59:41 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:59:41.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:59:41 smithi038 ceph-mon[92422]: pgmap v6322: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 114 KiB/s wr, 10 op/s 2023-03-09T17:59:41.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:59:41 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T17:59:43.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:59:42 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 10893.371118 secs 2023-03-09T17:59:43.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:59:42 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 10893.371118 secs 2023-03-09T17:59:43.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:59:42 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 10893.371118 secs 2023-03-09T17:59:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11355.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:59:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:59:43 smithi122 ceph-mon[93869]: pgmap v6323: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 124 KiB/s wr, 9 op/s 2023-03-09T17:59:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:59:43 smithi181 ceph-mon[94189]: pgmap v6323: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 124 KiB/s wr, 9 op/s 2023-03-09T17:59:44.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:59:43 smithi038 ceph-mon[92422]: pgmap v6323: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 124 KiB/s wr, 9 op/s 2023-03-09T17:59:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:59:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10896.084986 secs 2023-03-09T17:59:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:59:46 smithi122 ceph-mon[93869]: pgmap v6324: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 123 KiB/s wr, 8 op/s 2023-03-09T17:59:46.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:59:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10896.084986 secs 2023-03-09T17:59:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:59:46 smithi181 ceph-mon[94189]: pgmap v6324: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 123 KiB/s wr, 8 op/s 2023-03-09T17:59:46.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:59:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10896.084986 secs 2023-03-09T17:59:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:59:46 smithi038 ceph-mon[92422]: pgmap v6324: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 123 KiB/s wr, 8 op/s 2023-03-09T17:59:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:59:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10898.371244 secs 2023-03-09T17:59:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:59:48 smithi122 ceph-mon[93869]: pgmap v6325: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 194 KiB/s wr, 12 op/s 2023-03-09T17:59:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:59:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10898.371244 secs 2023-03-09T17:59:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:59:48 smithi181 ceph-mon[94189]: pgmap v6325: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 194 KiB/s wr, 12 op/s 2023-03-09T17:59:48.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:59:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10898.371244 secs 2023-03-09T17:59:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:59:48 smithi038 ceph-mon[92422]: pgmap v6325: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 194 KiB/s wr, 12 op/s 2023-03-09T17:59:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11360.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:59:49.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:59:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:59:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:59:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:59:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:59:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:59:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:59:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:59:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:59:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T17:59:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:59:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T17:59:50.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:59:50 smithi038 ceph-mon[92422]: pgmap v6326: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 137 KiB/s wr, 9 op/s 2023-03-09T17:59:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:59:50 smithi122 ceph-mon[93869]: pgmap v6326: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 137 KiB/s wr, 9 op/s 2023-03-09T17:59:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:59:50 smithi181 ceph-mon[94189]: pgmap v6326: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 137 KiB/s wr, 9 op/s 2023-03-09T17:59:51.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:59:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10901.085047 secs 2023-03-09T17:59:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:59:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10901.085047 secs 2023-03-09T17:59:51.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:59:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10901.085047 secs 2023-03-09T17:59:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:59:52 smithi122 ceph-mon[93869]: pgmap v6327: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 191 KiB/s wr, 11 op/s 2023-03-09T17:59:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:59:52 smithi181 ceph-mon[94189]: pgmap v6327: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 191 KiB/s wr, 11 op/s 2023-03-09T17:59:52.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:59:52 smithi038 ceph-mon[92422]: pgmap v6327: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 191 KiB/s wr, 11 op/s 2023-03-09T17:59:53.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:59:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10903.371357 secs 2023-03-09T17:59:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:59:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10903.371357 secs 2023-03-09T17:59:53.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:59:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10903.371357 secs 2023-03-09T17:59:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11365.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T17:59:54.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:59:54 smithi122 ceph-mon[93869]: pgmap v6328: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 135 KiB/s wr, 10 op/s 2023-03-09T17:59:54.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:59:54 smithi181 ceph-mon[94189]: pgmap v6328: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 135 KiB/s wr, 10 op/s 2023-03-09T17:59:54.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:59:54 smithi038 ceph-mon[92422]: pgmap v6328: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 135 KiB/s wr, 10 op/s 2023-03-09T17:59:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:59:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10906.085107 secs 2023-03-09T17:59:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:59:56 smithi122 ceph-mon[93869]: pgmap v6329: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 125 KiB/s wr, 8 op/s 2023-03-09T17:59:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:59:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10906.085107 secs 2023-03-09T17:59:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:59:56 smithi181 ceph-mon[94189]: pgmap v6329: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 125 KiB/s wr, 8 op/s 2023-03-09T17:59:56.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:59:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10906.085107 secs 2023-03-09T17:59:56.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:59:56 smithi038 ceph-mon[92422]: pgmap v6329: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 125 KiB/s wr, 8 op/s 2023-03-09T17:59:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:59:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10908.371483 secs 2023-03-09T17:59:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 17:59:58 smithi122 ceph-mon[93869]: pgmap v6330: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 235 KiB/s wr, 12 op/s 2023-03-09T17:59:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:59:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10908.371483 secs 2023-03-09T17:59:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 17:59:58 smithi181 ceph-mon[94189]: pgmap v6330: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 235 KiB/s wr, 12 op/s 2023-03-09T17:59:58.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:59:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10908.371483 secs 2023-03-09T17:59:58.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 17:59:58 smithi038 ceph-mon[92422]: pgmap v6330: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 235 KiB/s wr, 12 op/s 2023-03-09T17:59:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11370.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:00:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:00:00 smithi122 ceph-mon[93869]: pgmap v6331: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 164 KiB/s wr, 9 op/s 2023-03-09T18:00:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:00:00 smithi122 ceph-mon[93869]: overall HEALTH_WARN 2 MDSs report slow requests 2023-03-09T18:00:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:00:00 smithi181 ceph-mon[94189]: pgmap v6331: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 164 KiB/s wr, 9 op/s 2023-03-09T18:00:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:00:00 smithi181 ceph-mon[94189]: overall HEALTH_WARN 2 MDSs report slow requests 2023-03-09T18:00:00.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:00:00 smithi038 ceph-mon[92422]: pgmap v6331: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 164 KiB/s wr, 9 op/s 2023-03-09T18:00:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:00:00 smithi038 ceph-mon[92422]: overall HEALTH_WARN 2 MDSs report slow requests 2023-03-09T18:00:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:00:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10911.085193 secs 2023-03-09T18:00:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:00:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10911.085193 secs 2023-03-09T18:00:01.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:00:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10911.085193 secs 2023-03-09T18:00:02.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:00:02 smithi122 ceph-mon[93869]: pgmap v6332: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 233 KiB/s wr, 14 op/s 2023-03-09T18:00:02.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:00:02 smithi181 ceph-mon[94189]: pgmap v6332: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 233 KiB/s wr, 14 op/s 2023-03-09T18:00:02.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:00:02 smithi038 ceph-mon[92422]: pgmap v6332: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 233 KiB/s wr, 14 op/s 2023-03-09T18:00:03.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:00:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10913.371617 secs 2023-03-09T18:00:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:00:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10913.371617 secs 2023-03-09T18:00:03.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:00:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10913.371617 secs 2023-03-09T18:00:03.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11375.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:00:04.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:00:04 smithi122 ceph-mon[93869]: pgmap v6333: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 200 KiB/s wr, 24 op/s 2023-03-09T18:00:04.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:00:04 smithi181 ceph-mon[94189]: pgmap v6333: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 200 KiB/s wr, 24 op/s 2023-03-09T18:00:04.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:00:04 smithi038 ceph-mon[92422]: pgmap v6333: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 200 KiB/s wr, 24 op/s 2023-03-09T18:00:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:00:05 smithi122 ceph-mon[93869]: osdmap e609: 12 total, 12 up, 12 in 2023-03-09T18:00:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:00:05 smithi181 ceph-mon[94189]: osdmap e609: 12 total, 12 up, 12 in 2023-03-09T18:00:05.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:00:05 smithi038 ceph-mon[92422]: osdmap e609: 12 total, 12 up, 12 in 2023-03-09T18:00:06.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:00:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10916.085256 secs 2023-03-09T18:00:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:00:06 smithi122 ceph-mon[93869]: pgmap v6335: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 240 KiB/s wr, 27 op/s 2023-03-09T18:00:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:00:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10916.085256 secs 2023-03-09T18:00:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:00:06 smithi181 ceph-mon[94189]: pgmap v6335: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 240 KiB/s wr, 27 op/s 2023-03-09T18:00:06.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:00:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10916.085256 secs 2023-03-09T18:00:06.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:00:06 smithi038 ceph-mon[92422]: pgmap v6335: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 240 KiB/s wr, 27 op/s 2023-03-09T18:00:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:00:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10918.371738 secs 2023-03-09T18:00:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:00:08 smithi122 ceph-mon[93869]: pgmap v6336: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 169 KiB/s wr, 27 op/s 2023-03-09T18:00:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:00:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10918.371738 secs 2023-03-09T18:00:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:00:08 smithi181 ceph-mon[94189]: pgmap v6336: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 169 KiB/s wr, 27 op/s 2023-03-09T18:00:08.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:00:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10918.371738 secs 2023-03-09T18:00:08.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:00:08 smithi038 ceph-mon[92422]: pgmap v6336: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 169 KiB/s wr, 27 op/s 2023-03-09T18:00:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11380.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:00:10.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:00:10 smithi122 ceph-mon[93869]: pgmap v6337: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 169 KiB/s wr, 27 op/s 2023-03-09T18:00:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:00:10 smithi181 ceph-mon[94189]: pgmap v6337: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 169 KiB/s wr, 27 op/s 2023-03-09T18:00:10.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:00:10 smithi038 ceph-mon[92422]: pgmap v6337: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 169 KiB/s wr, 27 op/s 2023-03-09T18:00:11.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:00:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10921.085317 secs 2023-03-09T18:00:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:00:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10921.085317 secs 2023-03-09T18:00:11.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:00:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10921.085317 secs 2023-03-09T18:00:12.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:00:12 smithi122 ceph-mon[93869]: pgmap v6338: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 112 KiB/s wr, 23 op/s 2023-03-09T18:00:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:00:12 smithi181 ceph-mon[94189]: pgmap v6338: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 112 KiB/s wr, 23 op/s 2023-03-09T18:00:12.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:00:12 smithi038 ceph-mon[92422]: pgmap v6338: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 112 KiB/s wr, 23 op/s 2023-03-09T18:00:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:00:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10923.371799 secs 2023-03-09T18:00:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:00:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10923.371799 secs 2023-03-09T18:00:13.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:00:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10923.371799 secs 2023-03-09T18:00:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11385.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:00:14.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:00:14 smithi122 ceph-mon[93869]: pgmap v6339: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 98 KiB/s wr, 9 op/s 2023-03-09T18:00:14.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:00:14 smithi181 ceph-mon[94189]: pgmap v6339: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 98 KiB/s wr, 9 op/s 2023-03-09T18:00:14.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:00:14 smithi038 ceph-mon[92422]: pgmap v6339: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 98 KiB/s wr, 9 op/s 2023-03-09T18:00:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:00:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10926.085375 secs 2023-03-09T18:00:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:00:16 smithi122 ceph-mon[93869]: pgmap v6340: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 91 KiB/s wr, 8 op/s 2023-03-09T18:00:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:00:16 smithi122 ceph-mon[93869]: osdmap e610: 12 total, 12 up, 12 in 2023-03-09T18:00:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:00:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10926.085375 secs 2023-03-09T18:00:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:00:16 smithi181 ceph-mon[94189]: pgmap v6340: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 91 KiB/s wr, 8 op/s 2023-03-09T18:00:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:00:16 smithi181 ceph-mon[94189]: osdmap e610: 12 total, 12 up, 12 in 2023-03-09T18:00:16.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:00:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10926.085375 secs 2023-03-09T18:00:16.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:00:16 smithi038 ceph-mon[92422]: pgmap v6340: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 91 KiB/s wr, 8 op/s 2023-03-09T18:00:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:00:16 smithi038 ceph-mon[92422]: osdmap e610: 12 total, 12 up, 12 in 2023-03-09T18:00:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:00:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10928.371886 secs 2023-03-09T18:00:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:00:18 smithi122 ceph-mon[93869]: pgmap v6342: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 39 KiB/s wr, 8 op/s 2023-03-09T18:00:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:00:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10928.371886 secs 2023-03-09T18:00:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:00:18 smithi181 ceph-mon[94189]: pgmap v6342: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 39 KiB/s wr, 8 op/s 2023-03-09T18:00:18.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:00:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10928.371886 secs 2023-03-09T18:00:18.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:00:18 smithi038 ceph-mon[92422]: pgmap v6342: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 39 KiB/s wr, 8 op/s 2023-03-09T18:00:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11390.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:00:20.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:00:20 smithi122 ceph-mon[93869]: pgmap v6343: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 39 KiB/s wr, 8 op/s 2023-03-09T18:00:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:00:20 smithi181 ceph-mon[94189]: pgmap v6343: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 39 KiB/s wr, 8 op/s 2023-03-09T18:00:20.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:00:20 smithi038 ceph-mon[92422]: pgmap v6343: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 39 KiB/s wr, 8 op/s 2023-03-09T18:00:21.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:00:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10931.085436 secs 2023-03-09T18:00:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:00:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10931.085436 secs 2023-03-09T18:00:21.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:00:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10931.085436 secs 2023-03-09T18:00:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:00:22 smithi122 ceph-mon[93869]: pgmap v6344: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 88 KiB/s wr, 8 op/s 2023-03-09T18:00:22.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:00:22 smithi181 ceph-mon[94189]: pgmap v6344: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 88 KiB/s wr, 8 op/s 2023-03-09T18:00:22.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:00:22 smithi038 ceph-mon[92422]: pgmap v6344: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 88 KiB/s wr, 8 op/s 2023-03-09T18:00:23.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:00:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10933.371965 secs 2023-03-09T18:00:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:00:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10933.371965 secs 2023-03-09T18:00:23.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:00:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10933.371965 secs 2023-03-09T18:00:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11395.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:00:24.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:00:24 smithi122 ceph-mon[93869]: pgmap v6345: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 76 KiB/s wr, 8 op/s 2023-03-09T18:00:24.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:00:24 smithi181 ceph-mon[94189]: pgmap v6345: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 76 KiB/s wr, 8 op/s 2023-03-09T18:00:24.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:00:24 smithi038 ceph-mon[92422]: pgmap v6345: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 76 KiB/s wr, 8 op/s 2023-03-09T18:00:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:00:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10936.085499 secs 2023-03-09T18:00:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:00:26 smithi122 ceph-mon[93869]: pgmap v6346: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 76 KiB/s wr, 8 op/s 2023-03-09T18:00:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:00:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10936.085499 secs 2023-03-09T18:00:26.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:00:26 smithi181 ceph-mon[94189]: pgmap v6346: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 76 KiB/s wr, 8 op/s 2023-03-09T18:00:26.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:00:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10936.085499 secs 2023-03-09T18:00:26.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:00:26 smithi038 ceph-mon[92422]: pgmap v6346: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 76 KiB/s wr, 8 op/s 2023-03-09T18:00:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:00:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10938.372085 secs 2023-03-09T18:00:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:00:28 smithi122 ceph-mon[93869]: pgmap v6347: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 69 KiB/s wr, 8 op/s 2023-03-09T18:00:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:00:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10938.372085 secs 2023-03-09T18:00:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:00:28 smithi181 ceph-mon[94189]: pgmap v6347: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 69 KiB/s wr, 8 op/s 2023-03-09T18:00:28.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:00:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10938.372085 secs 2023-03-09T18:00:28.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:00:28 smithi038 ceph-mon[92422]: pgmap v6347: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 69 KiB/s wr, 8 op/s 2023-03-09T18:00:28.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11400.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:00:30.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:00:30 smithi122 ceph-mon[93869]: pgmap v6348: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 62 KiB/s wr, 6 op/s 2023-03-09T18:00:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:00:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10941.085565 secs 2023-03-09T18:00:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:00:30 smithi181 ceph-mon[94189]: pgmap v6348: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 62 KiB/s wr, 6 op/s 2023-03-09T18:00:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:00:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10941.085565 secs 2023-03-09T18:00:30.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:00:30 smithi038 ceph-mon[92422]: pgmap v6348: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 62 KiB/s wr, 6 op/s 2023-03-09T18:00:30.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:00:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10941.085565 secs 2023-03-09T18:00:32.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:00:32 smithi122 ceph-mon[93869]: pgmap v6349: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 80 KiB/s wr, 9 op/s 2023-03-09T18:00:32.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:00:32 smithi181 ceph-mon[94189]: pgmap v6349: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 80 KiB/s wr, 9 op/s 2023-03-09T18:00:32.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:00:32 smithi038 ceph-mon[92422]: pgmap v6349: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 80 KiB/s wr, 9 op/s 2023-03-09T18:00:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:00:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10943.372207 secs 2023-03-09T18:00:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:00:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:00:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:00:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10943.372207 secs 2023-03-09T18:00:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:00:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:00:33.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:00:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10943.372207 secs 2023-03-09T18:00:33.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:00:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:00:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11405.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:00:34.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:00:34 smithi122 ceph-mon[93869]: pgmap v6350: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 6 op/s 2023-03-09T18:00:34.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:00:34 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:00:34.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:00:34 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:00:34.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:00:34 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:00:34.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:00:34 smithi181 ceph-mon[94189]: pgmap v6350: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 6 op/s 2023-03-09T18:00:34.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:00:34 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:00:34.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:00:34 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:00:34.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:00:34 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:00:34.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:00:34 smithi038 ceph-mon[92422]: pgmap v6350: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 6 op/s 2023-03-09T18:00:34.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:00:34 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:00:34.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:00:34 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:00:34.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:00:34 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:00:35.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:00:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10946.085627 secs 2023-03-09T18:00:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:00:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10946.085627 secs 2023-03-09T18:00:35.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:00:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10946.085627 secs 2023-03-09T18:00:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:00:36 smithi122 ceph-mon[93869]: pgmap v6351: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 5 op/s 2023-03-09T18:00:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:00:36 smithi181 ceph-mon[94189]: pgmap v6351: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 5 op/s 2023-03-09T18:00:36.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:00:36 smithi038 ceph-mon[92422]: pgmap v6351: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 5 op/s 2023-03-09T18:00:38.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:00:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10948.372328 secs 2023-03-09T18:00:38.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:00:38 smithi038 ceph-mon[92422]: pgmap v6352: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 99 KiB/s wr, 9 op/s 2023-03-09T18:00:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11410.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:00:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:00:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10948.372328 secs 2023-03-09T18:00:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:00:38 smithi122 ceph-mon[93869]: pgmap v6352: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 99 KiB/s wr, 9 op/s 2023-03-09T18:00:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:00:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10948.372328 secs 2023-03-09T18:00:38.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:00:38 smithi181 ceph-mon[94189]: pgmap v6352: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 99 KiB/s wr, 9 op/s 2023-03-09T18:00:40.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:00:40 smithi038 ceph-mon[92422]: pgmap v6353: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 99 KiB/s wr, 6 op/s 2023-03-09T18:00:40.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:00:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10951.085685 secs 2023-03-09T18:00:40.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:00:40 smithi122 ceph-mon[93869]: pgmap v6353: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 99 KiB/s wr, 6 op/s 2023-03-09T18:00:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:00:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10951.085685 secs 2023-03-09T18:00:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:00:40 smithi181 ceph-mon[94189]: pgmap v6353: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 99 KiB/s wr, 6 op/s 2023-03-09T18:00:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:00:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10951.085685 secs 2023-03-09T18:00:42.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:00:42 smithi038 ceph-mon[92422]: pgmap v6354: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 133 KiB/s wr, 10 op/s 2023-03-09T18:00:42.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:00:42 smithi122 ceph-mon[93869]: pgmap v6354: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 133 KiB/s wr, 10 op/s 2023-03-09T18:00:42.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:00:42 smithi181 ceph-mon[94189]: pgmap v6354: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 133 KiB/s wr, 10 op/s 2023-03-09T18:00:43.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:00:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10953.372447 secs 2023-03-09T18:00:43.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:00:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10953.372447 secs 2023-03-09T18:00:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:00:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10953.372447 secs 2023-03-09T18:00:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11415.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:00:44.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:00:44 smithi038 ceph-mon[92422]: pgmap v6355: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 125 KiB/s wr, 9 op/s 2023-03-09T18:00:44.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:00:44 smithi122 ceph-mon[93869]: pgmap v6355: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 125 KiB/s wr, 9 op/s 2023-03-09T18:00:44.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:00:44 smithi181 ceph-mon[94189]: pgmap v6355: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 125 KiB/s wr, 9 op/s 2023-03-09T18:00:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:00:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10956.085747 secs 2023-03-09T18:00:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:00:46 smithi122 ceph-mon[93869]: pgmap v6356: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 152 KiB/s wr, 9 op/s 2023-03-09T18:00:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:00:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10956.085747 secs 2023-03-09T18:00:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:00:46 smithi181 ceph-mon[94189]: pgmap v6356: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 152 KiB/s wr, 9 op/s 2023-03-09T18:00:46.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:00:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10956.085747 secs 2023-03-09T18:00:46.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:00:46 smithi038 ceph-mon[92422]: pgmap v6356: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 152 KiB/s wr, 9 op/s 2023-03-09T18:00:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:00:48 smithi122 ceph-mon[93869]: pgmap v6357: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 210 KiB/s wr, 12 op/s 2023-03-09T18:00:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:00:48 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:00:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:00:48 smithi181 ceph-mon[94189]: pgmap v6357: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 210 KiB/s wr, 12 op/s 2023-03-09T18:00:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:00:48 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:00:48.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:00:48 smithi038 ceph-mon[92422]: pgmap v6357: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 210 KiB/s wr, 12 op/s 2023-03-09T18:00:48.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:00:48 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:00:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11420.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:00:49.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:00:49 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:00:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:00:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:00:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:00:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:00:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:00:49 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:00:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:00:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:00:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:00:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:00:49.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:00:49 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:00:49.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:00:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:00:49.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:00:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:00:50.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:00:50 smithi122 ceph-mon[93869]: pgmap v6358: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 163 KiB/s wr, 10 op/s 2023-03-09T18:00:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:00:50 smithi181 ceph-mon[94189]: pgmap v6358: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 163 KiB/s wr, 10 op/s 2023-03-09T18:00:50.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:00:50 smithi038 ceph-mon[92422]: pgmap v6358: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 163 KiB/s wr, 10 op/s 2023-03-09T18:00:51.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:00:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10961.085807 secs 2023-03-09T18:00:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:00:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10961.085807 secs 2023-03-09T18:00:51.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:00:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10961.085807 secs 2023-03-09T18:00:52.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:00:52 smithi122 ceph-mon[93869]: pgmap v6359: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 163 KiB/s wr, 12 op/s 2023-03-09T18:00:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:00:52 smithi181 ceph-mon[94189]: pgmap v6359: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 163 KiB/s wr, 12 op/s 2023-03-09T18:00:52.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:00:52 smithi038 ceph-mon[92422]: pgmap v6359: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 163 KiB/s wr, 12 op/s 2023-03-09T18:00:53.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11425.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:00:54.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:00:54 smithi122 ceph-mon[93869]: pgmap v6360: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 129 KiB/s wr, 10 op/s 2023-03-09T18:00:54.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:00:54 smithi181 ceph-mon[94189]: pgmap v6360: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 129 KiB/s wr, 10 op/s 2023-03-09T18:00:54.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:00:54 smithi038 ceph-mon[92422]: pgmap v6360: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 129 KiB/s wr, 10 op/s 2023-03-09T18:00:56.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:00:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10966.085869 secs 2023-03-09T18:00:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:00:56 smithi122 ceph-mon[93869]: pgmap v6361: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 199 KiB/s wr, 10 op/s 2023-03-09T18:00:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:00:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10966.085869 secs 2023-03-09T18:00:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:00:56 smithi181 ceph-mon[94189]: pgmap v6361: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 199 KiB/s wr, 10 op/s 2023-03-09T18:00:56.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:00:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10966.085869 secs 2023-03-09T18:00:56.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:00:56 smithi038 ceph-mon[92422]: pgmap v6361: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 199 KiB/s wr, 10 op/s 2023-03-09T18:00:58.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:00:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10968.372775 secs 2023-03-09T18:00:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:00:58 smithi122 ceph-mon[93869]: pgmap v6362: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 187 KiB/s wr, 12 op/s 2023-03-09T18:00:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:00:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10968.372775 secs 2023-03-09T18:00:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:00:58 smithi181 ceph-mon[94189]: pgmap v6362: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 187 KiB/s wr, 12 op/s 2023-03-09T18:00:58.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:00:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10968.372775 secs 2023-03-09T18:00:58.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:00:58 smithi038 ceph-mon[92422]: pgmap v6362: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 187 KiB/s wr, 12 op/s 2023-03-09T18:00:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11430.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:01:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:00 smithi122 ceph-mon[93869]: pgmap v6363: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 152 KiB/s wr, 9 op/s 2023-03-09T18:01:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:00 smithi181 ceph-mon[94189]: pgmap v6363: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 152 KiB/s wr, 9 op/s 2023-03-09T18:01:00.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:00 smithi038 ceph-mon[92422]: pgmap v6363: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 152 KiB/s wr, 9 op/s 2023-03-09T18:01:01.621 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10971.085926 secs 2023-03-09T18:01:01.622 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:01 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:01:01.622 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:01:01.622 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:01:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10971.085926 secs 2023-03-09T18:01:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:01 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:01:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:01:01.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:01:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10971.085926 secs 2023-03-09T18:01:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:01 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:01:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:01:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:01:02.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:02 smithi122 ceph-mon[93869]: pgmap v6364: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 141 KiB/s wr, 13 op/s 2023-03-09T18:01:02.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:02 smithi181 ceph-mon[94189]: pgmap v6364: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 141 KiB/s wr, 13 op/s 2023-03-09T18:01:02.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:02 smithi038 ceph-mon[92422]: pgmap v6364: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 141 KiB/s wr, 13 op/s 2023-03-09T18:01:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10973.372905 secs 2023-03-09T18:01:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10973.372905 secs 2023-03-09T18:01:03.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10973.372905 secs 2023-03-09T18:01:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11435.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:01:04.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:04 smithi122 ceph-mon[93869]: pgmap v6365: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 162 KiB/s wr, 23 op/s 2023-03-09T18:01:04.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:04 smithi181 ceph-mon[94189]: pgmap v6365: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 162 KiB/s wr, 23 op/s 2023-03-09T18:01:04.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:04 smithi038 ceph-mon[92422]: pgmap v6365: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 162 KiB/s wr, 23 op/s 2023-03-09T18:01:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:05 smithi122 ceph-mon[93869]: osdmap e611: 12 total, 12 up, 12 in 2023-03-09T18:01:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:05 smithi181 ceph-mon[94189]: osdmap e611: 12 total, 12 up, 12 in 2023-03-09T18:01:05.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:05 smithi038 ceph-mon[92422]: osdmap e611: 12 total, 12 up, 12 in 2023-03-09T18:01:06.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10976.086009 secs 2023-03-09T18:01:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:06 smithi122 ceph-mon[93869]: pgmap v6367: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 161 KiB/s wr, 27 op/s 2023-03-09T18:01:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10976.086009 secs 2023-03-09T18:01:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:06 smithi181 ceph-mon[94189]: pgmap v6367: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 161 KiB/s wr, 27 op/s 2023-03-09T18:01:06.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10976.086009 secs 2023-03-09T18:01:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:06 smithi038 ceph-mon[92422]: pgmap v6367: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 161 KiB/s wr, 27 op/s 2023-03-09T18:01:08.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10978.373003 secs 2023-03-09T18:01:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:08 smithi122 ceph-mon[93869]: pgmap v6368: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 144 KiB/s wr, 27 op/s 2023-03-09T18:01:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10978.373003 secs 2023-03-09T18:01:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:08 smithi181 ceph-mon[94189]: pgmap v6368: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 144 KiB/s wr, 27 op/s 2023-03-09T18:01:08.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10978.373003 secs 2023-03-09T18:01:08.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:08 smithi038 ceph-mon[92422]: pgmap v6368: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 144 KiB/s wr, 27 op/s 2023-03-09T18:01:08.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11440.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:01:10.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:10 smithi122 ceph-mon[93869]: pgmap v6369: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 140 KiB/s wr, 27 op/s 2023-03-09T18:01:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:10 smithi181 ceph-mon[94189]: pgmap v6369: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 140 KiB/s wr, 27 op/s 2023-03-09T18:01:10.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:10 smithi038 ceph-mon[92422]: pgmap v6369: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 140 KiB/s wr, 27 op/s 2023-03-09T18:01:11.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10981.086071 secs 2023-03-09T18:01:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10981.086071 secs 2023-03-09T18:01:11.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10981.086071 secs 2023-03-09T18:01:12.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:12 smithi122 ceph-mon[93869]: pgmap v6370: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 125 KiB/s wr, 24 op/s 2023-03-09T18:01:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:12 smithi181 ceph-mon[94189]: pgmap v6370: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 125 KiB/s wr, 24 op/s 2023-03-09T18:01:12.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:12 smithi038 ceph-mon[92422]: pgmap v6370: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 125 KiB/s wr, 24 op/s 2023-03-09T18:01:13.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10983.373171 secs 2023-03-09T18:01:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10983.373171 secs 2023-03-09T18:01:13.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10983.373171 secs 2023-03-09T18:01:13.964 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11445.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:01:14.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:14 smithi122 ceph-mon[93869]: pgmap v6371: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 99 KiB/s wr, 10 op/s 2023-03-09T18:01:14.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:14 smithi181 ceph-mon[94189]: pgmap v6371: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 99 KiB/s wr, 10 op/s 2023-03-09T18:01:14.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:14 smithi038 ceph-mon[92422]: pgmap v6371: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 99 KiB/s wr, 10 op/s 2023-03-09T18:01:16.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10986.086130 secs 2023-03-09T18:01:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:16 smithi122 ceph-mon[93869]: pgmap v6372: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 35 KiB/s wr, 8 op/s 2023-03-09T18:01:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:16 smithi122 ceph-mon[93869]: osdmap e612: 12 total, 12 up, 12 in 2023-03-09T18:01:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10986.086130 secs 2023-03-09T18:01:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:16 smithi181 ceph-mon[94189]: pgmap v6372: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 35 KiB/s wr, 8 op/s 2023-03-09T18:01:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:16 smithi181 ceph-mon[94189]: osdmap e612: 12 total, 12 up, 12 in 2023-03-09T18:01:16.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10986.086130 secs 2023-03-09T18:01:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:16 smithi038 ceph-mon[92422]: pgmap v6372: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 35 KiB/s wr, 8 op/s 2023-03-09T18:01:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:16 smithi038 ceph-mon[92422]: osdmap e612: 12 total, 12 up, 12 in 2023-03-09T18:01:18.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10988.373312 secs 2023-03-09T18:01:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:18 smithi122 ceph-mon[93869]: pgmap v6374: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 38 KiB/s wr, 8 op/s 2023-03-09T18:01:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10988.373312 secs 2023-03-09T18:01:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:18 smithi181 ceph-mon[94189]: pgmap v6374: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 38 KiB/s wr, 8 op/s 2023-03-09T18:01:18.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10988.373312 secs 2023-03-09T18:01:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:18 smithi038 ceph-mon[92422]: pgmap v6374: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 38 KiB/s wr, 8 op/s 2023-03-09T18:01:18.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11450.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:01:20.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:20 smithi122 ceph-mon[93869]: pgmap v6375: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 26 KiB/s wr, 8 op/s 2023-03-09T18:01:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:20 smithi181 ceph-mon[94189]: pgmap v6375: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 26 KiB/s wr, 8 op/s 2023-03-09T18:01:20.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:20 smithi038 ceph-mon[92422]: pgmap v6375: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 26 KiB/s wr, 8 op/s 2023-03-09T18:01:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10991.086191 secs 2023-03-09T18:01:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10991.086191 secs 2023-03-09T18:01:21.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10991.086191 secs 2023-03-09T18:01:22.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:22 smithi122 ceph-mon[93869]: pgmap v6376: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 15 KiB/s wr, 7 op/s 2023-03-09T18:01:22.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:22 smithi181 ceph-mon[94189]: pgmap v6376: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 15 KiB/s wr, 7 op/s 2023-03-09T18:01:22.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:22 smithi038 ceph-mon[92422]: pgmap v6376: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 15 KiB/s wr, 7 op/s 2023-03-09T18:01:23.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10993.373425 secs 2023-03-09T18:01:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10993.373425 secs 2023-03-09T18:01:23.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10993.373425 secs 2023-03-09T18:01:23.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11455.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:01:24.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:24 smithi122 ceph-mon[93869]: pgmap v6377: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 15 KiB/s wr, 7 op/s 2023-03-09T18:01:24.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:24 smithi181 ceph-mon[94189]: pgmap v6377: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 15 KiB/s wr, 7 op/s 2023-03-09T18:01:24.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:24 smithi038 ceph-mon[92422]: pgmap v6377: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 15 KiB/s wr, 7 op/s 2023-03-09T18:01:26.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10996.086249 secs 2023-03-09T18:01:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:26 smithi122 ceph-mon[93869]: pgmap v6378: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 51 KiB/s wr, 7 op/s 2023-03-09T18:01:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10996.086249 secs 2023-03-09T18:01:26.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:26 smithi181 ceph-mon[94189]: pgmap v6378: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 51 KiB/s wr, 7 op/s 2023-03-09T18:01:26.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10996.086249 secs 2023-03-09T18:01:26.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:26 smithi038 ceph-mon[92422]: pgmap v6378: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 51 KiB/s wr, 7 op/s 2023-03-09T18:01:28.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 10998.373580 secs 2023-03-09T18:01:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:28 smithi122 ceph-mon[93869]: pgmap v6379: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 48 KiB/s wr, 7 op/s 2023-03-09T18:01:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 10998.373580 secs 2023-03-09T18:01:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:28 smithi181 ceph-mon[94189]: pgmap v6379: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 48 KiB/s wr, 7 op/s 2023-03-09T18:01:28.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 10998.373580 secs 2023-03-09T18:01:28.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:28 smithi038 ceph-mon[92422]: pgmap v6379: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 48 KiB/s wr, 7 op/s 2023-03-09T18:01:28.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11460.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:01:30.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:30 smithi122 ceph-mon[93869]: pgmap v6380: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T18:01:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11001.086309 secs 2023-03-09T18:01:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:30 smithi181 ceph-mon[94189]: pgmap v6380: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T18:01:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11001.086309 secs 2023-03-09T18:01:30.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:30 smithi038 ceph-mon[92422]: pgmap v6380: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T18:01:30.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11001.086309 secs 2023-03-09T18:01:32.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:32 smithi122 ceph-mon[93869]: pgmap v6381: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 33 KiB/s wr, 9 op/s 2023-03-09T18:01:32.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:32 smithi181 ceph-mon[94189]: pgmap v6381: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 33 KiB/s wr, 9 op/s 2023-03-09T18:01:32.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:32 smithi038 ceph-mon[92422]: pgmap v6381: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 33 KiB/s wr, 9 op/s 2023-03-09T18:01:33.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11465.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:01:34.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:34 smithi122 ceph-mon[93869]: pgmap v6382: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T18:01:34.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:34 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:01:34.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:34 smithi181 ceph-mon[94189]: pgmap v6382: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T18:01:34.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:34 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:01:34.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:34 smithi038 ceph-mon[92422]: pgmap v6382: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T18:01:34.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:34 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:01:35.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11006.086366 secs 2023-03-09T18:01:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11006.086366 secs 2023-03-09T18:01:35.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11006.086366 secs 2023-03-09T18:01:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:36 smithi122 ceph-mon[93869]: pgmap v6383: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 118 KiB/s wr, 8 op/s 2023-03-09T18:01:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:36 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:01:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:36 smithi181 ceph-mon[94189]: pgmap v6383: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 118 KiB/s wr, 8 op/s 2023-03-09T18:01:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:36 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:01:36.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:36 smithi038 ceph-mon[92422]: pgmap v6383: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 118 KiB/s wr, 8 op/s 2023-03-09T18:01:36.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:36 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:01:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:37 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:01:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:01:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:01:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:01:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:01:37.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:01:37.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:01:37.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:01:37.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:01:37.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:01:37.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:01:37.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:01:37.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:01:37.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:01:37.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:01:37.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:37 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:01:37.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:01:37.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:01:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:01:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:01:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:01:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:01:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:01:37.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:01:37.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:01:37.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:01:37.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:01:37.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:01:37.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:01:37.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:01:37.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:37 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:01:37.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:01:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:01:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:01:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:01:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:01:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:01:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:01:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:01:37.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:01:37.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:01:37.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:01:37.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:01:37.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:01:37.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:01:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:38 smithi122 ceph-mon[93869]: pgmap v6384: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 87 KiB/s wr, 9 op/s 2023-03-09T18:01:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:38 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T18:01:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:38 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi038 to 197.2M 2023-03-09T18:01:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:38 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T18:01:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:38 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi038 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-03-09T18:01:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:38 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:01:38.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:38 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:01:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:38 smithi181 ceph-mon[94189]: pgmap v6384: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 87 KiB/s wr, 9 op/s 2023-03-09T18:01:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:38 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T18:01:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:38 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi038 to 197.2M 2023-03-09T18:01:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:38 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T18:01:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:38 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi038 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-03-09T18:01:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:38 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:01:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:38 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:01:38.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:38 smithi038 ceph-mon[92422]: pgmap v6384: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 87 KiB/s wr, 9 op/s 2023-03-09T18:01:38.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:38 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T18:01:38.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:38 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi038 to 197.2M 2023-03-09T18:01:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:38 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T18:01:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:38 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi038 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-03-09T18:01:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:38 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:01:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:38 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:01:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11470.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:01:39.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:01:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:01:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:01:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:01:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:01:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:01:39.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:01:39.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:01:39.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:01:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:01:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:01:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:01:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:01:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:01:39.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:01:39.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:01:39.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:01:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:01:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:01:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:01:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:01:40.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:40 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T18:01:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:40 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T18:01:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:40 smithi122 ceph-mon[93869]: pgmap v6385: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 87 KiB/s wr, 7 op/s 2023-03-09T18:01:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11011.086428 secs 2023-03-09T18:01:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:40 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T18:01:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:40 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T18:01:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:40 smithi181 ceph-mon[94189]: pgmap v6385: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 87 KiB/s wr, 7 op/s 2023-03-09T18:01:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11011.086428 secs 2023-03-09T18:01:40.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:40 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T18:01:40.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:40 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T18:01:40.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:40 smithi038 ceph-mon[92422]: pgmap v6385: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 87 KiB/s wr, 7 op/s 2023-03-09T18:01:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11011.086428 secs 2023-03-09T18:01:42.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:42 smithi122 ceph-mon[93869]: pgmap v6386: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 154 KiB/s wr, 11 op/s 2023-03-09T18:01:42.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:42 smithi181 ceph-mon[94189]: pgmap v6386: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 154 KiB/s wr, 11 op/s 2023-03-09T18:01:42.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:42 smithi038 ceph-mon[92422]: pgmap v6386: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 154 KiB/s wr, 11 op/s 2023-03-09T18:01:43.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11475.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:01:44.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:44 smithi181 ceph-mon[94189]: pgmap v6387: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 153 KiB/s wr, 9 op/s 2023-03-09T18:01:44.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:44 smithi038 ceph-mon[92422]: pgmap v6387: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 153 KiB/s wr, 9 op/s 2023-03-09T18:01:44.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:44 smithi122 ceph-mon[93869]: pgmap v6387: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 153 KiB/s wr, 9 op/s 2023-03-09T18:01:45.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11016.086487 secs 2023-03-09T18:01:45.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11016.086487 secs 2023-03-09T18:01:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11016.086487 secs 2023-03-09T18:01:46.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:46 smithi038 ceph-mon[92422]: pgmap v6388: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 163 KiB/s wr, 10 op/s 2023-03-09T18:01:46.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:46 smithi122 ceph-mon[93869]: pgmap v6388: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 163 KiB/s wr, 10 op/s 2023-03-09T18:01:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:46 smithi181 ceph-mon[94189]: pgmap v6388: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 163 KiB/s wr, 10 op/s 2023-03-09T18:01:48.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:48 smithi038 ceph-mon[92422]: pgmap v6389: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 175 KiB/s wr, 12 op/s 2023-03-09T18:01:48.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11480.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:01:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:48 smithi122 ceph-mon[93869]: pgmap v6389: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 175 KiB/s wr, 12 op/s 2023-03-09T18:01:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:48 smithi181 ceph-mon[94189]: pgmap v6389: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 175 KiB/s wr, 12 op/s 2023-03-09T18:01:50.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:50 smithi038 ceph-mon[92422]: pgmap v6390: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 174 KiB/s wr, 10 op/s 2023-03-09T18:01:50.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11021.086551 secs 2023-03-09T18:01:50.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:50 smithi122 ceph-mon[93869]: pgmap v6390: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 174 KiB/s wr, 10 op/s 2023-03-09T18:01:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11021.086551 secs 2023-03-09T18:01:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:50 smithi181 ceph-mon[94189]: pgmap v6390: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 174 KiB/s wr, 10 op/s 2023-03-09T18:01:50.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11021.086551 secs 2023-03-09T18:01:52.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:52 smithi122 ceph-mon[93869]: pgmap v6391: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 218 KiB/s wr, 13 op/s 2023-03-09T18:01:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:52 smithi181 ceph-mon[94189]: pgmap v6391: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 218 KiB/s wr, 13 op/s 2023-03-09T18:01:52.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:52 smithi038 ceph-mon[92422]: pgmap v6391: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 218 KiB/s wr, 13 op/s 2023-03-09T18:01:53.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11485.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:01:54.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:54 smithi122 ceph-mon[93869]: pgmap v6392: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 151 KiB/s wr, 10 op/s 2023-03-09T18:01:54.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:54 smithi181 ceph-mon[94189]: pgmap v6392: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 151 KiB/s wr, 10 op/s 2023-03-09T18:01:54.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:54 smithi038 ceph-mon[92422]: pgmap v6392: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 151 KiB/s wr, 10 op/s 2023-03-09T18:01:55.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11026.086611 secs 2023-03-09T18:01:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:55 smithi122 ceph-mon[93869]: pgmap v6393: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 160 KiB/s wr, 10 op/s 2023-03-09T18:01:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11026.086611 secs 2023-03-09T18:01:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:55 smithi181 ceph-mon[94189]: pgmap v6393: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 160 KiB/s wr, 10 op/s 2023-03-09T18:01:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11026.086611 secs 2023-03-09T18:01:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:55 smithi038 ceph-mon[92422]: pgmap v6393: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 160 KiB/s wr, 10 op/s 2023-03-09T18:01:58.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11028.374302 secs 2023-03-09T18:01:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:01:58 smithi122 ceph-mon[93869]: pgmap v6394: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 209 KiB/s wr, 11 op/s 2023-03-09T18:01:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11028.374302 secs 2023-03-09T18:01:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:01:58 smithi181 ceph-mon[94189]: pgmap v6394: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 209 KiB/s wr, 11 op/s 2023-03-09T18:01:58.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11028.374302 secs 2023-03-09T18:01:58.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:01:58 smithi038 ceph-mon[92422]: pgmap v6394: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 209 KiB/s wr, 11 op/s 2023-03-09T18:01:58.793 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11490.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:02:00.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:02:00 smithi122 ceph-mon[93869]: pgmap v6395: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 111 KiB/s wr, 8 op/s 2023-03-09T18:02:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:02:00 smithi181 ceph-mon[94189]: pgmap v6395: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 111 KiB/s wr, 8 op/s 2023-03-09T18:02:00.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:02:00 smithi038 ceph-mon[92422]: pgmap v6395: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 111 KiB/s wr, 8 op/s 2023-03-09T18:02:01.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:02:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11031.086670 secs 2023-03-09T18:02:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:02:01 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:02:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:02:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:02:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:02:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:02:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:02:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11031.086670 secs 2023-03-09T18:02:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:02:01 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:02:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:02:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:02:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:02:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:02:01.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:02:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11031.086670 secs 2023-03-09T18:02:01.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:02:01 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:02:01.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:02:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:02:01.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:02:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:02:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:02:02 smithi122 ceph-mon[93869]: pgmap v6396: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 177 KiB/s wr, 13 op/s 2023-03-09T18:02:02.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:02:02 smithi181 ceph-mon[94189]: pgmap v6396: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 177 KiB/s wr, 13 op/s 2023-03-09T18:02:02.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:02:02 smithi038 ceph-mon[92422]: pgmap v6396: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 177 KiB/s wr, 13 op/s 2023-03-09T18:02:03.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:02:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11033.374429 secs 2023-03-09T18:02:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:02:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11033.374429 secs 2023-03-09T18:02:03.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:02:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11033.374429 secs 2023-03-09T18:02:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11495.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:02:04.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:02:04 smithi122 ceph-mon[93869]: pgmap v6397: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 155 KiB/s wr, 21 op/s 2023-03-09T18:02:04.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:02:04 smithi181 ceph-mon[94189]: pgmap v6397: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 155 KiB/s wr, 21 op/s 2023-03-09T18:02:04.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:02:04 smithi038 ceph-mon[92422]: pgmap v6397: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 155 KiB/s wr, 21 op/s 2023-03-09T18:02:06.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:02:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11036.086735 secs 2023-03-09T18:02:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:02:06 smithi122 ceph-mon[93869]: pgmap v6398: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 155 KiB/s wr, 21 op/s 2023-03-09T18:02:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:02:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11036.086735 secs 2023-03-09T18:02:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:02:06 smithi181 ceph-mon[94189]: pgmap v6398: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 155 KiB/s wr, 21 op/s 2023-03-09T18:02:06.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:02:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11036.086735 secs 2023-03-09T18:02:06.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:02:06 smithi038 ceph-mon[92422]: pgmap v6398: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 155 KiB/s wr, 21 op/s 2023-03-09T18:02:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:02:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11038.374556 secs 2023-03-09T18:02:08.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:02:08 smithi122 ceph-mon[93869]: pgmap v6399: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 187 KiB/s wr, 24 op/s 2023-03-09T18:02:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:02:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11038.374556 secs 2023-03-09T18:02:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:02:08 smithi181 ceph-mon[94189]: pgmap v6399: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 187 KiB/s wr, 24 op/s 2023-03-09T18:02:08.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:02:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11038.374556 secs 2023-03-09T18:02:08.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:02:08 smithi038 ceph-mon[92422]: pgmap v6399: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 187 KiB/s wr, 24 op/s 2023-03-09T18:02:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11500.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:02:10.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:02:10 smithi122 ceph-mon[93869]: pgmap v6400: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 129 KiB/s wr, 21 op/s 2023-03-09T18:02:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:02:10 smithi122 ceph-mon[93869]: osdmap e613: 12 total, 12 up, 12 in 2023-03-09T18:02:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:02:10 smithi181 ceph-mon[94189]: pgmap v6400: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 129 KiB/s wr, 21 op/s 2023-03-09T18:02:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:02:10 smithi181 ceph-mon[94189]: osdmap e613: 12 total, 12 up, 12 in 2023-03-09T18:02:10.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:02:10 smithi038 ceph-mon[92422]: pgmap v6400: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 129 KiB/s wr, 21 op/s 2023-03-09T18:02:10.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:02:10 smithi038 ceph-mon[92422]: osdmap e613: 12 total, 12 up, 12 in 2023-03-09T18:02:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:02:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11041.086795 secs 2023-03-09T18:02:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:02:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11041.086795 secs 2023-03-09T18:02:11.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:02:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11041.086795 secs 2023-03-09T18:02:12.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:02:12 smithi122 ceph-mon[93869]: pgmap v6402: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 107 KiB/s wr, 24 op/s 2023-03-09T18:02:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:02:12 smithi181 ceph-mon[94189]: pgmap v6402: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 107 KiB/s wr, 24 op/s 2023-03-09T18:02:12.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:02:12 smithi038 ceph-mon[92422]: pgmap v6402: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 107 KiB/s wr, 24 op/s 2023-03-09T18:02:13.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:02:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11043.374673 secs 2023-03-09T18:02:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:02:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11043.374673 secs 2023-03-09T18:02:13.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:02:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11043.374673 secs 2023-03-09T18:02:13.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11505.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:02:14.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:02:14 smithi122 ceph-mon[93869]: pgmap v6403: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 81 KiB/s wr, 9 op/s 2023-03-09T18:02:14.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:02:14 smithi181 ceph-mon[94189]: pgmap v6403: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 81 KiB/s wr, 9 op/s 2023-03-09T18:02:14.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:02:14 smithi038 ceph-mon[92422]: pgmap v6403: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 81 KiB/s wr, 9 op/s 2023-03-09T18:02:16.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:02:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11046.086853 secs 2023-03-09T18:02:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:02:16 smithi122 ceph-mon[93869]: pgmap v6404: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 93 KiB/s wr, 9 op/s 2023-03-09T18:02:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:02:16 smithi122 ceph-mon[93869]: osdmap e614: 12 total, 12 up, 12 in 2023-03-09T18:02:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:02:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11046.086853 secs 2023-03-09T18:02:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:02:16 smithi181 ceph-mon[94189]: pgmap v6404: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 93 KiB/s wr, 9 op/s 2023-03-09T18:02:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:02:16 smithi181 ceph-mon[94189]: osdmap e614: 12 total, 12 up, 12 in 2023-03-09T18:02:16.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:02:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11046.086853 secs 2023-03-09T18:02:16.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:02:16 smithi038 ceph-mon[92422]: pgmap v6404: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 93 KiB/s wr, 9 op/s 2023-03-09T18:02:16.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:02:16 smithi038 ceph-mon[92422]: osdmap e614: 12 total, 12 up, 12 in 2023-03-09T18:02:18.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:02:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11048.374816 secs 2023-03-09T18:02:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:02:18 smithi122 ceph-mon[93869]: pgmap v6406: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 55 KiB/s wr, 11 op/s 2023-03-09T18:02:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:02:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11048.374816 secs 2023-03-09T18:02:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:02:18 smithi181 ceph-mon[94189]: pgmap v6406: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 55 KiB/s wr, 11 op/s 2023-03-09T18:02:18.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:02:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11048.374816 secs 2023-03-09T18:02:18.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:02:18 smithi038 ceph-mon[92422]: pgmap v6406: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 55 KiB/s wr, 11 op/s 2023-03-09T18:02:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11510.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:02:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:02:20 smithi122 ceph-mon[93869]: pgmap v6407: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 34 KiB/s wr, 8 op/s 2023-03-09T18:02:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:02:20 smithi181 ceph-mon[94189]: pgmap v6407: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 34 KiB/s wr, 8 op/s 2023-03-09T18:02:20.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:02:20 smithi038 ceph-mon[92422]: pgmap v6407: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 34 KiB/s wr, 8 op/s 2023-03-09T18:02:21.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:02:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11051.086914 secs 2023-03-09T18:02:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:02:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11051.086914 secs 2023-03-09T18:02:21.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:02:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11051.086914 secs 2023-03-09T18:02:22.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:02:22 smithi122 ceph-mon[93869]: pgmap v6408: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 8 op/s 2023-03-09T18:02:22.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:02:22 smithi181 ceph-mon[94189]: pgmap v6408: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 8 op/s 2023-03-09T18:02:22.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:02:22 smithi038 ceph-mon[92422]: pgmap v6408: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 8 op/s 2023-03-09T18:02:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:02:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11053.374912 secs 2023-03-09T18:02:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:02:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11053.374912 secs 2023-03-09T18:02:23.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:02:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11053.374912 secs 2023-03-09T18:02:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11515.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:02:24.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:02:24 smithi122 ceph-mon[93869]: pgmap v6409: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 9 op/s 2023-03-09T18:02:24.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:02:24 smithi181 ceph-mon[94189]: pgmap v6409: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 9 op/s 2023-03-09T18:02:24.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:02:24 smithi038 ceph-mon[92422]: pgmap v6409: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 9 op/s 2023-03-09T18:02:26.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:02:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11056.086981 secs 2023-03-09T18:02:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:02:26 smithi122 ceph-mon[93869]: pgmap v6410: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2023-03-09T18:02:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:02:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11056.086981 secs 2023-03-09T18:02:26.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:02:26 smithi181 ceph-mon[94189]: pgmap v6410: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2023-03-09T18:02:26.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:02:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11056.086981 secs 2023-03-09T18:02:26.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:02:26 smithi038 ceph-mon[92422]: pgmap v6410: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2023-03-09T18:02:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:02:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11058.375004 secs 2023-03-09T18:02:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:02:28 smithi122 ceph-mon[93869]: pgmap v6411: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 KiB/s wr, 8 op/s 2023-03-09T18:02:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:02:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11058.375004 secs 2023-03-09T18:02:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:02:28 smithi181 ceph-mon[94189]: pgmap v6411: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 KiB/s wr, 8 op/s 2023-03-09T18:02:28.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:02:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11058.375004 secs 2023-03-09T18:02:28.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:02:28 smithi038 ceph-mon[92422]: pgmap v6411: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 KiB/s wr, 8 op/s 2023-03-09T18:02:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11520.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:02:30.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:02:30 smithi122 ceph-mon[93869]: pgmap v6412: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T18:02:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:02:30 smithi181 ceph-mon[94189]: pgmap v6412: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T18:02:30.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:02:30 smithi038 ceph-mon[92422]: pgmap v6412: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T18:02:31.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:02:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11061.087039 secs 2023-03-09T18:02:31.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:02:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11061.087039 secs 2023-03-09T18:02:31.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:02:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11061.087039 secs 2023-03-09T18:02:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:02:32 smithi122 ceph-mon[93869]: pgmap v6413: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 32 KiB/s wr, 9 op/s 2023-03-09T18:02:32.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:02:32 smithi181 ceph-mon[94189]: pgmap v6413: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 32 KiB/s wr, 9 op/s 2023-03-09T18:02:32.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:02:32 smithi038 ceph-mon[92422]: pgmap v6413: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 32 KiB/s wr, 9 op/s 2023-03-09T18:02:33.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:02:33 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 11063.375138 secs 2023-03-09T18:02:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:02:33 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 11063.375138 secs 2023-03-09T18:02:33.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:02:33 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 11063.375138 secs 2023-03-09T18:02:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11525.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:02:34.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:02:34 smithi122 ceph-mon[93869]: pgmap v6414: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 32 KiB/s wr, 6 op/s 2023-03-09T18:02:34.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:02:34 smithi181 ceph-mon[94189]: pgmap v6414: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 32 KiB/s wr, 6 op/s 2023-03-09T18:02:34.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:02:34 smithi038 ceph-mon[92422]: pgmap v6414: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 32 KiB/s wr, 6 op/s 2023-03-09T18:02:36.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:02:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11066.087100 secs 2023-03-09T18:02:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:02:36 smithi122 ceph-mon[93869]: pgmap v6415: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 41 KiB/s wr, 6 op/s 2023-03-09T18:02:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:02:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11066.087100 secs 2023-03-09T18:02:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:02:36 smithi181 ceph-mon[94189]: pgmap v6415: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 41 KiB/s wr, 6 op/s 2023-03-09T18:02:36.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:02:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11066.087100 secs 2023-03-09T18:02:36.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:02:36 smithi038 ceph-mon[92422]: pgmap v6415: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 41 KiB/s wr, 6 op/s 2023-03-09T18:02:37.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:02:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:02:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:02:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:02:37.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:02:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:02:37.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:02:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:02:37.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:02:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:02:37.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:02:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:02:38.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:02:38 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 11068.375226 secs 2023-03-09T18:02:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:02:38 smithi122 ceph-mon[93869]: pgmap v6416: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 90 KiB/s wr, 9 op/s 2023-03-09T18:02:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:02:38 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 11068.375226 secs 2023-03-09T18:02:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:02:38 smithi181 ceph-mon[94189]: pgmap v6416: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 90 KiB/s wr, 9 op/s 2023-03-09T18:02:38.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:02:38 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 11068.375226 secs 2023-03-09T18:02:38.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:02:38 smithi038 ceph-mon[92422]: pgmap v6416: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 90 KiB/s wr, 9 op/s 2023-03-09T18:02:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11530.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:02:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:02:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:02:39.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:02:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:02:39.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:02:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:02:40.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:02:40 smithi122 ceph-mon[93869]: pgmap v6417: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 6 op/s 2023-03-09T18:02:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:02:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:02:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:02:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:02:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:02:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:02:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:02:40 smithi181 ceph-mon[94189]: pgmap v6417: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 6 op/s 2023-03-09T18:02:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:02:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:02:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:02:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:02:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:02:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:02:40.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:02:40 smithi038 ceph-mon[92422]: pgmap v6417: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 6 op/s 2023-03-09T18:02:40.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:02:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:02:40.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:02:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:02:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:02:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:02:41.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:02:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11071.087184 secs 2023-03-09T18:02:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:02:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11071.087184 secs 2023-03-09T18:02:41.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:02:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11071.087184 secs 2023-03-09T18:02:42.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:02:42 smithi122 ceph-mon[93869]: pgmap v6418: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 150 KiB/s wr, 11 op/s 2023-03-09T18:02:42.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:02:42 smithi181 ceph-mon[94189]: pgmap v6418: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 150 KiB/s wr, 11 op/s 2023-03-09T18:02:42.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:02:42 smithi038 ceph-mon[92422]: pgmap v6418: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 150 KiB/s wr, 11 op/s 2023-03-09T18:02:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:02:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11073.375306 secs 2023-03-09T18:02:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:02:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11073.375306 secs 2023-03-09T18:02:43.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:02:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11073.375306 secs 2023-03-09T18:02:43.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11535.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:02:44.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:02:44 smithi122 ceph-mon[93869]: pgmap v6419: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 150 KiB/s wr, 9 op/s 2023-03-09T18:02:44.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:02:44 smithi181 ceph-mon[94189]: pgmap v6419: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 150 KiB/s wr, 9 op/s 2023-03-09T18:02:44.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:02:44 smithi038 ceph-mon[92422]: pgmap v6419: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 150 KiB/s wr, 9 op/s 2023-03-09T18:02:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:02:45 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:02:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:02:45 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:02:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:02:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11076.087251 secs 2023-03-09T18:02:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:02:45 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:02:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:02:45 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:02:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:02:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11076.087251 secs 2023-03-09T18:02:45.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:02:45 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:02:45.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:02:45 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:02:45.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:02:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11076.087251 secs 2023-03-09T18:02:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:02:46 smithi181 ceph-mon[94189]: pgmap v6420: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 160 KiB/s wr, 10 op/s 2023-03-09T18:02:46.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:02:46 smithi038 ceph-mon[92422]: pgmap v6420: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 160 KiB/s wr, 10 op/s 2023-03-09T18:02:46.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:02:46 smithi122 ceph-mon[93869]: pgmap v6420: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 160 KiB/s wr, 10 op/s 2023-03-09T18:02:48.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:02:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11078.375418 secs 2023-03-09T18:02:48.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:02:48 smithi038 ceph-mon[92422]: pgmap v6421: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 204 KiB/s wr, 13 op/s 2023-03-09T18:02:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11540.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:02:48.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:02:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11078.375418 secs 2023-03-09T18:02:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:02:48 smithi122 ceph-mon[93869]: pgmap v6421: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 204 KiB/s wr, 13 op/s 2023-03-09T18:02:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:02:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11078.375418 secs 2023-03-09T18:02:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:02:48 smithi181 ceph-mon[94189]: pgmap v6421: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 204 KiB/s wr, 13 op/s 2023-03-09T18:02:50.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:02:50 smithi038 ceph-mon[92422]: pgmap v6422: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 155 KiB/s wr, 10 op/s 2023-03-09T18:02:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:02:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11081.087312 secs 2023-03-09T18:02:50.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:02:50 smithi122 ceph-mon[93869]: pgmap v6422: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 155 KiB/s wr, 10 op/s 2023-03-09T18:02:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:02:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11081.087312 secs 2023-03-09T18:02:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:02:50 smithi181 ceph-mon[94189]: pgmap v6422: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 155 KiB/s wr, 10 op/s 2023-03-09T18:02:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:02:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11081.087312 secs 2023-03-09T18:02:52.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:02:52 smithi038 ceph-mon[92422]: pgmap v6423: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 231 KiB/s wr, 14 op/s 2023-03-09T18:02:52.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:02:52 smithi122 ceph-mon[93869]: pgmap v6423: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 231 KiB/s wr, 14 op/s 2023-03-09T18:02:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:02:52 smithi181 ceph-mon[94189]: pgmap v6423: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 231 KiB/s wr, 14 op/s 2023-03-09T18:02:53.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:02:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11083.375513 secs 2023-03-09T18:02:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11545.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:02:53.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:02:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11083.375513 secs 2023-03-09T18:02:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:02:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11083.375513 secs 2023-03-09T18:02:54.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:02:54 smithi122 ceph-mon[93869]: pgmap v6424: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 139 KiB/s wr, 10 op/s 2023-03-09T18:02:54.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:02:54 smithi181 ceph-mon[94189]: pgmap v6424: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 139 KiB/s wr, 10 op/s 2023-03-09T18:02:54.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:02:54 smithi038 ceph-mon[92422]: pgmap v6424: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 139 KiB/s wr, 10 op/s 2023-03-09T18:02:55.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:02:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11086.087373 secs 2023-03-09T18:02:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:02:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11086.087373 secs 2023-03-09T18:02:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:02:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11086.087373 secs 2023-03-09T18:02:56.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:02:56 smithi122 ceph-mon[93869]: pgmap v6425: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 139 KiB/s wr, 9 op/s 2023-03-09T18:02:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:02:56 smithi181 ceph-mon[94189]: pgmap v6425: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 139 KiB/s wr, 9 op/s 2023-03-09T18:02:56.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:02:56 smithi038 ceph-mon[92422]: pgmap v6425: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 139 KiB/s wr, 9 op/s 2023-03-09T18:02:58.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11550.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:02:58.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:02:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11088.375635 secs 2023-03-09T18:02:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:02:58 smithi122 ceph-mon[93869]: pgmap v6426: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 172 KiB/s wr, 11 op/s 2023-03-09T18:02:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:02:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11088.375635 secs 2023-03-09T18:02:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:02:58 smithi181 ceph-mon[94189]: pgmap v6426: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 172 KiB/s wr, 11 op/s 2023-03-09T18:02:58.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:02:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11088.375635 secs 2023-03-09T18:02:58.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:02:58 smithi038 ceph-mon[92422]: pgmap v6426: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 172 KiB/s wr, 11 op/s 2023-03-09T18:03:00.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:03:00 smithi122 ceph-mon[93869]: pgmap v6427: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 119 KiB/s wr, 8 op/s 2023-03-09T18:03:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:03:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11091.087434 secs 2023-03-09T18:03:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:03:00 smithi181 ceph-mon[94189]: pgmap v6427: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 119 KiB/s wr, 8 op/s 2023-03-09T18:03:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:03:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11091.087434 secs 2023-03-09T18:03:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:03:00 smithi038 ceph-mon[92422]: pgmap v6427: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 119 KiB/s wr, 8 op/s 2023-03-09T18:03:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:03:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11091.087434 secs 2023-03-09T18:03:02.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:03:02 smithi122 ceph-mon[93869]: pgmap v6428: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 142 KiB/s wr, 10 op/s 2023-03-09T18:03:02.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:03:02 smithi181 ceph-mon[94189]: pgmap v6428: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 142 KiB/s wr, 10 op/s 2023-03-09T18:03:02.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:03:02 smithi038 ceph-mon[92422]: pgmap v6428: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 142 KiB/s wr, 10 op/s 2023-03-09T18:03:03.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:03:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11093.375756 secs 2023-03-09T18:03:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:03:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11093.375756 secs 2023-03-09T18:03:03.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:03:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11093.375756 secs 2023-03-09T18:03:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11555.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:03:04.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:03:04 smithi122 ceph-mon[93869]: pgmap v6429: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 93 KiB/s wr, 20 op/s 2023-03-09T18:03:04.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:03:04 smithi181 ceph-mon[94189]: pgmap v6429: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 93 KiB/s wr, 20 op/s 2023-03-09T18:03:04.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:03:04 smithi038 ceph-mon[92422]: pgmap v6429: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 93 KiB/s wr, 20 op/s 2023-03-09T18:03:06.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:03:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11096.087494 secs 2023-03-09T18:03:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:03:06 smithi122 ceph-mon[93869]: pgmap v6430: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 93 KiB/s wr, 19 op/s 2023-03-09T18:03:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:03:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11096.087494 secs 2023-03-09T18:03:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:03:06 smithi181 ceph-mon[94189]: pgmap v6430: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 93 KiB/s wr, 19 op/s 2023-03-09T18:03:06.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:03:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11096.087494 secs 2023-03-09T18:03:06.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:03:06 smithi038 ceph-mon[92422]: pgmap v6430: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 93 KiB/s wr, 19 op/s 2023-03-09T18:03:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:03:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11098.375883 secs 2023-03-09T18:03:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:03:08 smithi122 ceph-mon[93869]: pgmap v6431: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 115 KiB/s wr, 23 op/s 2023-03-09T18:03:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:03:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11098.375883 secs 2023-03-09T18:03:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:03:08 smithi181 ceph-mon[94189]: pgmap v6431: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 115 KiB/s wr, 23 op/s 2023-03-09T18:03:08.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:03:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11098.375883 secs 2023-03-09T18:03:08.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:03:08 smithi038 ceph-mon[92422]: pgmap v6431: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 115 KiB/s wr, 23 op/s 2023-03-09T18:03:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11560.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:03:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:03:10 smithi122 ceph-mon[93869]: pgmap v6432: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 73 KiB/s wr, 21 op/s 2023-03-09T18:03:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:03:10 smithi122 ceph-mon[93869]: osdmap e615: 12 total, 12 up, 12 in 2023-03-09T18:03:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:03:10 smithi181 ceph-mon[94189]: pgmap v6432: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 73 KiB/s wr, 21 op/s 2023-03-09T18:03:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:03:10 smithi181 ceph-mon[94189]: osdmap e615: 12 total, 12 up, 12 in 2023-03-09T18:03:10.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:03:10 smithi038 ceph-mon[92422]: pgmap v6432: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 73 KiB/s wr, 21 op/s 2023-03-09T18:03:10.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:03:10 smithi038 ceph-mon[92422]: osdmap e615: 12 total, 12 up, 12 in 2023-03-09T18:03:11.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:03:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11101.087573 secs 2023-03-09T18:03:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:03:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11101.087573 secs 2023-03-09T18:03:11.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:03:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11101.087573 secs 2023-03-09T18:03:12.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:03:12 smithi122 ceph-mon[93869]: pgmap v6434: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 60 KiB/s wr, 25 op/s 2023-03-09T18:03:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:03:12 smithi181 ceph-mon[94189]: pgmap v6434: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 60 KiB/s wr, 25 op/s 2023-03-09T18:03:12.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:03:12 smithi038 ceph-mon[92422]: pgmap v6434: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 60 KiB/s wr, 25 op/s 2023-03-09T18:03:13.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:03:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11103.375975 secs 2023-03-09T18:03:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:03:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11103.375975 secs 2023-03-09T18:03:13.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:03:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11103.375975 secs 2023-03-09T18:03:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11565.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:03:14.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:03:14 smithi122 ceph-mon[93869]: pgmap v6435: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 52 KiB/s wr, 10 op/s 2023-03-09T18:03:14.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:03:14 smithi181 ceph-mon[94189]: pgmap v6435: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 52 KiB/s wr, 10 op/s 2023-03-09T18:03:14.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:03:14 smithi038 ceph-mon[92422]: pgmap v6435: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 52 KiB/s wr, 10 op/s 2023-03-09T18:03:16.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:03:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11106.087633 secs 2023-03-09T18:03:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:03:16 smithi122 ceph-mon[93869]: pgmap v6436: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 53 KiB/s wr, 10 op/s 2023-03-09T18:03:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:03:16 smithi122 ceph-mon[93869]: osdmap e616: 12 total, 12 up, 12 in 2023-03-09T18:03:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:03:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11106.087633 secs 2023-03-09T18:03:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:03:16 smithi181 ceph-mon[94189]: pgmap v6436: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 53 KiB/s wr, 10 op/s 2023-03-09T18:03:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:03:16 smithi181 ceph-mon[94189]: osdmap e616: 12 total, 12 up, 12 in 2023-03-09T18:03:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:03:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11106.087633 secs 2023-03-09T18:03:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:03:16 smithi038 ceph-mon[92422]: pgmap v6436: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 53 KiB/s wr, 10 op/s 2023-03-09T18:03:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:03:16 smithi038 ceph-mon[92422]: osdmap e616: 12 total, 12 up, 12 in 2023-03-09T18:03:18.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:03:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11108.376133 secs 2023-03-09T18:03:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:03:18 smithi122 ceph-mon[93869]: pgmap v6438: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 32 KiB/s wr, 9 op/s 2023-03-09T18:03:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:03:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11108.376133 secs 2023-03-09T18:03:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:03:18 smithi181 ceph-mon[94189]: pgmap v6438: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 32 KiB/s wr, 9 op/s 2023-03-09T18:03:18.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:03:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11108.376133 secs 2023-03-09T18:03:18.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:03:18 smithi038 ceph-mon[92422]: pgmap v6438: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 32 KiB/s wr, 9 op/s 2023-03-09T18:03:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11570.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:03:20.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:03:20 smithi122 ceph-mon[93869]: pgmap v6439: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 65 KiB/s wr, 7 op/s 2023-03-09T18:03:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:03:20 smithi181 ceph-mon[94189]: pgmap v6439: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 65 KiB/s wr, 7 op/s 2023-03-09T18:03:20.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:03:20 smithi038 ceph-mon[92422]: pgmap v6439: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 65 KiB/s wr, 7 op/s 2023-03-09T18:03:21.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:03:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11111.087700 secs 2023-03-09T18:03:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:03:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11111.087700 secs 2023-03-09T18:03:21.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:03:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11111.087700 secs 2023-03-09T18:03:22.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:03:22 smithi122 ceph-mon[93869]: pgmap v6440: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 64 KiB/s wr, 8 op/s 2023-03-09T18:03:22.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:03:22 smithi181 ceph-mon[94189]: pgmap v6440: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 64 KiB/s wr, 8 op/s 2023-03-09T18:03:22.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:03:22 smithi038 ceph-mon[92422]: pgmap v6440: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 64 KiB/s wr, 8 op/s 2023-03-09T18:03:23.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:03:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11113.376259 secs 2023-03-09T18:03:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:03:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11113.376259 secs 2023-03-09T18:03:23.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:03:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11113.376259 secs 2023-03-09T18:03:23.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11575.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:03:24.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:03:24 smithi122 ceph-mon[93869]: pgmap v6441: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 39 KiB/s wr, 8 op/s 2023-03-09T18:03:24.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:03:24 smithi181 ceph-mon[94189]: pgmap v6441: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 39 KiB/s wr, 8 op/s 2023-03-09T18:03:24.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:03:24 smithi038 ceph-mon[92422]: pgmap v6441: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 39 KiB/s wr, 8 op/s 2023-03-09T18:03:26.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:03:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11116.087762 secs 2023-03-09T18:03:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:03:26 smithi122 ceph-mon[93869]: pgmap v6442: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 39 KiB/s wr, 8 op/s 2023-03-09T18:03:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:03:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11116.087762 secs 2023-03-09T18:03:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:03:26 smithi181 ceph-mon[94189]: pgmap v6442: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 39 KiB/s wr, 8 op/s 2023-03-09T18:03:26.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:03:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11116.087762 secs 2023-03-09T18:03:26.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:03:26 smithi038 ceph-mon[92422]: pgmap v6442: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 39 KiB/s wr, 8 op/s 2023-03-09T18:03:28.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:03:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11118.376374 secs 2023-03-09T18:03:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:03:28 smithi122 ceph-mon[93869]: pgmap v6443: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 KiB/s wr, 7 op/s 2023-03-09T18:03:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:03:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11118.376374 secs 2023-03-09T18:03:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:03:28 smithi181 ceph-mon[94189]: pgmap v6443: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 KiB/s wr, 7 op/s 2023-03-09T18:03:28.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:03:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11118.376374 secs 2023-03-09T18:03:28.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:03:28 smithi038 ceph-mon[92422]: pgmap v6443: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 KiB/s wr, 7 op/s 2023-03-09T18:03:28.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11580.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:03:30.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:03:30 smithi122 ceph-mon[93869]: pgmap v6444: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T18:03:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:03:30 smithi181 ceph-mon[94189]: pgmap v6444: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T18:03:30.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:03:30 smithi038 ceph-mon[92422]: pgmap v6444: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T18:03:31.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:03:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11121.087821 secs 2023-03-09T18:03:31.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:03:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11121.087821 secs 2023-03-09T18:03:31.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:03:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11121.087821 secs 2023-03-09T18:03:32.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:03:32 smithi122 ceph-mon[93869]: pgmap v6445: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 8 op/s 2023-03-09T18:03:32.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:03:32 smithi181 ceph-mon[94189]: pgmap v6445: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 8 op/s 2023-03-09T18:03:32.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:03:32 smithi038 ceph-mon[92422]: pgmap v6445: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 8 op/s 2023-03-09T18:03:33.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:03:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11123.376499 secs 2023-03-09T18:03:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:03:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11123.376499 secs 2023-03-09T18:03:33.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:03:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11123.376499 secs 2023-03-09T18:03:33.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11585.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:03:34.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:03:34 smithi122 ceph-mon[93869]: pgmap v6446: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-09T18:03:34.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:03:34 smithi181 ceph-mon[94189]: pgmap v6446: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-09T18:03:34.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:03:34 smithi038 ceph-mon[92422]: pgmap v6446: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-09T18:03:36.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:03:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11126.087881 secs 2023-03-09T18:03:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:03:36 smithi122 ceph-mon[93869]: pgmap v6447: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 6 op/s 2023-03-09T18:03:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:03:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11126.087881 secs 2023-03-09T18:03:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:03:36 smithi181 ceph-mon[94189]: pgmap v6447: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 6 op/s 2023-03-09T18:03:36.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:03:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11126.087881 secs 2023-03-09T18:03:36.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:03:36 smithi038 ceph-mon[92422]: pgmap v6447: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 6 op/s 2023-03-09T18:03:38.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:03:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11128.376603 secs 2023-03-09T18:03:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:03:38 smithi122 ceph-mon[93869]: pgmap v6448: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 341 B/s wr, 7 op/s 2023-03-09T18:03:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:03:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11128.376603 secs 2023-03-09T18:03:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:03:38 smithi181 ceph-mon[94189]: pgmap v6448: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 341 B/s wr, 7 op/s 2023-03-09T18:03:38.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:03:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11128.376603 secs 2023-03-09T18:03:38.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:03:38 smithi038 ceph-mon[92422]: pgmap v6448: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 341 B/s wr, 7 op/s 2023-03-09T18:03:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11590.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:03:40.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:03:40 smithi122 ceph-mon[93869]: pgmap v6449: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 32 KiB/s wr, 6 op/s 2023-03-09T18:03:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:03:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:03:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:03:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:03:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:03:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:03:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:03:40 smithi181 ceph-mon[94189]: pgmap v6449: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 32 KiB/s wr, 6 op/s 2023-03-09T18:03:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:03:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:03:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:03:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:03:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:03:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:03:40.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:03:40 smithi038 ceph-mon[92422]: pgmap v6449: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 32 KiB/s wr, 6 op/s 2023-03-09T18:03:40.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:03:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:03:40.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:03:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:03:40.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:03:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:03:41.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:03:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11131.087944 secs 2023-03-09T18:03:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:03:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:03:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:03:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:03:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:03:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:03:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:03:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11131.087944 secs 2023-03-09T18:03:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:03:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:03:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:03:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:03:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:03:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:03:41.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:03:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11131.087944 secs 2023-03-09T18:03:41.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:03:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:03:41.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:03:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:03:41.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:03:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:03:42.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:03:42 smithi122 ceph-mon[93869]: pgmap v6450: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 41 KiB/s wr, 8 op/s 2023-03-09T18:03:42.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:03:42 smithi181 ceph-mon[94189]: pgmap v6450: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 41 KiB/s wr, 8 op/s 2023-03-09T18:03:42.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:03:42 smithi038 ceph-mon[92422]: pgmap v6450: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 41 KiB/s wr, 8 op/s 2023-03-09T18:03:43.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:03:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11133.376739 secs 2023-03-09T18:03:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:03:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11133.376739 secs 2023-03-09T18:03:43.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:03:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11133.376739 secs 2023-03-09T18:03:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11595.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:03:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:03:44 smithi122 ceph-mon[93869]: pgmap v6451: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 97 KiB/s wr, 7 op/s 2023-03-09T18:03:44.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:03:44 smithi181 ceph-mon[94189]: pgmap v6451: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 97 KiB/s wr, 7 op/s 2023-03-09T18:03:44.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:03:44 smithi038 ceph-mon[92422]: pgmap v6451: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 97 KiB/s wr, 7 op/s 2023-03-09T18:03:45.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:03:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11136.088021 secs 2023-03-09T18:03:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:03:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11136.088021 secs 2023-03-09T18:03:45.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:03:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11136.088021 secs 2023-03-09T18:03:46.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:03:46 smithi122 ceph-mon[93869]: pgmap v6452: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 106 KiB/s wr, 7 op/s 2023-03-09T18:03:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:03:46 smithi181 ceph-mon[94189]: pgmap v6452: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 106 KiB/s wr, 7 op/s 2023-03-09T18:03:46.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:03:46 smithi038 ceph-mon[92422]: pgmap v6452: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 106 KiB/s wr, 7 op/s 2023-03-09T18:03:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:03:48 smithi181 ceph-mon[94189]: pgmap v6453: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 106 KiB/s wr, 9 op/s 2023-03-09T18:03:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:03:48 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:03:48.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:03:48 smithi038 ceph-mon[92422]: pgmap v6453: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 106 KiB/s wr, 9 op/s 2023-03-09T18:03:48.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:03:48 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:03:48.793 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11600.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:03:48.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:03:48 smithi122 ceph-mon[93869]: pgmap v6453: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 106 KiB/s wr, 9 op/s 2023-03-09T18:03:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:03:48 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:03:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:03:49 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:03:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:03:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:03:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:03:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:03:49.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:03:49 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:03:49.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:03:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:03:49.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:03:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:03:49.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:03:49 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:03:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:03:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:03:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:03:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:03:50.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:03:50 smithi038 ceph-mon[92422]: pgmap v6454: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 191 KiB/s wr, 9 op/s 2023-03-09T18:03:50.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:03:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11141.088097 secs 2023-03-09T18:03:50.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:03:50 smithi122 ceph-mon[93869]: pgmap v6454: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 191 KiB/s wr, 9 op/s 2023-03-09T18:03:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:03:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11141.088097 secs 2023-03-09T18:03:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:03:50 smithi181 ceph-mon[94189]: pgmap v6454: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 191 KiB/s wr, 9 op/s 2023-03-09T18:03:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:03:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11141.088097 secs 2023-03-09T18:03:52.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:03:52 smithi038 ceph-mon[92422]: pgmap v6455: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 168 KiB/s wr, 11 op/s 2023-03-09T18:03:52.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:03:52 smithi122 ceph-mon[93869]: pgmap v6455: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 168 KiB/s wr, 11 op/s 2023-03-09T18:03:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:03:52 smithi181 ceph-mon[94189]: pgmap v6455: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 168 KiB/s wr, 11 op/s 2023-03-09T18:03:53.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11605.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:03:54.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:03:54 smithi038 ceph-mon[92422]: pgmap v6456: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 195 KiB/s wr, 9 op/s 2023-03-09T18:03:54.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:03:54 smithi122 ceph-mon[93869]: pgmap v6456: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 195 KiB/s wr, 9 op/s 2023-03-09T18:03:54.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:03:54 smithi181 ceph-mon[94189]: pgmap v6456: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 195 KiB/s wr, 9 op/s 2023-03-09T18:03:55.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:03:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11146.088184 secs 2023-03-09T18:03:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:03:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11146.088184 secs 2023-03-09T18:03:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:03:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11146.088184 secs 2023-03-09T18:03:56.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:03:56 smithi122 ceph-mon[93869]: pgmap v6457: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 139 KiB/s wr, 9 op/s 2023-03-09T18:03:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:03:56 smithi181 ceph-mon[94189]: pgmap v6457: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 139 KiB/s wr, 9 op/s 2023-03-09T18:03:56.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:03:56 smithi038 ceph-mon[92422]: pgmap v6457: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 139 KiB/s wr, 9 op/s 2023-03-09T18:03:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11610.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:03:58.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:03:58 smithi122 ceph-mon[93869]: pgmap v6458: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 130 KiB/s wr, 10 op/s 2023-03-09T18:03:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:03:58 smithi181 ceph-mon[94189]: pgmap v6458: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 130 KiB/s wr, 10 op/s 2023-03-09T18:03:58.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:03:58 smithi038 ceph-mon[92422]: pgmap v6458: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 130 KiB/s wr, 10 op/s 2023-03-09T18:04:00.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:00 smithi122 ceph-mon[93869]: pgmap v6459: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 207 KiB/s wr, 10 op/s 2023-03-09T18:04:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11151.088243 secs 2023-03-09T18:04:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:00 smithi181 ceph-mon[94189]: pgmap v6459: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 207 KiB/s wr, 10 op/s 2023-03-09T18:04:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11151.088243 secs 2023-03-09T18:04:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:00 smithi038 ceph-mon[92422]: pgmap v6459: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 207 KiB/s wr, 10 op/s 2023-03-09T18:04:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11151.088243 secs 2023-03-09T18:04:02.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:02 smithi122 ceph-mon[93869]: pgmap v6460: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 191 KiB/s wr, 12 op/s 2023-03-09T18:04:02.987 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:02 smithi181 ceph-mon[94189]: pgmap v6460: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 191 KiB/s wr, 12 op/s 2023-03-09T18:04:02.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:02 smithi038 ceph-mon[92422]: pgmap v6460: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 191 KiB/s wr, 12 op/s 2023-03-09T18:04:03.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11615.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:04:03.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11153.377195 secs 2023-03-09T18:04:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:03 smithi122 ceph-mon[93869]: pgmap v6461: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 203 KiB/s wr, 22 op/s 2023-03-09T18:04:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11153.377195 secs 2023-03-09T18:04:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:03 smithi181 ceph-mon[94189]: pgmap v6461: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 203 KiB/s wr, 22 op/s 2023-03-09T18:04:03.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11153.377195 secs 2023-03-09T18:04:03.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:03 smithi038 ceph-mon[92422]: pgmap v6461: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 203 KiB/s wr, 22 op/s 2023-03-09T18:04:04.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:04 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:04:04.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:04 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:04:04.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:04 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:04:04.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:04 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:04:04.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:04 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:04:04.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:04 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:04:04.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:04 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:04:04.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:04 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:04:04.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:04 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:04:05.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11156.088314 secs 2023-03-09T18:04:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:05 smithi122 ceph-mon[93869]: pgmap v6462: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 176 KiB/s wr, 21 op/s 2023-03-09T18:04:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11156.088314 secs 2023-03-09T18:04:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:05 smithi181 ceph-mon[94189]: pgmap v6462: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 176 KiB/s wr, 21 op/s 2023-03-09T18:04:05.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11156.088314 secs 2023-03-09T18:04:05.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:05 smithi038 ceph-mon[92422]: pgmap v6462: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 176 KiB/s wr, 21 op/s 2023-03-09T18:04:08.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11158.377312 secs 2023-03-09T18:04:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:08 smithi122 ceph-mon[93869]: pgmap v6463: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 222 KiB/s wr, 26 op/s 2023-03-09T18:04:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11158.377312 secs 2023-03-09T18:04:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:08 smithi181 ceph-mon[94189]: pgmap v6463: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 222 KiB/s wr, 26 op/s 2023-03-09T18:04:08.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11158.377312 secs 2023-03-09T18:04:08.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:08 smithi038 ceph-mon[92422]: pgmap v6463: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 222 KiB/s wr, 26 op/s 2023-03-09T18:04:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11620.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:04:10.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:10 smithi122 ceph-mon[93869]: pgmap v6464: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 221 KiB/s wr, 24 op/s 2023-03-09T18:04:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:10 smithi122 ceph-mon[93869]: osdmap e617: 12 total, 12 up, 12 in 2023-03-09T18:04:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:10 smithi181 ceph-mon[94189]: pgmap v6464: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 221 KiB/s wr, 24 op/s 2023-03-09T18:04:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:10 smithi181 ceph-mon[94189]: osdmap e617: 12 total, 12 up, 12 in 2023-03-09T18:04:10.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:10 smithi038 ceph-mon[92422]: pgmap v6464: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 221 KiB/s wr, 24 op/s 2023-03-09T18:04:10.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:10 smithi038 ceph-mon[92422]: osdmap e617: 12 total, 12 up, 12 in 2023-03-09T18:04:11.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11161.088390 secs 2023-03-09T18:04:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11161.088390 secs 2023-03-09T18:04:11.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11161.088390 secs 2023-03-09T18:04:12.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:12 smithi122 ceph-mon[93869]: pgmap v6466: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 172 KiB/s wr, 27 op/s 2023-03-09T18:04:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:12 smithi181 ceph-mon[94189]: pgmap v6466: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 172 KiB/s wr, 27 op/s 2023-03-09T18:04:12.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:12 smithi038 ceph-mon[92422]: pgmap v6466: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 172 KiB/s wr, 27 op/s 2023-03-09T18:04:13.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11163.377432 secs 2023-03-09T18:04:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11163.377432 secs 2023-03-09T18:04:13.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11163.377432 secs 2023-03-09T18:04:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11625.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:04:14.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:14 smithi122 ceph-mon[93869]: pgmap v6467: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 146 KiB/s wr, 11 op/s 2023-03-09T18:04:14.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:14 smithi181 ceph-mon[94189]: pgmap v6467: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 146 KiB/s wr, 11 op/s 2023-03-09T18:04:14.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:14 smithi038 ceph-mon[92422]: pgmap v6467: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 146 KiB/s wr, 11 op/s 2023-03-09T18:04:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11166.088449 secs 2023-03-09T18:04:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:16 smithi122 ceph-mon[93869]: pgmap v6468: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 136 KiB/s wr, 11 op/s 2023-03-09T18:04:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:16 smithi122 ceph-mon[93869]: osdmap e618: 12 total, 12 up, 12 in 2023-03-09T18:04:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11166.088449 secs 2023-03-09T18:04:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:16 smithi181 ceph-mon[94189]: pgmap v6468: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 136 KiB/s wr, 11 op/s 2023-03-09T18:04:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:16 smithi181 ceph-mon[94189]: osdmap e618: 12 total, 12 up, 12 in 2023-03-09T18:04:16.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11166.088449 secs 2023-03-09T18:04:16.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:16 smithi038 ceph-mon[92422]: pgmap v6468: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 136 KiB/s wr, 11 op/s 2023-03-09T18:04:16.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:16 smithi038 ceph-mon[92422]: osdmap e618: 12 total, 12 up, 12 in 2023-03-09T18:04:18.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11168.377552 secs 2023-03-09T18:04:18.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:18 smithi122 ceph-mon[93869]: pgmap v6470: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 134 KiB/s wr, 9 op/s 2023-03-09T18:04:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11168.377552 secs 2023-03-09T18:04:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:18 smithi181 ceph-mon[94189]: pgmap v6470: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 134 KiB/s wr, 9 op/s 2023-03-09T18:04:18.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11168.377552 secs 2023-03-09T18:04:18.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:18 smithi038 ceph-mon[92422]: pgmap v6470: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 134 KiB/s wr, 9 op/s 2023-03-09T18:04:18.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11630.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:04:20.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:20 smithi122 ceph-mon[93869]: pgmap v6471: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 109 KiB/s wr, 6 op/s 2023-03-09T18:04:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:20 smithi181 ceph-mon[94189]: pgmap v6471: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 109 KiB/s wr, 6 op/s 2023-03-09T18:04:20.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:20 smithi038 ceph-mon[92422]: pgmap v6471: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 109 KiB/s wr, 6 op/s 2023-03-09T18:04:21.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11171.088510 secs 2023-03-09T18:04:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11171.088510 secs 2023-03-09T18:04:21.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11171.088510 secs 2023-03-09T18:04:22.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:22 smithi122 ceph-mon[93869]: pgmap v6472: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 37 KiB/s wr, 7 op/s 2023-03-09T18:04:22.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:22 smithi181 ceph-mon[94189]: pgmap v6472: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 37 KiB/s wr, 7 op/s 2023-03-09T18:04:22.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:22 smithi038 ceph-mon[92422]: pgmap v6472: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 37 KiB/s wr, 7 op/s 2023-03-09T18:04:23.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11173.377677 secs 2023-03-09T18:04:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11173.377677 secs 2023-03-09T18:04:23.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11173.377677 secs 2023-03-09T18:04:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11635.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:04:24.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:24 smithi122 ceph-mon[93869]: pgmap v6473: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 38 KiB/s wr, 7 op/s 2023-03-09T18:04:24.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:24 smithi181 ceph-mon[94189]: pgmap v6473: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 38 KiB/s wr, 7 op/s 2023-03-09T18:04:24.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:24 smithi038 ceph-mon[92422]: pgmap v6473: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 38 KiB/s wr, 7 op/s 2023-03-09T18:04:26.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11176.088575 secs 2023-03-09T18:04:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:26 smithi122 ceph-mon[93869]: pgmap v6474: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 38 KiB/s wr, 7 op/s 2023-03-09T18:04:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11176.088575 secs 2023-03-09T18:04:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:26 smithi181 ceph-mon[94189]: pgmap v6474: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 38 KiB/s wr, 7 op/s 2023-03-09T18:04:26.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11176.088575 secs 2023-03-09T18:04:26.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:26 smithi038 ceph-mon[92422]: pgmap v6474: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 38 KiB/s wr, 7 op/s 2023-03-09T18:04:28.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11178.377795 secs 2023-03-09T18:04:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:28 smithi122 ceph-mon[93869]: pgmap v6475: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 6 op/s 2023-03-09T18:04:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11178.377795 secs 2023-03-09T18:04:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:28 smithi181 ceph-mon[94189]: pgmap v6475: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 6 op/s 2023-03-09T18:04:28.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11178.377795 secs 2023-03-09T18:04:28.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:28 smithi038 ceph-mon[92422]: pgmap v6475: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 6 op/s 2023-03-09T18:04:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11640.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:04:30.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:30 smithi122 ceph-mon[93869]: pgmap v6476: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 10 KiB/s wr, 6 op/s 2023-03-09T18:04:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:30 smithi181 ceph-mon[94189]: pgmap v6476: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 10 KiB/s wr, 6 op/s 2023-03-09T18:04:30.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:30 smithi038 ceph-mon[92422]: pgmap v6476: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 10 KiB/s wr, 6 op/s 2023-03-09T18:04:31.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11181.088659 secs 2023-03-09T18:04:31.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11181.088659 secs 2023-03-09T18:04:31.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11181.088659 secs 2023-03-09T18:04:32.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:32 smithi122 ceph-mon[93869]: pgmap v6477: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 43 KiB/s wr, 9 op/s 2023-03-09T18:04:32.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:32 smithi181 ceph-mon[94189]: pgmap v6477: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 43 KiB/s wr, 9 op/s 2023-03-09T18:04:32.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:32 smithi038 ceph-mon[92422]: pgmap v6477: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 43 KiB/s wr, 9 op/s 2023-03-09T18:04:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11645.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:04:34.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:34 smithi122 ceph-mon[93869]: pgmap v6478: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 34 KiB/s wr, 7 op/s 2023-03-09T18:04:34.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:34 smithi181 ceph-mon[94189]: pgmap v6478: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 34 KiB/s wr, 7 op/s 2023-03-09T18:04:34.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:34 smithi038 ceph-mon[92422]: pgmap v6478: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 34 KiB/s wr, 7 op/s 2023-03-09T18:04:36.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11186.088719 secs 2023-03-09T18:04:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:36 smithi122 ceph-mon[93869]: pgmap v6479: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 34 KiB/s wr, 7 op/s 2023-03-09T18:04:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:36 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:04:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11186.088719 secs 2023-03-09T18:04:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:36 smithi181 ceph-mon[94189]: pgmap v6479: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 34 KiB/s wr, 7 op/s 2023-03-09T18:04:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:36 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:04:36.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11186.088719 secs 2023-03-09T18:04:36.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:36 smithi038 ceph-mon[92422]: pgmap v6479: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 34 KiB/s wr, 7 op/s 2023-03-09T18:04:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:36 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:04:37.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:37 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:04:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:04:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:04:37.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:37 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:04:37.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:04:37.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:04:37.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:37 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:04:37.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:04:37.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:04:38.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:38 smithi122 ceph-mon[93869]: pgmap v6480: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 109 KiB/s wr, 10 op/s 2023-03-09T18:04:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:38 smithi181 ceph-mon[94189]: pgmap v6480: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 109 KiB/s wr, 10 op/s 2023-03-09T18:04:38.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:38 smithi038 ceph-mon[92422]: pgmap v6480: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 109 KiB/s wr, 10 op/s 2023-03-09T18:04:38.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11650.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:04:40.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:40 smithi122 ceph-mon[93869]: pgmap v6481: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 109 KiB/s wr, 7 op/s 2023-03-09T18:04:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11191.088778 secs 2023-03-09T18:04:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:40 smithi181 ceph-mon[94189]: pgmap v6481: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 109 KiB/s wr, 7 op/s 2023-03-09T18:04:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11191.088778 secs 2023-03-09T18:04:40.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:40 smithi038 ceph-mon[92422]: pgmap v6481: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 109 KiB/s wr, 7 op/s 2023-03-09T18:04:40.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11191.088778 secs 2023-03-09T18:04:41.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:04:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:04:41.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:04:42.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:42 smithi122 ceph-mon[93869]: pgmap v6482: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 202 KiB/s wr, 12 op/s 2023-03-09T18:04:42.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:04:42.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:04:42.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:04:42.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:04:42.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:04:42.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:04:42.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:04:42.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:42 smithi181 ceph-mon[94189]: pgmap v6482: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 202 KiB/s wr, 12 op/s 2023-03-09T18:04:42.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:04:42.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:04:42.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:04:42.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:04:42.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:04:42.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:04:42.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:04:42.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:42 smithi038 ceph-mon[92422]: pgmap v6482: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 202 KiB/s wr, 12 op/s 2023-03-09T18:04:42.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:04:42.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:04:42.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:04:42.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:04:42.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:04:42.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:04:42.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:04:43.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11655.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:04:43.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:43 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 11193.378138 secs 2023-03-09T18:04:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:43 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 11193.378138 secs 2023-03-09T18:04:43.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:43 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 11193.378138 secs 2023-03-09T18:04:44.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:44 smithi122 ceph-mon[93869]: pgmap v6483: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 169 KiB/s wr, 9 op/s 2023-03-09T18:04:44.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:44 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:04:44.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:44 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:04:44.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:44 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:04:44.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:44 smithi181 ceph-mon[94189]: pgmap v6483: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 169 KiB/s wr, 9 op/s 2023-03-09T18:04:44.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:44 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:04:44.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:44 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:04:44.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:44 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:04:44.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:44 smithi038 ceph-mon[92422]: pgmap v6483: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 169 KiB/s wr, 9 op/s 2023-03-09T18:04:44.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:44 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:04:44.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:44 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:04:44.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:44 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:04:45.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11196.088839 secs 2023-03-09T18:04:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11196.088839 secs 2023-03-09T18:04:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11196.088839 secs 2023-03-09T18:04:46.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:46 smithi122 ceph-mon[93869]: pgmap v6484: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 242 KiB/s wr, 10 op/s 2023-03-09T18:04:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:46 smithi181 ceph-mon[94189]: pgmap v6484: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 242 KiB/s wr, 10 op/s 2023-03-09T18:04:46.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:46 smithi038 ceph-mon[92422]: pgmap v6484: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 242 KiB/s wr, 10 op/s 2023-03-09T18:04:47.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:47 smithi122 ceph-mon[93869]: pgmap v6485: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 242 KiB/s wr, 12 op/s 2023-03-09T18:04:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:47 smithi181 ceph-mon[94189]: pgmap v6485: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 242 KiB/s wr, 12 op/s 2023-03-09T18:04:47.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:47 smithi038 ceph-mon[92422]: pgmap v6485: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 242 KiB/s wr, 12 op/s 2023-03-09T18:04:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11660.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:04:48.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:48 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 11198.378290 secs 2023-03-09T18:04:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:48 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 11198.378290 secs 2023-03-09T18:04:48.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:48 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 11198.378290 secs 2023-03-09T18:04:49.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:49 smithi122 ceph-mon[93869]: pgmap v6486: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 206 KiB/s wr, 10 op/s 2023-03-09T18:04:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:49 smithi181 ceph-mon[94189]: pgmap v6486: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 206 KiB/s wr, 10 op/s 2023-03-09T18:04:49.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:49 smithi038 ceph-mon[92422]: pgmap v6486: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 206 KiB/s wr, 10 op/s 2023-03-09T18:04:50.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11201.088920 secs 2023-03-09T18:04:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11201.088920 secs 2023-03-09T18:04:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11201.088920 secs 2023-03-09T18:04:52.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:52 smithi122 ceph-mon[93869]: pgmap v6487: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 215 KiB/s wr, 12 op/s 2023-03-09T18:04:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:52 smithi181 ceph-mon[94189]: pgmap v6487: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 215 KiB/s wr, 12 op/s 2023-03-09T18:04:52.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:52 smithi038 ceph-mon[92422]: pgmap v6487: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 215 KiB/s wr, 12 op/s 2023-03-09T18:04:53.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11203.378413 secs 2023-03-09T18:04:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11203.378413 secs 2023-03-09T18:04:53.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11203.378413 secs 2023-03-09T18:04:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11665.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:04:54.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:54 smithi122 ceph-mon[93869]: pgmap v6488: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 123 KiB/s wr, 8 op/s 2023-03-09T18:04:54.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:54 smithi181 ceph-mon[94189]: pgmap v6488: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 123 KiB/s wr, 8 op/s 2023-03-09T18:04:54.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:54 smithi038 ceph-mon[92422]: pgmap v6488: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 123 KiB/s wr, 8 op/s 2023-03-09T18:04:56.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11206.088987 secs 2023-03-09T18:04:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:56 smithi122 ceph-mon[93869]: pgmap v6489: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 196 KiB/s wr, 10 op/s 2023-03-09T18:04:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11206.088987 secs 2023-03-09T18:04:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:56 smithi181 ceph-mon[94189]: pgmap v6489: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 196 KiB/s wr, 10 op/s 2023-03-09T18:04:56.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11206.088987 secs 2023-03-09T18:04:56.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:56 smithi038 ceph-mon[92422]: pgmap v6489: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 196 KiB/s wr, 10 op/s 2023-03-09T18:04:57.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:04:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:04:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:04:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:04:57.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:04:57.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:04:58.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11208.378530 secs 2023-03-09T18:04:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:04:58 smithi122 ceph-mon[93869]: pgmap v6490: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 122 KiB/s wr, 10 op/s 2023-03-09T18:04:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11208.378530 secs 2023-03-09T18:04:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:04:58 smithi181 ceph-mon[94189]: pgmap v6490: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 122 KiB/s wr, 10 op/s 2023-03-09T18:04:58.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11208.378530 secs 2023-03-09T18:04:58.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:04:58 smithi038 ceph-mon[92422]: pgmap v6490: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 122 KiB/s wr, 10 op/s 2023-03-09T18:04:58.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11670.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:05:00.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:05:00 smithi122 ceph-mon[93869]: pgmap v6491: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 182 KiB/s wr, 9 op/s 2023-03-09T18:05:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:05:00 smithi181 ceph-mon[94189]: pgmap v6491: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 182 KiB/s wr, 9 op/s 2023-03-09T18:05:00.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:05:00 smithi038 ceph-mon[92422]: pgmap v6491: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 182 KiB/s wr, 9 op/s 2023-03-09T18:05:01.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:05:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11211.089048 secs 2023-03-09T18:05:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:05:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11211.089048 secs 2023-03-09T18:05:01.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:05:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11211.089048 secs 2023-03-09T18:05:02.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:05:02 smithi122 ceph-mon[93869]: pgmap v6492: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 151 KiB/s wr, 10 op/s 2023-03-09T18:05:02.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:05:02 smithi181 ceph-mon[94189]: pgmap v6492: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 151 KiB/s wr, 10 op/s 2023-03-09T18:05:02.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:05:02 smithi038 ceph-mon[92422]: pgmap v6492: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 151 KiB/s wr, 10 op/s 2023-03-09T18:05:03.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:05:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11213.378644 secs 2023-03-09T18:05:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:05:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11213.378644 secs 2023-03-09T18:05:03.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:05:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11213.378644 secs 2023-03-09T18:05:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11675.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:05:04.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:05:04 smithi122 ceph-mon[93869]: pgmap v6493: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 163 KiB/s wr, 20 op/s 2023-03-09T18:05:04.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:05:04 smithi181 ceph-mon[94189]: pgmap v6493: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 163 KiB/s wr, 20 op/s 2023-03-09T18:05:04.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:05:04 smithi038 ceph-mon[92422]: pgmap v6493: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 163 KiB/s wr, 20 op/s 2023-03-09T18:05:06.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:05:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11216.089111 secs 2023-03-09T18:05:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:05:06 smithi122 ceph-mon[93869]: pgmap v6494: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 173 KiB/s wr, 21 op/s 2023-03-09T18:05:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:05:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11216.089111 secs 2023-03-09T18:05:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:05:06 smithi181 ceph-mon[94189]: pgmap v6494: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 173 KiB/s wr, 21 op/s 2023-03-09T18:05:06.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:05:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11216.089111 secs 2023-03-09T18:05:06.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:05:06 smithi038 ceph-mon[92422]: pgmap v6494: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 173 KiB/s wr, 21 op/s 2023-03-09T18:05:08.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:05:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11218.378734 secs 2023-03-09T18:05:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:05:08 smithi122 ceph-mon[93869]: pgmap v6495: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 123 KiB/s wr, 23 op/s 2023-03-09T18:05:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:05:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11218.378734 secs 2023-03-09T18:05:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:05:08 smithi181 ceph-mon[94189]: pgmap v6495: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 123 KiB/s wr, 23 op/s 2023-03-09T18:05:08.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:05:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11218.378734 secs 2023-03-09T18:05:08.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:05:08 smithi038 ceph-mon[92422]: pgmap v6495: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 123 KiB/s wr, 23 op/s 2023-03-09T18:05:08.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11680.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:05:10.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:05:10 smithi122 ceph-mon[93869]: pgmap v6496: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 123 KiB/s wr, 21 op/s 2023-03-09T18:05:10.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:05:10 smithi122 ceph-mon[93869]: osdmap e619: 12 total, 12 up, 12 in 2023-03-09T18:05:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:05:10 smithi181 ceph-mon[94189]: pgmap v6496: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 123 KiB/s wr, 21 op/s 2023-03-09T18:05:10.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:05:10 smithi181 ceph-mon[94189]: osdmap e619: 12 total, 12 up, 12 in 2023-03-09T18:05:10.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:05:10 smithi038 ceph-mon[92422]: pgmap v6496: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 123 KiB/s wr, 21 op/s 2023-03-09T18:05:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:05:10 smithi038 ceph-mon[92422]: osdmap e619: 12 total, 12 up, 12 in 2023-03-09T18:05:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:05:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11221.090003 secs 2023-03-09T18:05:11.731 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:05:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11221.090003 secs 2023-03-09T18:05:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:05:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11221.090003 secs 2023-03-09T18:05:12.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:05:12 smithi122 ceph-mon[93869]: pgmap v6498: 129 pgs: 8 active+clean+snaptrim, 29 active+clean+snaptrim_wait, 92 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 66 KiB/s wr, 25 op/s 2023-03-09T18:05:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:05:12 smithi181 ceph-mon[94189]: pgmap v6498: 129 pgs: 8 active+clean+snaptrim, 29 active+clean+snaptrim_wait, 92 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 66 KiB/s wr, 25 op/s 2023-03-09T18:05:12.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:05:12 smithi038 ceph-mon[92422]: pgmap v6498: 129 pgs: 8 active+clean+snaptrim, 29 active+clean+snaptrim_wait, 92 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 66 KiB/s wr, 25 op/s 2023-03-09T18:05:13.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:05:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11223.378801 secs 2023-03-09T18:05:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:05:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11223.378801 secs 2023-03-09T18:05:13.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:05:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11223.378801 secs 2023-03-09T18:05:13.793 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11685.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:05:14.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:05:14 smithi122 ceph-mon[93869]: pgmap v6499: 129 pgs: 8 active+clean+snaptrim, 29 active+clean+snaptrim_wait, 92 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 40 KiB/s wr, 10 op/s 2023-03-09T18:05:14.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:05:14 smithi181 ceph-mon[94189]: pgmap v6499: 129 pgs: 8 active+clean+snaptrim, 29 active+clean+snaptrim_wait, 92 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 40 KiB/s wr, 10 op/s 2023-03-09T18:05:14.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:05:14 smithi038 ceph-mon[92422]: pgmap v6499: 129 pgs: 8 active+clean+snaptrim, 29 active+clean+snaptrim_wait, 92 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 40 KiB/s wr, 10 op/s 2023-03-09T18:05:15.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:05:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11226.090063 secs 2023-03-09T18:05:15.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:05:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11226.090063 secs 2023-03-09T18:05:15.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:05:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11226.090063 secs 2023-03-09T18:05:16.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:05:16 smithi122 ceph-mon[93869]: pgmap v6500: 129 pgs: 8 active+clean+snaptrim, 29 active+clean+snaptrim_wait, 92 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 38 KiB/s wr, 10 op/s 2023-03-09T18:05:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:05:16 smithi181 ceph-mon[94189]: pgmap v6500: 129 pgs: 8 active+clean+snaptrim, 29 active+clean+snaptrim_wait, 92 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 38 KiB/s wr, 10 op/s 2023-03-09T18:05:16.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:05:16 smithi038 ceph-mon[92422]: pgmap v6500: 129 pgs: 8 active+clean+snaptrim, 29 active+clean+snaptrim_wait, 92 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 38 KiB/s wr, 10 op/s 2023-03-09T18:05:18.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:05:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11228.378921 secs 2023-03-09T18:05:18.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:05:18 smithi038 ceph-mon[92422]: pgmap v6501: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 7 op/s 2023-03-09T18:05:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11690.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:05:18.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:05:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11228.378921 secs 2023-03-09T18:05:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:05:18 smithi122 ceph-mon[93869]: pgmap v6501: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 7 op/s 2023-03-09T18:05:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:05:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11228.378921 secs 2023-03-09T18:05:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:05:18 smithi181 ceph-mon[94189]: pgmap v6501: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 7 op/s 2023-03-09T18:05:20.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:05:20 smithi038 ceph-mon[92422]: pgmap v6502: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 8 op/s 2023-03-09T18:05:20.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:05:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11231.090123 secs 2023-03-09T18:05:20.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:05:20 smithi122 ceph-mon[93869]: pgmap v6502: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 8 op/s 2023-03-09T18:05:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:05:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11231.090123 secs 2023-03-09T18:05:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:05:20 smithi181 ceph-mon[94189]: pgmap v6502: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 8 op/s 2023-03-09T18:05:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:05:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11231.090123 secs 2023-03-09T18:05:22.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:05:22 smithi122 ceph-mon[93869]: pgmap v6503: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 11 KiB/s wr, 8 op/s 2023-03-09T18:05:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:05:22 smithi122 ceph-mon[93869]: osdmap e620: 12 total, 12 up, 12 in 2023-03-09T18:05:22.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:05:22 smithi181 ceph-mon[94189]: pgmap v6503: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 11 KiB/s wr, 8 op/s 2023-03-09T18:05:22.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:05:22 smithi181 ceph-mon[94189]: osdmap e620: 12 total, 12 up, 12 in 2023-03-09T18:05:22.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:05:22 smithi038 ceph-mon[92422]: pgmap v6503: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 11 KiB/s wr, 8 op/s 2023-03-09T18:05:22.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:05:22 smithi038 ceph-mon[92422]: osdmap e620: 12 total, 12 up, 12 in 2023-03-09T18:05:23.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:05:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11233.379047 secs 2023-03-09T18:05:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:05:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11233.379047 secs 2023-03-09T18:05:23.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:05:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11233.379047 secs 2023-03-09T18:05:23.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11695.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:05:24.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:05:24 smithi122 ceph-mon[93869]: pgmap v6505: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 8 op/s 2023-03-09T18:05:24.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:05:24 smithi181 ceph-mon[94189]: pgmap v6505: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 8 op/s 2023-03-09T18:05:24.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:05:24 smithi038 ceph-mon[92422]: pgmap v6505: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 8 op/s 2023-03-09T18:05:26.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:05:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11236.090183 secs 2023-03-09T18:05:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:05:26 smithi122 ceph-mon[93869]: pgmap v6506: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 41 KiB/s wr, 8 op/s 2023-03-09T18:05:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:05:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11236.090183 secs 2023-03-09T18:05:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:05:26 smithi181 ceph-mon[94189]: pgmap v6506: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 41 KiB/s wr, 8 op/s 2023-03-09T18:05:26.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:05:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11236.090183 secs 2023-03-09T18:05:26.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:05:26 smithi038 ceph-mon[92422]: pgmap v6506: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 41 KiB/s wr, 8 op/s 2023-03-09T18:05:28.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:05:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11238.379186 secs 2023-03-09T18:05:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:05:28 smithi122 ceph-mon[93869]: pgmap v6507: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 40 KiB/s wr, 7 op/s 2023-03-09T18:05:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:05:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11238.379186 secs 2023-03-09T18:05:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:05:28 smithi181 ceph-mon[94189]: pgmap v6507: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 40 KiB/s wr, 7 op/s 2023-03-09T18:05:28.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:05:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11238.379186 secs 2023-03-09T18:05:28.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:05:28 smithi038 ceph-mon[92422]: pgmap v6507: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 40 KiB/s wr, 7 op/s 2023-03-09T18:05:28.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11700.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:05:30.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:05:30 smithi122 ceph-mon[93869]: pgmap v6508: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 75 KiB/s wr, 7 op/s 2023-03-09T18:05:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:05:30 smithi181 ceph-mon[94189]: pgmap v6508: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 75 KiB/s wr, 7 op/s 2023-03-09T18:05:30.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:05:30 smithi038 ceph-mon[92422]: pgmap v6508: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 75 KiB/s wr, 7 op/s 2023-03-09T18:05:31.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:05:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11241.090242 secs 2023-03-09T18:05:31.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:05:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11241.090242 secs 2023-03-09T18:05:31.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:05:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11241.090242 secs 2023-03-09T18:05:32.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:05:32 smithi122 ceph-mon[93869]: pgmap v6509: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 75 KiB/s wr, 7 op/s 2023-03-09T18:05:32.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:05:32 smithi181 ceph-mon[94189]: pgmap v6509: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 75 KiB/s wr, 7 op/s 2023-03-09T18:05:32.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:05:32 smithi038 ceph-mon[92422]: pgmap v6509: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 75 KiB/s wr, 7 op/s 2023-03-09T18:05:33.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:05:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11243.379319 secs 2023-03-09T18:05:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:05:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11243.379319 secs 2023-03-09T18:05:33.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:05:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11243.379319 secs 2023-03-09T18:05:33.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11705.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:05:34.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:05:34 smithi122 ceph-mon[93869]: pgmap v6510: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 63 KiB/s wr, 6 op/s 2023-03-09T18:05:34.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:05:34 smithi181 ceph-mon[94189]: pgmap v6510: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 63 KiB/s wr, 6 op/s 2023-03-09T18:05:34.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:05:34 smithi038 ceph-mon[92422]: pgmap v6510: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 63 KiB/s wr, 6 op/s 2023-03-09T18:05:36.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:05:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11246.090304 secs 2023-03-09T18:05:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:05:36 smithi122 ceph-mon[93869]: pgmap v6511: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 116 KiB/s wr, 7 op/s 2023-03-09T18:05:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:05:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11246.090304 secs 2023-03-09T18:05:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:05:36 smithi181 ceph-mon[94189]: pgmap v6511: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 116 KiB/s wr, 7 op/s 2023-03-09T18:05:36.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:05:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11246.090304 secs 2023-03-09T18:05:36.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:05:36 smithi038 ceph-mon[92422]: pgmap v6511: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 116 KiB/s wr, 7 op/s 2023-03-09T18:05:38.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:05:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11248.379456 secs 2023-03-09T18:05:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:05:38 smithi122 ceph-mon[93869]: pgmap v6512: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 90 KiB/s wr, 9 op/s 2023-03-09T18:05:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:05:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11248.379456 secs 2023-03-09T18:05:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:05:38 smithi181 ceph-mon[94189]: pgmap v6512: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 90 KiB/s wr, 9 op/s 2023-03-09T18:05:38.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:05:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11248.379456 secs 2023-03-09T18:05:38.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:05:38 smithi038 ceph-mon[92422]: pgmap v6512: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 90 KiB/s wr, 9 op/s 2023-03-09T18:05:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11710.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:05:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:05:40 smithi122 ceph-mon[93869]: pgmap v6513: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 90 KiB/s wr, 7 op/s 2023-03-09T18:05:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:05:40 smithi181 ceph-mon[94189]: pgmap v6513: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 90 KiB/s wr, 7 op/s 2023-03-09T18:05:40.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:05:40 smithi038 ceph-mon[92422]: pgmap v6513: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 90 KiB/s wr, 7 op/s 2023-03-09T18:05:41.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:05:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11251.090366 secs 2023-03-09T18:05:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:05:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11251.090366 secs 2023-03-09T18:05:41.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:05:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11251.090366 secs 2023-03-09T18:05:42.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:05:42 smithi122 ceph-mon[93869]: pgmap v6514: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 153 KiB/s wr, 11 op/s 2023-03-09T18:05:42.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:05:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:05:42.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:05:42 smithi181 ceph-mon[94189]: pgmap v6514: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 153 KiB/s wr, 11 op/s 2023-03-09T18:05:42.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:05:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:05:42.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:05:42 smithi038 ceph-mon[92422]: pgmap v6514: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 153 KiB/s wr, 11 op/s 2023-03-09T18:05:42.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:05:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:05:43.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:05:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11253.379575 secs 2023-03-09T18:05:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:05:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:05:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:05:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:05:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:05:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:05:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:05:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11253.379575 secs 2023-03-09T18:05:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:05:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:05:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:05:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:05:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:05:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:05:43.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:05:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11253.379575 secs 2023-03-09T18:05:43.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:05:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:05:43.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:05:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:05:43.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:05:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:05:43.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11715.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:05:44.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:05:44 smithi122 ceph-mon[93869]: pgmap v6515: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 153 KiB/s wr, 9 op/s 2023-03-09T18:05:44.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:05:44 smithi181 ceph-mon[94189]: pgmap v6515: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 153 KiB/s wr, 9 op/s 2023-03-09T18:05:44.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:05:44 smithi038 ceph-mon[92422]: pgmap v6515: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 153 KiB/s wr, 9 op/s 2023-03-09T18:05:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:05:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11256.090450 secs 2023-03-09T18:05:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:05:46 smithi122 ceph-mon[93869]: pgmap v6516: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 162 KiB/s wr, 10 op/s 2023-03-09T18:05:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:05:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11256.090450 secs 2023-03-09T18:05:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:05:46 smithi181 ceph-mon[94189]: pgmap v6516: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 162 KiB/s wr, 10 op/s 2023-03-09T18:05:46.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:05:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11256.090450 secs 2023-03-09T18:05:46.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:05:46 smithi038 ceph-mon[92422]: pgmap v6516: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 162 KiB/s wr, 10 op/s 2023-03-09T18:05:48.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:05:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11258.379697 secs 2023-03-09T18:05:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:05:48 smithi122 ceph-mon[93869]: pgmap v6517: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 163 KiB/s wr, 12 op/s 2023-03-09T18:05:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:05:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11258.379697 secs 2023-03-09T18:05:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:05:48 smithi181 ceph-mon[94189]: pgmap v6517: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 163 KiB/s wr, 12 op/s 2023-03-09T18:05:48.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:05:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11258.379697 secs 2023-03-09T18:05:48.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:05:48 smithi038 ceph-mon[92422]: pgmap v6517: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 163 KiB/s wr, 12 op/s 2023-03-09T18:05:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11720.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:05:50.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:05:50 smithi122 ceph-mon[93869]: pgmap v6518: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 157 KiB/s wr, 10 op/s 2023-03-09T18:05:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:05:50 smithi181 ceph-mon[94189]: pgmap v6518: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 157 KiB/s wr, 10 op/s 2023-03-09T18:05:50.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:05:50 smithi038 ceph-mon[92422]: pgmap v6518: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 157 KiB/s wr, 10 op/s 2023-03-09T18:05:51.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:05:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11261.090511 secs 2023-03-09T18:05:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:05:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11261.090511 secs 2023-03-09T18:05:51.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:05:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11261.090511 secs 2023-03-09T18:05:52.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:05:52 smithi122 ceph-mon[93869]: pgmap v6519: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 227 KiB/s wr, 13 op/s 2023-03-09T18:05:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:05:52 smithi181 ceph-mon[94189]: pgmap v6519: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 227 KiB/s wr, 13 op/s 2023-03-09T18:05:52.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:05:52 smithi038 ceph-mon[92422]: pgmap v6519: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 227 KiB/s wr, 13 op/s 2023-03-09T18:05:53.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:05:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11263.379837 secs 2023-03-09T18:05:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:05:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11263.379837 secs 2023-03-09T18:05:53.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:05:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11263.379837 secs 2023-03-09T18:05:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11725.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:05:54.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:05:54 smithi122 ceph-mon[93869]: pgmap v6520: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 135 KiB/s wr, 9 op/s 2023-03-09T18:05:54.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:05:54 smithi181 ceph-mon[94189]: pgmap v6520: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 135 KiB/s wr, 9 op/s 2023-03-09T18:05:54.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:05:54 smithi038 ceph-mon[92422]: pgmap v6520: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 135 KiB/s wr, 9 op/s 2023-03-09T18:05:55.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:05:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11266.090589 secs 2023-03-09T18:05:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:05:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11266.090589 secs 2023-03-09T18:05:55.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:05:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11266.090589 secs 2023-03-09T18:05:56.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:05:56 smithi122 ceph-mon[93869]: pgmap v6521: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 143 KiB/s wr, 9 op/s 2023-03-09T18:05:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:05:56 smithi181 ceph-mon[94189]: pgmap v6521: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 143 KiB/s wr, 9 op/s 2023-03-09T18:05:56.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:05:56 smithi038 ceph-mon[92422]: pgmap v6521: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 143 KiB/s wr, 9 op/s 2023-03-09T18:05:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:05:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11268.379956 secs 2023-03-09T18:05:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:05:58 smithi181 ceph-mon[94189]: pgmap v6522: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 227 KiB/s wr, 11 op/s 2023-03-09T18:05:58.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:05:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11268.379956 secs 2023-03-09T18:05:58.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:05:58 smithi038 ceph-mon[92422]: pgmap v6522: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 227 KiB/s wr, 11 op/s 2023-03-09T18:05:58.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11730.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:05:58.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:05:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11268.379956 secs 2023-03-09T18:05:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:05:58 smithi122 ceph-mon[93869]: pgmap v6522: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 227 KiB/s wr, 11 op/s 2023-03-09T18:06:00.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:06:00 smithi038 ceph-mon[92422]: pgmap v6523: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 172 KiB/s wr, 8 op/s 2023-03-09T18:06:00.780 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:06:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11271.090649 secs 2023-03-09T18:06:00.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:06:00 smithi122 ceph-mon[93869]: pgmap v6523: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 172 KiB/s wr, 8 op/s 2023-03-09T18:06:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:06:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11271.090649 secs 2023-03-09T18:06:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:06:00 smithi181 ceph-mon[94189]: pgmap v6523: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 172 KiB/s wr, 8 op/s 2023-03-09T18:06:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:06:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11271.090649 secs 2023-03-09T18:06:02.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:06:02 smithi038 ceph-mon[92422]: pgmap v6524: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 219 KiB/s wr, 11 op/s 2023-03-09T18:06:02.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:06:02 smithi122 ceph-mon[93869]: pgmap v6524: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 219 KiB/s wr, 11 op/s 2023-03-09T18:06:02.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:06:02 smithi181 ceph-mon[94189]: pgmap v6524: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 219 KiB/s wr, 11 op/s 2023-03-09T18:06:03.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:06:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11273.380077 secs 2023-03-09T18:06:03.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11735.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:06:03.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:06:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11273.380077 secs 2023-03-09T18:06:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:06:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11273.380077 secs 2023-03-09T18:06:04.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:06:04 smithi122 ceph-mon[93869]: pgmap v6525: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 204 KiB/s wr, 21 op/s 2023-03-09T18:06:04.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:06:04 smithi181 ceph-mon[94189]: pgmap v6525: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 204 KiB/s wr, 21 op/s 2023-03-09T18:06:04.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:06:04 smithi038 ceph-mon[92422]: pgmap v6525: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 204 KiB/s wr, 21 op/s 2023-03-09T18:06:05.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:06:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11276.090709 secs 2023-03-09T18:06:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:06:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11276.090709 secs 2023-03-09T18:06:05.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:06:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11276.090709 secs 2023-03-09T18:06:06.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:06:06 smithi122 ceph-mon[93869]: pgmap v6526: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 204 KiB/s wr, 21 op/s 2023-03-09T18:06:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:06:06 smithi181 ceph-mon[94189]: pgmap v6526: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 204 KiB/s wr, 21 op/s 2023-03-09T18:06:06.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:06:06 smithi038 ceph-mon[92422]: pgmap v6526: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 204 KiB/s wr, 21 op/s 2023-03-09T18:06:08.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11740.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:06:08.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:06:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11278.380178 secs 2023-03-09T18:06:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:06:08 smithi122 ceph-mon[93869]: pgmap v6527: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 239 KiB/s wr, 26 op/s 2023-03-09T18:06:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:06:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11278.380178 secs 2023-03-09T18:06:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:06:08 smithi181 ceph-mon[94189]: pgmap v6527: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 239 KiB/s wr, 26 op/s 2023-03-09T18:06:08.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:06:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11278.380178 secs 2023-03-09T18:06:08.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:06:08 smithi038 ceph-mon[92422]: pgmap v6527: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 239 KiB/s wr, 26 op/s 2023-03-09T18:06:10.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:06:10 smithi122 ceph-mon[93869]: pgmap v6528: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 147 KiB/s wr, 23 op/s 2023-03-09T18:06:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:06:10 smithi122 ceph-mon[93869]: osdmap e621: 12 total, 12 up, 12 in 2023-03-09T18:06:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:06:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11281.090772 secs 2023-03-09T18:06:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:06:10 smithi181 ceph-mon[94189]: pgmap v6528: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 147 KiB/s wr, 23 op/s 2023-03-09T18:06:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:06:10 smithi181 ceph-mon[94189]: osdmap e621: 12 total, 12 up, 12 in 2023-03-09T18:06:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:06:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11281.090772 secs 2023-03-09T18:06:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:06:10 smithi038 ceph-mon[92422]: pgmap v6528: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 147 KiB/s wr, 23 op/s 2023-03-09T18:06:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:06:10 smithi038 ceph-mon[92422]: osdmap e621: 12 total, 12 up, 12 in 2023-03-09T18:06:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:06:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11281.090772 secs 2023-03-09T18:06:12.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:06:12 smithi122 ceph-mon[93869]: pgmap v6530: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 130 KiB/s wr, 26 op/s 2023-03-09T18:06:12.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:06:12 smithi181 ceph-mon[94189]: pgmap v6530: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 130 KiB/s wr, 26 op/s 2023-03-09T18:06:12.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:06:12 smithi038 ceph-mon[92422]: pgmap v6530: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 130 KiB/s wr, 26 op/s 2023-03-09T18:06:13.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11745.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:06:13.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:06:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11283.380241 secs 2023-03-09T18:06:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:06:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11283.380241 secs 2023-03-09T18:06:13.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:06:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11283.380241 secs 2023-03-09T18:06:14.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:06:14 smithi122 ceph-mon[93869]: pgmap v6531: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 86 KiB/s wr, 11 op/s 2023-03-09T18:06:14.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:06:14 smithi181 ceph-mon[94189]: pgmap v6531: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 86 KiB/s wr, 11 op/s 2023-03-09T18:06:14.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:06:14 smithi038 ceph-mon[92422]: pgmap v6531: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 86 KiB/s wr, 11 op/s 2023-03-09T18:06:15.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:06:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11286.090832 secs 2023-03-09T18:06:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:06:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11286.090832 secs 2023-03-09T18:06:15.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:06:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11286.090832 secs 2023-03-09T18:06:16.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:06:16 smithi122 ceph-mon[93869]: pgmap v6532: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 97 KiB/s wr, 11 op/s 2023-03-09T18:06:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:06:16 smithi122 ceph-mon[93869]: osdmap e622: 12 total, 12 up, 12 in 2023-03-09T18:06:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:06:16 smithi181 ceph-mon[94189]: pgmap v6532: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 97 KiB/s wr, 11 op/s 2023-03-09T18:06:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:06:16 smithi181 ceph-mon[94189]: osdmap e622: 12 total, 12 up, 12 in 2023-03-09T18:06:16.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:06:16 smithi038 ceph-mon[92422]: pgmap v6532: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 97 KiB/s wr, 11 op/s 2023-03-09T18:06:16.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:06:16 smithi038 ceph-mon[92422]: osdmap e622: 12 total, 12 up, 12 in 2023-03-09T18:06:17.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:06:17 smithi122 ceph-mon[93869]: pgmap v6534: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 57 KiB/s wr, 11 op/s 2023-03-09T18:06:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:06:17 smithi181 ceph-mon[94189]: pgmap v6534: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 57 KiB/s wr, 11 op/s 2023-03-09T18:06:17.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:06:17 smithi038 ceph-mon[92422]: pgmap v6534: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 57 KiB/s wr, 11 op/s 2023-03-09T18:06:18.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11750.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:06:18.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:06:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11288.380353 secs 2023-03-09T18:06:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:06:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11288.380353 secs 2023-03-09T18:06:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:06:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11288.380353 secs 2023-03-09T18:06:19.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:06:19 smithi122 ceph-mon[93869]: pgmap v6535: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 36 KiB/s wr, 8 op/s 2023-03-09T18:06:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:06:19 smithi181 ceph-mon[94189]: pgmap v6535: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 36 KiB/s wr, 8 op/s 2023-03-09T18:06:19.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:06:19 smithi038 ceph-mon[92422]: pgmap v6535: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 36 KiB/s wr, 8 op/s 2023-03-09T18:06:20.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:06:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11291.090890 secs 2023-03-09T18:06:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:06:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11291.090890 secs 2023-03-09T18:06:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:06:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11291.090890 secs 2023-03-09T18:06:22.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:06:22 smithi122 ceph-mon[93869]: pgmap v6536: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 35 KiB/s wr, 8 op/s 2023-03-09T18:06:22.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:06:22 smithi181 ceph-mon[94189]: pgmap v6536: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 35 KiB/s wr, 8 op/s 2023-03-09T18:06:22.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:06:22 smithi038 ceph-mon[92422]: pgmap v6536: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 35 KiB/s wr, 8 op/s 2023-03-09T18:06:23.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:06:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11293.380470 secs 2023-03-09T18:06:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:06:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11293.380470 secs 2023-03-09T18:06:23.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:06:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11293.380470 secs 2023-03-09T18:06:23.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11755.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:06:24.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:06:24 smithi122 ceph-mon[93869]: pgmap v6537: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 7 op/s 2023-03-09T18:06:24.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:06:24 smithi181 ceph-mon[94189]: pgmap v6537: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 7 op/s 2023-03-09T18:06:24.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:06:24 smithi038 ceph-mon[92422]: pgmap v6537: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 7 op/s 2023-03-09T18:06:26.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:06:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11296.090950 secs 2023-03-09T18:06:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:06:26 smithi122 ceph-mon[93869]: pgmap v6538: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.9 KiB/s wr, 7 op/s 2023-03-09T18:06:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:06:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11296.090950 secs 2023-03-09T18:06:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:06:26 smithi181 ceph-mon[94189]: pgmap v6538: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.9 KiB/s wr, 7 op/s 2023-03-09T18:06:26.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:06:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11296.090950 secs 2023-03-09T18:06:26.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:06:26 smithi038 ceph-mon[92422]: pgmap v6538: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.9 KiB/s wr, 7 op/s 2023-03-09T18:06:28.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:06:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11298.380600 secs 2023-03-09T18:06:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:06:28 smithi122 ceph-mon[93869]: pgmap v6539: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 37 KiB/s wr, 7 op/s 2023-03-09T18:06:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:06:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11298.380600 secs 2023-03-09T18:06:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:06:28 smithi181 ceph-mon[94189]: pgmap v6539: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 37 KiB/s wr, 7 op/s 2023-03-09T18:06:28.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:06:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11298.380600 secs 2023-03-09T18:06:28.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:06:28 smithi038 ceph-mon[92422]: pgmap v6539: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 37 KiB/s wr, 7 op/s 2023-03-09T18:06:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11760.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:06:30.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:06:30 smithi122 ceph-mon[93869]: pgmap v6540: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 33 KiB/s wr, 6 op/s 2023-03-09T18:06:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:06:30 smithi181 ceph-mon[94189]: pgmap v6540: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 33 KiB/s wr, 6 op/s 2023-03-09T18:06:30.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:06:30 smithi038 ceph-mon[92422]: pgmap v6540: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 33 KiB/s wr, 6 op/s 2023-03-09T18:06:31.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:06:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11301.090988 secs 2023-03-09T18:06:31.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:06:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11301.090988 secs 2023-03-09T18:06:31.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:06:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11301.090988 secs 2023-03-09T18:06:32.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:06:32 smithi122 ceph-mon[93869]: pgmap v6541: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 33 KiB/s wr, 8 op/s 2023-03-09T18:06:32.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:06:32 smithi181 ceph-mon[94189]: pgmap v6541: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 33 KiB/s wr, 8 op/s 2023-03-09T18:06:32.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:06:32 smithi038 ceph-mon[92422]: pgmap v6541: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 33 KiB/s wr, 8 op/s 2023-03-09T18:06:33.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:06:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11303.380736 secs 2023-03-09T18:06:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:06:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11303.380736 secs 2023-03-09T18:06:33.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:06:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11303.380736 secs 2023-03-09T18:06:33.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11765.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:06:34.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:06:34 smithi122 ceph-mon[93869]: pgmap v6542: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 33 KiB/s wr, 6 op/s 2023-03-09T18:06:34.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:06:34 smithi181 ceph-mon[94189]: pgmap v6542: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 33 KiB/s wr, 6 op/s 2023-03-09T18:06:34.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:06:34 smithi038 ceph-mon[92422]: pgmap v6542: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 33 KiB/s wr, 6 op/s 2023-03-09T18:06:36.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:06:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11306.091068 secs 2023-03-09T18:06:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:06:36 smithi122 ceph-mon[93869]: pgmap v6543: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 42 KiB/s wr, 6 op/s 2023-03-09T18:06:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:06:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11306.091068 secs 2023-03-09T18:06:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:06:36 smithi181 ceph-mon[94189]: pgmap v6543: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 42 KiB/s wr, 6 op/s 2023-03-09T18:06:36.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:06:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11306.091068 secs 2023-03-09T18:06:36.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:06:36 smithi038 ceph-mon[92422]: pgmap v6543: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 42 KiB/s wr, 6 op/s 2023-03-09T18:06:38.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:06:38 smithi122 ceph-mon[93869]: pgmap v6544: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 93 KiB/s wr, 9 op/s 2023-03-09T18:06:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:06:38 smithi181 ceph-mon[94189]: pgmap v6544: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 93 KiB/s wr, 9 op/s 2023-03-09T18:06:38.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:06:38 smithi038 ceph-mon[92422]: pgmap v6544: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 93 KiB/s wr, 9 op/s 2023-03-09T18:06:38.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11770.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:06:40.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:06:40 smithi122 ceph-mon[93869]: pgmap v6545: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 61 KiB/s wr, 6 op/s 2023-03-09T18:06:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:06:40 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:06:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:06:40 smithi181 ceph-mon[94189]: pgmap v6545: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 61 KiB/s wr, 6 op/s 2023-03-09T18:06:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:06:40 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:06:40.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:06:40 smithi038 ceph-mon[92422]: pgmap v6545: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 61 KiB/s wr, 6 op/s 2023-03-09T18:06:40.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:06:40 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:06:41.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:06:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11311.091131 secs 2023-03-09T18:06:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:06:41 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:06:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:06:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:06:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:06:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:06:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:06:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11311.091131 secs 2023-03-09T18:06:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:06:41 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:06:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:06:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:06:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:06:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:06:41.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:06:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11311.091131 secs 2023-03-09T18:06:41.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:06:41 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:06:41.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:06:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:06:41.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:06:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:06:42.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:06:42 smithi122 ceph-mon[93869]: pgmap v6546: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 70 KiB/s wr, 9 op/s 2023-03-09T18:06:42.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:06:42 smithi181 ceph-mon[94189]: pgmap v6546: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 70 KiB/s wr, 9 op/s 2023-03-09T18:06:42.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:06:42 smithi038 ceph-mon[92422]: pgmap v6546: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 70 KiB/s wr, 9 op/s 2023-03-09T18:06:43.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:06:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:06:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:06:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:06:43.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:06:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:06:43.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11775.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:06:44.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:06:44 smithi122 ceph-mon[93869]: pgmap v6547: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 132 KiB/s wr, 9 op/s 2023-03-09T18:06:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:06:44 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:06:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:06:44 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:06:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:06:44 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:06:44.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:06:44 smithi181 ceph-mon[94189]: pgmap v6547: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 132 KiB/s wr, 9 op/s 2023-03-09T18:06:44.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:06:44 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:06:44.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:06:44 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:06:44.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:06:44 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:06:44.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:06:44 smithi038 ceph-mon[92422]: pgmap v6547: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 132 KiB/s wr, 9 op/s 2023-03-09T18:06:44.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:06:44 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:06:44.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:06:44 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:06:44.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:06:44 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:06:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:06:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11316.091189 secs 2023-03-09T18:06:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:06:46 smithi122 ceph-mon[93869]: pgmap v6548: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 132 KiB/s wr, 9 op/s 2023-03-09T18:06:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:06:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11316.091189 secs 2023-03-09T18:06:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:06:46 smithi181 ceph-mon[94189]: pgmap v6548: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 132 KiB/s wr, 9 op/s 2023-03-09T18:06:46.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:06:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11316.091189 secs 2023-03-09T18:06:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:06:46 smithi038 ceph-mon[92422]: pgmap v6548: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 132 KiB/s wr, 9 op/s 2023-03-09T18:06:48.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:06:48 smithi122 ceph-mon[93869]: pgmap v6549: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 188 KiB/s wr, 12 op/s 2023-03-09T18:06:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:06:48 smithi181 ceph-mon[94189]: pgmap v6549: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 188 KiB/s wr, 12 op/s 2023-03-09T18:06:48.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:06:48 smithi038 ceph-mon[92422]: pgmap v6549: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 188 KiB/s wr, 12 op/s 2023-03-09T18:06:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11780.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:06:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:06:50 smithi122 ceph-mon[93869]: pgmap v6550: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 147 KiB/s wr, 10 op/s 2023-03-09T18:06:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:06:50 smithi181 ceph-mon[94189]: pgmap v6550: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 147 KiB/s wr, 10 op/s 2023-03-09T18:06:50.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:06:50 smithi038 ceph-mon[92422]: pgmap v6550: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 147 KiB/s wr, 10 op/s 2023-03-09T18:06:51.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:06:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11321.091251 secs 2023-03-09T18:06:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:06:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11321.091251 secs 2023-03-09T18:06:51.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:06:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11321.091251 secs 2023-03-09T18:06:52.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:06:52 smithi122 ceph-mon[93869]: pgmap v6551: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 147 KiB/s wr, 12 op/s 2023-03-09T18:06:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:06:52 smithi181 ceph-mon[94189]: pgmap v6551: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 147 KiB/s wr, 12 op/s 2023-03-09T18:06:52.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:06:52 smithi038 ceph-mon[92422]: pgmap v6551: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 147 KiB/s wr, 12 op/s 2023-03-09T18:06:53.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11785.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:06:54.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:06:54 smithi122 ceph-mon[93869]: pgmap v6552: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 207 KiB/s wr, 11 op/s 2023-03-09T18:06:54.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:06:54 smithi181 ceph-mon[94189]: pgmap v6552: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 207 KiB/s wr, 11 op/s 2023-03-09T18:06:54.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:06:54 smithi038 ceph-mon[92422]: pgmap v6552: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 207 KiB/s wr, 11 op/s 2023-03-09T18:06:55.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:06:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11326.091310 secs 2023-03-09T18:06:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:06:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11326.091310 secs 2023-03-09T18:06:55.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:06:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11326.091310 secs 2023-03-09T18:06:56.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:06:56 smithi122 ceph-mon[93869]: pgmap v6553: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 212 KiB/s wr, 10 op/s 2023-03-09T18:06:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:06:56 smithi181 ceph-mon[94189]: pgmap v6553: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 212 KiB/s wr, 10 op/s 2023-03-09T18:06:56.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:06:56 smithi038 ceph-mon[92422]: pgmap v6553: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 212 KiB/s wr, 10 op/s 2023-03-09T18:06:58.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:06:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11328.381326 secs 2023-03-09T18:06:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:06:58 smithi122 ceph-mon[93869]: pgmap v6554: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 212 KiB/s wr, 12 op/s 2023-03-09T18:06:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:06:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11328.381326 secs 2023-03-09T18:06:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:06:58 smithi181 ceph-mon[94189]: pgmap v6554: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 212 KiB/s wr, 12 op/s 2023-03-09T18:06:58.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:06:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11328.381326 secs 2023-03-09T18:06:58.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:06:58 smithi038 ceph-mon[92422]: pgmap v6554: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 212 KiB/s wr, 12 op/s 2023-03-09T18:06:58.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11790.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:07:00.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:07:00 smithi122 ceph-mon[93869]: pgmap v6555: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 233 KiB/s wr, 10 op/s 2023-03-09T18:07:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:07:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11331.091372 secs 2023-03-09T18:07:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:07:00 smithi181 ceph-mon[94189]: pgmap v6555: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 233 KiB/s wr, 10 op/s 2023-03-09T18:07:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:07:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11331.091372 secs 2023-03-09T18:07:00.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:07:00 smithi038 ceph-mon[92422]: pgmap v6555: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 233 KiB/s wr, 10 op/s 2023-03-09T18:07:00.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:07:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11331.091372 secs 2023-03-09T18:07:01.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:07:01 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:07:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:07:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:07:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:07:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:07:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:07:01 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:07:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:07:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:07:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:07:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:07:01.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:07:01 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:07:01.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:07:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:07:01.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:07:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:07:02.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:07:02 smithi122 ceph-mon[93869]: pgmap v6556: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 246 KiB/s wr, 13 op/s 2023-03-09T18:07:02.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:07:02 smithi181 ceph-mon[94189]: pgmap v6556: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 246 KiB/s wr, 13 op/s 2023-03-09T18:07:02.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:07:02 smithi038 ceph-mon[92422]: pgmap v6556: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 246 KiB/s wr, 13 op/s 2023-03-09T18:07:03.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:07:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11333.381439 secs 2023-03-09T18:07:03.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11795.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:07:03.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:07:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11333.381439 secs 2023-03-09T18:07:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:07:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11333.381439 secs 2023-03-09T18:07:04.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:07:04 smithi122 ceph-mon[93869]: pgmap v6557: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 267 KiB/s wr, 24 op/s 2023-03-09T18:07:04.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:07:04 smithi181 ceph-mon[94189]: pgmap v6557: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 267 KiB/s wr, 24 op/s 2023-03-09T18:07:04.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:07:04 smithi038 ceph-mon[92422]: pgmap v6557: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 267 KiB/s wr, 24 op/s 2023-03-09T18:07:05.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:07:05 smithi122 ceph-mon[93869]: osdmap e623: 12 total, 12 up, 12 in 2023-03-09T18:07:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:07:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11336.091442 secs 2023-03-09T18:07:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:07:05 smithi181 ceph-mon[94189]: osdmap e623: 12 total, 12 up, 12 in 2023-03-09T18:07:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:07:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11336.091442 secs 2023-03-09T18:07:05.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:07:05 smithi038 ceph-mon[92422]: osdmap e623: 12 total, 12 up, 12 in 2023-03-09T18:07:05.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:07:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11336.091442 secs 2023-03-09T18:07:06.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:07:06 smithi122 ceph-mon[93869]: pgmap v6559: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 212 KiB/s wr, 27 op/s 2023-03-09T18:07:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:07:06 smithi181 ceph-mon[94189]: pgmap v6559: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 212 KiB/s wr, 27 op/s 2023-03-09T18:07:06.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:07:06 smithi038 ceph-mon[92422]: pgmap v6559: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 212 KiB/s wr, 27 op/s 2023-03-09T18:07:08.793 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11800.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:07:08.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:07:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11338.381534 secs 2023-03-09T18:07:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:07:08 smithi122 ceph-mon[93869]: pgmap v6560: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 213 KiB/s wr, 27 op/s 2023-03-09T18:07:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:07:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11338.381534 secs 2023-03-09T18:07:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:07:08 smithi181 ceph-mon[94189]: pgmap v6560: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 213 KiB/s wr, 27 op/s 2023-03-09T18:07:08.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:07:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11338.381534 secs 2023-03-09T18:07:08.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:07:08 smithi038 ceph-mon[92422]: pgmap v6560: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 213 KiB/s wr, 27 op/s 2023-03-09T18:07:10.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:07:10 smithi122 ceph-mon[93869]: pgmap v6561: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 145 KiB/s wr, 26 op/s 2023-03-09T18:07:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:07:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11341.091502 secs 2023-03-09T18:07:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:07:10 smithi181 ceph-mon[94189]: pgmap v6561: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 145 KiB/s wr, 26 op/s 2023-03-09T18:07:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:07:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11341.091502 secs 2023-03-09T18:07:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:07:10 smithi038 ceph-mon[92422]: pgmap v6561: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 145 KiB/s wr, 26 op/s 2023-03-09T18:07:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:07:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11341.091502 secs 2023-03-09T18:07:12.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:07:12 smithi122 ceph-mon[93869]: pgmap v6562: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 118 KiB/s wr, 23 op/s 2023-03-09T18:07:12.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:07:12 smithi122 ceph-mon[93869]: osdmap e624: 12 total, 12 up, 12 in 2023-03-09T18:07:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:07:12 smithi181 ceph-mon[94189]: pgmap v6562: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 118 KiB/s wr, 23 op/s 2023-03-09T18:07:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:07:12 smithi181 ceph-mon[94189]: osdmap e624: 12 total, 12 up, 12 in 2023-03-09T18:07:12.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:07:12 smithi038 ceph-mon[92422]: pgmap v6562: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 118 KiB/s wr, 23 op/s 2023-03-09T18:07:12.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:07:12 smithi038 ceph-mon[92422]: osdmap e624: 12 total, 12 up, 12 in 2023-03-09T18:07:13.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:07:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11343.381652 secs 2023-03-09T18:07:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:07:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11343.381652 secs 2023-03-09T18:07:13.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:07:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11343.381652 secs 2023-03-09T18:07:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11805.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:07:14.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:07:14 smithi122 ceph-mon[93869]: pgmap v6564: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 56 KiB/s wr, 8 op/s 2023-03-09T18:07:14.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:07:14 smithi181 ceph-mon[94189]: pgmap v6564: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 56 KiB/s wr, 8 op/s 2023-03-09T18:07:14.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:07:14 smithi038 ceph-mon[92422]: pgmap v6564: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 56 KiB/s wr, 8 op/s 2023-03-09T18:07:16.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:07:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11346.091563 secs 2023-03-09T18:07:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:07:16 smithi122 ceph-mon[93869]: pgmap v6565: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 49 KiB/s wr, 8 op/s 2023-03-09T18:07:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:07:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11346.091563 secs 2023-03-09T18:07:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:07:16 smithi181 ceph-mon[94189]: pgmap v6565: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 49 KiB/s wr, 8 op/s 2023-03-09T18:07:16.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:07:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11346.091563 secs 2023-03-09T18:07:16.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:07:16 smithi038 ceph-mon[92422]: pgmap v6565: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 49 KiB/s wr, 8 op/s 2023-03-09T18:07:18.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:07:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11348.381797 secs 2023-03-09T18:07:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:07:18 smithi122 ceph-mon[93869]: pgmap v6566: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 50 KiB/s wr, 8 op/s 2023-03-09T18:07:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:07:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11348.381797 secs 2023-03-09T18:07:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:07:18 smithi181 ceph-mon[94189]: pgmap v6566: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 50 KiB/s wr, 8 op/s 2023-03-09T18:07:18.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:07:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11348.381797 secs 2023-03-09T18:07:18.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:07:18 smithi038 ceph-mon[92422]: pgmap v6566: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 50 KiB/s wr, 8 op/s 2023-03-09T18:07:18.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11810.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:07:20.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:07:20 smithi122 ceph-mon[93869]: pgmap v6567: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 43 KiB/s wr, 8 op/s 2023-03-09T18:07:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:07:20 smithi181 ceph-mon[94189]: pgmap v6567: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 43 KiB/s wr, 8 op/s 2023-03-09T18:07:20.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:07:20 smithi038 ceph-mon[92422]: pgmap v6567: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 43 KiB/s wr, 8 op/s 2023-03-09T18:07:21.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:07:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11351.091624 secs 2023-03-09T18:07:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:07:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11351.091624 secs 2023-03-09T18:07:21.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:07:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11351.091624 secs 2023-03-09T18:07:22.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:07:22 smithi122 ceph-mon[93869]: pgmap v6568: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 43 KiB/s wr, 7 op/s 2023-03-09T18:07:22.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:07:22 smithi181 ceph-mon[94189]: pgmap v6568: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 43 KiB/s wr, 7 op/s 2023-03-09T18:07:22.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:07:22 smithi038 ceph-mon[92422]: pgmap v6568: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 43 KiB/s wr, 7 op/s 2023-03-09T18:07:23.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:07:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11353.381915 secs 2023-03-09T18:07:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:07:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11353.381915 secs 2023-03-09T18:07:23.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:07:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11353.381915 secs 2023-03-09T18:07:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11815.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:07:24.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:07:24 smithi122 ceph-mon[93869]: pgmap v6569: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 36 KiB/s wr, 7 op/s 2023-03-09T18:07:24.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:07:24 smithi181 ceph-mon[94189]: pgmap v6569: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 36 KiB/s wr, 7 op/s 2023-03-09T18:07:24.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:07:24 smithi038 ceph-mon[92422]: pgmap v6569: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 36 KiB/s wr, 7 op/s 2023-03-09T18:07:26.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:07:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11356.091699 secs 2023-03-09T18:07:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:07:26 smithi122 ceph-mon[93869]: pgmap v6570: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 60 KiB/s wr, 7 op/s 2023-03-09T18:07:26.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:07:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11356.091699 secs 2023-03-09T18:07:26.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:07:26 smithi181 ceph-mon[94189]: pgmap v6570: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 60 KiB/s wr, 7 op/s 2023-03-09T18:07:26.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:07:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11356.091699 secs 2023-03-09T18:07:26.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:07:26 smithi038 ceph-mon[92422]: pgmap v6570: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 60 KiB/s wr, 7 op/s 2023-03-09T18:07:28.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:07:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11358.382051 secs 2023-03-09T18:07:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:07:28 smithi122 ceph-mon[93869]: pgmap v6571: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 60 KiB/s wr, 8 op/s 2023-03-09T18:07:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:07:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11358.382051 secs 2023-03-09T18:07:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:07:28 smithi181 ceph-mon[94189]: pgmap v6571: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 60 KiB/s wr, 8 op/s 2023-03-09T18:07:28.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:07:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11358.382051 secs 2023-03-09T18:07:28.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:07:28 smithi038 ceph-mon[92422]: pgmap v6571: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 60 KiB/s wr, 8 op/s 2023-03-09T18:07:28.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11820.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:07:30.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:07:30 smithi122 ceph-mon[93869]: pgmap v6572: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 68 KiB/s wr, 8 op/s 2023-03-09T18:07:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:07:30 smithi181 ceph-mon[94189]: pgmap v6572: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 68 KiB/s wr, 8 op/s 2023-03-09T18:07:30.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:07:30 smithi038 ceph-mon[92422]: pgmap v6572: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 68 KiB/s wr, 8 op/s 2023-03-09T18:07:31.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:07:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11361.091759 secs 2023-03-09T18:07:31.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:07:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11361.091759 secs 2023-03-09T18:07:31.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:07:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11361.091759 secs 2023-03-09T18:07:32.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:07:32 smithi122 ceph-mon[93869]: pgmap v6573: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 43 KiB/s wr, 8 op/s 2023-03-09T18:07:32.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:07:32 smithi181 ceph-mon[94189]: pgmap v6573: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 43 KiB/s wr, 8 op/s 2023-03-09T18:07:32.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:07:32 smithi038 ceph-mon[92422]: pgmap v6573: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 43 KiB/s wr, 8 op/s 2023-03-09T18:07:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11825.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:07:34.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:07:34 smithi122 ceph-mon[93869]: pgmap v6574: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 43 KiB/s wr, 7 op/s 2023-03-09T18:07:34.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:07:34 smithi181 ceph-mon[94189]: pgmap v6574: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 43 KiB/s wr, 7 op/s 2023-03-09T18:07:34.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:07:34 smithi038 ceph-mon[92422]: pgmap v6574: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 43 KiB/s wr, 7 op/s 2023-03-09T18:07:36.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:07:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11366.091820 secs 2023-03-09T18:07:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:07:36 smithi122 ceph-mon[93869]: pgmap v6575: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 107 KiB/s wr, 7 op/s 2023-03-09T18:07:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:07:36 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:07:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:07:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11366.091820 secs 2023-03-09T18:07:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:07:36 smithi181 ceph-mon[94189]: pgmap v6575: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 107 KiB/s wr, 7 op/s 2023-03-09T18:07:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:07:36 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:07:36.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:07:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11366.091820 secs 2023-03-09T18:07:36.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:07:36 smithi038 ceph-mon[92422]: pgmap v6575: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 107 KiB/s wr, 7 op/s 2023-03-09T18:07:36.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:07:36 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:07:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:07:37 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:07:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:07:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:07:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:07:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:07:37.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:07:37 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:07:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:07:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:07:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:07:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:07:37.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:07:37 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:07:37.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:07:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:07:37.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:07:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:07:38.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:07:38 smithi122 ceph-mon[93869]: pgmap v6576: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 91 KiB/s wr, 9 op/s 2023-03-09T18:07:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:07:38 smithi181 ceph-mon[94189]: pgmap v6576: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 91 KiB/s wr, 9 op/s 2023-03-09T18:07:38.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:07:38 smithi038 ceph-mon[92422]: pgmap v6576: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 91 KiB/s wr, 9 op/s 2023-03-09T18:07:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11830.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:07:40.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:07:40 smithi122 ceph-mon[93869]: pgmap v6577: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 99 KiB/s wr, 8 op/s 2023-03-09T18:07:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:07:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11371.091881 secs 2023-03-09T18:07:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:07:40 smithi181 ceph-mon[94189]: pgmap v6577: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 99 KiB/s wr, 8 op/s 2023-03-09T18:07:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:07:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11371.091881 secs 2023-03-09T18:07:40.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:07:40 smithi038 ceph-mon[92422]: pgmap v6577: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 99 KiB/s wr, 8 op/s 2023-03-09T18:07:40.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:07:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11371.091881 secs 2023-03-09T18:07:42.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:07:42 smithi122 ceph-mon[93869]: pgmap v6578: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 91 KiB/s wr, 9 op/s 2023-03-09T18:07:42.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:07:42 smithi181 ceph-mon[94189]: pgmap v6578: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 91 KiB/s wr, 9 op/s 2023-03-09T18:07:42.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:07:42 smithi038 ceph-mon[92422]: pgmap v6578: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 91 KiB/s wr, 9 op/s 2023-03-09T18:07:43.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:07:43 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 11373.382356 secs 2023-03-09T18:07:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:07:43 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 11373.382356 secs 2023-03-09T18:07:43.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:07:43 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 11373.382356 secs 2023-03-09T18:07:43.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11835.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:07:44.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:07:44 smithi122 ceph-mon[93869]: pgmap v6579: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 170 KiB/s wr, 10 op/s 2023-03-09T18:07:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:07:44 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:07:44.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:07:44 smithi181 ceph-mon[94189]: pgmap v6579: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 170 KiB/s wr, 10 op/s 2023-03-09T18:07:44.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:07:44 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:07:44.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:07:44 smithi038 ceph-mon[92422]: pgmap v6579: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 170 KiB/s wr, 10 op/s 2023-03-09T18:07:44.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:07:44 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:07:45.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:07:45 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:07:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:07:45 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:07:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:07:45 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:07:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:07:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11376.091966 secs 2023-03-09T18:07:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:07:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:07:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:07:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:07:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:07:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:07:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:07:45 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:07:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:07:45 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:07:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:07:45 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:07:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:07:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11376.091966 secs 2023-03-09T18:07:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:07:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:07:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:07:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:07:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:07:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:07:45.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:07:45 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:07:45.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:07:45 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:07:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:07:45 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:07:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:07:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11376.091966 secs 2023-03-09T18:07:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:07:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:07:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:07:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:07:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:07:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:07:46.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:07:46 smithi122 ceph-mon[93869]: pgmap v6580: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 170 KiB/s wr, 10 op/s 2023-03-09T18:07:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:07:46 smithi181 ceph-mon[94189]: pgmap v6580: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 170 KiB/s wr, 10 op/s 2023-03-09T18:07:46.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:07:46 smithi038 ceph-mon[92422]: pgmap v6580: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 170 KiB/s wr, 10 op/s 2023-03-09T18:07:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:07:48 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 11378.382464 secs 2023-03-09T18:07:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:07:48 smithi181 ceph-mon[94189]: pgmap v6581: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 202 KiB/s wr, 12 op/s 2023-03-09T18:07:48.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:07:48 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 11378.382464 secs 2023-03-09T18:07:48.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:07:48 smithi038 ceph-mon[92422]: pgmap v6581: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 202 KiB/s wr, 12 op/s 2023-03-09T18:07:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11840.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:07:48.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:07:48 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 11378.382464 secs 2023-03-09T18:07:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:07:48 smithi122 ceph-mon[93869]: pgmap v6581: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 202 KiB/s wr, 12 op/s 2023-03-09T18:07:50.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:07:50 smithi038 ceph-mon[92422]: pgmap v6582: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 195 KiB/s wr, 10 op/s 2023-03-09T18:07:50.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:07:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11381.091991 secs 2023-03-09T18:07:50.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:07:50 smithi122 ceph-mon[93869]: pgmap v6582: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 195 KiB/s wr, 10 op/s 2023-03-09T18:07:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:07:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11381.091991 secs 2023-03-09T18:07:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:07:50 smithi181 ceph-mon[94189]: pgmap v6582: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 195 KiB/s wr, 10 op/s 2023-03-09T18:07:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:07:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11381.091991 secs 2023-03-09T18:07:52.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:07:52 smithi038 ceph-mon[92422]: pgmap v6583: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 186 KiB/s wr, 11 op/s 2023-03-09T18:07:52.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:07:52 smithi122 ceph-mon[93869]: pgmap v6583: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 186 KiB/s wr, 11 op/s 2023-03-09T18:07:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:07:52 smithi181 ceph-mon[94189]: pgmap v6583: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 186 KiB/s wr, 11 op/s 2023-03-09T18:07:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11845.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:07:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:07:53 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 11383.382582 secs 2023-03-09T18:07:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:07:53 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 11383.382582 secs 2023-03-09T18:07:53.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:07:53 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 11383.382582 secs 2023-03-09T18:07:54.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:07:54 smithi122 ceph-mon[93869]: pgmap v6584: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 240 KiB/s wr, 11 op/s 2023-03-09T18:07:54.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:07:54 smithi181 ceph-mon[94189]: pgmap v6584: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 240 KiB/s wr, 11 op/s 2023-03-09T18:07:54.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:07:54 smithi038 ceph-mon[92422]: pgmap v6584: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 240 KiB/s wr, 11 op/s 2023-03-09T18:07:55.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:07:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11386.092051 secs 2023-03-09T18:07:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:07:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11386.092051 secs 2023-03-09T18:07:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:07:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11386.092051 secs 2023-03-09T18:07:56.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:07:56 smithi122 ceph-mon[93869]: pgmap v6585: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 162 KiB/s wr, 9 op/s 2023-03-09T18:07:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:07:56 smithi181 ceph-mon[94189]: pgmap v6585: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 162 KiB/s wr, 9 op/s 2023-03-09T18:07:56.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:07:56 smithi038 ceph-mon[92422]: pgmap v6585: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 162 KiB/s wr, 9 op/s 2023-03-09T18:07:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11850.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:07:58.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:07:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11388.382712 secs 2023-03-09T18:07:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:07:58 smithi122 ceph-mon[93869]: pgmap v6586: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 224 KiB/s wr, 12 op/s 2023-03-09T18:07:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:07:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11388.382712 secs 2023-03-09T18:07:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:07:58 smithi181 ceph-mon[94189]: pgmap v6586: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 224 KiB/s wr, 12 op/s 2023-03-09T18:07:58.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:07:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11388.382712 secs 2023-03-09T18:07:58.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:07:58 smithi038 ceph-mon[92422]: pgmap v6586: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 224 KiB/s wr, 12 op/s 2023-03-09T18:08:00.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:08:00 smithi122 ceph-mon[93869]: pgmap v6587: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 128 KiB/s wr, 10 op/s 2023-03-09T18:08:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:08:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:08:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:08:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:08:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:08:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11391.092110 secs 2023-03-09T18:08:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:08:00 smithi181 ceph-mon[94189]: pgmap v6587: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 128 KiB/s wr, 10 op/s 2023-03-09T18:08:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:08:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:08:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:08:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:08:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:08:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11391.092110 secs 2023-03-09T18:08:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:08:00 smithi038 ceph-mon[92422]: pgmap v6587: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 128 KiB/s wr, 10 op/s 2023-03-09T18:08:01.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:08:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:08:01.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:08:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:08:01.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:08:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11391.092110 secs 2023-03-09T18:08:02.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:08:02 smithi122 ceph-mon[93869]: pgmap v6588: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 141 KiB/s wr, 12 op/s 2023-03-09T18:08:02.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:08:02 smithi181 ceph-mon[94189]: pgmap v6588: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 141 KiB/s wr, 12 op/s 2023-03-09T18:08:02.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:08:02 smithi038 ceph-mon[92422]: pgmap v6588: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 141 KiB/s wr, 12 op/s 2023-03-09T18:08:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11855.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:08:03.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:08:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11393.382828 secs 2023-03-09T18:08:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:08:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11393.382828 secs 2023-03-09T18:08:03.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:08:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11393.382828 secs 2023-03-09T18:08:04.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:08:04 smithi122 ceph-mon[93869]: pgmap v6589: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 213 KiB/s wr, 24 op/s 2023-03-09T18:08:04.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:08:04 smithi181 ceph-mon[94189]: pgmap v6589: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 213 KiB/s wr, 24 op/s 2023-03-09T18:08:04.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:08:04 smithi038 ceph-mon[92422]: pgmap v6589: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 213 KiB/s wr, 24 op/s 2023-03-09T18:08:05.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:08:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11396.092171 secs 2023-03-09T18:08:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:08:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11396.092171 secs 2023-03-09T18:08:05.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:08:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11396.092171 secs 2023-03-09T18:08:06.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:08:06 smithi122 ceph-mon[93869]: pgmap v6590: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 159 KiB/s wr, 22 op/s 2023-03-09T18:08:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:08:06 smithi181 ceph-mon[94189]: pgmap v6590: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 159 KiB/s wr, 22 op/s 2023-03-09T18:08:06.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:08:06 smithi038 ceph-mon[92422]: pgmap v6590: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 159 KiB/s wr, 22 op/s 2023-03-09T18:08:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11860.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:08:08.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:08:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11398.382949 secs 2023-03-09T18:08:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:08:08 smithi122 ceph-mon[93869]: pgmap v6591: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 159 KiB/s wr, 24 op/s 2023-03-09T18:08:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:08:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11398.382949 secs 2023-03-09T18:08:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:08:08 smithi181 ceph-mon[94189]: pgmap v6591: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 159 KiB/s wr, 24 op/s 2023-03-09T18:08:08.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:08:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11398.382949 secs 2023-03-09T18:08:08.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:08:08 smithi038 ceph-mon[92422]: pgmap v6591: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 159 KiB/s wr, 24 op/s 2023-03-09T18:08:10.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:08:10 smithi122 ceph-mon[93869]: pgmap v6592: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 105 KiB/s wr, 22 op/s 2023-03-09T18:08:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:08:10 smithi122 ceph-mon[93869]: osdmap e625: 12 total, 12 up, 12 in 2023-03-09T18:08:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:08:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11401.092236 secs 2023-03-09T18:08:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:08:10 smithi181 ceph-mon[94189]: pgmap v6592: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 105 KiB/s wr, 22 op/s 2023-03-09T18:08:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:08:10 smithi181 ceph-mon[94189]: osdmap e625: 12 total, 12 up, 12 in 2023-03-09T18:08:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:08:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11401.092236 secs 2023-03-09T18:08:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:08:10 smithi038 ceph-mon[92422]: pgmap v6592: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 105 KiB/s wr, 22 op/s 2023-03-09T18:08:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:08:10 smithi038 ceph-mon[92422]: osdmap e625: 12 total, 12 up, 12 in 2023-03-09T18:08:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:08:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11401.092236 secs 2023-03-09T18:08:12.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:08:12 smithi122 ceph-mon[93869]: pgmap v6594: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 122 KiB/s wr, 23 op/s 2023-03-09T18:08:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:08:12 smithi181 ceph-mon[94189]: pgmap v6594: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 122 KiB/s wr, 23 op/s 2023-03-09T18:08:12.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:08:12 smithi038 ceph-mon[92422]: pgmap v6594: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 122 KiB/s wr, 23 op/s 2023-03-09T18:08:13.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:08:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11403.383056 secs 2023-03-09T18:08:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:08:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11403.383056 secs 2023-03-09T18:08:13.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:08:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11403.383056 secs 2023-03-09T18:08:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11865.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:08:14.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:08:14 smithi122 ceph-mon[93869]: pgmap v6595: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 47 KiB/s wr, 8 op/s 2023-03-09T18:08:14.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:08:14 smithi181 ceph-mon[94189]: pgmap v6595: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 47 KiB/s wr, 8 op/s 2023-03-09T18:08:14.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:08:14 smithi038 ceph-mon[92422]: pgmap v6595: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 47 KiB/s wr, 8 op/s 2023-03-09T18:08:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:08:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11406.092297 secs 2023-03-09T18:08:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:08:16 smithi122 ceph-mon[93869]: pgmap v6596: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 47 KiB/s wr, 8 op/s 2023-03-09T18:08:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:08:16 smithi122 ceph-mon[93869]: osdmap e626: 12 total, 12 up, 12 in 2023-03-09T18:08:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:08:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11406.092297 secs 2023-03-09T18:08:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:08:16 smithi181 ceph-mon[94189]: pgmap v6596: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 47 KiB/s wr, 8 op/s 2023-03-09T18:08:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:08:16 smithi181 ceph-mon[94189]: osdmap e626: 12 total, 12 up, 12 in 2023-03-09T18:08:16.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:08:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11406.092297 secs 2023-03-09T18:08:16.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:08:16 smithi038 ceph-mon[92422]: pgmap v6596: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 47 KiB/s wr, 8 op/s 2023-03-09T18:08:16.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:08:16 smithi038 ceph-mon[92422]: osdmap e626: 12 total, 12 up, 12 in 2023-03-09T18:08:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:08:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11408.383177 secs 2023-03-09T18:08:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:08:18 smithi122 ceph-mon[93869]: pgmap v6598: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 47 KiB/s wr, 10 op/s 2023-03-09T18:08:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:08:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11408.383177 secs 2023-03-09T18:08:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:08:18 smithi181 ceph-mon[94189]: pgmap v6598: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 47 KiB/s wr, 10 op/s 2023-03-09T18:08:18.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:08:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11408.383177 secs 2023-03-09T18:08:18.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:08:18 smithi038 ceph-mon[92422]: pgmap v6598: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 47 KiB/s wr, 10 op/s 2023-03-09T18:08:18.793 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11870.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:08:20.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:08:20 smithi122 ceph-mon[93869]: pgmap v6599: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 39 KiB/s wr, 9 op/s 2023-03-09T18:08:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:08:20 smithi181 ceph-mon[94189]: pgmap v6599: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 39 KiB/s wr, 9 op/s 2023-03-09T18:08:20.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:08:20 smithi038 ceph-mon[92422]: pgmap v6599: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 39 KiB/s wr, 9 op/s 2023-03-09T18:08:21.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:08:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11411.092356 secs 2023-03-09T18:08:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:08:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11411.092356 secs 2023-03-09T18:08:21.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:08:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11411.092356 secs 2023-03-09T18:08:22.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:08:22 smithi122 ceph-mon[93869]: pgmap v6600: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 8 op/s 2023-03-09T18:08:22.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:08:22 smithi181 ceph-mon[94189]: pgmap v6600: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 8 op/s 2023-03-09T18:08:22.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:08:22 smithi038 ceph-mon[92422]: pgmap v6600: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 8 op/s 2023-03-09T18:08:23.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:08:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11413.383303 secs 2023-03-09T18:08:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:08:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11413.383303 secs 2023-03-09T18:08:23.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:08:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11413.383303 secs 2023-03-09T18:08:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11875.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:08:24.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:08:24 smithi122 ceph-mon[93869]: pgmap v6601: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 8 op/s 2023-03-09T18:08:24.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:08:24 smithi181 ceph-mon[94189]: pgmap v6601: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 8 op/s 2023-03-09T18:08:24.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:08:24 smithi038 ceph-mon[92422]: pgmap v6601: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 8 op/s 2023-03-09T18:08:26.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:08:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11416.092415 secs 2023-03-09T18:08:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:08:26 smithi122 ceph-mon[93869]: pgmap v6602: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 8 op/s 2023-03-09T18:08:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:08:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11416.092415 secs 2023-03-09T18:08:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:08:26 smithi181 ceph-mon[94189]: pgmap v6602: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 8 op/s 2023-03-09T18:08:26.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:08:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11416.092415 secs 2023-03-09T18:08:26.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:08:26 smithi038 ceph-mon[92422]: pgmap v6602: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 8 op/s 2023-03-09T18:08:28.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:08:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11418.383437 secs 2023-03-09T18:08:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:08:28 smithi122 ceph-mon[93869]: pgmap v6603: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 47 KiB/s wr, 8 op/s 2023-03-09T18:08:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:08:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11418.383437 secs 2023-03-09T18:08:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:08:28 smithi181 ceph-mon[94189]: pgmap v6603: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 47 KiB/s wr, 8 op/s 2023-03-09T18:08:28.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:08:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11418.383437 secs 2023-03-09T18:08:28.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:08:28 smithi038 ceph-mon[92422]: pgmap v6603: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 47 KiB/s wr, 8 op/s 2023-03-09T18:08:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11880.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:08:30.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:08:30 smithi122 ceph-mon[93869]: pgmap v6604: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 43 KiB/s wr, 8 op/s 2023-03-09T18:08:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:08:30 smithi181 ceph-mon[94189]: pgmap v6604: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 43 KiB/s wr, 8 op/s 2023-03-09T18:08:30.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:08:30 smithi038 ceph-mon[92422]: pgmap v6604: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 43 KiB/s wr, 8 op/s 2023-03-09T18:08:31.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:08:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11421.092477 secs 2023-03-09T18:08:31.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:08:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11421.092477 secs 2023-03-09T18:08:31.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:08:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11421.092477 secs 2023-03-09T18:08:32.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:08:32 smithi122 ceph-mon[93869]: pgmap v6605: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 42 KiB/s wr, 9 op/s 2023-03-09T18:08:32.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:08:32 smithi181 ceph-mon[94189]: pgmap v6605: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 42 KiB/s wr, 9 op/s 2023-03-09T18:08:32.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:08:32 smithi038 ceph-mon[92422]: pgmap v6605: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 42 KiB/s wr, 9 op/s 2023-03-09T18:08:33.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:08:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11423.383572 secs 2023-03-09T18:08:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:08:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11423.383572 secs 2023-03-09T18:08:33.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:08:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11423.383572 secs 2023-03-09T18:08:33.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11885.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:08:34.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:08:34 smithi122 ceph-mon[93869]: pgmap v6606: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 34 KiB/s wr, 8 op/s 2023-03-09T18:08:34.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:08:34 smithi181 ceph-mon[94189]: pgmap v6606: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 34 KiB/s wr, 8 op/s 2023-03-09T18:08:34.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:08:34 smithi038 ceph-mon[92422]: pgmap v6606: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 34 KiB/s wr, 8 op/s 2023-03-09T18:08:35.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:08:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11426.092536 secs 2023-03-09T18:08:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:08:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11426.092536 secs 2023-03-09T18:08:35.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:08:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11426.092536 secs 2023-03-09T18:08:36.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:08:36 smithi122 ceph-mon[93869]: pgmap v6607: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 34 KiB/s wr, 6 op/s 2023-03-09T18:08:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:08:36 smithi181 ceph-mon[94189]: pgmap v6607: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 34 KiB/s wr, 6 op/s 2023-03-09T18:08:36.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:08:36 smithi038 ceph-mon[92422]: pgmap v6607: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 34 KiB/s wr, 6 op/s 2023-03-09T18:08:38.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:08:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11428.383704 secs 2023-03-09T18:08:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:08:38 smithi122 ceph-mon[93869]: pgmap v6608: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 86 KiB/s wr, 9 op/s 2023-03-09T18:08:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:08:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11428.383704 secs 2023-03-09T18:08:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:08:38 smithi181 ceph-mon[94189]: pgmap v6608: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 86 KiB/s wr, 9 op/s 2023-03-09T18:08:38.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:08:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11428.383704 secs 2023-03-09T18:08:38.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:08:38 smithi038 ceph-mon[92422]: pgmap v6608: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 86 KiB/s wr, 9 op/s 2023-03-09T18:08:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11890.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:08:40.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:08:40 smithi038 ceph-mon[92422]: pgmap v6609: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 62 KiB/s wr, 8 op/s 2023-03-09T18:08:40.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:08:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11431.092595 secs 2023-03-09T18:08:40.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:08:40 smithi122 ceph-mon[93869]: pgmap v6609: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 62 KiB/s wr, 8 op/s 2023-03-09T18:08:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:08:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11431.092595 secs 2023-03-09T18:08:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:08:40 smithi181 ceph-mon[94189]: pgmap v6609: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 62 KiB/s wr, 8 op/s 2023-03-09T18:08:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:08:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11431.092595 secs 2023-03-09T18:08:42.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:08:42 smithi038 ceph-mon[92422]: pgmap v6610: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 124 KiB/s wr, 10 op/s 2023-03-09T18:08:42.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:08:42 smithi122 ceph-mon[93869]: pgmap v6610: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 124 KiB/s wr, 10 op/s 2023-03-09T18:08:42.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:08:42 smithi181 ceph-mon[94189]: pgmap v6610: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 124 KiB/s wr, 10 op/s 2023-03-09T18:08:43.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:08:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11433.383824 secs 2023-03-09T18:08:43.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11895.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:08:43.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:08:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11433.383824 secs 2023-03-09T18:08:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:08:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11433.383824 secs 2023-03-09T18:08:44.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:08:44 smithi038 ceph-mon[92422]: pgmap v6611: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 134 KiB/s wr, 9 op/s 2023-03-09T18:08:44.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:08:44 smithi122 ceph-mon[93869]: pgmap v6611: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 134 KiB/s wr, 9 op/s 2023-03-09T18:08:44.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:08:44 smithi181 ceph-mon[94189]: pgmap v6611: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 134 KiB/s wr, 9 op/s 2023-03-09T18:08:45.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:08:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11436.092678 secs 2023-03-09T18:08:45.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:08:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:08:45.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:08:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11436.092678 secs 2023-03-09T18:08:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:08:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:08:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:08:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11436.092678 secs 2023-03-09T18:08:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:08:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:08:46.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:08:46 smithi122 ceph-mon[93869]: pgmap v6612: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 134 KiB/s wr, 8 op/s 2023-03-09T18:08:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:08:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:08:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:08:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:08:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:08:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:08:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:08:46 smithi181 ceph-mon[94189]: pgmap v6612: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 134 KiB/s wr, 8 op/s 2023-03-09T18:08:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:08:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:08:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:08:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:08:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:08:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:08:46.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:08:46 smithi038 ceph-mon[92422]: pgmap v6612: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 134 KiB/s wr, 8 op/s 2023-03-09T18:08:46.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:08:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:08:46.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:08:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:08:46.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:08:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:08:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11900.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:08:48.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:08:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11438.383962 secs 2023-03-09T18:08:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:08:48 smithi122 ceph-mon[93869]: pgmap v6613: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 201 KiB/s wr, 11 op/s 2023-03-09T18:08:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:08:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11438.383962 secs 2023-03-09T18:08:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:08:48 smithi181 ceph-mon[94189]: pgmap v6613: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 201 KiB/s wr, 11 op/s 2023-03-09T18:08:48.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:08:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11438.383962 secs 2023-03-09T18:08:48.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:08:48 smithi038 ceph-mon[92422]: pgmap v6613: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 201 KiB/s wr, 11 op/s 2023-03-09T18:08:50.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:08:50 smithi122 ceph-mon[93869]: pgmap v6614: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 158 KiB/s wr, 9 op/s 2023-03-09T18:08:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:08:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11441.092738 secs 2023-03-09T18:08:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:08:50 smithi181 ceph-mon[94189]: pgmap v6614: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 158 KiB/s wr, 9 op/s 2023-03-09T18:08:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:08:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11441.092738 secs 2023-03-09T18:08:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:08:50 smithi038 ceph-mon[92422]: pgmap v6614: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 158 KiB/s wr, 9 op/s 2023-03-09T18:08:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:08:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11441.092738 secs 2023-03-09T18:08:52.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:08:52 smithi122 ceph-mon[93869]: pgmap v6615: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 208 KiB/s wr, 11 op/s 2023-03-09T18:08:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:08:52 smithi181 ceph-mon[94189]: pgmap v6615: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 208 KiB/s wr, 11 op/s 2023-03-09T18:08:52.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:08:52 smithi038 ceph-mon[92422]: pgmap v6615: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 208 KiB/s wr, 11 op/s 2023-03-09T18:08:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11905.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:08:53.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:08:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11443.384097 secs 2023-03-09T18:08:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:08:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11443.384097 secs 2023-03-09T18:08:53.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:08:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11443.384097 secs 2023-03-09T18:08:54.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:08:54 smithi122 ceph-mon[93869]: pgmap v6616: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 175 KiB/s wr, 10 op/s 2023-03-09T18:08:54.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:08:54 smithi181 ceph-mon[94189]: pgmap v6616: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 175 KiB/s wr, 10 op/s 2023-03-09T18:08:54.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:08:54 smithi038 ceph-mon[92422]: pgmap v6616: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 175 KiB/s wr, 10 op/s 2023-03-09T18:08:55.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:08:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11446.092797 secs 2023-03-09T18:08:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:08:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11446.092797 secs 2023-03-09T18:08:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:08:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11446.092797 secs 2023-03-09T18:08:56.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:08:56 smithi122 ceph-mon[93869]: pgmap v6617: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 166 KiB/s wr, 9 op/s 2023-03-09T18:08:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:08:56 smithi181 ceph-mon[94189]: pgmap v6617: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 166 KiB/s wr, 9 op/s 2023-03-09T18:08:56.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:08:56 smithi038 ceph-mon[92422]: pgmap v6617: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 166 KiB/s wr, 9 op/s 2023-03-09T18:08:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11910.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:08:58.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:08:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11448.384213 secs 2023-03-09T18:08:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:08:58 smithi122 ceph-mon[93869]: pgmap v6618: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 208 KiB/s wr, 11 op/s 2023-03-09T18:08:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:08:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11448.384213 secs 2023-03-09T18:08:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:08:58 smithi181 ceph-mon[94189]: pgmap v6618: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 208 KiB/s wr, 11 op/s 2023-03-09T18:08:58.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:08:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11448.384213 secs 2023-03-09T18:08:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:08:58 smithi038 ceph-mon[92422]: pgmap v6618: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 208 KiB/s wr, 11 op/s 2023-03-09T18:09:00.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:09:00 smithi122 ceph-mon[93869]: pgmap v6619: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 188 KiB/s wr, 10 op/s 2023-03-09T18:09:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:09:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11451.092881 secs 2023-03-09T18:09:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:09:00 smithi181 ceph-mon[94189]: pgmap v6619: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 188 KiB/s wr, 10 op/s 2023-03-09T18:09:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:09:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11451.092881 secs 2023-03-09T18:09:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:09:00 smithi038 ceph-mon[92422]: pgmap v6619: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 188 KiB/s wr, 10 op/s 2023-03-09T18:09:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:09:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11451.092881 secs 2023-03-09T18:09:02.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:09:02 smithi122 ceph-mon[93869]: pgmap v6620: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 201 KiB/s wr, 13 op/s 2023-03-09T18:09:02.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:09:02 smithi181 ceph-mon[94189]: pgmap v6620: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 201 KiB/s wr, 13 op/s 2023-03-09T18:09:02.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:09:02 smithi038 ceph-mon[92422]: pgmap v6620: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 201 KiB/s wr, 13 op/s 2023-03-09T18:09:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11915.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:09:03.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:09:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11453.384306 secs 2023-03-09T18:09:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:09:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11453.384306 secs 2023-03-09T18:09:03.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:09:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11453.384306 secs 2023-03-09T18:09:04.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:09:04 smithi122 ceph-mon[93869]: pgmap v6621: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 242 KiB/s wr, 25 op/s 2023-03-09T18:09:04.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:09:04 smithi181 ceph-mon[94189]: pgmap v6621: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 242 KiB/s wr, 25 op/s 2023-03-09T18:09:04.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:09:04 smithi038 ceph-mon[92422]: pgmap v6621: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 242 KiB/s wr, 25 op/s 2023-03-09T18:09:05.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:09:05 smithi122 ceph-mon[93869]: osdmap e627: 12 total, 12 up, 12 in 2023-03-09T18:09:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:09:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11456.092955 secs 2023-03-09T18:09:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:09:05 smithi122 ceph-mon[93869]: pgmap v6623: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 255 KiB/s wr, 28 op/s 2023-03-09T18:09:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:09:05 smithi181 ceph-mon[94189]: osdmap e627: 12 total, 12 up, 12 in 2023-03-09T18:09:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:09:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11456.092955 secs 2023-03-09T18:09:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:09:05 smithi181 ceph-mon[94189]: pgmap v6623: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 255 KiB/s wr, 28 op/s 2023-03-09T18:09:05.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:09:05 smithi038 ceph-mon[92422]: osdmap e627: 12 total, 12 up, 12 in 2023-03-09T18:09:05.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:09:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11456.092955 secs 2023-03-09T18:09:05.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:09:05 smithi038 ceph-mon[92422]: pgmap v6623: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 255 KiB/s wr, 28 op/s 2023-03-09T18:09:08.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:09:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11458.384422 secs 2023-03-09T18:09:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:09:08 smithi122 ceph-mon[93869]: pgmap v6624: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 204 KiB/s wr, 27 op/s 2023-03-09T18:09:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:09:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11458.384422 secs 2023-03-09T18:09:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:09:08 smithi181 ceph-mon[94189]: pgmap v6624: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 204 KiB/s wr, 27 op/s 2023-03-09T18:09:08.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:09:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11458.384422 secs 2023-03-09T18:09:08.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:09:08 smithi038 ceph-mon[92422]: pgmap v6624: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 204 KiB/s wr, 27 op/s 2023-03-09T18:09:08.793 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11920.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:09:10.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:09:10 smithi122 ceph-mon[93869]: pgmap v6625: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 159 KiB/s wr, 26 op/s 2023-03-09T18:09:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:09:10 smithi181 ceph-mon[94189]: pgmap v6625: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 159 KiB/s wr, 26 op/s 2023-03-09T18:09:10.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:09:10 smithi038 ceph-mon[92422]: pgmap v6625: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 159 KiB/s wr, 26 op/s 2023-03-09T18:09:11.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:09:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11461.092982 secs 2023-03-09T18:09:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:09:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11461.092982 secs 2023-03-09T18:09:11.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:09:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11461.092982 secs 2023-03-09T18:09:12.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:09:12 smithi122 ceph-mon[93869]: pgmap v6626: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 132 KiB/s wr, 23 op/s 2023-03-09T18:09:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:09:12 smithi122 ceph-mon[93869]: osdmap e628: 12 total, 12 up, 12 in 2023-03-09T18:09:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:09:12 smithi181 ceph-mon[94189]: pgmap v6626: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 132 KiB/s wr, 23 op/s 2023-03-09T18:09:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:09:12 smithi181 ceph-mon[94189]: osdmap e628: 12 total, 12 up, 12 in 2023-03-09T18:09:12.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:09:12 smithi038 ceph-mon[92422]: pgmap v6626: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 132 KiB/s wr, 23 op/s 2023-03-09T18:09:12.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:09:12 smithi038 ceph-mon[92422]: osdmap e628: 12 total, 12 up, 12 in 2023-03-09T18:09:13.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:09:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11463.384538 secs 2023-03-09T18:09:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:09:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11463.384538 secs 2023-03-09T18:09:13.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:09:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11463.384538 secs 2023-03-09T18:09:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11925.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:09:14.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:09:14 smithi122 ceph-mon[93869]: pgmap v6628: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 58 KiB/s wr, 9 op/s 2023-03-09T18:09:14.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:09:14 smithi181 ceph-mon[94189]: pgmap v6628: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 58 KiB/s wr, 9 op/s 2023-03-09T18:09:14.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:09:14 smithi038 ceph-mon[92422]: pgmap v6628: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 58 KiB/s wr, 9 op/s 2023-03-09T18:09:16.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:09:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11466.093040 secs 2023-03-09T18:09:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:09:16 smithi122 ceph-mon[93869]: pgmap v6629: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 8 op/s 2023-03-09T18:09:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:09:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11466.093040 secs 2023-03-09T18:09:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:09:16 smithi181 ceph-mon[94189]: pgmap v6629: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 8 op/s 2023-03-09T18:09:16.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:09:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11466.093040 secs 2023-03-09T18:09:16.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:09:16 smithi038 ceph-mon[92422]: pgmap v6629: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 8 op/s 2023-03-09T18:09:18.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:09:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11468.384655 secs 2023-03-09T18:09:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:09:18 smithi122 ceph-mon[93869]: pgmap v6630: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 8 op/s 2023-03-09T18:09:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:09:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11468.384655 secs 2023-03-09T18:09:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:09:18 smithi181 ceph-mon[94189]: pgmap v6630: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 8 op/s 2023-03-09T18:09:18.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:09:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11468.384655 secs 2023-03-09T18:09:18.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:09:18 smithi038 ceph-mon[92422]: pgmap v6630: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 8 op/s 2023-03-09T18:09:18.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11930.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:09:20.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:09:20 smithi122 ceph-mon[93869]: pgmap v6631: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 39 KiB/s wr, 8 op/s 2023-03-09T18:09:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:09:20 smithi181 ceph-mon[94189]: pgmap v6631: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 39 KiB/s wr, 8 op/s 2023-03-09T18:09:20.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:09:20 smithi038 ceph-mon[92422]: pgmap v6631: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 39 KiB/s wr, 8 op/s 2023-03-09T18:09:21.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:09:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11471.093124 secs 2023-03-09T18:09:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:09:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11471.093124 secs 2023-03-09T18:09:21.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:09:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11471.093124 secs 2023-03-09T18:09:22.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:09:22 smithi122 ceph-mon[93869]: pgmap v6632: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 40 KiB/s wr, 8 op/s 2023-03-09T18:09:22.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:09:22 smithi181 ceph-mon[94189]: pgmap v6632: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 40 KiB/s wr, 8 op/s 2023-03-09T18:09:22.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:09:22 smithi038 ceph-mon[92422]: pgmap v6632: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 40 KiB/s wr, 8 op/s 2023-03-09T18:09:23.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:09:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11473.384777 secs 2023-03-09T18:09:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:09:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11473.384777 secs 2023-03-09T18:09:23.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:09:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11473.384777 secs 2023-03-09T18:09:23.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11935.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:09:24.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:09:24 smithi122 ceph-mon[93869]: pgmap v6633: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 90 KiB/s wr, 8 op/s 2023-03-09T18:09:24.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:09:24 smithi181 ceph-mon[94189]: pgmap v6633: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 90 KiB/s wr, 8 op/s 2023-03-09T18:09:24.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:09:24 smithi038 ceph-mon[92422]: pgmap v6633: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 90 KiB/s wr, 8 op/s 2023-03-09T18:09:26.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:09:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11476.093185 secs 2023-03-09T18:09:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:09:26 smithi122 ceph-mon[93869]: pgmap v6634: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 56 KiB/s wr, 7 op/s 2023-03-09T18:09:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:09:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11476.093185 secs 2023-03-09T18:09:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:09:26 smithi181 ceph-mon[94189]: pgmap v6634: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 56 KiB/s wr, 7 op/s 2023-03-09T18:09:26.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:09:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11476.093185 secs 2023-03-09T18:09:26.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:09:26 smithi038 ceph-mon[92422]: pgmap v6634: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 56 KiB/s wr, 7 op/s 2023-03-09T18:09:28.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:09:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11478.384890 secs 2023-03-09T18:09:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:09:28 smithi122 ceph-mon[93869]: pgmap v6635: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 9 op/s 2023-03-09T18:09:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:09:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11478.384890 secs 2023-03-09T18:09:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:09:28 smithi181 ceph-mon[94189]: pgmap v6635: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 9 op/s 2023-03-09T18:09:28.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:09:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11478.384890 secs 2023-03-09T18:09:28.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:09:28 smithi038 ceph-mon[92422]: pgmap v6635: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 9 op/s 2023-03-09T18:09:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11940.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:09:30.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:09:30 smithi122 ceph-mon[93869]: pgmap v6636: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 57 KiB/s wr, 7 op/s 2023-03-09T18:09:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:09:30 smithi181 ceph-mon[94189]: pgmap v6636: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 57 KiB/s wr, 7 op/s 2023-03-09T18:09:30.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:09:30 smithi038 ceph-mon[92422]: pgmap v6636: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 57 KiB/s wr, 7 op/s 2023-03-09T18:09:31.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:09:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11481.093244 secs 2023-03-09T18:09:31.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:09:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11481.093244 secs 2023-03-09T18:09:31.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:09:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11481.093244 secs 2023-03-09T18:09:32.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:09:32 smithi122 ceph-mon[93869]: pgmap v6637: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 8 op/s 2023-03-09T18:09:32.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:09:32 smithi181 ceph-mon[94189]: pgmap v6637: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 8 op/s 2023-03-09T18:09:32.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:09:32 smithi038 ceph-mon[92422]: pgmap v6637: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 8 op/s 2023-03-09T18:09:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11945.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:09:34.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:09:34 smithi038 ceph-mon[92422]: pgmap v6638: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 132 KiB/s wr, 8 op/s 2023-03-09T18:09:34.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:09:34 smithi122 ceph-mon[93869]: pgmap v6638: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 132 KiB/s wr, 8 op/s 2023-03-09T18:09:34.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:09:34 smithi181 ceph-mon[94189]: pgmap v6638: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 132 KiB/s wr, 8 op/s 2023-03-09T18:09:35.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:09:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11486.093305 secs 2023-03-09T18:09:35.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:09:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11486.093305 secs 2023-03-09T18:09:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:09:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11486.093305 secs 2023-03-09T18:09:36.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:09:36 smithi122 ceph-mon[93869]: pgmap v6639: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 76 KiB/s wr, 6 op/s 2023-03-09T18:09:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:09:36 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:09:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:09:36 smithi181 ceph-mon[94189]: pgmap v6639: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 76 KiB/s wr, 6 op/s 2023-03-09T18:09:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:09:36 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:09:36.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:09:36 smithi038 ceph-mon[92422]: pgmap v6639: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 76 KiB/s wr, 6 op/s 2023-03-09T18:09:36.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:09:36 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:09:37.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:09:37 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:09:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:09:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:09:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:09:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:09:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:09:37 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:09:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:09:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:09:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:09:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:09:37.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:09:37 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:09:37.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:09:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:09:37.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:09:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:09:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11950.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:09:38.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:09:38 smithi122 ceph-mon[93869]: pgmap v6640: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 76 KiB/s wr, 8 op/s 2023-03-09T18:09:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:09:38 smithi181 ceph-mon[94189]: pgmap v6640: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 76 KiB/s wr, 8 op/s 2023-03-09T18:09:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:09:38 smithi038 ceph-mon[92422]: pgmap v6640: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 76 KiB/s wr, 8 op/s 2023-03-09T18:09:40.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:09:40 smithi122 ceph-mon[93869]: pgmap v6641: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 93 KiB/s wr, 8 op/s 2023-03-09T18:09:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:09:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11491.093382 secs 2023-03-09T18:09:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:09:40 smithi181 ceph-mon[94189]: pgmap v6641: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 93 KiB/s wr, 8 op/s 2023-03-09T18:09:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:09:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11491.093382 secs 2023-03-09T18:09:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:09:40 smithi038 ceph-mon[92422]: pgmap v6641: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 93 KiB/s wr, 8 op/s 2023-03-09T18:09:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:09:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11491.093382 secs 2023-03-09T18:09:42.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:09:42 smithi122 ceph-mon[93869]: pgmap v6642: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 110 KiB/s wr, 9 op/s 2023-03-09T18:09:42.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:09:42 smithi181 ceph-mon[94189]: pgmap v6642: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 110 KiB/s wr, 9 op/s 2023-03-09T18:09:42.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:09:42 smithi038 ceph-mon[92422]: pgmap v6642: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 110 KiB/s wr, 9 op/s 2023-03-09T18:09:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11955.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:09:44.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:09:44 smithi122 ceph-mon[93869]: pgmap v6643: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 166 KiB/s wr, 10 op/s 2023-03-09T18:09:44.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:09:44 smithi181 ceph-mon[94189]: pgmap v6643: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 166 KiB/s wr, 10 op/s 2023-03-09T18:09:44.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:09:44 smithi038 ceph-mon[92422]: pgmap v6643: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 166 KiB/s wr, 10 op/s 2023-03-09T18:09:45.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:09:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11496.093445 secs 2023-03-09T18:09:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:09:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11496.093445 secs 2023-03-09T18:09:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:09:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11496.093445 secs 2023-03-09T18:09:46.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:09:46 smithi122 ceph-mon[93869]: pgmap v6644: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 196 KiB/s wr, 10 op/s 2023-03-09T18:09:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:09:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:09:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:09:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:09:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:09:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:09:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:09:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:09:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:09:46 smithi181 ceph-mon[94189]: pgmap v6644: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 196 KiB/s wr, 10 op/s 2023-03-09T18:09:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:09:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:09:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:09:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:09:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:09:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:09:46.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:09:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:09:46.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:09:46 smithi038 ceph-mon[92422]: pgmap v6644: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 196 KiB/s wr, 10 op/s 2023-03-09T18:09:46.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:09:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:09:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:09:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:09:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:09:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:09:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:09:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:09:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11960.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:09:48.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:09:48 smithi122 ceph-mon[93869]: pgmap v6645: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 196 KiB/s wr, 12 op/s 2023-03-09T18:09:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:09:48 smithi181 ceph-mon[94189]: pgmap v6645: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 196 KiB/s wr, 12 op/s 2023-03-09T18:09:48.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:09:48 smithi038 ceph-mon[92422]: pgmap v6645: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 196 KiB/s wr, 12 op/s 2023-03-09T18:09:50.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:09:50 smithi122 ceph-mon[93869]: pgmap v6646: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 206 KiB/s wr, 11 op/s 2023-03-09T18:09:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:09:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11501.093503 secs 2023-03-09T18:09:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:09:50 smithi181 ceph-mon[94189]: pgmap v6646: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 206 KiB/s wr, 11 op/s 2023-03-09T18:09:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:09:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11501.093503 secs 2023-03-09T18:09:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:09:50 smithi038 ceph-mon[92422]: pgmap v6646: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 206 KiB/s wr, 11 op/s 2023-03-09T18:09:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:09:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11501.093503 secs 2023-03-09T18:09:52.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:09:52 smithi122 ceph-mon[93869]: pgmap v6647: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 226 KiB/s wr, 13 op/s 2023-03-09T18:09:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:09:52 smithi181 ceph-mon[94189]: pgmap v6647: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 226 KiB/s wr, 13 op/s 2023-03-09T18:09:52.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:09:52 smithi038 ceph-mon[92422]: pgmap v6647: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 226 KiB/s wr, 13 op/s 2023-03-09T18:09:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11965.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:09:54.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:09:54 smithi122 ceph-mon[93869]: pgmap v6648: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 227 KiB/s wr, 11 op/s 2023-03-09T18:09:54.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:09:54 smithi181 ceph-mon[94189]: pgmap v6648: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 227 KiB/s wr, 11 op/s 2023-03-09T18:09:54.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:09:54 smithi038 ceph-mon[92422]: pgmap v6648: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 227 KiB/s wr, 11 op/s 2023-03-09T18:09:55.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:09:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11506.093562 secs 2023-03-09T18:09:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:09:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11506.093562 secs 2023-03-09T18:09:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:09:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11506.093562 secs 2023-03-09T18:09:56.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:09:56 smithi122 ceph-mon[93869]: pgmap v6649: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 235 KiB/s wr, 10 op/s 2023-03-09T18:09:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:09:56 smithi181 ceph-mon[94189]: pgmap v6649: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 235 KiB/s wr, 10 op/s 2023-03-09T18:09:56.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:09:56 smithi038 ceph-mon[92422]: pgmap v6649: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 235 KiB/s wr, 10 op/s 2023-03-09T18:09:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11970.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:09:58.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:09:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11508.385614 secs 2023-03-09T18:09:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:09:58 smithi122 ceph-mon[93869]: pgmap v6650: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 129 KiB/s wr, 11 op/s 2023-03-09T18:09:58.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:09:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11508.385614 secs 2023-03-09T18:09:58.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:09:58 smithi181 ceph-mon[94189]: pgmap v6650: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 129 KiB/s wr, 11 op/s 2023-03-09T18:09:58.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:09:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11508.385614 secs 2023-03-09T18:09:58.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:09:58 smithi038 ceph-mon[92422]: pgmap v6650: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 129 KiB/s wr, 11 op/s 2023-03-09T18:09:59.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:09:59 smithi122 ceph-mon[93869]: pgmap v6651: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 130 KiB/s wr, 9 op/s 2023-03-09T18:09:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:09:59 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:09:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:09:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:09:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:09:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:09:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:09:59 smithi181 ceph-mon[94189]: pgmap v6651: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 130 KiB/s wr, 9 op/s 2023-03-09T18:09:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:09:59 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:09:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:09:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:09:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:09:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:09:59.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:09:59 smithi038 ceph-mon[92422]: pgmap v6651: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 130 KiB/s wr, 9 op/s 2023-03-09T18:09:59.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:09:59 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:09:59.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:09:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:09:59.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:09:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:10:00.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:10:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11511.093592 secs 2023-03-09T18:10:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:10:00 smithi122 ceph-mon[93869]: overall HEALTH_WARN 2 MDSs report slow requests 2023-03-09T18:10:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:10:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11511.093592 secs 2023-03-09T18:10:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:10:00 smithi181 ceph-mon[94189]: overall HEALTH_WARN 2 MDSs report slow requests 2023-03-09T18:10:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:10:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11511.093592 secs 2023-03-09T18:10:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:10:00 smithi038 ceph-mon[92422]: overall HEALTH_WARN 2 MDSs report slow requests 2023-03-09T18:10:02.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:10:02 smithi122 ceph-mon[93869]: pgmap v6652: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 201 KiB/s wr, 13 op/s 2023-03-09T18:10:02.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:10:02 smithi181 ceph-mon[94189]: pgmap v6652: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 201 KiB/s wr, 13 op/s 2023-03-09T18:10:02.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:10:02 smithi038 ceph-mon[92422]: pgmap v6652: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 201 KiB/s wr, 13 op/s 2023-03-09T18:10:03.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:10:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11513.385735 secs 2023-03-09T18:10:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:10:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11513.385735 secs 2023-03-09T18:10:03.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:10:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11513.385735 secs 2023-03-09T18:10:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11975.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:10:04.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:10:04 smithi122 ceph-mon[93869]: pgmap v6653: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 185 KiB/s wr, 23 op/s 2023-03-09T18:10:04.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:10:04 smithi181 ceph-mon[94189]: pgmap v6653: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 185 KiB/s wr, 23 op/s 2023-03-09T18:10:04.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:10:04 smithi038 ceph-mon[92422]: pgmap v6653: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 185 KiB/s wr, 23 op/s 2023-03-09T18:10:06.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:10:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11516.093648 secs 2023-03-09T18:10:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:10:06 smithi122 ceph-mon[93869]: pgmap v6654: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 199 KiB/s wr, 22 op/s 2023-03-09T18:10:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:10:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11516.093648 secs 2023-03-09T18:10:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:10:06 smithi181 ceph-mon[94189]: pgmap v6654: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 199 KiB/s wr, 22 op/s 2023-03-09T18:10:06.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:10:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11516.093648 secs 2023-03-09T18:10:06.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:10:06 smithi038 ceph-mon[92422]: pgmap v6654: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 199 KiB/s wr, 22 op/s 2023-03-09T18:10:08.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:10:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11518.385839 secs 2023-03-09T18:10:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:10:08 smithi122 ceph-mon[93869]: pgmap v6655: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 136 KiB/s wr, 23 op/s 2023-03-09T18:10:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:10:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11518.385839 secs 2023-03-09T18:10:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:10:08 smithi181 ceph-mon[94189]: pgmap v6655: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 136 KiB/s wr, 23 op/s 2023-03-09T18:10:08.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:10:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11518.385839 secs 2023-03-09T18:10:08.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:10:08 smithi038 ceph-mon[92422]: pgmap v6655: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 136 KiB/s wr, 23 op/s 2023-03-09T18:10:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11980.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:10:10.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:10:10 smithi122 ceph-mon[93869]: pgmap v6656: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 145 KiB/s wr, 22 op/s 2023-03-09T18:10:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:10:10 smithi122 ceph-mon[93869]: osdmap e629: 12 total, 12 up, 12 in 2023-03-09T18:10:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:10:10 smithi181 ceph-mon[94189]: pgmap v6656: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 145 KiB/s wr, 22 op/s 2023-03-09T18:10:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:10:10 smithi181 ceph-mon[94189]: osdmap e629: 12 total, 12 up, 12 in 2023-03-09T18:10:10.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:10:10 smithi038 ceph-mon[92422]: pgmap v6656: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 145 KiB/s wr, 22 op/s 2023-03-09T18:10:10.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:10:10 smithi038 ceph-mon[92422]: osdmap e629: 12 total, 12 up, 12 in 2023-03-09T18:10:11.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:10:11 smithi122 ceph-mon[93869]: 2 slow requests, 2 included below; oldest blocked for > 11521.093753 secs 2023-03-09T18:10:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:10:11 smithi122 ceph-mon[93869]: slow request 11521.093753 seconds old, received at 2023-03-09T14:58:08.564667+0000: peer_request:mds.2:599 currently acquired locks 2023-03-09T18:10:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:10:11 smithi122 ceph-mon[93869]: slow request 11521.083052 seconds old, received at 2023-03-09T14:58:08.575367+0000: peer_request(client.15375:6773.0 authpin) currently dispatched 2023-03-09T18:10:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:10:11 smithi181 ceph-mon[94189]: 2 slow requests, 2 included below; oldest blocked for > 11521.093753 secs 2023-03-09T18:10:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:10:11 smithi181 ceph-mon[94189]: slow request 11521.093753 seconds old, received at 2023-03-09T14:58:08.564667+0000: peer_request:mds.2:599 currently acquired locks 2023-03-09T18:10:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:10:11 smithi181 ceph-mon[94189]: slow request 11521.083052 seconds old, received at 2023-03-09T14:58:08.575367+0000: peer_request(client.15375:6773.0 authpin) currently dispatched 2023-03-09T18:10:11.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:10:11 smithi038 ceph-mon[92422]: 2 slow requests, 2 included below; oldest blocked for > 11521.093753 secs 2023-03-09T18:10:11.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:10:11 smithi038 ceph-mon[92422]: slow request 11521.093753 seconds old, received at 2023-03-09T14:58:08.564667+0000: peer_request:mds.2:599 currently acquired locks 2023-03-09T18:10:11.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:10:11 smithi038 ceph-mon[92422]: slow request 11521.083052 seconds old, received at 2023-03-09T14:58:08.575367+0000: peer_request(client.15375:6773.0 authpin) currently dispatched 2023-03-09T18:10:12.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:10:12 smithi122 ceph-mon[93869]: pgmap v6658: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 98 KiB/s wr, 23 op/s 2023-03-09T18:10:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:10:12 smithi181 ceph-mon[94189]: pgmap v6658: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 98 KiB/s wr, 23 op/s 2023-03-09T18:10:12.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:10:12 smithi038 ceph-mon[92422]: pgmap v6658: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 98 KiB/s wr, 23 op/s 2023-03-09T18:10:13.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:10:13 smithi122 ceph-mon[93869]: 2 slow requests, 2 included below; oldest blocked for > 11523.385951 secs 2023-03-09T18:10:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:10:13 smithi122 ceph-mon[93869]: slow request 11523.385950 seconds old, received at 2023-03-09T14:58:08.560137+0000: client_request(mds.2:599 rename #0x10000000007/0000000100000001000000EB #0x60f/1000000062a caller_uid=0, caller_gid=0{}) currently acquired locks 2023-03-09T18:10:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:10:13 smithi122 ceph-mon[93869]: slow request 11523.371570 seconds old, received at 2023-03-09T14:58:08.574517+0000: client_request(client.15375:6773 unlink #0x10000000007/0000000100000001000000EB 2023-03-09T14:58:08.569481+0000 caller_uid=26, caller_gid=26{26,}) currently requesting remote authpins 2023-03-09T18:10:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:10:13 smithi181 ceph-mon[94189]: 2 slow requests, 2 included below; oldest blocked for > 11523.385951 secs 2023-03-09T18:10:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:10:13 smithi181 ceph-mon[94189]: slow request 11523.385950 seconds old, received at 2023-03-09T14:58:08.560137+0000: client_request(mds.2:599 rename #0x10000000007/0000000100000001000000EB #0x60f/1000000062a caller_uid=0, caller_gid=0{}) currently acquired locks 2023-03-09T18:10:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:10:13 smithi181 ceph-mon[94189]: slow request 11523.371570 seconds old, received at 2023-03-09T14:58:08.574517+0000: client_request(client.15375:6773 unlink #0x10000000007/0000000100000001000000EB 2023-03-09T14:58:08.569481+0000 caller_uid=26, caller_gid=26{26,}) currently requesting remote authpins 2023-03-09T18:10:13.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:10:13 smithi038 ceph-mon[92422]: 2 slow requests, 2 included below; oldest blocked for > 11523.385951 secs 2023-03-09T18:10:13.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:10:13 smithi038 ceph-mon[92422]: slow request 11523.385950 seconds old, received at 2023-03-09T14:58:08.560137+0000: client_request(mds.2:599 rename #0x10000000007/0000000100000001000000EB #0x60f/1000000062a caller_uid=0, caller_gid=0{}) currently acquired locks 2023-03-09T18:10:13.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:10:13 smithi038 ceph-mon[92422]: slow request 11523.371570 seconds old, received at 2023-03-09T14:58:08.574517+0000: client_request(client.15375:6773 unlink #0x10000000007/0000000100000001000000EB 2023-03-09T14:58:08.569481+0000 caller_uid=26, caller_gid=26{26,}) currently requesting remote authpins 2023-03-09T18:10:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11985.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:10:14.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:10:14 smithi122 ceph-mon[93869]: pgmap v6659: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 83 KiB/s wr, 8 op/s 2023-03-09T18:10:14.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:10:14 smithi181 ceph-mon[94189]: pgmap v6659: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 83 KiB/s wr, 8 op/s 2023-03-09T18:10:14.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:10:14 smithi038 ceph-mon[92422]: pgmap v6659: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 83 KiB/s wr, 8 op/s 2023-03-09T18:10:16.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:10:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11526.093816 secs 2023-03-09T18:10:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:10:16 smithi122 ceph-mon[93869]: pgmap v6660: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 45 KiB/s wr, 7 op/s 2023-03-09T18:10:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:10:16 smithi122 ceph-mon[93869]: osdmap e630: 12 total, 12 up, 12 in 2023-03-09T18:10:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:10:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11526.093816 secs 2023-03-09T18:10:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:10:16 smithi181 ceph-mon[94189]: pgmap v6660: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 45 KiB/s wr, 7 op/s 2023-03-09T18:10:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:10:16 smithi181 ceph-mon[94189]: osdmap e630: 12 total, 12 up, 12 in 2023-03-09T18:10:16.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:10:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11526.093816 secs 2023-03-09T18:10:16.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:10:16 smithi038 ceph-mon[92422]: pgmap v6660: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 45 KiB/s wr, 7 op/s 2023-03-09T18:10:16.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:10:16 smithi038 ceph-mon[92422]: osdmap e630: 12 total, 12 up, 12 in 2023-03-09T18:10:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:10:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11528.386074 secs 2023-03-09T18:10:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:10:18 smithi181 ceph-mon[94189]: pgmap v6662: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 9 op/s 2023-03-09T18:10:18.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:10:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11528.386074 secs 2023-03-09T18:10:18.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:10:18 smithi038 ceph-mon[92422]: pgmap v6662: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 9 op/s 2023-03-09T18:10:18.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11990.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:10:18.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:10:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11528.386074 secs 2023-03-09T18:10:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:10:18 smithi122 ceph-mon[93869]: pgmap v6662: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 9 op/s 2023-03-09T18:10:20.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:10:20 smithi038 ceph-mon[92422]: pgmap v6663: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 KiB/s wr, 8 op/s 2023-03-09T18:10:20.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:10:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11531.093875 secs 2023-03-09T18:10:20.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:10:20 smithi122 ceph-mon[93869]: pgmap v6663: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 KiB/s wr, 8 op/s 2023-03-09T18:10:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:10:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11531.093875 secs 2023-03-09T18:10:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:10:20 smithi181 ceph-mon[94189]: pgmap v6663: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 KiB/s wr, 8 op/s 2023-03-09T18:10:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:10:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11531.093875 secs 2023-03-09T18:10:22.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:10:22 smithi038 ceph-mon[92422]: pgmap v6664: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 7 op/s 2023-03-09T18:10:22.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:10:22 smithi122 ceph-mon[93869]: pgmap v6664: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 7 op/s 2023-03-09T18:10:22.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:10:22 smithi181 ceph-mon[94189]: pgmap v6664: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 7 op/s 2023-03-09T18:10:23.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:10:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11533.386211 secs 2023-03-09T18:10:23.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 11995.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:10:23.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:10:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11533.386211 secs 2023-03-09T18:10:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:10:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11533.386211 secs 2023-03-09T18:10:24.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:10:24 smithi038 ceph-mon[92422]: pgmap v6665: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 8 op/s 2023-03-09T18:10:24.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:10:24 smithi122 ceph-mon[93869]: pgmap v6665: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 8 op/s 2023-03-09T18:10:24.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:10:24 smithi181 ceph-mon[94189]: pgmap v6665: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 8 op/s 2023-03-09T18:10:25.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:10:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11536.093961 secs 2023-03-09T18:10:25.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:10:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11536.093961 secs 2023-03-09T18:10:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:10:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11536.093961 secs 2023-03-09T18:10:26.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:10:26 smithi038 ceph-mon[92422]: pgmap v6666: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 8 op/s 2023-03-09T18:10:26.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:10:26 smithi122 ceph-mon[93869]: pgmap v6666: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 8 op/s 2023-03-09T18:10:26.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:10:26 smithi181 ceph-mon[94189]: pgmap v6666: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 8 op/s 2023-03-09T18:10:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12000.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:10:28.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:10:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11538.386334 secs 2023-03-09T18:10:28.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:10:28 smithi122 ceph-mon[93869]: pgmap v6667: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 38 KiB/s wr, 7 op/s 2023-03-09T18:10:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:10:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11538.386334 secs 2023-03-09T18:10:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:10:28 smithi181 ceph-mon[94189]: pgmap v6667: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 38 KiB/s wr, 7 op/s 2023-03-09T18:10:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:10:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11538.386334 secs 2023-03-09T18:10:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:10:28 smithi038 ceph-mon[92422]: pgmap v6667: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 38 KiB/s wr, 7 op/s 2023-03-09T18:10:30.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:10:30 smithi122 ceph-mon[93869]: pgmap v6668: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 35 KiB/s wr, 7 op/s 2023-03-09T18:10:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:10:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11541.093984 secs 2023-03-09T18:10:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:10:30 smithi181 ceph-mon[94189]: pgmap v6668: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 35 KiB/s wr, 7 op/s 2023-03-09T18:10:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:10:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11541.093984 secs 2023-03-09T18:10:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:10:30 smithi038 ceph-mon[92422]: pgmap v6668: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 35 KiB/s wr, 7 op/s 2023-03-09T18:10:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:10:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11541.093984 secs 2023-03-09T18:10:32.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:10:32 smithi122 ceph-mon[93869]: pgmap v6669: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 35 KiB/s wr, 9 op/s 2023-03-09T18:10:32.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:10:32 smithi181 ceph-mon[94189]: pgmap v6669: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 35 KiB/s wr, 9 op/s 2023-03-09T18:10:32.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:10:32 smithi038 ceph-mon[92422]: pgmap v6669: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 35 KiB/s wr, 9 op/s 2023-03-09T18:10:33.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12005.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:10:33.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:10:33 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 11543.386414 secs 2023-03-09T18:10:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:10:33 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 11543.386414 secs 2023-03-09T18:10:33.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:10:33 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 11543.386414 secs 2023-03-09T18:10:34.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:10:34 smithi122 ceph-mon[93869]: pgmap v6670: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 79 KiB/s wr, 8 op/s 2023-03-09T18:10:34.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:10:34 smithi181 ceph-mon[94189]: pgmap v6670: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 79 KiB/s wr, 8 op/s 2023-03-09T18:10:34.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:10:34 smithi038 ceph-mon[92422]: pgmap v6670: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 79 KiB/s wr, 8 op/s 2023-03-09T18:10:35.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:10:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11546.094071 secs 2023-03-09T18:10:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:10:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11546.094071 secs 2023-03-09T18:10:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:10:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11546.094071 secs 2023-03-09T18:10:36.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:10:36 smithi122 ceph-mon[93869]: pgmap v6671: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 56 KiB/s wr, 7 op/s 2023-03-09T18:10:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:10:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:10:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:10:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:10:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:10:36 smithi181 ceph-mon[94189]: pgmap v6671: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 56 KiB/s wr, 7 op/s 2023-03-09T18:10:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:10:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:10:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:10:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:10:36.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:10:36 smithi038 ceph-mon[92422]: pgmap v6671: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 56 KiB/s wr, 7 op/s 2023-03-09T18:10:36.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:10:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:10:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:10:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:10:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12010.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:10:38.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:10:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11548.386505 secs 2023-03-09T18:10:38.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:10:38 smithi122 ceph-mon[93869]: pgmap v6672: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 46 KiB/s wr, 8 op/s 2023-03-09T18:10:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:10:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11548.386505 secs 2023-03-09T18:10:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:10:38 smithi181 ceph-mon[94189]: pgmap v6672: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 46 KiB/s wr, 8 op/s 2023-03-09T18:10:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:10:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11548.386505 secs 2023-03-09T18:10:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:10:38 smithi038 ceph-mon[92422]: pgmap v6672: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 46 KiB/s wr, 8 op/s 2023-03-09T18:10:40.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:10:40 smithi122 ceph-mon[93869]: pgmap v6673: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 63 KiB/s wr, 8 op/s 2023-03-09T18:10:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:10:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:10:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:10:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:10:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:10:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11551.094133 secs 2023-03-09T18:10:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:10:40 smithi181 ceph-mon[94189]: pgmap v6673: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 63 KiB/s wr, 8 op/s 2023-03-09T18:10:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:10:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:10:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:10:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:10:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:10:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11551.094133 secs 2023-03-09T18:10:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:10:40 smithi038 ceph-mon[92422]: pgmap v6673: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 63 KiB/s wr, 8 op/s 2023-03-09T18:10:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:10:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:10:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:10:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:10:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:10:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11551.094133 secs 2023-03-09T18:10:42.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:10:42 smithi122 ceph-mon[93869]: pgmap v6674: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 63 KiB/s wr, 9 op/s 2023-03-09T18:10:42.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:10:42 smithi181 ceph-mon[94189]: pgmap v6674: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 63 KiB/s wr, 9 op/s 2023-03-09T18:10:42.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:10:42 smithi038 ceph-mon[92422]: pgmap v6674: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 63 KiB/s wr, 9 op/s 2023-03-09T18:10:43.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12015.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:10:43.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:10:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11553.386616 secs 2023-03-09T18:10:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:10:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11553.386616 secs 2023-03-09T18:10:43.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:10:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11553.386616 secs 2023-03-09T18:10:44.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:10:44 smithi122 ceph-mon[93869]: pgmap v6675: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 197 KiB/s wr, 10 op/s 2023-03-09T18:10:44.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:10:44 smithi181 ceph-mon[94189]: pgmap v6675: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 197 KiB/s wr, 10 op/s 2023-03-09T18:10:44.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:10:44 smithi038 ceph-mon[92422]: pgmap v6675: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 197 KiB/s wr, 10 op/s 2023-03-09T18:10:45.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:10:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11556.094194 secs 2023-03-09T18:10:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:10:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11556.094194 secs 2023-03-09T18:10:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:10:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11556.094194 secs 2023-03-09T18:10:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:10:46 smithi122 ceph-mon[93869]: pgmap v6676: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 152 KiB/s wr, 9 op/s 2023-03-09T18:10:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:10:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:10:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:10:46 smithi181 ceph-mon[94189]: pgmap v6676: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 152 KiB/s wr, 9 op/s 2023-03-09T18:10:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:10:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:10:46.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:10:46 smithi038 ceph-mon[92422]: pgmap v6676: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 152 KiB/s wr, 9 op/s 2023-03-09T18:10:46.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:10:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:10:47.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:10:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:10:47.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:10:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:10:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:10:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:10:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:10:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:10:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:10:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:10:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:10:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:10:47.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:10:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:10:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:10:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:10:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:10:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:10:48.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12020.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:10:48.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:10:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11558.386700 secs 2023-03-09T18:10:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:10:48 smithi122 ceph-mon[93869]: pgmap v6677: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 152 KiB/s wr, 11 op/s 2023-03-09T18:10:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:10:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11558.386700 secs 2023-03-09T18:10:48.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:10:48 smithi181 ceph-mon[94189]: pgmap v6677: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 152 KiB/s wr, 11 op/s 2023-03-09T18:10:48.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:10:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11558.386700 secs 2023-03-09T18:10:48.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:10:48 smithi038 ceph-mon[92422]: pgmap v6677: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 152 KiB/s wr, 11 op/s 2023-03-09T18:10:50.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:10:50 smithi122 ceph-mon[93869]: pgmap v6678: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 170 KiB/s wr, 11 op/s 2023-03-09T18:10:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:10:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11561.094273 secs 2023-03-09T18:10:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:10:50 smithi181 ceph-mon[94189]: pgmap v6678: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 170 KiB/s wr, 11 op/s 2023-03-09T18:10:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:10:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11561.094273 secs 2023-03-09T18:10:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:10:50 smithi038 ceph-mon[92422]: pgmap v6678: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 170 KiB/s wr, 11 op/s 2023-03-09T18:10:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:10:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11561.094273 secs 2023-03-09T18:10:52.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:10:52 smithi122 ceph-mon[93869]: pgmap v6679: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 153 KiB/s wr, 11 op/s 2023-03-09T18:10:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:10:52 smithi181 ceph-mon[94189]: pgmap v6679: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 153 KiB/s wr, 11 op/s 2023-03-09T18:10:52.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:10:52 smithi038 ceph-mon[92422]: pgmap v6679: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 153 KiB/s wr, 11 op/s 2023-03-09T18:10:53.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12025.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:10:53.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:10:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11563.386819 secs 2023-03-09T18:10:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:10:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11563.386819 secs 2023-03-09T18:10:53.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:10:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11563.386819 secs 2023-03-09T18:10:54.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:10:54 smithi122 ceph-mon[93869]: pgmap v6680: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 285 KiB/s wr, 11 op/s 2023-03-09T18:10:54.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:10:54 smithi181 ceph-mon[94189]: pgmap v6680: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 285 KiB/s wr, 11 op/s 2023-03-09T18:10:54.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:10:54 smithi038 ceph-mon[92422]: pgmap v6680: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 285 KiB/s wr, 11 op/s 2023-03-09T18:10:55.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:10:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11566.094332 secs 2023-03-09T18:10:55.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:10:55 smithi122 ceph-mon[93869]: pgmap v6681: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 151 KiB/s wr, 8 op/s 2023-03-09T18:10:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:10:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11566.094332 secs 2023-03-09T18:10:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:10:55 smithi181 ceph-mon[94189]: pgmap v6681: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 151 KiB/s wr, 8 op/s 2023-03-09T18:10:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:10:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11566.094332 secs 2023-03-09T18:10:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:10:55 smithi038 ceph-mon[92422]: pgmap v6681: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 151 KiB/s wr, 8 op/s 2023-03-09T18:10:58.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:10:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11568.386929 secs 2023-03-09T18:10:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:10:58 smithi122 ceph-mon[93869]: pgmap v6682: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 179 KiB/s wr, 10 op/s 2023-03-09T18:10:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:10:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11568.386929 secs 2023-03-09T18:10:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:10:58 smithi181 ceph-mon[94189]: pgmap v6682: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 179 KiB/s wr, 10 op/s 2023-03-09T18:10:58.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:10:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11568.386929 secs 2023-03-09T18:10:58.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:10:58 smithi038 ceph-mon[92422]: pgmap v6682: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 179 KiB/s wr, 10 op/s 2023-03-09T18:10:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12030.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:11:00.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:00 smithi122 ceph-mon[93869]: pgmap v6683: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 199 KiB/s wr, 10 op/s 2023-03-09T18:11:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:00 smithi181 ceph-mon[94189]: pgmap v6683: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 199 KiB/s wr, 10 op/s 2023-03-09T18:11:00.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:00 smithi038 ceph-mon[92422]: pgmap v6683: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 199 KiB/s wr, 10 op/s 2023-03-09T18:11:01.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11571.094411 secs 2023-03-09T18:11:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11571.094411 secs 2023-03-09T18:11:01.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11571.094411 secs 2023-03-09T18:11:02.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:02 smithi122 ceph-mon[93869]: pgmap v6684: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 231 KiB/s wr, 12 op/s 2023-03-09T18:11:02.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:02 smithi181 ceph-mon[94189]: pgmap v6684: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 231 KiB/s wr, 12 op/s 2023-03-09T18:11:02.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:02 smithi038 ceph-mon[92422]: pgmap v6684: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 231 KiB/s wr, 12 op/s 2023-03-09T18:11:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11573.387065 secs 2023-03-09T18:11:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11573.387065 secs 2023-03-09T18:11:03.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11573.387065 secs 2023-03-09T18:11:03.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12035.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:11:04.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:04 smithi038 ceph-mon[92422]: pgmap v6685: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 252 KiB/s wr, 24 op/s 2023-03-09T18:11:04.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:04 smithi122 ceph-mon[93869]: pgmap v6685: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 252 KiB/s wr, 24 op/s 2023-03-09T18:11:04.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:04 smithi181 ceph-mon[94189]: pgmap v6685: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 252 KiB/s wr, 24 op/s 2023-03-09T18:11:05.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:05 smithi038 ceph-mon[92422]: osdmap e631: 12 total, 12 up, 12 in 2023-03-09T18:11:05.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11576.095289 secs 2023-03-09T18:11:05.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:05 smithi122 ceph-mon[93869]: osdmap e631: 12 total, 12 up, 12 in 2023-03-09T18:11:05.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11576.095289 secs 2023-03-09T18:11:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:05 smithi181 ceph-mon[94189]: osdmap e631: 12 total, 12 up, 12 in 2023-03-09T18:11:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11576.095289 secs 2023-03-09T18:11:06.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:06 smithi122 ceph-mon[93869]: pgmap v6687: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 143 KiB/s wr, 26 op/s 2023-03-09T18:11:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:06 smithi181 ceph-mon[94189]: pgmap v6687: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 143 KiB/s wr, 26 op/s 2023-03-09T18:11:06.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:06 smithi038 ceph-mon[92422]: pgmap v6687: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 143 KiB/s wr, 26 op/s 2023-03-09T18:11:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12040.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:11:08.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11578.387156 secs 2023-03-09T18:11:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:08 smithi122 ceph-mon[93869]: pgmap v6688: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 178 KiB/s wr, 27 op/s 2023-03-09T18:11:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11578.387156 secs 2023-03-09T18:11:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:08 smithi181 ceph-mon[94189]: pgmap v6688: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 178 KiB/s wr, 27 op/s 2023-03-09T18:11:08.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11578.387156 secs 2023-03-09T18:11:08.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:08 smithi038 ceph-mon[92422]: pgmap v6688: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 178 KiB/s wr, 27 op/s 2023-03-09T18:11:10.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:10 smithi122 ceph-mon[93869]: pgmap v6689: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 164 KiB/s wr, 26 op/s 2023-03-09T18:11:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11581.095376 secs 2023-03-09T18:11:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:10 smithi181 ceph-mon[94189]: pgmap v6689: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 164 KiB/s wr, 26 op/s 2023-03-09T18:11:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11581.095376 secs 2023-03-09T18:11:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:10 smithi038 ceph-mon[92422]: pgmap v6689: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 164 KiB/s wr, 26 op/s 2023-03-09T18:11:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11581.095376 secs 2023-03-09T18:11:12.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:12 smithi122 ceph-mon[93869]: pgmap v6690: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 133 KiB/s wr, 24 op/s 2023-03-09T18:11:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:12 smithi122 ceph-mon[93869]: osdmap e632: 12 total, 12 up, 12 in 2023-03-09T18:11:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:12 smithi181 ceph-mon[94189]: pgmap v6690: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 133 KiB/s wr, 24 op/s 2023-03-09T18:11:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:12 smithi181 ceph-mon[94189]: osdmap e632: 12 total, 12 up, 12 in 2023-03-09T18:11:12.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:12 smithi038 ceph-mon[92422]: pgmap v6690: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 133 KiB/s wr, 24 op/s 2023-03-09T18:11:12.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:12 smithi038 ceph-mon[92422]: osdmap e632: 12 total, 12 up, 12 in 2023-03-09T18:11:13.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11583.387275 secs 2023-03-09T18:11:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11583.387275 secs 2023-03-09T18:11:13.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11583.387275 secs 2023-03-09T18:11:13.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12045.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:11:14.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:14 smithi122 ceph-mon[93869]: pgmap v6692: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 134 KiB/s wr, 11 op/s 2023-03-09T18:11:14.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:14 smithi181 ceph-mon[94189]: pgmap v6692: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 134 KiB/s wr, 11 op/s 2023-03-09T18:11:14.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:14 smithi038 ceph-mon[92422]: pgmap v6692: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 134 KiB/s wr, 11 op/s 2023-03-09T18:11:16.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11586.095439 secs 2023-03-09T18:11:16.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:16 smithi122 ceph-mon[93869]: pgmap v6693: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 118 KiB/s wr, 10 op/s 2023-03-09T18:11:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11586.095439 secs 2023-03-09T18:11:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:16 smithi181 ceph-mon[94189]: pgmap v6693: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 118 KiB/s wr, 10 op/s 2023-03-09T18:11:16.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11586.095439 secs 2023-03-09T18:11:16.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:16 smithi038 ceph-mon[92422]: pgmap v6693: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 118 KiB/s wr, 10 op/s 2023-03-09T18:11:18.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11588.387388 secs 2023-03-09T18:11:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:18 smithi122 ceph-mon[93869]: pgmap v6694: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 51 KiB/s wr, 9 op/s 2023-03-09T18:11:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11588.387388 secs 2023-03-09T18:11:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:18 smithi181 ceph-mon[94189]: pgmap v6694: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 51 KiB/s wr, 9 op/s 2023-03-09T18:11:18.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11588.387388 secs 2023-03-09T18:11:18.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:18 smithi038 ceph-mon[92422]: pgmap v6694: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 51 KiB/s wr, 9 op/s 2023-03-09T18:11:18.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12050.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:11:20.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:20 smithi122 ceph-mon[93869]: pgmap v6695: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 40 KiB/s wr, 8 op/s 2023-03-09T18:11:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:20 smithi181 ceph-mon[94189]: pgmap v6695: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 40 KiB/s wr, 8 op/s 2023-03-09T18:11:20.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:20 smithi038 ceph-mon[92422]: pgmap v6695: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 40 KiB/s wr, 8 op/s 2023-03-09T18:11:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11591.095519 secs 2023-03-09T18:11:21.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11591.095519 secs 2023-03-09T18:11:21.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11591.095519 secs 2023-03-09T18:11:22.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:22 smithi122 ceph-mon[93869]: pgmap v6696: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 35 KiB/s wr, 7 op/s 2023-03-09T18:11:22.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:22 smithi181 ceph-mon[94189]: pgmap v6696: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 35 KiB/s wr, 7 op/s 2023-03-09T18:11:22.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:22 smithi038 ceph-mon[92422]: pgmap v6696: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 35 KiB/s wr, 7 op/s 2023-03-09T18:11:23.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11593.387494 secs 2023-03-09T18:11:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12055.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:11:23.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11593.387494 secs 2023-03-09T18:11:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11593.387494 secs 2023-03-09T18:11:24.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:24 smithi038 ceph-mon[92422]: pgmap v6697: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 30 KiB/s wr, 7 op/s 2023-03-09T18:11:24.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:24 smithi122 ceph-mon[93869]: pgmap v6697: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 30 KiB/s wr, 7 op/s 2023-03-09T18:11:24.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:24 smithi181 ceph-mon[94189]: pgmap v6697: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 30 KiB/s wr, 7 op/s 2023-03-09T18:11:25.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11596.095581 secs 2023-03-09T18:11:25.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11596.095581 secs 2023-03-09T18:11:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11596.095581 secs 2023-03-09T18:11:26.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:26 smithi038 ceph-mon[92422]: pgmap v6698: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 6 op/s 2023-03-09T18:11:26.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:26 smithi122 ceph-mon[93869]: pgmap v6698: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 6 op/s 2023-03-09T18:11:26.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:26 smithi181 ceph-mon[94189]: pgmap v6698: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 6 op/s 2023-03-09T18:11:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12060.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:11:28.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11598.387613 secs 2023-03-09T18:11:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:28 smithi122 ceph-mon[93869]: pgmap v6699: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 55 KiB/s wr, 9 op/s 2023-03-09T18:11:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11598.387613 secs 2023-03-09T18:11:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:28 smithi181 ceph-mon[94189]: pgmap v6699: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 55 KiB/s wr, 9 op/s 2023-03-09T18:11:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11598.387613 secs 2023-03-09T18:11:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:28 smithi038 ceph-mon[92422]: pgmap v6699: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 55 KiB/s wr, 9 op/s 2023-03-09T18:11:30.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:30 smithi122 ceph-mon[93869]: pgmap v6700: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 54 KiB/s wr, 7 op/s 2023-03-09T18:11:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11601.095640 secs 2023-03-09T18:11:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:30 smithi181 ceph-mon[94189]: pgmap v6700: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 54 KiB/s wr, 7 op/s 2023-03-09T18:11:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11601.095640 secs 2023-03-09T18:11:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:30 smithi038 ceph-mon[92422]: pgmap v6700: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 54 KiB/s wr, 7 op/s 2023-03-09T18:11:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11601.095640 secs 2023-03-09T18:11:32.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:32 smithi122 ceph-mon[93869]: pgmap v6701: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 85 KiB/s wr, 9 op/s 2023-03-09T18:11:32.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:32 smithi181 ceph-mon[94189]: pgmap v6701: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 85 KiB/s wr, 9 op/s 2023-03-09T18:11:32.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:32 smithi038 ceph-mon[92422]: pgmap v6701: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 85 KiB/s wr, 9 op/s 2023-03-09T18:11:33.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12065.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:11:33.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11603.387728 secs 2023-03-09T18:11:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11603.387728 secs 2023-03-09T18:11:33.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11603.387728 secs 2023-03-09T18:11:34.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:34 smithi122 ceph-mon[93869]: pgmap v6702: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 66 KiB/s wr, 7 op/s 2023-03-09T18:11:34.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:34 smithi181 ceph-mon[94189]: pgmap v6702: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 66 KiB/s wr, 7 op/s 2023-03-09T18:11:34.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:34 smithi038 ceph-mon[92422]: pgmap v6702: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 66 KiB/s wr, 7 op/s 2023-03-09T18:11:35.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11606.095702 secs 2023-03-09T18:11:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11606.095702 secs 2023-03-09T18:11:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11606.095702 secs 2023-03-09T18:11:36.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:36 smithi122 ceph-mon[93869]: pgmap v6703: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 66 KiB/s wr, 6 op/s 2023-03-09T18:11:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:36 smithi181 ceph-mon[94189]: pgmap v6703: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 66 KiB/s wr, 6 op/s 2023-03-09T18:11:36.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:36 smithi038 ceph-mon[92422]: pgmap v6703: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 66 KiB/s wr, 6 op/s 2023-03-09T18:11:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12070.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:11:38.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11608.387860 secs 2023-03-09T18:11:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:38 smithi122 ceph-mon[93869]: pgmap v6704: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 120 KiB/s wr, 9 op/s 2023-03-09T18:11:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11608.387860 secs 2023-03-09T18:11:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:38 smithi181 ceph-mon[94189]: pgmap v6704: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 120 KiB/s wr, 9 op/s 2023-03-09T18:11:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11608.387860 secs 2023-03-09T18:11:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:38 smithi038 ceph-mon[92422]: pgmap v6704: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 120 KiB/s wr, 9 op/s 2023-03-09T18:11:40.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:40 smithi122 ceph-mon[93869]: pgmap v6705: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 95 KiB/s wr, 8 op/s 2023-03-09T18:11:40.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11611.095761 secs 2023-03-09T18:11:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:40 smithi181 ceph-mon[94189]: pgmap v6705: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 95 KiB/s wr, 8 op/s 2023-03-09T18:11:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11611.095761 secs 2023-03-09T18:11:40.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:40 smithi038 ceph-mon[92422]: pgmap v6705: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 95 KiB/s wr, 8 op/s 2023-03-09T18:11:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11611.095761 secs 2023-03-09T18:11:42.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:42 smithi122 ceph-mon[93869]: pgmap v6706: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 167 KiB/s wr, 11 op/s 2023-03-09T18:11:42.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:42 smithi181 ceph-mon[94189]: pgmap v6706: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 167 KiB/s wr, 11 op/s 2023-03-09T18:11:42.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:42 smithi038 ceph-mon[92422]: pgmap v6706: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 167 KiB/s wr, 11 op/s 2023-03-09T18:11:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12075.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:11:43.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:43 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:11:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:43 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:11:43.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:43 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:11:44.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:44 smithi122 ceph-mon[93869]: pgmap v6707: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 146 KiB/s wr, 10 op/s 2023-03-09T18:11:44.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:44 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:11:44.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:44 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:11:44.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:44 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:11:44.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:44 smithi181 ceph-mon[94189]: pgmap v6707: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 146 KiB/s wr, 10 op/s 2023-03-09T18:11:44.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:44 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:11:44.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:44 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:11:44.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:44 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:11:44.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:44 smithi038 ceph-mon[92422]: pgmap v6707: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 146 KiB/s wr, 10 op/s 2023-03-09T18:11:44.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:44 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:11:44.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:44 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:11:44.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:44 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:11:45.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11616.095822 secs 2023-03-09T18:11:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11616.095822 secs 2023-03-09T18:11:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11616.095822 secs 2023-03-09T18:11:46.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:46 smithi122 ceph-mon[93869]: pgmap v6708: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 146 KiB/s wr, 9 op/s 2023-03-09T18:11:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:46 smithi181 ceph-mon[94189]: pgmap v6708: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 146 KiB/s wr, 9 op/s 2023-03-09T18:11:46.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:46 smithi038 ceph-mon[92422]: pgmap v6708: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 146 KiB/s wr, 9 op/s 2023-03-09T18:11:47.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:11:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:11:47.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:11:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12080.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:11:48.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:48 smithi122 ceph-mon[93869]: pgmap v6709: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 246 KiB/s wr, 13 op/s 2023-03-09T18:11:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:48 smithi181 ceph-mon[94189]: pgmap v6709: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 246 KiB/s wr, 13 op/s 2023-03-09T18:11:48.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:48 smithi038 ceph-mon[92422]: pgmap v6709: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 246 KiB/s wr, 13 op/s 2023-03-09T18:11:50.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:50 smithi122 ceph-mon[93869]: pgmap v6710: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 202 KiB/s wr, 11 op/s 2023-03-09T18:11:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11621.095929 secs 2023-03-09T18:11:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:50 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:11:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:50 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:11:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:50 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:11:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:50 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:11:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:50 smithi181 ceph-mon[94189]: pgmap v6710: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 202 KiB/s wr, 11 op/s 2023-03-09T18:11:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11621.095929 secs 2023-03-09T18:11:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:50 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:11:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:50 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:11:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:50 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:11:50.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:50 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:11:50.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:50 smithi038 ceph-mon[92422]: pgmap v6710: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 202 KiB/s wr, 11 op/s 2023-03-09T18:11:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11621.095929 secs 2023-03-09T18:11:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:50 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:11:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:50 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:11:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:50 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:11:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:50 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:11:51.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:11:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:11:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:11:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:11:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:11:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:11:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:11:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:11:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:11:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:11:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:11:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:11:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:11:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:11:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:11:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:11:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:11:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:11:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:11:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:11:51.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:11:51.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:11:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:11:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:11:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:11:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:11:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:11:52.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:11:52.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:11:52.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:11:52.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:52 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T18:11:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:52 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T18:11:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:52 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi038 to 197.2M 2023-03-09T18:11:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:52 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi038 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-03-09T18:11:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:52 smithi122 ceph-mon[93869]: pgmap v6711: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 230 KiB/s wr, 12 op/s 2023-03-09T18:11:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:11:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:11:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:11:52.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:11:52.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:11:52.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:11:52.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:11:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:52 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T18:11:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:52 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T18:11:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:52 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi038 to 197.2M 2023-03-09T18:11:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:52 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi038 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-03-09T18:11:52.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:52 smithi181 ceph-mon[94189]: pgmap v6711: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 230 KiB/s wr, 12 op/s 2023-03-09T18:11:52.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:11:52.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:11:52.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:11:52.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:11:52.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:11:52.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:11:52.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:11:52.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:52 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T18:11:52.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:52 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T18:11:52.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:52 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi038 to 197.2M 2023-03-09T18:11:52.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:52 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi038 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-03-09T18:11:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:52 smithi038 ceph-mon[92422]: pgmap v6711: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 230 KiB/s wr, 12 op/s 2023-03-09T18:11:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:11:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:11:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:11:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:11:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:11:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:11:53.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:11:53.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12085.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:11:53.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:53 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T18:11:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:53 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T18:11:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:53 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T18:11:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:53 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T18:11:53.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:53 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T18:11:53.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:53 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T18:11:54.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:54 smithi122 ceph-mon[93869]: pgmap v6712: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 177 KiB/s wr, 11 op/s 2023-03-09T18:11:54.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:54 smithi181 ceph-mon[94189]: pgmap v6712: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 177 KiB/s wr, 11 op/s 2023-03-09T18:11:54.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:54 smithi038 ceph-mon[92422]: pgmap v6712: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 177 KiB/s wr, 11 op/s 2023-03-09T18:11:55.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11626.095982 secs 2023-03-09T18:11:55.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:55 smithi122 ceph-mon[93869]: pgmap v6713: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 167 KiB/s wr, 9 op/s 2023-03-09T18:11:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11626.095982 secs 2023-03-09T18:11:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:55 smithi181 ceph-mon[94189]: pgmap v6713: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 167 KiB/s wr, 9 op/s 2023-03-09T18:11:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11626.095982 secs 2023-03-09T18:11:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:55 smithi038 ceph-mon[92422]: pgmap v6713: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 167 KiB/s wr, 9 op/s 2023-03-09T18:11:58.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11628.388357 secs 2023-03-09T18:11:58.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:11:58 smithi122 ceph-mon[93869]: pgmap v6714: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 284 KiB/s wr, 12 op/s 2023-03-09T18:11:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11628.388357 secs 2023-03-09T18:11:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:11:58 smithi181 ceph-mon[94189]: pgmap v6714: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 284 KiB/s wr, 12 op/s 2023-03-09T18:11:58.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11628.388357 secs 2023-03-09T18:11:58.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:11:58 smithi038 ceph-mon[92422]: pgmap v6714: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 284 KiB/s wr, 12 op/s 2023-03-09T18:11:58.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12090.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:12:00.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:12:00 smithi122 ceph-mon[93869]: pgmap v6715: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 184 KiB/s wr, 9 op/s 2023-03-09T18:12:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:12:00 smithi181 ceph-mon[94189]: pgmap v6715: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 184 KiB/s wr, 9 op/s 2023-03-09T18:12:00.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:12:00 smithi038 ceph-mon[92422]: pgmap v6715: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 184 KiB/s wr, 9 op/s 2023-03-09T18:12:01.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:12:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11631.096044 secs 2023-03-09T18:12:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:12:01 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:12:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:12:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:12:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:12:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:12:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:12:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11631.096044 secs 2023-03-09T18:12:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:12:01 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:12:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:12:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:12:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:12:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:12:01.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:12:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11631.096044 secs 2023-03-09T18:12:01.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:12:01 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:12:01.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:12:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:12:01.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:12:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:12:02.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:12:02 smithi038 ceph-mon[92422]: pgmap v6716: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 228 KiB/s wr, 11 op/s 2023-03-09T18:12:02.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:12:02 smithi122 ceph-mon[93869]: pgmap v6716: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 228 KiB/s wr, 11 op/s 2023-03-09T18:12:02.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:12:02 smithi181 ceph-mon[94189]: pgmap v6716: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 228 KiB/s wr, 11 op/s 2023-03-09T18:12:03.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:12:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11633.388503 secs 2023-03-09T18:12:03.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12095.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:12:03.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:12:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11633.388503 secs 2023-03-09T18:12:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:12:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11633.388503 secs 2023-03-09T18:12:04.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:12:04 smithi038 ceph-mon[92422]: pgmap v6717: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 212 KiB/s wr, 21 op/s 2023-03-09T18:12:04.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:12:04 smithi122 ceph-mon[93869]: pgmap v6717: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 212 KiB/s wr, 21 op/s 2023-03-09T18:12:04.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:12:04 smithi181 ceph-mon[94189]: pgmap v6717: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 212 KiB/s wr, 21 op/s 2023-03-09T18:12:05.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:12:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11636.096106 secs 2023-03-09T18:12:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:12:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11636.096106 secs 2023-03-09T18:12:05.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:12:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11636.096106 secs 2023-03-09T18:12:06.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:12:06 smithi122 ceph-mon[93869]: pgmap v6718: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 193 KiB/s wr, 20 op/s 2023-03-09T18:12:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:12:06 smithi181 ceph-mon[94189]: pgmap v6718: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 193 KiB/s wr, 20 op/s 2023-03-09T18:12:06.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:12:06 smithi038 ceph-mon[92422]: pgmap v6718: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 193 KiB/s wr, 20 op/s 2023-03-09T18:12:08.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12100.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:12:08.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:12:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11638.388641 secs 2023-03-09T18:12:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:12:08 smithi122 ceph-mon[93869]: pgmap v6719: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 215 KiB/s wr, 24 op/s 2023-03-09T18:12:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:12:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11638.388641 secs 2023-03-09T18:12:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:12:08 smithi181 ceph-mon[94189]: pgmap v6719: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 215 KiB/s wr, 24 op/s 2023-03-09T18:12:08.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:12:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11638.388641 secs 2023-03-09T18:12:08.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:12:08 smithi038 ceph-mon[92422]: pgmap v6719: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 215 KiB/s wr, 24 op/s 2023-03-09T18:12:10.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:12:10 smithi122 ceph-mon[93869]: pgmap v6720: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 108 KiB/s wr, 22 op/s 2023-03-09T18:12:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:12:10 smithi122 ceph-mon[93869]: osdmap e633: 12 total, 12 up, 12 in 2023-03-09T18:12:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:12:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11641.096168 secs 2023-03-09T18:12:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:12:10 smithi181 ceph-mon[94189]: pgmap v6720: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 108 KiB/s wr, 22 op/s 2023-03-09T18:12:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:12:10 smithi181 ceph-mon[94189]: osdmap e633: 12 total, 12 up, 12 in 2023-03-09T18:12:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:12:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11641.096168 secs 2023-03-09T18:12:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:12:10 smithi038 ceph-mon[92422]: pgmap v6720: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 108 KiB/s wr, 22 op/s 2023-03-09T18:12:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:12:10 smithi038 ceph-mon[92422]: osdmap e633: 12 total, 12 up, 12 in 2023-03-09T18:12:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:12:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11641.096168 secs 2023-03-09T18:12:12.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:12:12 smithi122 ceph-mon[93869]: pgmap v6722: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 91 KiB/s wr, 24 op/s 2023-03-09T18:12:12.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:12:12 smithi181 ceph-mon[94189]: pgmap v6722: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 91 KiB/s wr, 24 op/s 2023-03-09T18:12:12.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:12:12 smithi038 ceph-mon[92422]: pgmap v6722: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 91 KiB/s wr, 24 op/s 2023-03-09T18:12:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12105.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:12:13.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:12:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11643.388781 secs 2023-03-09T18:12:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:12:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11643.388781 secs 2023-03-09T18:12:13.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:12:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11643.388781 secs 2023-03-09T18:12:14.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:12:14 smithi122 ceph-mon[93869]: pgmap v6723: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 77 KiB/s wr, 10 op/s 2023-03-09T18:12:14.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:12:14 smithi181 ceph-mon[94189]: pgmap v6723: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 77 KiB/s wr, 10 op/s 2023-03-09T18:12:14.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:12:14 smithi038 ceph-mon[92422]: pgmap v6723: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 77 KiB/s wr, 10 op/s 2023-03-09T18:12:15.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:12:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11646.096227 secs 2023-03-09T18:12:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:12:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11646.096227 secs 2023-03-09T18:12:15.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:12:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11646.096227 secs 2023-03-09T18:12:16.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:12:16 smithi122 ceph-mon[93869]: pgmap v6724: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 78 KiB/s wr, 10 op/s 2023-03-09T18:12:16.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:12:16 smithi122 ceph-mon[93869]: osdmap e634: 12 total, 12 up, 12 in 2023-03-09T18:12:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:12:16 smithi181 ceph-mon[94189]: pgmap v6724: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 78 KiB/s wr, 10 op/s 2023-03-09T18:12:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:12:16 smithi181 ceph-mon[94189]: osdmap e634: 12 total, 12 up, 12 in 2023-03-09T18:12:16.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:12:16 smithi038 ceph-mon[92422]: pgmap v6724: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 78 KiB/s wr, 10 op/s 2023-03-09T18:12:16.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:12:16 smithi038 ceph-mon[92422]: osdmap e634: 12 total, 12 up, 12 in 2023-03-09T18:12:18.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12110.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:12:18.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:12:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11648.388905 secs 2023-03-09T18:12:18.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:12:18 smithi122 ceph-mon[93869]: pgmap v6726: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 9 op/s 2023-03-09T18:12:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:12:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11648.388905 secs 2023-03-09T18:12:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:12:18 smithi181 ceph-mon[94189]: pgmap v6726: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 9 op/s 2023-03-09T18:12:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:12:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11648.388905 secs 2023-03-09T18:12:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:12:18 smithi038 ceph-mon[92422]: pgmap v6726: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 9 op/s 2023-03-09T18:12:20.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:12:20 smithi122 ceph-mon[93869]: pgmap v6727: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 41 KiB/s wr, 8 op/s 2023-03-09T18:12:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:12:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11651.096286 secs 2023-03-09T18:12:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:12:20 smithi181 ceph-mon[94189]: pgmap v6727: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 41 KiB/s wr, 8 op/s 2023-03-09T18:12:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:12:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11651.096286 secs 2023-03-09T18:12:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:12:20 smithi038 ceph-mon[92422]: pgmap v6727: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 41 KiB/s wr, 8 op/s 2023-03-09T18:12:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:12:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11651.096286 secs 2023-03-09T18:12:22.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:12:22 smithi122 ceph-mon[93869]: pgmap v6728: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 7 op/s 2023-03-09T18:12:22.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:12:22 smithi181 ceph-mon[94189]: pgmap v6728: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 7 op/s 2023-03-09T18:12:22.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:12:22 smithi038 ceph-mon[92422]: pgmap v6728: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 7 op/s 2023-03-09T18:12:23.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12115.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:12:23.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:12:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11653.389016 secs 2023-03-09T18:12:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:12:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11653.389016 secs 2023-03-09T18:12:23.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:12:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11653.389016 secs 2023-03-09T18:12:24.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:12:24 smithi122 ceph-mon[93869]: pgmap v6729: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 2.0 KiB/s wr, 8 op/s 2023-03-09T18:12:24.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:12:24 smithi181 ceph-mon[94189]: pgmap v6729: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 2.0 KiB/s wr, 8 op/s 2023-03-09T18:12:24.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:12:24 smithi038 ceph-mon[92422]: pgmap v6729: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 2.0 KiB/s wr, 8 op/s 2023-03-09T18:12:25.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:12:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11656.096370 secs 2023-03-09T18:12:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:12:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11656.096370 secs 2023-03-09T18:12:25.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:12:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11656.096370 secs 2023-03-09T18:12:26.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:12:26 smithi122 ceph-mon[93869]: pgmap v6730: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1023 B/s wr, 7 op/s 2023-03-09T18:12:26.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:12:26 smithi181 ceph-mon[94189]: pgmap v6730: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1023 B/s wr, 7 op/s 2023-03-09T18:12:26.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:12:26 smithi038 ceph-mon[92422]: pgmap v6730: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1023 B/s wr, 7 op/s 2023-03-09T18:12:28.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12120.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:12:28.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:12:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11658.389130 secs 2023-03-09T18:12:28.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:12:28 smithi122 ceph-mon[93869]: pgmap v6731: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 39 KiB/s wr, 8 op/s 2023-03-09T18:12:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:12:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11658.389130 secs 2023-03-09T18:12:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:12:28 smithi181 ceph-mon[94189]: pgmap v6731: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 39 KiB/s wr, 8 op/s 2023-03-09T18:12:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:12:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11658.389130 secs 2023-03-09T18:12:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:12:28 smithi038 ceph-mon[92422]: pgmap v6731: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 39 KiB/s wr, 8 op/s 2023-03-09T18:12:30.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:12:30 smithi122 ceph-mon[93869]: pgmap v6732: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 35 KiB/s wr, 7 op/s 2023-03-09T18:12:30.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:12:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11661.096429 secs 2023-03-09T18:12:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:12:30 smithi181 ceph-mon[94189]: pgmap v6732: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 35 KiB/s wr, 7 op/s 2023-03-09T18:12:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:12:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11661.096429 secs 2023-03-09T18:12:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:12:30 smithi038 ceph-mon[92422]: pgmap v6732: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 35 KiB/s wr, 7 op/s 2023-03-09T18:12:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:12:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11661.096429 secs 2023-03-09T18:12:32.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:12:32 smithi122 ceph-mon[93869]: pgmap v6733: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 35 KiB/s wr, 9 op/s 2023-03-09T18:12:32.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:12:32 smithi181 ceph-mon[94189]: pgmap v6733: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 35 KiB/s wr, 9 op/s 2023-03-09T18:12:32.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:12:32 smithi038 ceph-mon[92422]: pgmap v6733: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 35 KiB/s wr, 9 op/s 2023-03-09T18:12:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12125.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:12:34.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:12:34 smithi122 ceph-mon[93869]: pgmap v6734: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 35 KiB/s wr, 8 op/s 2023-03-09T18:12:34.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:12:34 smithi181 ceph-mon[94189]: pgmap v6734: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 35 KiB/s wr, 8 op/s 2023-03-09T18:12:34.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:12:34 smithi038 ceph-mon[92422]: pgmap v6734: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 35 KiB/s wr, 8 op/s 2023-03-09T18:12:35.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:12:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11666.096506 secs 2023-03-09T18:12:35.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:12:35 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:12:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:12:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11666.096506 secs 2023-03-09T18:12:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:12:35 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:12:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:12:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11666.096506 secs 2023-03-09T18:12:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:12:35 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:12:36.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:12:36 smithi122 ceph-mon[93869]: pgmap v6735: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 35 KiB/s wr, 7 op/s 2023-03-09T18:12:36.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:12:36 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:12:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:12:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:12:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:12:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:12:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:12:36 smithi181 ceph-mon[94189]: pgmap v6735: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 35 KiB/s wr, 7 op/s 2023-03-09T18:12:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:12:36 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:12:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:12:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:12:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:12:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:12:36.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:12:36 smithi038 ceph-mon[92422]: pgmap v6735: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 35 KiB/s wr, 7 op/s 2023-03-09T18:12:36.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:12:36 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:12:36.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:12:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:12:36.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:12:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:12:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12130.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:12:38.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:12:38 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 11668.389374 secs 2023-03-09T18:12:38.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:12:38 smithi122 ceph-mon[93869]: pgmap v6736: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 89 KiB/s wr, 10 op/s 2023-03-09T18:12:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:12:38 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 11668.389374 secs 2023-03-09T18:12:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:12:38 smithi181 ceph-mon[94189]: pgmap v6736: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 89 KiB/s wr, 10 op/s 2023-03-09T18:12:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:12:38 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 11668.389374 secs 2023-03-09T18:12:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:12:38 smithi038 ceph-mon[92422]: pgmap v6736: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 89 KiB/s wr, 10 op/s 2023-03-09T18:12:40.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:12:40 smithi122 ceph-mon[93869]: pgmap v6737: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 65 KiB/s wr, 8 op/s 2023-03-09T18:12:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:12:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:12:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:12:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:12:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:12:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11671.096570 secs 2023-03-09T18:12:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:12:40 smithi181 ceph-mon[94189]: pgmap v6737: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 65 KiB/s wr, 8 op/s 2023-03-09T18:12:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:12:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:12:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:12:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:12:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:12:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11671.096570 secs 2023-03-09T18:12:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:12:40 smithi038 ceph-mon[92422]: pgmap v6737: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 65 KiB/s wr, 8 op/s 2023-03-09T18:12:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:12:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:12:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:12:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:12:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:12:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11671.096570 secs 2023-03-09T18:12:41.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:12:41 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:12:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:12:41 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:12:41.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:12:41 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:12:42.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:12:42 smithi122 ceph-mon[93869]: pgmap v6738: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 137 KiB/s wr, 11 op/s 2023-03-09T18:12:42.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:12:42 smithi181 ceph-mon[94189]: pgmap v6738: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 137 KiB/s wr, 11 op/s 2023-03-09T18:12:42.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:12:42 smithi038 ceph-mon[92422]: pgmap v6738: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 137 KiB/s wr, 11 op/s 2023-03-09T18:12:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12135.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:12:43.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:12:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11673.389495 secs 2023-03-09T18:12:43.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:12:43 smithi122 ceph-mon[93869]: pgmap v6739: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 148 KiB/s wr, 10 op/s 2023-03-09T18:12:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:12:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11673.389495 secs 2023-03-09T18:12:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:12:43 smithi181 ceph-mon[94189]: pgmap v6739: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 148 KiB/s wr, 10 op/s 2023-03-09T18:12:43.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:12:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11673.389495 secs 2023-03-09T18:12:43.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:12:43 smithi038 ceph-mon[92422]: pgmap v6739: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 148 KiB/s wr, 10 op/s 2023-03-09T18:12:44.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:12:44 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:12:44.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:12:44 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:12:44.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:12:44 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:12:44.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:12:44 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:12:44.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:12:44 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:12:44.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:12:44 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:12:45.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:12:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11676.096628 secs 2023-03-09T18:12:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:12:45 smithi122 ceph-mon[93869]: pgmap v6740: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 148 KiB/s wr, 9 op/s 2023-03-09T18:12:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:12:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11676.096628 secs 2023-03-09T18:12:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:12:45 smithi181 ceph-mon[94189]: pgmap v6740: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 148 KiB/s wr, 9 op/s 2023-03-09T18:12:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:12:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11676.096628 secs 2023-03-09T18:12:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:12:45 smithi038 ceph-mon[92422]: pgmap v6740: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 148 KiB/s wr, 9 op/s 2023-03-09T18:12:48.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:12:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11678.389614 secs 2023-03-09T18:12:48.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:12:48 smithi122 ceph-mon[93869]: pgmap v6741: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 210 KiB/s wr, 12 op/s 2023-03-09T18:12:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:12:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11678.389614 secs 2023-03-09T18:12:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:12:48 smithi181 ceph-mon[94189]: pgmap v6741: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 210 KiB/s wr, 12 op/s 2023-03-09T18:12:48.747 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:12:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11678.389614 secs 2023-03-09T18:12:48.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:12:48 smithi038 ceph-mon[92422]: pgmap v6741: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 210 KiB/s wr, 12 op/s 2023-03-09T18:12:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12140.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:12:50.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:12:50 smithi038 ceph-mon[92422]: pgmap v6742: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 166 KiB/s wr, 10 op/s 2023-03-09T18:12:50.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:12:50 smithi122 ceph-mon[93869]: pgmap v6742: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 166 KiB/s wr, 10 op/s 2023-03-09T18:12:50.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:12:50 smithi181 ceph-mon[94189]: pgmap v6742: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 166 KiB/s wr, 10 op/s 2023-03-09T18:12:51.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:12:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11681.096700 secs 2023-03-09T18:12:51.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:12:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11681.096700 secs 2023-03-09T18:12:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:12:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11681.096700 secs 2023-03-09T18:12:52.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:12:52 smithi038 ceph-mon[92422]: pgmap v6743: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 217 KiB/s wr, 12 op/s 2023-03-09T18:12:52.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:12:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:12:52.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:12:52 smithi122 ceph-mon[93869]: pgmap v6743: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 217 KiB/s wr, 12 op/s 2023-03-09T18:12:52.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:12:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:12:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:12:52 smithi181 ceph-mon[94189]: pgmap v6743: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 217 KiB/s wr, 12 op/s 2023-03-09T18:12:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:12:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:12:53.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:12:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11683.389711 secs 2023-03-09T18:12:53.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:12:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:12:53.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:12:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:12:53.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:12:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:12:53.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12145.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:12:53.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:12:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11683.389711 secs 2023-03-09T18:12:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:12:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:12:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:12:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:12:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:12:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:12:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:12:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11683.389711 secs 2023-03-09T18:12:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:12:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:12:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:12:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:12:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:12:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:12:54.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:12:54 smithi122 ceph-mon[93869]: pgmap v6744: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 154 KiB/s wr, 10 op/s 2023-03-09T18:12:54.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:12:54 smithi181 ceph-mon[94189]: pgmap v6744: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 154 KiB/s wr, 10 op/s 2023-03-09T18:12:54.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:12:54 smithi038 ceph-mon[92422]: pgmap v6744: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 154 KiB/s wr, 10 op/s 2023-03-09T18:12:55.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:12:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11686.096781 secs 2023-03-09T18:12:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:12:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11686.096781 secs 2023-03-09T18:12:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:12:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11686.096781 secs 2023-03-09T18:12:56.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:12:56 smithi122 ceph-mon[93869]: pgmap v6745: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 144 KiB/s wr, 8 op/s 2023-03-09T18:12:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:12:56 smithi181 ceph-mon[94189]: pgmap v6745: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 144 KiB/s wr, 8 op/s 2023-03-09T18:12:56.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:12:56 smithi038 ceph-mon[92422]: pgmap v6745: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 144 KiB/s wr, 8 op/s 2023-03-09T18:12:58.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12150.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:12:58.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:12:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11688.389835 secs 2023-03-09T18:12:58.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:12:58 smithi122 ceph-mon[93869]: pgmap v6746: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 215 KiB/s wr, 11 op/s 2023-03-09T18:12:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:12:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11688.389835 secs 2023-03-09T18:12:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:12:58 smithi181 ceph-mon[94189]: pgmap v6746: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 215 KiB/s wr, 11 op/s 2023-03-09T18:12:58.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:12:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11688.389835 secs 2023-03-09T18:12:58.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:12:58 smithi038 ceph-mon[92422]: pgmap v6746: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 215 KiB/s wr, 11 op/s 2023-03-09T18:13:00.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:13:00 smithi122 ceph-mon[93869]: pgmap v6747: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 196 KiB/s wr, 9 op/s 2023-03-09T18:13:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:13:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11691.096843 secs 2023-03-09T18:13:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:13:00 smithi181 ceph-mon[94189]: pgmap v6747: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 196 KiB/s wr, 9 op/s 2023-03-09T18:13:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:13:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11691.096843 secs 2023-03-09T18:13:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:13:00 smithi038 ceph-mon[92422]: pgmap v6747: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 196 KiB/s wr, 9 op/s 2023-03-09T18:13:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:13:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11691.096843 secs 2023-03-09T18:13:02.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:13:02 smithi122 ceph-mon[93869]: pgmap v6748: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 186 KiB/s wr, 11 op/s 2023-03-09T18:13:02.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:13:02 smithi181 ceph-mon[94189]: pgmap v6748: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 186 KiB/s wr, 11 op/s 2023-03-09T18:13:02.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:13:02 smithi038 ceph-mon[92422]: pgmap v6748: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 186 KiB/s wr, 11 op/s 2023-03-09T18:13:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12155.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:13:03.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:13:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11693.389967 secs 2023-03-09T18:13:03.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:13:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11693.389967 secs 2023-03-09T18:13:03.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:13:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11693.389967 secs 2023-03-09T18:13:04.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:13:04 smithi122 ceph-mon[93869]: pgmap v6749: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 228 KiB/s wr, 22 op/s 2023-03-09T18:13:04.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:13:04 smithi181 ceph-mon[94189]: pgmap v6749: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 228 KiB/s wr, 22 op/s 2023-03-09T18:13:04.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:13:04 smithi038 ceph-mon[92422]: pgmap v6749: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 228 KiB/s wr, 22 op/s 2023-03-09T18:13:05.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:13:05 smithi122 ceph-mon[93869]: osdmap e635: 12 total, 12 up, 12 in 2023-03-09T18:13:05.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:13:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11696.096908 secs 2023-03-09T18:13:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:13:05 smithi181 ceph-mon[94189]: osdmap e635: 12 total, 12 up, 12 in 2023-03-09T18:13:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:13:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11696.096908 secs 2023-03-09T18:13:05.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:13:05 smithi038 ceph-mon[92422]: osdmap e635: 12 total, 12 up, 12 in 2023-03-09T18:13:05.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:13:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11696.096908 secs 2023-03-09T18:13:06.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:13:06 smithi122 ceph-mon[93869]: pgmap v6751: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 261 KiB/s wr, 25 op/s 2023-03-09T18:13:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:13:06 smithi181 ceph-mon[94189]: pgmap v6751: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 261 KiB/s wr, 25 op/s 2023-03-09T18:13:06.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:13:06 smithi038 ceph-mon[92422]: pgmap v6751: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 261 KiB/s wr, 25 op/s 2023-03-09T18:13:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12160.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:13:08.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:13:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11698.390087 secs 2023-03-09T18:13:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:13:08 smithi122 ceph-mon[93869]: pgmap v6752: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 203 KiB/s wr, 27 op/s 2023-03-09T18:13:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:13:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11698.390087 secs 2023-03-09T18:13:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:13:08 smithi181 ceph-mon[94189]: pgmap v6752: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 203 KiB/s wr, 27 op/s 2023-03-09T18:13:08.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:13:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11698.390087 secs 2023-03-09T18:13:08.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:13:08 smithi038 ceph-mon[92422]: pgmap v6752: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 203 KiB/s wr, 27 op/s 2023-03-09T18:13:10.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:13:10 smithi122 ceph-mon[93869]: pgmap v6753: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 163 KiB/s wr, 26 op/s 2023-03-09T18:13:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:13:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11701.096975 secs 2023-03-09T18:13:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:13:10 smithi181 ceph-mon[94189]: pgmap v6753: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 163 KiB/s wr, 26 op/s 2023-03-09T18:13:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:13:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11701.096975 secs 2023-03-09T18:13:10.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:13:10 smithi038 ceph-mon[92422]: pgmap v6753: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 163 KiB/s wr, 26 op/s 2023-03-09T18:13:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:13:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11701.096975 secs 2023-03-09T18:13:12.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:13:12 smithi038 ceph-mon[92422]: pgmap v6754: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 163 KiB/s wr, 26 op/s 2023-03-09T18:13:12.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:13:12 smithi038 ceph-mon[92422]: osdmap e636: 12 total, 12 up, 12 in 2023-03-09T18:13:12.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:13:12 smithi122 ceph-mon[93869]: pgmap v6754: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 163 KiB/s wr, 26 op/s 2023-03-09T18:13:12.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:13:12 smithi122 ceph-mon[93869]: osdmap e636: 12 total, 12 up, 12 in 2023-03-09T18:13:12.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:13:12 smithi181 ceph-mon[94189]: pgmap v6754: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 163 KiB/s wr, 26 op/s 2023-03-09T18:13:12.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:13:12 smithi181 ceph-mon[94189]: osdmap e636: 12 total, 12 up, 12 in 2023-03-09T18:13:13.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:13:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11703.390201 secs 2023-03-09T18:13:13.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12165.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:13:13.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:13:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11703.390201 secs 2023-03-09T18:13:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:13:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11703.390201 secs 2023-03-09T18:13:14.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:13:14 smithi038 ceph-mon[92422]: pgmap v6756: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 90 KiB/s wr, 12 op/s 2023-03-09T18:13:14.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:13:14 smithi122 ceph-mon[93869]: pgmap v6756: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 90 KiB/s wr, 12 op/s 2023-03-09T18:13:14.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:13:14 smithi181 ceph-mon[94189]: pgmap v6756: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 90 KiB/s wr, 12 op/s 2023-03-09T18:13:15.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:13:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11706.097054 secs 2023-03-09T18:13:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:13:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11706.097054 secs 2023-03-09T18:13:15.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:13:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11706.097054 secs 2023-03-09T18:13:16.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:13:16 smithi122 ceph-mon[93869]: pgmap v6757: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 79 KiB/s wr, 11 op/s 2023-03-09T18:13:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:13:16 smithi181 ceph-mon[94189]: pgmap v6757: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 79 KiB/s wr, 11 op/s 2023-03-09T18:13:16.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:13:16 smithi038 ceph-mon[92422]: pgmap v6757: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 79 KiB/s wr, 11 op/s 2023-03-09T18:13:18.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12170.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:13:18.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:13:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11708.390300 secs 2023-03-09T18:13:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:13:18 smithi122 ceph-mon[93869]: pgmap v6758: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 8 op/s 2023-03-09T18:13:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:13:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11708.390300 secs 2023-03-09T18:13:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:13:18 smithi181 ceph-mon[94189]: pgmap v6758: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 8 op/s 2023-03-09T18:13:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:13:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11708.390300 secs 2023-03-09T18:13:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:13:18 smithi038 ceph-mon[92422]: pgmap v6758: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 8 op/s 2023-03-09T18:13:20.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:13:20 smithi122 ceph-mon[93869]: pgmap v6759: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 41 KiB/s wr, 8 op/s 2023-03-09T18:13:20.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:13:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11711.097116 secs 2023-03-09T18:13:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:13:20 smithi181 ceph-mon[94189]: pgmap v6759: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 41 KiB/s wr, 8 op/s 2023-03-09T18:13:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:13:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11711.097116 secs 2023-03-09T18:13:20.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:13:20 smithi038 ceph-mon[92422]: pgmap v6759: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 41 KiB/s wr, 8 op/s 2023-03-09T18:13:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:13:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11711.097116 secs 2023-03-09T18:13:22.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:13:22 smithi122 ceph-mon[93869]: pgmap v6760: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 41 KiB/s wr, 8 op/s 2023-03-09T18:13:22.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:13:22 smithi181 ceph-mon[94189]: pgmap v6760: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 41 KiB/s wr, 8 op/s 2023-03-09T18:13:22.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:13:22 smithi038 ceph-mon[92422]: pgmap v6760: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 41 KiB/s wr, 8 op/s 2023-03-09T18:13:23.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12175.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:13:23.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:13:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11713.390394 secs 2023-03-09T18:13:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:13:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11713.390394 secs 2023-03-09T18:13:23.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:13:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11713.390394 secs 2023-03-09T18:13:24.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:13:24 smithi122 ceph-mon[93869]: pgmap v6761: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 93 KiB/s wr, 8 op/s 2023-03-09T18:13:24.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:13:24 smithi181 ceph-mon[94189]: pgmap v6761: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 93 KiB/s wr, 8 op/s 2023-03-09T18:13:24.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:13:24 smithi038 ceph-mon[92422]: pgmap v6761: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 93 KiB/s wr, 8 op/s 2023-03-09T18:13:25.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:13:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11716.097177 secs 2023-03-09T18:13:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:13:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11716.097177 secs 2023-03-09T18:13:25.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:13:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11716.097177 secs 2023-03-09T18:13:26.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:13:26 smithi122 ceph-mon[93869]: pgmap v6762: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 59 KiB/s wr, 6 op/s 2023-03-09T18:13:26.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:13:26 smithi181 ceph-mon[94189]: pgmap v6762: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 59 KiB/s wr, 6 op/s 2023-03-09T18:13:26.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:13:26 smithi038 ceph-mon[92422]: pgmap v6762: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 59 KiB/s wr, 6 op/s 2023-03-09T18:13:28.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12180.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:13:28.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:13:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11718.390514 secs 2023-03-09T18:13:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:13:28 smithi122 ceph-mon[93869]: pgmap v6763: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 60 KiB/s wr, 8 op/s 2023-03-09T18:13:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:13:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11718.390514 secs 2023-03-09T18:13:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:13:28 smithi181 ceph-mon[94189]: pgmap v6763: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 60 KiB/s wr, 8 op/s 2023-03-09T18:13:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:13:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11718.390514 secs 2023-03-09T18:13:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:13:28 smithi038 ceph-mon[92422]: pgmap v6763: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 60 KiB/s wr, 8 op/s 2023-03-09T18:13:30.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:13:30 smithi122 ceph-mon[93869]: pgmap v6764: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 7 op/s 2023-03-09T18:13:30.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:13:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11721.097264 secs 2023-03-09T18:13:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:13:30 smithi181 ceph-mon[94189]: pgmap v6764: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 7 op/s 2023-03-09T18:13:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:13:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11721.097264 secs 2023-03-09T18:13:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:13:30 smithi038 ceph-mon[92422]: pgmap v6764: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 7 op/s 2023-03-09T18:13:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:13:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11721.097264 secs 2023-03-09T18:13:32.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:13:32 smithi122 ceph-mon[93869]: pgmap v6765: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 59 KiB/s wr, 8 op/s 2023-03-09T18:13:32.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:13:32 smithi181 ceph-mon[94189]: pgmap v6765: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 59 KiB/s wr, 8 op/s 2023-03-09T18:13:32.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:13:32 smithi038 ceph-mon[92422]: pgmap v6765: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 59 KiB/s wr, 8 op/s 2023-03-09T18:13:33.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12185.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:13:33.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:13:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11723.390636 secs 2023-03-09T18:13:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:13:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11723.390636 secs 2023-03-09T18:13:33.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:13:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11723.390636 secs 2023-03-09T18:13:34.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:13:34 smithi122 ceph-mon[93869]: pgmap v6766: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 136 KiB/s wr, 8 op/s 2023-03-09T18:13:34.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:13:34 smithi181 ceph-mon[94189]: pgmap v6766: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 136 KiB/s wr, 8 op/s 2023-03-09T18:13:34.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:13:34 smithi038 ceph-mon[92422]: pgmap v6766: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 136 KiB/s wr, 8 op/s 2023-03-09T18:13:35.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:13:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11726.097326 secs 2023-03-09T18:13:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:13:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11726.097326 secs 2023-03-09T18:13:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:13:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11726.097326 secs 2023-03-09T18:13:36.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:13:36 smithi122 ceph-mon[93869]: pgmap v6767: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 79 KiB/s wr, 6 op/s 2023-03-09T18:13:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:13:36 smithi181 ceph-mon[94189]: pgmap v6767: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 79 KiB/s wr, 6 op/s 2023-03-09T18:13:36.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:13:36 smithi038 ceph-mon[92422]: pgmap v6767: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 79 KiB/s wr, 6 op/s 2023-03-09T18:13:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12190.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:13:38.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:13:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11728.390757 secs 2023-03-09T18:13:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:13:38 smithi122 ceph-mon[93869]: pgmap v6768: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 79 KiB/s wr, 9 op/s 2023-03-09T18:13:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:13:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11728.390757 secs 2023-03-09T18:13:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:13:38 smithi181 ceph-mon[94189]: pgmap v6768: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 79 KiB/s wr, 9 op/s 2023-03-09T18:13:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:13:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11728.390757 secs 2023-03-09T18:13:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:13:38 smithi038 ceph-mon[92422]: pgmap v6768: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 79 KiB/s wr, 9 op/s 2023-03-09T18:13:40.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:13:40 smithi122 ceph-mon[93869]: pgmap v6769: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 97 KiB/s wr, 8 op/s 2023-03-09T18:13:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:13:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11731.097387 secs 2023-03-09T18:13:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:13:40 smithi181 ceph-mon[94189]: pgmap v6769: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 97 KiB/s wr, 8 op/s 2023-03-09T18:13:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:13:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11731.097387 secs 2023-03-09T18:13:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:13:40 smithi038 ceph-mon[92422]: pgmap v6769: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 97 KiB/s wr, 8 op/s 2023-03-09T18:13:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:13:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11731.097387 secs 2023-03-09T18:13:42.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:13:42 smithi122 ceph-mon[93869]: pgmap v6770: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 97 KiB/s wr, 9 op/s 2023-03-09T18:13:42.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:13:42 smithi181 ceph-mon[94189]: pgmap v6770: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 97 KiB/s wr, 9 op/s 2023-03-09T18:13:42.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:13:42 smithi038 ceph-mon[92422]: pgmap v6770: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 97 KiB/s wr, 9 op/s 2023-03-09T18:13:43.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12195.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:13:43.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:13:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11733.390843 secs 2023-03-09T18:13:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:13:43 smithi122 ceph-mon[93869]: pgmap v6771: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 235 KiB/s wr, 11 op/s 2023-03-09T18:13:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:13:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11733.390843 secs 2023-03-09T18:13:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:13:43 smithi181 ceph-mon[94189]: pgmap v6771: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 235 KiB/s wr, 11 op/s 2023-03-09T18:13:43.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:13:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11733.390843 secs 2023-03-09T18:13:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:13:43 smithi038 ceph-mon[92422]: pgmap v6771: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 235 KiB/s wr, 11 op/s 2023-03-09T18:13:46.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:13:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11736.097449 secs 2023-03-09T18:13:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:13:46 smithi122 ceph-mon[93869]: pgmap v6772: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 157 KiB/s wr, 9 op/s 2023-03-09T18:13:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:13:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11736.097449 secs 2023-03-09T18:13:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:13:46 smithi181 ceph-mon[94189]: pgmap v6772: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 157 KiB/s wr, 9 op/s 2023-03-09T18:13:46.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:13:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11736.097449 secs 2023-03-09T18:13:46.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:13:46 smithi038 ceph-mon[92422]: pgmap v6772: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 157 KiB/s wr, 9 op/s 2023-03-09T18:13:48.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:13:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11738.390956 secs 2023-03-09T18:13:48.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:13:48 smithi038 ceph-mon[92422]: pgmap v6773: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 157 KiB/s wr, 11 op/s 2023-03-09T18:13:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12200.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:13:48.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:13:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11738.390956 secs 2023-03-09T18:13:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:13:48 smithi122 ceph-mon[93869]: pgmap v6773: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 157 KiB/s wr, 11 op/s 2023-03-09T18:13:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:13:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11738.390956 secs 2023-03-09T18:13:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:13:48 smithi181 ceph-mon[94189]: pgmap v6773: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 157 KiB/s wr, 11 op/s 2023-03-09T18:13:50.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:13:50 smithi122 ceph-mon[93869]: pgmap v6774: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 215 KiB/s wr, 12 op/s 2023-03-09T18:13:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:13:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11741.097510 secs 2023-03-09T18:13:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:13:50 smithi181 ceph-mon[94189]: pgmap v6774: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 215 KiB/s wr, 12 op/s 2023-03-09T18:13:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:13:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11741.097510 secs 2023-03-09T18:13:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:13:50 smithi038 ceph-mon[92422]: pgmap v6774: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 215 KiB/s wr, 12 op/s 2023-03-09T18:13:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:13:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11741.097510 secs 2023-03-09T18:13:52.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:13:52 smithi122 ceph-mon[93869]: pgmap v6775: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 197 KiB/s wr, 12 op/s 2023-03-09T18:13:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:13:52 smithi181 ceph-mon[94189]: pgmap v6775: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 197 KiB/s wr, 12 op/s 2023-03-09T18:13:52.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:13:52 smithi038 ceph-mon[92422]: pgmap v6775: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 197 KiB/s wr, 12 op/s 2023-03-09T18:13:53.803 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12205.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:13:53.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:13:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:13:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:13:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:13:53.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:13:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:13:54.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:13:54 smithi122 ceph-mon[93869]: pgmap v6776: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 374 KiB/s wr, 13 op/s 2023-03-09T18:13:54.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:13:54 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:13:54.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:13:54 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:13:54.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:13:54 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:13:54.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:13:54 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:13:54.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:13:54 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:13:54.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:13:54 smithi181 ceph-mon[94189]: pgmap v6776: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 374 KiB/s wr, 13 op/s 2023-03-09T18:13:54.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:13:54 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:13:54.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:13:54 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:13:54.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:13:54 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:13:54.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:13:54 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:13:54.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:13:54 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:13:54.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:13:54 smithi038 ceph-mon[92422]: pgmap v6776: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 374 KiB/s wr, 13 op/s 2023-03-09T18:13:54.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:13:54 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:13:54.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:13:54 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:13:54.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:13:54 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:13:54.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:13:54 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:13:54.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:13:54 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:13:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:13:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11746.097569 secs 2023-03-09T18:13:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:13:55 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:13:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:13:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11746.097569 secs 2023-03-09T18:13:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:13:55 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:13:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:13:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11746.097569 secs 2023-03-09T18:13:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:13:55 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:13:56.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:13:56 smithi122 ceph-mon[93869]: pgmap v6777: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 236 KiB/s wr, 9 op/s 2023-03-09T18:13:56.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:13:56 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:13:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:13:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:13:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:13:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:13:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:13:56 smithi181 ceph-mon[94189]: pgmap v6777: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 236 KiB/s wr, 9 op/s 2023-03-09T18:13:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:13:56 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:13:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:13:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:13:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:13:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:13:56.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:13:56 smithi038 ceph-mon[92422]: pgmap v6777: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 236 KiB/s wr, 9 op/s 2023-03-09T18:13:56.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:13:56 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:13:56.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:13:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:13:56.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:13:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:13:58.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12210.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:13:58.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:13:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11748.391173 secs 2023-03-09T18:13:58.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:13:58 smithi122 ceph-mon[93869]: pgmap v6778: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 237 KiB/s wr, 11 op/s 2023-03-09T18:13:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:13:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11748.391173 secs 2023-03-09T18:13:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:13:58 smithi181 ceph-mon[94189]: pgmap v6778: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 237 KiB/s wr, 11 op/s 2023-03-09T18:13:58.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:13:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11748.391173 secs 2023-03-09T18:13:58.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:13:58 smithi038 ceph-mon[92422]: pgmap v6778: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 237 KiB/s wr, 11 op/s 2023-03-09T18:14:00.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:14:00 smithi122 ceph-mon[93869]: pgmap v6779: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 237 KiB/s wr, 11 op/s 2023-03-09T18:14:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:14:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:14:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:14:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:14:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:14:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11751.097653 secs 2023-03-09T18:14:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:14:00 smithi181 ceph-mon[94189]: pgmap v6779: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 237 KiB/s wr, 11 op/s 2023-03-09T18:14:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:14:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:14:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:14:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:14:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:14:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11751.097653 secs 2023-03-09T18:14:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:14:00 smithi038 ceph-mon[92422]: pgmap v6779: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 237 KiB/s wr, 11 op/s 2023-03-09T18:14:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:14:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:14:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:14:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:14:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:14:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11751.097653 secs 2023-03-09T18:14:01.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:14:01 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:14:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:14:01 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:14:01.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:14:01 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:14:02.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:14:02 smithi122 ceph-mon[93869]: pgmap v6780: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 179 KiB/s wr, 10 op/s 2023-03-09T18:14:02.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:14:02 smithi181 ceph-mon[94189]: pgmap v6780: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 179 KiB/s wr, 10 op/s 2023-03-09T18:14:02.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:14:02 smithi038 ceph-mon[92422]: pgmap v6780: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 179 KiB/s wr, 10 op/s 2023-03-09T18:14:03.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12215.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:14:03.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:14:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11753.391294 secs 2023-03-09T18:14:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:14:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11753.391294 secs 2023-03-09T18:14:03.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:14:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11753.391294 secs 2023-03-09T18:14:04.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:14:04 smithi122 ceph-mon[93869]: pgmap v6781: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 252 KiB/s wr, 22 op/s 2023-03-09T18:14:04.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:14:04 smithi181 ceph-mon[94189]: pgmap v6781: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 252 KiB/s wr, 22 op/s 2023-03-09T18:14:04.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:14:04 smithi038 ceph-mon[92422]: pgmap v6781: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 252 KiB/s wr, 22 op/s 2023-03-09T18:14:05.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:14:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11756.097739 secs 2023-03-09T18:14:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:14:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11756.097739 secs 2023-03-09T18:14:05.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:14:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11756.097739 secs 2023-03-09T18:14:06.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:14:06 smithi122 ceph-mon[93869]: pgmap v6782: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 74 KiB/s wr, 19 op/s 2023-03-09T18:14:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:14:06 smithi181 ceph-mon[94189]: pgmap v6782: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 74 KiB/s wr, 19 op/s 2023-03-09T18:14:06.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:14:06 smithi038 ceph-mon[92422]: pgmap v6782: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 74 KiB/s wr, 19 op/s 2023-03-09T18:14:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12220.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:14:08.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:14:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11758.391397 secs 2023-03-09T18:14:08.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:14:08 smithi122 ceph-mon[93869]: pgmap v6783: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 96 KiB/s wr, 24 op/s 2023-03-09T18:14:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:14:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11758.391397 secs 2023-03-09T18:14:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:14:08 smithi181 ceph-mon[94189]: pgmap v6783: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 96 KiB/s wr, 24 op/s 2023-03-09T18:14:08.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:14:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11758.391397 secs 2023-03-09T18:14:08.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:14:08 smithi038 ceph-mon[92422]: pgmap v6783: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 96 KiB/s wr, 24 op/s 2023-03-09T18:14:10.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:14:10 smithi122 ceph-mon[93869]: pgmap v6784: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 108 KiB/s wr, 23 op/s 2023-03-09T18:14:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:14:10 smithi122 ceph-mon[93869]: osdmap e637: 12 total, 12 up, 12 in 2023-03-09T18:14:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:14:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11761.098992 secs 2023-03-09T18:14:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:14:10 smithi181 ceph-mon[94189]: pgmap v6784: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 108 KiB/s wr, 23 op/s 2023-03-09T18:14:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:14:10 smithi181 ceph-mon[94189]: osdmap e637: 12 total, 12 up, 12 in 2023-03-09T18:14:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:14:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11761.098992 secs 2023-03-09T18:14:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:14:10 smithi038 ceph-mon[92422]: pgmap v6784: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 108 KiB/s wr, 23 op/s 2023-03-09T18:14:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:14:10 smithi038 ceph-mon[92422]: osdmap e637: 12 total, 12 up, 12 in 2023-03-09T18:14:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:14:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11761.098992 secs 2023-03-09T18:14:12.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:14:12 smithi122 ceph-mon[93869]: pgmap v6786: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 129 KiB/s wr, 26 op/s 2023-03-09T18:14:12.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:14:12 smithi181 ceph-mon[94189]: pgmap v6786: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 129 KiB/s wr, 26 op/s 2023-03-09T18:14:12.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:14:12 smithi038 ceph-mon[92422]: pgmap v6786: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 129 KiB/s wr, 26 op/s 2023-03-09T18:14:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12225.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:14:13.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:14:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11763.391526 secs 2023-03-09T18:14:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:14:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11763.391526 secs 2023-03-09T18:14:13.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:14:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11763.391526 secs 2023-03-09T18:14:14.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:14:14 smithi122 ceph-mon[93869]: pgmap v6787: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 67 KiB/s wr, 10 op/s 2023-03-09T18:14:14.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:14:14 smithi181 ceph-mon[94189]: pgmap v6787: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 67 KiB/s wr, 10 op/s 2023-03-09T18:14:14.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:14:14 smithi038 ceph-mon[92422]: pgmap v6787: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 67 KiB/s wr, 10 op/s 2023-03-09T18:14:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:14:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11766.099054 secs 2023-03-09T18:14:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:14:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11766.099054 secs 2023-03-09T18:14:15.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:14:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11766.099054 secs 2023-03-09T18:14:16.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:14:16 smithi122 ceph-mon[93869]: pgmap v6788: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 67 KiB/s wr, 11 op/s 2023-03-09T18:14:16.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:14:16 smithi122 ceph-mon[93869]: osdmap e638: 12 total, 12 up, 12 in 2023-03-09T18:14:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:14:16 smithi181 ceph-mon[94189]: pgmap v6788: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 67 KiB/s wr, 11 op/s 2023-03-09T18:14:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:14:16 smithi181 ceph-mon[94189]: osdmap e638: 12 total, 12 up, 12 in 2023-03-09T18:14:16.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:14:16 smithi038 ceph-mon[92422]: pgmap v6788: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 67 KiB/s wr, 11 op/s 2023-03-09T18:14:16.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:14:16 smithi038 ceph-mon[92422]: osdmap e638: 12 total, 12 up, 12 in 2023-03-09T18:14:17.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:14:17 smithi122 ceph-mon[93869]: pgmap v6790: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 33 KiB/s wr, 8 op/s 2023-03-09T18:14:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:14:17 smithi181 ceph-mon[94189]: pgmap v6790: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 33 KiB/s wr, 8 op/s 2023-03-09T18:14:17.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:14:17 smithi038 ceph-mon[92422]: pgmap v6790: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 33 KiB/s wr, 8 op/s 2023-03-09T18:14:18.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12230.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:14:18.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:14:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11768.391654 secs 2023-03-09T18:14:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:14:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11768.391654 secs 2023-03-09T18:14:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:14:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11768.391654 secs 2023-03-09T18:14:19.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:14:19 smithi122 ceph-mon[93869]: pgmap v6791: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 51 KiB/s wr, 9 op/s 2023-03-09T18:14:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:14:19 smithi181 ceph-mon[94189]: pgmap v6791: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 51 KiB/s wr, 9 op/s 2023-03-09T18:14:19.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:14:19 smithi038 ceph-mon[92422]: pgmap v6791: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 51 KiB/s wr, 9 op/s 2023-03-09T18:14:20.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:14:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11771.099116 secs 2023-03-09T18:14:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:14:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11771.099116 secs 2023-03-09T18:14:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:14:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11771.099116 secs 2023-03-09T18:14:22.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:14:22 smithi122 ceph-mon[93869]: pgmap v6792: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 50 KiB/s wr, 8 op/s 2023-03-09T18:14:22.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:14:22 smithi181 ceph-mon[94189]: pgmap v6792: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 50 KiB/s wr, 8 op/s 2023-03-09T18:14:22.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:14:22 smithi038 ceph-mon[92422]: pgmap v6792: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 50 KiB/s wr, 8 op/s 2023-03-09T18:14:23.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12235.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:14:23.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:14:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11773.391768 secs 2023-03-09T18:14:23.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:14:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11773.391768 secs 2023-03-09T18:14:23.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:14:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11773.391768 secs 2023-03-09T18:14:24.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:14:24 smithi122 ceph-mon[93869]: pgmap v6793: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 25 KiB/s wr, 8 op/s 2023-03-09T18:14:24.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:14:24 smithi181 ceph-mon[94189]: pgmap v6793: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 25 KiB/s wr, 8 op/s 2023-03-09T18:14:24.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:14:24 smithi038 ceph-mon[92422]: pgmap v6793: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 25 KiB/s wr, 8 op/s 2023-03-09T18:14:25.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:14:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11776.099173 secs 2023-03-09T18:14:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:14:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11776.099173 secs 2023-03-09T18:14:25.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:14:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11776.099173 secs 2023-03-09T18:14:26.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:14:26 smithi122 ceph-mon[93869]: pgmap v6794: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 66 KiB/s wr, 8 op/s 2023-03-09T18:14:26.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:14:26 smithi181 ceph-mon[94189]: pgmap v6794: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 66 KiB/s wr, 8 op/s 2023-03-09T18:14:26.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:14:26 smithi038 ceph-mon[92422]: pgmap v6794: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 66 KiB/s wr, 8 op/s 2023-03-09T18:14:28.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12240.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:14:28.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:14:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11778.391903 secs 2023-03-09T18:14:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:14:28 smithi122 ceph-mon[93869]: pgmap v6795: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 60 KiB/s wr, 7 op/s 2023-03-09T18:14:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:14:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11778.391903 secs 2023-03-09T18:14:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:14:28 smithi181 ceph-mon[94189]: pgmap v6795: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 60 KiB/s wr, 7 op/s 2023-03-09T18:14:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:14:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11778.391903 secs 2023-03-09T18:14:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:14:28 smithi038 ceph-mon[92422]: pgmap v6795: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 60 KiB/s wr, 7 op/s 2023-03-09T18:14:30.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:14:30 smithi122 ceph-mon[93869]: pgmap v6796: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 55 KiB/s wr, 8 op/s 2023-03-09T18:14:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:14:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11781.099237 secs 2023-03-09T18:14:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:14:30 smithi181 ceph-mon[94189]: pgmap v6796: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 55 KiB/s wr, 8 op/s 2023-03-09T18:14:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:14:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11781.099237 secs 2023-03-09T18:14:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:14:30 smithi038 ceph-mon[92422]: pgmap v6796: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 55 KiB/s wr, 8 op/s 2023-03-09T18:14:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:14:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11781.099237 secs 2023-03-09T18:14:32.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:14:32 smithi122 ceph-mon[93869]: pgmap v6797: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 36 KiB/s wr, 8 op/s 2023-03-09T18:14:32.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:14:32 smithi181 ceph-mon[94189]: pgmap v6797: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 36 KiB/s wr, 8 op/s 2023-03-09T18:14:32.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:14:32 smithi038 ceph-mon[92422]: pgmap v6797: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 36 KiB/s wr, 8 op/s 2023-03-09T18:14:33.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12245.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:14:34.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:14:34 smithi122 ceph-mon[93869]: pgmap v6798: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 36 KiB/s wr, 7 op/s 2023-03-09T18:14:34.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:14:34 smithi181 ceph-mon[94189]: pgmap v6798: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 36 KiB/s wr, 7 op/s 2023-03-09T18:14:34.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:14:34 smithi038 ceph-mon[92422]: pgmap v6798: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 36 KiB/s wr, 7 op/s 2023-03-09T18:14:35.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:14:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11786.099300 secs 2023-03-09T18:14:35.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:14:35 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:14:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:14:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11786.099300 secs 2023-03-09T18:14:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:14:35 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:14:35.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:14:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11786.099300 secs 2023-03-09T18:14:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:14:35 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:14:36.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:14:36 smithi122 ceph-mon[93869]: pgmap v6799: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 90 KiB/s wr, 7 op/s 2023-03-09T18:14:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:14:36 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:14:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:14:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:14:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:14:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:14:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:14:36 smithi181 ceph-mon[94189]: pgmap v6799: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 90 KiB/s wr, 7 op/s 2023-03-09T18:14:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:14:36 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:14:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:14:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:14:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:14:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:14:36.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:14:36 smithi038 ceph-mon[92422]: pgmap v6799: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 90 KiB/s wr, 7 op/s 2023-03-09T18:14:36.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:14:36 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:14:36.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:14:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:14:36.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:14:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:14:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12250.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:14:38.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:14:38 smithi122 ceph-mon[93869]: pgmap v6800: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 56 KiB/s wr, 8 op/s 2023-03-09T18:14:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:14:38 smithi181 ceph-mon[94189]: pgmap v6800: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 56 KiB/s wr, 8 op/s 2023-03-09T18:14:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:14:38 smithi038 ceph-mon[92422]: pgmap v6800: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 56 KiB/s wr, 8 op/s 2023-03-09T18:14:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:14:40 smithi122 ceph-mon[93869]: pgmap v6801: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 140 KiB/s wr, 8 op/s 2023-03-09T18:14:40.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:14:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11791.099361 secs 2023-03-09T18:14:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:14:40 smithi181 ceph-mon[94189]: pgmap v6801: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 140 KiB/s wr, 8 op/s 2023-03-09T18:14:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:14:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11791.099361 secs 2023-03-09T18:14:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:14:40 smithi038 ceph-mon[92422]: pgmap v6801: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 140 KiB/s wr, 8 op/s 2023-03-09T18:14:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:14:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11791.099361 secs 2023-03-09T18:14:42.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:14:42 smithi122 ceph-mon[93869]: pgmap v6802: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 140 KiB/s wr, 9 op/s 2023-03-09T18:14:42.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:14:42 smithi181 ceph-mon[94189]: pgmap v6802: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 140 KiB/s wr, 9 op/s 2023-03-09T18:14:42.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:14:42 smithi038 ceph-mon[92422]: pgmap v6802: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 140 KiB/s wr, 9 op/s 2023-03-09T18:14:43.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12255.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:14:44.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:14:44 smithi122 ceph-mon[93869]: pgmap v6803: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 150 KiB/s wr, 9 op/s 2023-03-09T18:14:44.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:14:44 smithi181 ceph-mon[94189]: pgmap v6803: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 150 KiB/s wr, 9 op/s 2023-03-09T18:14:44.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:14:44 smithi038 ceph-mon[92422]: pgmap v6803: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 150 KiB/s wr, 9 op/s 2023-03-09T18:14:45.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:14:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11796.099422 secs 2023-03-09T18:14:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:14:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11796.099422 secs 2023-03-09T18:14:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:14:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11796.099422 secs 2023-03-09T18:14:46.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:14:46 smithi122 ceph-mon[93869]: pgmap v6804: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 213 KiB/s wr, 9 op/s 2023-03-09T18:14:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:14:46 smithi181 ceph-mon[94189]: pgmap v6804: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 213 KiB/s wr, 9 op/s 2023-03-09T18:14:46.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:14:46 smithi038 ceph-mon[92422]: pgmap v6804: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 213 KiB/s wr, 9 op/s 2023-03-09T18:14:48.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12260.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:14:48.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:14:48 smithi122 ceph-mon[93869]: pgmap v6805: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 158 KiB/s wr, 11 op/s 2023-03-09T18:14:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:14:48 smithi181 ceph-mon[94189]: pgmap v6805: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 158 KiB/s wr, 11 op/s 2023-03-09T18:14:48.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:14:48 smithi038 ceph-mon[92422]: pgmap v6805: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 158 KiB/s wr, 11 op/s 2023-03-09T18:14:50.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:14:50 smithi122 ceph-mon[93869]: pgmap v6806: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 231 KiB/s wr, 10 op/s 2023-03-09T18:14:50.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:14:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11801.099481 secs 2023-03-09T18:14:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:14:50 smithi181 ceph-mon[94189]: pgmap v6806: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 231 KiB/s wr, 10 op/s 2023-03-09T18:14:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:14:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11801.099481 secs 2023-03-09T18:14:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:14:50 smithi038 ceph-mon[92422]: pgmap v6806: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 231 KiB/s wr, 10 op/s 2023-03-09T18:14:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:14:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11801.099481 secs 2023-03-09T18:14:52.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:14:52 smithi122 ceph-mon[93869]: pgmap v6807: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 147 KiB/s wr, 11 op/s 2023-03-09T18:14:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:14:52 smithi181 ceph-mon[94189]: pgmap v6807: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 147 KiB/s wr, 11 op/s 2023-03-09T18:14:52.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:14:52 smithi038 ceph-mon[92422]: pgmap v6807: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 147 KiB/s wr, 11 op/s 2023-03-09T18:14:53.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12265.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:14:54.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:14:54 smithi122 ceph-mon[93869]: pgmap v6808: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 157 KiB/s wr, 10 op/s 2023-03-09T18:14:54.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:14:54 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:14:54.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:14:54 smithi181 ceph-mon[94189]: pgmap v6808: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 157 KiB/s wr, 10 op/s 2023-03-09T18:14:54.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:14:54 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:14:54.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:14:54 smithi038 ceph-mon[92422]: pgmap v6808: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 157 KiB/s wr, 10 op/s 2023-03-09T18:14:54.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:14:54 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:14:56.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:14:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11806.099542 secs 2023-03-09T18:14:56.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:14:55 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:14:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:14:55 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:14:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:14:55 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:14:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:14:55 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:14:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:14:55 smithi122 ceph-mon[93869]: pgmap v6809: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 179 KiB/s wr, 9 op/s 2023-03-09T18:14:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:14:55 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:14:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:14:55 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:14:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:14:55 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:14:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:14:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11806.099542 secs 2023-03-09T18:14:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:14:55 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:14:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:14:55 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:14:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:14:55 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:14:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:14:55 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:14:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:14:55 smithi181 ceph-mon[94189]: pgmap v6809: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 179 KiB/s wr, 9 op/s 2023-03-09T18:14:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:14:55 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:14:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:14:55 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:14:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:14:55 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:14:56.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:14:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11806.099542 secs 2023-03-09T18:14:56.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:14:55 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:14:56.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:14:55 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:14:56.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:14:55 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:14:56.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:14:55 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:14:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:14:55 smithi038 ceph-mon[92422]: pgmap v6809: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 179 KiB/s wr, 9 op/s 2023-03-09T18:14:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:14:55 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:14:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:14:55 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:14:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:14:55 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:14:58.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12270.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:14:58.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:14:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11808.392634 secs 2023-03-09T18:14:58.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:14:58 smithi122 ceph-mon[93869]: pgmap v6810: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 190 KiB/s wr, 11 op/s 2023-03-09T18:14:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:14:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11808.392634 secs 2023-03-09T18:14:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:14:58 smithi181 ceph-mon[94189]: pgmap v6810: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 190 KiB/s wr, 11 op/s 2023-03-09T18:14:58.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:14:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11808.392634 secs 2023-03-09T18:14:58.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:14:58 smithi038 ceph-mon[92422]: pgmap v6810: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 190 KiB/s wr, 11 op/s 2023-03-09T18:15:00.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:15:00 smithi122 ceph-mon[93869]: pgmap v6811: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 200 KiB/s wr, 10 op/s 2023-03-09T18:15:00.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:15:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11811.099620 secs 2023-03-09T18:15:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:15:00 smithi181 ceph-mon[94189]: pgmap v6811: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 200 KiB/s wr, 10 op/s 2023-03-09T18:15:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:15:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11811.099620 secs 2023-03-09T18:15:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:15:00 smithi038 ceph-mon[92422]: pgmap v6811: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 200 KiB/s wr, 10 op/s 2023-03-09T18:15:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:15:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11811.099620 secs 2023-03-09T18:15:01.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:15:01 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:15:01.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:15:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:15:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:15:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:15:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:15:01 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:15:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:15:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:15:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:15:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:15:01.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:15:01 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:15:01.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:15:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:15:01.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:15:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:15:02.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:15:02 smithi122 ceph-mon[93869]: pgmap v6812: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 150 KiB/s wr, 12 op/s 2023-03-09T18:15:02.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:15:02 smithi181 ceph-mon[94189]: pgmap v6812: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 150 KiB/s wr, 12 op/s 2023-03-09T18:15:02.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:15:02 smithi038 ceph-mon[92422]: pgmap v6812: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 150 KiB/s wr, 12 op/s 2023-03-09T18:15:03.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12275.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:15:03.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:15:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11813.392760 secs 2023-03-09T18:15:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:15:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11813.392760 secs 2023-03-09T18:15:03.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:15:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11813.392760 secs 2023-03-09T18:15:04.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:15:04 smithi122 ceph-mon[93869]: pgmap v6813: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 207 KiB/s wr, 24 op/s 2023-03-09T18:15:04.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:15:04 smithi181 ceph-mon[94189]: pgmap v6813: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 207 KiB/s wr, 24 op/s 2023-03-09T18:15:04.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:15:04 smithi038 ceph-mon[92422]: pgmap v6813: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 207 KiB/s wr, 24 op/s 2023-03-09T18:15:05.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:15:05 smithi122 ceph-mon[93869]: osdmap e639: 12 total, 12 up, 12 in 2023-03-09T18:15:05.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:15:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11816.099680 secs 2023-03-09T18:15:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:15:05 smithi181 ceph-mon[94189]: osdmap e639: 12 total, 12 up, 12 in 2023-03-09T18:15:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:15:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11816.099680 secs 2023-03-09T18:15:05.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:15:05 smithi038 ceph-mon[92422]: osdmap e639: 12 total, 12 up, 12 in 2023-03-09T18:15:05.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:15:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11816.099680 secs 2023-03-09T18:15:06.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:15:06 smithi122 ceph-mon[93869]: pgmap v6815: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 197 KiB/s wr, 27 op/s 2023-03-09T18:15:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:15:06 smithi181 ceph-mon[94189]: pgmap v6815: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 197 KiB/s wr, 27 op/s 2023-03-09T18:15:06.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:15:06 smithi038 ceph-mon[92422]: pgmap v6815: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 197 KiB/s wr, 27 op/s 2023-03-09T18:15:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12280.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:15:08.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:15:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11818.392884 secs 2023-03-09T18:15:08.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:15:08 smithi122 ceph-mon[93869]: pgmap v6816: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 165 KiB/s wr, 27 op/s 2023-03-09T18:15:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:15:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11818.392884 secs 2023-03-09T18:15:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:15:08 smithi181 ceph-mon[94189]: pgmap v6816: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 165 KiB/s wr, 27 op/s 2023-03-09T18:15:08.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:15:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11818.392884 secs 2023-03-09T18:15:08.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:15:08 smithi038 ceph-mon[92422]: pgmap v6816: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 165 KiB/s wr, 27 op/s 2023-03-09T18:15:10.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:15:10 smithi122 ceph-mon[93869]: pgmap v6817: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 165 KiB/s wr, 26 op/s 2023-03-09T18:15:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:15:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11821.099764 secs 2023-03-09T18:15:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:15:10 smithi181 ceph-mon[94189]: pgmap v6817: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 165 KiB/s wr, 26 op/s 2023-03-09T18:15:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:15:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11821.099764 secs 2023-03-09T18:15:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:15:10 smithi038 ceph-mon[92422]: pgmap v6817: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 165 KiB/s wr, 26 op/s 2023-03-09T18:15:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:15:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11821.099764 secs 2023-03-09T18:15:12.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:15:12 smithi122 ceph-mon[93869]: pgmap v6818: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 138 KiB/s wr, 24 op/s 2023-03-09T18:15:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:15:12 smithi122 ceph-mon[93869]: osdmap e640: 12 total, 12 up, 12 in 2023-03-09T18:15:12.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:15:12 smithi181 ceph-mon[94189]: pgmap v6818: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 138 KiB/s wr, 24 op/s 2023-03-09T18:15:12.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:15:12 smithi181 ceph-mon[94189]: osdmap e640: 12 total, 12 up, 12 in 2023-03-09T18:15:12.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:15:12 smithi038 ceph-mon[92422]: pgmap v6818: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 138 KiB/s wr, 24 op/s 2023-03-09T18:15:12.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:15:12 smithi038 ceph-mon[92422]: osdmap e640: 12 total, 12 up, 12 in 2023-03-09T18:15:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12285.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:15:13.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:15:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11823.392984 secs 2023-03-09T18:15:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:15:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11823.392984 secs 2023-03-09T18:15:13.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:15:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11823.392984 secs 2023-03-09T18:15:14.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:15:14 smithi122 ceph-mon[93869]: pgmap v6820: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 124 KiB/s wr, 9 op/s 2023-03-09T18:15:14.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:15:14 smithi181 ceph-mon[94189]: pgmap v6820: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 124 KiB/s wr, 9 op/s 2023-03-09T18:15:14.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:15:14 smithi038 ceph-mon[92422]: pgmap v6820: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 124 KiB/s wr, 9 op/s 2023-03-09T18:15:15.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:15:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11826.099824 secs 2023-03-09T18:15:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:15:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11826.099824 secs 2023-03-09T18:15:15.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:15:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11826.099824 secs 2023-03-09T18:15:16.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:15:16 smithi122 ceph-mon[93869]: pgmap v6821: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 109 KiB/s wr, 8 op/s 2023-03-09T18:15:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:15:16 smithi181 ceph-mon[94189]: pgmap v6821: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 109 KiB/s wr, 8 op/s 2023-03-09T18:15:16.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:15:16 smithi038 ceph-mon[92422]: pgmap v6821: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 109 KiB/s wr, 8 op/s 2023-03-09T18:15:18.793 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12290.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:15:18.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:15:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11828.393100 secs 2023-03-09T18:15:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:15:18 smithi122 ceph-mon[93869]: pgmap v6822: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 54 KiB/s wr, 8 op/s 2023-03-09T18:15:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:15:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11828.393100 secs 2023-03-09T18:15:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:15:18 smithi181 ceph-mon[94189]: pgmap v6822: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 54 KiB/s wr, 8 op/s 2023-03-09T18:15:18.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:15:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11828.393100 secs 2023-03-09T18:15:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:15:18 smithi038 ceph-mon[92422]: pgmap v6822: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 54 KiB/s wr, 8 op/s 2023-03-09T18:15:20.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:15:20 smithi122 ceph-mon[93869]: pgmap v6823: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 42 KiB/s wr, 8 op/s 2023-03-09T18:15:20.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:15:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11831.099902 secs 2023-03-09T18:15:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:15:20 smithi181 ceph-mon[94189]: pgmap v6823: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 42 KiB/s wr, 8 op/s 2023-03-09T18:15:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:15:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11831.099902 secs 2023-03-09T18:15:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:15:20 smithi038 ceph-mon[92422]: pgmap v6823: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 42 KiB/s wr, 8 op/s 2023-03-09T18:15:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:15:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11831.099902 secs 2023-03-09T18:15:22.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:15:22 smithi122 ceph-mon[93869]: pgmap v6824: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 42 KiB/s wr, 7 op/s 2023-03-09T18:15:22.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:15:22 smithi181 ceph-mon[94189]: pgmap v6824: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 42 KiB/s wr, 7 op/s 2023-03-09T18:15:22.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:15:22 smithi038 ceph-mon[92422]: pgmap v6824: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 42 KiB/s wr, 7 op/s 2023-03-09T18:15:23.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12295.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:15:23.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:15:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11833.393216 secs 2023-03-09T18:15:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:15:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11833.393216 secs 2023-03-09T18:15:23.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:15:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11833.393216 secs 2023-03-09T18:15:24.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:15:24 smithi122 ceph-mon[93869]: pgmap v6825: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 59 KiB/s wr, 8 op/s 2023-03-09T18:15:24.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:15:24 smithi181 ceph-mon[94189]: pgmap v6825: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 59 KiB/s wr, 8 op/s 2023-03-09T18:15:24.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:15:24 smithi038 ceph-mon[92422]: pgmap v6825: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 59 KiB/s wr, 8 op/s 2023-03-09T18:15:25.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:15:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11836.099965 secs 2023-03-09T18:15:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:15:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11836.099965 secs 2023-03-09T18:15:25.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:15:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11836.099965 secs 2023-03-09T18:15:26.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:15:26 smithi122 ceph-mon[93869]: pgmap v6826: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 25 KiB/s wr, 6 op/s 2023-03-09T18:15:26.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:15:26 smithi181 ceph-mon[94189]: pgmap v6826: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 25 KiB/s wr, 6 op/s 2023-03-09T18:15:26.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:15:26 smithi038 ceph-mon[92422]: pgmap v6826: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 25 KiB/s wr, 6 op/s 2023-03-09T18:15:28.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12300.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:15:28.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:15:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11838.393326 secs 2023-03-09T18:15:28.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:15:28 smithi122 ceph-mon[93869]: pgmap v6827: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 61 KiB/s wr, 9 op/s 2023-03-09T18:15:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:15:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11838.393326 secs 2023-03-09T18:15:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:15:28 smithi181 ceph-mon[94189]: pgmap v6827: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 61 KiB/s wr, 9 op/s 2023-03-09T18:15:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:15:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11838.393326 secs 2023-03-09T18:15:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:15:28 smithi038 ceph-mon[92422]: pgmap v6827: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 61 KiB/s wr, 9 op/s 2023-03-09T18:15:30.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:15:30 smithi122 ceph-mon[93869]: pgmap v6828: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 60 KiB/s wr, 7 op/s 2023-03-09T18:15:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:15:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11841.100048 secs 2023-03-09T18:15:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:15:30 smithi181 ceph-mon[94189]: pgmap v6828: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 60 KiB/s wr, 7 op/s 2023-03-09T18:15:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:15:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11841.100048 secs 2023-03-09T18:15:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:15:30 smithi038 ceph-mon[92422]: pgmap v6828: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 60 KiB/s wr, 7 op/s 2023-03-09T18:15:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:15:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11841.100048 secs 2023-03-09T18:15:32.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:15:32 smithi122 ceph-mon[93869]: pgmap v6829: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 60 KiB/s wr, 8 op/s 2023-03-09T18:15:32.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:15:32 smithi181 ceph-mon[94189]: pgmap v6829: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 60 KiB/s wr, 8 op/s 2023-03-09T18:15:32.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:15:32 smithi038 ceph-mon[92422]: pgmap v6829: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 60 KiB/s wr, 8 op/s 2023-03-09T18:15:33.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12305.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:15:33.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:15:33 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 11843.393395 secs 2023-03-09T18:15:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:15:33 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 11843.393395 secs 2023-03-09T18:15:33.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:15:33 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 11843.393395 secs 2023-03-09T18:15:34.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:15:34 smithi122 ceph-mon[93869]: pgmap v6830: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 92 KiB/s wr, 8 op/s 2023-03-09T18:15:34.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:15:34 smithi181 ceph-mon[94189]: pgmap v6830: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 92 KiB/s wr, 8 op/s 2023-03-09T18:15:34.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:15:34 smithi038 ceph-mon[92422]: pgmap v6830: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 92 KiB/s wr, 8 op/s 2023-03-09T18:15:35.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:15:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11846.100110 secs 2023-03-09T18:15:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:15:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11846.100110 secs 2023-03-09T18:15:35.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:15:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11846.100110 secs 2023-03-09T18:15:36.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:15:36 smithi122 ceph-mon[93869]: pgmap v6831: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 68 KiB/s wr, 6 op/s 2023-03-09T18:15:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:15:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:15:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:15:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:15:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:15:36 smithi181 ceph-mon[94189]: pgmap v6831: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 68 KiB/s wr, 6 op/s 2023-03-09T18:15:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:15:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:15:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:15:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:15:36.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:15:36 smithi038 ceph-mon[92422]: pgmap v6831: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 68 KiB/s wr, 6 op/s 2023-03-09T18:15:36.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:15:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:15:36.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:15:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:15:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12310.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:15:38.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:15:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11848.393529 secs 2023-03-09T18:15:38.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:15:38 smithi122 ceph-mon[93869]: pgmap v6832: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 141 KiB/s wr, 9 op/s 2023-03-09T18:15:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:15:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11848.393529 secs 2023-03-09T18:15:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:15:38 smithi181 ceph-mon[94189]: pgmap v6832: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 141 KiB/s wr, 9 op/s 2023-03-09T18:15:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:15:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11848.393529 secs 2023-03-09T18:15:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:15:38 smithi038 ceph-mon[92422]: pgmap v6832: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 141 KiB/s wr, 9 op/s 2023-03-09T18:15:40.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:15:40 smithi122 ceph-mon[93869]: pgmap v6833: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 116 KiB/s wr, 7 op/s 2023-03-09T18:15:40.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:15:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:15:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:15:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:15:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:15:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11851.100171 secs 2023-03-09T18:15:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:15:40 smithi181 ceph-mon[94189]: pgmap v6833: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 116 KiB/s wr, 7 op/s 2023-03-09T18:15:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:15:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:15:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:15:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:15:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:15:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11851.100171 secs 2023-03-09T18:15:40.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:15:40 smithi038 ceph-mon[92422]: pgmap v6833: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 116 KiB/s wr, 7 op/s 2023-03-09T18:15:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:15:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:15:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:15:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:15:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:15:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11851.100171 secs 2023-03-09T18:15:42.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:15:42 smithi122 ceph-mon[93869]: pgmap v6834: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 116 KiB/s wr, 9 op/s 2023-03-09T18:15:42.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:15:42 smithi181 ceph-mon[94189]: pgmap v6834: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 116 KiB/s wr, 9 op/s 2023-03-09T18:15:42.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:15:42 smithi038 ceph-mon[92422]: pgmap v6834: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 116 KiB/s wr, 9 op/s 2023-03-09T18:15:43.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12315.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:15:43.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:15:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11853.393655 secs 2023-03-09T18:15:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:15:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11853.393655 secs 2023-03-09T18:15:43.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:15:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11853.393655 secs 2023-03-09T18:15:44.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:15:44 smithi122 ceph-mon[93869]: pgmap v6835: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 176 KiB/s wr, 10 op/s 2023-03-09T18:15:44.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:15:44 smithi181 ceph-mon[94189]: pgmap v6835: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 176 KiB/s wr, 10 op/s 2023-03-09T18:15:44.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:15:44 smithi038 ceph-mon[92422]: pgmap v6835: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 176 KiB/s wr, 10 op/s 2023-03-09T18:15:45.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:15:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11856.100231 secs 2023-03-09T18:15:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:15:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11856.100231 secs 2023-03-09T18:15:45.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:15:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11856.100231 secs 2023-03-09T18:15:46.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:15:46 smithi122 ceph-mon[93869]: pgmap v6836: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 143 KiB/s wr, 9 op/s 2023-03-09T18:15:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:15:46 smithi181 ceph-mon[94189]: pgmap v6836: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 143 KiB/s wr, 9 op/s 2023-03-09T18:15:46.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:15:46 smithi038 ceph-mon[92422]: pgmap v6836: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 143 KiB/s wr, 9 op/s 2023-03-09T18:15:47.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:15:47 smithi122 ceph-mon[93869]: pgmap v6837: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 255 KiB/s wr, 13 op/s 2023-03-09T18:15:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:15:47 smithi181 ceph-mon[94189]: pgmap v6837: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 255 KiB/s wr, 13 op/s 2023-03-09T18:15:47.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:15:47 smithi038 ceph-mon[92422]: pgmap v6837: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 255 KiB/s wr, 13 op/s 2023-03-09T18:15:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12320.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:15:48.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:15:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11858.393778 secs 2023-03-09T18:15:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:15:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11858.393778 secs 2023-03-09T18:15:48.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:15:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11858.393778 secs 2023-03-09T18:15:49.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:15:49 smithi122 ceph-mon[93869]: pgmap v6838: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 193 KiB/s wr, 11 op/s 2023-03-09T18:15:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:15:49 smithi181 ceph-mon[94189]: pgmap v6838: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 193 KiB/s wr, 11 op/s 2023-03-09T18:15:49.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:15:49 smithi038 ceph-mon[92422]: pgmap v6838: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 193 KiB/s wr, 11 op/s 2023-03-09T18:15:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:15:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11861.100293 secs 2023-03-09T18:15:50.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:15:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11861.100293 secs 2023-03-09T18:15:51.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:15:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11861.100293 secs 2023-03-09T18:15:52.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:15:52 smithi122 ceph-mon[93869]: pgmap v6839: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 183 KiB/s wr, 12 op/s 2023-03-09T18:15:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:15:52 smithi181 ceph-mon[94189]: pgmap v6839: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 183 KiB/s wr, 12 op/s 2023-03-09T18:15:52.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:15:52 smithi038 ceph-mon[92422]: pgmap v6839: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 183 KiB/s wr, 12 op/s 2023-03-09T18:15:53.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12325.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:15:53.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:15:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11863.393905 secs 2023-03-09T18:15:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:15:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11863.393905 secs 2023-03-09T18:15:53.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:15:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11863.393905 secs 2023-03-09T18:15:54.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:15:54 smithi122 ceph-mon[93869]: pgmap v6840: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 243 KiB/s wr, 12 op/s 2023-03-09T18:15:54.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:15:54 smithi181 ceph-mon[94189]: pgmap v6840: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 243 KiB/s wr, 12 op/s 2023-03-09T18:15:54.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:15:54 smithi038 ceph-mon[92422]: pgmap v6840: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 243 KiB/s wr, 12 op/s 2023-03-09T18:15:55.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:15:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11866.100357 secs 2023-03-09T18:15:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:15:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11866.100357 secs 2023-03-09T18:15:55.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:15:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11866.100357 secs 2023-03-09T18:15:56.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:15:56 smithi122 ceph-mon[93869]: pgmap v6841: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 183 KiB/s wr, 9 op/s 2023-03-09T18:15:56.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:15:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:15:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:15:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:15:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:15:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:15:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:15:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:15:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:15:56 smithi181 ceph-mon[94189]: pgmap v6841: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 183 KiB/s wr, 9 op/s 2023-03-09T18:15:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:15:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:15:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:15:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:15:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:15:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:15:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:15:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:15:56.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:15:56 smithi038 ceph-mon[92422]: pgmap v6841: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 183 KiB/s wr, 9 op/s 2023-03-09T18:15:56.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:15:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:15:56.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:15:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:15:56.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:15:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:15:56.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:15:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:15:58.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12330.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:15:58.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:15:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11868.394020 secs 2023-03-09T18:15:58.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:15:58 smithi122 ceph-mon[93869]: pgmap v6842: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 277 KiB/s wr, 12 op/s 2023-03-09T18:15:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:15:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11868.394020 secs 2023-03-09T18:15:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:15:58 smithi181 ceph-mon[94189]: pgmap v6842: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 277 KiB/s wr, 12 op/s 2023-03-09T18:15:58.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:15:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11868.394020 secs 2023-03-09T18:15:58.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:15:58 smithi038 ceph-mon[92422]: pgmap v6842: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 277 KiB/s wr, 12 op/s 2023-03-09T18:16:00.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:16:00 smithi122 ceph-mon[93869]: pgmap v6843: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 165 KiB/s wr, 9 op/s 2023-03-09T18:16:00.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:16:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11871.100452 secs 2023-03-09T18:16:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:16:00 smithi181 ceph-mon[94189]: pgmap v6843: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 165 KiB/s wr, 9 op/s 2023-03-09T18:16:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:16:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11871.100452 secs 2023-03-09T18:16:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:16:00 smithi038 ceph-mon[92422]: pgmap v6843: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 165 KiB/s wr, 9 op/s 2023-03-09T18:16:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:16:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11871.100452 secs 2023-03-09T18:16:02.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:16:02 smithi122 ceph-mon[93869]: pgmap v6844: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 185 KiB/s wr, 13 op/s 2023-03-09T18:16:02.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:16:02 smithi181 ceph-mon[94189]: pgmap v6844: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 185 KiB/s wr, 13 op/s 2023-03-09T18:16:02.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:16:02 smithi038 ceph-mon[92422]: pgmap v6844: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 185 KiB/s wr, 13 op/s 2023-03-09T18:16:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12335.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:16:03.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:16:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11873.394171 secs 2023-03-09T18:16:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:16:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11873.394171 secs 2023-03-09T18:16:03.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:16:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11873.394171 secs 2023-03-09T18:16:04.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:16:04 smithi122 ceph-mon[93869]: pgmap v6845: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 254 KiB/s wr, 24 op/s 2023-03-09T18:16:04.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:16:04 smithi181 ceph-mon[94189]: pgmap v6845: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 254 KiB/s wr, 24 op/s 2023-03-09T18:16:04.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:16:04 smithi038 ceph-mon[92422]: pgmap v6845: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 254 KiB/s wr, 24 op/s 2023-03-09T18:16:05.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:16:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11876.100511 secs 2023-03-09T18:16:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:16:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11876.100511 secs 2023-03-09T18:16:05.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:16:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11876.100511 secs 2023-03-09T18:16:06.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:16:06 smithi122 ceph-mon[93869]: pgmap v6846: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 194 KiB/s wr, 22 op/s 2023-03-09T18:16:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:16:06 smithi181 ceph-mon[94189]: pgmap v6846: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 194 KiB/s wr, 22 op/s 2023-03-09T18:16:06.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:16:06 smithi038 ceph-mon[92422]: pgmap v6846: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 194 KiB/s wr, 22 op/s 2023-03-09T18:16:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12340.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:16:08.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:16:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11878.394291 secs 2023-03-09T18:16:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:16:08 smithi122 ceph-mon[93869]: pgmap v6847: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 195 KiB/s wr, 25 op/s 2023-03-09T18:16:08.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:16:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11878.394291 secs 2023-03-09T18:16:08.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:16:08 smithi181 ceph-mon[94189]: pgmap v6847: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 195 KiB/s wr, 25 op/s 2023-03-09T18:16:08.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:16:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11878.394291 secs 2023-03-09T18:16:08.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:16:08 smithi038 ceph-mon[92422]: pgmap v6847: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 195 KiB/s wr, 25 op/s 2023-03-09T18:16:10.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:16:10 smithi122 ceph-mon[93869]: pgmap v6848: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 111 KiB/s wr, 22 op/s 2023-03-09T18:16:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:16:10 smithi122 ceph-mon[93869]: osdmap e641: 12 total, 12 up, 12 in 2023-03-09T18:16:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:16:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11881.100576 secs 2023-03-09T18:16:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:16:10 smithi181 ceph-mon[94189]: pgmap v6848: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 111 KiB/s wr, 22 op/s 2023-03-09T18:16:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:16:10 smithi181 ceph-mon[94189]: osdmap e641: 12 total, 12 up, 12 in 2023-03-09T18:16:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:16:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11881.100576 secs 2023-03-09T18:16:10.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:16:10 smithi038 ceph-mon[92422]: pgmap v6848: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 111 KiB/s wr, 22 op/s 2023-03-09T18:16:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:16:10 smithi038 ceph-mon[92422]: osdmap e641: 12 total, 12 up, 12 in 2023-03-09T18:16:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:16:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11881.100576 secs 2023-03-09T18:16:12.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:16:12 smithi122 ceph-mon[93869]: pgmap v6850: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 124 KiB/s wr, 23 op/s 2023-03-09T18:16:12.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:16:12 smithi181 ceph-mon[94189]: pgmap v6850: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 124 KiB/s wr, 23 op/s 2023-03-09T18:16:12.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:16:12 smithi038 ceph-mon[92422]: pgmap v6850: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 124 KiB/s wr, 23 op/s 2023-03-09T18:16:13.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12345.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:16:13.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:16:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11883.394414 secs 2023-03-09T18:16:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:16:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11883.394414 secs 2023-03-09T18:16:13.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:16:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11883.394414 secs 2023-03-09T18:16:14.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:16:14 smithi122 ceph-mon[93869]: pgmap v6851: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 8 op/s 2023-03-09T18:16:14.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:16:14 smithi181 ceph-mon[94189]: pgmap v6851: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 8 op/s 2023-03-09T18:16:14.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:16:14 smithi038 ceph-mon[92422]: pgmap v6851: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 8 op/s 2023-03-09T18:16:15.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:16:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11886.100635 secs 2023-03-09T18:16:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:16:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11886.100635 secs 2023-03-09T18:16:15.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:16:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11886.100635 secs 2023-03-09T18:16:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:16:16 smithi122 ceph-mon[93869]: pgmap v6852: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 54 KiB/s wr, 8 op/s 2023-03-09T18:16:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:16:16 smithi122 ceph-mon[93869]: osdmap e642: 12 total, 12 up, 12 in 2023-03-09T18:16:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:16:16 smithi181 ceph-mon[94189]: pgmap v6852: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 54 KiB/s wr, 8 op/s 2023-03-09T18:16:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:16:16 smithi181 ceph-mon[94189]: osdmap e642: 12 total, 12 up, 12 in 2023-03-09T18:16:16.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:16:16 smithi038 ceph-mon[92422]: pgmap v6852: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 54 KiB/s wr, 8 op/s 2023-03-09T18:16:16.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:16:16 smithi038 ceph-mon[92422]: osdmap e642: 12 total, 12 up, 12 in 2023-03-09T18:16:18.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12350.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:16:18.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:16:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11888.394542 secs 2023-03-09T18:16:18.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:16:18 smithi122 ceph-mon[93869]: pgmap v6854: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 52 KiB/s wr, 9 op/s 2023-03-09T18:16:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:16:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11888.394542 secs 2023-03-09T18:16:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:16:18 smithi181 ceph-mon[94189]: pgmap v6854: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 52 KiB/s wr, 9 op/s 2023-03-09T18:16:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:16:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11888.394542 secs 2023-03-09T18:16:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:16:18 smithi038 ceph-mon[92422]: pgmap v6854: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 52 KiB/s wr, 9 op/s 2023-03-09T18:16:20.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:16:20 smithi122 ceph-mon[93869]: pgmap v6855: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 43 KiB/s wr, 8 op/s 2023-03-09T18:16:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:16:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11891.100696 secs 2023-03-09T18:16:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:16:20 smithi181 ceph-mon[94189]: pgmap v6855: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 43 KiB/s wr, 8 op/s 2023-03-09T18:16:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:16:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11891.100696 secs 2023-03-09T18:16:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:16:20 smithi038 ceph-mon[92422]: pgmap v6855: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 43 KiB/s wr, 8 op/s 2023-03-09T18:16:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:16:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11891.100696 secs 2023-03-09T18:16:22.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:16:22 smithi122 ceph-mon[93869]: pgmap v6856: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T18:16:22.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:16:22 smithi181 ceph-mon[94189]: pgmap v6856: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T18:16:22.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:16:22 smithi038 ceph-mon[92422]: pgmap v6856: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T18:16:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12355.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:16:23.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:16:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11893.394684 secs 2023-03-09T18:16:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:16:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11893.394684 secs 2023-03-09T18:16:23.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:16:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11893.394684 secs 2023-03-09T18:16:24.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:16:24 smithi122 ceph-mon[93869]: pgmap v6857: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 2.1 KiB/s wr, 7 op/s 2023-03-09T18:16:24.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:16:24 smithi181 ceph-mon[94189]: pgmap v6857: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 2.1 KiB/s wr, 7 op/s 2023-03-09T18:16:25.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:16:24 smithi038 ceph-mon[92422]: pgmap v6857: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 2.1 KiB/s wr, 7 op/s 2023-03-09T18:16:25.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:16:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11896.100757 secs 2023-03-09T18:16:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:16:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11896.100757 secs 2023-03-09T18:16:25.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:16:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11896.100757 secs 2023-03-09T18:16:26.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:16:26 smithi122 ceph-mon[93869]: pgmap v6858: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2023-03-09T18:16:26.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:16:26 smithi181 ceph-mon[94189]: pgmap v6858: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2023-03-09T18:16:26.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:16:26 smithi038 ceph-mon[92422]: pgmap v6858: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2023-03-09T18:16:28.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12360.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:16:28.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:16:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11898.394803 secs 2023-03-09T18:16:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:16:28 smithi122 ceph-mon[93869]: pgmap v6859: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 40 KiB/s wr, 7 op/s 2023-03-09T18:16:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:16:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11898.394803 secs 2023-03-09T18:16:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:16:28 smithi181 ceph-mon[94189]: pgmap v6859: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 40 KiB/s wr, 7 op/s 2023-03-09T18:16:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:16:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11898.394803 secs 2023-03-09T18:16:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:16:28 smithi038 ceph-mon[92422]: pgmap v6859: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 40 KiB/s wr, 7 op/s 2023-03-09T18:16:30.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:16:30 smithi122 ceph-mon[93869]: pgmap v6860: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 KiB/s wr, 7 op/s 2023-03-09T18:16:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:16:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11901.100814 secs 2023-03-09T18:16:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:16:30 smithi181 ceph-mon[94189]: pgmap v6860: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 KiB/s wr, 7 op/s 2023-03-09T18:16:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:16:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11901.100814 secs 2023-03-09T18:16:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:16:30 smithi038 ceph-mon[92422]: pgmap v6860: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 KiB/s wr, 7 op/s 2023-03-09T18:16:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:16:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11901.100814 secs 2023-03-09T18:16:32.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:16:32 smithi122 ceph-mon[93869]: pgmap v6861: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 36 KiB/s wr, 8 op/s 2023-03-09T18:16:32.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:16:32 smithi181 ceph-mon[94189]: pgmap v6861: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 36 KiB/s wr, 8 op/s 2023-03-09T18:16:32.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:16:32 smithi038 ceph-mon[92422]: pgmap v6861: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 36 KiB/s wr, 8 op/s 2023-03-09T18:16:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12365.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:16:33.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:16:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11903.394918 secs 2023-03-09T18:16:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:16:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11903.394918 secs 2023-03-09T18:16:33.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:16:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11903.394918 secs 2023-03-09T18:16:34.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:16:34 smithi122 ceph-mon[93869]: pgmap v6862: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 KiB/s wr, 7 op/s 2023-03-09T18:16:34.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:16:34 smithi181 ceph-mon[94189]: pgmap v6862: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 KiB/s wr, 7 op/s 2023-03-09T18:16:34.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:16:34 smithi038 ceph-mon[92422]: pgmap v6862: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 KiB/s wr, 7 op/s 2023-03-09T18:16:35.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:16:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11906.100876 secs 2023-03-09T18:16:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:16:35 smithi122 ceph-mon[93869]: pgmap v6863: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 KiB/s wr, 6 op/s 2023-03-09T18:16:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:16:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11906.100876 secs 2023-03-09T18:16:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:16:35 smithi181 ceph-mon[94189]: pgmap v6863: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 KiB/s wr, 6 op/s 2023-03-09T18:16:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:16:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11906.100876 secs 2023-03-09T18:16:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:16:35 smithi038 ceph-mon[92422]: pgmap v6863: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 KiB/s wr, 6 op/s 2023-03-09T18:16:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12370.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:16:38.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:16:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11908.395043 secs 2023-03-09T18:16:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:16:38 smithi122 ceph-mon[93869]: pgmap v6864: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 KiB/s wr, 8 op/s 2023-03-09T18:16:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:16:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11908.395043 secs 2023-03-09T18:16:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:16:38 smithi181 ceph-mon[94189]: pgmap v6864: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 KiB/s wr, 8 op/s 2023-03-09T18:16:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:16:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11908.395043 secs 2023-03-09T18:16:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:16:38 smithi038 ceph-mon[92422]: pgmap v6864: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 KiB/s wr, 8 op/s 2023-03-09T18:16:40.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:16:40 smithi122 ceph-mon[93869]: pgmap v6865: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 67 KiB/s wr, 7 op/s 2023-03-09T18:16:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:16:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11911.100938 secs 2023-03-09T18:16:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:16:40 smithi181 ceph-mon[94189]: pgmap v6865: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 67 KiB/s wr, 7 op/s 2023-03-09T18:16:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:16:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11911.100938 secs 2023-03-09T18:16:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:16:40 smithi038 ceph-mon[92422]: pgmap v6865: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 67 KiB/s wr, 7 op/s 2023-03-09T18:16:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:16:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11911.100938 secs 2023-03-09T18:16:42.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:16:42 smithi122 ceph-mon[93869]: pgmap v6866: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 142 KiB/s wr, 10 op/s 2023-03-09T18:16:42.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:16:42 smithi181 ceph-mon[94189]: pgmap v6866: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 142 KiB/s wr, 10 op/s 2023-03-09T18:16:42.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:16:42 smithi038 ceph-mon[92422]: pgmap v6866: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 142 KiB/s wr, 10 op/s 2023-03-09T18:16:43.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12375.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:16:43.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:16:43 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:16:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:16:43 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:16:43.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:16:43 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:16:44.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:16:44 smithi122 ceph-mon[93869]: pgmap v6867: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 153 KiB/s wr, 9 op/s 2023-03-09T18:16:44.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:16:44 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:16:44.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:16:44 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:16:44.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:16:44 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:16:44.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:16:44 smithi181 ceph-mon[94189]: pgmap v6867: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 153 KiB/s wr, 9 op/s 2023-03-09T18:16:44.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:16:44 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:16:44.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:16:44 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:16:44.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:16:44 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:16:44.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:16:44 smithi038 ceph-mon[92422]: pgmap v6867: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 153 KiB/s wr, 9 op/s 2023-03-09T18:16:44.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:16:44 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:16:44.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:16:44 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:16:44.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:16:44 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:16:45.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:16:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11916.100984 secs 2023-03-09T18:16:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:16:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11916.100984 secs 2023-03-09T18:16:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:16:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11916.100984 secs 2023-03-09T18:16:46.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:16:46 smithi122 ceph-mon[93869]: pgmap v6868: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 153 KiB/s wr, 8 op/s 2023-03-09T18:16:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:16:46 smithi181 ceph-mon[94189]: pgmap v6868: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 153 KiB/s wr, 8 op/s 2023-03-09T18:16:46.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:16:46 smithi038 ceph-mon[92422]: pgmap v6868: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 153 KiB/s wr, 8 op/s 2023-03-09T18:16:48.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12380.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:16:48.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:16:48 smithi122 ceph-mon[93869]: pgmap v6869: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 217 KiB/s wr, 12 op/s 2023-03-09T18:16:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:16:48 smithi181 ceph-mon[94189]: pgmap v6869: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 217 KiB/s wr, 12 op/s 2023-03-09T18:16:48.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:16:48 smithi038 ceph-mon[92422]: pgmap v6869: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 217 KiB/s wr, 12 op/s 2023-03-09T18:16:50.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:16:50 smithi122 ceph-mon[93869]: pgmap v6870: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 172 KiB/s wr, 10 op/s 2023-03-09T18:16:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:16:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11921.101068 secs 2023-03-09T18:16:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:16:50 smithi181 ceph-mon[94189]: pgmap v6870: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 172 KiB/s wr, 10 op/s 2023-03-09T18:16:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:16:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11921.101068 secs 2023-03-09T18:16:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:16:50 smithi038 ceph-mon[92422]: pgmap v6870: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 172 KiB/s wr, 10 op/s 2023-03-09T18:16:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:16:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11921.101068 secs 2023-03-09T18:16:52.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:16:52 smithi122 ceph-mon[93869]: pgmap v6871: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 225 KiB/s wr, 12 op/s 2023-03-09T18:16:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:16:52 smithi181 ceph-mon[94189]: pgmap v6871: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 225 KiB/s wr, 12 op/s 2023-03-09T18:16:52.997 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:16:52 smithi038 ceph-mon[92422]: pgmap v6871: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 225 KiB/s wr, 12 op/s 2023-03-09T18:16:53.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12385.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:16:54.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:16:54 smithi122 ceph-mon[93869]: pgmap v6872: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 160 KiB/s wr, 10 op/s 2023-03-09T18:16:54.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:16:54 smithi181 ceph-mon[94189]: pgmap v6872: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 160 KiB/s wr, 10 op/s 2023-03-09T18:16:54.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:16:54 smithi038 ceph-mon[92422]: pgmap v6872: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 160 KiB/s wr, 10 op/s 2023-03-09T18:16:55.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:16:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11926.101130 secs 2023-03-09T18:16:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:16:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11926.101130 secs 2023-03-09T18:16:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:16:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11926.101130 secs 2023-03-09T18:16:56.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:16:56 smithi122 ceph-mon[93869]: pgmap v6873: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 149 KiB/s wr, 8 op/s 2023-03-09T18:16:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:16:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:16:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:16:56 smithi181 ceph-mon[94189]: pgmap v6873: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 149 KiB/s wr, 8 op/s 2023-03-09T18:16:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:16:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:16:56.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:16:56 smithi038 ceph-mon[92422]: pgmap v6873: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 149 KiB/s wr, 8 op/s 2023-03-09T18:16:56.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:16:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:16:57.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:16:57 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:16:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:16:57 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:16:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:16:57 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:16:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:16:57 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:16:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:16:57 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:16:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:16:57 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:16:57.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:16:57 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:16:57.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:16:57 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:16:57.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:16:57 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:16:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12390.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:16:58.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:16:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11928.395475 secs 2023-03-09T18:16:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:16:58 smithi122 ceph-mon[93869]: pgmap v6874: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 257 KiB/s wr, 12 op/s 2023-03-09T18:16:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:16:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11928.395475 secs 2023-03-09T18:16:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:16:58 smithi181 ceph-mon[94189]: pgmap v6874: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 257 KiB/s wr, 12 op/s 2023-03-09T18:16:58.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:16:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11928.395475 secs 2023-03-09T18:16:58.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:16:58 smithi038 ceph-mon[92422]: pgmap v6874: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 257 KiB/s wr, 12 op/s 2023-03-09T18:16:59.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:16:59 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:16:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:16:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:16:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:16:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:16:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:16:59 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:16:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:16:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:16:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:16:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:16:59.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:16:59 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:16:59.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:16:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:16:59.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:16:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:17:00.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:17:00 smithi122 ceph-mon[93869]: pgmap v6875: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 214 KiB/s wr, 10 op/s 2023-03-09T18:17:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:17:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11931.101191 secs 2023-03-09T18:17:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:17:00 smithi181 ceph-mon[94189]: pgmap v6875: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 214 KiB/s wr, 10 op/s 2023-03-09T18:17:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:17:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11931.101191 secs 2023-03-09T18:17:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:17:00 smithi038 ceph-mon[92422]: pgmap v6875: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 214 KiB/s wr, 10 op/s 2023-03-09T18:17:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:17:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11931.101191 secs 2023-03-09T18:17:02.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:17:02 smithi122 ceph-mon[93869]: pgmap v6876: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 263 KiB/s wr, 15 op/s 2023-03-09T18:17:02.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:17:02 smithi181 ceph-mon[94189]: pgmap v6876: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 263 KiB/s wr, 15 op/s 2023-03-09T18:17:02.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:17:02 smithi038 ceph-mon[92422]: pgmap v6876: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 263 KiB/s wr, 15 op/s 2023-03-09T18:17:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12395.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:17:03.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:17:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11933.395596 secs 2023-03-09T18:17:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:17:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11933.395596 secs 2023-03-09T18:17:03.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:17:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11933.395596 secs 2023-03-09T18:17:04.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:17:04 smithi122 ceph-mon[93869]: pgmap v6877: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 222 KiB/s wr, 25 op/s 2023-03-09T18:17:04.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:17:04 smithi181 ceph-mon[94189]: pgmap v6877: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 222 KiB/s wr, 25 op/s 2023-03-09T18:17:04.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:17:04 smithi038 ceph-mon[92422]: pgmap v6877: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 222 KiB/s wr, 25 op/s 2023-03-09T18:17:05.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:17:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11936.101252 secs 2023-03-09T18:17:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:17:05 smithi122 ceph-mon[93869]: osdmap e643: 12 total, 12 up, 12 in 2023-03-09T18:17:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:17:05 smithi122 ceph-mon[93869]: pgmap v6879: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 253 KiB/s wr, 29 op/s 2023-03-09T18:17:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:17:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11936.101252 secs 2023-03-09T18:17:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:17:05 smithi181 ceph-mon[94189]: osdmap e643: 12 total, 12 up, 12 in 2023-03-09T18:17:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:17:05 smithi181 ceph-mon[94189]: pgmap v6879: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 253 KiB/s wr, 29 op/s 2023-03-09T18:17:05.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:17:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11936.101252 secs 2023-03-09T18:17:05.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:17:05 smithi038 ceph-mon[92422]: osdmap e643: 12 total, 12 up, 12 in 2023-03-09T18:17:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:17:05 smithi038 ceph-mon[92422]: pgmap v6879: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 253 KiB/s wr, 29 op/s 2023-03-09T18:17:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12400.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:17:08.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:17:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11938.395714 secs 2023-03-09T18:17:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:17:08 smithi122 ceph-mon[93869]: pgmap v6880: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 220 KiB/s wr, 28 op/s 2023-03-09T18:17:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:17:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11938.395714 secs 2023-03-09T18:17:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:17:08 smithi181 ceph-mon[94189]: pgmap v6880: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 220 KiB/s wr, 28 op/s 2023-03-09T18:17:08.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:17:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11938.395714 secs 2023-03-09T18:17:08.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:17:08 smithi038 ceph-mon[92422]: pgmap v6880: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 220 KiB/s wr, 28 op/s 2023-03-09T18:17:10.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:17:10 smithi122 ceph-mon[93869]: pgmap v6881: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 195 KiB/s wr, 26 op/s 2023-03-09T18:17:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:17:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11941.101308 secs 2023-03-09T18:17:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:17:10 smithi181 ceph-mon[94189]: pgmap v6881: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 195 KiB/s wr, 26 op/s 2023-03-09T18:17:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:17:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11941.101308 secs 2023-03-09T18:17:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:17:10 smithi038 ceph-mon[92422]: pgmap v6881: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 195 KiB/s wr, 26 op/s 2023-03-09T18:17:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:17:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11941.101308 secs 2023-03-09T18:17:12.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:17:12 smithi122 ceph-mon[93869]: pgmap v6882: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 152 KiB/s wr, 23 op/s 2023-03-09T18:17:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:17:12 smithi122 ceph-mon[93869]: osdmap e644: 12 total, 12 up, 12 in 2023-03-09T18:17:12.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:17:12 smithi181 ceph-mon[94189]: pgmap v6882: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 152 KiB/s wr, 23 op/s 2023-03-09T18:17:12.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:17:12 smithi181 ceph-mon[94189]: osdmap e644: 12 total, 12 up, 12 in 2023-03-09T18:17:12.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:17:12 smithi038 ceph-mon[92422]: pgmap v6882: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 152 KiB/s wr, 23 op/s 2023-03-09T18:17:12.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:17:12 smithi038 ceph-mon[92422]: osdmap e644: 12 total, 12 up, 12 in 2023-03-09T18:17:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12405.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:17:13.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:17:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11943.395779 secs 2023-03-09T18:17:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:17:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11943.395779 secs 2023-03-09T18:17:13.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:17:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11943.395779 secs 2023-03-09T18:17:14.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:17:14 smithi122 ceph-mon[93869]: pgmap v6884: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 144 KiB/s wr, 9 op/s 2023-03-09T18:17:14.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:17:14 smithi181 ceph-mon[94189]: pgmap v6884: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 144 KiB/s wr, 9 op/s 2023-03-09T18:17:14.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:17:14 smithi038 ceph-mon[92422]: pgmap v6884: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 144 KiB/s wr, 9 op/s 2023-03-09T18:17:15.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:17:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11946.101376 secs 2023-03-09T18:17:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:17:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11946.101376 secs 2023-03-09T18:17:15.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:17:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11946.101376 secs 2023-03-09T18:17:16.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:17:16 smithi122 ceph-mon[93869]: pgmap v6885: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 126 KiB/s wr, 7 op/s 2023-03-09T18:17:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:17:16 smithi181 ceph-mon[94189]: pgmap v6885: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 126 KiB/s wr, 7 op/s 2023-03-09T18:17:16.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:17:16 smithi038 ceph-mon[92422]: pgmap v6885: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 126 KiB/s wr, 7 op/s 2023-03-09T18:17:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12410.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:17:18.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:17:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11948.395896 secs 2023-03-09T18:17:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:17:18 smithi122 ceph-mon[93869]: pgmap v6886: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 29 KiB/s wr, 7 op/s 2023-03-09T18:17:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:17:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11948.395896 secs 2023-03-09T18:17:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:17:18 smithi181 ceph-mon[94189]: pgmap v6886: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 29 KiB/s wr, 7 op/s 2023-03-09T18:17:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:17:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11948.395896 secs 2023-03-09T18:17:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:17:18 smithi038 ceph-mon[92422]: pgmap v6886: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 29 KiB/s wr, 7 op/s 2023-03-09T18:17:20.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:17:20 smithi122 ceph-mon[93869]: pgmap v6887: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 29 KiB/s wr, 6 op/s 2023-03-09T18:17:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:17:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11951.101435 secs 2023-03-09T18:17:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:17:20 smithi181 ceph-mon[94189]: pgmap v6887: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 29 KiB/s wr, 6 op/s 2023-03-09T18:17:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:17:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11951.101435 secs 2023-03-09T18:17:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:17:20 smithi038 ceph-mon[92422]: pgmap v6887: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 29 KiB/s wr, 6 op/s 2023-03-09T18:17:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:17:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11951.101435 secs 2023-03-09T18:17:22.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:17:22 smithi122 ceph-mon[93869]: pgmap v6888: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 49 KiB/s wr, 6 op/s 2023-03-09T18:17:22.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:17:22 smithi181 ceph-mon[94189]: pgmap v6888: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 49 KiB/s wr, 6 op/s 2023-03-09T18:17:22.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:17:22 smithi038 ceph-mon[92422]: pgmap v6888: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 49 KiB/s wr, 6 op/s 2023-03-09T18:17:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12415.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:17:23.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:17:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11953.396033 secs 2023-03-09T18:17:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:17:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11953.396033 secs 2023-03-09T18:17:23.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:17:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11953.396033 secs 2023-03-09T18:17:24.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:17:24 smithi122 ceph-mon[93869]: pgmap v6889: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 41 KiB/s wr, 6 op/s 2023-03-09T18:17:24.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:17:24 smithi181 ceph-mon[94189]: pgmap v6889: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 41 KiB/s wr, 6 op/s 2023-03-09T18:17:24.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:17:24 smithi038 ceph-mon[92422]: pgmap v6889: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 41 KiB/s wr, 6 op/s 2023-03-09T18:17:25.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:17:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11956.101498 secs 2023-03-09T18:17:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:17:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11956.101498 secs 2023-03-09T18:17:25.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:17:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11956.101498 secs 2023-03-09T18:17:26.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:17:26 smithi122 ceph-mon[93869]: pgmap v6890: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 41 KiB/s wr, 6 op/s 2023-03-09T18:17:26.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:17:26 smithi181 ceph-mon[94189]: pgmap v6890: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 41 KiB/s wr, 6 op/s 2023-03-09T18:17:26.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:17:26 smithi038 ceph-mon[92422]: pgmap v6890: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 41 KiB/s wr, 6 op/s 2023-03-09T18:17:28.793 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12420.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:17:28.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:17:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11958.396149 secs 2023-03-09T18:17:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:17:28 smithi122 ceph-mon[93869]: pgmap v6891: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 41 KiB/s wr, 8 op/s 2023-03-09T18:17:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:17:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11958.396149 secs 2023-03-09T18:17:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:17:28 smithi181 ceph-mon[94189]: pgmap v6891: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 41 KiB/s wr, 8 op/s 2023-03-09T18:17:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:17:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11958.396149 secs 2023-03-09T18:17:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:17:28 smithi038 ceph-mon[92422]: pgmap v6891: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 41 KiB/s wr, 8 op/s 2023-03-09T18:17:30.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:17:30 smithi122 ceph-mon[93869]: pgmap v6892: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 41 KiB/s wr, 6 op/s 2023-03-09T18:17:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:17:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11961.101581 secs 2023-03-09T18:17:30.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:17:30 smithi181 ceph-mon[94189]: pgmap v6892: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 41 KiB/s wr, 6 op/s 2023-03-09T18:17:30.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:17:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11961.101581 secs 2023-03-09T18:17:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:17:30 smithi038 ceph-mon[92422]: pgmap v6892: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 41 KiB/s wr, 6 op/s 2023-03-09T18:17:30.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:17:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11961.101581 secs 2023-03-09T18:17:32.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:17:32 smithi122 ceph-mon[93869]: pgmap v6893: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 93 KiB/s wr, 9 op/s 2023-03-09T18:17:32.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:17:32 smithi181 ceph-mon[94189]: pgmap v6893: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 93 KiB/s wr, 9 op/s 2023-03-09T18:17:32.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:17:32 smithi038 ceph-mon[92422]: pgmap v6893: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 93 KiB/s wr, 9 op/s 2023-03-09T18:17:33.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12425.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:17:34.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:17:34 smithi122 ceph-mon[93869]: pgmap v6894: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 7 op/s 2023-03-09T18:17:34.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:17:34 smithi181 ceph-mon[94189]: pgmap v6894: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 7 op/s 2023-03-09T18:17:34.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:17:34 smithi038 ceph-mon[92422]: pgmap v6894: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 7 op/s 2023-03-09T18:17:35.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:17:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11966.101666 secs 2023-03-09T18:17:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:17:35 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:17:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:17:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11966.101666 secs 2023-03-09T18:17:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:17:35 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:17:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:17:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11966.101666 secs 2023-03-09T18:17:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:17:35 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:17:36.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:17:36 smithi122 ceph-mon[93869]: pgmap v6895: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 52 KiB/s wr, 6 op/s 2023-03-09T18:17:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:17:36 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:17:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:17:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:17:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:17:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:17:36.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:17:36 smithi181 ceph-mon[94189]: pgmap v6895: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 52 KiB/s wr, 6 op/s 2023-03-09T18:17:36.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:17:36 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:17:36.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:17:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:17:36.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:17:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:17:36.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:17:36 smithi038 ceph-mon[92422]: pgmap v6895: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 52 KiB/s wr, 6 op/s 2023-03-09T18:17:36.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:17:36 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:17:36.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:17:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:17:36.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:17:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:17:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12430.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:17:38.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:17:38 smithi122 ceph-mon[93869]: pgmap v6896: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 89 KiB/s wr, 9 op/s 2023-03-09T18:17:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:17:38 smithi181 ceph-mon[94189]: pgmap v6896: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 89 KiB/s wr, 9 op/s 2023-03-09T18:17:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:17:38 smithi038 ceph-mon[92422]: pgmap v6896: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 89 KiB/s wr, 9 op/s 2023-03-09T18:17:40.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:17:40 smithi122 ceph-mon[93869]: pgmap v6897: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 98 KiB/s wr, 7 op/s 2023-03-09T18:17:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:17:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11971.101751 secs 2023-03-09T18:17:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:17:40 smithi181 ceph-mon[94189]: pgmap v6897: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 98 KiB/s wr, 7 op/s 2023-03-09T18:17:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:17:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11971.101751 secs 2023-03-09T18:17:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:17:40 smithi038 ceph-mon[92422]: pgmap v6897: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 98 KiB/s wr, 7 op/s 2023-03-09T18:17:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:17:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11971.101751 secs 2023-03-09T18:17:42.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:17:42 smithi122 ceph-mon[93869]: pgmap v6898: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 206 KiB/s wr, 10 op/s 2023-03-09T18:17:42.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:17:42 smithi181 ceph-mon[94189]: pgmap v6898: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 206 KiB/s wr, 10 op/s 2023-03-09T18:17:42.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:17:42 smithi038 ceph-mon[92422]: pgmap v6898: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 206 KiB/s wr, 10 op/s 2023-03-09T18:17:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12435.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:17:43.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:17:43 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 11973.396404 secs 2023-03-09T18:17:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:17:43 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 11973.396404 secs 2023-03-09T18:17:43.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:17:43 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 11973.396404 secs 2023-03-09T18:17:44.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:17:44 smithi122 ceph-mon[93869]: pgmap v6899: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 165 KiB/s wr, 9 op/s 2023-03-09T18:17:44.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:17:44 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:17:44.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:17:44 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:17:44.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:17:44 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:17:44.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:17:44 smithi181 ceph-mon[94189]: pgmap v6899: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 165 KiB/s wr, 9 op/s 2023-03-09T18:17:44.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:17:44 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:17:44.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:17:44 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:17:44.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:17:44 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:17:44.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:17:44 smithi038 ceph-mon[92422]: pgmap v6899: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 165 KiB/s wr, 9 op/s 2023-03-09T18:17:44.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:17:44 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:17:44.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:17:44 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:17:44.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:17:44 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:17:45.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:17:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11976.101811 secs 2023-03-09T18:17:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:17:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11976.101811 secs 2023-03-09T18:17:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:17:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11976.101811 secs 2023-03-09T18:17:46.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:17:46 smithi122 ceph-mon[93869]: pgmap v6900: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 165 KiB/s wr, 9 op/s 2023-03-09T18:17:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:17:46 smithi181 ceph-mon[94189]: pgmap v6900: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 165 KiB/s wr, 9 op/s 2023-03-09T18:17:46.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:17:46 smithi038 ceph-mon[92422]: pgmap v6900: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 165 KiB/s wr, 9 op/s 2023-03-09T18:17:48.793 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12440.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:17:48.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:17:48 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 11978.396515 secs 2023-03-09T18:17:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:17:48 smithi122 ceph-mon[93869]: pgmap v6901: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 178 KiB/s wr, 12 op/s 2023-03-09T18:17:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:17:48 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 11978.396515 secs 2023-03-09T18:17:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:17:48 smithi181 ceph-mon[94189]: pgmap v6901: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 178 KiB/s wr, 12 op/s 2023-03-09T18:17:48.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:17:48 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 11978.396515 secs 2023-03-09T18:17:48.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:17:48 smithi038 ceph-mon[92422]: pgmap v6901: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 178 KiB/s wr, 12 op/s 2023-03-09T18:17:50.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:17:50 smithi122 ceph-mon[93869]: pgmap v6902: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 152 KiB/s wr, 10 op/s 2023-03-09T18:17:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:17:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11981.101896 secs 2023-03-09T18:17:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:17:50 smithi181 ceph-mon[94189]: pgmap v6902: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 152 KiB/s wr, 10 op/s 2023-03-09T18:17:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:17:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11981.101896 secs 2023-03-09T18:17:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:17:50 smithi038 ceph-mon[92422]: pgmap v6902: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 152 KiB/s wr, 10 op/s 2023-03-09T18:17:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:17:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11981.101896 secs 2023-03-09T18:17:52.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:17:52 smithi122 ceph-mon[93869]: pgmap v6903: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 261 KiB/s wr, 12 op/s 2023-03-09T18:17:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:17:52 smithi181 ceph-mon[94189]: pgmap v6903: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 261 KiB/s wr, 12 op/s 2023-03-09T18:17:52.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:17:52 smithi038 ceph-mon[92422]: pgmap v6903: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 261 KiB/s wr, 12 op/s 2023-03-09T18:17:53.793 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12445.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:17:53.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:17:53 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 11983.396631 secs 2023-03-09T18:17:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:17:53 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 11983.396631 secs 2023-03-09T18:17:53.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:17:53 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 11983.396631 secs 2023-03-09T18:17:54.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:17:54 smithi122 ceph-mon[93869]: pgmap v6904: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 154 KiB/s wr, 9 op/s 2023-03-09T18:17:54.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:17:54 smithi181 ceph-mon[94189]: pgmap v6904: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 154 KiB/s wr, 9 op/s 2023-03-09T18:17:54.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:17:54 smithi038 ceph-mon[92422]: pgmap v6904: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 154 KiB/s wr, 9 op/s 2023-03-09T18:17:55.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:17:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11986.101955 secs 2023-03-09T18:17:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:17:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11986.101955 secs 2023-03-09T18:17:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:17:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11986.101955 secs 2023-03-09T18:17:56.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:17:56 smithi122 ceph-mon[93869]: pgmap v6905: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 143 KiB/s wr, 8 op/s 2023-03-09T18:17:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:17:56 smithi181 ceph-mon[94189]: pgmap v6905: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 143 KiB/s wr, 8 op/s 2023-03-09T18:17:56.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:17:56 smithi038 ceph-mon[92422]: pgmap v6905: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 143 KiB/s wr, 8 op/s 2023-03-09T18:17:57.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:17:57 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:17:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:17:57 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:17:57.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:17:57 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:17:58.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12450.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:17:58.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:17:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11988.396767 secs 2023-03-09T18:17:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:17:58 smithi122 ceph-mon[93869]: pgmap v6906: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 160 KiB/s wr, 10 op/s 2023-03-09T18:17:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:17:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11988.396767 secs 2023-03-09T18:17:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:17:58 smithi181 ceph-mon[94189]: pgmap v6906: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 160 KiB/s wr, 10 op/s 2023-03-09T18:17:58.985 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:17:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11988.396767 secs 2023-03-09T18:17:58.985 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:17:58 smithi038 ceph-mon[92422]: pgmap v6906: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 160 KiB/s wr, 10 op/s 2023-03-09T18:18:00.885 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:18:00 smithi038 ceph-mon[92422]: pgmap v6907: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 157 KiB/s wr, 8 op/s 2023-03-09T18:18:00.885 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:18:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:18:00.886 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:18:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:18:00.886 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:18:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11991.101985 secs 2023-03-09T18:18:00.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:18:00 smithi122 ceph-mon[93869]: pgmap v6907: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 157 KiB/s wr, 8 op/s 2023-03-09T18:18:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:18:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:18:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:18:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:18:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:18:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11991.101985 secs 2023-03-09T18:18:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:18:00 smithi181 ceph-mon[94189]: pgmap v6907: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 157 KiB/s wr, 8 op/s 2023-03-09T18:18:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:18:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:18:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:18:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:18:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:18:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11991.101985 secs 2023-03-09T18:18:02.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:18:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:18:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:18:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:18:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:18:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:18:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:18:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:18:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:18:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:18:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:18:02 smithi122 ceph-mon[93869]: pgmap v6908: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 262 KiB/s wr, 13 op/s 2023-03-09T18:18:02.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:18:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:18:02.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:18:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:18:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:18:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:18:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:18:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:18:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:18:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:18:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:18:02 smithi181 ceph-mon[94189]: pgmap v6908: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 262 KiB/s wr, 13 op/s 2023-03-09T18:18:02.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:18:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:18:02.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:18:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:18:02.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:18:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:18:02.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:18:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:18:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:18:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:18:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:18:02 smithi038 ceph-mon[92422]: pgmap v6908: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 262 KiB/s wr, 13 op/s 2023-03-09T18:18:03.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:18:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11993.396903 secs 2023-03-09T18:18:03.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:18:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11993.396903 secs 2023-03-09T18:18:03.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:18:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11993.396903 secs 2023-03-09T18:18:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12455.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:18:04.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:18:04 smithi122 ceph-mon[93869]: pgmap v6909: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 174 KiB/s wr, 23 op/s 2023-03-09T18:18:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:18:04 smithi181 ceph-mon[94189]: pgmap v6909: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 174 KiB/s wr, 23 op/s 2023-03-09T18:18:04.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:18:04 smithi038 ceph-mon[92422]: pgmap v6909: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 174 KiB/s wr, 23 op/s 2023-03-09T18:18:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:18:05 smithi122 ceph-mon[93869]: osdmap e645: 12 total, 12 up, 12 in 2023-03-09T18:18:05.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:18:05 smithi181 ceph-mon[94189]: osdmap e645: 12 total, 12 up, 12 in 2023-03-09T18:18:05.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:18:05 smithi038 ceph-mon[92422]: osdmap e645: 12 total, 12 up, 12 in 2023-03-09T18:18:06.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:18:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11996.102046 secs 2023-03-09T18:18:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:18:06 smithi122 ceph-mon[93869]: pgmap v6911: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 208 KiB/s wr, 26 op/s 2023-03-09T18:18:06.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:18:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11996.102046 secs 2023-03-09T18:18:06.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:18:06 smithi181 ceph-mon[94189]: pgmap v6911: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 208 KiB/s wr, 26 op/s 2023-03-09T18:18:06.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:18:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11996.102046 secs 2023-03-09T18:18:06.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:18:06 smithi038 ceph-mon[92422]: pgmap v6911: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 208 KiB/s wr, 26 op/s 2023-03-09T18:18:08.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12460.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:18:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:18:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 11998.397044 secs 2023-03-09T18:18:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:18:08 smithi122 ceph-mon[93869]: pgmap v6912: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 280 KiB/s wr, 27 op/s 2023-03-09T18:18:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:18:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 11998.397044 secs 2023-03-09T18:18:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:18:08 smithi181 ceph-mon[94189]: pgmap v6912: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 280 KiB/s wr, 27 op/s 2023-03-09T18:18:08.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:18:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 11998.397044 secs 2023-03-09T18:18:08.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:18:08 smithi038 ceph-mon[92422]: pgmap v6912: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 280 KiB/s wr, 27 op/s 2023-03-09T18:18:10.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:18:10 smithi122 ceph-mon[93869]: pgmap v6913: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 280 KiB/s wr, 27 op/s 2023-03-09T18:18:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:18:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12001.102107 secs 2023-03-09T18:18:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:18:10 smithi181 ceph-mon[94189]: pgmap v6913: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 280 KiB/s wr, 27 op/s 2023-03-09T18:18:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:18:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12001.102107 secs 2023-03-09T18:18:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:18:10 smithi038 ceph-mon[92422]: pgmap v6913: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 280 KiB/s wr, 27 op/s 2023-03-09T18:18:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:18:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12001.102107 secs 2023-03-09T18:18:12.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:18:12 smithi122 ceph-mon[93869]: pgmap v6914: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 171 KiB/s wr, 24 op/s 2023-03-09T18:18:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:18:12 smithi122 ceph-mon[93869]: osdmap e646: 12 total, 12 up, 12 in 2023-03-09T18:18:12.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:18:12 smithi181 ceph-mon[94189]: pgmap v6914: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 171 KiB/s wr, 24 op/s 2023-03-09T18:18:12.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:18:12 smithi181 ceph-mon[94189]: osdmap e646: 12 total, 12 up, 12 in 2023-03-09T18:18:13.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:18:12 smithi038 ceph-mon[92422]: pgmap v6914: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 171 KiB/s wr, 24 op/s 2023-03-09T18:18:13.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:18:12 smithi038 ceph-mon[92422]: osdmap e646: 12 total, 12 up, 12 in 2023-03-09T18:18:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12465.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:18:13.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:18:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12003.397160 secs 2023-03-09T18:18:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:18:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12003.397160 secs 2023-03-09T18:18:13.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:18:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12003.397160 secs 2023-03-09T18:18:14.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:18:14 smithi122 ceph-mon[93869]: pgmap v6916: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 156 KiB/s wr, 10 op/s 2023-03-09T18:18:14.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:18:14 smithi181 ceph-mon[94189]: pgmap v6916: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 156 KiB/s wr, 10 op/s 2023-03-09T18:18:14.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:18:14 smithi038 ceph-mon[92422]: pgmap v6916: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 156 KiB/s wr, 10 op/s 2023-03-09T18:18:15.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:18:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12006.102168 secs 2023-03-09T18:18:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:18:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12006.102168 secs 2023-03-09T18:18:15.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:18:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12006.102168 secs 2023-03-09T18:18:16.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:18:16 smithi122 ceph-mon[93869]: pgmap v6917: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 146 KiB/s wr, 9 op/s 2023-03-09T18:18:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:18:16 smithi181 ceph-mon[94189]: pgmap v6917: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 146 KiB/s wr, 9 op/s 2023-03-09T18:18:16.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:18:16 smithi038 ceph-mon[92422]: pgmap v6917: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 146 KiB/s wr, 9 op/s 2023-03-09T18:18:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12470.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:18:18.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:18:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12008.397274 secs 2023-03-09T18:18:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:18:18 smithi122 ceph-mon[93869]: pgmap v6918: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 55 KiB/s wr, 9 op/s 2023-03-09T18:18:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:18:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12008.397274 secs 2023-03-09T18:18:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:18:18 smithi181 ceph-mon[94189]: pgmap v6918: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 55 KiB/s wr, 9 op/s 2023-03-09T18:18:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:18:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12008.397274 secs 2023-03-09T18:18:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:18:18 smithi038 ceph-mon[92422]: pgmap v6918: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 55 KiB/s wr, 9 op/s 2023-03-09T18:18:20.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:18:20 smithi122 ceph-mon[93869]: pgmap v6919: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 43 KiB/s wr, 9 op/s 2023-03-09T18:18:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:18:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12011.102228 secs 2023-03-09T18:18:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:18:20 smithi181 ceph-mon[94189]: pgmap v6919: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 43 KiB/s wr, 9 op/s 2023-03-09T18:18:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:18:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12011.102228 secs 2023-03-09T18:18:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:18:20 smithi038 ceph-mon[92422]: pgmap v6919: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 43 KiB/s wr, 9 op/s 2023-03-09T18:18:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:18:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12011.102228 secs 2023-03-09T18:18:22.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:18:22 smithi122 ceph-mon[93869]: pgmap v6920: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 8 op/s 2023-03-09T18:18:22.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:18:22 smithi181 ceph-mon[94189]: pgmap v6920: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 8 op/s 2023-03-09T18:18:22.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:18:22 smithi038 ceph-mon[92422]: pgmap v6920: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 8 op/s 2023-03-09T18:18:23.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12475.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:18:23.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:18:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12013.397370 secs 2023-03-09T18:18:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:18:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12013.397370 secs 2023-03-09T18:18:23.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:18:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12013.397370 secs 2023-03-09T18:18:24.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:18:24 smithi122 ceph-mon[93869]: pgmap v6921: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 36 KiB/s wr, 8 op/s 2023-03-09T18:18:24.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:18:24 smithi181 ceph-mon[94189]: pgmap v6921: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 36 KiB/s wr, 8 op/s 2023-03-09T18:18:24.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:18:24 smithi038 ceph-mon[92422]: pgmap v6921: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 36 KiB/s wr, 8 op/s 2023-03-09T18:18:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:18:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12016.102329 secs 2023-03-09T18:18:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:18:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12016.102329 secs 2023-03-09T18:18:25.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:18:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12016.102329 secs 2023-03-09T18:18:26.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:18:26 smithi122 ceph-mon[93869]: pgmap v6922: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 26 KiB/s wr, 7 op/s 2023-03-09T18:18:26.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:18:26 smithi181 ceph-mon[94189]: pgmap v6922: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 26 KiB/s wr, 7 op/s 2023-03-09T18:18:26.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:18:26 smithi038 ceph-mon[92422]: pgmap v6922: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 26 KiB/s wr, 7 op/s 2023-03-09T18:18:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12480.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:18:28.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:18:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12018.397483 secs 2023-03-09T18:18:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:18:28 smithi122 ceph-mon[93869]: pgmap v6923: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 61 KiB/s wr, 10 op/s 2023-03-09T18:18:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:18:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12018.397483 secs 2023-03-09T18:18:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:18:28 smithi181 ceph-mon[94189]: pgmap v6923: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 61 KiB/s wr, 10 op/s 2023-03-09T18:18:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:18:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12018.397483 secs 2023-03-09T18:18:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:18:28 smithi038 ceph-mon[92422]: pgmap v6923: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 61 KiB/s wr, 10 op/s 2023-03-09T18:18:30.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:18:30 smithi122 ceph-mon[93869]: pgmap v6924: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 61 KiB/s wr, 7 op/s 2023-03-09T18:18:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:18:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12021.102390 secs 2023-03-09T18:18:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:18:30 smithi181 ceph-mon[94189]: pgmap v6924: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 61 KiB/s wr, 7 op/s 2023-03-09T18:18:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:18:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12021.102390 secs 2023-03-09T18:18:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:18:30 smithi038 ceph-mon[92422]: pgmap v6924: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 61 KiB/s wr, 7 op/s 2023-03-09T18:18:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:18:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12021.102390 secs 2023-03-09T18:18:32.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:18:32 smithi122 ceph-mon[93869]: pgmap v6925: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 94 KiB/s wr, 9 op/s 2023-03-09T18:18:32.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:18:32 smithi181 ceph-mon[94189]: pgmap v6925: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 94 KiB/s wr, 9 op/s 2023-03-09T18:18:32.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:18:32 smithi038 ceph-mon[92422]: pgmap v6925: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 94 KiB/s wr, 9 op/s 2023-03-09T18:18:33.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12485.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:18:33.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:18:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12023.397577 secs 2023-03-09T18:18:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:18:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12023.397577 secs 2023-03-09T18:18:33.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:18:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12023.397577 secs 2023-03-09T18:18:34.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:18:34 smithi122 ceph-mon[93869]: pgmap v6926: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 70 KiB/s wr, 7 op/s 2023-03-09T18:18:34.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:18:34 smithi181 ceph-mon[94189]: pgmap v6926: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 70 KiB/s wr, 7 op/s 2023-03-09T18:18:34.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:18:34 smithi038 ceph-mon[92422]: pgmap v6926: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 70 KiB/s wr, 7 op/s 2023-03-09T18:18:35.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:18:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12026.102451 secs 2023-03-09T18:18:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:18:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12026.102451 secs 2023-03-09T18:18:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:18:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12026.102451 secs 2023-03-09T18:18:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:18:36 smithi122 ceph-mon[93869]: pgmap v6927: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 70 KiB/s wr, 6 op/s 2023-03-09T18:18:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:18:36 smithi181 ceph-mon[94189]: pgmap v6927: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 70 KiB/s wr, 6 op/s 2023-03-09T18:18:36.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:18:36 smithi038 ceph-mon[92422]: pgmap v6927: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 70 KiB/s wr, 6 op/s 2023-03-09T18:18:38.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12490.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:18:38.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:18:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12028.397694 secs 2023-03-09T18:18:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:18:38 smithi122 ceph-mon[93869]: pgmap v6928: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 106 KiB/s wr, 8 op/s 2023-03-09T18:18:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:18:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12028.397694 secs 2023-03-09T18:18:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:18:38 smithi181 ceph-mon[94189]: pgmap v6928: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 106 KiB/s wr, 8 op/s 2023-03-09T18:18:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:18:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12028.397694 secs 2023-03-09T18:18:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:18:38 smithi038 ceph-mon[92422]: pgmap v6928: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 106 KiB/s wr, 8 op/s 2023-03-09T18:18:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:18:40 smithi122 ceph-mon[93869]: pgmap v6929: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 80 KiB/s wr, 6 op/s 2023-03-09T18:18:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:18:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12031.102510 secs 2023-03-09T18:18:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:18:40 smithi181 ceph-mon[94189]: pgmap v6929: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 80 KiB/s wr, 6 op/s 2023-03-09T18:18:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:18:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12031.102510 secs 2023-03-09T18:18:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:18:40 smithi038 ceph-mon[92422]: pgmap v6929: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 80 KiB/s wr, 6 op/s 2023-03-09T18:18:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:18:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12031.102510 secs 2023-03-09T18:18:42.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:18:42 smithi122 ceph-mon[93869]: pgmap v6930: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 194 KiB/s wr, 10 op/s 2023-03-09T18:18:42.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:18:42 smithi181 ceph-mon[94189]: pgmap v6930: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 194 KiB/s wr, 10 op/s 2023-03-09T18:18:42.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:18:42 smithi038 ceph-mon[92422]: pgmap v6930: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 194 KiB/s wr, 10 op/s 2023-03-09T18:18:43.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12495.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:18:43.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:18:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12033.397758 secs 2023-03-09T18:18:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:18:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12033.397758 secs 2023-03-09T18:18:43.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:18:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12033.397758 secs 2023-03-09T18:18:44.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:18:44 smithi122 ceph-mon[93869]: pgmap v6931: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 171 KiB/s wr, 9 op/s 2023-03-09T18:18:44.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:18:44 smithi181 ceph-mon[94189]: pgmap v6931: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 171 KiB/s wr, 9 op/s 2023-03-09T18:18:44.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:18:44 smithi038 ceph-mon[92422]: pgmap v6931: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 171 KiB/s wr, 9 op/s 2023-03-09T18:18:45.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:18:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12036.102571 secs 2023-03-09T18:18:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:18:45 smithi122 ceph-mon[93869]: pgmap v6932: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 171 KiB/s wr, 9 op/s 2023-03-09T18:18:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:18:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12036.102571 secs 2023-03-09T18:18:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:18:45 smithi181 ceph-mon[94189]: pgmap v6932: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 171 KiB/s wr, 9 op/s 2023-03-09T18:18:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:18:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12036.102571 secs 2023-03-09T18:18:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:18:45 smithi038 ceph-mon[92422]: pgmap v6932: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 171 KiB/s wr, 9 op/s 2023-03-09T18:18:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12500.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:18:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:18:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12038.397909 secs 2023-03-09T18:18:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:18:48 smithi122 ceph-mon[93869]: pgmap v6933: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 230 KiB/s wr, 12 op/s 2023-03-09T18:18:48.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:18:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12038.397909 secs 2023-03-09T18:18:48.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:18:48 smithi181 ceph-mon[94189]: pgmap v6933: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 230 KiB/s wr, 12 op/s 2023-03-09T18:18:48.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:18:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12038.397909 secs 2023-03-09T18:18:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:18:48 smithi038 ceph-mon[92422]: pgmap v6933: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 230 KiB/s wr, 12 op/s 2023-03-09T18:18:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:18:50 smithi122 ceph-mon[93869]: pgmap v6934: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 203 KiB/s wr, 11 op/s 2023-03-09T18:18:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:18:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12041.102651 secs 2023-03-09T18:18:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:18:50 smithi181 ceph-mon[94189]: pgmap v6934: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 203 KiB/s wr, 11 op/s 2023-03-09T18:18:50.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:18:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12041.102651 secs 2023-03-09T18:18:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:18:50 smithi038 ceph-mon[92422]: pgmap v6934: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 203 KiB/s wr, 11 op/s 2023-03-09T18:18:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:18:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12041.102651 secs 2023-03-09T18:18:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:18:52 smithi122 ceph-mon[93869]: pgmap v6935: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 286 KiB/s wr, 13 op/s 2023-03-09T18:18:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:18:52 smithi181 ceph-mon[94189]: pgmap v6935: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 286 KiB/s wr, 13 op/s 2023-03-09T18:18:52.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:18:52 smithi038 ceph-mon[92422]: pgmap v6935: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 286 KiB/s wr, 13 op/s 2023-03-09T18:18:53.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12505.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:18:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:18:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12043.398033 secs 2023-03-09T18:18:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:18:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12043.398033 secs 2023-03-09T18:18:53.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:18:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12043.398033 secs 2023-03-09T18:18:54.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:18:54 smithi122 ceph-mon[93869]: pgmap v6936: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 183 KiB/s wr, 10 op/s 2023-03-09T18:18:54.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:18:54 smithi181 ceph-mon[94189]: pgmap v6936: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 183 KiB/s wr, 10 op/s 2023-03-09T18:18:54.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:18:54 smithi038 ceph-mon[92422]: pgmap v6936: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 183 KiB/s wr, 10 op/s 2023-03-09T18:18:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:18:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12046.102711 secs 2023-03-09T18:18:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:18:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12046.102711 secs 2023-03-09T18:18:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:18:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12046.102711 secs 2023-03-09T18:18:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:18:56 smithi122 ceph-mon[93869]: pgmap v6937: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 172 KiB/s wr, 9 op/s 2023-03-09T18:18:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:18:56 smithi181 ceph-mon[94189]: pgmap v6937: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 172 KiB/s wr, 9 op/s 2023-03-09T18:18:56.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:18:56 smithi038 ceph-mon[92422]: pgmap v6937: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 172 KiB/s wr, 9 op/s 2023-03-09T18:18:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12510.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:18:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:18:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12048.398170 secs 2023-03-09T18:18:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:18:58 smithi122 ceph-mon[93869]: pgmap v6938: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 189 KiB/s wr, 12 op/s 2023-03-09T18:18:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:18:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12048.398170 secs 2023-03-09T18:18:58.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:18:58 smithi181 ceph-mon[94189]: pgmap v6938: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 189 KiB/s wr, 12 op/s 2023-03-09T18:18:58.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:18:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12048.398170 secs 2023-03-09T18:18:58.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:18:58 smithi038 ceph-mon[92422]: pgmap v6938: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 189 KiB/s wr, 12 op/s 2023-03-09T18:19:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:19:00 smithi122 ceph-mon[93869]: pgmap v6939: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 141 KiB/s wr, 10 op/s 2023-03-09T18:19:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:19:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12051.102798 secs 2023-03-09T18:19:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:19:00 smithi181 ceph-mon[94189]: pgmap v6939: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 141 KiB/s wr, 10 op/s 2023-03-09T18:19:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:19:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12051.102798 secs 2023-03-09T18:19:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:19:00 smithi038 ceph-mon[92422]: pgmap v6939: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 141 KiB/s wr, 10 op/s 2023-03-09T18:19:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:19:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12051.102798 secs 2023-03-09T18:19:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:19:01 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:19:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:19:01 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:19:01.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:19:01 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:19:02.944 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:19:02 smithi122 ceph-mon[93869]: pgmap v6940: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 243 KiB/s wr, 14 op/s 2023-03-09T18:19:02.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:19:02 smithi181 ceph-mon[94189]: pgmap v6940: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 243 KiB/s wr, 14 op/s 2023-03-09T18:19:02.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:19:02 smithi038 ceph-mon[92422]: pgmap v6940: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 243 KiB/s wr, 14 op/s 2023-03-09T18:19:03.641 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:19:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12053.398274 secs 2023-03-09T18:19:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12515.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:19:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:19:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12053.398274 secs 2023-03-09T18:19:03.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:19:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12053.398274 secs 2023-03-09T18:19:04.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:19:04 smithi122 ceph-mon[93869]: pgmap v6941: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 172 KiB/s wr, 24 op/s 2023-03-09T18:19:04.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:19:04 smithi181 ceph-mon[94189]: pgmap v6941: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 172 KiB/s wr, 24 op/s 2023-03-09T18:19:04.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:19:04 smithi038 ceph-mon[92422]: pgmap v6941: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 172 KiB/s wr, 24 op/s 2023-03-09T18:19:06.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:19:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12056.102859 secs 2023-03-09T18:19:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:19:06 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:19:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:19:06 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:19:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:19:06 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:19:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:19:06 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:19:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:19:06 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:19:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:19:06 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:19:06.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:19:06 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:19:06.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:19:06 smithi038 ceph-mon[92422]: pgmap v6942: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 162 KiB/s wr, 22 op/s 2023-03-09T18:19:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:19:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12056.102859 secs 2023-03-09T18:19:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:19:06 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:19:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:19:06 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:19:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:19:06 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:19:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:19:06 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:19:06.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:19:06 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:19:06.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:19:06 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:19:06.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:19:06 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:19:06.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:19:06 smithi122 ceph-mon[93869]: pgmap v6942: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 162 KiB/s wr, 22 op/s 2023-03-09T18:19:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:19:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12056.102859 secs 2023-03-09T18:19:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:19:06 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:19:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:19:06 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:19:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:19:06 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:19:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:19:06 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:19:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:19:06 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:19:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:19:06 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:19:06.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:19:06 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:19:06.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:19:06 smithi181 ceph-mon[94189]: pgmap v6942: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 162 KiB/s wr, 22 op/s 2023-03-09T18:19:08.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12520.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:19:08.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:19:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12058.398353 secs 2023-03-09T18:19:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:19:08 smithi122 ceph-mon[93869]: pgmap v6943: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 186 KiB/s wr, 25 op/s 2023-03-09T18:19:08.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:19:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12058.398353 secs 2023-03-09T18:19:08.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:19:08 smithi181 ceph-mon[94189]: pgmap v6943: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 186 KiB/s wr, 25 op/s 2023-03-09T18:19:08.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:19:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12058.398353 secs 2023-03-09T18:19:08.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:19:08 smithi038 ceph-mon[92422]: pgmap v6943: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 186 KiB/s wr, 25 op/s 2023-03-09T18:19:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:19:10 smithi122 ceph-mon[93869]: pgmap v6944: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 179 KiB/s wr, 22 op/s 2023-03-09T18:19:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:19:10 smithi122 ceph-mon[93869]: osdmap e647: 12 total, 12 up, 12 in 2023-03-09T18:19:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:19:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12061.102922 secs 2023-03-09T18:19:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:19:10 smithi181 ceph-mon[94189]: pgmap v6944: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 179 KiB/s wr, 22 op/s 2023-03-09T18:19:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:19:10 smithi181 ceph-mon[94189]: osdmap e647: 12 total, 12 up, 12 in 2023-03-09T18:19:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:19:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12061.102922 secs 2023-03-09T18:19:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:19:10 smithi038 ceph-mon[92422]: pgmap v6944: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 179 KiB/s wr, 22 op/s 2023-03-09T18:19:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:19:10 smithi038 ceph-mon[92422]: osdmap e647: 12 total, 12 up, 12 in 2023-03-09T18:19:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:19:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12061.102922 secs 2023-03-09T18:19:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:19:12 smithi122 ceph-mon[93869]: pgmap v6946: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 68 KiB/s wr, 22 op/s 2023-03-09T18:19:12.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:19:12 smithi181 ceph-mon[94189]: pgmap v6946: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 68 KiB/s wr, 22 op/s 2023-03-09T18:19:12.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:19:12 smithi038 ceph-mon[92422]: pgmap v6946: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 68 KiB/s wr, 22 op/s 2023-03-09T18:19:13.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12525.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:19:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:19:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12063.398439 secs 2023-03-09T18:19:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:19:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12063.398439 secs 2023-03-09T18:19:13.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:19:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12063.398439 secs 2023-03-09T18:19:14.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:19:14 smithi122 ceph-mon[93869]: pgmap v6947: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 55 KiB/s wr, 8 op/s 2023-03-09T18:19:14.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:19:14 smithi181 ceph-mon[94189]: pgmap v6947: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 55 KiB/s wr, 8 op/s 2023-03-09T18:19:14.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:19:14 smithi038 ceph-mon[92422]: pgmap v6947: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 55 KiB/s wr, 8 op/s 2023-03-09T18:19:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:19:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12066.102987 secs 2023-03-09T18:19:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:19:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12066.102987 secs 2023-03-09T18:19:15.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:19:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12066.102987 secs 2023-03-09T18:19:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:19:16 smithi122 ceph-mon[93869]: pgmap v6948: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 55 KiB/s wr, 8 op/s 2023-03-09T18:19:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:19:16 smithi122 ceph-mon[93869]: osdmap e648: 12 total, 12 up, 12 in 2023-03-09T18:19:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:19:16 smithi181 ceph-mon[94189]: pgmap v6948: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 55 KiB/s wr, 8 op/s 2023-03-09T18:19:16.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:19:16 smithi181 ceph-mon[94189]: osdmap e648: 12 total, 12 up, 12 in 2023-03-09T18:19:16.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:19:16 smithi038 ceph-mon[92422]: pgmap v6948: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 55 KiB/s wr, 8 op/s 2023-03-09T18:19:16.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:19:16 smithi038 ceph-mon[92422]: osdmap e648: 12 total, 12 up, 12 in 2023-03-09T18:19:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12530.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:19:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:19:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12068.398574 secs 2023-03-09T18:19:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:19:18 smithi122 ceph-mon[93869]: pgmap v6950: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 9 op/s 2023-03-09T18:19:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:19:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12068.398574 secs 2023-03-09T18:19:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:19:18 smithi181 ceph-mon[94189]: pgmap v6950: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 9 op/s 2023-03-09T18:19:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:19:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12068.398574 secs 2023-03-09T18:19:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:19:18 smithi038 ceph-mon[92422]: pgmap v6950: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 9 op/s 2023-03-09T18:19:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:19:20 smithi122 ceph-mon[93869]: pgmap v6951: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 8 op/s 2023-03-09T18:19:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:19:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12071.103045 secs 2023-03-09T18:19:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:19:20 smithi181 ceph-mon[94189]: pgmap v6951: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 8 op/s 2023-03-09T18:19:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:19:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12071.103045 secs 2023-03-09T18:19:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:19:20 smithi038 ceph-mon[92422]: pgmap v6951: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 8 op/s 2023-03-09T18:19:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:19:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12071.103045 secs 2023-03-09T18:19:22.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:19:22 smithi122 ceph-mon[93869]: pgmap v6952: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T18:19:22.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:19:22 smithi181 ceph-mon[94189]: pgmap v6952: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T18:19:22.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:19:22 smithi038 ceph-mon[92422]: pgmap v6952: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T18:19:23.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12535.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:19:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:19:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12073.398731 secs 2023-03-09T18:19:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:19:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12073.398731 secs 2023-03-09T18:19:23.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:19:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12073.398731 secs 2023-03-09T18:19:24.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:19:24 smithi122 ceph-mon[93869]: pgmap v6953: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.1 KiB/s wr, 8 op/s 2023-03-09T18:19:24.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:19:24 smithi181 ceph-mon[94189]: pgmap v6953: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.1 KiB/s wr, 8 op/s 2023-03-09T18:19:24.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:19:24 smithi038 ceph-mon[92422]: pgmap v6953: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.1 KiB/s wr, 8 op/s 2023-03-09T18:19:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:19:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12076.103104 secs 2023-03-09T18:19:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:19:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12076.103104 secs 2023-03-09T18:19:25.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:19:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12076.103104 secs 2023-03-09T18:19:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:19:26 smithi122 ceph-mon[93869]: pgmap v6954: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.1 KiB/s wr, 8 op/s 2023-03-09T18:19:26.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:19:26 smithi181 ceph-mon[94189]: pgmap v6954: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.1 KiB/s wr, 8 op/s 2023-03-09T18:19:26.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:19:26 smithi038 ceph-mon[92422]: pgmap v6954: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.1 KiB/s wr, 8 op/s 2023-03-09T18:19:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12540.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:19:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:19:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12078.398852 secs 2023-03-09T18:19:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:19:28 smithi122 ceph-mon[93869]: pgmap v6955: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 41 KiB/s wr, 7 op/s 2023-03-09T18:19:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:19:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12078.398852 secs 2023-03-09T18:19:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:19:28 smithi181 ceph-mon[94189]: pgmap v6955: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 41 KiB/s wr, 7 op/s 2023-03-09T18:19:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:19:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12078.398852 secs 2023-03-09T18:19:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:19:28 smithi038 ceph-mon[92422]: pgmap v6955: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 41 KiB/s wr, 7 op/s 2023-03-09T18:19:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:19:30 smithi122 ceph-mon[93869]: pgmap v6956: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 37 KiB/s wr, 6 op/s 2023-03-09T18:19:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:19:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12081.103167 secs 2023-03-09T18:19:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:19:30 smithi181 ceph-mon[94189]: pgmap v6956: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 37 KiB/s wr, 6 op/s 2023-03-09T18:19:30.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:19:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12081.103167 secs 2023-03-09T18:19:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:19:30 smithi038 ceph-mon[92422]: pgmap v6956: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 37 KiB/s wr, 6 op/s 2023-03-09T18:19:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:19:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12081.103167 secs 2023-03-09T18:19:32.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:19:32 smithi122 ceph-mon[93869]: pgmap v6957: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 37 KiB/s wr, 8 op/s 2023-03-09T18:19:32.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:19:32 smithi181 ceph-mon[94189]: pgmap v6957: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 37 KiB/s wr, 8 op/s 2023-03-09T18:19:32.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:19:32 smithi038 ceph-mon[92422]: pgmap v6957: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 37 KiB/s wr, 8 op/s 2023-03-09T18:19:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12545.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:19:34.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:19:34 smithi122 ceph-mon[93869]: pgmap v6958: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 85 KiB/s wr, 7 op/s 2023-03-09T18:19:34.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:19:34 smithi181 ceph-mon[94189]: pgmap v6958: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 85 KiB/s wr, 7 op/s 2023-03-09T18:19:34.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:19:34 smithi038 ceph-mon[92422]: pgmap v6958: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 85 KiB/s wr, 7 op/s 2023-03-09T18:19:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:19:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12086.103227 secs 2023-03-09T18:19:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:19:35 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:19:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:19:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12086.103227 secs 2023-03-09T18:19:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:19:35 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:19:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:19:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12086.103227 secs 2023-03-09T18:19:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:19:35 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:19:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:19:36 smithi122 ceph-mon[93869]: pgmap v6959: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 84 KiB/s wr, 6 op/s 2023-03-09T18:19:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:19:36 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:19:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:19:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:19:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:19:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:19:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:19:36 smithi181 ceph-mon[94189]: pgmap v6959: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 84 KiB/s wr, 6 op/s 2023-03-09T18:19:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:19:36 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:19:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:19:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:19:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:19:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:19:36.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:19:36 smithi038 ceph-mon[92422]: pgmap v6959: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 84 KiB/s wr, 6 op/s 2023-03-09T18:19:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:19:36 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:19:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:19:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:19:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:19:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:19:38.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12550.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:19:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:19:38 smithi122 ceph-mon[93869]: pgmap v6960: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 86 KiB/s wr, 8 op/s 2023-03-09T18:19:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:19:38 smithi181 ceph-mon[94189]: pgmap v6960: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 86 KiB/s wr, 8 op/s 2023-03-09T18:19:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:19:38 smithi038 ceph-mon[92422]: pgmap v6960: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 86 KiB/s wr, 8 op/s 2023-03-09T18:19:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:19:40 smithi122 ceph-mon[93869]: pgmap v6961: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 69 KiB/s wr, 8 op/s 2023-03-09T18:19:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:19:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12091.103289 secs 2023-03-09T18:19:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:19:40 smithi181 ceph-mon[94189]: pgmap v6961: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 69 KiB/s wr, 8 op/s 2023-03-09T18:19:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:19:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12091.103289 secs 2023-03-09T18:19:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:19:40 smithi038 ceph-mon[92422]: pgmap v6961: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 69 KiB/s wr, 8 op/s 2023-03-09T18:19:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:19:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12091.103289 secs 2023-03-09T18:19:42.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:19:42 smithi122 ceph-mon[93869]: pgmap v6962: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 68 KiB/s wr, 10 op/s 2023-03-09T18:19:42.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:19:42 smithi181 ceph-mon[94189]: pgmap v6962: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 68 KiB/s wr, 10 op/s 2023-03-09T18:19:42.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:19:42 smithi038 ceph-mon[92422]: pgmap v6962: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 68 KiB/s wr, 10 op/s 2023-03-09T18:19:43.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12555.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:19:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:19:43 smithi122 ceph-mon[93869]: pgmap v6963: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 213 KiB/s wr, 11 op/s 2023-03-09T18:19:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:19:43 smithi181 ceph-mon[94189]: pgmap v6963: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 213 KiB/s wr, 11 op/s 2023-03-09T18:19:43.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:19:43 smithi038 ceph-mon[92422]: pgmap v6963: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 213 KiB/s wr, 11 op/s 2023-03-09T18:19:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:19:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12096.103352 secs 2023-03-09T18:19:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:19:46 smithi122 ceph-mon[93869]: pgmap v6964: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 165 KiB/s wr, 9 op/s 2023-03-09T18:19:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:19:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12096.103352 secs 2023-03-09T18:19:46.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:19:46 smithi181 ceph-mon[94189]: pgmap v6964: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 165 KiB/s wr, 9 op/s 2023-03-09T18:19:46.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:19:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12096.103352 secs 2023-03-09T18:19:46.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:19:46 smithi038 ceph-mon[92422]: pgmap v6964: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 165 KiB/s wr, 9 op/s 2023-03-09T18:19:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12560.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:19:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:19:48 smithi122 ceph-mon[93869]: pgmap v6965: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 166 KiB/s wr, 12 op/s 2023-03-09T18:19:48.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:19:48 smithi181 ceph-mon[94189]: pgmap v6965: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 166 KiB/s wr, 12 op/s 2023-03-09T18:19:48.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:19:48 smithi038 ceph-mon[92422]: pgmap v6965: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 166 KiB/s wr, 12 op/s 2023-03-09T18:19:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:19:50 smithi122 ceph-mon[93869]: pgmap v6966: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 184 KiB/s wr, 11 op/s 2023-03-09T18:19:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:19:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12101.103409 secs 2023-03-09T18:19:50.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:19:50 smithi181 ceph-mon[94189]: pgmap v6966: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 184 KiB/s wr, 11 op/s 2023-03-09T18:19:50.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:19:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12101.103409 secs 2023-03-09T18:19:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:19:50 smithi038 ceph-mon[92422]: pgmap v6966: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 184 KiB/s wr, 11 op/s 2023-03-09T18:19:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:19:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12101.103409 secs 2023-03-09T18:19:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:19:52 smithi122 ceph-mon[93869]: pgmap v6967: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 165 KiB/s wr, 12 op/s 2023-03-09T18:19:52.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:19:52 smithi181 ceph-mon[94189]: pgmap v6967: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 165 KiB/s wr, 12 op/s 2023-03-09T18:19:52.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:19:52 smithi038 ceph-mon[92422]: pgmap v6967: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 165 KiB/s wr, 12 op/s 2023-03-09T18:19:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12565.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:19:54.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:19:54 smithi122 ceph-mon[93869]: pgmap v6968: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 318 KiB/s wr, 12 op/s 2023-03-09T18:19:54.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:19:54 smithi181 ceph-mon[94189]: pgmap v6968: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 318 KiB/s wr, 12 op/s 2023-03-09T18:19:54.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:19:54 smithi038 ceph-mon[92422]: pgmap v6968: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 318 KiB/s wr, 12 op/s 2023-03-09T18:19:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:19:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12106.103494 secs 2023-03-09T18:19:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:19:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12106.103494 secs 2023-03-09T18:19:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:19:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12106.103494 secs 2023-03-09T18:19:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:19:56 smithi122 ceph-mon[93869]: pgmap v6969: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 234 KiB/s wr, 10 op/s 2023-03-09T18:19:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:19:56 smithi181 ceph-mon[94189]: pgmap v6969: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 234 KiB/s wr, 10 op/s 2023-03-09T18:19:56.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:19:56 smithi038 ceph-mon[92422]: pgmap v6969: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 234 KiB/s wr, 10 op/s 2023-03-09T18:19:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12570.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:19:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:19:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12108.399427 secs 2023-03-09T18:19:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:19:58 smithi122 ceph-mon[93869]: pgmap v6970: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 234 KiB/s wr, 11 op/s 2023-03-09T18:19:58.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:19:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12108.399427 secs 2023-03-09T18:19:58.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:19:58 smithi181 ceph-mon[94189]: pgmap v6970: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 234 KiB/s wr, 11 op/s 2023-03-09T18:19:58.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:19:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12108.399427 secs 2023-03-09T18:19:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:19:58 smithi038 ceph-mon[92422]: pgmap v6970: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 234 KiB/s wr, 11 op/s 2023-03-09T18:19:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:19:59 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:19:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:19:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:19:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:19:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:19:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:19:59 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:19:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:19:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:19:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:19:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:19:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:19:59 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:19:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:19:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:19:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:19:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:20:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:20:00 smithi122 ceph-mon[93869]: pgmap v6971: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 244 KiB/s wr, 10 op/s 2023-03-09T18:20:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:20:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12111.103556 secs 2023-03-09T18:20:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:20:00 smithi122 ceph-mon[93869]: overall HEALTH_WARN 2 MDSs report slow requests 2023-03-09T18:20:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:20:00 smithi181 ceph-mon[94189]: pgmap v6971: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 244 KiB/s wr, 10 op/s 2023-03-09T18:20:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:20:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12111.103556 secs 2023-03-09T18:20:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:20:00 smithi181 ceph-mon[94189]: overall HEALTH_WARN 2 MDSs report slow requests 2023-03-09T18:20:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:20:00 smithi038 ceph-mon[92422]: pgmap v6971: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 244 KiB/s wr, 10 op/s 2023-03-09T18:20:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:20:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12111.103556 secs 2023-03-09T18:20:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:20:00 smithi038 ceph-mon[92422]: overall HEALTH_WARN 2 MDSs report slow requests 2023-03-09T18:20:02.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:20:02 smithi122 ceph-mon[93869]: pgmap v6972: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 276 KiB/s wr, 13 op/s 2023-03-09T18:20:02.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:20:02 smithi181 ceph-mon[94189]: pgmap v6972: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 276 KiB/s wr, 13 op/s 2023-03-09T18:20:02.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:20:02 smithi038 ceph-mon[92422]: pgmap v6972: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 276 KiB/s wr, 13 op/s 2023-03-09T18:20:03.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12575.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:20:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:20:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12113.399553 secs 2023-03-09T18:20:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:20:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12113.399553 secs 2023-03-09T18:20:03.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:20:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12113.399553 secs 2023-03-09T18:20:04.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:20:04 smithi122 ceph-mon[93869]: pgmap v6973: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 297 KiB/s wr, 23 op/s 2023-03-09T18:20:04.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:20:04 smithi181 ceph-mon[94189]: pgmap v6973: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 297 KiB/s wr, 23 op/s 2023-03-09T18:20:04.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:20:04 smithi038 ceph-mon[92422]: pgmap v6973: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 297 KiB/s wr, 23 op/s 2023-03-09T18:20:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:20:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12116.103615 secs 2023-03-09T18:20:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:20:05 smithi038 ceph-mon[92422]: osdmap e649: 12 total, 12 up, 12 in 2023-03-09T18:20:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:20:05 smithi038 ceph-mon[92422]: pgmap v6975: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 160 KiB/s wr, 26 op/s 2023-03-09T18:20:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:20:05 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:20:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:20:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12116.103615 secs 2023-03-09T18:20:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:20:05 smithi122 ceph-mon[93869]: osdmap e649: 12 total, 12 up, 12 in 2023-03-09T18:20:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:20:05 smithi122 ceph-mon[93869]: pgmap v6975: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 160 KiB/s wr, 26 op/s 2023-03-09T18:20:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:20:05 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:20:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:20:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12116.103615 secs 2023-03-09T18:20:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:20:05 smithi181 ceph-mon[94189]: osdmap e649: 12 total, 12 up, 12 in 2023-03-09T18:20:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:20:05 smithi181 ceph-mon[94189]: pgmap v6975: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 160 KiB/s wr, 26 op/s 2023-03-09T18:20:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:20:05 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:20:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:20:06 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:20:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:20:06 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:20:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:20:06 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:20:06.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:20:06 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:20:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:20:06 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:20:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:20:06 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:20:07.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:20:06 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:20:07.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:20:06 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:20:07.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:20:06 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:20:07.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:20:07 smithi038 ceph-mon[92422]: pgmap v6976: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 173 KiB/s wr, 27 op/s 2023-03-09T18:20:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:20:07 smithi122 ceph-mon[93869]: pgmap v6976: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 173 KiB/s wr, 27 op/s 2023-03-09T18:20:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:20:07 smithi181 ceph-mon[94189]: pgmap v6976: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 173 KiB/s wr, 27 op/s 2023-03-09T18:20:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12580.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:20:08.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:20:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12118.399667 secs 2023-03-09T18:20:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:20:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12118.399667 secs 2023-03-09T18:20:09.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:20:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12118.399667 secs 2023-03-09T18:20:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:20:09 smithi122 ceph-mon[93869]: pgmap v6977: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 173 KiB/s wr, 26 op/s 2023-03-09T18:20:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:20:09 smithi181 ceph-mon[94189]: pgmap v6977: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 173 KiB/s wr, 26 op/s 2023-03-09T18:20:10.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:20:09 smithi038 ceph-mon[92422]: pgmap v6977: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 173 KiB/s wr, 26 op/s 2023-03-09T18:20:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:20:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12121.103674 secs 2023-03-09T18:20:11.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:20:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12121.103674 secs 2023-03-09T18:20:11.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:20:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12121.103674 secs 2023-03-09T18:20:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:20:12 smithi122 ceph-mon[93869]: pgmap v6978: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 141 KiB/s wr, 24 op/s 2023-03-09T18:20:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:20:12 smithi122 ceph-mon[93869]: osdmap e650: 12 total, 12 up, 12 in 2023-03-09T18:20:12.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:20:12 smithi181 ceph-mon[94189]: pgmap v6978: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 141 KiB/s wr, 24 op/s 2023-03-09T18:20:12.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:20:12 smithi181 ceph-mon[94189]: osdmap e650: 12 total, 12 up, 12 in 2023-03-09T18:20:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:20:12 smithi038 ceph-mon[92422]: pgmap v6978: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 141 KiB/s wr, 24 op/s 2023-03-09T18:20:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:20:12 smithi038 ceph-mon[92422]: osdmap e650: 12 total, 12 up, 12 in 2023-03-09T18:20:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12585.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:20:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:20:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12123.399807 secs 2023-03-09T18:20:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:20:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12123.399807 secs 2023-03-09T18:20:13.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:20:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12123.399807 secs 2023-03-09T18:20:14.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:20:14 smithi122 ceph-mon[93869]: pgmap v6980: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 147 KiB/s wr, 11 op/s 2023-03-09T18:20:14.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:20:14 smithi181 ceph-mon[94189]: pgmap v6980: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 147 KiB/s wr, 11 op/s 2023-03-09T18:20:14.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:20:14 smithi038 ceph-mon[92422]: pgmap v6980: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 147 KiB/s wr, 11 op/s 2023-03-09T18:20:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:20:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12126.103734 secs 2023-03-09T18:20:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:20:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12126.103734 secs 2023-03-09T18:20:15.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:20:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12126.103734 secs 2023-03-09T18:20:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:20:16 smithi122 ceph-mon[93869]: pgmap v6981: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 69 KiB/s wr, 8 op/s 2023-03-09T18:20:16.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:20:16 smithi181 ceph-mon[94189]: pgmap v6981: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 69 KiB/s wr, 8 op/s 2023-03-09T18:20:16.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:20:16 smithi038 ceph-mon[92422]: pgmap v6981: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 69 KiB/s wr, 8 op/s 2023-03-09T18:20:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12590.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:20:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:20:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12128.399925 secs 2023-03-09T18:20:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:20:18 smithi122 ceph-mon[93869]: pgmap v6982: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 55 KiB/s wr, 8 op/s 2023-03-09T18:20:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:20:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12128.399925 secs 2023-03-09T18:20:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:20:18 smithi181 ceph-mon[94189]: pgmap v6982: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 55 KiB/s wr, 8 op/s 2023-03-09T18:20:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:20:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12128.399925 secs 2023-03-09T18:20:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:20:18 smithi038 ceph-mon[92422]: pgmap v6982: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 55 KiB/s wr, 8 op/s 2023-03-09T18:20:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:20:20 smithi122 ceph-mon[93869]: pgmap v6983: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 43 KiB/s wr, 8 op/s 2023-03-09T18:20:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:20:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12131.103797 secs 2023-03-09T18:20:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:20:20 smithi181 ceph-mon[94189]: pgmap v6983: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 43 KiB/s wr, 8 op/s 2023-03-09T18:20:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:20:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12131.103797 secs 2023-03-09T18:20:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:20:20 smithi038 ceph-mon[92422]: pgmap v6983: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 43 KiB/s wr, 8 op/s 2023-03-09T18:20:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:20:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12131.103797 secs 2023-03-09T18:20:22.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:20:22 smithi122 ceph-mon[93869]: pgmap v6984: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 38 KiB/s wr, 7 op/s 2023-03-09T18:20:22.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:20:22 smithi181 ceph-mon[94189]: pgmap v6984: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 38 KiB/s wr, 7 op/s 2023-03-09T18:20:22.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:20:22 smithi038 ceph-mon[92422]: pgmap v6984: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 38 KiB/s wr, 7 op/s 2023-03-09T18:20:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12595.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:20:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:20:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12133.399991 secs 2023-03-09T18:20:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:20:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12133.399991 secs 2023-03-09T18:20:23.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:20:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12133.399991 secs 2023-03-09T18:20:24.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:20:24 smithi122 ceph-mon[93869]: pgmap v6985: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T18:20:24.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:20:24 smithi181 ceph-mon[94189]: pgmap v6985: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T18:20:24.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:20:24 smithi038 ceph-mon[92422]: pgmap v6985: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T18:20:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:20:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12136.103862 secs 2023-03-09T18:20:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:20:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12136.103862 secs 2023-03-09T18:20:25.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:20:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12136.103862 secs 2023-03-09T18:20:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:20:26 smithi122 ceph-mon[93869]: pgmap v6986: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 57 KiB/s wr, 7 op/s 2023-03-09T18:20:26.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:20:26 smithi181 ceph-mon[94189]: pgmap v6986: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 57 KiB/s wr, 7 op/s 2023-03-09T18:20:26.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:20:26 smithi038 ceph-mon[92422]: pgmap v6986: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 57 KiB/s wr, 7 op/s 2023-03-09T18:20:28.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12600.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:20:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:20:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12138.400082 secs 2023-03-09T18:20:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:20:28 smithi122 ceph-mon[93869]: pgmap v6987: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 9 op/s 2023-03-09T18:20:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:20:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12138.400082 secs 2023-03-09T18:20:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:20:28 smithi181 ceph-mon[94189]: pgmap v6987: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 9 op/s 2023-03-09T18:20:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:20:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12138.400082 secs 2023-03-09T18:20:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:20:28 smithi038 ceph-mon[92422]: pgmap v6987: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 9 op/s 2023-03-09T18:20:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:20:30 smithi122 ceph-mon[93869]: pgmap v6988: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 58 KiB/s wr, 7 op/s 2023-03-09T18:20:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:20:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12141.103923 secs 2023-03-09T18:20:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:20:30 smithi181 ceph-mon[94189]: pgmap v6988: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 58 KiB/s wr, 7 op/s 2023-03-09T18:20:30.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:20:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12141.103923 secs 2023-03-09T18:20:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:20:30 smithi038 ceph-mon[92422]: pgmap v6988: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 58 KiB/s wr, 7 op/s 2023-03-09T18:20:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:20:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12141.103923 secs 2023-03-09T18:20:32.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:20:32 smithi122 ceph-mon[93869]: pgmap v6989: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 91 KiB/s wr, 9 op/s 2023-03-09T18:20:32.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:20:32 smithi181 ceph-mon[94189]: pgmap v6989: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 91 KiB/s wr, 9 op/s 2023-03-09T18:20:32.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:20:32 smithi038 ceph-mon[92422]: pgmap v6989: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 91 KiB/s wr, 9 op/s 2023-03-09T18:20:33.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12605.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:20:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:20:33 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 12143.400151 secs 2023-03-09T18:20:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:20:33 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 12143.400151 secs 2023-03-09T18:20:33.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:20:33 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 12143.400151 secs 2023-03-09T18:20:34.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:20:34 smithi122 ceph-mon[93869]: pgmap v6990: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 71 KiB/s wr, 7 op/s 2023-03-09T18:20:34.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:20:34 smithi181 ceph-mon[94189]: pgmap v6990: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 71 KiB/s wr, 7 op/s 2023-03-09T18:20:34.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:20:34 smithi038 ceph-mon[92422]: pgmap v6990: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 71 KiB/s wr, 7 op/s 2023-03-09T18:20:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:20:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12146.103985 secs 2023-03-09T18:20:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:20:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12146.103985 secs 2023-03-09T18:20:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:20:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12146.103985 secs 2023-03-09T18:20:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:20:36 smithi122 ceph-mon[93869]: pgmap v6991: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 148 KiB/s wr, 7 op/s 2023-03-09T18:20:36.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:20:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:20:36.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:20:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:20:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:20:36 smithi181 ceph-mon[94189]: pgmap v6991: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 148 KiB/s wr, 7 op/s 2023-03-09T18:20:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:20:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:20:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:20:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:20:36.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:20:36 smithi038 ceph-mon[92422]: pgmap v6991: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 148 KiB/s wr, 7 op/s 2023-03-09T18:20:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:20:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:20:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:20:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:20:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:20:37 smithi122 ceph-mon[93869]: pgmap v6992: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 112 KiB/s wr, 9 op/s 2023-03-09T18:20:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:20:37 smithi181 ceph-mon[94189]: pgmap v6992: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 112 KiB/s wr, 9 op/s 2023-03-09T18:20:38.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:20:37 smithi038 ceph-mon[92422]: pgmap v6992: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 112 KiB/s wr, 9 op/s 2023-03-09T18:20:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12610.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:20:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:20:38 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 12148.400250 secs 2023-03-09T18:20:39.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:20:38 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 12148.400250 secs 2023-03-09T18:20:39.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:20:38 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 12148.400250 secs 2023-03-09T18:20:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:20:39 smithi122 ceph-mon[93869]: pgmap v6993: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 122 KiB/s wr, 8 op/s 2023-03-09T18:20:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:20:39 smithi181 ceph-mon[94189]: pgmap v6993: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 122 KiB/s wr, 8 op/s 2023-03-09T18:20:40.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:20:39 smithi038 ceph-mon[92422]: pgmap v6993: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 122 KiB/s wr, 8 op/s 2023-03-09T18:20:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:20:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12151.104047 secs 2023-03-09T18:20:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:20:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12151.104047 secs 2023-03-09T18:20:41.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:20:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12151.104047 secs 2023-03-09T18:20:42.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:20:42 smithi122 ceph-mon[93869]: pgmap v6994: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 181 KiB/s wr, 11 op/s 2023-03-09T18:20:42.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:20:42 smithi181 ceph-mon[94189]: pgmap v6994: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 181 KiB/s wr, 11 op/s 2023-03-09T18:20:42.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:20:42 smithi038 ceph-mon[92422]: pgmap v6994: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 181 KiB/s wr, 11 op/s 2023-03-09T18:20:43.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12615.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:20:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:20:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12153.400370 secs 2023-03-09T18:20:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:20:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12153.400370 secs 2023-03-09T18:20:43.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:20:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12153.400370 secs 2023-03-09T18:20:44.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:20:44 smithi122 ceph-mon[93869]: pgmap v6995: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 158 KiB/s wr, 10 op/s 2023-03-09T18:20:44.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:20:44 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:20:44.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:20:44 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:20:44.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:20:44 smithi181 ceph-mon[94189]: pgmap v6995: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 158 KiB/s wr, 10 op/s 2023-03-09T18:20:44.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:20:44 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:20:44.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:20:44 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:20:44.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:20:44 smithi038 ceph-mon[92422]: pgmap v6995: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 158 KiB/s wr, 10 op/s 2023-03-09T18:20:44.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:20:44 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:20:44.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:20:44 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:20:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:20:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12156.104108 secs 2023-03-09T18:20:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:20:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12156.104108 secs 2023-03-09T18:20:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:20:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12156.104108 secs 2023-03-09T18:20:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:20:46 smithi122 ceph-mon[93869]: pgmap v6996: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 158 KiB/s wr, 10 op/s 2023-03-09T18:20:46.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:20:46 smithi181 ceph-mon[94189]: pgmap v6996: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 158 KiB/s wr, 10 op/s 2023-03-09T18:20:46.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:20:46 smithi038 ceph-mon[92422]: pgmap v6996: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 158 KiB/s wr, 10 op/s 2023-03-09T18:20:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12620.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:20:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:20:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12158.400482 secs 2023-03-09T18:20:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:20:48 smithi122 ceph-mon[93869]: pgmap v6997: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 283 KiB/s rd, 189 KiB/s wr, 13 op/s 2023-03-09T18:20:48.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:20:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12158.400482 secs 2023-03-09T18:20:48.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:20:48 smithi181 ceph-mon[94189]: pgmap v6997: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 283 KiB/s rd, 189 KiB/s wr, 13 op/s 2023-03-09T18:20:48.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:20:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12158.400482 secs 2023-03-09T18:20:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:20:48 smithi038 ceph-mon[92422]: pgmap v6997: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 283 KiB/s rd, 189 KiB/s wr, 13 op/s 2023-03-09T18:20:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:20:50 smithi122 ceph-mon[93869]: pgmap v6998: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 200 KiB/s wr, 11 op/s 2023-03-09T18:20:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:20:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12161.104166 secs 2023-03-09T18:20:50.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:20:50 smithi181 ceph-mon[94189]: pgmap v6998: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 200 KiB/s wr, 11 op/s 2023-03-09T18:20:50.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:20:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12161.104166 secs 2023-03-09T18:20:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:20:50 smithi038 ceph-mon[92422]: pgmap v6998: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 200 KiB/s wr, 11 op/s 2023-03-09T18:20:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:20:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12161.104166 secs 2023-03-09T18:20:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:20:52 smithi122 ceph-mon[93869]: pgmap v6999: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 313 KiB/s rd, 231 KiB/s wr, 13 op/s 2023-03-09T18:20:52.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:20:52 smithi181 ceph-mon[94189]: pgmap v6999: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 313 KiB/s rd, 231 KiB/s wr, 13 op/s 2023-03-09T18:20:52.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:20:52 smithi038 ceph-mon[92422]: pgmap v6999: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 313 KiB/s rd, 231 KiB/s wr, 13 op/s 2023-03-09T18:20:53.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12625.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:20:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:20:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12163.400600 secs 2023-03-09T18:20:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:20:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12163.400600 secs 2023-03-09T18:20:53.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:20:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12163.400600 secs 2023-03-09T18:20:54.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:20:54 smithi122 ceph-mon[93869]: pgmap v7000: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 192 KiB/s wr, 11 op/s 2023-03-09T18:20:54.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:20:54 smithi181 ceph-mon[94189]: pgmap v7000: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 192 KiB/s wr, 11 op/s 2023-03-09T18:20:54.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:20:54 smithi038 ceph-mon[92422]: pgmap v7000: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 192 KiB/s wr, 11 op/s 2023-03-09T18:20:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:20:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12166.104226 secs 2023-03-09T18:20:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:20:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12166.104226 secs 2023-03-09T18:20:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:20:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12166.104226 secs 2023-03-09T18:20:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:20:56 smithi122 ceph-mon[93869]: pgmap v7001: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 182 KiB/s wr, 9 op/s 2023-03-09T18:20:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:20:56 smithi181 ceph-mon[94189]: pgmap v7001: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 182 KiB/s wr, 9 op/s 2023-03-09T18:20:56.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:20:56 smithi038 ceph-mon[92422]: pgmap v7001: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 182 KiB/s wr, 9 op/s 2023-03-09T18:20:58.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12630.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:20:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:20:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12168.400741 secs 2023-03-09T18:20:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:20:58 smithi122 ceph-mon[93869]: pgmap v7002: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 251 KiB/s wr, 12 op/s 2023-03-09T18:20:58.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:20:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12168.400741 secs 2023-03-09T18:20:58.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:20:58 smithi181 ceph-mon[94189]: pgmap v7002: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 251 KiB/s wr, 12 op/s 2023-03-09T18:20:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:20:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12168.400741 secs 2023-03-09T18:20:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:20:58 smithi038 ceph-mon[92422]: pgmap v7002: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 251 KiB/s wr, 12 op/s 2023-03-09T18:21:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:21:00 smithi122 ceph-mon[93869]: pgmap v7003: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 143 KiB/s wr, 9 op/s 2023-03-09T18:21:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:21:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12171.104305 secs 2023-03-09T18:21:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:21:00 smithi181 ceph-mon[94189]: pgmap v7003: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 143 KiB/s wr, 9 op/s 2023-03-09T18:21:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:21:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12171.104305 secs 2023-03-09T18:21:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:21:00 smithi038 ceph-mon[92422]: pgmap v7003: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 143 KiB/s wr, 9 op/s 2023-03-09T18:21:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:21:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12171.104305 secs 2023-03-09T18:21:02.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:21:02 smithi122 ceph-mon[93869]: pgmap v7004: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 191 KiB/s wr, 11 op/s 2023-03-09T18:21:02.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:21:02 smithi181 ceph-mon[94189]: pgmap v7004: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 191 KiB/s wr, 11 op/s 2023-03-09T18:21:02.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:21:02 smithi038 ceph-mon[92422]: pgmap v7004: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 191 KiB/s wr, 11 op/s 2023-03-09T18:21:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12635.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:21:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:21:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12173.400858 secs 2023-03-09T18:21:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:21:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12173.400858 secs 2023-03-09T18:21:03.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:21:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12173.400858 secs 2023-03-09T18:21:04.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:21:04 smithi122 ceph-mon[93869]: pgmap v7005: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 170 KiB/s wr, 20 op/s 2023-03-09T18:21:04.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:21:04 smithi181 ceph-mon[94189]: pgmap v7005: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 170 KiB/s wr, 20 op/s 2023-03-09T18:21:04.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:21:04 smithi038 ceph-mon[92422]: pgmap v7005: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 170 KiB/s wr, 20 op/s 2023-03-09T18:21:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:21:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12176.104373 secs 2023-03-09T18:21:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:21:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12176.104373 secs 2023-03-09T18:21:05.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:21:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12176.104373 secs 2023-03-09T18:21:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:21:06 smithi122 ceph-mon[93869]: pgmap v7006: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 150 KiB/s wr, 19 op/s 2023-03-09T18:21:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:21:06 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:21:06.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:21:06 smithi181 ceph-mon[94189]: pgmap v7006: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 150 KiB/s wr, 19 op/s 2023-03-09T18:21:06.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:21:06 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:21:06.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:21:06 smithi038 ceph-mon[92422]: pgmap v7006: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 150 KiB/s wr, 19 op/s 2023-03-09T18:21:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:21:06 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:21:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12640.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:21:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:21:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12178.400979 secs 2023-03-09T18:21:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:21:08 smithi122 ceph-mon[93869]: pgmap v7007: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 173 KiB/s wr, 24 op/s 2023-03-09T18:21:08.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:21:08 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:21:08.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:21:08 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:21:08.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:21:08 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:21:08.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:21:08 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:21:08.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:21:08 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:21:08.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:21:08 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:21:08.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:21:08 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:21:08.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:21:08 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:21:08.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:21:08 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:21:08.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:21:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12178.400979 secs 2023-03-09T18:21:08.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:21:08 smithi181 ceph-mon[94189]: pgmap v7007: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 173 KiB/s wr, 24 op/s 2023-03-09T18:21:08.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:21:08 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:21:08.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:21:08 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:21:08.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:21:08 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:21:08.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:21:08 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:21:08.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:21:08 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:21:08.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:21:08 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:21:08.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:21:08 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:21:08.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:21:08 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:21:08.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:21:08 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:21:08.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:21:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12178.400979 secs 2023-03-09T18:21:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:21:08 smithi038 ceph-mon[92422]: pgmap v7007: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 173 KiB/s wr, 24 op/s 2023-03-09T18:21:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:21:08 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:21:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:21:08 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:21:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:21:08 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:21:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:21:08 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:21:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:21:08 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:21:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:21:08 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:21:09.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:21:08 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:21:09.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:21:08 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:21:09.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:21:08 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:21:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:21:10 smithi122 ceph-mon[93869]: pgmap v7008: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 124 KiB/s wr, 22 op/s 2023-03-09T18:21:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:21:10 smithi122 ceph-mon[93869]: osdmap e651: 12 total, 12 up, 12 in 2023-03-09T18:21:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:21:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12181.104443 secs 2023-03-09T18:21:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:21:10 smithi181 ceph-mon[94189]: pgmap v7008: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 124 KiB/s wr, 22 op/s 2023-03-09T18:21:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:21:10 smithi181 ceph-mon[94189]: osdmap e651: 12 total, 12 up, 12 in 2023-03-09T18:21:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:21:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12181.104443 secs 2023-03-09T18:21:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:21:10 smithi038 ceph-mon[92422]: pgmap v7008: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 124 KiB/s wr, 22 op/s 2023-03-09T18:21:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:21:10 smithi038 ceph-mon[92422]: osdmap e651: 12 total, 12 up, 12 in 2023-03-09T18:21:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:21:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12181.104443 secs 2023-03-09T18:21:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:21:12 smithi122 ceph-mon[93869]: pgmap v7010: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 109 KiB/s wr, 26 op/s 2023-03-09T18:21:12.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:21:12 smithi181 ceph-mon[94189]: pgmap v7010: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 109 KiB/s wr, 26 op/s 2023-03-09T18:21:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:21:12 smithi038 ceph-mon[92422]: pgmap v7010: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 109 KiB/s wr, 26 op/s 2023-03-09T18:21:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12645.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:21:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:21:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12183.401134 secs 2023-03-09T18:21:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:21:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12183.401134 secs 2023-03-09T18:21:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:21:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12183.401134 secs 2023-03-09T18:21:14.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:21:14 smithi122 ceph-mon[93869]: pgmap v7011: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 84 KiB/s wr, 12 op/s 2023-03-09T18:21:14.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:21:14 smithi181 ceph-mon[94189]: pgmap v7011: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 84 KiB/s wr, 12 op/s 2023-03-09T18:21:14.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:21:14 smithi038 ceph-mon[92422]: pgmap v7011: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 84 KiB/s wr, 12 op/s 2023-03-09T18:21:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:21:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12186.104530 secs 2023-03-09T18:21:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:21:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12186.104530 secs 2023-03-09T18:21:15.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:21:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12186.104530 secs 2023-03-09T18:21:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:21:16 smithi122 ceph-mon[93869]: pgmap v7012: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 83 KiB/s wr, 12 op/s 2023-03-09T18:21:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:21:16 smithi122 ceph-mon[93869]: osdmap e652: 12 total, 12 up, 12 in 2023-03-09T18:21:16.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:21:16 smithi181 ceph-mon[94189]: pgmap v7012: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 83 KiB/s wr, 12 op/s 2023-03-09T18:21:16.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:21:16 smithi181 ceph-mon[94189]: osdmap e652: 12 total, 12 up, 12 in 2023-03-09T18:21:16.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:21:16 smithi038 ceph-mon[92422]: pgmap v7012: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 83 KiB/s wr, 12 op/s 2023-03-09T18:21:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:21:16 smithi038 ceph-mon[92422]: osdmap e652: 12 total, 12 up, 12 in 2023-03-09T18:21:18.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12650.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:21:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:21:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12188.401276 secs 2023-03-09T18:21:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:21:18 smithi122 ceph-mon[93869]: pgmap v7014: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 93 KiB/s wr, 10 op/s 2023-03-09T18:21:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:21:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12188.401276 secs 2023-03-09T18:21:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:21:18 smithi181 ceph-mon[94189]: pgmap v7014: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 93 KiB/s wr, 10 op/s 2023-03-09T18:21:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:21:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12188.401276 secs 2023-03-09T18:21:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:21:18 smithi038 ceph-mon[92422]: pgmap v7014: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 93 KiB/s wr, 10 op/s 2023-03-09T18:21:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:21:20 smithi122 ceph-mon[93869]: pgmap v7015: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 75 KiB/s wr, 9 op/s 2023-03-09T18:21:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:21:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12191.104606 secs 2023-03-09T18:21:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:21:20 smithi181 ceph-mon[94189]: pgmap v7015: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 75 KiB/s wr, 9 op/s 2023-03-09T18:21:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:21:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12191.104606 secs 2023-03-09T18:21:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:21:20 smithi038 ceph-mon[92422]: pgmap v7015: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 75 KiB/s wr, 9 op/s 2023-03-09T18:21:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:21:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12191.104606 secs 2023-03-09T18:21:22.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:21:22 smithi122 ceph-mon[93869]: pgmap v7016: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 45 KiB/s wr, 8 op/s 2023-03-09T18:21:22.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:21:22 smithi181 ceph-mon[94189]: pgmap v7016: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 45 KiB/s wr, 8 op/s 2023-03-09T18:21:22.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:21:22 smithi038 ceph-mon[92422]: pgmap v7016: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 45 KiB/s wr, 8 op/s 2023-03-09T18:21:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12655.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:21:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:21:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12193.401356 secs 2023-03-09T18:21:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:21:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12193.401356 secs 2023-03-09T18:21:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:21:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12193.401356 secs 2023-03-09T18:21:24.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:21:24 smithi122 ceph-mon[93869]: pgmap v7017: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 45 KiB/s wr, 8 op/s 2023-03-09T18:21:24.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:21:24 smithi181 ceph-mon[94189]: pgmap v7017: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 45 KiB/s wr, 8 op/s 2023-03-09T18:21:24.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:21:24 smithi038 ceph-mon[92422]: pgmap v7017: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 45 KiB/s wr, 8 op/s 2023-03-09T18:21:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:21:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12196.104669 secs 2023-03-09T18:21:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:21:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12196.104669 secs 2023-03-09T18:21:25.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:21:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12196.104669 secs 2023-03-09T18:21:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:21:26 smithi122 ceph-mon[93869]: pgmap v7018: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 45 KiB/s wr, 8 op/s 2023-03-09T18:21:26.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:21:26 smithi181 ceph-mon[94189]: pgmap v7018: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 45 KiB/s wr, 8 op/s 2023-03-09T18:21:26.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:21:26 smithi038 ceph-mon[92422]: pgmap v7018: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 45 KiB/s wr, 8 op/s 2023-03-09T18:21:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12660.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:21:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:21:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12198.401441 secs 2023-03-09T18:21:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:21:28 smithi122 ceph-mon[93869]: pgmap v7019: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 470 B/s wr, 7 op/s 2023-03-09T18:21:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:21:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12198.401441 secs 2023-03-09T18:21:28.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:21:28 smithi181 ceph-mon[94189]: pgmap v7019: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 470 B/s wr, 7 op/s 2023-03-09T18:21:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:21:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12198.401441 secs 2023-03-09T18:21:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:21:28 smithi038 ceph-mon[92422]: pgmap v7019: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 470 B/s wr, 7 op/s 2023-03-09T18:21:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:21:30 smithi122 ceph-mon[93869]: pgmap v7020: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 7 op/s 2023-03-09T18:21:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:21:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12201.104756 secs 2023-03-09T18:21:30.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:21:30 smithi181 ceph-mon[94189]: pgmap v7020: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 7 op/s 2023-03-09T18:21:30.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:21:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12201.104756 secs 2023-03-09T18:21:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:21:30 smithi038 ceph-mon[92422]: pgmap v7020: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 7 op/s 2023-03-09T18:21:30.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:21:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12201.104756 secs 2023-03-09T18:21:32.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:21:32 smithi122 ceph-mon[93869]: pgmap v7021: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 9 op/s 2023-03-09T18:21:32.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:21:32 smithi181 ceph-mon[94189]: pgmap v7021: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 9 op/s 2023-03-09T18:21:32.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:21:32 smithi038 ceph-mon[92422]: pgmap v7021: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 9 op/s 2023-03-09T18:21:33.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12665.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:21:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:21:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12203.401554 secs 2023-03-09T18:21:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:21:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12203.401554 secs 2023-03-09T18:21:33.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:21:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12203.401554 secs 2023-03-09T18:21:34.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:21:34 smithi122 ceph-mon[93869]: pgmap v7022: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 7 op/s 2023-03-09T18:21:34.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:21:34 smithi181 ceph-mon[94189]: pgmap v7022: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 7 op/s 2023-03-09T18:21:34.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:21:34 smithi038 ceph-mon[92422]: pgmap v7022: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 7 op/s 2023-03-09T18:21:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:21:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12206.104817 secs 2023-03-09T18:21:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:21:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12206.104817 secs 2023-03-09T18:21:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:21:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12206.104817 secs 2023-03-09T18:21:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:21:36 smithi122 ceph-mon[93869]: pgmap v7023: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 6 op/s 2023-03-09T18:21:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:21:36 smithi181 ceph-mon[94189]: pgmap v7023: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 6 op/s 2023-03-09T18:21:36.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:21:36 smithi038 ceph-mon[92422]: pgmap v7023: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 6 op/s 2023-03-09T18:21:38.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12670.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:21:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:21:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12208.401689 secs 2023-03-09T18:21:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:21:38 smithi122 ceph-mon[93869]: pgmap v7024: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 37 KiB/s wr, 8 op/s 2023-03-09T18:21:38.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:21:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12208.401689 secs 2023-03-09T18:21:38.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:21:38 smithi181 ceph-mon[94189]: pgmap v7024: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 37 KiB/s wr, 8 op/s 2023-03-09T18:21:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:21:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12208.401689 secs 2023-03-09T18:21:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:21:38 smithi038 ceph-mon[92422]: pgmap v7024: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 37 KiB/s wr, 8 op/s 2023-03-09T18:21:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:21:40 smithi122 ceph-mon[93869]: pgmap v7025: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 37 KiB/s wr, 6 op/s 2023-03-09T18:21:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:21:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12211.104939 secs 2023-03-09T18:21:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:21:40 smithi181 ceph-mon[94189]: pgmap v7025: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 37 KiB/s wr, 6 op/s 2023-03-09T18:21:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:21:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12211.104939 secs 2023-03-09T18:21:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:21:40 smithi038 ceph-mon[92422]: pgmap v7025: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 37 KiB/s wr, 6 op/s 2023-03-09T18:21:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:21:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12211.104939 secs 2023-03-09T18:21:42.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:21:42 smithi038 ceph-mon[92422]: pgmap v7026: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 104 KiB/s wr, 8 op/s 2023-03-09T18:21:43.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:21:42 smithi122 ceph-mon[93869]: pgmap v7026: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 104 KiB/s wr, 8 op/s 2023-03-09T18:21:43.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:21:42 smithi181 ceph-mon[94189]: pgmap v7026: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 104 KiB/s wr, 8 op/s 2023-03-09T18:21:43.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12675.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:21:43.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:21:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12213.401801 secs 2023-03-09T18:21:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:21:43 smithi038 ceph-mon[92422]: pgmap v7027: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 114 KiB/s wr, 7 op/s 2023-03-09T18:21:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:21:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12213.401801 secs 2023-03-09T18:21:44.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:21:43 smithi122 ceph-mon[93869]: pgmap v7027: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 114 KiB/s wr, 7 op/s 2023-03-09T18:21:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:21:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12213.401801 secs 2023-03-09T18:21:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:21:43 smithi181 ceph-mon[94189]: pgmap v7027: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 114 KiB/s wr, 7 op/s 2023-03-09T18:21:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:21:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12216.104988 secs 2023-03-09T18:21:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:21:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12216.104988 secs 2023-03-09T18:21:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:21:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12216.104988 secs 2023-03-09T18:21:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:21:46 smithi122 ceph-mon[93869]: pgmap v7028: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 114 KiB/s wr, 7 op/s 2023-03-09T18:21:46.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:21:46 smithi181 ceph-mon[94189]: pgmap v7028: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 114 KiB/s wr, 7 op/s 2023-03-09T18:21:46.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:21:46 smithi038 ceph-mon[92422]: pgmap v7028: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 114 KiB/s wr, 7 op/s 2023-03-09T18:21:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12680.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:21:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:21:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12218.401920 secs 2023-03-09T18:21:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:21:48 smithi122 ceph-mon[93869]: pgmap v7029: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 214 KiB/s wr, 10 op/s 2023-03-09T18:21:48.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:21:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12218.401920 secs 2023-03-09T18:21:48.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:21:48 smithi181 ceph-mon[94189]: pgmap v7029: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 214 KiB/s wr, 10 op/s 2023-03-09T18:21:48.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:21:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12218.401920 secs 2023-03-09T18:21:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:21:48 smithi038 ceph-mon[92422]: pgmap v7029: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 214 KiB/s wr, 10 op/s 2023-03-09T18:21:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:21:50 smithi122 ceph-mon[93869]: pgmap v7030: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 187 KiB/s wr, 9 op/s 2023-03-09T18:21:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:21:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12221.105047 secs 2023-03-09T18:21:50.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:21:50 smithi181 ceph-mon[94189]: pgmap v7030: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 187 KiB/s wr, 9 op/s 2023-03-09T18:21:50.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:21:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12221.105047 secs 2023-03-09T18:21:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:21:50 smithi038 ceph-mon[92422]: pgmap v7030: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 187 KiB/s wr, 9 op/s 2023-03-09T18:21:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:21:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12221.105047 secs 2023-03-09T18:21:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:21:52 smithi122 ceph-mon[93869]: pgmap v7031: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 301 KiB/s wr, 12 op/s 2023-03-09T18:21:52.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:21:52 smithi181 ceph-mon[94189]: pgmap v7031: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 301 KiB/s wr, 12 op/s 2023-03-09T18:21:52.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:21:52 smithi038 ceph-mon[92422]: pgmap v7031: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 301 KiB/s wr, 12 op/s 2023-03-09T18:21:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12685.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:21:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:21:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12223.401998 secs 2023-03-09T18:21:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:21:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12223.401998 secs 2023-03-09T18:21:53.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:21:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12223.401998 secs 2023-03-09T18:21:54.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:21:54 smithi122 ceph-mon[93869]: pgmap v7032: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 246 KiB/s wr, 11 op/s 2023-03-09T18:21:54.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:21:54 smithi181 ceph-mon[94189]: pgmap v7032: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 246 KiB/s wr, 11 op/s 2023-03-09T18:21:54.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:21:54 smithi038 ceph-mon[92422]: pgmap v7032: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 246 KiB/s wr, 11 op/s 2023-03-09T18:21:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:21:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12226.105108 secs 2023-03-09T18:21:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:21:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12226.105108 secs 2023-03-09T18:21:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:21:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12226.105108 secs 2023-03-09T18:21:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:21:56 smithi122 ceph-mon[93869]: pgmap v7033: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 235 KiB/s wr, 10 op/s 2023-03-09T18:21:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:21:56 smithi181 ceph-mon[94189]: pgmap v7033: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 235 KiB/s wr, 10 op/s 2023-03-09T18:21:56.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:21:56 smithi038 ceph-mon[92422]: pgmap v7033: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 235 KiB/s wr, 10 op/s 2023-03-09T18:21:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12690.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:21:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:21:58 smithi122 ceph-mon[93869]: pgmap v7034: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 252 KiB/s wr, 13 op/s 2023-03-09T18:21:58.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:21:58 smithi181 ceph-mon[94189]: pgmap v7034: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 252 KiB/s wr, 13 op/s 2023-03-09T18:21:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:21:58 smithi038 ceph-mon[92422]: pgmap v7034: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 252 KiB/s wr, 13 op/s 2023-03-09T18:21:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:21:59 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:21:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:21:59 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:21:59.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:21:59 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:22:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:00 smithi122 ceph-mon[93869]: pgmap v7035: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 157 KiB/s wr, 10 op/s 2023-03-09T18:22:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12231.105166 secs 2023-03-09T18:22:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:00 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:22:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:22:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:22:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:00 smithi181 ceph-mon[94189]: pgmap v7035: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 157 KiB/s wr, 10 op/s 2023-03-09T18:22:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12231.105166 secs 2023-03-09T18:22:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:00 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:22:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:22:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:22:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:00 smithi038 ceph-mon[92422]: pgmap v7035: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 157 KiB/s wr, 10 op/s 2023-03-09T18:22:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12231.105166 secs 2023-03-09T18:22:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:00 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:22:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:22:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:22:02.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:02 smithi122 ceph-mon[93869]: pgmap v7036: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 276 KiB/s wr, 13 op/s 2023-03-09T18:22:02.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:02 smithi181 ceph-mon[94189]: pgmap v7036: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 276 KiB/s wr, 13 op/s 2023-03-09T18:22:02.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:02 smithi038 ceph-mon[92422]: pgmap v7036: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 276 KiB/s wr, 13 op/s 2023-03-09T18:22:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12695.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:22:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12233.402242 secs 2023-03-09T18:22:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12233.402242 secs 2023-03-09T18:22:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12233.402242 secs 2023-03-09T18:22:04.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:04 smithi122 ceph-mon[93869]: pgmap v7037: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 183 KiB/s wr, 21 op/s 2023-03-09T18:22:04.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:04 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:22:04.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:04 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:22:04.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:04 smithi181 ceph-mon[94189]: pgmap v7037: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 183 KiB/s wr, 21 op/s 2023-03-09T18:22:04.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:04 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:22:04.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:04 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:22:04.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:04 smithi038 ceph-mon[92422]: pgmap v7037: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 183 KiB/s wr, 21 op/s 2023-03-09T18:22:04.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:04 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:22:04.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:04 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:22:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12236.105226 secs 2023-03-09T18:22:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12236.105226 secs 2023-03-09T18:22:05.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12236.105226 secs 2023-03-09T18:22:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:06 smithi122 ceph-mon[93869]: pgmap v7038: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 172 KiB/s wr, 20 op/s 2023-03-09T18:22:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:06 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:22:06.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:06 smithi181 ceph-mon[94189]: pgmap v7038: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 172 KiB/s wr, 20 op/s 2023-03-09T18:22:06.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:06 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:22:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:06 smithi038 ceph-mon[92422]: pgmap v7038: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 172 KiB/s wr, 20 op/s 2023-03-09T18:22:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:06 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:22:08.793 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12700.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:22:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12238.402338 secs 2023-03-09T18:22:08.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:08 smithi122 ceph-mon[93869]: pgmap v7039: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 221 KiB/s wr, 26 op/s 2023-03-09T18:22:08.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:08 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:22:08.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12238.402338 secs 2023-03-09T18:22:08.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:08 smithi181 ceph-mon[94189]: pgmap v7039: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 221 KiB/s wr, 26 op/s 2023-03-09T18:22:08.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:08 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:22:08.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12238.402338 secs 2023-03-09T18:22:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:08 smithi038 ceph-mon[92422]: pgmap v7039: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 221 KiB/s wr, 26 op/s 2023-03-09T18:22:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:08 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:22:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:10 smithi122 ceph-mon[93869]: pgmap v7040: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 204 KiB/s wr, 22 op/s 2023-03-09T18:22:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12241.105900 secs 2023-03-09T18:22:10.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:10 smithi122 ceph-mon[93869]: osdmap e653: 12 total, 12 up, 12 in 2023-03-09T18:22:10.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:22:10.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:22:10.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:22:10.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:22:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:10 smithi181 ceph-mon[94189]: pgmap v7040: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 204 KiB/s wr, 22 op/s 2023-03-09T18:22:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12241.105900 secs 2023-03-09T18:22:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:10 smithi181 ceph-mon[94189]: osdmap e653: 12 total, 12 up, 12 in 2023-03-09T18:22:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:22:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:22:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:22:10.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:22:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:10 smithi038 ceph-mon[92422]: pgmap v7040: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 204 KiB/s wr, 22 op/s 2023-03-09T18:22:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12241.105900 secs 2023-03-09T18:22:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:10 smithi038 ceph-mon[92422]: osdmap e653: 12 total, 12 up, 12 in 2023-03-09T18:22:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:22:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:22:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:22:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:22:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:11 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:22:11.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:11 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:22:11.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:11 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:22:11.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:11 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:22:11.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:11 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:22:11.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:11 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:22:11.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:11 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:22:11.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:11 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:22:11.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:11 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:22:11.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:11 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:22:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:11 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:22:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:11 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:22:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:11 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:22:11.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:11 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:22:11.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:11 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:22:11.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:11 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:22:11.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:11 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:22:11.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:11 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:22:11.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:11 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:22:11.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:11 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:22:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:11 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:22:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:11 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:22:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:11 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:22:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:11 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:22:12.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:11 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:22:12.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:11 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:22:12.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:11 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:22:12.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:11 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:22:12.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:11 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:22:12.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:11 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:22:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:12 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T18:22:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:12 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T18:22:12.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:12 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi038 to 197.2M 2023-03-09T18:22:12.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:12 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi038 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-03-09T18:22:12.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:12 smithi122 ceph-mon[93869]: pgmap v7042: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 97 KiB/s wr, 25 op/s 2023-03-09T18:22:12.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:12 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:22:12.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:12 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:22:12.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:12 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:22:12.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:12 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:22:12.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:12 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:22:12.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:12 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:22:12.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:12 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:22:12.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:12 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T18:22:12.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:12 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T18:22:12.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:12 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi038 to 197.2M 2023-03-09T18:22:12.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:12 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi038 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-03-09T18:22:12.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:12 smithi181 ceph-mon[94189]: pgmap v7042: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 97 KiB/s wr, 25 op/s 2023-03-09T18:22:12.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:12 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:22:12.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:12 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:22:12.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:12 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:22:12.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:12 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:22:12.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:12 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:22:12.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:12 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:22:12.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:12 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:22:12.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:12 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T18:22:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:12 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T18:22:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:12 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi038 to 197.2M 2023-03-09T18:22:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:12 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi038 to 206877081: error parsing value: Value '206877081' is below minimum 939524096 2023-03-09T18:22:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:12 smithi038 ceph-mon[92422]: pgmap v7042: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 97 KiB/s wr, 25 op/s 2023-03-09T18:22:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:12 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:22:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:12 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:22:13.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:12 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:22:13.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:12 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:22:13.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:12 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:22:13.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:12 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:22:13.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:12 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:22:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12705.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:22:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:13 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T18:22:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:13 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T18:22:13.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12243.402460 secs 2023-03-09T18:22:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:13 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T18:22:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:13 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T18:22:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12243.402460 secs 2023-03-09T18:22:13.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:13 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T18:22:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:13 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T18:22:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12243.402460 secs 2023-03-09T18:22:14.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:14 smithi181 ceph-mon[94189]: pgmap v7043: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 71 KiB/s wr, 11 op/s 2023-03-09T18:22:14.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:14 smithi038 ceph-mon[92422]: pgmap v7043: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 71 KiB/s wr, 11 op/s 2023-03-09T18:22:15.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:14 smithi122 ceph-mon[93869]: pgmap v7043: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 71 KiB/s wr, 11 op/s 2023-03-09T18:22:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12246.105983 secs 2023-03-09T18:22:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12246.105983 secs 2023-03-09T18:22:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12246.105983 secs 2023-03-09T18:22:16.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:16 smithi038 ceph-mon[92422]: pgmap v7044: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 71 KiB/s wr, 11 op/s 2023-03-09T18:22:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:16 smithi038 ceph-mon[92422]: osdmap e654: 12 total, 12 up, 12 in 2023-03-09T18:22:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:16 smithi122 ceph-mon[93869]: pgmap v7044: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 71 KiB/s wr, 11 op/s 2023-03-09T18:22:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:16 smithi122 ceph-mon[93869]: osdmap e654: 12 total, 12 up, 12 in 2023-03-09T18:22:17.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:16 smithi181 ceph-mon[94189]: pgmap v7044: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 71 KiB/s wr, 11 op/s 2023-03-09T18:22:17.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:16 smithi181 ceph-mon[94189]: osdmap e654: 12 total, 12 up, 12 in 2023-03-09T18:22:17.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:17 smithi038 ceph-mon[92422]: pgmap v7046: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 16 KiB/s wr, 9 op/s 2023-03-09T18:22:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:17 smithi122 ceph-mon[93869]: pgmap v7046: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 16 KiB/s wr, 9 op/s 2023-03-09T18:22:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:17 smithi181 ceph-mon[94189]: pgmap v7046: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 16 KiB/s wr, 9 op/s 2023-03-09T18:22:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12710.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:22:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12248.402579 secs 2023-03-09T18:22:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12248.402579 secs 2023-03-09T18:22:19.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12248.402579 secs 2023-03-09T18:22:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:19 smithi122 ceph-mon[93869]: pgmap v7047: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 7 op/s 2023-03-09T18:22:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:19 smithi181 ceph-mon[94189]: pgmap v7047: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 7 op/s 2023-03-09T18:22:20.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:19 smithi038 ceph-mon[92422]: pgmap v7047: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 7 op/s 2023-03-09T18:22:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12251.106041 secs 2023-03-09T18:22:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12251.106041 secs 2023-03-09T18:22:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12251.106041 secs 2023-03-09T18:22:22.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:22 smithi122 ceph-mon[93869]: pgmap v7048: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 7 op/s 2023-03-09T18:22:22.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:22 smithi181 ceph-mon[94189]: pgmap v7048: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 7 op/s 2023-03-09T18:22:22.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:22 smithi038 ceph-mon[92422]: pgmap v7048: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 12 KiB/s wr, 7 op/s 2023-03-09T18:22:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12715.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:22:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12253.402694 secs 2023-03-09T18:22:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12253.402694 secs 2023-03-09T18:22:23.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12253.402694 secs 2023-03-09T18:22:24.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:24 smithi122 ceph-mon[93869]: pgmap v7049: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 7 op/s 2023-03-09T18:22:24.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:24 smithi181 ceph-mon[94189]: pgmap v7049: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 7 op/s 2023-03-09T18:22:24.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:24 smithi038 ceph-mon[92422]: pgmap v7049: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 7 op/s 2023-03-09T18:22:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12256.106109 secs 2023-03-09T18:22:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12256.106109 secs 2023-03-09T18:22:25.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12256.106109 secs 2023-03-09T18:22:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:26 smithi122 ceph-mon[93869]: pgmap v7050: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 8 op/s 2023-03-09T18:22:26.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:26 smithi181 ceph-mon[94189]: pgmap v7050: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 8 op/s 2023-03-09T18:22:26.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:26 smithi038 ceph-mon[92422]: pgmap v7050: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 8 op/s 2023-03-09T18:22:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12720.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:22:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12258.402843 secs 2023-03-09T18:22:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:28 smithi122 ceph-mon[93869]: pgmap v7051: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 7 op/s 2023-03-09T18:22:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12258.402843 secs 2023-03-09T18:22:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:28 smithi181 ceph-mon[94189]: pgmap v7051: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 7 op/s 2023-03-09T18:22:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12258.402843 secs 2023-03-09T18:22:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:28 smithi038 ceph-mon[92422]: pgmap v7051: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 7 op/s 2023-03-09T18:22:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:30 smithi122 ceph-mon[93869]: pgmap v7052: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 6 op/s 2023-03-09T18:22:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12261.106168 secs 2023-03-09T18:22:30.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:30 smithi181 ceph-mon[94189]: pgmap v7052: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 6 op/s 2023-03-09T18:22:30.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12261.106168 secs 2023-03-09T18:22:30.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:30 smithi038 ceph-mon[92422]: pgmap v7052: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 6 op/s 2023-03-09T18:22:30.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12261.106168 secs 2023-03-09T18:22:32.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:32 smithi122 ceph-mon[93869]: pgmap v7053: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 49 KiB/s wr, 9 op/s 2023-03-09T18:22:32.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:32 smithi181 ceph-mon[94189]: pgmap v7053: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 49 KiB/s wr, 9 op/s 2023-03-09T18:22:32.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:32 smithi038 ceph-mon[92422]: pgmap v7053: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 49 KiB/s wr, 9 op/s 2023-03-09T18:22:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12725.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:22:34.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:34 smithi122 ceph-mon[93869]: pgmap v7054: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 39 KiB/s wr, 6 op/s 2023-03-09T18:22:34.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:34 smithi181 ceph-mon[94189]: pgmap v7054: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 39 KiB/s wr, 6 op/s 2023-03-09T18:22:34.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:34 smithi038 ceph-mon[92422]: pgmap v7054: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 39 KiB/s wr, 6 op/s 2023-03-09T18:22:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12266.106228 secs 2023-03-09T18:22:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:35 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:22:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12266.106228 secs 2023-03-09T18:22:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:35 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:22:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12266.106228 secs 2023-03-09T18:22:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:35 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:22:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:36 smithi122 ceph-mon[93869]: pgmap v7055: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 38 KiB/s wr, 6 op/s 2023-03-09T18:22:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:36 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:22:36.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:22:36.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:22:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:36 smithi181 ceph-mon[94189]: pgmap v7055: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 38 KiB/s wr, 6 op/s 2023-03-09T18:22:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:36 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:22:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:22:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:22:36.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:36 smithi038 ceph-mon[92422]: pgmap v7055: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 38 KiB/s wr, 6 op/s 2023-03-09T18:22:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:36 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:22:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:22:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:22:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12730.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:22:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:38 smithi122 ceph-mon[93869]: pgmap v7056: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 76 KiB/s wr, 9 op/s 2023-03-09T18:22:38.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:38 smithi181 ceph-mon[94189]: pgmap v7056: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 76 KiB/s wr, 9 op/s 2023-03-09T18:22:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:38 smithi038 ceph-mon[92422]: pgmap v7056: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 76 KiB/s wr, 9 op/s 2023-03-09T18:22:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:40 smithi122 ceph-mon[93869]: pgmap v7057: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 75 KiB/s wr, 6 op/s 2023-03-09T18:22:40.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12271.106290 secs 2023-03-09T18:22:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:40 smithi181 ceph-mon[94189]: pgmap v7057: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 75 KiB/s wr, 6 op/s 2023-03-09T18:22:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12271.106290 secs 2023-03-09T18:22:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:40 smithi038 ceph-mon[92422]: pgmap v7057: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 75 KiB/s wr, 6 op/s 2023-03-09T18:22:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12271.106290 secs 2023-03-09T18:22:42.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:42 smithi122 ceph-mon[93869]: pgmap v7058: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 180 KiB/s wr, 10 op/s 2023-03-09T18:22:42.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:42 smithi181 ceph-mon[94189]: pgmap v7058: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 180 KiB/s wr, 10 op/s 2023-03-09T18:22:42.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:42 smithi038 ceph-mon[92422]: pgmap v7058: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 180 KiB/s wr, 10 op/s 2023-03-09T18:22:43.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12735.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:22:44.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:44 smithi122 ceph-mon[93869]: pgmap v7059: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 171 KiB/s wr, 8 op/s 2023-03-09T18:22:44.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:44 smithi181 ceph-mon[94189]: pgmap v7059: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 171 KiB/s wr, 8 op/s 2023-03-09T18:22:44.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:44 smithi038 ceph-mon[92422]: pgmap v7059: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 171 KiB/s wr, 8 op/s 2023-03-09T18:22:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12276.106373 secs 2023-03-09T18:22:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12276.106373 secs 2023-03-09T18:22:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12276.106373 secs 2023-03-09T18:22:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:46 smithi038 ceph-mon[92422]: pgmap v7060: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 171 KiB/s wr, 8 op/s 2023-03-09T18:22:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:46 smithi122 ceph-mon[93869]: pgmap v7060: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 171 KiB/s wr, 8 op/s 2023-03-09T18:22:47.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:46 smithi181 ceph-mon[94189]: pgmap v7060: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 171 KiB/s wr, 8 op/s 2023-03-09T18:22:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12740.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:22:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:48 smithi038 ceph-mon[92422]: pgmap v7061: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 247 KiB/s wr, 12 op/s 2023-03-09T18:22:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:48 smithi122 ceph-mon[93869]: pgmap v7061: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 247 KiB/s wr, 12 op/s 2023-03-09T18:22:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:48 smithi181 ceph-mon[94189]: pgmap v7061: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 247 KiB/s wr, 12 op/s 2023-03-09T18:22:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:49 smithi122 ceph-mon[93869]: pgmap v7062: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 209 KiB/s wr, 9 op/s 2023-03-09T18:22:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:49 smithi181 ceph-mon[94189]: pgmap v7062: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 209 KiB/s wr, 9 op/s 2023-03-09T18:22:50.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:49 smithi038 ceph-mon[92422]: pgmap v7062: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 209 KiB/s wr, 9 op/s 2023-03-09T18:22:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12281.106430 secs 2023-03-09T18:22:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12281.106430 secs 2023-03-09T18:22:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12281.106430 secs 2023-03-09T18:22:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:52 smithi122 ceph-mon[93869]: pgmap v7063: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 220 KiB/s wr, 11 op/s 2023-03-09T18:22:52.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:52 smithi181 ceph-mon[94189]: pgmap v7063: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 220 KiB/s wr, 11 op/s 2023-03-09T18:22:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:52 smithi038 ceph-mon[92422]: pgmap v7063: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 220 KiB/s wr, 11 op/s 2023-03-09T18:22:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12745.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:22:54.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:54 smithi122 ceph-mon[93869]: pgmap v7064: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 236 KiB/s wr, 10 op/s 2023-03-09T18:22:54.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:54 smithi181 ceph-mon[94189]: pgmap v7064: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 236 KiB/s wr, 10 op/s 2023-03-09T18:22:54.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:54 smithi038 ceph-mon[92422]: pgmap v7064: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 236 KiB/s wr, 10 op/s 2023-03-09T18:22:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12286.106511 secs 2023-03-09T18:22:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12286.106511 secs 2023-03-09T18:22:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12286.106511 secs 2023-03-09T18:22:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:56 smithi122 ceph-mon[93869]: pgmap v7065: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 207 KiB/s wr, 8 op/s 2023-03-09T18:22:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:56 smithi181 ceph-mon[94189]: pgmap v7065: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 207 KiB/s wr, 8 op/s 2023-03-09T18:22:56.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:56 smithi038 ceph-mon[92422]: pgmap v7065: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 207 KiB/s wr, 8 op/s 2023-03-09T18:22:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12750.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:22:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12288.403593 secs 2023-03-09T18:22:58.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:58 smithi122 ceph-mon[93869]: pgmap v7066: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 288 KiB/s wr, 12 op/s 2023-03-09T18:22:58.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12288.403593 secs 2023-03-09T18:22:58.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:58 smithi181 ceph-mon[94189]: pgmap v7066: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 288 KiB/s wr, 12 op/s 2023-03-09T18:22:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12288.403593 secs 2023-03-09T18:22:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:58 smithi038 ceph-mon[92422]: pgmap v7066: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 288 KiB/s wr, 12 op/s 2023-03-09T18:22:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:59 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:22:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:22:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:22:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:23:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:59 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:23:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:23:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:22:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:23:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:59 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:23:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:23:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:22:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:23:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:23:00 smithi038 ceph-mon[92422]: pgmap v7067: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 212 KiB/s wr, 8 op/s 2023-03-09T18:23:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:23:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12291.106573 secs 2023-03-09T18:23:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:23:00 smithi122 ceph-mon[93869]: pgmap v7067: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 212 KiB/s wr, 8 op/s 2023-03-09T18:23:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:23:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12291.106573 secs 2023-03-09T18:23:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:23:00 smithi181 ceph-mon[94189]: pgmap v7067: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 212 KiB/s wr, 8 op/s 2023-03-09T18:23:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:23:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12291.106573 secs 2023-03-09T18:23:02.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:23:02 smithi038 ceph-mon[92422]: pgmap v7068: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 235 KiB/s wr, 13 op/s 2023-03-09T18:23:03.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:23:02 smithi122 ceph-mon[93869]: pgmap v7068: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 235 KiB/s wr, 13 op/s 2023-03-09T18:23:03.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:23:02 smithi181 ceph-mon[94189]: pgmap v7068: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 235 KiB/s wr, 13 op/s 2023-03-09T18:23:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12755.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:23:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:23:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12293.403697 secs 2023-03-09T18:23:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:23:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12293.403697 secs 2023-03-09T18:23:04.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:23:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12293.403697 secs 2023-03-09T18:23:05.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:23:04 smithi122 ceph-mon[93869]: pgmap v7069: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 267 KiB/s wr, 24 op/s 2023-03-09T18:23:05.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:23:04 smithi181 ceph-mon[94189]: pgmap v7069: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 267 KiB/s wr, 24 op/s 2023-03-09T18:23:05.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:23:04 smithi038 ceph-mon[92422]: pgmap v7069: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 267 KiB/s wr, 24 op/s 2023-03-09T18:23:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:23:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12296.106633 secs 2023-03-09T18:23:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:23:05 smithi122 ceph-mon[93869]: osdmap e655: 12 total, 12 up, 12 in 2023-03-09T18:23:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:23:05 smithi122 ceph-mon[93869]: pgmap v7071: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 176 KiB/s wr, 26 op/s 2023-03-09T18:23:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:23:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12296.106633 secs 2023-03-09T18:23:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:23:05 smithi181 ceph-mon[94189]: osdmap e655: 12 total, 12 up, 12 in 2023-03-09T18:23:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:23:05 smithi181 ceph-mon[94189]: pgmap v7071: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 176 KiB/s wr, 26 op/s 2023-03-09T18:23:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:23:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12296.106633 secs 2023-03-09T18:23:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:23:05 smithi038 ceph-mon[92422]: osdmap e655: 12 total, 12 up, 12 in 2023-03-09T18:23:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:23:05 smithi038 ceph-mon[92422]: pgmap v7071: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 176 KiB/s wr, 26 op/s 2023-03-09T18:23:08.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12760.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:23:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:23:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12298.403817 secs 2023-03-09T18:23:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:23:08 smithi122 ceph-mon[93869]: pgmap v7072: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 190 KiB/s wr, 26 op/s 2023-03-09T18:23:08.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:23:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12298.403817 secs 2023-03-09T18:23:08.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:23:08 smithi181 ceph-mon[94189]: pgmap v7072: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 190 KiB/s wr, 26 op/s 2023-03-09T18:23:08.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:23:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12298.403817 secs 2023-03-09T18:23:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:23:08 smithi038 ceph-mon[92422]: pgmap v7072: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 190 KiB/s wr, 26 op/s 2023-03-09T18:23:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:23:10 smithi122 ceph-mon[93869]: pgmap v7073: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 190 KiB/s wr, 26 op/s 2023-03-09T18:23:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:23:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12301.106779 secs 2023-03-09T18:23:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:23:10 smithi181 ceph-mon[94189]: pgmap v7073: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 190 KiB/s wr, 26 op/s 2023-03-09T18:23:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:23:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12301.106779 secs 2023-03-09T18:23:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:23:10 smithi038 ceph-mon[92422]: pgmap v7073: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 190 KiB/s wr, 26 op/s 2023-03-09T18:23:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:23:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12301.106779 secs 2023-03-09T18:23:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:23:12 smithi122 ceph-mon[93869]: pgmap v7074: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 175 KiB/s wr, 24 op/s 2023-03-09T18:23:12.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:23:12 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:23:12.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:23:12 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:23:12.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:23:12 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:23:12.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:23:12 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:23:12.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:23:12 smithi181 ceph-mon[94189]: pgmap v7074: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 175 KiB/s wr, 24 op/s 2023-03-09T18:23:12.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:23:12 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:23:12.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:23:12 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:23:12.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:23:12 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:23:12.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:23:12 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:23:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:23:12 smithi038 ceph-mon[92422]: pgmap v7074: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 175 KiB/s wr, 24 op/s 2023-03-09T18:23:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:23:12 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:23:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:23:12 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:23:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:23:12 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:23:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:23:12 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:23:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12765.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:23:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:23:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12303.403890 secs 2023-03-09T18:23:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:23:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12303.403890 secs 2023-03-09T18:23:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:23:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12303.403890 secs 2023-03-09T18:23:14.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:23:14 smithi122 ceph-mon[93869]: pgmap v7075: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 124 KiB/s wr, 9 op/s 2023-03-09T18:23:14.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:23:14 smithi181 ceph-mon[94189]: pgmap v7075: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 124 KiB/s wr, 9 op/s 2023-03-09T18:23:14.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:23:14 smithi038 ceph-mon[92422]: pgmap v7075: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 124 KiB/s wr, 9 op/s 2023-03-09T18:23:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:23:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12306.106864 secs 2023-03-09T18:23:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:23:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12306.106864 secs 2023-03-09T18:23:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:23:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12306.106864 secs 2023-03-09T18:23:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:23:16 smithi122 ceph-mon[93869]: pgmap v7076: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 115 KiB/s wr, 9 op/s 2023-03-09T18:23:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:23:16 smithi122 ceph-mon[93869]: osdmap e656: 12 total, 12 up, 12 in 2023-03-09T18:23:16.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:23:16 smithi181 ceph-mon[94189]: pgmap v7076: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 115 KiB/s wr, 9 op/s 2023-03-09T18:23:16.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:23:16 smithi181 ceph-mon[94189]: osdmap e656: 12 total, 12 up, 12 in 2023-03-09T18:23:16.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:23:16 smithi038 ceph-mon[92422]: pgmap v7076: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 115 KiB/s wr, 9 op/s 2023-03-09T18:23:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:23:16 smithi038 ceph-mon[92422]: osdmap e656: 12 total, 12 up, 12 in 2023-03-09T18:23:18.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12770.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:23:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:23:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12308.404019 secs 2023-03-09T18:23:18.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:23:18 smithi122 ceph-mon[93869]: pgmap v7078: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 26 KiB/s wr, 8 op/s 2023-03-09T18:23:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:23:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12308.404019 secs 2023-03-09T18:23:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:23:18 smithi181 ceph-mon[94189]: pgmap v7078: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 26 KiB/s wr, 8 op/s 2023-03-09T18:23:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:23:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12308.404019 secs 2023-03-09T18:23:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:23:18 smithi038 ceph-mon[92422]: pgmap v7078: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 26 KiB/s wr, 8 op/s 2023-03-09T18:23:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:23:20 smithi122 ceph-mon[93869]: pgmap v7079: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 26 KiB/s wr, 8 op/s 2023-03-09T18:23:20.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:23:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12311.106924 secs 2023-03-09T18:23:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:23:20 smithi181 ceph-mon[94189]: pgmap v7079: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 26 KiB/s wr, 8 op/s 2023-03-09T18:23:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:23:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12311.106924 secs 2023-03-09T18:23:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:23:20 smithi038 ceph-mon[92422]: pgmap v7079: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 26 KiB/s wr, 8 op/s 2023-03-09T18:23:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:23:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12311.106924 secs 2023-03-09T18:23:22.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:23:22 smithi122 ceph-mon[93869]: pgmap v7080: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 8 op/s 2023-03-09T18:23:22.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:23:22 smithi181 ceph-mon[94189]: pgmap v7080: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 8 op/s 2023-03-09T18:23:22.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:23:22 smithi038 ceph-mon[92422]: pgmap v7080: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 8 op/s 2023-03-09T18:23:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12775.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:23:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:23:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12313.404152 secs 2023-03-09T18:23:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:23:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12313.404152 secs 2023-03-09T18:23:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:23:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12313.404152 secs 2023-03-09T18:23:24.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:23:24 smithi122 ceph-mon[93869]: pgmap v7081: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 8 op/s 2023-03-09T18:23:24.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:23:24 smithi181 ceph-mon[94189]: pgmap v7081: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 8 op/s 2023-03-09T18:23:24.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:23:24 smithi038 ceph-mon[92422]: pgmap v7081: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 8 op/s 2023-03-09T18:23:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:23:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12316.106984 secs 2023-03-09T18:23:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:23:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12316.106984 secs 2023-03-09T18:23:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:23:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12316.106984 secs 2023-03-09T18:23:26.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:23:26 smithi038 ceph-mon[92422]: pgmap v7082: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 8 op/s 2023-03-09T18:23:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:23:26 smithi122 ceph-mon[93869]: pgmap v7082: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 8 op/s 2023-03-09T18:23:27.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:23:26 smithi181 ceph-mon[94189]: pgmap v7082: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 8 op/s 2023-03-09T18:23:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12780.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:23:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:23:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12318.404285 secs 2023-03-09T18:23:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:23:28 smithi038 ceph-mon[92422]: pgmap v7083: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 43 KiB/s wr, 8 op/s 2023-03-09T18:23:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:23:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12318.404285 secs 2023-03-09T18:23:29.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:23:28 smithi122 ceph-mon[93869]: pgmap v7083: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 43 KiB/s wr, 8 op/s 2023-03-09T18:23:29.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:23:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12318.404285 secs 2023-03-09T18:23:29.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:23:28 smithi181 ceph-mon[94189]: pgmap v7083: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 43 KiB/s wr, 8 op/s 2023-03-09T18:23:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:23:30 smithi122 ceph-mon[93869]: pgmap v7084: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 39 KiB/s wr, 7 op/s 2023-03-09T18:23:31.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:23:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12321.107045 secs 2023-03-09T18:23:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:23:30 smithi181 ceph-mon[94189]: pgmap v7084: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 39 KiB/s wr, 7 op/s 2023-03-09T18:23:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:23:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12321.107045 secs 2023-03-09T18:23:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:23:30 smithi038 ceph-mon[92422]: pgmap v7084: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 39 KiB/s wr, 7 op/s 2023-03-09T18:23:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:23:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12321.107045 secs 2023-03-09T18:23:32.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:23:32 smithi122 ceph-mon[93869]: pgmap v7085: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 73 KiB/s wr, 10 op/s 2023-03-09T18:23:32.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:23:32 smithi181 ceph-mon[94189]: pgmap v7085: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 73 KiB/s wr, 10 op/s 2023-03-09T18:23:32.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:23:32 smithi038 ceph-mon[92422]: pgmap v7085: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 73 KiB/s wr, 10 op/s 2023-03-09T18:23:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12785.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:23:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:23:33 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 12323.404396 secs 2023-03-09T18:23:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:23:33 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 12323.404396 secs 2023-03-09T18:23:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:23:33 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 12323.404396 secs 2023-03-09T18:23:34.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:23:34 smithi122 ceph-mon[93869]: pgmap v7086: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 73 KiB/s wr, 7 op/s 2023-03-09T18:23:34.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:23:34 smithi181 ceph-mon[94189]: pgmap v7086: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 73 KiB/s wr, 7 op/s 2023-03-09T18:23:34.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:23:34 smithi038 ceph-mon[92422]: pgmap v7086: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 73 KiB/s wr, 7 op/s 2023-03-09T18:23:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:23:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12326.107106 secs 2023-03-09T18:23:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:23:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12326.107106 secs 2023-03-09T18:23:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:23:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12326.107106 secs 2023-03-09T18:23:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:23:36 smithi122 ceph-mon[93869]: pgmap v7087: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 73 KiB/s wr, 7 op/s 2023-03-09T18:23:36.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:23:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:23:36.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:23:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:23:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:23:36 smithi181 ceph-mon[94189]: pgmap v7087: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 73 KiB/s wr, 7 op/s 2023-03-09T18:23:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:23:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:23:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:23:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:23:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:23:36 smithi038 ceph-mon[92422]: pgmap v7087: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 73 KiB/s wr, 7 op/s 2023-03-09T18:23:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:23:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:23:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:23:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:23:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12790.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:23:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:23:38 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 12328.404512 secs 2023-03-09T18:23:38.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:23:38 smithi122 ceph-mon[93869]: pgmap v7088: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 111 KiB/s wr, 10 op/s 2023-03-09T18:23:38.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:23:38 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 12328.404512 secs 2023-03-09T18:23:38.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:23:38 smithi181 ceph-mon[94189]: pgmap v7088: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 111 KiB/s wr, 10 op/s 2023-03-09T18:23:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:23:38 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 12328.404512 secs 2023-03-09T18:23:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:23:38 smithi038 ceph-mon[92422]: pgmap v7088: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 111 KiB/s wr, 10 op/s 2023-03-09T18:23:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:23:40 smithi122 ceph-mon[93869]: pgmap v7089: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 73 KiB/s wr, 7 op/s 2023-03-09T18:23:40.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:23:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12331.107166 secs 2023-03-09T18:23:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:23:40 smithi181 ceph-mon[94189]: pgmap v7089: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 73 KiB/s wr, 7 op/s 2023-03-09T18:23:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:23:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12331.107166 secs 2023-03-09T18:23:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:23:40 smithi038 ceph-mon[92422]: pgmap v7089: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 73 KiB/s wr, 7 op/s 2023-03-09T18:23:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:23:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12331.107166 secs 2023-03-09T18:23:42.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:23:42 smithi122 ceph-mon[93869]: pgmap v7090: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 208 KiB/s wr, 10 op/s 2023-03-09T18:23:42.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:23:42 smithi181 ceph-mon[94189]: pgmap v7090: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 208 KiB/s wr, 10 op/s 2023-03-09T18:23:42.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:23:42 smithi038 ceph-mon[92422]: pgmap v7090: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 208 KiB/s wr, 10 op/s 2023-03-09T18:23:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12795.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:23:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:23:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12333.404628 secs 2023-03-09T18:23:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:23:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12333.404628 secs 2023-03-09T18:23:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:23:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12333.404628 secs 2023-03-09T18:23:44.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:23:44 smithi122 ceph-mon[93869]: pgmap v7091: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 173 KiB/s wr, 8 op/s 2023-03-09T18:23:44.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:23:44 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:23:44.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:23:44 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:23:44.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:23:44 smithi181 ceph-mon[94189]: pgmap v7091: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 173 KiB/s wr, 8 op/s 2023-03-09T18:23:44.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:23:44 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:23:44.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:23:44 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:23:44.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:23:44 smithi038 ceph-mon[92422]: pgmap v7091: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 173 KiB/s wr, 8 op/s 2023-03-09T18:23:44.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:23:44 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:23:44.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:23:44 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:23:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:23:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12336.107228 secs 2023-03-09T18:23:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:23:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12336.107228 secs 2023-03-09T18:23:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:23:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12336.107228 secs 2023-03-09T18:23:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:23:46 smithi122 ceph-mon[93869]: pgmap v7092: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 173 KiB/s wr, 8 op/s 2023-03-09T18:23:46.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:23:46 smithi181 ceph-mon[94189]: pgmap v7092: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 173 KiB/s wr, 8 op/s 2023-03-09T18:23:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:23:46 smithi038 ceph-mon[92422]: pgmap v7092: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 173 KiB/s wr, 8 op/s 2023-03-09T18:23:48.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12800.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:23:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:23:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12338.404739 secs 2023-03-09T18:23:48.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:23:48 smithi122 ceph-mon[93869]: pgmap v7093: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 249 KiB/s wr, 12 op/s 2023-03-09T18:23:48.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:23:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12338.404739 secs 2023-03-09T18:23:48.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:23:48 smithi181 ceph-mon[94189]: pgmap v7093: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 249 KiB/s wr, 12 op/s 2023-03-09T18:23:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:23:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12338.404739 secs 2023-03-09T18:23:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:23:48 smithi038 ceph-mon[92422]: pgmap v7093: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 249 KiB/s wr, 12 op/s 2023-03-09T18:23:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:23:50 smithi038 ceph-mon[92422]: pgmap v7094: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 211 KiB/s wr, 9 op/s 2023-03-09T18:23:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:23:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12341.107287 secs 2023-03-09T18:23:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:23:50 smithi122 ceph-mon[93869]: pgmap v7094: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 211 KiB/s wr, 9 op/s 2023-03-09T18:23:51.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:23:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12341.107287 secs 2023-03-09T18:23:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:23:50 smithi181 ceph-mon[94189]: pgmap v7094: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 211 KiB/s wr, 9 op/s 2023-03-09T18:23:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:23:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12341.107287 secs 2023-03-09T18:23:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:23:52 smithi038 ceph-mon[92422]: pgmap v7095: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 318 KiB/s wr, 14 op/s 2023-03-09T18:23:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:23:52 smithi122 ceph-mon[93869]: pgmap v7095: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 318 KiB/s wr, 14 op/s 2023-03-09T18:23:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:23:52 smithi181 ceph-mon[94189]: pgmap v7095: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 318 KiB/s wr, 14 op/s 2023-03-09T18:23:53.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12805.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:23:53.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:23:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12343.404830 secs 2023-03-09T18:23:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:23:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12343.404830 secs 2023-03-09T18:23:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:23:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12343.404830 secs 2023-03-09T18:23:54.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:23:54 smithi038 ceph-mon[92422]: pgmap v7096: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 184 KiB/s wr, 10 op/s 2023-03-09T18:23:55.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:23:54 smithi122 ceph-mon[93869]: pgmap v7096: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 184 KiB/s wr, 10 op/s 2023-03-09T18:23:55.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:23:54 smithi181 ceph-mon[94189]: pgmap v7096: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 184 KiB/s wr, 10 op/s 2023-03-09T18:23:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:23:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12346.107348 secs 2023-03-09T18:23:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:23:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12346.107348 secs 2023-03-09T18:23:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:23:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12346.107348 secs 2023-03-09T18:23:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:23:56 smithi122 ceph-mon[93869]: pgmap v7097: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 184 KiB/s wr, 9 op/s 2023-03-09T18:23:57.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:23:56 smithi181 ceph-mon[94189]: pgmap v7097: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 184 KiB/s wr, 9 op/s 2023-03-09T18:23:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:23:56 smithi038 ceph-mon[92422]: pgmap v7097: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 184 KiB/s wr, 9 op/s 2023-03-09T18:23:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:23:57 smithi122 ceph-mon[93869]: pgmap v7098: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 212 KiB/s wr, 13 op/s 2023-03-09T18:23:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:23:57 smithi181 ceph-mon[94189]: pgmap v7098: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 212 KiB/s wr, 13 op/s 2023-03-09T18:23:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:23:57 smithi038 ceph-mon[92422]: pgmap v7098: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 212 KiB/s wr, 13 op/s 2023-03-09T18:23:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12810.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:23:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:23:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12348.404963 secs 2023-03-09T18:23:59.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:23:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12348.404963 secs 2023-03-09T18:23:59.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:23:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12348.404963 secs 2023-03-09T18:24:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:23:59 smithi122 ceph-mon[93869]: pgmap v7099: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 136 KiB/s wr, 9 op/s 2023-03-09T18:24:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:23:59 smithi181 ceph-mon[94189]: pgmap v7099: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 136 KiB/s wr, 9 op/s 2023-03-09T18:24:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:23:59 smithi038 ceph-mon[92422]: pgmap v7099: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 136 KiB/s wr, 9 op/s 2023-03-09T18:24:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:24:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12351.107411 secs 2023-03-09T18:24:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:24:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12351.107411 secs 2023-03-09T18:24:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:24:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12351.107411 secs 2023-03-09T18:24:02.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:24:02 smithi122 ceph-mon[93869]: pgmap v7100: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 257 KiB/s wr, 15 op/s 2023-03-09T18:24:02.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:24:02 smithi181 ceph-mon[94189]: pgmap v7100: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 257 KiB/s wr, 15 op/s 2023-03-09T18:24:02.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:24:02 smithi038 ceph-mon[92422]: pgmap v7100: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 257 KiB/s wr, 15 op/s 2023-03-09T18:24:03.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12815.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:24:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:24:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12353.405083 secs 2023-03-09T18:24:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:24:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12353.405083 secs 2023-03-09T18:24:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:24:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12353.405083 secs 2023-03-09T18:24:04.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:24:04 smithi122 ceph-mon[93869]: pgmap v7101: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 171 KiB/s wr, 23 op/s 2023-03-09T18:24:04.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:24:04 smithi181 ceph-mon[94189]: pgmap v7101: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 171 KiB/s wr, 23 op/s 2023-03-09T18:24:04.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:24:04 smithi038 ceph-mon[92422]: pgmap v7101: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 171 KiB/s wr, 23 op/s 2023-03-09T18:24:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:24:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12356.107470 secs 2023-03-09T18:24:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:24:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12356.107470 secs 2023-03-09T18:24:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:24:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12356.107470 secs 2023-03-09T18:24:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:24:06 smithi122 ceph-mon[93869]: pgmap v7102: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 171 KiB/s wr, 22 op/s 2023-03-09T18:24:06.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:24:06 smithi181 ceph-mon[94189]: pgmap v7102: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 171 KiB/s wr, 22 op/s 2023-03-09T18:24:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:24:06 smithi038 ceph-mon[92422]: pgmap v7102: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 171 KiB/s wr, 22 op/s 2023-03-09T18:24:08.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12820.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:24:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:24:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12358.405203 secs 2023-03-09T18:24:08.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:24:08 smithi122 ceph-mon[93869]: pgmap v7103: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 196 KiB/s wr, 25 op/s 2023-03-09T18:24:08.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:24:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12358.405203 secs 2023-03-09T18:24:08.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:24:08 smithi181 ceph-mon[94189]: pgmap v7103: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 196 KiB/s wr, 25 op/s 2023-03-09T18:24:08.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:24:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12358.405203 secs 2023-03-09T18:24:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:24:08 smithi038 ceph-mon[92422]: pgmap v7103: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 196 KiB/s wr, 25 op/s 2023-03-09T18:24:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:24:10 smithi181 ceph-mon[94189]: pgmap v7104: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 168 KiB/s wr, 22 op/s 2023-03-09T18:24:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:24:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12361.107531 secs 2023-03-09T18:24:10.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:24:10 smithi181 ceph-mon[94189]: osdmap e657: 12 total, 12 up, 12 in 2023-03-09T18:24:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:24:10 smithi038 ceph-mon[92422]: pgmap v7104: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 168 KiB/s wr, 22 op/s 2023-03-09T18:24:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:24:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12361.107531 secs 2023-03-09T18:24:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:24:10 smithi038 ceph-mon[92422]: osdmap e657: 12 total, 12 up, 12 in 2023-03-09T18:24:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:24:10 smithi122 ceph-mon[93869]: pgmap v7104: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 168 KiB/s wr, 22 op/s 2023-03-09T18:24:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:24:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12361.107531 secs 2023-03-09T18:24:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:24:10 smithi122 ceph-mon[93869]: osdmap e657: 12 total, 12 up, 12 in 2023-03-09T18:24:12.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:24:12 smithi181 ceph-mon[94189]: pgmap v7106: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 69 KiB/s wr, 22 op/s 2023-03-09T18:24:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:24:12 smithi038 ceph-mon[92422]: pgmap v7106: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 69 KiB/s wr, 22 op/s 2023-03-09T18:24:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:24:12 smithi122 ceph-mon[93869]: pgmap v7106: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 69 KiB/s wr, 22 op/s 2023-03-09T18:24:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12825.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:24:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:24:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12363.405305 secs 2023-03-09T18:24:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:24:13 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:24:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:24:13 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:24:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:24:13 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:24:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:24:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12363.405305 secs 2023-03-09T18:24:14.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:24:13 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:24:14.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:24:13 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:24:14.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:24:13 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:24:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:24:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12363.405305 secs 2023-03-09T18:24:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:24:13 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:24:14.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:24:13 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:24:14.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:24:13 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:24:14.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:24:14 smithi038 ceph-mon[92422]: pgmap v7107: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 43 KiB/s wr, 8 op/s 2023-03-09T18:24:14.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:24:14 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:24:14.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:24:14 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:24:14.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:24:14 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:24:15.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:24:14 smithi122 ceph-mon[93869]: pgmap v7107: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 43 KiB/s wr, 8 op/s 2023-03-09T18:24:15.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:24:14 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:24:15.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:24:14 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:24:15.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:24:14 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:24:15.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:24:14 smithi181 ceph-mon[94189]: pgmap v7107: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 43 KiB/s wr, 8 op/s 2023-03-09T18:24:15.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:24:14 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:24:15.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:24:14 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:24:15.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:24:14 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:24:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:24:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12366.107592 secs 2023-03-09T18:24:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:24:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12366.107592 secs 2023-03-09T18:24:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:24:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12366.107592 secs 2023-03-09T18:24:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:24:16 smithi038 ceph-mon[92422]: pgmap v7108: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 8 op/s 2023-03-09T18:24:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:24:16 smithi038 ceph-mon[92422]: osdmap e658: 12 total, 12 up, 12 in 2023-03-09T18:24:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:24:16 smithi122 ceph-mon[93869]: pgmap v7108: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 8 op/s 2023-03-09T18:24:17.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:24:16 smithi122 ceph-mon[93869]: osdmap e658: 12 total, 12 up, 12 in 2023-03-09T18:24:17.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:24:16 smithi181 ceph-mon[94189]: pgmap v7108: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 8 op/s 2023-03-09T18:24:17.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:24:16 smithi181 ceph-mon[94189]: osdmap e658: 12 total, 12 up, 12 in 2023-03-09T18:24:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12830.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:24:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:24:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12368.405409 secs 2023-03-09T18:24:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:24:18 smithi038 ceph-mon[92422]: pgmap v7110: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 32 KiB/s wr, 10 op/s 2023-03-09T18:24:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:24:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12368.405409 secs 2023-03-09T18:24:19.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:24:18 smithi122 ceph-mon[93869]: pgmap v7110: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 32 KiB/s wr, 10 op/s 2023-03-09T18:24:19.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:24:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12368.405409 secs 2023-03-09T18:24:19.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:24:18 smithi181 ceph-mon[94189]: pgmap v7110: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 32 KiB/s wr, 10 op/s 2023-03-09T18:24:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:24:20 smithi122 ceph-mon[93869]: pgmap v7111: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 8 op/s 2023-03-09T18:24:21.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:24:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12371.107671 secs 2023-03-09T18:24:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:24:20 smithi181 ceph-mon[94189]: pgmap v7111: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 8 op/s 2023-03-09T18:24:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:24:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12371.107671 secs 2023-03-09T18:24:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:24:20 smithi038 ceph-mon[92422]: pgmap v7111: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 8 op/s 2023-03-09T18:24:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:24:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12371.107671 secs 2023-03-09T18:24:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:24:22 smithi122 ceph-mon[93869]: pgmap v7112: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T18:24:23.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:24:22 smithi181 ceph-mon[94189]: pgmap v7112: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T18:24:23.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:24:22 smithi038 ceph-mon[92422]: pgmap v7112: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T18:24:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12835.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:24:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:24:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12373.405559 secs 2023-03-09T18:24:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:24:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12373.405559 secs 2023-03-09T18:24:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:24:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12373.405559 secs 2023-03-09T18:24:25.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:24:24 smithi122 ceph-mon[93869]: pgmap v7113: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 8 op/s 2023-03-09T18:24:25.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:24:24 smithi181 ceph-mon[94189]: pgmap v7113: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 8 op/s 2023-03-09T18:24:25.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:24:24 smithi038 ceph-mon[92422]: pgmap v7113: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 8 op/s 2023-03-09T18:24:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:24:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12376.107749 secs 2023-03-09T18:24:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:24:25 smithi122 ceph-mon[93869]: pgmap v7114: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 8 op/s 2023-03-09T18:24:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:24:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12376.107749 secs 2023-03-09T18:24:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:24:25 smithi181 ceph-mon[94189]: pgmap v7114: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 8 op/s 2023-03-09T18:24:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:24:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12376.107749 secs 2023-03-09T18:24:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:24:25 smithi038 ceph-mon[92422]: pgmap v7114: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 8 op/s 2023-03-09T18:24:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12840.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:24:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:24:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12378.405661 secs 2023-03-09T18:24:28.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:24:28 smithi122 ceph-mon[93869]: pgmap v7115: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 42 KiB/s wr, 7 op/s 2023-03-09T18:24:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:24:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12378.405661 secs 2023-03-09T18:24:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:24:28 smithi181 ceph-mon[94189]: pgmap v7115: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 42 KiB/s wr, 7 op/s 2023-03-09T18:24:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:24:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12378.405661 secs 2023-03-09T18:24:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:24:28 smithi038 ceph-mon[92422]: pgmap v7115: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 42 KiB/s wr, 7 op/s 2023-03-09T18:24:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:24:30 smithi122 ceph-mon[93869]: pgmap v7116: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 38 KiB/s wr, 6 op/s 2023-03-09T18:24:30.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:24:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12381.107807 secs 2023-03-09T18:24:30.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:24:30 smithi181 ceph-mon[94189]: pgmap v7116: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 38 KiB/s wr, 6 op/s 2023-03-09T18:24:30.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:24:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12381.107807 secs 2023-03-09T18:24:30.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:24:30 smithi038 ceph-mon[92422]: pgmap v7116: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 38 KiB/s wr, 6 op/s 2023-03-09T18:24:30.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:24:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12381.107807 secs 2023-03-09T18:24:32.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:24:32 smithi122 ceph-mon[93869]: pgmap v7117: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 38 KiB/s wr, 9 op/s 2023-03-09T18:24:32.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:24:32 smithi181 ceph-mon[94189]: pgmap v7117: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 38 KiB/s wr, 9 op/s 2023-03-09T18:24:32.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:24:32 smithi038 ceph-mon[92422]: pgmap v7117: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 38 KiB/s wr, 9 op/s 2023-03-09T18:24:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12845.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:24:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:24:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12383.405779 secs 2023-03-09T18:24:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:24:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12383.405779 secs 2023-03-09T18:24:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:24:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12383.405779 secs 2023-03-09T18:24:34.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:24:34 smithi122 ceph-mon[93869]: pgmap v7118: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 38 KiB/s wr, 6 op/s 2023-03-09T18:24:34.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:24:34 smithi181 ceph-mon[94189]: pgmap v7118: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 38 KiB/s wr, 6 op/s 2023-03-09T18:24:34.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:24:34 smithi038 ceph-mon[92422]: pgmap v7118: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 38 KiB/s wr, 6 op/s 2023-03-09T18:24:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:24:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12386.107865 secs 2023-03-09T18:24:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:24:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12386.107865 secs 2023-03-09T18:24:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:24:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12386.107865 secs 2023-03-09T18:24:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:24:36 smithi038 ceph-mon[92422]: pgmap v7119: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 38 KiB/s wr, 6 op/s 2023-03-09T18:24:37.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:24:36 smithi122 ceph-mon[93869]: pgmap v7119: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 38 KiB/s wr, 6 op/s 2023-03-09T18:24:37.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:24:36 smithi181 ceph-mon[94189]: pgmap v7119: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 38 KiB/s wr, 6 op/s 2023-03-09T18:24:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12850.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:24:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:24:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12388.405876 secs 2023-03-09T18:24:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:24:38 smithi038 ceph-mon[92422]: pgmap v7120: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 114 KiB/s wr, 9 op/s 2023-03-09T18:24:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:24:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12388.405876 secs 2023-03-09T18:24:39.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:24:38 smithi122 ceph-mon[93869]: pgmap v7120: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 114 KiB/s wr, 9 op/s 2023-03-09T18:24:39.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:24:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12388.405876 secs 2023-03-09T18:24:39.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:24:38 smithi181 ceph-mon[94189]: pgmap v7120: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 114 KiB/s wr, 9 op/s 2023-03-09T18:24:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:24:40 smithi038 ceph-mon[92422]: pgmap v7121: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 76 KiB/s wr, 7 op/s 2023-03-09T18:24:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:24:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12391.107949 secs 2023-03-09T18:24:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:24:40 smithi122 ceph-mon[93869]: pgmap v7121: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 76 KiB/s wr, 7 op/s 2023-03-09T18:24:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:24:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12391.107949 secs 2023-03-09T18:24:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:24:40 smithi181 ceph-mon[94189]: pgmap v7121: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 76 KiB/s wr, 7 op/s 2023-03-09T18:24:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:24:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12391.107949 secs 2023-03-09T18:24:42.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:24:42 smithi038 ceph-mon[92422]: pgmap v7122: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 153 KiB/s wr, 11 op/s 2023-03-09T18:24:43.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:24:42 smithi122 ceph-mon[93869]: pgmap v7122: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 153 KiB/s wr, 11 op/s 2023-03-09T18:24:43.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:24:42 smithi181 ceph-mon[94189]: pgmap v7122: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 153 KiB/s wr, 11 op/s 2023-03-09T18:24:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12855.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:24:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:24:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12393.406025 secs 2023-03-09T18:24:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:24:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12393.406025 secs 2023-03-09T18:24:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:24:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12393.406025 secs 2023-03-09T18:24:45.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:24:44 smithi122 ceph-mon[93869]: pgmap v7123: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 153 KiB/s wr, 9 op/s 2023-03-09T18:24:45.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:24:44 smithi181 ceph-mon[94189]: pgmap v7123: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 153 KiB/s wr, 9 op/s 2023-03-09T18:24:45.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:24:44 smithi038 ceph-mon[92422]: pgmap v7123: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 153 KiB/s wr, 9 op/s 2023-03-09T18:24:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:24:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12396.107984 secs 2023-03-09T18:24:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:24:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12396.107984 secs 2023-03-09T18:24:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:24:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12396.107984 secs 2023-03-09T18:24:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:24:46 smithi122 ceph-mon[93869]: pgmap v7124: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 153 KiB/s wr, 9 op/s 2023-03-09T18:24:47.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:24:46 smithi181 ceph-mon[94189]: pgmap v7124: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 153 KiB/s wr, 9 op/s 2023-03-09T18:24:47.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:24:46 smithi038 ceph-mon[92422]: pgmap v7124: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 153 KiB/s wr, 9 op/s 2023-03-09T18:24:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:24:47 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:24:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:24:47 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:24:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:24:47 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:24:48.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12860.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:24:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:24:48 smithi122 ceph-mon[93869]: pgmap v7125: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 241 KiB/s wr, 13 op/s 2023-03-09T18:24:49.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:24:48 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:24:49.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:24:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:24:49.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:24:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:24:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:24:48 smithi181 ceph-mon[94189]: pgmap v7125: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 241 KiB/s wr, 13 op/s 2023-03-09T18:24:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:24:48 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:24:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:24:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:24:49.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:24:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:24:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:24:48 smithi038 ceph-mon[92422]: pgmap v7125: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 241 KiB/s wr, 13 op/s 2023-03-09T18:24:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:24:48 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:24:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:24:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:24:49.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:24:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:24:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:24:49 smithi122 ceph-mon[93869]: pgmap v7126: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 165 KiB/s wr, 10 op/s 2023-03-09T18:24:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:24:49 smithi181 ceph-mon[94189]: pgmap v7126: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 165 KiB/s wr, 10 op/s 2023-03-09T18:24:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:24:49 smithi038 ceph-mon[92422]: pgmap v7126: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 165 KiB/s wr, 10 op/s 2023-03-09T18:24:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:24:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12401.108043 secs 2023-03-09T18:24:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:24:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12401.108043 secs 2023-03-09T18:24:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:24:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12401.108043 secs 2023-03-09T18:24:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:24:52 smithi122 ceph-mon[93869]: pgmap v7127: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 234 KiB/s wr, 13 op/s 2023-03-09T18:24:52.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:24:52 smithi181 ceph-mon[94189]: pgmap v7127: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 234 KiB/s wr, 13 op/s 2023-03-09T18:24:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:24:52 smithi038 ceph-mon[92422]: pgmap v7127: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 234 KiB/s wr, 13 op/s 2023-03-09T18:24:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12865.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:24:54.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:24:54 smithi122 ceph-mon[93869]: pgmap v7128: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 157 KiB/s wr, 10 op/s 2023-03-09T18:24:54.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:24:54 smithi181 ceph-mon[94189]: pgmap v7128: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 157 KiB/s wr, 10 op/s 2023-03-09T18:24:54.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:24:54 smithi038 ceph-mon[92422]: pgmap v7128: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 157 KiB/s wr, 10 op/s 2023-03-09T18:24:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:24:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12406.108101 secs 2023-03-09T18:24:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:24:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12406.108101 secs 2023-03-09T18:24:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:24:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12406.108101 secs 2023-03-09T18:24:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:24:56 smithi122 ceph-mon[93869]: pgmap v7129: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 157 KiB/s wr, 9 op/s 2023-03-09T18:24:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:24:56 smithi181 ceph-mon[94189]: pgmap v7129: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 157 KiB/s wr, 9 op/s 2023-03-09T18:24:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:24:56 smithi038 ceph-mon[92422]: pgmap v7129: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 157 KiB/s wr, 9 op/s 2023-03-09T18:24:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12870.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:24:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:24:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12408.406397 secs 2023-03-09T18:24:58.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:24:58 smithi122 ceph-mon[93869]: pgmap v7130: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 292 KiB/s wr, 13 op/s 2023-03-09T18:24:58.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:24:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12408.406397 secs 2023-03-09T18:24:58.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:24:58 smithi181 ceph-mon[94189]: pgmap v7130: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 292 KiB/s wr, 13 op/s 2023-03-09T18:24:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:24:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12408.406397 secs 2023-03-09T18:24:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:24:58 smithi038 ceph-mon[92422]: pgmap v7130: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 292 KiB/s wr, 13 op/s 2023-03-09T18:25:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:24:59 smithi122 ceph-mon[93869]: pgmap v7131: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 205 KiB/s wr, 9 op/s 2023-03-09T18:25:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:24:59 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:25:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:24:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:25:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:24:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:25:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:24:59 smithi181 ceph-mon[94189]: pgmap v7131: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 205 KiB/s wr, 9 op/s 2023-03-09T18:25:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:24:59 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:25:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:24:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:25:00.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:24:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:25:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:24:59 smithi038 ceph-mon[92422]: pgmap v7131: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 205 KiB/s wr, 9 op/s 2023-03-09T18:25:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:24:59 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:25:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:24:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:25:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:24:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:25:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:25:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12411.108163 secs 2023-03-09T18:25:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:25:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12411.108163 secs 2023-03-09T18:25:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:25:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12411.108163 secs 2023-03-09T18:25:02.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:25:02 smithi122 ceph-mon[93869]: pgmap v7132: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 265 KiB/s wr, 14 op/s 2023-03-09T18:25:02.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:25:02 smithi181 ceph-mon[94189]: pgmap v7132: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 265 KiB/s wr, 14 op/s 2023-03-09T18:25:02.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:25:02 smithi038 ceph-mon[92422]: pgmap v7132: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 265 KiB/s wr, 14 op/s 2023-03-09T18:25:03.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12875.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:25:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:25:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12413.406522 secs 2023-03-09T18:25:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:25:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12413.406522 secs 2023-03-09T18:25:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:25:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12413.406522 secs 2023-03-09T18:25:04.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:25:04 smithi122 ceph-mon[93869]: pgmap v7133: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 218 KiB/s wr, 22 op/s 2023-03-09T18:25:04.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:25:04 smithi181 ceph-mon[94189]: pgmap v7133: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 218 KiB/s wr, 22 op/s 2023-03-09T18:25:04.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:25:04 smithi038 ceph-mon[92422]: pgmap v7133: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 218 KiB/s wr, 22 op/s 2023-03-09T18:25:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:25:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12416.108224 secs 2023-03-09T18:25:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:25:05 smithi181 ceph-mon[94189]: osdmap e659: 12 total, 12 up, 12 in 2023-03-09T18:25:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:25:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12416.108224 secs 2023-03-09T18:25:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:25:05 smithi038 ceph-mon[92422]: osdmap e659: 12 total, 12 up, 12 in 2023-03-09T18:25:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:25:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12416.108224 secs 2023-03-09T18:25:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:25:05 smithi122 ceph-mon[93869]: osdmap e659: 12 total, 12 up, 12 in 2023-03-09T18:25:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:25:06 smithi038 ceph-mon[92422]: pgmap v7135: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 283 KiB/s rd, 324 KiB/s wr, 28 op/s 2023-03-09T18:25:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:25:06 smithi122 ceph-mon[93869]: pgmap v7135: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 283 KiB/s rd, 324 KiB/s wr, 28 op/s 2023-03-09T18:25:07.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:25:06 smithi181 ceph-mon[94189]: pgmap v7135: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 283 KiB/s rd, 324 KiB/s wr, 28 op/s 2023-03-09T18:25:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12880.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:25:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:25:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12418.406646 secs 2023-03-09T18:25:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:25:08 smithi038 ceph-mon[92422]: pgmap v7136: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 161 KiB/s wr, 26 op/s 2023-03-09T18:25:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:25:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12418.406646 secs 2023-03-09T18:25:09.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:25:08 smithi122 ceph-mon[93869]: pgmap v7136: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 161 KiB/s wr, 26 op/s 2023-03-09T18:25:09.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:25:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12418.406646 secs 2023-03-09T18:25:09.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:25:08 smithi181 ceph-mon[94189]: pgmap v7136: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 161 KiB/s wr, 26 op/s 2023-03-09T18:25:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:25:10 smithi038 ceph-mon[92422]: pgmap v7137: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 161 KiB/s wr, 26 op/s 2023-03-09T18:25:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:25:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12421.108282 secs 2023-03-09T18:25:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:25:10 smithi122 ceph-mon[93869]: pgmap v7137: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 161 KiB/s wr, 26 op/s 2023-03-09T18:25:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:25:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12421.108282 secs 2023-03-09T18:25:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:25:10 smithi181 ceph-mon[94189]: pgmap v7137: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 161 KiB/s wr, 26 op/s 2023-03-09T18:25:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:25:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12421.108282 secs 2023-03-09T18:25:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:25:12 smithi122 ceph-mon[93869]: pgmap v7138: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 132 KiB/s wr, 23 op/s 2023-03-09T18:25:12.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:25:12 smithi122 ceph-mon[93869]: osdmap e660: 12 total, 12 up, 12 in 2023-03-09T18:25:12.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:25:12 smithi181 ceph-mon[94189]: pgmap v7138: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 132 KiB/s wr, 23 op/s 2023-03-09T18:25:12.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:25:12 smithi181 ceph-mon[94189]: osdmap e660: 12 total, 12 up, 12 in 2023-03-09T18:25:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:25:12 smithi038 ceph-mon[92422]: pgmap v7138: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 132 KiB/s wr, 23 op/s 2023-03-09T18:25:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:25:12 smithi038 ceph-mon[92422]: osdmap e660: 12 total, 12 up, 12 in 2023-03-09T18:25:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12885.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:25:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:25:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12423.406781 secs 2023-03-09T18:25:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:25:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12423.406781 secs 2023-03-09T18:25:13.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:25:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12423.406781 secs 2023-03-09T18:25:14.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:25:14 smithi122 ceph-mon[93869]: pgmap v7140: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 119 KiB/s wr, 10 op/s 2023-03-09T18:25:14.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:25:14 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:25:14.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:25:14 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:25:14.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:25:14 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:25:14.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:25:14 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:25:14.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:25:14 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:25:14.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:25:14 smithi181 ceph-mon[94189]: pgmap v7140: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 119 KiB/s wr, 10 op/s 2023-03-09T18:25:14.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:25:14 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:25:14.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:25:14 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:25:14.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:25:14 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:25:14.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:25:14 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:25:14.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:25:14 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:25:14.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:25:14 smithi038 ceph-mon[92422]: pgmap v7140: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 119 KiB/s wr, 10 op/s 2023-03-09T18:25:14.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:25:14 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:25:14.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:25:14 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:25:14.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:25:14 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:25:14.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:25:14 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:25:14.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:25:14 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:25:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:25:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12426.108345 secs 2023-03-09T18:25:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:25:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:25:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:25:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:25:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:25:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:25:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:25:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12426.108345 secs 2023-03-09T18:25:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:25:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:25:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:25:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:25:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:25:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:25:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:25:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12426.108345 secs 2023-03-09T18:25:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:25:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:25:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:25:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:25:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:25:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:25:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:25:16 smithi122 ceph-mon[93869]: pgmap v7141: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 45 KiB/s wr, 8 op/s 2023-03-09T18:25:16.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:25:16 smithi181 ceph-mon[94189]: pgmap v7141: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 45 KiB/s wr, 8 op/s 2023-03-09T18:25:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:25:16 smithi038 ceph-mon[92422]: pgmap v7141: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 45 KiB/s wr, 8 op/s 2023-03-09T18:25:18.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12890.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:25:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:25:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12428.406910 secs 2023-03-09T18:25:18.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:25:18 smithi122 ceph-mon[93869]: pgmap v7142: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 57 KiB/s wr, 8 op/s 2023-03-09T18:25:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:25:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12428.406910 secs 2023-03-09T18:25:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:25:18 smithi181 ceph-mon[94189]: pgmap v7142: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 57 KiB/s wr, 8 op/s 2023-03-09T18:25:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:25:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12428.406910 secs 2023-03-09T18:25:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:25:18 smithi038 ceph-mon[92422]: pgmap v7142: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 57 KiB/s wr, 8 op/s 2023-03-09T18:25:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:25:20 smithi181 ceph-mon[94189]: pgmap v7143: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 57 KiB/s wr, 8 op/s 2023-03-09T18:25:20.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:25:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12431.108404 secs 2023-03-09T18:25:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:25:20 smithi038 ceph-mon[92422]: pgmap v7143: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 57 KiB/s wr, 8 op/s 2023-03-09T18:25:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:25:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12431.108404 secs 2023-03-09T18:25:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:25:20 smithi122 ceph-mon[93869]: pgmap v7143: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 57 KiB/s wr, 8 op/s 2023-03-09T18:25:21.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:25:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12431.108404 secs 2023-03-09T18:25:22.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:25:22 smithi038 ceph-mon[92422]: pgmap v7144: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T18:25:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:25:22 smithi122 ceph-mon[93869]: pgmap v7144: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T18:25:23.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:25:22 smithi181 ceph-mon[94189]: pgmap v7144: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T18:25:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12895.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:25:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:25:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12433.407034 secs 2023-03-09T18:25:24.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:25:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12433.407034 secs 2023-03-09T18:25:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:25:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12433.407034 secs 2023-03-09T18:25:24.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:25:24 smithi038 ceph-mon[92422]: pgmap v7145: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 37 KiB/s wr, 7 op/s 2023-03-09T18:25:25.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:25:24 smithi122 ceph-mon[93869]: pgmap v7145: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 37 KiB/s wr, 7 op/s 2023-03-09T18:25:25.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:25:24 smithi181 ceph-mon[94189]: pgmap v7145: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 37 KiB/s wr, 7 op/s 2023-03-09T18:25:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:25:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12436.108468 secs 2023-03-09T18:25:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:25:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12436.108468 secs 2023-03-09T18:25:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:25:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12436.108468 secs 2023-03-09T18:25:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:25:26 smithi122 ceph-mon[93869]: pgmap v7146: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 75 KiB/s wr, 7 op/s 2023-03-09T18:25:27.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:25:26 smithi181 ceph-mon[94189]: pgmap v7146: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 75 KiB/s wr, 7 op/s 2023-03-09T18:25:27.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:25:26 smithi038 ceph-mon[92422]: pgmap v7146: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 75 KiB/s wr, 7 op/s 2023-03-09T18:25:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12900.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:25:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:25:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12438.407138 secs 2023-03-09T18:25:29.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:25:28 smithi122 ceph-mon[93869]: pgmap v7147: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 75 KiB/s wr, 9 op/s 2023-03-09T18:25:29.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:25:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12438.407138 secs 2023-03-09T18:25:29.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:25:28 smithi181 ceph-mon[94189]: pgmap v7147: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 75 KiB/s wr, 9 op/s 2023-03-09T18:25:29.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:25:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12438.407138 secs 2023-03-09T18:25:29.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:25:28 smithi038 ceph-mon[92422]: pgmap v7147: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 75 KiB/s wr, 9 op/s 2023-03-09T18:25:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:25:30 smithi122 ceph-mon[93869]: pgmap v7148: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 64 KiB/s wr, 7 op/s 2023-03-09T18:25:31.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:25:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12441.108530 secs 2023-03-09T18:25:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:25:30 smithi181 ceph-mon[94189]: pgmap v7148: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 64 KiB/s wr, 7 op/s 2023-03-09T18:25:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:25:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12441.108530 secs 2023-03-09T18:25:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:25:30 smithi038 ceph-mon[92422]: pgmap v7148: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 64 KiB/s wr, 7 op/s 2023-03-09T18:25:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:25:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12441.108530 secs 2023-03-09T18:25:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:25:32 smithi122 ceph-mon[93869]: pgmap v7149: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 100 KiB/s wr, 9 op/s 2023-03-09T18:25:33.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:25:32 smithi181 ceph-mon[94189]: pgmap v7149: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 100 KiB/s wr, 9 op/s 2023-03-09T18:25:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:25:32 smithi038 ceph-mon[92422]: pgmap v7149: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 100 KiB/s wr, 9 op/s 2023-03-09T18:25:33.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12905.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:25:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:25:33 smithi122 ceph-mon[93869]: pgmap v7150: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 74 KiB/s wr, 7 op/s 2023-03-09T18:25:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:25:33 smithi181 ceph-mon[94189]: pgmap v7150: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 74 KiB/s wr, 7 op/s 2023-03-09T18:25:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:25:33 smithi038 ceph-mon[92422]: pgmap v7150: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 74 KiB/s wr, 7 op/s 2023-03-09T18:25:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:25:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12446.108588 secs 2023-03-09T18:25:35.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:25:35 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:25:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:25:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12446.108588 secs 2023-03-09T18:25:35.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:25:35 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:25:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:25:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12446.108588 secs 2023-03-09T18:25:36.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:25:35 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:25:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:25:36 smithi122 ceph-mon[93869]: pgmap v7151: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 134 KiB/s wr, 7 op/s 2023-03-09T18:25:36.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:25:36 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:25:36.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:25:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:25:36.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:25:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:25:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:25:36 smithi181 ceph-mon[94189]: pgmap v7151: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 134 KiB/s wr, 7 op/s 2023-03-09T18:25:36.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:25:36 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:25:36.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:25:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:25:36.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:25:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:25:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:25:36 smithi038 ceph-mon[92422]: pgmap v7151: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 134 KiB/s wr, 7 op/s 2023-03-09T18:25:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:25:36 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:25:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:25:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:25:37.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:25:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:25:38.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12910.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:25:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:25:38 smithi122 ceph-mon[93869]: pgmap v7152: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 107 KiB/s wr, 9 op/s 2023-03-09T18:25:38.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:25:38 smithi181 ceph-mon[94189]: pgmap v7152: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 107 KiB/s wr, 9 op/s 2023-03-09T18:25:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:25:38 smithi038 ceph-mon[92422]: pgmap v7152: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 107 KiB/s wr, 9 op/s 2023-03-09T18:25:40.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:25:40 smithi181 ceph-mon[94189]: pgmap v7153: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 106 KiB/s wr, 7 op/s 2023-03-09T18:25:40.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:25:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12451.108647 secs 2023-03-09T18:25:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:25:40 smithi038 ceph-mon[92422]: pgmap v7153: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 106 KiB/s wr, 7 op/s 2023-03-09T18:25:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:25:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12451.108647 secs 2023-03-09T18:25:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:25:40 smithi122 ceph-mon[93869]: pgmap v7153: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 106 KiB/s wr, 7 op/s 2023-03-09T18:25:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:25:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12451.108647 secs 2023-03-09T18:25:42.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:25:42 smithi038 ceph-mon[92422]: pgmap v7154: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 202 KiB/s wr, 11 op/s 2023-03-09T18:25:43.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:25:42 smithi122 ceph-mon[93869]: pgmap v7154: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 202 KiB/s wr, 11 op/s 2023-03-09T18:25:43.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:25:42 smithi181 ceph-mon[94189]: pgmap v7154: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 202 KiB/s wr, 11 op/s 2023-03-09T18:25:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12915.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:25:44.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:25:44 smithi038 ceph-mon[92422]: pgmap v7155: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 167 KiB/s wr, 9 op/s 2023-03-09T18:25:45.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:25:44 smithi122 ceph-mon[93869]: pgmap v7155: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 167 KiB/s wr, 9 op/s 2023-03-09T18:25:45.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:25:44 smithi181 ceph-mon[94189]: pgmap v7155: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 167 KiB/s wr, 9 op/s 2023-03-09T18:25:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:25:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12456.108708 secs 2023-03-09T18:25:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:25:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12456.108708 secs 2023-03-09T18:25:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:25:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12456.108708 secs 2023-03-09T18:25:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:25:46 smithi122 ceph-mon[93869]: pgmap v7156: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 167 KiB/s wr, 9 op/s 2023-03-09T18:25:47.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:25:46 smithi181 ceph-mon[94189]: pgmap v7156: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 167 KiB/s wr, 9 op/s 2023-03-09T18:25:47.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:25:46 smithi038 ceph-mon[92422]: pgmap v7156: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 167 KiB/s wr, 9 op/s 2023-03-09T18:25:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12920.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:25:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:25:48 smithi122 ceph-mon[93869]: pgmap v7157: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 227 KiB/s wr, 13 op/s 2023-03-09T18:25:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:25:48 smithi181 ceph-mon[94189]: pgmap v7157: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 227 KiB/s wr, 13 op/s 2023-03-09T18:25:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:25:48 smithi038 ceph-mon[92422]: pgmap v7157: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 227 KiB/s wr, 13 op/s 2023-03-09T18:25:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:25:50 smithi122 ceph-mon[93869]: pgmap v7158: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 217 KiB/s wr, 10 op/s 2023-03-09T18:25:51.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:25:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12461.108781 secs 2023-03-09T18:25:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:25:50 smithi181 ceph-mon[94189]: pgmap v7158: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 217 KiB/s wr, 10 op/s 2023-03-09T18:25:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:25:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12461.108781 secs 2023-03-09T18:25:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:25:50 smithi038 ceph-mon[92422]: pgmap v7158: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 217 KiB/s wr, 10 op/s 2023-03-09T18:25:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:25:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12461.108781 secs 2023-03-09T18:25:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:25:52 smithi122 ceph-mon[93869]: pgmap v7159: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 271 KiB/s wr, 14 op/s 2023-03-09T18:25:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:25:52 smithi181 ceph-mon[94189]: pgmap v7159: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 271 KiB/s wr, 14 op/s 2023-03-09T18:25:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:25:52 smithi038 ceph-mon[92422]: pgmap v7159: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 271 KiB/s wr, 14 op/s 2023-03-09T18:25:53.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12925.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:25:55.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:25:54 smithi122 ceph-mon[93869]: pgmap v7160: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 176 KiB/s wr, 10 op/s 2023-03-09T18:25:55.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:25:54 smithi181 ceph-mon[94189]: pgmap v7160: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 176 KiB/s wr, 10 op/s 2023-03-09T18:25:55.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:25:54 smithi038 ceph-mon[92422]: pgmap v7160: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 176 KiB/s wr, 10 op/s 2023-03-09T18:25:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:25:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12466.108855 secs 2023-03-09T18:25:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:25:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12466.108855 secs 2023-03-09T18:25:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:25:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12466.108855 secs 2023-03-09T18:25:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:25:56 smithi122 ceph-mon[93869]: pgmap v7161: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 175 KiB/s wr, 9 op/s 2023-03-09T18:25:57.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:25:56 smithi181 ceph-mon[94189]: pgmap v7161: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 175 KiB/s wr, 9 op/s 2023-03-09T18:25:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:25:56 smithi038 ceph-mon[92422]: pgmap v7161: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 175 KiB/s wr, 9 op/s 2023-03-09T18:25:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:25:57 smithi122 ceph-mon[93869]: pgmap v7162: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 258 KiB/s wr, 12 op/s 2023-03-09T18:25:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:25:57 smithi181 ceph-mon[94189]: pgmap v7162: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 258 KiB/s wr, 12 op/s 2023-03-09T18:25:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:25:57 smithi038 ceph-mon[92422]: pgmap v7162: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 258 KiB/s wr, 12 op/s 2023-03-09T18:25:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12930.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:25:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:25:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12468.407745 secs 2023-03-09T18:25:59.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:25:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12468.407745 secs 2023-03-09T18:25:59.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:25:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12468.407745 secs 2023-03-09T18:26:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:25:59 smithi122 ceph-mon[93869]: pgmap v7163: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 137 KiB/s wr, 9 op/s 2023-03-09T18:26:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:25:59 smithi181 ceph-mon[94189]: pgmap v7163: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 137 KiB/s wr, 9 op/s 2023-03-09T18:26:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:25:59 smithi038 ceph-mon[92422]: pgmap v7163: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 137 KiB/s wr, 9 op/s 2023-03-09T18:26:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:26:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12471.108915 secs 2023-03-09T18:26:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:26:00 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:26:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:26:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:26:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:26:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:26:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:26:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12471.108915 secs 2023-03-09T18:26:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:26:00 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:26:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:26:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:26:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:26:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:26:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:26:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12471.108915 secs 2023-03-09T18:26:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:26:00 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:26:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:26:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:26:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:26:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:26:02.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:26:02 smithi122 ceph-mon[93869]: pgmap v7164: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 200 KiB/s wr, 14 op/s 2023-03-09T18:26:02.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:26:02 smithi181 ceph-mon[94189]: pgmap v7164: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 200 KiB/s wr, 14 op/s 2023-03-09T18:26:02.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:26:02 smithi038 ceph-mon[92422]: pgmap v7164: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 200 KiB/s wr, 14 op/s 2023-03-09T18:26:03.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12935.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:26:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:26:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12473.407894 secs 2023-03-09T18:26:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:26:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12473.407894 secs 2023-03-09T18:26:04.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:26:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12473.407894 secs 2023-03-09T18:26:04.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:26:04 smithi038 ceph-mon[92422]: pgmap v7165: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 167 KiB/s wr, 23 op/s 2023-03-09T18:26:05.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:26:04 smithi122 ceph-mon[93869]: pgmap v7165: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 167 KiB/s wr, 23 op/s 2023-03-09T18:26:05.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:26:04 smithi181 ceph-mon[94189]: pgmap v7165: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 167 KiB/s wr, 23 op/s 2023-03-09T18:26:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:26:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12476.108982 secs 2023-03-09T18:26:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:26:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12476.108982 secs 2023-03-09T18:26:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:26:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12476.108982 secs 2023-03-09T18:26:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:26:06 smithi038 ceph-mon[92422]: pgmap v7166: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 167 KiB/s wr, 22 op/s 2023-03-09T18:26:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:26:06 smithi122 ceph-mon[93869]: pgmap v7166: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 167 KiB/s wr, 22 op/s 2023-03-09T18:26:07.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:26:06 smithi181 ceph-mon[94189]: pgmap v7166: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 167 KiB/s wr, 22 op/s 2023-03-09T18:26:08.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12940.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:26:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:26:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12478.408030 secs 2023-03-09T18:26:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:26:08 smithi038 ceph-mon[92422]: pgmap v7167: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 202 KiB/s wr, 25 op/s 2023-03-09T18:26:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:26:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12478.408030 secs 2023-03-09T18:26:09.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:26:08 smithi122 ceph-mon[93869]: pgmap v7167: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 202 KiB/s wr, 25 op/s 2023-03-09T18:26:09.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:26:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12478.408030 secs 2023-03-09T18:26:09.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:26:08 smithi181 ceph-mon[94189]: pgmap v7167: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 202 KiB/s wr, 25 op/s 2023-03-09T18:26:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:26:10 smithi122 ceph-mon[93869]: pgmap v7168: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 120 KiB/s wr, 21 op/s 2023-03-09T18:26:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:26:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12481.109043 secs 2023-03-09T18:26:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:26:10 smithi122 ceph-mon[93869]: osdmap e661: 12 total, 12 up, 12 in 2023-03-09T18:26:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:26:10 smithi181 ceph-mon[94189]: pgmap v7168: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 120 KiB/s wr, 21 op/s 2023-03-09T18:26:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:26:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12481.109043 secs 2023-03-09T18:26:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:26:10 smithi181 ceph-mon[94189]: osdmap e661: 12 total, 12 up, 12 in 2023-03-09T18:26:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:26:10 smithi038 ceph-mon[92422]: pgmap v7168: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 120 KiB/s wr, 21 op/s 2023-03-09T18:26:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:26:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12481.109043 secs 2023-03-09T18:26:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:26:10 smithi038 ceph-mon[92422]: osdmap e661: 12 total, 12 up, 12 in 2023-03-09T18:26:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:26:12 smithi122 ceph-mon[93869]: pgmap v7170: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 106 KiB/s wr, 22 op/s 2023-03-09T18:26:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:26:12 smithi181 ceph-mon[94189]: pgmap v7170: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 106 KiB/s wr, 22 op/s 2023-03-09T18:26:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:26:12 smithi038 ceph-mon[92422]: pgmap v7170: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 106 KiB/s wr, 22 op/s 2023-03-09T18:26:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12945.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:26:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:26:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12483.408168 secs 2023-03-09T18:26:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:26:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12483.408168 secs 2023-03-09T18:26:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:26:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12483.408168 secs 2023-03-09T18:26:15.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:26:14 smithi122 ceph-mon[93869]: pgmap v7171: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 81 KiB/s wr, 7 op/s 2023-03-09T18:26:15.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:26:14 smithi181 ceph-mon[94189]: pgmap v7171: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 81 KiB/s wr, 7 op/s 2023-03-09T18:26:15.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:26:14 smithi038 ceph-mon[92422]: pgmap v7171: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 81 KiB/s wr, 7 op/s 2023-03-09T18:26:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:26:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12486.109104 secs 2023-03-09T18:26:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:26:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:26:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:26:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12486.109104 secs 2023-03-09T18:26:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:26:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:26:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:26:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12486.109104 secs 2023-03-09T18:26:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:26:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:26:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:26:16 smithi122 ceph-mon[93869]: pgmap v7172: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 81 KiB/s wr, 7 op/s 2023-03-09T18:26:17.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:26:16 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:26:17.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:26:16 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:26:17.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:26:16 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:26:17.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:26:16 smithi122 ceph-mon[93869]: osdmap e662: 12 total, 12 up, 12 in 2023-03-09T18:26:17.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:26:16 smithi181 ceph-mon[94189]: pgmap v7172: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 81 KiB/s wr, 7 op/s 2023-03-09T18:26:17.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:26:16 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:26:17.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:26:16 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:26:17.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:26:16 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:26:17.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:26:16 smithi181 ceph-mon[94189]: osdmap e662: 12 total, 12 up, 12 in 2023-03-09T18:26:17.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:26:16 smithi038 ceph-mon[92422]: pgmap v7172: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 81 KiB/s wr, 7 op/s 2023-03-09T18:26:17.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:26:16 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:26:17.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:26:16 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:26:17.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:26:16 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:26:17.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:26:16 smithi038 ceph-mon[92422]: osdmap e662: 12 total, 12 up, 12 in 2023-03-09T18:26:18.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:26:17 smithi122 ceph-mon[93869]: pgmap v7174: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 49 KiB/s wr, 9 op/s 2023-03-09T18:26:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:26:17 smithi181 ceph-mon[94189]: pgmap v7174: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 49 KiB/s wr, 9 op/s 2023-03-09T18:26:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:26:17 smithi038 ceph-mon[92422]: pgmap v7174: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 49 KiB/s wr, 9 op/s 2023-03-09T18:26:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12950.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:26:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:26:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12488.408258 secs 2023-03-09T18:26:19.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:26:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12488.408258 secs 2023-03-09T18:26:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:26:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12488.408258 secs 2023-03-09T18:26:20.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:26:19 smithi122 ceph-mon[93869]: pgmap v7175: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 40 KiB/s wr, 7 op/s 2023-03-09T18:26:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:26:19 smithi181 ceph-mon[94189]: pgmap v7175: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 40 KiB/s wr, 7 op/s 2023-03-09T18:26:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:26:19 smithi038 ceph-mon[92422]: pgmap v7175: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 40 KiB/s wr, 7 op/s 2023-03-09T18:26:21.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:26:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12491.109163 secs 2023-03-09T18:26:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:26:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12491.109163 secs 2023-03-09T18:26:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:26:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12491.109163 secs 2023-03-09T18:26:22.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:26:22 smithi181 ceph-mon[94189]: pgmap v7176: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2023-03-09T18:26:22.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:26:22 smithi038 ceph-mon[92422]: pgmap v7176: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2023-03-09T18:26:23.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:26:22 smithi122 ceph-mon[93869]: pgmap v7176: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2023-03-09T18:26:23.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12955.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:26:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:26:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12493.408378 secs 2023-03-09T18:26:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:26:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12493.408378 secs 2023-03-09T18:26:24.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:26:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12493.408378 secs 2023-03-09T18:26:24.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:26:24 smithi038 ceph-mon[92422]: pgmap v7177: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.8 KiB/s wr, 7 op/s 2023-03-09T18:26:25.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:26:24 smithi122 ceph-mon[93869]: pgmap v7177: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.8 KiB/s wr, 7 op/s 2023-03-09T18:26:25.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:26:24 smithi181 ceph-mon[94189]: pgmap v7177: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.8 KiB/s wr, 7 op/s 2023-03-09T18:26:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:26:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12496.109242 secs 2023-03-09T18:26:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:26:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12496.109242 secs 2023-03-09T18:26:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:26:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12496.109242 secs 2023-03-09T18:26:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:26:26 smithi122 ceph-mon[93869]: pgmap v7178: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 818 B/s wr, 7 op/s 2023-03-09T18:26:27.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:26:26 smithi181 ceph-mon[94189]: pgmap v7178: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 818 B/s wr, 7 op/s 2023-03-09T18:26:27.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:26:26 smithi038 ceph-mon[92422]: pgmap v7178: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 818 B/s wr, 7 op/s 2023-03-09T18:26:28.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12960.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:26:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:26:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12498.408499 secs 2023-03-09T18:26:29.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:26:28 smithi122 ceph-mon[93869]: pgmap v7179: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 43 KiB/s wr, 7 op/s 2023-03-09T18:26:29.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:26:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12498.408499 secs 2023-03-09T18:26:29.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:26:28 smithi181 ceph-mon[94189]: pgmap v7179: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 43 KiB/s wr, 7 op/s 2023-03-09T18:26:29.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:26:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12498.408499 secs 2023-03-09T18:26:29.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:26:28 smithi038 ceph-mon[92422]: pgmap v7179: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 43 KiB/s wr, 7 op/s 2023-03-09T18:26:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:26:30 smithi122 ceph-mon[93869]: pgmap v7180: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 39 KiB/s wr, 6 op/s 2023-03-09T18:26:31.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:26:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12501.109301 secs 2023-03-09T18:26:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:26:30 smithi181 ceph-mon[94189]: pgmap v7180: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 39 KiB/s wr, 6 op/s 2023-03-09T18:26:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:26:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12501.109301 secs 2023-03-09T18:26:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:26:30 smithi038 ceph-mon[92422]: pgmap v7180: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 39 KiB/s wr, 6 op/s 2023-03-09T18:26:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:26:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12501.109301 secs 2023-03-09T18:26:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:26:32 smithi122 ceph-mon[93869]: pgmap v7181: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 39 KiB/s wr, 10 op/s 2023-03-09T18:26:33.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:26:32 smithi181 ceph-mon[94189]: pgmap v7181: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 39 KiB/s wr, 10 op/s 2023-03-09T18:26:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:26:32 smithi038 ceph-mon[92422]: pgmap v7181: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 39 KiB/s wr, 10 op/s 2023-03-09T18:26:33.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12965.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:26:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:26:33 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 12503.408612 secs 2023-03-09T18:26:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:26:33 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 12503.408612 secs 2023-03-09T18:26:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:26:33 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 12503.408612 secs 2023-03-09T18:26:35.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:26:34 smithi122 ceph-mon[93869]: pgmap v7182: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 39 KiB/s wr, 7 op/s 2023-03-09T18:26:35.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:26:34 smithi181 ceph-mon[94189]: pgmap v7182: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 39 KiB/s wr, 7 op/s 2023-03-09T18:26:35.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:26:34 smithi038 ceph-mon[92422]: pgmap v7182: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 39 KiB/s wr, 7 op/s 2023-03-09T18:26:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:26:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12506.109402 secs 2023-03-09T18:26:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:26:35 smithi122 ceph-mon[93869]: pgmap v7183: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 39 KiB/s wr, 7 op/s 2023-03-09T18:26:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:26:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12506.109402 secs 2023-03-09T18:26:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:26:35 smithi181 ceph-mon[94189]: pgmap v7183: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 39 KiB/s wr, 7 op/s 2023-03-09T18:26:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:26:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12506.109402 secs 2023-03-09T18:26:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:26:35 smithi038 ceph-mon[92422]: pgmap v7183: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 39 KiB/s wr, 7 op/s 2023-03-09T18:26:37.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:26:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:26:37.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:26:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:26:37.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:26:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:26:37.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:26:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:26:37.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:26:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:26:37.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:26:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:26:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:26:37 smithi122 ceph-mon[93869]: pgmap v7184: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 78 KiB/s wr, 10 op/s 2023-03-09T18:26:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:26:37 smithi181 ceph-mon[94189]: pgmap v7184: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 78 KiB/s wr, 10 op/s 2023-03-09T18:26:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:26:37 smithi038 ceph-mon[92422]: pgmap v7184: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 78 KiB/s wr, 10 op/s 2023-03-09T18:26:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12970.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:26:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:26:38 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 12508.408709 secs 2023-03-09T18:26:39.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:26:38 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 12508.408709 secs 2023-03-09T18:26:39.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:26:38 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 12508.408709 secs 2023-03-09T18:26:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:26:39 smithi122 ceph-mon[93869]: pgmap v7185: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 40 KiB/s wr, 7 op/s 2023-03-09T18:26:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:26:39 smithi181 ceph-mon[94189]: pgmap v7185: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 40 KiB/s wr, 7 op/s 2023-03-09T18:26:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:26:39 smithi038 ceph-mon[92422]: pgmap v7185: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 40 KiB/s wr, 7 op/s 2023-03-09T18:26:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:26:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12511.109463 secs 2023-03-09T18:26:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:26:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12511.109463 secs 2023-03-09T18:26:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:26:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12511.109463 secs 2023-03-09T18:26:42.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:26:42 smithi038 ceph-mon[92422]: pgmap v7186: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 177 KiB/s wr, 11 op/s 2023-03-09T18:26:43.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:26:42 smithi122 ceph-mon[93869]: pgmap v7186: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 177 KiB/s wr, 11 op/s 2023-03-09T18:26:43.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:26:42 smithi181 ceph-mon[94189]: pgmap v7186: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 177 KiB/s wr, 11 op/s 2023-03-09T18:26:43.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12975.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:26:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:26:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12513.408846 secs 2023-03-09T18:26:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:26:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12513.408846 secs 2023-03-09T18:26:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:26:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12513.408846 secs 2023-03-09T18:26:44.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:26:44 smithi038 ceph-mon[92422]: pgmap v7187: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 177 KiB/s wr, 9 op/s 2023-03-09T18:26:44.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:26:44 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:26:44.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:26:44 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:26:45.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:26:44 smithi122 ceph-mon[93869]: pgmap v7187: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 177 KiB/s wr, 9 op/s 2023-03-09T18:26:45.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:26:44 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:26:45.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:26:44 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:26:45.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:26:44 smithi181 ceph-mon[94189]: pgmap v7187: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 177 KiB/s wr, 9 op/s 2023-03-09T18:26:45.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:26:44 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:26:45.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:26:44 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:26:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:26:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12516.109542 secs 2023-03-09T18:26:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:26:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12516.109542 secs 2023-03-09T18:26:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:26:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12516.109542 secs 2023-03-09T18:26:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:26:46 smithi122 ceph-mon[93869]: pgmap v7188: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 177 KiB/s wr, 9 op/s 2023-03-09T18:26:47.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:26:46 smithi181 ceph-mon[94189]: pgmap v7188: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 177 KiB/s wr, 9 op/s 2023-03-09T18:26:47.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:26:46 smithi038 ceph-mon[92422]: pgmap v7188: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 177 KiB/s wr, 9 op/s 2023-03-09T18:26:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12980.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:26:49.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:26:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12518.408941 secs 2023-03-09T18:26:49.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:26:48 smithi122 ceph-mon[93869]: pgmap v7189: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 191 KiB/s wr, 11 op/s 2023-03-09T18:26:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:26:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12518.408941 secs 2023-03-09T18:26:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:26:48 smithi181 ceph-mon[94189]: pgmap v7189: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 191 KiB/s wr, 11 op/s 2023-03-09T18:26:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:26:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12518.408941 secs 2023-03-09T18:26:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:26:48 smithi038 ceph-mon[92422]: pgmap v7189: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 191 KiB/s wr, 11 op/s 2023-03-09T18:26:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:26:50 smithi122 ceph-mon[93869]: pgmap v7190: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 152 KiB/s wr, 9 op/s 2023-03-09T18:26:51.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:26:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12521.109601 secs 2023-03-09T18:26:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:26:50 smithi181 ceph-mon[94189]: pgmap v7190: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 152 KiB/s wr, 9 op/s 2023-03-09T18:26:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:26:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12521.109601 secs 2023-03-09T18:26:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:26:50 smithi038 ceph-mon[92422]: pgmap v7190: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 152 KiB/s wr, 9 op/s 2023-03-09T18:26:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:26:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12521.109601 secs 2023-03-09T18:26:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:26:52 smithi122 ceph-mon[93869]: pgmap v7191: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 292 KiB/s wr, 13 op/s 2023-03-09T18:26:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:26:52 smithi181 ceph-mon[94189]: pgmap v7191: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 292 KiB/s wr, 13 op/s 2023-03-09T18:26:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:26:52 smithi038 ceph-mon[92422]: pgmap v7191: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 292 KiB/s wr, 13 op/s 2023-03-09T18:26:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12985.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:26:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:26:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12523.409059 secs 2023-03-09T18:26:54.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:26:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12523.409059 secs 2023-03-09T18:26:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:26:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12523.409059 secs 2023-03-09T18:26:55.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:26:54 smithi122 ceph-mon[93869]: pgmap v7192: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 155 KiB/s wr, 9 op/s 2023-03-09T18:26:55.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:26:54 smithi181 ceph-mon[94189]: pgmap v7192: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 155 KiB/s wr, 9 op/s 2023-03-09T18:26:55.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:26:54 smithi038 ceph-mon[92422]: pgmap v7192: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 155 KiB/s wr, 9 op/s 2023-03-09T18:26:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:26:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12526.109663 secs 2023-03-09T18:26:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:26:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12526.109663 secs 2023-03-09T18:26:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:26:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12526.109663 secs 2023-03-09T18:26:57.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:26:56 smithi122 ceph-mon[93869]: pgmap v7193: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 199 KiB/s wr, 10 op/s 2023-03-09T18:26:57.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:26:56 smithi181 ceph-mon[94189]: pgmap v7193: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 199 KiB/s wr, 10 op/s 2023-03-09T18:26:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:26:56 smithi038 ceph-mon[92422]: pgmap v7193: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 199 KiB/s wr, 10 op/s 2023-03-09T18:26:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:26:57 smithi122 ceph-mon[93869]: pgmap v7194: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 237 KiB/s wr, 12 op/s 2023-03-09T18:26:58.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:26:57 smithi181 ceph-mon[94189]: pgmap v7194: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 237 KiB/s wr, 12 op/s 2023-03-09T18:26:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:26:57 smithi038 ceph-mon[92422]: pgmap v7194: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 237 KiB/s wr, 12 op/s 2023-03-09T18:26:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12990.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:26:59.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:26:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12528.409158 secs 2023-03-09T18:26:59.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:26:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12528.409158 secs 2023-03-09T18:26:59.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:26:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12528.409158 secs 2023-03-09T18:27:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:26:59 smithi122 ceph-mon[93869]: pgmap v7195: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 223 KiB/s wr, 10 op/s 2023-03-09T18:27:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:26:59 smithi181 ceph-mon[94189]: pgmap v7195: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 223 KiB/s wr, 10 op/s 2023-03-09T18:27:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:26:59 smithi038 ceph-mon[92422]: pgmap v7195: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 223 KiB/s wr, 10 op/s 2023-03-09T18:27:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:27:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12531.109725 secs 2023-03-09T18:27:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:27:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12531.109725 secs 2023-03-09T18:27:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:27:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12531.109725 secs 2023-03-09T18:27:02.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:27:02 smithi038 ceph-mon[92422]: pgmap v7196: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 267 KiB/s wr, 16 op/s 2023-03-09T18:27:03.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:27:02 smithi122 ceph-mon[93869]: pgmap v7196: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 267 KiB/s wr, 16 op/s 2023-03-09T18:27:03.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:27:02 smithi181 ceph-mon[94189]: pgmap v7196: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 267 KiB/s wr, 16 op/s 2023-03-09T18:27:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 12995.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:27:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:27:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12533.409253 secs 2023-03-09T18:27:04.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:27:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12533.409253 secs 2023-03-09T18:27:04.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:27:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12533.409253 secs 2023-03-09T18:27:04.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:27:04 smithi038 ceph-mon[92422]: pgmap v7197: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 149 KiB/s wr, 23 op/s 2023-03-09T18:27:05.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:27:04 smithi122 ceph-mon[93869]: pgmap v7197: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 149 KiB/s wr, 23 op/s 2023-03-09T18:27:05.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:27:04 smithi181 ceph-mon[94189]: pgmap v7197: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 149 KiB/s wr, 23 op/s 2023-03-09T18:27:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:27:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12536.109785 secs 2023-03-09T18:27:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:27:05 smithi122 ceph-mon[93869]: osdmap e663: 12 total, 12 up, 12 in 2023-03-09T18:27:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:27:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12536.109785 secs 2023-03-09T18:27:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:27:05 smithi181 ceph-mon[94189]: osdmap e663: 12 total, 12 up, 12 in 2023-03-09T18:27:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:27:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12536.109785 secs 2023-03-09T18:27:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:27:05 smithi038 ceph-mon[92422]: osdmap e663: 12 total, 12 up, 12 in 2023-03-09T18:27:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:27:06 smithi122 ceph-mon[93869]: pgmap v7199: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 230 KiB/s wr, 28 op/s 2023-03-09T18:27:07.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:27:06 smithi181 ceph-mon[94189]: pgmap v7199: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 230 KiB/s wr, 28 op/s 2023-03-09T18:27:07.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:27:06 smithi038 ceph-mon[92422]: pgmap v7199: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 230 KiB/s wr, 28 op/s 2023-03-09T18:27:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13000.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:27:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:27:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12538.409364 secs 2023-03-09T18:27:09.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:27:08 smithi122 ceph-mon[93869]: pgmap v7200: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 194 KiB/s wr, 27 op/s 2023-03-09T18:27:09.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:27:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12538.409364 secs 2023-03-09T18:27:09.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:27:08 smithi181 ceph-mon[94189]: pgmap v7200: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 194 KiB/s wr, 27 op/s 2023-03-09T18:27:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:27:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12538.409364 secs 2023-03-09T18:27:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:27:08 smithi038 ceph-mon[92422]: pgmap v7200: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 194 KiB/s wr, 27 op/s 2023-03-09T18:27:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:27:10 smithi122 ceph-mon[93869]: pgmap v7201: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 194 KiB/s wr, 26 op/s 2023-03-09T18:27:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:27:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12541.109846 secs 2023-03-09T18:27:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:27:10 smithi181 ceph-mon[94189]: pgmap v7201: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 194 KiB/s wr, 26 op/s 2023-03-09T18:27:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:27:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12541.109846 secs 2023-03-09T18:27:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:27:10 smithi038 ceph-mon[92422]: pgmap v7201: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 194 KiB/s wr, 26 op/s 2023-03-09T18:27:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:27:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12541.109846 secs 2023-03-09T18:27:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:27:12 smithi038 ceph-mon[92422]: pgmap v7202: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 153 KiB/s wr, 23 op/s 2023-03-09T18:27:13.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:27:12 smithi038 ceph-mon[92422]: osdmap e664: 12 total, 12 up, 12 in 2023-03-09T18:27:13.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:27:12 smithi122 ceph-mon[93869]: pgmap v7202: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 153 KiB/s wr, 23 op/s 2023-03-09T18:27:13.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:27:12 smithi122 ceph-mon[93869]: osdmap e664: 12 total, 12 up, 12 in 2023-03-09T18:27:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:27:12 smithi181 ceph-mon[94189]: pgmap v7202: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 153 KiB/s wr, 23 op/s 2023-03-09T18:27:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:27:12 smithi181 ceph-mon[94189]: osdmap e664: 12 total, 12 up, 12 in 2023-03-09T18:27:13.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13005.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:27:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:27:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12543.409464 secs 2023-03-09T18:27:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:27:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12543.409464 secs 2023-03-09T18:27:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:27:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12543.409464 secs 2023-03-09T18:27:15.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:27:14 smithi122 ceph-mon[93869]: pgmap v7204: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 144 KiB/s wr, 10 op/s 2023-03-09T18:27:15.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:27:14 smithi181 ceph-mon[94189]: pgmap v7204: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 144 KiB/s wr, 10 op/s 2023-03-09T18:27:15.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:27:14 smithi038 ceph-mon[92422]: pgmap v7204: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 144 KiB/s wr, 10 op/s 2023-03-09T18:27:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:27:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12546.109905 secs 2023-03-09T18:27:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:27:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12546.109905 secs 2023-03-09T18:27:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:27:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12546.109905 secs 2023-03-09T18:27:17.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:27:16 smithi122 ceph-mon[93869]: pgmap v7205: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 8 op/s 2023-03-09T18:27:17.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:27:16 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:27:17.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:27:16 smithi181 ceph-mon[94189]: pgmap v7205: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 8 op/s 2023-03-09T18:27:17.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:27:16 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:27:17.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:27:16 smithi038 ceph-mon[92422]: pgmap v7205: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 8 op/s 2023-03-09T18:27:17.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:27:16 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:27:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:27:17 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:27:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:27:17 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:27:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:27:17 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:27:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:27:17 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:27:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:27:17 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:27:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:27:17 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:27:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:27:17 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:27:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:27:17 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:27:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:27:17 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:27:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13010.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:27:19.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:27:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12548.409617 secs 2023-03-09T18:27:19.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:27:18 smithi122 ceph-mon[93869]: pgmap v7206: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 26 KiB/s wr, 8 op/s 2023-03-09T18:27:19.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:27:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12548.409617 secs 2023-03-09T18:27:19.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:27:18 smithi181 ceph-mon[94189]: pgmap v7206: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 26 KiB/s wr, 8 op/s 2023-03-09T18:27:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:27:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12548.409617 secs 2023-03-09T18:27:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:27:18 smithi038 ceph-mon[92422]: pgmap v7206: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 26 KiB/s wr, 8 op/s 2023-03-09T18:27:21.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:27:20 smithi122 ceph-mon[93869]: pgmap v7207: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 26 KiB/s wr, 8 op/s 2023-03-09T18:27:21.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:27:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12551.109964 secs 2023-03-09T18:27:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:27:20 smithi181 ceph-mon[94189]: pgmap v7207: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 26 KiB/s wr, 8 op/s 2023-03-09T18:27:21.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:27:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12551.109964 secs 2023-03-09T18:27:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:27:20 smithi038 ceph-mon[92422]: pgmap v7207: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 26 KiB/s wr, 8 op/s 2023-03-09T18:27:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:27:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12551.109964 secs 2023-03-09T18:27:23.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:27:22 smithi122 ceph-mon[93869]: pgmap v7208: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T18:27:23.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:27:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12553.409728 secs 2023-03-09T18:27:23.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:27:22 smithi181 ceph-mon[94189]: pgmap v7208: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T18:27:23.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:27:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12553.409728 secs 2023-03-09T18:27:23.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:27:22 smithi038 ceph-mon[92422]: pgmap v7208: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T18:27:23.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:27:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12553.409728 secs 2023-03-09T18:27:23.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13015.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:27:25.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:27:24 smithi122 ceph-mon[93869]: pgmap v7209: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 6 op/s 2023-03-09T18:27:25.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:27:24 smithi181 ceph-mon[94189]: pgmap v7209: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 6 op/s 2023-03-09T18:27:25.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:27:24 smithi038 ceph-mon[92422]: pgmap v7209: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 6 op/s 2023-03-09T18:27:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:27:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12556.109985 secs 2023-03-09T18:27:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:27:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12556.109985 secs 2023-03-09T18:27:26.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:27:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12556.109985 secs 2023-03-09T18:27:27.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:27:26 smithi122 ceph-mon[93869]: pgmap v7210: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 8 op/s 2023-03-09T18:27:27.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:27:26 smithi181 ceph-mon[94189]: pgmap v7210: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 8 op/s 2023-03-09T18:27:27.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:27:26 smithi038 ceph-mon[92422]: pgmap v7210: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 8 op/s 2023-03-09T18:27:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:27:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12558.409852 secs 2023-03-09T18:27:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:27:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12558.409852 secs 2023-03-09T18:27:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:27:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12558.409852 secs 2023-03-09T18:27:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13020.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:27:29.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:27:28 smithi122 ceph-mon[93869]: pgmap v7211: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 51 KiB/s wr, 8 op/s 2023-03-09T18:27:29.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:27:28 smithi181 ceph-mon[94189]: pgmap v7211: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 51 KiB/s wr, 8 op/s 2023-03-09T18:27:29.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:27:28 smithi038 ceph-mon[92422]: pgmap v7211: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 51 KiB/s wr, 8 op/s 2023-03-09T18:27:30.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:27:29 smithi122 ceph-mon[93869]: pgmap v7212: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 40 KiB/s wr, 6 op/s 2023-03-09T18:27:30.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:27:29 smithi181 ceph-mon[94189]: pgmap v7212: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 40 KiB/s wr, 6 op/s 2023-03-09T18:27:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:27:29 smithi038 ceph-mon[92422]: pgmap v7212: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 40 KiB/s wr, 6 op/s 2023-03-09T18:27:31.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:27:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12561.110048 secs 2023-03-09T18:27:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:27:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12561.110048 secs 2023-03-09T18:27:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:27:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12561.110048 secs 2023-03-09T18:27:33.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:27:32 smithi122 ceph-mon[93869]: pgmap v7213: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 75 KiB/s wr, 9 op/s 2023-03-09T18:27:33.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:27:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12563.409967 secs 2023-03-09T18:27:33.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:27:32 smithi181 ceph-mon[94189]: pgmap v7213: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 75 KiB/s wr, 9 op/s 2023-03-09T18:27:33.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:27:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12563.409967 secs 2023-03-09T18:27:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:27:32 smithi038 ceph-mon[92422]: pgmap v7213: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 75 KiB/s wr, 9 op/s 2023-03-09T18:27:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:27:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12563.409967 secs 2023-03-09T18:27:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13025.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:27:35.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:27:34 smithi122 ceph-mon[93869]: pgmap v7214: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 75 KiB/s wr, 7 op/s 2023-03-09T18:27:35.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:27:34 smithi181 ceph-mon[94189]: pgmap v7214: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 75 KiB/s wr, 7 op/s 2023-03-09T18:27:35.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:27:34 smithi038 ceph-mon[92422]: pgmap v7214: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 75 KiB/s wr, 7 op/s 2023-03-09T18:27:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:27:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12566.110108 secs 2023-03-09T18:27:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:27:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12566.110108 secs 2023-03-09T18:27:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:27:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12566.110108 secs 2023-03-09T18:27:37.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:27:36 smithi122 ceph-mon[93869]: pgmap v7215: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 114 KiB/s wr, 8 op/s 2023-03-09T18:27:37.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:27:36 smithi181 ceph-mon[94189]: pgmap v7215: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 114 KiB/s wr, 8 op/s 2023-03-09T18:27:37.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:27:36 smithi038 ceph-mon[92422]: pgmap v7215: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 114 KiB/s wr, 8 op/s 2023-03-09T18:27:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:27:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12568.410028 secs 2023-03-09T18:27:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:27:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12568.410028 secs 2023-03-09T18:27:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:27:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12568.410028 secs 2023-03-09T18:27:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13030.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:27:39.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:27:38 smithi122 ceph-mon[93869]: pgmap v7216: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 76 KiB/s wr, 8 op/s 2023-03-09T18:27:39.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:27:38 smithi181 ceph-mon[94189]: pgmap v7216: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 76 KiB/s wr, 8 op/s 2023-03-09T18:27:39.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:27:38 smithi038 ceph-mon[92422]: pgmap v7216: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 76 KiB/s wr, 8 op/s 2023-03-09T18:27:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:27:40 smithi122 ceph-mon[93869]: pgmap v7217: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 77 KiB/s wr, 7 op/s 2023-03-09T18:27:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:27:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12571.110166 secs 2023-03-09T18:27:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:27:40 smithi181 ceph-mon[94189]: pgmap v7217: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 77 KiB/s wr, 7 op/s 2023-03-09T18:27:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:27:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12571.110166 secs 2023-03-09T18:27:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:27:40 smithi038 ceph-mon[92422]: pgmap v7217: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 77 KiB/s wr, 7 op/s 2023-03-09T18:27:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:27:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12571.110166 secs 2023-03-09T18:27:43.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:27:42 smithi122 ceph-mon[93869]: pgmap v7218: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 175 KiB/s wr, 11 op/s 2023-03-09T18:27:43.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:27:42 smithi181 ceph-mon[94189]: pgmap v7218: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 175 KiB/s wr, 11 op/s 2023-03-09T18:27:43.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:27:42 smithi038 ceph-mon[92422]: pgmap v7218: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 175 KiB/s wr, 11 op/s 2023-03-09T18:27:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13035.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:27:44.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:27:43 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:27:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:27:43 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:27:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:27:43 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:27:45.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:27:44 smithi122 ceph-mon[93869]: pgmap v7219: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 179 KiB/s wr, 9 op/s 2023-03-09T18:27:45.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:27:44 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:27:45.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:27:44 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:27:45.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:27:44 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:27:45.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:27:44 smithi181 ceph-mon[94189]: pgmap v7219: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 179 KiB/s wr, 9 op/s 2023-03-09T18:27:45.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:27:44 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:27:45.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:27:44 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:27:45.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:27:44 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:27:45.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:27:44 smithi038 ceph-mon[92422]: pgmap v7219: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 179 KiB/s wr, 9 op/s 2023-03-09T18:27:45.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:27:44 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:27:45.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:27:44 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:27:45.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:27:44 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:27:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:27:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12576.110227 secs 2023-03-09T18:27:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:27:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12576.110227 secs 2023-03-09T18:27:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:27:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12576.110227 secs 2023-03-09T18:27:47.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:27:46 smithi122 ceph-mon[93869]: pgmap v7220: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 179 KiB/s wr, 10 op/s 2023-03-09T18:27:47.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:27:46 smithi181 ceph-mon[94189]: pgmap v7220: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 179 KiB/s wr, 10 op/s 2023-03-09T18:27:47.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:27:46 smithi038 ceph-mon[92422]: pgmap v7220: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 179 KiB/s wr, 10 op/s 2023-03-09T18:27:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13040.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:27:49.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:27:48 smithi122 ceph-mon[93869]: pgmap v7221: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 150 KiB/s wr, 11 op/s 2023-03-09T18:27:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:27:48 smithi181 ceph-mon[94189]: pgmap v7221: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 150 KiB/s wr, 11 op/s 2023-03-09T18:27:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:27:48 smithi038 ceph-mon[92422]: pgmap v7221: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 150 KiB/s wr, 11 op/s 2023-03-09T18:27:50.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:27:49 smithi122 ceph-mon[93869]: pgmap v7222: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 273 KiB/s wr, 11 op/s 2023-03-09T18:27:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:27:49 smithi181 ceph-mon[94189]: pgmap v7222: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 273 KiB/s wr, 11 op/s 2023-03-09T18:27:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:27:49 smithi038 ceph-mon[92422]: pgmap v7222: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 273 KiB/s wr, 11 op/s 2023-03-09T18:27:51.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:27:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12581.110288 secs 2023-03-09T18:27:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:27:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12581.110288 secs 2023-03-09T18:27:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:27:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12581.110288 secs 2023-03-09T18:27:53.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:27:52 smithi122 ceph-mon[93869]: pgmap v7223: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 282 KiB/s wr, 13 op/s 2023-03-09T18:27:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:27:52 smithi181 ceph-mon[94189]: pgmap v7223: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 282 KiB/s wr, 13 op/s 2023-03-09T18:27:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:27:52 smithi038 ceph-mon[92422]: pgmap v7223: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 282 KiB/s wr, 13 op/s 2023-03-09T18:27:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13045.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:27:55.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:27:54 smithi122 ceph-mon[93869]: pgmap v7224: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 229 KiB/s wr, 10 op/s 2023-03-09T18:27:55.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:27:54 smithi181 ceph-mon[94189]: pgmap v7224: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 229 KiB/s wr, 10 op/s 2023-03-09T18:27:55.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:27:54 smithi038 ceph-mon[92422]: pgmap v7224: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 229 KiB/s wr, 10 op/s 2023-03-09T18:27:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:27:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12586.110348 secs 2023-03-09T18:27:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:27:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12586.110348 secs 2023-03-09T18:27:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:27:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12586.110348 secs 2023-03-09T18:27:57.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:27:56 smithi122 ceph-mon[93869]: pgmap v7225: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 189 KiB/s wr, 9 op/s 2023-03-09T18:27:57.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:27:56 smithi181 ceph-mon[94189]: pgmap v7225: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 189 KiB/s wr, 9 op/s 2023-03-09T18:27:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:27:56 smithi038 ceph-mon[92422]: pgmap v7225: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 189 KiB/s wr, 9 op/s 2023-03-09T18:27:58.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:27:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12588.410466 secs 2023-03-09T18:27:58.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:27:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12588.410466 secs 2023-03-09T18:27:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:27:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12588.410466 secs 2023-03-09T18:27:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13050.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:27:59.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:27:58 smithi122 ceph-mon[93869]: pgmap v7226: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 199 KiB/s wr, 10 op/s 2023-03-09T18:27:59.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:27:58 smithi181 ceph-mon[94189]: pgmap v7226: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 199 KiB/s wr, 10 op/s 2023-03-09T18:27:59.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:27:58 smithi038 ceph-mon[92422]: pgmap v7226: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 199 KiB/s wr, 10 op/s 2023-03-09T18:28:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:27:59 smithi122 ceph-mon[93869]: pgmap v7227: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 318 KiB/s wr, 9 op/s 2023-03-09T18:28:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:27:59 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:28:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:27:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:28:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:27:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:28:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:27:59 smithi181 ceph-mon[94189]: pgmap v7227: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 318 KiB/s wr, 9 op/s 2023-03-09T18:28:00.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:27:59 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:28:00.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:27:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:28:00.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:27:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:28:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:27:59 smithi038 ceph-mon[92422]: pgmap v7227: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 318 KiB/s wr, 9 op/s 2023-03-09T18:28:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:27:59 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:28:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:27:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:28:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:27:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:28:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:28:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12591.110411 secs 2023-03-09T18:28:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:28:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12591.110411 secs 2023-03-09T18:28:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:28:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12591.110411 secs 2023-03-09T18:28:03.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:28:02 smithi122 ceph-mon[93869]: pgmap v7228: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 217 KiB/s wr, 12 op/s 2023-03-09T18:28:03.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:28:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12593.410583 secs 2023-03-09T18:28:03.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:28:02 smithi181 ceph-mon[94189]: pgmap v7228: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 217 KiB/s wr, 12 op/s 2023-03-09T18:28:03.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:28:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12593.410583 secs 2023-03-09T18:28:03.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:28:02 smithi038 ceph-mon[92422]: pgmap v7228: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 217 KiB/s wr, 12 op/s 2023-03-09T18:28:03.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:28:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12593.410583 secs 2023-03-09T18:28:03.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13055.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:28:05.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:28:04 smithi122 ceph-mon[93869]: pgmap v7229: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 274 KiB/s wr, 23 op/s 2023-03-09T18:28:05.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:28:04 smithi181 ceph-mon[94189]: pgmap v7229: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 274 KiB/s wr, 23 op/s 2023-03-09T18:28:05.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:28:04 smithi038 ceph-mon[92422]: pgmap v7229: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 274 KiB/s wr, 23 op/s 2023-03-09T18:28:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:28:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12596.110467 secs 2023-03-09T18:28:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:28:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12596.110467 secs 2023-03-09T18:28:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:28:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12596.110467 secs 2023-03-09T18:28:07.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:28:06 smithi122 ceph-mon[93869]: pgmap v7230: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 230 KiB/s wr, 23 op/s 2023-03-09T18:28:07.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:28:06 smithi181 ceph-mon[94189]: pgmap v7230: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 230 KiB/s wr, 23 op/s 2023-03-09T18:28:07.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:28:06 smithi038 ceph-mon[92422]: pgmap v7230: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 230 KiB/s wr, 23 op/s 2023-03-09T18:28:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:28:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12598.410719 secs 2023-03-09T18:28:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:28:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12598.410719 secs 2023-03-09T18:28:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:28:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12598.410719 secs 2023-03-09T18:28:08.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13060.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:28:09.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:28:08 smithi122 ceph-mon[93869]: pgmap v7231: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 257 KiB/s wr, 25 op/s 2023-03-09T18:28:09.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:28:08 smithi181 ceph-mon[94189]: pgmap v7231: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 257 KiB/s wr, 25 op/s 2023-03-09T18:28:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:28:08 smithi038 ceph-mon[92422]: pgmap v7231: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 257 KiB/s wr, 25 op/s 2023-03-09T18:28:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:28:10 smithi122 ceph-mon[93869]: pgmap v7232: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 246 KiB/s wr, 23 op/s 2023-03-09T18:28:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:28:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12601.110524 secs 2023-03-09T18:28:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:28:10 smithi122 ceph-mon[93869]: osdmap e665: 12 total, 12 up, 12 in 2023-03-09T18:28:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:28:10 smithi181 ceph-mon[94189]: pgmap v7232: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 246 KiB/s wr, 23 op/s 2023-03-09T18:28:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:28:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12601.110524 secs 2023-03-09T18:28:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:28:10 smithi181 ceph-mon[94189]: osdmap e665: 12 total, 12 up, 12 in 2023-03-09T18:28:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:28:10 smithi038 ceph-mon[92422]: pgmap v7232: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 246 KiB/s wr, 23 op/s 2023-03-09T18:28:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:28:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12601.110524 secs 2023-03-09T18:28:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:28:10 smithi038 ceph-mon[92422]: osdmap e665: 12 total, 12 up, 12 in 2023-03-09T18:28:13.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:28:12 smithi122 ceph-mon[93869]: pgmap v7234: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 125 KiB/s wr, 23 op/s 2023-03-09T18:28:13.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:28:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12603.410838 secs 2023-03-09T18:28:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:28:12 smithi181 ceph-mon[94189]: pgmap v7234: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 125 KiB/s wr, 23 op/s 2023-03-09T18:28:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:28:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12603.410838 secs 2023-03-09T18:28:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:28:12 smithi038 ceph-mon[92422]: pgmap v7234: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 125 KiB/s wr, 23 op/s 2023-03-09T18:28:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:28:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12603.410838 secs 2023-03-09T18:28:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13065.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:28:14.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:28:13 smithi122 ceph-mon[93869]: pgmap v7235: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 45 KiB/s wr, 8 op/s 2023-03-09T18:28:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:28:13 smithi181 ceph-mon[94189]: pgmap v7235: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 45 KiB/s wr, 8 op/s 2023-03-09T18:28:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:28:13 smithi038 ceph-mon[92422]: pgmap v7235: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 45 KiB/s wr, 8 op/s 2023-03-09T18:28:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:28:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12606.110583 secs 2023-03-09T18:28:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:28:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12606.110583 secs 2023-03-09T18:28:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:28:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12606.110583 secs 2023-03-09T18:28:16.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:28:16 smithi181 ceph-mon[94189]: pgmap v7236: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 46 KiB/s wr, 8 op/s 2023-03-09T18:28:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:28:16 smithi038 ceph-mon[92422]: pgmap v7236: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 46 KiB/s wr, 8 op/s 2023-03-09T18:28:17.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:28:16 smithi122 ceph-mon[93869]: pgmap v7236: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 46 KiB/s wr, 8 op/s 2023-03-09T18:28:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:28:17 smithi122 ceph-mon[93869]: osdmap e666: 12 total, 12 up, 12 in 2023-03-09T18:28:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:28:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12608.410967 secs 2023-03-09T18:28:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:28:17 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:28:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:28:17 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:28:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:28:17 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:28:18.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:28:17 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:28:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:28:17 smithi181 ceph-mon[94189]: osdmap e666: 12 total, 12 up, 12 in 2023-03-09T18:28:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:28:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12608.410967 secs 2023-03-09T18:28:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:28:17 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:28:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:28:17 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:28:18.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:28:17 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:28:18.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:28:17 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:28:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:28:17 smithi038 ceph-mon[92422]: osdmap e666: 12 total, 12 up, 12 in 2023-03-09T18:28:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:28:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12608.410967 secs 2023-03-09T18:28:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:28:17 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:28:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:28:17 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:28:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:28:17 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:28:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:28:17 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:28:18.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13070.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:28:19.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:28:18 smithi122 ceph-mon[93869]: pgmap v7238: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 32 KiB/s wr, 10 op/s 2023-03-09T18:28:19.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:28:18 smithi181 ceph-mon[94189]: pgmap v7238: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 32 KiB/s wr, 10 op/s 2023-03-09T18:28:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:28:18 smithi038 ceph-mon[92422]: pgmap v7238: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 32 KiB/s wr, 10 op/s 2023-03-09T18:28:21.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:28:20 smithi122 ceph-mon[93869]: pgmap v7239: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 26 KiB/s wr, 8 op/s 2023-03-09T18:28:21.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:28:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12611.110656 secs 2023-03-09T18:28:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:28:20 smithi181 ceph-mon[94189]: pgmap v7239: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 26 KiB/s wr, 8 op/s 2023-03-09T18:28:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:28:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12611.110656 secs 2023-03-09T18:28:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:28:20 smithi038 ceph-mon[92422]: pgmap v7239: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 26 KiB/s wr, 8 op/s 2023-03-09T18:28:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:28:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12611.110656 secs 2023-03-09T18:28:23.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:28:22 smithi122 ceph-mon[93869]: pgmap v7240: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T18:28:23.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:28:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12613.411087 secs 2023-03-09T18:28:23.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:28:22 smithi181 ceph-mon[94189]: pgmap v7240: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T18:28:23.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:28:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12613.411087 secs 2023-03-09T18:28:23.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:28:22 smithi038 ceph-mon[92422]: pgmap v7240: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T18:28:23.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:28:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12613.411087 secs 2023-03-09T18:28:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13075.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:28:25.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:28:24 smithi122 ceph-mon[93869]: pgmap v7241: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 8 op/s 2023-03-09T18:28:25.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:28:24 smithi181 ceph-mon[94189]: pgmap v7241: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 8 op/s 2023-03-09T18:28:25.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:28:24 smithi038 ceph-mon[92422]: pgmap v7241: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 8 op/s 2023-03-09T18:28:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:28:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12616.110716 secs 2023-03-09T18:28:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:28:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12616.110716 secs 2023-03-09T18:28:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:28:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12616.110716 secs 2023-03-09T18:28:27.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:28:26 smithi122 ceph-mon[93869]: pgmap v7242: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 7 op/s 2023-03-09T18:28:27.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:28:26 smithi181 ceph-mon[94189]: pgmap v7242: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 7 op/s 2023-03-09T18:28:27.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:28:26 smithi038 ceph-mon[92422]: pgmap v7242: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 7 op/s 2023-03-09T18:28:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:28:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12618.411193 secs 2023-03-09T18:28:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:28:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12618.411193 secs 2023-03-09T18:28:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:28:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12618.411193 secs 2023-03-09T18:28:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13080.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:28:29.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:28:28 smithi122 ceph-mon[93869]: pgmap v7243: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 8 op/s 2023-03-09T18:28:29.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:28:28 smithi181 ceph-mon[94189]: pgmap v7243: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 8 op/s 2023-03-09T18:28:29.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:28:28 smithi038 ceph-mon[92422]: pgmap v7243: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 8 op/s 2023-03-09T18:28:31.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:28:30 smithi122 ceph-mon[93869]: pgmap v7244: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 39 KiB/s wr, 6 op/s 2023-03-09T18:28:31.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:28:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12621.110777 secs 2023-03-09T18:28:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:28:30 smithi181 ceph-mon[94189]: pgmap v7244: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 39 KiB/s wr, 6 op/s 2023-03-09T18:28:31.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:28:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12621.110777 secs 2023-03-09T18:28:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:28:30 smithi038 ceph-mon[92422]: pgmap v7244: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 39 KiB/s wr, 6 op/s 2023-03-09T18:28:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:28:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12621.110777 secs 2023-03-09T18:28:33.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:28:32 smithi122 ceph-mon[93869]: pgmap v7245: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 40 KiB/s wr, 8 op/s 2023-03-09T18:28:33.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:28:32 smithi181 ceph-mon[94189]: pgmap v7245: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 40 KiB/s wr, 8 op/s 2023-03-09T18:28:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:28:32 smithi038 ceph-mon[92422]: pgmap v7245: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 40 KiB/s wr, 8 op/s 2023-03-09T18:28:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13085.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:28:35.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:28:34 smithi122 ceph-mon[93869]: pgmap v7246: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 40 KiB/s wr, 7 op/s 2023-03-09T18:28:35.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:28:34 smithi181 ceph-mon[94189]: pgmap v7246: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 40 KiB/s wr, 7 op/s 2023-03-09T18:28:35.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:28:34 smithi038 ceph-mon[92422]: pgmap v7246: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 40 KiB/s wr, 7 op/s 2023-03-09T18:28:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:28:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12626.110837 secs 2023-03-09T18:28:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:28:35 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:28:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:28:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12626.110837 secs 2023-03-09T18:28:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:28:35 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:28:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:28:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12626.110837 secs 2023-03-09T18:28:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:28:35 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:28:37.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:28:36 smithi122 ceph-mon[93869]: pgmap v7247: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 40 KiB/s wr, 7 op/s 2023-03-09T18:28:37.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:28:36 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:28:37.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:28:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:28:37.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:28:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:28:37.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:28:36 smithi181 ceph-mon[94189]: pgmap v7247: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 40 KiB/s wr, 7 op/s 2023-03-09T18:28:37.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:28:36 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:28:37.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:28:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:28:37.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:28:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:28:37.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:28:36 smithi038 ceph-mon[92422]: pgmap v7247: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 40 KiB/s wr, 7 op/s 2023-03-09T18:28:37.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:28:36 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:28:37.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:28:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:28:37.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:28:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:28:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13090.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:28:39.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:28:38 smithi122 ceph-mon[93869]: pgmap v7248: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 112 KiB/s wr, 9 op/s 2023-03-09T18:28:39.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:28:38 smithi181 ceph-mon[94189]: pgmap v7248: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 112 KiB/s wr, 9 op/s 2023-03-09T18:28:39.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:28:38 smithi038 ceph-mon[92422]: pgmap v7248: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 112 KiB/s wr, 9 op/s 2023-03-09T18:28:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:28:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12631.110898 secs 2023-03-09T18:28:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:28:40 smithi122 ceph-mon[93869]: pgmap v7249: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 73 KiB/s wr, 7 op/s 2023-03-09T18:28:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:28:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12631.110898 secs 2023-03-09T18:28:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:28:40 smithi181 ceph-mon[94189]: pgmap v7249: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 73 KiB/s wr, 7 op/s 2023-03-09T18:28:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:28:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12631.110898 secs 2023-03-09T18:28:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:28:40 smithi038 ceph-mon[92422]: pgmap v7249: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 73 KiB/s wr, 7 op/s 2023-03-09T18:28:43.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:28:42 smithi122 ceph-mon[93869]: pgmap v7250: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 83 KiB/s wr, 9 op/s 2023-03-09T18:28:43.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:28:42 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 12633.411566 secs 2023-03-09T18:28:43.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:28:42 smithi181 ceph-mon[94189]: pgmap v7250: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 83 KiB/s wr, 9 op/s 2023-03-09T18:28:43.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:28:42 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 12633.411566 secs 2023-03-09T18:28:43.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:28:42 smithi038 ceph-mon[92422]: pgmap v7250: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 83 KiB/s wr, 9 op/s 2023-03-09T18:28:43.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:28:42 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 12633.411566 secs 2023-03-09T18:28:43.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13095.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:28:44.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:28:43 smithi122 ceph-mon[93869]: pgmap v7251: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 167 KiB/s wr, 9 op/s 2023-03-09T18:28:44.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:28:43 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:28:44.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:28:43 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:28:44.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:28:43 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:28:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:28:43 smithi181 ceph-mon[94189]: pgmap v7251: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 167 KiB/s wr, 9 op/s 2023-03-09T18:28:44.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:28:43 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:28:44.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:28:43 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:28:44.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:28:43 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:28:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:28:43 smithi038 ceph-mon[92422]: pgmap v7251: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 167 KiB/s wr, 9 op/s 2023-03-09T18:28:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:28:43 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:28:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:28:43 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:28:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:28:43 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:28:45.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:28:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12636.110959 secs 2023-03-09T18:28:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:28:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12636.110959 secs 2023-03-09T18:28:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:28:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12636.110959 secs 2023-03-09T18:28:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:28:46 smithi038 ceph-mon[92422]: pgmap v7252: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 167 KiB/s wr, 9 op/s 2023-03-09T18:28:47.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:28:46 smithi122 ceph-mon[93869]: pgmap v7252: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 167 KiB/s wr, 9 op/s 2023-03-09T18:28:47.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:28:46 smithi181 ceph-mon[94189]: pgmap v7252: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 167 KiB/s wr, 9 op/s 2023-03-09T18:28:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:28:47 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 12638.411688 secs 2023-03-09T18:28:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:28:47 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 12638.411688 secs 2023-03-09T18:28:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:28:47 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 12638.411688 secs 2023-03-09T18:28:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13100.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:28:49.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:28:48 smithi122 ceph-mon[93869]: pgmap v7253: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 269 KiB/s wr, 12 op/s 2023-03-09T18:28:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:28:48 smithi181 ceph-mon[94189]: pgmap v7253: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 269 KiB/s wr, 12 op/s 2023-03-09T18:28:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:28:48 smithi038 ceph-mon[92422]: pgmap v7253: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 269 KiB/s wr, 12 op/s 2023-03-09T18:28:51.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:28:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12641.111023 secs 2023-03-09T18:28:51.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:28:50 smithi122 ceph-mon[93869]: pgmap v7254: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 197 KiB/s wr, 9 op/s 2023-03-09T18:28:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:28:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12641.111023 secs 2023-03-09T18:28:51.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:28:50 smithi181 ceph-mon[94189]: pgmap v7254: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 197 KiB/s wr, 9 op/s 2023-03-09T18:28:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:28:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12641.111023 secs 2023-03-09T18:28:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:28:50 smithi038 ceph-mon[92422]: pgmap v7254: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 197 KiB/s wr, 9 op/s 2023-03-09T18:28:53.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:28:52 smithi122 ceph-mon[93869]: pgmap v7255: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 247 KiB/s wr, 12 op/s 2023-03-09T18:28:53.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:28:52 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 12643.411813 secs 2023-03-09T18:28:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:28:52 smithi181 ceph-mon[94189]: pgmap v7255: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 247 KiB/s wr, 12 op/s 2023-03-09T18:28:53.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:28:52 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 12643.411813 secs 2023-03-09T18:28:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:28:52 smithi038 ceph-mon[92422]: pgmap v7255: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 247 KiB/s wr, 12 op/s 2023-03-09T18:28:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:28:52 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 12643.411813 secs 2023-03-09T18:28:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13105.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:28:55.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:28:54 smithi122 ceph-mon[93869]: pgmap v7256: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 237 KiB/s wr, 10 op/s 2023-03-09T18:28:55.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:28:54 smithi181 ceph-mon[94189]: pgmap v7256: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 237 KiB/s wr, 10 op/s 2023-03-09T18:28:55.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:28:54 smithi038 ceph-mon[92422]: pgmap v7256: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 237 KiB/s wr, 10 op/s 2023-03-09T18:28:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:28:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12646.111084 secs 2023-03-09T18:28:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:28:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12646.111084 secs 2023-03-09T18:28:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:28:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12646.111084 secs 2023-03-09T18:28:57.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:28:56 smithi122 ceph-mon[93869]: pgmap v7257: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 292 KiB/s wr, 12 op/s 2023-03-09T18:28:57.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:28:56 smithi181 ceph-mon[94189]: pgmap v7257: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 292 KiB/s wr, 12 op/s 2023-03-09T18:28:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:28:56 smithi038 ceph-mon[92422]: pgmap v7257: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 292 KiB/s wr, 12 op/s 2023-03-09T18:28:58.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:28:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12648.411933 secs 2023-03-09T18:28:58.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:28:57 smithi122 ceph-mon[93869]: pgmap v7258: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 292 KiB/s wr, 12 op/s 2023-03-09T18:28:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:28:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12648.411933 secs 2023-03-09T18:28:58.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:28:57 smithi181 ceph-mon[94189]: pgmap v7258: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 292 KiB/s wr, 12 op/s 2023-03-09T18:28:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:28:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12648.411933 secs 2023-03-09T18:28:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:28:57 smithi038 ceph-mon[92422]: pgmap v7258: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 292 KiB/s wr, 12 op/s 2023-03-09T18:28:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13110.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:29:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:29:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12651.111169 secs 2023-03-09T18:29:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:29:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:29:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:29:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:29:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:29:00 smithi122 ceph-mon[93869]: pgmap v7259: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 191 KiB/s wr, 8 op/s 2023-03-09T18:29:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:29:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12651.111169 secs 2023-03-09T18:29:00.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:29:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:29:00.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:29:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:29:00.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:29:00 smithi181 ceph-mon[94189]: pgmap v7259: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 191 KiB/s wr, 8 op/s 2023-03-09T18:29:01.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:29:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12651.111169 secs 2023-03-09T18:29:01.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:29:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:29:01.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:29:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:29:01.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:29:00 smithi038 ceph-mon[92422]: pgmap v7259: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 191 KiB/s wr, 8 op/s 2023-03-09T18:29:03.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:29:02 smithi122 ceph-mon[93869]: pgmap v7260: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 245 KiB/s wr, 13 op/s 2023-03-09T18:29:03.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:29:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12653.412056 secs 2023-03-09T18:29:03.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:29:02 smithi181 ceph-mon[94189]: pgmap v7260: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 245 KiB/s wr, 13 op/s 2023-03-09T18:29:03.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:29:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12653.412056 secs 2023-03-09T18:29:03.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:29:02 smithi038 ceph-mon[92422]: pgmap v7260: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 245 KiB/s wr, 13 op/s 2023-03-09T18:29:03.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:29:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12653.412056 secs 2023-03-09T18:29:03.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13115.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:29:05.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:29:04 smithi122 ceph-mon[93869]: pgmap v7261: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 216 KiB/s wr, 22 op/s 2023-03-09T18:29:05.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:29:04 smithi181 ceph-mon[94189]: pgmap v7261: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 216 KiB/s wr, 22 op/s 2023-03-09T18:29:05.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:29:04 smithi038 ceph-mon[92422]: pgmap v7261: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 216 KiB/s wr, 22 op/s 2023-03-09T18:29:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:29:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12656.111231 secs 2023-03-09T18:29:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:29:05 smithi122 ceph-mon[93869]: osdmap e667: 12 total, 12 up, 12 in 2023-03-09T18:29:06.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:29:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12656.111231 secs 2023-03-09T18:29:06.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:29:05 smithi181 ceph-mon[94189]: osdmap e667: 12 total, 12 up, 12 in 2023-03-09T18:29:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:29:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12656.111231 secs 2023-03-09T18:29:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:29:05 smithi038 ceph-mon[92422]: osdmap e667: 12 total, 12 up, 12 in 2023-03-09T18:29:07.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:29:06 smithi122 ceph-mon[93869]: pgmap v7263: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 156 KiB/s wr, 25 op/s 2023-03-09T18:29:07.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:29:06 smithi181 ceph-mon[94189]: pgmap v7263: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 156 KiB/s wr, 25 op/s 2023-03-09T18:29:07.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:29:06 smithi038 ceph-mon[92422]: pgmap v7263: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 156 KiB/s wr, 25 op/s 2023-03-09T18:29:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:29:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12658.412136 secs 2023-03-09T18:29:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:29:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12658.412136 secs 2023-03-09T18:29:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:29:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12658.412136 secs 2023-03-09T18:29:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13120.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:29:09.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:29:08 smithi122 ceph-mon[93869]: pgmap v7264: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 169 KiB/s wr, 26 op/s 2023-03-09T18:29:09.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:29:08 smithi181 ceph-mon[94189]: pgmap v7264: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 169 KiB/s wr, 26 op/s 2023-03-09T18:29:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:29:08 smithi038 ceph-mon[92422]: pgmap v7264: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 169 KiB/s wr, 26 op/s 2023-03-09T18:29:10.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:29:09 smithi122 ceph-mon[93869]: pgmap v7265: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 169 KiB/s wr, 26 op/s 2023-03-09T18:29:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:29:09 smithi181 ceph-mon[94189]: pgmap v7265: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 169 KiB/s wr, 26 op/s 2023-03-09T18:29:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:29:09 smithi038 ceph-mon[92422]: pgmap v7265: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 169 KiB/s wr, 26 op/s 2023-03-09T18:29:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:29:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12661.111290 secs 2023-03-09T18:29:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:29:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12661.111290 secs 2023-03-09T18:29:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:29:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12661.111290 secs 2023-03-09T18:29:13.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:29:12 smithi122 ceph-mon[93869]: pgmap v7266: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 148 KiB/s wr, 25 op/s 2023-03-09T18:29:13.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:29:12 smithi122 ceph-mon[93869]: osdmap e668: 12 total, 12 up, 12 in 2023-03-09T18:29:13.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:29:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12663.412256 secs 2023-03-09T18:29:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:29:12 smithi181 ceph-mon[94189]: pgmap v7266: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 148 KiB/s wr, 25 op/s 2023-03-09T18:29:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:29:12 smithi181 ceph-mon[94189]: osdmap e668: 12 total, 12 up, 12 in 2023-03-09T18:29:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:29:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12663.412256 secs 2023-03-09T18:29:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:29:12 smithi038 ceph-mon[92422]: pgmap v7266: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 148 KiB/s wr, 25 op/s 2023-03-09T18:29:13.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:29:12 smithi038 ceph-mon[92422]: osdmap e668: 12 total, 12 up, 12 in 2023-03-09T18:29:13.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:29:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12663.412256 secs 2023-03-09T18:29:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13125.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:29:15.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:29:14 smithi122 ceph-mon[93869]: pgmap v7268: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 137 KiB/s wr, 11 op/s 2023-03-09T18:29:15.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:29:14 smithi181 ceph-mon[94189]: pgmap v7268: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 137 KiB/s wr, 11 op/s 2023-03-09T18:29:15.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:29:14 smithi038 ceph-mon[92422]: pgmap v7268: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 137 KiB/s wr, 11 op/s 2023-03-09T18:29:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:29:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12666.111350 secs 2023-03-09T18:29:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:29:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12666.111350 secs 2023-03-09T18:29:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:29:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12666.111350 secs 2023-03-09T18:29:17.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:29:16 smithi122 ceph-mon[93869]: pgmap v7269: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 59 KiB/s wr, 9 op/s 2023-03-09T18:29:17.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:29:16 smithi181 ceph-mon[94189]: pgmap v7269: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 59 KiB/s wr, 9 op/s 2023-03-09T18:29:17.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:29:16 smithi038 ceph-mon[92422]: pgmap v7269: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 59 KiB/s wr, 9 op/s 2023-03-09T18:29:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:29:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12668.412379 secs 2023-03-09T18:29:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:29:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12668.412379 secs 2023-03-09T18:29:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:29:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12668.412379 secs 2023-03-09T18:29:18.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13130.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:29:19.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:29:18 smithi122 ceph-mon[93869]: pgmap v7270: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 45 KiB/s wr, 7 op/s 2023-03-09T18:29:19.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:29:18 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:29:19.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:29:18 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:29:19.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:29:18 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:29:19.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:29:18 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:29:19.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:29:18 smithi181 ceph-mon[94189]: pgmap v7270: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 45 KiB/s wr, 7 op/s 2023-03-09T18:29:19.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:29:18 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:29:19.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:29:18 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:29:19.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:29:18 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:29:19.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:29:18 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:29:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:29:18 smithi038 ceph-mon[92422]: pgmap v7270: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 45 KiB/s wr, 7 op/s 2023-03-09T18:29:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:29:18 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:29:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:29:18 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:29:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:29:18 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:29:19.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:29:18 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:29:20.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:29:19 smithi122 ceph-mon[93869]: pgmap v7271: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 45 KiB/s wr, 7 op/s 2023-03-09T18:29:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:29:19 smithi181 ceph-mon[94189]: pgmap v7271: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 45 KiB/s wr, 7 op/s 2023-03-09T18:29:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:29:19 smithi038 ceph-mon[92422]: pgmap v7271: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 45 KiB/s wr, 7 op/s 2023-03-09T18:29:21.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:29:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12671.111404 secs 2023-03-09T18:29:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:29:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12671.111404 secs 2023-03-09T18:29:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:29:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12671.111404 secs 2023-03-09T18:29:23.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:29:22 smithi122 ceph-mon[93869]: pgmap v7272: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T18:29:23.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:29:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12673.412481 secs 2023-03-09T18:29:23.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:29:22 smithi181 ceph-mon[94189]: pgmap v7272: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T18:29:23.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:29:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12673.412481 secs 2023-03-09T18:29:23.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:29:22 smithi038 ceph-mon[92422]: pgmap v7272: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-09T18:29:23.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:29:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12673.412481 secs 2023-03-09T18:29:23.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13135.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:29:25.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:29:24 smithi122 ceph-mon[93869]: pgmap v7273: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 23 KiB/s wr, 6 op/s 2023-03-09T18:29:25.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:29:24 smithi181 ceph-mon[94189]: pgmap v7273: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 23 KiB/s wr, 6 op/s 2023-03-09T18:29:25.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:29:24 smithi038 ceph-mon[92422]: pgmap v7273: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 23 KiB/s wr, 6 op/s 2023-03-09T18:29:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:29:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12676.111483 secs 2023-03-09T18:29:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:29:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12676.111483 secs 2023-03-09T18:29:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:29:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12676.111483 secs 2023-03-09T18:29:27.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:29:26 smithi122 ceph-mon[93869]: pgmap v7274: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 8 op/s 2023-03-09T18:29:27.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:29:26 smithi181 ceph-mon[94189]: pgmap v7274: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 8 op/s 2023-03-09T18:29:27.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:29:26 smithi038 ceph-mon[92422]: pgmap v7274: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 8 op/s 2023-03-09T18:29:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:29:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12678.412600 secs 2023-03-09T18:29:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:29:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12678.412600 secs 2023-03-09T18:29:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:29:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12678.412600 secs 2023-03-09T18:29:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13140.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:29:29.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:29:28 smithi122 ceph-mon[93869]: pgmap v7275: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 62 KiB/s wr, 7 op/s 2023-03-09T18:29:29.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:29:28 smithi181 ceph-mon[94189]: pgmap v7275: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 62 KiB/s wr, 7 op/s 2023-03-09T18:29:29.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:29:28 smithi038 ceph-mon[92422]: pgmap v7275: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 62 KiB/s wr, 7 op/s 2023-03-09T18:29:30.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:29:29 smithi122 ceph-mon[93869]: pgmap v7276: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 62 KiB/s wr, 7 op/s 2023-03-09T18:29:30.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:29:29 smithi181 ceph-mon[94189]: pgmap v7276: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 62 KiB/s wr, 7 op/s 2023-03-09T18:29:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:29:29 smithi038 ceph-mon[92422]: pgmap v7276: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 62 KiB/s wr, 7 op/s 2023-03-09T18:29:31.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:29:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12681.111542 secs 2023-03-09T18:29:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:29:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12681.111542 secs 2023-03-09T18:29:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:29:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12681.111542 secs 2023-03-09T18:29:33.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:29:32 smithi122 ceph-mon[93869]: pgmap v7277: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 98 KiB/s wr, 10 op/s 2023-03-09T18:29:33.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:29:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12683.412716 secs 2023-03-09T18:29:33.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:29:32 smithi181 ceph-mon[94189]: pgmap v7277: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 98 KiB/s wr, 10 op/s 2023-03-09T18:29:33.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:29:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12683.412716 secs 2023-03-09T18:29:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:29:32 smithi038 ceph-mon[92422]: pgmap v7277: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 98 KiB/s wr, 10 op/s 2023-03-09T18:29:33.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:29:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12683.412716 secs 2023-03-09T18:29:33.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13145.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:29:35.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:29:34 smithi122 ceph-mon[93869]: pgmap v7278: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 76 KiB/s wr, 8 op/s 2023-03-09T18:29:35.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:29:34 smithi181 ceph-mon[94189]: pgmap v7278: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 76 KiB/s wr, 8 op/s 2023-03-09T18:29:35.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:29:34 smithi038 ceph-mon[92422]: pgmap v7278: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 76 KiB/s wr, 8 op/s 2023-03-09T18:29:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:29:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12686.111603 secs 2023-03-09T18:29:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:29:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12686.111603 secs 2023-03-09T18:29:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:29:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12686.111603 secs 2023-03-09T18:29:37.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:29:36 smithi122 ceph-mon[93869]: pgmap v7279: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 87 KiB/s wr, 9 op/s 2023-03-09T18:29:37.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:29:36 smithi181 ceph-mon[94189]: pgmap v7279: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 87 KiB/s wr, 9 op/s 2023-03-09T18:29:37.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:29:36 smithi038 ceph-mon[92422]: pgmap v7279: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 87 KiB/s wr, 9 op/s 2023-03-09T18:29:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:29:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12688.412859 secs 2023-03-09T18:29:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:29:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12688.412859 secs 2023-03-09T18:29:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:29:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12688.412859 secs 2023-03-09T18:29:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13150.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:29:39.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:29:38 smithi122 ceph-mon[93869]: pgmap v7280: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 119 KiB/s wr, 9 op/s 2023-03-09T18:29:39.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:29:38 smithi181 ceph-mon[94189]: pgmap v7280: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 119 KiB/s wr, 9 op/s 2023-03-09T18:29:39.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:29:38 smithi038 ceph-mon[92422]: pgmap v7280: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 119 KiB/s wr, 9 op/s 2023-03-09T18:29:40.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:29:39 smithi122 ceph-mon[93869]: pgmap v7281: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 110 KiB/s wr, 8 op/s 2023-03-09T18:29:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:29:39 smithi181 ceph-mon[94189]: pgmap v7281: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 110 KiB/s wr, 8 op/s 2023-03-09T18:29:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:29:39 smithi038 ceph-mon[92422]: pgmap v7281: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 110 KiB/s wr, 8 op/s 2023-03-09T18:29:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:29:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12691.111661 secs 2023-03-09T18:29:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:29:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12691.111661 secs 2023-03-09T18:29:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:29:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12691.111661 secs 2023-03-09T18:29:43.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:29:42 smithi122 ceph-mon[93869]: pgmap v7282: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 196 KiB/s wr, 11 op/s 2023-03-09T18:29:43.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:29:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12693.412998 secs 2023-03-09T18:29:43.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:29:42 smithi181 ceph-mon[94189]: pgmap v7282: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 196 KiB/s wr, 11 op/s 2023-03-09T18:29:43.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:29:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12693.412998 secs 2023-03-09T18:29:43.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:29:42 smithi038 ceph-mon[92422]: pgmap v7282: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 196 KiB/s wr, 11 op/s 2023-03-09T18:29:43.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:29:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12693.412998 secs 2023-03-09T18:29:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13155.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:29:45.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:29:44 smithi122 ceph-mon[93869]: pgmap v7283: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 160 KiB/s wr, 9 op/s 2023-03-09T18:29:45.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:29:44 smithi181 ceph-mon[94189]: pgmap v7283: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 160 KiB/s wr, 9 op/s 2023-03-09T18:29:45.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:29:44 smithi038 ceph-mon[92422]: pgmap v7283: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 160 KiB/s wr, 9 op/s 2023-03-09T18:29:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:29:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12696.111746 secs 2023-03-09T18:29:46.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:29:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12696.111746 secs 2023-03-09T18:29:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:29:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12696.111746 secs 2023-03-09T18:29:47.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:29:46 smithi122 ceph-mon[93869]: pgmap v7284: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 170 KiB/s wr, 11 op/s 2023-03-09T18:29:47.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:29:46 smithi181 ceph-mon[94189]: pgmap v7284: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 170 KiB/s wr, 11 op/s 2023-03-09T18:29:47.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:29:46 smithi038 ceph-mon[92422]: pgmap v7284: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 170 KiB/s wr, 11 op/s 2023-03-09T18:29:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:29:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12698.413096 secs 2023-03-09T18:29:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:29:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12698.413096 secs 2023-03-09T18:29:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:29:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12698.413096 secs 2023-03-09T18:29:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13160.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:29:49.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:29:48 smithi122 ceph-mon[93869]: pgmap v7285: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 285 KiB/s wr, 11 op/s 2023-03-09T18:29:49.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:29:48 smithi181 ceph-mon[94189]: pgmap v7285: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 285 KiB/s wr, 11 op/s 2023-03-09T18:29:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:29:48 smithi038 ceph-mon[92422]: pgmap v7285: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 285 KiB/s wr, 11 op/s 2023-03-09T18:29:50.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:29:49 smithi122 ceph-mon[93869]: pgmap v7286: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 223 KiB/s wr, 9 op/s 2023-03-09T18:29:50.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:29:49 smithi181 ceph-mon[94189]: pgmap v7286: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 223 KiB/s wr, 9 op/s 2023-03-09T18:29:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:29:49 smithi038 ceph-mon[92422]: pgmap v7286: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 223 KiB/s wr, 9 op/s 2023-03-09T18:29:51.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:29:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12701.111821 secs 2023-03-09T18:29:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:29:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12701.111821 secs 2023-03-09T18:29:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:29:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12701.111821 secs 2023-03-09T18:29:53.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:29:52 smithi122 ceph-mon[93869]: pgmap v7287: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 267 KiB/s wr, 12 op/s 2023-03-09T18:29:53.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:29:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12703.413212 secs 2023-03-09T18:29:53.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:29:52 smithi181 ceph-mon[94189]: pgmap v7287: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 267 KiB/s wr, 12 op/s 2023-03-09T18:29:53.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:29:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12703.413212 secs 2023-03-09T18:29:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:29:52 smithi038 ceph-mon[92422]: pgmap v7287: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 267 KiB/s wr, 12 op/s 2023-03-09T18:29:53.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:29:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12703.413212 secs 2023-03-09T18:29:53.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13165.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:29:55.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:29:54 smithi122 ceph-mon[93869]: pgmap v7288: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 193 KiB/s wr, 10 op/s 2023-03-09T18:29:55.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:29:54 smithi181 ceph-mon[94189]: pgmap v7288: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 193 KiB/s wr, 10 op/s 2023-03-09T18:29:55.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:29:54 smithi038 ceph-mon[92422]: pgmap v7288: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 193 KiB/s wr, 10 op/s 2023-03-09T18:29:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:29:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12706.111880 secs 2023-03-09T18:29:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:29:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12706.111880 secs 2023-03-09T18:29:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:29:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12706.111880 secs 2023-03-09T18:29:57.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:29:56 smithi122 ceph-mon[93869]: pgmap v7289: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 203 KiB/s wr, 11 op/s 2023-03-09T18:29:57.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:29:56 smithi181 ceph-mon[94189]: pgmap v7289: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 203 KiB/s wr, 11 op/s 2023-03-09T18:29:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:29:56 smithi038 ceph-mon[92422]: pgmap v7289: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 203 KiB/s wr, 11 op/s 2023-03-09T18:29:58.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:29:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12708.413329 secs 2023-03-09T18:29:58.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:29:57 smithi122 ceph-mon[93869]: pgmap v7290: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 329 KiB/s rd, 328 KiB/s wr, 11 op/s 2023-03-09T18:29:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:29:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12708.413329 secs 2023-03-09T18:29:58.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:29:57 smithi181 ceph-mon[94189]: pgmap v7290: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 329 KiB/s rd, 328 KiB/s wr, 11 op/s 2023-03-09T18:29:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:29:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12708.413329 secs 2023-03-09T18:29:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:29:57 smithi038 ceph-mon[92422]: pgmap v7290: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 329 KiB/s rd, 328 KiB/s wr, 11 op/s 2023-03-09T18:29:58.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13170.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:30:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:30:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12711.111941 secs 2023-03-09T18:30:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:30:00 smithi122 ceph-mon[93869]: pgmap v7291: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 202 KiB/s wr, 9 op/s 2023-03-09T18:30:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:30:00 smithi122 ceph-mon[93869]: overall HEALTH_WARN 2 MDSs report slow requests 2023-03-09T18:30:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:30:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12711.111941 secs 2023-03-09T18:30:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:30:00 smithi181 ceph-mon[94189]: pgmap v7291: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 202 KiB/s wr, 9 op/s 2023-03-09T18:30:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:30:00 smithi181 ceph-mon[94189]: overall HEALTH_WARN 2 MDSs report slow requests 2023-03-09T18:30:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:30:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12711.111941 secs 2023-03-09T18:30:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:30:00 smithi038 ceph-mon[92422]: pgmap v7291: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 202 KiB/s wr, 9 op/s 2023-03-09T18:30:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:30:00 smithi038 ceph-mon[92422]: overall HEALTH_WARN 2 MDSs report slow requests 2023-03-09T18:30:03.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:30:02 smithi122 ceph-mon[93869]: pgmap v7292: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 203 KiB/s wr, 12 op/s 2023-03-09T18:30:03.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:30:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12713.413470 secs 2023-03-09T18:30:03.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:30:02 smithi181 ceph-mon[94189]: pgmap v7292: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 203 KiB/s wr, 12 op/s 2023-03-09T18:30:03.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:30:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12713.413470 secs 2023-03-09T18:30:03.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:30:02 smithi038 ceph-mon[92422]: pgmap v7292: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 203 KiB/s wr, 12 op/s 2023-03-09T18:30:03.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:30:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12713.413470 secs 2023-03-09T18:30:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13175.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:30:05.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:30:04 smithi122 ceph-mon[93869]: pgmap v7293: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 243 KiB/s wr, 22 op/s 2023-03-09T18:30:05.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:30:04 smithi181 ceph-mon[94189]: pgmap v7293: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 243 KiB/s wr, 22 op/s 2023-03-09T18:30:05.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:30:04 smithi038 ceph-mon[92422]: pgmap v7293: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 243 KiB/s wr, 22 op/s 2023-03-09T18:30:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:30:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12716.111995 secs 2023-03-09T18:30:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:30:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12716.111995 secs 2023-03-09T18:30:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:30:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12716.111995 secs 2023-03-09T18:30:07.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:30:06 smithi122 ceph-mon[93869]: pgmap v7294: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 232 KiB/s wr, 23 op/s 2023-03-09T18:30:07.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:30:06 smithi181 ceph-mon[94189]: pgmap v7294: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 232 KiB/s wr, 23 op/s 2023-03-09T18:30:07.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:30:06 smithi038 ceph-mon[92422]: pgmap v7294: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 232 KiB/s wr, 23 op/s 2023-03-09T18:30:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:30:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12718.413586 secs 2023-03-09T18:30:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:30:07 smithi122 ceph-mon[93869]: pgmap v7295: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 244 KiB/s wr, 24 op/s 2023-03-09T18:30:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:30:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12718.413586 secs 2023-03-09T18:30:08.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:30:07 smithi181 ceph-mon[94189]: pgmap v7295: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 244 KiB/s wr, 24 op/s 2023-03-09T18:30:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:30:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12718.413586 secs 2023-03-09T18:30:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:30:07 smithi038 ceph-mon[92422]: pgmap v7295: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 244 KiB/s wr, 24 op/s 2023-03-09T18:30:08.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13180.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:30:10.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:30:09 smithi122 ceph-mon[93869]: osdmap e669: 12 total, 12 up, 12 in 2023-03-09T18:30:10.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:30:09 smithi122 ceph-mon[93869]: pgmap v7297: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 130 KiB/s wr, 26 op/s 2023-03-09T18:30:10.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:30:09 smithi181 ceph-mon[94189]: osdmap e669: 12 total, 12 up, 12 in 2023-03-09T18:30:10.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:30:09 smithi181 ceph-mon[94189]: pgmap v7297: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 130 KiB/s wr, 26 op/s 2023-03-09T18:30:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:30:09 smithi038 ceph-mon[92422]: osdmap e669: 12 total, 12 up, 12 in 2023-03-09T18:30:10.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:30:09 smithi038 ceph-mon[92422]: pgmap v7297: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 130 KiB/s wr, 26 op/s 2023-03-09T18:30:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:30:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12721.112053 secs 2023-03-09T18:30:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:30:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12721.112053 secs 2023-03-09T18:30:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:30:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12721.112053 secs 2023-03-09T18:30:13.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:30:12 smithi122 ceph-mon[93869]: pgmap v7298: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 141 KiB/s wr, 26 op/s 2023-03-09T18:30:13.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:30:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12723.413700 secs 2023-03-09T18:30:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:30:12 smithi181 ceph-mon[94189]: pgmap v7298: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 141 KiB/s wr, 26 op/s 2023-03-09T18:30:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:30:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12723.413700 secs 2023-03-09T18:30:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:30:12 smithi038 ceph-mon[92422]: pgmap v7298: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 141 KiB/s wr, 26 op/s 2023-03-09T18:30:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:30:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12723.413700 secs 2023-03-09T18:30:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13185.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:30:15.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:30:14 smithi122 ceph-mon[93869]: pgmap v7299: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 71 KiB/s wr, 10 op/s 2023-03-09T18:30:15.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:30:14 smithi181 ceph-mon[94189]: pgmap v7299: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 71 KiB/s wr, 10 op/s 2023-03-09T18:30:15.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:30:14 smithi038 ceph-mon[92422]: pgmap v7299: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 71 KiB/s wr, 10 op/s 2023-03-09T18:30:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:30:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12726.112116 secs 2023-03-09T18:30:16.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:30:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12726.112116 secs 2023-03-09T18:30:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:30:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12726.112116 secs 2023-03-09T18:30:17.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:30:16 smithi122 ceph-mon[93869]: pgmap v7300: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 70 KiB/s wr, 10 op/s 2023-03-09T18:30:17.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:30:16 smithi122 ceph-mon[93869]: osdmap e670: 12 total, 12 up, 12 in 2023-03-09T18:30:17.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:30:16 smithi181 ceph-mon[94189]: pgmap v7300: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 70 KiB/s wr, 10 op/s 2023-03-09T18:30:17.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:30:16 smithi181 ceph-mon[94189]: osdmap e670: 12 total, 12 up, 12 in 2023-03-09T18:30:17.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:30:16 smithi038 ceph-mon[92422]: pgmap v7300: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 70 KiB/s wr, 10 op/s 2023-03-09T18:30:17.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:30:16 smithi038 ceph-mon[92422]: osdmap e670: 12 total, 12 up, 12 in 2023-03-09T18:30:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:30:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12728.413823 secs 2023-03-09T18:30:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:30:17 smithi122 ceph-mon[93869]: pgmap v7302: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 51 KiB/s wr, 9 op/s 2023-03-09T18:30:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:30:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12728.413823 secs 2023-03-09T18:30:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:30:17 smithi181 ceph-mon[94189]: pgmap v7302: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 51 KiB/s wr, 9 op/s 2023-03-09T18:30:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:30:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12728.413823 secs 2023-03-09T18:30:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:30:17 smithi038 ceph-mon[92422]: pgmap v7302: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 51 KiB/s wr, 9 op/s 2023-03-09T18:30:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13190.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:30:19.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:30:18 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:30:19.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:30:18 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:30:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:30:18 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:30:20.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:30:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:30:20.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:30:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:30:20.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:30:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:30:20.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:30:19 smithi122 ceph-mon[93869]: pgmap v7303: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 45 KiB/s wr, 8 op/s 2023-03-09T18:30:20.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:30:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:30:20.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:30:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:30:20.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:30:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:30:20.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:30:19 smithi181 ceph-mon[94189]: pgmap v7303: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 45 KiB/s wr, 8 op/s 2023-03-09T18:30:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:30:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:30:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:30:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:30:20.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:30:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:30:20.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:30:19 smithi038 ceph-mon[92422]: pgmap v7303: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 45 KiB/s wr, 8 op/s 2023-03-09T18:30:21.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:30:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12731.112175 secs 2023-03-09T18:30:21.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:30:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12731.112175 secs 2023-03-09T18:30:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:30:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12731.112175 secs 2023-03-09T18:30:23.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:30:22 smithi122 ceph-mon[93869]: pgmap v7304: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 33 KiB/s wr, 7 op/s 2023-03-09T18:30:23.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:30:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12733.413965 secs 2023-03-09T18:30:23.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:30:22 smithi181 ceph-mon[94189]: pgmap v7304: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 33 KiB/s wr, 7 op/s 2023-03-09T18:30:23.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:30:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12733.413965 secs 2023-03-09T18:30:23.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:30:22 smithi038 ceph-mon[92422]: pgmap v7304: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 33 KiB/s wr, 7 op/s 2023-03-09T18:30:23.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:30:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12733.413965 secs 2023-03-09T18:30:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13195.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:30:25.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:30:24 smithi122 ceph-mon[93869]: pgmap v7305: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2023-03-09T18:30:25.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:30:24 smithi181 ceph-mon[94189]: pgmap v7305: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2023-03-09T18:30:25.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:30:24 smithi038 ceph-mon[92422]: pgmap v7305: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2023-03-09T18:30:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:30:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12736.112233 secs 2023-03-09T18:30:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:30:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12736.112233 secs 2023-03-09T18:30:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:30:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12736.112233 secs 2023-03-09T18:30:27.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:30:26 smithi122 ceph-mon[93869]: pgmap v7306: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.9 KiB/s wr, 7 op/s 2023-03-09T18:30:27.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:30:26 smithi181 ceph-mon[94189]: pgmap v7306: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.9 KiB/s wr, 7 op/s 2023-03-09T18:30:27.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:30:26 smithi038 ceph-mon[92422]: pgmap v7306: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.9 KiB/s wr, 7 op/s 2023-03-09T18:30:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:30:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12738.414085 secs 2023-03-09T18:30:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:30:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12738.414085 secs 2023-03-09T18:30:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:30:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12738.414085 secs 2023-03-09T18:30:28.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13200.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:30:29.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:30:28 smithi122 ceph-mon[93869]: pgmap v7307: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T18:30:29.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:30:28 smithi181 ceph-mon[94189]: pgmap v7307: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T18:30:29.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:30:28 smithi038 ceph-mon[92422]: pgmap v7307: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T18:30:30.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:30:29 smithi122 ceph-mon[93869]: pgmap v7308: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 40 KiB/s wr, 6 op/s 2023-03-09T18:30:30.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:30:29 smithi181 ceph-mon[94189]: pgmap v7308: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 40 KiB/s wr, 6 op/s 2023-03-09T18:30:30.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:30:29 smithi038 ceph-mon[92422]: pgmap v7308: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 40 KiB/s wr, 6 op/s 2023-03-09T18:30:31.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:30:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12741.112302 secs 2023-03-09T18:30:31.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:30:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12741.112302 secs 2023-03-09T18:30:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:30:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12741.112302 secs 2023-03-09T18:30:33.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:30:32 smithi122 ceph-mon[93869]: pgmap v7309: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 40 KiB/s wr, 8 op/s 2023-03-09T18:30:33.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:30:32 smithi181 ceph-mon[94189]: pgmap v7309: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 40 KiB/s wr, 8 op/s 2023-03-09T18:30:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:30:32 smithi038 ceph-mon[92422]: pgmap v7309: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 40 KiB/s wr, 8 op/s 2023-03-09T18:30:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13205.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:30:35.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:30:34 smithi122 ceph-mon[93869]: pgmap v7310: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 40 KiB/s wr, 7 op/s 2023-03-09T18:30:35.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:30:34 smithi181 ceph-mon[94189]: pgmap v7310: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 40 KiB/s wr, 7 op/s 2023-03-09T18:30:35.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:30:34 smithi038 ceph-mon[92422]: pgmap v7310: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 40 KiB/s wr, 7 op/s 2023-03-09T18:30:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:30:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12746.112390 secs 2023-03-09T18:30:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:30:35 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:30:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:30:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12746.112390 secs 2023-03-09T18:30:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:30:35 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:30:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:30:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12746.112390 secs 2023-03-09T18:30:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:30:35 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:30:37.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:30:36 smithi122 ceph-mon[93869]: pgmap v7311: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 51 KiB/s wr, 8 op/s 2023-03-09T18:30:37.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:30:36 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:30:37.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:30:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:30:37.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:30:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:30:37.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:30:36 smithi181 ceph-mon[94189]: pgmap v7311: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 51 KiB/s wr, 8 op/s 2023-03-09T18:30:37.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:30:36 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:30:37.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:30:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:30:37.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:30:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:30:37.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:30:36 smithi038 ceph-mon[92422]: pgmap v7311: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 51 KiB/s wr, 8 op/s 2023-03-09T18:30:37.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:30:36 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:30:37.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:30:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:30:37.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:30:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:30:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:30:37 smithi122 ceph-mon[93869]: pgmap v7312: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 118 KiB/s wr, 8 op/s 2023-03-09T18:30:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:30:37 smithi181 ceph-mon[94189]: pgmap v7312: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 118 KiB/s wr, 8 op/s 2023-03-09T18:30:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:30:37 smithi038 ceph-mon[92422]: pgmap v7312: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 118 KiB/s wr, 8 op/s 2023-03-09T18:30:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13210.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:30:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:30:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12751.112465 secs 2023-03-09T18:30:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:30:40 smithi122 ceph-mon[93869]: pgmap v7313: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 160 KiB/s wr, 8 op/s 2023-03-09T18:30:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:30:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12751.112465 secs 2023-03-09T18:30:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:30:40 smithi181 ceph-mon[94189]: pgmap v7313: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 160 KiB/s wr, 8 op/s 2023-03-09T18:30:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:30:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12751.112465 secs 2023-03-09T18:30:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:30:40 smithi038 ceph-mon[92422]: pgmap v7313: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 160 KiB/s wr, 8 op/s 2023-03-09T18:30:43.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:30:42 smithi122 ceph-mon[93869]: pgmap v7314: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 170 KiB/s wr, 10 op/s 2023-03-09T18:30:43.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:30:42 smithi181 ceph-mon[94189]: pgmap v7314: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 170 KiB/s wr, 10 op/s 2023-03-09T18:30:43.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:30:42 smithi038 ceph-mon[92422]: pgmap v7314: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 170 KiB/s wr, 10 op/s 2023-03-09T18:30:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13215.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:30:45.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:30:44 smithi122 ceph-mon[93869]: pgmap v7315: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 170 KiB/s wr, 10 op/s 2023-03-09T18:30:45.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:30:44 smithi181 ceph-mon[94189]: pgmap v7315: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 170 KiB/s wr, 10 op/s 2023-03-09T18:30:45.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:30:44 smithi038 ceph-mon[92422]: pgmap v7315: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 170 KiB/s wr, 10 op/s 2023-03-09T18:30:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:30:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12756.112526 secs 2023-03-09T18:30:46.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:30:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12756.112526 secs 2023-03-09T18:30:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:30:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12756.112526 secs 2023-03-09T18:30:47.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:30:46 smithi122 ceph-mon[93869]: pgmap v7316: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 252 KiB/s wr, 12 op/s 2023-03-09T18:30:47.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:30:46 smithi181 ceph-mon[94189]: pgmap v7316: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 252 KiB/s wr, 12 op/s 2023-03-09T18:30:47.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:30:46 smithi038 ceph-mon[92422]: pgmap v7316: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 252 KiB/s wr, 12 op/s 2023-03-09T18:30:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:30:47 smithi122 ceph-mon[93869]: pgmap v7317: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 242 KiB/s wr, 11 op/s 2023-03-09T18:30:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:30:47 smithi181 ceph-mon[94189]: pgmap v7317: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 242 KiB/s wr, 11 op/s 2023-03-09T18:30:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:30:47 smithi038 ceph-mon[92422]: pgmap v7317: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 242 KiB/s wr, 11 op/s 2023-03-09T18:30:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13220.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:30:51.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:30:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12761.112601 secs 2023-03-09T18:30:51.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:30:50 smithi122 ceph-mon[93869]: pgmap v7318: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 246 KiB/s wr, 10 op/s 2023-03-09T18:30:51.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:30:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12761.112601 secs 2023-03-09T18:30:51.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:30:50 smithi181 ceph-mon[94189]: pgmap v7318: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 246 KiB/s wr, 10 op/s 2023-03-09T18:30:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:30:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12761.112601 secs 2023-03-09T18:30:51.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:30:50 smithi038 ceph-mon[92422]: pgmap v7318: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 246 KiB/s wr, 10 op/s 2023-03-09T18:30:53.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:30:52 smithi122 ceph-mon[93869]: pgmap v7319: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 165 KiB/s wr, 12 op/s 2023-03-09T18:30:53.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:30:52 smithi181 ceph-mon[94189]: pgmap v7319: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 165 KiB/s wr, 12 op/s 2023-03-09T18:30:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:30:52 smithi038 ceph-mon[92422]: pgmap v7319: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 165 KiB/s wr, 12 op/s 2023-03-09T18:30:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13225.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:30:55.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:30:54 smithi122 ceph-mon[93869]: pgmap v7320: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 155 KiB/s wr, 11 op/s 2023-03-09T18:30:55.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:30:54 smithi181 ceph-mon[94189]: pgmap v7320: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 155 KiB/s wr, 11 op/s 2023-03-09T18:30:55.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:30:54 smithi038 ceph-mon[92422]: pgmap v7320: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 155 KiB/s wr, 11 op/s 2023-03-09T18:30:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:30:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12766.112659 secs 2023-03-09T18:30:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:30:55 smithi122 ceph-mon[93869]: pgmap v7321: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 260 KiB/s wr, 12 op/s 2023-03-09T18:30:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:30:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12766.112659 secs 2023-03-09T18:30:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:30:55 smithi181 ceph-mon[94189]: pgmap v7321: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 260 KiB/s wr, 12 op/s 2023-03-09T18:30:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:30:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12766.112659 secs 2023-03-09T18:30:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:30:55 smithi038 ceph-mon[92422]: pgmap v7321: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 260 KiB/s wr, 12 op/s 2023-03-09T18:30:58.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:30:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12768.414784 secs 2023-03-09T18:30:58.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:30:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12768.414784 secs 2023-03-09T18:30:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:30:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12768.414784 secs 2023-03-09T18:30:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13230.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:30:59.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:30:58 smithi122 ceph-mon[93869]: pgmap v7322: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 181 KiB/s wr, 10 op/s 2023-03-09T18:30:59.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:30:58 smithi181 ceph-mon[94189]: pgmap v7322: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 181 KiB/s wr, 10 op/s 2023-03-09T18:30:59.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:30:58 smithi038 ceph-mon[92422]: pgmap v7322: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 181 KiB/s wr, 10 op/s 2023-03-09T18:31:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:30:59 smithi122 ceph-mon[93869]: pgmap v7323: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 214 KiB/s wr, 10 op/s 2023-03-09T18:31:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:30:59 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:31:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:30:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:31:00.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:30:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:31:00.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:30:59 smithi181 ceph-mon[94189]: pgmap v7323: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 214 KiB/s wr, 10 op/s 2023-03-09T18:31:00.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:30:59 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:31:00.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:30:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:31:00.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:30:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:31:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:30:59 smithi038 ceph-mon[92422]: pgmap v7323: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 214 KiB/s wr, 10 op/s 2023-03-09T18:31:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:30:59 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:31:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:30:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:31:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:30:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:31:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:31:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12771.112719 secs 2023-03-09T18:31:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:31:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12771.112719 secs 2023-03-09T18:31:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:31:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12771.112719 secs 2023-03-09T18:31:03.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:31:02 smithi122 ceph-mon[93869]: pgmap v7324: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 206 KiB/s wr, 14 op/s 2023-03-09T18:31:03.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:31:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12773.414926 secs 2023-03-09T18:31:03.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:31:02 smithi181 ceph-mon[94189]: pgmap v7324: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 206 KiB/s wr, 14 op/s 2023-03-09T18:31:03.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:31:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12773.414926 secs 2023-03-09T18:31:03.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:31:02 smithi038 ceph-mon[92422]: pgmap v7324: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 206 KiB/s wr, 14 op/s 2023-03-09T18:31:03.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:31:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12773.414926 secs 2023-03-09T18:31:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13235.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:31:05.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:31:04 smithi122 ceph-mon[93869]: pgmap v7325: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 227 KiB/s wr, 24 op/s 2023-03-09T18:31:05.269 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:31:04 smithi038 ceph-mon[92422]: pgmap v7325: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 227 KiB/s wr, 24 op/s 2023-03-09T18:31:05.270 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:31:04 smithi181 ceph-mon[94189]: pgmap v7325: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 227 KiB/s wr, 24 op/s 2023-03-09T18:31:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:31:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12776.112778 secs 2023-03-09T18:31:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:31:05 smithi122 ceph-mon[93869]: osdmap e671: 12 total, 12 up, 12 in 2023-03-09T18:31:06.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:31:05 smithi122 ceph-mon[93869]: pgmap v7327: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 146 KiB/s wr, 26 op/s 2023-03-09T18:31:06.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:31:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12776.112778 secs 2023-03-09T18:31:06.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:31:05 smithi181 ceph-mon[94189]: osdmap e671: 12 total, 12 up, 12 in 2023-03-09T18:31:06.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:31:05 smithi181 ceph-mon[94189]: pgmap v7327: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 146 KiB/s wr, 26 op/s 2023-03-09T18:31:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:31:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12776.112778 secs 2023-03-09T18:31:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:31:05 smithi038 ceph-mon[92422]: osdmap e671: 12 total, 12 up, 12 in 2023-03-09T18:31:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:31:05 smithi038 ceph-mon[92422]: pgmap v7327: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 146 KiB/s wr, 26 op/s 2023-03-09T18:31:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:31:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12778.415046 secs 2023-03-09T18:31:08.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:31:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12778.415046 secs 2023-03-09T18:31:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:31:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12778.415046 secs 2023-03-09T18:31:08.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13240.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:31:09.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:31:08 smithi122 ceph-mon[93869]: pgmap v7328: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 206 KiB/s wr, 27 op/s 2023-03-09T18:31:09.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:31:08 smithi181 ceph-mon[94189]: pgmap v7328: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 206 KiB/s wr, 27 op/s 2023-03-09T18:31:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:31:08 smithi038 ceph-mon[92422]: pgmap v7328: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 206 KiB/s wr, 27 op/s 2023-03-09T18:31:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:31:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12781.112837 secs 2023-03-09T18:31:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:31:10 smithi122 ceph-mon[93869]: pgmap v7329: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 166 KiB/s wr, 26 op/s 2023-03-09T18:31:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:31:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12781.112837 secs 2023-03-09T18:31:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:31:10 smithi181 ceph-mon[94189]: pgmap v7329: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 166 KiB/s wr, 26 op/s 2023-03-09T18:31:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:31:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12781.112837 secs 2023-03-09T18:31:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:31:10 smithi038 ceph-mon[92422]: pgmap v7329: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 166 KiB/s wr, 26 op/s 2023-03-09T18:31:13.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:31:12 smithi122 ceph-mon[93869]: pgmap v7330: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 135 KiB/s wr, 23 op/s 2023-03-09T18:31:13.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:31:12 smithi122 ceph-mon[93869]: osdmap e672: 12 total, 12 up, 12 in 2023-03-09T18:31:13.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:31:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12783.415212 secs 2023-03-09T18:31:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:31:12 smithi181 ceph-mon[94189]: pgmap v7330: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 135 KiB/s wr, 23 op/s 2023-03-09T18:31:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:31:12 smithi181 ceph-mon[94189]: osdmap e672: 12 total, 12 up, 12 in 2023-03-09T18:31:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:31:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12783.415212 secs 2023-03-09T18:31:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:31:12 smithi038 ceph-mon[92422]: pgmap v7330: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 135 KiB/s wr, 23 op/s 2023-03-09T18:31:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:31:12 smithi038 ceph-mon[92422]: osdmap e672: 12 total, 12 up, 12 in 2023-03-09T18:31:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:31:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12783.415212 secs 2023-03-09T18:31:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13245.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:31:15.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:31:14 smithi122 ceph-mon[93869]: pgmap v7332: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 122 KiB/s wr, 9 op/s 2023-03-09T18:31:15.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:31:14 smithi181 ceph-mon[94189]: pgmap v7332: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 122 KiB/s wr, 9 op/s 2023-03-09T18:31:15.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:31:14 smithi038 ceph-mon[92422]: pgmap v7332: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 122 KiB/s wr, 9 op/s 2023-03-09T18:31:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:31:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12786.112897 secs 2023-03-09T18:31:16.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:31:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12786.112897 secs 2023-03-09T18:31:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:31:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12786.112897 secs 2023-03-09T18:31:17.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:31:16 smithi122 ceph-mon[93869]: pgmap v7333: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 121 KiB/s wr, 10 op/s 2023-03-09T18:31:17.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:31:16 smithi181 ceph-mon[94189]: pgmap v7333: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 121 KiB/s wr, 10 op/s 2023-03-09T18:31:17.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:31:16 smithi038 ceph-mon[92422]: pgmap v7333: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 121 KiB/s wr, 10 op/s 2023-03-09T18:31:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:31:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12788.415280 secs 2023-03-09T18:31:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:31:17 smithi122 ceph-mon[93869]: pgmap v7334: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 59 KiB/s wr, 8 op/s 2023-03-09T18:31:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:31:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12788.415280 secs 2023-03-09T18:31:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:31:17 smithi181 ceph-mon[94189]: pgmap v7334: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 59 KiB/s wr, 8 op/s 2023-03-09T18:31:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:31:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12788.415280 secs 2023-03-09T18:31:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:31:17 smithi038 ceph-mon[92422]: pgmap v7334: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 59 KiB/s wr, 8 op/s 2023-03-09T18:31:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13250.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:31:19.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:31:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:31:19.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:31:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:31:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:31:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:31:20.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:31:20 smithi122 ceph-mon[93869]: pgmap v7335: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 59 KiB/s wr, 8 op/s 2023-03-09T18:31:20.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:31:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:31:20.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:31:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:31:20.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:31:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:31:20.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:31:20 smithi181 ceph-mon[94189]: pgmap v7335: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 59 KiB/s wr, 8 op/s 2023-03-09T18:31:20.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:31:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:31:20.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:31:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:31:20.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:31:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:31:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:31:20 smithi038 ceph-mon[92422]: pgmap v7335: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 59 KiB/s wr, 8 op/s 2023-03-09T18:31:21.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:31:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:31:21.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:31:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:31:21.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:31:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:31:21.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:31:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12791.112967 secs 2023-03-09T18:31:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:31:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12791.112967 secs 2023-03-09T18:31:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:31:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12791.112967 secs 2023-03-09T18:31:23.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:31:22 smithi122 ceph-mon[93869]: pgmap v7336: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 46 KiB/s wr, 7 op/s 2023-03-09T18:31:23.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:31:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12793.415364 secs 2023-03-09T18:31:23.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:31:22 smithi181 ceph-mon[94189]: pgmap v7336: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 46 KiB/s wr, 7 op/s 2023-03-09T18:31:23.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:31:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12793.415364 secs 2023-03-09T18:31:23.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:31:22 smithi038 ceph-mon[92422]: pgmap v7336: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 46 KiB/s wr, 7 op/s 2023-03-09T18:31:23.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:31:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12793.415364 secs 2023-03-09T18:31:23.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13255.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:31:25.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:31:24 smithi122 ceph-mon[93869]: pgmap v7337: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 38 KiB/s wr, 6 op/s 2023-03-09T18:31:25.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:31:24 smithi181 ceph-mon[94189]: pgmap v7337: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 38 KiB/s wr, 6 op/s 2023-03-09T18:31:25.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:31:24 smithi038 ceph-mon[92422]: pgmap v7337: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 38 KiB/s wr, 6 op/s 2023-03-09T18:31:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:31:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12796.113028 secs 2023-03-09T18:31:26.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:31:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12796.113028 secs 2023-03-09T18:31:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:31:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12796.113028 secs 2023-03-09T18:31:27.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:31:26 smithi122 ceph-mon[93869]: pgmap v7338: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 79 KiB/s wr, 9 op/s 2023-03-09T18:31:27.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:31:26 smithi181 ceph-mon[94189]: pgmap v7338: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 79 KiB/s wr, 9 op/s 2023-03-09T18:31:27.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:31:26 smithi038 ceph-mon[92422]: pgmap v7338: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 79 KiB/s wr, 9 op/s 2023-03-09T18:31:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:31:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12798.415490 secs 2023-03-09T18:31:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:31:27 smithi122 ceph-mon[93869]: pgmap v7339: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 68 KiB/s wr, 7 op/s 2023-03-09T18:31:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:31:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12798.415490 secs 2023-03-09T18:31:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:31:27 smithi181 ceph-mon[94189]: pgmap v7339: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 68 KiB/s wr, 7 op/s 2023-03-09T18:31:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:31:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12798.415490 secs 2023-03-09T18:31:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:31:27 smithi038 ceph-mon[92422]: pgmap v7339: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 68 KiB/s wr, 7 op/s 2023-03-09T18:31:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13260.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:31:31.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:31:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12801.113087 secs 2023-03-09T18:31:31.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:31:30 smithi122 ceph-mon[93869]: pgmap v7340: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 67 KiB/s wr, 6 op/s 2023-03-09T18:31:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:31:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12801.113087 secs 2023-03-09T18:31:31.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:31:30 smithi181 ceph-mon[94189]: pgmap v7340: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 67 KiB/s wr, 6 op/s 2023-03-09T18:31:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:31:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12801.113087 secs 2023-03-09T18:31:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:31:30 smithi038 ceph-mon[92422]: pgmap v7340: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 67 KiB/s wr, 6 op/s 2023-03-09T18:31:33.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:31:32 smithi122 ceph-mon[93869]: pgmap v7341: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 104 KiB/s wr, 9 op/s 2023-03-09T18:31:33.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:31:32 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 12803.415630 secs 2023-03-09T18:31:33.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:31:32 smithi181 ceph-mon[94189]: pgmap v7341: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 104 KiB/s wr, 9 op/s 2023-03-09T18:31:33.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:31:32 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 12803.415630 secs 2023-03-09T18:31:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:31:32 smithi038 ceph-mon[92422]: pgmap v7341: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 104 KiB/s wr, 9 op/s 2023-03-09T18:31:33.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:31:32 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 12803.415630 secs 2023-03-09T18:31:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13265.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:31:34.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:31:33 smithi122 ceph-mon[93869]: pgmap v7342: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 78 KiB/s wr, 7 op/s 2023-03-09T18:31:34.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:31:33 smithi181 ceph-mon[94189]: pgmap v7342: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 78 KiB/s wr, 7 op/s 2023-03-09T18:31:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:31:33 smithi038 ceph-mon[92422]: pgmap v7342: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 78 KiB/s wr, 7 op/s 2023-03-09T18:31:35.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:31:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12806.113147 secs 2023-03-09T18:31:36.058 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:31:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12806.113147 secs 2023-03-09T18:31:36.058 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:31:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12806.113147 secs 2023-03-09T18:31:36.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:31:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:31:36.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:31:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:31:36.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:31:36 smithi122 ceph-mon[93869]: pgmap v7343: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 152 KiB/s wr, 10 op/s 2023-03-09T18:31:36.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:31:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:31:36.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:31:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:31:36.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:31:36 smithi181 ceph-mon[94189]: pgmap v7343: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 152 KiB/s wr, 10 op/s 2023-03-09T18:31:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:31:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:31:37.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:31:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:31:37.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:31:36 smithi038 ceph-mon[92422]: pgmap v7343: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 152 KiB/s wr, 10 op/s 2023-03-09T18:31:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:31:37 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 12808.415756 secs 2023-03-09T18:31:38.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:31:37 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 12808.415756 secs 2023-03-09T18:31:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:31:37 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 12808.415756 secs 2023-03-09T18:31:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13270.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:31:39.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:31:38 smithi122 ceph-mon[93869]: pgmap v7344: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 111 KiB/s wr, 7 op/s 2023-03-09T18:31:39.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:31:38 smithi181 ceph-mon[94189]: pgmap v7344: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 111 KiB/s wr, 7 op/s 2023-03-09T18:31:39.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:31:38 smithi038 ceph-mon[92422]: pgmap v7344: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 111 KiB/s wr, 7 op/s 2023-03-09T18:31:40.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:31:39 smithi122 ceph-mon[93869]: pgmap v7345: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 111 KiB/s wr, 7 op/s 2023-03-09T18:31:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:31:39 smithi181 ceph-mon[94189]: pgmap v7345: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 111 KiB/s wr, 7 op/s 2023-03-09T18:31:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:31:39 smithi038 ceph-mon[92422]: pgmap v7345: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 111 KiB/s wr, 7 op/s 2023-03-09T18:31:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:31:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12811.113210 secs 2023-03-09T18:31:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:31:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12811.113210 secs 2023-03-09T18:31:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:31:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12811.113210 secs 2023-03-09T18:31:43.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:31:42 smithi122 ceph-mon[93869]: pgmap v7346: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 175 KiB/s wr, 10 op/s 2023-03-09T18:31:43.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:31:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12813.415875 secs 2023-03-09T18:31:43.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:31:42 smithi181 ceph-mon[94189]: pgmap v7346: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 175 KiB/s wr, 10 op/s 2023-03-09T18:31:43.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:31:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12813.415875 secs 2023-03-09T18:31:43.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:31:42 smithi038 ceph-mon[92422]: pgmap v7346: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 175 KiB/s wr, 10 op/s 2023-03-09T18:31:43.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:31:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12813.415875 secs 2023-03-09T18:31:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13275.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:31:44.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:31:43 smithi122 ceph-mon[93869]: pgmap v7347: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 234 KiB/s wr, 9 op/s 2023-03-09T18:31:44.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:31:43 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:31:44.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:31:43 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:31:44.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:31:43 smithi181 ceph-mon[94189]: pgmap v7347: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 234 KiB/s wr, 9 op/s 2023-03-09T18:31:44.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:31:43 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:31:44.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:31:43 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:31:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:31:43 smithi038 ceph-mon[92422]: pgmap v7347: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 234 KiB/s wr, 9 op/s 2023-03-09T18:31:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:31:43 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:31:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:31:43 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:31:45.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:31:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12816.113269 secs 2023-03-09T18:31:45.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:31:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12816.113269 secs 2023-03-09T18:31:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:31:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12816.113269 secs 2023-03-09T18:31:46.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:31:46 smithi122 ceph-mon[93869]: pgmap v7348: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 245 KiB/s wr, 12 op/s 2023-03-09T18:31:46.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:31:46 smithi181 ceph-mon[94189]: pgmap v7348: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 245 KiB/s wr, 12 op/s 2023-03-09T18:31:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:31:46 smithi038 ceph-mon[92422]: pgmap v7348: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 245 KiB/s wr, 12 op/s 2023-03-09T18:31:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:31:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12818.415995 secs 2023-03-09T18:31:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:31:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12818.415995 secs 2023-03-09T18:31:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:31:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12818.415995 secs 2023-03-09T18:31:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13280.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:31:49.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:31:48 smithi122 ceph-mon[93869]: pgmap v7349: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 171 KiB/s wr, 9 op/s 2023-03-09T18:31:49.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:31:48 smithi181 ceph-mon[94189]: pgmap v7349: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 171 KiB/s wr, 9 op/s 2023-03-09T18:31:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:31:48 smithi038 ceph-mon[92422]: pgmap v7349: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 171 KiB/s wr, 9 op/s 2023-03-09T18:31:50.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:31:49 smithi122 ceph-mon[93869]: pgmap v7350: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 237 KiB/s wr, 11 op/s 2023-03-09T18:31:50.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:31:49 smithi181 ceph-mon[94189]: pgmap v7350: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 237 KiB/s wr, 11 op/s 2023-03-09T18:31:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:31:49 smithi038 ceph-mon[92422]: pgmap v7350: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 237 KiB/s wr, 11 op/s 2023-03-09T18:31:51.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:31:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12821.113327 secs 2023-03-09T18:31:51.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:31:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12821.113327 secs 2023-03-09T18:31:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:31:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12821.113327 secs 2023-03-09T18:31:53.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:31:52 smithi122 ceph-mon[93869]: pgmap v7351: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 237 KiB/s wr, 13 op/s 2023-03-09T18:31:53.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:31:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12823.416137 secs 2023-03-09T18:31:53.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:31:52 smithi181 ceph-mon[94189]: pgmap v7351: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 237 KiB/s wr, 13 op/s 2023-03-09T18:31:53.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:31:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12823.416137 secs 2023-03-09T18:31:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:31:52 smithi038 ceph-mon[92422]: pgmap v7351: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 237 KiB/s wr, 13 op/s 2023-03-09T18:31:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:31:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12823.416137 secs 2023-03-09T18:31:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13285.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:31:54.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:31:53 smithi122 ceph-mon[93869]: pgmap v7352: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 295 KiB/s wr, 11 op/s 2023-03-09T18:31:54.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:31:53 smithi181 ceph-mon[94189]: pgmap v7352: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 295 KiB/s wr, 11 op/s 2023-03-09T18:31:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:31:53 smithi038 ceph-mon[92422]: pgmap v7352: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 295 KiB/s wr, 11 op/s 2023-03-09T18:31:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:31:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12826.113390 secs 2023-03-09T18:31:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:31:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12826.113390 secs 2023-03-09T18:31:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:31:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12826.113390 secs 2023-03-09T18:31:56.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:31:56 smithi122 ceph-mon[93869]: pgmap v7353: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 210 KiB/s wr, 13 op/s 2023-03-09T18:31:56.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:31:56 smithi181 ceph-mon[94189]: pgmap v7353: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 210 KiB/s wr, 13 op/s 2023-03-09T18:31:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:31:56 smithi038 ceph-mon[92422]: pgmap v7353: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 210 KiB/s wr, 13 op/s 2023-03-09T18:31:58.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:31:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12828.416260 secs 2023-03-09T18:31:58.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:31:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12828.416260 secs 2023-03-09T18:31:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:31:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12828.416260 secs 2023-03-09T18:31:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13290.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:31:59.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:31:58 smithi122 ceph-mon[93869]: pgmap v7354: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 198 KiB/s wr, 10 op/s 2023-03-09T18:31:59.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:31:58 smithi181 ceph-mon[94189]: pgmap v7354: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 198 KiB/s wr, 10 op/s 2023-03-09T18:31:59.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:31:58 smithi038 ceph-mon[92422]: pgmap v7354: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 198 KiB/s wr, 10 op/s 2023-03-09T18:32:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:31:59 smithi122 ceph-mon[93869]: pgmap v7355: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 230 KiB/s wr, 11 op/s 2023-03-09T18:32:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:31:59 smithi181 ceph-mon[94189]: pgmap v7355: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 230 KiB/s wr, 11 op/s 2023-03-09T18:32:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:31:59 smithi038 ceph-mon[92422]: pgmap v7355: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 230 KiB/s wr, 11 op/s 2023-03-09T18:32:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12831.113450 secs 2023-03-09T18:32:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12831.113450 secs 2023-03-09T18:32:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12831.113450 secs 2023-03-09T18:32:03.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:02 smithi122 ceph-mon[93869]: pgmap v7356: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 186 KiB/s wr, 13 op/s 2023-03-09T18:32:03.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12833.416364 secs 2023-03-09T18:32:03.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:02 smithi181 ceph-mon[94189]: pgmap v7356: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 186 KiB/s wr, 13 op/s 2023-03-09T18:32:03.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12833.416364 secs 2023-03-09T18:32:03.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:02 smithi038 ceph-mon[92422]: pgmap v7356: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 186 KiB/s wr, 13 op/s 2023-03-09T18:32:03.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12833.416364 secs 2023-03-09T18:32:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13295.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:32:04.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:03 smithi122 ceph-mon[93869]: pgmap v7357: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 272 KiB/s wr, 24 op/s 2023-03-09T18:32:04.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:03 smithi181 ceph-mon[94189]: pgmap v7357: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 272 KiB/s wr, 24 op/s 2023-03-09T18:32:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:03 smithi038 ceph-mon[92422]: pgmap v7357: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 272 KiB/s wr, 24 op/s 2023-03-09T18:32:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12836.113517 secs 2023-03-09T18:32:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12836.113517 secs 2023-03-09T18:32:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12836.113517 secs 2023-03-09T18:32:06.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:06 smithi122 ceph-mon[93869]: pgmap v7358: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 151 KiB/s wr, 24 op/s 2023-03-09T18:32:06.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:06 smithi181 ceph-mon[94189]: pgmap v7358: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 151 KiB/s wr, 24 op/s 2023-03-09T18:32:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:06 smithi038 ceph-mon[92422]: pgmap v7358: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 151 KiB/s wr, 24 op/s 2023-03-09T18:32:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12838.416452 secs 2023-03-09T18:32:08.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12838.416452 secs 2023-03-09T18:32:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12838.416452 secs 2023-03-09T18:32:08.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13300.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:32:09.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:08 smithi122 ceph-mon[93869]: pgmap v7359: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 140 KiB/s wr, 22 op/s 2023-03-09T18:32:09.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:08 smithi181 ceph-mon[94189]: pgmap v7359: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 140 KiB/s wr, 22 op/s 2023-03-09T18:32:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:08 smithi038 ceph-mon[92422]: pgmap v7359: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 140 KiB/s wr, 22 op/s 2023-03-09T18:32:10.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:09 smithi122 ceph-mon[93869]: pgmap v7360: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 150 KiB/s wr, 23 op/s 2023-03-09T18:32:10.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:09 smithi181 ceph-mon[94189]: pgmap v7360: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 150 KiB/s wr, 23 op/s 2023-03-09T18:32:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:09 smithi038 ceph-mon[92422]: pgmap v7360: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 150 KiB/s wr, 23 op/s 2023-03-09T18:32:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12841.113578 secs 2023-03-09T18:32:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:10 smithi122 ceph-mon[93869]: osdmap e673: 12 total, 12 up, 12 in 2023-03-09T18:32:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12841.113578 secs 2023-03-09T18:32:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:10 smithi181 ceph-mon[94189]: osdmap e673: 12 total, 12 up, 12 in 2023-03-09T18:32:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12841.113578 secs 2023-03-09T18:32:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:10 smithi038 ceph-mon[92422]: osdmap e673: 12 total, 12 up, 12 in 2023-03-09T18:32:12.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:11 smithi122 ceph-mon[93869]: pgmap v7362: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 116 KiB/s wr, 23 op/s 2023-03-09T18:32:12.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:11 smithi181 ceph-mon[94189]: pgmap v7362: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 116 KiB/s wr, 23 op/s 2023-03-09T18:32:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:11 smithi038 ceph-mon[92422]: pgmap v7362: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 116 KiB/s wr, 23 op/s 2023-03-09T18:32:13.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12843.416572 secs 2023-03-09T18:32:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12843.416572 secs 2023-03-09T18:32:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12843.416572 secs 2023-03-09T18:32:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13305.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:32:14.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:13 smithi122 ceph-mon[93869]: pgmap v7363: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 44 KiB/s wr, 8 op/s 2023-03-09T18:32:14.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:13 smithi181 ceph-mon[94189]: pgmap v7363: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 44 KiB/s wr, 8 op/s 2023-03-09T18:32:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:13 smithi038 ceph-mon[92422]: pgmap v7363: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 44 KiB/s wr, 8 op/s 2023-03-09T18:32:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12846.113664 secs 2023-03-09T18:32:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12846.113664 secs 2023-03-09T18:32:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12846.113664 secs 2023-03-09T18:32:16.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:16 smithi122 ceph-mon[93869]: pgmap v7364: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 56 KiB/s wr, 8 op/s 2023-03-09T18:32:16.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:16 smithi181 ceph-mon[94189]: pgmap v7364: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 56 KiB/s wr, 8 op/s 2023-03-09T18:32:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:16 smithi038 ceph-mon[92422]: pgmap v7364: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 56 KiB/s wr, 8 op/s 2023-03-09T18:32:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:17 smithi122 ceph-mon[93869]: osdmap e674: 12 total, 12 up, 12 in 2023-03-09T18:32:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12848.416696 secs 2023-03-09T18:32:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:17 smithi181 ceph-mon[94189]: osdmap e674: 12 total, 12 up, 12 in 2023-03-09T18:32:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12848.416696 secs 2023-03-09T18:32:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:17 smithi038 ceph-mon[92422]: osdmap e674: 12 total, 12 up, 12 in 2023-03-09T18:32:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12848.416696 secs 2023-03-09T18:32:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13310.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:32:19.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:18 smithi122 ceph-mon[93869]: pgmap v7366: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 55 KiB/s wr, 9 op/s 2023-03-09T18:32:19.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:18 smithi181 ceph-mon[94189]: pgmap v7366: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 55 KiB/s wr, 9 op/s 2023-03-09T18:32:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:18 smithi038 ceph-mon[92422]: pgmap v7366: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 55 KiB/s wr, 9 op/s 2023-03-09T18:32:20.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:19 smithi122 ceph-mon[93869]: pgmap v7367: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 46 KiB/s wr, 8 op/s 2023-03-09T18:32:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:19 smithi181 ceph-mon[94189]: pgmap v7367: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 46 KiB/s wr, 8 op/s 2023-03-09T18:32:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:19 smithi038 ceph-mon[92422]: pgmap v7367: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 46 KiB/s wr, 8 op/s 2023-03-09T18:32:21.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12851.113724 secs 2023-03-09T18:32:21.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:32:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12851.113724 secs 2023-03-09T18:32:21.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:32:21.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12851.113724 secs 2023-03-09T18:32:21.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:32:23.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:22 smithi122 ceph-mon[93869]: pgmap v7368: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 45 KiB/s wr, 8 op/s 2023-03-09T18:32:23.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12853.416762 secs 2023-03-09T18:32:23.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:22 smithi181 ceph-mon[94189]: pgmap v7368: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 45 KiB/s wr, 8 op/s 2023-03-09T18:32:23.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12853.416762 secs 2023-03-09T18:32:23.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:22 smithi038 ceph-mon[92422]: pgmap v7368: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 45 KiB/s wr, 8 op/s 2023-03-09T18:32:23.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12853.416762 secs 2023-03-09T18:32:23.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13315.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:32:24.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:32:24.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:32:24.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:32:24.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:32:24.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:32:24.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:32:24.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:24 smithi122 ceph-mon[93869]: pgmap v7369: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 8 op/s 2023-03-09T18:32:24.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:32:24.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:32:24.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:32:24.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:32:24.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:32:24.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:32:24.481 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:32:24.481 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:32:24.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:32:24.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:32:24.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:32:24.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:32:24.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:32:24.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:32:24.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:24 smithi181 ceph-mon[94189]: pgmap v7369: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 8 op/s 2023-03-09T18:32:24.485 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:32:24.485 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:32:24.485 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:32:24.485 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:32:24.486 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:32:24.486 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:32:24.486 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:32:24.486 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:32:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:32:24.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:32:24.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:32:24.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:32:24.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:32:24.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:32:24.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:24 smithi038 ceph-mon[92422]: pgmap v7369: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 8 op/s 2023-03-09T18:32:24.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:32:24.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:32:24.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:32:24.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:32:24.503 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:32:24.503 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:32:24.503 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:32:24.503 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:32:25.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:25 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi038 to 195.5M 2023-03-09T18:32:25.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:25 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T18:32:25.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:25 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi038 to 204998033: error parsing value: Value '204998033' is below minimum 939524096 2023-03-09T18:32:25.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:25 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T18:32:25.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:32:25.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:32:25.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:32:25.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:32:25.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:25 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T18:32:25.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:25 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T18:32:25.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:32:25.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:32:25.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:32:25.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:25 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi038 to 195.5M 2023-03-09T18:32:25.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:25 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T18:32:25.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:25 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi038 to 204998033: error parsing value: Value '204998033' is below minimum 939524096 2023-03-09T18:32:25.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:25 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T18:32:25.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:32:25.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:32:25.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:32:25.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:32:25.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:25 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T18:32:25.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:25 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T18:32:25.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:32:25.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:32:25.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:32:25.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:25 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi038 to 195.5M 2023-03-09T18:32:25.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:25 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T18:32:25.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:25 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi038 to 204998033: error parsing value: Value '204998033' is below minimum 939524096 2023-03-09T18:32:25.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:25 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T18:32:25.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:32:25.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:32:25.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:32:25.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:32:25.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:25 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T18:32:25.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:25 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T18:32:25.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:32:25.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:32:25.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:32:26.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12856.113786 secs 2023-03-09T18:32:26.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:26 smithi122 ceph-mon[93869]: pgmap v7370: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 2.1 KiB/s wr, 7 op/s 2023-03-09T18:32:26.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12856.113786 secs 2023-03-09T18:32:26.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:26 smithi181 ceph-mon[94189]: pgmap v7370: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 2.1 KiB/s wr, 7 op/s 2023-03-09T18:32:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12856.113786 secs 2023-03-09T18:32:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:26 smithi038 ceph-mon[92422]: pgmap v7370: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 2.1 KiB/s wr, 7 op/s 2023-03-09T18:32:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12858.416872 secs 2023-03-09T18:32:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:27 smithi122 ceph-mon[93869]: pgmap v7371: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 46 KiB/s wr, 7 op/s 2023-03-09T18:32:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12858.416872 secs 2023-03-09T18:32:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:27 smithi181 ceph-mon[94189]: pgmap v7371: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 46 KiB/s wr, 7 op/s 2023-03-09T18:32:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12858.416872 secs 2023-03-09T18:32:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:27 smithi038 ceph-mon[92422]: pgmap v7371: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 46 KiB/s wr, 7 op/s 2023-03-09T18:32:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13320.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:32:31.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12861.113845 secs 2023-03-09T18:32:31.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:30 smithi122 ceph-mon[93869]: pgmap v7372: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 42 KiB/s wr, 7 op/s 2023-03-09T18:32:31.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12861.113845 secs 2023-03-09T18:32:31.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:30 smithi181 ceph-mon[94189]: pgmap v7372: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 42 KiB/s wr, 7 op/s 2023-03-09T18:32:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12861.113845 secs 2023-03-09T18:32:31.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:30 smithi038 ceph-mon[92422]: pgmap v7372: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 42 KiB/s wr, 7 op/s 2023-03-09T18:32:33.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:32 smithi122 ceph-mon[93869]: pgmap v7373: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 41 KiB/s wr, 8 op/s 2023-03-09T18:32:33.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12863.417007 secs 2023-03-09T18:32:33.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:32 smithi181 ceph-mon[94189]: pgmap v7373: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 41 KiB/s wr, 8 op/s 2023-03-09T18:32:33.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12863.417007 secs 2023-03-09T18:32:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:32 smithi038 ceph-mon[92422]: pgmap v7373: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 41 KiB/s wr, 8 op/s 2023-03-09T18:32:33.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12863.417007 secs 2023-03-09T18:32:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13325.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:32:35.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:34 smithi122 ceph-mon[93869]: pgmap v7374: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 41 KiB/s wr, 7 op/s 2023-03-09T18:32:35.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:34 smithi181 ceph-mon[94189]: pgmap v7374: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 41 KiB/s wr, 7 op/s 2023-03-09T18:32:35.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:34 smithi038 ceph-mon[92422]: pgmap v7374: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 41 KiB/s wr, 7 op/s 2023-03-09T18:32:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12866.113927 secs 2023-03-09T18:32:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12866.113927 secs 2023-03-09T18:32:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12866.113927 secs 2023-03-09T18:32:37.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:36 smithi122 ceph-mon[93869]: pgmap v7375: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 41 KiB/s wr, 8 op/s 2023-03-09T18:32:37.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:36 smithi181 ceph-mon[94189]: pgmap v7375: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 41 KiB/s wr, 8 op/s 2023-03-09T18:32:37.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:36 smithi038 ceph-mon[92422]: pgmap v7375: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 41 KiB/s wr, 8 op/s 2023-03-09T18:32:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12868.417160 secs 2023-03-09T18:32:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:37 smithi122 ceph-mon[93869]: pgmap v7376: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 90 KiB/s wr, 6 op/s 2023-03-09T18:32:38.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12868.417160 secs 2023-03-09T18:32:38.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:37 smithi181 ceph-mon[94189]: pgmap v7376: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 90 KiB/s wr, 6 op/s 2023-03-09T18:32:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12868.417160 secs 2023-03-09T18:32:38.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:37 smithi038 ceph-mon[92422]: pgmap v7376: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 90 KiB/s wr, 6 op/s 2023-03-09T18:32:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13330.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:32:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12871.113986 secs 2023-03-09T18:32:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:40 smithi122 ceph-mon[93869]: pgmap v7377: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 60 KiB/s wr, 6 op/s 2023-03-09T18:32:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12871.113986 secs 2023-03-09T18:32:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:40 smithi181 ceph-mon[94189]: pgmap v7377: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 60 KiB/s wr, 6 op/s 2023-03-09T18:32:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12871.113986 secs 2023-03-09T18:32:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:40 smithi038 ceph-mon[92422]: pgmap v7377: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 60 KiB/s wr, 6 op/s 2023-03-09T18:32:43.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:42 smithi122 ceph-mon[93869]: pgmap v7378: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 60 KiB/s wr, 7 op/s 2023-03-09T18:32:43.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12873.417312 secs 2023-03-09T18:32:43.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:42 smithi181 ceph-mon[94189]: pgmap v7378: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 60 KiB/s wr, 7 op/s 2023-03-09T18:32:43.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12873.417312 secs 2023-03-09T18:32:43.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:42 smithi038 ceph-mon[92422]: pgmap v7378: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 60 KiB/s wr, 7 op/s 2023-03-09T18:32:43.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12873.417312 secs 2023-03-09T18:32:43.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13335.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:32:45.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:44 smithi122 ceph-mon[93869]: pgmap v7379: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 133 KiB/s wr, 7 op/s 2023-03-09T18:32:45.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:44 smithi181 ceph-mon[94189]: pgmap v7379: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 133 KiB/s wr, 7 op/s 2023-03-09T18:32:45.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:44 smithi038 ceph-mon[92422]: pgmap v7379: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 133 KiB/s wr, 7 op/s 2023-03-09T18:32:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12876.114046 secs 2023-03-09T18:32:46.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12876.114046 secs 2023-03-09T18:32:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12876.114046 secs 2023-03-09T18:32:47.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:46 smithi122 ceph-mon[93869]: pgmap v7380: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 143 KiB/s wr, 9 op/s 2023-03-09T18:32:47.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:46 smithi181 ceph-mon[94189]: pgmap v7380: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 143 KiB/s wr, 9 op/s 2023-03-09T18:32:47.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:46 smithi038 ceph-mon[92422]: pgmap v7380: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 143 KiB/s wr, 9 op/s 2023-03-09T18:32:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:47 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:32:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:47 smithi122 ceph-mon[93869]: pgmap v7381: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 189 KiB/s wr, 8 op/s 2023-03-09T18:32:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:47 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:32:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:47 smithi181 ceph-mon[94189]: pgmap v7381: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 189 KiB/s wr, 8 op/s 2023-03-09T18:32:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:47 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:32:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:47 smithi038 ceph-mon[92422]: pgmap v7381: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 189 KiB/s wr, 8 op/s 2023-03-09T18:32:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13340.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:32:49.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:48 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:32:49.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:32:49.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:32:49.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:48 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:32:49.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:32:49.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:32:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:48 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:32:49.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:32:49.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:32:50.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:49 smithi122 ceph-mon[93869]: pgmap v7382: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 151 KiB/s wr, 9 op/s 2023-03-09T18:32:50.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:49 smithi181 ceph-mon[94189]: pgmap v7382: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 151 KiB/s wr, 9 op/s 2023-03-09T18:32:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:49 smithi038 ceph-mon[92422]: pgmap v7382: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 151 KiB/s wr, 9 op/s 2023-03-09T18:32:51.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12881.114105 secs 2023-03-09T18:32:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12881.114105 secs 2023-03-09T18:32:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12881.114105 secs 2023-03-09T18:32:52.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:51 smithi122 ceph-mon[93869]: pgmap v7383: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 140 KiB/s wr, 10 op/s 2023-03-09T18:32:52.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:51 smithi181 ceph-mon[94189]: pgmap v7383: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 140 KiB/s wr, 10 op/s 2023-03-09T18:32:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:51 smithi038 ceph-mon[92422]: pgmap v7383: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 140 KiB/s wr, 10 op/s 2023-03-09T18:32:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13345.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:32:55.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:54 smithi122 ceph-mon[93869]: pgmap v7384: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 265 KiB/s wr, 11 op/s 2023-03-09T18:32:55.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:54 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12886.114165 secs 2023-03-09T18:32:55.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:54 smithi181 ceph-mon[94189]: pgmap v7384: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 265 KiB/s wr, 11 op/s 2023-03-09T18:32:55.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:54 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12886.114165 secs 2023-03-09T18:32:55.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:54 smithi038 ceph-mon[92422]: pgmap v7384: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 265 KiB/s wr, 11 op/s 2023-03-09T18:32:55.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:54 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12886.114165 secs 2023-03-09T18:32:57.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:56 smithi122 ceph-mon[93869]: pgmap v7385: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 205 KiB/s wr, 12 op/s 2023-03-09T18:32:57.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:56 smithi181 ceph-mon[94189]: pgmap v7385: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 205 KiB/s wr, 12 op/s 2023-03-09T18:32:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:56 smithi038 ceph-mon[92422]: pgmap v7385: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 205 KiB/s wr, 12 op/s 2023-03-09T18:32:58.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:57 smithi122 ceph-mon[93869]: pgmap v7386: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 194 KiB/s wr, 10 op/s 2023-03-09T18:32:58.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:57 smithi181 ceph-mon[94189]: pgmap v7386: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 194 KiB/s wr, 10 op/s 2023-03-09T18:32:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:57 smithi038 ceph-mon[92422]: pgmap v7386: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 194 KiB/s wr, 10 op/s 2023-03-09T18:32:58.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13350.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:33:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:32:59 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12891.114225 secs 2023-03-09T18:33:00.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:32:59 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12891.114225 secs 2023-03-09T18:33:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:32:59 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12891.114225 secs 2023-03-09T18:33:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:33:00 smithi122 ceph-mon[93869]: pgmap v7387: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 262 KiB/s wr, 11 op/s 2023-03-09T18:33:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:33:00 smithi181 ceph-mon[94189]: pgmap v7387: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 262 KiB/s wr, 11 op/s 2023-03-09T18:33:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:33:00 smithi038 ceph-mon[92422]: pgmap v7387: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 262 KiB/s wr, 11 op/s 2023-03-09T18:33:03.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:33:02 smithi122 ceph-mon[93869]: pgmap v7388: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 274 KiB/s wr, 14 op/s 2023-03-09T18:33:03.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:33:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12893.417714 secs 2023-03-09T18:33:03.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:33:02 smithi181 ceph-mon[94189]: pgmap v7388: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 274 KiB/s wr, 14 op/s 2023-03-09T18:33:03.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:33:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12893.417714 secs 2023-03-09T18:33:03.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:33:02 smithi038 ceph-mon[92422]: pgmap v7388: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 274 KiB/s wr, 14 op/s 2023-03-09T18:33:03.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:33:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12893.417714 secs 2023-03-09T18:33:03.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13355.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:33:04.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:33:03 smithi122 ceph-mon[93869]: pgmap v7389: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 372 KiB/s wr, 26 op/s 2023-03-09T18:33:04.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:33:03 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:33:04.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:33:03 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:33:04.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:33:03 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:33:04.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:33:03 smithi181 ceph-mon[94189]: pgmap v7389: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 372 KiB/s wr, 26 op/s 2023-03-09T18:33:04.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:33:03 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:33:04.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:33:03 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:33:04.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:33:03 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:33:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:33:03 smithi038 ceph-mon[92422]: pgmap v7389: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 372 KiB/s wr, 26 op/s 2023-03-09T18:33:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:33:03 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:33:04.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:33:03 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:33:04.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:33:03 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:33:05.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:33:04 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12896.114284 secs 2023-03-09T18:33:05.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:33:04 smithi122 ceph-mon[93869]: osdmap e675: 12 total, 12 up, 12 in 2023-03-09T18:33:05.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:33:04 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12896.114284 secs 2023-03-09T18:33:05.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:33:04 smithi181 ceph-mon[94189]: osdmap e675: 12 total, 12 up, 12 in 2023-03-09T18:33:05.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:33:04 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12896.114284 secs 2023-03-09T18:33:05.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:33:04 smithi038 ceph-mon[92422]: osdmap e675: 12 total, 12 up, 12 in 2023-03-09T18:33:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:33:05 smithi122 ceph-mon[93869]: pgmap v7391: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 284 KiB/s wr, 28 op/s 2023-03-09T18:33:06.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:33:05 smithi181 ceph-mon[94189]: pgmap v7391: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 284 KiB/s wr, 28 op/s 2023-03-09T18:33:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:33:05 smithi038 ceph-mon[92422]: pgmap v7391: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 284 KiB/s wr, 28 op/s 2023-03-09T18:33:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:33:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12898.417853 secs 2023-03-09T18:33:08.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:33:07 smithi122 ceph-mon[93869]: pgmap v7392: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 284 KiB/s wr, 28 op/s 2023-03-09T18:33:08.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:33:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12898.417853 secs 2023-03-09T18:33:08.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:33:07 smithi181 ceph-mon[94189]: pgmap v7392: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 284 KiB/s wr, 28 op/s 2023-03-09T18:33:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:33:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12898.417853 secs 2023-03-09T18:33:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:33:07 smithi038 ceph-mon[92422]: pgmap v7392: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 284 KiB/s wr, 28 op/s 2023-03-09T18:33:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13360.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:33:10.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:33:09 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12901.114346 secs 2023-03-09T18:33:10.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:33:09 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12901.114346 secs 2023-03-09T18:33:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:33:09 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12901.114346 secs 2023-03-09T18:33:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:33:10 smithi122 ceph-mon[93869]: pgmap v7393: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 194 KiB/s wr, 26 op/s 2023-03-09T18:33:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:33:10 smithi181 ceph-mon[94189]: pgmap v7393: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 194 KiB/s wr, 26 op/s 2023-03-09T18:33:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:33:10 smithi038 ceph-mon[92422]: pgmap v7393: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 194 KiB/s wr, 26 op/s 2023-03-09T18:33:13.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:33:12 smithi122 ceph-mon[93869]: pgmap v7394: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 167 KiB/s wr, 24 op/s 2023-03-09T18:33:13.267 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:33:12 smithi122 ceph-mon[93869]: osdmap e676: 12 total, 12 up, 12 in 2023-03-09T18:33:13.268 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:33:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12903.417970 secs 2023-03-09T18:33:13.269 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:33:12 smithi038 ceph-mon[92422]: pgmap v7394: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 167 KiB/s wr, 24 op/s 2023-03-09T18:33:13.269 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:33:12 smithi038 ceph-mon[92422]: osdmap e676: 12 total, 12 up, 12 in 2023-03-09T18:33:13.269 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:33:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12903.417970 secs 2023-03-09T18:33:13.269 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:33:12 smithi181 ceph-mon[94189]: pgmap v7394: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 167 KiB/s wr, 24 op/s 2023-03-09T18:33:13.269 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:33:12 smithi181 ceph-mon[94189]: osdmap e676: 12 total, 12 up, 12 in 2023-03-09T18:33:13.270 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:33:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12903.417970 secs 2023-03-09T18:33:13.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13365.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:33:15.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:33:14 smithi122 ceph-mon[93869]: pgmap v7396: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 56 KiB/s wr, 8 op/s 2023-03-09T18:33:15.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:33:14 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12906.114406 secs 2023-03-09T18:33:15.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:33:14 smithi181 ceph-mon[94189]: pgmap v7396: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 56 KiB/s wr, 8 op/s 2023-03-09T18:33:15.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:33:14 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12906.114406 secs 2023-03-09T18:33:15.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:33:14 smithi038 ceph-mon[92422]: pgmap v7396: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 56 KiB/s wr, 8 op/s 2023-03-09T18:33:15.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:33:14 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12906.114406 secs 2023-03-09T18:33:17.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:33:16 smithi122 ceph-mon[93869]: pgmap v7397: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 48 KiB/s wr, 7 op/s 2023-03-09T18:33:17.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:33:16 smithi181 ceph-mon[94189]: pgmap v7397: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 48 KiB/s wr, 7 op/s 2023-03-09T18:33:17.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:33:16 smithi038 ceph-mon[92422]: pgmap v7397: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 48 KiB/s wr, 7 op/s 2023-03-09T18:33:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:33:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12908.418070 secs 2023-03-09T18:33:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:33:17 smithi122 ceph-mon[93869]: pgmap v7398: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 48 KiB/s wr, 7 op/s 2023-03-09T18:33:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:33:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12908.418070 secs 2023-03-09T18:33:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:33:17 smithi181 ceph-mon[94189]: pgmap v7398: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 48 KiB/s wr, 7 op/s 2023-03-09T18:33:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:33:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12908.418070 secs 2023-03-09T18:33:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:33:17 smithi038 ceph-mon[92422]: pgmap v7398: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 48 KiB/s wr, 7 op/s 2023-03-09T18:33:18.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13370.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:33:20.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:33:19 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12911.114490 secs 2023-03-09T18:33:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:33:19 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12911.114490 secs 2023-03-09T18:33:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:33:19 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12911.114490 secs 2023-03-09T18:33:21.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:33:20 smithi122 ceph-mon[93869]: pgmap v7399: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 7 op/s 2023-03-09T18:33:21.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:33:20 smithi181 ceph-mon[94189]: pgmap v7399: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 7 op/s 2023-03-09T18:33:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:33:20 smithi038 ceph-mon[92422]: pgmap v7399: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 13 KiB/s wr, 7 op/s 2023-03-09T18:33:23.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:33:22 smithi122 ceph-mon[93869]: pgmap v7400: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T18:33:23.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:33:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12913.418156 secs 2023-03-09T18:33:23.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:33:22 smithi181 ceph-mon[94189]: pgmap v7400: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T18:33:23.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:33:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12913.418156 secs 2023-03-09T18:33:23.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:33:22 smithi038 ceph-mon[92422]: pgmap v7400: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T18:33:23.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:33:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12913.418156 secs 2023-03-09T18:33:23.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13375.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:33:25.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:33:24 smithi122 ceph-mon[93869]: pgmap v7401: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 6 op/s 2023-03-09T18:33:25.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:33:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:33:25.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:33:24 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12916.114552 secs 2023-03-09T18:33:25.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:33:24 smithi181 ceph-mon[94189]: pgmap v7401: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 6 op/s 2023-03-09T18:33:25.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:33:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:33:25.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:33:24 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12916.114552 secs 2023-03-09T18:33:25.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:33:24 smithi038 ceph-mon[92422]: pgmap v7401: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 6 op/s 2023-03-09T18:33:25.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:33:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:33:25.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:33:24 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12916.114552 secs 2023-03-09T18:33:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:33:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:33:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:33:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:33:26.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:33:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:33:26.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:33:25 smithi122 ceph-mon[93869]: pgmap v7402: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 8 op/s 2023-03-09T18:33:26.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:33:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:33:26.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:33:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:33:26.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:33:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:33:26.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:33:25 smithi181 ceph-mon[94189]: pgmap v7402: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 8 op/s 2023-03-09T18:33:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:33:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:33:26.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:33:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:33:26.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:33:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:33:26.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:33:25 smithi038 ceph-mon[92422]: pgmap v7402: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 8 op/s 2023-03-09T18:33:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:33:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12918.418296 secs 2023-03-09T18:33:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:33:27 smithi122 ceph-mon[93869]: pgmap v7403: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 6 op/s 2023-03-09T18:33:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:33:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12918.418296 secs 2023-03-09T18:33:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:33:27 smithi181 ceph-mon[94189]: pgmap v7403: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 6 op/s 2023-03-09T18:33:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:33:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12918.418296 secs 2023-03-09T18:33:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:33:27 smithi038 ceph-mon[92422]: pgmap v7403: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 6 op/s 2023-03-09T18:33:28.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13380.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:33:30.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:33:29 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12921.114614 secs 2023-03-09T18:33:30.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:33:29 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12921.114614 secs 2023-03-09T18:33:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:33:29 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12921.114614 secs 2023-03-09T18:33:31.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:33:30 smithi122 ceph-mon[93869]: pgmap v7404: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 52 KiB/s wr, 7 op/s 2023-03-09T18:33:31.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:33:30 smithi181 ceph-mon[94189]: pgmap v7404: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 52 KiB/s wr, 7 op/s 2023-03-09T18:33:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:33:30 smithi038 ceph-mon[92422]: pgmap v7404: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 52 KiB/s wr, 7 op/s 2023-03-09T18:33:33.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:33:32 smithi122 ceph-mon[93869]: pgmap v7405: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 8 op/s 2023-03-09T18:33:33.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:33:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12923.418382 secs 2023-03-09T18:33:33.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:33:32 smithi181 ceph-mon[94189]: pgmap v7405: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 8 op/s 2023-03-09T18:33:33.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:33:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12923.418382 secs 2023-03-09T18:33:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:33:32 smithi038 ceph-mon[92422]: pgmap v7405: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 8 op/s 2023-03-09T18:33:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:33:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12923.418382 secs 2023-03-09T18:33:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13385.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:33:35.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:33:34 smithi122 ceph-mon[93869]: pgmap v7406: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 41 KiB/s wr, 7 op/s 2023-03-09T18:33:35.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:33:34 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12926.114698 secs 2023-03-09T18:33:35.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:33:34 smithi181 ceph-mon[94189]: pgmap v7406: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 41 KiB/s wr, 7 op/s 2023-03-09T18:33:35.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:33:34 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12926.114698 secs 2023-03-09T18:33:35.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:33:34 smithi038 ceph-mon[92422]: pgmap v7406: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 41 KiB/s wr, 7 op/s 2023-03-09T18:33:35.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:33:34 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12926.114698 secs 2023-03-09T18:33:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:33:35 smithi122 ceph-mon[93869]: pgmap v7407: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 41 KiB/s wr, 8 op/s 2023-03-09T18:33:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:33:35 smithi181 ceph-mon[94189]: pgmap v7407: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 41 KiB/s wr, 8 op/s 2023-03-09T18:33:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:33:35 smithi038 ceph-mon[92422]: pgmap v7407: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 41 KiB/s wr, 8 op/s 2023-03-09T18:33:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:33:37 smithi122 ceph-mon[93869]: pgmap v7408: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 41 KiB/s wr, 6 op/s 2023-03-09T18:33:38.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:33:37 smithi181 ceph-mon[94189]: pgmap v7408: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 41 KiB/s wr, 6 op/s 2023-03-09T18:33:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:33:37 smithi038 ceph-mon[92422]: pgmap v7408: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 41 KiB/s wr, 6 op/s 2023-03-09T18:33:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13390.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:33:39.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:33:39 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:33:39.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:33:39 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:33:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:33:39 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:33:40.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:33:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12931.114757 secs 2023-03-09T18:33:40.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:33:40 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:33:40.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:33:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:33:40.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:33:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:33:40.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:33:40 smithi122 ceph-mon[93869]: pgmap v7409: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 116 KiB/s wr, 8 op/s 2023-03-09T18:33:40.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:33:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12931.114757 secs 2023-03-09T18:33:40.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:33:40 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:33:40.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:33:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:33:40.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:33:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:33:40.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:33:40 smithi181 ceph-mon[94189]: pgmap v7409: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 116 KiB/s wr, 8 op/s 2023-03-09T18:33:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:33:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12931.114757 secs 2023-03-09T18:33:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:33:40 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:33:41.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:33:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:33:41.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:33:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:33:41.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:33:40 smithi038 ceph-mon[92422]: pgmap v7409: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 116 KiB/s wr, 8 op/s 2023-03-09T18:33:43.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:33:42 smithi122 ceph-mon[93869]: pgmap v7410: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 75 KiB/s wr, 9 op/s 2023-03-09T18:33:43.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:33:42 smithi181 ceph-mon[94189]: pgmap v7410: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 75 KiB/s wr, 9 op/s 2023-03-09T18:33:43.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:33:42 smithi038 ceph-mon[92422]: pgmap v7410: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 75 KiB/s wr, 9 op/s 2023-03-09T18:33:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13395.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:33:45.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:33:44 smithi122 ceph-mon[93869]: pgmap v7411: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 186 KiB/s wr, 9 op/s 2023-03-09T18:33:45.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:33:44 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12936.114841 secs 2023-03-09T18:33:45.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:33:44 smithi181 ceph-mon[94189]: pgmap v7411: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 186 KiB/s wr, 9 op/s 2023-03-09T18:33:45.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:33:44 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12936.114841 secs 2023-03-09T18:33:45.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:33:44 smithi038 ceph-mon[92422]: pgmap v7411: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 186 KiB/s wr, 9 op/s 2023-03-09T18:33:45.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:33:44 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12936.114841 secs 2023-03-09T18:33:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:33:45 smithi122 ceph-mon[93869]: pgmap v7412: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 187 KiB/s wr, 11 op/s 2023-03-09T18:33:46.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:33:45 smithi181 ceph-mon[94189]: pgmap v7412: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 187 KiB/s wr, 11 op/s 2023-03-09T18:33:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:33:45 smithi038 ceph-mon[92422]: pgmap v7412: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 187 KiB/s wr, 11 op/s 2023-03-09T18:33:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:33:47 smithi122 ceph-mon[93869]: pgmap v7413: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 201 KiB/s wr, 10 op/s 2023-03-09T18:33:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:33:47 smithi181 ceph-mon[94189]: pgmap v7413: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 201 KiB/s wr, 10 op/s 2023-03-09T18:33:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:33:47 smithi038 ceph-mon[92422]: pgmap v7413: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 201 KiB/s wr, 10 op/s 2023-03-09T18:33:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13400.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:33:50.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:33:49 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12941.114924 secs 2023-03-09T18:33:50.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:33:49 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12941.114924 secs 2023-03-09T18:33:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:33:49 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12941.114924 secs 2023-03-09T18:33:51.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:33:50 smithi122 ceph-mon[93869]: pgmap v7414: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 211 KiB/s wr, 11 op/s 2023-03-09T18:33:51.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:33:50 smithi181 ceph-mon[94189]: pgmap v7414: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 211 KiB/s wr, 11 op/s 2023-03-09T18:33:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:33:50 smithi038 ceph-mon[92422]: pgmap v7414: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 211 KiB/s wr, 11 op/s 2023-03-09T18:33:53.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:33:52 smithi122 ceph-mon[93869]: pgmap v7415: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 137 KiB/s wr, 11 op/s 2023-03-09T18:33:53.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:33:52 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 12943.418885 secs 2023-03-09T18:33:53.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:33:52 smithi181 ceph-mon[94189]: pgmap v7415: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 137 KiB/s wr, 11 op/s 2023-03-09T18:33:53.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:33:52 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 12943.418885 secs 2023-03-09T18:33:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:33:52 smithi038 ceph-mon[92422]: pgmap v7415: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 137 KiB/s wr, 11 op/s 2023-03-09T18:33:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:33:52 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 12943.418885 secs 2023-03-09T18:33:53.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13405.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:33:54.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:33:53 smithi122 ceph-mon[93869]: pgmap v7416: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 287 KiB/s wr, 10 op/s 2023-03-09T18:33:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:33:53 smithi181 ceph-mon[94189]: pgmap v7416: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 287 KiB/s wr, 10 op/s 2023-03-09T18:33:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:33:53 smithi038 ceph-mon[92422]: pgmap v7416: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 287 KiB/s wr, 10 op/s 2023-03-09T18:33:55.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:33:54 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12946.114998 secs 2023-03-09T18:33:55.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:33:54 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12946.114998 secs 2023-03-09T18:33:55.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:33:54 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12946.114998 secs 2023-03-09T18:33:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:33:55 smithi122 ceph-mon[93869]: pgmap v7417: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 186 KiB/s wr, 11 op/s 2023-03-09T18:33:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:33:55 smithi181 ceph-mon[94189]: pgmap v7417: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 186 KiB/s wr, 11 op/s 2023-03-09T18:33:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:33:55 smithi038 ceph-mon[92422]: pgmap v7417: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 186 KiB/s wr, 11 op/s 2023-03-09T18:33:57.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:33:56 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:33:57.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:33:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:33:57.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:33:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:33:57.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:33:56 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:33:57.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:33:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:33:57.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:33:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:33:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:33:56 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:33:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:33:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:33:57.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:33:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:33:58.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:33:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12948.419024 secs 2023-03-09T18:33:58.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:33:57 smithi122 ceph-mon[93869]: pgmap v7418: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 252 KiB/s wr, 9 op/s 2023-03-09T18:33:58.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:33:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12948.419024 secs 2023-03-09T18:33:58.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:33:57 smithi181 ceph-mon[94189]: pgmap v7418: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 252 KiB/s wr, 9 op/s 2023-03-09T18:33:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:33:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12948.419024 secs 2023-03-09T18:33:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:33:57 smithi038 ceph-mon[92422]: pgmap v7418: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 252 KiB/s wr, 9 op/s 2023-03-09T18:33:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13410.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:34:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:34:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12951.115061 secs 2023-03-09T18:34:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:34:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:34:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:34:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:34:00.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:34:00 smithi122 ceph-mon[93869]: pgmap v7419: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 239 KiB/s wr, 9 op/s 2023-03-09T18:34:00.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:34:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12951.115061 secs 2023-03-09T18:34:00.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:34:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:34:00.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:34:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:34:00.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:34:00 smithi181 ceph-mon[94189]: pgmap v7419: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 239 KiB/s wr, 9 op/s 2023-03-09T18:34:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:34:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12951.115061 secs 2023-03-09T18:34:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:34:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:34:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:34:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:34:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:34:00 smithi038 ceph-mon[92422]: pgmap v7419: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 239 KiB/s wr, 9 op/s 2023-03-09T18:34:02.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:34:01 smithi122 ceph-mon[93869]: pgmap v7420: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 282 KiB/s wr, 13 op/s 2023-03-09T18:34:02.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:34:01 smithi181 ceph-mon[94189]: pgmap v7420: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 282 KiB/s wr, 13 op/s 2023-03-09T18:34:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:34:01 smithi038 ceph-mon[92422]: pgmap v7420: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 282 KiB/s wr, 13 op/s 2023-03-09T18:34:03.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:34:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12953.419081 secs 2023-03-09T18:34:03.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:34:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12953.419081 secs 2023-03-09T18:34:03.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:34:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12953.419081 secs 2023-03-09T18:34:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13415.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:34:04.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:34:03 smithi122 ceph-mon[93869]: pgmap v7421: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 304 KiB/s wr, 24 op/s 2023-03-09T18:34:04.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:34:03 smithi181 ceph-mon[94189]: pgmap v7421: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 304 KiB/s wr, 24 op/s 2023-03-09T18:34:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:34:03 smithi038 ceph-mon[92422]: pgmap v7421: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 304 KiB/s wr, 24 op/s 2023-03-09T18:34:05.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:34:04 smithi122 ceph-mon[93869]: osdmap e677: 12 total, 12 up, 12 in 2023-03-09T18:34:05.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:34:04 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12956.115121 secs 2023-03-09T18:34:05.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:34:04 smithi181 ceph-mon[94189]: osdmap e677: 12 total, 12 up, 12 in 2023-03-09T18:34:05.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:34:04 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12956.115121 secs 2023-03-09T18:34:05.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:34:04 smithi038 ceph-mon[92422]: osdmap e677: 12 total, 12 up, 12 in 2023-03-09T18:34:05.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:34:04 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12956.115121 secs 2023-03-09T18:34:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:34:05 smithi122 ceph-mon[93869]: pgmap v7423: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 282 KiB/s wr, 28 op/s 2023-03-09T18:34:06.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:34:05 smithi181 ceph-mon[94189]: pgmap v7423: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 282 KiB/s wr, 28 op/s 2023-03-09T18:34:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:34:05 smithi038 ceph-mon[92422]: pgmap v7423: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 282 KiB/s wr, 28 op/s 2023-03-09T18:34:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:34:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12958.419193 secs 2023-03-09T18:34:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:34:07 smithi122 ceph-mon[93869]: pgmap v7424: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 202 KiB/s wr, 27 op/s 2023-03-09T18:34:08.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:34:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12958.419193 secs 2023-03-09T18:34:08.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:34:07 smithi181 ceph-mon[94189]: pgmap v7424: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 202 KiB/s wr, 27 op/s 2023-03-09T18:34:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:34:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12958.419193 secs 2023-03-09T18:34:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:34:07 smithi038 ceph-mon[92422]: pgmap v7424: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 202 KiB/s wr, 27 op/s 2023-03-09T18:34:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13420.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:34:10.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:34:09 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12961.115182 secs 2023-03-09T18:34:10.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:34:09 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12961.115182 secs 2023-03-09T18:34:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:34:09 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12961.115182 secs 2023-03-09T18:34:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:34:10 smithi122 ceph-mon[93869]: pgmap v7425: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 213 KiB/s wr, 27 op/s 2023-03-09T18:34:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:34:10 smithi181 ceph-mon[94189]: pgmap v7425: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 213 KiB/s wr, 27 op/s 2023-03-09T18:34:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:34:10 smithi038 ceph-mon[92422]: pgmap v7425: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 213 KiB/s wr, 27 op/s 2023-03-09T18:34:12.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:34:11 smithi122 ceph-mon[93869]: pgmap v7426: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 171 KiB/s wr, 24 op/s 2023-03-09T18:34:12.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:34:11 smithi122 ceph-mon[93869]: osdmap e678: 12 total, 12 up, 12 in 2023-03-09T18:34:12.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:34:11 smithi181 ceph-mon[94189]: pgmap v7426: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 171 KiB/s wr, 24 op/s 2023-03-09T18:34:12.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:34:11 smithi181 ceph-mon[94189]: osdmap e678: 12 total, 12 up, 12 in 2023-03-09T18:34:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:34:11 smithi038 ceph-mon[92422]: pgmap v7426: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 171 KiB/s wr, 24 op/s 2023-03-09T18:34:12.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:34:11 smithi038 ceph-mon[92422]: osdmap e678: 12 total, 12 up, 12 in 2023-03-09T18:34:13.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:34:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12963.419336 secs 2023-03-09T18:34:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:34:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12963.419336 secs 2023-03-09T18:34:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:34:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12963.419336 secs 2023-03-09T18:34:13.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13425.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:34:14.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:34:13 smithi122 ceph-mon[93869]: pgmap v7428: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 146 KiB/s wr, 10 op/s 2023-03-09T18:34:14.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:34:13 smithi181 ceph-mon[94189]: pgmap v7428: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 146 KiB/s wr, 10 op/s 2023-03-09T18:34:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:34:13 smithi038 ceph-mon[92422]: pgmap v7428: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 146 KiB/s wr, 10 op/s 2023-03-09T18:34:15.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:34:14 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12966.115240 secs 2023-03-09T18:34:15.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:34:14 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12966.115240 secs 2023-03-09T18:34:15.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:34:14 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12966.115240 secs 2023-03-09T18:34:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:34:15 smithi122 ceph-mon[93869]: pgmap v7429: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 48 KiB/s wr, 9 op/s 2023-03-09T18:34:16.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:34:15 smithi181 ceph-mon[94189]: pgmap v7429: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 48 KiB/s wr, 9 op/s 2023-03-09T18:34:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:34:15 smithi038 ceph-mon[92422]: pgmap v7429: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 48 KiB/s wr, 9 op/s 2023-03-09T18:34:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:34:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12968.419462 secs 2023-03-09T18:34:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:34:17 smithi122 ceph-mon[93869]: pgmap v7430: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 48 KiB/s wr, 9 op/s 2023-03-09T18:34:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:34:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12968.419462 secs 2023-03-09T18:34:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:34:17 smithi181 ceph-mon[94189]: pgmap v7430: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 48 KiB/s wr, 9 op/s 2023-03-09T18:34:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:34:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12968.419462 secs 2023-03-09T18:34:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:34:17 smithi038 ceph-mon[92422]: pgmap v7430: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 48 KiB/s wr, 9 op/s 2023-03-09T18:34:18.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13430.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:34:20.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:34:19 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12971.115301 secs 2023-03-09T18:34:20.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:34:19 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12971.115301 secs 2023-03-09T18:34:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:34:19 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12971.115301 secs 2023-03-09T18:34:21.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:34:20 smithi122 ceph-mon[93869]: pgmap v7431: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 35 KiB/s wr, 8 op/s 2023-03-09T18:34:21.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:34:20 smithi181 ceph-mon[94189]: pgmap v7431: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 35 KiB/s wr, 8 op/s 2023-03-09T18:34:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:34:20 smithi038 ceph-mon[92422]: pgmap v7431: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 35 KiB/s wr, 8 op/s 2023-03-09T18:34:22.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:34:21 smithi122 ceph-mon[93869]: pgmap v7432: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 8 op/s 2023-03-09T18:34:22.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:34:21 smithi181 ceph-mon[94189]: pgmap v7432: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 8 op/s 2023-03-09T18:34:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:34:21 smithi038 ceph-mon[92422]: pgmap v7432: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 8 op/s 2023-03-09T18:34:23.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:34:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12973.419607 secs 2023-03-09T18:34:23.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:34:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12973.419607 secs 2023-03-09T18:34:23.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:34:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12973.419607 secs 2023-03-09T18:34:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13435.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:34:24.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:34:23 smithi122 ceph-mon[93869]: pgmap v7433: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 7 op/s 2023-03-09T18:34:24.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:34:23 smithi181 ceph-mon[94189]: pgmap v7433: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 7 op/s 2023-03-09T18:34:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:34:23 smithi038 ceph-mon[92422]: pgmap v7433: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 7 op/s 2023-03-09T18:34:25.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:34:24 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12976.151104 secs 2023-03-09T18:34:25.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:34:24 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12976.151104 secs 2023-03-09T18:34:25.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:34:24 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12976.151104 secs 2023-03-09T18:34:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:34:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:34:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:34:25 smithi122 ceph-mon[93869]: pgmap v7434: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 10 op/s 2023-03-09T18:34:26.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:34:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:34:26.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:34:25 smithi181 ceph-mon[94189]: pgmap v7434: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 10 op/s 2023-03-09T18:34:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:34:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:34:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:34:25 smithi038 ceph-mon[92422]: pgmap v7434: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 10 op/s 2023-03-09T18:34:27.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:34:26 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:34:27.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:34:26 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:34:27.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:34:26 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:34:27.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:34:26 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:34:27.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:34:26 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:34:27.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:34:26 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:34:27.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:34:26 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:34:27.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:34:26 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:34:27.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:34:26 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:34:27.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:34:26 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:34:27.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:34:26 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:34:27.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:34:26 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:34:27.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:34:26 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:34:27.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:34:26 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:34:27.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:34:26 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:34:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:34:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12978.419735 secs 2023-03-09T18:34:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:34:27 smithi122 ceph-mon[93869]: pgmap v7435: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 42 KiB/s wr, 7 op/s 2023-03-09T18:34:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:34:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12978.419735 secs 2023-03-09T18:34:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:34:27 smithi181 ceph-mon[94189]: pgmap v7435: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 42 KiB/s wr, 7 op/s 2023-03-09T18:34:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:34:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12978.419735 secs 2023-03-09T18:34:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:34:27 smithi038 ceph-mon[92422]: pgmap v7435: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 42 KiB/s wr, 7 op/s 2023-03-09T18:34:28.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13440.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:34:30.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:34:29 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12981.151161 secs 2023-03-09T18:34:30.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:34:29 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12981.151161 secs 2023-03-09T18:34:30.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:34:29 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12981.151161 secs 2023-03-09T18:34:31.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:34:30 smithi122 ceph-mon[93869]: pgmap v7436: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 7 op/s 2023-03-09T18:34:31.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:34:30 smithi181 ceph-mon[94189]: pgmap v7436: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 7 op/s 2023-03-09T18:34:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:34:30 smithi038 ceph-mon[92422]: pgmap v7436: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 7 op/s 2023-03-09T18:34:32.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:34:31 smithi122 ceph-mon[93869]: pgmap v7437: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 79 KiB/s wr, 9 op/s 2023-03-09T18:34:32.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:34:31 smithi181 ceph-mon[94189]: pgmap v7437: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 79 KiB/s wr, 9 op/s 2023-03-09T18:34:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:34:31 smithi038 ceph-mon[92422]: pgmap v7437: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 79 KiB/s wr, 9 op/s 2023-03-09T18:34:33.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:34:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12983.419856 secs 2023-03-09T18:34:33.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:34:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12983.419856 secs 2023-03-09T18:34:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:34:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12983.419856 secs 2023-03-09T18:34:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13445.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:34:34.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:34:33 smithi122 ceph-mon[93869]: pgmap v7438: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 79 KiB/s wr, 7 op/s 2023-03-09T18:34:34.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:34:33 smithi181 ceph-mon[94189]: pgmap v7438: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 79 KiB/s wr, 7 op/s 2023-03-09T18:34:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:34:33 smithi038 ceph-mon[92422]: pgmap v7438: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 79 KiB/s wr, 7 op/s 2023-03-09T18:34:35.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:34:34 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12986.151220 secs 2023-03-09T18:34:35.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:34:34 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12986.151220 secs 2023-03-09T18:34:35.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:34:34 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12986.151220 secs 2023-03-09T18:34:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:34:35 smithi122 ceph-mon[93869]: pgmap v7439: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 129 KiB/s wr, 9 op/s 2023-03-09T18:34:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:34:35 smithi181 ceph-mon[94189]: pgmap v7439: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 129 KiB/s wr, 9 op/s 2023-03-09T18:34:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:34:35 smithi038 ceph-mon[92422]: pgmap v7439: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 129 KiB/s wr, 9 op/s 2023-03-09T18:34:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:34:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12988.419952 secs 2023-03-09T18:34:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:34:37 smithi122 ceph-mon[93869]: pgmap v7440: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 88 KiB/s wr, 6 op/s 2023-03-09T18:34:38.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:34:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12988.419952 secs 2023-03-09T18:34:38.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:34:37 smithi181 ceph-mon[94189]: pgmap v7440: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 88 KiB/s wr, 6 op/s 2023-03-09T18:34:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:34:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12988.419952 secs 2023-03-09T18:34:38.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:34:37 smithi038 ceph-mon[92422]: pgmap v7440: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 88 KiB/s wr, 6 op/s 2023-03-09T18:34:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13450.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:34:40.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:34:39 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12991.151282 secs 2023-03-09T18:34:40.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:34:39 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12991.151282 secs 2023-03-09T18:34:40.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:34:39 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12991.151282 secs 2023-03-09T18:34:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:34:40 smithi122 ceph-mon[93869]: pgmap v7441: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 99 KiB/s wr, 7 op/s 2023-03-09T18:34:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:34:40 smithi181 ceph-mon[94189]: pgmap v7441: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 99 KiB/s wr, 7 op/s 2023-03-09T18:34:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:34:40 smithi038 ceph-mon[92422]: pgmap v7441: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 99 KiB/s wr, 7 op/s 2023-03-09T18:34:42.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:34:41 smithi122 ceph-mon[93869]: pgmap v7442: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 182 KiB/s wr, 9 op/s 2023-03-09T18:34:42.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:34:41 smithi181 ceph-mon[94189]: pgmap v7442: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 182 KiB/s wr, 9 op/s 2023-03-09T18:34:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:34:41 smithi038 ceph-mon[92422]: pgmap v7442: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 182 KiB/s wr, 9 op/s 2023-03-09T18:34:43.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:34:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12993.420077 secs 2023-03-09T18:34:43.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:34:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12993.420077 secs 2023-03-09T18:34:43.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:34:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12993.420077 secs 2023-03-09T18:34:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13455.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:34:44.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:34:43 smithi122 ceph-mon[93869]: pgmap v7443: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 145 KiB/s wr, 7 op/s 2023-03-09T18:34:44.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:34:43 smithi181 ceph-mon[94189]: pgmap v7443: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 145 KiB/s wr, 7 op/s 2023-03-09T18:34:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:34:43 smithi038 ceph-mon[92422]: pgmap v7443: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 145 KiB/s wr, 7 op/s 2023-03-09T18:34:45.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:34:44 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12996.151342 secs 2023-03-09T18:34:45.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:34:44 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12996.151342 secs 2023-03-09T18:34:45.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:34:44 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12996.151342 secs 2023-03-09T18:34:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:34:45 smithi122 ceph-mon[93869]: pgmap v7444: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 212 KiB/s wr, 12 op/s 2023-03-09T18:34:46.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:34:45 smithi181 ceph-mon[94189]: pgmap v7444: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 212 KiB/s wr, 12 op/s 2023-03-09T18:34:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:34:45 smithi038 ceph-mon[92422]: pgmap v7444: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 212 KiB/s wr, 12 op/s 2023-03-09T18:34:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:34:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 12998.420210 secs 2023-03-09T18:34:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:34:47 smithi122 ceph-mon[93869]: pgmap v7445: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 162 KiB/s wr, 9 op/s 2023-03-09T18:34:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:34:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 12998.420210 secs 2023-03-09T18:34:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:34:47 smithi181 ceph-mon[94189]: pgmap v7445: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 162 KiB/s wr, 9 op/s 2023-03-09T18:34:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:34:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 12998.420210 secs 2023-03-09T18:34:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:34:47 smithi038 ceph-mon[92422]: pgmap v7445: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 162 KiB/s wr, 9 op/s 2023-03-09T18:34:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13460.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:34:50.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:34:49 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13001.151403 secs 2023-03-09T18:34:50.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:34:49 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13001.151403 secs 2023-03-09T18:34:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:34:49 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13001.151403 secs 2023-03-09T18:34:51.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:34:50 smithi122 ceph-mon[93869]: pgmap v7446: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 172 KiB/s wr, 10 op/s 2023-03-09T18:34:51.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:34:50 smithi181 ceph-mon[94189]: pgmap v7446: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 172 KiB/s wr, 10 op/s 2023-03-09T18:34:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:34:50 smithi038 ceph-mon[92422]: pgmap v7446: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 172 KiB/s wr, 10 op/s 2023-03-09T18:34:52.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:34:51 smithi122 ceph-mon[93869]: pgmap v7447: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 267 KiB/s wr, 13 op/s 2023-03-09T18:34:52.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:34:51 smithi181 ceph-mon[94189]: pgmap v7447: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 267 KiB/s wr, 13 op/s 2023-03-09T18:34:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:34:51 smithi038 ceph-mon[92422]: pgmap v7447: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 267 KiB/s wr, 13 op/s 2023-03-09T18:34:53.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:34:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13003.420337 secs 2023-03-09T18:34:53.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:34:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13003.420337 secs 2023-03-09T18:34:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:34:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13003.420337 secs 2023-03-09T18:34:53.801 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13465.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:34:54.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:34:53 smithi122 ceph-mon[93869]: pgmap v7448: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 247 KiB/s rd, 184 KiB/s wr, 10 op/s 2023-03-09T18:34:54.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:34:53 smithi181 ceph-mon[94189]: pgmap v7448: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 247 KiB/s rd, 184 KiB/s wr, 10 op/s 2023-03-09T18:34:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:34:53 smithi038 ceph-mon[92422]: pgmap v7448: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 247 KiB/s rd, 184 KiB/s wr, 10 op/s 2023-03-09T18:34:55.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:34:54 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13006.151485 secs 2023-03-09T18:34:55.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:34:54 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13006.151485 secs 2023-03-09T18:34:55.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:34:54 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13006.151485 secs 2023-03-09T18:34:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:34:55 smithi122 ceph-mon[93869]: pgmap v7449: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 214 KiB/s wr, 14 op/s 2023-03-09T18:34:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:34:55 smithi181 ceph-mon[94189]: pgmap v7449: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 214 KiB/s wr, 14 op/s 2023-03-09T18:34:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:34:55 smithi038 ceph-mon[92422]: pgmap v7449: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 214 KiB/s wr, 14 op/s 2023-03-09T18:34:58.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:34:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13008.420400 secs 2023-03-09T18:34:58.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:34:57 smithi122 ceph-mon[93869]: pgmap v7450: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 146 KiB/s wr, 9 op/s 2023-03-09T18:34:58.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:34:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13008.420400 secs 2023-03-09T18:34:58.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:34:57 smithi181 ceph-mon[94189]: pgmap v7450: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 146 KiB/s wr, 9 op/s 2023-03-09T18:34:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:34:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13008.420400 secs 2023-03-09T18:34:58.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:34:57 smithi038 ceph-mon[92422]: pgmap v7450: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 146 KiB/s wr, 9 op/s 2023-03-09T18:34:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13470.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:35:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:34:59 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13011.151544 secs 2023-03-09T18:35:00.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:34:59 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13011.151544 secs 2023-03-09T18:35:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:34:59 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13011.151544 secs 2023-03-09T18:35:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:35:00 smithi122 ceph-mon[93869]: pgmap v7451: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 156 KiB/s wr, 10 op/s 2023-03-09T18:35:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:35:00 smithi181 ceph-mon[94189]: pgmap v7451: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 156 KiB/s wr, 10 op/s 2023-03-09T18:35:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:35:00 smithi038 ceph-mon[92422]: pgmap v7451: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 156 KiB/s wr, 10 op/s 2023-03-09T18:35:02.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:35:01 smithi122 ceph-mon[93869]: pgmap v7452: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 257 KiB/s wr, 13 op/s 2023-03-09T18:35:02.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:35:01 smithi181 ceph-mon[94189]: pgmap v7452: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 257 KiB/s wr, 13 op/s 2023-03-09T18:35:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:35:01 smithi038 ceph-mon[92422]: pgmap v7452: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 257 KiB/s wr, 13 op/s 2023-03-09T18:35:03.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:35:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13013.420537 secs 2023-03-09T18:35:03.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:35:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13013.420537 secs 2023-03-09T18:35:03.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:35:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13013.420537 secs 2023-03-09T18:35:03.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13475.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:35:04.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:35:03 smithi122 ceph-mon[93869]: pgmap v7453: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 173 KiB/s wr, 22 op/s 2023-03-09T18:35:04.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:35:03 smithi181 ceph-mon[94189]: pgmap v7453: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 173 KiB/s wr, 22 op/s 2023-03-09T18:35:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:35:03 smithi038 ceph-mon[92422]: pgmap v7453: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 173 KiB/s wr, 22 op/s 2023-03-09T18:35:05.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:35:04 smithi122 ceph-mon[93869]: osdmap e679: 12 total, 12 up, 12 in 2023-03-09T18:35:05.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:35:04 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13016.151603 secs 2023-03-09T18:35:05.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:35:04 smithi181 ceph-mon[94189]: osdmap e679: 12 total, 12 up, 12 in 2023-03-09T18:35:05.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:35:04 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13016.151603 secs 2023-03-09T18:35:05.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:35:04 smithi038 ceph-mon[92422]: osdmap e679: 12 total, 12 up, 12 in 2023-03-09T18:35:05.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:35:04 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13016.151603 secs 2023-03-09T18:35:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:35:05 smithi122 ceph-mon[93869]: pgmap v7455: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 294 KiB/s wr, 27 op/s 2023-03-09T18:35:06.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:35:05 smithi181 ceph-mon[94189]: pgmap v7455: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 294 KiB/s wr, 27 op/s 2023-03-09T18:35:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:35:05 smithi038 ceph-mon[92422]: pgmap v7455: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 294 KiB/s wr, 27 op/s 2023-03-09T18:35:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:35:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13018.420658 secs 2023-03-09T18:35:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:35:07 smithi122 ceph-mon[93869]: pgmap v7456: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 294 KiB/s wr, 27 op/s 2023-03-09T18:35:08.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:35:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13018.420658 secs 2023-03-09T18:35:08.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:35:07 smithi181 ceph-mon[94189]: pgmap v7456: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 294 KiB/s wr, 27 op/s 2023-03-09T18:35:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:35:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13018.420658 secs 2023-03-09T18:35:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:35:07 smithi038 ceph-mon[92422]: pgmap v7456: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 294 KiB/s wr, 27 op/s 2023-03-09T18:35:08.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13480.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:35:10.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:35:09 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13021.151664 secs 2023-03-09T18:35:10.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:35:09 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13021.151664 secs 2023-03-09T18:35:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:35:09 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13021.151664 secs 2023-03-09T18:35:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:35:10 smithi122 ceph-mon[93869]: pgmap v7457: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 294 KiB/s wr, 27 op/s 2023-03-09T18:35:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:35:10 smithi181 ceph-mon[94189]: pgmap v7457: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 294 KiB/s wr, 27 op/s 2023-03-09T18:35:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:35:10 smithi038 ceph-mon[92422]: pgmap v7457: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 294 KiB/s wr, 27 op/s 2023-03-09T18:35:12.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:35:11 smithi122 ceph-mon[93869]: pgmap v7458: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 184 KiB/s wr, 25 op/s 2023-03-09T18:35:12.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:35:11 smithi122 ceph-mon[93869]: osdmap e680: 12 total, 12 up, 12 in 2023-03-09T18:35:12.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:35:11 smithi181 ceph-mon[94189]: pgmap v7458: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 184 KiB/s wr, 25 op/s 2023-03-09T18:35:12.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:35:11 smithi181 ceph-mon[94189]: osdmap e680: 12 total, 12 up, 12 in 2023-03-09T18:35:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:35:11 smithi038 ceph-mon[92422]: pgmap v7458: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 184 KiB/s wr, 25 op/s 2023-03-09T18:35:12.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:35:11 smithi038 ceph-mon[92422]: osdmap e680: 12 total, 12 up, 12 in 2023-03-09T18:35:13.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:35:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13023.420777 secs 2023-03-09T18:35:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:35:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13023.420777 secs 2023-03-09T18:35:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:35:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13023.420777 secs 2023-03-09T18:35:13.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13485.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:35:14.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:35:13 smithi122 ceph-mon[93869]: pgmap v7460: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 160 KiB/s wr, 11 op/s 2023-03-09T18:35:14.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:35:13 smithi181 ceph-mon[94189]: pgmap v7460: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 160 KiB/s wr, 11 op/s 2023-03-09T18:35:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:35:13 smithi038 ceph-mon[92422]: pgmap v7460: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 160 KiB/s wr, 11 op/s 2023-03-09T18:35:15.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:35:14 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13026.151723 secs 2023-03-09T18:35:15.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:35:14 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13026.151723 secs 2023-03-09T18:35:15.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:35:14 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13026.151723 secs 2023-03-09T18:35:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:35:15 smithi122 ceph-mon[93869]: pgmap v7461: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 35 KiB/s wr, 8 op/s 2023-03-09T18:35:16.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:35:15 smithi181 ceph-mon[94189]: pgmap v7461: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 35 KiB/s wr, 8 op/s 2023-03-09T18:35:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:35:15 smithi038 ceph-mon[92422]: pgmap v7461: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 35 KiB/s wr, 8 op/s 2023-03-09T18:35:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:35:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13028.420878 secs 2023-03-09T18:35:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:35:17 smithi122 ceph-mon[93869]: pgmap v7462: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 35 KiB/s wr, 8 op/s 2023-03-09T18:35:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:35:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13028.420878 secs 2023-03-09T18:35:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:35:17 smithi181 ceph-mon[94189]: pgmap v7462: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 35 KiB/s wr, 8 op/s 2023-03-09T18:35:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:35:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13028.420878 secs 2023-03-09T18:35:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:35:17 smithi038 ceph-mon[92422]: pgmap v7462: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 35 KiB/s wr, 8 op/s 2023-03-09T18:35:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13490.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:35:20.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:35:19 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13031.151792 secs 2023-03-09T18:35:20.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:35:19 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13031.151792 secs 2023-03-09T18:35:20.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:35:19 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13031.151792 secs 2023-03-09T18:35:21.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:35:20 smithi122 ceph-mon[93869]: pgmap v7463: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 35 KiB/s wr, 8 op/s 2023-03-09T18:35:21.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:35:20 smithi181 ceph-mon[94189]: pgmap v7463: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 35 KiB/s wr, 8 op/s 2023-03-09T18:35:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:35:20 smithi038 ceph-mon[92422]: pgmap v7463: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 35 KiB/s wr, 8 op/s 2023-03-09T18:35:22.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:35:21 smithi122 ceph-mon[93869]: pgmap v7464: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T18:35:22.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:35:21 smithi181 ceph-mon[94189]: pgmap v7464: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T18:35:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:35:21 smithi038 ceph-mon[92422]: pgmap v7464: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T18:35:23.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:35:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13033.421003 secs 2023-03-09T18:35:23.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:35:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13033.421003 secs 2023-03-09T18:35:23.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:35:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13033.421003 secs 2023-03-09T18:35:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13495.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:35:24.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:35:23 smithi122 ceph-mon[93869]: pgmap v7465: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 6 op/s 2023-03-09T18:35:24.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:35:23 smithi181 ceph-mon[94189]: pgmap v7465: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 6 op/s 2023-03-09T18:35:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:35:23 smithi038 ceph-mon[92422]: pgmap v7465: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 6 op/s 2023-03-09T18:35:25.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:35:24 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13036.151880 secs 2023-03-09T18:35:25.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:35:24 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13036.151880 secs 2023-03-09T18:35:25.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:35:24 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13036.151880 secs 2023-03-09T18:35:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:35:25 smithi122 ceph-mon[93869]: pgmap v7466: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 8 op/s 2023-03-09T18:35:26.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:35:25 smithi181 ceph-mon[94189]: pgmap v7466: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 8 op/s 2023-03-09T18:35:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:35:25 smithi038 ceph-mon[92422]: pgmap v7466: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 12 KiB/s wr, 8 op/s 2023-03-09T18:35:27.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:35:26 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:35:27.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:35:26 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:35:27.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:35:26 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:35:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:35:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13038.421094 secs 2023-03-09T18:35:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:35:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:35:28.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:35:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:35:28.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:35:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:35:28.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:35:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:35:28.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:35:27 smithi122 ceph-mon[93869]: pgmap v7467: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 6 op/s 2023-03-09T18:35:28.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:35:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:35:28.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:35:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:35:28.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:35:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:35:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:35:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13038.421094 secs 2023-03-09T18:35:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:35:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:35:28.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:35:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:35:28.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:35:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:35:28.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:35:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:35:28.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:35:27 smithi181 ceph-mon[94189]: pgmap v7467: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 6 op/s 2023-03-09T18:35:28.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:35:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:35:28.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:35:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:35:28.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:35:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:35:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:35:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13038.421094 secs 2023-03-09T18:35:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:35:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:35:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:35:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:35:28.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:35:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:35:28.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:35:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:35:28.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:35:27 smithi038 ceph-mon[92422]: pgmap v7467: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 6 op/s 2023-03-09T18:35:28.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:35:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:35:28.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:35:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:35:28.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:35:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:35:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13500.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:35:30.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:35:29 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13041.151939 secs 2023-03-09T18:35:30.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:35:29 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13041.151939 secs 2023-03-09T18:35:30.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:35:29 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13041.151939 secs 2023-03-09T18:35:31.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:35:30 smithi122 ceph-mon[93869]: pgmap v7468: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 6 op/s 2023-03-09T18:35:31.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:35:30 smithi181 ceph-mon[94189]: pgmap v7468: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 6 op/s 2023-03-09T18:35:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:35:30 smithi038 ceph-mon[92422]: pgmap v7468: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 11 KiB/s wr, 6 op/s 2023-03-09T18:35:32.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:35:31 smithi122 ceph-mon[93869]: pgmap v7469: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 42 KiB/s wr, 8 op/s 2023-03-09T18:35:32.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:35:31 smithi181 ceph-mon[94189]: pgmap v7469: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 42 KiB/s wr, 8 op/s 2023-03-09T18:35:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:35:31 smithi038 ceph-mon[92422]: pgmap v7469: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 42 KiB/s wr, 8 op/s 2023-03-09T18:35:33.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:35:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13043.421209 secs 2023-03-09T18:35:33.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:35:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13043.421209 secs 2023-03-09T18:35:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:35:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13043.421209 secs 2023-03-09T18:35:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13505.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:35:34.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:35:33 smithi122 ceph-mon[93869]: pgmap v7470: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 41 KiB/s wr, 6 op/s 2023-03-09T18:35:34.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:35:33 smithi181 ceph-mon[94189]: pgmap v7470: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 41 KiB/s wr, 6 op/s 2023-03-09T18:35:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:35:33 smithi038 ceph-mon[92422]: pgmap v7470: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 41 KiB/s wr, 6 op/s 2023-03-09T18:35:35.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:35:34 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13046.151983 secs 2023-03-09T18:35:35.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:35:34 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13046.151983 secs 2023-03-09T18:35:35.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:35:34 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13046.151983 secs 2023-03-09T18:35:36.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:35:35 smithi122 ceph-mon[93869]: pgmap v7471: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 83 KiB/s wr, 9 op/s 2023-03-09T18:35:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:35:35 smithi181 ceph-mon[94189]: pgmap v7471: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 83 KiB/s wr, 9 op/s 2023-03-09T18:35:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:35:35 smithi038 ceph-mon[92422]: pgmap v7471: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 83 KiB/s wr, 9 op/s 2023-03-09T18:35:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:35:37 smithi122 ceph-mon[93869]: pgmap v7472: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 83 KiB/s wr, 6 op/s 2023-03-09T18:35:38.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:35:37 smithi181 ceph-mon[94189]: pgmap v7472: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 83 KiB/s wr, 6 op/s 2023-03-09T18:35:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:35:37 smithi038 ceph-mon[92422]: pgmap v7472: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 83 KiB/s wr, 6 op/s 2023-03-09T18:35:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13510.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:35:39.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:35:39 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:35:39.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:35:39 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:35:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:35:39 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:35:41.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:35:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13051.152042 secs 2023-03-09T18:35:41.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:35:40 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:35:41.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:35:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:35:41.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:35:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:35:41.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:35:40 smithi038 ceph-mon[92422]: pgmap v7473: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 104 KiB/s wr, 7 op/s 2023-03-09T18:35:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:35:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13051.152042 secs 2023-03-09T18:35:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:35:40 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:35:41.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:35:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:35:41.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:35:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:35:41.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:35:40 smithi122 ceph-mon[93869]: pgmap v7473: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 104 KiB/s wr, 7 op/s 2023-03-09T18:35:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:35:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13051.152042 secs 2023-03-09T18:35:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:35:40 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:35:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:35:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:35:41.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:35:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:35:41.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:35:40 smithi181 ceph-mon[94189]: pgmap v7473: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 104 KiB/s wr, 7 op/s 2023-03-09T18:35:42.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:35:41 smithi122 ceph-mon[93869]: pgmap v7474: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 229 KiB/s wr, 10 op/s 2023-03-09T18:35:42.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:35:41 smithi181 ceph-mon[94189]: pgmap v7474: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 229 KiB/s wr, 10 op/s 2023-03-09T18:35:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:35:41 smithi038 ceph-mon[92422]: pgmap v7474: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 229 KiB/s wr, 10 op/s 2023-03-09T18:35:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13515.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:35:44.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:35:43 smithi122 ceph-mon[93869]: pgmap v7475: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 188 KiB/s wr, 8 op/s 2023-03-09T18:35:44.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:35:43 smithi181 ceph-mon[94189]: pgmap v7475: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 188 KiB/s wr, 8 op/s 2023-03-09T18:35:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:35:43 smithi038 ceph-mon[92422]: pgmap v7475: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 188 KiB/s wr, 8 op/s 2023-03-09T18:35:45.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:35:44 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13056.152103 secs 2023-03-09T18:35:45.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:35:44 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13056.152103 secs 2023-03-09T18:35:45.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:35:44 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13056.152103 secs 2023-03-09T18:35:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:35:45 smithi122 ceph-mon[93869]: pgmap v7476: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 203 KiB/s wr, 11 op/s 2023-03-09T18:35:46.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:35:45 smithi181 ceph-mon[94189]: pgmap v7476: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 203 KiB/s wr, 11 op/s 2023-03-09T18:35:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:35:45 smithi038 ceph-mon[92422]: pgmap v7476: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 203 KiB/s wr, 11 op/s 2023-03-09T18:35:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:35:47 smithi122 ceph-mon[93869]: pgmap v7477: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 161 KiB/s wr, 8 op/s 2023-03-09T18:35:48.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:35:47 smithi181 ceph-mon[94189]: pgmap v7477: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 161 KiB/s wr, 8 op/s 2023-03-09T18:35:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:35:47 smithi038 ceph-mon[92422]: pgmap v7477: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 161 KiB/s wr, 8 op/s 2023-03-09T18:35:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13520.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:35:50.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:35:49 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13061.152162 secs 2023-03-09T18:35:50.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:35:49 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13061.152162 secs 2023-03-09T18:35:50.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:35:49 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13061.152162 secs 2023-03-09T18:35:51.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:35:50 smithi122 ceph-mon[93869]: pgmap v7478: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 172 KiB/s wr, 9 op/s 2023-03-09T18:35:51.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:35:50 smithi181 ceph-mon[94189]: pgmap v7478: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 172 KiB/s wr, 9 op/s 2023-03-09T18:35:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:35:50 smithi038 ceph-mon[92422]: pgmap v7478: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 172 KiB/s wr, 9 op/s 2023-03-09T18:35:52.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:35:51 smithi122 ceph-mon[93869]: pgmap v7479: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 223 KiB/s wr, 11 op/s 2023-03-09T18:35:52.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:35:51 smithi181 ceph-mon[94189]: pgmap v7479: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 223 KiB/s wr, 11 op/s 2023-03-09T18:35:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:35:51 smithi038 ceph-mon[92422]: pgmap v7479: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 223 KiB/s wr, 11 op/s 2023-03-09T18:35:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13525.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:35:54.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:35:53 smithi122 ceph-mon[93869]: pgmap v7480: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 165 KiB/s wr, 9 op/s 2023-03-09T18:35:54.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:35:53 smithi181 ceph-mon[94189]: pgmap v7480: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 165 KiB/s wr, 9 op/s 2023-03-09T18:35:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:35:53 smithi038 ceph-mon[92422]: pgmap v7480: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 165 KiB/s wr, 9 op/s 2023-03-09T18:35:55.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:35:54 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13066.152221 secs 2023-03-09T18:35:55.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:35:54 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13066.152221 secs 2023-03-09T18:35:55.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:35:54 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13066.152221 secs 2023-03-09T18:35:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:35:55 smithi122 ceph-mon[93869]: pgmap v7481: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 175 KiB/s wr, 12 op/s 2023-03-09T18:35:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:35:55 smithi181 ceph-mon[94189]: pgmap v7481: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 175 KiB/s wr, 12 op/s 2023-03-09T18:35:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:35:55 smithi038 ceph-mon[92422]: pgmap v7481: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 175 KiB/s wr, 12 op/s 2023-03-09T18:35:58.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:35:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13068.421829 secs 2023-03-09T18:35:58.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:35:57 smithi122 ceph-mon[93869]: pgmap v7482: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 237 KiB/s wr, 10 op/s 2023-03-09T18:35:58.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:35:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13068.421829 secs 2023-03-09T18:35:58.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:35:57 smithi181 ceph-mon[94189]: pgmap v7482: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 237 KiB/s wr, 10 op/s 2023-03-09T18:35:58.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:35:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13068.421829 secs 2023-03-09T18:35:58.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:35:57 smithi038 ceph-mon[92422]: pgmap v7482: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 237 KiB/s wr, 10 op/s 2023-03-09T18:35:58.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13530.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:36:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:35:59 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13071.152281 secs 2023-03-09T18:36:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:35:59 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:36:00.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:35:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:36:00.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:35:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:36:00.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:35:59 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13071.152281 secs 2023-03-09T18:36:00.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:35:59 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:36:00.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:35:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:36:00.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:35:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:36:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:35:59 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13071.152281 secs 2023-03-09T18:36:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:35:59 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:36:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:35:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:36:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:35:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:36:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:36:00 smithi122 ceph-mon[93869]: pgmap v7483: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 238 KiB/s wr, 11 op/s 2023-03-09T18:36:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:36:00 smithi181 ceph-mon[94189]: pgmap v7483: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 238 KiB/s wr, 11 op/s 2023-03-09T18:36:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:36:00 smithi038 ceph-mon[92422]: pgmap v7483: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 238 KiB/s wr, 11 op/s 2023-03-09T18:36:02.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:36:01 smithi122 ceph-mon[93869]: pgmap v7484: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 228 KiB/s wr, 12 op/s 2023-03-09T18:36:02.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:36:01 smithi181 ceph-mon[94189]: pgmap v7484: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 228 KiB/s wr, 12 op/s 2023-03-09T18:36:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:36:01 smithi038 ceph-mon[92422]: pgmap v7484: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 228 KiB/s wr, 12 op/s 2023-03-09T18:36:03.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:36:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13073.421977 secs 2023-03-09T18:36:03.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:36:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13073.421977 secs 2023-03-09T18:36:03.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:36:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13073.421977 secs 2023-03-09T18:36:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13535.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:36:04.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:36:03 smithi122 ceph-mon[93869]: pgmap v7485: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 201 KiB/s wr, 22 op/s 2023-03-09T18:36:04.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:36:03 smithi181 ceph-mon[94189]: pgmap v7485: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 201 KiB/s wr, 22 op/s 2023-03-09T18:36:04.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:36:03 smithi038 ceph-mon[92422]: pgmap v7485: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 201 KiB/s wr, 22 op/s 2023-03-09T18:36:05.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:36:04 smithi122 ceph-mon[93869]: osdmap e681: 12 total, 12 up, 12 in 2023-03-09T18:36:05.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:36:04 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13076.152343 secs 2023-03-09T18:36:05.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:36:04 smithi181 ceph-mon[94189]: osdmap e681: 12 total, 12 up, 12 in 2023-03-09T18:36:05.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:36:04 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13076.152343 secs 2023-03-09T18:36:05.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:36:04 smithi038 ceph-mon[92422]: osdmap e681: 12 total, 12 up, 12 in 2023-03-09T18:36:05.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:36:04 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13076.152343 secs 2023-03-09T18:36:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:36:05 smithi122 ceph-mon[93869]: pgmap v7487: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 186 KiB/s wr, 26 op/s 2023-03-09T18:36:06.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:36:05 smithi181 ceph-mon[94189]: pgmap v7487: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 186 KiB/s wr, 26 op/s 2023-03-09T18:36:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:36:05 smithi038 ceph-mon[92422]: pgmap v7487: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 186 KiB/s wr, 26 op/s 2023-03-09T18:36:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:36:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13078.422091 secs 2023-03-09T18:36:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:36:07 smithi122 ceph-mon[93869]: pgmap v7488: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 243 KiB/s wr, 26 op/s 2023-03-09T18:36:08.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:36:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13078.422091 secs 2023-03-09T18:36:08.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:36:07 smithi181 ceph-mon[94189]: pgmap v7488: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 243 KiB/s wr, 26 op/s 2023-03-09T18:36:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:36:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13078.422091 secs 2023-03-09T18:36:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:36:07 smithi038 ceph-mon[92422]: pgmap v7488: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 243 KiB/s wr, 26 op/s 2023-03-09T18:36:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13540.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:36:10.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:36:09 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13081.152402 secs 2023-03-09T18:36:10.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:36:09 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13081.152402 secs 2023-03-09T18:36:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:36:09 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13081.152402 secs 2023-03-09T18:36:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:36:10 smithi122 ceph-mon[93869]: pgmap v7489: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 254 KiB/s wr, 26 op/s 2023-03-09T18:36:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:36:10 smithi181 ceph-mon[94189]: pgmap v7489: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 254 KiB/s wr, 26 op/s 2023-03-09T18:36:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:36:10 smithi038 ceph-mon[92422]: pgmap v7489: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 254 KiB/s wr, 26 op/s 2023-03-09T18:36:12.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:36:11 smithi122 ceph-mon[93869]: pgmap v7490: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 254 KiB/s wr, 26 op/s 2023-03-09T18:36:12.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:36:11 smithi122 ceph-mon[93869]: osdmap e682: 12 total, 12 up, 12 in 2023-03-09T18:36:12.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:36:11 smithi181 ceph-mon[94189]: pgmap v7490: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 254 KiB/s wr, 26 op/s 2023-03-09T18:36:12.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:36:11 smithi181 ceph-mon[94189]: osdmap e682: 12 total, 12 up, 12 in 2023-03-09T18:36:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:36:11 smithi038 ceph-mon[92422]: pgmap v7490: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 254 KiB/s wr, 26 op/s 2023-03-09T18:36:12.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:36:11 smithi038 ceph-mon[92422]: osdmap e682: 12 total, 12 up, 12 in 2023-03-09T18:36:13.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:36:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13083.422220 secs 2023-03-09T18:36:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:36:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13083.422220 secs 2023-03-09T18:36:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:36:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13083.422220 secs 2023-03-09T18:36:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13545.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:36:14.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:36:13 smithi122 ceph-mon[93869]: pgmap v7492: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 203 KiB/s wr, 11 op/s 2023-03-09T18:36:14.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:36:13 smithi181 ceph-mon[94189]: pgmap v7492: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 203 KiB/s wr, 11 op/s 2023-03-09T18:36:14.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:36:13 smithi038 ceph-mon[92422]: pgmap v7492: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 203 KiB/s wr, 11 op/s 2023-03-09T18:36:15.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:36:14 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13086.152462 secs 2023-03-09T18:36:15.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:36:14 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13086.152462 secs 2023-03-09T18:36:15.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:36:14 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13086.152462 secs 2023-03-09T18:36:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:36:15 smithi122 ceph-mon[93869]: pgmap v7493: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 175 KiB/s wr, 8 op/s 2023-03-09T18:36:16.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:36:15 smithi181 ceph-mon[94189]: pgmap v7493: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 175 KiB/s wr, 8 op/s 2023-03-09T18:36:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:36:15 smithi038 ceph-mon[92422]: pgmap v7493: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 175 KiB/s wr, 8 op/s 2023-03-09T18:36:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:36:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13088.422337 secs 2023-03-09T18:36:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:36:17 smithi122 ceph-mon[93869]: pgmap v7494: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 26 KiB/s wr, 8 op/s 2023-03-09T18:36:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:36:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13088.422337 secs 2023-03-09T18:36:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:36:17 smithi181 ceph-mon[94189]: pgmap v7494: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 26 KiB/s wr, 8 op/s 2023-03-09T18:36:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:36:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13088.422337 secs 2023-03-09T18:36:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:36:17 smithi038 ceph-mon[92422]: pgmap v7494: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 26 KiB/s wr, 8 op/s 2023-03-09T18:36:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13550.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:36:20.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:36:19 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13091.152520 secs 2023-03-09T18:36:20.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:36:19 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13091.152520 secs 2023-03-09T18:36:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:36:19 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13091.152520 secs 2023-03-09T18:36:21.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:36:20 smithi122 ceph-mon[93869]: pgmap v7495: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T18:36:21.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:36:20 smithi181 ceph-mon[94189]: pgmap v7495: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T18:36:21.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:36:20 smithi038 ceph-mon[92422]: pgmap v7495: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T18:36:22.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:36:21 smithi122 ceph-mon[93869]: pgmap v7496: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 8 op/s 2023-03-09T18:36:22.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:36:21 smithi181 ceph-mon[94189]: pgmap v7496: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 8 op/s 2023-03-09T18:36:22.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:36:21 smithi038 ceph-mon[92422]: pgmap v7496: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 8 op/s 2023-03-09T18:36:23.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:36:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13093.422460 secs 2023-03-09T18:36:23.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:36:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13093.422460 secs 2023-03-09T18:36:23.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:36:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13093.422460 secs 2023-03-09T18:36:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13555.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:36:24.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:36:23 smithi122 ceph-mon[93869]: pgmap v7497: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 7 op/s 2023-03-09T18:36:24.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:36:23 smithi181 ceph-mon[94189]: pgmap v7497: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 7 op/s 2023-03-09T18:36:24.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:36:23 smithi038 ceph-mon[92422]: pgmap v7497: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 7 op/s 2023-03-09T18:36:25.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:36:24 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13096.152582 secs 2023-03-09T18:36:25.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:36:24 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13096.152582 secs 2023-03-09T18:36:25.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:36:24 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13096.152582 secs 2023-03-09T18:36:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:36:25 smithi122 ceph-mon[93869]: pgmap v7498: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 53 KiB/s wr, 9 op/s 2023-03-09T18:36:26.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:36:25 smithi181 ceph-mon[94189]: pgmap v7498: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 53 KiB/s wr, 9 op/s 2023-03-09T18:36:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:36:25 smithi038 ceph-mon[92422]: pgmap v7498: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 53 KiB/s wr, 9 op/s 2023-03-09T18:36:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:36:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13098.422577 secs 2023-03-09T18:36:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:36:27 smithi122 ceph-mon[93869]: pgmap v7499: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 7 op/s 2023-03-09T18:36:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:36:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13098.422577 secs 2023-03-09T18:36:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:36:27 smithi181 ceph-mon[94189]: pgmap v7499: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 7 op/s 2023-03-09T18:36:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:36:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13098.422577 secs 2023-03-09T18:36:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:36:27 smithi038 ceph-mon[92422]: pgmap v7499: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 7 op/s 2023-03-09T18:36:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13560.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:36:29.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:36:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:36:29.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:36:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:36:29.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:36:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:36:29.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:36:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:36:29.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:36:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:36:29.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:36:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:36:29.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:36:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:36:29.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:36:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:36:29.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:36:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:36:29.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:36:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:36:29.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:36:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:36:29.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:36:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:36:30.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:36:29 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13101.152668 secs 2023-03-09T18:36:30.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:36:29 smithi122 ceph-mon[93869]: pgmap v7500: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 43 KiB/s wr, 7 op/s 2023-03-09T18:36:30.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:36:29 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13101.152668 secs 2023-03-09T18:36:30.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:36:29 smithi181 ceph-mon[94189]: pgmap v7500: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 43 KiB/s wr, 7 op/s 2023-03-09T18:36:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:36:29 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13101.152668 secs 2023-03-09T18:36:30.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:36:29 smithi038 ceph-mon[92422]: pgmap v7500: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 43 KiB/s wr, 7 op/s 2023-03-09T18:36:32.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:36:31 smithi122 ceph-mon[93869]: pgmap v7501: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 122 KiB/s wr, 10 op/s 2023-03-09T18:36:32.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:36:31 smithi181 ceph-mon[94189]: pgmap v7501: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 122 KiB/s wr, 10 op/s 2023-03-09T18:36:32.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:36:31 smithi038 ceph-mon[92422]: pgmap v7501: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 122 KiB/s wr, 10 op/s 2023-03-09T18:36:33.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:36:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13103.422699 secs 2023-03-09T18:36:33.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:36:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13103.422699 secs 2023-03-09T18:36:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:36:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13103.422699 secs 2023-03-09T18:36:33.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13565.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:36:34.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:36:33 smithi122 ceph-mon[93869]: pgmap v7502: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 121 KiB/s wr, 7 op/s 2023-03-09T18:36:34.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:36:33 smithi181 ceph-mon[94189]: pgmap v7502: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 121 KiB/s wr, 7 op/s 2023-03-09T18:36:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:36:33 smithi038 ceph-mon[92422]: pgmap v7502: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 121 KiB/s wr, 7 op/s 2023-03-09T18:36:35.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:36:34 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13106.152730 secs 2023-03-09T18:36:35.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:36:34 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13106.152730 secs 2023-03-09T18:36:35.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:36:34 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13106.152730 secs 2023-03-09T18:36:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:36:35 smithi122 ceph-mon[93869]: pgmap v7503: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 122 KiB/s wr, 9 op/s 2023-03-09T18:36:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:36:35 smithi181 ceph-mon[94189]: pgmap v7503: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 122 KiB/s wr, 9 op/s 2023-03-09T18:36:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:36:35 smithi038 ceph-mon[92422]: pgmap v7503: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 122 KiB/s wr, 9 op/s 2023-03-09T18:36:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:36:37 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 13108.422808 secs 2023-03-09T18:36:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:36:37 smithi122 ceph-mon[93869]: pgmap v7504: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 122 KiB/s wr, 7 op/s 2023-03-09T18:36:38.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:36:37 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 13108.422808 secs 2023-03-09T18:36:38.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:36:37 smithi181 ceph-mon[94189]: pgmap v7504: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 122 KiB/s wr, 7 op/s 2023-03-09T18:36:38.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:36:37 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 13108.422808 secs 2023-03-09T18:36:38.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:36:37 smithi038 ceph-mon[92422]: pgmap v7504: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 122 KiB/s wr, 7 op/s 2023-03-09T18:36:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13570.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:36:41.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:36:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13111.152791 secs 2023-03-09T18:36:41.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:36:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:36:41.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:36:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:36:41.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:36:40 smithi038 ceph-mon[92422]: pgmap v7505: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 127 KiB/s wr, 7 op/s 2023-03-09T18:36:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:36:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13111.152791 secs 2023-03-09T18:36:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:36:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:36:41.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:36:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:36:41.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:36:40 smithi122 ceph-mon[93869]: pgmap v7505: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 127 KiB/s wr, 7 op/s 2023-03-09T18:36:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:36:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13111.152791 secs 2023-03-09T18:36:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:36:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:36:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:36:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:36:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:36:40 smithi181 ceph-mon[94189]: pgmap v7505: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 127 KiB/s wr, 7 op/s 2023-03-09T18:36:42.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:36:41 smithi122 ceph-mon[93869]: pgmap v7506: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 207 KiB/s wr, 10 op/s 2023-03-09T18:36:42.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:36:41 smithi181 ceph-mon[94189]: pgmap v7506: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 207 KiB/s wr, 10 op/s 2023-03-09T18:36:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:36:41 smithi038 ceph-mon[92422]: pgmap v7506: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 207 KiB/s wr, 10 op/s 2023-03-09T18:36:43.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:36:42 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 13113.422933 secs 2023-03-09T18:36:43.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:36:42 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 13113.422933 secs 2023-03-09T18:36:43.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:36:42 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 13113.422933 secs 2023-03-09T18:36:43.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13575.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:36:44.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:36:43 smithi122 ceph-mon[93869]: pgmap v7507: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 128 KiB/s wr, 8 op/s 2023-03-09T18:36:44.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:36:43 smithi181 ceph-mon[94189]: pgmap v7507: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 128 KiB/s wr, 8 op/s 2023-03-09T18:36:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:36:43 smithi038 ceph-mon[92422]: pgmap v7507: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 128 KiB/s wr, 8 op/s 2023-03-09T18:36:45.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:36:44 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13116.152850 secs 2023-03-09T18:36:45.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:36:44 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13116.152850 secs 2023-03-09T18:36:45.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:36:44 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13116.152850 secs 2023-03-09T18:36:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:36:45 smithi122 ceph-mon[93869]: pgmap v7508: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 196 KiB/s wr, 11 op/s 2023-03-09T18:36:46.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:36:45 smithi181 ceph-mon[94189]: pgmap v7508: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 196 KiB/s wr, 11 op/s 2023-03-09T18:36:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:36:45 smithi038 ceph-mon[92422]: pgmap v7508: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 196 KiB/s wr, 11 op/s 2023-03-09T18:36:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:36:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13118.423030 secs 2023-03-09T18:36:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:36:47 smithi122 ceph-mon[93869]: pgmap v7509: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 166 KiB/s wr, 9 op/s 2023-03-09T18:36:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:36:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13118.423030 secs 2023-03-09T18:36:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:36:47 smithi181 ceph-mon[94189]: pgmap v7509: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 166 KiB/s wr, 9 op/s 2023-03-09T18:36:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:36:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13118.423030 secs 2023-03-09T18:36:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:36:47 smithi038 ceph-mon[92422]: pgmap v7509: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 166 KiB/s wr, 9 op/s 2023-03-09T18:36:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13580.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:36:49.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:36:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:36:49.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:36:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:36:49.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:36:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:36:49.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:36:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:36:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:36:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:36:49.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:36:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:36:50.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:36:49 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13121.152908 secs 2023-03-09T18:36:50.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:36:49 smithi122 ceph-mon[93869]: pgmap v7510: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 166 KiB/s wr, 10 op/s 2023-03-09T18:36:50.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:36:49 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13121.152908 secs 2023-03-09T18:36:50.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:36:49 smithi181 ceph-mon[94189]: pgmap v7510: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 166 KiB/s wr, 10 op/s 2023-03-09T18:36:50.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:36:49 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13121.152908 secs 2023-03-09T18:36:50.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:36:49 smithi038 ceph-mon[92422]: pgmap v7510: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 166 KiB/s wr, 10 op/s 2023-03-09T18:36:52.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:36:51 smithi122 ceph-mon[93869]: pgmap v7511: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 293 KiB/s rd, 278 KiB/s wr, 14 op/s 2023-03-09T18:36:52.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:36:51 smithi181 ceph-mon[94189]: pgmap v7511: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 293 KiB/s rd, 278 KiB/s wr, 14 op/s 2023-03-09T18:36:52.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:36:51 smithi038 ceph-mon[92422]: pgmap v7511: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 293 KiB/s rd, 278 KiB/s wr, 14 op/s 2023-03-09T18:36:53.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:36:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13123.423176 secs 2023-03-09T18:36:53.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:36:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13123.423176 secs 2023-03-09T18:36:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:36:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13123.423176 secs 2023-03-09T18:36:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13585.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:36:54.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:36:53 smithi122 ceph-mon[93869]: pgmap v7512: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 198 KiB/s wr, 10 op/s 2023-03-09T18:36:54.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:36:53 smithi181 ceph-mon[94189]: pgmap v7512: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 198 KiB/s wr, 10 op/s 2023-03-09T18:36:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:36:53 smithi038 ceph-mon[92422]: pgmap v7512: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 198 KiB/s wr, 10 op/s 2023-03-09T18:36:55.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:36:54 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13126.152966 secs 2023-03-09T18:36:55.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:36:54 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13126.152966 secs 2023-03-09T18:36:55.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:36:54 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13126.152966 secs 2023-03-09T18:36:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:36:55 smithi122 ceph-mon[93869]: pgmap v7513: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 218 KiB/s wr, 14 op/s 2023-03-09T18:36:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:36:55 smithi181 ceph-mon[94189]: pgmap v7513: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 218 KiB/s wr, 14 op/s 2023-03-09T18:36:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:36:55 smithi038 ceph-mon[92422]: pgmap v7513: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 218 KiB/s wr, 14 op/s 2023-03-09T18:36:58.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:36:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13128.423322 secs 2023-03-09T18:36:58.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:36:57 smithi122 ceph-mon[93869]: pgmap v7514: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 161 KiB/s wr, 11 op/s 2023-03-09T18:36:58.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:36:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13128.423322 secs 2023-03-09T18:36:58.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:36:57 smithi181 ceph-mon[94189]: pgmap v7514: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 161 KiB/s wr, 11 op/s 2023-03-09T18:36:58.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:36:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13128.423322 secs 2023-03-09T18:36:58.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:36:57 smithi038 ceph-mon[92422]: pgmap v7514: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 161 KiB/s wr, 11 op/s 2023-03-09T18:36:58.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13590.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:37:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:36:59 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13131.153036 secs 2023-03-09T18:37:00.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:36:59 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13131.153036 secs 2023-03-09T18:37:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:36:59 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13131.153036 secs 2023-03-09T18:37:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:37:00 smithi122 ceph-mon[93869]: pgmap v7515: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 149 KiB/s wr, 10 op/s 2023-03-09T18:37:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:37:00 smithi181 ceph-mon[94189]: pgmap v7515: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 149 KiB/s wr, 10 op/s 2023-03-09T18:37:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:37:00 smithi038 ceph-mon[92422]: pgmap v7515: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 149 KiB/s wr, 10 op/s 2023-03-09T18:37:02.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:37:01 smithi122 ceph-mon[93869]: pgmap v7516: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 247 KiB/s rd, 244 KiB/s wr, 14 op/s 2023-03-09T18:37:02.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:37:01 smithi181 ceph-mon[94189]: pgmap v7516: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 247 KiB/s rd, 244 KiB/s wr, 14 op/s 2023-03-09T18:37:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:37:01 smithi038 ceph-mon[92422]: pgmap v7516: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 247 KiB/s rd, 244 KiB/s wr, 14 op/s 2023-03-09T18:37:03.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:37:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13133.423404 secs 2023-03-09T18:37:03.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:37:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13133.423404 secs 2023-03-09T18:37:03.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:37:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13133.423404 secs 2023-03-09T18:37:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13595.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:37:04.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:37:03 smithi122 ceph-mon[93869]: pgmap v7517: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 148 KiB/s wr, 21 op/s 2023-03-09T18:37:04.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:37:03 smithi181 ceph-mon[94189]: pgmap v7517: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 148 KiB/s wr, 21 op/s 2023-03-09T18:37:04.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:37:03 smithi038 ceph-mon[92422]: pgmap v7517: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 148 KiB/s wr, 21 op/s 2023-03-09T18:37:05.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:37:04 smithi122 ceph-mon[93869]: osdmap e683: 12 total, 12 up, 12 in 2023-03-09T18:37:05.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:37:04 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13136.153094 secs 2023-03-09T18:37:05.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:37:04 smithi181 ceph-mon[94189]: osdmap e683: 12 total, 12 up, 12 in 2023-03-09T18:37:05.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:37:04 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13136.153094 secs 2023-03-09T18:37:05.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:37:04 smithi038 ceph-mon[92422]: osdmap e683: 12 total, 12 up, 12 in 2023-03-09T18:37:05.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:37:04 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13136.153094 secs 2023-03-09T18:37:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:37:05 smithi122 ceph-mon[93869]: pgmap v7519: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 287 KiB/s wr, 28 op/s 2023-03-09T18:37:06.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:37:05 smithi181 ceph-mon[94189]: pgmap v7519: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 287 KiB/s wr, 28 op/s 2023-03-09T18:37:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:37:05 smithi038 ceph-mon[92422]: pgmap v7519: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 287 KiB/s wr, 28 op/s 2023-03-09T18:37:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:37:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13138.423480 secs 2023-03-09T18:37:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:37:07 smithi122 ceph-mon[93869]: pgmap v7520: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 274 KiB/s wr, 27 op/s 2023-03-09T18:37:08.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:37:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13138.423480 secs 2023-03-09T18:37:08.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:37:07 smithi181 ceph-mon[94189]: pgmap v7520: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 274 KiB/s wr, 27 op/s 2023-03-09T18:37:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:37:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13138.423480 secs 2023-03-09T18:37:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:37:07 smithi038 ceph-mon[92422]: pgmap v7520: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 274 KiB/s wr, 27 op/s 2023-03-09T18:37:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13600.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:37:10.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:37:09 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13141.153157 secs 2023-03-09T18:37:10.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:37:09 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13141.153157 secs 2023-03-09T18:37:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:37:09 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13141.153157 secs 2023-03-09T18:37:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:37:10 smithi122 ceph-mon[93869]: pgmap v7521: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 274 KiB/s wr, 27 op/s 2023-03-09T18:37:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:37:10 smithi181 ceph-mon[94189]: pgmap v7521: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 274 KiB/s wr, 27 op/s 2023-03-09T18:37:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:37:10 smithi038 ceph-mon[92422]: pgmap v7521: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 274 KiB/s wr, 27 op/s 2023-03-09T18:37:12.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:37:11 smithi122 ceph-mon[93869]: pgmap v7522: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 206 KiB/s wr, 27 op/s 2023-03-09T18:37:12.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:37:11 smithi181 ceph-mon[94189]: pgmap v7522: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 206 KiB/s wr, 27 op/s 2023-03-09T18:37:12.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:37:11 smithi038 ceph-mon[92422]: pgmap v7522: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 206 KiB/s wr, 27 op/s 2023-03-09T18:37:13.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:37:12 smithi122 ceph-mon[93869]: osdmap e684: 12 total, 12 up, 12 in 2023-03-09T18:37:13.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:37:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13143.423603 secs 2023-03-09T18:37:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:37:12 smithi181 ceph-mon[94189]: osdmap e684: 12 total, 12 up, 12 in 2023-03-09T18:37:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:37:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13143.423603 secs 2023-03-09T18:37:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:37:12 smithi038 ceph-mon[92422]: osdmap e684: 12 total, 12 up, 12 in 2023-03-09T18:37:13.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:37:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13143.423603 secs 2023-03-09T18:37:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13605.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:37:14.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:37:13 smithi122 ceph-mon[93869]: pgmap v7524: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 182 KiB/s wr, 12 op/s 2023-03-09T18:37:14.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:37:13 smithi181 ceph-mon[94189]: pgmap v7524: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 182 KiB/s wr, 12 op/s 2023-03-09T18:37:14.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:37:13 smithi038 ceph-mon[92422]: pgmap v7524: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 182 KiB/s wr, 12 op/s 2023-03-09T18:37:15.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:37:14 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13146.153217 secs 2023-03-09T18:37:15.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:37:14 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13146.153217 secs 2023-03-09T18:37:15.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:37:14 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13146.153217 secs 2023-03-09T18:37:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:37:15 smithi122 ceph-mon[93869]: pgmap v7525: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 47 KiB/s wr, 8 op/s 2023-03-09T18:37:16.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:37:15 smithi181 ceph-mon[94189]: pgmap v7525: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 47 KiB/s wr, 8 op/s 2023-03-09T18:37:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:37:15 smithi038 ceph-mon[92422]: pgmap v7525: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 47 KiB/s wr, 8 op/s 2023-03-09T18:37:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:37:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13148.423758 secs 2023-03-09T18:37:18.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:37:17 smithi122 ceph-mon[93869]: pgmap v7526: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 47 KiB/s wr, 7 op/s 2023-03-09T18:37:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:37:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13148.423758 secs 2023-03-09T18:37:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:37:17 smithi181 ceph-mon[94189]: pgmap v7526: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 47 KiB/s wr, 7 op/s 2023-03-09T18:37:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:37:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13148.423758 secs 2023-03-09T18:37:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:37:17 smithi038 ceph-mon[92422]: pgmap v7526: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 47 KiB/s wr, 7 op/s 2023-03-09T18:37:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13610.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:37:20.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:37:19 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13151.153274 secs 2023-03-09T18:37:20.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:37:19 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13151.153274 secs 2023-03-09T18:37:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:37:19 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13151.153274 secs 2023-03-09T18:37:21.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:37:20 smithi122 ceph-mon[93869]: pgmap v7527: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 47 KiB/s wr, 7 op/s 2023-03-09T18:37:21.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:37:20 smithi181 ceph-mon[94189]: pgmap v7527: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 47 KiB/s wr, 7 op/s 2023-03-09T18:37:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:37:20 smithi038 ceph-mon[92422]: pgmap v7527: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 47 KiB/s wr, 7 op/s 2023-03-09T18:37:22.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:37:21 smithi122 ceph-mon[93869]: pgmap v7528: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 40 KiB/s wr, 7 op/s 2023-03-09T18:37:22.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:37:21 smithi181 ceph-mon[94189]: pgmap v7528: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 40 KiB/s wr, 7 op/s 2023-03-09T18:37:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:37:21 smithi038 ceph-mon[92422]: pgmap v7528: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 40 KiB/s wr, 7 op/s 2023-03-09T18:37:23.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:37:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13153.423880 secs 2023-03-09T18:37:23.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:37:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13153.423880 secs 2023-03-09T18:37:23.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:37:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13153.423880 secs 2023-03-09T18:37:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13615.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:37:24.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:37:23 smithi122 ceph-mon[93869]: pgmap v7529: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 35 KiB/s wr, 6 op/s 2023-03-09T18:37:24.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:37:23 smithi181 ceph-mon[94189]: pgmap v7529: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 35 KiB/s wr, 6 op/s 2023-03-09T18:37:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:37:23 smithi038 ceph-mon[92422]: pgmap v7529: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 35 KiB/s wr, 6 op/s 2023-03-09T18:37:25.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:37:24 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13156.153348 secs 2023-03-09T18:37:25.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:37:24 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13156.153348 secs 2023-03-09T18:37:25.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:37:24 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13156.153348 secs 2023-03-09T18:37:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:37:25 smithi122 ceph-mon[93869]: pgmap v7530: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 34 KiB/s wr, 8 op/s 2023-03-09T18:37:26.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:37:25 smithi181 ceph-mon[94189]: pgmap v7530: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 34 KiB/s wr, 8 op/s 2023-03-09T18:37:26.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:37:25 smithi038 ceph-mon[92422]: pgmap v7530: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 34 KiB/s wr, 8 op/s 2023-03-09T18:37:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:37:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13158.423943 secs 2023-03-09T18:37:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:37:27 smithi122 ceph-mon[93869]: pgmap v7531: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 35 KiB/s wr, 7 op/s 2023-03-09T18:37:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:37:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13158.423943 secs 2023-03-09T18:37:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:37:27 smithi181 ceph-mon[94189]: pgmap v7531: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 35 KiB/s wr, 7 op/s 2023-03-09T18:37:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:37:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13158.423943 secs 2023-03-09T18:37:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:37:27 smithi038 ceph-mon[92422]: pgmap v7531: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 35 KiB/s wr, 7 op/s 2023-03-09T18:37:28.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13620.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:37:29.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:37:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:37:29.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:37:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:37:29.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:37:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:37:30.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:37:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:37:30.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:37:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:37:30.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:37:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:37:30.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:37:29 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13161.153411 secs 2023-03-09T18:37:30.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:37:29 smithi122 ceph-mon[93869]: pgmap v7532: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 35 KiB/s wr, 6 op/s 2023-03-09T18:37:30.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:37:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:37:30.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:37:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:37:30.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:37:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:37:30.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:37:29 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13161.153411 secs 2023-03-09T18:37:30.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:37:29 smithi181 ceph-mon[94189]: pgmap v7532: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 35 KiB/s wr, 6 op/s 2023-03-09T18:37:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:37:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:37:30.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:37:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:37:30.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:37:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:37:30.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:37:29 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13161.153411 secs 2023-03-09T18:37:30.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:37:29 smithi038 ceph-mon[92422]: pgmap v7532: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 35 KiB/s wr, 6 op/s 2023-03-09T18:37:32.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:37:31 smithi122 ceph-mon[93869]: pgmap v7533: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 76 KiB/s wr, 9 op/s 2023-03-09T18:37:32.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:37:31 smithi181 ceph-mon[94189]: pgmap v7533: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 76 KiB/s wr, 9 op/s 2023-03-09T18:37:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:37:31 smithi038 ceph-mon[92422]: pgmap v7533: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 76 KiB/s wr, 9 op/s 2023-03-09T18:37:33.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:37:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13163.424072 secs 2023-03-09T18:37:33.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:37:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13163.424072 secs 2023-03-09T18:37:33.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:37:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13163.424072 secs 2023-03-09T18:37:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13625.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:37:34.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:37:33 smithi122 ceph-mon[93869]: pgmap v7534: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 6 op/s 2023-03-09T18:37:34.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:37:33 smithi181 ceph-mon[94189]: pgmap v7534: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 6 op/s 2023-03-09T18:37:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:37:33 smithi038 ceph-mon[92422]: pgmap v7534: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 6 op/s 2023-03-09T18:37:35.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:37:34 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13166.153471 secs 2023-03-09T18:37:35.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:37:34 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13166.153471 secs 2023-03-09T18:37:35.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:37:34 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13166.153471 secs 2023-03-09T18:37:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:37:35 smithi122 ceph-mon[93869]: pgmap v7535: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 84 KiB/s wr, 9 op/s 2023-03-09T18:37:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:37:35 smithi181 ceph-mon[94189]: pgmap v7535: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 84 KiB/s wr, 9 op/s 2023-03-09T18:37:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:37:35 smithi038 ceph-mon[92422]: pgmap v7535: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 84 KiB/s wr, 9 op/s 2023-03-09T18:37:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:37:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13168.424227 secs 2023-03-09T18:37:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:37:37 smithi122 ceph-mon[93869]: pgmap v7536: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 85 KiB/s wr, 7 op/s 2023-03-09T18:37:38.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:37:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13168.424227 secs 2023-03-09T18:37:38.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:37:37 smithi181 ceph-mon[94189]: pgmap v7536: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 85 KiB/s wr, 7 op/s 2023-03-09T18:37:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:37:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13168.424227 secs 2023-03-09T18:37:38.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:37:37 smithi038 ceph-mon[92422]: pgmap v7536: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 85 KiB/s wr, 7 op/s 2023-03-09T18:37:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13630.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:37:40.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:37:39 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13171.153531 secs 2023-03-09T18:37:40.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:37:39 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13171.153531 secs 2023-03-09T18:37:40.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:37:39 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13171.153531 secs 2023-03-09T18:37:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:37:40 smithi122 ceph-mon[93869]: pgmap v7537: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 84 KiB/s wr, 7 op/s 2023-03-09T18:37:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:37:40 smithi181 ceph-mon[94189]: pgmap v7537: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 84 KiB/s wr, 7 op/s 2023-03-09T18:37:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:37:40 smithi038 ceph-mon[92422]: pgmap v7537: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 84 KiB/s wr, 7 op/s 2023-03-09T18:37:42.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:37:41 smithi122 ceph-mon[93869]: pgmap v7538: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 232 KiB/s wr, 11 op/s 2023-03-09T18:37:42.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:37:41 smithi181 ceph-mon[94189]: pgmap v7538: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 232 KiB/s wr, 11 op/s 2023-03-09T18:37:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:37:41 smithi038 ceph-mon[92422]: pgmap v7538: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 232 KiB/s wr, 11 op/s 2023-03-09T18:37:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13635.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:37:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:37:43 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:37:44.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:37:43 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:37:44.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:37:43 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:37:45.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:37:44 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:37:45.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:37:44 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:37:45.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:37:44 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:37:45.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:37:44 smithi038 ceph-mon[92422]: pgmap v7539: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 191 KiB/s wr, 9 op/s 2023-03-09T18:37:45.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:37:44 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:37:45.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:37:44 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:37:45.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:37:44 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:37:45.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:37:44 smithi122 ceph-mon[93869]: pgmap v7539: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 191 KiB/s wr, 9 op/s 2023-03-09T18:37:45.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:37:44 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:37:45.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:37:44 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:37:45.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:37:44 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:37:45.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:37:44 smithi181 ceph-mon[94189]: pgmap v7539: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 191 KiB/s wr, 9 op/s 2023-03-09T18:37:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:37:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13176.153601 secs 2023-03-09T18:37:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:37:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13176.153601 secs 2023-03-09T18:37:46.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:37:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13176.153601 secs 2023-03-09T18:37:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:37:46 smithi038 ceph-mon[92422]: pgmap v7540: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 207 KiB/s wr, 11 op/s 2023-03-09T18:37:47.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:37:46 smithi122 ceph-mon[93869]: pgmap v7540: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 207 KiB/s wr, 11 op/s 2023-03-09T18:37:47.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:37:46 smithi181 ceph-mon[94189]: pgmap v7540: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 207 KiB/s wr, 11 op/s 2023-03-09T18:37:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:37:47 smithi122 ceph-mon[93869]: pgmap v7541: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 164 KiB/s wr, 9 op/s 2023-03-09T18:37:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:37:47 smithi181 ceph-mon[94189]: pgmap v7541: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 164 KiB/s wr, 9 op/s 2023-03-09T18:37:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:37:47 smithi038 ceph-mon[92422]: pgmap v7541: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 164 KiB/s wr, 9 op/s 2023-03-09T18:37:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13640.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:37:50.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:37:49 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13181.153665 secs 2023-03-09T18:37:50.262 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:37:49 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13181.153665 secs 2023-03-09T18:37:50.262 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:37:49 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13181.153665 secs 2023-03-09T18:37:51.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:37:50 smithi122 ceph-mon[93869]: pgmap v7542: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 164 KiB/s wr, 9 op/s 2023-03-09T18:37:51.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:37:50 smithi181 ceph-mon[94189]: pgmap v7542: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 164 KiB/s wr, 9 op/s 2023-03-09T18:37:51.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:37:50 smithi038 ceph-mon[92422]: pgmap v7542: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 164 KiB/s wr, 9 op/s 2023-03-09T18:37:52.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:37:51 smithi122 ceph-mon[93869]: pgmap v7543: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 380 KiB/s wr, 13 op/s 2023-03-09T18:37:52.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:37:51 smithi181 ceph-mon[94189]: pgmap v7543: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 380 KiB/s wr, 13 op/s 2023-03-09T18:37:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:37:51 smithi038 ceph-mon[92422]: pgmap v7543: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 380 KiB/s wr, 13 op/s 2023-03-09T18:37:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13645.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:37:54.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:37:53 smithi122 ceph-mon[93869]: pgmap v7544: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 232 KiB/s wr, 9 op/s 2023-03-09T18:37:54.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:37:53 smithi181 ceph-mon[94189]: pgmap v7544: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 232 KiB/s wr, 9 op/s 2023-03-09T18:37:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:37:53 smithi038 ceph-mon[92422]: pgmap v7544: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 232 KiB/s wr, 9 op/s 2023-03-09T18:37:55.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:37:54 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13186.153723 secs 2023-03-09T18:37:55.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:37:54 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13186.153723 secs 2023-03-09T18:37:55.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:37:54 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13186.153723 secs 2023-03-09T18:37:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:37:55 smithi122 ceph-mon[93869]: pgmap v7545: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 313 KiB/s rd, 311 KiB/s wr, 11 op/s 2023-03-09T18:37:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:37:55 smithi181 ceph-mon[94189]: pgmap v7545: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 313 KiB/s rd, 311 KiB/s wr, 11 op/s 2023-03-09T18:37:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:37:55 smithi038 ceph-mon[92422]: pgmap v7545: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 313 KiB/s rd, 311 KiB/s wr, 11 op/s 2023-03-09T18:37:58.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:37:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13188.424719 secs 2023-03-09T18:37:58.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:37:57 smithi122 ceph-mon[93869]: pgmap v7546: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 307 KiB/s wr, 10 op/s 2023-03-09T18:37:58.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:37:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13188.424719 secs 2023-03-09T18:37:58.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:37:57 smithi181 ceph-mon[94189]: pgmap v7546: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 307 KiB/s wr, 10 op/s 2023-03-09T18:37:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:37:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13188.424719 secs 2023-03-09T18:37:58.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:37:57 smithi038 ceph-mon[92422]: pgmap v7546: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 307 KiB/s wr, 10 op/s 2023-03-09T18:37:58.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13650.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:38:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:37:59 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13191.153783 secs 2023-03-09T18:38:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:37:59 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:38:00.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:37:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:38:00.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:37:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:38:00.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:37:59 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13191.153783 secs 2023-03-09T18:38:00.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:37:59 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:38:00.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:37:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:38:00.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:37:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:38:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:37:59 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13191.153783 secs 2023-03-09T18:38:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:37:59 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:38:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:37:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:38:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:37:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:38:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:38:00 smithi122 ceph-mon[93869]: pgmap v7547: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 307 KiB/s wr, 9 op/s 2023-03-09T18:38:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:38:00 smithi181 ceph-mon[94189]: pgmap v7547: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 307 KiB/s wr, 9 op/s 2023-03-09T18:38:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:38:00 smithi038 ceph-mon[92422]: pgmap v7547: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 307 KiB/s wr, 9 op/s 2023-03-09T18:38:02.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:38:01 smithi122 ceph-mon[93869]: pgmap v7548: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 330 KiB/s wr, 12 op/s 2023-03-09T18:38:02.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:38:01 smithi181 ceph-mon[94189]: pgmap v7548: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 330 KiB/s wr, 12 op/s 2023-03-09T18:38:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:38:01 smithi038 ceph-mon[92422]: pgmap v7548: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 330 KiB/s wr, 12 op/s 2023-03-09T18:38:03.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:38:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13193.424862 secs 2023-03-09T18:38:03.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:38:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13193.424862 secs 2023-03-09T18:38:03.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:38:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13193.424862 secs 2023-03-09T18:38:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13655.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:38:04.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:38:03 smithi122 ceph-mon[93869]: pgmap v7549: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 135 KiB/s wr, 20 op/s 2023-03-09T18:38:04.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:38:03 smithi181 ceph-mon[94189]: pgmap v7549: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 135 KiB/s wr, 20 op/s 2023-03-09T18:38:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:38:03 smithi038 ceph-mon[92422]: pgmap v7549: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 135 KiB/s wr, 20 op/s 2023-03-09T18:38:05.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:38:04 smithi122 ceph-mon[93869]: osdmap e685: 12 total, 12 up, 12 in 2023-03-09T18:38:05.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:38:04 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13196.153840 secs 2023-03-09T18:38:05.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:38:04 smithi181 ceph-mon[94189]: osdmap e685: 12 total, 12 up, 12 in 2023-03-09T18:38:05.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:38:04 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13196.153840 secs 2023-03-09T18:38:05.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:38:04 smithi038 ceph-mon[92422]: osdmap e685: 12 total, 12 up, 12 in 2023-03-09T18:38:05.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:38:04 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13196.153840 secs 2023-03-09T18:38:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:38:05 smithi122 ceph-mon[93869]: pgmap v7551: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 208 KiB/s wr, 26 op/s 2023-03-09T18:38:06.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:38:05 smithi181 ceph-mon[94189]: pgmap v7551: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 208 KiB/s wr, 26 op/s 2023-03-09T18:38:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:38:05 smithi038 ceph-mon[92422]: pgmap v7551: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 208 KiB/s wr, 26 op/s 2023-03-09T18:38:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:38:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13198.424974 secs 2023-03-09T18:38:08.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:38:07 smithi122 ceph-mon[93869]: pgmap v7552: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 205 KiB/s wr, 25 op/s 2023-03-09T18:38:08.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:38:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13198.424974 secs 2023-03-09T18:38:08.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:38:07 smithi181 ceph-mon[94189]: pgmap v7552: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 205 KiB/s wr, 25 op/s 2023-03-09T18:38:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:38:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13198.424974 secs 2023-03-09T18:38:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:38:07 smithi038 ceph-mon[92422]: pgmap v7552: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 205 KiB/s wr, 25 op/s 2023-03-09T18:38:08.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13660.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:38:10.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:38:09 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13201.153922 secs 2023-03-09T18:38:10.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:38:09 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13201.153922 secs 2023-03-09T18:38:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:38:09 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13201.153922 secs 2023-03-09T18:38:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:38:10 smithi122 ceph-mon[93869]: pgmap v7553: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 205 KiB/s wr, 25 op/s 2023-03-09T18:38:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:38:10 smithi181 ceph-mon[94189]: pgmap v7553: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 205 KiB/s wr, 25 op/s 2023-03-09T18:38:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:38:10 smithi038 ceph-mon[92422]: pgmap v7553: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 205 KiB/s wr, 25 op/s 2023-03-09T18:38:12.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:38:11 smithi122 ceph-mon[93869]: pgmap v7554: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 190 KiB/s wr, 26 op/s 2023-03-09T18:38:12.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:38:11 smithi181 ceph-mon[94189]: pgmap v7554: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 190 KiB/s wr, 26 op/s 2023-03-09T18:38:12.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:38:11 smithi038 ceph-mon[92422]: pgmap v7554: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 190 KiB/s wr, 26 op/s 2023-03-09T18:38:13.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:38:12 smithi122 ceph-mon[93869]: osdmap e686: 12 total, 12 up, 12 in 2023-03-09T18:38:13.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:38:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13203.425102 secs 2023-03-09T18:38:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:38:12 smithi181 ceph-mon[94189]: osdmap e686: 12 total, 12 up, 12 in 2023-03-09T18:38:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:38:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13203.425102 secs 2023-03-09T18:38:13.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:38:12 smithi038 ceph-mon[92422]: osdmap e686: 12 total, 12 up, 12 in 2023-03-09T18:38:13.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:38:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13203.425102 secs 2023-03-09T18:38:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13665.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:38:14.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:38:13 smithi122 ceph-mon[93869]: pgmap v7556: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 166 KiB/s wr, 11 op/s 2023-03-09T18:38:14.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:38:13 smithi181 ceph-mon[94189]: pgmap v7556: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 166 KiB/s wr, 11 op/s 2023-03-09T18:38:14.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:38:13 smithi038 ceph-mon[92422]: pgmap v7556: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 166 KiB/s wr, 11 op/s 2023-03-09T18:38:15.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:38:14 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13206.153986 secs 2023-03-09T18:38:15.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:38:14 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13206.153986 secs 2023-03-09T18:38:15.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:38:14 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13206.153986 secs 2023-03-09T18:38:16.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:38:15 smithi181 ceph-mon[94189]: pgmap v7557: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 7 op/s 2023-03-09T18:38:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:38:15 smithi038 ceph-mon[92422]: pgmap v7557: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 7 op/s 2023-03-09T18:38:16.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:38:15 smithi122 ceph-mon[93869]: pgmap v7557: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 7 op/s 2023-03-09T18:38:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:38:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13208.425239 secs 2023-03-09T18:38:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:38:17 smithi122 ceph-mon[93869]: pgmap v7558: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 8 op/s 2023-03-09T18:38:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:38:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13208.425239 secs 2023-03-09T18:38:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:38:17 smithi181 ceph-mon[94189]: pgmap v7558: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 8 op/s 2023-03-09T18:38:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:38:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13208.425239 secs 2023-03-09T18:38:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:38:17 smithi038 ceph-mon[92422]: pgmap v7558: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 8 op/s 2023-03-09T18:38:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13670.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:38:20.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:38:19 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13211.154046 secs 2023-03-09T18:38:20.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:38:19 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13211.154046 secs 2023-03-09T18:38:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:38:19 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13211.154046 secs 2023-03-09T18:38:21.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:38:20 smithi122 ceph-mon[93869]: pgmap v7559: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 8 op/s 2023-03-09T18:38:21.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:38:20 smithi181 ceph-mon[94189]: pgmap v7559: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 8 op/s 2023-03-09T18:38:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:38:20 smithi038 ceph-mon[92422]: pgmap v7559: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 8 op/s 2023-03-09T18:38:22.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:38:21 smithi122 ceph-mon[93869]: pgmap v7560: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T18:38:22.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:38:21 smithi181 ceph-mon[94189]: pgmap v7560: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T18:38:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:38:21 smithi038 ceph-mon[92422]: pgmap v7560: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T18:38:23.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:38:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13213.425377 secs 2023-03-09T18:38:23.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:38:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13213.425377 secs 2023-03-09T18:38:23.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:38:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13213.425377 secs 2023-03-09T18:38:23.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13675.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:38:24.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:38:23 smithi122 ceph-mon[93869]: pgmap v7561: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 6 op/s 2023-03-09T18:38:24.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:38:23 smithi181 ceph-mon[94189]: pgmap v7561: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 6 op/s 2023-03-09T18:38:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:38:23 smithi038 ceph-mon[92422]: pgmap v7561: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 6 op/s 2023-03-09T18:38:25.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:38:24 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13216.154107 secs 2023-03-09T18:38:25.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:38:24 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13216.154107 secs 2023-03-09T18:38:25.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:38:24 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13216.154107 secs 2023-03-09T18:38:27.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:38:26 smithi122 ceph-mon[93869]: pgmap v7562: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 54 KiB/s wr, 8 op/s 2023-03-09T18:38:27.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:38:26 smithi181 ceph-mon[94189]: pgmap v7562: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 54 KiB/s wr, 8 op/s 2023-03-09T18:38:27.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:38:26 smithi038 ceph-mon[92422]: pgmap v7562: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 54 KiB/s wr, 8 op/s 2023-03-09T18:38:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:38:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13218.425445 secs 2023-03-09T18:38:28.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:38:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13218.425445 secs 2023-03-09T18:38:28.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:38:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13218.425445 secs 2023-03-09T18:38:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13680.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:38:29.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:38:28 smithi038 ceph-mon[92422]: pgmap v7563: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 7 op/s 2023-03-09T18:38:29.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:38:28 smithi122 ceph-mon[93869]: pgmap v7563: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 7 op/s 2023-03-09T18:38:29.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:38:28 smithi181 ceph-mon[94189]: pgmap v7563: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 7 op/s 2023-03-09T18:38:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:38:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:38:30.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:38:29 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13221.154168 secs 2023-03-09T18:38:30.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:38:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:38:30.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:38:29 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13221.154168 secs 2023-03-09T18:38:30.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:38:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:38:30.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:38:29 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13221.154168 secs 2023-03-09T18:38:31.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:38:30 smithi038 ceph-mon[92422]: pgmap v7564: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 43 KiB/s wr, 6 op/s 2023-03-09T18:38:31.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:38:30 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:38:31.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:38:30 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:38:31.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:38:30 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:38:31.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:38:30 smithi122 ceph-mon[93869]: pgmap v7564: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 43 KiB/s wr, 6 op/s 2023-03-09T18:38:31.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:38:30 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:38:31.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:38:30 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:38:31.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:38:30 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:38:31.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:38:30 smithi181 ceph-mon[94189]: pgmap v7564: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 43 KiB/s wr, 6 op/s 2023-03-09T18:38:31.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:38:30 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:38:31.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:38:30 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:38:31.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:38:30 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:38:33.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:38:32 smithi122 ceph-mon[93869]: pgmap v7565: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 82 KiB/s wr, 9 op/s 2023-03-09T18:38:33.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:38:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13223.425539 secs 2023-03-09T18:38:33.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:38:32 smithi181 ceph-mon[94189]: pgmap v7565: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 82 KiB/s wr, 9 op/s 2023-03-09T18:38:33.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:38:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13223.425539 secs 2023-03-09T18:38:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:38:32 smithi038 ceph-mon[92422]: pgmap v7565: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 82 KiB/s wr, 9 op/s 2023-03-09T18:38:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:38:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13223.425539 secs 2023-03-09T18:38:33.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13685.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:38:35.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:38:35 smithi122 ceph-mon[93869]: pgmap v7566: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 82 KiB/s wr, 7 op/s 2023-03-09T18:38:35.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:38:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13226.154242 secs 2023-03-09T18:38:35.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:38:35 smithi181 ceph-mon[94189]: pgmap v7566: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 82 KiB/s wr, 7 op/s 2023-03-09T18:38:35.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:38:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13226.154242 secs 2023-03-09T18:38:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:38:35 smithi038 ceph-mon[92422]: pgmap v7566: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 82 KiB/s wr, 7 op/s 2023-03-09T18:38:35.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:38:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13226.154242 secs 2023-03-09T18:38:37.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:38:37 smithi122 ceph-mon[93869]: pgmap v7567: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 139 KiB/s wr, 9 op/s 2023-03-09T18:38:37.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:38:37 smithi181 ceph-mon[94189]: pgmap v7567: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 139 KiB/s wr, 9 op/s 2023-03-09T18:38:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:38:37 smithi038 ceph-mon[92422]: pgmap v7567: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 139 KiB/s wr, 9 op/s 2023-03-09T18:38:38.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13690.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:38:39.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:38:39 smithi122 ceph-mon[93869]: pgmap v7568: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 97 KiB/s wr, 7 op/s 2023-03-09T18:38:39.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:38:39 smithi181 ceph-mon[94189]: pgmap v7568: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 97 KiB/s wr, 7 op/s 2023-03-09T18:38:39.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:38:39 smithi038 ceph-mon[92422]: pgmap v7568: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 97 KiB/s wr, 7 op/s 2023-03-09T18:38:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:38:40 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:38:40.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:38:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13231.154301 secs 2023-03-09T18:38:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:38:40 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:38:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:38:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13231.154301 secs 2023-03-09T18:38:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:38:40 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:38:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:38:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13231.154301 secs 2023-03-09T18:38:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:38:41 smithi122 ceph-mon[93869]: pgmap v7569: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 96 KiB/s wr, 6 op/s 2023-03-09T18:38:41.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:38:41 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:38:41.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:38:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:38:41.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:38:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:38:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:38:41 smithi181 ceph-mon[94189]: pgmap v7569: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 96 KiB/s wr, 6 op/s 2023-03-09T18:38:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:38:41 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:38:41.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:38:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:38:41.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:38:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:38:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:38:41 smithi038 ceph-mon[92422]: pgmap v7569: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 96 KiB/s wr, 6 op/s 2023-03-09T18:38:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:38:41 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:38:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:38:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:38:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:38:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:38:43.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:38:43 smithi122 ceph-mon[93869]: pgmap v7570: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 231 KiB/s wr, 11 op/s 2023-03-09T18:38:43.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:38:43 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 13233.425797 secs 2023-03-09T18:38:43.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:38:43 smithi181 ceph-mon[94189]: pgmap v7570: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 231 KiB/s wr, 11 op/s 2023-03-09T18:38:43.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:38:43 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 13233.425797 secs 2023-03-09T18:38:43.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:38:43 smithi038 ceph-mon[92422]: pgmap v7570: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 231 KiB/s wr, 11 op/s 2023-03-09T18:38:43.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:38:43 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 13233.425797 secs 2023-03-09T18:38:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13695.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:38:45.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:38:45 smithi122 ceph-mon[93869]: pgmap v7571: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 193 KiB/s wr, 9 op/s 2023-03-09T18:38:45.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:38:45 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:38:45.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:38:45 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:38:45.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:38:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13236.154365 secs 2023-03-09T18:38:45.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:38:45 smithi181 ceph-mon[94189]: pgmap v7571: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 193 KiB/s wr, 9 op/s 2023-03-09T18:38:45.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:38:45 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:38:45.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:38:45 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:38:45.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:38:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13236.154365 secs 2023-03-09T18:38:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:38:45 smithi038 ceph-mon[92422]: pgmap v7571: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 193 KiB/s wr, 9 op/s 2023-03-09T18:38:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:38:45 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:38:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:38:45 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:38:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:38:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13236.154365 secs 2023-03-09T18:38:47.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:38:47 smithi122 ceph-mon[93869]: pgmap v7572: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 193 KiB/s wr, 11 op/s 2023-03-09T18:38:47.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:38:47 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:38:47.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:38:47 smithi181 ceph-mon[94189]: pgmap v7572: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 193 KiB/s wr, 11 op/s 2023-03-09T18:38:47.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:38:47 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:38:47.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:38:47 smithi038 ceph-mon[92422]: pgmap v7572: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 193 KiB/s wr, 11 op/s 2023-03-09T18:38:47.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:38:47 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:38:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:38:48 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 13238.425918 secs 2023-03-09T18:38:48.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:38:48 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 13238.425918 secs 2023-03-09T18:38:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:38:48 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 13238.425918 secs 2023-03-09T18:38:48.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13700.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:38:49.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:38:49 smithi122 ceph-mon[93869]: pgmap v7573: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 221 KiB/s wr, 11 op/s 2023-03-09T18:38:49.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:38:49 smithi181 ceph-mon[94189]: pgmap v7573: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 221 KiB/s wr, 11 op/s 2023-03-09T18:38:49.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:38:49 smithi038 ceph-mon[92422]: pgmap v7573: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 221 KiB/s wr, 11 op/s 2023-03-09T18:38:50.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:38:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13241.154424 secs 2023-03-09T18:38:50.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:38:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13241.154424 secs 2023-03-09T18:38:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:38:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13241.154424 secs 2023-03-09T18:38:51.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:38:51 smithi122 ceph-mon[93869]: pgmap v7574: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 220 KiB/s wr, 10 op/s 2023-03-09T18:38:51.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:38:51 smithi181 ceph-mon[94189]: pgmap v7574: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 220 KiB/s wr, 10 op/s 2023-03-09T18:38:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:38:51 smithi038 ceph-mon[92422]: pgmap v7574: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 220 KiB/s wr, 10 op/s 2023-03-09T18:38:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:38:53 smithi122 ceph-mon[93869]: pgmap v7575: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 340 KiB/s wr, 14 op/s 2023-03-09T18:38:53.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:38:53 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 13243.426015 secs 2023-03-09T18:38:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:38:53 smithi181 ceph-mon[94189]: pgmap v7575: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 340 KiB/s wr, 14 op/s 2023-03-09T18:38:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:38:53 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 13243.426015 secs 2023-03-09T18:38:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:38:53 smithi038 ceph-mon[92422]: pgmap v7575: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 340 KiB/s wr, 14 op/s 2023-03-09T18:38:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:38:53 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 13243.426015 secs 2023-03-09T18:38:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13705.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:38:55.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:38:55 smithi122 ceph-mon[93869]: pgmap v7576: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 205 KiB/s wr, 10 op/s 2023-03-09T18:38:55.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:38:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13246.154502 secs 2023-03-09T18:38:55.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:38:55 smithi181 ceph-mon[94189]: pgmap v7576: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 205 KiB/s wr, 10 op/s 2023-03-09T18:38:55.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:38:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13246.154502 secs 2023-03-09T18:38:55.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:38:55 smithi038 ceph-mon[92422]: pgmap v7576: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 205 KiB/s wr, 10 op/s 2023-03-09T18:38:55.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:38:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13246.154502 secs 2023-03-09T18:38:57.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:38:57 smithi122 ceph-mon[93869]: pgmap v7577: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 205 KiB/s wr, 11 op/s 2023-03-09T18:38:57.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:38:57 smithi181 ceph-mon[94189]: pgmap v7577: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 205 KiB/s wr, 11 op/s 2023-03-09T18:38:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:38:57 smithi038 ceph-mon[92422]: pgmap v7577: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 205 KiB/s wr, 11 op/s 2023-03-09T18:38:58.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:38:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13248.426136 secs 2023-03-09T18:38:58.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:38:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13248.426136 secs 2023-03-09T18:38:58.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:38:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13248.426136 secs 2023-03-09T18:38:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13710.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:38:59.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:38:59 smithi122 ceph-mon[93869]: pgmap v7578: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 265 KiB/s wr, 11 op/s 2023-03-09T18:38:59.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:38:59 smithi181 ceph-mon[94189]: pgmap v7578: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 265 KiB/s wr, 11 op/s 2023-03-09T18:38:59.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:38:59 smithi038 ceph-mon[92422]: pgmap v7578: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 265 KiB/s wr, 11 op/s 2023-03-09T18:39:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:39:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13251.154564 secs 2023-03-09T18:39:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:39:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13251.154564 secs 2023-03-09T18:39:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:39:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13251.154564 secs 2023-03-09T18:39:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:39:01 smithi122 ceph-mon[93869]: pgmap v7579: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 180 KiB/s wr, 8 op/s 2023-03-09T18:39:01.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:39:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:39:01.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:39:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:39:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:39:01 smithi181 ceph-mon[94189]: pgmap v7579: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 180 KiB/s wr, 8 op/s 2023-03-09T18:39:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:39:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:39:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:39:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:39:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:39:01 smithi038 ceph-mon[92422]: pgmap v7579: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 180 KiB/s wr, 8 op/s 2023-03-09T18:39:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:39:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:39:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:39:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:39:03.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:39:03 smithi122 ceph-mon[93869]: pgmap v7580: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 229 KiB/s wr, 13 op/s 2023-03-09T18:39:03.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:39:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13253.426256 secs 2023-03-09T18:39:03.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:39:03 smithi181 ceph-mon[94189]: pgmap v7580: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 229 KiB/s wr, 13 op/s 2023-03-09T18:39:03.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:39:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13253.426256 secs 2023-03-09T18:39:03.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:39:03 smithi038 ceph-mon[92422]: pgmap v7580: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 229 KiB/s wr, 13 op/s 2023-03-09T18:39:03.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:39:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13253.426256 secs 2023-03-09T18:39:03.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13715.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:39:05.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:39:05 smithi122 ceph-mon[93869]: pgmap v7581: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 180 KiB/s wr, 23 op/s 2023-03-09T18:39:05.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:39:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13256.154646 secs 2023-03-09T18:39:05.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:39:05 smithi181 ceph-mon[94189]: pgmap v7581: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 180 KiB/s wr, 23 op/s 2023-03-09T18:39:05.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:39:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13256.154646 secs 2023-03-09T18:39:05.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:39:05 smithi038 ceph-mon[92422]: pgmap v7581: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 180 KiB/s wr, 23 op/s 2023-03-09T18:39:05.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:39:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13256.154646 secs 2023-03-09T18:39:07.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:39:07 smithi122 ceph-mon[93869]: pgmap v7582: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 180 KiB/s wr, 24 op/s 2023-03-09T18:39:07.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:39:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13258.426329 secs 2023-03-09T18:39:07.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:39:07 smithi181 ceph-mon[94189]: pgmap v7582: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 180 KiB/s wr, 24 op/s 2023-03-09T18:39:07.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:39:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13258.426329 secs 2023-03-09T18:39:07.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:39:07 smithi038 ceph-mon[92422]: pgmap v7582: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 180 KiB/s wr, 24 op/s 2023-03-09T18:39:07.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:39:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13258.426329 secs 2023-03-09T18:39:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13720.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:39:09.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:39:09 smithi122 ceph-mon[93869]: pgmap v7583: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 209 KiB/s wr, 23 op/s 2023-03-09T18:39:09.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:39:09 smithi181 ceph-mon[94189]: pgmap v7583: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 209 KiB/s wr, 23 op/s 2023-03-09T18:39:09.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:39:09 smithi038 ceph-mon[92422]: pgmap v7583: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 209 KiB/s wr, 23 op/s 2023-03-09T18:39:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:39:10 smithi122 ceph-mon[93869]: osdmap e687: 12 total, 12 up, 12 in 2023-03-09T18:39:10.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:39:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13261.154707 secs 2023-03-09T18:39:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:39:10 smithi181 ceph-mon[94189]: osdmap e687: 12 total, 12 up, 12 in 2023-03-09T18:39:10.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:39:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13261.154707 secs 2023-03-09T18:39:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:39:10 smithi038 ceph-mon[92422]: osdmap e687: 12 total, 12 up, 12 in 2023-03-09T18:39:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:39:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13261.154707 secs 2023-03-09T18:39:11.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:39:11 smithi122 ceph-mon[93869]: pgmap v7585: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 179 KiB/s wr, 26 op/s 2023-03-09T18:39:11.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:39:11 smithi181 ceph-mon[94189]: pgmap v7585: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 179 KiB/s wr, 26 op/s 2023-03-09T18:39:11.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:39:11 smithi038 ceph-mon[92422]: pgmap v7585: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 179 KiB/s wr, 26 op/s 2023-03-09T18:39:13.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:39:12 smithi122 ceph-mon[93869]: pgmap v7586: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 133 KiB/s wr, 23 op/s 2023-03-09T18:39:13.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:39:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13263.426443 secs 2023-03-09T18:39:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:39:12 smithi181 ceph-mon[94189]: pgmap v7586: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 133 KiB/s wr, 23 op/s 2023-03-09T18:39:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:39:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13263.426443 secs 2023-03-09T18:39:13.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:39:12 smithi038 ceph-mon[92422]: pgmap v7586: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 133 KiB/s wr, 23 op/s 2023-03-09T18:39:13.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:39:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13263.426443 secs 2023-03-09T18:39:13.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13725.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:39:15.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:39:14 smithi122 ceph-mon[93869]: pgmap v7587: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 47 KiB/s wr, 7 op/s 2023-03-09T18:39:15.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:39:14 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13266.154766 secs 2023-03-09T18:39:15.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:39:14 smithi181 ceph-mon[94189]: pgmap v7587: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 47 KiB/s wr, 7 op/s 2023-03-09T18:39:15.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:39:14 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13266.154766 secs 2023-03-09T18:39:15.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:39:14 smithi038 ceph-mon[92422]: pgmap v7587: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 47 KiB/s wr, 7 op/s 2023-03-09T18:39:15.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:39:14 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13266.154766 secs 2023-03-09T18:39:17.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:39:16 smithi038 ceph-mon[92422]: pgmap v7588: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 48 KiB/s wr, 8 op/s 2023-03-09T18:39:17.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:39:16 smithi038 ceph-mon[92422]: osdmap e688: 12 total, 12 up, 12 in 2023-03-09T18:39:17.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:39:16 smithi122 ceph-mon[93869]: pgmap v7588: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 48 KiB/s wr, 8 op/s 2023-03-09T18:39:17.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:39:16 smithi122 ceph-mon[93869]: osdmap e688: 12 total, 12 up, 12 in 2023-03-09T18:39:17.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:39:16 smithi181 ceph-mon[94189]: pgmap v7588: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 48 KiB/s wr, 8 op/s 2023-03-09T18:39:17.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:39:16 smithi181 ceph-mon[94189]: osdmap e688: 12 total, 12 up, 12 in 2023-03-09T18:39:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:39:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13268.426575 secs 2023-03-09T18:39:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:39:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13268.426575 secs 2023-03-09T18:39:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:39:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13268.426575 secs 2023-03-09T18:39:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13730.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:39:19.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:39:18 smithi038 ceph-mon[92422]: pgmap v7590: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 9 op/s 2023-03-09T18:39:19.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:39:18 smithi122 ceph-mon[93869]: pgmap v7590: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 9 op/s 2023-03-09T18:39:19.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:39:18 smithi181 ceph-mon[94189]: pgmap v7590: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 9 op/s 2023-03-09T18:39:20.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:39:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13271.154828 secs 2023-03-09T18:39:20.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:39:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13271.154828 secs 2023-03-09T18:39:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:39:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13271.154828 secs 2023-03-09T18:39:21.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:39:21 smithi122 ceph-mon[93869]: pgmap v7591: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 8 op/s 2023-03-09T18:39:21.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:39:21 smithi181 ceph-mon[94189]: pgmap v7591: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 8 op/s 2023-03-09T18:39:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:39:21 smithi038 ceph-mon[92422]: pgmap v7591: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 27 KiB/s wr, 8 op/s 2023-03-09T18:39:23.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:39:23 smithi122 ceph-mon[93869]: pgmap v7592: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T18:39:23.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:39:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13273.426694 secs 2023-03-09T18:39:23.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:39:23 smithi181 ceph-mon[94189]: pgmap v7592: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T18:39:23.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:39:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13273.426694 secs 2023-03-09T18:39:23.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:39:23 smithi038 ceph-mon[92422]: pgmap v7592: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T18:39:23.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:39:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13273.426694 secs 2023-03-09T18:39:23.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13735.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:39:25.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:39:25 smithi122 ceph-mon[93869]: pgmap v7593: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 65 KiB/s wr, 7 op/s 2023-03-09T18:39:25.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:39:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13276.154890 secs 2023-03-09T18:39:25.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:39:25 smithi181 ceph-mon[94189]: pgmap v7593: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 65 KiB/s wr, 7 op/s 2023-03-09T18:39:25.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:39:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13276.154890 secs 2023-03-09T18:39:25.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:39:25 smithi038 ceph-mon[92422]: pgmap v7593: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 65 KiB/s wr, 7 op/s 2023-03-09T18:39:25.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:39:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13276.154890 secs 2023-03-09T18:39:27.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:39:27 smithi122 ceph-mon[93869]: pgmap v7594: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 64 KiB/s wr, 7 op/s 2023-03-09T18:39:27.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:39:27 smithi181 ceph-mon[94189]: pgmap v7594: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 64 KiB/s wr, 7 op/s 2023-03-09T18:39:27.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:39:27 smithi038 ceph-mon[92422]: pgmap v7594: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 64 KiB/s wr, 7 op/s 2023-03-09T18:39:28.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:39:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13278.426862 secs 2023-03-09T18:39:28.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:39:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13278.426862 secs 2023-03-09T18:39:28.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:39:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13278.426862 secs 2023-03-09T18:39:28.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13740.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:39:29.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:39:29 smithi122 ceph-mon[93869]: pgmap v7595: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 47 KiB/s wr, 7 op/s 2023-03-09T18:39:29.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:39:29 smithi181 ceph-mon[94189]: pgmap v7595: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 47 KiB/s wr, 7 op/s 2023-03-09T18:39:29.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:39:29 smithi038 ceph-mon[92422]: pgmap v7595: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 47 KiB/s wr, 7 op/s 2023-03-09T18:39:30.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:39:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13281.154949 secs 2023-03-09T18:39:30.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:39:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13281.154949 secs 2023-03-09T18:39:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:39:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13281.154949 secs 2023-03-09T18:39:31.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:39:31 smithi122 ceph-mon[93869]: pgmap v7596: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 43 KiB/s wr, 6 op/s 2023-03-09T18:39:31.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:39:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:39:31.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:39:31 smithi181 ceph-mon[94189]: pgmap v7596: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 43 KiB/s wr, 6 op/s 2023-03-09T18:39:31.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:39:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:39:31.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:39:31 smithi038 ceph-mon[92422]: pgmap v7596: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 43 KiB/s wr, 6 op/s 2023-03-09T18:39:31.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:39:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:39:32.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:39:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:39:32.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:39:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:39:32.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:39:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:39:32.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:39:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:39:32.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:39:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:39:32.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:39:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:39:32.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:39:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:39:32.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:39:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:39:32.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:39:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:39:33.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:39:33 smithi122 ceph-mon[93869]: pgmap v7597: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 43 KiB/s wr, 8 op/s 2023-03-09T18:39:33.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:39:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13283.426979 secs 2023-03-09T18:39:33.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:39:33 smithi181 ceph-mon[94189]: pgmap v7597: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 43 KiB/s wr, 8 op/s 2023-03-09T18:39:33.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:39:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13283.426979 secs 2023-03-09T18:39:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:39:33 smithi038 ceph-mon[92422]: pgmap v7597: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 43 KiB/s wr, 8 op/s 2023-03-09T18:39:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:39:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13283.426979 secs 2023-03-09T18:39:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13745.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:39:35.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:39:35 smithi122 ceph-mon[93869]: pgmap v7598: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 84 KiB/s wr, 7 op/s 2023-03-09T18:39:35.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:39:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13286.154984 secs 2023-03-09T18:39:35.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:39:35 smithi181 ceph-mon[94189]: pgmap v7598: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 84 KiB/s wr, 7 op/s 2023-03-09T18:39:35.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:39:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13286.154984 secs 2023-03-09T18:39:35.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:39:35 smithi038 ceph-mon[92422]: pgmap v7598: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 84 KiB/s wr, 7 op/s 2023-03-09T18:39:35.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:39:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13286.154984 secs 2023-03-09T18:39:37.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:39:37 smithi122 ceph-mon[93869]: pgmap v7599: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 43 KiB/s wr, 8 op/s 2023-03-09T18:39:37.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:39:37 smithi181 ceph-mon[94189]: pgmap v7599: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 43 KiB/s wr, 8 op/s 2023-03-09T18:39:37.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:39:37 smithi038 ceph-mon[92422]: pgmap v7599: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 43 KiB/s wr, 8 op/s 2023-03-09T18:39:38.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:39:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13288.427077 secs 2023-03-09T18:39:38.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:39:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13288.427077 secs 2023-03-09T18:39:38.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:39:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13288.427077 secs 2023-03-09T18:39:38.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13750.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:39:39.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:39:39 smithi122 ceph-mon[93869]: pgmap v7600: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 7 op/s 2023-03-09T18:39:39.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:39:39 smithi181 ceph-mon[94189]: pgmap v7600: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 7 op/s 2023-03-09T18:39:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:39:39 smithi038 ceph-mon[92422]: pgmap v7600: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 7 op/s 2023-03-09T18:39:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:39:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13291.155072 secs 2023-03-09T18:39:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:39:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13291.155072 secs 2023-03-09T18:39:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:39:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13291.155072 secs 2023-03-09T18:39:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:39:41 smithi122 ceph-mon[93869]: pgmap v7601: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 6 op/s 2023-03-09T18:39:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:39:41 smithi181 ceph-mon[94189]: pgmap v7601: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 6 op/s 2023-03-09T18:39:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:39:41 smithi038 ceph-mon[92422]: pgmap v7601: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 6 op/s 2023-03-09T18:39:43.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:39:43 smithi122 ceph-mon[93869]: pgmap v7602: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 63 KiB/s wr, 8 op/s 2023-03-09T18:39:43.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:39:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13293.427192 secs 2023-03-09T18:39:43.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:39:43 smithi181 ceph-mon[94189]: pgmap v7602: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 63 KiB/s wr, 8 op/s 2023-03-09T18:39:43.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:39:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13293.427192 secs 2023-03-09T18:39:43.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:39:43 smithi038 ceph-mon[92422]: pgmap v7602: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 63 KiB/s wr, 8 op/s 2023-03-09T18:39:43.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:39:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13293.427192 secs 2023-03-09T18:39:43.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13755.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:39:45.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:39:45 smithi122 ceph-mon[93869]: pgmap v7603: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 140 KiB/s wr, 7 op/s 2023-03-09T18:39:45.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:39:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13296.155130 secs 2023-03-09T18:39:45.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:39:45 smithi181 ceph-mon[94189]: pgmap v7603: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 140 KiB/s wr, 7 op/s 2023-03-09T18:39:45.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:39:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13296.155130 secs 2023-03-09T18:39:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:39:45 smithi038 ceph-mon[92422]: pgmap v7603: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 140 KiB/s wr, 7 op/s 2023-03-09T18:39:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:39:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13296.155130 secs 2023-03-09T18:39:47.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:39:47 smithi122 ceph-mon[93869]: pgmap v7604: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 98 KiB/s wr, 9 op/s 2023-03-09T18:39:47.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:39:47 smithi181 ceph-mon[94189]: pgmap v7604: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 98 KiB/s wr, 9 op/s 2023-03-09T18:39:47.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:39:47 smithi038 ceph-mon[92422]: pgmap v7604: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 98 KiB/s wr, 9 op/s 2023-03-09T18:39:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:39:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13298.427288 secs 2023-03-09T18:39:48.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:39:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13298.427288 secs 2023-03-09T18:39:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:39:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13298.427288 secs 2023-03-09T18:39:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13760.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:39:49.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:39:49 smithi122 ceph-mon[93869]: pgmap v7605: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 174 KiB/s wr, 10 op/s 2023-03-09T18:39:49.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:39:49 smithi181 ceph-mon[94189]: pgmap v7605: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 174 KiB/s wr, 10 op/s 2023-03-09T18:39:49.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:39:49 smithi038 ceph-mon[92422]: pgmap v7605: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 174 KiB/s wr, 10 op/s 2023-03-09T18:39:50.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:39:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13301.155190 secs 2023-03-09T18:39:50.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:39:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13301.155190 secs 2023-03-09T18:39:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:39:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13301.155190 secs 2023-03-09T18:39:51.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:39:51 smithi122 ceph-mon[93869]: pgmap v7606: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 164 KiB/s wr, 9 op/s 2023-03-09T18:39:51.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:39:51 smithi181 ceph-mon[94189]: pgmap v7606: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 164 KiB/s wr, 9 op/s 2023-03-09T18:39:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:39:51 smithi038 ceph-mon[92422]: pgmap v7606: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 164 KiB/s wr, 9 op/s 2023-03-09T18:39:53.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:39:52 smithi038 ceph-mon[92422]: pgmap v7607: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 290 KiB/s wr, 13 op/s 2023-03-09T18:39:53.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:39:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13303.427407 secs 2023-03-09T18:39:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:39:52 smithi122 ceph-mon[93869]: pgmap v7607: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 290 KiB/s wr, 13 op/s 2023-03-09T18:39:53.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:39:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13303.427407 secs 2023-03-09T18:39:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:39:52 smithi181 ceph-mon[94189]: pgmap v7607: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 290 KiB/s wr, 13 op/s 2023-03-09T18:39:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:39:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13303.427407 secs 2023-03-09T18:39:53.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13765.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:39:55.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:39:54 smithi038 ceph-mon[92422]: pgmap v7608: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 279 KiB/s wr, 11 op/s 2023-03-09T18:39:55.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:39:54 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13306.155252 secs 2023-03-09T18:39:55.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:39:54 smithi122 ceph-mon[93869]: pgmap v7608: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 279 KiB/s wr, 11 op/s 2023-03-09T18:39:55.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:39:54 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13306.155252 secs 2023-03-09T18:39:55.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:39:54 smithi181 ceph-mon[94189]: pgmap v7608: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 279 KiB/s wr, 11 op/s 2023-03-09T18:39:55.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:39:54 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13306.155252 secs 2023-03-09T18:39:57.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:39:56 smithi038 ceph-mon[92422]: pgmap v7609: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 205 KiB/s wr, 12 op/s 2023-03-09T18:39:57.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:39:56 smithi122 ceph-mon[93869]: pgmap v7609: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 205 KiB/s wr, 12 op/s 2023-03-09T18:39:57.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:39:56 smithi181 ceph-mon[94189]: pgmap v7609: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 205 KiB/s wr, 12 op/s 2023-03-09T18:39:58.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:39:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13308.427470 secs 2023-03-09T18:39:58.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:39:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13308.427470 secs 2023-03-09T18:39:58.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:39:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13308.427470 secs 2023-03-09T18:39:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13770.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:39:59.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:39:58 smithi038 ceph-mon[92422]: pgmap v7610: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 225 KiB/s wr, 11 op/s 2023-03-09T18:39:59.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:39:58 smithi122 ceph-mon[93869]: pgmap v7610: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 225 KiB/s wr, 11 op/s 2023-03-09T18:39:59.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:39:58 smithi181 ceph-mon[94189]: pgmap v7610: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 225 KiB/s wr, 11 op/s 2023-03-09T18:40:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:40:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13311.155313 secs 2023-03-09T18:40:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:40:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13311.155313 secs 2023-03-09T18:40:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:40:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13311.155313 secs 2023-03-09T18:40:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:40:01 smithi122 ceph-mon[93869]: pgmap v7611: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 149 KiB/s wr, 9 op/s 2023-03-09T18:40:01.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:40:01 smithi122 ceph-mon[93869]: overall HEALTH_WARN 2 MDSs report slow requests 2023-03-09T18:40:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:40:01 smithi181 ceph-mon[94189]: pgmap v7611: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 149 KiB/s wr, 9 op/s 2023-03-09T18:40:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:40:01 smithi181 ceph-mon[94189]: overall HEALTH_WARN 2 MDSs report slow requests 2023-03-09T18:40:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:40:01 smithi038 ceph-mon[92422]: pgmap v7611: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 149 KiB/s wr, 9 op/s 2023-03-09T18:40:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:40:01 smithi038 ceph-mon[92422]: overall HEALTH_WARN 2 MDSs report slow requests 2023-03-09T18:40:03.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:40:03 smithi122 ceph-mon[93869]: pgmap v7612: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 321 KiB/s wr, 15 op/s 2023-03-09T18:40:03.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:40:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13313.427534 secs 2023-03-09T18:40:03.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:40:03 smithi181 ceph-mon[94189]: pgmap v7612: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 321 KiB/s wr, 15 op/s 2023-03-09T18:40:03.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:40:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13313.427534 secs 2023-03-09T18:40:03.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:40:03 smithi038 ceph-mon[92422]: pgmap v7612: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 321 KiB/s wr, 15 op/s 2023-03-09T18:40:03.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:40:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13313.427534 secs 2023-03-09T18:40:03.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13775.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:40:05.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:40:05 smithi122 ceph-mon[93869]: pgmap v7613: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 216 KiB/s wr, 23 op/s 2023-03-09T18:40:05.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:40:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13316.155374 secs 2023-03-09T18:40:05.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:40:05 smithi181 ceph-mon[94189]: pgmap v7613: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 216 KiB/s wr, 23 op/s 2023-03-09T18:40:05.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:40:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13316.155374 secs 2023-03-09T18:40:05.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:40:05 smithi038 ceph-mon[92422]: pgmap v7613: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 216 KiB/s wr, 23 op/s 2023-03-09T18:40:05.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:40:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13316.155374 secs 2023-03-09T18:40:07.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:40:07 smithi122 ceph-mon[93869]: pgmap v7614: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 216 KiB/s wr, 25 op/s 2023-03-09T18:40:07.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:40:07 smithi181 ceph-mon[94189]: pgmap v7614: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 216 KiB/s wr, 25 op/s 2023-03-09T18:40:07.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:40:07 smithi038 ceph-mon[92422]: pgmap v7614: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 216 KiB/s wr, 25 op/s 2023-03-09T18:40:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:40:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13318.427601 secs 2023-03-09T18:40:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:40:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13318.427601 secs 2023-03-09T18:40:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:40:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13318.427601 secs 2023-03-09T18:40:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13780.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:40:09.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:40:09 smithi122 ceph-mon[93869]: pgmap v7615: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 243 KiB/s wr, 24 op/s 2023-03-09T18:40:09.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:40:09 smithi181 ceph-mon[94189]: pgmap v7615: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 243 KiB/s wr, 24 op/s 2023-03-09T18:40:09.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:40:09 smithi038 ceph-mon[92422]: pgmap v7615: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 243 KiB/s wr, 24 op/s 2023-03-09T18:40:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:40:10 smithi122 ceph-mon[93869]: osdmap e689: 12 total, 12 up, 12 in 2023-03-09T18:40:10.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:40:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13321.155468 secs 2023-03-09T18:40:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:40:10 smithi181 ceph-mon[94189]: osdmap e689: 12 total, 12 up, 12 in 2023-03-09T18:40:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:40:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13321.155468 secs 2023-03-09T18:40:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:40:10 smithi038 ceph-mon[92422]: osdmap e689: 12 total, 12 up, 12 in 2023-03-09T18:40:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:40:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13321.155468 secs 2023-03-09T18:40:11.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:40:11 smithi122 ceph-mon[93869]: pgmap v7617: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 266 KiB/s wr, 26 op/s 2023-03-09T18:40:11.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:40:11 smithi181 ceph-mon[94189]: pgmap v7617: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 266 KiB/s wr, 26 op/s 2023-03-09T18:40:11.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:40:11 smithi038 ceph-mon[92422]: pgmap v7617: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 266 KiB/s wr, 26 op/s 2023-03-09T18:40:13.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:40:13 smithi122 ceph-mon[93869]: pgmap v7618: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 73 KiB/s wr, 22 op/s 2023-03-09T18:40:13.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:40:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13323.427679 secs 2023-03-09T18:40:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:40:13 smithi181 ceph-mon[94189]: pgmap v7618: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 73 KiB/s wr, 22 op/s 2023-03-09T18:40:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:40:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13323.427679 secs 2023-03-09T18:40:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:40:13 smithi038 ceph-mon[92422]: pgmap v7618: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 73 KiB/s wr, 22 op/s 2023-03-09T18:40:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:40:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13323.427679 secs 2023-03-09T18:40:13.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13785.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:40:15.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:40:15 smithi122 ceph-mon[93869]: pgmap v7619: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 48 KiB/s wr, 8 op/s 2023-03-09T18:40:15.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:40:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13326.155537 secs 2023-03-09T18:40:15.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:40:15 smithi181 ceph-mon[94189]: pgmap v7619: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 48 KiB/s wr, 8 op/s 2023-03-09T18:40:15.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:40:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13326.155537 secs 2023-03-09T18:40:15.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:40:15 smithi038 ceph-mon[92422]: pgmap v7619: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 48 KiB/s wr, 8 op/s 2023-03-09T18:40:15.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:40:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13326.155537 secs 2023-03-09T18:40:17.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:40:17 smithi122 ceph-mon[93869]: pgmap v7620: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 48 KiB/s wr, 8 op/s 2023-03-09T18:40:17.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:40:17 smithi122 ceph-mon[93869]: osdmap e690: 12 total, 12 up, 12 in 2023-03-09T18:40:17.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:40:17 smithi181 ceph-mon[94189]: pgmap v7620: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 48 KiB/s wr, 8 op/s 2023-03-09T18:40:17.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:40:17 smithi181 ceph-mon[94189]: osdmap e690: 12 total, 12 up, 12 in 2023-03-09T18:40:17.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:40:17 smithi038 ceph-mon[92422]: pgmap v7620: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 48 KiB/s wr, 8 op/s 2023-03-09T18:40:17.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:40:17 smithi038 ceph-mon[92422]: osdmap e690: 12 total, 12 up, 12 in 2023-03-09T18:40:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:40:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13328.427744 secs 2023-03-09T18:40:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:40:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13328.427744 secs 2023-03-09T18:40:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:40:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13328.427744 secs 2023-03-09T18:40:18.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13790.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:40:19.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:40:19 smithi122 ceph-mon[93869]: pgmap v7622: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 15 KiB/s wr, 8 op/s 2023-03-09T18:40:19.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:40:19 smithi181 ceph-mon[94189]: pgmap v7622: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 15 KiB/s wr, 8 op/s 2023-03-09T18:40:19.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:40:19 smithi038 ceph-mon[92422]: pgmap v7622: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 15 KiB/s wr, 8 op/s 2023-03-09T18:40:20.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:40:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13331.155598 secs 2023-03-09T18:40:20.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:40:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13331.155598 secs 2023-03-09T18:40:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:40:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13331.155598 secs 2023-03-09T18:40:21.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:40:21 smithi122 ceph-mon[93869]: pgmap v7623: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 7 op/s 2023-03-09T18:40:21.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:40:21 smithi181 ceph-mon[94189]: pgmap v7623: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 7 op/s 2023-03-09T18:40:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:40:21 smithi038 ceph-mon[92422]: pgmap v7623: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 13 KiB/s wr, 7 op/s 2023-03-09T18:40:23.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:40:23 smithi122 ceph-mon[93869]: pgmap v7624: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2023-03-09T18:40:23.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:40:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13333.427866 secs 2023-03-09T18:40:23.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:40:23 smithi181 ceph-mon[94189]: pgmap v7624: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2023-03-09T18:40:23.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:40:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13333.427866 secs 2023-03-09T18:40:23.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:40:23 smithi038 ceph-mon[92422]: pgmap v7624: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2023-03-09T18:40:23.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:40:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13333.427866 secs 2023-03-09T18:40:23.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13795.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:40:25.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:40:25 smithi122 ceph-mon[93869]: pgmap v7625: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2023-03-09T18:40:25.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:40:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13336.155657 secs 2023-03-09T18:40:25.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:40:25 smithi181 ceph-mon[94189]: pgmap v7625: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2023-03-09T18:40:25.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:40:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13336.155657 secs 2023-03-09T18:40:25.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:40:25 smithi038 ceph-mon[92422]: pgmap v7625: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2023-03-09T18:40:25.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:40:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13336.155657 secs 2023-03-09T18:40:27.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:40:27 smithi122 ceph-mon[93869]: pgmap v7626: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2023-03-09T18:40:27.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:40:27 smithi181 ceph-mon[94189]: pgmap v7626: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2023-03-09T18:40:27.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:40:27 smithi038 ceph-mon[92422]: pgmap v7626: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2023-03-09T18:40:28.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:40:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13338.427998 secs 2023-03-09T18:40:28.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:40:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13338.427998 secs 2023-03-09T18:40:28.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:40:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13338.427998 secs 2023-03-09T18:40:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13800.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:40:29.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:40:29 smithi122 ceph-mon[93869]: pgmap v7627: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2023-03-09T18:40:29.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:40:29 smithi181 ceph-mon[94189]: pgmap v7627: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2023-03-09T18:40:29.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:40:29 smithi038 ceph-mon[92422]: pgmap v7627: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2023-03-09T18:40:30.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:40:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13341.155716 secs 2023-03-09T18:40:30.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:40:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13341.155716 secs 2023-03-09T18:40:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:40:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13341.155716 secs 2023-03-09T18:40:31.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:40:31 smithi122 ceph-mon[93869]: pgmap v7628: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2023-03-09T18:40:31.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:40:31 smithi181 ceph-mon[94189]: pgmap v7628: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2023-03-09T18:40:31.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:40:31 smithi038 ceph-mon[92422]: pgmap v7628: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2023-03-09T18:40:32.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:40:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:40:32.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:40:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13343.428154 secs 2023-03-09T18:40:32.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:40:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:40:32.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:40:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:40:32.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:40:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:40:32.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:40:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:40:32.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:40:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13343.428154 secs 2023-03-09T18:40:32.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:40:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:40:32.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:40:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:40:32.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:40:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:40:32.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:40:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:40:32.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:40:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13343.428154 secs 2023-03-09T18:40:32.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:40:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:40:32.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:40:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:40:32.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:40:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:40:33.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:40:33 smithi122 ceph-mon[93869]: pgmap v7629: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 9 op/s 2023-03-09T18:40:33.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:40:33 smithi181 ceph-mon[94189]: pgmap v7629: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 9 op/s 2023-03-09T18:40:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:40:33 smithi038 ceph-mon[92422]: pgmap v7629: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 9 op/s 2023-03-09T18:40:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13805.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:40:35.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:40:35 smithi122 ceph-mon[93869]: pgmap v7630: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 43 KiB/s wr, 6 op/s 2023-03-09T18:40:35.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:40:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13346.155816 secs 2023-03-09T18:40:35.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:40:35 smithi181 ceph-mon[94189]: pgmap v7630: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 43 KiB/s wr, 6 op/s 2023-03-09T18:40:35.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:40:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13346.155816 secs 2023-03-09T18:40:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:40:35 smithi038 ceph-mon[92422]: pgmap v7630: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 43 KiB/s wr, 6 op/s 2023-03-09T18:40:35.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:40:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13346.155816 secs 2023-03-09T18:40:37.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:40:37 smithi122 ceph-mon[93869]: pgmap v7631: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 43 KiB/s wr, 8 op/s 2023-03-09T18:40:37.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:40:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13348.428238 secs 2023-03-09T18:40:37.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:40:37 smithi181 ceph-mon[94189]: pgmap v7631: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 43 KiB/s wr, 8 op/s 2023-03-09T18:40:37.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:40:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13348.428238 secs 2023-03-09T18:40:37.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:40:37 smithi038 ceph-mon[92422]: pgmap v7631: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 43 KiB/s wr, 8 op/s 2023-03-09T18:40:37.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:40:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13348.428238 secs 2023-03-09T18:40:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13810.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:40:39.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:40:39 smithi122 ceph-mon[93869]: pgmap v7632: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 173 KiB/s wr, 9 op/s 2023-03-09T18:40:39.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:40:39 smithi181 ceph-mon[94189]: pgmap v7632: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 173 KiB/s wr, 9 op/s 2023-03-09T18:40:39.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:40:39 smithi038 ceph-mon[92422]: pgmap v7632: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 173 KiB/s wr, 9 op/s 2023-03-09T18:40:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:40:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13351.155899 secs 2023-03-09T18:40:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:40:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13351.155899 secs 2023-03-09T18:40:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:40:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13351.155899 secs 2023-03-09T18:40:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:40:41 smithi122 ceph-mon[93869]: pgmap v7633: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 172 KiB/s wr, 7 op/s 2023-03-09T18:40:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:40:41 smithi181 ceph-mon[94189]: pgmap v7633: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 172 KiB/s wr, 7 op/s 2023-03-09T18:40:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:40:41 smithi038 ceph-mon[92422]: pgmap v7633: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 172 KiB/s wr, 7 op/s 2023-03-09T18:40:43.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:40:43 smithi122 ceph-mon[93869]: pgmap v7634: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 194 KiB/s wr, 10 op/s 2023-03-09T18:40:43.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:40:43 smithi181 ceph-mon[94189]: pgmap v7634: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 194 KiB/s wr, 10 op/s 2023-03-09T18:40:43.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:40:43 smithi038 ceph-mon[92422]: pgmap v7634: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 194 KiB/s wr, 10 op/s 2023-03-09T18:40:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13815.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:40:44.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:40:44 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:40:44.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:40:44 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:40:44.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:40:44 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:40:45.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:40:45 smithi122 ceph-mon[93869]: pgmap v7635: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 195 KiB/s wr, 9 op/s 2023-03-09T18:40:45.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:40:45 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:40:45.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:40:45 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:40:45.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:40:45 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:40:45.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:40:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13356.155961 secs 2023-03-09T18:40:45.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:40:45 smithi181 ceph-mon[94189]: pgmap v7635: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 195 KiB/s wr, 9 op/s 2023-03-09T18:40:45.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:40:45 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:40:45.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:40:45 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:40:45.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:40:45 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:40:45.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:40:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13356.155961 secs 2023-03-09T18:40:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:40:45 smithi038 ceph-mon[92422]: pgmap v7635: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 195 KiB/s wr, 9 op/s 2023-03-09T18:40:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:40:45 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:40:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:40:45 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:40:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:40:45 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:40:45.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:40:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13356.155961 secs 2023-03-09T18:40:47.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:40:47 smithi122 ceph-mon[93869]: pgmap v7636: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 196 KiB/s wr, 11 op/s 2023-03-09T18:40:47.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:40:47 smithi181 ceph-mon[94189]: pgmap v7636: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 196 KiB/s wr, 11 op/s 2023-03-09T18:40:47.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:40:47 smithi038 ceph-mon[92422]: pgmap v7636: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 196 KiB/s wr, 11 op/s 2023-03-09T18:40:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13820.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:40:49.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:40:49 smithi122 ceph-mon[93869]: pgmap v7637: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 311 KiB/s wr, 12 op/s 2023-03-09T18:40:49.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:40:49 smithi181 ceph-mon[94189]: pgmap v7637: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 311 KiB/s wr, 12 op/s 2023-03-09T18:40:49.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:40:49 smithi038 ceph-mon[92422]: pgmap v7637: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 311 KiB/s wr, 12 op/s 2023-03-09T18:40:50.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:40:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13361.156017 secs 2023-03-09T18:40:50.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:40:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13361.156017 secs 2023-03-09T18:40:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:40:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13361.156017 secs 2023-03-09T18:40:51.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:40:51 smithi122 ceph-mon[93869]: pgmap v7638: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 181 KiB/s wr, 9 op/s 2023-03-09T18:40:51.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:40:51 smithi181 ceph-mon[94189]: pgmap v7638: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 181 KiB/s wr, 9 op/s 2023-03-09T18:40:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:40:51 smithi038 ceph-mon[92422]: pgmap v7638: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 181 KiB/s wr, 9 op/s 2023-03-09T18:40:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:40:53 smithi122 ceph-mon[93869]: pgmap v7639: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 193 KiB/s wr, 12 op/s 2023-03-09T18:40:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:40:53 smithi181 ceph-mon[94189]: pgmap v7639: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 193 KiB/s wr, 12 op/s 2023-03-09T18:40:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:40:53 smithi038 ceph-mon[92422]: pgmap v7639: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 193 KiB/s wr, 12 op/s 2023-03-09T18:40:53.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13825.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:40:55.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:40:55 smithi122 ceph-mon[93869]: pgmap v7640: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 215 KiB/s wr, 10 op/s 2023-03-09T18:40:55.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:40:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13366.156079 secs 2023-03-09T18:40:55.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:40:55 smithi181 ceph-mon[94189]: pgmap v7640: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 215 KiB/s wr, 10 op/s 2023-03-09T18:40:55.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:40:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13366.156079 secs 2023-03-09T18:40:55.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:40:55 smithi038 ceph-mon[92422]: pgmap v7640: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 215 KiB/s wr, 10 op/s 2023-03-09T18:40:55.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:40:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13366.156079 secs 2023-03-09T18:40:57.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:40:57 smithi122 ceph-mon[93869]: pgmap v7641: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 233 KiB/s wr, 11 op/s 2023-03-09T18:40:57.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:40:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13368.428646 secs 2023-03-09T18:40:57.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:40:57 smithi181 ceph-mon[94189]: pgmap v7641: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 233 KiB/s wr, 11 op/s 2023-03-09T18:40:57.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:40:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13368.428646 secs 2023-03-09T18:40:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:40:57 smithi038 ceph-mon[92422]: pgmap v7641: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 233 KiB/s wr, 11 op/s 2023-03-09T18:40:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:40:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13368.428646 secs 2023-03-09T18:40:58.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13830.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:40:59.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:40:59 smithi122 ceph-mon[93869]: pgmap v7642: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 263 KiB/s wr, 11 op/s 2023-03-09T18:40:59.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:40:59 smithi181 ceph-mon[94189]: pgmap v7642: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 263 KiB/s wr, 11 op/s 2023-03-09T18:40:59.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:40:59 smithi038 ceph-mon[92422]: pgmap v7642: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 263 KiB/s wr, 11 op/s 2023-03-09T18:41:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:41:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13371.156141 secs 2023-03-09T18:41:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:41:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13371.156141 secs 2023-03-09T18:41:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:41:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13371.156141 secs 2023-03-09T18:41:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:41:01 smithi122 ceph-mon[93869]: pgmap v7643: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 148 KiB/s wr, 8 op/s 2023-03-09T18:41:01.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:41:01 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:41:01.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:41:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:41:01.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:41:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:41:01.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:41:01 smithi181 ceph-mon[94189]: pgmap v7643: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 148 KiB/s wr, 8 op/s 2023-03-09T18:41:01.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:41:01 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:41:01.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:41:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:41:01.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:41:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:41:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:41:01 smithi038 ceph-mon[92422]: pgmap v7643: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 148 KiB/s wr, 8 op/s 2023-03-09T18:41:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:41:01 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:41:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:41:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:41:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:41:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:41:03.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:41:03 smithi122 ceph-mon[93869]: pgmap v7644: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 194 KiB/s wr, 14 op/s 2023-03-09T18:41:03.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:41:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13373.428769 secs 2023-03-09T18:41:03.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:41:03 smithi181 ceph-mon[94189]: pgmap v7644: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 194 KiB/s wr, 14 op/s 2023-03-09T18:41:03.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:41:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13373.428769 secs 2023-03-09T18:41:03.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:41:03 smithi038 ceph-mon[92422]: pgmap v7644: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 194 KiB/s wr, 14 op/s 2023-03-09T18:41:03.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:41:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13373.428769 secs 2023-03-09T18:41:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13835.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:41:05.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:41:05 smithi122 ceph-mon[93869]: pgmap v7645: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 203 KiB/s wr, 23 op/s 2023-03-09T18:41:05.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:41:05 smithi122 ceph-mon[93869]: osdmap e691: 12 total, 12 up, 12 in 2023-03-09T18:41:05.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:41:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13376.156203 secs 2023-03-09T18:41:05.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:41:05 smithi181 ceph-mon[94189]: pgmap v7645: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 203 KiB/s wr, 23 op/s 2023-03-09T18:41:05.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:41:05 smithi181 ceph-mon[94189]: osdmap e691: 12 total, 12 up, 12 in 2023-03-09T18:41:05.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:41:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13376.156203 secs 2023-03-09T18:41:05.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:41:05 smithi038 ceph-mon[92422]: pgmap v7645: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 203 KiB/s wr, 23 op/s 2023-03-09T18:41:05.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:41:05 smithi038 ceph-mon[92422]: osdmap e691: 12 total, 12 up, 12 in 2023-03-09T18:41:05.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:41:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13376.156203 secs 2023-03-09T18:41:07.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:41:07 smithi122 ceph-mon[93869]: pgmap v7647: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 234 KiB/s wr, 27 op/s 2023-03-09T18:41:07.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:41:07 smithi181 ceph-mon[94189]: pgmap v7647: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 234 KiB/s wr, 27 op/s 2023-03-09T18:41:07.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:41:07 smithi038 ceph-mon[92422]: pgmap v7647: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 234 KiB/s wr, 27 op/s 2023-03-09T18:41:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:41:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13378.428858 secs 2023-03-09T18:41:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:41:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13378.428858 secs 2023-03-09T18:41:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:41:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13378.428858 secs 2023-03-09T18:41:08.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13840.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:41:09.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:41:09 smithi122 ceph-mon[93869]: pgmap v7648: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 216 KiB/s wr, 26 op/s 2023-03-09T18:41:09.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:41:09 smithi181 ceph-mon[94189]: pgmap v7648: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 216 KiB/s wr, 26 op/s 2023-03-09T18:41:09.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:41:09 smithi038 ceph-mon[92422]: pgmap v7648: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 216 KiB/s wr, 26 op/s 2023-03-09T18:41:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:41:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13381.156262 secs 2023-03-09T18:41:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:41:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13381.156262 secs 2023-03-09T18:41:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:41:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13381.156262 secs 2023-03-09T18:41:11.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:41:11 smithi122 ceph-mon[93869]: pgmap v7649: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 216 KiB/s wr, 26 op/s 2023-03-09T18:41:11.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:41:11 smithi181 ceph-mon[94189]: pgmap v7649: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 216 KiB/s wr, 26 op/s 2023-03-09T18:41:11.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:41:11 smithi038 ceph-mon[92422]: pgmap v7649: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 216 KiB/s wr, 26 op/s 2023-03-09T18:41:13.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:41:13 smithi122 ceph-mon[93869]: pgmap v7650: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 209 KiB/s wr, 25 op/s 2023-03-09T18:41:13.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:41:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13383.428984 secs 2023-03-09T18:41:13.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:41:13 smithi122 ceph-mon[93869]: osdmap e692: 12 total, 12 up, 12 in 2023-03-09T18:41:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:41:13 smithi181 ceph-mon[94189]: pgmap v7650: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 209 KiB/s wr, 25 op/s 2023-03-09T18:41:13.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:41:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13383.428984 secs 2023-03-09T18:41:13.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:41:13 smithi181 ceph-mon[94189]: osdmap e692: 12 total, 12 up, 12 in 2023-03-09T18:41:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:41:13 smithi038 ceph-mon[92422]: pgmap v7650: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 209 KiB/s wr, 25 op/s 2023-03-09T18:41:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:41:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13383.428984 secs 2023-03-09T18:41:13.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:41:13 smithi038 ceph-mon[92422]: osdmap e692: 12 total, 12 up, 12 in 2023-03-09T18:41:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13845.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:41:15.481 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:41:15 smithi122 ceph-mon[93869]: pgmap v7652: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 184 KiB/s wr, 10 op/s 2023-03-09T18:41:15.482 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:41:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13386.156342 secs 2023-03-09T18:41:15.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:41:15 smithi181 ceph-mon[94189]: pgmap v7652: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 184 KiB/s wr, 10 op/s 2023-03-09T18:41:15.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:41:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13386.156342 secs 2023-03-09T18:41:15.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:41:15 smithi038 ceph-mon[92422]: pgmap v7652: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 184 KiB/s wr, 10 op/s 2023-03-09T18:41:15.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:41:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13386.156342 secs 2023-03-09T18:41:17.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:41:17 smithi122 ceph-mon[93869]: pgmap v7653: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 128 KiB/s wr, 10 op/s 2023-03-09T18:41:17.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:41:17 smithi181 ceph-mon[94189]: pgmap v7653: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 128 KiB/s wr, 10 op/s 2023-03-09T18:41:17.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:41:17 smithi038 ceph-mon[92422]: pgmap v7653: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 128 KiB/s wr, 10 op/s 2023-03-09T18:41:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:41:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13388.429100 secs 2023-03-09T18:41:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:41:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13388.429100 secs 2023-03-09T18:41:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:41:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13388.429100 secs 2023-03-09T18:41:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13850.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:41:19.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:41:19 smithi122 ceph-mon[93869]: pgmap v7654: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 109 KiB/s wr, 9 op/s 2023-03-09T18:41:19.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:41:19 smithi181 ceph-mon[94189]: pgmap v7654: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 109 KiB/s wr, 9 op/s 2023-03-09T18:41:19.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:41:19 smithi038 ceph-mon[92422]: pgmap v7654: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 109 KiB/s wr, 9 op/s 2023-03-09T18:41:20.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:41:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13391.156403 secs 2023-03-09T18:41:20.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:41:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13391.156403 secs 2023-03-09T18:41:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:41:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13391.156403 secs 2023-03-09T18:41:21.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:41:21 smithi122 ceph-mon[93869]: pgmap v7655: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 109 KiB/s wr, 9 op/s 2023-03-09T18:41:21.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:41:21 smithi181 ceph-mon[94189]: pgmap v7655: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 109 KiB/s wr, 9 op/s 2023-03-09T18:41:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:41:21 smithi038 ceph-mon[92422]: pgmap v7655: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 109 KiB/s wr, 9 op/s 2023-03-09T18:41:23.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:41:23 smithi122 ceph-mon[93869]: pgmap v7656: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 112 KiB/s wr, 8 op/s 2023-03-09T18:41:23.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:41:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13393.429189 secs 2023-03-09T18:41:23.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:41:23 smithi181 ceph-mon[94189]: pgmap v7656: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 112 KiB/s wr, 8 op/s 2023-03-09T18:41:23.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:41:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13393.429189 secs 2023-03-09T18:41:23.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:41:23 smithi038 ceph-mon[92422]: pgmap v7656: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 112 KiB/s wr, 8 op/s 2023-03-09T18:41:23.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:41:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13393.429189 secs 2023-03-09T18:41:23.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13855.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:41:25.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:41:25 smithi122 ceph-mon[93869]: pgmap v7657: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 94 KiB/s wr, 7 op/s 2023-03-09T18:41:25.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:41:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13396.156486 secs 2023-03-09T18:41:25.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:41:25 smithi181 ceph-mon[94189]: pgmap v7657: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 94 KiB/s wr, 7 op/s 2023-03-09T18:41:25.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:41:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13396.156486 secs 2023-03-09T18:41:25.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:41:25 smithi038 ceph-mon[92422]: pgmap v7657: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 94 KiB/s wr, 7 op/s 2023-03-09T18:41:25.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:41:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13396.156486 secs 2023-03-09T18:41:27.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:41:27 smithi122 ceph-mon[93869]: pgmap v7658: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 94 KiB/s wr, 9 op/s 2023-03-09T18:41:27.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:41:27 smithi181 ceph-mon[94189]: pgmap v7658: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 94 KiB/s wr, 9 op/s 2023-03-09T18:41:27.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:41:27 smithi038 ceph-mon[92422]: pgmap v7658: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 94 KiB/s wr, 9 op/s 2023-03-09T18:41:28.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:41:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13398.429300 secs 2023-03-09T18:41:28.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:41:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13398.429300 secs 2023-03-09T18:41:28.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:41:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13398.429300 secs 2023-03-09T18:41:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13860.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:41:29.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:41:29 smithi122 ceph-mon[93869]: pgmap v7659: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 43 KiB/s wr, 6 op/s 2023-03-09T18:41:29.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:41:29 smithi181 ceph-mon[94189]: pgmap v7659: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 43 KiB/s wr, 6 op/s 2023-03-09T18:41:29.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:41:29 smithi038 ceph-mon[92422]: pgmap v7659: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 43 KiB/s wr, 6 op/s 2023-03-09T18:41:30.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:41:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13401.156549 secs 2023-03-09T18:41:30.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:41:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13401.156549 secs 2023-03-09T18:41:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:41:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13401.156549 secs 2023-03-09T18:41:31.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:41:31 smithi122 ceph-mon[93869]: pgmap v7660: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 43 KiB/s wr, 6 op/s 2023-03-09T18:41:31.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:41:31 smithi181 ceph-mon[94189]: pgmap v7660: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 43 KiB/s wr, 6 op/s 2023-03-09T18:41:31.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:41:31 smithi038 ceph-mon[92422]: pgmap v7660: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 43 KiB/s wr, 6 op/s 2023-03-09T18:41:33.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:41:33 smithi122 ceph-mon[93869]: pgmap v7661: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 83 KiB/s wr, 9 op/s 2023-03-09T18:41:33.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:41:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13403.429415 secs 2023-03-09T18:41:33.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:41:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:41:33.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:41:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:41:33.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:41:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:41:33.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:41:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:41:33.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:41:33 smithi181 ceph-mon[94189]: pgmap v7661: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 83 KiB/s wr, 9 op/s 2023-03-09T18:41:33.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:41:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13403.429415 secs 2023-03-09T18:41:33.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:41:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:41:33.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:41:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:41:33.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:41:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:41:33.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:41:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:41:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:41:33 smithi038 ceph-mon[92422]: pgmap v7661: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 83 KiB/s wr, 9 op/s 2023-03-09T18:41:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:41:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13403.429415 secs 2023-03-09T18:41:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:41:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:41:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:41:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:41:33.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:41:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:41:33.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:41:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:41:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13865.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:41:35.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:41:35 smithi122 ceph-mon[93869]: pgmap v7662: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 40 KiB/s wr, 5 op/s 2023-03-09T18:41:35.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:41:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13406.156610 secs 2023-03-09T18:41:35.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:41:35 smithi181 ceph-mon[94189]: pgmap v7662: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 40 KiB/s wr, 5 op/s 2023-03-09T18:41:35.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:41:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13406.156610 secs 2023-03-09T18:41:35.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:41:35 smithi038 ceph-mon[92422]: pgmap v7662: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 40 KiB/s wr, 5 op/s 2023-03-09T18:41:35.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:41:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13406.156610 secs 2023-03-09T18:41:37.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:41:37 smithi122 ceph-mon[93869]: pgmap v7663: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 161 KiB/s wr, 10 op/s 2023-03-09T18:41:37.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:41:37 smithi181 ceph-mon[94189]: pgmap v7663: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 161 KiB/s wr, 10 op/s 2023-03-09T18:41:37.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:41:37 smithi038 ceph-mon[92422]: pgmap v7663: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 161 KiB/s wr, 10 op/s 2023-03-09T18:41:38.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:41:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13408.429472 secs 2023-03-09T18:41:38.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:41:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13408.429472 secs 2023-03-09T18:41:38.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:41:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13408.429472 secs 2023-03-09T18:41:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13870.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:41:39.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:41:39 smithi122 ceph-mon[93869]: pgmap v7664: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 161 KiB/s wr, 41 op/s 2023-03-09T18:41:39.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:41:39 smithi181 ceph-mon[94189]: pgmap v7664: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 161 KiB/s wr, 41 op/s 2023-03-09T18:41:39.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:41:39 smithi038 ceph-mon[92422]: pgmap v7664: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 161 KiB/s wr, 41 op/s 2023-03-09T18:41:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:41:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13411.156717 secs 2023-03-09T18:41:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:41:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13411.156717 secs 2023-03-09T18:41:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:41:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13411.156717 secs 2023-03-09T18:41:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:41:41 smithi122 ceph-mon[93869]: pgmap v7665: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 161 KiB/s wr, 41 op/s 2023-03-09T18:41:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:41:41 smithi181 ceph-mon[94189]: pgmap v7665: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 161 KiB/s wr, 41 op/s 2023-03-09T18:41:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:41:41 smithi038 ceph-mon[92422]: pgmap v7665: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 161 KiB/s wr, 41 op/s 2023-03-09T18:41:43.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:41:43 smithi122 ceph-mon[93869]: pgmap v7666: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 206 KiB/s wr, 45 op/s 2023-03-09T18:41:43.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:41:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13413.429585 secs 2023-03-09T18:41:43.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:41:43 smithi181 ceph-mon[94189]: pgmap v7666: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 206 KiB/s wr, 45 op/s 2023-03-09T18:41:43.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:41:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13413.429585 secs 2023-03-09T18:41:43.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:41:43 smithi038 ceph-mon[92422]: pgmap v7666: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 206 KiB/s wr, 45 op/s 2023-03-09T18:41:43.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:41:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13413.429585 secs 2023-03-09T18:41:43.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13875.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:41:45.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:41:45 smithi122 ceph-mon[93869]: pgmap v7667: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 167 KiB/s wr, 43 op/s 2023-03-09T18:41:45.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:41:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13416.156777 secs 2023-03-09T18:41:45.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:41:45 smithi181 ceph-mon[94189]: pgmap v7667: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 167 KiB/s wr, 43 op/s 2023-03-09T18:41:45.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:41:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13416.156777 secs 2023-03-09T18:41:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:41:45 smithi038 ceph-mon[92422]: pgmap v7667: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 167 KiB/s wr, 43 op/s 2023-03-09T18:41:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:41:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13416.156777 secs 2023-03-09T18:41:47.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:41:47 smithi122 ceph-mon[93869]: pgmap v7668: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 255 KiB/s wr, 47 op/s 2023-03-09T18:41:47.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:41:47 smithi181 ceph-mon[94189]: pgmap v7668: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 255 KiB/s wr, 47 op/s 2023-03-09T18:41:47.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:41:47 smithi038 ceph-mon[92422]: pgmap v7668: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 255 KiB/s wr, 47 op/s 2023-03-09T18:41:48.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:41:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13418.429720 secs 2023-03-09T18:41:48.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:41:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13418.429720 secs 2023-03-09T18:41:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:41:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13418.429720 secs 2023-03-09T18:41:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13880.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:41:49.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:41:49 smithi122 ceph-mon[93869]: pgmap v7669: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 134 KiB/s wr, 42 op/s 2023-03-09T18:41:49.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:41:49 smithi181 ceph-mon[94189]: pgmap v7669: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 134 KiB/s wr, 42 op/s 2023-03-09T18:41:49.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:41:49 smithi038 ceph-mon[92422]: pgmap v7669: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 134 KiB/s wr, 42 op/s 2023-03-09T18:41:50.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:41:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13421.156838 secs 2023-03-09T18:41:50.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:41:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13421.156838 secs 2023-03-09T18:41:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:41:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13421.156838 secs 2023-03-09T18:41:51.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:41:51 smithi122 ceph-mon[93869]: pgmap v7670: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 134 KiB/s wr, 8 op/s 2023-03-09T18:41:51.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:41:51 smithi181 ceph-mon[94189]: pgmap v7670: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 134 KiB/s wr, 8 op/s 2023-03-09T18:41:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:41:51 smithi038 ceph-mon[92422]: pgmap v7670: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 134 KiB/s wr, 8 op/s 2023-03-09T18:41:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:41:53 smithi122 ceph-mon[93869]: pgmap v7671: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 222 KiB/s wr, 12 op/s 2023-03-09T18:41:53.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:41:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13423.429836 secs 2023-03-09T18:41:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:41:53 smithi181 ceph-mon[94189]: pgmap v7671: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 222 KiB/s wr, 12 op/s 2023-03-09T18:41:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:41:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13423.429836 secs 2023-03-09T18:41:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:41:53 smithi038 ceph-mon[92422]: pgmap v7671: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 222 KiB/s wr, 12 op/s 2023-03-09T18:41:53.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:41:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13423.429836 secs 2023-03-09T18:41:53.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13885.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:41:55.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:41:55 smithi122 ceph-mon[93869]: pgmap v7672: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 178 KiB/s wr, 9 op/s 2023-03-09T18:41:55.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:41:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13426.156897 secs 2023-03-09T18:41:55.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:41:55 smithi181 ceph-mon[94189]: pgmap v7672: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 178 KiB/s wr, 9 op/s 2023-03-09T18:41:55.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:41:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13426.156897 secs 2023-03-09T18:41:55.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:41:55 smithi038 ceph-mon[92422]: pgmap v7672: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 178 KiB/s wr, 9 op/s 2023-03-09T18:41:55.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:41:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13426.156897 secs 2023-03-09T18:41:57.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:41:57 smithi122 ceph-mon[93869]: pgmap v7673: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 209 KiB/s wr, 12 op/s 2023-03-09T18:41:57.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:41:57 smithi181 ceph-mon[94189]: pgmap v7673: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 209 KiB/s wr, 12 op/s 2023-03-09T18:41:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:41:57 smithi038 ceph-mon[92422]: pgmap v7673: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 209 KiB/s wr, 12 op/s 2023-03-09T18:41:58.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:41:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13428.429953 secs 2023-03-09T18:41:58.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:41:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13428.429953 secs 2023-03-09T18:41:58.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:41:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13428.429953 secs 2023-03-09T18:41:58.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13890.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:41:59.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:41:59 smithi122 ceph-mon[93869]: pgmap v7674: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 120 KiB/s wr, 8 op/s 2023-03-09T18:41:59.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:41:59 smithi181 ceph-mon[94189]: pgmap v7674: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 120 KiB/s wr, 8 op/s 2023-03-09T18:41:59.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:41:59 smithi038 ceph-mon[92422]: pgmap v7674: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 120 KiB/s wr, 8 op/s 2023-03-09T18:42:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13431.156957 secs 2023-03-09T18:42:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13431.156957 secs 2023-03-09T18:42:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13431.156957 secs 2023-03-09T18:42:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:01 smithi122 ceph-mon[93869]: pgmap v7675: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 120 KiB/s wr, 8 op/s 2023-03-09T18:42:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:01 smithi181 ceph-mon[94189]: pgmap v7675: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 120 KiB/s wr, 8 op/s 2023-03-09T18:42:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:01 smithi038 ceph-mon[92422]: pgmap v7675: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 120 KiB/s wr, 8 op/s 2023-03-09T18:42:03.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:03 smithi122 ceph-mon[93869]: pgmap v7676: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 239 KiB/s wr, 14 op/s 2023-03-09T18:42:03.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13433.430066 secs 2023-03-09T18:42:03.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:03 smithi181 ceph-mon[94189]: pgmap v7676: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 239 KiB/s wr, 14 op/s 2023-03-09T18:42:03.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13433.430066 secs 2023-03-09T18:42:03.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:03 smithi038 ceph-mon[92422]: pgmap v7676: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 239 KiB/s wr, 14 op/s 2023-03-09T18:42:03.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13433.430066 secs 2023-03-09T18:42:03.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13895.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:42:05.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:05 smithi122 ceph-mon[93869]: pgmap v7677: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 172 KiB/s wr, 22 op/s 2023-03-09T18:42:05.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13436.156982 secs 2023-03-09T18:42:05.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:05 smithi181 ceph-mon[94189]: pgmap v7677: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 172 KiB/s wr, 22 op/s 2023-03-09T18:42:05.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13436.156982 secs 2023-03-09T18:42:05.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:05 smithi038 ceph-mon[92422]: pgmap v7677: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 172 KiB/s wr, 22 op/s 2023-03-09T18:42:05.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13436.156982 secs 2023-03-09T18:42:07.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:07 smithi122 ceph-mon[93869]: pgmap v7678: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 173 KiB/s wr, 25 op/s 2023-03-09T18:42:07.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13438.430182 secs 2023-03-09T18:42:07.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:07 smithi181 ceph-mon[94189]: pgmap v7678: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 173 KiB/s wr, 25 op/s 2023-03-09T18:42:07.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13438.430182 secs 2023-03-09T18:42:07.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:07 smithi038 ceph-mon[92422]: pgmap v7678: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 173 KiB/s wr, 25 op/s 2023-03-09T18:42:07.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13438.430182 secs 2023-03-09T18:42:08.801 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13900.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:42:09.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:09 smithi122 ceph-mon[93869]: pgmap v7679: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 141 KiB/s wr, 22 op/s 2023-03-09T18:42:09.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:09 smithi181 ceph-mon[94189]: pgmap v7679: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 141 KiB/s wr, 22 op/s 2023-03-09T18:42:09.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:09 smithi038 ceph-mon[92422]: pgmap v7679: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 141 KiB/s wr, 22 op/s 2023-03-09T18:42:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:10 smithi122 ceph-mon[93869]: osdmap e693: 12 total, 12 up, 12 in 2023-03-09T18:42:10.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13441.157089 secs 2023-03-09T18:42:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:10 smithi181 ceph-mon[94189]: osdmap e693: 12 total, 12 up, 12 in 2023-03-09T18:42:10.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13441.157089 secs 2023-03-09T18:42:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:10 smithi038 ceph-mon[92422]: osdmap e693: 12 total, 12 up, 12 in 2023-03-09T18:42:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13441.157089 secs 2023-03-09T18:42:11.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:11 smithi122 ceph-mon[93869]: pgmap v7681: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 170 KiB/s wr, 26 op/s 2023-03-09T18:42:11.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:11 smithi181 ceph-mon[94189]: pgmap v7681: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 170 KiB/s wr, 26 op/s 2023-03-09T18:42:11.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:11 smithi038 ceph-mon[92422]: pgmap v7681: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 170 KiB/s wr, 26 op/s 2023-03-09T18:42:13.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:13 smithi122 ceph-mon[93869]: pgmap v7682: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 68 KiB/s wr, 21 op/s 2023-03-09T18:42:13.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13443.430274 secs 2023-03-09T18:42:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:13 smithi181 ceph-mon[94189]: pgmap v7682: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 68 KiB/s wr, 21 op/s 2023-03-09T18:42:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13443.430274 secs 2023-03-09T18:42:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:13 smithi038 ceph-mon[92422]: pgmap v7682: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 68 KiB/s wr, 21 op/s 2023-03-09T18:42:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13443.430274 secs 2023-03-09T18:42:13.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13905.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:42:15.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:15 smithi122 ceph-mon[93869]: pgmap v7683: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 42 KiB/s wr, 7 op/s 2023-03-09T18:42:15.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13446.157150 secs 2023-03-09T18:42:15.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:15 smithi181 ceph-mon[94189]: pgmap v7683: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 42 KiB/s wr, 7 op/s 2023-03-09T18:42:15.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13446.157150 secs 2023-03-09T18:42:15.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:15 smithi038 ceph-mon[92422]: pgmap v7683: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 42 KiB/s wr, 7 op/s 2023-03-09T18:42:15.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13446.157150 secs 2023-03-09T18:42:17.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:17 smithi122 ceph-mon[93869]: pgmap v7684: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 6 op/s 2023-03-09T18:42:17.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:17 smithi181 ceph-mon[94189]: pgmap v7684: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 6 op/s 2023-03-09T18:42:17.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:17 smithi038 ceph-mon[92422]: pgmap v7684: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 6 op/s 2023-03-09T18:42:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13448.430389 secs 2023-03-09T18:42:18.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:18 smithi122 ceph-mon[93869]: osdmap e694: 12 total, 12 up, 12 in 2023-03-09T18:42:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13448.430389 secs 2023-03-09T18:42:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:18 smithi181 ceph-mon[94189]: osdmap e694: 12 total, 12 up, 12 in 2023-03-09T18:42:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13448.430389 secs 2023-03-09T18:42:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:18 smithi038 ceph-mon[92422]: osdmap e694: 12 total, 12 up, 12 in 2023-03-09T18:42:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13910.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:42:19.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:19 smithi122 ceph-mon[93869]: pgmap v7686: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 47 KiB/s wr, 8 op/s 2023-03-09T18:42:19.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:19 smithi181 ceph-mon[94189]: pgmap v7686: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 47 KiB/s wr, 8 op/s 2023-03-09T18:42:19.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:19 smithi038 ceph-mon[92422]: pgmap v7686: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 47 KiB/s wr, 8 op/s 2023-03-09T18:42:20.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13451.157212 secs 2023-03-09T18:42:20.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13451.157212 secs 2023-03-09T18:42:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13451.157212 secs 2023-03-09T18:42:21.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:21 smithi122 ceph-mon[93869]: pgmap v7687: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 41 KiB/s wr, 7 op/s 2023-03-09T18:42:21.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:21 smithi181 ceph-mon[94189]: pgmap v7687: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 41 KiB/s wr, 7 op/s 2023-03-09T18:42:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:21 smithi038 ceph-mon[92422]: pgmap v7687: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 41 KiB/s wr, 7 op/s 2023-03-09T18:42:23.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:23 smithi122 ceph-mon[93869]: pgmap v7688: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 8 op/s 2023-03-09T18:42:23.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13453.430466 secs 2023-03-09T18:42:23.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:23 smithi181 ceph-mon[94189]: pgmap v7688: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 8 op/s 2023-03-09T18:42:23.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13453.430466 secs 2023-03-09T18:42:23.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:23 smithi038 ceph-mon[92422]: pgmap v7688: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 8 op/s 2023-03-09T18:42:23.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13453.430466 secs 2023-03-09T18:42:23.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13915.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:42:25.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:25 smithi122 ceph-mon[93869]: pgmap v7689: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 8 op/s 2023-03-09T18:42:25.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13456.157269 secs 2023-03-09T18:42:25.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:25 smithi181 ceph-mon[94189]: pgmap v7689: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 8 op/s 2023-03-09T18:42:25.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13456.157269 secs 2023-03-09T18:42:25.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:25 smithi038 ceph-mon[92422]: pgmap v7689: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 8 op/s 2023-03-09T18:42:25.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13456.157269 secs 2023-03-09T18:42:27.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:27 smithi122 ceph-mon[93869]: pgmap v7690: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 100 KiB/s wr, 8 op/s 2023-03-09T18:42:27.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:27 smithi181 ceph-mon[94189]: pgmap v7690: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 100 KiB/s wr, 8 op/s 2023-03-09T18:42:27.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:27 smithi038 ceph-mon[92422]: pgmap v7690: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 100 KiB/s wr, 8 op/s 2023-03-09T18:42:28.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13458.430576 secs 2023-03-09T18:42:28.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13458.430576 secs 2023-03-09T18:42:28.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13458.430576 secs 2023-03-09T18:42:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13920.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:42:29.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:29 smithi122 ceph-mon[93869]: pgmap v7691: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 91 KiB/s wr, 7 op/s 2023-03-09T18:42:29.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:29 smithi181 ceph-mon[94189]: pgmap v7691: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 91 KiB/s wr, 7 op/s 2023-03-09T18:42:29.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:29 smithi038 ceph-mon[92422]: pgmap v7691: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 91 KiB/s wr, 7 op/s 2023-03-09T18:42:30.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13461.157329 secs 2023-03-09T18:42:30.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13461.157329 secs 2023-03-09T18:42:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13461.157329 secs 2023-03-09T18:42:31.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:31 smithi122 ceph-mon[93869]: pgmap v7692: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 83 KiB/s wr, 6 op/s 2023-03-09T18:42:31.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:31 smithi181 ceph-mon[94189]: pgmap v7692: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 83 KiB/s wr, 6 op/s 2023-03-09T18:42:31.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:31 smithi038 ceph-mon[92422]: pgmap v7692: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 83 KiB/s wr, 6 op/s 2023-03-09T18:42:33.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:33 smithi122 ceph-mon[93869]: pgmap v7693: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 84 KiB/s wr, 9 op/s 2023-03-09T18:42:33.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13463.430691 secs 2023-03-09T18:42:33.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:42:33.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:33 smithi181 ceph-mon[94189]: pgmap v7693: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 84 KiB/s wr, 9 op/s 2023-03-09T18:42:33.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13463.430691 secs 2023-03-09T18:42:33.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:42:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:33 smithi038 ceph-mon[92422]: pgmap v7693: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 84 KiB/s wr, 9 op/s 2023-03-09T18:42:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13463.430691 secs 2023-03-09T18:42:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:42:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13925.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:42:35.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:35 smithi122 ceph-mon[93869]: pgmap v7694: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 40 KiB/s wr, 6 op/s 2023-03-09T18:42:35.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13466.157389 secs 2023-03-09T18:42:35.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:35 smithi181 ceph-mon[94189]: pgmap v7694: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 40 KiB/s wr, 6 op/s 2023-03-09T18:42:35.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13466.157389 secs 2023-03-09T18:42:35.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:35 smithi038 ceph-mon[92422]: pgmap v7694: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 40 KiB/s wr, 6 op/s 2023-03-09T18:42:35.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13466.157389 secs 2023-03-09T18:42:37.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:42:37.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:42:37.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:42:37.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:42:37.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:42:37.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:42:37.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:42:37.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:42:37.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:42:37.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:42:37.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:42:37.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:42:37.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:42:37.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:42:37.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:42:37.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:42:37.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:42:37.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:42:37.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:42:37.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:42:37.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:42:37.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:42:37.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:42:37.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:42:37.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:42:37.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:42:37.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:42:37.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:42:37.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:42:37.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:42:37.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:42:37.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:42:37.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:42:37.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:42:37.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:42:37.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:42:37.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:42:37.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:42:37.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:42:37.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:42:37.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:42:37.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:42:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:37 smithi122 ceph-mon[93869]: pgmap v7695: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 94 KiB/s wr, 9 op/s 2023-03-09T18:42:38.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:37 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi038 to 186.5M 2023-03-09T18:42:38.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:37 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T18:42:38.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:37 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi038 to 195602792: error parsing value: Value '195602792' is below minimum 939524096 2023-03-09T18:42:38.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:37 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T18:42:38.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:42:38.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:42:38.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:42:38.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:42:38.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:42:38.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:42:38.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:42:38.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:37 smithi181 ceph-mon[94189]: pgmap v7695: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 94 KiB/s wr, 9 op/s 2023-03-09T18:42:38.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:37 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi038 to 186.5M 2023-03-09T18:42:38.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:37 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T18:42:38.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:37 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi038 to 195602792: error parsing value: Value '195602792' is below minimum 939524096 2023-03-09T18:42:38.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:37 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T18:42:38.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:42:38.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:42:38.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:42:38.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:42:38.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:42:38.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:42:38.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:42:38.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:37 smithi038 ceph-mon[92422]: pgmap v7695: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 94 KiB/s wr, 9 op/s 2023-03-09T18:42:38.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:37 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi038 to 186.5M 2023-03-09T18:42:38.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:37 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T18:42:38.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:37 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi038 to 195602792: error parsing value: Value '195602792' is below minimum 939524096 2023-03-09T18:42:38.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:37 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T18:42:38.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:42:38.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:42:38.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:42:38.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:42:38.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:42:38.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:42:38.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:42:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13930.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:42:39.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:38 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T18:42:39.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:38 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T18:42:39.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:38 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T18:42:39.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:38 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T18:42:39.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:38 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T18:42:39.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:38 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T18:42:40.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:39 smithi122 ceph-mon[93869]: pgmap v7696: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 54 KiB/s wr, 6 op/s 2023-03-09T18:42:40.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:39 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:42:40.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:39 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13471.157470 secs 2023-03-09T18:42:40.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:39 smithi181 ceph-mon[94189]: pgmap v7696: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 54 KiB/s wr, 6 op/s 2023-03-09T18:42:40.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:39 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:42:40.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:39 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13471.157470 secs 2023-03-09T18:42:40.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:39 smithi038 ceph-mon[92422]: pgmap v7696: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 54 KiB/s wr, 6 op/s 2023-03-09T18:42:40.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:39 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:42:40.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:39 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13471.157470 secs 2023-03-09T18:42:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:40 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:42:41.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:42:41.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:42:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:40 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:42:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:42:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:42:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:40 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:42:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:42:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:42:42.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:41 smithi122 ceph-mon[93869]: pgmap v7697: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 54 KiB/s wr, 6 op/s 2023-03-09T18:42:42.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:41 smithi181 ceph-mon[94189]: pgmap v7697: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 54 KiB/s wr, 6 op/s 2023-03-09T18:42:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:41 smithi038 ceph-mon[92422]: pgmap v7697: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 54 KiB/s wr, 6 op/s 2023-03-09T18:42:43.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13935.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:42:44.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:43 smithi122 ceph-mon[93869]: pgmap v7698: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 177 KiB/s wr, 10 op/s 2023-03-09T18:42:44.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:43 smithi181 ceph-mon[94189]: pgmap v7698: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 177 KiB/s wr, 10 op/s 2023-03-09T18:42:44.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:43 smithi038 ceph-mon[92422]: pgmap v7698: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 177 KiB/s wr, 10 op/s 2023-03-09T18:42:45.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:44 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13476.157536 secs 2023-03-09T18:42:45.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:44 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13476.157536 secs 2023-03-09T18:42:45.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:44 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13476.157536 secs 2023-03-09T18:42:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:45 smithi122 ceph-mon[93869]: pgmap v7699: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 176 KiB/s wr, 8 op/s 2023-03-09T18:42:46.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:45 smithi181 ceph-mon[94189]: pgmap v7699: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 176 KiB/s wr, 8 op/s 2023-03-09T18:42:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:45 smithi038 ceph-mon[92422]: pgmap v7699: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 176 KiB/s wr, 8 op/s 2023-03-09T18:42:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:47 smithi122 ceph-mon[93869]: pgmap v7700: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 187 KiB/s wr, 11 op/s 2023-03-09T18:42:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:47 smithi181 ceph-mon[94189]: pgmap v7700: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 187 KiB/s wr, 11 op/s 2023-03-09T18:42:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:47 smithi038 ceph-mon[92422]: pgmap v7700: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 187 KiB/s wr, 11 op/s 2023-03-09T18:42:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13940.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:42:50.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:49 smithi122 ceph-mon[93869]: pgmap v7701: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 134 KiB/s wr, 9 op/s 2023-03-09T18:42:50.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:49 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13481.157595 secs 2023-03-09T18:42:50.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:49 smithi181 ceph-mon[94189]: pgmap v7701: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 134 KiB/s wr, 9 op/s 2023-03-09T18:42:50.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:49 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13481.157595 secs 2023-03-09T18:42:50.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:49 smithi038 ceph-mon[92422]: pgmap v7701: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 134 KiB/s wr, 9 op/s 2023-03-09T18:42:50.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:49 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13481.157595 secs 2023-03-09T18:42:52.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:51 smithi122 ceph-mon[93869]: pgmap v7702: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 134 KiB/s wr, 9 op/s 2023-03-09T18:42:52.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:51 smithi181 ceph-mon[94189]: pgmap v7702: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 134 KiB/s wr, 9 op/s 2023-03-09T18:42:52.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:51 smithi038 ceph-mon[92422]: pgmap v7702: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 134 KiB/s wr, 9 op/s 2023-03-09T18:42:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13945.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:42:54.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:53 smithi122 ceph-mon[93869]: pgmap v7703: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 300 KiB/s wr, 13 op/s 2023-03-09T18:42:54.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:53 smithi181 ceph-mon[94189]: pgmap v7703: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 300 KiB/s wr, 13 op/s 2023-03-09T18:42:54.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:53 smithi038 ceph-mon[92422]: pgmap v7703: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 300 KiB/s wr, 13 op/s 2023-03-09T18:42:55.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:54 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13486.157655 secs 2023-03-09T18:42:55.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:54 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13486.157655 secs 2023-03-09T18:42:55.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:54 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13486.157655 secs 2023-03-09T18:42:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:55 smithi122 ceph-mon[93869]: pgmap v7704: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 178 KiB/s wr, 9 op/s 2023-03-09T18:42:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:55 smithi181 ceph-mon[94189]: pgmap v7704: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 178 KiB/s wr, 9 op/s 2023-03-09T18:42:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:55 smithi038 ceph-mon[92422]: pgmap v7704: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 178 KiB/s wr, 9 op/s 2023-03-09T18:42:57.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:56 smithi122 ceph-mon[93869]: pgmap v7705: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 277 KiB/s wr, 12 op/s 2023-03-09T18:42:57.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:56 smithi181 ceph-mon[94189]: pgmap v7705: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 277 KiB/s wr, 12 op/s 2023-03-09T18:42:57.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:56 smithi038 ceph-mon[92422]: pgmap v7705: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 277 KiB/s wr, 12 op/s 2023-03-09T18:42:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13950.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:42:59.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:42:59 smithi122 ceph-mon[93869]: pgmap v7706: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 266 KiB/s wr, 8 op/s 2023-03-09T18:42:59.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:42:59 smithi181 ceph-mon[94189]: pgmap v7706: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 266 KiB/s wr, 8 op/s 2023-03-09T18:42:59.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:42:59 smithi038 ceph-mon[92422]: pgmap v7706: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 266 KiB/s wr, 8 op/s 2023-03-09T18:43:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:43:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13491.157717 secs 2023-03-09T18:43:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:43:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13491.157717 secs 2023-03-09T18:43:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:43:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13491.157717 secs 2023-03-09T18:43:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:43:01 smithi122 ceph-mon[93869]: pgmap v7707: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 265 KiB/s wr, 8 op/s 2023-03-09T18:43:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:43:01 smithi181 ceph-mon[94189]: pgmap v7707: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 265 KiB/s wr, 8 op/s 2023-03-09T18:43:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:43:01 smithi038 ceph-mon[92422]: pgmap v7707: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 265 KiB/s wr, 8 op/s 2023-03-09T18:43:03.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:43:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13493.431479 secs 2023-03-09T18:43:03.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:43:03 smithi122 ceph-mon[93869]: pgmap v7708: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 357 KiB/s wr, 13 op/s 2023-03-09T18:43:03.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:43:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13493.431479 secs 2023-03-09T18:43:03.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:43:03 smithi181 ceph-mon[94189]: pgmap v7708: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 357 KiB/s wr, 13 op/s 2023-03-09T18:43:03.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:43:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13493.431479 secs 2023-03-09T18:43:03.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:43:03 smithi038 ceph-mon[92422]: pgmap v7708: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 357 KiB/s wr, 13 op/s 2023-03-09T18:43:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13955.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:43:04.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:43:04 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:43:04.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:43:04 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:43:04.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:43:04 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:43:04.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:43:04 smithi122 ceph-mon[93869]: osdmap e695: 12 total, 12 up, 12 in 2023-03-09T18:43:04.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:43:04 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:43:04.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:43:04 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:43:04.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:43:04 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:43:04.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:43:04 smithi181 ceph-mon[94189]: osdmap e695: 12 total, 12 up, 12 in 2023-03-09T18:43:04.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:43:04 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:43:04.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:43:04 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:43:04.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:43:04 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:43:04.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:43:04 smithi038 ceph-mon[92422]: osdmap e695: 12 total, 12 up, 12 in 2023-03-09T18:43:05.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:43:05 smithi122 ceph-mon[93869]: pgmap v7709: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 212 KiB/s wr, 21 op/s 2023-03-09T18:43:05.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:43:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13496.157794 secs 2023-03-09T18:43:05.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:43:05 smithi181 ceph-mon[94189]: pgmap v7709: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 212 KiB/s wr, 21 op/s 2023-03-09T18:43:05.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:43:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13496.157794 secs 2023-03-09T18:43:05.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:43:05 smithi038 ceph-mon[92422]: pgmap v7709: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 212 KiB/s wr, 21 op/s 2023-03-09T18:43:05.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:43:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13496.157794 secs 2023-03-09T18:43:07.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:43:07 smithi122 ceph-mon[93869]: pgmap v7711: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 218 KiB/s wr, 27 op/s 2023-03-09T18:43:07.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:43:07 smithi181 ceph-mon[94189]: pgmap v7711: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 218 KiB/s wr, 27 op/s 2023-03-09T18:43:07.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:43:07 smithi038 ceph-mon[92422]: pgmap v7711: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 218 KiB/s wr, 27 op/s 2023-03-09T18:43:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:43:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13498.431633 secs 2023-03-09T18:43:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:43:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13498.431633 secs 2023-03-09T18:43:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:43:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13498.431633 secs 2023-03-09T18:43:08.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13960.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:43:09.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:43:09 smithi122 ceph-mon[93869]: pgmap v7712: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 218 KiB/s wr, 27 op/s 2023-03-09T18:43:09.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:43:09 smithi181 ceph-mon[94189]: pgmap v7712: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 218 KiB/s wr, 27 op/s 2023-03-09T18:43:09.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:43:09 smithi038 ceph-mon[92422]: pgmap v7712: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 218 KiB/s wr, 27 op/s 2023-03-09T18:43:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:43:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13501.157854 secs 2023-03-09T18:43:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:43:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13501.157854 secs 2023-03-09T18:43:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:43:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13501.157854 secs 2023-03-09T18:43:11.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:43:11 smithi122 ceph-mon[93869]: pgmap v7713: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 218 KiB/s wr, 27 op/s 2023-03-09T18:43:11.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:43:11 smithi181 ceph-mon[94189]: pgmap v7713: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 218 KiB/s wr, 27 op/s 2023-03-09T18:43:11.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:43:11 smithi038 ceph-mon[92422]: pgmap v7713: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 218 KiB/s wr, 27 op/s 2023-03-09T18:43:13.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:43:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13503.431749 secs 2023-03-09T18:43:13.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:43:13 smithi122 ceph-mon[93869]: pgmap v7714: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 169 KiB/s wr, 25 op/s 2023-03-09T18:43:13.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:43:13 smithi122 ceph-mon[93869]: osdmap e696: 12 total, 12 up, 12 in 2023-03-09T18:43:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:43:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13503.431749 secs 2023-03-09T18:43:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:43:13 smithi181 ceph-mon[94189]: pgmap v7714: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 169 KiB/s wr, 25 op/s 2023-03-09T18:43:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:43:13 smithi181 ceph-mon[94189]: osdmap e696: 12 total, 12 up, 12 in 2023-03-09T18:43:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:43:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13503.431749 secs 2023-03-09T18:43:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:43:13 smithi038 ceph-mon[92422]: pgmap v7714: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 169 KiB/s wr, 25 op/s 2023-03-09T18:43:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:43:13 smithi038 ceph-mon[92422]: osdmap e696: 12 total, 12 up, 12 in 2023-03-09T18:43:13.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13965.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:43:15.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:43:15 smithi122 ceph-mon[93869]: pgmap v7716: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 144 KiB/s wr, 11 op/s 2023-03-09T18:43:15.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:43:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13506.157911 secs 2023-03-09T18:43:15.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:43:15 smithi181 ceph-mon[94189]: pgmap v7716: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 144 KiB/s wr, 11 op/s 2023-03-09T18:43:15.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:43:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13506.157911 secs 2023-03-09T18:43:15.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:43:15 smithi038 ceph-mon[92422]: pgmap v7716: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 144 KiB/s wr, 11 op/s 2023-03-09T18:43:15.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:43:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13506.157911 secs 2023-03-09T18:43:17.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:43:17 smithi122 ceph-mon[93869]: pgmap v7717: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 61 KiB/s wr, 9 op/s 2023-03-09T18:43:17.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:43:17 smithi181 ceph-mon[94189]: pgmap v7717: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 61 KiB/s wr, 9 op/s 2023-03-09T18:43:17.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:43:17 smithi038 ceph-mon[92422]: pgmap v7717: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 61 KiB/s wr, 9 op/s 2023-03-09T18:43:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:43:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13508.431871 secs 2023-03-09T18:43:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:43:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13508.431871 secs 2023-03-09T18:43:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:43:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13508.431871 secs 2023-03-09T18:43:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13970.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:43:19.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:43:19 smithi122 ceph-mon[93869]: pgmap v7718: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 62 KiB/s wr, 8 op/s 2023-03-09T18:43:19.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:43:19 smithi181 ceph-mon[94189]: pgmap v7718: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 62 KiB/s wr, 8 op/s 2023-03-09T18:43:19.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:43:19 smithi038 ceph-mon[92422]: pgmap v7718: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 62 KiB/s wr, 8 op/s 2023-03-09T18:43:20.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:43:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13511.157996 secs 2023-03-09T18:43:20.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:43:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13511.157996 secs 2023-03-09T18:43:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:43:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13511.157996 secs 2023-03-09T18:43:21.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:43:21 smithi122 ceph-mon[93869]: pgmap v7719: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 62 KiB/s wr, 8 op/s 2023-03-09T18:43:21.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:43:21 smithi181 ceph-mon[94189]: pgmap v7719: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 62 KiB/s wr, 8 op/s 2023-03-09T18:43:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:43:21 smithi038 ceph-mon[92422]: pgmap v7719: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 62 KiB/s wr, 8 op/s 2023-03-09T18:43:23.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:43:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13513.432015 secs 2023-03-09T18:43:23.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:43:23 smithi122 ceph-mon[93869]: pgmap v7720: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 8 op/s 2023-03-09T18:43:23.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:43:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13513.432015 secs 2023-03-09T18:43:23.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:43:23 smithi181 ceph-mon[94189]: pgmap v7720: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 8 op/s 2023-03-09T18:43:23.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:43:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13513.432015 secs 2023-03-09T18:43:23.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:43:23 smithi038 ceph-mon[92422]: pgmap v7720: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 15 KiB/s wr, 8 op/s 2023-03-09T18:43:23.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13975.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:43:25.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:43:25 smithi122 ceph-mon[93869]: pgmap v7721: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 7 op/s 2023-03-09T18:43:25.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:43:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13516.158062 secs 2023-03-09T18:43:25.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:43:25 smithi181 ceph-mon[94189]: pgmap v7721: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 7 op/s 2023-03-09T18:43:25.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:43:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13516.158062 secs 2023-03-09T18:43:25.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:43:25 smithi038 ceph-mon[92422]: pgmap v7721: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 12 KiB/s wr, 7 op/s 2023-03-09T18:43:25.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:43:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13516.158062 secs 2023-03-09T18:43:27.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:43:27 smithi122 ceph-mon[93869]: pgmap v7722: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 56 KiB/s wr, 10 op/s 2023-03-09T18:43:27.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:43:27 smithi181 ceph-mon[94189]: pgmap v7722: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 56 KiB/s wr, 10 op/s 2023-03-09T18:43:27.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:43:27 smithi038 ceph-mon[92422]: pgmap v7722: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 56 KiB/s wr, 10 op/s 2023-03-09T18:43:28.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:43:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13518.432152 secs 2023-03-09T18:43:28.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:43:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13518.432152 secs 2023-03-09T18:43:28.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:43:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13518.432152 secs 2023-03-09T18:43:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13980.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:43:29.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:43:29 smithi122 ceph-mon[93869]: pgmap v7723: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 56 KiB/s wr, 7 op/s 2023-03-09T18:43:29.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:43:29 smithi181 ceph-mon[94189]: pgmap v7723: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 56 KiB/s wr, 7 op/s 2023-03-09T18:43:29.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:43:29 smithi038 ceph-mon[92422]: pgmap v7723: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 56 KiB/s wr, 7 op/s 2023-03-09T18:43:30.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:43:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13521.158123 secs 2023-03-09T18:43:30.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:43:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13521.158123 secs 2023-03-09T18:43:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:43:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13521.158123 secs 2023-03-09T18:43:31.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:43:31 smithi122 ceph-mon[93869]: pgmap v7724: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 55 KiB/s wr, 7 op/s 2023-03-09T18:43:31.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:43:31 smithi181 ceph-mon[94189]: pgmap v7724: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 55 KiB/s wr, 7 op/s 2023-03-09T18:43:31.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:43:31 smithi038 ceph-mon[92422]: pgmap v7724: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 55 KiB/s wr, 7 op/s 2023-03-09T18:43:33.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:43:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13523.432304 secs 2023-03-09T18:43:33.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:43:33 smithi122 ceph-mon[93869]: pgmap v7725: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 95 KiB/s wr, 10 op/s 2023-03-09T18:43:33.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:43:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13523.432304 secs 2023-03-09T18:43:33.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:43:33 smithi181 ceph-mon[94189]: pgmap v7725: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 95 KiB/s wr, 10 op/s 2023-03-09T18:43:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:43:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13523.432304 secs 2023-03-09T18:43:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:43:33 smithi038 ceph-mon[92422]: pgmap v7725: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 95 KiB/s wr, 10 op/s 2023-03-09T18:43:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13985.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:43:35.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:43:35 smithi122 ceph-mon[93869]: pgmap v7726: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 84 KiB/s wr, 7 op/s 2023-03-09T18:43:35.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:43:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13526.158208 secs 2023-03-09T18:43:35.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:43:35 smithi181 ceph-mon[94189]: pgmap v7726: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 84 KiB/s wr, 7 op/s 2023-03-09T18:43:35.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:43:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13526.158208 secs 2023-03-09T18:43:35.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:43:35 smithi038 ceph-mon[92422]: pgmap v7726: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 84 KiB/s wr, 7 op/s 2023-03-09T18:43:35.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:43:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13526.158208 secs 2023-03-09T18:43:37.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:43:37 smithi122 ceph-mon[93869]: pgmap v7727: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 129 KiB/s wr, 9 op/s 2023-03-09T18:43:37.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:43:37 smithi181 ceph-mon[94189]: pgmap v7727: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 129 KiB/s wr, 9 op/s 2023-03-09T18:43:37.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:43:37 smithi038 ceph-mon[92422]: pgmap v7727: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 129 KiB/s wr, 9 op/s 2023-03-09T18:43:38.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:43:38 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 13528.432421 secs 2023-03-09T18:43:38.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:43:38 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:43:38.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:43:38 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:43:38.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:43:38 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:43:38.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:43:38 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:43:38.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:43:38 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 13528.432421 secs 2023-03-09T18:43:38.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:43:38 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:43:38.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:43:38 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:43:38.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:43:38 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:43:38.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:43:38 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:43:38.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:43:38 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 13528.432421 secs 2023-03-09T18:43:38.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:43:38 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:43:38.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:43:38 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:43:38.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:43:38 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:43:38.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:43:38 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:43:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13990.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:43:39.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:43:39 smithi122 ceph-mon[93869]: pgmap v7728: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 85 KiB/s wr, 6 op/s 2023-03-09T18:43:39.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:43:39 smithi181 ceph-mon[94189]: pgmap v7728: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 85 KiB/s wr, 6 op/s 2023-03-09T18:43:39.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:43:39 smithi038 ceph-mon[92422]: pgmap v7728: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 85 KiB/s wr, 6 op/s 2023-03-09T18:43:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:43:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13531.158270 secs 2023-03-09T18:43:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:43:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13531.158270 secs 2023-03-09T18:43:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:43:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13531.158270 secs 2023-03-09T18:43:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:43:41 smithi122 ceph-mon[93869]: pgmap v7729: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 85 KiB/s wr, 6 op/s 2023-03-09T18:43:41.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:43:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:43:41.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:43:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:43:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:43:41 smithi181 ceph-mon[94189]: pgmap v7729: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 85 KiB/s wr, 6 op/s 2023-03-09T18:43:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:43:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:43:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:43:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:43:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:43:41 smithi038 ceph-mon[92422]: pgmap v7729: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 85 KiB/s wr, 6 op/s 2023-03-09T18:43:41.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:43:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:43:41.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:43:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:43:43.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:43:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13533.432530 secs 2023-03-09T18:43:43.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:43:43 smithi122 ceph-mon[93869]: pgmap v7730: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 252 KiB/s wr, 11 op/s 2023-03-09T18:43:43.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:43:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13533.432530 secs 2023-03-09T18:43:43.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:43:43 smithi181 ceph-mon[94189]: pgmap v7730: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 252 KiB/s wr, 11 op/s 2023-03-09T18:43:43.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:43:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13533.432530 secs 2023-03-09T18:43:43.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:43:43 smithi038 ceph-mon[92422]: pgmap v7730: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 252 KiB/s wr, 11 op/s 2023-03-09T18:43:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 13995.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:43:45.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:43:45 smithi122 ceph-mon[93869]: pgmap v7731: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 211 KiB/s wr, 9 op/s 2023-03-09T18:43:45.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:43:45 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:43:45.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:43:45 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:43:45.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:43:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13536.158333 secs 2023-03-09T18:43:45.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:43:45 smithi181 ceph-mon[94189]: pgmap v7731: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 211 KiB/s wr, 9 op/s 2023-03-09T18:43:45.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:43:45 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:43:45.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:43:45 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:43:45.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:43:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13536.158333 secs 2023-03-09T18:43:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:43:45 smithi038 ceph-mon[92422]: pgmap v7731: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 211 KiB/s wr, 9 op/s 2023-03-09T18:43:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:43:45 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:43:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:43:45 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:43:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:43:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13536.158333 secs 2023-03-09T18:43:47.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:43:47 smithi122 ceph-mon[93869]: pgmap v7732: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 227 KiB/s wr, 12 op/s 2023-03-09T18:43:47.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:43:47 smithi181 ceph-mon[94189]: pgmap v7732: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 227 KiB/s wr, 12 op/s 2023-03-09T18:43:47.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:43:47 smithi038 ceph-mon[92422]: pgmap v7732: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 227 KiB/s wr, 12 op/s 2023-03-09T18:43:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:43:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13538.432655 secs 2023-03-09T18:43:48.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:43:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13538.432655 secs 2023-03-09T18:43:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:43:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13538.432655 secs 2023-03-09T18:43:48.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14000.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:43:49.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:43:49 smithi122 ceph-mon[93869]: pgmap v7733: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 183 KiB/s wr, 10 op/s 2023-03-09T18:43:49.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:43:49 smithi181 ceph-mon[94189]: pgmap v7733: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 183 KiB/s wr, 10 op/s 2023-03-09T18:43:49.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:43:49 smithi038 ceph-mon[92422]: pgmap v7733: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 183 KiB/s wr, 10 op/s 2023-03-09T18:43:50.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:43:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13541.158418 secs 2023-03-09T18:43:50.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:43:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13541.158418 secs 2023-03-09T18:43:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:43:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13541.158418 secs 2023-03-09T18:43:51.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:43:51 smithi122 ceph-mon[93869]: pgmap v7734: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 356 KiB/s wr, 11 op/s 2023-03-09T18:43:51.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:43:51 smithi181 ceph-mon[94189]: pgmap v7734: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 356 KiB/s wr, 11 op/s 2023-03-09T18:43:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:43:51 smithi038 ceph-mon[92422]: pgmap v7734: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 356 KiB/s wr, 11 op/s 2023-03-09T18:43:53.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:43:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13543.432779 secs 2023-03-09T18:43:53.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:43:53 smithi122 ceph-mon[93869]: pgmap v7735: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 326 KiB/s rd, 367 KiB/s wr, 14 op/s 2023-03-09T18:43:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:43:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13543.432779 secs 2023-03-09T18:43:53.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:43:53 smithi181 ceph-mon[94189]: pgmap v7735: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 326 KiB/s rd, 367 KiB/s wr, 14 op/s 2023-03-09T18:43:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:43:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13543.432779 secs 2023-03-09T18:43:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:43:53 smithi038 ceph-mon[92422]: pgmap v7735: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 326 KiB/s rd, 367 KiB/s wr, 14 op/s 2023-03-09T18:43:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14005.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:43:55.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:43:55 smithi122 ceph-mon[93869]: pgmap v7736: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 201 KiB/s wr, 9 op/s 2023-03-09T18:43:55.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:43:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13546.158477 secs 2023-03-09T18:43:55.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:43:55 smithi181 ceph-mon[94189]: pgmap v7736: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 201 KiB/s wr, 9 op/s 2023-03-09T18:43:55.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:43:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13546.158477 secs 2023-03-09T18:43:55.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:43:55 smithi038 ceph-mon[92422]: pgmap v7736: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 201 KiB/s wr, 9 op/s 2023-03-09T18:43:55.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:43:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13546.158477 secs 2023-03-09T18:43:57.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:43:57 smithi122 ceph-mon[93869]: pgmap v7737: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 267 KiB/s wr, 12 op/s 2023-03-09T18:43:57.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:43:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13548.432899 secs 2023-03-09T18:43:57.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:43:57 smithi181 ceph-mon[94189]: pgmap v7737: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 267 KiB/s wr, 12 op/s 2023-03-09T18:43:57.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:43:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13548.432899 secs 2023-03-09T18:43:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:43:57 smithi038 ceph-mon[92422]: pgmap v7737: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 267 KiB/s wr, 12 op/s 2023-03-09T18:43:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:43:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13548.432899 secs 2023-03-09T18:43:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14010.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:43:59.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:43:59 smithi122 ceph-mon[93869]: pgmap v7738: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 251 KiB/s wr, 9 op/s 2023-03-09T18:43:59.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:43:59 smithi181 ceph-mon[94189]: pgmap v7738: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 251 KiB/s wr, 9 op/s 2023-03-09T18:43:59.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:43:59 smithi038 ceph-mon[92422]: pgmap v7738: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 251 KiB/s wr, 9 op/s 2023-03-09T18:44:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:44:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13551.158537 secs 2023-03-09T18:44:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:44:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13551.158537 secs 2023-03-09T18:44:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:44:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13551.158537 secs 2023-03-09T18:44:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:44:01 smithi122 ceph-mon[93869]: pgmap v7739: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 348 KiB/s wr, 9 op/s 2023-03-09T18:44:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:44:01 smithi181 ceph-mon[94189]: pgmap v7739: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 348 KiB/s wr, 9 op/s 2023-03-09T18:44:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:44:01 smithi038 ceph-mon[92422]: pgmap v7739: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 348 KiB/s wr, 9 op/s 2023-03-09T18:44:03.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:44:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13553.433014 secs 2023-03-09T18:44:03.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:44:03 smithi122 ceph-mon[93869]: pgmap v7740: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 175 KiB/s wr, 11 op/s 2023-03-09T18:44:03.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:44:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13553.433014 secs 2023-03-09T18:44:03.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:44:03 smithi181 ceph-mon[94189]: pgmap v7740: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 175 KiB/s wr, 11 op/s 2023-03-09T18:44:03.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:44:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13553.433014 secs 2023-03-09T18:44:03.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:44:03 smithi038 ceph-mon[92422]: pgmap v7740: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 175 KiB/s wr, 11 op/s 2023-03-09T18:44:03.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14015.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:44:05.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:44:05 smithi122 ceph-mon[93869]: pgmap v7741: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 185 KiB/s wr, 20 op/s 2023-03-09T18:44:05.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:44:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13556.158595 secs 2023-03-09T18:44:05.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:44:05 smithi181 ceph-mon[94189]: pgmap v7741: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 185 KiB/s wr, 20 op/s 2023-03-09T18:44:05.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:44:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13556.158595 secs 2023-03-09T18:44:05.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:44:05 smithi038 ceph-mon[92422]: pgmap v7741: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 185 KiB/s wr, 20 op/s 2023-03-09T18:44:05.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:44:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13556.158595 secs 2023-03-09T18:44:07.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:44:07 smithi122 ceph-mon[93869]: pgmap v7742: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 254 KiB/s wr, 26 op/s 2023-03-09T18:44:07.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:44:07 smithi181 ceph-mon[94189]: pgmap v7742: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 254 KiB/s wr, 26 op/s 2023-03-09T18:44:07.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:44:07 smithi038 ceph-mon[92422]: pgmap v7742: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 254 KiB/s wr, 26 op/s 2023-03-09T18:44:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:44:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13558.433128 secs 2023-03-09T18:44:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:44:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13558.433128 secs 2023-03-09T18:44:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:44:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13558.433128 secs 2023-03-09T18:44:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14020.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:44:09.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:44:09 smithi122 ceph-mon[93869]: pgmap v7743: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 188 KiB/s wr, 22 op/s 2023-03-09T18:44:09.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:44:09 smithi181 ceph-mon[94189]: pgmap v7743: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 188 KiB/s wr, 22 op/s 2023-03-09T18:44:09.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:44:09 smithi038 ceph-mon[92422]: pgmap v7743: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 188 KiB/s wr, 22 op/s 2023-03-09T18:44:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:44:10 smithi122 ceph-mon[93869]: osdmap e697: 12 total, 12 up, 12 in 2023-03-09T18:44:10.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:44:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13561.158656 secs 2023-03-09T18:44:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:44:10 smithi181 ceph-mon[94189]: osdmap e697: 12 total, 12 up, 12 in 2023-03-09T18:44:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:44:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13561.158656 secs 2023-03-09T18:44:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:44:10 smithi038 ceph-mon[92422]: osdmap e697: 12 total, 12 up, 12 in 2023-03-09T18:44:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:44:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13561.158656 secs 2023-03-09T18:44:11.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:44:11 smithi122 ceph-mon[93869]: pgmap v7745: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 110 KiB/s wr, 26 op/s 2023-03-09T18:44:11.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:44:11 smithi181 ceph-mon[94189]: pgmap v7745: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 110 KiB/s wr, 26 op/s 2023-03-09T18:44:11.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:44:11 smithi038 ceph-mon[92422]: pgmap v7745: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 110 KiB/s wr, 26 op/s 2023-03-09T18:44:12.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:44:12 smithi122 ceph-mon[93869]: Health check failed: 1 MDSs behind on trimming (MDS_TRIM) 2023-03-09T18:44:12.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:44:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T18:44:12.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:44:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:44:12.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:44:12 smithi181 ceph-mon[94189]: Health check failed: 1 MDSs behind on trimming (MDS_TRIM) 2023-03-09T18:44:12.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:44:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T18:44:12.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:44:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:44:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:44:12 smithi038 ceph-mon[92422]: Health check failed: 1 MDSs behind on trimming (MDS_TRIM) 2023-03-09T18:44:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:44:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T18:44:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:44:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:44:13.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:44:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13563.433241 secs 2023-03-09T18:44:13.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:44:13 smithi122 ceph-mon[93869]: pgmap v7746: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 124 KiB/s wr, 27 op/s 2023-03-09T18:44:13.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:44:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T18:44:13.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:44:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:44:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:44:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13563.433241 secs 2023-03-09T18:44:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:44:13 smithi181 ceph-mon[94189]: pgmap v7746: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 124 KiB/s wr, 27 op/s 2023-03-09T18:44:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:44:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T18:44:13.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:44:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:44:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:44:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13563.433241 secs 2023-03-09T18:44:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:44:13 smithi038 ceph-mon[92422]: pgmap v7746: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 124 KiB/s wr, 27 op/s 2023-03-09T18:44:13.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:44:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T18:44:13.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:44:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:44:13.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14025.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:44:15.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:44:15 smithi122 ceph-mon[93869]: pgmap v7747: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 99 KiB/s wr, 12 op/s 2023-03-09T18:44:15.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:44:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13566.158748 secs 2023-03-09T18:44:15.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:44:15 smithi181 ceph-mon[94189]: pgmap v7747: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 99 KiB/s wr, 12 op/s 2023-03-09T18:44:15.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:44:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13566.158748 secs 2023-03-09T18:44:15.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:44:15 smithi038 ceph-mon[92422]: pgmap v7747: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 99 KiB/s wr, 12 op/s 2023-03-09T18:44:15.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:44:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13566.158748 secs 2023-03-09T18:44:17.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:44:17 smithi122 ceph-mon[93869]: pgmap v7748: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 58 KiB/s wr, 9 op/s 2023-03-09T18:44:17.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:44:17 smithi122 ceph-mon[93869]: Health check update: 2 MDSs behind on trimming (MDS_TRIM) 2023-03-09T18:44:17.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:44:17 smithi122 ceph-mon[93869]: osdmap e698: 12 total, 12 up, 12 in 2023-03-09T18:44:17.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:44:17 smithi181 ceph-mon[94189]: pgmap v7748: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 58 KiB/s wr, 9 op/s 2023-03-09T18:44:17.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:44:17 smithi181 ceph-mon[94189]: Health check update: 2 MDSs behind on trimming (MDS_TRIM) 2023-03-09T18:44:17.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:44:17 smithi181 ceph-mon[94189]: osdmap e698: 12 total, 12 up, 12 in 2023-03-09T18:44:17.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:44:17 smithi038 ceph-mon[92422]: pgmap v7748: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 58 KiB/s wr, 9 op/s 2023-03-09T18:44:17.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:44:17 smithi038 ceph-mon[92422]: Health check update: 2 MDSs behind on trimming (MDS_TRIM) 2023-03-09T18:44:17.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:44:17 smithi038 ceph-mon[92422]: osdmap e698: 12 total, 12 up, 12 in 2023-03-09T18:44:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:44:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13568.433362 secs 2023-03-09T18:44:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:44:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13568.433362 secs 2023-03-09T18:44:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:44:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13568.433362 secs 2023-03-09T18:44:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14030.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:44:19.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:44:19 smithi122 ceph-mon[93869]: pgmap v7750: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 65 KiB/s wr, 10 op/s 2023-03-09T18:44:19.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:44:19 smithi181 ceph-mon[94189]: pgmap v7750: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 65 KiB/s wr, 10 op/s 2023-03-09T18:44:19.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:44:19 smithi038 ceph-mon[92422]: pgmap v7750: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 65 KiB/s wr, 10 op/s 2023-03-09T18:44:20.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:44:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13571.158810 secs 2023-03-09T18:44:20.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:44:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13571.158810 secs 2023-03-09T18:44:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:44:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13571.158810 secs 2023-03-09T18:44:21.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:44:21 smithi122 ceph-mon[93869]: pgmap v7751: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 58 KiB/s wr, 9 op/s 2023-03-09T18:44:21.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:44:21 smithi181 ceph-mon[94189]: pgmap v7751: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 58 KiB/s wr, 9 op/s 2023-03-09T18:44:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:44:21 smithi038 ceph-mon[92422]: pgmap v7751: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 58 KiB/s wr, 9 op/s 2023-03-09T18:44:23.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:44:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13573.433480 secs 2023-03-09T18:44:23.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:44:23 smithi122 ceph-mon[93869]: pgmap v7752: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 43 KiB/s wr, 8 op/s 2023-03-09T18:44:23.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:44:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13573.433480 secs 2023-03-09T18:44:23.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:44:23 smithi181 ceph-mon[94189]: pgmap v7752: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 43 KiB/s wr, 8 op/s 2023-03-09T18:44:23.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:44:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13573.433480 secs 2023-03-09T18:44:23.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:44:23 smithi038 ceph-mon[92422]: pgmap v7752: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 43 KiB/s wr, 8 op/s 2023-03-09T18:44:23.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14035.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:44:25.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:44:25 smithi122 ceph-mon[93869]: pgmap v7753: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 43 KiB/s wr, 8 op/s 2023-03-09T18:44:25.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:44:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13576.158881 secs 2023-03-09T18:44:25.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:44:25 smithi181 ceph-mon[94189]: pgmap v7753: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 43 KiB/s wr, 8 op/s 2023-03-09T18:44:25.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:44:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13576.158881 secs 2023-03-09T18:44:25.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:44:25 smithi038 ceph-mon[92422]: pgmap v7753: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 43 KiB/s wr, 8 op/s 2023-03-09T18:44:25.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:44:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13576.158881 secs 2023-03-09T18:44:27.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:44:27 smithi122 ceph-mon[93869]: pgmap v7754: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 7 op/s 2023-03-09T18:44:27.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:44:27 smithi181 ceph-mon[94189]: pgmap v7754: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 7 op/s 2023-03-09T18:44:27.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:44:27 smithi038 ceph-mon[92422]: pgmap v7754: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 53 KiB/s wr, 7 op/s 2023-03-09T18:44:28.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:44:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13578.433596 secs 2023-03-09T18:44:28.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:44:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13578.433596 secs 2023-03-09T18:44:28.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:44:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13578.433596 secs 2023-03-09T18:44:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14040.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:44:29.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:44:29 smithi122 ceph-mon[93869]: pgmap v7755: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 48 KiB/s wr, 7 op/s 2023-03-09T18:44:29.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:44:29 smithi181 ceph-mon[94189]: pgmap v7755: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 48 KiB/s wr, 7 op/s 2023-03-09T18:44:29.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:44:29 smithi038 ceph-mon[92422]: pgmap v7755: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 48 KiB/s wr, 7 op/s 2023-03-09T18:44:30.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:44:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13581.158943 secs 2023-03-09T18:44:30.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:44:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13581.158943 secs 2023-03-09T18:44:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:44:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13581.158943 secs 2023-03-09T18:44:31.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:44:31 smithi122 ceph-mon[93869]: pgmap v7756: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 6 op/s 2023-03-09T18:44:31.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:44:31 smithi181 ceph-mon[94189]: pgmap v7756: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 6 op/s 2023-03-09T18:44:31.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:44:31 smithi038 ceph-mon[92422]: pgmap v7756: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 6 op/s 2023-03-09T18:44:33.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:44:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13583.433728 secs 2023-03-09T18:44:33.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:44:33 smithi122 ceph-mon[93869]: pgmap v7757: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 44 KiB/s wr, 9 op/s 2023-03-09T18:44:33.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:44:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13583.433728 secs 2023-03-09T18:44:33.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:44:33 smithi181 ceph-mon[94189]: pgmap v7757: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 44 KiB/s wr, 9 op/s 2023-03-09T18:44:33.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:44:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13583.433728 secs 2023-03-09T18:44:33.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:44:33 smithi038 ceph-mon[92422]: pgmap v7757: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 44 KiB/s wr, 9 op/s 2023-03-09T18:44:33.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14045.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:44:35.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:44:35 smithi122 ceph-mon[93869]: pgmap v7758: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 6 op/s 2023-03-09T18:44:35.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:44:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13586.159004 secs 2023-03-09T18:44:35.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:44:35 smithi181 ceph-mon[94189]: pgmap v7758: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 6 op/s 2023-03-09T18:44:35.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:44:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13586.159004 secs 2023-03-09T18:44:35.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:44:35 smithi038 ceph-mon[92422]: pgmap v7758: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 6 op/s 2023-03-09T18:44:35.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:44:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13586.159004 secs 2023-03-09T18:44:37.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:44:37 smithi122 ceph-mon[93869]: pgmap v7759: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 98 KiB/s wr, 9 op/s 2023-03-09T18:44:37.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:44:37 smithi181 ceph-mon[94189]: pgmap v7759: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 98 KiB/s wr, 9 op/s 2023-03-09T18:44:37.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:44:37 smithi038 ceph-mon[92422]: pgmap v7759: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 98 KiB/s wr, 9 op/s 2023-03-09T18:44:38.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:44:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13588.433867 secs 2023-03-09T18:44:38.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:44:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13588.433867 secs 2023-03-09T18:44:38.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:44:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13588.433867 secs 2023-03-09T18:44:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14050.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:44:39.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:44:39 smithi122 ceph-mon[93869]: pgmap v7760: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 55 KiB/s wr, 6 op/s 2023-03-09T18:44:39.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:44:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:44:39.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:44:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:44:39.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:44:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:44:39.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:44:39 smithi181 ceph-mon[94189]: pgmap v7760: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 55 KiB/s wr, 6 op/s 2023-03-09T18:44:39.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:44:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:44:39.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:44:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:44:39.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:44:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:44:39.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:44:39 smithi038 ceph-mon[92422]: pgmap v7760: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 55 KiB/s wr, 6 op/s 2023-03-09T18:44:39.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:44:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:44:39.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:44:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:44:39.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:44:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:44:40.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:44:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:44:40.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:44:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:44:40.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:44:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:44:40.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:44:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13591.159065 secs 2023-03-09T18:44:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:44:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:44:40.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:44:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:44:40.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:44:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:44:40.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:44:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13591.159065 secs 2023-03-09T18:44:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:44:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:44:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:44:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:44:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:44:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:44:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:44:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13591.159065 secs 2023-03-09T18:44:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:44:41 smithi122 ceph-mon[93869]: pgmap v7761: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 55 KiB/s wr, 6 op/s 2023-03-09T18:44:41.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:44:41 smithi181 ceph-mon[94189]: pgmap v7761: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 55 KiB/s wr, 6 op/s 2023-03-09T18:44:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:44:41 smithi038 ceph-mon[92422]: pgmap v7761: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 55 KiB/s wr, 6 op/s 2023-03-09T18:44:43.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:44:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13593.433983 secs 2023-03-09T18:44:43.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:44:43 smithi122 ceph-mon[93869]: pgmap v7762: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 145 KiB/s wr, 9 op/s 2023-03-09T18:44:43.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:44:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13593.433983 secs 2023-03-09T18:44:43.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:44:43 smithi181 ceph-mon[94189]: pgmap v7762: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 145 KiB/s wr, 9 op/s 2023-03-09T18:44:43.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:44:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13593.433983 secs 2023-03-09T18:44:43.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:44:43 smithi038 ceph-mon[92422]: pgmap v7762: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 145 KiB/s wr, 9 op/s 2023-03-09T18:44:43.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14055.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:44:45.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:44:45 smithi122 ceph-mon[93869]: pgmap v7763: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 145 KiB/s wr, 7 op/s 2023-03-09T18:44:45.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:44:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13596.159123 secs 2023-03-09T18:44:45.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:44:45 smithi181 ceph-mon[94189]: pgmap v7763: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 145 KiB/s wr, 7 op/s 2023-03-09T18:44:45.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:44:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13596.159123 secs 2023-03-09T18:44:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:44:45 smithi038 ceph-mon[92422]: pgmap v7763: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 145 KiB/s wr, 7 op/s 2023-03-09T18:44:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:44:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13596.159123 secs 2023-03-09T18:44:47.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:44:47 smithi122 ceph-mon[93869]: pgmap v7764: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 172 KiB/s wr, 11 op/s 2023-03-09T18:44:47.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:44:47 smithi181 ceph-mon[94189]: pgmap v7764: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 172 KiB/s wr, 11 op/s 2023-03-09T18:44:47.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:44:47 smithi038 ceph-mon[92422]: pgmap v7764: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 172 KiB/s wr, 11 op/s 2023-03-09T18:44:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:44:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13598.434073 secs 2023-03-09T18:44:48.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:44:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13598.434073 secs 2023-03-09T18:44:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:44:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13598.434073 secs 2023-03-09T18:44:48.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14060.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:44:49.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:44:49 smithi122 ceph-mon[93869]: pgmap v7765: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 119 KiB/s wr, 8 op/s 2023-03-09T18:44:49.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:44:49 smithi181 ceph-mon[94189]: pgmap v7765: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 119 KiB/s wr, 8 op/s 2023-03-09T18:44:49.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:44:49 smithi038 ceph-mon[92422]: pgmap v7765: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 119 KiB/s wr, 8 op/s 2023-03-09T18:44:50.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:44:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13601.159184 secs 2023-03-09T18:44:50.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:44:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13601.159184 secs 2023-03-09T18:44:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:44:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13601.159184 secs 2023-03-09T18:44:51.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:44:51 smithi122 ceph-mon[93869]: pgmap v7766: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 118 KiB/s wr, 8 op/s 2023-03-09T18:44:51.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:44:51 smithi181 ceph-mon[94189]: pgmap v7766: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 118 KiB/s wr, 8 op/s 2023-03-09T18:44:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:44:51 smithi038 ceph-mon[92422]: pgmap v7766: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 118 KiB/s wr, 8 op/s 2023-03-09T18:44:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:44:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13603.434181 secs 2023-03-09T18:44:53.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:44:53 smithi122 ceph-mon[93869]: pgmap v7767: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 283 KiB/s wr, 13 op/s 2023-03-09T18:44:53.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:44:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T18:44:53.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:44:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:44:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:44:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13603.434181 secs 2023-03-09T18:44:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:44:53 smithi181 ceph-mon[94189]: pgmap v7767: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 283 KiB/s wr, 13 op/s 2023-03-09T18:44:53.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:44:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T18:44:53.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:44:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:44:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:44:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13603.434181 secs 2023-03-09T18:44:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:44:53 smithi038 ceph-mon[92422]: pgmap v7767: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 283 KiB/s wr, 13 op/s 2023-03-09T18:44:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:44:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T18:44:53.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:44:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:44:53.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14065.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:44:55.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:44:55 smithi122 ceph-mon[93869]: pgmap v7768: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 192 KiB/s wr, 10 op/s 2023-03-09T18:44:55.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:44:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13606.159245 secs 2023-03-09T18:44:55.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:44:55 smithi181 ceph-mon[94189]: pgmap v7768: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 192 KiB/s wr, 10 op/s 2023-03-09T18:44:55.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:44:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13606.159245 secs 2023-03-09T18:44:55.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:44:55 smithi038 ceph-mon[92422]: pgmap v7768: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 192 KiB/s wr, 10 op/s 2023-03-09T18:44:55.503 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:44:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13606.159245 secs 2023-03-09T18:44:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:44:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T18:44:56.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:44:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:44:56.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:44:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T18:44:56.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:44:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:44:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:44:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T18:44:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:44:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:44:57.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:44:57 smithi122 ceph-mon[93869]: pgmap v7769: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 208 KiB/s wr, 13 op/s 2023-03-09T18:44:57.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:44:57 smithi181 ceph-mon[94189]: pgmap v7769: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 208 KiB/s wr, 13 op/s 2023-03-09T18:44:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:44:57 smithi038 ceph-mon[92422]: pgmap v7769: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 208 KiB/s wr, 13 op/s 2023-03-09T18:44:58.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14070.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:44:59.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:44:59 smithi122 ceph-mon[93869]: pgmap v7770: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 181 KiB/s wr, 10 op/s 2023-03-09T18:44:59.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:44:59 smithi181 ceph-mon[94189]: pgmap v7770: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 181 KiB/s wr, 10 op/s 2023-03-09T18:44:59.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:44:59 smithi038 ceph-mon[92422]: pgmap v7770: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 181 KiB/s wr, 10 op/s 2023-03-09T18:45:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13611.159308 secs 2023-03-09T18:45:00.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:00 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:45:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13611.159308 secs 2023-03-09T18:45:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:00 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:45:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13611.159308 secs 2023-03-09T18:45:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:00 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:45:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:01 smithi122 ceph-mon[93869]: pgmap v7771: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 181 KiB/s wr, 9 op/s 2023-03-09T18:45:01.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:01 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:45:01.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:45:01.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:45:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:01 smithi181 ceph-mon[94189]: pgmap v7771: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 181 KiB/s wr, 9 op/s 2023-03-09T18:45:01.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:01 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:45:01.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:45:01.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:45:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:01 smithi038 ceph-mon[92422]: pgmap v7771: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 181 KiB/s wr, 9 op/s 2023-03-09T18:45:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:01 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:45:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:45:01.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:45:03.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13613.434439 secs 2023-03-09T18:45:03.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:03 smithi122 ceph-mon[93869]: pgmap v7772: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 444 KiB/s rd, 441 KiB/s wr, 14 op/s 2023-03-09T18:45:03.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13613.434439 secs 2023-03-09T18:45:03.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:03 smithi181 ceph-mon[94189]: pgmap v7772: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 444 KiB/s rd, 441 KiB/s wr, 14 op/s 2023-03-09T18:45:03.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13613.434439 secs 2023-03-09T18:45:03.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:03 smithi038 ceph-mon[92422]: pgmap v7772: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 444 KiB/s rd, 441 KiB/s wr, 14 op/s 2023-03-09T18:45:03.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14075.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:45:05.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:05 smithi122 ceph-mon[93869]: pgmap v7773: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 298 KiB/s wr, 22 op/s 2023-03-09T18:45:05.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:05 smithi122 ceph-mon[93869]: osdmap e699: 12 total, 12 up, 12 in 2023-03-09T18:45:05.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:05 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:45:05.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:05 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:45:05.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13616.159385 secs 2023-03-09T18:45:05.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:05 smithi181 ceph-mon[94189]: pgmap v7773: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 298 KiB/s wr, 22 op/s 2023-03-09T18:45:05.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:05 smithi181 ceph-mon[94189]: osdmap e699: 12 total, 12 up, 12 in 2023-03-09T18:45:05.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:05 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:45:05.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:05 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:45:05.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13616.159385 secs 2023-03-09T18:45:05.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:05 smithi038 ceph-mon[92422]: pgmap v7773: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 298 KiB/s wr, 22 op/s 2023-03-09T18:45:05.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:05 smithi038 ceph-mon[92422]: osdmap e699: 12 total, 12 up, 12 in 2023-03-09T18:45:05.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:05 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:45:05.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:05 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:45:05.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13616.159385 secs 2023-03-09T18:45:07.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:07 smithi122 ceph-mon[93869]: pgmap v7775: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 460 KiB/s wr, 30 op/s 2023-03-09T18:45:07.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:07 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:45:07.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:07 smithi181 ceph-mon[94189]: pgmap v7775: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 460 KiB/s wr, 30 op/s 2023-03-09T18:45:07.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:07 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:45:07.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:07 smithi038 ceph-mon[92422]: pgmap v7775: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 460 KiB/s wr, 30 op/s 2023-03-09T18:45:07.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:07 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:45:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13618.434552 secs 2023-03-09T18:45:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13618.434552 secs 2023-03-09T18:45:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13618.434552 secs 2023-03-09T18:45:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14080.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:45:09.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:09 smithi122 ceph-mon[93869]: pgmap v7776: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 460 KiB/s wr, 30 op/s 2023-03-09T18:45:09.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:09 smithi181 ceph-mon[94189]: pgmap v7776: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 460 KiB/s wr, 30 op/s 2023-03-09T18:45:09.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:09 smithi038 ceph-mon[92422]: pgmap v7776: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 460 KiB/s wr, 30 op/s 2023-03-09T18:45:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13621.159464 secs 2023-03-09T18:45:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13621.159464 secs 2023-03-09T18:45:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13621.159464 secs 2023-03-09T18:45:11.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:11 smithi122 ceph-mon[93869]: pgmap v7777: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 460 KiB/s wr, 30 op/s 2023-03-09T18:45:11.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:11 smithi181 ceph-mon[94189]: pgmap v7777: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 460 KiB/s wr, 30 op/s 2023-03-09T18:45:11.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:11 smithi038 ceph-mon[92422]: pgmap v7777: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 460 KiB/s wr, 30 op/s 2023-03-09T18:45:12.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T18:45:12.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:45:12.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:12 smithi122 ceph-mon[93869]: osdmap e700: 12 total, 12 up, 12 in 2023-03-09T18:45:12.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T18:45:12.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:45:12.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:12 smithi181 ceph-mon[94189]: osdmap e700: 12 total, 12 up, 12 in 2023-03-09T18:45:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T18:45:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:45:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:12 smithi038 ceph-mon[92422]: osdmap e700: 12 total, 12 up, 12 in 2023-03-09T18:45:13.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13623.434678 secs 2023-03-09T18:45:13.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:13 smithi122 ceph-mon[93869]: pgmap v7778: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 274 KiB/s wr, 28 op/s 2023-03-09T18:45:13.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T18:45:13.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:45:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13623.434678 secs 2023-03-09T18:45:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:13 smithi181 ceph-mon[94189]: pgmap v7778: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 274 KiB/s wr, 28 op/s 2023-03-09T18:45:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T18:45:13.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:45:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13623.434678 secs 2023-03-09T18:45:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:13 smithi038 ceph-mon[92422]: pgmap v7778: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 274 KiB/s wr, 28 op/s 2023-03-09T18:45:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T18:45:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:45:13.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14085.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:45:15.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:15 smithi122 ceph-mon[93869]: pgmap v7780: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 249 KiB/s wr, 14 op/s 2023-03-09T18:45:15.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13626.159522 secs 2023-03-09T18:45:15.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:15 smithi181 ceph-mon[94189]: pgmap v7780: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 249 KiB/s wr, 14 op/s 2023-03-09T18:45:15.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13626.159522 secs 2023-03-09T18:45:15.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:15 smithi038 ceph-mon[92422]: pgmap v7780: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 249 KiB/s wr, 14 op/s 2023-03-09T18:45:15.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13626.159522 secs 2023-03-09T18:45:17.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:17 smithi122 ceph-mon[93869]: pgmap v7781: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 163 KiB/s wr, 8 op/s 2023-03-09T18:45:17.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13628.434795 secs 2023-03-09T18:45:17.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:17 smithi181 ceph-mon[94189]: pgmap v7781: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 163 KiB/s wr, 8 op/s 2023-03-09T18:45:17.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13628.434795 secs 2023-03-09T18:45:17.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:17 smithi038 ceph-mon[92422]: pgmap v7781: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 163 KiB/s wr, 8 op/s 2023-03-09T18:45:17.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13628.434795 secs 2023-03-09T18:45:18.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14090.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:45:19.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:19 smithi122 ceph-mon[93869]: pgmap v7782: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 162 KiB/s wr, 8 op/s 2023-03-09T18:45:19.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:19 smithi181 ceph-mon[94189]: pgmap v7782: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 162 KiB/s wr, 8 op/s 2023-03-09T18:45:19.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:19 smithi038 ceph-mon[92422]: pgmap v7782: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 162 KiB/s wr, 8 op/s 2023-03-09T18:45:20.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13631.159582 secs 2023-03-09T18:45:20.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13631.159582 secs 2023-03-09T18:45:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13631.159582 secs 2023-03-09T18:45:21.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:21 smithi122 ceph-mon[93869]: pgmap v7783: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 162 KiB/s wr, 8 op/s 2023-03-09T18:45:21.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:21 smithi181 ceph-mon[94189]: pgmap v7783: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 162 KiB/s wr, 8 op/s 2023-03-09T18:45:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:21 smithi038 ceph-mon[92422]: pgmap v7783: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 162 KiB/s wr, 8 op/s 2023-03-09T18:45:23.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13633.434947 secs 2023-03-09T18:45:23.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:23 smithi122 ceph-mon[93869]: pgmap v7784: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 37 KiB/s wr, 6 op/s 2023-03-09T18:45:23.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13633.434947 secs 2023-03-09T18:45:23.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:23 smithi181 ceph-mon[94189]: pgmap v7784: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 37 KiB/s wr, 6 op/s 2023-03-09T18:45:23.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13633.434947 secs 2023-03-09T18:45:23.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:23 smithi038 ceph-mon[92422]: pgmap v7784: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 37 KiB/s wr, 6 op/s 2023-03-09T18:45:23.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14095.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:45:25.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:25 smithi122 ceph-mon[93869]: pgmap v7785: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 31 KiB/s wr, 5 op/s 2023-03-09T18:45:25.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13636.159667 secs 2023-03-09T18:45:25.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:25 smithi181 ceph-mon[94189]: pgmap v7785: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 31 KiB/s wr, 5 op/s 2023-03-09T18:45:25.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13636.159667 secs 2023-03-09T18:45:25.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:25 smithi038 ceph-mon[92422]: pgmap v7785: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 31 KiB/s wr, 5 op/s 2023-03-09T18:45:25.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13636.159667 secs 2023-03-09T18:45:27.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:27 smithi122 ceph-mon[93869]: pgmap v7786: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 60 KiB/s wr, 8 op/s 2023-03-09T18:45:27.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:27 smithi181 ceph-mon[94189]: pgmap v7786: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 60 KiB/s wr, 8 op/s 2023-03-09T18:45:27.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:27 smithi038 ceph-mon[92422]: pgmap v7786: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 60 KiB/s wr, 8 op/s 2023-03-09T18:45:28.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13638.435063 secs 2023-03-09T18:45:28.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13638.435063 secs 2023-03-09T18:45:28.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13638.435063 secs 2023-03-09T18:45:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14100.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:45:29.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:29 smithi122 ceph-mon[93869]: pgmap v7787: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 6 op/s 2023-03-09T18:45:29.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:29 smithi181 ceph-mon[94189]: pgmap v7787: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 6 op/s 2023-03-09T18:45:29.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:29 smithi038 ceph-mon[92422]: pgmap v7787: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 6 op/s 2023-03-09T18:45:30.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13641.159728 secs 2023-03-09T18:45:30.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13641.159728 secs 2023-03-09T18:45:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13641.159728 secs 2023-03-09T18:45:31.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:31 smithi122 ceph-mon[93869]: pgmap v7788: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 5 op/s 2023-03-09T18:45:31.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:31 smithi181 ceph-mon[94189]: pgmap v7788: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 5 op/s 2023-03-09T18:45:31.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:31 smithi038 ceph-mon[92422]: pgmap v7788: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 5 op/s 2023-03-09T18:45:33.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13643.435201 secs 2023-03-09T18:45:33.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:33 smithi122 ceph-mon[93869]: pgmap v7789: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 75 KiB/s wr, 9 op/s 2023-03-09T18:45:33.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13643.435201 secs 2023-03-09T18:45:33.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:33 smithi181 ceph-mon[94189]: pgmap v7789: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 75 KiB/s wr, 9 op/s 2023-03-09T18:45:33.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13643.435201 secs 2023-03-09T18:45:33.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:33 smithi038 ceph-mon[92422]: pgmap v7789: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 75 KiB/s wr, 9 op/s 2023-03-09T18:45:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14105.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:45:35.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:35 smithi122 ceph-mon[93869]: pgmap v7790: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 74 KiB/s wr, 7 op/s 2023-03-09T18:45:35.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13646.159788 secs 2023-03-09T18:45:35.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:35 smithi181 ceph-mon[94189]: pgmap v7790: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 74 KiB/s wr, 7 op/s 2023-03-09T18:45:35.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13646.159788 secs 2023-03-09T18:45:35.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:35 smithi038 ceph-mon[92422]: pgmap v7790: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 74 KiB/s wr, 7 op/s 2023-03-09T18:45:35.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13646.159788 secs 2023-03-09T18:45:37.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:37 smithi122 ceph-mon[93869]: pgmap v7791: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 193 KiB/s wr, 10 op/s 2023-03-09T18:45:37.485 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:37 smithi181 ceph-mon[94189]: pgmap v7791: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 193 KiB/s wr, 10 op/s 2023-03-09T18:45:37.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:37 smithi038 ceph-mon[92422]: pgmap v7791: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 193 KiB/s wr, 10 op/s 2023-03-09T18:45:38.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14110.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:45:39.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:39 smithi122 ceph-mon[93869]: pgmap v7792: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 163 KiB/s wr, 8 op/s 2023-03-09T18:45:39.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:39 smithi181 ceph-mon[94189]: pgmap v7792: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 163 KiB/s wr, 8 op/s 2023-03-09T18:45:39.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:39 smithi038 ceph-mon[92422]: pgmap v7792: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 163 KiB/s wr, 8 op/s 2023-03-09T18:45:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T18:45:40.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:45:40.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:45:40.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13651.159855 secs 2023-03-09T18:45:40.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:40 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:45:40.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:45:40.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:45:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T18:45:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:45:40.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:45:40.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13651.159855 secs 2023-03-09T18:45:40.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:40 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:45:40.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:45:40.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:45:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T18:45:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:45:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:45:40.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13651.159855 secs 2023-03-09T18:45:40.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:40 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:45:40.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:45:40.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:45:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:41 smithi122 ceph-mon[93869]: pgmap v7793: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 163 KiB/s wr, 8 op/s 2023-03-09T18:45:41.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:41 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:45:41.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T18:45:41.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:45:41.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:45:41.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:45:41.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:45:41.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:45:41.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:45:41.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:45:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:41 smithi181 ceph-mon[94189]: pgmap v7793: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 163 KiB/s wr, 8 op/s 2023-03-09T18:45:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:41 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:45:41.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T18:45:41.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:45:41.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:45:41.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:45:41.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:45:41.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:45:41.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:45:41.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:45:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:41 smithi038 ceph-mon[92422]: pgmap v7793: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 163 KiB/s wr, 8 op/s 2023-03-09T18:45:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:41 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:45:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T18:45:41.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:45:41.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:45:41.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:45:41.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:45:41.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:45:41.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:45:41.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:45:43.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:43 smithi122 ceph-mon[93869]: pgmap v7794: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 275 KiB/s wr, 12 op/s 2023-03-09T18:45:43.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:43 smithi181 ceph-mon[94189]: pgmap v7794: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 275 KiB/s wr, 12 op/s 2023-03-09T18:45:43.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:43 smithi038 ceph-mon[92422]: pgmap v7794: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 275 KiB/s wr, 12 op/s 2023-03-09T18:45:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14115.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:45:45.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:45 smithi122 ceph-mon[93869]: pgmap v7795: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 230 KiB/s wr, 9 op/s 2023-03-09T18:45:45.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13656.159914 secs 2023-03-09T18:45:45.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:45 smithi181 ceph-mon[94189]: pgmap v7795: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 230 KiB/s wr, 9 op/s 2023-03-09T18:45:45.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13656.159914 secs 2023-03-09T18:45:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:45 smithi038 ceph-mon[92422]: pgmap v7795: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 230 KiB/s wr, 9 op/s 2023-03-09T18:45:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13656.159914 secs 2023-03-09T18:45:47.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:47 smithi122 ceph-mon[93869]: pgmap v7796: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 336 KiB/s wr, 13 op/s 2023-03-09T18:45:47.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:47 smithi181 ceph-mon[94189]: pgmap v7796: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 336 KiB/s wr, 13 op/s 2023-03-09T18:45:47.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:47 smithi038 ceph-mon[92422]: pgmap v7796: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 336 KiB/s wr, 13 op/s 2023-03-09T18:45:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14120.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:45:49.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:49 smithi122 ceph-mon[93869]: pgmap v7797: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 218 KiB/s wr, 10 op/s 2023-03-09T18:45:49.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:49 smithi181 ceph-mon[94189]: pgmap v7797: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 218 KiB/s wr, 10 op/s 2023-03-09T18:45:49.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:49 smithi038 ceph-mon[92422]: pgmap v7797: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 218 KiB/s wr, 10 op/s 2023-03-09T18:45:50.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13661.159996 secs 2023-03-09T18:45:50.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13661.159996 secs 2023-03-09T18:45:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13661.159996 secs 2023-03-09T18:45:51.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:51 smithi122 ceph-mon[93869]: pgmap v7798: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 218 KiB/s wr, 9 op/s 2023-03-09T18:45:51.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:51 smithi181 ceph-mon[94189]: pgmap v7798: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 218 KiB/s wr, 9 op/s 2023-03-09T18:45:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:51 smithi038 ceph-mon[92422]: pgmap v7798: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 218 KiB/s wr, 9 op/s 2023-03-09T18:45:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13663.435658 secs 2023-03-09T18:45:53.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:53 smithi122 ceph-mon[93869]: pgmap v7799: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 319 KiB/s wr, 13 op/s 2023-03-09T18:45:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13663.435658 secs 2023-03-09T18:45:53.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:53 smithi181 ceph-mon[94189]: pgmap v7799: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 319 KiB/s wr, 13 op/s 2023-03-09T18:45:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13663.435658 secs 2023-03-09T18:45:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:53 smithi038 ceph-mon[92422]: pgmap v7799: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 319 KiB/s wr, 13 op/s 2023-03-09T18:45:53.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14125.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:45:55.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:55 smithi122 ceph-mon[93869]: pgmap v7800: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 208 KiB/s wr, 9 op/s 2023-03-09T18:45:55.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13666.160055 secs 2023-03-09T18:45:55.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:55 smithi181 ceph-mon[94189]: pgmap v7800: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 208 KiB/s wr, 9 op/s 2023-03-09T18:45:55.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13666.160055 secs 2023-03-09T18:45:55.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:55 smithi038 ceph-mon[92422]: pgmap v7800: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 208 KiB/s wr, 9 op/s 2023-03-09T18:45:55.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13666.160055 secs 2023-03-09T18:45:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:56 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:45:56.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:45:56.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:45:56.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:56 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:45:56.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:45:56.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:45:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:56 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:45:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:45:56.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:45:57.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:57 smithi122 ceph-mon[93869]: pgmap v7801: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 276 KiB/s wr, 12 op/s 2023-03-09T18:45:57.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13668.435783 secs 2023-03-09T18:45:57.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:57 smithi181 ceph-mon[94189]: pgmap v7801: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 276 KiB/s wr, 12 op/s 2023-03-09T18:45:57.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13668.435783 secs 2023-03-09T18:45:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:57 smithi038 ceph-mon[92422]: pgmap v7801: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 276 KiB/s wr, 12 op/s 2023-03-09T18:45:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13668.435783 secs 2023-03-09T18:45:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14130.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:45:59.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:45:59 smithi122 ceph-mon[93869]: pgmap v7802: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 170 KiB/s wr, 9 op/s 2023-03-09T18:45:59.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:45:59 smithi181 ceph-mon[94189]: pgmap v7802: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 170 KiB/s wr, 9 op/s 2023-03-09T18:45:59.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:45:59 smithi038 ceph-mon[92422]: pgmap v7802: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 170 KiB/s wr, 9 op/s 2023-03-09T18:46:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:46:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T18:46:00.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:46:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:46:00.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:46:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13671.160115 secs 2023-03-09T18:46:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:46:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T18:46:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:46:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:46:00.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:46:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13671.160115 secs 2023-03-09T18:46:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:46:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T18:46:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:46:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:46:00.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:46:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13671.160115 secs 2023-03-09T18:46:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:46:01 smithi122 ceph-mon[93869]: pgmap v7803: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 174 KiB/s wr, 8 op/s 2023-03-09T18:46:01.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:46:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T18:46:01.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:46:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:46:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:46:01 smithi181 ceph-mon[94189]: pgmap v7803: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 174 KiB/s wr, 8 op/s 2023-03-09T18:46:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:46:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T18:46:01.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:46:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:46:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:46:01 smithi038 ceph-mon[92422]: pgmap v7803: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 174 KiB/s wr, 8 op/s 2023-03-09T18:46:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:46:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T18:46:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:46:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:46:03.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:46:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13673.435908 secs 2023-03-09T18:46:03.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:46:03 smithi122 ceph-mon[93869]: pgmap v7804: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 263 KiB/s wr, 12 op/s 2023-03-09T18:46:03.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:46:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13673.435908 secs 2023-03-09T18:46:03.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:46:03 smithi181 ceph-mon[94189]: pgmap v7804: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 263 KiB/s wr, 12 op/s 2023-03-09T18:46:03.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:46:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13673.435908 secs 2023-03-09T18:46:03.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:46:03 smithi038 ceph-mon[92422]: pgmap v7804: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 263 KiB/s wr, 12 op/s 2023-03-09T18:46:03.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14135.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:46:05.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:46:05 smithi122 ceph-mon[93869]: pgmap v7805: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 183 KiB/s wr, 20 op/s 2023-03-09T18:46:05.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:46:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13676.160173 secs 2023-03-09T18:46:05.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:46:05 smithi181 ceph-mon[94189]: pgmap v7805: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 183 KiB/s wr, 20 op/s 2023-03-09T18:46:05.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:46:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13676.160173 secs 2023-03-09T18:46:05.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:46:05 smithi038 ceph-mon[92422]: pgmap v7805: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 183 KiB/s wr, 20 op/s 2023-03-09T18:46:05.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:46:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13676.160173 secs 2023-03-09T18:46:07.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:46:07 smithi122 ceph-mon[93869]: pgmap v7806: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 241 KiB/s wr, 25 op/s 2023-03-09T18:46:07.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:46:07 smithi181 ceph-mon[94189]: pgmap v7806: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 241 KiB/s wr, 25 op/s 2023-03-09T18:46:07.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:46:07 smithi038 ceph-mon[92422]: pgmap v7806: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 241 KiB/s wr, 25 op/s 2023-03-09T18:46:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:46:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13678.435979 secs 2023-03-09T18:46:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:46:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13678.435979 secs 2023-03-09T18:46:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:46:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13678.435979 secs 2023-03-09T18:46:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14140.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:46:09.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:46:09 smithi122 ceph-mon[93869]: pgmap v7807: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 173 KiB/s wr, 22 op/s 2023-03-09T18:46:09.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:46:09 smithi181 ceph-mon[94189]: pgmap v7807: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 173 KiB/s wr, 22 op/s 2023-03-09T18:46:09.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:46:09 smithi038 ceph-mon[92422]: pgmap v7807: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 173 KiB/s wr, 22 op/s 2023-03-09T18:46:10.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:46:10 smithi122 ceph-mon[93869]: osdmap e701: 12 total, 12 up, 12 in 2023-03-09T18:46:10.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:46:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13681.160232 secs 2023-03-09T18:46:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:46:10 smithi181 ceph-mon[94189]: osdmap e701: 12 total, 12 up, 12 in 2023-03-09T18:46:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:46:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13681.160232 secs 2023-03-09T18:46:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:46:10 smithi038 ceph-mon[92422]: osdmap e701: 12 total, 12 up, 12 in 2023-03-09T18:46:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:46:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13681.160232 secs 2023-03-09T18:46:11.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:46:11 smithi122 ceph-mon[93869]: pgmap v7809: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 215 KiB/s wr, 26 op/s 2023-03-09T18:46:11.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:46:11 smithi181 ceph-mon[94189]: pgmap v7809: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 215 KiB/s wr, 26 op/s 2023-03-09T18:46:11.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:46:11 smithi038 ceph-mon[92422]: pgmap v7809: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 215 KiB/s wr, 26 op/s 2023-03-09T18:46:13.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:46:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13683.436100 secs 2023-03-09T18:46:13.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:46:13 smithi122 ceph-mon[93869]: pgmap v7810: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 110 KiB/s wr, 26 op/s 2023-03-09T18:46:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:46:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13683.436100 secs 2023-03-09T18:46:13.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:46:13 smithi181 ceph-mon[94189]: pgmap v7810: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 110 KiB/s wr, 26 op/s 2023-03-09T18:46:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:46:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13683.436100 secs 2023-03-09T18:46:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:46:13 smithi038 ceph-mon[92422]: pgmap v7810: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 110 KiB/s wr, 26 op/s 2023-03-09T18:46:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14145.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:46:15.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:46:15 smithi122 ceph-mon[93869]: pgmap v7811: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 98 KiB/s wr, 12 op/s 2023-03-09T18:46:15.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:46:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13686.160295 secs 2023-03-09T18:46:15.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:46:15 smithi181 ceph-mon[94189]: pgmap v7811: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 98 KiB/s wr, 12 op/s 2023-03-09T18:46:15.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:46:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13686.160295 secs 2023-03-09T18:46:15.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:46:15 smithi038 ceph-mon[92422]: pgmap v7811: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 98 KiB/s wr, 12 op/s 2023-03-09T18:46:15.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:46:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13686.160295 secs 2023-03-09T18:46:17.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:46:17 smithi122 ceph-mon[93869]: pgmap v7812: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 28 KiB/s wr, 8 op/s 2023-03-09T18:46:17.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:46:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13688.436215 secs 2023-03-09T18:46:17.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:46:17 smithi122 ceph-mon[93869]: osdmap e702: 12 total, 12 up, 12 in 2023-03-09T18:46:17.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:46:17 smithi181 ceph-mon[94189]: pgmap v7812: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 28 KiB/s wr, 8 op/s 2023-03-09T18:46:17.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:46:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13688.436215 secs 2023-03-09T18:46:17.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:46:17 smithi181 ceph-mon[94189]: osdmap e702: 12 total, 12 up, 12 in 2023-03-09T18:46:17.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:46:17 smithi038 ceph-mon[92422]: pgmap v7812: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 28 KiB/s wr, 8 op/s 2023-03-09T18:46:17.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:46:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13688.436215 secs 2023-03-09T18:46:17.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:46:17 smithi038 ceph-mon[92422]: osdmap e702: 12 total, 12 up, 12 in 2023-03-09T18:46:18.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14150.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:46:19.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:46:19 smithi122 ceph-mon[93869]: pgmap v7814: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 17 KiB/s wr, 9 op/s 2023-03-09T18:46:19.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:46:19 smithi181 ceph-mon[94189]: pgmap v7814: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 17 KiB/s wr, 9 op/s 2023-03-09T18:46:19.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:46:19 smithi038 ceph-mon[92422]: pgmap v7814: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 17 KiB/s wr, 9 op/s 2023-03-09T18:46:20.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:46:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13691.160357 secs 2023-03-09T18:46:20.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:46:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13691.160357 secs 2023-03-09T18:46:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:46:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13691.160357 secs 2023-03-09T18:46:21.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:46:21 smithi122 ceph-mon[93869]: pgmap v7815: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 KiB/s wr, 9 op/s 2023-03-09T18:46:21.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:46:21 smithi181 ceph-mon[94189]: pgmap v7815: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 KiB/s wr, 9 op/s 2023-03-09T18:46:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:46:21 smithi038 ceph-mon[92422]: pgmap v7815: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 15 KiB/s wr, 9 op/s 2023-03-09T18:46:23.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:46:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13693.436300 secs 2023-03-09T18:46:23.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:46:23 smithi122 ceph-mon[93869]: pgmap v7816: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 67 KiB/s wr, 8 op/s 2023-03-09T18:46:23.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:46:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13693.436300 secs 2023-03-09T18:46:23.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:46:23 smithi181 ceph-mon[94189]: pgmap v7816: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 67 KiB/s wr, 8 op/s 2023-03-09T18:46:23.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:46:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13693.436300 secs 2023-03-09T18:46:23.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:46:23 smithi038 ceph-mon[92422]: pgmap v7816: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 67 KiB/s wr, 8 op/s 2023-03-09T18:46:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14155.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:46:25.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:46:25 smithi122 ceph-mon[93869]: pgmap v7817: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 54 KiB/s wr, 8 op/s 2023-03-09T18:46:25.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:46:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13696.160416 secs 2023-03-09T18:46:25.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:46:25 smithi181 ceph-mon[94189]: pgmap v7817: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 54 KiB/s wr, 8 op/s 2023-03-09T18:46:25.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:46:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13696.160416 secs 2023-03-09T18:46:25.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:46:25 smithi038 ceph-mon[92422]: pgmap v7817: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 54 KiB/s wr, 8 op/s 2023-03-09T18:46:25.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:46:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13696.160416 secs 2023-03-09T18:46:27.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:46:27 smithi122 ceph-mon[93869]: pgmap v7818: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 103 KiB/s wr, 8 op/s 2023-03-09T18:46:27.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:46:27 smithi181 ceph-mon[94189]: pgmap v7818: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 103 KiB/s wr, 8 op/s 2023-03-09T18:46:27.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:46:27 smithi038 ceph-mon[92422]: pgmap v7818: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 103 KiB/s wr, 8 op/s 2023-03-09T18:46:28.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:46:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13698.436435 secs 2023-03-09T18:46:28.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:46:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13698.436435 secs 2023-03-09T18:46:28.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:46:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13698.436435 secs 2023-03-09T18:46:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14160.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:46:29.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:46:29 smithi122 ceph-mon[93869]: pgmap v7819: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 94 KiB/s wr, 7 op/s 2023-03-09T18:46:29.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:46:29 smithi181 ceph-mon[94189]: pgmap v7819: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 94 KiB/s wr, 7 op/s 2023-03-09T18:46:29.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:46:29 smithi038 ceph-mon[92422]: pgmap v7819: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 94 KiB/s wr, 7 op/s 2023-03-09T18:46:30.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:46:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13701.160475 secs 2023-03-09T18:46:30.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:46:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13701.160475 secs 2023-03-09T18:46:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:46:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13701.160475 secs 2023-03-09T18:46:31.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:46:31 smithi122 ceph-mon[93869]: pgmap v7820: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 86 KiB/s wr, 6 op/s 2023-03-09T18:46:31.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:46:31 smithi181 ceph-mon[94189]: pgmap v7820: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 86 KiB/s wr, 6 op/s 2023-03-09T18:46:31.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:46:31 smithi038 ceph-mon[92422]: pgmap v7820: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 86 KiB/s wr, 6 op/s 2023-03-09T18:46:33.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:46:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13703.436573 secs 2023-03-09T18:46:33.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:46:33 smithi122 ceph-mon[93869]: pgmap v7821: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 86 KiB/s wr, 8 op/s 2023-03-09T18:46:33.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:46:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13703.436573 secs 2023-03-09T18:46:33.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:46:33 smithi181 ceph-mon[94189]: pgmap v7821: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 86 KiB/s wr, 8 op/s 2023-03-09T18:46:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:46:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13703.436573 secs 2023-03-09T18:46:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:46:33 smithi038 ceph-mon[92422]: pgmap v7821: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 86 KiB/s wr, 8 op/s 2023-03-09T18:46:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14165.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:46:35.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:46:35 smithi122 ceph-mon[93869]: pgmap v7822: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 41 KiB/s wr, 6 op/s 2023-03-09T18:46:35.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:46:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13706.160535 secs 2023-03-09T18:46:35.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:46:35 smithi181 ceph-mon[94189]: pgmap v7822: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 41 KiB/s wr, 6 op/s 2023-03-09T18:46:35.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:46:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13706.160535 secs 2023-03-09T18:46:35.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:46:35 smithi038 ceph-mon[92422]: pgmap v7822: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 41 KiB/s wr, 6 op/s 2023-03-09T18:46:35.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:46:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13706.160535 secs 2023-03-09T18:46:37.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:46:37 smithi122 ceph-mon[93869]: pgmap v7823: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 96 KiB/s wr, 8 op/s 2023-03-09T18:46:37.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:46:37 smithi181 ceph-mon[94189]: pgmap v7823: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 96 KiB/s wr, 8 op/s 2023-03-09T18:46:37.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:46:37 smithi038 ceph-mon[92422]: pgmap v7823: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 96 KiB/s wr, 8 op/s 2023-03-09T18:46:38.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:46:38 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 13708.436716 secs 2023-03-09T18:46:38.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:46:38 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 13708.436716 secs 2023-03-09T18:46:38.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:46:38 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 13708.436716 secs 2023-03-09T18:46:38.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14170.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:46:39.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:46:39 smithi122 ceph-mon[93869]: pgmap v7824: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 56 KiB/s wr, 6 op/s 2023-03-09T18:46:39.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:46:39 smithi181 ceph-mon[94189]: pgmap v7824: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 56 KiB/s wr, 6 op/s 2023-03-09T18:46:39.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:46:39 smithi038 ceph-mon[92422]: pgmap v7824: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 56 KiB/s wr, 6 op/s 2023-03-09T18:46:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:46:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13711.160594 secs 2023-03-09T18:46:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:46:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13711.160594 secs 2023-03-09T18:46:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:46:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13711.160594 secs 2023-03-09T18:46:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:46:41 smithi122 ceph-mon[93869]: pgmap v7825: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 67 KiB/s wr, 7 op/s 2023-03-09T18:46:41.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:46:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:46:41.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:46:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:46:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:46:41 smithi181 ceph-mon[94189]: pgmap v7825: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 67 KiB/s wr, 7 op/s 2023-03-09T18:46:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:46:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:46:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:46:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:46:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:46:41 smithi038 ceph-mon[92422]: pgmap v7825: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 67 KiB/s wr, 7 op/s 2023-03-09T18:46:41.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:46:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:46:41.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:46:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:46:42.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:46:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:46:42.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:46:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:46:42.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:46:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:46:42.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:46:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:46:42.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:46:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13713.436800 secs 2023-03-09T18:46:42.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:46:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:46:42.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:46:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:46:42.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:46:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:46:42.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:46:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:46:42.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:46:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13713.436800 secs 2023-03-09T18:46:42.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:46:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:46:42.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:46:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:46:42.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:46:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:46:42.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:46:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:46:42.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:46:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13713.436800 secs 2023-03-09T18:46:43.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14175.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:46:44.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:46:43 smithi038 ceph-mon[92422]: pgmap v7826: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 186 KiB/s wr, 10 op/s 2023-03-09T18:46:44.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:46:43 smithi122 ceph-mon[93869]: pgmap v7826: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 186 KiB/s wr, 10 op/s 2023-03-09T18:46:44.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:46:43 smithi181 ceph-mon[94189]: pgmap v7826: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 186 KiB/s wr, 10 op/s 2023-03-09T18:46:45.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:46:45 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:46:45.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:46:45 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:46:45.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:46:45 smithi122 ceph-mon[93869]: pgmap v7827: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 188 KiB/s wr, 34 op/s 2023-03-09T18:46:45.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:46:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13716.160678 secs 2023-03-09T18:46:45.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:46:45 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:46:45.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:46:45 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:46:45.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:46:45 smithi181 ceph-mon[94189]: pgmap v7827: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 188 KiB/s wr, 34 op/s 2023-03-09T18:46:45.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:46:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13716.160678 secs 2023-03-09T18:46:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:46:45 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:46:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:46:45 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:46:45.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:46:45 smithi038 ceph-mon[92422]: pgmap v7827: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 188 KiB/s wr, 34 op/s 2023-03-09T18:46:45.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:46:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13716.160678 secs 2023-03-09T18:46:47.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:46:47 smithi122 ceph-mon[93869]: pgmap v7828: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 200 KiB/s wr, 37 op/s 2023-03-09T18:46:47.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:46:47 smithi181 ceph-mon[94189]: pgmap v7828: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 200 KiB/s wr, 37 op/s 2023-03-09T18:46:47.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:46:47 smithi038 ceph-mon[92422]: pgmap v7828: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 200 KiB/s wr, 37 op/s 2023-03-09T18:46:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:46:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13718.436897 secs 2023-03-09T18:46:48.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:46:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T18:46:48.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:46:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:46:48.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:46:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13718.436897 secs 2023-03-09T18:46:48.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:46:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T18:46:48.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:46:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:46:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:46:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13718.436897 secs 2023-03-09T18:46:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:46:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T18:46:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:46:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:46:48.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14180.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:46:49.481 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:46:49 smithi122 ceph-mon[93869]: pgmap v7829: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 145 KiB/s wr, 45 op/s 2023-03-09T18:46:49.481 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:46:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T18:46:49.482 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:46:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:46:49.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:46:49 smithi181 ceph-mon[94189]: pgmap v7829: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 145 KiB/s wr, 45 op/s 2023-03-09T18:46:49.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:46:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T18:46:49.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:46:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:46:49.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:46:49 smithi038 ceph-mon[92422]: pgmap v7829: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 145 KiB/s wr, 45 op/s 2023-03-09T18:46:49.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:46:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T18:46:49.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:46:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:46:50.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:46:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13721.160747 secs 2023-03-09T18:46:50.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:46:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13721.160747 secs 2023-03-09T18:46:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:46:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13721.160747 secs 2023-03-09T18:46:51.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:46:51 smithi122 ceph-mon[93869]: pgmap v7830: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 165 KiB/s wr, 46 op/s 2023-03-09T18:46:51.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:46:51 smithi181 ceph-mon[94189]: pgmap v7830: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 165 KiB/s wr, 46 op/s 2023-03-09T18:46:51.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:46:51 smithi038 ceph-mon[92422]: pgmap v7830: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 165 KiB/s wr, 46 op/s 2023-03-09T18:46:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:46:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13723.437037 secs 2023-03-09T18:46:53.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:46:53 smithi122 ceph-mon[93869]: pgmap v7831: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 303 KiB/s wr, 49 op/s 2023-03-09T18:46:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:46:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13723.437037 secs 2023-03-09T18:46:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:46:53 smithi181 ceph-mon[94189]: pgmap v7831: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 303 KiB/s wr, 49 op/s 2023-03-09T18:46:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:46:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13723.437037 secs 2023-03-09T18:46:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:46:53 smithi038 ceph-mon[92422]: pgmap v7831: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 303 KiB/s wr, 49 op/s 2023-03-09T18:46:53.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14185.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:46:55.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:46:55 smithi122 ceph-mon[93869]: pgmap v7832: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 195 KiB/s wr, 46 op/s 2023-03-09T18:46:55.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:46:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13726.160842 secs 2023-03-09T18:46:55.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:46:55 smithi181 ceph-mon[94189]: pgmap v7832: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 195 KiB/s wr, 46 op/s 2023-03-09T18:46:55.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:46:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13726.160842 secs 2023-03-09T18:46:55.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:46:55 smithi038 ceph-mon[92422]: pgmap v7832: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 195 KiB/s wr, 46 op/s 2023-03-09T18:46:55.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:46:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13726.160842 secs 2023-03-09T18:46:57.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:46:57 smithi122 ceph-mon[93869]: pgmap v7833: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 214 KiB/s wr, 22 op/s 2023-03-09T18:46:57.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:46:57 smithi181 ceph-mon[94189]: pgmap v7833: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 214 KiB/s wr, 22 op/s 2023-03-09T18:46:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:46:57 smithi038 ceph-mon[92422]: pgmap v7833: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 214 KiB/s wr, 22 op/s 2023-03-09T18:46:58.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:46:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13728.437184 secs 2023-03-09T18:46:58.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:46:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13728.437184 secs 2023-03-09T18:46:58.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:46:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13728.437184 secs 2023-03-09T18:46:58.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14190.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:46:59.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:46:59 smithi122 ceph-mon[93869]: pgmap v7834: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 203 KiB/s wr, 18 op/s 2023-03-09T18:46:59.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:46:59 smithi181 ceph-mon[94189]: pgmap v7834: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 203 KiB/s wr, 18 op/s 2023-03-09T18:46:59.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:46:59 smithi038 ceph-mon[92422]: pgmap v7834: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 203 KiB/s wr, 18 op/s 2023-03-09T18:47:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:47:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13731.160900 secs 2023-03-09T18:47:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:47:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13731.160900 secs 2023-03-09T18:47:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:47:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13731.160900 secs 2023-03-09T18:47:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:47:01 smithi122 ceph-mon[93869]: pgmap v7835: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 214 KiB/s wr, 8 op/s 2023-03-09T18:47:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:47:01 smithi181 ceph-mon[94189]: pgmap v7835: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 214 KiB/s wr, 8 op/s 2023-03-09T18:47:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:47:01 smithi038 ceph-mon[92422]: pgmap v7835: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 214 KiB/s wr, 8 op/s 2023-03-09T18:47:03.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:47:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13733.437264 secs 2023-03-09T18:47:03.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:47:03 smithi122 ceph-mon[93869]: pgmap v7836: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 287 KiB/s wr, 13 op/s 2023-03-09T18:47:03.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:47:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13733.437264 secs 2023-03-09T18:47:03.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:47:03 smithi181 ceph-mon[94189]: pgmap v7836: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 287 KiB/s wr, 13 op/s 2023-03-09T18:47:03.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:47:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13733.437264 secs 2023-03-09T18:47:03.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:47:03 smithi038 ceph-mon[92422]: pgmap v7836: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 287 KiB/s wr, 13 op/s 2023-03-09T18:47:03.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14195.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:47:05.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:47:05 smithi122 ceph-mon[93869]: pgmap v7837: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 159 KiB/s wr, 22 op/s 2023-03-09T18:47:05.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:47:05 smithi122 ceph-mon[93869]: osdmap e703: 12 total, 12 up, 12 in 2023-03-09T18:47:05.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:47:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13736.160964 secs 2023-03-09T18:47:05.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:47:05 smithi181 ceph-mon[94189]: pgmap v7837: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 159 KiB/s wr, 22 op/s 2023-03-09T18:47:05.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:47:05 smithi181 ceph-mon[94189]: osdmap e703: 12 total, 12 up, 12 in 2023-03-09T18:47:05.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:47:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13736.160964 secs 2023-03-09T18:47:05.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:47:05 smithi038 ceph-mon[92422]: pgmap v7837: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 159 KiB/s wr, 22 op/s 2023-03-09T18:47:05.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:47:05 smithi038 ceph-mon[92422]: osdmap e703: 12 total, 12 up, 12 in 2023-03-09T18:47:05.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:47:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13736.160964 secs 2023-03-09T18:47:07.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:47:07 smithi122 ceph-mon[93869]: pgmap v7839: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 195 KiB/s wr, 26 op/s 2023-03-09T18:47:07.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:47:07 smithi181 ceph-mon[94189]: pgmap v7839: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 195 KiB/s wr, 26 op/s 2023-03-09T18:47:07.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:47:07 smithi038 ceph-mon[92422]: pgmap v7839: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 195 KiB/s wr, 26 op/s 2023-03-09T18:47:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:47:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13738.437386 secs 2023-03-09T18:47:08.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:47:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T18:47:08.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:47:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:47:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:47:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13738.437386 secs 2023-03-09T18:47:08.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:47:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T18:47:08.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:47:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:47:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:47:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13738.437386 secs 2023-03-09T18:47:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:47:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T18:47:08.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:47:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:47:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14200.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:47:09.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:47:09 smithi122 ceph-mon[93869]: pgmap v7840: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 264 KiB/s wr, 27 op/s 2023-03-09T18:47:09.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:47:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T18:47:09.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:47:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:47:09.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:47:09 smithi181 ceph-mon[94189]: pgmap v7840: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 264 KiB/s wr, 27 op/s 2023-03-09T18:47:09.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:47:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T18:47:09.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:47:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:47:09.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:47:09 smithi038 ceph-mon[92422]: pgmap v7840: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 264 KiB/s wr, 27 op/s 2023-03-09T18:47:09.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:47:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T18:47:09.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:47:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:47:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:47:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13741.160986 secs 2023-03-09T18:47:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:47:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13741.160986 secs 2023-03-09T18:47:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:47:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13741.160986 secs 2023-03-09T18:47:11.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:47:11 smithi122 ceph-mon[93869]: pgmap v7841: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 265 KiB/s wr, 27 op/s 2023-03-09T18:47:11.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:47:11 smithi181 ceph-mon[94189]: pgmap v7841: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 265 KiB/s wr, 27 op/s 2023-03-09T18:47:11.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:47:11 smithi038 ceph-mon[92422]: pgmap v7841: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 265 KiB/s wr, 27 op/s 2023-03-09T18:47:13.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:47:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13743.437512 secs 2023-03-09T18:47:13.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:47:13 smithi122 ceph-mon[93869]: pgmap v7842: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 188 KiB/s wr, 24 op/s 2023-03-09T18:47:13.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:47:13 smithi122 ceph-mon[93869]: osdmap e704: 12 total, 12 up, 12 in 2023-03-09T18:47:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:47:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13743.437512 secs 2023-03-09T18:47:13.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:47:13 smithi181 ceph-mon[94189]: pgmap v7842: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 188 KiB/s wr, 24 op/s 2023-03-09T18:47:13.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:47:13 smithi181 ceph-mon[94189]: osdmap e704: 12 total, 12 up, 12 in 2023-03-09T18:47:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:47:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13743.437512 secs 2023-03-09T18:47:13.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:47:13 smithi038 ceph-mon[92422]: pgmap v7842: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 188 KiB/s wr, 24 op/s 2023-03-09T18:47:13.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:47:13 smithi038 ceph-mon[92422]: osdmap e704: 12 total, 12 up, 12 in 2023-03-09T18:47:13.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14205.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:47:15.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:47:15 smithi122 ceph-mon[93869]: pgmap v7844: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 164 KiB/s wr, 9 op/s 2023-03-09T18:47:15.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:47:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13746.161046 secs 2023-03-09T18:47:15.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:47:15 smithi181 ceph-mon[94189]: pgmap v7844: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 164 KiB/s wr, 9 op/s 2023-03-09T18:47:15.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:47:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13746.161046 secs 2023-03-09T18:47:15.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:47:15 smithi038 ceph-mon[92422]: pgmap v7844: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 164 KiB/s wr, 9 op/s 2023-03-09T18:47:15.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:47:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13746.161046 secs 2023-03-09T18:47:17.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:47:17 smithi122 ceph-mon[93869]: pgmap v7845: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 120 KiB/s wr, 8 op/s 2023-03-09T18:47:17.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:47:17 smithi181 ceph-mon[94189]: pgmap v7845: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 120 KiB/s wr, 8 op/s 2023-03-09T18:47:17.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:47:17 smithi038 ceph-mon[92422]: pgmap v7845: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 120 KiB/s wr, 8 op/s 2023-03-09T18:47:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:47:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13748.437647 secs 2023-03-09T18:47:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:47:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13748.437647 secs 2023-03-09T18:47:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:47:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13748.437647 secs 2023-03-09T18:47:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14210.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:47:19.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:47:19 smithi122 ceph-mon[93869]: pgmap v7846: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 7 op/s 2023-03-09T18:47:19.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:47:19 smithi181 ceph-mon[94189]: pgmap v7846: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 7 op/s 2023-03-09T18:47:19.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:47:19 smithi038 ceph-mon[92422]: pgmap v7846: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 7 op/s 2023-03-09T18:47:20.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:47:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13751.161108 secs 2023-03-09T18:47:20.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:47:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13751.161108 secs 2023-03-09T18:47:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:47:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13751.161108 secs 2023-03-09T18:47:21.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:47:21 smithi122 ceph-mon[93869]: pgmap v7847: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 37 KiB/s wr, 7 op/s 2023-03-09T18:47:21.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:47:21 smithi181 ceph-mon[94189]: pgmap v7847: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 37 KiB/s wr, 7 op/s 2023-03-09T18:47:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:47:21 smithi038 ceph-mon[92422]: pgmap v7847: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 37 KiB/s wr, 7 op/s 2023-03-09T18:47:23.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:47:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13753.437783 secs 2023-03-09T18:47:23.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:47:23 smithi122 ceph-mon[93869]: pgmap v7848: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2023-03-09T18:47:23.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:47:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13753.437783 secs 2023-03-09T18:47:23.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:47:23 smithi181 ceph-mon[94189]: pgmap v7848: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2023-03-09T18:47:23.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:47:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13753.437783 secs 2023-03-09T18:47:23.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:47:23 smithi038 ceph-mon[92422]: pgmap v7848: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2023-03-09T18:47:23.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14215.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:47:25.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:47:25 smithi122 ceph-mon[93869]: pgmap v7849: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2023-03-09T18:47:25.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:47:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13756.161170 secs 2023-03-09T18:47:25.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:47:25 smithi181 ceph-mon[94189]: pgmap v7849: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2023-03-09T18:47:25.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:47:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13756.161170 secs 2023-03-09T18:47:25.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:47:25 smithi038 ceph-mon[92422]: pgmap v7849: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2023-03-09T18:47:25.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:47:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13756.161170 secs 2023-03-09T18:47:27.481 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:47:27 smithi122 ceph-mon[93869]: pgmap v7850: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 2.2 KiB/s wr, 8 op/s 2023-03-09T18:47:27.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:47:27 smithi181 ceph-mon[94189]: pgmap v7850: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 2.2 KiB/s wr, 8 op/s 2023-03-09T18:47:27.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:47:27 smithi038 ceph-mon[92422]: pgmap v7850: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 2.2 KiB/s wr, 8 op/s 2023-03-09T18:47:28.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:47:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13758.437883 secs 2023-03-09T18:47:28.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:47:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13758.437883 secs 2023-03-09T18:47:28.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:47:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13758.437883 secs 2023-03-09T18:47:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14220.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:47:29.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:47:29 smithi122 ceph-mon[93869]: pgmap v7851: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 45 KiB/s wr, 7 op/s 2023-03-09T18:47:29.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:47:29 smithi181 ceph-mon[94189]: pgmap v7851: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 45 KiB/s wr, 7 op/s 2023-03-09T18:47:29.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:47:29 smithi038 ceph-mon[92422]: pgmap v7851: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 45 KiB/s wr, 7 op/s 2023-03-09T18:47:30.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:47:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13761.161231 secs 2023-03-09T18:47:30.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:47:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13761.161231 secs 2023-03-09T18:47:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:47:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13761.161231 secs 2023-03-09T18:47:31.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:47:31 smithi122 ceph-mon[93869]: pgmap v7852: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 46 KiB/s wr, 7 op/s 2023-03-09T18:47:31.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:47:31 smithi181 ceph-mon[94189]: pgmap v7852: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 46 KiB/s wr, 7 op/s 2023-03-09T18:47:31.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:47:31 smithi038 ceph-mon[92422]: pgmap v7852: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 46 KiB/s wr, 7 op/s 2023-03-09T18:47:33.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:47:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13763.437995 secs 2023-03-09T18:47:33.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:47:33 smithi122 ceph-mon[93869]: pgmap v7853: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 87 KiB/s wr, 9 op/s 2023-03-09T18:47:33.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:47:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13763.437995 secs 2023-03-09T18:47:33.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:47:33 smithi181 ceph-mon[94189]: pgmap v7853: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 87 KiB/s wr, 9 op/s 2023-03-09T18:47:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:47:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13763.437995 secs 2023-03-09T18:47:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:47:33 smithi038 ceph-mon[92422]: pgmap v7853: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 87 KiB/s wr, 9 op/s 2023-03-09T18:47:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14225.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:47:35.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:47:35 smithi122 ceph-mon[93869]: pgmap v7854: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 87 KiB/s wr, 7 op/s 2023-03-09T18:47:35.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:47:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13766.161289 secs 2023-03-09T18:47:35.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:47:35 smithi181 ceph-mon[94189]: pgmap v7854: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 87 KiB/s wr, 7 op/s 2023-03-09T18:47:35.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:47:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13766.161289 secs 2023-03-09T18:47:35.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:47:35 smithi038 ceph-mon[92422]: pgmap v7854: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 87 KiB/s wr, 7 op/s 2023-03-09T18:47:35.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:47:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13766.161289 secs 2023-03-09T18:47:37.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:47:37 smithi122 ceph-mon[93869]: pgmap v7855: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 86 KiB/s wr, 8 op/s 2023-03-09T18:47:37.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:47:37 smithi181 ceph-mon[94189]: pgmap v7855: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 86 KiB/s wr, 8 op/s 2023-03-09T18:47:37.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:47:37 smithi038 ceph-mon[92422]: pgmap v7855: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 86 KiB/s wr, 8 op/s 2023-03-09T18:47:38.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14230.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:47:39.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:47:39 smithi122 ceph-mon[93869]: pgmap v7856: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 141 KiB/s wr, 7 op/s 2023-03-09T18:47:39.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:47:39 smithi181 ceph-mon[94189]: pgmap v7856: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 141 KiB/s wr, 7 op/s 2023-03-09T18:47:39.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:47:39 smithi038 ceph-mon[92422]: pgmap v7856: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 141 KiB/s wr, 7 op/s 2023-03-09T18:47:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:47:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13771.161347 secs 2023-03-09T18:47:40.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:47:40 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:47:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:47:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13771.161347 secs 2023-03-09T18:47:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:47:40 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:47:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:47:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13771.161347 secs 2023-03-09T18:47:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:47:40 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:47:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:47:41 smithi122 ceph-mon[93869]: pgmap v7857: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 108 KiB/s wr, 7 op/s 2023-03-09T18:47:41.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:47:41 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:47:41.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:47:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:47:41.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:47:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:47:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:47:41 smithi181 ceph-mon[94189]: pgmap v7857: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 108 KiB/s wr, 7 op/s 2023-03-09T18:47:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:47:41 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:47:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:47:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:47:41.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:47:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:47:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:47:41 smithi038 ceph-mon[92422]: pgmap v7857: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 108 KiB/s wr, 7 op/s 2023-03-09T18:47:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:47:41 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:47:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:47:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:47:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:47:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:47:42.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:47:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:47:42.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:47:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:47:42.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:47:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:47:43.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:47:43 smithi122 ceph-mon[93869]: pgmap v7858: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 189 KiB/s wr, 9 op/s 2023-03-09T18:47:43.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:47:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:47:43.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:47:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:47:43.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:47:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:47:43.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:47:43 smithi181 ceph-mon[94189]: pgmap v7858: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 189 KiB/s wr, 9 op/s 2023-03-09T18:47:43.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:47:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:47:43.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:47:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:47:43.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:47:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:47:43.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:47:43 smithi038 ceph-mon[92422]: pgmap v7858: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 189 KiB/s wr, 9 op/s 2023-03-09T18:47:43.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:47:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:47:43.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:47:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:47:43.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:47:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:47:43.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14235.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:47:45.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:47:45 smithi122 ceph-mon[93869]: pgmap v7859: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 159 KiB/s wr, 8 op/s 2023-03-09T18:47:45.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:47:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13776.161408 secs 2023-03-09T18:47:45.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:47:45 smithi181 ceph-mon[94189]: pgmap v7859: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 159 KiB/s wr, 8 op/s 2023-03-09T18:47:45.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:47:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13776.161408 secs 2023-03-09T18:47:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:47:45 smithi038 ceph-mon[92422]: pgmap v7859: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 159 KiB/s wr, 8 op/s 2023-03-09T18:47:45.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:47:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13776.161408 secs 2023-03-09T18:47:47.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:47:47 smithi122 ceph-mon[93869]: pgmap v7860: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 159 KiB/s wr, 9 op/s 2023-03-09T18:47:47.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:47:47 smithi181 ceph-mon[94189]: pgmap v7860: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 159 KiB/s wr, 9 op/s 2023-03-09T18:47:47.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:47:47 smithi038 ceph-mon[92422]: pgmap v7860: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 159 KiB/s wr, 9 op/s 2023-03-09T18:47:48.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14240.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:47:49.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:47:49 smithi122 ceph-mon[93869]: pgmap v7861: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 233 KiB/s wr, 10 op/s 2023-03-09T18:47:49.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:47:49 smithi181 ceph-mon[94189]: pgmap v7861: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 233 KiB/s wr, 10 op/s 2023-03-09T18:47:49.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:47:49 smithi038 ceph-mon[92422]: pgmap v7861: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 233 KiB/s wr, 10 op/s 2023-03-09T18:47:50.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:47:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13781.161467 secs 2023-03-09T18:47:50.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:47:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13781.161467 secs 2023-03-09T18:47:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:47:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13781.161467 secs 2023-03-09T18:47:51.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:47:51 smithi122 ceph-mon[93869]: pgmap v7862: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 189 KiB/s wr, 9 op/s 2023-03-09T18:47:51.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:47:51 smithi181 ceph-mon[94189]: pgmap v7862: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 189 KiB/s wr, 9 op/s 2023-03-09T18:47:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:47:51 smithi038 ceph-mon[92422]: pgmap v7862: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 189 KiB/s wr, 9 op/s 2023-03-09T18:47:52.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:47:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T18:47:52.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:47:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:47:52.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:47:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T18:47:52.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:47:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:47:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:47:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T18:47:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:47:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:47:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:47:53 smithi122 ceph-mon[93869]: pgmap v7863: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 294 KiB/s wr, 13 op/s 2023-03-09T18:47:53.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:47:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T18:47:53.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:47:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:47:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:47:53 smithi181 ceph-mon[94189]: pgmap v7863: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 294 KiB/s wr, 13 op/s 2023-03-09T18:47:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:47:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T18:47:53.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:47:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:47:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:47:53 smithi038 ceph-mon[92422]: pgmap v7863: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 294 KiB/s wr, 13 op/s 2023-03-09T18:47:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:47:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T18:47:53.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:47:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:47:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14245.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:47:55.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:47:55 smithi122 ceph-mon[93869]: pgmap v7864: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 224 KiB/s wr, 11 op/s 2023-03-09T18:47:55.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:47:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13786.161529 secs 2023-03-09T18:47:55.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:47:55 smithi181 ceph-mon[94189]: pgmap v7864: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 224 KiB/s wr, 11 op/s 2023-03-09T18:47:55.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:47:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13786.161529 secs 2023-03-09T18:47:55.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:47:55 smithi038 ceph-mon[92422]: pgmap v7864: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 224 KiB/s wr, 11 op/s 2023-03-09T18:47:55.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:47:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13786.161529 secs 2023-03-09T18:47:57.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:47:57 smithi181 ceph-mon[94189]: pgmap v7865: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 213 KiB/s wr, 12 op/s 2023-03-09T18:47:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:47:57 smithi038 ceph-mon[92422]: pgmap v7865: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 213 KiB/s wr, 12 op/s 2023-03-09T18:47:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:47:57 smithi122 ceph-mon[93869]: pgmap v7865: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 213 KiB/s wr, 12 op/s 2023-03-09T18:47:58.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14250.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:47:59.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:47:59 smithi038 ceph-mon[92422]: pgmap v7866: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 305 KiB/s wr, 11 op/s 2023-03-09T18:47:59.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:47:59 smithi122 ceph-mon[93869]: pgmap v7866: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 305 KiB/s wr, 11 op/s 2023-03-09T18:47:59.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:47:59 smithi181 ceph-mon[94189]: pgmap v7866: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 305 KiB/s wr, 11 op/s 2023-03-09T18:48:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13791.161608 secs 2023-03-09T18:48:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13791.161608 secs 2023-03-09T18:48:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13791.161608 secs 2023-03-09T18:48:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:01 smithi122 ceph-mon[93869]: pgmap v7867: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 241 KiB/s wr, 10 op/s 2023-03-09T18:48:01.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:01 smithi181 ceph-mon[94189]: pgmap v7867: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 241 KiB/s wr, 10 op/s 2023-03-09T18:48:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:01 smithi038 ceph-mon[92422]: pgmap v7867: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 241 KiB/s wr, 10 op/s 2023-03-09T18:48:03.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13793.438713 secs 2023-03-09T18:48:03.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:03 smithi122 ceph-mon[93869]: pgmap v7868: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 346 KiB/s wr, 15 op/s 2023-03-09T18:48:03.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13793.438713 secs 2023-03-09T18:48:03.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:03 smithi181 ceph-mon[94189]: pgmap v7868: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 346 KiB/s wr, 15 op/s 2023-03-09T18:48:03.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13793.438713 secs 2023-03-09T18:48:03.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:03 smithi038 ceph-mon[92422]: pgmap v7868: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 346 KiB/s wr, 15 op/s 2023-03-09T18:48:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14255.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:48:04.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:04 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:48:04.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:04 smithi122 ceph-mon[93869]: osdmap e705: 12 total, 12 up, 12 in 2023-03-09T18:48:04.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:04 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:48:04.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:04 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:48:04.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:04 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:48:04.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:04 smithi181 ceph-mon[94189]: osdmap e705: 12 total, 12 up, 12 in 2023-03-09T18:48:04.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:04 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:48:04.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:04 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:48:04.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:04 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:48:04.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:04 smithi038 ceph-mon[92422]: osdmap e705: 12 total, 12 up, 12 in 2023-03-09T18:48:04.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:04 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:48:04.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:04 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:48:05.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:05 smithi122 ceph-mon[93869]: pgmap v7869: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 252 KiB/s wr, 23 op/s 2023-03-09T18:48:05.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13796.161667 secs 2023-03-09T18:48:05.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:05 smithi181 ceph-mon[94189]: pgmap v7869: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 252 KiB/s wr, 23 op/s 2023-03-09T18:48:05.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13796.161667 secs 2023-03-09T18:48:05.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:05 smithi038 ceph-mon[92422]: pgmap v7869: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 252 KiB/s wr, 23 op/s 2023-03-09T18:48:05.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13796.161667 secs 2023-03-09T18:48:07.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:07 smithi122 ceph-mon[93869]: pgmap v7871: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 403 KiB/s wr, 28 op/s 2023-03-09T18:48:07.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13798.438780 secs 2023-03-09T18:48:07.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:07 smithi181 ceph-mon[94189]: pgmap v7871: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 403 KiB/s wr, 28 op/s 2023-03-09T18:48:07.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13798.438780 secs 2023-03-09T18:48:07.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:07 smithi038 ceph-mon[92422]: pgmap v7871: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 403 KiB/s wr, 28 op/s 2023-03-09T18:48:07.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13798.438780 secs 2023-03-09T18:48:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T18:48:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:48:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T18:48:08.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:48:08.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T18:48:08.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:48:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14260.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:48:09.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:09 smithi038 ceph-mon[92422]: pgmap v7872: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 295 KiB/s wr, 27 op/s 2023-03-09T18:48:09.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T18:48:09.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:48:09.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:09 smithi122 ceph-mon[93869]: pgmap v7872: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 295 KiB/s wr, 27 op/s 2023-03-09T18:48:09.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T18:48:09.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:48:09.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:09 smithi181 ceph-mon[94189]: pgmap v7872: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 295 KiB/s wr, 27 op/s 2023-03-09T18:48:09.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T18:48:09.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:48:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13801.161729 secs 2023-03-09T18:48:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13801.161729 secs 2023-03-09T18:48:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13801.161729 secs 2023-03-09T18:48:11.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:11 smithi122 ceph-mon[93869]: pgmap v7873: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 295 KiB/s wr, 27 op/s 2023-03-09T18:48:11.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:11 smithi181 ceph-mon[94189]: pgmap v7873: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 295 KiB/s wr, 27 op/s 2023-03-09T18:48:11.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:11 smithi038 ceph-mon[92422]: pgmap v7873: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 295 KiB/s wr, 27 op/s 2023-03-09T18:48:13.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13803.438903 secs 2023-03-09T18:48:13.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:13 smithi122 ceph-mon[93869]: pgmap v7874: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 250 KiB/s wr, 25 op/s 2023-03-09T18:48:13.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:13 smithi122 ceph-mon[93869]: osdmap e706: 12 total, 12 up, 12 in 2023-03-09T18:48:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13803.438903 secs 2023-03-09T18:48:13.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:13 smithi181 ceph-mon[94189]: pgmap v7874: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 250 KiB/s wr, 25 op/s 2023-03-09T18:48:13.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:13 smithi181 ceph-mon[94189]: osdmap e706: 12 total, 12 up, 12 in 2023-03-09T18:48:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13803.438903 secs 2023-03-09T18:48:13.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:13 smithi038 ceph-mon[92422]: pgmap v7874: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 250 KiB/s wr, 25 op/s 2023-03-09T18:48:13.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:13 smithi038 ceph-mon[92422]: osdmap e706: 12 total, 12 up, 12 in 2023-03-09T18:48:13.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14265.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:48:15.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:15 smithi122 ceph-mon[93869]: pgmap v7876: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 225 KiB/s wr, 10 op/s 2023-03-09T18:48:15.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13806.161813 secs 2023-03-09T18:48:15.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:15 smithi181 ceph-mon[94189]: pgmap v7876: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 225 KiB/s wr, 10 op/s 2023-03-09T18:48:15.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13806.161813 secs 2023-03-09T18:48:15.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:15 smithi038 ceph-mon[92422]: pgmap v7876: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 225 KiB/s wr, 10 op/s 2023-03-09T18:48:15.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13806.161813 secs 2023-03-09T18:48:17.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:17 smithi122 ceph-mon[93869]: pgmap v7877: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 110 KiB/s wr, 9 op/s 2023-03-09T18:48:17.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:17 smithi181 ceph-mon[94189]: pgmap v7877: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 110 KiB/s wr, 9 op/s 2023-03-09T18:48:17.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:17 smithi038 ceph-mon[92422]: pgmap v7877: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 110 KiB/s wr, 9 op/s 2023-03-09T18:48:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13808.439025 secs 2023-03-09T18:48:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13808.439025 secs 2023-03-09T18:48:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13808.439025 secs 2023-03-09T18:48:18.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14270.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:48:19.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:19 smithi122 ceph-mon[93869]: pgmap v7878: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 108 KiB/s wr, 8 op/s 2023-03-09T18:48:19.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:19 smithi181 ceph-mon[94189]: pgmap v7878: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 108 KiB/s wr, 8 op/s 2023-03-09T18:48:19.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:19 smithi038 ceph-mon[92422]: pgmap v7878: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 108 KiB/s wr, 8 op/s 2023-03-09T18:48:20.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13811.161873 secs 2023-03-09T18:48:20.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13811.161873 secs 2023-03-09T18:48:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13811.161873 secs 2023-03-09T18:48:21.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:21 smithi122 ceph-mon[93869]: pgmap v7879: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 107 KiB/s wr, 8 op/s 2023-03-09T18:48:21.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:21 smithi181 ceph-mon[94189]: pgmap v7879: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 107 KiB/s wr, 8 op/s 2023-03-09T18:48:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:21 smithi038 ceph-mon[92422]: pgmap v7879: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 107 KiB/s wr, 8 op/s 2023-03-09T18:48:23.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13813.439168 secs 2023-03-09T18:48:23.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:23 smithi122 ceph-mon[93869]: pgmap v7880: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T18:48:23.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13813.439168 secs 2023-03-09T18:48:23.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:23 smithi181 ceph-mon[94189]: pgmap v7880: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T18:48:23.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13813.439168 secs 2023-03-09T18:48:23.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:23 smithi038 ceph-mon[92422]: pgmap v7880: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T18:48:23.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14275.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:48:25.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:25 smithi122 ceph-mon[93869]: pgmap v7881: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 37 KiB/s wr, 6 op/s 2023-03-09T18:48:25.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13816.161946 secs 2023-03-09T18:48:25.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:25 smithi181 ceph-mon[94189]: pgmap v7881: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 37 KiB/s wr, 6 op/s 2023-03-09T18:48:25.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13816.161946 secs 2023-03-09T18:48:25.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:25 smithi038 ceph-mon[92422]: pgmap v7881: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 37 KiB/s wr, 6 op/s 2023-03-09T18:48:25.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13816.161946 secs 2023-03-09T18:48:27.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:27 smithi122 ceph-mon[93869]: pgmap v7882: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 37 KiB/s wr, 8 op/s 2023-03-09T18:48:27.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13818.439285 secs 2023-03-09T18:48:27.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:27 smithi181 ceph-mon[94189]: pgmap v7882: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 37 KiB/s wr, 8 op/s 2023-03-09T18:48:27.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13818.439285 secs 2023-03-09T18:48:27.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:27 smithi038 ceph-mon[92422]: pgmap v7882: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 37 KiB/s wr, 8 op/s 2023-03-09T18:48:27.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13818.439285 secs 2023-03-09T18:48:28.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14280.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:48:29.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:29 smithi122 ceph-mon[93869]: pgmap v7883: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 37 KiB/s wr, 6 op/s 2023-03-09T18:48:29.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:29 smithi181 ceph-mon[94189]: pgmap v7883: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 37 KiB/s wr, 6 op/s 2023-03-09T18:48:29.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:29 smithi038 ceph-mon[92422]: pgmap v7883: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 37 KiB/s wr, 6 op/s 2023-03-09T18:48:30.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13821.161986 secs 2023-03-09T18:48:30.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13821.161986 secs 2023-03-09T18:48:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13821.161986 secs 2023-03-09T18:48:31.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:31 smithi122 ceph-mon[93869]: pgmap v7884: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 37 KiB/s wr, 7 op/s 2023-03-09T18:48:31.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:31 smithi181 ceph-mon[94189]: pgmap v7884: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 37 KiB/s wr, 7 op/s 2023-03-09T18:48:31.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:31 smithi038 ceph-mon[92422]: pgmap v7884: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 37 KiB/s wr, 7 op/s 2023-03-09T18:48:33.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13823.439392 secs 2023-03-09T18:48:33.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:33 smithi122 ceph-mon[93869]: pgmap v7885: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 71 KiB/s wr, 9 op/s 2023-03-09T18:48:33.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13823.439392 secs 2023-03-09T18:48:33.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:33 smithi181 ceph-mon[94189]: pgmap v7885: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 71 KiB/s wr, 9 op/s 2023-03-09T18:48:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13823.439392 secs 2023-03-09T18:48:33.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:33 smithi038 ceph-mon[92422]: pgmap v7885: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 71 KiB/s wr, 9 op/s 2023-03-09T18:48:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14285.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:48:35.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:35 smithi122 ceph-mon[93869]: pgmap v7886: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 45 KiB/s wr, 6 op/s 2023-03-09T18:48:35.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13826.162045 secs 2023-03-09T18:48:35.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:35 smithi181 ceph-mon[94189]: pgmap v7886: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 45 KiB/s wr, 6 op/s 2023-03-09T18:48:35.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13826.162045 secs 2023-03-09T18:48:35.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:35 smithi038 ceph-mon[92422]: pgmap v7886: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 45 KiB/s wr, 6 op/s 2023-03-09T18:48:35.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13826.162045 secs 2023-03-09T18:48:37.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:37 smithi122 ceph-mon[93869]: pgmap v7887: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 91 KiB/s wr, 8 op/s 2023-03-09T18:48:37.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:37 smithi181 ceph-mon[94189]: pgmap v7887: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 91 KiB/s wr, 8 op/s 2023-03-09T18:48:37.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:37 smithi038 ceph-mon[92422]: pgmap v7887: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 91 KiB/s wr, 8 op/s 2023-03-09T18:48:38.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:38 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 13828.439510 secs 2023-03-09T18:48:38.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:38 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 13828.439510 secs 2023-03-09T18:48:38.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:38 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 13828.439510 secs 2023-03-09T18:48:38.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14290.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:48:39.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:39 smithi122 ceph-mon[93869]: pgmap v7888: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 91 KiB/s wr, 6 op/s 2023-03-09T18:48:39.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:39 smithi181 ceph-mon[94189]: pgmap v7888: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 91 KiB/s wr, 6 op/s 2023-03-09T18:48:39.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:39 smithi038 ceph-mon[92422]: pgmap v7888: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 91 KiB/s wr, 6 op/s 2023-03-09T18:48:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13831.162107 secs 2023-03-09T18:48:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13831.162107 secs 2023-03-09T18:48:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13831.162107 secs 2023-03-09T18:48:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:41 smithi122 ceph-mon[93869]: pgmap v7889: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 113 KiB/s wr, 7 op/s 2023-03-09T18:48:41.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T18:48:41.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:48:41.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:48:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:41 smithi181 ceph-mon[94189]: pgmap v7889: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 113 KiB/s wr, 7 op/s 2023-03-09T18:48:41.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T18:48:41.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:48:41.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:48:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:41 smithi038 ceph-mon[92422]: pgmap v7889: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 113 KiB/s wr, 7 op/s 2023-03-09T18:48:41.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T18:48:41.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:48:41.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:48:43.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13833.439629 secs 2023-03-09T18:48:43.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:43 smithi122 ceph-mon[93869]: pgmap v7890: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 251 KiB/s wr, 10 op/s 2023-03-09T18:48:43.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:48:43.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13833.439629 secs 2023-03-09T18:48:43.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:43 smithi181 ceph-mon[94189]: pgmap v7890: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 251 KiB/s wr, 10 op/s 2023-03-09T18:48:43.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:48:43.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13833.439629 secs 2023-03-09T18:48:43.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:43 smithi038 ceph-mon[92422]: pgmap v7890: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 251 KiB/s wr, 10 op/s 2023-03-09T18:48:43.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:48:43.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14295.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:48:45.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:45 smithi122 ceph-mon[93869]: pgmap v7891: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 206 KiB/s wr, 9 op/s 2023-03-09T18:48:45.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:45 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:48:45.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:45 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:48:45.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13836.162167 secs 2023-03-09T18:48:45.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:45 smithi181 ceph-mon[94189]: pgmap v7891: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 206 KiB/s wr, 9 op/s 2023-03-09T18:48:45.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:45 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:48:45.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:45 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:48:45.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13836.162167 secs 2023-03-09T18:48:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:45 smithi038 ceph-mon[92422]: pgmap v7891: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 206 KiB/s wr, 9 op/s 2023-03-09T18:48:45.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:45 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:48:45.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:45 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:48:45.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13836.162167 secs 2023-03-09T18:48:47.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:47 smithi038 ceph-mon[92422]: pgmap v7892: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 224 KiB/s wr, 11 op/s 2023-03-09T18:48:47.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:48:47.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:48:47.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:48:47.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:48:47.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:48:47.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:47 smithi122 ceph-mon[93869]: pgmap v7892: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 224 KiB/s wr, 11 op/s 2023-03-09T18:48:47.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:48:47.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:48:47.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:48:47.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:48:47.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:48:47.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:47 smithi181 ceph-mon[94189]: pgmap v7892: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 224 KiB/s wr, 11 op/s 2023-03-09T18:48:47.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:48:47.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:48:47.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:48:47.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:48:47.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:48:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13838.439745 secs 2023-03-09T18:48:48.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T18:48:48.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:48:48.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13838.439745 secs 2023-03-09T18:48:48.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T18:48:48.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:48:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13838.439745 secs 2023-03-09T18:48:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T18:48:48.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:48:48.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14300.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:48:49.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:49 smithi122 ceph-mon[93869]: pgmap v7893: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 178 KiB/s wr, 9 op/s 2023-03-09T18:48:49.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:49 smithi181 ceph-mon[94189]: pgmap v7893: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 178 KiB/s wr, 9 op/s 2023-03-09T18:48:49.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:49 smithi038 ceph-mon[92422]: pgmap v7893: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 178 KiB/s wr, 9 op/s 2023-03-09T18:48:50.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13841.162226 secs 2023-03-09T18:48:50.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13841.162226 secs 2023-03-09T18:48:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13841.162226 secs 2023-03-09T18:48:51.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:51 smithi122 ceph-mon[93869]: pgmap v7894: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 200 KiB/s wr, 10 op/s 2023-03-09T18:48:51.519 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:51 smithi038 ceph-mon[92422]: pgmap v7894: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 200 KiB/s wr, 10 op/s 2023-03-09T18:48:51.519 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:51 smithi181 ceph-mon[94189]: pgmap v7894: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 200 KiB/s wr, 10 op/s 2023-03-09T18:48:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13843.439862 secs 2023-03-09T18:48:53.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:53 smithi122 ceph-mon[93869]: pgmap v7895: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 430 KiB/s rd, 427 KiB/s wr, 13 op/s 2023-03-09T18:48:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13843.439862 secs 2023-03-09T18:48:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:53 smithi181 ceph-mon[94189]: pgmap v7895: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 430 KiB/s rd, 427 KiB/s wr, 13 op/s 2023-03-09T18:48:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13843.439862 secs 2023-03-09T18:48:53.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:53 smithi038 ceph-mon[92422]: pgmap v7895: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 430 KiB/s rd, 427 KiB/s wr, 13 op/s 2023-03-09T18:48:53.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14305.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:48:55.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:55 smithi122 ceph-mon[93869]: pgmap v7896: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 301 KiB/s wr, 10 op/s 2023-03-09T18:48:55.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13846.162284 secs 2023-03-09T18:48:55.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:55 smithi181 ceph-mon[94189]: pgmap v7896: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 301 KiB/s wr, 10 op/s 2023-03-09T18:48:55.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13846.162284 secs 2023-03-09T18:48:55.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:55 smithi038 ceph-mon[92422]: pgmap v7896: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 301 KiB/s wr, 10 op/s 2023-03-09T18:48:55.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13846.162284 secs 2023-03-09T18:48:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T18:48:56.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:48:56.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T18:48:56.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:48:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T18:48:56.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:48:57.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:57 smithi122 ceph-mon[93869]: pgmap v7897: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 347 KiB/s wr, 12 op/s 2023-03-09T18:48:57.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T18:48:57.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:48:57.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:57 smithi181 ceph-mon[94189]: pgmap v7897: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 347 KiB/s wr, 12 op/s 2023-03-09T18:48:57.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T18:48:57.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:48:57.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:57 smithi038 ceph-mon[92422]: pgmap v7897: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 347 KiB/s wr, 12 op/s 2023-03-09T18:48:57.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T18:48:57.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:48:58.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13848.439981 secs 2023-03-09T18:48:58.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13848.439981 secs 2023-03-09T18:48:58.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13848.439981 secs 2023-03-09T18:48:58.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14310.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:48:59.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:48:59 smithi122 ceph-mon[93869]: pgmap v7898: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 330 KiB/s wr, 10 op/s 2023-03-09T18:48:59.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:48:59 smithi181 ceph-mon[94189]: pgmap v7898: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 330 KiB/s wr, 10 op/s 2023-03-09T18:48:59.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:48:59 smithi038 ceph-mon[92422]: pgmap v7898: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 330 KiB/s wr, 10 op/s 2023-03-09T18:49:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:49:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13851.162344 secs 2023-03-09T18:49:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:49:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13851.162344 secs 2023-03-09T18:49:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:49:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13851.162344 secs 2023-03-09T18:49:01.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:49:01 smithi122 ceph-mon[93869]: pgmap v7899: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 330 KiB/s wr, 10 op/s 2023-03-09T18:49:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:49:01 smithi181 ceph-mon[94189]: pgmap v7899: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 330 KiB/s wr, 10 op/s 2023-03-09T18:49:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:49:01 smithi038 ceph-mon[92422]: pgmap v7899: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 330 KiB/s wr, 10 op/s 2023-03-09T18:49:03.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:49:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13853.440096 secs 2023-03-09T18:49:03.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:49:03 smithi122 ceph-mon[93869]: pgmap v7900: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 416 KiB/s rd, 425 KiB/s wr, 15 op/s 2023-03-09T18:49:03.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:49:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13853.440096 secs 2023-03-09T18:49:03.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:49:03 smithi181 ceph-mon[94189]: pgmap v7900: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 416 KiB/s rd, 425 KiB/s wr, 15 op/s 2023-03-09T18:49:03.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:49:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13853.440096 secs 2023-03-09T18:49:03.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:49:03 smithi038 ceph-mon[92422]: pgmap v7900: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 416 KiB/s rd, 425 KiB/s wr, 15 op/s 2023-03-09T18:49:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14315.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:49:05.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:49:05 smithi122 ceph-mon[93869]: pgmap v7901: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 198 KiB/s wr, 23 op/s 2023-03-09T18:49:05.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:49:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13856.162401 secs 2023-03-09T18:49:05.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:49:05 smithi181 ceph-mon[94189]: pgmap v7901: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 198 KiB/s wr, 23 op/s 2023-03-09T18:49:05.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:49:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13856.162401 secs 2023-03-09T18:49:05.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:49:05 smithi038 ceph-mon[92422]: pgmap v7901: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 198 KiB/s wr, 23 op/s 2023-03-09T18:49:05.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:49:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13856.162401 secs 2023-03-09T18:49:07.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:49:07 smithi122 ceph-mon[93869]: pgmap v7902: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 187 KiB/s wr, 25 op/s 2023-03-09T18:49:07.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:49:07 smithi181 ceph-mon[94189]: pgmap v7902: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 187 KiB/s wr, 25 op/s 2023-03-09T18:49:07.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:49:07 smithi038 ceph-mon[92422]: pgmap v7902: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 187 KiB/s wr, 25 op/s 2023-03-09T18:49:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:49:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13858.440214 secs 2023-03-09T18:49:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:49:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13858.440214 secs 2023-03-09T18:49:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:49:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13858.440214 secs 2023-03-09T18:49:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14320.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:49:09.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:49:09 smithi122 ceph-mon[93869]: pgmap v7903: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 140 KiB/s wr, 22 op/s 2023-03-09T18:49:09.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:49:09 smithi181 ceph-mon[94189]: pgmap v7903: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 140 KiB/s wr, 22 op/s 2023-03-09T18:49:09.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:49:09 smithi038 ceph-mon[92422]: pgmap v7903: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 140 KiB/s wr, 22 op/s 2023-03-09T18:49:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:49:10 smithi122 ceph-mon[93869]: osdmap e707: 12 total, 12 up, 12 in 2023-03-09T18:49:10.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:49:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13861.162461 secs 2023-03-09T18:49:10.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:49:10 smithi181 ceph-mon[94189]: osdmap e707: 12 total, 12 up, 12 in 2023-03-09T18:49:10.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:49:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13861.162461 secs 2023-03-09T18:49:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:49:10 smithi038 ceph-mon[92422]: osdmap e707: 12 total, 12 up, 12 in 2023-03-09T18:49:10.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:49:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13861.162461 secs 2023-03-09T18:49:11.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:49:11 smithi122 ceph-mon[93869]: pgmap v7905: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 168 KiB/s wr, 25 op/s 2023-03-09T18:49:11.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:49:11 smithi181 ceph-mon[94189]: pgmap v7905: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 168 KiB/s wr, 25 op/s 2023-03-09T18:49:11.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:49:11 smithi038 ceph-mon[92422]: pgmap v7905: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 168 KiB/s wr, 25 op/s 2023-03-09T18:49:13.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:49:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13863.440361 secs 2023-03-09T18:49:13.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:49:13 smithi122 ceph-mon[93869]: pgmap v7906: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 69 KiB/s wr, 21 op/s 2023-03-09T18:49:13.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:49:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13863.440361 secs 2023-03-09T18:49:13.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:49:13 smithi181 ceph-mon[94189]: pgmap v7906: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 69 KiB/s wr, 21 op/s 2023-03-09T18:49:13.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:49:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13863.440361 secs 2023-03-09T18:49:13.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:49:13 smithi038 ceph-mon[92422]: pgmap v7906: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 69 KiB/s wr, 21 op/s 2023-03-09T18:49:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14325.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:49:15.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:49:15 smithi122 ceph-mon[93869]: pgmap v7907: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 43 KiB/s wr, 6 op/s 2023-03-09T18:49:15.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:49:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13866.162520 secs 2023-03-09T18:49:15.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:49:15 smithi181 ceph-mon[94189]: pgmap v7907: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 43 KiB/s wr, 6 op/s 2023-03-09T18:49:15.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:49:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13866.162520 secs 2023-03-09T18:49:15.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:49:15 smithi038 ceph-mon[92422]: pgmap v7907: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 43 KiB/s wr, 6 op/s 2023-03-09T18:49:15.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:49:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13866.162520 secs 2023-03-09T18:49:17.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:49:17 smithi122 ceph-mon[93869]: pgmap v7908: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 43 KiB/s wr, 6 op/s 2023-03-09T18:49:17.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:49:17 smithi122 ceph-mon[93869]: osdmap e708: 12 total, 12 up, 12 in 2023-03-09T18:49:17.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:49:17 smithi181 ceph-mon[94189]: pgmap v7908: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 43 KiB/s wr, 6 op/s 2023-03-09T18:49:17.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:49:17 smithi181 ceph-mon[94189]: osdmap e708: 12 total, 12 up, 12 in 2023-03-09T18:49:17.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:49:17 smithi038 ceph-mon[92422]: pgmap v7908: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 43 KiB/s wr, 6 op/s 2023-03-09T18:49:17.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:49:17 smithi038 ceph-mon[92422]: osdmap e708: 12 total, 12 up, 12 in 2023-03-09T18:49:18.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:49:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13868.440490 secs 2023-03-09T18:49:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:49:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13868.440490 secs 2023-03-09T18:49:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:49:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13868.440490 secs 2023-03-09T18:49:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14330.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:49:19.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:49:19 smithi122 ceph-mon[93869]: pgmap v7910: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 48 KiB/s wr, 6 op/s 2023-03-09T18:49:19.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:49:19 smithi181 ceph-mon[94189]: pgmap v7910: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 48 KiB/s wr, 6 op/s 2023-03-09T18:49:19.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:49:19 smithi038 ceph-mon[92422]: pgmap v7910: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 48 KiB/s wr, 6 op/s 2023-03-09T18:49:20.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:49:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13871.162590 secs 2023-03-09T18:49:20.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:49:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13871.162590 secs 2023-03-09T18:49:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:49:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13871.162590 secs 2023-03-09T18:49:21.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:49:21 smithi122 ceph-mon[93869]: pgmap v7911: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 43 KiB/s wr, 6 op/s 2023-03-09T18:49:21.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:49:21 smithi181 ceph-mon[94189]: pgmap v7911: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 43 KiB/s wr, 6 op/s 2023-03-09T18:49:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:49:21 smithi038 ceph-mon[92422]: pgmap v7911: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 43 KiB/s wr, 6 op/s 2023-03-09T18:49:23.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:49:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13873.440562 secs 2023-03-09T18:49:23.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:49:23 smithi122 ceph-mon[93869]: pgmap v7912: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 409 B/s wr, 6 op/s 2023-03-09T18:49:23.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:49:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13873.440562 secs 2023-03-09T18:49:23.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:49:23 smithi181 ceph-mon[94189]: pgmap v7912: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 409 B/s wr, 6 op/s 2023-03-09T18:49:23.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:49:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13873.440562 secs 2023-03-09T18:49:23.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:49:23 smithi038 ceph-mon[92422]: pgmap v7912: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 409 B/s wr, 6 op/s 2023-03-09T18:49:23.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14335.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:49:25.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:49:25 smithi122 ceph-mon[93869]: pgmap v7913: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 614 B/s wr, 7 op/s 2023-03-09T18:49:25.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:49:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13876.162651 secs 2023-03-09T18:49:25.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:49:25 smithi181 ceph-mon[94189]: pgmap v7913: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 614 B/s wr, 7 op/s 2023-03-09T18:49:25.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:49:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13876.162651 secs 2023-03-09T18:49:25.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:49:25 smithi038 ceph-mon[92422]: pgmap v7913: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 614 B/s wr, 7 op/s 2023-03-09T18:49:25.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:49:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13876.162651 secs 2023-03-09T18:49:27.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:49:27 smithi122 ceph-mon[93869]: pgmap v7914: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 614 B/s wr, 7 op/s 2023-03-09T18:49:27.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:49:27 smithi181 ceph-mon[94189]: pgmap v7914: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 614 B/s wr, 7 op/s 2023-03-09T18:49:27.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:49:27 smithi038 ceph-mon[92422]: pgmap v7914: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 614 B/s wr, 7 op/s 2023-03-09T18:49:28.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:49:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13878.440681 secs 2023-03-09T18:49:28.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:49:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13878.440681 secs 2023-03-09T18:49:28.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:49:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13878.440681 secs 2023-03-09T18:49:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14340.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:49:29.481 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:49:29 smithi122 ceph-mon[93869]: pgmap v7915: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 652 B/s wr, 6 op/s 2023-03-09T18:49:29.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:49:29 smithi181 ceph-mon[94189]: pgmap v7915: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 652 B/s wr, 6 op/s 2023-03-09T18:49:29.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:49:29 smithi038 ceph-mon[92422]: pgmap v7915: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 652 B/s wr, 6 op/s 2023-03-09T18:49:30.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:49:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13881.162708 secs 2023-03-09T18:49:30.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:49:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13881.162708 secs 2023-03-09T18:49:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:49:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13881.162708 secs 2023-03-09T18:49:31.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:49:31 smithi122 ceph-mon[93869]: pgmap v7916: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 767 B/s wr, 6 op/s 2023-03-09T18:49:31.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:49:31 smithi181 ceph-mon[94189]: pgmap v7916: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 767 B/s wr, 6 op/s 2023-03-09T18:49:31.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:49:31 smithi038 ceph-mon[92422]: pgmap v7916: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 767 B/s wr, 6 op/s 2023-03-09T18:49:33.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:49:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13883.440774 secs 2023-03-09T18:49:33.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:49:33 smithi122 ceph-mon[93869]: pgmap v7917: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 46 KiB/s wr, 8 op/s 2023-03-09T18:49:33.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:49:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13883.440774 secs 2023-03-09T18:49:33.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:49:33 smithi181 ceph-mon[94189]: pgmap v7917: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 46 KiB/s wr, 8 op/s 2023-03-09T18:49:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:49:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13883.440774 secs 2023-03-09T18:49:33.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:49:33 smithi038 ceph-mon[92422]: pgmap v7917: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 46 KiB/s wr, 8 op/s 2023-03-09T18:49:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14345.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:49:35.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:49:35 smithi122 ceph-mon[93869]: pgmap v7918: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 45 KiB/s wr, 7 op/s 2023-03-09T18:49:35.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:49:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13886.162794 secs 2023-03-09T18:49:35.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:49:35 smithi181 ceph-mon[94189]: pgmap v7918: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 45 KiB/s wr, 7 op/s 2023-03-09T18:49:35.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:49:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13886.162794 secs 2023-03-09T18:49:35.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:49:35 smithi038 ceph-mon[92422]: pgmap v7918: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 45 KiB/s wr, 7 op/s 2023-03-09T18:49:35.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:49:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13886.162794 secs 2023-03-09T18:49:37.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:49:37 smithi122 ceph-mon[93869]: pgmap v7919: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 KiB/s wr, 8 op/s 2023-03-09T18:49:37.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:49:37 smithi181 ceph-mon[94189]: pgmap v7919: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 KiB/s wr, 8 op/s 2023-03-09T18:49:37.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:49:37 smithi038 ceph-mon[92422]: pgmap v7919: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 KiB/s wr, 8 op/s 2023-03-09T18:49:38.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:49:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13888.440914 secs 2023-03-09T18:49:38.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:49:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13888.440914 secs 2023-03-09T18:49:38.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:49:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13888.440914 secs 2023-03-09T18:49:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14350.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:49:39.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:49:39 smithi122 ceph-mon[93869]: pgmap v7920: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 92 KiB/s wr, 6 op/s 2023-03-09T18:49:39.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:49:39 smithi181 ceph-mon[94189]: pgmap v7920: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 92 KiB/s wr, 6 op/s 2023-03-09T18:49:39.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:49:39 smithi038 ceph-mon[92422]: pgmap v7920: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 92 KiB/s wr, 6 op/s 2023-03-09T18:49:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:49:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13891.162869 secs 2023-03-09T18:49:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:49:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13891.162869 secs 2023-03-09T18:49:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:49:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13891.162869 secs 2023-03-09T18:49:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:49:41 smithi122 ceph-mon[93869]: pgmap v7921: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 158 KiB/s wr, 7 op/s 2023-03-09T18:49:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:49:41 smithi181 ceph-mon[94189]: pgmap v7921: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 158 KiB/s wr, 7 op/s 2023-03-09T18:49:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:49:41 smithi038 ceph-mon[92422]: pgmap v7921: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 158 KiB/s wr, 7 op/s 2023-03-09T18:49:43.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:49:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13893.441048 secs 2023-03-09T18:49:43.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:49:43 smithi122 ceph-mon[93869]: pgmap v7922: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 226 KiB/s wr, 9 op/s 2023-03-09T18:49:43.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:49:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13893.441048 secs 2023-03-09T18:49:43.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:49:43 smithi181 ceph-mon[94189]: pgmap v7922: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 226 KiB/s wr, 9 op/s 2023-03-09T18:49:43.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:49:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13893.441048 secs 2023-03-09T18:49:43.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:49:43 smithi038 ceph-mon[92422]: pgmap v7922: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 226 KiB/s wr, 9 op/s 2023-03-09T18:49:43.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14355.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:49:45.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:49:45 smithi122 ceph-mon[93869]: pgmap v7923: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 209 KiB/s wr, 8 op/s 2023-03-09T18:49:45.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:49:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13896.162928 secs 2023-03-09T18:49:45.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:49:45 smithi181 ceph-mon[94189]: pgmap v7923: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 209 KiB/s wr, 8 op/s 2023-03-09T18:49:45.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:49:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13896.162928 secs 2023-03-09T18:49:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:49:45 smithi038 ceph-mon[92422]: pgmap v7923: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 209 KiB/s wr, 8 op/s 2023-03-09T18:49:45.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:49:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13896.162928 secs 2023-03-09T18:49:47.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:49:47 smithi122 ceph-mon[93869]: pgmap v7924: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 209 KiB/s wr, 10 op/s 2023-03-09T18:49:47.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:49:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:49:47.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:49:47 smithi181 ceph-mon[94189]: pgmap v7924: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 209 KiB/s wr, 10 op/s 2023-03-09T18:49:47.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:49:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:49:47.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:49:47 smithi038 ceph-mon[92422]: pgmap v7924: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 209 KiB/s wr, 10 op/s 2023-03-09T18:49:47.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:49:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:49:48.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:49:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T18:49:48.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:49:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:49:48.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:49:48 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:49:48.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:49:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T18:49:48.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:49:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:49:48.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:49:48 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:49:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:49:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T18:49:48.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:49:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:49:48.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:49:48 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:49:48.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14360.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:49:49.392 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:49:49 smithi181 ceph-mon[94189]: pgmap v7925: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 164 KiB/s wr, 9 op/s 2023-03-09T18:49:49.392 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:49:49 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:49:49.392 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:49:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T18:49:49.392 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:49:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:49:49.392 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:49:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:49:49.431 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:49:49 smithi122 ceph-mon[93869]: pgmap v7925: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 164 KiB/s wr, 9 op/s 2023-03-09T18:49:49.431 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:49:49 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:49:49.431 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:49:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T18:49:49.432 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:49:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:49:49.432 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:49:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:49:49.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:49:49 smithi038 ceph-mon[92422]: pgmap v7925: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 164 KiB/s wr, 9 op/s 2023-03-09T18:49:49.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:49:49 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:49:49.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:49:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T18:49:49.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:49:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:49:49.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:49:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:49:50.467 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:49:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13901.162987 secs 2023-03-09T18:49:50.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:49:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13901.162987 secs 2023-03-09T18:49:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:49:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13901.162987 secs 2023-03-09T18:49:52.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:49:51 smithi122 ceph-mon[93869]: pgmap v7926: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 329 KiB/s rd, 352 KiB/s wr, 11 op/s 2023-03-09T18:49:52.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:49:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:49:52.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:49:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:49:52.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:49:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:49:52.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:49:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:49:52.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:49:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:49:52.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:49:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:49:52.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:49:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:49:52.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:49:51 smithi181 ceph-mon[94189]: pgmap v7926: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 329 KiB/s rd, 352 KiB/s wr, 11 op/s 2023-03-09T18:49:52.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:49:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:49:52.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:49:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:49:52.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:49:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:49:52.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:49:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:49:52.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:49:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:49:52.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:49:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:49:52.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:49:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:49:52.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:49:51 smithi038 ceph-mon[92422]: pgmap v7926: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 329 KiB/s rd, 352 KiB/s wr, 11 op/s 2023-03-09T18:49:52.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:49:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:49:52.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:49:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:49:52.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:49:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:49:52.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:49:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:49:52.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:49:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:49:52.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:49:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:49:52.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:49:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:49:53.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14365.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:49:54.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:49:53 smithi122 ceph-mon[93869]: pgmap v7927: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 286 KiB/s wr, 13 op/s 2023-03-09T18:49:54.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:49:53 smithi181 ceph-mon[94189]: pgmap v7927: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 286 KiB/s wr, 13 op/s 2023-03-09T18:49:54.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:49:53 smithi038 ceph-mon[92422]: pgmap v7927: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 286 KiB/s wr, 13 op/s 2023-03-09T18:49:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:49:55 smithi122 ceph-mon[93869]: pgmap v7928: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 261 KiB/s wr, 12 op/s 2023-03-09T18:49:56.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:49:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13906.163049 secs 2023-03-09T18:49:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:49:55 smithi181 ceph-mon[94189]: pgmap v7928: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 261 KiB/s wr, 12 op/s 2023-03-09T18:49:56.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:49:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13906.163049 secs 2023-03-09T18:49:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:49:55 smithi038 ceph-mon[92422]: pgmap v7928: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 261 KiB/s wr, 12 op/s 2023-03-09T18:49:56.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:49:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13906.163049 secs 2023-03-09T18:49:58.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:49:57 smithi122 ceph-mon[93869]: pgmap v7929: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 233 KiB/s wr, 12 op/s 2023-03-09T18:49:58.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:49:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13908.441396 secs 2023-03-09T18:49:58.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:49:57 smithi181 ceph-mon[94189]: pgmap v7929: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 233 KiB/s wr, 12 op/s 2023-03-09T18:49:58.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:49:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13908.441396 secs 2023-03-09T18:49:58.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:49:57 smithi038 ceph-mon[92422]: pgmap v7929: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 233 KiB/s wr, 12 op/s 2023-03-09T18:49:58.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:49:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13908.441396 secs 2023-03-09T18:49:58.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14370.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:50:00.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:49:59 smithi122 ceph-mon[93869]: pgmap v7930: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 233 KiB/s wr, 10 op/s 2023-03-09T18:50:00.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:49:59 smithi181 ceph-mon[94189]: pgmap v7930: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 233 KiB/s wr, 10 op/s 2023-03-09T18:50:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:49:59 smithi038 ceph-mon[92422]: pgmap v7930: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 233 KiB/s wr, 10 op/s 2023-03-09T18:50:01.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13911.163110 secs 2023-03-09T18:50:01.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:00 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:50:01.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:50:01.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:50:01.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:00 smithi122 ceph-mon[93869]: Health detail: HEALTH_WARN 2 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-09T18:50:01.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:00 smithi122 ceph-mon[93869]: [WRN] MDS_SLOW_REQUEST: 2 MDSs report slow requests 2023-03-09T18:50:01.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:00 smithi122 ceph-mon[93869]: mds.i(mds.2): 2 slow requests are blocked > 180 secs 2023-03-09T18:50:01.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:00 smithi122 ceph-mon[93869]: mds.h(mds.1): 2 slow requests are blocked > 180 secs 2023-03-09T18:50:01.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:00 smithi122 ceph-mon[93869]: [WRN] MDS_TRIM: 2 MDSs behind on trimming 2023-03-09T18:50:01.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:00 smithi122 ceph-mon[93869]: mds.e(mds.0): Behind on trimming (268/128) max_segments: 128, num_segments: 268 2023-03-09T18:50:01.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:00 smithi122 ceph-mon[93869]: mds.b(mds.0): Behind on trimming (268/128) max_segments: 128, num_segments: 268 2023-03-09T18:50:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13911.163110 secs 2023-03-09T18:50:01.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:00 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:50:01.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:50:01.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:50:01.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:00 smithi181 ceph-mon[94189]: Health detail: HEALTH_WARN 2 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-09T18:50:01.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:00 smithi181 ceph-mon[94189]: [WRN] MDS_SLOW_REQUEST: 2 MDSs report slow requests 2023-03-09T18:50:01.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:00 smithi181 ceph-mon[94189]: mds.i(mds.2): 2 slow requests are blocked > 180 secs 2023-03-09T18:50:01.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:00 smithi181 ceph-mon[94189]: mds.h(mds.1): 2 slow requests are blocked > 180 secs 2023-03-09T18:50:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:00 smithi181 ceph-mon[94189]: [WRN] MDS_TRIM: 2 MDSs behind on trimming 2023-03-09T18:50:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:00 smithi181 ceph-mon[94189]: mds.e(mds.0): Behind on trimming (268/128) max_segments: 128, num_segments: 268 2023-03-09T18:50:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:00 smithi181 ceph-mon[94189]: mds.b(mds.0): Behind on trimming (268/128) max_segments: 128, num_segments: 268 2023-03-09T18:50:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13911.163110 secs 2023-03-09T18:50:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:00 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:50:01.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:50:01.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:50:01.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:00 smithi038 ceph-mon[92422]: Health detail: HEALTH_WARN 2 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-09T18:50:01.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:00 smithi038 ceph-mon[92422]: [WRN] MDS_SLOW_REQUEST: 2 MDSs report slow requests 2023-03-09T18:50:01.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:00 smithi038 ceph-mon[92422]: mds.i(mds.2): 2 slow requests are blocked > 180 secs 2023-03-09T18:50:01.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:00 smithi038 ceph-mon[92422]: mds.h(mds.1): 2 slow requests are blocked > 180 secs 2023-03-09T18:50:01.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:00 smithi038 ceph-mon[92422]: [WRN] MDS_TRIM: 2 MDSs behind on trimming 2023-03-09T18:50:01.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:00 smithi038 ceph-mon[92422]: mds.e(mds.0): Behind on trimming (268/128) max_segments: 128, num_segments: 268 2023-03-09T18:50:01.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:00 smithi038 ceph-mon[92422]: mds.b(mds.0): Behind on trimming (268/128) max_segments: 128, num_segments: 268 2023-03-09T18:50:02.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:01 smithi122 ceph-mon[93869]: pgmap v7931: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 306 KiB/s wr, 11 op/s 2023-03-09T18:50:02.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:01 smithi181 ceph-mon[94189]: pgmap v7931: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 306 KiB/s wr, 11 op/s 2023-03-09T18:50:02.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:01 smithi038 ceph-mon[92422]: pgmap v7931: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 306 KiB/s wr, 11 op/s 2023-03-09T18:50:03.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13913.441522 secs 2023-03-09T18:50:03.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:03 smithi122 ceph-mon[93869]: pgmap v7932: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 168 KiB/s wr, 13 op/s 2023-03-09T18:50:03.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13913.441522 secs 2023-03-09T18:50:03.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:03 smithi181 ceph-mon[94189]: pgmap v7932: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 168 KiB/s wr, 13 op/s 2023-03-09T18:50:03.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13913.441522 secs 2023-03-09T18:50:03.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:03 smithi038 ceph-mon[92422]: pgmap v7932: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 168 KiB/s wr, 13 op/s 2023-03-09T18:50:03.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14375.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:50:05.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:05 smithi122 ceph-mon[93869]: pgmap v7933: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 190 KiB/s wr, 26 op/s 2023-03-09T18:50:05.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13916.163170 secs 2023-03-09T18:50:05.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:05 smithi181 ceph-mon[94189]: pgmap v7933: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 190 KiB/s wr, 26 op/s 2023-03-09T18:50:05.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13916.163170 secs 2023-03-09T18:50:05.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:05 smithi038 ceph-mon[92422]: pgmap v7933: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 190 KiB/s wr, 26 op/s 2023-03-09T18:50:05.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13916.163170 secs 2023-03-09T18:50:07.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:07 smithi122 ceph-mon[93869]: pgmap v7934: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 147 KiB/s wr, 31 op/s 2023-03-09T18:50:07.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13918.441671 secs 2023-03-09T18:50:07.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:07 smithi181 ceph-mon[94189]: pgmap v7934: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 147 KiB/s wr, 31 op/s 2023-03-09T18:50:07.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13918.441671 secs 2023-03-09T18:50:07.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:07 smithi038 ceph-mon[92422]: pgmap v7934: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 147 KiB/s wr, 31 op/s 2023-03-09T18:50:07.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13918.441671 secs 2023-03-09T18:50:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T18:50:08.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:50:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T18:50:08.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:50:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T18:50:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:50:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14380.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:50:09.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:09 smithi122 ceph-mon[93869]: pgmap v7935: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 177 KiB/s wr, 34 op/s 2023-03-09T18:50:09.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T18:50:09.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:50:09.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:09 smithi181 ceph-mon[94189]: pgmap v7935: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 177 KiB/s wr, 34 op/s 2023-03-09T18:50:09.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T18:50:09.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:50:09.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:09 smithi038 ceph-mon[92422]: pgmap v7935: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 177 KiB/s wr, 34 op/s 2023-03-09T18:50:09.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T18:50:09.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:50:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:10 smithi122 ceph-mon[93869]: osdmap e709: 12 total, 12 up, 12 in 2023-03-09T18:50:10.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13921.163230 secs 2023-03-09T18:50:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:10 smithi181 ceph-mon[94189]: osdmap e709: 12 total, 12 up, 12 in 2023-03-09T18:50:10.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13921.163230 secs 2023-03-09T18:50:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:10 smithi038 ceph-mon[92422]: osdmap e709: 12 total, 12 up, 12 in 2023-03-09T18:50:10.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13921.163230 secs 2023-03-09T18:50:11.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:11 smithi122 ceph-mon[93869]: pgmap v7937: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 123 KiB/s wr, 39 op/s 2023-03-09T18:50:11.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:11 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): Behind on trimming (269/128) 2023-03-09T18:50:11.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:11 smithi181 ceph-mon[94189]: pgmap v7937: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 123 KiB/s wr, 39 op/s 2023-03-09T18:50:11.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:11 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): Behind on trimming (269/128) 2023-03-09T18:50:11.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:11 smithi038 ceph-mon[92422]: pgmap v7937: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 123 KiB/s wr, 39 op/s 2023-03-09T18:50:11.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:11 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): Behind on trimming (269/128) 2023-03-09T18:50:12.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:12 smithi122 ceph-mon[93869]: Health check update: 1 MDSs behind on trimming (MDS_TRIM) 2023-03-09T18:50:12.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T18:50:12.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:50:12.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:12 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): Behind on trimming (269/128) 2023-03-09T18:50:12.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13923.441791 secs 2023-03-09T18:50:12.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:12 smithi181 ceph-mon[94189]: Health check update: 1 MDSs behind on trimming (MDS_TRIM) 2023-03-09T18:50:12.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T18:50:12.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:50:12.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:12 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): Behind on trimming (269/128) 2023-03-09T18:50:12.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13923.441791 secs 2023-03-09T18:50:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:12 smithi038 ceph-mon[92422]: Health check update: 1 MDSs behind on trimming (MDS_TRIM) 2023-03-09T18:50:12.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T18:50:12.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:50:12.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:12 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): Behind on trimming (269/128) 2023-03-09T18:50:12.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13923.441791 secs 2023-03-09T18:50:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:13 smithi038 ceph-mon[92422]: pgmap v7938: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 63 KiB/s wr, 36 op/s 2023-03-09T18:50:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:13 smithi038 ceph-mon[92422]: Health check cleared: MDS_TRIM (was: 1 MDSs behind on trimming) 2023-03-09T18:50:13.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T18:50:13.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:50:13.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:13 smithi122 ceph-mon[93869]: pgmap v7938: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 63 KiB/s wr, 36 op/s 2023-03-09T18:50:13.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:13 smithi122 ceph-mon[93869]: Health check cleared: MDS_TRIM (was: 1 MDSs behind on trimming) 2023-03-09T18:50:13.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T18:50:13.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:50:13.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:13 smithi181 ceph-mon[94189]: pgmap v7938: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 63 KiB/s wr, 36 op/s 2023-03-09T18:50:13.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:13 smithi181 ceph-mon[94189]: Health check cleared: MDS_TRIM (was: 1 MDSs behind on trimming) 2023-03-09T18:50:13.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T18:50:13.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:50:13.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14385.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:50:15.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:15 smithi038 ceph-mon[92422]: pgmap v7939: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 37 KiB/s wr, 18 op/s 2023-03-09T18:50:15.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13926.163289 secs 2023-03-09T18:50:15.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:15 smithi122 ceph-mon[93869]: pgmap v7939: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 37 KiB/s wr, 18 op/s 2023-03-09T18:50:15.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13926.163289 secs 2023-03-09T18:50:15.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:15 smithi181 ceph-mon[94189]: pgmap v7939: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 37 KiB/s wr, 18 op/s 2023-03-09T18:50:15.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13926.163289 secs 2023-03-09T18:50:17.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:17 smithi038 ceph-mon[92422]: pgmap v7940: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 37 KiB/s wr, 11 op/s 2023-03-09T18:50:17.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13928.441905 secs 2023-03-09T18:50:17.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:17 smithi038 ceph-mon[92422]: osdmap e710: 12 total, 12 up, 12 in 2023-03-09T18:50:17.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:17 smithi122 ceph-mon[93869]: pgmap v7940: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 37 KiB/s wr, 11 op/s 2023-03-09T18:50:17.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13928.441905 secs 2023-03-09T18:50:17.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:17 smithi122 ceph-mon[93869]: osdmap e710: 12 total, 12 up, 12 in 2023-03-09T18:50:17.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:17 smithi181 ceph-mon[94189]: pgmap v7940: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 37 KiB/s wr, 11 op/s 2023-03-09T18:50:17.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13928.441905 secs 2023-03-09T18:50:17.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:17 smithi181 ceph-mon[94189]: osdmap e710: 12 total, 12 up, 12 in 2023-03-09T18:50:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14390.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:50:19.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:19 smithi038 ceph-mon[92422]: pgmap v7942: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 228 B/s wr, 6 op/s 2023-03-09T18:50:19.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:19 smithi122 ceph-mon[93869]: pgmap v7942: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 228 B/s wr, 6 op/s 2023-03-09T18:50:19.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:19 smithi181 ceph-mon[94189]: pgmap v7942: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 228 B/s wr, 6 op/s 2023-03-09T18:50:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13931.163363 secs 2023-03-09T18:50:20.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13931.163363 secs 2023-03-09T18:50:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13931.163363 secs 2023-03-09T18:50:21.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:21 smithi122 ceph-mon[93869]: pgmap v7943: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T18:50:21.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:21 smithi181 ceph-mon[94189]: pgmap v7943: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T18:50:21.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:21 smithi038 ceph-mon[92422]: pgmap v7943: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T18:50:23.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13933.442022 secs 2023-03-09T18:50:23.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:23 smithi122 ceph-mon[93869]: pgmap v7944: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T18:50:23.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13933.442022 secs 2023-03-09T18:50:23.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:23 smithi181 ceph-mon[94189]: pgmap v7944: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T18:50:23.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13933.442022 secs 2023-03-09T18:50:23.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:23 smithi038 ceph-mon[92422]: pgmap v7944: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T18:50:23.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14395.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:50:25.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:25 smithi122 ceph-mon[93869]: pgmap v7945: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T18:50:25.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13936.163422 secs 2023-03-09T18:50:25.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:25 smithi181 ceph-mon[94189]: pgmap v7945: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T18:50:25.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13936.163422 secs 2023-03-09T18:50:25.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:25 smithi038 ceph-mon[92422]: pgmap v7945: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T18:50:25.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13936.163422 secs 2023-03-09T18:50:27.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:27 smithi122 ceph-mon[93869]: pgmap v7946: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T18:50:27.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13938.442157 secs 2023-03-09T18:50:27.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:27 smithi181 ceph-mon[94189]: pgmap v7946: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T18:50:27.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13938.442157 secs 2023-03-09T18:50:27.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:27 smithi038 ceph-mon[92422]: pgmap v7946: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T18:50:27.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13938.442157 secs 2023-03-09T18:50:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14400.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:50:29.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:29 smithi122 ceph-mon[93869]: pgmap v7947: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T18:50:29.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:29 smithi181 ceph-mon[94189]: pgmap v7947: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T18:50:29.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:29 smithi038 ceph-mon[92422]: pgmap v7947: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T18:50:30.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13941.163482 secs 2023-03-09T18:50:30.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13941.163482 secs 2023-03-09T18:50:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13941.163482 secs 2023-03-09T18:50:31.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:31 smithi181 ceph-mon[94189]: pgmap v7948: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2023-03-09T18:50:31.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:31 smithi038 ceph-mon[92422]: pgmap v7948: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2023-03-09T18:50:31.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:31 smithi122 ceph-mon[93869]: pgmap v7948: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2023-03-09T18:50:33.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13943.442231 secs 2023-03-09T18:50:33.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:33 smithi122 ceph-mon[93869]: pgmap v7949: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 36 KiB/s wr, 8 op/s 2023-03-09T18:50:33.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13943.442231 secs 2023-03-09T18:50:33.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:33 smithi181 ceph-mon[94189]: pgmap v7949: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 36 KiB/s wr, 8 op/s 2023-03-09T18:50:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13943.442231 secs 2023-03-09T18:50:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:33 smithi038 ceph-mon[92422]: pgmap v7949: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 36 KiB/s wr, 8 op/s 2023-03-09T18:50:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14405.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:50:35.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:35 smithi122 ceph-mon[93869]: pgmap v7950: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 KiB/s wr, 6 op/s 2023-03-09T18:50:35.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13946.163553 secs 2023-03-09T18:50:35.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:35 smithi181 ceph-mon[94189]: pgmap v7950: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 KiB/s wr, 6 op/s 2023-03-09T18:50:35.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13946.163553 secs 2023-03-09T18:50:35.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:35 smithi038 ceph-mon[92422]: pgmap v7950: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 KiB/s wr, 6 op/s 2023-03-09T18:50:35.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13946.163553 secs 2023-03-09T18:50:37.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:37 smithi122 ceph-mon[93869]: pgmap v7951: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 36 KiB/s wr, 8 op/s 2023-03-09T18:50:37.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:37 smithi181 ceph-mon[94189]: pgmap v7951: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 36 KiB/s wr, 8 op/s 2023-03-09T18:50:37.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:37 smithi038 ceph-mon[92422]: pgmap v7951: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 36 KiB/s wr, 8 op/s 2023-03-09T18:50:38.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14410.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:50:39.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:39 smithi122 ceph-mon[93869]: pgmap v7952: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 93 KiB/s wr, 7 op/s 2023-03-09T18:50:39.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:39 smithi181 ceph-mon[94189]: pgmap v7952: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 93 KiB/s wr, 7 op/s 2023-03-09T18:50:39.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:39 smithi038 ceph-mon[92422]: pgmap v7952: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 93 KiB/s wr, 7 op/s 2023-03-09T18:50:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13951.163615 secs 2023-03-09T18:50:40.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:40 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:50:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13951.163615 secs 2023-03-09T18:50:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:40 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:50:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13951.163615 secs 2023-03-09T18:50:40.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:40 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:50:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:41 smithi122 ceph-mon[93869]: pgmap v7953: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 104 KiB/s wr, 7 op/s 2023-03-09T18:50:41.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:41 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:50:41.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:50:41.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:50:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:41 smithi181 ceph-mon[94189]: pgmap v7953: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 104 KiB/s wr, 7 op/s 2023-03-09T18:50:41.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:41 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:50:41.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:50:41.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:50:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:41 smithi038 ceph-mon[92422]: pgmap v7953: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 104 KiB/s wr, 7 op/s 2023-03-09T18:50:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:41 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:50:41.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:50:41.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:50:43.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:43 smithi122 ceph-mon[93869]: pgmap v7954: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 187 KiB/s wr, 9 op/s 2023-03-09T18:50:43.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:43 smithi181 ceph-mon[94189]: pgmap v7954: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 187 KiB/s wr, 9 op/s 2023-03-09T18:50:43.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:43 smithi038 ceph-mon[92422]: pgmap v7954: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 187 KiB/s wr, 9 op/s 2023-03-09T18:50:43.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14415.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:50:45.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:45 smithi181 ceph-mon[94189]: pgmap v7955: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 162 KiB/s wr, 8 op/s 2023-03-09T18:50:45.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13956.163672 secs 2023-03-09T18:50:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:45 smithi038 ceph-mon[92422]: pgmap v7955: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 162 KiB/s wr, 8 op/s 2023-03-09T18:50:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13956.163672 secs 2023-03-09T18:50:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:45 smithi122 ceph-mon[93869]: pgmap v7955: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 162 KiB/s wr, 8 op/s 2023-03-09T18:50:45.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13956.163672 secs 2023-03-09T18:50:47.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:47 smithi038 ceph-mon[92422]: pgmap v7956: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 162 KiB/s wr, 9 op/s 2023-03-09T18:50:47.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:47 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 13958.442535 secs 2023-03-09T18:50:47.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:47 smithi122 ceph-mon[93869]: pgmap v7956: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 162 KiB/s wr, 9 op/s 2023-03-09T18:50:47.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:47 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 13958.442535 secs 2023-03-09T18:50:47.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:47 smithi181 ceph-mon[94189]: pgmap v7956: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 162 KiB/s wr, 9 op/s 2023-03-09T18:50:47.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:47 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 13958.442535 secs 2023-03-09T18:50:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:48 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:50:48.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:50:48.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:50:48.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:48 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:50:48.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:50:48.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:50:48.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:48 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:50:48.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:50:48.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:50:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14420.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:50:49.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:49 smithi122 ceph-mon[93869]: pgmap v7957: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 193 KiB/s wr, 9 op/s 2023-03-09T18:50:49.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:49 smithi181 ceph-mon[94189]: pgmap v7957: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 193 KiB/s wr, 9 op/s 2023-03-09T18:50:49.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:49 smithi038 ceph-mon[92422]: pgmap v7957: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 193 KiB/s wr, 9 op/s 2023-03-09T18:50:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13961.163732 secs 2023-03-09T18:50:50.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13961.163732 secs 2023-03-09T18:50:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13961.163732 secs 2023-03-09T18:50:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:51 smithi122 ceph-mon[93869]: pgmap v7958: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 148 KiB/s wr, 9 op/s 2023-03-09T18:50:51.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:51 smithi181 ceph-mon[94189]: pgmap v7958: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 148 KiB/s wr, 9 op/s 2023-03-09T18:50:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:51 smithi038 ceph-mon[92422]: pgmap v7958: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 148 KiB/s wr, 9 op/s 2023-03-09T18:50:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:50:52.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:52 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 13963.442628 secs 2023-03-09T18:50:52.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:50:52.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:50:52.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:50:52.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:50:52.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:52 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 13963.442628 secs 2023-03-09T18:50:52.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:50:52.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:50:52.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:50:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:50:52.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:52 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 13963.442628 secs 2023-03-09T18:50:52.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:50:52.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:50:52.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:50:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:53 smithi122 ceph-mon[93869]: pgmap v7959: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 230 KiB/s wr, 11 op/s 2023-03-09T18:50:53.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:53 smithi181 ceph-mon[94189]: pgmap v7959: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 230 KiB/s wr, 11 op/s 2023-03-09T18:50:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:53 smithi038 ceph-mon[92422]: pgmap v7959: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 230 KiB/s wr, 11 op/s 2023-03-09T18:50:53.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14425.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:50:55.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:55 smithi122 ceph-mon[93869]: pgmap v7960: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 200 KiB/s wr, 10 op/s 2023-03-09T18:50:55.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13966.163791 secs 2023-03-09T18:50:55.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:55 smithi181 ceph-mon[94189]: pgmap v7960: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 200 KiB/s wr, 10 op/s 2023-03-09T18:50:55.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13966.163791 secs 2023-03-09T18:50:55.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:55 smithi038 ceph-mon[92422]: pgmap v7960: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 200 KiB/s wr, 10 op/s 2023-03-09T18:50:55.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13966.163791 secs 2023-03-09T18:50:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:57 smithi122 ceph-mon[93869]: pgmap v7961: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 189 KiB/s wr, 11 op/s 2023-03-09T18:50:57.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:57 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 13968.442746 secs 2023-03-09T18:50:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:57 smithi181 ceph-mon[94189]: pgmap v7961: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 189 KiB/s wr, 11 op/s 2023-03-09T18:50:57.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:57 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 13968.442746 secs 2023-03-09T18:50:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:57 smithi038 ceph-mon[92422]: pgmap v7961: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 189 KiB/s wr, 11 op/s 2023-03-09T18:50:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:57 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 13968.442746 secs 2023-03-09T18:50:58.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14430.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:50:59.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:50:59 smithi122 ceph-mon[93869]: pgmap v7962: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 318 KiB/s wr, 11 op/s 2023-03-09T18:50:59.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:50:59 smithi181 ceph-mon[94189]: pgmap v7962: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 318 KiB/s wr, 11 op/s 2023-03-09T18:50:59.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:50:59 smithi038 ceph-mon[92422]: pgmap v7962: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 318 KiB/s wr, 11 op/s 2023-03-09T18:51:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:51:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13971.163851 secs 2023-03-09T18:51:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:51:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13971.163851 secs 2023-03-09T18:51:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:51:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13971.163851 secs 2023-03-09T18:51:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:51:01 smithi122 ceph-mon[93869]: pgmap v7963: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 310 KiB/s rd, 309 KiB/s wr, 10 op/s 2023-03-09T18:51:01.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:51:01 smithi181 ceph-mon[94189]: pgmap v7963: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 310 KiB/s rd, 309 KiB/s wr, 10 op/s 2023-03-09T18:51:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:51:01 smithi038 ceph-mon[92422]: pgmap v7963: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 310 KiB/s rd, 309 KiB/s wr, 10 op/s 2023-03-09T18:51:03.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:51:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13973.442861 secs 2023-03-09T18:51:03.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:51:03 smithi122 ceph-mon[93869]: pgmap v7964: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 310 KiB/s wr, 13 op/s 2023-03-09T18:51:03.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:51:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13973.442861 secs 2023-03-09T18:51:03.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:51:03 smithi181 ceph-mon[94189]: pgmap v7964: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 310 KiB/s wr, 13 op/s 2023-03-09T18:51:03.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:51:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13973.442861 secs 2023-03-09T18:51:03.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:51:03 smithi038 ceph-mon[92422]: pgmap v7964: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 310 KiB/s wr, 13 op/s 2023-03-09T18:51:03.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14435.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:51:05.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:51:05 smithi122 ceph-mon[93869]: pgmap v7965: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 342 KiB/s wr, 25 op/s 2023-03-09T18:51:05.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:51:05 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:51:05.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:51:05 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:51:05.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:51:05 smithi122 ceph-mon[93869]: osdmap e711: 12 total, 12 up, 12 in 2023-03-09T18:51:05.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:51:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13976.163934 secs 2023-03-09T18:51:05.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:51:05 smithi181 ceph-mon[94189]: pgmap v7965: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 342 KiB/s wr, 25 op/s 2023-03-09T18:51:05.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:51:05 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:51:05.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:51:05 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:51:05.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:51:05 smithi181 ceph-mon[94189]: osdmap e711: 12 total, 12 up, 12 in 2023-03-09T18:51:05.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:51:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13976.163934 secs 2023-03-09T18:51:05.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:51:05 smithi038 ceph-mon[92422]: pgmap v7965: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 342 KiB/s wr, 25 op/s 2023-03-09T18:51:05.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:51:05 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:51:05.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:51:05 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:51:05.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:51:05 smithi038 ceph-mon[92422]: osdmap e711: 12 total, 12 up, 12 in 2023-03-09T18:51:05.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:51:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13976.163934 secs 2023-03-09T18:51:07.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:51:07 smithi122 ceph-mon[93869]: pgmap v7967: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 361 KiB/s wr, 29 op/s 2023-03-09T18:51:07.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:51:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13978.442980 secs 2023-03-09T18:51:07.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:51:07 smithi181 ceph-mon[94189]: pgmap v7967: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 361 KiB/s wr, 29 op/s 2023-03-09T18:51:07.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:51:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13978.442980 secs 2023-03-09T18:51:07.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:51:07 smithi038 ceph-mon[92422]: pgmap v7967: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 361 KiB/s wr, 29 op/s 2023-03-09T18:51:07.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:51:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13978.442980 secs 2023-03-09T18:51:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14440.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:51:09.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:51:09 smithi122 ceph-mon[93869]: pgmap v7968: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 260 KiB/s wr, 28 op/s 2023-03-09T18:51:09.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:51:09 smithi181 ceph-mon[94189]: pgmap v7968: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 260 KiB/s wr, 28 op/s 2023-03-09T18:51:09.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:51:09 smithi038 ceph-mon[92422]: pgmap v7968: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 260 KiB/s wr, 28 op/s 2023-03-09T18:51:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:51:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13981.163985 secs 2023-03-09T18:51:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:51:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13981.163985 secs 2023-03-09T18:51:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:51:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13981.163985 secs 2023-03-09T18:51:11.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:51:11 smithi122 ceph-mon[93869]: pgmap v7969: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 233 KiB/s wr, 28 op/s 2023-03-09T18:51:11.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:51:11 smithi181 ceph-mon[94189]: pgmap v7969: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 233 KiB/s wr, 28 op/s 2023-03-09T18:51:11.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:51:11 smithi038 ceph-mon[92422]: pgmap v7969: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 233 KiB/s wr, 28 op/s 2023-03-09T18:51:13.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:51:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13983.443101 secs 2023-03-09T18:51:13.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:51:13 smithi122 ceph-mon[93869]: pgmap v7970: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 346 KiB/s wr, 28 op/s 2023-03-09T18:51:13.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:51:13 smithi122 ceph-mon[93869]: osdmap e712: 12 total, 12 up, 12 in 2023-03-09T18:51:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:51:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13983.443101 secs 2023-03-09T18:51:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:51:13 smithi181 ceph-mon[94189]: pgmap v7970: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 346 KiB/s wr, 28 op/s 2023-03-09T18:51:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:51:13 smithi181 ceph-mon[94189]: osdmap e712: 12 total, 12 up, 12 in 2023-03-09T18:51:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:51:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13983.443101 secs 2023-03-09T18:51:13.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:51:13 smithi038 ceph-mon[92422]: pgmap v7970: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 346 KiB/s wr, 28 op/s 2023-03-09T18:51:13.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:51:13 smithi038 ceph-mon[92422]: osdmap e712: 12 total, 12 up, 12 in 2023-03-09T18:51:13.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14445.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:51:15.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:51:15 smithi122 ceph-mon[93869]: pgmap v7972: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 195 KiB/s wr, 11 op/s 2023-03-09T18:51:15.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:51:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13986.164042 secs 2023-03-09T18:51:15.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:51:15 smithi181 ceph-mon[94189]: pgmap v7972: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 195 KiB/s wr, 11 op/s 2023-03-09T18:51:15.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:51:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13986.164042 secs 2023-03-09T18:51:15.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:51:15 smithi038 ceph-mon[92422]: pgmap v7972: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 195 KiB/s wr, 11 op/s 2023-03-09T18:51:15.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:51:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13986.164042 secs 2023-03-09T18:51:17.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:51:17 smithi122 ceph-mon[93869]: pgmap v7973: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 219 KiB/s wr, 11 op/s 2023-03-09T18:51:17.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:51:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13988.443219 secs 2023-03-09T18:51:17.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:51:17 smithi181 ceph-mon[94189]: pgmap v7973: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 219 KiB/s wr, 11 op/s 2023-03-09T18:51:17.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:51:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13988.443219 secs 2023-03-09T18:51:17.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:51:17 smithi038 ceph-mon[92422]: pgmap v7973: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 219 KiB/s wr, 11 op/s 2023-03-09T18:51:17.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:51:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13988.443219 secs 2023-03-09T18:51:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14450.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:51:19.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:51:19 smithi122 ceph-mon[93869]: pgmap v7974: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 164 KiB/s wr, 10 op/s 2023-03-09T18:51:19.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:51:19 smithi181 ceph-mon[94189]: pgmap v7974: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 164 KiB/s wr, 10 op/s 2023-03-09T18:51:19.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:51:19 smithi038 ceph-mon[92422]: pgmap v7974: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 164 KiB/s wr, 10 op/s 2023-03-09T18:51:20.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:51:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13991.164101 secs 2023-03-09T18:51:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:51:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13991.164101 secs 2023-03-09T18:51:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:51:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13991.164101 secs 2023-03-09T18:51:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:51:21 smithi038 ceph-mon[92422]: pgmap v7975: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 164 KiB/s wr, 9 op/s 2023-03-09T18:51:21.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:51:21 smithi122 ceph-mon[93869]: pgmap v7975: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 164 KiB/s wr, 9 op/s 2023-03-09T18:51:21.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:51:21 smithi181 ceph-mon[94189]: pgmap v7975: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 164 KiB/s wr, 9 op/s 2023-03-09T18:51:23.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:51:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13993.443285 secs 2023-03-09T18:51:23.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:51:23 smithi122 ceph-mon[93869]: pgmap v7976: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 51 KiB/s wr, 8 op/s 2023-03-09T18:51:23.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:51:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13993.443285 secs 2023-03-09T18:51:23.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:51:23 smithi181 ceph-mon[94189]: pgmap v7976: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 51 KiB/s wr, 8 op/s 2023-03-09T18:51:23.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:51:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13993.443285 secs 2023-03-09T18:51:23.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:51:23 smithi038 ceph-mon[92422]: pgmap v7976: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 51 KiB/s wr, 8 op/s 2023-03-09T18:51:23.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14455.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:51:25.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:51:25 smithi122 ceph-mon[93869]: pgmap v7977: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 43 KiB/s wr, 7 op/s 2023-03-09T18:51:25.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:51:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13996.164160 secs 2023-03-09T18:51:25.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:51:25 smithi181 ceph-mon[94189]: pgmap v7977: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 43 KiB/s wr, 7 op/s 2023-03-09T18:51:25.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:51:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13996.164160 secs 2023-03-09T18:51:25.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:51:25 smithi038 ceph-mon[92422]: pgmap v7977: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 43 KiB/s wr, 7 op/s 2023-03-09T18:51:25.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:51:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13996.164160 secs 2023-03-09T18:51:27.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:51:27 smithi122 ceph-mon[93869]: pgmap v7978: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 74 KiB/s wr, 9 op/s 2023-03-09T18:51:27.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:51:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 13998.443400 secs 2023-03-09T18:51:27.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:51:27 smithi181 ceph-mon[94189]: pgmap v7978: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 74 KiB/s wr, 9 op/s 2023-03-09T18:51:27.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:51:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 13998.443400 secs 2023-03-09T18:51:27.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:51:27 smithi038 ceph-mon[92422]: pgmap v7978: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 74 KiB/s wr, 9 op/s 2023-03-09T18:51:27.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:51:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 13998.443400 secs 2023-03-09T18:51:28.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14460.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:51:29.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:51:29 smithi122 ceph-mon[93869]: pgmap v7979: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 43 KiB/s wr, 6 op/s 2023-03-09T18:51:29.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:51:29 smithi181 ceph-mon[94189]: pgmap v7979: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 43 KiB/s wr, 6 op/s 2023-03-09T18:51:29.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:51:29 smithi038 ceph-mon[92422]: pgmap v7979: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 43 KiB/s wr, 6 op/s 2023-03-09T18:51:30.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:51:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14001.164222 secs 2023-03-09T18:51:30.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:51:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14001.164222 secs 2023-03-09T18:51:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:51:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14001.164222 secs 2023-03-09T18:51:31.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:51:31 smithi122 ceph-mon[93869]: pgmap v7980: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 43 KiB/s wr, 6 op/s 2023-03-09T18:51:31.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:51:31 smithi181 ceph-mon[94189]: pgmap v7980: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 43 KiB/s wr, 6 op/s 2023-03-09T18:51:31.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:51:31 smithi038 ceph-mon[92422]: pgmap v7980: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 43 KiB/s wr, 6 op/s 2023-03-09T18:51:33.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:51:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14003.443523 secs 2023-03-09T18:51:33.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:51:33 smithi122 ceph-mon[93869]: pgmap v7981: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 89 KiB/s wr, 9 op/s 2023-03-09T18:51:33.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:51:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14003.443523 secs 2023-03-09T18:51:33.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:51:33 smithi181 ceph-mon[94189]: pgmap v7981: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 89 KiB/s wr, 9 op/s 2023-03-09T18:51:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:51:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14003.443523 secs 2023-03-09T18:51:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:51:33 smithi038 ceph-mon[92422]: pgmap v7981: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 89 KiB/s wr, 9 op/s 2023-03-09T18:51:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14465.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:51:35.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:51:35 smithi122 ceph-mon[93869]: pgmap v7982: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 78 KiB/s wr, 6 op/s 2023-03-09T18:51:35.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:51:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14006.164281 secs 2023-03-09T18:51:35.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:51:35 smithi181 ceph-mon[94189]: pgmap v7982: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 78 KiB/s wr, 6 op/s 2023-03-09T18:51:35.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:51:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14006.164281 secs 2023-03-09T18:51:35.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:51:35 smithi038 ceph-mon[92422]: pgmap v7982: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 78 KiB/s wr, 6 op/s 2023-03-09T18:51:35.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:51:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14006.164281 secs 2023-03-09T18:51:37.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:51:37 smithi122 ceph-mon[93869]: pgmap v7983: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 201 KiB/s wr, 10 op/s 2023-03-09T18:51:37.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:51:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14008.443659 secs 2023-03-09T18:51:37.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:51:37 smithi181 ceph-mon[94189]: pgmap v7983: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 201 KiB/s wr, 10 op/s 2023-03-09T18:51:37.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:51:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14008.443659 secs 2023-03-09T18:51:37.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:51:37 smithi038 ceph-mon[92422]: pgmap v7983: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 201 KiB/s wr, 10 op/s 2023-03-09T18:51:37.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:51:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14008.443659 secs 2023-03-09T18:51:38.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14470.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:51:39.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:51:39 smithi122 ceph-mon[93869]: pgmap v7984: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 170 KiB/s wr, 8 op/s 2023-03-09T18:51:39.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:51:39 smithi181 ceph-mon[94189]: pgmap v7984: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 170 KiB/s wr, 8 op/s 2023-03-09T18:51:39.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:51:39 smithi038 ceph-mon[92422]: pgmap v7984: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 170 KiB/s wr, 8 op/s 2023-03-09T18:51:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:51:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14011.164339 secs 2023-03-09T18:51:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:51:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14011.164339 secs 2023-03-09T18:51:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:51:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14011.164339 secs 2023-03-09T18:51:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:51:41 smithi122 ceph-mon[93869]: pgmap v7985: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 170 KiB/s wr, 7 op/s 2023-03-09T18:51:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:51:41 smithi181 ceph-mon[94189]: pgmap v7985: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 170 KiB/s wr, 7 op/s 2023-03-09T18:51:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:51:41 smithi038 ceph-mon[92422]: pgmap v7985: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 170 KiB/s wr, 7 op/s 2023-03-09T18:51:43.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:51:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14013.443767 secs 2023-03-09T18:51:43.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:51:43 smithi122 ceph-mon[93869]: pgmap v7986: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 286 KiB/s wr, 12 op/s 2023-03-09T18:51:43.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:51:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14013.443767 secs 2023-03-09T18:51:43.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:51:43 smithi181 ceph-mon[94189]: pgmap v7986: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 286 KiB/s wr, 12 op/s 2023-03-09T18:51:43.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:51:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14013.443767 secs 2023-03-09T18:51:43.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:51:43 smithi038 ceph-mon[92422]: pgmap v7986: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 286 KiB/s wr, 12 op/s 2023-03-09T18:51:43.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14475.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:51:45.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:51:45 smithi122 ceph-mon[93869]: pgmap v7987: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 240 KiB/s wr, 9 op/s 2023-03-09T18:51:45.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:51:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14016.164402 secs 2023-03-09T18:51:45.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:51:45 smithi181 ceph-mon[94189]: pgmap v7987: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 240 KiB/s wr, 9 op/s 2023-03-09T18:51:45.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:51:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14016.164402 secs 2023-03-09T18:51:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:51:45 smithi038 ceph-mon[92422]: pgmap v7987: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 240 KiB/s wr, 9 op/s 2023-03-09T18:51:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:51:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14016.164402 secs 2023-03-09T18:51:47.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:51:47 smithi122 ceph-mon[93869]: pgmap v7988: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 350 KiB/s wr, 14 op/s 2023-03-09T18:51:47.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:51:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14018.443883 secs 2023-03-09T18:51:47.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:51:47 smithi181 ceph-mon[94189]: pgmap v7988: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 350 KiB/s wr, 14 op/s 2023-03-09T18:51:47.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:51:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14018.443883 secs 2023-03-09T18:51:47.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:51:47 smithi038 ceph-mon[92422]: pgmap v7988: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 350 KiB/s wr, 14 op/s 2023-03-09T18:51:47.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:51:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14018.443883 secs 2023-03-09T18:51:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14480.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:51:49.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:51:49 smithi038 ceph-mon[92422]: pgmap v7989: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 226 KiB/s wr, 11 op/s 2023-03-09T18:51:49.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:51:49 smithi122 ceph-mon[93869]: pgmap v7989: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 226 KiB/s wr, 11 op/s 2023-03-09T18:51:49.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:51:49 smithi181 ceph-mon[94189]: pgmap v7989: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 226 KiB/s wr, 11 op/s 2023-03-09T18:51:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:51:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14021.164461 secs 2023-03-09T18:51:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:51:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14021.164461 secs 2023-03-09T18:51:50.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:51:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14021.164461 secs 2023-03-09T18:51:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:51:51 smithi038 ceph-mon[92422]: pgmap v7990: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 226 KiB/s wr, 10 op/s 2023-03-09T18:51:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:51:51 smithi122 ceph-mon[93869]: pgmap v7990: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 226 KiB/s wr, 10 op/s 2023-03-09T18:51:51.785 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:51:51 smithi181 ceph-mon[94189]: pgmap v7990: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 226 KiB/s wr, 10 op/s 2023-03-09T18:51:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:51:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14023.444023 secs 2023-03-09T18:51:53.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:51:53 smithi122 ceph-mon[93869]: pgmap v7991: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 335 KiB/s rd, 331 KiB/s wr, 14 op/s 2023-03-09T18:51:53.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:51:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:51:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:51:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14023.444023 secs 2023-03-09T18:51:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:51:53 smithi181 ceph-mon[94189]: pgmap v7991: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 335 KiB/s rd, 331 KiB/s wr, 14 op/s 2023-03-09T18:51:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:51:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:51:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:51:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14023.444023 secs 2023-03-09T18:51:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:51:53 smithi038 ceph-mon[92422]: pgmap v7991: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 335 KiB/s rd, 331 KiB/s wr, 14 op/s 2023-03-09T18:51:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:51:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:51:53.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14485.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:51:54.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:51:54 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:51:54.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:51:54 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:51:54.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:51:54 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:51:54.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:51:54 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:51:54.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:51:54 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:51:54.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:51:54 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:51:54.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:51:54 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:51:54.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:51:54 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:51:54.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:51:54 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:51:54.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:51:54 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:51:54.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:51:54 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:51:54.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:51:54 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:51:54.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:51:54 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:51:54.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:51:54 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:51:54.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:51:54 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:51:54.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:51:54 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:51:54.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:51:54 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:51:54.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:51:54 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:51:54.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:51:54 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:51:54.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:51:54 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:51:54.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:51:54 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:51:54.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:51:54 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:51:54.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:51:54 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:51:54.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:51:54 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:51:54.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:51:54 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:51:54.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:51:54 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:51:54.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:51:54 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:51:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:51:55 smithi122 ceph-mon[93869]: pgmap v7992: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 216 KiB/s wr, 9 op/s 2023-03-09T18:51:55.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:51:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14026.164521 secs 2023-03-09T18:51:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:51:55 smithi181 ceph-mon[94189]: pgmap v7992: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 216 KiB/s wr, 9 op/s 2023-03-09T18:51:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:51:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14026.164521 secs 2023-03-09T18:51:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:51:55 smithi038 ceph-mon[92422]: pgmap v7992: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 216 KiB/s wr, 9 op/s 2023-03-09T18:51:56.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:51:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14026.164521 secs 2023-03-09T18:51:57.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:51:57 smithi122 ceph-mon[93869]: pgmap v7993: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 288 KiB/s wr, 13 op/s 2023-03-09T18:51:57.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:51:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14028.444159 secs 2023-03-09T18:51:57.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:51:57 smithi181 ceph-mon[94189]: pgmap v7993: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 288 KiB/s wr, 13 op/s 2023-03-09T18:51:57.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:51:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14028.444159 secs 2023-03-09T18:51:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:51:57 smithi038 ceph-mon[92422]: pgmap v7993: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 288 KiB/s wr, 13 op/s 2023-03-09T18:51:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:51:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14028.444159 secs 2023-03-09T18:51:58.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14490.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:51:59.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:51:59 smithi122 ceph-mon[93869]: pgmap v7994: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 178 KiB/s wr, 9 op/s 2023-03-09T18:51:59.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:51:59 smithi181 ceph-mon[94189]: pgmap v7994: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 178 KiB/s wr, 9 op/s 2023-03-09T18:52:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:51:59 smithi038 ceph-mon[92422]: pgmap v7994: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 178 KiB/s wr, 9 op/s 2023-03-09T18:52:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14031.164579 secs 2023-03-09T18:52:00.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14031.164579 secs 2023-03-09T18:52:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14031.164579 secs 2023-03-09T18:52:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:01 smithi122 ceph-mon[93869]: pgmap v7995: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 177 KiB/s wr, 8 op/s 2023-03-09T18:52:01.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:01 smithi181 ceph-mon[94189]: pgmap v7995: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 177 KiB/s wr, 8 op/s 2023-03-09T18:52:02.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:01 smithi038 ceph-mon[92422]: pgmap v7995: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 177 KiB/s wr, 8 op/s 2023-03-09T18:52:02.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14033.444277 secs 2023-03-09T18:52:02.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14033.444277 secs 2023-03-09T18:52:03.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14033.444277 secs 2023-03-09T18:52:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14495.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:52:03.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:03 smithi122 ceph-mon[93869]: pgmap v7996: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 304 KiB/s wr, 13 op/s 2023-03-09T18:52:03.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:03 smithi181 ceph-mon[94189]: pgmap v7996: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 304 KiB/s wr, 13 op/s 2023-03-09T18:52:04.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:03 smithi038 ceph-mon[92422]: pgmap v7996: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 304 KiB/s wr, 13 op/s 2023-03-09T18:52:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:05 smithi122 ceph-mon[93869]: pgmap v7997: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 220 KiB/s wr, 20 op/s 2023-03-09T18:52:05.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:05 smithi122 ceph-mon[93869]: osdmap e713: 12 total, 12 up, 12 in 2023-03-09T18:52:05.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14036.164639 secs 2023-03-09T18:52:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:05 smithi181 ceph-mon[94189]: pgmap v7997: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 220 KiB/s wr, 20 op/s 2023-03-09T18:52:05.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:05 smithi181 ceph-mon[94189]: osdmap e713: 12 total, 12 up, 12 in 2023-03-09T18:52:05.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14036.164639 secs 2023-03-09T18:52:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:05 smithi038 ceph-mon[92422]: pgmap v7997: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 220 KiB/s wr, 20 op/s 2023-03-09T18:52:06.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:05 smithi038 ceph-mon[92422]: osdmap e713: 12 total, 12 up, 12 in 2023-03-09T18:52:06.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14036.164639 secs 2023-03-09T18:52:07.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:07 smithi122 ceph-mon[93869]: pgmap v7999: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 261 KiB/s wr, 27 op/s 2023-03-09T18:52:07.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14038.444344 secs 2023-03-09T18:52:07.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:07 smithi181 ceph-mon[94189]: pgmap v7999: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 261 KiB/s wr, 27 op/s 2023-03-09T18:52:07.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14038.444344 secs 2023-03-09T18:52:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:07 smithi038 ceph-mon[92422]: pgmap v7999: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 261 KiB/s wr, 27 op/s 2023-03-09T18:52:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14038.444344 secs 2023-03-09T18:52:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14500.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:52:09.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:09 smithi122 ceph-mon[93869]: pgmap v8000: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 261 KiB/s wr, 27 op/s 2023-03-09T18:52:09.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:09 smithi181 ceph-mon[94189]: pgmap v8000: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 261 KiB/s wr, 27 op/s 2023-03-09T18:52:10.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:09 smithi038 ceph-mon[92422]: pgmap v8000: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 261 KiB/s wr, 27 op/s 2023-03-09T18:52:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14041.164698 secs 2023-03-09T18:52:10.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14041.164698 secs 2023-03-09T18:52:11.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14041.164698 secs 2023-03-09T18:52:11.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:11 smithi122 ceph-mon[93869]: pgmap v8001: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 261 KiB/s wr, 27 op/s 2023-03-09T18:52:11.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:11 smithi181 ceph-mon[94189]: pgmap v8001: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 261 KiB/s wr, 27 op/s 2023-03-09T18:52:12.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:11 smithi038 ceph-mon[92422]: pgmap v8001: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 261 KiB/s wr, 27 op/s 2023-03-09T18:52:12.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14043.444425 secs 2023-03-09T18:52:12.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:12 smithi122 ceph-mon[93869]: osdmap e714: 12 total, 12 up, 12 in 2023-03-09T18:52:12.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14043.444425 secs 2023-03-09T18:52:12.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:12 smithi181 ceph-mon[94189]: osdmap e714: 12 total, 12 up, 12 in 2023-03-09T18:52:13.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14043.444425 secs 2023-03-09T18:52:13.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:12 smithi038 ceph-mon[92422]: osdmap e714: 12 total, 12 up, 12 in 2023-03-09T18:52:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14505.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:52:13.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:13 smithi122 ceph-mon[93869]: pgmap v8002: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 140 KiB/s wr, 25 op/s 2023-03-09T18:52:13.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:13 smithi181 ceph-mon[94189]: pgmap v8002: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 140 KiB/s wr, 25 op/s 2023-03-09T18:52:14.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:13 smithi038 ceph-mon[92422]: pgmap v8002: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 140 KiB/s wr, 25 op/s 2023-03-09T18:52:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:15 smithi122 ceph-mon[93869]: pgmap v8004: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 119 KiB/s wr, 11 op/s 2023-03-09T18:52:15.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14046.164757 secs 2023-03-09T18:52:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:15 smithi181 ceph-mon[94189]: pgmap v8004: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 119 KiB/s wr, 11 op/s 2023-03-09T18:52:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14046.164757 secs 2023-03-09T18:52:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:15 smithi038 ceph-mon[92422]: pgmap v8004: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 119 KiB/s wr, 11 op/s 2023-03-09T18:52:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14046.164757 secs 2023-03-09T18:52:17.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:17 smithi122 ceph-mon[93869]: pgmap v8005: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 30 KiB/s wr, 6 op/s 2023-03-09T18:52:17.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14048.444577 secs 2023-03-09T18:52:17.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:17 smithi181 ceph-mon[94189]: pgmap v8005: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 30 KiB/s wr, 6 op/s 2023-03-09T18:52:17.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14048.444577 secs 2023-03-09T18:52:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:17 smithi038 ceph-mon[92422]: pgmap v8005: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 30 KiB/s wr, 6 op/s 2023-03-09T18:52:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14048.444577 secs 2023-03-09T18:52:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14510.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:52:19.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:19 smithi122 ceph-mon[93869]: pgmap v8006: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 30 KiB/s wr, 6 op/s 2023-03-09T18:52:19.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:19 smithi181 ceph-mon[94189]: pgmap v8006: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 30 KiB/s wr, 6 op/s 2023-03-09T18:52:20.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:19 smithi038 ceph-mon[92422]: pgmap v8006: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 30 KiB/s wr, 6 op/s 2023-03-09T18:52:20.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14051.164817 secs 2023-03-09T18:52:20.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14051.164817 secs 2023-03-09T18:52:21.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14051.164817 secs 2023-03-09T18:52:21.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:21 smithi122 ceph-mon[93869]: pgmap v8007: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 30 KiB/s wr, 6 op/s 2023-03-09T18:52:21.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:21 smithi181 ceph-mon[94189]: pgmap v8007: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 30 KiB/s wr, 6 op/s 2023-03-09T18:52:22.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:21 smithi038 ceph-mon[92422]: pgmap v8007: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 30 KiB/s wr, 6 op/s 2023-03-09T18:52:22.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14053.444692 secs 2023-03-09T18:52:22.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14053.444692 secs 2023-03-09T18:52:23.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14053.444692 secs 2023-03-09T18:52:23.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14515.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:52:23.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:23 smithi122 ceph-mon[93869]: pgmap v8008: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T18:52:23.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:23 smithi181 ceph-mon[94189]: pgmap v8008: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T18:52:24.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:23 smithi038 ceph-mon[92422]: pgmap v8008: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T18:52:25.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:25 smithi122 ceph-mon[93869]: pgmap v8009: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T18:52:25.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14056.164894 secs 2023-03-09T18:52:25.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:25 smithi181 ceph-mon[94189]: pgmap v8009: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T18:52:25.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14056.164894 secs 2023-03-09T18:52:26.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:25 smithi038 ceph-mon[92422]: pgmap v8009: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T18:52:26.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14056.164894 secs 2023-03-09T18:52:27.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:27 smithi122 ceph-mon[93869]: pgmap v8010: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 341 B/s wr, 8 op/s 2023-03-09T18:52:27.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14058.444839 secs 2023-03-09T18:52:27.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:27 smithi181 ceph-mon[94189]: pgmap v8010: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 341 B/s wr, 8 op/s 2023-03-09T18:52:27.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14058.444839 secs 2023-03-09T18:52:28.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:27 smithi038 ceph-mon[92422]: pgmap v8010: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 341 B/s wr, 8 op/s 2023-03-09T18:52:28.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14058.444839 secs 2023-03-09T18:52:28.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14520.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:52:29.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:29 smithi122 ceph-mon[93869]: pgmap v8011: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2023-03-09T18:52:29.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:29 smithi181 ceph-mon[94189]: pgmap v8011: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2023-03-09T18:52:30.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:29 smithi038 ceph-mon[92422]: pgmap v8011: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2023-03-09T18:52:30.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14061.164978 secs 2023-03-09T18:52:30.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14061.164978 secs 2023-03-09T18:52:31.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14061.164978 secs 2023-03-09T18:52:31.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:31 smithi122 ceph-mon[93869]: pgmap v8012: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2023-03-09T18:52:31.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:31 smithi181 ceph-mon[94189]: pgmap v8012: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2023-03-09T18:52:32.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:31 smithi038 ceph-mon[92422]: pgmap v8012: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2023-03-09T18:52:32.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14063.444957 secs 2023-03-09T18:52:32.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14063.444957 secs 2023-03-09T18:52:33.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14063.444957 secs 2023-03-09T18:52:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14525.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:52:33.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:33 smithi122 ceph-mon[93869]: pgmap v8013: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 46 KiB/s wr, 8 op/s 2023-03-09T18:52:33.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:33 smithi181 ceph-mon[94189]: pgmap v8013: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 46 KiB/s wr, 8 op/s 2023-03-09T18:52:34.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:33 smithi038 ceph-mon[92422]: pgmap v8013: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 46 KiB/s wr, 8 op/s 2023-03-09T18:52:35.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:35 smithi122 ceph-mon[93869]: pgmap v8014: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 46 KiB/s wr, 6 op/s 2023-03-09T18:52:35.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14066.165038 secs 2023-03-09T18:52:35.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:35 smithi181 ceph-mon[94189]: pgmap v8014: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 46 KiB/s wr, 6 op/s 2023-03-09T18:52:35.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14066.165038 secs 2023-03-09T18:52:36.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:35 smithi038 ceph-mon[92422]: pgmap v8014: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 46 KiB/s wr, 6 op/s 2023-03-09T18:52:36.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14066.165038 secs 2023-03-09T18:52:37.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:37 smithi181 ceph-mon[94189]: pgmap v8015: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 94 KiB/s wr, 8 op/s 2023-03-09T18:52:38.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:37 smithi038 ceph-mon[92422]: pgmap v8015: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 94 KiB/s wr, 8 op/s 2023-03-09T18:52:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:37 smithi122 ceph-mon[93869]: pgmap v8015: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 94 KiB/s wr, 8 op/s 2023-03-09T18:52:38.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14530.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:52:40.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:39 smithi038 ceph-mon[92422]: pgmap v8016: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 107 KiB/s wr, 6 op/s 2023-03-09T18:52:40.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:39 smithi122 ceph-mon[93869]: pgmap v8016: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 107 KiB/s wr, 6 op/s 2023-03-09T18:52:40.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:39 smithi181 ceph-mon[94189]: pgmap v8016: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 107 KiB/s wr, 6 op/s 2023-03-09T18:52:41.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14071.165099 secs 2023-03-09T18:52:41.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:40 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:52:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14071.165099 secs 2023-03-09T18:52:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:40 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:52:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14071.165099 secs 2023-03-09T18:52:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:40 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:52:42.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:41 smithi038 ceph-mon[92422]: pgmap v8017: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 107 KiB/s wr, 6 op/s 2023-03-09T18:52:42.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:41 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:52:42.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:52:42.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:52:42.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:41 smithi122 ceph-mon[93869]: pgmap v8017: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 107 KiB/s wr, 6 op/s 2023-03-09T18:52:42.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:41 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:52:42.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:52:42.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:52:42.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:41 smithi181 ceph-mon[94189]: pgmap v8017: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 107 KiB/s wr, 6 op/s 2023-03-09T18:52:42.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:41 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:52:42.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:52:42.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:52:43.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14535.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:52:44.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:43 smithi038 ceph-mon[92422]: pgmap v8018: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 209 KiB/s wr, 10 op/s 2023-03-09T18:52:44.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:43 smithi122 ceph-mon[93869]: pgmap v8018: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 209 KiB/s wr, 10 op/s 2023-03-09T18:52:44.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:43 smithi181 ceph-mon[94189]: pgmap v8018: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 209 KiB/s wr, 10 op/s 2023-03-09T18:52:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:45 smithi122 ceph-mon[93869]: pgmap v8019: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 276 KiB/s wr, 9 op/s 2023-03-09T18:52:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14076.165158 secs 2023-03-09T18:52:46.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:45 smithi181 ceph-mon[94189]: pgmap v8019: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 276 KiB/s wr, 9 op/s 2023-03-09T18:52:46.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14076.165158 secs 2023-03-09T18:52:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:45 smithi038 ceph-mon[92422]: pgmap v8019: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 276 KiB/s wr, 9 op/s 2023-03-09T18:52:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14076.165158 secs 2023-03-09T18:52:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:47 smithi122 ceph-mon[93869]: pgmap v8020: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 289 KiB/s wr, 12 op/s 2023-03-09T18:52:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:47 smithi181 ceph-mon[94189]: pgmap v8020: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 289 KiB/s wr, 12 op/s 2023-03-09T18:52:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:47 smithi038 ceph-mon[92422]: pgmap v8020: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 289 KiB/s wr, 12 op/s 2023-03-09T18:52:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14540.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:52:50.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:49 smithi122 ceph-mon[93869]: pgmap v8021: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 319 KiB/s wr, 11 op/s 2023-03-09T18:52:50.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:49 smithi181 ceph-mon[94189]: pgmap v8021: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 319 KiB/s wr, 11 op/s 2023-03-09T18:52:50.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:49 smithi038 ceph-mon[92422]: pgmap v8021: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 319 KiB/s wr, 11 op/s 2023-03-09T18:52:51.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14081.165218 secs 2023-03-09T18:52:51.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14081.165218 secs 2023-03-09T18:52:51.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14081.165218 secs 2023-03-09T18:52:52.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:51 smithi122 ceph-mon[93869]: pgmap v8022: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 318 KiB/s rd, 305 KiB/s wr, 10 op/s 2023-03-09T18:52:52.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:51 smithi181 ceph-mon[94189]: pgmap v8022: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 318 KiB/s rd, 305 KiB/s wr, 10 op/s 2023-03-09T18:52:52.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:51 smithi038 ceph-mon[92422]: pgmap v8022: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 318 KiB/s rd, 305 KiB/s wr, 10 op/s 2023-03-09T18:52:53.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14083.445408 secs 2023-03-09T18:52:53.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14083.445408 secs 2023-03-09T18:52:53.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14083.445408 secs 2023-03-09T18:52:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14545.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:52:54.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:53 smithi122 ceph-mon[93869]: pgmap v8023: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 328 KiB/s wr, 13 op/s 2023-03-09T18:52:54.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:52:54.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:53 smithi181 ceph-mon[94189]: pgmap v8023: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 328 KiB/s wr, 13 op/s 2023-03-09T18:52:54.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:52:54.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:53 smithi038 ceph-mon[92422]: pgmap v8023: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 328 KiB/s wr, 13 op/s 2023-03-09T18:52:54.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:52:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:55 smithi122 ceph-mon[93869]: pgmap v8024: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 279 KiB/s wr, 10 op/s 2023-03-09T18:52:55.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14086.165278 secs 2023-03-09T18:52:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:55 smithi181 ceph-mon[94189]: pgmap v8024: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 279 KiB/s wr, 10 op/s 2023-03-09T18:52:56.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14086.165278 secs 2023-03-09T18:52:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:55 smithi038 ceph-mon[92422]: pgmap v8024: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 279 KiB/s wr, 10 op/s 2023-03-09T18:52:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14086.165278 secs 2023-03-09T18:52:57.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:56 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:52:57.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:52:57.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:52:57.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:52:57.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:52:57.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:52:57.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:52:57.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:56 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:52:57.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:52:57.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:52:57.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:52:57.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:52:57.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:52:57.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:52:57.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:56 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:52:57.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:52:57.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:52:57.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:52:57.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:52:57.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:52:57.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:52:58.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:57 smithi122 ceph-mon[93869]: pgmap v8025: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 167 KiB/s wr, 11 op/s 2023-03-09T18:52:58.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14088.445560 secs 2023-03-09T18:52:58.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:57 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:52:58.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:57 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:52:58.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:57 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:52:58.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:57 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:52:58.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:57 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:52:58.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:57 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:52:58.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:57 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:52:58.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:57 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:52:58.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:57 smithi181 ceph-mon[94189]: pgmap v8025: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 167 KiB/s wr, 11 op/s 2023-03-09T18:52:58.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14088.445560 secs 2023-03-09T18:52:58.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:57 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:52:58.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:57 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:52:58.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:57 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:52:58.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:57 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:52:58.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:57 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:52:58.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:57 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:52:58.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:57 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:52:58.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:57 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:52:58.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:57 smithi038 ceph-mon[92422]: pgmap v8025: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 167 KiB/s wr, 11 op/s 2023-03-09T18:52:58.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14088.445560 secs 2023-03-09T18:52:58.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:57 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:52:58.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:57 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:52:58.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:57 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:52:58.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:57 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:52:58.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:57 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:52:58.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:57 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:52:58.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:57 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:52:58.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:57 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:52:58.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14550.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:52:59.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:58 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi038 to 176.8M 2023-03-09T18:52:59.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:58 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi038 to 185402245: error parsing value: Value '185402245' is below minimum 939524096 2023-03-09T18:52:59.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:58 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T18:52:59.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:58 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T18:52:59.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:52:59.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:52:59.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:52:59.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:52:59.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:52:59.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:52:59.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:52:59.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:52:59.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:52:59.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:58 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi038 to 176.8M 2023-03-09T18:52:59.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:58 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi038 to 185402245: error parsing value: Value '185402245' is below minimum 939524096 2023-03-09T18:52:59.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:58 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T18:52:59.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:58 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T18:52:59.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:52:59.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:52:59.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:52:59.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:52:59.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:52:59.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:52:59.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:52:59.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:52:59.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:52:59.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:58 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi038 to 176.8M 2023-03-09T18:52:59.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:58 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi038 to 185402245: error parsing value: Value '185402245' is below minimum 939524096 2023-03-09T18:52:59.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:58 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T18:52:59.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:58 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T18:52:59.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:52:59.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:52:59.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:52:59.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:52:59.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:52:59.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T18:52:59.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:52:59.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:52:59.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:53:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:59 smithi122 ceph-mon[93869]: pgmap v8026: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 215 KiB/s wr, 10 op/s 2023-03-09T18:53:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:59 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T18:53:00.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:52:59 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T18:53:00.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:59 smithi181 ceph-mon[94189]: pgmap v8026: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 215 KiB/s wr, 10 op/s 2023-03-09T18:53:00.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:59 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T18:53:00.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:52:59 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T18:53:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:59 smithi038 ceph-mon[92422]: pgmap v8026: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 215 KiB/s wr, 10 op/s 2023-03-09T18:53:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:59 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T18:53:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:52:59 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T18:53:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:53:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14091.165337 secs 2023-03-09T18:53:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:53:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14091.165337 secs 2023-03-09T18:53:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:53:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14091.165337 secs 2023-03-09T18:53:02.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:53:01 smithi122 ceph-mon[93869]: pgmap v8027: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 137 KiB/s wr, 8 op/s 2023-03-09T18:53:02.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:53:01 smithi181 ceph-mon[94189]: pgmap v8027: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 137 KiB/s wr, 8 op/s 2023-03-09T18:53:02.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:53:01 smithi038 ceph-mon[92422]: pgmap v8027: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 137 KiB/s wr, 8 op/s 2023-03-09T18:53:03.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:53:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14093.445678 secs 2023-03-09T18:53:03.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:53:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14093.445678 secs 2023-03-09T18:53:03.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:53:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14093.445678 secs 2023-03-09T18:53:03.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14555.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:53:04.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:53:03 smithi122 ceph-mon[93869]: pgmap v8028: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 151 KiB/s wr, 11 op/s 2023-03-09T18:53:04.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:53:03 smithi181 ceph-mon[94189]: pgmap v8028: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 151 KiB/s wr, 11 op/s 2023-03-09T18:53:04.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:53:03 smithi038 ceph-mon[92422]: pgmap v8028: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 151 KiB/s wr, 11 op/s 2023-03-09T18:53:05.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:53:04 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14096.165395 secs 2023-03-09T18:53:05.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:53:04 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14096.165395 secs 2023-03-09T18:53:05.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:53:04 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14096.165395 secs 2023-03-09T18:53:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:53:05 smithi122 ceph-mon[93869]: pgmap v8029: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 174 KiB/s wr, 21 op/s 2023-03-09T18:53:06.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:53:05 smithi181 ceph-mon[94189]: pgmap v8029: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 174 KiB/s wr, 21 op/s 2023-03-09T18:53:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:53:05 smithi038 ceph-mon[92422]: pgmap v8029: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 174 KiB/s wr, 21 op/s 2023-03-09T18:53:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:53:07 smithi122 ceph-mon[93869]: pgmap v8030: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 143 KiB/s wr, 24 op/s 2023-03-09T18:53:08.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:53:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14098.445801 secs 2023-03-09T18:53:08.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:53:07 smithi181 ceph-mon[94189]: pgmap v8030: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 143 KiB/s wr, 24 op/s 2023-03-09T18:53:08.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:53:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14098.445801 secs 2023-03-09T18:53:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:53:07 smithi038 ceph-mon[92422]: pgmap v8030: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 143 KiB/s wr, 24 op/s 2023-03-09T18:53:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:53:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14098.445801 secs 2023-03-09T18:53:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14560.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:53:10.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:53:09 smithi122 ceph-mon[93869]: pgmap v8031: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 173 KiB/s wr, 22 op/s 2023-03-09T18:53:10.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:53:09 smithi122 ceph-mon[93869]: osdmap e715: 12 total, 12 up, 12 in 2023-03-09T18:53:10.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:53:09 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14101.165456 secs 2023-03-09T18:53:10.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:53:09 smithi181 ceph-mon[94189]: pgmap v8031: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 173 KiB/s wr, 22 op/s 2023-03-09T18:53:10.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:53:09 smithi181 ceph-mon[94189]: osdmap e715: 12 total, 12 up, 12 in 2023-03-09T18:53:10.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:53:09 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14101.165456 secs 2023-03-09T18:53:10.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:53:09 smithi038 ceph-mon[92422]: pgmap v8031: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 173 KiB/s wr, 22 op/s 2023-03-09T18:53:10.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:53:09 smithi038 ceph-mon[92422]: osdmap e715: 12 total, 12 up, 12 in 2023-03-09T18:53:10.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:53:09 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14101.165456 secs 2023-03-09T18:53:12.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:53:11 smithi122 ceph-mon[93869]: pgmap v8033: 129 pgs: 9 active+clean+snaptrim_wait, 120 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 135 KiB/s wr, 25 op/s 2023-03-09T18:53:12.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:53:11 smithi181 ceph-mon[94189]: pgmap v8033: 129 pgs: 9 active+clean+snaptrim_wait, 120 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 135 KiB/s wr, 25 op/s 2023-03-09T18:53:12.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:53:11 smithi038 ceph-mon[92422]: pgmap v8033: 129 pgs: 9 active+clean+snaptrim_wait, 120 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 135 KiB/s wr, 25 op/s 2023-03-09T18:53:13.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:53:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14103.445940 secs 2023-03-09T18:53:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:53:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14103.445940 secs 2023-03-09T18:53:13.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:53:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14103.445940 secs 2023-03-09T18:53:13.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14565.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:53:14.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:53:13 smithi122 ceph-mon[93869]: pgmap v8034: 129 pgs: 9 active+clean+snaptrim_wait, 120 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 120 KiB/s wr, 24 op/s 2023-03-09T18:53:14.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:53:13 smithi181 ceph-mon[94189]: pgmap v8034: 129 pgs: 9 active+clean+snaptrim_wait, 120 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 120 KiB/s wr, 24 op/s 2023-03-09T18:53:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:53:13 smithi038 ceph-mon[92422]: pgmap v8034: 129 pgs: 9 active+clean+snaptrim_wait, 120 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 120 KiB/s wr, 24 op/s 2023-03-09T18:53:15.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:53:14 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14106.165516 secs 2023-03-09T18:53:15.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:53:14 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14106.165516 secs 2023-03-09T18:53:15.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:53:14 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14106.165516 secs 2023-03-09T18:53:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:53:15 smithi122 ceph-mon[93869]: pgmap v8035: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 64 KiB/s wr, 10 op/s 2023-03-09T18:53:16.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:53:15 smithi181 ceph-mon[94189]: pgmap v8035: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 64 KiB/s wr, 10 op/s 2023-03-09T18:53:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:53:15 smithi038 ceph-mon[92422]: pgmap v8035: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 64 KiB/s wr, 10 op/s 2023-03-09T18:53:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:53:17 smithi122 ceph-mon[93869]: pgmap v8036: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 37 KiB/s wr, 7 op/s 2023-03-09T18:53:18.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:53:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14108.446078 secs 2023-03-09T18:53:18.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:53:17 smithi122 ceph-mon[93869]: osdmap e716: 12 total, 12 up, 12 in 2023-03-09T18:53:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:53:17 smithi181 ceph-mon[94189]: pgmap v8036: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 37 KiB/s wr, 7 op/s 2023-03-09T18:53:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:53:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14108.446078 secs 2023-03-09T18:53:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:53:17 smithi181 ceph-mon[94189]: osdmap e716: 12 total, 12 up, 12 in 2023-03-09T18:53:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:53:17 smithi038 ceph-mon[92422]: pgmap v8036: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 37 KiB/s wr, 7 op/s 2023-03-09T18:53:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:53:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14108.446078 secs 2023-03-09T18:53:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:53:17 smithi038 ceph-mon[92422]: osdmap e716: 12 total, 12 up, 12 in 2023-03-09T18:53:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14570.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:53:20.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:53:19 smithi181 ceph-mon[94189]: pgmap v8038: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 220 B/s wr, 7 op/s 2023-03-09T18:53:20.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:53:19 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14111.165619 secs 2023-03-09T18:53:20.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:53:19 smithi038 ceph-mon[92422]: pgmap v8038: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 220 B/s wr, 7 op/s 2023-03-09T18:53:20.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:53:19 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14111.165619 secs 2023-03-09T18:53:20.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:53:19 smithi122 ceph-mon[93869]: pgmap v8038: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 220 B/s wr, 7 op/s 2023-03-09T18:53:20.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:53:19 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14111.165619 secs 2023-03-09T18:53:22.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:53:21 smithi038 ceph-mon[92422]: pgmap v8039: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 7 op/s 2023-03-09T18:53:22.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:53:21 smithi122 ceph-mon[93869]: pgmap v8039: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 7 op/s 2023-03-09T18:53:22.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:53:21 smithi181 ceph-mon[94189]: pgmap v8039: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 7 op/s 2023-03-09T18:53:23.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:53:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14113.446129 secs 2023-03-09T18:53:23.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:53:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14113.446129 secs 2023-03-09T18:53:23.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:53:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14113.446129 secs 2023-03-09T18:53:23.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14575.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:53:24.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:53:23 smithi038 ceph-mon[92422]: pgmap v8040: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T18:53:24.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:53:23 smithi122 ceph-mon[93869]: pgmap v8040: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T18:53:24.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:53:23 smithi181 ceph-mon[94189]: pgmap v8040: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T18:53:25.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:53:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14116.165750 secs 2023-03-09T18:53:25.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:53:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14116.165750 secs 2023-03-09T18:53:25.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:53:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14116.165750 secs 2023-03-09T18:53:26.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:53:26 smithi122 ceph-mon[93869]: pgmap v8041: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T18:53:26.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:53:26 smithi181 ceph-mon[94189]: pgmap v8041: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T18:53:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:53:26 smithi038 ceph-mon[92422]: pgmap v8041: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T18:53:28.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:53:28 smithi122 ceph-mon[93869]: pgmap v8042: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 45 KiB/s wr, 8 op/s 2023-03-09T18:53:28.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:53:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14118.446192 secs 2023-03-09T18:53:28.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:53:28 smithi181 ceph-mon[94189]: pgmap v8042: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 45 KiB/s wr, 8 op/s 2023-03-09T18:53:28.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:53:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14118.446192 secs 2023-03-09T18:53:28.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:53:28 smithi038 ceph-mon[92422]: pgmap v8042: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 45 KiB/s wr, 8 op/s 2023-03-09T18:53:28.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:53:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14118.446192 secs 2023-03-09T18:53:28.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14580.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:53:30.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:53:30 smithi122 ceph-mon[93869]: pgmap v8043: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 87 KiB/s wr, 7 op/s 2023-03-09T18:53:30.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:53:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14121.165810 secs 2023-03-09T18:53:30.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:53:30 smithi181 ceph-mon[94189]: pgmap v8043: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 87 KiB/s wr, 7 op/s 2023-03-09T18:53:30.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:53:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14121.165810 secs 2023-03-09T18:53:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:53:30 smithi038 ceph-mon[92422]: pgmap v8043: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 87 KiB/s wr, 7 op/s 2023-03-09T18:53:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:53:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14121.165810 secs 2023-03-09T18:53:32.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:53:32 smithi122 ceph-mon[93869]: pgmap v8044: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 80 KiB/s wr, 6 op/s 2023-03-09T18:53:32.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:53:32 smithi181 ceph-mon[94189]: pgmap v8044: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 80 KiB/s wr, 6 op/s 2023-03-09T18:53:32.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:53:32 smithi038 ceph-mon[92422]: pgmap v8044: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 80 KiB/s wr, 6 op/s 2023-03-09T18:53:33.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:53:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14123.446294 secs 2023-03-09T18:53:33.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:53:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14123.446294 secs 2023-03-09T18:53:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:53:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14123.446294 secs 2023-03-09T18:53:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14585.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:53:34.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:53:34 smithi122 ceph-mon[93869]: pgmap v8045: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 126 KiB/s wr, 9 op/s 2023-03-09T18:53:34.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:53:34 smithi181 ceph-mon[94189]: pgmap v8045: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 126 KiB/s wr, 9 op/s 2023-03-09T18:53:34.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:53:34 smithi038 ceph-mon[92422]: pgmap v8045: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 126 KiB/s wr, 9 op/s 2023-03-09T18:53:35.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:53:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14126.165871 secs 2023-03-09T18:53:35.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:53:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14126.165871 secs 2023-03-09T18:53:35.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:53:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14126.165871 secs 2023-03-09T18:53:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:53:36 smithi122 ceph-mon[93869]: pgmap v8046: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 89 KiB/s wr, 6 op/s 2023-03-09T18:53:36.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:53:36 smithi181 ceph-mon[94189]: pgmap v8046: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 89 KiB/s wr, 6 op/s 2023-03-09T18:53:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:53:36 smithi038 ceph-mon[92422]: pgmap v8046: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 89 KiB/s wr, 6 op/s 2023-03-09T18:53:38.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:53:38 smithi122 ceph-mon[93869]: pgmap v8047: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 90 KiB/s wr, 8 op/s 2023-03-09T18:53:38.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:53:38 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 14128.446402 secs 2023-03-09T18:53:38.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:53:38 smithi181 ceph-mon[94189]: pgmap v8047: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 90 KiB/s wr, 8 op/s 2023-03-09T18:53:38.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:53:38 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 14128.446402 secs 2023-03-09T18:53:38.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:53:38 smithi038 ceph-mon[92422]: pgmap v8047: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 90 KiB/s wr, 8 op/s 2023-03-09T18:53:38.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:53:38 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 14128.446402 secs 2023-03-09T18:53:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14590.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:53:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:53:40 smithi122 ceph-mon[93869]: pgmap v8048: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 137 KiB/s wr, 7 op/s 2023-03-09T18:53:40.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:53:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14131.165943 secs 2023-03-09T18:53:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:53:40 smithi181 ceph-mon[94189]: pgmap v8048: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 137 KiB/s wr, 7 op/s 2023-03-09T18:53:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:53:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14131.165943 secs 2023-03-09T18:53:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:53:40 smithi038 ceph-mon[92422]: pgmap v8048: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 137 KiB/s wr, 7 op/s 2023-03-09T18:53:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:53:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14131.165943 secs 2023-03-09T18:53:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:53:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:53:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:53:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:53:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:53:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:53:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:53:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:53:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:53:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:53:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:53:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:53:42.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:53:42 smithi122 ceph-mon[93869]: pgmap v8049: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 95 KiB/s wr, 6 op/s 2023-03-09T18:53:42.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:53:42 smithi181 ceph-mon[94189]: pgmap v8049: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 95 KiB/s wr, 6 op/s 2023-03-09T18:53:42.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:53:42 smithi038 ceph-mon[92422]: pgmap v8049: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 95 KiB/s wr, 6 op/s 2023-03-09T18:53:43.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:53:43 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 14133.446528 secs 2023-03-09T18:53:43.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:53:43 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 14133.446528 secs 2023-03-09T18:53:43.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:53:43 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 14133.446528 secs 2023-03-09T18:53:43.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14595.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:53:44.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:53:44 smithi122 ceph-mon[93869]: pgmap v8050: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 232 KiB/s wr, 9 op/s 2023-03-09T18:53:44.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:53:44 smithi181 ceph-mon[94189]: pgmap v8050: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 232 KiB/s wr, 9 op/s 2023-03-09T18:53:44.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:53:44 smithi038 ceph-mon[92422]: pgmap v8050: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 232 KiB/s wr, 9 op/s 2023-03-09T18:53:45.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:53:45 smithi122 ceph-mon[93869]: pgmap v8051: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 240 KiB/s wr, 8 op/s 2023-03-09T18:53:45.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:53:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14136.165985 secs 2023-03-09T18:53:45.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:53:45 smithi181 ceph-mon[94189]: pgmap v8051: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 240 KiB/s wr, 8 op/s 2023-03-09T18:53:45.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:53:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14136.165985 secs 2023-03-09T18:53:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:53:45 smithi038 ceph-mon[92422]: pgmap v8051: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 240 KiB/s wr, 8 op/s 2023-03-09T18:53:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:53:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14136.165985 secs 2023-03-09T18:53:47.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:53:47 smithi122 ceph-mon[93869]: pgmap v8052: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 253 KiB/s wr, 11 op/s 2023-03-09T18:53:47.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:53:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14138.446641 secs 2023-03-09T18:53:47.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:53:47 smithi181 ceph-mon[94189]: pgmap v8052: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 253 KiB/s wr, 11 op/s 2023-03-09T18:53:47.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:53:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14138.446641 secs 2023-03-09T18:53:47.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:53:47 smithi038 ceph-mon[92422]: pgmap v8052: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 253 KiB/s wr, 11 op/s 2023-03-09T18:53:47.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:53:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14138.446641 secs 2023-03-09T18:53:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14600.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:53:49.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:53:49 smithi122 ceph-mon[93869]: pgmap v8053: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 253 KiB/s wr, 10 op/s 2023-03-09T18:53:49.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:53:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:53:49.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:53:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:53:49.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:53:49 smithi181 ceph-mon[94189]: pgmap v8053: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 253 KiB/s wr, 10 op/s 2023-03-09T18:53:49.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:53:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:53:49.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:53:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:53:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:53:49 smithi038 ceph-mon[92422]: pgmap v8053: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 253 KiB/s wr, 10 op/s 2023-03-09T18:53:49.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:53:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:53:49.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:53:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:53:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:53:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14141.166047 secs 2023-03-09T18:53:50.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:53:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14141.166047 secs 2023-03-09T18:53:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:53:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14141.166047 secs 2023-03-09T18:53:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:53:51 smithi122 ceph-mon[93869]: pgmap v8054: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 349 KiB/s wr, 10 op/s 2023-03-09T18:53:51.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:53:51 smithi181 ceph-mon[94189]: pgmap v8054: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 349 KiB/s wr, 10 op/s 2023-03-09T18:53:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:53:51 smithi038 ceph-mon[92422]: pgmap v8054: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 349 KiB/s wr, 10 op/s 2023-03-09T18:53:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:53:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14143.446755 secs 2023-03-09T18:53:52.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:53:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14143.446755 secs 2023-03-09T18:53:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:53:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14143.446755 secs 2023-03-09T18:53:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:53:53 smithi122 ceph-mon[93869]: pgmap v8055: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 363 KiB/s wr, 14 op/s 2023-03-09T18:53:53.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:53:53 smithi181 ceph-mon[94189]: pgmap v8055: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 363 KiB/s wr, 14 op/s 2023-03-09T18:53:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:53:53 smithi038 ceph-mon[92422]: pgmap v8055: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 363 KiB/s wr, 14 op/s 2023-03-09T18:53:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14605.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:53:55.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:53:55 smithi122 ceph-mon[93869]: pgmap v8056: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 251 KiB/s wr, 12 op/s 2023-03-09T18:53:55.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:53:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14146.166107 secs 2023-03-09T18:53:55.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:53:55 smithi181 ceph-mon[94189]: pgmap v8056: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 251 KiB/s wr, 12 op/s 2023-03-09T18:53:55.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:53:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14146.166107 secs 2023-03-09T18:53:55.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:53:55 smithi038 ceph-mon[92422]: pgmap v8056: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 251 KiB/s wr, 12 op/s 2023-03-09T18:53:55.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:53:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14146.166107 secs 2023-03-09T18:53:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:53:57 smithi122 ceph-mon[93869]: pgmap v8057: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 208 KiB/s wr, 14 op/s 2023-03-09T18:53:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:53:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14148.446891 secs 2023-03-09T18:53:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:53:57 smithi181 ceph-mon[94189]: pgmap v8057: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 208 KiB/s wr, 14 op/s 2023-03-09T18:53:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:53:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14148.446891 secs 2023-03-09T18:53:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:53:57 smithi038 ceph-mon[92422]: pgmap v8057: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 208 KiB/s wr, 14 op/s 2023-03-09T18:53:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:53:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14148.446891 secs 2023-03-09T18:53:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14610.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:53:59.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:53:59 smithi122 ceph-mon[93869]: pgmap v8058: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 195 KiB/s wr, 11 op/s 2023-03-09T18:53:59.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:53:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:53:59.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:53:59 smithi181 ceph-mon[94189]: pgmap v8058: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 195 KiB/s wr, 11 op/s 2023-03-09T18:53:59.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:53:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:53:59.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:53:59 smithi038 ceph-mon[92422]: pgmap v8058: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 195 KiB/s wr, 11 op/s 2023-03-09T18:53:59.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:53:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:54:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:54:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:54:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:54:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:54:00.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:54:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:54:00.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:54:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14151.166170 secs 2023-03-09T18:54:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:54:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:54:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:54:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:54:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:54:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:54:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:54:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14151.166170 secs 2023-03-09T18:54:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:54:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:54:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:54:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:54:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:54:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:54:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:54:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14151.166170 secs 2023-03-09T18:54:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:54:01 smithi122 ceph-mon[93869]: pgmap v8059: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 310 KiB/s rd, 298 KiB/s wr, 11 op/s 2023-03-09T18:54:01.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:54:01 smithi181 ceph-mon[94189]: pgmap v8059: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 310 KiB/s rd, 298 KiB/s wr, 11 op/s 2023-03-09T18:54:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:54:01 smithi038 ceph-mon[92422]: pgmap v8059: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 310 KiB/s rd, 298 KiB/s wr, 11 op/s 2023-03-09T18:54:02.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:54:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14153.447044 secs 2023-03-09T18:54:02.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:54:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14153.447044 secs 2023-03-09T18:54:02.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:54:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14153.447044 secs 2023-03-09T18:54:03.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:54:03 smithi122 ceph-mon[93869]: pgmap v8060: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 176 KiB/s wr, 14 op/s 2023-03-09T18:54:03.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:54:03 smithi181 ceph-mon[94189]: pgmap v8060: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 176 KiB/s wr, 14 op/s 2023-03-09T18:54:03.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:54:03 smithi038 ceph-mon[92422]: pgmap v8060: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 176 KiB/s wr, 14 op/s 2023-03-09T18:54:03.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14615.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:54:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:54:05 smithi122 ceph-mon[93869]: pgmap v8061: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 199 KiB/s wr, 24 op/s 2023-03-09T18:54:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:54:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14156.166230 secs 2023-03-09T18:54:05.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:54:05 smithi181 ceph-mon[94189]: pgmap v8061: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 199 KiB/s wr, 24 op/s 2023-03-09T18:54:05.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:54:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14156.166230 secs 2023-03-09T18:54:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:54:05 smithi038 ceph-mon[92422]: pgmap v8061: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 199 KiB/s wr, 24 op/s 2023-03-09T18:54:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:54:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14156.166230 secs 2023-03-09T18:54:07.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:54:07 smithi122 ceph-mon[93869]: pgmap v8062: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 174 KiB/s wr, 25 op/s 2023-03-09T18:54:07.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:54:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14158.447165 secs 2023-03-09T18:54:07.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:54:07 smithi181 ceph-mon[94189]: pgmap v8062: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 174 KiB/s wr, 25 op/s 2023-03-09T18:54:07.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:54:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14158.447165 secs 2023-03-09T18:54:07.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:54:07 smithi038 ceph-mon[92422]: pgmap v8062: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 174 KiB/s wr, 25 op/s 2023-03-09T18:54:07.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:54:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14158.447165 secs 2023-03-09T18:54:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14620.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:54:09.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:54:09 smithi122 ceph-mon[93869]: pgmap v8063: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 180 KiB/s wr, 22 op/s 2023-03-09T18:54:09.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:54:09 smithi181 ceph-mon[94189]: pgmap v8063: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 180 KiB/s wr, 22 op/s 2023-03-09T18:54:09.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:54:09 smithi038 ceph-mon[92422]: pgmap v8063: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 180 KiB/s wr, 22 op/s 2023-03-09T18:54:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:54:10 smithi122 ceph-mon[93869]: osdmap e717: 12 total, 12 up, 12 in 2023-03-09T18:54:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:54:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14161.166291 secs 2023-03-09T18:54:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:54:10 smithi181 ceph-mon[94189]: osdmap e717: 12 total, 12 up, 12 in 2023-03-09T18:54:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:54:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14161.166291 secs 2023-03-09T18:54:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:54:10 smithi038 ceph-mon[92422]: osdmap e717: 12 total, 12 up, 12 in 2023-03-09T18:54:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:54:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14161.166291 secs 2023-03-09T18:54:11.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:54:11 smithi122 ceph-mon[93869]: pgmap v8065: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 92 KiB/s wr, 26 op/s 2023-03-09T18:54:11.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:54:11 smithi181 ceph-mon[94189]: pgmap v8065: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 92 KiB/s wr, 26 op/s 2023-03-09T18:54:11.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:54:11 smithi038 ceph-mon[92422]: pgmap v8065: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 92 KiB/s wr, 26 op/s 2023-03-09T18:54:12.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:54:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14163.447285 secs 2023-03-09T18:54:12.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:54:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14163.447285 secs 2023-03-09T18:54:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:54:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14163.447285 secs 2023-03-09T18:54:13.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:54:13 smithi122 ceph-mon[93869]: pgmap v8066: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 65 KiB/s wr, 23 op/s 2023-03-09T18:54:13.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:54:13 smithi181 ceph-mon[94189]: pgmap v8066: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 65 KiB/s wr, 23 op/s 2023-03-09T18:54:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:54:13 smithi038 ceph-mon[92422]: pgmap v8066: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 65 KiB/s wr, 23 op/s 2023-03-09T18:54:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14625.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:54:15.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:54:15 smithi122 ceph-mon[93869]: pgmap v8067: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 20 KiB/s wr, 8 op/s 2023-03-09T18:54:15.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:54:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14166.166350 secs 2023-03-09T18:54:15.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:54:15 smithi181 ceph-mon[94189]: pgmap v8067: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 20 KiB/s wr, 8 op/s 2023-03-09T18:54:15.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:54:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14166.166350 secs 2023-03-09T18:54:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:54:15 smithi038 ceph-mon[92422]: pgmap v8067: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 20 KiB/s wr, 8 op/s 2023-03-09T18:54:15.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:54:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14166.166350 secs 2023-03-09T18:54:17.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:54:17 smithi122 ceph-mon[93869]: pgmap v8068: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 19 KiB/s wr, 8 op/s 2023-03-09T18:54:17.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:54:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14168.447401 secs 2023-03-09T18:54:17.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:54:17 smithi122 ceph-mon[93869]: osdmap e718: 12 total, 12 up, 12 in 2023-03-09T18:54:17.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:54:17 smithi181 ceph-mon[94189]: pgmap v8068: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 19 KiB/s wr, 8 op/s 2023-03-09T18:54:17.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:54:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14168.447401 secs 2023-03-09T18:54:17.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:54:17 smithi181 ceph-mon[94189]: osdmap e718: 12 total, 12 up, 12 in 2023-03-09T18:54:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:54:17 smithi038 ceph-mon[92422]: pgmap v8068: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 19 KiB/s wr, 8 op/s 2023-03-09T18:54:17.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:54:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14168.447401 secs 2023-03-09T18:54:17.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:54:17 smithi038 ceph-mon[92422]: osdmap e718: 12 total, 12 up, 12 in 2023-03-09T18:54:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14630.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:54:19.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:54:19 smithi122 ceph-mon[93869]: pgmap v8070: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 348 B/s wr, 9 op/s 2023-03-09T18:54:19.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:54:19 smithi181 ceph-mon[94189]: pgmap v8070: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 348 B/s wr, 9 op/s 2023-03-09T18:54:19.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:54:19 smithi038 ceph-mon[92422]: pgmap v8070: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 348 B/s wr, 9 op/s 2023-03-09T18:54:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:54:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14171.166412 secs 2023-03-09T18:54:20.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:54:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14171.166412 secs 2023-03-09T18:54:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:54:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14171.166412 secs 2023-03-09T18:54:21.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:54:21 smithi122 ceph-mon[93869]: pgmap v8071: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 8 op/s 2023-03-09T18:54:21.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:54:21 smithi181 ceph-mon[94189]: pgmap v8071: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 8 op/s 2023-03-09T18:54:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:54:21 smithi038 ceph-mon[92422]: pgmap v8071: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 8 op/s 2023-03-09T18:54:22.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:54:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14173.447523 secs 2023-03-09T18:54:22.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:54:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14173.447523 secs 2023-03-09T18:54:22.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:54:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14173.447523 secs 2023-03-09T18:54:23.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:54:23 smithi122 ceph-mon[93869]: pgmap v8072: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 8 op/s 2023-03-09T18:54:23.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:54:23 smithi181 ceph-mon[94189]: pgmap v8072: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 8 op/s 2023-03-09T18:54:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:54:23 smithi038 ceph-mon[92422]: pgmap v8072: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 8 op/s 2023-03-09T18:54:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14635.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:54:25.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:54:25 smithi038 ceph-mon[92422]: pgmap v8073: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 8 op/s 2023-03-09T18:54:25.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:54:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14176.166475 secs 2023-03-09T18:54:25.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:54:25 smithi122 ceph-mon[93869]: pgmap v8073: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 8 op/s 2023-03-09T18:54:25.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:54:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14176.166475 secs 2023-03-09T18:54:25.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:54:25 smithi181 ceph-mon[94189]: pgmap v8073: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 8 op/s 2023-03-09T18:54:25.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:54:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14176.166475 secs 2023-03-09T18:54:27.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:54:27 smithi038 ceph-mon[92422]: pgmap v8074: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 614 B/s wr, 8 op/s 2023-03-09T18:54:27.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:54:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14178.447648 secs 2023-03-09T18:54:27.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:54:27 smithi122 ceph-mon[93869]: pgmap v8074: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 614 B/s wr, 8 op/s 2023-03-09T18:54:27.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:54:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14178.447648 secs 2023-03-09T18:54:27.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:54:27 smithi181 ceph-mon[94189]: pgmap v8074: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 614 B/s wr, 8 op/s 2023-03-09T18:54:27.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:54:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14178.447648 secs 2023-03-09T18:54:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14640.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:54:29.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:54:29 smithi038 ceph-mon[92422]: pgmap v8075: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 465 B/s wr, 8 op/s 2023-03-09T18:54:29.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:54:29 smithi122 ceph-mon[93869]: pgmap v8075: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 465 B/s wr, 8 op/s 2023-03-09T18:54:29.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:54:29 smithi181 ceph-mon[94189]: pgmap v8075: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 465 B/s wr, 8 op/s 2023-03-09T18:54:30.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:54:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14181.166532 secs 2023-03-09T18:54:30.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:54:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14181.166532 secs 2023-03-09T18:54:30.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:54:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14181.166532 secs 2023-03-09T18:54:31.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:54:31 smithi122 ceph-mon[93869]: pgmap v8076: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 7 op/s 2023-03-09T18:54:31.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:54:31 smithi181 ceph-mon[94189]: pgmap v8076: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 7 op/s 2023-03-09T18:54:32.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:54:31 smithi038 ceph-mon[92422]: pgmap v8076: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 7 op/s 2023-03-09T18:54:32.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:54:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14183.447742 secs 2023-03-09T18:54:32.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:54:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14183.447742 secs 2023-03-09T18:54:32.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:54:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14183.447742 secs 2023-03-09T18:54:33.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14645.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:54:33.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:54:33 smithi122 ceph-mon[93869]: pgmap v8077: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 47 KiB/s wr, 10 op/s 2023-03-09T18:54:33.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:54:33 smithi181 ceph-mon[94189]: pgmap v8077: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 47 KiB/s wr, 10 op/s 2023-03-09T18:54:34.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:54:33 smithi038 ceph-mon[92422]: pgmap v8077: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 47 KiB/s wr, 10 op/s 2023-03-09T18:54:35.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:54:35 smithi122 ceph-mon[93869]: pgmap v8078: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 47 KiB/s wr, 7 op/s 2023-03-09T18:54:35.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:54:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14186.166593 secs 2023-03-09T18:54:35.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:54:35 smithi181 ceph-mon[94189]: pgmap v8078: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 47 KiB/s wr, 7 op/s 2023-03-09T18:54:35.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:54:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14186.166593 secs 2023-03-09T18:54:36.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:54:35 smithi038 ceph-mon[92422]: pgmap v8078: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 47 KiB/s wr, 7 op/s 2023-03-09T18:54:36.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:54:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14186.166593 secs 2023-03-09T18:54:37.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:54:37 smithi122 ceph-mon[93869]: pgmap v8079: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 47 KiB/s wr, 10 op/s 2023-03-09T18:54:37.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:54:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14188.447859 secs 2023-03-09T18:54:37.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:54:37 smithi181 ceph-mon[94189]: pgmap v8079: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 47 KiB/s wr, 10 op/s 2023-03-09T18:54:37.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:54:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14188.447859 secs 2023-03-09T18:54:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:54:37 smithi038 ceph-mon[92422]: pgmap v8079: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 47 KiB/s wr, 10 op/s 2023-03-09T18:54:38.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:54:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14188.447859 secs 2023-03-09T18:54:38.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14650.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:54:39.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:54:39 smithi122 ceph-mon[93869]: pgmap v8080: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 95 KiB/s wr, 8 op/s 2023-03-09T18:54:39.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:54:39 smithi181 ceph-mon[94189]: pgmap v8080: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 95 KiB/s wr, 8 op/s 2023-03-09T18:54:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:54:39 smithi038 ceph-mon[92422]: pgmap v8080: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 95 KiB/s wr, 8 op/s 2023-03-09T18:54:40.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:54:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14191.166653 secs 2023-03-09T18:54:40.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:54:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14191.166653 secs 2023-03-09T18:54:41.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:54:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14191.166653 secs 2023-03-09T18:54:41.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:54:41 smithi122 ceph-mon[93869]: pgmap v8081: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 95 KiB/s wr, 7 op/s 2023-03-09T18:54:41.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:54:41 smithi181 ceph-mon[94189]: pgmap v8081: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 95 KiB/s wr, 7 op/s 2023-03-09T18:54:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:54:41 smithi038 ceph-mon[92422]: pgmap v8081: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 95 KiB/s wr, 7 op/s 2023-03-09T18:54:42.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:54:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14193.447983 secs 2023-03-09T18:54:42.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:54:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14193.447983 secs 2023-03-09T18:54:42.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:54:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14193.447983 secs 2023-03-09T18:54:43.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14655.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:54:43.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:54:43 smithi122 ceph-mon[93869]: pgmap v8082: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 245 KiB/s wr, 10 op/s 2023-03-09T18:54:43.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:54:43 smithi181 ceph-mon[94189]: pgmap v8082: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 245 KiB/s wr, 10 op/s 2023-03-09T18:54:44.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:54:43 smithi038 ceph-mon[92422]: pgmap v8082: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 245 KiB/s wr, 10 op/s 2023-03-09T18:54:45.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:54:45 smithi122 ceph-mon[93869]: pgmap v8083: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 199 KiB/s wr, 7 op/s 2023-03-09T18:54:45.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:54:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14196.166715 secs 2023-03-09T18:54:45.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:54:45 smithi181 ceph-mon[94189]: pgmap v8083: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 199 KiB/s wr, 7 op/s 2023-03-09T18:54:45.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:54:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14196.166715 secs 2023-03-09T18:54:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:54:45 smithi038 ceph-mon[92422]: pgmap v8083: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 199 KiB/s wr, 7 op/s 2023-03-09T18:54:46.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:54:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14196.166715 secs 2023-03-09T18:54:47.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:54:47 smithi122 ceph-mon[93869]: pgmap v8084: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 212 KiB/s wr, 11 op/s 2023-03-09T18:54:47.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:54:47 smithi181 ceph-mon[94189]: pgmap v8084: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 212 KiB/s wr, 11 op/s 2023-03-09T18:54:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:54:47 smithi038 ceph-mon[92422]: pgmap v8084: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 212 KiB/s wr, 11 op/s 2023-03-09T18:54:48.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14660.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:54:48.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:54:48 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:54:48.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:54:48 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:54:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:54:48 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:54:49.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:54:49 smithi122 ceph-mon[93869]: pgmap v8085: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 306 KiB/s wr, 10 op/s 2023-03-09T18:54:49.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:54:49 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:54:49.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:54:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:54:49.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:54:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:54:49.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:54:49 smithi181 ceph-mon[94189]: pgmap v8085: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 306 KiB/s wr, 10 op/s 2023-03-09T18:54:49.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:54:49 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:54:49.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:54:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:54:49.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:54:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:54:50.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:54:49 smithi038 ceph-mon[92422]: pgmap v8085: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 306 KiB/s wr, 10 op/s 2023-03-09T18:54:50.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:54:49 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:54:50.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:54:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:54:50.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:54:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:54:50.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:54:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14201.166774 secs 2023-03-09T18:54:50.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:54:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14201.166774 secs 2023-03-09T18:54:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:54:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14201.166774 secs 2023-03-09T18:54:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:54:51 smithi122 ceph-mon[93869]: pgmap v8086: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 340 KiB/s wr, 10 op/s 2023-03-09T18:54:51.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:54:51 smithi181 ceph-mon[94189]: pgmap v8086: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 340 KiB/s wr, 10 op/s 2023-03-09T18:54:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:54:51 smithi038 ceph-mon[92422]: pgmap v8086: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 340 KiB/s wr, 10 op/s 2023-03-09T18:54:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:54:53 smithi122 ceph-mon[93869]: pgmap v8087: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 352 KiB/s wr, 14 op/s 2023-03-09T18:54:53.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:54:53 smithi181 ceph-mon[94189]: pgmap v8087: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 352 KiB/s wr, 14 op/s 2023-03-09T18:54:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:54:53 smithi038 ceph-mon[92422]: pgmap v8087: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 352 KiB/s wr, 14 op/s 2023-03-09T18:54:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14665.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:54:55.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:54:55 smithi122 ceph-mon[93869]: pgmap v8088: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 239 KiB/s wr, 11 op/s 2023-03-09T18:54:55.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:54:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14206.166834 secs 2023-03-09T18:54:55.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:54:55 smithi181 ceph-mon[94189]: pgmap v8088: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 239 KiB/s wr, 11 op/s 2023-03-09T18:54:55.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:54:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14206.166834 secs 2023-03-09T18:54:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:54:55 smithi038 ceph-mon[92422]: pgmap v8088: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 239 KiB/s wr, 11 op/s 2023-03-09T18:54:55.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:54:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14206.166834 secs 2023-03-09T18:54:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:54:57 smithi122 ceph-mon[93869]: pgmap v8089: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 260 KiB/s wr, 13 op/s 2023-03-09T18:54:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:54:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14208.448370 secs 2023-03-09T18:54:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:54:57 smithi181 ceph-mon[94189]: pgmap v8089: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 260 KiB/s wr, 13 op/s 2023-03-09T18:54:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:54:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14208.448370 secs 2023-03-09T18:54:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:54:57 smithi038 ceph-mon[92422]: pgmap v8089: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 260 KiB/s wr, 13 op/s 2023-03-09T18:54:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:54:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14208.448370 secs 2023-03-09T18:54:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14670.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:54:59.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:54:59 smithi122 ceph-mon[93869]: pgmap v8090: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 247 KiB/s wr, 11 op/s 2023-03-09T18:54:59.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:54:59 smithi181 ceph-mon[94189]: pgmap v8090: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 247 KiB/s wr, 11 op/s 2023-03-09T18:54:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:54:59 smithi038 ceph-mon[92422]: pgmap v8090: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 247 KiB/s wr, 11 op/s 2023-03-09T18:55:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:55:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:55:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:55:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14211.166897 secs 2023-03-09T18:55:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:55:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:55:00.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:55:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:55:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:55:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:55:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:55:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14211.166897 secs 2023-03-09T18:55:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:55:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:55:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:55:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:55:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:55:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:55:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:55:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14211.166897 secs 2023-03-09T18:55:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:55:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:55:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:55:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:55:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:55:01 smithi122 ceph-mon[93869]: pgmap v8091: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 285 KiB/s wr, 10 op/s 2023-03-09T18:55:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:55:01 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:55:01.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:55:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:55:01.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:55:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:55:01.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:55:01 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:55:01.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:55:01 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:55:01.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:55:01 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:55:01.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:55:01 smithi181 ceph-mon[94189]: pgmap v8091: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 285 KiB/s wr, 10 op/s 2023-03-09T18:55:01.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:55:01 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:55:01.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:55:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:55:01.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:55:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:55:01.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:55:01 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:55:01.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:55:01 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:55:01.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:55:01 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:55:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:55:01 smithi038 ceph-mon[92422]: pgmap v8091: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 285 KiB/s wr, 10 op/s 2023-03-09T18:55:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:55:01 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:55:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:55:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:55:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:55:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:55:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:55:01 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:55:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:55:01 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:55:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:55:01 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:55:02.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:55:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14213.448507 secs 2023-03-09T18:55:02.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:55:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14213.448507 secs 2023-03-09T18:55:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:55:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14213.448507 secs 2023-03-09T18:55:03.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:55:03 smithi122 ceph-mon[93869]: pgmap v8092: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 226 KiB/s wr, 15 op/s 2023-03-09T18:55:03.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:55:03 smithi181 ceph-mon[94189]: pgmap v8092: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 226 KiB/s wr, 15 op/s 2023-03-09T18:55:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:55:03 smithi038 ceph-mon[92422]: pgmap v8092: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 226 KiB/s wr, 15 op/s 2023-03-09T18:55:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14675.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:55:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:55:05 smithi122 ceph-mon[93869]: pgmap v8093: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 256 KiB/s wr, 24 op/s 2023-03-09T18:55:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:55:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14216.166954 secs 2023-03-09T18:55:05.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:55:05 smithi181 ceph-mon[94189]: pgmap v8093: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 256 KiB/s wr, 24 op/s 2023-03-09T18:55:05.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:55:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14216.166954 secs 2023-03-09T18:55:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:55:05 smithi038 ceph-mon[92422]: pgmap v8093: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 256 KiB/s wr, 24 op/s 2023-03-09T18:55:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:55:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14216.166954 secs 2023-03-09T18:55:07.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:55:07 smithi122 ceph-mon[93869]: pgmap v8094: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 219 KiB/s wr, 25 op/s 2023-03-09T18:55:07.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:55:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14218.448633 secs 2023-03-09T18:55:07.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:55:07 smithi181 ceph-mon[94189]: pgmap v8094: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 219 KiB/s wr, 25 op/s 2023-03-09T18:55:07.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:55:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14218.448633 secs 2023-03-09T18:55:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:55:07 smithi038 ceph-mon[92422]: pgmap v8094: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 219 KiB/s wr, 25 op/s 2023-03-09T18:55:07.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:55:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14218.448633 secs 2023-03-09T18:55:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14680.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:55:09.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:55:09 smithi122 ceph-mon[93869]: pgmap v8095: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 198 KiB/s wr, 23 op/s 2023-03-09T18:55:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:55:09 smithi181 ceph-mon[94189]: pgmap v8095: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 198 KiB/s wr, 23 op/s 2023-03-09T18:55:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:55:09 smithi038 ceph-mon[92422]: pgmap v8095: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 198 KiB/s wr, 23 op/s 2023-03-09T18:55:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:55:10 smithi122 ceph-mon[93869]: osdmap e719: 12 total, 12 up, 12 in 2023-03-09T18:55:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:55:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14221.166982 secs 2023-03-09T18:55:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:55:10 smithi181 ceph-mon[94189]: osdmap e719: 12 total, 12 up, 12 in 2023-03-09T18:55:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:55:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14221.166982 secs 2023-03-09T18:55:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:55:10 smithi038 ceph-mon[92422]: osdmap e719: 12 total, 12 up, 12 in 2023-03-09T18:55:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:55:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14221.166982 secs 2023-03-09T18:55:11.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:55:11 smithi122 ceph-mon[93869]: pgmap v8097: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 131 KiB/s wr, 26 op/s 2023-03-09T18:55:11.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:55:11 smithi181 ceph-mon[94189]: pgmap v8097: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 131 KiB/s wr, 26 op/s 2023-03-09T18:55:11.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:55:11 smithi038 ceph-mon[92422]: pgmap v8097: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 131 KiB/s wr, 26 op/s 2023-03-09T18:55:12.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:55:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14223.448753 secs 2023-03-09T18:55:12.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:55:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14223.448753 secs 2023-03-09T18:55:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:55:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14223.448753 secs 2023-03-09T18:55:13.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:55:13 smithi122 ceph-mon[93869]: pgmap v8098: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 103 KiB/s wr, 22 op/s 2023-03-09T18:55:13.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:55:13 smithi181 ceph-mon[94189]: pgmap v8098: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 103 KiB/s wr, 22 op/s 2023-03-09T18:55:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:55:13 smithi038 ceph-mon[92422]: pgmap v8098: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 103 KiB/s wr, 22 op/s 2023-03-09T18:55:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14685.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:55:15.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:55:15 smithi122 ceph-mon[93869]: pgmap v8099: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 52 KiB/s wr, 6 op/s 2023-03-09T18:55:15.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:55:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14226.167041 secs 2023-03-09T18:55:15.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:55:15 smithi181 ceph-mon[94189]: pgmap v8099: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 52 KiB/s wr, 6 op/s 2023-03-09T18:55:15.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:55:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14226.167041 secs 2023-03-09T18:55:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:55:15 smithi038 ceph-mon[92422]: pgmap v8099: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 52 KiB/s wr, 6 op/s 2023-03-09T18:55:15.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:55:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14226.167041 secs 2023-03-09T18:55:17.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:55:17 smithi122 ceph-mon[93869]: pgmap v8100: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 51 KiB/s wr, 6 op/s 2023-03-09T18:55:17.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:55:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14228.448871 secs 2023-03-09T18:55:17.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:55:17 smithi122 ceph-mon[93869]: osdmap e720: 12 total, 12 up, 12 in 2023-03-09T18:55:17.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:55:17 smithi181 ceph-mon[94189]: pgmap v8100: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 51 KiB/s wr, 6 op/s 2023-03-09T18:55:17.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:55:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14228.448871 secs 2023-03-09T18:55:17.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:55:17 smithi181 ceph-mon[94189]: osdmap e720: 12 total, 12 up, 12 in 2023-03-09T18:55:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:55:17 smithi038 ceph-mon[92422]: pgmap v8100: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 51 KiB/s wr, 6 op/s 2023-03-09T18:55:17.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:55:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14228.448871 secs 2023-03-09T18:55:17.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:55:17 smithi038 ceph-mon[92422]: osdmap e720: 12 total, 12 up, 12 in 2023-03-09T18:55:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14690.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:55:19.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:55:19 smithi122 ceph-mon[93869]: pgmap v8102: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 114 B/s wr, 6 op/s 2023-03-09T18:55:19.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:55:19 smithi181 ceph-mon[94189]: pgmap v8102: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 114 B/s wr, 6 op/s 2023-03-09T18:55:19.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:55:19 smithi038 ceph-mon[92422]: pgmap v8102: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 114 B/s wr, 6 op/s 2023-03-09T18:55:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:55:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14231.167105 secs 2023-03-09T18:55:20.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:55:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14231.167105 secs 2023-03-09T18:55:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:55:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14231.167105 secs 2023-03-09T18:55:21.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:55:21 smithi122 ceph-mon[93869]: pgmap v8103: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T18:55:21.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:55:21 smithi181 ceph-mon[94189]: pgmap v8103: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T18:55:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:55:21 smithi038 ceph-mon[92422]: pgmap v8103: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T18:55:22.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:55:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14233.448987 secs 2023-03-09T18:55:22.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:55:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14233.448987 secs 2023-03-09T18:55:22.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:55:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14233.448987 secs 2023-03-09T18:55:23.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:55:23 smithi122 ceph-mon[93869]: pgmap v8104: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T18:55:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:55:23 smithi181 ceph-mon[94189]: pgmap v8104: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T18:55:23.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:55:23 smithi038 ceph-mon[92422]: pgmap v8104: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T18:55:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14695.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:55:25.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:55:25 smithi122 ceph-mon[93869]: pgmap v8105: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 33 KiB/s wr, 6 op/s 2023-03-09T18:55:25.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:55:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14236.167160 secs 2023-03-09T18:55:25.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:55:25 smithi181 ceph-mon[94189]: pgmap v8105: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 33 KiB/s wr, 6 op/s 2023-03-09T18:55:25.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:55:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14236.167160 secs 2023-03-09T18:55:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:55:25 smithi038 ceph-mon[92422]: pgmap v8105: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 33 KiB/s wr, 6 op/s 2023-03-09T18:55:25.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:55:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14236.167160 secs 2023-03-09T18:55:27.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:55:27 smithi122 ceph-mon[93869]: pgmap v8106: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 34 KiB/s wr, 6 op/s 2023-03-09T18:55:27.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:55:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14238.449118 secs 2023-03-09T18:55:27.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:55:27 smithi181 ceph-mon[94189]: pgmap v8106: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 34 KiB/s wr, 6 op/s 2023-03-09T18:55:27.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:55:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14238.449118 secs 2023-03-09T18:55:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:55:27 smithi038 ceph-mon[92422]: pgmap v8106: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 34 KiB/s wr, 6 op/s 2023-03-09T18:55:27.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:55:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14238.449118 secs 2023-03-09T18:55:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14700.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:55:29.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:55:29 smithi122 ceph-mon[93869]: pgmap v8107: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 92 KiB/s wr, 7 op/s 2023-03-09T18:55:29.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:55:29 smithi181 ceph-mon[94189]: pgmap v8107: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 92 KiB/s wr, 7 op/s 2023-03-09T18:55:29.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:55:29 smithi038 ceph-mon[92422]: pgmap v8107: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 92 KiB/s wr, 7 op/s 2023-03-09T18:55:30.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:55:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14241.167220 secs 2023-03-09T18:55:30.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:55:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14241.167220 secs 2023-03-09T18:55:30.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:55:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14241.167220 secs 2023-03-09T18:55:31.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:55:31 smithi122 ceph-mon[93869]: pgmap v8108: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 84 KiB/s wr, 7 op/s 2023-03-09T18:55:31.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:55:31 smithi181 ceph-mon[94189]: pgmap v8108: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 84 KiB/s wr, 7 op/s 2023-03-09T18:55:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:55:31 smithi038 ceph-mon[92422]: pgmap v8108: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 84 KiB/s wr, 7 op/s 2023-03-09T18:55:32.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:55:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14243.449220 secs 2023-03-09T18:55:32.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:55:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14243.449220 secs 2023-03-09T18:55:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:55:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14243.449220 secs 2023-03-09T18:55:33.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:55:33 smithi122 ceph-mon[93869]: pgmap v8109: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 84 KiB/s wr, 9 op/s 2023-03-09T18:55:33.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:55:33 smithi181 ceph-mon[94189]: pgmap v8109: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 84 KiB/s wr, 9 op/s 2023-03-09T18:55:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:55:33 smithi038 ceph-mon[92422]: pgmap v8109: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 84 KiB/s wr, 9 op/s 2023-03-09T18:55:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14705.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:55:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:55:35 smithi122 ceph-mon[93869]: pgmap v8110: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 84 KiB/s wr, 8 op/s 2023-03-09T18:55:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:55:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14246.167289 secs 2023-03-09T18:55:35.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:55:35 smithi181 ceph-mon[94189]: pgmap v8110: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 84 KiB/s wr, 8 op/s 2023-03-09T18:55:35.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:55:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14246.167289 secs 2023-03-09T18:55:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:55:35 smithi038 ceph-mon[92422]: pgmap v8110: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 84 KiB/s wr, 8 op/s 2023-03-09T18:55:35.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:55:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14246.167289 secs 2023-03-09T18:55:37.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:55:37 smithi122 ceph-mon[93869]: pgmap v8111: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 57 KiB/s wr, 9 op/s 2023-03-09T18:55:37.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:55:37 smithi181 ceph-mon[94189]: pgmap v8111: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 57 KiB/s wr, 9 op/s 2023-03-09T18:55:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:55:37 smithi038 ceph-mon[92422]: pgmap v8111: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 57 KiB/s wr, 9 op/s 2023-03-09T18:55:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14710.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:55:39.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:55:39 smithi122 ceph-mon[93869]: pgmap v8112: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 105 KiB/s wr, 8 op/s 2023-03-09T18:55:39.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:55:39 smithi181 ceph-mon[94189]: pgmap v8112: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 105 KiB/s wr, 8 op/s 2023-03-09T18:55:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:55:39 smithi038 ceph-mon[92422]: pgmap v8112: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 105 KiB/s wr, 8 op/s 2023-03-09T18:55:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:55:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14251.167351 secs 2023-03-09T18:55:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:55:40 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:55:40.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:55:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14251.167351 secs 2023-03-09T18:55:40.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:55:40 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:55:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:55:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14251.167351 secs 2023-03-09T18:55:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:55:40 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:55:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:55:41 smithi122 ceph-mon[93869]: pgmap v8113: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 49 KiB/s wr, 7 op/s 2023-03-09T18:55:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:55:41 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:55:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:55:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:55:41.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:55:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:55:41.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:55:41 smithi181 ceph-mon[94189]: pgmap v8113: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 49 KiB/s wr, 7 op/s 2023-03-09T18:55:41.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:55:41 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:55:41.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:55:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:55:41.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:55:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:55:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:55:41 smithi038 ceph-mon[92422]: pgmap v8113: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 49 KiB/s wr, 7 op/s 2023-03-09T18:55:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:55:41 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:55:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:55:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:55:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:55:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:55:43.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:55:43 smithi122 ceph-mon[93869]: pgmap v8114: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 60 KiB/s wr, 9 op/s 2023-03-09T18:55:43.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:55:43 smithi181 ceph-mon[94189]: pgmap v8114: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 60 KiB/s wr, 9 op/s 2023-03-09T18:55:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:55:43 smithi038 ceph-mon[92422]: pgmap v8114: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 60 KiB/s wr, 9 op/s 2023-03-09T18:55:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14715.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:55:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:55:45 smithi122 ceph-mon[93869]: pgmap v8115: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 200 KiB/s wr, 8 op/s 2023-03-09T18:55:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:55:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14256.167405 secs 2023-03-09T18:55:45.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:55:45 smithi181 ceph-mon[94189]: pgmap v8115: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 200 KiB/s wr, 8 op/s 2023-03-09T18:55:45.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:55:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14256.167405 secs 2023-03-09T18:55:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:55:45 smithi038 ceph-mon[92422]: pgmap v8115: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 200 KiB/s wr, 8 op/s 2023-03-09T18:55:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:55:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14256.167405 secs 2023-03-09T18:55:47.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:55:47 smithi122 ceph-mon[93869]: pgmap v8116: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 214 KiB/s wr, 10 op/s 2023-03-09T18:55:47.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:55:47 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 14258.449538 secs 2023-03-09T18:55:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:55:47 smithi181 ceph-mon[94189]: pgmap v8116: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 214 KiB/s wr, 10 op/s 2023-03-09T18:55:47.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:55:47 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 14258.449538 secs 2023-03-09T18:55:47.753 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:55:47 smithi038 ceph-mon[92422]: pgmap v8116: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 214 KiB/s wr, 10 op/s 2023-03-09T18:55:47.753 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:55:47 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 14258.449538 secs 2023-03-09T18:55:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:55:48 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:55:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:55:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:55:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:55:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:55:48.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:55:48 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:55:48.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:55:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:55:48.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:55:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:55:48.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:55:48 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:55:48.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:55:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:55:48.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:55:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:55:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14720.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:55:49.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:55:49 smithi122 ceph-mon[93869]: pgmap v8117: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 326 KiB/s rd, 312 KiB/s wr, 9 op/s 2023-03-09T18:55:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:55:49 smithi181 ceph-mon[94189]: pgmap v8117: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 326 KiB/s rd, 312 KiB/s wr, 9 op/s 2023-03-09T18:55:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:55:49 smithi038 ceph-mon[92422]: pgmap v8117: 129 pgs: 129 active+clean; 9.0 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 326 KiB/s rd, 312 KiB/s wr, 9 op/s 2023-03-09T18:55:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:55:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14261.167485 secs 2023-03-09T18:55:50.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:55:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14261.167485 secs 2023-03-09T18:55:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:55:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14261.167485 secs 2023-03-09T18:55:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:55:51 smithi122 ceph-mon[93869]: pgmap v8118: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 379 KiB/s wr, 10 op/s 2023-03-09T18:55:51.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:55:51 smithi181 ceph-mon[94189]: pgmap v8118: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 379 KiB/s wr, 10 op/s 2023-03-09T18:55:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:55:51 smithi038 ceph-mon[92422]: pgmap v8118: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 379 KiB/s wr, 10 op/s 2023-03-09T18:55:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:55:52 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 14263.449617 secs 2023-03-09T18:55:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:55:52 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 14263.449617 secs 2023-03-09T18:55:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:55:52 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 14263.449617 secs 2023-03-09T18:55:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:55:53 smithi122 ceph-mon[93869]: pgmap v8119: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 406 KiB/s rd, 391 KiB/s wr, 13 op/s 2023-03-09T18:55:53.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:55:53 smithi181 ceph-mon[94189]: pgmap v8119: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 406 KiB/s rd, 391 KiB/s wr, 13 op/s 2023-03-09T18:55:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:55:53 smithi038 ceph-mon[92422]: pgmap v8119: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 406 KiB/s rd, 391 KiB/s wr, 13 op/s 2023-03-09T18:55:53.801 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14725.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:55:55.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:55:55 smithi122 ceph-mon[93869]: pgmap v8120: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 464 KiB/s rd, 451 KiB/s wr, 12 op/s 2023-03-09T18:55:55.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:55:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14266.167558 secs 2023-03-09T18:55:55.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:55:55 smithi181 ceph-mon[94189]: pgmap v8120: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 464 KiB/s rd, 451 KiB/s wr, 12 op/s 2023-03-09T18:55:55.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:55:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14266.167558 secs 2023-03-09T18:55:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:55:55 smithi038 ceph-mon[92422]: pgmap v8120: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 464 KiB/s rd, 451 KiB/s wr, 12 op/s 2023-03-09T18:55:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:55:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14266.167558 secs 2023-03-09T18:55:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:55:57 smithi122 ceph-mon[93869]: pgmap v8121: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 336 KiB/s rd, 333 KiB/s wr, 13 op/s 2023-03-09T18:55:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:55:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14268.449761 secs 2023-03-09T18:55:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:55:57 smithi181 ceph-mon[94189]: pgmap v8121: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 336 KiB/s rd, 333 KiB/s wr, 13 op/s 2023-03-09T18:55:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:55:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14268.449761 secs 2023-03-09T18:55:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:55:57 smithi038 ceph-mon[92422]: pgmap v8121: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 336 KiB/s rd, 333 KiB/s wr, 13 op/s 2023-03-09T18:55:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:55:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14268.449761 secs 2023-03-09T18:55:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14730.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:55:59.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:55:59 smithi122 ceph-mon[93869]: pgmap v8122: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 319 KiB/s wr, 11 op/s 2023-03-09T18:55:59.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:55:59 smithi181 ceph-mon[94189]: pgmap v8122: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 319 KiB/s wr, 11 op/s 2023-03-09T18:55:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:55:59 smithi038 ceph-mon[92422]: pgmap v8122: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 319 KiB/s wr, 11 op/s 2023-03-09T18:56:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:56:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14271.167619 secs 2023-03-09T18:56:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:56:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14271.167619 secs 2023-03-09T18:56:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:56:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14271.167619 secs 2023-03-09T18:56:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:56:01 smithi122 ceph-mon[93869]: pgmap v8123: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 303 KiB/s wr, 10 op/s 2023-03-09T18:56:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:56:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:56:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:56:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:56:01.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:56:01 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:56:01.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:56:01 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:56:01.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:56:01 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:56:01.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:56:01 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:56:01.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:56:01 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:56:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:56:01 smithi181 ceph-mon[94189]: pgmap v8123: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 303 KiB/s wr, 10 op/s 2023-03-09T18:56:01.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:56:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:56:01.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:56:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:56:01.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:56:01 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:56:01.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:56:01 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:56:01.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:56:01 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:56:01.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:56:01 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:56:01.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:56:01 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:56:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:56:01 smithi038 ceph-mon[92422]: pgmap v8123: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 303 KiB/s wr, 10 op/s 2023-03-09T18:56:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:56:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:56:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:56:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:56:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:56:01 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:56:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:56:01 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:56:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:56:01 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:56:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:56:01 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:56:01.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:56:01 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:56:02.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:56:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14273.449889 secs 2023-03-09T18:56:02.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:56:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:56:02.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:56:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:56:02.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:56:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:56:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:56:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14273.449889 secs 2023-03-09T18:56:02.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:56:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:56:02.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:56:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:56:02.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:56:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:56:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:56:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14273.449889 secs 2023-03-09T18:56:02.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:56:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:56:02.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:56:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:56:02.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:56:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:56:03.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:56:03 smithi122 ceph-mon[93869]: pgmap v8124: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 210 KiB/s wr, 14 op/s 2023-03-09T18:56:03.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:56:03 smithi181 ceph-mon[94189]: pgmap v8124: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 210 KiB/s wr, 14 op/s 2023-03-09T18:56:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:56:03 smithi038 ceph-mon[92422]: pgmap v8124: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 210 KiB/s wr, 14 op/s 2023-03-09T18:56:03.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14735.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:56:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:56:05 smithi122 ceph-mon[93869]: pgmap v8125: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 263 KiB/s wr, 24 op/s 2023-03-09T18:56:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:56:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14276.167699 secs 2023-03-09T18:56:05.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:56:05 smithi181 ceph-mon[94189]: pgmap v8125: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 263 KiB/s wr, 24 op/s 2023-03-09T18:56:05.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:56:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14276.167699 secs 2023-03-09T18:56:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:56:05 smithi038 ceph-mon[92422]: pgmap v8125: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 263 KiB/s wr, 24 op/s 2023-03-09T18:56:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:56:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14276.167699 secs 2023-03-09T18:56:07.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:56:07 smithi122 ceph-mon[93869]: pgmap v8126: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 193 KiB/s wr, 24 op/s 2023-03-09T18:56:07.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:56:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14278.449988 secs 2023-03-09T18:56:07.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:56:07 smithi181 ceph-mon[94189]: pgmap v8126: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 193 KiB/s wr, 24 op/s 2023-03-09T18:56:07.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:56:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14278.449988 secs 2023-03-09T18:56:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:56:07 smithi038 ceph-mon[92422]: pgmap v8126: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 193 KiB/s wr, 24 op/s 2023-03-09T18:56:07.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:56:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14278.449988 secs 2023-03-09T18:56:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14740.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:56:09.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:56:09 smithi181 ceph-mon[94189]: pgmap v8127: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 171 KiB/s wr, 22 op/s 2023-03-09T18:56:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:56:09 smithi038 ceph-mon[92422]: pgmap v8127: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 171 KiB/s wr, 22 op/s 2023-03-09T18:56:09.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:56:09 smithi122 ceph-mon[93869]: pgmap v8127: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 171 KiB/s wr, 22 op/s 2023-03-09T18:56:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:56:10 smithi038 ceph-mon[92422]: osdmap e721: 12 total, 12 up, 12 in 2023-03-09T18:56:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:56:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14281.167762 secs 2023-03-09T18:56:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:56:10 smithi122 ceph-mon[93869]: osdmap e721: 12 total, 12 up, 12 in 2023-03-09T18:56:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:56:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14281.167762 secs 2023-03-09T18:56:10.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:56:10 smithi181 ceph-mon[94189]: osdmap e721: 12 total, 12 up, 12 in 2023-03-09T18:56:10.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:56:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14281.167762 secs 2023-03-09T18:56:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:56:11 smithi038 ceph-mon[92422]: pgmap v8129: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 106 KiB/s wr, 25 op/s 2023-03-09T18:56:11.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:56:11 smithi122 ceph-mon[93869]: pgmap v8129: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 106 KiB/s wr, 25 op/s 2023-03-09T18:56:11.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:56:11 smithi181 ceph-mon[94189]: pgmap v8129: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 106 KiB/s wr, 25 op/s 2023-03-09T18:56:12.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:56:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14283.450115 secs 2023-03-09T18:56:12.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:56:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14283.450115 secs 2023-03-09T18:56:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:56:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14283.450115 secs 2023-03-09T18:56:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14745.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:56:13.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:56:13 smithi122 ceph-mon[93869]: pgmap v8130: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 79 KiB/s wr, 22 op/s 2023-03-09T18:56:13.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:56:13 smithi181 ceph-mon[94189]: pgmap v8130: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 79 KiB/s wr, 22 op/s 2023-03-09T18:56:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:56:13 smithi038 ceph-mon[92422]: pgmap v8130: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 79 KiB/s wr, 22 op/s 2023-03-09T18:56:15.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:56:15 smithi122 ceph-mon[93869]: pgmap v8131: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 716 B/s wr, 6 op/s 2023-03-09T18:56:15.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:56:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14286.167822 secs 2023-03-09T18:56:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:56:15 smithi181 ceph-mon[94189]: pgmap v8131: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 716 B/s wr, 6 op/s 2023-03-09T18:56:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:56:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14286.167822 secs 2023-03-09T18:56:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:56:15 smithi038 ceph-mon[92422]: pgmap v8131: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 716 B/s wr, 6 op/s 2023-03-09T18:56:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:56:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14286.167822 secs 2023-03-09T18:56:17.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:56:17 smithi122 ceph-mon[93869]: pgmap v8132: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2023-03-09T18:56:17.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:56:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14288.450215 secs 2023-03-09T18:56:17.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:56:17 smithi122 ceph-mon[93869]: osdmap e722: 12 total, 12 up, 12 in 2023-03-09T18:56:17.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:56:17 smithi181 ceph-mon[94189]: pgmap v8132: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2023-03-09T18:56:17.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:56:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14288.450215 secs 2023-03-09T18:56:17.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:56:17 smithi181 ceph-mon[94189]: osdmap e722: 12 total, 12 up, 12 in 2023-03-09T18:56:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:56:17 smithi038 ceph-mon[92422]: pgmap v8132: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2023-03-09T18:56:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:56:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14288.450215 secs 2023-03-09T18:56:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:56:17 smithi038 ceph-mon[92422]: osdmap e722: 12 total, 12 up, 12 in 2023-03-09T18:56:18.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14750.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:56:19.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:56:19 smithi122 ceph-mon[93869]: pgmap v8134: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 232 B/s wr, 6 op/s 2023-03-09T18:56:19.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:56:19 smithi181 ceph-mon[94189]: pgmap v8134: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 232 B/s wr, 6 op/s 2023-03-09T18:56:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:56:19 smithi038 ceph-mon[92422]: pgmap v8134: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 232 B/s wr, 6 op/s 2023-03-09T18:56:20.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:56:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14291.167883 secs 2023-03-09T18:56:20.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:56:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14291.167883 secs 2023-03-09T18:56:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:56:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14291.167883 secs 2023-03-09T18:56:21.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:56:21 smithi122 ceph-mon[93869]: pgmap v8135: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T18:56:21.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:56:21 smithi181 ceph-mon[94189]: pgmap v8135: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T18:56:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:56:21 smithi038 ceph-mon[92422]: pgmap v8135: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T18:56:22.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:56:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14293.450312 secs 2023-03-09T18:56:22.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:56:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14293.450312 secs 2023-03-09T18:56:22.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:56:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14293.450312 secs 2023-03-09T18:56:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14755.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:56:23.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:56:23 smithi122 ceph-mon[93869]: pgmap v8136: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T18:56:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:56:23 smithi181 ceph-mon[94189]: pgmap v8136: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T18:56:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:56:23 smithi038 ceph-mon[92422]: pgmap v8136: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T18:56:25.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:56:25 smithi122 ceph-mon[93869]: pgmap v8137: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-09T18:56:25.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:56:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14296.167957 secs 2023-03-09T18:56:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:56:25 smithi181 ceph-mon[94189]: pgmap v8137: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-09T18:56:25.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:56:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14296.167957 secs 2023-03-09T18:56:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:56:25 smithi038 ceph-mon[92422]: pgmap v8137: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-09T18:56:26.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:56:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14296.167957 secs 2023-03-09T18:56:27.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:56:27 smithi122 ceph-mon[93869]: pgmap v8138: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 409 B/s wr, 6 op/s 2023-03-09T18:56:27.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:56:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14298.450394 secs 2023-03-09T18:56:27.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:56:27 smithi181 ceph-mon[94189]: pgmap v8138: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 409 B/s wr, 6 op/s 2023-03-09T18:56:27.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:56:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14298.450394 secs 2023-03-09T18:56:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:56:27 smithi038 ceph-mon[92422]: pgmap v8138: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 409 B/s wr, 6 op/s 2023-03-09T18:56:28.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:56:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14298.450394 secs 2023-03-09T18:56:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14760.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:56:29.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:56:29 smithi122 ceph-mon[93869]: pgmap v8139: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 464 B/s wr, 6 op/s 2023-03-09T18:56:29.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:56:29 smithi181 ceph-mon[94189]: pgmap v8139: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 464 B/s wr, 6 op/s 2023-03-09T18:56:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:56:29 smithi038 ceph-mon[92422]: pgmap v8139: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 464 B/s wr, 6 op/s 2023-03-09T18:56:30.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:56:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14301.168011 secs 2023-03-09T18:56:30.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:56:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14301.168011 secs 2023-03-09T18:56:30.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:56:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14301.168011 secs 2023-03-09T18:56:31.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:56:31 smithi122 ceph-mon[93869]: pgmap v8140: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 6 op/s 2023-03-09T18:56:31.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:56:31 smithi181 ceph-mon[94189]: pgmap v8140: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 6 op/s 2023-03-09T18:56:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:56:31 smithi038 ceph-mon[92422]: pgmap v8140: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 6 op/s 2023-03-09T18:56:32.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:56:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14303.450511 secs 2023-03-09T18:56:32.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:56:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14303.450511 secs 2023-03-09T18:56:32.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:56:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14303.450511 secs 2023-03-09T18:56:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14765.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:56:33.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:56:33 smithi122 ceph-mon[93869]: pgmap v8141: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 47 KiB/s wr, 8 op/s 2023-03-09T18:56:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:56:33 smithi181 ceph-mon[94189]: pgmap v8141: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 47 KiB/s wr, 8 op/s 2023-03-09T18:56:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:56:33 smithi038 ceph-mon[92422]: pgmap v8141: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 47 KiB/s wr, 8 op/s 2023-03-09T18:56:35.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:56:35 smithi122 ceph-mon[93869]: pgmap v8142: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 47 KiB/s wr, 7 op/s 2023-03-09T18:56:35.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:56:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14306.168072 secs 2023-03-09T18:56:35.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:56:35 smithi181 ceph-mon[94189]: pgmap v8142: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 47 KiB/s wr, 7 op/s 2023-03-09T18:56:35.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:56:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14306.168072 secs 2023-03-09T18:56:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:56:35 smithi038 ceph-mon[92422]: pgmap v8142: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 47 KiB/s wr, 7 op/s 2023-03-09T18:56:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:56:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14306.168072 secs 2023-03-09T18:56:37.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:56:37 smithi122 ceph-mon[93869]: pgmap v8143: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 48 KiB/s wr, 9 op/s 2023-03-09T18:56:37.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:56:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14308.450627 secs 2023-03-09T18:56:37.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:56:37 smithi181 ceph-mon[94189]: pgmap v8143: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 48 KiB/s wr, 9 op/s 2023-03-09T18:56:37.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:56:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14308.450627 secs 2023-03-09T18:56:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:56:37 smithi038 ceph-mon[92422]: pgmap v8143: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 48 KiB/s wr, 9 op/s 2023-03-09T18:56:38.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:56:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14308.450627 secs 2023-03-09T18:56:38.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14770.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:56:39.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:56:39 smithi122 ceph-mon[93869]: pgmap v8144: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 96 KiB/s wr, 7 op/s 2023-03-09T18:56:39.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:56:39 smithi181 ceph-mon[94189]: pgmap v8144: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 96 KiB/s wr, 7 op/s 2023-03-09T18:56:40.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:56:39 smithi038 ceph-mon[92422]: pgmap v8144: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 96 KiB/s wr, 7 op/s 2023-03-09T18:56:40.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:56:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14311.168143 secs 2023-03-09T18:56:40.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:56:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14311.168143 secs 2023-03-09T18:56:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:56:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14311.168143 secs 2023-03-09T18:56:41.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:56:41 smithi122 ceph-mon[93869]: pgmap v8145: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 96 KiB/s wr, 7 op/s 2023-03-09T18:56:41.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:56:41 smithi181 ceph-mon[94189]: pgmap v8145: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 96 KiB/s wr, 7 op/s 2023-03-09T18:56:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:56:41 smithi038 ceph-mon[92422]: pgmap v8145: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 96 KiB/s wr, 7 op/s 2023-03-09T18:56:42.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:56:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14313.450747 secs 2023-03-09T18:56:42.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:56:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14313.450747 secs 2023-03-09T18:56:42.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:56:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14313.450747 secs 2023-03-09T18:56:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14775.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:56:43.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:56:43 smithi122 ceph-mon[93869]: pgmap v8146: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 219 KiB/s wr, 10 op/s 2023-03-09T18:56:43.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:56:43 smithi181 ceph-mon[94189]: pgmap v8146: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 219 KiB/s wr, 10 op/s 2023-03-09T18:56:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:56:43 smithi038 ceph-mon[92422]: pgmap v8146: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 219 KiB/s wr, 10 op/s 2023-03-09T18:56:45.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:56:45 smithi122 ceph-mon[93869]: pgmap v8147: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 201 KiB/s wr, 8 op/s 2023-03-09T18:56:45.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:56:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14316.168203 secs 2023-03-09T18:56:45.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:56:45 smithi181 ceph-mon[94189]: pgmap v8147: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 201 KiB/s wr, 8 op/s 2023-03-09T18:56:45.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:56:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14316.168203 secs 2023-03-09T18:56:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:56:45 smithi038 ceph-mon[92422]: pgmap v8147: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 201 KiB/s wr, 8 op/s 2023-03-09T18:56:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:56:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14316.168203 secs 2023-03-09T18:56:47.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:56:47 smithi122 ceph-mon[93869]: pgmap v8148: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 214 KiB/s wr, 10 op/s 2023-03-09T18:56:47.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:56:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14318.450864 secs 2023-03-09T18:56:47.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:56:47 smithi181 ceph-mon[94189]: pgmap v8148: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 214 KiB/s wr, 10 op/s 2023-03-09T18:56:47.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:56:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14318.450864 secs 2023-03-09T18:56:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:56:47 smithi038 ceph-mon[92422]: pgmap v8148: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 214 KiB/s wr, 10 op/s 2023-03-09T18:56:48.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:56:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14318.450864 secs 2023-03-09T18:56:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14780.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:56:49.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:56:49 smithi122 ceph-mon[93869]: pgmap v8149: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 325 KiB/s wr, 10 op/s 2023-03-09T18:56:49.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:56:49 smithi181 ceph-mon[94189]: pgmap v8149: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 325 KiB/s wr, 10 op/s 2023-03-09T18:56:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:56:49 smithi038 ceph-mon[92422]: pgmap v8149: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 325 KiB/s wr, 10 op/s 2023-03-09T18:56:50.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:56:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14321.168263 secs 2023-03-09T18:56:50.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:56:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14321.168263 secs 2023-03-09T18:56:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:56:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14321.168263 secs 2023-03-09T18:56:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:56:51 smithi122 ceph-mon[93869]: pgmap v8150: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 276 KiB/s wr, 9 op/s 2023-03-09T18:56:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:56:51 smithi181 ceph-mon[94189]: pgmap v8150: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 276 KiB/s wr, 9 op/s 2023-03-09T18:56:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:56:51 smithi038 ceph-mon[92422]: pgmap v8150: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 276 KiB/s wr, 9 op/s 2023-03-09T18:56:52.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:56:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14323.450980 secs 2023-03-09T18:56:52.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:56:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14323.450980 secs 2023-03-09T18:56:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:56:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14323.450980 secs 2023-03-09T18:56:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14785.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:56:53.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:56:53 smithi122 ceph-mon[93869]: pgmap v8151: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 394 KiB/s wr, 14 op/s 2023-03-09T18:56:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:56:53 smithi181 ceph-mon[94189]: pgmap v8151: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 394 KiB/s wr, 14 op/s 2023-03-09T18:56:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:56:53 smithi038 ceph-mon[92422]: pgmap v8151: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 394 KiB/s wr, 14 op/s 2023-03-09T18:56:55.647 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:56:55 smithi122 ceph-mon[93869]: pgmap v8152: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 242 KiB/s wr, 11 op/s 2023-03-09T18:56:55.648 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:56:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14326.168322 secs 2023-03-09T18:56:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:56:55 smithi181 ceph-mon[94189]: pgmap v8152: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 242 KiB/s wr, 11 op/s 2023-03-09T18:56:55.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:56:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14326.168322 secs 2023-03-09T18:56:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:56:55 smithi038 ceph-mon[92422]: pgmap v8152: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 242 KiB/s wr, 11 op/s 2023-03-09T18:56:56.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:56:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14326.168322 secs 2023-03-09T18:56:57.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:56:57 smithi122 ceph-mon[93869]: pgmap v8153: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 313 KiB/s wr, 13 op/s 2023-03-09T18:56:57.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:56:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14328.451129 secs 2023-03-09T18:56:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:56:57 smithi181 ceph-mon[94189]: pgmap v8153: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 313 KiB/s wr, 13 op/s 2023-03-09T18:56:57.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:56:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14328.451129 secs 2023-03-09T18:56:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:56:57 smithi038 ceph-mon[92422]: pgmap v8153: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 313 KiB/s wr, 13 op/s 2023-03-09T18:56:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:56:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14328.451129 secs 2023-03-09T18:56:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14790.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:56:59.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:56:59 smithi122 ceph-mon[93869]: pgmap v8154: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 324 KiB/s wr, 12 op/s 2023-03-09T18:56:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:56:59 smithi181 ceph-mon[94189]: pgmap v8154: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 324 KiB/s wr, 12 op/s 2023-03-09T18:56:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:56:59 smithi038 ceph-mon[92422]: pgmap v8154: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 324 KiB/s wr, 12 op/s 2023-03-09T18:57:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:57:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14331.168382 secs 2023-03-09T18:57:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:57:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14331.168382 secs 2023-03-09T18:57:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:57:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14331.168382 secs 2023-03-09T18:57:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:57:01 smithi122 ceph-mon[93869]: pgmap v8155: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 213 KiB/s wr, 10 op/s 2023-03-09T18:57:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:57:01 smithi181 ceph-mon[94189]: pgmap v8155: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 213 KiB/s wr, 10 op/s 2023-03-09T18:57:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:57:01 smithi038 ceph-mon[92422]: pgmap v8155: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 213 KiB/s wr, 10 op/s 2023-03-09T18:57:02.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:57:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14333.451251 secs 2023-03-09T18:57:02.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:57:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:57:02.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:57:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14333.451251 secs 2023-03-09T18:57:02.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:57:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:57:02.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:57:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14333.451251 secs 2023-03-09T18:57:03.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:57:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:57:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14795.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:57:03.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:57:03 smithi122 ceph-mon[93869]: pgmap v8156: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 319 KiB/s wr, 16 op/s 2023-03-09T18:57:03.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:57:03 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:57:03.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:57:03 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:57:03.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:57:03 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:57:03.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:57:03 smithi181 ceph-mon[94189]: pgmap v8156: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 319 KiB/s wr, 16 op/s 2023-03-09T18:57:03.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:57:03 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:57:03.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:57:03 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:57:03.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:57:03 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:57:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:57:03 smithi038 ceph-mon[92422]: pgmap v8156: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 319 KiB/s wr, 16 op/s 2023-03-09T18:57:04.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:57:03 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:57:04.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:57:03 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:57:04.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:57:03 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:57:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:57:05 smithi122 ceph-mon[93869]: pgmap v8157: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 223 KiB/s wr, 24 op/s 2023-03-09T18:57:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:57:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14336.168465 secs 2023-03-09T18:57:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:57:05 smithi181 ceph-mon[94189]: pgmap v8157: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 223 KiB/s wr, 24 op/s 2023-03-09T18:57:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:57:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14336.168465 secs 2023-03-09T18:57:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:57:05 smithi038 ceph-mon[92422]: pgmap v8157: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 223 KiB/s wr, 24 op/s 2023-03-09T18:57:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:57:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14336.168465 secs 2023-03-09T18:57:07.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:57:07 smithi122 ceph-mon[93869]: pgmap v8158: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 266 KiB/s wr, 26 op/s 2023-03-09T18:57:07.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:57:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14338.451371 secs 2023-03-09T18:57:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:57:07 smithi181 ceph-mon[94189]: pgmap v8158: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 266 KiB/s wr, 26 op/s 2023-03-09T18:57:07.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:57:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14338.451371 secs 2023-03-09T18:57:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:57:07 smithi038 ceph-mon[92422]: pgmap v8158: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 266 KiB/s wr, 26 op/s 2023-03-09T18:57:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:57:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14338.451371 secs 2023-03-09T18:57:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14800.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:57:09.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:57:09 smithi122 ceph-mon[93869]: pgmap v8159: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 195 KiB/s wr, 23 op/s 2023-03-09T18:57:09.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:57:09 smithi181 ceph-mon[94189]: pgmap v8159: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 195 KiB/s wr, 23 op/s 2023-03-09T18:57:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:57:09 smithi038 ceph-mon[92422]: pgmap v8159: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 195 KiB/s wr, 23 op/s 2023-03-09T18:57:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:57:10 smithi122 ceph-mon[93869]: osdmap e723: 12 total, 12 up, 12 in 2023-03-09T18:57:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:57:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14341.168529 secs 2023-03-09T18:57:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:57:10 smithi181 ceph-mon[94189]: osdmap e723: 12 total, 12 up, 12 in 2023-03-09T18:57:10.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:57:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14341.168529 secs 2023-03-09T18:57:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:57:10 smithi038 ceph-mon[92422]: osdmap e723: 12 total, 12 up, 12 in 2023-03-09T18:57:11.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:57:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14341.168529 secs 2023-03-09T18:57:11.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:57:11 smithi122 ceph-mon[93869]: pgmap v8161: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 206 KiB/s wr, 26 op/s 2023-03-09T18:57:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:57:11 smithi181 ceph-mon[94189]: pgmap v8161: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 206 KiB/s wr, 26 op/s 2023-03-09T18:57:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:57:11 smithi038 ceph-mon[92422]: pgmap v8161: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 206 KiB/s wr, 26 op/s 2023-03-09T18:57:12.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:57:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14343.451488 secs 2023-03-09T18:57:12.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:57:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14343.451488 secs 2023-03-09T18:57:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:57:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14343.451488 secs 2023-03-09T18:57:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14805.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:57:13.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:57:13 smithi181 ceph-mon[94189]: pgmap v8162: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 78 KiB/s wr, 21 op/s 2023-03-09T18:57:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:57:13 smithi038 ceph-mon[92422]: pgmap v8162: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 78 KiB/s wr, 21 op/s 2023-03-09T18:57:14.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:57:13 smithi122 ceph-mon[93869]: pgmap v8162: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 78 KiB/s wr, 21 op/s 2023-03-09T18:57:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:57:15 smithi038 ceph-mon[92422]: pgmap v8163: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 52 KiB/s wr, 7 op/s 2023-03-09T18:57:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:57:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14346.168589 secs 2023-03-09T18:57:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:57:15 smithi122 ceph-mon[93869]: pgmap v8163: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 52 KiB/s wr, 7 op/s 2023-03-09T18:57:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:57:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14346.168589 secs 2023-03-09T18:57:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:57:15 smithi181 ceph-mon[94189]: pgmap v8163: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 52 KiB/s wr, 7 op/s 2023-03-09T18:57:16.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:57:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14346.168589 secs 2023-03-09T18:57:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:57:17 smithi038 ceph-mon[92422]: pgmap v8164: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 614 B/s wr, 6 op/s 2023-03-09T18:57:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:57:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14348.451610 secs 2023-03-09T18:57:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:57:17 smithi038 ceph-mon[92422]: osdmap e724: 12 total, 12 up, 12 in 2023-03-09T18:57:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:57:17 smithi122 ceph-mon[93869]: pgmap v8164: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 614 B/s wr, 6 op/s 2023-03-09T18:57:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:57:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14348.451610 secs 2023-03-09T18:57:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:57:17 smithi122 ceph-mon[93869]: osdmap e724: 12 total, 12 up, 12 in 2023-03-09T18:57:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:57:17 smithi181 ceph-mon[94189]: pgmap v8164: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 614 B/s wr, 6 op/s 2023-03-09T18:57:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:57:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14348.451610 secs 2023-03-09T18:57:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:57:17 smithi181 ceph-mon[94189]: osdmap e724: 12 total, 12 up, 12 in 2023-03-09T18:57:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14810.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:57:20.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:57:19 smithi122 ceph-mon[93869]: pgmap v8166: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 237 B/s wr, 7 op/s 2023-03-09T18:57:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:57:19 smithi181 ceph-mon[94189]: pgmap v8166: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 237 B/s wr, 7 op/s 2023-03-09T18:57:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:57:19 smithi038 ceph-mon[92422]: pgmap v8166: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 237 B/s wr, 7 op/s 2023-03-09T18:57:21.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:57:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14351.168656 secs 2023-03-09T18:57:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:57:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14351.168656 secs 2023-03-09T18:57:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:57:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14351.168656 secs 2023-03-09T18:57:22.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:57:21 smithi122 ceph-mon[93869]: pgmap v8167: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T18:57:22.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:57:21 smithi181 ceph-mon[94189]: pgmap v8167: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T18:57:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:57:21 smithi038 ceph-mon[92422]: pgmap v8167: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T18:57:23.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:57:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14353.451728 secs 2023-03-09T18:57:23.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:57:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14353.451728 secs 2023-03-09T18:57:23.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:57:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14353.451728 secs 2023-03-09T18:57:23.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14815.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:57:24.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:57:23 smithi122 ceph-mon[93869]: pgmap v8168: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T18:57:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:57:23 smithi181 ceph-mon[94189]: pgmap v8168: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T18:57:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:57:23 smithi038 ceph-mon[92422]: pgmap v8168: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T18:57:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:57:25 smithi122 ceph-mon[93869]: pgmap v8169: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T18:57:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:57:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14356.168716 secs 2023-03-09T18:57:26.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:57:25 smithi181 ceph-mon[94189]: pgmap v8169: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T18:57:26.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:57:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14356.168716 secs 2023-03-09T18:57:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:57:25 smithi038 ceph-mon[92422]: pgmap v8169: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T18:57:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:57:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14356.168716 secs 2023-03-09T18:57:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:57:27 smithi122 ceph-mon[93869]: pgmap v8170: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 45 KiB/s wr, 6 op/s 2023-03-09T18:57:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:57:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14358.451850 secs 2023-03-09T18:57:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:57:27 smithi181 ceph-mon[94189]: pgmap v8170: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 45 KiB/s wr, 6 op/s 2023-03-09T18:57:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:57:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14358.451850 secs 2023-03-09T18:57:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:57:27 smithi038 ceph-mon[92422]: pgmap v8170: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 45 KiB/s wr, 6 op/s 2023-03-09T18:57:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:57:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14358.451850 secs 2023-03-09T18:57:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14820.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:57:30.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:57:29 smithi122 ceph-mon[93869]: pgmap v8171: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 41 KiB/s wr, 6 op/s 2023-03-09T18:57:30.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:57:29 smithi181 ceph-mon[94189]: pgmap v8171: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 41 KiB/s wr, 6 op/s 2023-03-09T18:57:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:57:29 smithi038 ceph-mon[92422]: pgmap v8171: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 41 KiB/s wr, 6 op/s 2023-03-09T18:57:31.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:57:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14361.168795 secs 2023-03-09T18:57:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:57:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14361.168795 secs 2023-03-09T18:57:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:57:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14361.168795 secs 2023-03-09T18:57:32.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:57:31 smithi122 ceph-mon[93869]: pgmap v8172: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 58 KiB/s wr, 6 op/s 2023-03-09T18:57:32.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:57:31 smithi181 ceph-mon[94189]: pgmap v8172: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 58 KiB/s wr, 6 op/s 2023-03-09T18:57:32.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:57:31 smithi038 ceph-mon[92422]: pgmap v8172: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 58 KiB/s wr, 6 op/s 2023-03-09T18:57:33.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:57:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14363.451967 secs 2023-03-09T18:57:33.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:57:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14363.451967 secs 2023-03-09T18:57:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:57:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14363.451967 secs 2023-03-09T18:57:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14825.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:57:34.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:57:33 smithi122 ceph-mon[93869]: pgmap v8173: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 85 KiB/s wr, 9 op/s 2023-03-09T18:57:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:57:33 smithi181 ceph-mon[94189]: pgmap v8173: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 85 KiB/s wr, 9 op/s 2023-03-09T18:57:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:57:33 smithi038 ceph-mon[92422]: pgmap v8173: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 85 KiB/s wr, 9 op/s 2023-03-09T18:57:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:57:35 smithi122 ceph-mon[93869]: pgmap v8174: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 85 KiB/s wr, 6 op/s 2023-03-09T18:57:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:57:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14366.168856 secs 2023-03-09T18:57:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:57:35 smithi181 ceph-mon[94189]: pgmap v8174: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 85 KiB/s wr, 6 op/s 2023-03-09T18:57:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:57:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14366.168856 secs 2023-03-09T18:57:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:57:35 smithi038 ceph-mon[92422]: pgmap v8174: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 85 KiB/s wr, 6 op/s 2023-03-09T18:57:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:57:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14366.168856 secs 2023-03-09T18:57:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:57:37 smithi122 ceph-mon[93869]: pgmap v8175: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 134 KiB/s wr, 9 op/s 2023-03-09T18:57:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:57:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14368.452055 secs 2023-03-09T18:57:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:57:37 smithi181 ceph-mon[94189]: pgmap v8175: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 134 KiB/s wr, 9 op/s 2023-03-09T18:57:38.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:57:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14368.452055 secs 2023-03-09T18:57:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:57:37 smithi038 ceph-mon[92422]: pgmap v8175: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 134 KiB/s wr, 9 op/s 2023-03-09T18:57:38.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:57:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14368.452055 secs 2023-03-09T18:57:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14830.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:57:40.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:57:39 smithi122 ceph-mon[93869]: pgmap v8176: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 97 KiB/s wr, 8 op/s 2023-03-09T18:57:40.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:57:39 smithi181 ceph-mon[94189]: pgmap v8176: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 97 KiB/s wr, 8 op/s 2023-03-09T18:57:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:57:39 smithi038 ceph-mon[92422]: pgmap v8176: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 97 KiB/s wr, 8 op/s 2023-03-09T18:57:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:57:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14371.168918 secs 2023-03-09T18:57:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:57:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14371.168918 secs 2023-03-09T18:57:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:57:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14371.168918 secs 2023-03-09T18:57:42.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:57:41 smithi122 ceph-mon[93869]: pgmap v8177: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 242 KiB/s wr, 8 op/s 2023-03-09T18:57:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:57:41 smithi181 ceph-mon[94189]: pgmap v8177: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 242 KiB/s wr, 8 op/s 2023-03-09T18:57:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:57:41 smithi038 ceph-mon[92422]: pgmap v8177: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 242 KiB/s wr, 8 op/s 2023-03-09T18:57:43.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:57:43 smithi122 ceph-mon[93869]: pgmap v8178: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 247 KiB/s wr, 11 op/s 2023-03-09T18:57:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:57:43 smithi181 ceph-mon[94189]: pgmap v8178: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 247 KiB/s wr, 11 op/s 2023-03-09T18:57:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:57:43 smithi038 ceph-mon[92422]: pgmap v8178: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 247 KiB/s wr, 11 op/s 2023-03-09T18:57:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14835.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:57:44.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:57:44 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:57:44.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:57:44 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:57:44.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:57:44 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:57:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:57:45 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:57:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:57:45 smithi122 ceph-mon[93869]: pgmap v8179: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 221 KiB/s wr, 9 op/s 2023-03-09T18:57:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:57:45 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:57:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:57:45 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:57:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:57:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14376.168993 secs 2023-03-09T18:57:45.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:57:45 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:57:45.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:57:45 smithi181 ceph-mon[94189]: pgmap v8179: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 221 KiB/s wr, 9 op/s 2023-03-09T18:57:45.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:57:45 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:57:45.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:57:45 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:57:45.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:57:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14376.168993 secs 2023-03-09T18:57:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:57:45 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:57:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:57:45 smithi038 ceph-mon[92422]: pgmap v8179: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 221 KiB/s wr, 9 op/s 2023-03-09T18:57:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:57:45 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:57:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:57:45 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:57:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:57:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14376.168993 secs 2023-03-09T18:57:47.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:57:47 smithi122 ceph-mon[93869]: pgmap v8180: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 239 KiB/s wr, 12 op/s 2023-03-09T18:57:47.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:57:47 smithi181 ceph-mon[94189]: pgmap v8180: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 239 KiB/s wr, 12 op/s 2023-03-09T18:57:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:57:47 smithi038 ceph-mon[92422]: pgmap v8180: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 239 KiB/s wr, 12 op/s 2023-03-09T18:57:48.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14840.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:57:49.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:57:49 smithi122 ceph-mon[93869]: pgmap v8181: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 190 KiB/s wr, 10 op/s 2023-03-09T18:57:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:57:49 smithi181 ceph-mon[94189]: pgmap v8181: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 190 KiB/s wr, 10 op/s 2023-03-09T18:57:49.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:57:49 smithi038 ceph-mon[92422]: pgmap v8181: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 190 KiB/s wr, 10 op/s 2023-03-09T18:57:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:57:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14381.169056 secs 2023-03-09T18:57:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:57:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14381.169056 secs 2023-03-09T18:57:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:57:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14381.169056 secs 2023-03-09T18:57:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:57:51 smithi122 ceph-mon[93869]: pgmap v8182: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 313 KiB/s rd, 299 KiB/s wr, 10 op/s 2023-03-09T18:57:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:57:51 smithi181 ceph-mon[94189]: pgmap v8182: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 313 KiB/s rd, 299 KiB/s wr, 10 op/s 2023-03-09T18:57:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:57:51 smithi038 ceph-mon[92422]: pgmap v8182: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 313 KiB/s rd, 299 KiB/s wr, 10 op/s 2023-03-09T18:57:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:57:53 smithi122 ceph-mon[93869]: pgmap v8183: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 223 KiB/s wr, 12 op/s 2023-03-09T18:57:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:57:53 smithi181 ceph-mon[94189]: pgmap v8183: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 223 KiB/s wr, 12 op/s 2023-03-09T18:57:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:57:53 smithi038 ceph-mon[92422]: pgmap v8183: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 223 KiB/s wr, 12 op/s 2023-03-09T18:57:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14845.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:57:55.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:57:55 smithi122 ceph-mon[93869]: pgmap v8184: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 199 KiB/s wr, 9 op/s 2023-03-09T18:57:55.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:57:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14386.169128 secs 2023-03-09T18:57:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:57:55 smithi181 ceph-mon[94189]: pgmap v8184: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 199 KiB/s wr, 9 op/s 2023-03-09T18:57:55.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:57:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14386.169128 secs 2023-03-09T18:57:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:57:55 smithi038 ceph-mon[92422]: pgmap v8184: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 199 KiB/s wr, 9 op/s 2023-03-09T18:57:55.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:57:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14386.169128 secs 2023-03-09T18:57:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:57:57 smithi122 ceph-mon[93869]: pgmap v8185: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 199 KiB/s wr, 11 op/s 2023-03-09T18:57:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:57:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14388.452486 secs 2023-03-09T18:57:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:57:57 smithi181 ceph-mon[94189]: pgmap v8185: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 199 KiB/s wr, 11 op/s 2023-03-09T18:57:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:57:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14388.452486 secs 2023-03-09T18:57:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:57:57 smithi038 ceph-mon[92422]: pgmap v8185: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 199 KiB/s wr, 11 op/s 2023-03-09T18:57:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:57:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14388.452486 secs 2023-03-09T18:57:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14850.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:57:59.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:57:59 smithi122 ceph-mon[93869]: pgmap v8186: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 366 KiB/s wr, 11 op/s 2023-03-09T18:57:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:57:59 smithi181 ceph-mon[94189]: pgmap v8186: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 366 KiB/s wr, 11 op/s 2023-03-09T18:57:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:57:59 smithi038 ceph-mon[92422]: pgmap v8186: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 366 KiB/s wr, 11 op/s 2023-03-09T18:58:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:58:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14391.169188 secs 2023-03-09T18:58:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:58:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14391.169188 secs 2023-03-09T18:58:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:58:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14391.169188 secs 2023-03-09T18:58:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:58:01 smithi122 ceph-mon[93869]: pgmap v8187: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 366 KiB/s wr, 10 op/s 2023-03-09T18:58:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:58:01 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:58:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:58:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:58:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:58:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:58:01.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:58:01 smithi181 ceph-mon[94189]: pgmap v8187: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 366 KiB/s wr, 10 op/s 2023-03-09T18:58:01.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:58:01 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:58:01.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:58:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:58:01.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:58:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:58:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:58:01 smithi038 ceph-mon[92422]: pgmap v8187: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 366 KiB/s wr, 10 op/s 2023-03-09T18:58:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:58:01 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:58:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:58:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:58:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:58:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:58:02.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:58:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14393.452571 secs 2023-03-09T18:58:02.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:58:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14393.452571 secs 2023-03-09T18:58:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:58:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14393.452571 secs 2023-03-09T18:58:03.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:58:03 smithi122 ceph-mon[93869]: pgmap v8188: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 306 KiB/s wr, 16 op/s 2023-03-09T18:58:03.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:58:03 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:58:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:58:03 smithi181 ceph-mon[94189]: pgmap v8188: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 306 KiB/s wr, 16 op/s 2023-03-09T18:58:03.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:58:03 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:58:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:58:03 smithi038 ceph-mon[92422]: pgmap v8188: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 306 KiB/s wr, 16 op/s 2023-03-09T18:58:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:58:03 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:58:03.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14855.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:58:04.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:58:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:58:04.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:58:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:58:04.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:58:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:58:04.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:58:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:58:04.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:58:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:58:04.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:58:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:58:04.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:58:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:58:04.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:58:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:58:04.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:58:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:58:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:58:05 smithi038 ceph-mon[92422]: pgmap v8189: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 258 KiB/s wr, 24 op/s 2023-03-09T18:58:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:58:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14396.169248 secs 2023-03-09T18:58:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:58:05 smithi038 ceph-mon[92422]: osdmap e725: 12 total, 12 up, 12 in 2023-03-09T18:58:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:58:05 smithi122 ceph-mon[93869]: pgmap v8189: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 258 KiB/s wr, 24 op/s 2023-03-09T18:58:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:58:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14396.169248 secs 2023-03-09T18:58:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:58:05 smithi122 ceph-mon[93869]: osdmap e725: 12 total, 12 up, 12 in 2023-03-09T18:58:06.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:58:05 smithi181 ceph-mon[94189]: pgmap v8189: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 258 KiB/s wr, 24 op/s 2023-03-09T18:58:06.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:58:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14396.169248 secs 2023-03-09T18:58:06.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:58:05 smithi181 ceph-mon[94189]: osdmap e725: 12 total, 12 up, 12 in 2023-03-09T18:58:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:58:07 smithi038 ceph-mon[92422]: pgmap v8191: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 331 KiB/s wr, 27 op/s 2023-03-09T18:58:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:58:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14398.452685 secs 2023-03-09T18:58:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:58:07 smithi122 ceph-mon[93869]: pgmap v8191: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 331 KiB/s wr, 27 op/s 2023-03-09T18:58:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:58:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14398.452685 secs 2023-03-09T18:58:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:58:07 smithi181 ceph-mon[94189]: pgmap v8191: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 331 KiB/s wr, 27 op/s 2023-03-09T18:58:08.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:58:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14398.452685 secs 2023-03-09T18:58:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14860.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:58:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:58:09 smithi038 ceph-mon[92422]: pgmap v8192: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 225 KiB/s wr, 26 op/s 2023-03-09T18:58:10.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:58:09 smithi122 ceph-mon[93869]: pgmap v8192: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 225 KiB/s wr, 26 op/s 2023-03-09T18:58:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:58:09 smithi181 ceph-mon[94189]: pgmap v8192: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 225 KiB/s wr, 26 op/s 2023-03-09T18:58:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:58:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14401.169308 secs 2023-03-09T18:58:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:58:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14401.169308 secs 2023-03-09T18:58:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:58:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14401.169308 secs 2023-03-09T18:58:12.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:58:11 smithi122 ceph-mon[93869]: pgmap v8193: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 225 KiB/s wr, 26 op/s 2023-03-09T18:58:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:58:11 smithi181 ceph-mon[94189]: pgmap v8193: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 225 KiB/s wr, 26 op/s 2023-03-09T18:58:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:58:11 smithi038 ceph-mon[92422]: pgmap v8193: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 225 KiB/s wr, 26 op/s 2023-03-09T18:58:13.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:58:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14403.452839 secs 2023-03-09T18:58:13.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:58:12 smithi122 ceph-mon[93869]: osdmap e726: 12 total, 12 up, 12 in 2023-03-09T18:58:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:58:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14403.452839 secs 2023-03-09T18:58:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:58:12 smithi181 ceph-mon[94189]: osdmap e726: 12 total, 12 up, 12 in 2023-03-09T18:58:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:58:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14403.452839 secs 2023-03-09T18:58:13.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:58:12 smithi038 ceph-mon[92422]: osdmap e726: 12 total, 12 up, 12 in 2023-03-09T18:58:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14865.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:58:14.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:58:13 smithi122 ceph-mon[93869]: pgmap v8195: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 177 KiB/s wr, 11 op/s 2023-03-09T18:58:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:58:13 smithi181 ceph-mon[94189]: pgmap v8195: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 177 KiB/s wr, 11 op/s 2023-03-09T18:58:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:58:13 smithi038 ceph-mon[92422]: pgmap v8195: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 177 KiB/s wr, 11 op/s 2023-03-09T18:58:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:58:15 smithi122 ceph-mon[93869]: pgmap v8196: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 147 KiB/s wr, 9 op/s 2023-03-09T18:58:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:58:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14406.169369 secs 2023-03-09T18:58:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:58:15 smithi181 ceph-mon[94189]: pgmap v8196: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 147 KiB/s wr, 9 op/s 2023-03-09T18:58:16.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:58:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14406.169369 secs 2023-03-09T18:58:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:58:15 smithi038 ceph-mon[92422]: pgmap v8196: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 147 KiB/s wr, 9 op/s 2023-03-09T18:58:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:58:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14406.169369 secs 2023-03-09T18:58:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:58:17 smithi122 ceph-mon[93869]: pgmap v8197: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 171 KiB/s wr, 9 op/s 2023-03-09T18:58:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:58:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14408.452953 secs 2023-03-09T18:58:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:58:17 smithi181 ceph-mon[94189]: pgmap v8197: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 171 KiB/s wr, 9 op/s 2023-03-09T18:58:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:58:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14408.452953 secs 2023-03-09T18:58:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:58:17 smithi038 ceph-mon[92422]: pgmap v8197: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 171 KiB/s wr, 9 op/s 2023-03-09T18:58:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:58:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14408.452953 secs 2023-03-09T18:58:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14870.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:58:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:58:19 smithi122 ceph-mon[93869]: pgmap v8198: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 54 KiB/s wr, 8 op/s 2023-03-09T18:58:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:58:19 smithi181 ceph-mon[94189]: pgmap v8198: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 54 KiB/s wr, 8 op/s 2023-03-09T18:58:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:58:19 smithi038 ceph-mon[92422]: pgmap v8198: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 54 KiB/s wr, 8 op/s 2023-03-09T18:58:21.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:58:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14411.169429 secs 2023-03-09T18:58:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:58:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14411.169429 secs 2023-03-09T18:58:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:58:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14411.169429 secs 2023-03-09T18:58:22.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:58:21 smithi122 ceph-mon[93869]: pgmap v8199: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 54 KiB/s wr, 8 op/s 2023-03-09T18:58:22.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:58:21 smithi181 ceph-mon[94189]: pgmap v8199: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 54 KiB/s wr, 8 op/s 2023-03-09T18:58:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:58:21 smithi038 ceph-mon[92422]: pgmap v8199: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 54 KiB/s wr, 8 op/s 2023-03-09T18:58:23.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:58:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14413.453082 secs 2023-03-09T18:58:23.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:58:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14413.453082 secs 2023-03-09T18:58:23.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:58:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14413.453082 secs 2023-03-09T18:58:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14875.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:58:24.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:58:23 smithi122 ceph-mon[93869]: pgmap v8200: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 53 KiB/s wr, 7 op/s 2023-03-09T18:58:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:58:23 smithi181 ceph-mon[94189]: pgmap v8200: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 53 KiB/s wr, 7 op/s 2023-03-09T18:58:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:58:23 smithi038 ceph-mon[92422]: pgmap v8200: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 53 KiB/s wr, 7 op/s 2023-03-09T18:58:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:58:25 smithi122 ceph-mon[93869]: pgmap v8201: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 44 KiB/s wr, 6 op/s 2023-03-09T18:58:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:58:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14416.169489 secs 2023-03-09T18:58:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:58:25 smithi181 ceph-mon[94189]: pgmap v8201: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 44 KiB/s wr, 6 op/s 2023-03-09T18:58:26.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:58:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14416.169489 secs 2023-03-09T18:58:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:58:25 smithi038 ceph-mon[92422]: pgmap v8201: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 44 KiB/s wr, 6 op/s 2023-03-09T18:58:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:58:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14416.169489 secs 2023-03-09T18:58:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:58:27 smithi122 ceph-mon[93869]: pgmap v8202: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 44 KiB/s wr, 8 op/s 2023-03-09T18:58:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:58:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14418.453218 secs 2023-03-09T18:58:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:58:27 smithi181 ceph-mon[94189]: pgmap v8202: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 44 KiB/s wr, 8 op/s 2023-03-09T18:58:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:58:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14418.453218 secs 2023-03-09T18:58:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:58:27 smithi038 ceph-mon[92422]: pgmap v8202: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 44 KiB/s wr, 8 op/s 2023-03-09T18:58:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:58:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14418.453218 secs 2023-03-09T18:58:28.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14880.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:58:30.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:58:29 smithi122 ceph-mon[93869]: pgmap v8203: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 6 op/s 2023-03-09T18:58:30.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:58:29 smithi181 ceph-mon[94189]: pgmap v8203: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 6 op/s 2023-03-09T18:58:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:58:29 smithi038 ceph-mon[92422]: pgmap v8203: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 6 op/s 2023-03-09T18:58:31.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:58:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14421.169549 secs 2023-03-09T18:58:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:58:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14421.169549 secs 2023-03-09T18:58:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:58:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14421.169549 secs 2023-03-09T18:58:32.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:58:31 smithi122 ceph-mon[93869]: pgmap v8204: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2023-03-09T18:58:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:58:31 smithi181 ceph-mon[94189]: pgmap v8204: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2023-03-09T18:58:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:58:31 smithi038 ceph-mon[92422]: pgmap v8204: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2023-03-09T18:58:33.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:58:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14423.453340 secs 2023-03-09T18:58:33.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:58:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14423.453340 secs 2023-03-09T18:58:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:58:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14423.453340 secs 2023-03-09T18:58:33.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14885.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:58:34.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:58:33 smithi122 ceph-mon[93869]: pgmap v8205: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 170 B/s wr, 8 op/s 2023-03-09T18:58:34.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:58:33 smithi181 ceph-mon[94189]: pgmap v8205: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 170 B/s wr, 8 op/s 2023-03-09T18:58:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:58:33 smithi038 ceph-mon[92422]: pgmap v8205: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 170 B/s wr, 8 op/s 2023-03-09T18:58:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:58:35 smithi122 ceph-mon[93869]: pgmap v8206: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T18:58:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:58:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14426.169628 secs 2023-03-09T18:58:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:58:35 smithi181 ceph-mon[94189]: pgmap v8206: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T18:58:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:58:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14426.169628 secs 2023-03-09T18:58:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:58:35 smithi038 ceph-mon[92422]: pgmap v8206: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T18:58:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:58:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14426.169628 secs 2023-03-09T18:58:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:58:37 smithi122 ceph-mon[93869]: pgmap v8207: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 55 KiB/s wr, 7 op/s 2023-03-09T18:58:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:58:37 smithi181 ceph-mon[94189]: pgmap v8207: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 55 KiB/s wr, 7 op/s 2023-03-09T18:58:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:58:37 smithi038 ceph-mon[92422]: pgmap v8207: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 55 KiB/s wr, 7 op/s 2023-03-09T18:58:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14890.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:58:39.889 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:58:39 smithi038 ceph-mon[92422]: pgmap v8208: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 56 KiB/s wr, 6 op/s 2023-03-09T18:58:39.890 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:58:39 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:58:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:58:39 smithi122 ceph-mon[93869]: pgmap v8208: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 56 KiB/s wr, 6 op/s 2023-03-09T18:58:40.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:58:39 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:58:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:58:39 smithi181 ceph-mon[94189]: pgmap v8208: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 56 KiB/s wr, 6 op/s 2023-03-09T18:58:40.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:58:39 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T18:58:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:58:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14431.169689 secs 2023-03-09T18:58:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:58:40 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:58:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:58:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:58:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:58:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:58:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:58:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14431.169689 secs 2023-03-09T18:58:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:58:40 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:58:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:58:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:58:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:58:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:58:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:58:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14431.169689 secs 2023-03-09T18:58:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:58:40 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:58:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:58:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:58:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:58:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:58:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:58:41 smithi122 ceph-mon[93869]: pgmap v8209: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 56 KiB/s wr, 6 op/s 2023-03-09T18:58:42.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:58:41 smithi181 ceph-mon[94189]: pgmap v8209: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 56 KiB/s wr, 6 op/s 2023-03-09T18:58:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:58:41 smithi038 ceph-mon[92422]: pgmap v8209: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 56 KiB/s wr, 6 op/s 2023-03-09T18:58:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:58:43 smithi122 ceph-mon[93869]: pgmap v8210: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 227 KiB/s wr, 10 op/s 2023-03-09T18:58:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:58:43 smithi181 ceph-mon[94189]: pgmap v8210: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 227 KiB/s wr, 10 op/s 2023-03-09T18:58:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:58:43 smithi038 ceph-mon[92422]: pgmap v8210: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 227 KiB/s wr, 10 op/s 2023-03-09T18:58:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14895.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:58:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:58:45 smithi122 ceph-mon[93869]: pgmap v8211: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 227 KiB/s wr, 8 op/s 2023-03-09T18:58:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:58:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14436.169750 secs 2023-03-09T18:58:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:58:45 smithi181 ceph-mon[94189]: pgmap v8211: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 227 KiB/s wr, 8 op/s 2023-03-09T18:58:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:58:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14436.169750 secs 2023-03-09T18:58:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:58:45 smithi038 ceph-mon[92422]: pgmap v8211: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 227 KiB/s wr, 8 op/s 2023-03-09T18:58:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:58:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14436.169750 secs 2023-03-09T18:58:47.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:58:47 smithi122 ceph-mon[93869]: pgmap v8212: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 245 KiB/s wr, 11 op/s 2023-03-09T18:58:47.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:58:47 smithi181 ceph-mon[94189]: pgmap v8212: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 245 KiB/s wr, 11 op/s 2023-03-09T18:58:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:58:47 smithi038 ceph-mon[92422]: pgmap v8212: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 245 KiB/s wr, 11 op/s 2023-03-09T18:58:48.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14900.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:58:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:58:49 smithi122 ceph-mon[93869]: pgmap v8213: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 191 KiB/s wr, 9 op/s 2023-03-09T18:58:49.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:58:49 smithi181 ceph-mon[94189]: pgmap v8213: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 191 KiB/s wr, 9 op/s 2023-03-09T18:58:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:58:49 smithi038 ceph-mon[92422]: pgmap v8213: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 191 KiB/s wr, 9 op/s 2023-03-09T18:58:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:58:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14441.169835 secs 2023-03-09T18:58:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:58:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14441.169835 secs 2023-03-09T18:58:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:58:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14441.169835 secs 2023-03-09T18:58:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:58:51 smithi122 ceph-mon[93869]: pgmap v8214: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 190 KiB/s wr, 8 op/s 2023-03-09T18:58:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:58:51 smithi181 ceph-mon[94189]: pgmap v8214: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 190 KiB/s wr, 8 op/s 2023-03-09T18:58:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:58:51 smithi038 ceph-mon[92422]: pgmap v8214: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 190 KiB/s wr, 8 op/s 2023-03-09T18:58:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:58:53 smithi122 ceph-mon[93869]: pgmap v8215: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 371 KiB/s wr, 13 op/s 2023-03-09T18:58:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:58:53 smithi181 ceph-mon[94189]: pgmap v8215: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 371 KiB/s wr, 13 op/s 2023-03-09T18:58:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:58:53 smithi038 ceph-mon[92422]: pgmap v8215: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 371 KiB/s wr, 13 op/s 2023-03-09T18:58:53.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14905.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:58:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:58:55 smithi122 ceph-mon[93869]: pgmap v8216: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 200 KiB/s wr, 9 op/s 2023-03-09T18:58:55.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:58:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14446.169896 secs 2023-03-09T18:58:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:58:55 smithi181 ceph-mon[94189]: pgmap v8216: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 200 KiB/s wr, 9 op/s 2023-03-09T18:58:55.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:58:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14446.169896 secs 2023-03-09T18:58:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:58:55 smithi038 ceph-mon[92422]: pgmap v8216: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 200 KiB/s wr, 9 op/s 2023-03-09T18:58:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:58:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14446.169896 secs 2023-03-09T18:58:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:58:57 smithi122 ceph-mon[93869]: pgmap v8217: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 201 KiB/s wr, 10 op/s 2023-03-09T18:58:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:58:57 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 14448.453891 secs 2023-03-09T18:58:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:58:57 smithi181 ceph-mon[94189]: pgmap v8217: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 201 KiB/s wr, 10 op/s 2023-03-09T18:58:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:58:57 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 14448.453891 secs 2023-03-09T18:58:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:58:57 smithi038 ceph-mon[92422]: pgmap v8217: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 201 KiB/s wr, 10 op/s 2023-03-09T18:58:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:58:57 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 14448.453891 secs 2023-03-09T18:58:58.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14910.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:58:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:58:59 smithi122 ceph-mon[93869]: pgmap v8218: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 299 KiB/s wr, 10 op/s 2023-03-09T18:58:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:58:59 smithi181 ceph-mon[94189]: pgmap v8218: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 299 KiB/s wr, 10 op/s 2023-03-09T18:58:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:58:59 smithi038 ceph-mon[92422]: pgmap v8218: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 299 KiB/s wr, 10 op/s 2023-03-09T18:59:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:59:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14451.169984 secs 2023-03-09T18:59:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:59:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14451.169984 secs 2023-03-09T18:59:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:59:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14451.169984 secs 2023-03-09T18:59:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:59:01 smithi122 ceph-mon[93869]: pgmap v8219: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 298 KiB/s wr, 9 op/s 2023-03-09T18:59:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:59:01 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:59:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:59:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:59:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:59:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:59:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:59:01 smithi181 ceph-mon[94189]: pgmap v8219: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 298 KiB/s wr, 9 op/s 2023-03-09T18:59:01.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:59:01 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:59:01.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:59:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:59:01.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:59:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:59:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:59:01 smithi038 ceph-mon[92422]: pgmap v8219: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 298 KiB/s wr, 9 op/s 2023-03-09T18:59:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:59:01 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T18:59:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:59:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:59:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:59:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:59:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:59:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14453.454011 secs 2023-03-09T18:59:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:59:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14453.454011 secs 2023-03-09T18:59:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:59:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14453.454011 secs 2023-03-09T18:59:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:59:03 smithi122 ceph-mon[93869]: pgmap v8220: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 421 KiB/s wr, 14 op/s 2023-03-09T18:59:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:59:03 smithi181 ceph-mon[94189]: pgmap v8220: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 421 KiB/s wr, 14 op/s 2023-03-09T18:59:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:59:03 smithi038 ceph-mon[92422]: pgmap v8220: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 421 KiB/s wr, 14 op/s 2023-03-09T18:59:03.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14915.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:59:04.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:59:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:59:04.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:59:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:59:04.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:59:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T18:59:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:59:05 smithi122 ceph-mon[93869]: pgmap v8221: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 297 KiB/s wr, 22 op/s 2023-03-09T18:59:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:59:05 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:59:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:59:05 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:59:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:59:05 smithi122 ceph-mon[93869]: osdmap e727: 12 total, 12 up, 12 in 2023-03-09T18:59:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:59:05 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:59:05.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:59:05 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:59:05.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:59:05 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:59:05.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:59:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14456.170049 secs 2023-03-09T18:59:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:59:05 smithi181 ceph-mon[94189]: pgmap v8221: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 297 KiB/s wr, 22 op/s 2023-03-09T18:59:05.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:59:05 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:59:05.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:59:05 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:59:05.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:59:05 smithi181 ceph-mon[94189]: osdmap e727: 12 total, 12 up, 12 in 2023-03-09T18:59:05.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:59:05 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:59:05.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:59:05 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:59:05.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:59:05 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:59:05.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:59:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14456.170049 secs 2023-03-09T18:59:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:59:05 smithi038 ceph-mon[92422]: pgmap v8221: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 297 KiB/s wr, 22 op/s 2023-03-09T18:59:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:59:05 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:59:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:59:05 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:59:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:59:05 smithi038 ceph-mon[92422]: osdmap e727: 12 total, 12 up, 12 in 2023-03-09T18:59:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:59:05 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T18:59:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:59:05 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T18:59:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:59:05 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T18:59:05.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:59:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14456.170049 secs 2023-03-09T18:59:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:59:07 smithi122 ceph-mon[93869]: pgmap v8223: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 328 KiB/s rd, 369 KiB/s wr, 27 op/s 2023-03-09T18:59:07.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:59:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14458.454141 secs 2023-03-09T18:59:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:59:07 smithi181 ceph-mon[94189]: pgmap v8223: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 328 KiB/s rd, 369 KiB/s wr, 27 op/s 2023-03-09T18:59:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:59:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14458.454141 secs 2023-03-09T18:59:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:59:07 smithi038 ceph-mon[92422]: pgmap v8223: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 328 KiB/s rd, 369 KiB/s wr, 27 op/s 2023-03-09T18:59:07.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:59:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14458.454141 secs 2023-03-09T18:59:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14920.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:59:09.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:59:09 smithi122 ceph-mon[93869]: pgmap v8224: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 264 KiB/s wr, 28 op/s 2023-03-09T18:59:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:59:09 smithi181 ceph-mon[94189]: pgmap v8224: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 264 KiB/s wr, 28 op/s 2023-03-09T18:59:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:59:09 smithi038 ceph-mon[92422]: pgmap v8224: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 264 KiB/s wr, 28 op/s 2023-03-09T18:59:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:59:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14461.170109 secs 2023-03-09T18:59:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:59:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14461.170109 secs 2023-03-09T18:59:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:59:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14461.170109 secs 2023-03-09T18:59:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:59:11 smithi122 ceph-mon[93869]: pgmap v8225: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 264 KiB/s wr, 28 op/s 2023-03-09T18:59:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:59:11 smithi181 ceph-mon[94189]: pgmap v8225: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 264 KiB/s wr, 28 op/s 2023-03-09T18:59:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:59:11 smithi038 ceph-mon[92422]: pgmap v8225: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 264 KiB/s wr, 28 op/s 2023-03-09T18:59:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:59:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14463.454220 secs 2023-03-09T18:59:12.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:59:12 smithi122 ceph-mon[93869]: osdmap e728: 12 total, 12 up, 12 in 2023-03-09T18:59:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:59:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14463.454220 secs 2023-03-09T18:59:12.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:59:12 smithi181 ceph-mon[94189]: osdmap e728: 12 total, 12 up, 12 in 2023-03-09T18:59:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:59:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14463.454220 secs 2023-03-09T18:59:12.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:59:12 smithi038 ceph-mon[92422]: osdmap e728: 12 total, 12 up, 12 in 2023-03-09T18:59:13.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:59:13 smithi122 ceph-mon[93869]: pgmap v8227: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 62 KiB/s wr, 11 op/s 2023-03-09T18:59:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:59:13 smithi181 ceph-mon[94189]: pgmap v8227: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 62 KiB/s wr, 11 op/s 2023-03-09T18:59:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:59:13 smithi038 ceph-mon[92422]: pgmap v8227: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 62 KiB/s wr, 11 op/s 2023-03-09T18:59:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14925.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:59:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:59:15 smithi122 ceph-mon[93869]: pgmap v8228: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 103 KiB/s wr, 10 op/s 2023-03-09T18:59:15.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:59:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14466.170167 secs 2023-03-09T18:59:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:59:15 smithi181 ceph-mon[94189]: pgmap v8228: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 103 KiB/s wr, 10 op/s 2023-03-09T18:59:15.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:59:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14466.170167 secs 2023-03-09T18:59:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:59:15 smithi038 ceph-mon[92422]: pgmap v8228: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 103 KiB/s wr, 10 op/s 2023-03-09T18:59:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:59:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14466.170167 secs 2023-03-09T18:59:17.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:59:17 smithi122 ceph-mon[93869]: pgmap v8229: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 89 KiB/s wr, 8 op/s 2023-03-09T18:59:17.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:59:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14468.454335 secs 2023-03-09T18:59:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:59:17 smithi181 ceph-mon[94189]: pgmap v8229: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 89 KiB/s wr, 8 op/s 2023-03-09T18:59:17.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:59:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14468.454335 secs 2023-03-09T18:59:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:59:17 smithi038 ceph-mon[92422]: pgmap v8229: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 89 KiB/s wr, 8 op/s 2023-03-09T18:59:17.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:59:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14468.454335 secs 2023-03-09T18:59:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14930.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:59:19.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:59:19 smithi181 ceph-mon[94189]: pgmap v8230: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 107 KiB/s wr, 7 op/s 2023-03-09T18:59:19.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:59:19 smithi038 ceph-mon[92422]: pgmap v8230: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 107 KiB/s wr, 7 op/s 2023-03-09T18:59:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:59:19 smithi122 ceph-mon[93869]: pgmap v8230: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 107 KiB/s wr, 7 op/s 2023-03-09T18:59:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:59:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14471.170229 secs 2023-03-09T18:59:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:59:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14471.170229 secs 2023-03-09T18:59:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:59:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14471.170229 secs 2023-03-09T18:59:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:59:21 smithi038 ceph-mon[92422]: pgmap v8231: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 107 KiB/s wr, 7 op/s 2023-03-09T18:59:21.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:59:21 smithi122 ceph-mon[93869]: pgmap v8231: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 107 KiB/s wr, 7 op/s 2023-03-09T18:59:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:59:21 smithi181 ceph-mon[94189]: pgmap v8231: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 107 KiB/s wr, 7 op/s 2023-03-09T18:59:22.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:59:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14473.454471 secs 2023-03-09T18:59:22.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:59:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14473.454471 secs 2023-03-09T18:59:22.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:59:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14473.454471 secs 2023-03-09T18:59:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:59:23 smithi038 ceph-mon[92422]: pgmap v8232: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 106 KiB/s wr, 6 op/s 2023-03-09T18:59:23.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14935.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:59:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:59:23 smithi122 ceph-mon[93869]: pgmap v8232: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 106 KiB/s wr, 6 op/s 2023-03-09T18:59:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:59:23 smithi181 ceph-mon[94189]: pgmap v8232: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 106 KiB/s wr, 6 op/s 2023-03-09T18:59:25.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:59:25 smithi122 ceph-mon[93869]: pgmap v8233: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 109 KiB/s wr, 6 op/s 2023-03-09T18:59:25.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:59:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14476.170314 secs 2023-03-09T18:59:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:59:25 smithi181 ceph-mon[94189]: pgmap v8233: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 109 KiB/s wr, 6 op/s 2023-03-09T18:59:25.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:59:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14476.170314 secs 2023-03-09T18:59:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:59:25 smithi038 ceph-mon[92422]: pgmap v8233: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 109 KiB/s wr, 6 op/s 2023-03-09T18:59:26.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:59:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14476.170314 secs 2023-03-09T18:59:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:59:27 smithi122 ceph-mon[93869]: pgmap v8234: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 65 KiB/s wr, 7 op/s 2023-03-09T18:59:27.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:59:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14478.454612 secs 2023-03-09T18:59:27.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:59:27 smithi181 ceph-mon[94189]: pgmap v8234: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 65 KiB/s wr, 7 op/s 2023-03-09T18:59:27.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:59:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14478.454612 secs 2023-03-09T18:59:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:59:27 smithi038 ceph-mon[92422]: pgmap v8234: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 65 KiB/s wr, 7 op/s 2023-03-09T18:59:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:59:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14478.454612 secs 2023-03-09T18:59:28.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14940.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:59:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:59:29 smithi122 ceph-mon[93869]: pgmap v8235: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 65 KiB/s wr, 6 op/s 2023-03-09T18:59:29.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:59:29 smithi181 ceph-mon[94189]: pgmap v8235: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 65 KiB/s wr, 6 op/s 2023-03-09T18:59:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:59:29 smithi038 ceph-mon[92422]: pgmap v8235: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 65 KiB/s wr, 6 op/s 2023-03-09T18:59:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:59:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14481.170377 secs 2023-03-09T18:59:30.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:59:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14481.170377 secs 2023-03-09T18:59:30.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:59:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14481.170377 secs 2023-03-09T18:59:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:59:31 smithi122 ceph-mon[93869]: pgmap v8236: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 21 KiB/s wr, 5 op/s 2023-03-09T18:59:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:59:31 smithi181 ceph-mon[94189]: pgmap v8236: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 21 KiB/s wr, 5 op/s 2023-03-09T18:59:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:59:31 smithi038 ceph-mon[92422]: pgmap v8236: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 21 KiB/s wr, 5 op/s 2023-03-09T18:59:32.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:59:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14483.454689 secs 2023-03-09T18:59:32.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:59:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14483.454689 secs 2023-03-09T18:59:32.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:59:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14483.454689 secs 2023-03-09T18:59:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14945.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:59:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:59:33 smithi122 ceph-mon[93869]: pgmap v8237: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 8 op/s 2023-03-09T18:59:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:59:33 smithi181 ceph-mon[94189]: pgmap v8237: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 8 op/s 2023-03-09T18:59:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:59:33 smithi038 ceph-mon[92422]: pgmap v8237: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 8 op/s 2023-03-09T18:59:35.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:59:35 smithi122 ceph-mon[93869]: pgmap v8238: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 69 KiB/s wr, 6 op/s 2023-03-09T18:59:35.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:59:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14486.170439 secs 2023-03-09T18:59:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:59:35 smithi181 ceph-mon[94189]: pgmap v8238: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 69 KiB/s wr, 6 op/s 2023-03-09T18:59:35.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:59:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14486.170439 secs 2023-03-09T18:59:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:59:35 smithi038 ceph-mon[92422]: pgmap v8238: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 69 KiB/s wr, 6 op/s 2023-03-09T18:59:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:59:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14486.170439 secs 2023-03-09T18:59:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:59:37 smithi122 ceph-mon[93869]: pgmap v8239: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 48 KiB/s wr, 7 op/s 2023-03-09T18:59:37.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:59:37 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 14488.454806 secs 2023-03-09T18:59:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:59:37 smithi181 ceph-mon[94189]: pgmap v8239: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 48 KiB/s wr, 7 op/s 2023-03-09T18:59:37.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:59:37 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 14488.454806 secs 2023-03-09T18:59:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:59:37 smithi038 ceph-mon[92422]: pgmap v8239: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 48 KiB/s wr, 7 op/s 2023-03-09T18:59:38.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:59:37 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 14488.454806 secs 2023-03-09T18:59:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14950.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:59:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:59:39 smithi122 ceph-mon[93869]: pgmap v8240: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 60 KiB/s wr, 7 op/s 2023-03-09T18:59:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:59:39 smithi181 ceph-mon[94189]: pgmap v8240: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 60 KiB/s wr, 7 op/s 2023-03-09T18:59:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:59:39 smithi038 ceph-mon[92422]: pgmap v8240: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 60 KiB/s wr, 7 op/s 2023-03-09T18:59:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:59:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14491.170499 secs 2023-03-09T18:59:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:59:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14491.170499 secs 2023-03-09T18:59:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:59:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14491.170499 secs 2023-03-09T18:59:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:59:41 smithi122 ceph-mon[93869]: pgmap v8241: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 60 KiB/s wr, 6 op/s 2023-03-09T18:59:41.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:59:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:59:41.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:59:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:59:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:59:41 smithi181 ceph-mon[94189]: pgmap v8241: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 60 KiB/s wr, 6 op/s 2023-03-09T18:59:41.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:59:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:59:41.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:59:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:59:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:59:41 smithi038 ceph-mon[92422]: pgmap v8241: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 60 KiB/s wr, 6 op/s 2023-03-09T18:59:42.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:59:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:59:42.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:59:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:59:42.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:59:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14493.454923 secs 2023-03-09T18:59:42.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:59:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14493.454923 secs 2023-03-09T18:59:42.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:59:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14493.454923 secs 2023-03-09T18:59:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14955.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:59:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:59:43 smithi122 ceph-mon[93869]: pgmap v8242: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 73 KiB/s wr, 9 op/s 2023-03-09T18:59:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:59:43 smithi181 ceph-mon[94189]: pgmap v8242: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 73 KiB/s wr, 9 op/s 2023-03-09T18:59:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:59:43 smithi038 ceph-mon[92422]: pgmap v8242: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 73 KiB/s wr, 9 op/s 2023-03-09T18:59:45.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:59:45 smithi122 ceph-mon[93869]: pgmap v8243: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 216 KiB/s wr, 8 op/s 2023-03-09T18:59:45.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:59:45 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:59:45.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:59:45 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:59:45.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:59:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14496.170585 secs 2023-03-09T18:59:45.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:59:45 smithi181 ceph-mon[94189]: pgmap v8243: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 216 KiB/s wr, 8 op/s 2023-03-09T18:59:45.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:59:45 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:59:45.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:59:45 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:59:45.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:59:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14496.170585 secs 2023-03-09T18:59:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:59:45 smithi038 ceph-mon[92422]: pgmap v8243: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 216 KiB/s wr, 8 op/s 2023-03-09T18:59:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:59:45 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T18:59:46.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:59:45 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T18:59:46.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:59:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14496.170585 secs 2023-03-09T18:59:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:59:47 smithi122 ceph-mon[93869]: pgmap v8244: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 169 KiB/s wr, 10 op/s 2023-03-09T18:59:47.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:59:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14498.455040 secs 2023-03-09T18:59:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:59:47 smithi181 ceph-mon[94189]: pgmap v8244: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 169 KiB/s wr, 10 op/s 2023-03-09T18:59:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:59:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14498.455040 secs 2023-03-09T18:59:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:59:47 smithi038 ceph-mon[92422]: pgmap v8244: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 169 KiB/s wr, 10 op/s 2023-03-09T18:59:48.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:59:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14498.455040 secs 2023-03-09T18:59:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14960.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:59:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:59:49 smithi122 ceph-mon[93869]: pgmap v8245: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 365 KiB/s wr, 12 op/s 2023-03-09T18:59:49.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:59:49 smithi181 ceph-mon[94189]: pgmap v8245: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 365 KiB/s wr, 12 op/s 2023-03-09T18:59:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:59:49 smithi038 ceph-mon[92422]: pgmap v8245: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 365 KiB/s wr, 12 op/s 2023-03-09T18:59:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:59:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14501.170655 secs 2023-03-09T18:59:50.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:59:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14501.170655 secs 2023-03-09T18:59:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:59:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14501.170655 secs 2023-03-09T18:59:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:59:51 smithi122 ceph-mon[93869]: pgmap v8246: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 353 KiB/s wr, 10 op/s 2023-03-09T18:59:51.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:59:51 smithi181 ceph-mon[94189]: pgmap v8246: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 353 KiB/s wr, 10 op/s 2023-03-09T18:59:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:59:51 smithi038 ceph-mon[92422]: pgmap v8246: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 353 KiB/s wr, 10 op/s 2023-03-09T18:59:52.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:59:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14503.455159 secs 2023-03-09T18:59:52.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:59:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14503.455159 secs 2023-03-09T18:59:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:59:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14503.455159 secs 2023-03-09T18:59:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14965.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:59:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:59:53 smithi122 ceph-mon[93869]: pgmap v8247: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 365 KiB/s wr, 14 op/s 2023-03-09T18:59:53.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:59:53 smithi181 ceph-mon[94189]: pgmap v8247: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 365 KiB/s wr, 14 op/s 2023-03-09T18:59:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:59:53 smithi038 ceph-mon[92422]: pgmap v8247: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 365 KiB/s wr, 14 op/s 2023-03-09T18:59:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:59:55 smithi122 ceph-mon[93869]: pgmap v8248: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 391 KiB/s wr, 12 op/s 2023-03-09T18:59:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:59:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14506.170765 secs 2023-03-09T18:59:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:59:55 smithi181 ceph-mon[94189]: pgmap v8248: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 391 KiB/s wr, 12 op/s 2023-03-09T18:59:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:59:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14506.170765 secs 2023-03-09T18:59:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:59:55 smithi038 ceph-mon[92422]: pgmap v8248: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 391 KiB/s wr, 12 op/s 2023-03-09T18:59:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:59:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14506.170765 secs 2023-03-09T18:59:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:59:57 smithi122 ceph-mon[93869]: pgmap v8249: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 248 KiB/s wr, 12 op/s 2023-03-09T18:59:57.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:59:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14508.455284 secs 2023-03-09T18:59:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:59:57 smithi181 ceph-mon[94189]: pgmap v8249: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 248 KiB/s wr, 12 op/s 2023-03-09T18:59:57.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:59:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14508.455284 secs 2023-03-09T18:59:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:59:57 smithi038 ceph-mon[92422]: pgmap v8249: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 248 KiB/s wr, 12 op/s 2023-03-09T18:59:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:59:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14508.455284 secs 2023-03-09T18:59:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14970.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T18:59:59.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 18:59:59 smithi122 ceph-mon[93869]: pgmap v8250: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 388 KiB/s wr, 12 op/s 2023-03-09T18:59:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 18:59:59 smithi181 ceph-mon[94189]: pgmap v8250: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 388 KiB/s wr, 12 op/s 2023-03-09T18:59:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 18:59:59 smithi038 ceph-mon[92422]: pgmap v8250: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 388 KiB/s wr, 12 op/s 2023-03-09T19:00:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:00:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14511.170846 secs 2023-03-09T19:00:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:00:00 smithi122 ceph-mon[93869]: Health detail: HEALTH_WARN 2 MDSs report slow requests 2023-03-09T19:00:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:00:00 smithi122 ceph-mon[93869]: [WRN] MDS_SLOW_REQUEST: 2 MDSs report slow requests 2023-03-09T19:00:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:00:00 smithi122 ceph-mon[93869]: mds.i(mds.2): 2 slow requests are blocked > 180 secs 2023-03-09T19:00:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:00:00 smithi122 ceph-mon[93869]: mds.h(mds.1): 2 slow requests are blocked > 180 secs 2023-03-09T19:00:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:00:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14511.170846 secs 2023-03-09T19:00:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:00:00 smithi181 ceph-mon[94189]: Health detail: HEALTH_WARN 2 MDSs report slow requests 2023-03-09T19:00:00.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:00:00 smithi181 ceph-mon[94189]: [WRN] MDS_SLOW_REQUEST: 2 MDSs report slow requests 2023-03-09T19:00:00.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:00:00 smithi181 ceph-mon[94189]: mds.i(mds.2): 2 slow requests are blocked > 180 secs 2023-03-09T19:00:00.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:00:00 smithi181 ceph-mon[94189]: mds.h(mds.1): 2 slow requests are blocked > 180 secs 2023-03-09T19:00:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:00:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14511.170846 secs 2023-03-09T19:00:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:00:00 smithi038 ceph-mon[92422]: Health detail: HEALTH_WARN 2 MDSs report slow requests 2023-03-09T19:00:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:00:00 smithi038 ceph-mon[92422]: [WRN] MDS_SLOW_REQUEST: 2 MDSs report slow requests 2023-03-09T19:00:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:00:00 smithi038 ceph-mon[92422]: mds.i(mds.2): 2 slow requests are blocked > 180 secs 2023-03-09T19:00:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:00:00 smithi038 ceph-mon[92422]: mds.h(mds.1): 2 slow requests are blocked > 180 secs 2023-03-09T19:00:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:00:01 smithi122 ceph-mon[93869]: pgmap v8251: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 193 KiB/s wr, 8 op/s 2023-03-09T19:00:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:00:01 smithi181 ceph-mon[94189]: pgmap v8251: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 193 KiB/s wr, 8 op/s 2023-03-09T19:00:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:00:01 smithi038 ceph-mon[92422]: pgmap v8251: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 193 KiB/s wr, 8 op/s 2023-03-09T19:00:02.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:00:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14513.455414 secs 2023-03-09T19:00:02.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:00:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14513.455414 secs 2023-03-09T19:00:02.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:00:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14513.455414 secs 2023-03-09T19:00:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14975.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:00:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:00:03 smithi122 ceph-mon[93869]: pgmap v8252: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 231 KiB/s wr, 12 op/s 2023-03-09T19:00:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:00:03 smithi181 ceph-mon[94189]: pgmap v8252: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 231 KiB/s wr, 12 op/s 2023-03-09T19:00:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:00:03 smithi038 ceph-mon[92422]: pgmap v8252: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 231 KiB/s wr, 12 op/s 2023-03-09T19:00:05.634 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:00:05 smithi122 ceph-mon[93869]: pgmap v8253: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 240 KiB/s wr, 21 op/s 2023-03-09T19:00:05.634 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:00:05 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:00:05.634 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:00:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14516.170908 secs 2023-03-09T19:00:05.634 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:00:05 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:00:05.635 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:00:05 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:00:05.635 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:00:05 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:00:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:00:05 smithi181 ceph-mon[94189]: pgmap v8253: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 240 KiB/s wr, 21 op/s 2023-03-09T19:00:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:00:05 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:00:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:00:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14516.170908 secs 2023-03-09T19:00:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:00:05 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:00:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:00:05 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:00:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:00:05 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:00:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:00:05 smithi038 ceph-mon[92422]: pgmap v8253: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 240 KiB/s wr, 21 op/s 2023-03-09T19:00:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:00:05 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:00:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:00:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14516.170908 secs 2023-03-09T19:00:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:00:05 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:00:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:00:05 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:00:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:00:05 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:00:07.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:00:07 smithi122 ceph-mon[93869]: pgmap v8254: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 269 KiB/s wr, 25 op/s 2023-03-09T19:00:07.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:00:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14518.455522 secs 2023-03-09T19:00:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:00:07 smithi181 ceph-mon[94189]: pgmap v8254: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 269 KiB/s wr, 25 op/s 2023-03-09T19:00:07.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:00:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14518.455522 secs 2023-03-09T19:00:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:00:07 smithi038 ceph-mon[92422]: pgmap v8254: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 269 KiB/s wr, 25 op/s 2023-03-09T19:00:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:00:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14518.455522 secs 2023-03-09T19:00:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14980.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:00:09.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:00:09 smithi122 ceph-mon[93869]: pgmap v8255: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 269 KiB/s wr, 24 op/s 2023-03-09T19:00:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:00:09 smithi181 ceph-mon[94189]: pgmap v8255: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 269 KiB/s wr, 24 op/s 2023-03-09T19:00:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:00:09 smithi038 ceph-mon[92422]: pgmap v8255: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 269 KiB/s wr, 24 op/s 2023-03-09T19:00:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:00:10 smithi122 ceph-mon[93869]: osdmap e729: 12 total, 12 up, 12 in 2023-03-09T19:00:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:00:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14521.170967 secs 2023-03-09T19:00:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:00:10 smithi181 ceph-mon[94189]: osdmap e729: 12 total, 12 up, 12 in 2023-03-09T19:00:10.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:00:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14521.170967 secs 2023-03-09T19:00:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:00:10 smithi038 ceph-mon[92422]: osdmap e729: 12 total, 12 up, 12 in 2023-03-09T19:00:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:00:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14521.170967 secs 2023-03-09T19:00:11.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:00:11 smithi122 ceph-mon[93869]: pgmap v8257: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 154 KiB/s wr, 26 op/s 2023-03-09T19:00:11.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:00:11 smithi181 ceph-mon[94189]: pgmap v8257: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 154 KiB/s wr, 26 op/s 2023-03-09T19:00:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:00:11 smithi038 ceph-mon[92422]: pgmap v8257: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 154 KiB/s wr, 26 op/s 2023-03-09T19:00:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:00:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14523.455622 secs 2023-03-09T19:00:12.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:00:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14523.455622 secs 2023-03-09T19:00:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:00:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14523.455622 secs 2023-03-09T19:00:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14985.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:00:13.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:00:13 smithi122 ceph-mon[93869]: pgmap v8258: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 107 KiB/s wr, 25 op/s 2023-03-09T19:00:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:00:13 smithi181 ceph-mon[94189]: pgmap v8258: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 107 KiB/s wr, 25 op/s 2023-03-09T19:00:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:00:13 smithi038 ceph-mon[92422]: pgmap v8258: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 107 KiB/s wr, 25 op/s 2023-03-09T19:00:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:00:15 smithi122 ceph-mon[93869]: pgmap v8259: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 82 KiB/s wr, 10 op/s 2023-03-09T19:00:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:00:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14526.170987 secs 2023-03-09T19:00:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:00:15 smithi181 ceph-mon[94189]: pgmap v8259: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 82 KiB/s wr, 10 op/s 2023-03-09T19:00:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:00:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14526.170987 secs 2023-03-09T19:00:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:00:15 smithi038 ceph-mon[92422]: pgmap v8259: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 82 KiB/s wr, 10 op/s 2023-03-09T19:00:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:00:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14526.170987 secs 2023-03-09T19:00:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:00:17 smithi122 ceph-mon[93869]: pgmap v8260: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 716 B/s wr, 7 op/s 2023-03-09T19:00:17.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:00:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14528.455755 secs 2023-03-09T19:00:17.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:00:17 smithi122 ceph-mon[93869]: osdmap e730: 12 total, 12 up, 12 in 2023-03-09T19:00:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:00:17 smithi181 ceph-mon[94189]: pgmap v8260: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 716 B/s wr, 7 op/s 2023-03-09T19:00:17.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:00:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14528.455755 secs 2023-03-09T19:00:17.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:00:17 smithi181 ceph-mon[94189]: osdmap e730: 12 total, 12 up, 12 in 2023-03-09T19:00:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:00:17 smithi038 ceph-mon[92422]: pgmap v8260: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 716 B/s wr, 7 op/s 2023-03-09T19:00:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:00:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14528.455755 secs 2023-03-09T19:00:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:00:17 smithi038 ceph-mon[92422]: osdmap e730: 12 total, 12 up, 12 in 2023-03-09T19:00:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14990.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:00:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:00:19 smithi122 ceph-mon[93869]: pgmap v8262: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 352 B/s wr, 7 op/s 2023-03-09T19:00:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:00:19 smithi181 ceph-mon[94189]: pgmap v8262: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 352 B/s wr, 7 op/s 2023-03-09T19:00:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:00:19 smithi038 ceph-mon[92422]: pgmap v8262: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 352 B/s wr, 7 op/s 2023-03-09T19:00:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:00:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14531.171046 secs 2023-03-09T19:00:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:00:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14531.171046 secs 2023-03-09T19:00:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:00:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14531.171046 secs 2023-03-09T19:00:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:00:21 smithi122 ceph-mon[93869]: pgmap v8263: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 307 B/s wr, 6 op/s 2023-03-09T19:00:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:00:21 smithi181 ceph-mon[94189]: pgmap v8263: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 307 B/s wr, 6 op/s 2023-03-09T19:00:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:00:21 smithi038 ceph-mon[92422]: pgmap v8263: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 307 B/s wr, 6 op/s 2023-03-09T19:00:22.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:00:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14533.455907 secs 2023-03-09T19:00:22.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:00:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14533.455907 secs 2023-03-09T19:00:22.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:00:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14533.455907 secs 2023-03-09T19:00:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 14995.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:00:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:00:23 smithi122 ceph-mon[93869]: pgmap v8264: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T19:00:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:00:23 smithi181 ceph-mon[94189]: pgmap v8264: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T19:00:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:00:23 smithi038 ceph-mon[92422]: pgmap v8264: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T19:00:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:00:25 smithi122 ceph-mon[93869]: pgmap v8265: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 5 op/s 2023-03-09T19:00:25.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:00:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14536.171110 secs 2023-03-09T19:00:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:00:25 smithi181 ceph-mon[94189]: pgmap v8265: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 5 op/s 2023-03-09T19:00:25.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:00:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14536.171110 secs 2023-03-09T19:00:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:00:25 smithi038 ceph-mon[92422]: pgmap v8265: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 5 op/s 2023-03-09T19:00:26.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:00:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14536.171110 secs 2023-03-09T19:00:27.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:00:27 smithi181 ceph-mon[94189]: pgmap v8266: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 46 KiB/s wr, 6 op/s 2023-03-09T19:00:27.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:00:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14538.456041 secs 2023-03-09T19:00:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:00:27 smithi038 ceph-mon[92422]: pgmap v8266: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 46 KiB/s wr, 6 op/s 2023-03-09T19:00:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:00:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14538.456041 secs 2023-03-09T19:00:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:00:27 smithi122 ceph-mon[93869]: pgmap v8266: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 46 KiB/s wr, 6 op/s 2023-03-09T19:00:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:00:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14538.456041 secs 2023-03-09T19:00:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15000.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:00:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:00:29 smithi038 ceph-mon[92422]: pgmap v8267: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 6 op/s 2023-03-09T19:00:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:00:29 smithi122 ceph-mon[93869]: pgmap v8267: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 6 op/s 2023-03-09T19:00:30.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:00:29 smithi181 ceph-mon[94189]: pgmap v8267: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 6 op/s 2023-03-09T19:00:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:00:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14541.171170 secs 2023-03-09T19:00:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:00:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14541.171170 secs 2023-03-09T19:00:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:00:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14541.171170 secs 2023-03-09T19:00:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:00:31 smithi122 ceph-mon[93869]: pgmap v8268: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 38 KiB/s wr, 5 op/s 2023-03-09T19:00:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:00:31 smithi181 ceph-mon[94189]: pgmap v8268: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 38 KiB/s wr, 5 op/s 2023-03-09T19:00:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:00:31 smithi038 ceph-mon[92422]: pgmap v8268: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 38 KiB/s wr, 5 op/s 2023-03-09T19:00:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:00:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14543.456192 secs 2023-03-09T19:00:33.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:00:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14543.456192 secs 2023-03-09T19:00:33.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:00:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14543.456192 secs 2023-03-09T19:00:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15005.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:00:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:00:33 smithi122 ceph-mon[93869]: pgmap v8269: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 83 KiB/s wr, 8 op/s 2023-03-09T19:00:34.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:00:33 smithi181 ceph-mon[94189]: pgmap v8269: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 83 KiB/s wr, 8 op/s 2023-03-09T19:00:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:00:33 smithi038 ceph-mon[92422]: pgmap v8269: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 83 KiB/s wr, 8 op/s 2023-03-09T19:00:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:00:35 smithi122 ceph-mon[93869]: pgmap v8270: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 83 KiB/s wr, 6 op/s 2023-03-09T19:00:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:00:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14546.171229 secs 2023-03-09T19:00:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:00:35 smithi181 ceph-mon[94189]: pgmap v8270: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 83 KiB/s wr, 6 op/s 2023-03-09T19:00:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:00:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14546.171229 secs 2023-03-09T19:00:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:00:35 smithi038 ceph-mon[92422]: pgmap v8270: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 83 KiB/s wr, 6 op/s 2023-03-09T19:00:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:00:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14546.171229 secs 2023-03-09T19:00:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:00:37 smithi122 ceph-mon[93869]: pgmap v8271: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 143 KiB/s wr, 9 op/s 2023-03-09T19:00:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:00:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14548.456267 secs 2023-03-09T19:00:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:00:37 smithi181 ceph-mon[94189]: pgmap v8271: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 143 KiB/s wr, 9 op/s 2023-03-09T19:00:38.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:00:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14548.456267 secs 2023-03-09T19:00:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:00:37 smithi038 ceph-mon[92422]: pgmap v8271: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 143 KiB/s wr, 9 op/s 2023-03-09T19:00:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:00:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14548.456267 secs 2023-03-09T19:00:38.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15010.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:00:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:00:39 smithi122 ceph-mon[93869]: pgmap v8272: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 105 KiB/s wr, 7 op/s 2023-03-09T19:00:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:00:39 smithi181 ceph-mon[94189]: pgmap v8272: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 105 KiB/s wr, 7 op/s 2023-03-09T19:00:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:00:39 smithi038 ceph-mon[92422]: pgmap v8272: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 105 KiB/s wr, 7 op/s 2023-03-09T19:00:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:00:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14551.171292 secs 2023-03-09T19:00:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:00:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14551.171292 secs 2023-03-09T19:00:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:00:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14551.171292 secs 2023-03-09T19:00:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:00:41 smithi122 ceph-mon[93869]: pgmap v8273: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 134 KiB/s wr, 6 op/s 2023-03-09T19:00:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:00:41 smithi181 ceph-mon[94189]: pgmap v8273: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 134 KiB/s wr, 6 op/s 2023-03-09T19:00:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:00:41 smithi038 ceph-mon[92422]: pgmap v8273: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 134 KiB/s wr, 6 op/s 2023-03-09T19:00:43.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:00:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14553.456346 secs 2023-03-09T19:00:43.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:00:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14553.456346 secs 2023-03-09T19:00:43.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:00:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14553.456346 secs 2023-03-09T19:00:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15015.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:00:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:00:43 smithi122 ceph-mon[93869]: pgmap v8274: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 262 KiB/s wr, 10 op/s 2023-03-09T19:00:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:00:43 smithi181 ceph-mon[94189]: pgmap v8274: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 262 KiB/s wr, 10 op/s 2023-03-09T19:00:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:00:43 smithi038 ceph-mon[92422]: pgmap v8274: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 262 KiB/s wr, 10 op/s 2023-03-09T19:00:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:00:45 smithi122 ceph-mon[93869]: pgmap v8275: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 330 KiB/s wr, 8 op/s 2023-03-09T19:00:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:00:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14556.171376 secs 2023-03-09T19:00:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:00:45 smithi181 ceph-mon[94189]: pgmap v8275: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 330 KiB/s wr, 8 op/s 2023-03-09T19:00:46.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:00:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14556.171376 secs 2023-03-09T19:00:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:00:45 smithi038 ceph-mon[92422]: pgmap v8275: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 330 KiB/s wr, 8 op/s 2023-03-09T19:00:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:00:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14556.171376 secs 2023-03-09T19:00:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:00:47 smithi122 ceph-mon[93869]: pgmap v8276: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 330 KiB/s wr, 11 op/s 2023-03-09T19:00:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:00:47 smithi181 ceph-mon[94189]: pgmap v8276: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 330 KiB/s wr, 11 op/s 2023-03-09T19:00:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:00:47 smithi038 ceph-mon[92422]: pgmap v8276: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 330 KiB/s wr, 11 op/s 2023-03-09T19:00:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15020.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:00:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:00:48 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:00:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:00:48 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:00:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:00:48 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:00:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:00:49 smithi122 ceph-mon[93869]: pgmap v8277: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 285 KiB/s wr, 10 op/s 2023-03-09T19:00:50.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:00:49 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:00:50.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:00:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:00:50.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:00:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:00:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:00:49 smithi181 ceph-mon[94189]: pgmap v8277: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 285 KiB/s wr, 10 op/s 2023-03-09T19:00:50.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:00:49 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:00:50.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:00:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:00:50.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:00:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:00:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:00:49 smithi038 ceph-mon[92422]: pgmap v8277: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 285 KiB/s wr, 10 op/s 2023-03-09T19:00:50.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:00:49 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:00:50.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:00:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:00:50.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:00:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:00:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:00:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14561.171436 secs 2023-03-09T19:00:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:00:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14561.171436 secs 2023-03-09T19:00:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:00:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14561.171436 secs 2023-03-09T19:00:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:00:51 smithi122 ceph-mon[93869]: pgmap v8278: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 393 KiB/s wr, 10 op/s 2023-03-09T19:00:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:00:51 smithi181 ceph-mon[94189]: pgmap v8278: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 393 KiB/s wr, 10 op/s 2023-03-09T19:00:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:00:51 smithi038 ceph-mon[92422]: pgmap v8278: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 393 KiB/s wr, 10 op/s 2023-03-09T19:00:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:00:53 smithi122 ceph-mon[93869]: pgmap v8279: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 378 KiB/s wr, 14 op/s 2023-03-09T19:00:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:00:53 smithi181 ceph-mon[94189]: pgmap v8279: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 378 KiB/s wr, 14 op/s 2023-03-09T19:00:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:00:53 smithi038 ceph-mon[92422]: pgmap v8279: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 378 KiB/s wr, 14 op/s 2023-03-09T19:00:53.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15025.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:00:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:00:55 smithi122 ceph-mon[93869]: pgmap v8280: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 335 KiB/s wr, 12 op/s 2023-03-09T19:00:55.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:00:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14566.171497 secs 2023-03-09T19:00:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:00:55 smithi181 ceph-mon[94189]: pgmap v8280: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 335 KiB/s wr, 12 op/s 2023-03-09T19:00:55.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:00:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14566.171497 secs 2023-03-09T19:00:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:00:55 smithi038 ceph-mon[92422]: pgmap v8280: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 335 KiB/s wr, 12 op/s 2023-03-09T19:00:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:00:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14566.171497 secs 2023-03-09T19:00:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:00:57 smithi122 ceph-mon[93869]: pgmap v8281: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 223 KiB/s wr, 12 op/s 2023-03-09T19:00:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:00:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14568.456719 secs 2023-03-09T19:00:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:00:57 smithi181 ceph-mon[94189]: pgmap v8281: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 223 KiB/s wr, 12 op/s 2023-03-09T19:00:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:00:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14568.456719 secs 2023-03-09T19:00:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:00:57 smithi038 ceph-mon[92422]: pgmap v8281: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 223 KiB/s wr, 12 op/s 2023-03-09T19:00:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:00:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14568.456719 secs 2023-03-09T19:00:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15030.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:00:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:00:59 smithi122 ceph-mon[93869]: pgmap v8282: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 245 KiB/s wr, 11 op/s 2023-03-09T19:00:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:00:59 smithi181 ceph-mon[94189]: pgmap v8282: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 245 KiB/s wr, 11 op/s 2023-03-09T19:00:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:00:59 smithi038 ceph-mon[92422]: pgmap v8282: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 245 KiB/s wr, 11 op/s 2023-03-09T19:01:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:01:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14571.171593 secs 2023-03-09T19:01:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:01:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14571.171593 secs 2023-03-09T19:01:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:01:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14571.171593 secs 2023-03-09T19:01:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:01:01 smithi122 ceph-mon[93869]: pgmap v8283: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 238 KiB/s wr, 10 op/s 2023-03-09T19:01:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:01:01 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:01:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:01:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:01:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:01:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:01:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:01:01 smithi181 ceph-mon[94189]: pgmap v8283: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 238 KiB/s wr, 10 op/s 2023-03-09T19:01:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:01:01 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:01:01.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:01:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:01:01.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:01:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:01:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:01:01 smithi038 ceph-mon[92422]: pgmap v8283: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 238 KiB/s wr, 10 op/s 2023-03-09T19:01:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:01:01 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:01:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:01:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:01:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:01:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:01:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:01:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14573.456848 secs 2023-03-09T19:01:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:01:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14573.456848 secs 2023-03-09T19:01:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:01:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14573.456848 secs 2023-03-09T19:01:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:01:03 smithi122 ceph-mon[93869]: pgmap v8284: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 247 KiB/s rd, 206 KiB/s wr, 12 op/s 2023-03-09T19:01:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:01:03 smithi181 ceph-mon[94189]: pgmap v8284: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 247 KiB/s rd, 206 KiB/s wr, 12 op/s 2023-03-09T19:01:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:01:03 smithi038 ceph-mon[92422]: pgmap v8284: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 247 KiB/s rd, 206 KiB/s wr, 12 op/s 2023-03-09T19:01:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15035.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:01:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:01:05 smithi122 ceph-mon[93869]: pgmap v8285: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 215 KiB/s wr, 21 op/s 2023-03-09T19:01:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:01:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14576.171650 secs 2023-03-09T19:01:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:01:05 smithi181 ceph-mon[94189]: pgmap v8285: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 215 KiB/s wr, 21 op/s 2023-03-09T19:01:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:01:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14576.171650 secs 2023-03-09T19:01:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:01:05 smithi038 ceph-mon[92422]: pgmap v8285: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 215 KiB/s wr, 21 op/s 2023-03-09T19:01:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:01:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14576.171650 secs 2023-03-09T19:01:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:01:06 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:01:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:01:06 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:01:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:01:06 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:01:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:01:06 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:01:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:01:06 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:01:06.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:01:06 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:01:06.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:01:06 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:01:06.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:01:06 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:01:06.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:01:06 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:01:06.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:01:06 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:01:06.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:01:06 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:01:06.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:01:06 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:01:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:01:07 smithi038 ceph-mon[92422]: pgmap v8286: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 197 KiB/s wr, 25 op/s 2023-03-09T19:01:07.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:01:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14578.456921 secs 2023-03-09T19:01:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:01:07 smithi122 ceph-mon[93869]: pgmap v8286: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 197 KiB/s wr, 25 op/s 2023-03-09T19:01:07.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:01:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14578.456921 secs 2023-03-09T19:01:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:01:07 smithi181 ceph-mon[94189]: pgmap v8286: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 197 KiB/s wr, 25 op/s 2023-03-09T19:01:07.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:01:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14578.456921 secs 2023-03-09T19:01:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15040.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:01:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:01:09 smithi122 ceph-mon[93869]: pgmap v8287: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 197 KiB/s wr, 23 op/s 2023-03-09T19:01:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:01:09 smithi181 ceph-mon[94189]: pgmap v8287: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 197 KiB/s wr, 23 op/s 2023-03-09T19:01:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:01:09 smithi038 ceph-mon[92422]: pgmap v8287: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 197 KiB/s wr, 23 op/s 2023-03-09T19:01:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:01:10 smithi122 ceph-mon[93869]: osdmap e731: 12 total, 12 up, 12 in 2023-03-09T19:01:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:01:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14581.171714 secs 2023-03-09T19:01:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:01:10 smithi181 ceph-mon[94189]: osdmap e731: 12 total, 12 up, 12 in 2023-03-09T19:01:10.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:01:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14581.171714 secs 2023-03-09T19:01:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:01:10 smithi038 ceph-mon[92422]: osdmap e731: 12 total, 12 up, 12 in 2023-03-09T19:01:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:01:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14581.171714 secs 2023-03-09T19:01:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:01:11 smithi122 ceph-mon[93869]: pgmap v8289: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 200 KiB/s wr, 26 op/s 2023-03-09T19:01:11.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:01:11 smithi181 ceph-mon[94189]: pgmap v8289: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 200 KiB/s wr, 26 op/s 2023-03-09T19:01:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:01:11 smithi038 ceph-mon[92422]: pgmap v8289: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 200 KiB/s wr, 26 op/s 2023-03-09T19:01:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:01:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14583.457042 secs 2023-03-09T19:01:12.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:01:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14583.457042 secs 2023-03-09T19:01:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:01:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14583.457042 secs 2023-03-09T19:01:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15045.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:01:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:01:13 smithi122 ceph-mon[93869]: pgmap v8290: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 108 KiB/s wr, 25 op/s 2023-03-09T19:01:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:01:13 smithi181 ceph-mon[94189]: pgmap v8290: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 108 KiB/s wr, 25 op/s 2023-03-09T19:01:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:01:13 smithi038 ceph-mon[92422]: pgmap v8290: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 108 KiB/s wr, 25 op/s 2023-03-09T19:01:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:01:15 smithi122 ceph-mon[93869]: pgmap v8291: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 82 KiB/s wr, 10 op/s 2023-03-09T19:01:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:01:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14586.171774 secs 2023-03-09T19:01:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:01:15 smithi181 ceph-mon[94189]: pgmap v8291: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 82 KiB/s wr, 10 op/s 2023-03-09T19:01:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:01:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14586.171774 secs 2023-03-09T19:01:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:01:15 smithi038 ceph-mon[92422]: pgmap v8291: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 82 KiB/s wr, 10 op/s 2023-03-09T19:01:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:01:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14586.171774 secs 2023-03-09T19:01:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:01:17 smithi122 ceph-mon[93869]: pgmap v8292: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 39 KiB/s wr, 7 op/s 2023-03-09T19:01:17.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:01:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14588.457178 secs 2023-03-09T19:01:17.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:01:17 smithi122 ceph-mon[93869]: osdmap e732: 12 total, 12 up, 12 in 2023-03-09T19:01:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:01:17 smithi181 ceph-mon[94189]: pgmap v8292: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 39 KiB/s wr, 7 op/s 2023-03-09T19:01:17.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:01:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14588.457178 secs 2023-03-09T19:01:17.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:01:17 smithi181 ceph-mon[94189]: osdmap e732: 12 total, 12 up, 12 in 2023-03-09T19:01:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:01:17 smithi038 ceph-mon[92422]: pgmap v8292: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 39 KiB/s wr, 7 op/s 2023-03-09T19:01:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:01:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14588.457178 secs 2023-03-09T19:01:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:01:17 smithi038 ceph-mon[92422]: osdmap e732: 12 total, 12 up, 12 in 2023-03-09T19:01:18.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15050.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:01:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:01:19 smithi122 ceph-mon[93869]: pgmap v8294: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T19:01:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:01:19 smithi181 ceph-mon[94189]: pgmap v8294: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T19:01:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:01:19 smithi038 ceph-mon[92422]: pgmap v8294: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T19:01:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:01:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14591.171858 secs 2023-03-09T19:01:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:01:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14591.171858 secs 2023-03-09T19:01:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:01:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14591.171858 secs 2023-03-09T19:01:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:01:21 smithi122 ceph-mon[93869]: pgmap v8295: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 39 KiB/s wr, 6 op/s 2023-03-09T19:01:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:01:21 smithi181 ceph-mon[94189]: pgmap v8295: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 39 KiB/s wr, 6 op/s 2023-03-09T19:01:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:01:21 smithi038 ceph-mon[92422]: pgmap v8295: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 39 KiB/s wr, 6 op/s 2023-03-09T19:01:22.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:01:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14593.457299 secs 2023-03-09T19:01:22.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:01:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14593.457299 secs 2023-03-09T19:01:22.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:01:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14593.457299 secs 2023-03-09T19:01:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15055.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:01:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:01:23 smithi122 ceph-mon[93869]: pgmap v8296: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 39 KiB/s wr, 6 op/s 2023-03-09T19:01:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:01:23 smithi181 ceph-mon[94189]: pgmap v8296: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 39 KiB/s wr, 6 op/s 2023-03-09T19:01:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:01:23 smithi038 ceph-mon[92422]: pgmap v8296: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 39 KiB/s wr, 6 op/s 2023-03-09T19:01:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:01:25 smithi122 ceph-mon[93869]: pgmap v8297: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 39 KiB/s wr, 6 op/s 2023-03-09T19:01:25.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:01:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14596.171924 secs 2023-03-09T19:01:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:01:25 smithi181 ceph-mon[94189]: pgmap v8297: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 39 KiB/s wr, 6 op/s 2023-03-09T19:01:25.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:01:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14596.171924 secs 2023-03-09T19:01:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:01:25 smithi038 ceph-mon[92422]: pgmap v8297: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 39 KiB/s wr, 6 op/s 2023-03-09T19:01:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:01:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14596.171924 secs 2023-03-09T19:01:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:01:27 smithi122 ceph-mon[93869]: pgmap v8298: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-09T19:01:27.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:01:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14598.457422 secs 2023-03-09T19:01:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:01:27 smithi181 ceph-mon[94189]: pgmap v8298: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-09T19:01:27.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:01:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14598.457422 secs 2023-03-09T19:01:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:01:27 smithi038 ceph-mon[92422]: pgmap v8298: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-09T19:01:28.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:01:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14598.457422 secs 2023-03-09T19:01:28.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15060.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:01:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:01:29 smithi122 ceph-mon[93869]: pgmap v8299: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T19:01:29.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:01:29 smithi181 ceph-mon[94189]: pgmap v8299: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T19:01:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:01:29 smithi038 ceph-mon[92422]: pgmap v8299: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T19:01:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:01:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14601.172002 secs 2023-03-09T19:01:30.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:01:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14601.172002 secs 2023-03-09T19:01:30.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:01:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14601.172002 secs 2023-03-09T19:01:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:01:31 smithi122 ceph-mon[93869]: pgmap v8300: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T19:01:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:01:31 smithi181 ceph-mon[94189]: pgmap v8300: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T19:01:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:01:31 smithi038 ceph-mon[92422]: pgmap v8300: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T19:01:32.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:01:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14603.457538 secs 2023-03-09T19:01:32.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:01:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14603.457538 secs 2023-03-09T19:01:32.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:01:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14603.457538 secs 2023-03-09T19:01:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15065.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:01:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:01:33 smithi122 ceph-mon[93869]: pgmap v8301: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 8 op/s 2023-03-09T19:01:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:01:33 smithi181 ceph-mon[94189]: pgmap v8301: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 8 op/s 2023-03-09T19:01:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:01:33 smithi038 ceph-mon[92422]: pgmap v8301: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 8 op/s 2023-03-09T19:01:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:01:35 smithi122 ceph-mon[93869]: pgmap v8302: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 597 B/s wr, 5 op/s 2023-03-09T19:01:35.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:01:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14606.172071 secs 2023-03-09T19:01:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:01:35 smithi181 ceph-mon[94189]: pgmap v8302: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 597 B/s wr, 5 op/s 2023-03-09T19:01:35.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:01:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14606.172071 secs 2023-03-09T19:01:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:01:35 smithi038 ceph-mon[92422]: pgmap v8302: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 597 B/s wr, 5 op/s 2023-03-09T19:01:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:01:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14606.172071 secs 2023-03-09T19:01:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:01:37 smithi122 ceph-mon[93869]: pgmap v8303: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 61 KiB/s wr, 8 op/s 2023-03-09T19:01:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:01:37 smithi181 ceph-mon[94189]: pgmap v8303: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 61 KiB/s wr, 8 op/s 2023-03-09T19:01:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:01:37 smithi038 ceph-mon[92422]: pgmap v8303: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 61 KiB/s wr, 8 op/s 2023-03-09T19:01:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15070.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:01:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:01:39 smithi122 ceph-mon[93869]: pgmap v8304: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 61 KiB/s wr, 6 op/s 2023-03-09T19:01:39.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:01:39 smithi181 ceph-mon[94189]: pgmap v8304: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 61 KiB/s wr, 6 op/s 2023-03-09T19:01:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:01:39 smithi038 ceph-mon[92422]: pgmap v8304: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 61 KiB/s wr, 6 op/s 2023-03-09T19:01:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:01:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14611.172130 secs 2023-03-09T19:01:40.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:01:40 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:01:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:01:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14611.172130 secs 2023-03-09T19:01:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:01:40 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:01:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:01:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14611.172130 secs 2023-03-09T19:01:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:01:40 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:01:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:01:41 smithi122 ceph-mon[93869]: pgmap v8305: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 61 KiB/s wr, 6 op/s 2023-03-09T19:01:41.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:01:41 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:01:41.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:01:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:01:41.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:01:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:01:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:01:41 smithi181 ceph-mon[94189]: pgmap v8305: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 61 KiB/s wr, 6 op/s 2023-03-09T19:01:41.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:01:41 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:01:41.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:01:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:01:41.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:01:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:01:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:01:41 smithi038 ceph-mon[92422]: pgmap v8305: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 61 KiB/s wr, 6 op/s 2023-03-09T19:01:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:01:41 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:01:42.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:01:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:01:42.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:01:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:01:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:01:43 smithi122 ceph-mon[93869]: pgmap v8306: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 219 KiB/s wr, 10 op/s 2023-03-09T19:01:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:01:43 smithi181 ceph-mon[94189]: pgmap v8306: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 219 KiB/s wr, 10 op/s 2023-03-09T19:01:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:01:43 smithi038 ceph-mon[92422]: pgmap v8306: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 219 KiB/s wr, 10 op/s 2023-03-09T19:01:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15075.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:01:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:01:45 smithi122 ceph-mon[93869]: pgmap v8307: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 218 KiB/s wr, 7 op/s 2023-03-09T19:01:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:01:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14616.172206 secs 2023-03-09T19:01:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:01:45 smithi181 ceph-mon[94189]: pgmap v8307: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 218 KiB/s wr, 7 op/s 2023-03-09T19:01:45.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:01:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14616.172206 secs 2023-03-09T19:01:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:01:45 smithi038 ceph-mon[92422]: pgmap v8307: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 218 KiB/s wr, 7 op/s 2023-03-09T19:01:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:01:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14616.172206 secs 2023-03-09T19:01:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:01:47 smithi181 ceph-mon[94189]: pgmap v8308: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 218 KiB/s wr, 10 op/s 2023-03-09T19:01:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:01:47 smithi038 ceph-mon[92422]: pgmap v8308: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 218 KiB/s wr, 10 op/s 2023-03-09T19:01:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:01:47 smithi122 ceph-mon[93869]: pgmap v8308: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 218 KiB/s wr, 10 op/s 2023-03-09T19:01:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15080.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:01:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:01:49 smithi038 ceph-mon[92422]: pgmap v8309: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 287 KiB/s wr, 10 op/s 2023-03-09T19:01:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:01:49 smithi122 ceph-mon[93869]: pgmap v8309: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 287 KiB/s wr, 10 op/s 2023-03-09T19:01:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:01:49 smithi181 ceph-mon[94189]: pgmap v8309: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 287 KiB/s wr, 10 op/s 2023-03-09T19:01:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:01:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14621.172268 secs 2023-03-09T19:01:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:01:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14621.172268 secs 2023-03-09T19:01:50.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:01:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14621.172268 secs 2023-03-09T19:01:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:01:51 smithi038 ceph-mon[92422]: pgmap v8310: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 287 KiB/s wr, 10 op/s 2023-03-09T19:01:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:01:51 smithi122 ceph-mon[93869]: pgmap v8310: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 287 KiB/s wr, 10 op/s 2023-03-09T19:01:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:01:51 smithi181 ceph-mon[94189]: pgmap v8310: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 287 KiB/s wr, 10 op/s 2023-03-09T19:01:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:01:52 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 14623.458078 secs 2023-03-09T19:01:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:01:52 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 14623.458078 secs 2023-03-09T19:01:52.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:01:52 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 14623.458078 secs 2023-03-09T19:01:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:01:53 smithi038 ceph-mon[92422]: pgmap v8311: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 409 KiB/s wr, 14 op/s 2023-03-09T19:01:53.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15085.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:01:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:01:53 smithi122 ceph-mon[93869]: pgmap v8311: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 409 KiB/s wr, 14 op/s 2023-03-09T19:01:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:01:53 smithi181 ceph-mon[94189]: pgmap v8311: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 409 KiB/s wr, 14 op/s 2023-03-09T19:01:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:01:55 smithi122 ceph-mon[93869]: pgmap v8312: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 251 KiB/s wr, 10 op/s 2023-03-09T19:01:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:01:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14626.172326 secs 2023-03-09T19:01:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:01:55 smithi181 ceph-mon[94189]: pgmap v8312: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 251 KiB/s wr, 10 op/s 2023-03-09T19:01:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:01:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14626.172326 secs 2023-03-09T19:01:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:01:55 smithi038 ceph-mon[92422]: pgmap v8312: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 251 KiB/s wr, 10 op/s 2023-03-09T19:01:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:01:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14626.172326 secs 2023-03-09T19:01:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:01:56 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:01:56.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:01:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:01:56.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:01:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:01:56.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:01:56 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:01:56.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:01:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:01:56.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:01:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:01:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:01:56 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:01:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:01:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:01:57.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:01:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:01:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:01:57 smithi122 ceph-mon[93869]: pgmap v8313: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 251 KiB/s wr, 12 op/s 2023-03-09T19:01:57.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:01:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14628.458198 secs 2023-03-09T19:01:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:01:57 smithi181 ceph-mon[94189]: pgmap v8313: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 251 KiB/s wr, 12 op/s 2023-03-09T19:01:57.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:01:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14628.458198 secs 2023-03-09T19:01:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:01:57 smithi038 ceph-mon[92422]: pgmap v8313: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 251 KiB/s wr, 12 op/s 2023-03-09T19:01:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:01:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14628.458198 secs 2023-03-09T19:01:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15090.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:01:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:01:59 smithi122 ceph-mon[93869]: pgmap v8314: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 347 KiB/s wr, 12 op/s 2023-03-09T19:01:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:01:59 smithi181 ceph-mon[94189]: pgmap v8314: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 347 KiB/s wr, 12 op/s 2023-03-09T19:01:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:01:59 smithi038 ceph-mon[92422]: pgmap v8314: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 347 KiB/s wr, 12 op/s 2023-03-09T19:02:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:02:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14631.172412 secs 2023-03-09T19:02:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:02:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14631.172412 secs 2023-03-09T19:02:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:02:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14631.172412 secs 2023-03-09T19:02:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:02:01 smithi122 ceph-mon[93869]: pgmap v8315: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 219 KiB/s wr, 9 op/s 2023-03-09T19:02:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:02:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:02:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:02:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:02:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:02:01 smithi181 ceph-mon[94189]: pgmap v8315: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 219 KiB/s wr, 9 op/s 2023-03-09T19:02:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:02:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:02:01.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:02:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:02:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:02:01 smithi038 ceph-mon[92422]: pgmap v8315: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 219 KiB/s wr, 9 op/s 2023-03-09T19:02:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:02:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:02:02.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:02:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:02:02.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:02:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14633.458306 secs 2023-03-09T19:02:02.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:02:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14633.458306 secs 2023-03-09T19:02:02.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:02:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14633.458306 secs 2023-03-09T19:02:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15095.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:02:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:02:03 smithi122 ceph-mon[93869]: pgmap v8316: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 328 KiB/s wr, 15 op/s 2023-03-09T19:02:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:02:03 smithi181 ceph-mon[94189]: pgmap v8316: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 328 KiB/s wr, 15 op/s 2023-03-09T19:02:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:02:03 smithi038 ceph-mon[92422]: pgmap v8316: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 328 KiB/s wr, 15 op/s 2023-03-09T19:02:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:02:05 smithi122 ceph-mon[93869]: pgmap v8317: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 227 KiB/s wr, 23 op/s 2023-03-09T19:02:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:02:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14636.172498 secs 2023-03-09T19:02:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:02:05 smithi181 ceph-mon[94189]: pgmap v8317: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 227 KiB/s wr, 23 op/s 2023-03-09T19:02:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:02:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14636.172498 secs 2023-03-09T19:02:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:02:05 smithi038 ceph-mon[92422]: pgmap v8317: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 227 KiB/s wr, 23 op/s 2023-03-09T19:02:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:02:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14636.172498 secs 2023-03-09T19:02:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:02:06 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:02:06.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:02:06 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:02:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:02:06 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:02:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:02:07 smithi122 ceph-mon[93869]: pgmap v8318: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 227 KiB/s wr, 24 op/s 2023-03-09T19:02:07.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:02:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14638.458369 secs 2023-03-09T19:02:07.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:02:07 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:02:07.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:02:07 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:02:07.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:02:07 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:02:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:02:07 smithi181 ceph-mon[94189]: pgmap v8318: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 227 KiB/s wr, 24 op/s 2023-03-09T19:02:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:02:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14638.458369 secs 2023-03-09T19:02:07.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:02:07 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:02:07.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:02:07 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:02:07.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:02:07 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:02:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:02:07 smithi038 ceph-mon[92422]: pgmap v8318: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 227 KiB/s wr, 24 op/s 2023-03-09T19:02:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:02:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14638.458369 secs 2023-03-09T19:02:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:02:07 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:02:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:02:07 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:02:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:02:07 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:02:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15100.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:02:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:02:09 smithi122 ceph-mon[93869]: pgmap v8319: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 272 KiB/s wr, 24 op/s 2023-03-09T19:02:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:02:09 smithi181 ceph-mon[94189]: pgmap v8319: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 272 KiB/s wr, 24 op/s 2023-03-09T19:02:10.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:02:09 smithi038 ceph-mon[92422]: pgmap v8319: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 272 KiB/s wr, 24 op/s 2023-03-09T19:02:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:02:10 smithi122 ceph-mon[93869]: osdmap e733: 12 total, 12 up, 12 in 2023-03-09T19:02:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:02:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14641.172562 secs 2023-03-09T19:02:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:02:10 smithi181 ceph-mon[94189]: osdmap e733: 12 total, 12 up, 12 in 2023-03-09T19:02:10.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:02:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14641.172562 secs 2023-03-09T19:02:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:02:10 smithi038 ceph-mon[92422]: osdmap e733: 12 total, 12 up, 12 in 2023-03-09T19:02:11.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:02:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14641.172562 secs 2023-03-09T19:02:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:02:11 smithi122 ceph-mon[93869]: pgmap v8321: 129 pgs: 1 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 122 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 211 KiB/s wr, 26 op/s 2023-03-09T19:02:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:02:11 smithi181 ceph-mon[94189]: pgmap v8321: 129 pgs: 1 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 122 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 211 KiB/s wr, 26 op/s 2023-03-09T19:02:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:02:11 smithi038 ceph-mon[92422]: pgmap v8321: 129 pgs: 1 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 122 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 211 KiB/s wr, 26 op/s 2023-03-09T19:02:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:02:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14643.458519 secs 2023-03-09T19:02:12.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:02:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14643.458519 secs 2023-03-09T19:02:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:02:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14643.458519 secs 2023-03-09T19:02:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15105.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:02:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:02:13 smithi122 ceph-mon[93869]: pgmap v8322: 129 pgs: 1 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 122 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 80 KiB/s wr, 21 op/s 2023-03-09T19:02:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:02:13 smithi181 ceph-mon[94189]: pgmap v8322: 129 pgs: 1 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 122 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 80 KiB/s wr, 21 op/s 2023-03-09T19:02:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:02:13 smithi038 ceph-mon[92422]: pgmap v8322: 129 pgs: 1 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 122 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 80 KiB/s wr, 21 op/s 2023-03-09T19:02:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:02:15 smithi122 ceph-mon[93869]: pgmap v8323: 129 pgs: 1 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 122 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 54 KiB/s wr, 7 op/s 2023-03-09T19:02:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:02:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14646.172623 secs 2023-03-09T19:02:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:02:15 smithi181 ceph-mon[94189]: pgmap v8323: 129 pgs: 1 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 122 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 54 KiB/s wr, 7 op/s 2023-03-09T19:02:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:02:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14646.172623 secs 2023-03-09T19:02:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:02:15 smithi038 ceph-mon[92422]: pgmap v8323: 129 pgs: 1 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 122 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 54 KiB/s wr, 7 op/s 2023-03-09T19:02:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:02:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14646.172623 secs 2023-03-09T19:02:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:02:17 smithi122 ceph-mon[93869]: pgmap v8324: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 54 KiB/s wr, 7 op/s 2023-03-09T19:02:17.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:02:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14648.458657 secs 2023-03-09T19:02:17.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:02:17 smithi122 ceph-mon[93869]: osdmap e734: 12 total, 12 up, 12 in 2023-03-09T19:02:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:02:17 smithi181 ceph-mon[94189]: pgmap v8324: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 54 KiB/s wr, 7 op/s 2023-03-09T19:02:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:02:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14648.458657 secs 2023-03-09T19:02:17.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:02:17 smithi181 ceph-mon[94189]: osdmap e734: 12 total, 12 up, 12 in 2023-03-09T19:02:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:02:17 smithi038 ceph-mon[92422]: pgmap v8324: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 54 KiB/s wr, 7 op/s 2023-03-09T19:02:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:02:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14648.458657 secs 2023-03-09T19:02:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:02:17 smithi038 ceph-mon[92422]: osdmap e734: 12 total, 12 up, 12 in 2023-03-09T19:02:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15110.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:02:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:02:19 smithi122 ceph-mon[93869]: pgmap v8326: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 232 B/s wr, 7 op/s 2023-03-09T19:02:19.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:02:19 smithi181 ceph-mon[94189]: pgmap v8326: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 232 B/s wr, 7 op/s 2023-03-09T19:02:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:02:19 smithi038 ceph-mon[92422]: pgmap v8326: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 232 B/s wr, 7 op/s 2023-03-09T19:02:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:02:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14651.172682 secs 2023-03-09T19:02:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:02:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14651.172682 secs 2023-03-09T19:02:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:02:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14651.172682 secs 2023-03-09T19:02:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:02:21 smithi122 ceph-mon[93869]: pgmap v8327: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T19:02:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:02:21 smithi181 ceph-mon[94189]: pgmap v8327: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T19:02:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:02:21 smithi038 ceph-mon[92422]: pgmap v8327: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T19:02:22.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:02:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14653.458777 secs 2023-03-09T19:02:22.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:02:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14653.458777 secs 2023-03-09T19:02:22.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:02:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14653.458777 secs 2023-03-09T19:02:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15115.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:02:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:02:23 smithi122 ceph-mon[93869]: pgmap v8328: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T19:02:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:02:23 smithi181 ceph-mon[94189]: pgmap v8328: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T19:02:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:02:23 smithi038 ceph-mon[92422]: pgmap v8328: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T19:02:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:02:25 smithi122 ceph-mon[93869]: pgmap v8329: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T19:02:25.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:02:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14656.172741 secs 2023-03-09T19:02:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:02:25 smithi181 ceph-mon[94189]: pgmap v8329: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T19:02:25.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:02:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14656.172741 secs 2023-03-09T19:02:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:02:25 smithi038 ceph-mon[92422]: pgmap v8329: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T19:02:26.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:02:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14656.172741 secs 2023-03-09T19:02:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:02:27 smithi122 ceph-mon[93869]: pgmap v8330: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T19:02:27.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:02:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14658.458895 secs 2023-03-09T19:02:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:02:27 smithi181 ceph-mon[94189]: pgmap v8330: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T19:02:27.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:02:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14658.458895 secs 2023-03-09T19:02:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:02:27 smithi038 ceph-mon[92422]: pgmap v8330: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T19:02:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:02:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14658.458895 secs 2023-03-09T19:02:28.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15120.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:02:29.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:02:29 smithi122 ceph-mon[93869]: pgmap v8331: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2023-03-09T19:02:29.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:02:29 smithi181 ceph-mon[94189]: pgmap v8331: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2023-03-09T19:02:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:02:29 smithi038 ceph-mon[92422]: pgmap v8331: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2023-03-09T19:02:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:02:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14661.172827 secs 2023-03-09T19:02:30.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:02:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14661.172827 secs 2023-03-09T19:02:30.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:02:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14661.172827 secs 2023-03-09T19:02:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:02:31 smithi122 ceph-mon[93869]: pgmap v8332: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T19:02:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:02:31 smithi181 ceph-mon[94189]: pgmap v8332: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T19:02:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:02:31 smithi038 ceph-mon[92422]: pgmap v8332: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T19:02:32.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:02:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14663.458991 secs 2023-03-09T19:02:32.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:02:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14663.458991 secs 2023-03-09T19:02:32.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:02:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14663.458991 secs 2023-03-09T19:02:33.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15125.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:02:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:02:33 smithi122 ceph-mon[93869]: pgmap v8333: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 39 KiB/s wr, 8 op/s 2023-03-09T19:02:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:02:33 smithi181 ceph-mon[94189]: pgmap v8333: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 39 KiB/s wr, 8 op/s 2023-03-09T19:02:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:02:33 smithi038 ceph-mon[92422]: pgmap v8333: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 39 KiB/s wr, 8 op/s 2023-03-09T19:02:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:02:35 smithi122 ceph-mon[93869]: pgmap v8334: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 39 KiB/s wr, 6 op/s 2023-03-09T19:02:35.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:02:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14666.172889 secs 2023-03-09T19:02:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:02:35 smithi181 ceph-mon[94189]: pgmap v8334: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 39 KiB/s wr, 6 op/s 2023-03-09T19:02:35.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:02:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14666.172889 secs 2023-03-09T19:02:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:02:35 smithi038 ceph-mon[92422]: pgmap v8334: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 39 KiB/s wr, 6 op/s 2023-03-09T19:02:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:02:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14666.172889 secs 2023-03-09T19:02:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:02:37 smithi181 ceph-mon[94189]: pgmap v8335: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 39 KiB/s wr, 7 op/s 2023-03-09T19:02:37.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:02:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14668.459126 secs 2023-03-09T19:02:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:02:37 smithi038 ceph-mon[92422]: pgmap v8335: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 39 KiB/s wr, 7 op/s 2023-03-09T19:02:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:02:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14668.459126 secs 2023-03-09T19:02:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:02:37 smithi122 ceph-mon[93869]: pgmap v8335: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 39 KiB/s wr, 7 op/s 2023-03-09T19:02:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:02:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14668.459126 secs 2023-03-09T19:02:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15130.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:02:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:02:39 smithi038 ceph-mon[92422]: pgmap v8336: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 100 KiB/s wr, 7 op/s 2023-03-09T19:02:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:02:39 smithi122 ceph-mon[93869]: pgmap v8336: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 100 KiB/s wr, 7 op/s 2023-03-09T19:02:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:02:39 smithi181 ceph-mon[94189]: pgmap v8336: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 100 KiB/s wr, 7 op/s 2023-03-09T19:02:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:02:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14671.172953 secs 2023-03-09T19:02:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:02:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14671.172953 secs 2023-03-09T19:02:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:02:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14671.172953 secs 2023-03-09T19:02:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:02:41 smithi038 ceph-mon[92422]: pgmap v8337: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 100 KiB/s wr, 6 op/s 2023-03-09T19:02:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:02:41 smithi122 ceph-mon[93869]: pgmap v8337: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 100 KiB/s wr, 6 op/s 2023-03-09T19:02:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:02:41 smithi181 ceph-mon[94189]: pgmap v8337: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 100 KiB/s wr, 6 op/s 2023-03-09T19:02:42.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:02:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14673.459252 secs 2023-03-09T19:02:43.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:02:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14673.459252 secs 2023-03-09T19:02:43.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:02:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14673.459252 secs 2023-03-09T19:02:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15135.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:02:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:02:43 smithi038 ceph-mon[92422]: pgmap v8338: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 253 KiB/s wr, 10 op/s 2023-03-09T19:02:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:02:43 smithi122 ceph-mon[93869]: pgmap v8338: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 253 KiB/s wr, 10 op/s 2023-03-09T19:02:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:02:43 smithi181 ceph-mon[94189]: pgmap v8338: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 253 KiB/s wr, 10 op/s 2023-03-09T19:02:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:02:45 smithi122 ceph-mon[93869]: pgmap v8339: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 220 KiB/s wr, 8 op/s 2023-03-09T19:02:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:02:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14676.173005 secs 2023-03-09T19:02:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:02:45 smithi181 ceph-mon[94189]: pgmap v8339: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 220 KiB/s wr, 8 op/s 2023-03-09T19:02:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:02:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14676.173005 secs 2023-03-09T19:02:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:02:45 smithi038 ceph-mon[92422]: pgmap v8339: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 220 KiB/s wr, 8 op/s 2023-03-09T19:02:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:02:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14676.173005 secs 2023-03-09T19:02:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:02:47 smithi122 ceph-mon[93869]: pgmap v8340: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 236 KiB/s wr, 11 op/s 2023-03-09T19:02:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:02:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14678.459374 secs 2023-03-09T19:02:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:02:47 smithi181 ceph-mon[94189]: pgmap v8340: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 236 KiB/s wr, 11 op/s 2023-03-09T19:02:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:02:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14678.459374 secs 2023-03-09T19:02:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:02:47 smithi038 ceph-mon[92422]: pgmap v8340: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 236 KiB/s wr, 11 op/s 2023-03-09T19:02:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:02:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14678.459374 secs 2023-03-09T19:02:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15140.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:02:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:02:49 smithi122 ceph-mon[93869]: pgmap v8341: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 435 KiB/s rd, 421 KiB/s wr, 11 op/s 2023-03-09T19:02:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:02:49 smithi181 ceph-mon[94189]: pgmap v8341: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 435 KiB/s rd, 421 KiB/s wr, 11 op/s 2023-03-09T19:02:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:02:49 smithi038 ceph-mon[92422]: pgmap v8341: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 435 KiB/s rd, 421 KiB/s wr, 11 op/s 2023-03-09T19:02:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:02:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14681.173091 secs 2023-03-09T19:02:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:02:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14681.173091 secs 2023-03-09T19:02:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:02:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14681.173091 secs 2023-03-09T19:02:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:02:51 smithi122 ceph-mon[93869]: pgmap v8342: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 361 KiB/s wr, 10 op/s 2023-03-09T19:02:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:02:51 smithi181 ceph-mon[94189]: pgmap v8342: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 361 KiB/s wr, 10 op/s 2023-03-09T19:02:52.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:02:51 smithi038 ceph-mon[92422]: pgmap v8342: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 361 KiB/s wr, 10 op/s 2023-03-09T19:02:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:02:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14683.459491 secs 2023-03-09T19:02:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:02:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14683.459491 secs 2023-03-09T19:02:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:02:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14683.459491 secs 2023-03-09T19:02:53.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15145.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:02:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:02:53 smithi122 ceph-mon[93869]: pgmap v8343: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 373 KiB/s wr, 13 op/s 2023-03-09T19:02:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:02:53 smithi181 ceph-mon[94189]: pgmap v8343: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 373 KiB/s wr, 13 op/s 2023-03-09T19:02:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:02:53 smithi038 ceph-mon[92422]: pgmap v8343: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 373 KiB/s wr, 13 op/s 2023-03-09T19:02:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:02:55 smithi122 ceph-mon[93869]: pgmap v8344: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 260 KiB/s wr, 11 op/s 2023-03-09T19:02:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:02:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14686.173151 secs 2023-03-09T19:02:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:02:55 smithi181 ceph-mon[94189]: pgmap v8344: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 260 KiB/s wr, 11 op/s 2023-03-09T19:02:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:02:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14686.173151 secs 2023-03-09T19:02:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:02:55 smithi038 ceph-mon[92422]: pgmap v8344: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 260 KiB/s wr, 11 op/s 2023-03-09T19:02:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:02:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14686.173151 secs 2023-03-09T19:02:58.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:02:57 smithi122 ceph-mon[93869]: pgmap v8345: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 276 KiB/s wr, 12 op/s 2023-03-09T19:02:58.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:02:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14688.459607 secs 2023-03-09T19:02:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:02:57 smithi181 ceph-mon[94189]: pgmap v8345: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 276 KiB/s wr, 12 op/s 2023-03-09T19:02:58.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:02:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14688.459607 secs 2023-03-09T19:02:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:02:57 smithi038 ceph-mon[92422]: pgmap v8345: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 276 KiB/s wr, 12 op/s 2023-03-09T19:02:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:02:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14688.459607 secs 2023-03-09T19:02:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15150.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:03:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:02:59 smithi122 ceph-mon[93869]: pgmap v8346: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 373 KiB/s wr, 11 op/s 2023-03-09T19:03:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:02:59 smithi181 ceph-mon[94189]: pgmap v8346: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 373 KiB/s wr, 11 op/s 2023-03-09T19:03:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:02:59 smithi038 ceph-mon[92422]: pgmap v8346: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 373 KiB/s wr, 11 op/s 2023-03-09T19:03:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14691.173210 secs 2023-03-09T19:03:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14691.173210 secs 2023-03-09T19:03:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14691.173210 secs 2023-03-09T19:03:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:01 smithi122 ceph-mon[93869]: pgmap v8347: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 188 KiB/s wr, 9 op/s 2023-03-09T19:03:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:01 smithi181 ceph-mon[94189]: pgmap v8347: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 188 KiB/s wr, 9 op/s 2023-03-09T19:03:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:01 smithi038 ceph-mon[92422]: pgmap v8347: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 188 KiB/s wr, 9 op/s 2023-03-09T19:03:03.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14693.459726 secs 2023-03-09T19:03:03.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14693.459726 secs 2023-03-09T19:03:03.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14693.459726 secs 2023-03-09T19:03:03.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15155.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:03:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:03 smithi122 ceph-mon[93869]: pgmap v8348: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 271 KiB/s wr, 15 op/s 2023-03-09T19:03:04.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:03 smithi181 ceph-mon[94189]: pgmap v8348: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 271 KiB/s wr, 15 op/s 2023-03-09T19:03:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:03 smithi038 ceph-mon[92422]: pgmap v8348: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 271 KiB/s wr, 15 op/s 2023-03-09T19:03:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:05 smithi122 ceph-mon[93869]: pgmap v8349: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 279 KiB/s wr, 24 op/s 2023-03-09T19:03:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14696.173270 secs 2023-03-09T19:03:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:05 smithi181 ceph-mon[94189]: pgmap v8349: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 279 KiB/s wr, 24 op/s 2023-03-09T19:03:06.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14696.173270 secs 2023-03-09T19:03:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:05 smithi038 ceph-mon[92422]: pgmap v8349: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 279 KiB/s wr, 24 op/s 2023-03-09T19:03:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14696.173270 secs 2023-03-09T19:03:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:07 smithi122 ceph-mon[93869]: pgmap v8350: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 285 KiB/s wr, 26 op/s 2023-03-09T19:03:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14698.459850 secs 2023-03-09T19:03:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:07 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:03:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:07 smithi181 ceph-mon[94189]: pgmap v8350: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 285 KiB/s wr, 26 op/s 2023-03-09T19:03:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14698.459850 secs 2023-03-09T19:03:08.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:07 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:03:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:07 smithi038 ceph-mon[92422]: pgmap v8350: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 285 KiB/s wr, 26 op/s 2023-03-09T19:03:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14698.459850 secs 2023-03-09T19:03:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:07 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:03:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15160.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:03:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:09 smithi122 ceph-mon[93869]: pgmap v8351: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 263 KiB/s wr, 23 op/s 2023-03-09T19:03:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:09 smithi181 ceph-mon[94189]: pgmap v8351: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 263 KiB/s wr, 23 op/s 2023-03-09T19:03:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:09 smithi038 ceph-mon[92422]: pgmap v8351: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 263 KiB/s wr, 23 op/s 2023-03-09T19:03:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14701.173333 secs 2023-03-09T19:03:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:10 smithi122 ceph-mon[93869]: osdmap e735: 12 total, 12 up, 12 in 2023-03-09T19:03:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:03:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:03:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:03:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:03:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:03:11.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:03:11.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:03:11.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:03:11.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:03:11.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:03:11.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:03:11.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:03:11.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:03:11.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:03:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14701.173333 secs 2023-03-09T19:03:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:10 smithi181 ceph-mon[94189]: osdmap e735: 12 total, 12 up, 12 in 2023-03-09T19:03:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:03:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:03:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:03:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:03:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:03:11.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:03:11.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:03:11.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:03:11.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:03:11.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:03:11.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:03:11.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:03:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:03:11.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:03:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14701.173333 secs 2023-03-09T19:03:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:10 smithi038 ceph-mon[92422]: osdmap e735: 12 total, 12 up, 12 in 2023-03-09T19:03:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:03:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:03:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:03:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:03:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:03:11.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:03:11.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:03:11.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:03:11.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:03:11.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:03:11.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:03:11.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:03:11.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:03:11.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:03:12.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:12 smithi122 ceph-mon[93869]: pgmap v8353: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 180 KiB/s wr, 25 op/s 2023-03-09T19:03:12.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:12 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi038 to 167.8M 2023-03-09T19:03:12.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:12 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T19:03:12.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:12 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi038 to 176007004: error parsing value: Value '176007004' is below minimum 939524096 2023-03-09T19:03:12.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:12 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T19:03:12.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:12 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:03:12.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:12 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:03:12.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:12 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:03:12.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:12 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:03:12.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:12 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:03:12.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:12 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:03:12.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:12 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:03:12.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:12 smithi181 ceph-mon[94189]: pgmap v8353: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 180 KiB/s wr, 25 op/s 2023-03-09T19:03:12.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:12 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi038 to 167.8M 2023-03-09T19:03:12.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:12 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T19:03:12.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:12 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi038 to 176007004: error parsing value: Value '176007004' is below minimum 939524096 2023-03-09T19:03:12.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:12 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T19:03:12.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:12 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:03:12.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:12 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:03:12.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:12 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:03:12.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:12 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:03:12.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:12 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:03:12.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:12 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:03:12.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:12 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:03:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:12 smithi038 ceph-mon[92422]: pgmap v8353: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 180 KiB/s wr, 25 op/s 2023-03-09T19:03:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:12 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi038 to 167.8M 2023-03-09T19:03:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:12 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T19:03:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:12 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi038 to 176007004: error parsing value: Value '176007004' is below minimum 939524096 2023-03-09T19:03:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:12 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T19:03:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:12 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:03:12.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:12 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:03:12.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:12 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:03:12.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:12 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:03:12.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:12 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:03:12.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:12 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:03:12.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:12 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:03:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:13 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T19:03:13.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:13 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T19:03:13.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14703.459971 secs 2023-03-09T19:03:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:13 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T19:03:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:13 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T19:03:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14703.459971 secs 2023-03-09T19:03:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:13 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T19:03:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:13 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T19:03:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14703.459971 secs 2023-03-09T19:03:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15165.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:03:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:14 smithi122 ceph-mon[93869]: pgmap v8354: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 80 KiB/s wr, 22 op/s 2023-03-09T19:03:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:14 smithi181 ceph-mon[94189]: pgmap v8354: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 80 KiB/s wr, 22 op/s 2023-03-09T19:03:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:14 smithi038 ceph-mon[92422]: pgmap v8354: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 80 KiB/s wr, 22 op/s 2023-03-09T19:03:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14706.173399 secs 2023-03-09T19:03:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14706.173399 secs 2023-03-09T19:03:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14706.173399 secs 2023-03-09T19:03:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:16 smithi122 ceph-mon[93869]: pgmap v8355: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 55 KiB/s wr, 7 op/s 2023-03-09T19:03:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:16 smithi181 ceph-mon[94189]: pgmap v8355: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 55 KiB/s wr, 7 op/s 2023-03-09T19:03:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:16 smithi038 ceph-mon[92422]: pgmap v8355: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 55 KiB/s wr, 7 op/s 2023-03-09T19:03:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:18 smithi122 ceph-mon[93869]: pgmap v8356: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 40 KiB/s wr, 6 op/s 2023-03-09T19:03:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14708.460101 secs 2023-03-09T19:03:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:18 smithi122 ceph-mon[93869]: osdmap e736: 12 total, 12 up, 12 in 2023-03-09T19:03:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:18 smithi181 ceph-mon[94189]: pgmap v8356: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 40 KiB/s wr, 6 op/s 2023-03-09T19:03:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14708.460101 secs 2023-03-09T19:03:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:18 smithi181 ceph-mon[94189]: osdmap e736: 12 total, 12 up, 12 in 2023-03-09T19:03:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:18 smithi038 ceph-mon[92422]: pgmap v8356: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 40 KiB/s wr, 6 op/s 2023-03-09T19:03:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14708.460101 secs 2023-03-09T19:03:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:18 smithi038 ceph-mon[92422]: osdmap e736: 12 total, 12 up, 12 in 2023-03-09T19:03:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15170.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:03:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:20 smithi122 ceph-mon[93869]: pgmap v8358: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 46 KiB/s wr, 7 op/s 2023-03-09T19:03:20.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14711.173475 secs 2023-03-09T19:03:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:20 smithi181 ceph-mon[94189]: pgmap v8358: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 46 KiB/s wr, 7 op/s 2023-03-09T19:03:20.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14711.173475 secs 2023-03-09T19:03:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:20 smithi038 ceph-mon[92422]: pgmap v8358: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 46 KiB/s wr, 7 op/s 2023-03-09T19:03:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14711.173475 secs 2023-03-09T19:03:22.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:22 smithi122 ceph-mon[93869]: pgmap v8359: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 40 KiB/s wr, 6 op/s 2023-03-09T19:03:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:22 smithi181 ceph-mon[94189]: pgmap v8359: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 40 KiB/s wr, 6 op/s 2023-03-09T19:03:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:22 smithi038 ceph-mon[92422]: pgmap v8359: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 40 KiB/s wr, 6 op/s 2023-03-09T19:03:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14713.460212 secs 2023-03-09T19:03:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14713.460212 secs 2023-03-09T19:03:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14713.460212 secs 2023-03-09T19:03:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15175.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:03:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:24 smithi122 ceph-mon[93869]: pgmap v8360: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 39 KiB/s wr, 6 op/s 2023-03-09T19:03:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:24 smithi181 ceph-mon[94189]: pgmap v8360: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 39 KiB/s wr, 6 op/s 2023-03-09T19:03:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:24 smithi038 ceph-mon[92422]: pgmap v8360: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 39 KiB/s wr, 6 op/s 2023-03-09T19:03:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14716.173547 secs 2023-03-09T19:03:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14716.173547 secs 2023-03-09T19:03:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14716.173547 secs 2023-03-09T19:03:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:26 smithi122 ceph-mon[93869]: pgmap v8361: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 39 KiB/s wr, 6 op/s 2023-03-09T19:03:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:26 smithi181 ceph-mon[94189]: pgmap v8361: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 39 KiB/s wr, 6 op/s 2023-03-09T19:03:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:26 smithi038 ceph-mon[92422]: pgmap v8361: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 39 KiB/s wr, 6 op/s 2023-03-09T19:03:27.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:27 smithi122 ceph-mon[93869]: pgmap v8362: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 409 B/s wr, 6 op/s 2023-03-09T19:03:27.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:27 smithi181 ceph-mon[94189]: pgmap v8362: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 409 B/s wr, 6 op/s 2023-03-09T19:03:27.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:27 smithi038 ceph-mon[92422]: pgmap v8362: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 409 B/s wr, 6 op/s 2023-03-09T19:03:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14718.460309 secs 2023-03-09T19:03:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14718.460309 secs 2023-03-09T19:03:28.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14718.460309 secs 2023-03-09T19:03:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15180.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:03:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:29 smithi122 ceph-mon[93869]: pgmap v8363: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 463 B/s wr, 6 op/s 2023-03-09T19:03:29.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:29 smithi181 ceph-mon[94189]: pgmap v8363: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 463 B/s wr, 6 op/s 2023-03-09T19:03:29.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:29 smithi038 ceph-mon[92422]: pgmap v8363: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 463 B/s wr, 6 op/s 2023-03-09T19:03:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14721.173615 secs 2023-03-09T19:03:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14721.173615 secs 2023-03-09T19:03:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14721.173615 secs 2023-03-09T19:03:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:31 smithi122 ceph-mon[93869]: pgmap v8364: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2023-03-09T19:03:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:31 smithi181 ceph-mon[94189]: pgmap v8364: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2023-03-09T19:03:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:31 smithi038 ceph-mon[92422]: pgmap v8364: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2023-03-09T19:03:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14723.460421 secs 2023-03-09T19:03:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14723.460421 secs 2023-03-09T19:03:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14723.460421 secs 2023-03-09T19:03:33.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:33 smithi122 ceph-mon[93869]: pgmap v8365: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 49 KiB/s wr, 8 op/s 2023-03-09T19:03:33.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:33 smithi181 ceph-mon[94189]: pgmap v8365: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 49 KiB/s wr, 8 op/s 2023-03-09T19:03:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:33 smithi038 ceph-mon[92422]: pgmap v8365: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 49 KiB/s wr, 8 op/s 2023-03-09T19:03:33.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15185.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:03:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:35 smithi122 ceph-mon[93869]: pgmap v8366: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 49 KiB/s wr, 6 op/s 2023-03-09T19:03:35.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14726.173698 secs 2023-03-09T19:03:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:35 smithi181 ceph-mon[94189]: pgmap v8366: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 49 KiB/s wr, 6 op/s 2023-03-09T19:03:35.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14726.173698 secs 2023-03-09T19:03:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:35 smithi038 ceph-mon[92422]: pgmap v8366: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 49 KiB/s wr, 6 op/s 2023-03-09T19:03:36.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14726.173698 secs 2023-03-09T19:03:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:37 smithi122 ceph-mon[93869]: pgmap v8367: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 101 KiB/s wr, 9 op/s 2023-03-09T19:03:37.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14728.460534 secs 2023-03-09T19:03:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:37 smithi181 ceph-mon[94189]: pgmap v8367: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 101 KiB/s wr, 9 op/s 2023-03-09T19:03:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14728.460534 secs 2023-03-09T19:03:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:37 smithi038 ceph-mon[92422]: pgmap v8367: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 101 KiB/s wr, 9 op/s 2023-03-09T19:03:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14728.460534 secs 2023-03-09T19:03:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15190.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:03:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:39 smithi122 ceph-mon[93869]: pgmap v8368: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 100 KiB/s wr, 6 op/s 2023-03-09T19:03:39.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:39 smithi181 ceph-mon[94189]: pgmap v8368: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 100 KiB/s wr, 6 op/s 2023-03-09T19:03:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:39 smithi038 ceph-mon[92422]: pgmap v8368: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 100 KiB/s wr, 6 op/s 2023-03-09T19:03:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14731.173760 secs 2023-03-09T19:03:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14731.173760 secs 2023-03-09T19:03:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14731.173760 secs 2023-03-09T19:03:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:41 smithi122 ceph-mon[93869]: pgmap v8369: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 100 KiB/s wr, 6 op/s 2023-03-09T19:03:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:41 smithi181 ceph-mon[94189]: pgmap v8369: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 100 KiB/s wr, 6 op/s 2023-03-09T19:03:42.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:41 smithi038 ceph-mon[92422]: pgmap v8369: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 100 KiB/s wr, 6 op/s 2023-03-09T19:03:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:43 smithi122 ceph-mon[93869]: pgmap v8370: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 164 KiB/s wr, 9 op/s 2023-03-09T19:03:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:43 smithi181 ceph-mon[94189]: pgmap v8370: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 164 KiB/s wr, 9 op/s 2023-03-09T19:03:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:43 smithi038 ceph-mon[92422]: pgmap v8370: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 164 KiB/s wr, 9 op/s 2023-03-09T19:03:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15195.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:03:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:44 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:03:44.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:44 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:03:44.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:44 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:03:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:45 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:03:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:45 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:03:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:45 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:03:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:45 smithi122 ceph-mon[93869]: pgmap v8371: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 115 KiB/s wr, 6 op/s 2023-03-09T19:03:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14736.173829 secs 2023-03-09T19:03:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:45 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:03:45.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:45 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:03:45.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:45 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:03:45.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:45 smithi181 ceph-mon[94189]: pgmap v8371: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 115 KiB/s wr, 6 op/s 2023-03-09T19:03:45.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14736.173829 secs 2023-03-09T19:03:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:45 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:03:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:45 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:03:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:45 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:03:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:45 smithi038 ceph-mon[92422]: pgmap v8371: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 115 KiB/s wr, 6 op/s 2023-03-09T19:03:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14736.173829 secs 2023-03-09T19:03:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:47 smithi122 ceph-mon[93869]: pgmap v8372: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 180 KiB/s wr, 10 op/s 2023-03-09T19:03:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:47 smithi181 ceph-mon[94189]: pgmap v8372: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 180 KiB/s wr, 10 op/s 2023-03-09T19:03:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:47 smithi038 ceph-mon[92422]: pgmap v8372: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 180 KiB/s wr, 10 op/s 2023-03-09T19:03:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15200.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:03:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:49 smithi122 ceph-mon[93869]: pgmap v8373: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 129 KiB/s wr, 8 op/s 2023-03-09T19:03:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:49 smithi181 ceph-mon[94189]: pgmap v8373: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 129 KiB/s wr, 8 op/s 2023-03-09T19:03:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:49 smithi038 ceph-mon[92422]: pgmap v8373: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 129 KiB/s wr, 8 op/s 2023-03-09T19:03:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14741.173913 secs 2023-03-09T19:03:50.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14741.173913 secs 2023-03-09T19:03:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14741.173913 secs 2023-03-09T19:03:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:51 smithi122 ceph-mon[93869]: pgmap v8374: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 129 KiB/s wr, 8 op/s 2023-03-09T19:03:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:51 smithi181 ceph-mon[94189]: pgmap v8374: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 129 KiB/s wr, 8 op/s 2023-03-09T19:03:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:51 smithi038 ceph-mon[92422]: pgmap v8374: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 129 KiB/s wr, 8 op/s 2023-03-09T19:03:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:53 smithi122 ceph-mon[93869]: pgmap v8375: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 435 KiB/s rd, 280 KiB/s wr, 12 op/s 2023-03-09T19:03:53.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:53 smithi181 ceph-mon[94189]: pgmap v8375: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 435 KiB/s rd, 280 KiB/s wr, 12 op/s 2023-03-09T19:03:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:53 smithi038 ceph-mon[92422]: pgmap v8375: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 435 KiB/s rd, 280 KiB/s wr, 12 op/s 2023-03-09T19:03:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15205.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:03:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:55 smithi122 ceph-mon[93869]: pgmap v8376: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 217 KiB/s wr, 9 op/s 2023-03-09T19:03:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14746.173999 secs 2023-03-09T19:03:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:55 smithi181 ceph-mon[94189]: pgmap v8376: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 217 KiB/s wr, 9 op/s 2023-03-09T19:03:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14746.173999 secs 2023-03-09T19:03:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:55 smithi038 ceph-mon[92422]: pgmap v8376: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 217 KiB/s wr, 9 op/s 2023-03-09T19:03:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14746.173999 secs 2023-03-09T19:03:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:57 smithi122 ceph-mon[93869]: pgmap v8377: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 236 KiB/s wr, 12 op/s 2023-03-09T19:03:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:57 smithi181 ceph-mon[94189]: pgmap v8377: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 236 KiB/s wr, 12 op/s 2023-03-09T19:03:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:57 smithi038 ceph-mon[92422]: pgmap v8377: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 236 KiB/s wr, 12 op/s 2023-03-09T19:03:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15210.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:03:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:03:59 smithi122 ceph-mon[93869]: pgmap v8378: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 171 KiB/s wr, 9 op/s 2023-03-09T19:03:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:03:59 smithi181 ceph-mon[94189]: pgmap v8378: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 171 KiB/s wr, 9 op/s 2023-03-09T19:03:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:03:59 smithi038 ceph-mon[92422]: pgmap v8378: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 171 KiB/s wr, 9 op/s 2023-03-09T19:04:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:04:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14751.174058 secs 2023-03-09T19:04:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:04:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14751.174058 secs 2023-03-09T19:04:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:04:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14751.174058 secs 2023-03-09T19:04:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:04:01 smithi122 ceph-mon[93869]: pgmap v8379: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 170 KiB/s wr, 9 op/s 2023-03-09T19:04:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:04:01 smithi181 ceph-mon[94189]: pgmap v8379: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 170 KiB/s wr, 9 op/s 2023-03-09T19:04:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:04:01 smithi038 ceph-mon[92422]: pgmap v8379: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 170 KiB/s wr, 9 op/s 2023-03-09T19:04:02.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:04:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14753.461069 secs 2023-03-09T19:04:02.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:04:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14753.461069 secs 2023-03-09T19:04:02.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:04:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14753.461069 secs 2023-03-09T19:04:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15215.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:04:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:04:03 smithi122 ceph-mon[93869]: pgmap v8380: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 404 KiB/s wr, 16 op/s 2023-03-09T19:04:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:04:03 smithi181 ceph-mon[94189]: pgmap v8380: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 404 KiB/s wr, 16 op/s 2023-03-09T19:04:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:04:03 smithi038 ceph-mon[92422]: pgmap v8380: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 404 KiB/s wr, 16 op/s 2023-03-09T19:04:04.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:04:04 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:04:04.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:04:04 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:04:04.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:04:04 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:04:04.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:04:04 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:04:04.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:04:04 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:04:04.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:04:04 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:04:04.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:04:04 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:04:05.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:04:04 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:04:05.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:04:04 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:04:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:04:05 smithi122 ceph-mon[93869]: pgmap v8381: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 274 KiB/s wr, 23 op/s 2023-03-09T19:04:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:04:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14756.174119 secs 2023-03-09T19:04:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:04:05 smithi181 ceph-mon[94189]: pgmap v8381: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 274 KiB/s wr, 23 op/s 2023-03-09T19:04:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:04:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14756.174119 secs 2023-03-09T19:04:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:04:05 smithi038 ceph-mon[92422]: pgmap v8381: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 274 KiB/s wr, 23 op/s 2023-03-09T19:04:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:04:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14756.174119 secs 2023-03-09T19:04:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:04:07 smithi122 ceph-mon[93869]: pgmap v8382: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 274 KiB/s wr, 26 op/s 2023-03-09T19:04:07.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:04:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14758.461152 secs 2023-03-09T19:04:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:04:07 smithi181 ceph-mon[94189]: pgmap v8382: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 274 KiB/s wr, 26 op/s 2023-03-09T19:04:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:04:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14758.461152 secs 2023-03-09T19:04:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:04:07 smithi038 ceph-mon[92422]: pgmap v8382: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 274 KiB/s wr, 26 op/s 2023-03-09T19:04:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:04:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14758.461152 secs 2023-03-09T19:04:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15220.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:04:09.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:04:09 smithi122 ceph-mon[93869]: pgmap v8383: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 289 KiB/s wr, 24 op/s 2023-03-09T19:04:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:04:09 smithi181 ceph-mon[94189]: pgmap v8383: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 289 KiB/s wr, 24 op/s 2023-03-09T19:04:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:04:09 smithi038 ceph-mon[92422]: pgmap v8383: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 289 KiB/s wr, 24 op/s 2023-03-09T19:04:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:04:10 smithi122 ceph-mon[93869]: osdmap e737: 12 total, 12 up, 12 in 2023-03-09T19:04:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:04:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14761.174182 secs 2023-03-09T19:04:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:04:10 smithi181 ceph-mon[94189]: osdmap e737: 12 total, 12 up, 12 in 2023-03-09T19:04:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:04:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14761.174182 secs 2023-03-09T19:04:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:04:10 smithi038 ceph-mon[92422]: osdmap e737: 12 total, 12 up, 12 in 2023-03-09T19:04:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:04:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14761.174182 secs 2023-03-09T19:04:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:04:11 smithi122 ceph-mon[93869]: pgmap v8385: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 346 KiB/s wr, 28 op/s 2023-03-09T19:04:11.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:04:11 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:04:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:04:11 smithi181 ceph-mon[94189]: pgmap v8385: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 346 KiB/s wr, 28 op/s 2023-03-09T19:04:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:04:11 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:04:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:04:11 smithi038 ceph-mon[92422]: pgmap v8385: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 346 KiB/s wr, 28 op/s 2023-03-09T19:04:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:04:11 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:04:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:04:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14763.461240 secs 2023-03-09T19:04:12.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:04:12 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:04:12.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:04:12 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:04:12.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:04:12 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:04:12.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:04:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14763.461240 secs 2023-03-09T19:04:12.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:04:12 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:04:12.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:04:12 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:04:12.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:04:12 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:04:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:04:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14763.461240 secs 2023-03-09T19:04:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:04:12 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:04:13.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:04:12 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:04:13.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:04:12 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:04:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15225.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:04:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:04:13 smithi122 ceph-mon[93869]: pgmap v8386: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 206 KiB/s wr, 24 op/s 2023-03-09T19:04:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:04:13 smithi181 ceph-mon[94189]: pgmap v8386: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 206 KiB/s wr, 24 op/s 2023-03-09T19:04:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:04:13 smithi038 ceph-mon[92422]: pgmap v8386: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 206 KiB/s wr, 24 op/s 2023-03-09T19:04:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:04:15 smithi122 ceph-mon[93869]: pgmap v8387: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 180 KiB/s wr, 9 op/s 2023-03-09T19:04:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:04:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14766.174242 secs 2023-03-09T19:04:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:04:15 smithi181 ceph-mon[94189]: pgmap v8387: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 180 KiB/s wr, 9 op/s 2023-03-09T19:04:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:04:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14766.174242 secs 2023-03-09T19:04:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:04:15 smithi038 ceph-mon[92422]: pgmap v8387: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 180 KiB/s wr, 9 op/s 2023-03-09T19:04:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:04:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14766.174242 secs 2023-03-09T19:04:17.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:04:17 smithi122 ceph-mon[93869]: pgmap v8388: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 195 KiB/s wr, 9 op/s 2023-03-09T19:04:17.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:04:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14768.461392 secs 2023-03-09T19:04:17.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:04:17 smithi122 ceph-mon[93869]: osdmap e738: 12 total, 12 up, 12 in 2023-03-09T19:04:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:04:17 smithi181 ceph-mon[94189]: pgmap v8388: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 195 KiB/s wr, 9 op/s 2023-03-09T19:04:17.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:04:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14768.461392 secs 2023-03-09T19:04:17.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:04:17 smithi181 ceph-mon[94189]: osdmap e738: 12 total, 12 up, 12 in 2023-03-09T19:04:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:04:17 smithi038 ceph-mon[92422]: pgmap v8388: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 195 KiB/s wr, 9 op/s 2023-03-09T19:04:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:04:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14768.461392 secs 2023-03-09T19:04:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:04:17 smithi038 ceph-mon[92422]: osdmap e738: 12 total, 12 up, 12 in 2023-03-09T19:04:18.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15230.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:04:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:04:19 smithi122 ceph-mon[93869]: pgmap v8390: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 221 KiB/s wr, 10 op/s 2023-03-09T19:04:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:04:19 smithi181 ceph-mon[94189]: pgmap v8390: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 221 KiB/s wr, 10 op/s 2023-03-09T19:04:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:04:19 smithi038 ceph-mon[92422]: pgmap v8390: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 221 KiB/s wr, 10 op/s 2023-03-09T19:04:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:04:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14771.174302 secs 2023-03-09T19:04:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:04:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14771.174302 secs 2023-03-09T19:04:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:04:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14771.174302 secs 2023-03-09T19:04:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:04:21 smithi122 ceph-mon[93869]: pgmap v8391: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 195 KiB/s wr, 8 op/s 2023-03-09T19:04:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:04:21 smithi181 ceph-mon[94189]: pgmap v8391: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 195 KiB/s wr, 8 op/s 2023-03-09T19:04:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:04:21 smithi038 ceph-mon[92422]: pgmap v8391: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 195 KiB/s wr, 8 op/s 2023-03-09T19:04:22.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:04:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14773.461514 secs 2023-03-09T19:04:22.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:04:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14773.461514 secs 2023-03-09T19:04:22.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:04:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14773.461514 secs 2023-03-09T19:04:23.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15235.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:04:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:04:23 smithi122 ceph-mon[93869]: pgmap v8392: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 56 KiB/s wr, 7 op/s 2023-03-09T19:04:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:04:23 smithi181 ceph-mon[94189]: pgmap v8392: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 56 KiB/s wr, 7 op/s 2023-03-09T19:04:24.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:04:23 smithi038 ceph-mon[92422]: pgmap v8392: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 56 KiB/s wr, 7 op/s 2023-03-09T19:04:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:04:25 smithi122 ceph-mon[93869]: pgmap v8393: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 56 KiB/s wr, 7 op/s 2023-03-09T19:04:25.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:04:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14776.174380 secs 2023-03-09T19:04:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:04:25 smithi181 ceph-mon[94189]: pgmap v8393: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 56 KiB/s wr, 7 op/s 2023-03-09T19:04:25.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:04:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14776.174380 secs 2023-03-09T19:04:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:04:25 smithi038 ceph-mon[92422]: pgmap v8393: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 56 KiB/s wr, 7 op/s 2023-03-09T19:04:26.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:04:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14776.174380 secs 2023-03-09T19:04:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:04:27 smithi122 ceph-mon[93869]: pgmap v8394: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 41 KiB/s wr, 7 op/s 2023-03-09T19:04:27.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:04:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14778.461631 secs 2023-03-09T19:04:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:04:27 smithi181 ceph-mon[94189]: pgmap v8394: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 41 KiB/s wr, 7 op/s 2023-03-09T19:04:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:04:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14778.461631 secs 2023-03-09T19:04:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:04:27 smithi038 ceph-mon[92422]: pgmap v8394: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 41 KiB/s wr, 7 op/s 2023-03-09T19:04:28.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:04:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14778.461631 secs 2023-03-09T19:04:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15240.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:04:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:04:29 smithi122 ceph-mon[93869]: pgmap v8395: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 55 KiB/s wr, 7 op/s 2023-03-09T19:04:29.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:04:29 smithi181 ceph-mon[94189]: pgmap v8395: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 55 KiB/s wr, 7 op/s 2023-03-09T19:04:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:04:29 smithi038 ceph-mon[92422]: pgmap v8395: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 55 KiB/s wr, 7 op/s 2023-03-09T19:04:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:04:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14781.174475 secs 2023-03-09T19:04:30.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:04:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14781.174475 secs 2023-03-09T19:04:30.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:04:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14781.174475 secs 2023-03-09T19:04:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:04:31 smithi122 ceph-mon[93869]: pgmap v8396: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 6 op/s 2023-03-09T19:04:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:04:31 smithi181 ceph-mon[94189]: pgmap v8396: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 6 op/s 2023-03-09T19:04:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:04:31 smithi038 ceph-mon[92422]: pgmap v8396: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 6 op/s 2023-03-09T19:04:32.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:04:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14783.461703 secs 2023-03-09T19:04:32.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:04:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14783.461703 secs 2023-03-09T19:04:32.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:04:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14783.461703 secs 2023-03-09T19:04:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15245.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:04:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:04:33 smithi122 ceph-mon[93869]: pgmap v8397: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 96 KiB/s wr, 9 op/s 2023-03-09T19:04:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:04:33 smithi181 ceph-mon[94189]: pgmap v8397: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 96 KiB/s wr, 9 op/s 2023-03-09T19:04:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:04:33 smithi038 ceph-mon[92422]: pgmap v8397: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 96 KiB/s wr, 9 op/s 2023-03-09T19:04:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:04:35 smithi122 ceph-mon[93869]: pgmap v8398: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 96 KiB/s wr, 7 op/s 2023-03-09T19:04:35.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:04:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14786.174559 secs 2023-03-09T19:04:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:04:35 smithi181 ceph-mon[94189]: pgmap v8398: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 96 KiB/s wr, 7 op/s 2023-03-09T19:04:35.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:04:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14786.174559 secs 2023-03-09T19:04:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:04:35 smithi038 ceph-mon[92422]: pgmap v8398: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 96 KiB/s wr, 7 op/s 2023-03-09T19:04:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:04:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14786.174559 secs 2023-03-09T19:04:37.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:04:37 smithi122 ceph-mon[93869]: pgmap v8399: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 96 KiB/s wr, 9 op/s 2023-03-09T19:04:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:04:37 smithi181 ceph-mon[94189]: pgmap v8399: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 96 KiB/s wr, 9 op/s 2023-03-09T19:04:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:04:37 smithi038 ceph-mon[92422]: pgmap v8399: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 96 KiB/s wr, 9 op/s 2023-03-09T19:04:38.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15250.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:04:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:04:39 smithi038 ceph-mon[92422]: pgmap v8400: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 147 KiB/s wr, 7 op/s 2023-03-09T19:04:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:04:39 smithi122 ceph-mon[93869]: pgmap v8400: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 147 KiB/s wr, 7 op/s 2023-03-09T19:04:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:04:39 smithi181 ceph-mon[94189]: pgmap v8400: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 147 KiB/s wr, 7 op/s 2023-03-09T19:04:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:04:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14791.174619 secs 2023-03-09T19:04:41.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:04:40 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:04:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:04:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14791.174619 secs 2023-03-09T19:04:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:04:40 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:04:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:04:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14791.174619 secs 2023-03-09T19:04:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:04:40 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:04:41.741 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:04:41 smithi181 ceph-mon[94189]: pgmap v8401: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 98 KiB/s wr, 6 op/s 2023-03-09T19:04:41.741 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:04:41 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:04:41.742 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:04:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:04:41.742 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:04:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:04:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:04:41 smithi038 ceph-mon[92422]: pgmap v8401: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 98 KiB/s wr, 6 op/s 2023-03-09T19:04:42.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:04:41 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:04:42.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:04:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:04:42.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:04:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:04:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:04:41 smithi122 ceph-mon[93869]: pgmap v8401: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 98 KiB/s wr, 6 op/s 2023-03-09T19:04:42.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:04:41 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:04:42.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:04:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:04:42.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:04:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:04:42.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:04:42 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 14793.461941 secs 2023-03-09T19:04:43.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:04:42 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 14793.461941 secs 2023-03-09T19:04:43.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:04:42 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 14793.461941 secs 2023-03-09T19:04:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15255.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:04:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:04:43 smithi038 ceph-mon[92422]: pgmap v8402: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 236 KiB/s wr, 11 op/s 2023-03-09T19:04:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:04:43 smithi122 ceph-mon[93869]: pgmap v8402: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 236 KiB/s wr, 11 op/s 2023-03-09T19:04:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:04:43 smithi181 ceph-mon[94189]: pgmap v8402: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 236 KiB/s wr, 11 op/s 2023-03-09T19:04:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:04:45 smithi122 ceph-mon[93869]: pgmap v8403: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 190 KiB/s wr, 9 op/s 2023-03-09T19:04:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:04:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14796.174704 secs 2023-03-09T19:04:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:04:45 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:04:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:04:45 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:04:46.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:04:45 smithi181 ceph-mon[94189]: pgmap v8403: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 190 KiB/s wr, 9 op/s 2023-03-09T19:04:46.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:04:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14796.174704 secs 2023-03-09T19:04:46.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:04:45 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:04:46.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:04:45 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:04:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:04:45 smithi038 ceph-mon[92422]: pgmap v8403: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 190 KiB/s wr, 9 op/s 2023-03-09T19:04:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:04:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14796.174704 secs 2023-03-09T19:04:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:04:45 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:04:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:04:45 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:04:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:04:47 smithi122 ceph-mon[93869]: pgmap v8404: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 291 KiB/s wr, 12 op/s 2023-03-09T19:04:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:04:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14798.462077 secs 2023-03-09T19:04:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:04:47 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:04:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:04:47 smithi181 ceph-mon[94189]: pgmap v8404: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 291 KiB/s wr, 12 op/s 2023-03-09T19:04:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:04:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14798.462077 secs 2023-03-09T19:04:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:04:47 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:04:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:04:47 smithi038 ceph-mon[92422]: pgmap v8404: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 291 KiB/s wr, 12 op/s 2023-03-09T19:04:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:04:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14798.462077 secs 2023-03-09T19:04:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:04:47 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:04:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15260.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:04:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:04:49 smithi122 ceph-mon[93869]: pgmap v8405: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 291 KiB/s wr, 11 op/s 2023-03-09T19:04:50.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:04:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:04:50.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:04:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:04:50.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:04:49 smithi181 ceph-mon[94189]: pgmap v8405: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 291 KiB/s wr, 11 op/s 2023-03-09T19:04:50.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:04:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:04:50.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:04:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:04:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:04:49 smithi038 ceph-mon[92422]: pgmap v8405: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 291 KiB/s wr, 11 op/s 2023-03-09T19:04:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:04:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:04:50.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:04:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:04:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:04:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14801.174764 secs 2023-03-09T19:04:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:04:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14801.174764 secs 2023-03-09T19:04:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:04:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14801.174764 secs 2023-03-09T19:04:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:04:51 smithi122 ceph-mon[93869]: pgmap v8406: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 240 KiB/s wr, 10 op/s 2023-03-09T19:04:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:04:51 smithi181 ceph-mon[94189]: pgmap v8406: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 240 KiB/s wr, 10 op/s 2023-03-09T19:04:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:04:51 smithi038 ceph-mon[92422]: pgmap v8406: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 240 KiB/s wr, 10 op/s 2023-03-09T19:04:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:04:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14803.462192 secs 2023-03-09T19:04:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:04:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14803.462192 secs 2023-03-09T19:04:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:04:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14803.462192 secs 2023-03-09T19:04:53.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15265.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:04:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:04:53 smithi122 ceph-mon[93869]: pgmap v8407: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 381 KiB/s wr, 15 op/s 2023-03-09T19:04:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:04:53 smithi181 ceph-mon[94189]: pgmap v8407: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 381 KiB/s wr, 15 op/s 2023-03-09T19:04:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:04:53 smithi038 ceph-mon[92422]: pgmap v8407: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 381 KiB/s wr, 15 op/s 2023-03-09T19:04:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:04:55 smithi122 ceph-mon[93869]: pgmap v8408: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 243 KiB/s wr, 10 op/s 2023-03-09T19:04:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:04:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14806.174823 secs 2023-03-09T19:04:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:04:55 smithi181 ceph-mon[94189]: pgmap v8408: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 243 KiB/s wr, 10 op/s 2023-03-09T19:04:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:04:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14806.174823 secs 2023-03-09T19:04:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:04:55 smithi038 ceph-mon[92422]: pgmap v8408: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 243 KiB/s wr, 10 op/s 2023-03-09T19:04:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:04:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14806.174823 secs 2023-03-09T19:04:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:04:57 smithi122 ceph-mon[93869]: pgmap v8409: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 283 KiB/s rd, 280 KiB/s wr, 12 op/s 2023-03-09T19:04:58.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:04:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14808.462306 secs 2023-03-09T19:04:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:04:57 smithi181 ceph-mon[94189]: pgmap v8409: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 283 KiB/s rd, 280 KiB/s wr, 12 op/s 2023-03-09T19:04:58.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:04:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14808.462306 secs 2023-03-09T19:04:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:04:57 smithi038 ceph-mon[92422]: pgmap v8409: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 283 KiB/s rd, 280 KiB/s wr, 12 op/s 2023-03-09T19:04:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:04:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14808.462306 secs 2023-03-09T19:04:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15270.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:05:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:04:59 smithi122 ceph-mon[93869]: pgmap v8410: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 179 KiB/s wr, 9 op/s 2023-03-09T19:05:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:04:59 smithi181 ceph-mon[94189]: pgmap v8410: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 179 KiB/s wr, 9 op/s 2023-03-09T19:05:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:04:59 smithi038 ceph-mon[92422]: pgmap v8410: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 179 KiB/s wr, 9 op/s 2023-03-09T19:05:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:05:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14811.174919 secs 2023-03-09T19:05:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:05:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14811.174919 secs 2023-03-09T19:05:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:05:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14811.174919 secs 2023-03-09T19:05:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:05:01 smithi122 ceph-mon[93869]: pgmap v8411: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 278 KiB/s wr, 9 op/s 2023-03-09T19:05:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:05:01 smithi181 ceph-mon[94189]: pgmap v8411: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 278 KiB/s wr, 9 op/s 2023-03-09T19:05:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:05:01 smithi038 ceph-mon[92422]: pgmap v8411: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 278 KiB/s wr, 9 op/s 2023-03-09T19:05:03.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:05:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14813.462413 secs 2023-03-09T19:05:03.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:05:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14813.462413 secs 2023-03-09T19:05:03.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:05:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14813.462413 secs 2023-03-09T19:05:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15275.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:05:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:05:03 smithi122 ceph-mon[93869]: pgmap v8412: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 301 KiB/s wr, 14 op/s 2023-03-09T19:05:04.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:05:03 smithi181 ceph-mon[94189]: pgmap v8412: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 301 KiB/s wr, 14 op/s 2023-03-09T19:05:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:05:03 smithi038 ceph-mon[92422]: pgmap v8412: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 301 KiB/s wr, 14 op/s 2023-03-09T19:05:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:05:05 smithi122 ceph-mon[93869]: pgmap v8413: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 221 KiB/s wr, 22 op/s 2023-03-09T19:05:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:05:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14816.174983 secs 2023-03-09T19:05:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:05:05 smithi181 ceph-mon[94189]: pgmap v8413: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 221 KiB/s wr, 22 op/s 2023-03-09T19:05:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:05:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14816.174983 secs 2023-03-09T19:05:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:05:05 smithi038 ceph-mon[92422]: pgmap v8413: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 221 KiB/s wr, 22 op/s 2023-03-09T19:05:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:05:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14816.174983 secs 2023-03-09T19:05:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:05:07 smithi122 ceph-mon[93869]: pgmap v8414: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 233 KiB/s wr, 26 op/s 2023-03-09T19:05:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:05:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14818.462553 secs 2023-03-09T19:05:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:05:07 smithi181 ceph-mon[94189]: pgmap v8414: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 233 KiB/s wr, 26 op/s 2023-03-09T19:05:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:05:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14818.462553 secs 2023-03-09T19:05:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:05:07 smithi038 ceph-mon[92422]: pgmap v8414: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 233 KiB/s wr, 26 op/s 2023-03-09T19:05:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:05:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14818.462553 secs 2023-03-09T19:05:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15280.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:05:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:05:09 smithi122 ceph-mon[93869]: pgmap v8415: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 196 KiB/s wr, 23 op/s 2023-03-09T19:05:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:05:09 smithi181 ceph-mon[94189]: pgmap v8415: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 196 KiB/s wr, 23 op/s 2023-03-09T19:05:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:05:09 smithi038 ceph-mon[92422]: pgmap v8415: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 196 KiB/s wr, 23 op/s 2023-03-09T19:05:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:05:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14821.175042 secs 2023-03-09T19:05:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:05:10 smithi122 ceph-mon[93869]: osdmap e739: 12 total, 12 up, 12 in 2023-03-09T19:05:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:05:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14821.175042 secs 2023-03-09T19:05:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:05:10 smithi181 ceph-mon[94189]: osdmap e739: 12 total, 12 up, 12 in 2023-03-09T19:05:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:05:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14821.175042 secs 2023-03-09T19:05:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:05:10 smithi038 ceph-mon[92422]: osdmap e739: 12 total, 12 up, 12 in 2023-03-09T19:05:12.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:05:11 smithi122 ceph-mon[93869]: pgmap v8417: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 139 KiB/s wr, 27 op/s 2023-03-09T19:05:12.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:05:11 smithi181 ceph-mon[94189]: pgmap v8417: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 139 KiB/s wr, 27 op/s 2023-03-09T19:05:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:05:11 smithi038 ceph-mon[92422]: pgmap v8417: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 139 KiB/s wr, 27 op/s 2023-03-09T19:05:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:05:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14823.462681 secs 2023-03-09T19:05:13.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:05:12 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:05:13.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:05:12 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:05:13.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:05:12 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:05:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:05:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14823.462681 secs 2023-03-09T19:05:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:05:12 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:05:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:05:12 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:05:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:05:12 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:05:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:05:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14823.462681 secs 2023-03-09T19:05:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:05:12 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:05:13.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:05:12 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:05:13.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:05:12 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:05:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15285.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:05:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:05:13 smithi122 ceph-mon[93869]: pgmap v8418: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 126 KiB/s wr, 25 op/s 2023-03-09T19:05:14.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:05:13 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:05:14.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:05:13 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:05:14.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:05:13 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:05:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:05:13 smithi181 ceph-mon[94189]: pgmap v8418: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 126 KiB/s wr, 25 op/s 2023-03-09T19:05:14.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:05:13 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:05:14.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:05:13 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:05:14.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:05:13 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:05:14.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:05:13 smithi038 ceph-mon[92422]: pgmap v8418: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 126 KiB/s wr, 25 op/s 2023-03-09T19:05:14.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:05:13 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:05:14.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:05:13 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:05:14.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:05:13 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:05:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:05:15 smithi122 ceph-mon[93869]: pgmap v8419: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 84 KiB/s wr, 9 op/s 2023-03-09T19:05:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:05:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14826.175101 secs 2023-03-09T19:05:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:05:15 smithi181 ceph-mon[94189]: pgmap v8419: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 84 KiB/s wr, 9 op/s 2023-03-09T19:05:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:05:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14826.175101 secs 2023-03-09T19:05:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:05:15 smithi038 ceph-mon[92422]: pgmap v8419: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 84 KiB/s wr, 9 op/s 2023-03-09T19:05:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:05:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14826.175101 secs 2023-03-09T19:05:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:05:17 smithi122 ceph-mon[93869]: pgmap v8420: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 86 KiB/s wr, 9 op/s 2023-03-09T19:05:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:05:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14828.462782 secs 2023-03-09T19:05:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:05:17 smithi122 ceph-mon[93869]: osdmap e740: 12 total, 12 up, 12 in 2023-03-09T19:05:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:05:17 smithi181 ceph-mon[94189]: pgmap v8420: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 86 KiB/s wr, 9 op/s 2023-03-09T19:05:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:05:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14828.462782 secs 2023-03-09T19:05:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:05:17 smithi181 ceph-mon[94189]: osdmap e740: 12 total, 12 up, 12 in 2023-03-09T19:05:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:05:17 smithi038 ceph-mon[92422]: pgmap v8420: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 86 KiB/s wr, 9 op/s 2023-03-09T19:05:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:05:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14828.462782 secs 2023-03-09T19:05:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:05:17 smithi038 ceph-mon[92422]: osdmap e740: 12 total, 12 up, 12 in 2023-03-09T19:05:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15290.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:05:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:05:19 smithi122 ceph-mon[93869]: pgmap v8422: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 73 KiB/s wr, 9 op/s 2023-03-09T19:05:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:05:19 smithi181 ceph-mon[94189]: pgmap v8422: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 73 KiB/s wr, 9 op/s 2023-03-09T19:05:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:05:19 smithi038 ceph-mon[92422]: pgmap v8422: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 73 KiB/s wr, 9 op/s 2023-03-09T19:05:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:05:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14831.175160 secs 2023-03-09T19:05:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:05:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14831.175160 secs 2023-03-09T19:05:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:05:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14831.175160 secs 2023-03-09T19:05:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:05:21 smithi122 ceph-mon[93869]: pgmap v8423: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 63 KiB/s wr, 9 op/s 2023-03-09T19:05:22.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:05:21 smithi181 ceph-mon[94189]: pgmap v8423: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 63 KiB/s wr, 9 op/s 2023-03-09T19:05:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:05:21 smithi038 ceph-mon[92422]: pgmap v8423: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 63 KiB/s wr, 9 op/s 2023-03-09T19:05:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:05:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14833.462924 secs 2023-03-09T19:05:23.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:05:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14833.462924 secs 2023-03-09T19:05:23.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:05:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14833.462924 secs 2023-03-09T19:05:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15295.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:05:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:05:23 smithi122 ceph-mon[93869]: pgmap v8424: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 49 KiB/s wr, 8 op/s 2023-03-09T19:05:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:05:23 smithi181 ceph-mon[94189]: pgmap v8424: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 49 KiB/s wr, 8 op/s 2023-03-09T19:05:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:05:23 smithi038 ceph-mon[92422]: pgmap v8424: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 49 KiB/s wr, 8 op/s 2023-03-09T19:05:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:05:25 smithi122 ceph-mon[93869]: pgmap v8425: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 76 KiB/s wr, 8 op/s 2023-03-09T19:05:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:05:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14836.175221 secs 2023-03-09T19:05:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:05:25 smithi181 ceph-mon[94189]: pgmap v8425: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 76 KiB/s wr, 8 op/s 2023-03-09T19:05:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:05:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14836.175221 secs 2023-03-09T19:05:26.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:05:25 smithi038 ceph-mon[92422]: pgmap v8425: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 76 KiB/s wr, 8 op/s 2023-03-09T19:05:26.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:05:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14836.175221 secs 2023-03-09T19:05:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:05:27 smithi122 ceph-mon[93869]: pgmap v8426: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 60 KiB/s wr, 7 op/s 2023-03-09T19:05:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:05:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14838.463047 secs 2023-03-09T19:05:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:05:27 smithi181 ceph-mon[94189]: pgmap v8426: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 60 KiB/s wr, 7 op/s 2023-03-09T19:05:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:05:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14838.463047 secs 2023-03-09T19:05:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:05:27 smithi038 ceph-mon[92422]: pgmap v8426: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 60 KiB/s wr, 7 op/s 2023-03-09T19:05:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:05:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14838.463047 secs 2023-03-09T19:05:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15300.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:05:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:05:29 smithi122 ceph-mon[93869]: pgmap v8427: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 54 KiB/s wr, 7 op/s 2023-03-09T19:05:30.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:05:29 smithi181 ceph-mon[94189]: pgmap v8427: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 54 KiB/s wr, 7 op/s 2023-03-09T19:05:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:05:29 smithi038 ceph-mon[92422]: pgmap v8427: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 54 KiB/s wr, 7 op/s 2023-03-09T19:05:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:05:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14841.175283 secs 2023-03-09T19:05:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:05:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14841.175283 secs 2023-03-09T19:05:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:05:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14841.175283 secs 2023-03-09T19:05:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:05:31 smithi122 ceph-mon[93869]: pgmap v8428: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 50 KiB/s wr, 6 op/s 2023-03-09T19:05:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:05:31 smithi181 ceph-mon[94189]: pgmap v8428: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 50 KiB/s wr, 6 op/s 2023-03-09T19:05:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:05:31 smithi038 ceph-mon[92422]: pgmap v8428: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 50 KiB/s wr, 6 op/s 2023-03-09T19:05:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:05:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14843.463193 secs 2023-03-09T19:05:33.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:05:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14843.463193 secs 2023-03-09T19:05:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:05:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14843.463193 secs 2023-03-09T19:05:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15305.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:05:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:05:33 smithi122 ceph-mon[93869]: pgmap v8429: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 50 KiB/s wr, 8 op/s 2023-03-09T19:05:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:05:33 smithi181 ceph-mon[94189]: pgmap v8429: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 50 KiB/s wr, 8 op/s 2023-03-09T19:05:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:05:33 smithi038 ceph-mon[92422]: pgmap v8429: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 50 KiB/s wr, 8 op/s 2023-03-09T19:05:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:05:35 smithi122 ceph-mon[93869]: pgmap v8430: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 102 KiB/s wr, 6 op/s 2023-03-09T19:05:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:05:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14846.175354 secs 2023-03-09T19:05:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:05:35 smithi181 ceph-mon[94189]: pgmap v8430: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 102 KiB/s wr, 6 op/s 2023-03-09T19:05:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:05:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14846.175354 secs 2023-03-09T19:05:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:05:35 smithi038 ceph-mon[92422]: pgmap v8430: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 102 KiB/s wr, 6 op/s 2023-03-09T19:05:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:05:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14846.175354 secs 2023-03-09T19:05:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:05:37 smithi122 ceph-mon[93869]: pgmap v8431: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 52 KiB/s wr, 8 op/s 2023-03-09T19:05:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:05:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14848.463302 secs 2023-03-09T19:05:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:05:37 smithi181 ceph-mon[94189]: pgmap v8431: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 52 KiB/s wr, 8 op/s 2023-03-09T19:05:38.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:05:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14848.463302 secs 2023-03-09T19:05:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:05:37 smithi038 ceph-mon[92422]: pgmap v8431: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 52 KiB/s wr, 8 op/s 2023-03-09T19:05:38.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:05:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14848.463302 secs 2023-03-09T19:05:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15310.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:05:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:05:39 smithi122 ceph-mon[93869]: pgmap v8432: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 52 KiB/s wr, 6 op/s 2023-03-09T19:05:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:05:39 smithi181 ceph-mon[94189]: pgmap v8432: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 52 KiB/s wr, 6 op/s 2023-03-09T19:05:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:05:39 smithi038 ceph-mon[92422]: pgmap v8432: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 52 KiB/s wr, 6 op/s 2023-03-09T19:05:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:05:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14851.175415 secs 2023-03-09T19:05:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:05:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14851.175415 secs 2023-03-09T19:05:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:05:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14851.175415 secs 2023-03-09T19:05:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:05:41 smithi181 ceph-mon[94189]: pgmap v8433: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 108 KiB/s wr, 6 op/s 2023-03-09T19:05:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:05:41 smithi038 ceph-mon[92422]: pgmap v8433: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 108 KiB/s wr, 6 op/s 2023-03-09T19:05:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:05:41 smithi122 ceph-mon[93869]: pgmap v8433: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 108 KiB/s wr, 6 op/s 2023-03-09T19:05:43.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:05:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14853.463407 secs 2023-03-09T19:05:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:05:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14853.463407 secs 2023-03-09T19:05:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:05:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14853.463407 secs 2023-03-09T19:05:43.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15315.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:05:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:05:43 smithi038 ceph-mon[92422]: pgmap v8434: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 120 KiB/s wr, 8 op/s 2023-03-09T19:05:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:05:43 smithi122 ceph-mon[93869]: pgmap v8434: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 120 KiB/s wr, 8 op/s 2023-03-09T19:05:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:05:43 smithi181 ceph-mon[94189]: pgmap v8434: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 120 KiB/s wr, 8 op/s 2023-03-09T19:05:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:05:45 smithi038 ceph-mon[92422]: pgmap v8435: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 120 KiB/s wr, 7 op/s 2023-03-09T19:05:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:05:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14856.175495 secs 2023-03-09T19:05:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:05:45 smithi122 ceph-mon[93869]: pgmap v8435: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 120 KiB/s wr, 7 op/s 2023-03-09T19:05:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:05:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14856.175495 secs 2023-03-09T19:05:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:05:45 smithi181 ceph-mon[94189]: pgmap v8435: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 120 KiB/s wr, 7 op/s 2023-03-09T19:05:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:05:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14856.175495 secs 2023-03-09T19:05:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:05:48 smithi122 ceph-mon[93869]: pgmap v8436: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 135 KiB/s wr, 10 op/s 2023-03-09T19:05:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:05:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14858.463474 secs 2023-03-09T19:05:48.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:05:48 smithi181 ceph-mon[94189]: pgmap v8436: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 135 KiB/s wr, 10 op/s 2023-03-09T19:05:48.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:05:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14858.463474 secs 2023-03-09T19:05:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:05:48 smithi038 ceph-mon[92422]: pgmap v8436: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 135 KiB/s wr, 10 op/s 2023-03-09T19:05:48.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:05:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14858.463474 secs 2023-03-09T19:05:48.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15320.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:05:50.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:05:50 smithi122 ceph-mon[93869]: pgmap v8437: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 135 KiB/s wr, 8 op/s 2023-03-09T19:05:50.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:05:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14861.175558 secs 2023-03-09T19:05:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:05:50 smithi181 ceph-mon[94189]: pgmap v8437: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 135 KiB/s wr, 8 op/s 2023-03-09T19:05:50.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:05:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14861.175558 secs 2023-03-09T19:05:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:05:50 smithi038 ceph-mon[92422]: pgmap v8437: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 135 KiB/s wr, 8 op/s 2023-03-09T19:05:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:05:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14861.175558 secs 2023-03-09T19:05:52.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:05:52 smithi122 ceph-mon[93869]: pgmap v8438: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 355 KiB/s rd, 271 KiB/s wr, 10 op/s 2023-03-09T19:05:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:05:52 smithi181 ceph-mon[94189]: pgmap v8438: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 355 KiB/s rd, 271 KiB/s wr, 10 op/s 2023-03-09T19:05:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:05:52 smithi038 ceph-mon[92422]: pgmap v8438: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 355 KiB/s rd, 271 KiB/s wr, 10 op/s 2023-03-09T19:05:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:05:53 smithi122 ceph-mon[93869]: pgmap v8439: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 318 KiB/s rd, 229 KiB/s wr, 12 op/s 2023-03-09T19:05:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:05:53 smithi181 ceph-mon[94189]: pgmap v8439: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 318 KiB/s rd, 229 KiB/s wr, 12 op/s 2023-03-09T19:05:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:05:53 smithi038 ceph-mon[92422]: pgmap v8439: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 318 KiB/s rd, 229 KiB/s wr, 12 op/s 2023-03-09T19:05:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15325.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:05:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:05:55 smithi122 ceph-mon[93869]: pgmap v8440: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 216 KiB/s wr, 10 op/s 2023-03-09T19:05:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:05:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14866.175640 secs 2023-03-09T19:05:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:05:55 smithi181 ceph-mon[94189]: pgmap v8440: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 216 KiB/s wr, 10 op/s 2023-03-09T19:05:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:05:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14866.175640 secs 2023-03-09T19:05:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:05:55 smithi038 ceph-mon[92422]: pgmap v8440: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 216 KiB/s wr, 10 op/s 2023-03-09T19:05:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:05:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14866.175640 secs 2023-03-09T19:05:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:05:56 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:05:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:05:56 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:05:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:05:56 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:05:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:05:57 smithi122 ceph-mon[93869]: pgmap v8441: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 235 KiB/s wr, 12 op/s 2023-03-09T19:05:57.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:05:57 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:05:57.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:05:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:05:57.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:05:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:05:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:05:57 smithi181 ceph-mon[94189]: pgmap v8441: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 235 KiB/s wr, 12 op/s 2023-03-09T19:05:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:05:57 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:05:57.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:05:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:05:57.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:05:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:05:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:05:57 smithi038 ceph-mon[92422]: pgmap v8441: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 235 KiB/s wr, 12 op/s 2023-03-09T19:05:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:05:57 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:05:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:05:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:05:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:05:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:05:58.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15330.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:05:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:05:59 smithi122 ceph-mon[93869]: pgmap v8442: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 169 KiB/s wr, 9 op/s 2023-03-09T19:05:59.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:05:59 smithi181 ceph-mon[94189]: pgmap v8442: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 169 KiB/s wr, 9 op/s 2023-03-09T19:05:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:05:59 smithi038 ceph-mon[92422]: pgmap v8442: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 169 KiB/s wr, 9 op/s 2023-03-09T19:06:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:06:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14871.175699 secs 2023-03-09T19:06:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:06:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14871.175699 secs 2023-03-09T19:06:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:06:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14871.175699 secs 2023-03-09T19:06:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:06:01 smithi122 ceph-mon[93869]: pgmap v8443: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 301 KiB/s wr, 10 op/s 2023-03-09T19:06:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:06:01 smithi181 ceph-mon[94189]: pgmap v8443: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 301 KiB/s wr, 10 op/s 2023-03-09T19:06:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:06:01 smithi038 ceph-mon[92422]: pgmap v8443: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 301 KiB/s wr, 10 op/s 2023-03-09T19:06:02.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:06:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14873.463862 secs 2023-03-09T19:06:02.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:06:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14873.463862 secs 2023-03-09T19:06:02.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:06:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14873.463862 secs 2023-03-09T19:06:03.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15335.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:06:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:06:03 smithi122 ceph-mon[93869]: pgmap v8444: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 247 KiB/s rd, 245 KiB/s wr, 12 op/s 2023-03-09T19:06:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:06:03 smithi181 ceph-mon[94189]: pgmap v8444: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 247 KiB/s rd, 245 KiB/s wr, 12 op/s 2023-03-09T19:06:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:06:03 smithi038 ceph-mon[92422]: pgmap v8444: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 247 KiB/s rd, 245 KiB/s wr, 12 op/s 2023-03-09T19:06:04.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:06:04 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:06:04.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:06:04 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:06:04.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:06:04 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:06:04.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:06:04 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:06:04.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:06:04 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:06:04.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:06:04 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:06:04.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:06:04 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:06:04.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:06:04 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:06:05.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:06:04 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:06:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:06:05 smithi122 ceph-mon[93869]: pgmap v8445: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 254 KiB/s wr, 21 op/s 2023-03-09T19:06:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:06:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14876.175758 secs 2023-03-09T19:06:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:06:05 smithi181 ceph-mon[94189]: pgmap v8445: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 254 KiB/s wr, 21 op/s 2023-03-09T19:06:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:06:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14876.175758 secs 2023-03-09T19:06:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:06:05 smithi038 ceph-mon[92422]: pgmap v8445: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 254 KiB/s wr, 21 op/s 2023-03-09T19:06:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:06:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14876.175758 secs 2023-03-09T19:06:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:06:07 smithi122 ceph-mon[93869]: pgmap v8446: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 311 KiB/s wr, 27 op/s 2023-03-09T19:06:07.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:06:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14878.463917 secs 2023-03-09T19:06:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:06:07 smithi181 ceph-mon[94189]: pgmap v8446: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 311 KiB/s wr, 27 op/s 2023-03-09T19:06:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:06:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14878.463917 secs 2023-03-09T19:06:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:06:07 smithi038 ceph-mon[92422]: pgmap v8446: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 311 KiB/s wr, 27 op/s 2023-03-09T19:06:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:06:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14878.463917 secs 2023-03-09T19:06:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15340.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:06:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:06:09 smithi122 ceph-mon[93869]: pgmap v8447: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 292 KiB/s wr, 24 op/s 2023-03-09T19:06:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:06:09 smithi181 ceph-mon[94189]: pgmap v8447: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 292 KiB/s wr, 24 op/s 2023-03-09T19:06:10.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:06:09 smithi038 ceph-mon[92422]: pgmap v8447: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 292 KiB/s wr, 24 op/s 2023-03-09T19:06:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:06:10 smithi122 ceph-mon[93869]: osdmap e741: 12 total, 12 up, 12 in 2023-03-09T19:06:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:06:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14881.175820 secs 2023-03-09T19:06:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:06:10 smithi181 ceph-mon[94189]: osdmap e741: 12 total, 12 up, 12 in 2023-03-09T19:06:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:06:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14881.175820 secs 2023-03-09T19:06:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:06:10 smithi038 ceph-mon[92422]: osdmap e741: 12 total, 12 up, 12 in 2023-03-09T19:06:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:06:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14881.175820 secs 2023-03-09T19:06:11.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:06:11 smithi122 ceph-mon[93869]: pgmap v8449: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 191 KiB/s wr, 27 op/s 2023-03-09T19:06:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:06:11 smithi181 ceph-mon[94189]: pgmap v8449: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 191 KiB/s wr, 27 op/s 2023-03-09T19:06:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:06:11 smithi038 ceph-mon[92422]: pgmap v8449: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 191 KiB/s wr, 27 op/s 2023-03-09T19:06:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:06:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14883.464013 secs 2023-03-09T19:06:12.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:06:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14883.464013 secs 2023-03-09T19:06:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:06:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14883.464013 secs 2023-03-09T19:06:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15345.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:06:13.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:06:13 smithi122 ceph-mon[93869]: pgmap v8450: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 236 KiB/s wr, 27 op/s 2023-03-09T19:06:13.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:06:13 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:06:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:06:13 smithi181 ceph-mon[94189]: pgmap v8450: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 236 KiB/s wr, 27 op/s 2023-03-09T19:06:13.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:06:13 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:06:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:06:13 smithi038 ceph-mon[92422]: pgmap v8450: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 236 KiB/s wr, 27 op/s 2023-03-09T19:06:14.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:06:13 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:06:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:06:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:06:15.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:06:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:06:15.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:06:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:06:15.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:06:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:06:15.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:06:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:06:15.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:06:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:06:15.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:06:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:06:15.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:06:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14886.175886 secs 2023-03-09T19:06:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:06:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:06:15.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:06:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:06:15.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:06:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:06:15.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:06:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:06:15.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:06:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:06:15.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:06:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:06:15.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:06:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:06:15.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:06:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14886.175886 secs 2023-03-09T19:06:15.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:06:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:06:15.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:06:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:06:15.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:06:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:06:15.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:06:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:06:15.503 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:06:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:06:15.503 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:06:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:06:15.503 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:06:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:06:15.503 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:06:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14886.175886 secs 2023-03-09T19:06:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:06:16 smithi122 ceph-mon[93869]: pgmap v8451: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 211 KiB/s wr, 12 op/s 2023-03-09T19:06:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:06:16 smithi181 ceph-mon[94189]: pgmap v8451: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 211 KiB/s wr, 12 op/s 2023-03-09T19:06:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:06:16 smithi038 ceph-mon[92422]: pgmap v8451: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 211 KiB/s wr, 12 op/s 2023-03-09T19:06:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:06:18 smithi122 ceph-mon[93869]: pgmap v8452: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 184 KiB/s wr, 9 op/s 2023-03-09T19:06:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:06:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14888.464129 secs 2023-03-09T19:06:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:06:18 smithi122 ceph-mon[93869]: osdmap e742: 12 total, 12 up, 12 in 2023-03-09T19:06:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:06:18 smithi181 ceph-mon[94189]: pgmap v8452: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 184 KiB/s wr, 9 op/s 2023-03-09T19:06:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:06:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14888.464129 secs 2023-03-09T19:06:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:06:18 smithi181 ceph-mon[94189]: osdmap e742: 12 total, 12 up, 12 in 2023-03-09T19:06:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:06:18 smithi038 ceph-mon[92422]: pgmap v8452: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 184 KiB/s wr, 9 op/s 2023-03-09T19:06:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:06:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14888.464129 secs 2023-03-09T19:06:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:06:18 smithi038 ceph-mon[92422]: osdmap e742: 12 total, 12 up, 12 in 2023-03-09T19:06:18.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15350.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:06:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:06:20 smithi122 ceph-mon[93869]: pgmap v8454: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 208 KiB/s wr, 9 op/s 2023-03-09T19:06:20.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:06:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14891.175948 secs 2023-03-09T19:06:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:06:20 smithi181 ceph-mon[94189]: pgmap v8454: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 208 KiB/s wr, 9 op/s 2023-03-09T19:06:20.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:06:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14891.175948 secs 2023-03-09T19:06:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:06:20 smithi038 ceph-mon[92422]: pgmap v8454: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 208 KiB/s wr, 9 op/s 2023-03-09T19:06:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:06:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14891.175948 secs 2023-03-09T19:06:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:06:22 smithi122 ceph-mon[93869]: pgmap v8455: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 184 KiB/s wr, 8 op/s 2023-03-09T19:06:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:06:22 smithi181 ceph-mon[94189]: pgmap v8455: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 184 KiB/s wr, 8 op/s 2023-03-09T19:06:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:06:22 smithi038 ceph-mon[92422]: pgmap v8455: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 184 KiB/s wr, 8 op/s 2023-03-09T19:06:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:06:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14893.464246 secs 2023-03-09T19:06:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:06:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14893.464246 secs 2023-03-09T19:06:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:06:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14893.464246 secs 2023-03-09T19:06:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15355.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:06:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:06:24 smithi122 ceph-mon[93869]: pgmap v8456: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 113 KiB/s wr, 8 op/s 2023-03-09T19:06:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:06:24 smithi181 ceph-mon[94189]: pgmap v8456: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 113 KiB/s wr, 8 op/s 2023-03-09T19:06:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:06:24 smithi038 ceph-mon[92422]: pgmap v8456: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 113 KiB/s wr, 8 op/s 2023-03-09T19:06:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:06:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14896.175980 secs 2023-03-09T19:06:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:06:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14896.175980 secs 2023-03-09T19:06:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:06:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14896.175980 secs 2023-03-09T19:06:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:06:26 smithi122 ceph-mon[93869]: pgmap v8457: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 113 KiB/s wr, 8 op/s 2023-03-09T19:06:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:06:26 smithi181 ceph-mon[94189]: pgmap v8457: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 113 KiB/s wr, 8 op/s 2023-03-09T19:06:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:06:26 smithi038 ceph-mon[92422]: pgmap v8457: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 113 KiB/s wr, 8 op/s 2023-03-09T19:06:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:06:28 smithi122 ceph-mon[93869]: pgmap v8458: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 130 KiB/s wr, 7 op/s 2023-03-09T19:06:28.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:06:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14898.464364 secs 2023-03-09T19:06:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:06:28 smithi181 ceph-mon[94189]: pgmap v8458: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 130 KiB/s wr, 7 op/s 2023-03-09T19:06:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:06:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14898.464364 secs 2023-03-09T19:06:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:06:28 smithi038 ceph-mon[92422]: pgmap v8458: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 130 KiB/s wr, 7 op/s 2023-03-09T19:06:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:06:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14898.464364 secs 2023-03-09T19:06:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15360.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:06:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:06:30 smithi122 ceph-mon[93869]: pgmap v8459: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 118 KiB/s wr, 7 op/s 2023-03-09T19:06:30.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:06:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14901.176041 secs 2023-03-09T19:06:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:06:30 smithi181 ceph-mon[94189]: pgmap v8459: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 118 KiB/s wr, 7 op/s 2023-03-09T19:06:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:06:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14901.176041 secs 2023-03-09T19:06:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:06:30 smithi038 ceph-mon[92422]: pgmap v8459: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 118 KiB/s wr, 7 op/s 2023-03-09T19:06:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:06:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14901.176041 secs 2023-03-09T19:06:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:06:32 smithi122 ceph-mon[93869]: pgmap v8460: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 108 KiB/s wr, 6 op/s 2023-03-09T19:06:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:06:32 smithi181 ceph-mon[94189]: pgmap v8460: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 108 KiB/s wr, 6 op/s 2023-03-09T19:06:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:06:32 smithi038 ceph-mon[92422]: pgmap v8460: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 108 KiB/s wr, 6 op/s 2023-03-09T19:06:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:06:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14903.464470 secs 2023-03-09T19:06:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:06:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14903.464470 secs 2023-03-09T19:06:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:06:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14903.464470 secs 2023-03-09T19:06:33.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15365.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:06:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:06:34 smithi122 ceph-mon[93869]: pgmap v8461: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 109 KiB/s wr, 9 op/s 2023-03-09T19:06:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:06:34 smithi181 ceph-mon[94189]: pgmap v8461: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 109 KiB/s wr, 9 op/s 2023-03-09T19:06:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:06:34 smithi038 ceph-mon[92422]: pgmap v8461: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 109 KiB/s wr, 9 op/s 2023-03-09T19:06:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:06:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14906.176142 secs 2023-03-09T19:06:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:06:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14906.176142 secs 2023-03-09T19:06:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:06:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14906.176142 secs 2023-03-09T19:06:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:06:36 smithi122 ceph-mon[93869]: pgmap v8462: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 6 op/s 2023-03-09T19:06:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:06:36 smithi181 ceph-mon[94189]: pgmap v8462: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 6 op/s 2023-03-09T19:06:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:06:36 smithi038 ceph-mon[92422]: pgmap v8462: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 6 op/s 2023-03-09T19:06:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:06:38 smithi122 ceph-mon[93869]: pgmap v8463: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 213 KiB/s wr, 10 op/s 2023-03-09T19:06:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:06:38 smithi181 ceph-mon[94189]: pgmap v8463: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 213 KiB/s wr, 10 op/s 2023-03-09T19:06:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:06:38 smithi038 ceph-mon[92422]: pgmap v8463: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 213 KiB/s wr, 10 op/s 2023-03-09T19:06:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15370.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:06:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:06:40 smithi122 ceph-mon[93869]: pgmap v8464: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 164 KiB/s wr, 7 op/s 2023-03-09T19:06:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:06:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14911.176205 secs 2023-03-09T19:06:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:06:40 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:06:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:06:40 smithi181 ceph-mon[94189]: pgmap v8464: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 164 KiB/s wr, 7 op/s 2023-03-09T19:06:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:06:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14911.176205 secs 2023-03-09T19:06:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:06:40 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:06:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:06:40 smithi038 ceph-mon[92422]: pgmap v8464: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 164 KiB/s wr, 7 op/s 2023-03-09T19:06:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:06:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14911.176205 secs 2023-03-09T19:06:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:06:40 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:06:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:06:41 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:06:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:06:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:06:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:06:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:06:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:06:41 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:06:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:06:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:06:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:06:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:06:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:06:41 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:06:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:06:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:06:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:06:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:06:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:06:42 smithi122 ceph-mon[93869]: pgmap v8465: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 164 KiB/s wr, 7 op/s 2023-03-09T19:06:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:06:42 smithi181 ceph-mon[94189]: pgmap v8465: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 164 KiB/s wr, 7 op/s 2023-03-09T19:06:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:06:42 smithi038 ceph-mon[92422]: pgmap v8465: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 164 KiB/s wr, 7 op/s 2023-03-09T19:06:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:06:43 smithi038 ceph-mon[92422]: pgmap v8466: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 229 KiB/s wr, 11 op/s 2023-03-09T19:06:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15375.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:06:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:06:43 smithi122 ceph-mon[93869]: pgmap v8466: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 229 KiB/s wr, 11 op/s 2023-03-09T19:06:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:06:43 smithi181 ceph-mon[94189]: pgmap v8466: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 229 KiB/s wr, 11 op/s 2023-03-09T19:06:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:06:45 smithi122 ceph-mon[93869]: pgmap v8467: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 228 KiB/s wr, 9 op/s 2023-03-09T19:06:45.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:06:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14916.176263 secs 2023-03-09T19:06:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:06:45 smithi181 ceph-mon[94189]: pgmap v8467: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 228 KiB/s wr, 9 op/s 2023-03-09T19:06:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:06:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14916.176263 secs 2023-03-09T19:06:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:06:45 smithi038 ceph-mon[92422]: pgmap v8467: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 228 KiB/s wr, 9 op/s 2023-03-09T19:06:46.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:06:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14916.176263 secs 2023-03-09T19:06:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:06:47 smithi122 ceph-mon[93869]: pgmap v8468: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 330 KiB/s wr, 13 op/s 2023-03-09T19:06:47.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:06:47 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 14918.464860 secs 2023-03-09T19:06:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:06:47 smithi181 ceph-mon[94189]: pgmap v8468: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 330 KiB/s wr, 13 op/s 2023-03-09T19:06:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:06:47 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 14918.464860 secs 2023-03-09T19:06:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:06:47 smithi038 ceph-mon[92422]: pgmap v8468: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 330 KiB/s wr, 13 op/s 2023-03-09T19:06:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:06:47 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 14918.464860 secs 2023-03-09T19:06:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15380.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:06:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:06:48 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:06:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:06:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:06:48.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:06:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:06:48.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:06:48 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:06:48.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:06:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:06:48.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:06:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:06:49.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:06:48 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:06:49.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:06:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:06:49.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:06:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:06:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:06:49 smithi122 ceph-mon[93869]: pgmap v8469: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 177 KiB/s wr, 10 op/s 2023-03-09T19:06:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:06:49 smithi181 ceph-mon[94189]: pgmap v8469: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 177 KiB/s wr, 10 op/s 2023-03-09T19:06:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:06:49 smithi038 ceph-mon[92422]: pgmap v8469: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 177 KiB/s wr, 10 op/s 2023-03-09T19:06:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:06:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14921.176331 secs 2023-03-09T19:06:50.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:06:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14921.176331 secs 2023-03-09T19:06:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:06:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14921.176331 secs 2023-03-09T19:06:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:06:51 smithi122 ceph-mon[93869]: pgmap v8470: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 177 KiB/s wr, 10 op/s 2023-03-09T19:06:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:06:51 smithi181 ceph-mon[94189]: pgmap v8470: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 177 KiB/s wr, 10 op/s 2023-03-09T19:06:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:06:51 smithi038 ceph-mon[92422]: pgmap v8470: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 177 KiB/s wr, 10 op/s 2023-03-09T19:06:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:06:52 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 14923.464971 secs 2023-03-09T19:06:52.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:06:52 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 14923.464971 secs 2023-03-09T19:06:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:06:52 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 14923.464971 secs 2023-03-09T19:06:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15385.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:06:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:06:53 smithi122 ceph-mon[93869]: pgmap v8471: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 238 KiB/s wr, 13 op/s 2023-03-09T19:06:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:06:53 smithi181 ceph-mon[94189]: pgmap v8471: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 238 KiB/s wr, 13 op/s 2023-03-09T19:06:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:06:53 smithi038 ceph-mon[92422]: pgmap v8471: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 238 KiB/s wr, 13 op/s 2023-03-09T19:06:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:06:55 smithi122 ceph-mon[93869]: pgmap v8472: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 315 KiB/s wr, 11 op/s 2023-03-09T19:06:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:06:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14926.176391 secs 2023-03-09T19:06:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:06:55 smithi181 ceph-mon[94189]: pgmap v8472: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 315 KiB/s wr, 11 op/s 2023-03-09T19:06:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:06:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14926.176391 secs 2023-03-09T19:06:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:06:55 smithi038 ceph-mon[92422]: pgmap v8472: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 315 KiB/s wr, 11 op/s 2023-03-09T19:06:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:06:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14926.176391 secs 2023-03-09T19:06:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:06:57 smithi122 ceph-mon[93869]: pgmap v8473: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 315 KiB/s wr, 12 op/s 2023-03-09T19:06:57.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:06:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14928.465100 secs 2023-03-09T19:06:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:06:57 smithi181 ceph-mon[94189]: pgmap v8473: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 315 KiB/s wr, 12 op/s 2023-03-09T19:06:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:06:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14928.465100 secs 2023-03-09T19:06:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:06:57 smithi038 ceph-mon[92422]: pgmap v8473: 129 pgs: 129 active+clean; 9.1 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 315 KiB/s wr, 12 op/s 2023-03-09T19:06:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:06:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14928.465100 secs 2023-03-09T19:06:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15390.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:06:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:06:59 smithi122 ceph-mon[93869]: pgmap v8474: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 284 KiB/s wr, 10 op/s 2023-03-09T19:06:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:06:59 smithi181 ceph-mon[94189]: pgmap v8474: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 284 KiB/s wr, 10 op/s 2023-03-09T19:06:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:06:59 smithi038 ceph-mon[92422]: pgmap v8474: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 284 KiB/s wr, 10 op/s 2023-03-09T19:07:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:07:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14931.176448 secs 2023-03-09T19:07:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:07:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14931.176448 secs 2023-03-09T19:07:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:07:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14931.176448 secs 2023-03-09T19:07:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:07:01 smithi122 ceph-mon[93869]: pgmap v8475: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 274 KiB/s wr, 9 op/s 2023-03-09T19:07:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:07:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:07:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:07:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:07:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:07:01 smithi181 ceph-mon[94189]: pgmap v8475: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 274 KiB/s wr, 9 op/s 2023-03-09T19:07:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:07:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:07:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:07:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:07:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:07:01 smithi038 ceph-mon[92422]: pgmap v8475: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 274 KiB/s wr, 9 op/s 2023-03-09T19:07:02.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:07:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:07:02.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:07:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:07:02.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:07:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14933.465190 secs 2023-03-09T19:07:02.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:07:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14933.465190 secs 2023-03-09T19:07:02.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:07:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14933.465190 secs 2023-03-09T19:07:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15395.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:07:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:07:03 smithi122 ceph-mon[93869]: pgmap v8476: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 274 KiB/s wr, 12 op/s 2023-03-09T19:07:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:07:03 smithi181 ceph-mon[94189]: pgmap v8476: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 274 KiB/s wr, 12 op/s 2023-03-09T19:07:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:07:03 smithi038 ceph-mon[92422]: pgmap v8476: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 274 KiB/s wr, 12 op/s 2023-03-09T19:07:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:07:05 smithi122 ceph-mon[93869]: pgmap v8477: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 336 KiB/s rd, 349 KiB/s wr, 22 op/s 2023-03-09T19:07:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:07:05 smithi122 ceph-mon[93869]: osdmap e743: 12 total, 12 up, 12 in 2023-03-09T19:07:05.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:07:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14936.176509 secs 2023-03-09T19:07:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:07:05 smithi181 ceph-mon[94189]: pgmap v8477: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 336 KiB/s rd, 349 KiB/s wr, 22 op/s 2023-03-09T19:07:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:07:05 smithi181 ceph-mon[94189]: osdmap e743: 12 total, 12 up, 12 in 2023-03-09T19:07:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:07:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14936.176509 secs 2023-03-09T19:07:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:07:05 smithi038 ceph-mon[92422]: pgmap v8477: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 336 KiB/s rd, 349 KiB/s wr, 22 op/s 2023-03-09T19:07:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:07:05 smithi038 ceph-mon[92422]: osdmap e743: 12 total, 12 up, 12 in 2023-03-09T19:07:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:07:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14936.176509 secs 2023-03-09T19:07:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:07:07 smithi122 ceph-mon[93869]: pgmap v8479: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 275 KiB/s wr, 27 op/s 2023-03-09T19:07:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:07:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14938.465311 secs 2023-03-09T19:07:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:07:07 smithi181 ceph-mon[94189]: pgmap v8479: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 275 KiB/s wr, 27 op/s 2023-03-09T19:07:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:07:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14938.465311 secs 2023-03-09T19:07:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:07:07 smithi038 ceph-mon[92422]: pgmap v8479: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 275 KiB/s wr, 27 op/s 2023-03-09T19:07:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:07:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14938.465311 secs 2023-03-09T19:07:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15400.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:07:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:07:09 smithi122 ceph-mon[93869]: pgmap v8480: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 190 KiB/s wr, 26 op/s 2023-03-09T19:07:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:07:09 smithi181 ceph-mon[94189]: pgmap v8480: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 190 KiB/s wr, 26 op/s 2023-03-09T19:07:10.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:07:09 smithi038 ceph-mon[92422]: pgmap v8480: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 190 KiB/s wr, 26 op/s 2023-03-09T19:07:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:07:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14941.176567 secs 2023-03-09T19:07:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:07:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14941.176567 secs 2023-03-09T19:07:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:07:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14941.176567 secs 2023-03-09T19:07:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:07:11 smithi038 ceph-mon[92422]: pgmap v8481: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 190 KiB/s wr, 26 op/s 2023-03-09T19:07:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:07:11 smithi122 ceph-mon[93869]: pgmap v8481: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 190 KiB/s wr, 26 op/s 2023-03-09T19:07:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:07:11 smithi181 ceph-mon[94189]: pgmap v8481: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 190 KiB/s wr, 26 op/s 2023-03-09T19:07:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:07:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14943.465395 secs 2023-03-09T19:07:13.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:07:12 smithi038 ceph-mon[92422]: osdmap e744: 12 total, 12 up, 12 in 2023-03-09T19:07:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:07:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14943.465395 secs 2023-03-09T19:07:13.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:07:12 smithi122 ceph-mon[93869]: osdmap e744: 12 total, 12 up, 12 in 2023-03-09T19:07:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:07:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14943.465395 secs 2023-03-09T19:07:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:07:12 smithi181 ceph-mon[94189]: osdmap e744: 12 total, 12 up, 12 in 2023-03-09T19:07:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15405.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:07:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:07:13 smithi038 ceph-mon[92422]: pgmap v8483: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 34 KiB/s wr, 10 op/s 2023-03-09T19:07:14.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:07:13 smithi122 ceph-mon[93869]: pgmap v8483: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 34 KiB/s wr, 10 op/s 2023-03-09T19:07:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:07:13 smithi181 ceph-mon[94189]: pgmap v8483: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 34 KiB/s wr, 10 op/s 2023-03-09T19:07:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:07:15 smithi038 ceph-mon[92422]: pgmap v8484: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 69 KiB/s wr, 9 op/s 2023-03-09T19:07:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:07:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14946.176629 secs 2023-03-09T19:07:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:07:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:07:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:07:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:07:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:07:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:07:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:07:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:07:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:07:15 smithi122 ceph-mon[93869]: pgmap v8484: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 69 KiB/s wr, 9 op/s 2023-03-09T19:07:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:07:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14946.176629 secs 2023-03-09T19:07:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:07:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:07:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:07:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:07:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:07:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:07:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:07:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:07:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:07:15 smithi181 ceph-mon[94189]: pgmap v8484: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 69 KiB/s wr, 9 op/s 2023-03-09T19:07:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:07:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14946.176629 secs 2023-03-09T19:07:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:07:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:07:16.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:07:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:07:16.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:07:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:07:16.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:07:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:07:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:07:17 smithi122 ceph-mon[93869]: pgmap v8485: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 41 KiB/s wr, 6 op/s 2023-03-09T19:07:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:07:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14948.465512 secs 2023-03-09T19:07:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:07:17 smithi181 ceph-mon[94189]: pgmap v8485: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 41 KiB/s wr, 6 op/s 2023-03-09T19:07:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:07:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14948.465512 secs 2023-03-09T19:07:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:07:17 smithi038 ceph-mon[92422]: pgmap v8485: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 41 KiB/s wr, 6 op/s 2023-03-09T19:07:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:07:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14948.465512 secs 2023-03-09T19:07:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15410.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:07:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:07:19 smithi122 ceph-mon[93869]: pgmap v8486: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 41 KiB/s wr, 6 op/s 2023-03-09T19:07:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:07:19 smithi181 ceph-mon[94189]: pgmap v8486: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 41 KiB/s wr, 6 op/s 2023-03-09T19:07:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:07:19 smithi038 ceph-mon[92422]: pgmap v8486: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 41 KiB/s wr, 6 op/s 2023-03-09T19:07:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:07:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14951.176691 secs 2023-03-09T19:07:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:07:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14951.176691 secs 2023-03-09T19:07:21.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:07:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14951.176691 secs 2023-03-09T19:07:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:07:21 smithi122 ceph-mon[93869]: pgmap v8487: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 41 KiB/s wr, 6 op/s 2023-03-09T19:07:22.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:07:21 smithi181 ceph-mon[94189]: pgmap v8487: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 41 KiB/s wr, 6 op/s 2023-03-09T19:07:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:07:21 smithi038 ceph-mon[92422]: pgmap v8487: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 41 KiB/s wr, 6 op/s 2023-03-09T19:07:23.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:07:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14953.465650 secs 2023-03-09T19:07:23.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:07:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14953.465650 secs 2023-03-09T19:07:23.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:07:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14953.465650 secs 2023-03-09T19:07:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15415.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:07:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:07:23 smithi122 ceph-mon[93869]: pgmap v8488: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 71 KiB/s wr, 6 op/s 2023-03-09T19:07:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:07:23 smithi181 ceph-mon[94189]: pgmap v8488: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 71 KiB/s wr, 6 op/s 2023-03-09T19:07:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:07:23 smithi038 ceph-mon[92422]: pgmap v8488: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 71 KiB/s wr, 6 op/s 2023-03-09T19:07:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:07:25 smithi122 ceph-mon[93869]: pgmap v8489: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 67 KiB/s wr, 7 op/s 2023-03-09T19:07:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:07:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14956.176775 secs 2023-03-09T19:07:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:07:25 smithi181 ceph-mon[94189]: pgmap v8489: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 67 KiB/s wr, 7 op/s 2023-03-09T19:07:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:07:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14956.176775 secs 2023-03-09T19:07:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:07:25 smithi038 ceph-mon[92422]: pgmap v8489: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 67 KiB/s wr, 7 op/s 2023-03-09T19:07:26.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:07:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14956.176775 secs 2023-03-09T19:07:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:07:27 smithi122 ceph-mon[93869]: pgmap v8490: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 34 KiB/s wr, 8 op/s 2023-03-09T19:07:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:07:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14958.465770 secs 2023-03-09T19:07:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:07:27 smithi181 ceph-mon[94189]: pgmap v8490: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 34 KiB/s wr, 8 op/s 2023-03-09T19:07:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:07:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14958.465770 secs 2023-03-09T19:07:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:07:27 smithi038 ceph-mon[92422]: pgmap v8490: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 34 KiB/s wr, 8 op/s 2023-03-09T19:07:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:07:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14958.465770 secs 2023-03-09T19:07:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15420.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:07:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:07:29 smithi122 ceph-mon[93869]: pgmap v8491: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 34 KiB/s wr, 7 op/s 2023-03-09T19:07:30.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:07:29 smithi181 ceph-mon[94189]: pgmap v8491: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 34 KiB/s wr, 7 op/s 2023-03-09T19:07:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:07:29 smithi038 ceph-mon[92422]: pgmap v8491: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 34 KiB/s wr, 7 op/s 2023-03-09T19:07:31.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:07:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14961.176834 secs 2023-03-09T19:07:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:07:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14961.176834 secs 2023-03-09T19:07:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:07:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14961.176834 secs 2023-03-09T19:07:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:07:31 smithi122 ceph-mon[93869]: pgmap v8492: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 34 KiB/s wr, 6 op/s 2023-03-09T19:07:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:07:31 smithi181 ceph-mon[94189]: pgmap v8492: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 34 KiB/s wr, 6 op/s 2023-03-09T19:07:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:07:31 smithi038 ceph-mon[92422]: pgmap v8492: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 34 KiB/s wr, 6 op/s 2023-03-09T19:07:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:07:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14963.465884 secs 2023-03-09T19:07:33.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:07:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14963.465884 secs 2023-03-09T19:07:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:07:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14963.465884 secs 2023-03-09T19:07:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15425.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:07:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:07:33 smithi122 ceph-mon[93869]: pgmap v8493: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 34 KiB/s wr, 9 op/s 2023-03-09T19:07:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:07:33 smithi181 ceph-mon[94189]: pgmap v8493: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 34 KiB/s wr, 9 op/s 2023-03-09T19:07:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:07:33 smithi038 ceph-mon[92422]: pgmap v8493: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 34 KiB/s wr, 9 op/s 2023-03-09T19:07:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:07:35 smithi122 ceph-mon[93869]: pgmap v8494: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.4 KiB/s wr, 7 op/s 2023-03-09T19:07:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:07:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14966.176941 secs 2023-03-09T19:07:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:07:35 smithi181 ceph-mon[94189]: pgmap v8494: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.4 KiB/s wr, 7 op/s 2023-03-09T19:07:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:07:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14966.176941 secs 2023-03-09T19:07:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:07:35 smithi038 ceph-mon[92422]: pgmap v8494: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.4 KiB/s wr, 7 op/s 2023-03-09T19:07:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:07:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14966.176941 secs 2023-03-09T19:07:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:07:37 smithi122 ceph-mon[93869]: pgmap v8495: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 63 KiB/s wr, 8 op/s 2023-03-09T19:07:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:07:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14968.465974 secs 2023-03-09T19:07:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:07:37 smithi181 ceph-mon[94189]: pgmap v8495: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 63 KiB/s wr, 8 op/s 2023-03-09T19:07:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:07:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14968.465974 secs 2023-03-09T19:07:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:07:37 smithi038 ceph-mon[92422]: pgmap v8495: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 63 KiB/s wr, 8 op/s 2023-03-09T19:07:38.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:07:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14968.465974 secs 2023-03-09T19:07:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15430.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:07:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:07:39 smithi122 ceph-mon[93869]: pgmap v8496: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 63 KiB/s wr, 7 op/s 2023-03-09T19:07:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:07:39 smithi181 ceph-mon[94189]: pgmap v8496: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 63 KiB/s wr, 7 op/s 2023-03-09T19:07:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:07:39 smithi038 ceph-mon[92422]: pgmap v8496: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 63 KiB/s wr, 7 op/s 2023-03-09T19:07:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:07:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14971.177006 secs 2023-03-09T19:07:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:07:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14971.177006 secs 2023-03-09T19:07:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:07:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14971.177006 secs 2023-03-09T19:07:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:07:41 smithi122 ceph-mon[93869]: pgmap v8497: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 63 KiB/s wr, 6 op/s 2023-03-09T19:07:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:07:41 smithi181 ceph-mon[94189]: pgmap v8497: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 63 KiB/s wr, 6 op/s 2023-03-09T19:07:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:07:41 smithi038 ceph-mon[92422]: pgmap v8497: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 63 KiB/s wr, 6 op/s 2023-03-09T19:07:43.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:07:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14973.466121 secs 2023-03-09T19:07:43.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:07:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14973.466121 secs 2023-03-09T19:07:43.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:07:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14973.466121 secs 2023-03-09T19:07:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15435.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:07:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:07:43 smithi122 ceph-mon[93869]: pgmap v8498: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 214 KiB/s wr, 9 op/s 2023-03-09T19:07:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:07:43 smithi181 ceph-mon[94189]: pgmap v8498: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 214 KiB/s wr, 9 op/s 2023-03-09T19:07:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:07:43 smithi038 ceph-mon[92422]: pgmap v8498: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 214 KiB/s wr, 9 op/s 2023-03-09T19:07:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:07:45 smithi122 ceph-mon[93869]: pgmap v8499: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 226 KiB/s wr, 8 op/s 2023-03-09T19:07:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:07:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14976.177089 secs 2023-03-09T19:07:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:07:45 smithi181 ceph-mon[94189]: pgmap v8499: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 226 KiB/s wr, 8 op/s 2023-03-09T19:07:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:07:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14976.177089 secs 2023-03-09T19:07:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:07:45 smithi038 ceph-mon[92422]: pgmap v8499: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 226 KiB/s wr, 8 op/s 2023-03-09T19:07:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:07:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14976.177089 secs 2023-03-09T19:07:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:07:47 smithi122 ceph-mon[93869]: pgmap v8500: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 345 KiB/s wr, 10 op/s 2023-03-09T19:07:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:07:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14978.466250 secs 2023-03-09T19:07:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:07:47 smithi181 ceph-mon[94189]: pgmap v8500: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 345 KiB/s wr, 10 op/s 2023-03-09T19:07:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:07:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14978.466250 secs 2023-03-09T19:07:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:07:47 smithi038 ceph-mon[92422]: pgmap v8500: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 345 KiB/s wr, 10 op/s 2023-03-09T19:07:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:07:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14978.466250 secs 2023-03-09T19:07:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15440.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:07:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:07:49 smithi122 ceph-mon[93869]: pgmap v8501: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 310 KiB/s rd, 297 KiB/s wr, 9 op/s 2023-03-09T19:07:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:07:49 smithi181 ceph-mon[94189]: pgmap v8501: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 310 KiB/s rd, 297 KiB/s wr, 9 op/s 2023-03-09T19:07:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:07:49 smithi038 ceph-mon[92422]: pgmap v8501: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 310 KiB/s rd, 297 KiB/s wr, 9 op/s 2023-03-09T19:07:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:07:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14981.177150 secs 2023-03-09T19:07:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:07:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14981.177150 secs 2023-03-09T19:07:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:07:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14981.177150 secs 2023-03-09T19:07:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:07:51 smithi122 ceph-mon[93869]: pgmap v8502: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 422 KiB/s rd, 409 KiB/s wr, 10 op/s 2023-03-09T19:07:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:07:51 smithi181 ceph-mon[94189]: pgmap v8502: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 422 KiB/s rd, 409 KiB/s wr, 10 op/s 2023-03-09T19:07:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:07:51 smithi038 ceph-mon[92422]: pgmap v8502: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 422 KiB/s rd, 409 KiB/s wr, 10 op/s 2023-03-09T19:07:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:07:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14983.466368 secs 2023-03-09T19:07:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:07:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14983.466368 secs 2023-03-09T19:07:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:07:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14983.466368 secs 2023-03-09T19:07:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15445.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:07:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:07:53 smithi122 ceph-mon[93869]: pgmap v8503: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 410 KiB/s wr, 12 op/s 2023-03-09T19:07:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:07:53 smithi181 ceph-mon[94189]: pgmap v8503: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 410 KiB/s wr, 12 op/s 2023-03-09T19:07:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:07:53 smithi038 ceph-mon[92422]: pgmap v8503: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 410 KiB/s wr, 12 op/s 2023-03-09T19:07:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:07:55 smithi122 ceph-mon[93869]: pgmap v8504: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 282 KiB/s wr, 11 op/s 2023-03-09T19:07:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:07:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14986.177209 secs 2023-03-09T19:07:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:07:55 smithi181 ceph-mon[94189]: pgmap v8504: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 282 KiB/s wr, 11 op/s 2023-03-09T19:07:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:07:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14986.177209 secs 2023-03-09T19:07:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:07:55 smithi038 ceph-mon[92422]: pgmap v8504: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 282 KiB/s wr, 11 op/s 2023-03-09T19:07:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:07:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14986.177209 secs 2023-03-09T19:07:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:07:57 smithi122 ceph-mon[93869]: pgmap v8505: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 347 KiB/s wr, 12 op/s 2023-03-09T19:07:58.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:07:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14988.466489 secs 2023-03-09T19:07:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:07:57 smithi181 ceph-mon[94189]: pgmap v8505: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 347 KiB/s wr, 12 op/s 2023-03-09T19:07:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:07:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14988.466489 secs 2023-03-09T19:07:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:07:57 smithi038 ceph-mon[92422]: pgmap v8505: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 347 KiB/s wr, 12 op/s 2023-03-09T19:07:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:07:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14988.466489 secs 2023-03-09T19:07:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15450.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:08:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:07:59 smithi122 ceph-mon[93869]: pgmap v8506: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 240 KiB/s wr, 10 op/s 2023-03-09T19:08:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:07:59 smithi181 ceph-mon[94189]: pgmap v8506: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 240 KiB/s wr, 10 op/s 2023-03-09T19:08:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:07:59 smithi038 ceph-mon[92422]: pgmap v8506: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 240 KiB/s wr, 10 op/s 2023-03-09T19:08:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:08:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14991.177276 secs 2023-03-09T19:08:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:08:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14991.177276 secs 2023-03-09T19:08:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:08:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14991.177276 secs 2023-03-09T19:08:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:08:01 smithi122 ceph-mon[93869]: pgmap v8507: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 314 KiB/s wr, 10 op/s 2023-03-09T19:08:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:08:01 smithi181 ceph-mon[94189]: pgmap v8507: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 314 KiB/s wr, 10 op/s 2023-03-09T19:08:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:08:01 smithi038 ceph-mon[92422]: pgmap v8507: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 314 KiB/s wr, 10 op/s 2023-03-09T19:08:03.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:08:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14993.466609 secs 2023-03-09T19:08:03.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:08:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14993.466609 secs 2023-03-09T19:08:03.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:08:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14993.466609 secs 2023-03-09T19:08:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15455.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:08:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:08:03 smithi122 ceph-mon[93869]: pgmap v8508: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 201 KiB/s wr, 10 op/s 2023-03-09T19:08:04.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:08:03 smithi181 ceph-mon[94189]: pgmap v8508: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 201 KiB/s wr, 10 op/s 2023-03-09T19:08:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:08:03 smithi038 ceph-mon[92422]: pgmap v8508: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 201 KiB/s wr, 10 op/s 2023-03-09T19:08:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:08:05 smithi122 ceph-mon[93869]: pgmap v8509: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 222 KiB/s wr, 20 op/s 2023-03-09T19:08:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:08:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14996.177345 secs 2023-03-09T19:08:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:08:05 smithi181 ceph-mon[94189]: pgmap v8509: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 222 KiB/s wr, 20 op/s 2023-03-09T19:08:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:08:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14996.177345 secs 2023-03-09T19:08:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:08:05 smithi038 ceph-mon[92422]: pgmap v8509: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 222 KiB/s wr, 20 op/s 2023-03-09T19:08:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:08:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14996.177345 secs 2023-03-09T19:08:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:08:07 smithi122 ceph-mon[93869]: pgmap v8510: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 269 KiB/s wr, 24 op/s 2023-03-09T19:08:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:08:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 14998.466726 secs 2023-03-09T19:08:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:08:07 smithi181 ceph-mon[94189]: pgmap v8510: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 269 KiB/s wr, 24 op/s 2023-03-09T19:08:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:08:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 14998.466726 secs 2023-03-09T19:08:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:08:07 smithi038 ceph-mon[92422]: pgmap v8510: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 269 KiB/s wr, 24 op/s 2023-03-09T19:08:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:08:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 14998.466726 secs 2023-03-09T19:08:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15460.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:08:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:08:09 smithi122 ceph-mon[93869]: pgmap v8511: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 192 KiB/s wr, 22 op/s 2023-03-09T19:08:10.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:08:09 smithi122 ceph-mon[93869]: osdmap e745: 12 total, 12 up, 12 in 2023-03-09T19:08:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:08:09 smithi181 ceph-mon[94189]: pgmap v8511: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 192 KiB/s wr, 22 op/s 2023-03-09T19:08:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:08:09 smithi181 ceph-mon[94189]: osdmap e745: 12 total, 12 up, 12 in 2023-03-09T19:08:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:08:09 smithi038 ceph-mon[92422]: pgmap v8511: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 192 KiB/s wr, 22 op/s 2023-03-09T19:08:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:08:09 smithi038 ceph-mon[92422]: osdmap e745: 12 total, 12 up, 12 in 2023-03-09T19:08:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:08:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15001.177405 secs 2023-03-09T19:08:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:08:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15001.177405 secs 2023-03-09T19:08:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:08:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15001.177405 secs 2023-03-09T19:08:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:08:11 smithi122 ceph-mon[93869]: pgmap v8513: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 110 KiB/s wr, 25 op/s 2023-03-09T19:08:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:08:11 smithi181 ceph-mon[94189]: pgmap v8513: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 110 KiB/s wr, 25 op/s 2023-03-09T19:08:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:08:11 smithi038 ceph-mon[92422]: pgmap v8513: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 110 KiB/s wr, 25 op/s 2023-03-09T19:08:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:08:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15003.466874 secs 2023-03-09T19:08:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:08:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15003.466874 secs 2023-03-09T19:08:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:08:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15003.466874 secs 2023-03-09T19:08:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15465.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:08:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:08:13 smithi122 ceph-mon[93869]: pgmap v8514: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 110 KiB/s wr, 24 op/s 2023-03-09T19:08:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:08:13 smithi181 ceph-mon[94189]: pgmap v8514: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 110 KiB/s wr, 24 op/s 2023-03-09T19:08:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:08:13 smithi038 ceph-mon[92422]: pgmap v8514: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 110 KiB/s wr, 24 op/s 2023-03-09T19:08:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:08:15 smithi181 ceph-mon[94189]: pgmap v8515: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 84 KiB/s wr, 9 op/s 2023-03-09T19:08:16.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:08:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15006.177466 secs 2023-03-09T19:08:16.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:08:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:08:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:08:15 smithi038 ceph-mon[92422]: pgmap v8515: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 84 KiB/s wr, 9 op/s 2023-03-09T19:08:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:08:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15006.177466 secs 2023-03-09T19:08:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:08:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:08:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:08:15 smithi122 ceph-mon[93869]: pgmap v8515: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 84 KiB/s wr, 9 op/s 2023-03-09T19:08:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:08:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15006.177466 secs 2023-03-09T19:08:16.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:08:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:08:17.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:08:16 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:08:17.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:08:16 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:08:17.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:08:16 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:08:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:08:16 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:08:17.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:08:16 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:08:17.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:08:16 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:08:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:08:16 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:08:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:08:16 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:08:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:08:16 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:08:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:08:17 smithi038 ceph-mon[92422]: pgmap v8516: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T19:08:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:08:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15008.466992 secs 2023-03-09T19:08:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:08:17 smithi038 ceph-mon[92422]: osdmap e746: 12 total, 12 up, 12 in 2023-03-09T19:08:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:08:17 smithi122 ceph-mon[93869]: pgmap v8516: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T19:08:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:08:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15008.466992 secs 2023-03-09T19:08:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:08:17 smithi122 ceph-mon[93869]: osdmap e746: 12 total, 12 up, 12 in 2023-03-09T19:08:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:08:17 smithi181 ceph-mon[94189]: pgmap v8516: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T19:08:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:08:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15008.466992 secs 2023-03-09T19:08:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:08:17 smithi181 ceph-mon[94189]: osdmap e746: 12 total, 12 up, 12 in 2023-03-09T19:08:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15470.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:08:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:08:19 smithi038 ceph-mon[92422]: pgmap v8518: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 213 B/s wr, 6 op/s 2023-03-09T19:08:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:08:19 smithi122 ceph-mon[93869]: pgmap v8518: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 213 B/s wr, 6 op/s 2023-03-09T19:08:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:08:19 smithi181 ceph-mon[94189]: pgmap v8518: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 213 B/s wr, 6 op/s 2023-03-09T19:08:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:08:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15011.177526 secs 2023-03-09T19:08:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:08:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15011.177526 secs 2023-03-09T19:08:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:08:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15011.177526 secs 2023-03-09T19:08:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:08:21 smithi038 ceph-mon[92422]: pgmap v8519: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T19:08:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:08:21 smithi122 ceph-mon[93869]: pgmap v8519: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T19:08:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:08:21 smithi181 ceph-mon[94189]: pgmap v8519: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T19:08:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:08:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15013.467125 secs 2023-03-09T19:08:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:08:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15013.467125 secs 2023-03-09T19:08:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:08:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15013.467125 secs 2023-03-09T19:08:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15475.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:08:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:08:24 smithi122 ceph-mon[93869]: pgmap v8520: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T19:08:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:08:24 smithi181 ceph-mon[94189]: pgmap v8520: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T19:08:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:08:24 smithi038 ceph-mon[92422]: pgmap v8520: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T19:08:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:08:26 smithi122 ceph-mon[93869]: pgmap v8521: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T19:08:26.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:08:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15016.177611 secs 2023-03-09T19:08:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:08:26 smithi181 ceph-mon[94189]: pgmap v8521: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T19:08:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:08:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15016.177611 secs 2023-03-09T19:08:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:08:26 smithi038 ceph-mon[92422]: pgmap v8521: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T19:08:26.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:08:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15016.177611 secs 2023-03-09T19:08:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:08:28 smithi122 ceph-mon[93869]: pgmap v8522: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 49 KiB/s wr, 6 op/s 2023-03-09T19:08:28.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:08:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15018.467220 secs 2023-03-09T19:08:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:08:28 smithi181 ceph-mon[94189]: pgmap v8522: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 49 KiB/s wr, 6 op/s 2023-03-09T19:08:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:08:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15018.467220 secs 2023-03-09T19:08:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:08:28 smithi038 ceph-mon[92422]: pgmap v8522: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 49 KiB/s wr, 6 op/s 2023-03-09T19:08:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:08:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15018.467220 secs 2023-03-09T19:08:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15480.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:08:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:08:30 smithi122 ceph-mon[93869]: pgmap v8523: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T19:08:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:08:30 smithi181 ceph-mon[94189]: pgmap v8523: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T19:08:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:08:30 smithi038 ceph-mon[92422]: pgmap v8523: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T19:08:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:08:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15021.177689 secs 2023-03-09T19:08:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:08:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15021.177689 secs 2023-03-09T19:08:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:08:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15021.177689 secs 2023-03-09T19:08:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:08:32 smithi122 ceph-mon[93869]: pgmap v8524: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 41 KiB/s wr, 6 op/s 2023-03-09T19:08:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:08:32 smithi181 ceph-mon[94189]: pgmap v8524: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 41 KiB/s wr, 6 op/s 2023-03-09T19:08:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:08:32 smithi038 ceph-mon[92422]: pgmap v8524: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 41 KiB/s wr, 6 op/s 2023-03-09T19:08:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:08:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15023.467285 secs 2023-03-09T19:08:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:08:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15023.467285 secs 2023-03-09T19:08:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:08:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15023.467285 secs 2023-03-09T19:08:33.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15485.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:08:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:08:34 smithi122 ceph-mon[93869]: pgmap v8525: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 88 KiB/s wr, 8 op/s 2023-03-09T19:08:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:08:34 smithi181 ceph-mon[94189]: pgmap v8525: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 88 KiB/s wr, 8 op/s 2023-03-09T19:08:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:08:34 smithi038 ceph-mon[92422]: pgmap v8525: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 88 KiB/s wr, 8 op/s 2023-03-09T19:08:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:08:36 smithi122 ceph-mon[93869]: pgmap v8526: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 88 KiB/s wr, 7 op/s 2023-03-09T19:08:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:08:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15026.177773 secs 2023-03-09T19:08:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:08:36 smithi181 ceph-mon[94189]: pgmap v8526: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 88 KiB/s wr, 7 op/s 2023-03-09T19:08:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:08:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15026.177773 secs 2023-03-09T19:08:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:08:36 smithi038 ceph-mon[92422]: pgmap v8526: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 88 KiB/s wr, 7 op/s 2023-03-09T19:08:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:08:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15026.177773 secs 2023-03-09T19:08:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:08:38 smithi122 ceph-mon[93869]: pgmap v8527: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 191 KiB/s wr, 10 op/s 2023-03-09T19:08:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:08:38 smithi181 ceph-mon[94189]: pgmap v8527: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 191 KiB/s wr, 10 op/s 2023-03-09T19:08:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:08:38 smithi038 ceph-mon[92422]: pgmap v8527: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 191 KiB/s wr, 10 op/s 2023-03-09T19:08:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15490.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:08:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:08:40 smithi122 ceph-mon[93869]: pgmap v8528: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 151 KiB/s wr, 8 op/s 2023-03-09T19:08:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:08:40 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:08:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:08:40 smithi181 ceph-mon[94189]: pgmap v8528: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 151 KiB/s wr, 8 op/s 2023-03-09T19:08:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:08:40 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:08:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:08:40 smithi038 ceph-mon[92422]: pgmap v8528: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 151 KiB/s wr, 8 op/s 2023-03-09T19:08:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:08:40 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:08:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:08:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15031.177857 secs 2023-03-09T19:08:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:08:41 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:08:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:08:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:08:41.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:08:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:08:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:08:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15031.177857 secs 2023-03-09T19:08:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:08:41 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:08:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:08:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:08:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:08:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:08:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:08:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15031.177857 secs 2023-03-09T19:08:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:08:41 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:08:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:08:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:08:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:08:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:08:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:08:42 smithi122 ceph-mon[93869]: pgmap v8529: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 150 KiB/s wr, 8 op/s 2023-03-09T19:08:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:08:42 smithi181 ceph-mon[94189]: pgmap v8529: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 150 KiB/s wr, 8 op/s 2023-03-09T19:08:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:08:42 smithi038 ceph-mon[92422]: pgmap v8529: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 150 KiB/s wr, 8 op/s 2023-03-09T19:08:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15495.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:08:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:08:43 smithi122 ceph-mon[93869]: pgmap v8530: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 305 KiB/s wr, 10 op/s 2023-03-09T19:08:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:08:43 smithi181 ceph-mon[94189]: pgmap v8530: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 305 KiB/s wr, 10 op/s 2023-03-09T19:08:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:08:43 smithi038 ceph-mon[92422]: pgmap v8530: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 305 KiB/s wr, 10 op/s 2023-03-09T19:08:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:08:45 smithi122 ceph-mon[93869]: pgmap v8531: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 281 KiB/s wr, 9 op/s 2023-03-09T19:08:45.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:08:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15036.177918 secs 2023-03-09T19:08:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:08:45 smithi181 ceph-mon[94189]: pgmap v8531: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 281 KiB/s wr, 9 op/s 2023-03-09T19:08:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:08:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15036.177918 secs 2023-03-09T19:08:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:08:45 smithi038 ceph-mon[92422]: pgmap v8531: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 281 KiB/s wr, 9 op/s 2023-03-09T19:08:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:08:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15036.177918 secs 2023-03-09T19:08:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:08:47 smithi122 ceph-mon[93869]: pgmap v8532: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 549 KiB/s rd, 281 KiB/s wr, 12 op/s 2023-03-09T19:08:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:08:47 smithi181 ceph-mon[94189]: pgmap v8532: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 549 KiB/s rd, 281 KiB/s wr, 12 op/s 2023-03-09T19:08:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:08:47 smithi038 ceph-mon[92422]: pgmap v8532: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 549 KiB/s rd, 281 KiB/s wr, 12 op/s 2023-03-09T19:08:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15500.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:08:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:08:49 smithi122 ceph-mon[93869]: pgmap v8533: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 193 KiB/s wr, 10 op/s 2023-03-09T19:08:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:08:49 smithi181 ceph-mon[94189]: pgmap v8533: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 193 KiB/s wr, 10 op/s 2023-03-09T19:08:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:08:49 smithi038 ceph-mon[92422]: pgmap v8533: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 193 KiB/s wr, 10 op/s 2023-03-09T19:08:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:08:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15041.177981 secs 2023-03-09T19:08:50.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:08:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15041.177981 secs 2023-03-09T19:08:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:08:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15041.177981 secs 2023-03-09T19:08:51.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:08:51 smithi122 ceph-mon[93869]: pgmap v8534: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 560 KiB/s rd, 371 KiB/s wr, 10 op/s 2023-03-09T19:08:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:08:51 smithi181 ceph-mon[94189]: pgmap v8534: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 560 KiB/s rd, 371 KiB/s wr, 10 op/s 2023-03-09T19:08:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:08:51 smithi038 ceph-mon[92422]: pgmap v8534: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 560 KiB/s rd, 371 KiB/s wr, 10 op/s 2023-03-09T19:08:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15505.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:08:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:08:53 smithi122 ceph-mon[93869]: pgmap v8535: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 561 KiB/s rd, 372 KiB/s wr, 13 op/s 2023-03-09T19:08:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:08:53 smithi181 ceph-mon[94189]: pgmap v8535: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 561 KiB/s rd, 372 KiB/s wr, 13 op/s 2023-03-09T19:08:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:08:53 smithi038 ceph-mon[92422]: pgmap v8535: 129 pgs: 129 active+clean; 9.2 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 561 KiB/s rd, 372 KiB/s wr, 13 op/s 2023-03-09T19:08:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:08:55 smithi122 ceph-mon[93869]: pgmap v8536: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 469 KiB/s rd, 330 KiB/s wr, 12 op/s 2023-03-09T19:08:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:08:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15046.178039 secs 2023-03-09T19:08:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:08:55 smithi181 ceph-mon[94189]: pgmap v8536: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 469 KiB/s rd, 330 KiB/s wr, 12 op/s 2023-03-09T19:08:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:08:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15046.178039 secs 2023-03-09T19:08:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:08:55 smithi038 ceph-mon[92422]: pgmap v8536: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 469 KiB/s rd, 330 KiB/s wr, 12 op/s 2023-03-09T19:08:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:08:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15046.178039 secs 2023-03-09T19:08:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:08:57 smithi122 ceph-mon[93869]: pgmap v8537: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 308 KiB/s wr, 13 op/s 2023-03-09T19:08:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:08:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15048.467926 secs 2023-03-09T19:08:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:08:57 smithi181 ceph-mon[94189]: pgmap v8537: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 308 KiB/s wr, 13 op/s 2023-03-09T19:08:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:08:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15048.467926 secs 2023-03-09T19:08:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:08:57 smithi038 ceph-mon[92422]: pgmap v8537: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 308 KiB/s wr, 13 op/s 2023-03-09T19:08:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:08:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15048.467926 secs 2023-03-09T19:08:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15510.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:08:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:08:59 smithi122 ceph-mon[93869]: pgmap v8538: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 319 KiB/s wr, 11 op/s 2023-03-09T19:08:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:08:59 smithi181 ceph-mon[94189]: pgmap v8538: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 319 KiB/s wr, 11 op/s 2023-03-09T19:08:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:08:59 smithi038 ceph-mon[92422]: pgmap v8538: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 319 KiB/s wr, 11 op/s 2023-03-09T19:09:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:09:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15051.178113 secs 2023-03-09T19:09:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:09:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15051.178113 secs 2023-03-09T19:09:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:09:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15051.178113 secs 2023-03-09T19:09:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:09:01 smithi122 ceph-mon[93869]: pgmap v8539: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 394 KiB/s wr, 11 op/s 2023-03-09T19:09:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:09:01 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:09:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:09:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:09:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:09:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:09:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:09:01 smithi181 ceph-mon[94189]: pgmap v8539: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 394 KiB/s wr, 11 op/s 2023-03-09T19:09:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:09:01 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:09:01.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:09:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:09:01.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:09:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:09:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:09:01 smithi038 ceph-mon[92422]: pgmap v8539: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 394 KiB/s wr, 11 op/s 2023-03-09T19:09:02.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:09:01 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:09:02.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:09:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:09:02.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:09:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:09:02.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:09:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15053.468037 secs 2023-03-09T19:09:02.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:09:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15053.468037 secs 2023-03-09T19:09:02.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:09:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15053.468037 secs 2023-03-09T19:09:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15515.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:09:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:09:03 smithi122 ceph-mon[93869]: pgmap v8540: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 237 KiB/s wr, 14 op/s 2023-03-09T19:09:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:09:03 smithi181 ceph-mon[94189]: pgmap v8540: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 237 KiB/s wr, 14 op/s 2023-03-09T19:09:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:09:03 smithi038 ceph-mon[92422]: pgmap v8540: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 237 KiB/s wr, 14 op/s 2023-03-09T19:09:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:09:05 smithi122 ceph-mon[93869]: pgmap v8541: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 306 KiB/s wr, 25 op/s 2023-03-09T19:09:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:09:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15056.178174 secs 2023-03-09T19:09:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:09:05 smithi181 ceph-mon[94189]: pgmap v8541: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 306 KiB/s wr, 25 op/s 2023-03-09T19:09:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:09:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15056.178174 secs 2023-03-09T19:09:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:09:05 smithi038 ceph-mon[92422]: pgmap v8541: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 306 KiB/s wr, 25 op/s 2023-03-09T19:09:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:09:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15056.178174 secs 2023-03-09T19:09:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:09:07 smithi122 ceph-mon[93869]: pgmap v8542: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 193 KiB/s wr, 25 op/s 2023-03-09T19:09:07.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:09:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15058.468170 secs 2023-03-09T19:09:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:09:07 smithi181 ceph-mon[94189]: pgmap v8542: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 193 KiB/s wr, 25 op/s 2023-03-09T19:09:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:09:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15058.468170 secs 2023-03-09T19:09:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:09:07 smithi038 ceph-mon[92422]: pgmap v8542: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 193 KiB/s wr, 25 op/s 2023-03-09T19:09:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:09:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15058.468170 secs 2023-03-09T19:09:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15520.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:09:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:09:09 smithi122 ceph-mon[93869]: pgmap v8543: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 193 KiB/s wr, 23 op/s 2023-03-09T19:09:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:09:09 smithi181 ceph-mon[94189]: pgmap v8543: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 193 KiB/s wr, 23 op/s 2023-03-09T19:09:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:09:09 smithi038 ceph-mon[92422]: pgmap v8543: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 193 KiB/s wr, 23 op/s 2023-03-09T19:09:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:09:10 smithi122 ceph-mon[93869]: osdmap e747: 12 total, 12 up, 12 in 2023-03-09T19:09:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:09:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15061.179073 secs 2023-03-09T19:09:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:09:10 smithi181 ceph-mon[94189]: osdmap e747: 12 total, 12 up, 12 in 2023-03-09T19:09:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:09:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15061.179073 secs 2023-03-09T19:09:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:09:10 smithi038 ceph-mon[92422]: osdmap e747: 12 total, 12 up, 12 in 2023-03-09T19:09:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:09:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15061.179073 secs 2023-03-09T19:09:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:09:11 smithi122 ceph-mon[93869]: pgmap v8545: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 111 KiB/s wr, 26 op/s 2023-03-09T19:09:11.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:09:11 smithi181 ceph-mon[94189]: pgmap v8545: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 111 KiB/s wr, 26 op/s 2023-03-09T19:09:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:09:11 smithi038 ceph-mon[92422]: pgmap v8545: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 111 KiB/s wr, 26 op/s 2023-03-09T19:09:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:09:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15063.468297 secs 2023-03-09T19:09:12.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:09:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15063.468297 secs 2023-03-09T19:09:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:09:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15063.468297 secs 2023-03-09T19:09:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15525.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:09:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:09:13 smithi122 ceph-mon[93869]: pgmap v8546: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 84 KiB/s wr, 23 op/s 2023-03-09T19:09:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:09:13 smithi181 ceph-mon[94189]: pgmap v8546: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 84 KiB/s wr, 23 op/s 2023-03-09T19:09:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:09:13 smithi038 ceph-mon[92422]: pgmap v8546: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 84 KiB/s wr, 23 op/s 2023-03-09T19:09:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:09:15 smithi122 ceph-mon[93869]: pgmap v8547: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 41 KiB/s wr, 7 op/s 2023-03-09T19:09:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:09:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15066.179133 secs 2023-03-09T19:09:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:09:15 smithi181 ceph-mon[94189]: pgmap v8547: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 41 KiB/s wr, 7 op/s 2023-03-09T19:09:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:09:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15066.179133 secs 2023-03-09T19:09:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:09:15 smithi038 ceph-mon[92422]: pgmap v8547: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 41 KiB/s wr, 7 op/s 2023-03-09T19:09:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:09:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15066.179133 secs 2023-03-09T19:09:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:09:16 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:09:16.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:09:16 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:09:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:09:16 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:09:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:09:17 smithi181 ceph-mon[94189]: pgmap v8548: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 41 KiB/s wr, 7 op/s 2023-03-09T19:09:17.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:09:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15068.468371 secs 2023-03-09T19:09:17.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:09:17 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:09:17.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:09:17 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:09:17.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:09:17 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:09:17.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:09:17 smithi181 ceph-mon[94189]: osdmap e748: 12 total, 12 up, 12 in 2023-03-09T19:09:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:09:17 smithi038 ceph-mon[92422]: pgmap v8548: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 41 KiB/s wr, 7 op/s 2023-03-09T19:09:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:09:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15068.468371 secs 2023-03-09T19:09:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:09:17 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:09:18.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:09:17 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:09:18.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:09:17 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:09:18.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:09:17 smithi038 ceph-mon[92422]: osdmap e748: 12 total, 12 up, 12 in 2023-03-09T19:09:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:09:17 smithi122 ceph-mon[93869]: pgmap v8548: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 41 KiB/s wr, 7 op/s 2023-03-09T19:09:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:09:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15068.468371 secs 2023-03-09T19:09:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:09:17 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:09:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:09:17 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:09:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:09:17 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:09:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:09:17 smithi122 ceph-mon[93869]: osdmap e748: 12 total, 12 up, 12 in 2023-03-09T19:09:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15530.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:09:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:09:19 smithi038 ceph-mon[92422]: pgmap v8550: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 46 KiB/s wr, 6 op/s 2023-03-09T19:09:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:09:19 smithi122 ceph-mon[93869]: pgmap v8550: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 46 KiB/s wr, 6 op/s 2023-03-09T19:09:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:09:19 smithi181 ceph-mon[94189]: pgmap v8550: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 46 KiB/s wr, 6 op/s 2023-03-09T19:09:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:09:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15071.179196 secs 2023-03-09T19:09:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:09:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15071.179196 secs 2023-03-09T19:09:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:09:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15071.179196 secs 2023-03-09T19:09:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:09:21 smithi038 ceph-mon[92422]: pgmap v8551: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 41 KiB/s wr, 6 op/s 2023-03-09T19:09:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:09:21 smithi122 ceph-mon[93869]: pgmap v8551: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 41 KiB/s wr, 6 op/s 2023-03-09T19:09:22.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:09:21 smithi181 ceph-mon[94189]: pgmap v8551: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 41 KiB/s wr, 6 op/s 2023-03-09T19:09:22.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:09:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15073.468501 secs 2023-03-09T19:09:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:09:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15073.468501 secs 2023-03-09T19:09:23.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:09:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15073.468501 secs 2023-03-09T19:09:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15535.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:09:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:09:23 smithi122 ceph-mon[93869]: pgmap v8552: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 41 KiB/s wr, 6 op/s 2023-03-09T19:09:24.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:09:23 smithi181 ceph-mon[94189]: pgmap v8552: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 41 KiB/s wr, 6 op/s 2023-03-09T19:09:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:09:23 smithi038 ceph-mon[92422]: pgmap v8552: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 41 KiB/s wr, 6 op/s 2023-03-09T19:09:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:09:25 smithi122 ceph-mon[93869]: pgmap v8553: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 6 op/s 2023-03-09T19:09:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:09:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15076.179255 secs 2023-03-09T19:09:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:09:25 smithi181 ceph-mon[94189]: pgmap v8553: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 6 op/s 2023-03-09T19:09:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:09:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15076.179255 secs 2023-03-09T19:09:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:09:25 smithi038 ceph-mon[92422]: pgmap v8553: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 6 op/s 2023-03-09T19:09:26.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:09:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15076.179255 secs 2023-03-09T19:09:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:09:27 smithi122 ceph-mon[93869]: pgmap v8554: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 6 op/s 2023-03-09T19:09:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:09:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15078.468561 secs 2023-03-09T19:09:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:09:27 smithi181 ceph-mon[94189]: pgmap v8554: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 6 op/s 2023-03-09T19:09:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:09:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15078.468561 secs 2023-03-09T19:09:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:09:27 smithi038 ceph-mon[92422]: pgmap v8554: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 6 op/s 2023-03-09T19:09:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:09:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15078.468561 secs 2023-03-09T19:09:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15540.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:09:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:09:29 smithi122 ceph-mon[93869]: pgmap v8555: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 277 B/s wr, 6 op/s 2023-03-09T19:09:30.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:09:29 smithi181 ceph-mon[94189]: pgmap v8555: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 277 B/s wr, 6 op/s 2023-03-09T19:09:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:09:29 smithi038 ceph-mon[92422]: pgmap v8555: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 277 B/s wr, 6 op/s 2023-03-09T19:09:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:09:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15081.179315 secs 2023-03-09T19:09:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:09:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15081.179315 secs 2023-03-09T19:09:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:09:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15081.179315 secs 2023-03-09T19:09:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:09:31 smithi122 ceph-mon[93869]: pgmap v8556: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 255 B/s wr, 5 op/s 2023-03-09T19:09:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:09:31 smithi181 ceph-mon[94189]: pgmap v8556: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 255 B/s wr, 5 op/s 2023-03-09T19:09:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:09:31 smithi038 ceph-mon[92422]: pgmap v8556: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 255 B/s wr, 5 op/s 2023-03-09T19:09:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:09:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15083.468686 secs 2023-03-09T19:09:33.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:09:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15083.468686 secs 2023-03-09T19:09:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:09:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15083.468686 secs 2023-03-09T19:09:33.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15545.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:09:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:09:33 smithi122 ceph-mon[93869]: pgmap v8557: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 7 op/s 2023-03-09T19:09:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:09:33 smithi181 ceph-mon[94189]: pgmap v8557: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 7 op/s 2023-03-09T19:09:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:09:33 smithi038 ceph-mon[92422]: pgmap v8557: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 7 op/s 2023-03-09T19:09:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:09:35 smithi122 ceph-mon[93869]: pgmap v8558: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 104 KiB/s wr, 6 op/s 2023-03-09T19:09:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:09:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15086.179374 secs 2023-03-09T19:09:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:09:35 smithi181 ceph-mon[94189]: pgmap v8558: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 104 KiB/s wr, 6 op/s 2023-03-09T19:09:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:09:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15086.179374 secs 2023-03-09T19:09:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:09:35 smithi038 ceph-mon[92422]: pgmap v8558: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 104 KiB/s wr, 6 op/s 2023-03-09T19:09:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:09:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15086.179374 secs 2023-03-09T19:09:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:09:37 smithi122 ceph-mon[93869]: pgmap v8559: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 104 KiB/s wr, 7 op/s 2023-03-09T19:09:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:09:37 smithi181 ceph-mon[94189]: pgmap v8559: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 104 KiB/s wr, 7 op/s 2023-03-09T19:09:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:09:37 smithi038 ceph-mon[92422]: pgmap v8559: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 104 KiB/s wr, 7 op/s 2023-03-09T19:09:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15550.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:09:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:09:39 smithi122 ceph-mon[93869]: pgmap v8560: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 104 KiB/s wr, 7 op/s 2023-03-09T19:09:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:09:39 smithi181 ceph-mon[94189]: pgmap v8560: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 104 KiB/s wr, 7 op/s 2023-03-09T19:09:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:09:39 smithi038 ceph-mon[92422]: pgmap v8560: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 104 KiB/s wr, 7 op/s 2023-03-09T19:09:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:09:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15091.179434 secs 2023-03-09T19:09:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:09:40 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:09:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:09:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15091.179434 secs 2023-03-09T19:09:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:09:40 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:09:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:09:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15091.179434 secs 2023-03-09T19:09:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:09:40 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:09:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:09:41 smithi122 ceph-mon[93869]: pgmap v8561: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 236 KiB/s wr, 7 op/s 2023-03-09T19:09:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:09:41 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:09:42.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:09:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:09:42.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:09:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:09:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:09:41 smithi181 ceph-mon[94189]: pgmap v8561: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 236 KiB/s wr, 7 op/s 2023-03-09T19:09:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:09:41 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:09:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:09:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:09:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:09:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:09:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:09:41 smithi038 ceph-mon[92422]: pgmap v8561: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 236 KiB/s wr, 7 op/s 2023-03-09T19:09:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:09:41 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:09:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:09:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:09:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:09:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:09:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15555.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:09:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:09:43 smithi122 ceph-mon[93869]: pgmap v8562: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 236 KiB/s wr, 8 op/s 2023-03-09T19:09:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:09:43 smithi181 ceph-mon[94189]: pgmap v8562: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 236 KiB/s wr, 8 op/s 2023-03-09T19:09:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:09:43 smithi038 ceph-mon[92422]: pgmap v8562: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 236 KiB/s wr, 8 op/s 2023-03-09T19:09:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:09:45 smithi122 ceph-mon[93869]: pgmap v8563: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 282 KiB/s wr, 9 op/s 2023-03-09T19:09:45.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:09:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15096.179519 secs 2023-03-09T19:09:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:09:45 smithi181 ceph-mon[94189]: pgmap v8563: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 282 KiB/s wr, 9 op/s 2023-03-09T19:09:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:09:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15096.179519 secs 2023-03-09T19:09:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:09:45 smithi038 ceph-mon[92422]: pgmap v8563: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 282 KiB/s wr, 9 op/s 2023-03-09T19:09:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:09:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15096.179519 secs 2023-03-09T19:09:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:09:47 smithi122 ceph-mon[93869]: pgmap v8564: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 180 KiB/s wr, 10 op/s 2023-03-09T19:09:47.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:09:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15098.469053 secs 2023-03-09T19:09:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:09:47 smithi181 ceph-mon[94189]: pgmap v8564: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 180 KiB/s wr, 10 op/s 2023-03-09T19:09:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:09:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15098.469053 secs 2023-03-09T19:09:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:09:47 smithi038 ceph-mon[92422]: pgmap v8564: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 180 KiB/s wr, 10 op/s 2023-03-09T19:09:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:09:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15098.469053 secs 2023-03-09T19:09:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15560.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:09:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:09:48 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:09:48.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:09:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:09:48.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:09:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:09:48.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:09:48 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:09:48.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:09:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:09:48.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:09:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:09:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:09:48 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:09:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:09:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:09:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:09:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:09:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:09:49 smithi122 ceph-mon[93869]: pgmap v8565: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 194 KiB/s wr, 10 op/s 2023-03-09T19:09:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:09:49 smithi181 ceph-mon[94189]: pgmap v8565: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 194 KiB/s wr, 10 op/s 2023-03-09T19:09:50.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:09:49 smithi038 ceph-mon[92422]: pgmap v8565: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 194 KiB/s wr, 10 op/s 2023-03-09T19:09:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:09:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15101.179581 secs 2023-03-09T19:09:50.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:09:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15101.179581 secs 2023-03-09T19:09:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:09:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15101.179581 secs 2023-03-09T19:09:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:09:51 smithi122 ceph-mon[93869]: pgmap v8566: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 503 KiB/s rd, 374 KiB/s wr, 11 op/s 2023-03-09T19:09:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:09:51 smithi181 ceph-mon[94189]: pgmap v8566: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 503 KiB/s rd, 374 KiB/s wr, 11 op/s 2023-03-09T19:09:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:09:51 smithi038 ceph-mon[92422]: pgmap v8566: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 503 KiB/s rd, 374 KiB/s wr, 11 op/s 2023-03-09T19:09:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:09:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15103.469187 secs 2023-03-09T19:09:52.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:09:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15103.469187 secs 2023-03-09T19:09:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:09:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15103.469187 secs 2023-03-09T19:09:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15565.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:09:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:09:53 smithi038 ceph-mon[92422]: pgmap v8567: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 243 KiB/s wr, 12 op/s 2023-03-09T19:09:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:09:53 smithi122 ceph-mon[93869]: pgmap v8567: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 243 KiB/s wr, 12 op/s 2023-03-09T19:09:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:09:53 smithi181 ceph-mon[94189]: pgmap v8567: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 243 KiB/s wr, 12 op/s 2023-03-09T19:09:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:09:55 smithi038 ceph-mon[92422]: pgmap v8568: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 574 KiB/s rd, 343 KiB/s wr, 13 op/s 2023-03-09T19:09:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:09:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15106.179662 secs 2023-03-09T19:09:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:09:55 smithi122 ceph-mon[93869]: pgmap v8568: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 574 KiB/s rd, 343 KiB/s wr, 13 op/s 2023-03-09T19:09:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:09:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15106.179662 secs 2023-03-09T19:09:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:09:55 smithi181 ceph-mon[94189]: pgmap v8568: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 574 KiB/s rd, 343 KiB/s wr, 13 op/s 2023-03-09T19:09:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:09:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15106.179662 secs 2023-03-09T19:09:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:09:57 smithi038 ceph-mon[92422]: pgmap v8569: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 297 KiB/s wr, 12 op/s 2023-03-09T19:09:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:09:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15108.469246 secs 2023-03-09T19:09:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:09:57 smithi122 ceph-mon[93869]: pgmap v8569: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 297 KiB/s wr, 12 op/s 2023-03-09T19:09:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:09:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15108.469246 secs 2023-03-09T19:09:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:09:57 smithi181 ceph-mon[94189]: pgmap v8569: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 297 KiB/s wr, 12 op/s 2023-03-09T19:09:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:09:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15108.469246 secs 2023-03-09T19:09:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15570.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:09:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:09:59 smithi038 ceph-mon[92422]: pgmap v8570: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 362 KiB/s rd, 325 KiB/s wr, 11 op/s 2023-03-09T19:10:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:09:59 smithi122 ceph-mon[93869]: pgmap v8570: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 362 KiB/s rd, 325 KiB/s wr, 11 op/s 2023-03-09T19:10:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:09:59 smithi181 ceph-mon[94189]: pgmap v8570: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 362 KiB/s rd, 325 KiB/s wr, 11 op/s 2023-03-09T19:10:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:10:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15111.179743 secs 2023-03-09T19:10:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:10:00 smithi038 ceph-mon[92422]: overall HEALTH_WARN 2 MDSs report slow requests 2023-03-09T19:10:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:10:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15111.179743 secs 2023-03-09T19:10:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:10:00 smithi122 ceph-mon[93869]: overall HEALTH_WARN 2 MDSs report slow requests 2023-03-09T19:10:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:10:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15111.179743 secs 2023-03-09T19:10:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:10:00 smithi181 ceph-mon[94189]: overall HEALTH_WARN 2 MDSs report slow requests 2023-03-09T19:10:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:10:01 smithi038 ceph-mon[92422]: pgmap v8571: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 399 KiB/s rd, 361 KiB/s wr, 10 op/s 2023-03-09T19:10:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:10:01 smithi122 ceph-mon[93869]: pgmap v8571: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 399 KiB/s rd, 361 KiB/s wr, 10 op/s 2023-03-09T19:10:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:10:01 smithi181 ceph-mon[94189]: pgmap v8571: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 399 KiB/s rd, 361 KiB/s wr, 10 op/s 2023-03-09T19:10:02.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:10:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15113.469324 secs 2023-03-09T19:10:03.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:10:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15113.469324 secs 2023-03-09T19:10:03.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:10:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15113.469324 secs 2023-03-09T19:10:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15575.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:10:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:10:03 smithi122 ceph-mon[93869]: pgmap v8572: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 203 KiB/s wr, 13 op/s 2023-03-09T19:10:04.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:10:03 smithi181 ceph-mon[94189]: pgmap v8572: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 203 KiB/s wr, 13 op/s 2023-03-09T19:10:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:10:03 smithi038 ceph-mon[92422]: pgmap v8572: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 203 KiB/s wr, 13 op/s 2023-03-09T19:10:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:10:05 smithi122 ceph-mon[93869]: pgmap v8573: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 248 KiB/s wr, 24 op/s 2023-03-09T19:10:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:10:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15116.179800 secs 2023-03-09T19:10:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:10:05 smithi181 ceph-mon[94189]: pgmap v8573: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 248 KiB/s wr, 24 op/s 2023-03-09T19:10:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:10:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15116.179800 secs 2023-03-09T19:10:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:10:05 smithi038 ceph-mon[92422]: pgmap v8573: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 248 KiB/s wr, 24 op/s 2023-03-09T19:10:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:10:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15116.179800 secs 2023-03-09T19:10:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:10:07 smithi122 ceph-mon[93869]: pgmap v8574: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 147 KiB/s wr, 24 op/s 2023-03-09T19:10:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:10:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15118.469433 secs 2023-03-09T19:10:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:10:07 smithi181 ceph-mon[94189]: pgmap v8574: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 147 KiB/s wr, 24 op/s 2023-03-09T19:10:08.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:10:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15118.469433 secs 2023-03-09T19:10:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:10:07 smithi038 ceph-mon[92422]: pgmap v8574: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 147 KiB/s wr, 24 op/s 2023-03-09T19:10:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:10:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15118.469433 secs 2023-03-09T19:10:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15580.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:10:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:10:09 smithi122 ceph-mon[93869]: pgmap v8575: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 194 KiB/s wr, 24 op/s 2023-03-09T19:10:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:10:09 smithi181 ceph-mon[94189]: pgmap v8575: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 194 KiB/s wr, 24 op/s 2023-03-09T19:10:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:10:09 smithi038 ceph-mon[92422]: pgmap v8575: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 194 KiB/s wr, 24 op/s 2023-03-09T19:10:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:10:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15121.179858 secs 2023-03-09T19:10:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:10:10 smithi122 ceph-mon[93869]: osdmap e749: 12 total, 12 up, 12 in 2023-03-09T19:10:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:10:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15121.179858 secs 2023-03-09T19:10:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:10:10 smithi181 ceph-mon[94189]: osdmap e749: 12 total, 12 up, 12 in 2023-03-09T19:10:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:10:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15121.179858 secs 2023-03-09T19:10:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:10:10 smithi038 ceph-mon[92422]: osdmap e749: 12 total, 12 up, 12 in 2023-03-09T19:10:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:10:11 smithi122 ceph-mon[93869]: pgmap v8577: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 138 KiB/s wr, 26 op/s 2023-03-09T19:10:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:10:11 smithi181 ceph-mon[94189]: pgmap v8577: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 138 KiB/s wr, 26 op/s 2023-03-09T19:10:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:10:11 smithi038 ceph-mon[92422]: pgmap v8577: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 138 KiB/s wr, 26 op/s 2023-03-09T19:10:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:10:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15123.469544 secs 2023-03-09T19:10:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:10:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15123.469544 secs 2023-03-09T19:10:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:10:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15123.469544 secs 2023-03-09T19:10:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15585.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:10:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:10:13 smithi122 ceph-mon[93869]: pgmap v8578: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 111 KiB/s wr, 23 op/s 2023-03-09T19:10:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:10:13 smithi181 ceph-mon[94189]: pgmap v8578: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 111 KiB/s wr, 23 op/s 2023-03-09T19:10:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:10:13 smithi038 ceph-mon[92422]: pgmap v8578: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 111 KiB/s wr, 23 op/s 2023-03-09T19:10:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:10:15 smithi122 ceph-mon[93869]: pgmap v8579: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 7 op/s 2023-03-09T19:10:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:10:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15126.179920 secs 2023-03-09T19:10:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:10:15 smithi181 ceph-mon[94189]: pgmap v8579: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 7 op/s 2023-03-09T19:10:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:10:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15126.179920 secs 2023-03-09T19:10:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:10:15 smithi038 ceph-mon[92422]: pgmap v8579: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 7 op/s 2023-03-09T19:10:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:10:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15126.179920 secs 2023-03-09T19:10:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:10:17 smithi122 ceph-mon[93869]: pgmap v8580: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 7 op/s 2023-03-09T19:10:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:10:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15128.469657 secs 2023-03-09T19:10:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:10:17 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:10:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:10:17 smithi122 ceph-mon[93869]: osdmap e750: 12 total, 12 up, 12 in 2023-03-09T19:10:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:10:17 smithi181 ceph-mon[94189]: pgmap v8580: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 7 op/s 2023-03-09T19:10:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:10:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15128.469657 secs 2023-03-09T19:10:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:10:17 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:10:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:10:17 smithi181 ceph-mon[94189]: osdmap e750: 12 total, 12 up, 12 in 2023-03-09T19:10:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:10:17 smithi038 ceph-mon[92422]: pgmap v8580: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 7 op/s 2023-03-09T19:10:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:10:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15128.469657 secs 2023-03-09T19:10:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:10:17 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:10:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:10:17 smithi038 ceph-mon[92422]: osdmap e750: 12 total, 12 up, 12 in 2023-03-09T19:10:18.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15590.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:10:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:10:18 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:10:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:10:18 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:10:19.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:10:18 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:10:19.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:10:18 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:10:19.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:10:18 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:10:19.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:10:18 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:10:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:10:18 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:10:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:10:18 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:10:19.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:10:18 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:10:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:10:20 smithi122 ceph-mon[93869]: pgmap v8582: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 231 B/s wr, 7 op/s 2023-03-09T19:10:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:10:20 smithi181 ceph-mon[94189]: pgmap v8582: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 231 B/s wr, 7 op/s 2023-03-09T19:10:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:10:20 smithi038 ceph-mon[92422]: pgmap v8582: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 231 B/s wr, 7 op/s 2023-03-09T19:10:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:10:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15131.179991 secs 2023-03-09T19:10:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:10:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15131.179991 secs 2023-03-09T19:10:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:10:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15131.179991 secs 2023-03-09T19:10:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:10:22 smithi122 ceph-mon[93869]: pgmap v8583: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T19:10:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:10:22 smithi181 ceph-mon[94189]: pgmap v8583: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T19:10:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:10:22 smithi038 ceph-mon[92422]: pgmap v8583: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T19:10:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:10:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15133.469732 secs 2023-03-09T19:10:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:10:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15133.469732 secs 2023-03-09T19:10:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:10:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15133.469732 secs 2023-03-09T19:10:23.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15595.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:10:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:10:24 smithi122 ceph-mon[93869]: pgmap v8584: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T19:10:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:10:24 smithi181 ceph-mon[94189]: pgmap v8584: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T19:10:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:10:24 smithi038 ceph-mon[92422]: pgmap v8584: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T19:10:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:10:26 smithi122 ceph-mon[93869]: pgmap v8585: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 409 B/s wr, 6 op/s 2023-03-09T19:10:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:10:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15136.180053 secs 2023-03-09T19:10:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:10:26 smithi181 ceph-mon[94189]: pgmap v8585: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 409 B/s wr, 6 op/s 2023-03-09T19:10:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:10:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15136.180053 secs 2023-03-09T19:10:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:10:26 smithi038 ceph-mon[92422]: pgmap v8585: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 409 B/s wr, 6 op/s 2023-03-09T19:10:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:10:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15136.180053 secs 2023-03-09T19:10:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:10:28 smithi122 ceph-mon[93869]: pgmap v8586: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 614 B/s wr, 6 op/s 2023-03-09T19:10:28.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:10:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15138.469824 secs 2023-03-09T19:10:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:10:28 smithi181 ceph-mon[94189]: pgmap v8586: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 614 B/s wr, 6 op/s 2023-03-09T19:10:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:10:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15138.469824 secs 2023-03-09T19:10:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:10:28 smithi038 ceph-mon[92422]: pgmap v8586: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 614 B/s wr, 6 op/s 2023-03-09T19:10:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:10:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15138.469824 secs 2023-03-09T19:10:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15600.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:10:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:10:30 smithi122 ceph-mon[93869]: pgmap v8587: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 100 KiB/s wr, 7 op/s 2023-03-09T19:10:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:10:30 smithi181 ceph-mon[94189]: pgmap v8587: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 100 KiB/s wr, 7 op/s 2023-03-09T19:10:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:10:30 smithi038 ceph-mon[92422]: pgmap v8587: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 100 KiB/s wr, 7 op/s 2023-03-09T19:10:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:10:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15141.180116 secs 2023-03-09T19:10:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:10:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15141.180116 secs 2023-03-09T19:10:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:10:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15141.180116 secs 2023-03-09T19:10:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:10:32 smithi122 ceph-mon[93869]: pgmap v8588: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 92 KiB/s wr, 6 op/s 2023-03-09T19:10:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:10:32 smithi181 ceph-mon[94189]: pgmap v8588: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 92 KiB/s wr, 6 op/s 2023-03-09T19:10:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:10:32 smithi038 ceph-mon[92422]: pgmap v8588: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 92 KiB/s wr, 6 op/s 2023-03-09T19:10:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:10:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15143.469962 secs 2023-03-09T19:10:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:10:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15143.469962 secs 2023-03-09T19:10:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:10:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15143.469962 secs 2023-03-09T19:10:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15605.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:10:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:10:34 smithi122 ceph-mon[93869]: pgmap v8589: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 92 KiB/s wr, 8 op/s 2023-03-09T19:10:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:10:34 smithi181 ceph-mon[94189]: pgmap v8589: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 92 KiB/s wr, 8 op/s 2023-03-09T19:10:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:10:34 smithi038 ceph-mon[92422]: pgmap v8589: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 92 KiB/s wr, 8 op/s 2023-03-09T19:10:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:10:36 smithi122 ceph-mon[93869]: pgmap v8590: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 92 KiB/s wr, 7 op/s 2023-03-09T19:10:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:10:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15146.180182 secs 2023-03-09T19:10:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:10:36 smithi181 ceph-mon[94189]: pgmap v8590: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 92 KiB/s wr, 7 op/s 2023-03-09T19:10:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:10:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15146.180182 secs 2023-03-09T19:10:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:10:36 smithi038 ceph-mon[92422]: pgmap v8590: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 92 KiB/s wr, 7 op/s 2023-03-09T19:10:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:10:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15146.180182 secs 2023-03-09T19:10:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:10:38 smithi122 ceph-mon[93869]: pgmap v8591: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 92 KiB/s wr, 8 op/s 2023-03-09T19:10:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:10:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15148.470057 secs 2023-03-09T19:10:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:10:38 smithi181 ceph-mon[94189]: pgmap v8591: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 92 KiB/s wr, 8 op/s 2023-03-09T19:10:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:10:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15148.470057 secs 2023-03-09T19:10:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:10:38 smithi038 ceph-mon[92422]: pgmap v8591: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 92 KiB/s wr, 8 op/s 2023-03-09T19:10:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:10:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15148.470057 secs 2023-03-09T19:10:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15610.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:10:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:10:40 smithi122 ceph-mon[93869]: pgmap v8592: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 145 KiB/s wr, 7 op/s 2023-03-09T19:10:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:10:40 smithi181 ceph-mon[94189]: pgmap v8592: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 145 KiB/s wr, 7 op/s 2023-03-09T19:10:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:10:40 smithi038 ceph-mon[92422]: pgmap v8592: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 145 KiB/s wr, 7 op/s 2023-03-09T19:10:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:10:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15151.180266 secs 2023-03-09T19:10:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:10:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15151.180266 secs 2023-03-09T19:10:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:10:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15151.180266 secs 2023-03-09T19:10:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:10:42 smithi122 ceph-mon[93869]: pgmap v8593: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 54 KiB/s wr, 5 op/s 2023-03-09T19:10:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:10:42 smithi181 ceph-mon[94189]: pgmap v8593: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 54 KiB/s wr, 5 op/s 2023-03-09T19:10:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:10:42 smithi038 ceph-mon[92422]: pgmap v8593: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 54 KiB/s wr, 5 op/s 2023-03-09T19:10:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:10:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15153.470180 secs 2023-03-09T19:10:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:10:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15153.470180 secs 2023-03-09T19:10:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:10:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15153.470180 secs 2023-03-09T19:10:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15615.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:10:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:10:44 smithi122 ceph-mon[93869]: pgmap v8594: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 56 KiB/s wr, 8 op/s 2023-03-09T19:10:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:10:44 smithi181 ceph-mon[94189]: pgmap v8594: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 56 KiB/s wr, 8 op/s 2023-03-09T19:10:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:10:44 smithi038 ceph-mon[92422]: pgmap v8594: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 56 KiB/s wr, 8 op/s 2023-03-09T19:10:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:10:46 smithi122 ceph-mon[93869]: pgmap v8595: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 179 KiB/s wr, 8 op/s 2023-03-09T19:10:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:10:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15156.180329 secs 2023-03-09T19:10:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:10:46 smithi181 ceph-mon[94189]: pgmap v8595: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 179 KiB/s wr, 8 op/s 2023-03-09T19:10:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:10:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15156.180329 secs 2023-03-09T19:10:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:10:46 smithi038 ceph-mon[92422]: pgmap v8595: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 179 KiB/s wr, 8 op/s 2023-03-09T19:10:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:10:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15156.180329 secs 2023-03-09T19:10:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:10:48 smithi122 ceph-mon[93869]: pgmap v8596: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 300 KiB/s wr, 10 op/s 2023-03-09T19:10:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:10:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15158.470305 secs 2023-03-09T19:10:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:10:48 smithi181 ceph-mon[94189]: pgmap v8596: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 300 KiB/s wr, 10 op/s 2023-03-09T19:10:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:10:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15158.470305 secs 2023-03-09T19:10:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:10:48 smithi038 ceph-mon[92422]: pgmap v8596: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 300 KiB/s wr, 10 op/s 2023-03-09T19:10:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:10:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15158.470305 secs 2023-03-09T19:10:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15620.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:10:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:10:50 smithi122 ceph-mon[93869]: pgmap v8597: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 329 KiB/s rd, 315 KiB/s wr, 11 op/s 2023-03-09T19:10:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:10:50 smithi181 ceph-mon[94189]: pgmap v8597: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 329 KiB/s rd, 315 KiB/s wr, 11 op/s 2023-03-09T19:10:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:10:50 smithi038 ceph-mon[92422]: pgmap v8597: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 329 KiB/s rd, 315 KiB/s wr, 11 op/s 2023-03-09T19:10:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:10:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15161.180388 secs 2023-03-09T19:10:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:10:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15161.180388 secs 2023-03-09T19:10:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:10:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15161.180388 secs 2023-03-09T19:10:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:10:52 smithi122 ceph-mon[93869]: pgmap v8598: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 262 KiB/s wr, 9 op/s 2023-03-09T19:10:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:10:52 smithi181 ceph-mon[94189]: pgmap v8598: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 262 KiB/s wr, 9 op/s 2023-03-09T19:10:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:10:52 smithi038 ceph-mon[92422]: pgmap v8598: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 262 KiB/s wr, 9 op/s 2023-03-09T19:10:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15625.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:10:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:10:53 smithi122 ceph-mon[93869]: pgmap v8599: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 377 KiB/s wr, 13 op/s 2023-03-09T19:10:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:10:53 smithi181 ceph-mon[94189]: pgmap v8599: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 377 KiB/s wr, 13 op/s 2023-03-09T19:10:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:10:53 smithi038 ceph-mon[92422]: pgmap v8599: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 377 KiB/s wr, 13 op/s 2023-03-09T19:10:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:10:55 smithi122 ceph-mon[93869]: pgmap v8600: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 397 KiB/s wr, 12 op/s 2023-03-09T19:10:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:10:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15166.180450 secs 2023-03-09T19:10:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:10:55 smithi181 ceph-mon[94189]: pgmap v8600: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 397 KiB/s wr, 12 op/s 2023-03-09T19:10:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:10:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15166.180450 secs 2023-03-09T19:10:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:10:55 smithi038 ceph-mon[92422]: pgmap v8600: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 397 KiB/s wr, 12 op/s 2023-03-09T19:10:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:10:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15166.180450 secs 2023-03-09T19:10:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:10:56 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:10:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:10:56 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:10:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:10:56 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:10:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:10:57 smithi122 ceph-mon[93869]: pgmap v8601: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 365 KiB/s wr, 13 op/s 2023-03-09T19:10:57.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:10:57 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:10:57.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:10:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:10:57.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:10:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:10:57.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:10:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15168.470539 secs 2023-03-09T19:10:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:10:57 smithi181 ceph-mon[94189]: pgmap v8601: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 365 KiB/s wr, 13 op/s 2023-03-09T19:10:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:10:57 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:10:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:10:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:10:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:10:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:10:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:10:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15168.470539 secs 2023-03-09T19:10:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:10:57 smithi038 ceph-mon[92422]: pgmap v8601: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 365 KiB/s wr, 13 op/s 2023-03-09T19:10:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:10:57 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:10:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:10:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:10:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:10:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:10:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:10:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15168.470539 secs 2023-03-09T19:10:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15630.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:10:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:10:59 smithi122 ceph-mon[93869]: pgmap v8602: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 257 KiB/s wr, 11 op/s 2023-03-09T19:10:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:10:59 smithi181 ceph-mon[94189]: pgmap v8602: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 257 KiB/s wr, 11 op/s 2023-03-09T19:10:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:10:59 smithi038 ceph-mon[92422]: pgmap v8602: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 257 KiB/s wr, 11 op/s 2023-03-09T19:11:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:11:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15171.180510 secs 2023-03-09T19:11:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:11:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15171.180510 secs 2023-03-09T19:11:01.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:11:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15171.180510 secs 2023-03-09T19:11:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:11:01 smithi122 ceph-mon[93869]: pgmap v8603: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 242 KiB/s wr, 9 op/s 2023-03-09T19:11:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:11:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:11:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:11:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:11:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:11:01 smithi181 ceph-mon[94189]: pgmap v8603: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 242 KiB/s wr, 9 op/s 2023-03-09T19:11:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:11:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:11:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:11:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:11:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:11:01 smithi038 ceph-mon[92422]: pgmap v8603: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 242 KiB/s wr, 9 op/s 2023-03-09T19:11:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:11:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:11:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:11:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:11:02.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:11:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15173.470677 secs 2023-03-09T19:11:02.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:11:02 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:11:02.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:11:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15173.470677 secs 2023-03-09T19:11:02.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:11:02 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:11:02.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:11:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15173.470677 secs 2023-03-09T19:11:02.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:11:02 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:11:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15635.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:11:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:11:03 smithi122 ceph-mon[93869]: pgmap v8604: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 336 KiB/s rd, 355 KiB/s wr, 15 op/s 2023-03-09T19:11:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:11:03 smithi181 ceph-mon[94189]: pgmap v8604: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 336 KiB/s rd, 355 KiB/s wr, 15 op/s 2023-03-09T19:11:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:11:03 smithi038 ceph-mon[92422]: pgmap v8604: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 336 KiB/s rd, 355 KiB/s wr, 15 op/s 2023-03-09T19:11:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:11:05 smithi122 ceph-mon[93869]: pgmap v8605: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 262 KiB/s wr, 24 op/s 2023-03-09T19:11:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:11:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15176.180571 secs 2023-03-09T19:11:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:11:05 smithi181 ceph-mon[94189]: pgmap v8605: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 262 KiB/s wr, 24 op/s 2023-03-09T19:11:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:11:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15176.180571 secs 2023-03-09T19:11:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:11:05 smithi038 ceph-mon[92422]: pgmap v8605: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 262 KiB/s wr, 24 op/s 2023-03-09T19:11:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:11:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15176.180571 secs 2023-03-09T19:11:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:11:07 smithi122 ceph-mon[93869]: pgmap v8606: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 239 KiB/s wr, 25 op/s 2023-03-09T19:11:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:11:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15178.470797 secs 2023-03-09T19:11:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:11:07 smithi181 ceph-mon[94189]: pgmap v8606: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 239 KiB/s wr, 25 op/s 2023-03-09T19:11:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:11:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15178.470797 secs 2023-03-09T19:11:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:11:07 smithi038 ceph-mon[92422]: pgmap v8606: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 239 KiB/s wr, 25 op/s 2023-03-09T19:11:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:11:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15178.470797 secs 2023-03-09T19:11:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15640.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:11:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:11:09 smithi122 ceph-mon[93869]: pgmap v8607: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 195 KiB/s wr, 23 op/s 2023-03-09T19:11:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:11:09 smithi181 ceph-mon[94189]: pgmap v8607: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 195 KiB/s wr, 23 op/s 2023-03-09T19:11:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:11:09 smithi038 ceph-mon[92422]: pgmap v8607: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 195 KiB/s wr, 23 op/s 2023-03-09T19:11:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:11:10 smithi038 ceph-mon[92422]: osdmap e751: 12 total, 12 up, 12 in 2023-03-09T19:11:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:11:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15181.180635 secs 2023-03-09T19:11:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:11:10 smithi122 ceph-mon[93869]: osdmap e751: 12 total, 12 up, 12 in 2023-03-09T19:11:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:11:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15181.180635 secs 2023-03-09T19:11:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:11:10 smithi181 ceph-mon[94189]: osdmap e751: 12 total, 12 up, 12 in 2023-03-09T19:11:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:11:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15181.180635 secs 2023-03-09T19:11:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:11:11 smithi038 ceph-mon[92422]: pgmap v8609: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 219 KiB/s wr, 28 op/s 2023-03-09T19:11:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:11:11 smithi122 ceph-mon[93869]: pgmap v8609: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 219 KiB/s wr, 28 op/s 2023-03-09T19:11:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:11:11 smithi181 ceph-mon[94189]: pgmap v8609: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 219 KiB/s wr, 28 op/s 2023-03-09T19:11:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:11:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15183.470924 secs 2023-03-09T19:11:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:11:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15183.470924 secs 2023-03-09T19:11:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:11:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15183.470924 secs 2023-03-09T19:11:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15645.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:11:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:11:13 smithi038 ceph-mon[92422]: pgmap v8610: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 83 KiB/s wr, 22 op/s 2023-03-09T19:11:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:11:13 smithi122 ceph-mon[93869]: pgmap v8610: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 83 KiB/s wr, 22 op/s 2023-03-09T19:11:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:11:13 smithi181 ceph-mon[94189]: pgmap v8610: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 83 KiB/s wr, 22 op/s 2023-03-09T19:11:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:11:15 smithi122 ceph-mon[93869]: pgmap v8611: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 7 op/s 2023-03-09T19:11:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:11:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15186.180694 secs 2023-03-09T19:11:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:11:15 smithi181 ceph-mon[94189]: pgmap v8611: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 7 op/s 2023-03-09T19:11:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:11:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15186.180694 secs 2023-03-09T19:11:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:11:15 smithi038 ceph-mon[92422]: pgmap v8611: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 7 op/s 2023-03-09T19:11:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:11:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15186.180694 secs 2023-03-09T19:11:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:11:17 smithi122 ceph-mon[93869]: pgmap v8612: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 7 op/s 2023-03-09T19:11:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:11:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15188.471034 secs 2023-03-09T19:11:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:11:17 smithi122 ceph-mon[93869]: osdmap e752: 12 total, 12 up, 12 in 2023-03-09T19:11:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:11:17 smithi181 ceph-mon[94189]: pgmap v8612: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 7 op/s 2023-03-09T19:11:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:11:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15188.471034 secs 2023-03-09T19:11:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:11:17 smithi181 ceph-mon[94189]: osdmap e752: 12 total, 12 up, 12 in 2023-03-09T19:11:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:11:17 smithi038 ceph-mon[92422]: pgmap v8612: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 7 op/s 2023-03-09T19:11:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:11:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15188.471034 secs 2023-03-09T19:11:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:11:17 smithi038 ceph-mon[92422]: osdmap e752: 12 total, 12 up, 12 in 2023-03-09T19:11:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15650.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:11:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:11:18 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:11:19.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:11:18 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:11:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:11:18 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:11:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:11:19 smithi122 ceph-mon[93869]: pgmap v8614: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 47 KiB/s wr, 7 op/s 2023-03-09T19:11:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:11:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:11:20.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:11:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:11:20.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:11:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:11:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:11:19 smithi181 ceph-mon[94189]: pgmap v8614: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 47 KiB/s wr, 7 op/s 2023-03-09T19:11:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:11:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:11:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:11:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:11:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:11:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:11:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:11:19 smithi038 ceph-mon[92422]: pgmap v8614: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 47 KiB/s wr, 7 op/s 2023-03-09T19:11:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:11:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:11:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:11:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:11:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:11:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:11:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:11:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15191.180754 secs 2023-03-09T19:11:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:11:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15191.180754 secs 2023-03-09T19:11:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:11:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15191.180754 secs 2023-03-09T19:11:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:11:21 smithi122 ceph-mon[93869]: pgmap v8615: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 41 KiB/s wr, 6 op/s 2023-03-09T19:11:22.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:11:21 smithi181 ceph-mon[94189]: pgmap v8615: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 41 KiB/s wr, 6 op/s 2023-03-09T19:11:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:11:21 smithi038 ceph-mon[92422]: pgmap v8615: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 41 KiB/s wr, 6 op/s 2023-03-09T19:11:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:11:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15193.471149 secs 2023-03-09T19:11:23.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:11:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15193.471149 secs 2023-03-09T19:11:23.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:11:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15193.471149 secs 2023-03-09T19:11:23.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15655.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:11:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:11:23 smithi122 ceph-mon[93869]: pgmap v8616: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 41 KiB/s wr, 6 op/s 2023-03-09T19:11:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:11:23 smithi181 ceph-mon[94189]: pgmap v8616: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 41 KiB/s wr, 6 op/s 2023-03-09T19:11:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:11:23 smithi038 ceph-mon[92422]: pgmap v8616: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 41 KiB/s wr, 6 op/s 2023-03-09T19:11:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:11:25 smithi122 ceph-mon[93869]: pgmap v8617: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 6 op/s 2023-03-09T19:11:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:11:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15196.180817 secs 2023-03-09T19:11:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:11:25 smithi181 ceph-mon[94189]: pgmap v8617: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 6 op/s 2023-03-09T19:11:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:11:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15196.180817 secs 2023-03-09T19:11:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:11:25 smithi038 ceph-mon[92422]: pgmap v8617: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 6 op/s 2023-03-09T19:11:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:11:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15196.180817 secs 2023-03-09T19:11:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:11:27 smithi122 ceph-mon[93869]: pgmap v8618: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 6 op/s 2023-03-09T19:11:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:11:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15198.471216 secs 2023-03-09T19:11:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:11:27 smithi181 ceph-mon[94189]: pgmap v8618: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 6 op/s 2023-03-09T19:11:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:11:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15198.471216 secs 2023-03-09T19:11:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:11:27 smithi038 ceph-mon[92422]: pgmap v8618: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 42 KiB/s wr, 6 op/s 2023-03-09T19:11:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:11:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15198.471216 secs 2023-03-09T19:11:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15660.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:11:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:11:29 smithi122 ceph-mon[93869]: pgmap v8619: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 740 B/s wr, 6 op/s 2023-03-09T19:11:30.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:11:29 smithi181 ceph-mon[94189]: pgmap v8619: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 740 B/s wr, 6 op/s 2023-03-09T19:11:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:11:29 smithi038 ceph-mon[92422]: pgmap v8619: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 740 B/s wr, 6 op/s 2023-03-09T19:11:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:11:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15201.180875 secs 2023-03-09T19:11:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:11:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15201.180875 secs 2023-03-09T19:11:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:11:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15201.180875 secs 2023-03-09T19:11:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:11:31 smithi122 ceph-mon[93869]: pgmap v8620: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 767 B/s wr, 6 op/s 2023-03-09T19:11:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:11:31 smithi181 ceph-mon[94189]: pgmap v8620: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 767 B/s wr, 6 op/s 2023-03-09T19:11:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:11:31 smithi038 ceph-mon[92422]: pgmap v8620: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 767 B/s wr, 6 op/s 2023-03-09T19:11:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:11:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15203.471280 secs 2023-03-09T19:11:33.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:11:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15203.471280 secs 2023-03-09T19:11:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:11:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15203.471280 secs 2023-03-09T19:11:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15665.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:11:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:11:33 smithi122 ceph-mon[93869]: pgmap v8621: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 52 KiB/s wr, 7 op/s 2023-03-09T19:11:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:11:33 smithi181 ceph-mon[94189]: pgmap v8621: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 52 KiB/s wr, 7 op/s 2023-03-09T19:11:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:11:33 smithi038 ceph-mon[92422]: pgmap v8621: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 52 KiB/s wr, 7 op/s 2023-03-09T19:11:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:11:35 smithi122 ceph-mon[93869]: pgmap v8622: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 52 KiB/s wr, 7 op/s 2023-03-09T19:11:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:11:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15206.180934 secs 2023-03-09T19:11:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:11:35 smithi181 ceph-mon[94189]: pgmap v8622: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 52 KiB/s wr, 7 op/s 2023-03-09T19:11:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:11:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15206.180934 secs 2023-03-09T19:11:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:11:35 smithi038 ceph-mon[92422]: pgmap v8622: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 52 KiB/s wr, 7 op/s 2023-03-09T19:11:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:11:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15206.180934 secs 2023-03-09T19:11:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:11:37 smithi122 ceph-mon[93869]: pgmap v8623: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 52 KiB/s wr, 8 op/s 2023-03-09T19:11:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:11:37 smithi181 ceph-mon[94189]: pgmap v8623: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 52 KiB/s wr, 8 op/s 2023-03-09T19:11:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:11:37 smithi038 ceph-mon[92422]: pgmap v8623: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 52 KiB/s wr, 8 op/s 2023-03-09T19:11:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15670.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:11:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:11:39 smithi122 ceph-mon[93869]: pgmap v8624: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 105 KiB/s wr, 7 op/s 2023-03-09T19:11:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:11:39 smithi181 ceph-mon[94189]: pgmap v8624: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 105 KiB/s wr, 7 op/s 2023-03-09T19:11:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:11:39 smithi038 ceph-mon[92422]: pgmap v8624: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 105 KiB/s wr, 7 op/s 2023-03-09T19:11:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:11:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15211.180985 secs 2023-03-09T19:11:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:11:40 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:11:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:11:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15211.180985 secs 2023-03-09T19:11:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:11:40 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:11:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:11:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15211.180985 secs 2023-03-09T19:11:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:11:40 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:11:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:11:41 smithi122 ceph-mon[93869]: pgmap v8625: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 105 KiB/s wr, 6 op/s 2023-03-09T19:11:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:11:41 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:11:42.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:11:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:11:42.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:11:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:11:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:11:41 smithi181 ceph-mon[94189]: pgmap v8625: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 105 KiB/s wr, 6 op/s 2023-03-09T19:11:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:11:41 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:11:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:11:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:11:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:11:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:11:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:11:41 smithi038 ceph-mon[92422]: pgmap v8625: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 105 KiB/s wr, 6 op/s 2023-03-09T19:11:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:11:41 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:11:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:11:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:11:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:11:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:11:43.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15675.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:11:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:11:43 smithi122 ceph-mon[93869]: pgmap v8626: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 219 KiB/s wr, 8 op/s 2023-03-09T19:11:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:11:43 smithi181 ceph-mon[94189]: pgmap v8626: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 219 KiB/s wr, 8 op/s 2023-03-09T19:11:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:11:43 smithi038 ceph-mon[92422]: pgmap v8626: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 219 KiB/s wr, 8 op/s 2023-03-09T19:11:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:11:45 smithi122 ceph-mon[93869]: pgmap v8627: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 180 KiB/s wr, 7 op/s 2023-03-09T19:11:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:11:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15216.181066 secs 2023-03-09T19:11:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:11:45 smithi181 ceph-mon[94189]: pgmap v8627: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 180 KiB/s wr, 7 op/s 2023-03-09T19:11:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:11:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15216.181066 secs 2023-03-09T19:11:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:11:45 smithi038 ceph-mon[92422]: pgmap v8627: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 180 KiB/s wr, 7 op/s 2023-03-09T19:11:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:11:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15216.181066 secs 2023-03-09T19:11:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:11:47 smithi122 ceph-mon[93869]: pgmap v8628: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 181 KiB/s wr, 9 op/s 2023-03-09T19:11:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:11:47 smithi181 ceph-mon[94189]: pgmap v8628: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 181 KiB/s wr, 9 op/s 2023-03-09T19:11:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:11:47 smithi038 ceph-mon[92422]: pgmap v8628: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 181 KiB/s wr, 9 op/s 2023-03-09T19:11:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15680.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:11:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:11:49 smithi122 ceph-mon[93869]: pgmap v8629: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 316 KiB/s wr, 10 op/s 2023-03-09T19:11:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:11:49 smithi181 ceph-mon[94189]: pgmap v8629: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 316 KiB/s wr, 10 op/s 2023-03-09T19:11:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:11:49 smithi038 ceph-mon[92422]: pgmap v8629: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 316 KiB/s wr, 10 op/s 2023-03-09T19:11:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:11:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15221.181125 secs 2023-03-09T19:11:50.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:11:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15221.181125 secs 2023-03-09T19:11:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:11:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15221.181125 secs 2023-03-09T19:11:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:11:51 smithi122 ceph-mon[93869]: pgmap v8630: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 263 KiB/s wr, 10 op/s 2023-03-09T19:11:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:11:51 smithi181 ceph-mon[94189]: pgmap v8630: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 263 KiB/s wr, 10 op/s 2023-03-09T19:11:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:11:51 smithi038 ceph-mon[92422]: pgmap v8630: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 263 KiB/s wr, 10 op/s 2023-03-09T19:11:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:11:52 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 15223.471746 secs 2023-03-09T19:11:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:11:52 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 15223.471746 secs 2023-03-09T19:11:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:11:52 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 15223.471746 secs 2023-03-09T19:11:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15685.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:11:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:11:53 smithi038 ceph-mon[92422]: pgmap v8631: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 378 KiB/s wr, 12 op/s 2023-03-09T19:11:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:11:53 smithi122 ceph-mon[93869]: pgmap v8631: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 378 KiB/s wr, 12 op/s 2023-03-09T19:11:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:11:53 smithi181 ceph-mon[94189]: pgmap v8631: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 378 KiB/s wr, 12 op/s 2023-03-09T19:11:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:11:55 smithi038 ceph-mon[92422]: pgmap v8632: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 288 KiB/s wr, 11 op/s 2023-03-09T19:11:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:11:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15226.181183 secs 2023-03-09T19:11:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:11:55 smithi122 ceph-mon[93869]: pgmap v8632: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 288 KiB/s wr, 11 op/s 2023-03-09T19:11:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:11:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15226.181183 secs 2023-03-09T19:11:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:11:55 smithi181 ceph-mon[94189]: pgmap v8632: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 288 KiB/s wr, 11 op/s 2023-03-09T19:11:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:11:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15226.181183 secs 2023-03-09T19:11:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:11:56 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:11:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:11:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:11:57.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:11:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:11:57.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:11:56 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:11:57.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:11:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:11:57.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:11:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:11:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:11:56 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:11:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:11:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:11:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:11:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:11:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:11:57 smithi122 ceph-mon[93869]: pgmap v8633: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 285 KiB/s wr, 12 op/s 2023-03-09T19:11:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:11:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15228.471884 secs 2023-03-09T19:11:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:11:57 smithi181 ceph-mon[94189]: pgmap v8633: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 285 KiB/s wr, 12 op/s 2023-03-09T19:11:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:11:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15228.471884 secs 2023-03-09T19:11:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:11:57 smithi038 ceph-mon[92422]: pgmap v8633: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 285 KiB/s wr, 12 op/s 2023-03-09T19:11:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:11:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15228.471884 secs 2023-03-09T19:11:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15690.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:12:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:11:59 smithi122 ceph-mon[93869]: pgmap v8634: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 367 KiB/s wr, 12 op/s 2023-03-09T19:12:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:11:59 smithi181 ceph-mon[94189]: pgmap v8634: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 367 KiB/s wr, 12 op/s 2023-03-09T19:12:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:11:59 smithi038 ceph-mon[92422]: pgmap v8634: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 367 KiB/s wr, 12 op/s 2023-03-09T19:12:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:12:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15231.181242 secs 2023-03-09T19:12:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:12:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15231.181242 secs 2023-03-09T19:12:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:12:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15231.181242 secs 2023-03-09T19:12:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:12:01 smithi122 ceph-mon[93869]: pgmap v8635: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 322 KiB/s wr, 11 op/s 2023-03-09T19:12:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:12:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:12:02.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:12:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:12:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:12:01 smithi181 ceph-mon[94189]: pgmap v8635: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 322 KiB/s wr, 11 op/s 2023-03-09T19:12:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:12:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:12:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:12:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:12:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:12:01 smithi038 ceph-mon[92422]: pgmap v8635: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 322 KiB/s wr, 11 op/s 2023-03-09T19:12:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:12:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:12:02.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:12:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:12:03.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:12:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15233.472020 secs 2023-03-09T19:12:03.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:12:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15233.472020 secs 2023-03-09T19:12:03.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:12:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15233.472020 secs 2023-03-09T19:12:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15695.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:12:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:12:03 smithi122 ceph-mon[93869]: pgmap v8636: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 343 KiB/s wr, 14 op/s 2023-03-09T19:12:04.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:12:03 smithi181 ceph-mon[94189]: pgmap v8636: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 343 KiB/s wr, 14 op/s 2023-03-09T19:12:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:12:03 smithi038 ceph-mon[92422]: pgmap v8636: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 343 KiB/s wr, 14 op/s 2023-03-09T19:12:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:12:05 smithi122 ceph-mon[93869]: pgmap v8637: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 249 KiB/s wr, 24 op/s 2023-03-09T19:12:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:12:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15236.181300 secs 2023-03-09T19:12:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:12:05 smithi181 ceph-mon[94189]: pgmap v8637: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 249 KiB/s wr, 24 op/s 2023-03-09T19:12:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:12:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15236.181300 secs 2023-03-09T19:12:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:12:05 smithi038 ceph-mon[92422]: pgmap v8637: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 249 KiB/s wr, 24 op/s 2023-03-09T19:12:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:12:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15236.181300 secs 2023-03-09T19:12:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:12:07 smithi122 ceph-mon[93869]: pgmap v8638: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 275 KiB/s wr, 26 op/s 2023-03-09T19:12:08.238 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:12:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15238.472146 secs 2023-03-09T19:12:08.239 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:12:07 smithi181 ceph-mon[94189]: pgmap v8638: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 275 KiB/s wr, 26 op/s 2023-03-09T19:12:08.239 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:12:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15238.472146 secs 2023-03-09T19:12:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:12:07 smithi038 ceph-mon[92422]: pgmap v8638: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 275 KiB/s wr, 26 op/s 2023-03-09T19:12:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:12:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15238.472146 secs 2023-03-09T19:12:08.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15700.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:12:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:12:09 smithi122 ceph-mon[93869]: pgmap v8639: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 265 KiB/s wr, 24 op/s 2023-03-09T19:12:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:12:09 smithi181 ceph-mon[94189]: pgmap v8639: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 265 KiB/s wr, 24 op/s 2023-03-09T19:12:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:12:09 smithi038 ceph-mon[92422]: pgmap v8639: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 265 KiB/s wr, 24 op/s 2023-03-09T19:12:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:12:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15241.181372 secs 2023-03-09T19:12:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:12:10 smithi122 ceph-mon[93869]: osdmap e753: 12 total, 12 up, 12 in 2023-03-09T19:12:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:12:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15241.181372 secs 2023-03-09T19:12:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:12:10 smithi181 ceph-mon[94189]: osdmap e753: 12 total, 12 up, 12 in 2023-03-09T19:12:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:12:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15241.181372 secs 2023-03-09T19:12:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:12:10 smithi038 ceph-mon[92422]: osdmap e753: 12 total, 12 up, 12 in 2023-03-09T19:12:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:12:11 smithi122 ceph-mon[93869]: pgmap v8641: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 112 KiB/s wr, 26 op/s 2023-03-09T19:12:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:12:11 smithi181 ceph-mon[94189]: pgmap v8641: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 112 KiB/s wr, 26 op/s 2023-03-09T19:12:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:12:11 smithi038 ceph-mon[92422]: pgmap v8641: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 112 KiB/s wr, 26 op/s 2023-03-09T19:12:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:12:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15243.472210 secs 2023-03-09T19:12:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:12:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15243.472210 secs 2023-03-09T19:12:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:12:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15243.472210 secs 2023-03-09T19:12:13.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15705.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:12:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:12:13 smithi122 ceph-mon[93869]: pgmap v8642: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 85 KiB/s wr, 23 op/s 2023-03-09T19:12:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:12:13 smithi181 ceph-mon[94189]: pgmap v8642: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 85 KiB/s wr, 23 op/s 2023-03-09T19:12:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:12:13 smithi038 ceph-mon[92422]: pgmap v8642: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 85 KiB/s wr, 23 op/s 2023-03-09T19:12:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:12:15 smithi122 ceph-mon[93869]: pgmap v8643: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 59 KiB/s wr, 8 op/s 2023-03-09T19:12:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:12:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15246.181431 secs 2023-03-09T19:12:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:12:15 smithi181 ceph-mon[94189]: pgmap v8643: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 59 KiB/s wr, 8 op/s 2023-03-09T19:12:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:12:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15246.181431 secs 2023-03-09T19:12:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:12:15 smithi038 ceph-mon[92422]: pgmap v8643: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 59 KiB/s wr, 8 op/s 2023-03-09T19:12:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:12:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15246.181431 secs 2023-03-09T19:12:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:12:17 smithi122 ceph-mon[93869]: pgmap v8644: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 7 op/s 2023-03-09T19:12:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:12:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15248.472308 secs 2023-03-09T19:12:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:12:17 smithi122 ceph-mon[93869]: osdmap e754: 12 total, 12 up, 12 in 2023-03-09T19:12:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:12:17 smithi181 ceph-mon[94189]: pgmap v8644: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 7 op/s 2023-03-09T19:12:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:12:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15248.472308 secs 2023-03-09T19:12:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:12:17 smithi181 ceph-mon[94189]: osdmap e754: 12 total, 12 up, 12 in 2023-03-09T19:12:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:12:17 smithi038 ceph-mon[92422]: pgmap v8644: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 7 op/s 2023-03-09T19:12:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:12:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15248.472308 secs 2023-03-09T19:12:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:12:17 smithi038 ceph-mon[92422]: osdmap e754: 12 total, 12 up, 12 in 2023-03-09T19:12:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15710.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:12:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:12:19 smithi122 ceph-mon[93869]: pgmap v8646: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 232 B/s wr, 8 op/s 2023-03-09T19:12:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:12:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:12:20.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:12:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:12:20.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:12:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:12:20.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:12:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:12:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:12:19 smithi181 ceph-mon[94189]: pgmap v8646: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 232 B/s wr, 8 op/s 2023-03-09T19:12:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:12:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:12:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:12:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:12:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:12:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:12:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:12:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:12:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:12:19 smithi038 ceph-mon[92422]: pgmap v8646: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 232 B/s wr, 8 op/s 2023-03-09T19:12:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:12:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:12:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:12:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:12:20.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:12:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:12:20.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:12:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:12:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:12:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15251.181514 secs 2023-03-09T19:12:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:12:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15251.181514 secs 2023-03-09T19:12:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:12:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15251.181514 secs 2023-03-09T19:12:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:12:21 smithi122 ceph-mon[93869]: pgmap v8647: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 7 op/s 2023-03-09T19:12:22.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:12:21 smithi181 ceph-mon[94189]: pgmap v8647: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 7 op/s 2023-03-09T19:12:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:12:21 smithi038 ceph-mon[92422]: pgmap v8647: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 7 op/s 2023-03-09T19:12:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:12:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15253.472388 secs 2023-03-09T19:12:23.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:12:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15253.472388 secs 2023-03-09T19:12:23.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:12:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15253.472388 secs 2023-03-09T19:12:23.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15715.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:12:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:12:23 smithi122 ceph-mon[93869]: pgmap v8648: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 7 op/s 2023-03-09T19:12:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:12:23 smithi181 ceph-mon[94189]: pgmap v8648: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 7 op/s 2023-03-09T19:12:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:12:23 smithi038 ceph-mon[92422]: pgmap v8648: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 7 op/s 2023-03-09T19:12:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:12:25 smithi122 ceph-mon[93869]: pgmap v8649: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 7 op/s 2023-03-09T19:12:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:12:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15256.181607 secs 2023-03-09T19:12:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:12:25 smithi181 ceph-mon[94189]: pgmap v8649: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 7 op/s 2023-03-09T19:12:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:12:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15256.181607 secs 2023-03-09T19:12:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:12:25 smithi038 ceph-mon[92422]: pgmap v8649: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 7 op/s 2023-03-09T19:12:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:12:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15256.181607 secs 2023-03-09T19:12:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:12:27 smithi122 ceph-mon[93869]: pgmap v8650: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 7 op/s 2023-03-09T19:12:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:12:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15258.472507 secs 2023-03-09T19:12:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:12:27 smithi181 ceph-mon[94189]: pgmap v8650: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 7 op/s 2023-03-09T19:12:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:12:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15258.472507 secs 2023-03-09T19:12:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:12:27 smithi038 ceph-mon[92422]: pgmap v8650: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 7 op/s 2023-03-09T19:12:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:12:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15258.472507 secs 2023-03-09T19:12:28.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15720.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:12:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:12:29 smithi122 ceph-mon[93869]: pgmap v8651: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 45 KiB/s wr, 7 op/s 2023-03-09T19:12:30.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:12:29 smithi181 ceph-mon[94189]: pgmap v8651: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 45 KiB/s wr, 7 op/s 2023-03-09T19:12:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:12:29 smithi038 ceph-mon[92422]: pgmap v8651: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 45 KiB/s wr, 7 op/s 2023-03-09T19:12:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:12:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15261.181668 secs 2023-03-09T19:12:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:12:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15261.181668 secs 2023-03-09T19:12:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:12:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15261.181668 secs 2023-03-09T19:12:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:12:31 smithi122 ceph-mon[93869]: pgmap v8652: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 93 KiB/s wr, 7 op/s 2023-03-09T19:12:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:12:31 smithi181 ceph-mon[94189]: pgmap v8652: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 93 KiB/s wr, 7 op/s 2023-03-09T19:12:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:12:31 smithi038 ceph-mon[92422]: pgmap v8652: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 93 KiB/s wr, 7 op/s 2023-03-09T19:12:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:12:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15263.472623 secs 2023-03-09T19:12:33.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:12:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15263.472623 secs 2023-03-09T19:12:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:12:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15263.472623 secs 2023-03-09T19:12:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15725.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:12:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:12:33 smithi122 ceph-mon[93869]: pgmap v8653: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 93 KiB/s wr, 8 op/s 2023-03-09T19:12:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:12:33 smithi181 ceph-mon[94189]: pgmap v8653: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 93 KiB/s wr, 8 op/s 2023-03-09T19:12:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:12:33 smithi038 ceph-mon[92422]: pgmap v8653: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 93 KiB/s wr, 8 op/s 2023-03-09T19:12:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:12:35 smithi122 ceph-mon[93869]: pgmap v8654: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 93 KiB/s wr, 7 op/s 2023-03-09T19:12:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:12:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15266.181725 secs 2023-03-09T19:12:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:12:35 smithi181 ceph-mon[94189]: pgmap v8654: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 93 KiB/s wr, 7 op/s 2023-03-09T19:12:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:12:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15266.181725 secs 2023-03-09T19:12:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:12:35 smithi038 ceph-mon[92422]: pgmap v8654: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 93 KiB/s wr, 7 op/s 2023-03-09T19:12:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:12:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15266.181725 secs 2023-03-09T19:12:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:12:37 smithi122 ceph-mon[93869]: pgmap v8655: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 147 KiB/s wr, 9 op/s 2023-03-09T19:12:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:12:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15268.472738 secs 2023-03-09T19:12:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:12:37 smithi181 ceph-mon[94189]: pgmap v8655: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 147 KiB/s wr, 9 op/s 2023-03-09T19:12:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:12:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15268.472738 secs 2023-03-09T19:12:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:12:37 smithi038 ceph-mon[92422]: pgmap v8655: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 147 KiB/s wr, 9 op/s 2023-03-09T19:12:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:12:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15268.472738 secs 2023-03-09T19:12:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15730.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:12:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:12:39 smithi122 ceph-mon[93869]: pgmap v8656: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 106 KiB/s wr, 6 op/s 2023-03-09T19:12:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:12:39 smithi181 ceph-mon[94189]: pgmap v8656: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 106 KiB/s wr, 6 op/s 2023-03-09T19:12:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:12:39 smithi038 ceph-mon[92422]: pgmap v8656: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 106 KiB/s wr, 6 op/s 2023-03-09T19:12:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:12:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15271.181785 secs 2023-03-09T19:12:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:12:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15271.181785 secs 2023-03-09T19:12:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:12:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15271.181785 secs 2023-03-09T19:12:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:12:41 smithi122 ceph-mon[93869]: pgmap v8657: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 156 KiB/s wr, 7 op/s 2023-03-09T19:12:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:12:41 smithi181 ceph-mon[94189]: pgmap v8657: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 156 KiB/s wr, 7 op/s 2023-03-09T19:12:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:12:41 smithi038 ceph-mon[92422]: pgmap v8657: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 156 KiB/s wr, 7 op/s 2023-03-09T19:12:43.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:12:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15273.472858 secs 2023-03-09T19:12:43.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:12:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15273.472858 secs 2023-03-09T19:12:43.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:12:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15273.472858 secs 2023-03-09T19:12:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15735.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:12:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:12:43 smithi122 ceph-mon[93869]: pgmap v8658: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 142 KiB/s wr, 8 op/s 2023-03-09T19:12:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:12:43 smithi181 ceph-mon[94189]: pgmap v8658: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 142 KiB/s wr, 8 op/s 2023-03-09T19:12:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:12:43 smithi038 ceph-mon[92422]: pgmap v8658: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 142 KiB/s wr, 8 op/s 2023-03-09T19:12:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:12:45 smithi122 ceph-mon[93869]: pgmap v8659: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 142 KiB/s wr, 7 op/s 2023-03-09T19:12:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:12:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15276.181845 secs 2023-03-09T19:12:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:12:45 smithi181 ceph-mon[94189]: pgmap v8659: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 142 KiB/s wr, 7 op/s 2023-03-09T19:12:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:12:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15276.181845 secs 2023-03-09T19:12:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:12:45 smithi038 ceph-mon[92422]: pgmap v8659: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 142 KiB/s wr, 7 op/s 2023-03-09T19:12:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:12:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15276.181845 secs 2023-03-09T19:12:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:12:47 smithi122 ceph-mon[93869]: pgmap v8660: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 237 KiB/s wr, 11 op/s 2023-03-09T19:12:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:12:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15278.472968 secs 2023-03-09T19:12:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:12:47 smithi181 ceph-mon[94189]: pgmap v8660: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 237 KiB/s wr, 11 op/s 2023-03-09T19:12:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:12:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15278.472968 secs 2023-03-09T19:12:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:12:47 smithi038 ceph-mon[92422]: pgmap v8660: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 237 KiB/s wr, 11 op/s 2023-03-09T19:12:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:12:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15278.472968 secs 2023-03-09T19:12:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15740.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:12:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:12:49 smithi122 ceph-mon[93869]: pgmap v8661: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 183 KiB/s wr, 9 op/s 2023-03-09T19:12:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:12:49 smithi181 ceph-mon[94189]: pgmap v8661: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 183 KiB/s wr, 9 op/s 2023-03-09T19:12:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:12:49 smithi038 ceph-mon[92422]: pgmap v8661: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 183 KiB/s wr, 9 op/s 2023-03-09T19:12:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:12:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15281.181905 secs 2023-03-09T19:12:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:12:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15281.181905 secs 2023-03-09T19:12:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:12:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15281.181905 secs 2023-03-09T19:12:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:12:51 smithi122 ceph-mon[93869]: pgmap v8662: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 183 KiB/s wr, 11 op/s 2023-03-09T19:12:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:12:51 smithi181 ceph-mon[94189]: pgmap v8662: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 183 KiB/s wr, 11 op/s 2023-03-09T19:12:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:12:51 smithi038 ceph-mon[92422]: pgmap v8662: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 183 KiB/s wr, 11 op/s 2023-03-09T19:12:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:12:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15283.473085 secs 2023-03-09T19:12:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:12:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15283.473085 secs 2023-03-09T19:12:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:12:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15283.473085 secs 2023-03-09T19:12:53.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15745.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:12:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:12:53 smithi122 ceph-mon[93869]: pgmap v8663: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 275 KiB/s wr, 13 op/s 2023-03-09T19:12:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:12:53 smithi181 ceph-mon[94189]: pgmap v8663: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 275 KiB/s wr, 13 op/s 2023-03-09T19:12:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:12:53 smithi038 ceph-mon[92422]: pgmap v8663: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 275 KiB/s wr, 13 op/s 2023-03-09T19:12:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:12:55 smithi181 ceph-mon[94189]: pgmap v8664: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 238 KiB/s wr, 10 op/s 2023-03-09T19:12:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:12:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15286.181966 secs 2023-03-09T19:12:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:12:55 smithi038 ceph-mon[92422]: pgmap v8664: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 238 KiB/s wr, 10 op/s 2023-03-09T19:12:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:12:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15286.181966 secs 2023-03-09T19:12:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:12:55 smithi122 ceph-mon[93869]: pgmap v8664: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 238 KiB/s wr, 10 op/s 2023-03-09T19:12:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:12:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15286.181966 secs 2023-03-09T19:12:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:12:57 smithi038 ceph-mon[92422]: pgmap v8665: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 328 KiB/s wr, 14 op/s 2023-03-09T19:12:58.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:12:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15288.473195 secs 2023-03-09T19:12:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:12:57 smithi122 ceph-mon[93869]: pgmap v8665: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 328 KiB/s wr, 14 op/s 2023-03-09T19:12:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:12:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15288.473195 secs 2023-03-09T19:12:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:12:57 smithi181 ceph-mon[94189]: pgmap v8665: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 328 KiB/s wr, 14 op/s 2023-03-09T19:12:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:12:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15288.473195 secs 2023-03-09T19:12:58.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15750.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:13:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:12:59 smithi038 ceph-mon[92422]: pgmap v8666: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 233 KiB/s wr, 10 op/s 2023-03-09T19:13:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:12:59 smithi122 ceph-mon[93869]: pgmap v8666: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 233 KiB/s wr, 10 op/s 2023-03-09T19:13:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:12:59 smithi181 ceph-mon[94189]: pgmap v8666: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 233 KiB/s wr, 10 op/s 2023-03-09T19:13:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15291.182026 secs 2023-03-09T19:13:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15291.182026 secs 2023-03-09T19:13:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15291.182026 secs 2023-03-09T19:13:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:01 smithi038 ceph-mon[92422]: pgmap v8667: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 300 KiB/s wr, 13 op/s 2023-03-09T19:13:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:01 smithi122 ceph-mon[93869]: pgmap v8667: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 300 KiB/s wr, 13 op/s 2023-03-09T19:13:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:01 smithi181 ceph-mon[94189]: pgmap v8667: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 300 KiB/s wr, 13 op/s 2023-03-09T19:13:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15293.473311 secs 2023-03-09T19:13:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15293.473311 secs 2023-03-09T19:13:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15293.473311 secs 2023-03-09T19:13:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15755.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:13:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:04 smithi122 ceph-mon[93869]: pgmap v8668: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 353 KiB/s wr, 13 op/s 2023-03-09T19:13:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:04 smithi181 ceph-mon[94189]: pgmap v8668: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 353 KiB/s wr, 13 op/s 2023-03-09T19:13:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:04 smithi038 ceph-mon[92422]: pgmap v8668: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 353 KiB/s wr, 13 op/s 2023-03-09T19:13:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:06 smithi122 ceph-mon[93869]: pgmap v8669: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 233 KiB/s wr, 22 op/s 2023-03-09T19:13:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15296.182110 secs 2023-03-09T19:13:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:06 smithi181 ceph-mon[94189]: pgmap v8669: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 233 KiB/s wr, 22 op/s 2023-03-09T19:13:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15296.182110 secs 2023-03-09T19:13:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:06 smithi038 ceph-mon[92422]: pgmap v8669: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 233 KiB/s wr, 22 op/s 2023-03-09T19:13:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15296.182110 secs 2023-03-09T19:13:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:08 smithi122 ceph-mon[93869]: pgmap v8670: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 247 KiB/s rd, 280 KiB/s wr, 26 op/s 2023-03-09T19:13:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15298.473449 secs 2023-03-09T19:13:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:08 smithi181 ceph-mon[94189]: pgmap v8670: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 247 KiB/s rd, 280 KiB/s wr, 26 op/s 2023-03-09T19:13:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15298.473449 secs 2023-03-09T19:13:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:08 smithi038 ceph-mon[92422]: pgmap v8670: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 247 KiB/s rd, 280 KiB/s wr, 26 op/s 2023-03-09T19:13:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15298.473449 secs 2023-03-09T19:13:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15760.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:13:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:10 smithi122 ceph-mon[93869]: pgmap v8671: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 190 KiB/s wr, 23 op/s 2023-03-09T19:13:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:10 smithi122 ceph-mon[93869]: osdmap e755: 12 total, 12 up, 12 in 2023-03-09T19:13:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:10 smithi181 ceph-mon[94189]: pgmap v8671: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 190 KiB/s wr, 23 op/s 2023-03-09T19:13:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:10 smithi181 ceph-mon[94189]: osdmap e755: 12 total, 12 up, 12 in 2023-03-09T19:13:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:10 smithi038 ceph-mon[92422]: pgmap v8671: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 190 KiB/s wr, 23 op/s 2023-03-09T19:13:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:10 smithi038 ceph-mon[92422]: osdmap e755: 12 total, 12 up, 12 in 2023-03-09T19:13:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15301.182170 secs 2023-03-09T19:13:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15301.182170 secs 2023-03-09T19:13:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15301.182170 secs 2023-03-09T19:13:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:12 smithi122 ceph-mon[93869]: pgmap v8673: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 181 KiB/s wr, 26 op/s 2023-03-09T19:13:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:12 smithi181 ceph-mon[94189]: pgmap v8673: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 181 KiB/s wr, 26 op/s 2023-03-09T19:13:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:12 smithi038 ceph-mon[92422]: pgmap v8673: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 181 KiB/s wr, 26 op/s 2023-03-09T19:13:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15303.473561 secs 2023-03-09T19:13:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15303.473561 secs 2023-03-09T19:13:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15303.473561 secs 2023-03-09T19:13:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15765.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:13:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:14 smithi122 ceph-mon[93869]: pgmap v8674: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 117 KiB/s wr, 24 op/s 2023-03-09T19:13:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:14 smithi181 ceph-mon[94189]: pgmap v8674: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 117 KiB/s wr, 24 op/s 2023-03-09T19:13:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:14 smithi038 ceph-mon[92422]: pgmap v8674: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 117 KiB/s wr, 24 op/s 2023-03-09T19:13:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:16 smithi122 ceph-mon[93869]: pgmap v8675: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 91 KiB/s wr, 9 op/s 2023-03-09T19:13:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15306.182229 secs 2023-03-09T19:13:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:16 smithi181 ceph-mon[94189]: pgmap v8675: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 91 KiB/s wr, 9 op/s 2023-03-09T19:13:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15306.182229 secs 2023-03-09T19:13:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:16 smithi038 ceph-mon[92422]: pgmap v8675: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 91 KiB/s wr, 9 op/s 2023-03-09T19:13:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15306.182229 secs 2023-03-09T19:13:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:18 smithi122 ceph-mon[93869]: pgmap v8676: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 34 KiB/s wr, 7 op/s 2023-03-09T19:13:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15308.473693 secs 2023-03-09T19:13:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:18 smithi122 ceph-mon[93869]: osdmap e756: 12 total, 12 up, 12 in 2023-03-09T19:13:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:18 smithi181 ceph-mon[94189]: pgmap v8676: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 34 KiB/s wr, 7 op/s 2023-03-09T19:13:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15308.473693 secs 2023-03-09T19:13:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:18 smithi181 ceph-mon[94189]: osdmap e756: 12 total, 12 up, 12 in 2023-03-09T19:13:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:18 smithi038 ceph-mon[92422]: pgmap v8676: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 34 KiB/s wr, 7 op/s 2023-03-09T19:13:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15308.473693 secs 2023-03-09T19:13:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:18 smithi038 ceph-mon[92422]: osdmap e756: 12 total, 12 up, 12 in 2023-03-09T19:13:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15770.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:13:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:20 smithi122 ceph-mon[93869]: pgmap v8678: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 KiB/s wr, 7 op/s 2023-03-09T19:13:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:13:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:20 smithi181 ceph-mon[94189]: pgmap v8678: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 KiB/s wr, 7 op/s 2023-03-09T19:13:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:13:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:20 smithi038 ceph-mon[92422]: pgmap v8678: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 36 KiB/s wr, 7 op/s 2023-03-09T19:13:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:13:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15311.182336 secs 2023-03-09T19:13:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15311.182336 secs 2023-03-09T19:13:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15311.182336 secs 2023-03-09T19:13:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:22 smithi122 ceph-mon[93869]: pgmap v8679: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T19:13:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:22 smithi181 ceph-mon[94189]: pgmap v8679: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T19:13:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:22 smithi038 ceph-mon[92422]: pgmap v8679: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T19:13:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15313.473838 secs 2023-03-09T19:13:23.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:23 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:13:23.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:23 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:13:23.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:23 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:13:23.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:23 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:13:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15313.473838 secs 2023-03-09T19:13:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:23 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:13:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:23 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:13:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:23 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:13:23.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:23 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:13:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15313.473838 secs 2023-03-09T19:13:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:23 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:13:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:23 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:13:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:23 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:13:23.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:23 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:13:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15775.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:13:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:24 smithi122 ceph-mon[93869]: pgmap v8680: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T19:13:24.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:13:24.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:13:24.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:13:24.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:13:24.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:24 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T19:13:24.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:24 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T19:13:24.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:13:24.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:13:24.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:13:24.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:13:24.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:13:24.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:13:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:24 smithi181 ceph-mon[94189]: pgmap v8680: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T19:13:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:13:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:13:24.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:13:24.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:13:24.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:24 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T19:13:24.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:24 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T19:13:24.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:13:24.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:13:24.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:13:24.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:13:24.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:13:24.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:13:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:24 smithi038 ceph-mon[92422]: pgmap v8680: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T19:13:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:13:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:13:24.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:13:24.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:13:24.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:24 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T19:13:24.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:24 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T19:13:24.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:13:24.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:13:24.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:13:24.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:13:24.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:13:24.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:13:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:25 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi038 to 158.6M 2023-03-09T19:13:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:25 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi038 to 166343327: error parsing value: Value '166343327' is below minimum 939524096 2023-03-09T19:13:25.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:13:25.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:13:25.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:13:25.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:13:25.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:13:25.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:13:25.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:13:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:25 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi038 to 158.6M 2023-03-09T19:13:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:25 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi038 to 166343327: error parsing value: Value '166343327' is below minimum 939524096 2023-03-09T19:13:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:13:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:13:25.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:13:25.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:13:25.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:13:25.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:13:25.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:13:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:25 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi038 to 158.6M 2023-03-09T19:13:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:25 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi038 to 166343327: error parsing value: Value '166343327' is below minimum 939524096 2023-03-09T19:13:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:13:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:13:25.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:13:25.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:13:25.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:13:25.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:13:25.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:13:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:26 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T19:13:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:26 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T19:13:26.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:26 smithi122 ceph-mon[93869]: pgmap v8681: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T19:13:26.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15316.182396 secs 2023-03-09T19:13:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:26 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T19:13:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:26 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T19:13:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:26 smithi181 ceph-mon[94189]: pgmap v8681: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T19:13:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15316.182396 secs 2023-03-09T19:13:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:26 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T19:13:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:26 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T19:13:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:26 smithi038 ceph-mon[92422]: pgmap v8681: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T19:13:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15316.182396 secs 2023-03-09T19:13:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:28 smithi122 ceph-mon[93869]: pgmap v8682: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 409 B/s wr, 6 op/s 2023-03-09T19:13:28.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15318.473982 secs 2023-03-09T19:13:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:28 smithi181 ceph-mon[94189]: pgmap v8682: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 409 B/s wr, 6 op/s 2023-03-09T19:13:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15318.473982 secs 2023-03-09T19:13:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:28 smithi038 ceph-mon[92422]: pgmap v8682: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 409 B/s wr, 6 op/s 2023-03-09T19:13:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15318.473982 secs 2023-03-09T19:13:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15780.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:13:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:30 smithi122 ceph-mon[93869]: pgmap v8683: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 462 B/s wr, 6 op/s 2023-03-09T19:13:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:30 smithi181 ceph-mon[94189]: pgmap v8683: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 462 B/s wr, 6 op/s 2023-03-09T19:13:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:30 smithi038 ceph-mon[92422]: pgmap v8683: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 462 B/s wr, 6 op/s 2023-03-09T19:13:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15321.182456 secs 2023-03-09T19:13:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15321.182456 secs 2023-03-09T19:13:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15321.182456 secs 2023-03-09T19:13:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:32 smithi122 ceph-mon[93869]: pgmap v8684: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 52 KiB/s wr, 8 op/s 2023-03-09T19:13:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:32 smithi181 ceph-mon[94189]: pgmap v8684: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 52 KiB/s wr, 8 op/s 2023-03-09T19:13:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:32 smithi038 ceph-mon[92422]: pgmap v8684: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 52 KiB/s wr, 8 op/s 2023-03-09T19:13:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15323.474126 secs 2023-03-09T19:13:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15323.474126 secs 2023-03-09T19:13:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15323.474126 secs 2023-03-09T19:13:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15785.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:13:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:34 smithi122 ceph-mon[93869]: pgmap v8685: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 52 KiB/s wr, 6 op/s 2023-03-09T19:13:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:34 smithi181 ceph-mon[94189]: pgmap v8685: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 52 KiB/s wr, 6 op/s 2023-03-09T19:13:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:34 smithi038 ceph-mon[92422]: pgmap v8685: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 52 KiB/s wr, 6 op/s 2023-03-09T19:13:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:36 smithi122 ceph-mon[93869]: pgmap v8686: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 52 KiB/s wr, 6 op/s 2023-03-09T19:13:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15326.182513 secs 2023-03-09T19:13:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:36 smithi181 ceph-mon[94189]: pgmap v8686: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 52 KiB/s wr, 6 op/s 2023-03-09T19:13:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15326.182513 secs 2023-03-09T19:13:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:36 smithi038 ceph-mon[92422]: pgmap v8686: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 52 KiB/s wr, 6 op/s 2023-03-09T19:13:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15326.182513 secs 2023-03-09T19:13:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:38 smithi122 ceph-mon[93869]: pgmap v8687: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 107 KiB/s wr, 8 op/s 2023-03-09T19:13:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15328.474241 secs 2023-03-09T19:13:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:38 smithi181 ceph-mon[94189]: pgmap v8687: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 107 KiB/s wr, 8 op/s 2023-03-09T19:13:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15328.474241 secs 2023-03-09T19:13:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:38 smithi038 ceph-mon[92422]: pgmap v8687: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 107 KiB/s wr, 8 op/s 2023-03-09T19:13:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15328.474241 secs 2023-03-09T19:13:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15790.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:13:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:40 smithi122 ceph-mon[93869]: pgmap v8688: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 107 KiB/s wr, 6 op/s 2023-03-09T19:13:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:40 smithi181 ceph-mon[94189]: pgmap v8688: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 107 KiB/s wr, 6 op/s 2023-03-09T19:13:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:40 smithi038 ceph-mon[92422]: pgmap v8688: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 107 KiB/s wr, 6 op/s 2023-03-09T19:13:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15331.182575 secs 2023-03-09T19:13:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15331.182575 secs 2023-03-09T19:13:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15331.182575 secs 2023-03-09T19:13:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:42 smithi122 ceph-mon[93869]: pgmap v8689: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 236 KiB/s wr, 10 op/s 2023-03-09T19:13:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:42 smithi181 ceph-mon[94189]: pgmap v8689: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 236 KiB/s wr, 10 op/s 2023-03-09T19:13:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:42 smithi038 ceph-mon[92422]: pgmap v8689: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 236 KiB/s wr, 10 op/s 2023-03-09T19:13:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15795.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:13:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:43 smithi122 ceph-mon[93869]: pgmap v8690: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 184 KiB/s wr, 7 op/s 2023-03-09T19:13:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:43 smithi181 ceph-mon[94189]: pgmap v8690: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 184 KiB/s wr, 7 op/s 2023-03-09T19:13:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:43 smithi038 ceph-mon[92422]: pgmap v8690: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 184 KiB/s wr, 7 op/s 2023-03-09T19:13:44.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:44 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:13:44.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:44 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:13:44.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:44 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:13:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:45 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:13:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:45 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:13:45.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:45 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:13:45.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:45 smithi122 ceph-mon[93869]: pgmap v8691: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 199 KiB/s wr, 9 op/s 2023-03-09T19:13:45.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15336.182655 secs 2023-03-09T19:13:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:45 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:13:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:45 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:13:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:45 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:13:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:45 smithi181 ceph-mon[94189]: pgmap v8691: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 199 KiB/s wr, 9 op/s 2023-03-09T19:13:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15336.182655 secs 2023-03-09T19:13:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:45 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:13:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:45 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:13:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:45 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:13:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:45 smithi038 ceph-mon[92422]: pgmap v8691: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 199 KiB/s wr, 9 op/s 2023-03-09T19:13:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15336.182655 secs 2023-03-09T19:13:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:47 smithi122 ceph-mon[93869]: pgmap v8692: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 200 KiB/s wr, 12 op/s 2023-03-09T19:13:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:47 smithi181 ceph-mon[94189]: pgmap v8692: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 200 KiB/s wr, 12 op/s 2023-03-09T19:13:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:47 smithi038 ceph-mon[92422]: pgmap v8692: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 200 KiB/s wr, 12 op/s 2023-03-09T19:13:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15800.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:13:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:49 smithi122 ceph-mon[93869]: pgmap v8693: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 288 KiB/s wr, 10 op/s 2023-03-09T19:13:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:49 smithi181 ceph-mon[94189]: pgmap v8693: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 288 KiB/s wr, 10 op/s 2023-03-09T19:13:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:49 smithi038 ceph-mon[92422]: pgmap v8693: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 288 KiB/s wr, 10 op/s 2023-03-09T19:13:50.981 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15341.182715 secs 2023-03-09T19:13:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15341.182715 secs 2023-03-09T19:13:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15341.182715 secs 2023-03-09T19:13:52.051 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:51 smithi038 ceph-mon[92422]: pgmap v8694: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 302 KiB/s wr, 13 op/s 2023-03-09T19:13:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:51 smithi122 ceph-mon[93869]: pgmap v8694: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 302 KiB/s wr, 13 op/s 2023-03-09T19:13:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:51 smithi181 ceph-mon[94189]: pgmap v8694: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 302 KiB/s wr, 13 op/s 2023-03-09T19:13:53.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15805.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:13:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:53 smithi122 ceph-mon[93869]: pgmap v8695: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 173 KiB/s wr, 9 op/s 2023-03-09T19:13:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:53 smithi181 ceph-mon[94189]: pgmap v8695: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 173 KiB/s wr, 9 op/s 2023-03-09T19:13:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:53 smithi038 ceph-mon[92422]: pgmap v8695: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 173 KiB/s wr, 9 op/s 2023-03-09T19:13:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:55 smithi038 ceph-mon[92422]: pgmap v8696: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 227 KiB/s wr, 10 op/s 2023-03-09T19:13:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15346.182778 secs 2023-03-09T19:13:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:55 smithi122 ceph-mon[93869]: pgmap v8696: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 227 KiB/s wr, 10 op/s 2023-03-09T19:13:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15346.182778 secs 2023-03-09T19:13:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:55 smithi181 ceph-mon[94189]: pgmap v8696: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 227 KiB/s wr, 10 op/s 2023-03-09T19:13:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15346.182778 secs 2023-03-09T19:13:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:57 smithi038 ceph-mon[92422]: pgmap v8697: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 226 KiB/s wr, 12 op/s 2023-03-09T19:13:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15348.474717 secs 2023-03-09T19:13:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:57 smithi122 ceph-mon[93869]: pgmap v8697: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 226 KiB/s wr, 12 op/s 2023-03-09T19:13:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15348.474717 secs 2023-03-09T19:13:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:57 smithi181 ceph-mon[94189]: pgmap v8697: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 226 KiB/s wr, 12 op/s 2023-03-09T19:13:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15348.474717 secs 2023-03-09T19:13:58.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15810.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:14:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:13:59 smithi122 ceph-mon[93869]: pgmap v8698: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 357 KiB/s wr, 11 op/s 2023-03-09T19:14:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:13:59 smithi181 ceph-mon[94189]: pgmap v8698: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 357 KiB/s wr, 11 op/s 2023-03-09T19:14:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:13:59 smithi038 ceph-mon[92422]: pgmap v8698: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 357 KiB/s wr, 11 op/s 2023-03-09T19:14:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:14:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15351.182840 secs 2023-03-09T19:14:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:14:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15351.182840 secs 2023-03-09T19:14:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:14:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15351.182840 secs 2023-03-09T19:14:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:14:01 smithi122 ceph-mon[93869]: pgmap v8699: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 236 KiB/s wr, 12 op/s 2023-03-09T19:14:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:14:01 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:14:02.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:14:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:14:02.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:14:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:14:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:14:01 smithi181 ceph-mon[94189]: pgmap v8699: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 236 KiB/s wr, 12 op/s 2023-03-09T19:14:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:14:01 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:14:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:14:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:14:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:14:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:14:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:14:01 smithi038 ceph-mon[92422]: pgmap v8699: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 236 KiB/s wr, 12 op/s 2023-03-09T19:14:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:14:01 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:14:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:14:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:14:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:14:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:14:03.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:14:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15353.474811 secs 2023-03-09T19:14:03.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:14:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15353.474811 secs 2023-03-09T19:14:03.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:14:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15353.474811 secs 2023-03-09T19:14:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15815.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:14:04.108 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:14:03 smithi038 ceph-mon[92422]: pgmap v8700: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 222 KiB/s wr, 9 op/s 2023-03-09T19:14:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:14:03 smithi122 ceph-mon[93869]: pgmap v8700: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 222 KiB/s wr, 9 op/s 2023-03-09T19:14:04.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:14:03 smithi181 ceph-mon[94189]: pgmap v8700: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 222 KiB/s wr, 9 op/s 2023-03-09T19:14:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:14:05 smithi122 ceph-mon[93869]: pgmap v8701: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 325 KiB/s wr, 23 op/s 2023-03-09T19:14:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:14:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15356.182899 secs 2023-03-09T19:14:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:14:05 smithi122 ceph-mon[93869]: osdmap e757: 12 total, 12 up, 12 in 2023-03-09T19:14:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:14:05 smithi181 ceph-mon[94189]: pgmap v8701: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 325 KiB/s wr, 23 op/s 2023-03-09T19:14:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:14:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15356.182899 secs 2023-03-09T19:14:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:14:05 smithi181 ceph-mon[94189]: osdmap e757: 12 total, 12 up, 12 in 2023-03-09T19:14:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:14:05 smithi038 ceph-mon[92422]: pgmap v8701: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 325 KiB/s wr, 23 op/s 2023-03-09T19:14:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:14:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15356.182899 secs 2023-03-09T19:14:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:14:05 smithi038 ceph-mon[92422]: osdmap e757: 12 total, 12 up, 12 in 2023-03-09T19:14:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:14:07 smithi122 ceph-mon[93869]: pgmap v8703: 129 pgs: 1 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 120 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 347 KiB/s wr, 27 op/s 2023-03-09T19:14:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:14:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15358.474884 secs 2023-03-09T19:14:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:14:07 smithi181 ceph-mon[94189]: pgmap v8703: 129 pgs: 1 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 120 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 347 KiB/s wr, 27 op/s 2023-03-09T19:14:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:14:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15358.474884 secs 2023-03-09T19:14:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:14:07 smithi038 ceph-mon[92422]: pgmap v8703: 129 pgs: 1 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 120 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 347 KiB/s wr, 27 op/s 2023-03-09T19:14:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:14:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15358.474884 secs 2023-03-09T19:14:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15820.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:14:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:14:09 smithi122 ceph-mon[93869]: pgmap v8704: 129 pgs: 1 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 120 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 190 KiB/s wr, 26 op/s 2023-03-09T19:14:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:14:09 smithi181 ceph-mon[94189]: pgmap v8704: 129 pgs: 1 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 120 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 190 KiB/s wr, 26 op/s 2023-03-09T19:14:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:14:09 smithi038 ceph-mon[92422]: pgmap v8704: 129 pgs: 1 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 120 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 190 KiB/s wr, 26 op/s 2023-03-09T19:14:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:14:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15361.182958 secs 2023-03-09T19:14:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:14:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15361.182958 secs 2023-03-09T19:14:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:14:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15361.182958 secs 2023-03-09T19:14:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:14:11 smithi122 ceph-mon[93869]: pgmap v8705: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 285 KiB/s wr, 26 op/s 2023-03-09T19:14:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:14:11 smithi181 ceph-mon[94189]: pgmap v8705: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 285 KiB/s wr, 26 op/s 2023-03-09T19:14:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:14:11 smithi038 ceph-mon[92422]: pgmap v8705: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 285 KiB/s wr, 26 op/s 2023-03-09T19:14:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:14:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15363.474985 secs 2023-03-09T19:14:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:14:12 smithi122 ceph-mon[93869]: osdmap e758: 12 total, 12 up, 12 in 2023-03-09T19:14:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:14:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15363.474985 secs 2023-03-09T19:14:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:14:12 smithi181 ceph-mon[94189]: osdmap e758: 12 total, 12 up, 12 in 2023-03-09T19:14:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:14:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15363.474985 secs 2023-03-09T19:14:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:14:12 smithi038 ceph-mon[92422]: osdmap e758: 12 total, 12 up, 12 in 2023-03-09T19:14:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15825.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:14:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:14:13 smithi122 ceph-mon[93869]: pgmap v8707: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 201 KiB/s wr, 12 op/s 2023-03-09T19:14:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:14:13 smithi181 ceph-mon[94189]: pgmap v8707: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 201 KiB/s wr, 12 op/s 2023-03-09T19:14:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:14:13 smithi038 ceph-mon[92422]: pgmap v8707: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 201 KiB/s wr, 12 op/s 2023-03-09T19:14:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:14:15 smithi122 ceph-mon[93869]: pgmap v8708: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 162 KiB/s wr, 10 op/s 2023-03-09T19:14:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:14:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15366.182980 secs 2023-03-09T19:14:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:14:15 smithi181 ceph-mon[94189]: pgmap v8708: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 162 KiB/s wr, 10 op/s 2023-03-09T19:14:16.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:14:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15366.182980 secs 2023-03-09T19:14:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:14:15 smithi038 ceph-mon[92422]: pgmap v8708: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 162 KiB/s wr, 10 op/s 2023-03-09T19:14:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:14:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15366.182980 secs 2023-03-09T19:14:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:14:17 smithi122 ceph-mon[93869]: pgmap v8709: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 122 KiB/s wr, 8 op/s 2023-03-09T19:14:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:14:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15368.475112 secs 2023-03-09T19:14:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:14:17 smithi181 ceph-mon[94189]: pgmap v8709: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 122 KiB/s wr, 8 op/s 2023-03-09T19:14:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:14:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15368.475112 secs 2023-03-09T19:14:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:14:17 smithi038 ceph-mon[92422]: pgmap v8709: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 122 KiB/s wr, 8 op/s 2023-03-09T19:14:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:14:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15368.475112 secs 2023-03-09T19:14:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15830.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:14:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:14:19 smithi122 ceph-mon[93869]: pgmap v8710: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 121 KiB/s wr, 7 op/s 2023-03-09T19:14:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:14:19 smithi181 ceph-mon[94189]: pgmap v8710: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 121 KiB/s wr, 7 op/s 2023-03-09T19:14:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:14:19 smithi038 ceph-mon[92422]: pgmap v8710: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 121 KiB/s wr, 7 op/s 2023-03-09T19:14:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:14:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15371.183039 secs 2023-03-09T19:14:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:14:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15371.183039 secs 2023-03-09T19:14:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:14:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15371.183039 secs 2023-03-09T19:14:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:14:21 smithi122 ceph-mon[93869]: pgmap v8711: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 7 op/s 2023-03-09T19:14:22.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:14:21 smithi181 ceph-mon[94189]: pgmap v8711: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 7 op/s 2023-03-09T19:14:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:14:21 smithi038 ceph-mon[92422]: pgmap v8711: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 7 op/s 2023-03-09T19:14:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:14:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15373.475175 secs 2023-03-09T19:14:23.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:14:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15373.475175 secs 2023-03-09T19:14:23.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:14:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15373.475175 secs 2023-03-09T19:14:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15835.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:14:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:14:23 smithi122 ceph-mon[93869]: pgmap v8712: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 7 op/s 2023-03-09T19:14:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:14:23 smithi181 ceph-mon[94189]: pgmap v8712: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 7 op/s 2023-03-09T19:14:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:14:23 smithi038 ceph-mon[92422]: pgmap v8712: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 7 op/s 2023-03-09T19:14:25.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:14:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:14:25.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:14:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:14:25.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:14:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:14:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:14:25 smithi122 ceph-mon[93869]: pgmap v8713: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 101 KiB/s wr, 6 op/s 2023-03-09T19:14:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:14:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15376.183116 secs 2023-03-09T19:14:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:14:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:14:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:14:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:14:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:14:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:14:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:14:25 smithi181 ceph-mon[94189]: pgmap v8713: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 101 KiB/s wr, 6 op/s 2023-03-09T19:14:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:14:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15376.183116 secs 2023-03-09T19:14:26.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:14:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:14:26.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:14:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:14:26.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:14:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:14:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:14:25 smithi038 ceph-mon[92422]: pgmap v8713: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 101 KiB/s wr, 6 op/s 2023-03-09T19:14:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:14:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15376.183116 secs 2023-03-09T19:14:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:14:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:14:26.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:14:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:14:26.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:14:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:14:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:14:27 smithi122 ceph-mon[93869]: pgmap v8714: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 101 KiB/s wr, 8 op/s 2023-03-09T19:14:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:14:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15378.475292 secs 2023-03-09T19:14:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:14:27 smithi181 ceph-mon[94189]: pgmap v8714: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 101 KiB/s wr, 8 op/s 2023-03-09T19:14:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:14:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15378.475292 secs 2023-03-09T19:14:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:14:27 smithi038 ceph-mon[92422]: pgmap v8714: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 101 KiB/s wr, 8 op/s 2023-03-09T19:14:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:14:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15378.475292 secs 2023-03-09T19:14:28.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15840.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:14:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:14:29 smithi122 ceph-mon[93869]: pgmap v8715: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 100 KiB/s wr, 6 op/s 2023-03-09T19:14:30.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:14:29 smithi181 ceph-mon[94189]: pgmap v8715: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 100 KiB/s wr, 6 op/s 2023-03-09T19:14:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:14:29 smithi038 ceph-mon[92422]: pgmap v8715: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 100 KiB/s wr, 6 op/s 2023-03-09T19:14:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:14:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15381.183177 secs 2023-03-09T19:14:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:14:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15381.183177 secs 2023-03-09T19:14:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:14:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15381.183177 secs 2023-03-09T19:14:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:14:31 smithi122 ceph-mon[93869]: pgmap v8716: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 101 KiB/s wr, 8 op/s 2023-03-09T19:14:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:14:31 smithi181 ceph-mon[94189]: pgmap v8716: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 101 KiB/s wr, 8 op/s 2023-03-09T19:14:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:14:31 smithi038 ceph-mon[92422]: pgmap v8716: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 101 KiB/s wr, 8 op/s 2023-03-09T19:14:33.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:14:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15383.475353 secs 2023-03-09T19:14:33.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:14:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15383.475353 secs 2023-03-09T19:14:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:14:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15383.475353 secs 2023-03-09T19:14:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15845.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:14:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:14:33 smithi122 ceph-mon[93869]: pgmap v8717: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 52 KiB/s wr, 5 op/s 2023-03-09T19:14:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:14:33 smithi181 ceph-mon[94189]: pgmap v8717: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 52 KiB/s wr, 5 op/s 2023-03-09T19:14:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:14:33 smithi038 ceph-mon[92422]: pgmap v8717: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 52 KiB/s wr, 5 op/s 2023-03-09T19:14:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:14:35 smithi122 ceph-mon[93869]: pgmap v8718: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 91 KiB/s wr, 6 op/s 2023-03-09T19:14:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:14:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15386.183236 secs 2023-03-09T19:14:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:14:35 smithi181 ceph-mon[94189]: pgmap v8718: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 91 KiB/s wr, 6 op/s 2023-03-09T19:14:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:14:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15386.183236 secs 2023-03-09T19:14:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:14:35 smithi038 ceph-mon[92422]: pgmap v8718: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 91 KiB/s wr, 6 op/s 2023-03-09T19:14:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:14:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15386.183236 secs 2023-03-09T19:14:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:14:37 smithi122 ceph-mon[93869]: pgmap v8719: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 55 KiB/s wr, 8 op/s 2023-03-09T19:14:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:14:37 smithi181 ceph-mon[94189]: pgmap v8719: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 55 KiB/s wr, 8 op/s 2023-03-09T19:14:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:14:37 smithi038 ceph-mon[92422]: pgmap v8719: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 55 KiB/s wr, 8 op/s 2023-03-09T19:14:38.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15850.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:14:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:14:39 smithi122 ceph-mon[93869]: pgmap v8720: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 55 KiB/s wr, 6 op/s 2023-03-09T19:14:40.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:14:39 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:14:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:14:39 smithi181 ceph-mon[94189]: pgmap v8720: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 55 KiB/s wr, 6 op/s 2023-03-09T19:14:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:14:39 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:14:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:14:39 smithi038 ceph-mon[92422]: pgmap v8720: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 55 KiB/s wr, 6 op/s 2023-03-09T19:14:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:14:39 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:14:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:14:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15391.183313 secs 2023-03-09T19:14:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:14:40 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:14:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:14:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:14:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:14:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:14:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:14:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15391.183313 secs 2023-03-09T19:14:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:14:40 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:14:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:14:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:14:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:14:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:14:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:14:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15391.183313 secs 2023-03-09T19:14:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:14:40 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:14:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:14:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:14:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:14:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:14:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:14:41 smithi122 ceph-mon[93869]: pgmap v8721: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 200 KiB/s wr, 11 op/s 2023-03-09T19:14:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:14:41 smithi181 ceph-mon[94189]: pgmap v8721: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 200 KiB/s wr, 11 op/s 2023-03-09T19:14:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:14:41 smithi038 ceph-mon[92422]: pgmap v8721: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 200 KiB/s wr, 11 op/s 2023-03-09T19:14:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15855.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:14:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:14:43 smithi122 ceph-mon[93869]: pgmap v8722: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 200 KiB/s wr, 8 op/s 2023-03-09T19:14:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:14:43 smithi181 ceph-mon[94189]: pgmap v8722: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 200 KiB/s wr, 8 op/s 2023-03-09T19:14:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:14:43 smithi038 ceph-mon[92422]: pgmap v8722: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 200 KiB/s wr, 8 op/s 2023-03-09T19:14:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:14:45 smithi122 ceph-mon[93869]: pgmap v8723: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 201 KiB/s wr, 9 op/s 2023-03-09T19:14:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:14:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15396.183371 secs 2023-03-09T19:14:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:14:45 smithi181 ceph-mon[94189]: pgmap v8723: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 201 KiB/s wr, 9 op/s 2023-03-09T19:14:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:14:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15396.183371 secs 2023-03-09T19:14:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:14:45 smithi038 ceph-mon[92422]: pgmap v8723: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 201 KiB/s wr, 9 op/s 2023-03-09T19:14:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:14:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15396.183371 secs 2023-03-09T19:14:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:14:47 smithi122 ceph-mon[93869]: pgmap v8724: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 182 KiB/s wr, 12 op/s 2023-03-09T19:14:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:14:47 smithi181 ceph-mon[94189]: pgmap v8724: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 182 KiB/s wr, 12 op/s 2023-03-09T19:14:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:14:47 smithi038 ceph-mon[92422]: pgmap v8724: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 182 KiB/s wr, 12 op/s 2023-03-09T19:14:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15860.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:14:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:14:49 smithi122 ceph-mon[93869]: pgmap v8725: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 167 KiB/s wr, 10 op/s 2023-03-09T19:14:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:14:49 smithi181 ceph-mon[94189]: pgmap v8725: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 167 KiB/s wr, 10 op/s 2023-03-09T19:14:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:14:49 smithi038 ceph-mon[92422]: pgmap v8725: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 167 KiB/s wr, 10 op/s 2023-03-09T19:14:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:14:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15401.183432 secs 2023-03-09T19:14:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:14:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15401.183432 secs 2023-03-09T19:14:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:14:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15401.183432 secs 2023-03-09T19:14:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:14:51 smithi122 ceph-mon[93869]: pgmap v8726: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 366 KiB/s wr, 14 op/s 2023-03-09T19:14:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:14:51 smithi181 ceph-mon[94189]: pgmap v8726: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 366 KiB/s wr, 14 op/s 2023-03-09T19:14:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:14:51 smithi038 ceph-mon[92422]: pgmap v8726: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 366 KiB/s wr, 14 op/s 2023-03-09T19:14:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:14:52 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 15403.475838 secs 2023-03-09T19:14:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:14:52 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 15403.475838 secs 2023-03-09T19:14:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:14:52 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 15403.475838 secs 2023-03-09T19:14:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15865.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:14:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:14:53 smithi122 ceph-mon[93869]: pgmap v8727: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 221 KiB/s wr, 9 op/s 2023-03-09T19:14:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:14:53 smithi181 ceph-mon[94189]: pgmap v8727: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 221 KiB/s wr, 9 op/s 2023-03-09T19:14:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:14:53 smithi038 ceph-mon[92422]: pgmap v8727: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 221 KiB/s wr, 9 op/s 2023-03-09T19:14:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:14:55 smithi122 ceph-mon[93869]: pgmap v8728: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 221 KiB/s wr, 9 op/s 2023-03-09T19:14:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:14:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15406.183491 secs 2023-03-09T19:14:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:14:55 smithi181 ceph-mon[94189]: pgmap v8728: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 221 KiB/s wr, 9 op/s 2023-03-09T19:14:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:14:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15406.183491 secs 2023-03-09T19:14:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:14:55 smithi038 ceph-mon[92422]: pgmap v8728: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 221 KiB/s wr, 9 op/s 2023-03-09T19:14:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:14:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15406.183491 secs 2023-03-09T19:14:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:14:56 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:14:57.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:14:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:14:57.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:14:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:14:57.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:14:56 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:14:57.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:14:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:14:57.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:14:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:14:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:14:56 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:14:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:14:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:14:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:14:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:14:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:14:57 smithi122 ceph-mon[93869]: pgmap v8729: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 431 KiB/s wr, 13 op/s 2023-03-09T19:14:58.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:14:57 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 15408.475935 secs 2023-03-09T19:14:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:14:57 smithi181 ceph-mon[94189]: pgmap v8729: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 431 KiB/s wr, 13 op/s 2023-03-09T19:14:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:14:57 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 15408.475935 secs 2023-03-09T19:14:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:14:57 smithi038 ceph-mon[92422]: pgmap v8729: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 431 KiB/s wr, 13 op/s 2023-03-09T19:14:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:14:57 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 15408.475935 secs 2023-03-09T19:14:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15870.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:15:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:14:59 smithi122 ceph-mon[93869]: pgmap v8730: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 411 KiB/s wr, 10 op/s 2023-03-09T19:15:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:14:59 smithi181 ceph-mon[94189]: pgmap v8730: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 411 KiB/s wr, 10 op/s 2023-03-09T19:15:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:14:59 smithi038 ceph-mon[92422]: pgmap v8730: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 411 KiB/s wr, 10 op/s 2023-03-09T19:15:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:15:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15411.183549 secs 2023-03-09T19:15:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:15:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15411.183549 secs 2023-03-09T19:15:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:15:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15411.183549 secs 2023-03-09T19:15:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:15:01 smithi122 ceph-mon[93869]: pgmap v8731: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 490 KiB/s wr, 13 op/s 2023-03-09T19:15:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:15:01 smithi181 ceph-mon[94189]: pgmap v8731: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 490 KiB/s wr, 13 op/s 2023-03-09T19:15:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:15:01 smithi038 ceph-mon[92422]: pgmap v8731: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 490 KiB/s wr, 13 op/s 2023-03-09T19:15:03.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:15:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15413.476057 secs 2023-03-09T19:15:03.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:15:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15413.476057 secs 2023-03-09T19:15:03.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:15:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15413.476057 secs 2023-03-09T19:15:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15875.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:15:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:15:03 smithi122 ceph-mon[93869]: pgmap v8732: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 291 KiB/s wr, 9 op/s 2023-03-09T19:15:04.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:15:03 smithi181 ceph-mon[94189]: pgmap v8732: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 291 KiB/s wr, 9 op/s 2023-03-09T19:15:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:15:03 smithi038 ceph-mon[92422]: pgmap v8732: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 291 KiB/s wr, 9 op/s 2023-03-09T19:15:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:15:05 smithi122 ceph-mon[93869]: pgmap v8733: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 312 KiB/s wr, 21 op/s 2023-03-09T19:15:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:15:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15416.183610 secs 2023-03-09T19:15:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:15:05 smithi122 ceph-mon[93869]: osdmap e759: 12 total, 12 up, 12 in 2023-03-09T19:15:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:15:05 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:15:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:15:05 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:15:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:15:05 smithi181 ceph-mon[94189]: pgmap v8733: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 312 KiB/s wr, 21 op/s 2023-03-09T19:15:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:15:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15416.183610 secs 2023-03-09T19:15:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:15:05 smithi181 ceph-mon[94189]: osdmap e759: 12 total, 12 up, 12 in 2023-03-09T19:15:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:15:05 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:15:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:15:05 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:15:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:15:05 smithi038 ceph-mon[92422]: pgmap v8733: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 312 KiB/s wr, 21 op/s 2023-03-09T19:15:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:15:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15416.183610 secs 2023-03-09T19:15:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:15:05 smithi038 ceph-mon[92422]: osdmap e759: 12 total, 12 up, 12 in 2023-03-09T19:15:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:15:05 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:15:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:15:05 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:15:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:15:07 smithi122 ceph-mon[93869]: pgmap v8735: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 284 KiB/s wr, 28 op/s 2023-03-09T19:15:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:15:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15418.476166 secs 2023-03-09T19:15:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:15:07 smithi181 ceph-mon[94189]: pgmap v8735: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 284 KiB/s wr, 28 op/s 2023-03-09T19:15:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:15:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15418.476166 secs 2023-03-09T19:15:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:15:07 smithi038 ceph-mon[92422]: pgmap v8735: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 284 KiB/s wr, 28 op/s 2023-03-09T19:15:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:15:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15418.476166 secs 2023-03-09T19:15:08.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15880.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:15:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:15:09 smithi122 ceph-mon[93869]: pgmap v8736: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 284 KiB/s wr, 28 op/s 2023-03-09T19:15:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:15:09 smithi181 ceph-mon[94189]: pgmap v8736: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 284 KiB/s wr, 28 op/s 2023-03-09T19:15:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:15:09 smithi038 ceph-mon[92422]: pgmap v8736: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 284 KiB/s wr, 28 op/s 2023-03-09T19:15:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:15:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15421.183669 secs 2023-03-09T19:15:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:15:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15421.183669 secs 2023-03-09T19:15:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:15:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15421.183669 secs 2023-03-09T19:15:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:15:11 smithi122 ceph-mon[93869]: pgmap v8737: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 231 KiB/s wr, 28 op/s 2023-03-09T19:15:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:15:11 smithi181 ceph-mon[94189]: pgmap v8737: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 231 KiB/s wr, 28 op/s 2023-03-09T19:15:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:15:11 smithi038 ceph-mon[92422]: pgmap v8737: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 231 KiB/s wr, 28 op/s 2023-03-09T19:15:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:15:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15423.476305 secs 2023-03-09T19:15:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:15:12 smithi122 ceph-mon[93869]: osdmap e760: 12 total, 12 up, 12 in 2023-03-09T19:15:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:15:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15423.476305 secs 2023-03-09T19:15:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:15:12 smithi181 ceph-mon[94189]: osdmap e760: 12 total, 12 up, 12 in 2023-03-09T19:15:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:15:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15423.476305 secs 2023-03-09T19:15:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:15:12 smithi038 ceph-mon[92422]: osdmap e760: 12 total, 12 up, 12 in 2023-03-09T19:15:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15885.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:15:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:15:13 smithi122 ceph-mon[93869]: pgmap v8739: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 257 KiB/s wr, 16 op/s 2023-03-09T19:15:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:15:13 smithi181 ceph-mon[94189]: pgmap v8739: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 257 KiB/s wr, 16 op/s 2023-03-09T19:15:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:15:13 smithi038 ceph-mon[92422]: pgmap v8739: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 257 KiB/s wr, 16 op/s 2023-03-09T19:15:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:15:15 smithi122 ceph-mon[93869]: pgmap v8740: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 209 KiB/s wr, 13 op/s 2023-03-09T19:15:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:15:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15426.183736 secs 2023-03-09T19:15:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:15:15 smithi181 ceph-mon[94189]: pgmap v8740: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 209 KiB/s wr, 13 op/s 2023-03-09T19:15:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:15:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15426.183736 secs 2023-03-09T19:15:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:15:15 smithi038 ceph-mon[92422]: pgmap v8740: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 209 KiB/s wr, 13 op/s 2023-03-09T19:15:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:15:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15426.183736 secs 2023-03-09T19:15:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:15:17 smithi122 ceph-mon[93869]: pgmap v8741: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 102 KiB/s wr, 8 op/s 2023-03-09T19:15:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:15:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15428.476441 secs 2023-03-09T19:15:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:15:17 smithi181 ceph-mon[94189]: pgmap v8741: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 102 KiB/s wr, 8 op/s 2023-03-09T19:15:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:15:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15428.476441 secs 2023-03-09T19:15:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:15:17 smithi038 ceph-mon[92422]: pgmap v8741: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 102 KiB/s wr, 8 op/s 2023-03-09T19:15:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:15:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15428.476441 secs 2023-03-09T19:15:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15890.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:15:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:15:19 smithi122 ceph-mon[93869]: pgmap v8742: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 101 KiB/s wr, 8 op/s 2023-03-09T19:15:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:15:19 smithi181 ceph-mon[94189]: pgmap v8742: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 101 KiB/s wr, 8 op/s 2023-03-09T19:15:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:15:19 smithi038 ceph-mon[92422]: pgmap v8742: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 101 KiB/s wr, 8 op/s 2023-03-09T19:15:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:15:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15431.183841 secs 2023-03-09T19:15:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:15:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15431.183841 secs 2023-03-09T19:15:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:15:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15431.183841 secs 2023-03-09T19:15:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:15:21 smithi122 ceph-mon[93869]: pgmap v8743: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 86 KiB/s wr, 7 op/s 2023-03-09T19:15:22.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:15:21 smithi181 ceph-mon[94189]: pgmap v8743: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 86 KiB/s wr, 7 op/s 2023-03-09T19:15:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:15:21 smithi038 ceph-mon[92422]: pgmap v8743: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 86 KiB/s wr, 7 op/s 2023-03-09T19:15:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:15:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15433.476566 secs 2023-03-09T19:15:23.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:15:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15433.476566 secs 2023-03-09T19:15:23.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:15:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15433.476566 secs 2023-03-09T19:15:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15895.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:15:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:15:23 smithi122 ceph-mon[93869]: pgmap v8744: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 85 KiB/s wr, 7 op/s 2023-03-09T19:15:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:15:23 smithi181 ceph-mon[94189]: pgmap v8744: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 85 KiB/s wr, 7 op/s 2023-03-09T19:15:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:15:23 smithi038 ceph-mon[92422]: pgmap v8744: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 85 KiB/s wr, 7 op/s 2023-03-09T19:15:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:15:25 smithi122 ceph-mon[93869]: pgmap v8745: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 71 KiB/s wr, 6 op/s 2023-03-09T19:15:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:15:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15436.183918 secs 2023-03-09T19:15:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:15:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:15:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:15:25 smithi181 ceph-mon[94189]: pgmap v8745: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 71 KiB/s wr, 6 op/s 2023-03-09T19:15:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:15:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15436.183918 secs 2023-03-09T19:15:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:15:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:15:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:15:25 smithi038 ceph-mon[92422]: pgmap v8745: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 71 KiB/s wr, 6 op/s 2023-03-09T19:15:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:15:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15436.183918 secs 2023-03-09T19:15:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:15:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:15:27.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:15:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:15:27.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:15:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:15:27.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:15:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:15:27.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:15:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:15:27.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:15:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:15:27.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:15:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:15:27.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:15:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:15:27.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:15:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:15:27.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:15:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:15:27.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:15:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:15:27.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:15:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:15:27.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:15:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:15:27.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:15:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:15:27.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:15:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:15:27.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:15:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:15:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:15:28 smithi122 ceph-mon[93869]: pgmap v8746: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 71 KiB/s wr, 9 op/s 2023-03-09T19:15:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:15:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15438.476684 secs 2023-03-09T19:15:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:15:28 smithi181 ceph-mon[94189]: pgmap v8746: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 71 KiB/s wr, 9 op/s 2023-03-09T19:15:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:15:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15438.476684 secs 2023-03-09T19:15:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:15:28 smithi038 ceph-mon[92422]: pgmap v8746: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 71 KiB/s wr, 9 op/s 2023-03-09T19:15:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:15:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15438.476684 secs 2023-03-09T19:15:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15900.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:15:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:15:30 smithi122 ceph-mon[93869]: pgmap v8747: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 6 op/s 2023-03-09T19:15:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:15:30 smithi181 ceph-mon[94189]: pgmap v8747: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 6 op/s 2023-03-09T19:15:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:15:30 smithi038 ceph-mon[92422]: pgmap v8747: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 23 KiB/s wr, 6 op/s 2023-03-09T19:15:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:15:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15441.184000 secs 2023-03-09T19:15:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:15:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15441.184000 secs 2023-03-09T19:15:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:15:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15441.184000 secs 2023-03-09T19:15:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:15:32 smithi122 ceph-mon[93869]: pgmap v8748: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 23 KiB/s wr, 9 op/s 2023-03-09T19:15:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:15:32 smithi181 ceph-mon[94189]: pgmap v8748: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 23 KiB/s wr, 9 op/s 2023-03-09T19:15:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:15:32 smithi038 ceph-mon[92422]: pgmap v8748: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 23 KiB/s wr, 9 op/s 2023-03-09T19:15:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:15:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15443.476805 secs 2023-03-09T19:15:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:15:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15443.476805 secs 2023-03-09T19:15:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:15:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15443.476805 secs 2023-03-09T19:15:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15905.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:15:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:15:34 smithi122 ceph-mon[93869]: pgmap v8749: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T19:15:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:15:34 smithi181 ceph-mon[94189]: pgmap v8749: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T19:15:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:15:34 smithi038 ceph-mon[92422]: pgmap v8749: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T19:15:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:15:36 smithi122 ceph-mon[93869]: pgmap v8750: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T19:15:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:15:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15446.184073 secs 2023-03-09T19:15:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:15:36 smithi181 ceph-mon[94189]: pgmap v8750: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T19:15:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:15:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15446.184073 secs 2023-03-09T19:15:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:15:36 smithi038 ceph-mon[92422]: pgmap v8750: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T19:15:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:15:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15446.184073 secs 2023-03-09T19:15:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:15:38 smithi122 ceph-mon[93869]: pgmap v8751: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 108 KiB/s wr, 8 op/s 2023-03-09T19:15:38.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:15:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15448.476891 secs 2023-03-09T19:15:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:15:38 smithi181 ceph-mon[94189]: pgmap v8751: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 108 KiB/s wr, 8 op/s 2023-03-09T19:15:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:15:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15448.476891 secs 2023-03-09T19:15:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:15:38 smithi038 ceph-mon[92422]: pgmap v8751: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 108 KiB/s wr, 8 op/s 2023-03-09T19:15:38.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:15:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15448.476891 secs 2023-03-09T19:15:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15910.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:15:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:15:40 smithi122 ceph-mon[93869]: pgmap v8752: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 108 KiB/s wr, 6 op/s 2023-03-09T19:15:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:15:40 smithi181 ceph-mon[94189]: pgmap v8752: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 108 KiB/s wr, 6 op/s 2023-03-09T19:15:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:15:40 smithi038 ceph-mon[92422]: pgmap v8752: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 108 KiB/s wr, 6 op/s 2023-03-09T19:15:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:15:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15451.184135 secs 2023-03-09T19:15:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:15:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15451.184135 secs 2023-03-09T19:15:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:15:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15451.184135 secs 2023-03-09T19:15:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:15:42 smithi122 ceph-mon[93869]: pgmap v8753: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 237 KiB/s wr, 10 op/s 2023-03-09T19:15:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:15:42 smithi181 ceph-mon[94189]: pgmap v8753: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 237 KiB/s wr, 10 op/s 2023-03-09T19:15:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:15:42 smithi038 ceph-mon[92422]: pgmap v8753: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 237 KiB/s wr, 10 op/s 2023-03-09T19:15:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:15:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15453.477022 secs 2023-03-09T19:15:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:15:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15453.477022 secs 2023-03-09T19:15:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:15:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15453.477022 secs 2023-03-09T19:15:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15915.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:15:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:15:44 smithi122 ceph-mon[93869]: pgmap v8754: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 237 KiB/s wr, 7 op/s 2023-03-09T19:15:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:15:44 smithi181 ceph-mon[94189]: pgmap v8754: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 237 KiB/s wr, 7 op/s 2023-03-09T19:15:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:15:44 smithi038 ceph-mon[92422]: pgmap v8754: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 237 KiB/s wr, 7 op/s 2023-03-09T19:15:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:15:46 smithi122 ceph-mon[93869]: pgmap v8755: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 237 KiB/s wr, 7 op/s 2023-03-09T19:15:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:15:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15456.184197 secs 2023-03-09T19:15:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:15:46 smithi181 ceph-mon[94189]: pgmap v8755: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 237 KiB/s wr, 7 op/s 2023-03-09T19:15:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:15:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15456.184197 secs 2023-03-09T19:15:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:15:46 smithi038 ceph-mon[92422]: pgmap v8755: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 237 KiB/s wr, 7 op/s 2023-03-09T19:15:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:15:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15456.184197 secs 2023-03-09T19:15:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:15:48 smithi122 ceph-mon[93869]: pgmap v8756: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 376 KiB/s wr, 12 op/s 2023-03-09T19:15:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:15:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15458.477139 secs 2023-03-09T19:15:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:15:48 smithi181 ceph-mon[94189]: pgmap v8756: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 376 KiB/s wr, 12 op/s 2023-03-09T19:15:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:15:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15458.477139 secs 2023-03-09T19:15:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:15:48 smithi038 ceph-mon[92422]: pgmap v8756: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 376 KiB/s wr, 12 op/s 2023-03-09T19:15:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:15:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15458.477139 secs 2023-03-09T19:15:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15920.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:15:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:15:50 smithi122 ceph-mon[93869]: pgmap v8757: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 269 KiB/s wr, 10 op/s 2023-03-09T19:15:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:15:50 smithi181 ceph-mon[94189]: pgmap v8757: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 269 KiB/s wr, 10 op/s 2023-03-09T19:15:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:15:50 smithi038 ceph-mon[92422]: pgmap v8757: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 269 KiB/s wr, 10 op/s 2023-03-09T19:15:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:15:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15461.184260 secs 2023-03-09T19:15:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:15:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15461.184260 secs 2023-03-09T19:15:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:15:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15461.184260 secs 2023-03-09T19:15:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:15:52 smithi122 ceph-mon[93869]: pgmap v8758: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 400 KiB/s wr, 14 op/s 2023-03-09T19:15:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:15:52 smithi181 ceph-mon[94189]: pgmap v8758: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 400 KiB/s wr, 14 op/s 2023-03-09T19:15:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:15:52 smithi038 ceph-mon[92422]: pgmap v8758: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 400 KiB/s wr, 14 op/s 2023-03-09T19:15:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:15:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15463.477264 secs 2023-03-09T19:15:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:15:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15463.477264 secs 2023-03-09T19:15:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:15:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15463.477264 secs 2023-03-09T19:15:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15925.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:15:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:15:54 smithi122 ceph-mon[93869]: pgmap v8759: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 271 KiB/s wr, 10 op/s 2023-03-09T19:15:54.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:15:54 smithi181 ceph-mon[94189]: pgmap v8759: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 271 KiB/s wr, 10 op/s 2023-03-09T19:15:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:15:54 smithi038 ceph-mon[92422]: pgmap v8759: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 271 KiB/s wr, 10 op/s 2023-03-09T19:15:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:15:56 smithi122 ceph-mon[93869]: pgmap v8760: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 271 KiB/s wr, 10 op/s 2023-03-09T19:15:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:15:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15466.184338 secs 2023-03-09T19:15:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:15:56 smithi181 ceph-mon[94189]: pgmap v8760: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 271 KiB/s wr, 10 op/s 2023-03-09T19:15:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:15:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15466.184338 secs 2023-03-09T19:15:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:15:56 smithi038 ceph-mon[92422]: pgmap v8760: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 271 KiB/s wr, 10 op/s 2023-03-09T19:15:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:15:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15466.184338 secs 2023-03-09T19:15:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:15:58 smithi122 ceph-mon[93869]: pgmap v8761: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 371 KiB/s wr, 14 op/s 2023-03-09T19:15:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:15:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15468.477380 secs 2023-03-09T19:15:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:15:58 smithi181 ceph-mon[94189]: pgmap v8761: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 371 KiB/s wr, 14 op/s 2023-03-09T19:15:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:15:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15468.477380 secs 2023-03-09T19:15:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:15:58 smithi038 ceph-mon[92422]: pgmap v8761: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 371 KiB/s wr, 14 op/s 2023-03-09T19:15:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:15:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15468.477380 secs 2023-03-09T19:15:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15930.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:16:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:16:00 smithi122 ceph-mon[93869]: pgmap v8762: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 236 KiB/s wr, 10 op/s 2023-03-09T19:16:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:16:00 smithi181 ceph-mon[94189]: pgmap v8762: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 236 KiB/s wr, 10 op/s 2023-03-09T19:16:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:16:00 smithi038 ceph-mon[92422]: pgmap v8762: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 236 KiB/s wr, 10 op/s 2023-03-09T19:16:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:16:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15471.184397 secs 2023-03-09T19:16:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:16:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15471.184397 secs 2023-03-09T19:16:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:16:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15471.184397 secs 2023-03-09T19:16:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:16:02 smithi122 ceph-mon[93869]: pgmap v8763: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 258 KiB/s wr, 14 op/s 2023-03-09T19:16:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:16:02 smithi181 ceph-mon[94189]: pgmap v8763: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 258 KiB/s wr, 14 op/s 2023-03-09T19:16:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:16:02 smithi038 ceph-mon[92422]: pgmap v8763: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 258 KiB/s wr, 14 op/s 2023-03-09T19:16:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:16:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15473.477496 secs 2023-03-09T19:16:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:16:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15473.477496 secs 2023-03-09T19:16:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:16:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15473.477496 secs 2023-03-09T19:16:03.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15935.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:16:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:16:04 smithi122 ceph-mon[93869]: pgmap v8764: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 220 KiB/s wr, 11 op/s 2023-03-09T19:16:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:16:04 smithi181 ceph-mon[94189]: pgmap v8764: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 220 KiB/s wr, 11 op/s 2023-03-09T19:16:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:16:04 smithi038 ceph-mon[92422]: pgmap v8764: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 220 KiB/s wr, 11 op/s 2023-03-09T19:16:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:16:06 smithi122 ceph-mon[93869]: pgmap v8765: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 241 KiB/s wr, 23 op/s 2023-03-09T19:16:06.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:16:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15476.184458 secs 2023-03-09T19:16:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:16:06 smithi181 ceph-mon[94189]: pgmap v8765: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 241 KiB/s wr, 23 op/s 2023-03-09T19:16:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:16:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15476.184458 secs 2023-03-09T19:16:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:16:06 smithi038 ceph-mon[92422]: pgmap v8765: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 241 KiB/s wr, 23 op/s 2023-03-09T19:16:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:16:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15476.184458 secs 2023-03-09T19:16:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:16:08 smithi122 ceph-mon[93869]: pgmap v8766: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 242 KiB/s wr, 26 op/s 2023-03-09T19:16:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:16:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15478.477576 secs 2023-03-09T19:16:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:16:08 smithi181 ceph-mon[94189]: pgmap v8766: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 242 KiB/s wr, 26 op/s 2023-03-09T19:16:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:16:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15478.477576 secs 2023-03-09T19:16:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:16:08 smithi038 ceph-mon[92422]: pgmap v8766: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 242 KiB/s wr, 26 op/s 2023-03-09T19:16:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:16:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15478.477576 secs 2023-03-09T19:16:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15940.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:16:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:16:10 smithi122 ceph-mon[93869]: pgmap v8767: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 190 KiB/s wr, 23 op/s 2023-03-09T19:16:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:16:10 smithi122 ceph-mon[93869]: osdmap e761: 12 total, 12 up, 12 in 2023-03-09T19:16:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:16:10 smithi181 ceph-mon[94189]: pgmap v8767: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 190 KiB/s wr, 23 op/s 2023-03-09T19:16:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:16:10 smithi181 ceph-mon[94189]: osdmap e761: 12 total, 12 up, 12 in 2023-03-09T19:16:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:16:10 smithi038 ceph-mon[92422]: pgmap v8767: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 190 KiB/s wr, 23 op/s 2023-03-09T19:16:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:16:10 smithi038 ceph-mon[92422]: osdmap e761: 12 total, 12 up, 12 in 2023-03-09T19:16:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:16:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15481.184519 secs 2023-03-09T19:16:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:16:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15481.184519 secs 2023-03-09T19:16:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:16:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15481.184519 secs 2023-03-09T19:16:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:16:12 smithi122 ceph-mon[93869]: pgmap v8769: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 196 KiB/s wr, 23 op/s 2023-03-09T19:16:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:16:12 smithi181 ceph-mon[94189]: pgmap v8769: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 196 KiB/s wr, 23 op/s 2023-03-09T19:16:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:16:12 smithi038 ceph-mon[92422]: pgmap v8769: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 196 KiB/s wr, 23 op/s 2023-03-09T19:16:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:16:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15483.477650 secs 2023-03-09T19:16:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:16:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15483.477650 secs 2023-03-09T19:16:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:16:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15483.477650 secs 2023-03-09T19:16:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15945.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:16:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:16:14 smithi122 ceph-mon[93869]: pgmap v8770: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 85 KiB/s wr, 22 op/s 2023-03-09T19:16:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:16:14 smithi181 ceph-mon[94189]: pgmap v8770: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 85 KiB/s wr, 22 op/s 2023-03-09T19:16:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:16:14 smithi038 ceph-mon[92422]: pgmap v8770: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 85 KiB/s wr, 22 op/s 2023-03-09T19:16:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:16:16 smithi122 ceph-mon[93869]: pgmap v8771: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 59 KiB/s wr, 7 op/s 2023-03-09T19:16:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:16:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15486.184580 secs 2023-03-09T19:16:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:16:16 smithi181 ceph-mon[94189]: pgmap v8771: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 59 KiB/s wr, 7 op/s 2023-03-09T19:16:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:16:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15486.184580 secs 2023-03-09T19:16:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:16:16 smithi038 ceph-mon[92422]: pgmap v8771: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 59 KiB/s wr, 7 op/s 2023-03-09T19:16:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:16:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15486.184580 secs 2023-03-09T19:16:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:16:18 smithi122 ceph-mon[93869]: pgmap v8772: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 7 op/s 2023-03-09T19:16:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:16:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15488.477706 secs 2023-03-09T19:16:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:16:18 smithi122 ceph-mon[93869]: osdmap e762: 12 total, 12 up, 12 in 2023-03-09T19:16:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:16:18 smithi181 ceph-mon[94189]: pgmap v8772: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 7 op/s 2023-03-09T19:16:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:16:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15488.477706 secs 2023-03-09T19:16:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:16:18 smithi181 ceph-mon[94189]: osdmap e762: 12 total, 12 up, 12 in 2023-03-09T19:16:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:16:18 smithi038 ceph-mon[92422]: pgmap v8772: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 7 op/s 2023-03-09T19:16:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:16:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15488.477706 secs 2023-03-09T19:16:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:16:18 smithi038 ceph-mon[92422]: osdmap e762: 12 total, 12 up, 12 in 2023-03-09T19:16:18.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15950.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:16:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:16:20 smithi122 ceph-mon[93869]: pgmap v8774: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 320 B/s wr, 7 op/s 2023-03-09T19:16:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:16:20 smithi181 ceph-mon[94189]: pgmap v8774: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 320 B/s wr, 7 op/s 2023-03-09T19:16:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:16:20 smithi038 ceph-mon[92422]: pgmap v8774: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 320 B/s wr, 7 op/s 2023-03-09T19:16:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:16:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15491.184642 secs 2023-03-09T19:16:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:16:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15491.184642 secs 2023-03-09T19:16:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:16:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15491.184642 secs 2023-03-09T19:16:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:16:22 smithi122 ceph-mon[93869]: pgmap v8775: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T19:16:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:16:22 smithi181 ceph-mon[94189]: pgmap v8775: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T19:16:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:16:22 smithi038 ceph-mon[92422]: pgmap v8775: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T19:16:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:16:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15493.477824 secs 2023-03-09T19:16:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:16:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15493.477824 secs 2023-03-09T19:16:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:16:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15493.477824 secs 2023-03-09T19:16:23.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15955.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:16:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:16:24 smithi122 ceph-mon[93869]: pgmap v8776: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T19:16:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:16:24 smithi181 ceph-mon[94189]: pgmap v8776: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T19:16:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:16:24 smithi038 ceph-mon[92422]: pgmap v8776: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T19:16:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:16:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15496.184709 secs 2023-03-09T19:16:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:16:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15496.184709 secs 2023-03-09T19:16:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:16:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15496.184709 secs 2023-03-09T19:16:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:16:26 smithi122 ceph-mon[93869]: pgmap v8777: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T19:16:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:16:26 smithi181 ceph-mon[94189]: pgmap v8777: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T19:16:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:16:26 smithi038 ceph-mon[92422]: pgmap v8777: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T19:16:27.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:16:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:16:27.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:16:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:16:27.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:16:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:16:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:16:28 smithi122 ceph-mon[93869]: pgmap v8778: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T19:16:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:16:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15498.477907 secs 2023-03-09T19:16:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:16:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:16:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:16:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:16:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:16:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:16:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:16:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:16:28.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:16:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:16:28.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:16:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:16:28.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:16:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:16:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:16:28 smithi181 ceph-mon[94189]: pgmap v8778: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T19:16:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:16:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15498.477907 secs 2023-03-09T19:16:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:16:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:16:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:16:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:16:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:16:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:16:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:16:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:16:28.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:16:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:16:28.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:16:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:16:28.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:16:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:16:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:16:28 smithi038 ceph-mon[92422]: pgmap v8778: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T19:16:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:16:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15498.477907 secs 2023-03-09T19:16:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:16:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:16:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:16:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:16:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:16:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:16:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:16:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:16:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:16:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:16:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:16:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:16:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:16:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:16:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15960.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:16:29.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:16:29 smithi122 ceph-mon[93869]: pgmap v8779: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 46 KiB/s wr, 6 op/s 2023-03-09T19:16:29.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:16:29 smithi181 ceph-mon[94189]: pgmap v8779: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 46 KiB/s wr, 6 op/s 2023-03-09T19:16:29.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:16:29 smithi038 ceph-mon[92422]: pgmap v8779: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 46 KiB/s wr, 6 op/s 2023-03-09T19:16:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:16:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15501.184770 secs 2023-03-09T19:16:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:16:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15501.184770 secs 2023-03-09T19:16:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:16:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15501.184770 secs 2023-03-09T19:16:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:16:31 smithi122 ceph-mon[93869]: pgmap v8780: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 43 KiB/s wr, 8 op/s 2023-03-09T19:16:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:16:31 smithi181 ceph-mon[94189]: pgmap v8780: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 43 KiB/s wr, 8 op/s 2023-03-09T19:16:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:16:31 smithi038 ceph-mon[92422]: pgmap v8780: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 43 KiB/s wr, 8 op/s 2023-03-09T19:16:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:16:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15503.478023 secs 2023-03-09T19:16:32.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:16:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15503.478023 secs 2023-03-09T19:16:32.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:16:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15503.478023 secs 2023-03-09T19:16:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15965.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:16:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:16:33 smithi122 ceph-mon[93869]: pgmap v8781: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 92 KiB/s wr, 6 op/s 2023-03-09T19:16:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:16:33 smithi181 ceph-mon[94189]: pgmap v8781: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 92 KiB/s wr, 6 op/s 2023-03-09T19:16:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:16:33 smithi038 ceph-mon[92422]: pgmap v8781: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 92 KiB/s wr, 6 op/s 2023-03-09T19:16:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:16:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15506.184829 secs 2023-03-09T19:16:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:16:35 smithi122 ceph-mon[93869]: pgmap v8782: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 92 KiB/s wr, 6 op/s 2023-03-09T19:16:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:16:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15506.184829 secs 2023-03-09T19:16:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:16:35 smithi181 ceph-mon[94189]: pgmap v8782: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 92 KiB/s wr, 6 op/s 2023-03-09T19:16:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:16:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15506.184829 secs 2023-03-09T19:16:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:16:35 smithi038 ceph-mon[92422]: pgmap v8782: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 92 KiB/s wr, 6 op/s 2023-03-09T19:16:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:16:37 smithi122 ceph-mon[93869]: pgmap v8783: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 KiB/s wr, 8 op/s 2023-03-09T19:16:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:16:37 smithi181 ceph-mon[94189]: pgmap v8783: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 KiB/s wr, 8 op/s 2023-03-09T19:16:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:16:37 smithi038 ceph-mon[92422]: pgmap v8783: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 KiB/s wr, 8 op/s 2023-03-09T19:16:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15970.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:16:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:16:39 smithi122 ceph-mon[93869]: pgmap v8784: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 199 KiB/s wr, 8 op/s 2023-03-09T19:16:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:16:39 smithi181 ceph-mon[94189]: pgmap v8784: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 199 KiB/s wr, 8 op/s 2023-03-09T19:16:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:16:39 smithi038 ceph-mon[92422]: pgmap v8784: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 199 KiB/s wr, 8 op/s 2023-03-09T19:16:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:16:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15511.184891 secs 2023-03-09T19:16:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:16:40 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:16:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:16:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15511.184891 secs 2023-03-09T19:16:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:16:40 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:16:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:16:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15511.184891 secs 2023-03-09T19:16:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:16:40 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:16:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:16:41 smithi122 ceph-mon[93869]: pgmap v8785: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 170 KiB/s wr, 10 op/s 2023-03-09T19:16:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:16:41 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:16:42.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:16:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:16:42.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:16:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:16:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:16:41 smithi181 ceph-mon[94189]: pgmap v8785: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 170 KiB/s wr, 10 op/s 2023-03-09T19:16:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:16:41 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:16:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:16:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:16:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:16:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:16:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:16:41 smithi038 ceph-mon[92422]: pgmap v8785: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 170 KiB/s wr, 10 op/s 2023-03-09T19:16:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:16:41 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:16:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:16:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:16:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:16:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:16:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15975.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:16:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:16:43 smithi122 ceph-mon[93869]: pgmap v8786: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 170 KiB/s wr, 8 op/s 2023-03-09T19:16:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:16:43 smithi181 ceph-mon[94189]: pgmap v8786: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 170 KiB/s wr, 8 op/s 2023-03-09T19:16:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:16:43 smithi038 ceph-mon[92422]: pgmap v8786: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 170 KiB/s wr, 8 op/s 2023-03-09T19:16:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:16:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15516.184957 secs 2023-03-09T19:16:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:16:45 smithi122 ceph-mon[93869]: pgmap v8787: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 238 KiB/s wr, 8 op/s 2023-03-09T19:16:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:16:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15516.184957 secs 2023-03-09T19:16:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:16:45 smithi181 ceph-mon[94189]: pgmap v8787: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 238 KiB/s wr, 8 op/s 2023-03-09T19:16:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:16:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15516.184957 secs 2023-03-09T19:16:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:16:45 smithi038 ceph-mon[92422]: pgmap v8787: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 238 KiB/s wr, 8 op/s 2023-03-09T19:16:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:16:47 smithi122 ceph-mon[93869]: pgmap v8788: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 254 KiB/s wr, 11 op/s 2023-03-09T19:16:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:16:47 smithi181 ceph-mon[94189]: pgmap v8788: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 254 KiB/s wr, 11 op/s 2023-03-09T19:16:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:16:47 smithi038 ceph-mon[92422]: pgmap v8788: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 254 KiB/s wr, 11 op/s 2023-03-09T19:16:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15980.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:16:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:16:49 smithi122 ceph-mon[93869]: pgmap v8789: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 472 KiB/s rd, 454 KiB/s wr, 11 op/s 2023-03-09T19:16:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:16:49 smithi181 ceph-mon[94189]: pgmap v8789: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 472 KiB/s rd, 454 KiB/s wr, 11 op/s 2023-03-09T19:16:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:16:49 smithi038 ceph-mon[92422]: pgmap v8789: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 472 KiB/s rd, 454 KiB/s wr, 11 op/s 2023-03-09T19:16:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:16:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15521.184981 secs 2023-03-09T19:16:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:16:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15521.184981 secs 2023-03-09T19:16:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:16:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15521.184981 secs 2023-03-09T19:16:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:16:51 smithi122 ceph-mon[93869]: pgmap v8790: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 380 KiB/s wr, 13 op/s 2023-03-09T19:16:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:16:51 smithi181 ceph-mon[94189]: pgmap v8790: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 380 KiB/s wr, 13 op/s 2023-03-09T19:16:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:16:51 smithi038 ceph-mon[92422]: pgmap v8790: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 380 KiB/s wr, 13 op/s 2023-03-09T19:16:53.801 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15985.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:16:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:16:53 smithi122 ceph-mon[93869]: pgmap v8791: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 367 KiB/s wr, 10 op/s 2023-03-09T19:16:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:16:53 smithi181 ceph-mon[94189]: pgmap v8791: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 367 KiB/s wr, 10 op/s 2023-03-09T19:16:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:16:53 smithi038 ceph-mon[92422]: pgmap v8791: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 367 KiB/s wr, 10 op/s 2023-03-09T19:16:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:16:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15526.185041 secs 2023-03-09T19:16:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:16:55 smithi122 ceph-mon[93869]: pgmap v8792: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 389 KiB/s wr, 11 op/s 2023-03-09T19:16:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:16:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15526.185041 secs 2023-03-09T19:16:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:16:55 smithi181 ceph-mon[94189]: pgmap v8792: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 389 KiB/s wr, 11 op/s 2023-03-09T19:16:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:16:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15526.185041 secs 2023-03-09T19:16:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:16:55 smithi038 ceph-mon[92422]: pgmap v8792: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 389 KiB/s wr, 11 op/s 2023-03-09T19:16:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:16:57 smithi122 ceph-mon[93869]: pgmap v8793: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 390 KiB/s wr, 13 op/s 2023-03-09T19:16:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:16:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15528.478595 secs 2023-03-09T19:16:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:16:57 smithi181 ceph-mon[94189]: pgmap v8793: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 390 KiB/s wr, 13 op/s 2023-03-09T19:16:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:16:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15528.478595 secs 2023-03-09T19:16:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:16:57 smithi038 ceph-mon[92422]: pgmap v8793: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 390 KiB/s wr, 13 op/s 2023-03-09T19:16:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:16:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15528.478595 secs 2023-03-09T19:16:58.802 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15990.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:17:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:16:59 smithi122 ceph-mon[93869]: pgmap v8794: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 375 KiB/s wr, 11 op/s 2023-03-09T19:17:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:16:59 smithi181 ceph-mon[94189]: pgmap v8794: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 375 KiB/s wr, 11 op/s 2023-03-09T19:17:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:16:59 smithi038 ceph-mon[92422]: pgmap v8794: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 375 KiB/s wr, 11 op/s 2023-03-09T19:17:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:17:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15531.185101 secs 2023-03-09T19:17:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:17:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15531.185101 secs 2023-03-09T19:17:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:17:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15531.185101 secs 2023-03-09T19:17:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:17:01 smithi122 ceph-mon[93869]: pgmap v8795: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 290 KiB/s wr, 15 op/s 2023-03-09T19:17:02.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:17:01 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:17:02.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:17:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:17:02.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:17:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:17:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:17:01 smithi181 ceph-mon[94189]: pgmap v8795: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 290 KiB/s wr, 15 op/s 2023-03-09T19:17:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:17:01 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:17:02.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:17:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:17:02.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:17:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:17:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:17:01 smithi038 ceph-mon[92422]: pgmap v8795: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 290 KiB/s wr, 15 op/s 2023-03-09T19:17:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:17:01 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:17:02.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:17:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:17:02.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:17:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:17:03.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:17:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15533.478666 secs 2023-03-09T19:17:03.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:17:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15533.478666 secs 2023-03-09T19:17:03.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:17:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15533.478666 secs 2023-03-09T19:17:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 15995.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:17:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:17:03 smithi122 ceph-mon[93869]: pgmap v8796: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 257 KiB/s wr, 11 op/s 2023-03-09T19:17:04.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:17:03 smithi181 ceph-mon[94189]: pgmap v8796: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 257 KiB/s wr, 11 op/s 2023-03-09T19:17:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:17:03 smithi038 ceph-mon[92422]: pgmap v8796: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 257 KiB/s wr, 11 op/s 2023-03-09T19:17:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:17:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15536.185183 secs 2023-03-09T19:17:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:17:05 smithi122 ceph-mon[93869]: pgmap v8797: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 278 KiB/s wr, 23 op/s 2023-03-09T19:17:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:17:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15536.185183 secs 2023-03-09T19:17:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:17:05 smithi181 ceph-mon[94189]: pgmap v8797: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 278 KiB/s wr, 23 op/s 2023-03-09T19:17:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:17:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15536.185183 secs 2023-03-09T19:17:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:17:05 smithi038 ceph-mon[92422]: pgmap v8797: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 278 KiB/s wr, 23 op/s 2023-03-09T19:17:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:17:07 smithi122 ceph-mon[93869]: pgmap v8798: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 305 KiB/s wr, 25 op/s 2023-03-09T19:17:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:17:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15538.478791 secs 2023-03-09T19:17:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:17:07 smithi181 ceph-mon[94189]: pgmap v8798: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 305 KiB/s wr, 25 op/s 2023-03-09T19:17:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:17:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15538.478791 secs 2023-03-09T19:17:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:17:07 smithi038 ceph-mon[92422]: pgmap v8798: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 305 KiB/s wr, 25 op/s 2023-03-09T19:17:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:17:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15538.478791 secs 2023-03-09T19:17:08.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16000.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:17:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:17:09 smithi122 ceph-mon[93869]: pgmap v8799: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 187 KiB/s wr, 22 op/s 2023-03-09T19:17:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:17:09 smithi181 ceph-mon[94189]: pgmap v8799: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 187 KiB/s wr, 22 op/s 2023-03-09T19:17:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:17:09 smithi038 ceph-mon[92422]: pgmap v8799: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 187 KiB/s wr, 22 op/s 2023-03-09T19:17:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:17:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15541.185263 secs 2023-03-09T19:17:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:17:10 smithi122 ceph-mon[93869]: osdmap e763: 12 total, 12 up, 12 in 2023-03-09T19:17:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:17:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15541.185263 secs 2023-03-09T19:17:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:17:10 smithi181 ceph-mon[94189]: osdmap e763: 12 total, 12 up, 12 in 2023-03-09T19:17:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:17:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15541.185263 secs 2023-03-09T19:17:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:17:10 smithi038 ceph-mon[92422]: osdmap e763: 12 total, 12 up, 12 in 2023-03-09T19:17:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:17:11 smithi122 ceph-mon[93869]: pgmap v8801: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 85 KiB/s wr, 22 op/s 2023-03-09T19:17:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:17:11 smithi181 ceph-mon[94189]: pgmap v8801: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 85 KiB/s wr, 22 op/s 2023-03-09T19:17:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:17:11 smithi038 ceph-mon[92422]: pgmap v8801: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 85 KiB/s wr, 22 op/s 2023-03-09T19:17:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:17:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15543.478927 secs 2023-03-09T19:17:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:17:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15543.478927 secs 2023-03-09T19:17:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:17:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15543.478927 secs 2023-03-09T19:17:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16005.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:17:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:17:13 smithi122 ceph-mon[93869]: pgmap v8802: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 85 KiB/s wr, 21 op/s 2023-03-09T19:17:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:17:13 smithi181 ceph-mon[94189]: pgmap v8802: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 85 KiB/s wr, 21 op/s 2023-03-09T19:17:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:17:13 smithi038 ceph-mon[92422]: pgmap v8802: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 85 KiB/s wr, 21 op/s 2023-03-09T19:17:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:17:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15546.185325 secs 2023-03-09T19:17:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:17:15 smithi122 ceph-mon[93869]: pgmap v8803: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 59 KiB/s wr, 7 op/s 2023-03-09T19:17:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:17:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15546.185325 secs 2023-03-09T19:17:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:17:15 smithi181 ceph-mon[94189]: pgmap v8803: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 59 KiB/s wr, 7 op/s 2023-03-09T19:17:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:17:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15546.185325 secs 2023-03-09T19:17:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:17:15 smithi038 ceph-mon[92422]: pgmap v8803: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 59 KiB/s wr, 7 op/s 2023-03-09T19:17:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:17:17 smithi122 ceph-mon[93869]: pgmap v8804: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 43 KiB/s wr, 6 op/s 2023-03-09T19:17:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:17:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15548.479049 secs 2023-03-09T19:17:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:17:17 smithi122 ceph-mon[93869]: osdmap e764: 12 total, 12 up, 12 in 2023-03-09T19:17:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:17:17 smithi181 ceph-mon[94189]: pgmap v8804: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 43 KiB/s wr, 6 op/s 2023-03-09T19:17:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:17:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15548.479049 secs 2023-03-09T19:17:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:17:17 smithi181 ceph-mon[94189]: osdmap e764: 12 total, 12 up, 12 in 2023-03-09T19:17:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:17:17 smithi038 ceph-mon[92422]: pgmap v8804: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 43 KiB/s wr, 6 op/s 2023-03-09T19:17:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:17:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15548.479049 secs 2023-03-09T19:17:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:17:17 smithi038 ceph-mon[92422]: osdmap e764: 12 total, 12 up, 12 in 2023-03-09T19:17:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16010.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:17:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:17:19 smithi122 ceph-mon[93869]: pgmap v8806: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 48 KiB/s wr, 7 op/s 2023-03-09T19:17:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:17:19 smithi181 ceph-mon[94189]: pgmap v8806: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 48 KiB/s wr, 7 op/s 2023-03-09T19:17:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:17:19 smithi038 ceph-mon[92422]: pgmap v8806: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 48 KiB/s wr, 7 op/s 2023-03-09T19:17:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:17:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15551.185411 secs 2023-03-09T19:17:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:17:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15551.185411 secs 2023-03-09T19:17:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:17:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15551.185411 secs 2023-03-09T19:17:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:17:21 smithi122 ceph-mon[93869]: pgmap v8807: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 43 KiB/s wr, 6 op/s 2023-03-09T19:17:22.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:17:21 smithi181 ceph-mon[94189]: pgmap v8807: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 43 KiB/s wr, 6 op/s 2023-03-09T19:17:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:17:21 smithi038 ceph-mon[92422]: pgmap v8807: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 43 KiB/s wr, 6 op/s 2023-03-09T19:17:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:17:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15553.479181 secs 2023-03-09T19:17:23.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:17:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15553.479181 secs 2023-03-09T19:17:23.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:17:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15553.479181 secs 2023-03-09T19:17:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16015.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:17:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:17:23 smithi122 ceph-mon[93869]: pgmap v8808: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 43 KiB/s wr, 6 op/s 2023-03-09T19:17:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:17:23 smithi181 ceph-mon[94189]: pgmap v8808: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 43 KiB/s wr, 6 op/s 2023-03-09T19:17:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:17:23 smithi038 ceph-mon[92422]: pgmap v8808: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 43 KiB/s wr, 6 op/s 2023-03-09T19:17:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:17:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15556.185471 secs 2023-03-09T19:17:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:17:25 smithi122 ceph-mon[93869]: pgmap v8809: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 43 KiB/s wr, 6 op/s 2023-03-09T19:17:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:17:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15556.185471 secs 2023-03-09T19:17:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:17:25 smithi181 ceph-mon[94189]: pgmap v8809: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 43 KiB/s wr, 6 op/s 2023-03-09T19:17:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:17:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15556.185471 secs 2023-03-09T19:17:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:17:25 smithi038 ceph-mon[92422]: pgmap v8809: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 43 KiB/s wr, 6 op/s 2023-03-09T19:17:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:17:27 smithi122 ceph-mon[93869]: pgmap v8810: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-09T19:17:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:17:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15558.479284 secs 2023-03-09T19:17:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:17:27 smithi181 ceph-mon[94189]: pgmap v8810: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-09T19:17:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:17:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15558.479284 secs 2023-03-09T19:17:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:17:27 smithi038 ceph-mon[92422]: pgmap v8810: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-09T19:17:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:17:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15558.479284 secs 2023-03-09T19:17:28.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16020.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:17:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:17:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:17:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:17:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:17:29.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:17:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:17:29.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:17:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:17:29.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:17:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:17:29.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:17:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:17:29.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:17:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:17:29.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:17:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:17:29.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:17:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:17:29.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:17:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:17:29.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:17:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:17:29.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:17:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:17:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:17:29 smithi122 ceph-mon[93869]: pgmap v8811: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T19:17:30.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:17:29 smithi181 ceph-mon[94189]: pgmap v8811: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T19:17:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:17:29 smithi038 ceph-mon[92422]: pgmap v8811: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T19:17:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:17:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15561.185531 secs 2023-03-09T19:17:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:17:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15561.185531 secs 2023-03-09T19:17:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:17:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15561.185531 secs 2023-03-09T19:17:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:17:31 smithi122 ceph-mon[93869]: pgmap v8812: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 511 B/s wr, 7 op/s 2023-03-09T19:17:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:17:31 smithi181 ceph-mon[94189]: pgmap v8812: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 511 B/s wr, 7 op/s 2023-03-09T19:17:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:17:31 smithi038 ceph-mon[92422]: pgmap v8812: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 511 B/s wr, 7 op/s 2023-03-09T19:17:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:17:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15563.479399 secs 2023-03-09T19:17:33.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:17:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15563.479399 secs 2023-03-09T19:17:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:17:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15563.479399 secs 2023-03-09T19:17:33.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16025.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:17:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:17:33 smithi122 ceph-mon[93869]: pgmap v8813: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 597 B/s wr, 5 op/s 2023-03-09T19:17:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:17:33 smithi181 ceph-mon[94189]: pgmap v8813: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 597 B/s wr, 5 op/s 2023-03-09T19:17:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:17:33 smithi038 ceph-mon[92422]: pgmap v8813: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 597 B/s wr, 5 op/s 2023-03-09T19:17:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:17:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15566.185591 secs 2023-03-09T19:17:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:17:35 smithi122 ceph-mon[93869]: pgmap v8814: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 597 B/s wr, 5 op/s 2023-03-09T19:17:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:17:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15566.185591 secs 2023-03-09T19:17:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:17:35 smithi181 ceph-mon[94189]: pgmap v8814: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 597 B/s wr, 5 op/s 2023-03-09T19:17:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:17:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15566.185591 secs 2023-03-09T19:17:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:17:35 smithi038 ceph-mon[92422]: pgmap v8814: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 597 B/s wr, 5 op/s 2023-03-09T19:17:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:17:37 smithi122 ceph-mon[93869]: pgmap v8815: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 109 KiB/s wr, 9 op/s 2023-03-09T19:17:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:17:37 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 15568.479510 secs 2023-03-09T19:17:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:17:37 smithi181 ceph-mon[94189]: pgmap v8815: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 109 KiB/s wr, 9 op/s 2023-03-09T19:17:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:17:37 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 15568.479510 secs 2023-03-09T19:17:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:17:37 smithi038 ceph-mon[92422]: pgmap v8815: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 109 KiB/s wr, 9 op/s 2023-03-09T19:17:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:17:37 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 15568.479510 secs 2023-03-09T19:17:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16030.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:17:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:17:39 smithi122 ceph-mon[93869]: pgmap v8816: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 109 KiB/s wr, 6 op/s 2023-03-09T19:17:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:17:39 smithi181 ceph-mon[94189]: pgmap v8816: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 109 KiB/s wr, 6 op/s 2023-03-09T19:17:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:17:39 smithi038 ceph-mon[92422]: pgmap v8816: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 109 KiB/s wr, 6 op/s 2023-03-09T19:17:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:17:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15571.185651 secs 2023-03-09T19:17:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:17:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:17:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:17:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:17:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:17:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15571.185651 secs 2023-03-09T19:17:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:17:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:17:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:17:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:17:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:17:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15571.185651 secs 2023-03-09T19:17:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:17:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:17:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:17:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:17:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:17:41 smithi122 ceph-mon[93869]: pgmap v8817: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 240 KiB/s wr, 10 op/s 2023-03-09T19:17:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:17:41 smithi181 ceph-mon[94189]: pgmap v8817: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 240 KiB/s wr, 10 op/s 2023-03-09T19:17:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:17:41 smithi038 ceph-mon[92422]: pgmap v8817: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 240 KiB/s wr, 10 op/s 2023-03-09T19:17:43.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:17:42 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 15573.479621 secs 2023-03-09T19:17:43.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:17:42 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 15573.479621 secs 2023-03-09T19:17:43.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:17:42 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 15573.479621 secs 2023-03-09T19:17:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16035.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:17:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:17:43 smithi122 ceph-mon[93869]: pgmap v8818: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 239 KiB/s wr, 7 op/s 2023-03-09T19:17:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:17:43 smithi181 ceph-mon[94189]: pgmap v8818: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 239 KiB/s wr, 7 op/s 2023-03-09T19:17:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:17:43 smithi038 ceph-mon[92422]: pgmap v8818: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 239 KiB/s wr, 7 op/s 2023-03-09T19:17:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:17:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15576.185739 secs 2023-03-09T19:17:46.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:17:45 smithi181 ceph-mon[94189]: pgmap v8819: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 239 KiB/s wr, 8 op/s 2023-03-09T19:17:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:17:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15576.185739 secs 2023-03-09T19:17:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:17:45 smithi038 ceph-mon[92422]: pgmap v8819: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 239 KiB/s wr, 8 op/s 2023-03-09T19:17:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:17:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15576.185739 secs 2023-03-09T19:17:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:17:45 smithi122 ceph-mon[93869]: pgmap v8819: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 239 KiB/s wr, 8 op/s 2023-03-09T19:17:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:17:47 smithi038 ceph-mon[92422]: pgmap v8820: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 395 KiB/s rd, 380 KiB/s wr, 13 op/s 2023-03-09T19:17:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:17:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15578.479744 secs 2023-03-09T19:17:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:17:47 smithi122 ceph-mon[93869]: pgmap v8820: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 395 KiB/s rd, 380 KiB/s wr, 13 op/s 2023-03-09T19:17:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:17:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15578.479744 secs 2023-03-09T19:17:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:17:47 smithi181 ceph-mon[94189]: pgmap v8820: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 395 KiB/s rd, 380 KiB/s wr, 13 op/s 2023-03-09T19:17:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:17:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15578.479744 secs 2023-03-09T19:17:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16040.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:17:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:17:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:17:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:17:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:17:49.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:17:49 smithi122 ceph-mon[93869]: pgmap v8821: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 272 KiB/s wr, 10 op/s 2023-03-09T19:17:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:17:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:17:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:17:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:17:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:17:49 smithi181 ceph-mon[94189]: pgmap v8821: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 272 KiB/s wr, 10 op/s 2023-03-09T19:17:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:17:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:17:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:17:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:17:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:17:49 smithi038 ceph-mon[92422]: pgmap v8821: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 272 KiB/s wr, 10 op/s 2023-03-09T19:17:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:17:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15581.185821 secs 2023-03-09T19:17:50.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:17:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15581.185821 secs 2023-03-09T19:17:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:17:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15581.185821 secs 2023-03-09T19:17:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:17:51 smithi122 ceph-mon[93869]: pgmap v8822: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 404 KiB/s wr, 15 op/s 2023-03-09T19:17:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:17:51 smithi181 ceph-mon[94189]: pgmap v8822: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 404 KiB/s wr, 15 op/s 2023-03-09T19:17:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:17:51 smithi038 ceph-mon[92422]: pgmap v8822: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 404 KiB/s wr, 15 op/s 2023-03-09T19:17:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:17:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15583.479822 secs 2023-03-09T19:17:52.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:17:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15583.479822 secs 2023-03-09T19:17:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:17:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15583.479822 secs 2023-03-09T19:17:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16045.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:17:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:17:53 smithi122 ceph-mon[93869]: pgmap v8823: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 273 KiB/s wr, 11 op/s 2023-03-09T19:17:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:17:53 smithi181 ceph-mon[94189]: pgmap v8823: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 273 KiB/s wr, 11 op/s 2023-03-09T19:17:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:17:53 smithi038 ceph-mon[92422]: pgmap v8823: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 273 KiB/s wr, 11 op/s 2023-03-09T19:17:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:17:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15586.185880 secs 2023-03-09T19:17:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:17:55 smithi122 ceph-mon[93869]: pgmap v8824: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 273 KiB/s wr, 12 op/s 2023-03-09T19:17:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:17:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15586.185880 secs 2023-03-09T19:17:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:17:55 smithi181 ceph-mon[94189]: pgmap v8824: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 273 KiB/s wr, 12 op/s 2023-03-09T19:17:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:17:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15586.185880 secs 2023-03-09T19:17:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:17:55 smithi038 ceph-mon[92422]: pgmap v8824: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 273 KiB/s wr, 12 op/s 2023-03-09T19:17:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:17:57 smithi122 ceph-mon[93869]: pgmap v8825: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 378 KiB/s wr, 16 op/s 2023-03-09T19:17:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:17:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15588.479939 secs 2023-03-09T19:17:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:17:57 smithi181 ceph-mon[94189]: pgmap v8825: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 378 KiB/s wr, 16 op/s 2023-03-09T19:17:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:17:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15588.479939 secs 2023-03-09T19:17:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:17:57 smithi038 ceph-mon[92422]: pgmap v8825: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 378 KiB/s wr, 16 op/s 2023-03-09T19:17:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:17:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15588.479939 secs 2023-03-09T19:17:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16050.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:18:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:17:59 smithi122 ceph-mon[93869]: pgmap v8826: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 238 KiB/s wr, 11 op/s 2023-03-09T19:18:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:17:59 smithi181 ceph-mon[94189]: pgmap v8826: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 238 KiB/s wr, 11 op/s 2023-03-09T19:18:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:17:59 smithi038 ceph-mon[92422]: pgmap v8826: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 238 KiB/s wr, 11 op/s 2023-03-09T19:18:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:18:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15591.185940 secs 2023-03-09T19:18:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:18:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15591.185940 secs 2023-03-09T19:18:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:18:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15591.185940 secs 2023-03-09T19:18:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:18:01 smithi122 ceph-mon[93869]: pgmap v8827: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 354 KiB/s wr, 17 op/s 2023-03-09T19:18:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:18:01 smithi181 ceph-mon[94189]: pgmap v8827: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 354 KiB/s wr, 17 op/s 2023-03-09T19:18:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:18:01 smithi038 ceph-mon[92422]: pgmap v8827: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 354 KiB/s wr, 17 op/s 2023-03-09T19:18:03.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:18:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15593.480062 secs 2023-03-09T19:18:03.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:18:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15593.480062 secs 2023-03-09T19:18:03.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:18:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15593.480062 secs 2023-03-09T19:18:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16055.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:18:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:18:03 smithi122 ceph-mon[93869]: pgmap v8828: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 222 KiB/s wr, 12 op/s 2023-03-09T19:18:04.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:18:03 smithi181 ceph-mon[94189]: pgmap v8828: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 222 KiB/s wr, 12 op/s 2023-03-09T19:18:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:18:03 smithi038 ceph-mon[92422]: pgmap v8828: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 222 KiB/s wr, 12 op/s 2023-03-09T19:18:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:18:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15596.185981 secs 2023-03-09T19:18:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:18:05 smithi122 ceph-mon[93869]: pgmap v8829: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 244 KiB/s wr, 24 op/s 2023-03-09T19:18:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:18:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15596.185981 secs 2023-03-09T19:18:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:18:05 smithi181 ceph-mon[94189]: pgmap v8829: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 244 KiB/s wr, 24 op/s 2023-03-09T19:18:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:18:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15596.185981 secs 2023-03-09T19:18:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:18:05 smithi038 ceph-mon[92422]: pgmap v8829: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 244 KiB/s wr, 24 op/s 2023-03-09T19:18:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:18:07 smithi122 ceph-mon[93869]: pgmap v8830: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 293 KiB/s wr, 27 op/s 2023-03-09T19:18:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:18:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15598.480178 secs 2023-03-09T19:18:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:18:07 smithi181 ceph-mon[94189]: pgmap v8830: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 293 KiB/s wr, 27 op/s 2023-03-09T19:18:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:18:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15598.480178 secs 2023-03-09T19:18:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:18:07 smithi038 ceph-mon[92422]: pgmap v8830: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 293 KiB/s wr, 27 op/s 2023-03-09T19:18:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:18:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15598.480178 secs 2023-03-09T19:18:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16060.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:18:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:18:09 smithi122 ceph-mon[93869]: pgmap v8831: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 188 KiB/s wr, 22 op/s 2023-03-09T19:18:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:18:09 smithi181 ceph-mon[94189]: pgmap v8831: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 188 KiB/s wr, 22 op/s 2023-03-09T19:18:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:18:09 smithi038 ceph-mon[92422]: pgmap v8831: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 188 KiB/s wr, 22 op/s 2023-03-09T19:18:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:18:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15601.186042 secs 2023-03-09T19:18:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:18:10 smithi122 ceph-mon[93869]: osdmap e765: 12 total, 12 up, 12 in 2023-03-09T19:18:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:18:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15601.186042 secs 2023-03-09T19:18:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:18:10 smithi181 ceph-mon[94189]: osdmap e765: 12 total, 12 up, 12 in 2023-03-09T19:18:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:18:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15601.186042 secs 2023-03-09T19:18:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:18:10 smithi038 ceph-mon[92422]: osdmap e765: 12 total, 12 up, 12 in 2023-03-09T19:18:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:18:11 smithi122 ceph-mon[93869]: pgmap v8833: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 86 KiB/s wr, 22 op/s 2023-03-09T19:18:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:18:11 smithi181 ceph-mon[94189]: pgmap v8833: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 86 KiB/s wr, 22 op/s 2023-03-09T19:18:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:18:11 smithi038 ceph-mon[92422]: pgmap v8833: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 86 KiB/s wr, 22 op/s 2023-03-09T19:18:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:18:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15603.480285 secs 2023-03-09T19:18:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:18:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15603.480285 secs 2023-03-09T19:18:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:18:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15603.480285 secs 2023-03-09T19:18:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16065.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:18:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:18:13 smithi122 ceph-mon[93869]: pgmap v8834: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 85 KiB/s wr, 21 op/s 2023-03-09T19:18:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:18:13 smithi181 ceph-mon[94189]: pgmap v8834: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 85 KiB/s wr, 21 op/s 2023-03-09T19:18:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:18:13 smithi038 ceph-mon[92422]: pgmap v8834: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 85 KiB/s wr, 21 op/s 2023-03-09T19:18:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:18:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15606.186125 secs 2023-03-09T19:18:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:18:15 smithi122 ceph-mon[93869]: pgmap v8835: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 60 KiB/s wr, 7 op/s 2023-03-09T19:18:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:18:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15606.186125 secs 2023-03-09T19:18:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:18:15 smithi181 ceph-mon[94189]: pgmap v8835: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 60 KiB/s wr, 7 op/s 2023-03-09T19:18:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:18:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15606.186125 secs 2023-03-09T19:18:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:18:15 smithi038 ceph-mon[92422]: pgmap v8835: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 60 KiB/s wr, 7 op/s 2023-03-09T19:18:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:18:17 smithi122 ceph-mon[93869]: pgmap v8836: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T19:18:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:18:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15608.480366 secs 2023-03-09T19:18:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:18:17 smithi122 ceph-mon[93869]: osdmap e766: 12 total, 12 up, 12 in 2023-03-09T19:18:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:18:17 smithi181 ceph-mon[94189]: pgmap v8836: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T19:18:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:18:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15608.480366 secs 2023-03-09T19:18:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:18:17 smithi181 ceph-mon[94189]: osdmap e766: 12 total, 12 up, 12 in 2023-03-09T19:18:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:18:17 smithi038 ceph-mon[92422]: pgmap v8836: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T19:18:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:18:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15608.480366 secs 2023-03-09T19:18:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:18:17 smithi038 ceph-mon[92422]: osdmap e766: 12 total, 12 up, 12 in 2023-03-09T19:18:18.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16070.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:18:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:18:19 smithi122 ceph-mon[93869]: pgmap v8838: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 230 B/s wr, 7 op/s 2023-03-09T19:18:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:18:19 smithi181 ceph-mon[94189]: pgmap v8838: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 230 B/s wr, 7 op/s 2023-03-09T19:18:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:18:19 smithi038 ceph-mon[92422]: pgmap v8838: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 230 B/s wr, 7 op/s 2023-03-09T19:18:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:18:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15611.186185 secs 2023-03-09T19:18:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:18:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15611.186185 secs 2023-03-09T19:18:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:18:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15611.186185 secs 2023-03-09T19:18:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:18:21 smithi122 ceph-mon[93869]: pgmap v8839: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T19:18:22.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:18:21 smithi181 ceph-mon[94189]: pgmap v8839: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T19:18:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:18:21 smithi038 ceph-mon[92422]: pgmap v8839: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T19:18:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:18:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15613.480446 secs 2023-03-09T19:18:23.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:18:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15613.480446 secs 2023-03-09T19:18:23.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:18:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15613.480446 secs 2023-03-09T19:18:23.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16075.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:18:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:18:23 smithi122 ceph-mon[93869]: pgmap v8840: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T19:18:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:18:23 smithi181 ceph-mon[94189]: pgmap v8840: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T19:18:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:18:23 smithi038 ceph-mon[92422]: pgmap v8840: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T19:18:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:18:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15616.186245 secs 2023-03-09T19:18:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:18:25 smithi122 ceph-mon[93869]: pgmap v8841: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T19:18:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:18:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15616.186245 secs 2023-03-09T19:18:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:18:25 smithi181 ceph-mon[94189]: pgmap v8841: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T19:18:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:18:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15616.186245 secs 2023-03-09T19:18:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:18:25 smithi038 ceph-mon[92422]: pgmap v8841: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T19:18:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:18:27 smithi122 ceph-mon[93869]: pgmap v8842: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 45 KiB/s wr, 6 op/s 2023-03-09T19:18:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:18:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15618.480566 secs 2023-03-09T19:18:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:18:27 smithi181 ceph-mon[94189]: pgmap v8842: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 45 KiB/s wr, 6 op/s 2023-03-09T19:18:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:18:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15618.480566 secs 2023-03-09T19:18:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:18:27 smithi038 ceph-mon[92422]: pgmap v8842: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 45 KiB/s wr, 6 op/s 2023-03-09T19:18:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:18:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15618.480566 secs 2023-03-09T19:18:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16080.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:18:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:18:29 smithi038 ceph-mon[92422]: pgmap v8843: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 41 KiB/s wr, 5 op/s 2023-03-09T19:18:30.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:18:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:18:30.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:18:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:18:30.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:18:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:18:30.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:18:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:18:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:18:29 smithi122 ceph-mon[93869]: pgmap v8843: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 41 KiB/s wr, 5 op/s 2023-03-09T19:18:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:18:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:18:30.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:18:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:18:30.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:18:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:18:30.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:18:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:18:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:18:29 smithi181 ceph-mon[94189]: pgmap v8843: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 41 KiB/s wr, 5 op/s 2023-03-09T19:18:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:18:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:18:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:18:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:18:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:18:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:18:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:18:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:18:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:18:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15621.186310 secs 2023-03-09T19:18:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:18:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15621.186310 secs 2023-03-09T19:18:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:18:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15621.186310 secs 2023-03-09T19:18:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:18:31 smithi038 ceph-mon[92422]: pgmap v8844: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 43 KiB/s wr, 8 op/s 2023-03-09T19:18:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:18:31 smithi122 ceph-mon[93869]: pgmap v8844: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 43 KiB/s wr, 8 op/s 2023-03-09T19:18:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:18:31 smithi181 ceph-mon[94189]: pgmap v8844: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 43 KiB/s wr, 8 op/s 2023-03-09T19:18:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:18:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15623.480703 secs 2023-03-09T19:18:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:18:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15623.480703 secs 2023-03-09T19:18:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:18:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15623.480703 secs 2023-03-09T19:18:33.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16085.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:18:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:18:34 smithi122 ceph-mon[93869]: pgmap v8845: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 43 KiB/s wr, 6 op/s 2023-03-09T19:18:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:18:34 smithi181 ceph-mon[94189]: pgmap v8845: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 43 KiB/s wr, 6 op/s 2023-03-09T19:18:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:18:34 smithi038 ceph-mon[92422]: pgmap v8845: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 43 KiB/s wr, 6 op/s 2023-03-09T19:18:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:18:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15626.186369 secs 2023-03-09T19:18:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:18:36 smithi122 ceph-mon[93869]: pgmap v8846: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 43 KiB/s wr, 6 op/s 2023-03-09T19:18:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:18:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15626.186369 secs 2023-03-09T19:18:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:18:36 smithi181 ceph-mon[94189]: pgmap v8846: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 43 KiB/s wr, 6 op/s 2023-03-09T19:18:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:18:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15626.186369 secs 2023-03-09T19:18:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:18:36 smithi038 ceph-mon[92422]: pgmap v8846: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 43 KiB/s wr, 6 op/s 2023-03-09T19:18:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:18:38 smithi122 ceph-mon[93869]: pgmap v8847: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 152 KiB/s wr, 9 op/s 2023-03-09T19:18:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:18:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15628.480848 secs 2023-03-09T19:18:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:18:38 smithi181 ceph-mon[94189]: pgmap v8847: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 152 KiB/s wr, 9 op/s 2023-03-09T19:18:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:18:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15628.480848 secs 2023-03-09T19:18:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:18:38 smithi038 ceph-mon[92422]: pgmap v8847: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 152 KiB/s wr, 9 op/s 2023-03-09T19:18:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:18:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15628.480848 secs 2023-03-09T19:18:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16090.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:18:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:18:40 smithi122 ceph-mon[93869]: pgmap v8848: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 114 KiB/s wr, 7 op/s 2023-03-09T19:18:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:18:40 smithi181 ceph-mon[94189]: pgmap v8848: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 114 KiB/s wr, 7 op/s 2023-03-09T19:18:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:18:40 smithi038 ceph-mon[92422]: pgmap v8848: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 114 KiB/s wr, 7 op/s 2023-03-09T19:18:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:18:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15631.186430 secs 2023-03-09T19:18:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:18:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15631.186430 secs 2023-03-09T19:18:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:18:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15631.186430 secs 2023-03-09T19:18:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:18:42 smithi122 ceph-mon[93869]: pgmap v8849: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 245 KiB/s wr, 11 op/s 2023-03-09T19:18:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:18:42 smithi181 ceph-mon[94189]: pgmap v8849: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 245 KiB/s wr, 11 op/s 2023-03-09T19:18:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:18:42 smithi038 ceph-mon[92422]: pgmap v8849: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 245 KiB/s wr, 11 op/s 2023-03-09T19:18:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:18:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15633.480933 secs 2023-03-09T19:18:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:18:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15633.480933 secs 2023-03-09T19:18:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:18:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15633.480933 secs 2023-03-09T19:18:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16095.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:18:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:18:44 smithi122 ceph-mon[93869]: pgmap v8850: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 240 KiB/s wr, 8 op/s 2023-03-09T19:18:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:18:44 smithi181 ceph-mon[94189]: pgmap v8850: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 240 KiB/s wr, 8 op/s 2023-03-09T19:18:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:18:44 smithi038 ceph-mon[92422]: pgmap v8850: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 240 KiB/s wr, 8 op/s 2023-03-09T19:18:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:18:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15636.186490 secs 2023-03-09T19:18:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:18:46 smithi122 ceph-mon[93869]: pgmap v8851: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 240 KiB/s wr, 8 op/s 2023-03-09T19:18:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:18:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15636.186490 secs 2023-03-09T19:18:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:18:46 smithi181 ceph-mon[94189]: pgmap v8851: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 240 KiB/s wr, 8 op/s 2023-03-09T19:18:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:18:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15636.186490 secs 2023-03-09T19:18:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:18:46 smithi038 ceph-mon[92422]: pgmap v8851: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 240 KiB/s wr, 8 op/s 2023-03-09T19:18:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:18:48 smithi122 ceph-mon[93869]: pgmap v8852: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 365 KiB/s wr, 12 op/s 2023-03-09T19:18:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:18:48 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:18:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:18:48 smithi181 ceph-mon[94189]: pgmap v8852: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 365 KiB/s wr, 12 op/s 2023-03-09T19:18:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:18:48 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:18:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:18:48 smithi038 ceph-mon[92422]: pgmap v8852: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 365 KiB/s wr, 12 op/s 2023-03-09T19:18:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:18:48 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:18:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16100.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:18:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:18:49 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:18:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:18:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:18:49.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:18:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:18:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:18:49 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:18:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:18:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:18:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:18:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:18:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:18:49 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:18:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:18:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:18:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:18:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:18:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:18:50 smithi122 ceph-mon[93869]: pgmap v8853: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 272 KiB/s wr, 10 op/s 2023-03-09T19:18:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:18:50 smithi181 ceph-mon[94189]: pgmap v8853: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 272 KiB/s wr, 10 op/s 2023-03-09T19:18:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:18:50 smithi038 ceph-mon[92422]: pgmap v8853: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 272 KiB/s wr, 10 op/s 2023-03-09T19:18:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:18:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15641.186576 secs 2023-03-09T19:18:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:18:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15641.186576 secs 2023-03-09T19:18:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:18:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15641.186576 secs 2023-03-09T19:18:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:18:52 smithi122 ceph-mon[93869]: pgmap v8854: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 285 KiB/s wr, 13 op/s 2023-03-09T19:18:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:18:52 smithi181 ceph-mon[94189]: pgmap v8854: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 285 KiB/s wr, 13 op/s 2023-03-09T19:18:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:18:52 smithi038 ceph-mon[92422]: pgmap v8854: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 285 KiB/s wr, 13 op/s 2023-03-09T19:18:53.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16105.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:18:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:18:53 smithi122 ceph-mon[93869]: pgmap v8855: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 274 KiB/s wr, 11 op/s 2023-03-09T19:18:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:18:53 smithi181 ceph-mon[94189]: pgmap v8855: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 274 KiB/s wr, 11 op/s 2023-03-09T19:18:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:18:53 smithi038 ceph-mon[92422]: pgmap v8855: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 274 KiB/s wr, 11 op/s 2023-03-09T19:18:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:18:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15646.186635 secs 2023-03-09T19:18:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:18:55 smithi122 ceph-mon[93869]: pgmap v8856: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 274 KiB/s wr, 11 op/s 2023-03-09T19:18:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:18:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15646.186635 secs 2023-03-09T19:18:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:18:55 smithi181 ceph-mon[94189]: pgmap v8856: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 274 KiB/s wr, 11 op/s 2023-03-09T19:18:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:18:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15646.186635 secs 2023-03-09T19:18:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:18:55 smithi038 ceph-mon[92422]: pgmap v8856: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 274 KiB/s wr, 11 op/s 2023-03-09T19:18:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:18:57 smithi122 ceph-mon[93869]: pgmap v8857: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 297 KiB/s wr, 14 op/s 2023-03-09T19:18:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:18:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15648.481327 secs 2023-03-09T19:18:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:18:57 smithi181 ceph-mon[94189]: pgmap v8857: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 297 KiB/s wr, 14 op/s 2023-03-09T19:18:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:18:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15648.481327 secs 2023-03-09T19:18:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:18:57 smithi038 ceph-mon[92422]: pgmap v8857: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 297 KiB/s wr, 14 op/s 2023-03-09T19:18:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:18:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15648.481327 secs 2023-03-09T19:18:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16110.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:19:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:18:59 smithi122 ceph-mon[93869]: pgmap v8858: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 269 KiB/s wr, 11 op/s 2023-03-09T19:19:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:18:59 smithi181 ceph-mon[94189]: pgmap v8858: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 269 KiB/s wr, 11 op/s 2023-03-09T19:19:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:18:59 smithi038 ceph-mon[92422]: pgmap v8858: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 269 KiB/s wr, 11 op/s 2023-03-09T19:19:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:19:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15651.186695 secs 2023-03-09T19:19:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:19:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15651.186695 secs 2023-03-09T19:19:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:19:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15651.186695 secs 2023-03-09T19:19:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:19:01 smithi122 ceph-mon[93869]: pgmap v8859: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 275 KiB/s wr, 15 op/s 2023-03-09T19:19:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:19:01 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:19:02.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:19:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:19:02.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:19:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:19:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:19:01 smithi181 ceph-mon[94189]: pgmap v8859: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 275 KiB/s wr, 15 op/s 2023-03-09T19:19:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:19:01 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:19:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:19:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:19:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:19:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:19:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:19:01 smithi038 ceph-mon[92422]: pgmap v8859: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 275 KiB/s wr, 15 op/s 2023-03-09T19:19:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:19:01 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:19:02.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:19:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:19:02.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:19:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:19:03.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:19:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15653.481445 secs 2023-03-09T19:19:03.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:19:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15653.481445 secs 2023-03-09T19:19:03.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:19:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15653.481445 secs 2023-03-09T19:19:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16115.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:19:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:19:03 smithi122 ceph-mon[93869]: pgmap v8860: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 336 KiB/s rd, 356 KiB/s wr, 13 op/s 2023-03-09T19:19:04.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:19:03 smithi181 ceph-mon[94189]: pgmap v8860: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 336 KiB/s rd, 356 KiB/s wr, 13 op/s 2023-03-09T19:19:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:19:03 smithi038 ceph-mon[92422]: pgmap v8860: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 336 KiB/s rd, 356 KiB/s wr, 13 op/s 2023-03-09T19:19:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:19:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15656.186752 secs 2023-03-09T19:19:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:19:05 smithi122 ceph-mon[93869]: pgmap v8861: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 258 KiB/s wr, 23 op/s 2023-03-09T19:19:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:19:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15656.186752 secs 2023-03-09T19:19:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:19:05 smithi181 ceph-mon[94189]: pgmap v8861: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 258 KiB/s wr, 23 op/s 2023-03-09T19:19:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:19:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15656.186752 secs 2023-03-09T19:19:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:19:05 smithi038 ceph-mon[92422]: pgmap v8861: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 258 KiB/s wr, 23 op/s 2023-03-09T19:19:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:19:07 smithi122 ceph-mon[93869]: pgmap v8862: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 258 KiB/s wr, 25 op/s 2023-03-09T19:19:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:19:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15658.481568 secs 2023-03-09T19:19:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:19:07 smithi181 ceph-mon[94189]: pgmap v8862: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 258 KiB/s wr, 25 op/s 2023-03-09T19:19:08.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:19:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15658.481568 secs 2023-03-09T19:19:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:19:07 smithi038 ceph-mon[92422]: pgmap v8862: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 258 KiB/s wr, 25 op/s 2023-03-09T19:19:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:19:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15658.481568 secs 2023-03-09T19:19:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16120.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:19:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:19:09 smithi122 ceph-mon[93869]: pgmap v8863: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 285 KiB/s wr, 24 op/s 2023-03-09T19:19:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:19:09 smithi181 ceph-mon[94189]: pgmap v8863: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 285 KiB/s wr, 24 op/s 2023-03-09T19:19:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:19:09 smithi038 ceph-mon[92422]: pgmap v8863: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 285 KiB/s wr, 24 op/s 2023-03-09T19:19:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:19:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15661.186838 secs 2023-03-09T19:19:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:19:10 smithi122 ceph-mon[93869]: osdmap e767: 12 total, 12 up, 12 in 2023-03-09T19:19:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:19:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15661.186838 secs 2023-03-09T19:19:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:19:10 smithi181 ceph-mon[94189]: osdmap e767: 12 total, 12 up, 12 in 2023-03-09T19:19:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:19:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15661.186838 secs 2023-03-09T19:19:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:19:10 smithi038 ceph-mon[92422]: osdmap e767: 12 total, 12 up, 12 in 2023-03-09T19:19:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:19:11 smithi122 ceph-mon[93869]: pgmap v8865: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 199 KiB/s wr, 23 op/s 2023-03-09T19:19:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:19:11 smithi181 ceph-mon[94189]: pgmap v8865: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 199 KiB/s wr, 23 op/s 2023-03-09T19:19:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:19:11 smithi038 ceph-mon[92422]: pgmap v8865: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 199 KiB/s wr, 23 op/s 2023-03-09T19:19:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:19:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15663.481681 secs 2023-03-09T19:19:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:19:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15663.481681 secs 2023-03-09T19:19:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:19:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15663.481681 secs 2023-03-09T19:19:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16125.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:19:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:19:13 smithi122 ceph-mon[93869]: pgmap v8866: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 86 KiB/s wr, 22 op/s 2023-03-09T19:19:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:19:13 smithi181 ceph-mon[94189]: pgmap v8866: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 86 KiB/s wr, 22 op/s 2023-03-09T19:19:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:19:13 smithi038 ceph-mon[92422]: pgmap v8866: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 86 KiB/s wr, 22 op/s 2023-03-09T19:19:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:19:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15666.186898 secs 2023-03-09T19:19:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:19:15 smithi122 ceph-mon[93869]: pgmap v8867: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 60 KiB/s wr, 8 op/s 2023-03-09T19:19:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:19:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15666.186898 secs 2023-03-09T19:19:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:19:15 smithi181 ceph-mon[94189]: pgmap v8867: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 60 KiB/s wr, 8 op/s 2023-03-09T19:19:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:19:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15666.186898 secs 2023-03-09T19:19:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:19:15 smithi038 ceph-mon[92422]: pgmap v8867: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 60 KiB/s wr, 8 op/s 2023-03-09T19:19:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:19:17 smithi122 ceph-mon[93869]: pgmap v8868: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 60 KiB/s wr, 8 op/s 2023-03-09T19:19:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:19:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15668.481796 secs 2023-03-09T19:19:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:19:17 smithi122 ceph-mon[93869]: osdmap e768: 12 total, 12 up, 12 in 2023-03-09T19:19:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:19:17 smithi181 ceph-mon[94189]: pgmap v8868: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 60 KiB/s wr, 8 op/s 2023-03-09T19:19:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:19:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15668.481796 secs 2023-03-09T19:19:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:19:17 smithi181 ceph-mon[94189]: osdmap e768: 12 total, 12 up, 12 in 2023-03-09T19:19:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:19:17 smithi038 ceph-mon[92422]: pgmap v8868: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 60 KiB/s wr, 8 op/s 2023-03-09T19:19:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:19:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15668.481796 secs 2023-03-09T19:19:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:19:17 smithi038 ceph-mon[92422]: osdmap e768: 12 total, 12 up, 12 in 2023-03-09T19:19:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16130.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:19:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:19:19 smithi122 ceph-mon[93869]: pgmap v8870: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 49 KiB/s wr, 8 op/s 2023-03-09T19:19:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:19:19 smithi181 ceph-mon[94189]: pgmap v8870: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 49 KiB/s wr, 8 op/s 2023-03-09T19:19:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:19:19 smithi038 ceph-mon[92422]: pgmap v8870: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 49 KiB/s wr, 8 op/s 2023-03-09T19:19:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:19:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15671.186960 secs 2023-03-09T19:19:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:19:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15671.186960 secs 2023-03-09T19:19:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:19:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15671.186960 secs 2023-03-09T19:19:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:19:21 smithi122 ceph-mon[93869]: pgmap v8871: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 43 KiB/s wr, 7 op/s 2023-03-09T19:19:22.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:19:21 smithi181 ceph-mon[94189]: pgmap v8871: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 43 KiB/s wr, 7 op/s 2023-03-09T19:19:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:19:21 smithi038 ceph-mon[92422]: pgmap v8871: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 43 KiB/s wr, 7 op/s 2023-03-09T19:19:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:19:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15673.481912 secs 2023-03-09T19:19:23.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:19:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15673.481912 secs 2023-03-09T19:19:23.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:19:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15673.481912 secs 2023-03-09T19:19:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16135.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:19:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:19:23 smithi122 ceph-mon[93869]: pgmap v8872: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 43 KiB/s wr, 6 op/s 2023-03-09T19:19:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:19:23 smithi181 ceph-mon[94189]: pgmap v8872: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 43 KiB/s wr, 6 op/s 2023-03-09T19:19:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:19:23 smithi038 ceph-mon[92422]: pgmap v8872: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 43 KiB/s wr, 6 op/s 2023-03-09T19:19:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:19:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15676.187054 secs 2023-03-09T19:19:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:19:25 smithi122 ceph-mon[93869]: pgmap v8873: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 43 KiB/s wr, 6 op/s 2023-03-09T19:19:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:19:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15676.187054 secs 2023-03-09T19:19:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:19:25 smithi181 ceph-mon[94189]: pgmap v8873: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 43 KiB/s wr, 6 op/s 2023-03-09T19:19:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:19:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15676.187054 secs 2023-03-09T19:19:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:19:25 smithi038 ceph-mon[92422]: pgmap v8873: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 43 KiB/s wr, 6 op/s 2023-03-09T19:19:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:19:27 smithi122 ceph-mon[93869]: pgmap v8874: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 6 op/s 2023-03-09T19:19:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:19:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15678.482033 secs 2023-03-09T19:19:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:19:27 smithi181 ceph-mon[94189]: pgmap v8874: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 6 op/s 2023-03-09T19:19:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:19:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15678.482033 secs 2023-03-09T19:19:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:19:27 smithi038 ceph-mon[92422]: pgmap v8874: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 6 op/s 2023-03-09T19:19:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:19:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15678.482033 secs 2023-03-09T19:19:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16140.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:19:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:19:29 smithi122 ceph-mon[93869]: pgmap v8875: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 97 KiB/s wr, 7 op/s 2023-03-09T19:19:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:19:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:19:30.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:19:29 smithi181 ceph-mon[94189]: pgmap v8875: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 97 KiB/s wr, 7 op/s 2023-03-09T19:19:30.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:19:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:19:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:19:29 smithi038 ceph-mon[92422]: pgmap v8875: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 97 KiB/s wr, 7 op/s 2023-03-09T19:19:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:19:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:19:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:19:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15681.187116 secs 2023-03-09T19:19:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:19:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15681.187116 secs 2023-03-09T19:19:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:19:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15681.187116 secs 2023-03-09T19:19:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:19:31 smithi122 ceph-mon[93869]: pgmap v8876: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 54 KiB/s wr, 8 op/s 2023-03-09T19:19:32.227 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:19:31 smithi038 ceph-mon[92422]: pgmap v8876: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 54 KiB/s wr, 8 op/s 2023-03-09T19:19:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:19:31 smithi181 ceph-mon[94189]: pgmap v8876: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 54 KiB/s wr, 8 op/s 2023-03-09T19:19:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:19:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15683.482151 secs 2023-03-09T19:19:33.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:19:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15683.482151 secs 2023-03-09T19:19:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:19:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15683.482151 secs 2023-03-09T19:19:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16145.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:19:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:19:33 smithi122 ceph-mon[93869]: pgmap v8877: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 54 KiB/s wr, 6 op/s 2023-03-09T19:19:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:19:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:19:34.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:19:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:19:34.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:19:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:19:34.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:19:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:19:34.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:19:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:19:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:19:33 smithi181 ceph-mon[94189]: pgmap v8877: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 54 KiB/s wr, 6 op/s 2023-03-09T19:19:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:19:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:19:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:19:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:19:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:19:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:19:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:19:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:19:34.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:19:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:19:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:19:33 smithi038 ceph-mon[92422]: pgmap v8877: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 54 KiB/s wr, 6 op/s 2023-03-09T19:19:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:19:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:19:34.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:19:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:19:34.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:19:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:19:34.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:19:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:19:34.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:19:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:19:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:19:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15686.187194 secs 2023-03-09T19:19:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:19:35 smithi038 ceph-mon[92422]: pgmap v8878: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 54 KiB/s wr, 6 op/s 2023-03-09T19:19:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:19:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15686.187194 secs 2023-03-09T19:19:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:19:35 smithi122 ceph-mon[93869]: pgmap v8878: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 54 KiB/s wr, 6 op/s 2023-03-09T19:19:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:19:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15686.187194 secs 2023-03-09T19:19:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:19:35 smithi181 ceph-mon[94189]: pgmap v8878: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 54 KiB/s wr, 6 op/s 2023-03-09T19:19:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:19:37 smithi038 ceph-mon[92422]: pgmap v8879: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 54 KiB/s wr, 8 op/s 2023-03-09T19:19:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:19:37 smithi122 ceph-mon[93869]: pgmap v8879: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 54 KiB/s wr, 8 op/s 2023-03-09T19:19:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:19:37 smithi181 ceph-mon[94189]: pgmap v8879: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 54 KiB/s wr, 8 op/s 2023-03-09T19:19:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16150.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:19:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:19:39 smithi038 ceph-mon[92422]: pgmap v8880: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 110 KiB/s wr, 7 op/s 2023-03-09T19:19:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:19:39 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:19:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:19:39 smithi122 ceph-mon[93869]: pgmap v8880: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 110 KiB/s wr, 7 op/s 2023-03-09T19:19:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:19:39 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:19:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:19:39 smithi181 ceph-mon[94189]: pgmap v8880: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 110 KiB/s wr, 7 op/s 2023-03-09T19:19:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:19:39 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:19:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:19:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15691.187254 secs 2023-03-09T19:19:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:19:41 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:19:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:19:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:19:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:19:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:19:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:19:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15691.187254 secs 2023-03-09T19:19:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:19:41 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:19:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:19:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:19:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:19:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:19:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:19:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15691.187254 secs 2023-03-09T19:19:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:19:41 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:19:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:19:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:19:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:19:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:19:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:19:42 smithi122 ceph-mon[93869]: pgmap v8881: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 69 KiB/s wr, 8 op/s 2023-03-09T19:19:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:19:42 smithi181 ceph-mon[94189]: pgmap v8881: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 69 KiB/s wr, 8 op/s 2023-03-09T19:19:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:19:42 smithi038 ceph-mon[92422]: pgmap v8881: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 69 KiB/s wr, 8 op/s 2023-03-09T19:19:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16155.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:19:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:19:43 smithi181 ceph-mon[94189]: pgmap v8882: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 69 KiB/s wr, 6 op/s 2023-03-09T19:19:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:19:43 smithi038 ceph-mon[92422]: pgmap v8882: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 69 KiB/s wr, 6 op/s 2023-03-09T19:19:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:19:43 smithi122 ceph-mon[93869]: pgmap v8882: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 69 KiB/s wr, 6 op/s 2023-03-09T19:19:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:19:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15696.187313 secs 2023-03-09T19:19:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:19:45 smithi122 ceph-mon[93869]: pgmap v8883: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 232 KiB/s wr, 7 op/s 2023-03-09T19:19:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:19:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15696.187313 secs 2023-03-09T19:19:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:19:45 smithi181 ceph-mon[94189]: pgmap v8883: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 232 KiB/s wr, 7 op/s 2023-03-09T19:19:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:19:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15696.187313 secs 2023-03-09T19:19:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:19:45 smithi038 ceph-mon[92422]: pgmap v8883: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 232 KiB/s wr, 7 op/s 2023-03-09T19:19:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:19:47 smithi122 ceph-mon[93869]: pgmap v8884: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 246 KiB/s wr, 10 op/s 2023-03-09T19:19:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:19:47 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 15698.482492 secs 2023-03-09T19:19:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:19:47 smithi181 ceph-mon[94189]: pgmap v8884: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 246 KiB/s wr, 10 op/s 2023-03-09T19:19:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:19:47 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 15698.482492 secs 2023-03-09T19:19:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:19:47 smithi038 ceph-mon[92422]: pgmap v8884: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 246 KiB/s wr, 10 op/s 2023-03-09T19:19:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:19:47 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 15698.482492 secs 2023-03-09T19:19:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16160.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:19:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:19:48 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:19:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:19:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:19:49.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:19:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:19:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:19:48 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:19:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:19:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:19:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:19:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:19:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:19:48 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:19:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:19:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:19:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:19:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:19:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:19:49 smithi122 ceph-mon[93869]: pgmap v8885: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 636 KiB/s rd, 408 KiB/s wr, 10 op/s 2023-03-09T19:19:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:19:49 smithi181 ceph-mon[94189]: pgmap v8885: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 636 KiB/s rd, 408 KiB/s wr, 10 op/s 2023-03-09T19:19:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:19:49 smithi038 ceph-mon[92422]: pgmap v8885: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 636 KiB/s rd, 408 KiB/s wr, 10 op/s 2023-03-09T19:19:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:19:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15701.187415 secs 2023-03-09T19:19:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:19:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15701.187415 secs 2023-03-09T19:19:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:19:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15701.187415 secs 2023-03-09T19:19:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:19:51 smithi122 ceph-mon[93869]: pgmap v8886: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 594 KiB/s rd, 364 KiB/s wr, 13 op/s 2023-03-09T19:19:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:19:51 smithi181 ceph-mon[94189]: pgmap v8886: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 594 KiB/s rd, 364 KiB/s wr, 13 op/s 2023-03-09T19:19:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:19:51 smithi038 ceph-mon[92422]: pgmap v8886: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 594 KiB/s rd, 364 KiB/s wr, 13 op/s 2023-03-09T19:19:53.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:19:52 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 15703.482616 secs 2023-03-09T19:19:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:19:52 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 15703.482616 secs 2023-03-09T19:19:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:19:52 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 15703.482616 secs 2023-03-09T19:19:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16165.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:19:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:19:53 smithi122 ceph-mon[93869]: pgmap v8887: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 581 KiB/s rd, 352 KiB/s wr, 11 op/s 2023-03-09T19:19:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:19:53 smithi181 ceph-mon[94189]: pgmap v8887: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 581 KiB/s rd, 352 KiB/s wr, 11 op/s 2023-03-09T19:19:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:19:53 smithi038 ceph-mon[92422]: pgmap v8887: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 581 KiB/s rd, 352 KiB/s wr, 11 op/s 2023-03-09T19:19:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:19:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15706.187476 secs 2023-03-09T19:19:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:19:55 smithi122 ceph-mon[93869]: pgmap v8888: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 395 KiB/s wr, 12 op/s 2023-03-09T19:19:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:19:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15706.187476 secs 2023-03-09T19:19:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:19:55 smithi181 ceph-mon[94189]: pgmap v8888: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 395 KiB/s wr, 12 op/s 2023-03-09T19:19:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:19:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15706.187476 secs 2023-03-09T19:19:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:19:55 smithi038 ceph-mon[92422]: pgmap v8888: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 395 KiB/s wr, 12 op/s 2023-03-09T19:19:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:19:57 smithi122 ceph-mon[93869]: pgmap v8889: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 256 KiB/s wr, 13 op/s 2023-03-09T19:19:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:19:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15708.482759 secs 2023-03-09T19:19:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:19:57 smithi181 ceph-mon[94189]: pgmap v8889: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 256 KiB/s wr, 13 op/s 2023-03-09T19:19:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:19:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15708.482759 secs 2023-03-09T19:19:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:19:57 smithi038 ceph-mon[92422]: pgmap v8889: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 256 KiB/s wr, 13 op/s 2023-03-09T19:19:58.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:19:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15708.482759 secs 2023-03-09T19:19:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16170.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:20:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:19:59 smithi122 ceph-mon[93869]: pgmap v8890: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 466 KiB/s rd, 380 KiB/s wr, 11 op/s 2023-03-09T19:20:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:19:59 smithi181 ceph-mon[94189]: pgmap v8890: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 466 KiB/s rd, 380 KiB/s wr, 11 op/s 2023-03-09T19:20:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:19:59 smithi038 ceph-mon[92422]: pgmap v8890: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 466 KiB/s rd, 380 KiB/s wr, 11 op/s 2023-03-09T19:20:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:20:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15711.187537 secs 2023-03-09T19:20:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:20:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:20:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:20:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:20:01.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:20:00 smithi122 ceph-mon[93869]: overall HEALTH_WARN 2 MDSs report slow requests 2023-03-09T19:20:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:20:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15711.187537 secs 2023-03-09T19:20:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:20:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:20:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:20:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:20:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:20:00 smithi181 ceph-mon[94189]: overall HEALTH_WARN 2 MDSs report slow requests 2023-03-09T19:20:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:20:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15711.187537 secs 2023-03-09T19:20:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:20:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:20:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:20:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:20:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:20:00 smithi038 ceph-mon[92422]: overall HEALTH_WARN 2 MDSs report slow requests 2023-03-09T19:20:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:20:01 smithi122 ceph-mon[93869]: pgmap v8891: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 269 KiB/s wr, 13 op/s 2023-03-09T19:20:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:20:01 smithi181 ceph-mon[94189]: pgmap v8891: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 269 KiB/s wr, 13 op/s 2023-03-09T19:20:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:20:01 smithi038 ceph-mon[92422]: pgmap v8891: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 269 KiB/s wr, 13 op/s 2023-03-09T19:20:03.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:20:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15713.482891 secs 2023-03-09T19:20:03.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:20:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15713.482891 secs 2023-03-09T19:20:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:20:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15713.482891 secs 2023-03-09T19:20:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16175.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:20:04.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:20:03 smithi181 ceph-mon[94189]: pgmap v8892: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 257 KiB/s wr, 10 op/s 2023-03-09T19:20:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:20:03 smithi038 ceph-mon[92422]: pgmap v8892: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 257 KiB/s wr, 10 op/s 2023-03-09T19:20:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:20:03 smithi122 ceph-mon[93869]: pgmap v8892: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 257 KiB/s wr, 10 op/s 2023-03-09T19:20:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:20:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15716.187596 secs 2023-03-09T19:20:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:20:05 smithi038 ceph-mon[92422]: pgmap v8893: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 278 KiB/s wr, 23 op/s 2023-03-09T19:20:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:20:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15716.187596 secs 2023-03-09T19:20:06.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:20:05 smithi122 ceph-mon[93869]: pgmap v8893: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 278 KiB/s wr, 23 op/s 2023-03-09T19:20:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:20:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15716.187596 secs 2023-03-09T19:20:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:20:05 smithi181 ceph-mon[94189]: pgmap v8893: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 278 KiB/s wr, 23 op/s 2023-03-09T19:20:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:20:07 smithi038 ceph-mon[92422]: pgmap v8894: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 296 KiB/s wr, 26 op/s 2023-03-09T19:20:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:20:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15718.482991 secs 2023-03-09T19:20:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:20:07 smithi122 ceph-mon[93869]: pgmap v8894: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 296 KiB/s wr, 26 op/s 2023-03-09T19:20:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:20:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15718.482991 secs 2023-03-09T19:20:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:20:07 smithi181 ceph-mon[94189]: pgmap v8894: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 296 KiB/s wr, 26 op/s 2023-03-09T19:20:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:20:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15718.482991 secs 2023-03-09T19:20:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16180.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:20:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:20:09 smithi038 ceph-mon[92422]: pgmap v8895: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 272 KiB/s wr, 23 op/s 2023-03-09T19:20:10.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:20:09 smithi038 ceph-mon[92422]: osdmap e769: 12 total, 12 up, 12 in 2023-03-09T19:20:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:20:09 smithi122 ceph-mon[93869]: pgmap v8895: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 272 KiB/s wr, 23 op/s 2023-03-09T19:20:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:20:09 smithi122 ceph-mon[93869]: osdmap e769: 12 total, 12 up, 12 in 2023-03-09T19:20:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:20:09 smithi181 ceph-mon[94189]: pgmap v8895: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 272 KiB/s wr, 23 op/s 2023-03-09T19:20:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:20:09 smithi181 ceph-mon[94189]: osdmap e769: 12 total, 12 up, 12 in 2023-03-09T19:20:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:20:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15721.187673 secs 2023-03-09T19:20:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:20:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15721.187673 secs 2023-03-09T19:20:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:20:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15721.187673 secs 2023-03-09T19:20:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:20:12 smithi122 ceph-mon[93869]: pgmap v8897: 129 pgs: 10 active+clean+snaptrim_wait, 119 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 100 KiB/s wr, 24 op/s 2023-03-09T19:20:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:20:12 smithi181 ceph-mon[94189]: pgmap v8897: 129 pgs: 10 active+clean+snaptrim_wait, 119 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 100 KiB/s wr, 24 op/s 2023-03-09T19:20:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:20:12 smithi038 ceph-mon[92422]: pgmap v8897: 129 pgs: 10 active+clean+snaptrim_wait, 119 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 100 KiB/s wr, 24 op/s 2023-03-09T19:20:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:20:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15723.483120 secs 2023-03-09T19:20:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:20:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15723.483120 secs 2023-03-09T19:20:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:20:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15723.483120 secs 2023-03-09T19:20:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16185.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:20:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:20:14 smithi122 ceph-mon[93869]: pgmap v8898: 129 pgs: 10 active+clean+snaptrim_wait, 119 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 100 KiB/s wr, 24 op/s 2023-03-09T19:20:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:20:14 smithi181 ceph-mon[94189]: pgmap v8898: 129 pgs: 10 active+clean+snaptrim_wait, 119 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 100 KiB/s wr, 24 op/s 2023-03-09T19:20:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:20:14 smithi038 ceph-mon[92422]: pgmap v8898: 129 pgs: 10 active+clean+snaptrim_wait, 119 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 100 KiB/s wr, 24 op/s 2023-03-09T19:20:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:20:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15726.187732 secs 2023-03-09T19:20:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:20:16 smithi122 ceph-mon[93869]: pgmap v8899: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 74 KiB/s wr, 8 op/s 2023-03-09T19:20:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:20:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15726.187732 secs 2023-03-09T19:20:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:20:16 smithi181 ceph-mon[94189]: pgmap v8899: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 74 KiB/s wr, 8 op/s 2023-03-09T19:20:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:20:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15726.187732 secs 2023-03-09T19:20:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:20:16 smithi038 ceph-mon[92422]: pgmap v8899: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 74 KiB/s wr, 8 op/s 2023-03-09T19:20:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:20:18 smithi122 ceph-mon[93869]: pgmap v8900: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T19:20:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:20:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15728.483232 secs 2023-03-09T19:20:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:20:18 smithi122 ceph-mon[93869]: osdmap e770: 12 total, 12 up, 12 in 2023-03-09T19:20:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:20:18 smithi181 ceph-mon[94189]: pgmap v8900: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T19:20:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:20:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15728.483232 secs 2023-03-09T19:20:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:20:18 smithi181 ceph-mon[94189]: osdmap e770: 12 total, 12 up, 12 in 2023-03-09T19:20:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:20:18 smithi038 ceph-mon[92422]: pgmap v8900: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T19:20:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:20:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15728.483232 secs 2023-03-09T19:20:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:20:18 smithi038 ceph-mon[92422]: osdmap e770: 12 total, 12 up, 12 in 2023-03-09T19:20:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16190.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:20:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:20:20 smithi122 ceph-mon[93869]: pgmap v8902: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 208 B/s wr, 6 op/s 2023-03-09T19:20:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:20:20 smithi181 ceph-mon[94189]: pgmap v8902: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 208 B/s wr, 6 op/s 2023-03-09T19:20:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:20:20 smithi038 ceph-mon[92422]: pgmap v8902: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 208 B/s wr, 6 op/s 2023-03-09T19:20:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:20:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15731.187809 secs 2023-03-09T19:20:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:20:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15731.187809 secs 2023-03-09T19:20:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:20:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15731.187809 secs 2023-03-09T19:20:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:20:22 smithi122 ceph-mon[93869]: pgmap v8903: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T19:20:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:20:22 smithi181 ceph-mon[94189]: pgmap v8903: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T19:20:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:20:22 smithi038 ceph-mon[92422]: pgmap v8903: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T19:20:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:20:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15733.483349 secs 2023-03-09T19:20:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:20:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15733.483349 secs 2023-03-09T19:20:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:20:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15733.483349 secs 2023-03-09T19:20:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16195.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:20:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:20:24 smithi122 ceph-mon[93869]: pgmap v8904: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T19:20:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:20:24 smithi181 ceph-mon[94189]: pgmap v8904: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T19:20:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:20:24 smithi038 ceph-mon[92422]: pgmap v8904: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T19:20:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:20:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15736.187895 secs 2023-03-09T19:20:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:20:26 smithi122 ceph-mon[93869]: pgmap v8905: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T19:20:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:20:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15736.187895 secs 2023-03-09T19:20:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:20:26 smithi181 ceph-mon[94189]: pgmap v8905: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T19:20:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:20:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15736.187895 secs 2023-03-09T19:20:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:20:26 smithi038 ceph-mon[92422]: pgmap v8905: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T19:20:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:20:28 smithi122 ceph-mon[93869]: pgmap v8906: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 52 KiB/s wr, 6 op/s 2023-03-09T19:20:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:20:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15738.483459 secs 2023-03-09T19:20:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:20:28 smithi181 ceph-mon[94189]: pgmap v8906: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 52 KiB/s wr, 6 op/s 2023-03-09T19:20:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:20:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15738.483459 secs 2023-03-09T19:20:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:20:28 smithi038 ceph-mon[92422]: pgmap v8906: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 52 KiB/s wr, 6 op/s 2023-03-09T19:20:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:20:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15738.483459 secs 2023-03-09T19:20:28.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16200.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:20:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:20:30 smithi122 ceph-mon[93869]: pgmap v8907: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 47 KiB/s wr, 6 op/s 2023-03-09T19:20:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:20:30 smithi181 ceph-mon[94189]: pgmap v8907: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 47 KiB/s wr, 6 op/s 2023-03-09T19:20:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:20:30 smithi038 ceph-mon[92422]: pgmap v8907: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 47 KiB/s wr, 6 op/s 2023-03-09T19:20:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:20:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15741.187955 secs 2023-03-09T19:20:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:20:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15741.187955 secs 2023-03-09T19:20:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:20:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15741.187955 secs 2023-03-09T19:20:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:20:32 smithi122 ceph-mon[93869]: pgmap v8908: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 97 KiB/s wr, 8 op/s 2023-03-09T19:20:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:20:32 smithi181 ceph-mon[94189]: pgmap v8908: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 97 KiB/s wr, 8 op/s 2023-03-09T19:20:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:20:32 smithi038 ceph-mon[92422]: pgmap v8908: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 97 KiB/s wr, 8 op/s 2023-03-09T19:20:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:20:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15743.483580 secs 2023-03-09T19:20:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:20:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15743.483580 secs 2023-03-09T19:20:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:20:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15743.483580 secs 2023-03-09T19:20:33.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16205.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:20:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:20:34 smithi122 ceph-mon[93869]: pgmap v8909: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 97 KiB/s wr, 6 op/s 2023-03-09T19:20:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:20:34 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:20:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:20:34 smithi181 ceph-mon[94189]: pgmap v8909: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 97 KiB/s wr, 6 op/s 2023-03-09T19:20:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:20:34 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:20:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:20:34 smithi038 ceph-mon[92422]: pgmap v8909: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 97 KiB/s wr, 6 op/s 2023-03-09T19:20:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:20:34 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:20:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:20:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15746.187989 secs 2023-03-09T19:20:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:20:36 smithi122 ceph-mon[93869]: pgmap v8910: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 97 KiB/s wr, 6 op/s 2023-03-09T19:20:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:20:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15746.187989 secs 2023-03-09T19:20:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:20:36 smithi181 ceph-mon[94189]: pgmap v8910: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 97 KiB/s wr, 6 op/s 2023-03-09T19:20:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:20:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15746.187989 secs 2023-03-09T19:20:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:20:36 smithi038 ceph-mon[92422]: pgmap v8910: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 97 KiB/s wr, 6 op/s 2023-03-09T19:20:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:20:38 smithi122 ceph-mon[93869]: pgmap v8911: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 154 KiB/s wr, 9 op/s 2023-03-09T19:20:38.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:20:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15748.483695 secs 2023-03-09T19:20:38.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:20:38 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:20:38.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:20:38 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:20:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:20:38 smithi181 ceph-mon[94189]: pgmap v8911: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 154 KiB/s wr, 9 op/s 2023-03-09T19:20:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:20:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15748.483695 secs 2023-03-09T19:20:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:20:38 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:20:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:20:38 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:20:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:20:38 smithi038 ceph-mon[92422]: pgmap v8911: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 154 KiB/s wr, 9 op/s 2023-03-09T19:20:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:20:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15748.483695 secs 2023-03-09T19:20:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:20:38 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:20:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:20:38 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:20:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16210.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:20:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:20:40 smithi122 ceph-mon[93869]: pgmap v8912: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 111 KiB/s wr, 6 op/s 2023-03-09T19:20:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:20:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:20:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:20:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:20:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:20:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:20:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:20:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:20:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:20:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:20:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:20:40 smithi181 ceph-mon[94189]: pgmap v8912: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 111 KiB/s wr, 6 op/s 2023-03-09T19:20:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:20:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:20:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:20:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:20:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:20:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:20:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:20:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:20:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:20:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:20:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:20:40 smithi038 ceph-mon[92422]: pgmap v8912: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 111 KiB/s wr, 6 op/s 2023-03-09T19:20:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:20:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:20:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:20:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:20:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:20:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:20:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:20:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:20:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:20:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:20:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:20:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15751.188050 secs 2023-03-09T19:20:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:20:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15751.188050 secs 2023-03-09T19:20:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:20:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15751.188050 secs 2023-03-09T19:20:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:20:42 smithi122 ceph-mon[93869]: pgmap v8913: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 243 KiB/s wr, 9 op/s 2023-03-09T19:20:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:20:42 smithi181 ceph-mon[94189]: pgmap v8913: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 243 KiB/s wr, 9 op/s 2023-03-09T19:20:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:20:42 smithi038 ceph-mon[92422]: pgmap v8913: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 243 KiB/s wr, 9 op/s 2023-03-09T19:20:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:20:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15753.483796 secs 2023-03-09T19:20:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:20:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15753.483796 secs 2023-03-09T19:20:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:20:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15753.483796 secs 2023-03-09T19:20:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16215.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:20:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:20:44 smithi122 ceph-mon[93869]: pgmap v8914: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 190 KiB/s wr, 7 op/s 2023-03-09T19:20:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:20:44 smithi181 ceph-mon[94189]: pgmap v8914: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 190 KiB/s wr, 7 op/s 2023-03-09T19:20:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:20:44 smithi038 ceph-mon[92422]: pgmap v8914: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 190 KiB/s wr, 7 op/s 2023-03-09T19:20:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:20:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15756.188109 secs 2023-03-09T19:20:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:20:46 smithi122 ceph-mon[93869]: pgmap v8915: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 190 KiB/s wr, 7 op/s 2023-03-09T19:20:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:20:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15756.188109 secs 2023-03-09T19:20:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:20:46 smithi181 ceph-mon[94189]: pgmap v8915: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 190 KiB/s wr, 7 op/s 2023-03-09T19:20:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:20:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15756.188109 secs 2023-03-09T19:20:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:20:46 smithi038 ceph-mon[92422]: pgmap v8915: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 190 KiB/s wr, 7 op/s 2023-03-09T19:20:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:20:48 smithi181 ceph-mon[94189]: pgmap v8916: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 331 KiB/s wr, 11 op/s 2023-03-09T19:20:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:20:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15758.483921 secs 2023-03-09T19:20:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:20:48 smithi038 ceph-mon[92422]: pgmap v8916: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 331 KiB/s wr, 11 op/s 2023-03-09T19:20:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:20:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15758.483921 secs 2023-03-09T19:20:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:20:48 smithi122 ceph-mon[93869]: pgmap v8916: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 331 KiB/s wr, 11 op/s 2023-03-09T19:20:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:20:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15758.483921 secs 2023-03-09T19:20:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16220.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:20:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:20:50 smithi038 ceph-mon[92422]: pgmap v8917: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 275 KiB/s wr, 9 op/s 2023-03-09T19:20:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:20:50 smithi122 ceph-mon[93869]: pgmap v8917: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 275 KiB/s wr, 9 op/s 2023-03-09T19:20:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:20:50 smithi181 ceph-mon[94189]: pgmap v8917: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 275 KiB/s wr, 9 op/s 2023-03-09T19:20:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:20:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15761.188172 secs 2023-03-09T19:20:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:20:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15761.188172 secs 2023-03-09T19:20:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:20:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15761.188172 secs 2023-03-09T19:20:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:20:52 smithi122 ceph-mon[93869]: pgmap v8918: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 409 KiB/s wr, 14 op/s 2023-03-09T19:20:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:20:52 smithi181 ceph-mon[94189]: pgmap v8918: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 409 KiB/s wr, 14 op/s 2023-03-09T19:20:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:20:52 smithi038 ceph-mon[92422]: pgmap v8918: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 409 KiB/s wr, 14 op/s 2023-03-09T19:20:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:20:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15763.484064 secs 2023-03-09T19:20:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:20:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15763.484064 secs 2023-03-09T19:20:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:20:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15763.484064 secs 2023-03-09T19:20:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16225.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:20:54.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:20:54 smithi122 ceph-mon[93869]: pgmap v8919: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 277 KiB/s wr, 10 op/s 2023-03-09T19:20:54.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:20:54 smithi181 ceph-mon[94189]: pgmap v8919: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 277 KiB/s wr, 10 op/s 2023-03-09T19:20:54.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:20:54 smithi038 ceph-mon[92422]: pgmap v8919: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 277 KiB/s wr, 10 op/s 2023-03-09T19:20:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:20:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15766.188233 secs 2023-03-09T19:20:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:20:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15766.188233 secs 2023-03-09T19:20:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:20:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15766.188233 secs 2023-03-09T19:20:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:20:56 smithi122 ceph-mon[93869]: pgmap v8920: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 277 KiB/s wr, 10 op/s 2023-03-09T19:20:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:20:56 smithi181 ceph-mon[94189]: pgmap v8920: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 277 KiB/s wr, 10 op/s 2023-03-09T19:20:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:20:56 smithi038 ceph-mon[92422]: pgmap v8920: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 277 KiB/s wr, 10 op/s 2023-03-09T19:20:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:20:58 smithi122 ceph-mon[93869]: pgmap v8921: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 397 KiB/s wr, 14 op/s 2023-03-09T19:20:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:20:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15768.484194 secs 2023-03-09T19:20:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:20:58 smithi181 ceph-mon[94189]: pgmap v8921: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 397 KiB/s wr, 14 op/s 2023-03-09T19:20:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:20:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15768.484194 secs 2023-03-09T19:20:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:20:58 smithi038 ceph-mon[92422]: pgmap v8921: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 397 KiB/s wr, 14 op/s 2023-03-09T19:20:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:20:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15768.484194 secs 2023-03-09T19:20:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16230.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:21:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:21:00 smithi122 ceph-mon[93869]: pgmap v8922: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 257 KiB/s wr, 11 op/s 2023-03-09T19:21:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:21:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15771.188293 secs 2023-03-09T19:21:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:21:00 smithi181 ceph-mon[94189]: pgmap v8922: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 257 KiB/s wr, 11 op/s 2023-03-09T19:21:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:21:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15771.188293 secs 2023-03-09T19:21:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:21:00 smithi038 ceph-mon[92422]: pgmap v8922: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 257 KiB/s wr, 11 op/s 2023-03-09T19:21:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:21:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15771.188293 secs 2023-03-09T19:21:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:21:02 smithi122 ceph-mon[93869]: pgmap v8923: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 350 KiB/s wr, 14 op/s 2023-03-09T19:21:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:21:02 smithi181 ceph-mon[94189]: pgmap v8923: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 350 KiB/s wr, 14 op/s 2023-03-09T19:21:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:21:02 smithi038 ceph-mon[92422]: pgmap v8923: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 350 KiB/s wr, 14 op/s 2023-03-09T19:21:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:21:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15773.484310 secs 2023-03-09T19:21:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:21:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15773.484310 secs 2023-03-09T19:21:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:21:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15773.484310 secs 2023-03-09T19:21:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16235.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:21:04.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:21:04 smithi122 ceph-mon[93869]: pgmap v8924: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 216 KiB/s wr, 9 op/s 2023-03-09T19:21:04.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:21:04 smithi181 ceph-mon[94189]: pgmap v8924: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 216 KiB/s wr, 9 op/s 2023-03-09T19:21:04.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:21:04 smithi038 ceph-mon[92422]: pgmap v8924: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 216 KiB/s wr, 9 op/s 2023-03-09T19:21:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:21:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15776.188355 secs 2023-03-09T19:21:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:21:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15776.188355 secs 2023-03-09T19:21:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:21:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15776.188355 secs 2023-03-09T19:21:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:21:06 smithi122 ceph-mon[93869]: pgmap v8925: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 237 KiB/s wr, 21 op/s 2023-03-09T19:21:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:21:06 smithi181 ceph-mon[94189]: pgmap v8925: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 237 KiB/s wr, 21 op/s 2023-03-09T19:21:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:21:06 smithi038 ceph-mon[92422]: pgmap v8925: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 237 KiB/s wr, 21 op/s 2023-03-09T19:21:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:21:08 smithi122 ceph-mon[93869]: pgmap v8926: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 311 KiB/s wr, 27 op/s 2023-03-09T19:21:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:21:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15778.484439 secs 2023-03-09T19:21:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:21:08 smithi181 ceph-mon[94189]: pgmap v8926: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 311 KiB/s wr, 27 op/s 2023-03-09T19:21:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:21:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15778.484439 secs 2023-03-09T19:21:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:21:08 smithi038 ceph-mon[92422]: pgmap v8926: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 311 KiB/s wr, 27 op/s 2023-03-09T19:21:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:21:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15778.484439 secs 2023-03-09T19:21:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16240.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:21:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:21:10 smithi122 ceph-mon[93869]: pgmap v8927: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 190 KiB/s wr, 22 op/s 2023-03-09T19:21:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:21:10 smithi122 ceph-mon[93869]: osdmap e771: 12 total, 12 up, 12 in 2023-03-09T19:21:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:21:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15781.188417 secs 2023-03-09T19:21:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:21:10 smithi181 ceph-mon[94189]: pgmap v8927: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 190 KiB/s wr, 22 op/s 2023-03-09T19:21:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:21:10 smithi181 ceph-mon[94189]: osdmap e771: 12 total, 12 up, 12 in 2023-03-09T19:21:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:21:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15781.188417 secs 2023-03-09T19:21:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:21:10 smithi038 ceph-mon[92422]: pgmap v8927: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 190 KiB/s wr, 22 op/s 2023-03-09T19:21:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:21:10 smithi038 ceph-mon[92422]: osdmap e771: 12 total, 12 up, 12 in 2023-03-09T19:21:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:21:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15781.188417 secs 2023-03-09T19:21:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:21:11 smithi122 ceph-mon[93869]: pgmap v8929: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 114 KiB/s wr, 25 op/s 2023-03-09T19:21:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:21:11 smithi181 ceph-mon[94189]: pgmap v8929: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 114 KiB/s wr, 25 op/s 2023-03-09T19:21:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:21:11 smithi038 ceph-mon[92422]: pgmap v8929: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 114 KiB/s wr, 25 op/s 2023-03-09T19:21:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:21:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15783.484562 secs 2023-03-09T19:21:12.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:21:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15783.484562 secs 2023-03-09T19:21:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:21:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15783.484562 secs 2023-03-09T19:21:13.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16245.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:21:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:21:13 smithi122 ceph-mon[93869]: pgmap v8930: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 114 KiB/s wr, 25 op/s 2023-03-09T19:21:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:21:13 smithi181 ceph-mon[94189]: pgmap v8930: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 114 KiB/s wr, 25 op/s 2023-03-09T19:21:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:21:13 smithi038 ceph-mon[92422]: pgmap v8930: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 114 KiB/s wr, 25 op/s 2023-03-09T19:21:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:21:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15786.188479 secs 2023-03-09T19:21:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:21:15 smithi122 ceph-mon[93869]: pgmap v8931: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 88 KiB/s wr, 10 op/s 2023-03-09T19:21:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:21:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15786.188479 secs 2023-03-09T19:21:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:21:15 smithi181 ceph-mon[94189]: pgmap v8931: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 88 KiB/s wr, 10 op/s 2023-03-09T19:21:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:21:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15786.188479 secs 2023-03-09T19:21:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:21:15 smithi038 ceph-mon[92422]: pgmap v8931: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 88 KiB/s wr, 10 op/s 2023-03-09T19:21:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:21:17 smithi122 ceph-mon[93869]: pgmap v8932: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 44 KiB/s wr, 6 op/s 2023-03-09T19:21:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:21:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15788.484704 secs 2023-03-09T19:21:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:21:17 smithi122 ceph-mon[93869]: osdmap e772: 12 total, 12 up, 12 in 2023-03-09T19:21:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:21:17 smithi181 ceph-mon[94189]: pgmap v8932: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 44 KiB/s wr, 6 op/s 2023-03-09T19:21:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:21:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15788.484704 secs 2023-03-09T19:21:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:21:17 smithi181 ceph-mon[94189]: osdmap e772: 12 total, 12 up, 12 in 2023-03-09T19:21:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:21:17 smithi038 ceph-mon[92422]: pgmap v8932: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 44 KiB/s wr, 6 op/s 2023-03-09T19:21:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:21:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15788.484704 secs 2023-03-09T19:21:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:21:17 smithi038 ceph-mon[92422]: osdmap e772: 12 total, 12 up, 12 in 2023-03-09T19:21:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16250.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:21:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:21:19 smithi122 ceph-mon[93869]: pgmap v8934: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 46 KiB/s wr, 7 op/s 2023-03-09T19:21:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:21:19 smithi181 ceph-mon[94189]: pgmap v8934: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 46 KiB/s wr, 7 op/s 2023-03-09T19:21:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:21:19 smithi038 ceph-mon[92422]: pgmap v8934: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 46 KiB/s wr, 7 op/s 2023-03-09T19:21:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:21:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15791.188540 secs 2023-03-09T19:21:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:21:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15791.188540 secs 2023-03-09T19:21:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:21:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15791.188540 secs 2023-03-09T19:21:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:21:21 smithi122 ceph-mon[93869]: pgmap v8935: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 44 KiB/s wr, 6 op/s 2023-03-09T19:21:22.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:21:21 smithi181 ceph-mon[94189]: pgmap v8935: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 44 KiB/s wr, 6 op/s 2023-03-09T19:21:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:21:21 smithi038 ceph-mon[92422]: pgmap v8935: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 44 KiB/s wr, 6 op/s 2023-03-09T19:21:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:21:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15793.484827 secs 2023-03-09T19:21:23.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:21:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15793.484827 secs 2023-03-09T19:21:23.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:21:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15793.484827 secs 2023-03-09T19:21:23.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16255.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:21:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:21:23 smithi122 ceph-mon[93869]: pgmap v8936: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 44 KiB/s wr, 6 op/s 2023-03-09T19:21:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:21:23 smithi181 ceph-mon[94189]: pgmap v8936: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 44 KiB/s wr, 6 op/s 2023-03-09T19:21:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:21:23 smithi038 ceph-mon[92422]: pgmap v8936: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 44 KiB/s wr, 6 op/s 2023-03-09T19:21:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:21:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15796.188600 secs 2023-03-09T19:21:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:21:25 smithi122 ceph-mon[93869]: pgmap v8937: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 44 KiB/s wr, 6 op/s 2023-03-09T19:21:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:21:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15796.188600 secs 2023-03-09T19:21:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:21:25 smithi181 ceph-mon[94189]: pgmap v8937: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 44 KiB/s wr, 6 op/s 2023-03-09T19:21:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:21:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15796.188600 secs 2023-03-09T19:21:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:21:25 smithi038 ceph-mon[92422]: pgmap v8937: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 44 KiB/s wr, 6 op/s 2023-03-09T19:21:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:21:27 smithi122 ceph-mon[93869]: pgmap v8938: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 409 B/s wr, 6 op/s 2023-03-09T19:21:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:21:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15798.484940 secs 2023-03-09T19:21:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:21:27 smithi181 ceph-mon[94189]: pgmap v8938: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 409 B/s wr, 6 op/s 2023-03-09T19:21:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:21:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15798.484940 secs 2023-03-09T19:21:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:21:27 smithi038 ceph-mon[92422]: pgmap v8938: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 409 B/s wr, 6 op/s 2023-03-09T19:21:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:21:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15798.484940 secs 2023-03-09T19:21:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16260.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:21:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:21:29 smithi122 ceph-mon[93869]: pgmap v8939: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 460 B/s wr, 6 op/s 2023-03-09T19:21:30.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:21:29 smithi181 ceph-mon[94189]: pgmap v8939: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 460 B/s wr, 6 op/s 2023-03-09T19:21:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:21:29 smithi038 ceph-mon[92422]: pgmap v8939: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 460 B/s wr, 6 op/s 2023-03-09T19:21:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:21:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15801.188663 secs 2023-03-09T19:21:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:21:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15801.188663 secs 2023-03-09T19:21:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:21:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15801.188663 secs 2023-03-09T19:21:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:21:31 smithi122 ceph-mon[93869]: pgmap v8940: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 54 KiB/s wr, 8 op/s 2023-03-09T19:21:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:21:31 smithi181 ceph-mon[94189]: pgmap v8940: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 54 KiB/s wr, 8 op/s 2023-03-09T19:21:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:21:31 smithi038 ceph-mon[92422]: pgmap v8940: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 54 KiB/s wr, 8 op/s 2023-03-09T19:21:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:21:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15803.485053 secs 2023-03-09T19:21:33.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:21:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15803.485053 secs 2023-03-09T19:21:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:21:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15803.485053 secs 2023-03-09T19:21:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16265.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:21:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:21:33 smithi122 ceph-mon[93869]: pgmap v8941: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 55 KiB/s wr, 6 op/s 2023-03-09T19:21:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:21:33 smithi181 ceph-mon[94189]: pgmap v8941: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 55 KiB/s wr, 6 op/s 2023-03-09T19:21:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:21:33 smithi038 ceph-mon[92422]: pgmap v8941: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 55 KiB/s wr, 6 op/s 2023-03-09T19:21:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:21:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15806.188722 secs 2023-03-09T19:21:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:21:35 smithi122 ceph-mon[93869]: pgmap v8942: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 55 KiB/s wr, 6 op/s 2023-03-09T19:21:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:21:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15806.188722 secs 2023-03-09T19:21:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:21:35 smithi181 ceph-mon[94189]: pgmap v8942: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 55 KiB/s wr, 6 op/s 2023-03-09T19:21:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:21:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15806.188722 secs 2023-03-09T19:21:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:21:35 smithi038 ceph-mon[92422]: pgmap v8942: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 55 KiB/s wr, 6 op/s 2023-03-09T19:21:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:21:37 smithi122 ceph-mon[93869]: pgmap v8943: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 112 KiB/s wr, 9 op/s 2023-03-09T19:21:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:21:37 smithi181 ceph-mon[94189]: pgmap v8943: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 112 KiB/s wr, 9 op/s 2023-03-09T19:21:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:21:37 smithi038 ceph-mon[92422]: pgmap v8943: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 112 KiB/s wr, 9 op/s 2023-03-09T19:21:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16270.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:21:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:21:39 smithi122 ceph-mon[93869]: pgmap v8944: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 111 KiB/s wr, 6 op/s 2023-03-09T19:21:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:21:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:21:40.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:21:39 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:21:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:21:39 smithi181 ceph-mon[94189]: pgmap v8944: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 111 KiB/s wr, 6 op/s 2023-03-09T19:21:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:21:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:21:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:21:39 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:21:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:21:39 smithi038 ceph-mon[92422]: pgmap v8944: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 111 KiB/s wr, 6 op/s 2023-03-09T19:21:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:21:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:21:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:21:39 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:21:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:21:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15811.188783 secs 2023-03-09T19:21:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:21:40 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:21:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:21:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:21:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:21:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:21:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:21:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:21:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:21:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:21:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:21:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:21:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:21:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15811.188783 secs 2023-03-09T19:21:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:21:40 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:21:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:21:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:21:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:21:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:21:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:21:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:21:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:21:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:21:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:21:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:21:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:21:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15811.188783 secs 2023-03-09T19:21:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:21:40 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:21:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:21:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:21:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:21:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:21:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:21:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:21:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:21:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:21:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:21:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:21:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:21:41 smithi038 ceph-mon[92422]: pgmap v8945: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 245 KiB/s wr, 9 op/s 2023-03-09T19:21:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:21:41 smithi122 ceph-mon[93869]: pgmap v8945: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 245 KiB/s wr, 9 op/s 2023-03-09T19:21:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:21:41 smithi181 ceph-mon[94189]: pgmap v8945: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 245 KiB/s wr, 9 op/s 2023-03-09T19:21:43.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16275.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:21:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:21:43 smithi122 ceph-mon[93869]: pgmap v8946: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 191 KiB/s wr, 7 op/s 2023-03-09T19:21:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:21:43 smithi181 ceph-mon[94189]: pgmap v8946: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 191 KiB/s wr, 7 op/s 2023-03-09T19:21:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:21:43 smithi038 ceph-mon[92422]: pgmap v8946: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 191 KiB/s wr, 7 op/s 2023-03-09T19:21:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:21:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15816.188842 secs 2023-03-09T19:21:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:21:45 smithi122 ceph-mon[93869]: pgmap v8947: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 191 KiB/s wr, 7 op/s 2023-03-09T19:21:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:21:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15816.188842 secs 2023-03-09T19:21:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:21:45 smithi181 ceph-mon[94189]: pgmap v8947: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 191 KiB/s wr, 7 op/s 2023-03-09T19:21:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:21:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15816.188842 secs 2023-03-09T19:21:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:21:45 smithi038 ceph-mon[92422]: pgmap v8947: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 191 KiB/s wr, 7 op/s 2023-03-09T19:21:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:21:47 smithi122 ceph-mon[93869]: pgmap v8948: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 333 KiB/s wr, 11 op/s 2023-03-09T19:21:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:21:47 smithi181 ceph-mon[94189]: pgmap v8948: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 333 KiB/s wr, 11 op/s 2023-03-09T19:21:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:21:47 smithi038 ceph-mon[92422]: pgmap v8948: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 333 KiB/s wr, 11 op/s 2023-03-09T19:21:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16280.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:21:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:21:49 smithi122 ceph-mon[93869]: pgmap v8949: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 277 KiB/s wr, 9 op/s 2023-03-09T19:21:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:21:49 smithi181 ceph-mon[94189]: pgmap v8949: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 277 KiB/s wr, 9 op/s 2023-03-09T19:21:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:21:49 smithi038 ceph-mon[92422]: pgmap v8949: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 277 KiB/s wr, 9 op/s 2023-03-09T19:21:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:21:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15821.188926 secs 2023-03-09T19:21:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:21:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15821.188926 secs 2023-03-09T19:21:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:21:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15821.188926 secs 2023-03-09T19:21:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:21:51 smithi122 ceph-mon[93869]: pgmap v8950: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 412 KiB/s wr, 13 op/s 2023-03-09T19:21:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:21:51 smithi181 ceph-mon[94189]: pgmap v8950: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 412 KiB/s wr, 13 op/s 2023-03-09T19:21:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:21:51 smithi038 ceph-mon[92422]: pgmap v8950: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 412 KiB/s wr, 13 op/s 2023-03-09T19:21:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16285.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:21:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:21:53 smithi122 ceph-mon[93869]: pgmap v8951: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 278 KiB/s wr, 10 op/s 2023-03-09T19:21:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:21:53 smithi181 ceph-mon[94189]: pgmap v8951: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 278 KiB/s wr, 10 op/s 2023-03-09T19:21:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:21:53 smithi038 ceph-mon[92422]: pgmap v8951: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 278 KiB/s wr, 10 op/s 2023-03-09T19:21:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:21:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15826.188983 secs 2023-03-09T19:21:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:21:55 smithi122 ceph-mon[93869]: pgmap v8952: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 278 KiB/s wr, 10 op/s 2023-03-09T19:21:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:21:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15826.188983 secs 2023-03-09T19:21:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:21:55 smithi181 ceph-mon[94189]: pgmap v8952: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 278 KiB/s wr, 10 op/s 2023-03-09T19:21:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:21:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15826.188983 secs 2023-03-09T19:21:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:21:55 smithi038 ceph-mon[92422]: pgmap v8952: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 278 KiB/s wr, 10 op/s 2023-03-09T19:21:58.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:21:57 smithi122 ceph-mon[93869]: pgmap v8953: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 386 KiB/s wr, 14 op/s 2023-03-09T19:21:58.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:21:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15828.485709 secs 2023-03-09T19:21:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:21:57 smithi181 ceph-mon[94189]: pgmap v8953: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 386 KiB/s wr, 14 op/s 2023-03-09T19:21:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:21:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15828.485709 secs 2023-03-09T19:21:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:21:57 smithi038 ceph-mon[92422]: pgmap v8953: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 386 KiB/s wr, 14 op/s 2023-03-09T19:21:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:21:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15828.485709 secs 2023-03-09T19:21:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16290.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:22:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:21:59 smithi122 ceph-mon[93869]: pgmap v8954: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 244 KiB/s wr, 10 op/s 2023-03-09T19:22:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:21:59 smithi181 ceph-mon[94189]: pgmap v8954: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 244 KiB/s wr, 10 op/s 2023-03-09T19:22:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:21:59 smithi038 ceph-mon[92422]: pgmap v8954: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 244 KiB/s wr, 10 op/s 2023-03-09T19:22:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15831.189044 secs 2023-03-09T19:22:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:00 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:22:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:22:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:22:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15831.189044 secs 2023-03-09T19:22:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:00 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:22:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:22:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:22:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15831.189044 secs 2023-03-09T19:22:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:00 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:22:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:22:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:22:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:01 smithi122 ceph-mon[93869]: pgmap v8955: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 338 KiB/s wr, 13 op/s 2023-03-09T19:22:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:01 smithi181 ceph-mon[94189]: pgmap v8955: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 338 KiB/s wr, 13 op/s 2023-03-09T19:22:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:01 smithi038 ceph-mon[92422]: pgmap v8955: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 338 KiB/s wr, 13 op/s 2023-03-09T19:22:03.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15833.485818 secs 2023-03-09T19:22:03.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15833.485818 secs 2023-03-09T19:22:03.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15833.485818 secs 2023-03-09T19:22:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16295.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:22:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:03 smithi122 ceph-mon[93869]: pgmap v8956: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 203 KiB/s wr, 9 op/s 2023-03-09T19:22:04.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:03 smithi181 ceph-mon[94189]: pgmap v8956: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 203 KiB/s wr, 9 op/s 2023-03-09T19:22:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:03 smithi038 ceph-mon[92422]: pgmap v8956: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 203 KiB/s wr, 9 op/s 2023-03-09T19:22:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15836.189104 secs 2023-03-09T19:22:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:05 smithi122 ceph-mon[93869]: pgmap v8957: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 224 KiB/s wr, 21 op/s 2023-03-09T19:22:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15836.189104 secs 2023-03-09T19:22:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:05 smithi181 ceph-mon[94189]: pgmap v8957: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 224 KiB/s wr, 21 op/s 2023-03-09T19:22:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15836.189104 secs 2023-03-09T19:22:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:05 smithi038 ceph-mon[92422]: pgmap v8957: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 224 KiB/s wr, 21 op/s 2023-03-09T19:22:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:07 smithi122 ceph-mon[93869]: pgmap v8958: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 298 KiB/s wr, 26 op/s 2023-03-09T19:22:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15838.485906 secs 2023-03-09T19:22:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:07 smithi181 ceph-mon[94189]: pgmap v8958: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 298 KiB/s wr, 26 op/s 2023-03-09T19:22:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15838.485906 secs 2023-03-09T19:22:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:07 smithi038 ceph-mon[92422]: pgmap v8958: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 298 KiB/s wr, 26 op/s 2023-03-09T19:22:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15838.485906 secs 2023-03-09T19:22:08.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16300.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:22:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:09 smithi122 ceph-mon[93869]: pgmap v8959: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 191 KiB/s wr, 22 op/s 2023-03-09T19:22:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:09 smithi122 ceph-mon[93869]: osdmap e773: 12 total, 12 up, 12 in 2023-03-09T19:22:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:09 smithi181 ceph-mon[94189]: pgmap v8959: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 191 KiB/s wr, 22 op/s 2023-03-09T19:22:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:09 smithi181 ceph-mon[94189]: osdmap e773: 12 total, 12 up, 12 in 2023-03-09T19:22:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:09 smithi038 ceph-mon[92422]: pgmap v8959: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 191 KiB/s wr, 22 op/s 2023-03-09T19:22:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:09 smithi038 ceph-mon[92422]: osdmap e773: 12 total, 12 up, 12 in 2023-03-09T19:22:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15841.189166 secs 2023-03-09T19:22:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15841.189166 secs 2023-03-09T19:22:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15841.189166 secs 2023-03-09T19:22:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:11 smithi122 ceph-mon[93869]: pgmap v8961: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 115 KiB/s wr, 24 op/s 2023-03-09T19:22:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:11 smithi181 ceph-mon[94189]: pgmap v8961: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 115 KiB/s wr, 24 op/s 2023-03-09T19:22:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:11 smithi038 ceph-mon[92422]: pgmap v8961: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 115 KiB/s wr, 24 op/s 2023-03-09T19:22:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15843.486057 secs 2023-03-09T19:22:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15843.486057 secs 2023-03-09T19:22:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15843.486057 secs 2023-03-09T19:22:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16305.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:22:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:13 smithi038 ceph-mon[92422]: pgmap v8962: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 115 KiB/s wr, 24 op/s 2023-03-09T19:22:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:13 smithi122 ceph-mon[93869]: pgmap v8962: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 115 KiB/s wr, 24 op/s 2023-03-09T19:22:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:13 smithi181 ceph-mon[94189]: pgmap v8962: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 115 KiB/s wr, 24 op/s 2023-03-09T19:22:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15846.189225 secs 2023-03-09T19:22:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:15 smithi038 ceph-mon[92422]: pgmap v8963: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 89 KiB/s wr, 10 op/s 2023-03-09T19:22:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15846.189225 secs 2023-03-09T19:22:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:15 smithi122 ceph-mon[93869]: pgmap v8963: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 89 KiB/s wr, 10 op/s 2023-03-09T19:22:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15846.189225 secs 2023-03-09T19:22:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:15 smithi181 ceph-mon[94189]: pgmap v8963: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 89 KiB/s wr, 10 op/s 2023-03-09T19:22:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:17 smithi038 ceph-mon[92422]: pgmap v8964: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T19:22:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15848.486180 secs 2023-03-09T19:22:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:17 smithi038 ceph-mon[92422]: osdmap e774: 12 total, 12 up, 12 in 2023-03-09T19:22:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:17 smithi122 ceph-mon[93869]: pgmap v8964: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T19:22:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15848.486180 secs 2023-03-09T19:22:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:17 smithi122 ceph-mon[93869]: osdmap e774: 12 total, 12 up, 12 in 2023-03-09T19:22:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:17 smithi181 ceph-mon[94189]: pgmap v8964: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T19:22:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15848.486180 secs 2023-03-09T19:22:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:17 smithi181 ceph-mon[94189]: osdmap e774: 12 total, 12 up, 12 in 2023-03-09T19:22:18.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16310.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:22:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:20 smithi122 ceph-mon[93869]: pgmap v8966: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 413 B/s wr, 6 op/s 2023-03-09T19:22:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:20 smithi181 ceph-mon[94189]: pgmap v8966: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 413 B/s wr, 6 op/s 2023-03-09T19:22:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:20 smithi038 ceph-mon[92422]: pgmap v8966: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 413 B/s wr, 6 op/s 2023-03-09T19:22:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15851.189286 secs 2023-03-09T19:22:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15851.189286 secs 2023-03-09T19:22:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15851.189286 secs 2023-03-09T19:22:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:22 smithi122 ceph-mon[93869]: pgmap v8967: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 716 B/s wr, 6 op/s 2023-03-09T19:22:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:22 smithi181 ceph-mon[94189]: pgmap v8967: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 716 B/s wr, 6 op/s 2023-03-09T19:22:22.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:22 smithi038 ceph-mon[92422]: pgmap v8967: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 716 B/s wr, 6 op/s 2023-03-09T19:22:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15853.486306 secs 2023-03-09T19:22:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15853.486306 secs 2023-03-09T19:22:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15853.486306 secs 2023-03-09T19:22:23.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16315.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:22:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:24 smithi122 ceph-mon[93869]: pgmap v8968: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 716 B/s wr, 6 op/s 2023-03-09T19:22:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:24 smithi181 ceph-mon[94189]: pgmap v8968: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 716 B/s wr, 6 op/s 2023-03-09T19:22:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:24 smithi038 ceph-mon[92422]: pgmap v8968: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 716 B/s wr, 6 op/s 2023-03-09T19:22:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15856.189347 secs 2023-03-09T19:22:26.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:26 smithi122 ceph-mon[93869]: pgmap v8969: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 716 B/s wr, 6 op/s 2023-03-09T19:22:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15856.189347 secs 2023-03-09T19:22:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:26 smithi181 ceph-mon[94189]: pgmap v8969: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 716 B/s wr, 6 op/s 2023-03-09T19:22:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15856.189347 secs 2023-03-09T19:22:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:26 smithi038 ceph-mon[92422]: pgmap v8969: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 716 B/s wr, 6 op/s 2023-03-09T19:22:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:28 smithi122 ceph-mon[93869]: pgmap v8970: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 74 KiB/s wr, 7 op/s 2023-03-09T19:22:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15858.486412 secs 2023-03-09T19:22:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:28 smithi181 ceph-mon[94189]: pgmap v8970: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 74 KiB/s wr, 7 op/s 2023-03-09T19:22:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15858.486412 secs 2023-03-09T19:22:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:28 smithi038 ceph-mon[92422]: pgmap v8970: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 74 KiB/s wr, 7 op/s 2023-03-09T19:22:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15858.486412 secs 2023-03-09T19:22:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16320.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:22:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:30 smithi122 ceph-mon[93869]: pgmap v8971: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 67 KiB/s wr, 7 op/s 2023-03-09T19:22:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:30 smithi181 ceph-mon[94189]: pgmap v8971: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 67 KiB/s wr, 7 op/s 2023-03-09T19:22:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:30 smithi038 ceph-mon[92422]: pgmap v8971: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 67 KiB/s wr, 7 op/s 2023-03-09T19:22:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15861.189409 secs 2023-03-09T19:22:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15861.189409 secs 2023-03-09T19:22:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15861.189409 secs 2023-03-09T19:22:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:32 smithi122 ceph-mon[93869]: pgmap v8972: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 99 KiB/s wr, 9 op/s 2023-03-09T19:22:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:32 smithi181 ceph-mon[94189]: pgmap v8972: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 99 KiB/s wr, 9 op/s 2023-03-09T19:22:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:32 smithi038 ceph-mon[92422]: pgmap v8972: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 99 KiB/s wr, 9 op/s 2023-03-09T19:22:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15863.486522 secs 2023-03-09T19:22:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15863.486522 secs 2023-03-09T19:22:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15863.486522 secs 2023-03-09T19:22:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16325.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:22:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:34 smithi122 ceph-mon[93869]: pgmap v8973: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 98 KiB/s wr, 6 op/s 2023-03-09T19:22:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:34 smithi181 ceph-mon[94189]: pgmap v8973: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 98 KiB/s wr, 6 op/s 2023-03-09T19:22:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:34 smithi038 ceph-mon[92422]: pgmap v8973: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 98 KiB/s wr, 6 op/s 2023-03-09T19:22:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15866.189467 secs 2023-03-09T19:22:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:36 smithi122 ceph-mon[93869]: pgmap v8974: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 98 KiB/s wr, 6 op/s 2023-03-09T19:22:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15866.189467 secs 2023-03-09T19:22:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:36 smithi181 ceph-mon[94189]: pgmap v8974: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 98 KiB/s wr, 6 op/s 2023-03-09T19:22:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15866.189467 secs 2023-03-09T19:22:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:36 smithi038 ceph-mon[92422]: pgmap v8974: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 98 KiB/s wr, 6 op/s 2023-03-09T19:22:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:38 smithi122 ceph-mon[93869]: pgmap v8975: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 210 KiB/s wr, 9 op/s 2023-03-09T19:22:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:38 smithi181 ceph-mon[94189]: pgmap v8975: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 210 KiB/s wr, 9 op/s 2023-03-09T19:22:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:38 smithi038 ceph-mon[92422]: pgmap v8975: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 210 KiB/s wr, 9 op/s 2023-03-09T19:22:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16330.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:22:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:40 smithi122 ceph-mon[93869]: pgmap v8976: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 149 KiB/s wr, 7 op/s 2023-03-09T19:22:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:40 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:22:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:40 smithi181 ceph-mon[94189]: pgmap v8976: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 149 KiB/s wr, 7 op/s 2023-03-09T19:22:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:40 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:22:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:40 smithi038 ceph-mon[92422]: pgmap v8976: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 149 KiB/s wr, 7 op/s 2023-03-09T19:22:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:40 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:22:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15871.189525 secs 2023-03-09T19:22:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:41 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:22:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:22:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:22:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:22:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15871.189525 secs 2023-03-09T19:22:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:41 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:22:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:22:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:22:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:22:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15871.189525 secs 2023-03-09T19:22:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:41 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:22:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:22:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:22:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:22:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:42 smithi122 ceph-mon[93869]: pgmap v8977: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 219 KiB/s wr, 10 op/s 2023-03-09T19:22:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:22:42.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:22:42.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:22:42.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:22:42.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:22:42.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:22:42.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:22:42.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:22:42.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:22:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:42 smithi181 ceph-mon[94189]: pgmap v8977: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 219 KiB/s wr, 10 op/s 2023-03-09T19:22:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:22:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:22:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:22:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:22:42.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:22:42.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:22:42.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:22:42.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:22:42.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:22:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:42 smithi038 ceph-mon[92422]: pgmap v8977: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 219 KiB/s wr, 10 op/s 2023-03-09T19:22:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:22:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:22:42.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:22:42.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:22:42.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:22:42.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:22:42.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:22:42.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:22:42.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:22:43.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16335.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:22:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:43 smithi122 ceph-mon[93869]: pgmap v8978: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 182 KiB/s wr, 8 op/s 2023-03-09T19:22:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:43 smithi181 ceph-mon[94189]: pgmap v8978: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 182 KiB/s wr, 8 op/s 2023-03-09T19:22:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:43 smithi038 ceph-mon[92422]: pgmap v8978: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 182 KiB/s wr, 8 op/s 2023-03-09T19:22:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15876.189585 secs 2023-03-09T19:22:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:45 smithi122 ceph-mon[93869]: pgmap v8979: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 182 KiB/s wr, 8 op/s 2023-03-09T19:22:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15876.189585 secs 2023-03-09T19:22:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:45 smithi181 ceph-mon[94189]: pgmap v8979: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 182 KiB/s wr, 8 op/s 2023-03-09T19:22:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15876.189585 secs 2023-03-09T19:22:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:45 smithi038 ceph-mon[92422]: pgmap v8979: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 182 KiB/s wr, 8 op/s 2023-03-09T19:22:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:47 smithi122 ceph-mon[93869]: pgmap v8980: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 216 KiB/s wr, 12 op/s 2023-03-09T19:22:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15878.486889 secs 2023-03-09T19:22:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:47 smithi181 ceph-mon[94189]: pgmap v8980: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 216 KiB/s wr, 12 op/s 2023-03-09T19:22:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15878.486889 secs 2023-03-09T19:22:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:47 smithi038 ceph-mon[92422]: pgmap v8980: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 216 KiB/s wr, 12 op/s 2023-03-09T19:22:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15878.486889 secs 2023-03-09T19:22:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16340.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:22:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:48 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:22:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:22:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:22:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:48 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:22:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:22:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:22:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:48 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:22:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:22:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:22:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:49 smithi122 ceph-mon[93869]: pgmap v8981: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 105 KiB/s wr, 9 op/s 2023-03-09T19:22:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:49 smithi181 ceph-mon[94189]: pgmap v8981: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 105 KiB/s wr, 9 op/s 2023-03-09T19:22:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:49 smithi038 ceph-mon[92422]: pgmap v8981: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 105 KiB/s wr, 9 op/s 2023-03-09T19:22:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15881.189648 secs 2023-03-09T19:22:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15881.189648 secs 2023-03-09T19:22:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15881.189648 secs 2023-03-09T19:22:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:51 smithi181 ceph-mon[94189]: pgmap v8982: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 280 KiB/s wr, 12 op/s 2023-03-09T19:22:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:51 smithi038 ceph-mon[92422]: pgmap v8982: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 280 KiB/s wr, 12 op/s 2023-03-09T19:22:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:51 smithi122 ceph-mon[93869]: pgmap v8982: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 280 KiB/s wr, 12 op/s 2023-03-09T19:22:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15883.487006 secs 2023-03-09T19:22:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15883.487006 secs 2023-03-09T19:22:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15883.487006 secs 2023-03-09T19:22:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16345.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:22:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:53 smithi038 ceph-mon[92422]: pgmap v8983: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 241 KiB/s wr, 9 op/s 2023-03-09T19:22:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:53 smithi122 ceph-mon[93869]: pgmap v8983: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 241 KiB/s wr, 9 op/s 2023-03-09T19:22:54.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:53 smithi181 ceph-mon[94189]: pgmap v8983: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 241 KiB/s wr, 9 op/s 2023-03-09T19:22:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15886.189731 secs 2023-03-09T19:22:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:55 smithi038 ceph-mon[92422]: pgmap v8984: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 241 KiB/s wr, 9 op/s 2023-03-09T19:22:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15886.189731 secs 2023-03-09T19:22:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:55 smithi122 ceph-mon[93869]: pgmap v8984: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 241 KiB/s wr, 9 op/s 2023-03-09T19:22:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15886.189731 secs 2023-03-09T19:22:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:55 smithi181 ceph-mon[94189]: pgmap v8984: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 241 KiB/s wr, 9 op/s 2023-03-09T19:22:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:57 smithi038 ceph-mon[92422]: pgmap v8985: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 241 KiB/s wr, 12 op/s 2023-03-09T19:22:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:22:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15888.487146 secs 2023-03-09T19:22:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:57 smithi122 ceph-mon[93869]: pgmap v8985: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 241 KiB/s wr, 12 op/s 2023-03-09T19:22:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:22:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15888.487146 secs 2023-03-09T19:22:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:57 smithi181 ceph-mon[94189]: pgmap v8985: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 241 KiB/s wr, 12 op/s 2023-03-09T19:22:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:22:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15888.487146 secs 2023-03-09T19:22:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16350.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:23:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:00 smithi122 ceph-mon[93869]: pgmap v8986: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 228 KiB/s wr, 9 op/s 2023-03-09T19:23:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:00 smithi181 ceph-mon[94189]: pgmap v8986: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 228 KiB/s wr, 9 op/s 2023-03-09T19:23:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:00 smithi038 ceph-mon[92422]: pgmap v8986: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 228 KiB/s wr, 9 op/s 2023-03-09T19:23:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15891.189789 secs 2023-03-09T19:23:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15891.189789 secs 2023-03-09T19:23:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15891.189789 secs 2023-03-09T19:23:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:02 smithi122 ceph-mon[93869]: pgmap v8987: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 253 KiB/s wr, 11 op/s 2023-03-09T19:23:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:02 smithi181 ceph-mon[94189]: pgmap v8987: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 253 KiB/s wr, 11 op/s 2023-03-09T19:23:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:02 smithi038 ceph-mon[92422]: pgmap v8987: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 253 KiB/s wr, 11 op/s 2023-03-09T19:23:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15893.487201 secs 2023-03-09T19:23:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15893.487201 secs 2023-03-09T19:23:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15893.487201 secs 2023-03-09T19:23:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16355.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:23:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:04 smithi122 ceph-mon[93869]: pgmap v8988: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 293 KiB/s wr, 10 op/s 2023-03-09T19:23:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:04 smithi181 ceph-mon[94189]: pgmap v8988: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 293 KiB/s wr, 10 op/s 2023-03-09T19:23:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:04 smithi038 ceph-mon[92422]: pgmap v8988: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 293 KiB/s wr, 10 op/s 2023-03-09T19:23:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15896.189849 secs 2023-03-09T19:23:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:06 smithi122 ceph-mon[93869]: pgmap v8989: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 284 KiB/s wr, 21 op/s 2023-03-09T19:23:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15896.189849 secs 2023-03-09T19:23:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:06 smithi181 ceph-mon[94189]: pgmap v8989: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 284 KiB/s wr, 21 op/s 2023-03-09T19:23:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15896.189849 secs 2023-03-09T19:23:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:06 smithi038 ceph-mon[92422]: pgmap v8989: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 284 KiB/s wr, 21 op/s 2023-03-09T19:23:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:08 smithi122 ceph-mon[93869]: pgmap v8990: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 306 KiB/s wr, 26 op/s 2023-03-09T19:23:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15898.487310 secs 2023-03-09T19:23:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:08 smithi181 ceph-mon[94189]: pgmap v8990: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 306 KiB/s wr, 26 op/s 2023-03-09T19:23:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15898.487310 secs 2023-03-09T19:23:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:08 smithi038 ceph-mon[92422]: pgmap v8990: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 306 KiB/s wr, 26 op/s 2023-03-09T19:23:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15898.487310 secs 2023-03-09T19:23:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16360.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:23:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:10 smithi122 ceph-mon[93869]: pgmap v8991: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 343 KiB/s wr, 24 op/s 2023-03-09T19:23:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:10 smithi122 ceph-mon[93869]: osdmap e775: 12 total, 12 up, 12 in 2023-03-09T19:23:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:10 smithi181 ceph-mon[94189]: pgmap v8991: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 343 KiB/s wr, 24 op/s 2023-03-09T19:23:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:10 smithi181 ceph-mon[94189]: osdmap e775: 12 total, 12 up, 12 in 2023-03-09T19:23:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:10 smithi038 ceph-mon[92422]: pgmap v8991: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 343 KiB/s wr, 24 op/s 2023-03-09T19:23:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:10 smithi038 ceph-mon[92422]: osdmap e775: 12 total, 12 up, 12 in 2023-03-09T19:23:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15901.189910 secs 2023-03-09T19:23:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15901.189910 secs 2023-03-09T19:23:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15901.189910 secs 2023-03-09T19:23:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:12 smithi122 ceph-mon[93869]: pgmap v8993: 129 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+snaptrim, 124 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 371 KiB/s wr, 28 op/s 2023-03-09T19:23:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:12 smithi181 ceph-mon[94189]: pgmap v8993: 129 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+snaptrim, 124 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 371 KiB/s wr, 28 op/s 2023-03-09T19:23:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:12 smithi038 ceph-mon[92422]: pgmap v8993: 129 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+snaptrim, 124 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 371 KiB/s wr, 28 op/s 2023-03-09T19:23:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15903.487390 secs 2023-03-09T19:23:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15903.487390 secs 2023-03-09T19:23:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15903.487390 secs 2023-03-09T19:23:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16365.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:23:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:14 smithi122 ceph-mon[93869]: pgmap v8994: 129 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+snaptrim, 124 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 252 KiB/s wr, 27 op/s 2023-03-09T19:23:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:14 smithi181 ceph-mon[94189]: pgmap v8994: 129 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+snaptrim, 124 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 252 KiB/s wr, 27 op/s 2023-03-09T19:23:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:14 smithi038 ceph-mon[92422]: pgmap v8994: 129 pgs: 4 active+clean+snaptrim_wait, 1 active+clean+snaptrim, 124 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 252 KiB/s wr, 27 op/s 2023-03-09T19:23:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15906.189968 secs 2023-03-09T19:23:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:16 smithi122 ceph-mon[93869]: pgmap v8995: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 226 KiB/s wr, 13 op/s 2023-03-09T19:23:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15906.189968 secs 2023-03-09T19:23:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:16 smithi181 ceph-mon[94189]: pgmap v8995: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 226 KiB/s wr, 13 op/s 2023-03-09T19:23:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15906.189968 secs 2023-03-09T19:23:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:16 smithi038 ceph-mon[92422]: pgmap v8995: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 226 KiB/s wr, 13 op/s 2023-03-09T19:23:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:18 smithi122 ceph-mon[93869]: pgmap v8996: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 243 KiB/s wr, 10 op/s 2023-03-09T19:23:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15908.487470 secs 2023-03-09T19:23:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:18 smithi122 ceph-mon[93869]: osdmap e776: 12 total, 12 up, 12 in 2023-03-09T19:23:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:18 smithi181 ceph-mon[94189]: pgmap v8996: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 243 KiB/s wr, 10 op/s 2023-03-09T19:23:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15908.487470 secs 2023-03-09T19:23:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:18 smithi181 ceph-mon[94189]: osdmap e776: 12 total, 12 up, 12 in 2023-03-09T19:23:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:18 smithi038 ceph-mon[92422]: pgmap v8996: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 243 KiB/s wr, 10 op/s 2023-03-09T19:23:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15908.487470 secs 2023-03-09T19:23:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:18 smithi038 ceph-mon[92422]: osdmap e776: 12 total, 12 up, 12 in 2023-03-09T19:23:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16370.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:23:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:20 smithi122 ceph-mon[93869]: pgmap v8998: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 203 KiB/s wr, 9 op/s 2023-03-09T19:23:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:20 smithi181 ceph-mon[94189]: pgmap v8998: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 203 KiB/s wr, 9 op/s 2023-03-09T19:23:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:20 smithi038 ceph-mon[92422]: pgmap v8998: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 203 KiB/s wr, 9 op/s 2023-03-09T19:23:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15911.190009 secs 2023-03-09T19:23:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15911.190009 secs 2023-03-09T19:23:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15911.190009 secs 2023-03-09T19:23:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:22 smithi122 ceph-mon[93869]: pgmap v8999: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 185 KiB/s wr, 8 op/s 2023-03-09T19:23:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:22 smithi181 ceph-mon[94189]: pgmap v8999: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 185 KiB/s wr, 8 op/s 2023-03-09T19:23:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:22 smithi038 ceph-mon[92422]: pgmap v8999: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 185 KiB/s wr, 8 op/s 2023-03-09T19:23:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15913.487587 secs 2023-03-09T19:23:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15913.487587 secs 2023-03-09T19:23:23.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15913.487587 secs 2023-03-09T19:23:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16375.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:23:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:24 smithi122 ceph-mon[93869]: pgmap v9000: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 46 KiB/s wr, 6 op/s 2023-03-09T19:23:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:24 smithi181 ceph-mon[94189]: pgmap v9000: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 46 KiB/s wr, 6 op/s 2023-03-09T19:23:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:24 smithi038 ceph-mon[92422]: pgmap v9000: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 46 KiB/s wr, 6 op/s 2023-03-09T19:23:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15916.190068 secs 2023-03-09T19:23:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:26 smithi122 ceph-mon[93869]: pgmap v9001: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 46 KiB/s wr, 6 op/s 2023-03-09T19:23:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15916.190068 secs 2023-03-09T19:23:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:26 smithi181 ceph-mon[94189]: pgmap v9001: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 46 KiB/s wr, 6 op/s 2023-03-09T19:23:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15916.190068 secs 2023-03-09T19:23:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:26 smithi038 ceph-mon[92422]: pgmap v9001: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 46 KiB/s wr, 6 op/s 2023-03-09T19:23:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:28 smithi122 ceph-mon[93869]: pgmap v9002: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 68 KiB/s wr, 7 op/s 2023-03-09T19:23:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15918.487701 secs 2023-03-09T19:23:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:28 smithi181 ceph-mon[94189]: pgmap v9002: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 68 KiB/s wr, 7 op/s 2023-03-09T19:23:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15918.487701 secs 2023-03-09T19:23:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:28 smithi038 ceph-mon[92422]: pgmap v9002: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 68 KiB/s wr, 7 op/s 2023-03-09T19:23:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15918.487701 secs 2023-03-09T19:23:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16380.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:23:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:30 smithi122 ceph-mon[93869]: pgmap v9003: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 61 KiB/s wr, 7 op/s 2023-03-09T19:23:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:30 smithi181 ceph-mon[94189]: pgmap v9003: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 61 KiB/s wr, 7 op/s 2023-03-09T19:23:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:30 smithi038 ceph-mon[92422]: pgmap v9003: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 61 KiB/s wr, 7 op/s 2023-03-09T19:23:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15921.190132 secs 2023-03-09T19:23:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15921.190132 secs 2023-03-09T19:23:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15921.190132 secs 2023-03-09T19:23:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:32 smithi122 ceph-mon[93869]: pgmap v9004: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 56 KiB/s wr, 8 op/s 2023-03-09T19:23:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:32 smithi181 ceph-mon[94189]: pgmap v9004: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 56 KiB/s wr, 8 op/s 2023-03-09T19:23:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:32 smithi038 ceph-mon[92422]: pgmap v9004: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 56 KiB/s wr, 8 op/s 2023-03-09T19:23:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15923.487821 secs 2023-03-09T19:23:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15923.487821 secs 2023-03-09T19:23:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15923.487821 secs 2023-03-09T19:23:33.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16385.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:23:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:34 smithi122 ceph-mon[93869]: pgmap v9005: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 56 KiB/s wr, 7 op/s 2023-03-09T19:23:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:34 smithi181 ceph-mon[94189]: pgmap v9005: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 56 KiB/s wr, 7 op/s 2023-03-09T19:23:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:34 smithi038 ceph-mon[92422]: pgmap v9005: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 56 KiB/s wr, 7 op/s 2023-03-09T19:23:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15926.190192 secs 2023-03-09T19:23:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:36 smithi122 ceph-mon[93869]: pgmap v9006: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 56 KiB/s wr, 6 op/s 2023-03-09T19:23:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15926.190192 secs 2023-03-09T19:23:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:36 smithi181 ceph-mon[94189]: pgmap v9006: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 56 KiB/s wr, 6 op/s 2023-03-09T19:23:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15926.190192 secs 2023-03-09T19:23:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:36 smithi038 ceph-mon[92422]: pgmap v9006: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 56 KiB/s wr, 6 op/s 2023-03-09T19:23:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:38 smithi122 ceph-mon[93869]: pgmap v9007: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 234 KiB/s wr, 10 op/s 2023-03-09T19:23:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15928.487952 secs 2023-03-09T19:23:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:38 smithi181 ceph-mon[94189]: pgmap v9007: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 234 KiB/s wr, 10 op/s 2023-03-09T19:23:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15928.487952 secs 2023-03-09T19:23:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:38 smithi038 ceph-mon[92422]: pgmap v9007: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 234 KiB/s wr, 10 op/s 2023-03-09T19:23:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15928.487952 secs 2023-03-09T19:23:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16390.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:23:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:40 smithi122 ceph-mon[93869]: pgmap v9008: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 179 KiB/s wr, 7 op/s 2023-03-09T19:23:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:40 smithi181 ceph-mon[94189]: pgmap v9008: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 179 KiB/s wr, 7 op/s 2023-03-09T19:23:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:40 smithi038 ceph-mon[92422]: pgmap v9008: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 179 KiB/s wr, 7 op/s 2023-03-09T19:23:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15931.190253 secs 2023-03-09T19:23:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15931.190253 secs 2023-03-09T19:23:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15931.190253 secs 2023-03-09T19:23:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:42 smithi122 ceph-mon[93869]: pgmap v9009: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 326 KiB/s rd, 310 KiB/s wr, 11 op/s 2023-03-09T19:23:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:23:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:42 smithi181 ceph-mon[94189]: pgmap v9009: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 326 KiB/s rd, 310 KiB/s wr, 11 op/s 2023-03-09T19:23:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:23:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:42 smithi038 ceph-mon[92422]: pgmap v9009: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 326 KiB/s rd, 310 KiB/s wr, 11 op/s 2023-03-09T19:23:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:23:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15933.488028 secs 2023-03-09T19:23:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15933.488028 secs 2023-03-09T19:23:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15933.488028 secs 2023-03-09T19:23:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16395.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:23:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:44 smithi122 ceph-mon[93869]: pgmap v9010: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 318 KiB/s wr, 9 op/s 2023-03-09T19:23:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:44 smithi181 ceph-mon[94189]: pgmap v9010: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 318 KiB/s wr, 9 op/s 2023-03-09T19:23:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:44 smithi038 ceph-mon[92422]: pgmap v9010: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 318 KiB/s wr, 9 op/s 2023-03-09T19:23:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15936.190314 secs 2023-03-09T19:23:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:46 smithi122 ceph-mon[93869]: pgmap v9011: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 318 KiB/s wr, 9 op/s 2023-03-09T19:23:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:23:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:23:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:23:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:23:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:23:46.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:23:46.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:23:46.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:23:46.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:23:46.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:23:46.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:23:46.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:23:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15936.190314 secs 2023-03-09T19:23:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:46 smithi181 ceph-mon[94189]: pgmap v9011: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 318 KiB/s wr, 9 op/s 2023-03-09T19:23:46.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:23:46.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:23:46.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:23:46.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:23:46.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:23:46.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:23:46.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:23:46.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:23:46.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:23:46.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:23:46.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:23:46.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:23:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15936.190314 secs 2023-03-09T19:23:46.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:46 smithi038 ceph-mon[92422]: pgmap v9011: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 318 KiB/s wr, 9 op/s 2023-03-09T19:23:46.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:23:46.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:23:46.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:23:46.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:23:46.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:23:46.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:23:46.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:23:46.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:23:46.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:23:46.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:23:46.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:23:46.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:23:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:47 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T19:23:47.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:47 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi038 to 150.4M 2023-03-09T19:23:47.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:47 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T19:23:47.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:47 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi038 to 157753393: error parsing value: Value '157753393' is below minimum 939524096 2023-03-09T19:23:47.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:47 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T19:23:47.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:47 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi038 to 150.4M 2023-03-09T19:23:47.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:47 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T19:23:47.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:47 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi038 to 157753393: error parsing value: Value '157753393' is below minimum 939524096 2023-03-09T19:23:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:47 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T19:23:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:47 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi038 to 150.4M 2023-03-09T19:23:47.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:47 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T19:23:47.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:47 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi038 to 157753393: error parsing value: Value '157753393' is below minimum 939524096 2023-03-09T19:23:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:48 smithi122 ceph-mon[93869]: pgmap v9012: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 514 KiB/s rd, 500 KiB/s wr, 15 op/s 2023-03-09T19:23:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15938.488141 secs 2023-03-09T19:23:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:48 smithi181 ceph-mon[94189]: pgmap v9012: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 514 KiB/s rd, 500 KiB/s wr, 15 op/s 2023-03-09T19:23:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15938.488141 secs 2023-03-09T19:23:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:48 smithi038 ceph-mon[92422]: pgmap v9012: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 514 KiB/s rd, 500 KiB/s wr, 15 op/s 2023-03-09T19:23:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15938.488141 secs 2023-03-09T19:23:48.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16400.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:23:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:50 smithi122 ceph-mon[93869]: pgmap v9013: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 335 KiB/s rd, 322 KiB/s wr, 12 op/s 2023-03-09T19:23:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:50 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:23:50.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:50 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:23:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:50 smithi181 ceph-mon[94189]: pgmap v9013: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 335 KiB/s rd, 322 KiB/s wr, 12 op/s 2023-03-09T19:23:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:50 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:23:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:50 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:23:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:50 smithi038 ceph-mon[92422]: pgmap v9013: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 335 KiB/s rd, 322 KiB/s wr, 12 op/s 2023-03-09T19:23:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:50 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:23:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:50 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:23:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15941.190374 secs 2023-03-09T19:23:51.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:23:51.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:23:51.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:23:51.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:23:51.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:23:51.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:23:51.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:23:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15941.190374 secs 2023-03-09T19:23:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:23:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:23:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:23:51.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:23:51.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:23:51.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:23:51.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:23:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15941.190374 secs 2023-03-09T19:23:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:23:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:23:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:23:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:23:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:23:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:23:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:23:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:52 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T19:23:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:52 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T19:23:52.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:52 smithi122 ceph-mon[93869]: pgmap v9014: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 387 KiB/s wr, 15 op/s 2023-03-09T19:23:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:52 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T19:23:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:52 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T19:23:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:52 smithi181 ceph-mon[94189]: pgmap v9014: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 387 KiB/s wr, 15 op/s 2023-03-09T19:23:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:52 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T19:23:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:52 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T19:23:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:52 smithi038 ceph-mon[92422]: pgmap v9014: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 387 KiB/s wr, 15 op/s 2023-03-09T19:23:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15943.488256 secs 2023-03-09T19:23:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15943.488256 secs 2023-03-09T19:23:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15943.488256 secs 2023-03-09T19:23:53.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16405.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:23:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:54 smithi122 ceph-mon[93869]: pgmap v9015: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 255 KiB/s wr, 11 op/s 2023-03-09T19:23:54.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:54 smithi181 ceph-mon[94189]: pgmap v9015: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 255 KiB/s wr, 11 op/s 2023-03-09T19:23:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:54 smithi038 ceph-mon[92422]: pgmap v9015: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 255 KiB/s wr, 11 op/s 2023-03-09T19:23:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15946.190432 secs 2023-03-09T19:23:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:56 smithi122 ceph-mon[93869]: pgmap v9016: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 248 KiB/s wr, 10 op/s 2023-03-09T19:23:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15946.190432 secs 2023-03-09T19:23:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:56 smithi181 ceph-mon[94189]: pgmap v9016: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 248 KiB/s wr, 10 op/s 2023-03-09T19:23:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15946.190432 secs 2023-03-09T19:23:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:56 smithi038 ceph-mon[92422]: pgmap v9016: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 248 KiB/s wr, 10 op/s 2023-03-09T19:23:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:58 smithi122 ceph-mon[93869]: pgmap v9017: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 331 KiB/s wr, 13 op/s 2023-03-09T19:23:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:23:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15948.488381 secs 2023-03-09T19:23:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:58 smithi181 ceph-mon[94189]: pgmap v9017: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 331 KiB/s wr, 13 op/s 2023-03-09T19:23:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:23:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15948.488381 secs 2023-03-09T19:23:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:58 smithi038 ceph-mon[92422]: pgmap v9017: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 331 KiB/s wr, 13 op/s 2023-03-09T19:23:58.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:23:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15948.488381 secs 2023-03-09T19:23:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16410.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:24:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:24:00 smithi122 ceph-mon[93869]: pgmap v9018: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 148 KiB/s wr, 8 op/s 2023-03-09T19:24:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:24:00 smithi181 ceph-mon[94189]: pgmap v9018: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 148 KiB/s wr, 8 op/s 2023-03-09T19:24:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:24:00 smithi038 ceph-mon[92422]: pgmap v9018: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 148 KiB/s wr, 8 op/s 2023-03-09T19:24:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:24:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15951.190493 secs 2023-03-09T19:24:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:24:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15951.190493 secs 2023-03-09T19:24:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:24:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15951.190493 secs 2023-03-09T19:24:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:24:02 smithi122 ceph-mon[93869]: pgmap v9019: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 215 KiB/s wr, 13 op/s 2023-03-09T19:24:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:24:02 smithi181 ceph-mon[94189]: pgmap v9019: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 215 KiB/s wr, 13 op/s 2023-03-09T19:24:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:24:02 smithi038 ceph-mon[92422]: pgmap v9019: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 215 KiB/s wr, 13 op/s 2023-03-09T19:24:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:24:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15953.488498 secs 2023-03-09T19:24:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:24:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15953.488498 secs 2023-03-09T19:24:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:24:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15953.488498 secs 2023-03-09T19:24:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16415.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:24:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:24:04 smithi122 ceph-mon[93869]: pgmap v9020: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 150 KiB/s wr, 9 op/s 2023-03-09T19:24:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:24:04 smithi181 ceph-mon[94189]: pgmap v9020: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 150 KiB/s wr, 9 op/s 2023-03-09T19:24:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:24:04 smithi038 ceph-mon[92422]: pgmap v9020: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 150 KiB/s wr, 9 op/s 2023-03-09T19:24:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:24:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15956.190554 secs 2023-03-09T19:24:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:24:06 smithi122 ceph-mon[93869]: pgmap v9021: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 172 KiB/s wr, 21 op/s 2023-03-09T19:24:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:24:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15956.190554 secs 2023-03-09T19:24:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:24:06 smithi181 ceph-mon[94189]: pgmap v9021: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 172 KiB/s wr, 21 op/s 2023-03-09T19:24:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:24:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15956.190554 secs 2023-03-09T19:24:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:24:06 smithi038 ceph-mon[92422]: pgmap v9021: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 172 KiB/s wr, 21 op/s 2023-03-09T19:24:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:24:08 smithi122 ceph-mon[93869]: pgmap v9022: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 172 KiB/s wr, 24 op/s 2023-03-09T19:24:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:24:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15958.488621 secs 2023-03-09T19:24:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:24:08 smithi181 ceph-mon[94189]: pgmap v9022: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 172 KiB/s wr, 24 op/s 2023-03-09T19:24:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:24:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15958.488621 secs 2023-03-09T19:24:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:24:08 smithi038 ceph-mon[92422]: pgmap v9022: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 172 KiB/s wr, 24 op/s 2023-03-09T19:24:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:24:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15958.488621 secs 2023-03-09T19:24:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16420.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:24:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:24:10 smithi122 ceph-mon[93869]: pgmap v9023: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 89 KiB/s wr, 21 op/s 2023-03-09T19:24:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:24:10 smithi122 ceph-mon[93869]: osdmap e777: 12 total, 12 up, 12 in 2023-03-09T19:24:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:24:10 smithi181 ceph-mon[94189]: pgmap v9023: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 89 KiB/s wr, 21 op/s 2023-03-09T19:24:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:24:10 smithi181 ceph-mon[94189]: osdmap e777: 12 total, 12 up, 12 in 2023-03-09T19:24:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:24:10 smithi038 ceph-mon[92422]: pgmap v9023: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 89 KiB/s wr, 21 op/s 2023-03-09T19:24:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:24:10 smithi038 ceph-mon[92422]: osdmap e777: 12 total, 12 up, 12 in 2023-03-09T19:24:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:24:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15961.190616 secs 2023-03-09T19:24:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:24:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15961.190616 secs 2023-03-09T19:24:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:24:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15961.190616 secs 2023-03-09T19:24:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:24:12 smithi122 ceph-mon[93869]: pgmap v9025: 129 pgs: 10 active+clean+snaptrim_wait, 119 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 27 KiB/s wr, 22 op/s 2023-03-09T19:24:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:24:12 smithi181 ceph-mon[94189]: pgmap v9025: 129 pgs: 10 active+clean+snaptrim_wait, 119 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 27 KiB/s wr, 22 op/s 2023-03-09T19:24:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:24:12 smithi038 ceph-mon[92422]: pgmap v9025: 129 pgs: 10 active+clean+snaptrim_wait, 119 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 27 KiB/s wr, 22 op/s 2023-03-09T19:24:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:24:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15963.488755 secs 2023-03-09T19:24:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:24:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15963.488755 secs 2023-03-09T19:24:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:24:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15963.488755 secs 2023-03-09T19:24:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16425.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:24:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:24:14 smithi122 ceph-mon[93869]: pgmap v9026: 129 pgs: 10 active+clean+snaptrim_wait, 119 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 27 KiB/s wr, 22 op/s 2023-03-09T19:24:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:24:14 smithi181 ceph-mon[94189]: pgmap v9026: 129 pgs: 10 active+clean+snaptrim_wait, 119 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 27 KiB/s wr, 22 op/s 2023-03-09T19:24:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:24:14 smithi038 ceph-mon[92422]: pgmap v9026: 129 pgs: 10 active+clean+snaptrim_wait, 119 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 27 KiB/s wr, 22 op/s 2023-03-09T19:24:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:24:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15966.190676 secs 2023-03-09T19:24:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:24:16 smithi122 ceph-mon[93869]: pgmap v9027: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1023 B/s wr, 7 op/s 2023-03-09T19:24:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:24:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15966.190676 secs 2023-03-09T19:24:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:24:16 smithi181 ceph-mon[94189]: pgmap v9027: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1023 B/s wr, 7 op/s 2023-03-09T19:24:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:24:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15966.190676 secs 2023-03-09T19:24:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:24:16 smithi038 ceph-mon[92422]: pgmap v9027: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1023 B/s wr, 7 op/s 2023-03-09T19:24:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:24:18 smithi122 ceph-mon[93869]: pgmap v9028: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2023-03-09T19:24:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:24:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15968.488824 secs 2023-03-09T19:24:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:24:18 smithi122 ceph-mon[93869]: osdmap e778: 12 total, 12 up, 12 in 2023-03-09T19:24:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:24:18 smithi181 ceph-mon[94189]: pgmap v9028: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2023-03-09T19:24:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:24:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15968.488824 secs 2023-03-09T19:24:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:24:18 smithi181 ceph-mon[94189]: osdmap e778: 12 total, 12 up, 12 in 2023-03-09T19:24:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:24:18 smithi038 ceph-mon[92422]: pgmap v9028: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2023-03-09T19:24:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:24:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15968.488824 secs 2023-03-09T19:24:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:24:18 smithi038 ceph-mon[92422]: osdmap e778: 12 total, 12 up, 12 in 2023-03-09T19:24:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16430.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:24:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:24:20 smithi122 ceph-mon[93869]: pgmap v9030: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 417 B/s wr, 7 op/s 2023-03-09T19:24:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:24:20 smithi181 ceph-mon[94189]: pgmap v9030: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 417 B/s wr, 7 op/s 2023-03-09T19:24:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:24:20 smithi038 ceph-mon[92422]: pgmap v9030: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 417 B/s wr, 7 op/s 2023-03-09T19:24:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:24:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15971.190737 secs 2023-03-09T19:24:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:24:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15971.190737 secs 2023-03-09T19:24:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:24:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15971.190737 secs 2023-03-09T19:24:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:24:22 smithi122 ceph-mon[93869]: pgmap v9031: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 53 KiB/s wr, 7 op/s 2023-03-09T19:24:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:24:22 smithi181 ceph-mon[94189]: pgmap v9031: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 53 KiB/s wr, 7 op/s 2023-03-09T19:24:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:24:22 smithi038 ceph-mon[92422]: pgmap v9031: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 53 KiB/s wr, 7 op/s 2023-03-09T19:24:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:24:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15973.488923 secs 2023-03-09T19:24:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:24:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15973.488923 secs 2023-03-09T19:24:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:24:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15973.488923 secs 2023-03-09T19:24:23.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16435.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:24:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:24:24 smithi122 ceph-mon[93869]: pgmap v9032: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 53 KiB/s wr, 7 op/s 2023-03-09T19:24:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:24:24 smithi181 ceph-mon[94189]: pgmap v9032: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 53 KiB/s wr, 7 op/s 2023-03-09T19:24:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:24:24 smithi038 ceph-mon[92422]: pgmap v9032: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 53 KiB/s wr, 7 op/s 2023-03-09T19:24:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:24:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15976.190814 secs 2023-03-09T19:24:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:24:26 smithi122 ceph-mon[93869]: pgmap v9033: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 53 KiB/s wr, 7 op/s 2023-03-09T19:24:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:24:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15976.190814 secs 2023-03-09T19:24:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:24:26 smithi181 ceph-mon[94189]: pgmap v9033: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 53 KiB/s wr, 7 op/s 2023-03-09T19:24:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:24:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15976.190814 secs 2023-03-09T19:24:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:24:26 smithi038 ceph-mon[92422]: pgmap v9033: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 53 KiB/s wr, 7 op/s 2023-03-09T19:24:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:24:28 smithi122 ceph-mon[93869]: pgmap v9034: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 180 KiB/s wr, 8 op/s 2023-03-09T19:24:28.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:24:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15978.489040 secs 2023-03-09T19:24:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:24:28 smithi181 ceph-mon[94189]: pgmap v9034: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 180 KiB/s wr, 8 op/s 2023-03-09T19:24:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:24:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15978.489040 secs 2023-03-09T19:24:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:24:28 smithi038 ceph-mon[92422]: pgmap v9034: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 180 KiB/s wr, 8 op/s 2023-03-09T19:24:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:24:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15978.489040 secs 2023-03-09T19:24:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16440.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:24:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:24:30 smithi122 ceph-mon[93869]: pgmap v9035: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 162 KiB/s wr, 7 op/s 2023-03-09T19:24:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:24:30 smithi181 ceph-mon[94189]: pgmap v9035: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 162 KiB/s wr, 7 op/s 2023-03-09T19:24:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:24:30 smithi038 ceph-mon[92422]: pgmap v9035: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 162 KiB/s wr, 7 op/s 2023-03-09T19:24:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:24:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15981.190895 secs 2023-03-09T19:24:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:24:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15981.190895 secs 2023-03-09T19:24:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:24:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15981.190895 secs 2023-03-09T19:24:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:24:32 smithi122 ceph-mon[93869]: pgmap v9036: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 151 KiB/s wr, 9 op/s 2023-03-09T19:24:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:24:32 smithi181 ceph-mon[94189]: pgmap v9036: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 151 KiB/s wr, 9 op/s 2023-03-09T19:24:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:24:32 smithi038 ceph-mon[92422]: pgmap v9036: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 151 KiB/s wr, 9 op/s 2023-03-09T19:24:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:24:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15983.489156 secs 2023-03-09T19:24:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:24:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15983.489156 secs 2023-03-09T19:24:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:24:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15983.489156 secs 2023-03-09T19:24:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16445.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:24:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:24:34 smithi122 ceph-mon[93869]: pgmap v9037: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 106 KiB/s wr, 6 op/s 2023-03-09T19:24:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:24:34 smithi181 ceph-mon[94189]: pgmap v9037: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 106 KiB/s wr, 6 op/s 2023-03-09T19:24:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:24:34 smithi038 ceph-mon[92422]: pgmap v9037: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 106 KiB/s wr, 6 op/s 2023-03-09T19:24:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:24:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15986.190955 secs 2023-03-09T19:24:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:24:36 smithi122 ceph-mon[93869]: pgmap v9038: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 106 KiB/s wr, 6 op/s 2023-03-09T19:24:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:24:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15986.190955 secs 2023-03-09T19:24:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:24:36 smithi181 ceph-mon[94189]: pgmap v9038: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 106 KiB/s wr, 6 op/s 2023-03-09T19:24:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:24:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15986.190955 secs 2023-03-09T19:24:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:24:36 smithi038 ceph-mon[92422]: pgmap v9038: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 106 KiB/s wr, 6 op/s 2023-03-09T19:24:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:24:38 smithi122 ceph-mon[93869]: pgmap v9039: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 164 KiB/s wr, 8 op/s 2023-03-09T19:24:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:24:38 smithi181 ceph-mon[94189]: pgmap v9039: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 164 KiB/s wr, 8 op/s 2023-03-09T19:24:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:24:38 smithi038 ceph-mon[92422]: pgmap v9039: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 164 KiB/s wr, 8 op/s 2023-03-09T19:24:38.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16450.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:24:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:24:40 smithi038 ceph-mon[92422]: pgmap v9040: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 5 op/s 2023-03-09T19:24:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:24:40 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:24:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:24:40 smithi122 ceph-mon[93869]: pgmap v9040: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 5 op/s 2023-03-09T19:24:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:24:40 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:24:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:24:40 smithi181 ceph-mon[94189]: pgmap v9040: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 5 op/s 2023-03-09T19:24:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:24:40 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:24:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:24:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15991.190985 secs 2023-03-09T19:24:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:24:41 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:24:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:24:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:24:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:24:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:24:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:24:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15991.190985 secs 2023-03-09T19:24:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:24:41 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:24:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:24:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:24:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:24:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:24:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:24:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15991.190985 secs 2023-03-09T19:24:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:24:41 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:24:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:24:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:24:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:24:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:24:42.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:24:42 smithi122 ceph-mon[93869]: pgmap v9041: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 92 KiB/s wr, 9 op/s 2023-03-09T19:24:42.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:24:42 smithi181 ceph-mon[94189]: pgmap v9041: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 92 KiB/s wr, 9 op/s 2023-03-09T19:24:42.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:24:42 smithi038 ceph-mon[92422]: pgmap v9041: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 92 KiB/s wr, 9 op/s 2023-03-09T19:24:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16455.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:24:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:24:43 smithi122 ceph-mon[93869]: pgmap v9042: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 129 KiB/s wr, 7 op/s 2023-03-09T19:24:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:24:43 smithi181 ceph-mon[94189]: pgmap v9042: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 129 KiB/s wr, 7 op/s 2023-03-09T19:24:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:24:43 smithi038 ceph-mon[92422]: pgmap v9042: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 129 KiB/s wr, 7 op/s 2023-03-09T19:24:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:24:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 15996.191069 secs 2023-03-09T19:24:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:24:45 smithi038 ceph-mon[92422]: pgmap v9043: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 129 KiB/s wr, 7 op/s 2023-03-09T19:24:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:24:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 15996.191069 secs 2023-03-09T19:24:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:24:45 smithi122 ceph-mon[93869]: pgmap v9043: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 129 KiB/s wr, 7 op/s 2023-03-09T19:24:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:24:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 15996.191069 secs 2023-03-09T19:24:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:24:45 smithi181 ceph-mon[94189]: pgmap v9043: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 129 KiB/s wr, 7 op/s 2023-03-09T19:24:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:24:48 smithi122 ceph-mon[93869]: pgmap v9044: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 206 KiB/s wr, 11 op/s 2023-03-09T19:24:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:24:48 smithi181 ceph-mon[94189]: pgmap v9044: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 206 KiB/s wr, 11 op/s 2023-03-09T19:24:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:24:48 smithi038 ceph-mon[92422]: pgmap v9044: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 206 KiB/s wr, 11 op/s 2023-03-09T19:24:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16460.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:24:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:24:50 smithi122 ceph-mon[93869]: pgmap v9045: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 149 KiB/s wr, 9 op/s 2023-03-09T19:24:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:24:50 smithi181 ceph-mon[94189]: pgmap v9045: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 149 KiB/s wr, 9 op/s 2023-03-09T19:24:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:24:50 smithi038 ceph-mon[92422]: pgmap v9045: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 149 KiB/s wr, 9 op/s 2023-03-09T19:24:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:24:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16001.191130 secs 2023-03-09T19:24:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:24:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:24:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:24:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16001.191130 secs 2023-03-09T19:24:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:24:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:24:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:24:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16001.191130 secs 2023-03-09T19:24:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:24:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:24:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:24:52 smithi122 ceph-mon[93869]: pgmap v9046: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 161 KiB/s wr, 11 op/s 2023-03-09T19:24:52.530 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:24:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:24:52.530 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:24:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:24:52.531 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:24:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:24:52.532 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:24:52 smithi038 ceph-mon[92422]: pgmap v9046: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 161 KiB/s wr, 11 op/s 2023-03-09T19:24:52.532 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:24:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:24:52.533 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:24:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:24:52.533 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:24:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:24:52.533 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:24:52 smithi181 ceph-mon[94189]: pgmap v9046: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 161 KiB/s wr, 11 op/s 2023-03-09T19:24:52.533 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:24:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:24:52.534 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:24:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:24:52.534 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:24:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:24:53.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16465.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:24:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:24:53 smithi122 ceph-mon[93869]: pgmap v9047: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 279 KiB/s wr, 9 op/s 2023-03-09T19:24:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:24:53 smithi181 ceph-mon[94189]: pgmap v9047: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 279 KiB/s wr, 9 op/s 2023-03-09T19:24:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:24:53 smithi038 ceph-mon[92422]: pgmap v9047: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 279 KiB/s wr, 9 op/s 2023-03-09T19:24:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:24:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16006.191187 secs 2023-03-09T19:24:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:24:55 smithi038 ceph-mon[92422]: pgmap v9048: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 242 KiB/s wr, 9 op/s 2023-03-09T19:24:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:24:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16006.191187 secs 2023-03-09T19:24:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:24:55 smithi122 ceph-mon[93869]: pgmap v9048: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 242 KiB/s wr, 9 op/s 2023-03-09T19:24:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:24:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16006.191187 secs 2023-03-09T19:24:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:24:55 smithi181 ceph-mon[94189]: pgmap v9048: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 242 KiB/s wr, 9 op/s 2023-03-09T19:24:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:24:57 smithi038 ceph-mon[92422]: pgmap v9049: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 264 KiB/s wr, 12 op/s 2023-03-09T19:24:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:24:57 smithi122 ceph-mon[93869]: pgmap v9049: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 264 KiB/s wr, 12 op/s 2023-03-09T19:24:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:24:57 smithi181 ceph-mon[94189]: pgmap v9049: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 264 KiB/s wr, 12 op/s 2023-03-09T19:24:58.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16470.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:25:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:24:59 smithi122 ceph-mon[93869]: pgmap v9050: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 249 KiB/s wr, 9 op/s 2023-03-09T19:25:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:24:59 smithi181 ceph-mon[94189]: pgmap v9050: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 249 KiB/s wr, 9 op/s 2023-03-09T19:25:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:24:59 smithi038 ceph-mon[92422]: pgmap v9050: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 249 KiB/s wr, 9 op/s 2023-03-09T19:25:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:25:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16011.191262 secs 2023-03-09T19:25:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:25:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16011.191262 secs 2023-03-09T19:25:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:25:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16011.191262 secs 2023-03-09T19:25:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:25:02 smithi122 ceph-mon[93869]: pgmap v9051: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 296 KiB/s wr, 13 op/s 2023-03-09T19:25:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:25:02 smithi181 ceph-mon[94189]: pgmap v9051: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 296 KiB/s wr, 13 op/s 2023-03-09T19:25:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:25:02 smithi038 ceph-mon[92422]: pgmap v9051: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 296 KiB/s wr, 13 op/s 2023-03-09T19:25:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:25:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16013.489889 secs 2023-03-09T19:25:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:25:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16013.489889 secs 2023-03-09T19:25:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:25:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16013.489889 secs 2023-03-09T19:25:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16475.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:25:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:25:04 smithi122 ceph-mon[93869]: pgmap v9052: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 431 KiB/s wr, 12 op/s 2023-03-09T19:25:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:25:04 smithi181 ceph-mon[94189]: pgmap v9052: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 431 KiB/s wr, 12 op/s 2023-03-09T19:25:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:25:04 smithi038 ceph-mon[92422]: pgmap v9052: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 431 KiB/s wr, 12 op/s 2023-03-09T19:25:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:25:05 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:25:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:25:05 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:25:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:25:05 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:25:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:25:05 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:25:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:25:05 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:25:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:25:05 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:25:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:25:05 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:25:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:25:05 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:25:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:25:05 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:25:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:25:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16016.191322 secs 2023-03-09T19:25:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:25:06 smithi122 ceph-mon[93869]: pgmap v9053: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 301 KiB/s wr, 23 op/s 2023-03-09T19:25:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:25:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16016.191322 secs 2023-03-09T19:25:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:25:06 smithi181 ceph-mon[94189]: pgmap v9053: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 301 KiB/s wr, 23 op/s 2023-03-09T19:25:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:25:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16016.191322 secs 2023-03-09T19:25:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:25:06 smithi038 ceph-mon[92422]: pgmap v9053: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 301 KiB/s wr, 23 op/s 2023-03-09T19:25:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:25:08 smithi122 ceph-mon[93869]: pgmap v9054: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 301 KiB/s wr, 25 op/s 2023-03-09T19:25:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:25:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16018.490016 secs 2023-03-09T19:25:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:25:08 smithi181 ceph-mon[94189]: pgmap v9054: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 301 KiB/s wr, 25 op/s 2023-03-09T19:25:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:25:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16018.490016 secs 2023-03-09T19:25:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:25:08 smithi038 ceph-mon[92422]: pgmap v9054: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 301 KiB/s wr, 25 op/s 2023-03-09T19:25:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:25:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16018.490016 secs 2023-03-09T19:25:08.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16480.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:25:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:25:10 smithi122 ceph-mon[93869]: pgmap v9055: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 283 KiB/s rd, 390 KiB/s wr, 23 op/s 2023-03-09T19:25:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:25:10 smithi122 ceph-mon[93869]: osdmap e779: 12 total, 12 up, 12 in 2023-03-09T19:25:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:25:10 smithi181 ceph-mon[94189]: pgmap v9055: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 283 KiB/s rd, 390 KiB/s wr, 23 op/s 2023-03-09T19:25:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:25:10 smithi181 ceph-mon[94189]: osdmap e779: 12 total, 12 up, 12 in 2023-03-09T19:25:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:25:10 smithi038 ceph-mon[92422]: pgmap v9055: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 283 KiB/s rd, 390 KiB/s wr, 23 op/s 2023-03-09T19:25:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:25:10 smithi038 ceph-mon[92422]: osdmap e779: 12 total, 12 up, 12 in 2023-03-09T19:25:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:25:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16021.191386 secs 2023-03-09T19:25:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:25:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16021.191386 secs 2023-03-09T19:25:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:25:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16021.191386 secs 2023-03-09T19:25:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:25:12 smithi122 ceph-mon[93869]: pgmap v9057: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 350 KiB/s wr, 24 op/s 2023-03-09T19:25:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:25:12 smithi181 ceph-mon[94189]: pgmap v9057: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 350 KiB/s wr, 24 op/s 2023-03-09T19:25:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:25:12 smithi038 ceph-mon[92422]: pgmap v9057: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 350 KiB/s wr, 24 op/s 2023-03-09T19:25:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:25:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16023.490135 secs 2023-03-09T19:25:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:25:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16023.490135 secs 2023-03-09T19:25:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:25:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16023.490135 secs 2023-03-09T19:25:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16485.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:25:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:25:14 smithi122 ceph-mon[93869]: pgmap v9058: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 226 KiB/s wr, 23 op/s 2023-03-09T19:25:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:25:14 smithi181 ceph-mon[94189]: pgmap v9058: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 226 KiB/s wr, 23 op/s 2023-03-09T19:25:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:25:14 smithi038 ceph-mon[92422]: pgmap v9058: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 226 KiB/s wr, 23 op/s 2023-03-09T19:25:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:25:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16026.191446 secs 2023-03-09T19:25:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:25:16 smithi122 ceph-mon[93869]: pgmap v9059: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 200 KiB/s wr, 8 op/s 2023-03-09T19:25:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:25:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16026.191446 secs 2023-03-09T19:25:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:25:16 smithi181 ceph-mon[94189]: pgmap v9059: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 200 KiB/s wr, 8 op/s 2023-03-09T19:25:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:25:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16026.191446 secs 2023-03-09T19:25:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:25:16 smithi038 ceph-mon[92422]: pgmap v9059: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 200 KiB/s wr, 8 op/s 2023-03-09T19:25:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:25:18 smithi122 ceph-mon[93869]: pgmap v9060: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 215 KiB/s wr, 8 op/s 2023-03-09T19:25:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:25:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16028.490260 secs 2023-03-09T19:25:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:25:18 smithi122 ceph-mon[93869]: osdmap e780: 12 total, 12 up, 12 in 2023-03-09T19:25:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:25:18 smithi181 ceph-mon[94189]: pgmap v9060: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 215 KiB/s wr, 8 op/s 2023-03-09T19:25:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:25:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16028.490260 secs 2023-03-09T19:25:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:25:18 smithi181 ceph-mon[94189]: osdmap e780: 12 total, 12 up, 12 in 2023-03-09T19:25:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:25:18 smithi038 ceph-mon[92422]: pgmap v9060: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 215 KiB/s wr, 8 op/s 2023-03-09T19:25:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:25:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16028.490260 secs 2023-03-09T19:25:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:25:18 smithi038 ceph-mon[92422]: osdmap e780: 12 total, 12 up, 12 in 2023-03-09T19:25:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16490.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:25:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:25:20 smithi122 ceph-mon[93869]: pgmap v9062: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 130 KiB/s wr, 8 op/s 2023-03-09T19:25:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:25:20 smithi181 ceph-mon[94189]: pgmap v9062: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 130 KiB/s wr, 8 op/s 2023-03-09T19:25:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:25:20 smithi038 ceph-mon[92422]: pgmap v9062: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 130 KiB/s wr, 8 op/s 2023-03-09T19:25:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:25:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16031.191508 secs 2023-03-09T19:25:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:25:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16031.191508 secs 2023-03-09T19:25:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:25:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16031.191508 secs 2023-03-09T19:25:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:25:22 smithi122 ceph-mon[93869]: pgmap v9063: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 114 KiB/s wr, 8 op/s 2023-03-09T19:25:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:25:22 smithi181 ceph-mon[94189]: pgmap v9063: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 114 KiB/s wr, 8 op/s 2023-03-09T19:25:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:25:22 smithi038 ceph-mon[92422]: pgmap v9063: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 114 KiB/s wr, 8 op/s 2023-03-09T19:25:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:25:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16033.490353 secs 2023-03-09T19:25:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:25:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16033.490353 secs 2023-03-09T19:25:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:25:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16033.490353 secs 2023-03-09T19:25:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16495.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:25:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:25:24 smithi122 ceph-mon[93869]: pgmap v9064: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 71 KiB/s wr, 8 op/s 2023-03-09T19:25:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:25:24 smithi181 ceph-mon[94189]: pgmap v9064: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 71 KiB/s wr, 8 op/s 2023-03-09T19:25:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:25:24 smithi038 ceph-mon[92422]: pgmap v9064: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 71 KiB/s wr, 8 op/s 2023-03-09T19:25:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:25:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16036.191594 secs 2023-03-09T19:25:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:25:26 smithi122 ceph-mon[93869]: pgmap v9065: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 127 KiB/s wr, 7 op/s 2023-03-09T19:25:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:25:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16036.191594 secs 2023-03-09T19:25:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:25:26 smithi181 ceph-mon[94189]: pgmap v9065: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 127 KiB/s wr, 7 op/s 2023-03-09T19:25:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:25:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16036.191594 secs 2023-03-09T19:25:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:25:26 smithi038 ceph-mon[92422]: pgmap v9065: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 127 KiB/s wr, 7 op/s 2023-03-09T19:25:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:25:28 smithi122 ceph-mon[93869]: pgmap v9066: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 113 KiB/s wr, 7 op/s 2023-03-09T19:25:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:25:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16038.490473 secs 2023-03-09T19:25:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:25:28 smithi181 ceph-mon[94189]: pgmap v9066: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 113 KiB/s wr, 7 op/s 2023-03-09T19:25:28.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:25:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16038.490473 secs 2023-03-09T19:25:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:25:28 smithi038 ceph-mon[92422]: pgmap v9066: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 113 KiB/s wr, 7 op/s 2023-03-09T19:25:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:25:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16038.490473 secs 2023-03-09T19:25:28.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16500.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:25:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:25:30 smithi122 ceph-mon[93869]: pgmap v9067: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 102 KiB/s wr, 8 op/s 2023-03-09T19:25:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:25:30 smithi181 ceph-mon[94189]: pgmap v9067: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 102 KiB/s wr, 8 op/s 2023-03-09T19:25:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:25:30 smithi038 ceph-mon[92422]: pgmap v9067: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 102 KiB/s wr, 8 op/s 2023-03-09T19:25:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:25:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16041.191658 secs 2023-03-09T19:25:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:25:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16041.191658 secs 2023-03-09T19:25:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:25:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16041.191658 secs 2023-03-09T19:25:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:25:32 smithi122 ceph-mon[93869]: pgmap v9068: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 56 KiB/s wr, 9 op/s 2023-03-09T19:25:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:25:32 smithi181 ceph-mon[94189]: pgmap v9068: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 56 KiB/s wr, 9 op/s 2023-03-09T19:25:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:25:32 smithi038 ceph-mon[92422]: pgmap v9068: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 56 KiB/s wr, 9 op/s 2023-03-09T19:25:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:25:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16043.490573 secs 2023-03-09T19:25:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:25:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16043.490573 secs 2023-03-09T19:25:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:25:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16043.490573 secs 2023-03-09T19:25:33.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16505.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:25:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:25:34 smithi122 ceph-mon[93869]: pgmap v9069: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 56 KiB/s wr, 7 op/s 2023-03-09T19:25:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:25:34 smithi181 ceph-mon[94189]: pgmap v9069: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 56 KiB/s wr, 7 op/s 2023-03-09T19:25:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:25:34 smithi038 ceph-mon[92422]: pgmap v9069: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 56 KiB/s wr, 7 op/s 2023-03-09T19:25:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:25:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16046.191742 secs 2023-03-09T19:25:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:25:36 smithi122 ceph-mon[93869]: pgmap v9070: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 48 KiB/s wr, 7 op/s 2023-03-09T19:25:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:25:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16046.191742 secs 2023-03-09T19:25:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:25:36 smithi181 ceph-mon[94189]: pgmap v9070: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 48 KiB/s wr, 7 op/s 2023-03-09T19:25:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:25:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16046.191742 secs 2023-03-09T19:25:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:25:36 smithi038 ceph-mon[92422]: pgmap v9070: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 48 KiB/s wr, 7 op/s 2023-03-09T19:25:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:25:38 smithi122 ceph-mon[93869]: pgmap v9071: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 14 KiB/s wr, 10 op/s 2023-03-09T19:25:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:25:38 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 16048.490686 secs 2023-03-09T19:25:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:25:38 smithi181 ceph-mon[94189]: pgmap v9071: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 14 KiB/s wr, 10 op/s 2023-03-09T19:25:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:25:38 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 16048.490686 secs 2023-03-09T19:25:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:25:38 smithi038 ceph-mon[92422]: pgmap v9071: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 14 KiB/s wr, 10 op/s 2023-03-09T19:25:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:25:38 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 16048.490686 secs 2023-03-09T19:25:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16510.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:25:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:25:40 smithi122 ceph-mon[93869]: pgmap v9072: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 250 KiB/s wr, 9 op/s 2023-03-09T19:25:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:25:40 smithi181 ceph-mon[94189]: pgmap v9072: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 250 KiB/s wr, 9 op/s 2023-03-09T19:25:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:25:40 smithi038 ceph-mon[92422]: pgmap v9072: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 250 KiB/s wr, 9 op/s 2023-03-09T19:25:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:25:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16051.191805 secs 2023-03-09T19:25:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:25:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:25:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:25:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:25:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:25:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16051.191805 secs 2023-03-09T19:25:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:25:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:25:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:25:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:25:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:25:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16051.191805 secs 2023-03-09T19:25:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:25:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:25:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:25:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:25:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:25:42 smithi122 ceph-mon[93869]: pgmap v9073: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 264 KiB/s wr, 11 op/s 2023-03-09T19:25:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:25:42 smithi181 ceph-mon[94189]: pgmap v9073: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 264 KiB/s wr, 11 op/s 2023-03-09T19:25:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:25:42 smithi038 ceph-mon[92422]: pgmap v9073: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 264 KiB/s wr, 11 op/s 2023-03-09T19:25:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:25:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16053.490842 secs 2023-03-09T19:25:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:25:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16053.490842 secs 2023-03-09T19:25:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:25:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16053.490842 secs 2023-03-09T19:25:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16515.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:25:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:25:44 smithi122 ceph-mon[93869]: pgmap v9074: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 264 KiB/s wr, 9 op/s 2023-03-09T19:25:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:25:44 smithi181 ceph-mon[94189]: pgmap v9074: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 264 KiB/s wr, 9 op/s 2023-03-09T19:25:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:25:44 smithi038 ceph-mon[92422]: pgmap v9074: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 264 KiB/s wr, 9 op/s 2023-03-09T19:25:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:25:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16056.191866 secs 2023-03-09T19:25:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:25:45 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:25:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:25:45 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:25:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:25:45 smithi122 ceph-mon[93869]: pgmap v9075: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 336 KiB/s rd, 321 KiB/s wr, 10 op/s 2023-03-09T19:25:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:25:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16056.191866 secs 2023-03-09T19:25:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:25:45 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:25:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:25:45 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:25:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:25:45 smithi181 ceph-mon[94189]: pgmap v9075: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 336 KiB/s rd, 321 KiB/s wr, 10 op/s 2023-03-09T19:25:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:25:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16056.191866 secs 2023-03-09T19:25:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:25:45 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:25:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:25:45 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:25:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:25:45 smithi038 ceph-mon[92422]: pgmap v9075: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 336 KiB/s rd, 321 KiB/s wr, 10 op/s 2023-03-09T19:25:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:25:48 smithi122 ceph-mon[93869]: pgmap v9076: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 418 KiB/s wr, 13 op/s 2023-03-09T19:25:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:25:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16058.490927 secs 2023-03-09T19:25:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:25:48 smithi181 ceph-mon[94189]: pgmap v9076: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 418 KiB/s wr, 13 op/s 2023-03-09T19:25:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:25:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16058.490927 secs 2023-03-09T19:25:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:25:48 smithi038 ceph-mon[92422]: pgmap v9076: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 418 KiB/s wr, 13 op/s 2023-03-09T19:25:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:25:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16058.490927 secs 2023-03-09T19:25:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16520.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:25:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:25:50 smithi122 ceph-mon[93869]: pgmap v9077: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 405 KiB/s wr, 11 op/s 2023-03-09T19:25:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:25:50 smithi181 ceph-mon[94189]: pgmap v9077: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 405 KiB/s wr, 11 op/s 2023-03-09T19:25:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:25:50 smithi038 ceph-mon[92422]: pgmap v9077: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 405 KiB/s wr, 11 op/s 2023-03-09T19:25:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:25:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16061.191926 secs 2023-03-09T19:25:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:25:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16061.191926 secs 2023-03-09T19:25:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:25:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16061.191926 secs 2023-03-09T19:25:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:25:52 smithi122 ceph-mon[93869]: pgmap v9078: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 334 KiB/s wr, 13 op/s 2023-03-09T19:25:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:25:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:25:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:25:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:25:52.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:25:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:25:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:25:52 smithi181 ceph-mon[94189]: pgmap v9078: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 334 KiB/s wr, 13 op/s 2023-03-09T19:25:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:25:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:25:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:25:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:25:52.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:25:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:25:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:25:52 smithi038 ceph-mon[92422]: pgmap v9078: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 334 KiB/s wr, 13 op/s 2023-03-09T19:25:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:25:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:25:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:25:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:25:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:25:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:25:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:25:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16063.491053 secs 2023-03-09T19:25:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:25:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:25:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:25:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:25:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:25:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:25:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:25:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16063.491053 secs 2023-03-09T19:25:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:25:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:25:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:25:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:25:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:25:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:25:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:25:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16063.491053 secs 2023-03-09T19:25:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:25:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:25:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:25:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:25:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:25:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:25:53.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16525.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:25:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:25:54 smithi122 ceph-mon[93869]: pgmap v9079: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 320 KiB/s wr, 10 op/s 2023-03-09T19:25:54.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:25:54 smithi181 ceph-mon[94189]: pgmap v9079: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 320 KiB/s wr, 10 op/s 2023-03-09T19:25:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:25:54 smithi038 ceph-mon[92422]: pgmap v9079: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 320 KiB/s wr, 10 op/s 2023-03-09T19:25:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:25:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16066.191987 secs 2023-03-09T19:25:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:25:56 smithi122 ceph-mon[93869]: pgmap v9080: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 320 KiB/s wr, 10 op/s 2023-03-09T19:25:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:25:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16066.191987 secs 2023-03-09T19:25:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:25:56 smithi181 ceph-mon[94189]: pgmap v9080: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 320 KiB/s wr, 10 op/s 2023-03-09T19:25:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:25:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16066.191987 secs 2023-03-09T19:25:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:25:56 smithi038 ceph-mon[92422]: pgmap v9080: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 320 KiB/s wr, 10 op/s 2023-03-09T19:25:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:25:58 smithi122 ceph-mon[93869]: pgmap v9081: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 318 KiB/s wr, 12 op/s 2023-03-09T19:25:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:25:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16068.491173 secs 2023-03-09T19:25:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:25:58 smithi181 ceph-mon[94189]: pgmap v9081: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 318 KiB/s wr, 12 op/s 2023-03-09T19:25:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:25:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16068.491173 secs 2023-03-09T19:25:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:25:58 smithi038 ceph-mon[92422]: pgmap v9081: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 318 KiB/s wr, 12 op/s 2023-03-09T19:25:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:25:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16068.491173 secs 2023-03-09T19:25:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16530.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:26:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:26:00 smithi122 ceph-mon[93869]: pgmap v9082: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 221 KiB/s wr, 8 op/s 2023-03-09T19:26:00.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:26:00 smithi181 ceph-mon[94189]: pgmap v9082: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 221 KiB/s wr, 8 op/s 2023-03-09T19:26:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:26:00 smithi038 ceph-mon[92422]: pgmap v9082: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 221 KiB/s wr, 8 op/s 2023-03-09T19:26:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:26:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16071.192048 secs 2023-03-09T19:26:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:26:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16071.192048 secs 2023-03-09T19:26:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:26:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16071.192048 secs 2023-03-09T19:26:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:26:02 smithi122 ceph-mon[93869]: pgmap v9083: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 305 KiB/s wr, 13 op/s 2023-03-09T19:26:02.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:26:02 smithi181 ceph-mon[94189]: pgmap v9083: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 305 KiB/s wr, 13 op/s 2023-03-09T19:26:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:26:02 smithi038 ceph-mon[92422]: pgmap v9083: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 305 KiB/s wr, 13 op/s 2023-03-09T19:26:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:26:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16073.491300 secs 2023-03-09T19:26:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:26:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16073.491300 secs 2023-03-09T19:26:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:26:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16073.491300 secs 2023-03-09T19:26:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16535.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:26:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:26:04 smithi122 ceph-mon[93869]: pgmap v9084: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 196 KiB/s wr, 10 op/s 2023-03-09T19:26:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:26:04 smithi181 ceph-mon[94189]: pgmap v9084: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 196 KiB/s wr, 10 op/s 2023-03-09T19:26:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:26:04 smithi038 ceph-mon[92422]: pgmap v9084: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 196 KiB/s wr, 10 op/s 2023-03-09T19:26:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:26:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16076.192106 secs 2023-03-09T19:26:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:26:06 smithi122 ceph-mon[93869]: pgmap v9085: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 218 KiB/s wr, 22 op/s 2023-03-09T19:26:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:26:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16076.192106 secs 2023-03-09T19:26:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:26:06 smithi181 ceph-mon[94189]: pgmap v9085: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 218 KiB/s wr, 22 op/s 2023-03-09T19:26:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:26:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16076.192106 secs 2023-03-09T19:26:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:26:06 smithi038 ceph-mon[92422]: pgmap v9085: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 218 KiB/s wr, 22 op/s 2023-03-09T19:26:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:26:08 smithi122 ceph-mon[93869]: pgmap v9086: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 218 KiB/s wr, 25 op/s 2023-03-09T19:26:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:26:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16078.491378 secs 2023-03-09T19:26:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:26:08 smithi181 ceph-mon[94189]: pgmap v9086: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 218 KiB/s wr, 25 op/s 2023-03-09T19:26:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:26:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16078.491378 secs 2023-03-09T19:26:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:26:08 smithi038 ceph-mon[92422]: pgmap v9086: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 218 KiB/s wr, 25 op/s 2023-03-09T19:26:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:26:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16078.491378 secs 2023-03-09T19:26:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16540.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:26:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:26:10 smithi122 ceph-mon[93869]: pgmap v9087: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 163 KiB/s wr, 22 op/s 2023-03-09T19:26:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:26:10 smithi122 ceph-mon[93869]: osdmap e781: 12 total, 12 up, 12 in 2023-03-09T19:26:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:26:10 smithi181 ceph-mon[94189]: pgmap v9087: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 163 KiB/s wr, 22 op/s 2023-03-09T19:26:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:26:10 smithi181 ceph-mon[94189]: osdmap e781: 12 total, 12 up, 12 in 2023-03-09T19:26:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:26:10 smithi038 ceph-mon[92422]: pgmap v9087: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 163 KiB/s wr, 22 op/s 2023-03-09T19:26:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:26:10 smithi038 ceph-mon[92422]: osdmap e781: 12 total, 12 up, 12 in 2023-03-09T19:26:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:26:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16081.192165 secs 2023-03-09T19:26:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:26:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16081.192165 secs 2023-03-09T19:26:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:26:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16081.192165 secs 2023-03-09T19:26:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:26:12 smithi122 ceph-mon[93869]: pgmap v9089: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 110 KiB/s wr, 23 op/s 2023-03-09T19:26:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:26:12 smithi181 ceph-mon[94189]: pgmap v9089: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 110 KiB/s wr, 23 op/s 2023-03-09T19:26:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:26:12 smithi038 ceph-mon[92422]: pgmap v9089: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 110 KiB/s wr, 23 op/s 2023-03-09T19:26:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:26:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16083.491490 secs 2023-03-09T19:26:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:26:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16083.491490 secs 2023-03-09T19:26:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:26:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16083.491490 secs 2023-03-09T19:26:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16545.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:26:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:26:14 smithi122 ceph-mon[93869]: pgmap v9090: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 95 KiB/s wr, 22 op/s 2023-03-09T19:26:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:26:14 smithi181 ceph-mon[94189]: pgmap v9090: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 95 KiB/s wr, 22 op/s 2023-03-09T19:26:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:26:14 smithi038 ceph-mon[92422]: pgmap v9090: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 95 KiB/s wr, 22 op/s 2023-03-09T19:26:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:26:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16086.192225 secs 2023-03-09T19:26:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:26:16 smithi122 ceph-mon[93869]: pgmap v9091: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 70 KiB/s wr, 8 op/s 2023-03-09T19:26:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:26:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16086.192225 secs 2023-03-09T19:26:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:26:16 smithi181 ceph-mon[94189]: pgmap v9091: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 70 KiB/s wr, 8 op/s 2023-03-09T19:26:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:26:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16086.192225 secs 2023-03-09T19:26:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:26:16 smithi038 ceph-mon[92422]: pgmap v9091: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 70 KiB/s wr, 8 op/s 2023-03-09T19:26:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:26:18 smithi122 ceph-mon[93869]: pgmap v9092: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 84 KiB/s wr, 8 op/s 2023-03-09T19:26:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:26:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16088.491574 secs 2023-03-09T19:26:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:26:18 smithi122 ceph-mon[93869]: osdmap e782: 12 total, 12 up, 12 in 2023-03-09T19:26:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:26:18 smithi181 ceph-mon[94189]: pgmap v9092: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 84 KiB/s wr, 8 op/s 2023-03-09T19:26:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:26:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16088.491574 secs 2023-03-09T19:26:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:26:18 smithi181 ceph-mon[94189]: osdmap e782: 12 total, 12 up, 12 in 2023-03-09T19:26:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:26:18 smithi038 ceph-mon[92422]: pgmap v9092: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 84 KiB/s wr, 8 op/s 2023-03-09T19:26:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:26:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16088.491574 secs 2023-03-09T19:26:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:26:18 smithi038 ceph-mon[92422]: osdmap e782: 12 total, 12 up, 12 in 2023-03-09T19:26:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16550.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:26:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:26:20 smithi122 ceph-mon[93869]: pgmap v9094: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 86 KiB/s wr, 8 op/s 2023-03-09T19:26:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:26:20 smithi181 ceph-mon[94189]: pgmap v9094: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 86 KiB/s wr, 8 op/s 2023-03-09T19:26:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:26:20 smithi038 ceph-mon[92422]: pgmap v9094: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 86 KiB/s wr, 8 op/s 2023-03-09T19:26:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:26:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16091.192287 secs 2023-03-09T19:26:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:26:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16091.192287 secs 2023-03-09T19:26:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:26:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16091.192287 secs 2023-03-09T19:26:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:26:22 smithi122 ceph-mon[93869]: pgmap v9095: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 69 KiB/s wr, 8 op/s 2023-03-09T19:26:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:26:22 smithi181 ceph-mon[94189]: pgmap v9095: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 69 KiB/s wr, 8 op/s 2023-03-09T19:26:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:26:22 smithi038 ceph-mon[92422]: pgmap v9095: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 69 KiB/s wr, 8 op/s 2023-03-09T19:26:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:26:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16093.491689 secs 2023-03-09T19:26:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:26:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16093.491689 secs 2023-03-09T19:26:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:26:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16093.491689 secs 2023-03-09T19:26:23.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16555.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:26:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:26:24 smithi122 ceph-mon[93869]: pgmap v9096: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 82 KiB/s wr, 8 op/s 2023-03-09T19:26:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:26:24 smithi181 ceph-mon[94189]: pgmap v9096: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 82 KiB/s wr, 8 op/s 2023-03-09T19:26:24.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:26:24 smithi038 ceph-mon[92422]: pgmap v9096: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 82 KiB/s wr, 8 op/s 2023-03-09T19:26:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:26:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16096.192354 secs 2023-03-09T19:26:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:26:26 smithi122 ceph-mon[93869]: pgmap v9097: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 82 KiB/s wr, 8 op/s 2023-03-09T19:26:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:26:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16096.192354 secs 2023-03-09T19:26:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:26:26 smithi181 ceph-mon[94189]: pgmap v9097: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 82 KiB/s wr, 8 op/s 2023-03-09T19:26:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:26:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16096.192354 secs 2023-03-09T19:26:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:26:26 smithi038 ceph-mon[92422]: pgmap v9097: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 82 KiB/s wr, 8 op/s 2023-03-09T19:26:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:26:28 smithi122 ceph-mon[93869]: pgmap v9098: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 130 KiB/s wr, 8 op/s 2023-03-09T19:26:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:26:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16098.491797 secs 2023-03-09T19:26:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:26:28 smithi181 ceph-mon[94189]: pgmap v9098: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 130 KiB/s wr, 8 op/s 2023-03-09T19:26:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:26:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16098.491797 secs 2023-03-09T19:26:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:26:28 smithi038 ceph-mon[92422]: pgmap v9098: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 130 KiB/s wr, 8 op/s 2023-03-09T19:26:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:26:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16098.491797 secs 2023-03-09T19:26:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16560.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:26:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:26:30 smithi122 ceph-mon[93869]: pgmap v9099: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 116 KiB/s wr, 7 op/s 2023-03-09T19:26:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:26:30 smithi181 ceph-mon[94189]: pgmap v9099: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 116 KiB/s wr, 7 op/s 2023-03-09T19:26:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:26:30 smithi038 ceph-mon[92422]: pgmap v9099: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 116 KiB/s wr, 7 op/s 2023-03-09T19:26:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:26:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16101.192416 secs 2023-03-09T19:26:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:26:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16101.192416 secs 2023-03-09T19:26:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:26:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16101.192416 secs 2023-03-09T19:26:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:26:32 smithi122 ceph-mon[93869]: pgmap v9100: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 108 KiB/s wr, 9 op/s 2023-03-09T19:26:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:26:32 smithi181 ceph-mon[94189]: pgmap v9100: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 108 KiB/s wr, 9 op/s 2023-03-09T19:26:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:26:32 smithi038 ceph-mon[92422]: pgmap v9100: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 108 KiB/s wr, 9 op/s 2023-03-09T19:26:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:26:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16103.491947 secs 2023-03-09T19:26:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:26:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16103.491947 secs 2023-03-09T19:26:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:26:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16103.491947 secs 2023-03-09T19:26:33.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16565.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:26:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:26:34 smithi122 ceph-mon[93869]: pgmap v9101: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 162 KiB/s wr, 7 op/s 2023-03-09T19:26:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:26:34 smithi181 ceph-mon[94189]: pgmap v9101: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 162 KiB/s wr, 7 op/s 2023-03-09T19:26:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:26:34 smithi038 ceph-mon[92422]: pgmap v9101: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 162 KiB/s wr, 7 op/s 2023-03-09T19:26:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:26:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16106.192484 secs 2023-03-09T19:26:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:26:36 smithi122 ceph-mon[93869]: pgmap v9102: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 107 KiB/s wr, 6 op/s 2023-03-09T19:26:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:26:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16106.192484 secs 2023-03-09T19:26:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:26:36 smithi181 ceph-mon[94189]: pgmap v9102: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 107 KiB/s wr, 6 op/s 2023-03-09T19:26:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:26:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16106.192484 secs 2023-03-09T19:26:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:26:36 smithi038 ceph-mon[92422]: pgmap v9102: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 107 KiB/s wr, 6 op/s 2023-03-09T19:26:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:26:38 smithi122 ceph-mon[93869]: pgmap v9103: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 166 KiB/s wr, 9 op/s 2023-03-09T19:26:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:26:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16108.492063 secs 2023-03-09T19:26:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:26:38 smithi181 ceph-mon[94189]: pgmap v9103: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 166 KiB/s wr, 9 op/s 2023-03-09T19:26:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:26:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16108.492063 secs 2023-03-09T19:26:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:26:38 smithi038 ceph-mon[92422]: pgmap v9103: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 166 KiB/s wr, 9 op/s 2023-03-09T19:26:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:26:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16108.492063 secs 2023-03-09T19:26:38.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16570.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:26:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:26:40 smithi122 ceph-mon[93869]: pgmap v9104: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 114 KiB/s wr, 6 op/s 2023-03-09T19:26:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:26:40 smithi181 ceph-mon[94189]: pgmap v9104: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 114 KiB/s wr, 6 op/s 2023-03-09T19:26:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:26:40 smithi038 ceph-mon[92422]: pgmap v9104: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 114 KiB/s wr, 6 op/s 2023-03-09T19:26:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:26:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16111.192563 secs 2023-03-09T19:26:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:26:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16111.192563 secs 2023-03-09T19:26:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:26:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16111.192563 secs 2023-03-09T19:26:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:26:42 smithi122 ceph-mon[93869]: pgmap v9105: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 209 KiB/s wr, 17 op/s 2023-03-09T19:26:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:26:42 smithi181 ceph-mon[94189]: pgmap v9105: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 209 KiB/s wr, 17 op/s 2023-03-09T19:26:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:26:42 smithi038 ceph-mon[92422]: pgmap v9105: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 209 KiB/s wr, 17 op/s 2023-03-09T19:26:43.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16575.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:26:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:26:43 smithi122 ceph-mon[93869]: pgmap v9106: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 230 KiB/s wr, 16 op/s 2023-03-09T19:26:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:26:43 smithi181 ceph-mon[94189]: pgmap v9106: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 230 KiB/s wr, 16 op/s 2023-03-09T19:26:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:26:43 smithi038 ceph-mon[92422]: pgmap v9106: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 230 KiB/s wr, 16 op/s 2023-03-09T19:26:45.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:26:44 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:26:45.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:26:44 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:26:45.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:26:44 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:26:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:26:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16116.192622 secs 2023-03-09T19:26:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:26:45 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:26:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:26:45 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:26:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:26:45 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:26:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:26:45 smithi122 ceph-mon[93869]: pgmap v9107: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 175 KiB/s wr, 16 op/s 2023-03-09T19:26:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:26:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16116.192622 secs 2023-03-09T19:26:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:26:45 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:26:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:26:45 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:26:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:26:45 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:26:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:26:45 smithi181 ceph-mon[94189]: pgmap v9107: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 175 KiB/s wr, 16 op/s 2023-03-09T19:26:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:26:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16116.192622 secs 2023-03-09T19:26:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:26:45 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:26:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:26:45 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:26:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:26:45 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:26:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:26:45 smithi038 ceph-mon[92422]: pgmap v9107: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 175 KiB/s wr, 16 op/s 2023-03-09T19:26:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:26:48 smithi122 ceph-mon[93869]: pgmap v9108: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 190 KiB/s wr, 48 op/s 2023-03-09T19:26:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:26:48 smithi181 ceph-mon[94189]: pgmap v9108: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 190 KiB/s wr, 48 op/s 2023-03-09T19:26:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:26:48 smithi038 ceph-mon[92422]: pgmap v9108: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 190 KiB/s wr, 48 op/s 2023-03-09T19:26:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16580.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:26:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:26:50 smithi122 ceph-mon[93869]: pgmap v9109: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 131 KiB/s wr, 45 op/s 2023-03-09T19:26:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:26:50 smithi181 ceph-mon[94189]: pgmap v9109: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 131 KiB/s wr, 45 op/s 2023-03-09T19:26:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:26:50 smithi038 ceph-mon[92422]: pgmap v9109: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 131 KiB/s wr, 45 op/s 2023-03-09T19:26:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:26:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16121.192683 secs 2023-03-09T19:26:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:26:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16121.192683 secs 2023-03-09T19:26:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:26:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16121.192683 secs 2023-03-09T19:26:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:26:52 smithi122 ceph-mon[93869]: pgmap v9110: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 345 KiB/s wr, 49 op/s 2023-03-09T19:26:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:26:52 smithi181 ceph-mon[94189]: pgmap v9110: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 345 KiB/s wr, 49 op/s 2023-03-09T19:26:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:26:52 smithi038 ceph-mon[92422]: pgmap v9110: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 345 KiB/s wr, 49 op/s 2023-03-09T19:26:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:26:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:26:53.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:26:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:26:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:26:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:26:53.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16585.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:26:54.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:26:54 smithi122 ceph-mon[93869]: pgmap v9111: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 250 KiB/s wr, 37 op/s 2023-03-09T19:26:54.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:26:54 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:26:54.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:26:54 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:26:54.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:26:54 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:26:54.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:26:54 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:26:54.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:26:54 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:26:54.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:26:54 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:26:54.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:26:54 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:26:54.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:26:54 smithi181 ceph-mon[94189]: pgmap v9111: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 250 KiB/s wr, 37 op/s 2023-03-09T19:26:54.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:26:54 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:26:54.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:26:54 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:26:54.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:26:54 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:26:54.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:26:54 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:26:54.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:26:54 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:26:54.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:26:54 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:26:54.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:26:54 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:26:54.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:26:54 smithi038 ceph-mon[92422]: pgmap v9111: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 250 KiB/s wr, 37 op/s 2023-03-09T19:26:54.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:26:54 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:26:54.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:26:54 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:26:54.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:26:54 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:26:54.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:26:54 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:26:54.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:26:54 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:26:54.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:26:54 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:26:54.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:26:54 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:26:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:26:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16126.192743 secs 2023-03-09T19:26:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:26:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16126.192743 secs 2023-03-09T19:26:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:26:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16126.192743 secs 2023-03-09T19:26:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:26:56 smithi122 ceph-mon[93869]: pgmap v9112: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 230 KiB/s wr, 36 op/s 2023-03-09T19:26:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:26:56 smithi181 ceph-mon[94189]: pgmap v9112: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 230 KiB/s wr, 36 op/s 2023-03-09T19:26:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:26:56 smithi038 ceph-mon[92422]: pgmap v9112: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 230 KiB/s wr, 36 op/s 2023-03-09T19:26:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:26:58 smithi122 ceph-mon[93869]: pgmap v9113: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 357 KiB/s rd, 300 KiB/s wr, 40 op/s 2023-03-09T19:26:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:26:58 smithi181 ceph-mon[94189]: pgmap v9113: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 357 KiB/s rd, 300 KiB/s wr, 40 op/s 2023-03-09T19:26:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:26:58 smithi038 ceph-mon[92422]: pgmap v9113: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 357 KiB/s rd, 300 KiB/s wr, 40 op/s 2023-03-09T19:26:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16590.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:27:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:27:00 smithi122 ceph-mon[93869]: pgmap v9114: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 285 KiB/s wr, 9 op/s 2023-03-09T19:27:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:27:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16131.192818 secs 2023-03-09T19:27:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:27:00 smithi181 ceph-mon[94189]: pgmap v9114: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 285 KiB/s wr, 9 op/s 2023-03-09T19:27:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:27:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16131.192818 secs 2023-03-09T19:27:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:27:00 smithi038 ceph-mon[92422]: pgmap v9114: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 285 KiB/s wr, 9 op/s 2023-03-09T19:27:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:27:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16131.192818 secs 2023-03-09T19:27:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:27:02 smithi122 ceph-mon[93869]: pgmap v9115: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 309 KiB/s wr, 13 op/s 2023-03-09T19:27:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:27:02 smithi181 ceph-mon[94189]: pgmap v9115: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 309 KiB/s wr, 13 op/s 2023-03-09T19:27:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:27:02 smithi038 ceph-mon[92422]: pgmap v9115: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 309 KiB/s wr, 13 op/s 2023-03-09T19:27:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:27:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16133.492647 secs 2023-03-09T19:27:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:27:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16133.492647 secs 2023-03-09T19:27:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:27:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16133.492647 secs 2023-03-09T19:27:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16595.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:27:04.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:27:04 smithi122 ceph-mon[93869]: pgmap v9116: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 281 KiB/s wr, 24 op/s 2023-03-09T19:27:04.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:27:04 smithi181 ceph-mon[94189]: pgmap v9116: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 281 KiB/s wr, 24 op/s 2023-03-09T19:27:04.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:27:04 smithi038 ceph-mon[92422]: pgmap v9116: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 281 KiB/s wr, 24 op/s 2023-03-09T19:27:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:27:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16136.192881 secs 2023-03-09T19:27:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:27:05 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:27:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:27:05 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:27:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:27:05 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:27:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:27:05 smithi122 ceph-mon[93869]: osdmap e783: 12 total, 12 up, 12 in 2023-03-09T19:27:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:27:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16136.192881 secs 2023-03-09T19:27:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:27:05 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:27:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:27:05 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:27:05.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:27:05 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:27:05.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:27:05 smithi181 ceph-mon[94189]: osdmap e783: 12 total, 12 up, 12 in 2023-03-09T19:27:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:27:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16136.192881 secs 2023-03-09T19:27:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:27:05 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:27:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:27:05 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:27:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:27:05 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:27:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:27:05 smithi038 ceph-mon[92422]: osdmap e783: 12 total, 12 up, 12 in 2023-03-09T19:27:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:27:06 smithi122 ceph-mon[93869]: pgmap v9118: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 337 KiB/s wr, 28 op/s 2023-03-09T19:27:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:27:06 smithi181 ceph-mon[94189]: pgmap v9118: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 337 KiB/s wr, 28 op/s 2023-03-09T19:27:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:27:06 smithi038 ceph-mon[92422]: pgmap v9118: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 337 KiB/s wr, 28 op/s 2023-03-09T19:27:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:27:08 smithi122 ceph-mon[93869]: pgmap v9119: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 280 KiB/s wr, 28 op/s 2023-03-09T19:27:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:27:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16138.492747 secs 2023-03-09T19:27:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:27:08 smithi181 ceph-mon[94189]: pgmap v9119: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 280 KiB/s wr, 28 op/s 2023-03-09T19:27:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:27:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16138.492747 secs 2023-03-09T19:27:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:27:08 smithi038 ceph-mon[92422]: pgmap v9119: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 280 KiB/s wr, 28 op/s 2023-03-09T19:27:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:27:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16138.492747 secs 2023-03-09T19:27:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16600.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:27:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:27:10 smithi122 ceph-mon[93869]: pgmap v9120: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 333 KiB/s wr, 29 op/s 2023-03-09T19:27:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:27:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16141.192966 secs 2023-03-09T19:27:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:27:10 smithi181 ceph-mon[94189]: pgmap v9120: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 333 KiB/s wr, 29 op/s 2023-03-09T19:27:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:27:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16141.192966 secs 2023-03-09T19:27:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:27:10 smithi038 ceph-mon[92422]: pgmap v9120: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 333 KiB/s wr, 29 op/s 2023-03-09T19:27:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:27:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16141.192966 secs 2023-03-09T19:27:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:27:12 smithi122 ceph-mon[93869]: pgmap v9121: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 304 KiB/s wr, 26 op/s 2023-03-09T19:27:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:27:12 smithi181 ceph-mon[94189]: pgmap v9121: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 304 KiB/s wr, 26 op/s 2023-03-09T19:27:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:27:12 smithi038 ceph-mon[92422]: pgmap v9121: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 304 KiB/s wr, 26 op/s 2023-03-09T19:27:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:27:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16143.492863 secs 2023-03-09T19:27:13.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:27:13 smithi122 ceph-mon[93869]: osdmap e784: 12 total, 12 up, 12 in 2023-03-09T19:27:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:27:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16143.492863 secs 2023-03-09T19:27:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:27:13 smithi181 ceph-mon[94189]: osdmap e784: 12 total, 12 up, 12 in 2023-03-09T19:27:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:27:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16143.492863 secs 2023-03-09T19:27:13.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:27:13 smithi038 ceph-mon[92422]: osdmap e784: 12 total, 12 up, 12 in 2023-03-09T19:27:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16605.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:27:14.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:27:14 smithi122 ceph-mon[93869]: pgmap v9123: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 156 KiB/s wr, 13 op/s 2023-03-09T19:27:14.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:27:14 smithi181 ceph-mon[94189]: pgmap v9123: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 156 KiB/s wr, 13 op/s 2023-03-09T19:27:14.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:27:14 smithi038 ceph-mon[92422]: pgmap v9123: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 156 KiB/s wr, 13 op/s 2023-03-09T19:27:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:27:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16146.192983 secs 2023-03-09T19:27:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:27:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16146.192983 secs 2023-03-09T19:27:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:27:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16146.192983 secs 2023-03-09T19:27:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:27:16 smithi122 ceph-mon[93869]: pgmap v9124: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 127 KiB/s wr, 10 op/s 2023-03-09T19:27:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:27:16 smithi181 ceph-mon[94189]: pgmap v9124: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 127 KiB/s wr, 10 op/s 2023-03-09T19:27:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:27:16 smithi038 ceph-mon[92422]: pgmap v9124: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 127 KiB/s wr, 10 op/s 2023-03-09T19:27:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:27:18 smithi122 ceph-mon[93869]: pgmap v9125: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 99 KiB/s wr, 8 op/s 2023-03-09T19:27:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:27:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16148.492927 secs 2023-03-09T19:27:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:27:18 smithi181 ceph-mon[94189]: pgmap v9125: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 99 KiB/s wr, 8 op/s 2023-03-09T19:27:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:27:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16148.492927 secs 2023-03-09T19:27:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:27:18 smithi038 ceph-mon[92422]: pgmap v9125: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 99 KiB/s wr, 8 op/s 2023-03-09T19:27:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:27:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16148.492927 secs 2023-03-09T19:27:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16610.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:27:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:27:20 smithi122 ceph-mon[93869]: pgmap v9126: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 46 KiB/s wr, 7 op/s 2023-03-09T19:27:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:27:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16151.193043 secs 2023-03-09T19:27:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:27:20 smithi181 ceph-mon[94189]: pgmap v9126: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 46 KiB/s wr, 7 op/s 2023-03-09T19:27:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:27:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16151.193043 secs 2023-03-09T19:27:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:27:20 smithi038 ceph-mon[92422]: pgmap v9126: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 46 KiB/s wr, 7 op/s 2023-03-09T19:27:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:27:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16151.193043 secs 2023-03-09T19:27:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:27:22 smithi122 ceph-mon[93869]: pgmap v9127: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 45 KiB/s wr, 7 op/s 2023-03-09T19:27:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:27:22 smithi181 ceph-mon[94189]: pgmap v9127: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 45 KiB/s wr, 7 op/s 2023-03-09T19:27:22.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:27:22 smithi038 ceph-mon[92422]: pgmap v9127: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 45 KiB/s wr, 7 op/s 2023-03-09T19:27:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:27:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16153.493061 secs 2023-03-09T19:27:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:27:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16153.493061 secs 2023-03-09T19:27:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:27:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16153.493061 secs 2023-03-09T19:27:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16615.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:27:24.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:27:24 smithi122 ceph-mon[93869]: pgmap v9128: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 81 KiB/s wr, 7 op/s 2023-03-09T19:27:24.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:27:24 smithi181 ceph-mon[94189]: pgmap v9128: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 81 KiB/s wr, 7 op/s 2023-03-09T19:27:24.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:27:24 smithi038 ceph-mon[92422]: pgmap v9128: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 81 KiB/s wr, 7 op/s 2023-03-09T19:27:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:27:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16156.193104 secs 2023-03-09T19:27:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:27:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16156.193104 secs 2023-03-09T19:27:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:27:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16156.193104 secs 2023-03-09T19:27:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:27:26 smithi122 ceph-mon[93869]: pgmap v9129: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 30 KiB/s wr, 5 op/s 2023-03-09T19:27:26.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:27:26 smithi181 ceph-mon[94189]: pgmap v9129: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 30 KiB/s wr, 5 op/s 2023-03-09T19:27:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:27:26 smithi038 ceph-mon[92422]: pgmap v9129: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 30 KiB/s wr, 5 op/s 2023-03-09T19:27:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:27:28 smithi122 ceph-mon[93869]: pgmap v9130: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 31 KiB/s wr, 8 op/s 2023-03-09T19:27:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:27:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16158.493101 secs 2023-03-09T19:27:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:27:28 smithi181 ceph-mon[94189]: pgmap v9130: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 31 KiB/s wr, 8 op/s 2023-03-09T19:27:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:27:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16158.493101 secs 2023-03-09T19:27:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:27:28 smithi038 ceph-mon[92422]: pgmap v9130: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 31 KiB/s wr, 8 op/s 2023-03-09T19:27:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:27:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16158.493101 secs 2023-03-09T19:27:28.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16620.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:27:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:27:30 smithi122 ceph-mon[93869]: pgmap v9131: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 86 KiB/s wr, 6 op/s 2023-03-09T19:27:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:27:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16161.193192 secs 2023-03-09T19:27:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:27:30 smithi181 ceph-mon[94189]: pgmap v9131: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 86 KiB/s wr, 6 op/s 2023-03-09T19:27:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:27:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16161.193192 secs 2023-03-09T19:27:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:27:30 smithi038 ceph-mon[92422]: pgmap v9131: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 86 KiB/s wr, 6 op/s 2023-03-09T19:27:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:27:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16161.193192 secs 2023-03-09T19:27:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:27:32 smithi122 ceph-mon[93869]: pgmap v9132: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 87 KiB/s wr, 8 op/s 2023-03-09T19:27:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:27:32 smithi181 ceph-mon[94189]: pgmap v9132: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 87 KiB/s wr, 8 op/s 2023-03-09T19:27:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:27:32 smithi038 ceph-mon[92422]: pgmap v9132: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 87 KiB/s wr, 8 op/s 2023-03-09T19:27:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:27:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16163.493217 secs 2023-03-09T19:27:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:27:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16163.493217 secs 2023-03-09T19:27:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:27:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16163.493217 secs 2023-03-09T19:27:33.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16625.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:27:34.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:27:34 smithi122 ceph-mon[93869]: pgmap v9133: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 87 KiB/s wr, 6 op/s 2023-03-09T19:27:34.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:27:34 smithi181 ceph-mon[94189]: pgmap v9133: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 87 KiB/s wr, 6 op/s 2023-03-09T19:27:34.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:27:34 smithi038 ceph-mon[92422]: pgmap v9133: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 87 KiB/s wr, 6 op/s 2023-03-09T19:27:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:27:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16166.193251 secs 2023-03-09T19:27:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:27:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16166.193251 secs 2023-03-09T19:27:35.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:27:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16166.193251 secs 2023-03-09T19:27:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:27:36 smithi122 ceph-mon[93869]: pgmap v9134: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 56 KiB/s wr, 6 op/s 2023-03-09T19:27:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:27:36 smithi181 ceph-mon[94189]: pgmap v9134: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 56 KiB/s wr, 6 op/s 2023-03-09T19:27:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:27:36 smithi038 ceph-mon[92422]: pgmap v9134: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 56 KiB/s wr, 6 op/s 2023-03-09T19:27:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:27:38 smithi122 ceph-mon[93869]: pgmap v9135: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 8 op/s 2023-03-09T19:27:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:27:38 smithi181 ceph-mon[94189]: pgmap v9135: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 8 op/s 2023-03-09T19:27:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:27:38 smithi038 ceph-mon[92422]: pgmap v9135: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 8 op/s 2023-03-09T19:27:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16630.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:27:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:27:39 smithi122 ceph-mon[93869]: pgmap v9136: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 185 KiB/s wr, 7 op/s 2023-03-09T19:27:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:27:39 smithi181 ceph-mon[94189]: pgmap v9136: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 185 KiB/s wr, 7 op/s 2023-03-09T19:27:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:27:39 smithi038 ceph-mon[92422]: pgmap v9136: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 185 KiB/s wr, 7 op/s 2023-03-09T19:27:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:27:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16171.193311 secs 2023-03-09T19:27:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:27:40 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:27:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:27:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16171.193311 secs 2023-03-09T19:27:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:27:40 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:27:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:27:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16171.193311 secs 2023-03-09T19:27:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:27:40 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:27:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:27:41 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:27:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:27:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:27:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:27:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:27:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:27:41 smithi122 ceph-mon[93869]: pgmap v9137: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 157 KiB/s wr, 10 op/s 2023-03-09T19:27:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:27:41 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:27:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:27:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:27:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:27:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:27:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:27:41 smithi181 ceph-mon[94189]: pgmap v9137: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 157 KiB/s wr, 10 op/s 2023-03-09T19:27:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:27:41 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:27:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:27:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:27:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:27:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:27:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:27:41 smithi038 ceph-mon[92422]: pgmap v9137: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 157 KiB/s wr, 10 op/s 2023-03-09T19:27:42.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:27:42 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 16173.493449 secs 2023-03-09T19:27:42.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:27:42 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 16173.493449 secs 2023-03-09T19:27:42.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:27:42 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 16173.493449 secs 2023-03-09T19:27:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16635.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:27:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:27:43 smithi122 ceph-mon[93869]: pgmap v9138: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 253 KiB/s wr, 10 op/s 2023-03-09T19:27:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:27:43 smithi181 ceph-mon[94189]: pgmap v9138: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 253 KiB/s wr, 10 op/s 2023-03-09T19:27:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:27:43 smithi038 ceph-mon[92422]: pgmap v9138: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 253 KiB/s wr, 10 op/s 2023-03-09T19:27:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:27:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16176.193371 secs 2023-03-09T19:27:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:27:45 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:27:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:27:45 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:27:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:27:45 smithi122 ceph-mon[93869]: pgmap v9139: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 253 KiB/s wr, 9 op/s 2023-03-09T19:27:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:27:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16176.193371 secs 2023-03-09T19:27:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:27:45 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:27:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:27:45 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:27:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:27:45 smithi181 ceph-mon[94189]: pgmap v9139: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 253 KiB/s wr, 9 op/s 2023-03-09T19:27:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:27:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16176.193371 secs 2023-03-09T19:27:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:27:45 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:27:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:27:45 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:27:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:27:45 smithi038 ceph-mon[92422]: pgmap v9139: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 253 KiB/s wr, 9 op/s 2023-03-09T19:27:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:27:48 smithi122 ceph-mon[93869]: pgmap v9140: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 253 KiB/s wr, 13 op/s 2023-03-09T19:27:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:27:48 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 16178.493527 secs 2023-03-09T19:27:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:27:48 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:27:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:27:48 smithi181 ceph-mon[94189]: pgmap v9140: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 253 KiB/s wr, 13 op/s 2023-03-09T19:27:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:27:48 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 16178.493527 secs 2023-03-09T19:27:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:27:48 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:27:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:27:48 smithi038 ceph-mon[92422]: pgmap v9140: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 253 KiB/s wr, 13 op/s 2023-03-09T19:27:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:27:48 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 16178.493527 secs 2023-03-09T19:27:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:27:48 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:27:48.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16640.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:27:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:27:50 smithi122 ceph-mon[93869]: pgmap v9141: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 381 KiB/s wr, 12 op/s 2023-03-09T19:27:50.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:27:50 smithi181 ceph-mon[94189]: pgmap v9141: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 381 KiB/s wr, 12 op/s 2023-03-09T19:27:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:27:50 smithi038 ceph-mon[92422]: pgmap v9141: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 381 KiB/s wr, 12 op/s 2023-03-09T19:27:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:27:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16181.193461 secs 2023-03-09T19:27:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:27:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16181.193461 secs 2023-03-09T19:27:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:27:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16181.193461 secs 2023-03-09T19:27:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:27:52 smithi122 ceph-mon[93869]: pgmap v9142: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 367 KiB/s wr, 14 op/s 2023-03-09T19:27:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:27:52 smithi181 ceph-mon[94189]: pgmap v9142: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 367 KiB/s wr, 14 op/s 2023-03-09T19:27:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:27:52 smithi038 ceph-mon[92422]: pgmap v9142: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 367 KiB/s wr, 14 op/s 2023-03-09T19:27:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:27:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16183.493642 secs 2023-03-09T19:27:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:27:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16183.493642 secs 2023-03-09T19:27:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:27:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16183.493642 secs 2023-03-09T19:27:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16645.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:27:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:27:54 smithi122 ceph-mon[93869]: pgmap v9143: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 339 KiB/s wr, 11 op/s 2023-03-09T19:27:54.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:27:54 smithi181 ceph-mon[94189]: pgmap v9143: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 339 KiB/s wr, 11 op/s 2023-03-09T19:27:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:27:54 smithi038 ceph-mon[92422]: pgmap v9143: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 339 KiB/s wr, 11 op/s 2023-03-09T19:27:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:27:55 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:27:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:27:55 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:27:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:27:55 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:27:55.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:27:55 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:27:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:27:55 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:27:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:27:55 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:27:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:27:55 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:27:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:27:55 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:27:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:27:55 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:27:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:27:55 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:27:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:27:55 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:27:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:27:55 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:27:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:27:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16186.193521 secs 2023-03-09T19:27:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:27:56 smithi122 ceph-mon[93869]: pgmap v9144: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 244 KiB/s wr, 10 op/s 2023-03-09T19:27:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:27:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16186.193521 secs 2023-03-09T19:27:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:27:56 smithi181 ceph-mon[94189]: pgmap v9144: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 244 KiB/s wr, 10 op/s 2023-03-09T19:27:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:27:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16186.193521 secs 2023-03-09T19:27:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:27:56 smithi038 ceph-mon[92422]: pgmap v9144: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 244 KiB/s wr, 10 op/s 2023-03-09T19:27:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:27:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:27:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:27:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:27:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:27:57 smithi122 ceph-mon[93869]: pgmap v9145: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 247 KiB/s rd, 244 KiB/s wr, 13 op/s 2023-03-09T19:27:58.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:27:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16188.493778 secs 2023-03-09T19:27:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:27:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:27:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:27:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:27:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:27:57 smithi181 ceph-mon[94189]: pgmap v9145: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 247 KiB/s rd, 244 KiB/s wr, 13 op/s 2023-03-09T19:27:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:27:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16188.493778 secs 2023-03-09T19:27:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:27:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:27:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:27:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:27:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:27:57 smithi038 ceph-mon[92422]: pgmap v9145: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 247 KiB/s rd, 244 KiB/s wr, 13 op/s 2023-03-09T19:27:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:27:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16188.493778 secs 2023-03-09T19:27:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16650.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:28:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:28:00 smithi122 ceph-mon[93869]: pgmap v9146: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 244 KiB/s wr, 10 op/s 2023-03-09T19:28:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:28:00 smithi181 ceph-mon[94189]: pgmap v9146: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 244 KiB/s wr, 10 op/s 2023-03-09T19:28:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:28:00 smithi038 ceph-mon[92422]: pgmap v9146: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 244 KiB/s wr, 10 op/s 2023-03-09T19:28:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:28:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16191.193581 secs 2023-03-09T19:28:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:28:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16191.193581 secs 2023-03-09T19:28:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:28:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16191.193581 secs 2023-03-09T19:28:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:28:02 smithi122 ceph-mon[93869]: pgmap v9147: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 234 KiB/s wr, 12 op/s 2023-03-09T19:28:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:28:02 smithi181 ceph-mon[94189]: pgmap v9147: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 234 KiB/s wr, 12 op/s 2023-03-09T19:28:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:28:02 smithi038 ceph-mon[92422]: pgmap v9147: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 234 KiB/s wr, 12 op/s 2023-03-09T19:28:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:28:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16193.493908 secs 2023-03-09T19:28:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:28:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16193.493908 secs 2023-03-09T19:28:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:28:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16193.493908 secs 2023-03-09T19:28:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16655.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:28:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:28:04 smithi122 ceph-mon[93869]: pgmap v9148: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 140 KiB/s wr, 20 op/s 2023-03-09T19:28:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:28:04 smithi181 ceph-mon[94189]: pgmap v9148: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 140 KiB/s wr, 20 op/s 2023-03-09T19:28:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:28:04 smithi038 ceph-mon[92422]: pgmap v9148: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 140 KiB/s wr, 20 op/s 2023-03-09T19:28:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:28:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16196.193641 secs 2023-03-09T19:28:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:28:06 smithi122 ceph-mon[93869]: pgmap v9149: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 140 KiB/s wr, 19 op/s 2023-03-09T19:28:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:28:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16196.193641 secs 2023-03-09T19:28:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:28:06 smithi181 ceph-mon[94189]: pgmap v9149: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 140 KiB/s wr, 19 op/s 2023-03-09T19:28:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:28:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16196.193641 secs 2023-03-09T19:28:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:28:06 smithi038 ceph-mon[92422]: pgmap v9149: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 140 KiB/s wr, 19 op/s 2023-03-09T19:28:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:28:08 smithi122 ceph-mon[93869]: pgmap v9150: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 202 KiB/s wr, 25 op/s 2023-03-09T19:28:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:28:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16198.494048 secs 2023-03-09T19:28:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:28:08 smithi181 ceph-mon[94189]: pgmap v9150: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 202 KiB/s wr, 25 op/s 2023-03-09T19:28:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:28:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16198.494048 secs 2023-03-09T19:28:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:28:08 smithi038 ceph-mon[92422]: pgmap v9150: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 202 KiB/s wr, 25 op/s 2023-03-09T19:28:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:28:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16198.494048 secs 2023-03-09T19:28:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16660.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:28:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:28:10 smithi122 ceph-mon[93869]: pgmap v9151: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 201 KiB/s wr, 21 op/s 2023-03-09T19:28:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:28:10 smithi122 ceph-mon[93869]: osdmap e785: 12 total, 12 up, 12 in 2023-03-09T19:28:10.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:28:10 smithi181 ceph-mon[94189]: pgmap v9151: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 201 KiB/s wr, 21 op/s 2023-03-09T19:28:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:28:10 smithi181 ceph-mon[94189]: osdmap e785: 12 total, 12 up, 12 in 2023-03-09T19:28:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:28:10 smithi038 ceph-mon[92422]: pgmap v9151: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 201 KiB/s wr, 21 op/s 2023-03-09T19:28:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:28:10 smithi038 ceph-mon[92422]: osdmap e785: 12 total, 12 up, 12 in 2023-03-09T19:28:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:28:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16201.193701 secs 2023-03-09T19:28:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:28:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16201.193701 secs 2023-03-09T19:28:11.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:28:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16201.193701 secs 2023-03-09T19:28:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:28:12 smithi122 ceph-mon[93869]: pgmap v9153: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 100 KiB/s wr, 24 op/s 2023-03-09T19:28:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:28:12 smithi181 ceph-mon[94189]: pgmap v9153: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 100 KiB/s wr, 24 op/s 2023-03-09T19:28:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:28:12 smithi038 ceph-mon[92422]: pgmap v9153: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 100 KiB/s wr, 24 op/s 2023-03-09T19:28:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:28:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16203.494160 secs 2023-03-09T19:28:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:28:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16203.494160 secs 2023-03-09T19:28:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:28:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16203.494160 secs 2023-03-09T19:28:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16665.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:28:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:28:14 smithi122 ceph-mon[93869]: pgmap v9154: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 74 KiB/s wr, 9 op/s 2023-03-09T19:28:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:28:14 smithi181 ceph-mon[94189]: pgmap v9154: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 74 KiB/s wr, 9 op/s 2023-03-09T19:28:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:28:14 smithi038 ceph-mon[92422]: pgmap v9154: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 74 KiB/s wr, 9 op/s 2023-03-09T19:28:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:28:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16206.193763 secs 2023-03-09T19:28:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:28:16 smithi122 ceph-mon[93869]: pgmap v9155: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 74 KiB/s wr, 9 op/s 2023-03-09T19:28:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:28:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16206.193763 secs 2023-03-09T19:28:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:28:16 smithi181 ceph-mon[94189]: pgmap v9155: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 74 KiB/s wr, 9 op/s 2023-03-09T19:28:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:28:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16206.193763 secs 2023-03-09T19:28:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:28:16 smithi038 ceph-mon[92422]: pgmap v9155: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 74 KiB/s wr, 9 op/s 2023-03-09T19:28:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:28:18 smithi122 ceph-mon[93869]: pgmap v9156: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 818 B/s wr, 7 op/s 2023-03-09T19:28:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:28:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16208.494278 secs 2023-03-09T19:28:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:28:18 smithi122 ceph-mon[93869]: osdmap e786: 12 total, 12 up, 12 in 2023-03-09T19:28:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:28:18 smithi181 ceph-mon[94189]: pgmap v9156: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 818 B/s wr, 7 op/s 2023-03-09T19:28:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:28:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16208.494278 secs 2023-03-09T19:28:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:28:18 smithi181 ceph-mon[94189]: osdmap e786: 12 total, 12 up, 12 in 2023-03-09T19:28:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:28:18 smithi038 ceph-mon[92422]: pgmap v9156: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 818 B/s wr, 7 op/s 2023-03-09T19:28:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:28:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16208.494278 secs 2023-03-09T19:28:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:28:18 smithi038 ceph-mon[92422]: osdmap e786: 12 total, 12 up, 12 in 2023-03-09T19:28:18.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16670.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:28:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:28:20 smithi122 ceph-mon[93869]: pgmap v9158: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 826 B/s wr, 7 op/s 2023-03-09T19:28:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:28:20 smithi181 ceph-mon[94189]: pgmap v9158: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 826 B/s wr, 7 op/s 2023-03-09T19:28:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:28:20 smithi038 ceph-mon[92422]: pgmap v9158: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 826 B/s wr, 7 op/s 2023-03-09T19:28:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:28:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16211.193826 secs 2023-03-09T19:28:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:28:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16211.193826 secs 2023-03-09T19:28:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:28:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16211.193826 secs 2023-03-09T19:28:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:28:22 smithi122 ceph-mon[93869]: pgmap v9159: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 122 KiB/s wr, 7 op/s 2023-03-09T19:28:22.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:28:22 smithi181 ceph-mon[94189]: pgmap v9159: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 122 KiB/s wr, 7 op/s 2023-03-09T19:28:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:28:22 smithi038 ceph-mon[92422]: pgmap v9159: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 122 KiB/s wr, 7 op/s 2023-03-09T19:28:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:28:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16213.494366 secs 2023-03-09T19:28:23.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:28:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16213.494366 secs 2023-03-09T19:28:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:28:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16213.494366 secs 2023-03-09T19:28:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16675.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:28:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:28:24 smithi122 ceph-mon[93869]: pgmap v9160: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 122 KiB/s wr, 8 op/s 2023-03-09T19:28:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:28:24 smithi181 ceph-mon[94189]: pgmap v9160: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 122 KiB/s wr, 8 op/s 2023-03-09T19:28:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:28:24 smithi038 ceph-mon[92422]: pgmap v9160: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 122 KiB/s wr, 8 op/s 2023-03-09T19:28:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:28:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16216.193885 secs 2023-03-09T19:28:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:28:26 smithi122 ceph-mon[93869]: pgmap v9161: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 122 KiB/s wr, 8 op/s 2023-03-09T19:28:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:28:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16216.193885 secs 2023-03-09T19:28:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:28:26 smithi181 ceph-mon[94189]: pgmap v9161: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 122 KiB/s wr, 8 op/s 2023-03-09T19:28:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:28:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16216.193885 secs 2023-03-09T19:28:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:28:26 smithi038 ceph-mon[92422]: pgmap v9161: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 122 KiB/s wr, 8 op/s 2023-03-09T19:28:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:28:28 smithi122 ceph-mon[93869]: pgmap v9162: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 184 KiB/s wr, 8 op/s 2023-03-09T19:28:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:28:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16218.494489 secs 2023-03-09T19:28:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:28:28 smithi181 ceph-mon[94189]: pgmap v9162: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 184 KiB/s wr, 8 op/s 2023-03-09T19:28:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:28:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16218.494489 secs 2023-03-09T19:28:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:28:28 smithi038 ceph-mon[92422]: pgmap v9162: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 184 KiB/s wr, 8 op/s 2023-03-09T19:28:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:28:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16218.494489 secs 2023-03-09T19:28:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16680.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:28:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:28:30 smithi122 ceph-mon[93869]: pgmap v9163: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 166 KiB/s wr, 7 op/s 2023-03-09T19:28:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:28:30 smithi181 ceph-mon[94189]: pgmap v9163: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 166 KiB/s wr, 7 op/s 2023-03-09T19:28:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:28:30 smithi038 ceph-mon[92422]: pgmap v9163: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 166 KiB/s wr, 7 op/s 2023-03-09T19:28:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:28:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16221.193945 secs 2023-03-09T19:28:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:28:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16221.193945 secs 2023-03-09T19:28:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:28:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16221.193945 secs 2023-03-09T19:28:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:28:32 smithi122 ceph-mon[93869]: pgmap v9164: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 209 KiB/s wr, 10 op/s 2023-03-09T19:28:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:28:32 smithi181 ceph-mon[94189]: pgmap v9164: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 209 KiB/s wr, 10 op/s 2023-03-09T19:28:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:28:32 smithi038 ceph-mon[92422]: pgmap v9164: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 209 KiB/s wr, 10 op/s 2023-03-09T19:28:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:28:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16223.494602 secs 2023-03-09T19:28:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:28:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16223.494602 secs 2023-03-09T19:28:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:28:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16223.494602 secs 2023-03-09T19:28:33.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16685.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:28:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:28:34 smithi122 ceph-mon[93869]: pgmap v9165: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 108 KiB/s wr, 7 op/s 2023-03-09T19:28:34.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:28:34 smithi181 ceph-mon[94189]: pgmap v9165: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 108 KiB/s wr, 7 op/s 2023-03-09T19:28:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:28:34 smithi038 ceph-mon[92422]: pgmap v9165: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 108 KiB/s wr, 7 op/s 2023-03-09T19:28:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:28:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16226.194004 secs 2023-03-09T19:28:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:28:36 smithi122 ceph-mon[93869]: pgmap v9166: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 108 KiB/s wr, 6 op/s 2023-03-09T19:28:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:28:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16226.194004 secs 2023-03-09T19:28:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:28:36 smithi181 ceph-mon[94189]: pgmap v9166: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 108 KiB/s wr, 6 op/s 2023-03-09T19:28:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:28:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16226.194004 secs 2023-03-09T19:28:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:28:36 smithi038 ceph-mon[92422]: pgmap v9166: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 108 KiB/s wr, 6 op/s 2023-03-09T19:28:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:28:38 smithi122 ceph-mon[93869]: pgmap v9167: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 175 KiB/s wr, 10 op/s 2023-03-09T19:28:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:28:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16228.494720 secs 2023-03-09T19:28:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:28:38 smithi181 ceph-mon[94189]: pgmap v9167: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 175 KiB/s wr, 10 op/s 2023-03-09T19:28:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:28:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16228.494720 secs 2023-03-09T19:28:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:28:38 smithi038 ceph-mon[92422]: pgmap v9167: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 175 KiB/s wr, 10 op/s 2023-03-09T19:28:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:28:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16228.494720 secs 2023-03-09T19:28:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16690.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:28:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:28:40 smithi122 ceph-mon[93869]: pgmap v9168: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 123 KiB/s wr, 7 op/s 2023-03-09T19:28:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:28:40 smithi181 ceph-mon[94189]: pgmap v9168: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 123 KiB/s wr, 7 op/s 2023-03-09T19:28:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:28:40 smithi038 ceph-mon[92422]: pgmap v9168: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 123 KiB/s wr, 7 op/s 2023-03-09T19:28:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:28:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16231.194064 secs 2023-03-09T19:28:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:28:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16231.194064 secs 2023-03-09T19:28:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:28:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16231.194064 secs 2023-03-09T19:28:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:28:42 smithi122 ceph-mon[93869]: pgmap v9169: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 195 KiB/s wr, 10 op/s 2023-03-09T19:28:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:28:42 smithi181 ceph-mon[94189]: pgmap v9169: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 195 KiB/s wr, 10 op/s 2023-03-09T19:28:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:28:42 smithi038 ceph-mon[92422]: pgmap v9169: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 195 KiB/s wr, 10 op/s 2023-03-09T19:28:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:28:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16233.494836 secs 2023-03-09T19:28:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:28:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16233.494836 secs 2023-03-09T19:28:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:28:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16233.494836 secs 2023-03-09T19:28:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16695.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:28:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:28:44 smithi122 ceph-mon[93869]: pgmap v9170: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 139 KiB/s wr, 7 op/s 2023-03-09T19:28:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:28:44 smithi181 ceph-mon[94189]: pgmap v9170: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 139 KiB/s wr, 7 op/s 2023-03-09T19:28:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:28:44 smithi038 ceph-mon[92422]: pgmap v9170: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 139 KiB/s wr, 7 op/s 2023-03-09T19:28:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:28:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16236.194142 secs 2023-03-09T19:28:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:28:46 smithi122 ceph-mon[93869]: pgmap v9171: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 139 KiB/s wr, 7 op/s 2023-03-09T19:28:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:28:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16236.194142 secs 2023-03-09T19:28:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:28:46 smithi181 ceph-mon[94189]: pgmap v9171: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 139 KiB/s wr, 7 op/s 2023-03-09T19:28:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:28:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16236.194142 secs 2023-03-09T19:28:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:28:46 smithi038 ceph-mon[92422]: pgmap v9171: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 139 KiB/s wr, 7 op/s 2023-03-09T19:28:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:28:48 smithi122 ceph-mon[93869]: pgmap v9172: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 219 KiB/s wr, 11 op/s 2023-03-09T19:28:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:28:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16238.494968 secs 2023-03-09T19:28:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:28:48 smithi181 ceph-mon[94189]: pgmap v9172: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 219 KiB/s wr, 11 op/s 2023-03-09T19:28:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:28:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16238.494968 secs 2023-03-09T19:28:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:28:48 smithi038 ceph-mon[92422]: pgmap v9172: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 219 KiB/s wr, 11 op/s 2023-03-09T19:28:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:28:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16238.494968 secs 2023-03-09T19:28:48.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16700.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:28:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:28:50 smithi122 ceph-mon[93869]: pgmap v9173: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 153 KiB/s wr, 8 op/s 2023-03-09T19:28:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:28:50 smithi181 ceph-mon[94189]: pgmap v9173: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 153 KiB/s wr, 8 op/s 2023-03-09T19:28:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:28:50 smithi038 ceph-mon[92422]: pgmap v9173: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 153 KiB/s wr, 8 op/s 2023-03-09T19:28:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:28:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16241.194225 secs 2023-03-09T19:28:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:28:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16241.194225 secs 2023-03-09T19:28:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:28:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16241.194225 secs 2023-03-09T19:28:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:28:52 smithi181 ceph-mon[94189]: pgmap v9174: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 337 KiB/s rd, 321 KiB/s wr, 12 op/s 2023-03-09T19:28:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:28:52 smithi038 ceph-mon[92422]: pgmap v9174: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 337 KiB/s rd, 321 KiB/s wr, 12 op/s 2023-03-09T19:28:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:28:52 smithi122 ceph-mon[93869]: pgmap v9174: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 337 KiB/s rd, 321 KiB/s wr, 12 op/s 2023-03-09T19:28:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:28:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16243.495081 secs 2023-03-09T19:28:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:28:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16243.495081 secs 2023-03-09T19:28:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:28:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16243.495081 secs 2023-03-09T19:28:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16705.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:28:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:28:54 smithi038 ceph-mon[92422]: pgmap v9175: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 249 KiB/s wr, 10 op/s 2023-03-09T19:28:54.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:28:54 smithi122 ceph-mon[93869]: pgmap v9175: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 249 KiB/s wr, 10 op/s 2023-03-09T19:28:54.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:28:54 smithi181 ceph-mon[94189]: pgmap v9175: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 249 KiB/s wr, 10 op/s 2023-03-09T19:28:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:28:55 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:28:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:28:55 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:28:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:28:55 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:28:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:28:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16246.194286 secs 2023-03-09T19:28:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:28:56 smithi038 ceph-mon[92422]: pgmap v9176: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 249 KiB/s wr, 9 op/s 2023-03-09T19:28:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:28:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:28:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:28:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:28:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:28:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:28:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:28:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16246.194286 secs 2023-03-09T19:28:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:28:56 smithi122 ceph-mon[93869]: pgmap v9176: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 249 KiB/s wr, 9 op/s 2023-03-09T19:28:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:28:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:28:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:28:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:28:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:28:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:28:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:28:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16246.194286 secs 2023-03-09T19:28:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:28:56 smithi181 ceph-mon[94189]: pgmap v9176: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 249 KiB/s wr, 9 op/s 2023-03-09T19:28:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:28:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:28:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:28:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:28:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:28:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:28:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:28:58 smithi122 ceph-mon[93869]: pgmap v9177: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 271 KiB/s wr, 13 op/s 2023-03-09T19:28:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:28:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16248.495201 secs 2023-03-09T19:28:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:28:58 smithi181 ceph-mon[94189]: pgmap v9177: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 271 KiB/s wr, 13 op/s 2023-03-09T19:28:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:28:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16248.495201 secs 2023-03-09T19:28:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:28:58 smithi038 ceph-mon[92422]: pgmap v9177: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 271 KiB/s wr, 13 op/s 2023-03-09T19:28:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:28:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16248.495201 secs 2023-03-09T19:28:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16710.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:29:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:29:00 smithi122 ceph-mon[93869]: pgmap v9178: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 191 KiB/s wr, 9 op/s 2023-03-09T19:29:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:29:00 smithi181 ceph-mon[94189]: pgmap v9178: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 191 KiB/s wr, 9 op/s 2023-03-09T19:29:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:29:00 smithi038 ceph-mon[92422]: pgmap v9178: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 191 KiB/s wr, 9 op/s 2023-03-09T19:29:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:29:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16251.194347 secs 2023-03-09T19:29:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:29:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16251.194347 secs 2023-03-09T19:29:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:29:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16251.194347 secs 2023-03-09T19:29:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:29:02 smithi122 ceph-mon[93869]: pgmap v9179: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 432 KiB/s rd, 429 KiB/s wr, 13 op/s 2023-03-09T19:29:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:29:02 smithi181 ceph-mon[94189]: pgmap v9179: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 432 KiB/s rd, 429 KiB/s wr, 13 op/s 2023-03-09T19:29:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:29:02 smithi038 ceph-mon[92422]: pgmap v9179: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 432 KiB/s rd, 429 KiB/s wr, 13 op/s 2023-03-09T19:29:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:29:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16253.495307 secs 2023-03-09T19:29:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:29:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16253.495307 secs 2023-03-09T19:29:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:29:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16253.495307 secs 2023-03-09T19:29:03.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16715.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:29:04.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:29:04 smithi122 ceph-mon[93869]: pgmap v9180: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 285 KiB/s wr, 21 op/s 2023-03-09T19:29:04.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:29:04 smithi181 ceph-mon[94189]: pgmap v9180: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 285 KiB/s wr, 21 op/s 2023-03-09T19:29:04.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:29:04 smithi038 ceph-mon[92422]: pgmap v9180: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 285 KiB/s wr, 21 op/s 2023-03-09T19:29:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:29:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16256.194407 secs 2023-03-09T19:29:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:29:06 smithi122 ceph-mon[93869]: pgmap v9181: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 284 KiB/s wr, 21 op/s 2023-03-09T19:29:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:29:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16256.194407 secs 2023-03-09T19:29:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:29:06 smithi181 ceph-mon[94189]: pgmap v9181: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 284 KiB/s wr, 21 op/s 2023-03-09T19:29:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:29:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16256.194407 secs 2023-03-09T19:29:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:29:06 smithi038 ceph-mon[92422]: pgmap v9181: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 284 KiB/s wr, 21 op/s 2023-03-09T19:29:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:29:08 smithi122 ceph-mon[93869]: pgmap v9182: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 307 KiB/s wr, 26 op/s 2023-03-09T19:29:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:29:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16258.495397 secs 2023-03-09T19:29:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:29:08 smithi181 ceph-mon[94189]: pgmap v9182: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 307 KiB/s wr, 26 op/s 2023-03-09T19:29:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:29:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16258.495397 secs 2023-03-09T19:29:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:29:08 smithi038 ceph-mon[92422]: pgmap v9182: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 307 KiB/s wr, 26 op/s 2023-03-09T19:29:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:29:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16258.495397 secs 2023-03-09T19:29:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16720.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:29:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:29:10 smithi122 ceph-mon[93869]: pgmap v9183: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 363 KiB/s rd, 398 KiB/s wr, 24 op/s 2023-03-09T19:29:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:29:10 smithi122 ceph-mon[93869]: osdmap e787: 12 total, 12 up, 12 in 2023-03-09T19:29:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:29:10 smithi181 ceph-mon[94189]: pgmap v9183: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 363 KiB/s rd, 398 KiB/s wr, 24 op/s 2023-03-09T19:29:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:29:10 smithi181 ceph-mon[94189]: osdmap e787: 12 total, 12 up, 12 in 2023-03-09T19:29:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:29:10 smithi038 ceph-mon[92422]: pgmap v9183: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 363 KiB/s rd, 398 KiB/s wr, 24 op/s 2023-03-09T19:29:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:29:10 smithi038 ceph-mon[92422]: osdmap e787: 12 total, 12 up, 12 in 2023-03-09T19:29:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:29:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16261.194465 secs 2023-03-09T19:29:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:29:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16261.194465 secs 2023-03-09T19:29:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:29:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16261.194465 secs 2023-03-09T19:29:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:29:12 smithi122 ceph-mon[93869]: pgmap v9185: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 206 KiB/s wr, 26 op/s 2023-03-09T19:29:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:29:12 smithi181 ceph-mon[94189]: pgmap v9185: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 206 KiB/s wr, 26 op/s 2023-03-09T19:29:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:29:12 smithi038 ceph-mon[92422]: pgmap v9185: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 206 KiB/s wr, 26 op/s 2023-03-09T19:29:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:29:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16263.495514 secs 2023-03-09T19:29:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:29:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16263.495514 secs 2023-03-09T19:29:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:29:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16263.495514 secs 2023-03-09T19:29:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16725.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:29:14.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:29:14 smithi122 ceph-mon[93869]: pgmap v9186: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 232 KiB/s wr, 12 op/s 2023-03-09T19:29:14.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:29:14 smithi181 ceph-mon[94189]: pgmap v9186: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 232 KiB/s wr, 12 op/s 2023-03-09T19:29:14.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:29:14 smithi038 ceph-mon[92422]: pgmap v9186: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 232 KiB/s wr, 12 op/s 2023-03-09T19:29:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:29:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16266.194526 secs 2023-03-09T19:29:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:29:16 smithi122 ceph-mon[93869]: pgmap v9187: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 232 KiB/s wr, 12 op/s 2023-03-09T19:29:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:29:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16266.194526 secs 2023-03-09T19:29:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:29:16 smithi181 ceph-mon[94189]: pgmap v9187: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 232 KiB/s wr, 12 op/s 2023-03-09T19:29:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:29:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16266.194526 secs 2023-03-09T19:29:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:29:16 smithi038 ceph-mon[92422]: pgmap v9187: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 232 KiB/s wr, 12 op/s 2023-03-09T19:29:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:29:18 smithi122 ceph-mon[93869]: pgmap v9188: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 206 KiB/s wr, 8 op/s 2023-03-09T19:29:18.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:29:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16268.495629 secs 2023-03-09T19:29:18.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:29:18 smithi122 ceph-mon[93869]: osdmap e788: 12 total, 12 up, 12 in 2023-03-09T19:29:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:29:18 smithi181 ceph-mon[94189]: pgmap v9188: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 206 KiB/s wr, 8 op/s 2023-03-09T19:29:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:29:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16268.495629 secs 2023-03-09T19:29:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:29:18 smithi181 ceph-mon[94189]: osdmap e788: 12 total, 12 up, 12 in 2023-03-09T19:29:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:29:18 smithi038 ceph-mon[92422]: pgmap v9188: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 206 KiB/s wr, 8 op/s 2023-03-09T19:29:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:29:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16268.495629 secs 2023-03-09T19:29:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:29:18 smithi038 ceph-mon[92422]: osdmap e788: 12 total, 12 up, 12 in 2023-03-09T19:29:18.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16730.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:29:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:29:20 smithi122 ceph-mon[93869]: pgmap v9190: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 120 KiB/s wr, 8 op/s 2023-03-09T19:29:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:29:20 smithi181 ceph-mon[94189]: pgmap v9190: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 120 KiB/s wr, 8 op/s 2023-03-09T19:29:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:29:20 smithi038 ceph-mon[92422]: pgmap v9190: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 120 KiB/s wr, 8 op/s 2023-03-09T19:29:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:29:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16271.194603 secs 2023-03-09T19:29:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:29:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16271.194603 secs 2023-03-09T19:29:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:29:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16271.194603 secs 2023-03-09T19:29:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:29:22 smithi122 ceph-mon[93869]: pgmap v9191: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 101 KiB/s wr, 7 op/s 2023-03-09T19:29:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:29:22 smithi181 ceph-mon[94189]: pgmap v9191: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 101 KiB/s wr, 7 op/s 2023-03-09T19:29:22.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:29:22 smithi038 ceph-mon[92422]: pgmap v9191: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 101 KiB/s wr, 7 op/s 2023-03-09T19:29:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:29:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16273.495742 secs 2023-03-09T19:29:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:29:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16273.495742 secs 2023-03-09T19:29:23.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:29:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16273.495742 secs 2023-03-09T19:29:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16735.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:29:24.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:29:24 smithi122 ceph-mon[93869]: pgmap v9192: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 115 KiB/s wr, 7 op/s 2023-03-09T19:29:24.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:29:24 smithi181 ceph-mon[94189]: pgmap v9192: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 115 KiB/s wr, 7 op/s 2023-03-09T19:29:24.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:29:24 smithi038 ceph-mon[92422]: pgmap v9192: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 115 KiB/s wr, 7 op/s 2023-03-09T19:29:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:29:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16276.194662 secs 2023-03-09T19:29:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:29:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16276.194662 secs 2023-03-09T19:29:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:29:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16276.194662 secs 2023-03-09T19:29:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:29:26 smithi122 ceph-mon[93869]: pgmap v9193: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 115 KiB/s wr, 7 op/s 2023-03-09T19:29:26.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:29:26 smithi181 ceph-mon[94189]: pgmap v9193: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 115 KiB/s wr, 7 op/s 2023-03-09T19:29:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:29:26 smithi038 ceph-mon[92422]: pgmap v9193: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 115 KiB/s wr, 7 op/s 2023-03-09T19:29:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:29:28 smithi122 ceph-mon[93869]: pgmap v9194: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 114 KiB/s wr, 8 op/s 2023-03-09T19:29:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:29:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16278.495856 secs 2023-03-09T19:29:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:29:28 smithi181 ceph-mon[94189]: pgmap v9194: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 114 KiB/s wr, 8 op/s 2023-03-09T19:29:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:29:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16278.495856 secs 2023-03-09T19:29:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:29:28 smithi038 ceph-mon[92422]: pgmap v9194: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 114 KiB/s wr, 8 op/s 2023-03-09T19:29:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:29:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16278.495856 secs 2023-03-09T19:29:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16740.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:29:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:29:30 smithi122 ceph-mon[93869]: pgmap v9195: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 103 KiB/s wr, 8 op/s 2023-03-09T19:29:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:29:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16281.194724 secs 2023-03-09T19:29:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:29:30 smithi181 ceph-mon[94189]: pgmap v9195: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 103 KiB/s wr, 8 op/s 2023-03-09T19:29:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:29:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16281.194724 secs 2023-03-09T19:29:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:29:30 smithi038 ceph-mon[92422]: pgmap v9195: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 103 KiB/s wr, 8 op/s 2023-03-09T19:29:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:29:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16281.194724 secs 2023-03-09T19:29:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:29:32 smithi122 ceph-mon[93869]: pgmap v9196: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 9 op/s 2023-03-09T19:29:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:29:32 smithi181 ceph-mon[94189]: pgmap v9196: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 9 op/s 2023-03-09T19:29:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:29:32 smithi038 ceph-mon[92422]: pgmap v9196: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 9 op/s 2023-03-09T19:29:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:29:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16283.495971 secs 2023-03-09T19:29:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:29:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16283.495971 secs 2023-03-09T19:29:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:29:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16283.495971 secs 2023-03-09T19:29:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16745.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:29:34.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:29:34 smithi122 ceph-mon[93869]: pgmap v9197: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 57 KiB/s wr, 7 op/s 2023-03-09T19:29:34.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:29:34 smithi181 ceph-mon[94189]: pgmap v9197: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 57 KiB/s wr, 7 op/s 2023-03-09T19:29:34.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:29:34 smithi038 ceph-mon[92422]: pgmap v9197: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 57 KiB/s wr, 7 op/s 2023-03-09T19:29:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:29:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16286.194783 secs 2023-03-09T19:29:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:29:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16286.194783 secs 2023-03-09T19:29:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:29:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16286.194783 secs 2023-03-09T19:29:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:29:36 smithi122 ceph-mon[93869]: pgmap v9198: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2023-03-09T19:29:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:29:36 smithi181 ceph-mon[94189]: pgmap v9198: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2023-03-09T19:29:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:29:36 smithi038 ceph-mon[92422]: pgmap v9198: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2023-03-09T19:29:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:29:38 smithi122 ceph-mon[93869]: pgmap v9199: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 97 KiB/s wr, 9 op/s 2023-03-09T19:29:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:29:38 smithi181 ceph-mon[94189]: pgmap v9199: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 97 KiB/s wr, 9 op/s 2023-03-09T19:29:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:29:38 smithi038 ceph-mon[92422]: pgmap v9199: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 97 KiB/s wr, 9 op/s 2023-03-09T19:29:38.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16750.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:29:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:29:40 smithi122 ceph-mon[93869]: pgmap v9200: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 185 KiB/s wr, 8 op/s 2023-03-09T19:29:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:29:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16291.194841 secs 2023-03-09T19:29:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:29:40 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:29:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:29:40 smithi181 ceph-mon[94189]: pgmap v9200: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 185 KiB/s wr, 8 op/s 2023-03-09T19:29:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:29:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16291.194841 secs 2023-03-09T19:29:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:29:40 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:29:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:29:40 smithi038 ceph-mon[92422]: pgmap v9200: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 185 KiB/s wr, 8 op/s 2023-03-09T19:29:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:29:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16291.194841 secs 2023-03-09T19:29:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:29:40 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:29:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:29:41 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:29:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:29:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:29:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:29:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:29:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:29:41 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:29:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:29:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:29:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:29:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:29:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:29:41 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:29:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:29:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:29:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:29:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:29:42.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:29:42 smithi122 ceph-mon[93869]: pgmap v9201: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 329 KiB/s wr, 11 op/s 2023-03-09T19:29:42.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:29:42 smithi181 ceph-mon[94189]: pgmap v9201: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 329 KiB/s wr, 11 op/s 2023-03-09T19:29:42.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:29:42 smithi038 ceph-mon[92422]: pgmap v9201: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 329 KiB/s wr, 11 op/s 2023-03-09T19:29:43.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16755.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:29:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:29:43 smithi122 ceph-mon[93869]: pgmap v9202: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 329 KiB/s wr, 10 op/s 2023-03-09T19:29:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:29:43 smithi181 ceph-mon[94189]: pgmap v9202: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 329 KiB/s wr, 10 op/s 2023-03-09T19:29:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:29:43 smithi038 ceph-mon[92422]: pgmap v9202: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 329 KiB/s wr, 10 op/s 2023-03-09T19:29:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:29:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16296.194903 secs 2023-03-09T19:29:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:29:46 smithi122 ceph-mon[93869]: pgmap v9203: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 329 KiB/s wr, 9 op/s 2023-03-09T19:29:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:29:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16296.194903 secs 2023-03-09T19:29:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:29:46 smithi181 ceph-mon[94189]: pgmap v9203: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 329 KiB/s wr, 9 op/s 2023-03-09T19:29:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:29:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16296.194903 secs 2023-03-09T19:29:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:29:46 smithi038 ceph-mon[92422]: pgmap v9203: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 329 KiB/s wr, 9 op/s 2023-03-09T19:29:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:29:48 smithi122 ceph-mon[93869]: pgmap v9204: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 529 KiB/s rd, 470 KiB/s wr, 13 op/s 2023-03-09T19:29:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:29:48 smithi181 ceph-mon[94189]: pgmap v9204: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 529 KiB/s rd, 470 KiB/s wr, 13 op/s 2023-03-09T19:29:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:29:48 smithi038 ceph-mon[92422]: pgmap v9204: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 529 KiB/s rd, 470 KiB/s wr, 13 op/s 2023-03-09T19:29:48.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16760.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:29:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:29:50 smithi122 ceph-mon[93869]: pgmap v9205: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 374 KiB/s wr, 10 op/s 2023-03-09T19:29:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:29:50 smithi181 ceph-mon[94189]: pgmap v9205: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 374 KiB/s wr, 10 op/s 2023-03-09T19:29:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:29:50 smithi038 ceph-mon[92422]: pgmap v9205: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 374 KiB/s wr, 10 op/s 2023-03-09T19:29:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:29:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16301.194965 secs 2023-03-09T19:29:51.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:29:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16301.194965 secs 2023-03-09T19:29:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:29:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16301.194965 secs 2023-03-09T19:29:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:29:52 smithi122 ceph-mon[93869]: pgmap v9206: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 643 KiB/s rd, 413 KiB/s wr, 14 op/s 2023-03-09T19:29:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:29:52 smithi181 ceph-mon[94189]: pgmap v9206: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 643 KiB/s rd, 413 KiB/s wr, 14 op/s 2023-03-09T19:29:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:29:52 smithi038 ceph-mon[92422]: pgmap v9206: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 643 KiB/s rd, 413 KiB/s wr, 14 op/s 2023-03-09T19:29:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:29:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16303.496450 secs 2023-03-09T19:29:53.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:29:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16303.496450 secs 2023-03-09T19:29:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:29:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16303.496450 secs 2023-03-09T19:29:53.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16765.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:29:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:29:54 smithi122 ceph-mon[93869]: pgmap v9207: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 485 KiB/s rd, 269 KiB/s wr, 9 op/s 2023-03-09T19:29:54.575 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:29:54 smithi038 ceph-mon[92422]: pgmap v9207: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 485 KiB/s rd, 269 KiB/s wr, 9 op/s 2023-03-09T19:29:54.575 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:29:54 smithi181 ceph-mon[94189]: pgmap v9207: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 485 KiB/s rd, 269 KiB/s wr, 9 op/s 2023-03-09T19:29:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:29:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16306.194985 secs 2023-03-09T19:29:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:29:56 smithi122 ceph-mon[93869]: pgmap v9208: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 485 KiB/s rd, 269 KiB/s wr, 9 op/s 2023-03-09T19:29:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:29:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:29:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:29:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16306.194985 secs 2023-03-09T19:29:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:29:56 smithi181 ceph-mon[94189]: pgmap v9208: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 485 KiB/s rd, 269 KiB/s wr, 9 op/s 2023-03-09T19:29:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:29:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:29:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:29:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16306.194985 secs 2023-03-09T19:29:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:29:56 smithi038 ceph-mon[92422]: pgmap v9208: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 485 KiB/s rd, 269 KiB/s wr, 9 op/s 2023-03-09T19:29:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:29:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:29:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:29:57 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:29:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:29:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:29:57.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:29:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:29:57.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:29:57 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:29:57.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:29:57 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:29:57.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:29:57 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:29:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:29:57 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:29:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:29:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:29:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:29:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:29:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:29:57 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:29:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:29:57 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:29:57.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:29:57 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:29:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:29:57 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:29:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:29:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:29:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:29:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:29:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:29:57 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:29:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:29:57 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:29:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:29:57 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:29:58.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:29:58 smithi122 ceph-mon[93869]: pgmap v9209: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 542 KiB/s rd, 325 KiB/s wr, 12 op/s 2023-03-09T19:29:58.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:29:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16308.496532 secs 2023-03-09T19:29:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:29:58 smithi181 ceph-mon[94189]: pgmap v9209: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 542 KiB/s rd, 325 KiB/s wr, 12 op/s 2023-03-09T19:29:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:29:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16308.496532 secs 2023-03-09T19:29:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:29:58 smithi038 ceph-mon[92422]: pgmap v9209: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 542 KiB/s rd, 325 KiB/s wr, 12 op/s 2023-03-09T19:29:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:29:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16308.496532 secs 2023-03-09T19:29:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16770.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:30:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:30:00 smithi122 ceph-mon[93869]: pgmap v9210: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 357 KiB/s rd, 184 KiB/s wr, 8 op/s 2023-03-09T19:30:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:30:00 smithi122 ceph-mon[93869]: overall HEALTH_WARN 2 MDSs report slow requests 2023-03-09T19:30:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:30:00 smithi181 ceph-mon[94189]: pgmap v9210: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 357 KiB/s rd, 184 KiB/s wr, 8 op/s 2023-03-09T19:30:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:30:00 smithi181 ceph-mon[94189]: overall HEALTH_WARN 2 MDSs report slow requests 2023-03-09T19:30:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:30:00 smithi038 ceph-mon[92422]: pgmap v9210: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 357 KiB/s rd, 184 KiB/s wr, 8 op/s 2023-03-09T19:30:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:30:00 smithi038 ceph-mon[92422]: overall HEALTH_WARN 2 MDSs report slow requests 2023-03-09T19:30:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:30:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16311.195049 secs 2023-03-09T19:30:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:30:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16311.195049 secs 2023-03-09T19:30:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:30:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16311.195049 secs 2023-03-09T19:30:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:30:02 smithi122 ceph-mon[93869]: pgmap v9211: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 320 KiB/s wr, 13 op/s 2023-03-09T19:30:02.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:30:02 smithi181 ceph-mon[94189]: pgmap v9211: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 320 KiB/s wr, 13 op/s 2023-03-09T19:30:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:30:02 smithi038 ceph-mon[92422]: pgmap v9211: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 320 KiB/s wr, 13 op/s 2023-03-09T19:30:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:30:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16313.496650 secs 2023-03-09T19:30:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:30:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16313.496650 secs 2023-03-09T19:30:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:30:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16313.496650 secs 2023-03-09T19:30:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16775.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:30:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:30:04 smithi122 ceph-mon[93869]: pgmap v9212: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 214 KiB/s wr, 21 op/s 2023-03-09T19:30:04.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:30:04 smithi181 ceph-mon[94189]: pgmap v9212: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 214 KiB/s wr, 21 op/s 2023-03-09T19:30:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:30:04 smithi038 ceph-mon[92422]: pgmap v9212: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 214 KiB/s wr, 21 op/s 2023-03-09T19:30:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:30:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16316.195109 secs 2023-03-09T19:30:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:30:05 smithi122 ceph-mon[93869]: osdmap e789: 12 total, 12 up, 12 in 2023-03-09T19:30:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:30:05 smithi122 ceph-mon[93869]: pgmap v9214: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 257 KiB/s wr, 24 op/s 2023-03-09T19:30:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:30:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16316.195109 secs 2023-03-09T19:30:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:30:05 smithi181 ceph-mon[94189]: osdmap e789: 12 total, 12 up, 12 in 2023-03-09T19:30:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:30:05 smithi181 ceph-mon[94189]: pgmap v9214: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 257 KiB/s wr, 24 op/s 2023-03-09T19:30:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:30:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16316.195109 secs 2023-03-09T19:30:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:30:05 smithi038 ceph-mon[92422]: osdmap e789: 12 total, 12 up, 12 in 2023-03-09T19:30:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:30:05 smithi038 ceph-mon[92422]: pgmap v9214: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 257 KiB/s wr, 24 op/s 2023-03-09T19:30:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:30:08 smithi122 ceph-mon[93869]: pgmap v9215: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 348 KiB/s wr, 27 op/s 2023-03-09T19:30:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:30:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16318.496787 secs 2023-03-09T19:30:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:30:08 smithi181 ceph-mon[94189]: pgmap v9215: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 348 KiB/s wr, 27 op/s 2023-03-09T19:30:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:30:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16318.496787 secs 2023-03-09T19:30:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:30:08 smithi038 ceph-mon[92422]: pgmap v9215: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 348 KiB/s wr, 27 op/s 2023-03-09T19:30:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:30:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16318.496787 secs 2023-03-09T19:30:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16780.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:30:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:30:10 smithi122 ceph-mon[93869]: pgmap v9216: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 348 KiB/s wr, 27 op/s 2023-03-09T19:30:10.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:30:10 smithi181 ceph-mon[94189]: pgmap v9216: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 348 KiB/s wr, 27 op/s 2023-03-09T19:30:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:30:10 smithi038 ceph-mon[92422]: pgmap v9216: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 348 KiB/s wr, 27 op/s 2023-03-09T19:30:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:30:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16321.195194 secs 2023-03-09T19:30:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:30:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16321.195194 secs 2023-03-09T19:30:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:30:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16321.195194 secs 2023-03-09T19:30:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:30:12 smithi122 ceph-mon[93869]: pgmap v9217: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 231 KiB/s wr, 26 op/s 2023-03-09T19:30:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:30:12 smithi181 ceph-mon[94189]: pgmap v9217: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 231 KiB/s wr, 26 op/s 2023-03-09T19:30:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:30:12 smithi038 ceph-mon[92422]: pgmap v9217: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 231 KiB/s wr, 26 op/s 2023-03-09T19:30:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:30:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16323.496907 secs 2023-03-09T19:30:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:30:13 smithi122 ceph-mon[93869]: osdmap e790: 12 total, 12 up, 12 in 2023-03-09T19:30:13.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:30:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16323.496907 secs 2023-03-09T19:30:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:30:13 smithi181 ceph-mon[94189]: osdmap e790: 12 total, 12 up, 12 in 2023-03-09T19:30:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:30:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16323.496907 secs 2023-03-09T19:30:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:30:13 smithi038 ceph-mon[92422]: osdmap e790: 12 total, 12 up, 12 in 2023-03-09T19:30:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16785.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:30:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:30:14 smithi122 ceph-mon[93869]: pgmap v9219: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 253 KiB/s wr, 14 op/s 2023-03-09T19:30:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:30:14 smithi181 ceph-mon[94189]: pgmap v9219: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 253 KiB/s wr, 14 op/s 2023-03-09T19:30:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:30:14 smithi038 ceph-mon[92422]: pgmap v9219: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 253 KiB/s wr, 14 op/s 2023-03-09T19:30:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:30:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16326.195254 secs 2023-03-09T19:30:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:30:16 smithi122 ceph-mon[93869]: pgmap v9220: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 206 KiB/s wr, 11 op/s 2023-03-09T19:30:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:30:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16326.195254 secs 2023-03-09T19:30:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:30:16 smithi181 ceph-mon[94189]: pgmap v9220: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 206 KiB/s wr, 11 op/s 2023-03-09T19:30:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:30:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16326.195254 secs 2023-03-09T19:30:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:30:16 smithi038 ceph-mon[92422]: pgmap v9220: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 206 KiB/s wr, 11 op/s 2023-03-09T19:30:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:30:18 smithi122 ceph-mon[93869]: pgmap v9221: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 46 KiB/s wr, 8 op/s 2023-03-09T19:30:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:30:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16328.497044 secs 2023-03-09T19:30:18.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:30:18 smithi181 ceph-mon[94189]: pgmap v9221: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 46 KiB/s wr, 8 op/s 2023-03-09T19:30:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:30:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16328.497044 secs 2023-03-09T19:30:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:30:18 smithi038 ceph-mon[92422]: pgmap v9221: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 46 KiB/s wr, 8 op/s 2023-03-09T19:30:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:30:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16328.497044 secs 2023-03-09T19:30:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16790.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:30:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:30:20 smithi122 ceph-mon[93869]: pgmap v9222: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 46 KiB/s wr, 8 op/s 2023-03-09T19:30:20.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:30:20 smithi181 ceph-mon[94189]: pgmap v9222: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 46 KiB/s wr, 8 op/s 2023-03-09T19:30:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:30:20 smithi038 ceph-mon[92422]: pgmap v9222: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 46 KiB/s wr, 8 op/s 2023-03-09T19:30:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:30:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16331.195325 secs 2023-03-09T19:30:21.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:30:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16331.195325 secs 2023-03-09T19:30:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:30:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16331.195325 secs 2023-03-09T19:30:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:30:22 smithi122 ceph-mon[93869]: pgmap v9223: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 7 op/s 2023-03-09T19:30:22.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:30:22 smithi181 ceph-mon[94189]: pgmap v9223: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 7 op/s 2023-03-09T19:30:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:30:22 smithi038 ceph-mon[92422]: pgmap v9223: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 7 op/s 2023-03-09T19:30:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:30:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16333.497177 secs 2023-03-09T19:30:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:30:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16333.497177 secs 2023-03-09T19:30:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:30:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16333.497177 secs 2023-03-09T19:30:23.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16795.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:30:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:30:24 smithi122 ceph-mon[93869]: pgmap v9224: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 202 B/s wr, 7 op/s 2023-03-09T19:30:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:30:24 smithi181 ceph-mon[94189]: pgmap v9224: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 202 B/s wr, 7 op/s 2023-03-09T19:30:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:30:24 smithi038 ceph-mon[92422]: pgmap v9224: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 202 B/s wr, 7 op/s 2023-03-09T19:30:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:30:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16336.195386 secs 2023-03-09T19:30:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:30:26 smithi122 ceph-mon[93869]: pgmap v9225: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-09T19:30:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:30:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16336.195386 secs 2023-03-09T19:30:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:30:26 smithi181 ceph-mon[94189]: pgmap v9225: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-09T19:30:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:30:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16336.195386 secs 2023-03-09T19:30:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:30:26 smithi038 ceph-mon[92422]: pgmap v9225: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-09T19:30:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:30:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16338.497289 secs 2023-03-09T19:30:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:30:28 smithi122 ceph-mon[93869]: pgmap v9226: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 9 op/s 2023-03-09T19:30:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:30:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16338.497289 secs 2023-03-09T19:30:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:30:28 smithi181 ceph-mon[94189]: pgmap v9226: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 9 op/s 2023-03-09T19:30:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:30:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16338.497289 secs 2023-03-09T19:30:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:30:28 smithi038 ceph-mon[92422]: pgmap v9226: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 9 op/s 2023-03-09T19:30:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16800.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:30:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:30:30 smithi122 ceph-mon[93869]: pgmap v9227: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 170 B/s wr, 6 op/s 2023-03-09T19:30:30.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:30:30 smithi181 ceph-mon[94189]: pgmap v9227: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 170 B/s wr, 6 op/s 2023-03-09T19:30:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:30:30 smithi038 ceph-mon[92422]: pgmap v9227: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 170 B/s wr, 6 op/s 2023-03-09T19:30:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:30:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16341.195446 secs 2023-03-09T19:30:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:30:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16341.195446 secs 2023-03-09T19:30:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:30:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16341.195446 secs 2023-03-09T19:30:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:30:32 smithi122 ceph-mon[93869]: pgmap v9228: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 9 op/s 2023-03-09T19:30:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:30:32 smithi181 ceph-mon[94189]: pgmap v9228: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 9 op/s 2023-03-09T19:30:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:30:32 smithi038 ceph-mon[92422]: pgmap v9228: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 9 op/s 2023-03-09T19:30:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:30:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16343.497398 secs 2023-03-09T19:30:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:30:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16343.497398 secs 2023-03-09T19:30:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:30:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16343.497398 secs 2023-03-09T19:30:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16805.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:30:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:30:34 smithi122 ceph-mon[93869]: pgmap v9229: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 170 B/s wr, 6 op/s 2023-03-09T19:30:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:30:34 smithi181 ceph-mon[94189]: pgmap v9229: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 170 B/s wr, 6 op/s 2023-03-09T19:30:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:30:34 smithi038 ceph-mon[92422]: pgmap v9229: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 170 B/s wr, 6 op/s 2023-03-09T19:30:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:30:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16346.195517 secs 2023-03-09T19:30:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:30:36 smithi122 ceph-mon[93869]: pgmap v9230: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-09T19:30:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:30:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16346.195517 secs 2023-03-09T19:30:36.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:30:36 smithi181 ceph-mon[94189]: pgmap v9230: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-09T19:30:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:30:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16346.195517 secs 2023-03-09T19:30:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:30:36 smithi038 ceph-mon[92422]: pgmap v9230: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-09T19:30:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:30:38 smithi122 ceph-mon[93869]: pgmap v9231: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 67 KiB/s wr, 9 op/s 2023-03-09T19:30:38.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:30:38 smithi181 ceph-mon[94189]: pgmap v9231: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 67 KiB/s wr, 9 op/s 2023-03-09T19:30:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:30:38 smithi038 ceph-mon[92422]: pgmap v9231: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 67 KiB/s wr, 9 op/s 2023-03-09T19:30:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16810.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:30:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:30:40 smithi122 ceph-mon[93869]: pgmap v9232: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 67 KiB/s wr, 6 op/s 2023-03-09T19:30:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:30:40 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:30:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:30:40 smithi181 ceph-mon[94189]: pgmap v9232: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 67 KiB/s wr, 6 op/s 2023-03-09T19:30:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:30:40 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:30:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:30:40 smithi038 ceph-mon[92422]: pgmap v9232: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 67 KiB/s wr, 6 op/s 2023-03-09T19:30:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:30:40 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:30:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:30:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16351.195587 secs 2023-03-09T19:30:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:30:41 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:30:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:30:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:30:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:30:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:30:41.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:30:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16351.195587 secs 2023-03-09T19:30:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:30:41 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:30:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:30:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:30:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:30:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:30:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:30:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16351.195587 secs 2023-03-09T19:30:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:30:41 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:30:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:30:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:30:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:30:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:30:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:30:42 smithi122 ceph-mon[93869]: pgmap v9233: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 140 KiB/s wr, 9 op/s 2023-03-09T19:30:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:30:42 smithi181 ceph-mon[94189]: pgmap v9233: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 140 KiB/s wr, 9 op/s 2023-03-09T19:30:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:30:42 smithi038 ceph-mon[92422]: pgmap v9233: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 140 KiB/s wr, 9 op/s 2023-03-09T19:30:43.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16815.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:30:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:30:43 smithi122 ceph-mon[93869]: pgmap v9234: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 141 KiB/s wr, 6 op/s 2023-03-09T19:30:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:30:43 smithi181 ceph-mon[94189]: pgmap v9234: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 141 KiB/s wr, 6 op/s 2023-03-09T19:30:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:30:43 smithi038 ceph-mon[92422]: pgmap v9234: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 141 KiB/s wr, 6 op/s 2023-03-09T19:30:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:30:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16356.195652 secs 2023-03-09T19:30:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:30:46 smithi122 ceph-mon[93869]: pgmap v9235: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 141 KiB/s wr, 7 op/s 2023-03-09T19:30:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:30:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16356.195652 secs 2023-03-09T19:30:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:30:46 smithi181 ceph-mon[94189]: pgmap v9235: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 141 KiB/s wr, 7 op/s 2023-03-09T19:30:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:30:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16356.195652 secs 2023-03-09T19:30:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:30:46 smithi038 ceph-mon[92422]: pgmap v9235: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 141 KiB/s wr, 7 op/s 2023-03-09T19:30:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:30:48 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 16358.497797 secs 2023-03-09T19:30:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:30:48 smithi122 ceph-mon[93869]: pgmap v9236: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 176 KiB/s wr, 11 op/s 2023-03-09T19:30:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:30:48 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 16358.497797 secs 2023-03-09T19:30:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:30:48 smithi181 ceph-mon[94189]: pgmap v9236: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 176 KiB/s wr, 11 op/s 2023-03-09T19:30:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:30:48 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 16358.497797 secs 2023-03-09T19:30:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:30:48 smithi038 ceph-mon[92422]: pgmap v9236: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 176 KiB/s wr, 11 op/s 2023-03-09T19:30:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16820.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:30:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:30:49 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:30:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:30:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:30:49.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:30:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:30:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:30:49 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:30:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:30:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:30:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:30:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:30:49.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:30:49 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:30:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:30:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:30:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:30:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:30:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:30:50 smithi122 ceph-mon[93869]: pgmap v9237: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 109 KiB/s wr, 8 op/s 2023-03-09T19:30:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:30:50 smithi181 ceph-mon[94189]: pgmap v9237: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 109 KiB/s wr, 8 op/s 2023-03-09T19:30:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:30:50 smithi038 ceph-mon[92422]: pgmap v9237: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 109 KiB/s wr, 8 op/s 2023-03-09T19:30:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:30:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16361.195711 secs 2023-03-09T19:30:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:30:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16361.195711 secs 2023-03-09T19:30:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:30:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16361.195711 secs 2023-03-09T19:30:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:30:52 smithi122 ceph-mon[93869]: pgmap v9238: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 324 KiB/s wr, 12 op/s 2023-03-09T19:30:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:30:52 smithi181 ceph-mon[94189]: pgmap v9238: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 324 KiB/s wr, 12 op/s 2023-03-09T19:30:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:30:52 smithi038 ceph-mon[92422]: pgmap v9238: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 324 KiB/s wr, 12 op/s 2023-03-09T19:30:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:30:53 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 16363.497913 secs 2023-03-09T19:30:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:30:53 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 16363.497913 secs 2023-03-09T19:30:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:30:53 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 16363.497913 secs 2023-03-09T19:30:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16825.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:30:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:30:54 smithi122 ceph-mon[93869]: pgmap v9239: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 252 KiB/s wr, 10 op/s 2023-03-09T19:30:54.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:30:54 smithi181 ceph-mon[94189]: pgmap v9239: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 252 KiB/s wr, 10 op/s 2023-03-09T19:30:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:30:54 smithi038 ceph-mon[92422]: pgmap v9239: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 252 KiB/s wr, 10 op/s 2023-03-09T19:30:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:30:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16366.195788 secs 2023-03-09T19:30:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:30:56 smithi122 ceph-mon[93869]: pgmap v9240: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 251 KiB/s wr, 9 op/s 2023-03-09T19:30:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:30:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16366.195788 secs 2023-03-09T19:30:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:30:56 smithi181 ceph-mon[94189]: pgmap v9240: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 251 KiB/s wr, 9 op/s 2023-03-09T19:30:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:30:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16366.195788 secs 2023-03-09T19:30:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:30:56 smithi038 ceph-mon[92422]: pgmap v9240: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 251 KiB/s wr, 9 op/s 2023-03-09T19:30:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:30:57 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:30:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:30:57 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:30:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:30:57 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:30:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:30:58 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 16368.497989 secs 2023-03-09T19:30:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:30:58 smithi122 ceph-mon[93869]: pgmap v9241: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 319 KiB/s wr, 13 op/s 2023-03-09T19:30:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:30:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:30:58.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:30:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:30:58.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:30:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:30:58.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:30:58 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 16368.497989 secs 2023-03-09T19:30:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:30:58 smithi181 ceph-mon[94189]: pgmap v9241: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 319 KiB/s wr, 13 op/s 2023-03-09T19:30:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:30:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:30:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:30:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:30:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:30:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:30:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:30:58 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 16368.497989 secs 2023-03-09T19:30:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:30:58 smithi038 ceph-mon[92422]: pgmap v9241: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 319 KiB/s wr, 13 op/s 2023-03-09T19:30:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:30:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:30:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:30:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:30:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:30:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:30:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16830.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:31:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:31:00 smithi122 ceph-mon[93869]: pgmap v9242: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 337 KiB/s rd, 335 KiB/s wr, 9 op/s 2023-03-09T19:31:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:31:00 smithi181 ceph-mon[94189]: pgmap v9242: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 337 KiB/s rd, 335 KiB/s wr, 9 op/s 2023-03-09T19:31:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:31:00 smithi038 ceph-mon[92422]: pgmap v9242: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 337 KiB/s rd, 335 KiB/s wr, 9 op/s 2023-03-09T19:31:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:31:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16371.195847 secs 2023-03-09T19:31:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:31:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16371.195847 secs 2023-03-09T19:31:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:31:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16371.195847 secs 2023-03-09T19:31:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:31:02 smithi122 ceph-mon[93869]: pgmap v9243: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 547 KiB/s rd, 566 KiB/s wr, 15 op/s 2023-03-09T19:31:02.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:31:02 smithi181 ceph-mon[94189]: pgmap v9243: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 547 KiB/s rd, 566 KiB/s wr, 15 op/s 2023-03-09T19:31:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:31:02 smithi038 ceph-mon[92422]: pgmap v9243: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 547 KiB/s rd, 566 KiB/s wr, 15 op/s 2023-03-09T19:31:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:31:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16373.498120 secs 2023-03-09T19:31:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:31:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16373.498120 secs 2023-03-09T19:31:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:31:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16373.498120 secs 2023-03-09T19:31:03.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16835.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:31:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:31:04 smithi122 ceph-mon[93869]: pgmap v9244: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 457 KiB/s wr, 24 op/s 2023-03-09T19:31:04.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:31:04 smithi181 ceph-mon[94189]: pgmap v9244: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 457 KiB/s wr, 24 op/s 2023-03-09T19:31:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:31:04 smithi038 ceph-mon[92422]: pgmap v9244: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 457 KiB/s wr, 24 op/s 2023-03-09T19:31:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:31:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16376.195908 secs 2023-03-09T19:31:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:31:05 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:31:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:31:05 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:31:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:31:05 smithi122 ceph-mon[93869]: pgmap v9245: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 457 KiB/s wr, 24 op/s 2023-03-09T19:31:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:31:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16376.195908 secs 2023-03-09T19:31:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:31:05 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:31:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:31:05 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:31:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:31:05 smithi181 ceph-mon[94189]: pgmap v9245: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 457 KiB/s wr, 24 op/s 2023-03-09T19:31:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:31:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16376.195908 secs 2023-03-09T19:31:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:31:05 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:31:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:31:05 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:31:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:31:05 smithi038 ceph-mon[92422]: pgmap v9245: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 457 KiB/s wr, 24 op/s 2023-03-09T19:31:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:31:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16378.498239 secs 2023-03-09T19:31:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:31:08 smithi122 ceph-mon[93869]: pgmap v9246: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 459 KiB/s wr, 27 op/s 2023-03-09T19:31:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:31:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16378.498239 secs 2023-03-09T19:31:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:31:08 smithi181 ceph-mon[94189]: pgmap v9246: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 459 KiB/s wr, 27 op/s 2023-03-09T19:31:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:31:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16378.498239 secs 2023-03-09T19:31:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:31:08 smithi038 ceph-mon[92422]: pgmap v9246: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 459 KiB/s wr, 27 op/s 2023-03-09T19:31:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16840.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:31:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:31:10 smithi122 ceph-mon[93869]: pgmap v9247: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 478 KiB/s rd, 513 KiB/s wr, 25 op/s 2023-03-09T19:31:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:31:10 smithi122 ceph-mon[93869]: osdmap e791: 12 total, 12 up, 12 in 2023-03-09T19:31:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:31:10 smithi181 ceph-mon[94189]: pgmap v9247: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 478 KiB/s rd, 513 KiB/s wr, 25 op/s 2023-03-09T19:31:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:31:10 smithi181 ceph-mon[94189]: osdmap e791: 12 total, 12 up, 12 in 2023-03-09T19:31:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:31:10 smithi038 ceph-mon[92422]: pgmap v9247: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 478 KiB/s rd, 513 KiB/s wr, 25 op/s 2023-03-09T19:31:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:31:10 smithi038 ceph-mon[92422]: osdmap e791: 12 total, 12 up, 12 in 2023-03-09T19:31:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:31:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16381.195967 secs 2023-03-09T19:31:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:31:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16381.195967 secs 2023-03-09T19:31:11.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:31:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16381.195967 secs 2023-03-09T19:31:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:31:12 smithi122 ceph-mon[93869]: pgmap v9249: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 292 KiB/s wr, 25 op/s 2023-03-09T19:31:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:31:12 smithi181 ceph-mon[94189]: pgmap v9249: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 292 KiB/s wr, 25 op/s 2023-03-09T19:31:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:31:12 smithi038 ceph-mon[92422]: pgmap v9249: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 292 KiB/s wr, 25 op/s 2023-03-09T19:31:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:31:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16383.498349 secs 2023-03-09T19:31:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:31:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16383.498349 secs 2023-03-09T19:31:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:31:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16383.498349 secs 2023-03-09T19:31:13.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16845.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:31:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:31:14 smithi122 ceph-mon[93869]: pgmap v9250: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 218 KiB/s wr, 9 op/s 2023-03-09T19:31:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:31:14 smithi181 ceph-mon[94189]: pgmap v9250: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 218 KiB/s wr, 9 op/s 2023-03-09T19:31:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:31:14 smithi038 ceph-mon[92422]: pgmap v9250: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 218 KiB/s wr, 9 op/s 2023-03-09T19:31:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:31:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16386.196010 secs 2023-03-09T19:31:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:31:16 smithi122 ceph-mon[93869]: pgmap v9251: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 218 KiB/s wr, 9 op/s 2023-03-09T19:31:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:31:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16386.196010 secs 2023-03-09T19:31:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:31:16 smithi181 ceph-mon[94189]: pgmap v9251: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 218 KiB/s wr, 9 op/s 2023-03-09T19:31:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:31:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16386.196010 secs 2023-03-09T19:31:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:31:16 smithi038 ceph-mon[92422]: pgmap v9251: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 218 KiB/s wr, 9 op/s 2023-03-09T19:31:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:31:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16388.498487 secs 2023-03-09T19:31:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:31:18 smithi122 ceph-mon[93869]: pgmap v9252: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 218 KiB/s wr, 8 op/s 2023-03-09T19:31:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:31:18 smithi122 ceph-mon[93869]: osdmap e792: 12 total, 12 up, 12 in 2023-03-09T19:31:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:31:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16388.498487 secs 2023-03-09T19:31:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:31:18 smithi181 ceph-mon[94189]: pgmap v9252: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 218 KiB/s wr, 8 op/s 2023-03-09T19:31:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:31:18 smithi181 ceph-mon[94189]: osdmap e792: 12 total, 12 up, 12 in 2023-03-09T19:31:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:31:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16388.498487 secs 2023-03-09T19:31:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:31:18 smithi038 ceph-mon[92422]: pgmap v9252: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 218 KiB/s wr, 8 op/s 2023-03-09T19:31:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:31:18 smithi038 ceph-mon[92422]: osdmap e792: 12 total, 12 up, 12 in 2023-03-09T19:31:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16850.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:31:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:31:20 smithi122 ceph-mon[93869]: pgmap v9254: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 119 KiB/s wr, 7 op/s 2023-03-09T19:31:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:31:20 smithi181 ceph-mon[94189]: pgmap v9254: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 119 KiB/s wr, 7 op/s 2023-03-09T19:31:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:31:20 smithi038 ceph-mon[92422]: pgmap v9254: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 119 KiB/s wr, 7 op/s 2023-03-09T19:31:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:31:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16391.196094 secs 2023-03-09T19:31:21.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:31:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16391.196094 secs 2023-03-09T19:31:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:31:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16391.196094 secs 2023-03-09T19:31:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:31:22 smithi122 ceph-mon[93869]: pgmap v9255: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 103 KiB/s wr, 7 op/s 2023-03-09T19:31:22.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:31:22 smithi181 ceph-mon[94189]: pgmap v9255: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 103 KiB/s wr, 7 op/s 2023-03-09T19:31:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:31:22 smithi038 ceph-mon[92422]: pgmap v9255: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 103 KiB/s wr, 7 op/s 2023-03-09T19:31:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:31:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16393.498615 secs 2023-03-09T19:31:23.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:31:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16393.498615 secs 2023-03-09T19:31:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:31:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16393.498615 secs 2023-03-09T19:31:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16855.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:31:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:31:24 smithi122 ceph-mon[93869]: pgmap v9256: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 116 KiB/s wr, 7 op/s 2023-03-09T19:31:24.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:31:24 smithi181 ceph-mon[94189]: pgmap v9256: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 116 KiB/s wr, 7 op/s 2023-03-09T19:31:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:31:24 smithi038 ceph-mon[92422]: pgmap v9256: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 116 KiB/s wr, 7 op/s 2023-03-09T19:31:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:31:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16396.196153 secs 2023-03-09T19:31:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:31:26 smithi122 ceph-mon[93869]: pgmap v9257: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 116 KiB/s wr, 7 op/s 2023-03-09T19:31:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:31:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16396.196153 secs 2023-03-09T19:31:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:31:26 smithi181 ceph-mon[94189]: pgmap v9257: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 116 KiB/s wr, 7 op/s 2023-03-09T19:31:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:31:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16396.196153 secs 2023-03-09T19:31:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:31:26 smithi038 ceph-mon[92422]: pgmap v9257: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 116 KiB/s wr, 7 op/s 2023-03-09T19:31:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:31:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16398.498734 secs 2023-03-09T19:31:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:31:28 smithi122 ceph-mon[93869]: pgmap v9258: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 115 KiB/s wr, 8 op/s 2023-03-09T19:31:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:31:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16398.498734 secs 2023-03-09T19:31:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:31:28 smithi181 ceph-mon[94189]: pgmap v9258: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 115 KiB/s wr, 8 op/s 2023-03-09T19:31:28.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:31:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16398.498734 secs 2023-03-09T19:31:28.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:31:28 smithi038 ceph-mon[92422]: pgmap v9258: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 115 KiB/s wr, 8 op/s 2023-03-09T19:31:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16860.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:31:30.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:31:30 smithi122 ceph-mon[93869]: pgmap v9259: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 104 KiB/s wr, 8 op/s 2023-03-09T19:31:30.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:31:30 smithi181 ceph-mon[94189]: pgmap v9259: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 104 KiB/s wr, 8 op/s 2023-03-09T19:31:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:31:30 smithi038 ceph-mon[92422]: pgmap v9259: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 104 KiB/s wr, 8 op/s 2023-03-09T19:31:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:31:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16401.196215 secs 2023-03-09T19:31:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:31:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16401.196215 secs 2023-03-09T19:31:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:31:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16401.196215 secs 2023-03-09T19:31:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:31:32 smithi122 ceph-mon[93869]: pgmap v9260: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 9 op/s 2023-03-09T19:31:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:31:32 smithi181 ceph-mon[94189]: pgmap v9260: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 9 op/s 2023-03-09T19:31:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:31:32 smithi038 ceph-mon[92422]: pgmap v9260: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 9 op/s 2023-03-09T19:31:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:31:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16403.498853 secs 2023-03-09T19:31:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:31:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16403.498853 secs 2023-03-09T19:31:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:31:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16403.498853 secs 2023-03-09T19:31:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16865.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:31:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:31:34 smithi181 ceph-mon[94189]: pgmap v9261: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 58 KiB/s wr, 8 op/s 2023-03-09T19:31:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:31:34 smithi038 ceph-mon[92422]: pgmap v9261: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 58 KiB/s wr, 8 op/s 2023-03-09T19:31:34.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:31:34 smithi122 ceph-mon[93869]: pgmap v9261: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 58 KiB/s wr, 8 op/s 2023-03-09T19:31:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:31:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16406.196310 secs 2023-03-09T19:31:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:31:36 smithi038 ceph-mon[92422]: pgmap v9262: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1023 B/s wr, 7 op/s 2023-03-09T19:31:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:31:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16406.196310 secs 2023-03-09T19:31:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:31:36 smithi122 ceph-mon[93869]: pgmap v9262: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1023 B/s wr, 7 op/s 2023-03-09T19:31:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:31:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16406.196310 secs 2023-03-09T19:31:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:31:36 smithi181 ceph-mon[94189]: pgmap v9262: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1023 B/s wr, 7 op/s 2023-03-09T19:31:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:31:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16408.498986 secs 2023-03-09T19:31:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:31:38 smithi038 ceph-mon[92422]: pgmap v9263: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 10 op/s 2023-03-09T19:31:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:31:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16408.498986 secs 2023-03-09T19:31:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:31:38 smithi122 ceph-mon[93869]: pgmap v9263: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 10 op/s 2023-03-09T19:31:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:31:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16408.498986 secs 2023-03-09T19:31:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:31:38 smithi181 ceph-mon[94189]: pgmap v9263: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 10 op/s 2023-03-09T19:31:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16870.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:31:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:31:40 smithi122 ceph-mon[93869]: pgmap v9264: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 182 KiB/s wr, 9 op/s 2023-03-09T19:31:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:31:40 smithi181 ceph-mon[94189]: pgmap v9264: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 182 KiB/s wr, 9 op/s 2023-03-09T19:31:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:31:40 smithi038 ceph-mon[92422]: pgmap v9264: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 182 KiB/s wr, 9 op/s 2023-03-09T19:31:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:31:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16411.196370 secs 2023-03-09T19:31:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:31:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16411.196370 secs 2023-03-09T19:31:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:31:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16411.196370 secs 2023-03-09T19:31:42.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:31:42 smithi122 ceph-mon[93869]: pgmap v9265: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 195 KiB/s wr, 11 op/s 2023-03-09T19:31:42.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:31:42 smithi181 ceph-mon[94189]: pgmap v9265: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 195 KiB/s wr, 11 op/s 2023-03-09T19:31:42.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:31:42 smithi038 ceph-mon[92422]: pgmap v9265: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 195 KiB/s wr, 11 op/s 2023-03-09T19:31:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:31:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16413.499124 secs 2023-03-09T19:31:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:31:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16413.499124 secs 2023-03-09T19:31:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:31:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16413.499124 secs 2023-03-09T19:31:43.803 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16875.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:31:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:31:44 smithi122 ceph-mon[93869]: pgmap v9266: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 313 KiB/s wr, 11 op/s 2023-03-09T19:31:44.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:31:44 smithi181 ceph-mon[94189]: pgmap v9266: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 313 KiB/s wr, 11 op/s 2023-03-09T19:31:44.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:31:44 smithi038 ceph-mon[92422]: pgmap v9266: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 313 KiB/s wr, 11 op/s 2023-03-09T19:31:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:31:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16416.196430 secs 2023-03-09T19:31:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:31:46 smithi122 ceph-mon[93869]: pgmap v9267: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 313 KiB/s wr, 9 op/s 2023-03-09T19:31:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:31:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16416.196430 secs 2023-03-09T19:31:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:31:46 smithi181 ceph-mon[94189]: pgmap v9267: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 313 KiB/s wr, 9 op/s 2023-03-09T19:31:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:31:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16416.196430 secs 2023-03-09T19:31:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:31:46 smithi038 ceph-mon[92422]: pgmap v9267: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 313 KiB/s wr, 9 op/s 2023-03-09T19:31:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:31:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16418.499217 secs 2023-03-09T19:31:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:31:48 smithi122 ceph-mon[93869]: pgmap v9268: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 394 KiB/s wr, 13 op/s 2023-03-09T19:31:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:31:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16418.499217 secs 2023-03-09T19:31:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:31:48 smithi181 ceph-mon[94189]: pgmap v9268: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 394 KiB/s wr, 13 op/s 2023-03-09T19:31:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:31:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16418.499217 secs 2023-03-09T19:31:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:31:48 smithi038 ceph-mon[92422]: pgmap v9268: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 394 KiB/s wr, 13 op/s 2023-03-09T19:31:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16880.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:31:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:31:50 smithi122 ceph-mon[93869]: pgmap v9269: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 395 KiB/s rd, 381 KiB/s wr, 11 op/s 2023-03-09T19:31:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:31:50 smithi181 ceph-mon[94189]: pgmap v9269: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 395 KiB/s rd, 381 KiB/s wr, 11 op/s 2023-03-09T19:31:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:31:50 smithi038 ceph-mon[92422]: pgmap v9269: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 395 KiB/s rd, 381 KiB/s wr, 11 op/s 2023-03-09T19:31:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:31:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16421.196491 secs 2023-03-09T19:31:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:31:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16421.196491 secs 2023-03-09T19:31:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:31:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16421.196491 secs 2023-03-09T19:31:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:31:52 smithi122 ceph-mon[93869]: pgmap v9270: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 346 KiB/s wr, 13 op/s 2023-03-09T19:31:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:31:52 smithi181 ceph-mon[94189]: pgmap v9270: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 346 KiB/s wr, 13 op/s 2023-03-09T19:31:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:31:52 smithi038 ceph-mon[92422]: pgmap v9270: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 346 KiB/s wr, 13 op/s 2023-03-09T19:31:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:31:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16423.499308 secs 2023-03-09T19:31:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:31:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16423.499308 secs 2023-03-09T19:31:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:31:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16423.499308 secs 2023-03-09T19:31:53.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16885.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:31:54.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:31:54 smithi122 ceph-mon[93869]: pgmap v9271: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 333 KiB/s wr, 11 op/s 2023-03-09T19:31:54.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:31:54 smithi181 ceph-mon[94189]: pgmap v9271: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 333 KiB/s wr, 11 op/s 2023-03-09T19:31:54.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:31:54 smithi038 ceph-mon[92422]: pgmap v9271: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 333 KiB/s wr, 11 op/s 2023-03-09T19:31:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:31:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16426.196549 secs 2023-03-09T19:31:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:31:56 smithi122 ceph-mon[93869]: pgmap v9272: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 216 KiB/s wr, 9 op/s 2023-03-09T19:31:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:31:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16426.196549 secs 2023-03-09T19:31:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:31:56 smithi181 ceph-mon[94189]: pgmap v9272: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 216 KiB/s wr, 9 op/s 2023-03-09T19:31:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:31:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16426.196549 secs 2023-03-09T19:31:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:31:56 smithi038 ceph-mon[92422]: pgmap v9272: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 216 KiB/s wr, 9 op/s 2023-03-09T19:31:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:31:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16428.499388 secs 2023-03-09T19:31:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:31:58 smithi122 ceph-mon[93869]: pgmap v9273: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 363 KiB/s rd, 304 KiB/s wr, 12 op/s 2023-03-09T19:31:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:31:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:31:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:31:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16428.499388 secs 2023-03-09T19:31:58.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:31:58 smithi181 ceph-mon[94189]: pgmap v9273: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 363 KiB/s rd, 304 KiB/s wr, 12 op/s 2023-03-09T19:31:58.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:31:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:31:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:31:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16428.499388 secs 2023-03-09T19:31:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:31:58 smithi038 ceph-mon[92422]: pgmap v9273: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 363 KiB/s rd, 304 KiB/s wr, 12 op/s 2023-03-09T19:31:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:31:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:31:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16890.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:31:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:31:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:31:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:31:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:31:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:31:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:31:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:31:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:31:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:31:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:31:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:31:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:31:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:31:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:31:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:31:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:31:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:31:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:32:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:32:00 smithi122 ceph-mon[93869]: pgmap v9274: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 222 KiB/s wr, 9 op/s 2023-03-09T19:32:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:32:00 smithi181 ceph-mon[94189]: pgmap v9274: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 222 KiB/s wr, 9 op/s 2023-03-09T19:32:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:32:00 smithi038 ceph-mon[92422]: pgmap v9274: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 222 KiB/s wr, 9 op/s 2023-03-09T19:32:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:32:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16431.196611 secs 2023-03-09T19:32:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:32:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16431.196611 secs 2023-03-09T19:32:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:32:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16431.196611 secs 2023-03-09T19:32:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:32:02 smithi122 ceph-mon[93869]: pgmap v9275: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 323 KiB/s wr, 14 op/s 2023-03-09T19:32:02.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:32:02 smithi181 ceph-mon[94189]: pgmap v9275: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 323 KiB/s wr, 14 op/s 2023-03-09T19:32:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:32:02 smithi038 ceph-mon[92422]: pgmap v9275: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 323 KiB/s wr, 14 op/s 2023-03-09T19:32:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:32:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16433.499514 secs 2023-03-09T19:32:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:32:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16433.499514 secs 2023-03-09T19:32:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:32:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16433.499514 secs 2023-03-09T19:32:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16895.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:32:04.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:32:04 smithi122 ceph-mon[93869]: pgmap v9276: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 254 KiB/s wr, 23 op/s 2023-03-09T19:32:04.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:32:04 smithi181 ceph-mon[94189]: pgmap v9276: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 254 KiB/s wr, 23 op/s 2023-03-09T19:32:04.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:32:04 smithi038 ceph-mon[92422]: pgmap v9276: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 254 KiB/s wr, 23 op/s 2023-03-09T19:32:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:32:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16436.196689 secs 2023-03-09T19:32:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:32:06 smithi122 ceph-mon[93869]: pgmap v9277: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 254 KiB/s wr, 22 op/s 2023-03-09T19:32:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:32:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16436.196689 secs 2023-03-09T19:32:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:32:06 smithi181 ceph-mon[94189]: pgmap v9277: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 254 KiB/s wr, 22 op/s 2023-03-09T19:32:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:32:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16436.196689 secs 2023-03-09T19:32:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:32:06 smithi038 ceph-mon[92422]: pgmap v9277: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 254 KiB/s wr, 22 op/s 2023-03-09T19:32:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:32:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16438.499630 secs 2023-03-09T19:32:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:32:08 smithi122 ceph-mon[93869]: pgmap v9278: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 254 KiB/s wr, 25 op/s 2023-03-09T19:32:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:32:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16438.499630 secs 2023-03-09T19:32:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:32:08 smithi181 ceph-mon[94189]: pgmap v9278: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 254 KiB/s wr, 25 op/s 2023-03-09T19:32:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:32:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16438.499630 secs 2023-03-09T19:32:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:32:08 smithi038 ceph-mon[92422]: pgmap v9278: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 254 KiB/s wr, 25 op/s 2023-03-09T19:32:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16900.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:32:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:32:10 smithi122 ceph-mon[93869]: pgmap v9279: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 212 KiB/s wr, 22 op/s 2023-03-09T19:32:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:32:10 smithi122 ceph-mon[93869]: osdmap e793: 12 total, 12 up, 12 in 2023-03-09T19:32:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:32:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16441.196750 secs 2023-03-09T19:32:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:32:10 smithi181 ceph-mon[94189]: pgmap v9279: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 212 KiB/s wr, 22 op/s 2023-03-09T19:32:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:32:10 smithi181 ceph-mon[94189]: osdmap e793: 12 total, 12 up, 12 in 2023-03-09T19:32:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:32:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16441.196750 secs 2023-03-09T19:32:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:32:10 smithi038 ceph-mon[92422]: pgmap v9279: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 212 KiB/s wr, 22 op/s 2023-03-09T19:32:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:32:10 smithi038 ceph-mon[92422]: osdmap e793: 12 total, 12 up, 12 in 2023-03-09T19:32:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:32:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16441.196750 secs 2023-03-09T19:32:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:32:12 smithi122 ceph-mon[93869]: pgmap v9281: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 149 KiB/s wr, 23 op/s 2023-03-09T19:32:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:32:12 smithi181 ceph-mon[94189]: pgmap v9281: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 149 KiB/s wr, 23 op/s 2023-03-09T19:32:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:32:12 smithi038 ceph-mon[92422]: pgmap v9281: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 149 KiB/s wr, 23 op/s 2023-03-09T19:32:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:32:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16443.499747 secs 2023-03-09T19:32:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:32:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16443.499747 secs 2023-03-09T19:32:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:32:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16443.499747 secs 2023-03-09T19:32:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16905.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:32:14.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:32:14 smithi122 ceph-mon[93869]: pgmap v9282: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 9 op/s 2023-03-09T19:32:14.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:32:14 smithi181 ceph-mon[94189]: pgmap v9282: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 9 op/s 2023-03-09T19:32:14.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:32:14 smithi038 ceph-mon[92422]: pgmap v9282: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 9 op/s 2023-03-09T19:32:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:32:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16446.196827 secs 2023-03-09T19:32:15.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:32:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16446.196827 secs 2023-03-09T19:32:15.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:32:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16446.196827 secs 2023-03-09T19:32:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:32:16 smithi122 ceph-mon[93869]: pgmap v9283: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 9 op/s 2023-03-09T19:32:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:32:16 smithi181 ceph-mon[94189]: pgmap v9283: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 9 op/s 2023-03-09T19:32:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:32:16 smithi038 ceph-mon[92422]: pgmap v9283: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 9 op/s 2023-03-09T19:32:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:32:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16448.499883 secs 2023-03-09T19:32:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:32:18 smithi122 ceph-mon[93869]: pgmap v9284: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 88 KiB/s wr, 9 op/s 2023-03-09T19:32:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:32:18 smithi122 ceph-mon[93869]: osdmap e794: 12 total, 12 up, 12 in 2023-03-09T19:32:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:32:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16448.499883 secs 2023-03-09T19:32:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:32:18 smithi181 ceph-mon[94189]: pgmap v9284: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 88 KiB/s wr, 9 op/s 2023-03-09T19:32:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:32:18 smithi181 ceph-mon[94189]: osdmap e794: 12 total, 12 up, 12 in 2023-03-09T19:32:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:32:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16448.499883 secs 2023-03-09T19:32:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:32:18 smithi038 ceph-mon[92422]: pgmap v9284: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 88 KiB/s wr, 9 op/s 2023-03-09T19:32:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:32:18 smithi038 ceph-mon[92422]: osdmap e794: 12 total, 12 up, 12 in 2023-03-09T19:32:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16910.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:32:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:32:20 smithi122 ceph-mon[93869]: pgmap v9286: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 104 KiB/s wr, 9 op/s 2023-03-09T19:32:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:32:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16451.196889 secs 2023-03-09T19:32:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:32:20 smithi181 ceph-mon[94189]: pgmap v9286: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 104 KiB/s wr, 9 op/s 2023-03-09T19:32:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:32:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16451.196889 secs 2023-03-09T19:32:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:32:20 smithi038 ceph-mon[92422]: pgmap v9286: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 104 KiB/s wr, 9 op/s 2023-03-09T19:32:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:32:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16451.196889 secs 2023-03-09T19:32:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:32:22 smithi122 ceph-mon[93869]: pgmap v9287: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 86 KiB/s wr, 8 op/s 2023-03-09T19:32:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:32:22 smithi181 ceph-mon[94189]: pgmap v9287: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 86 KiB/s wr, 8 op/s 2023-03-09T19:32:22.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:32:22 smithi038 ceph-mon[92422]: pgmap v9287: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 86 KiB/s wr, 8 op/s 2023-03-09T19:32:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:32:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16453.499998 secs 2023-03-09T19:32:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:32:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16453.499998 secs 2023-03-09T19:32:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:32:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16453.499998 secs 2023-03-09T19:32:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16915.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:32:24.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:32:24 smithi122 ceph-mon[93869]: pgmap v9288: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 85 KiB/s wr, 9 op/s 2023-03-09T19:32:24.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:32:24 smithi181 ceph-mon[94189]: pgmap v9288: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 85 KiB/s wr, 9 op/s 2023-03-09T19:32:24.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:32:24 smithi038 ceph-mon[92422]: pgmap v9288: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 85 KiB/s wr, 9 op/s 2023-03-09T19:32:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:32:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16456.196973 secs 2023-03-09T19:32:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:32:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16456.196973 secs 2023-03-09T19:32:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:32:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16456.196973 secs 2023-03-09T19:32:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:32:26 smithi122 ceph-mon[93869]: pgmap v9289: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 85 KiB/s wr, 9 op/s 2023-03-09T19:32:26.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:32:26 smithi181 ceph-mon[94189]: pgmap v9289: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 85 KiB/s wr, 9 op/s 2023-03-09T19:32:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:32:26 smithi038 ceph-mon[92422]: pgmap v9289: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 85 KiB/s wr, 9 op/s 2023-03-09T19:32:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:32:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16458.500136 secs 2023-03-09T19:32:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:32:28 smithi122 ceph-mon[93869]: pgmap v9290: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 70 KiB/s wr, 8 op/s 2023-03-09T19:32:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:32:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16458.500136 secs 2023-03-09T19:32:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:32:28 smithi181 ceph-mon[94189]: pgmap v9290: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 70 KiB/s wr, 8 op/s 2023-03-09T19:32:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:32:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16458.500136 secs 2023-03-09T19:32:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:32:28 smithi038 ceph-mon[92422]: pgmap v9290: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 70 KiB/s wr, 8 op/s 2023-03-09T19:32:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16920.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:32:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:32:30 smithi122 ceph-mon[93869]: pgmap v9291: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 124 KiB/s wr, 8 op/s 2023-03-09T19:32:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:32:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16461.197032 secs 2023-03-09T19:32:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:32:30 smithi181 ceph-mon[94189]: pgmap v9291: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 124 KiB/s wr, 8 op/s 2023-03-09T19:32:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:32:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16461.197032 secs 2023-03-09T19:32:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:32:30 smithi038 ceph-mon[92422]: pgmap v9291: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 124 KiB/s wr, 8 op/s 2023-03-09T19:32:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:32:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16461.197032 secs 2023-03-09T19:32:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:32:32 smithi122 ceph-mon[93869]: pgmap v9292: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 8 op/s 2023-03-09T19:32:32.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:32:32 smithi181 ceph-mon[94189]: pgmap v9292: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 8 op/s 2023-03-09T19:32:32.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:32:32 smithi038 ceph-mon[92422]: pgmap v9292: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 8 op/s 2023-03-09T19:32:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:32:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16463.500249 secs 2023-03-09T19:32:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:32:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16463.500249 secs 2023-03-09T19:32:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:32:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16463.500249 secs 2023-03-09T19:32:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16925.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:32:34.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:32:34 smithi122 ceph-mon[93869]: pgmap v9293: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 111 KiB/s wr, 7 op/s 2023-03-09T19:32:34.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:32:34 smithi181 ceph-mon[94189]: pgmap v9293: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 111 KiB/s wr, 7 op/s 2023-03-09T19:32:34.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:32:34 smithi038 ceph-mon[92422]: pgmap v9293: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 111 KiB/s wr, 7 op/s 2023-03-09T19:32:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:32:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16466.197093 secs 2023-03-09T19:32:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:32:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16466.197093 secs 2023-03-09T19:32:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:32:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16466.197093 secs 2023-03-09T19:32:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:32:36 smithi122 ceph-mon[93869]: pgmap v9294: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 110 KiB/s wr, 5 op/s 2023-03-09T19:32:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:32:36 smithi181 ceph-mon[94189]: pgmap v9294: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 110 KiB/s wr, 5 op/s 2023-03-09T19:32:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:32:36 smithi038 ceph-mon[92422]: pgmap v9294: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 110 KiB/s wr, 5 op/s 2023-03-09T19:32:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:32:38 smithi122 ceph-mon[93869]: pgmap v9295: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 111 KiB/s wr, 8 op/s 2023-03-09T19:32:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:32:38 smithi181 ceph-mon[94189]: pgmap v9295: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 111 KiB/s wr, 8 op/s 2023-03-09T19:32:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:32:38 smithi038 ceph-mon[92422]: pgmap v9295: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 111 KiB/s wr, 8 op/s 2023-03-09T19:32:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16930.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:32:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:32:40 smithi122 ceph-mon[93869]: pgmap v9296: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 171 KiB/s wr, 6 op/s 2023-03-09T19:32:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:32:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16471.197154 secs 2023-03-09T19:32:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:32:40 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:32:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:32:40 smithi181 ceph-mon[94189]: pgmap v9296: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 171 KiB/s wr, 6 op/s 2023-03-09T19:32:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:32:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16471.197154 secs 2023-03-09T19:32:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:32:40 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:32:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:32:40 smithi038 ceph-mon[92422]: pgmap v9296: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 171 KiB/s wr, 6 op/s 2023-03-09T19:32:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:32:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16471.197154 secs 2023-03-09T19:32:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:32:40 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:32:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:32:41 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:32:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:32:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:32:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:32:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:32:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:32:41 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:32:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:32:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:32:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:32:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:32:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:32:41 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:32:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:32:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:32:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:32:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:32:42.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:32:42 smithi122 ceph-mon[93869]: pgmap v9297: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 127 KiB/s wr, 8 op/s 2023-03-09T19:32:42.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:32:42 smithi181 ceph-mon[94189]: pgmap v9297: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 127 KiB/s wr, 8 op/s 2023-03-09T19:32:42.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:32:42 smithi038 ceph-mon[92422]: pgmap v9297: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 127 KiB/s wr, 8 op/s 2023-03-09T19:32:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16935.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:32:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:32:43 smithi122 ceph-mon[93869]: pgmap v9298: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 247 KiB/s rd, 188 KiB/s wr, 8 op/s 2023-03-09T19:32:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:32:43 smithi181 ceph-mon[94189]: pgmap v9298: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 247 KiB/s rd, 188 KiB/s wr, 8 op/s 2023-03-09T19:32:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:32:43 smithi038 ceph-mon[92422]: pgmap v9298: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 247 KiB/s rd, 188 KiB/s wr, 8 op/s 2023-03-09T19:32:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:32:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16476.197214 secs 2023-03-09T19:32:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:32:46 smithi122 ceph-mon[93869]: pgmap v9299: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 135 KiB/s wr, 7 op/s 2023-03-09T19:32:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:32:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16476.197214 secs 2023-03-09T19:32:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:32:46 smithi181 ceph-mon[94189]: pgmap v9299: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 135 KiB/s wr, 7 op/s 2023-03-09T19:32:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:32:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16476.197214 secs 2023-03-09T19:32:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:32:46 smithi038 ceph-mon[92422]: pgmap v9299: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 135 KiB/s wr, 7 op/s 2023-03-09T19:32:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:32:48 smithi122 ceph-mon[93869]: pgmap v9300: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 246 KiB/s wr, 11 op/s 2023-03-09T19:32:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:32:48 smithi181 ceph-mon[94189]: pgmap v9300: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 246 KiB/s wr, 11 op/s 2023-03-09T19:32:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:32:48 smithi038 ceph-mon[92422]: pgmap v9300: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 246 KiB/s wr, 11 op/s 2023-03-09T19:32:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16940.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:32:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:32:50 smithi122 ceph-mon[93869]: pgmap v9301: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 246 KiB/s wr, 10 op/s 2023-03-09T19:32:50.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:32:50 smithi181 ceph-mon[94189]: pgmap v9301: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 246 KiB/s wr, 10 op/s 2023-03-09T19:32:50.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:32:50 smithi038 ceph-mon[92422]: pgmap v9301: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 246 KiB/s wr, 10 op/s 2023-03-09T19:32:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:32:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16481.197273 secs 2023-03-09T19:32:51.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:32:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16481.197273 secs 2023-03-09T19:32:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:32:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16481.197273 secs 2023-03-09T19:32:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:32:52 smithi122 ceph-mon[93869]: pgmap v9302: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 199 KiB/s wr, 12 op/s 2023-03-09T19:32:52.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:32:52 smithi181 ceph-mon[94189]: pgmap v9302: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 199 KiB/s wr, 12 op/s 2023-03-09T19:32:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:32:52 smithi038 ceph-mon[92422]: pgmap v9302: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 199 KiB/s wr, 12 op/s 2023-03-09T19:32:53.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16945.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:32:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:32:53 smithi181 ceph-mon[94189]: pgmap v9303: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 337 KiB/s wr, 11 op/s 2023-03-09T19:32:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:32:53 smithi038 ceph-mon[92422]: pgmap v9303: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 337 KiB/s wr, 11 op/s 2023-03-09T19:32:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:32:53 smithi122 ceph-mon[93869]: pgmap v9303: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 337 KiB/s wr, 11 op/s 2023-03-09T19:32:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:32:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16486.197350 secs 2023-03-09T19:32:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:32:56 smithi122 ceph-mon[93869]: pgmap v9304: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 276 KiB/s wr, 10 op/s 2023-03-09T19:32:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:32:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16486.197350 secs 2023-03-09T19:32:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:32:56 smithi181 ceph-mon[94189]: pgmap v9304: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 276 KiB/s wr, 10 op/s 2023-03-09T19:32:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:32:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16486.197350 secs 2023-03-09T19:32:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:32:56 smithi038 ceph-mon[92422]: pgmap v9304: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 276 KiB/s wr, 10 op/s 2023-03-09T19:32:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:32:58 smithi122 ceph-mon[93869]: pgmap v9305: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 347 KiB/s wr, 13 op/s 2023-03-09T19:32:58.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:32:58 smithi181 ceph-mon[94189]: pgmap v9305: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 347 KiB/s wr, 13 op/s 2023-03-09T19:32:58.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:32:58 smithi038 ceph-mon[92422]: pgmap v9305: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 347 KiB/s wr, 13 op/s 2023-03-09T19:32:58.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16950.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:32:59.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:32:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:32:59.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:32:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:32:59.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:32:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:33:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:33:00 smithi122 ceph-mon[93869]: pgmap v9306: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 300 KiB/s wr, 10 op/s 2023-03-09T19:33:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:33:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:33:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:33:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:33:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:33:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:33:00.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:33:00 smithi181 ceph-mon[94189]: pgmap v9306: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 300 KiB/s wr, 10 op/s 2023-03-09T19:33:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:33:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:33:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:33:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:33:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:33:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:33:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:33:00 smithi038 ceph-mon[92422]: pgmap v9306: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 300 KiB/s wr, 10 op/s 2023-03-09T19:33:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:33:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:33:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:33:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:33:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:33:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:33:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:33:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16491.197411 secs 2023-03-09T19:33:01.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:33:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16491.197411 secs 2023-03-09T19:33:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:33:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16491.197411 secs 2023-03-09T19:33:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:33:02 smithi122 ceph-mon[93869]: pgmap v9307: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 335 KiB/s wr, 14 op/s 2023-03-09T19:33:02.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:33:02 smithi181 ceph-mon[94189]: pgmap v9307: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 335 KiB/s wr, 14 op/s 2023-03-09T19:33:02.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:33:02 smithi038 ceph-mon[92422]: pgmap v9307: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 335 KiB/s wr, 14 op/s 2023-03-09T19:33:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:33:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16493.500942 secs 2023-03-09T19:33:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:33:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16493.500942 secs 2023-03-09T19:33:03.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:33:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16493.500942 secs 2023-03-09T19:33:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16955.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:33:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:33:04 smithi122 ceph-mon[93869]: pgmap v9308: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 386 KiB/s rd, 343 KiB/s wr, 24 op/s 2023-03-09T19:33:04.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:33:04 smithi181 ceph-mon[94189]: pgmap v9308: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 386 KiB/s rd, 343 KiB/s wr, 24 op/s 2023-03-09T19:33:04.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:33:04 smithi038 ceph-mon[92422]: pgmap v9308: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 386 KiB/s rd, 343 KiB/s wr, 24 op/s 2023-03-09T19:33:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:33:05 smithi122 ceph-mon[93869]: osdmap e795: 12 total, 12 up, 12 in 2023-03-09T19:33:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:33:05 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:33:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:33:05 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:33:05.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:33:05 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:33:05.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:33:05 smithi181 ceph-mon[94189]: osdmap e795: 12 total, 12 up, 12 in 2023-03-09T19:33:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:33:05 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:33:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:33:05 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:33:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:33:05 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:33:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:33:05 smithi038 ceph-mon[92422]: osdmap e795: 12 total, 12 up, 12 in 2023-03-09T19:33:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:33:05 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:33:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:33:05 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:33:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:33:05 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:33:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:33:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16496.197472 secs 2023-03-09T19:33:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:33:06 smithi122 ceph-mon[93869]: pgmap v9310: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 485 KiB/s rd, 412 KiB/s wr, 28 op/s 2023-03-09T19:33:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:33:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16496.197472 secs 2023-03-09T19:33:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:33:06 smithi181 ceph-mon[94189]: pgmap v9310: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 485 KiB/s rd, 412 KiB/s wr, 28 op/s 2023-03-09T19:33:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:33:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16496.197472 secs 2023-03-09T19:33:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:33:06 smithi038 ceph-mon[92422]: pgmap v9310: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 485 KiB/s rd, 412 KiB/s wr, 28 op/s 2023-03-09T19:33:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:33:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16498.501056 secs 2023-03-09T19:33:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:33:08 smithi122 ceph-mon[93869]: pgmap v9311: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 327 KiB/s wr, 27 op/s 2023-03-09T19:33:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:33:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16498.501056 secs 2023-03-09T19:33:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:33:08 smithi181 ceph-mon[94189]: pgmap v9311: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 327 KiB/s wr, 27 op/s 2023-03-09T19:33:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:33:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16498.501056 secs 2023-03-09T19:33:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:33:08 smithi038 ceph-mon[92422]: pgmap v9311: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 327 KiB/s wr, 27 op/s 2023-03-09T19:33:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16960.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:33:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:33:10 smithi122 ceph-mon[93869]: pgmap v9312: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 297 KiB/s wr, 27 op/s 2023-03-09T19:33:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:33:10 smithi181 ceph-mon[94189]: pgmap v9312: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 297 KiB/s wr, 27 op/s 2023-03-09T19:33:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:33:10 smithi038 ceph-mon[92422]: pgmap v9312: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 297 KiB/s wr, 27 op/s 2023-03-09T19:33:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:33:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16501.197533 secs 2023-03-09T19:33:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:33:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16501.197533 secs 2023-03-09T19:33:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:33:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16501.197533 secs 2023-03-09T19:33:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:33:12 smithi038 ceph-mon[92422]: pgmap v9313: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 271 KiB/s wr, 25 op/s 2023-03-09T19:33:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:33:12 smithi122 ceph-mon[93869]: pgmap v9313: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 271 KiB/s wr, 25 op/s 2023-03-09T19:33:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:33:12 smithi181 ceph-mon[94189]: pgmap v9313: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 271 KiB/s wr, 25 op/s 2023-03-09T19:33:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:33:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16503.501177 secs 2023-03-09T19:33:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:33:13 smithi038 ceph-mon[92422]: osdmap e796: 12 total, 12 up, 12 in 2023-03-09T19:33:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:33:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16503.501177 secs 2023-03-09T19:33:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:33:13 smithi122 ceph-mon[93869]: osdmap e796: 12 total, 12 up, 12 in 2023-03-09T19:33:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:33:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16503.501177 secs 2023-03-09T19:33:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:33:13 smithi181 ceph-mon[94189]: osdmap e796: 12 total, 12 up, 12 in 2023-03-09T19:33:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16965.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:33:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:33:14 smithi038 ceph-mon[92422]: pgmap v9315: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 71 KiB/s wr, 8 op/s 2023-03-09T19:33:14.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:33:14 smithi122 ceph-mon[93869]: pgmap v9315: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 71 KiB/s wr, 8 op/s 2023-03-09T19:33:14.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:33:14 smithi181 ceph-mon[94189]: pgmap v9315: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 71 KiB/s wr, 8 op/s 2023-03-09T19:33:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:33:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16506.197592 secs 2023-03-09T19:33:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:33:16 smithi038 ceph-mon[92422]: pgmap v9316: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 63 KiB/s wr, 8 op/s 2023-03-09T19:33:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:33:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16506.197592 secs 2023-03-09T19:33:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:33:16 smithi122 ceph-mon[93869]: pgmap v9316: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 63 KiB/s wr, 8 op/s 2023-03-09T19:33:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:33:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16506.197592 secs 2023-03-09T19:33:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:33:16 smithi181 ceph-mon[94189]: pgmap v9316: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 63 KiB/s wr, 8 op/s 2023-03-09T19:33:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:33:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16508.501294 secs 2023-03-09T19:33:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:33:18 smithi038 ceph-mon[92422]: pgmap v9317: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 80 KiB/s wr, 8 op/s 2023-03-09T19:33:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:33:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16508.501294 secs 2023-03-09T19:33:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:33:18 smithi122 ceph-mon[93869]: pgmap v9317: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 80 KiB/s wr, 8 op/s 2023-03-09T19:33:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:33:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16508.501294 secs 2023-03-09T19:33:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:33:18 smithi181 ceph-mon[94189]: pgmap v9317: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 80 KiB/s wr, 8 op/s 2023-03-09T19:33:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16970.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:33:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:33:20 smithi122 ceph-mon[93869]: pgmap v9318: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 70 KiB/s wr, 7 op/s 2023-03-09T19:33:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:33:20 smithi181 ceph-mon[94189]: pgmap v9318: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 70 KiB/s wr, 7 op/s 2023-03-09T19:33:20.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:33:20 smithi038 ceph-mon[92422]: pgmap v9318: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 70 KiB/s wr, 7 op/s 2023-03-09T19:33:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:33:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16511.197659 secs 2023-03-09T19:33:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:33:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16511.197659 secs 2023-03-09T19:33:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:33:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16511.197659 secs 2023-03-09T19:33:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:33:22 smithi122 ceph-mon[93869]: pgmap v9319: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 55 KiB/s wr, 7 op/s 2023-03-09T19:33:22.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:33:22 smithi181 ceph-mon[94189]: pgmap v9319: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 55 KiB/s wr, 7 op/s 2023-03-09T19:33:22.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:33:22 smithi038 ceph-mon[92422]: pgmap v9319: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 55 KiB/s wr, 7 op/s 2023-03-09T19:33:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:33:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16513.501411 secs 2023-03-09T19:33:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:33:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16513.501411 secs 2023-03-09T19:33:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:33:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16513.501411 secs 2023-03-09T19:33:23.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16975.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:33:24.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:33:24 smithi122 ceph-mon[93869]: pgmap v9320: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 54 KiB/s wr, 7 op/s 2023-03-09T19:33:24.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:33:24 smithi181 ceph-mon[94189]: pgmap v9320: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 54 KiB/s wr, 7 op/s 2023-03-09T19:33:24.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:33:24 smithi038 ceph-mon[92422]: pgmap v9320: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 54 KiB/s wr, 7 op/s 2023-03-09T19:33:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:33:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16516.197721 secs 2023-03-09T19:33:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:33:26 smithi122 ceph-mon[93869]: pgmap v9321: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 102 KiB/s wr, 7 op/s 2023-03-09T19:33:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:33:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16516.197721 secs 2023-03-09T19:33:26.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:33:26 smithi181 ceph-mon[94189]: pgmap v9321: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 102 KiB/s wr, 7 op/s 2023-03-09T19:33:26.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:33:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16516.197721 secs 2023-03-09T19:33:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:33:26 smithi038 ceph-mon[92422]: pgmap v9321: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 102 KiB/s wr, 7 op/s 2023-03-09T19:33:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:33:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16518.501534 secs 2023-03-09T19:33:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:33:28 smithi122 ceph-mon[93869]: pgmap v9322: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 103 KiB/s wr, 9 op/s 2023-03-09T19:33:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:33:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16518.501534 secs 2023-03-09T19:33:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:33:28 smithi181 ceph-mon[94189]: pgmap v9322: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 103 KiB/s wr, 9 op/s 2023-03-09T19:33:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:33:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16518.501534 secs 2023-03-09T19:33:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:33:28 smithi038 ceph-mon[92422]: pgmap v9322: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 103 KiB/s wr, 9 op/s 2023-03-09T19:33:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16980.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:33:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:33:30 smithi122 ceph-mon[93869]: pgmap v9323: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 89 KiB/s wr, 7 op/s 2023-03-09T19:33:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:33:30 smithi181 ceph-mon[94189]: pgmap v9323: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 89 KiB/s wr, 7 op/s 2023-03-09T19:33:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:33:30 smithi038 ceph-mon[92422]: pgmap v9323: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 89 KiB/s wr, 7 op/s 2023-03-09T19:33:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:33:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16521.197806 secs 2023-03-09T19:33:31.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:33:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16521.197806 secs 2023-03-09T19:33:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:33:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16521.197806 secs 2023-03-09T19:33:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:33:32 smithi122 ceph-mon[93869]: pgmap v9324: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 58 KiB/s wr, 9 op/s 2023-03-09T19:33:32.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:33:32 smithi181 ceph-mon[94189]: pgmap v9324: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 58 KiB/s wr, 9 op/s 2023-03-09T19:33:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:33:32 smithi038 ceph-mon[92422]: pgmap v9324: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 58 KiB/s wr, 9 op/s 2023-03-09T19:33:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:33:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16523.501654 secs 2023-03-09T19:33:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:33:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16523.501654 secs 2023-03-09T19:33:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:33:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16523.501654 secs 2023-03-09T19:33:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16985.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:33:34.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:33:34 smithi122 ceph-mon[93869]: pgmap v9325: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 58 KiB/s wr, 7 op/s 2023-03-09T19:33:34.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:33:34 smithi181 ceph-mon[94189]: pgmap v9325: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 58 KiB/s wr, 7 op/s 2023-03-09T19:33:34.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:33:34 smithi038 ceph-mon[92422]: pgmap v9325: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 58 KiB/s wr, 7 op/s 2023-03-09T19:33:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:33:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16526.197864 secs 2023-03-09T19:33:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:33:36 smithi122 ceph-mon[93869]: pgmap v9326: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 58 KiB/s wr, 7 op/s 2023-03-09T19:33:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:33:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16526.197864 secs 2023-03-09T19:33:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:33:36 smithi181 ceph-mon[94189]: pgmap v9326: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 58 KiB/s wr, 7 op/s 2023-03-09T19:33:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:33:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16526.197864 secs 2023-03-09T19:33:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:33:36 smithi038 ceph-mon[92422]: pgmap v9326: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 58 KiB/s wr, 7 op/s 2023-03-09T19:33:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:33:38 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 16528.501766 secs 2023-03-09T19:33:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:33:38 smithi122 ceph-mon[93869]: pgmap v9327: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 137 KiB/s wr, 10 op/s 2023-03-09T19:33:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:33:38 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 16528.501766 secs 2023-03-09T19:33:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:33:38 smithi181 ceph-mon[94189]: pgmap v9327: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 137 KiB/s wr, 10 op/s 2023-03-09T19:33:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:33:38 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 16528.501766 secs 2023-03-09T19:33:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:33:38 smithi038 ceph-mon[92422]: pgmap v9327: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 137 KiB/s wr, 10 op/s 2023-03-09T19:33:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16990.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:33:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:33:40 smithi122 ceph-mon[93869]: pgmap v9328: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 137 KiB/s wr, 7 op/s 2023-03-09T19:33:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:33:40 smithi181 ceph-mon[94189]: pgmap v9328: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 137 KiB/s wr, 7 op/s 2023-03-09T19:33:40.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:33:40 smithi038 ceph-mon[92422]: pgmap v9328: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 137 KiB/s wr, 7 op/s 2023-03-09T19:33:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:33:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16531.197922 secs 2023-03-09T19:33:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:33:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:33:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:33:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:33:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:33:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16531.197922 secs 2023-03-09T19:33:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:33:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:33:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:33:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:33:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:33:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16531.197922 secs 2023-03-09T19:33:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:33:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:33:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:33:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:33:42.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:33:42 smithi122 ceph-mon[93869]: pgmap v9329: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 150 KiB/s wr, 9 op/s 2023-03-09T19:33:42.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:33:42 smithi181 ceph-mon[94189]: pgmap v9329: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 150 KiB/s wr, 9 op/s 2023-03-09T19:33:42.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:33:42 smithi038 ceph-mon[92422]: pgmap v9329: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 150 KiB/s wr, 9 op/s 2023-03-09T19:33:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:33:43 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 16533.501878 secs 2023-03-09T19:33:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:33:43 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 16533.501878 secs 2023-03-09T19:33:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:33:43 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 16533.501878 secs 2023-03-09T19:33:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 16995.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:33:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:33:44 smithi122 ceph-mon[93869]: pgmap v9330: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 211 KiB/s wr, 8 op/s 2023-03-09T19:33:44.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:33:44 smithi181 ceph-mon[94189]: pgmap v9330: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 211 KiB/s wr, 8 op/s 2023-03-09T19:33:44.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:33:44 smithi038 ceph-mon[92422]: pgmap v9330: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 211 KiB/s wr, 8 op/s 2023-03-09T19:33:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:33:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16536.197985 secs 2023-03-09T19:33:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:33:46 smithi122 ceph-mon[93869]: pgmap v9331: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 211 KiB/s wr, 7 op/s 2023-03-09T19:33:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:33:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16536.197985 secs 2023-03-09T19:33:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:33:46 smithi181 ceph-mon[94189]: pgmap v9331: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 211 KiB/s wr, 7 op/s 2023-03-09T19:33:46.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:33:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16536.197985 secs 2023-03-09T19:33:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:33:46 smithi038 ceph-mon[92422]: pgmap v9331: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 211 KiB/s wr, 7 op/s 2023-03-09T19:33:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:33:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16538.502026 secs 2023-03-09T19:33:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:33:48 smithi122 ceph-mon[93869]: pgmap v9332: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 247 KiB/s wr, 11 op/s 2023-03-09T19:33:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:33:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16538.502026 secs 2023-03-09T19:33:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:33:48 smithi181 ceph-mon[94189]: pgmap v9332: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 247 KiB/s wr, 11 op/s 2023-03-09T19:33:48.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:33:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16538.502026 secs 2023-03-09T19:33:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:33:48 smithi038 ceph-mon[92422]: pgmap v9332: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 247 KiB/s wr, 11 op/s 2023-03-09T19:33:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17000.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:33:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:33:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:33:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:33:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:33:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:33:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:33:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:33:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:33:49.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:33:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:33:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:33:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:33:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:33:50 smithi122 ceph-mon[93869]: pgmap v9333: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 195 KiB/s wr, 9 op/s 2023-03-09T19:33:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:33:50 smithi181 ceph-mon[94189]: pgmap v9333: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 195 KiB/s wr, 9 op/s 2023-03-09T19:33:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:33:50 smithi038 ceph-mon[92422]: pgmap v9333: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 195 KiB/s wr, 9 op/s 2023-03-09T19:33:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:33:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16541.198044 secs 2023-03-09T19:33:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:33:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16541.198044 secs 2023-03-09T19:33:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:33:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16541.198044 secs 2023-03-09T19:33:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:33:52 smithi122 ceph-mon[93869]: pgmap v9334: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 208 KiB/s wr, 11 op/s 2023-03-09T19:33:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:33:52 smithi181 ceph-mon[94189]: pgmap v9334: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 208 KiB/s wr, 11 op/s 2023-03-09T19:33:52.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:33:52 smithi038 ceph-mon[92422]: pgmap v9334: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 208 KiB/s wr, 11 op/s 2023-03-09T19:33:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:33:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16543.502121 secs 2023-03-09T19:33:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:33:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16543.502121 secs 2023-03-09T19:33:53.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:33:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16543.502121 secs 2023-03-09T19:33:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17005.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:33:54.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:33:54 smithi122 ceph-mon[93869]: pgmap v9335: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 317 KiB/s wr, 10 op/s 2023-03-09T19:33:54.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:33:54 smithi181 ceph-mon[94189]: pgmap v9335: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 317 KiB/s wr, 10 op/s 2023-03-09T19:33:54.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:33:54 smithi038 ceph-mon[92422]: pgmap v9335: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 317 KiB/s wr, 10 op/s 2023-03-09T19:33:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:33:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16546.198104 secs 2023-03-09T19:33:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:33:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16546.198104 secs 2023-03-09T19:33:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:33:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16546.198104 secs 2023-03-09T19:33:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:33:56 smithi122 ceph-mon[93869]: pgmap v9336: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 277 KiB/s wr, 9 op/s 2023-03-09T19:33:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:33:56 smithi181 ceph-mon[94189]: pgmap v9336: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 277 KiB/s wr, 9 op/s 2023-03-09T19:33:56.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:33:56 smithi038 ceph-mon[92422]: pgmap v9336: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 277 KiB/s wr, 9 op/s 2023-03-09T19:33:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:33:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16548.502245 secs 2023-03-09T19:33:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:33:58 smithi122 ceph-mon[93869]: pgmap v9337: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 278 KiB/s wr, 12 op/s 2023-03-09T19:33:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:33:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16548.502245 secs 2023-03-09T19:33:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:33:58 smithi181 ceph-mon[94189]: pgmap v9337: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 278 KiB/s wr, 12 op/s 2023-03-09T19:33:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:33:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16548.502245 secs 2023-03-09T19:33:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:33:58 smithi038 ceph-mon[92422]: pgmap v9337: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 278 KiB/s wr, 12 op/s 2023-03-09T19:33:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17010.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:34:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:00 smithi122 ceph-mon[93869]: pgmap v9338: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 447 KiB/s wr, 10 op/s 2023-03-09T19:34:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:34:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16551.198165 secs 2023-03-09T19:34:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:00 smithi181 ceph-mon[94189]: pgmap v9338: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 447 KiB/s wr, 10 op/s 2023-03-09T19:34:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:34:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16551.198165 secs 2023-03-09T19:34:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:00 smithi038 ceph-mon[92422]: pgmap v9338: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 447 KiB/s wr, 10 op/s 2023-03-09T19:34:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:34:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16551.198165 secs 2023-03-09T19:34:02.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:02 smithi122 ceph-mon[93869]: pgmap v9339: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 412 KiB/s wr, 14 op/s 2023-03-09T19:34:02.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:34:02.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:34:02.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:34:02.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:02 smithi181 ceph-mon[94189]: pgmap v9339: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 412 KiB/s wr, 14 op/s 2023-03-09T19:34:02.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:34:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:34:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:34:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:02 smithi038 ceph-mon[92422]: pgmap v9339: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 412 KiB/s wr, 14 op/s 2023-03-09T19:34:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:34:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:34:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:34:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16553.502313 secs 2023-03-09T19:34:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:03 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:34:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:03 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:34:03.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:03 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:34:03.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:03 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:34:03.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:03 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:34:03.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:03 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:34:03.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:03 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:34:03.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:03 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:34:03.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:03 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:34:03.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:03 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:34:03.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:03 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:34:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16553.502313 secs 2023-03-09T19:34:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:03 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:34:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:03 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:34:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:03 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:34:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:03 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:34:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:03 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:34:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:03 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:34:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:03 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:34:03.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:03 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:34:03.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:03 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:34:03.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:03 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:34:03.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:03 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:34:03.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16553.502313 secs 2023-03-09T19:34:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:03 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:34:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:03 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:34:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:03 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:34:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:03 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:34:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:03 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:34:03.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:03 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:34:03.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:03 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:34:03.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:03 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:34:03.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:03 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:34:03.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:03 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:34:03.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:03 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:34:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17015.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:34:04.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:04 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T19:34:04.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:04 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi038 to 141.2M 2023-03-09T19:34:04.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:04 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T19:34:04.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:04 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi038 to 148089716: error parsing value: Value '148089716' is below minimum 939524096 2023-03-09T19:34:04.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:04 smithi122 ceph-mon[93869]: pgmap v9340: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 420 KiB/s wr, 23 op/s 2023-03-09T19:34:04.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:34:04.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:34:04.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:34:04.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:34:04.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:34:04.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:34:04.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:34:04.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:04 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T19:34:04.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:04 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi038 to 141.2M 2023-03-09T19:34:04.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:04 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T19:34:04.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:04 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi038 to 148089716: error parsing value: Value '148089716' is below minimum 939524096 2023-03-09T19:34:04.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:04 smithi181 ceph-mon[94189]: pgmap v9340: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 420 KiB/s wr, 23 op/s 2023-03-09T19:34:04.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:34:04.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:34:04.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:34:04.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:34:04.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:34:04.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:34:04.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:34:04.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:04 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T19:34:04.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:04 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi038 to 141.2M 2023-03-09T19:34:04.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:04 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T19:34:04.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:04 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi038 to 148089716: error parsing value: Value '148089716' is below minimum 939524096 2023-03-09T19:34:04.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:04 smithi038 ceph-mon[92422]: pgmap v9340: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 420 KiB/s wr, 23 op/s 2023-03-09T19:34:04.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:34:04.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:34:04.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:34:04.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:34:04.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:34:04.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:34:04.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:34:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:05 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T19:34:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:05 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T19:34:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16556.198227 secs 2023-03-09T19:34:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:05 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T19:34:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:05 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T19:34:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16556.198227 secs 2023-03-09T19:34:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:05 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T19:34:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:05 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T19:34:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16556.198227 secs 2023-03-09T19:34:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:06 smithi122 ceph-mon[93869]: pgmap v9341: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 348 KiB/s rd, 329 KiB/s wr, 23 op/s 2023-03-09T19:34:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:06 smithi181 ceph-mon[94189]: pgmap v9341: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 348 KiB/s rd, 329 KiB/s wr, 23 op/s 2023-03-09T19:34:06.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:06 smithi038 ceph-mon[92422]: pgmap v9341: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 348 KiB/s rd, 329 KiB/s wr, 23 op/s 2023-03-09T19:34:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16558.502424 secs 2023-03-09T19:34:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:08 smithi122 ceph-mon[93869]: pgmap v9342: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 307 KiB/s wr, 25 op/s 2023-03-09T19:34:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16558.502424 secs 2023-03-09T19:34:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:08 smithi181 ceph-mon[94189]: pgmap v9342: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 307 KiB/s wr, 25 op/s 2023-03-09T19:34:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16558.502424 secs 2023-03-09T19:34:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:08 smithi038 ceph-mon[92422]: pgmap v9342: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 307 KiB/s wr, 25 op/s 2023-03-09T19:34:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17020.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:34:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:10 smithi122 ceph-mon[93869]: pgmap v9343: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 395 KiB/s rd, 407 KiB/s wr, 23 op/s 2023-03-09T19:34:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:10 smithi122 ceph-mon[93869]: osdmap e797: 12 total, 12 up, 12 in 2023-03-09T19:34:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16561.198288 secs 2023-03-09T19:34:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:10 smithi181 ceph-mon[94189]: pgmap v9343: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 395 KiB/s rd, 407 KiB/s wr, 23 op/s 2023-03-09T19:34:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:10 smithi181 ceph-mon[94189]: osdmap e797: 12 total, 12 up, 12 in 2023-03-09T19:34:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16561.198288 secs 2023-03-09T19:34:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:10 smithi038 ceph-mon[92422]: pgmap v9343: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 395 KiB/s rd, 407 KiB/s wr, 23 op/s 2023-03-09T19:34:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:10 smithi038 ceph-mon[92422]: osdmap e797: 12 total, 12 up, 12 in 2023-03-09T19:34:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16561.198288 secs 2023-03-09T19:34:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:12 smithi122 ceph-mon[93869]: pgmap v9345: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 199 KiB/s wr, 24 op/s 2023-03-09T19:34:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:12 smithi181 ceph-mon[94189]: pgmap v9345: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 199 KiB/s wr, 24 op/s 2023-03-09T19:34:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:12 smithi038 ceph-mon[92422]: pgmap v9345: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 199 KiB/s wr, 24 op/s 2023-03-09T19:34:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16563.502543 secs 2023-03-09T19:34:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16563.502543 secs 2023-03-09T19:34:13.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16563.502543 secs 2023-03-09T19:34:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17025.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:34:14.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:14 smithi122 ceph-mon[93869]: pgmap v9346: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 313 KiB/s rd, 229 KiB/s wr, 10 op/s 2023-03-09T19:34:14.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:14 smithi181 ceph-mon[94189]: pgmap v9346: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 313 KiB/s rd, 229 KiB/s wr, 10 op/s 2023-03-09T19:34:14.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:14 smithi038 ceph-mon[92422]: pgmap v9346: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 313 KiB/s rd, 229 KiB/s wr, 10 op/s 2023-03-09T19:34:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16566.198347 secs 2023-03-09T19:34:15.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16566.198347 secs 2023-03-09T19:34:15.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16566.198347 secs 2023-03-09T19:34:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:16 smithi122 ceph-mon[93869]: pgmap v9347: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 192 KiB/s wr, 9 op/s 2023-03-09T19:34:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:16 smithi181 ceph-mon[94189]: pgmap v9347: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 192 KiB/s wr, 9 op/s 2023-03-09T19:34:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:16 smithi038 ceph-mon[92422]: pgmap v9347: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 192 KiB/s wr, 9 op/s 2023-03-09T19:34:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16568.502665 secs 2023-03-09T19:34:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:18 smithi122 ceph-mon[93869]: pgmap v9348: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 208 KiB/s wr, 9 op/s 2023-03-09T19:34:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:18 smithi122 ceph-mon[93869]: osdmap e798: 12 total, 12 up, 12 in 2023-03-09T19:34:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16568.502665 secs 2023-03-09T19:34:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:18 smithi181 ceph-mon[94189]: pgmap v9348: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 208 KiB/s wr, 9 op/s 2023-03-09T19:34:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:18 smithi181 ceph-mon[94189]: osdmap e798: 12 total, 12 up, 12 in 2023-03-09T19:34:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16568.502665 secs 2023-03-09T19:34:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:18 smithi038 ceph-mon[92422]: pgmap v9348: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 208 KiB/s wr, 9 op/s 2023-03-09T19:34:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:18 smithi038 ceph-mon[92422]: osdmap e798: 12 total, 12 up, 12 in 2023-03-09T19:34:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17030.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:34:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:20 smithi122 ceph-mon[93869]: pgmap v9350: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 139 KiB/s wr, 8 op/s 2023-03-09T19:34:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16571.198409 secs 2023-03-09T19:34:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:20 smithi181 ceph-mon[94189]: pgmap v9350: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 139 KiB/s wr, 8 op/s 2023-03-09T19:34:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16571.198409 secs 2023-03-09T19:34:20.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:20 smithi038 ceph-mon[92422]: pgmap v9350: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 139 KiB/s wr, 8 op/s 2023-03-09T19:34:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16571.198409 secs 2023-03-09T19:34:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:22 smithi122 ceph-mon[93869]: pgmap v9351: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 120 KiB/s wr, 7 op/s 2023-03-09T19:34:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:22 smithi181 ceph-mon[94189]: pgmap v9351: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 120 KiB/s wr, 7 op/s 2023-03-09T19:34:22.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:22 smithi038 ceph-mon[92422]: pgmap v9351: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 120 KiB/s wr, 7 op/s 2023-03-09T19:34:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16573.502795 secs 2023-03-09T19:34:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16573.502795 secs 2023-03-09T19:34:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16573.502795 secs 2023-03-09T19:34:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17035.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:34:24.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:24 smithi122 ceph-mon[93869]: pgmap v9352: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 133 KiB/s wr, 7 op/s 2023-03-09T19:34:24.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:24 smithi181 ceph-mon[94189]: pgmap v9352: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 133 KiB/s wr, 7 op/s 2023-03-09T19:34:24.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:24 smithi038 ceph-mon[92422]: pgmap v9352: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 133 KiB/s wr, 7 op/s 2023-03-09T19:34:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16576.198471 secs 2023-03-09T19:34:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16576.198471 secs 2023-03-09T19:34:25.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16576.198471 secs 2023-03-09T19:34:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:26 smithi122 ceph-mon[93869]: pgmap v9353: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 133 KiB/s wr, 7 op/s 2023-03-09T19:34:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:26 smithi181 ceph-mon[94189]: pgmap v9353: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 133 KiB/s wr, 7 op/s 2023-03-09T19:34:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:26 smithi038 ceph-mon[92422]: pgmap v9353: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 133 KiB/s wr, 7 op/s 2023-03-09T19:34:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16578.502949 secs 2023-03-09T19:34:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:28 smithi038 ceph-mon[92422]: pgmap v9354: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 118 KiB/s wr, 7 op/s 2023-03-09T19:34:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17040.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:34:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16578.502949 secs 2023-03-09T19:34:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:28 smithi122 ceph-mon[93869]: pgmap v9354: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 118 KiB/s wr, 7 op/s 2023-03-09T19:34:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16578.502949 secs 2023-03-09T19:34:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:28 smithi181 ceph-mon[94189]: pgmap v9354: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 118 KiB/s wr, 7 op/s 2023-03-09T19:34:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:30 smithi038 ceph-mon[92422]: pgmap v9355: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 106 KiB/s wr, 7 op/s 2023-03-09T19:34:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16581.198531 secs 2023-03-09T19:34:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:30 smithi122 ceph-mon[93869]: pgmap v9355: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 106 KiB/s wr, 7 op/s 2023-03-09T19:34:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16581.198531 secs 2023-03-09T19:34:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:30 smithi181 ceph-mon[94189]: pgmap v9355: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 106 KiB/s wr, 7 op/s 2023-03-09T19:34:30.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16581.198531 secs 2023-03-09T19:34:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:31 smithi122 ceph-mon[93869]: pgmap v9356: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 59 KiB/s wr, 8 op/s 2023-03-09T19:34:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:31 smithi181 ceph-mon[94189]: pgmap v9356: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 59 KiB/s wr, 8 op/s 2023-03-09T19:34:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:31 smithi038 ceph-mon[92422]: pgmap v9356: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 59 KiB/s wr, 8 op/s 2023-03-09T19:34:32.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16583.503082 secs 2023-03-09T19:34:32.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16583.503082 secs 2023-03-09T19:34:32.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16583.503082 secs 2023-03-09T19:34:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17045.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:34:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:34 smithi122 ceph-mon[93869]: pgmap v9357: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 7 op/s 2023-03-09T19:34:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:34 smithi181 ceph-mon[94189]: pgmap v9357: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 7 op/s 2023-03-09T19:34:34.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:34 smithi038 ceph-mon[92422]: pgmap v9357: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 7 op/s 2023-03-09T19:34:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16586.198617 secs 2023-03-09T19:34:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:36 smithi122 ceph-mon[93869]: pgmap v9358: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2023-03-09T19:34:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16586.198617 secs 2023-03-09T19:34:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:36 smithi181 ceph-mon[94189]: pgmap v9358: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2023-03-09T19:34:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16586.198617 secs 2023-03-09T19:34:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:36 smithi038 ceph-mon[92422]: pgmap v9358: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2023-03-09T19:34:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16588.503203 secs 2023-03-09T19:34:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:38 smithi122 ceph-mon[93869]: pgmap v9359: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 15 KiB/s wr, 9 op/s 2023-03-09T19:34:38.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16588.503203 secs 2023-03-09T19:34:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:38 smithi181 ceph-mon[94189]: pgmap v9359: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 15 KiB/s wr, 9 op/s 2023-03-09T19:34:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16588.503203 secs 2023-03-09T19:34:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:38 smithi038 ceph-mon[92422]: pgmap v9359: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 15 KiB/s wr, 9 op/s 2023-03-09T19:34:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17050.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:34:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:40 smithi122 ceph-mon[93869]: pgmap v9360: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 113 KiB/s wr, 8 op/s 2023-03-09T19:34:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:40 smithi181 ceph-mon[94189]: pgmap v9360: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 113 KiB/s wr, 8 op/s 2023-03-09T19:34:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:40 smithi038 ceph-mon[92422]: pgmap v9360: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 113 KiB/s wr, 8 op/s 2023-03-09T19:34:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16591.198698 secs 2023-03-09T19:34:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16591.198698 secs 2023-03-09T19:34:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16591.198698 secs 2023-03-09T19:34:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:42 smithi038 ceph-mon[92422]: pgmap v9361: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 126 KiB/s wr, 10 op/s 2023-03-09T19:34:42.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:42 smithi122 ceph-mon[93869]: pgmap v9361: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 126 KiB/s wr, 10 op/s 2023-03-09T19:34:42.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:42 smithi181 ceph-mon[94189]: pgmap v9361: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 126 KiB/s wr, 10 op/s 2023-03-09T19:34:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16593.503306 secs 2023-03-09T19:34:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16593.503306 secs 2023-03-09T19:34:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16593.503306 secs 2023-03-09T19:34:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17055.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:34:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:44 smithi038 ceph-mon[92422]: pgmap v9362: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 260 KiB/s wr, 10 op/s 2023-03-09T19:34:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:44 smithi122 ceph-mon[93869]: pgmap v9362: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 260 KiB/s wr, 10 op/s 2023-03-09T19:34:44.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:44 smithi181 ceph-mon[94189]: pgmap v9362: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 260 KiB/s wr, 10 op/s 2023-03-09T19:34:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16596.198756 secs 2023-03-09T19:34:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:46 smithi122 ceph-mon[93869]: pgmap v9363: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 260 KiB/s wr, 9 op/s 2023-03-09T19:34:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16596.198756 secs 2023-03-09T19:34:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:46 smithi181 ceph-mon[94189]: pgmap v9363: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 260 KiB/s wr, 9 op/s 2023-03-09T19:34:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16596.198756 secs 2023-03-09T19:34:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:46 smithi038 ceph-mon[92422]: pgmap v9363: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 260 KiB/s wr, 9 op/s 2023-03-09T19:34:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:48 smithi122 ceph-mon[93869]: pgmap v9364: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 416 KiB/s rd, 400 KiB/s wr, 13 op/s 2023-03-09T19:34:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:48 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:34:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:48 smithi181 ceph-mon[94189]: pgmap v9364: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 416 KiB/s rd, 400 KiB/s wr, 13 op/s 2023-03-09T19:34:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:48 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:34:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:48 smithi038 ceph-mon[92422]: pgmap v9364: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 416 KiB/s rd, 400 KiB/s wr, 13 op/s 2023-03-09T19:34:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:48 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:34:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17060.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:34:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:49 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:34:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:34:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:34:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:49 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:34:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:34:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:34:49.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:49 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:34:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:34:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:34:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:50 smithi122 ceph-mon[93869]: pgmap v9365: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 387 KiB/s wr, 10 op/s 2023-03-09T19:34:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:50 smithi181 ceph-mon[94189]: pgmap v9365: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 387 KiB/s wr, 10 op/s 2023-03-09T19:34:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:50 smithi038 ceph-mon[92422]: pgmap v9365: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 387 KiB/s wr, 10 op/s 2023-03-09T19:34:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16601.198827 secs 2023-03-09T19:34:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16601.198827 secs 2023-03-09T19:34:51.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16601.198827 secs 2023-03-09T19:34:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:52 smithi122 ceph-mon[93869]: pgmap v9366: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 303 KiB/s wr, 12 op/s 2023-03-09T19:34:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:52 smithi181 ceph-mon[94189]: pgmap v9366: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 303 KiB/s wr, 12 op/s 2023-03-09T19:34:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:52 smithi038 ceph-mon[92422]: pgmap v9366: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 303 KiB/s wr, 12 op/s 2023-03-09T19:34:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16603.503521 secs 2023-03-09T19:34:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16603.503521 secs 2023-03-09T19:34:53.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16603.503521 secs 2023-03-09T19:34:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17065.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:34:54.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:54 smithi122 ceph-mon[93869]: pgmap v9367: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 371 KiB/s wr, 10 op/s 2023-03-09T19:34:54.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:54 smithi181 ceph-mon[94189]: pgmap v9367: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 371 KiB/s wr, 10 op/s 2023-03-09T19:34:54.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:54 smithi038 ceph-mon[92422]: pgmap v9367: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 371 KiB/s wr, 10 op/s 2023-03-09T19:34:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16606.198909 secs 2023-03-09T19:34:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:56 smithi122 ceph-mon[93869]: pgmap v9368: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 365 KiB/s wr, 9 op/s 2023-03-09T19:34:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16606.198909 secs 2023-03-09T19:34:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:56 smithi181 ceph-mon[94189]: pgmap v9368: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 365 KiB/s wr, 9 op/s 2023-03-09T19:34:56.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16606.198909 secs 2023-03-09T19:34:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:56 smithi038 ceph-mon[92422]: pgmap v9368: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 365 KiB/s wr, 9 op/s 2023-03-09T19:34:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:57 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:34:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:34:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:34:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:57 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:34:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:34:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:34:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:57 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:34:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:34:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:34:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16608.503659 secs 2023-03-09T19:34:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:34:58 smithi122 ceph-mon[93869]: pgmap v9369: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 366 KiB/s wr, 12 op/s 2023-03-09T19:34:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16608.503659 secs 2023-03-09T19:34:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:34:58 smithi181 ceph-mon[94189]: pgmap v9369: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 366 KiB/s wr, 12 op/s 2023-03-09T19:34:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16608.503659 secs 2023-03-09T19:34:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:34:58 smithi038 ceph-mon[92422]: pgmap v9369: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 366 KiB/s wr, 12 op/s 2023-03-09T19:34:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17070.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:35:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:35:00 smithi122 ceph-mon[93869]: pgmap v9370: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 337 KiB/s wr, 9 op/s 2023-03-09T19:35:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:35:00 smithi181 ceph-mon[94189]: pgmap v9370: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 337 KiB/s wr, 9 op/s 2023-03-09T19:35:00.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:35:00 smithi038 ceph-mon[92422]: pgmap v9370: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 337 KiB/s wr, 9 op/s 2023-03-09T19:35:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:35:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16611.198974 secs 2023-03-09T19:35:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:35:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16611.198974 secs 2023-03-09T19:35:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:35:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16611.198974 secs 2023-03-09T19:35:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:35:02 smithi122 ceph-mon[93869]: pgmap v9371: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 359 KiB/s wr, 12 op/s 2023-03-09T19:35:02.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:35:02 smithi181 ceph-mon[94189]: pgmap v9371: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 359 KiB/s wr, 12 op/s 2023-03-09T19:35:02.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:35:02 smithi038 ceph-mon[92422]: pgmap v9371: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 359 KiB/s wr, 12 op/s 2023-03-09T19:35:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:35:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16613.503794 secs 2023-03-09T19:35:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:35:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16613.503794 secs 2023-03-09T19:35:03.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:35:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16613.503794 secs 2023-03-09T19:35:03.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17075.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:35:04.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:35:04 smithi122 ceph-mon[93869]: pgmap v9372: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 366 KiB/s wr, 21 op/s 2023-03-09T19:35:04.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:35:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:35:04.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:35:04 smithi181 ceph-mon[94189]: pgmap v9372: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 366 KiB/s wr, 21 op/s 2023-03-09T19:35:04.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:35:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:35:04.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:35:04 smithi038 ceph-mon[92422]: pgmap v9372: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 366 KiB/s wr, 21 op/s 2023-03-09T19:35:04.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:35:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:35:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:35:05 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:35:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:35:05 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:35:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:35:05 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:35:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:35:05 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:35:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:35:05 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:35:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:35:05 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:35:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:35:05 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:35:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:35:05 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:35:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:35:05 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:35:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:35:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16616.198985 secs 2023-03-09T19:35:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:35:06 smithi122 ceph-mon[93869]: pgmap v9373: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 285 KiB/s wr, 21 op/s 2023-03-09T19:35:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:35:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16616.198985 secs 2023-03-09T19:35:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:35:06 smithi181 ceph-mon[94189]: pgmap v9373: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 285 KiB/s wr, 21 op/s 2023-03-09T19:35:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:35:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16616.198985 secs 2023-03-09T19:35:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:35:06 smithi038 ceph-mon[92422]: pgmap v9373: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 285 KiB/s wr, 21 op/s 2023-03-09T19:35:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:35:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16618.503928 secs 2023-03-09T19:35:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:35:08 smithi122 ceph-mon[93869]: pgmap v9374: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 168 KiB/s wr, 23 op/s 2023-03-09T19:35:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:35:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16618.503928 secs 2023-03-09T19:35:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:35:08 smithi181 ceph-mon[94189]: pgmap v9374: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 168 KiB/s wr, 23 op/s 2023-03-09T19:35:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:35:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16618.503928 secs 2023-03-09T19:35:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:35:08 smithi038 ceph-mon[92422]: pgmap v9374: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 168 KiB/s wr, 23 op/s 2023-03-09T19:35:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17080.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:35:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:35:10 smithi122 ceph-mon[93869]: pgmap v9375: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 168 KiB/s wr, 21 op/s 2023-03-09T19:35:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:35:10 smithi122 ceph-mon[93869]: osdmap e799: 12 total, 12 up, 12 in 2023-03-09T19:35:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:35:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16621.199045 secs 2023-03-09T19:35:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:35:10 smithi181 ceph-mon[94189]: pgmap v9375: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 168 KiB/s wr, 21 op/s 2023-03-09T19:35:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:35:10 smithi181 ceph-mon[94189]: osdmap e799: 12 total, 12 up, 12 in 2023-03-09T19:35:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:35:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16621.199045 secs 2023-03-09T19:35:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:35:10 smithi038 ceph-mon[92422]: pgmap v9375: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 168 KiB/s wr, 21 op/s 2023-03-09T19:35:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:35:10 smithi038 ceph-mon[92422]: osdmap e799: 12 total, 12 up, 12 in 2023-03-09T19:35:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:35:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16621.199045 secs 2023-03-09T19:35:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:35:12 smithi122 ceph-mon[93869]: pgmap v9377: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 59 KiB/s wr, 24 op/s 2023-03-09T19:35:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:35:12 smithi181 ceph-mon[94189]: pgmap v9377: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 59 KiB/s wr, 24 op/s 2023-03-09T19:35:12.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:35:12 smithi038 ceph-mon[92422]: pgmap v9377: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 59 KiB/s wr, 24 op/s 2023-03-09T19:35:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:35:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16623.504015 secs 2023-03-09T19:35:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:35:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16623.504015 secs 2023-03-09T19:35:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:35:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16623.504015 secs 2023-03-09T19:35:13.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17085.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:35:14.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:35:14 smithi122 ceph-mon[93869]: pgmap v9378: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 34 KiB/s wr, 10 op/s 2023-03-09T19:35:14.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:35:14 smithi181 ceph-mon[94189]: pgmap v9378: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 34 KiB/s wr, 10 op/s 2023-03-09T19:35:14.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:35:14 smithi038 ceph-mon[92422]: pgmap v9378: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 34 KiB/s wr, 10 op/s 2023-03-09T19:35:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:35:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16626.199105 secs 2023-03-09T19:35:15.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:35:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16626.199105 secs 2023-03-09T19:35:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:35:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16626.199105 secs 2023-03-09T19:35:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:35:16 smithi122 ceph-mon[93869]: pgmap v9379: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 33 KiB/s wr, 9 op/s 2023-03-09T19:35:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:35:16 smithi181 ceph-mon[94189]: pgmap v9379: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 33 KiB/s wr, 9 op/s 2023-03-09T19:35:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:35:16 smithi038 ceph-mon[92422]: pgmap v9379: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 33 KiB/s wr, 9 op/s 2023-03-09T19:35:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:35:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16628.504134 secs 2023-03-09T19:35:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:35:18 smithi122 ceph-mon[93869]: pgmap v9380: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 33 KiB/s wr, 8 op/s 2023-03-09T19:35:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:35:18 smithi122 ceph-mon[93869]: osdmap e800: 12 total, 12 up, 12 in 2023-03-09T19:35:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:35:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16628.504134 secs 2023-03-09T19:35:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:35:18 smithi181 ceph-mon[94189]: pgmap v9380: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 33 KiB/s wr, 8 op/s 2023-03-09T19:35:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:35:18 smithi181 ceph-mon[94189]: osdmap e800: 12 total, 12 up, 12 in 2023-03-09T19:35:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:35:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16628.504134 secs 2023-03-09T19:35:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:35:18 smithi038 ceph-mon[92422]: pgmap v9380: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 33 KiB/s wr, 8 op/s 2023-03-09T19:35:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:35:18 smithi038 ceph-mon[92422]: osdmap e800: 12 total, 12 up, 12 in 2023-03-09T19:35:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17090.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:35:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:35:20 smithi122 ceph-mon[93869]: pgmap v9382: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 105 KiB/s wr, 8 op/s 2023-03-09T19:35:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:35:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16631.199166 secs 2023-03-09T19:35:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:35:20 smithi181 ceph-mon[94189]: pgmap v9382: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 105 KiB/s wr, 8 op/s 2023-03-09T19:35:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:35:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16631.199166 secs 2023-03-09T19:35:20.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:35:20 smithi038 ceph-mon[92422]: pgmap v9382: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 105 KiB/s wr, 8 op/s 2023-03-09T19:35:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:35:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16631.199166 secs 2023-03-09T19:35:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:35:22 smithi122 ceph-mon[93869]: pgmap v9383: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 85 KiB/s wr, 8 op/s 2023-03-09T19:35:22.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:35:22 smithi181 ceph-mon[94189]: pgmap v9383: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 85 KiB/s wr, 8 op/s 2023-03-09T19:35:22.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:35:22 smithi038 ceph-mon[92422]: pgmap v9383: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 85 KiB/s wr, 8 op/s 2023-03-09T19:35:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:35:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16633.504204 secs 2023-03-09T19:35:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:35:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16633.504204 secs 2023-03-09T19:35:23.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:35:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16633.504204 secs 2023-03-09T19:35:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17095.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:35:24.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:35:24 smithi122 ceph-mon[93869]: pgmap v9384: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 84 KiB/s wr, 8 op/s 2023-03-09T19:35:24.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:35:24 smithi181 ceph-mon[94189]: pgmap v9384: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 84 KiB/s wr, 8 op/s 2023-03-09T19:35:24.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:35:24 smithi038 ceph-mon[92422]: pgmap v9384: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 84 KiB/s wr, 8 op/s 2023-03-09T19:35:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:35:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16636.199251 secs 2023-03-09T19:35:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:35:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16636.199251 secs 2023-03-09T19:35:25.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:35:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16636.199251 secs 2023-03-09T19:35:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:35:26 smithi122 ceph-mon[93869]: pgmap v9385: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 84 KiB/s wr, 8 op/s 2023-03-09T19:35:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:35:26 smithi181 ceph-mon[94189]: pgmap v9385: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 84 KiB/s wr, 8 op/s 2023-03-09T19:35:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:35:26 smithi038 ceph-mon[92422]: pgmap v9385: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 84 KiB/s wr, 8 op/s 2023-03-09T19:35:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:35:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16638.504335 secs 2023-03-09T19:35:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:35:28 smithi122 ceph-mon[93869]: pgmap v9386: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 71 KiB/s wr, 8 op/s 2023-03-09T19:35:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:35:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16638.504335 secs 2023-03-09T19:35:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:35:28 smithi181 ceph-mon[94189]: pgmap v9386: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 71 KiB/s wr, 8 op/s 2023-03-09T19:35:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:35:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16638.504335 secs 2023-03-09T19:35:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:35:28 smithi038 ceph-mon[92422]: pgmap v9386: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 71 KiB/s wr, 8 op/s 2023-03-09T19:35:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17100.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:35:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:35:30 smithi122 ceph-mon[93869]: pgmap v9387: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 126 KiB/s wr, 8 op/s 2023-03-09T19:35:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:35:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16641.199310 secs 2023-03-09T19:35:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:35:30 smithi181 ceph-mon[94189]: pgmap v9387: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 126 KiB/s wr, 8 op/s 2023-03-09T19:35:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:35:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16641.199310 secs 2023-03-09T19:35:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:35:30 smithi038 ceph-mon[92422]: pgmap v9387: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 126 KiB/s wr, 8 op/s 2023-03-09T19:35:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:35:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16641.199310 secs 2023-03-09T19:35:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:35:32 smithi122 ceph-mon[93869]: pgmap v9388: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 59 KiB/s wr, 9 op/s 2023-03-09T19:35:32.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:35:32 smithi181 ceph-mon[94189]: pgmap v9388: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 59 KiB/s wr, 9 op/s 2023-03-09T19:35:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:35:32 smithi038 ceph-mon[92422]: pgmap v9388: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 59 KiB/s wr, 9 op/s 2023-03-09T19:35:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:35:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16643.504430 secs 2023-03-09T19:35:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:35:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16643.504430 secs 2023-03-09T19:35:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:35:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16643.504430 secs 2023-03-09T19:35:33.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17105.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:35:34.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:35:34 smithi122 ceph-mon[93869]: pgmap v9389: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 113 KiB/s wr, 7 op/s 2023-03-09T19:35:34.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:35:34 smithi181 ceph-mon[94189]: pgmap v9389: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 113 KiB/s wr, 7 op/s 2023-03-09T19:35:34.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:35:34 smithi038 ceph-mon[92422]: pgmap v9389: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 113 KiB/s wr, 7 op/s 2023-03-09T19:35:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:35:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16646.199394 secs 2023-03-09T19:35:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:35:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16646.199394 secs 2023-03-09T19:35:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:35:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16646.199394 secs 2023-03-09T19:35:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:35:36 smithi122 ceph-mon[93869]: pgmap v9390: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 112 KiB/s wr, 6 op/s 2023-03-09T19:35:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:35:36 smithi181 ceph-mon[94189]: pgmap v9390: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 112 KiB/s wr, 6 op/s 2023-03-09T19:35:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:35:36 smithi038 ceph-mon[92422]: pgmap v9390: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 112 KiB/s wr, 6 op/s 2023-03-09T19:35:38.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:35:38 smithi122 ceph-mon[93869]: pgmap v9391: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 113 KiB/s wr, 8 op/s 2023-03-09T19:35:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:35:38 smithi181 ceph-mon[94189]: pgmap v9391: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 113 KiB/s wr, 8 op/s 2023-03-09T19:35:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:35:38 smithi038 ceph-mon[92422]: pgmap v9391: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 113 KiB/s wr, 8 op/s 2023-03-09T19:35:38.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17110.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:35:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:35:40 smithi122 ceph-mon[93869]: pgmap v9392: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 112 KiB/s wr, 6 op/s 2023-03-09T19:35:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:35:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16651.199456 secs 2023-03-09T19:35:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:35:40 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:35:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:35:40 smithi181 ceph-mon[94189]: pgmap v9392: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 112 KiB/s wr, 6 op/s 2023-03-09T19:35:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:35:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16651.199456 secs 2023-03-09T19:35:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:35:40 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:35:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:35:40 smithi038 ceph-mon[92422]: pgmap v9392: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 112 KiB/s wr, 6 op/s 2023-03-09T19:35:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:35:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16651.199456 secs 2023-03-09T19:35:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:35:40 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:35:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:35:41 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:35:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:35:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:35:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:35:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:35:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:35:41 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:35:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:35:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:35:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:35:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:35:41.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:35:41 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:35:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:35:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:35:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:35:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:35:42.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:35:42 smithi038 ceph-mon[92422]: pgmap v9393: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 55 KiB/s wr, 8 op/s 2023-03-09T19:35:42.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:35:42 smithi122 ceph-mon[93869]: pgmap v9393: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 55 KiB/s wr, 8 op/s 2023-03-09T19:35:42.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:35:42 smithi181 ceph-mon[94189]: pgmap v9393: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 55 KiB/s wr, 8 op/s 2023-03-09T19:35:43.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17115.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:35:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:35:44 smithi122 ceph-mon[93869]: pgmap v9394: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 124 KiB/s wr, 6 op/s 2023-03-09T19:35:44.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:35:44 smithi181 ceph-mon[94189]: pgmap v9394: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 124 KiB/s wr, 6 op/s 2023-03-09T19:35:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:35:44 smithi038 ceph-mon[92422]: pgmap v9394: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 124 KiB/s wr, 6 op/s 2023-03-09T19:35:46.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:35:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16656.199516 secs 2023-03-09T19:35:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:35:46 smithi038 ceph-mon[92422]: pgmap v9395: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 70 KiB/s wr, 6 op/s 2023-03-09T19:35:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:35:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16656.199516 secs 2023-03-09T19:35:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:35:46 smithi122 ceph-mon[93869]: pgmap v9395: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 70 KiB/s wr, 6 op/s 2023-03-09T19:35:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:35:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16656.199516 secs 2023-03-09T19:35:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:35:46 smithi181 ceph-mon[94189]: pgmap v9395: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 70 KiB/s wr, 6 op/s 2023-03-09T19:35:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:35:48 smithi038 ceph-mon[92422]: pgmap v9396: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 85 KiB/s wr, 8 op/s 2023-03-09T19:35:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:35:48 smithi122 ceph-mon[93869]: pgmap v9396: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 85 KiB/s wr, 8 op/s 2023-03-09T19:35:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:35:48 smithi181 ceph-mon[94189]: pgmap v9396: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 85 KiB/s wr, 8 op/s 2023-03-09T19:35:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17120.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:35:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:35:50 smithi038 ceph-mon[92422]: pgmap v9397: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 238 KiB/s wr, 8 op/s 2023-03-09T19:35:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:35:50 smithi122 ceph-mon[93869]: pgmap v9397: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 238 KiB/s wr, 8 op/s 2023-03-09T19:35:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:35:50 smithi181 ceph-mon[94189]: pgmap v9397: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 238 KiB/s wr, 8 op/s 2023-03-09T19:35:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:35:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16661.199575 secs 2023-03-09T19:35:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:35:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16661.199575 secs 2023-03-09T19:35:51.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:35:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16661.199575 secs 2023-03-09T19:35:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:35:52 smithi122 ceph-mon[93869]: pgmap v9398: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 238 KiB/s wr, 45 op/s 2023-03-09T19:35:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:35:52 smithi181 ceph-mon[94189]: pgmap v9398: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 238 KiB/s wr, 45 op/s 2023-03-09T19:35:52.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:35:52 smithi038 ceph-mon[92422]: pgmap v9398: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 238 KiB/s wr, 45 op/s 2023-03-09T19:35:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17125.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:35:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:35:54 smithi122 ceph-mon[93869]: pgmap v9399: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 290 KiB/s wr, 45 op/s 2023-03-09T19:35:54.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:35:54 smithi181 ceph-mon[94189]: pgmap v9399: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 290 KiB/s wr, 45 op/s 2023-03-09T19:35:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:35:54 smithi038 ceph-mon[92422]: pgmap v9399: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 290 KiB/s wr, 45 op/s 2023-03-09T19:35:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:35:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16666.199633 secs 2023-03-09T19:35:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:35:56 smithi038 ceph-mon[92422]: pgmap v9400: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 221 KiB/s wr, 44 op/s 2023-03-09T19:35:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:35:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16666.199633 secs 2023-03-09T19:35:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:35:56 smithi122 ceph-mon[93869]: pgmap v9400: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 221 KiB/s wr, 44 op/s 2023-03-09T19:35:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:35:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16666.199633 secs 2023-03-09T19:35:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:35:56 smithi181 ceph-mon[94189]: pgmap v9400: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 221 KiB/s wr, 44 op/s 2023-03-09T19:35:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:35:58 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 16668.504976 secs 2023-03-09T19:35:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:35:58 smithi038 ceph-mon[92422]: pgmap v9401: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 603 KiB/s rd, 496 KiB/s wr, 48 op/s 2023-03-09T19:35:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:35:58 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 16668.504976 secs 2023-03-09T19:35:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:35:58 smithi122 ceph-mon[93869]: pgmap v9401: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 603 KiB/s rd, 496 KiB/s wr, 48 op/s 2023-03-09T19:35:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:35:58 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 16668.504976 secs 2023-03-09T19:35:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:35:58 smithi181 ceph-mon[94189]: pgmap v9401: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 603 KiB/s rd, 496 KiB/s wr, 48 op/s 2023-03-09T19:35:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17130.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:36:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:36:00 smithi038 ceph-mon[92422]: pgmap v9402: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 585 KiB/s rd, 481 KiB/s wr, 46 op/s 2023-03-09T19:36:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:36:00 smithi122 ceph-mon[93869]: pgmap v9402: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 585 KiB/s rd, 481 KiB/s wr, 46 op/s 2023-03-09T19:36:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:36:00 smithi181 ceph-mon[94189]: pgmap v9402: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 585 KiB/s rd, 481 KiB/s wr, 46 op/s 2023-03-09T19:36:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:36:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16671.199692 secs 2023-03-09T19:36:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:36:01 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:36:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:36:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:36:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:36:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:36:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:36:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16671.199692 secs 2023-03-09T19:36:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:36:01 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:36:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:36:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:36:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:36:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:36:01.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:36:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16671.199692 secs 2023-03-09T19:36:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:36:01 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:36:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:36:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:36:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:36:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:36:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:36:02 smithi122 ceph-mon[93869]: pgmap v9403: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 560 KiB/s rd, 403 KiB/s wr, 49 op/s 2023-03-09T19:36:02.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:36:02 smithi181 ceph-mon[94189]: pgmap v9403: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 560 KiB/s rd, 403 KiB/s wr, 49 op/s 2023-03-09T19:36:02.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:36:02 smithi038 ceph-mon[92422]: pgmap v9403: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 560 KiB/s rd, 403 KiB/s wr, 49 op/s 2023-03-09T19:36:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:36:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16673.505090 secs 2023-03-09T19:36:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:36:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16673.505090 secs 2023-03-09T19:36:03.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:36:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16673.505090 secs 2023-03-09T19:36:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17135.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:36:04.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:36:04 smithi122 ceph-mon[93869]: pgmap v9404: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 520 KiB/s rd, 424 KiB/s wr, 24 op/s 2023-03-09T19:36:04.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:36:04 smithi181 ceph-mon[94189]: pgmap v9404: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 520 KiB/s rd, 424 KiB/s wr, 24 op/s 2023-03-09T19:36:04.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:36:04 smithi038 ceph-mon[92422]: pgmap v9404: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 520 KiB/s rd, 424 KiB/s wr, 24 op/s 2023-03-09T19:36:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:36:05 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:36:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:36:05 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:36:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:36:05 smithi122 ceph-mon[93869]: osdmap e801: 12 total, 12 up, 12 in 2023-03-09T19:36:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:36:05 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:36:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:36:05 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:36:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:36:05 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:36:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:36:05 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:36:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:36:05 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:36:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:36:05 smithi181 ceph-mon[94189]: osdmap e801: 12 total, 12 up, 12 in 2023-03-09T19:36:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:36:05 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:36:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:36:05 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:36:05.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:36:05 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:36:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:36:05 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:36:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:36:05 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:36:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:36:05 smithi038 ceph-mon[92422]: osdmap e801: 12 total, 12 up, 12 in 2023-03-09T19:36:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:36:05 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:36:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:36:05 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:36:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:36:05 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:36:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:36:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16676.199756 secs 2023-03-09T19:36:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:36:06 smithi122 ceph-mon[93869]: pgmap v9406: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 446 KiB/s wr, 26 op/s 2023-03-09T19:36:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:36:06 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:36:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:36:06 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:36:06.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:36:06 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:36:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:36:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16676.199756 secs 2023-03-09T19:36:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:36:06 smithi181 ceph-mon[94189]: pgmap v9406: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 446 KiB/s wr, 26 op/s 2023-03-09T19:36:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:36:06 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:36:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:36:06 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:36:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:36:06 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:36:06.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:36:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16676.199756 secs 2023-03-09T19:36:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:36:06 smithi038 ceph-mon[92422]: pgmap v9406: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 446 KiB/s wr, 26 op/s 2023-03-09T19:36:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:36:06 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:36:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:36:06 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:36:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:36:06 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:36:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:36:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16678.505209 secs 2023-03-09T19:36:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:36:08 smithi122 ceph-mon[93869]: pgmap v9407: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 302 KiB/s wr, 29 op/s 2023-03-09T19:36:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:36:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16678.505209 secs 2023-03-09T19:36:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:36:08 smithi181 ceph-mon[94189]: pgmap v9407: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 302 KiB/s wr, 29 op/s 2023-03-09T19:36:08.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:36:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16678.505209 secs 2023-03-09T19:36:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:36:08 smithi038 ceph-mon[92422]: pgmap v9407: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 302 KiB/s wr, 29 op/s 2023-03-09T19:36:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17140.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:36:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:36:10 smithi122 ceph-mon[93869]: pgmap v9408: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 302 KiB/s wr, 29 op/s 2023-03-09T19:36:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:36:10 smithi181 ceph-mon[94189]: pgmap v9408: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 302 KiB/s wr, 29 op/s 2023-03-09T19:36:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:36:10 smithi038 ceph-mon[92422]: pgmap v9408: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 302 KiB/s wr, 29 op/s 2023-03-09T19:36:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:36:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16681.199815 secs 2023-03-09T19:36:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:36:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16681.199815 secs 2023-03-09T19:36:11.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:36:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16681.199815 secs 2023-03-09T19:36:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:36:12 smithi122 ceph-mon[93869]: pgmap v9409: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 276 KiB/s wr, 28 op/s 2023-03-09T19:36:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:36:12 smithi181 ceph-mon[94189]: pgmap v9409: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 276 KiB/s wr, 28 op/s 2023-03-09T19:36:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:36:12 smithi038 ceph-mon[92422]: pgmap v9409: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 276 KiB/s wr, 28 op/s 2023-03-09T19:36:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:36:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16683.505319 secs 2023-03-09T19:36:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:36:13 smithi122 ceph-mon[93869]: osdmap e802: 12 total, 12 up, 12 in 2023-03-09T19:36:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:36:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16683.505319 secs 2023-03-09T19:36:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:36:13 smithi181 ceph-mon[94189]: osdmap e802: 12 total, 12 up, 12 in 2023-03-09T19:36:13.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:36:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16683.505319 secs 2023-03-09T19:36:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:36:13 smithi038 ceph-mon[92422]: osdmap e802: 12 total, 12 up, 12 in 2023-03-09T19:36:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17145.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:36:14.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:36:14 smithi122 ceph-mon[93869]: pgmap v9411: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 281 KiB/s wr, 15 op/s 2023-03-09T19:36:14.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:36:14 smithi181 ceph-mon[94189]: pgmap v9411: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 281 KiB/s wr, 15 op/s 2023-03-09T19:36:14.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:36:14 smithi038 ceph-mon[92422]: pgmap v9411: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 281 KiB/s wr, 15 op/s 2023-03-09T19:36:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:36:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16686.199893 secs 2023-03-09T19:36:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:36:16 smithi122 ceph-mon[93869]: pgmap v9412: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 250 KiB/s wr, 13 op/s 2023-03-09T19:36:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:36:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16686.199893 secs 2023-03-09T19:36:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:36:16 smithi181 ceph-mon[94189]: pgmap v9412: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 250 KiB/s wr, 13 op/s 2023-03-09T19:36:16.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:36:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16686.199893 secs 2023-03-09T19:36:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:36:16 smithi038 ceph-mon[92422]: pgmap v9412: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 250 KiB/s wr, 13 op/s 2023-03-09T19:36:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:36:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16688.505453 secs 2023-03-09T19:36:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:36:18 smithi122 ceph-mon[93869]: pgmap v9413: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 95 KiB/s wr, 9 op/s 2023-03-09T19:36:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:36:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16688.505453 secs 2023-03-09T19:36:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:36:18 smithi181 ceph-mon[94189]: pgmap v9413: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 95 KiB/s wr, 9 op/s 2023-03-09T19:36:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:36:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16688.505453 secs 2023-03-09T19:36:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:36:18 smithi038 ceph-mon[92422]: pgmap v9413: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 95 KiB/s wr, 9 op/s 2023-03-09T19:36:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17150.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:36:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:36:20 smithi122 ceph-mon[93869]: pgmap v9414: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 95 KiB/s wr, 8 op/s 2023-03-09T19:36:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:36:20 smithi181 ceph-mon[94189]: pgmap v9414: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 95 KiB/s wr, 8 op/s 2023-03-09T19:36:20.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:36:20 smithi038 ceph-mon[92422]: pgmap v9414: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 95 KiB/s wr, 8 op/s 2023-03-09T19:36:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:36:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16691.199967 secs 2023-03-09T19:36:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:36:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16691.199967 secs 2023-03-09T19:36:21.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:36:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16691.199967 secs 2023-03-09T19:36:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:36:22 smithi122 ceph-mon[93869]: pgmap v9415: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 32 KiB/s wr, 8 op/s 2023-03-09T19:36:22.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:36:22 smithi181 ceph-mon[94189]: pgmap v9415: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 32 KiB/s wr, 8 op/s 2023-03-09T19:36:22.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:36:22 smithi038 ceph-mon[92422]: pgmap v9415: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 32 KiB/s wr, 8 op/s 2023-03-09T19:36:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:36:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16693.505550 secs 2023-03-09T19:36:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:36:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16693.505550 secs 2023-03-09T19:36:23.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:36:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16693.505550 secs 2023-03-09T19:36:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17155.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:36:24.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:36:24 smithi122 ceph-mon[93869]: pgmap v9416: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 31 KiB/s wr, 8 op/s 2023-03-09T19:36:24.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:36:24 smithi181 ceph-mon[94189]: pgmap v9416: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 31 KiB/s wr, 8 op/s 2023-03-09T19:36:24.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:36:24 smithi038 ceph-mon[92422]: pgmap v9416: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 31 KiB/s wr, 8 op/s 2023-03-09T19:36:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:36:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16696.200026 secs 2023-03-09T19:36:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:36:26 smithi122 ceph-mon[93869]: pgmap v9417: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 26 KiB/s wr, 6 op/s 2023-03-09T19:36:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:36:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16696.200026 secs 2023-03-09T19:36:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:36:26 smithi181 ceph-mon[94189]: pgmap v9417: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 26 KiB/s wr, 6 op/s 2023-03-09T19:36:26.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:36:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16696.200026 secs 2023-03-09T19:36:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:36:26 smithi038 ceph-mon[92422]: pgmap v9417: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 26 KiB/s wr, 6 op/s 2023-03-09T19:36:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:36:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16698.505670 secs 2023-03-09T19:36:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:36:28 smithi122 ceph-mon[93869]: pgmap v9418: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 85 KiB/s wr, 10 op/s 2023-03-09T19:36:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:36:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16698.505670 secs 2023-03-09T19:36:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:36:28 smithi181 ceph-mon[94189]: pgmap v9418: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 85 KiB/s wr, 10 op/s 2023-03-09T19:36:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:36:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16698.505670 secs 2023-03-09T19:36:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:36:28 smithi038 ceph-mon[92422]: pgmap v9418: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 85 KiB/s wr, 10 op/s 2023-03-09T19:36:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17160.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:36:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:36:30 smithi122 ceph-mon[93869]: pgmap v9419: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 6 op/s 2023-03-09T19:36:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:36:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16701.200110 secs 2023-03-09T19:36:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:36:30 smithi181 ceph-mon[94189]: pgmap v9419: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 6 op/s 2023-03-09T19:36:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:36:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16701.200110 secs 2023-03-09T19:36:30.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:36:30 smithi038 ceph-mon[92422]: pgmap v9419: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 6 op/s 2023-03-09T19:36:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:36:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16701.200110 secs 2023-03-09T19:36:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:36:32 smithi122 ceph-mon[93869]: pgmap v9420: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 59 KiB/s wr, 9 op/s 2023-03-09T19:36:32.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:36:32 smithi181 ceph-mon[94189]: pgmap v9420: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 59 KiB/s wr, 9 op/s 2023-03-09T19:36:32.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:36:32 smithi038 ceph-mon[92422]: pgmap v9420: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 59 KiB/s wr, 9 op/s 2023-03-09T19:36:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:36:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16703.505781 secs 2023-03-09T19:36:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:36:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16703.505781 secs 2023-03-09T19:36:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:36:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16703.505781 secs 2023-03-09T19:36:33.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17165.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:36:34.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:36:34 smithi122 ceph-mon[93869]: pgmap v9421: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 6 op/s 2023-03-09T19:36:34.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:36:34 smithi181 ceph-mon[94189]: pgmap v9421: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 6 op/s 2023-03-09T19:36:34.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:36:34 smithi038 ceph-mon[92422]: pgmap v9421: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 6 op/s 2023-03-09T19:36:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:36:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16706.200171 secs 2023-03-09T19:36:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:36:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16706.200171 secs 2023-03-09T19:36:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:36:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16706.200171 secs 2023-03-09T19:36:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:36:36 smithi122 ceph-mon[93869]: pgmap v9422: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 6 op/s 2023-03-09T19:36:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:36:36 smithi181 ceph-mon[94189]: pgmap v9422: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 6 op/s 2023-03-09T19:36:36.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:36:36 smithi038 ceph-mon[92422]: pgmap v9422: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 6 op/s 2023-03-09T19:36:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:36:38 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 16708.505893 secs 2023-03-09T19:36:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:36:38 smithi122 ceph-mon[93869]: pgmap v9423: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 117 KiB/s wr, 9 op/s 2023-03-09T19:36:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:36:38 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 16708.505893 secs 2023-03-09T19:36:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:36:38 smithi181 ceph-mon[94189]: pgmap v9423: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 117 KiB/s wr, 9 op/s 2023-03-09T19:36:38.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:36:38 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 16708.505893 secs 2023-03-09T19:36:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:36:38 smithi038 ceph-mon[92422]: pgmap v9423: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 117 KiB/s wr, 9 op/s 2023-03-09T19:36:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17170.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:36:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:36:40 smithi122 ceph-mon[93869]: pgmap v9424: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 6 op/s 2023-03-09T19:36:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:36:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16711.200243 secs 2023-03-09T19:36:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:36:40 smithi181 ceph-mon[94189]: pgmap v9424: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 6 op/s 2023-03-09T19:36:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:36:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16711.200243 secs 2023-03-09T19:36:40.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:36:40 smithi038 ceph-mon[92422]: pgmap v9424: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 6 op/s 2023-03-09T19:36:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:36:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16711.200243 secs 2023-03-09T19:36:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:36:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:36:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:36:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:36:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:36:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:36:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:36:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:36:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:36:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:36:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:36:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:36:42.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:36:42 smithi122 ceph-mon[93869]: pgmap v9425: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 131 KiB/s wr, 8 op/s 2023-03-09T19:36:42.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:36:42 smithi181 ceph-mon[94189]: pgmap v9425: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 131 KiB/s wr, 8 op/s 2023-03-09T19:36:42.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:36:42 smithi038 ceph-mon[92422]: pgmap v9425: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 131 KiB/s wr, 8 op/s 2023-03-09T19:36:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:36:43 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 16713.506003 secs 2023-03-09T19:36:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:36:43 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 16713.506003 secs 2023-03-09T19:36:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:36:43 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 16713.506003 secs 2023-03-09T19:36:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17175.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:36:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:36:44 smithi122 ceph-mon[93869]: pgmap v9426: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 130 KiB/s wr, 6 op/s 2023-03-09T19:36:44.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:36:44 smithi181 ceph-mon[94189]: pgmap v9426: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 130 KiB/s wr, 6 op/s 2023-03-09T19:36:44.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:36:44 smithi038 ceph-mon[92422]: pgmap v9426: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 130 KiB/s wr, 6 op/s 2023-03-09T19:36:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:36:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16716.200316 secs 2023-03-09T19:36:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:36:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16716.200316 secs 2023-03-09T19:36:45.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:36:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16716.200316 secs 2023-03-09T19:36:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:36:46 smithi122 ceph-mon[93869]: pgmap v9427: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 130 KiB/s wr, 6 op/s 2023-03-09T19:36:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:36:46 smithi181 ceph-mon[94189]: pgmap v9427: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 130 KiB/s wr, 6 op/s 2023-03-09T19:36:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:36:46 smithi038 ceph-mon[92422]: pgmap v9427: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 130 KiB/s wr, 6 op/s 2023-03-09T19:36:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:36:48 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 16718.506138 secs 2023-03-09T19:36:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:36:48 smithi122 ceph-mon[93869]: pgmap v9428: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 242 KiB/s wr, 9 op/s 2023-03-09T19:36:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:36:48 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 16718.506138 secs 2023-03-09T19:36:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:36:48 smithi181 ceph-mon[94189]: pgmap v9428: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 242 KiB/s wr, 9 op/s 2023-03-09T19:36:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:36:48 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 16718.506138 secs 2023-03-09T19:36:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:36:48 smithi038 ceph-mon[92422]: pgmap v9428: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 242 KiB/s wr, 9 op/s 2023-03-09T19:36:48.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17180.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:36:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:36:50 smithi122 ceph-mon[93869]: pgmap v9429: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 184 KiB/s wr, 7 op/s 2023-03-09T19:36:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:36:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16721.200377 secs 2023-03-09T19:36:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:36:50 smithi181 ceph-mon[94189]: pgmap v9429: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 184 KiB/s wr, 7 op/s 2023-03-09T19:36:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:36:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16721.200377 secs 2023-03-09T19:36:50.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:36:50 smithi038 ceph-mon[92422]: pgmap v9429: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 184 KiB/s wr, 7 op/s 2023-03-09T19:36:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:36:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16721.200377 secs 2023-03-09T19:36:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:36:52 smithi122 ceph-mon[93869]: pgmap v9430: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 221 KiB/s wr, 11 op/s 2023-03-09T19:36:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:36:52 smithi181 ceph-mon[94189]: pgmap v9430: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 221 KiB/s wr, 11 op/s 2023-03-09T19:36:52.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:36:52 smithi038 ceph-mon[92422]: pgmap v9430: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 221 KiB/s wr, 11 op/s 2023-03-09T19:36:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:36:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16723.506254 secs 2023-03-09T19:36:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:36:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16723.506254 secs 2023-03-09T19:36:53.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:36:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16723.506254 secs 2023-03-09T19:36:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17185.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:36:54.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:36:54 smithi122 ceph-mon[93869]: pgmap v9431: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 149 KiB/s wr, 9 op/s 2023-03-09T19:36:54.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:36:54 smithi181 ceph-mon[94189]: pgmap v9431: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 149 KiB/s wr, 9 op/s 2023-03-09T19:36:54.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:36:54 smithi038 ceph-mon[92422]: pgmap v9431: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 149 KiB/s wr, 9 op/s 2023-03-09T19:36:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:36:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16726.200441 secs 2023-03-09T19:36:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:36:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16726.200441 secs 2023-03-09T19:36:55.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:36:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16726.200441 secs 2023-03-09T19:36:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:36:56 smithi181 ceph-mon[94189]: pgmap v9432: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 149 KiB/s wr, 9 op/s 2023-03-09T19:36:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:36:56 smithi038 ceph-mon[92422]: pgmap v9432: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 149 KiB/s wr, 9 op/s 2023-03-09T19:36:56.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:36:56 smithi122 ceph-mon[93869]: pgmap v9432: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 149 KiB/s wr, 9 op/s 2023-03-09T19:36:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:36:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:36:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:36:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:36:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:36:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:36:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:36:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:36:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:36:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:36:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:36:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:36:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17190.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:36:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:36:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16728.506392 secs 2023-03-09T19:36:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:36:58 smithi122 ceph-mon[93869]: pgmap v9433: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 383 KiB/s wr, 14 op/s 2023-03-09T19:36:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:36:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16728.506392 secs 2023-03-09T19:36:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:36:58 smithi181 ceph-mon[94189]: pgmap v9433: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 383 KiB/s wr, 14 op/s 2023-03-09T19:36:58.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:36:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16728.506392 secs 2023-03-09T19:36:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:36:58 smithi038 ceph-mon[92422]: pgmap v9433: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 383 KiB/s wr, 14 op/s 2023-03-09T19:37:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:37:00 smithi122 ceph-mon[93869]: pgmap v9434: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 271 KiB/s wr, 10 op/s 2023-03-09T19:37:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:37:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16731.200502 secs 2023-03-09T19:37:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:37:00 smithi181 ceph-mon[94189]: pgmap v9434: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 271 KiB/s wr, 10 op/s 2023-03-09T19:37:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:37:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16731.200502 secs 2023-03-09T19:37:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:37:00 smithi038 ceph-mon[92422]: pgmap v9434: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 271 KiB/s wr, 10 op/s 2023-03-09T19:37:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:37:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16731.200502 secs 2023-03-09T19:37:02.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:37:02 smithi122 ceph-mon[93869]: pgmap v9435: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 314 KiB/s wr, 14 op/s 2023-03-09T19:37:02.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:37:02 smithi181 ceph-mon[94189]: pgmap v9435: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 314 KiB/s wr, 14 op/s 2023-03-09T19:37:02.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:37:02 smithi038 ceph-mon[92422]: pgmap v9435: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 314 KiB/s wr, 14 op/s 2023-03-09T19:37:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17195.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:37:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:37:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16733.506506 secs 2023-03-09T19:37:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:37:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16733.506506 secs 2023-03-09T19:37:03.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:37:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16733.506506 secs 2023-03-09T19:37:04.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:37:04 smithi122 ceph-mon[93869]: pgmap v9436: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 299 KiB/s wr, 22 op/s 2023-03-09T19:37:04.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:37:04 smithi181 ceph-mon[94189]: pgmap v9436: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 299 KiB/s wr, 22 op/s 2023-03-09T19:37:04.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:37:04 smithi038 ceph-mon[92422]: pgmap v9436: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 299 KiB/s wr, 22 op/s 2023-03-09T19:37:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:37:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16736.200564 secs 2023-03-09T19:37:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:37:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16736.200564 secs 2023-03-09T19:37:05.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:37:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16736.200564 secs 2023-03-09T19:37:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:37:06 smithi122 ceph-mon[93869]: pgmap v9437: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 516 KiB/s wr, 23 op/s 2023-03-09T19:37:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:37:06 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:37:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:37:06 smithi181 ceph-mon[94189]: pgmap v9437: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 516 KiB/s wr, 23 op/s 2023-03-09T19:37:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:37:06 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:37:06.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:37:06 smithi038 ceph-mon[92422]: pgmap v9437: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 516 KiB/s wr, 23 op/s 2023-03-09T19:37:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:37:06 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:37:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:37:07 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:37:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:37:07 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:37:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:37:07 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:37:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:37:07 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:37:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:37:07 smithi122 ceph-mon[93869]: pgmap v9438: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 464 KiB/s rd, 553 KiB/s wr, 29 op/s 2023-03-09T19:37:07.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:37:07 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:37:07.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:37:07 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:37:07.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:37:07 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:37:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:37:07 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:37:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:37:07 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:37:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:37:07 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:37:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:37:07 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:37:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:37:07 smithi181 ceph-mon[94189]: pgmap v9438: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 464 KiB/s rd, 553 KiB/s wr, 29 op/s 2023-03-09T19:37:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:37:07 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:37:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:37:07 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:37:07.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:37:07 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:37:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:37:07 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:37:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:37:07 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:37:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:37:07 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:37:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:37:07 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:37:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:37:07 smithi038 ceph-mon[92422]: pgmap v9438: 129 pgs: 129 active+clean; 9.4 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 464 KiB/s rd, 553 KiB/s wr, 29 op/s 2023-03-09T19:37:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:37:07 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:37:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:37:07 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:37:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:37:07 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:37:08.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17200.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:37:08.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:37:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16738.506577 secs 2023-03-09T19:37:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:37:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16738.506577 secs 2023-03-09T19:37:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:37:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16738.506577 secs 2023-03-09T19:37:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:37:09 smithi122 ceph-mon[93869]: pgmap v9439: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 415 KiB/s wr, 25 op/s 2023-03-09T19:37:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:37:09 smithi181 ceph-mon[94189]: pgmap v9439: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 415 KiB/s wr, 25 op/s 2023-03-09T19:37:09.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:37:09 smithi038 ceph-mon[92422]: pgmap v9439: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 415 KiB/s wr, 25 op/s 2023-03-09T19:37:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:37:10 smithi122 ceph-mon[93869]: osdmap e803: 12 total, 12 up, 12 in 2023-03-09T19:37:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:37:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16741.200629 secs 2023-03-09T19:37:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:37:10 smithi181 ceph-mon[94189]: osdmap e803: 12 total, 12 up, 12 in 2023-03-09T19:37:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:37:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16741.200629 secs 2023-03-09T19:37:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:37:10 smithi038 ceph-mon[92422]: osdmap e803: 12 total, 12 up, 12 in 2023-03-09T19:37:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:37:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16741.200629 secs 2023-03-09T19:37:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:37:11 smithi122 ceph-mon[93869]: pgmap v9441: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 395 KiB/s rd, 464 KiB/s wr, 29 op/s 2023-03-09T19:37:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:37:11 smithi181 ceph-mon[94189]: pgmap v9441: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 395 KiB/s rd, 464 KiB/s wr, 29 op/s 2023-03-09T19:37:11.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:37:11 smithi038 ceph-mon[92422]: pgmap v9441: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 395 KiB/s rd, 464 KiB/s wr, 29 op/s 2023-03-09T19:37:12.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:37:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16743.506696 secs 2023-03-09T19:37:12.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:37:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16743.506696 secs 2023-03-09T19:37:12.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:37:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16743.506696 secs 2023-03-09T19:37:13.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17205.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:37:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:37:14 smithi122 ceph-mon[93869]: pgmap v9442: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 440 KiB/s wr, 15 op/s 2023-03-09T19:37:14.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:37:14 smithi181 ceph-mon[94189]: pgmap v9442: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 440 KiB/s wr, 15 op/s 2023-03-09T19:37:14.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:37:14 smithi038 ceph-mon[92422]: pgmap v9442: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 440 KiB/s wr, 15 op/s 2023-03-09T19:37:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:37:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16746.200688 secs 2023-03-09T19:37:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:37:16 smithi122 ceph-mon[93869]: pgmap v9443: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 179 KiB/s wr, 14 op/s 2023-03-09T19:37:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:37:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16746.200688 secs 2023-03-09T19:37:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:37:16 smithi181 ceph-mon[94189]: pgmap v9443: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 179 KiB/s wr, 14 op/s 2023-03-09T19:37:16.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:37:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16746.200688 secs 2023-03-09T19:37:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:37:16 smithi038 ceph-mon[92422]: pgmap v9443: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 179 KiB/s wr, 14 op/s 2023-03-09T19:37:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:37:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16748.506825 secs 2023-03-09T19:37:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:37:18 smithi122 ceph-mon[93869]: pgmap v9444: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 136 KiB/s wr, 10 op/s 2023-03-09T19:37:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:37:18 smithi122 ceph-mon[93869]: osdmap e804: 12 total, 12 up, 12 in 2023-03-09T19:37:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:37:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16748.506825 secs 2023-03-09T19:37:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:37:18 smithi181 ceph-mon[94189]: pgmap v9444: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 136 KiB/s wr, 10 op/s 2023-03-09T19:37:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:37:18 smithi181 ceph-mon[94189]: osdmap e804: 12 total, 12 up, 12 in 2023-03-09T19:37:18.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:37:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16748.506825 secs 2023-03-09T19:37:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:37:18 smithi038 ceph-mon[92422]: pgmap v9444: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 136 KiB/s wr, 10 op/s 2023-03-09T19:37:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:37:18 smithi038 ceph-mon[92422]: osdmap e804: 12 total, 12 up, 12 in 2023-03-09T19:37:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17210.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:37:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:37:20 smithi122 ceph-mon[93869]: pgmap v9446: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 21 KiB/s wr, 10 op/s 2023-03-09T19:37:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:37:20 smithi181 ceph-mon[94189]: pgmap v9446: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 21 KiB/s wr, 10 op/s 2023-03-09T19:37:20.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:37:20 smithi038 ceph-mon[92422]: pgmap v9446: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 21 KiB/s wr, 10 op/s 2023-03-09T19:37:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:37:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16751.200749 secs 2023-03-09T19:37:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:37:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16751.200749 secs 2023-03-09T19:37:21.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:37:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16751.200749 secs 2023-03-09T19:37:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:37:22 smithi122 ceph-mon[93869]: pgmap v9447: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 2.8 KiB/s wr, 8 op/s 2023-03-09T19:37:22.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:37:22 smithi181 ceph-mon[94189]: pgmap v9447: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 2.8 KiB/s wr, 8 op/s 2023-03-09T19:37:22.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:37:22 smithi038 ceph-mon[92422]: pgmap v9447: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 2.8 KiB/s wr, 8 op/s 2023-03-09T19:37:23.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:37:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16753.506947 secs 2023-03-09T19:37:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:37:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16753.506947 secs 2023-03-09T19:37:23.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:37:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16753.506947 secs 2023-03-09T19:37:23.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17215.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:37:24.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:37:24 smithi122 ceph-mon[93869]: pgmap v9448: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 1.9 KiB/s wr, 8 op/s 2023-03-09T19:37:24.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:37:24 smithi181 ceph-mon[94189]: pgmap v9448: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 1.9 KiB/s wr, 8 op/s 2023-03-09T19:37:24.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:37:24 smithi038 ceph-mon[92422]: pgmap v9448: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 1.9 KiB/s wr, 8 op/s 2023-03-09T19:37:26.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:37:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16756.200811 secs 2023-03-09T19:37:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:37:26 smithi122 ceph-mon[93869]: pgmap v9449: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.9 KiB/s wr, 7 op/s 2023-03-09T19:37:26.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:37:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16756.200811 secs 2023-03-09T19:37:26.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:37:26 smithi181 ceph-mon[94189]: pgmap v9449: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.9 KiB/s wr, 7 op/s 2023-03-09T19:37:26.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:37:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16756.200811 secs 2023-03-09T19:37:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:37:26 smithi038 ceph-mon[92422]: pgmap v9449: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.9 KiB/s wr, 7 op/s 2023-03-09T19:37:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:37:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16758.507066 secs 2023-03-09T19:37:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:37:28 smithi122 ceph-mon[93869]: pgmap v9450: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.3 KiB/s wr, 7 op/s 2023-03-09T19:37:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:37:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16758.507066 secs 2023-03-09T19:37:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:37:28 smithi181 ceph-mon[94189]: pgmap v9450: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.3 KiB/s wr, 7 op/s 2023-03-09T19:37:28.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:37:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16758.507066 secs 2023-03-09T19:37:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:37:28 smithi038 ceph-mon[92422]: pgmap v9450: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.3 KiB/s wr, 7 op/s 2023-03-09T19:37:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17220.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:37:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:37:30 smithi122 ceph-mon[93869]: pgmap v9451: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 64 KiB/s wr, 7 op/s 2023-03-09T19:37:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:37:30 smithi181 ceph-mon[94189]: pgmap v9451: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 64 KiB/s wr, 7 op/s 2023-03-09T19:37:30.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:37:30 smithi038 ceph-mon[92422]: pgmap v9451: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 64 KiB/s wr, 7 op/s 2023-03-09T19:37:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:37:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16761.200872 secs 2023-03-09T19:37:31.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:37:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16761.200872 secs 2023-03-09T19:37:31.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:37:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16761.200872 secs 2023-03-09T19:37:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:37:32 smithi122 ceph-mon[93869]: pgmap v9452: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 60 KiB/s wr, 8 op/s 2023-03-09T19:37:32.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:37:32 smithi181 ceph-mon[94189]: pgmap v9452: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 60 KiB/s wr, 8 op/s 2023-03-09T19:37:32.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:37:32 smithi038 ceph-mon[92422]: pgmap v9452: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 60 KiB/s wr, 8 op/s 2023-03-09T19:37:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:37:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16763.507204 secs 2023-03-09T19:37:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:37:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16763.507204 secs 2023-03-09T19:37:33.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:37:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16763.507204 secs 2023-03-09T19:37:33.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17225.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:37:34.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:37:34 smithi122 ceph-mon[93869]: pgmap v9453: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 60 KiB/s wr, 7 op/s 2023-03-09T19:37:34.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:37:34 smithi181 ceph-mon[94189]: pgmap v9453: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 60 KiB/s wr, 7 op/s 2023-03-09T19:37:34.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:37:34 smithi038 ceph-mon[92422]: pgmap v9453: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 60 KiB/s wr, 7 op/s 2023-03-09T19:37:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:37:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16766.200956 secs 2023-03-09T19:37:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:37:36 smithi122 ceph-mon[93869]: pgmap v9454: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 7 op/s 2023-03-09T19:37:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:37:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16766.200956 secs 2023-03-09T19:37:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:37:36 smithi181 ceph-mon[94189]: pgmap v9454: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 7 op/s 2023-03-09T19:37:36.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:37:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16766.200956 secs 2023-03-09T19:37:36.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:37:36 smithi038 ceph-mon[92422]: pgmap v9454: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 7 op/s 2023-03-09T19:37:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:37:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16768.507269 secs 2023-03-09T19:37:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:37:38 smithi122 ceph-mon[93869]: pgmap v9455: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 60 KiB/s wr, 9 op/s 2023-03-09T19:37:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:37:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16768.507269 secs 2023-03-09T19:37:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:37:38 smithi181 ceph-mon[94189]: pgmap v9455: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 60 KiB/s wr, 9 op/s 2023-03-09T19:37:38.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:37:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16768.507269 secs 2023-03-09T19:37:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:37:38 smithi038 ceph-mon[92422]: pgmap v9455: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 60 KiB/s wr, 9 op/s 2023-03-09T19:37:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17230.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:37:40.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:37:40 smithi122 ceph-mon[93869]: pgmap v9456: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 60 KiB/s wr, 7 op/s 2023-03-09T19:37:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:37:40 smithi181 ceph-mon[94189]: pgmap v9456: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 60 KiB/s wr, 7 op/s 2023-03-09T19:37:40.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:37:40 smithi038 ceph-mon[92422]: pgmap v9456: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 60 KiB/s wr, 7 op/s 2023-03-09T19:37:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:37:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16771.200981 secs 2023-03-09T19:37:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:37:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16771.200981 secs 2023-03-09T19:37:41.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:37:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16771.200981 secs 2023-03-09T19:37:42.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:37:42 smithi122 ceph-mon[93869]: pgmap v9457: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1023 B/s wr, 8 op/s 2023-03-09T19:37:42.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:37:42 smithi181 ceph-mon[94189]: pgmap v9457: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1023 B/s wr, 8 op/s 2023-03-09T19:37:42.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:37:42 smithi038 ceph-mon[92422]: pgmap v9457: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1023 B/s wr, 8 op/s 2023-03-09T19:37:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:37:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16773.507359 secs 2023-03-09T19:37:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:37:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16773.507359 secs 2023-03-09T19:37:43.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:37:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16773.507359 secs 2023-03-09T19:37:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17235.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:37:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:37:44 smithi122 ceph-mon[93869]: pgmap v9458: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 184 KiB/s wr, 7 op/s 2023-03-09T19:37:44.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:37:44 smithi181 ceph-mon[94189]: pgmap v9458: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 184 KiB/s wr, 7 op/s 2023-03-09T19:37:44.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:37:44 smithi038 ceph-mon[92422]: pgmap v9458: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 184 KiB/s wr, 7 op/s 2023-03-09T19:37:45.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:37:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16776.201043 secs 2023-03-09T19:37:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:37:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16776.201043 secs 2023-03-09T19:37:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:37:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16776.201043 secs 2023-03-09T19:37:46.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:37:46 smithi122 ceph-mon[93869]: pgmap v9459: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 184 KiB/s wr, 6 op/s 2023-03-09T19:37:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:37:46 smithi181 ceph-mon[94189]: pgmap v9459: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 184 KiB/s wr, 6 op/s 2023-03-09T19:37:46.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:37:46 smithi038 ceph-mon[92422]: pgmap v9459: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 184 KiB/s wr, 6 op/s 2023-03-09T19:37:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:37:48 smithi122 ceph-mon[93869]: pgmap v9460: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 210 KiB/s wr, 9 op/s 2023-03-09T19:37:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:37:48 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:37:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:37:48 smithi181 ceph-mon[94189]: pgmap v9460: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 210 KiB/s wr, 9 op/s 2023-03-09T19:37:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:37:48 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:37:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:37:48 smithi038 ceph-mon[92422]: pgmap v9460: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 210 KiB/s wr, 9 op/s 2023-03-09T19:37:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:37:48 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:37:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17240.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:37:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:37:49 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:37:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:37:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:37:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:37:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:37:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:37:49 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:37:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:37:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:37:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:37:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:37:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:37:49 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:37:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:37:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:37:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:37:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:37:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:37:50 smithi122 ceph-mon[93869]: pgmap v9461: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 272 KiB/s wr, 8 op/s 2023-03-09T19:37:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:37:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16781.201102 secs 2023-03-09T19:37:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:37:50 smithi181 ceph-mon[94189]: pgmap v9461: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 272 KiB/s wr, 8 op/s 2023-03-09T19:37:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:37:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16781.201102 secs 2023-03-09T19:37:50.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:37:50 smithi038 ceph-mon[92422]: pgmap v9461: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 272 KiB/s wr, 8 op/s 2023-03-09T19:37:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:37:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16781.201102 secs 2023-03-09T19:37:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:37:52 smithi122 ceph-mon[93869]: pgmap v9462: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 272 KiB/s wr, 10 op/s 2023-03-09T19:37:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:37:52 smithi181 ceph-mon[94189]: pgmap v9462: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 272 KiB/s wr, 10 op/s 2023-03-09T19:37:52.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:37:52 smithi038 ceph-mon[92422]: pgmap v9462: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 272 KiB/s wr, 10 op/s 2023-03-09T19:37:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17245.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:37:54.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:37:54 smithi122 ceph-mon[93869]: pgmap v9463: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 309 KiB/s wr, 9 op/s 2023-03-09T19:37:54.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:37:54 smithi181 ceph-mon[94189]: pgmap v9463: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 309 KiB/s wr, 9 op/s 2023-03-09T19:37:54.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:37:54 smithi038 ceph-mon[92422]: pgmap v9463: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 309 KiB/s wr, 9 op/s 2023-03-09T19:37:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:37:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16786.201161 secs 2023-03-09T19:37:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:37:56 smithi122 ceph-mon[93869]: pgmap v9464: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 126 KiB/s wr, 8 op/s 2023-03-09T19:37:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:37:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16786.201161 secs 2023-03-09T19:37:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:37:56 smithi181 ceph-mon[94189]: pgmap v9464: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 126 KiB/s wr, 8 op/s 2023-03-09T19:37:56.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:37:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16786.201161 secs 2023-03-09T19:37:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:37:56 smithi038 ceph-mon[92422]: pgmap v9464: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 126 KiB/s wr, 8 op/s 2023-03-09T19:37:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:37:58 smithi122 ceph-mon[93869]: pgmap v9465: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 328 KiB/s wr, 12 op/s 2023-03-09T19:37:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:37:58 smithi181 ceph-mon[94189]: pgmap v9465: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 328 KiB/s wr, 12 op/s 2023-03-09T19:37:58.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:37:58 smithi038 ceph-mon[92422]: pgmap v9465: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 328 KiB/s wr, 12 op/s 2023-03-09T19:37:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17250.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:38:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:38:00 smithi122 ceph-mon[93869]: pgmap v9466: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 302 KiB/s wr, 10 op/s 2023-03-09T19:38:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:38:00 smithi181 ceph-mon[94189]: pgmap v9466: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 302 KiB/s wr, 10 op/s 2023-03-09T19:38:00.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:38:00 smithi038 ceph-mon[92422]: pgmap v9466: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 302 KiB/s wr, 10 op/s 2023-03-09T19:38:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:38:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16791.201222 secs 2023-03-09T19:38:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:38:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16791.201222 secs 2023-03-09T19:38:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:38:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16791.201222 secs 2023-03-09T19:38:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:38:02 smithi122 ceph-mon[93869]: pgmap v9467: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 277 KiB/s wr, 14 op/s 2023-03-09T19:38:02.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:38:02 smithi181 ceph-mon[94189]: pgmap v9467: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 277 KiB/s wr, 14 op/s 2023-03-09T19:38:02.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:38:02 smithi038 ceph-mon[92422]: pgmap v9467: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 277 KiB/s wr, 14 op/s 2023-03-09T19:38:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:38:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16793.507804 secs 2023-03-09T19:38:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:38:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16793.507804 secs 2023-03-09T19:38:03.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:38:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16793.507804 secs 2023-03-09T19:38:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17255.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:38:04.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:38:04 smithi122 ceph-mon[93869]: pgmap v9468: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 415 KiB/s wr, 25 op/s 2023-03-09T19:38:04.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:38:04 smithi181 ceph-mon[94189]: pgmap v9468: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 415 KiB/s wr, 25 op/s 2023-03-09T19:38:04.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:38:04 smithi038 ceph-mon[92422]: pgmap v9468: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 415 KiB/s wr, 25 op/s 2023-03-09T19:38:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:38:05 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:38:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:38:05 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:38:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:38:05 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:38:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:38:05 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:38:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:38:05 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:38:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:38:05 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:38:05.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:38:05 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:38:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:38:05 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:38:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:38:05 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:38:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:38:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16796.201283 secs 2023-03-09T19:38:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:38:06 smithi122 ceph-mon[93869]: pgmap v9469: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 357 KiB/s rd, 378 KiB/s wr, 24 op/s 2023-03-09T19:38:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:38:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16796.201283 secs 2023-03-09T19:38:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:38:06 smithi181 ceph-mon[94189]: pgmap v9469: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 357 KiB/s rd, 378 KiB/s wr, 24 op/s 2023-03-09T19:38:06.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:38:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16796.201283 secs 2023-03-09T19:38:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:38:06 smithi038 ceph-mon[92422]: pgmap v9469: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 357 KiB/s rd, 378 KiB/s wr, 24 op/s 2023-03-09T19:38:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:38:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16798.507945 secs 2023-03-09T19:38:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:38:08 smithi122 ceph-mon[93869]: pgmap v9470: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 415 KiB/s wr, 27 op/s 2023-03-09T19:38:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:38:08 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:38:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:38:08 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:38:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:38:08 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:38:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:38:08 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:38:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:38:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16798.507945 secs 2023-03-09T19:38:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:38:08 smithi181 ceph-mon[94189]: pgmap v9470: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 415 KiB/s wr, 27 op/s 2023-03-09T19:38:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:38:08 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:38:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:38:08 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:38:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:38:08 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:38:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:38:08 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:38:08.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:38:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16798.507945 secs 2023-03-09T19:38:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:38:08 smithi038 ceph-mon[92422]: pgmap v9470: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 415 KiB/s wr, 27 op/s 2023-03-09T19:38:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:38:08 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:38:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:38:08 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:38:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:38:08 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:38:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:38:08 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:38:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17260.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:38:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:38:10 smithi122 ceph-mon[93869]: pgmap v9471: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 213 KiB/s wr, 23 op/s 2023-03-09T19:38:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:38:10 smithi122 ceph-mon[93869]: osdmap e805: 12 total, 12 up, 12 in 2023-03-09T19:38:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:38:10 smithi181 ceph-mon[94189]: pgmap v9471: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 213 KiB/s wr, 23 op/s 2023-03-09T19:38:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:38:10 smithi181 ceph-mon[94189]: osdmap e805: 12 total, 12 up, 12 in 2023-03-09T19:38:10.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:38:10 smithi038 ceph-mon[92422]: pgmap v9471: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 213 KiB/s wr, 23 op/s 2023-03-09T19:38:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:38:10 smithi038 ceph-mon[92422]: osdmap e805: 12 total, 12 up, 12 in 2023-03-09T19:38:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:38:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16801.201363 secs 2023-03-09T19:38:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:38:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16801.201363 secs 2023-03-09T19:38:11.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:38:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16801.201363 secs 2023-03-09T19:38:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:38:12 smithi122 ceph-mon[93869]: pgmap v9473: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 241 KiB/s wr, 24 op/s 2023-03-09T19:38:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:38:12 smithi181 ceph-mon[94189]: pgmap v9473: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 241 KiB/s wr, 24 op/s 2023-03-09T19:38:12.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:38:12 smithi038 ceph-mon[92422]: pgmap v9473: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 241 KiB/s wr, 24 op/s 2023-03-09T19:38:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:38:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16803.508059 secs 2023-03-09T19:38:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:38:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16803.508059 secs 2023-03-09T19:38:13.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:38:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16803.508059 secs 2023-03-09T19:38:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17265.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:38:14.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:38:14 smithi122 ceph-mon[93869]: pgmap v9474: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 285 KiB/s wr, 9 op/s 2023-03-09T19:38:14.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:38:14 smithi181 ceph-mon[94189]: pgmap v9474: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 285 KiB/s wr, 9 op/s 2023-03-09T19:38:14.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:38:14 smithi038 ceph-mon[92422]: pgmap v9474: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 285 KiB/s wr, 9 op/s 2023-03-09T19:38:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:38:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16806.201424 secs 2023-03-09T19:38:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:38:16 smithi122 ceph-mon[93869]: pgmap v9475: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 285 KiB/s wr, 9 op/s 2023-03-09T19:38:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:38:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16806.201424 secs 2023-03-09T19:38:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:38:16 smithi181 ceph-mon[94189]: pgmap v9475: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 285 KiB/s wr, 9 op/s 2023-03-09T19:38:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:38:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16806.201424 secs 2023-03-09T19:38:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:38:16 smithi038 ceph-mon[92422]: pgmap v9475: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 285 KiB/s wr, 9 op/s 2023-03-09T19:38:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:38:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16808.508154 secs 2023-03-09T19:38:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:38:18 smithi122 ceph-mon[93869]: pgmap v9476: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 241 KiB/s wr, 9 op/s 2023-03-09T19:38:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:38:18 smithi122 ceph-mon[93869]: osdmap e806: 12 total, 12 up, 12 in 2023-03-09T19:38:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:38:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16808.508154 secs 2023-03-09T19:38:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:38:18 smithi181 ceph-mon[94189]: pgmap v9476: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 241 KiB/s wr, 9 op/s 2023-03-09T19:38:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:38:18 smithi181 ceph-mon[94189]: osdmap e806: 12 total, 12 up, 12 in 2023-03-09T19:38:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:38:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16808.508154 secs 2023-03-09T19:38:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:38:18 smithi038 ceph-mon[92422]: pgmap v9476: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 241 KiB/s wr, 9 op/s 2023-03-09T19:38:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:38:18 smithi038 ceph-mon[92422]: osdmap e806: 12 total, 12 up, 12 in 2023-03-09T19:38:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17270.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:38:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:38:20 smithi122 ceph-mon[93869]: pgmap v9478: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 243 KiB/s wr, 9 op/s 2023-03-09T19:38:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:38:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16811.201501 secs 2023-03-09T19:38:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:38:20 smithi181 ceph-mon[94189]: pgmap v9478: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 243 KiB/s wr, 9 op/s 2023-03-09T19:38:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:38:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16811.201501 secs 2023-03-09T19:38:20.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:38:20 smithi038 ceph-mon[92422]: pgmap v9478: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 243 KiB/s wr, 9 op/s 2023-03-09T19:38:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:38:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16811.201501 secs 2023-03-09T19:38:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:38:22 smithi122 ceph-mon[93869]: pgmap v9479: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 228 KiB/s wr, 9 op/s 2023-03-09T19:38:22.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:38:22 smithi181 ceph-mon[94189]: pgmap v9479: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 228 KiB/s wr, 9 op/s 2023-03-09T19:38:22.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:38:22 smithi038 ceph-mon[92422]: pgmap v9479: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 228 KiB/s wr, 9 op/s 2023-03-09T19:38:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:38:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16813.508246 secs 2023-03-09T19:38:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:38:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16813.508246 secs 2023-03-09T19:38:23.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:38:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16813.508246 secs 2023-03-09T19:38:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17275.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:38:24.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:38:24 smithi122 ceph-mon[93869]: pgmap v9480: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 18 KiB/s wr, 7 op/s 2023-03-09T19:38:24.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:38:24 smithi181 ceph-mon[94189]: pgmap v9480: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 18 KiB/s wr, 7 op/s 2023-03-09T19:38:24.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:38:24 smithi038 ceph-mon[92422]: pgmap v9480: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 18 KiB/s wr, 7 op/s 2023-03-09T19:38:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:38:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16816.201559 secs 2023-03-09T19:38:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:38:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16816.201559 secs 2023-03-09T19:38:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:38:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16816.201559 secs 2023-03-09T19:38:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:38:26 smithi122 ceph-mon[93869]: pgmap v9481: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 18 KiB/s wr, 7 op/s 2023-03-09T19:38:26.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:38:26 smithi181 ceph-mon[94189]: pgmap v9481: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 18 KiB/s wr, 7 op/s 2023-03-09T19:38:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:38:26 smithi038 ceph-mon[92422]: pgmap v9481: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 18 KiB/s wr, 7 op/s 2023-03-09T19:38:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:38:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16818.508323 secs 2023-03-09T19:38:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:38:28 smithi122 ceph-mon[93869]: pgmap v9482: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 88 KiB/s wr, 7 op/s 2023-03-09T19:38:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:38:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16818.508323 secs 2023-03-09T19:38:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:38:28 smithi181 ceph-mon[94189]: pgmap v9482: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 88 KiB/s wr, 7 op/s 2023-03-09T19:38:28.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:38:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16818.508323 secs 2023-03-09T19:38:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:38:28 smithi038 ceph-mon[92422]: pgmap v9482: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 88 KiB/s wr, 7 op/s 2023-03-09T19:38:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17280.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:38:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:38:30 smithi122 ceph-mon[93869]: pgmap v9483: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 78 KiB/s wr, 6 op/s 2023-03-09T19:38:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:38:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16821.201621 secs 2023-03-09T19:38:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:38:30 smithi181 ceph-mon[94189]: pgmap v9483: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 78 KiB/s wr, 6 op/s 2023-03-09T19:38:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:38:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16821.201621 secs 2023-03-09T19:38:30.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:38:30 smithi038 ceph-mon[92422]: pgmap v9483: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 78 KiB/s wr, 6 op/s 2023-03-09T19:38:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:38:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16821.201621 secs 2023-03-09T19:38:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:38:32 smithi122 ceph-mon[93869]: pgmap v9484: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 74 KiB/s wr, 9 op/s 2023-03-09T19:38:32.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:38:32 smithi181 ceph-mon[94189]: pgmap v9484: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 74 KiB/s wr, 9 op/s 2023-03-09T19:38:32.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:38:32 smithi038 ceph-mon[92422]: pgmap v9484: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 74 KiB/s wr, 9 op/s 2023-03-09T19:38:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:38:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16823.508457 secs 2023-03-09T19:38:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:38:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16823.508457 secs 2023-03-09T19:38:33.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:38:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16823.508457 secs 2023-03-09T19:38:33.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17285.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:38:34.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:38:34 smithi122 ceph-mon[93869]: pgmap v9485: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 60 KiB/s wr, 7 op/s 2023-03-09T19:38:34.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:38:34 smithi181 ceph-mon[94189]: pgmap v9485: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 60 KiB/s wr, 7 op/s 2023-03-09T19:38:34.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:38:34 smithi038 ceph-mon[92422]: pgmap v9485: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 60 KiB/s wr, 7 op/s 2023-03-09T19:38:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:38:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16826.201680 secs 2023-03-09T19:38:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:38:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16826.201680 secs 2023-03-09T19:38:35.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:38:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16826.201680 secs 2023-03-09T19:38:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:38:36 smithi122 ceph-mon[93869]: pgmap v9486: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 60 KiB/s wr, 6 op/s 2023-03-09T19:38:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:38:36 smithi181 ceph-mon[94189]: pgmap v9486: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 60 KiB/s wr, 6 op/s 2023-03-09T19:38:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:38:36 smithi038 ceph-mon[92422]: pgmap v9486: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 60 KiB/s wr, 6 op/s 2023-03-09T19:38:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:38:38 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 16828.508576 secs 2023-03-09T19:38:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:38:38 smithi122 ceph-mon[93869]: pgmap v9487: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 197 KiB/s wr, 10 op/s 2023-03-09T19:38:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:38:38 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 16828.508576 secs 2023-03-09T19:38:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:38:38 smithi181 ceph-mon[94189]: pgmap v9487: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 197 KiB/s wr, 10 op/s 2023-03-09T19:38:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:38:38 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 16828.508576 secs 2023-03-09T19:38:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:38:38 smithi038 ceph-mon[92422]: pgmap v9487: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 197 KiB/s wr, 10 op/s 2023-03-09T19:38:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17290.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:38:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:38:40 smithi038 ceph-mon[92422]: pgmap v9488: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 138 KiB/s wr, 7 op/s 2023-03-09T19:38:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:38:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16831.201741 secs 2023-03-09T19:38:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:38:40 smithi122 ceph-mon[93869]: pgmap v9488: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 138 KiB/s wr, 7 op/s 2023-03-09T19:38:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:38:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16831.201741 secs 2023-03-09T19:38:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:38:40 smithi181 ceph-mon[94189]: pgmap v9488: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 138 KiB/s wr, 7 op/s 2023-03-09T19:38:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:38:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16831.201741 secs 2023-03-09T19:38:41.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:38:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:38:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:38:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:38:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:38:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:38:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:38:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:38:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:38:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:38:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:38:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:38:42.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:38:42 smithi122 ceph-mon[93869]: pgmap v9489: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 167 KiB/s wr, 10 op/s 2023-03-09T19:38:42.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:38:42 smithi181 ceph-mon[94189]: pgmap v9489: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 167 KiB/s wr, 10 op/s 2023-03-09T19:38:42.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:38:42 smithi038 ceph-mon[92422]: pgmap v9489: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 167 KiB/s wr, 10 op/s 2023-03-09T19:38:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17295.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:38:43.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:38:43 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 16833.508658 secs 2023-03-09T19:38:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:38:43 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 16833.508658 secs 2023-03-09T19:38:43.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:38:43 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 16833.508658 secs 2023-03-09T19:38:44.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:38:44 smithi122 ceph-mon[93869]: pgmap v9490: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 166 KiB/s wr, 9 op/s 2023-03-09T19:38:44.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:38:44 smithi181 ceph-mon[94189]: pgmap v9490: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 166 KiB/s wr, 9 op/s 2023-03-09T19:38:44.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:38:44 smithi038 ceph-mon[92422]: pgmap v9490: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 166 KiB/s wr, 9 op/s 2023-03-09T19:38:45.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:38:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16836.201801 secs 2023-03-09T19:38:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:38:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16836.201801 secs 2023-03-09T19:38:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:38:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16836.201801 secs 2023-03-09T19:38:46.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:38:46 smithi122 ceph-mon[93869]: pgmap v9491: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 166 KiB/s wr, 8 op/s 2023-03-09T19:38:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:38:46 smithi181 ceph-mon[94189]: pgmap v9491: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 166 KiB/s wr, 8 op/s 2023-03-09T19:38:46.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:38:46 smithi038 ceph-mon[92422]: pgmap v9491: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 166 KiB/s wr, 8 op/s 2023-03-09T19:38:48.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17300.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:38:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:38:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16838.508782 secs 2023-03-09T19:38:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:38:48 smithi122 ceph-mon[93869]: pgmap v9492: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 313 KiB/s wr, 13 op/s 2023-03-09T19:38:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:38:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16838.508782 secs 2023-03-09T19:38:48.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:38:48 smithi181 ceph-mon[94189]: pgmap v9492: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 313 KiB/s wr, 13 op/s 2023-03-09T19:38:48.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:38:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16838.508782 secs 2023-03-09T19:38:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:38:48 smithi038 ceph-mon[92422]: pgmap v9492: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 313 KiB/s wr, 13 op/s 2023-03-09T19:38:49.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:38:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:38:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:38:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:38:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:38:49 smithi122 ceph-mon[93869]: pgmap v9493: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 176 KiB/s wr, 9 op/s 2023-03-09T19:38:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:38:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:38:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:38:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:38:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:38:49 smithi181 ceph-mon[94189]: pgmap v9493: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 176 KiB/s wr, 9 op/s 2023-03-09T19:38:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:38:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:38:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:38:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:38:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:38:49 smithi038 ceph-mon[92422]: pgmap v9493: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 176 KiB/s wr, 9 op/s 2023-03-09T19:38:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:38:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16841.201861 secs 2023-03-09T19:38:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:38:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16841.201861 secs 2023-03-09T19:38:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:38:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16841.201861 secs 2023-03-09T19:38:51.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:38:51 smithi122 ceph-mon[93869]: pgmap v9494: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 355 KiB/s wr, 14 op/s 2023-03-09T19:38:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:38:51 smithi181 ceph-mon[94189]: pgmap v9494: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 355 KiB/s wr, 14 op/s 2023-03-09T19:38:51.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:38:51 smithi038 ceph-mon[92422]: pgmap v9494: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 355 KiB/s wr, 14 op/s 2023-03-09T19:38:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:38:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16843.508878 secs 2023-03-09T19:38:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:38:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16843.508878 secs 2023-03-09T19:38:52.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:38:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16843.508878 secs 2023-03-09T19:38:53.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17305.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:38:54.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:38:54 smithi122 ceph-mon[93869]: pgmap v9495: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 340 KiB/s wr, 11 op/s 2023-03-09T19:38:54.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:38:54 smithi181 ceph-mon[94189]: pgmap v9495: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 340 KiB/s wr, 11 op/s 2023-03-09T19:38:54.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:38:54 smithi038 ceph-mon[92422]: pgmap v9495: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 340 KiB/s wr, 11 op/s 2023-03-09T19:38:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:38:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16846.201920 secs 2023-03-09T19:38:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:38:56 smithi122 ceph-mon[93869]: pgmap v9496: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 340 KiB/s wr, 10 op/s 2023-03-09T19:38:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:38:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16846.201920 secs 2023-03-09T19:38:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:38:56 smithi181 ceph-mon[94189]: pgmap v9496: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 340 KiB/s wr, 10 op/s 2023-03-09T19:38:56.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:38:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16846.201920 secs 2023-03-09T19:38:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:38:56 smithi038 ceph-mon[92422]: pgmap v9496: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 340 KiB/s wr, 10 op/s 2023-03-09T19:38:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:38:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16848.508969 secs 2023-03-09T19:38:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:38:58 smithi122 ceph-mon[93869]: pgmap v9497: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 424 KiB/s wr, 14 op/s 2023-03-09T19:38:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:38:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16848.508969 secs 2023-03-09T19:38:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:38:58 smithi181 ceph-mon[94189]: pgmap v9497: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 424 KiB/s wr, 14 op/s 2023-03-09T19:38:58.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:38:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16848.508969 secs 2023-03-09T19:38:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:38:58 smithi038 ceph-mon[92422]: pgmap v9497: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 424 KiB/s wr, 14 op/s 2023-03-09T19:38:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17310.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:39:00.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:39:00 smithi122 ceph-mon[93869]: pgmap v9498: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 277 KiB/s wr, 10 op/s 2023-03-09T19:39:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:39:00 smithi181 ceph-mon[94189]: pgmap v9498: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 277 KiB/s wr, 10 op/s 2023-03-09T19:39:00.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:39:00 smithi038 ceph-mon[92422]: pgmap v9498: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 277 KiB/s wr, 10 op/s 2023-03-09T19:39:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:39:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16851.201985 secs 2023-03-09T19:39:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:39:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16851.201985 secs 2023-03-09T19:39:01.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:39:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16851.201985 secs 2023-03-09T19:39:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:39:02 smithi122 ceph-mon[93869]: pgmap v9499: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 313 KiB/s wr, 14 op/s 2023-03-09T19:39:02.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:39:02 smithi181 ceph-mon[94189]: pgmap v9499: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 313 KiB/s wr, 14 op/s 2023-03-09T19:39:02.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:39:02 smithi038 ceph-mon[92422]: pgmap v9499: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 313 KiB/s wr, 14 op/s 2023-03-09T19:39:03.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:39:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16853.509069 secs 2023-03-09T19:39:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:39:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16853.509069 secs 2023-03-09T19:39:03.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:39:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16853.509069 secs 2023-03-09T19:39:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17315.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:39:04.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:39:04 smithi122 ceph-mon[93869]: pgmap v9500: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 271 KiB/s wr, 24 op/s 2023-03-09T19:39:04.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:39:04 smithi181 ceph-mon[94189]: pgmap v9500: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 271 KiB/s wr, 24 op/s 2023-03-09T19:39:04.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:39:04 smithi038 ceph-mon[92422]: pgmap v9500: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 271 KiB/s wr, 24 op/s 2023-03-09T19:39:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:39:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16856.202044 secs 2023-03-09T19:39:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:39:06 smithi122 ceph-mon[93869]: pgmap v9501: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 257 KiB/s wr, 23 op/s 2023-03-09T19:39:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:39:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16856.202044 secs 2023-03-09T19:39:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:39:06 smithi181 ceph-mon[94189]: pgmap v9501: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 257 KiB/s wr, 23 op/s 2023-03-09T19:39:06.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:39:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16856.202044 secs 2023-03-09T19:39:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:39:06 smithi038 ceph-mon[92422]: pgmap v9501: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 257 KiB/s wr, 23 op/s 2023-03-09T19:39:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:39:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16858.509173 secs 2023-03-09T19:39:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:39:08 smithi122 ceph-mon[93869]: pgmap v9502: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 357 KiB/s wr, 27 op/s 2023-03-09T19:39:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:39:08 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:39:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:39:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16858.509173 secs 2023-03-09T19:39:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:39:08 smithi181 ceph-mon[94189]: pgmap v9502: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 357 KiB/s wr, 27 op/s 2023-03-09T19:39:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:39:08 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:39:08.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:39:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16858.509173 secs 2023-03-09T19:39:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:39:08 smithi038 ceph-mon[92422]: pgmap v9502: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 357 KiB/s wr, 27 op/s 2023-03-09T19:39:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:39:08 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:39:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17320.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:39:09.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:39:09 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:39:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:39:09 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:39:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:39:09 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:39:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:39:09 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:39:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:39:09 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:39:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:39:09 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:39:09.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:39:09 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:39:09.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:39:09 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:39:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:39:09 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:39:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:39:10 smithi122 ceph-mon[93869]: pgmap v9503: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 273 KiB/s wr, 23 op/s 2023-03-09T19:39:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:39:10 smithi122 ceph-mon[93869]: osdmap e807: 12 total, 12 up, 12 in 2023-03-09T19:39:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:39:10 smithi181 ceph-mon[94189]: pgmap v9503: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 273 KiB/s wr, 23 op/s 2023-03-09T19:39:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:39:10 smithi181 ceph-mon[94189]: osdmap e807: 12 total, 12 up, 12 in 2023-03-09T19:39:10.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:39:10 smithi038 ceph-mon[92422]: pgmap v9503: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 273 KiB/s wr, 23 op/s 2023-03-09T19:39:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:39:10 smithi038 ceph-mon[92422]: osdmap e807: 12 total, 12 up, 12 in 2023-03-09T19:39:11.733 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:39:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16861.202106 secs 2023-03-09T19:39:11.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:39:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16861.202106 secs 2023-03-09T19:39:11.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:39:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16861.202106 secs 2023-03-09T19:39:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:39:12 smithi122 ceph-mon[93869]: pgmap v9505: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 300 KiB/s wr, 24 op/s 2023-03-09T19:39:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:39:12 smithi181 ceph-mon[94189]: pgmap v9505: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 300 KiB/s wr, 24 op/s 2023-03-09T19:39:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:39:12 smithi038 ceph-mon[92422]: pgmap v9505: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 300 KiB/s wr, 24 op/s 2023-03-09T19:39:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:39:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16863.509287 secs 2023-03-09T19:39:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:39:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16863.509287 secs 2023-03-09T19:39:13.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:39:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16863.509287 secs 2023-03-09T19:39:13.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17325.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:39:14.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:39:14 smithi122 ceph-mon[93869]: pgmap v9506: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 135 KiB/s wr, 9 op/s 2023-03-09T19:39:14.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:39:14 smithi181 ceph-mon[94189]: pgmap v9506: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 135 KiB/s wr, 9 op/s 2023-03-09T19:39:14.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:39:14 smithi038 ceph-mon[92422]: pgmap v9506: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 135 KiB/s wr, 9 op/s 2023-03-09T19:39:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:39:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16866.202191 secs 2023-03-09T19:39:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:39:16 smithi122 ceph-mon[93869]: pgmap v9507: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 135 KiB/s wr, 9 op/s 2023-03-09T19:39:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:39:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16866.202191 secs 2023-03-09T19:39:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:39:16 smithi181 ceph-mon[94189]: pgmap v9507: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 135 KiB/s wr, 9 op/s 2023-03-09T19:39:16.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:39:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16866.202191 secs 2023-03-09T19:39:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:39:16 smithi038 ceph-mon[92422]: pgmap v9507: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 135 KiB/s wr, 9 op/s 2023-03-09T19:39:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:39:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16868.509364 secs 2023-03-09T19:39:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:39:18 smithi122 ceph-mon[93869]: pgmap v9508: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 80 KiB/s wr, 8 op/s 2023-03-09T19:39:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:39:18 smithi122 ceph-mon[93869]: osdmap e808: 12 total, 12 up, 12 in 2023-03-09T19:39:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:39:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16868.509364 secs 2023-03-09T19:39:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:39:18 smithi181 ceph-mon[94189]: pgmap v9508: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 80 KiB/s wr, 8 op/s 2023-03-09T19:39:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:39:18 smithi181 ceph-mon[94189]: osdmap e808: 12 total, 12 up, 12 in 2023-03-09T19:39:18.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:39:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16868.509364 secs 2023-03-09T19:39:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:39:18 smithi038 ceph-mon[92422]: pgmap v9508: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 80 KiB/s wr, 8 op/s 2023-03-09T19:39:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:39:18 smithi038 ceph-mon[92422]: osdmap e808: 12 total, 12 up, 12 in 2023-03-09T19:39:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17330.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:39:20.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:39:20 smithi122 ceph-mon[93869]: pgmap v9510: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 81 KiB/s wr, 8 op/s 2023-03-09T19:39:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:39:20 smithi181 ceph-mon[94189]: pgmap v9510: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 81 KiB/s wr, 8 op/s 2023-03-09T19:39:20.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:39:20 smithi038 ceph-mon[92422]: pgmap v9510: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 81 KiB/s wr, 8 op/s 2023-03-09T19:39:21.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:39:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16871.202253 secs 2023-03-09T19:39:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:39:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16871.202253 secs 2023-03-09T19:39:21.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:39:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16871.202253 secs 2023-03-09T19:39:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:39:22 smithi122 ceph-mon[93869]: pgmap v9511: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 64 KiB/s wr, 7 op/s 2023-03-09T19:39:22.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:39:22 smithi181 ceph-mon[94189]: pgmap v9511: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 64 KiB/s wr, 7 op/s 2023-03-09T19:39:22.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:39:22 smithi038 ceph-mon[92422]: pgmap v9511: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 64 KiB/s wr, 7 op/s 2023-03-09T19:39:23.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:39:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16873.509488 secs 2023-03-09T19:39:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:39:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16873.509488 secs 2023-03-09T19:39:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:39:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16873.509488 secs 2023-03-09T19:39:23.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17335.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:39:24.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:39:24 smithi122 ceph-mon[93869]: pgmap v9512: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 130 KiB/s wr, 7 op/s 2023-03-09T19:39:24.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:39:24 smithi181 ceph-mon[94189]: pgmap v9512: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 130 KiB/s wr, 7 op/s 2023-03-09T19:39:24.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:39:24 smithi038 ceph-mon[92422]: pgmap v9512: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 130 KiB/s wr, 7 op/s 2023-03-09T19:39:25.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:39:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16876.202313 secs 2023-03-09T19:39:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:39:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16876.202313 secs 2023-03-09T19:39:25.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:39:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16876.202313 secs 2023-03-09T19:39:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:39:26 smithi122 ceph-mon[93869]: pgmap v9513: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 130 KiB/s wr, 8 op/s 2023-03-09T19:39:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:39:26 smithi181 ceph-mon[94189]: pgmap v9513: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 130 KiB/s wr, 8 op/s 2023-03-09T19:39:26.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:39:26 smithi038 ceph-mon[92422]: pgmap v9513: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 130 KiB/s wr, 8 op/s 2023-03-09T19:39:28.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:39:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16878.509568 secs 2023-03-09T19:39:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:39:28 smithi122 ceph-mon[93869]: pgmap v9514: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 7 op/s 2023-03-09T19:39:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:39:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16878.509568 secs 2023-03-09T19:39:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:39:28 smithi181 ceph-mon[94189]: pgmap v9514: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 7 op/s 2023-03-09T19:39:28.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:39:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16878.509568 secs 2023-03-09T19:39:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:39:28 smithi038 ceph-mon[92422]: pgmap v9514: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 7 op/s 2023-03-09T19:39:28.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17340.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:39:30.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:39:30 smithi122 ceph-mon[93869]: pgmap v9515: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 64 KiB/s wr, 7 op/s 2023-03-09T19:39:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:39:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16881.202373 secs 2023-03-09T19:39:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:39:30 smithi181 ceph-mon[94189]: pgmap v9515: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 64 KiB/s wr, 7 op/s 2023-03-09T19:39:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:39:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16881.202373 secs 2023-03-09T19:39:30.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:39:30 smithi038 ceph-mon[92422]: pgmap v9515: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 64 KiB/s wr, 7 op/s 2023-03-09T19:39:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:39:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16881.202373 secs 2023-03-09T19:39:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:39:32 smithi122 ceph-mon[93869]: pgmap v9516: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 60 KiB/s wr, 9 op/s 2023-03-09T19:39:32.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:39:32 smithi181 ceph-mon[94189]: pgmap v9516: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 60 KiB/s wr, 9 op/s 2023-03-09T19:39:32.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:39:32 smithi038 ceph-mon[92422]: pgmap v9516: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 60 KiB/s wr, 9 op/s 2023-03-09T19:39:33.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:39:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16883.509629 secs 2023-03-09T19:39:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:39:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16883.509629 secs 2023-03-09T19:39:33.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:39:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16883.509629 secs 2023-03-09T19:39:33.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17345.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:39:34.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:39:34 smithi122 ceph-mon[93869]: pgmap v9517: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 119 KiB/s wr, 8 op/s 2023-03-09T19:39:34.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:39:34 smithi181 ceph-mon[94189]: pgmap v9517: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 119 KiB/s wr, 8 op/s 2023-03-09T19:39:34.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:39:34 smithi038 ceph-mon[92422]: pgmap v9517: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 119 KiB/s wr, 8 op/s 2023-03-09T19:39:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:39:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16886.202436 secs 2023-03-09T19:39:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:39:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16886.202436 secs 2023-03-09T19:39:35.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:39:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16886.202436 secs 2023-03-09T19:39:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:39:36 smithi122 ceph-mon[93869]: pgmap v9518: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 64 KiB/s wr, 7 op/s 2023-03-09T19:39:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:39:36 smithi181 ceph-mon[94189]: pgmap v9518: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 64 KiB/s wr, 7 op/s 2023-03-09T19:39:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:39:36 smithi038 ceph-mon[92422]: pgmap v9518: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 64 KiB/s wr, 7 op/s 2023-03-09T19:39:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:39:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16888.509747 secs 2023-03-09T19:39:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:39:38 smithi122 ceph-mon[93869]: pgmap v9519: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 64 KiB/s wr, 10 op/s 2023-03-09T19:39:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:39:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16888.509747 secs 2023-03-09T19:39:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:39:38 smithi181 ceph-mon[94189]: pgmap v9519: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 64 KiB/s wr, 10 op/s 2023-03-09T19:39:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:39:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16888.509747 secs 2023-03-09T19:39:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:39:38 smithi038 ceph-mon[92422]: pgmap v9519: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 64 KiB/s wr, 10 op/s 2023-03-09T19:39:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17350.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:39:40.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:39:40 smithi122 ceph-mon[93869]: pgmap v9520: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 59 KiB/s wr, 7 op/s 2023-03-09T19:39:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:39:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16891.202497 secs 2023-03-09T19:39:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:39:40 smithi181 ceph-mon[94189]: pgmap v9520: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 59 KiB/s wr, 7 op/s 2023-03-09T19:39:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:39:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16891.202497 secs 2023-03-09T19:39:40.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:39:40 smithi038 ceph-mon[92422]: pgmap v9520: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 59 KiB/s wr, 7 op/s 2023-03-09T19:39:40.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:39:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16891.202497 secs 2023-03-09T19:39:42.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:39:42 smithi122 ceph-mon[93869]: pgmap v9521: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 60 KiB/s wr, 9 op/s 2023-03-09T19:39:42.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:39:42 smithi181 ceph-mon[94189]: pgmap v9521: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 60 KiB/s wr, 9 op/s 2023-03-09T19:39:42.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:39:42 smithi038 ceph-mon[92422]: pgmap v9521: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 60 KiB/s wr, 9 op/s 2023-03-09T19:39:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:39:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16893.509859 secs 2023-03-09T19:39:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:39:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16893.509859 secs 2023-03-09T19:39:43.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:39:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16893.509859 secs 2023-03-09T19:39:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17355.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:39:44.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:39:44 smithi122 ceph-mon[93869]: pgmap v9522: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 130 KiB/s wr, 8 op/s 2023-03-09T19:39:44.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:39:44 smithi181 ceph-mon[94189]: pgmap v9522: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 130 KiB/s wr, 8 op/s 2023-03-09T19:39:44.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:39:44 smithi038 ceph-mon[92422]: pgmap v9522: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 130 KiB/s wr, 8 op/s 2023-03-09T19:39:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:39:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16896.202556 secs 2023-03-09T19:39:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:39:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16896.202556 secs 2023-03-09T19:39:45.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:39:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16896.202556 secs 2023-03-09T19:39:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:39:46 smithi122 ceph-mon[93869]: pgmap v9523: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 71 KiB/s wr, 6 op/s 2023-03-09T19:39:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:39:46 smithi181 ceph-mon[94189]: pgmap v9523: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 71 KiB/s wr, 6 op/s 2023-03-09T19:39:46.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:39:46 smithi038 ceph-mon[92422]: pgmap v9523: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 71 KiB/s wr, 6 op/s 2023-03-09T19:39:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:39:48 smithi122 ceph-mon[93869]: pgmap v9524: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 216 KiB/s wr, 9 op/s 2023-03-09T19:39:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:39:48 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:39:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:39:48 smithi181 ceph-mon[94189]: pgmap v9524: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 216 KiB/s wr, 9 op/s 2023-03-09T19:39:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:39:48 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:39:48.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:39:48 smithi038 ceph-mon[92422]: pgmap v9524: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 216 KiB/s wr, 9 op/s 2023-03-09T19:39:48.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:39:48 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:39:48.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17360.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:39:49.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:39:49 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:39:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:39:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:39:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:39:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:39:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:39:49 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:39:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:39:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:39:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:39:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:39:49.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:39:49 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:39:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:39:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:39:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:39:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:39:50.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:39:50 smithi122 ceph-mon[93869]: pgmap v9525: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 374 KiB/s wr, 32 op/s 2023-03-09T19:39:50.731 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:39:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16901.202619 secs 2023-03-09T19:39:50.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:39:50 smithi181 ceph-mon[94189]: pgmap v9525: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 374 KiB/s wr, 32 op/s 2023-03-09T19:39:50.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:39:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16901.202619 secs 2023-03-09T19:39:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:39:50 smithi038 ceph-mon[92422]: pgmap v9525: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 374 KiB/s wr, 32 op/s 2023-03-09T19:39:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:39:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16901.202619 secs 2023-03-09T19:39:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:39:52 smithi122 ceph-mon[93869]: pgmap v9526: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 430 KiB/s rd, 390 KiB/s wr, 35 op/s 2023-03-09T19:39:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:39:52 smithi181 ceph-mon[94189]: pgmap v9526: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 430 KiB/s rd, 390 KiB/s wr, 35 op/s 2023-03-09T19:39:52.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:39:52 smithi038 ceph-mon[92422]: pgmap v9526: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 430 KiB/s rd, 390 KiB/s wr, 35 op/s 2023-03-09T19:39:53.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17365.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:39:54.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:39:54 smithi122 ceph-mon[93869]: pgmap v9527: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 390 KiB/s wr, 43 op/s 2023-03-09T19:39:54.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:39:54 smithi181 ceph-mon[94189]: pgmap v9527: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 390 KiB/s wr, 43 op/s 2023-03-09T19:39:54.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:39:54 smithi038 ceph-mon[92422]: pgmap v9527: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 390 KiB/s wr, 43 op/s 2023-03-09T19:39:56.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:39:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16906.202678 secs 2023-03-09T19:39:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:39:56 smithi122 ceph-mon[93869]: pgmap v9528: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 319 KiB/s wr, 42 op/s 2023-03-09T19:39:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:39:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16906.202678 secs 2023-03-09T19:39:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:39:56 smithi181 ceph-mon[94189]: pgmap v9528: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 319 KiB/s wr, 42 op/s 2023-03-09T19:39:56.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:39:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16906.202678 secs 2023-03-09T19:39:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:39:56 smithi038 ceph-mon[92422]: pgmap v9528: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 319 KiB/s wr, 42 op/s 2023-03-09T19:39:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:39:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16908.510132 secs 2023-03-09T19:39:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:39:58 smithi122 ceph-mon[93869]: pgmap v9529: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 593 KiB/s rd, 540 KiB/s wr, 47 op/s 2023-03-09T19:39:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:39:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16908.510132 secs 2023-03-09T19:39:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:39:58 smithi181 ceph-mon[94189]: pgmap v9529: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 593 KiB/s rd, 540 KiB/s wr, 47 op/s 2023-03-09T19:39:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:39:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16908.510132 secs 2023-03-09T19:39:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:39:58 smithi038 ceph-mon[92422]: pgmap v9529: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 593 KiB/s rd, 540 KiB/s wr, 47 op/s 2023-03-09T19:39:58.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17370.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:40:00.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:40:00 smithi122 ceph-mon[93869]: pgmap v9530: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 444 KiB/s rd, 395 KiB/s wr, 44 op/s 2023-03-09T19:40:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:40:00 smithi122 ceph-mon[93869]: Health detail: HEALTH_WARN 1 MDSs report slow requests 2023-03-09T19:40:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:40:00 smithi122 ceph-mon[93869]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2023-03-09T19:40:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:40:00 smithi122 ceph-mon[93869]: mds.h(mds.1): 2 slow requests are blocked > 180 secs 2023-03-09T19:40:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:40:00 smithi181 ceph-mon[94189]: pgmap v9530: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 444 KiB/s rd, 395 KiB/s wr, 44 op/s 2023-03-09T19:40:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:40:00 smithi181 ceph-mon[94189]: Health detail: HEALTH_WARN 1 MDSs report slow requests 2023-03-09T19:40:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:40:00 smithi181 ceph-mon[94189]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2023-03-09T19:40:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:40:00 smithi181 ceph-mon[94189]: mds.h(mds.1): 2 slow requests are blocked > 180 secs 2023-03-09T19:40:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:40:00 smithi038 ceph-mon[92422]: pgmap v9530: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 444 KiB/s rd, 395 KiB/s wr, 44 op/s 2023-03-09T19:40:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:40:00 smithi038 ceph-mon[92422]: Health detail: HEALTH_WARN 1 MDSs report slow requests 2023-03-09T19:40:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:40:00 smithi038 ceph-mon[92422]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2023-03-09T19:40:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:40:00 smithi038 ceph-mon[92422]: mds.h(mds.1): 2 slow requests are blocked > 180 secs 2023-03-09T19:40:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:40:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16911.202739 secs 2023-03-09T19:40:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:40:01 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:40:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:40:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:40:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:40:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:40:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:40:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16911.202739 secs 2023-03-09T19:40:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:40:01 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:40:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:40:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:40:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:40:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:40:01.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:40:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16911.202739 secs 2023-03-09T19:40:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:40:01 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:40:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:40:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:40:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:40:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:40:02.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:40:02 smithi122 ceph-mon[93869]: pgmap v9531: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 251 KiB/s wr, 23 op/s 2023-03-09T19:40:02.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:40:02 smithi181 ceph-mon[94189]: pgmap v9531: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 251 KiB/s wr, 23 op/s 2023-03-09T19:40:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:40:02 smithi038 ceph-mon[92422]: pgmap v9531: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 251 KiB/s wr, 23 op/s 2023-03-09T19:40:03.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:40:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16913.510253 secs 2023-03-09T19:40:03.795 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:40:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16913.510253 secs 2023-03-09T19:40:03.796 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:40:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16913.510253 secs 2023-03-09T19:40:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17375.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:40:04.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:40:04 smithi122 ceph-mon[93869]: pgmap v9532: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 263 KiB/s wr, 22 op/s 2023-03-09T19:40:04.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:40:04 smithi181 ceph-mon[94189]: pgmap v9532: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 263 KiB/s wr, 22 op/s 2023-03-09T19:40:04.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:40:04 smithi038 ceph-mon[92422]: pgmap v9532: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 263 KiB/s wr, 22 op/s 2023-03-09T19:40:06.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:40:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16916.202821 secs 2023-03-09T19:40:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:40:06 smithi122 ceph-mon[93869]: pgmap v9533: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 263 KiB/s wr, 11 op/s 2023-03-09T19:40:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:40:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16916.202821 secs 2023-03-09T19:40:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:40:06 smithi181 ceph-mon[94189]: pgmap v9533: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 263 KiB/s wr, 11 op/s 2023-03-09T19:40:06.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:40:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16916.202821 secs 2023-03-09T19:40:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:40:06 smithi038 ceph-mon[92422]: pgmap v9533: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 263 KiB/s wr, 11 op/s 2023-03-09T19:40:08.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:40:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16918.510332 secs 2023-03-09T19:40:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:40:08 smithi122 ceph-mon[93869]: pgmap v9534: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 298 KiB/s wr, 17 op/s 2023-03-09T19:40:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:40:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16918.510332 secs 2023-03-09T19:40:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:40:08 smithi181 ceph-mon[94189]: pgmap v9534: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 298 KiB/s wr, 17 op/s 2023-03-09T19:40:08.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:40:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16918.510332 secs 2023-03-09T19:40:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:40:08 smithi038 ceph-mon[92422]: pgmap v9534: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 298 KiB/s wr, 17 op/s 2023-03-09T19:40:08.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17380.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:40:09.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:40:09 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:40:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:40:09 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:40:09.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:40:09 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:40:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:40:10 smithi122 ceph-mon[93869]: pgmap v9535: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 304 KiB/s wr, 25 op/s 2023-03-09T19:40:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:40:10 smithi122 ceph-mon[93869]: osdmap e809: 12 total, 12 up, 12 in 2023-03-09T19:40:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:40:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:40:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:40:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:40:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:40:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:40:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:40:10 smithi181 ceph-mon[94189]: pgmap v9535: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 304 KiB/s wr, 25 op/s 2023-03-09T19:40:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:40:10 smithi181 ceph-mon[94189]: osdmap e809: 12 total, 12 up, 12 in 2023-03-09T19:40:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:40:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:40:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:40:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:40:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:40:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:40:10.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:40:10 smithi038 ceph-mon[92422]: pgmap v9535: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 304 KiB/s wr, 25 op/s 2023-03-09T19:40:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:40:10 smithi038 ceph-mon[92422]: osdmap e809: 12 total, 12 up, 12 in 2023-03-09T19:40:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:40:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:40:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:40:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:40:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:40:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:40:11.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:40:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16921.202881 secs 2023-03-09T19:40:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:40:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16921.202881 secs 2023-03-09T19:40:11.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:40:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16921.202881 secs 2023-03-09T19:40:12.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:40:12 smithi122 ceph-mon[93869]: pgmap v9537: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 366 KiB/s wr, 29 op/s 2023-03-09T19:40:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:40:12 smithi181 ceph-mon[94189]: pgmap v9537: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 366 KiB/s wr, 29 op/s 2023-03-09T19:40:12.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:40:12 smithi038 ceph-mon[92422]: pgmap v9537: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 366 KiB/s wr, 29 op/s 2023-03-09T19:40:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:40:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16923.510446 secs 2023-03-09T19:40:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:40:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16923.510446 secs 2023-03-09T19:40:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:40:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16923.510446 secs 2023-03-09T19:40:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17385.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:40:14.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:40:14 smithi122 ceph-mon[93869]: pgmap v9538: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 390 KiB/s wr, 28 op/s 2023-03-09T19:40:14.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:40:14 smithi181 ceph-mon[94189]: pgmap v9538: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 390 KiB/s wr, 28 op/s 2023-03-09T19:40:14.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:40:14 smithi038 ceph-mon[92422]: pgmap v9538: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 390 KiB/s wr, 28 op/s 2023-03-09T19:40:16.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:40:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16926.202942 secs 2023-03-09T19:40:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:40:16 smithi122 ceph-mon[93869]: pgmap v9539: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 390 KiB/s wr, 28 op/s 2023-03-09T19:40:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:40:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16926.202942 secs 2023-03-09T19:40:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:40:16 smithi181 ceph-mon[94189]: pgmap v9539: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 390 KiB/s wr, 28 op/s 2023-03-09T19:40:16.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:40:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16926.202942 secs 2023-03-09T19:40:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:40:16 smithi038 ceph-mon[92422]: pgmap v9539: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 390 KiB/s wr, 28 op/s 2023-03-09T19:40:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:40:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16928.510562 secs 2023-03-09T19:40:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:40:18 smithi122 ceph-mon[93869]: pgmap v9540: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 348 KiB/s wr, 24 op/s 2023-03-09T19:40:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:40:18 smithi122 ceph-mon[93869]: osdmap e810: 12 total, 12 up, 12 in 2023-03-09T19:40:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:40:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16928.510562 secs 2023-03-09T19:40:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:40:18 smithi181 ceph-mon[94189]: pgmap v9540: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 348 KiB/s wr, 24 op/s 2023-03-09T19:40:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:40:18 smithi181 ceph-mon[94189]: osdmap e810: 12 total, 12 up, 12 in 2023-03-09T19:40:18.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:40:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16928.510562 secs 2023-03-09T19:40:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:40:18 smithi038 ceph-mon[92422]: pgmap v9540: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 348 KiB/s wr, 24 op/s 2023-03-09T19:40:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:40:18 smithi038 ceph-mon[92422]: osdmap e810: 12 total, 12 up, 12 in 2023-03-09T19:40:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17390.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:40:20.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:40:20 smithi122 ceph-mon[93869]: pgmap v9542: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 225 KiB/s wr, 10 op/s 2023-03-09T19:40:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:40:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16931.203000 secs 2023-03-09T19:40:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:40:20 smithi181 ceph-mon[94189]: pgmap v9542: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 225 KiB/s wr, 10 op/s 2023-03-09T19:40:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:40:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16931.203000 secs 2023-03-09T19:40:20.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:40:20 smithi038 ceph-mon[92422]: pgmap v9542: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 225 KiB/s wr, 10 op/s 2023-03-09T19:40:20.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:40:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16931.203000 secs 2023-03-09T19:40:22.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:40:22 smithi122 ceph-mon[93869]: pgmap v9543: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 206 KiB/s wr, 9 op/s 2023-03-09T19:40:22.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:40:22 smithi181 ceph-mon[94189]: pgmap v9543: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 206 KiB/s wr, 9 op/s 2023-03-09T19:40:22.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:40:22 smithi038 ceph-mon[92422]: pgmap v9543: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 206 KiB/s wr, 9 op/s 2023-03-09T19:40:23.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:40:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16933.510681 secs 2023-03-09T19:40:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:40:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16933.510681 secs 2023-03-09T19:40:23.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:40:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16933.510681 secs 2023-03-09T19:40:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17395.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:40:24.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:40:24 smithi122 ceph-mon[93869]: pgmap v9544: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 148 KiB/s wr, 8 op/s 2023-03-09T19:40:24.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:40:24 smithi181 ceph-mon[94189]: pgmap v9544: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 148 KiB/s wr, 8 op/s 2023-03-09T19:40:24.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:40:24 smithi038 ceph-mon[92422]: pgmap v9544: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 148 KiB/s wr, 8 op/s 2023-03-09T19:40:25.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:40:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16936.203061 secs 2023-03-09T19:40:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:40:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16936.203061 secs 2023-03-09T19:40:25.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:40:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16936.203061 secs 2023-03-09T19:40:26.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:40:26 smithi122 ceph-mon[93869]: pgmap v9545: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 187 KiB/s wr, 8 op/s 2023-03-09T19:40:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:40:26 smithi181 ceph-mon[94189]: pgmap v9545: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 187 KiB/s wr, 8 op/s 2023-03-09T19:40:26.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:40:26 smithi038 ceph-mon[92422]: pgmap v9545: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 187 KiB/s wr, 8 op/s 2023-03-09T19:40:28.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:40:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16938.510775 secs 2023-03-09T19:40:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:40:28 smithi122 ceph-mon[93869]: pgmap v9546: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 187 KiB/s wr, 9 op/s 2023-03-09T19:40:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:40:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16938.510775 secs 2023-03-09T19:40:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:40:28 smithi181 ceph-mon[94189]: pgmap v9546: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 187 KiB/s wr, 9 op/s 2023-03-09T19:40:28.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:40:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16938.510775 secs 2023-03-09T19:40:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:40:28 smithi038 ceph-mon[92422]: pgmap v9546: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 187 KiB/s wr, 9 op/s 2023-03-09T19:40:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17400.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:40:30.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:40:30 smithi122 ceph-mon[93869]: pgmap v9547: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 100 KiB/s wr, 8 op/s 2023-03-09T19:40:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:40:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16941.203122 secs 2023-03-09T19:40:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:40:30 smithi181 ceph-mon[94189]: pgmap v9547: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 100 KiB/s wr, 8 op/s 2023-03-09T19:40:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:40:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16941.203122 secs 2023-03-09T19:40:30.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:40:30 smithi038 ceph-mon[92422]: pgmap v9547: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 100 KiB/s wr, 8 op/s 2023-03-09T19:40:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:40:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16941.203122 secs 2023-03-09T19:40:32.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:40:32 smithi122 ceph-mon[93869]: pgmap v9548: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 33 KiB/s wr, 9 op/s 2023-03-09T19:40:32.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:40:32 smithi181 ceph-mon[94189]: pgmap v9548: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 33 KiB/s wr, 9 op/s 2023-03-09T19:40:32.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:40:32 smithi038 ceph-mon[92422]: pgmap v9548: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 33 KiB/s wr, 9 op/s 2023-03-09T19:40:33.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:40:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16943.510892 secs 2023-03-09T19:40:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:40:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16943.510892 secs 2023-03-09T19:40:33.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:40:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16943.510892 secs 2023-03-09T19:40:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17405.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:40:34.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:40:34 smithi122 ceph-mon[93869]: pgmap v9549: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 33 KiB/s wr, 7 op/s 2023-03-09T19:40:34.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:40:34 smithi038 ceph-mon[92422]: pgmap v9549: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 33 KiB/s wr, 7 op/s 2023-03-09T19:40:34.750 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:40:34 smithi181 ceph-mon[94189]: pgmap v9549: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 33 KiB/s wr, 7 op/s 2023-03-09T19:40:35.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:40:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16946.203183 secs 2023-03-09T19:40:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:40:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16946.203183 secs 2023-03-09T19:40:35.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:40:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16946.203183 secs 2023-03-09T19:40:36.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:40:36 smithi122 ceph-mon[93869]: pgmap v9550: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 114 KiB/s wr, 7 op/s 2023-03-09T19:40:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:40:36 smithi181 ceph-mon[94189]: pgmap v9550: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 114 KiB/s wr, 7 op/s 2023-03-09T19:40:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:40:36 smithi038 ceph-mon[92422]: pgmap v9550: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 114 KiB/s wr, 7 op/s 2023-03-09T19:40:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:40:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16948.511013 secs 2023-03-09T19:40:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:40:38 smithi122 ceph-mon[93869]: pgmap v9551: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 82 KiB/s wr, 9 op/s 2023-03-09T19:40:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:40:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16948.511013 secs 2023-03-09T19:40:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:40:38 smithi181 ceph-mon[94189]: pgmap v9551: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 82 KiB/s wr, 9 op/s 2023-03-09T19:40:38.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:40:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16948.511013 secs 2023-03-09T19:40:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:40:38 smithi038 ceph-mon[92422]: pgmap v9551: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 82 KiB/s wr, 9 op/s 2023-03-09T19:40:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17410.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:40:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:40:40 smithi122 ceph-mon[93869]: pgmap v9552: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 126 KiB/s wr, 7 op/s 2023-03-09T19:40:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:40:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16951.203243 secs 2023-03-09T19:40:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:40:40 smithi181 ceph-mon[94189]: pgmap v9552: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 126 KiB/s wr, 7 op/s 2023-03-09T19:40:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:40:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16951.203243 secs 2023-03-09T19:40:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:40:40 smithi038 ceph-mon[92422]: pgmap v9552: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 126 KiB/s wr, 7 op/s 2023-03-09T19:40:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:40:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16951.203243 secs 2023-03-09T19:40:42.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:40:42 smithi122 ceph-mon[93869]: pgmap v9553: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 126 KiB/s wr, 8 op/s 2023-03-09T19:40:42.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:40:42 smithi181 ceph-mon[94189]: pgmap v9553: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 126 KiB/s wr, 8 op/s 2023-03-09T19:40:42.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:40:42 smithi038 ceph-mon[92422]: pgmap v9553: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 126 KiB/s wr, 8 op/s 2023-03-09T19:40:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:40:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16953.511150 secs 2023-03-09T19:40:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:40:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16953.511150 secs 2023-03-09T19:40:43.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:40:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16953.511150 secs 2023-03-09T19:40:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17415.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:40:44.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:40:44 smithi122 ceph-mon[93869]: pgmap v9554: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 126 KiB/s wr, 6 op/s 2023-03-09T19:40:44.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:40:44 smithi181 ceph-mon[94189]: pgmap v9554: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 126 KiB/s wr, 6 op/s 2023-03-09T19:40:44.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:40:44 smithi038 ceph-mon[92422]: pgmap v9554: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 126 KiB/s wr, 6 op/s 2023-03-09T19:40:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:40:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16956.203305 secs 2023-03-09T19:40:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:40:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16956.203305 secs 2023-03-09T19:40:45.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:40:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16956.203305 secs 2023-03-09T19:40:46.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:40:46 smithi122 ceph-mon[93869]: pgmap v9555: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 249 KiB/s wr, 7 op/s 2023-03-09T19:40:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:40:46 smithi181 ceph-mon[94189]: pgmap v9555: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 249 KiB/s wr, 7 op/s 2023-03-09T19:40:46.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:40:46 smithi038 ceph-mon[92422]: pgmap v9555: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 249 KiB/s wr, 7 op/s 2023-03-09T19:40:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:40:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16958.511266 secs 2023-03-09T19:40:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:40:48 smithi038 ceph-mon[92422]: pgmap v9556: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 181 KiB/s wr, 8 op/s 2023-03-09T19:40:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17420.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:40:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:40:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16958.511266 secs 2023-03-09T19:40:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:40:48 smithi122 ceph-mon[93869]: pgmap v9556: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 181 KiB/s wr, 8 op/s 2023-03-09T19:40:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:40:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16958.511266 secs 2023-03-09T19:40:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:40:48 smithi181 ceph-mon[94189]: pgmap v9556: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 181 KiB/s wr, 8 op/s 2023-03-09T19:40:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:40:50 smithi038 ceph-mon[92422]: pgmap v9557: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 181 KiB/s wr, 7 op/s 2023-03-09T19:40:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:40:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16961.203367 secs 2023-03-09T19:40:50.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:40:50 smithi122 ceph-mon[93869]: pgmap v9557: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 181 KiB/s wr, 7 op/s 2023-03-09T19:40:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:40:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16961.203367 secs 2023-03-09T19:40:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:40:50 smithi181 ceph-mon[94189]: pgmap v9557: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 181 KiB/s wr, 7 op/s 2023-03-09T19:40:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:40:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16961.203367 secs 2023-03-09T19:40:52.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:40:52 smithi038 ceph-mon[92422]: pgmap v9558: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 162 KiB/s wr, 10 op/s 2023-03-09T19:40:52.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:40:52 smithi122 ceph-mon[93869]: pgmap v9558: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 162 KiB/s wr, 10 op/s 2023-03-09T19:40:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:40:52 smithi181 ceph-mon[94189]: pgmap v9558: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 162 KiB/s wr, 10 op/s 2023-03-09T19:40:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:40:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16963.511332 secs 2023-03-09T19:40:53.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17425.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:40:53.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:40:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16963.511332 secs 2023-03-09T19:40:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:40:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16963.511332 secs 2023-03-09T19:40:54.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:40:54 smithi038 ceph-mon[92422]: pgmap v9559: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 162 KiB/s wr, 9 op/s 2023-03-09T19:40:54.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:40:54 smithi122 ceph-mon[93869]: pgmap v9559: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 162 KiB/s wr, 9 op/s 2023-03-09T19:40:54.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:40:54 smithi181 ceph-mon[94189]: pgmap v9559: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 162 KiB/s wr, 9 op/s 2023-03-09T19:40:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:40:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16966.203448 secs 2023-03-09T19:40:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:40:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16966.203448 secs 2023-03-09T19:40:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:40:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16966.203448 secs 2023-03-09T19:40:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:40:56 smithi122 ceph-mon[93869]: pgmap v9560: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 162 KiB/s wr, 9 op/s 2023-03-09T19:40:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:40:56 smithi181 ceph-mon[94189]: pgmap v9560: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 162 KiB/s wr, 9 op/s 2023-03-09T19:40:56.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:40:56 smithi038 ceph-mon[92422]: pgmap v9560: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 162 KiB/s wr, 9 op/s 2023-03-09T19:40:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17430.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:40:58.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:40:58 smithi122 ceph-mon[93869]: pgmap v9561: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 265 KiB/s wr, 12 op/s 2023-03-09T19:40:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:40:58 smithi181 ceph-mon[94189]: pgmap v9561: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 265 KiB/s wr, 12 op/s 2023-03-09T19:40:58.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:40:58 smithi038 ceph-mon[92422]: pgmap v9561: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 265 KiB/s wr, 12 op/s 2023-03-09T19:41:00.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:41:00 smithi122 ceph-mon[93869]: pgmap v9562: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 252 KiB/s wr, 10 op/s 2023-03-09T19:41:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:41:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16971.203508 secs 2023-03-09T19:41:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:41:00 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:41:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:41:00 smithi181 ceph-mon[94189]: pgmap v9562: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 252 KiB/s wr, 10 op/s 2023-03-09T19:41:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:41:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16971.203508 secs 2023-03-09T19:41:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:41:00 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:41:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:41:00 smithi038 ceph-mon[92422]: pgmap v9562: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 252 KiB/s wr, 10 op/s 2023-03-09T19:41:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:41:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16971.203508 secs 2023-03-09T19:41:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:41:00 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:41:01.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:41:01 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:41:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:41:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:41:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:41:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:41:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:41:01 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:41:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:41:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:41:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:41:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:41:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:41:01 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:41:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:41:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:41:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:41:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:41:02.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:41:02 smithi122 ceph-mon[93869]: pgmap v9563: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 370 KiB/s wr, 13 op/s 2023-03-09T19:41:02.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:41:02 smithi181 ceph-mon[94189]: pgmap v9563: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 370 KiB/s wr, 13 op/s 2023-03-09T19:41:02.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:41:02 smithi038 ceph-mon[92422]: pgmap v9563: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 370 KiB/s wr, 13 op/s 2023-03-09T19:41:03.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17435.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:41:04.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:41:04 smithi122 ceph-mon[93869]: pgmap v9564: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 357 KiB/s wr, 21 op/s 2023-03-09T19:41:04.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:41:04 smithi181 ceph-mon[94189]: pgmap v9564: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 357 KiB/s wr, 21 op/s 2023-03-09T19:41:04.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:41:04 smithi038 ceph-mon[92422]: pgmap v9564: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 357 KiB/s wr, 21 op/s 2023-03-09T19:41:06.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:41:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16976.203602 secs 2023-03-09T19:41:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:41:06 smithi122 ceph-mon[93869]: pgmap v9565: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 531 KiB/s wr, 22 op/s 2023-03-09T19:41:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:41:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16976.203602 secs 2023-03-09T19:41:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:41:06 smithi181 ceph-mon[94189]: pgmap v9565: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 531 KiB/s wr, 22 op/s 2023-03-09T19:41:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:41:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16976.203602 secs 2023-03-09T19:41:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:41:06 smithi038 ceph-mon[92422]: pgmap v9565: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 531 KiB/s wr, 22 op/s 2023-03-09T19:41:08.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:41:08 smithi122 ceph-mon[93869]: pgmap v9566: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 669 KiB/s rd, 557 KiB/s wr, 26 op/s 2023-03-09T19:41:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:41:08 smithi181 ceph-mon[94189]: pgmap v9566: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 669 KiB/s rd, 557 KiB/s wr, 26 op/s 2023-03-09T19:41:08.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:41:08 smithi038 ceph-mon[92422]: pgmap v9566: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 669 KiB/s rd, 557 KiB/s wr, 26 op/s 2023-03-09T19:41:08.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17440.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:41:10.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:41:10 smithi122 ceph-mon[93869]: pgmap v9567: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 330 KiB/s wr, 22 op/s 2023-03-09T19:41:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:41:10 smithi181 ceph-mon[94189]: pgmap v9567: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 330 KiB/s wr, 22 op/s 2023-03-09T19:41:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:41:10 smithi038 ceph-mon[92422]: pgmap v9567: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 330 KiB/s wr, 22 op/s 2023-03-09T19:41:11.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:41:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16981.203666 secs 2023-03-09T19:41:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:41:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16981.203666 secs 2023-03-09T19:41:11.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:41:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16981.203666 secs 2023-03-09T19:41:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:41:12 smithi122 ceph-mon[93869]: pgmap v9568: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 560 KiB/s rd, 450 KiB/s wr, 25 op/s 2023-03-09T19:41:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:41:12 smithi181 ceph-mon[94189]: pgmap v9568: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 560 KiB/s rd, 450 KiB/s wr, 25 op/s 2023-03-09T19:41:12.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:41:12 smithi038 ceph-mon[92422]: pgmap v9568: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 560 KiB/s rd, 450 KiB/s wr, 25 op/s 2023-03-09T19:41:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17445.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:41:14.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:41:14 smithi122 ceph-mon[93869]: pgmap v9569: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 332 KiB/s wr, 22 op/s 2023-03-09T19:41:14.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:41:14 smithi181 ceph-mon[94189]: pgmap v9569: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 332 KiB/s wr, 22 op/s 2023-03-09T19:41:14.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:41:14 smithi038 ceph-mon[92422]: pgmap v9569: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 332 KiB/s wr, 22 op/s 2023-03-09T19:41:16.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:41:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16986.203726 secs 2023-03-09T19:41:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:41:16 smithi122 ceph-mon[93869]: pgmap v9570: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 381 KiB/s wr, 11 op/s 2023-03-09T19:41:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:41:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16986.203726 secs 2023-03-09T19:41:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:41:16 smithi181 ceph-mon[94189]: pgmap v9570: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 381 KiB/s wr, 11 op/s 2023-03-09T19:41:16.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:41:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16986.203726 secs 2023-03-09T19:41:16.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:41:16 smithi038 ceph-mon[92422]: pgmap v9570: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 381 KiB/s wr, 11 op/s 2023-03-09T19:41:18.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:41:18 smithi122 ceph-mon[93869]: pgmap v9571: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 208 KiB/s wr, 12 op/s 2023-03-09T19:41:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:41:18 smithi181 ceph-mon[94189]: pgmap v9571: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 208 KiB/s wr, 12 op/s 2023-03-09T19:41:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:41:18 smithi038 ceph-mon[92422]: pgmap v9571: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 208 KiB/s wr, 12 op/s 2023-03-09T19:41:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17450.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:41:19.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:41:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:41:19.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:41:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:41:19.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:41:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:41:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:41:20 smithi122 ceph-mon[93869]: pgmap v9572: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 183 KiB/s wr, 7 op/s 2023-03-09T19:41:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:41:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:41:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:41:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:41:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:41:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:41:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:41:20 smithi181 ceph-mon[94189]: pgmap v9572: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 183 KiB/s wr, 7 op/s 2023-03-09T19:41:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:41:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:41:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:41:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:41:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:41:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:41:20.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:41:20 smithi038 ceph-mon[92422]: pgmap v9572: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 183 KiB/s wr, 7 op/s 2023-03-09T19:41:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:41:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:41:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:41:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:41:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:41:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:41:21.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:41:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16991.203794 secs 2023-03-09T19:41:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:41:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16991.203794 secs 2023-03-09T19:41:21.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:41:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16991.203794 secs 2023-03-09T19:41:22.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:41:22 smithi122 ceph-mon[93869]: pgmap v9573: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 208 KiB/s wr, 11 op/s 2023-03-09T19:41:22.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:41:22 smithi181 ceph-mon[94189]: pgmap v9573: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 208 KiB/s wr, 11 op/s 2023-03-09T19:41:22.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:41:22 smithi038 ceph-mon[92422]: pgmap v9573: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 208 KiB/s wr, 11 op/s 2023-03-09T19:41:23.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:41:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16993.512035 secs 2023-03-09T19:41:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:41:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16993.512035 secs 2023-03-09T19:41:23.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:41:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16993.512035 secs 2023-03-09T19:41:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17455.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:41:24.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:41:24 smithi122 ceph-mon[93869]: pgmap v9574: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 98 KiB/s wr, 9 op/s 2023-03-09T19:41:24.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:41:24 smithi181 ceph-mon[94189]: pgmap v9574: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 98 KiB/s wr, 9 op/s 2023-03-09T19:41:24.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:41:24 smithi038 ceph-mon[92422]: pgmap v9574: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 98 KiB/s wr, 9 op/s 2023-03-09T19:41:25.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:41:25 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:41:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:41:25 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:41:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:41:25 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:41:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:41:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16996.203853 secs 2023-03-09T19:41:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:41:25 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:41:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:41:25 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:41:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:41:25 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:41:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:41:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16996.203853 secs 2023-03-09T19:41:25.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:41:25 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:41:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:41:25 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:41:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:41:25 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:41:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:41:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16996.203853 secs 2023-03-09T19:41:26.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:41:26 smithi122 ceph-mon[93869]: pgmap v9575: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 166 KiB/s wr, 10 op/s 2023-03-09T19:41:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:41:26 smithi181 ceph-mon[94189]: pgmap v9575: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 166 KiB/s wr, 10 op/s 2023-03-09T19:41:26.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:41:26 smithi038 ceph-mon[92422]: pgmap v9575: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 166 KiB/s wr, 10 op/s 2023-03-09T19:41:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:41:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 16998.512158 secs 2023-03-09T19:41:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:41:28 smithi122 ceph-mon[93869]: pgmap v9576: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 108 KiB/s wr, 13 op/s 2023-03-09T19:41:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:41:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 16998.512158 secs 2023-03-09T19:41:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:41:28 smithi181 ceph-mon[94189]: pgmap v9576: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 108 KiB/s wr, 13 op/s 2023-03-09T19:41:28.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:41:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 16998.512158 secs 2023-03-09T19:41:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:41:28 smithi038 ceph-mon[92422]: pgmap v9576: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 108 KiB/s wr, 13 op/s 2023-03-09T19:41:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17460.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:41:30.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:41:30 smithi122 ceph-mon[93869]: pgmap v9577: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 435 KiB/s rd, 253 KiB/s wr, 11 op/s 2023-03-09T19:41:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:41:30 smithi122 ceph-mon[93869]: osdmap e811: 12 total, 12 up, 12 in 2023-03-09T19:41:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:41:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17001.203915 secs 2023-03-09T19:41:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:41:30 smithi181 ceph-mon[94189]: pgmap v9577: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 435 KiB/s rd, 253 KiB/s wr, 11 op/s 2023-03-09T19:41:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:41:30 smithi181 ceph-mon[94189]: osdmap e811: 12 total, 12 up, 12 in 2023-03-09T19:41:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:41:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17001.203915 secs 2023-03-09T19:41:30.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:41:30 smithi038 ceph-mon[92422]: pgmap v9577: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 435 KiB/s rd, 253 KiB/s wr, 11 op/s 2023-03-09T19:41:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:41:30 smithi038 ceph-mon[92422]: osdmap e811: 12 total, 12 up, 12 in 2023-03-09T19:41:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:41:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17001.203915 secs 2023-03-09T19:41:32.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:41:32 smithi122 ceph-mon[93869]: pgmap v9579: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 276 KiB/s wr, 12 op/s 2023-03-09T19:41:32.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:41:32 smithi181 ceph-mon[94189]: pgmap v9579: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 276 KiB/s wr, 12 op/s 2023-03-09T19:41:32.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:41:32 smithi038 ceph-mon[92422]: pgmap v9579: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 276 KiB/s wr, 12 op/s 2023-03-09T19:41:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:41:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17003.512270 secs 2023-03-09T19:41:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:41:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17003.512270 secs 2023-03-09T19:41:33.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:41:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17003.512270 secs 2023-03-09T19:41:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17465.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:41:34.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:41:34 smithi122 ceph-mon[93869]: pgmap v9580: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 263 KiB/s wr, 11 op/s 2023-03-09T19:41:34.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:41:34 smithi181 ceph-mon[94189]: pgmap v9580: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 263 KiB/s wr, 11 op/s 2023-03-09T19:41:34.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:41:34 smithi038 ceph-mon[92422]: pgmap v9580: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 263 KiB/s wr, 11 op/s 2023-03-09T19:41:35.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:41:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17006.203979 secs 2023-03-09T19:41:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:41:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17006.203979 secs 2023-03-09T19:41:35.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:41:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17006.203979 secs 2023-03-09T19:41:36.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:41:36 smithi122 ceph-mon[93869]: pgmap v9581: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 221 KiB/s wr, 10 op/s 2023-03-09T19:41:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:41:36 smithi181 ceph-mon[94189]: pgmap v9581: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 221 KiB/s wr, 10 op/s 2023-03-09T19:41:36.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:41:36 smithi038 ceph-mon[92422]: pgmap v9581: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 221 KiB/s wr, 10 op/s 2023-03-09T19:41:38.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:41:38 smithi122 ceph-mon[93869]: pgmap v9582: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 220 KiB/s wr, 10 op/s 2023-03-09T19:41:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:41:38 smithi122 ceph-mon[93869]: osdmap e812: 12 total, 12 up, 12 in 2023-03-09T19:41:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:41:38 smithi181 ceph-mon[94189]: pgmap v9582: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 220 KiB/s wr, 10 op/s 2023-03-09T19:41:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:41:38 smithi181 ceph-mon[94189]: osdmap e812: 12 total, 12 up, 12 in 2023-03-09T19:41:38.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:41:38 smithi038 ceph-mon[92422]: pgmap v9582: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 220 KiB/s wr, 10 op/s 2023-03-09T19:41:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:41:38 smithi038 ceph-mon[92422]: osdmap e812: 12 total, 12 up, 12 in 2023-03-09T19:41:38.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17470.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:41:40.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:41:40 smithi122 ceph-mon[93869]: pgmap v9584: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 93 KiB/s wr, 9 op/s 2023-03-09T19:41:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:41:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17011.204041 secs 2023-03-09T19:41:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:41:40 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:41:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:41:40 smithi181 ceph-mon[94189]: pgmap v9584: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 93 KiB/s wr, 9 op/s 2023-03-09T19:41:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:41:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17011.204041 secs 2023-03-09T19:41:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:41:40 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:41:40.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:41:40 smithi038 ceph-mon[92422]: pgmap v9584: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 93 KiB/s wr, 9 op/s 2023-03-09T19:41:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:41:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17011.204041 secs 2023-03-09T19:41:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:41:40 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:41:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:41:41 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:41:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:41:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:41:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:41:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:41:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:41:41 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:41:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:41:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:41:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:41:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:41:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:41:41 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:41:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:41:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:41:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:41:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:41:42.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:41:42 smithi122 ceph-mon[93869]: pgmap v9585: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 106 KiB/s wr, 8 op/s 2023-03-09T19:41:42.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:41:42 smithi181 ceph-mon[94189]: pgmap v9585: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 106 KiB/s wr, 8 op/s 2023-03-09T19:41:42.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:41:42 smithi038 ceph-mon[92422]: pgmap v9585: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 106 KiB/s wr, 8 op/s 2023-03-09T19:41:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17475.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:41:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:41:44 smithi122 ceph-mon[93869]: pgmap v9586: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 183 KiB/s wr, 9 op/s 2023-03-09T19:41:44.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:41:44 smithi181 ceph-mon[94189]: pgmap v9586: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 183 KiB/s wr, 9 op/s 2023-03-09T19:41:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:41:44 smithi038 ceph-mon[92422]: pgmap v9586: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 183 KiB/s wr, 9 op/s 2023-03-09T19:41:46.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:41:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17016.204102 secs 2023-03-09T19:41:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:41:46 smithi122 ceph-mon[93869]: pgmap v9587: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 209 KiB/s wr, 9 op/s 2023-03-09T19:41:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:41:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17016.204102 secs 2023-03-09T19:41:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:41:46 smithi181 ceph-mon[94189]: pgmap v9587: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 209 KiB/s wr, 9 op/s 2023-03-09T19:41:46.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:41:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17016.204102 secs 2023-03-09T19:41:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:41:46 smithi038 ceph-mon[92422]: pgmap v9587: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 209 KiB/s wr, 9 op/s 2023-03-09T19:41:48.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:41:48 smithi122 ceph-mon[93869]: pgmap v9588: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 301 KiB/s wr, 11 op/s 2023-03-09T19:41:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:41:48 smithi181 ceph-mon[94189]: pgmap v9588: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 301 KiB/s wr, 11 op/s 2023-03-09T19:41:48.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:41:48 smithi038 ceph-mon[92422]: pgmap v9588: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 301 KiB/s wr, 11 op/s 2023-03-09T19:41:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17480.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:41:50.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:41:50 smithi122 ceph-mon[93869]: pgmap v9589: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 310 KiB/s rd, 270 KiB/s wr, 11 op/s 2023-03-09T19:41:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:41:50 smithi181 ceph-mon[94189]: pgmap v9589: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 310 KiB/s rd, 270 KiB/s wr, 11 op/s 2023-03-09T19:41:50.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:41:50 smithi038 ceph-mon[92422]: pgmap v9589: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 310 KiB/s rd, 270 KiB/s wr, 11 op/s 2023-03-09T19:41:51.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:41:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17021.204164 secs 2023-03-09T19:41:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:41:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17021.204164 secs 2023-03-09T19:41:51.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:41:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17021.204164 secs 2023-03-09T19:41:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:41:52 smithi122 ceph-mon[93869]: pgmap v9590: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 225 KiB/s wr, 12 op/s 2023-03-09T19:41:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:41:52 smithi181 ceph-mon[94189]: pgmap v9590: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 225 KiB/s wr, 12 op/s 2023-03-09T19:41:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:41:52 smithi038 ceph-mon[92422]: pgmap v9590: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 225 KiB/s wr, 12 op/s 2023-03-09T19:41:53.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:41:53 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 17023.512750 secs 2023-03-09T19:41:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:41:53 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 17023.512750 secs 2023-03-09T19:41:53.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:41:53 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 17023.512750 secs 2023-03-09T19:41:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17485.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:41:54.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:41:54 smithi122 ceph-mon[93869]: pgmap v9591: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 371 KiB/s wr, 11 op/s 2023-03-09T19:41:54.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:41:54 smithi181 ceph-mon[94189]: pgmap v9591: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 371 KiB/s wr, 11 op/s 2023-03-09T19:41:54.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:41:54 smithi038 ceph-mon[92422]: pgmap v9591: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 371 KiB/s wr, 11 op/s 2023-03-09T19:41:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:41:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17026.204225 secs 2023-03-09T19:41:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:41:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17026.204225 secs 2023-03-09T19:41:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:41:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17026.204225 secs 2023-03-09T19:41:56.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:41:56 smithi122 ceph-mon[93869]: pgmap v9592: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 379 KiB/s wr, 11 op/s 2023-03-09T19:41:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:41:56 smithi181 ceph-mon[94189]: pgmap v9592: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 379 KiB/s wr, 11 op/s 2023-03-09T19:41:56.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:41:56 smithi038 ceph-mon[92422]: pgmap v9592: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 379 KiB/s wr, 11 op/s 2023-03-09T19:41:57.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:41:57 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:41:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:41:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:41:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:41:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:41:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:41:57 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:41:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:41:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:41:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:41:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:41:57.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:41:57 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:41:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:41:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:41:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:41:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:41:58.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:41:58 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 17028.512848 secs 2023-03-09T19:41:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:41:58 smithi122 ceph-mon[93869]: pgmap v9593: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 392 KiB/s wr, 14 op/s 2023-03-09T19:41:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:41:58 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 17028.512848 secs 2023-03-09T19:41:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:41:58 smithi181 ceph-mon[94189]: pgmap v9593: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 392 KiB/s wr, 14 op/s 2023-03-09T19:41:58.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:41:58 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 17028.512848 secs 2023-03-09T19:41:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:41:58 smithi038 ceph-mon[92422]: pgmap v9593: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 392 KiB/s wr, 14 op/s 2023-03-09T19:41:58.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17490.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:42:00.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:42:00 smithi122 ceph-mon[93869]: pgmap v9594: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 314 KiB/s wr, 10 op/s 2023-03-09T19:42:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:42:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17031.204305 secs 2023-03-09T19:42:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:42:00 smithi181 ceph-mon[94189]: pgmap v9594: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 314 KiB/s wr, 10 op/s 2023-03-09T19:42:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:42:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17031.204305 secs 2023-03-09T19:42:00.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:42:00 smithi038 ceph-mon[92422]: pgmap v9594: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 314 KiB/s wr, 10 op/s 2023-03-09T19:42:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:42:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17031.204305 secs 2023-03-09T19:42:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:42:02 smithi122 ceph-mon[93869]: pgmap v9595: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 445 KiB/s wr, 15 op/s 2023-03-09T19:42:02.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:42:02 smithi181 ceph-mon[94189]: pgmap v9595: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 445 KiB/s wr, 15 op/s 2023-03-09T19:42:02.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:42:02 smithi038 ceph-mon[92422]: pgmap v9595: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 445 KiB/s wr, 15 op/s 2023-03-09T19:42:03.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:42:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17033.512969 secs 2023-03-09T19:42:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:42:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17033.512969 secs 2023-03-09T19:42:03.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:42:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17033.512969 secs 2023-03-09T19:42:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17495.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:42:04.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:42:04 smithi122 ceph-mon[93869]: pgmap v9596: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 452 KiB/s wr, 25 op/s 2023-03-09T19:42:04.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:42:04 smithi181 ceph-mon[94189]: pgmap v9596: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 452 KiB/s wr, 25 op/s 2023-03-09T19:42:04.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:42:04 smithi038 ceph-mon[92422]: pgmap v9596: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 452 KiB/s wr, 25 op/s 2023-03-09T19:42:05.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:42:05 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:42:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:42:05 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:42:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:42:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17036.204365 secs 2023-03-09T19:42:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:42:05 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:42:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:42:05 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:42:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:42:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17036.204365 secs 2023-03-09T19:42:05.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:42:05 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:42:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:42:05 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:42:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:42:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17036.204365 secs 2023-03-09T19:42:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:42:06 smithi122 ceph-mon[93869]: pgmap v9597: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 343 KiB/s wr, 24 op/s 2023-03-09T19:42:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:42:06 smithi181 ceph-mon[94189]: pgmap v9597: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 343 KiB/s wr, 24 op/s 2023-03-09T19:42:06.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:42:06 smithi038 ceph-mon[92422]: pgmap v9597: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 343 KiB/s wr, 24 op/s 2023-03-09T19:42:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:42:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17038.513085 secs 2023-03-09T19:42:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:42:08 smithi122 ceph-mon[93869]: pgmap v9598: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 293 KiB/s rd, 271 KiB/s wr, 26 op/s 2023-03-09T19:42:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:42:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17038.513085 secs 2023-03-09T19:42:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:42:08 smithi181 ceph-mon[94189]: pgmap v9598: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 293 KiB/s rd, 271 KiB/s wr, 26 op/s 2023-03-09T19:42:08.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:42:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17038.513085 secs 2023-03-09T19:42:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:42:08 smithi038 ceph-mon[92422]: pgmap v9598: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 293 KiB/s rd, 271 KiB/s wr, 26 op/s 2023-03-09T19:42:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17500.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:42:10.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:42:10 smithi038 ceph-mon[92422]: pgmap v9599: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 203 KiB/s wr, 22 op/s 2023-03-09T19:42:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:42:10 smithi038 ceph-mon[92422]: osdmap e813: 12 total, 12 up, 12 in 2023-03-09T19:42:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:42:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17041.204424 secs 2023-03-09T19:42:10.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:42:10 smithi122 ceph-mon[93869]: pgmap v9599: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 203 KiB/s wr, 22 op/s 2023-03-09T19:42:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:42:10 smithi122 ceph-mon[93869]: osdmap e813: 12 total, 12 up, 12 in 2023-03-09T19:42:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:42:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17041.204424 secs 2023-03-09T19:42:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:42:10 smithi181 ceph-mon[94189]: pgmap v9599: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 203 KiB/s wr, 22 op/s 2023-03-09T19:42:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:42:10 smithi181 ceph-mon[94189]: osdmap e813: 12 total, 12 up, 12 in 2023-03-09T19:42:10.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:42:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17041.204424 secs 2023-03-09T19:42:12.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:42:12 smithi038 ceph-mon[92422]: pgmap v9601: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 103 KiB/s wr, 23 op/s 2023-03-09T19:42:12.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:42:12 smithi122 ceph-mon[93869]: pgmap v9601: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 103 KiB/s wr, 23 op/s 2023-03-09T19:42:12.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:42:12 smithi181 ceph-mon[94189]: pgmap v9601: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 103 KiB/s wr, 23 op/s 2023-03-09T19:42:13.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:42:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17043.513197 secs 2023-03-09T19:42:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17505.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:42:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:42:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17043.513197 secs 2023-03-09T19:42:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:42:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17043.513197 secs 2023-03-09T19:42:14.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:42:14 smithi038 ceph-mon[92422]: pgmap v9602: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 77 KiB/s wr, 8 op/s 2023-03-09T19:42:14.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:42:14 smithi122 ceph-mon[93869]: pgmap v9602: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 77 KiB/s wr, 8 op/s 2023-03-09T19:42:14.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:42:14 smithi181 ceph-mon[94189]: pgmap v9602: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 77 KiB/s wr, 8 op/s 2023-03-09T19:42:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:42:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17046.204484 secs 2023-03-09T19:42:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:42:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17046.204484 secs 2023-03-09T19:42:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:42:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17046.204484 secs 2023-03-09T19:42:16.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:42:16 smithi122 ceph-mon[93869]: pgmap v9603: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 93 KiB/s wr, 8 op/s 2023-03-09T19:42:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:42:16 smithi181 ceph-mon[94189]: pgmap v9603: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 93 KiB/s wr, 8 op/s 2023-03-09T19:42:16.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:42:16 smithi038 ceph-mon[92422]: pgmap v9603: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 93 KiB/s wr, 8 op/s 2023-03-09T19:42:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17510.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:42:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:42:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17048.513327 secs 2023-03-09T19:42:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:42:18 smithi122 ceph-mon[93869]: pgmap v9604: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 109 KiB/s wr, 8 op/s 2023-03-09T19:42:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:42:18 smithi122 ceph-mon[93869]: osdmap e814: 12 total, 12 up, 12 in 2023-03-09T19:42:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:42:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17048.513327 secs 2023-03-09T19:42:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:42:18 smithi181 ceph-mon[94189]: pgmap v9604: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 109 KiB/s wr, 8 op/s 2023-03-09T19:42:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:42:18 smithi181 ceph-mon[94189]: osdmap e814: 12 total, 12 up, 12 in 2023-03-09T19:42:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:42:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17048.513327 secs 2023-03-09T19:42:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:42:18 smithi038 ceph-mon[92422]: pgmap v9604: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 109 KiB/s wr, 8 op/s 2023-03-09T19:42:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:42:18 smithi038 ceph-mon[92422]: osdmap e814: 12 total, 12 up, 12 in 2023-03-09T19:42:20.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:42:20 smithi122 ceph-mon[93869]: pgmap v9606: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 110 KiB/s wr, 8 op/s 2023-03-09T19:42:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:42:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17051.204543 secs 2023-03-09T19:42:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:42:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:42:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:42:20 smithi181 ceph-mon[94189]: pgmap v9606: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 110 KiB/s wr, 8 op/s 2023-03-09T19:42:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:42:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17051.204543 secs 2023-03-09T19:42:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:42:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:42:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:42:20 smithi038 ceph-mon[92422]: pgmap v9606: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 110 KiB/s wr, 8 op/s 2023-03-09T19:42:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:42:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17051.204543 secs 2023-03-09T19:42:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:42:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:42:21.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:42:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:42:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:42:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:42:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:42:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:42:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:42:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:42:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:42:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:42:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:42:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:42:21.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:42:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:42:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:42:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:42:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:42:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:42:22.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:42:22 smithi122 ceph-mon[93869]: pgmap v9607: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 94 KiB/s wr, 7 op/s 2023-03-09T19:42:22.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:42:22 smithi181 ceph-mon[94189]: pgmap v9607: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 94 KiB/s wr, 7 op/s 2023-03-09T19:42:22.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:42:22 smithi038 ceph-mon[92422]: pgmap v9607: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 94 KiB/s wr, 7 op/s 2023-03-09T19:42:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17515.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:42:23.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:42:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17053.513410 secs 2023-03-09T19:42:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:42:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17053.513410 secs 2023-03-09T19:42:23.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:42:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17053.513410 secs 2023-03-09T19:42:24.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:42:24 smithi122 ceph-mon[93869]: pgmap v9608: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 94 KiB/s wr, 8 op/s 2023-03-09T19:42:24.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:42:24 smithi181 ceph-mon[94189]: pgmap v9608: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 94 KiB/s wr, 8 op/s 2023-03-09T19:42:24.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:42:24 smithi038 ceph-mon[92422]: pgmap v9608: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 94 KiB/s wr, 8 op/s 2023-03-09T19:42:25.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:42:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17056.204603 secs 2023-03-09T19:42:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:42:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17056.204603 secs 2023-03-09T19:42:25.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:42:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17056.204603 secs 2023-03-09T19:42:26.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:42:26 smithi122 ceph-mon[93869]: pgmap v9609: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 92 KiB/s wr, 7 op/s 2023-03-09T19:42:26.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:42:26 smithi181 ceph-mon[94189]: pgmap v9609: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 92 KiB/s wr, 7 op/s 2023-03-09T19:42:26.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:42:26 smithi038 ceph-mon[92422]: pgmap v9609: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 92 KiB/s wr, 7 op/s 2023-03-09T19:42:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17520.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:42:28.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:42:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17058.513490 secs 2023-03-09T19:42:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:42:28 smithi122 ceph-mon[93869]: pgmap v9610: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 114 KiB/s wr, 8 op/s 2023-03-09T19:42:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:42:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17058.513490 secs 2023-03-09T19:42:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:42:28 smithi181 ceph-mon[94189]: pgmap v9610: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 114 KiB/s wr, 8 op/s 2023-03-09T19:42:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:42:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17058.513490 secs 2023-03-09T19:42:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:42:28 smithi038 ceph-mon[92422]: pgmap v9610: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 114 KiB/s wr, 8 op/s 2023-03-09T19:42:30.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:42:30 smithi122 ceph-mon[93869]: pgmap v9611: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 102 KiB/s wr, 7 op/s 2023-03-09T19:42:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:42:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17061.204687 secs 2023-03-09T19:42:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:42:30 smithi181 ceph-mon[94189]: pgmap v9611: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 102 KiB/s wr, 7 op/s 2023-03-09T19:42:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:42:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17061.204687 secs 2023-03-09T19:42:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:42:30 smithi038 ceph-mon[92422]: pgmap v9611: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 102 KiB/s wr, 7 op/s 2023-03-09T19:42:30.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:42:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17061.204687 secs 2023-03-09T19:42:32.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:42:32 smithi122 ceph-mon[93869]: pgmap v9612: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 96 KiB/s wr, 10 op/s 2023-03-09T19:42:32.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:42:32 smithi181 ceph-mon[94189]: pgmap v9612: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 96 KiB/s wr, 10 op/s 2023-03-09T19:42:32.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:42:32 smithi038 ceph-mon[92422]: pgmap v9612: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 96 KiB/s wr, 10 op/s 2023-03-09T19:42:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17525.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:42:33.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:42:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17063.513623 secs 2023-03-09T19:42:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:42:33 smithi122 ceph-mon[93869]: pgmap v9613: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 95 KiB/s wr, 8 op/s 2023-03-09T19:42:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:42:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17063.513623 secs 2023-03-09T19:42:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:42:33 smithi181 ceph-mon[94189]: pgmap v9613: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 95 KiB/s wr, 8 op/s 2023-03-09T19:42:33.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:42:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17063.513623 secs 2023-03-09T19:42:33.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:42:33 smithi038 ceph-mon[92422]: pgmap v9613: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 95 KiB/s wr, 8 op/s 2023-03-09T19:42:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:42:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17066.204746 secs 2023-03-09T19:42:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:42:36 smithi122 ceph-mon[93869]: pgmap v9614: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 94 KiB/s wr, 8 op/s 2023-03-09T19:42:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:42:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17066.204746 secs 2023-03-09T19:42:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:42:36 smithi181 ceph-mon[94189]: pgmap v9614: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 94 KiB/s wr, 8 op/s 2023-03-09T19:42:36.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:42:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17066.204746 secs 2023-03-09T19:42:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:42:36 smithi038 ceph-mon[92422]: pgmap v9614: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 94 KiB/s wr, 8 op/s 2023-03-09T19:42:38.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:42:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17068.513734 secs 2023-03-09T19:42:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:42:38 smithi122 ceph-mon[93869]: pgmap v9615: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 184 KiB/s wr, 10 op/s 2023-03-09T19:42:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:42:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17068.513734 secs 2023-03-09T19:42:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:42:38 smithi181 ceph-mon[94189]: pgmap v9615: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 184 KiB/s wr, 10 op/s 2023-03-09T19:42:38.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:42:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17068.513734 secs 2023-03-09T19:42:38.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:42:38 smithi038 ceph-mon[92422]: pgmap v9615: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 184 KiB/s wr, 10 op/s 2023-03-09T19:42:38.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17530.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:42:40.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:42:40 smithi122 ceph-mon[93869]: pgmap v9616: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 153 KiB/s wr, 7 op/s 2023-03-09T19:42:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:42:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17071.204804 secs 2023-03-09T19:42:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:42:40 smithi181 ceph-mon[94189]: pgmap v9616: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 153 KiB/s wr, 7 op/s 2023-03-09T19:42:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:42:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17071.204804 secs 2023-03-09T19:42:40.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:42:40 smithi038 ceph-mon[92422]: pgmap v9616: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 153 KiB/s wr, 7 op/s 2023-03-09T19:42:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:42:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17071.204804 secs 2023-03-09T19:42:42.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:42:42 smithi122 ceph-mon[93869]: pgmap v9617: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 182 KiB/s wr, 11 op/s 2023-03-09T19:42:42.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:42:42 smithi181 ceph-mon[94189]: pgmap v9617: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 182 KiB/s wr, 11 op/s 2023-03-09T19:42:42.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:42:42 smithi038 ceph-mon[92422]: pgmap v9617: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 182 KiB/s wr, 11 op/s 2023-03-09T19:42:43.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:42:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17073.513865 secs 2023-03-09T19:42:43.792 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:42:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17073.513865 secs 2023-03-09T19:42:43.792 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:42:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17073.513865 secs 2023-03-09T19:42:43.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17535.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:42:44.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:42:44 smithi122 ceph-mon[93869]: pgmap v9618: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 182 KiB/s wr, 8 op/s 2023-03-09T19:42:44.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:42:44 smithi181 ceph-mon[94189]: pgmap v9618: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 182 KiB/s wr, 8 op/s 2023-03-09T19:42:44.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:42:44 smithi038 ceph-mon[92422]: pgmap v9618: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 182 KiB/s wr, 8 op/s 2023-03-09T19:42:45.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:42:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17076.204863 secs 2023-03-09T19:42:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:42:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17076.204863 secs 2023-03-09T19:42:45.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:42:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17076.204863 secs 2023-03-09T19:42:46.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:42:46 smithi122 ceph-mon[93869]: pgmap v9619: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 182 KiB/s wr, 8 op/s 2023-03-09T19:42:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:42:46 smithi181 ceph-mon[94189]: pgmap v9619: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 182 KiB/s wr, 8 op/s 2023-03-09T19:42:46.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:42:46 smithi038 ceph-mon[92422]: pgmap v9619: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 182 KiB/s wr, 8 op/s 2023-03-09T19:42:48.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:42:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17078.513951 secs 2023-03-09T19:42:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:42:48 smithi122 ceph-mon[93869]: pgmap v9620: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 338 KiB/s wr, 11 op/s 2023-03-09T19:42:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:42:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17078.513951 secs 2023-03-09T19:42:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:42:48 smithi181 ceph-mon[94189]: pgmap v9620: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 338 KiB/s wr, 11 op/s 2023-03-09T19:42:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:42:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17078.513951 secs 2023-03-09T19:42:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:42:48 smithi038 ceph-mon[92422]: pgmap v9620: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 338 KiB/s wr, 11 op/s 2023-03-09T19:42:48.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17540.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:42:50.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:42:50 smithi122 ceph-mon[93869]: pgmap v9621: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 187 KiB/s wr, 9 op/s 2023-03-09T19:42:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:42:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17081.204922 secs 2023-03-09T19:42:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:42:50 smithi181 ceph-mon[94189]: pgmap v9621: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 187 KiB/s wr, 9 op/s 2023-03-09T19:42:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:42:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17081.204922 secs 2023-03-09T19:42:50.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:42:50 smithi038 ceph-mon[92422]: pgmap v9621: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 187 KiB/s wr, 9 op/s 2023-03-09T19:42:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:42:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17081.204922 secs 2023-03-09T19:42:52.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:42:52 smithi122 ceph-mon[93869]: pgmap v9622: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 201 KiB/s wr, 12 op/s 2023-03-09T19:42:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:42:52 smithi181 ceph-mon[94189]: pgmap v9622: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 201 KiB/s wr, 12 op/s 2023-03-09T19:42:52.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:42:52 smithi038 ceph-mon[92422]: pgmap v9622: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 201 KiB/s wr, 12 op/s 2023-03-09T19:42:53.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:42:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17083.514059 secs 2023-03-09T19:42:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:42:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17083.514059 secs 2023-03-09T19:42:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:42:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17083.514059 secs 2023-03-09T19:42:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17545.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:42:54.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:42:54 smithi122 ceph-mon[93869]: pgmap v9623: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 348 KiB/s wr, 10 op/s 2023-03-09T19:42:54.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:42:54 smithi181 ceph-mon[94189]: pgmap v9623: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 348 KiB/s wr, 10 op/s 2023-03-09T19:42:54.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:42:54 smithi038 ceph-mon[92422]: pgmap v9623: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 348 KiB/s wr, 10 op/s 2023-03-09T19:42:55.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:42:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17086.204985 secs 2023-03-09T19:42:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:42:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17086.204985 secs 2023-03-09T19:42:55.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:42:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17086.204985 secs 2023-03-09T19:42:56.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:42:56 smithi122 ceph-mon[93869]: pgmap v9624: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 348 KiB/s wr, 10 op/s 2023-03-09T19:42:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:42:56 smithi181 ceph-mon[94189]: pgmap v9624: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 348 KiB/s wr, 10 op/s 2023-03-09T19:42:56.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:42:56 smithi038 ceph-mon[92422]: pgmap v9624: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 348 KiB/s wr, 10 op/s 2023-03-09T19:42:58.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:42:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17088.514196 secs 2023-03-09T19:42:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:42:58 smithi122 ceph-mon[93869]: pgmap v9625: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 435 KiB/s wr, 14 op/s 2023-03-09T19:42:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:42:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17088.514196 secs 2023-03-09T19:42:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:42:58 smithi181 ceph-mon[94189]: pgmap v9625: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 435 KiB/s wr, 14 op/s 2023-03-09T19:42:58.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:42:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17088.514196 secs 2023-03-09T19:42:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:42:58 smithi038 ceph-mon[92422]: pgmap v9625: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 435 KiB/s wr, 14 op/s 2023-03-09T19:42:58.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17550.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:43:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:43:00 smithi122 ceph-mon[93869]: pgmap v9626: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 303 KiB/s wr, 11 op/s 2023-03-09T19:43:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:43:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17091.205045 secs 2023-03-09T19:43:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:43:00 smithi181 ceph-mon[94189]: pgmap v9626: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 303 KiB/s wr, 11 op/s 2023-03-09T19:43:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:43:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17091.205045 secs 2023-03-09T19:43:00.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:43:00 smithi038 ceph-mon[92422]: pgmap v9626: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 303 KiB/s wr, 11 op/s 2023-03-09T19:43:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:43:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17091.205045 secs 2023-03-09T19:43:02.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:43:02 smithi122 ceph-mon[93869]: pgmap v9627: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 339 KiB/s wr, 15 op/s 2023-03-09T19:43:02.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:43:02 smithi181 ceph-mon[94189]: pgmap v9627: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 339 KiB/s wr, 15 op/s 2023-03-09T19:43:02.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:43:02 smithi038 ceph-mon[92422]: pgmap v9627: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 339 KiB/s wr, 15 op/s 2023-03-09T19:43:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:43:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17093.514259 secs 2023-03-09T19:43:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:43:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17093.514259 secs 2023-03-09T19:43:03.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:43:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17093.514259 secs 2023-03-09T19:43:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17555.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:43:04.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:43:04 smithi181 ceph-mon[94189]: pgmap v9628: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 439 KiB/s wr, 25 op/s 2023-03-09T19:43:04.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:43:04 smithi038 ceph-mon[92422]: pgmap v9628: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 439 KiB/s wr, 25 op/s 2023-03-09T19:43:04.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:43:04 smithi122 ceph-mon[93869]: pgmap v9628: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 439 KiB/s wr, 25 op/s 2023-03-09T19:43:05.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:43:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17096.205110 secs 2023-03-09T19:43:05.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:43:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17096.205110 secs 2023-03-09T19:43:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:43:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17096.205110 secs 2023-03-09T19:43:06.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:43:06 smithi038 ceph-mon[92422]: pgmap v9629: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 263 KiB/s wr, 24 op/s 2023-03-09T19:43:06.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:43:06 smithi122 ceph-mon[93869]: pgmap v9629: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 263 KiB/s wr, 24 op/s 2023-03-09T19:43:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:43:06 smithi181 ceph-mon[94189]: pgmap v9629: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 263 KiB/s wr, 24 op/s 2023-03-09T19:43:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:43:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17098.514355 secs 2023-03-09T19:43:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:43:08 smithi038 ceph-mon[92422]: pgmap v9630: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 264 KiB/s wr, 27 op/s 2023-03-09T19:43:08.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17560.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:43:08.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:43:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17098.514355 secs 2023-03-09T19:43:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:43:08 smithi122 ceph-mon[93869]: pgmap v9630: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 264 KiB/s wr, 27 op/s 2023-03-09T19:43:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:43:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17098.514355 secs 2023-03-09T19:43:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:43:08 smithi181 ceph-mon[94189]: pgmap v9630: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 264 KiB/s wr, 27 op/s 2023-03-09T19:43:10.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:43:10 smithi122 ceph-mon[93869]: pgmap v9631: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 278 KiB/s wr, 23 op/s 2023-03-09T19:43:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:43:10 smithi122 ceph-mon[93869]: osdmap e815: 12 total, 12 up, 12 in 2023-03-09T19:43:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:43:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17101.205169 secs 2023-03-09T19:43:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:43:10 smithi181 ceph-mon[94189]: pgmap v9631: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 278 KiB/s wr, 23 op/s 2023-03-09T19:43:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:43:10 smithi181 ceph-mon[94189]: osdmap e815: 12 total, 12 up, 12 in 2023-03-09T19:43:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:43:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17101.205169 secs 2023-03-09T19:43:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:43:10 smithi038 ceph-mon[92422]: pgmap v9631: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 278 KiB/s wr, 23 op/s 2023-03-09T19:43:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:43:10 smithi038 ceph-mon[92422]: osdmap e815: 12 total, 12 up, 12 in 2023-03-09T19:43:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:43:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17101.205169 secs 2023-03-09T19:43:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:43:12 smithi122 ceph-mon[93869]: pgmap v9633: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 277 KiB/s wr, 25 op/s 2023-03-09T19:43:12.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:43:12 smithi181 ceph-mon[94189]: pgmap v9633: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 277 KiB/s wr, 25 op/s 2023-03-09T19:43:12.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:43:12 smithi038 ceph-mon[92422]: pgmap v9633: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 277 KiB/s wr, 25 op/s 2023-03-09T19:43:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17565.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:43:13.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:43:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17103.514418 secs 2023-03-09T19:43:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:43:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17103.514418 secs 2023-03-09T19:43:13.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:43:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17103.514418 secs 2023-03-09T19:43:14.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:43:14 smithi122 ceph-mon[93869]: pgmap v9634: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 140 KiB/s wr, 10 op/s 2023-03-09T19:43:14.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:43:14 smithi181 ceph-mon[94189]: pgmap v9634: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 140 KiB/s wr, 10 op/s 2023-03-09T19:43:14.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:43:14 smithi038 ceph-mon[92422]: pgmap v9634: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 140 KiB/s wr, 10 op/s 2023-03-09T19:43:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:43:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17106.205230 secs 2023-03-09T19:43:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:43:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17106.205230 secs 2023-03-09T19:43:15.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:43:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17106.205230 secs 2023-03-09T19:43:16.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:43:16 smithi122 ceph-mon[93869]: pgmap v9635: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 139 KiB/s wr, 9 op/s 2023-03-09T19:43:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:43:16 smithi181 ceph-mon[94189]: pgmap v9635: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 139 KiB/s wr, 9 op/s 2023-03-09T19:43:16.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:43:16 smithi038 ceph-mon[92422]: pgmap v9635: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 139 KiB/s wr, 9 op/s 2023-03-09T19:43:18.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17570.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:43:18.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:43:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17108.514522 secs 2023-03-09T19:43:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:43:18 smithi122 ceph-mon[93869]: pgmap v9636: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 155 KiB/s wr, 9 op/s 2023-03-09T19:43:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:43:18 smithi122 ceph-mon[93869]: osdmap e816: 12 total, 12 up, 12 in 2023-03-09T19:43:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:43:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17108.514522 secs 2023-03-09T19:43:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:43:18 smithi181 ceph-mon[94189]: pgmap v9636: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 155 KiB/s wr, 9 op/s 2023-03-09T19:43:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:43:18 smithi181 ceph-mon[94189]: osdmap e816: 12 total, 12 up, 12 in 2023-03-09T19:43:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:43:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17108.514522 secs 2023-03-09T19:43:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:43:18 smithi038 ceph-mon[92422]: pgmap v9636: 129 pgs: 129 active+clean; 9.5 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 155 KiB/s wr, 9 op/s 2023-03-09T19:43:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:43:18 smithi038 ceph-mon[92422]: osdmap e816: 12 total, 12 up, 12 in 2023-03-09T19:43:20.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:43:20 smithi122 ceph-mon[93869]: pgmap v9638: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 98 KiB/s wr, 9 op/s 2023-03-09T19:43:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:43:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17111.205294 secs 2023-03-09T19:43:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:43:20 smithi181 ceph-mon[94189]: pgmap v9638: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 98 KiB/s wr, 9 op/s 2023-03-09T19:43:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:43:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17111.205294 secs 2023-03-09T19:43:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:43:20 smithi038 ceph-mon[92422]: pgmap v9638: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 98 KiB/s wr, 9 op/s 2023-03-09T19:43:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:43:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17111.205294 secs 2023-03-09T19:43:21.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:43:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:43:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:43:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:43:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:43:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:43:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:43:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:43:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:43:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:43:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:43:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:43:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:43:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:43:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:43:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:43:21.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:43:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:43:21.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:43:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:43:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:43:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:43:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:43:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:43:22.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:43:22 smithi122 ceph-mon[93869]: pgmap v9639: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 80 KiB/s wr, 8 op/s 2023-03-09T19:43:22.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:43:22 smithi181 ceph-mon[94189]: pgmap v9639: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 80 KiB/s wr, 8 op/s 2023-03-09T19:43:22.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:43:22 smithi038 ceph-mon[92422]: pgmap v9639: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 80 KiB/s wr, 8 op/s 2023-03-09T19:43:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17575.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:43:23.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:43:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17113.514613 secs 2023-03-09T19:43:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:43:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17113.514613 secs 2023-03-09T19:43:23.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:43:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17113.514613 secs 2023-03-09T19:43:24.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:43:24 smithi122 ceph-mon[93869]: pgmap v9640: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 80 KiB/s wr, 8 op/s 2023-03-09T19:43:24.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:43:24 smithi181 ceph-mon[94189]: pgmap v9640: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 80 KiB/s wr, 8 op/s 2023-03-09T19:43:24.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:43:24 smithi038 ceph-mon[92422]: pgmap v9640: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 80 KiB/s wr, 8 op/s 2023-03-09T19:43:25.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:43:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17116.205355 secs 2023-03-09T19:43:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:43:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17116.205355 secs 2023-03-09T19:43:25.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:43:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17116.205355 secs 2023-03-09T19:43:26.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:43:26 smithi122 ceph-mon[93869]: pgmap v9641: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 152 KiB/s wr, 8 op/s 2023-03-09T19:43:26.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:43:26 smithi181 ceph-mon[94189]: pgmap v9641: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 152 KiB/s wr, 8 op/s 2023-03-09T19:43:26.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:43:26 smithi038 ceph-mon[92422]: pgmap v9641: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 152 KiB/s wr, 8 op/s 2023-03-09T19:43:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17580.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:43:28.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:43:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17118.514695 secs 2023-03-09T19:43:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:43:28 smithi122 ceph-mon[93869]: pgmap v9642: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 135 KiB/s wr, 7 op/s 2023-03-09T19:43:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:43:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17118.514695 secs 2023-03-09T19:43:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:43:28 smithi181 ceph-mon[94189]: pgmap v9642: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 135 KiB/s wr, 7 op/s 2023-03-09T19:43:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:43:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17118.514695 secs 2023-03-09T19:43:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:43:28 smithi038 ceph-mon[92422]: pgmap v9642: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 135 KiB/s wr, 7 op/s 2023-03-09T19:43:30.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:43:30 smithi122 ceph-mon[93869]: pgmap v9643: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 122 KiB/s wr, 7 op/s 2023-03-09T19:43:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:43:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17121.205414 secs 2023-03-09T19:43:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:43:30 smithi181 ceph-mon[94189]: pgmap v9643: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 122 KiB/s wr, 7 op/s 2023-03-09T19:43:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:43:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17121.205414 secs 2023-03-09T19:43:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:43:30 smithi038 ceph-mon[92422]: pgmap v9643: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 122 KiB/s wr, 7 op/s 2023-03-09T19:43:30.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:43:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17121.205414 secs 2023-03-09T19:43:32.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:43:32 smithi122 ceph-mon[93869]: pgmap v9644: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 62 KiB/s wr, 8 op/s 2023-03-09T19:43:32.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:43:32 smithi181 ceph-mon[94189]: pgmap v9644: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 62 KiB/s wr, 8 op/s 2023-03-09T19:43:32.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:43:32 smithi038 ceph-mon[92422]: pgmap v9644: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 62 KiB/s wr, 8 op/s 2023-03-09T19:43:33.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17585.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:43:33.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:43:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17123.514808 secs 2023-03-09T19:43:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:43:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17123.514808 secs 2023-03-09T19:43:33.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:43:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17123.514808 secs 2023-03-09T19:43:34.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:43:34 smithi122 ceph-mon[93869]: pgmap v9645: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 62 KiB/s wr, 6 op/s 2023-03-09T19:43:34.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:43:34 smithi181 ceph-mon[94189]: pgmap v9645: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 62 KiB/s wr, 6 op/s 2023-03-09T19:43:34.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:43:34 smithi038 ceph-mon[92422]: pgmap v9645: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 62 KiB/s wr, 6 op/s 2023-03-09T19:43:35.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:43:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17126.205473 secs 2023-03-09T19:43:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:43:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17126.205473 secs 2023-03-09T19:43:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:43:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17126.205473 secs 2023-03-09T19:43:36.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:43:36 smithi122 ceph-mon[93869]: pgmap v9646: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 122 KiB/s wr, 7 op/s 2023-03-09T19:43:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:43:36 smithi181 ceph-mon[94189]: pgmap v9646: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 122 KiB/s wr, 7 op/s 2023-03-09T19:43:36.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:43:36 smithi038 ceph-mon[92422]: pgmap v9646: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 122 KiB/s wr, 7 op/s 2023-03-09T19:43:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17590.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:43:38.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:43:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17128.514924 secs 2023-03-09T19:43:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:43:38 smithi122 ceph-mon[93869]: pgmap v9647: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 62 KiB/s wr, 9 op/s 2023-03-09T19:43:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:43:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17128.514924 secs 2023-03-09T19:43:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:43:38 smithi181 ceph-mon[94189]: pgmap v9647: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 62 KiB/s wr, 9 op/s 2023-03-09T19:43:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:43:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17128.514924 secs 2023-03-09T19:43:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:43:38 smithi038 ceph-mon[92422]: pgmap v9647: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 62 KiB/s wr, 9 op/s 2023-03-09T19:43:40.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:43:40 smithi122 ceph-mon[93869]: pgmap v9648: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 103 KiB/s wr, 7 op/s 2023-03-09T19:43:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:43:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17131.205536 secs 2023-03-09T19:43:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:43:40 smithi181 ceph-mon[94189]: pgmap v9648: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 103 KiB/s wr, 7 op/s 2023-03-09T19:43:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:43:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17131.205536 secs 2023-03-09T19:43:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:43:40 smithi038 ceph-mon[92422]: pgmap v9648: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 103 KiB/s wr, 7 op/s 2023-03-09T19:43:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:43:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17131.205536 secs 2023-03-09T19:43:41.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:43:41 smithi122 ceph-mon[93869]: pgmap v9649: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 126 KiB/s wr, 9 op/s 2023-03-09T19:43:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:43:41 smithi181 ceph-mon[94189]: pgmap v9649: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 126 KiB/s wr, 9 op/s 2023-03-09T19:43:41.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:43:41 smithi038 ceph-mon[92422]: pgmap v9649: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 126 KiB/s wr, 9 op/s 2023-03-09T19:43:43.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:43:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17133.515044 secs 2023-03-09T19:43:43.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:43:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17133.515044 secs 2023-03-09T19:43:43.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:43:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17133.515044 secs 2023-03-09T19:43:43.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17595.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:43:44.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:43:44 smithi122 ceph-mon[93869]: pgmap v9650: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 293 KiB/s rd, 126 KiB/s wr, 7 op/s 2023-03-09T19:43:44.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:43:44 smithi181 ceph-mon[94189]: pgmap v9650: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 293 KiB/s rd, 126 KiB/s wr, 7 op/s 2023-03-09T19:43:44.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:43:44 smithi038 ceph-mon[92422]: pgmap v9650: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 293 KiB/s rd, 126 KiB/s wr, 7 op/s 2023-03-09T19:43:46.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:43:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17136.205598 secs 2023-03-09T19:43:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:43:46 smithi122 ceph-mon[93869]: pgmap v9651: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 125 KiB/s wr, 7 op/s 2023-03-09T19:43:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:43:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17136.205598 secs 2023-03-09T19:43:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:43:46 smithi181 ceph-mon[94189]: pgmap v9651: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 125 KiB/s wr, 7 op/s 2023-03-09T19:43:46.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:43:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17136.205598 secs 2023-03-09T19:43:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:43:46 smithi038 ceph-mon[92422]: pgmap v9651: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 125 KiB/s wr, 7 op/s 2023-03-09T19:43:48.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:43:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17138.515181 secs 2023-03-09T19:43:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:43:48 smithi122 ceph-mon[93869]: pgmap v9652: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 402 KiB/s rd, 192 KiB/s wr, 9 op/s 2023-03-09T19:43:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:43:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17138.515181 secs 2023-03-09T19:43:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:43:48 smithi181 ceph-mon[94189]: pgmap v9652: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 402 KiB/s rd, 192 KiB/s wr, 9 op/s 2023-03-09T19:43:48.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:43:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17138.515181 secs 2023-03-09T19:43:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:43:48 smithi038 ceph-mon[92422]: pgmap v9652: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 402 KiB/s rd, 192 KiB/s wr, 9 op/s 2023-03-09T19:43:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17600.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:43:50.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:43:50 smithi122 ceph-mon[93869]: pgmap v9653: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 197 KiB/s wr, 8 op/s 2023-03-09T19:43:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:43:50 smithi181 ceph-mon[94189]: pgmap v9653: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 197 KiB/s wr, 8 op/s 2023-03-09T19:43:50.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:43:50 smithi038 ceph-mon[92422]: pgmap v9653: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 197 KiB/s wr, 8 op/s 2023-03-09T19:43:51.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:43:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17141.205658 secs 2023-03-09T19:43:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:43:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17141.205658 secs 2023-03-09T19:43:51.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:43:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17141.205658 secs 2023-03-09T19:43:52.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:43:52 smithi122 ceph-mon[93869]: pgmap v9654: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 189 KiB/s wr, 11 op/s 2023-03-09T19:43:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:43:52 smithi181 ceph-mon[94189]: pgmap v9654: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 189 KiB/s wr, 11 op/s 2023-03-09T19:43:52.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:43:52 smithi038 ceph-mon[92422]: pgmap v9654: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 189 KiB/s wr, 11 op/s 2023-03-09T19:43:53.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17605.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:43:54.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:43:54 smithi122 ceph-mon[93869]: pgmap v9655: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 166 KiB/s wr, 9 op/s 2023-03-09T19:43:54.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:43:54 smithi181 ceph-mon[94189]: pgmap v9655: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 166 KiB/s wr, 9 op/s 2023-03-09T19:43:54.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:43:54 smithi038 ceph-mon[92422]: pgmap v9655: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 166 KiB/s wr, 9 op/s 2023-03-09T19:43:56.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:43:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17146.205719 secs 2023-03-09T19:43:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:43:56 smithi122 ceph-mon[93869]: pgmap v9656: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 180 KiB/s wr, 10 op/s 2023-03-09T19:43:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:43:56 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:43:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:43:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17146.205719 secs 2023-03-09T19:43:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:43:56 smithi181 ceph-mon[94189]: pgmap v9656: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 180 KiB/s wr, 10 op/s 2023-03-09T19:43:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:43:56 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:43:56.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:43:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17146.205719 secs 2023-03-09T19:43:56.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:43:56 smithi038 ceph-mon[92422]: pgmap v9656: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 180 KiB/s wr, 10 op/s 2023-03-09T19:43:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:43:56 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:43:57.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:43:57 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:43:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:43:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:43:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:43:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:43:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:43:57 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:43:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:43:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:43:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:43:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:43:57.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:43:57 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:43:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:43:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:43:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:43:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:43:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:43:58 smithi122 ceph-mon[93869]: pgmap v9657: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 363 KiB/s rd, 362 KiB/s wr, 13 op/s 2023-03-09T19:43:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:43:58 smithi181 ceph-mon[94189]: pgmap v9657: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 363 KiB/s rd, 362 KiB/s wr, 13 op/s 2023-03-09T19:43:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:43:58 smithi038 ceph-mon[92422]: pgmap v9657: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 363 KiB/s rd, 362 KiB/s wr, 13 op/s 2023-03-09T19:43:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17610.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:44:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:00 smithi122 ceph-mon[93869]: pgmap v9658: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 311 KiB/s wr, 11 op/s 2023-03-09T19:44:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17151.205780 secs 2023-03-09T19:44:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:00 smithi181 ceph-mon[94189]: pgmap v9658: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 311 KiB/s wr, 11 op/s 2023-03-09T19:44:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17151.205780 secs 2023-03-09T19:44:00.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:00 smithi038 ceph-mon[92422]: pgmap v9658: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 311 KiB/s wr, 11 op/s 2023-03-09T19:44:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17151.205780 secs 2023-03-09T19:44:02.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:02 smithi122 ceph-mon[93869]: pgmap v9659: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 307 KiB/s wr, 13 op/s 2023-03-09T19:44:02.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:02 smithi181 ceph-mon[94189]: pgmap v9659: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 307 KiB/s wr, 13 op/s 2023-03-09T19:44:02.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:02 smithi038 ceph-mon[92422]: pgmap v9659: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 307 KiB/s wr, 13 op/s 2023-03-09T19:44:03.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17615.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:44:04.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:04 smithi122 ceph-mon[93869]: pgmap v9660: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 482 KiB/s wr, 24 op/s 2023-03-09T19:44:04.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:04 smithi181 ceph-mon[94189]: pgmap v9660: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 482 KiB/s wr, 24 op/s 2023-03-09T19:44:04.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:04 smithi038 ceph-mon[92422]: pgmap v9660: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 482 KiB/s wr, 24 op/s 2023-03-09T19:44:06.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17156.205839 secs 2023-03-09T19:44:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:06 smithi122 ceph-mon[93869]: pgmap v9661: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 496 KiB/s wr, 24 op/s 2023-03-09T19:44:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17156.205839 secs 2023-03-09T19:44:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:06 smithi181 ceph-mon[94189]: pgmap v9661: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 496 KiB/s wr, 24 op/s 2023-03-09T19:44:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17156.205839 secs 2023-03-09T19:44:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:06 smithi038 ceph-mon[92422]: pgmap v9661: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 496 KiB/s wr, 24 op/s 2023-03-09T19:44:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17158.515555 secs 2023-03-09T19:44:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:08 smithi122 ceph-mon[93869]: pgmap v9662: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 505 KiB/s wr, 28 op/s 2023-03-09T19:44:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17158.515555 secs 2023-03-09T19:44:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:08 smithi181 ceph-mon[94189]: pgmap v9662: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 505 KiB/s wr, 28 op/s 2023-03-09T19:44:08.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17158.515555 secs 2023-03-09T19:44:08.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:08 smithi038 ceph-mon[92422]: pgmap v9662: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 505 KiB/s wr, 28 op/s 2023-03-09T19:44:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17620.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:44:09.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:09 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:44:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:44:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:44:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:09 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:44:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:44:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:44:09.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:09 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:44:09.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:44:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:44:10.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:10 smithi122 ceph-mon[93869]: pgmap v9663: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 384 KiB/s rd, 431 KiB/s wr, 26 op/s 2023-03-09T19:44:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:10 smithi122 ceph-mon[93869]: osdmap e817: 12 total, 12 up, 12 in 2023-03-09T19:44:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17161.205899 secs 2023-03-09T19:44:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:10 smithi181 ceph-mon[94189]: pgmap v9663: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 384 KiB/s rd, 431 KiB/s wr, 26 op/s 2023-03-09T19:44:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:10 smithi181 ceph-mon[94189]: osdmap e817: 12 total, 12 up, 12 in 2023-03-09T19:44:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17161.205899 secs 2023-03-09T19:44:10.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:10 smithi038 ceph-mon[92422]: pgmap v9663: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 384 KiB/s rd, 431 KiB/s wr, 26 op/s 2023-03-09T19:44:10.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:10 smithi038 ceph-mon[92422]: osdmap e817: 12 total, 12 up, 12 in 2023-03-09T19:44:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17161.205899 secs 2023-03-09T19:44:12.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:12 smithi122 ceph-mon[93869]: pgmap v9665: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 446 KiB/s rd, 503 KiB/s wr, 30 op/s 2023-03-09T19:44:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:12 smithi181 ceph-mon[94189]: pgmap v9665: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 446 KiB/s rd, 503 KiB/s wr, 30 op/s 2023-03-09T19:44:12.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:12 smithi038 ceph-mon[92422]: pgmap v9665: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 446 KiB/s rd, 503 KiB/s wr, 30 op/s 2023-03-09T19:44:13.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17163.515688 secs 2023-03-09T19:44:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17163.515688 secs 2023-03-09T19:44:13.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17163.515688 secs 2023-03-09T19:44:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17625.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:44:14.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:14 smithi122 ceph-mon[93869]: pgmap v9666: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 254 KiB/s wr, 14 op/s 2023-03-09T19:44:14.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:14 smithi181 ceph-mon[94189]: pgmap v9666: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 254 KiB/s wr, 14 op/s 2023-03-09T19:44:14.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:14 smithi038 ceph-mon[92422]: pgmap v9666: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 254 KiB/s wr, 14 op/s 2023-03-09T19:44:15.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17166.205961 secs 2023-03-09T19:44:15.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17166.205961 secs 2023-03-09T19:44:15.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17166.205961 secs 2023-03-09T19:44:16.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:16 smithi122 ceph-mon[93869]: pgmap v9667: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 238 KiB/s wr, 14 op/s 2023-03-09T19:44:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:16 smithi181 ceph-mon[94189]: pgmap v9667: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 238 KiB/s wr, 14 op/s 2023-03-09T19:44:16.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:16 smithi038 ceph-mon[92422]: pgmap v9667: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 238 KiB/s wr, 14 op/s 2023-03-09T19:44:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17168.515808 secs 2023-03-09T19:44:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:18 smithi122 ceph-mon[93869]: pgmap v9668: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 210 KiB/s wr, 10 op/s 2023-03-09T19:44:18.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:18 smithi122 ceph-mon[93869]: osdmap e818: 12 total, 12 up, 12 in 2023-03-09T19:44:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17168.515808 secs 2023-03-09T19:44:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:18 smithi181 ceph-mon[94189]: pgmap v9668: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 210 KiB/s wr, 10 op/s 2023-03-09T19:44:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:18 smithi181 ceph-mon[94189]: osdmap e818: 12 total, 12 up, 12 in 2023-03-09T19:44:18.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17168.515808 secs 2023-03-09T19:44:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:18 smithi038 ceph-mon[92422]: pgmap v9668: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 210 KiB/s wr, 10 op/s 2023-03-09T19:44:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:18 smithi038 ceph-mon[92422]: osdmap e818: 12 total, 12 up, 12 in 2023-03-09T19:44:18.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17630.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:44:20.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:20 smithi122 ceph-mon[93869]: pgmap v9670: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 81 KiB/s wr, 9 op/s 2023-03-09T19:44:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17171.205981 secs 2023-03-09T19:44:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:20 smithi181 ceph-mon[94189]: pgmap v9670: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 81 KiB/s wr, 9 op/s 2023-03-09T19:44:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17171.205981 secs 2023-03-09T19:44:20.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:20 smithi038 ceph-mon[92422]: pgmap v9670: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 81 KiB/s wr, 9 op/s 2023-03-09T19:44:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17171.205981 secs 2023-03-09T19:44:22.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:22 smithi122 ceph-mon[93869]: pgmap v9671: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 2.0 KiB/s wr, 8 op/s 2023-03-09T19:44:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:22 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:44:22.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:22 smithi181 ceph-mon[94189]: pgmap v9671: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 2.0 KiB/s wr, 8 op/s 2023-03-09T19:44:22.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:22 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:44:22.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:22 smithi038 ceph-mon[92422]: pgmap v9671: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 2.0 KiB/s wr, 8 op/s 2023-03-09T19:44:22.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:22 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:44:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17173.515872 secs 2023-03-09T19:44:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17173.515872 secs 2023-03-09T19:44:23.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17173.515872 secs 2023-03-09T19:44:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17635.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:44:24.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:24 smithi181 ceph-mon[94189]: pgmap v9672: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.8 KiB/s wr, 7 op/s 2023-03-09T19:44:24.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:44:24.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:44:24.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:44:24.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:44:24.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:24 smithi038 ceph-mon[92422]: pgmap v9672: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.8 KiB/s wr, 7 op/s 2023-03-09T19:44:24.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:44:24.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:44:24.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:44:24.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:44:24.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:24 smithi122 ceph-mon[93869]: pgmap v9672: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.8 KiB/s wr, 7 op/s 2023-03-09T19:44:24.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:44:24.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:44:24.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:44:24.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:44:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:44:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:44:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:44:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:44:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17176.206041 secs 2023-03-09T19:44:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:44:25.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:44:25.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:44:25.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:44:25.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:44:25.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:44:25.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:44:25.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:44:25.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:44:25.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:44:25.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:44:25.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:44:25.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:44:25.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:44:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:44:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:44:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:44:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17176.206041 secs 2023-03-09T19:44:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:44:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:44:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:44:25.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:44:25.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:44:25.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:44:25.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:44:25.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:44:25.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:44:25.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:44:25.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:44:25.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:44:25.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:44:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:44:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:44:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:44:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:44:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17176.206041 secs 2023-03-09T19:44:25.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:44:25.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:44:25.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:44:25.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:44:25.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:44:25.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:44:25.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:44:25.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:44:25.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:44:25.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:44:25.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:44:25.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:44:25.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:44:26.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:26 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T19:44:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:26 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T19:44:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:26 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi038 to 133.2M 2023-03-09T19:44:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:26 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi038 to 139768217: error parsing value: Value '139768217' is below minimum 939524096 2023-03-09T19:44:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:26 smithi038 ceph-mon[92422]: pgmap v9673: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.3 KiB/s wr, 7 op/s 2023-03-09T19:44:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:26 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T19:44:26.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:26 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T19:44:26.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:26 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T19:44:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:26 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T19:44:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:26 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi038 to 133.2M 2023-03-09T19:44:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:26 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi038 to 139768217: error parsing value: Value '139768217' is below minimum 939524096 2023-03-09T19:44:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:26 smithi122 ceph-mon[93869]: pgmap v9673: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.3 KiB/s wr, 7 op/s 2023-03-09T19:44:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:26 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T19:44:26.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:26 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T19:44:26.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:26 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T19:44:26.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:26 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T19:44:26.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:26 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi038 to 133.2M 2023-03-09T19:44:26.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:26 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi038 to 139768217: error parsing value: Value '139768217' is below minimum 939524096 2023-03-09T19:44:26.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:26 smithi181 ceph-mon[94189]: pgmap v9673: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.3 KiB/s wr, 7 op/s 2023-03-09T19:44:26.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:26 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T19:44:26.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:26 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T19:44:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17178.515987 secs 2023-03-09T19:44:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:28 smithi038 ceph-mon[92422]: pgmap v9674: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.4 KiB/s wr, 7 op/s 2023-03-09T19:44:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17640.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:44:28.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17178.515987 secs 2023-03-09T19:44:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:28 smithi122 ceph-mon[93869]: pgmap v9674: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.4 KiB/s wr, 7 op/s 2023-03-09T19:44:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17178.515987 secs 2023-03-09T19:44:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:28 smithi181 ceph-mon[94189]: pgmap v9674: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.4 KiB/s wr, 7 op/s 2023-03-09T19:44:30.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:30 smithi038 ceph-mon[92422]: pgmap v9675: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.4 KiB/s wr, 7 op/s 2023-03-09T19:44:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17181.206102 secs 2023-03-09T19:44:30.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:30 smithi122 ceph-mon[93869]: pgmap v9675: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.4 KiB/s wr, 7 op/s 2023-03-09T19:44:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17181.206102 secs 2023-03-09T19:44:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:30 smithi181 ceph-mon[94189]: pgmap v9675: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.4 KiB/s wr, 7 op/s 2023-03-09T19:44:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17181.206102 secs 2023-03-09T19:44:32.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:32 smithi038 ceph-mon[92422]: pgmap v9676: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 62 KiB/s wr, 8 op/s 2023-03-09T19:44:32.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:32 smithi122 ceph-mon[93869]: pgmap v9676: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 62 KiB/s wr, 8 op/s 2023-03-09T19:44:32.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:32 smithi181 ceph-mon[94189]: pgmap v9676: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 62 KiB/s wr, 8 op/s 2023-03-09T19:44:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17183.516101 secs 2023-03-09T19:44:33.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17645.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:44:33.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17183.516101 secs 2023-03-09T19:44:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17183.516101 secs 2023-03-09T19:44:34.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:34 smithi038 ceph-mon[92422]: pgmap v9677: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 61 KiB/s wr, 7 op/s 2023-03-09T19:44:34.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:34 smithi122 ceph-mon[93869]: pgmap v9677: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 61 KiB/s wr, 7 op/s 2023-03-09T19:44:34.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:34 smithi181 ceph-mon[94189]: pgmap v9677: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 61 KiB/s wr, 7 op/s 2023-03-09T19:44:35.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17186.206162 secs 2023-03-09T19:44:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17186.206162 secs 2023-03-09T19:44:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17186.206162 secs 2023-03-09T19:44:36.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:36 smithi122 ceph-mon[93869]: pgmap v9678: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 62 KiB/s wr, 7 op/s 2023-03-09T19:44:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:36 smithi181 ceph-mon[94189]: pgmap v9678: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 62 KiB/s wr, 7 op/s 2023-03-09T19:44:37.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:36 smithi038 ceph-mon[92422]: pgmap v9678: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 62 KiB/s wr, 7 op/s 2023-03-09T19:44:38.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17650.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:44:38.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:38 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 17188.516165 secs 2023-03-09T19:44:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:38 smithi122 ceph-mon[93869]: pgmap v9679: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 61 KiB/s wr, 8 op/s 2023-03-09T19:44:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:38 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 17188.516165 secs 2023-03-09T19:44:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:38 smithi181 ceph-mon[94189]: pgmap v9679: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 61 KiB/s wr, 8 op/s 2023-03-09T19:44:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:38 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 17188.516165 secs 2023-03-09T19:44:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:38 smithi038 ceph-mon[92422]: pgmap v9679: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 61 KiB/s wr, 8 op/s 2023-03-09T19:44:40.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:40 smithi122 ceph-mon[93869]: pgmap v9680: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 62 KiB/s wr, 7 op/s 2023-03-09T19:44:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17191.206242 secs 2023-03-09T19:44:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:40 smithi181 ceph-mon[94189]: pgmap v9680: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 62 KiB/s wr, 7 op/s 2023-03-09T19:44:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17191.206242 secs 2023-03-09T19:44:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:40 smithi038 ceph-mon[92422]: pgmap v9680: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 62 KiB/s wr, 7 op/s 2023-03-09T19:44:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17191.206242 secs 2023-03-09T19:44:41.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:44:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:44:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:44:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:44:41.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:44:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:44:42.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:42 smithi122 ceph-mon[93869]: pgmap v9681: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 126 KiB/s wr, 8 op/s 2023-03-09T19:44:42.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:42 smithi181 ceph-mon[94189]: pgmap v9681: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 126 KiB/s wr, 8 op/s 2023-03-09T19:44:42.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:42 smithi038 ceph-mon[92422]: pgmap v9681: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 126 KiB/s wr, 8 op/s 2023-03-09T19:44:43.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17655.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:44:43.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:43 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 17193.516249 secs 2023-03-09T19:44:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:43 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 17193.516249 secs 2023-03-09T19:44:43.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:43 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 17193.516249 secs 2023-03-09T19:44:44.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:44 smithi122 ceph-mon[93869]: pgmap v9682: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 66 KiB/s wr, 6 op/s 2023-03-09T19:44:44.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:44 smithi181 ceph-mon[94189]: pgmap v9682: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 66 KiB/s wr, 6 op/s 2023-03-09T19:44:44.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:44 smithi038 ceph-mon[92422]: pgmap v9682: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 66 KiB/s wr, 6 op/s 2023-03-09T19:44:45.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17196.206305 secs 2023-03-09T19:44:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17196.206305 secs 2023-03-09T19:44:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17196.206305 secs 2023-03-09T19:44:46.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:46 smithi122 ceph-mon[93869]: pgmap v9683: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 245 KiB/s wr, 8 op/s 2023-03-09T19:44:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:46 smithi181 ceph-mon[94189]: pgmap v9683: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 245 KiB/s wr, 8 op/s 2023-03-09T19:44:46.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:46 smithi038 ceph-mon[92422]: pgmap v9683: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 245 KiB/s wr, 8 op/s 2023-03-09T19:44:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17660.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:44:48.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17198.516302 secs 2023-03-09T19:44:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:48 smithi122 ceph-mon[93869]: pgmap v9684: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 285 KiB/s wr, 10 op/s 2023-03-09T19:44:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17198.516302 secs 2023-03-09T19:44:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:48 smithi181 ceph-mon[94189]: pgmap v9684: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 285 KiB/s wr, 10 op/s 2023-03-09T19:44:48.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17198.516302 secs 2023-03-09T19:44:48.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:48 smithi038 ceph-mon[92422]: pgmap v9684: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 285 KiB/s wr, 10 op/s 2023-03-09T19:44:49.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:44:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:44:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:44:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:44:49.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:44:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:44:50.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:50 smithi122 ceph-mon[93869]: pgmap v9685: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 285 KiB/s wr, 9 op/s 2023-03-09T19:44:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17201.206365 secs 2023-03-09T19:44:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:50 smithi181 ceph-mon[94189]: pgmap v9685: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 285 KiB/s wr, 9 op/s 2023-03-09T19:44:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17201.206365 secs 2023-03-09T19:44:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:50 smithi038 ceph-mon[92422]: pgmap v9685: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 285 KiB/s wr, 9 op/s 2023-03-09T19:44:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17201.206365 secs 2023-03-09T19:44:52.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:52 smithi122 ceph-mon[93869]: pgmap v9686: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 285 KiB/s wr, 10 op/s 2023-03-09T19:44:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:52 smithi181 ceph-mon[94189]: pgmap v9686: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 285 KiB/s wr, 10 op/s 2023-03-09T19:44:52.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:52 smithi038 ceph-mon[92422]: pgmap v9686: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 285 KiB/s wr, 10 op/s 2023-03-09T19:44:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17665.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:44:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17203.516367 secs 2023-03-09T19:44:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17203.516367 secs 2023-03-09T19:44:53.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17203.516367 secs 2023-03-09T19:44:54.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:54 smithi122 ceph-mon[93869]: pgmap v9687: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 244 KiB/s wr, 9 op/s 2023-03-09T19:44:54.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:54 smithi181 ceph-mon[94189]: pgmap v9687: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 244 KiB/s wr, 9 op/s 2023-03-09T19:44:54.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:54 smithi038 ceph-mon[92422]: pgmap v9687: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 244 KiB/s wr, 9 op/s 2023-03-09T19:44:55.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17206.206425 secs 2023-03-09T19:44:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17206.206425 secs 2023-03-09T19:44:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17206.206425 secs 2023-03-09T19:44:56.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:56 smithi122 ceph-mon[93869]: pgmap v9688: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 258 KiB/s wr, 10 op/s 2023-03-09T19:44:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:56 smithi181 ceph-mon[94189]: pgmap v9688: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 258 KiB/s wr, 10 op/s 2023-03-09T19:44:56.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:56 smithi038 ceph-mon[92422]: pgmap v9688: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 258 KiB/s wr, 10 op/s 2023-03-09T19:44:58.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17670.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:44:58.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17208.516441 secs 2023-03-09T19:44:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:58 smithi122 ceph-mon[93869]: pgmap v9689: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 289 KiB/s wr, 12 op/s 2023-03-09T19:44:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17208.516441 secs 2023-03-09T19:44:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:58 smithi181 ceph-mon[94189]: pgmap v9689: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 289 KiB/s wr, 12 op/s 2023-03-09T19:44:58.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17208.516441 secs 2023-03-09T19:44:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:58 smithi038 ceph-mon[92422]: pgmap v9689: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 289 KiB/s wr, 12 op/s 2023-03-09T19:44:59.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:44:59 smithi122 ceph-mon[93869]: pgmap v9690: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 264 KiB/s wr, 10 op/s 2023-03-09T19:44:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:44:59 smithi181 ceph-mon[94189]: pgmap v9690: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 264 KiB/s wr, 10 op/s 2023-03-09T19:44:59.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:44:59 smithi038 ceph-mon[92422]: pgmap v9690: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 264 KiB/s wr, 10 op/s 2023-03-09T19:45:00.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:45:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17211.206485 secs 2023-03-09T19:45:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:45:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17211.206485 secs 2023-03-09T19:45:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:45:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17211.206485 secs 2023-03-09T19:45:01.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:45:01 smithi122 ceph-mon[93869]: pgmap v9691: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 339 KiB/s wr, 14 op/s 2023-03-09T19:45:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:45:01 smithi181 ceph-mon[94189]: pgmap v9691: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 339 KiB/s wr, 14 op/s 2023-03-09T19:45:01.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:45:01 smithi038 ceph-mon[92422]: pgmap v9691: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 339 KiB/s wr, 14 op/s 2023-03-09T19:45:03.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:45:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17213.516511 secs 2023-03-09T19:45:03.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:45:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17213.516511 secs 2023-03-09T19:45:03.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:45:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17213.516511 secs 2023-03-09T19:45:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17675.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:45:04.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:45:04 smithi122 ceph-mon[93869]: pgmap v9692: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 430 KiB/s rd, 428 KiB/s wr, 25 op/s 2023-03-09T19:45:04.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:45:04 smithi181 ceph-mon[94189]: pgmap v9692: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 430 KiB/s rd, 428 KiB/s wr, 25 op/s 2023-03-09T19:45:04.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:45:04 smithi038 ceph-mon[92422]: pgmap v9692: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 430 KiB/s rd, 428 KiB/s wr, 25 op/s 2023-03-09T19:45:05.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:45:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17216.206570 secs 2023-03-09T19:45:05.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:45:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17216.206570 secs 2023-03-09T19:45:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:45:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17216.206570 secs 2023-03-09T19:45:06.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:45:06 smithi038 ceph-mon[92422]: pgmap v9693: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 443 KiB/s rd, 441 KiB/s wr, 25 op/s 2023-03-09T19:45:06.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:45:06 smithi122 ceph-mon[93869]: pgmap v9693: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 443 KiB/s rd, 441 KiB/s wr, 25 op/s 2023-03-09T19:45:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:45:06 smithi181 ceph-mon[94189]: pgmap v9693: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 443 KiB/s rd, 441 KiB/s wr, 25 op/s 2023-03-09T19:45:08.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:45:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17218.516621 secs 2023-03-09T19:45:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:45:08 smithi038 ceph-mon[92422]: pgmap v9694: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 430 KiB/s rd, 427 KiB/s wr, 27 op/s 2023-03-09T19:45:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17680.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:45:08.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:45:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17218.516621 secs 2023-03-09T19:45:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:45:08 smithi122 ceph-mon[93869]: pgmap v9694: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 430 KiB/s rd, 427 KiB/s wr, 27 op/s 2023-03-09T19:45:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:45:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17218.516621 secs 2023-03-09T19:45:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:45:08 smithi181 ceph-mon[94189]: pgmap v9694: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 430 KiB/s rd, 427 KiB/s wr, 27 op/s 2023-03-09T19:45:10.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:45:10 smithi122 ceph-mon[93869]: pgmap v9695: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 242 KiB/s wr, 24 op/s 2023-03-09T19:45:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:45:10 smithi122 ceph-mon[93869]: osdmap e819: 12 total, 12 up, 12 in 2023-03-09T19:45:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:45:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17221.206629 secs 2023-03-09T19:45:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:45:10 smithi181 ceph-mon[94189]: pgmap v9695: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 242 KiB/s wr, 24 op/s 2023-03-09T19:45:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:45:10 smithi181 ceph-mon[94189]: osdmap e819: 12 total, 12 up, 12 in 2023-03-09T19:45:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:45:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17221.206629 secs 2023-03-09T19:45:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:45:10 smithi038 ceph-mon[92422]: pgmap v9695: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 242 KiB/s wr, 24 op/s 2023-03-09T19:45:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:45:10 smithi038 ceph-mon[92422]: osdmap e819: 12 total, 12 up, 12 in 2023-03-09T19:45:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:45:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17221.206629 secs 2023-03-09T19:45:12.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:45:12 smithi122 ceph-mon[93869]: pgmap v9697: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 436 KiB/s rd, 408 KiB/s wr, 25 op/s 2023-03-09T19:45:12.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:45:12 smithi181 ceph-mon[94189]: pgmap v9697: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 436 KiB/s rd, 408 KiB/s wr, 25 op/s 2023-03-09T19:45:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:45:12 smithi038 ceph-mon[92422]: pgmap v9697: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 436 KiB/s rd, 408 KiB/s wr, 25 op/s 2023-03-09T19:45:13.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17685.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:45:13.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:45:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17223.516733 secs 2023-03-09T19:45:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:45:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17223.516733 secs 2023-03-09T19:45:13.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:45:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17223.516733 secs 2023-03-09T19:45:14.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:45:14 smithi122 ceph-mon[93869]: pgmap v9698: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 301 KiB/s wr, 10 op/s 2023-03-09T19:45:14.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:45:14 smithi181 ceph-mon[94189]: pgmap v9698: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 301 KiB/s wr, 10 op/s 2023-03-09T19:45:14.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:45:14 smithi038 ceph-mon[92422]: pgmap v9698: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 301 KiB/s wr, 10 op/s 2023-03-09T19:45:15.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:45:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17226.206689 secs 2023-03-09T19:45:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:45:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17226.206689 secs 2023-03-09T19:45:15.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:45:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17226.206689 secs 2023-03-09T19:45:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:45:16 smithi122 ceph-mon[93869]: pgmap v9699: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 257 KiB/s wr, 10 op/s 2023-03-09T19:45:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:45:16 smithi181 ceph-mon[94189]: pgmap v9699: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 257 KiB/s wr, 10 op/s 2023-03-09T19:45:16.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:45:16 smithi038 ceph-mon[92422]: pgmap v9699: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 257 KiB/s wr, 10 op/s 2023-03-09T19:45:18.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17690.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:45:18.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:45:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17228.516842 secs 2023-03-09T19:45:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:45:18 smithi122 ceph-mon[93869]: pgmap v9700: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 257 KiB/s wr, 9 op/s 2023-03-09T19:45:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:45:18 smithi122 ceph-mon[93869]: osdmap e820: 12 total, 12 up, 12 in 2023-03-09T19:45:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:45:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17228.516842 secs 2023-03-09T19:45:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:45:18 smithi181 ceph-mon[94189]: pgmap v9700: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 257 KiB/s wr, 9 op/s 2023-03-09T19:45:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:45:18 smithi181 ceph-mon[94189]: osdmap e820: 12 total, 12 up, 12 in 2023-03-09T19:45:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:45:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17228.516842 secs 2023-03-09T19:45:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:45:18 smithi038 ceph-mon[92422]: pgmap v9700: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 257 KiB/s wr, 9 op/s 2023-03-09T19:45:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:45:18 smithi038 ceph-mon[92422]: osdmap e820: 12 total, 12 up, 12 in 2023-03-09T19:45:20.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:45:20 smithi122 ceph-mon[93869]: pgmap v9702: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 245 KiB/s wr, 9 op/s 2023-03-09T19:45:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:45:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17231.206750 secs 2023-03-09T19:45:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:45:20 smithi181 ceph-mon[94189]: pgmap v9702: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 245 KiB/s wr, 9 op/s 2023-03-09T19:45:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:45:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17231.206750 secs 2023-03-09T19:45:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:45:20 smithi038 ceph-mon[92422]: pgmap v9702: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 245 KiB/s wr, 9 op/s 2023-03-09T19:45:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:45:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17231.206750 secs 2023-03-09T19:45:22.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:45:22 smithi122 ceph-mon[93869]: pgmap v9703: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 8 op/s 2023-03-09T19:45:22.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:45:22 smithi181 ceph-mon[94189]: pgmap v9703: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 8 op/s 2023-03-09T19:45:22.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:45:22 smithi038 ceph-mon[92422]: pgmap v9703: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 8 op/s 2023-03-09T19:45:23.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17695.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:45:23.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:45:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17233.516975 secs 2023-03-09T19:45:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:45:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17233.516975 secs 2023-03-09T19:45:23.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:45:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17233.516975 secs 2023-03-09T19:45:24.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:45:24 smithi122 ceph-mon[93869]: pgmap v9704: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T19:45:24.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:45:24 smithi181 ceph-mon[94189]: pgmap v9704: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T19:45:24.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:45:24 smithi038 ceph-mon[92422]: pgmap v9704: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T19:45:25.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:45:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17236.206810 secs 2023-03-09T19:45:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:45:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17236.206810 secs 2023-03-09T19:45:25.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:45:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17236.206810 secs 2023-03-09T19:45:26.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:45:26 smithi122 ceph-mon[93869]: pgmap v9705: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 92 KiB/s wr, 7 op/s 2023-03-09T19:45:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:45:26 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:45:26.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:45:26 smithi181 ceph-mon[94189]: pgmap v9705: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 92 KiB/s wr, 7 op/s 2023-03-09T19:45:26.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:45:26 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:45:26.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:45:26 smithi038 ceph-mon[92422]: pgmap v9705: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 92 KiB/s wr, 7 op/s 2023-03-09T19:45:26.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:45:26 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:45:27.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:45:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:45:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:45:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:45:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:45:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:45:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:45:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:45:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:45:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:45:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:45:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:45:27.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:45:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:45:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:45:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:45:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:45:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:45:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17700.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:45:28.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:45:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17238.517100 secs 2023-03-09T19:45:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:45:28 smithi122 ceph-mon[93869]: pgmap v9706: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 92 KiB/s wr, 7 op/s 2023-03-09T19:45:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:45:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17238.517100 secs 2023-03-09T19:45:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:45:28 smithi181 ceph-mon[94189]: pgmap v9706: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 92 KiB/s wr, 7 op/s 2023-03-09T19:45:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:45:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17238.517100 secs 2023-03-09T19:45:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:45:28 smithi038 ceph-mon[92422]: pgmap v9706: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 92 KiB/s wr, 7 op/s 2023-03-09T19:45:30.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:45:30 smithi122 ceph-mon[93869]: pgmap v9707: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 82 KiB/s wr, 7 op/s 2023-03-09T19:45:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:45:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17241.206871 secs 2023-03-09T19:45:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:45:30 smithi181 ceph-mon[94189]: pgmap v9707: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 82 KiB/s wr, 7 op/s 2023-03-09T19:45:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:45:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17241.206871 secs 2023-03-09T19:45:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:45:30 smithi038 ceph-mon[92422]: pgmap v9707: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 82 KiB/s wr, 7 op/s 2023-03-09T19:45:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:45:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17241.206871 secs 2023-03-09T19:45:32.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:45:32 smithi122 ceph-mon[93869]: pgmap v9708: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 63 KiB/s wr, 9 op/s 2023-03-09T19:45:32.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:45:32 smithi181 ceph-mon[94189]: pgmap v9708: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 63 KiB/s wr, 9 op/s 2023-03-09T19:45:32.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:45:32 smithi038 ceph-mon[92422]: pgmap v9708: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 63 KiB/s wr, 9 op/s 2023-03-09T19:45:33.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17705.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:45:33.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:45:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17243.517220 secs 2023-03-09T19:45:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:45:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17243.517220 secs 2023-03-09T19:45:33.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:45:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17243.517220 secs 2023-03-09T19:45:34.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:45:34 smithi122 ceph-mon[93869]: pgmap v9709: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 62 KiB/s wr, 6 op/s 2023-03-09T19:45:34.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:45:34 smithi181 ceph-mon[94189]: pgmap v9709: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 62 KiB/s wr, 6 op/s 2023-03-09T19:45:34.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:45:34 smithi038 ceph-mon[92422]: pgmap v9709: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 62 KiB/s wr, 6 op/s 2023-03-09T19:45:35.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:45:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17246.206955 secs 2023-03-09T19:45:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:45:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17246.206955 secs 2023-03-09T19:45:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:45:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17246.206955 secs 2023-03-09T19:45:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:45:36 smithi122 ceph-mon[93869]: pgmap v9710: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 68 KiB/s wr, 7 op/s 2023-03-09T19:45:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:45:36 smithi181 ceph-mon[94189]: pgmap v9710: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 68 KiB/s wr, 7 op/s 2023-03-09T19:45:36.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:45:36 smithi038 ceph-mon[92422]: pgmap v9710: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 68 KiB/s wr, 7 op/s 2023-03-09T19:45:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17710.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:45:38.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:45:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17248.517284 secs 2023-03-09T19:45:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:45:38 smithi122 ceph-mon[93869]: pgmap v9711: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 143 KiB/s wr, 8 op/s 2023-03-09T19:45:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:45:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17248.517284 secs 2023-03-09T19:45:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:45:38 smithi181 ceph-mon[94189]: pgmap v9711: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 143 KiB/s wr, 8 op/s 2023-03-09T19:45:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:45:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17248.517284 secs 2023-03-09T19:45:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:45:38 smithi038 ceph-mon[92422]: pgmap v9711: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 143 KiB/s wr, 8 op/s 2023-03-09T19:45:40.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:45:40 smithi122 ceph-mon[93869]: pgmap v9712: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 158 KiB/s wr, 7 op/s 2023-03-09T19:45:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:45:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17251.206985 secs 2023-03-09T19:45:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:45:40 smithi181 ceph-mon[94189]: pgmap v9712: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 158 KiB/s wr, 7 op/s 2023-03-09T19:45:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:45:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17251.206985 secs 2023-03-09T19:45:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:45:40 smithi038 ceph-mon[92422]: pgmap v9712: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 158 KiB/s wr, 7 op/s 2023-03-09T19:45:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:45:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17251.206985 secs 2023-03-09T19:45:42.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:45:42 smithi122 ceph-mon[93869]: pgmap v9713: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 174 KiB/s wr, 9 op/s 2023-03-09T19:45:42.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:45:42 smithi181 ceph-mon[94189]: pgmap v9713: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 174 KiB/s wr, 9 op/s 2023-03-09T19:45:42.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:45:42 smithi038 ceph-mon[92422]: pgmap v9713: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 174 KiB/s wr, 9 op/s 2023-03-09T19:45:43.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17715.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:45:43.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:45:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17253.517399 secs 2023-03-09T19:45:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:45:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17253.517399 secs 2023-03-09T19:45:43.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:45:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17253.517399 secs 2023-03-09T19:45:44.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:45:44 smithi122 ceph-mon[93869]: pgmap v9714: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 173 KiB/s wr, 7 op/s 2023-03-09T19:45:44.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:45:44 smithi181 ceph-mon[94189]: pgmap v9714: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 173 KiB/s wr, 7 op/s 2023-03-09T19:45:44.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:45:44 smithi038 ceph-mon[92422]: pgmap v9714: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 173 KiB/s wr, 7 op/s 2023-03-09T19:45:45.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:45:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17256.207044 secs 2023-03-09T19:45:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:45:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17256.207044 secs 2023-03-09T19:45:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:45:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17256.207044 secs 2023-03-09T19:45:46.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:45:46 smithi122 ceph-mon[93869]: pgmap v9715: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 187 KiB/s wr, 9 op/s 2023-03-09T19:45:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:45:46 smithi181 ceph-mon[94189]: pgmap v9715: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 187 KiB/s wr, 9 op/s 2023-03-09T19:45:46.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:45:46 smithi038 ceph-mon[92422]: pgmap v9715: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 187 KiB/s wr, 9 op/s 2023-03-09T19:45:47.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:45:47 smithi122 ceph-mon[93869]: pgmap v9716: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 406 KiB/s rd, 390 KiB/s wr, 12 op/s 2023-03-09T19:45:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:45:47 smithi181 ceph-mon[94189]: pgmap v9716: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 406 KiB/s rd, 390 KiB/s wr, 12 op/s 2023-03-09T19:45:47.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:45:47 smithi038 ceph-mon[92422]: pgmap v9716: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 406 KiB/s rd, 390 KiB/s wr, 12 op/s 2023-03-09T19:45:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17720.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:45:48.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:45:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17258.517491 secs 2023-03-09T19:45:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:45:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17258.517491 secs 2023-03-09T19:45:48.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:45:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17258.517491 secs 2023-03-09T19:45:49.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:45:49 smithi122 ceph-mon[93869]: pgmap v9717: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 283 KiB/s rd, 268 KiB/s wr, 11 op/s 2023-03-09T19:45:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:45:49 smithi181 ceph-mon[94189]: pgmap v9717: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 283 KiB/s rd, 268 KiB/s wr, 11 op/s 2023-03-09T19:45:49.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:45:49 smithi038 ceph-mon[92422]: pgmap v9717: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 283 KiB/s rd, 268 KiB/s wr, 11 op/s 2023-03-09T19:45:50.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:45:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17261.207120 secs 2023-03-09T19:45:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:45:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17261.207120 secs 2023-03-09T19:45:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:45:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17261.207120 secs 2023-03-09T19:45:51.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:45:51 smithi122 ceph-mon[93869]: pgmap v9718: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 474 KiB/s rd, 445 KiB/s wr, 14 op/s 2023-03-09T19:45:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:45:51 smithi181 ceph-mon[94189]: pgmap v9718: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 474 KiB/s rd, 445 KiB/s wr, 14 op/s 2023-03-09T19:45:51.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:45:51 smithi038 ceph-mon[92422]: pgmap v9718: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 474 KiB/s rd, 445 KiB/s wr, 14 op/s 2023-03-09T19:45:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17725.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:45:54.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:45:54 smithi122 ceph-mon[93869]: pgmap v9719: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 456 KiB/s rd, 429 KiB/s wr, 12 op/s 2023-03-09T19:45:54.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:45:54 smithi181 ceph-mon[94189]: pgmap v9719: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 456 KiB/s rd, 429 KiB/s wr, 12 op/s 2023-03-09T19:45:54.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:45:54 smithi038 ceph-mon[92422]: pgmap v9719: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 456 KiB/s rd, 429 KiB/s wr, 12 op/s 2023-03-09T19:45:56.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:45:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17266.207180 secs 2023-03-09T19:45:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:45:56 smithi122 ceph-mon[93869]: pgmap v9720: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 471 KiB/s rd, 457 KiB/s wr, 12 op/s 2023-03-09T19:45:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:45:56 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:45:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:45:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17266.207180 secs 2023-03-09T19:45:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:45:56 smithi181 ceph-mon[94189]: pgmap v9720: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 471 KiB/s rd, 457 KiB/s wr, 12 op/s 2023-03-09T19:45:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:45:56 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:45:56.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:45:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17266.207180 secs 2023-03-09T19:45:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:45:56 smithi038 ceph-mon[92422]: pgmap v9720: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 471 KiB/s rd, 457 KiB/s wr, 12 op/s 2023-03-09T19:45:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:45:56 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:45:57.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:45:57 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:45:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:45:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:45:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:45:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:45:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:45:57 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:45:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:45:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:45:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:45:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:45:57.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:45:57 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:45:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:45:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:45:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:45:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:45:58.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:45:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17268.517732 secs 2023-03-09T19:45:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:45:58 smithi038 ceph-mon[92422]: pgmap v9721: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 508 KiB/s wr, 13 op/s 2023-03-09T19:45:58.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17730.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:45:58.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:45:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17268.517732 secs 2023-03-09T19:45:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:45:58 smithi122 ceph-mon[93869]: pgmap v9721: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 508 KiB/s wr, 13 op/s 2023-03-09T19:45:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:45:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17268.517732 secs 2023-03-09T19:45:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:45:58 smithi181 ceph-mon[94189]: pgmap v9721: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 508 KiB/s wr, 13 op/s 2023-03-09T19:46:00.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:00 smithi038 ceph-mon[92422]: pgmap v9722: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 300 KiB/s wr, 10 op/s 2023-03-09T19:46:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17271.207242 secs 2023-03-09T19:46:00.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:00 smithi122 ceph-mon[93869]: pgmap v9722: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 300 KiB/s wr, 10 op/s 2023-03-09T19:46:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17271.207242 secs 2023-03-09T19:46:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:00 smithi181 ceph-mon[94189]: pgmap v9722: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 300 KiB/s wr, 10 op/s 2023-03-09T19:46:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17271.207242 secs 2023-03-09T19:46:01.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:46:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:46:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:46:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:46:01.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:46:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:46:02.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:02 smithi122 ceph-mon[93869]: pgmap v9723: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 455 KiB/s wr, 15 op/s 2023-03-09T19:46:02.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:02 smithi181 ceph-mon[94189]: pgmap v9723: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 455 KiB/s wr, 15 op/s 2023-03-09T19:46:02.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:02 smithi038 ceph-mon[92422]: pgmap v9723: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 455 KiB/s wr, 15 op/s 2023-03-09T19:46:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17735.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:46:03.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17273.517847 secs 2023-03-09T19:46:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:03 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:46:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17273.517847 secs 2023-03-09T19:46:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:03 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:46:03.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17273.517847 secs 2023-03-09T19:46:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:03 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:46:04.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:04 smithi122 ceph-mon[93869]: pgmap v9724: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 285 KiB/s wr, 23 op/s 2023-03-09T19:46:04.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:04 smithi181 ceph-mon[94189]: pgmap v9724: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 285 KiB/s wr, 23 op/s 2023-03-09T19:46:04.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:04 smithi038 ceph-mon[92422]: pgmap v9724: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 285 KiB/s wr, 23 op/s 2023-03-09T19:46:05.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17276.207316 secs 2023-03-09T19:46:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17276.207316 secs 2023-03-09T19:46:05.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17276.207316 secs 2023-03-09T19:46:06.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:06 smithi122 ceph-mon[93869]: pgmap v9725: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 286 KiB/s wr, 23 op/s 2023-03-09T19:46:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:06 smithi181 ceph-mon[94189]: pgmap v9725: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 286 KiB/s wr, 23 op/s 2023-03-09T19:46:06.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:06 smithi038 ceph-mon[92422]: pgmap v9725: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 286 KiB/s wr, 23 op/s 2023-03-09T19:46:07.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:07 smithi122 ceph-mon[93869]: Health check failed: 1 MDSs behind on trimming (MDS_TRIM) 2023-03-09T19:46:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:46:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:46:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:07 smithi181 ceph-mon[94189]: Health check failed: 1 MDSs behind on trimming (MDS_TRIM) 2023-03-09T19:46:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:46:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:46:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:07 smithi038 ceph-mon[92422]: Health check failed: 1 MDSs behind on trimming (MDS_TRIM) 2023-03-09T19:46:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:46:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:46:08.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17740.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:46:08.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17278.517976 secs 2023-03-09T19:46:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:08 smithi122 ceph-mon[93869]: pgmap v9726: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 307 KiB/s wr, 25 op/s 2023-03-09T19:46:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:46:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:46:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17278.517976 secs 2023-03-09T19:46:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:08 smithi181 ceph-mon[94189]: pgmap v9726: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 307 KiB/s wr, 25 op/s 2023-03-09T19:46:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:46:08.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:46:08.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17278.517976 secs 2023-03-09T19:46:08.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:08 smithi038 ceph-mon[92422]: pgmap v9726: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 307 KiB/s wr, 25 op/s 2023-03-09T19:46:08.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:46:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:46:10.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:10 smithi122 ceph-mon[93869]: pgmap v9727: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 256 KiB/s wr, 23 op/s 2023-03-09T19:46:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:10 smithi122 ceph-mon[93869]: osdmap e821: 12 total, 12 up, 12 in 2023-03-09T19:46:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17281.207375 secs 2023-03-09T19:46:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:10 smithi181 ceph-mon[94189]: pgmap v9727: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 256 KiB/s wr, 23 op/s 2023-03-09T19:46:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:10 smithi181 ceph-mon[94189]: osdmap e821: 12 total, 12 up, 12 in 2023-03-09T19:46:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17281.207375 secs 2023-03-09T19:46:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:10 smithi038 ceph-mon[92422]: pgmap v9727: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 256 KiB/s wr, 23 op/s 2023-03-09T19:46:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:10 smithi038 ceph-mon[92422]: osdmap e821: 12 total, 12 up, 12 in 2023-03-09T19:46:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17281.207375 secs 2023-03-09T19:46:12.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:12 smithi122 ceph-mon[93869]: pgmap v9729: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 104 KiB/s wr, 23 op/s 2023-03-09T19:46:12.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:12 smithi181 ceph-mon[94189]: pgmap v9729: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 104 KiB/s wr, 23 op/s 2023-03-09T19:46:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:12 smithi038 ceph-mon[92422]: pgmap v9729: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 104 KiB/s wr, 23 op/s 2023-03-09T19:46:13.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17745.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:46:13.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17283.518086 secs 2023-03-09T19:46:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:13 smithi122 ceph-mon[93869]: Health check update: 2 MDSs behind on trimming (MDS_TRIM) 2023-03-09T19:46:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17283.518086 secs 2023-03-09T19:46:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:13 smithi181 ceph-mon[94189]: Health check update: 2 MDSs behind on trimming (MDS_TRIM) 2023-03-09T19:46:13.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17283.518086 secs 2023-03-09T19:46:13.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:13 smithi038 ceph-mon[92422]: Health check update: 2 MDSs behind on trimming (MDS_TRIM) 2023-03-09T19:46:14.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:14 smithi122 ceph-mon[93869]: pgmap v9730: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 78 KiB/s wr, 8 op/s 2023-03-09T19:46:14.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:14 smithi181 ceph-mon[94189]: pgmap v9730: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 78 KiB/s wr, 8 op/s 2023-03-09T19:46:14.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:14 smithi038 ceph-mon[92422]: pgmap v9730: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 78 KiB/s wr, 8 op/s 2023-03-09T19:46:15.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17286.207439 secs 2023-03-09T19:46:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17286.207439 secs 2023-03-09T19:46:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17286.207439 secs 2023-03-09T19:46:16.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:16 smithi122 ceph-mon[93869]: pgmap v9731: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 94 KiB/s wr, 8 op/s 2023-03-09T19:46:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:16 smithi181 ceph-mon[94189]: pgmap v9731: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 94 KiB/s wr, 8 op/s 2023-03-09T19:46:16.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:16 smithi038 ceph-mon[92422]: pgmap v9731: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 94 KiB/s wr, 8 op/s 2023-03-09T19:46:18.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17750.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:46:18.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17288.518194 secs 2023-03-09T19:46:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:18 smithi122 ceph-mon[93869]: pgmap v9732: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 85 KiB/s wr, 8 op/s 2023-03-09T19:46:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:18 smithi122 ceph-mon[93869]: osdmap e822: 12 total, 12 up, 12 in 2023-03-09T19:46:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17288.518194 secs 2023-03-09T19:46:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:18 smithi181 ceph-mon[94189]: pgmap v9732: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 85 KiB/s wr, 8 op/s 2023-03-09T19:46:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:18 smithi181 ceph-mon[94189]: osdmap e822: 12 total, 12 up, 12 in 2023-03-09T19:46:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17288.518194 secs 2023-03-09T19:46:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:18 smithi038 ceph-mon[92422]: pgmap v9732: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 85 KiB/s wr, 8 op/s 2023-03-09T19:46:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:18 smithi038 ceph-mon[92422]: osdmap e822: 12 total, 12 up, 12 in 2023-03-09T19:46:20.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:20 smithi122 ceph-mon[93869]: pgmap v9734: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 69 KiB/s wr, 8 op/s 2023-03-09T19:46:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17291.207498 secs 2023-03-09T19:46:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:20 smithi181 ceph-mon[94189]: pgmap v9734: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 69 KiB/s wr, 8 op/s 2023-03-09T19:46:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17291.207498 secs 2023-03-09T19:46:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:20 smithi038 ceph-mon[92422]: pgmap v9734: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 69 KiB/s wr, 8 op/s 2023-03-09T19:46:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17291.207498 secs 2023-03-09T19:46:22.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:22 smithi122 ceph-mon[93869]: pgmap v9735: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 68 KiB/s wr, 7 op/s 2023-03-09T19:46:22.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:22 smithi181 ceph-mon[94189]: pgmap v9735: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 68 KiB/s wr, 7 op/s 2023-03-09T19:46:22.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:22 smithi038 ceph-mon[92422]: pgmap v9735: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 68 KiB/s wr, 7 op/s 2023-03-09T19:46:23.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17755.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:46:23.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17293.518257 secs 2023-03-09T19:46:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17293.518257 secs 2023-03-09T19:46:23.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17293.518257 secs 2023-03-09T19:46:24.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:24 smithi122 ceph-mon[93869]: pgmap v9736: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 68 KiB/s wr, 7 op/s 2023-03-09T19:46:24.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:24 smithi181 ceph-mon[94189]: pgmap v9736: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 68 KiB/s wr, 7 op/s 2023-03-09T19:46:24.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:24 smithi038 ceph-mon[92422]: pgmap v9736: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 68 KiB/s wr, 7 op/s 2023-03-09T19:46:25.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17296.207583 secs 2023-03-09T19:46:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17296.207583 secs 2023-03-09T19:46:25.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17296.207583 secs 2023-03-09T19:46:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:26 smithi122 ceph-mon[93869]: pgmap v9737: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 52 KiB/s wr, 7 op/s 2023-03-09T19:46:26.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:26 smithi181 ceph-mon[94189]: pgmap v9737: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 52 KiB/s wr, 7 op/s 2023-03-09T19:46:26.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:26 smithi038 ceph-mon[92422]: pgmap v9737: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 52 KiB/s wr, 7 op/s 2023-03-09T19:46:27.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:46:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:46:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:46:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:46:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:46:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:46:27.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:46:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:46:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:46:28.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17760.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:46:28.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17298.518330 secs 2023-03-09T19:46:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:28 smithi122 ceph-mon[93869]: pgmap v9738: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 75 KiB/s wr, 7 op/s 2023-03-09T19:46:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:46:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:46:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:46:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17298.518330 secs 2023-03-09T19:46:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:28 smithi181 ceph-mon[94189]: pgmap v9738: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 75 KiB/s wr, 7 op/s 2023-03-09T19:46:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:46:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:46:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:46:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17298.518330 secs 2023-03-09T19:46:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:28 smithi038 ceph-mon[92422]: pgmap v9738: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 75 KiB/s wr, 7 op/s 2023-03-09T19:46:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:46:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:46:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:46:30.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:30 smithi122 ceph-mon[93869]: pgmap v9739: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 67 KiB/s wr, 7 op/s 2023-03-09T19:46:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17301.207642 secs 2023-03-09T19:46:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:30 smithi181 ceph-mon[94189]: pgmap v9739: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 67 KiB/s wr, 7 op/s 2023-03-09T19:46:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17301.207642 secs 2023-03-09T19:46:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:30 smithi038 ceph-mon[92422]: pgmap v9739: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 67 KiB/s wr, 7 op/s 2023-03-09T19:46:30.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17301.207642 secs 2023-03-09T19:46:31.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:31 smithi122 ceph-mon[93869]: pgmap v9740: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 62 KiB/s wr, 8 op/s 2023-03-09T19:46:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:31 smithi181 ceph-mon[94189]: pgmap v9740: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 62 KiB/s wr, 8 op/s 2023-03-09T19:46:31.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:31 smithi038 ceph-mon[92422]: pgmap v9740: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 62 KiB/s wr, 8 op/s 2023-03-09T19:46:33.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17303.518445 secs 2023-03-09T19:46:33.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17303.518445 secs 2023-03-09T19:46:33.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17303.518445 secs 2023-03-09T19:46:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17765.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:46:34.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:34 smithi181 ceph-mon[94189]: pgmap v9741: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 62 KiB/s wr, 6 op/s 2023-03-09T19:46:34.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:34 smithi038 ceph-mon[92422]: pgmap v9741: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 62 KiB/s wr, 6 op/s 2023-03-09T19:46:34.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:34 smithi122 ceph-mon[93869]: pgmap v9741: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 62 KiB/s wr, 6 op/s 2023-03-09T19:46:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17306.207730 secs 2023-03-09T19:46:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:36 smithi038 ceph-mon[92422]: pgmap v9742: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 62 KiB/s wr, 6 op/s 2023-03-09T19:46:36.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17306.207730 secs 2023-03-09T19:46:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:36 smithi122 ceph-mon[93869]: pgmap v9742: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 62 KiB/s wr, 6 op/s 2023-03-09T19:46:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17306.207730 secs 2023-03-09T19:46:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:36 smithi181 ceph-mon[94189]: pgmap v9742: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 62 KiB/s wr, 6 op/s 2023-03-09T19:46:38.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:38 smithi038 ceph-mon[92422]: pgmap v9743: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 62 KiB/s wr, 8 op/s 2023-03-09T19:46:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17770.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:46:38.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:38 smithi122 ceph-mon[93869]: pgmap v9743: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 62 KiB/s wr, 8 op/s 2023-03-09T19:46:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:38 smithi181 ceph-mon[94189]: pgmap v9743: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 62 KiB/s wr, 8 op/s 2023-03-09T19:46:40.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:40 smithi038 ceph-mon[92422]: pgmap v9744: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 6 op/s 2023-03-09T19:46:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17311.207788 secs 2023-03-09T19:46:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:40 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:46:40.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:40 smithi122 ceph-mon[93869]: pgmap v9744: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 6 op/s 2023-03-09T19:46:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17311.207788 secs 2023-03-09T19:46:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:40 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:46:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:40 smithi181 ceph-mon[94189]: pgmap v9744: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 6 op/s 2023-03-09T19:46:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17311.207788 secs 2023-03-09T19:46:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:40 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:46:41.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:41 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:46:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:46:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:46:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:41 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:46:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:46:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:46:41.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:41 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:46:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:46:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:46:42.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:42 smithi122 ceph-mon[93869]: pgmap v9745: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 110 KiB/s wr, 7 op/s 2023-03-09T19:46:42.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:42 smithi181 ceph-mon[94189]: pgmap v9745: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 110 KiB/s wr, 7 op/s 2023-03-09T19:46:42.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:42 smithi038 ceph-mon[92422]: pgmap v9745: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 110 KiB/s wr, 7 op/s 2023-03-09T19:46:43.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17775.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:46:44.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:44 smithi038 ceph-mon[92422]: pgmap v9746: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 111 KiB/s wr, 5 op/s 2023-03-09T19:46:44.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:44 smithi122 ceph-mon[93869]: pgmap v9746: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 111 KiB/s wr, 5 op/s 2023-03-09T19:46:44.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:44 smithi181 ceph-mon[94189]: pgmap v9746: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 111 KiB/s wr, 5 op/s 2023-03-09T19:46:46.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17316.207849 secs 2023-03-09T19:46:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:46 smithi038 ceph-mon[92422]: pgmap v9747: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 125 KiB/s wr, 6 op/s 2023-03-09T19:46:46.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17316.207849 secs 2023-03-09T19:46:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:46 smithi122 ceph-mon[93869]: pgmap v9747: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 125 KiB/s wr, 6 op/s 2023-03-09T19:46:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17316.207849 secs 2023-03-09T19:46:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:46 smithi181 ceph-mon[94189]: pgmap v9747: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 125 KiB/s wr, 6 op/s 2023-03-09T19:46:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:48 smithi038 ceph-mon[92422]: pgmap v9748: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 211 KiB/s wr, 9 op/s 2023-03-09T19:46:48.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17780.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:46:48.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:48 smithi122 ceph-mon[93869]: pgmap v9748: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 211 KiB/s wr, 9 op/s 2023-03-09T19:46:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:48 smithi181 ceph-mon[94189]: pgmap v9748: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 211 KiB/s wr, 9 op/s 2023-03-09T19:46:49.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:46:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:46:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:46:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:46:49.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:46:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:46:50.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:50 smithi122 ceph-mon[93869]: pgmap v9749: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 225 KiB/s wr, 7 op/s 2023-03-09T19:46:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17321.207913 secs 2023-03-09T19:46:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:50 smithi181 ceph-mon[94189]: pgmap v9749: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 225 KiB/s wr, 7 op/s 2023-03-09T19:46:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17321.207913 secs 2023-03-09T19:46:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:50 smithi038 ceph-mon[92422]: pgmap v9749: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 225 KiB/s wr, 7 op/s 2023-03-09T19:46:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17321.207913 secs 2023-03-09T19:46:51.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:46:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:46:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:46:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:46:51.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:46:51.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:46:52.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:52 smithi122 ceph-mon[93869]: pgmap v9750: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 517 KiB/s rd, 291 KiB/s wr, 10 op/s 2023-03-09T19:46:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:52 smithi181 ceph-mon[94189]: pgmap v9750: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 517 KiB/s rd, 291 KiB/s wr, 10 op/s 2023-03-09T19:46:53.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:52 smithi038 ceph-mon[92422]: pgmap v9750: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 517 KiB/s rd, 291 KiB/s wr, 10 op/s 2023-03-09T19:46:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17785.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:46:54.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:54 smithi038 ceph-mon[92422]: pgmap v9751: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 182 KiB/s wr, 9 op/s 2023-03-09T19:46:54.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:54 smithi122 ceph-mon[93869]: pgmap v9751: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 182 KiB/s wr, 9 op/s 2023-03-09T19:46:54.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:54 smithi181 ceph-mon[94189]: pgmap v9751: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 182 KiB/s wr, 9 op/s 2023-03-09T19:46:56.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17326.207975 secs 2023-03-09T19:46:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:56 smithi038 ceph-mon[92422]: pgmap v9752: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 196 KiB/s wr, 9 op/s 2023-03-09T19:46:56.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17326.207975 secs 2023-03-09T19:46:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:56 smithi122 ceph-mon[93869]: pgmap v9752: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 196 KiB/s wr, 9 op/s 2023-03-09T19:46:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17326.207975 secs 2023-03-09T19:46:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:56 smithi181 ceph-mon[94189]: pgmap v9752: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 196 KiB/s wr, 9 op/s 2023-03-09T19:46:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17790.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:46:58.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:46:58 smithi122 ceph-mon[93869]: pgmap v9753: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 539 KiB/s rd, 345 KiB/s wr, 12 op/s 2023-03-09T19:46:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:46:58 smithi181 ceph-mon[94189]: pgmap v9753: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 539 KiB/s rd, 345 KiB/s wr, 12 op/s 2023-03-09T19:46:58.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:46:58 smithi038 ceph-mon[92422]: pgmap v9753: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 539 KiB/s rd, 345 KiB/s wr, 12 op/s 2023-03-09T19:47:00.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:00 smithi122 ceph-mon[93869]: pgmap v9754: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 259 KiB/s wr, 10 op/s 2023-03-09T19:47:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17331.208036 secs 2023-03-09T19:47:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:00 smithi181 ceph-mon[94189]: pgmap v9754: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 259 KiB/s wr, 10 op/s 2023-03-09T19:47:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17331.208036 secs 2023-03-09T19:47:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:00 smithi038 ceph-mon[92422]: pgmap v9754: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 259 KiB/s wr, 10 op/s 2023-03-09T19:47:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17331.208036 secs 2023-03-09T19:47:02.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:02 smithi122 ceph-mon[93869]: pgmap v9755: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 271 KiB/s wr, 12 op/s 2023-03-09T19:47:02.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:02 smithi181 ceph-mon[94189]: pgmap v9755: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 271 KiB/s wr, 12 op/s 2023-03-09T19:47:02.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:02 smithi038 ceph-mon[92422]: pgmap v9755: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 271 KiB/s wr, 12 op/s 2023-03-09T19:47:03.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17795.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:47:04.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:04 smithi038 ceph-mon[92422]: pgmap v9756: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 248 KiB/s wr, 20 op/s 2023-03-09T19:47:04.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:04 smithi122 ceph-mon[93869]: pgmap v9756: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 248 KiB/s wr, 20 op/s 2023-03-09T19:47:04.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:04 smithi181 ceph-mon[94189]: pgmap v9756: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 248 KiB/s wr, 20 op/s 2023-03-09T19:47:06.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17336.208098 secs 2023-03-09T19:47:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:06 smithi122 ceph-mon[93869]: pgmap v9757: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 262 KiB/s wr, 20 op/s 2023-03-09T19:47:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17336.208098 secs 2023-03-09T19:47:06.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:06 smithi181 ceph-mon[94189]: pgmap v9757: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 262 KiB/s wr, 20 op/s 2023-03-09T19:47:06.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17336.208098 secs 2023-03-09T19:47:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:06 smithi038 ceph-mon[92422]: pgmap v9757: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 262 KiB/s wr, 20 op/s 2023-03-09T19:47:07.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:47:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:47:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:47:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:47:07.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:47:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:47:08.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17800.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:47:08.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:08 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 17338.519221 secs 2023-03-09T19:47:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:08 smithi122 ceph-mon[93869]: pgmap v9758: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 446 KiB/s wr, 25 op/s 2023-03-09T19:47:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:08 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 17338.519221 secs 2023-03-09T19:47:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:08 smithi181 ceph-mon[94189]: pgmap v9758: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 446 KiB/s wr, 25 op/s 2023-03-09T19:47:08.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:08 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 17338.519221 secs 2023-03-09T19:47:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:08 smithi038 ceph-mon[92422]: pgmap v9758: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 446 KiB/s wr, 25 op/s 2023-03-09T19:47:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:09 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:47:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:47:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:47:09.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:47:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:09 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:47:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:47:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:47:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:47:09.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:09 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:47:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:47:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:47:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:47:10.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:10 smithi122 ceph-mon[93869]: pgmap v9759: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 297 KiB/s wr, 22 op/s 2023-03-09T19:47:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17341.208157 secs 2023-03-09T19:47:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:10 smithi181 ceph-mon[94189]: pgmap v9759: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 297 KiB/s wr, 22 op/s 2023-03-09T19:47:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17341.208157 secs 2023-03-09T19:47:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:10 smithi038 ceph-mon[92422]: pgmap v9759: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 297 KiB/s wr, 22 op/s 2023-03-09T19:47:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17341.208157 secs 2023-03-09T19:47:12.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:12 smithi122 ceph-mon[93869]: pgmap v9760: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 297 KiB/s wr, 23 op/s 2023-03-09T19:47:12.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:12 smithi181 ceph-mon[94189]: pgmap v9760: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 297 KiB/s wr, 23 op/s 2023-03-09T19:47:12.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:12 smithi038 ceph-mon[92422]: pgmap v9760: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 297 KiB/s wr, 23 op/s 2023-03-09T19:47:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17805.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:47:13.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:13 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 17343.519330 secs 2023-03-09T19:47:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:13 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 17343.519330 secs 2023-03-09T19:47:13.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:13 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 17343.519330 secs 2023-03-09T19:47:14.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:14 smithi122 ceph-mon[93869]: pgmap v9761: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 312 KiB/s wr, 21 op/s 2023-03-09T19:47:14.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:14 smithi181 ceph-mon[94189]: pgmap v9761: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 312 KiB/s wr, 21 op/s 2023-03-09T19:47:14.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:14 smithi038 ceph-mon[92422]: pgmap v9761: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 312 KiB/s wr, 21 op/s 2023-03-09T19:47:15.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17346.208239 secs 2023-03-09T19:47:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17346.208239 secs 2023-03-09T19:47:15.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17346.208239 secs 2023-03-09T19:47:16.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:16 smithi122 ceph-mon[93869]: pgmap v9762: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 270 KiB/s wr, 10 op/s 2023-03-09T19:47:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:16 smithi181 ceph-mon[94189]: pgmap v9762: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 270 KiB/s wr, 10 op/s 2023-03-09T19:47:16.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:16 smithi038 ceph-mon[92422]: pgmap v9762: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 270 KiB/s wr, 10 op/s 2023-03-09T19:47:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17810.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:47:18.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:18 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 17348.519480 secs 2023-03-09T19:47:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:18 smithi122 ceph-mon[93869]: pgmap v9763: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 290 KiB/s wr, 12 op/s 2023-03-09T19:47:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:18 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 17348.519480 secs 2023-03-09T19:47:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:18 smithi181 ceph-mon[94189]: pgmap v9763: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 290 KiB/s wr, 12 op/s 2023-03-09T19:47:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:18 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 17348.519480 secs 2023-03-09T19:47:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:18 smithi038 ceph-mon[92422]: pgmap v9763: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 290 KiB/s wr, 12 op/s 2023-03-09T19:47:20.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:47:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:47:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:20 smithi038 ceph-mon[92422]: pgmap v9764: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 116 KiB/s wr, 9 op/s 2023-03-09T19:47:20.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:47:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:47:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:20 smithi122 ceph-mon[93869]: pgmap v9764: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 116 KiB/s wr, 9 op/s 2023-03-09T19:47:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:47:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:47:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:20 smithi181 ceph-mon[94189]: pgmap v9764: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 116 KiB/s wr, 9 op/s 2023-03-09T19:47:21.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17351.208299 secs 2023-03-09T19:47:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:47:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:47:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17351.208299 secs 2023-03-09T19:47:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:47:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:47:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17351.208299 secs 2023-03-09T19:47:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:47:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:47:22.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:22 smithi122 ceph-mon[93869]: pgmap v9765: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 261 KiB/s wr, 13 op/s 2023-03-09T19:47:22.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:22 smithi181 ceph-mon[94189]: pgmap v9765: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 261 KiB/s wr, 13 op/s 2023-03-09T19:47:22.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:22 smithi038 ceph-mon[92422]: pgmap v9765: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 261 KiB/s wr, 13 op/s 2023-03-09T19:47:23.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17353.519592 secs 2023-03-09T19:47:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17353.519592 secs 2023-03-09T19:47:23.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17353.519592 secs 2023-03-09T19:47:23.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17815.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:47:24.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:24 smithi122 ceph-mon[93869]: pgmap v9766: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 308 KiB/s wr, 12 op/s 2023-03-09T19:47:24.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:24 smithi181 ceph-mon[94189]: pgmap v9766: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 308 KiB/s wr, 12 op/s 2023-03-09T19:47:24.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:24 smithi038 ceph-mon[92422]: pgmap v9766: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 308 KiB/s wr, 12 op/s 2023-03-09T19:47:25.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:25 smithi122 ceph-mon[93869]: osdmap e823: 12 total, 12 up, 12 in 2023-03-09T19:47:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:25 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:47:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:25 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:47:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:25 smithi181 ceph-mon[94189]: osdmap e823: 12 total, 12 up, 12 in 2023-03-09T19:47:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:25 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:47:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:25 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:47:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:25 smithi038 ceph-mon[92422]: osdmap e823: 12 total, 12 up, 12 in 2023-03-09T19:47:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:25 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:47:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:25 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:47:26.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17356.208359 secs 2023-03-09T19:47:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:26 smithi122 ceph-mon[93869]: pgmap v9768: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 320 KiB/s wr, 13 op/s 2023-03-09T19:47:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17356.208359 secs 2023-03-09T19:47:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:26 smithi181 ceph-mon[94189]: pgmap v9768: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 320 KiB/s wr, 13 op/s 2023-03-09T19:47:26.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17356.208359 secs 2023-03-09T19:47:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:26 smithi038 ceph-mon[92422]: pgmap v9768: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 320 KiB/s wr, 13 op/s 2023-03-09T19:47:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17820.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:47:28.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17358.519703 secs 2023-03-09T19:47:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:28 smithi122 ceph-mon[93869]: pgmap v9769: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 342 KiB/s wr, 14 op/s 2023-03-09T19:47:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:47:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17358.519703 secs 2023-03-09T19:47:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:28 smithi181 ceph-mon[94189]: pgmap v9769: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 342 KiB/s wr, 14 op/s 2023-03-09T19:47:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:47:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17358.519703 secs 2023-03-09T19:47:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:28 smithi038 ceph-mon[92422]: pgmap v9769: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 342 KiB/s wr, 14 op/s 2023-03-09T19:47:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:47:29.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:47:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:47:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:47:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:47:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:29 smithi122 ceph-mon[93869]: pgmap v9770: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 311 KiB/s wr, 12 op/s 2023-03-09T19:47:29.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:47:29.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:47:29.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:47:29.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:47:29.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:29 smithi181 ceph-mon[94189]: pgmap v9770: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 311 KiB/s wr, 12 op/s 2023-03-09T19:47:29.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:47:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:47:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:47:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:47:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:29 smithi038 ceph-mon[92422]: pgmap v9770: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 311 KiB/s wr, 12 op/s 2023-03-09T19:47:30.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17361.208435 secs 2023-03-09T19:47:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:30 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:47:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:30 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:47:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:30 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:47:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17361.208435 secs 2023-03-09T19:47:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:30 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:47:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:30 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:47:30.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:30 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:47:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17361.208435 secs 2023-03-09T19:47:30.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:30 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:47:30.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:30 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:47:30.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:30 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:47:31.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:31 smithi122 ceph-mon[93869]: pgmap v9771: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 230 KiB/s wr, 10 op/s 2023-03-09T19:47:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:31 smithi181 ceph-mon[94189]: pgmap v9771: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 230 KiB/s wr, 10 op/s 2023-03-09T19:47:31.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:31 smithi038 ceph-mon[92422]: pgmap v9771: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 230 KiB/s wr, 10 op/s 2023-03-09T19:47:33.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17363.519841 secs 2023-03-09T19:47:33.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17363.519841 secs 2023-03-09T19:47:33.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17363.519841 secs 2023-03-09T19:47:33.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17825.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:47:34.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:34 smithi038 ceph-mon[92422]: osdmap e824: 12 total, 12 up, 12 in 2023-03-09T19:47:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:34 smithi038 ceph-mon[92422]: pgmap v9773: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 189 KiB/s wr, 10 op/s 2023-03-09T19:47:34.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:34 smithi122 ceph-mon[93869]: osdmap e824: 12 total, 12 up, 12 in 2023-03-09T19:47:34.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:34 smithi122 ceph-mon[93869]: pgmap v9773: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 189 KiB/s wr, 10 op/s 2023-03-09T19:47:34.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:34 smithi181 ceph-mon[94189]: osdmap e824: 12 total, 12 up, 12 in 2023-03-09T19:47:34.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:34 smithi181 ceph-mon[94189]: pgmap v9773: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 189 KiB/s wr, 10 op/s 2023-03-09T19:47:35.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17366.208497 secs 2023-03-09T19:47:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17366.208497 secs 2023-03-09T19:47:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17366.208497 secs 2023-03-09T19:47:36.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:36 smithi122 ceph-mon[93869]: pgmap v9774: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 174 KiB/s wr, 9 op/s 2023-03-09T19:47:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:36 smithi181 ceph-mon[94189]: pgmap v9774: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 174 KiB/s wr, 9 op/s 2023-03-09T19:47:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:36 smithi038 ceph-mon[92422]: pgmap v9774: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 174 KiB/s wr, 9 op/s 2023-03-09T19:47:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17830.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:47:38.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:38 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 17368.519957 secs 2023-03-09T19:47:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:38 smithi122 ceph-mon[93869]: pgmap v9775: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 175 KiB/s wr, 9 op/s 2023-03-09T19:47:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:38 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 17368.519957 secs 2023-03-09T19:47:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:38 smithi181 ceph-mon[94189]: pgmap v9775: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 175 KiB/s wr, 9 op/s 2023-03-09T19:47:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:38 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 17368.519957 secs 2023-03-09T19:47:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:38 smithi038 ceph-mon[92422]: pgmap v9775: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 175 KiB/s wr, 9 op/s 2023-03-09T19:47:40.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:40 smithi122 ceph-mon[93869]: pgmap v9776: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 175 KiB/s wr, 9 op/s 2023-03-09T19:47:40.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:47:40.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:47:40.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17371.208556 secs 2023-03-09T19:47:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:40 smithi181 ceph-mon[94189]: pgmap v9776: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 175 KiB/s wr, 9 op/s 2023-03-09T19:47:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:47:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:47:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17371.208556 secs 2023-03-09T19:47:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:40 smithi038 ceph-mon[92422]: pgmap v9776: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 175 KiB/s wr, 9 op/s 2023-03-09T19:47:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:47:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:47:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17371.208556 secs 2023-03-09T19:47:41.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:47:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:47:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:47:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:47:41.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:47:41.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:47:42.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:42 smithi122 ceph-mon[93869]: pgmap v9777: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 209 KiB/s wr, 10 op/s 2023-03-09T19:47:42.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:42 smithi181 ceph-mon[94189]: pgmap v9777: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 209 KiB/s wr, 10 op/s 2023-03-09T19:47:42.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:42 smithi038 ceph-mon[92422]: pgmap v9777: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 209 KiB/s wr, 10 op/s 2023-03-09T19:47:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17835.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:47:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17373.520077 secs 2023-03-09T19:47:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17373.520077 secs 2023-03-09T19:47:43.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17373.520077 secs 2023-03-09T19:47:44.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:44 smithi122 ceph-mon[93869]: pgmap v9778: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 220 KiB/s wr, 9 op/s 2023-03-09T19:47:44.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:44 smithi181 ceph-mon[94189]: pgmap v9778: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 220 KiB/s wr, 9 op/s 2023-03-09T19:47:44.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:44 smithi038 ceph-mon[92422]: pgmap v9778: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 220 KiB/s wr, 9 op/s 2023-03-09T19:47:45.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:45 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:47:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:45 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:47:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17376.208634 secs 2023-03-09T19:47:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:45 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:47:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:45 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:47:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17376.208634 secs 2023-03-09T19:47:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:45 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:47:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:45 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:47:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17376.208634 secs 2023-03-09T19:47:46.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:46 smithi122 ceph-mon[93869]: pgmap v9779: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 187 KiB/s wr, 8 op/s 2023-03-09T19:47:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:46 smithi181 ceph-mon[94189]: pgmap v9779: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 187 KiB/s wr, 8 op/s 2023-03-09T19:47:46.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:46 smithi038 ceph-mon[92422]: pgmap v9779: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 187 KiB/s wr, 8 op/s 2023-03-09T19:47:47.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:47:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:47:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:47:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:47:47.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:47:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:47:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17840.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:47:48.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17378.520218 secs 2023-03-09T19:47:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:48 smithi122 ceph-mon[93869]: pgmap v9780: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 261 KiB/s wr, 11 op/s 2023-03-09T19:47:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17378.520218 secs 2023-03-09T19:47:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:48 smithi181 ceph-mon[94189]: pgmap v9780: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 261 KiB/s wr, 11 op/s 2023-03-09T19:47:48.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17378.520218 secs 2023-03-09T19:47:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:48 smithi038 ceph-mon[92422]: pgmap v9780: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 261 KiB/s wr, 11 op/s 2023-03-09T19:47:50.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:50 smithi122 ceph-mon[93869]: pgmap v9781: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 222 KiB/s wr, 10 op/s 2023-03-09T19:47:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17381.208709 secs 2023-03-09T19:47:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:50 smithi181 ceph-mon[94189]: pgmap v9781: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 222 KiB/s wr, 10 op/s 2023-03-09T19:47:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17381.208709 secs 2023-03-09T19:47:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:50 smithi038 ceph-mon[92422]: pgmap v9781: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 222 KiB/s wr, 10 op/s 2023-03-09T19:47:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17381.208709 secs 2023-03-09T19:47:52.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:52 smithi122 ceph-mon[93869]: pgmap v9782: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 371 KiB/s wr, 12 op/s 2023-03-09T19:47:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:52 smithi181 ceph-mon[94189]: pgmap v9782: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 371 KiB/s wr, 12 op/s 2023-03-09T19:47:52.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:52 smithi038 ceph-mon[92422]: pgmap v9782: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 371 KiB/s wr, 12 op/s 2023-03-09T19:47:53.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17845.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:47:53.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17383.520335 secs 2023-03-09T19:47:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17383.520335 secs 2023-03-09T19:47:53.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17383.520335 secs 2023-03-09T19:47:54.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:54 smithi122 ceph-mon[93869]: pgmap v9783: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 282 KiB/s wr, 11 op/s 2023-03-09T19:47:54.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:54 smithi181 ceph-mon[94189]: pgmap v9783: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 282 KiB/s wr, 11 op/s 2023-03-09T19:47:54.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:54 smithi038 ceph-mon[92422]: pgmap v9783: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 282 KiB/s wr, 11 op/s 2023-03-09T19:47:56.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17386.208792 secs 2023-03-09T19:47:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:47:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:47:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:56 smithi122 ceph-mon[93869]: pgmap v9784: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 283 KiB/s rd, 268 KiB/s wr, 10 op/s 2023-03-09T19:47:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17386.208792 secs 2023-03-09T19:47:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:47:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:47:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:56 smithi181 ceph-mon[94189]: pgmap v9784: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 283 KiB/s rd, 268 KiB/s wr, 10 op/s 2023-03-09T19:47:56.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17386.208792 secs 2023-03-09T19:47:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:47:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:47:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:56 smithi038 ceph-mon[92422]: pgmap v9784: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 283 KiB/s rd, 268 KiB/s wr, 10 op/s 2023-03-09T19:47:57.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:47:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:47:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:47:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:47:57.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:47:57.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:47:58.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17388.520452 secs 2023-03-09T19:47:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:47:58 smithi122 ceph-mon[93869]: pgmap v9785: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 392 KiB/s wr, 13 op/s 2023-03-09T19:47:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17388.520452 secs 2023-03-09T19:47:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:47:58 smithi181 ceph-mon[94189]: pgmap v9785: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 392 KiB/s wr, 13 op/s 2023-03-09T19:47:58.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17388.520452 secs 2023-03-09T19:47:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:47:58 smithi038 ceph-mon[92422]: pgmap v9785: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 392 KiB/s wr, 13 op/s 2023-03-09T19:47:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17850.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:48:00.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:48:00 smithi122 ceph-mon[93869]: pgmap v9786: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 346 KiB/s wr, 10 op/s 2023-03-09T19:48:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:48:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17391.208854 secs 2023-03-09T19:48:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:48:00 smithi181 ceph-mon[94189]: pgmap v9786: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 346 KiB/s wr, 10 op/s 2023-03-09T19:48:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:48:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17391.208854 secs 2023-03-09T19:48:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:48:00 smithi038 ceph-mon[92422]: pgmap v9786: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 346 KiB/s wr, 10 op/s 2023-03-09T19:48:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:48:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17391.208854 secs 2023-03-09T19:48:02.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:48:02 smithi122 ceph-mon[93869]: pgmap v9787: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 354 KiB/s wr, 13 op/s 2023-03-09T19:48:02.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:48:02 smithi181 ceph-mon[94189]: pgmap v9787: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 354 KiB/s wr, 13 op/s 2023-03-09T19:48:02.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:48:02 smithi038 ceph-mon[92422]: pgmap v9787: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 354 KiB/s wr, 13 op/s 2023-03-09T19:48:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17855.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:48:03.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:48:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17393.520572 secs 2023-03-09T19:48:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:48:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17393.520572 secs 2023-03-09T19:48:03.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:48:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17393.520572 secs 2023-03-09T19:48:04.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:48:04 smithi122 ceph-mon[93869]: pgmap v9788: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 326 KiB/s wr, 24 op/s 2023-03-09T19:48:04.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:48:04 smithi181 ceph-mon[94189]: pgmap v9788: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 326 KiB/s wr, 24 op/s 2023-03-09T19:48:04.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:48:04 smithi038 ceph-mon[92422]: pgmap v9788: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 326 KiB/s wr, 24 op/s 2023-03-09T19:48:05.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:48:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17396.208916 secs 2023-03-09T19:48:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:48:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17396.208916 secs 2023-03-09T19:48:05.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:48:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17396.208916 secs 2023-03-09T19:48:06.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:48:06 smithi122 ceph-mon[93869]: pgmap v9789: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 295 KiB/s wr, 23 op/s 2023-03-09T19:48:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:48:06 smithi181 ceph-mon[94189]: pgmap v9789: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 295 KiB/s wr, 23 op/s 2023-03-09T19:48:06.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:48:06 smithi038 ceph-mon[92422]: pgmap v9789: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 295 KiB/s wr, 23 op/s 2023-03-09T19:48:08.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17860.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:48:08.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:48:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17398.520696 secs 2023-03-09T19:48:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:48:08 smithi122 ceph-mon[93869]: pgmap v9790: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 313 KiB/s rd, 346 KiB/s wr, 25 op/s 2023-03-09T19:48:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:48:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17398.520696 secs 2023-03-09T19:48:08.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:48:08 smithi181 ceph-mon[94189]: pgmap v9790: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 313 KiB/s rd, 346 KiB/s wr, 25 op/s 2023-03-09T19:48:08.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:48:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17398.520696 secs 2023-03-09T19:48:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:48:08 smithi038 ceph-mon[92422]: pgmap v9790: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 313 KiB/s rd, 346 KiB/s wr, 25 op/s 2023-03-09T19:48:10.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:48:10 smithi122 ceph-mon[93869]: pgmap v9791: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 236 KiB/s wr, 22 op/s 2023-03-09T19:48:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:48:10 smithi122 ceph-mon[93869]: osdmap e825: 12 total, 12 up, 12 in 2023-03-09T19:48:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:48:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17401.208979 secs 2023-03-09T19:48:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:48:10 smithi181 ceph-mon[94189]: pgmap v9791: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 236 KiB/s wr, 22 op/s 2023-03-09T19:48:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:48:10 smithi181 ceph-mon[94189]: osdmap e825: 12 total, 12 up, 12 in 2023-03-09T19:48:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:48:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17401.208979 secs 2023-03-09T19:48:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:48:10 smithi038 ceph-mon[92422]: pgmap v9791: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 236 KiB/s wr, 22 op/s 2023-03-09T19:48:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:48:10 smithi038 ceph-mon[92422]: osdmap e825: 12 total, 12 up, 12 in 2023-03-09T19:48:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:48:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17401.208979 secs 2023-03-09T19:48:12.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:48:12 smithi122 ceph-mon[93869]: pgmap v9793: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 223 KiB/s wr, 23 op/s 2023-03-09T19:48:12.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:48:12 smithi181 ceph-mon[94189]: pgmap v9793: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 223 KiB/s wr, 23 op/s 2023-03-09T19:48:12.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:48:12 smithi038 ceph-mon[92422]: pgmap v9793: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 223 KiB/s wr, 23 op/s 2023-03-09T19:48:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17865.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:48:13.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:48:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17403.520816 secs 2023-03-09T19:48:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:48:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17403.520816 secs 2023-03-09T19:48:13.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:48:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17403.520816 secs 2023-03-09T19:48:14.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:48:14 smithi122 ceph-mon[93869]: pgmap v9794: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 94 KiB/s wr, 7 op/s 2023-03-09T19:48:14.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:48:14 smithi181 ceph-mon[94189]: pgmap v9794: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 94 KiB/s wr, 7 op/s 2023-03-09T19:48:14.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:48:14 smithi038 ceph-mon[92422]: pgmap v9794: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 94 KiB/s wr, 7 op/s 2023-03-09T19:48:15.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:48:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17406.209040 secs 2023-03-09T19:48:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:48:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17406.209040 secs 2023-03-09T19:48:15.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:48:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17406.209040 secs 2023-03-09T19:48:16.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:48:16 smithi122 ceph-mon[93869]: pgmap v9795: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 94 KiB/s wr, 7 op/s 2023-03-09T19:48:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:48:16 smithi181 ceph-mon[94189]: pgmap v9795: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 94 KiB/s wr, 7 op/s 2023-03-09T19:48:16.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:48:16 smithi038 ceph-mon[92422]: pgmap v9795: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 94 KiB/s wr, 7 op/s 2023-03-09T19:48:18.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17870.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:48:18.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:48:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17408.520920 secs 2023-03-09T19:48:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:48:18 smithi122 ceph-mon[93869]: pgmap v9796: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 104 KiB/s wr, 8 op/s 2023-03-09T19:48:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:48:18 smithi122 ceph-mon[93869]: osdmap e826: 12 total, 12 up, 12 in 2023-03-09T19:48:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:48:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17408.520920 secs 2023-03-09T19:48:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:48:18 smithi181 ceph-mon[94189]: pgmap v9796: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 104 KiB/s wr, 8 op/s 2023-03-09T19:48:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:48:18 smithi181 ceph-mon[94189]: osdmap e826: 12 total, 12 up, 12 in 2023-03-09T19:48:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:48:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17408.520920 secs 2023-03-09T19:48:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:48:18 smithi038 ceph-mon[92422]: pgmap v9796: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 104 KiB/s wr, 8 op/s 2023-03-09T19:48:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:48:18 smithi038 ceph-mon[92422]: osdmap e826: 12 total, 12 up, 12 in 2023-03-09T19:48:20.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:48:20 smithi122 ceph-mon[93869]: pgmap v9798: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 89 KiB/s wr, 8 op/s 2023-03-09T19:48:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:48:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17411.209101 secs 2023-03-09T19:48:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:48:20 smithi181 ceph-mon[94189]: pgmap v9798: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 89 KiB/s wr, 8 op/s 2023-03-09T19:48:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:48:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17411.209101 secs 2023-03-09T19:48:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:48:20 smithi038 ceph-mon[92422]: pgmap v9798: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 89 KiB/s wr, 8 op/s 2023-03-09T19:48:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:48:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17411.209101 secs 2023-03-09T19:48:22.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:48:22 smithi122 ceph-mon[93869]: pgmap v9799: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 89 KiB/s wr, 8 op/s 2023-03-09T19:48:22.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:48:22 smithi181 ceph-mon[94189]: pgmap v9799: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 89 KiB/s wr, 8 op/s 2023-03-09T19:48:22.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:48:22 smithi038 ceph-mon[92422]: pgmap v9799: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 89 KiB/s wr, 8 op/s 2023-03-09T19:48:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17875.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:48:23.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:48:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17413.521034 secs 2023-03-09T19:48:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:48:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17413.521034 secs 2023-03-09T19:48:23.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:48:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17413.521034 secs 2023-03-09T19:48:24.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:48:24 smithi122 ceph-mon[93869]: pgmap v9800: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 72 KiB/s wr, 8 op/s 2023-03-09T19:48:24.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:48:24 smithi181 ceph-mon[94189]: pgmap v9800: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 72 KiB/s wr, 8 op/s 2023-03-09T19:48:24.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:48:24 smithi038 ceph-mon[92422]: pgmap v9800: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 72 KiB/s wr, 8 op/s 2023-03-09T19:48:25.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:48:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17416.209160 secs 2023-03-09T19:48:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:48:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17416.209160 secs 2023-03-09T19:48:25.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:48:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17416.209160 secs 2023-03-09T19:48:26.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:48:26 smithi122 ceph-mon[93869]: pgmap v9801: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 72 KiB/s wr, 8 op/s 2023-03-09T19:48:26.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:48:26 smithi181 ceph-mon[94189]: pgmap v9801: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 72 KiB/s wr, 8 op/s 2023-03-09T19:48:26.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:48:26 smithi038 ceph-mon[92422]: pgmap v9801: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 72 KiB/s wr, 8 op/s 2023-03-09T19:48:27.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:48:27 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:48:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:48:27 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:48:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:48:27 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:48:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:48:27 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:48:27.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:48:27 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:48:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:48:27 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:48:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17880.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:48:28.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:48:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17418.521181 secs 2023-03-09T19:48:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:48:28 smithi122 ceph-mon[93869]: pgmap v9802: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 117 KiB/s wr, 7 op/s 2023-03-09T19:48:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:48:28 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:48:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:48:28 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:48:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:48:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17418.521181 secs 2023-03-09T19:48:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:48:28 smithi181 ceph-mon[94189]: pgmap v9802: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 117 KiB/s wr, 7 op/s 2023-03-09T19:48:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:48:28 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:48:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:48:28 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:48:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:48:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17418.521181 secs 2023-03-09T19:48:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:48:28 smithi038 ceph-mon[92422]: pgmap v9802: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 117 KiB/s wr, 7 op/s 2023-03-09T19:48:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:48:28 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:48:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:48:28 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:48:30.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:48:30 smithi122 ceph-mon[93869]: pgmap v9803: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 104 KiB/s wr, 7 op/s 2023-03-09T19:48:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:48:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17421.209223 secs 2023-03-09T19:48:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:48:30 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:48:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:48:30 smithi181 ceph-mon[94189]: pgmap v9803: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 104 KiB/s wr, 7 op/s 2023-03-09T19:48:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:48:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17421.209223 secs 2023-03-09T19:48:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:48:30 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:48:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:48:30 smithi038 ceph-mon[92422]: pgmap v9803: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 104 KiB/s wr, 7 op/s 2023-03-09T19:48:30.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:48:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17421.209223 secs 2023-03-09T19:48:30.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:48:30 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:48:31.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:48:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:48:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:48:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:48:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:48:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:48:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:48:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:48:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:48:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:48:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:48:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:48:31.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:48:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:48:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:48:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:48:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:48:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:48:32.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:48:32 smithi122 ceph-mon[93869]: pgmap v9804: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 98 KiB/s wr, 9 op/s 2023-03-09T19:48:32.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:48:32 smithi181 ceph-mon[94189]: pgmap v9804: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 98 KiB/s wr, 9 op/s 2023-03-09T19:48:32.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:48:32 smithi038 ceph-mon[92422]: pgmap v9804: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 98 KiB/s wr, 9 op/s 2023-03-09T19:48:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17885.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:48:33.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:48:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17423.521268 secs 2023-03-09T19:48:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:48:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17423.521268 secs 2023-03-09T19:48:33.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:48:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17423.521268 secs 2023-03-09T19:48:34.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:48:34 smithi122 ceph-mon[93869]: pgmap v9805: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 98 KiB/s wr, 9 op/s 2023-03-09T19:48:34.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:48:34 smithi181 ceph-mon[94189]: pgmap v9805: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 98 KiB/s wr, 9 op/s 2023-03-09T19:48:34.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:48:34 smithi038 ceph-mon[92422]: pgmap v9805: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 98 KiB/s wr, 9 op/s 2023-03-09T19:48:35.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:48:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17426.209282 secs 2023-03-09T19:48:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:48:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17426.209282 secs 2023-03-09T19:48:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:48:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17426.209282 secs 2023-03-09T19:48:36.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:48:36 smithi122 ceph-mon[93869]: pgmap v9806: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 98 KiB/s wr, 7 op/s 2023-03-09T19:48:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:48:36 smithi181 ceph-mon[94189]: pgmap v9806: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 98 KiB/s wr, 7 op/s 2023-03-09T19:48:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:48:36 smithi038 ceph-mon[92422]: pgmap v9806: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 98 KiB/s wr, 7 op/s 2023-03-09T19:48:38.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17890.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:48:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:48:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17428.521382 secs 2023-03-09T19:48:38.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:48:38 smithi122 ceph-mon[93869]: pgmap v9807: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 243 KiB/s wr, 10 op/s 2023-03-09T19:48:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:48:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17428.521382 secs 2023-03-09T19:48:38.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:48:38 smithi181 ceph-mon[94189]: pgmap v9807: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 243 KiB/s wr, 10 op/s 2023-03-09T19:48:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:48:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17428.521382 secs 2023-03-09T19:48:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:48:38 smithi038 ceph-mon[92422]: pgmap v9807: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 243 KiB/s wr, 10 op/s 2023-03-09T19:48:40.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:48:40 smithi122 ceph-mon[93869]: pgmap v9808: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 147 KiB/s wr, 8 op/s 2023-03-09T19:48:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:48:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17431.209341 secs 2023-03-09T19:48:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:48:40 smithi181 ceph-mon[94189]: pgmap v9808: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 147 KiB/s wr, 8 op/s 2023-03-09T19:48:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:48:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17431.209341 secs 2023-03-09T19:48:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:48:40 smithi038 ceph-mon[92422]: pgmap v9808: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 147 KiB/s wr, 8 op/s 2023-03-09T19:48:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:48:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17431.209341 secs 2023-03-09T19:48:41.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:48:41 smithi122 ceph-mon[93869]: pgmap v9809: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 146 KiB/s wr, 9 op/s 2023-03-09T19:48:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:48:41 smithi181 ceph-mon[94189]: pgmap v9809: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 146 KiB/s wr, 9 op/s 2023-03-09T19:48:41.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:48:41 smithi038 ceph-mon[92422]: pgmap v9809: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 146 KiB/s wr, 9 op/s 2023-03-09T19:48:43.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:48:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17433.521479 secs 2023-03-09T19:48:43.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:48:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17433.521479 secs 2023-03-09T19:48:43.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:48:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17433.521479 secs 2023-03-09T19:48:43.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17895.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:48:44.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:48:44 smithi122 ceph-mon[93869]: pgmap v9810: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 176 KiB/s wr, 8 op/s 2023-03-09T19:48:44.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:48:44 smithi181 ceph-mon[94189]: pgmap v9810: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 176 KiB/s wr, 8 op/s 2023-03-09T19:48:44.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:48:44 smithi038 ceph-mon[92422]: pgmap v9810: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 176 KiB/s wr, 8 op/s 2023-03-09T19:48:45.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:48:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17436.209402 secs 2023-03-09T19:48:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:48:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17436.209402 secs 2023-03-09T19:48:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:48:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17436.209402 secs 2023-03-09T19:48:46.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:48:46 smithi122 ceph-mon[93869]: pgmap v9811: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 176 KiB/s wr, 7 op/s 2023-03-09T19:48:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:48:46 smithi181 ceph-mon[94189]: pgmap v9811: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 176 KiB/s wr, 7 op/s 2023-03-09T19:48:46.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:48:46 smithi038 ceph-mon[92422]: pgmap v9811: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 176 KiB/s wr, 7 op/s 2023-03-09T19:48:48.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17900.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:48:48.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:48:48 smithi122 ceph-mon[93869]: pgmap v9812: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 318 KiB/s rd, 302 KiB/s wr, 11 op/s 2023-03-09T19:48:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:48:48 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:48:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:48:48 smithi181 ceph-mon[94189]: pgmap v9812: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 318 KiB/s rd, 302 KiB/s wr, 11 op/s 2023-03-09T19:48:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:48:48 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:48:49.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:48:48 smithi038 ceph-mon[92422]: pgmap v9812: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 318 KiB/s rd, 302 KiB/s wr, 11 op/s 2023-03-09T19:48:49.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:48:48 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:48:49.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:48:49 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:48:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:48:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:48:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:48:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:48:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:48:49 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:48:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:48:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:48:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:48:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:48:49.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:48:49 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:48:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:48:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:48:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:48:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:48:50.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:48:50 smithi122 ceph-mon[93869]: pgmap v9813: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 171 KiB/s wr, 9 op/s 2023-03-09T19:48:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:48:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17441.209460 secs 2023-03-09T19:48:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:48:50 smithi181 ceph-mon[94189]: pgmap v9813: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 171 KiB/s wr, 9 op/s 2023-03-09T19:48:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:48:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17441.209460 secs 2023-03-09T19:48:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:48:50 smithi038 ceph-mon[92422]: pgmap v9813: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 171 KiB/s wr, 9 op/s 2023-03-09T19:48:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:48:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17441.209460 secs 2023-03-09T19:48:51.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:48:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:48:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:48:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:48:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:48:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:48:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:48:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:48:51.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:48:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:48:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:48:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:48:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:48:52 smithi122 ceph-mon[93869]: pgmap v9814: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 171 KiB/s wr, 11 op/s 2023-03-09T19:48:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:48:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:48:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:48:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:48:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:48:52 smithi181 ceph-mon[94189]: pgmap v9814: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 171 KiB/s wr, 11 op/s 2023-03-09T19:48:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:48:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:48:52.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:48:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:48:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:48:52 smithi038 ceph-mon[92422]: pgmap v9814: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 171 KiB/s wr, 11 op/s 2023-03-09T19:48:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:48:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:48:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:48:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:48:53.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17905.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:48:54.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:48:54 smithi122 ceph-mon[93869]: pgmap v9815: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 406 KiB/s rd, 392 KiB/s wr, 12 op/s 2023-03-09T19:48:54.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:48:54 smithi181 ceph-mon[94189]: pgmap v9815: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 406 KiB/s rd, 392 KiB/s wr, 12 op/s 2023-03-09T19:48:54.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:48:54 smithi038 ceph-mon[92422]: pgmap v9815: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 406 KiB/s rd, 392 KiB/s wr, 12 op/s 2023-03-09T19:48:55.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:48:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17446.209523 secs 2023-03-09T19:48:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:48:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17446.209523 secs 2023-03-09T19:48:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:48:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17446.209523 secs 2023-03-09T19:48:56.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:48:56 smithi122 ceph-mon[93869]: pgmap v9816: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 361 KiB/s wr, 10 op/s 2023-03-09T19:48:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:48:56 smithi181 ceph-mon[94189]: pgmap v9816: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 361 KiB/s wr, 10 op/s 2023-03-09T19:48:56.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:48:56 smithi038 ceph-mon[92422]: pgmap v9816: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 361 KiB/s wr, 10 op/s 2023-03-09T19:48:58.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17910.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:48:58.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:48:58 smithi122 ceph-mon[93869]: pgmap v9817: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 485 KiB/s rd, 451 KiB/s wr, 13 op/s 2023-03-09T19:48:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:48:58 smithi181 ceph-mon[94189]: pgmap v9817: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 485 KiB/s rd, 451 KiB/s wr, 13 op/s 2023-03-09T19:48:58.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:48:58 smithi038 ceph-mon[92422]: pgmap v9817: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 485 KiB/s rd, 451 KiB/s wr, 13 op/s 2023-03-09T19:49:00.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:49:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:49:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:00 smithi122 ceph-mon[93869]: pgmap v9818: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 569 KiB/s wr, 11 op/s 2023-03-09T19:49:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:49:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:49:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:00 smithi181 ceph-mon[94189]: pgmap v9818: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 569 KiB/s wr, 11 op/s 2023-03-09T19:49:00.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:49:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:49:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:00 smithi038 ceph-mon[92422]: pgmap v9818: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 569 KiB/s wr, 11 op/s 2023-03-09T19:49:01.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17451.209601 secs 2023-03-09T19:49:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:49:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:49:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17451.209601 secs 2023-03-09T19:49:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:49:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:49:01.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17451.209601 secs 2023-03-09T19:49:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:49:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:49:02.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:02 smithi122 ceph-mon[93869]: pgmap v9819: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 588 KiB/s rd, 567 KiB/s wr, 13 op/s 2023-03-09T19:49:02.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:02 smithi181 ceph-mon[94189]: pgmap v9819: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 588 KiB/s rd, 567 KiB/s wr, 13 op/s 2023-03-09T19:49:02.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:02 smithi038 ceph-mon[92422]: pgmap v9819: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 588 KiB/s rd, 567 KiB/s wr, 13 op/s 2023-03-09T19:49:03.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17453.521970 secs 2023-03-09T19:49:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17453.521970 secs 2023-03-09T19:49:03.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17453.521970 secs 2023-03-09T19:49:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17915.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:49:04.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:04 smithi122 ceph-mon[93869]: pgmap v9820: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 609 KiB/s rd, 588 KiB/s wr, 25 op/s 2023-03-09T19:49:04.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:04 smithi181 ceph-mon[94189]: pgmap v9820: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 609 KiB/s rd, 588 KiB/s wr, 25 op/s 2023-03-09T19:49:04.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:04 smithi038 ceph-mon[92422]: pgmap v9820: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 609 KiB/s rd, 588 KiB/s wr, 25 op/s 2023-03-09T19:49:05.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:05 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:49:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:05 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:49:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:05 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:49:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:05 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:49:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:05 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:49:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:05 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:49:05.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:05 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:49:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:05 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:49:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:05 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:49:06.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17456.209668 secs 2023-03-09T19:49:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:06 smithi122 ceph-mon[93869]: pgmap v9821: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 384 KiB/s wr, 23 op/s 2023-03-09T19:49:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17456.209668 secs 2023-03-09T19:49:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:06 smithi181 ceph-mon[94189]: pgmap v9821: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 384 KiB/s wr, 23 op/s 2023-03-09T19:49:06.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17456.209668 secs 2023-03-09T19:49:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:06 smithi038 ceph-mon[92422]: pgmap v9821: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 384 KiB/s wr, 23 op/s 2023-03-09T19:49:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17920.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:49:08.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17458.522068 secs 2023-03-09T19:49:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:08 smithi122 ceph-mon[93869]: pgmap v9822: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 396 KiB/s wr, 26 op/s 2023-03-09T19:49:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17458.522068 secs 2023-03-09T19:49:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:08 smithi181 ceph-mon[94189]: pgmap v9822: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 396 KiB/s wr, 26 op/s 2023-03-09T19:49:08.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17458.522068 secs 2023-03-09T19:49:08.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:08 smithi038 ceph-mon[92422]: pgmap v9822: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 396 KiB/s wr, 26 op/s 2023-03-09T19:49:10.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:10 smithi122 ceph-mon[93869]: pgmap v9823: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 409 KiB/s wr, 24 op/s 2023-03-09T19:49:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:10 smithi122 ceph-mon[93869]: osdmap e827: 12 total, 12 up, 12 in 2023-03-09T19:49:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17461.209737 secs 2023-03-09T19:49:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:10 smithi181 ceph-mon[94189]: pgmap v9823: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 409 KiB/s wr, 24 op/s 2023-03-09T19:49:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:10 smithi181 ceph-mon[94189]: osdmap e827: 12 total, 12 up, 12 in 2023-03-09T19:49:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17461.209737 secs 2023-03-09T19:49:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:10 smithi038 ceph-mon[92422]: pgmap v9823: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 409 KiB/s wr, 24 op/s 2023-03-09T19:49:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:10 smithi038 ceph-mon[92422]: osdmap e827: 12 total, 12 up, 12 in 2023-03-09T19:49:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17461.209737 secs 2023-03-09T19:49:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:12 smithi122 ceph-mon[93869]: pgmap v9825: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 184 KiB/s wr, 25 op/s 2023-03-09T19:49:12.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:12 smithi181 ceph-mon[94189]: pgmap v9825: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 184 KiB/s wr, 25 op/s 2023-03-09T19:49:12.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:12 smithi038 ceph-mon[92422]: pgmap v9825: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 184 KiB/s wr, 25 op/s 2023-03-09T19:49:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17925.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:49:13.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17463.522197 secs 2023-03-09T19:49:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17463.522197 secs 2023-03-09T19:49:13.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17463.522197 secs 2023-03-09T19:49:14.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:14 smithi122 ceph-mon[93869]: pgmap v9826: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 175 KiB/s wr, 10 op/s 2023-03-09T19:49:14.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:14 smithi181 ceph-mon[94189]: pgmap v9826: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 175 KiB/s wr, 10 op/s 2023-03-09T19:49:14.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:14 smithi038 ceph-mon[92422]: pgmap v9826: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 175 KiB/s wr, 10 op/s 2023-03-09T19:49:15.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17466.209824 secs 2023-03-09T19:49:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17466.209824 secs 2023-03-09T19:49:15.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17466.209824 secs 2023-03-09T19:49:16.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:16 smithi122 ceph-mon[93869]: pgmap v9827: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 172 KiB/s wr, 9 op/s 2023-03-09T19:49:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:16 smithi181 ceph-mon[94189]: pgmap v9827: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 172 KiB/s wr, 9 op/s 2023-03-09T19:49:16.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:16 smithi038 ceph-mon[92422]: pgmap v9827: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 172 KiB/s wr, 9 op/s 2023-03-09T19:49:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17930.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:49:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17468.522322 secs 2023-03-09T19:49:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:18 smithi122 ceph-mon[93869]: pgmap v9828: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 186 KiB/s wr, 8 op/s 2023-03-09T19:49:18.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:18 smithi122 ceph-mon[93869]: osdmap e828: 12 total, 12 up, 12 in 2023-03-09T19:49:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17468.522322 secs 2023-03-09T19:49:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:18 smithi181 ceph-mon[94189]: pgmap v9828: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 186 KiB/s wr, 8 op/s 2023-03-09T19:49:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:18 smithi181 ceph-mon[94189]: osdmap e828: 12 total, 12 up, 12 in 2023-03-09T19:49:19.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17468.522322 secs 2023-03-09T19:49:19.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:18 smithi038 ceph-mon[92422]: pgmap v9828: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 186 KiB/s wr, 8 op/s 2023-03-09T19:49:19.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:18 smithi038 ceph-mon[92422]: osdmap e828: 12 total, 12 up, 12 in 2023-03-09T19:49:20.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:20 smithi122 ceph-mon[93869]: pgmap v9830: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 69 KiB/s wr, 8 op/s 2023-03-09T19:49:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17471.209883 secs 2023-03-09T19:49:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:20 smithi181 ceph-mon[94189]: pgmap v9830: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 69 KiB/s wr, 8 op/s 2023-03-09T19:49:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17471.209883 secs 2023-03-09T19:49:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:20 smithi038 ceph-mon[92422]: pgmap v9830: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 69 KiB/s wr, 8 op/s 2023-03-09T19:49:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17471.209883 secs 2023-03-09T19:49:22.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:22 smithi122 ceph-mon[93869]: pgmap v9831: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 68 KiB/s wr, 8 op/s 2023-03-09T19:49:22.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:22 smithi181 ceph-mon[94189]: pgmap v9831: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 68 KiB/s wr, 8 op/s 2023-03-09T19:49:22.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:22 smithi038 ceph-mon[92422]: pgmap v9831: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 68 KiB/s wr, 8 op/s 2023-03-09T19:49:23.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17935.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:49:23.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17473.522437 secs 2023-03-09T19:49:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17473.522437 secs 2023-03-09T19:49:23.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17473.522437 secs 2023-03-09T19:49:24.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:24 smithi122 ceph-mon[93869]: pgmap v9832: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 52 KiB/s wr, 8 op/s 2023-03-09T19:49:24.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:24 smithi181 ceph-mon[94189]: pgmap v9832: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 52 KiB/s wr, 8 op/s 2023-03-09T19:49:24.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:24 smithi038 ceph-mon[92422]: pgmap v9832: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 52 KiB/s wr, 8 op/s 2023-03-09T19:49:25.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17476.209945 secs 2023-03-09T19:49:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17476.209945 secs 2023-03-09T19:49:25.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17476.209945 secs 2023-03-09T19:49:26.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:26 smithi122 ceph-mon[93869]: pgmap v9833: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 34 KiB/s wr, 8 op/s 2023-03-09T19:49:26.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:26 smithi181 ceph-mon[94189]: pgmap v9833: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 34 KiB/s wr, 8 op/s 2023-03-09T19:49:26.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:26 smithi038 ceph-mon[92422]: pgmap v9833: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 34 KiB/s wr, 8 op/s 2023-03-09T19:49:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:27 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:49:27.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:27 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:49:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:27 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:49:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:27 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:49:27.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:27 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:49:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:27 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:49:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17940.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:49:28.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17478.522576 secs 2023-03-09T19:49:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:28 smithi122 ceph-mon[93869]: pgmap v9834: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 82 KiB/s wr, 8 op/s 2023-03-09T19:49:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:28 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:49:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:28 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:49:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17478.522576 secs 2023-03-09T19:49:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:28 smithi181 ceph-mon[94189]: pgmap v9834: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 82 KiB/s wr, 8 op/s 2023-03-09T19:49:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:28 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:49:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:28 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:49:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17478.522576 secs 2023-03-09T19:49:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:28 smithi038 ceph-mon[92422]: pgmap v9834: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 82 KiB/s wr, 8 op/s 2023-03-09T19:49:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:28 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:49:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:28 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:49:30.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:30 smithi122 ceph-mon[93869]: pgmap v9835: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 73 KiB/s wr, 7 op/s 2023-03-09T19:49:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17481.210004 secs 2023-03-09T19:49:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:30 smithi181 ceph-mon[94189]: pgmap v9835: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 73 KiB/s wr, 7 op/s 2023-03-09T19:49:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17481.210004 secs 2023-03-09T19:49:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:30 smithi038 ceph-mon[92422]: pgmap v9835: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 73 KiB/s wr, 7 op/s 2023-03-09T19:49:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17481.210004 secs 2023-03-09T19:49:31.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:49:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:49:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:49:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:49:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:49:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:49:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:49:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:49:31.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:49:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:49:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:49:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:49:32.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:32 smithi122 ceph-mon[93869]: pgmap v9836: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 63 KiB/s wr, 8 op/s 2023-03-09T19:49:32.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:32 smithi181 ceph-mon[94189]: pgmap v9836: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 63 KiB/s wr, 8 op/s 2023-03-09T19:49:32.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:32 smithi038 ceph-mon[92422]: pgmap v9836: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 63 KiB/s wr, 8 op/s 2023-03-09T19:49:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17945.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:49:33.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17483.522686 secs 2023-03-09T19:49:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17483.522686 secs 2023-03-09T19:49:33.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17483.522686 secs 2023-03-09T19:49:34.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:34 smithi122 ceph-mon[93869]: pgmap v9837: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 63 KiB/s wr, 7 op/s 2023-03-09T19:49:34.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:34 smithi181 ceph-mon[94189]: pgmap v9837: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 63 KiB/s wr, 7 op/s 2023-03-09T19:49:34.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:34 smithi038 ceph-mon[92422]: pgmap v9837: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 63 KiB/s wr, 7 op/s 2023-03-09T19:49:35.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17486.210063 secs 2023-03-09T19:49:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17486.210063 secs 2023-03-09T19:49:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17486.210063 secs 2023-03-09T19:49:36.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:36 smithi122 ceph-mon[93869]: pgmap v9838: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 63 KiB/s wr, 6 op/s 2023-03-09T19:49:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:36 smithi181 ceph-mon[94189]: pgmap v9838: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 63 KiB/s wr, 6 op/s 2023-03-09T19:49:36.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:36 smithi038 ceph-mon[92422]: pgmap v9838: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 63 KiB/s wr, 6 op/s 2023-03-09T19:49:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17950.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:49:38.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:38 smithi122 ceph-mon[93869]: pgmap v9839: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 125 KiB/s wr, 8 op/s 2023-03-09T19:49:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:38 smithi181 ceph-mon[94189]: pgmap v9839: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 125 KiB/s wr, 8 op/s 2023-03-09T19:49:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:38 smithi038 ceph-mon[92422]: pgmap v9839: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 125 KiB/s wr, 8 op/s 2023-03-09T19:49:40.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:40 smithi122 ceph-mon[93869]: pgmap v9840: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 63 KiB/s wr, 6 op/s 2023-03-09T19:49:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17491.210127 secs 2023-03-09T19:49:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:40 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:49:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:40 smithi181 ceph-mon[94189]: pgmap v9840: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 63 KiB/s wr, 6 op/s 2023-03-09T19:49:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17491.210127 secs 2023-03-09T19:49:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:40 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:49:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:40 smithi038 ceph-mon[92422]: pgmap v9840: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 63 KiB/s wr, 6 op/s 2023-03-09T19:49:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17491.210127 secs 2023-03-09T19:49:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:40 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:49:41.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:41 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:49:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:49:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:49:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:41 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:49:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:49:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:49:41.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:41 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:49:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:49:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:49:42.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:42 smithi122 ceph-mon[93869]: pgmap v9841: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 130 KiB/s wr, 8 op/s 2023-03-09T19:49:42.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:42 smithi181 ceph-mon[94189]: pgmap v9841: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 130 KiB/s wr, 8 op/s 2023-03-09T19:49:42.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:42 smithi038 ceph-mon[92422]: pgmap v9841: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 130 KiB/s wr, 8 op/s 2023-03-09T19:49:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17955.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:49:44.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:44 smithi122 ceph-mon[93869]: pgmap v9842: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 130 KiB/s wr, 7 op/s 2023-03-09T19:49:44.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:44 smithi181 ceph-mon[94189]: pgmap v9842: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 130 KiB/s wr, 7 op/s 2023-03-09T19:49:44.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:44 smithi038 ceph-mon[92422]: pgmap v9842: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 130 KiB/s wr, 7 op/s 2023-03-09T19:49:45.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17496.210187 secs 2023-03-09T19:49:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17496.210187 secs 2023-03-09T19:49:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17496.210187 secs 2023-03-09T19:49:46.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:46 smithi122 ceph-mon[93869]: pgmap v9843: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 130 KiB/s wr, 6 op/s 2023-03-09T19:49:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:46 smithi181 ceph-mon[94189]: pgmap v9843: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 130 KiB/s wr, 6 op/s 2023-03-09T19:49:46.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:46 smithi038 ceph-mon[92422]: pgmap v9843: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 130 KiB/s wr, 6 op/s 2023-03-09T19:49:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17960.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:49:48.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:48 smithi122 ceph-mon[93869]: pgmap v9844: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 258 KiB/s wr, 8 op/s 2023-03-09T19:49:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:48 smithi181 ceph-mon[94189]: pgmap v9844: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 258 KiB/s wr, 8 op/s 2023-03-09T19:49:48.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:48 smithi038 ceph-mon[92422]: pgmap v9844: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 258 KiB/s wr, 8 op/s 2023-03-09T19:49:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:50 smithi122 ceph-mon[93869]: pgmap v9845: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 224 KiB/s wr, 7 op/s 2023-03-09T19:49:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17501.210248 secs 2023-03-09T19:49:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:50 smithi181 ceph-mon[94189]: pgmap v9845: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 224 KiB/s wr, 7 op/s 2023-03-09T19:49:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17501.210248 secs 2023-03-09T19:49:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:50 smithi038 ceph-mon[92422]: pgmap v9845: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 224 KiB/s wr, 7 op/s 2023-03-09T19:49:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17501.210248 secs 2023-03-09T19:49:52.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:52 smithi122 ceph-mon[93869]: pgmap v9846: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 224 KiB/s wr, 10 op/s 2023-03-09T19:49:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:52 smithi181 ceph-mon[94189]: pgmap v9846: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 224 KiB/s wr, 10 op/s 2023-03-09T19:49:52.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:52 smithi038 ceph-mon[92422]: pgmap v9846: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 224 KiB/s wr, 10 op/s 2023-03-09T19:49:53.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17965.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:49:53.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:53 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 17503.523139 secs 2023-03-09T19:49:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:53 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 17503.523139 secs 2023-03-09T19:49:53.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:53 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 17503.523139 secs 2023-03-09T19:49:54.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:54 smithi122 ceph-mon[93869]: pgmap v9847: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 172 KiB/s wr, 9 op/s 2023-03-09T19:49:54.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:54 smithi181 ceph-mon[94189]: pgmap v9847: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 172 KiB/s wr, 9 op/s 2023-03-09T19:49:54.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:54 smithi038 ceph-mon[92422]: pgmap v9847: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 172 KiB/s wr, 9 op/s 2023-03-09T19:49:56.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17506.210332 secs 2023-03-09T19:49:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:49:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:49:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:56 smithi122 ceph-mon[93869]: pgmap v9848: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 397 KiB/s wr, 10 op/s 2023-03-09T19:49:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17506.210332 secs 2023-03-09T19:49:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:49:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:49:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:56 smithi181 ceph-mon[94189]: pgmap v9848: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 397 KiB/s wr, 10 op/s 2023-03-09T19:49:56.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17506.210332 secs 2023-03-09T19:49:56.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:49:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:49:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:56 smithi038 ceph-mon[92422]: pgmap v9848: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 397 KiB/s wr, 10 op/s 2023-03-09T19:49:57.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:57 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:49:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:49:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:49:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:49:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:57 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:49:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:49:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:49:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:49:57.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:57 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:49:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:49:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:49:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:49:58.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:58 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 17508.523258 secs 2023-03-09T19:49:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:49:58 smithi122 ceph-mon[93869]: pgmap v9849: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 397 KiB/s wr, 12 op/s 2023-03-09T19:49:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:58 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 17508.523258 secs 2023-03-09T19:49:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:49:58 smithi181 ceph-mon[94189]: pgmap v9849: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 397 KiB/s wr, 12 op/s 2023-03-09T19:49:58.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:58 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 17508.523258 secs 2023-03-09T19:49:58.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:49:58 smithi038 ceph-mon[92422]: pgmap v9849: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 397 KiB/s wr, 12 op/s 2023-03-09T19:49:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17970.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:50:00.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:00 smithi122 ceph-mon[93869]: pgmap v9850: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 328 KiB/s wr, 12 op/s 2023-03-09T19:50:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17511.210390 secs 2023-03-09T19:50:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:00 smithi122 ceph-mon[93869]: Health detail: HEALTH_WARN 2 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-09T19:50:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:00 smithi122 ceph-mon[93869]: [WRN] MDS_SLOW_REQUEST: 2 MDSs report slow requests 2023-03-09T19:50:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:00 smithi122 ceph-mon[93869]: mds.i(mds.2): 1 slow requests are blocked > 180 secs 2023-03-09T19:50:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:00 smithi122 ceph-mon[93869]: mds.h(mds.1): 2 slow requests are blocked > 180 secs 2023-03-09T19:50:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:00 smithi122 ceph-mon[93869]: [WRN] MDS_TRIM: 2 MDSs behind on trimming 2023-03-09T19:50:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:00 smithi122 ceph-mon[93869]: mds.e(mds.0): Behind on trimming (267/128) max_segments: 128, num_segments: 267 2023-03-09T19:50:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:00 smithi122 ceph-mon[93869]: mds.b(mds.0): Behind on trimming (267/128) max_segments: 128, num_segments: 267 2023-03-09T19:50:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:00 smithi181 ceph-mon[94189]: pgmap v9850: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 328 KiB/s wr, 12 op/s 2023-03-09T19:50:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17511.210390 secs 2023-03-09T19:50:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:00 smithi181 ceph-mon[94189]: Health detail: HEALTH_WARN 2 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-09T19:50:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:00 smithi181 ceph-mon[94189]: [WRN] MDS_SLOW_REQUEST: 2 MDSs report slow requests 2023-03-09T19:50:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:00 smithi181 ceph-mon[94189]: mds.i(mds.2): 1 slow requests are blocked > 180 secs 2023-03-09T19:50:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:00 smithi181 ceph-mon[94189]: mds.h(mds.1): 2 slow requests are blocked > 180 secs 2023-03-09T19:50:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:00 smithi181 ceph-mon[94189]: [WRN] MDS_TRIM: 2 MDSs behind on trimming 2023-03-09T19:50:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:00 smithi181 ceph-mon[94189]: mds.e(mds.0): Behind on trimming (267/128) max_segments: 128, num_segments: 267 2023-03-09T19:50:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:00 smithi181 ceph-mon[94189]: mds.b(mds.0): Behind on trimming (267/128) max_segments: 128, num_segments: 267 2023-03-09T19:50:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:00 smithi038 ceph-mon[92422]: pgmap v9850: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 328 KiB/s wr, 12 op/s 2023-03-09T19:50:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17511.210390 secs 2023-03-09T19:50:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:00 smithi038 ceph-mon[92422]: Health detail: HEALTH_WARN 2 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-09T19:50:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:00 smithi038 ceph-mon[92422]: [WRN] MDS_SLOW_REQUEST: 2 MDSs report slow requests 2023-03-09T19:50:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:00 smithi038 ceph-mon[92422]: mds.i(mds.2): 1 slow requests are blocked > 180 secs 2023-03-09T19:50:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:00 smithi038 ceph-mon[92422]: mds.h(mds.1): 2 slow requests are blocked > 180 secs 2023-03-09T19:50:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:00 smithi038 ceph-mon[92422]: [WRN] MDS_TRIM: 2 MDSs behind on trimming 2023-03-09T19:50:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:00 smithi038 ceph-mon[92422]: mds.e(mds.0): Behind on trimming (267/128) max_segments: 128, num_segments: 267 2023-03-09T19:50:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:00 smithi038 ceph-mon[92422]: mds.b(mds.0): Behind on trimming (267/128) max_segments: 128, num_segments: 267 2023-03-09T19:50:02.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:02 smithi122 ceph-mon[93869]: pgmap v9851: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 357 KiB/s rd, 301 KiB/s wr, 12 op/s 2023-03-09T19:50:02.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:02 smithi181 ceph-mon[94189]: pgmap v9851: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 357 KiB/s rd, 301 KiB/s wr, 12 op/s 2023-03-09T19:50:03.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:02 smithi038 ceph-mon[92422]: pgmap v9851: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 357 KiB/s rd, 301 KiB/s wr, 12 op/s 2023-03-09T19:50:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17975.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:50:03.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:03 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 17513.523351 secs 2023-03-09T19:50:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:03 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 17513.523351 secs 2023-03-09T19:50:03.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:03 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 17513.523351 secs 2023-03-09T19:50:04.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:04 smithi122 ceph-mon[93869]: pgmap v9852: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 442 KiB/s wr, 23 op/s 2023-03-09T19:50:04.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:04 smithi181 ceph-mon[94189]: pgmap v9852: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 442 KiB/s wr, 23 op/s 2023-03-09T19:50:04.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:04 smithi038 ceph-mon[92422]: pgmap v9852: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 442 KiB/s wr, 23 op/s 2023-03-09T19:50:05.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17516.210453 secs 2023-03-09T19:50:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17516.210453 secs 2023-03-09T19:50:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17516.210453 secs 2023-03-09T19:50:06.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:06 smithi122 ceph-mon[93869]: pgmap v9853: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 487 KiB/s wr, 23 op/s 2023-03-09T19:50:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:06 smithi181 ceph-mon[94189]: pgmap v9853: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 487 KiB/s wr, 23 op/s 2023-03-09T19:50:06.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:06 smithi038 ceph-mon[92422]: pgmap v9853: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 487 KiB/s wr, 23 op/s 2023-03-09T19:50:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:50:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:50:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:50:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:50:07.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:50:07.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:50:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17980.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:50:08.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17518.523470 secs 2023-03-09T19:50:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:08 smithi122 ceph-mon[93869]: pgmap v9854: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 326 KiB/s rd, 284 KiB/s wr, 26 op/s 2023-03-09T19:50:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:50:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:50:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17518.523470 secs 2023-03-09T19:50:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:08 smithi181 ceph-mon[94189]: pgmap v9854: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 326 KiB/s rd, 284 KiB/s wr, 26 op/s 2023-03-09T19:50:08.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:50:08.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:50:08.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17518.523470 secs 2023-03-09T19:50:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:08 smithi038 ceph-mon[92422]: pgmap v9854: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 326 KiB/s rd, 284 KiB/s wr, 26 op/s 2023-03-09T19:50:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:50:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:50:09.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:50:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:50:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:50:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:50:09.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:50:09.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:50:10.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:10 smithi122 ceph-mon[93869]: pgmap v9855: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 398 KiB/s rd, 357 KiB/s wr, 25 op/s 2023-03-09T19:50:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:10 smithi122 ceph-mon[93869]: osdmap e829: 12 total, 12 up, 12 in 2023-03-09T19:50:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17521.210513 secs 2023-03-09T19:50:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:10 smithi181 ceph-mon[94189]: pgmap v9855: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 398 KiB/s rd, 357 KiB/s wr, 25 op/s 2023-03-09T19:50:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:10 smithi181 ceph-mon[94189]: osdmap e829: 12 total, 12 up, 12 in 2023-03-09T19:50:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17521.210513 secs 2023-03-09T19:50:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:10 smithi038 ceph-mon[92422]: pgmap v9855: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 398 KiB/s rd, 357 KiB/s wr, 25 op/s 2023-03-09T19:50:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:10 smithi038 ceph-mon[92422]: osdmap e829: 12 total, 12 up, 12 in 2023-03-09T19:50:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17521.210513 secs 2023-03-09T19:50:12.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:12 smithi122 ceph-mon[93869]: pgmap v9857: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 335 KiB/s rd, 356 KiB/s wr, 28 op/s 2023-03-09T19:50:12.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:12 smithi181 ceph-mon[94189]: pgmap v9857: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 335 KiB/s rd, 356 KiB/s wr, 28 op/s 2023-03-09T19:50:12.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:12 smithi038 ceph-mon[92422]: pgmap v9857: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 335 KiB/s rd, 356 KiB/s wr, 28 op/s 2023-03-09T19:50:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17985.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:50:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17523.523560 secs 2023-03-09T19:50:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17523.523560 secs 2023-03-09T19:50:13.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17523.523560 secs 2023-03-09T19:50:14.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:14 smithi122 ceph-mon[93869]: pgmap v9858: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 238 KiB/s wr, 12 op/s 2023-03-09T19:50:14.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:14 smithi181 ceph-mon[94189]: pgmap v9858: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 238 KiB/s wr, 12 op/s 2023-03-09T19:50:14.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:14 smithi038 ceph-mon[92422]: pgmap v9858: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 238 KiB/s wr, 12 op/s 2023-03-09T19:50:15.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17526.210589 secs 2023-03-09T19:50:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17526.210589 secs 2023-03-09T19:50:15.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17526.210589 secs 2023-03-09T19:50:16.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:16 smithi122 ceph-mon[93869]: pgmap v9859: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 167 KiB/s wr, 11 op/s 2023-03-09T19:50:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:16 smithi181 ceph-mon[94189]: pgmap v9859: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 167 KiB/s wr, 11 op/s 2023-03-09T19:50:16.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:16 smithi038 ceph-mon[92422]: pgmap v9859: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 167 KiB/s wr, 11 op/s 2023-03-09T19:50:18.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17990.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:50:18.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17528.523675 secs 2023-03-09T19:50:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:18 smithi122 ceph-mon[93869]: pgmap v9860: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 140 KiB/s wr, 8 op/s 2023-03-09T19:50:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:18 smithi122 ceph-mon[93869]: osdmap e830: 12 total, 12 up, 12 in 2023-03-09T19:50:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17528.523675 secs 2023-03-09T19:50:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:18 smithi181 ceph-mon[94189]: pgmap v9860: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 140 KiB/s wr, 8 op/s 2023-03-09T19:50:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:18 smithi181 ceph-mon[94189]: osdmap e830: 12 total, 12 up, 12 in 2023-03-09T19:50:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17528.523675 secs 2023-03-09T19:50:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:18 smithi038 ceph-mon[92422]: pgmap v9860: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 140 KiB/s wr, 8 op/s 2023-03-09T19:50:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:18 smithi038 ceph-mon[92422]: osdmap e830: 12 total, 12 up, 12 in 2023-03-09T19:50:20.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:20 smithi122 ceph-mon[93869]: pgmap v9862: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 52 KiB/s wr, 7 op/s 2023-03-09T19:50:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17531.210648 secs 2023-03-09T19:50:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:20 smithi181 ceph-mon[94189]: pgmap v9862: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 52 KiB/s wr, 7 op/s 2023-03-09T19:50:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17531.210648 secs 2023-03-09T19:50:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:20 smithi038 ceph-mon[92422]: pgmap v9862: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 52 KiB/s wr, 7 op/s 2023-03-09T19:50:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17531.210648 secs 2023-03-09T19:50:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:22 smithi181 ceph-mon[94189]: pgmap v9863: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 52 KiB/s wr, 7 op/s 2023-03-09T19:50:22.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:22 smithi122 ceph-mon[93869]: pgmap v9863: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 52 KiB/s wr, 7 op/s 2023-03-09T19:50:22.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:22 smithi038 ceph-mon[92422]: pgmap v9863: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 52 KiB/s wr, 7 op/s 2023-03-09T19:50:23.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 17995.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:50:23.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17533.523821 secs 2023-03-09T19:50:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17533.523821 secs 2023-03-09T19:50:23.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17533.523821 secs 2023-03-09T19:50:24.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:24 smithi122 ceph-mon[93869]: pgmap v9864: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2023-03-09T19:50:24.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:24 smithi181 ceph-mon[94189]: pgmap v9864: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2023-03-09T19:50:24.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:24 smithi038 ceph-mon[92422]: pgmap v9864: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2023-03-09T19:50:25.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17536.210711 secs 2023-03-09T19:50:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17536.210711 secs 2023-03-09T19:50:25.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17536.210711 secs 2023-03-09T19:50:26.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:26 smithi122 ceph-mon[93869]: pgmap v9865: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2023-03-09T19:50:26.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:26 smithi181 ceph-mon[94189]: pgmap v9865: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2023-03-09T19:50:26.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:26 smithi038 ceph-mon[92422]: pgmap v9865: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2023-03-09T19:50:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18000.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:50:28.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17538.523958 secs 2023-03-09T19:50:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:28 smithi122 ceph-mon[93869]: pgmap v9866: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2023-03-09T19:50:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17538.523958 secs 2023-03-09T19:50:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:28 smithi181 ceph-mon[94189]: pgmap v9866: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2023-03-09T19:50:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17538.523958 secs 2023-03-09T19:50:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:28 smithi038 ceph-mon[92422]: pgmap v9866: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.5 KiB/s wr, 6 op/s 2023-03-09T19:50:30.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:30 smithi122 ceph-mon[93869]: pgmap v9867: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 68 KiB/s wr, 6 op/s 2023-03-09T19:50:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17541.210790 secs 2023-03-09T19:50:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:30 smithi181 ceph-mon[94189]: pgmap v9867: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 68 KiB/s wr, 6 op/s 2023-03-09T19:50:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17541.210790 secs 2023-03-09T19:50:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:30 smithi038 ceph-mon[92422]: pgmap v9867: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 68 KiB/s wr, 6 op/s 2023-03-09T19:50:30.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17541.210790 secs 2023-03-09T19:50:31.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:50:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:50:31.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:50:32.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:32 smithi122 ceph-mon[93869]: pgmap v9868: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 64 KiB/s wr, 7 op/s 2023-03-09T19:50:32.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:32 smithi181 ceph-mon[94189]: pgmap v9868: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 64 KiB/s wr, 7 op/s 2023-03-09T19:50:32.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:32 smithi038 ceph-mon[92422]: pgmap v9868: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 64 KiB/s wr, 7 op/s 2023-03-09T19:50:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18005.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:50:33.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17543.524050 secs 2023-03-09T19:50:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:33 smithi122 ceph-mon[93869]: pgmap v9869: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 64 KiB/s wr, 7 op/s 2023-03-09T19:50:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17543.524050 secs 2023-03-09T19:50:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:33 smithi181 ceph-mon[94189]: pgmap v9869: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 64 KiB/s wr, 7 op/s 2023-03-09T19:50:33.986 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17543.524050 secs 2023-03-09T19:50:33.986 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:33 smithi038 ceph-mon[92422]: pgmap v9869: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 64 KiB/s wr, 7 op/s 2023-03-09T19:50:36.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17546.210848 secs 2023-03-09T19:50:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:50:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:50:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:50:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:50:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:50:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:36 smithi122 ceph-mon[93869]: pgmap v9870: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 63 KiB/s wr, 6 op/s 2023-03-09T19:50:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17546.210848 secs 2023-03-09T19:50:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:50:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:50:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:50:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:50:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:50:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:36 smithi181 ceph-mon[94189]: pgmap v9870: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 63 KiB/s wr, 6 op/s 2023-03-09T19:50:36.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17546.210848 secs 2023-03-09T19:50:36.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:50:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:50:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:50:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:50:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:50:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:36 smithi038 ceph-mon[92422]: pgmap v9870: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 63 KiB/s wr, 6 op/s 2023-03-09T19:50:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18010.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:50:38.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17548.524173 secs 2023-03-09T19:50:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:38 smithi122 ceph-mon[93869]: pgmap v9871: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 63 KiB/s wr, 8 op/s 2023-03-09T19:50:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17548.524173 secs 2023-03-09T19:50:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:38 smithi181 ceph-mon[94189]: pgmap v9871: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 63 KiB/s wr, 8 op/s 2023-03-09T19:50:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17548.524173 secs 2023-03-09T19:50:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:38 smithi038 ceph-mon[92422]: pgmap v9871: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 63 KiB/s wr, 8 op/s 2023-03-09T19:50:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:40 smithi122 ceph-mon[93869]: pgmap v9872: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 64 KiB/s wr, 7 op/s 2023-03-09T19:50:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17551.210908 secs 2023-03-09T19:50:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:40 smithi181 ceph-mon[94189]: pgmap v9872: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 64 KiB/s wr, 7 op/s 2023-03-09T19:50:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17551.210908 secs 2023-03-09T19:50:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:40 smithi038 ceph-mon[92422]: pgmap v9872: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 64 KiB/s wr, 7 op/s 2023-03-09T19:50:41.048 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17551.210908 secs 2023-03-09T19:50:42.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:42 smithi122 ceph-mon[93869]: pgmap v9873: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 767 B/s wr, 7 op/s 2023-03-09T19:50:42.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:42 smithi181 ceph-mon[94189]: pgmap v9873: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 767 B/s wr, 7 op/s 2023-03-09T19:50:42.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:42 smithi038 ceph-mon[92422]: pgmap v9873: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 767 B/s wr, 7 op/s 2023-03-09T19:50:43.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18015.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:50:43.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17553.524263 secs 2023-03-09T19:50:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17553.524263 secs 2023-03-09T19:50:43.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17553.524263 secs 2023-03-09T19:50:44.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:44 smithi122 ceph-mon[93869]: pgmap v9874: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 212 KiB/s wr, 8 op/s 2023-03-09T19:50:44.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:44 smithi181 ceph-mon[94189]: pgmap v9874: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 212 KiB/s wr, 8 op/s 2023-03-09T19:50:44.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:44 smithi038 ceph-mon[92422]: pgmap v9874: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 212 KiB/s wr, 8 op/s 2023-03-09T19:50:45.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17556.210990 secs 2023-03-09T19:50:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17556.210990 secs 2023-03-09T19:50:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17556.210990 secs 2023-03-09T19:50:46.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:46 smithi122 ceph-mon[93869]: pgmap v9875: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 212 KiB/s wr, 7 op/s 2023-03-09T19:50:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:46 smithi181 ceph-mon[94189]: pgmap v9875: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 212 KiB/s wr, 7 op/s 2023-03-09T19:50:46.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:46 smithi038 ceph-mon[92422]: pgmap v9875: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 212 KiB/s wr, 7 op/s 2023-03-09T19:50:47.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:50:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:50:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:50:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:50:47.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:50:47.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:50:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18020.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:50:48.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17558.524378 secs 2023-03-09T19:50:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:48 smithi122 ceph-mon[93869]: pgmap v9876: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 212 KiB/s wr, 9 op/s 2023-03-09T19:50:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:50:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:50:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17558.524378 secs 2023-03-09T19:50:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:48 smithi181 ceph-mon[94189]: pgmap v9876: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 212 KiB/s wr, 9 op/s 2023-03-09T19:50:48.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:50:48.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:50:48.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17558.524378 secs 2023-03-09T19:50:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:48 smithi038 ceph-mon[92422]: pgmap v9876: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 212 KiB/s wr, 9 op/s 2023-03-09T19:50:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:50:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:50:50.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:50 smithi122 ceph-mon[93869]: pgmap v9877: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 294 KiB/s wr, 9 op/s 2023-03-09T19:50:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17561.211077 secs 2023-03-09T19:50:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:50 smithi181 ceph-mon[94189]: pgmap v9877: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 294 KiB/s wr, 9 op/s 2023-03-09T19:50:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17561.211077 secs 2023-03-09T19:50:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:50 smithi038 ceph-mon[92422]: pgmap v9877: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 294 KiB/s wr, 9 op/s 2023-03-09T19:50:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17561.211077 secs 2023-03-09T19:50:52.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:52 smithi122 ceph-mon[93869]: pgmap v9878: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 294 KiB/s wr, 10 op/s 2023-03-09T19:50:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:52 smithi181 ceph-mon[94189]: pgmap v9878: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 294 KiB/s wr, 10 op/s 2023-03-09T19:50:52.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:52 smithi038 ceph-mon[92422]: pgmap v9878: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 294 KiB/s wr, 10 op/s 2023-03-09T19:50:53.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18025.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:50:53.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17563.524515 secs 2023-03-09T19:50:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17563.524515 secs 2023-03-09T19:50:53.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17563.524515 secs 2023-03-09T19:50:54.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:54 smithi122 ceph-mon[93869]: pgmap v9879: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 471 KiB/s wr, 10 op/s 2023-03-09T19:50:54.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:54 smithi181 ceph-mon[94189]: pgmap v9879: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 471 KiB/s wr, 10 op/s 2023-03-09T19:50:54.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:54 smithi038 ceph-mon[92422]: pgmap v9879: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 471 KiB/s wr, 10 op/s 2023-03-09T19:50:55.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17566.211156 secs 2023-03-09T19:50:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17566.211156 secs 2023-03-09T19:50:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17566.211156 secs 2023-03-09T19:50:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:56 smithi122 ceph-mon[93869]: pgmap v9880: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 260 KiB/s wr, 8 op/s 2023-03-09T19:50:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:50:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:50:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:56 smithi181 ceph-mon[94189]: pgmap v9880: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 260 KiB/s wr, 8 op/s 2023-03-09T19:50:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:50:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:50:56.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:56 smithi038 ceph-mon[92422]: pgmap v9880: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 260 KiB/s wr, 8 op/s 2023-03-09T19:50:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:50:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:50:58.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18030.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:50:58.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17568.524636 secs 2023-03-09T19:50:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:58 smithi122 ceph-mon[93869]: pgmap v9881: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 340 KiB/s wr, 11 op/s 2023-03-09T19:50:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17568.524636 secs 2023-03-09T19:50:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:58 smithi181 ceph-mon[94189]: pgmap v9881: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 340 KiB/s wr, 11 op/s 2023-03-09T19:50:58.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17568.524636 secs 2023-03-09T19:50:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:58 smithi038 ceph-mon[92422]: pgmap v9881: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 340 KiB/s wr, 11 op/s 2023-03-09T19:50:59.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:50:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:50:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:50:59 smithi122 ceph-mon[93869]: pgmap v9882: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 341 KiB/s wr, 10 op/s 2023-03-09T19:50:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:50:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:50:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:50:59 smithi181 ceph-mon[94189]: pgmap v9882: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 341 KiB/s wr, 10 op/s 2023-03-09T19:50:59.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:50:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:50:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:50:59 smithi038 ceph-mon[92422]: pgmap v9882: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 341 KiB/s wr, 10 op/s 2023-03-09T19:51:00.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:51:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17571.211240 secs 2023-03-09T19:51:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:51:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17571.211240 secs 2023-03-09T19:51:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:51:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17571.211240 secs 2023-03-09T19:51:01.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:51:01 smithi122 ceph-mon[93869]: pgmap v9883: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 414 KiB/s wr, 14 op/s 2023-03-09T19:51:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:51:01 smithi181 ceph-mon[94189]: pgmap v9883: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 414 KiB/s wr, 14 op/s 2023-03-09T19:51:01.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:51:01 smithi038 ceph-mon[92422]: pgmap v9883: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 414 KiB/s wr, 14 op/s 2023-03-09T19:51:03.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:51:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17573.524770 secs 2023-03-09T19:51:03.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:51:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17573.524770 secs 2023-03-09T19:51:03.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:51:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17573.524770 secs 2023-03-09T19:51:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18035.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:51:04.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:51:04 smithi122 ceph-mon[93869]: pgmap v9884: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 444 KiB/s wr, 25 op/s 2023-03-09T19:51:04.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:51:04 smithi181 ceph-mon[94189]: pgmap v9884: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 444 KiB/s wr, 25 op/s 2023-03-09T19:51:04.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:51:04 smithi038 ceph-mon[92422]: pgmap v9884: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 444 KiB/s wr, 25 op/s 2023-03-09T19:51:05.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:51:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17576.211319 secs 2023-03-09T19:51:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:51:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17576.211319 secs 2023-03-09T19:51:05.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:51:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17576.211319 secs 2023-03-09T19:51:06.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:51:06 smithi122 ceph-mon[93869]: pgmap v9885: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 267 KiB/s wr, 23 op/s 2023-03-09T19:51:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:51:06 smithi181 ceph-mon[94189]: pgmap v9885: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 267 KiB/s wr, 23 op/s 2023-03-09T19:51:06.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:51:06 smithi038 ceph-mon[92422]: pgmap v9885: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 267 KiB/s wr, 23 op/s 2023-03-09T19:51:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18040.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:51:08.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:51:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17578.524867 secs 2023-03-09T19:51:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:51:08 smithi122 ceph-mon[93869]: pgmap v9886: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 302 KiB/s wr, 26 op/s 2023-03-09T19:51:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:51:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17578.524867 secs 2023-03-09T19:51:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:51:08 smithi181 ceph-mon[94189]: pgmap v9886: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 302 KiB/s wr, 26 op/s 2023-03-09T19:51:08.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:51:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17578.524867 secs 2023-03-09T19:51:08.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:51:08 smithi038 ceph-mon[92422]: pgmap v9886: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 302 KiB/s wr, 26 op/s 2023-03-09T19:51:10.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:51:10 smithi122 ceph-mon[93869]: pgmap v9887: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 222 KiB/s wr, 24 op/s 2023-03-09T19:51:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:51:10 smithi122 ceph-mon[93869]: osdmap e831: 12 total, 12 up, 12 in 2023-03-09T19:51:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:51:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17581.211382 secs 2023-03-09T19:51:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:51:10 smithi181 ceph-mon[94189]: pgmap v9887: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 222 KiB/s wr, 24 op/s 2023-03-09T19:51:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:51:10 smithi181 ceph-mon[94189]: osdmap e831: 12 total, 12 up, 12 in 2023-03-09T19:51:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:51:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17581.211382 secs 2023-03-09T19:51:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:51:10 smithi038 ceph-mon[92422]: pgmap v9887: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 222 KiB/s wr, 24 op/s 2023-03-09T19:51:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:51:10 smithi038 ceph-mon[92422]: osdmap e831: 12 total, 12 up, 12 in 2023-03-09T19:51:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:51:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17581.211382 secs 2023-03-09T19:51:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:51:12 smithi122 ceph-mon[93869]: pgmap v9889: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 282 KiB/s wr, 25 op/s 2023-03-09T19:51:12.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:51:12 smithi181 ceph-mon[94189]: pgmap v9889: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 282 KiB/s wr, 25 op/s 2023-03-09T19:51:12.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:51:12 smithi038 ceph-mon[92422]: pgmap v9889: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 282 KiB/s wr, 25 op/s 2023-03-09T19:51:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18045.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:51:13.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:51:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17583.525004 secs 2023-03-09T19:51:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:51:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17583.525004 secs 2023-03-09T19:51:13.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:51:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17583.525004 secs 2023-03-09T19:51:14.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:51:14 smithi122 ceph-mon[93869]: pgmap v9890: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 245 KiB/s wr, 10 op/s 2023-03-09T19:51:14.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:51:14 smithi181 ceph-mon[94189]: pgmap v9890: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 245 KiB/s wr, 10 op/s 2023-03-09T19:51:14.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:51:14 smithi038 ceph-mon[92422]: pgmap v9890: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 245 KiB/s wr, 10 op/s 2023-03-09T19:51:15.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:51:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17586.211461 secs 2023-03-09T19:51:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:51:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17586.211461 secs 2023-03-09T19:51:15.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:51:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17586.211461 secs 2023-03-09T19:51:16.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:51:16 smithi122 ceph-mon[93869]: pgmap v9891: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 245 KiB/s wr, 10 op/s 2023-03-09T19:51:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:51:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:51:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:51:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:51:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:51:16 smithi181 ceph-mon[94189]: pgmap v9891: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 245 KiB/s wr, 10 op/s 2023-03-09T19:51:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:51:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:51:16.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:51:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:51:16.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:51:16 smithi038 ceph-mon[92422]: pgmap v9891: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 245 KiB/s wr, 10 op/s 2023-03-09T19:51:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:51:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:51:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:51:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:51:18.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18050.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:51:18.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:51:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17588.525148 secs 2023-03-09T19:51:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:51:18 smithi122 ceph-mon[93869]: pgmap v9892: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 224 KiB/s wr, 10 op/s 2023-03-09T19:51:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:51:18 smithi122 ceph-mon[93869]: osdmap e832: 12 total, 12 up, 12 in 2023-03-09T19:51:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:51:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17588.525148 secs 2023-03-09T19:51:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:51:18 smithi181 ceph-mon[94189]: pgmap v9892: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 224 KiB/s wr, 10 op/s 2023-03-09T19:51:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:51:18 smithi181 ceph-mon[94189]: osdmap e832: 12 total, 12 up, 12 in 2023-03-09T19:51:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:51:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17588.525148 secs 2023-03-09T19:51:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:51:18 smithi038 ceph-mon[92422]: pgmap v9892: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 224 KiB/s wr, 10 op/s 2023-03-09T19:51:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:51:18 smithi038 ceph-mon[92422]: osdmap e832: 12 total, 12 up, 12 in 2023-03-09T19:51:19.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:51:19 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:51:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:51:19 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:51:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:51:19 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:51:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:51:19 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:51:19.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:51:19 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:51:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:51:19 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:51:20.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:51:20 smithi122 ceph-mon[93869]: pgmap v9894: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 225 KiB/s wr, 10 op/s 2023-03-09T19:51:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:51:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17591.211535 secs 2023-03-09T19:51:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:51:20 smithi181 ceph-mon[94189]: pgmap v9894: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 225 KiB/s wr, 10 op/s 2023-03-09T19:51:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:51:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17591.211535 secs 2023-03-09T19:51:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:51:20 smithi038 ceph-mon[92422]: pgmap v9894: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 225 KiB/s wr, 10 op/s 2023-03-09T19:51:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:51:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17591.211535 secs 2023-03-09T19:51:22.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:51:22 smithi122 ceph-mon[93869]: pgmap v9895: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 80 KiB/s wr, 9 op/s 2023-03-09T19:51:22.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:51:22 smithi181 ceph-mon[94189]: pgmap v9895: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 80 KiB/s wr, 9 op/s 2023-03-09T19:51:22.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:51:22 smithi038 ceph-mon[92422]: pgmap v9895: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 80 KiB/s wr, 9 op/s 2023-03-09T19:51:23.793 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18055.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:51:23.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:51:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17593.525259 secs 2023-03-09T19:51:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:51:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17593.525259 secs 2023-03-09T19:51:23.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:51:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17593.525259 secs 2023-03-09T19:51:24.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:51:24 smithi122 ceph-mon[93869]: pgmap v9896: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 80 KiB/s wr, 8 op/s 2023-03-09T19:51:24.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:51:24 smithi181 ceph-mon[94189]: pgmap v9896: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 80 KiB/s wr, 8 op/s 2023-03-09T19:51:24.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:51:24 smithi038 ceph-mon[92422]: pgmap v9896: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 80 KiB/s wr, 8 op/s 2023-03-09T19:51:25.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:51:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17596.211598 secs 2023-03-09T19:51:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:51:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17596.211598 secs 2023-03-09T19:51:25.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:51:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17596.211598 secs 2023-03-09T19:51:26.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:51:26 smithi122 ceph-mon[93869]: pgmap v9897: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 80 KiB/s wr, 8 op/s 2023-03-09T19:51:26.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:51:26 smithi181 ceph-mon[94189]: pgmap v9897: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 80 KiB/s wr, 8 op/s 2023-03-09T19:51:26.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:51:26 smithi038 ceph-mon[92422]: pgmap v9897: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 80 KiB/s wr, 8 op/s 2023-03-09T19:51:27.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:51:27 smithi122 ceph-mon[93869]: pgmap v9898: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 135 KiB/s wr, 8 op/s 2023-03-09T19:51:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:51:27 smithi181 ceph-mon[94189]: pgmap v9898: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 135 KiB/s wr, 8 op/s 2023-03-09T19:51:27.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:51:27 smithi038 ceph-mon[92422]: pgmap v9898: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 135 KiB/s wr, 8 op/s 2023-03-09T19:51:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18060.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:51:28.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:51:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17598.525340 secs 2023-03-09T19:51:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:51:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17598.525340 secs 2023-03-09T19:51:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:51:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17598.525340 secs 2023-03-09T19:51:29.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:51:29 smithi122 ceph-mon[93869]: pgmap v9899: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 120 KiB/s wr, 7 op/s 2023-03-09T19:51:29.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:51:29 smithi181 ceph-mon[94189]: pgmap v9899: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 120 KiB/s wr, 7 op/s 2023-03-09T19:51:29.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:51:29 smithi038 ceph-mon[92422]: pgmap v9899: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 120 KiB/s wr, 7 op/s 2023-03-09T19:51:30.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:51:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17601.211658 secs 2023-03-09T19:51:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:51:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17601.211658 secs 2023-03-09T19:51:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:51:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17601.211658 secs 2023-03-09T19:51:31.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:51:31 smithi122 ceph-mon[93869]: pgmap v9900: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 114 KiB/s wr, 9 op/s 2023-03-09T19:51:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:51:31 smithi181 ceph-mon[94189]: pgmap v9900: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 114 KiB/s wr, 9 op/s 2023-03-09T19:51:31.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:51:31 smithi038 ceph-mon[92422]: pgmap v9900: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 114 KiB/s wr, 9 op/s 2023-03-09T19:51:33.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:51:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17603.525437 secs 2023-03-09T19:51:33.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:51:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17603.525437 secs 2023-03-09T19:51:33.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:51:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17603.525437 secs 2023-03-09T19:51:33.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18065.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:51:34.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:51:34 smithi122 ceph-mon[93869]: pgmap v9901: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 64 KiB/s wr, 6 op/s 2023-03-09T19:51:34.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:51:34 smithi181 ceph-mon[94189]: pgmap v9901: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 64 KiB/s wr, 6 op/s 2023-03-09T19:51:34.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:51:34 smithi038 ceph-mon[92422]: pgmap v9901: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 64 KiB/s wr, 6 op/s 2023-03-09T19:51:35.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:51:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17606.211717 secs 2023-03-09T19:51:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:51:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17606.211717 secs 2023-03-09T19:51:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:51:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17606.211717 secs 2023-03-09T19:51:36.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:51:36 smithi122 ceph-mon[93869]: pgmap v9902: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 64 KiB/s wr, 6 op/s 2023-03-09T19:51:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:51:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:51:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:51:36 smithi181 ceph-mon[94189]: pgmap v9902: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 64 KiB/s wr, 6 op/s 2023-03-09T19:51:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:51:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:51:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:51:36 smithi038 ceph-mon[92422]: pgmap v9902: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 64 KiB/s wr, 6 op/s 2023-03-09T19:51:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:51:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:51:38.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18070.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:51:38.964 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:51:38 smithi181 ceph-mon[94189]: pgmap v9903: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 212 KiB/s wr, 10 op/s 2023-03-09T19:51:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:51:38 smithi122 ceph-mon[93869]: pgmap v9903: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 212 KiB/s wr, 10 op/s 2023-03-09T19:51:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:51:38 smithi038 ceph-mon[92422]: pgmap v9903: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 212 KiB/s wr, 10 op/s 2023-03-09T19:51:40.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:51:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:51:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:51:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:51:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:51:40 smithi122 ceph-mon[93869]: pgmap v9904: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 149 KiB/s wr, 7 op/s 2023-03-09T19:51:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:51:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:51:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:51:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:51:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:51:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:51:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:51:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:51:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:51:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:51:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:51:40 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:51:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:51:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:51:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:51:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:51:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:51:40 smithi181 ceph-mon[94189]: pgmap v9904: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 149 KiB/s wr, 7 op/s 2023-03-09T19:51:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:51:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:51:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:51:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:51:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:51:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:51:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:51:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:51:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:51:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:51:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:51:40 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:51:40.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:51:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:51:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:51:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:51:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:51:40 smithi038 ceph-mon[92422]: pgmap v9904: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 149 KiB/s wr, 7 op/s 2023-03-09T19:51:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:51:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:51:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:51:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:51:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:51:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:51:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:51:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:51:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:51:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:51:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:51:40 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:51:41.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:51:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17611.211780 secs 2023-03-09T19:51:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:51:41 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:51:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:51:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:51:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:51:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:51:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:51:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17611.211780 secs 2023-03-09T19:51:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:51:41 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:51:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:51:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:51:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:51:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:51:41.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:51:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17611.211780 secs 2023-03-09T19:51:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:51:41 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:51:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:51:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:51:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:51:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:51:42.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:51:42 smithi122 ceph-mon[93869]: pgmap v9905: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 203 KiB/s wr, 10 op/s 2023-03-09T19:51:42.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:51:42 smithi181 ceph-mon[94189]: pgmap v9905: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 203 KiB/s wr, 10 op/s 2023-03-09T19:51:42.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:51:42 smithi038 ceph-mon[92422]: pgmap v9905: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 203 KiB/s wr, 10 op/s 2023-03-09T19:51:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18075.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:51:44.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:51:44 smithi122 ceph-mon[93869]: pgmap v9906: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 202 KiB/s wr, 7 op/s 2023-03-09T19:51:44.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:51:44 smithi181 ceph-mon[94189]: pgmap v9906: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 202 KiB/s wr, 7 op/s 2023-03-09T19:51:44.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:51:44 smithi038 ceph-mon[92422]: pgmap v9906: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 202 KiB/s wr, 7 op/s 2023-03-09T19:51:45.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:51:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17616.211841 secs 2023-03-09T19:51:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:51:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17616.211841 secs 2023-03-09T19:51:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:51:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17616.211841 secs 2023-03-09T19:51:46.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:51:46 smithi122 ceph-mon[93869]: pgmap v9907: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 322 KiB/s wr, 8 op/s 2023-03-09T19:51:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:51:46 smithi181 ceph-mon[94189]: pgmap v9907: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 322 KiB/s wr, 8 op/s 2023-03-09T19:51:46.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:51:46 smithi038 ceph-mon[92422]: pgmap v9907: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 322 KiB/s wr, 8 op/s 2023-03-09T19:51:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18080.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:51:48.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:51:48 smithi122 ceph-mon[93869]: pgmap v9908: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 337 KiB/s wr, 12 op/s 2023-03-09T19:51:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:51:48 smithi181 ceph-mon[94189]: pgmap v9908: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 337 KiB/s wr, 12 op/s 2023-03-09T19:51:48.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:51:48 smithi038 ceph-mon[92422]: pgmap v9908: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 337 KiB/s wr, 12 op/s 2023-03-09T19:51:50.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:51:50 smithi122 ceph-mon[93869]: pgmap v9909: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 190 KiB/s wr, 9 op/s 2023-03-09T19:51:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:51:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17621.211926 secs 2023-03-09T19:51:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:51:50 smithi181 ceph-mon[94189]: pgmap v9909: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 190 KiB/s wr, 9 op/s 2023-03-09T19:51:50.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:51:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17621.211926 secs 2023-03-09T19:51:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:51:50 smithi038 ceph-mon[92422]: pgmap v9909: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 190 KiB/s wr, 9 op/s 2023-03-09T19:51:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:51:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17621.211926 secs 2023-03-09T19:51:51.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:51:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:51:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:51:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:51:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:51:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:51:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:51:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:51:51.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:51:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:51:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:51:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:51:52.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:51:52 smithi122 ceph-mon[93869]: pgmap v9910: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 453 KiB/s rd, 398 KiB/s wr, 13 op/s 2023-03-09T19:51:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:51:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:51:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:51:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:51:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:51:52 smithi181 ceph-mon[94189]: pgmap v9910: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 453 KiB/s rd, 398 KiB/s wr, 13 op/s 2023-03-09T19:51:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:51:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:51:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:51:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:51:52.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:51:52 smithi038 ceph-mon[92422]: pgmap v9910: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 453 KiB/s rd, 398 KiB/s wr, 13 op/s 2023-03-09T19:51:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:51:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:51:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:51:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:51:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18085.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:51:54.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:51:54 smithi122 ceph-mon[93869]: pgmap v9911: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 344 KiB/s wr, 10 op/s 2023-03-09T19:51:54.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:51:54 smithi181 ceph-mon[94189]: pgmap v9911: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 344 KiB/s wr, 10 op/s 2023-03-09T19:51:54.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:51:54 smithi038 ceph-mon[92422]: pgmap v9911: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 344 KiB/s wr, 10 op/s 2023-03-09T19:51:55.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:51:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17626.211997 secs 2023-03-09T19:51:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:51:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17626.211997 secs 2023-03-09T19:51:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:51:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17626.211997 secs 2023-03-09T19:51:56.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:51:56 smithi122 ceph-mon[93869]: pgmap v9912: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 372 KiB/s wr, 11 op/s 2023-03-09T19:51:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:51:56 smithi181 ceph-mon[94189]: pgmap v9912: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 372 KiB/s wr, 11 op/s 2023-03-09T19:51:56.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:51:56 smithi038 ceph-mon[92422]: pgmap v9912: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 372 KiB/s wr, 11 op/s 2023-03-09T19:51:58.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18090.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:51:58.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:51:58 smithi122 ceph-mon[93869]: pgmap v9913: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 313 KiB/s wr, 14 op/s 2023-03-09T19:51:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:51:58 smithi181 ceph-mon[94189]: pgmap v9913: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 313 KiB/s wr, 14 op/s 2023-03-09T19:51:58.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:51:58 smithi038 ceph-mon[92422]: pgmap v9913: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 313 KiB/s wr, 14 op/s 2023-03-09T19:51:59.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:51:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:51:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:51:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:51:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:51:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:51:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:51:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:51:59.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:51:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:51:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:51:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:52:00.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:00 smithi122 ceph-mon[93869]: pgmap v9914: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 299 KiB/s wr, 10 op/s 2023-03-09T19:52:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:52:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:52:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17631.212058 secs 2023-03-09T19:52:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:00 smithi181 ceph-mon[94189]: pgmap v9914: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 299 KiB/s wr, 10 op/s 2023-03-09T19:52:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:52:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:52:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17631.212058 secs 2023-03-09T19:52:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:00 smithi038 ceph-mon[92422]: pgmap v9914: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 299 KiB/s wr, 10 op/s 2023-03-09T19:52:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:52:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:52:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17631.212058 secs 2023-03-09T19:52:02.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:02 smithi122 ceph-mon[93869]: pgmap v9915: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 324 KiB/s wr, 14 op/s 2023-03-09T19:52:02.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:02 smithi181 ceph-mon[94189]: pgmap v9915: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 324 KiB/s wr, 14 op/s 2023-03-09T19:52:02.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:02 smithi038 ceph-mon[92422]: pgmap v9915: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 324 KiB/s wr, 14 op/s 2023-03-09T19:52:03.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18095.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:52:03.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17633.526133 secs 2023-03-09T19:52:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17633.526133 secs 2023-03-09T19:52:03.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17633.526133 secs 2023-03-09T19:52:04.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:04 smithi122 ceph-mon[93869]: pgmap v9916: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 263 KiB/s wr, 24 op/s 2023-03-09T19:52:04.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:04 smithi181 ceph-mon[94189]: pgmap v9916: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 263 KiB/s wr, 24 op/s 2023-03-09T19:52:04.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:04 smithi038 ceph-mon[92422]: pgmap v9916: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 263 KiB/s wr, 24 op/s 2023-03-09T19:52:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:05 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:52:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:05 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:52:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:05 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:52:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17636.212117 secs 2023-03-09T19:52:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:05 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:52:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:05 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:52:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:05 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:52:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17636.212117 secs 2023-03-09T19:52:05.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:05 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:52:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:05 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:52:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:05 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:52:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17636.212117 secs 2023-03-09T19:52:06.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:06 smithi122 ceph-mon[93869]: pgmap v9917: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 263 KiB/s wr, 24 op/s 2023-03-09T19:52:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:06 smithi181 ceph-mon[94189]: pgmap v9917: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 263 KiB/s wr, 24 op/s 2023-03-09T19:52:06.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:06 smithi038 ceph-mon[92422]: pgmap v9917: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 263 KiB/s wr, 24 op/s 2023-03-09T19:52:08.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18100.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:52:08.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17638.526250 secs 2023-03-09T19:52:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:08 smithi122 ceph-mon[93869]: pgmap v9918: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 345 KiB/s wr, 27 op/s 2023-03-09T19:52:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17638.526250 secs 2023-03-09T19:52:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:08 smithi181 ceph-mon[94189]: pgmap v9918: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 345 KiB/s wr, 27 op/s 2023-03-09T19:52:08.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17638.526250 secs 2023-03-09T19:52:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:08 smithi038 ceph-mon[92422]: pgmap v9918: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 345 KiB/s wr, 27 op/s 2023-03-09T19:52:10.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:10 smithi122 ceph-mon[93869]: pgmap v9919: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 284 KiB/s wr, 24 op/s 2023-03-09T19:52:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:10 smithi122 ceph-mon[93869]: osdmap e833: 12 total, 12 up, 12 in 2023-03-09T19:52:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17641.212189 secs 2023-03-09T19:52:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:10 smithi181 ceph-mon[94189]: pgmap v9919: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 284 KiB/s wr, 24 op/s 2023-03-09T19:52:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:10 smithi181 ceph-mon[94189]: osdmap e833: 12 total, 12 up, 12 in 2023-03-09T19:52:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17641.212189 secs 2023-03-09T19:52:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:10 smithi038 ceph-mon[92422]: pgmap v9919: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 284 KiB/s wr, 24 op/s 2023-03-09T19:52:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:10 smithi038 ceph-mon[92422]: osdmap e833: 12 total, 12 up, 12 in 2023-03-09T19:52:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17641.212189 secs 2023-03-09T19:52:11.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:11 smithi122 ceph-mon[93869]: pgmap v9921: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 326 KiB/s wr, 25 op/s 2023-03-09T19:52:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:11 smithi181 ceph-mon[94189]: pgmap v9921: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 326 KiB/s wr, 25 op/s 2023-03-09T19:52:11.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:11 smithi038 ceph-mon[92422]: pgmap v9921: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 326 KiB/s wr, 25 op/s 2023-03-09T19:52:13.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17643.526357 secs 2023-03-09T19:52:13.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17643.526357 secs 2023-03-09T19:52:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17643.526357 secs 2023-03-09T19:52:13.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18105.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:52:14.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:14 smithi122 ceph-mon[93869]: pgmap v9922: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 150 KiB/s wr, 9 op/s 2023-03-09T19:52:14.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:14 smithi181 ceph-mon[94189]: pgmap v9922: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 150 KiB/s wr, 9 op/s 2023-03-09T19:52:14.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:14 smithi038 ceph-mon[92422]: pgmap v9922: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 150 KiB/s wr, 9 op/s 2023-03-09T19:52:15.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17646.212284 secs 2023-03-09T19:52:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17646.212284 secs 2023-03-09T19:52:15.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17646.212284 secs 2023-03-09T19:52:16.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:16 smithi122 ceph-mon[93869]: pgmap v9923: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 150 KiB/s wr, 9 op/s 2023-03-09T19:52:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:16 smithi181 ceph-mon[94189]: pgmap v9923: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 150 KiB/s wr, 9 op/s 2023-03-09T19:52:16.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:16 smithi038 ceph-mon[92422]: pgmap v9923: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 150 KiB/s wr, 9 op/s 2023-03-09T19:52:18.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18110.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:52:18.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17648.526480 secs 2023-03-09T19:52:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:18 smithi122 ceph-mon[93869]: pgmap v9924: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 68 KiB/s wr, 7 op/s 2023-03-09T19:52:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:18 smithi122 ceph-mon[93869]: osdmap e834: 12 total, 12 up, 12 in 2023-03-09T19:52:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17648.526480 secs 2023-03-09T19:52:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:18 smithi181 ceph-mon[94189]: pgmap v9924: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 68 KiB/s wr, 7 op/s 2023-03-09T19:52:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:18 smithi181 ceph-mon[94189]: osdmap e834: 12 total, 12 up, 12 in 2023-03-09T19:52:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17648.526480 secs 2023-03-09T19:52:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:18 smithi038 ceph-mon[92422]: pgmap v9924: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 68 KiB/s wr, 7 op/s 2023-03-09T19:52:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:18 smithi038 ceph-mon[92422]: osdmap e834: 12 total, 12 up, 12 in 2023-03-09T19:52:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:20 smithi122 ceph-mon[93869]: pgmap v9926: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 69 KiB/s wr, 7 op/s 2023-03-09T19:52:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17651.212344 secs 2023-03-09T19:52:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:20 smithi181 ceph-mon[94189]: pgmap v9926: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 69 KiB/s wr, 7 op/s 2023-03-09T19:52:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17651.212344 secs 2023-03-09T19:52:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:20 smithi038 ceph-mon[92422]: pgmap v9926: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 69 KiB/s wr, 7 op/s 2023-03-09T19:52:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17651.212344 secs 2023-03-09T19:52:22.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:22 smithi122 ceph-mon[93869]: pgmap v9927: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 52 KiB/s wr, 7 op/s 2023-03-09T19:52:22.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:22 smithi181 ceph-mon[94189]: pgmap v9927: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 52 KiB/s wr, 7 op/s 2023-03-09T19:52:22.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:22 smithi038 ceph-mon[92422]: pgmap v9927: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 52 KiB/s wr, 7 op/s 2023-03-09T19:52:23.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18115.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:52:23.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17653.526626 secs 2023-03-09T19:52:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17653.526626 secs 2023-03-09T19:52:23.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17653.526626 secs 2023-03-09T19:52:24.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:24 smithi122 ceph-mon[93869]: pgmap v9928: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 52 KiB/s wr, 7 op/s 2023-03-09T19:52:24.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:24 smithi181 ceph-mon[94189]: pgmap v9928: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 52 KiB/s wr, 7 op/s 2023-03-09T19:52:24.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:24 smithi038 ceph-mon[92422]: pgmap v9928: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 52 KiB/s wr, 7 op/s 2023-03-09T19:52:25.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17656.212407 secs 2023-03-09T19:52:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17656.212407 secs 2023-03-09T19:52:25.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17656.212407 secs 2023-03-09T19:52:26.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:26 smithi122 ceph-mon[93869]: pgmap v9929: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 52 KiB/s wr, 7 op/s 2023-03-09T19:52:26.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:26 smithi181 ceph-mon[94189]: pgmap v9929: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 52 KiB/s wr, 7 op/s 2023-03-09T19:52:26.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:26 smithi038 ceph-mon[92422]: pgmap v9929: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 52 KiB/s wr, 7 op/s 2023-03-09T19:52:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18120.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:52:28.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17658.526700 secs 2023-03-09T19:52:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:28 smithi122 ceph-mon[93869]: pgmap v9930: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 111 KiB/s wr, 8 op/s 2023-03-09T19:52:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17658.526700 secs 2023-03-09T19:52:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:28 smithi181 ceph-mon[94189]: pgmap v9930: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 111 KiB/s wr, 8 op/s 2023-03-09T19:52:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17658.526700 secs 2023-03-09T19:52:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:28 smithi038 ceph-mon[92422]: pgmap v9930: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 111 KiB/s wr, 8 op/s 2023-03-09T19:52:30.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:30 smithi122 ceph-mon[93869]: pgmap v9931: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 98 KiB/s wr, 7 op/s 2023-03-09T19:52:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17661.212468 secs 2023-03-09T19:52:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:30 smithi181 ceph-mon[94189]: pgmap v9931: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 98 KiB/s wr, 7 op/s 2023-03-09T19:52:30.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17661.212468 secs 2023-03-09T19:52:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:30 smithi038 ceph-mon[92422]: pgmap v9931: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 98 KiB/s wr, 7 op/s 2023-03-09T19:52:30.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17661.212468 secs 2023-03-09T19:52:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:31 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:52:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:31 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:52:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:31 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:52:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:31 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:52:31.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:31 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:52:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:31 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:52:32.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:32 smithi122 ceph-mon[93869]: pgmap v9932: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 KiB/s wr, 9 op/s 2023-03-09T19:52:32.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:32 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:52:32.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:32 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:52:32.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:32 smithi181 ceph-mon[94189]: pgmap v9932: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 KiB/s wr, 9 op/s 2023-03-09T19:52:32.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:32 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:52:32.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:32 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:52:32.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:32 smithi038 ceph-mon[92422]: pgmap v9932: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 KiB/s wr, 9 op/s 2023-03-09T19:52:32.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:32 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:52:32.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:32 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:52:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18125.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:52:33.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17663.526778 secs 2023-03-09T19:52:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17663.526778 secs 2023-03-09T19:52:33.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17663.526778 secs 2023-03-09T19:52:34.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:34 smithi122 ceph-mon[93869]: pgmap v9933: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 91 KiB/s wr, 8 op/s 2023-03-09T19:52:34.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:34 smithi181 ceph-mon[94189]: pgmap v9933: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 91 KiB/s wr, 8 op/s 2023-03-09T19:52:34.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:34 smithi038 ceph-mon[92422]: pgmap v9933: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 91 KiB/s wr, 8 op/s 2023-03-09T19:52:35.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17666.212529 secs 2023-03-09T19:52:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17666.212529 secs 2023-03-09T19:52:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17666.212529 secs 2023-03-09T19:52:36.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:36 smithi122 ceph-mon[93869]: pgmap v9934: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 91 KiB/s wr, 7 op/s 2023-03-09T19:52:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:36 smithi181 ceph-mon[94189]: pgmap v9934: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 91 KiB/s wr, 7 op/s 2023-03-09T19:52:36.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:36 smithi038 ceph-mon[92422]: pgmap v9934: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 91 KiB/s wr, 7 op/s 2023-03-09T19:52:37.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:37 smithi122 ceph-mon[93869]: pgmap v9935: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 KiB/s wr, 10 op/s 2023-03-09T19:52:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:37 smithi181 ceph-mon[94189]: pgmap v9935: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 KiB/s wr, 10 op/s 2023-03-09T19:52:37.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:37 smithi038 ceph-mon[92422]: pgmap v9935: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 KiB/s wr, 10 op/s 2023-03-09T19:52:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18130.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:52:40.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:40 smithi122 ceph-mon[93869]: pgmap v9936: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 767 B/s wr, 7 op/s 2023-03-09T19:52:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17671.212606 secs 2023-03-09T19:52:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:52:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:40 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:52:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:52:40.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:52:40.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:52:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:40 smithi181 ceph-mon[94189]: pgmap v9936: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 767 B/s wr, 7 op/s 2023-03-09T19:52:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17671.212606 secs 2023-03-09T19:52:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:52:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:40 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:52:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:52:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:52:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:52:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:40 smithi038 ceph-mon[92422]: pgmap v9936: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 767 B/s wr, 7 op/s 2023-03-09T19:52:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17671.212606 secs 2023-03-09T19:52:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:52:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:40 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:52:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:52:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:52:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:52:41.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:41 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:52:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:52:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:52:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:41 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:52:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:52:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:52:41.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:41 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:52:41.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:52:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:52:42.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:42 smithi122 ceph-mon[93869]: pgmap v9937: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 163 KiB/s wr, 10 op/s 2023-03-09T19:52:42.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:42 smithi181 ceph-mon[94189]: pgmap v9937: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 163 KiB/s wr, 10 op/s 2023-03-09T19:52:42.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:42 smithi038 ceph-mon[92422]: pgmap v9937: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 163 KiB/s wr, 10 op/s 2023-03-09T19:52:43.793 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18135.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:52:43.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:43 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 17673.527019 secs 2023-03-09T19:52:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:43 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 17673.527019 secs 2023-03-09T19:52:43.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:43 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 17673.527019 secs 2023-03-09T19:52:44.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:44 smithi122 ceph-mon[93869]: pgmap v9938: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 163 KiB/s wr, 8 op/s 2023-03-09T19:52:44.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:44 smithi181 ceph-mon[94189]: pgmap v9938: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 163 KiB/s wr, 8 op/s 2023-03-09T19:52:44.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:44 smithi038 ceph-mon[92422]: pgmap v9938: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 163 KiB/s wr, 8 op/s 2023-03-09T19:52:45.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:45 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:52:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:45 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:52:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17676.212669 secs 2023-03-09T19:52:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:45 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:52:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:45 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:52:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17676.212669 secs 2023-03-09T19:52:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:45 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:52:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:45 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:52:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17676.212669 secs 2023-03-09T19:52:46.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:46 smithi122 ceph-mon[93869]: pgmap v9939: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 163 KiB/s wr, 7 op/s 2023-03-09T19:52:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:46 smithi181 ceph-mon[94189]: pgmap v9939: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 163 KiB/s wr, 7 op/s 2023-03-09T19:52:46.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:46 smithi038 ceph-mon[92422]: pgmap v9939: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 163 KiB/s wr, 7 op/s 2023-03-09T19:52:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18140.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:52:48.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:48 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 17678.527137 secs 2023-03-09T19:52:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:48 smithi122 ceph-mon[93869]: pgmap v9940: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 337 KiB/s rd, 309 KiB/s wr, 11 op/s 2023-03-09T19:52:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:48 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:52:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:48 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 17678.527137 secs 2023-03-09T19:52:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:48 smithi181 ceph-mon[94189]: pgmap v9940: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 337 KiB/s rd, 309 KiB/s wr, 11 op/s 2023-03-09T19:52:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:48 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:52:48.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:48 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 17678.527137 secs 2023-03-09T19:52:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:48 smithi038 ceph-mon[92422]: pgmap v9940: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 337 KiB/s rd, 309 KiB/s wr, 11 op/s 2023-03-09T19:52:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:48 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:52:50.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:50 smithi122 ceph-mon[93869]: pgmap v9941: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 432 KiB/s rd, 422 KiB/s wr, 9 op/s 2023-03-09T19:52:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17681.212729 secs 2023-03-09T19:52:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:50 smithi181 ceph-mon[94189]: pgmap v9941: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 432 KiB/s rd, 422 KiB/s wr, 9 op/s 2023-03-09T19:52:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17681.212729 secs 2023-03-09T19:52:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:50 smithi038 ceph-mon[92422]: pgmap v9941: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 432 KiB/s rd, 422 KiB/s wr, 9 op/s 2023-03-09T19:52:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17681.212729 secs 2023-03-09T19:52:51.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:52:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:52:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:52:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:52:51.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:52:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:52:52.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:52 smithi122 ceph-mon[93869]: pgmap v9942: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 472 KiB/s rd, 437 KiB/s wr, 12 op/s 2023-03-09T19:52:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:52:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:52:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:52 smithi181 ceph-mon[94189]: pgmap v9942: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 472 KiB/s rd, 437 KiB/s wr, 12 op/s 2023-03-09T19:52:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:52:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:52:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:52 smithi038 ceph-mon[92422]: pgmap v9942: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 472 KiB/s rd, 437 KiB/s wr, 12 op/s 2023-03-09T19:52:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:52:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:52:53.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18145.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:52:53.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17683.527256 secs 2023-03-09T19:52:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:52:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:52:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:53 smithi122 ceph-mon[93869]: pgmap v9943: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 275 KiB/s wr, 9 op/s 2023-03-09T19:52:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17683.527256 secs 2023-03-09T19:52:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:52:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:52:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:53 smithi181 ceph-mon[94189]: pgmap v9943: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 275 KiB/s wr, 9 op/s 2023-03-09T19:52:53.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17683.527256 secs 2023-03-09T19:52:53.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:52:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:52:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:53 smithi038 ceph-mon[92422]: pgmap v9943: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 275 KiB/s wr, 9 op/s 2023-03-09T19:52:55.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17686.212790 secs 2023-03-09T19:52:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17686.212790 secs 2023-03-09T19:52:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17686.212790 secs 2023-03-09T19:52:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:56 smithi122 ceph-mon[93869]: pgmap v9944: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 326 KiB/s rd, 393 KiB/s wr, 10 op/s 2023-03-09T19:52:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:56 smithi181 ceph-mon[94189]: pgmap v9944: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 326 KiB/s rd, 393 KiB/s wr, 10 op/s 2023-03-09T19:52:56.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:56 smithi038 ceph-mon[92422]: pgmap v9944: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 326 KiB/s rd, 393 KiB/s wr, 10 op/s 2023-03-09T19:52:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18150.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:52:58.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17688.527404 secs 2023-03-09T19:52:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:52:58 smithi122 ceph-mon[93869]: pgmap v9945: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 394 KiB/s wr, 13 op/s 2023-03-09T19:52:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17688.527404 secs 2023-03-09T19:52:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:52:58 smithi181 ceph-mon[94189]: pgmap v9945: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 394 KiB/s wr, 13 op/s 2023-03-09T19:52:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17688.527404 secs 2023-03-09T19:52:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:52:58 smithi038 ceph-mon[92422]: pgmap v9945: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 394 KiB/s wr, 13 op/s 2023-03-09T19:53:00.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:00 smithi122 ceph-mon[93869]: pgmap v9946: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 383 KiB/s rd, 358 KiB/s wr, 10 op/s 2023-03-09T19:53:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:53:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:53:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17691.212851 secs 2023-03-09T19:53:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:00 smithi181 ceph-mon[94189]: pgmap v9946: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 383 KiB/s rd, 358 KiB/s wr, 10 op/s 2023-03-09T19:53:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:53:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:53:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17691.212851 secs 2023-03-09T19:53:01.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:00 smithi038 ceph-mon[92422]: pgmap v9946: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 383 KiB/s rd, 358 KiB/s wr, 10 op/s 2023-03-09T19:53:01.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:53:01.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:53:01.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17691.212851 secs 2023-03-09T19:53:02.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:02 smithi122 ceph-mon[93869]: pgmap v9947: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 356 KiB/s wr, 12 op/s 2023-03-09T19:53:02.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:02 smithi181 ceph-mon[94189]: pgmap v9947: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 356 KiB/s wr, 12 op/s 2023-03-09T19:53:02.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:02 smithi038 ceph-mon[92422]: pgmap v9947: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 356 KiB/s wr, 12 op/s 2023-03-09T19:53:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18155.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:53:03.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17693.527521 secs 2023-03-09T19:53:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17693.527521 secs 2023-03-09T19:53:03.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17693.527521 secs 2023-03-09T19:53:04.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:04 smithi122 ceph-mon[93869]: pgmap v9948: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 363 KiB/s wr, 22 op/s 2023-03-09T19:53:04.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:04 smithi181 ceph-mon[94189]: pgmap v9948: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 363 KiB/s wr, 22 op/s 2023-03-09T19:53:04.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:04 smithi038 ceph-mon[92422]: pgmap v9948: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 363 KiB/s wr, 22 op/s 2023-03-09T19:53:05.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17696.212913 secs 2023-03-09T19:53:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17696.212913 secs 2023-03-09T19:53:05.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17696.212913 secs 2023-03-09T19:53:06.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:06 smithi122 ceph-mon[93869]: pgmap v9949: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 363 KiB/s wr, 21 op/s 2023-03-09T19:53:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:06 smithi181 ceph-mon[94189]: pgmap v9949: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 363 KiB/s wr, 21 op/s 2023-03-09T19:53:06.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:06 smithi038 ceph-mon[92422]: pgmap v9949: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 363 KiB/s wr, 21 op/s 2023-03-09T19:53:07.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:53:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:53:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:53:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:53:07.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:53:07.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:53:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18160.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:53:08.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17698.527677 secs 2023-03-09T19:53:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:08 smithi122 ceph-mon[93869]: pgmap v9950: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 328 KiB/s wr, 27 op/s 2023-03-09T19:53:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17698.527677 secs 2023-03-09T19:53:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:08 smithi181 ceph-mon[94189]: pgmap v9950: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 328 KiB/s wr, 27 op/s 2023-03-09T19:53:08.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17698.527677 secs 2023-03-09T19:53:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:08 smithi038 ceph-mon[92422]: pgmap v9950: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 328 KiB/s wr, 27 op/s 2023-03-09T19:53:09.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:09 smithi122 ceph-mon[93869]: pgmap v9951: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 328 KiB/s wr, 24 op/s 2023-03-09T19:53:10.015 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:09 smithi038 ceph-mon[92422]: pgmap v9951: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 328 KiB/s wr, 24 op/s 2023-03-09T19:53:10.015 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:09 smithi181 ceph-mon[94189]: pgmap v9951: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 328 KiB/s wr, 24 op/s 2023-03-09T19:53:10.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:10 smithi122 ceph-mon[93869]: osdmap e835: 12 total, 12 up, 12 in 2023-03-09T19:53:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17701.214001 secs 2023-03-09T19:53:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:10 smithi181 ceph-mon[94189]: osdmap e835: 12 total, 12 up, 12 in 2023-03-09T19:53:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17701.214001 secs 2023-03-09T19:53:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:10 smithi038 ceph-mon[92422]: osdmap e835: 12 total, 12 up, 12 in 2023-03-09T19:53:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17701.214001 secs 2023-03-09T19:53:11.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:11 smithi122 ceph-mon[93869]: pgmap v9953: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 245 KiB/s wr, 27 op/s 2023-03-09T19:53:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:11 smithi181 ceph-mon[94189]: pgmap v9953: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 245 KiB/s wr, 27 op/s 2023-03-09T19:53:11.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:11 smithi038 ceph-mon[92422]: pgmap v9953: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 245 KiB/s wr, 27 op/s 2023-03-09T19:53:13.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17703.527795 secs 2023-03-09T19:53:13.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17703.527795 secs 2023-03-09T19:53:13.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17703.527795 secs 2023-03-09T19:53:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18165.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:53:14.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:14 smithi122 ceph-mon[93869]: pgmap v9954: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 219 KiB/s wr, 12 op/s 2023-03-09T19:53:14.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:14 smithi181 ceph-mon[94189]: pgmap v9954: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 219 KiB/s wr, 12 op/s 2023-03-09T19:53:14.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:14 smithi038 ceph-mon[92422]: pgmap v9954: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 219 KiB/s wr, 12 op/s 2023-03-09T19:53:15.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17706.214061 secs 2023-03-09T19:53:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17706.214061 secs 2023-03-09T19:53:15.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17706.214061 secs 2023-03-09T19:53:16.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:16 smithi122 ceph-mon[93869]: pgmap v9955: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 219 KiB/s wr, 12 op/s 2023-03-09T19:53:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:16 smithi181 ceph-mon[94189]: pgmap v9955: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 219 KiB/s wr, 12 op/s 2023-03-09T19:53:16.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:16 smithi038 ceph-mon[92422]: pgmap v9955: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 219 KiB/s wr, 12 op/s 2023-03-09T19:53:18.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18170.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:53:18.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17708.527886 secs 2023-03-09T19:53:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:18 smithi122 ceph-mon[93869]: pgmap v9956: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 188 KiB/s wr, 9 op/s 2023-03-09T19:53:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:18 smithi122 ceph-mon[93869]: osdmap e836: 12 total, 12 up, 12 in 2023-03-09T19:53:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17708.527886 secs 2023-03-09T19:53:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:18 smithi181 ceph-mon[94189]: pgmap v9956: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 188 KiB/s wr, 9 op/s 2023-03-09T19:53:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:18 smithi181 ceph-mon[94189]: osdmap e836: 12 total, 12 up, 12 in 2023-03-09T19:53:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17708.527886 secs 2023-03-09T19:53:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:18 smithi038 ceph-mon[92422]: pgmap v9956: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 188 KiB/s wr, 9 op/s 2023-03-09T19:53:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:18 smithi038 ceph-mon[92422]: osdmap e836: 12 total, 12 up, 12 in 2023-03-09T19:53:20.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:20 smithi122 ceph-mon[93869]: pgmap v9958: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 190 KiB/s wr, 9 op/s 2023-03-09T19:53:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17711.214120 secs 2023-03-09T19:53:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:20 smithi181 ceph-mon[94189]: pgmap v9958: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 190 KiB/s wr, 9 op/s 2023-03-09T19:53:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17711.214120 secs 2023-03-09T19:53:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:20 smithi038 ceph-mon[92422]: pgmap v9958: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 190 KiB/s wr, 9 op/s 2023-03-09T19:53:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17711.214120 secs 2023-03-09T19:53:22.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:22 smithi122 ceph-mon[93869]: pgmap v9959: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 71 KiB/s wr, 8 op/s 2023-03-09T19:53:22.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:22 smithi181 ceph-mon[94189]: pgmap v9959: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 71 KiB/s wr, 8 op/s 2023-03-09T19:53:22.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:22 smithi038 ceph-mon[92422]: pgmap v9959: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 71 KiB/s wr, 8 op/s 2023-03-09T19:53:23.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18175.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:53:23.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17713.528003 secs 2023-03-09T19:53:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17713.528003 secs 2023-03-09T19:53:23.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17713.528003 secs 2023-03-09T19:53:24.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:24 smithi122 ceph-mon[93869]: pgmap v9960: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 71 KiB/s wr, 8 op/s 2023-03-09T19:53:24.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:24 smithi181 ceph-mon[94189]: pgmap v9960: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 71 KiB/s wr, 8 op/s 2023-03-09T19:53:24.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:24 smithi038 ceph-mon[92422]: pgmap v9960: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 71 KiB/s wr, 8 op/s 2023-03-09T19:53:25.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17716.214181 secs 2023-03-09T19:53:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17716.214181 secs 2023-03-09T19:53:25.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17716.214181 secs 2023-03-09T19:53:26.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:26 smithi122 ceph-mon[93869]: pgmap v9961: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 71 KiB/s wr, 7 op/s 2023-03-09T19:53:26.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:26 smithi181 ceph-mon[94189]: pgmap v9961: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 71 KiB/s wr, 7 op/s 2023-03-09T19:53:26.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:26 smithi038 ceph-mon[92422]: pgmap v9961: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 71 KiB/s wr, 7 op/s 2023-03-09T19:53:28.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18180.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:53:28.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17718.528136 secs 2023-03-09T19:53:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:28 smithi122 ceph-mon[93869]: pgmap v9962: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 119 KiB/s wr, 7 op/s 2023-03-09T19:53:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17718.528136 secs 2023-03-09T19:53:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:28 smithi181 ceph-mon[94189]: pgmap v9962: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 119 KiB/s wr, 7 op/s 2023-03-09T19:53:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17718.528136 secs 2023-03-09T19:53:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:28 smithi038 ceph-mon[92422]: pgmap v9962: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 119 KiB/s wr, 7 op/s 2023-03-09T19:53:30.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:30 smithi122 ceph-mon[93869]: pgmap v9963: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 106 KiB/s wr, 7 op/s 2023-03-09T19:53:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17721.214267 secs 2023-03-09T19:53:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:30 smithi181 ceph-mon[94189]: pgmap v9963: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 106 KiB/s wr, 7 op/s 2023-03-09T19:53:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17721.214267 secs 2023-03-09T19:53:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:30 smithi038 ceph-mon[92422]: pgmap v9963: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 106 KiB/s wr, 7 op/s 2023-03-09T19:53:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17721.214267 secs 2023-03-09T19:53:32.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:32 smithi122 ceph-mon[93869]: pgmap v9964: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 100 KiB/s wr, 10 op/s 2023-03-09T19:53:32.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:32 smithi181 ceph-mon[94189]: pgmap v9964: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 100 KiB/s wr, 10 op/s 2023-03-09T19:53:32.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:32 smithi038 ceph-mon[92422]: pgmap v9964: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 100 KiB/s wr, 10 op/s 2023-03-09T19:53:33.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18185.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:53:33.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17723.528246 secs 2023-03-09T19:53:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17723.528246 secs 2023-03-09T19:53:33.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17723.528246 secs 2023-03-09T19:53:34.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:34 smithi122 ceph-mon[93869]: pgmap v9965: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 100 KiB/s wr, 7 op/s 2023-03-09T19:53:34.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:34 smithi181 ceph-mon[94189]: pgmap v9965: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 100 KiB/s wr, 7 op/s 2023-03-09T19:53:34.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:34 smithi038 ceph-mon[92422]: pgmap v9965: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 100 KiB/s wr, 7 op/s 2023-03-09T19:53:35.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17726.214328 secs 2023-03-09T19:53:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17726.214328 secs 2023-03-09T19:53:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17726.214328 secs 2023-03-09T19:53:36.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:36 smithi122 ceph-mon[93869]: pgmap v9966: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 100 KiB/s wr, 7 op/s 2023-03-09T19:53:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:36 smithi181 ceph-mon[94189]: pgmap v9966: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 100 KiB/s wr, 7 op/s 2023-03-09T19:53:36.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:36 smithi038 ceph-mon[92422]: pgmap v9966: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 100 KiB/s wr, 7 op/s 2023-03-09T19:53:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18190.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:53:38.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17728.528342 secs 2023-03-09T19:53:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:38 smithi122 ceph-mon[93869]: pgmap v9967: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 185 KiB/s wr, 11 op/s 2023-03-09T19:53:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17728.528342 secs 2023-03-09T19:53:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:38 smithi181 ceph-mon[94189]: pgmap v9967: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 185 KiB/s wr, 11 op/s 2023-03-09T19:53:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17728.528342 secs 2023-03-09T19:53:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:38 smithi038 ceph-mon[92422]: pgmap v9967: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 185 KiB/s wr, 11 op/s 2023-03-09T19:53:39.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:39 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:53:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:39 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:53:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:39 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:53:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:39 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:53:39.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:39 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:53:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:39 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:53:40.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:40 smithi122 ceph-mon[93869]: pgmap v9968: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 86 KiB/s wr, 7 op/s 2023-03-09T19:53:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:53:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:53:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17731.214389 secs 2023-03-09T19:53:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:40 smithi181 ceph-mon[94189]: pgmap v9968: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 86 KiB/s wr, 7 op/s 2023-03-09T19:53:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:53:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:53:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17731.214389 secs 2023-03-09T19:53:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:40 smithi038 ceph-mon[92422]: pgmap v9968: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 86 KiB/s wr, 7 op/s 2023-03-09T19:53:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:53:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:53:41.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17731.214389 secs 2023-03-09T19:53:41.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:53:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:53:41.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:53:42.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:42 smithi122 ceph-mon[93869]: pgmap v9969: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 212 KiB/s wr, 11 op/s 2023-03-09T19:53:42.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:53:42.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:53:42.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:53:42.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:53:42.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:53:42.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:53:42.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:53:42.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:53:42.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:53:42.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:42 smithi181 ceph-mon[94189]: pgmap v9969: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 212 KiB/s wr, 11 op/s 2023-03-09T19:53:42.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:53:42.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:53:42.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:53:42.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:53:42.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:53:42.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:53:42.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:53:42.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:53:42.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:53:42.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:42 smithi038 ceph-mon[92422]: pgmap v9969: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 212 KiB/s wr, 11 op/s 2023-03-09T19:53:42.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:53:42.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:53:42.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:53:42.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:53:42.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:53:42.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:53:42.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:53:43.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:53:43.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:53:43.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18195.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:53:43.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17733.528438 secs 2023-03-09T19:53:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17733.528438 secs 2023-03-09T19:53:43.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17733.528438 secs 2023-03-09T19:53:44.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:44 smithi122 ceph-mon[93869]: pgmap v9970: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 212 KiB/s wr, 8 op/s 2023-03-09T19:53:44.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:44 smithi181 ceph-mon[94189]: pgmap v9970: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 212 KiB/s wr, 8 op/s 2023-03-09T19:53:44.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:44 smithi038 ceph-mon[92422]: pgmap v9970: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 212 KiB/s wr, 8 op/s 2023-03-09T19:53:45.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17736.214451 secs 2023-03-09T19:53:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17736.214451 secs 2023-03-09T19:53:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17736.214451 secs 2023-03-09T19:53:46.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:46 smithi122 ceph-mon[93869]: pgmap v9971: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 212 KiB/s wr, 8 op/s 2023-03-09T19:53:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:46 smithi181 ceph-mon[94189]: pgmap v9971: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 212 KiB/s wr, 8 op/s 2023-03-09T19:53:46.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:46 smithi038 ceph-mon[92422]: pgmap v9971: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 212 KiB/s wr, 8 op/s 2023-03-09T19:53:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18200.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:53:48.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17738.528555 secs 2023-03-09T19:53:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:48 smithi122 ceph-mon[93869]: pgmap v9972: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 362 KiB/s rd, 343 KiB/s wr, 12 op/s 2023-03-09T19:53:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17738.528555 secs 2023-03-09T19:53:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:48 smithi181 ceph-mon[94189]: pgmap v9972: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 362 KiB/s rd, 343 KiB/s wr, 12 op/s 2023-03-09T19:53:48.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17738.528555 secs 2023-03-09T19:53:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:48 smithi038 ceph-mon[92422]: pgmap v9972: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 362 KiB/s rd, 343 KiB/s wr, 12 op/s 2023-03-09T19:53:49.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:49 smithi038 ceph-mon[92422]: pgmap v9973: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 258 KiB/s wr, 9 op/s 2023-03-09T19:53:50.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:49 smithi122 ceph-mon[93869]: pgmap v9973: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 258 KiB/s wr, 9 op/s 2023-03-09T19:53:50.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:49 smithi181 ceph-mon[94189]: pgmap v9973: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 258 KiB/s wr, 9 op/s 2023-03-09T19:53:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17741.214510 secs 2023-03-09T19:53:51.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17741.214510 secs 2023-03-09T19:53:51.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17741.214510 secs 2023-03-09T19:53:52.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:53:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:53:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:51 smithi122 ceph-mon[93869]: pgmap v9974: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 503 KiB/s wr, 13 op/s 2023-03-09T19:53:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:53:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:53:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:51 smithi181 ceph-mon[94189]: pgmap v9974: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 503 KiB/s wr, 13 op/s 2023-03-09T19:53:52.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:53:52.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:53:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:51 smithi038 ceph-mon[92422]: pgmap v9974: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 503 KiB/s wr, 13 op/s 2023-03-09T19:53:53.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:53:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:53:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:52 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:53:53.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:53:53.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:53:53.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:52 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:53:53.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:53:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:53:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:52 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:53:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18205.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:53:54.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:53 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:53:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:53:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:53:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:53 smithi122 ceph-mon[93869]: pgmap v9975: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 378 KiB/s wr, 10 op/s 2023-03-09T19:53:54.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:53 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:53:54.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:53:54.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:53:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:53 smithi181 ceph-mon[94189]: pgmap v9975: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 378 KiB/s wr, 10 op/s 2023-03-09T19:53:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:53 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:53:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:53:54.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:53:54.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:53 smithi038 ceph-mon[92422]: pgmap v9975: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 378 KiB/s wr, 10 op/s 2023-03-09T19:53:55.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17746.214570 secs 2023-03-09T19:53:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17746.214570 secs 2023-03-09T19:53:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17746.214570 secs 2023-03-09T19:53:56.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:56 smithi122 ceph-mon[93869]: pgmap v9976: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 377 KiB/s wr, 9 op/s 2023-03-09T19:53:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:56 smithi181 ceph-mon[94189]: pgmap v9976: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 377 KiB/s wr, 9 op/s 2023-03-09T19:53:56.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:56 smithi038 ceph-mon[92422]: pgmap v9976: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 377 KiB/s wr, 9 op/s 2023-03-09T19:53:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18210.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:53:58.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17748.528732 secs 2023-03-09T19:53:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:58 smithi122 ceph-mon[93869]: pgmap v9977: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 527 KiB/s rd, 517 KiB/s wr, 13 op/s 2023-03-09T19:53:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17748.528732 secs 2023-03-09T19:53:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:58 smithi181 ceph-mon[94189]: pgmap v9977: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 527 KiB/s rd, 517 KiB/s wr, 13 op/s 2023-03-09T19:53:58.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17748.528732 secs 2023-03-09T19:53:58.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:58 smithi038 ceph-mon[92422]: pgmap v9977: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 527 KiB/s rd, 517 KiB/s wr, 13 op/s 2023-03-09T19:53:59.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:53:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:53:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:53:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:53:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:53:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:53:59.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:53:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:53:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:54:00.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:00 smithi122 ceph-mon[93869]: pgmap v9978: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 473 KiB/s wr, 10 op/s 2023-03-09T19:54:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:54:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:54:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17751.214629 secs 2023-03-09T19:54:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:00 smithi181 ceph-mon[94189]: pgmap v9978: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 473 KiB/s wr, 10 op/s 2023-03-09T19:54:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:54:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:54:00.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17751.214629 secs 2023-03-09T19:54:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:00 smithi038 ceph-mon[92422]: pgmap v9978: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 473 KiB/s wr, 10 op/s 2023-03-09T19:54:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:54:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:54:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17751.214629 secs 2023-03-09T19:54:01.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:01 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:54:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:54:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:54:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:01 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:54:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:54:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:54:01.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:01 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:54:01.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:54:01.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:54:02.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:02 smithi122 ceph-mon[93869]: pgmap v9979: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 577 KiB/s rd, 473 KiB/s wr, 12 op/s 2023-03-09T19:54:02.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:02 smithi181 ceph-mon[94189]: pgmap v9979: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 577 KiB/s rd, 473 KiB/s wr, 12 op/s 2023-03-09T19:54:02.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:02 smithi038 ceph-mon[92422]: pgmap v9979: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 577 KiB/s rd, 473 KiB/s wr, 12 op/s 2023-03-09T19:54:03.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18215.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:54:03.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17753.528863 secs 2023-03-09T19:54:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17753.528863 secs 2023-03-09T19:54:03.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17753.528863 secs 2023-03-09T19:54:04.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:04 smithi122 ceph-mon[93869]: pgmap v9980: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 249 KiB/s wr, 20 op/s 2023-03-09T19:54:04.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:04 smithi181 ceph-mon[94189]: pgmap v9980: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 249 KiB/s wr, 20 op/s 2023-03-09T19:54:04.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:04 smithi038 ceph-mon[92422]: pgmap v9980: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 249 KiB/s wr, 20 op/s 2023-03-09T19:54:05.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17756.214688 secs 2023-03-09T19:54:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17756.214688 secs 2023-03-09T19:54:05.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17756.214688 secs 2023-03-09T19:54:06.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:06 smithi122 ceph-mon[93869]: pgmap v9981: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 250 KiB/s wr, 20 op/s 2023-03-09T19:54:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:06 smithi181 ceph-mon[94189]: pgmap v9981: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 250 KiB/s wr, 20 op/s 2023-03-09T19:54:06.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:06 smithi038 ceph-mon[92422]: pgmap v9981: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 250 KiB/s wr, 20 op/s 2023-03-09T19:54:07.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:07 smithi122 ceph-mon[93869]: pgmap v9982: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 336 KiB/s rd, 273 KiB/s wr, 25 op/s 2023-03-09T19:54:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:07 smithi181 ceph-mon[94189]: pgmap v9982: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 336 KiB/s rd, 273 KiB/s wr, 25 op/s 2023-03-09T19:54:07.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:07 smithi038 ceph-mon[92422]: pgmap v9982: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 336 KiB/s rd, 273 KiB/s wr, 25 op/s 2023-03-09T19:54:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18220.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:54:08.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17758.528979 secs 2023-03-09T19:54:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17758.528979 secs 2023-03-09T19:54:08.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17758.528979 secs 2023-03-09T19:54:09.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:09 smithi122 ceph-mon[93869]: pgmap v9983: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 193 KiB/s wr, 22 op/s 2023-03-09T19:54:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:09 smithi181 ceph-mon[94189]: pgmap v9983: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 193 KiB/s wr, 22 op/s 2023-03-09T19:54:09.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:09 smithi038 ceph-mon[92422]: pgmap v9983: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 193 KiB/s wr, 22 op/s 2023-03-09T19:54:10.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:10 smithi122 ceph-mon[93869]: osdmap e837: 12 total, 12 up, 12 in 2023-03-09T19:54:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17761.214751 secs 2023-03-09T19:54:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:10 smithi181 ceph-mon[94189]: osdmap e837: 12 total, 12 up, 12 in 2023-03-09T19:54:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17761.214751 secs 2023-03-09T19:54:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:10 smithi038 ceph-mon[92422]: osdmap e837: 12 total, 12 up, 12 in 2023-03-09T19:54:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17761.214751 secs 2023-03-09T19:54:11.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:11 smithi122 ceph-mon[93869]: pgmap v9985: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 144 KiB/s wr, 26 op/s 2023-03-09T19:54:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:11 smithi181 ceph-mon[94189]: pgmap v9985: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 144 KiB/s wr, 26 op/s 2023-03-09T19:54:11.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:11 smithi038 ceph-mon[92422]: pgmap v9985: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 144 KiB/s wr, 26 op/s 2023-03-09T19:54:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17763.529100 secs 2023-03-09T19:54:13.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17763.529100 secs 2023-03-09T19:54:13.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17763.529100 secs 2023-03-09T19:54:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18225.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:54:14.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:14 smithi122 ceph-mon[93869]: pgmap v9986: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 119 KiB/s wr, 11 op/s 2023-03-09T19:54:14.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:14 smithi181 ceph-mon[94189]: pgmap v9986: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 119 KiB/s wr, 11 op/s 2023-03-09T19:54:14.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:14 smithi038 ceph-mon[92422]: pgmap v9986: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 119 KiB/s wr, 11 op/s 2023-03-09T19:54:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17766.214812 secs 2023-03-09T19:54:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17766.214812 secs 2023-03-09T19:54:15.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17766.214812 secs 2023-03-09T19:54:16.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:16 smithi122 ceph-mon[93869]: pgmap v9987: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 118 KiB/s wr, 11 op/s 2023-03-09T19:54:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:16 smithi181 ceph-mon[94189]: pgmap v9987: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 118 KiB/s wr, 11 op/s 2023-03-09T19:54:16.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:16 smithi038 ceph-mon[92422]: pgmap v9987: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 118 KiB/s wr, 11 op/s 2023-03-09T19:54:18.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18230.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:54:18.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17768.529204 secs 2023-03-09T19:54:18.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:18 smithi122 ceph-mon[93869]: pgmap v9988: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 108 KiB/s wr, 9 op/s 2023-03-09T19:54:18.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:18 smithi122 ceph-mon[93869]: osdmap e838: 12 total, 12 up, 12 in 2023-03-09T19:54:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17768.529204 secs 2023-03-09T19:54:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:18 smithi181 ceph-mon[94189]: pgmap v9988: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 108 KiB/s wr, 9 op/s 2023-03-09T19:54:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:18 smithi181 ceph-mon[94189]: osdmap e838: 12 total, 12 up, 12 in 2023-03-09T19:54:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17768.529204 secs 2023-03-09T19:54:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:18 smithi038 ceph-mon[92422]: pgmap v9988: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 108 KiB/s wr, 9 op/s 2023-03-09T19:54:19.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:18 smithi038 ceph-mon[92422]: osdmap e838: 12 total, 12 up, 12 in 2023-03-09T19:54:20.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:20 smithi122 ceph-mon[93869]: pgmap v9990: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 166 KiB/s wr, 9 op/s 2023-03-09T19:54:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17771.214887 secs 2023-03-09T19:54:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:20 smithi181 ceph-mon[94189]: pgmap v9990: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 166 KiB/s wr, 9 op/s 2023-03-09T19:54:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17771.214887 secs 2023-03-09T19:54:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:20 smithi038 ceph-mon[92422]: pgmap v9990: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 166 KiB/s wr, 9 op/s 2023-03-09T19:54:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17771.214887 secs 2023-03-09T19:54:22.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:22 smithi122 ceph-mon[93869]: pgmap v9991: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 148 KiB/s wr, 8 op/s 2023-03-09T19:54:22.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:22 smithi181 ceph-mon[94189]: pgmap v9991: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 148 KiB/s wr, 8 op/s 2023-03-09T19:54:22.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:22 smithi038 ceph-mon[92422]: pgmap v9991: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 148 KiB/s wr, 8 op/s 2023-03-09T19:54:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18235.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:54:23.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17773.529342 secs 2023-03-09T19:54:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17773.529342 secs 2023-03-09T19:54:23.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17773.529342 secs 2023-03-09T19:54:24.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:24 smithi122 ceph-mon[93869]: pgmap v9992: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 147 KiB/s wr, 8 op/s 2023-03-09T19:54:24.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:24 smithi181 ceph-mon[94189]: pgmap v9992: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 147 KiB/s wr, 8 op/s 2023-03-09T19:54:24.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:24 smithi038 ceph-mon[92422]: pgmap v9992: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 147 KiB/s wr, 8 op/s 2023-03-09T19:54:25.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17776.214947 secs 2023-03-09T19:54:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17776.214947 secs 2023-03-09T19:54:25.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17776.214947 secs 2023-03-09T19:54:26.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:26 smithi122 ceph-mon[93869]: pgmap v9993: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 147 KiB/s wr, 9 op/s 2023-03-09T19:54:26.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:26 smithi181 ceph-mon[94189]: pgmap v9993: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 147 KiB/s wr, 9 op/s 2023-03-09T19:54:26.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:26 smithi038 ceph-mon[92422]: pgmap v9993: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 147 KiB/s wr, 9 op/s 2023-03-09T19:54:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18240.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:54:28.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17778.529461 secs 2023-03-09T19:54:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:28 smithi122 ceph-mon[93869]: pgmap v9994: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 130 KiB/s wr, 8 op/s 2023-03-09T19:54:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17778.529461 secs 2023-03-09T19:54:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:28 smithi181 ceph-mon[94189]: pgmap v9994: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 130 KiB/s wr, 8 op/s 2023-03-09T19:54:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17778.529461 secs 2023-03-09T19:54:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:28 smithi038 ceph-mon[92422]: pgmap v9994: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 130 KiB/s wr, 8 op/s 2023-03-09T19:54:30.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:30 smithi122 ceph-mon[93869]: pgmap v9995: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 184 KiB/s wr, 8 op/s 2023-03-09T19:54:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17781.214985 secs 2023-03-09T19:54:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:30 smithi181 ceph-mon[94189]: pgmap v9995: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 184 KiB/s wr, 8 op/s 2023-03-09T19:54:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17781.214985 secs 2023-03-09T19:54:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:30 smithi038 ceph-mon[92422]: pgmap v9995: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 184 KiB/s wr, 8 op/s 2023-03-09T19:54:30.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17781.214985 secs 2023-03-09T19:54:32.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:32 smithi122 ceph-mon[93869]: pgmap v9996: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 65 KiB/s wr, 8 op/s 2023-03-09T19:54:32.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:32 smithi181 ceph-mon[94189]: pgmap v9996: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 65 KiB/s wr, 8 op/s 2023-03-09T19:54:32.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:32 smithi038 ceph-mon[92422]: pgmap v9996: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 65 KiB/s wr, 8 op/s 2023-03-09T19:54:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18245.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:54:33.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17783.529529 secs 2023-03-09T19:54:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17783.529529 secs 2023-03-09T19:54:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17783.529529 secs 2023-03-09T19:54:34.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:34 smithi122 ceph-mon[93869]: pgmap v9997: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 103 KiB/s wr, 6 op/s 2023-03-09T19:54:34.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:34 smithi181 ceph-mon[94189]: pgmap v9997: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 103 KiB/s wr, 6 op/s 2023-03-09T19:54:34.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:34 smithi038 ceph-mon[92422]: pgmap v9997: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 103 KiB/s wr, 6 op/s 2023-03-09T19:54:35.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17786.215045 secs 2023-03-09T19:54:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:35 smithi122 ceph-mon[93869]: pgmap v9998: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 125 KiB/s wr, 6 op/s 2023-03-09T19:54:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17786.215045 secs 2023-03-09T19:54:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:35 smithi181 ceph-mon[94189]: pgmap v9998: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 125 KiB/s wr, 6 op/s 2023-03-09T19:54:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17786.215045 secs 2023-03-09T19:54:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:35 smithi038 ceph-mon[92422]: pgmap v9998: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 125 KiB/s wr, 6 op/s 2023-03-09T19:54:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18250.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:54:38.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:38 smithi122 ceph-mon[93869]: pgmap v9999: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 125 KiB/s wr, 9 op/s 2023-03-09T19:54:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:38 smithi181 ceph-mon[94189]: pgmap v9999: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 125 KiB/s wr, 9 op/s 2023-03-09T19:54:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:38 smithi038 ceph-mon[92422]: pgmap v9999: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 125 KiB/s wr, 9 op/s 2023-03-09T19:54:40.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:40 smithi122 ceph-mon[93869]: pgmap v10000: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 124 KiB/s wr, 7 op/s 2023-03-09T19:54:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:54:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:54:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17791.215105 secs 2023-03-09T19:54:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:40 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:54:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:40 smithi181 ceph-mon[94189]: pgmap v10000: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 124 KiB/s wr, 7 op/s 2023-03-09T19:54:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:54:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:54:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17791.215105 secs 2023-03-09T19:54:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:40 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:54:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:40 smithi038 ceph-mon[92422]: pgmap v10000: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 124 KiB/s wr, 7 op/s 2023-03-09T19:54:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:54:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:54:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17791.215105 secs 2023-03-09T19:54:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:40 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:54:41.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:41 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:54:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:54:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:54:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:41 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:54:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:54:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:54:41.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:41 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:54:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:54:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:54:42.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:42 smithi122 ceph-mon[93869]: pgmap v10001: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 61 KiB/s wr, 8 op/s 2023-03-09T19:54:42.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:54:42.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:42 smithi181 ceph-mon[94189]: pgmap v10001: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 61 KiB/s wr, 8 op/s 2023-03-09T19:54:42.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:54:42.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:42 smithi038 ceph-mon[92422]: pgmap v10001: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 61 KiB/s wr, 8 op/s 2023-03-09T19:54:42.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:54:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18255.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:54:44.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:44 smithi122 ceph-mon[93869]: pgmap v10002: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 129 KiB/s wr, 7 op/s 2023-03-09T19:54:44.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:44 smithi181 ceph-mon[94189]: pgmap v10002: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 129 KiB/s wr, 7 op/s 2023-03-09T19:54:44.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:44 smithi038 ceph-mon[92422]: pgmap v10002: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 129 KiB/s wr, 7 op/s 2023-03-09T19:54:46.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:54:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17796.215165 secs 2023-03-09T19:54:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:54:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:54:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:54:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:54:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:54:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:54:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:54:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:54:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:54:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:54:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:54:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:54:46.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:54:46.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:45 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi038 to 124.3M 2023-03-09T19:54:46.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:45 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T19:54:46.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:45 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi038 to 130372976: error parsing value: Value '130372976' is below minimum 939524096 2023-03-09T19:54:46.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:45 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T19:54:46.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:45 smithi122 ceph-mon[93869]: pgmap v10003: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 92 KiB/s wr, 8 op/s 2023-03-09T19:54:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:54:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17796.215165 secs 2023-03-09T19:54:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:54:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:54:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:54:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:54:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:54:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:54:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:54:46.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:54:46.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:54:46.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:54:46.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:54:46.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:54:46.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:54:46.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:45 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi038 to 124.3M 2023-03-09T19:54:46.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:45 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T19:54:46.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:45 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi038 to 130372976: error parsing value: Value '130372976' is below minimum 939524096 2023-03-09T19:54:46.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:45 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T19:54:46.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:45 smithi181 ceph-mon[94189]: pgmap v10003: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 92 KiB/s wr, 8 op/s 2023-03-09T19:54:46.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:54:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17796.215165 secs 2023-03-09T19:54:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:54:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:54:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:54:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:54:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:54:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:54:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:54:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:54:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:54:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:54:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:54:46.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:54:46.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:54:46.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:45 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi038 to 124.3M 2023-03-09T19:54:46.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:45 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T19:54:46.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:45 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi038 to 130372976: error parsing value: Value '130372976' is below minimum 939524096 2023-03-09T19:54:46.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:45 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T19:54:46.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:45 smithi038 ceph-mon[92422]: pgmap v10003: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 92 KiB/s wr, 8 op/s 2023-03-09T19:54:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:54:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:54:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:54:47.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:54:47.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:46 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T19:54:47.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:46 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T19:54:47.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:54:47.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:54:47.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:54:47.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:54:47.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:54:47.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:54:47.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:54:47.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:46 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T19:54:47.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:46 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T19:54:47.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:54:47.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:54:47.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:54:47.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:54:47.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:54:47.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:54:47.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T19:54:47.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:46 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T19:54:47.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:46 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T19:54:47.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:54:47.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:54:47.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:54:48.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:54:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:54:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:48 smithi122 ceph-mon[93869]: pgmap v10004: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 257 KiB/s wr, 11 op/s 2023-03-09T19:54:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:54:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:54:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:48 smithi181 ceph-mon[94189]: pgmap v10004: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 257 KiB/s wr, 11 op/s 2023-03-09T19:54:48.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:54:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:54:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:48 smithi038 ceph-mon[92422]: pgmap v10004: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 257 KiB/s wr, 11 op/s 2023-03-09T19:54:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18260.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:54:50.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:50 smithi122 ceph-mon[93869]: pgmap v10005: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 256 KiB/s wr, 9 op/s 2023-03-09T19:54:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17801.215226 secs 2023-03-09T19:54:50.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:50 smithi181 ceph-mon[94189]: pgmap v10005: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 256 KiB/s wr, 9 op/s 2023-03-09T19:54:50.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17801.215226 secs 2023-03-09T19:54:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:50 smithi038 ceph-mon[92422]: pgmap v10005: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 256 KiB/s wr, 9 op/s 2023-03-09T19:54:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17801.215226 secs 2023-03-09T19:54:52.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:52 smithi122 ceph-mon[93869]: pgmap v10006: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 257 KiB/s wr, 11 op/s 2023-03-09T19:54:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:52 smithi181 ceph-mon[94189]: pgmap v10006: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 257 KiB/s wr, 11 op/s 2023-03-09T19:54:52.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:52 smithi038 ceph-mon[92422]: pgmap v10006: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 257 KiB/s wr, 11 op/s 2023-03-09T19:54:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18265.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:54:54.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:54 smithi122 ceph-mon[93869]: pgmap v10007: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 321 KiB/s wr, 10 op/s 2023-03-09T19:54:54.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:54 smithi181 ceph-mon[94189]: pgmap v10007: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 321 KiB/s wr, 10 op/s 2023-03-09T19:54:54.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:54 smithi038 ceph-mon[92422]: pgmap v10007: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 321 KiB/s wr, 10 op/s 2023-03-09T19:54:55.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:54:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:54:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:54:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:54:55.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:54:55.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:54:56.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17806.215286 secs 2023-03-09T19:54:56.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:56 smithi122 ceph-mon[93869]: pgmap v10008: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 427 KiB/s wr, 10 op/s 2023-03-09T19:54:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17806.215286 secs 2023-03-09T19:54:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:56 smithi181 ceph-mon[94189]: pgmap v10008: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 427 KiB/s wr, 10 op/s 2023-03-09T19:54:56.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17806.215286 secs 2023-03-09T19:54:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:56 smithi038 ceph-mon[92422]: pgmap v10008: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 427 KiB/s wr, 10 op/s 2023-03-09T19:54:57.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:54:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:54:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:54:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:54:57.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:54:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:54:58.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:54:58 smithi122 ceph-mon[93869]: pgmap v10009: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 557 KiB/s rd, 468 KiB/s wr, 13 op/s 2023-03-09T19:54:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:54:58 smithi181 ceph-mon[94189]: pgmap v10009: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 557 KiB/s rd, 468 KiB/s wr, 13 op/s 2023-03-09T19:54:58.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:54:58 smithi038 ceph-mon[92422]: pgmap v10009: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 557 KiB/s rd, 468 KiB/s wr, 13 op/s 2023-03-09T19:54:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18270.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:55:00.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:00 smithi122 ceph-mon[93869]: pgmap v10010: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 494 KiB/s rd, 408 KiB/s wr, 10 op/s 2023-03-09T19:55:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17811.215348 secs 2023-03-09T19:55:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:00 smithi181 ceph-mon[94189]: pgmap v10010: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 494 KiB/s rd, 408 KiB/s wr, 10 op/s 2023-03-09T19:55:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17811.215348 secs 2023-03-09T19:55:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:00 smithi038 ceph-mon[92422]: pgmap v10010: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 494 KiB/s rd, 408 KiB/s wr, 10 op/s 2023-03-09T19:55:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17811.215348 secs 2023-03-09T19:55:02.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:02 smithi122 ceph-mon[93869]: pgmap v10011: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 422 KiB/s wr, 13 op/s 2023-03-09T19:55:02.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:02 smithi181 ceph-mon[94189]: pgmap v10011: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 422 KiB/s wr, 13 op/s 2023-03-09T19:55:02.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:02 smithi038 ceph-mon[92422]: pgmap v10011: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 422 KiB/s wr, 13 op/s 2023-03-09T19:55:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18275.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:55:03.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17813.530097 secs 2023-03-09T19:55:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17813.530097 secs 2023-03-09T19:55:03.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17813.530097 secs 2023-03-09T19:55:04.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:04 smithi122 ceph-mon[93869]: pgmap v10012: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 550 KiB/s rd, 489 KiB/s wr, 23 op/s 2023-03-09T19:55:04.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:04 smithi181 ceph-mon[94189]: pgmap v10012: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 550 KiB/s rd, 489 KiB/s wr, 23 op/s 2023-03-09T19:55:04.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:04 smithi038 ceph-mon[92422]: pgmap v10012: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 550 KiB/s rd, 489 KiB/s wr, 23 op/s 2023-03-09T19:55:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:05 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:55:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:05 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:55:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:05 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:55:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:05 smithi122 ceph-mon[93869]: osdmap e839: 12 total, 12 up, 12 in 2023-03-09T19:55:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17816.216097 secs 2023-03-09T19:55:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:05 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:55:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:05 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:55:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:05 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:55:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:05 smithi181 ceph-mon[94189]: osdmap e839: 12 total, 12 up, 12 in 2023-03-09T19:55:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17816.216097 secs 2023-03-09T19:55:05.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:05 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:55:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:05 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:55:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:05 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:55:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:05 smithi038 ceph-mon[92422]: osdmap e839: 12 total, 12 up, 12 in 2023-03-09T19:55:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17816.216097 secs 2023-03-09T19:55:06.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:06 smithi122 ceph-mon[93869]: pgmap v10014: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 349 KiB/s wr, 26 op/s 2023-03-09T19:55:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:06 smithi181 ceph-mon[94189]: pgmap v10014: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 349 KiB/s wr, 26 op/s 2023-03-09T19:55:06.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:06 smithi038 ceph-mon[92422]: pgmap v10014: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 349 KiB/s wr, 26 op/s 2023-03-09T19:55:07.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:55:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:55:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:07 smithi122 ceph-mon[93869]: pgmap v10015: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 327 KiB/s wr, 27 op/s 2023-03-09T19:55:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:55:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:55:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:07 smithi181 ceph-mon[94189]: pgmap v10015: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 327 KiB/s wr, 27 op/s 2023-03-09T19:55:07.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:55:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:55:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:07 smithi038 ceph-mon[92422]: pgmap v10015: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 327 KiB/s wr, 27 op/s 2023-03-09T19:55:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18280.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:55:08.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17818.530221 secs 2023-03-09T19:55:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:55:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:55:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17818.530221 secs 2023-03-09T19:55:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:55:08.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:55:08.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17818.530221 secs 2023-03-09T19:55:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:55:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:55:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:09 smithi181 ceph-mon[94189]: pgmap v10016: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 237 KiB/s wr, 28 op/s 2023-03-09T19:55:09.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:09 smithi038 ceph-mon[92422]: pgmap v10016: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 237 KiB/s wr, 28 op/s 2023-03-09T19:55:10.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:09 smithi122 ceph-mon[93869]: pgmap v10016: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 237 KiB/s wr, 28 op/s 2023-03-09T19:55:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17821.216172 secs 2023-03-09T19:55:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17821.216172 secs 2023-03-09T19:55:11.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17821.216172 secs 2023-03-09T19:55:11.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:11 smithi038 ceph-mon[92422]: pgmap v10017: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 237 KiB/s wr, 27 op/s 2023-03-09T19:55:12.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:11 smithi122 ceph-mon[93869]: pgmap v10017: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 237 KiB/s wr, 27 op/s 2023-03-09T19:55:12.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:11 smithi181 ceph-mon[94189]: pgmap v10017: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 237 KiB/s wr, 27 op/s 2023-03-09T19:55:13.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17823.530331 secs 2023-03-09T19:55:13.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17823.530331 secs 2023-03-09T19:55:13.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17823.530331 secs 2023-03-09T19:55:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18285.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:55:14.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:14 smithi122 ceph-mon[93869]: osdmap e840: 12 total, 12 up, 12 in 2023-03-09T19:55:14.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:14 smithi122 ceph-mon[93869]: pgmap v10019: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 120 KiB/s wr, 12 op/s 2023-03-09T19:55:14.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:14 smithi181 ceph-mon[94189]: osdmap e840: 12 total, 12 up, 12 in 2023-03-09T19:55:14.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:14 smithi181 ceph-mon[94189]: pgmap v10019: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 120 KiB/s wr, 12 op/s 2023-03-09T19:55:14.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:14 smithi038 ceph-mon[92422]: osdmap e840: 12 total, 12 up, 12 in 2023-03-09T19:55:14.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:14 smithi038 ceph-mon[92422]: pgmap v10019: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 120 KiB/s wr, 12 op/s 2023-03-09T19:55:15.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17826.216230 secs 2023-03-09T19:55:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17826.216230 secs 2023-03-09T19:55:15.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17826.216230 secs 2023-03-09T19:55:16.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:16 smithi122 ceph-mon[93869]: pgmap v10020: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 108 KiB/s wr, 12 op/s 2023-03-09T19:55:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:16 smithi181 ceph-mon[94189]: pgmap v10020: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 108 KiB/s wr, 12 op/s 2023-03-09T19:55:16.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:16 smithi038 ceph-mon[92422]: pgmap v10020: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 108 KiB/s wr, 12 op/s 2023-03-09T19:55:18.793 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18290.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:55:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17828.530455 secs 2023-03-09T19:55:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:18 smithi122 ceph-mon[93869]: pgmap v10021: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 81 KiB/s wr, 9 op/s 2023-03-09T19:55:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17828.530455 secs 2023-03-09T19:55:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:18 smithi181 ceph-mon[94189]: pgmap v10021: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 81 KiB/s wr, 9 op/s 2023-03-09T19:55:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17828.530455 secs 2023-03-09T19:55:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:18 smithi038 ceph-mon[92422]: pgmap v10021: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 81 KiB/s wr, 9 op/s 2023-03-09T19:55:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:20 smithi122 ceph-mon[93869]: pgmap v10022: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 91 KiB/s wr, 8 op/s 2023-03-09T19:55:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17831.216291 secs 2023-03-09T19:55:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:20 smithi181 ceph-mon[94189]: pgmap v10022: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 91 KiB/s wr, 8 op/s 2023-03-09T19:55:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17831.216291 secs 2023-03-09T19:55:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:20 smithi038 ceph-mon[92422]: pgmap v10022: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 91 KiB/s wr, 8 op/s 2023-03-09T19:55:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17831.216291 secs 2023-03-09T19:55:22.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:22 smithi122 ceph-mon[93869]: pgmap v10023: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 91 KiB/s wr, 8 op/s 2023-03-09T19:55:22.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:22 smithi181 ceph-mon[94189]: pgmap v10023: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 91 KiB/s wr, 8 op/s 2023-03-09T19:55:22.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:22 smithi038 ceph-mon[92422]: pgmap v10023: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 91 KiB/s wr, 8 op/s 2023-03-09T19:55:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18295.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:55:23.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17833.530572 secs 2023-03-09T19:55:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17833.530572 secs 2023-03-09T19:55:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17833.530572 secs 2023-03-09T19:55:24.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:24 smithi122 ceph-mon[93869]: pgmap v10024: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 164 KiB/s wr, 8 op/s 2023-03-09T19:55:24.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:24 smithi181 ceph-mon[94189]: pgmap v10024: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 164 KiB/s wr, 8 op/s 2023-03-09T19:55:24.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:24 smithi038 ceph-mon[92422]: pgmap v10024: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 164 KiB/s wr, 8 op/s 2023-03-09T19:55:25.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17836.216350 secs 2023-03-09T19:55:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17836.216350 secs 2023-03-09T19:55:25.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17836.216350 secs 2023-03-09T19:55:26.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:26 smithi122 ceph-mon[93869]: pgmap v10025: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 140 KiB/s wr, 8 op/s 2023-03-09T19:55:26.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:26 smithi181 ceph-mon[94189]: pgmap v10025: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 140 KiB/s wr, 8 op/s 2023-03-09T19:55:26.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:26 smithi038 ceph-mon[92422]: pgmap v10025: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 140 KiB/s wr, 8 op/s 2023-03-09T19:55:27.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:27 smithi038 ceph-mon[92422]: pgmap v10026: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 140 KiB/s wr, 9 op/s 2023-03-09T19:55:28.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:27 smithi122 ceph-mon[93869]: pgmap v10026: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 140 KiB/s wr, 9 op/s 2023-03-09T19:55:28.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:27 smithi181 ceph-mon[94189]: pgmap v10026: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 140 KiB/s wr, 9 op/s 2023-03-09T19:55:28.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18300.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:55:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17838.530692 secs 2023-03-09T19:55:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17838.530692 secs 2023-03-09T19:55:29.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17838.530692 secs 2023-03-09T19:55:29.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:29 smithi038 ceph-mon[92422]: pgmap v10027: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 174 KiB/s wr, 8 op/s 2023-03-09T19:55:30.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:29 smithi122 ceph-mon[93869]: pgmap v10027: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 174 KiB/s wr, 8 op/s 2023-03-09T19:55:30.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:29 smithi181 ceph-mon[94189]: pgmap v10027: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 174 KiB/s wr, 8 op/s 2023-03-09T19:55:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17841.216410 secs 2023-03-09T19:55:31.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17841.216410 secs 2023-03-09T19:55:31.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17841.216410 secs 2023-03-09T19:55:32.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:31 smithi122 ceph-mon[93869]: pgmap v10028: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 114 KiB/s wr, 9 op/s 2023-03-09T19:55:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:31 smithi181 ceph-mon[94189]: pgmap v10028: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 114 KiB/s wr, 9 op/s 2023-03-09T19:55:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:31 smithi038 ceph-mon[92422]: pgmap v10028: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 114 KiB/s wr, 9 op/s 2023-03-09T19:55:33.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17843.530816 secs 2023-03-09T19:55:33.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17843.530816 secs 2023-03-09T19:55:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17843.530816 secs 2023-03-09T19:55:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18305.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:55:34.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:34 smithi122 ceph-mon[93869]: pgmap v10029: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 164 KiB/s wr, 7 op/s 2023-03-09T19:55:34.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:34 smithi181 ceph-mon[94189]: pgmap v10029: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 164 KiB/s wr, 7 op/s 2023-03-09T19:55:34.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:34 smithi038 ceph-mon[92422]: pgmap v10029: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 164 KiB/s wr, 7 op/s 2023-03-09T19:55:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17846.216469 secs 2023-03-09T19:55:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17846.216469 secs 2023-03-09T19:55:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17846.216469 secs 2023-03-09T19:55:36.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:36 smithi122 ceph-mon[93869]: pgmap v10030: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 100 KiB/s wr, 7 op/s 2023-03-09T19:55:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:36 smithi181 ceph-mon[94189]: pgmap v10030: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 100 KiB/s wr, 7 op/s 2023-03-09T19:55:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:36 smithi038 ceph-mon[92422]: pgmap v10030: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 100 KiB/s wr, 7 op/s 2023-03-09T19:55:38.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18310.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:55:38.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:38 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 17848.530954 secs 2023-03-09T19:55:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:38 smithi122 ceph-mon[93869]: pgmap v10031: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 100 KiB/s wr, 7 op/s 2023-03-09T19:55:38.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:38 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 17848.530954 secs 2023-03-09T19:55:38.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:38 smithi181 ceph-mon[94189]: pgmap v10031: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 100 KiB/s wr, 7 op/s 2023-03-09T19:55:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:38 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 17848.530954 secs 2023-03-09T19:55:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:38 smithi038 ceph-mon[92422]: pgmap v10031: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 100 KiB/s wr, 7 op/s 2023-03-09T19:55:40.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:40 smithi122 ceph-mon[93869]: pgmap v10032: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 120 KiB/s wr, 7 op/s 2023-03-09T19:55:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:55:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:55:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17851.216531 secs 2023-03-09T19:55:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:40 smithi181 ceph-mon[94189]: pgmap v10032: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 120 KiB/s wr, 7 op/s 2023-03-09T19:55:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:55:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:55:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17851.216531 secs 2023-03-09T19:55:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:40 smithi038 ceph-mon[92422]: pgmap v10032: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 120 KiB/s wr, 7 op/s 2023-03-09T19:55:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:55:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:55:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17851.216531 secs 2023-03-09T19:55:41.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:55:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:55:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:41 smithi122 ceph-mon[93869]: pgmap v10033: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 134 KiB/s wr, 8 op/s 2023-03-09T19:55:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:55:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:55:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:41 smithi181 ceph-mon[94189]: pgmap v10033: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 134 KiB/s wr, 8 op/s 2023-03-09T19:55:41.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:55:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:55:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:41 smithi038 ceph-mon[92422]: pgmap v10033: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 134 KiB/s wr, 8 op/s 2023-03-09T19:55:43.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:42 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 17853.531028 secs 2023-03-09T19:55:43.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:42 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 17853.531028 secs 2023-03-09T19:55:43.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:42 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 17853.531028 secs 2023-03-09T19:55:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18315.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:55:44.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:44 smithi122 ceph-mon[93869]: pgmap v10034: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 134 KiB/s wr, 7 op/s 2023-03-09T19:55:44.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:44 smithi181 ceph-mon[94189]: pgmap v10034: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 134 KiB/s wr, 7 op/s 2023-03-09T19:55:44.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:44 smithi038 ceph-mon[92422]: pgmap v10034: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 134 KiB/s wr, 7 op/s 2023-03-09T19:55:45.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17856.216594 secs 2023-03-09T19:55:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17856.216594 secs 2023-03-09T19:55:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17856.216594 secs 2023-03-09T19:55:46.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:46 smithi122 ceph-mon[93869]: pgmap v10035: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 190 KiB/s wr, 8 op/s 2023-03-09T19:55:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:55:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:46 smithi181 ceph-mon[94189]: pgmap v10035: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 190 KiB/s wr, 8 op/s 2023-03-09T19:55:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:55:46.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:46 smithi038 ceph-mon[92422]: pgmap v10035: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 190 KiB/s wr, 8 op/s 2023-03-09T19:55:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:55:47.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:55:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:55:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:55:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:55:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:55:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:55:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:55:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:55:47.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:55:47.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:55:47.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:55:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:55:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:55:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:55:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:55:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18320.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:55:48.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17858.531121 secs 2023-03-09T19:55:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:48 smithi122 ceph-mon[93869]: pgmap v10036: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 204 KiB/s wr, 9 op/s 2023-03-09T19:55:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17858.531121 secs 2023-03-09T19:55:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:48 smithi181 ceph-mon[94189]: pgmap v10036: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 204 KiB/s wr, 9 op/s 2023-03-09T19:55:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17858.531121 secs 2023-03-09T19:55:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:48 smithi038 ceph-mon[92422]: pgmap v10036: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 204 KiB/s wr, 9 op/s 2023-03-09T19:55:49.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:55:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:55:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:55:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:55:49.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:55:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:55:50.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:50 smithi122 ceph-mon[93869]: pgmap v10037: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 318 KiB/s rd, 204 KiB/s wr, 8 op/s 2023-03-09T19:55:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17861.216654 secs 2023-03-09T19:55:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:50 smithi181 ceph-mon[94189]: pgmap v10037: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 318 KiB/s rd, 204 KiB/s wr, 8 op/s 2023-03-09T19:55:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17861.216654 secs 2023-03-09T19:55:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:50 smithi038 ceph-mon[92422]: pgmap v10037: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 318 KiB/s rd, 204 KiB/s wr, 8 op/s 2023-03-09T19:55:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17861.216654 secs 2023-03-09T19:55:52.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:52 smithi122 ceph-mon[93869]: pgmap v10038: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 224 KiB/s wr, 11 op/s 2023-03-09T19:55:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:52 smithi181 ceph-mon[94189]: pgmap v10038: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 224 KiB/s wr, 11 op/s 2023-03-09T19:55:52.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:52 smithi038 ceph-mon[92422]: pgmap v10038: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 224 KiB/s wr, 11 op/s 2023-03-09T19:55:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18325.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:55:53.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17863.531234 secs 2023-03-09T19:55:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17863.531234 secs 2023-03-09T19:55:53.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17863.531234 secs 2023-03-09T19:55:54.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:54 smithi122 ceph-mon[93869]: pgmap v10039: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 175 KiB/s wr, 9 op/s 2023-03-09T19:55:54.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:54 smithi181 ceph-mon[94189]: pgmap v10039: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 175 KiB/s wr, 9 op/s 2023-03-09T19:55:54.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:54 smithi038 ceph-mon[92422]: pgmap v10039: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 175 KiB/s wr, 9 op/s 2023-03-09T19:55:55.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17866.216715 secs 2023-03-09T19:55:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17866.216715 secs 2023-03-09T19:55:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17866.216715 secs 2023-03-09T19:55:56.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:56 smithi122 ceph-mon[93869]: pgmap v10040: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 337 KiB/s rd, 322 KiB/s wr, 10 op/s 2023-03-09T19:55:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:55:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:55:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:56 smithi181 ceph-mon[94189]: pgmap v10040: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 337 KiB/s rd, 322 KiB/s wr, 10 op/s 2023-03-09T19:55:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:55:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:55:56.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:56 smithi038 ceph-mon[92422]: pgmap v10040: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 337 KiB/s rd, 322 KiB/s wr, 10 op/s 2023-03-09T19:55:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:55:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:55:58.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:57 smithi122 ceph-mon[93869]: pgmap v10041: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 301 KiB/s wr, 11 op/s 2023-03-09T19:55:58.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:57 smithi181 ceph-mon[94189]: pgmap v10041: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 301 KiB/s wr, 11 op/s 2023-03-09T19:55:58.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:57 smithi038 ceph-mon[92422]: pgmap v10041: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 301 KiB/s wr, 11 op/s 2023-03-09T19:55:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18330.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:55:59.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17868.531323 secs 2023-03-09T19:55:59.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17868.531323 secs 2023-03-09T19:55:59.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17868.531323 secs 2023-03-09T19:56:00.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:56:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:56:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:55:59 smithi122 ceph-mon[93869]: pgmap v10042: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 287 KiB/s wr, 10 op/s 2023-03-09T19:56:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:56:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:56:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:55:59 smithi181 ceph-mon[94189]: pgmap v10042: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 287 KiB/s wr, 10 op/s 2023-03-09T19:56:00.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:56:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:56:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:55:59 smithi038 ceph-mon[92422]: pgmap v10042: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 287 KiB/s wr, 10 op/s 2023-03-09T19:56:01.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17871.216777 secs 2023-03-09T19:56:01.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17871.216777 secs 2023-03-09T19:56:01.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17871.216777 secs 2023-03-09T19:56:02.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:01 smithi122 ceph-mon[93869]: pgmap v10043: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 312 KiB/s wr, 13 op/s 2023-03-09T19:56:02.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:01 smithi181 ceph-mon[94189]: pgmap v10043: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 312 KiB/s wr, 13 op/s 2023-03-09T19:56:02.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:01 smithi038 ceph-mon[92422]: pgmap v10043: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 312 KiB/s wr, 13 op/s 2023-03-09T19:56:03.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17873.531459 secs 2023-03-09T19:56:03.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17873.531459 secs 2023-03-09T19:56:03.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17873.531459 secs 2023-03-09T19:56:03.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18335.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:56:04.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:04 smithi122 ceph-mon[93869]: pgmap v10044: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 284 KiB/s wr, 20 op/s 2023-03-09T19:56:04.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:04 smithi181 ceph-mon[94189]: pgmap v10044: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 284 KiB/s wr, 20 op/s 2023-03-09T19:56:04.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:04 smithi038 ceph-mon[92422]: pgmap v10044: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 284 KiB/s wr, 20 op/s 2023-03-09T19:56:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17876.216837 secs 2023-03-09T19:56:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17876.216837 secs 2023-03-09T19:56:05.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17876.216837 secs 2023-03-09T19:56:06.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:06 smithi122 ceph-mon[93869]: pgmap v10045: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 310 KiB/s wr, 21 op/s 2023-03-09T19:56:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:06 smithi181 ceph-mon[94189]: pgmap v10045: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 310 KiB/s wr, 21 op/s 2023-03-09T19:56:06.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:06 smithi038 ceph-mon[92422]: pgmap v10045: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 310 KiB/s wr, 21 op/s 2023-03-09T19:56:07.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:56:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:56:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:56:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:56:07.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:56:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:56:08.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18340.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:56:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17878.531572 secs 2023-03-09T19:56:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:56:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:56:08.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:08 smithi122 ceph-mon[93869]: pgmap v10046: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 421 KiB/s wr, 23 op/s 2023-03-09T19:56:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17878.531572 secs 2023-03-09T19:56:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:56:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:56:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:08 smithi181 ceph-mon[94189]: pgmap v10046: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 421 KiB/s wr, 23 op/s 2023-03-09T19:56:08.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17878.531572 secs 2023-03-09T19:56:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:56:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:56:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:08 smithi038 ceph-mon[92422]: pgmap v10046: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 421 KiB/s wr, 23 op/s 2023-03-09T19:56:10.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:10 smithi122 ceph-mon[93869]: pgmap v10047: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 460 KiB/s wr, 22 op/s 2023-03-09T19:56:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17881.216897 secs 2023-03-09T19:56:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:10 smithi181 ceph-mon[94189]: pgmap v10047: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 460 KiB/s wr, 22 op/s 2023-03-09T19:56:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17881.216897 secs 2023-03-09T19:56:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:10 smithi038 ceph-mon[92422]: pgmap v10047: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 460 KiB/s wr, 22 op/s 2023-03-09T19:56:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17881.216897 secs 2023-03-09T19:56:12.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:12 smithi122 ceph-mon[93869]: pgmap v10048: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 475 KiB/s wr, 24 op/s 2023-03-09T19:56:12.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:12 smithi181 ceph-mon[94189]: pgmap v10048: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 475 KiB/s wr, 24 op/s 2023-03-09T19:56:12.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:12 smithi038 ceph-mon[92422]: pgmap v10048: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 475 KiB/s wr, 24 op/s 2023-03-09T19:56:13.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18345.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:56:13.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17883.531646 secs 2023-03-09T19:56:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17883.531646 secs 2023-03-09T19:56:13.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17883.531646 secs 2023-03-09T19:56:14.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:14 smithi122 ceph-mon[93869]: pgmap v10049: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 436 KiB/s rd, 449 KiB/s wr, 21 op/s 2023-03-09T19:56:14.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:14 smithi181 ceph-mon[94189]: pgmap v10049: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 436 KiB/s rd, 449 KiB/s wr, 21 op/s 2023-03-09T19:56:14.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:14 smithi038 ceph-mon[92422]: pgmap v10049: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 436 KiB/s rd, 449 KiB/s wr, 21 op/s 2023-03-09T19:56:15.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17886.216957 secs 2023-03-09T19:56:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17886.216957 secs 2023-03-09T19:56:15.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17886.216957 secs 2023-03-09T19:56:16.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:16 smithi122 ceph-mon[93869]: pgmap v10050: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 494 KiB/s rd, 502 KiB/s wr, 12 op/s 2023-03-09T19:56:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:16 smithi181 ceph-mon[94189]: pgmap v10050: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 494 KiB/s rd, 502 KiB/s wr, 12 op/s 2023-03-09T19:56:16.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:16 smithi038 ceph-mon[92422]: pgmap v10050: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 494 KiB/s rd, 502 KiB/s wr, 12 op/s 2023-03-09T19:56:17.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:17 smithi122 ceph-mon[93869]: pgmap v10051: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 477 KiB/s wr, 12 op/s 2023-03-09T19:56:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:17 smithi181 ceph-mon[94189]: pgmap v10051: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 477 KiB/s wr, 12 op/s 2023-03-09T19:56:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:17 smithi038 ceph-mon[92422]: pgmap v10051: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 477 KiB/s wr, 12 op/s 2023-03-09T19:56:18.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18350.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:56:18.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17888.531747 secs 2023-03-09T19:56:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17888.531747 secs 2023-03-09T19:56:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17888.531747 secs 2023-03-09T19:56:19.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:19 smithi038 ceph-mon[92422]: pgmap v10052: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 220 KiB/s wr, 8 op/s 2023-03-09T19:56:20.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:19 smithi122 ceph-mon[93869]: pgmap v10052: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 220 KiB/s wr, 8 op/s 2023-03-09T19:56:20.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:19 smithi181 ceph-mon[94189]: pgmap v10052: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 220 KiB/s wr, 8 op/s 2023-03-09T19:56:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17891.217021 secs 2023-03-09T19:56:21.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17891.217021 secs 2023-03-09T19:56:21.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17891.217021 secs 2023-03-09T19:56:21.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:21 smithi038 ceph-mon[92422]: pgmap v10053: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 150 KiB/s wr, 11 op/s 2023-03-09T19:56:22.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:21 smithi122 ceph-mon[93869]: pgmap v10053: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 150 KiB/s wr, 11 op/s 2023-03-09T19:56:22.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:21 smithi181 ceph-mon[94189]: pgmap v10053: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 150 KiB/s wr, 11 op/s 2023-03-09T19:56:23.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17893.531843 secs 2023-03-09T19:56:23.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17893.531843 secs 2023-03-09T19:56:23.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17893.531843 secs 2023-03-09T19:56:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18355.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:56:24.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:24 smithi122 ceph-mon[93869]: pgmap v10054: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 147 KiB/s wr, 9 op/s 2023-03-09T19:56:24.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:24 smithi181 ceph-mon[94189]: pgmap v10054: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 147 KiB/s wr, 9 op/s 2023-03-09T19:56:24.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:24 smithi038 ceph-mon[92422]: pgmap v10054: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 147 KiB/s wr, 9 op/s 2023-03-09T19:56:25.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:25 smithi122 ceph-mon[93869]: osdmap e841: 12 total, 12 up, 12 in 2023-03-09T19:56:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:25 smithi181 ceph-mon[94189]: osdmap e841: 12 total, 12 up, 12 in 2023-03-09T19:56:25.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:25 smithi038 ceph-mon[92422]: osdmap e841: 12 total, 12 up, 12 in 2023-03-09T19:56:26.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17896.217080 secs 2023-03-09T19:56:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:26 smithi122 ceph-mon[93869]: pgmap v10056: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 234 KiB/s wr, 11 op/s 2023-03-09T19:56:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17896.217080 secs 2023-03-09T19:56:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:26 smithi181 ceph-mon[94189]: pgmap v10056: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 234 KiB/s wr, 11 op/s 2023-03-09T19:56:26.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17896.217080 secs 2023-03-09T19:56:26.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:26 smithi038 ceph-mon[92422]: pgmap v10056: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 234 KiB/s wr, 11 op/s 2023-03-09T19:56:27.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:27 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:56:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:27 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:56:27.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:27 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:56:27.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:27 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:56:27.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:27 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:56:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:27 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:56:28.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17898.531964 secs 2023-03-09T19:56:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:28 smithi122 ceph-mon[93869]: pgmap v10057: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 309 KiB/s wr, 12 op/s 2023-03-09T19:56:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17898.531964 secs 2023-03-09T19:56:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:28 smithi181 ceph-mon[94189]: pgmap v10057: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 309 KiB/s wr, 12 op/s 2023-03-09T19:56:28.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17898.531964 secs 2023-03-09T19:56:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:28 smithi038 ceph-mon[92422]: pgmap v10057: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 309 KiB/s wr, 12 op/s 2023-03-09T19:56:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18360.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:56:29.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:29 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:56:29.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:29 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:56:29.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:29 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:56:29.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:29 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:56:29.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:29 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:56:29.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:29 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:56:30.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:30 smithi122 ceph-mon[93869]: pgmap v10058: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 309 KiB/s wr, 12 op/s 2023-03-09T19:56:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:30 smithi181 ceph-mon[94189]: pgmap v10058: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 309 KiB/s wr, 12 op/s 2023-03-09T19:56:30.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:30 smithi038 ceph-mon[92422]: pgmap v10058: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 309 KiB/s wr, 12 op/s 2023-03-09T19:56:31.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17901.217145 secs 2023-03-09T19:56:31.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17901.217145 secs 2023-03-09T19:56:31.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17901.217145 secs 2023-03-09T19:56:32.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:32 smithi122 ceph-mon[93869]: pgmap v10059: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 383 KiB/s rd, 394 KiB/s wr, 12 op/s 2023-03-09T19:56:32.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:32 smithi181 ceph-mon[94189]: pgmap v10059: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 383 KiB/s rd, 394 KiB/s wr, 12 op/s 2023-03-09T19:56:32.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:32 smithi038 ceph-mon[92422]: pgmap v10059: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 383 KiB/s rd, 394 KiB/s wr, 12 op/s 2023-03-09T19:56:33.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17903.532052 secs 2023-03-09T19:56:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:33 smithi122 ceph-mon[93869]: osdmap e842: 12 total, 12 up, 12 in 2023-03-09T19:56:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17903.532052 secs 2023-03-09T19:56:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:33 smithi181 ceph-mon[94189]: osdmap e842: 12 total, 12 up, 12 in 2023-03-09T19:56:33.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17903.532052 secs 2023-03-09T19:56:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:33 smithi038 ceph-mon[92422]: osdmap e842: 12 total, 12 up, 12 in 2023-03-09T19:56:33.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18365.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:56:34.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:34 smithi122 ceph-mon[93869]: pgmap v10061: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 267 KiB/s wr, 10 op/s 2023-03-09T19:56:34.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:34 smithi181 ceph-mon[94189]: pgmap v10061: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 267 KiB/s wr, 10 op/s 2023-03-09T19:56:34.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:34 smithi038 ceph-mon[92422]: pgmap v10061: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 267 KiB/s wr, 10 op/s 2023-03-09T19:56:35.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17906.217203 secs 2023-03-09T19:56:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17906.217203 secs 2023-03-09T19:56:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17906.217203 secs 2023-03-09T19:56:36.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:36 smithi122 ceph-mon[93869]: pgmap v10062: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 247 KiB/s wr, 9 op/s 2023-03-09T19:56:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:36 smithi181 ceph-mon[94189]: pgmap v10062: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 247 KiB/s wr, 9 op/s 2023-03-09T19:56:36.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:36 smithi038 ceph-mon[92422]: pgmap v10062: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 247 KiB/s wr, 9 op/s 2023-03-09T19:56:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18370.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:56:38.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17908.532171 secs 2023-03-09T19:56:38.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:38 smithi122 ceph-mon[93869]: pgmap v10063: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 223 KiB/s wr, 9 op/s 2023-03-09T19:56:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17908.532171 secs 2023-03-09T19:56:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:38 smithi181 ceph-mon[94189]: pgmap v10063: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 223 KiB/s wr, 9 op/s 2023-03-09T19:56:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17908.532171 secs 2023-03-09T19:56:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:38 smithi038 ceph-mon[92422]: pgmap v10063: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 223 KiB/s wr, 9 op/s 2023-03-09T19:56:40.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:40 smithi122 ceph-mon[93869]: pgmap v10064: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 223 KiB/s wr, 8 op/s 2023-03-09T19:56:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17911.217262 secs 2023-03-09T19:56:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:40 smithi181 ceph-mon[94189]: pgmap v10064: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 223 KiB/s wr, 8 op/s 2023-03-09T19:56:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17911.217262 secs 2023-03-09T19:56:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:40 smithi038 ceph-mon[92422]: pgmap v10064: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 223 KiB/s wr, 8 op/s 2023-03-09T19:56:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17911.217262 secs 2023-03-09T19:56:42.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:42 smithi122 ceph-mon[93869]: pgmap v10065: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 134 KiB/s wr, 8 op/s 2023-03-09T19:56:42.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:42 smithi181 ceph-mon[94189]: pgmap v10065: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 134 KiB/s wr, 8 op/s 2023-03-09T19:56:42.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:42 smithi038 ceph-mon[92422]: pgmap v10065: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 134 KiB/s wr, 8 op/s 2023-03-09T19:56:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18375.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:56:43.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17913.532247 secs 2023-03-09T19:56:43.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:43 smithi122 ceph-mon[93869]: pgmap v10066: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 131 KiB/s wr, 8 op/s 2023-03-09T19:56:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17913.532247 secs 2023-03-09T19:56:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:43 smithi181 ceph-mon[94189]: pgmap v10066: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 131 KiB/s wr, 8 op/s 2023-03-09T19:56:43.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17913.532247 secs 2023-03-09T19:56:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:43 smithi038 ceph-mon[92422]: pgmap v10066: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 131 KiB/s wr, 8 op/s 2023-03-09T19:56:45.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17916.217333 secs 2023-03-09T19:56:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17916.217333 secs 2023-03-09T19:56:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17916.217333 secs 2023-03-09T19:56:46.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:46 smithi122 ceph-mon[93869]: pgmap v10067: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 259 KiB/s wr, 9 op/s 2023-03-09T19:56:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:46 smithi181 ceph-mon[94189]: pgmap v10067: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 259 KiB/s wr, 9 op/s 2023-03-09T19:56:46.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:46 smithi038 ceph-mon[92422]: pgmap v10067: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 259 KiB/s wr, 9 op/s 2023-03-09T19:56:47.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:56:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:56:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:56:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:56:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:56:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:56:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:56:47.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:56:47.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:56:47.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:56:47.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:56:47.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:56:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:56:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:56:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:56:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18380.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:56:48.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:48 smithi122 ceph-mon[93869]: pgmap v10068: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 274 KiB/s wr, 11 op/s 2023-03-09T19:56:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:48 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:56:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:56:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:56:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:56:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:48 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:56:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:56:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:56:48.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:56:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:48 smithi181 ceph-mon[94189]: pgmap v10068: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 274 KiB/s wr, 11 op/s 2023-03-09T19:56:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:48 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:56:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:56:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:56:48.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:56:48.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:48 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:56:48.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:56:48.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:56:48.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:56:48.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:48 smithi038 ceph-mon[92422]: pgmap v10068: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 274 KiB/s wr, 11 op/s 2023-03-09T19:56:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:48 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:56:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:56:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:56:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:56:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:48 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:56:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:56:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:56:49.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:56:50.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:50 smithi122 ceph-mon[93869]: pgmap v10069: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 355 KiB/s rd, 423 KiB/s wr, 10 op/s 2023-03-09T19:56:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17921.217402 secs 2023-03-09T19:56:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:50 smithi181 ceph-mon[94189]: pgmap v10069: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 355 KiB/s rd, 423 KiB/s wr, 10 op/s 2023-03-09T19:56:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17921.217402 secs 2023-03-09T19:56:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:50 smithi038 ceph-mon[92422]: pgmap v10069: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 355 KiB/s rd, 423 KiB/s wr, 10 op/s 2023-03-09T19:56:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17921.217402 secs 2023-03-09T19:56:52.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:52 smithi122 ceph-mon[93869]: pgmap v10070: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 438 KiB/s wr, 13 op/s 2023-03-09T19:56:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:52 smithi181 ceph-mon[94189]: pgmap v10070: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 438 KiB/s wr, 13 op/s 2023-03-09T19:56:52.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:52 smithi038 ceph-mon[92422]: pgmap v10070: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 438 KiB/s wr, 13 op/s 2023-03-09T19:56:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18385.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:56:54.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:54 smithi122 ceph-mon[93869]: pgmap v10071: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 372 KiB/s wr, 11 op/s 2023-03-09T19:56:54.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:54 smithi181 ceph-mon[94189]: pgmap v10071: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 372 KiB/s wr, 11 op/s 2023-03-09T19:56:54.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:54 smithi038 ceph-mon[92422]: pgmap v10071: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 372 KiB/s wr, 11 op/s 2023-03-09T19:56:55.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17926.217479 secs 2023-03-09T19:56:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17926.217479 secs 2023-03-09T19:56:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17926.217479 secs 2023-03-09T19:56:56.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:56 smithi122 ceph-mon[93869]: pgmap v10072: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 582 KiB/s rd, 480 KiB/s wr, 13 op/s 2023-03-09T19:56:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:56 smithi181 ceph-mon[94189]: pgmap v10072: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 582 KiB/s rd, 480 KiB/s wr, 13 op/s 2023-03-09T19:56:56.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:56 smithi038 ceph-mon[92422]: pgmap v10072: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 582 KiB/s rd, 480 KiB/s wr, 13 op/s 2023-03-09T19:56:58.793 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18390.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:56:58.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:58 smithi122 ceph-mon[93869]: pgmap v10073: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 507 KiB/s rd, 347 KiB/s wr, 13 op/s 2023-03-09T19:56:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:58 smithi181 ceph-mon[94189]: pgmap v10073: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 507 KiB/s rd, 347 KiB/s wr, 13 op/s 2023-03-09T19:56:58.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:58 smithi038 ceph-mon[92422]: pgmap v10073: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 507 KiB/s rd, 347 KiB/s wr, 13 op/s 2023-03-09T19:56:59.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:56:59.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:56:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:56:59 smithi038 ceph-mon[92422]: pgmap v10074: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 332 KiB/s wr, 11 op/s 2023-03-09T19:57:00.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:57:00.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:57:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:56:59 smithi122 ceph-mon[93869]: pgmap v10074: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 332 KiB/s wr, 11 op/s 2023-03-09T19:57:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:57:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:57:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:56:59 smithi181 ceph-mon[94189]: pgmap v10074: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 332 KiB/s wr, 11 op/s 2023-03-09T19:57:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:57:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:57:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17931.217538 secs 2023-03-09T19:57:01.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:57:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:57:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17931.217538 secs 2023-03-09T19:57:01.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:57:01.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:57:01.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17931.217538 secs 2023-03-09T19:57:01.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:01 smithi038 ceph-mon[92422]: pgmap v10075: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 279 KiB/s wr, 15 op/s 2023-03-09T19:57:02.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:01 smithi122 ceph-mon[93869]: pgmap v10075: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 279 KiB/s wr, 15 op/s 2023-03-09T19:57:02.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:01 smithi181 ceph-mon[94189]: pgmap v10075: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 279 KiB/s wr, 15 op/s 2023-03-09T19:57:03.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17933.532664 secs 2023-03-09T19:57:03.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17933.532664 secs 2023-03-09T19:57:03.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17933.532664 secs 2023-03-09T19:57:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18395.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:57:04.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:04 smithi122 ceph-mon[93869]: pgmap v10076: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 442 KiB/s rd, 285 KiB/s wr, 24 op/s 2023-03-09T19:57:04.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:04 smithi181 ceph-mon[94189]: pgmap v10076: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 442 KiB/s rd, 285 KiB/s wr, 24 op/s 2023-03-09T19:57:04.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:04 smithi038 ceph-mon[92422]: pgmap v10076: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 442 KiB/s rd, 285 KiB/s wr, 24 op/s 2023-03-09T19:57:05.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:05 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:57:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:05 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:57:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:05 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:57:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:05 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:57:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:05 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:57:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:05 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:57:05.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:05 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:57:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:05 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:57:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:05 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:57:06.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17936.217597 secs 2023-03-09T19:57:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:06 smithi122 ceph-mon[93869]: pgmap v10077: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 338 KiB/s wr, 25 op/s 2023-03-09T19:57:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17936.217597 secs 2023-03-09T19:57:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:06 smithi181 ceph-mon[94189]: pgmap v10077: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 338 KiB/s wr, 25 op/s 2023-03-09T19:57:06.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17936.217597 secs 2023-03-09T19:57:06.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:06 smithi038 ceph-mon[92422]: pgmap v10077: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 338 KiB/s wr, 25 op/s 2023-03-09T19:57:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18400.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:57:08.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17938.532784 secs 2023-03-09T19:57:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:08 smithi122 ceph-mon[93869]: pgmap v10078: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 231 KiB/s wr, 24 op/s 2023-03-09T19:57:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17938.532784 secs 2023-03-09T19:57:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:08 smithi181 ceph-mon[94189]: pgmap v10078: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 231 KiB/s wr, 24 op/s 2023-03-09T19:57:08.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17938.532784 secs 2023-03-09T19:57:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:08 smithi038 ceph-mon[92422]: pgmap v10078: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 231 KiB/s wr, 24 op/s 2023-03-09T19:57:10.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:09 smithi122 ceph-mon[93869]: pgmap v10079: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 216 KiB/s wr, 22 op/s 2023-03-09T19:57:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:09 smithi122 ceph-mon[93869]: osdmap e843: 12 total, 12 up, 12 in 2023-03-09T19:57:10.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:09 smithi181 ceph-mon[94189]: pgmap v10079: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 216 KiB/s wr, 22 op/s 2023-03-09T19:57:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:09 smithi181 ceph-mon[94189]: osdmap e843: 12 total, 12 up, 12 in 2023-03-09T19:57:10.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:09 smithi038 ceph-mon[92422]: pgmap v10079: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 216 KiB/s wr, 22 op/s 2023-03-09T19:57:10.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:09 smithi038 ceph-mon[92422]: osdmap e843: 12 total, 12 up, 12 in 2023-03-09T19:57:11.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17941.219378 secs 2023-03-09T19:57:11.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17941.219378 secs 2023-03-09T19:57:11.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17941.219378 secs 2023-03-09T19:57:12.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:11 smithi122 ceph-mon[93869]: pgmap v10081: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 90 KiB/s wr, 21 op/s 2023-03-09T19:57:12.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:11 smithi181 ceph-mon[94189]: pgmap v10081: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 90 KiB/s wr, 21 op/s 2023-03-09T19:57:12.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:11 smithi038 ceph-mon[92422]: pgmap v10081: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 90 KiB/s wr, 21 op/s 2023-03-09T19:57:13.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17943.532925 secs 2023-03-09T19:57:13.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17943.532925 secs 2023-03-09T19:57:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17943.532925 secs 2023-03-09T19:57:13.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18405.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:57:14.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:14 smithi122 ceph-mon[93869]: pgmap v10082: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 64 KiB/s wr, 7 op/s 2023-03-09T19:57:14.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:14 smithi181 ceph-mon[94189]: pgmap v10082: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 64 KiB/s wr, 7 op/s 2023-03-09T19:57:14.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:14 smithi038 ceph-mon[92422]: pgmap v10082: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 64 KiB/s wr, 7 op/s 2023-03-09T19:57:15.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17946.219438 secs 2023-03-09T19:57:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17946.219438 secs 2023-03-09T19:57:15.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17946.219438 secs 2023-03-09T19:57:16.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:16 smithi122 ceph-mon[93869]: pgmap v10083: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 KiB/s wr, 9 op/s 2023-03-09T19:57:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:16 smithi181 ceph-mon[94189]: pgmap v10083: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 KiB/s wr, 9 op/s 2023-03-09T19:57:16.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:16 smithi038 ceph-mon[92422]: pgmap v10083: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 KiB/s wr, 9 op/s 2023-03-09T19:57:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18410.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:57:18.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17948.533004 secs 2023-03-09T19:57:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:18 smithi122 ceph-mon[93869]: pgmap v10084: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 93 KiB/s wr, 8 op/s 2023-03-09T19:57:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:18 smithi122 ceph-mon[93869]: osdmap e844: 12 total, 12 up, 12 in 2023-03-09T19:57:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17948.533004 secs 2023-03-09T19:57:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:18 smithi181 ceph-mon[94189]: pgmap v10084: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 93 KiB/s wr, 8 op/s 2023-03-09T19:57:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:18 smithi181 ceph-mon[94189]: osdmap e844: 12 total, 12 up, 12 in 2023-03-09T19:57:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17948.533004 secs 2023-03-09T19:57:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:18 smithi038 ceph-mon[92422]: pgmap v10084: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 93 KiB/s wr, 8 op/s 2023-03-09T19:57:19.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:18 smithi038 ceph-mon[92422]: osdmap e844: 12 total, 12 up, 12 in 2023-03-09T19:57:20.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:19 smithi122 ceph-mon[93869]: pgmap v10086: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 93 KiB/s wr, 8 op/s 2023-03-09T19:57:20.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:19 smithi181 ceph-mon[94189]: pgmap v10086: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 93 KiB/s wr, 8 op/s 2023-03-09T19:57:20.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:19 smithi038 ceph-mon[92422]: pgmap v10086: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 93 KiB/s wr, 8 op/s 2023-03-09T19:57:21.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17951.219497 secs 2023-03-09T19:57:21.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17951.219497 secs 2023-03-09T19:57:21.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17951.219497 secs 2023-03-09T19:57:22.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:21 smithi122 ceph-mon[93869]: pgmap v10087: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 113 KiB/s wr, 8 op/s 2023-03-09T19:57:22.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:21 smithi181 ceph-mon[94189]: pgmap v10087: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 113 KiB/s wr, 8 op/s 2023-03-09T19:57:22.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:21 smithi038 ceph-mon[92422]: pgmap v10087: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 113 KiB/s wr, 8 op/s 2023-03-09T19:57:23.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17953.533139 secs 2023-03-09T19:57:23.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17953.533139 secs 2023-03-09T19:57:23.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17953.533139 secs 2023-03-09T19:57:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18415.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:57:24.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:24 smithi122 ceph-mon[93869]: pgmap v10088: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 244 KiB/s wr, 9 op/s 2023-03-09T19:57:24.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:24 smithi181 ceph-mon[94189]: pgmap v10088: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 244 KiB/s wr, 9 op/s 2023-03-09T19:57:24.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:24 smithi038 ceph-mon[92422]: pgmap v10088: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 244 KiB/s wr, 9 op/s 2023-03-09T19:57:25.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17956.219559 secs 2023-03-09T19:57:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17956.219559 secs 2023-03-09T19:57:25.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17956.219559 secs 2023-03-09T19:57:26.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:26 smithi122 ceph-mon[93869]: pgmap v10089: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 153 KiB/s wr, 7 op/s 2023-03-09T19:57:26.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:26 smithi181 ceph-mon[94189]: pgmap v10089: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 153 KiB/s wr, 7 op/s 2023-03-09T19:57:26.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:26 smithi038 ceph-mon[92422]: pgmap v10089: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 153 KiB/s wr, 7 op/s 2023-03-09T19:57:27.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:27 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:57:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:27 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:57:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:27 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:57:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:27 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:57:27.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:27 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:57:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:27 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:57:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18420.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:57:28.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17958.533290 secs 2023-03-09T19:57:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:28 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:57:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:28 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:57:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:28 smithi122 ceph-mon[93869]: pgmap v10090: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 152 KiB/s wr, 7 op/s 2023-03-09T19:57:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17958.533290 secs 2023-03-09T19:57:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:28 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:57:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:28 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:57:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:28 smithi181 ceph-mon[94189]: pgmap v10090: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 152 KiB/s wr, 7 op/s 2023-03-09T19:57:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17958.533290 secs 2023-03-09T19:57:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:28 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:57:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:28 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:57:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:28 smithi038 ceph-mon[92422]: pgmap v10090: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 152 KiB/s wr, 7 op/s 2023-03-09T19:57:30.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:29 smithi122 ceph-mon[93869]: pgmap v10091: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 173 KiB/s wr, 7 op/s 2023-03-09T19:57:30.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:29 smithi181 ceph-mon[94189]: pgmap v10091: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 173 KiB/s wr, 7 op/s 2023-03-09T19:57:30.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:29 smithi038 ceph-mon[92422]: pgmap v10091: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 173 KiB/s wr, 7 op/s 2023-03-09T19:57:31.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17961.219620 secs 2023-03-09T19:57:31.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17961.219620 secs 2023-03-09T19:57:31.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17961.219620 secs 2023-03-09T19:57:32.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:31 smithi122 ceph-mon[93869]: pgmap v10092: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 163 KiB/s wr, 10 op/s 2023-03-09T19:57:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:31 smithi181 ceph-mon[94189]: pgmap v10092: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 163 KiB/s wr, 10 op/s 2023-03-09T19:57:32.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:31 smithi038 ceph-mon[92422]: pgmap v10092: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 163 KiB/s wr, 10 op/s 2023-03-09T19:57:33.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17963.533405 secs 2023-03-09T19:57:33.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17963.533405 secs 2023-03-09T19:57:33.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17963.533405 secs 2023-03-09T19:57:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18425.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:57:34.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:34 smithi122 ceph-mon[93869]: pgmap v10093: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 146 KiB/s wr, 7 op/s 2023-03-09T19:57:34.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:34 smithi181 ceph-mon[94189]: pgmap v10093: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 146 KiB/s wr, 7 op/s 2023-03-09T19:57:34.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:34 smithi038 ceph-mon[92422]: pgmap v10093: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 146 KiB/s wr, 7 op/s 2023-03-09T19:57:35.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17966.219679 secs 2023-03-09T19:57:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17966.219679 secs 2023-03-09T19:57:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17966.219679 secs 2023-03-09T19:57:36.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:36 smithi122 ceph-mon[93869]: pgmap v10094: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 36 KiB/s wr, 9 op/s 2023-03-09T19:57:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:36 smithi181 ceph-mon[94189]: pgmap v10094: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 36 KiB/s wr, 9 op/s 2023-03-09T19:57:36.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:36 smithi038 ceph-mon[92422]: pgmap v10094: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 36 KiB/s wr, 9 op/s 2023-03-09T19:57:38.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:37 smithi122 ceph-mon[93869]: pgmap v10095: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 36 KiB/s wr, 7 op/s 2023-03-09T19:57:38.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:37 smithi181 ceph-mon[94189]: pgmap v10095: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 36 KiB/s wr, 7 op/s 2023-03-09T19:57:38.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:37 smithi038 ceph-mon[92422]: pgmap v10095: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 36 KiB/s wr, 7 op/s 2023-03-09T19:57:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18430.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:57:40.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:39 smithi122 ceph-mon[93869]: pgmap v10096: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 123 KiB/s wr, 7 op/s 2023-03-09T19:57:40.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:39 smithi181 ceph-mon[94189]: pgmap v10096: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 123 KiB/s wr, 7 op/s 2023-03-09T19:57:40.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:39 smithi038 ceph-mon[92422]: pgmap v10096: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 123 KiB/s wr, 7 op/s 2023-03-09T19:57:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17971.219769 secs 2023-03-09T19:57:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:40 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:57:41.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17971.219769 secs 2023-03-09T19:57:41.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:40 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:57:41.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17971.219769 secs 2023-03-09T19:57:41.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:40 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:57:42.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:41 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:57:42.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:57:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:57:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:41 smithi122 ceph-mon[93869]: pgmap v10097: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 102 KiB/s wr, 9 op/s 2023-03-09T19:57:42.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:41 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:57:42.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:57:42.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:57:42.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:41 smithi181 ceph-mon[94189]: pgmap v10097: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 102 KiB/s wr, 9 op/s 2023-03-09T19:57:42.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:41 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:57:42.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:57:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:57:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:41 smithi038 ceph-mon[92422]: pgmap v10097: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 102 KiB/s wr, 9 op/s 2023-03-09T19:57:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18435.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:57:44.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:44 smithi122 ceph-mon[93869]: pgmap v10098: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 119 KiB/s wr, 7 op/s 2023-03-09T19:57:44.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:44 smithi181 ceph-mon[94189]: pgmap v10098: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 119 KiB/s wr, 7 op/s 2023-03-09T19:57:44.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:44 smithi038 ceph-mon[92422]: pgmap v10098: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 119 KiB/s wr, 7 op/s 2023-03-09T19:57:45.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17976.219843 secs 2023-03-09T19:57:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17976.219843 secs 2023-03-09T19:57:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17976.219843 secs 2023-03-09T19:57:46.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:46 smithi122 ceph-mon[93869]: pgmap v10099: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 134 KiB/s wr, 10 op/s 2023-03-09T19:57:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:46 smithi181 ceph-mon[94189]: pgmap v10099: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 134 KiB/s wr, 10 op/s 2023-03-09T19:57:46.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:46 smithi038 ceph-mon[92422]: pgmap v10099: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 134 KiB/s wr, 10 op/s 2023-03-09T19:57:48.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:47 smithi122 ceph-mon[93869]: pgmap v10100: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 173 KiB/s wr, 8 op/s 2023-03-09T19:57:48.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:47 smithi181 ceph-mon[94189]: pgmap v10100: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 173 KiB/s wr, 8 op/s 2023-03-09T19:57:48.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:47 smithi038 ceph-mon[92422]: pgmap v10100: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 173 KiB/s wr, 8 op/s 2023-03-09T19:57:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18440.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:57:49.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:48 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 17978.533774 secs 2023-03-09T19:57:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:57:49.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:48 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 17978.533774 secs 2023-03-09T19:57:49.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:57:49.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:48 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 17978.533774 secs 2023-03-09T19:57:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:57:50.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:49 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:57:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:57:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:57:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:57:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:57:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:57:50.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:57:50.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:57:50.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:57:50.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:57:50.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:49 smithi122 ceph-mon[93869]: pgmap v10101: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 265 KiB/s wr, 9 op/s 2023-03-09T19:57:50.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:49 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:57:50.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:57:50.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:57:50.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:57:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:57:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:57:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:57:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:57:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:57:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:57:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:49 smithi181 ceph-mon[94189]: pgmap v10101: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 265 KiB/s wr, 9 op/s 2023-03-09T19:57:50.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:49 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:57:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:57:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:57:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:57:50.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:57:50.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:57:50.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:57:50.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:57:50.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:57:50.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:57:50.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:49 smithi038 ceph-mon[92422]: pgmap v10101: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 265 KiB/s wr, 9 op/s 2023-03-09T19:57:51.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17981.219907 secs 2023-03-09T19:57:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:57:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:57:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17981.219907 secs 2023-03-09T19:57:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:57:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:57:51.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17981.219907 secs 2023-03-09T19:57:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:57:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:57:52.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:52 smithi122 ceph-mon[93869]: pgmap v10102: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 194 KiB/s wr, 11 op/s 2023-03-09T19:57:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:52 smithi181 ceph-mon[94189]: pgmap v10102: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 194 KiB/s wr, 11 op/s 2023-03-09T19:57:52.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:52 smithi038 ceph-mon[92422]: pgmap v10102: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 194 KiB/s wr, 11 op/s 2023-03-09T19:57:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18445.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:57:53.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:53 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 17983.533831 secs 2023-03-09T19:57:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:57:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:57:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:53 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 17983.533831 secs 2023-03-09T19:57:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:57:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:57:53.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:53 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 17983.533831 secs 2023-03-09T19:57:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:57:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:57:54.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:54 smithi122 ceph-mon[93869]: pgmap v10103: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 318 KiB/s wr, 11 op/s 2023-03-09T19:57:54.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:54 smithi181 ceph-mon[94189]: pgmap v10103: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 318 KiB/s wr, 11 op/s 2023-03-09T19:57:54.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:54 smithi038 ceph-mon[92422]: pgmap v10103: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 318 KiB/s wr, 11 op/s 2023-03-09T19:57:55.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17986.219966 secs 2023-03-09T19:57:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17986.219966 secs 2023-03-09T19:57:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17986.219966 secs 2023-03-09T19:57:56.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:56 smithi122 ceph-mon[93869]: pgmap v10104: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 301 KiB/s wr, 12 op/s 2023-03-09T19:57:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:56 smithi181 ceph-mon[94189]: pgmap v10104: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 301 KiB/s wr, 12 op/s 2023-03-09T19:57:56.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:56 smithi038 ceph-mon[92422]: pgmap v10104: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 301 KiB/s wr, 12 op/s 2023-03-09T19:57:58.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:57 smithi122 ceph-mon[93869]: pgmap v10105: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 409 KiB/s wr, 11 op/s 2023-03-09T19:57:58.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:57 smithi181 ceph-mon[94189]: pgmap v10105: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 409 KiB/s wr, 11 op/s 2023-03-09T19:57:58.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:57 smithi038 ceph-mon[92422]: pgmap v10105: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 409 KiB/s wr, 11 op/s 2023-03-09T19:57:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18450.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:57:59.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:58 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 17988.533933 secs 2023-03-09T19:57:59.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:58 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 17988.533933 secs 2023-03-09T19:57:59.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:58 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 17988.533933 secs 2023-03-09T19:58:00.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:58:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:58:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:57:59 smithi122 ceph-mon[93869]: pgmap v10106: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 370 KiB/s wr, 10 op/s 2023-03-09T19:58:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:58:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:58:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:57:59 smithi181 ceph-mon[94189]: pgmap v10106: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 370 KiB/s wr, 10 op/s 2023-03-09T19:58:00.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:58:00.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:58:00.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:57:59 smithi038 ceph-mon[92422]: pgmap v10106: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 370 KiB/s wr, 10 op/s 2023-03-09T19:58:01.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17991.219984 secs 2023-03-09T19:58:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:58:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:58:01.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17991.219984 secs 2023-03-09T19:58:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:58:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:58:01.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17991.219984 secs 2023-03-09T19:58:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:58:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:58:02.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:01 smithi122 ceph-mon[93869]: pgmap v10107: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 318 KiB/s wr, 13 op/s 2023-03-09T19:58:02.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:01 smithi181 ceph-mon[94189]: pgmap v10107: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 318 KiB/s wr, 13 op/s 2023-03-09T19:58:02.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:01 smithi038 ceph-mon[92422]: pgmap v10107: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 318 KiB/s wr, 13 op/s 2023-03-09T19:58:03.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17993.534024 secs 2023-03-09T19:58:03.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17993.534024 secs 2023-03-09T19:58:03.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17993.534024 secs 2023-03-09T19:58:03.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18455.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:58:04.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:04 smithi122 ceph-mon[93869]: pgmap v10108: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 464 KiB/s rd, 487 KiB/s wr, 25 op/s 2023-03-09T19:58:04.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:04 smithi181 ceph-mon[94189]: pgmap v10108: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 464 KiB/s rd, 487 KiB/s wr, 25 op/s 2023-03-09T19:58:04.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:04 smithi038 ceph-mon[92422]: pgmap v10108: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 464 KiB/s rd, 487 KiB/s wr, 25 op/s 2023-03-09T19:58:05.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:05 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:58:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:05 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:58:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:05 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:58:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:05 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:58:05.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:05 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:58:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:05 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:58:06.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17996.220046 secs 2023-03-09T19:58:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:06 smithi122 ceph-mon[93869]: pgmap v10109: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 360 KiB/s wr, 26 op/s 2023-03-09T19:58:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17996.220046 secs 2023-03-09T19:58:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:06 smithi181 ceph-mon[94189]: pgmap v10109: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 360 KiB/s wr, 26 op/s 2023-03-09T19:58:06.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17996.220046 secs 2023-03-09T19:58:06.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:06 smithi038 ceph-mon[92422]: pgmap v10109: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 360 KiB/s wr, 26 op/s 2023-03-09T19:58:08.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:07 smithi122 ceph-mon[93869]: pgmap v10110: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 437 KiB/s wr, 25 op/s 2023-03-09T19:58:08.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:07 smithi181 ceph-mon[94189]: pgmap v10110: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 437 KiB/s wr, 25 op/s 2023-03-09T19:58:08.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:07 smithi038 ceph-mon[92422]: pgmap v10110: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 437 KiB/s wr, 25 op/s 2023-03-09T19:58:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18460.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:58:09.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 17998.534140 secs 2023-03-09T19:58:09.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 17998.534140 secs 2023-03-09T19:58:09.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 17998.534140 secs 2023-03-09T19:58:10.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:09 smithi122 ceph-mon[93869]: pgmap v10111: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 314 KiB/s wr, 24 op/s 2023-03-09T19:58:10.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:09 smithi181 ceph-mon[94189]: pgmap v10111: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 314 KiB/s wr, 24 op/s 2023-03-09T19:58:10.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:09 smithi038 ceph-mon[92422]: pgmap v10111: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 314 KiB/s wr, 24 op/s 2023-03-09T19:58:11.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18001.220108 secs 2023-03-09T19:58:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:10 smithi122 ceph-mon[93869]: osdmap e845: 12 total, 12 up, 12 in 2023-03-09T19:58:11.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18001.220108 secs 2023-03-09T19:58:11.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:10 smithi181 ceph-mon[94189]: osdmap e845: 12 total, 12 up, 12 in 2023-03-09T19:58:11.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18001.220108 secs 2023-03-09T19:58:11.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:10 smithi038 ceph-mon[92422]: osdmap e845: 12 total, 12 up, 12 in 2023-03-09T19:58:12.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:11 smithi122 ceph-mon[93869]: pgmap v10113: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 329 KiB/s wr, 26 op/s 2023-03-09T19:58:12.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:11 smithi181 ceph-mon[94189]: pgmap v10113: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 329 KiB/s wr, 26 op/s 2023-03-09T19:58:12.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:11 smithi038 ceph-mon[92422]: pgmap v10113: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 329 KiB/s wr, 26 op/s 2023-03-09T19:58:13.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18003.534264 secs 2023-03-09T19:58:13.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18003.534264 secs 2023-03-09T19:58:13.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18003.534264 secs 2023-03-09T19:58:13.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18465.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:58:14.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:14 smithi122 ceph-mon[93869]: pgmap v10114: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 210 KiB/s wr, 10 op/s 2023-03-09T19:58:14.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:14 smithi181 ceph-mon[94189]: pgmap v10114: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 210 KiB/s wr, 10 op/s 2023-03-09T19:58:14.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:14 smithi038 ceph-mon[92422]: pgmap v10114: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 210 KiB/s wr, 10 op/s 2023-03-09T19:58:15.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18006.220167 secs 2023-03-09T19:58:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18006.220167 secs 2023-03-09T19:58:15.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18006.220167 secs 2023-03-09T19:58:16.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:16 smithi122 ceph-mon[93869]: pgmap v10115: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 197 KiB/s wr, 9 op/s 2023-03-09T19:58:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:16 smithi181 ceph-mon[94189]: pgmap v10115: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 197 KiB/s wr, 9 op/s 2023-03-09T19:58:16.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:16 smithi038 ceph-mon[92422]: pgmap v10115: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 197 KiB/s wr, 9 op/s 2023-03-09T19:58:18.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:17 smithi122 ceph-mon[93869]: pgmap v10116: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 158 KiB/s wr, 8 op/s 2023-03-09T19:58:18.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:17 smithi181 ceph-mon[94189]: pgmap v10116: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 158 KiB/s wr, 8 op/s 2023-03-09T19:58:18.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:17 smithi038 ceph-mon[92422]: pgmap v10116: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 158 KiB/s wr, 8 op/s 2023-03-09T19:58:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18470.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:58:19.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18008.534369 secs 2023-03-09T19:58:19.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:18 smithi122 ceph-mon[93869]: osdmap e846: 12 total, 12 up, 12 in 2023-03-09T19:58:19.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18008.534369 secs 2023-03-09T19:58:19.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:18 smithi181 ceph-mon[94189]: osdmap e846: 12 total, 12 up, 12 in 2023-03-09T19:58:19.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18008.534369 secs 2023-03-09T19:58:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:18 smithi038 ceph-mon[92422]: osdmap e846: 12 total, 12 up, 12 in 2023-03-09T19:58:20.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:19 smithi122 ceph-mon[93869]: pgmap v10118: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 159 KiB/s wr, 8 op/s 2023-03-09T19:58:20.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:19 smithi181 ceph-mon[94189]: pgmap v10118: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 159 KiB/s wr, 8 op/s 2023-03-09T19:58:20.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:19 smithi038 ceph-mon[92422]: pgmap v10118: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 159 KiB/s wr, 8 op/s 2023-03-09T19:58:21.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18011.220247 secs 2023-03-09T19:58:21.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18011.220247 secs 2023-03-09T19:58:21.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18011.220247 secs 2023-03-09T19:58:22.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:21 smithi122 ceph-mon[93869]: pgmap v10119: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 157 KiB/s wr, 8 op/s 2023-03-09T19:58:22.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:21 smithi181 ceph-mon[94189]: pgmap v10119: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 157 KiB/s wr, 8 op/s 2023-03-09T19:58:22.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:21 smithi038 ceph-mon[92422]: pgmap v10119: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 157 KiB/s wr, 8 op/s 2023-03-09T19:58:23.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18013.534486 secs 2023-03-09T19:58:23.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18013.534486 secs 2023-03-09T19:58:23.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18013.534486 secs 2023-03-09T19:58:23.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18475.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:58:24.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:24 smithi122 ceph-mon[93869]: pgmap v10120: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 54 KiB/s wr, 7 op/s 2023-03-09T19:58:24.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:24 smithi181 ceph-mon[94189]: pgmap v10120: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 54 KiB/s wr, 7 op/s 2023-03-09T19:58:24.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:24 smithi038 ceph-mon[92422]: pgmap v10120: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 54 KiB/s wr, 7 op/s 2023-03-09T19:58:25.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18016.220305 secs 2023-03-09T19:58:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18016.220305 secs 2023-03-09T19:58:25.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18016.220305 secs 2023-03-09T19:58:26.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:26 smithi122 ceph-mon[93869]: pgmap v10121: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 54 KiB/s wr, 7 op/s 2023-03-09T19:58:26.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:26 smithi181 ceph-mon[94189]: pgmap v10121: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 54 KiB/s wr, 7 op/s 2023-03-09T19:58:26.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:26 smithi038 ceph-mon[92422]: pgmap v10121: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 54 KiB/s wr, 7 op/s 2023-03-09T19:58:28.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:27 smithi122 ceph-mon[93869]: pgmap v10122: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T19:58:28.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:27 smithi181 ceph-mon[94189]: pgmap v10122: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T19:58:28.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:27 smithi038 ceph-mon[92422]: pgmap v10122: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T19:58:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18480.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:58:29.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18018.534603 secs 2023-03-09T19:58:29.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18018.534603 secs 2023-03-09T19:58:29.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18018.534603 secs 2023-03-09T19:58:30.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:29 smithi122 ceph-mon[93869]: pgmap v10123: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 39 KiB/s wr, 7 op/s 2023-03-09T19:58:30.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:29 smithi181 ceph-mon[94189]: pgmap v10123: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 39 KiB/s wr, 7 op/s 2023-03-09T19:58:30.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:29 smithi038 ceph-mon[92422]: pgmap v10123: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 39 KiB/s wr, 7 op/s 2023-03-09T19:58:31.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18021.220388 secs 2023-03-09T19:58:31.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18021.220388 secs 2023-03-09T19:58:31.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18021.220388 secs 2023-03-09T19:58:32.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:31 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:58:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:31 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:58:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:31 smithi122 ceph-mon[93869]: pgmap v10124: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 163 KiB/s wr, 9 op/s 2023-03-09T19:58:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:31 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:58:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:31 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:58:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:31 smithi181 ceph-mon[94189]: pgmap v10124: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 163 KiB/s wr, 9 op/s 2023-03-09T19:58:32.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:31 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:58:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:31 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:58:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:31 smithi038 ceph-mon[92422]: pgmap v10124: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 163 KiB/s wr, 9 op/s 2023-03-09T19:58:33.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:32 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:58:33.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:32 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:58:33.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:32 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:58:33.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:32 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:58:33.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:32 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:58:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:32 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:58:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18485.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:58:34.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18023.534717 secs 2023-03-09T19:58:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:33 smithi122 ceph-mon[93869]: pgmap v10125: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 164 KiB/s wr, 8 op/s 2023-03-09T19:58:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18023.534717 secs 2023-03-09T19:58:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:33 smithi181 ceph-mon[94189]: pgmap v10125: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 164 KiB/s wr, 8 op/s 2023-03-09T19:58:34.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18023.534717 secs 2023-03-09T19:58:34.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:33 smithi038 ceph-mon[92422]: pgmap v10125: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 164 KiB/s wr, 8 op/s 2023-03-09T19:58:35.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18026.220461 secs 2023-03-09T19:58:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18026.220461 secs 2023-03-09T19:58:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18026.220461 secs 2023-03-09T19:58:36.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:36 smithi122 ceph-mon[93869]: pgmap v10126: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 164 KiB/s wr, 9 op/s 2023-03-09T19:58:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:36 smithi181 ceph-mon[94189]: pgmap v10126: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 164 KiB/s wr, 9 op/s 2023-03-09T19:58:36.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:36 smithi038 ceph-mon[92422]: pgmap v10126: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 164 KiB/s wr, 9 op/s 2023-03-09T19:58:38.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:37 smithi122 ceph-mon[93869]: pgmap v10127: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 163 KiB/s wr, 7 op/s 2023-03-09T19:58:38.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:37 smithi181 ceph-mon[94189]: pgmap v10127: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 163 KiB/s wr, 7 op/s 2023-03-09T19:58:38.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:37 smithi038 ceph-mon[92422]: pgmap v10127: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 163 KiB/s wr, 7 op/s 2023-03-09T19:58:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18490.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:58:39.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18028.534853 secs 2023-03-09T19:58:39.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18028.534853 secs 2023-03-09T19:58:39.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18028.534853 secs 2023-03-09T19:58:40.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:39 smithi122 ceph-mon[93869]: pgmap v10128: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 134 KiB/s wr, 6 op/s 2023-03-09T19:58:40.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:39 smithi181 ceph-mon[94189]: pgmap v10128: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 134 KiB/s wr, 6 op/s 2023-03-09T19:58:40.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:39 smithi038 ceph-mon[92422]: pgmap v10128: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 134 KiB/s wr, 6 op/s 2023-03-09T19:58:41.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18031.220520 secs 2023-03-09T19:58:41.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18031.220520 secs 2023-03-09T19:58:41.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18031.220520 secs 2023-03-09T19:58:42.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:41 smithi122 ceph-mon[93869]: pgmap v10129: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 362 KiB/s rd, 335 KiB/s wr, 10 op/s 2023-03-09T19:58:42.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:41 smithi181 ceph-mon[94189]: pgmap v10129: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 362 KiB/s rd, 335 KiB/s wr, 10 op/s 2023-03-09T19:58:42.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:41 smithi038 ceph-mon[92422]: pgmap v10129: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 362 KiB/s rd, 335 KiB/s wr, 10 op/s 2023-03-09T19:58:43.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18033.534986 secs 2023-03-09T19:58:43.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18033.534986 secs 2023-03-09T19:58:43.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18033.534986 secs 2023-03-09T19:58:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18495.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:58:44.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:44 smithi122 ceph-mon[93869]: pgmap v10130: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 208 KiB/s wr, 7 op/s 2023-03-09T19:58:44.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:44 smithi181 ceph-mon[94189]: pgmap v10130: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 208 KiB/s wr, 7 op/s 2023-03-09T19:58:44.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:44 smithi038 ceph-mon[92422]: pgmap v10130: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 208 KiB/s wr, 7 op/s 2023-03-09T19:58:45.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18036.220618 secs 2023-03-09T19:58:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18036.220618 secs 2023-03-09T19:58:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18036.220618 secs 2023-03-09T19:58:46.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:46 smithi122 ceph-mon[93869]: pgmap v10131: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 355 KiB/s rd, 358 KiB/s wr, 11 op/s 2023-03-09T19:58:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:46 smithi181 ceph-mon[94189]: pgmap v10131: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 355 KiB/s rd, 358 KiB/s wr, 11 op/s 2023-03-09T19:58:46.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:46 smithi038 ceph-mon[92422]: pgmap v10131: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 355 KiB/s rd, 358 KiB/s wr, 11 op/s 2023-03-09T19:58:48.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:47 smithi122 ceph-mon[93869]: pgmap v10132: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 358 KiB/s wr, 10 op/s 2023-03-09T19:58:48.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:47 smithi181 ceph-mon[94189]: pgmap v10132: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 358 KiB/s wr, 10 op/s 2023-03-09T19:58:48.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:47 smithi038 ceph-mon[92422]: pgmap v10132: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 358 KiB/s wr, 10 op/s 2023-03-09T19:58:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18500.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:58:49.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18038.535132 secs 2023-03-09T19:58:49.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18038.535132 secs 2023-03-09T19:58:49.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18038.535132 secs 2023-03-09T19:58:50.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:58:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:49 smithi122 ceph-mon[93869]: pgmap v10133: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 508 KiB/s rd, 372 KiB/s wr, 10 op/s 2023-03-09T19:58:50.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:58:50.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:49 smithi181 ceph-mon[94189]: pgmap v10133: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 508 KiB/s rd, 372 KiB/s wr, 10 op/s 2023-03-09T19:58:50.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:58:50.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:49 smithi038 ceph-mon[92422]: pgmap v10133: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 508 KiB/s rd, 372 KiB/s wr, 10 op/s 2023-03-09T19:58:51.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18041.220677 secs 2023-03-09T19:58:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:50 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:58:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:50 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:58:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:50 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:58:51.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18041.220677 secs 2023-03-09T19:58:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:50 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:58:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:50 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:58:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:50 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:58:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18041.220677 secs 2023-03-09T19:58:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:50 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:58:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:50 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:58:51.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:50 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:58:52.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:58:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:58:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:51 smithi122 ceph-mon[93869]: pgmap v10134: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 656 KiB/s rd, 478 KiB/s wr, 14 op/s 2023-03-09T19:58:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:58:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:58:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:51 smithi181 ceph-mon[94189]: pgmap v10134: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 656 KiB/s rd, 478 KiB/s wr, 14 op/s 2023-03-09T19:58:52.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:58:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:58:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:51 smithi038 ceph-mon[92422]: pgmap v10134: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 656 KiB/s rd, 478 KiB/s wr, 14 op/s 2023-03-09T19:58:53.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:58:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:58:53.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:58:53.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:58:53.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:58:53.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:58:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18505.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:58:54.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18043.535205 secs 2023-03-09T19:58:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:53 smithi122 ceph-mon[93869]: pgmap v10135: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 278 KiB/s wr, 11 op/s 2023-03-09T19:58:54.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18043.535205 secs 2023-03-09T19:58:54.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:53 smithi181 ceph-mon[94189]: pgmap v10135: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 278 KiB/s wr, 11 op/s 2023-03-09T19:58:54.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18043.535205 secs 2023-03-09T19:58:54.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:53 smithi038 ceph-mon[92422]: pgmap v10135: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 278 KiB/s wr, 11 op/s 2023-03-09T19:58:55.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18046.220766 secs 2023-03-09T19:58:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18046.220766 secs 2023-03-09T19:58:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18046.220766 secs 2023-03-09T19:58:56.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:56 smithi122 ceph-mon[93869]: pgmap v10136: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 519 KiB/s rd, 386 KiB/s wr, 14 op/s 2023-03-09T19:58:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:56 smithi181 ceph-mon[94189]: pgmap v10136: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 519 KiB/s rd, 386 KiB/s wr, 14 op/s 2023-03-09T19:58:56.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:56 smithi038 ceph-mon[92422]: pgmap v10136: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 519 KiB/s rd, 386 KiB/s wr, 14 op/s 2023-03-09T19:58:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:57 smithi122 ceph-mon[93869]: pgmap v10137: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 274 KiB/s wr, 11 op/s 2023-03-09T19:58:58.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:57 smithi181 ceph-mon[94189]: pgmap v10137: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 274 KiB/s wr, 11 op/s 2023-03-09T19:58:58.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:57 smithi038 ceph-mon[92422]: pgmap v10137: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 274 KiB/s wr, 11 op/s 2023-03-09T19:58:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18510.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:58:59.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18048.535304 secs 2023-03-09T19:58:59.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18048.535304 secs 2023-03-09T19:58:59.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18048.535304 secs 2023-03-09T19:59:00.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:59:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:59:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:58:59 smithi122 ceph-mon[93869]: pgmap v10138: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 288 KiB/s wr, 11 op/s 2023-03-09T19:59:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:59:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:59:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:58:59 smithi181 ceph-mon[94189]: pgmap v10138: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 288 KiB/s wr, 11 op/s 2023-03-09T19:59:00.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:59:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:59:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:58:59 smithi038 ceph-mon[92422]: pgmap v10138: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 288 KiB/s wr, 11 op/s 2023-03-09T19:59:01.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18051.220854 secs 2023-03-09T19:59:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:59:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:59:01.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18051.220854 secs 2023-03-09T19:59:01.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:59:01.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:59:01.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18051.220854 secs 2023-03-09T19:59:01.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:59:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:59:02.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:01 smithi122 ceph-mon[93869]: pgmap v10139: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 310 KiB/s wr, 13 op/s 2023-03-09T19:59:02.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:01 smithi181 ceph-mon[94189]: pgmap v10139: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 310 KiB/s wr, 13 op/s 2023-03-09T19:59:02.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:01 smithi038 ceph-mon[92422]: pgmap v10139: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 310 KiB/s wr, 13 op/s 2023-03-09T19:59:03.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18053.535438 secs 2023-03-09T19:59:03.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18053.535438 secs 2023-03-09T19:59:03.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18053.535438 secs 2023-03-09T19:59:03.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18515.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:59:04.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:04 smithi122 ceph-mon[93869]: pgmap v10140: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 218 KiB/s wr, 22 op/s 2023-03-09T19:59:04.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:04 smithi181 ceph-mon[94189]: pgmap v10140: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 218 KiB/s wr, 22 op/s 2023-03-09T19:59:04.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:04 smithi038 ceph-mon[92422]: pgmap v10140: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 218 KiB/s wr, 22 op/s 2023-03-09T19:59:05.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:05 smithi122 ceph-mon[93869]: osdmap e847: 12 total, 12 up, 12 in 2023-03-09T19:59:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:05 smithi181 ceph-mon[94189]: osdmap e847: 12 total, 12 up, 12 in 2023-03-09T19:59:05.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:05 smithi038 ceph-mon[92422]: osdmap e847: 12 total, 12 up, 12 in 2023-03-09T19:59:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18056.220938 secs 2023-03-09T19:59:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:06 smithi181 ceph-mon[94189]: pgmap v10142: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 159 KiB/s wr, 26 op/s 2023-03-09T19:59:06.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18056.220938 secs 2023-03-09T19:59:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:06 smithi038 ceph-mon[92422]: pgmap v10142: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 159 KiB/s wr, 26 op/s 2023-03-09T19:59:06.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18056.220938 secs 2023-03-09T19:59:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:06 smithi122 ceph-mon[93869]: pgmap v10142: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 159 KiB/s wr, 26 op/s 2023-03-09T19:59:08.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:07 smithi122 ceph-mon[93869]: pgmap v10143: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 317 KiB/s wr, 28 op/s 2023-03-09T19:59:08.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:07 smithi181 ceph-mon[94189]: pgmap v10143: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 317 KiB/s wr, 28 op/s 2023-03-09T19:59:08.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:07 smithi038 ceph-mon[92422]: pgmap v10143: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 317 KiB/s wr, 28 op/s 2023-03-09T19:59:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18520.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:59:09.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18058.535584 secs 2023-03-09T19:59:09.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18058.535584 secs 2023-03-09T19:59:09.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18058.535584 secs 2023-03-09T19:59:10.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:09 smithi122 ceph-mon[93869]: pgmap v10144: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 293 KiB/s rd, 317 KiB/s wr, 28 op/s 2023-03-09T19:59:10.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:09 smithi181 ceph-mon[94189]: pgmap v10144: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 293 KiB/s rd, 317 KiB/s wr, 28 op/s 2023-03-09T19:59:10.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:09 smithi038 ceph-mon[92422]: pgmap v10144: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 293 KiB/s rd, 317 KiB/s wr, 28 op/s 2023-03-09T19:59:11.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18061.220987 secs 2023-03-09T19:59:11.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18061.220987 secs 2023-03-09T19:59:11.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18061.220987 secs 2023-03-09T19:59:12.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:11 smithi122 ceph-mon[93869]: pgmap v10145: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 394 KiB/s wr, 27 op/s 2023-03-09T19:59:12.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:11 smithi181 ceph-mon[94189]: pgmap v10145: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 394 KiB/s wr, 27 op/s 2023-03-09T19:59:12.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:11 smithi038 ceph-mon[92422]: pgmap v10145: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 394 KiB/s wr, 27 op/s 2023-03-09T19:59:13.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18063.535695 secs 2023-03-09T19:59:13.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18063.535695 secs 2023-03-09T19:59:13.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18063.535695 secs 2023-03-09T19:59:13.793 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18525.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:59:14.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:14 smithi122 ceph-mon[93869]: osdmap e848: 12 total, 12 up, 12 in 2023-03-09T19:59:14.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:14 smithi122 ceph-mon[93869]: pgmap v10147: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 399 KiB/s wr, 13 op/s 2023-03-09T19:59:14.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:14 smithi181 ceph-mon[94189]: osdmap e848: 12 total, 12 up, 12 in 2023-03-09T19:59:14.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:14 smithi181 ceph-mon[94189]: pgmap v10147: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 399 KiB/s wr, 13 op/s 2023-03-09T19:59:14.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:14 smithi038 ceph-mon[92422]: osdmap e848: 12 total, 12 up, 12 in 2023-03-09T19:59:14.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:14 smithi038 ceph-mon[92422]: pgmap v10147: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 399 KiB/s wr, 13 op/s 2023-03-09T19:59:15.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18066.221048 secs 2023-03-09T19:59:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18066.221048 secs 2023-03-09T19:59:15.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18066.221048 secs 2023-03-09T19:59:16.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:16 smithi122 ceph-mon[93869]: pgmap v10148: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 405 KiB/s wr, 10 op/s 2023-03-09T19:59:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:16 smithi181 ceph-mon[94189]: pgmap v10148: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 405 KiB/s wr, 10 op/s 2023-03-09T19:59:16.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:16 smithi038 ceph-mon[92422]: pgmap v10148: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 405 KiB/s wr, 10 op/s 2023-03-09T19:59:18.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:17 smithi122 ceph-mon[93869]: pgmap v10149: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 202 KiB/s wr, 8 op/s 2023-03-09T19:59:18.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:17 smithi181 ceph-mon[94189]: pgmap v10149: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 202 KiB/s wr, 8 op/s 2023-03-09T19:59:18.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:17 smithi038 ceph-mon[92422]: pgmap v10149: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 202 KiB/s wr, 8 op/s 2023-03-09T19:59:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18530.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:59:19.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18068.535828 secs 2023-03-09T19:59:19.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18068.535828 secs 2023-03-09T19:59:19.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18068.535828 secs 2023-03-09T19:59:20.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:19 smithi122 ceph-mon[93869]: pgmap v10150: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 185 KiB/s wr, 8 op/s 2023-03-09T19:59:20.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:19 smithi181 ceph-mon[94189]: pgmap v10150: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 185 KiB/s wr, 8 op/s 2023-03-09T19:59:20.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:19 smithi038 ceph-mon[92422]: pgmap v10150: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 185 KiB/s wr, 8 op/s 2023-03-09T19:59:21.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18071.221109 secs 2023-03-09T19:59:21.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18071.221109 secs 2023-03-09T19:59:21.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18071.221109 secs 2023-03-09T19:59:22.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:21 smithi122 ceph-mon[93869]: pgmap v10151: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 66 KiB/s wr, 8 op/s 2023-03-09T19:59:22.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:21 smithi181 ceph-mon[94189]: pgmap v10151: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 66 KiB/s wr, 8 op/s 2023-03-09T19:59:22.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:21 smithi038 ceph-mon[92422]: pgmap v10151: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 66 KiB/s wr, 8 op/s 2023-03-09T19:59:23.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18073.535941 secs 2023-03-09T19:59:23.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18073.535941 secs 2023-03-09T19:59:23.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18073.535941 secs 2023-03-09T19:59:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18535.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:59:24.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:24 smithi122 ceph-mon[93869]: pgmap v10152: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 64 KiB/s wr, 7 op/s 2023-03-09T19:59:24.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:24 smithi181 ceph-mon[94189]: pgmap v10152: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 64 KiB/s wr, 7 op/s 2023-03-09T19:59:24.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:24 smithi038 ceph-mon[92422]: pgmap v10152: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 64 KiB/s wr, 7 op/s 2023-03-09T19:59:25.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18076.221171 secs 2023-03-09T19:59:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18076.221171 secs 2023-03-09T19:59:25.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18076.221171 secs 2023-03-09T19:59:26.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:26 smithi122 ceph-mon[93869]: pgmap v10153: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 121 KiB/s wr, 9 op/s 2023-03-09T19:59:26.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:26 smithi181 ceph-mon[94189]: pgmap v10153: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 121 KiB/s wr, 9 op/s 2023-03-09T19:59:26.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:26 smithi038 ceph-mon[92422]: pgmap v10153: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 121 KiB/s wr, 9 op/s 2023-03-09T19:59:27.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:27 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:59:27.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:27 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:59:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:27 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:59:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:27 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:59:27.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:27 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:59:27.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:27 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:59:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18540.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:59:28.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18078.536076 secs 2023-03-09T19:59:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:28 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:59:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:28 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:59:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:28 smithi122 ceph-mon[93869]: pgmap v10154: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 68 KiB/s wr, 7 op/s 2023-03-09T19:59:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18078.536076 secs 2023-03-09T19:59:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:28 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:59:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:28 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:59:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:28 smithi181 ceph-mon[94189]: pgmap v10154: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 68 KiB/s wr, 7 op/s 2023-03-09T19:59:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18078.536076 secs 2023-03-09T19:59:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:28 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:59:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:28 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:59:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:28 smithi038 ceph-mon[92422]: pgmap v10154: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 68 KiB/s wr, 7 op/s 2023-03-09T19:59:30.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:29 smithi122 ceph-mon[93869]: pgmap v10155: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 67 KiB/s wr, 7 op/s 2023-03-09T19:59:30.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:29 smithi181 ceph-mon[94189]: pgmap v10155: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 67 KiB/s wr, 7 op/s 2023-03-09T19:59:30.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:29 smithi038 ceph-mon[92422]: pgmap v10155: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 67 KiB/s wr, 7 op/s 2023-03-09T19:59:31.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18081.221255 secs 2023-03-09T19:59:31.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18081.221255 secs 2023-03-09T19:59:31.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18081.221255 secs 2023-03-09T19:59:32.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:31 smithi122 ceph-mon[93869]: pgmap v10156: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 67 KiB/s wr, 9 op/s 2023-03-09T19:59:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:31 smithi181 ceph-mon[94189]: pgmap v10156: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 67 KiB/s wr, 9 op/s 2023-03-09T19:59:32.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:31 smithi038 ceph-mon[92422]: pgmap v10156: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 67 KiB/s wr, 9 op/s 2023-03-09T19:59:33.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18083.536191 secs 2023-03-09T19:59:33.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18083.536191 secs 2023-03-09T19:59:33.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18083.536191 secs 2023-03-09T19:59:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18545.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:59:34.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:34 smithi122 ceph-mon[93869]: pgmap v10157: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 66 KiB/s wr, 6 op/s 2023-03-09T19:59:34.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:34 smithi181 ceph-mon[94189]: pgmap v10157: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 66 KiB/s wr, 6 op/s 2023-03-09T19:59:34.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:34 smithi038 ceph-mon[92422]: pgmap v10157: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 66 KiB/s wr, 6 op/s 2023-03-09T19:59:35.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18086.221315 secs 2023-03-09T19:59:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18086.221315 secs 2023-03-09T19:59:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18086.221315 secs 2023-03-09T19:59:36.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:36 smithi122 ceph-mon[93869]: pgmap v10158: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 132 KiB/s wr, 9 op/s 2023-03-09T19:59:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:36 smithi181 ceph-mon[94189]: pgmap v10158: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 132 KiB/s wr, 9 op/s 2023-03-09T19:59:36.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:36 smithi038 ceph-mon[92422]: pgmap v10158: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 132 KiB/s wr, 9 op/s 2023-03-09T19:59:38.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:37 smithi122 ceph-mon[93869]: pgmap v10159: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 66 KiB/s wr, 6 op/s 2023-03-09T19:59:38.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:37 smithi181 ceph-mon[94189]: pgmap v10159: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 66 KiB/s wr, 6 op/s 2023-03-09T19:59:38.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:37 smithi038 ceph-mon[92422]: pgmap v10159: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 66 KiB/s wr, 6 op/s 2023-03-09T19:59:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18550.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:59:39.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18088.536285 secs 2023-03-09T19:59:39.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18088.536285 secs 2023-03-09T19:59:39.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18088.536285 secs 2023-03-09T19:59:40.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:39 smithi122 ceph-mon[93869]: pgmap v10160: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 66 KiB/s wr, 6 op/s 2023-03-09T19:59:40.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:39 smithi181 ceph-mon[94189]: pgmap v10160: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 66 KiB/s wr, 6 op/s 2023-03-09T19:59:40.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:39 smithi038 ceph-mon[92422]: pgmap v10160: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 66 KiB/s wr, 6 op/s 2023-03-09T19:59:41.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18091.221394 secs 2023-03-09T19:59:41.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18091.221394 secs 2023-03-09T19:59:41.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18091.221394 secs 2023-03-09T19:59:42.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:41 smithi122 ceph-mon[93869]: pgmap v10161: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 136 KiB/s wr, 8 op/s 2023-03-09T19:59:42.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:41 smithi181 ceph-mon[94189]: pgmap v10161: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 136 KiB/s wr, 8 op/s 2023-03-09T19:59:42.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:41 smithi038 ceph-mon[92422]: pgmap v10161: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 136 KiB/s wr, 8 op/s 2023-03-09T19:59:43.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18093.536402 secs 2023-03-09T19:59:43.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18093.536402 secs 2023-03-09T19:59:43.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18093.536402 secs 2023-03-09T19:59:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18555.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:59:44.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:44 smithi122 ceph-mon[93869]: pgmap v10162: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 137 KiB/s wr, 6 op/s 2023-03-09T19:59:44.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:44 smithi181 ceph-mon[94189]: pgmap v10162: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 137 KiB/s wr, 6 op/s 2023-03-09T19:59:44.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:44 smithi038 ceph-mon[92422]: pgmap v10162: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 137 KiB/s wr, 6 op/s 2023-03-09T19:59:45.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18096.221455 secs 2023-03-09T19:59:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18096.221455 secs 2023-03-09T19:59:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18096.221455 secs 2023-03-09T19:59:46.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:46 smithi122 ceph-mon[93869]: pgmap v10163: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 274 KiB/s wr, 9 op/s 2023-03-09T19:59:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:46 smithi181 ceph-mon[94189]: pgmap v10163: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 274 KiB/s wr, 9 op/s 2023-03-09T19:59:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:46 smithi038 ceph-mon[92422]: pgmap v10163: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 274 KiB/s wr, 9 op/s 2023-03-09T19:59:48.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:47 smithi122 ceph-mon[93869]: pgmap v10164: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 209 KiB/s wr, 6 op/s 2023-03-09T19:59:48.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:47 smithi181 ceph-mon[94189]: pgmap v10164: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 209 KiB/s wr, 6 op/s 2023-03-09T19:59:48.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:47 smithi038 ceph-mon[92422]: pgmap v10164: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 209 KiB/s wr, 6 op/s 2023-03-09T19:59:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18560.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:59:49.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:48 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:59:49.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:48 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:59:49.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:48 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T19:59:50.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:49 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:59:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:59:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:59:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:49 smithi122 ceph-mon[93869]: pgmap v10165: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 223 KiB/s wr, 7 op/s 2023-03-09T19:59:50.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:49 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:59:50.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:59:50.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:59:50.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:49 smithi181 ceph-mon[94189]: pgmap v10165: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 223 KiB/s wr, 7 op/s 2023-03-09T19:59:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:49 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T19:59:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T19:59:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:59:50.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:49 smithi038 ceph-mon[92422]: pgmap v10165: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 223 KiB/s wr, 7 op/s 2023-03-09T19:59:51.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18101.221514 secs 2023-03-09T19:59:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:50 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:59:51.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18101.221514 secs 2023-03-09T19:59:51.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:50 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:59:51.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18101.221514 secs 2023-03-09T19:59:51.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:50 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T19:59:52.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:59:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:59:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:59:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:59:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:59:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:51 smithi122 ceph-mon[93869]: pgmap v10166: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 409 KiB/s wr, 10 op/s 2023-03-09T19:59:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:59:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:59:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:59:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:59:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:59:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:51 smithi181 ceph-mon[94189]: pgmap v10166: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 409 KiB/s wr, 10 op/s 2023-03-09T19:59:52.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T19:59:52.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:59:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T19:59:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T19:59:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T19:59:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:51 smithi038 ceph-mon[92422]: pgmap v10166: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 409 KiB/s wr, 10 op/s 2023-03-09T19:59:53.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:59:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:59:53.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:59:53.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:59:53.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T19:59:53.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T19:59:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18565.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T19:59:54.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:54 smithi181 ceph-mon[94189]: pgmap v10167: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 339 KiB/s wr, 9 op/s 2023-03-09T19:59:54.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:54 smithi038 ceph-mon[92422]: pgmap v10167: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 339 KiB/s wr, 9 op/s 2023-03-09T19:59:54.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:54 smithi122 ceph-mon[93869]: pgmap v10167: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 339 KiB/s wr, 9 op/s 2023-03-09T19:59:55.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18106.221574 secs 2023-03-09T19:59:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18106.221574 secs 2023-03-09T19:59:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18106.221574 secs 2023-03-09T19:59:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:56 smithi122 ceph-mon[93869]: pgmap v10168: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 566 KiB/s wr, 13 op/s 2023-03-09T19:59:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:56 smithi181 ceph-mon[94189]: pgmap v10168: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 566 KiB/s wr, 13 op/s 2023-03-09T19:59:56.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:56 smithi038 ceph-mon[92422]: pgmap v10168: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 566 KiB/s wr, 13 op/s 2023-03-09T19:59:58.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:57 smithi122 ceph-mon[93869]: pgmap v10169: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 443 KiB/s rd, 428 KiB/s wr, 10 op/s 2023-03-09T19:59:58.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:57 smithi181 ceph-mon[94189]: pgmap v10169: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 443 KiB/s rd, 428 KiB/s wr, 10 op/s 2023-03-09T19:59:58.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:57 smithi038 ceph-mon[92422]: pgmap v10169: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 443 KiB/s rd, 428 KiB/s wr, 10 op/s 2023-03-09T19:59:58.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18570.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:00:00.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:00:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:00:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 19:59:59 smithi122 ceph-mon[93869]: pgmap v10170: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 457 KiB/s rd, 443 KiB/s wr, 11 op/s 2023-03-09T20:00:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:00:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:00:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 19:59:59 smithi181 ceph-mon[94189]: pgmap v10170: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 457 KiB/s rd, 443 KiB/s wr, 11 op/s 2023-03-09T20:00:00.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:00:00.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:00:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 19:59:59 smithi038 ceph-mon[92422]: pgmap v10170: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 457 KiB/s rd, 443 KiB/s wr, 11 op/s 2023-03-09T20:00:01.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18111.221635 secs 2023-03-09T20:00:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:00:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:00:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:00 smithi122 ceph-mon[93869]: Health detail: HEALTH_WARN 1 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-09T20:00:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:00 smithi122 ceph-mon[93869]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2023-03-09T20:00:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:00 smithi122 ceph-mon[93869]: mds.h(mds.1): 2 slow requests are blocked > 180 secs 2023-03-09T20:00:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:00 smithi122 ceph-mon[93869]: [WRN] MDS_TRIM: 2 MDSs behind on trimming 2023-03-09T20:00:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:00 smithi122 ceph-mon[93869]: mds.e(mds.0): Behind on trimming (297/128) max_segments: 128, num_segments: 297 2023-03-09T20:00:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:00 smithi122 ceph-mon[93869]: mds.b(mds.0): Behind on trimming (297/128) max_segments: 128, num_segments: 297 2023-03-09T20:00:01.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18111.221635 secs 2023-03-09T20:00:01.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:00:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:00:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:00 smithi181 ceph-mon[94189]: Health detail: HEALTH_WARN 1 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-09T20:00:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:00 smithi181 ceph-mon[94189]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2023-03-09T20:00:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:00 smithi181 ceph-mon[94189]: mds.h(mds.1): 2 slow requests are blocked > 180 secs 2023-03-09T20:00:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:00 smithi181 ceph-mon[94189]: [WRN] MDS_TRIM: 2 MDSs behind on trimming 2023-03-09T20:00:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:00 smithi181 ceph-mon[94189]: mds.e(mds.0): Behind on trimming (297/128) max_segments: 128, num_segments: 297 2023-03-09T20:00:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:00 smithi181 ceph-mon[94189]: mds.b(mds.0): Behind on trimming (297/128) max_segments: 128, num_segments: 297 2023-03-09T20:00:01.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18111.221635 secs 2023-03-09T20:00:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:00:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:00:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:00 smithi038 ceph-mon[92422]: Health detail: HEALTH_WARN 1 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-09T20:00:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:00 smithi038 ceph-mon[92422]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2023-03-09T20:00:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:00 smithi038 ceph-mon[92422]: mds.h(mds.1): 2 slow requests are blocked > 180 secs 2023-03-09T20:00:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:00 smithi038 ceph-mon[92422]: [WRN] MDS_TRIM: 2 MDSs behind on trimming 2023-03-09T20:00:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:00 smithi038 ceph-mon[92422]: mds.e(mds.0): Behind on trimming (297/128) max_segments: 128, num_segments: 297 2023-03-09T20:00:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:00 smithi038 ceph-mon[92422]: mds.b(mds.0): Behind on trimming (297/128) max_segments: 128, num_segments: 297 2023-03-09T20:00:02.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:01 smithi122 ceph-mon[93869]: pgmap v10171: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 452 KiB/s rd, 448 KiB/s wr, 13 op/s 2023-03-09T20:00:02.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:01 smithi181 ceph-mon[94189]: pgmap v10171: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 452 KiB/s rd, 448 KiB/s wr, 13 op/s 2023-03-09T20:00:02.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:01 smithi038 ceph-mon[92422]: pgmap v10171: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 452 KiB/s rd, 448 KiB/s wr, 13 op/s 2023-03-09T20:00:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18575.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:00:04.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:04 smithi181 ceph-mon[94189]: pgmap v10172: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 283 KiB/s wr, 22 op/s 2023-03-09T20:00:04.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:04 smithi038 ceph-mon[92422]: pgmap v10172: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 283 KiB/s wr, 22 op/s 2023-03-09T20:00:04.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:04 smithi122 ceph-mon[93869]: pgmap v10172: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 283 KiB/s wr, 22 op/s 2023-03-09T20:00:05.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18116.221694 secs 2023-03-09T20:00:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18116.221694 secs 2023-03-09T20:00:05.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18116.221694 secs 2023-03-09T20:00:06.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:06 smithi122 ceph-mon[93869]: pgmap v10173: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 521 KiB/s rd, 539 KiB/s wr, 28 op/s 2023-03-09T20:00:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:06 smithi181 ceph-mon[94189]: pgmap v10173: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 521 KiB/s rd, 539 KiB/s wr, 28 op/s 2023-03-09T20:00:06.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:06 smithi038 ceph-mon[92422]: pgmap v10173: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 521 KiB/s rd, 539 KiB/s wr, 28 op/s 2023-03-09T20:00:08.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:07 smithi122 ceph-mon[93869]: pgmap v10174: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 312 KiB/s wr, 24 op/s 2023-03-09T20:00:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:07 smithi181 ceph-mon[94189]: pgmap v10174: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 312 KiB/s wr, 24 op/s 2023-03-09T20:00:08.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:07 smithi038 ceph-mon[92422]: pgmap v10174: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 312 KiB/s wr, 24 op/s 2023-03-09T20:00:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18580.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:00:09.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18118.537025 secs 2023-03-09T20:00:09.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18118.537025 secs 2023-03-09T20:00:09.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18118.537025 secs 2023-03-09T20:00:10.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:09 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:00:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:00:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:00:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:09 smithi122 ceph-mon[93869]: pgmap v10175: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 326 KiB/s wr, 24 op/s 2023-03-09T20:00:10.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:09 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:00:10.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:00:10.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:00:10.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:09 smithi181 ceph-mon[94189]: pgmap v10175: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 326 KiB/s wr, 24 op/s 2023-03-09T20:00:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:09 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:00:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:00:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:00:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:09 smithi038 ceph-mon[92422]: pgmap v10175: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 326 KiB/s wr, 24 op/s 2023-03-09T20:00:11.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18121.221752 secs 2023-03-09T20:00:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:10 smithi122 ceph-mon[93869]: osdmap e849: 12 total, 12 up, 12 in 2023-03-09T20:00:11.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18121.221752 secs 2023-03-09T20:00:11.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:10 smithi181 ceph-mon[94189]: osdmap e849: 12 total, 12 up, 12 in 2023-03-09T20:00:11.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18121.221752 secs 2023-03-09T20:00:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:10 smithi038 ceph-mon[92422]: osdmap e849: 12 total, 12 up, 12 in 2023-03-09T20:00:12.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:11 smithi122 ceph-mon[93869]: pgmap v10177: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 364 KiB/s wr, 27 op/s 2023-03-09T20:00:12.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:11 smithi181 ceph-mon[94189]: pgmap v10177: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 364 KiB/s wr, 27 op/s 2023-03-09T20:00:12.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:11 smithi038 ceph-mon[92422]: pgmap v10177: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 364 KiB/s wr, 27 op/s 2023-03-09T20:00:13.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18123.537141 secs 2023-03-09T20:00:13.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18123.537141 secs 2023-03-09T20:00:13.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18123.537141 secs 2023-03-09T20:00:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18585.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:00:14.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:14 smithi038 ceph-mon[92422]: pgmap v10178: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 380 KiB/s wr, 13 op/s 2023-03-09T20:00:14.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:14 smithi122 ceph-mon[93869]: pgmap v10178: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 380 KiB/s wr, 13 op/s 2023-03-09T20:00:14.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:14 smithi181 ceph-mon[94189]: pgmap v10178: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 380 KiB/s wr, 13 op/s 2023-03-09T20:00:15.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18126.221812 secs 2023-03-09T20:00:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:15 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:00:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:15 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:00:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18126.221812 secs 2023-03-09T20:00:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:15 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:00:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:15 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:00:15.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18126.221812 secs 2023-03-09T20:00:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:15 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:00:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:15 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:00:16.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:16 smithi122 ceph-mon[93869]: pgmap v10179: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 8 op/s 2023-03-09T20:00:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:16 smithi181 ceph-mon[94189]: pgmap v10179: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 8 op/s 2023-03-09T20:00:16.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:16 smithi038 ceph-mon[92422]: pgmap v10179: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 8 op/s 2023-03-09T20:00:18.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:17 smithi122 ceph-mon[93869]: pgmap v10180: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 71 KiB/s wr, 8 op/s 2023-03-09T20:00:18.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:17 smithi181 ceph-mon[94189]: pgmap v10180: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 71 KiB/s wr, 8 op/s 2023-03-09T20:00:18.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:17 smithi038 ceph-mon[92422]: pgmap v10180: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 71 KiB/s wr, 8 op/s 2023-03-09T20:00:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18590.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:00:19.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18128.537233 secs 2023-03-09T20:00:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:18 smithi122 ceph-mon[93869]: osdmap e850: 12 total, 12 up, 12 in 2023-03-09T20:00:19.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18128.537233 secs 2023-03-09T20:00:19.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:18 smithi181 ceph-mon[94189]: osdmap e850: 12 total, 12 up, 12 in 2023-03-09T20:00:19.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18128.537233 secs 2023-03-09T20:00:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:18 smithi038 ceph-mon[92422]: osdmap e850: 12 total, 12 up, 12 in 2023-03-09T20:00:20.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:19 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:00:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:19 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:00:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:19 smithi122 ceph-mon[93869]: pgmap v10182: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 9 op/s 2023-03-09T20:00:20.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:19 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:00:20.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:19 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:00:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:19 smithi181 ceph-mon[94189]: pgmap v10182: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 9 op/s 2023-03-09T20:00:20.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:19 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:00:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:19 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:00:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:19 smithi038 ceph-mon[92422]: pgmap v10182: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 9 op/s 2023-03-09T20:00:21.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18131.221873 secs 2023-03-09T20:00:21.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18131.221873 secs 2023-03-09T20:00:21.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18131.221873 secs 2023-03-09T20:00:22.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:21 smithi122 ceph-mon[93869]: pgmap v10183: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 8 op/s 2023-03-09T20:00:22.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:21 smithi181 ceph-mon[94189]: pgmap v10183: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 8 op/s 2023-03-09T20:00:22.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:21 smithi038 ceph-mon[92422]: pgmap v10183: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 8 op/s 2023-03-09T20:00:23.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18133.537345 secs 2023-03-09T20:00:23.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18133.537345 secs 2023-03-09T20:00:23.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18133.537345 secs 2023-03-09T20:00:23.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18595.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:00:24.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:24 smithi038 ceph-mon[92422]: pgmap v10184: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 KiB/s wr, 8 op/s 2023-03-09T20:00:24.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:24 smithi122 ceph-mon[93869]: pgmap v10184: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 KiB/s wr, 8 op/s 2023-03-09T20:00:24.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:24 smithi181 ceph-mon[94189]: pgmap v10184: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 17 KiB/s wr, 8 op/s 2023-03-09T20:00:25.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18136.221955 secs 2023-03-09T20:00:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18136.221955 secs 2023-03-09T20:00:25.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18136.221955 secs 2023-03-09T20:00:26.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:26 smithi122 ceph-mon[93869]: pgmap v10185: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 8 op/s 2023-03-09T20:00:26.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:26 smithi181 ceph-mon[94189]: pgmap v10185: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 8 op/s 2023-03-09T20:00:26.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:26 smithi038 ceph-mon[92422]: pgmap v10185: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 8 op/s 2023-03-09T20:00:28.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:27 smithi122 ceph-mon[93869]: pgmap v10186: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 8 op/s 2023-03-09T20:00:28.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:27 smithi181 ceph-mon[94189]: pgmap v10186: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 8 op/s 2023-03-09T20:00:28.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:27 smithi038 ceph-mon[92422]: pgmap v10186: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 18 KiB/s wr, 8 op/s 2023-03-09T20:00:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18600.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:00:29.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18138.537493 secs 2023-03-09T20:00:29.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18138.537493 secs 2023-03-09T20:00:29.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18138.537493 secs 2023-03-09T20:00:30.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:29 smithi122 ceph-mon[93869]: pgmap v10187: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 87 KiB/s wr, 8 op/s 2023-03-09T20:00:30.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:29 smithi181 ceph-mon[94189]: pgmap v10187: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 87 KiB/s wr, 8 op/s 2023-03-09T20:00:30.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:29 smithi038 ceph-mon[92422]: pgmap v10187: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 87 KiB/s wr, 8 op/s 2023-03-09T20:00:31.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18141.222015 secs 2023-03-09T20:00:31.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18141.222015 secs 2023-03-09T20:00:31.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18141.222015 secs 2023-03-09T20:00:32.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:31 smithi122 ceph-mon[93869]: pgmap v10188: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 68 KiB/s wr, 9 op/s 2023-03-09T20:00:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:31 smithi181 ceph-mon[94189]: pgmap v10188: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 68 KiB/s wr, 9 op/s 2023-03-09T20:00:32.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:31 smithi038 ceph-mon[92422]: pgmap v10188: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 68 KiB/s wr, 9 op/s 2023-03-09T20:00:33.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18143.537605 secs 2023-03-09T20:00:33.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18143.537605 secs 2023-03-09T20:00:33.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18143.537605 secs 2023-03-09T20:00:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18605.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:00:34.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:34 smithi038 ceph-mon[92422]: pgmap v10189: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 67 KiB/s wr, 7 op/s 2023-03-09T20:00:34.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:34 smithi122 ceph-mon[93869]: pgmap v10189: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 67 KiB/s wr, 7 op/s 2023-03-09T20:00:34.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:34 smithi181 ceph-mon[94189]: pgmap v10189: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 67 KiB/s wr, 7 op/s 2023-03-09T20:00:35.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18146.222077 secs 2023-03-09T20:00:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18146.222077 secs 2023-03-09T20:00:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18146.222077 secs 2023-03-09T20:00:36.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:36 smithi122 ceph-mon[93869]: pgmap v10190: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 68 KiB/s wr, 9 op/s 2023-03-09T20:00:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:36 smithi181 ceph-mon[94189]: pgmap v10190: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 68 KiB/s wr, 9 op/s 2023-03-09T20:00:36.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:36 smithi038 ceph-mon[92422]: pgmap v10190: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 68 KiB/s wr, 9 op/s 2023-03-09T20:00:38.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:37 smithi122 ceph-mon[93869]: pgmap v10191: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 67 KiB/s wr, 6 op/s 2023-03-09T20:00:38.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:37 smithi181 ceph-mon[94189]: pgmap v10191: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 67 KiB/s wr, 6 op/s 2023-03-09T20:00:38.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:37 smithi038 ceph-mon[92422]: pgmap v10191: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 67 KiB/s wr, 6 op/s 2023-03-09T20:00:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18610.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:00:40.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:39 smithi122 ceph-mon[93869]: pgmap v10192: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 218 KiB/s wr, 7 op/s 2023-03-09T20:00:40.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:39 smithi181 ceph-mon[94189]: pgmap v10192: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 218 KiB/s wr, 7 op/s 2023-03-09T20:00:40.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:39 smithi038 ceph-mon[92422]: pgmap v10192: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 218 KiB/s wr, 7 op/s 2023-03-09T20:00:41.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18151.222150 secs 2023-03-09T20:00:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:40 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:00:41.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18151.222150 secs 2023-03-09T20:00:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:40 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:00:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18151.222150 secs 2023-03-09T20:00:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:40 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:00:42.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:41 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:00:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:00:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:00:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:41 smithi122 ceph-mon[93869]: pgmap v10193: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 152 KiB/s wr, 8 op/s 2023-03-09T20:00:42.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:41 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:00:42.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:00:42.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:00:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:41 smithi181 ceph-mon[94189]: pgmap v10193: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 152 KiB/s wr, 8 op/s 2023-03-09T20:00:42.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:41 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:00:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:00:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:00:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:41 smithi038 ceph-mon[92422]: pgmap v10193: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 152 KiB/s wr, 8 op/s 2023-03-09T20:00:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18615.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:00:44.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:44 smithi038 ceph-mon[92422]: pgmap v10194: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 169 KiB/s wr, 7 op/s 2023-03-09T20:00:44.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:44 smithi122 ceph-mon[93869]: pgmap v10194: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 169 KiB/s wr, 7 op/s 2023-03-09T20:00:44.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:44 smithi181 ceph-mon[94189]: pgmap v10194: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 169 KiB/s wr, 7 op/s 2023-03-09T20:00:45.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18156.222208 secs 2023-03-09T20:00:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18156.222208 secs 2023-03-09T20:00:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18156.222208 secs 2023-03-09T20:00:46.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:46 smithi122 ceph-mon[93869]: pgmap v10195: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 184 KiB/s wr, 9 op/s 2023-03-09T20:00:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:46 smithi181 ceph-mon[94189]: pgmap v10195: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 184 KiB/s wr, 9 op/s 2023-03-09T20:00:46.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:46 smithi038 ceph-mon[92422]: pgmap v10195: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 184 KiB/s wr, 9 op/s 2023-03-09T20:00:48.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:47 smithi122 ceph-mon[93869]: pgmap v10196: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 220 KiB/s wr, 7 op/s 2023-03-09T20:00:48.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:47 smithi181 ceph-mon[94189]: pgmap v10196: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 220 KiB/s wr, 7 op/s 2023-03-09T20:00:48.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:47 smithi038 ceph-mon[92422]: pgmap v10196: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 220 KiB/s wr, 7 op/s 2023-03-09T20:00:48.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18620.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:00:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:48 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 18158.537938 secs 2023-03-09T20:00:49.235 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:48 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 18158.537938 secs 2023-03-09T20:00:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:48 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 18158.537938 secs 2023-03-09T20:00:49.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:49 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:00:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:00:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:00:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:49 smithi122 ceph-mon[93869]: pgmap v10197: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 322 KiB/s wr, 9 op/s 2023-03-09T20:00:50.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:49 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:00:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:00:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:00:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:49 smithi181 ceph-mon[94189]: pgmap v10197: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 322 KiB/s wr, 9 op/s 2023-03-09T20:00:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:49 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:00:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:00:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:00:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:49 smithi038 ceph-mon[92422]: pgmap v10197: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 322 KiB/s wr, 9 op/s 2023-03-09T20:00:50.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18161.222270 secs 2023-03-09T20:00:51.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18161.222270 secs 2023-03-09T20:00:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18161.222270 secs 2023-03-09T20:00:52.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:00:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:00:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:00:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:51 smithi122 ceph-mon[93869]: pgmap v10198: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 318 KiB/s rd, 303 KiB/s wr, 11 op/s 2023-03-09T20:00:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:00:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:00:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:00:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:51 smithi181 ceph-mon[94189]: pgmap v10198: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 318 KiB/s rd, 303 KiB/s wr, 11 op/s 2023-03-09T20:00:52.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:00:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:00:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:00:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:51 smithi038 ceph-mon[92422]: pgmap v10198: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 318 KiB/s rd, 303 KiB/s wr, 11 op/s 2023-03-09T20:00:53.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:00:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:00:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:00:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:00:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:00:53.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:00:53.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:00:53.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:00:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:00:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:00:53.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:00:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:00:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:00:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:00:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:00:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18625.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:00:54.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:53 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 18163.538023 secs 2023-03-09T20:00:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:53 smithi122 ceph-mon[93869]: pgmap v10199: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 303 KiB/s wr, 10 op/s 2023-03-09T20:00:54.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:53 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 18163.538023 secs 2023-03-09T20:00:54.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:53 smithi181 ceph-mon[94189]: pgmap v10199: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 303 KiB/s wr, 10 op/s 2023-03-09T20:00:54.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:53 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 18163.538023 secs 2023-03-09T20:00:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:53 smithi038 ceph-mon[92422]: pgmap v10199: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 303 KiB/s wr, 10 op/s 2023-03-09T20:00:55.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18166.222355 secs 2023-03-09T20:00:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18166.222355 secs 2023-03-09T20:00:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18166.222355 secs 2023-03-09T20:00:56.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:56 smithi122 ceph-mon[93869]: pgmap v10200: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 286 KiB/s wr, 12 op/s 2023-03-09T20:00:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:56 smithi181 ceph-mon[94189]: pgmap v10200: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 286 KiB/s wr, 12 op/s 2023-03-09T20:00:56.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:56 smithi038 ceph-mon[92422]: pgmap v10200: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 286 KiB/s wr, 12 op/s 2023-03-09T20:00:58.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:57 smithi122 ceph-mon[93869]: pgmap v10201: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 421 KiB/s wr, 10 op/s 2023-03-09T20:00:58.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:57 smithi181 ceph-mon[94189]: pgmap v10201: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 421 KiB/s wr, 10 op/s 2023-03-09T20:00:58.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:57 smithi038 ceph-mon[92422]: pgmap v10201: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 421 KiB/s wr, 10 op/s 2023-03-09T20:00:58.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18630.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:00:59.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:58 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 18168.538178 secs 2023-03-09T20:00:59.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:58 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 18168.538178 secs 2023-03-09T20:00:59.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:58 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 18168.538178 secs 2023-03-09T20:01:00.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:01:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:01:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:00:59 smithi122 ceph-mon[93869]: pgmap v10202: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 412 KiB/s wr, 11 op/s 2023-03-09T20:01:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:01:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:01:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:00:59 smithi181 ceph-mon[94189]: pgmap v10202: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 412 KiB/s wr, 11 op/s 2023-03-09T20:01:00.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:01:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:01:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:00:59 smithi038 ceph-mon[92422]: pgmap v10202: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 412 KiB/s wr, 11 op/s 2023-03-09T20:01:01.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18171.222416 secs 2023-03-09T20:01:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:01:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:01:01.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18171.222416 secs 2023-03-09T20:01:01.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:01:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:01:01.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18171.222416 secs 2023-03-09T20:01:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:01:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:01:01.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:01 smithi122 ceph-mon[93869]: pgmap v10203: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 522 KiB/s rd, 527 KiB/s wr, 15 op/s 2023-03-09T20:01:02.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:01 smithi181 ceph-mon[94189]: pgmap v10203: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 522 KiB/s rd, 527 KiB/s wr, 15 op/s 2023-03-09T20:01:02.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:01 smithi038 ceph-mon[92422]: pgmap v10203: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 522 KiB/s rd, 527 KiB/s wr, 15 op/s 2023-03-09T20:01:03.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18173.538292 secs 2023-03-09T20:01:03.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18173.538292 secs 2023-03-09T20:01:03.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18173.538292 secs 2023-03-09T20:01:03.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18635.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:01:04.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:04 smithi122 ceph-mon[93869]: pgmap v10204: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 417 KiB/s wr, 25 op/s 2023-03-09T20:01:04.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:04 smithi181 ceph-mon[94189]: pgmap v10204: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 417 KiB/s wr, 25 op/s 2023-03-09T20:01:04.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:04 smithi038 ceph-mon[92422]: pgmap v10204: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 417 KiB/s wr, 25 op/s 2023-03-09T20:01:05.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:05 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:01:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:05 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:01:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:05 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:01:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:05 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:01:05.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:05 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:01:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:05 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:01:06.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18176.222473 secs 2023-03-09T20:01:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:06 smithi122 ceph-mon[93869]: pgmap v10205: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 398 KiB/s rd, 417 KiB/s wr, 26 op/s 2023-03-09T20:01:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18176.222473 secs 2023-03-09T20:01:06.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:06 smithi181 ceph-mon[94189]: pgmap v10205: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 398 KiB/s rd, 417 KiB/s wr, 26 op/s 2023-03-09T20:01:06.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18176.222473 secs 2023-03-09T20:01:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:06 smithi038 ceph-mon[92422]: pgmap v10205: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 398 KiB/s rd, 417 KiB/s wr, 26 op/s 2023-03-09T20:01:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:07 smithi122 ceph-mon[93869]: pgmap v10206: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 445 KiB/s rd, 479 KiB/s wr, 24 op/s 2023-03-09T20:01:08.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:07 smithi181 ceph-mon[94189]: pgmap v10206: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 445 KiB/s rd, 479 KiB/s wr, 24 op/s 2023-03-09T20:01:08.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:07 smithi038 ceph-mon[92422]: pgmap v10206: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 445 KiB/s rd, 479 KiB/s wr, 24 op/s 2023-03-09T20:01:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18640.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:01:09.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18178.538419 secs 2023-03-09T20:01:09.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18178.538419 secs 2023-03-09T20:01:09.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18178.538419 secs 2023-03-09T20:01:10.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:09 smithi122 ceph-mon[93869]: pgmap v10207: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 310 KiB/s rd, 344 KiB/s wr, 24 op/s 2023-03-09T20:01:10.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:09 smithi181 ceph-mon[94189]: pgmap v10207: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 310 KiB/s rd, 344 KiB/s wr, 24 op/s 2023-03-09T20:01:10.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:09 smithi038 ceph-mon[92422]: pgmap v10207: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 310 KiB/s rd, 344 KiB/s wr, 24 op/s 2023-03-09T20:01:11.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18181.222536 secs 2023-03-09T20:01:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:10 smithi122 ceph-mon[93869]: osdmap e851: 12 total, 12 up, 12 in 2023-03-09T20:01:11.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18181.222536 secs 2023-03-09T20:01:11.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:10 smithi181 ceph-mon[94189]: osdmap e851: 12 total, 12 up, 12 in 2023-03-09T20:01:11.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18181.222536 secs 2023-03-09T20:01:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:10 smithi038 ceph-mon[92422]: osdmap e851: 12 total, 12 up, 12 in 2023-03-09T20:01:12.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:11 smithi122 ceph-mon[93869]: pgmap v10209: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 223 KiB/s wr, 23 op/s 2023-03-09T20:01:12.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:11 smithi181 ceph-mon[94189]: pgmap v10209: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 223 KiB/s wr, 23 op/s 2023-03-09T20:01:12.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:11 smithi038 ceph-mon[92422]: pgmap v10209: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 223 KiB/s wr, 23 op/s 2023-03-09T20:01:13.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18183.538515 secs 2023-03-09T20:01:13.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18183.538515 secs 2023-03-09T20:01:13.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18183.538515 secs 2023-03-09T20:01:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18645.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:01:14.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:14 smithi122 ceph-mon[93869]: pgmap v10210: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 198 KiB/s wr, 8 op/s 2023-03-09T20:01:14.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:14 smithi181 ceph-mon[94189]: pgmap v10210: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 198 KiB/s wr, 8 op/s 2023-03-09T20:01:14.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:14 smithi038 ceph-mon[92422]: pgmap v10210: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 198 KiB/s wr, 8 op/s 2023-03-09T20:01:15.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18186.222595 secs 2023-03-09T20:01:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18186.222595 secs 2023-03-09T20:01:15.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18186.222595 secs 2023-03-09T20:01:16.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:16 smithi122 ceph-mon[93869]: pgmap v10211: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 242 KiB/s wr, 9 op/s 2023-03-09T20:01:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:16 smithi181 ceph-mon[94189]: pgmap v10211: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 242 KiB/s wr, 9 op/s 2023-03-09T20:01:16.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:16 smithi038 ceph-mon[92422]: pgmap v10211: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 242 KiB/s wr, 9 op/s 2023-03-09T20:01:18.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:17 smithi122 ceph-mon[93869]: pgmap v10212: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 167 KiB/s wr, 8 op/s 2023-03-09T20:01:18.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:17 smithi181 ceph-mon[94189]: pgmap v10212: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 167 KiB/s wr, 8 op/s 2023-03-09T20:01:18.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:17 smithi038 ceph-mon[92422]: pgmap v10212: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 167 KiB/s wr, 8 op/s 2023-03-09T20:01:18.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18650.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:01:19.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18188.538632 secs 2023-03-09T20:01:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:18 smithi122 ceph-mon[93869]: osdmap e852: 12 total, 12 up, 12 in 2023-03-09T20:01:19.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18188.538632 secs 2023-03-09T20:01:19.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:18 smithi181 ceph-mon[94189]: osdmap e852: 12 total, 12 up, 12 in 2023-03-09T20:01:19.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18188.538632 secs 2023-03-09T20:01:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:18 smithi038 ceph-mon[92422]: osdmap e852: 12 total, 12 up, 12 in 2023-03-09T20:01:20.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:19 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:01:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:19 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:01:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:19 smithi122 ceph-mon[93869]: pgmap v10214: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 179 KiB/s wr, 8 op/s 2023-03-09T20:01:20.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:19 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:01:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:19 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:01:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:19 smithi181 ceph-mon[94189]: pgmap v10214: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 179 KiB/s wr, 8 op/s 2023-03-09T20:01:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:19 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:01:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:19 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:01:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:19 smithi038 ceph-mon[92422]: pgmap v10214: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 179 KiB/s wr, 8 op/s 2023-03-09T20:01:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18191.222680 secs 2023-03-09T20:01:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:01:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:01:21.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18191.222680 secs 2023-03-09T20:01:21.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:01:21.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:01:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18191.222680 secs 2023-03-09T20:01:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:01:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:01:22.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:21 smithi122 ceph-mon[93869]: pgmap v10215: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 73 KiB/s wr, 8 op/s 2023-03-09T20:01:22.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:21 smithi181 ceph-mon[94189]: pgmap v10215: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 73 KiB/s wr, 8 op/s 2023-03-09T20:01:22.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:21 smithi038 ceph-mon[92422]: pgmap v10215: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 73 KiB/s wr, 8 op/s 2023-03-09T20:01:23.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18193.538694 secs 2023-03-09T20:01:23.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18193.538694 secs 2023-03-09T20:01:23.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18193.538694 secs 2023-03-09T20:01:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18655.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:01:24.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:24 smithi122 ceph-mon[93869]: pgmap v10216: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 152 KiB/s wr, 9 op/s 2023-03-09T20:01:24.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:24 smithi181 ceph-mon[94189]: pgmap v10216: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 152 KiB/s wr, 9 op/s 2023-03-09T20:01:25.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:24 smithi038 ceph-mon[92422]: pgmap v10216: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 152 KiB/s wr, 9 op/s 2023-03-09T20:01:25.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18196.222765 secs 2023-03-09T20:01:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18196.222765 secs 2023-03-09T20:01:25.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18196.222765 secs 2023-03-09T20:01:26.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:26 smithi122 ceph-mon[93869]: pgmap v10217: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 108 KiB/s wr, 8 op/s 2023-03-09T20:01:26.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:26 smithi181 ceph-mon[94189]: pgmap v10217: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 108 KiB/s wr, 8 op/s 2023-03-09T20:01:26.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:26 smithi038 ceph-mon[92422]: pgmap v10217: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 108 KiB/s wr, 8 op/s 2023-03-09T20:01:28.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:27 smithi122 ceph-mon[93869]: pgmap v10218: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 108 KiB/s wr, 8 op/s 2023-03-09T20:01:28.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:27 smithi181 ceph-mon[94189]: pgmap v10218: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 108 KiB/s wr, 8 op/s 2023-03-09T20:01:28.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:27 smithi038 ceph-mon[92422]: pgmap v10218: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 108 KiB/s wr, 8 op/s 2023-03-09T20:01:28.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18660.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:01:29.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18198.538828 secs 2023-03-09T20:01:29.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18198.538828 secs 2023-03-09T20:01:29.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18198.538828 secs 2023-03-09T20:01:30.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:29 smithi122 ceph-mon[93869]: pgmap v10219: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 102 KiB/s wr, 8 op/s 2023-03-09T20:01:30.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:29 smithi181 ceph-mon[94189]: pgmap v10219: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 102 KiB/s wr, 8 op/s 2023-03-09T20:01:30.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:29 smithi038 ceph-mon[92422]: pgmap v10219: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 102 KiB/s wr, 8 op/s 2023-03-09T20:01:31.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18201.222846 secs 2023-03-09T20:01:31.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18201.222846 secs 2023-03-09T20:01:31.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18201.222846 secs 2023-03-09T20:01:32.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:31 smithi122 ceph-mon[93869]: pgmap v10220: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 96 KiB/s wr, 9 op/s 2023-03-09T20:01:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:31 smithi181 ceph-mon[94189]: pgmap v10220: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 96 KiB/s wr, 9 op/s 2023-03-09T20:01:32.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:31 smithi038 ceph-mon[92422]: pgmap v10220: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 96 KiB/s wr, 9 op/s 2023-03-09T20:01:33.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18203.538945 secs 2023-03-09T20:01:33.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18203.538945 secs 2023-03-09T20:01:33.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18203.538945 secs 2023-03-09T20:01:33.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18665.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:01:34.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:34 smithi122 ceph-mon[93869]: pgmap v10221: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 96 KiB/s wr, 8 op/s 2023-03-09T20:01:34.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:34 smithi181 ceph-mon[94189]: pgmap v10221: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 96 KiB/s wr, 8 op/s 2023-03-09T20:01:34.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:34 smithi038 ceph-mon[92422]: pgmap v10221: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 96 KiB/s wr, 8 op/s 2023-03-09T20:01:35.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18206.222905 secs 2023-03-09T20:01:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18206.222905 secs 2023-03-09T20:01:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18206.222905 secs 2023-03-09T20:01:36.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:36 smithi122 ceph-mon[93869]: pgmap v10222: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 30 KiB/s wr, 10 op/s 2023-03-09T20:01:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:36 smithi181 ceph-mon[94189]: pgmap v10222: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 30 KiB/s wr, 10 op/s 2023-03-09T20:01:36.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:36 smithi038 ceph-mon[92422]: pgmap v10222: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 30 KiB/s wr, 10 op/s 2023-03-09T20:01:38.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:37 smithi122 ceph-mon[93869]: pgmap v10223: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 7 op/s 2023-03-09T20:01:38.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:37 smithi181 ceph-mon[94189]: pgmap v10223: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 7 op/s 2023-03-09T20:01:38.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:37 smithi038 ceph-mon[92422]: pgmap v10223: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 7 op/s 2023-03-09T20:01:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18670.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:01:39.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18208.539060 secs 2023-03-09T20:01:39.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18208.539060 secs 2023-03-09T20:01:39.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18208.539060 secs 2023-03-09T20:01:40.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:39 smithi122 ceph-mon[93869]: pgmap v10224: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 231 KiB/s wr, 9 op/s 2023-03-09T20:01:40.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:39 smithi181 ceph-mon[94189]: pgmap v10224: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 231 KiB/s wr, 9 op/s 2023-03-09T20:01:40.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:39 smithi038 ceph-mon[92422]: pgmap v10224: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 231 KiB/s wr, 9 op/s 2023-03-09T20:01:41.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18211.222965 secs 2023-03-09T20:01:41.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18211.222965 secs 2023-03-09T20:01:41.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18211.222965 secs 2023-03-09T20:01:42.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:41 smithi122 ceph-mon[93869]: pgmap v10225: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 202 KiB/s wr, 10 op/s 2023-03-09T20:01:42.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:41 smithi181 ceph-mon[94189]: pgmap v10225: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 202 KiB/s wr, 10 op/s 2023-03-09T20:01:42.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:41 smithi038 ceph-mon[92422]: pgmap v10225: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 202 KiB/s wr, 10 op/s 2023-03-09T20:01:43.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18213.539182 secs 2023-03-09T20:01:43.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18213.539182 secs 2023-03-09T20:01:43.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18213.539182 secs 2023-03-09T20:01:43.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18675.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:01:45.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:45 smithi122 ceph-mon[93869]: pgmap v10226: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 254 KiB/s wr, 9 op/s 2023-03-09T20:01:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:45 smithi181 ceph-mon[94189]: pgmap v10226: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 254 KiB/s wr, 9 op/s 2023-03-09T20:01:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:45 smithi038 ceph-mon[92422]: pgmap v10226: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 254 KiB/s wr, 9 op/s 2023-03-09T20:01:46.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18216.223002 secs 2023-03-09T20:01:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18216.223002 secs 2023-03-09T20:01:46.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18216.223002 secs 2023-03-09T20:01:47.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:47 smithi122 ceph-mon[93869]: pgmap v10227: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 354 KiB/s wr, 12 op/s 2023-03-09T20:01:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:47 smithi181 ceph-mon[94189]: pgmap v10227: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 354 KiB/s wr, 12 op/s 2023-03-09T20:01:47.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:47 smithi038 ceph-mon[92422]: pgmap v10227: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 354 KiB/s wr, 12 op/s 2023-03-09T20:01:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18680.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:01:48.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18218.539321 secs 2023-03-09T20:01:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18218.539321 secs 2023-03-09T20:01:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18218.539321 secs 2023-03-09T20:01:49.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:49 smithi122 ceph-mon[93869]: pgmap v10228: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 472 KiB/s rd, 353 KiB/s wr, 9 op/s 2023-03-09T20:01:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:01:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:01:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:49 smithi181 ceph-mon[94189]: pgmap v10228: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 472 KiB/s rd, 353 KiB/s wr, 9 op/s 2023-03-09T20:01:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:01:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:01:49.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:49 smithi038 ceph-mon[92422]: pgmap v10228: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 472 KiB/s rd, 353 KiB/s wr, 9 op/s 2023-03-09T20:01:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:01:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:01:51.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:51 smithi122 ceph-mon[93869]: pgmap v10229: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 555 KiB/s rd, 368 KiB/s wr, 11 op/s 2023-03-09T20:01:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18221.223062 secs 2023-03-09T20:01:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:01:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:01:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:51 smithi181 ceph-mon[94189]: pgmap v10229: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 555 KiB/s rd, 368 KiB/s wr, 11 op/s 2023-03-09T20:01:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18221.223062 secs 2023-03-09T20:01:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:01:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:01:51.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:51 smithi038 ceph-mon[92422]: pgmap v10229: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 555 KiB/s rd, 368 KiB/s wr, 11 op/s 2023-03-09T20:01:51.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18221.223062 secs 2023-03-09T20:01:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:01:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:01:52.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:01:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:01:52.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:01:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18685.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:01:53.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:53 smithi122 ceph-mon[93869]: pgmap v10230: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 476 KiB/s rd, 396 KiB/s wr, 12 op/s 2023-03-09T20:01:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18223.539393 secs 2023-03-09T20:01:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:01:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:01:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:01:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:53 smithi181 ceph-mon[94189]: pgmap v10230: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 476 KiB/s rd, 396 KiB/s wr, 12 op/s 2023-03-09T20:01:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18223.539393 secs 2023-03-09T20:01:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:01:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:01:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:01:53.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:53 smithi038 ceph-mon[92422]: pgmap v10230: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 476 KiB/s rd, 396 KiB/s wr, 12 op/s 2023-03-09T20:01:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18223.539393 secs 2023-03-09T20:01:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:01:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:01:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:01:55.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:55 smithi122 ceph-mon[93869]: pgmap v10231: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 396 KiB/s wr, 10 op/s 2023-03-09T20:01:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:55 smithi181 ceph-mon[94189]: pgmap v10231: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 396 KiB/s wr, 10 op/s 2023-03-09T20:01:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:55 smithi038 ceph-mon[92422]: pgmap v10231: 129 pgs: 129 active+clean; 9.7 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 396 KiB/s wr, 10 op/s 2023-03-09T20:01:56.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18226.223122 secs 2023-03-09T20:01:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18226.223122 secs 2023-03-09T20:01:56.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18226.223122 secs 2023-03-09T20:01:57.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:57 smithi122 ceph-mon[93869]: pgmap v10232: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 386 KiB/s wr, 12 op/s 2023-03-09T20:01:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:57 smithi181 ceph-mon[94189]: pgmap v10232: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 386 KiB/s wr, 12 op/s 2023-03-09T20:01:57.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:57 smithi038 ceph-mon[92422]: pgmap v10232: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 386 KiB/s wr, 12 op/s 2023-03-09T20:01:58.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18690.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:01:58.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18228.539492 secs 2023-03-09T20:01:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18228.539492 secs 2023-03-09T20:01:58.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18228.539492 secs 2023-03-09T20:01:59.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:59 smithi122 ceph-mon[93869]: pgmap v10233: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 353 KiB/s wr, 10 op/s 2023-03-09T20:01:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:01:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:01:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:01:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:59 smithi181 ceph-mon[94189]: pgmap v10233: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 353 KiB/s wr, 10 op/s 2023-03-09T20:01:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:01:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:01:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:01:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:59 smithi038 ceph-mon[92422]: pgmap v10233: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 353 KiB/s wr, 10 op/s 2023-03-09T20:01:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:01:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:01:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:02:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:02:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:02:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:02:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:02:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:02:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18231.223180 secs 2023-03-09T20:02:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:02:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:02:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:02:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:02:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:02:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18231.223180 secs 2023-03-09T20:02:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:02:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:02:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:02:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:02:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:02:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18231.223180 secs 2023-03-09T20:02:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:02:01 smithi122 ceph-mon[93869]: pgmap v10234: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 353 KiB/s wr, 11 op/s 2023-03-09T20:02:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:02:01 smithi181 ceph-mon[94189]: pgmap v10234: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 353 KiB/s wr, 11 op/s 2023-03-09T20:02:01.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:02:01 smithi038 ceph-mon[92422]: pgmap v10234: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 353 KiB/s wr, 11 op/s 2023-03-09T20:02:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18695.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:02:03.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:02:03 smithi122 ceph-mon[93869]: pgmap v10235: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 360 KiB/s wr, 14 op/s 2023-03-09T20:02:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:02:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18233.539613 secs 2023-03-09T20:02:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:02:03 smithi181 ceph-mon[94189]: pgmap v10235: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 360 KiB/s wr, 14 op/s 2023-03-09T20:02:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:02:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18233.539613 secs 2023-03-09T20:02:03.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:02:03 smithi038 ceph-mon[92422]: pgmap v10235: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 360 KiB/s wr, 14 op/s 2023-03-09T20:02:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:02:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18233.539613 secs 2023-03-09T20:02:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:02:05 smithi122 ceph-mon[93869]: pgmap v10236: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 295 KiB/s wr, 24 op/s 2023-03-09T20:02:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:02:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18236.223239 secs 2023-03-09T20:02:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:02:05 smithi181 ceph-mon[94189]: pgmap v10236: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 295 KiB/s wr, 24 op/s 2023-03-09T20:02:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:02:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18236.223239 secs 2023-03-09T20:02:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:02:05 smithi038 ceph-mon[92422]: pgmap v10236: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 295 KiB/s wr, 24 op/s 2023-03-09T20:02:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:02:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18236.223239 secs 2023-03-09T20:02:07.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:02:07 smithi122 ceph-mon[93869]: pgmap v10237: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 295 KiB/s wr, 26 op/s 2023-03-09T20:02:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:02:07 smithi181 ceph-mon[94189]: pgmap v10237: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 295 KiB/s wr, 26 op/s 2023-03-09T20:02:07.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:02:07 smithi038 ceph-mon[92422]: pgmap v10237: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 295 KiB/s wr, 26 op/s 2023-03-09T20:02:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18700.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:02:08.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:02:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18238.539724 secs 2023-03-09T20:02:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:02:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18238.539724 secs 2023-03-09T20:02:08.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:02:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18238.539724 secs 2023-03-09T20:02:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:02:09 smithi122 ceph-mon[93869]: pgmap v10238: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 316 KiB/s wr, 23 op/s 2023-03-09T20:02:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:02:09 smithi181 ceph-mon[94189]: pgmap v10238: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 316 KiB/s wr, 23 op/s 2023-03-09T20:02:09.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:02:09 smithi038 ceph-mon[92422]: pgmap v10238: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 316 KiB/s wr, 23 op/s 2023-03-09T20:02:10.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:02:10 smithi122 ceph-mon[93869]: osdmap e853: 12 total, 12 up, 12 in 2023-03-09T20:02:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:02:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18241.227007 secs 2023-03-09T20:02:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:02:10 smithi181 ceph-mon[94189]: osdmap e853: 12 total, 12 up, 12 in 2023-03-09T20:02:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:02:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18241.227007 secs 2023-03-09T20:02:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:02:10 smithi038 ceph-mon[92422]: osdmap e853: 12 total, 12 up, 12 in 2023-03-09T20:02:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:02:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18241.227007 secs 2023-03-09T20:02:11.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:02:11 smithi122 ceph-mon[93869]: pgmap v10240: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 299 KiB/s wr, 26 op/s 2023-03-09T20:02:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:02:11 smithi181 ceph-mon[94189]: pgmap v10240: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 299 KiB/s wr, 26 op/s 2023-03-09T20:02:11.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:02:11 smithi038 ceph-mon[92422]: pgmap v10240: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 299 KiB/s wr, 26 op/s 2023-03-09T20:02:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18705.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:02:13.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:02:13 smithi122 ceph-mon[93869]: pgmap v10241: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 290 KiB/s wr, 24 op/s 2023-03-09T20:02:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:02:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18243.539840 secs 2023-03-09T20:02:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:02:13 smithi181 ceph-mon[94189]: pgmap v10241: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 290 KiB/s wr, 24 op/s 2023-03-09T20:02:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:02:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18243.539840 secs 2023-03-09T20:02:13.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:02:13 smithi038 ceph-mon[92422]: pgmap v10241: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 290 KiB/s wr, 24 op/s 2023-03-09T20:02:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:02:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18243.539840 secs 2023-03-09T20:02:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:02:15 smithi122 ceph-mon[93869]: pgmap v10242: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 94 KiB/s wr, 9 op/s 2023-03-09T20:02:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:02:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18246.227069 secs 2023-03-09T20:02:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:02:15 smithi181 ceph-mon[94189]: pgmap v10242: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 94 KiB/s wr, 9 op/s 2023-03-09T20:02:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:02:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18246.227069 secs 2023-03-09T20:02:15.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:02:15 smithi038 ceph-mon[92422]: pgmap v10242: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 94 KiB/s wr, 9 op/s 2023-03-09T20:02:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:02:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18246.227069 secs 2023-03-09T20:02:17.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:02:17 smithi122 ceph-mon[93869]: pgmap v10243: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 111 KiB/s wr, 9 op/s 2023-03-09T20:02:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:02:17 smithi181 ceph-mon[94189]: pgmap v10243: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 111 KiB/s wr, 9 op/s 2023-03-09T20:02:17.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:02:17 smithi038 ceph-mon[92422]: pgmap v10243: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 111 KiB/s wr, 9 op/s 2023-03-09T20:02:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18710.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:02:18.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:02:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18248.539960 secs 2023-03-09T20:02:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:02:18 smithi122 ceph-mon[93869]: osdmap e854: 12 total, 12 up, 12 in 2023-03-09T20:02:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:02:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18248.539960 secs 2023-03-09T20:02:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:02:18 smithi181 ceph-mon[94189]: osdmap e854: 12 total, 12 up, 12 in 2023-03-09T20:02:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:02:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18248.539960 secs 2023-03-09T20:02:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:02:18 smithi038 ceph-mon[92422]: osdmap e854: 12 total, 12 up, 12 in 2023-03-09T20:02:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:02:19 smithi122 ceph-mon[93869]: pgmap v10244: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 90 KiB/s wr, 9 op/s 2023-03-09T20:02:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:02:19 smithi181 ceph-mon[94189]: pgmap v10244: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 90 KiB/s wr, 9 op/s 2023-03-09T20:02:19.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:02:19 smithi038 ceph-mon[92422]: pgmap v10244: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 90 KiB/s wr, 9 op/s 2023-03-09T20:02:20.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:02:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18251.227130 secs 2023-03-09T20:02:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:02:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18251.227130 secs 2023-03-09T20:02:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:02:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18251.227130 secs 2023-03-09T20:02:21.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:02:21 smithi122 ceph-mon[93869]: pgmap v10246: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 90 KiB/s wr, 9 op/s 2023-03-09T20:02:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:02:21 smithi181 ceph-mon[94189]: pgmap v10246: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 90 KiB/s wr, 9 op/s 2023-03-09T20:02:21.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:02:21 smithi038 ceph-mon[92422]: pgmap v10246: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 90 KiB/s wr, 9 op/s 2023-03-09T20:02:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18715.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:02:23.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:02:23 smithi122 ceph-mon[93869]: pgmap v10247: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 121 KiB/s wr, 8 op/s 2023-03-09T20:02:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:02:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18253.540076 secs 2023-03-09T20:02:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:02:23 smithi181 ceph-mon[94189]: pgmap v10247: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 121 KiB/s wr, 8 op/s 2023-03-09T20:02:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:02:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18253.540076 secs 2023-03-09T20:02:23.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:02:23 smithi038 ceph-mon[92422]: pgmap v10247: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 121 KiB/s wr, 8 op/s 2023-03-09T20:02:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:02:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18253.540076 secs 2023-03-09T20:02:25.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:02:25 smithi122 ceph-mon[93869]: pgmap v10248: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 152 KiB/s wr, 9 op/s 2023-03-09T20:02:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:02:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18256.227190 secs 2023-03-09T20:02:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:02:25 smithi181 ceph-mon[94189]: pgmap v10248: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 152 KiB/s wr, 9 op/s 2023-03-09T20:02:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:02:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18256.227190 secs 2023-03-09T20:02:25.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:02:25 smithi038 ceph-mon[92422]: pgmap v10248: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 152 KiB/s wr, 9 op/s 2023-03-09T20:02:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:02:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18256.227190 secs 2023-03-09T20:02:27.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:02:27 smithi122 ceph-mon[93869]: pgmap v10249: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 136 KiB/s wr, 9 op/s 2023-03-09T20:02:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:02:27 smithi181 ceph-mon[94189]: pgmap v10249: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 136 KiB/s wr, 9 op/s 2023-03-09T20:02:27.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:02:27 smithi038 ceph-mon[92422]: pgmap v10249: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 136 KiB/s wr, 9 op/s 2023-03-09T20:02:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18720.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:02:28.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:02:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18258.540176 secs 2023-03-09T20:02:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:02:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18258.540176 secs 2023-03-09T20:02:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:02:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18258.540176 secs 2023-03-09T20:02:29.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:02:29 smithi122 ceph-mon[93869]: pgmap v10250: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 81 KiB/s wr, 9 op/s 2023-03-09T20:02:29.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:02:29 smithi181 ceph-mon[94189]: pgmap v10250: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 81 KiB/s wr, 9 op/s 2023-03-09T20:02:29.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:02:29 smithi038 ceph-mon[92422]: pgmap v10250: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 81 KiB/s wr, 9 op/s 2023-03-09T20:02:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:02:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18261.227250 secs 2023-03-09T20:02:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:02:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18261.227250 secs 2023-03-09T20:02:31.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:02:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18261.227250 secs 2023-03-09T20:02:31.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:02:31 smithi038 ceph-mon[92422]: pgmap v10251: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 8 op/s 2023-03-09T20:02:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:02:31 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:02:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:02:31 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:02:32.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:02:31 smithi122 ceph-mon[93869]: pgmap v10251: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 8 op/s 2023-03-09T20:02:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:02:31 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:02:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:02:31 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:02:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:02:31 smithi181 ceph-mon[94189]: pgmap v10251: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 8 op/s 2023-03-09T20:02:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:02:31 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:02:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:02:31 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:02:33.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:02:32 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:02:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:02:32 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:02:33.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:02:32 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:02:33.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:02:32 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:02:33.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:02:32 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:02:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:02:32 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:02:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18725.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:02:34.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:02:33 smithi122 ceph-mon[93869]: pgmap v10252: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 197 KiB/s wr, 10 op/s 2023-03-09T20:02:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:02:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18263.540256 secs 2023-03-09T20:02:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:02:33 smithi181 ceph-mon[94189]: pgmap v10252: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 197 KiB/s wr, 10 op/s 2023-03-09T20:02:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:02:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18263.540256 secs 2023-03-09T20:02:34.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:02:33 smithi038 ceph-mon[92422]: pgmap v10252: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 197 KiB/s wr, 10 op/s 2023-03-09T20:02:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:02:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18263.540256 secs 2023-03-09T20:02:36.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:02:35 smithi122 ceph-mon[93869]: pgmap v10253: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 156 KiB/s wr, 7 op/s 2023-03-09T20:02:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:02:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18266.227313 secs 2023-03-09T20:02:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:02:35 smithi181 ceph-mon[94189]: pgmap v10253: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 156 KiB/s wr, 7 op/s 2023-03-09T20:02:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:02:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18266.227313 secs 2023-03-09T20:02:36.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:02:35 smithi038 ceph-mon[92422]: pgmap v10253: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 156 KiB/s wr, 7 op/s 2023-03-09T20:02:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:02:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18266.227313 secs 2023-03-09T20:02:38.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:02:37 smithi122 ceph-mon[93869]: pgmap v10254: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 130 KiB/s wr, 9 op/s 2023-03-09T20:02:38.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:02:37 smithi181 ceph-mon[94189]: pgmap v10254: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 130 KiB/s wr, 9 op/s 2023-03-09T20:02:38.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:02:37 smithi038 ceph-mon[92422]: pgmap v10254: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 130 KiB/s wr, 9 op/s 2023-03-09T20:02:38.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18730.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:02:40.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:02:39 smithi122 ceph-mon[93869]: pgmap v10255: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 129 KiB/s wr, 6 op/s 2023-03-09T20:02:40.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:02:39 smithi181 ceph-mon[94189]: pgmap v10255: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 129 KiB/s wr, 6 op/s 2023-03-09T20:02:40.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:02:39 smithi038 ceph-mon[92422]: pgmap v10255: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 129 KiB/s wr, 6 op/s 2023-03-09T20:02:41.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:02:40 smithi122 ceph-mon[93869]: pgmap v10256: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 129 KiB/s wr, 6 op/s 2023-03-09T20:02:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:02:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18271.227374 secs 2023-03-09T20:02:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:02:40 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:02:41.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:02:40 smithi181 ceph-mon[94189]: pgmap v10256: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 129 KiB/s wr, 6 op/s 2023-03-09T20:02:41.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:02:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18271.227374 secs 2023-03-09T20:02:41.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:02:40 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:02:41.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:02:40 smithi038 ceph-mon[92422]: pgmap v10256: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 129 KiB/s wr, 6 op/s 2023-03-09T20:02:41.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:02:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18271.227374 secs 2023-03-09T20:02:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:02:40 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:02:42.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:02:41 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:02:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:02:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:02:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:02:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:02:42.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:02:41 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:02:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:02:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:02:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:02:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:02:42.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:02:41 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:02:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:02:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:02:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:02:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:02:43.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:02:42 smithi122 ceph-mon[93869]: pgmap v10257: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 210 KiB/s wr, 9 op/s 2023-03-09T20:02:43.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:02:42 smithi181 ceph-mon[94189]: pgmap v10257: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 210 KiB/s wr, 9 op/s 2023-03-09T20:02:43.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:02:42 smithi038 ceph-mon[92422]: pgmap v10257: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 210 KiB/s wr, 9 op/s 2023-03-09T20:02:43.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18735.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:02:45.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:02:45 smithi122 ceph-mon[93869]: pgmap v10258: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 81 KiB/s wr, 6 op/s 2023-03-09T20:02:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:02:45 smithi181 ceph-mon[94189]: pgmap v10258: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 81 KiB/s wr, 6 op/s 2023-03-09T20:02:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:02:45 smithi038 ceph-mon[92422]: pgmap v10258: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 81 KiB/s wr, 6 op/s 2023-03-09T20:02:46.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:02:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18276.227432 secs 2023-03-09T20:02:46.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:02:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18276.227432 secs 2023-03-09T20:02:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:02:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18276.227432 secs 2023-03-09T20:02:47.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:02:47 smithi122 ceph-mon[93869]: pgmap v10259: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 222 KiB/s wr, 9 op/s 2023-03-09T20:02:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:02:47 smithi181 ceph-mon[94189]: pgmap v10259: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 222 KiB/s wr, 9 op/s 2023-03-09T20:02:47.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:02:47 smithi038 ceph-mon[92422]: pgmap v10259: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 222 KiB/s wr, 9 op/s 2023-03-09T20:02:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18740.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:02:49.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:02:49 smithi122 ceph-mon[93869]: pgmap v10260: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 222 KiB/s wr, 6 op/s 2023-03-09T20:02:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:02:49 smithi181 ceph-mon[94189]: pgmap v10260: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 222 KiB/s wr, 6 op/s 2023-03-09T20:02:49.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:02:49 smithi038 ceph-mon[92422]: pgmap v10260: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 222 KiB/s wr, 6 op/s 2023-03-09T20:02:51.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:02:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18281.227495 secs 2023-03-09T20:02:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:02:51 smithi122 ceph-mon[93869]: pgmap v10261: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 301 KiB/s wr, 35 op/s 2023-03-09T20:02:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:02:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:02:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:02:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:02:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:02:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18281.227495 secs 2023-03-09T20:02:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:02:51 smithi181 ceph-mon[94189]: pgmap v10261: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 301 KiB/s wr, 35 op/s 2023-03-09T20:02:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:02:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:02:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:02:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:02:51.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:02:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18281.227495 secs 2023-03-09T20:02:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:02:51 smithi038 ceph-mon[92422]: pgmap v10261: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 301 KiB/s wr, 35 op/s 2023-03-09T20:02:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:02:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:02:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:02:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:02:52.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:02:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:02:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:02:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:02:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:02:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:02:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:02:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:02:52.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:02:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:02:52.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:02:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:02:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18745.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:02:53.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:02:53 smithi122 ceph-mon[93869]: pgmap v10262: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 328 KiB/s rd, 302 KiB/s wr, 39 op/s 2023-03-09T20:02:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:02:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:02:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:02:53 smithi181 ceph-mon[94189]: pgmap v10262: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 328 KiB/s rd, 302 KiB/s wr, 39 op/s 2023-03-09T20:02:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:02:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:02:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:02:53 smithi038 ceph-mon[92422]: pgmap v10262: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 328 KiB/s rd, 302 KiB/s wr, 39 op/s 2023-03-09T20:02:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:02:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:02:54.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:02:54 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:02:54.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:02:54 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:02:54.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:02:54 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:02:54.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:02:54 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:02:54.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:02:54 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:02:54.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:02:54 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:02:54.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:02:54 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:02:54.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:02:54 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:02:54.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:02:54 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:02:55.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:02:55 smithi122 ceph-mon[93869]: pgmap v10263: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 222 KiB/s wr, 44 op/s 2023-03-09T20:02:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:02:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18286.227557 secs 2023-03-09T20:02:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:02:55 smithi181 ceph-mon[94189]: pgmap v10263: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 222 KiB/s wr, 44 op/s 2023-03-09T20:02:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:02:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18286.227557 secs 2023-03-09T20:02:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:02:55 smithi038 ceph-mon[92422]: pgmap v10263: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 222 KiB/s wr, 44 op/s 2023-03-09T20:02:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:02:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18286.227557 secs 2023-03-09T20:02:57.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:02:57 smithi122 ceph-mon[93869]: pgmap v10264: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 478 KiB/s wr, 48 op/s 2023-03-09T20:02:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:02:57 smithi181 ceph-mon[94189]: pgmap v10264: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 478 KiB/s wr, 48 op/s 2023-03-09T20:02:57.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:02:57 smithi038 ceph-mon[92422]: pgmap v10264: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 478 KiB/s wr, 48 op/s 2023-03-09T20:02:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18750.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:02:59.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:02:59 smithi122 ceph-mon[93869]: pgmap v10265: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 340 KiB/s wr, 45 op/s 2023-03-09T20:02:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:02:59 smithi181 ceph-mon[94189]: pgmap v10265: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 340 KiB/s wr, 45 op/s 2023-03-09T20:02:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:02:59 smithi038 ceph-mon[92422]: pgmap v10265: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 340 KiB/s wr, 45 op/s 2023-03-09T20:03:00.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:03:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:03:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18291.227615 secs 2023-03-09T20:03:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:03:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:03:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18291.227615 secs 2023-03-09T20:03:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:03:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:03:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18291.227615 secs 2023-03-09T20:03:01.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:01 smithi122 ceph-mon[93869]: pgmap v10266: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 422 KiB/s rd, 340 KiB/s wr, 46 op/s 2023-03-09T20:03:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:01 smithi181 ceph-mon[94189]: pgmap v10266: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 422 KiB/s rd, 340 KiB/s wr, 46 op/s 2023-03-09T20:03:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:01 smithi038 ceph-mon[92422]: pgmap v10266: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 422 KiB/s rd, 340 KiB/s wr, 46 op/s 2023-03-09T20:03:03.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18755.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:03:03.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:03 smithi122 ceph-mon[93869]: pgmap v10267: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 286 KiB/s wr, 21 op/s 2023-03-09T20:03:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18293.540845 secs 2023-03-09T20:03:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:03 smithi181 ceph-mon[94189]: pgmap v10267: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 286 KiB/s wr, 21 op/s 2023-03-09T20:03:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18293.540845 secs 2023-03-09T20:03:03.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:03 smithi038 ceph-mon[92422]: pgmap v10267: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 286 KiB/s wr, 21 op/s 2023-03-09T20:03:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18293.540845 secs 2023-03-09T20:03:05.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:04 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:03:05.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:04 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:03:05.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:04 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:03:05.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:04 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:03:05.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:04 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:03:05.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:04 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:03:05.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:04 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:03:05.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:04 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:03:05.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:04 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:03:06.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:05 smithi122 ceph-mon[93869]: pgmap v10268: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 442 KiB/s rd, 442 KiB/s wr, 32 op/s 2023-03-09T20:03:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18296.227676 secs 2023-03-09T20:03:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:05 smithi181 ceph-mon[94189]: pgmap v10268: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 442 KiB/s rd, 442 KiB/s wr, 32 op/s 2023-03-09T20:03:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18296.227676 secs 2023-03-09T20:03:06.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:05 smithi038 ceph-mon[92422]: pgmap v10268: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 442 KiB/s rd, 442 KiB/s wr, 32 op/s 2023-03-09T20:03:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18296.227676 secs 2023-03-09T20:03:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:06 smithi122 ceph-mon[93869]: pgmap v10269: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 453 KiB/s wr, 28 op/s 2023-03-09T20:03:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:06 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:03:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:03:07.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:06 smithi181 ceph-mon[94189]: pgmap v10269: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 453 KiB/s wr, 28 op/s 2023-03-09T20:03:07.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:06 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:03:07.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:03:07.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:06 smithi038 ceph-mon[92422]: pgmap v10269: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 453 KiB/s wr, 28 op/s 2023-03-09T20:03:07.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:06 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:03:07.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:03:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18760.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:03:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18298.540966 secs 2023-03-09T20:03:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:08 smithi122 ceph-mon[93869]: pgmap v10270: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 234 KiB/s wr, 25 op/s 2023-03-09T20:03:09.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18298.540966 secs 2023-03-09T20:03:09.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:08 smithi181 ceph-mon[94189]: pgmap v10270: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 234 KiB/s wr, 25 op/s 2023-03-09T20:03:09.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18298.540966 secs 2023-03-09T20:03:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:08 smithi038 ceph-mon[92422]: pgmap v10270: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 234 KiB/s wr, 25 op/s 2023-03-09T20:03:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18301.227737 secs 2023-03-09T20:03:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:10 smithi122 ceph-mon[93869]: pgmap v10271: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 232 KiB/s wr, 24 op/s 2023-03-09T20:03:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:10 smithi122 ceph-mon[93869]: osdmap e855: 12 total, 12 up, 12 in 2023-03-09T20:03:11.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18301.227737 secs 2023-03-09T20:03:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:10 smithi181 ceph-mon[94189]: pgmap v10271: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 232 KiB/s wr, 24 op/s 2023-03-09T20:03:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:10 smithi181 ceph-mon[94189]: osdmap e855: 12 total, 12 up, 12 in 2023-03-09T20:03:11.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18301.227737 secs 2023-03-09T20:03:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:10 smithi038 ceph-mon[92422]: pgmap v10271: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 232 KiB/s wr, 24 op/s 2023-03-09T20:03:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:10 smithi038 ceph-mon[92422]: osdmap e855: 12 total, 12 up, 12 in 2023-03-09T20:03:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:12 smithi122 ceph-mon[93869]: pgmap v10273: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 278 KiB/s wr, 27 op/s 2023-03-09T20:03:13.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:12 smithi181 ceph-mon[94189]: pgmap v10273: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 278 KiB/s wr, 27 op/s 2023-03-09T20:03:13.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:12 smithi038 ceph-mon[92422]: pgmap v10273: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 278 KiB/s wr, 27 op/s 2023-03-09T20:03:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18765.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:03:14.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18303.541098 secs 2023-03-09T20:03:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18303.541098 secs 2023-03-09T20:03:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18303.541098 secs 2023-03-09T20:03:15.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:15 smithi122 ceph-mon[93869]: pgmap v10274: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 272 KiB/s wr, 11 op/s 2023-03-09T20:03:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:15 smithi181 ceph-mon[94189]: pgmap v10274: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 272 KiB/s wr, 11 op/s 2023-03-09T20:03:15.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:15 smithi038 ceph-mon[92422]: pgmap v10274: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 272 KiB/s wr, 11 op/s 2023-03-09T20:03:16.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18306.227798 secs 2023-03-09T20:03:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18306.227798 secs 2023-03-09T20:03:16.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18306.227798 secs 2023-03-09T20:03:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:17 smithi122 ceph-mon[93869]: pgmap v10275: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 258 KiB/s wr, 8 op/s 2023-03-09T20:03:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:17 smithi181 ceph-mon[94189]: pgmap v10275: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 258 KiB/s wr, 8 op/s 2023-03-09T20:03:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:17 smithi038 ceph-mon[92422]: pgmap v10275: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 258 KiB/s wr, 8 op/s 2023-03-09T20:03:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18770.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:03:18.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18308.541216 secs 2023-03-09T20:03:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:18 smithi122 ceph-mon[93869]: osdmap e856: 12 total, 12 up, 12 in 2023-03-09T20:03:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18308.541216 secs 2023-03-09T20:03:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:18 smithi181 ceph-mon[94189]: osdmap e856: 12 total, 12 up, 12 in 2023-03-09T20:03:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18308.541216 secs 2023-03-09T20:03:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:18 smithi038 ceph-mon[92422]: osdmap e856: 12 total, 12 up, 12 in 2023-03-09T20:03:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:19 smithi122 ceph-mon[93869]: pgmap v10276: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 215 KiB/s wr, 8 op/s 2023-03-09T20:03:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:19 smithi181 ceph-mon[94189]: pgmap v10276: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 215 KiB/s wr, 8 op/s 2023-03-09T20:03:19.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:19 smithi038 ceph-mon[92422]: pgmap v10276: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 215 KiB/s wr, 8 op/s 2023-03-09T20:03:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18311.227875 secs 2023-03-09T20:03:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:21 smithi122 ceph-mon[93869]: pgmap v10278: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 216 KiB/s wr, 8 op/s 2023-03-09T20:03:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18311.227875 secs 2023-03-09T20:03:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:21 smithi181 ceph-mon[94189]: pgmap v10278: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 216 KiB/s wr, 8 op/s 2023-03-09T20:03:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18311.227875 secs 2023-03-09T20:03:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:21 smithi038 ceph-mon[92422]: pgmap v10278: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 216 KiB/s wr, 8 op/s 2023-03-09T20:03:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18775.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:03:23.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:23 smithi122 ceph-mon[93869]: pgmap v10279: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 183 KiB/s wr, 7 op/s 2023-03-09T20:03:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18313.541325 secs 2023-03-09T20:03:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:23 smithi181 ceph-mon[94189]: pgmap v10279: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 183 KiB/s wr, 7 op/s 2023-03-09T20:03:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18313.541325 secs 2023-03-09T20:03:23.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:23 smithi038 ceph-mon[92422]: pgmap v10279: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 183 KiB/s wr, 7 op/s 2023-03-09T20:03:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18313.541325 secs 2023-03-09T20:03:25.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:25 smithi122 ceph-mon[93869]: pgmap v10280: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.8 KiB/s wr, 6 op/s 2023-03-09T20:03:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:25 smithi181 ceph-mon[94189]: pgmap v10280: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.8 KiB/s wr, 6 op/s 2023-03-09T20:03:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:25 smithi038 ceph-mon[92422]: pgmap v10280: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.8 KiB/s wr, 6 op/s 2023-03-09T20:03:26.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18316.227966 secs 2023-03-09T20:03:26.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18316.227966 secs 2023-03-09T20:03:26.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18316.227966 secs 2023-03-09T20:03:27.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:27 smithi122 ceph-mon[93869]: pgmap v10281: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 61 KiB/s wr, 7 op/s 2023-03-09T20:03:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:27 smithi181 ceph-mon[94189]: pgmap v10281: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 61 KiB/s wr, 7 op/s 2023-03-09T20:03:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:27 smithi038 ceph-mon[92422]: pgmap v10281: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 61 KiB/s wr, 7 op/s 2023-03-09T20:03:28.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18780.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:03:28.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18318.541432 secs 2023-03-09T20:03:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18318.541432 secs 2023-03-09T20:03:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18318.541432 secs 2023-03-09T20:03:29.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:29 smithi122 ceph-mon[93869]: pgmap v10282: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 80 KiB/s wr, 7 op/s 2023-03-09T20:03:29.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:29 smithi181 ceph-mon[94189]: pgmap v10282: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 80 KiB/s wr, 7 op/s 2023-03-09T20:03:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:29 smithi038 ceph-mon[92422]: pgmap v10282: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 80 KiB/s wr, 7 op/s 2023-03-09T20:03:31.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18321.228026 secs 2023-03-09T20:03:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:31 smithi122 ceph-mon[93869]: pgmap v10283: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 71 KiB/s wr, 6 op/s 2023-03-09T20:03:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18321.228026 secs 2023-03-09T20:03:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:31 smithi181 ceph-mon[94189]: pgmap v10283: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 71 KiB/s wr, 6 op/s 2023-03-09T20:03:31.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18321.228026 secs 2023-03-09T20:03:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:31 smithi038 ceph-mon[92422]: pgmap v10283: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 71 KiB/s wr, 6 op/s 2023-03-09T20:03:33.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18785.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:03:33.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:33 smithi122 ceph-mon[93869]: pgmap v10284: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 67 KiB/s wr, 9 op/s 2023-03-09T20:03:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18323.541542 secs 2023-03-09T20:03:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:33 smithi181 ceph-mon[94189]: pgmap v10284: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 67 KiB/s wr, 9 op/s 2023-03-09T20:03:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18323.541542 secs 2023-03-09T20:03:33.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:33 smithi038 ceph-mon[92422]: pgmap v10284: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 67 KiB/s wr, 9 op/s 2023-03-09T20:03:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18323.541542 secs 2023-03-09T20:03:35.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:35 smithi122 ceph-mon[93869]: pgmap v10285: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 67 KiB/s wr, 7 op/s 2023-03-09T20:03:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:35 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:03:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:35 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:03:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:35 smithi181 ceph-mon[94189]: pgmap v10285: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 67 KiB/s wr, 7 op/s 2023-03-09T20:03:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:35 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:03:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:35 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:03:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:35 smithi038 ceph-mon[92422]: pgmap v10285: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 67 KiB/s wr, 7 op/s 2023-03-09T20:03:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:35 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:03:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:35 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:03:36.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18326.228085 secs 2023-03-09T20:03:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:03:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:03:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18326.228085 secs 2023-03-09T20:03:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:03:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:03:36.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18326.228085 secs 2023-03-09T20:03:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:03:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:03:37.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:37 smithi122 ceph-mon[93869]: pgmap v10286: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 135 KiB/s wr, 10 op/s 2023-03-09T20:03:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:37 smithi181 ceph-mon[94189]: pgmap v10286: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 135 KiB/s wr, 10 op/s 2023-03-09T20:03:37.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:37 smithi038 ceph-mon[92422]: pgmap v10286: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 135 KiB/s wr, 10 op/s 2023-03-09T20:03:38.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18790.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:03:39.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:39 smithi122 ceph-mon[93869]: pgmap v10287: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 86 KiB/s wr, 8 op/s 2023-03-09T20:03:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:39 smithi181 ceph-mon[94189]: pgmap v10287: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 86 KiB/s wr, 8 op/s 2023-03-09T20:03:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:39 smithi038 ceph-mon[92422]: pgmap v10287: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 86 KiB/s wr, 8 op/s 2023-03-09T20:03:40.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:40 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:03:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:40 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:03:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:40 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:03:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18331.228146 secs 2023-03-09T20:03:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:41 smithi122 ceph-mon[93869]: pgmap v10288: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 68 KiB/s wr, 7 op/s 2023-03-09T20:03:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:41 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:03:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:03:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:03:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18331.228146 secs 2023-03-09T20:03:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:41 smithi181 ceph-mon[94189]: pgmap v10288: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 68 KiB/s wr, 7 op/s 2023-03-09T20:03:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:41 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:03:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:03:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:03:41.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18331.228146 secs 2023-03-09T20:03:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:41 smithi038 ceph-mon[92422]: pgmap v10288: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 68 KiB/s wr, 7 op/s 2023-03-09T20:03:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:41 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:03:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:03:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:03:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18795.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:03:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:43 smithi122 ceph-mon[93869]: pgmap v10289: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 261 KiB/s wr, 11 op/s 2023-03-09T20:03:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:43 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 18333.541793 secs 2023-03-09T20:03:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:43 smithi181 ceph-mon[94189]: pgmap v10289: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 261 KiB/s wr, 11 op/s 2023-03-09T20:03:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:43 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 18333.541793 secs 2023-03-09T20:03:43.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:43 smithi038 ceph-mon[92422]: pgmap v10289: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 261 KiB/s wr, 11 op/s 2023-03-09T20:03:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:43 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 18333.541793 secs 2023-03-09T20:03:45.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:45 smithi122 ceph-mon[93869]: pgmap v10290: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 261 KiB/s wr, 8 op/s 2023-03-09T20:03:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:45 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:03:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:45 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:03:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:45 smithi181 ceph-mon[94189]: pgmap v10290: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 261 KiB/s wr, 8 op/s 2023-03-09T20:03:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:45 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:03:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:45 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:03:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:45 smithi038 ceph-mon[92422]: pgmap v10290: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 261 KiB/s wr, 8 op/s 2023-03-09T20:03:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:45 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:03:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:45 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:03:46.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18336.228221 secs 2023-03-09T20:03:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18336.228221 secs 2023-03-09T20:03:46.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18336.228221 secs 2023-03-09T20:03:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:47 smithi122 ceph-mon[93869]: pgmap v10291: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 348 KiB/s wr, 20 op/s 2023-03-09T20:03:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:47 smithi181 ceph-mon[94189]: pgmap v10291: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 348 KiB/s wr, 20 op/s 2023-03-09T20:03:47.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:47 smithi038 ceph-mon[92422]: pgmap v10291: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 348 KiB/s wr, 20 op/s 2023-03-09T20:03:48.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18800.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:03:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18338.541935 secs 2023-03-09T20:03:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:48 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:03:48.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18338.541935 secs 2023-03-09T20:03:48.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:48 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:03:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18338.541935 secs 2023-03-09T20:03:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:48 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:03:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:49 smithi122 ceph-mon[93869]: pgmap v10292: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 280 KiB/s wr, 17 op/s 2023-03-09T20:03:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:03:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:03:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:49 smithi181 ceph-mon[94189]: pgmap v10292: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 280 KiB/s wr, 17 op/s 2023-03-09T20:03:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:03:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:03:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:49 smithi038 ceph-mon[92422]: pgmap v10292: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 280 KiB/s wr, 17 op/s 2023-03-09T20:03:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:03:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:03:50.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18341.228279 secs 2023-03-09T20:03:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18341.228279 secs 2023-03-09T20:03:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18341.228279 secs 2023-03-09T20:03:51.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:51 smithi122 ceph-mon[93869]: pgmap v10293: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 280 KiB/s wr, 17 op/s 2023-03-09T20:03:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:03:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:03:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:51 smithi181 ceph-mon[94189]: pgmap v10293: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 280 KiB/s wr, 17 op/s 2023-03-09T20:03:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:03:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:03:51.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:51 smithi038 ceph-mon[92422]: pgmap v10293: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 280 KiB/s wr, 17 op/s 2023-03-09T20:03:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:03:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:03:52.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:03:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:03:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:03:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:03:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:03:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:03:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18805.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:03:53.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:53 smithi038 ceph-mon[92422]: pgmap v10294: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 473 KiB/s wr, 21 op/s 2023-03-09T20:03:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18343.542046 secs 2023-03-09T20:03:54.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:53 smithi122 ceph-mon[93869]: pgmap v10294: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 473 KiB/s wr, 21 op/s 2023-03-09T20:03:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18343.542046 secs 2023-03-09T20:03:54.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:53 smithi181 ceph-mon[94189]: pgmap v10294: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 473 KiB/s wr, 21 op/s 2023-03-09T20:03:54.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18343.542046 secs 2023-03-09T20:03:54.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:54 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:03:54.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:54 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:03:54.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:54 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:03:54.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:54 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:03:55.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:54 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:03:55.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:54 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:03:55.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:54 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:03:55.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:54 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:03:55.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:54 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:03:55.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:54 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:03:55.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:54 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:03:55.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:54 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:03:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:55 smithi038 ceph-mon[92422]: pgmap v10295: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 280 KiB/s wr, 19 op/s 2023-03-09T20:03:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18346.228339 secs 2023-03-09T20:03:56.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:55 smithi122 ceph-mon[93869]: pgmap v10295: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 280 KiB/s wr, 19 op/s 2023-03-09T20:03:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18346.228339 secs 2023-03-09T20:03:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:55 smithi181 ceph-mon[94189]: pgmap v10295: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 280 KiB/s wr, 19 op/s 2023-03-09T20:03:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18346.228339 secs 2023-03-09T20:03:58.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:57 smithi122 ceph-mon[93869]: pgmap v10296: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 456 KiB/s rd, 433 KiB/s wr, 22 op/s 2023-03-09T20:03:58.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:57 smithi181 ceph-mon[94189]: pgmap v10296: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 456 KiB/s rd, 433 KiB/s wr, 22 op/s 2023-03-09T20:03:58.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:57 smithi038 ceph-mon[92422]: pgmap v10296: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 456 KiB/s rd, 433 KiB/s wr, 22 op/s 2023-03-09T20:03:58.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18810.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:03:59.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18348.542193 secs 2023-03-09T20:03:59.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18348.542193 secs 2023-03-09T20:03:59.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18348.542193 secs 2023-03-09T20:04:00.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:59 smithi122 ceph-mon[93869]: pgmap v10297: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 346 KiB/s wr, 10 op/s 2023-03-09T20:04:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:04:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:03:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:04:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:59 smithi181 ceph-mon[94189]: pgmap v10297: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 346 KiB/s wr, 10 op/s 2023-03-09T20:04:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:04:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:03:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:04:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:59 smithi038 ceph-mon[92422]: pgmap v10297: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 346 KiB/s wr, 10 op/s 2023-03-09T20:04:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:04:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:03:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:04:01.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:04:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:04:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18351.228424 secs 2023-03-09T20:04:01.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:04:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:04:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18351.228424 secs 2023-03-09T20:04:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:04:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:04:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18351.228424 secs 2023-03-09T20:04:02.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:01 smithi122 ceph-mon[93869]: pgmap v10298: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 447 KiB/s wr, 10 op/s 2023-03-09T20:04:02.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:01 smithi181 ceph-mon[94189]: pgmap v10298: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 447 KiB/s wr, 10 op/s 2023-03-09T20:04:02.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:01 smithi038 ceph-mon[92422]: pgmap v10298: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 447 KiB/s wr, 10 op/s 2023-03-09T20:04:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18815.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:04:04.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:03 smithi122 ceph-mon[93869]: pgmap v10299: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 461 KiB/s wr, 13 op/s 2023-03-09T20:04:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18353.542339 secs 2023-03-09T20:04:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:03 smithi181 ceph-mon[94189]: pgmap v10299: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 461 KiB/s wr, 13 op/s 2023-03-09T20:04:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18353.542339 secs 2023-03-09T20:04:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:03 smithi038 ceph-mon[92422]: pgmap v10299: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 461 KiB/s wr, 13 op/s 2023-03-09T20:04:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18353.542339 secs 2023-03-09T20:04:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:05 smithi122 ceph-mon[93869]: pgmap v10300: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 328 KiB/s wr, 22 op/s 2023-03-09T20:04:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18356.228506 secs 2023-03-09T20:04:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:05 smithi181 ceph-mon[94189]: pgmap v10300: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 328 KiB/s wr, 22 op/s 2023-03-09T20:04:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18356.228506 secs 2023-03-09T20:04:06.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:05 smithi038 ceph-mon[92422]: pgmap v10300: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 328 KiB/s wr, 22 op/s 2023-03-09T20:04:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18356.228506 secs 2023-03-09T20:04:07.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:06 smithi122 ceph-mon[93869]: pgmap v10301: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 350 KiB/s wr, 26 op/s 2023-03-09T20:04:07.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:06 smithi181 ceph-mon[94189]: pgmap v10301: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 350 KiB/s wr, 26 op/s 2023-03-09T20:04:07.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:06 smithi038 ceph-mon[92422]: pgmap v10301: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 350 KiB/s wr, 26 op/s 2023-03-09T20:04:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18820.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:04:09.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18358.542455 secs 2023-03-09T20:04:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:08 smithi122 ceph-mon[93869]: pgmap v10302: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 198 KiB/s wr, 23 op/s 2023-03-09T20:04:09.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18358.542455 secs 2023-03-09T20:04:09.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:08 smithi181 ceph-mon[94189]: pgmap v10302: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 198 KiB/s wr, 23 op/s 2023-03-09T20:04:09.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18358.542455 secs 2023-03-09T20:04:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:08 smithi038 ceph-mon[92422]: pgmap v10302: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 198 KiB/s wr, 23 op/s 2023-03-09T20:04:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18361.228566 secs 2023-03-09T20:04:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:10 smithi122 ceph-mon[93869]: pgmap v10303: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 247 KiB/s rd, 252 KiB/s wr, 23 op/s 2023-03-09T20:04:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:10 smithi122 ceph-mon[93869]: osdmap e857: 12 total, 12 up, 12 in 2023-03-09T20:04:11.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18361.228566 secs 2023-03-09T20:04:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:10 smithi181 ceph-mon[94189]: pgmap v10303: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 247 KiB/s rd, 252 KiB/s wr, 23 op/s 2023-03-09T20:04:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:10 smithi181 ceph-mon[94189]: osdmap e857: 12 total, 12 up, 12 in 2023-03-09T20:04:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18361.228566 secs 2023-03-09T20:04:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:10 smithi038 ceph-mon[92422]: pgmap v10303: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 247 KiB/s rd, 252 KiB/s wr, 23 op/s 2023-03-09T20:04:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:10 smithi038 ceph-mon[92422]: osdmap e857: 12 total, 12 up, 12 in 2023-03-09T20:04:13.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:12 smithi122 ceph-mon[93869]: pgmap v10305: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 181 KiB/s wr, 27 op/s 2023-03-09T20:04:13.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:12 smithi181 ceph-mon[94189]: pgmap v10305: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 181 KiB/s wr, 27 op/s 2023-03-09T20:04:13.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:12 smithi038 ceph-mon[92422]: pgmap v10305: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 181 KiB/s wr, 27 op/s 2023-03-09T20:04:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18825.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:04:14.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18363.542571 secs 2023-03-09T20:04:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18363.542571 secs 2023-03-09T20:04:14.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18363.542571 secs 2023-03-09T20:04:15.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:15 smithi122 ceph-mon[93869]: pgmap v10306: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 110 KiB/s wr, 12 op/s 2023-03-09T20:04:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:15 smithi181 ceph-mon[94189]: pgmap v10306: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 110 KiB/s wr, 12 op/s 2023-03-09T20:04:15.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:15 smithi038 ceph-mon[92422]: pgmap v10306: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 110 KiB/s wr, 12 op/s 2023-03-09T20:04:16.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18366.228651 secs 2023-03-09T20:04:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18366.228651 secs 2023-03-09T20:04:16.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18366.228651 secs 2023-03-09T20:04:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:17 smithi122 ceph-mon[93869]: pgmap v10307: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 83 KiB/s wr, 8 op/s 2023-03-09T20:04:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:17 smithi181 ceph-mon[94189]: pgmap v10307: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 83 KiB/s wr, 8 op/s 2023-03-09T20:04:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:17 smithi038 ceph-mon[92422]: pgmap v10307: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 83 KiB/s wr, 8 op/s 2023-03-09T20:04:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18830.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:04:18.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18368.542698 secs 2023-03-09T20:04:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:18 smithi122 ceph-mon[93869]: osdmap e858: 12 total, 12 up, 12 in 2023-03-09T20:04:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18368.542698 secs 2023-03-09T20:04:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:18 smithi181 ceph-mon[94189]: osdmap e858: 12 total, 12 up, 12 in 2023-03-09T20:04:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18368.542698 secs 2023-03-09T20:04:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:18 smithi038 ceph-mon[92422]: osdmap e858: 12 total, 12 up, 12 in 2023-03-09T20:04:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:19 smithi122 ceph-mon[93869]: pgmap v10308: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 83 KiB/s wr, 8 op/s 2023-03-09T20:04:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:19 smithi181 ceph-mon[94189]: pgmap v10308: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 83 KiB/s wr, 8 op/s 2023-03-09T20:04:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:19 smithi038 ceph-mon[92422]: pgmap v10308: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 83 KiB/s wr, 8 op/s 2023-03-09T20:04:21.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18371.228712 secs 2023-03-09T20:04:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:21 smithi122 ceph-mon[93869]: pgmap v10310: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 83 KiB/s wr, 8 op/s 2023-03-09T20:04:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18371.228712 secs 2023-03-09T20:04:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:21 smithi181 ceph-mon[94189]: pgmap v10310: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 83 KiB/s wr, 8 op/s 2023-03-09T20:04:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18371.228712 secs 2023-03-09T20:04:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:21 smithi038 ceph-mon[92422]: pgmap v10310: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 83 KiB/s wr, 8 op/s 2023-03-09T20:04:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18835.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:04:23.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:23 smithi122 ceph-mon[93869]: pgmap v10311: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 67 KiB/s wr, 7 op/s 2023-03-09T20:04:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18373.542818 secs 2023-03-09T20:04:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:23 smithi181 ceph-mon[94189]: pgmap v10311: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 67 KiB/s wr, 7 op/s 2023-03-09T20:04:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18373.542818 secs 2023-03-09T20:04:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:23 smithi038 ceph-mon[92422]: pgmap v10311: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 67 KiB/s wr, 7 op/s 2023-03-09T20:04:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18373.542818 secs 2023-03-09T20:04:25.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:25 smithi122 ceph-mon[93869]: pgmap v10312: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 147 KiB/s wr, 7 op/s 2023-03-09T20:04:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:25 smithi181 ceph-mon[94189]: pgmap v10312: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 147 KiB/s wr, 7 op/s 2023-03-09T20:04:25.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:25 smithi038 ceph-mon[92422]: pgmap v10312: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 147 KiB/s wr, 7 op/s 2023-03-09T20:04:26.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18376.228774 secs 2023-03-09T20:04:26.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18376.228774 secs 2023-03-09T20:04:26.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18376.228774 secs 2023-03-09T20:04:27.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:27 smithi122 ceph-mon[93869]: pgmap v10313: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 147 KiB/s wr, 7 op/s 2023-03-09T20:04:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:27 smithi181 ceph-mon[94189]: pgmap v10313: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 147 KiB/s wr, 7 op/s 2023-03-09T20:04:27.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:27 smithi038 ceph-mon[92422]: pgmap v10313: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 147 KiB/s wr, 7 op/s 2023-03-09T20:04:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18840.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:04:28.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18378.542937 secs 2023-03-09T20:04:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18378.542937 secs 2023-03-09T20:04:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18378.542937 secs 2023-03-09T20:04:29.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:29 smithi122 ceph-mon[93869]: pgmap v10314: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 147 KiB/s wr, 7 op/s 2023-03-09T20:04:29.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:29 smithi181 ceph-mon[94189]: pgmap v10314: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 147 KiB/s wr, 7 op/s 2023-03-09T20:04:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:29 smithi038 ceph-mon[92422]: pgmap v10314: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 147 KiB/s wr, 7 op/s 2023-03-09T20:04:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18381.228833 secs 2023-03-09T20:04:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:31 smithi122 ceph-mon[93869]: pgmap v10315: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 73 KiB/s wr, 6 op/s 2023-03-09T20:04:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18381.228833 secs 2023-03-09T20:04:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:31 smithi181 ceph-mon[94189]: pgmap v10315: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 73 KiB/s wr, 6 op/s 2023-03-09T20:04:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18381.228833 secs 2023-03-09T20:04:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:31 smithi038 ceph-mon[92422]: pgmap v10315: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 73 KiB/s wr, 6 op/s 2023-03-09T20:04:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18845.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:04:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:33 smithi122 ceph-mon[93869]: pgmap v10316: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 69 KiB/s wr, 9 op/s 2023-03-09T20:04:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18383.543059 secs 2023-03-09T20:04:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:33 smithi181 ceph-mon[94189]: pgmap v10316: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 69 KiB/s wr, 9 op/s 2023-03-09T20:04:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18383.543059 secs 2023-03-09T20:04:33.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:33 smithi038 ceph-mon[92422]: pgmap v10316: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 69 KiB/s wr, 9 op/s 2023-03-09T20:04:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18383.543059 secs 2023-03-09T20:04:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:35 smithi122 ceph-mon[93869]: pgmap v10317: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 135 KiB/s wr, 7 op/s 2023-03-09T20:04:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:35 smithi181 ceph-mon[94189]: pgmap v10317: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 135 KiB/s wr, 7 op/s 2023-03-09T20:04:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:35 smithi038 ceph-mon[92422]: pgmap v10317: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 135 KiB/s wr, 7 op/s 2023-03-09T20:04:36.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18386.228895 secs 2023-03-09T20:04:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18386.228895 secs 2023-03-09T20:04:36.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18386.228895 secs 2023-03-09T20:04:37.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:37 smithi122 ceph-mon[93869]: pgmap v10318: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 68 KiB/s wr, 8 op/s 2023-03-09T20:04:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:37 smithi181 ceph-mon[94189]: pgmap v10318: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 68 KiB/s wr, 8 op/s 2023-03-09T20:04:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:37 smithi038 ceph-mon[92422]: pgmap v10318: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 68 KiB/s wr, 8 op/s 2023-03-09T20:04:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18850.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:04:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18388.543193 secs 2023-03-09T20:04:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18388.543193 secs 2023-03-09T20:04:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18388.543193 secs 2023-03-09T20:04:39.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:39 smithi122 ceph-mon[93869]: pgmap v10319: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 68 KiB/s wr, 6 op/s 2023-03-09T20:04:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:39 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:04:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:39 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:04:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:39 smithi181 ceph-mon[94189]: pgmap v10319: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 68 KiB/s wr, 6 op/s 2023-03-09T20:04:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:39 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:04:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:39 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:04:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:39 smithi038 ceph-mon[92422]: pgmap v10319: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 68 KiB/s wr, 6 op/s 2023-03-09T20:04:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:39 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:04:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:39 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:04:40.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:04:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:04:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:04:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:04:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:04:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:04:41.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18391.228953 secs 2023-03-09T20:04:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:41 smithi122 ceph-mon[93869]: pgmap v10320: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 158 KiB/s wr, 7 op/s 2023-03-09T20:04:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18391.228953 secs 2023-03-09T20:04:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:41 smithi181 ceph-mon[94189]: pgmap v10320: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 158 KiB/s wr, 7 op/s 2023-03-09T20:04:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18391.228953 secs 2023-03-09T20:04:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:41 smithi038 ceph-mon[92422]: pgmap v10320: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 158 KiB/s wr, 7 op/s 2023-03-09T20:04:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18855.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:04:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:43 smithi122 ceph-mon[93869]: pgmap v10321: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 172 KiB/s wr, 9 op/s 2023-03-09T20:04:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18393.543314 secs 2023-03-09T20:04:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:43 smithi181 ceph-mon[94189]: pgmap v10321: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 172 KiB/s wr, 9 op/s 2023-03-09T20:04:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18393.543314 secs 2023-03-09T20:04:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:43 smithi038 ceph-mon[92422]: pgmap v10321: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 172 KiB/s wr, 9 op/s 2023-03-09T20:04:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18393.543314 secs 2023-03-09T20:04:45.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:45 smithi122 ceph-mon[93869]: pgmap v10322: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 190 KiB/s wr, 8 op/s 2023-03-09T20:04:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:45 smithi181 ceph-mon[94189]: pgmap v10322: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 190 KiB/s wr, 8 op/s 2023-03-09T20:04:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:45 smithi038 ceph-mon[92422]: pgmap v10322: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 190 KiB/s wr, 8 op/s 2023-03-09T20:04:46.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18396.229014 secs 2023-03-09T20:04:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18396.229014 secs 2023-03-09T20:04:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18396.229014 secs 2023-03-09T20:04:47.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:47 smithi122 ceph-mon[93869]: pgmap v10323: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 137 KiB/s wr, 11 op/s 2023-03-09T20:04:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:47 smithi181 ceph-mon[94189]: pgmap v10323: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 137 KiB/s wr, 11 op/s 2023-03-09T20:04:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:47 smithi038 ceph-mon[92422]: pgmap v10323: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 137 KiB/s wr, 11 op/s 2023-03-09T20:04:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18860.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:04:48.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:48 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:04:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:48 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:04:48.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:48 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:04:49.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:49 smithi122 ceph-mon[93869]: pgmap v10324: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 137 KiB/s wr, 9 op/s 2023-03-09T20:04:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:49 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:04:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:04:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:04:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:49 smithi181 ceph-mon[94189]: pgmap v10324: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 137 KiB/s wr, 9 op/s 2023-03-09T20:04:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:49 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:04:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:04:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:04:49.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:49 smithi038 ceph-mon[92422]: pgmap v10324: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 137 KiB/s wr, 9 op/s 2023-03-09T20:04:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:49 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:04:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:04:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:04:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18401.229072 secs 2023-03-09T20:04:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:51 smithi122 ceph-mon[93869]: pgmap v10325: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 397 KiB/s wr, 11 op/s 2023-03-09T20:04:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:04:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:04:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18401.229072 secs 2023-03-09T20:04:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:51 smithi181 ceph-mon[94189]: pgmap v10325: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 397 KiB/s wr, 11 op/s 2023-03-09T20:04:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:04:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:04:51.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18401.229072 secs 2023-03-09T20:04:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:51 smithi038 ceph-mon[92422]: pgmap v10325: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 397 KiB/s wr, 11 op/s 2023-03-09T20:04:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:04:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:04:52.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:04:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:04:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:04:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:04:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:04:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:04:53.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18865.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:04:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:53 smithi122 ceph-mon[93869]: pgmap v10326: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 322 KiB/s wr, 12 op/s 2023-03-09T20:04:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:53 smithi181 ceph-mon[94189]: pgmap v10326: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 322 KiB/s wr, 12 op/s 2023-03-09T20:04:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:53 smithi038 ceph-mon[92422]: pgmap v10326: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 322 KiB/s wr, 12 op/s 2023-03-09T20:04:54.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:54 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:04:54.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:54 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:04:54.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:54 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:04:55.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:55 smithi122 ceph-mon[93869]: pgmap v10327: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 407 KiB/s wr, 12 op/s 2023-03-09T20:04:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18406.229134 secs 2023-03-09T20:04:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:55 smithi181 ceph-mon[94189]: pgmap v10327: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 407 KiB/s wr, 12 op/s 2023-03-09T20:04:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18406.229134 secs 2023-03-09T20:04:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:55 smithi038 ceph-mon[92422]: pgmap v10327: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 407 KiB/s wr, 12 op/s 2023-03-09T20:04:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18406.229134 secs 2023-03-09T20:04:57.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:57 smithi122 ceph-mon[93869]: pgmap v10328: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 403 KiB/s wr, 13 op/s 2023-03-09T20:04:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:57 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:04:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:57 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:04:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:57 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:04:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:57 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:04:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:57 smithi181 ceph-mon[94189]: pgmap v10328: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 403 KiB/s wr, 13 op/s 2023-03-09T20:04:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:57 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:04:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:57 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:04:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:57 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:04:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:57 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:04:57.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:57 smithi038 ceph-mon[92422]: pgmap v10328: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 403 KiB/s wr, 13 op/s 2023-03-09T20:04:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:57 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:04:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:57 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:04:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:57 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:04:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:57 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:04:58.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18870.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:04:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:04:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:04:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:04:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:04:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:04:58.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:04:58.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:04:58.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:04:58.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:04:58.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:04:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:04:58.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:04:58.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:04:58.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:04:58.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:04:58.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:04:58.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:04:58.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:04:58.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:04:58.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:04:58.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:04:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:04:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:04:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:04:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:04:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:04:59.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:04:59.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:04:59.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:04:59.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:04:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:59 smithi181 ceph-mon[94189]: pgmap v10329: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 388 KiB/s wr, 10 op/s 2023-03-09T20:04:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:59 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T20:04:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:59 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T20:04:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:59 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi038 to 115.8M 2023-03-09T20:04:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:59 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi038 to 121514606: error parsing value: Value '121514606' is below minimum 939524096 2023-03-09T20:04:59.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:04:59.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:04:59.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:04:59.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:04:59.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:59 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T20:04:59.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:59 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T20:04:59.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:04:59.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:04:59.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:04:59.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:04:59.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:04:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:04:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:59 smithi038 ceph-mon[92422]: pgmap v10329: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 388 KiB/s wr, 10 op/s 2023-03-09T20:04:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:59 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T20:04:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:59 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T20:04:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:59 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi038 to 115.8M 2023-03-09T20:04:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:59 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi038 to 121514606: error parsing value: Value '121514606' is below minimum 939524096 2023-03-09T20:05:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:05:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:05:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:05:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:05:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:59 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T20:05:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:59 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T20:05:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:05:00.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:05:00.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:05:00.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:05:00.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:04:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:05:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:59 smithi122 ceph-mon[93869]: pgmap v10329: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 388 KiB/s wr, 10 op/s 2023-03-09T20:05:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:59 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T20:05:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:59 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T20:05:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:59 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi038 to 115.8M 2023-03-09T20:05:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:59 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi038 to 121514606: error parsing value: Value '121514606' is below minimum 939524096 2023-03-09T20:05:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:05:00.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:05:00.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:05:00.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:05:00.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:59 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T20:05:00.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:59 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T20:05:00.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:05:00.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:05:00.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:05:00.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:05:00.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:04:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:05:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18411.229194 secs 2023-03-09T20:05:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:01 smithi122 ceph-mon[93869]: pgmap v10330: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 567 KiB/s rd, 566 KiB/s wr, 12 op/s 2023-03-09T20:05:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:05:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:05:01.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18411.229194 secs 2023-03-09T20:05:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:01 smithi181 ceph-mon[94189]: pgmap v10330: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 567 KiB/s rd, 566 KiB/s wr, 12 op/s 2023-03-09T20:05:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:05:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:05:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18411.229194 secs 2023-03-09T20:05:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:01 smithi038 ceph-mon[92422]: pgmap v10330: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 567 KiB/s rd, 566 KiB/s wr, 12 op/s 2023-03-09T20:05:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:05:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:05:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:03 smithi122 ceph-mon[93869]: pgmap v10331: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 320 KiB/s wr, 12 op/s 2023-03-09T20:05:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18413.543814 secs 2023-03-09T20:05:03.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:03 smithi181 ceph-mon[94189]: pgmap v10331: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 320 KiB/s wr, 12 op/s 2023-03-09T20:05:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18413.543814 secs 2023-03-09T20:05:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:03 smithi038 ceph-mon[92422]: pgmap v10331: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 320 KiB/s wr, 12 op/s 2023-03-09T20:05:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18413.543814 secs 2023-03-09T20:05:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18875.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:05:04.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:04 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:05:04.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:04 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:05:04.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:04 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:05:04.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:04 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:05:04.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:04 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:05:04.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:04 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:05:04.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:04 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:05:04.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:04 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:05:04.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:04 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:05:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:05 smithi122 ceph-mon[93869]: pgmap v10332: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 557 KiB/s rd, 482 KiB/s wr, 24 op/s 2023-03-09T20:05:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:05 smithi181 ceph-mon[94189]: pgmap v10332: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 557 KiB/s rd, 482 KiB/s wr, 24 op/s 2023-03-09T20:05:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:05 smithi038 ceph-mon[92422]: pgmap v10332: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 557 KiB/s rd, 482 KiB/s wr, 24 op/s 2023-03-09T20:05:06.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18416.229256 secs 2023-03-09T20:05:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18416.229256 secs 2023-03-09T20:05:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18416.229256 secs 2023-03-09T20:05:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:07 smithi122 ceph-mon[93869]: pgmap v10333: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 405 KiB/s wr, 26 op/s 2023-03-09T20:05:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:07 smithi181 ceph-mon[94189]: pgmap v10333: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 405 KiB/s wr, 26 op/s 2023-03-09T20:05:07.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:07 smithi038 ceph-mon[92422]: pgmap v10333: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 405 KiB/s wr, 26 op/s 2023-03-09T20:05:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18880.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:05:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18418.543932 secs 2023-03-09T20:05:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18418.543932 secs 2023-03-09T20:05:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18418.543932 secs 2023-03-09T20:05:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:09 smithi122 ceph-mon[93869]: pgmap v10334: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 497 KiB/s wr, 25 op/s 2023-03-09T20:05:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:09 smithi181 ceph-mon[94189]: pgmap v10334: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 497 KiB/s wr, 25 op/s 2023-03-09T20:05:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:09 smithi038 ceph-mon[92422]: pgmap v10334: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 497 KiB/s wr, 25 op/s 2023-03-09T20:05:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:10 smithi122 ceph-mon[93869]: osdmap e859: 12 total, 12 up, 12 in 2023-03-09T20:05:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:10 smithi181 ceph-mon[94189]: osdmap e859: 12 total, 12 up, 12 in 2023-03-09T20:05:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:10 smithi038 ceph-mon[92422]: osdmap e859: 12 total, 12 up, 12 in 2023-03-09T20:05:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18421.229316 secs 2023-03-09T20:05:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:11 smithi122 ceph-mon[93869]: pgmap v10336: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 383 KiB/s wr, 27 op/s 2023-03-09T20:05:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18421.229316 secs 2023-03-09T20:05:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:11 smithi181 ceph-mon[94189]: pgmap v10336: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 383 KiB/s wr, 27 op/s 2023-03-09T20:05:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18421.229316 secs 2023-03-09T20:05:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:11 smithi038 ceph-mon[92422]: pgmap v10336: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 383 KiB/s wr, 27 op/s 2023-03-09T20:05:13.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18885.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:05:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:13 smithi122 ceph-mon[93869]: pgmap v10337: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 383 KiB/s wr, 28 op/s 2023-03-09T20:05:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18423.544054 secs 2023-03-09T20:05:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:13 smithi181 ceph-mon[94189]: pgmap v10337: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 383 KiB/s wr, 28 op/s 2023-03-09T20:05:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18423.544054 secs 2023-03-09T20:05:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:13 smithi038 ceph-mon[92422]: pgmap v10337: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 383 KiB/s wr, 28 op/s 2023-03-09T20:05:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18423.544054 secs 2023-03-09T20:05:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:15 smithi122 ceph-mon[93869]: pgmap v10338: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 172 KiB/s wr, 11 op/s 2023-03-09T20:05:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:15 smithi181 ceph-mon[94189]: pgmap v10338: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 172 KiB/s wr, 11 op/s 2023-03-09T20:05:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:15 smithi038 ceph-mon[92422]: pgmap v10338: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 172 KiB/s wr, 11 op/s 2023-03-09T20:05:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18426.229376 secs 2023-03-09T20:05:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18426.229376 secs 2023-03-09T20:05:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18426.229376 secs 2023-03-09T20:05:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:17 smithi122 ceph-mon[93869]: pgmap v10339: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 161 KiB/s wr, 8 op/s 2023-03-09T20:05:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:17 smithi181 ceph-mon[94189]: pgmap v10339: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 161 KiB/s wr, 8 op/s 2023-03-09T20:05:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:17 smithi038 ceph-mon[92422]: pgmap v10339: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 161 KiB/s wr, 8 op/s 2023-03-09T20:05:18.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18890.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:05:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18428.544194 secs 2023-03-09T20:05:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:18 smithi122 ceph-mon[93869]: osdmap e860: 12 total, 12 up, 12 in 2023-03-09T20:05:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18428.544194 secs 2023-03-09T20:05:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:18 smithi181 ceph-mon[94189]: osdmap e860: 12 total, 12 up, 12 in 2023-03-09T20:05:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18428.544194 secs 2023-03-09T20:05:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:18 smithi038 ceph-mon[92422]: osdmap e860: 12 total, 12 up, 12 in 2023-03-09T20:05:19.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:19 smithi122 ceph-mon[93869]: pgmap v10340: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 88 KiB/s wr, 8 op/s 2023-03-09T20:05:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:19 smithi181 ceph-mon[94189]: pgmap v10340: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 88 KiB/s wr, 8 op/s 2023-03-09T20:05:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:19 smithi038 ceph-mon[92422]: pgmap v10340: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 88 KiB/s wr, 8 op/s 2023-03-09T20:05:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18431.229438 secs 2023-03-09T20:05:21.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:21 smithi122 ceph-mon[93869]: pgmap v10342: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 88 KiB/s wr, 8 op/s 2023-03-09T20:05:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18431.229438 secs 2023-03-09T20:05:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:21 smithi181 ceph-mon[94189]: pgmap v10342: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 88 KiB/s wr, 8 op/s 2023-03-09T20:05:21.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18431.229438 secs 2023-03-09T20:05:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:21 smithi038 ceph-mon[92422]: pgmap v10342: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 88 KiB/s wr, 8 op/s 2023-03-09T20:05:23.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18895.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:05:23.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:23 smithi122 ceph-mon[93869]: pgmap v10343: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 7 op/s 2023-03-09T20:05:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18433.544272 secs 2023-03-09T20:05:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:23 smithi181 ceph-mon[94189]: pgmap v10343: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 7 op/s 2023-03-09T20:05:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18433.544272 secs 2023-03-09T20:05:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:23 smithi038 ceph-mon[92422]: pgmap v10343: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 7 op/s 2023-03-09T20:05:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18433.544272 secs 2023-03-09T20:05:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:25 smithi122 ceph-mon[93869]: pgmap v10344: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 7 op/s 2023-03-09T20:05:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:25 smithi181 ceph-mon[94189]: pgmap v10344: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 7 op/s 2023-03-09T20:05:25.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:25 smithi038 ceph-mon[92422]: pgmap v10344: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 7 op/s 2023-03-09T20:05:26.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18436.229497 secs 2023-03-09T20:05:26.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18436.229497 secs 2023-03-09T20:05:27.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18436.229497 secs 2023-03-09T20:05:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:27 smithi181 ceph-mon[94189]: pgmap v10345: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 127 KiB/s wr, 8 op/s 2023-03-09T20:05:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:27 smithi038 ceph-mon[92422]: pgmap v10345: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 127 KiB/s wr, 8 op/s 2023-03-09T20:05:28.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:27 smithi122 ceph-mon[93869]: pgmap v10345: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 127 KiB/s wr, 8 op/s 2023-03-09T20:05:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18900.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:05:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18438.544391 secs 2023-03-09T20:05:29.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18438.544391 secs 2023-03-09T20:05:29.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18438.544391 secs 2023-03-09T20:05:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:29 smithi038 ceph-mon[92422]: pgmap v10346: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 117 KiB/s wr, 8 op/s 2023-03-09T20:05:30.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:29 smithi122 ceph-mon[93869]: pgmap v10346: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 117 KiB/s wr, 8 op/s 2023-03-09T20:05:30.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:29 smithi181 ceph-mon[94189]: pgmap v10346: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 117 KiB/s wr, 8 op/s 2023-03-09T20:05:31.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18441.229591 secs 2023-03-09T20:05:31.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18441.229591 secs 2023-03-09T20:05:31.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18441.229591 secs 2023-03-09T20:05:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:31 smithi122 ceph-mon[93869]: pgmap v10347: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 104 KiB/s wr, 7 op/s 2023-03-09T20:05:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:31 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:05:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:31 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:05:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:31 smithi181 ceph-mon[94189]: pgmap v10347: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 104 KiB/s wr, 7 op/s 2023-03-09T20:05:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:31 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:05:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:31 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:05:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:31 smithi038 ceph-mon[92422]: pgmap v10347: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 104 KiB/s wr, 7 op/s 2023-03-09T20:05:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:31 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:05:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:31 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:05:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:32 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:05:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:32 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:05:33.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:32 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:05:33.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:32 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:05:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:32 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:05:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:32 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:05:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18905.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:05:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:33 smithi122 ceph-mon[93869]: pgmap v10348: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 98 KiB/s wr, 9 op/s 2023-03-09T20:05:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18443.544497 secs 2023-03-09T20:05:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:33 smithi181 ceph-mon[94189]: pgmap v10348: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 98 KiB/s wr, 9 op/s 2023-03-09T20:05:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18443.544497 secs 2023-03-09T20:05:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:33 smithi038 ceph-mon[92422]: pgmap v10348: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 98 KiB/s wr, 9 op/s 2023-03-09T20:05:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18443.544497 secs 2023-03-09T20:05:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:35 smithi122 ceph-mon[93869]: pgmap v10349: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 97 KiB/s wr, 8 op/s 2023-03-09T20:05:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18446.229676 secs 2023-03-09T20:05:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:35 smithi181 ceph-mon[94189]: pgmap v10349: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 97 KiB/s wr, 8 op/s 2023-03-09T20:05:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18446.229676 secs 2023-03-09T20:05:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:35 smithi038 ceph-mon[92422]: pgmap v10349: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 97 KiB/s wr, 8 op/s 2023-03-09T20:05:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18446.229676 secs 2023-03-09T20:05:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:37 smithi122 ceph-mon[93869]: pgmap v10350: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 164 KiB/s wr, 10 op/s 2023-03-09T20:05:38.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:37 smithi181 ceph-mon[94189]: pgmap v10350: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 164 KiB/s wr, 10 op/s 2023-03-09T20:05:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:37 smithi038 ceph-mon[92422]: pgmap v10350: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 164 KiB/s wr, 10 op/s 2023-03-09T20:05:38.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18910.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:05:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:39 smithi122 ceph-mon[93869]: pgmap v10351: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 104 KiB/s wr, 7 op/s 2023-03-09T20:05:40.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:39 smithi181 ceph-mon[94189]: pgmap v10351: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 104 KiB/s wr, 7 op/s 2023-03-09T20:05:40.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:39 smithi038 ceph-mon[92422]: pgmap v10351: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 104 KiB/s wr, 7 op/s 2023-03-09T20:05:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18451.229738 secs 2023-03-09T20:05:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:40 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:05:41.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18451.229738 secs 2023-03-09T20:05:41.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:40 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:05:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18451.229738 secs 2023-03-09T20:05:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:40 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:05:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:41 smithi122 ceph-mon[93869]: pgmap v10352: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 68 KiB/s wr, 6 op/s 2023-03-09T20:05:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:41 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:05:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:05:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:05:42.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:41 smithi181 ceph-mon[94189]: pgmap v10352: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 68 KiB/s wr, 6 op/s 2023-03-09T20:05:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:41 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:05:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:05:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:05:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:41 smithi038 ceph-mon[92422]: pgmap v10352: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 68 KiB/s wr, 6 op/s 2023-03-09T20:05:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:41 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:05:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:05:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:05:43.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:42 smithi122 ceph-mon[93869]: pgmap v10353: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 140 KiB/s wr, 9 op/s 2023-03-09T20:05:43.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:42 smithi181 ceph-mon[94189]: pgmap v10353: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 140 KiB/s wr, 9 op/s 2023-03-09T20:05:43.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:42 smithi038 ceph-mon[92422]: pgmap v10353: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 140 KiB/s wr, 9 op/s 2023-03-09T20:05:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18915.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:05:45.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:45 smithi122 ceph-mon[93869]: pgmap v10354: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 140 KiB/s wr, 7 op/s 2023-03-09T20:05:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:45 smithi181 ceph-mon[94189]: pgmap v10354: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 140 KiB/s wr, 7 op/s 2023-03-09T20:05:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:45 smithi038 ceph-mon[92422]: pgmap v10354: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 140 KiB/s wr, 7 op/s 2023-03-09T20:05:46.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18456.229799 secs 2023-03-09T20:05:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18456.229799 secs 2023-03-09T20:05:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18456.229799 secs 2023-03-09T20:05:47.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:47 smithi122 ceph-mon[93869]: pgmap v10355: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 297 KiB/s wr, 10 op/s 2023-03-09T20:05:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:47 smithi181 ceph-mon[94189]: pgmap v10355: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 297 KiB/s wr, 10 op/s 2023-03-09T20:05:47.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:47 smithi038 ceph-mon[92422]: pgmap v10355: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 297 KiB/s wr, 10 op/s 2023-03-09T20:05:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18920.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:05:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:49 smithi122 ceph-mon[93869]: pgmap v10356: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 247 KiB/s rd, 230 KiB/s wr, 7 op/s 2023-03-09T20:05:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:49 smithi181 ceph-mon[94189]: pgmap v10356: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 247 KiB/s rd, 230 KiB/s wr, 7 op/s 2023-03-09T20:05:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:49 smithi038 ceph-mon[92422]: pgmap v10356: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 247 KiB/s rd, 230 KiB/s wr, 7 op/s 2023-03-09T20:05:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18461.229893 secs 2023-03-09T20:05:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:51 smithi122 ceph-mon[93869]: pgmap v10357: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 230 KiB/s wr, 7 op/s 2023-03-09T20:05:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18461.229893 secs 2023-03-09T20:05:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:51 smithi181 ceph-mon[94189]: pgmap v10357: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 230 KiB/s wr, 7 op/s 2023-03-09T20:05:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18461.229893 secs 2023-03-09T20:05:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:51 smithi038 ceph-mon[92422]: pgmap v10357: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 230 KiB/s wr, 7 op/s 2023-03-09T20:05:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:52 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:05:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:05:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:05:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:52 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:05:52.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:05:52.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:05:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:52 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:05:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:05:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:05:53.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18925.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:05:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:53 smithi122 ceph-mon[93869]: pgmap v10358: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 256 KiB/s wr, 11 op/s 2023-03-09T20:05:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:53 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 18463.544888 secs 2023-03-09T20:05:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:53 smithi181 ceph-mon[94189]: pgmap v10358: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 256 KiB/s wr, 11 op/s 2023-03-09T20:05:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:53 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 18463.544888 secs 2023-03-09T20:05:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:53 smithi038 ceph-mon[92422]: pgmap v10358: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 256 KiB/s wr, 11 op/s 2023-03-09T20:05:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:53 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 18463.544888 secs 2023-03-09T20:05:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:55 smithi122 ceph-mon[93869]: pgmap v10359: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 184 KiB/s wr, 9 op/s 2023-03-09T20:05:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:05:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:05:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:55 smithi181 ceph-mon[94189]: pgmap v10359: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 184 KiB/s wr, 9 op/s 2023-03-09T20:05:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:05:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:05:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:55 smithi038 ceph-mon[92422]: pgmap v10359: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 184 KiB/s wr, 9 op/s 2023-03-09T20:05:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:05:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:05:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18466.229952 secs 2023-03-09T20:05:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:05:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:05:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18466.229952 secs 2023-03-09T20:05:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:05:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:05:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18466.229952 secs 2023-03-09T20:05:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:05:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:05:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:57 smithi122 ceph-mon[93869]: pgmap v10360: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 472 KiB/s wr, 13 op/s 2023-03-09T20:05:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:57 smithi181 ceph-mon[94189]: pgmap v10360: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 472 KiB/s wr, 13 op/s 2023-03-09T20:05:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:57 smithi038 ceph-mon[92422]: pgmap v10360: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 472 KiB/s wr, 13 op/s 2023-03-09T20:05:58.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18930.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:05:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:58 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 18468.545011 secs 2023-03-09T20:05:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:58 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 18468.545011 secs 2023-03-09T20:05:58.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:58 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 18468.545011 secs 2023-03-09T20:05:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:59 smithi122 ceph-mon[93869]: pgmap v10361: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 315 KiB/s wr, 10 op/s 2023-03-09T20:05:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:05:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:05:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:05:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:05:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:05:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:59 smithi181 ceph-mon[94189]: pgmap v10361: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 315 KiB/s wr, 10 op/s 2023-03-09T20:05:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:05:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:05:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:05:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:05:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:05:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:59 smithi038 ceph-mon[92422]: pgmap v10361: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 315 KiB/s wr, 10 op/s 2023-03-09T20:05:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:05:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:05:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:05:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:05:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:06:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:06:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18471.230036 secs 2023-03-09T20:06:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:06:01 smithi122 ceph-mon[93869]: pgmap v10362: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 315 KiB/s wr, 11 op/s 2023-03-09T20:06:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:06:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18471.230036 secs 2023-03-09T20:06:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:06:01 smithi181 ceph-mon[94189]: pgmap v10362: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 315 KiB/s wr, 11 op/s 2023-03-09T20:06:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:06:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18471.230036 secs 2023-03-09T20:06:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:06:01 smithi038 ceph-mon[92422]: pgmap v10362: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 315 KiB/s wr, 11 op/s 2023-03-09T20:06:03.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18935.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:06:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:06:03 smithi122 ceph-mon[93869]: pgmap v10363: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 454 KiB/s rd, 388 KiB/s wr, 15 op/s 2023-03-09T20:06:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:06:03 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 18473.545148 secs 2023-03-09T20:06:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:06:03 smithi181 ceph-mon[94189]: pgmap v10363: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 454 KiB/s rd, 388 KiB/s wr, 15 op/s 2023-03-09T20:06:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:06:03 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 18473.545148 secs 2023-03-09T20:06:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:06:03 smithi038 ceph-mon[92422]: pgmap v10363: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 454 KiB/s rd, 388 KiB/s wr, 15 op/s 2023-03-09T20:06:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:06:03 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 18473.545148 secs 2023-03-09T20:06:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:06:05 smithi122 ceph-mon[93869]: pgmap v10364: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 383 KiB/s wr, 22 op/s 2023-03-09T20:06:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:06:05 smithi181 ceph-mon[94189]: pgmap v10364: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 383 KiB/s wr, 22 op/s 2023-03-09T20:06:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:06:05 smithi038 ceph-mon[92422]: pgmap v10364: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 383 KiB/s wr, 22 op/s 2023-03-09T20:06:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:06:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18476.230097 secs 2023-03-09T20:06:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:06:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18476.230097 secs 2023-03-09T20:06:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:06:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18476.230097 secs 2023-03-09T20:06:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:06:07 smithi122 ceph-mon[93869]: pgmap v10365: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 503 KiB/s rd, 420 KiB/s wr, 28 op/s 2023-03-09T20:06:07.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:06:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:06:07.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:06:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:06:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:06:07 smithi181 ceph-mon[94189]: pgmap v10365: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 503 KiB/s rd, 420 KiB/s wr, 28 op/s 2023-03-09T20:06:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:06:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:06:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:06:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:06:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:06:07 smithi038 ceph-mon[92422]: pgmap v10365: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 503 KiB/s rd, 420 KiB/s wr, 28 op/s 2023-03-09T20:06:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:06:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:06:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:06:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:06:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18940.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:06:08.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:06:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18478.545231 secs 2023-03-09T20:06:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:06:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:06:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:06:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:06:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:06:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18478.545231 secs 2023-03-09T20:06:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:06:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:06:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:06:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:06:09.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:06:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18478.545231 secs 2023-03-09T20:06:09.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:06:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:06:09.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:06:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:06:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:06:09 smithi038 ceph-mon[92422]: pgmap v10366: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 432 KiB/s rd, 320 KiB/s wr, 25 op/s 2023-03-09T20:06:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:06:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:06:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:06:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:06:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:06:09 smithi122 ceph-mon[93869]: pgmap v10366: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 432 KiB/s rd, 320 KiB/s wr, 25 op/s 2023-03-09T20:06:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:06:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:06:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:06:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:06:10.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:06:09 smithi181 ceph-mon[94189]: pgmap v10366: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 432 KiB/s rd, 320 KiB/s wr, 25 op/s 2023-03-09T20:06:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:06:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:06:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:06:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:06:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:06:10 smithi122 ceph-mon[93869]: osdmap e861: 12 total, 12 up, 12 in 2023-03-09T20:06:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:06:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18481.230161 secs 2023-03-09T20:06:11.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:06:10 smithi181 ceph-mon[94189]: osdmap e861: 12 total, 12 up, 12 in 2023-03-09T20:06:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:06:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18481.230161 secs 2023-03-09T20:06:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:06:10 smithi038 ceph-mon[92422]: osdmap e861: 12 total, 12 up, 12 in 2023-03-09T20:06:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:06:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18481.230161 secs 2023-03-09T20:06:12.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:06:11 smithi122 ceph-mon[93869]: pgmap v10368: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 383 KiB/s wr, 29 op/s 2023-03-09T20:06:12.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:06:11 smithi181 ceph-mon[94189]: pgmap v10368: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 383 KiB/s wr, 29 op/s 2023-03-09T20:06:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:06:11 smithi038 ceph-mon[92422]: pgmap v10368: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 383 KiB/s wr, 29 op/s 2023-03-09T20:06:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18945.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:06:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:06:13 smithi122 ceph-mon[93869]: pgmap v10369: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 314 KiB/s wr, 28 op/s 2023-03-09T20:06:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:06:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18483.545311 secs 2023-03-09T20:06:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:06:13 smithi181 ceph-mon[94189]: pgmap v10369: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 314 KiB/s wr, 28 op/s 2023-03-09T20:06:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:06:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18483.545311 secs 2023-03-09T20:06:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:06:13 smithi038 ceph-mon[92422]: pgmap v10369: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 314 KiB/s wr, 28 op/s 2023-03-09T20:06:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:06:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18483.545311 secs 2023-03-09T20:06:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:06:15 smithi122 ceph-mon[93869]: pgmap v10370: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 343 KiB/s wr, 14 op/s 2023-03-09T20:06:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:06:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18486.230221 secs 2023-03-09T20:06:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:06:15 smithi181 ceph-mon[94189]: pgmap v10370: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 343 KiB/s wr, 14 op/s 2023-03-09T20:06:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:06:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18486.230221 secs 2023-03-09T20:06:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:06:15 smithi038 ceph-mon[92422]: pgmap v10370: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 343 KiB/s wr, 14 op/s 2023-03-09T20:06:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:06:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18486.230221 secs 2023-03-09T20:06:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:06:17 smithi122 ceph-mon[93869]: pgmap v10371: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 299 KiB/s wr, 10 op/s 2023-03-09T20:06:18.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:06:17 smithi181 ceph-mon[94189]: pgmap v10371: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 299 KiB/s wr, 10 op/s 2023-03-09T20:06:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:06:17 smithi038 ceph-mon[92422]: pgmap v10371: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 299 KiB/s wr, 10 op/s 2023-03-09T20:06:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18950.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:06:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:06:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18488.545377 secs 2023-03-09T20:06:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:06:18 smithi122 ceph-mon[93869]: osdmap e862: 12 total, 12 up, 12 in 2023-03-09T20:06:19.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:06:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18488.545377 secs 2023-03-09T20:06:19.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:06:18 smithi181 ceph-mon[94189]: osdmap e862: 12 total, 12 up, 12 in 2023-03-09T20:06:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:06:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18488.545377 secs 2023-03-09T20:06:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:06:18 smithi038 ceph-mon[92422]: osdmap e862: 12 total, 12 up, 12 in 2023-03-09T20:06:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:06:19 smithi122 ceph-mon[93869]: pgmap v10372: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 73 KiB/s wr, 8 op/s 2023-03-09T20:06:20.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:06:19 smithi181 ceph-mon[94189]: pgmap v10372: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 73 KiB/s wr, 8 op/s 2023-03-09T20:06:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:06:19 smithi038 ceph-mon[92422]: pgmap v10372: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 73 KiB/s wr, 8 op/s 2023-03-09T20:06:21.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:06:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18491.230281 secs 2023-03-09T20:06:21.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:06:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18491.230281 secs 2023-03-09T20:06:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:06:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18491.230281 secs 2023-03-09T20:06:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:06:21 smithi122 ceph-mon[93869]: pgmap v10374: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 73 KiB/s wr, 8 op/s 2023-03-09T20:06:22.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:06:21 smithi181 ceph-mon[94189]: pgmap v10374: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 73 KiB/s wr, 8 op/s 2023-03-09T20:06:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:06:21 smithi038 ceph-mon[92422]: pgmap v10374: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 73 KiB/s wr, 8 op/s 2023-03-09T20:06:23.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18955.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:06:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:06:23 smithi122 ceph-mon[93869]: pgmap v10375: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 55 KiB/s wr, 7 op/s 2023-03-09T20:06:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:06:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18493.545452 secs 2023-03-09T20:06:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:06:23 smithi181 ceph-mon[94189]: pgmap v10375: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 55 KiB/s wr, 7 op/s 2023-03-09T20:06:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:06:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18493.545452 secs 2023-03-09T20:06:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:06:23 smithi038 ceph-mon[92422]: pgmap v10375: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 55 KiB/s wr, 7 op/s 2023-03-09T20:06:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:06:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18493.545452 secs 2023-03-09T20:06:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:06:25 smithi122 ceph-mon[93869]: pgmap v10376: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T20:06:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:06:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18496.230342 secs 2023-03-09T20:06:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:06:25 smithi181 ceph-mon[94189]: pgmap v10376: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T20:06:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:06:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18496.230342 secs 2023-03-09T20:06:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:06:25 smithi038 ceph-mon[92422]: pgmap v10376: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T20:06:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:06:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18496.230342 secs 2023-03-09T20:06:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:06:26 smithi122 ceph-mon[93869]: pgmap v10377: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2023-03-09T20:06:27.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:06:26 smithi181 ceph-mon[94189]: pgmap v10377: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2023-03-09T20:06:27.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:06:26 smithi038 ceph-mon[92422]: pgmap v10377: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2023-03-09T20:06:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:06:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18498.545570 secs 2023-03-09T20:06:28.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:06:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18498.545570 secs 2023-03-09T20:06:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:06:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18498.545570 secs 2023-03-09T20:06:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18960.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:06:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:06:29 smithi122 ceph-mon[93869]: pgmap v10378: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2023-03-09T20:06:29.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:06:29 smithi181 ceph-mon[94189]: pgmap v10378: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2023-03-09T20:06:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:06:29 smithi038 ceph-mon[92422]: pgmap v10378: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2023-03-09T20:06:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:06:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18501.230401 secs 2023-03-09T20:06:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:06:31 smithi122 ceph-mon[93869]: pgmap v10379: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2023-03-09T20:06:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:06:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18501.230401 secs 2023-03-09T20:06:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:06:31 smithi181 ceph-mon[94189]: pgmap v10379: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2023-03-09T20:06:32.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:06:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18501.230401 secs 2023-03-09T20:06:32.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:06:31 smithi038 ceph-mon[92422]: pgmap v10379: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2023-03-09T20:06:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18965.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:06:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:06:33 smithi122 ceph-mon[93869]: pgmap v10380: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2023-03-09T20:06:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:06:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18503.545693 secs 2023-03-09T20:06:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:06:33 smithi181 ceph-mon[94189]: pgmap v10380: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2023-03-09T20:06:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:06:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18503.545693 secs 2023-03-09T20:06:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:06:33 smithi038 ceph-mon[92422]: pgmap v10380: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2023-03-09T20:06:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:06:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18503.545693 secs 2023-03-09T20:06:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:06:35 smithi122 ceph-mon[93869]: pgmap v10381: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 69 KiB/s wr, 6 op/s 2023-03-09T20:06:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:06:35 smithi181 ceph-mon[94189]: pgmap v10381: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 69 KiB/s wr, 6 op/s 2023-03-09T20:06:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:06:35 smithi038 ceph-mon[92422]: pgmap v10381: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 69 KiB/s wr, 6 op/s 2023-03-09T20:06:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:06:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18506.230460 secs 2023-03-09T20:06:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:06:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18506.230460 secs 2023-03-09T20:06:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:06:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18506.230460 secs 2023-03-09T20:06:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:06:37 smithi122 ceph-mon[93869]: pgmap v10382: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 69 KiB/s wr, 8 op/s 2023-03-09T20:06:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:06:37 smithi181 ceph-mon[94189]: pgmap v10382: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 69 KiB/s wr, 8 op/s 2023-03-09T20:06:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:06:37 smithi038 ceph-mon[92422]: pgmap v10382: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 69 KiB/s wr, 8 op/s 2023-03-09T20:06:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18970.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:06:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:06:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18508.545819 secs 2023-03-09T20:06:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:06:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18508.545819 secs 2023-03-09T20:06:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:06:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18508.545819 secs 2023-03-09T20:06:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:06:39 smithi122 ceph-mon[93869]: pgmap v10383: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 109 KiB/s wr, 7 op/s 2023-03-09T20:06:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:06:39 smithi181 ceph-mon[94189]: pgmap v10383: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 109 KiB/s wr, 7 op/s 2023-03-09T20:06:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:06:39 smithi038 ceph-mon[92422]: pgmap v10383: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 109 KiB/s wr, 7 op/s 2023-03-09T20:06:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:06:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:06:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:06:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:06:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:06:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:06:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:06:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:06:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:06:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:06:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:06:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:06:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:06:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18511.230519 secs 2023-03-09T20:06:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:06:41 smithi122 ceph-mon[93869]: pgmap v10384: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 109 KiB/s wr, 6 op/s 2023-03-09T20:06:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:06:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18511.230519 secs 2023-03-09T20:06:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:06:41 smithi181 ceph-mon[94189]: pgmap v10384: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 109 KiB/s wr, 6 op/s 2023-03-09T20:06:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:06:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18511.230519 secs 2023-03-09T20:06:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:06:41 smithi038 ceph-mon[92422]: pgmap v10384: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 109 KiB/s wr, 6 op/s 2023-03-09T20:06:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18975.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:06:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:06:43 smithi122 ceph-mon[93869]: pgmap v10385: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 141 KiB/s wr, 9 op/s 2023-03-09T20:06:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:06:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18513.545932 secs 2023-03-09T20:06:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:06:43 smithi181 ceph-mon[94189]: pgmap v10385: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 141 KiB/s wr, 9 op/s 2023-03-09T20:06:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:06:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18513.545932 secs 2023-03-09T20:06:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:06:43 smithi038 ceph-mon[92422]: pgmap v10385: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 141 KiB/s wr, 9 op/s 2023-03-09T20:06:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:06:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18513.545932 secs 2023-03-09T20:06:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:06:45 smithi122 ceph-mon[93869]: pgmap v10386: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 141 KiB/s wr, 7 op/s 2023-03-09T20:06:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:06:45 smithi181 ceph-mon[94189]: pgmap v10386: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 141 KiB/s wr, 7 op/s 2023-03-09T20:06:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:06:45 smithi038 ceph-mon[92422]: pgmap v10386: 129 pgs: 129 active+clean; 9.8 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 141 KiB/s wr, 7 op/s 2023-03-09T20:06:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:06:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18516.230581 secs 2023-03-09T20:06:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:06:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18516.230581 secs 2023-03-09T20:06:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:06:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18516.230581 secs 2023-03-09T20:06:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:06:47 smithi122 ceph-mon[93869]: pgmap v10387: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 316 KiB/s wr, 11 op/s 2023-03-09T20:06:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:06:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:06:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:06:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:06:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:06:47 smithi181 ceph-mon[94189]: pgmap v10387: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 316 KiB/s wr, 11 op/s 2023-03-09T20:06:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:06:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:06:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:06:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:06:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:06:47 smithi038 ceph-mon[92422]: pgmap v10387: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 316 KiB/s wr, 11 op/s 2023-03-09T20:06:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:06:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:06:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:06:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:06:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18980.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:06:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:06:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18518.546082 secs 2023-03-09T20:06:48.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:06:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18518.546082 secs 2023-03-09T20:06:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:06:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18518.546082 secs 2023-03-09T20:06:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:06:49 smithi122 ceph-mon[93869]: pgmap v10388: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 494 KiB/s rd, 316 KiB/s wr, 9 op/s 2023-03-09T20:06:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:06:49 smithi181 ceph-mon[94189]: pgmap v10388: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 494 KiB/s rd, 316 KiB/s wr, 9 op/s 2023-03-09T20:06:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:06:49 smithi038 ceph-mon[92422]: pgmap v10388: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 494 KiB/s rd, 316 KiB/s wr, 9 op/s 2023-03-09T20:06:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:06:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18521.230643 secs 2023-03-09T20:06:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:06:51 smithi122 ceph-mon[93869]: pgmap v10389: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 275 KiB/s wr, 8 op/s 2023-03-09T20:06:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:06:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18521.230643 secs 2023-03-09T20:06:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:06:51 smithi181 ceph-mon[94189]: pgmap v10389: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 275 KiB/s wr, 8 op/s 2023-03-09T20:06:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:06:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18521.230643 secs 2023-03-09T20:06:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:06:51 smithi038 ceph-mon[92422]: pgmap v10389: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 275 KiB/s wr, 8 op/s 2023-03-09T20:06:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18985.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:06:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:06:53 smithi122 ceph-mon[93869]: pgmap v10390: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 442 KiB/s rd, 303 KiB/s wr, 11 op/s 2023-03-09T20:06:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:06:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18523.546196 secs 2023-03-09T20:06:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:06:53 smithi181 ceph-mon[94189]: pgmap v10390: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 442 KiB/s rd, 303 KiB/s wr, 11 op/s 2023-03-09T20:06:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:06:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18523.546196 secs 2023-03-09T20:06:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:06:53 smithi038 ceph-mon[92422]: pgmap v10390: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 442 KiB/s rd, 303 KiB/s wr, 11 op/s 2023-03-09T20:06:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:06:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18523.546196 secs 2023-03-09T20:06:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:06:55 smithi122 ceph-mon[93869]: pgmap v10391: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 271 KiB/s wr, 9 op/s 2023-03-09T20:06:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:06:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:06:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:06:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:06:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:06:55 smithi181 ceph-mon[94189]: pgmap v10391: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 271 KiB/s wr, 9 op/s 2023-03-09T20:06:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:06:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:06:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:06:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:06:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:06:55 smithi038 ceph-mon[92422]: pgmap v10391: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 271 KiB/s wr, 9 op/s 2023-03-09T20:06:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:06:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:06:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:06:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:06:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:06:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18526.230705 secs 2023-03-09T20:06:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:06:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:06:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:06:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:06:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:06:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18526.230705 secs 2023-03-09T20:06:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:06:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:06:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:06:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:06:57.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:06:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18526.230705 secs 2023-03-09T20:06:57.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:06:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:06:57.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:06:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:06:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:06:57 smithi181 ceph-mon[94189]: pgmap v10392: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 537 KiB/s rd, 521 KiB/s wr, 13 op/s 2023-03-09T20:06:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:06:57 smithi038 ceph-mon[92422]: pgmap v10392: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 537 KiB/s rd, 521 KiB/s wr, 13 op/s 2023-03-09T20:06:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:06:57 smithi122 ceph-mon[93869]: pgmap v10392: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 537 KiB/s rd, 521 KiB/s wr, 13 op/s 2023-03-09T20:06:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18990.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:06:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:06:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18528.546339 secs 2023-03-09T20:06:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:06:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18528.546339 secs 2023-03-09T20:06:59.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:06:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18528.546339 secs 2023-03-09T20:06:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:06:59 smithi038 ceph-mon[92422]: pgmap v10393: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 279 KiB/s wr, 9 op/s 2023-03-09T20:06:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:06:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:07:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:06:59 smithi122 ceph-mon[93869]: pgmap v10393: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 279 KiB/s wr, 9 op/s 2023-03-09T20:07:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:06:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:07:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:06:59 smithi181 ceph-mon[94189]: pgmap v10393: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 279 KiB/s wr, 9 op/s 2023-03-09T20:07:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:06:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:07:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:07:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18531.230767 secs 2023-03-09T20:07:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:07:01 smithi122 ceph-mon[93869]: pgmap v10394: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 279 KiB/s wr, 9 op/s 2023-03-09T20:07:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:07:01 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:07:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:07:01 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:07:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:07:01 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:07:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:07:01 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:07:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:07:01 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:07:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:07:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18531.230767 secs 2023-03-09T20:07:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:07:01 smithi181 ceph-mon[94189]: pgmap v10394: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 279 KiB/s wr, 9 op/s 2023-03-09T20:07:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:07:01 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:07:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:07:01 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:07:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:07:01 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:07:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:07:01 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:07:01.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:07:01 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:07:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:07:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18531.230767 secs 2023-03-09T20:07:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:07:01 smithi038 ceph-mon[92422]: pgmap v10394: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 279 KiB/s wr, 9 op/s 2023-03-09T20:07:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:07:01 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:07:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:07:01 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:07:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:07:01 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:07:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:07:01 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:07:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:07:01 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:07:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:07:03 smithi122 ceph-mon[93869]: pgmap v10395: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 454 KiB/s wr, 15 op/s 2023-03-09T20:07:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:07:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18533.546456 secs 2023-03-09T20:07:03.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:07:03 smithi181 ceph-mon[94189]: pgmap v10395: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 454 KiB/s wr, 15 op/s 2023-03-09T20:07:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:07:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18533.546456 secs 2023-03-09T20:07:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:07:03 smithi038 ceph-mon[92422]: pgmap v10395: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 454 KiB/s wr, 15 op/s 2023-03-09T20:07:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:07:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18533.546456 secs 2023-03-09T20:07:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 18995.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:07:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:07:05 smithi122 ceph-mon[93869]: pgmap v10396: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 448 KiB/s wr, 24 op/s 2023-03-09T20:07:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:07:05 smithi181 ceph-mon[94189]: pgmap v10396: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 448 KiB/s wr, 24 op/s 2023-03-09T20:07:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:07:05 smithi038 ceph-mon[92422]: pgmap v10396: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 448 KiB/s wr, 24 op/s 2023-03-09T20:07:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:07:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18536.230826 secs 2023-03-09T20:07:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:07:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18536.230826 secs 2023-03-09T20:07:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:07:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18536.230826 secs 2023-03-09T20:07:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:07:07 smithi122 ceph-mon[93869]: pgmap v10397: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 452 KiB/s rd, 485 KiB/s wr, 27 op/s 2023-03-09T20:07:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:07:07 smithi181 ceph-mon[94189]: pgmap v10397: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 452 KiB/s rd, 485 KiB/s wr, 27 op/s 2023-03-09T20:07:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:07:07 smithi038 ceph-mon[92422]: pgmap v10397: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 452 KiB/s rd, 485 KiB/s wr, 27 op/s 2023-03-09T20:07:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19000.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:07:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:07:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18538.546577 secs 2023-03-09T20:07:08.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:07:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18538.546577 secs 2023-03-09T20:07:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:07:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18538.546577 secs 2023-03-09T20:07:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:07:09 smithi122 ceph-mon[93869]: pgmap v10398: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 235 KiB/s wr, 22 op/s 2023-03-09T20:07:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:07:09 smithi181 ceph-mon[94189]: pgmap v10398: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 235 KiB/s wr, 22 op/s 2023-03-09T20:07:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:07:09 smithi038 ceph-mon[92422]: pgmap v10398: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 235 KiB/s wr, 22 op/s 2023-03-09T20:07:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:07:10 smithi122 ceph-mon[93869]: osdmap e863: 12 total, 12 up, 12 in 2023-03-09T20:07:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:07:10 smithi181 ceph-mon[94189]: osdmap e863: 12 total, 12 up, 12 in 2023-03-09T20:07:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:07:10 smithi038 ceph-mon[92422]: osdmap e863: 12 total, 12 up, 12 in 2023-03-09T20:07:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:07:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18541.230890 secs 2023-03-09T20:07:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:07:11 smithi122 ceph-mon[93869]: pgmap v10400: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 386 KiB/s wr, 27 op/s 2023-03-09T20:07:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:07:11 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:07:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:07:11 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:07:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:07:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18541.230890 secs 2023-03-09T20:07:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:07:11 smithi181 ceph-mon[94189]: pgmap v10400: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 386 KiB/s wr, 27 op/s 2023-03-09T20:07:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:07:11 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:07:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:07:11 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:07:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:07:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18541.230890 secs 2023-03-09T20:07:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:07:11 smithi038 ceph-mon[92422]: pgmap v10400: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 386 KiB/s wr, 27 op/s 2023-03-09T20:07:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:07:11 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:07:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:07:11 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:07:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:07:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:07:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:07:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:07:12.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:07:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:07:12.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:07:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:07:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:07:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:07:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:07:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:07:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19005.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:07:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:07:13 smithi122 ceph-mon[93869]: pgmap v10401: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 211 KiB/s wr, 23 op/s 2023-03-09T20:07:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:07:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18543.546697 secs 2023-03-09T20:07:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:07:13 smithi181 ceph-mon[94189]: pgmap v10401: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 211 KiB/s wr, 23 op/s 2023-03-09T20:07:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:07:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18543.546697 secs 2023-03-09T20:07:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:07:13 smithi038 ceph-mon[92422]: pgmap v10401: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 211 KiB/s wr, 23 op/s 2023-03-09T20:07:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:07:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18543.546697 secs 2023-03-09T20:07:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:07:15 smithi122 ceph-mon[93869]: pgmap v10402: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 263 KiB/s wr, 9 op/s 2023-03-09T20:07:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:07:15 smithi181 ceph-mon[94189]: pgmap v10402: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 263 KiB/s wr, 9 op/s 2023-03-09T20:07:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:07:15 smithi038 ceph-mon[92422]: pgmap v10402: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 263 KiB/s wr, 9 op/s 2023-03-09T20:07:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:07:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18546.230951 secs 2023-03-09T20:07:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:07:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18546.230951 secs 2023-03-09T20:07:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:07:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18546.230951 secs 2023-03-09T20:07:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:07:17 smithi122 ceph-mon[93869]: pgmap v10403: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 218 KiB/s wr, 9 op/s 2023-03-09T20:07:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:07:17 smithi181 ceph-mon[94189]: pgmap v10403: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 218 KiB/s wr, 9 op/s 2023-03-09T20:07:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:07:17 smithi038 ceph-mon[92422]: pgmap v10403: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 218 KiB/s wr, 9 op/s 2023-03-09T20:07:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19010.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:07:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:07:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18548.546773 secs 2023-03-09T20:07:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:07:18 smithi122 ceph-mon[93869]: osdmap e864: 12 total, 12 up, 12 in 2023-03-09T20:07:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:07:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18548.546773 secs 2023-03-09T20:07:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:07:18 smithi181 ceph-mon[94189]: osdmap e864: 12 total, 12 up, 12 in 2023-03-09T20:07:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:07:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18548.546773 secs 2023-03-09T20:07:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:07:18 smithi038 ceph-mon[92422]: osdmap e864: 12 total, 12 up, 12 in 2023-03-09T20:07:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:07:19 smithi122 ceph-mon[93869]: pgmap v10404: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 218 KiB/s wr, 8 op/s 2023-03-09T20:07:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:07:19 smithi181 ceph-mon[94189]: pgmap v10404: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 218 KiB/s wr, 8 op/s 2023-03-09T20:07:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:07:19 smithi038 ceph-mon[92422]: pgmap v10404: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 218 KiB/s wr, 8 op/s 2023-03-09T20:07:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:07:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18551.230981 secs 2023-03-09T20:07:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:07:21 smithi122 ceph-mon[93869]: pgmap v10406: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 136 KiB/s wr, 9 op/s 2023-03-09T20:07:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:07:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18551.230981 secs 2023-03-09T20:07:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:07:21 smithi181 ceph-mon[94189]: pgmap v10406: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 136 KiB/s wr, 9 op/s 2023-03-09T20:07:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:07:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18551.230981 secs 2023-03-09T20:07:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:07:21 smithi038 ceph-mon[92422]: pgmap v10406: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 136 KiB/s wr, 9 op/s 2023-03-09T20:07:23.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19015.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:07:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:07:23 smithi122 ceph-mon[93869]: pgmap v10407: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 118 KiB/s wr, 8 op/s 2023-03-09T20:07:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:07:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18553.546889 secs 2023-03-09T20:07:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:07:23 smithi181 ceph-mon[94189]: pgmap v10407: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 118 KiB/s wr, 8 op/s 2023-03-09T20:07:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:07:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18553.546889 secs 2023-03-09T20:07:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:07:23 smithi038 ceph-mon[92422]: pgmap v10407: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 118 KiB/s wr, 8 op/s 2023-03-09T20:07:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:07:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18553.546889 secs 2023-03-09T20:07:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:07:25 smithi122 ceph-mon[93869]: pgmap v10408: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 167 KiB/s wr, 8 op/s 2023-03-09T20:07:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:07:25 smithi181 ceph-mon[94189]: pgmap v10408: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 167 KiB/s wr, 8 op/s 2023-03-09T20:07:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:07:25 smithi038 ceph-mon[92422]: pgmap v10408: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 167 KiB/s wr, 8 op/s 2023-03-09T20:07:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:07:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18556.231006 secs 2023-03-09T20:07:26.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:07:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18556.231006 secs 2023-03-09T20:07:26.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:07:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18556.231006 secs 2023-03-09T20:07:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:07:27 smithi122 ceph-mon[93869]: pgmap v10409: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 166 KiB/s wr, 8 op/s 2023-03-09T20:07:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:07:27 smithi181 ceph-mon[94189]: pgmap v10409: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 166 KiB/s wr, 8 op/s 2023-03-09T20:07:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:07:27 smithi038 ceph-mon[92422]: pgmap v10409: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 166 KiB/s wr, 8 op/s 2023-03-09T20:07:28.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19020.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:07:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:07:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18558.547025 secs 2023-03-09T20:07:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:07:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18558.547025 secs 2023-03-09T20:07:29.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:07:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18558.547025 secs 2023-03-09T20:07:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:07:29 smithi038 ceph-mon[92422]: pgmap v10410: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 166 KiB/s wr, 8 op/s 2023-03-09T20:07:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:07:29 smithi122 ceph-mon[93869]: pgmap v10410: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 166 KiB/s wr, 8 op/s 2023-03-09T20:07:30.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:07:29 smithi181 ceph-mon[94189]: pgmap v10410: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 166 KiB/s wr, 8 op/s 2023-03-09T20:07:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:07:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18561.231066 secs 2023-03-09T20:07:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:07:31 smithi038 ceph-mon[92422]: pgmap v10411: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 148 KiB/s wr, 8 op/s 2023-03-09T20:07:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:07:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18561.231066 secs 2023-03-09T20:07:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:07:31 smithi122 ceph-mon[93869]: pgmap v10411: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 148 KiB/s wr, 8 op/s 2023-03-09T20:07:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:07:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18561.231066 secs 2023-03-09T20:07:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:07:31 smithi181 ceph-mon[94189]: pgmap v10411: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 148 KiB/s wr, 8 op/s 2023-03-09T20:07:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19025.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:07:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:07:33 smithi038 ceph-mon[92422]: pgmap v10412: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 121 KiB/s wr, 9 op/s 2023-03-09T20:07:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:07:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18563.547143 secs 2023-03-09T20:07:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:07:33 smithi122 ceph-mon[93869]: pgmap v10412: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 121 KiB/s wr, 9 op/s 2023-03-09T20:07:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:07:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18563.547143 secs 2023-03-09T20:07:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:07:33 smithi181 ceph-mon[94189]: pgmap v10412: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 121 KiB/s wr, 9 op/s 2023-03-09T20:07:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:07:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18563.547143 secs 2023-03-09T20:07:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:07:35 smithi038 ceph-mon[92422]: pgmap v10413: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 173 KiB/s wr, 7 op/s 2023-03-09T20:07:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:07:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18566.231126 secs 2023-03-09T20:07:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:07:35 smithi122 ceph-mon[93869]: pgmap v10413: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 173 KiB/s wr, 7 op/s 2023-03-09T20:07:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:07:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18566.231126 secs 2023-03-09T20:07:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:07:35 smithi181 ceph-mon[94189]: pgmap v10413: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 173 KiB/s wr, 7 op/s 2023-03-09T20:07:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:07:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18566.231126 secs 2023-03-09T20:07:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:07:37 smithi122 ceph-mon[93869]: pgmap v10414: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 133 KiB/s wr, 9 op/s 2023-03-09T20:07:38.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:07:37 smithi181 ceph-mon[94189]: pgmap v10414: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 133 KiB/s wr, 9 op/s 2023-03-09T20:07:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:07:37 smithi038 ceph-mon[92422]: pgmap v10414: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 133 KiB/s wr, 9 op/s 2023-03-09T20:07:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19030.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:07:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:07:39 smithi122 ceph-mon[93869]: pgmap v10415: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 133 KiB/s wr, 7 op/s 2023-03-09T20:07:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:07:39 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:07:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:07:39 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:07:40.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:07:39 smithi181 ceph-mon[94189]: pgmap v10415: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 133 KiB/s wr, 7 op/s 2023-03-09T20:07:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:07:39 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:07:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:07:39 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:07:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:07:39 smithi038 ceph-mon[92422]: pgmap v10415: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 133 KiB/s wr, 7 op/s 2023-03-09T20:07:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:07:39 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:07:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:07:39 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:07:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:07:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:07:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:07:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:07:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:07:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18571.231187 secs 2023-03-09T20:07:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:07:40 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:07:41.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:07:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:07:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:07:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:07:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:07:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18571.231187 secs 2023-03-09T20:07:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:07:40 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:07:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:07:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:07:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:07:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:07:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:07:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18571.231187 secs 2023-03-09T20:07:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:07:40 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:07:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:07:41 smithi122 ceph-mon[93869]: pgmap v10416: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 133 KiB/s wr, 7 op/s 2023-03-09T20:07:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:07:41 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:07:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:07:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:07:42.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:07:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:07:42.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:07:41 smithi181 ceph-mon[94189]: pgmap v10416: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 133 KiB/s wr, 7 op/s 2023-03-09T20:07:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:07:41 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:07:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:07:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:07:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:07:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:07:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:07:41 smithi038 ceph-mon[92422]: pgmap v10416: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 133 KiB/s wr, 7 op/s 2023-03-09T20:07:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:07:41 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:07:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:07:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:07:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:07:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:07:43.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19035.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:07:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:07:43 smithi122 ceph-mon[93869]: pgmap v10417: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 152 KiB/s wr, 10 op/s 2023-03-09T20:07:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:07:43 smithi181 ceph-mon[94189]: pgmap v10417: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 152 KiB/s wr, 10 op/s 2023-03-09T20:07:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:07:43 smithi038 ceph-mon[92422]: pgmap v10417: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 152 KiB/s wr, 10 op/s 2023-03-09T20:07:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:07:45 smithi122 ceph-mon[93869]: pgmap v10418: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 151 KiB/s wr, 8 op/s 2023-03-09T20:07:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:07:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18576.231253 secs 2023-03-09T20:07:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:07:45 smithi181 ceph-mon[94189]: pgmap v10418: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 151 KiB/s wr, 8 op/s 2023-03-09T20:07:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:07:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18576.231253 secs 2023-03-09T20:07:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:07:45 smithi038 ceph-mon[92422]: pgmap v10418: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 151 KiB/s wr, 8 op/s 2023-03-09T20:07:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:07:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18576.231253 secs 2023-03-09T20:07:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:07:47 smithi122 ceph-mon[93869]: pgmap v10419: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 227 KiB/s wr, 11 op/s 2023-03-09T20:07:48.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:07:47 smithi181 ceph-mon[94189]: pgmap v10419: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 227 KiB/s wr, 11 op/s 2023-03-09T20:07:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:07:47 smithi038 ceph-mon[92422]: pgmap v10419: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 227 KiB/s wr, 11 op/s 2023-03-09T20:07:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19040.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:07:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:07:49 smithi122 ceph-mon[93869]: pgmap v10420: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 162 KiB/s wr, 7 op/s 2023-03-09T20:07:50.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:07:49 smithi181 ceph-mon[94189]: pgmap v10420: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 162 KiB/s wr, 7 op/s 2023-03-09T20:07:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:07:49 smithi038 ceph-mon[92422]: pgmap v10420: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 162 KiB/s wr, 7 op/s 2023-03-09T20:07:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:07:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18581.231330 secs 2023-03-09T20:07:51.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:07:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18581.231330 secs 2023-03-09T20:07:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:07:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18581.231330 secs 2023-03-09T20:07:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:07:51 smithi122 ceph-mon[93869]: pgmap v10421: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 162 KiB/s wr, 8 op/s 2023-03-09T20:07:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:07:51 smithi181 ceph-mon[94189]: pgmap v10421: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 162 KiB/s wr, 8 op/s 2023-03-09T20:07:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:07:51 smithi038 ceph-mon[92422]: pgmap v10421: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 162 KiB/s wr, 8 op/s 2023-03-09T20:07:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19045.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:07:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:07:53 smithi122 ceph-mon[93869]: pgmap v10422: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 295 KiB/s wr, 11 op/s 2023-03-09T20:07:54.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:07:53 smithi181 ceph-mon[94189]: pgmap v10422: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 295 KiB/s wr, 11 op/s 2023-03-09T20:07:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:07:53 smithi038 ceph-mon[92422]: pgmap v10422: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 295 KiB/s wr, 11 op/s 2023-03-09T20:07:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:07:55 smithi122 ceph-mon[93869]: pgmap v10423: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 277 KiB/s wr, 9 op/s 2023-03-09T20:07:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:07:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18586.231392 secs 2023-03-09T20:07:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:07:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:07:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:07:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:07:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:07:55 smithi181 ceph-mon[94189]: pgmap v10423: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 277 KiB/s wr, 9 op/s 2023-03-09T20:07:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:07:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18586.231392 secs 2023-03-09T20:07:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:07:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:07:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:07:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:07:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:07:55 smithi038 ceph-mon[92422]: pgmap v10423: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 277 KiB/s wr, 9 op/s 2023-03-09T20:07:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:07:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18586.231392 secs 2023-03-09T20:07:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:07:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:07:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:07:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:07:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:07:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:07:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:07:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:07:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:07:56 smithi122 ceph-mon[93869]: pgmap v10424: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 304 KiB/s wr, 12 op/s 2023-03-09T20:07:57.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:07:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:07:57.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:07:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:07:57.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:07:56 smithi181 ceph-mon[94189]: pgmap v10424: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 304 KiB/s wr, 12 op/s 2023-03-09T20:07:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:07:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:07:57.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:07:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:07:57.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:07:56 smithi038 ceph-mon[92422]: pgmap v10424: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 304 KiB/s wr, 12 op/s 2023-03-09T20:07:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19050.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:07:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:07:59 smithi122 ceph-mon[93869]: pgmap v10425: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 330 KiB/s wr, 10 op/s 2023-03-09T20:07:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:07:59 smithi181 ceph-mon[94189]: pgmap v10425: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 330 KiB/s wr, 10 op/s 2023-03-09T20:07:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:07:59 smithi038 ceph-mon[92422]: pgmap v10425: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 330 KiB/s wr, 10 op/s 2023-03-09T20:08:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18591.231451 secs 2023-03-09T20:08:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:01 smithi122 ceph-mon[93869]: pgmap v10426: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 330 KiB/s wr, 10 op/s 2023-03-09T20:08:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:01 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:08:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:01 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:08:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:01 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:08:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:01 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:08:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:01 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:08:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18591.231451 secs 2023-03-09T20:08:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:01 smithi181 ceph-mon[94189]: pgmap v10426: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 330 KiB/s wr, 10 op/s 2023-03-09T20:08:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:01 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:08:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:01 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:08:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:01 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:08:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:01 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:08:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:01 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:08:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18591.231451 secs 2023-03-09T20:08:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:01 smithi038 ceph-mon[92422]: pgmap v10426: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 330 KiB/s wr, 10 op/s 2023-03-09T20:08:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:01 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:08:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:01 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:08:02.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:01 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:08:02.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:01 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:08:02.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:01 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:08:02.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:08:02.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:08:02.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:08:02.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:08:02.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:08:02.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:08:02.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:08:02.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:08:02.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:08:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19055.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:08:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:03 smithi122 ceph-mon[93869]: pgmap v10427: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 345 KiB/s wr, 13 op/s 2023-03-09T20:08:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:03 smithi181 ceph-mon[94189]: pgmap v10427: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 345 KiB/s wr, 13 op/s 2023-03-09T20:08:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:03 smithi038 ceph-mon[92422]: pgmap v10427: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 345 KiB/s wr, 13 op/s 2023-03-09T20:08:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:05 smithi122 ceph-mon[93869]: pgmap v10428: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 248 KiB/s wr, 11 op/s 2023-03-09T20:08:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:05 smithi181 ceph-mon[94189]: pgmap v10428: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 248 KiB/s wr, 11 op/s 2023-03-09T20:08:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:05 smithi038 ceph-mon[92422]: pgmap v10428: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 248 KiB/s wr, 11 op/s 2023-03-09T20:08:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18596.231511 secs 2023-03-09T20:08:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18596.231511 secs 2023-03-09T20:08:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18596.231511 secs 2023-03-09T20:08:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:07 smithi122 ceph-mon[93869]: pgmap v10429: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 285 KiB/s wr, 16 op/s 2023-03-09T20:08:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:07 smithi181 ceph-mon[94189]: pgmap v10429: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 285 KiB/s wr, 16 op/s 2023-03-09T20:08:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:07 smithi038 ceph-mon[92422]: pgmap v10429: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 285 KiB/s wr, 16 op/s 2023-03-09T20:08:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19060.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:08:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18598.547852 secs 2023-03-09T20:08:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18598.547852 secs 2023-03-09T20:08:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18598.547852 secs 2023-03-09T20:08:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:09 smithi122 ceph-mon[93869]: pgmap v10430: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 426 KiB/s wr, 25 op/s 2023-03-09T20:08:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:09 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:08:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:08:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:08:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:09 smithi181 ceph-mon[94189]: pgmap v10430: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 426 KiB/s wr, 25 op/s 2023-03-09T20:08:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:09 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:08:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:08:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:08:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:09 smithi038 ceph-mon[92422]: pgmap v10430: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 426 KiB/s wr, 25 op/s 2023-03-09T20:08:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:09 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:08:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:08:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:08:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:10 smithi122 ceph-mon[93869]: osdmap e865: 12 total, 12 up, 12 in 2023-03-09T20:08:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:10 smithi181 ceph-mon[94189]: osdmap e865: 12 total, 12 up, 12 in 2023-03-09T20:08:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:10 smithi038 ceph-mon[92422]: osdmap e865: 12 total, 12 up, 12 in 2023-03-09T20:08:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18601.231577 secs 2023-03-09T20:08:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:11 smithi122 ceph-mon[93869]: pgmap v10432: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 308 KiB/s wr, 28 op/s 2023-03-09T20:08:11.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:11 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:08:11.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:11 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:08:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18601.231577 secs 2023-03-09T20:08:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:11 smithi181 ceph-mon[94189]: pgmap v10432: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 308 KiB/s wr, 28 op/s 2023-03-09T20:08:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:11 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:08:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:11 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:08:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18601.231577 secs 2023-03-09T20:08:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:11 smithi038 ceph-mon[92422]: pgmap v10432: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 308 KiB/s wr, 28 op/s 2023-03-09T20:08:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:11 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:08:12.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:11 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:08:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:08:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:08:12.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:08:12.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:08:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:08:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:08:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19065.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:08:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:13 smithi122 ceph-mon[93869]: pgmap v10433: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 308 KiB/s wr, 28 op/s 2023-03-09T20:08:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18603.547967 secs 2023-03-09T20:08:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:13 smithi181 ceph-mon[94189]: pgmap v10433: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 308 KiB/s wr, 28 op/s 2023-03-09T20:08:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18603.547967 secs 2023-03-09T20:08:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:13 smithi038 ceph-mon[92422]: pgmap v10433: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 308 KiB/s wr, 28 op/s 2023-03-09T20:08:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18603.547967 secs 2023-03-09T20:08:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:15 smithi122 ceph-mon[93869]: pgmap v10434: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 454 KiB/s wr, 28 op/s 2023-03-09T20:08:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:15 smithi181 ceph-mon[94189]: pgmap v10434: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 454 KiB/s wr, 28 op/s 2023-03-09T20:08:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:15 smithi038 ceph-mon[92422]: pgmap v10434: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 454 KiB/s wr, 28 op/s 2023-03-09T20:08:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18606.231636 secs 2023-03-09T20:08:16.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18606.231636 secs 2023-03-09T20:08:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18606.231636 secs 2023-03-09T20:08:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:17 smithi122 ceph-mon[93869]: pgmap v10435: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 395 KiB/s rd, 412 KiB/s wr, 24 op/s 2023-03-09T20:08:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:17 smithi181 ceph-mon[94189]: pgmap v10435: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 395 KiB/s rd, 412 KiB/s wr, 24 op/s 2023-03-09T20:08:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:17 smithi038 ceph-mon[92422]: pgmap v10435: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 395 KiB/s rd, 412 KiB/s wr, 24 op/s 2023-03-09T20:08:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19070.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:08:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18608.548058 secs 2023-03-09T20:08:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:18 smithi122 ceph-mon[93869]: osdmap e866: 12 total, 12 up, 12 in 2023-03-09T20:08:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18608.548058 secs 2023-03-09T20:08:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:18 smithi181 ceph-mon[94189]: osdmap e866: 12 total, 12 up, 12 in 2023-03-09T20:08:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18608.548058 secs 2023-03-09T20:08:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:18 smithi038 ceph-mon[92422]: osdmap e866: 12 total, 12 up, 12 in 2023-03-09T20:08:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:19 smithi122 ceph-mon[93869]: pgmap v10436: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 335 KiB/s rd, 333 KiB/s wr, 10 op/s 2023-03-09T20:08:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:19 smithi181 ceph-mon[94189]: pgmap v10436: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 335 KiB/s rd, 333 KiB/s wr, 10 op/s 2023-03-09T20:08:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:19 smithi038 ceph-mon[92422]: pgmap v10436: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 335 KiB/s rd, 333 KiB/s wr, 10 op/s 2023-03-09T20:08:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18611.231694 secs 2023-03-09T20:08:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:21 smithi122 ceph-mon[93869]: pgmap v10438: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 335 KiB/s rd, 333 KiB/s wr, 10 op/s 2023-03-09T20:08:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18611.231694 secs 2023-03-09T20:08:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:21 smithi181 ceph-mon[94189]: pgmap v10438: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 335 KiB/s rd, 333 KiB/s wr, 10 op/s 2023-03-09T20:08:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18611.231694 secs 2023-03-09T20:08:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:21 smithi038 ceph-mon[92422]: pgmap v10438: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 335 KiB/s rd, 333 KiB/s wr, 10 op/s 2023-03-09T20:08:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19075.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:08:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:23 smithi122 ceph-mon[93869]: pgmap v10439: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 336 KiB/s rd, 333 KiB/s wr, 11 op/s 2023-03-09T20:08:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18613.548177 secs 2023-03-09T20:08:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:23 smithi181 ceph-mon[94189]: pgmap v10439: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 336 KiB/s rd, 333 KiB/s wr, 11 op/s 2023-03-09T20:08:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18613.548177 secs 2023-03-09T20:08:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:23 smithi038 ceph-mon[92422]: pgmap v10439: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 336 KiB/s rd, 333 KiB/s wr, 11 op/s 2023-03-09T20:08:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18613.548177 secs 2023-03-09T20:08:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:25 smithi122 ceph-mon[93869]: pgmap v10440: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 143 KiB/s wr, 8 op/s 2023-03-09T20:08:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:25 smithi181 ceph-mon[94189]: pgmap v10440: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 143 KiB/s wr, 8 op/s 2023-03-09T20:08:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:25 smithi038 ceph-mon[92422]: pgmap v10440: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 143 KiB/s wr, 8 op/s 2023-03-09T20:08:26.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18616.231756 secs 2023-03-09T20:08:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18616.231756 secs 2023-03-09T20:08:27.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18616.231756 secs 2023-03-09T20:08:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:27 smithi038 ceph-mon[92422]: pgmap v10441: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 143 KiB/s wr, 8 op/s 2023-03-09T20:08:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:27 smithi122 ceph-mon[93869]: pgmap v10441: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 143 KiB/s wr, 8 op/s 2023-03-09T20:08:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:27 smithi181 ceph-mon[94189]: pgmap v10441: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 143 KiB/s wr, 8 op/s 2023-03-09T20:08:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19080.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:08:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18618.548299 secs 2023-03-09T20:08:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18618.548299 secs 2023-03-09T20:08:29.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18618.548299 secs 2023-03-09T20:08:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:29 smithi038 ceph-mon[92422]: pgmap v10442: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 64 KiB/s wr, 8 op/s 2023-03-09T20:08:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:29 smithi122 ceph-mon[93869]: pgmap v10442: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 64 KiB/s wr, 8 op/s 2023-03-09T20:08:30.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:29 smithi181 ceph-mon[94189]: pgmap v10442: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 64 KiB/s wr, 8 op/s 2023-03-09T20:08:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18621.231815 secs 2023-03-09T20:08:31.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18621.231815 secs 2023-03-09T20:08:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18621.231815 secs 2023-03-09T20:08:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:31 smithi122 ceph-mon[93869]: pgmap v10443: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 57 KiB/s wr, 7 op/s 2023-03-09T20:08:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:31 smithi181 ceph-mon[94189]: pgmap v10443: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 57 KiB/s wr, 7 op/s 2023-03-09T20:08:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:31 smithi038 ceph-mon[92422]: pgmap v10443: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 57 KiB/s wr, 7 op/s 2023-03-09T20:08:33.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19085.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:08:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:33 smithi122 ceph-mon[93869]: pgmap v10444: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 122 KiB/s wr, 10 op/s 2023-03-09T20:08:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18623.548420 secs 2023-03-09T20:08:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:33 smithi181 ceph-mon[94189]: pgmap v10444: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 122 KiB/s wr, 10 op/s 2023-03-09T20:08:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18623.548420 secs 2023-03-09T20:08:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:33 smithi038 ceph-mon[92422]: pgmap v10444: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 122 KiB/s wr, 10 op/s 2023-03-09T20:08:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18623.548420 secs 2023-03-09T20:08:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:35 smithi122 ceph-mon[93869]: pgmap v10445: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 107 KiB/s wr, 7 op/s 2023-03-09T20:08:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18626.231900 secs 2023-03-09T20:08:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:35 smithi181 ceph-mon[94189]: pgmap v10445: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 107 KiB/s wr, 7 op/s 2023-03-09T20:08:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18626.231900 secs 2023-03-09T20:08:36.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:35 smithi038 ceph-mon[92422]: pgmap v10445: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 107 KiB/s wr, 7 op/s 2023-03-09T20:08:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18626.231900 secs 2023-03-09T20:08:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:37 smithi122 ceph-mon[93869]: pgmap v10446: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 107 KiB/s wr, 10 op/s 2023-03-09T20:08:38.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:37 smithi181 ceph-mon[94189]: pgmap v10446: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 107 KiB/s wr, 10 op/s 2023-03-09T20:08:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:37 smithi038 ceph-mon[92422]: pgmap v10446: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 107 KiB/s wr, 10 op/s 2023-03-09T20:08:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19090.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:08:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:38 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 18628.548537 secs 2023-03-09T20:08:39.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:38 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 18628.548537 secs 2023-03-09T20:08:39.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:38 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 18628.548537 secs 2023-03-09T20:08:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:39 smithi122 ceph-mon[93869]: pgmap v10447: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 106 KiB/s wr, 8 op/s 2023-03-09T20:08:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:39 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:08:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:39 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:08:40.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:39 smithi181 ceph-mon[94189]: pgmap v10447: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 106 KiB/s wr, 8 op/s 2023-03-09T20:08:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:39 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:08:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:39 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:08:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:39 smithi038 ceph-mon[92422]: pgmap v10447: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 106 KiB/s wr, 8 op/s 2023-03-09T20:08:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:39 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:08:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:39 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:08:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:08:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:08:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18631.231963 secs 2023-03-09T20:08:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:08:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:08:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18631.231963 secs 2023-03-09T20:08:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:08:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:08:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18631.231963 secs 2023-03-09T20:08:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:41 smithi122 ceph-mon[93869]: pgmap v10448: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 69 KiB/s wr, 7 op/s 2023-03-09T20:08:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:08:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:08:42.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:41 smithi181 ceph-mon[94189]: pgmap v10448: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 69 KiB/s wr, 7 op/s 2023-03-09T20:08:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:08:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:08:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:41 smithi038 ceph-mon[92422]: pgmap v10448: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 69 KiB/s wr, 7 op/s 2023-03-09T20:08:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:08:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:08:43.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19095.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:08:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:43 smithi122 ceph-mon[93869]: pgmap v10449: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 247 KiB/s rd, 244 KiB/s wr, 11 op/s 2023-03-09T20:08:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:43 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 18633.548645 secs 2023-03-09T20:08:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:43 smithi181 ceph-mon[94189]: pgmap v10449: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 247 KiB/s rd, 244 KiB/s wr, 11 op/s 2023-03-09T20:08:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:43 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 18633.548645 secs 2023-03-09T20:08:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:43 smithi038 ceph-mon[92422]: pgmap v10449: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 247 KiB/s rd, 244 KiB/s wr, 11 op/s 2023-03-09T20:08:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:43 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 18633.548645 secs 2023-03-09T20:08:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:45 smithi122 ceph-mon[93869]: pgmap v10450: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 175 KiB/s wr, 7 op/s 2023-03-09T20:08:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18636.231983 secs 2023-03-09T20:08:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:45 smithi181 ceph-mon[94189]: pgmap v10450: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 175 KiB/s wr, 7 op/s 2023-03-09T20:08:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18636.231983 secs 2023-03-09T20:08:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:45 smithi038 ceph-mon[92422]: pgmap v10450: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 175 KiB/s wr, 7 op/s 2023-03-09T20:08:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18636.231983 secs 2023-03-09T20:08:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:47 smithi122 ceph-mon[93869]: pgmap v10451: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 338 KiB/s wr, 11 op/s 2023-03-09T20:08:48.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:47 smithi181 ceph-mon[94189]: pgmap v10451: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 338 KiB/s wr, 11 op/s 2023-03-09T20:08:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:47 smithi038 ceph-mon[92422]: pgmap v10451: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 338 KiB/s wr, 11 op/s 2023-03-09T20:08:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19100.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:08:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18638.548763 secs 2023-03-09T20:08:49.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18638.548763 secs 2023-03-09T20:08:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18638.548763 secs 2023-03-09T20:08:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:49 smithi122 ceph-mon[93869]: pgmap v10452: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 338 KiB/s wr, 9 op/s 2023-03-09T20:08:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:08:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:08:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:49 smithi181 ceph-mon[94189]: pgmap v10452: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 338 KiB/s wr, 9 op/s 2023-03-09T20:08:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:08:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:08:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:49 smithi038 ceph-mon[92422]: pgmap v10452: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 338 KiB/s wr, 9 op/s 2023-03-09T20:08:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:08:50.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:08:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18641.232067 secs 2023-03-09T20:08:51.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18641.232067 secs 2023-03-09T20:08:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18641.232067 secs 2023-03-09T20:08:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:51 smithi122 ceph-mon[93869]: pgmap v10453: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 455 KiB/s wr, 9 op/s 2023-03-09T20:08:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:08:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:08:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:51 smithi181 ceph-mon[94189]: pgmap v10453: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 455 KiB/s wr, 9 op/s 2023-03-09T20:08:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:08:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:08:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:51 smithi038 ceph-mon[92422]: pgmap v10453: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 455 KiB/s wr, 9 op/s 2023-03-09T20:08:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:08:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:08:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:08:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:08:53.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:52 smithi122 ceph-mon[93869]: pgmap v10454: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 470 KiB/s rd, 470 KiB/s wr, 13 op/s 2023-03-09T20:08:53.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:08:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:08:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:52 smithi181 ceph-mon[94189]: pgmap v10454: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 470 KiB/s rd, 470 KiB/s wr, 13 op/s 2023-03-09T20:08:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:08:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:08:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:52 smithi038 ceph-mon[92422]: pgmap v10454: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 470 KiB/s rd, 470 KiB/s wr, 13 op/s 2023-03-09T20:08:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19105.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:08:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18643.548898 secs 2023-03-09T20:08:54.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18643.548898 secs 2023-03-09T20:08:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18643.548898 secs 2023-03-09T20:08:55.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:54 smithi122 ceph-mon[93869]: pgmap v10455: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 295 KiB/s wr, 10 op/s 2023-03-09T20:08:55.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:54 smithi181 ceph-mon[94189]: pgmap v10455: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 295 KiB/s wr, 10 op/s 2023-03-09T20:08:55.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:54 smithi038 ceph-mon[92422]: pgmap v10455: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 295 KiB/s wr, 10 op/s 2023-03-09T20:08:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18646.232149 secs 2023-03-09T20:08:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18646.232149 secs 2023-03-09T20:08:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18646.232149 secs 2023-03-09T20:08:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:56 smithi122 ceph-mon[93869]: pgmap v10456: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 296 KiB/s wr, 13 op/s 2023-03-09T20:08:57.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:56 smithi181 ceph-mon[94189]: pgmap v10456: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 296 KiB/s wr, 13 op/s 2023-03-09T20:08:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:56 smithi038 ceph-mon[92422]: pgmap v10456: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 296 KiB/s wr, 13 op/s 2023-03-09T20:08:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18648.548997 secs 2023-03-09T20:08:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18648.548997 secs 2023-03-09T20:08:58.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18648.548997 secs 2023-03-09T20:08:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19110.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:08:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:59 smithi122 ceph-mon[93869]: pgmap v10457: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 134 KiB/s wr, 10 op/s 2023-03-09T20:08:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:08:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:08:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:08:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:59 smithi181 ceph-mon[94189]: pgmap v10457: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 134 KiB/s wr, 10 op/s 2023-03-09T20:08:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:08:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:08:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:08:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:59 smithi038 ceph-mon[92422]: pgmap v10457: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 134 KiB/s wr, 10 op/s 2023-03-09T20:08:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:08:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:08:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:09:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18651.232204 secs 2023-03-09T20:09:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:00 smithi038 ceph-mon[92422]: pgmap v10458: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 358 KiB/s wr, 10 op/s 2023-03-09T20:09:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:09:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:09:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18651.232204 secs 2023-03-09T20:09:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:00 smithi122 ceph-mon[93869]: pgmap v10458: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 358 KiB/s wr, 10 op/s 2023-03-09T20:09:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:09:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:09:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18651.232204 secs 2023-03-09T20:09:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:00 smithi181 ceph-mon[94189]: pgmap v10458: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 358 KiB/s wr, 10 op/s 2023-03-09T20:09:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:09:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:09:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:03 smithi122 ceph-mon[93869]: pgmap v10459: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 434 KiB/s wr, 15 op/s 2023-03-09T20:09:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18653.549134 secs 2023-03-09T20:09:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:03 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:09:03.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:03 smithi181 ceph-mon[94189]: pgmap v10459: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 434 KiB/s wr, 15 op/s 2023-03-09T20:09:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18653.549134 secs 2023-03-09T20:09:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:03 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:09:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:03 smithi038 ceph-mon[92422]: pgmap v10459: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 434 KiB/s wr, 15 op/s 2023-03-09T20:09:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18653.549134 secs 2023-03-09T20:09:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:03 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:09:03.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19115.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:09:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:09:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:09:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:09:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:09:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:09:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:09:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:09:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:09:04.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:09:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:05 smithi122 ceph-mon[93869]: pgmap v10460: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 441 KiB/s wr, 24 op/s 2023-03-09T20:09:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:05 smithi181 ceph-mon[94189]: pgmap v10460: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 441 KiB/s wr, 24 op/s 2023-03-09T20:09:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:05 smithi038 ceph-mon[92422]: pgmap v10460: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 441 KiB/s wr, 24 op/s 2023-03-09T20:09:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18656.232264 secs 2023-03-09T20:09:06.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18656.232264 secs 2023-03-09T20:09:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18656.232264 secs 2023-03-09T20:09:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:07 smithi122 ceph-mon[93869]: pgmap v10461: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 496 KiB/s wr, 27 op/s 2023-03-09T20:09:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:07 smithi181 ceph-mon[94189]: pgmap v10461: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 496 KiB/s wr, 27 op/s 2023-03-09T20:09:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:07 smithi038 ceph-mon[92422]: pgmap v10461: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 496 KiB/s wr, 27 op/s 2023-03-09T20:09:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19120.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:09:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18658.549256 secs 2023-03-09T20:09:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18658.549256 secs 2023-03-09T20:09:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18658.549256 secs 2023-03-09T20:09:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:09 smithi122 ceph-mon[93869]: pgmap v10462: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 496 KiB/s wr, 24 op/s 2023-03-09T20:09:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:09 smithi181 ceph-mon[94189]: pgmap v10462: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 496 KiB/s wr, 24 op/s 2023-03-09T20:09:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:09 smithi038 ceph-mon[92422]: pgmap v10462: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 496 KiB/s wr, 24 op/s 2023-03-09T20:09:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:10 smithi122 ceph-mon[93869]: osdmap e867: 12 total, 12 up, 12 in 2023-03-09T20:09:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:10 smithi181 ceph-mon[94189]: osdmap e867: 12 total, 12 up, 12 in 2023-03-09T20:09:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:10 smithi038 ceph-mon[92422]: osdmap e867: 12 total, 12 up, 12 in 2023-03-09T20:09:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18661.232333 secs 2023-03-09T20:09:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:11 smithi122 ceph-mon[93869]: pgmap v10464: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 325 KiB/s wr, 28 op/s 2023-03-09T20:09:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18661.232333 secs 2023-03-09T20:09:11.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:11 smithi181 ceph-mon[94189]: pgmap v10464: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 325 KiB/s wr, 28 op/s 2023-03-09T20:09:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18661.232333 secs 2023-03-09T20:09:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:11 smithi038 ceph-mon[92422]: pgmap v10464: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 325 KiB/s wr, 28 op/s 2023-03-09T20:09:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19125.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:09:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:13 smithi122 ceph-mon[93869]: pgmap v10465: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 219 KiB/s wr, 25 op/s 2023-03-09T20:09:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18663.549371 secs 2023-03-09T20:09:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:13 smithi181 ceph-mon[94189]: pgmap v10465: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 219 KiB/s wr, 25 op/s 2023-03-09T20:09:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18663.549371 secs 2023-03-09T20:09:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:13 smithi038 ceph-mon[92422]: pgmap v10465: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 219 KiB/s wr, 25 op/s 2023-03-09T20:09:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18663.549371 secs 2023-03-09T20:09:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:15 smithi122 ceph-mon[93869]: pgmap v10466: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 194 KiB/s wr, 10 op/s 2023-03-09T20:09:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:15 smithi181 ceph-mon[94189]: pgmap v10466: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 194 KiB/s wr, 10 op/s 2023-03-09T20:09:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:15 smithi038 ceph-mon[92422]: pgmap v10466: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 194 KiB/s wr, 10 op/s 2023-03-09T20:09:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18666.232390 secs 2023-03-09T20:09:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18666.232390 secs 2023-03-09T20:09:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18666.232390 secs 2023-03-09T20:09:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:17 smithi122 ceph-mon[93869]: pgmap v10467: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 183 KiB/s wr, 10 op/s 2023-03-09T20:09:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:17 smithi181 ceph-mon[94189]: pgmap v10467: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 183 KiB/s wr, 10 op/s 2023-03-09T20:09:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:17 smithi038 ceph-mon[92422]: pgmap v10467: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 183 KiB/s wr, 10 op/s 2023-03-09T20:09:18.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19130.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:09:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18668.549491 secs 2023-03-09T20:09:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:18 smithi122 ceph-mon[93869]: osdmap e868: 12 total, 12 up, 12 in 2023-03-09T20:09:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18668.549491 secs 2023-03-09T20:09:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:18 smithi181 ceph-mon[94189]: osdmap e868: 12 total, 12 up, 12 in 2023-03-09T20:09:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18668.549491 secs 2023-03-09T20:09:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:18 smithi038 ceph-mon[92422]: osdmap e868: 12 total, 12 up, 12 in 2023-03-09T20:09:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:19 smithi122 ceph-mon[93869]: pgmap v10468: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 183 KiB/s wr, 10 op/s 2023-03-09T20:09:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:19 smithi181 ceph-mon[94189]: pgmap v10468: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 183 KiB/s wr, 10 op/s 2023-03-09T20:09:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:19 smithi038 ceph-mon[92422]: pgmap v10468: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 183 KiB/s wr, 10 op/s 2023-03-09T20:09:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18671.232451 secs 2023-03-09T20:09:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:21 smithi122 ceph-mon[93869]: pgmap v10470: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 183 KiB/s wr, 9 op/s 2023-03-09T20:09:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18671.232451 secs 2023-03-09T20:09:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:21 smithi181 ceph-mon[94189]: pgmap v10470: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 183 KiB/s wr, 9 op/s 2023-03-09T20:09:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18671.232451 secs 2023-03-09T20:09:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:21 smithi038 ceph-mon[92422]: pgmap v10470: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 183 KiB/s wr, 9 op/s 2023-03-09T20:09:23.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19135.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:09:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:23 smithi122 ceph-mon[93869]: pgmap v10471: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 58 KiB/s wr, 8 op/s 2023-03-09T20:09:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18673.549604 secs 2023-03-09T20:09:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:23 smithi181 ceph-mon[94189]: pgmap v10471: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 58 KiB/s wr, 8 op/s 2023-03-09T20:09:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18673.549604 secs 2023-03-09T20:09:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:23 smithi038 ceph-mon[92422]: pgmap v10471: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 58 KiB/s wr, 8 op/s 2023-03-09T20:09:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18673.549604 secs 2023-03-09T20:09:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:25 smithi181 ceph-mon[94189]: pgmap v10472: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 58 KiB/s wr, 7 op/s 2023-03-09T20:09:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:25 smithi038 ceph-mon[92422]: pgmap v10472: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 58 KiB/s wr, 7 op/s 2023-03-09T20:09:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:25 smithi122 ceph-mon[93869]: pgmap v10472: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 58 KiB/s wr, 7 op/s 2023-03-09T20:09:26.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18676.232512 secs 2023-03-09T20:09:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18676.232512 secs 2023-03-09T20:09:27.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18676.232512 secs 2023-03-09T20:09:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:27 smithi038 ceph-mon[92422]: pgmap v10473: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 129 KiB/s wr, 7 op/s 2023-03-09T20:09:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:27 smithi122 ceph-mon[93869]: pgmap v10473: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 129 KiB/s wr, 7 op/s 2023-03-09T20:09:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:27 smithi181 ceph-mon[94189]: pgmap v10473: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 129 KiB/s wr, 7 op/s 2023-03-09T20:09:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19140.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:09:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18678.549714 secs 2023-03-09T20:09:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18678.549714 secs 2023-03-09T20:09:29.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18678.549714 secs 2023-03-09T20:09:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:29 smithi038 ceph-mon[92422]: pgmap v10474: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 129 KiB/s wr, 8 op/s 2023-03-09T20:09:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:29 smithi122 ceph-mon[93869]: pgmap v10474: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 129 KiB/s wr, 8 op/s 2023-03-09T20:09:30.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:29 smithi181 ceph-mon[94189]: pgmap v10474: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 129 KiB/s wr, 8 op/s 2023-03-09T20:09:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18681.232572 secs 2023-03-09T20:09:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:31 smithi038 ceph-mon[92422]: pgmap v10475: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 114 KiB/s wr, 7 op/s 2023-03-09T20:09:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:31 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:09:32.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:31 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:09:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18681.232572 secs 2023-03-09T20:09:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:31 smithi122 ceph-mon[93869]: pgmap v10475: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 114 KiB/s wr, 7 op/s 2023-03-09T20:09:32.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:31 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:09:32.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:31 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:09:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18681.232572 secs 2023-03-09T20:09:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:31 smithi181 ceph-mon[94189]: pgmap v10475: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 114 KiB/s wr, 7 op/s 2023-03-09T20:09:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:31 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:09:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:31 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:09:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:32 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:09:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:32 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:09:33.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:32 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:09:33.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:32 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:09:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:32 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:09:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:32 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:09:33.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19145.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:09:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:33 smithi122 ceph-mon[93869]: pgmap v10476: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 108 KiB/s wr, 10 op/s 2023-03-09T20:09:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18683.549834 secs 2023-03-09T20:09:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:33 smithi181 ceph-mon[94189]: pgmap v10476: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 108 KiB/s wr, 10 op/s 2023-03-09T20:09:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18683.549834 secs 2023-03-09T20:09:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:33 smithi038 ceph-mon[92422]: pgmap v10476: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 108 KiB/s wr, 10 op/s 2023-03-09T20:09:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18683.549834 secs 2023-03-09T20:09:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:35 smithi122 ceph-mon[93869]: pgmap v10477: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 107 KiB/s wr, 7 op/s 2023-03-09T20:09:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18686.232632 secs 2023-03-09T20:09:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:35 smithi181 ceph-mon[94189]: pgmap v10477: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 107 KiB/s wr, 7 op/s 2023-03-09T20:09:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18686.232632 secs 2023-03-09T20:09:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:35 smithi038 ceph-mon[92422]: pgmap v10477: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 107 KiB/s wr, 7 op/s 2023-03-09T20:09:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18686.232632 secs 2023-03-09T20:09:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:37 smithi122 ceph-mon[93869]: pgmap v10478: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 172 KiB/s wr, 11 op/s 2023-03-09T20:09:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:37 smithi181 ceph-mon[94189]: pgmap v10478: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 172 KiB/s wr, 11 op/s 2023-03-09T20:09:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:37 smithi038 ceph-mon[92422]: pgmap v10478: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 172 KiB/s wr, 11 op/s 2023-03-09T20:09:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19150.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:09:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18688.549952 secs 2023-03-09T20:09:39.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18688.549952 secs 2023-03-09T20:09:39.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18688.549952 secs 2023-03-09T20:09:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:39 smithi122 ceph-mon[93869]: pgmap v10479: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 66 KiB/s wr, 7 op/s 2023-03-09T20:09:40.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:39 smithi181 ceph-mon[94189]: pgmap v10479: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 66 KiB/s wr, 7 op/s 2023-03-09T20:09:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:39 smithi038 ceph-mon[92422]: pgmap v10479: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 66 KiB/s wr, 7 op/s 2023-03-09T20:09:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18691.232705 secs 2023-03-09T20:09:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18691.232705 secs 2023-03-09T20:09:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18691.232705 secs 2023-03-09T20:09:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:41 smithi122 ceph-mon[93869]: pgmap v10480: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 66 KiB/s wr, 7 op/s 2023-03-09T20:09:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:41 smithi181 ceph-mon[94189]: pgmap v10480: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 66 KiB/s wr, 7 op/s 2023-03-09T20:09:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:41 smithi038 ceph-mon[92422]: pgmap v10480: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 66 KiB/s wr, 7 op/s 2023-03-09T20:09:43.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19155.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:09:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:43 smithi122 ceph-mon[93869]: pgmap v10481: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 186 KiB/s wr, 10 op/s 2023-03-09T20:09:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18693.550068 secs 2023-03-09T20:09:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:43 smithi181 ceph-mon[94189]: pgmap v10481: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 186 KiB/s wr, 10 op/s 2023-03-09T20:09:44.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18693.550068 secs 2023-03-09T20:09:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:43 smithi038 ceph-mon[92422]: pgmap v10481: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 186 KiB/s wr, 10 op/s 2023-03-09T20:09:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18693.550068 secs 2023-03-09T20:09:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:45 smithi122 ceph-mon[93869]: pgmap v10482: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 185 KiB/s wr, 7 op/s 2023-03-09T20:09:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18696.232764 secs 2023-03-09T20:09:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:45 smithi181 ceph-mon[94189]: pgmap v10482: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 185 KiB/s wr, 7 op/s 2023-03-09T20:09:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18696.232764 secs 2023-03-09T20:09:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:45 smithi038 ceph-mon[92422]: pgmap v10482: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 185 KiB/s wr, 7 op/s 2023-03-09T20:09:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18696.232764 secs 2023-03-09T20:09:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:47 smithi122 ceph-mon[93869]: pgmap v10483: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 274 KiB/s wr, 11 op/s 2023-03-09T20:09:48.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:47 smithi181 ceph-mon[94189]: pgmap v10483: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 274 KiB/s wr, 11 op/s 2023-03-09T20:09:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:47 smithi038 ceph-mon[92422]: pgmap v10483: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 274 KiB/s wr, 11 op/s 2023-03-09T20:09:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19160.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:09:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:48 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:09:49.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:48 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:09:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:48 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:09:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:49 smithi122 ceph-mon[93869]: pgmap v10484: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 383 KiB/s wr, 9 op/s 2023-03-09T20:09:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:49 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:09:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:09:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:09:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:49 smithi181 ceph-mon[94189]: pgmap v10484: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 383 KiB/s wr, 9 op/s 2023-03-09T20:09:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:49 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:09:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:09:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:09:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:49 smithi038 ceph-mon[92422]: pgmap v10484: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 383 KiB/s wr, 9 op/s 2023-03-09T20:09:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:49 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:09:50.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:09:50.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:09:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18701.232824 secs 2023-03-09T20:09:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18701.232824 secs 2023-03-09T20:09:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18701.232824 secs 2023-03-09T20:09:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:51 smithi122 ceph-mon[93869]: pgmap v10485: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 383 KiB/s wr, 9 op/s 2023-03-09T20:09:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:09:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:09:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:51 smithi181 ceph-mon[94189]: pgmap v10485: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 383 KiB/s wr, 9 op/s 2023-03-09T20:09:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:09:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:09:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:51 smithi038 ceph-mon[92422]: pgmap v10485: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 383 KiB/s wr, 9 op/s 2023-03-09T20:09:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:09:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:09:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:09:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:09:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:09:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:09:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:09:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:09:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19165.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:09:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:53 smithi122 ceph-mon[93869]: pgmap v10486: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 398 KiB/s wr, 12 op/s 2023-03-09T20:09:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:53 smithi181 ceph-mon[94189]: pgmap v10486: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 398 KiB/s wr, 12 op/s 2023-03-09T20:09:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:53 smithi038 ceph-mon[92422]: pgmap v10486: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 398 KiB/s wr, 12 op/s 2023-03-09T20:09:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:55 smithi122 ceph-mon[93869]: pgmap v10487: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 379 KiB/s wr, 10 op/s 2023-03-09T20:09:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18706.232884 secs 2023-03-09T20:09:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:55 smithi181 ceph-mon[94189]: pgmap v10487: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 379 KiB/s wr, 10 op/s 2023-03-09T20:09:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18706.232884 secs 2023-03-09T20:09:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:55 smithi038 ceph-mon[92422]: pgmap v10487: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 379 KiB/s wr, 10 op/s 2023-03-09T20:09:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18706.232884 secs 2023-03-09T20:09:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:57 smithi122 ceph-mon[93869]: pgmap v10488: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 410 KiB/s wr, 14 op/s 2023-03-09T20:09:58.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:57 smithi181 ceph-mon[94189]: pgmap v10488: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 410 KiB/s wr, 14 op/s 2023-03-09T20:09:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:57 smithi038 ceph-mon[92422]: pgmap v10488: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 410 KiB/s wr, 14 op/s 2023-03-09T20:09:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19170.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:10:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:59 smithi122 ceph-mon[93869]: pgmap v10489: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 388 KiB/s wr, 11 op/s 2023-03-09T20:10:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:10:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:09:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:10:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:59 smithi181 ceph-mon[94189]: pgmap v10489: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 388 KiB/s wr, 11 op/s 2023-03-09T20:10:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:10:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:09:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:10:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:59 smithi038 ceph-mon[92422]: pgmap v10489: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 388 KiB/s wr, 11 op/s 2023-03-09T20:10:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:10:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:09:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:10:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18711.232943 secs 2023-03-09T20:10:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:10:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:10:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:00 smithi122 ceph-mon[93869]: pgmap v10490: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 215 KiB/s wr, 9 op/s 2023-03-09T20:10:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:00 smithi122 ceph-mon[93869]: overall HEALTH_WARN 1 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-09T20:10:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18711.232943 secs 2023-03-09T20:10:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:10:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:10:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:00 smithi181 ceph-mon[94189]: pgmap v10490: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 215 KiB/s wr, 9 op/s 2023-03-09T20:10:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:00 smithi181 ceph-mon[94189]: overall HEALTH_WARN 1 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-09T20:10:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18711.232943 secs 2023-03-09T20:10:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:10:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:10:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:00 smithi038 ceph-mon[92422]: pgmap v10490: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 215 KiB/s wr, 9 op/s 2023-03-09T20:10:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:00 smithi038 ceph-mon[92422]: overall HEALTH_WARN 1 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-09T20:10:03.467 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:03 smithi122 ceph-mon[93869]: pgmap v10491: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 478 KiB/s rd, 380 KiB/s wr, 15 op/s 2023-03-09T20:10:03.468 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18713.550518 secs 2023-03-09T20:10:03.469 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:03 smithi181 ceph-mon[94189]: pgmap v10491: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 478 KiB/s rd, 380 KiB/s wr, 15 op/s 2023-03-09T20:10:03.469 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18713.550518 secs 2023-03-09T20:10:03.484 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:03 smithi038 ceph-mon[92422]: pgmap v10491: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 478 KiB/s rd, 380 KiB/s wr, 15 op/s 2023-03-09T20:10:03.484 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18713.550518 secs 2023-03-09T20:10:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19175.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:10:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:10:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:10:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:10:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:10:04.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:10:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:10:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:10:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:10:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:10:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:10:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:10:04.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:10:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:05 smithi122 ceph-mon[93869]: pgmap v10492: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 469 KiB/s rd, 386 KiB/s wr, 25 op/s 2023-03-09T20:10:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:05 smithi181 ceph-mon[94189]: pgmap v10492: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 469 KiB/s rd, 386 KiB/s wr, 25 op/s 2023-03-09T20:10:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:05 smithi038 ceph-mon[92422]: pgmap v10492: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 469 KiB/s rd, 386 KiB/s wr, 25 op/s 2023-03-09T20:10:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18716.232985 secs 2023-03-09T20:10:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:06 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:10:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:06 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:10:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:10:06.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18716.232985 secs 2023-03-09T20:10:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:06 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:10:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:06 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:10:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:10:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18716.232985 secs 2023-03-09T20:10:06.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:06 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:10:06.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:06 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:10:06.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:10:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:07 smithi122 ceph-mon[93869]: pgmap v10493: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 386 KiB/s wr, 27 op/s 2023-03-09T20:10:07.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:07 smithi181 ceph-mon[94189]: pgmap v10493: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 386 KiB/s wr, 27 op/s 2023-03-09T20:10:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:07 smithi038 ceph-mon[92422]: pgmap v10493: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 386 KiB/s wr, 27 op/s 2023-03-09T20:10:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18718.550652 secs 2023-03-09T20:10:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18718.550652 secs 2023-03-09T20:10:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18718.550652 secs 2023-03-09T20:10:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19180.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:10:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:09 smithi122 ceph-mon[93869]: pgmap v10494: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 321 KiB/s rd, 356 KiB/s wr, 24 op/s 2023-03-09T20:10:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:09 smithi181 ceph-mon[94189]: pgmap v10494: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 321 KiB/s rd, 356 KiB/s wr, 24 op/s 2023-03-09T20:10:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:09 smithi038 ceph-mon[92422]: pgmap v10494: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 321 KiB/s rd, 356 KiB/s wr, 24 op/s 2023-03-09T20:10:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:10 smithi122 ceph-mon[93869]: osdmap e869: 12 total, 12 up, 12 in 2023-03-09T20:10:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:10 smithi181 ceph-mon[94189]: osdmap e869: 12 total, 12 up, 12 in 2023-03-09T20:10:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:10 smithi038 ceph-mon[92422]: osdmap e869: 12 total, 12 up, 12 in 2023-03-09T20:10:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18721.233050 secs 2023-03-09T20:10:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:11 smithi122 ceph-mon[93869]: pgmap v10496: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 345 KiB/s wr, 27 op/s 2023-03-09T20:10:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18721.233050 secs 2023-03-09T20:10:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:11 smithi181 ceph-mon[94189]: pgmap v10496: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 345 KiB/s wr, 27 op/s 2023-03-09T20:10:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18721.233050 secs 2023-03-09T20:10:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:11 smithi038 ceph-mon[92422]: pgmap v10496: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 345 KiB/s wr, 27 op/s 2023-03-09T20:10:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19185.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:10:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:13 smithi122 ceph-mon[93869]: pgmap v10497: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 165 KiB/s wr, 24 op/s 2023-03-09T20:10:13.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18723.550760 secs 2023-03-09T20:10:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:13 smithi181 ceph-mon[94189]: pgmap v10497: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 165 KiB/s wr, 24 op/s 2023-03-09T20:10:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18723.550760 secs 2023-03-09T20:10:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:13 smithi038 ceph-mon[92422]: pgmap v10497: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 165 KiB/s wr, 24 op/s 2023-03-09T20:10:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18723.550760 secs 2023-03-09T20:10:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:15 smithi122 ceph-mon[93869]: pgmap v10498: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 139 KiB/s wr, 9 op/s 2023-03-09T20:10:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:15 smithi181 ceph-mon[94189]: pgmap v10498: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 139 KiB/s wr, 9 op/s 2023-03-09T20:10:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:15 smithi038 ceph-mon[92422]: pgmap v10498: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 139 KiB/s wr, 9 op/s 2023-03-09T20:10:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18726.233108 secs 2023-03-09T20:10:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18726.233108 secs 2023-03-09T20:10:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18726.233108 secs 2023-03-09T20:10:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:17 smithi122 ceph-mon[93869]: pgmap v10499: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 90 KiB/s wr, 9 op/s 2023-03-09T20:10:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:17 smithi181 ceph-mon[94189]: pgmap v10499: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 90 KiB/s wr, 9 op/s 2023-03-09T20:10:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:17 smithi038 ceph-mon[92422]: pgmap v10499: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 90 KiB/s wr, 9 op/s 2023-03-09T20:10:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19190.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:10:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18728.550879 secs 2023-03-09T20:10:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:18 smithi122 ceph-mon[93869]: osdmap e870: 12 total, 12 up, 12 in 2023-03-09T20:10:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18728.550879 secs 2023-03-09T20:10:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:18 smithi181 ceph-mon[94189]: osdmap e870: 12 total, 12 up, 12 in 2023-03-09T20:10:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18728.550879 secs 2023-03-09T20:10:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:18 smithi038 ceph-mon[92422]: osdmap e870: 12 total, 12 up, 12 in 2023-03-09T20:10:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:19 smithi122 ceph-mon[93869]: pgmap v10500: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 90 KiB/s wr, 9 op/s 2023-03-09T20:10:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:19 smithi181 ceph-mon[94189]: pgmap v10500: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 90 KiB/s wr, 9 op/s 2023-03-09T20:10:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:19 smithi038 ceph-mon[92422]: pgmap v10500: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 90 KiB/s wr, 9 op/s 2023-03-09T20:10:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18731.233169 secs 2023-03-09T20:10:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:21 smithi122 ceph-mon[93869]: pgmap v10502: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 90 KiB/s wr, 9 op/s 2023-03-09T20:10:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18731.233169 secs 2023-03-09T20:10:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:21 smithi181 ceph-mon[94189]: pgmap v10502: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 90 KiB/s wr, 9 op/s 2023-03-09T20:10:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18731.233169 secs 2023-03-09T20:10:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:21 smithi038 ceph-mon[92422]: pgmap v10502: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 90 KiB/s wr, 9 op/s 2023-03-09T20:10:23.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19195.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:10:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:23 smithi122 ceph-mon[93869]: pgmap v10503: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 74 KiB/s wr, 8 op/s 2023-03-09T20:10:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18733.551000 secs 2023-03-09T20:10:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:23 smithi181 ceph-mon[94189]: pgmap v10503: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 74 KiB/s wr, 8 op/s 2023-03-09T20:10:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18733.551000 secs 2023-03-09T20:10:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:23 smithi038 ceph-mon[92422]: pgmap v10503: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 74 KiB/s wr, 8 op/s 2023-03-09T20:10:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18733.551000 secs 2023-03-09T20:10:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:25 smithi122 ceph-mon[93869]: pgmap v10504: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 94 KiB/s wr, 8 op/s 2023-03-09T20:10:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:25 smithi181 ceph-mon[94189]: pgmap v10504: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 94 KiB/s wr, 8 op/s 2023-03-09T20:10:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:25 smithi038 ceph-mon[92422]: pgmap v10504: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 94 KiB/s wr, 8 op/s 2023-03-09T20:10:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18736.233231 secs 2023-03-09T20:10:26.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18736.233231 secs 2023-03-09T20:10:26.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18736.233231 secs 2023-03-09T20:10:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:27 smithi122 ceph-mon[93869]: pgmap v10505: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 84 KiB/s wr, 7 op/s 2023-03-09T20:10:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:27 smithi181 ceph-mon[94189]: pgmap v10505: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 84 KiB/s wr, 7 op/s 2023-03-09T20:10:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:27 smithi038 ceph-mon[92422]: pgmap v10505: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 84 KiB/s wr, 7 op/s 2023-03-09T20:10:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19200.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:10:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18738.551092 secs 2023-03-09T20:10:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18738.551092 secs 2023-03-09T20:10:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18738.551092 secs 2023-03-09T20:10:29.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:29 smithi181 ceph-mon[94189]: pgmap v10506: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 84 KiB/s wr, 7 op/s 2023-03-09T20:10:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:29 smithi038 ceph-mon[92422]: pgmap v10506: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 84 KiB/s wr, 7 op/s 2023-03-09T20:10:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:29 smithi122 ceph-mon[93869]: pgmap v10506: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 84 KiB/s wr, 7 op/s 2023-03-09T20:10:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18741.233290 secs 2023-03-09T20:10:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:31 smithi038 ceph-mon[92422]: pgmap v10507: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 74 KiB/s wr, 7 op/s 2023-03-09T20:10:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18741.233290 secs 2023-03-09T20:10:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:31 smithi122 ceph-mon[93869]: pgmap v10507: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 74 KiB/s wr, 7 op/s 2023-03-09T20:10:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18741.233290 secs 2023-03-09T20:10:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:31 smithi181 ceph-mon[94189]: pgmap v10507: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 74 KiB/s wr, 7 op/s 2023-03-09T20:10:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19205.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:10:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:33 smithi038 ceph-mon[92422]: pgmap v10508: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 70 KiB/s wr, 10 op/s 2023-03-09T20:10:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18743.551203 secs 2023-03-09T20:10:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:33 smithi122 ceph-mon[93869]: pgmap v10508: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 70 KiB/s wr, 10 op/s 2023-03-09T20:10:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18743.551203 secs 2023-03-09T20:10:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:33 smithi181 ceph-mon[94189]: pgmap v10508: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 70 KiB/s wr, 10 op/s 2023-03-09T20:10:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18743.551203 secs 2023-03-09T20:10:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:35 smithi122 ceph-mon[93869]: pgmap v10509: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 138 KiB/s wr, 7 op/s 2023-03-09T20:10:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:35 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:10:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:35 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:10:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:35 smithi181 ceph-mon[94189]: pgmap v10509: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 138 KiB/s wr, 7 op/s 2023-03-09T20:10:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:35 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:10:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:35 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:10:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:35 smithi038 ceph-mon[92422]: pgmap v10509: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 138 KiB/s wr, 7 op/s 2023-03-09T20:10:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:35 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:10:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:35 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:10:37.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18746.233375 secs 2023-03-09T20:10:37.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:10:37.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:10:37.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18746.233375 secs 2023-03-09T20:10:37.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:10:37.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:10:37.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18746.233375 secs 2023-03-09T20:10:37.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:10:37.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:10:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:37 smithi122 ceph-mon[93869]: pgmap v10510: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 122 KiB/s wr, 9 op/s 2023-03-09T20:10:38.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:37 smithi181 ceph-mon[94189]: pgmap v10510: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 122 KiB/s wr, 9 op/s 2023-03-09T20:10:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:37 smithi038 ceph-mon[92422]: pgmap v10510: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 122 KiB/s wr, 9 op/s 2023-03-09T20:10:38.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19210.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:10:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:39 smithi122 ceph-mon[93869]: pgmap v10511: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 69 KiB/s wr, 7 op/s 2023-03-09T20:10:40.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:39 smithi181 ceph-mon[94189]: pgmap v10511: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 69 KiB/s wr, 7 op/s 2023-03-09T20:10:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:39 smithi038 ceph-mon[92422]: pgmap v10511: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 69 KiB/s wr, 7 op/s 2023-03-09T20:10:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:40 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:10:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:40 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:10:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:40 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:10:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18751.233436 secs 2023-03-09T20:10:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:41 smithi122 ceph-mon[93869]: pgmap v10512: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 69 KiB/s wr, 7 op/s 2023-03-09T20:10:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:41 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:10:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:10:42.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:10:42.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18751.233436 secs 2023-03-09T20:10:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:41 smithi181 ceph-mon[94189]: pgmap v10512: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 69 KiB/s wr, 7 op/s 2023-03-09T20:10:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:41 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:10:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:10:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:10:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18751.233436 secs 2023-03-09T20:10:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:41 smithi038 ceph-mon[92422]: pgmap v10512: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 69 KiB/s wr, 7 op/s 2023-03-09T20:10:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:41 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:10:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:10:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:10:43.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19215.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:10:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:43 smithi122 ceph-mon[93869]: pgmap v10513: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 70 KiB/s wr, 9 op/s 2023-03-09T20:10:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:43 smithi181 ceph-mon[94189]: pgmap v10513: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 70 KiB/s wr, 9 op/s 2023-03-09T20:10:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:43 smithi038 ceph-mon[92422]: pgmap v10513: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 70 KiB/s wr, 9 op/s 2023-03-09T20:10:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:45 smithi122 ceph-mon[93869]: pgmap v10514: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 152 KiB/s wr, 6 op/s 2023-03-09T20:10:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:45 smithi181 ceph-mon[94189]: pgmap v10514: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 152 KiB/s wr, 6 op/s 2023-03-09T20:10:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:45 smithi038 ceph-mon[92422]: pgmap v10514: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 152 KiB/s wr, 6 op/s 2023-03-09T20:10:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18756.233494 secs 2023-03-09T20:10:47.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18756.233494 secs 2023-03-09T20:10:47.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18756.233494 secs 2023-03-09T20:10:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:47 smithi122 ceph-mon[93869]: pgmap v10515: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 99 KiB/s wr, 9 op/s 2023-03-09T20:10:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:47 smithi181 ceph-mon[94189]: pgmap v10515: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 99 KiB/s wr, 9 op/s 2023-03-09T20:10:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:47 smithi038 ceph-mon[92422]: pgmap v10515: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 99 KiB/s wr, 9 op/s 2023-03-09T20:10:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19220.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:10:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:49 smithi122 ceph-mon[93869]: pgmap v10516: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 99 KiB/s wr, 7 op/s 2023-03-09T20:10:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:49 smithi181 ceph-mon[94189]: pgmap v10516: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 99 KiB/s wr, 7 op/s 2023-03-09T20:10:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:49 smithi038 ceph-mon[92422]: pgmap v10516: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 99 KiB/s wr, 7 op/s 2023-03-09T20:10:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18761.233558 secs 2023-03-09T20:10:51.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18761.233558 secs 2023-03-09T20:10:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18761.233558 secs 2023-03-09T20:10:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:51 smithi122 ceph-mon[93869]: pgmap v10517: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 229 KiB/s wr, 7 op/s 2023-03-09T20:10:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:51 smithi181 ceph-mon[94189]: pgmap v10517: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 229 KiB/s wr, 7 op/s 2023-03-09T20:10:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:51 smithi038 ceph-mon[92422]: pgmap v10517: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 229 KiB/s wr, 7 op/s 2023-03-09T20:10:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:52 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:10:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:10:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:10:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:52 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:10:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:10:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:10:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:52 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:10:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:10:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:10:53.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19225.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:10:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:53 smithi122 ceph-mon[93869]: pgmap v10518: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 306 KiB/s wr, 11 op/s 2023-03-09T20:10:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:53 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 18763.551646 secs 2023-03-09T20:10:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:53 smithi181 ceph-mon[94189]: pgmap v10518: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 306 KiB/s wr, 11 op/s 2023-03-09T20:10:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:53 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 18763.551646 secs 2023-03-09T20:10:54.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:53 smithi038 ceph-mon[92422]: pgmap v10518: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 306 KiB/s wr, 11 op/s 2023-03-09T20:10:54.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:53 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 18763.551646 secs 2023-03-09T20:10:55.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:54 smithi122 ceph-mon[93869]: pgmap v10519: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 350 KiB/s wr, 10 op/s 2023-03-09T20:10:55.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:54 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18766.233626 secs 2023-03-09T20:10:55.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:54 smithi181 ceph-mon[94189]: pgmap v10519: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 350 KiB/s wr, 10 op/s 2023-03-09T20:10:55.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:54 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18766.233626 secs 2023-03-09T20:10:55.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:54 smithi038 ceph-mon[92422]: pgmap v10519: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 350 KiB/s wr, 10 op/s 2023-03-09T20:10:55.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:54 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18766.233626 secs 2023-03-09T20:10:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:10:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:10:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:10:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:10:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:10:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:10:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:10:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:10:57.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:56 smithi122 ceph-mon[93869]: pgmap v10520: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 466 KiB/s wr, 13 op/s 2023-03-09T20:10:57.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:10:57.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:10:57.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:56 smithi181 ceph-mon[94189]: pgmap v10520: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 466 KiB/s wr, 13 op/s 2023-03-09T20:10:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:10:57.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:10:57.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:56 smithi038 ceph-mon[92422]: pgmap v10520: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 466 KiB/s wr, 13 op/s 2023-03-09T20:10:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:58 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 18768.551782 secs 2023-03-09T20:10:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:58 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 18768.551782 secs 2023-03-09T20:10:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:58 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 18768.551782 secs 2023-03-09T20:10:58.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19230.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:10:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:10:59 smithi122 ceph-mon[93869]: pgmap v10521: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 451 KiB/s wr, 11 op/s 2023-03-09T20:11:00.065 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:10:59 smithi038 ceph-mon[92422]: pgmap v10521: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 451 KiB/s wr, 11 op/s 2023-03-09T20:11:00.066 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:10:59 smithi181 ceph-mon[94189]: pgmap v10521: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 451 KiB/s wr, 11 op/s 2023-03-09T20:11:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18771.233688 secs 2023-03-09T20:11:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18771.233688 secs 2023-03-09T20:11:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18771.233688 secs 2023-03-09T20:11:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:01 smithi122 ceph-mon[93869]: pgmap v10522: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 451 KiB/s wr, 11 op/s 2023-03-09T20:11:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:01 smithi181 ceph-mon[94189]: pgmap v10522: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 451 KiB/s wr, 11 op/s 2023-03-09T20:11:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:01 smithi038 ceph-mon[92422]: pgmap v10522: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 451 KiB/s wr, 11 op/s 2023-03-09T20:11:03.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19235.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:11:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:03 smithi122 ceph-mon[93869]: pgmap v10523: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 513 KiB/s rd, 433 KiB/s wr, 14 op/s 2023-03-09T20:11:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:03 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 18773.551913 secs 2023-03-09T20:11:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:03 smithi181 ceph-mon[94189]: pgmap v10523: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 513 KiB/s rd, 433 KiB/s wr, 14 op/s 2023-03-09T20:11:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:03 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 18773.551913 secs 2023-03-09T20:11:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:03 smithi038 ceph-mon[92422]: pgmap v10523: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 513 KiB/s rd, 433 KiB/s wr, 14 op/s 2023-03-09T20:11:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:03 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 18773.551913 secs 2023-03-09T20:11:04.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:11:04.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:11:04.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:11:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:05 smithi122 ceph-mon[93869]: pgmap v10524: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 356 KiB/s wr, 12 op/s 2023-03-09T20:11:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:05 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:11:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:05 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:11:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:05 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:11:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18776.233749 secs 2023-03-09T20:11:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:05 smithi181 ceph-mon[94189]: pgmap v10524: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 356 KiB/s wr, 12 op/s 2023-03-09T20:11:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:05 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:11:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:05 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:11:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:05 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:11:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18776.233749 secs 2023-03-09T20:11:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:05 smithi038 ceph-mon[92422]: pgmap v10524: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 356 KiB/s wr, 12 op/s 2023-03-09T20:11:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:05 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:11:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:05 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:11:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:05 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:11:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18776.233749 secs 2023-03-09T20:11:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:07 smithi122 ceph-mon[93869]: pgmap v10525: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 629 KiB/s rd, 350 KiB/s wr, 16 op/s 2023-03-09T20:11:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:07 smithi181 ceph-mon[94189]: pgmap v10525: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 629 KiB/s rd, 350 KiB/s wr, 16 op/s 2023-03-09T20:11:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:07 smithi038 ceph-mon[92422]: pgmap v10525: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 629 KiB/s rd, 350 KiB/s wr, 16 op/s 2023-03-09T20:11:08.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19240.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:11:08.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18778.552021 secs 2023-03-09T20:11:08.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:11:08.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:11:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18778.552021 secs 2023-03-09T20:11:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:11:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:11:09.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18778.552021 secs 2023-03-09T20:11:09.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:11:09.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:11:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:09 smithi181 ceph-mon[94189]: pgmap v10526: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 589 KiB/s rd, 325 KiB/s wr, 25 op/s 2023-03-09T20:11:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:11:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:11:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:09 smithi038 ceph-mon[92422]: pgmap v10526: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 589 KiB/s rd, 325 KiB/s wr, 25 op/s 2023-03-09T20:11:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:11:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:11:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:09 smithi122 ceph-mon[93869]: pgmap v10526: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 589 KiB/s rd, 325 KiB/s wr, 25 op/s 2023-03-09T20:11:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:11:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:11:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:10 smithi038 ceph-mon[92422]: osdmap e871: 12 total, 12 up, 12 in 2023-03-09T20:11:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18781.233810 secs 2023-03-09T20:11:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:10 smithi122 ceph-mon[93869]: osdmap e871: 12 total, 12 up, 12 in 2023-03-09T20:11:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18781.233810 secs 2023-03-09T20:11:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:10 smithi181 ceph-mon[94189]: osdmap e871: 12 total, 12 up, 12 in 2023-03-09T20:11:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18781.233810 secs 2023-03-09T20:11:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:11 smithi038 ceph-mon[92422]: pgmap v10528: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 707 KiB/s rd, 390 KiB/s wr, 28 op/s 2023-03-09T20:11:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:11 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:11:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:11 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:11:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:11 smithi122 ceph-mon[93869]: pgmap v10528: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 707 KiB/s rd, 390 KiB/s wr, 28 op/s 2023-03-09T20:11:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:11 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:11:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:11 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:11:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:11 smithi181 ceph-mon[94189]: pgmap v10528: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 707 KiB/s rd, 390 KiB/s wr, 28 op/s 2023-03-09T20:11:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:11 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:11:12.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:11 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:11:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19245.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:11:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:13 smithi038 ceph-mon[92422]: pgmap v10529: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 724 KiB/s rd, 480 KiB/s wr, 30 op/s 2023-03-09T20:11:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18783.552129 secs 2023-03-09T20:11:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:13 smithi122 ceph-mon[93869]: pgmap v10529: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 724 KiB/s rd, 480 KiB/s wr, 30 op/s 2023-03-09T20:11:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18783.552129 secs 2023-03-09T20:11:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:13 smithi181 ceph-mon[94189]: pgmap v10529: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 724 KiB/s rd, 480 KiB/s wr, 30 op/s 2023-03-09T20:11:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18783.552129 secs 2023-03-09T20:11:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:15 smithi122 ceph-mon[93869]: pgmap v10530: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 724 KiB/s rd, 480 KiB/s wr, 28 op/s 2023-03-09T20:11:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18786.233869 secs 2023-03-09T20:11:16.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:15 smithi181 ceph-mon[94189]: pgmap v10530: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 724 KiB/s rd, 480 KiB/s wr, 28 op/s 2023-03-09T20:11:16.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18786.233869 secs 2023-03-09T20:11:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:15 smithi038 ceph-mon[92422]: pgmap v10530: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 724 KiB/s rd, 480 KiB/s wr, 28 op/s 2023-03-09T20:11:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18786.233869 secs 2023-03-09T20:11:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:17 smithi122 ceph-mon[93869]: pgmap v10531: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 492 KiB/s wr, 25 op/s 2023-03-09T20:11:18.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:17 smithi181 ceph-mon[94189]: pgmap v10531: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 492 KiB/s wr, 25 op/s 2023-03-09T20:11:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:17 smithi038 ceph-mon[92422]: pgmap v10531: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 492 KiB/s wr, 25 op/s 2023-03-09T20:11:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19250.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:11:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18788.552183 secs 2023-03-09T20:11:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:18 smithi122 ceph-mon[93869]: osdmap e872: 12 total, 12 up, 12 in 2023-03-09T20:11:19.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18788.552183 secs 2023-03-09T20:11:19.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:18 smithi181 ceph-mon[94189]: osdmap e872: 12 total, 12 up, 12 in 2023-03-09T20:11:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18788.552183 secs 2023-03-09T20:11:19.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:18 smithi038 ceph-mon[92422]: osdmap e872: 12 total, 12 up, 12 in 2023-03-09T20:11:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:19 smithi122 ceph-mon[93869]: pgmap v10532: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 302 KiB/s wr, 10 op/s 2023-03-09T20:11:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:19 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:11:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:19 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:11:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:19 smithi181 ceph-mon[94189]: pgmap v10532: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 302 KiB/s wr, 10 op/s 2023-03-09T20:11:20.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:19 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:11:20.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:19 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:11:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:19 smithi038 ceph-mon[92422]: pgmap v10532: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 302 KiB/s wr, 10 op/s 2023-03-09T20:11:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:19 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:11:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:19 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:11:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:11:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:11:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18791.233930 secs 2023-03-09T20:11:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:11:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:11:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18791.233930 secs 2023-03-09T20:11:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:11:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:11:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18791.233930 secs 2023-03-09T20:11:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:21 smithi122 ceph-mon[93869]: pgmap v10534: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 302 KiB/s wr, 11 op/s 2023-03-09T20:11:22.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:21 smithi181 ceph-mon[94189]: pgmap v10534: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 302 KiB/s wr, 11 op/s 2023-03-09T20:11:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:21 smithi038 ceph-mon[92422]: pgmap v10534: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 302 KiB/s wr, 11 op/s 2023-03-09T20:11:23.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19255.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:11:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:23 smithi122 ceph-mon[93869]: pgmap v10535: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 79 KiB/s wr, 8 op/s 2023-03-09T20:11:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18793.552303 secs 2023-03-09T20:11:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:23 smithi181 ceph-mon[94189]: pgmap v10535: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 79 KiB/s wr, 8 op/s 2023-03-09T20:11:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18793.552303 secs 2023-03-09T20:11:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:23 smithi038 ceph-mon[92422]: pgmap v10535: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 79 KiB/s wr, 8 op/s 2023-03-09T20:11:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18793.552303 secs 2023-03-09T20:11:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:25 smithi122 ceph-mon[93869]: pgmap v10536: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 78 KiB/s wr, 8 op/s 2023-03-09T20:11:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18796.233987 secs 2023-03-09T20:11:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:25 smithi181 ceph-mon[94189]: pgmap v10536: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 78 KiB/s wr, 8 op/s 2023-03-09T20:11:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18796.233987 secs 2023-03-09T20:11:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:25 smithi038 ceph-mon[92422]: pgmap v10536: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 78 KiB/s wr, 8 op/s 2023-03-09T20:11:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18796.233987 secs 2023-03-09T20:11:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:27 smithi122 ceph-mon[93869]: pgmap v10537: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 67 KiB/s wr, 8 op/s 2023-03-09T20:11:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:27 smithi181 ceph-mon[94189]: pgmap v10537: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 67 KiB/s wr, 8 op/s 2023-03-09T20:11:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:27 smithi038 ceph-mon[92422]: pgmap v10537: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 67 KiB/s wr, 8 op/s 2023-03-09T20:11:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19260.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:11:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18798.552419 secs 2023-03-09T20:11:29.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18798.552419 secs 2023-03-09T20:11:29.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18798.552419 secs 2023-03-09T20:11:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:29 smithi122 ceph-mon[93869]: pgmap v10538: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 47 KiB/s wr, 8 op/s 2023-03-09T20:11:30.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:29 smithi181 ceph-mon[94189]: pgmap v10538: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 47 KiB/s wr, 8 op/s 2023-03-09T20:11:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:29 smithi038 ceph-mon[92422]: pgmap v10538: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 47 KiB/s wr, 8 op/s 2023-03-09T20:11:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18801.234048 secs 2023-03-09T20:11:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18801.234048 secs 2023-03-09T20:11:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18801.234048 secs 2023-03-09T20:11:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:31 smithi122 ceph-mon[93869]: pgmap v10539: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 115 KiB/s wr, 8 op/s 2023-03-09T20:11:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:31 smithi181 ceph-mon[94189]: pgmap v10539: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 115 KiB/s wr, 8 op/s 2023-03-09T20:11:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:31 smithi038 ceph-mon[92422]: pgmap v10539: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 115 KiB/s wr, 8 op/s 2023-03-09T20:11:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19265.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:11:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:33 smithi122 ceph-mon[93869]: pgmap v10540: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 108 KiB/s wr, 9 op/s 2023-03-09T20:11:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18803.552545 secs 2023-03-09T20:11:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:33 smithi181 ceph-mon[94189]: pgmap v10540: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 108 KiB/s wr, 9 op/s 2023-03-09T20:11:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18803.552545 secs 2023-03-09T20:11:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:33 smithi038 ceph-mon[92422]: pgmap v10540: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 108 KiB/s wr, 9 op/s 2023-03-09T20:11:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18803.552545 secs 2023-03-09T20:11:35.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:34 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18806.234110 secs 2023-03-09T20:11:35.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:34 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18806.234110 secs 2023-03-09T20:11:35.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:34 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18806.234110 secs 2023-03-09T20:11:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:35 smithi122 ceph-mon[93869]: pgmap v10541: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 108 KiB/s wr, 7 op/s 2023-03-09T20:11:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:35 smithi181 ceph-mon[94189]: pgmap v10541: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 108 KiB/s wr, 7 op/s 2023-03-09T20:11:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:35 smithi038 ceph-mon[92422]: pgmap v10541: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 108 KiB/s wr, 7 op/s 2023-03-09T20:11:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:37 smithi122 ceph-mon[93869]: pgmap v10542: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 108 KiB/s wr, 9 op/s 2023-03-09T20:11:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:37 smithi181 ceph-mon[94189]: pgmap v10542: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 108 KiB/s wr, 9 op/s 2023-03-09T20:11:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:37 smithi038 ceph-mon[92422]: pgmap v10542: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 108 KiB/s wr, 9 op/s 2023-03-09T20:11:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19270.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:11:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18808.552683 secs 2023-03-09T20:11:39.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18808.552683 secs 2023-03-09T20:11:39.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18808.552683 secs 2023-03-09T20:11:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:39 smithi122 ceph-mon[93869]: pgmap v10543: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 70 KiB/s wr, 6 op/s 2023-03-09T20:11:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:39 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18811.234180 secs 2023-03-09T20:11:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:39 smithi181 ceph-mon[94189]: pgmap v10543: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 70 KiB/s wr, 6 op/s 2023-03-09T20:11:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:39 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18811.234180 secs 2023-03-09T20:11:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:39 smithi038 ceph-mon[92422]: pgmap v10543: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 70 KiB/s wr, 6 op/s 2023-03-09T20:11:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:39 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18811.234180 secs 2023-03-09T20:11:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:41 smithi122 ceph-mon[93869]: pgmap v10544: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 144 KiB/s wr, 6 op/s 2023-03-09T20:11:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:41 smithi181 ceph-mon[94189]: pgmap v10544: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 144 KiB/s wr, 6 op/s 2023-03-09T20:11:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:41 smithi038 ceph-mon[92422]: pgmap v10544: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 144 KiB/s wr, 6 op/s 2023-03-09T20:11:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19275.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:11:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:43 smithi122 ceph-mon[93869]: pgmap v10545: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 75 KiB/s wr, 8 op/s 2023-03-09T20:11:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18813.552744 secs 2023-03-09T20:11:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:43 smithi181 ceph-mon[94189]: pgmap v10545: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 75 KiB/s wr, 8 op/s 2023-03-09T20:11:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18813.552744 secs 2023-03-09T20:11:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:43 smithi038 ceph-mon[92422]: pgmap v10545: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 75 KiB/s wr, 8 op/s 2023-03-09T20:11:44.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18813.552744 secs 2023-03-09T20:11:45.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:44 smithi122 ceph-mon[93869]: pgmap v10546: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 75 KiB/s wr, 6 op/s 2023-03-09T20:11:45.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:44 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18816.234245 secs 2023-03-09T20:11:45.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:44 smithi181 ceph-mon[94189]: pgmap v10546: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 75 KiB/s wr, 6 op/s 2023-03-09T20:11:45.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:44 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18816.234245 secs 2023-03-09T20:11:45.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:44 smithi038 ceph-mon[92422]: pgmap v10546: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 75 KiB/s wr, 6 op/s 2023-03-09T20:11:45.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:44 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18816.234245 secs 2023-03-09T20:11:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:47 smithi122 ceph-mon[93869]: pgmap v10547: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 204 KiB/s wr, 9 op/s 2023-03-09T20:11:47.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:47 smithi181 ceph-mon[94189]: pgmap v10547: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 204 KiB/s wr, 9 op/s 2023-03-09T20:11:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:47 smithi038 ceph-mon[92422]: pgmap v10547: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 204 KiB/s wr, 9 op/s 2023-03-09T20:11:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18818.552862 secs 2023-03-09T20:11:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18818.552862 secs 2023-03-09T20:11:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18818.552862 secs 2023-03-09T20:11:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19280.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:11:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:49 smithi122 ceph-mon[93869]: pgmap v10548: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 221 KiB/s wr, 7 op/s 2023-03-09T20:11:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:49 smithi181 ceph-mon[94189]: pgmap v10548: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 221 KiB/s wr, 7 op/s 2023-03-09T20:11:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:49 smithi038 ceph-mon[92422]: pgmap v10548: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 221 KiB/s wr, 7 op/s 2023-03-09T20:11:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18821.234331 secs 2023-03-09T20:11:50.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18821.234331 secs 2023-03-09T20:11:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18821.234331 secs 2023-03-09T20:11:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:51 smithi122 ceph-mon[93869]: pgmap v10549: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 221 KiB/s wr, 7 op/s 2023-03-09T20:11:51.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:11:51.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:11:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:51 smithi181 ceph-mon[94189]: pgmap v10549: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 221 KiB/s wr, 7 op/s 2023-03-09T20:11:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:11:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:11:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:51 smithi038 ceph-mon[92422]: pgmap v10549: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 221 KiB/s wr, 7 op/s 2023-03-09T20:11:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:11:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:11:52.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:11:52.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:11:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:11:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:11:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:11:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:11:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19285.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:11:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:53 smithi038 ceph-mon[92422]: pgmap v10550: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 161 KiB/s wr, 10 op/s 2023-03-09T20:11:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18823.552974 secs 2023-03-09T20:11:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:53 smithi122 ceph-mon[93869]: pgmap v10550: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 161 KiB/s wr, 10 op/s 2023-03-09T20:11:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18823.552974 secs 2023-03-09T20:11:54.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:53 smithi181 ceph-mon[94189]: pgmap v10550: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 161 KiB/s wr, 10 op/s 2023-03-09T20:11:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18823.552974 secs 2023-03-09T20:11:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:55 smithi038 ceph-mon[92422]: pgmap v10551: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 481 KiB/s rd, 392 KiB/s wr, 9 op/s 2023-03-09T20:11:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18826.234390 secs 2023-03-09T20:11:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:55 smithi122 ceph-mon[93869]: pgmap v10551: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 481 KiB/s rd, 392 KiB/s wr, 9 op/s 2023-03-09T20:11:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18826.234390 secs 2023-03-09T20:11:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:55 smithi181 ceph-mon[94189]: pgmap v10551: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 481 KiB/s rd, 392 KiB/s wr, 9 op/s 2023-03-09T20:11:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18826.234390 secs 2023-03-09T20:11:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:57 smithi038 ceph-mon[92422]: pgmap v10552: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 423 KiB/s wr, 13 op/s 2023-03-09T20:11:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:57 smithi122 ceph-mon[93869]: pgmap v10552: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 423 KiB/s wr, 13 op/s 2023-03-09T20:11:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:57 smithi181 ceph-mon[94189]: pgmap v10552: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 423 KiB/s wr, 13 op/s 2023-03-09T20:11:58.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19290.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:11:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18828.553091 secs 2023-03-09T20:11:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18828.553091 secs 2023-03-09T20:11:59.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18828.553091 secs 2023-03-09T20:11:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:11:59 smithi038 ceph-mon[92422]: pgmap v10553: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 546 KiB/s rd, 389 KiB/s wr, 11 op/s 2023-03-09T20:12:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:11:59 smithi122 ceph-mon[93869]: pgmap v10553: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 546 KiB/s rd, 389 KiB/s wr, 11 op/s 2023-03-09T20:12:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:11:59 smithi181 ceph-mon[94189]: pgmap v10553: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 546 KiB/s rd, 389 KiB/s wr, 11 op/s 2023-03-09T20:12:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18831.234451 secs 2023-03-09T20:12:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18831.234451 secs 2023-03-09T20:12:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18831.234451 secs 2023-03-09T20:12:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:01 smithi122 ceph-mon[93869]: pgmap v10554: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 530 KiB/s rd, 372 KiB/s wr, 11 op/s 2023-03-09T20:12:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:01 smithi181 ceph-mon[94189]: pgmap v10554: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 530 KiB/s rd, 372 KiB/s wr, 11 op/s 2023-03-09T20:12:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:01 smithi038 ceph-mon[92422]: pgmap v10554: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 530 KiB/s rd, 372 KiB/s wr, 11 op/s 2023-03-09T20:12:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19295.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:12:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:03 smithi122 ceph-mon[93869]: pgmap v10555: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 438 KiB/s wr, 17 op/s 2023-03-09T20:12:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18833.553203 secs 2023-03-09T20:12:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:03 smithi181 ceph-mon[94189]: pgmap v10555: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 438 KiB/s wr, 17 op/s 2023-03-09T20:12:04.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18833.553203 secs 2023-03-09T20:12:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:03 smithi038 ceph-mon[92422]: pgmap v10555: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 438 KiB/s wr, 17 op/s 2023-03-09T20:12:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18833.553203 secs 2023-03-09T20:12:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:05 smithi122 ceph-mon[93869]: pgmap v10556: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 444 KiB/s wr, 26 op/s 2023-03-09T20:12:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:05 smithi122 ceph-mon[93869]: osdmap e873: 12 total, 12 up, 12 in 2023-03-09T20:12:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18836.234514 secs 2023-03-09T20:12:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:05 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:12:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:05 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:12:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:05 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:12:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:05 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:12:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:05 smithi181 ceph-mon[94189]: pgmap v10556: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 444 KiB/s wr, 26 op/s 2023-03-09T20:12:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:05 smithi181 ceph-mon[94189]: osdmap e873: 12 total, 12 up, 12 in 2023-03-09T20:12:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18836.234514 secs 2023-03-09T20:12:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:05 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:12:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:05 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:12:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:05 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:12:06.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:05 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:12:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:05 smithi038 ceph-mon[92422]: pgmap v10556: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 444 KiB/s wr, 26 op/s 2023-03-09T20:12:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:05 smithi038 ceph-mon[92422]: osdmap e873: 12 total, 12 up, 12 in 2023-03-09T20:12:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18836.234514 secs 2023-03-09T20:12:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:05 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:12:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:05 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:12:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:05 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:12:06.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:05 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:12:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:07 smithi122 ceph-mon[93869]: pgmap v10558: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 408 KiB/s wr, 30 op/s 2023-03-09T20:12:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:12:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:12:08.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:07 smithi181 ceph-mon[94189]: pgmap v10558: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 408 KiB/s wr, 30 op/s 2023-03-09T20:12:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:12:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:12:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:07 smithi038 ceph-mon[92422]: pgmap v10558: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 408 KiB/s wr, 30 op/s 2023-03-09T20:12:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:12:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:12:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19300.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:12:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18838.553320 secs 2023-03-09T20:12:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:12:09.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:12:09.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18838.553320 secs 2023-03-09T20:12:09.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:12:09.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:12:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18838.553320 secs 2023-03-09T20:12:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:12:09.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:12:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:09 smithi122 ceph-mon[93869]: pgmap v10559: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 294 KiB/s wr, 29 op/s 2023-03-09T20:12:10.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:09 smithi181 ceph-mon[94189]: pgmap v10559: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 294 KiB/s wr, 29 op/s 2023-03-09T20:12:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:09 smithi038 ceph-mon[92422]: pgmap v10559: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 294 KiB/s wr, 29 op/s 2023-03-09T20:12:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18841.234573 secs 2023-03-09T20:12:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18841.234573 secs 2023-03-09T20:12:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18841.234573 secs 2023-03-09T20:12:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:11 smithi122 ceph-mon[93869]: pgmap v10560: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 294 KiB/s wr, 28 op/s 2023-03-09T20:12:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:11 smithi181 ceph-mon[94189]: pgmap v10560: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 294 KiB/s wr, 28 op/s 2023-03-09T20:12:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:11 smithi038 ceph-mon[92422]: pgmap v10560: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 294 KiB/s wr, 28 op/s 2023-03-09T20:12:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19305.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:12:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:13 smithi122 ceph-mon[93869]: pgmap v10561: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 272 KiB/s wr, 24 op/s 2023-03-09T20:12:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18843.553440 secs 2023-03-09T20:12:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:13 smithi122 ceph-mon[93869]: osdmap e874: 12 total, 12 up, 12 in 2023-03-09T20:12:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:13 smithi181 ceph-mon[94189]: pgmap v10561: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 272 KiB/s wr, 24 op/s 2023-03-09T20:12:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18843.553440 secs 2023-03-09T20:12:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:13 smithi181 ceph-mon[94189]: osdmap e874: 12 total, 12 up, 12 in 2023-03-09T20:12:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:13 smithi038 ceph-mon[92422]: pgmap v10561: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 272 KiB/s wr, 24 op/s 2023-03-09T20:12:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18843.553440 secs 2023-03-09T20:12:14.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:13 smithi038 ceph-mon[92422]: osdmap e874: 12 total, 12 up, 12 in 2023-03-09T20:12:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:15 smithi122 ceph-mon[93869]: pgmap v10563: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 267 KiB/s wr, 10 op/s 2023-03-09T20:12:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18846.234634 secs 2023-03-09T20:12:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:15 smithi181 ceph-mon[94189]: pgmap v10563: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 267 KiB/s wr, 10 op/s 2023-03-09T20:12:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18846.234634 secs 2023-03-09T20:12:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:15 smithi038 ceph-mon[92422]: pgmap v10563: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 267 KiB/s wr, 10 op/s 2023-03-09T20:12:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18846.234634 secs 2023-03-09T20:12:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:17 smithi122 ceph-mon[93869]: pgmap v10564: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 8 op/s 2023-03-09T20:12:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:17 smithi181 ceph-mon[94189]: pgmap v10564: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 8 op/s 2023-03-09T20:12:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:17 smithi038 ceph-mon[92422]: pgmap v10564: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 8 op/s 2023-03-09T20:12:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19310.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:12:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18848.553562 secs 2023-03-09T20:12:19.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18848.553562 secs 2023-03-09T20:12:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18848.553562 secs 2023-03-09T20:12:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:19 smithi122 ceph-mon[93869]: pgmap v10565: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 7 op/s 2023-03-09T20:12:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:19 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18851.234695 secs 2023-03-09T20:12:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:19 smithi181 ceph-mon[94189]: pgmap v10565: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 7 op/s 2023-03-09T20:12:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:19 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18851.234695 secs 2023-03-09T20:12:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:19 smithi038 ceph-mon[92422]: pgmap v10565: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 7 op/s 2023-03-09T20:12:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:19 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18851.234695 secs 2023-03-09T20:12:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:21 smithi122 ceph-mon[93869]: pgmap v10566: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 7 op/s 2023-03-09T20:12:22.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:21 smithi181 ceph-mon[94189]: pgmap v10566: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 7 op/s 2023-03-09T20:12:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:21 smithi038 ceph-mon[92422]: pgmap v10566: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 7 op/s 2023-03-09T20:12:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19315.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:12:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:23 smithi122 ceph-mon[93869]: pgmap v10567: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-03-09T20:12:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18853.553693 secs 2023-03-09T20:12:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:23 smithi181 ceph-mon[94189]: pgmap v10567: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-03-09T20:12:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18853.553693 secs 2023-03-09T20:12:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:23 smithi038 ceph-mon[92422]: pgmap v10567: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-03-09T20:12:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18853.553693 secs 2023-03-09T20:12:25.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:24 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18856.234757 secs 2023-03-09T20:12:25.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:24 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18856.234757 secs 2023-03-09T20:12:25.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:24 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18856.234757 secs 2023-03-09T20:12:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:25 smithi122 ceph-mon[93869]: pgmap v10568: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2023-03-09T20:12:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:25 smithi181 ceph-mon[94189]: pgmap v10568: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2023-03-09T20:12:26.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:25 smithi038 ceph-mon[92422]: pgmap v10568: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2023-03-09T20:12:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:26 smithi122 ceph-mon[93869]: pgmap v10569: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T20:12:27.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:26 smithi181 ceph-mon[94189]: pgmap v10569: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T20:12:27.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:26 smithi038 ceph-mon[92422]: pgmap v10569: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T20:12:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18858.553830 secs 2023-03-09T20:12:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18858.553830 secs 2023-03-09T20:12:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18858.553830 secs 2023-03-09T20:12:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19320.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:12:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:29 smithi122 ceph-mon[93869]: pgmap v10570: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T20:12:29.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:29 smithi181 ceph-mon[94189]: pgmap v10570: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T20:12:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:29 smithi038 ceph-mon[92422]: pgmap v10570: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T20:12:30.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18861.234818 secs 2023-03-09T20:12:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18861.234818 secs 2023-03-09T20:12:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18861.234818 secs 2023-03-09T20:12:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:31 smithi038 ceph-mon[92422]: pgmap v10571: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T20:12:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:31 smithi122 ceph-mon[93869]: pgmap v10571: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T20:12:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:31 smithi181 ceph-mon[94189]: pgmap v10571: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T20:12:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19325.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:12:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:33 smithi038 ceph-mon[92422]: pgmap v10572: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T20:12:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18863.553954 secs 2023-03-09T20:12:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:33 smithi122 ceph-mon[93869]: pgmap v10572: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T20:12:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18863.553954 secs 2023-03-09T20:12:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:33 smithi181 ceph-mon[94189]: pgmap v10572: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T20:12:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18863.553954 secs 2023-03-09T20:12:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:35 smithi038 ceph-mon[92422]: pgmap v10573: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T20:12:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18866.234879 secs 2023-03-09T20:12:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:35 smithi122 ceph-mon[93869]: pgmap v10573: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T20:12:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18866.234879 secs 2023-03-09T20:12:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:35 smithi181 ceph-mon[94189]: pgmap v10573: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T20:12:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18866.234879 secs 2023-03-09T20:12:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:37 smithi038 ceph-mon[92422]: pgmap v10574: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 110 KiB/s wr, 9 op/s 2023-03-09T20:12:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:37 smithi122 ceph-mon[93869]: pgmap v10574: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 110 KiB/s wr, 9 op/s 2023-03-09T20:12:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:37 smithi181 ceph-mon[94189]: pgmap v10574: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 110 KiB/s wr, 9 op/s 2023-03-09T20:12:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19330.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:12:39.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18868.554071 secs 2023-03-09T20:12:39.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18868.554071 secs 2023-03-09T20:12:39.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18868.554071 secs 2023-03-09T20:12:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:39 smithi122 ceph-mon[93869]: pgmap v10575: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 110 KiB/s wr, 7 op/s 2023-03-09T20:12:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:39 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:12:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:39 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:12:40.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:39 smithi181 ceph-mon[94189]: pgmap v10575: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 110 KiB/s wr, 7 op/s 2023-03-09T20:12:40.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:39 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:12:40.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:39 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:12:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:39 smithi038 ceph-mon[92422]: pgmap v10575: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 110 KiB/s wr, 7 op/s 2023-03-09T20:12:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:39 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:12:40.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:39 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:12:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:12:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:12:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18871.234966 secs 2023-03-09T20:12:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:12:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:12:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18871.234966 secs 2023-03-09T20:12:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:12:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:12:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18871.234966 secs 2023-03-09T20:12:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:41 smithi122 ceph-mon[93869]: pgmap v10576: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 201 KiB/s wr, 7 op/s 2023-03-09T20:12:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:41 smithi181 ceph-mon[94189]: pgmap v10576: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 201 KiB/s wr, 7 op/s 2023-03-09T20:12:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:41 smithi038 ceph-mon[92422]: pgmap v10576: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 201 KiB/s wr, 7 op/s 2023-03-09T20:12:43.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19335.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:12:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:43 smithi122 ceph-mon[93869]: pgmap v10577: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 326 KiB/s rd, 217 KiB/s wr, 10 op/s 2023-03-09T20:12:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:43 smithi181 ceph-mon[94189]: pgmap v10577: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 326 KiB/s rd, 217 KiB/s wr, 10 op/s 2023-03-09T20:12:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:43 smithi038 ceph-mon[92422]: pgmap v10577: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 326 KiB/s rd, 217 KiB/s wr, 10 op/s 2023-03-09T20:12:45.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:44 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:12:45.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:44 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:12:45.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:44 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:12:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:45 smithi122 ceph-mon[93869]: pgmap v10578: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 235 KiB/s wr, 9 op/s 2023-03-09T20:12:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:45 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:12:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:45 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:12:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:45 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:12:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18876.234990 secs 2023-03-09T20:12:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:45 smithi181 ceph-mon[94189]: pgmap v10578: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 235 KiB/s wr, 9 op/s 2023-03-09T20:12:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:45 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:12:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:45 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:12:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:45 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:12:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18876.234990 secs 2023-03-09T20:12:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:45 smithi038 ceph-mon[92422]: pgmap v10578: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 235 KiB/s wr, 9 op/s 2023-03-09T20:12:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:45 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:12:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:45 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:12:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:45 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:12:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18876.234990 secs 2023-03-09T20:12:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:47 smithi122 ceph-mon[93869]: pgmap v10579: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 250 KiB/s wr, 11 op/s 2023-03-09T20:12:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:47 smithi181 ceph-mon[94189]: pgmap v10579: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 250 KiB/s wr, 11 op/s 2023-03-09T20:12:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:47 smithi038 ceph-mon[92422]: pgmap v10579: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 250 KiB/s wr, 11 op/s 2023-03-09T20:12:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19340.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:12:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:49 smithi122 ceph-mon[93869]: pgmap v10580: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 141 KiB/s wr, 9 op/s 2023-03-09T20:12:50.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:49 smithi181 ceph-mon[94189]: pgmap v10580: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 141 KiB/s wr, 9 op/s 2023-03-09T20:12:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:49 smithi038 ceph-mon[92422]: pgmap v10580: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 141 KiB/s wr, 9 op/s 2023-03-09T20:12:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18881.235054 secs 2023-03-09T20:12:51.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18881.235054 secs 2023-03-09T20:12:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18881.235054 secs 2023-03-09T20:12:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:51 smithi122 ceph-mon[93869]: pgmap v10581: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 493 KiB/s rd, 373 KiB/s wr, 10 op/s 2023-03-09T20:12:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:51 smithi181 ceph-mon[94189]: pgmap v10581: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 493 KiB/s rd, 373 KiB/s wr, 10 op/s 2023-03-09T20:12:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:51 smithi038 ceph-mon[92422]: pgmap v10581: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 493 KiB/s rd, 373 KiB/s wr, 10 op/s 2023-03-09T20:12:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:12:53.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:12:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:12:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:12:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:12:53.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:12:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19345.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:12:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:53 smithi122 ceph-mon[93869]: pgmap v10582: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 543 KiB/s rd, 422 KiB/s wr, 14 op/s 2023-03-09T20:12:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:53 smithi181 ceph-mon[94189]: pgmap v10582: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 543 KiB/s rd, 422 KiB/s wr, 14 op/s 2023-03-09T20:12:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:53 smithi038 ceph-mon[92422]: pgmap v10582: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 543 KiB/s rd, 422 KiB/s wr, 14 op/s 2023-03-09T20:12:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:55 smithi122 ceph-mon[93869]: pgmap v10583: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 407 KiB/s wr, 11 op/s 2023-03-09T20:12:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18886.235139 secs 2023-03-09T20:12:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:12:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:12:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:55 smithi181 ceph-mon[94189]: pgmap v10583: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 407 KiB/s wr, 11 op/s 2023-03-09T20:12:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18886.235139 secs 2023-03-09T20:12:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:12:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:12:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:55 smithi038 ceph-mon[92422]: pgmap v10583: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 407 KiB/s wr, 11 op/s 2023-03-09T20:12:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18886.235139 secs 2023-03-09T20:12:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:12:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:12:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:57 smithi122 ceph-mon[93869]: pgmap v10584: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 404 KiB/s wr, 13 op/s 2023-03-09T20:12:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:57 smithi181 ceph-mon[94189]: pgmap v10584: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 404 KiB/s wr, 13 op/s 2023-03-09T20:12:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:57 smithi038 ceph-mon[92422]: pgmap v10584: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 404 KiB/s wr, 13 op/s 2023-03-09T20:12:58.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19350.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:12:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18888.554464 secs 2023-03-09T20:12:59.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18888.554464 secs 2023-03-09T20:12:59.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18888.554464 secs 2023-03-09T20:13:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:59 smithi122 ceph-mon[93869]: pgmap v10585: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 473 KiB/s wr, 11 op/s 2023-03-09T20:13:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:12:59 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18891.235206 secs 2023-03-09T20:13:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:59 smithi181 ceph-mon[94189]: pgmap v10585: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 473 KiB/s wr, 11 op/s 2023-03-09T20:13:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:12:59 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18891.235206 secs 2023-03-09T20:13:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:59 smithi038 ceph-mon[92422]: pgmap v10585: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 473 KiB/s wr, 11 op/s 2023-03-09T20:13:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:12:59 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18891.235206 secs 2023-03-09T20:13:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:00 smithi122 ceph-mon[93869]: pgmap v10586: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 473 KiB/s wr, 10 op/s 2023-03-09T20:13:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:00 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:13:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:13:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:13:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:00 smithi181 ceph-mon[94189]: pgmap v10586: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 473 KiB/s wr, 10 op/s 2023-03-09T20:13:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:00 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:13:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:13:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:13:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:00 smithi038 ceph-mon[92422]: pgmap v10586: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 473 KiB/s wr, 10 op/s 2023-03-09T20:13:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:00 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:13:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:13:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:13:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:03 smithi122 ceph-mon[93869]: pgmap v10587: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 369 KiB/s wr, 16 op/s 2023-03-09T20:13:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18893.554580 secs 2023-03-09T20:13:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:03 smithi181 ceph-mon[94189]: pgmap v10587: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 369 KiB/s wr, 16 op/s 2023-03-09T20:13:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18893.554580 secs 2023-03-09T20:13:03.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:03 smithi038 ceph-mon[92422]: pgmap v10587: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 369 KiB/s wr, 16 op/s 2023-03-09T20:13:03.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18893.554580 secs 2023-03-09T20:13:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19355.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:13:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:05 smithi181 ceph-mon[94189]: pgmap v10588: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 250 KiB/s wr, 23 op/s 2023-03-09T20:13:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18896.235269 secs 2023-03-09T20:13:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:05 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:13:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:05 smithi038 ceph-mon[92422]: pgmap v10588: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 250 KiB/s wr, 23 op/s 2023-03-09T20:13:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18896.235269 secs 2023-03-09T20:13:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:05 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:13:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:05 smithi122 ceph-mon[93869]: pgmap v10588: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 250 KiB/s wr, 23 op/s 2023-03-09T20:13:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18896.235269 secs 2023-03-09T20:13:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:05 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:13:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:06 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:13:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:06 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:13:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:06 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:13:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:06 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:13:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:06 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:13:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:06 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:13:07.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:06 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:13:07.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:06 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:13:07.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:06 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:13:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:07 smithi038 ceph-mon[92422]: pgmap v10589: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 273 KiB/s wr, 25 op/s 2023-03-09T20:13:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:07 smithi122 ceph-mon[93869]: pgmap v10589: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 273 KiB/s wr, 25 op/s 2023-03-09T20:13:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:07 smithi181 ceph-mon[94189]: pgmap v10589: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 273 KiB/s wr, 25 op/s 2023-03-09T20:13:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19360.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:13:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18898.554679 secs 2023-03-09T20:13:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18898.554679 secs 2023-03-09T20:13:09.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18898.554679 secs 2023-03-09T20:13:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:09 smithi122 ceph-mon[93869]: pgmap v10590: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 301 KiB/s wr, 24 op/s 2023-03-09T20:13:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:09 smithi181 ceph-mon[94189]: pgmap v10590: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 301 KiB/s wr, 24 op/s 2023-03-09T20:13:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:09 smithi038 ceph-mon[92422]: pgmap v10590: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 301 KiB/s wr, 24 op/s 2023-03-09T20:13:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:10 smithi122 ceph-mon[93869]: osdmap e875: 12 total, 12 up, 12 in 2023-03-09T20:13:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18901.235340 secs 2023-03-09T20:13:11.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:10 smithi181 ceph-mon[94189]: osdmap e875: 12 total, 12 up, 12 in 2023-03-09T20:13:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18901.235340 secs 2023-03-09T20:13:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:10 smithi038 ceph-mon[92422]: osdmap e875: 12 total, 12 up, 12 in 2023-03-09T20:13:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18901.235340 secs 2023-03-09T20:13:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:11 smithi122 ceph-mon[93869]: pgmap v10592: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 260 KiB/s wr, 26 op/s 2023-03-09T20:13:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:11 smithi181 ceph-mon[94189]: pgmap v10592: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 260 KiB/s wr, 26 op/s 2023-03-09T20:13:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:11 smithi038 ceph-mon[92422]: pgmap v10592: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 260 KiB/s wr, 26 op/s 2023-03-09T20:13:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19365.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:13:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:13 smithi122 ceph-mon[93869]: pgmap v10593: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 106 KiB/s wr, 22 op/s 2023-03-09T20:13:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18903.554795 secs 2023-03-09T20:13:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:13 smithi181 ceph-mon[94189]: pgmap v10593: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 106 KiB/s wr, 22 op/s 2023-03-09T20:13:14.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18903.554795 secs 2023-03-09T20:13:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:13 smithi038 ceph-mon[92422]: pgmap v10593: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 106 KiB/s wr, 22 op/s 2023-03-09T20:13:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18903.554795 secs 2023-03-09T20:13:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:15 smithi122 ceph-mon[93869]: pgmap v10594: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 80 KiB/s wr, 8 op/s 2023-03-09T20:13:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18906.235399 secs 2023-03-09T20:13:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:15 smithi181 ceph-mon[94189]: pgmap v10594: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 80 KiB/s wr, 8 op/s 2023-03-09T20:13:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18906.235399 secs 2023-03-09T20:13:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:15 smithi038 ceph-mon[92422]: pgmap v10594: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 80 KiB/s wr, 8 op/s 2023-03-09T20:13:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18906.235399 secs 2023-03-09T20:13:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:13:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:13:17.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:13:17.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:13:17.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:13:17.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:13:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:17 smithi122 ceph-mon[93869]: pgmap v10595: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 52 KiB/s wr, 8 op/s 2023-03-09T20:13:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:17 smithi181 ceph-mon[94189]: pgmap v10595: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 52 KiB/s wr, 8 op/s 2023-03-09T20:13:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:17 smithi038 ceph-mon[92422]: pgmap v10595: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 52 KiB/s wr, 8 op/s 2023-03-09T20:13:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19370.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:13:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18908.554935 secs 2023-03-09T20:13:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:18 smithi122 ceph-mon[93869]: osdmap e876: 12 total, 12 up, 12 in 2023-03-09T20:13:19.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18908.554935 secs 2023-03-09T20:13:19.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:18 smithi181 ceph-mon[94189]: osdmap e876: 12 total, 12 up, 12 in 2023-03-09T20:13:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18908.554935 secs 2023-03-09T20:13:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:18 smithi038 ceph-mon[92422]: osdmap e876: 12 total, 12 up, 12 in 2023-03-09T20:13:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:19 smithi122 ceph-mon[93869]: pgmap v10596: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 7 op/s 2023-03-09T20:13:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:19 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:13:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:19 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:13:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:19 smithi181 ceph-mon[94189]: pgmap v10596: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 7 op/s 2023-03-09T20:13:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:19 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:13:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:19 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:13:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:19 smithi038 ceph-mon[92422]: pgmap v10596: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 7 op/s 2023-03-09T20:13:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:19 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:13:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:19 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:13:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18911.235459 secs 2023-03-09T20:13:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18911.235459 secs 2023-03-09T20:13:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18911.235459 secs 2023-03-09T20:13:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:21 smithi122 ceph-mon[93869]: pgmap v10598: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 7 op/s 2023-03-09T20:13:22.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:21 smithi181 ceph-mon[94189]: pgmap v10598: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 7 op/s 2023-03-09T20:13:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:21 smithi038 ceph-mon[92422]: pgmap v10598: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 7 op/s 2023-03-09T20:13:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19375.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:13:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:23 smithi122 ceph-mon[93869]: pgmap v10599: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 7 op/s 2023-03-09T20:13:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18913.555072 secs 2023-03-09T20:13:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:23 smithi181 ceph-mon[94189]: pgmap v10599: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 7 op/s 2023-03-09T20:13:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18913.555072 secs 2023-03-09T20:13:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:23 smithi038 ceph-mon[92422]: pgmap v10599: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 7 op/s 2023-03-09T20:13:24.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18913.555072 secs 2023-03-09T20:13:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:25 smithi122 ceph-mon[93869]: pgmap v10600: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 7 op/s 2023-03-09T20:13:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18916.235518 secs 2023-03-09T20:13:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:25 smithi181 ceph-mon[94189]: pgmap v10600: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 7 op/s 2023-03-09T20:13:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18916.235518 secs 2023-03-09T20:13:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:25 smithi038 ceph-mon[92422]: pgmap v10600: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 7 op/s 2023-03-09T20:13:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18916.235518 secs 2023-03-09T20:13:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:27 smithi122 ceph-mon[93869]: pgmap v10601: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T20:13:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:27 smithi181 ceph-mon[94189]: pgmap v10601: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T20:13:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:27 smithi038 ceph-mon[92422]: pgmap v10601: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T20:13:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19380.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:13:29.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18918.555141 secs 2023-03-09T20:13:29.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18918.555141 secs 2023-03-09T20:13:29.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18918.555141 secs 2023-03-09T20:13:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:29 smithi122 ceph-mon[93869]: pgmap v10602: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T20:13:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:29 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18921.235607 secs 2023-03-09T20:13:30.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:29 smithi181 ceph-mon[94189]: pgmap v10602: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T20:13:30.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:29 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18921.235607 secs 2023-03-09T20:13:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:29 smithi038 ceph-mon[92422]: pgmap v10602: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T20:13:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:29 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18921.235607 secs 2023-03-09T20:13:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:31 smithi122 ceph-mon[93869]: pgmap v10603: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T20:13:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:31 smithi181 ceph-mon[94189]: pgmap v10603: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T20:13:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:31 smithi038 ceph-mon[92422]: pgmap v10603: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T20:13:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19385.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:13:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:33 smithi122 ceph-mon[93869]: pgmap v10604: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 9 op/s 2023-03-09T20:13:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18923.555178 secs 2023-03-09T20:13:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:33 smithi181 ceph-mon[94189]: pgmap v10604: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 9 op/s 2023-03-09T20:13:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18923.555178 secs 2023-03-09T20:13:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:33 smithi038 ceph-mon[92422]: pgmap v10604: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 9 op/s 2023-03-09T20:13:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18923.555178 secs 2023-03-09T20:13:35.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:34 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18926.235667 secs 2023-03-09T20:13:35.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:34 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18926.235667 secs 2023-03-09T20:13:35.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:34 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18926.235667 secs 2023-03-09T20:13:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:35 smithi122 ceph-mon[93869]: pgmap v10605: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-09T20:13:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:35 smithi181 ceph-mon[94189]: pgmap v10605: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-09T20:13:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:35 smithi038 ceph-mon[92422]: pgmap v10605: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-09T20:13:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:37 smithi122 ceph-mon[93869]: pgmap v10606: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 170 B/s wr, 8 op/s 2023-03-09T20:13:38.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:37 smithi181 ceph-mon[94189]: pgmap v10606: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 170 B/s wr, 8 op/s 2023-03-09T20:13:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:37 smithi038 ceph-mon[92422]: pgmap v10606: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 170 B/s wr, 8 op/s 2023-03-09T20:13:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19390.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:13:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:39 smithi122 ceph-mon[93869]: pgmap v10607: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 7 op/s 2023-03-09T20:13:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:39 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18931.235730 secs 2023-03-09T20:13:40.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:39 smithi181 ceph-mon[94189]: pgmap v10607: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 7 op/s 2023-03-09T20:13:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:39 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18931.235730 secs 2023-03-09T20:13:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:39 smithi038 ceph-mon[92422]: pgmap v10607: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 7 op/s 2023-03-09T20:13:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:39 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18931.235730 secs 2023-03-09T20:13:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:40 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:13:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:40 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:13:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:40 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:13:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:41 smithi122 ceph-mon[93869]: pgmap v10608: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 6 op/s 2023-03-09T20:13:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:41 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:13:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:13:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:13:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:41 smithi181 ceph-mon[94189]: pgmap v10608: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 6 op/s 2023-03-09T20:13:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:41 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:13:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:13:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:13:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:41 smithi038 ceph-mon[92422]: pgmap v10608: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 6 op/s 2023-03-09T20:13:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:41 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:13:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:13:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:13:43.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:42 smithi122 ceph-mon[93869]: pgmap v10609: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 189 KiB/s wr, 10 op/s 2023-03-09T20:13:43.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:42 smithi181 ceph-mon[94189]: pgmap v10609: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 189 KiB/s wr, 10 op/s 2023-03-09T20:13:43.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:42 smithi038 ceph-mon[92422]: pgmap v10609: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 189 KiB/s wr, 10 op/s 2023-03-09T20:13:43.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19395.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:13:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:45 smithi038 ceph-mon[92422]: pgmap v10610: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 189 KiB/s wr, 7 op/s 2023-03-09T20:13:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18936.235790 secs 2023-03-09T20:13:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:45 smithi122 ceph-mon[93869]: pgmap v10610: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 189 KiB/s wr, 7 op/s 2023-03-09T20:13:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18936.235790 secs 2023-03-09T20:13:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:45 smithi181 ceph-mon[94189]: pgmap v10610: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 189 KiB/s wr, 7 op/s 2023-03-09T20:13:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18936.235790 secs 2023-03-09T20:13:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:47 smithi038 ceph-mon[92422]: pgmap v10611: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 363 KiB/s rd, 334 KiB/s wr, 10 op/s 2023-03-09T20:13:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:47 smithi122 ceph-mon[93869]: pgmap v10611: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 363 KiB/s rd, 334 KiB/s wr, 10 op/s 2023-03-09T20:13:48.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:47 smithi181 ceph-mon[94189]: pgmap v10611: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 363 KiB/s rd, 334 KiB/s wr, 10 op/s 2023-03-09T20:13:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19400.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:13:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:48 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 18938.555458 secs 2023-03-09T20:13:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:48 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 18938.555458 secs 2023-03-09T20:13:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:48 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 18938.555458 secs 2023-03-09T20:13:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:49 smithi122 ceph-mon[93869]: pgmap v10612: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 349 KiB/s wr, 10 op/s 2023-03-09T20:13:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:49 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:13:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:13:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:13:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:49 smithi181 ceph-mon[94189]: pgmap v10612: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 349 KiB/s wr, 10 op/s 2023-03-09T20:13:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:49 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:13:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:13:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:13:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:49 smithi038 ceph-mon[92422]: pgmap v10612: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 349 KiB/s wr, 10 op/s 2023-03-09T20:13:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:49 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:13:50.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:13:50.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:13:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18941.235851 secs 2023-03-09T20:13:51.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18941.235851 secs 2023-03-09T20:13:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18941.235851 secs 2023-03-09T20:13:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:51 smithi122 ceph-mon[93869]: pgmap v10613: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 349 KiB/s wr, 8 op/s 2023-03-09T20:13:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:13:52.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:13:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:51 smithi181 ceph-mon[94189]: pgmap v10613: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 349 KiB/s wr, 8 op/s 2023-03-09T20:13:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:13:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:13:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:51 smithi038 ceph-mon[92422]: pgmap v10613: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 349 KiB/s wr, 8 op/s 2023-03-09T20:13:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:13:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:13:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:13:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:13:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:13:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:13:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:13:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:13:53.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19405.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:13:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:53 smithi122 ceph-mon[93869]: pgmap v10614: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 577 KiB/s wr, 13 op/s 2023-03-09T20:13:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:53 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 18943.555595 secs 2023-03-09T20:13:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:53 smithi181 ceph-mon[94189]: pgmap v10614: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 577 KiB/s wr, 13 op/s 2023-03-09T20:13:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:53 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 18943.555595 secs 2023-03-09T20:13:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:53 smithi038 ceph-mon[92422]: pgmap v10614: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 577 KiB/s wr, 13 op/s 2023-03-09T20:13:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:53 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 18943.555595 secs 2023-03-09T20:13:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:55 smithi122 ceph-mon[93869]: pgmap v10615: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 443 KiB/s rd, 390 KiB/s wr, 10 op/s 2023-03-09T20:13:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18946.235910 secs 2023-03-09T20:13:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:55 smithi181 ceph-mon[94189]: pgmap v10615: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 443 KiB/s rd, 390 KiB/s wr, 10 op/s 2023-03-09T20:13:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18946.235910 secs 2023-03-09T20:13:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:55 smithi038 ceph-mon[92422]: pgmap v10615: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 443 KiB/s rd, 390 KiB/s wr, 10 op/s 2023-03-09T20:13:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18946.235910 secs 2023-03-09T20:13:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:57 smithi122 ceph-mon[93869]: pgmap v10616: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 390 KiB/s wr, 12 op/s 2023-03-09T20:13:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:57 smithi181 ceph-mon[94189]: pgmap v10616: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 390 KiB/s wr, 12 op/s 2023-03-09T20:13:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:57 smithi038 ceph-mon[92422]: pgmap v10616: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 390 KiB/s wr, 12 op/s 2023-03-09T20:13:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19410.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:13:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18948.555698 secs 2023-03-09T20:13:59.242 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18948.555698 secs 2023-03-09T20:13:59.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18948.555698 secs 2023-03-09T20:14:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:59 smithi122 ceph-mon[93869]: pgmap v10617: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 344 KiB/s wr, 11 op/s 2023-03-09T20:14:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:14:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:13:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:14:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:59 smithi181 ceph-mon[94189]: pgmap v10617: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 344 KiB/s wr, 11 op/s 2023-03-09T20:14:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:14:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:13:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:14:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:59 smithi038 ceph-mon[92422]: pgmap v10617: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 344 KiB/s wr, 11 op/s 2023-03-09T20:14:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:14:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:13:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:14:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:14:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18951.235969 secs 2023-03-09T20:14:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:14:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:14:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:14:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:14:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:14:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18951.235969 secs 2023-03-09T20:14:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:14:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:14:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:14:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:14:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:14:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18951.235969 secs 2023-03-09T20:14:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:14:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:14:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:14:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:14:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:14:01 smithi122 ceph-mon[93869]: pgmap v10618: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 329 KiB/s wr, 10 op/s 2023-03-09T20:14:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:14:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:14:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:14:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:14:02.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:14:01 smithi181 ceph-mon[94189]: pgmap v10618: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 329 KiB/s wr, 10 op/s 2023-03-09T20:14:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:14:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:14:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:14:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:14:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:14:01 smithi038 ceph-mon[92422]: pgmap v10618: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 329 KiB/s wr, 10 op/s 2023-03-09T20:14:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:14:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:14:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:14:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:14:03.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19415.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:14:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:14:03 smithi122 ceph-mon[93869]: pgmap v10619: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 464 KiB/s rd, 458 KiB/s wr, 16 op/s 2023-03-09T20:14:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:14:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18953.555814 secs 2023-03-09T20:14:04.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:14:03 smithi181 ceph-mon[94189]: pgmap v10619: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 464 KiB/s rd, 458 KiB/s wr, 16 op/s 2023-03-09T20:14:04.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:14:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18953.555814 secs 2023-03-09T20:14:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:14:03 smithi038 ceph-mon[92422]: pgmap v10619: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 464 KiB/s rd, 458 KiB/s wr, 16 op/s 2023-03-09T20:14:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:14:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18953.555814 secs 2023-03-09T20:14:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:14:05 smithi122 ceph-mon[93869]: pgmap v10620: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 251 KiB/s wr, 23 op/s 2023-03-09T20:14:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:14:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18956.236029 secs 2023-03-09T20:14:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:14:05 smithi181 ceph-mon[94189]: pgmap v10620: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 251 KiB/s wr, 23 op/s 2023-03-09T20:14:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:14:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18956.236029 secs 2023-03-09T20:14:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:14:05 smithi038 ceph-mon[92422]: pgmap v10620: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 251 KiB/s wr, 23 op/s 2023-03-09T20:14:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:14:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18956.236029 secs 2023-03-09T20:14:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:14:06 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:14:07.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:14:06 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:14:07.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:14:06 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:14:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:14:07 smithi122 ceph-mon[93869]: pgmap v10621: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 251 KiB/s wr, 25 op/s 2023-03-09T20:14:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:14:07 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:14:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:14:07 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:14:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:14:07 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:14:08.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:14:07 smithi181 ceph-mon[94189]: pgmap v10621: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 251 KiB/s wr, 25 op/s 2023-03-09T20:14:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:14:07 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:14:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:14:07 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:14:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:14:07 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:14:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:14:07 smithi038 ceph-mon[92422]: pgmap v10621: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 251 KiB/s wr, 25 op/s 2023-03-09T20:14:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:14:07 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:14:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:14:07 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:14:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:14:07 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:14:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19420.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:14:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:14:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18958.555960 secs 2023-03-09T20:14:09.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:14:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18958.555960 secs 2023-03-09T20:14:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:14:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18958.555960 secs 2023-03-09T20:14:10.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:14:09 smithi122 ceph-mon[93869]: pgmap v10622: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 283 KiB/s rd, 317 KiB/s wr, 24 op/s 2023-03-09T20:14:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:14:09 smithi181 ceph-mon[94189]: pgmap v10622: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 283 KiB/s rd, 317 KiB/s wr, 24 op/s 2023-03-09T20:14:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:14:09 smithi038 ceph-mon[92422]: pgmap v10622: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 283 KiB/s rd, 317 KiB/s wr, 24 op/s 2023-03-09T20:14:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:14:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18961.236090 secs 2023-03-09T20:14:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:14:10 smithi122 ceph-mon[93869]: osdmap e877: 12 total, 12 up, 12 in 2023-03-09T20:14:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:14:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18961.236090 secs 2023-03-09T20:14:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:14:10 smithi181 ceph-mon[94189]: osdmap e877: 12 total, 12 up, 12 in 2023-03-09T20:14:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:14:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18961.236090 secs 2023-03-09T20:14:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:14:10 smithi038 ceph-mon[92422]: osdmap e877: 12 total, 12 up, 12 in 2023-03-09T20:14:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:14:11 smithi122 ceph-mon[93869]: pgmap v10624: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 261 KiB/s wr, 26 op/s 2023-03-09T20:14:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:14:11 smithi181 ceph-mon[94189]: pgmap v10624: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 261 KiB/s wr, 26 op/s 2023-03-09T20:14:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:14:11 smithi038 ceph-mon[92422]: pgmap v10624: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 261 KiB/s wr, 26 op/s 2023-03-09T20:14:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19425.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:14:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:14:13 smithi122 ceph-mon[93869]: pgmap v10625: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 106 KiB/s wr, 22 op/s 2023-03-09T20:14:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:14:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18963.556100 secs 2023-03-09T20:14:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:14:13 smithi181 ceph-mon[94189]: pgmap v10625: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 106 KiB/s wr, 22 op/s 2023-03-09T20:14:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:14:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18963.556100 secs 2023-03-09T20:14:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:14:13 smithi038 ceph-mon[92422]: pgmap v10625: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 106 KiB/s wr, 22 op/s 2023-03-09T20:14:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:14:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18963.556100 secs 2023-03-09T20:14:15.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:14:14 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18966.236151 secs 2023-03-09T20:14:15.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:14:14 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18966.236151 secs 2023-03-09T20:14:15.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:14:14 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18966.236151 secs 2023-03-09T20:14:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:14:15 smithi122 ceph-mon[93869]: pgmap v10626: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 80 KiB/s wr, 7 op/s 2023-03-09T20:14:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:14:15 smithi181 ceph-mon[94189]: pgmap v10626: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 80 KiB/s wr, 7 op/s 2023-03-09T20:14:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:14:15 smithi038 ceph-mon[92422]: pgmap v10626: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 80 KiB/s wr, 7 op/s 2023-03-09T20:14:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:14:17 smithi122 ceph-mon[93869]: pgmap v10627: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 80 KiB/s wr, 7 op/s 2023-03-09T20:14:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:14:17 smithi181 ceph-mon[94189]: pgmap v10627: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 80 KiB/s wr, 7 op/s 2023-03-09T20:14:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:14:17 smithi038 ceph-mon[92422]: pgmap v10627: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 80 KiB/s wr, 7 op/s 2023-03-09T20:14:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19430.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:14:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:14:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18968.556181 secs 2023-03-09T20:14:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:14:18 smithi122 ceph-mon[93869]: osdmap e878: 12 total, 12 up, 12 in 2023-03-09T20:14:19.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:14:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18968.556181 secs 2023-03-09T20:14:19.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:14:18 smithi181 ceph-mon[94189]: osdmap e878: 12 total, 12 up, 12 in 2023-03-09T20:14:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:14:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18968.556181 secs 2023-03-09T20:14:19.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:14:18 smithi038 ceph-mon[92422]: osdmap e878: 12 total, 12 up, 12 in 2023-03-09T20:14:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:14:19 smithi122 ceph-mon[93869]: pgmap v10628: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2023-03-09T20:14:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:14:19 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18971.236214 secs 2023-03-09T20:14:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:14:19 smithi181 ceph-mon[94189]: pgmap v10628: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2023-03-09T20:14:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:14:19 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18971.236214 secs 2023-03-09T20:14:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:14:19 smithi038 ceph-mon[92422]: pgmap v10628: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2023-03-09T20:14:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:14:19 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18971.236214 secs 2023-03-09T20:14:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:14:21 smithi122 ceph-mon[93869]: pgmap v10630: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2023-03-09T20:14:22.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:14:21 smithi181 ceph-mon[94189]: pgmap v10630: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2023-03-09T20:14:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:14:21 smithi038 ceph-mon[92422]: pgmap v10630: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2023-03-09T20:14:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:14:22 smithi122 ceph-mon[93869]: pgmap v10631: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T20:14:23.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:14:22 smithi181 ceph-mon[94189]: pgmap v10631: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T20:14:23.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:14:22 smithi038 ceph-mon[92422]: pgmap v10631: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T20:14:23.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19435.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:14:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:14:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18973.556244 secs 2023-03-09T20:14:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:14:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18973.556244 secs 2023-03-09T20:14:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:14:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18973.556244 secs 2023-03-09T20:14:25.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:14:24 smithi122 ceph-mon[93869]: pgmap v10632: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T20:14:25.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:14:24 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18976.236279 secs 2023-03-09T20:14:25.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:14:24 smithi181 ceph-mon[94189]: pgmap v10632: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T20:14:25.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:14:24 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18976.236279 secs 2023-03-09T20:14:25.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:14:24 smithi038 ceph-mon[92422]: pgmap v10632: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T20:14:25.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:14:24 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18976.236279 secs 2023-03-09T20:14:27.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:14:27 smithi122 ceph-mon[93869]: pgmap v10633: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T20:14:27.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:14:27 smithi181 ceph-mon[94189]: pgmap v10633: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T20:14:27.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:14:27 smithi038 ceph-mon[92422]: pgmap v10633: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T20:14:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:14:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18978.556402 secs 2023-03-09T20:14:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:14:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18978.556402 secs 2023-03-09T20:14:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:14:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18978.556402 secs 2023-03-09T20:14:28.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19440.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:14:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:14:29 smithi122 ceph-mon[93869]: pgmap v10634: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T20:14:30.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:14:29 smithi181 ceph-mon[94189]: pgmap v10634: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T20:14:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:14:29 smithi038 ceph-mon[92422]: pgmap v10634: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T20:14:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:14:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18981.236337 secs 2023-03-09T20:14:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:14:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18981.236337 secs 2023-03-09T20:14:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:14:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18981.236337 secs 2023-03-09T20:14:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:14:31 smithi122 ceph-mon[93869]: pgmap v10635: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 90 B/s wr, 5 op/s 2023-03-09T20:14:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:14:31 smithi181 ceph-mon[94189]: pgmap v10635: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 90 B/s wr, 5 op/s 2023-03-09T20:14:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:14:31 smithi038 ceph-mon[92422]: pgmap v10635: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 90 B/s wr, 5 op/s 2023-03-09T20:14:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19445.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:14:34.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:14:33 smithi122 ceph-mon[93869]: pgmap v10636: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 8 op/s 2023-03-09T20:14:34.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:14:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18983.556479 secs 2023-03-09T20:14:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:14:33 smithi181 ceph-mon[94189]: pgmap v10636: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 8 op/s 2023-03-09T20:14:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:14:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18983.556479 secs 2023-03-09T20:14:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:14:33 smithi038 ceph-mon[92422]: pgmap v10636: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 8 op/s 2023-03-09T20:14:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:14:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18983.556479 secs 2023-03-09T20:14:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:14:35 smithi122 ceph-mon[93869]: pgmap v10637: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 767 B/s wr, 6 op/s 2023-03-09T20:14:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:14:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18986.236400 secs 2023-03-09T20:14:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:14:35 smithi181 ceph-mon[94189]: pgmap v10637: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 767 B/s wr, 6 op/s 2023-03-09T20:14:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:14:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18986.236400 secs 2023-03-09T20:14:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:14:35 smithi038 ceph-mon[92422]: pgmap v10637: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 767 B/s wr, 6 op/s 2023-03-09T20:14:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:14:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18986.236400 secs 2023-03-09T20:14:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:14:37 smithi122 ceph-mon[93869]: pgmap v10638: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 853 B/s wr, 7 op/s 2023-03-09T20:14:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:14:37 smithi181 ceph-mon[94189]: pgmap v10638: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 853 B/s wr, 7 op/s 2023-03-09T20:14:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:14:37 smithi038 ceph-mon[92422]: pgmap v10638: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 853 B/s wr, 7 op/s 2023-03-09T20:14:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19450.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:14:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:14:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18988.556598 secs 2023-03-09T20:14:39.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:14:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18988.556598 secs 2023-03-09T20:14:39.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:14:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18988.556598 secs 2023-03-09T20:14:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:14:39 smithi122 ceph-mon[93869]: pgmap v10639: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 71 KiB/s wr, 7 op/s 2023-03-09T20:14:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:14:39 smithi181 ceph-mon[94189]: pgmap v10639: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 71 KiB/s wr, 7 op/s 2023-03-09T20:14:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:14:39 smithi038 ceph-mon[92422]: pgmap v10639: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 71 KiB/s wr, 7 op/s 2023-03-09T20:14:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:14:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18991.236462 secs 2023-03-09T20:14:41.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:14:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18991.236462 secs 2023-03-09T20:14:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:14:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18991.236462 secs 2023-03-09T20:14:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:14:41 smithi122 ceph-mon[93869]: pgmap v10640: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 71 KiB/s wr, 6 op/s 2023-03-09T20:14:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:14:41 smithi181 ceph-mon[94189]: pgmap v10640: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 71 KiB/s wr, 6 op/s 2023-03-09T20:14:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:14:41 smithi038 ceph-mon[92422]: pgmap v10640: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 71 KiB/s wr, 6 op/s 2023-03-09T20:14:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19455.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:14:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:14:43 smithi122 ceph-mon[93869]: pgmap v10641: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 230 KiB/s wr, 9 op/s 2023-03-09T20:14:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:14:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18993.556710 secs 2023-03-09T20:14:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:14:43 smithi181 ceph-mon[94189]: pgmap v10641: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 230 KiB/s wr, 9 op/s 2023-03-09T20:14:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:14:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18993.556710 secs 2023-03-09T20:14:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:14:43 smithi038 ceph-mon[92422]: pgmap v10641: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 230 KiB/s wr, 9 op/s 2023-03-09T20:14:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:14:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18993.556710 secs 2023-03-09T20:14:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:14:45 smithi122 ceph-mon[93869]: pgmap v10642: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 230 KiB/s wr, 6 op/s 2023-03-09T20:14:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:14:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18996.236521 secs 2023-03-09T20:14:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:14:45 smithi181 ceph-mon[94189]: pgmap v10642: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 230 KiB/s wr, 6 op/s 2023-03-09T20:14:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:14:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18996.236521 secs 2023-03-09T20:14:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:14:45 smithi038 ceph-mon[92422]: pgmap v10642: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 230 KiB/s wr, 6 op/s 2023-03-09T20:14:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:14:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18996.236521 secs 2023-03-09T20:14:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:14:47 smithi122 ceph-mon[93869]: pgmap v10643: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 345 KiB/s wr, 8 op/s 2023-03-09T20:14:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:14:47 smithi181 ceph-mon[94189]: pgmap v10643: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 345 KiB/s wr, 8 op/s 2023-03-09T20:14:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:14:47 smithi038 ceph-mon[92422]: pgmap v10643: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 345 KiB/s wr, 8 op/s 2023-03-09T20:14:48.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19460.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:14:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:14:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 18998.556830 secs 2023-03-09T20:14:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:14:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 18998.556830 secs 2023-03-09T20:14:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:14:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 18998.556830 secs 2023-03-09T20:14:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:14:49 smithi122 ceph-mon[93869]: pgmap v10644: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 359 KiB/s wr, 8 op/s 2023-03-09T20:14:50.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:14:49 smithi181 ceph-mon[94189]: pgmap v10644: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 359 KiB/s wr, 8 op/s 2023-03-09T20:14:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:14:49 smithi038 ceph-mon[92422]: pgmap v10644: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 359 KiB/s wr, 8 op/s 2023-03-09T20:14:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:14:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19001.236580 secs 2023-03-09T20:14:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:14:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19001.236580 secs 2023-03-09T20:14:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:14:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19001.236580 secs 2023-03-09T20:14:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:14:51 smithi122 ceph-mon[93869]: pgmap v10645: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 289 KiB/s wr, 7 op/s 2023-03-09T20:14:52.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:14:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:14:52.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:14:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:14:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:14:51 smithi181 ceph-mon[94189]: pgmap v10645: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 289 KiB/s wr, 7 op/s 2023-03-09T20:14:52.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:14:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:14:52.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:14:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:14:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:14:51 smithi038 ceph-mon[92422]: pgmap v10645: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 289 KiB/s wr, 7 op/s 2023-03-09T20:14:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:14:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:14:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:14:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:14:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:14:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:14:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:14:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:14:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:14:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:14:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:14:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:14:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:14:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:14:53.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:14:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:14:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19465.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:14:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:14:53 smithi122 ceph-mon[93869]: pgmap v10646: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 619 KiB/s rd, 534 KiB/s wr, 11 op/s 2023-03-09T20:14:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:14:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19003.556939 secs 2023-03-09T20:14:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:14:53 smithi181 ceph-mon[94189]: pgmap v10646: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 619 KiB/s rd, 534 KiB/s wr, 11 op/s 2023-03-09T20:14:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:14:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19003.556939 secs 2023-03-09T20:14:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:14:53 smithi038 ceph-mon[92422]: pgmap v10646: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 619 KiB/s rd, 534 KiB/s wr, 11 op/s 2023-03-09T20:14:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:14:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19003.556939 secs 2023-03-09T20:14:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:14:55 smithi122 ceph-mon[93869]: pgmap v10647: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 486 KiB/s wr, 10 op/s 2023-03-09T20:14:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:14:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19006.236642 secs 2023-03-09T20:14:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:14:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:14:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:14:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:14:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:14:55 smithi181 ceph-mon[94189]: pgmap v10647: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 486 KiB/s wr, 10 op/s 2023-03-09T20:14:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:14:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19006.236642 secs 2023-03-09T20:14:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:14:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:14:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:14:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:14:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:14:55 smithi038 ceph-mon[92422]: pgmap v10647: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 486 KiB/s wr, 10 op/s 2023-03-09T20:14:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:14:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19006.236642 secs 2023-03-09T20:14:56.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:14:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:14:56.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:14:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:14:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:14:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:14:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:14:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:14:57.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:14:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:14:57.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:14:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:14:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:14:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:14:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:14:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:14:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:14:57 smithi122 ceph-mon[93869]: pgmap v10648: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 554 KiB/s rd, 486 KiB/s wr, 12 op/s 2023-03-09T20:14:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:14:57 smithi181 ceph-mon[94189]: pgmap v10648: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 554 KiB/s rd, 486 KiB/s wr, 12 op/s 2023-03-09T20:14:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:14:57 smithi038 ceph-mon[92422]: pgmap v10648: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 554 KiB/s rd, 486 KiB/s wr, 12 op/s 2023-03-09T20:14:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19470.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:14:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:14:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19008.557058 secs 2023-03-09T20:14:59.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:14:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19008.557058 secs 2023-03-09T20:14:59.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:14:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19008.557058 secs 2023-03-09T20:15:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:14:59 smithi122 ceph-mon[93869]: pgmap v10649: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 453 KiB/s rd, 387 KiB/s wr, 11 op/s 2023-03-09T20:15:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:14:59 smithi181 ceph-mon[94189]: pgmap v10649: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 453 KiB/s rd, 387 KiB/s wr, 11 op/s 2023-03-09T20:15:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:14:59 smithi038 ceph-mon[92422]: pgmap v10649: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 453 KiB/s rd, 387 KiB/s wr, 11 op/s 2023-03-09T20:15:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19011.236703 secs 2023-03-09T20:15:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:00 smithi122 ceph-mon[93869]: pgmap v10650: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 471 KiB/s rd, 458 KiB/s wr, 11 op/s 2023-03-09T20:15:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19011.236703 secs 2023-03-09T20:15:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:00 smithi181 ceph-mon[94189]: pgmap v10650: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 471 KiB/s rd, 458 KiB/s wr, 11 op/s 2023-03-09T20:15:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19011.236703 secs 2023-03-09T20:15:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:00 smithi038 ceph-mon[92422]: pgmap v10650: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 471 KiB/s rd, 458 KiB/s wr, 11 op/s 2023-03-09T20:15:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:03 smithi122 ceph-mon[93869]: pgmap v10651: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 508 KiB/s wr, 16 op/s 2023-03-09T20:15:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19013.557176 secs 2023-03-09T20:15:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:03 smithi181 ceph-mon[94189]: pgmap v10651: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 508 KiB/s wr, 16 op/s 2023-03-09T20:15:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19013.557176 secs 2023-03-09T20:15:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:03 smithi038 ceph-mon[92422]: pgmap v10651: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 508 KiB/s wr, 16 op/s 2023-03-09T20:15:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19013.557176 secs 2023-03-09T20:15:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19475.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:15:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:05 smithi122 ceph-mon[93869]: pgmap v10652: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 470 KiB/s wr, 26 op/s 2023-03-09T20:15:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:05 smithi122 ceph-mon[93869]: osdmap e879: 12 total, 12 up, 12 in 2023-03-09T20:15:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19016.236766 secs 2023-03-09T20:15:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:05 smithi181 ceph-mon[94189]: pgmap v10652: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 470 KiB/s wr, 26 op/s 2023-03-09T20:15:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:05 smithi181 ceph-mon[94189]: osdmap e879: 12 total, 12 up, 12 in 2023-03-09T20:15:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19016.236766 secs 2023-03-09T20:15:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:05 smithi038 ceph-mon[92422]: pgmap v10652: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 470 KiB/s wr, 26 op/s 2023-03-09T20:15:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:05 smithi038 ceph-mon[92422]: osdmap e879: 12 total, 12 up, 12 in 2023-03-09T20:15:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19016.236766 secs 2023-03-09T20:15:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:07 smithi122 ceph-mon[93869]: pgmap v10654: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 429 KiB/s wr, 29 op/s 2023-03-09T20:15:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:07 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:15:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:07 smithi181 ceph-mon[94189]: pgmap v10654: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 429 KiB/s wr, 29 op/s 2023-03-09T20:15:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:07 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:15:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:07 smithi038 ceph-mon[92422]: pgmap v10654: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 429 KiB/s wr, 29 op/s 2023-03-09T20:15:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:07 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:15:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19480.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:15:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19018.557289 secs 2023-03-09T20:15:09.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19018.557289 secs 2023-03-09T20:15:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19018.557289 secs 2023-03-09T20:15:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:09 smithi122 ceph-mon[93869]: pgmap v10655: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 411 KiB/s wr, 28 op/s 2023-03-09T20:15:10.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:09 smithi181 ceph-mon[94189]: pgmap v10655: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 411 KiB/s wr, 28 op/s 2023-03-09T20:15:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:09 smithi038 ceph-mon[92422]: pgmap v10655: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 411 KiB/s wr, 28 op/s 2023-03-09T20:15:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19021.236828 secs 2023-03-09T20:15:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:15:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:15:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:15:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:15:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:15:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:15:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:15:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:15:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:15:11.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:15:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19021.236828 secs 2023-03-09T20:15:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:15:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:15:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:15:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:15:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:15:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:15:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:15:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:15:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:15:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:15:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19021.236828 secs 2023-03-09T20:15:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:15:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:15:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:15:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:15:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:15:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:15:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:15:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:15:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:15:11.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:15:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:11 smithi122 ceph-mon[93869]: pgmap v10656: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 382 KiB/s wr, 29 op/s 2023-03-09T20:15:12.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:11 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T20:15:12.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:11 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T20:15:12.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:11 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:15:12.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:11 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:15:12.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:11 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:15:12.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:11 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:15:12.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:11 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi038 to 107.4M 2023-03-09T20:15:12.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:11 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi038 to 112656236: error parsing value: Value '112656236' is below minimum 939524096 2023-03-09T20:15:12.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:11 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:15:12.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:11 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:15:12.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:11 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:15:12.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:11 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:15:12.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:11 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:15:12.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:11 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:15:12.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:11 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:15:12.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:11 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:15:12.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:11 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:15:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:11 smithi181 ceph-mon[94189]: pgmap v10656: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 382 KiB/s wr, 29 op/s 2023-03-09T20:15:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:11 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T20:15:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:11 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T20:15:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:11 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:15:12.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:11 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:15:12.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:11 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:15:12.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:11 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:15:12.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:11 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi038 to 107.4M 2023-03-09T20:15:12.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:11 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi038 to 112656236: error parsing value: Value '112656236' is below minimum 939524096 2023-03-09T20:15:12.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:11 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:15:12.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:11 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:15:12.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:11 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:15:12.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:11 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:15:12.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:11 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:15:12.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:11 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:15:12.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:11 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:15:12.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:11 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:15:12.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:11 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:15:12.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:11 smithi038 ceph-mon[92422]: pgmap v10656: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 382 KiB/s wr, 29 op/s 2023-03-09T20:15:12.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:11 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T20:15:12.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:11 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T20:15:12.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:11 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:15:12.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:11 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:15:12.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:11 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:15:12.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:11 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:15:12.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:11 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi038 to 107.4M 2023-03-09T20:15:12.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:11 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi038 to 112656236: error parsing value: Value '112656236' is below minimum 939524096 2023-03-09T20:15:12.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:11 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:15:12.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:11 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:15:12.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:11 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:15:12.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:11 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:15:12.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:11 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:15:12.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:11 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:15:12.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:11 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:15:12.254 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:11 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:15:12.254 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:11 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:15:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:12 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T20:15:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:12 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T20:15:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:15:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:15:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:12 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T20:15:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:12 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T20:15:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:15:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:15:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:12 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T20:15:13.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:12 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T20:15:13.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:15:13.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:15:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19485.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:15:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:13 smithi122 ceph-mon[93869]: pgmap v10657: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 322 KiB/s wr, 25 op/s 2023-03-09T20:15:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19023.557398 secs 2023-03-09T20:15:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:13 smithi122 ceph-mon[93869]: osdmap e880: 12 total, 12 up, 12 in 2023-03-09T20:15:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:13 smithi181 ceph-mon[94189]: pgmap v10657: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 322 KiB/s wr, 25 op/s 2023-03-09T20:15:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19023.557398 secs 2023-03-09T20:15:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:13 smithi181 ceph-mon[94189]: osdmap e880: 12 total, 12 up, 12 in 2023-03-09T20:15:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:13 smithi038 ceph-mon[92422]: pgmap v10657: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 322 KiB/s wr, 25 op/s 2023-03-09T20:15:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19023.557398 secs 2023-03-09T20:15:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:13 smithi038 ceph-mon[92422]: osdmap e880: 12 total, 12 up, 12 in 2023-03-09T20:15:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:15 smithi122 ceph-mon[93869]: pgmap v10659: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 81 KiB/s wr, 10 op/s 2023-03-09T20:15:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19026.236891 secs 2023-03-09T20:15:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:15 smithi181 ceph-mon[94189]: pgmap v10659: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 81 KiB/s wr, 10 op/s 2023-03-09T20:15:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19026.236891 secs 2023-03-09T20:15:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:15 smithi038 ceph-mon[92422]: pgmap v10659: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 81 KiB/s wr, 10 op/s 2023-03-09T20:15:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19026.236891 secs 2023-03-09T20:15:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:17 smithi122 ceph-mon[93869]: pgmap v10660: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 75 KiB/s wr, 9 op/s 2023-03-09T20:15:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:17 smithi181 ceph-mon[94189]: pgmap v10660: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 75 KiB/s wr, 9 op/s 2023-03-09T20:15:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:17 smithi038 ceph-mon[92422]: pgmap v10660: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 75 KiB/s wr, 9 op/s 2023-03-09T20:15:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19490.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:15:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19028.557472 secs 2023-03-09T20:15:19.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19028.557472 secs 2023-03-09T20:15:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19028.557472 secs 2023-03-09T20:15:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:19 smithi122 ceph-mon[93869]: pgmap v10661: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 74 KiB/s wr, 8 op/s 2023-03-09T20:15:20.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:19 smithi181 ceph-mon[94189]: pgmap v10661: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 74 KiB/s wr, 8 op/s 2023-03-09T20:15:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:19 smithi038 ceph-mon[92422]: pgmap v10661: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 74 KiB/s wr, 8 op/s 2023-03-09T20:15:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19031.236949 secs 2023-03-09T20:15:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19031.236949 secs 2023-03-09T20:15:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19031.236949 secs 2023-03-09T20:15:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:21 smithi122 ceph-mon[93869]: pgmap v10662: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 511 B/s wr, 7 op/s 2023-03-09T20:15:22.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:21 smithi181 ceph-mon[94189]: pgmap v10662: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 511 B/s wr, 7 op/s 2023-03-09T20:15:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:21 smithi038 ceph-mon[92422]: pgmap v10662: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 511 B/s wr, 7 op/s 2023-03-09T20:15:23.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19495.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:15:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:23 smithi122 ceph-mon[93869]: pgmap v10663: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T20:15:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19033.557585 secs 2023-03-09T20:15:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:23 smithi181 ceph-mon[94189]: pgmap v10663: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T20:15:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19033.557585 secs 2023-03-09T20:15:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:23 smithi038 ceph-mon[92422]: pgmap v10663: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T20:15:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19033.557585 secs 2023-03-09T20:15:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:25 smithi122 ceph-mon[93869]: pgmap v10664: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 0 B/s wr, 8 op/s 2023-03-09T20:15:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19036.236982 secs 2023-03-09T20:15:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:25 smithi181 ceph-mon[94189]: pgmap v10664: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 0 B/s wr, 8 op/s 2023-03-09T20:15:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19036.236982 secs 2023-03-09T20:15:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:25 smithi038 ceph-mon[92422]: pgmap v10664: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 0 B/s wr, 8 op/s 2023-03-09T20:15:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19036.236982 secs 2023-03-09T20:15:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:27 smithi122 ceph-mon[93869]: pgmap v10665: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T20:15:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:27 smithi181 ceph-mon[94189]: pgmap v10665: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T20:15:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:27 smithi038 ceph-mon[92422]: pgmap v10665: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T20:15:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19500.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:15:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19038.557678 secs 2023-03-09T20:15:29.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19038.557678 secs 2023-03-09T20:15:29.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19038.557678 secs 2023-03-09T20:15:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:29 smithi122 ceph-mon[93869]: pgmap v10666: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T20:15:30.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:29 smithi181 ceph-mon[94189]: pgmap v10666: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T20:15:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:29 smithi038 ceph-mon[92422]: pgmap v10666: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T20:15:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19041.237042 secs 2023-03-09T20:15:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19041.237042 secs 2023-03-09T20:15:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19041.237042 secs 2023-03-09T20:15:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:31 smithi122 ceph-mon[93869]: pgmap v10667: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T20:15:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:31 smithi181 ceph-mon[94189]: pgmap v10667: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T20:15:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:31 smithi038 ceph-mon[92422]: pgmap v10667: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T20:15:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19505.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:15:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:33 smithi122 ceph-mon[93869]: pgmap v10668: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 426 B/s wr, 8 op/s 2023-03-09T20:15:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19043.557797 secs 2023-03-09T20:15:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:33 smithi181 ceph-mon[94189]: pgmap v10668: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 426 B/s wr, 8 op/s 2023-03-09T20:15:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19043.557797 secs 2023-03-09T20:15:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:33 smithi038 ceph-mon[92422]: pgmap v10668: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 426 B/s wr, 8 op/s 2023-03-09T20:15:34.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19043.557797 secs 2023-03-09T20:15:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:35 smithi122 ceph-mon[93869]: pgmap v10669: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 71 KiB/s wr, 7 op/s 2023-03-09T20:15:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19046.237099 secs 2023-03-09T20:15:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:35 smithi181 ceph-mon[94189]: pgmap v10669: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 71 KiB/s wr, 7 op/s 2023-03-09T20:15:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19046.237099 secs 2023-03-09T20:15:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:35 smithi038 ceph-mon[92422]: pgmap v10669: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 71 KiB/s wr, 7 op/s 2023-03-09T20:15:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19046.237099 secs 2023-03-09T20:15:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:37 smithi122 ceph-mon[93869]: pgmap v10670: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 71 KiB/s wr, 8 op/s 2023-03-09T20:15:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:37 smithi181 ceph-mon[94189]: pgmap v10670: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 71 KiB/s wr, 8 op/s 2023-03-09T20:15:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:37 smithi038 ceph-mon[92422]: pgmap v10670: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 71 KiB/s wr, 8 op/s 2023-03-09T20:15:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19510.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:15:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:39 smithi122 ceph-mon[93869]: pgmap v10671: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 71 KiB/s wr, 7 op/s 2023-03-09T20:15:40.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:39 smithi181 ceph-mon[94189]: pgmap v10671: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 71 KiB/s wr, 7 op/s 2023-03-09T20:15:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:39 smithi038 ceph-mon[92422]: pgmap v10671: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 71 KiB/s wr, 7 op/s 2023-03-09T20:15:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19051.237160 secs 2023-03-09T20:15:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:40 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:15:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19051.237160 secs 2023-03-09T20:15:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:40 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:15:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19051.237160 secs 2023-03-09T20:15:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:40 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:15:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:41 smithi122 ceph-mon[93869]: pgmap v10672: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 147 KiB/s wr, 7 op/s 2023-03-09T20:15:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:41 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:15:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:15:42.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:15:42.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:41 smithi181 ceph-mon[94189]: pgmap v10672: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 147 KiB/s wr, 7 op/s 2023-03-09T20:15:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:41 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:15:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:15:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:15:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:41 smithi038 ceph-mon[92422]: pgmap v10672: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 147 KiB/s wr, 7 op/s 2023-03-09T20:15:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:41 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:15:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:15:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:15:43.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:42 smithi122 ceph-mon[93869]: pgmap v10673: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 147 KiB/s wr, 9 op/s 2023-03-09T20:15:43.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:42 smithi181 ceph-mon[94189]: pgmap v10673: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 147 KiB/s wr, 9 op/s 2023-03-09T20:15:43.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:42 smithi038 ceph-mon[92422]: pgmap v10673: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 147 KiB/s wr, 9 op/s 2023-03-09T20:15:43.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19515.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:15:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:45 smithi122 ceph-mon[93869]: pgmap v10674: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 242 KiB/s wr, 7 op/s 2023-03-09T20:15:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19056.237218 secs 2023-03-09T20:15:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:45 smithi181 ceph-mon[94189]: pgmap v10674: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 242 KiB/s wr, 7 op/s 2023-03-09T20:15:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19056.237218 secs 2023-03-09T20:15:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:45 smithi038 ceph-mon[92422]: pgmap v10674: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 242 KiB/s wr, 7 op/s 2023-03-09T20:15:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19056.237218 secs 2023-03-09T20:15:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:47 smithi122 ceph-mon[93869]: pgmap v10675: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 172 KiB/s wr, 8 op/s 2023-03-09T20:15:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:47 smithi181 ceph-mon[94189]: pgmap v10675: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 172 KiB/s wr, 8 op/s 2023-03-09T20:15:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:47 smithi038 ceph-mon[92422]: pgmap v10675: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 172 KiB/s wr, 8 op/s 2023-03-09T20:15:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19520.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:15:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:49 smithi122 ceph-mon[93869]: pgmap v10676: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 291 KiB/s wr, 8 op/s 2023-03-09T20:15:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:49 smithi181 ceph-mon[94189]: pgmap v10676: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 291 KiB/s wr, 8 op/s 2023-03-09T20:15:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:49 smithi038 ceph-mon[92422]: pgmap v10676: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 291 KiB/s wr, 8 op/s 2023-03-09T20:15:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19061.237278 secs 2023-03-09T20:15:51.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19061.237278 secs 2023-03-09T20:15:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19061.237278 secs 2023-03-09T20:15:52.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:51 smithi122 ceph-mon[93869]: pgmap v10677: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 291 KiB/s wr, 7 op/s 2023-03-09T20:15:52.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:15:52.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:15:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:51 smithi181 ceph-mon[94189]: pgmap v10677: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 291 KiB/s wr, 7 op/s 2023-03-09T20:15:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:15:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:15:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:51 smithi038 ceph-mon[92422]: pgmap v10677: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 291 KiB/s wr, 7 op/s 2023-03-09T20:15:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:15:52.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:15:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:15:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:15:53.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:15:53.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:15:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:15:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:15:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19525.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:15:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:53 smithi122 ceph-mon[93869]: pgmap v10678: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 561 KiB/s rd, 356 KiB/s wr, 11 op/s 2023-03-09T20:15:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:53 smithi181 ceph-mon[94189]: pgmap v10678: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 561 KiB/s rd, 356 KiB/s wr, 11 op/s 2023-03-09T20:15:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:53 smithi038 ceph-mon[92422]: pgmap v10678: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 561 KiB/s rd, 356 KiB/s wr, 11 op/s 2023-03-09T20:15:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:55 smithi122 ceph-mon[93869]: pgmap v10679: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 560 KiB/s rd, 356 KiB/s wr, 9 op/s 2023-03-09T20:15:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19066.237337 secs 2023-03-09T20:15:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:55 smithi181 ceph-mon[94189]: pgmap v10679: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 560 KiB/s rd, 356 KiB/s wr, 9 op/s 2023-03-09T20:15:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19066.237337 secs 2023-03-09T20:15:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:55 smithi038 ceph-mon[92422]: pgmap v10679: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 560 KiB/s rd, 356 KiB/s wr, 9 op/s 2023-03-09T20:15:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19066.237337 secs 2023-03-09T20:15:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:57 smithi122 ceph-mon[93869]: pgmap v10680: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 262 KiB/s wr, 11 op/s 2023-03-09T20:15:58.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:57 smithi181 ceph-mon[94189]: pgmap v10680: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 262 KiB/s wr, 11 op/s 2023-03-09T20:15:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:57 smithi038 ceph-mon[92422]: pgmap v10680: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 262 KiB/s wr, 11 op/s 2023-03-09T20:15:58.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19530.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:16:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:59 smithi122 ceph-mon[93869]: pgmap v10681: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 465 KiB/s wr, 12 op/s 2023-03-09T20:16:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:16:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:15:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:16:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:59 smithi181 ceph-mon[94189]: pgmap v10681: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 465 KiB/s wr, 12 op/s 2023-03-09T20:16:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:16:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:15:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:16:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:59 smithi038 ceph-mon[92422]: pgmap v10681: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 465 KiB/s wr, 12 op/s 2023-03-09T20:16:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:16:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:15:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:16:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19071.237422 secs 2023-03-09T20:16:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:16:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:16:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19071.237422 secs 2023-03-09T20:16:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:16:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:16:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19071.237422 secs 2023-03-09T20:16:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:16:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:16:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:01 smithi122 ceph-mon[93869]: pgmap v10682: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 345 KiB/s wr, 10 op/s 2023-03-09T20:16:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:01 smithi181 ceph-mon[94189]: pgmap v10682: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 345 KiB/s wr, 10 op/s 2023-03-09T20:16:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:01 smithi038 ceph-mon[92422]: pgmap v10682: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 345 KiB/s wr, 10 op/s 2023-03-09T20:16:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19535.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:16:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:03 smithi122 ceph-mon[93869]: pgmap v10683: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 385 KiB/s wr, 15 op/s 2023-03-09T20:16:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19073.558477 secs 2023-03-09T20:16:04.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:03 smithi181 ceph-mon[94189]: pgmap v10683: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 385 KiB/s wr, 15 op/s 2023-03-09T20:16:04.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19073.558477 secs 2023-03-09T20:16:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:03 smithi038 ceph-mon[92422]: pgmap v10683: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 385 KiB/s wr, 15 op/s 2023-03-09T20:16:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19073.558477 secs 2023-03-09T20:16:05.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:04 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:16:05.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:04 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:16:05.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:04 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:16:05.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:04 smithi122 ceph-mon[93869]: osdmap e881: 12 total, 12 up, 12 in 2023-03-09T20:16:05.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:04 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:16:05.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:04 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:16:05.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:04 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:16:05.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:04 smithi181 ceph-mon[94189]: osdmap e881: 12 total, 12 up, 12 in 2023-03-09T20:16:05.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:04 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:16:05.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:04 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:16:05.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:04 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:16:05.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:04 smithi038 ceph-mon[92422]: osdmap e881: 12 total, 12 up, 12 in 2023-03-09T20:16:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:05 smithi122 ceph-mon[93869]: pgmap v10684: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 397 KiB/s wr, 24 op/s 2023-03-09T20:16:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19076.237481 secs 2023-03-09T20:16:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:05 smithi181 ceph-mon[94189]: pgmap v10684: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 397 KiB/s wr, 24 op/s 2023-03-09T20:16:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19076.237481 secs 2023-03-09T20:16:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:05 smithi038 ceph-mon[92422]: pgmap v10684: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 397 KiB/s wr, 24 op/s 2023-03-09T20:16:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19076.237481 secs 2023-03-09T20:16:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:07 smithi122 ceph-mon[93869]: pgmap v10686: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 337 KiB/s rd, 490 KiB/s wr, 29 op/s 2023-03-09T20:16:08.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:07 smithi181 ceph-mon[94189]: pgmap v10686: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 337 KiB/s rd, 490 KiB/s wr, 29 op/s 2023-03-09T20:16:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:07 smithi038 ceph-mon[92422]: pgmap v10686: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 337 KiB/s rd, 490 KiB/s wr, 29 op/s 2023-03-09T20:16:08.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19540.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:16:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19078.558607 secs 2023-03-09T20:16:09.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19078.558607 secs 2023-03-09T20:16:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19078.558607 secs 2023-03-09T20:16:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:09 smithi122 ceph-mon[93869]: pgmap v10687: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 326 KiB/s wr, 28 op/s 2023-03-09T20:16:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:09 smithi181 ceph-mon[94189]: pgmap v10687: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 326 KiB/s wr, 28 op/s 2023-03-09T20:16:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:09 smithi038 ceph-mon[92422]: pgmap v10687: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 326 KiB/s wr, 28 op/s 2023-03-09T20:16:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19081.237566 secs 2023-03-09T20:16:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19081.237566 secs 2023-03-09T20:16:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19081.237566 secs 2023-03-09T20:16:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:11 smithi122 ceph-mon[93869]: pgmap v10688: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 326 KiB/s wr, 28 op/s 2023-03-09T20:16:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:11 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:16:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:11 smithi181 ceph-mon[94189]: pgmap v10688: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 326 KiB/s wr, 28 op/s 2023-03-09T20:16:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:11 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:16:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:11 smithi038 ceph-mon[92422]: pgmap v10688: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 326 KiB/s wr, 28 op/s 2023-03-09T20:16:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:11 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:16:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:12 smithi122 ceph-mon[93869]: pgmap v10689: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 279 KiB/s wr, 25 op/s 2023-03-09T20:16:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:12 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:16:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:12 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:16:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:12 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:16:13.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:12 smithi181 ceph-mon[94189]: pgmap v10689: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 279 KiB/s wr, 25 op/s 2023-03-09T20:16:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:12 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:16:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:12 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:16:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:12 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:16:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:12 smithi038 ceph-mon[92422]: pgmap v10689: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 279 KiB/s wr, 25 op/s 2023-03-09T20:16:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:12 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:16:13.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:12 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:16:13.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:12 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:16:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19545.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:16:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19083.558685 secs 2023-03-09T20:16:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:13 smithi122 ceph-mon[93869]: osdmap e882: 12 total, 12 up, 12 in 2023-03-09T20:16:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19083.558685 secs 2023-03-09T20:16:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:13 smithi181 ceph-mon[94189]: osdmap e882: 12 total, 12 up, 12 in 2023-03-09T20:16:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19083.558685 secs 2023-03-09T20:16:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:13 smithi038 ceph-mon[92422]: osdmap e882: 12 total, 12 up, 12 in 2023-03-09T20:16:15.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:14 smithi122 ceph-mon[93869]: pgmap v10691: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 165 KiB/s wr, 10 op/s 2023-03-09T20:16:15.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:14 smithi181 ceph-mon[94189]: pgmap v10691: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 165 KiB/s wr, 10 op/s 2023-03-09T20:16:15.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:14 smithi038 ceph-mon[92422]: pgmap v10691: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 165 KiB/s wr, 10 op/s 2023-03-09T20:16:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19086.237625 secs 2023-03-09T20:16:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19086.237625 secs 2023-03-09T20:16:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19086.237625 secs 2023-03-09T20:16:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:16 smithi122 ceph-mon[93869]: pgmap v10692: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 139 KiB/s wr, 7 op/s 2023-03-09T20:16:17.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:16 smithi181 ceph-mon[94189]: pgmap v10692: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 139 KiB/s wr, 7 op/s 2023-03-09T20:16:17.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:16 smithi038 ceph-mon[92422]: pgmap v10692: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 139 KiB/s wr, 7 op/s 2023-03-09T20:16:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19088.558817 secs 2023-03-09T20:16:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19088.558817 secs 2023-03-09T20:16:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19088.558817 secs 2023-03-09T20:16:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19550.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:16:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:19 smithi122 ceph-mon[93869]: pgmap v10693: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 6 op/s 2023-03-09T20:16:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:19 smithi181 ceph-mon[94189]: pgmap v10693: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 6 op/s 2023-03-09T20:16:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:19 smithi038 ceph-mon[92422]: pgmap v10693: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 6 op/s 2023-03-09T20:16:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19091.237712 secs 2023-03-09T20:16:21.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19091.237712 secs 2023-03-09T20:16:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19091.237712 secs 2023-03-09T20:16:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:21 smithi122 ceph-mon[93869]: pgmap v10694: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 6 op/s 2023-03-09T20:16:22.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:21 smithi181 ceph-mon[94189]: pgmap v10694: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 6 op/s 2023-03-09T20:16:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:21 smithi038 ceph-mon[92422]: pgmap v10694: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 6 op/s 2023-03-09T20:16:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19555.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:16:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:23 smithi122 ceph-mon[93869]: pgmap v10695: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 58 KiB/s wr, 6 op/s 2023-03-09T20:16:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19093.558932 secs 2023-03-09T20:16:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:23 smithi181 ceph-mon[94189]: pgmap v10695: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 58 KiB/s wr, 6 op/s 2023-03-09T20:16:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19093.558932 secs 2023-03-09T20:16:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:23 smithi038 ceph-mon[92422]: pgmap v10695: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 58 KiB/s wr, 6 op/s 2023-03-09T20:16:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19093.558932 secs 2023-03-09T20:16:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:25 smithi122 ceph-mon[93869]: pgmap v10696: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2023-03-09T20:16:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19096.237772 secs 2023-03-09T20:16:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:25 smithi181 ceph-mon[94189]: pgmap v10696: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2023-03-09T20:16:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19096.237772 secs 2023-03-09T20:16:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:25 smithi038 ceph-mon[92422]: pgmap v10696: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2023-03-09T20:16:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19096.237772 secs 2023-03-09T20:16:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:27 smithi122 ceph-mon[93869]: pgmap v10697: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-03-09T20:16:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:27 smithi181 ceph-mon[94189]: pgmap v10697: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-03-09T20:16:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:27 smithi038 ceph-mon[92422]: pgmap v10697: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-03-09T20:16:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19560.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:16:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19098.559051 secs 2023-03-09T20:16:29.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19098.559051 secs 2023-03-09T20:16:29.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19098.559051 secs 2023-03-09T20:16:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:29 smithi122 ceph-mon[93869]: pgmap v10698: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 5 op/s 2023-03-09T20:16:30.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:29 smithi181 ceph-mon[94189]: pgmap v10698: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 5 op/s 2023-03-09T20:16:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:29 smithi038 ceph-mon[92422]: pgmap v10698: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 5 op/s 2023-03-09T20:16:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19101.237850 secs 2023-03-09T20:16:31.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19101.237850 secs 2023-03-09T20:16:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19101.237850 secs 2023-03-09T20:16:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:31 smithi122 ceph-mon[93869]: pgmap v10699: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 5 op/s 2023-03-09T20:16:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:31 smithi181 ceph-mon[94189]: pgmap v10699: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 5 op/s 2023-03-09T20:16:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:31 smithi038 ceph-mon[92422]: pgmap v10699: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 22 KiB/s wr, 5 op/s 2023-03-09T20:16:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19565.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:16:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:33 smithi122 ceph-mon[93869]: pgmap v10700: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 22 KiB/s wr, 7 op/s 2023-03-09T20:16:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19103.559174 secs 2023-03-09T20:16:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:33 smithi181 ceph-mon[94189]: pgmap v10700: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 22 KiB/s wr, 7 op/s 2023-03-09T20:16:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19103.559174 secs 2023-03-09T20:16:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:33 smithi038 ceph-mon[92422]: pgmap v10700: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 22 KiB/s wr, 7 op/s 2023-03-09T20:16:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19103.559174 secs 2023-03-09T20:16:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:35 smithi122 ceph-mon[93869]: pgmap v10701: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 128 KiB/s wr, 7 op/s 2023-03-09T20:16:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19106.237925 secs 2023-03-09T20:16:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:35 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:16:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:35 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:16:36.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:35 smithi181 ceph-mon[94189]: pgmap v10701: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 128 KiB/s wr, 7 op/s 2023-03-09T20:16:36.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19106.237925 secs 2023-03-09T20:16:36.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:35 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:16:36.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:35 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:16:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:35 smithi038 ceph-mon[92422]: pgmap v10701: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 128 KiB/s wr, 7 op/s 2023-03-09T20:16:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19106.237925 secs 2023-03-09T20:16:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:35 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:16:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:35 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:16:37.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:16:37.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:16:37.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:16:37.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:16:37.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:16:37.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:16:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:37 smithi122 ceph-mon[93869]: pgmap v10702: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 128 KiB/s wr, 8 op/s 2023-03-09T20:16:38.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:37 smithi181 ceph-mon[94189]: pgmap v10702: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 128 KiB/s wr, 8 op/s 2023-03-09T20:16:38.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:37 smithi038 ceph-mon[92422]: pgmap v10702: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 128 KiB/s wr, 8 op/s 2023-03-09T20:16:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19570.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:16:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:38 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 19108.559277 secs 2023-03-09T20:16:39.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:38 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 19108.559277 secs 2023-03-09T20:16:39.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:38 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 19108.559277 secs 2023-03-09T20:16:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:39 smithi122 ceph-mon[93869]: pgmap v10703: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 129 KiB/s wr, 8 op/s 2023-03-09T20:16:40.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:39 smithi181 ceph-mon[94189]: pgmap v10703: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 129 KiB/s wr, 8 op/s 2023-03-09T20:16:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:39 smithi038 ceph-mon[92422]: pgmap v10703: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 129 KiB/s wr, 8 op/s 2023-03-09T20:16:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19111.237988 secs 2023-03-09T20:16:41.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19111.237988 secs 2023-03-09T20:16:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19111.237988 secs 2023-03-09T20:16:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:41 smithi122 ceph-mon[93869]: pgmap v10704: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 107 KiB/s wr, 7 op/s 2023-03-09T20:16:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:16:42.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:16:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:41 smithi181 ceph-mon[94189]: pgmap v10704: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 107 KiB/s wr, 7 op/s 2023-03-09T20:16:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:16:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:16:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:41 smithi038 ceph-mon[92422]: pgmap v10704: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 107 KiB/s wr, 7 op/s 2023-03-09T20:16:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:16:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:16:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19575.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:16:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:43 smithi122 ceph-mon[93869]: pgmap v10705: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 108 KiB/s wr, 10 op/s 2023-03-09T20:16:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:43 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 19113.559343 secs 2023-03-09T20:16:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:43 smithi181 ceph-mon[94189]: pgmap v10705: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 108 KiB/s wr, 10 op/s 2023-03-09T20:16:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:43 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 19113.559343 secs 2023-03-09T20:16:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:43 smithi038 ceph-mon[92422]: pgmap v10705: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 108 KiB/s wr, 10 op/s 2023-03-09T20:16:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:43 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 19113.559343 secs 2023-03-09T20:16:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:45 smithi122 ceph-mon[93869]: pgmap v10706: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 193 KiB/s wr, 8 op/s 2023-03-09T20:16:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19116.238049 secs 2023-03-09T20:16:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:45 smithi181 ceph-mon[94189]: pgmap v10706: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 193 KiB/s wr, 8 op/s 2023-03-09T20:16:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19116.238049 secs 2023-03-09T20:16:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:45 smithi038 ceph-mon[92422]: pgmap v10706: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 193 KiB/s wr, 8 op/s 2023-03-09T20:16:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19116.238049 secs 2023-03-09T20:16:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:47 smithi122 ceph-mon[93869]: pgmap v10707: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 137 KiB/s wr, 9 op/s 2023-03-09T20:16:48.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:47 smithi181 ceph-mon[94189]: pgmap v10707: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 137 KiB/s wr, 9 op/s 2023-03-09T20:16:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:47 smithi038 ceph-mon[92422]: pgmap v10707: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 137 KiB/s wr, 9 op/s 2023-03-09T20:16:48.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19580.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:16:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:48 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 19118.559460 secs 2023-03-09T20:16:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:48 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 19118.559460 secs 2023-03-09T20:16:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:48 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 19118.559460 secs 2023-03-09T20:16:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:49 smithi122 ceph-mon[93869]: pgmap v10708: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 302 KiB/s wr, 9 op/s 2023-03-09T20:16:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:49 smithi181 ceph-mon[94189]: pgmap v10708: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 302 KiB/s wr, 9 op/s 2023-03-09T20:16:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:49 smithi038 ceph-mon[92422]: pgmap v10708: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 302 KiB/s wr, 9 op/s 2023-03-09T20:16:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19121.238113 secs 2023-03-09T20:16:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19121.238113 secs 2023-03-09T20:16:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19121.238113 secs 2023-03-09T20:16:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:51 smithi122 ceph-mon[93869]: pgmap v10709: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 302 KiB/s wr, 7 op/s 2023-03-09T20:16:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:16:52.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:16:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:51 smithi181 ceph-mon[94189]: pgmap v10709: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 302 KiB/s wr, 7 op/s 2023-03-09T20:16:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:16:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:16:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:51 smithi038 ceph-mon[92422]: pgmap v10709: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 302 KiB/s wr, 7 op/s 2023-03-09T20:16:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:16:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:16:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:16:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:16:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:52 smithi122 ceph-mon[93869]: pgmap v10710: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 443 KiB/s wr, 12 op/s 2023-03-09T20:16:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:16:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:16:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:52 smithi181 ceph-mon[94189]: pgmap v10710: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 443 KiB/s wr, 12 op/s 2023-03-09T20:16:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:16:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:16:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:52 smithi038 ceph-mon[92422]: pgmap v10710: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 443 KiB/s wr, 12 op/s 2023-03-09T20:16:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19585.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:16:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19123.559591 secs 2023-03-09T20:16:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:16:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:16:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19123.559591 secs 2023-03-09T20:16:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:16:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:16:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19123.559591 secs 2023-03-09T20:16:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:16:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:16:55.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:54 smithi122 ceph-mon[93869]: pgmap v10711: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 457 KiB/s rd, 443 KiB/s wr, 10 op/s 2023-03-09T20:16:55.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:54 smithi181 ceph-mon[94189]: pgmap v10711: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 457 KiB/s rd, 443 KiB/s wr, 10 op/s 2023-03-09T20:16:55.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:54 smithi038 ceph-mon[92422]: pgmap v10711: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 457 KiB/s rd, 443 KiB/s wr, 10 op/s 2023-03-09T20:16:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19126.238173 secs 2023-03-09T20:16:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19126.238173 secs 2023-03-09T20:16:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19126.238173 secs 2023-03-09T20:16:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:56 smithi122 ceph-mon[93869]: pgmap v10712: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 540 KiB/s rd, 525 KiB/s wr, 12 op/s 2023-03-09T20:16:57.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:56 smithi181 ceph-mon[94189]: pgmap v10712: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 540 KiB/s rd, 525 KiB/s wr, 12 op/s 2023-03-09T20:16:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:56 smithi038 ceph-mon[92422]: pgmap v10712: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 540 KiB/s rd, 525 KiB/s wr, 12 op/s 2023-03-09T20:16:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19128.559728 secs 2023-03-09T20:16:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19128.559728 secs 2023-03-09T20:16:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19128.559728 secs 2023-03-09T20:16:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19590.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:17:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:59 smithi122 ceph-mon[93869]: pgmap v10713: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 503 KiB/s rd, 490 KiB/s wr, 12 op/s 2023-03-09T20:17:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:17:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:16:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:17:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:59 smithi181 ceph-mon[94189]: pgmap v10713: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 503 KiB/s rd, 490 KiB/s wr, 12 op/s 2023-03-09T20:17:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:17:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:16:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:17:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:59 smithi038 ceph-mon[92422]: pgmap v10713: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 503 KiB/s rd, 490 KiB/s wr, 12 op/s 2023-03-09T20:17:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:17:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:16:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:17:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19131.238236 secs 2023-03-09T20:17:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:17:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:17:01.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19131.238236 secs 2023-03-09T20:17:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:17:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:17:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19131.238236 secs 2023-03-09T20:17:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:17:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:17:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:01 smithi122 ceph-mon[93869]: pgmap v10714: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 347 KiB/s wr, 11 op/s 2023-03-09T20:17:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:01 smithi181 ceph-mon[94189]: pgmap v10714: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 347 KiB/s wr, 11 op/s 2023-03-09T20:17:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:01 smithi038 ceph-mon[92422]: pgmap v10714: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 347 KiB/s wr, 11 op/s 2023-03-09T20:17:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19595.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:17:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:03 smithi122 ceph-mon[93869]: pgmap v10715: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 375 KiB/s wr, 14 op/s 2023-03-09T20:17:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19133.559858 secs 2023-03-09T20:17:04.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:03 smithi181 ceph-mon[94189]: pgmap v10715: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 375 KiB/s wr, 14 op/s 2023-03-09T20:17:04.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19133.559858 secs 2023-03-09T20:17:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:03 smithi038 ceph-mon[92422]: pgmap v10715: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 375 KiB/s wr, 14 op/s 2023-03-09T20:17:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19133.559858 secs 2023-03-09T20:17:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:05 smithi122 ceph-mon[93869]: pgmap v10716: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 388 KiB/s wr, 24 op/s 2023-03-09T20:17:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19136.238293 secs 2023-03-09T20:17:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:05 smithi122 ceph-mon[93869]: osdmap e883: 12 total, 12 up, 12 in 2023-03-09T20:17:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:05 smithi181 ceph-mon[94189]: pgmap v10716: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 388 KiB/s wr, 24 op/s 2023-03-09T20:17:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19136.238293 secs 2023-03-09T20:17:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:05 smithi181 ceph-mon[94189]: osdmap e883: 12 total, 12 up, 12 in 2023-03-09T20:17:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:05 smithi038 ceph-mon[92422]: pgmap v10716: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 388 KiB/s wr, 24 op/s 2023-03-09T20:17:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19136.238293 secs 2023-03-09T20:17:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:05 smithi038 ceph-mon[92422]: osdmap e883: 12 total, 12 up, 12 in 2023-03-09T20:17:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:07 smithi122 ceph-mon[93869]: pgmap v10718: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 291 KiB/s wr, 28 op/s 2023-03-09T20:17:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:07 smithi181 ceph-mon[94189]: pgmap v10718: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 291 KiB/s wr, 28 op/s 2023-03-09T20:17:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:07 smithi038 ceph-mon[92422]: pgmap v10718: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 291 KiB/s wr, 28 op/s 2023-03-09T20:17:08.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19600.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:17:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19138.559990 secs 2023-03-09T20:17:09.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19138.559990 secs 2023-03-09T20:17:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19138.559990 secs 2023-03-09T20:17:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:09 smithi122 ceph-mon[93869]: pgmap v10719: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 272 KiB/s wr, 27 op/s 2023-03-09T20:17:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:09 smithi181 ceph-mon[94189]: pgmap v10719: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 272 KiB/s wr, 27 op/s 2023-03-09T20:17:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:09 smithi038 ceph-mon[92422]: pgmap v10719: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 272 KiB/s wr, 27 op/s 2023-03-09T20:17:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19141.238357 secs 2023-03-09T20:17:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19141.238357 secs 2023-03-09T20:17:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19141.238357 secs 2023-03-09T20:17:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:11 smithi122 ceph-mon[93869]: pgmap v10720: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 327 KiB/s wr, 27 op/s 2023-03-09T20:17:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:11 smithi181 ceph-mon[94189]: pgmap v10720: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 327 KiB/s wr, 27 op/s 2023-03-09T20:17:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:11 smithi038 ceph-mon[92422]: pgmap v10720: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 327 KiB/s wr, 27 op/s 2023-03-09T20:17:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:12 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:17:13.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:12 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:17:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:12 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:17:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19605.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:17:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:13 smithi122 ceph-mon[93869]: pgmap v10721: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 293 KiB/s wr, 26 op/s 2023-03-09T20:17:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19143.560141 secs 2023-03-09T20:17:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:13 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:17:14.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:13 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:17:14.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:13 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:17:14.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:13 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:17:14.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:13 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:17:14.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:13 smithi122 ceph-mon[93869]: osdmap e884: 12 total, 12 up, 12 in 2023-03-09T20:17:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:13 smithi181 ceph-mon[94189]: pgmap v10721: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 293 KiB/s wr, 26 op/s 2023-03-09T20:17:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19143.560141 secs 2023-03-09T20:17:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:13 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:17:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:13 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:17:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:13 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:17:14.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:13 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:17:14.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:13 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:17:14.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:13 smithi181 ceph-mon[94189]: osdmap e884: 12 total, 12 up, 12 in 2023-03-09T20:17:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:13 smithi038 ceph-mon[92422]: pgmap v10721: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 293 KiB/s wr, 26 op/s 2023-03-09T20:17:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19143.560141 secs 2023-03-09T20:17:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:13 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:17:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:13 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:17:14.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:13 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:17:14.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:13 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:17:14.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:13 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:17:14.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:13 smithi038 ceph-mon[92422]: osdmap e884: 12 total, 12 up, 12 in 2023-03-09T20:17:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:15 smithi122 ceph-mon[93869]: pgmap v10723: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 179 KiB/s wr, 11 op/s 2023-03-09T20:17:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19146.238434 secs 2023-03-09T20:17:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:15 smithi181 ceph-mon[94189]: pgmap v10723: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 179 KiB/s wr, 11 op/s 2023-03-09T20:17:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19146.238434 secs 2023-03-09T20:17:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:15 smithi038 ceph-mon[92422]: pgmap v10723: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 179 KiB/s wr, 11 op/s 2023-03-09T20:17:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19146.238434 secs 2023-03-09T20:17:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:17 smithi122 ceph-mon[93869]: pgmap v10724: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 139 KiB/s wr, 8 op/s 2023-03-09T20:17:18.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:17 smithi181 ceph-mon[94189]: pgmap v10724: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 139 KiB/s wr, 8 op/s 2023-03-09T20:17:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:17 smithi038 ceph-mon[92422]: pgmap v10724: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 139 KiB/s wr, 8 op/s 2023-03-09T20:17:18.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19610.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:17:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19148.560259 secs 2023-03-09T20:17:19.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19148.560259 secs 2023-03-09T20:17:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19148.560259 secs 2023-03-09T20:17:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:19 smithi122 ceph-mon[93869]: pgmap v10725: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 139 KiB/s wr, 7 op/s 2023-03-09T20:17:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:19 smithi181 ceph-mon[94189]: pgmap v10725: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 139 KiB/s wr, 7 op/s 2023-03-09T20:17:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:19 smithi038 ceph-mon[92422]: pgmap v10725: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 139 KiB/s wr, 7 op/s 2023-03-09T20:17:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19151.238495 secs 2023-03-09T20:17:21.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19151.238495 secs 2023-03-09T20:17:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19151.238495 secs 2023-03-09T20:17:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:21 smithi122 ceph-mon[93869]: pgmap v10726: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 7 op/s 2023-03-09T20:17:22.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:21 smithi181 ceph-mon[94189]: pgmap v10726: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 7 op/s 2023-03-09T20:17:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:21 smithi038 ceph-mon[92422]: pgmap v10726: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 7 op/s 2023-03-09T20:17:23.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19615.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:17:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:23 smithi122 ceph-mon[93869]: pgmap v10727: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 6 op/s 2023-03-09T20:17:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19153.560377 secs 2023-03-09T20:17:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:23 smithi181 ceph-mon[94189]: pgmap v10727: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 6 op/s 2023-03-09T20:17:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19153.560377 secs 2023-03-09T20:17:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:23 smithi038 ceph-mon[92422]: pgmap v10727: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 6 op/s 2023-03-09T20:17:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19153.560377 secs 2023-03-09T20:17:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:25 smithi122 ceph-mon[93869]: pgmap v10728: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 56 KiB/s wr, 6 op/s 2023-03-09T20:17:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19156.238555 secs 2023-03-09T20:17:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:25 smithi181 ceph-mon[94189]: pgmap v10728: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 56 KiB/s wr, 6 op/s 2023-03-09T20:17:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19156.238555 secs 2023-03-09T20:17:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:25 smithi038 ceph-mon[92422]: pgmap v10728: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 56 KiB/s wr, 6 op/s 2023-03-09T20:17:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19156.238555 secs 2023-03-09T20:17:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:27 smithi122 ceph-mon[93869]: pgmap v10729: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T20:17:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:27 smithi181 ceph-mon[94189]: pgmap v10729: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T20:17:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:27 smithi038 ceph-mon[92422]: pgmap v10729: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T20:17:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19620.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:17:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19158.560492 secs 2023-03-09T20:17:29.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19158.560492 secs 2023-03-09T20:17:29.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19158.560492 secs 2023-03-09T20:17:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:29 smithi122 ceph-mon[93869]: pgmap v10730: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T20:17:30.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:29 smithi181 ceph-mon[94189]: pgmap v10730: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T20:17:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:29 smithi038 ceph-mon[92422]: pgmap v10730: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T20:17:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19161.238615 secs 2023-03-09T20:17:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:30 smithi122 ceph-mon[93869]: pgmap v10731: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T20:17:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19161.238615 secs 2023-03-09T20:17:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:30 smithi181 ceph-mon[94189]: pgmap v10731: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T20:17:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19161.238615 secs 2023-03-09T20:17:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:30 smithi038 ceph-mon[92422]: pgmap v10731: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T20:17:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:33 smithi122 ceph-mon[93869]: pgmap v10732: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 426 B/s wr, 7 op/s 2023-03-09T20:17:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19163.560610 secs 2023-03-09T20:17:33.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:33 smithi181 ceph-mon[94189]: pgmap v10732: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 426 B/s wr, 7 op/s 2023-03-09T20:17:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19163.560610 secs 2023-03-09T20:17:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:33 smithi038 ceph-mon[92422]: pgmap v10732: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 426 B/s wr, 7 op/s 2023-03-09T20:17:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19163.560610 secs 2023-03-09T20:17:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19625.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:17:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:35 smithi122 ceph-mon[93869]: pgmap v10733: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 71 KiB/s wr, 6 op/s 2023-03-09T20:17:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19166.238675 secs 2023-03-09T20:17:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:35 smithi181 ceph-mon[94189]: pgmap v10733: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 71 KiB/s wr, 6 op/s 2023-03-09T20:17:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19166.238675 secs 2023-03-09T20:17:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:35 smithi038 ceph-mon[92422]: pgmap v10733: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 71 KiB/s wr, 6 op/s 2023-03-09T20:17:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19166.238675 secs 2023-03-09T20:17:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:37 smithi122 ceph-mon[93869]: pgmap v10734: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 71 KiB/s wr, 7 op/s 2023-03-09T20:17:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:37 smithi181 ceph-mon[94189]: pgmap v10734: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 71 KiB/s wr, 7 op/s 2023-03-09T20:17:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:37 smithi038 ceph-mon[92422]: pgmap v10734: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 71 KiB/s wr, 7 op/s 2023-03-09T20:17:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19630.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:17:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19168.560728 secs 2023-03-09T20:17:39.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19168.560728 secs 2023-03-09T20:17:39.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19168.560728 secs 2023-03-09T20:17:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:39 smithi122 ceph-mon[93869]: pgmap v10735: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 6 op/s 2023-03-09T20:17:40.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:39 smithi181 ceph-mon[94189]: pgmap v10735: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 6 op/s 2023-03-09T20:17:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:39 smithi038 ceph-mon[92422]: pgmap v10735: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 6 op/s 2023-03-09T20:17:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19171.238738 secs 2023-03-09T20:17:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:17:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:17:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19171.238738 secs 2023-03-09T20:17:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:17:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:17:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19171.238738 secs 2023-03-09T20:17:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:17:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:17:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:41 smithi122 ceph-mon[93869]: pgmap v10736: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 74 KiB/s wr, 7 op/s 2023-03-09T20:17:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:41 smithi181 ceph-mon[94189]: pgmap v10736: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 74 KiB/s wr, 7 op/s 2023-03-09T20:17:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:41 smithi038 ceph-mon[92422]: pgmap v10736: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 74 KiB/s wr, 7 op/s 2023-03-09T20:17:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19635.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:17:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19173.560844 secs 2023-03-09T20:17:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:43 smithi122 ceph-mon[93869]: pgmap v10737: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 148 KiB/s wr, 9 op/s 2023-03-09T20:17:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19173.560844 secs 2023-03-09T20:17:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:43 smithi181 ceph-mon[94189]: pgmap v10737: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 148 KiB/s wr, 9 op/s 2023-03-09T20:17:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19173.560844 secs 2023-03-09T20:17:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:43 smithi038 ceph-mon[92422]: pgmap v10737: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 148 KiB/s wr, 9 op/s 2023-03-09T20:17:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:45 smithi122 ceph-mon[93869]: pgmap v10738: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 148 KiB/s wr, 7 op/s 2023-03-09T20:17:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19176.238797 secs 2023-03-09T20:17:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:45 smithi181 ceph-mon[94189]: pgmap v10738: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 148 KiB/s wr, 7 op/s 2023-03-09T20:17:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19176.238797 secs 2023-03-09T20:17:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:45 smithi038 ceph-mon[92422]: pgmap v10738: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 148 KiB/s wr, 7 op/s 2023-03-09T20:17:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19176.238797 secs 2023-03-09T20:17:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:47 smithi122 ceph-mon[93869]: pgmap v10739: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 159 KiB/s wr, 9 op/s 2023-03-09T20:17:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:17:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:17:48.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:47 smithi181 ceph-mon[94189]: pgmap v10739: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 159 KiB/s wr, 9 op/s 2023-03-09T20:17:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:17:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:17:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:47 smithi038 ceph-mon[92422]: pgmap v10739: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 159 KiB/s wr, 9 op/s 2023-03-09T20:17:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:17:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:17:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19640.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:17:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19178.560965 secs 2023-03-09T20:17:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:17:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:17:49.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19178.560965 secs 2023-03-09T20:17:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:17:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:17:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19178.560965 secs 2023-03-09T20:17:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:17:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:17:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:49 smithi122 ceph-mon[93869]: pgmap v10740: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 456 KiB/s wr, 10 op/s 2023-03-09T20:17:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:49 smithi181 ceph-mon[94189]: pgmap v10740: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 456 KiB/s wr, 10 op/s 2023-03-09T20:17:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:49 smithi038 ceph-mon[92422]: pgmap v10740: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 456 KiB/s wr, 10 op/s 2023-03-09T20:17:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19181.238855 secs 2023-03-09T20:17:51.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19181.238855 secs 2023-03-09T20:17:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19181.238855 secs 2023-03-09T20:17:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:51 smithi122 ceph-mon[93869]: pgmap v10741: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 456 KiB/s wr, 10 op/s 2023-03-09T20:17:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:17:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:17:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:51 smithi181 ceph-mon[94189]: pgmap v10741: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 456 KiB/s wr, 10 op/s 2023-03-09T20:17:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:17:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:17:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:51 smithi038 ceph-mon[92422]: pgmap v10741: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 456 KiB/s wr, 10 op/s 2023-03-09T20:17:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:17:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:17:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19645.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:17:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19183.561082 secs 2023-03-09T20:17:54.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:53 smithi122 ceph-mon[93869]: pgmap v10742: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 602 KiB/s rd, 585 KiB/s wr, 14 op/s 2023-03-09T20:17:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19183.561082 secs 2023-03-09T20:17:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:53 smithi181 ceph-mon[94189]: pgmap v10742: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 602 KiB/s rd, 585 KiB/s wr, 14 op/s 2023-03-09T20:17:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19183.561082 secs 2023-03-09T20:17:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:53 smithi038 ceph-mon[92422]: pgmap v10742: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 602 KiB/s rd, 585 KiB/s wr, 14 op/s 2023-03-09T20:17:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:55 smithi122 ceph-mon[93869]: pgmap v10743: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 535 KiB/s wr, 12 op/s 2023-03-09T20:17:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19186.238914 secs 2023-03-09T20:17:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:55 smithi181 ceph-mon[94189]: pgmap v10743: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 535 KiB/s wr, 12 op/s 2023-03-09T20:17:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19186.238914 secs 2023-03-09T20:17:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:55 smithi038 ceph-mon[92422]: pgmap v10743: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 535 KiB/s wr, 12 op/s 2023-03-09T20:17:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19186.238914 secs 2023-03-09T20:17:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:17:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:17:57.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:17:57.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:17:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:17:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:17:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:57 smithi122 ceph-mon[93869]: pgmap v10744: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 715 KiB/s rd, 711 KiB/s wr, 15 op/s 2023-03-09T20:17:58.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:57 smithi181 ceph-mon[94189]: pgmap v10744: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 715 KiB/s rd, 711 KiB/s wr, 15 op/s 2023-03-09T20:17:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:57 smithi038 ceph-mon[92422]: pgmap v10744: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 715 KiB/s rd, 711 KiB/s wr, 15 op/s 2023-03-09T20:17:58.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19650.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:17:59.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19188.561203 secs 2023-03-09T20:17:59.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19188.561203 secs 2023-03-09T20:17:59.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19188.561203 secs 2023-03-09T20:18:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:59 smithi122 ceph-mon[93869]: pgmap v10745: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 630 KiB/s rd, 630 KiB/s wr, 13 op/s 2023-03-09T20:18:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:18:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:17:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:18:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:59 smithi181 ceph-mon[94189]: pgmap v10745: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 630 KiB/s rd, 630 KiB/s wr, 13 op/s 2023-03-09T20:18:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:18:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:17:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:18:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:59 smithi038 ceph-mon[92422]: pgmap v10745: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 630 KiB/s rd, 630 KiB/s wr, 13 op/s 2023-03-09T20:18:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:18:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:17:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:18:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19191.238983 secs 2023-03-09T20:18:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19191.238983 secs 2023-03-09T20:18:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19191.238983 secs 2023-03-09T20:18:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:01 smithi122 ceph-mon[93869]: pgmap v10746: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 419 KiB/s wr, 11 op/s 2023-03-09T20:18:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:01 smithi181 ceph-mon[94189]: pgmap v10746: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 419 KiB/s wr, 11 op/s 2023-03-09T20:18:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:01 smithi038 ceph-mon[92422]: pgmap v10746: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 419 KiB/s wr, 11 op/s 2023-03-09T20:18:03.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:02 smithi122 ceph-mon[93869]: pgmap v10747: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 419 KiB/s wr, 14 op/s 2023-03-09T20:18:03.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:02 smithi181 ceph-mon[94189]: pgmap v10747: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 419 KiB/s wr, 14 op/s 2023-03-09T20:18:03.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:02 smithi038 ceph-mon[92422]: pgmap v10747: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 419 KiB/s wr, 14 op/s 2023-03-09T20:18:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19655.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:18:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19193.561288 secs 2023-03-09T20:18:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19193.561288 secs 2023-03-09T20:18:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19193.561288 secs 2023-03-09T20:18:05.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:04 smithi038 ceph-mon[92422]: pgmap v10748: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 404 KiB/s wr, 23 op/s 2023-03-09T20:18:05.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:04 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19196.239043 secs 2023-03-09T20:18:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:04 smithi122 ceph-mon[93869]: pgmap v10748: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 404 KiB/s wr, 23 op/s 2023-03-09T20:18:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:04 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19196.239043 secs 2023-03-09T20:18:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:04 smithi181 ceph-mon[94189]: pgmap v10748: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 404 KiB/s wr, 23 op/s 2023-03-09T20:18:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:04 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19196.239043 secs 2023-03-09T20:18:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:05 smithi038 ceph-mon[92422]: osdmap e885: 12 total, 12 up, 12 in 2023-03-09T20:18:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:05 smithi122 ceph-mon[93869]: osdmap e885: 12 total, 12 up, 12 in 2023-03-09T20:18:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:05 smithi181 ceph-mon[94189]: osdmap e885: 12 total, 12 up, 12 in 2023-03-09T20:18:07.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:06 smithi038 ceph-mon[92422]: pgmap v10750: 129 pgs: 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 125 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 271 KiB/s wr, 27 op/s 2023-03-09T20:18:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:06 smithi122 ceph-mon[93869]: pgmap v10750: 129 pgs: 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 125 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 271 KiB/s wr, 27 op/s 2023-03-09T20:18:07.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:06 smithi181 ceph-mon[94189]: pgmap v10750: 129 pgs: 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 125 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 271 KiB/s wr, 27 op/s 2023-03-09T20:18:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19198.561415 secs 2023-03-09T20:18:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19198.561415 secs 2023-03-09T20:18:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19198.561415 secs 2023-03-09T20:18:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19660.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:18:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:09 smithi122 ceph-mon[93869]: pgmap v10751: 129 pgs: 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 125 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 271 KiB/s wr, 27 op/s 2023-03-09T20:18:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:09 smithi181 ceph-mon[94189]: pgmap v10751: 129 pgs: 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 125 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 271 KiB/s wr, 27 op/s 2023-03-09T20:18:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:09 smithi038 ceph-mon[92422]: pgmap v10751: 129 pgs: 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 125 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 271 KiB/s wr, 27 op/s 2023-03-09T20:18:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19201.239103 secs 2023-03-09T20:18:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19201.239103 secs 2023-03-09T20:18:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19201.239103 secs 2023-03-09T20:18:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:11 smithi122 ceph-mon[93869]: pgmap v10752: 129 pgs: 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 125 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 237 KiB/s wr, 26 op/s 2023-03-09T20:18:12.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:11 smithi181 ceph-mon[94189]: pgmap v10752: 129 pgs: 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 125 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 237 KiB/s wr, 26 op/s 2023-03-09T20:18:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:11 smithi038 ceph-mon[92422]: pgmap v10752: 129 pgs: 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 125 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 237 KiB/s wr, 26 op/s 2023-03-09T20:18:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19665.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:18:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19203.561529 secs 2023-03-09T20:18:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:13 smithi122 ceph-mon[93869]: pgmap v10753: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 237 KiB/s wr, 25 op/s 2023-03-09T20:18:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:13 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:18:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:13 smithi122 ceph-mon[93869]: osdmap e886: 12 total, 12 up, 12 in 2023-03-09T20:18:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19203.561529 secs 2023-03-09T20:18:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:13 smithi181 ceph-mon[94189]: pgmap v10753: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 237 KiB/s wr, 25 op/s 2023-03-09T20:18:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:13 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:18:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:13 smithi181 ceph-mon[94189]: osdmap e886: 12 total, 12 up, 12 in 2023-03-09T20:18:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19203.561529 secs 2023-03-09T20:18:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:13 smithi038 ceph-mon[92422]: pgmap v10753: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 237 KiB/s wr, 25 op/s 2023-03-09T20:18:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:13 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:18:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:13 smithi038 ceph-mon[92422]: osdmap e886: 12 total, 12 up, 12 in 2023-03-09T20:18:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:15 smithi122 ceph-mon[93869]: pgmap v10755: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 169 KiB/s wr, 11 op/s 2023-03-09T20:18:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:18:15.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:18:15.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19206.239171 secs 2023-03-09T20:18:15.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:18:15.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:18:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:15 smithi181 ceph-mon[94189]: pgmap v10755: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 169 KiB/s wr, 11 op/s 2023-03-09T20:18:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:18:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:18:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19206.239171 secs 2023-03-09T20:18:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:18:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:18:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:15 smithi038 ceph-mon[92422]: pgmap v10755: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 169 KiB/s wr, 11 op/s 2023-03-09T20:18:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:18:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:18:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19206.239171 secs 2023-03-09T20:18:15.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:18:15.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:18:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:16 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:18:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:16 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:18:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:16 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:18:16.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:16 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:18:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:16 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:18:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:16 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:18:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:16 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:18:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:16 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:18:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:16 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:18:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:17 smithi181 ceph-mon[94189]: pgmap v10756: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 128 KiB/s wr, 7 op/s 2023-03-09T20:18:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:17 smithi038 ceph-mon[92422]: pgmap v10756: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 128 KiB/s wr, 7 op/s 2023-03-09T20:18:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:17 smithi122 ceph-mon[93869]: pgmap v10756: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 128 KiB/s wr, 7 op/s 2023-03-09T20:18:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19208.561653 secs 2023-03-09T20:18:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19670.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:18:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19208.561653 secs 2023-03-09T20:18:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19208.561653 secs 2023-03-09T20:18:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:19 smithi122 ceph-mon[93869]: pgmap v10757: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 128 KiB/s wr, 6 op/s 2023-03-09T20:18:20.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:19 smithi181 ceph-mon[94189]: pgmap v10757: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 128 KiB/s wr, 6 op/s 2023-03-09T20:18:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:19 smithi038 ceph-mon[92422]: pgmap v10757: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 128 KiB/s wr, 6 op/s 2023-03-09T20:18:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19211.239253 secs 2023-03-09T20:18:21.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19211.239253 secs 2023-03-09T20:18:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19211.239253 secs 2023-03-09T20:18:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:21 smithi122 ceph-mon[93869]: pgmap v10758: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 6 op/s 2023-03-09T20:18:22.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:21 smithi181 ceph-mon[94189]: pgmap v10758: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 6 op/s 2023-03-09T20:18:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:21 smithi038 ceph-mon[92422]: pgmap v10758: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 6 op/s 2023-03-09T20:18:23.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19675.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:18:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19213.561767 secs 2023-03-09T20:18:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:23 smithi122 ceph-mon[93869]: pgmap v10759: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 6 op/s 2023-03-09T20:18:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19213.561767 secs 2023-03-09T20:18:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:23 smithi181 ceph-mon[94189]: pgmap v10759: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 6 op/s 2023-03-09T20:18:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19213.561767 secs 2023-03-09T20:18:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:23 smithi038 ceph-mon[92422]: pgmap v10759: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 6 op/s 2023-03-09T20:18:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:25 smithi122 ceph-mon[93869]: pgmap v10760: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 91 KiB/s wr, 6 op/s 2023-03-09T20:18:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19216.239316 secs 2023-03-09T20:18:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:25 smithi181 ceph-mon[94189]: pgmap v10760: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 91 KiB/s wr, 6 op/s 2023-03-09T20:18:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19216.239316 secs 2023-03-09T20:18:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:25 smithi038 ceph-mon[92422]: pgmap v10760: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 91 KiB/s wr, 6 op/s 2023-03-09T20:18:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19216.239316 secs 2023-03-09T20:18:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:27 smithi122 ceph-mon[93869]: pgmap v10761: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 58 KiB/s wr, 7 op/s 2023-03-09T20:18:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:27 smithi181 ceph-mon[94189]: pgmap v10761: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 58 KiB/s wr, 7 op/s 2023-03-09T20:18:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:27 smithi038 ceph-mon[92422]: pgmap v10761: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 58 KiB/s wr, 7 op/s 2023-03-09T20:18:28.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19680.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:18:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19218.561896 secs 2023-03-09T20:18:29.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19218.561896 secs 2023-03-09T20:18:29.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19218.561896 secs 2023-03-09T20:18:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:29 smithi122 ceph-mon[93869]: pgmap v10762: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 58 KiB/s wr, 6 op/s 2023-03-09T20:18:30.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:29 smithi181 ceph-mon[94189]: pgmap v10762: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 58 KiB/s wr, 6 op/s 2023-03-09T20:18:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:29 smithi038 ceph-mon[92422]: pgmap v10762: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 58 KiB/s wr, 6 op/s 2023-03-09T20:18:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19221.239376 secs 2023-03-09T20:18:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19221.239376 secs 2023-03-09T20:18:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19221.239376 secs 2023-03-09T20:18:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:31 smithi122 ceph-mon[93869]: pgmap v10763: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 58 KiB/s wr, 6 op/s 2023-03-09T20:18:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:31 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:18:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:31 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:18:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:31 smithi181 ceph-mon[94189]: pgmap v10763: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 58 KiB/s wr, 6 op/s 2023-03-09T20:18:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:31 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:18:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:31 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:18:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:31 smithi038 ceph-mon[92422]: pgmap v10763: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 58 KiB/s wr, 6 op/s 2023-03-09T20:18:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:31 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:18:32.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:31 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:18:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:32 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:18:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:32 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:18:33.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:32 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:18:33.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:32 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:18:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:32 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:18:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:32 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:18:33.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19685.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:18:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19223.562029 secs 2023-03-09T20:18:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:33 smithi122 ceph-mon[93869]: pgmap v10764: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 130 KiB/s wr, 9 op/s 2023-03-09T20:18:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19223.562029 secs 2023-03-09T20:18:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:33 smithi181 ceph-mon[94189]: pgmap v10764: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 130 KiB/s wr, 9 op/s 2023-03-09T20:18:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19223.562029 secs 2023-03-09T20:18:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:33 smithi038 ceph-mon[92422]: pgmap v10764: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 130 KiB/s wr, 9 op/s 2023-03-09T20:18:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:35 smithi122 ceph-mon[93869]: pgmap v10765: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 130 KiB/s wr, 7 op/s 2023-03-09T20:18:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19226.239436 secs 2023-03-09T20:18:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:35 smithi181 ceph-mon[94189]: pgmap v10765: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 130 KiB/s wr, 7 op/s 2023-03-09T20:18:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19226.239436 secs 2023-03-09T20:18:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:35 smithi038 ceph-mon[92422]: pgmap v10765: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 130 KiB/s wr, 7 op/s 2023-03-09T20:18:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19226.239436 secs 2023-03-09T20:18:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:37 smithi122 ceph-mon[93869]: pgmap v10766: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 100 KiB/s wr, 9 op/s 2023-03-09T20:18:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:37 smithi181 ceph-mon[94189]: pgmap v10766: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 100 KiB/s wr, 9 op/s 2023-03-09T20:18:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:37 smithi038 ceph-mon[92422]: pgmap v10766: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 100 KiB/s wr, 9 op/s 2023-03-09T20:18:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19690.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:18:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:39 smithi122 ceph-mon[93869]: pgmap v10767: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 72 KiB/s wr, 7 op/s 2023-03-09T20:18:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:39 smithi181 ceph-mon[94189]: pgmap v10767: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 72 KiB/s wr, 7 op/s 2023-03-09T20:18:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:39 smithi038 ceph-mon[92422]: pgmap v10767: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 72 KiB/s wr, 7 op/s 2023-03-09T20:18:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19231.239494 secs 2023-03-09T20:18:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:40 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:18:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19231.239494 secs 2023-03-09T20:18:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:40 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:18:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19231.239494 secs 2023-03-09T20:18:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:40 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:18:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:41 smithi122 ceph-mon[93869]: pgmap v10768: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 71 KiB/s wr, 7 op/s 2023-03-09T20:18:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:41 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:18:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:18:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:18:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:41 smithi181 ceph-mon[94189]: pgmap v10768: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 71 KiB/s wr, 7 op/s 2023-03-09T20:18:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:41 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:18:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:18:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:18:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:41 smithi038 ceph-mon[92422]: pgmap v10768: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 71 KiB/s wr, 7 op/s 2023-03-09T20:18:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:41 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:18:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:18:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:18:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19695.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:18:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:43 smithi122 ceph-mon[93869]: pgmap v10769: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 254 KiB/s wr, 10 op/s 2023-03-09T20:18:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:43 smithi181 ceph-mon[94189]: pgmap v10769: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 254 KiB/s wr, 10 op/s 2023-03-09T20:18:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:43 smithi038 ceph-mon[92422]: pgmap v10769: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 254 KiB/s wr, 10 op/s 2023-03-09T20:18:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:45 smithi122 ceph-mon[93869]: pgmap v10770: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 183 KiB/s wr, 7 op/s 2023-03-09T20:18:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19236.239554 secs 2023-03-09T20:18:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:45 smithi181 ceph-mon[94189]: pgmap v10770: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 183 KiB/s wr, 7 op/s 2023-03-09T20:18:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19236.239554 secs 2023-03-09T20:18:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:45 smithi038 ceph-mon[92422]: pgmap v10770: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 183 KiB/s wr, 7 op/s 2023-03-09T20:18:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19236.239554 secs 2023-03-09T20:18:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:47 smithi122 ceph-mon[93869]: pgmap v10771: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 274 KiB/s wr, 11 op/s 2023-03-09T20:18:48.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:47 smithi181 ceph-mon[94189]: pgmap v10771: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 274 KiB/s wr, 11 op/s 2023-03-09T20:18:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:47 smithi038 ceph-mon[92422]: pgmap v10771: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 274 KiB/s wr, 11 op/s 2023-03-09T20:18:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19700.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:18:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:49 smithi122 ceph-mon[93869]: pgmap v10772: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 348 KiB/s rd, 333 KiB/s wr, 9 op/s 2023-03-09T20:18:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:49 smithi181 ceph-mon[94189]: pgmap v10772: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 348 KiB/s rd, 333 KiB/s wr, 9 op/s 2023-03-09T20:18:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:49 smithi038 ceph-mon[92422]: pgmap v10772: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 348 KiB/s rd, 333 KiB/s wr, 9 op/s 2023-03-09T20:18:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19241.239638 secs 2023-03-09T20:18:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19241.239638 secs 2023-03-09T20:18:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19241.239638 secs 2023-03-09T20:18:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:51 smithi122 ceph-mon[93869]: pgmap v10773: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 348 KiB/s rd, 333 KiB/s wr, 9 op/s 2023-03-09T20:18:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:18:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:18:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:51 smithi181 ceph-mon[94189]: pgmap v10773: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 348 KiB/s rd, 333 KiB/s wr, 9 op/s 2023-03-09T20:18:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:18:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:18:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:51 smithi038 ceph-mon[92422]: pgmap v10773: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 348 KiB/s rd, 333 KiB/s wr, 9 op/s 2023-03-09T20:18:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:18:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:18:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:18:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:18:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:18:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:18:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:18:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:18:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19705.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:18:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:53 smithi122 ceph-mon[93869]: pgmap v10774: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 595 KiB/s rd, 580 KiB/s wr, 14 op/s 2023-03-09T20:18:54.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:53 smithi181 ceph-mon[94189]: pgmap v10774: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 595 KiB/s rd, 580 KiB/s wr, 14 op/s 2023-03-09T20:18:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:53 smithi038 ceph-mon[92422]: pgmap v10774: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 595 KiB/s rd, 580 KiB/s wr, 14 op/s 2023-03-09T20:18:55.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:54 smithi122 ceph-mon[93869]: pgmap v10775: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 398 KiB/s wr, 10 op/s 2023-03-09T20:18:55.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:54 smithi181 ceph-mon[94189]: pgmap v10775: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 398 KiB/s wr, 10 op/s 2023-03-09T20:18:55.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:54 smithi038 ceph-mon[92422]: pgmap v10775: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 398 KiB/s wr, 10 op/s 2023-03-09T20:18:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19246.239698 secs 2023-03-09T20:18:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19246.239698 secs 2023-03-09T20:18:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19246.239698 secs 2023-03-09T20:18:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:56 smithi122 ceph-mon[93869]: pgmap v10776: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 465 KiB/s wr, 14 op/s 2023-03-09T20:18:57.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:56 smithi181 ceph-mon[94189]: pgmap v10776: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 465 KiB/s wr, 14 op/s 2023-03-09T20:18:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:56 smithi038 ceph-mon[92422]: pgmap v10776: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 465 KiB/s wr, 14 op/s 2023-03-09T20:18:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19248.562593 secs 2023-03-09T20:18:58.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19248.562593 secs 2023-03-09T20:18:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19248.562593 secs 2023-03-09T20:18:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19710.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:19:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:59 smithi122 ceph-mon[93869]: pgmap v10777: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 429 KiB/s wr, 12 op/s 2023-03-09T20:19:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:19:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:18:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:19:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:59 smithi181 ceph-mon[94189]: pgmap v10777: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 429 KiB/s wr, 12 op/s 2023-03-09T20:19:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:19:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:18:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:19:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:59 smithi038 ceph-mon[92422]: pgmap v10777: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 429 KiB/s wr, 12 op/s 2023-03-09T20:19:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:19:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:18:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:19:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19251.239758 secs 2023-03-09T20:19:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:19:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:19:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19251.239758 secs 2023-03-09T20:19:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:19:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:19:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19251.239758 secs 2023-03-09T20:19:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:19:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:19:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:01 smithi122 ceph-mon[93869]: pgmap v10778: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 369 KiB/s wr, 10 op/s 2023-03-09T20:19:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:01 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:19:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:19:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:19:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:01 smithi181 ceph-mon[94189]: pgmap v10778: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 369 KiB/s wr, 10 op/s 2023-03-09T20:19:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:01 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:19:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:19:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:19:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:01 smithi038 ceph-mon[92422]: pgmap v10778: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 369 KiB/s wr, 10 op/s 2023-03-09T20:19:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:01 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:19:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:19:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:19:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19715.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:19:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19253.562718 secs 2023-03-09T20:19:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:03 smithi122 ceph-mon[93869]: pgmap v10779: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 478 KiB/s rd, 499 KiB/s wr, 16 op/s 2023-03-09T20:19:04.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19253.562718 secs 2023-03-09T20:19:04.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:03 smithi181 ceph-mon[94189]: pgmap v10779: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 478 KiB/s rd, 499 KiB/s wr, 16 op/s 2023-03-09T20:19:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19253.562718 secs 2023-03-09T20:19:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:03 smithi038 ceph-mon[92422]: pgmap v10779: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 478 KiB/s rd, 499 KiB/s wr, 16 op/s 2023-03-09T20:19:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:05 smithi122 ceph-mon[93869]: pgmap v10780: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 274 KiB/s wr, 24 op/s 2023-03-09T20:19:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19256.239818 secs 2023-03-09T20:19:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:05 smithi181 ceph-mon[94189]: pgmap v10780: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 274 KiB/s wr, 24 op/s 2023-03-09T20:19:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19256.239818 secs 2023-03-09T20:19:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:05 smithi038 ceph-mon[92422]: pgmap v10780: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 274 KiB/s wr, 24 op/s 2023-03-09T20:19:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19256.239818 secs 2023-03-09T20:19:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:07 smithi122 ceph-mon[93869]: pgmap v10781: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 342 KiB/s wr, 27 op/s 2023-03-09T20:19:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:07 smithi181 ceph-mon[94189]: pgmap v10781: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 342 KiB/s wr, 27 op/s 2023-03-09T20:19:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:07 smithi038 ceph-mon[92422]: pgmap v10781: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 342 KiB/s wr, 27 op/s 2023-03-09T20:19:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19720.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:19:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19258.562835 secs 2023-03-09T20:19:09.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19258.562835 secs 2023-03-09T20:19:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19258.562835 secs 2023-03-09T20:19:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:09 smithi122 ceph-mon[93869]: pgmap v10782: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 275 KiB/s wr, 23 op/s 2023-03-09T20:19:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:09 smithi181 ceph-mon[94189]: pgmap v10782: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 275 KiB/s wr, 23 op/s 2023-03-09T20:19:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:09 smithi038 ceph-mon[92422]: pgmap v10782: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 275 KiB/s wr, 23 op/s 2023-03-09T20:19:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19261.239879 secs 2023-03-09T20:19:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:10 smithi122 ceph-mon[93869]: osdmap e887: 12 total, 12 up, 12 in 2023-03-09T20:19:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19261.239879 secs 2023-03-09T20:19:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:10 smithi181 ceph-mon[94189]: osdmap e887: 12 total, 12 up, 12 in 2023-03-09T20:19:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19261.239879 secs 2023-03-09T20:19:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:10 smithi038 ceph-mon[92422]: osdmap e887: 12 total, 12 up, 12 in 2023-03-09T20:19:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:11 smithi122 ceph-mon[93869]: pgmap v10784: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 265 KiB/s wr, 26 op/s 2023-03-09T20:19:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:11 smithi181 ceph-mon[94189]: pgmap v10784: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 265 KiB/s wr, 26 op/s 2023-03-09T20:19:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:11 smithi038 ceph-mon[92422]: pgmap v10784: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 265 KiB/s wr, 26 op/s 2023-03-09T20:19:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19725.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:19:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19263.562924 secs 2023-03-09T20:19:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:13 smithi122 ceph-mon[93869]: pgmap v10785: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 108 KiB/s wr, 22 op/s 2023-03-09T20:19:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19263.562924 secs 2023-03-09T20:19:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:13 smithi181 ceph-mon[94189]: pgmap v10785: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 108 KiB/s wr, 22 op/s 2023-03-09T20:19:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19263.562924 secs 2023-03-09T20:19:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:13 smithi038 ceph-mon[92422]: pgmap v10785: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 108 KiB/s wr, 22 op/s 2023-03-09T20:19:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:15 smithi122 ceph-mon[93869]: pgmap v10786: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 83 KiB/s wr, 8 op/s 2023-03-09T20:19:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19266.239939 secs 2023-03-09T20:19:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:19:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:15 smithi181 ceph-mon[94189]: pgmap v10786: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 83 KiB/s wr, 8 op/s 2023-03-09T20:19:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19266.239939 secs 2023-03-09T20:19:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:19:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:15 smithi038 ceph-mon[92422]: pgmap v10786: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 83 KiB/s wr, 8 op/s 2023-03-09T20:19:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19266.239939 secs 2023-03-09T20:19:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:19:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:16 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:19:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:16 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:19:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:16 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:19:17.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:16 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:19:17.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:16 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:19:17.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:16 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:19:17.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:16 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:19:17.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:16 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:19:17.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:16 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:19:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:17 smithi122 ceph-mon[93869]: pgmap v10787: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 7 op/s 2023-03-09T20:19:18.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:17 smithi181 ceph-mon[94189]: pgmap v10787: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 7 op/s 2023-03-09T20:19:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:17 smithi038 ceph-mon[92422]: pgmap v10787: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 7 op/s 2023-03-09T20:19:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19730.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:19:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19268.563038 secs 2023-03-09T20:19:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:18 smithi122 ceph-mon[93869]: osdmap e888: 12 total, 12 up, 12 in 2023-03-09T20:19:19.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19268.563038 secs 2023-03-09T20:19:19.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:18 smithi181 ceph-mon[94189]: osdmap e888: 12 total, 12 up, 12 in 2023-03-09T20:19:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19268.563038 secs 2023-03-09T20:19:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:18 smithi038 ceph-mon[92422]: osdmap e888: 12 total, 12 up, 12 in 2023-03-09T20:19:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:19 smithi122 ceph-mon[93869]: pgmap v10788: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 7 op/s 2023-03-09T20:19:20.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:19 smithi181 ceph-mon[94189]: pgmap v10788: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 7 op/s 2023-03-09T20:19:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:19 smithi038 ceph-mon[92422]: pgmap v10788: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 7 op/s 2023-03-09T20:19:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19271.239985 secs 2023-03-09T20:19:21.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19271.239985 secs 2023-03-09T20:19:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19271.239985 secs 2023-03-09T20:19:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:21 smithi122 ceph-mon[93869]: pgmap v10790: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 7 op/s 2023-03-09T20:19:22.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:21 smithi181 ceph-mon[94189]: pgmap v10790: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 7 op/s 2023-03-09T20:19:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:21 smithi038 ceph-mon[92422]: pgmap v10790: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 7 op/s 2023-03-09T20:19:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:22 smithi122 ceph-mon[93869]: pgmap v10791: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 6 op/s 2023-03-09T20:19:23.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:22 smithi181 ceph-mon[94189]: pgmap v10791: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 6 op/s 2023-03-09T20:19:23.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:22 smithi038 ceph-mon[92422]: pgmap v10791: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 6 op/s 2023-03-09T20:19:23.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19735.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:19:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19273.563149 secs 2023-03-09T20:19:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19273.563149 secs 2023-03-09T20:19:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19273.563149 secs 2023-03-09T20:19:25.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:24 smithi038 ceph-mon[92422]: pgmap v10792: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 6 op/s 2023-03-09T20:19:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:24 smithi122 ceph-mon[93869]: pgmap v10792: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 6 op/s 2023-03-09T20:19:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:24 smithi181 ceph-mon[94189]: pgmap v10792: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 6 op/s 2023-03-09T20:19:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19276.240043 secs 2023-03-09T20:19:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19276.240043 secs 2023-03-09T20:19:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19276.240043 secs 2023-03-09T20:19:27.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:26 smithi038 ceph-mon[92422]: pgmap v10793: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-03-09T20:19:27.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:26 smithi122 ceph-mon[93869]: pgmap v10793: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-03-09T20:19:27.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:27 smithi181 ceph-mon[94189]: pgmap v10793: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-03-09T20:19:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19278.563270 secs 2023-03-09T20:19:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19278.563270 secs 2023-03-09T20:19:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19278.563270 secs 2023-03-09T20:19:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19740.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:19:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:29 smithi122 ceph-mon[93869]: pgmap v10794: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T20:19:30.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:29 smithi181 ceph-mon[94189]: pgmap v10794: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T20:19:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:29 smithi038 ceph-mon[92422]: pgmap v10794: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T20:19:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19281.240103 secs 2023-03-09T20:19:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19281.240103 secs 2023-03-09T20:19:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19281.240103 secs 2023-03-09T20:19:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:31 smithi122 ceph-mon[93869]: pgmap v10795: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T20:19:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:31 smithi181 ceph-mon[94189]: pgmap v10795: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T20:19:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:31 smithi038 ceph-mon[92422]: pgmap v10795: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T20:19:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19745.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:19:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19283.563366 secs 2023-03-09T20:19:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:33 smithi122 ceph-mon[93869]: pgmap v10796: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 8 op/s 2023-03-09T20:19:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19283.563366 secs 2023-03-09T20:19:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:33 smithi181 ceph-mon[94189]: pgmap v10796: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 8 op/s 2023-03-09T20:19:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19283.563366 secs 2023-03-09T20:19:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:33 smithi038 ceph-mon[92422]: pgmap v10796: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 8 op/s 2023-03-09T20:19:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:35 smithi122 ceph-mon[93869]: pgmap v10797: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-09T20:19:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19286.240163 secs 2023-03-09T20:19:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:35 smithi181 ceph-mon[94189]: pgmap v10797: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-09T20:19:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19286.240163 secs 2023-03-09T20:19:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:35 smithi038 ceph-mon[92422]: pgmap v10797: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-09T20:19:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19286.240163 secs 2023-03-09T20:19:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:37 smithi122 ceph-mon[93869]: pgmap v10798: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 8 op/s 2023-03-09T20:19:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:37 smithi181 ceph-mon[94189]: pgmap v10798: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 8 op/s 2023-03-09T20:19:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:37 smithi038 ceph-mon[92422]: pgmap v10798: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 8 op/s 2023-03-09T20:19:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19750.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:19:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:38 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 19288.563474 secs 2023-03-09T20:19:39.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:38 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 19288.563474 secs 2023-03-09T20:19:39.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:38 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 19288.563474 secs 2023-03-09T20:19:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:39 smithi122 ceph-mon[93869]: pgmap v10799: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 6 op/s 2023-03-09T20:19:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:39 smithi181 ceph-mon[94189]: pgmap v10799: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 6 op/s 2023-03-09T20:19:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:39 smithi038 ceph-mon[92422]: pgmap v10799: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 6 op/s 2023-03-09T20:19:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19291.240226 secs 2023-03-09T20:19:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:19:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:19:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19291.240226 secs 2023-03-09T20:19:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:19:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:19:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19291.240226 secs 2023-03-09T20:19:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:19:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:19:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:41 smithi122 ceph-mon[93869]: pgmap v10800: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 5 op/s 2023-03-09T20:19:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:19:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:19:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:41 smithi181 ceph-mon[94189]: pgmap v10800: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 5 op/s 2023-03-09T20:19:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:19:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:19:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:41 smithi038 ceph-mon[92422]: pgmap v10800: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 5 op/s 2023-03-09T20:19:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:19:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:19:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19755.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:19:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:43 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 19293.563591 secs 2023-03-09T20:19:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:43 smithi122 ceph-mon[93869]: pgmap v10801: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 184 KiB/s wr, 10 op/s 2023-03-09T20:19:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:43 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 19293.563591 secs 2023-03-09T20:19:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:43 smithi181 ceph-mon[94189]: pgmap v10801: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 184 KiB/s wr, 10 op/s 2023-03-09T20:19:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:43 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 19293.563591 secs 2023-03-09T20:19:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:43 smithi038 ceph-mon[92422]: pgmap v10801: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 184 KiB/s wr, 10 op/s 2023-03-09T20:19:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:45 smithi122 ceph-mon[93869]: pgmap v10802: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 184 KiB/s wr, 7 op/s 2023-03-09T20:19:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19296.240285 secs 2023-03-09T20:19:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:45 smithi181 ceph-mon[94189]: pgmap v10802: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 184 KiB/s wr, 7 op/s 2023-03-09T20:19:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19296.240285 secs 2023-03-09T20:19:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:45 smithi038 ceph-mon[92422]: pgmap v10802: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 184 KiB/s wr, 7 op/s 2023-03-09T20:19:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19296.240285 secs 2023-03-09T20:19:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:47 smithi122 ceph-mon[93869]: pgmap v10803: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 200 KiB/s wr, 11 op/s 2023-03-09T20:19:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:19:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:19:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:47 smithi181 ceph-mon[94189]: pgmap v10803: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 200 KiB/s wr, 11 op/s 2023-03-09T20:19:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:19:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:19:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:47 smithi038 ceph-mon[92422]: pgmap v10803: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 200 KiB/s wr, 11 op/s 2023-03-09T20:19:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:19:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:19:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19760.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:19:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19298.563730 secs 2023-03-09T20:19:49.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19298.563730 secs 2023-03-09T20:19:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19298.563730 secs 2023-03-09T20:19:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:49 smithi122 ceph-mon[93869]: pgmap v10804: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 335 KiB/s wr, 10 op/s 2023-03-09T20:19:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:19:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:19:50.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:49 smithi181 ceph-mon[94189]: pgmap v10804: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 335 KiB/s wr, 10 op/s 2023-03-09T20:19:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:19:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:19:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:49 smithi038 ceph-mon[92422]: pgmap v10804: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 335 KiB/s wr, 10 op/s 2023-03-09T20:19:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:19:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:19:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19301.240345 secs 2023-03-09T20:19:51.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19301.240345 secs 2023-03-09T20:19:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19301.240345 secs 2023-03-09T20:19:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:51 smithi122 ceph-mon[93869]: pgmap v10805: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 335 KiB/s wr, 9 op/s 2023-03-09T20:19:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:51 smithi181 ceph-mon[94189]: pgmap v10805: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 335 KiB/s wr, 9 op/s 2023-03-09T20:19:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:51 smithi038 ceph-mon[92422]: pgmap v10805: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 335 KiB/s wr, 9 op/s 2023-03-09T20:19:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:19:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:19:53.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:19:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:19:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:19:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:19:53.801 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19765.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:19:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19303.563863 secs 2023-03-09T20:19:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:53 smithi122 ceph-mon[93869]: pgmap v10806: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 597 KiB/s rd, 582 KiB/s wr, 15 op/s 2023-03-09T20:19:54.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19303.563863 secs 2023-03-09T20:19:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:53 smithi181 ceph-mon[94189]: pgmap v10806: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 597 KiB/s rd, 582 KiB/s wr, 15 op/s 2023-03-09T20:19:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19303.563863 secs 2023-03-09T20:19:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:53 smithi038 ceph-mon[92422]: pgmap v10806: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 597 KiB/s rd, 582 KiB/s wr, 15 op/s 2023-03-09T20:19:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:55 smithi122 ceph-mon[93869]: pgmap v10807: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 400 KiB/s wr, 11 op/s 2023-03-09T20:19:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19306.240431 secs 2023-03-09T20:19:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:19:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:19:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:55 smithi181 ceph-mon[94189]: pgmap v10807: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 400 KiB/s wr, 11 op/s 2023-03-09T20:19:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19306.240431 secs 2023-03-09T20:19:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:19:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:19:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:55 smithi038 ceph-mon[92422]: pgmap v10807: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 400 KiB/s wr, 11 op/s 2023-03-09T20:19:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19306.240431 secs 2023-03-09T20:19:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:19:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:19:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:57 smithi122 ceph-mon[93869]: pgmap v10808: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 505 KiB/s rd, 502 KiB/s wr, 14 op/s 2023-03-09T20:19:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:57 smithi181 ceph-mon[94189]: pgmap v10808: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 505 KiB/s rd, 502 KiB/s wr, 14 op/s 2023-03-09T20:19:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:57 smithi038 ceph-mon[92422]: pgmap v10808: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 505 KiB/s rd, 502 KiB/s wr, 14 op/s 2023-03-09T20:19:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19770.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:19:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19308.563986 secs 2023-03-09T20:19:59.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19308.563986 secs 2023-03-09T20:19:59.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19308.563986 secs 2023-03-09T20:20:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:19:59 smithi122 ceph-mon[93869]: pgmap v10809: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 487 KiB/s wr, 12 op/s 2023-03-09T20:20:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:19:59 smithi181 ceph-mon[94189]: pgmap v10809: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 487 KiB/s wr, 12 op/s 2023-03-09T20:20:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:19:59 smithi038 ceph-mon[92422]: pgmap v10809: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 487 KiB/s wr, 12 op/s 2023-03-09T20:20:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19311.240489 secs 2023-03-09T20:20:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:00 smithi181 ceph-mon[94189]: Health detail: HEALTH_WARN 2 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-09T20:20:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:00 smithi181 ceph-mon[94189]: [WRN] MDS_SLOW_REQUEST: 2 MDSs report slow requests 2023-03-09T20:20:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:00 smithi181 ceph-mon[94189]: mds.i(mds.2): 2 slow requests are blocked > 180 secs 2023-03-09T20:20:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:00 smithi181 ceph-mon[94189]: mds.h(mds.1): 2 slow requests are blocked > 180 secs 2023-03-09T20:20:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:00 smithi181 ceph-mon[94189]: [WRN] MDS_TRIM: 2 MDSs behind on trimming 2023-03-09T20:20:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:00 smithi181 ceph-mon[94189]: mds.e(mds.0): Behind on trimming (354/128) max_segments: 128, num_segments: 354 2023-03-09T20:20:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:00 smithi181 ceph-mon[94189]: mds.b(mds.0): Behind on trimming (354/128) max_segments: 128, num_segments: 354 2023-03-09T20:20:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:00 smithi181 ceph-mon[94189]: pgmap v10810: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 351 KiB/s wr, 10 op/s 2023-03-09T20:20:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19311.240489 secs 2023-03-09T20:20:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:00 smithi038 ceph-mon[92422]: Health detail: HEALTH_WARN 2 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-09T20:20:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:00 smithi038 ceph-mon[92422]: [WRN] MDS_SLOW_REQUEST: 2 MDSs report slow requests 2023-03-09T20:20:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:00 smithi038 ceph-mon[92422]: mds.i(mds.2): 2 slow requests are blocked > 180 secs 2023-03-09T20:20:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:00 smithi038 ceph-mon[92422]: mds.h(mds.1): 2 slow requests are blocked > 180 secs 2023-03-09T20:20:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:00 smithi038 ceph-mon[92422]: [WRN] MDS_TRIM: 2 MDSs behind on trimming 2023-03-09T20:20:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:00 smithi038 ceph-mon[92422]: mds.e(mds.0): Behind on trimming (354/128) max_segments: 128, num_segments: 354 2023-03-09T20:20:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:00 smithi038 ceph-mon[92422]: mds.b(mds.0): Behind on trimming (354/128) max_segments: 128, num_segments: 354 2023-03-09T20:20:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:00 smithi038 ceph-mon[92422]: pgmap v10810: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 351 KiB/s wr, 10 op/s 2023-03-09T20:20:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19311.240489 secs 2023-03-09T20:20:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:00 smithi122 ceph-mon[93869]: Health detail: HEALTH_WARN 2 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-09T20:20:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:00 smithi122 ceph-mon[93869]: [WRN] MDS_SLOW_REQUEST: 2 MDSs report slow requests 2023-03-09T20:20:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:00 smithi122 ceph-mon[93869]: mds.i(mds.2): 2 slow requests are blocked > 180 secs 2023-03-09T20:20:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:00 smithi122 ceph-mon[93869]: mds.h(mds.1): 2 slow requests are blocked > 180 secs 2023-03-09T20:20:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:00 smithi122 ceph-mon[93869]: [WRN] MDS_TRIM: 2 MDSs behind on trimming 2023-03-09T20:20:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:00 smithi122 ceph-mon[93869]: mds.e(mds.0): Behind on trimming (354/128) max_segments: 128, num_segments: 354 2023-03-09T20:20:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:00 smithi122 ceph-mon[93869]: mds.b(mds.0): Behind on trimming (354/128) max_segments: 128, num_segments: 354 2023-03-09T20:20:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:00 smithi122 ceph-mon[93869]: pgmap v10810: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 351 KiB/s wr, 10 op/s 2023-03-09T20:20:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19313.564053 secs 2023-03-09T20:20:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:03 smithi038 ceph-mon[92422]: pgmap v10811: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 482 KiB/s wr, 16 op/s 2023-03-09T20:20:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19313.564053 secs 2023-03-09T20:20:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:03 smithi122 ceph-mon[93869]: pgmap v10811: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 482 KiB/s wr, 16 op/s 2023-03-09T20:20:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19313.564053 secs 2023-03-09T20:20:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:03 smithi181 ceph-mon[94189]: pgmap v10811: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 482 KiB/s wr, 16 op/s 2023-03-09T20:20:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19775.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:20:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:05 smithi122 ceph-mon[93869]: pgmap v10812: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 256 KiB/s wr, 23 op/s 2023-03-09T20:20:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19316.240573 secs 2023-03-09T20:20:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:05 smithi181 ceph-mon[94189]: pgmap v10812: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 256 KiB/s wr, 23 op/s 2023-03-09T20:20:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19316.240573 secs 2023-03-09T20:20:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:05 smithi038 ceph-mon[92422]: pgmap v10812: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 256 KiB/s wr, 23 op/s 2023-03-09T20:20:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19316.240573 secs 2023-03-09T20:20:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:07 smithi122 ceph-mon[93869]: pgmap v10813: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 303 KiB/s wr, 26 op/s 2023-03-09T20:20:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:07 smithi181 ceph-mon[94189]: pgmap v10813: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 303 KiB/s wr, 26 op/s 2023-03-09T20:20:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:07 smithi038 ceph-mon[92422]: pgmap v10813: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 303 KiB/s wr, 26 op/s 2023-03-09T20:20:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19780.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:20:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19318.564202 secs 2023-03-09T20:20:09.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19318.564202 secs 2023-03-09T20:20:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19318.564202 secs 2023-03-09T20:20:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:09 smithi122 ceph-mon[93869]: pgmap v10814: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 221 KiB/s wr, 22 op/s 2023-03-09T20:20:10.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:09 smithi181 ceph-mon[94189]: pgmap v10814: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 221 KiB/s wr, 22 op/s 2023-03-09T20:20:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:09 smithi038 ceph-mon[92422]: pgmap v10814: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 221 KiB/s wr, 22 op/s 2023-03-09T20:20:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19321.240635 secs 2023-03-09T20:20:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:10 smithi122 ceph-mon[93869]: osdmap e889: 12 total, 12 up, 12 in 2023-03-09T20:20:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19321.240635 secs 2023-03-09T20:20:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:10 smithi181 ceph-mon[94189]: osdmap e889: 12 total, 12 up, 12 in 2023-03-09T20:20:11.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19321.240635 secs 2023-03-09T20:20:11.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:10 smithi038 ceph-mon[92422]: osdmap e889: 12 total, 12 up, 12 in 2023-03-09T20:20:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:11 smithi122 ceph-mon[93869]: pgmap v10816: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 265 KiB/s wr, 26 op/s 2023-03-09T20:20:12.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:11 smithi181 ceph-mon[94189]: pgmap v10816: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 265 KiB/s wr, 26 op/s 2023-03-09T20:20:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:11 smithi038 ceph-mon[92422]: pgmap v10816: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 265 KiB/s wr, 26 op/s 2023-03-09T20:20:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19785.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:20:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19323.564334 secs 2023-03-09T20:20:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:13 smithi122 ceph-mon[93869]: pgmap v10817: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 108 KiB/s wr, 22 op/s 2023-03-09T20:20:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19323.564334 secs 2023-03-09T20:20:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:13 smithi181 ceph-mon[94189]: pgmap v10817: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 108 KiB/s wr, 22 op/s 2023-03-09T20:20:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19323.564334 secs 2023-03-09T20:20:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:13 smithi038 ceph-mon[92422]: pgmap v10817: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 108 KiB/s wr, 22 op/s 2023-03-09T20:20:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:15 smithi122 ceph-mon[93869]: pgmap v10818: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 83 KiB/s wr, 7 op/s 2023-03-09T20:20:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19326.240704 secs 2023-03-09T20:20:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:15 smithi181 ceph-mon[94189]: pgmap v10818: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 83 KiB/s wr, 7 op/s 2023-03-09T20:20:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19326.240704 secs 2023-03-09T20:20:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:15 smithi038 ceph-mon[92422]: pgmap v10818: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 83 KiB/s wr, 7 op/s 2023-03-09T20:20:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19326.240704 secs 2023-03-09T20:20:16.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:20:16.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:20:16.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:16 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:20:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:20:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:20:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:16 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:20:17.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:20:17.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:20:17.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:16 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:20:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:17 smithi122 ceph-mon[93869]: pgmap v10819: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 25 KiB/s wr, 6 op/s 2023-03-09T20:20:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:17 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:20:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:17 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:20:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:17 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:20:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:17 smithi181 ceph-mon[94189]: pgmap v10819: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 25 KiB/s wr, 6 op/s 2023-03-09T20:20:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:17 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:20:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:17 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:20:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:17 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:20:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:17 smithi038 ceph-mon[92422]: pgmap v10819: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 25 KiB/s wr, 6 op/s 2023-03-09T20:20:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:17 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:20:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:17 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:20:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:17 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:20:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19790.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:20:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19328.564446 secs 2023-03-09T20:20:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:18 smithi122 ceph-mon[93869]: osdmap e890: 12 total, 12 up, 12 in 2023-03-09T20:20:19.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19328.564446 secs 2023-03-09T20:20:19.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:18 smithi181 ceph-mon[94189]: osdmap e890: 12 total, 12 up, 12 in 2023-03-09T20:20:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19328.564446 secs 2023-03-09T20:20:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:18 smithi038 ceph-mon[92422]: osdmap e890: 12 total, 12 up, 12 in 2023-03-09T20:20:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:19 smithi122 ceph-mon[93869]: pgmap v10820: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 716 B/s wr, 7 op/s 2023-03-09T20:20:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:19 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:20:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:19 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:20:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:19 smithi181 ceph-mon[94189]: pgmap v10820: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 716 B/s wr, 7 op/s 2023-03-09T20:20:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:19 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:20:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:19 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:20:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:19 smithi038 ceph-mon[92422]: pgmap v10820: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 716 B/s wr, 7 op/s 2023-03-09T20:20:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:19 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:20:20.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:19 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:20:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19331.240764 secs 2023-03-09T20:20:21.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19331.240764 secs 2023-03-09T20:20:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19331.240764 secs 2023-03-09T20:20:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:21 smithi122 ceph-mon[93869]: pgmap v10822: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 716 B/s wr, 7 op/s 2023-03-09T20:20:22.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:21 smithi181 ceph-mon[94189]: pgmap v10822: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 716 B/s wr, 7 op/s 2023-03-09T20:20:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:21 smithi038 ceph-mon[92422]: pgmap v10822: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 716 B/s wr, 7 op/s 2023-03-09T20:20:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19795.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:20:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19333.564539 secs 2023-03-09T20:20:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:23 smithi122 ceph-mon[93869]: pgmap v10823: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 86 KiB/s wr, 6 op/s 2023-03-09T20:20:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19333.564539 secs 2023-03-09T20:20:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:23 smithi181 ceph-mon[94189]: pgmap v10823: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 86 KiB/s wr, 6 op/s 2023-03-09T20:20:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19333.564539 secs 2023-03-09T20:20:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:23 smithi038 ceph-mon[92422]: pgmap v10823: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 86 KiB/s wr, 6 op/s 2023-03-09T20:20:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:25 smithi122 ceph-mon[93869]: pgmap v10824: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 86 KiB/s wr, 6 op/s 2023-03-09T20:20:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19336.240849 secs 2023-03-09T20:20:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:25 smithi181 ceph-mon[94189]: pgmap v10824: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 86 KiB/s wr, 6 op/s 2023-03-09T20:20:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19336.240849 secs 2023-03-09T20:20:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:25 smithi038 ceph-mon[92422]: pgmap v10824: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 86 KiB/s wr, 6 op/s 2023-03-09T20:20:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19336.240849 secs 2023-03-09T20:20:27.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:27 smithi122 ceph-mon[93869]: pgmap v10825: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 86 KiB/s wr, 7 op/s 2023-03-09T20:20:27.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:27 smithi181 ceph-mon[94189]: pgmap v10825: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 86 KiB/s wr, 7 op/s 2023-03-09T20:20:27.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:27 smithi038 ceph-mon[92422]: pgmap v10825: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 86 KiB/s wr, 7 op/s 2023-03-09T20:20:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19338.564661 secs 2023-03-09T20:20:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19338.564661 secs 2023-03-09T20:20:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19338.564661 secs 2023-03-09T20:20:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19800.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:20:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:29 smithi122 ceph-mon[93869]: pgmap v10826: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 86 KiB/s wr, 6 op/s 2023-03-09T20:20:30.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:29 smithi181 ceph-mon[94189]: pgmap v10826: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 86 KiB/s wr, 6 op/s 2023-03-09T20:20:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:29 smithi038 ceph-mon[92422]: pgmap v10826: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 86 KiB/s wr, 6 op/s 2023-03-09T20:20:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19341.240907 secs 2023-03-09T20:20:31.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19341.240907 secs 2023-03-09T20:20:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19341.240907 secs 2023-03-09T20:20:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:31 smithi122 ceph-mon[93869]: pgmap v10827: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 76 KiB/s wr, 6 op/s 2023-03-09T20:20:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:31 smithi181 ceph-mon[94189]: pgmap v10827: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 76 KiB/s wr, 6 op/s 2023-03-09T20:20:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:31 smithi038 ceph-mon[92422]: pgmap v10827: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 76 KiB/s wr, 6 op/s 2023-03-09T20:20:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19805.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:20:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19343.564756 secs 2023-03-09T20:20:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:33 smithi122 ceph-mon[93869]: pgmap v10828: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 130 KiB/s wr, 8 op/s 2023-03-09T20:20:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19343.564756 secs 2023-03-09T20:20:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:33 smithi181 ceph-mon[94189]: pgmap v10828: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 130 KiB/s wr, 8 op/s 2023-03-09T20:20:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19343.564756 secs 2023-03-09T20:20:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:33 smithi038 ceph-mon[92422]: pgmap v10828: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 130 KiB/s wr, 8 op/s 2023-03-09T20:20:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:35 smithi122 ceph-mon[93869]: pgmap v10829: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 6 op/s 2023-03-09T20:20:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19346.240969 secs 2023-03-09T20:20:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:35 smithi181 ceph-mon[94189]: pgmap v10829: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 6 op/s 2023-03-09T20:20:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19346.240969 secs 2023-03-09T20:20:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:35 smithi038 ceph-mon[92422]: pgmap v10829: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 6 op/s 2023-03-09T20:20:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19346.240969 secs 2023-03-09T20:20:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:37 smithi122 ceph-mon[93869]: pgmap v10830: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 59 KiB/s wr, 8 op/s 2023-03-09T20:20:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:37 smithi181 ceph-mon[94189]: pgmap v10830: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 59 KiB/s wr, 8 op/s 2023-03-09T20:20:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:37 smithi038 ceph-mon[92422]: pgmap v10830: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 59 KiB/s wr, 8 op/s 2023-03-09T20:20:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19810.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:20:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19348.564906 secs 2023-03-09T20:20:39.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19348.564906 secs 2023-03-09T20:20:39.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19348.564906 secs 2023-03-09T20:20:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:39 smithi122 ceph-mon[93869]: pgmap v10831: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 6 op/s 2023-03-09T20:20:40.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:39 smithi181 ceph-mon[94189]: pgmap v10831: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 6 op/s 2023-03-09T20:20:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:39 smithi038 ceph-mon[92422]: pgmap v10831: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 6 op/s 2023-03-09T20:20:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19351.240985 secs 2023-03-09T20:20:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19351.240985 secs 2023-03-09T20:20:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19351.240985 secs 2023-03-09T20:20:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:41 smithi122 ceph-mon[93869]: pgmap v10832: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 6 op/s 2023-03-09T20:20:42.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:41 smithi181 ceph-mon[94189]: pgmap v10832: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 6 op/s 2023-03-09T20:20:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:41 smithi038 ceph-mon[92422]: pgmap v10832: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 6 op/s 2023-03-09T20:20:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19815.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:20:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19353.565041 secs 2023-03-09T20:20:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:43 smithi122 ceph-mon[93869]: pgmap v10833: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 265 KiB/s wr, 10 op/s 2023-03-09T20:20:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19353.565041 secs 2023-03-09T20:20:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:43 smithi181 ceph-mon[94189]: pgmap v10833: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 265 KiB/s wr, 10 op/s 2023-03-09T20:20:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19353.565041 secs 2023-03-09T20:20:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:43 smithi038 ceph-mon[92422]: pgmap v10833: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 265 KiB/s wr, 10 op/s 2023-03-09T20:20:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:45 smithi122 ceph-mon[93869]: pgmap v10834: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 207 KiB/s wr, 7 op/s 2023-03-09T20:20:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19356.241048 secs 2023-03-09T20:20:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:45 smithi181 ceph-mon[94189]: pgmap v10834: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 207 KiB/s wr, 7 op/s 2023-03-09T20:20:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19356.241048 secs 2023-03-09T20:20:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:45 smithi038 ceph-mon[92422]: pgmap v10834: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 207 KiB/s wr, 7 op/s 2023-03-09T20:20:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19356.241048 secs 2023-03-09T20:20:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:47 smithi122 ceph-mon[93869]: pgmap v10835: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 222 KiB/s wr, 10 op/s 2023-03-09T20:20:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:47 smithi181 ceph-mon[94189]: pgmap v10835: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 222 KiB/s wr, 10 op/s 2023-03-09T20:20:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:47 smithi038 ceph-mon[92422]: pgmap v10835: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 222 KiB/s wr, 10 op/s 2023-03-09T20:20:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19820.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:20:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19358.565163 secs 2023-03-09T20:20:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19358.565163 secs 2023-03-09T20:20:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19358.565163 secs 2023-03-09T20:20:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:49 smithi122 ceph-mon[93869]: pgmap v10836: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 336 KiB/s wr, 9 op/s 2023-03-09T20:20:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:49 smithi181 ceph-mon[94189]: pgmap v10836: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 336 KiB/s wr, 9 op/s 2023-03-09T20:20:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:49 smithi038 ceph-mon[92422]: pgmap v10836: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 336 KiB/s wr, 9 op/s 2023-03-09T20:20:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19361.241106 secs 2023-03-09T20:20:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19361.241106 secs 2023-03-09T20:20:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19361.241106 secs 2023-03-09T20:20:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:51 smithi122 ceph-mon[93869]: pgmap v10837: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 336 KiB/s wr, 9 op/s 2023-03-09T20:20:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:20:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:20:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:51 smithi181 ceph-mon[94189]: pgmap v10837: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 336 KiB/s wr, 9 op/s 2023-03-09T20:20:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:20:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:20:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:51 smithi038 ceph-mon[92422]: pgmap v10837: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 336 KiB/s wr, 9 op/s 2023-03-09T20:20:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:20:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:20:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:20:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:20:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:20:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:20:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:20:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:20:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19825.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:20:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19363.565212 secs 2023-03-09T20:20:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:53 smithi122 ceph-mon[93869]: pgmap v10838: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 623 KiB/s rd, 495 KiB/s wr, 13 op/s 2023-03-09T20:20:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19363.565212 secs 2023-03-09T20:20:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:53 smithi181 ceph-mon[94189]: pgmap v10838: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 623 KiB/s rd, 495 KiB/s wr, 13 op/s 2023-03-09T20:20:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19363.565212 secs 2023-03-09T20:20:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:53 smithi038 ceph-mon[92422]: pgmap v10838: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 623 KiB/s rd, 495 KiB/s wr, 13 op/s 2023-03-09T20:20:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:55 smithi122 ceph-mon[93869]: pgmap v10839: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 288 KiB/s wr, 10 op/s 2023-03-09T20:20:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19366.241167 secs 2023-03-09T20:20:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:55 smithi181 ceph-mon[94189]: pgmap v10839: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 288 KiB/s wr, 10 op/s 2023-03-09T20:20:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19366.241167 secs 2023-03-09T20:20:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:55 smithi038 ceph-mon[92422]: pgmap v10839: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 288 KiB/s wr, 10 op/s 2023-03-09T20:20:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19366.241167 secs 2023-03-09T20:20:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:56 smithi038 ceph-mon[92422]: pgmap v10840: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 289 KiB/s wr, 12 op/s 2023-03-09T20:20:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:56 smithi122 ceph-mon[93869]: pgmap v10840: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 289 KiB/s wr, 12 op/s 2023-03-09T20:20:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:56 smithi181 ceph-mon[94189]: pgmap v10840: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 289 KiB/s wr, 12 op/s 2023-03-09T20:20:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19368.565324 secs 2023-03-09T20:20:58.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19368.565324 secs 2023-03-09T20:20:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19368.565324 secs 2023-03-09T20:20:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19830.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:21:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:59 smithi122 ceph-mon[93869]: pgmap v10841: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 359 KiB/s wr, 11 op/s 2023-03-09T20:21:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:21:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:20:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:21:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:59 smithi181 ceph-mon[94189]: pgmap v10841: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 359 KiB/s wr, 11 op/s 2023-03-09T20:21:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:21:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:20:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:21:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:59 smithi038 ceph-mon[92422]: pgmap v10841: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 359 KiB/s wr, 11 op/s 2023-03-09T20:21:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:21:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:20:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:21:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19371.241239 secs 2023-03-09T20:21:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:21:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:21:01.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19371.241239 secs 2023-03-09T20:21:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:21:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:21:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19371.241239 secs 2023-03-09T20:21:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:21:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:21:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:01 smithi122 ceph-mon[93869]: pgmap v10842: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 440 KiB/s rd, 439 KiB/s wr, 10 op/s 2023-03-09T20:21:02.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:01 smithi181 ceph-mon[94189]: pgmap v10842: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 440 KiB/s rd, 439 KiB/s wr, 10 op/s 2023-03-09T20:21:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:01 smithi038 ceph-mon[92422]: pgmap v10842: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 440 KiB/s rd, 439 KiB/s wr, 10 op/s 2023-03-09T20:21:03.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19835.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:21:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19373.565461 secs 2023-03-09T20:21:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:03 smithi122 ceph-mon[93869]: pgmap v10843: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 492 KiB/s wr, 15 op/s 2023-03-09T20:21:04.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19373.565461 secs 2023-03-09T20:21:04.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:03 smithi181 ceph-mon[94189]: pgmap v10843: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 492 KiB/s wr, 15 op/s 2023-03-09T20:21:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19373.565461 secs 2023-03-09T20:21:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:03 smithi038 ceph-mon[92422]: pgmap v10843: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 492 KiB/s wr, 15 op/s 2023-03-09T20:21:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:05 smithi122 ceph-mon[93869]: pgmap v10844: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 502 KiB/s wr, 25 op/s 2023-03-09T20:21:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19376.241296 secs 2023-03-09T20:21:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:05 smithi122 ceph-mon[93869]: osdmap e891: 12 total, 12 up, 12 in 2023-03-09T20:21:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:05 smithi181 ceph-mon[94189]: pgmap v10844: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 502 KiB/s wr, 25 op/s 2023-03-09T20:21:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19376.241296 secs 2023-03-09T20:21:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:05 smithi181 ceph-mon[94189]: osdmap e891: 12 total, 12 up, 12 in 2023-03-09T20:21:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:05 smithi038 ceph-mon[92422]: pgmap v10844: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 502 KiB/s wr, 25 op/s 2023-03-09T20:21:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19376.241296 secs 2023-03-09T20:21:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:05 smithi038 ceph-mon[92422]: osdmap e891: 12 total, 12 up, 12 in 2023-03-09T20:21:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:07 smithi122 ceph-mon[93869]: pgmap v10846: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 561 KiB/s rd, 603 KiB/s wr, 30 op/s 2023-03-09T20:21:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:07 smithi181 ceph-mon[94189]: pgmap v10846: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 561 KiB/s rd, 603 KiB/s wr, 30 op/s 2023-03-09T20:21:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:07 smithi038 ceph-mon[92422]: pgmap v10846: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 561 KiB/s rd, 603 KiB/s wr, 30 op/s 2023-03-09T20:21:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19840.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:21:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19378.565556 secs 2023-03-09T20:21:09.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19378.565556 secs 2023-03-09T20:21:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19378.565556 secs 2023-03-09T20:21:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:09 smithi122 ceph-mon[93869]: pgmap v10847: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 500 KiB/s wr, 28 op/s 2023-03-09T20:21:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:09 smithi181 ceph-mon[94189]: pgmap v10847: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 500 KiB/s wr, 28 op/s 2023-03-09T20:21:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:09 smithi038 ceph-mon[92422]: pgmap v10847: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 500 KiB/s wr, 28 op/s 2023-03-09T20:21:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19381.241356 secs 2023-03-09T20:21:11.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19381.241356 secs 2023-03-09T20:21:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19381.241356 secs 2023-03-09T20:21:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:11 smithi122 ceph-mon[93869]: pgmap v10848: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 502 KiB/s wr, 29 op/s 2023-03-09T20:21:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:11 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:21:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:11 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:21:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:11 smithi181 ceph-mon[94189]: pgmap v10848: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 502 KiB/s wr, 29 op/s 2023-03-09T20:21:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:11 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:21:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:11 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:21:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:11 smithi038 ceph-mon[92422]: pgmap v10848: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 502 KiB/s wr, 29 op/s 2023-03-09T20:21:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:11 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:21:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:11 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:21:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:21:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:21:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:21:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:21:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:21:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:21:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19845.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:21:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19383.565673 secs 2023-03-09T20:21:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:13 smithi122 ceph-mon[93869]: pgmap v10849: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 443 KiB/s rd, 459 KiB/s wr, 26 op/s 2023-03-09T20:21:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:13 smithi122 ceph-mon[93869]: osdmap e892: 12 total, 12 up, 12 in 2023-03-09T20:21:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19383.565673 secs 2023-03-09T20:21:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:13 smithi181 ceph-mon[94189]: pgmap v10849: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 443 KiB/s rd, 459 KiB/s wr, 26 op/s 2023-03-09T20:21:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:13 smithi181 ceph-mon[94189]: osdmap e892: 12 total, 12 up, 12 in 2023-03-09T20:21:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19383.565673 secs 2023-03-09T20:21:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:13 smithi038 ceph-mon[92422]: pgmap v10849: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 443 KiB/s rd, 459 KiB/s wr, 26 op/s 2023-03-09T20:21:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:13 smithi038 ceph-mon[92422]: osdmap e892: 12 total, 12 up, 12 in 2023-03-09T20:21:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:15 smithi122 ceph-mon[93869]: pgmap v10851: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 274 KiB/s wr, 11 op/s 2023-03-09T20:21:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19386.241415 secs 2023-03-09T20:21:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:15 smithi181 ceph-mon[94189]: pgmap v10851: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 274 KiB/s wr, 11 op/s 2023-03-09T20:21:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19386.241415 secs 2023-03-09T20:21:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:15 smithi038 ceph-mon[92422]: pgmap v10851: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 274 KiB/s wr, 11 op/s 2023-03-09T20:21:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19386.241415 secs 2023-03-09T20:21:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:17 smithi122 ceph-mon[93869]: pgmap v10852: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 256 KiB/s wr, 10 op/s 2023-03-09T20:21:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:17 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:21:18.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:17 smithi181 ceph-mon[94189]: pgmap v10852: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 256 KiB/s wr, 10 op/s 2023-03-09T20:21:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:17 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:21:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:17 smithi038 ceph-mon[92422]: pgmap v10852: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 256 KiB/s wr, 10 op/s 2023-03-09T20:21:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:17 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:21:18.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19850.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:21:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19388.565792 secs 2023-03-09T20:21:19.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19388.565792 secs 2023-03-09T20:21:19.234 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19388.565792 secs 2023-03-09T20:21:20.078 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:19 smithi038 ceph-mon[92422]: pgmap v10853: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 256 KiB/s wr, 10 op/s 2023-03-09T20:21:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:19 smithi181 ceph-mon[94189]: pgmap v10853: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 256 KiB/s wr, 10 op/s 2023-03-09T20:21:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:19 smithi122 ceph-mon[93869]: pgmap v10853: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 256 KiB/s wr, 10 op/s 2023-03-09T20:21:21.190 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19391.241477 secs 2023-03-09T20:21:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19391.241477 secs 2023-03-09T20:21:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19391.241477 secs 2023-03-09T20:21:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:21 smithi038 ceph-mon[92422]: pgmap v10854: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 91 KiB/s wr, 8 op/s 2023-03-09T20:21:22.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:21:22.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:21:22.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:21:22.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:21:22.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:21:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:21 smithi122 ceph-mon[93869]: pgmap v10854: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 91 KiB/s wr, 8 op/s 2023-03-09T20:21:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:21:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:21:22.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:21:22.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:21:22.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:21:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:21 smithi181 ceph-mon[94189]: pgmap v10854: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 91 KiB/s wr, 8 op/s 2023-03-09T20:21:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:21:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:21:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:21:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:21:22.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:21:23.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:22 smithi038 ceph-mon[92422]: pgmap v10855: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 72 KiB/s wr, 8 op/s 2023-03-09T20:21:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:22 smithi122 ceph-mon[93869]: pgmap v10855: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 72 KiB/s wr, 8 op/s 2023-03-09T20:21:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:22 smithi181 ceph-mon[94189]: pgmap v10855: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 72 KiB/s wr, 8 op/s 2023-03-09T20:21:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19855.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:21:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19393.565906 secs 2023-03-09T20:21:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19393.565906 secs 2023-03-09T20:21:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19393.565906 secs 2023-03-09T20:21:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:25 smithi122 ceph-mon[93869]: pgmap v10856: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 153 KiB/s wr, 8 op/s 2023-03-09T20:21:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:25 smithi181 ceph-mon[94189]: pgmap v10856: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 153 KiB/s wr, 8 op/s 2023-03-09T20:21:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:25 smithi038 ceph-mon[92422]: pgmap v10856: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 153 KiB/s wr, 8 op/s 2023-03-09T20:21:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19396.241537 secs 2023-03-09T20:21:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19396.241537 secs 2023-03-09T20:21:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19396.241537 secs 2023-03-09T20:21:27.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:27 smithi122 ceph-mon[93869]: pgmap v10857: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 132 KiB/s wr, 8 op/s 2023-03-09T20:21:27.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:27 smithi181 ceph-mon[94189]: pgmap v10857: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 132 KiB/s wr, 8 op/s 2023-03-09T20:21:27.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:27 smithi038 ceph-mon[92422]: pgmap v10857: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 132 KiB/s wr, 8 op/s 2023-03-09T20:21:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19398.566009 secs 2023-03-09T20:21:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19398.566009 secs 2023-03-09T20:21:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19398.566009 secs 2023-03-09T20:21:28.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19860.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:21:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:29 smithi122 ceph-mon[93869]: pgmap v10858: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 190 KiB/s wr, 7 op/s 2023-03-09T20:21:30.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:29 smithi181 ceph-mon[94189]: pgmap v10858: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 190 KiB/s wr, 7 op/s 2023-03-09T20:21:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:29 smithi038 ceph-mon[92422]: pgmap v10858: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 190 KiB/s wr, 7 op/s 2023-03-09T20:21:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19401.241608 secs 2023-03-09T20:21:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19401.241608 secs 2023-03-09T20:21:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19401.241608 secs 2023-03-09T20:21:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:31 smithi122 ceph-mon[93869]: pgmap v10859: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 207 KiB/s wr, 7 op/s 2023-03-09T20:21:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:31 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:21:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:31 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:21:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:31 smithi181 ceph-mon[94189]: pgmap v10859: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 207 KiB/s wr, 7 op/s 2023-03-09T20:21:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:31 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:21:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:31 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:21:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:31 smithi038 ceph-mon[92422]: pgmap v10859: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 207 KiB/s wr, 7 op/s 2023-03-09T20:21:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:31 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:21:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:31 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:21:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:32 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:21:33.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:32 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:21:33.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:32 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:21:33.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:32 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:21:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:32 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:21:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:32 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:21:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19865.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:21:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19403.566124 secs 2023-03-09T20:21:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:33 smithi122 ceph-mon[93869]: pgmap v10860: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 150 KiB/s wr, 10 op/s 2023-03-09T20:21:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19403.566124 secs 2023-03-09T20:21:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:33 smithi181 ceph-mon[94189]: pgmap v10860: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 150 KiB/s wr, 10 op/s 2023-03-09T20:21:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19403.566124 secs 2023-03-09T20:21:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:33 smithi038 ceph-mon[92422]: pgmap v10860: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 150 KiB/s wr, 10 op/s 2023-03-09T20:21:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:35 smithi122 ceph-mon[93869]: pgmap v10861: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 149 KiB/s wr, 8 op/s 2023-03-09T20:21:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19406.241669 secs 2023-03-09T20:21:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:35 smithi181 ceph-mon[94189]: pgmap v10861: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 149 KiB/s wr, 8 op/s 2023-03-09T20:21:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19406.241669 secs 2023-03-09T20:21:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:35 smithi038 ceph-mon[92422]: pgmap v10861: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 149 KiB/s wr, 8 op/s 2023-03-09T20:21:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19406.241669 secs 2023-03-09T20:21:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:37 smithi122 ceph-mon[93869]: pgmap v10862: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 77 KiB/s wr, 9 op/s 2023-03-09T20:21:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:37 smithi181 ceph-mon[94189]: pgmap v10862: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 77 KiB/s wr, 9 op/s 2023-03-09T20:21:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:37 smithi038 ceph-mon[92422]: pgmap v10862: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 77 KiB/s wr, 9 op/s 2023-03-09T20:21:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19870.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:21:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:39 smithi122 ceph-mon[93869]: pgmap v10863: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 108 KiB/s wr, 8 op/s 2023-03-09T20:21:40.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:39 smithi181 ceph-mon[94189]: pgmap v10863: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 108 KiB/s wr, 8 op/s 2023-03-09T20:21:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:39 smithi038 ceph-mon[92422]: pgmap v10863: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 108 KiB/s wr, 8 op/s 2023-03-09T20:21:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19411.241733 secs 2023-03-09T20:21:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:40 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:21:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19411.241733 secs 2023-03-09T20:21:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:40 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:21:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19411.241733 secs 2023-03-09T20:21:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:40 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:21:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:41 smithi122 ceph-mon[93869]: pgmap v10864: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 7 op/s 2023-03-09T20:21:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:41 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:21:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:21:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:21:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:41 smithi181 ceph-mon[94189]: pgmap v10864: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 7 op/s 2023-03-09T20:21:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:41 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:21:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:21:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:21:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:41 smithi038 ceph-mon[92422]: pgmap v10864: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 7 op/s 2023-03-09T20:21:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:41 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:21:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:21:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:21:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19875.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:21:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:43 smithi122 ceph-mon[93869]: pgmap v10865: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 231 KiB/s wr, 10 op/s 2023-03-09T20:21:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:43 smithi181 ceph-mon[94189]: pgmap v10865: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 231 KiB/s wr, 10 op/s 2023-03-09T20:21:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:43 smithi038 ceph-mon[92422]: pgmap v10865: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 231 KiB/s wr, 10 op/s 2023-03-09T20:21:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:45 smithi122 ceph-mon[93869]: pgmap v10866: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 231 KiB/s wr, 7 op/s 2023-03-09T20:21:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19416.241817 secs 2023-03-09T20:21:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:45 smithi181 ceph-mon[94189]: pgmap v10866: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 231 KiB/s wr, 7 op/s 2023-03-09T20:21:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19416.241817 secs 2023-03-09T20:21:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:45 smithi038 ceph-mon[92422]: pgmap v10866: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 231 KiB/s wr, 7 op/s 2023-03-09T20:21:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19416.241817 secs 2023-03-09T20:21:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:47 smithi122 ceph-mon[93869]: pgmap v10867: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 246 KiB/s wr, 10 op/s 2023-03-09T20:21:48.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:47 smithi181 ceph-mon[94189]: pgmap v10867: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 246 KiB/s wr, 10 op/s 2023-03-09T20:21:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:47 smithi038 ceph-mon[92422]: pgmap v10867: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 246 KiB/s wr, 10 op/s 2023-03-09T20:21:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19880.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:21:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:49 smithi122 ceph-mon[93869]: pgmap v10868: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 324 KiB/s wr, 9 op/s 2023-03-09T20:21:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:49 smithi181 ceph-mon[94189]: pgmap v10868: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 324 KiB/s wr, 9 op/s 2023-03-09T20:21:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:49 smithi038 ceph-mon[92422]: pgmap v10868: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 324 KiB/s wr, 9 op/s 2023-03-09T20:21:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19421.241902 secs 2023-03-09T20:21:51.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19421.241902 secs 2023-03-09T20:21:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19421.241902 secs 2023-03-09T20:21:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:51 smithi122 ceph-mon[93869]: pgmap v10869: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 294 KiB/s wr, 8 op/s 2023-03-09T20:21:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:21:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:21:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:51 smithi181 ceph-mon[94189]: pgmap v10869: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 294 KiB/s wr, 8 op/s 2023-03-09T20:21:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:21:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:21:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:51 smithi038 ceph-mon[92422]: pgmap v10869: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 294 KiB/s wr, 8 op/s 2023-03-09T20:21:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:21:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:21:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:21:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:21:53.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:21:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:21:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:21:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:21:53.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19885.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:21:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:53 smithi038 ceph-mon[92422]: pgmap v10870: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 502 KiB/s wr, 13 op/s 2023-03-09T20:21:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:53 smithi122 ceph-mon[93869]: pgmap v10870: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 502 KiB/s wr, 13 op/s 2023-03-09T20:21:54.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:53 smithi181 ceph-mon[94189]: pgmap v10870: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 502 KiB/s wr, 13 op/s 2023-03-09T20:21:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:55 smithi038 ceph-mon[92422]: pgmap v10871: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 303 KiB/s wr, 10 op/s 2023-03-09T20:21:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19426.241962 secs 2023-03-09T20:21:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:55 smithi122 ceph-mon[93869]: pgmap v10871: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 303 KiB/s wr, 10 op/s 2023-03-09T20:21:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19426.241962 secs 2023-03-09T20:21:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:55 smithi181 ceph-mon[94189]: pgmap v10871: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 303 KiB/s wr, 10 op/s 2023-03-09T20:21:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19426.241962 secs 2023-03-09T20:21:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:56 smithi038 ceph-mon[92422]: pgmap v10872: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 303 KiB/s wr, 12 op/s 2023-03-09T20:21:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:56 smithi122 ceph-mon[93869]: pgmap v10872: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 303 KiB/s wr, 12 op/s 2023-03-09T20:21:57.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:56 smithi181 ceph-mon[94189]: pgmap v10872: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 303 KiB/s wr, 12 op/s 2023-03-09T20:21:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19890.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:22:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:59 smithi122 ceph-mon[93869]: pgmap v10873: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 375 KiB/s wr, 10 op/s 2023-03-09T20:22:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:22:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:21:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:22:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:59 smithi181 ceph-mon[94189]: pgmap v10873: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 375 KiB/s wr, 10 op/s 2023-03-09T20:22:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:22:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:21:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:22:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:59 smithi038 ceph-mon[92422]: pgmap v10873: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 375 KiB/s wr, 10 op/s 2023-03-09T20:22:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:22:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:21:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:22:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19431.241983 secs 2023-03-09T20:22:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:22:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:22:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19431.241983 secs 2023-03-09T20:22:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:22:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:22:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19431.241983 secs 2023-03-09T20:22:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:22:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:22:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:01 smithi122 ceph-mon[93869]: pgmap v10874: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 427 KiB/s wr, 9 op/s 2023-03-09T20:22:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:01 smithi181 ceph-mon[94189]: pgmap v10874: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 427 KiB/s wr, 9 op/s 2023-03-09T20:22:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:01 smithi038 ceph-mon[92422]: pgmap v10874: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 427 KiB/s wr, 9 op/s 2023-03-09T20:22:03.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19895.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:22:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19433.566824 secs 2023-03-09T20:22:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:03 smithi122 ceph-mon[93869]: pgmap v10875: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 513 KiB/s rd, 522 KiB/s wr, 14 op/s 2023-03-09T20:22:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19433.566824 secs 2023-03-09T20:22:04.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:03 smithi181 ceph-mon[94189]: pgmap v10875: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 513 KiB/s rd, 522 KiB/s wr, 14 op/s 2023-03-09T20:22:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19433.566824 secs 2023-03-09T20:22:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:03 smithi038 ceph-mon[92422]: pgmap v10875: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 513 KiB/s rd, 522 KiB/s wr, 14 op/s 2023-03-09T20:22:05.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:04 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:22:05.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:04 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:22:05.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:04 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:22:05.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:04 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:22:05.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:04 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:22:05.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:04 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:22:05.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:04 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:22:05.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:04 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:22:05.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:04 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:22:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:05 smithi122 ceph-mon[93869]: pgmap v10876: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 335 KiB/s wr, 21 op/s 2023-03-09T20:22:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19436.242045 secs 2023-03-09T20:22:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:05 smithi181 ceph-mon[94189]: pgmap v10876: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 335 KiB/s wr, 21 op/s 2023-03-09T20:22:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19436.242045 secs 2023-03-09T20:22:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:05 smithi038 ceph-mon[92422]: pgmap v10876: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 335 KiB/s wr, 21 op/s 2023-03-09T20:22:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19436.242045 secs 2023-03-09T20:22:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:07 smithi122 ceph-mon[93869]: pgmap v10877: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 434 KiB/s wr, 26 op/s 2023-03-09T20:22:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:07 smithi181 ceph-mon[94189]: pgmap v10877: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 434 KiB/s wr, 26 op/s 2023-03-09T20:22:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:07 smithi038 ceph-mon[92422]: pgmap v10877: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 434 KiB/s wr, 26 op/s 2023-03-09T20:22:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19900.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:22:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19438.566944 secs 2023-03-09T20:22:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19438.566944 secs 2023-03-09T20:22:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19438.566944 secs 2023-03-09T20:22:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:09 smithi038 ceph-mon[92422]: pgmap v10878: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 434 KiB/s wr, 24 op/s 2023-03-09T20:22:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:09 smithi038 ceph-mon[92422]: osdmap e893: 12 total, 12 up, 12 in 2023-03-09T20:22:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:09 smithi122 ceph-mon[93869]: pgmap v10878: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 434 KiB/s wr, 24 op/s 2023-03-09T20:22:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:09 smithi122 ceph-mon[93869]: osdmap e893: 12 total, 12 up, 12 in 2023-03-09T20:22:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:09 smithi181 ceph-mon[94189]: pgmap v10878: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 434 KiB/s wr, 24 op/s 2023-03-09T20:22:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:09 smithi181 ceph-mon[94189]: osdmap e893: 12 total, 12 up, 12 in 2023-03-09T20:22:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19441.242130 secs 2023-03-09T20:22:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:10 smithi122 ceph-mon[93869]: pgmap v10880: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 373 KiB/s wr, 28 op/s 2023-03-09T20:22:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19441.242130 secs 2023-03-09T20:22:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:10 smithi181 ceph-mon[94189]: pgmap v10880: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 373 KiB/s wr, 28 op/s 2023-03-09T20:22:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19441.242130 secs 2023-03-09T20:22:11.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:10 smithi038 ceph-mon[92422]: pgmap v10880: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 373 KiB/s wr, 28 op/s 2023-03-09T20:22:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19443.567060 secs 2023-03-09T20:22:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:13 smithi122 ceph-mon[93869]: pgmap v10881: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 278 KiB/s wr, 26 op/s 2023-03-09T20:22:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19443.567060 secs 2023-03-09T20:22:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:13 smithi181 ceph-mon[94189]: pgmap v10881: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 278 KiB/s wr, 26 op/s 2023-03-09T20:22:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19443.567060 secs 2023-03-09T20:22:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:13 smithi038 ceph-mon[92422]: pgmap v10881: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 278 KiB/s wr, 26 op/s 2023-03-09T20:22:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19905.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:22:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:15 smithi122 ceph-mon[93869]: pgmap v10882: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 253 KiB/s wr, 11 op/s 2023-03-09T20:22:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19446.242190 secs 2023-03-09T20:22:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:15 smithi181 ceph-mon[94189]: pgmap v10882: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 253 KiB/s wr, 11 op/s 2023-03-09T20:22:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19446.242190 secs 2023-03-09T20:22:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:15 smithi038 ceph-mon[92422]: pgmap v10882: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 253 KiB/s wr, 11 op/s 2023-03-09T20:22:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19446.242190 secs 2023-03-09T20:22:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:17 smithi122 ceph-mon[93869]: pgmap v10883: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 186 KiB/s wr, 10 op/s 2023-03-09T20:22:18.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:17 smithi181 ceph-mon[94189]: pgmap v10883: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 186 KiB/s wr, 10 op/s 2023-03-09T20:22:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:17 smithi038 ceph-mon[92422]: pgmap v10883: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 186 KiB/s wr, 10 op/s 2023-03-09T20:22:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19910.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:22:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19448.567182 secs 2023-03-09T20:22:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:18 smithi122 ceph-mon[93869]: osdmap e894: 12 total, 12 up, 12 in 2023-03-09T20:22:19.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19448.567182 secs 2023-03-09T20:22:19.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:18 smithi181 ceph-mon[94189]: osdmap e894: 12 total, 12 up, 12 in 2023-03-09T20:22:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19448.567182 secs 2023-03-09T20:22:19.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:18 smithi038 ceph-mon[92422]: osdmap e894: 12 total, 12 up, 12 in 2023-03-09T20:22:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:19 smithi122 ceph-mon[93869]: pgmap v10884: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 187 KiB/s wr, 9 op/s 2023-03-09T20:22:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:19 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:22:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:19 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:22:20.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:19 smithi181 ceph-mon[94189]: pgmap v10884: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 187 KiB/s wr, 9 op/s 2023-03-09T20:22:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:19 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:22:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:19 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:22:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:19 smithi038 ceph-mon[92422]: pgmap v10884: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 187 KiB/s wr, 9 op/s 2023-03-09T20:22:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:19 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:22:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:19 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:22:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19451.242245 secs 2023-03-09T20:22:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:22:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:22:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19451.242245 secs 2023-03-09T20:22:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:22:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:22:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19451.242245 secs 2023-03-09T20:22:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:22:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:22:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:21 smithi122 ceph-mon[93869]: pgmap v10886: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 98 KiB/s wr, 9 op/s 2023-03-09T20:22:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:22:22.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:21 smithi181 ceph-mon[94189]: pgmap v10886: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 98 KiB/s wr, 9 op/s 2023-03-09T20:22:22.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:22:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:21 smithi038 ceph-mon[92422]: pgmap v10886: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 98 KiB/s wr, 9 op/s 2023-03-09T20:22:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:22:23.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19915.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:22:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19453.567244 secs 2023-03-09T20:22:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:23 smithi122 ceph-mon[93869]: pgmap v10887: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 80 KiB/s wr, 9 op/s 2023-03-09T20:22:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19453.567244 secs 2023-03-09T20:22:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:23 smithi181 ceph-mon[94189]: pgmap v10887: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 80 KiB/s wr, 9 op/s 2023-03-09T20:22:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19453.567244 secs 2023-03-09T20:22:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:23 smithi038 ceph-mon[92422]: pgmap v10887: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 80 KiB/s wr, 9 op/s 2023-03-09T20:22:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:25 smithi122 ceph-mon[93869]: pgmap v10888: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 166 KiB/s wr, 9 op/s 2023-03-09T20:22:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19456.242305 secs 2023-03-09T20:22:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:22:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:22:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:25 smithi181 ceph-mon[94189]: pgmap v10888: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 166 KiB/s wr, 9 op/s 2023-03-09T20:22:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19456.242305 secs 2023-03-09T20:22:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:22:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:22:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:25 smithi038 ceph-mon[92422]: pgmap v10888: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 166 KiB/s wr, 9 op/s 2023-03-09T20:22:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19456.242305 secs 2023-03-09T20:22:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:22:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:22:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:27 smithi122 ceph-mon[93869]: pgmap v10889: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 114 KiB/s wr, 8 op/s 2023-03-09T20:22:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:22:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:22:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:22:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:22:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:22:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:27 smithi181 ceph-mon[94189]: pgmap v10889: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 114 KiB/s wr, 8 op/s 2023-03-09T20:22:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:22:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:22:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:22:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:22:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:22:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:27 smithi038 ceph-mon[92422]: pgmap v10889: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 114 KiB/s wr, 8 op/s 2023-03-09T20:22:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:22:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:22:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:22:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:22:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:22:28.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19920.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:22:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19458.567363 secs 2023-03-09T20:22:29.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19458.567363 secs 2023-03-09T20:22:29.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19458.567363 secs 2023-03-09T20:22:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:29 smithi122 ceph-mon[93869]: pgmap v10890: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 113 KiB/s wr, 8 op/s 2023-03-09T20:22:30.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:29 smithi181 ceph-mon[94189]: pgmap v10890: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 113 KiB/s wr, 8 op/s 2023-03-09T20:22:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:29 smithi038 ceph-mon[92422]: pgmap v10890: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 113 KiB/s wr, 8 op/s 2023-03-09T20:22:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19461.242364 secs 2023-03-09T20:22:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19461.242364 secs 2023-03-09T20:22:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19461.242364 secs 2023-03-09T20:22:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:31 smithi122 ceph-mon[93869]: pgmap v10891: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 133 KiB/s wr, 8 op/s 2023-03-09T20:22:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:31 smithi181 ceph-mon[94189]: pgmap v10891: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 133 KiB/s wr, 8 op/s 2023-03-09T20:22:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:31 smithi038 ceph-mon[92422]: pgmap v10891: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 133 KiB/s wr, 8 op/s 2023-03-09T20:22:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19925.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:22:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19463.567498 secs 2023-03-09T20:22:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:33 smithi122 ceph-mon[93869]: pgmap v10892: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 105 KiB/s wr, 9 op/s 2023-03-09T20:22:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19463.567498 secs 2023-03-09T20:22:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:33 smithi181 ceph-mon[94189]: pgmap v10892: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 105 KiB/s wr, 9 op/s 2023-03-09T20:22:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19463.567498 secs 2023-03-09T20:22:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:33 smithi038 ceph-mon[92422]: pgmap v10892: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 105 KiB/s wr, 9 op/s 2023-03-09T20:22:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:35 smithi122 ceph-mon[93869]: pgmap v10893: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 104 KiB/s wr, 7 op/s 2023-03-09T20:22:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19466.242444 secs 2023-03-09T20:22:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:35 smithi181 ceph-mon[94189]: pgmap v10893: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 104 KiB/s wr, 7 op/s 2023-03-09T20:22:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19466.242444 secs 2023-03-09T20:22:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:35 smithi038 ceph-mon[92422]: pgmap v10893: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 104 KiB/s wr, 7 op/s 2023-03-09T20:22:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19466.242444 secs 2023-03-09T20:22:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:37 smithi122 ceph-mon[93869]: pgmap v10894: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 33 KiB/s wr, 9 op/s 2023-03-09T20:22:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:37 smithi181 ceph-mon[94189]: pgmap v10894: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 33 KiB/s wr, 9 op/s 2023-03-09T20:22:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:37 smithi038 ceph-mon[92422]: pgmap v10894: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 33 KiB/s wr, 9 op/s 2023-03-09T20:22:38.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19930.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:22:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:38 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 19468.567588 secs 2023-03-09T20:22:39.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:38 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 19468.567588 secs 2023-03-09T20:22:39.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:38 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 19468.567588 secs 2023-03-09T20:22:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:39 smithi122 ceph-mon[93869]: pgmap v10895: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 32 KiB/s wr, 6 op/s 2023-03-09T20:22:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:39 smithi181 ceph-mon[94189]: pgmap v10895: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 32 KiB/s wr, 6 op/s 2023-03-09T20:22:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:39 smithi038 ceph-mon[92422]: pgmap v10895: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 32 KiB/s wr, 6 op/s 2023-03-09T20:22:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19471.242503 secs 2023-03-09T20:22:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19471.242503 secs 2023-03-09T20:22:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19471.242503 secs 2023-03-09T20:22:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:41 smithi038 ceph-mon[92422]: pgmap v10896: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 192 KiB/s wr, 7 op/s 2023-03-09T20:22:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:22:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:22:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:41 smithi122 ceph-mon[93869]: pgmap v10896: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 192 KiB/s wr, 7 op/s 2023-03-09T20:22:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:22:42.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:22:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:41 smithi181 ceph-mon[94189]: pgmap v10896: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 192 KiB/s wr, 7 op/s 2023-03-09T20:22:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:22:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:22:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19935.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:22:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:43 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 19473.567686 secs 2023-03-09T20:22:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:43 smithi038 ceph-mon[92422]: pgmap v10897: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 176 KiB/s wr, 9 op/s 2023-03-09T20:22:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:43 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 19473.567686 secs 2023-03-09T20:22:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:43 smithi122 ceph-mon[93869]: pgmap v10897: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 176 KiB/s wr, 9 op/s 2023-03-09T20:22:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:43 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 19473.567686 secs 2023-03-09T20:22:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:43 smithi181 ceph-mon[94189]: pgmap v10897: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 176 KiB/s wr, 9 op/s 2023-03-09T20:22:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:45 smithi038 ceph-mon[92422]: pgmap v10898: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 234 KiB/s wr, 8 op/s 2023-03-09T20:22:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19476.242587 secs 2023-03-09T20:22:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:45 smithi122 ceph-mon[93869]: pgmap v10898: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 234 KiB/s wr, 8 op/s 2023-03-09T20:22:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19476.242587 secs 2023-03-09T20:22:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:45 smithi181 ceph-mon[94189]: pgmap v10898: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 234 KiB/s wr, 8 op/s 2023-03-09T20:22:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19476.242587 secs 2023-03-09T20:22:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:47 smithi122 ceph-mon[93869]: pgmap v10899: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 250 KiB/s wr, 10 op/s 2023-03-09T20:22:47.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:47 smithi181 ceph-mon[94189]: pgmap v10899: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 250 KiB/s wr, 10 op/s 2023-03-09T20:22:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:47 smithi038 ceph-mon[92422]: pgmap v10899: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 250 KiB/s wr, 10 op/s 2023-03-09T20:22:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19478.567806 secs 2023-03-09T20:22:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19478.567806 secs 2023-03-09T20:22:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19478.567806 secs 2023-03-09T20:22:48.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19940.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:22:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:49 smithi122 ceph-mon[93869]: pgmap v10900: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 349 KiB/s wr, 9 op/s 2023-03-09T20:22:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:22:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:22:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:49 smithi181 ceph-mon[94189]: pgmap v10900: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 349 KiB/s wr, 9 op/s 2023-03-09T20:22:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:22:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:22:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:49 smithi038 ceph-mon[92422]: pgmap v10900: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 349 KiB/s wr, 9 op/s 2023-03-09T20:22:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:22:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:22:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19481.242648 secs 2023-03-09T20:22:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19481.242648 secs 2023-03-09T20:22:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19481.242648 secs 2023-03-09T20:22:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:51 smithi122 ceph-mon[93869]: pgmap v10901: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 555 KiB/s rd, 349 KiB/s wr, 10 op/s 2023-03-09T20:22:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:22:52.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:22:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:51 smithi181 ceph-mon[94189]: pgmap v10901: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 555 KiB/s rd, 349 KiB/s wr, 10 op/s 2023-03-09T20:22:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:22:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:22:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:51 smithi038 ceph-mon[92422]: pgmap v10901: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 555 KiB/s rd, 349 KiB/s wr, 10 op/s 2023-03-09T20:22:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:22:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:22:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:22:53.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:22:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:22:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:22:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:22:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:22:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19945.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:22:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19483.567920 secs 2023-03-09T20:22:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:53 smithi122 ceph-mon[93869]: pgmap v10902: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 644 KiB/s rd, 437 KiB/s wr, 13 op/s 2023-03-09T20:22:54.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19483.567920 secs 2023-03-09T20:22:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:53 smithi181 ceph-mon[94189]: pgmap v10902: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 644 KiB/s rd, 437 KiB/s wr, 13 op/s 2023-03-09T20:22:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19483.567920 secs 2023-03-09T20:22:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:53 smithi038 ceph-mon[92422]: pgmap v10902: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 644 KiB/s rd, 437 KiB/s wr, 13 op/s 2023-03-09T20:22:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:55 smithi122 ceph-mon[93869]: pgmap v10903: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 627 KiB/s rd, 421 KiB/s wr, 11 op/s 2023-03-09T20:22:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19486.242711 secs 2023-03-09T20:22:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:55 smithi181 ceph-mon[94189]: pgmap v10903: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 627 KiB/s rd, 421 KiB/s wr, 11 op/s 2023-03-09T20:22:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19486.242711 secs 2023-03-09T20:22:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:55 smithi038 ceph-mon[92422]: pgmap v10903: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 627 KiB/s rd, 421 KiB/s wr, 11 op/s 2023-03-09T20:22:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19486.242711 secs 2023-03-09T20:22:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:57 smithi122 ceph-mon[93869]: pgmap v10904: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 422 KiB/s rd, 380 KiB/s wr, 13 op/s 2023-03-09T20:22:58.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:57 smithi181 ceph-mon[94189]: pgmap v10904: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 422 KiB/s rd, 380 KiB/s wr, 13 op/s 2023-03-09T20:22:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:57 smithi038 ceph-mon[92422]: pgmap v10904: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 422 KiB/s rd, 380 KiB/s wr, 13 op/s 2023-03-09T20:22:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19950.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:22:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19488.568058 secs 2023-03-09T20:22:59.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19488.568058 secs 2023-03-09T20:22:59.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19488.568058 secs 2023-03-09T20:23:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:59 smithi122 ceph-mon[93869]: pgmap v10905: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 452 KiB/s wr, 11 op/s 2023-03-09T20:23:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:23:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:22:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:23:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:59 smithi181 ceph-mon[94189]: pgmap v10905: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 452 KiB/s wr, 11 op/s 2023-03-09T20:23:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:23:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:22:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:23:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:59 smithi038 ceph-mon[92422]: pgmap v10905: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 452 KiB/s wr, 11 op/s 2023-03-09T20:23:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:23:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:22:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:23:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19491.242773 secs 2023-03-09T20:23:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:23:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:23:01.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19491.242773 secs 2023-03-09T20:23:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:23:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:23:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19491.242773 secs 2023-03-09T20:23:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:23:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:23:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:01 smithi122 ceph-mon[93869]: pgmap v10906: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 353 KiB/s wr, 10 op/s 2023-03-09T20:23:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:01 smithi181 ceph-mon[94189]: pgmap v10906: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 353 KiB/s wr, 10 op/s 2023-03-09T20:23:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:01 smithi038 ceph-mon[92422]: pgmap v10906: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 353 KiB/s wr, 10 op/s 2023-03-09T20:23:03.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19955.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:23:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19493.568160 secs 2023-03-09T20:23:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:03 smithi122 ceph-mon[93869]: pgmap v10907: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 514 KiB/s wr, 14 op/s 2023-03-09T20:23:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19493.568160 secs 2023-03-09T20:23:04.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:03 smithi181 ceph-mon[94189]: pgmap v10907: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 514 KiB/s wr, 14 op/s 2023-03-09T20:23:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19493.568160 secs 2023-03-09T20:23:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:03 smithi038 ceph-mon[92422]: pgmap v10907: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 514 KiB/s wr, 14 op/s 2023-03-09T20:23:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:05 smithi122 ceph-mon[93869]: pgmap v10908: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 288 KiB/s wr, 22 op/s 2023-03-09T20:23:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19496.242833 secs 2023-03-09T20:23:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:05 smithi181 ceph-mon[94189]: pgmap v10908: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 288 KiB/s wr, 22 op/s 2023-03-09T20:23:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19496.242833 secs 2023-03-09T20:23:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:05 smithi038 ceph-mon[92422]: pgmap v10908: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 288 KiB/s wr, 22 op/s 2023-03-09T20:23:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19496.242833 secs 2023-03-09T20:23:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:07 smithi122 ceph-mon[93869]: pgmap v10909: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 310 KiB/s wr, 26 op/s 2023-03-09T20:23:08.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:07 smithi181 ceph-mon[94189]: pgmap v10909: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 310 KiB/s wr, 26 op/s 2023-03-09T20:23:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:07 smithi038 ceph-mon[92422]: pgmap v10909: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 310 KiB/s wr, 26 op/s 2023-03-09T20:23:08.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19960.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:23:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19498.568281 secs 2023-03-09T20:23:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19498.568281 secs 2023-03-09T20:23:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19498.568281 secs 2023-03-09T20:23:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:09 smithi038 ceph-mon[92422]: pgmap v10910: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 401 KiB/s wr, 25 op/s 2023-03-09T20:23:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:09 smithi038 ceph-mon[92422]: osdmap e895: 12 total, 12 up, 12 in 2023-03-09T20:23:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:09 smithi122 ceph-mon[93869]: pgmap v10910: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 401 KiB/s wr, 25 op/s 2023-03-09T20:23:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:09 smithi122 ceph-mon[93869]: osdmap e895: 12 total, 12 up, 12 in 2023-03-09T20:23:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:09 smithi181 ceph-mon[94189]: pgmap v10910: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 401 KiB/s wr, 25 op/s 2023-03-09T20:23:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:09 smithi181 ceph-mon[94189]: osdmap e895: 12 total, 12 up, 12 in 2023-03-09T20:23:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19501.242893 secs 2023-03-09T20:23:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19501.242893 secs 2023-03-09T20:23:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19501.242893 secs 2023-03-09T20:23:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:11 smithi038 ceph-mon[92422]: pgmap v10912: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 375 KiB/s wr, 28 op/s 2023-03-09T20:23:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:11 smithi122 ceph-mon[93869]: pgmap v10912: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 375 KiB/s wr, 28 op/s 2023-03-09T20:23:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:11 smithi181 ceph-mon[94189]: pgmap v10912: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 375 KiB/s wr, 28 op/s 2023-03-09T20:23:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19965.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:23:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19503.568413 secs 2023-03-09T20:23:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:13 smithi038 ceph-mon[92422]: pgmap v10913: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 200 KiB/s wr, 26 op/s 2023-03-09T20:23:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19503.568413 secs 2023-03-09T20:23:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:13 smithi122 ceph-mon[93869]: pgmap v10913: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 200 KiB/s wr, 26 op/s 2023-03-09T20:23:14.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19503.568413 secs 2023-03-09T20:23:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:13 smithi181 ceph-mon[94189]: pgmap v10913: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 200 KiB/s wr, 26 op/s 2023-03-09T20:23:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:15 smithi038 ceph-mon[92422]: pgmap v10914: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 174 KiB/s wr, 11 op/s 2023-03-09T20:23:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19506.242952 secs 2023-03-09T20:23:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:15 smithi122 ceph-mon[93869]: pgmap v10914: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 174 KiB/s wr, 11 op/s 2023-03-09T20:23:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19506.242952 secs 2023-03-09T20:23:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:15 smithi181 ceph-mon[94189]: pgmap v10914: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 174 KiB/s wr, 11 op/s 2023-03-09T20:23:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19506.242952 secs 2023-03-09T20:23:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:17 smithi122 ceph-mon[93869]: pgmap v10915: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 188 KiB/s wr, 9 op/s 2023-03-09T20:23:17.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:17 smithi181 ceph-mon[94189]: pgmap v10915: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 188 KiB/s wr, 9 op/s 2023-03-09T20:23:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:17 smithi038 ceph-mon[92422]: pgmap v10915: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 188 KiB/s wr, 9 op/s 2023-03-09T20:23:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19508.568526 secs 2023-03-09T20:23:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19508.568526 secs 2023-03-09T20:23:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19508.568526 secs 2023-03-09T20:23:18.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19970.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:23:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:19 smithi122 ceph-mon[93869]: pgmap v10916: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 77 KiB/s wr, 8 op/s 2023-03-09T20:23:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:19 smithi122 ceph-mon[93869]: osdmap e896: 12 total, 12 up, 12 in 2023-03-09T20:23:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:19 smithi181 ceph-mon[94189]: pgmap v10916: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 77 KiB/s wr, 8 op/s 2023-03-09T20:23:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:19 smithi181 ceph-mon[94189]: osdmap e896: 12 total, 12 up, 12 in 2023-03-09T20:23:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:19 smithi038 ceph-mon[92422]: pgmap v10916: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 77 KiB/s wr, 8 op/s 2023-03-09T20:23:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:19 smithi038 ceph-mon[92422]: osdmap e896: 12 total, 12 up, 12 in 2023-03-09T20:23:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19511.242984 secs 2023-03-09T20:23:21.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19511.242984 secs 2023-03-09T20:23:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19511.242984 secs 2023-03-09T20:23:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:21 smithi122 ceph-mon[93869]: pgmap v10918: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 77 KiB/s wr, 8 op/s 2023-03-09T20:23:22.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:21 smithi181 ceph-mon[94189]: pgmap v10918: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 77 KiB/s wr, 8 op/s 2023-03-09T20:23:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:21 smithi038 ceph-mon[92422]: pgmap v10918: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 77 KiB/s wr, 8 op/s 2023-03-09T20:23:23.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19975.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:23:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19513.568648 secs 2023-03-09T20:23:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:23 smithi122 ceph-mon[93869]: pgmap v10919: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 8 op/s 2023-03-09T20:23:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19513.568648 secs 2023-03-09T20:23:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:23 smithi181 ceph-mon[94189]: pgmap v10919: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 8 op/s 2023-03-09T20:23:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19513.568648 secs 2023-03-09T20:23:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:23 smithi038 ceph-mon[92422]: pgmap v10919: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 8 op/s 2023-03-09T20:23:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:25 smithi122 ceph-mon[93869]: pgmap v10920: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 8 op/s 2023-03-09T20:23:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19516.243043 secs 2023-03-09T20:23:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:25 smithi181 ceph-mon[94189]: pgmap v10920: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 8 op/s 2023-03-09T20:23:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19516.243043 secs 2023-03-09T20:23:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:25 smithi038 ceph-mon[92422]: pgmap v10920: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 8 op/s 2023-03-09T20:23:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19516.243043 secs 2023-03-09T20:23:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:27 smithi122 ceph-mon[93869]: pgmap v10921: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 105 KiB/s wr, 8 op/s 2023-03-09T20:23:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:27 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:23:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:27 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:23:28.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:27 smithi181 ceph-mon[94189]: pgmap v10921: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 105 KiB/s wr, 8 op/s 2023-03-09T20:23:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:27 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:23:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:27 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:23:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:27 smithi038 ceph-mon[92422]: pgmap v10921: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 105 KiB/s wr, 8 op/s 2023-03-09T20:23:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:27 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:23:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:27 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:23:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19980.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:23:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19518.568782 secs 2023-03-09T20:23:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:28 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:23:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:28 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:23:29.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:23:29.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:23:29.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:23:29.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:23:29.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19518.568782 secs 2023-03-09T20:23:29.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:28 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:23:29.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:28 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:23:29.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:23:29.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:23:29.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:23:29.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:23:29.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19518.568782 secs 2023-03-09T20:23:29.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:28 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:23:29.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:28 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:23:29.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:23:29.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:23:29.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:23:29.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:23:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:29 smithi122 ceph-mon[93869]: pgmap v10922: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 89 KiB/s wr, 8 op/s 2023-03-09T20:23:30.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:29 smithi181 ceph-mon[94189]: pgmap v10922: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 89 KiB/s wr, 8 op/s 2023-03-09T20:23:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:29 smithi038 ceph-mon[92422]: pgmap v10922: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 89 KiB/s wr, 8 op/s 2023-03-09T20:23:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19521.243104 secs 2023-03-09T20:23:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19521.243104 secs 2023-03-09T20:23:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19521.243104 secs 2023-03-09T20:23:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:31 smithi122 ceph-mon[93869]: pgmap v10923: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 78 KiB/s wr, 7 op/s 2023-03-09T20:23:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:31 smithi181 ceph-mon[94189]: pgmap v10923: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 78 KiB/s wr, 7 op/s 2023-03-09T20:23:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:31 smithi038 ceph-mon[92422]: pgmap v10923: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 78 KiB/s wr, 7 op/s 2023-03-09T20:23:33.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19985.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:23:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19523.568897 secs 2023-03-09T20:23:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:33 smithi122 ceph-mon[93869]: pgmap v10924: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 74 KiB/s wr, 10 op/s 2023-03-09T20:23:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19523.568897 secs 2023-03-09T20:23:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:33 smithi181 ceph-mon[94189]: pgmap v10924: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 74 KiB/s wr, 10 op/s 2023-03-09T20:23:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19523.568897 secs 2023-03-09T20:23:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:33 smithi038 ceph-mon[92422]: pgmap v10924: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 74 KiB/s wr, 10 op/s 2023-03-09T20:23:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:35 smithi122 ceph-mon[93869]: pgmap v10925: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 73 KiB/s wr, 7 op/s 2023-03-09T20:23:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19526.243163 secs 2023-03-09T20:23:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:35 smithi181 ceph-mon[94189]: pgmap v10925: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 73 KiB/s wr, 7 op/s 2023-03-09T20:23:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19526.243163 secs 2023-03-09T20:23:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:35 smithi038 ceph-mon[92422]: pgmap v10925: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 73 KiB/s wr, 7 op/s 2023-03-09T20:23:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19526.243163 secs 2023-03-09T20:23:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:37 smithi122 ceph-mon[93869]: pgmap v10926: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 74 KiB/s wr, 10 op/s 2023-03-09T20:23:38.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:37 smithi181 ceph-mon[94189]: pgmap v10926: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 74 KiB/s wr, 10 op/s 2023-03-09T20:23:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:37 smithi038 ceph-mon[92422]: pgmap v10926: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 74 KiB/s wr, 10 op/s 2023-03-09T20:23:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19990.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:23:39.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19528.568969 secs 2023-03-09T20:23:39.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19528.568969 secs 2023-03-09T20:23:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19528.568969 secs 2023-03-09T20:23:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:39 smithi038 ceph-mon[92422]: pgmap v10927: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 938 B/s wr, 8 op/s 2023-03-09T20:23:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:39 smithi122 ceph-mon[93869]: pgmap v10927: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 938 B/s wr, 8 op/s 2023-03-09T20:23:40.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:39 smithi181 ceph-mon[94189]: pgmap v10927: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 938 B/s wr, 8 op/s 2023-03-09T20:23:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19531.243248 secs 2023-03-09T20:23:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19531.243248 secs 2023-03-09T20:23:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19531.243248 secs 2023-03-09T20:23:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:41 smithi038 ceph-mon[92422]: pgmap v10928: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 767 B/s wr, 6 op/s 2023-03-09T20:23:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:41 smithi122 ceph-mon[93869]: pgmap v10928: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 767 B/s wr, 6 op/s 2023-03-09T20:23:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:41 smithi181 ceph-mon[94189]: pgmap v10928: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 767 B/s wr, 6 op/s 2023-03-09T20:23:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 19995.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:23:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19533.569088 secs 2023-03-09T20:23:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:43 smithi038 ceph-mon[92422]: pgmap v10929: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 251 KiB/s wr, 10 op/s 2023-03-09T20:23:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19533.569088 secs 2023-03-09T20:23:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:43 smithi122 ceph-mon[93869]: pgmap v10929: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 251 KiB/s wr, 10 op/s 2023-03-09T20:23:44.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19533.569088 secs 2023-03-09T20:23:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:43 smithi181 ceph-mon[94189]: pgmap v10929: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 251 KiB/s wr, 10 op/s 2023-03-09T20:23:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:45 smithi038 ceph-mon[92422]: pgmap v10930: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 329 KiB/s rd, 251 KiB/s wr, 7 op/s 2023-03-09T20:23:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19536.243310 secs 2023-03-09T20:23:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:45 smithi122 ceph-mon[93869]: pgmap v10930: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 329 KiB/s rd, 251 KiB/s wr, 7 op/s 2023-03-09T20:23:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19536.243310 secs 2023-03-09T20:23:46.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:45 smithi181 ceph-mon[94189]: pgmap v10930: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 329 KiB/s rd, 251 KiB/s wr, 7 op/s 2023-03-09T20:23:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19536.243310 secs 2023-03-09T20:23:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:47 smithi038 ceph-mon[92422]: pgmap v10931: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 677 KiB/s rd, 341 KiB/s wr, 12 op/s 2023-03-09T20:23:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:47 smithi122 ceph-mon[93869]: pgmap v10931: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 677 KiB/s rd, 341 KiB/s wr, 12 op/s 2023-03-09T20:23:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:48 smithi181 ceph-mon[94189]: pgmap v10931: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 677 KiB/s rd, 341 KiB/s wr, 12 op/s 2023-03-09T20:23:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20000.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:23:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:48 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:23:49.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:48 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:23:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:48 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:23:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:50 smithi122 ceph-mon[93869]: pgmap v10932: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 676 KiB/s rd, 341 KiB/s wr, 9 op/s 2023-03-09T20:23:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:50 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:23:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:50 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:23:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:50 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:23:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:50 smithi181 ceph-mon[94189]: pgmap v10932: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 676 KiB/s rd, 341 KiB/s wr, 9 op/s 2023-03-09T20:23:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:50 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:23:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:50 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:23:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:50 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:23:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:50 smithi038 ceph-mon[92422]: pgmap v10932: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 676 KiB/s rd, 341 KiB/s wr, 9 op/s 2023-03-09T20:23:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:50 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:23:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:50 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:23:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:50 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:23:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19541.243371 secs 2023-03-09T20:23:51.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19541.243371 secs 2023-03-09T20:23:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19541.243371 secs 2023-03-09T20:23:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:52 smithi122 ceph-mon[93869]: pgmap v10933: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 676 KiB/s rd, 341 KiB/s wr, 9 op/s 2023-03-09T20:23:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:23:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:23:52.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:52 smithi181 ceph-mon[94189]: pgmap v10933: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 676 KiB/s rd, 341 KiB/s wr, 9 op/s 2023-03-09T20:23:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:23:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:23:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:52 smithi038 ceph-mon[92422]: pgmap v10933: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 676 KiB/s rd, 341 KiB/s wr, 9 op/s 2023-03-09T20:23:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:23:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:23:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:23:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:23:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:53 smithi122 ceph-mon[93869]: pgmap v10934: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 838 KiB/s rd, 502 KiB/s wr, 13 op/s 2023-03-09T20:23:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:23:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:23:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:53 smithi181 ceph-mon[94189]: pgmap v10934: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 838 KiB/s rd, 502 KiB/s wr, 13 op/s 2023-03-09T20:23:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:23:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:23:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:53 smithi038 ceph-mon[92422]: pgmap v10934: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 838 KiB/s rd, 502 KiB/s wr, 13 op/s 2023-03-09T20:23:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20005.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:23:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:55 smithi122 ceph-mon[93869]: pgmap v10935: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 522 KiB/s rd, 266 KiB/s wr, 10 op/s 2023-03-09T20:23:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19546.243429 secs 2023-03-09T20:23:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:55 smithi181 ceph-mon[94189]: pgmap v10935: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 522 KiB/s rd, 266 KiB/s wr, 10 op/s 2023-03-09T20:23:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19546.243429 secs 2023-03-09T20:23:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:55 smithi038 ceph-mon[92422]: pgmap v10935: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 522 KiB/s rd, 266 KiB/s wr, 10 op/s 2023-03-09T20:23:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19546.243429 secs 2023-03-09T20:23:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:57 smithi122 ceph-mon[93869]: pgmap v10936: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 636 KiB/s rd, 389 KiB/s wr, 13 op/s 2023-03-09T20:23:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:57 smithi181 ceph-mon[94189]: pgmap v10936: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 636 KiB/s rd, 389 KiB/s wr, 13 op/s 2023-03-09T20:23:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:57 smithi038 ceph-mon[92422]: pgmap v10936: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 636 KiB/s rd, 389 KiB/s wr, 13 op/s 2023-03-09T20:23:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20010.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:24:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:59 smithi122 ceph-mon[93869]: pgmap v10937: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 300 KiB/s wr, 10 op/s 2023-03-09T20:24:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:24:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:23:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:24:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:59 smithi181 ceph-mon[94189]: pgmap v10937: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 300 KiB/s wr, 10 op/s 2023-03-09T20:24:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:24:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:23:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:24:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:59 smithi038 ceph-mon[92422]: pgmap v10937: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 300 KiB/s wr, 10 op/s 2023-03-09T20:24:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:24:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:23:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:24:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19551.243513 secs 2023-03-09T20:24:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:24:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:24:01.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19551.243513 secs 2023-03-09T20:24:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:24:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:24:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19551.243513 secs 2023-03-09T20:24:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:24:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:24:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:01 smithi122 ceph-mon[93869]: pgmap v10938: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 436 KiB/s rd, 455 KiB/s wr, 10 op/s 2023-03-09T20:24:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:01 smithi181 ceph-mon[94189]: pgmap v10938: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 436 KiB/s rd, 455 KiB/s wr, 10 op/s 2023-03-09T20:24:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:01 smithi038 ceph-mon[92422]: pgmap v10938: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 436 KiB/s rd, 455 KiB/s wr, 10 op/s 2023-03-09T20:24:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20015.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:24:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19553.569493 secs 2023-03-09T20:24:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:03 smithi122 ceph-mon[93869]: pgmap v10939: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 469 KiB/s rd, 486 KiB/s wr, 13 op/s 2023-03-09T20:24:04.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19553.569493 secs 2023-03-09T20:24:04.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:03 smithi181 ceph-mon[94189]: pgmap v10939: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 469 KiB/s rd, 486 KiB/s wr, 13 op/s 2023-03-09T20:24:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19553.569493 secs 2023-03-09T20:24:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:03 smithi038 ceph-mon[92422]: pgmap v10939: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 469 KiB/s rd, 486 KiB/s wr, 13 op/s 2023-03-09T20:24:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:05 smithi122 ceph-mon[93869]: pgmap v10940: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 485 KiB/s wr, 22 op/s 2023-03-09T20:24:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:05 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:24:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:05 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:24:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:05 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:24:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:05 smithi181 ceph-mon[94189]: pgmap v10940: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 485 KiB/s wr, 22 op/s 2023-03-09T20:24:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:05 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:24:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:05 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:24:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:05 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:24:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:05 smithi038 ceph-mon[92422]: pgmap v10940: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 485 KiB/s wr, 22 op/s 2023-03-09T20:24:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:05 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:24:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:05 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:24:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:05 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:24:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19556.243591 secs 2023-03-09T20:24:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19556.243591 secs 2023-03-09T20:24:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19556.243591 secs 2023-03-09T20:24:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:07 smithi122 ceph-mon[93869]: pgmap v10941: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 493 KiB/s wr, 27 op/s 2023-03-09T20:24:07.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:07 smithi181 ceph-mon[94189]: pgmap v10941: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 493 KiB/s wr, 27 op/s 2023-03-09T20:24:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:07 smithi038 ceph-mon[92422]: pgmap v10941: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 493 KiB/s wr, 27 op/s 2023-03-09T20:24:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19558.569606 secs 2023-03-09T20:24:08.511 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19558.569606 secs 2023-03-09T20:24:08.511 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19558.569606 secs 2023-03-09T20:24:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20020.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:24:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:09 smithi122 ceph-mon[93869]: pgmap v10942: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 369 KiB/s wr, 24 op/s 2023-03-09T20:24:10.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:09 smithi181 ceph-mon[94189]: pgmap v10942: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 369 KiB/s wr, 24 op/s 2023-03-09T20:24:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:09 smithi038 ceph-mon[92422]: pgmap v10942: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 369 KiB/s wr, 24 op/s 2023-03-09T20:24:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19561.243653 secs 2023-03-09T20:24:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:10 smithi122 ceph-mon[93869]: osdmap e897: 12 total, 12 up, 12 in 2023-03-09T20:24:11.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19561.243653 secs 2023-03-09T20:24:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:10 smithi181 ceph-mon[94189]: osdmap e897: 12 total, 12 up, 12 in 2023-03-09T20:24:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19561.243653 secs 2023-03-09T20:24:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:10 smithi038 ceph-mon[92422]: osdmap e897: 12 total, 12 up, 12 in 2023-03-09T20:24:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:11 smithi122 ceph-mon[93869]: pgmap v10944: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 299 KiB/s wr, 27 op/s 2023-03-09T20:24:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:11 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:24:12.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:11 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:24:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:11 smithi181 ceph-mon[94189]: pgmap v10944: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 299 KiB/s wr, 27 op/s 2023-03-09T20:24:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:11 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:24:12.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:11 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:24:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:11 smithi038 ceph-mon[92422]: pgmap v10944: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 299 KiB/s wr, 27 op/s 2023-03-09T20:24:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:11 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:24:12.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:11 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:24:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:24:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:24:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:24:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:24:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:24:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:24:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20025.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:24:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19563.569724 secs 2023-03-09T20:24:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:13 smithi122 ceph-mon[93869]: pgmap v10945: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 337 KiB/s rd, 354 KiB/s wr, 28 op/s 2023-03-09T20:24:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19563.569724 secs 2023-03-09T20:24:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:13 smithi181 ceph-mon[94189]: pgmap v10945: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 337 KiB/s rd, 354 KiB/s wr, 28 op/s 2023-03-09T20:24:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19563.569724 secs 2023-03-09T20:24:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:13 smithi038 ceph-mon[92422]: pgmap v10945: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 337 KiB/s rd, 354 KiB/s wr, 28 op/s 2023-03-09T20:24:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:15 smithi122 ceph-mon[93869]: pgmap v10946: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 162 KiB/s wr, 12 op/s 2023-03-09T20:24:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19566.243714 secs 2023-03-09T20:24:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:15 smithi181 ceph-mon[94189]: pgmap v10946: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 162 KiB/s wr, 12 op/s 2023-03-09T20:24:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19566.243714 secs 2023-03-09T20:24:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:15 smithi038 ceph-mon[92422]: pgmap v10946: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 162 KiB/s wr, 12 op/s 2023-03-09T20:24:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19566.243714 secs 2023-03-09T20:24:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:17 smithi122 ceph-mon[93869]: pgmap v10947: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 154 KiB/s wr, 10 op/s 2023-03-09T20:24:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:17 smithi181 ceph-mon[94189]: pgmap v10947: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 154 KiB/s wr, 10 op/s 2023-03-09T20:24:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:17 smithi038 ceph-mon[92422]: pgmap v10947: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 154 KiB/s wr, 10 op/s 2023-03-09T20:24:18.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20030.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:24:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19568.569843 secs 2023-03-09T20:24:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:18 smithi122 ceph-mon[93869]: osdmap e898: 12 total, 12 up, 12 in 2023-03-09T20:24:19.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19568.569843 secs 2023-03-09T20:24:19.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:18 smithi181 ceph-mon[94189]: osdmap e898: 12 total, 12 up, 12 in 2023-03-09T20:24:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19568.569843 secs 2023-03-09T20:24:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:18 smithi038 ceph-mon[92422]: osdmap e898: 12 total, 12 up, 12 in 2023-03-09T20:24:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:19 smithi122 ceph-mon[93869]: pgmap v10948: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 214 KiB/s wr, 10 op/s 2023-03-09T20:24:20.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:19 smithi181 ceph-mon[94189]: pgmap v10948: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 214 KiB/s wr, 10 op/s 2023-03-09T20:24:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:19 smithi038 ceph-mon[92422]: pgmap v10948: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 214 KiB/s wr, 10 op/s 2023-03-09T20:24:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19571.243811 secs 2023-03-09T20:24:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19571.243811 secs 2023-03-09T20:24:21.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19571.243811 secs 2023-03-09T20:24:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:21 smithi038 ceph-mon[92422]: pgmap v10950: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 172 KiB/s wr, 9 op/s 2023-03-09T20:24:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:21 smithi122 ceph-mon[93869]: pgmap v10950: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 172 KiB/s wr, 9 op/s 2023-03-09T20:24:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:21 smithi181 ceph-mon[94189]: pgmap v10950: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 172 KiB/s wr, 9 op/s 2023-03-09T20:24:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20035.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:24:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19573.569980 secs 2023-03-09T20:24:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:23 smithi038 ceph-mon[92422]: pgmap v10951: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 80 KiB/s wr, 8 op/s 2023-03-09T20:24:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19573.569980 secs 2023-03-09T20:24:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:23 smithi122 ceph-mon[93869]: pgmap v10951: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 80 KiB/s wr, 8 op/s 2023-03-09T20:24:24.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19573.569980 secs 2023-03-09T20:24:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:23 smithi181 ceph-mon[94189]: pgmap v10951: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 80 KiB/s wr, 8 op/s 2023-03-09T20:24:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:25 smithi038 ceph-mon[92422]: pgmap v10952: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 80 KiB/s wr, 8 op/s 2023-03-09T20:24:26.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19576.243887 secs 2023-03-09T20:24:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:25 smithi122 ceph-mon[93869]: pgmap v10952: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 80 KiB/s wr, 8 op/s 2023-03-09T20:24:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19576.243887 secs 2023-03-09T20:24:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:25 smithi181 ceph-mon[94189]: pgmap v10952: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 80 KiB/s wr, 8 op/s 2023-03-09T20:24:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19576.243887 secs 2023-03-09T20:24:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:27 smithi038 ceph-mon[92422]: pgmap v10953: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 61 KiB/s wr, 7 op/s 2023-03-09T20:24:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:27 smithi122 ceph-mon[93869]: pgmap v10953: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 61 KiB/s wr, 7 op/s 2023-03-09T20:24:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:27 smithi181 ceph-mon[94189]: pgmap v10953: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 61 KiB/s wr, 7 op/s 2023-03-09T20:24:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20040.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:24:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:29 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19578.570113 secs 2023-03-09T20:24:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:24:29.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:29 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19578.570113 secs 2023-03-09T20:24:29.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:24:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:29 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19578.570113 secs 2023-03-09T20:24:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:24:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:30 smithi122 ceph-mon[93869]: pgmap v10954: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 137 KiB/s wr, 7 op/s 2023-03-09T20:24:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:30 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:24:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:30 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:24:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:30 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:24:30.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:30 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:24:30.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:30 smithi181 ceph-mon[94189]: pgmap v10954: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 137 KiB/s wr, 7 op/s 2023-03-09T20:24:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:30 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:24:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:30 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:24:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:30 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:24:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:30 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:24:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:30 smithi038 ceph-mon[92422]: pgmap v10954: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 137 KiB/s wr, 7 op/s 2023-03-09T20:24:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:30 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:24:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:30 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:24:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:30 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:24:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:30 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:24:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19581.243950 secs 2023-03-09T20:24:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:24:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:24:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:24:31.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:24:31.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:24:31.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:31 smithi122 ceph-mon[93869]: pgmap v10955: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 121 KiB/s wr, 7 op/s 2023-03-09T20:24:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19581.243950 secs 2023-03-09T20:24:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:24:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:24:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:24:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:24:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:24:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:31 smithi181 ceph-mon[94189]: pgmap v10955: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 121 KiB/s wr, 7 op/s 2023-03-09T20:24:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19581.243950 secs 2023-03-09T20:24:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:24:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:24:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:24:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:24:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:24:31.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:31 smithi038 ceph-mon[92422]: pgmap v10955: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 121 KiB/s wr, 7 op/s 2023-03-09T20:24:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19583.570220 secs 2023-03-09T20:24:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:33 smithi122 ceph-mon[93869]: pgmap v10956: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 115 KiB/s wr, 9 op/s 2023-03-09T20:24:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19583.570220 secs 2023-03-09T20:24:33.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:33 smithi181 ceph-mon[94189]: pgmap v10956: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 115 KiB/s wr, 9 op/s 2023-03-09T20:24:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19583.570220 secs 2023-03-09T20:24:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:33 smithi038 ceph-mon[92422]: pgmap v10956: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 115 KiB/s wr, 9 op/s 2023-03-09T20:24:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20045.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:24:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:35 smithi122 ceph-mon[93869]: pgmap v10957: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 114 KiB/s wr, 6 op/s 2023-03-09T20:24:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19586.243986 secs 2023-03-09T20:24:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:35 smithi181 ceph-mon[94189]: pgmap v10957: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 114 KiB/s wr, 6 op/s 2023-03-09T20:24:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19586.243986 secs 2023-03-09T20:24:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:35 smithi038 ceph-mon[92422]: pgmap v10957: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 114 KiB/s wr, 6 op/s 2023-03-09T20:24:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19586.243986 secs 2023-03-09T20:24:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:37 smithi122 ceph-mon[93869]: pgmap v10958: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 114 KiB/s wr, 9 op/s 2023-03-09T20:24:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:37 smithi181 ceph-mon[94189]: pgmap v10958: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 114 KiB/s wr, 9 op/s 2023-03-09T20:24:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:37 smithi038 ceph-mon[92422]: pgmap v10958: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 114 KiB/s wr, 9 op/s 2023-03-09T20:24:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20050.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:24:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:39 smithi122 ceph-mon[93869]: pgmap v10959: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 182 KiB/s wr, 7 op/s 2023-03-09T20:24:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:39 smithi181 ceph-mon[94189]: pgmap v10959: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 182 KiB/s wr, 7 op/s 2023-03-09T20:24:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:39 smithi038 ceph-mon[92422]: pgmap v10959: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 182 KiB/s wr, 7 op/s 2023-03-09T20:24:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19591.244045 secs 2023-03-09T20:24:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:24:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:24:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:40 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:24:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19591.244045 secs 2023-03-09T20:24:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:24:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:24:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:40 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:24:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19591.244045 secs 2023-03-09T20:24:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:24:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:24:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:40 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:24:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:41 smithi122 ceph-mon[93869]: pgmap v10960: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 132 KiB/s wr, 7 op/s 2023-03-09T20:24:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:41 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:24:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:24:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:24:42.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:41 smithi181 ceph-mon[94189]: pgmap v10960: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 132 KiB/s wr, 7 op/s 2023-03-09T20:24:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:41 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:24:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:24:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:24:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:41 smithi038 ceph-mon[92422]: pgmap v10960: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 132 KiB/s wr, 7 op/s 2023-03-09T20:24:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:41 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:24:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:24:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:24:43.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20055.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:24:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:43 smithi122 ceph-mon[93869]: pgmap v10961: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 198 KiB/s wr, 10 op/s 2023-03-09T20:24:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:43 smithi181 ceph-mon[94189]: pgmap v10961: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 198 KiB/s wr, 10 op/s 2023-03-09T20:24:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:43 smithi038 ceph-mon[92422]: pgmap v10961: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 198 KiB/s wr, 10 op/s 2023-03-09T20:24:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:45 smithi122 ceph-mon[93869]: pgmap v10962: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 197 KiB/s wr, 7 op/s 2023-03-09T20:24:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19596.244131 secs 2023-03-09T20:24:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:45 smithi181 ceph-mon[94189]: pgmap v10962: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 197 KiB/s wr, 7 op/s 2023-03-09T20:24:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19596.244131 secs 2023-03-09T20:24:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:45 smithi038 ceph-mon[92422]: pgmap v10962: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 197 KiB/s wr, 7 op/s 2023-03-09T20:24:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19596.244131 secs 2023-03-09T20:24:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:47 smithi038 ceph-mon[92422]: pgmap v10963: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 352 KiB/s wr, 11 op/s 2023-03-09T20:24:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:24:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:24:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:47 smithi122 ceph-mon[93869]: pgmap v10963: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 352 KiB/s wr, 11 op/s 2023-03-09T20:24:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:24:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:24:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:47 smithi181 ceph-mon[94189]: pgmap v10963: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 352 KiB/s wr, 11 op/s 2023-03-09T20:24:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:24:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:24:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20060.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:24:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:49 smithi038 ceph-mon[92422]: pgmap v10964: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 352 KiB/s wr, 10 op/s 2023-03-09T20:24:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:49 smithi122 ceph-mon[93869]: pgmap v10964: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 352 KiB/s wr, 10 op/s 2023-03-09T20:24:50.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:49 smithi181 ceph-mon[94189]: pgmap v10964: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 352 KiB/s wr, 10 op/s 2023-03-09T20:24:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19601.244191 secs 2023-03-09T20:24:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19601.244191 secs 2023-03-09T20:24:51.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19601.244191 secs 2023-03-09T20:24:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:51 smithi122 ceph-mon[93869]: pgmap v10965: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 424 KiB/s wr, 10 op/s 2023-03-09T20:24:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:52 smithi181 ceph-mon[94189]: pgmap v10965: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 424 KiB/s wr, 10 op/s 2023-03-09T20:24:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:51 smithi038 ceph-mon[92422]: pgmap v10965: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 424 KiB/s wr, 10 op/s 2023-03-09T20:24:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:53 smithi122 ceph-mon[93869]: pgmap v10966: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 395 KiB/s rd, 379 KiB/s wr, 13 op/s 2023-03-09T20:24:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:53 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:24:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:24:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:24:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:53 smithi181 ceph-mon[94189]: pgmap v10966: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 395 KiB/s rd, 379 KiB/s wr, 13 op/s 2023-03-09T20:24:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:53 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:24:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:24:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:24:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:53 smithi038 ceph-mon[92422]: pgmap v10966: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 395 KiB/s rd, 379 KiB/s wr, 13 op/s 2023-03-09T20:24:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:53 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:24:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:24:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:24:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20065.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:24:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:54 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 19603.570650 secs 2023-03-09T20:24:54.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:54 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 19603.570650 secs 2023-03-09T20:24:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:54 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 19603.570650 secs 2023-03-09T20:24:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:55 smithi122 ceph-mon[93869]: pgmap v10967: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 405 KiB/s wr, 10 op/s 2023-03-09T20:24:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:55 smithi181 ceph-mon[94189]: pgmap v10967: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 405 KiB/s wr, 10 op/s 2023-03-09T20:24:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:55 smithi038 ceph-mon[92422]: pgmap v10967: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 405 KiB/s wr, 10 op/s 2023-03-09T20:24:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19606.244252 secs 2023-03-09T20:24:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:24:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:24:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19606.244252 secs 2023-03-09T20:24:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:24:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:24:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19606.244252 secs 2023-03-09T20:24:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:24:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:24:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:24:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:24:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:57 smithi122 ceph-mon[93869]: pgmap v10968: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 421 KiB/s wr, 13 op/s 2023-03-09T20:24:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:24:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:24:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:57 smithi181 ceph-mon[94189]: pgmap v10968: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 421 KiB/s wr, 13 op/s 2023-03-09T20:24:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:24:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:24:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:57 smithi038 ceph-mon[92422]: pgmap v10968: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 421 KiB/s wr, 13 op/s 2023-03-09T20:24:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:58 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 19608.570769 secs 2023-03-09T20:24:58.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:58 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 19608.570769 secs 2023-03-09T20:24:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:58 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 19608.570769 secs 2023-03-09T20:24:58.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20070.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:25:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:24:59 smithi122 ceph-mon[93869]: pgmap v10969: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 491 KiB/s wr, 12 op/s 2023-03-09T20:25:00.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:24:59 smithi181 ceph-mon[94189]: pgmap v10969: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 491 KiB/s wr, 12 op/s 2023-03-09T20:25:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:24:59 smithi038 ceph-mon[92422]: pgmap v10969: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 491 KiB/s wr, 12 op/s 2023-03-09T20:25:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19611.244331 secs 2023-03-09T20:25:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19611.244331 secs 2023-03-09T20:25:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19611.244331 secs 2023-03-09T20:25:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:01 smithi122 ceph-mon[93869]: pgmap v10970: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 491 KiB/s wr, 10 op/s 2023-03-09T20:25:02.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:01 smithi181 ceph-mon[94189]: pgmap v10970: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 491 KiB/s wr, 10 op/s 2023-03-09T20:25:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:01 smithi038 ceph-mon[92422]: pgmap v10970: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 491 KiB/s wr, 10 op/s 2023-03-09T20:25:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20075.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:25:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19613.570915 secs 2023-03-09T20:25:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:03 smithi122 ceph-mon[93869]: pgmap v10971: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 366 KiB/s wr, 12 op/s 2023-03-09T20:25:04.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19613.570915 secs 2023-03-09T20:25:04.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:03 smithi181 ceph-mon[94189]: pgmap v10971: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 366 KiB/s wr, 12 op/s 2023-03-09T20:25:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19613.570915 secs 2023-03-09T20:25:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:03 smithi038 ceph-mon[92422]: pgmap v10971: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 366 KiB/s wr, 12 op/s 2023-03-09T20:25:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:05 smithi181 ceph-mon[94189]: pgmap v10972: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 372 KiB/s wr, 22 op/s 2023-03-09T20:25:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19616.244400 secs 2023-03-09T20:25:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:05 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:25:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:05 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:25:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:05 smithi038 ceph-mon[92422]: pgmap v10972: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 372 KiB/s wr, 22 op/s 2023-03-09T20:25:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19616.244400 secs 2023-03-09T20:25:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:05 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:25:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:05 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:25:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:05 smithi122 ceph-mon[93869]: pgmap v10972: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 372 KiB/s wr, 22 op/s 2023-03-09T20:25:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19616.244400 secs 2023-03-09T20:25:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:05 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:25:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:05 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:25:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:07 smithi038 ceph-mon[92422]: pgmap v10973: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 302 KiB/s wr, 25 op/s 2023-03-09T20:25:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:07 smithi122 ceph-mon[93869]: pgmap v10973: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 302 KiB/s wr, 25 op/s 2023-03-09T20:25:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:07 smithi181 ceph-mon[94189]: pgmap v10973: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 302 KiB/s wr, 25 op/s 2023-03-09T20:25:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20080.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:25:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19618.571036 secs 2023-03-09T20:25:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19618.571036 secs 2023-03-09T20:25:09.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19618.571036 secs 2023-03-09T20:25:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:10 smithi122 ceph-mon[93869]: pgmap v10974: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 405 KiB/s wr, 24 op/s 2023-03-09T20:25:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:10 smithi122 ceph-mon[93869]: osdmap e899: 12 total, 12 up, 12 in 2023-03-09T20:25:10.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:10 smithi181 ceph-mon[94189]: pgmap v10974: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 405 KiB/s wr, 24 op/s 2023-03-09T20:25:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:10 smithi181 ceph-mon[94189]: osdmap e899: 12 total, 12 up, 12 in 2023-03-09T20:25:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:10 smithi038 ceph-mon[92422]: pgmap v10974: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 405 KiB/s wr, 24 op/s 2023-03-09T20:25:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:10 smithi038 ceph-mon[92422]: osdmap e899: 12 total, 12 up, 12 in 2023-03-09T20:25:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19621.244461 secs 2023-03-09T20:25:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19621.244461 secs 2023-03-09T20:25:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19621.244461 secs 2023-03-09T20:25:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:12 smithi122 ceph-mon[93869]: pgmap v10976: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 216 KiB/s wr, 26 op/s 2023-03-09T20:25:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:25:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:25:12.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:12 smithi181 ceph-mon[94189]: pgmap v10976: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 216 KiB/s wr, 26 op/s 2023-03-09T20:25:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:25:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:25:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:12 smithi038 ceph-mon[92422]: pgmap v10976: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 216 KiB/s wr, 26 op/s 2023-03-09T20:25:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:25:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:25:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:25:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:25:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:13 smithi122 ceph-mon[93869]: pgmap v10977: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 203 KiB/s wr, 27 op/s 2023-03-09T20:25:13.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:25:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:25:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:13 smithi181 ceph-mon[94189]: pgmap v10977: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 203 KiB/s wr, 27 op/s 2023-03-09T20:25:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:25:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:25:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:13 smithi038 ceph-mon[92422]: pgmap v10977: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 203 KiB/s wr, 27 op/s 2023-03-09T20:25:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20085.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:25:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:14 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19623.571187 secs 2023-03-09T20:25:14.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:14 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19623.571187 secs 2023-03-09T20:25:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:14 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19623.571187 secs 2023-03-09T20:25:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:15 smithi122 ceph-mon[93869]: pgmap v10978: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 195 KiB/s wr, 13 op/s 2023-03-09T20:25:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:15 smithi181 ceph-mon[94189]: pgmap v10978: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 195 KiB/s wr, 13 op/s 2023-03-09T20:25:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:15 smithi038 ceph-mon[92422]: pgmap v10978: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 195 KiB/s wr, 13 op/s 2023-03-09T20:25:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19626.244519 secs 2023-03-09T20:25:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19626.244519 secs 2023-03-09T20:25:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19626.244519 secs 2023-03-09T20:25:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:17 smithi122 ceph-mon[93869]: pgmap v10979: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 168 KiB/s wr, 10 op/s 2023-03-09T20:25:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:17 smithi181 ceph-mon[94189]: pgmap v10979: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 168 KiB/s wr, 10 op/s 2023-03-09T20:25:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:17 smithi038 ceph-mon[92422]: pgmap v10979: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 168 KiB/s wr, 10 op/s 2023-03-09T20:25:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19628.571256 secs 2023-03-09T20:25:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19628.571256 secs 2023-03-09T20:25:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19628.571256 secs 2023-03-09T20:25:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20090.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:25:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:19 smithi122 ceph-mon[93869]: pgmap v10980: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 84 KiB/s wr, 10 op/s 2023-03-09T20:25:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:19 smithi122 ceph-mon[93869]: osdmap e900: 12 total, 12 up, 12 in 2023-03-09T20:25:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:19 smithi181 ceph-mon[94189]: pgmap v10980: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 84 KiB/s wr, 10 op/s 2023-03-09T20:25:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:19 smithi181 ceph-mon[94189]: osdmap e900: 12 total, 12 up, 12 in 2023-03-09T20:25:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:19 smithi038 ceph-mon[92422]: pgmap v10980: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 84 KiB/s wr, 10 op/s 2023-03-09T20:25:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:19 smithi038 ceph-mon[92422]: osdmap e900: 12 total, 12 up, 12 in 2023-03-09T20:25:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19631.244581 secs 2023-03-09T20:25:21.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19631.244581 secs 2023-03-09T20:25:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19631.244581 secs 2023-03-09T20:25:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:21 smithi122 ceph-mon[93869]: pgmap v10982: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 85 KiB/s wr, 11 op/s 2023-03-09T20:25:22.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:21 smithi181 ceph-mon[94189]: pgmap v10982: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 85 KiB/s wr, 11 op/s 2023-03-09T20:25:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:21 smithi038 ceph-mon[92422]: pgmap v10982: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 85 KiB/s wr, 11 op/s 2023-03-09T20:25:23.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20095.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:25:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19633.571319 secs 2023-03-09T20:25:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:23 smithi122 ceph-mon[93869]: pgmap v10983: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 79 KiB/s wr, 9 op/s 2023-03-09T20:25:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19633.571319 secs 2023-03-09T20:25:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:23 smithi181 ceph-mon[94189]: pgmap v10983: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 79 KiB/s wr, 9 op/s 2023-03-09T20:25:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19633.571319 secs 2023-03-09T20:25:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:23 smithi038 ceph-mon[92422]: pgmap v10983: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 79 KiB/s wr, 9 op/s 2023-03-09T20:25:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:25 smithi122 ceph-mon[93869]: pgmap v10984: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 148 KiB/s wr, 8 op/s 2023-03-09T20:25:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19636.244642 secs 2023-03-09T20:25:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:25 smithi181 ceph-mon[94189]: pgmap v10984: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 148 KiB/s wr, 8 op/s 2023-03-09T20:25:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19636.244642 secs 2023-03-09T20:25:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:25 smithi038 ceph-mon[92422]: pgmap v10984: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 148 KiB/s wr, 8 op/s 2023-03-09T20:25:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19636.244642 secs 2023-03-09T20:25:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:27 smithi181 ceph-mon[94189]: pgmap v10985: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 148 KiB/s wr, 8 op/s 2023-03-09T20:25:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:27 smithi038 ceph-mon[92422]: pgmap v10985: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 148 KiB/s wr, 8 op/s 2023-03-09T20:25:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:27 smithi122 ceph-mon[93869]: pgmap v10985: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 148 KiB/s wr, 8 op/s 2023-03-09T20:25:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20100.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:25:29.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19638.571401 secs 2023-03-09T20:25:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19638.571401 secs 2023-03-09T20:25:29.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19638.571401 secs 2023-03-09T20:25:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:29 smithi038 ceph-mon[92422]: pgmap v10986: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 90 KiB/s wr, 7 op/s 2023-03-09T20:25:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:29 smithi122 ceph-mon[93869]: pgmap v10986: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 90 KiB/s wr, 7 op/s 2023-03-09T20:25:30.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:29 smithi181 ceph-mon[94189]: pgmap v10986: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 90 KiB/s wr, 7 op/s 2023-03-09T20:25:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19641.244702 secs 2023-03-09T20:25:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:25:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19641.244702 secs 2023-03-09T20:25:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:25:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19641.244702 secs 2023-03-09T20:25:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:25:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:32 smithi122 ceph-mon[93869]: pgmap v10987: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 79 KiB/s wr, 7 op/s 2023-03-09T20:25:32.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:32 smithi181 ceph-mon[94189]: pgmap v10987: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 79 KiB/s wr, 7 op/s 2023-03-09T20:25:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:32 smithi038 ceph-mon[92422]: pgmap v10987: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 79 KiB/s wr, 7 op/s 2023-03-09T20:25:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20105.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:25:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19643.571491 secs 2023-03-09T20:25:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:33 smithi122 ceph-mon[93869]: pgmap v10988: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 74 KiB/s wr, 8 op/s 2023-03-09T20:25:34.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:25:34.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:25:34.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:25:34.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:25:34.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:25:34.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:25:34.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:25:34.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:25:34.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:25:34.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:25:34.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:25:34.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:25:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19643.571491 secs 2023-03-09T20:25:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:33 smithi181 ceph-mon[94189]: pgmap v10988: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 74 KiB/s wr, 8 op/s 2023-03-09T20:25:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:25:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:25:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:25:34.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:25:34.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:25:34.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:25:34.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:25:34.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:25:34.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:25:34.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:25:34.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:25:34.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:25:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19643.571491 secs 2023-03-09T20:25:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:33 smithi038 ceph-mon[92422]: pgmap v10988: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 74 KiB/s wr, 8 op/s 2023-03-09T20:25:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:25:34.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:25:34.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:25:34.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:25:34.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:25:34.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:25:34.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:25:34.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:25:34.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:25:34.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:25:34.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:25:34.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:25:35.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:34 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T20:25:35.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:34 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T20:25:35.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:34 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi038 to 98.73M 2023-03-09T20:25:35.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:34 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi038 to 103529431: error parsing value: Value '103529431' is below minimum 939524096 2023-03-09T20:25:35.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:34 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:25:35.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:34 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:25:35.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:34 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T20:25:35.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:34 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T20:25:35.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:34 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi038 to 98.73M 2023-03-09T20:25:35.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:34 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi038 to 103529431: error parsing value: Value '103529431' is below minimum 939524096 2023-03-09T20:25:35.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:34 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:25:35.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:34 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:25:35.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:34 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T20:25:35.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:34 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T20:25:35.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:34 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi038 to 98.73M 2023-03-09T20:25:35.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:34 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi038 to 103529431: error parsing value: Value '103529431' is below minimum 939524096 2023-03-09T20:25:35.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:34 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:25:35.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:34 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:25:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:35 smithi122 ceph-mon[93869]: pgmap v10989: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 147 KiB/s wr, 7 op/s 2023-03-09T20:25:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19646.244782 secs 2023-03-09T20:25:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:25:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:25:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:25:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:25:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:35 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T20:25:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:35 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T20:25:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:25:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:25:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:25:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:35 smithi181 ceph-mon[94189]: pgmap v10989: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 147 KiB/s wr, 7 op/s 2023-03-09T20:25:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19646.244782 secs 2023-03-09T20:25:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:25:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:25:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:25:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:25:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:35 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T20:25:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:35 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T20:25:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:25:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:25:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:25:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:35 smithi038 ceph-mon[92422]: pgmap v10989: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 147 KiB/s wr, 7 op/s 2023-03-09T20:25:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19646.244782 secs 2023-03-09T20:25:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:25:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:25:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:25:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:25:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:35 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T20:25:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:35 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T20:25:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:25:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:25:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:25:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:37 smithi122 ceph-mon[93869]: pgmap v10990: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 74 KiB/s wr, 8 op/s 2023-03-09T20:25:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:37 smithi181 ceph-mon[94189]: pgmap v10990: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 74 KiB/s wr, 8 op/s 2023-03-09T20:25:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:37 smithi038 ceph-mon[92422]: pgmap v10990: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 74 KiB/s wr, 8 op/s 2023-03-09T20:25:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20110.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:25:39.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19648.571633 secs 2023-03-09T20:25:39.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19648.571633 secs 2023-03-09T20:25:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19648.571633 secs 2023-03-09T20:25:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:39 smithi038 ceph-mon[92422]: pgmap v10991: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 138 KiB/s wr, 7 op/s 2023-03-09T20:25:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:39 smithi122 ceph-mon[93869]: pgmap v10991: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 138 KiB/s wr, 7 op/s 2023-03-09T20:25:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:39 smithi181 ceph-mon[94189]: pgmap v10991: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 138 KiB/s wr, 7 op/s 2023-03-09T20:25:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19651.244843 secs 2023-03-09T20:25:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:25:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:25:41.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19651.244843 secs 2023-03-09T20:25:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:25:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:25:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19651.244843 secs 2023-03-09T20:25:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:25:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:25:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:42 smithi122 ceph-mon[93869]: pgmap v10992: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 138 KiB/s wr, 6 op/s 2023-03-09T20:25:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:42 smithi181 ceph-mon[94189]: pgmap v10992: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 138 KiB/s wr, 6 op/s 2023-03-09T20:25:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:42 smithi038 ceph-mon[92422]: pgmap v10992: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 138 KiB/s wr, 6 op/s 2023-03-09T20:25:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20115.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:25:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:44 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19653.571759 secs 2023-03-09T20:25:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:44 smithi122 ceph-mon[93869]: pgmap v10993: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 155 KiB/s wr, 8 op/s 2023-03-09T20:25:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:44 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19653.571759 secs 2023-03-09T20:25:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:44 smithi181 ceph-mon[94189]: pgmap v10993: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 155 KiB/s wr, 8 op/s 2023-03-09T20:25:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:44 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19653.571759 secs 2023-03-09T20:25:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:44 smithi038 ceph-mon[92422]: pgmap v10993: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 155 KiB/s wr, 8 op/s 2023-03-09T20:25:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:45 smithi122 ceph-mon[93869]: pgmap v10994: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 155 KiB/s wr, 8 op/s 2023-03-09T20:25:45.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:45 smithi181 ceph-mon[94189]: pgmap v10994: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 155 KiB/s wr, 8 op/s 2023-03-09T20:25:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:45 smithi038 ceph-mon[92422]: pgmap v10994: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 155 KiB/s wr, 8 op/s 2023-03-09T20:25:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19656.244904 secs 2023-03-09T20:25:46.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19656.244904 secs 2023-03-09T20:25:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19656.244904 secs 2023-03-09T20:25:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:47 smithi122 ceph-mon[93869]: pgmap v10995: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 82 KiB/s wr, 8 op/s 2023-03-09T20:25:47.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:47 smithi181 ceph-mon[94189]: pgmap v10995: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 82 KiB/s wr, 8 op/s 2023-03-09T20:25:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:47 smithi038 ceph-mon[92422]: pgmap v10995: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 82 KiB/s wr, 8 op/s 2023-03-09T20:25:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:25:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:25:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:25:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:25:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:25:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:25:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20120.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:25:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:49 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19658.571892 secs 2023-03-09T20:25:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:49 smithi122 ceph-mon[93869]: pgmap v10996: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 493 KiB/s rd, 236 KiB/s wr, 8 op/s 2023-03-09T20:25:49.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:49 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19658.571892 secs 2023-03-09T20:25:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:49 smithi181 ceph-mon[94189]: pgmap v10996: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 493 KiB/s rd, 236 KiB/s wr, 8 op/s 2023-03-09T20:25:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:49 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19658.571892 secs 2023-03-09T20:25:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:49 smithi038 ceph-mon[92422]: pgmap v10996: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 493 KiB/s rd, 236 KiB/s wr, 8 op/s 2023-03-09T20:25:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19661.244965 secs 2023-03-09T20:25:50.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19661.244965 secs 2023-03-09T20:25:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19661.244965 secs 2023-03-09T20:25:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:51 smithi122 ceph-mon[93869]: pgmap v10997: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 172 KiB/s wr, 7 op/s 2023-03-09T20:25:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:51 smithi181 ceph-mon[94189]: pgmap v10997: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 172 KiB/s wr, 7 op/s 2023-03-09T20:25:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:51 smithi038 ceph-mon[92422]: pgmap v10997: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 172 KiB/s wr, 7 op/s 2023-03-09T20:25:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19663.572045 secs 2023-03-09T20:25:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:53 smithi122 ceph-mon[93869]: pgmap v10998: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 202 KiB/s wr, 10 op/s 2023-03-09T20:25:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19663.572045 secs 2023-03-09T20:25:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:53 smithi181 ceph-mon[94189]: pgmap v10998: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 202 KiB/s wr, 10 op/s 2023-03-09T20:25:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19663.572045 secs 2023-03-09T20:25:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:53 smithi038 ceph-mon[92422]: pgmap v10998: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 202 KiB/s wr, 10 op/s 2023-03-09T20:25:53.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20125.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:25:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:55 smithi122 ceph-mon[93869]: pgmap v10999: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 308 KiB/s wr, 10 op/s 2023-03-09T20:25:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19666.245025 secs 2023-03-09T20:25:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:55 smithi181 ceph-mon[94189]: pgmap v10999: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 308 KiB/s wr, 10 op/s 2023-03-09T20:25:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19666.245025 secs 2023-03-09T20:25:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:55 smithi038 ceph-mon[92422]: pgmap v10999: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 308 KiB/s wr, 10 op/s 2023-03-09T20:25:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19666.245025 secs 2023-03-09T20:25:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:25:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:25:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:25:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:25:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:25:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:25:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:57 smithi038 ceph-mon[92422]: pgmap v11000: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 308 KiB/s wr, 11 op/s 2023-03-09T20:25:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:57 smithi122 ceph-mon[93869]: pgmap v11000: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 308 KiB/s wr, 11 op/s 2023-03-09T20:25:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:57 smithi181 ceph-mon[94189]: pgmap v11000: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 308 KiB/s wr, 11 op/s 2023-03-09T20:25:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20130.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:25:59.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19668.572161 secs 2023-03-09T20:25:59.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19668.572161 secs 2023-03-09T20:25:59.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19668.572161 secs 2023-03-09T20:26:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:59 smithi038 ceph-mon[92422]: pgmap v11001: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 482 KiB/s wr, 11 op/s 2023-03-09T20:26:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:26:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:25:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:26:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:59 smithi122 ceph-mon[93869]: pgmap v11001: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 482 KiB/s wr, 11 op/s 2023-03-09T20:26:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:26:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:25:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:26:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:59 smithi181 ceph-mon[94189]: pgmap v11001: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 482 KiB/s wr, 11 op/s 2023-03-09T20:26:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:26:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:25:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:26:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19671.245099 secs 2023-03-09T20:26:01.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19671.245099 secs 2023-03-09T20:26:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19671.245099 secs 2023-03-09T20:26:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:02 smithi122 ceph-mon[93869]: pgmap v11002: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 357 KiB/s wr, 10 op/s 2023-03-09T20:26:02.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:02 smithi181 ceph-mon[94189]: pgmap v11002: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 357 KiB/s wr, 10 op/s 2023-03-09T20:26:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:02 smithi038 ceph-mon[92422]: pgmap v11002: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 357 KiB/s wr, 10 op/s 2023-03-09T20:26:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20135.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:26:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:04 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19673.572251 secs 2023-03-09T20:26:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:04 smithi122 ceph-mon[93869]: pgmap v11003: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 358 KiB/s wr, 12 op/s 2023-03-09T20:26:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:04 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19673.572251 secs 2023-03-09T20:26:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:04 smithi181 ceph-mon[94189]: pgmap v11003: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 358 KiB/s wr, 12 op/s 2023-03-09T20:26:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:04 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19673.572251 secs 2023-03-09T20:26:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:04 smithi038 ceph-mon[92422]: pgmap v11003: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 358 KiB/s wr, 12 op/s 2023-03-09T20:26:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:06 smithi122 ceph-mon[93869]: pgmap v11004: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 541 KiB/s rd, 560 KiB/s wr, 22 op/s 2023-03-09T20:26:06.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19676.245160 secs 2023-03-09T20:26:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:06 smithi181 ceph-mon[94189]: pgmap v11004: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 541 KiB/s rd, 560 KiB/s wr, 22 op/s 2023-03-09T20:26:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19676.245160 secs 2023-03-09T20:26:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:06 smithi038 ceph-mon[92422]: pgmap v11004: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 541 KiB/s rd, 560 KiB/s wr, 22 op/s 2023-03-09T20:26:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19676.245160 secs 2023-03-09T20:26:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:07 smithi122 ceph-mon[93869]: pgmap v11005: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 699 KiB/s rd, 637 KiB/s wr, 24 op/s 2023-03-09T20:26:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:26:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:26:07.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:07 smithi181 ceph-mon[94189]: pgmap v11005: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 699 KiB/s rd, 637 KiB/s wr, 24 op/s 2023-03-09T20:26:07.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:26:07.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:26:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:07 smithi038 ceph-mon[92422]: pgmap v11005: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 699 KiB/s rd, 637 KiB/s wr, 24 op/s 2023-03-09T20:26:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:26:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:26:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20140.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:26:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:09 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19678.572316 secs 2023-03-09T20:26:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:26:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:26:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:09 smithi122 ceph-mon[93869]: pgmap v11006: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 653 KiB/s wr, 24 op/s 2023-03-09T20:26:09.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:09 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19678.572316 secs 2023-03-09T20:26:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:26:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:26:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:09 smithi181 ceph-mon[94189]: pgmap v11006: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 653 KiB/s wr, 24 op/s 2023-03-09T20:26:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:09 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19678.572316 secs 2023-03-09T20:26:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:26:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:26:09.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:09 smithi038 ceph-mon[92422]: pgmap v11006: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 653 KiB/s wr, 24 op/s 2023-03-09T20:26:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19681.245220 secs 2023-03-09T20:26:10.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19681.245220 secs 2023-03-09T20:26:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19681.245220 secs 2023-03-09T20:26:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:11 smithi122 ceph-mon[93869]: pgmap v11007: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 539 KiB/s rd, 478 KiB/s wr, 22 op/s 2023-03-09T20:26:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:11 smithi181 ceph-mon[94189]: pgmap v11007: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 539 KiB/s rd, 478 KiB/s wr, 22 op/s 2023-03-09T20:26:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:11 smithi038 ceph-mon[92422]: pgmap v11007: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 539 KiB/s rd, 478 KiB/s wr, 22 op/s 2023-03-09T20:26:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19683.572451 secs 2023-03-09T20:26:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:13 smithi122 ceph-mon[93869]: pgmap v11008: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 584 KiB/s wr, 24 op/s 2023-03-09T20:26:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19683.572451 secs 2023-03-09T20:26:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:13 smithi181 ceph-mon[94189]: pgmap v11008: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 584 KiB/s wr, 24 op/s 2023-03-09T20:26:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19683.572451 secs 2023-03-09T20:26:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:13 smithi038 ceph-mon[92422]: pgmap v11008: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 584 KiB/s wr, 24 op/s 2023-03-09T20:26:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20145.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:26:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:15 smithi122 ceph-mon[93869]: pgmap v11009: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 584 KiB/s wr, 22 op/s 2023-03-09T20:26:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19686.245283 secs 2023-03-09T20:26:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:15 smithi181 ceph-mon[94189]: pgmap v11009: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 584 KiB/s wr, 22 op/s 2023-03-09T20:26:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19686.245283 secs 2023-03-09T20:26:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:15 smithi038 ceph-mon[92422]: pgmap v11009: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 584 KiB/s wr, 22 op/s 2023-03-09T20:26:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19686.245283 secs 2023-03-09T20:26:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:17 smithi181 ceph-mon[94189]: pgmap v11010: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 398 KiB/s rd, 371 KiB/s wr, 12 op/s 2023-03-09T20:26:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:17 smithi038 ceph-mon[92422]: pgmap v11010: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 398 KiB/s rd, 371 KiB/s wr, 12 op/s 2023-03-09T20:26:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:17 smithi122 ceph-mon[93869]: pgmap v11010: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 398 KiB/s rd, 371 KiB/s wr, 12 op/s 2023-03-09T20:26:18.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20150.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:26:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19688.572545 secs 2023-03-09T20:26:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19688.572545 secs 2023-03-09T20:26:19.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19688.572545 secs 2023-03-09T20:26:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:19 smithi038 ceph-mon[92422]: pgmap v11011: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 171 KiB/s wr, 8 op/s 2023-03-09T20:26:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:19 smithi122 ceph-mon[93869]: pgmap v11011: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 171 KiB/s wr, 8 op/s 2023-03-09T20:26:20.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:19 smithi181 ceph-mon[94189]: pgmap v11011: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 171 KiB/s wr, 8 op/s 2023-03-09T20:26:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19691.245343 secs 2023-03-09T20:26:21.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19691.245343 secs 2023-03-09T20:26:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19691.245343 secs 2023-03-09T20:26:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:22 smithi122 ceph-mon[93869]: pgmap v11012: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 156 KiB/s wr, 7 op/s 2023-03-09T20:26:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:22 smithi181 ceph-mon[94189]: pgmap v11012: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 156 KiB/s wr, 7 op/s 2023-03-09T20:26:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:22 smithi038 ceph-mon[92422]: pgmap v11012: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 156 KiB/s wr, 7 op/s 2023-03-09T20:26:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20155.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:26:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:24 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19693.572682 secs 2023-03-09T20:26:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:24 smithi122 ceph-mon[93869]: pgmap v11013: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 327 KiB/s wr, 11 op/s 2023-03-09T20:26:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:24 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19693.572682 secs 2023-03-09T20:26:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:24 smithi181 ceph-mon[94189]: pgmap v11013: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 327 KiB/s wr, 11 op/s 2023-03-09T20:26:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:24 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19693.572682 secs 2023-03-09T20:26:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:24 smithi038 ceph-mon[92422]: pgmap v11013: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 327 KiB/s wr, 11 op/s 2023-03-09T20:26:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:25 smithi181 ceph-mon[94189]: pgmap v11014: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 219 KiB/s wr, 10 op/s 2023-03-09T20:26:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19696.245403 secs 2023-03-09T20:26:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:25 smithi181 ceph-mon[94189]: osdmap e901: 12 total, 12 up, 12 in 2023-03-09T20:26:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:25 smithi038 ceph-mon[92422]: pgmap v11014: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 219 KiB/s wr, 10 op/s 2023-03-09T20:26:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19696.245403 secs 2023-03-09T20:26:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:25 smithi038 ceph-mon[92422]: osdmap e901: 12 total, 12 up, 12 in 2023-03-09T20:26:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:25 smithi122 ceph-mon[93869]: pgmap v11014: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 219 KiB/s wr, 10 op/s 2023-03-09T20:26:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19696.245403 secs 2023-03-09T20:26:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:25 smithi122 ceph-mon[93869]: osdmap e901: 12 total, 12 up, 12 in 2023-03-09T20:26:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:27 smithi038 ceph-mon[92422]: pgmap v11016: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 328 KiB/s wr, 12 op/s 2023-03-09T20:26:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:27 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:26:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:27 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:26:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:27 smithi122 ceph-mon[93869]: pgmap v11016: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 328 KiB/s wr, 12 op/s 2023-03-09T20:26:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:27 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:26:28.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:27 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:26:28.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:27 smithi181 ceph-mon[94189]: pgmap v11016: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 328 KiB/s wr, 12 op/s 2023-03-09T20:26:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:27 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:26:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:27 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:26:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20160.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:26:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:29 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19698.572805 secs 2023-03-09T20:26:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:29 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:26:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:29 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:26:29.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:29 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19698.572805 secs 2023-03-09T20:26:29.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:29 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:26:29.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:29 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:26:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:29 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19698.572805 secs 2023-03-09T20:26:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:29 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:26:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:29 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:26:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:30 smithi122 ceph-mon[93869]: pgmap v11017: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 329 KiB/s wr, 13 op/s 2023-03-09T20:26:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:30 smithi181 ceph-mon[94189]: pgmap v11017: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 329 KiB/s wr, 13 op/s 2023-03-09T20:26:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:30 smithi038 ceph-mon[92422]: pgmap v11017: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 329 KiB/s wr, 13 op/s 2023-03-09T20:26:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19701.245515 secs 2023-03-09T20:26:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19701.245515 secs 2023-03-09T20:26:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19701.245515 secs 2023-03-09T20:26:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:32 smithi122 ceph-mon[93869]: pgmap v11018: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 329 KiB/s wr, 13 op/s 2023-03-09T20:26:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:32 smithi181 ceph-mon[94189]: pgmap v11018: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 329 KiB/s wr, 13 op/s 2023-03-09T20:26:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:32 smithi038 ceph-mon[92422]: pgmap v11018: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 329 KiB/s wr, 13 op/s 2023-03-09T20:26:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:33 smithi122 ceph-mon[93869]: pgmap v11019: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 314 KiB/s wr, 13 op/s 2023-03-09T20:26:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:33 smithi181 ceph-mon[94189]: pgmap v11019: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 314 KiB/s wr, 13 op/s 2023-03-09T20:26:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:33 smithi038 ceph-mon[92422]: pgmap v11019: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 314 KiB/s wr, 13 op/s 2023-03-09T20:26:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20165.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:26:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:34 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19703.572914 secs 2023-03-09T20:26:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:34 smithi122 ceph-mon[93869]: osdmap e902: 12 total, 12 up, 12 in 2023-03-09T20:26:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:34 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19703.572914 secs 2023-03-09T20:26:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:34 smithi181 ceph-mon[94189]: osdmap e902: 12 total, 12 up, 12 in 2023-03-09T20:26:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:34 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19703.572914 secs 2023-03-09T20:26:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:34 smithi038 ceph-mon[92422]: osdmap e902: 12 total, 12 up, 12 in 2023-03-09T20:26:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:35 smithi122 ceph-mon[93869]: pgmap v11021: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 303 KiB/s wr, 12 op/s 2023-03-09T20:26:35.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:35 smithi181 ceph-mon[94189]: pgmap v11021: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 303 KiB/s wr, 12 op/s 2023-03-09T20:26:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:35 smithi038 ceph-mon[92422]: pgmap v11021: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 303 KiB/s wr, 12 op/s 2023-03-09T20:26:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19706.245578 secs 2023-03-09T20:26:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:26:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:26:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:26:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:26:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:26:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:26:36.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19706.245578 secs 2023-03-09T20:26:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:26:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:26:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:26:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:26:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:26:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:26:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19706.245578 secs 2023-03-09T20:26:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:26:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:26:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:26:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:26:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:26:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:26:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:26:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:26:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:37 smithi122 ceph-mon[93869]: pgmap v11022: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 193 KiB/s wr, 10 op/s 2023-03-09T20:26:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:26:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:26:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:37 smithi181 ceph-mon[94189]: pgmap v11022: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 193 KiB/s wr, 10 op/s 2023-03-09T20:26:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:26:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:26:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:37 smithi038 ceph-mon[92422]: pgmap v11022: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 193 KiB/s wr, 10 op/s 2023-03-09T20:26:38.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20170.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:26:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:39 smithi038 ceph-mon[92422]: pgmap v11023: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 253 KiB/s wr, 11 op/s 2023-03-09T20:26:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:39 smithi122 ceph-mon[93869]: pgmap v11023: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 253 KiB/s wr, 11 op/s 2023-03-09T20:26:40.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:39 smithi181 ceph-mon[94189]: pgmap v11023: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 253 KiB/s wr, 11 op/s 2023-03-09T20:26:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19711.245640 secs 2023-03-09T20:26:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:40 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:26:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19711.245640 secs 2023-03-09T20:26:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:40 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:26:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19711.245640 secs 2023-03-09T20:26:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:40 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:26:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:41 smithi038 ceph-mon[92422]: pgmap v11024: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 253 KiB/s wr, 11 op/s 2023-03-09T20:26:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:41 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:26:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:26:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:26:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:41 smithi122 ceph-mon[93869]: pgmap v11024: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 253 KiB/s wr, 11 op/s 2023-03-09T20:26:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:41 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:26:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:26:42.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:26:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:41 smithi181 ceph-mon[94189]: pgmap v11024: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 253 KiB/s wr, 11 op/s 2023-03-09T20:26:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:41 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:26:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:26:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:26:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20175.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:26:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:43 smithi038 ceph-mon[92422]: pgmap v11025: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 289 KiB/s wr, 9 op/s 2023-03-09T20:26:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:43 smithi122 ceph-mon[93869]: pgmap v11025: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 289 KiB/s wr, 9 op/s 2023-03-09T20:26:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:43 smithi181 ceph-mon[94189]: pgmap v11025: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 289 KiB/s wr, 9 op/s 2023-03-09T20:26:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:45 smithi038 ceph-mon[92422]: pgmap v11026: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 313 KiB/s rd, 297 KiB/s wr, 9 op/s 2023-03-09T20:26:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19716.245711 secs 2023-03-09T20:26:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:45 smithi122 ceph-mon[93869]: pgmap v11026: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 313 KiB/s rd, 297 KiB/s wr, 9 op/s 2023-03-09T20:26:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19716.245711 secs 2023-03-09T20:26:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:45 smithi181 ceph-mon[94189]: pgmap v11026: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 313 KiB/s rd, 297 KiB/s wr, 9 op/s 2023-03-09T20:26:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19716.245711 secs 2023-03-09T20:26:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:48 smithi122 ceph-mon[93869]: pgmap v11027: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 284 KiB/s wr, 10 op/s 2023-03-09T20:26:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:48 smithi181 ceph-mon[94189]: pgmap v11027: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 284 KiB/s wr, 10 op/s 2023-03-09T20:26:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:48 smithi038 ceph-mon[92422]: pgmap v11027: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 284 KiB/s wr, 10 op/s 2023-03-09T20:26:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20180.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:26:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:50 smithi122 ceph-mon[93869]: pgmap v11028: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 352 KiB/s wr, 10 op/s 2023-03-09T20:26:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:50 smithi181 ceph-mon[94189]: pgmap v11028: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 352 KiB/s wr, 10 op/s 2023-03-09T20:26:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:50 smithi038 ceph-mon[92422]: pgmap v11028: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 352 KiB/s wr, 10 op/s 2023-03-09T20:26:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19721.245770 secs 2023-03-09T20:26:51.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19721.245770 secs 2023-03-09T20:26:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19721.245770 secs 2023-03-09T20:26:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:52 smithi122 ceph-mon[93869]: pgmap v11029: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 302 KiB/s wr, 9 op/s 2023-03-09T20:26:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:26:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:26:52.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:52 smithi181 ceph-mon[94189]: pgmap v11029: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 302 KiB/s wr, 9 op/s 2023-03-09T20:26:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:26:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:26:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:52 smithi038 ceph-mon[92422]: pgmap v11029: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 302 KiB/s wr, 9 op/s 2023-03-09T20:26:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:26:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:26:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:26:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:26:53.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:26:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:26:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:26:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:26:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20185.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:26:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:54 smithi122 ceph-mon[93869]: pgmap v11030: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 515 KiB/s rd, 500 KiB/s wr, 13 op/s 2023-03-09T20:26:54.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:54 smithi181 ceph-mon[94189]: pgmap v11030: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 515 KiB/s rd, 500 KiB/s wr, 13 op/s 2023-03-09T20:26:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:54 smithi038 ceph-mon[92422]: pgmap v11030: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 515 KiB/s rd, 500 KiB/s wr, 13 op/s 2023-03-09T20:26:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:56 smithi122 ceph-mon[93869]: pgmap v11031: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 327 KiB/s wr, 11 op/s 2023-03-09T20:26:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19726.245833 secs 2023-03-09T20:26:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:56 smithi181 ceph-mon[94189]: pgmap v11031: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 327 KiB/s wr, 11 op/s 2023-03-09T20:26:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19726.245833 secs 2023-03-09T20:26:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:56 smithi038 ceph-mon[92422]: pgmap v11031: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 327 KiB/s wr, 11 op/s 2023-03-09T20:26:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19726.245833 secs 2023-03-09T20:26:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:57 smithi122 ceph-mon[93869]: pgmap v11032: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 428 KiB/s wr, 12 op/s 2023-03-09T20:26:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:57 smithi181 ceph-mon[94189]: pgmap v11032: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 428 KiB/s wr, 12 op/s 2023-03-09T20:26:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:57 smithi038 ceph-mon[92422]: pgmap v11032: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 428 KiB/s wr, 12 op/s 2023-03-09T20:26:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20190.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:26:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:59 smithi122 ceph-mon[93869]: pgmap v11033: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 539 KiB/s rd, 581 KiB/s wr, 12 op/s 2023-03-09T20:26:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:26:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:26:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:26:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:59 smithi181 ceph-mon[94189]: pgmap v11033: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 539 KiB/s rd, 581 KiB/s wr, 12 op/s 2023-03-09T20:26:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:26:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:26:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:26:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:59 smithi038 ceph-mon[92422]: pgmap v11033: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 539 KiB/s rd, 581 KiB/s wr, 12 op/s 2023-03-09T20:26:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:26:59.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:26:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:27:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19731.245891 secs 2023-03-09T20:27:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19731.245891 secs 2023-03-09T20:27:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19731.245891 secs 2023-03-09T20:27:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:01 smithi122 ceph-mon[93869]: pgmap v11034: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 471 KiB/s rd, 512 KiB/s wr, 9 op/s 2023-03-09T20:27:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:27:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:27:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:01 smithi181 ceph-mon[94189]: pgmap v11034: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 471 KiB/s rd, 512 KiB/s wr, 9 op/s 2023-03-09T20:27:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:27:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:27:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:01 smithi038 ceph-mon[92422]: pgmap v11034: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 471 KiB/s rd, 512 KiB/s wr, 9 op/s 2023-03-09T20:27:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:27:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:27:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19733.573635 secs 2023-03-09T20:27:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:03 smithi122 ceph-mon[93869]: pgmap v11035: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 535 KiB/s wr, 14 op/s 2023-03-09T20:27:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19733.573635 secs 2023-03-09T20:27:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:03 smithi181 ceph-mon[94189]: pgmap v11035: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 535 KiB/s wr, 14 op/s 2023-03-09T20:27:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19733.573635 secs 2023-03-09T20:27:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:03 smithi038 ceph-mon[92422]: pgmap v11035: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 535 KiB/s wr, 14 op/s 2023-03-09T20:27:03.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20195.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:27:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:05 smithi122 ceph-mon[93869]: pgmap v11036: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 400 KiB/s wr, 24 op/s 2023-03-09T20:27:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:05 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:27:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:05 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:27:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:05 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:27:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:05 smithi181 ceph-mon[94189]: pgmap v11036: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 400 KiB/s wr, 24 op/s 2023-03-09T20:27:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:05 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:27:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:05 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:27:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:05 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:27:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:05 smithi038 ceph-mon[92422]: pgmap v11036: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 400 KiB/s wr, 24 op/s 2023-03-09T20:27:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:05 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:27:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:05 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:27:05.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:05 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:27:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19736.245950 secs 2023-03-09T20:27:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19736.245950 secs 2023-03-09T20:27:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19736.245950 secs 2023-03-09T20:27:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:07 smithi122 ceph-mon[93869]: pgmap v11037: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 384 KiB/s wr, 25 op/s 2023-03-09T20:27:07.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:07 smithi181 ceph-mon[94189]: pgmap v11037: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 384 KiB/s wr, 25 op/s 2023-03-09T20:27:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:07 smithi038 ceph-mon[92422]: pgmap v11037: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 384 KiB/s wr, 25 op/s 2023-03-09T20:27:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19738.573751 secs 2023-03-09T20:27:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19738.573751 secs 2023-03-09T20:27:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19738.573751 secs 2023-03-09T20:27:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20200.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:27:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:09 smithi038 ceph-mon[92422]: pgmap v11038: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 321 KiB/s rd, 328 KiB/s wr, 24 op/s 2023-03-09T20:27:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:09 smithi038 ceph-mon[92422]: osdmap e903: 12 total, 12 up, 12 in 2023-03-09T20:27:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:09 smithi122 ceph-mon[93869]: pgmap v11038: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 321 KiB/s rd, 328 KiB/s wr, 24 op/s 2023-03-09T20:27:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:09 smithi122 ceph-mon[93869]: osdmap e903: 12 total, 12 up, 12 in 2023-03-09T20:27:10.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:09 smithi181 ceph-mon[94189]: pgmap v11038: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 321 KiB/s rd, 328 KiB/s wr, 24 op/s 2023-03-09T20:27:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:09 smithi181 ceph-mon[94189]: osdmap e903: 12 total, 12 up, 12 in 2023-03-09T20:27:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19741.245984 secs 2023-03-09T20:27:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19741.245984 secs 2023-03-09T20:27:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19741.245984 secs 2023-03-09T20:27:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:12 smithi122 ceph-mon[93869]: pgmap v11040: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 176 KiB/s wr, 26 op/s 2023-03-09T20:27:12.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:12 smithi181 ceph-mon[94189]: pgmap v11040: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 176 KiB/s wr, 26 op/s 2023-03-09T20:27:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:12 smithi038 ceph-mon[92422]: pgmap v11040: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 176 KiB/s wr, 26 op/s 2023-03-09T20:27:13.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20205.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:27:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:14 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19743.573863 secs 2023-03-09T20:27:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:14 smithi122 ceph-mon[93869]: pgmap v11041: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 149 KiB/s wr, 23 op/s 2023-03-09T20:27:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:14 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19743.573863 secs 2023-03-09T20:27:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:14 smithi181 ceph-mon[94189]: pgmap v11041: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 149 KiB/s wr, 23 op/s 2023-03-09T20:27:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:14 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19743.573863 secs 2023-03-09T20:27:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:14 smithi038 ceph-mon[92422]: pgmap v11041: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 149 KiB/s wr, 23 op/s 2023-03-09T20:27:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:16 smithi122 ceph-mon[93869]: pgmap v11042: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 92 KiB/s wr, 8 op/s 2023-03-09T20:27:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19746.246042 secs 2023-03-09T20:27:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:16 smithi181 ceph-mon[94189]: pgmap v11042: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 92 KiB/s wr, 8 op/s 2023-03-09T20:27:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19746.246042 secs 2023-03-09T20:27:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:16 smithi038 ceph-mon[92422]: pgmap v11042: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 92 KiB/s wr, 8 op/s 2023-03-09T20:27:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19746.246042 secs 2023-03-09T20:27:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:18 smithi122 ceph-mon[93869]: pgmap v11043: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 92 KiB/s wr, 8 op/s 2023-03-09T20:27:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:18 smithi181 ceph-mon[94189]: pgmap v11043: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 92 KiB/s wr, 8 op/s 2023-03-09T20:27:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:18 smithi038 ceph-mon[92422]: pgmap v11043: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 92 KiB/s wr, 8 op/s 2023-03-09T20:27:18.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20210.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:27:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:19 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19748.573998 secs 2023-03-09T20:27:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:19 smithi122 ceph-mon[93869]: osdmap e904: 12 total, 12 up, 12 in 2023-03-09T20:27:19.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:19 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19748.573998 secs 2023-03-09T20:27:19.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:19 smithi181 ceph-mon[94189]: osdmap e904: 12 total, 12 up, 12 in 2023-03-09T20:27:19.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:19 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19748.573998 secs 2023-03-09T20:27:19.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:19 smithi038 ceph-mon[92422]: osdmap e904: 12 total, 12 up, 12 in 2023-03-09T20:27:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:20 smithi122 ceph-mon[93869]: pgmap v11044: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 111 KiB/s wr, 8 op/s 2023-03-09T20:27:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:20 smithi181 ceph-mon[94189]: pgmap v11044: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 111 KiB/s wr, 8 op/s 2023-03-09T20:27:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:20 smithi038 ceph-mon[92422]: pgmap v11044: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 111 KiB/s wr, 8 op/s 2023-03-09T20:27:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19751.246103 secs 2023-03-09T20:27:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19751.246103 secs 2023-03-09T20:27:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19751.246103 secs 2023-03-09T20:27:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:22 smithi122 ceph-mon[93869]: pgmap v11046: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 112 KiB/s wr, 8 op/s 2023-03-09T20:27:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:22 smithi181 ceph-mon[94189]: pgmap v11046: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 112 KiB/s wr, 8 op/s 2023-03-09T20:27:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:22 smithi038 ceph-mon[92422]: pgmap v11046: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 112 KiB/s wr, 8 op/s 2023-03-09T20:27:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:23 smithi122 ceph-mon[93869]: pgmap v11047: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 112 KiB/s wr, 8 op/s 2023-03-09T20:27:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:23 smithi181 ceph-mon[94189]: pgmap v11047: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 112 KiB/s wr, 8 op/s 2023-03-09T20:27:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:23 smithi038 ceph-mon[92422]: pgmap v11047: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 112 KiB/s wr, 8 op/s 2023-03-09T20:27:23.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20215.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:27:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:24 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19753.574114 secs 2023-03-09T20:27:24.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:24 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19753.574114 secs 2023-03-09T20:27:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:24 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19753.574114 secs 2023-03-09T20:27:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:25 smithi122 ceph-mon[93869]: pgmap v11048: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 93 KiB/s wr, 7 op/s 2023-03-09T20:27:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:25 smithi181 ceph-mon[94189]: pgmap v11048: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 93 KiB/s wr, 7 op/s 2023-03-09T20:27:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:25 smithi038 ceph-mon[92422]: pgmap v11048: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 93 KiB/s wr, 7 op/s 2023-03-09T20:27:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19756.246163 secs 2023-03-09T20:27:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19756.246163 secs 2023-03-09T20:27:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19756.246163 secs 2023-03-09T20:27:27.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:27 smithi122 ceph-mon[93869]: pgmap v11049: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 129 KiB/s wr, 8 op/s 2023-03-09T20:27:27.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:27 smithi181 ceph-mon[94189]: pgmap v11049: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 129 KiB/s wr, 8 op/s 2023-03-09T20:27:27.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:27 smithi038 ceph-mon[92422]: pgmap v11049: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 129 KiB/s wr, 8 op/s 2023-03-09T20:27:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19758.574198 secs 2023-03-09T20:27:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19758.574198 secs 2023-03-09T20:27:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19758.574198 secs 2023-03-09T20:27:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20220.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:27:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:29 smithi038 ceph-mon[92422]: pgmap v11050: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 74 KiB/s wr, 8 op/s 2023-03-09T20:27:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:29 smithi122 ceph-mon[93869]: pgmap v11050: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 74 KiB/s wr, 8 op/s 2023-03-09T20:27:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:29 smithi181 ceph-mon[94189]: pgmap v11050: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 74 KiB/s wr, 8 op/s 2023-03-09T20:27:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19761.246227 secs 2023-03-09T20:27:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19761.246227 secs 2023-03-09T20:27:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19761.246227 secs 2023-03-09T20:27:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:32 smithi122 ceph-mon[93869]: pgmap v11051: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 66 KiB/s wr, 7 op/s 2023-03-09T20:27:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:32 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:27:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:32 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:27:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:32 smithi181 ceph-mon[94189]: pgmap v11051: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 66 KiB/s wr, 7 op/s 2023-03-09T20:27:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:32 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:27:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:32 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:27:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:32 smithi038 ceph-mon[92422]: pgmap v11051: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 66 KiB/s wr, 7 op/s 2023-03-09T20:27:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:32 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:27:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:32 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:27:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:33 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:27:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:33 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:27:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:33 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:27:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:33 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:27:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:33 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:27:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:33 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:27:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20225.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:27:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:34 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19763.574309 secs 2023-03-09T20:27:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:34 smithi122 ceph-mon[93869]: pgmap v11052: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 136 KiB/s wr, 9 op/s 2023-03-09T20:27:34.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:34 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19763.574309 secs 2023-03-09T20:27:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:34 smithi181 ceph-mon[94189]: pgmap v11052: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 136 KiB/s wr, 9 op/s 2023-03-09T20:27:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:34 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19763.574309 secs 2023-03-09T20:27:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:34 smithi038 ceph-mon[92422]: pgmap v11052: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 136 KiB/s wr, 9 op/s 2023-03-09T20:27:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:36 smithi122 ceph-mon[93869]: pgmap v11053: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 136 KiB/s wr, 8 op/s 2023-03-09T20:27:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19766.246287 secs 2023-03-09T20:27:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:27:36.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:36 smithi181 ceph-mon[94189]: pgmap v11053: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 136 KiB/s wr, 8 op/s 2023-03-09T20:27:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19766.246287 secs 2023-03-09T20:27:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:27:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:36 smithi038 ceph-mon[92422]: pgmap v11053: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 136 KiB/s wr, 8 op/s 2023-03-09T20:27:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19766.246287 secs 2023-03-09T20:27:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:27:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:37 smithi122 ceph-mon[93869]: pgmap v11054: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 205 KiB/s wr, 10 op/s 2023-03-09T20:27:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:27:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:27:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:27:37.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:27:37.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:27:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:37 smithi181 ceph-mon[94189]: pgmap v11054: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 205 KiB/s wr, 10 op/s 2023-03-09T20:27:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:27:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:27:37.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:27:37.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:27:37.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:27:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:37 smithi038 ceph-mon[92422]: pgmap v11054: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 205 KiB/s wr, 10 op/s 2023-03-09T20:27:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:27:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:27:37.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:27:37.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:27:37.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:27:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20230.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:27:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:39 smithi038 ceph-mon[92422]: pgmap v11055: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 175 KiB/s wr, 9 op/s 2023-03-09T20:27:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:39 smithi122 ceph-mon[93869]: pgmap v11055: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 175 KiB/s wr, 9 op/s 2023-03-09T20:27:40.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:39 smithi181 ceph-mon[94189]: pgmap v11055: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 175 KiB/s wr, 9 op/s 2023-03-09T20:27:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19771.246365 secs 2023-03-09T20:27:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:41 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:27:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19771.246365 secs 2023-03-09T20:27:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:41 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:27:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19771.246365 secs 2023-03-09T20:27:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:41 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:27:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:42 smithi122 ceph-mon[93869]: pgmap v11056: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 145 KiB/s wr, 7 op/s 2023-03-09T20:27:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:42 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:27:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:42 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:27:42.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:42 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:27:42.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:42 smithi181 ceph-mon[94189]: pgmap v11056: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 145 KiB/s wr, 7 op/s 2023-03-09T20:27:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:42 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:27:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:42 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:27:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:42 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:27:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:42 smithi038 ceph-mon[92422]: pgmap v11056: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 145 KiB/s wr, 7 op/s 2023-03-09T20:27:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:42 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:27:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:42 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:27:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:42 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:27:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20235.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:27:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:44 smithi122 ceph-mon[93869]: pgmap v11057: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 313 KiB/s wr, 10 op/s 2023-03-09T20:27:44.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:44 smithi181 ceph-mon[94189]: pgmap v11057: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 313 KiB/s wr, 10 op/s 2023-03-09T20:27:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:44 smithi038 ceph-mon[92422]: pgmap v11057: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 313 KiB/s wr, 10 op/s 2023-03-09T20:27:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:46 smithi122 ceph-mon[93869]: pgmap v11058: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 255 KiB/s wr, 8 op/s 2023-03-09T20:27:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19776.246463 secs 2023-03-09T20:27:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:46 smithi181 ceph-mon[94189]: pgmap v11058: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 255 KiB/s wr, 8 op/s 2023-03-09T20:27:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19776.246463 secs 2023-03-09T20:27:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:46 smithi038 ceph-mon[92422]: pgmap v11058: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 255 KiB/s wr, 8 op/s 2023-03-09T20:27:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19776.246463 secs 2023-03-09T20:27:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:48 smithi122 ceph-mon[93869]: pgmap v11059: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 601 KiB/s rd, 361 KiB/s wr, 11 op/s 2023-03-09T20:27:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:48 smithi181 ceph-mon[94189]: pgmap v11059: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 601 KiB/s rd, 361 KiB/s wr, 11 op/s 2023-03-09T20:27:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:48 smithi038 ceph-mon[92422]: pgmap v11059: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 601 KiB/s rd, 361 KiB/s wr, 11 op/s 2023-03-09T20:27:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20240.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:27:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:49 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 19778.574661 secs 2023-03-09T20:27:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:49 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 19778.574661 secs 2023-03-09T20:27:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:49 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 19778.574661 secs 2023-03-09T20:27:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:50 smithi122 ceph-mon[93869]: pgmap v11060: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 546 KiB/s rd, 308 KiB/s wr, 10 op/s 2023-03-09T20:27:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:50 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:27:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:50 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:27:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:50 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:27:50.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:50 smithi181 ceph-mon[94189]: pgmap v11060: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 546 KiB/s rd, 308 KiB/s wr, 10 op/s 2023-03-09T20:27:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:50 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:27:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:50 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:27:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:50 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:27:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:50 smithi038 ceph-mon[92422]: pgmap v11060: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 546 KiB/s rd, 308 KiB/s wr, 10 op/s 2023-03-09T20:27:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:50 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:27:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:50 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:27:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:50 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:27:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19781.246528 secs 2023-03-09T20:27:51.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19781.246528 secs 2023-03-09T20:27:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19781.246528 secs 2023-03-09T20:27:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:52 smithi122 ceph-mon[93869]: pgmap v11061: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 782 KiB/s rd, 544 KiB/s wr, 11 op/s 2023-03-09T20:27:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:27:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:27:52.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:52 smithi181 ceph-mon[94189]: pgmap v11061: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 782 KiB/s rd, 544 KiB/s wr, 11 op/s 2023-03-09T20:27:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:27:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:27:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:52 smithi038 ceph-mon[92422]: pgmap v11061: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 782 KiB/s rd, 544 KiB/s wr, 11 op/s 2023-03-09T20:27:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:27:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:27:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:27:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:27:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:27:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:27:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:27:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:27:53.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20245.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:27:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:54 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19783.574754 secs 2023-03-09T20:27:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:54 smithi122 ceph-mon[93869]: pgmap v11062: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 783 KiB/s rd, 544 KiB/s wr, 13 op/s 2023-03-09T20:27:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:54 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:27:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:54 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:27:54.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:54 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19783.574754 secs 2023-03-09T20:27:54.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:54 smithi181 ceph-mon[94189]: pgmap v11062: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 783 KiB/s rd, 544 KiB/s wr, 13 op/s 2023-03-09T20:27:54.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:54 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:27:54.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:54 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:27:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:54 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19783.574754 secs 2023-03-09T20:27:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:54 smithi038 ceph-mon[92422]: pgmap v11062: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 783 KiB/s rd, 544 KiB/s wr, 13 op/s 2023-03-09T20:27:54.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:54 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:27:54.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:54 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:27:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:55 smithi122 ceph-mon[93869]: pgmap v11063: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 630 KiB/s rd, 392 KiB/s wr, 11 op/s 2023-03-09T20:27:55.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:55 smithi181 ceph-mon[94189]: pgmap v11063: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 630 KiB/s rd, 392 KiB/s wr, 11 op/s 2023-03-09T20:27:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:55 smithi038 ceph-mon[92422]: pgmap v11063: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 630 KiB/s rd, 392 KiB/s wr, 11 op/s 2023-03-09T20:27:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19786.246588 secs 2023-03-09T20:27:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19786.246588 secs 2023-03-09T20:27:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19786.246588 secs 2023-03-09T20:27:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:57 smithi122 ceph-mon[93869]: pgmap v11064: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 646 KiB/s rd, 376 KiB/s wr, 13 op/s 2023-03-09T20:27:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:57 smithi181 ceph-mon[94189]: pgmap v11064: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 646 KiB/s rd, 376 KiB/s wr, 13 op/s 2023-03-09T20:27:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:57 smithi038 ceph-mon[92422]: pgmap v11064: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 646 KiB/s rd, 376 KiB/s wr, 13 op/s 2023-03-09T20:27:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:27:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19788.574873 secs 2023-03-09T20:27:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:27:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19788.574873 secs 2023-03-09T20:27:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:27:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19788.574873 secs 2023-03-09T20:27:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20250.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:28:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:00 smithi122 ceph-mon[93869]: pgmap v11065: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 383 KiB/s wr, 12 op/s 2023-03-09T20:28:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:28:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:28:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:00 smithi181 ceph-mon[94189]: pgmap v11065: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 383 KiB/s wr, 12 op/s 2023-03-09T20:28:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:28:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:28:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:00 smithi038 ceph-mon[92422]: pgmap v11065: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 383 KiB/s wr, 12 op/s 2023-03-09T20:28:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:28:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:28:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19791.246648 secs 2023-03-09T20:28:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:28:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:28:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19791.246648 secs 2023-03-09T20:28:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:28:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:28:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19791.246648 secs 2023-03-09T20:28:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:28:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:28:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:02 smithi122 ceph-mon[93869]: pgmap v11066: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 367 KiB/s wr, 10 op/s 2023-03-09T20:28:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:02 smithi181 ceph-mon[94189]: pgmap v11066: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 367 KiB/s wr, 10 op/s 2023-03-09T20:28:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:02 smithi038 ceph-mon[92422]: pgmap v11066: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 367 KiB/s wr, 10 op/s 2023-03-09T20:28:03.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20255.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:28:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:04 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19793.574985 secs 2023-03-09T20:28:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:04 smithi122 ceph-mon[93869]: pgmap v11067: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 281 KiB/s wr, 14 op/s 2023-03-09T20:28:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:04 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19793.574985 secs 2023-03-09T20:28:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:04 smithi181 ceph-mon[94189]: pgmap v11067: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 281 KiB/s wr, 14 op/s 2023-03-09T20:28:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:04 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19793.574985 secs 2023-03-09T20:28:04.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:04 smithi038 ceph-mon[92422]: pgmap v11067: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 281 KiB/s wr, 14 op/s 2023-03-09T20:28:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:06 smithi122 ceph-mon[93869]: pgmap v11068: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 303 KiB/s wr, 24 op/s 2023-03-09T20:28:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19796.246720 secs 2023-03-09T20:28:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:06 smithi181 ceph-mon[94189]: pgmap v11068: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 303 KiB/s wr, 24 op/s 2023-03-09T20:28:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19796.246720 secs 2023-03-09T20:28:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:06 smithi038 ceph-mon[92422]: pgmap v11068: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 303 KiB/s wr, 24 op/s 2023-03-09T20:28:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19796.246720 secs 2023-03-09T20:28:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:08 smithi122 ceph-mon[93869]: pgmap v11069: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 298 KiB/s wr, 26 op/s 2023-03-09T20:28:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:08 smithi181 ceph-mon[94189]: pgmap v11069: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 298 KiB/s wr, 26 op/s 2023-03-09T20:28:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:08 smithi038 ceph-mon[92422]: pgmap v11069: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 298 KiB/s wr, 26 op/s 2023-03-09T20:28:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20260.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:28:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:09 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19798.575123 secs 2023-03-09T20:28:09.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:09 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19798.575123 secs 2023-03-09T20:28:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:09 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19798.575123 secs 2023-03-09T20:28:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:10 smithi122 ceph-mon[93869]: pgmap v11070: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 422 KiB/s wr, 25 op/s 2023-03-09T20:28:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:10 smithi122 ceph-mon[93869]: osdmap e905: 12 total, 12 up, 12 in 2023-03-09T20:28:10.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:10 smithi181 ceph-mon[94189]: pgmap v11070: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 422 KiB/s wr, 25 op/s 2023-03-09T20:28:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:10 smithi181 ceph-mon[94189]: osdmap e905: 12 total, 12 up, 12 in 2023-03-09T20:28:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:10 smithi038 ceph-mon[92422]: pgmap v11070: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 422 KiB/s wr, 25 op/s 2023-03-09T20:28:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:10 smithi038 ceph-mon[92422]: osdmap e905: 12 total, 12 up, 12 in 2023-03-09T20:28:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19801.246778 secs 2023-03-09T20:28:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19801.246778 secs 2023-03-09T20:28:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19801.246778 secs 2023-03-09T20:28:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:12 smithi122 ceph-mon[93869]: pgmap v11072: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 329 KiB/s rd, 370 KiB/s wr, 28 op/s 2023-03-09T20:28:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:12 smithi181 ceph-mon[94189]: pgmap v11072: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 329 KiB/s rd, 370 KiB/s wr, 28 op/s 2023-03-09T20:28:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:12 smithi038 ceph-mon[92422]: pgmap v11072: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 329 KiB/s rd, 370 KiB/s wr, 28 op/s 2023-03-09T20:28:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:13 smithi122 ceph-mon[93869]: pgmap v11073: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 190 KiB/s wr, 25 op/s 2023-03-09T20:28:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:13 smithi181 ceph-mon[94189]: pgmap v11073: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 190 KiB/s wr, 25 op/s 2023-03-09T20:28:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:13 smithi038 ceph-mon[92422]: pgmap v11073: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 190 KiB/s wr, 25 op/s 2023-03-09T20:28:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20265.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:28:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:14 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19803.575272 secs 2023-03-09T20:28:14.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:14 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19803.575272 secs 2023-03-09T20:28:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:14 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19803.575272 secs 2023-03-09T20:28:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:15 smithi122 ceph-mon[93869]: pgmap v11074: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 164 KiB/s wr, 10 op/s 2023-03-09T20:28:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:15 smithi181 ceph-mon[94189]: pgmap v11074: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 164 KiB/s wr, 10 op/s 2023-03-09T20:28:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:15 smithi038 ceph-mon[92422]: pgmap v11074: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 164 KiB/s wr, 10 op/s 2023-03-09T20:28:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19806.246838 secs 2023-03-09T20:28:16.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19806.246838 secs 2023-03-09T20:28:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19806.246838 secs 2023-03-09T20:28:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:17 smithi122 ceph-mon[93869]: pgmap v11075: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 150 KiB/s wr, 9 op/s 2023-03-09T20:28:17.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:17 smithi181 ceph-mon[94189]: pgmap v11075: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 150 KiB/s wr, 9 op/s 2023-03-09T20:28:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:17 smithi038 ceph-mon[92422]: pgmap v11075: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 150 KiB/s wr, 9 op/s 2023-03-09T20:28:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19808.575341 secs 2023-03-09T20:28:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19808.575341 secs 2023-03-09T20:28:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19808.575341 secs 2023-03-09T20:28:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20270.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:28:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:20 smithi122 ceph-mon[93869]: pgmap v11076: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 60 KiB/s wr, 9 op/s 2023-03-09T20:28:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:20 smithi122 ceph-mon[93869]: osdmap e906: 12 total, 12 up, 12 in 2023-03-09T20:28:20.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:20 smithi181 ceph-mon[94189]: pgmap v11076: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 60 KiB/s wr, 9 op/s 2023-03-09T20:28:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:20 smithi181 ceph-mon[94189]: osdmap e906: 12 total, 12 up, 12 in 2023-03-09T20:28:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:20 smithi038 ceph-mon[92422]: pgmap v11076: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 60 KiB/s wr, 9 op/s 2023-03-09T20:28:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:20 smithi038 ceph-mon[92422]: osdmap e906: 12 total, 12 up, 12 in 2023-03-09T20:28:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19811.246897 secs 2023-03-09T20:28:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19811.246897 secs 2023-03-09T20:28:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19811.246897 secs 2023-03-09T20:28:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:22 smithi122 ceph-mon[93869]: pgmap v11078: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 60 KiB/s wr, 8 op/s 2023-03-09T20:28:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:22 smithi181 ceph-mon[94189]: pgmap v11078: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 60 KiB/s wr, 8 op/s 2023-03-09T20:28:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:22 smithi038 ceph-mon[92422]: pgmap v11078: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 60 KiB/s wr, 8 op/s 2023-03-09T20:28:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20275.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:28:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:24 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19813.575408 secs 2023-03-09T20:28:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:24 smithi122 ceph-mon[93869]: pgmap v11079: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 61 KiB/s wr, 8 op/s 2023-03-09T20:28:24.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:24 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19813.575408 secs 2023-03-09T20:28:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:24 smithi181 ceph-mon[94189]: pgmap v11079: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 61 KiB/s wr, 8 op/s 2023-03-09T20:28:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:24 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19813.575408 secs 2023-03-09T20:28:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:24 smithi038 ceph-mon[92422]: pgmap v11079: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 61 KiB/s wr, 8 op/s 2023-03-09T20:28:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:26 smithi122 ceph-mon[93869]: pgmap v11080: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 150 KiB/s wr, 8 op/s 2023-03-09T20:28:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19816.246956 secs 2023-03-09T20:28:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:26 smithi181 ceph-mon[94189]: pgmap v11080: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 150 KiB/s wr, 8 op/s 2023-03-09T20:28:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19816.246956 secs 2023-03-09T20:28:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:26 smithi038 ceph-mon[92422]: pgmap v11080: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 150 KiB/s wr, 8 op/s 2023-03-09T20:28:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19816.246956 secs 2023-03-09T20:28:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:28 smithi122 ceph-mon[93869]: pgmap v11081: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 150 KiB/s wr, 8 op/s 2023-03-09T20:28:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:28 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:28:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:28 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:28:28.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:28 smithi181 ceph-mon[94189]: pgmap v11081: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 150 KiB/s wr, 8 op/s 2023-03-09T20:28:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:28 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:28:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:28 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:28:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:28 smithi038 ceph-mon[92422]: pgmap v11081: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 150 KiB/s wr, 8 op/s 2023-03-09T20:28:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:28 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:28:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:28 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:28:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20280.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:28:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:29 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19818.575526 secs 2023-03-09T20:28:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:29 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:28:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:29 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:28:29.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:29 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19818.575526 secs 2023-03-09T20:28:29.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:29 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:28:29.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:29 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:28:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:29 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19818.575526 secs 2023-03-09T20:28:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:29 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:28:29.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:29 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:28:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:30 smithi122 ceph-mon[93869]: pgmap v11082: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 91 KiB/s wr, 8 op/s 2023-03-09T20:28:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:30 smithi181 ceph-mon[94189]: pgmap v11082: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 91 KiB/s wr, 8 op/s 2023-03-09T20:28:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:30 smithi038 ceph-mon[92422]: pgmap v11082: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 91 KiB/s wr, 8 op/s 2023-03-09T20:28:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19821.246984 secs 2023-03-09T20:28:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19821.246984 secs 2023-03-09T20:28:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19821.246984 secs 2023-03-09T20:28:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:32 smithi122 ceph-mon[93869]: pgmap v11083: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 80 KiB/s wr, 7 op/s 2023-03-09T20:28:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:32 smithi181 ceph-mon[94189]: pgmap v11083: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 80 KiB/s wr, 7 op/s 2023-03-09T20:28:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:32 smithi038 ceph-mon[92422]: pgmap v11083: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 80 KiB/s wr, 7 op/s 2023-03-09T20:28:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:33 smithi122 ceph-mon[93869]: pgmap v11084: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 76 KiB/s wr, 9 op/s 2023-03-09T20:28:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:33 smithi181 ceph-mon[94189]: pgmap v11084: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 76 KiB/s wr, 9 op/s 2023-03-09T20:28:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:33 smithi038 ceph-mon[92422]: pgmap v11084: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 76 KiB/s wr, 9 op/s 2023-03-09T20:28:33.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20285.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:28:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:34 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19823.575637 secs 2023-03-09T20:28:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:34 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19823.575637 secs 2023-03-09T20:28:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:34 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19823.575637 secs 2023-03-09T20:28:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:35 smithi122 ceph-mon[93869]: pgmap v11085: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 75 KiB/s wr, 8 op/s 2023-03-09T20:28:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:35 smithi181 ceph-mon[94189]: pgmap v11085: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 75 KiB/s wr, 8 op/s 2023-03-09T20:28:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:35 smithi038 ceph-mon[92422]: pgmap v11085: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 75 KiB/s wr, 8 op/s 2023-03-09T20:28:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19826.247044 secs 2023-03-09T20:28:36.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19826.247044 secs 2023-03-09T20:28:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19826.247044 secs 2023-03-09T20:28:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:37 smithi122 ceph-mon[93869]: pgmap v11086: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 597 B/s wr, 8 op/s 2023-03-09T20:28:37.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:37 smithi181 ceph-mon[94189]: pgmap v11086: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 597 B/s wr, 8 op/s 2023-03-09T20:28:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:37 smithi038 ceph-mon[92422]: pgmap v11086: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 597 B/s wr, 8 op/s 2023-03-09T20:28:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:38 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:28:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:38 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:28:38.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:38 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:28:38.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:38 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:28:38.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:38 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:28:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:38 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:28:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:38 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:28:38.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:38 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:28:38.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:38 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:28:38.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:38 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:28:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:38 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:28:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:38 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:28:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:38 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:28:38.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:38 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:28:38.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:38 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:28:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20290.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:28:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:39 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19828.575779 secs 2023-03-09T20:28:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:39 smithi122 ceph-mon[93869]: pgmap v11087: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 767 B/s wr, 8 op/s 2023-03-09T20:28:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:28:39.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:28:39.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:28:39.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:39 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19828.575779 secs 2023-03-09T20:28:39.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:39 smithi181 ceph-mon[94189]: pgmap v11087: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 767 B/s wr, 8 op/s 2023-03-09T20:28:39.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:28:39.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:28:39.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:28:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:39 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19828.575779 secs 2023-03-09T20:28:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:39 smithi038 ceph-mon[92422]: pgmap v11087: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 767 B/s wr, 8 op/s 2023-03-09T20:28:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:28:39.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:28:39.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:28:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19831.247104 secs 2023-03-09T20:28:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19831.247104 secs 2023-03-09T20:28:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19831.247104 secs 2023-03-09T20:28:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:41 smithi122 ceph-mon[93869]: pgmap v11088: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T20:28:41.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:41 smithi181 ceph-mon[94189]: pgmap v11088: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T20:28:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:41 smithi038 ceph-mon[92422]: pgmap v11088: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T20:28:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19833.575873 secs 2023-03-09T20:28:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:43 smithi122 ceph-mon[93869]: pgmap v11089: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 682 B/s wr, 8 op/s 2023-03-09T20:28:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19833.575873 secs 2023-03-09T20:28:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:43 smithi181 ceph-mon[94189]: pgmap v11089: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 682 B/s wr, 8 op/s 2023-03-09T20:28:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19833.575873 secs 2023-03-09T20:28:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:43 smithi038 ceph-mon[92422]: pgmap v11089: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 682 B/s wr, 8 op/s 2023-03-09T20:28:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20295.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:28:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:46 smithi122 ceph-mon[93869]: pgmap v11090: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 241 KiB/s wr, 8 op/s 2023-03-09T20:28:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19836.247165 secs 2023-03-09T20:28:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:46 smithi181 ceph-mon[94189]: pgmap v11090: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 241 KiB/s wr, 8 op/s 2023-03-09T20:28:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19836.247165 secs 2023-03-09T20:28:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:46 smithi038 ceph-mon[92422]: pgmap v11090: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 241 KiB/s wr, 8 op/s 2023-03-09T20:28:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19836.247165 secs 2023-03-09T20:28:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:48 smithi122 ceph-mon[93869]: pgmap v11091: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 358 KiB/s wr, 10 op/s 2023-03-09T20:28:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:48 smithi181 ceph-mon[94189]: pgmap v11091: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 358 KiB/s wr, 10 op/s 2023-03-09T20:28:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:48 smithi038 ceph-mon[92422]: pgmap v11091: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 358 KiB/s wr, 10 op/s 2023-03-09T20:28:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20300.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:28:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:28:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:28:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:49 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:28:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:28:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:28:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:49 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:28:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:28:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:28:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:49 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:28:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:50 smithi122 ceph-mon[93869]: pgmap v11092: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 359 KiB/s wr, 10 op/s 2023-03-09T20:28:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:50 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:28:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:50 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:28:50.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:50 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:28:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:50 smithi181 ceph-mon[94189]: pgmap v11092: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 359 KiB/s wr, 10 op/s 2023-03-09T20:28:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:50 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:28:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:50 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:28:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:50 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:28:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:50 smithi038 ceph-mon[92422]: pgmap v11092: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 359 KiB/s wr, 10 op/s 2023-03-09T20:28:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:50 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:28:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:50 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:28:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:50 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:28:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19841.247224 secs 2023-03-09T20:28:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19841.247224 secs 2023-03-09T20:28:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19841.247224 secs 2023-03-09T20:28:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:52 smithi122 ceph-mon[93869]: pgmap v11093: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 358 KiB/s wr, 9 op/s 2023-03-09T20:28:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:28:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:28:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:52 smithi181 ceph-mon[94189]: pgmap v11093: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 358 KiB/s wr, 9 op/s 2023-03-09T20:28:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:28:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:28:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:52 smithi038 ceph-mon[92422]: pgmap v11093: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 358 KiB/s wr, 9 op/s 2023-03-09T20:28:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:28:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:28:53.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20305.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:28:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:54 smithi122 ceph-mon[93869]: pgmap v11094: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 389 KiB/s wr, 12 op/s 2023-03-09T20:28:54.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:54 smithi181 ceph-mon[94189]: pgmap v11094: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 389 KiB/s wr, 12 op/s 2023-03-09T20:28:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:54 smithi038 ceph-mon[92422]: pgmap v11094: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 389 KiB/s wr, 12 op/s 2023-03-09T20:28:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:56 smithi122 ceph-mon[93869]: pgmap v11095: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 389 KiB/s wr, 11 op/s 2023-03-09T20:28:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19846.247286 secs 2023-03-09T20:28:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:56 smithi181 ceph-mon[94189]: pgmap v11095: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 389 KiB/s wr, 11 op/s 2023-03-09T20:28:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19846.247286 secs 2023-03-09T20:28:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:56 smithi038 ceph-mon[92422]: pgmap v11095: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 389 KiB/s wr, 11 op/s 2023-03-09T20:28:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19846.247286 secs 2023-03-09T20:28:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:28:57 smithi122 ceph-mon[93869]: pgmap v11096: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 423 KiB/s wr, 14 op/s 2023-03-09T20:28:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:28:57 smithi181 ceph-mon[94189]: pgmap v11096: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 423 KiB/s wr, 14 op/s 2023-03-09T20:28:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:28:57 smithi038 ceph-mon[92422]: pgmap v11096: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 423 KiB/s wr, 14 op/s 2023-03-09T20:28:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20310.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:29:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:00 smithi122 ceph-mon[93869]: pgmap v11097: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 307 KiB/s wr, 11 op/s 2023-03-09T20:29:00.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:00 smithi181 ceph-mon[94189]: pgmap v11097: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 307 KiB/s wr, 11 op/s 2023-03-09T20:29:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:00 smithi038 ceph-mon[92422]: pgmap v11097: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 307 KiB/s wr, 11 op/s 2023-03-09T20:29:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19851.247345 secs 2023-03-09T20:29:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:29:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:29:01.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19851.247345 secs 2023-03-09T20:29:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:29:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:29:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19851.247345 secs 2023-03-09T20:29:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:29:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:29:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:02 smithi122 ceph-mon[93869]: pgmap v11098: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 306 KiB/s wr, 9 op/s 2023-03-09T20:29:02.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:02 smithi181 ceph-mon[94189]: pgmap v11098: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 306 KiB/s wr, 9 op/s 2023-03-09T20:29:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:02 smithi038 ceph-mon[92422]: pgmap v11098: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 306 KiB/s wr, 9 op/s 2023-03-09T20:29:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20315.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:29:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:04 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19853.576327 secs 2023-03-09T20:29:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:04 smithi122 ceph-mon[93869]: pgmap v11099: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 474 KiB/s wr, 14 op/s 2023-03-09T20:29:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:04 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19853.576327 secs 2023-03-09T20:29:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:04 smithi181 ceph-mon[94189]: pgmap v11099: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 474 KiB/s wr, 14 op/s 2023-03-09T20:29:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:04 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19853.576327 secs 2023-03-09T20:29:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:04 smithi038 ceph-mon[92422]: pgmap v11099: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 474 KiB/s wr, 14 op/s 2023-03-09T20:29:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:05 smithi122 ceph-mon[93869]: pgmap v11100: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 453 KiB/s rd, 465 KiB/s wr, 23 op/s 2023-03-09T20:29:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:05 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:29:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:05 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:29:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:05 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:29:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:05 smithi181 ceph-mon[94189]: pgmap v11100: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 453 KiB/s rd, 465 KiB/s wr, 23 op/s 2023-03-09T20:29:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:05 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:29:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:05 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:29:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:05 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:29:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:05 smithi038 ceph-mon[92422]: pgmap v11100: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 453 KiB/s rd, 465 KiB/s wr, 23 op/s 2023-03-09T20:29:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:05 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:29:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:05 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:29:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:05 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:29:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19856.247405 secs 2023-03-09T20:29:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19856.247405 secs 2023-03-09T20:29:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19856.247405 secs 2023-03-09T20:29:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:07 smithi122 ceph-mon[93869]: pgmap v11101: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 529 KiB/s wr, 28 op/s 2023-03-09T20:29:07.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:07 smithi181 ceph-mon[94189]: pgmap v11101: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 529 KiB/s wr, 28 op/s 2023-03-09T20:29:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:07 smithi038 ceph-mon[92422]: pgmap v11101: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 529 KiB/s wr, 28 op/s 2023-03-09T20:29:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:29:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:29:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:29:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:29:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:29:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:29:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20320.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:29:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:09 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19858.576442 secs 2023-03-09T20:29:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:09 smithi122 ceph-mon[93869]: pgmap v11102: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 254 KiB/s wr, 24 op/s 2023-03-09T20:29:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:09 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19858.576442 secs 2023-03-09T20:29:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:09 smithi181 ceph-mon[94189]: pgmap v11102: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 254 KiB/s wr, 24 op/s 2023-03-09T20:29:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:09 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19858.576442 secs 2023-03-09T20:29:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:09 smithi038 ceph-mon[92422]: pgmap v11102: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 254 KiB/s wr, 24 op/s 2023-03-09T20:29:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:10 smithi122 ceph-mon[93869]: osdmap e907: 12 total, 12 up, 12 in 2023-03-09T20:29:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19861.247467 secs 2023-03-09T20:29:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:10 smithi181 ceph-mon[94189]: osdmap e907: 12 total, 12 up, 12 in 2023-03-09T20:29:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19861.247467 secs 2023-03-09T20:29:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:10 smithi038 ceph-mon[92422]: osdmap e907: 12 total, 12 up, 12 in 2023-03-09T20:29:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19861.247467 secs 2023-03-09T20:29:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:11 smithi122 ceph-mon[93869]: pgmap v11104: 129 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+snaptrim, 127 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 305 KiB/s wr, 27 op/s 2023-03-09T20:29:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:11 smithi181 ceph-mon[94189]: pgmap v11104: 129 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+snaptrim, 127 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 305 KiB/s wr, 27 op/s 2023-03-09T20:29:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:11 smithi038 ceph-mon[92422]: pgmap v11104: 129 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+snaptrim, 127 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 305 KiB/s wr, 27 op/s 2023-03-09T20:29:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19863.576523 secs 2023-03-09T20:29:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:13 smithi122 ceph-mon[93869]: pgmap v11105: 129 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+snaptrim, 127 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 342 KiB/s wr, 27 op/s 2023-03-09T20:29:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19863.576523 secs 2023-03-09T20:29:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:13 smithi181 ceph-mon[94189]: pgmap v11105: 129 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+snaptrim, 127 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 342 KiB/s wr, 27 op/s 2023-03-09T20:29:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19863.576523 secs 2023-03-09T20:29:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:13 smithi038 ceph-mon[92422]: pgmap v11105: 129 pgs: 1 active+clean+snaptrim_wait, 1 active+clean+snaptrim, 127 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 342 KiB/s wr, 27 op/s 2023-03-09T20:29:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20325.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:29:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:16 smithi122 ceph-mon[93869]: pgmap v11106: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 316 KiB/s wr, 13 op/s 2023-03-09T20:29:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19866.247527 secs 2023-03-09T20:29:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:29:16.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:29:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:16 smithi181 ceph-mon[94189]: pgmap v11106: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 316 KiB/s wr, 13 op/s 2023-03-09T20:29:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19866.247527 secs 2023-03-09T20:29:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:29:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:29:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:16 smithi038 ceph-mon[92422]: pgmap v11106: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 316 KiB/s wr, 13 op/s 2023-03-09T20:29:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19866.247527 secs 2023-03-09T20:29:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:29:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:29:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:29:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:29:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:29:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:29:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:29:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:29:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:18 smithi122 ceph-mon[93869]: pgmap v11107: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 239 KiB/s wr, 9 op/s 2023-03-09T20:29:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:18 smithi181 ceph-mon[94189]: pgmap v11107: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 239 KiB/s wr, 9 op/s 2023-03-09T20:29:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:18 smithi038 ceph-mon[92422]: pgmap v11107: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 239 KiB/s wr, 9 op/s 2023-03-09T20:29:18.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20330.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:29:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:19 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19868.576639 secs 2023-03-09T20:29:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:19 smithi122 ceph-mon[93869]: osdmap e908: 12 total, 12 up, 12 in 2023-03-09T20:29:19.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:19 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19868.576639 secs 2023-03-09T20:29:19.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:19 smithi181 ceph-mon[94189]: osdmap e908: 12 total, 12 up, 12 in 2023-03-09T20:29:19.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:19 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19868.576639 secs 2023-03-09T20:29:19.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:19 smithi038 ceph-mon[92422]: osdmap e908: 12 total, 12 up, 12 in 2023-03-09T20:29:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:20 smithi122 ceph-mon[93869]: pgmap v11108: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 239 KiB/s wr, 9 op/s 2023-03-09T20:29:20.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:20 smithi181 ceph-mon[94189]: pgmap v11108: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 239 KiB/s wr, 9 op/s 2023-03-09T20:29:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:20 smithi038 ceph-mon[92422]: pgmap v11108: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 239 KiB/s wr, 9 op/s 2023-03-09T20:29:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19871.247589 secs 2023-03-09T20:29:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:21 smithi122 ceph-mon[93869]: pgmap v11110: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 240 KiB/s wr, 9 op/s 2023-03-09T20:29:21.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19871.247589 secs 2023-03-09T20:29:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:21 smithi181 ceph-mon[94189]: pgmap v11110: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 240 KiB/s wr, 9 op/s 2023-03-09T20:29:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19871.247589 secs 2023-03-09T20:29:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:21 smithi038 ceph-mon[92422]: pgmap v11110: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 240 KiB/s wr, 9 op/s 2023-03-09T20:29:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19873.576778 secs 2023-03-09T20:29:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:23 smithi122 ceph-mon[93869]: pgmap v11111: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 21 KiB/s wr, 8 op/s 2023-03-09T20:29:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19873.576778 secs 2023-03-09T20:29:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:23 smithi181 ceph-mon[94189]: pgmap v11111: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 21 KiB/s wr, 8 op/s 2023-03-09T20:29:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19873.576778 secs 2023-03-09T20:29:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:23 smithi038 ceph-mon[92422]: pgmap v11111: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 21 KiB/s wr, 8 op/s 2023-03-09T20:29:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20335.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:29:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:26 smithi122 ceph-mon[93869]: pgmap v11112: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 21 KiB/s wr, 8 op/s 2023-03-09T20:29:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19876.247648 secs 2023-03-09T20:29:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:26 smithi181 ceph-mon[94189]: pgmap v11112: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 21 KiB/s wr, 8 op/s 2023-03-09T20:29:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19876.247648 secs 2023-03-09T20:29:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:26 smithi038 ceph-mon[92422]: pgmap v11112: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 21 KiB/s wr, 8 op/s 2023-03-09T20:29:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19876.247648 secs 2023-03-09T20:29:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:28 smithi122 ceph-mon[93869]: pgmap v11113: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 110 KiB/s wr, 8 op/s 2023-03-09T20:29:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:28 smithi181 ceph-mon[94189]: pgmap v11113: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 110 KiB/s wr, 8 op/s 2023-03-09T20:29:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:28 smithi038 ceph-mon[92422]: pgmap v11113: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 110 KiB/s wr, 8 op/s 2023-03-09T20:29:28.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20340.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:29:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:29 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19878.576923 secs 2023-03-09T20:29:29.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:29 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19878.576923 secs 2023-03-09T20:29:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:29 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19878.576923 secs 2023-03-09T20:29:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:30 smithi122 ceph-mon[93869]: pgmap v11114: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 110 KiB/s wr, 9 op/s 2023-03-09T20:29:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:30 smithi181 ceph-mon[94189]: pgmap v11114: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 110 KiB/s wr, 9 op/s 2023-03-09T20:29:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:30 smithi038 ceph-mon[92422]: pgmap v11114: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 110 KiB/s wr, 9 op/s 2023-03-09T20:29:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19881.247708 secs 2023-03-09T20:29:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19881.247708 secs 2023-03-09T20:29:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19881.247708 secs 2023-03-09T20:29:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:32 smithi122 ceph-mon[93869]: pgmap v11115: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 97 KiB/s wr, 8 op/s 2023-03-09T20:29:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:32 smithi181 ceph-mon[94189]: pgmap v11115: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 97 KiB/s wr, 8 op/s 2023-03-09T20:29:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:32 smithi038 ceph-mon[92422]: pgmap v11115: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 97 KiB/s wr, 8 op/s 2023-03-09T20:29:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20345.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:29:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:34 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19883.577057 secs 2023-03-09T20:29:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:34 smithi122 ceph-mon[93869]: pgmap v11116: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 KiB/s wr, 11 op/s 2023-03-09T20:29:34.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:34 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19883.577057 secs 2023-03-09T20:29:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:34 smithi181 ceph-mon[94189]: pgmap v11116: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 KiB/s wr, 11 op/s 2023-03-09T20:29:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:34 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19883.577057 secs 2023-03-09T20:29:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:34 smithi038 ceph-mon[92422]: pgmap v11116: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 KiB/s wr, 11 op/s 2023-03-09T20:29:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:35 smithi122 ceph-mon[93869]: pgmap v11117: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 76 KiB/s wr, 7 op/s 2023-03-09T20:29:35.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:35 smithi181 ceph-mon[94189]: pgmap v11117: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 76 KiB/s wr, 7 op/s 2023-03-09T20:29:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:35 smithi038 ceph-mon[92422]: pgmap v11117: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 76 KiB/s wr, 7 op/s 2023-03-09T20:29:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19886.247768 secs 2023-03-09T20:29:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19886.247768 secs 2023-03-09T20:29:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19886.247768 secs 2023-03-09T20:29:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:37 smithi122 ceph-mon[93869]: pgmap v11118: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 220 KiB/s wr, 10 op/s 2023-03-09T20:29:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:37 smithi181 ceph-mon[94189]: pgmap v11118: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 220 KiB/s wr, 10 op/s 2023-03-09T20:29:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:37 smithi038 ceph-mon[92422]: pgmap v11118: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 220 KiB/s wr, 10 op/s 2023-03-09T20:29:38.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20350.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:29:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:39 smithi122 ceph-mon[93869]: pgmap v11119: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 146 KiB/s wr, 7 op/s 2023-03-09T20:29:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:29:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:29:39.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:29:39.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:29:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:39 smithi181 ceph-mon[94189]: pgmap v11119: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 146 KiB/s wr, 7 op/s 2023-03-09T20:29:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:29:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:29:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:29:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:29:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:39 smithi038 ceph-mon[92422]: pgmap v11119: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 146 KiB/s wr, 7 op/s 2023-03-09T20:29:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:29:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:29:39.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:29:39.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:29:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19891.247828 secs 2023-03-09T20:29:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:40 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:29:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19891.247828 secs 2023-03-09T20:29:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:40 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:29:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19891.247828 secs 2023-03-09T20:29:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:40 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:29:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:41 smithi122 ceph-mon[93869]: pgmap v11120: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 146 KiB/s wr, 7 op/s 2023-03-09T20:29:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:41 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:29:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:29:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:29:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:41 smithi181 ceph-mon[94189]: pgmap v11120: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 146 KiB/s wr, 7 op/s 2023-03-09T20:29:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:41 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:29:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:29:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:29:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:41 smithi038 ceph-mon[92422]: pgmap v11120: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 146 KiB/s wr, 7 op/s 2023-03-09T20:29:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:41 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:29:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:29:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:29:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:43 smithi122 ceph-mon[93869]: pgmap v11121: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 207 KiB/s wr, 10 op/s 2023-03-09T20:29:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:43 smithi181 ceph-mon[94189]: pgmap v11121: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 207 KiB/s wr, 10 op/s 2023-03-09T20:29:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:43 smithi038 ceph-mon[92422]: pgmap v11121: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 207 KiB/s wr, 10 op/s 2023-03-09T20:29:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20355.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:29:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:46 smithi122 ceph-mon[93869]: pgmap v11122: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 206 KiB/s wr, 7 op/s 2023-03-09T20:29:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19896.247914 secs 2023-03-09T20:29:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:46 smithi181 ceph-mon[94189]: pgmap v11122: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 206 KiB/s wr, 7 op/s 2023-03-09T20:29:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19896.247914 secs 2023-03-09T20:29:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:46 smithi038 ceph-mon[92422]: pgmap v11122: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 206 KiB/s wr, 7 op/s 2023-03-09T20:29:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19896.247914 secs 2023-03-09T20:29:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:48 smithi122 ceph-mon[93869]: pgmap v11123: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 527 KiB/s rd, 323 KiB/s wr, 11 op/s 2023-03-09T20:29:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:48 smithi181 ceph-mon[94189]: pgmap v11123: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 527 KiB/s rd, 323 KiB/s wr, 11 op/s 2023-03-09T20:29:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:48 smithi038 ceph-mon[92422]: pgmap v11123: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 527 KiB/s rd, 323 KiB/s wr, 11 op/s 2023-03-09T20:29:48.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20360.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:29:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:50 smithi122 ceph-mon[93869]: pgmap v11124: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 178 KiB/s wr, 7 op/s 2023-03-09T20:29:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:50 smithi181 ceph-mon[94189]: pgmap v11124: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 178 KiB/s wr, 7 op/s 2023-03-09T20:29:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:50 smithi038 ceph-mon[92422]: pgmap v11124: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 178 KiB/s wr, 7 op/s 2023-03-09T20:29:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19901.247980 secs 2023-03-09T20:29:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19901.247980 secs 2023-03-09T20:29:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19901.247980 secs 2023-03-09T20:29:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:52 smithi122 ceph-mon[93869]: pgmap v11125: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 178 KiB/s wr, 7 op/s 2023-03-09T20:29:52.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:29:52.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:29:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:52 smithi181 ceph-mon[94189]: pgmap v11125: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 178 KiB/s wr, 7 op/s 2023-03-09T20:29:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:29:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:29:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:52 smithi038 ceph-mon[92422]: pgmap v11125: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 178 KiB/s wr, 7 op/s 2023-03-09T20:29:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:29:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:29:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:29:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:29:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:53 smithi122 ceph-mon[93869]: pgmap v11126: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 543 KiB/s rd, 326 KiB/s wr, 12 op/s 2023-03-09T20:29:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:29:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:29:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:53 smithi181 ceph-mon[94189]: pgmap v11126: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 543 KiB/s rd, 326 KiB/s wr, 12 op/s 2023-03-09T20:29:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:29:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:29:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:53 smithi038 ceph-mon[92422]: pgmap v11126: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 543 KiB/s rd, 326 KiB/s wr, 12 op/s 2023-03-09T20:29:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20365.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:29:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:54 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 19903.577439 secs 2023-03-09T20:29:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:54 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:29:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:54 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:29:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:54 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:29:54.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:54 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 19903.577439 secs 2023-03-09T20:29:54.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:54 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:29:54.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:54 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:29:54.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:54 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:29:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:54 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 19903.577439 secs 2023-03-09T20:29:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:54 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:29:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:54 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:29:54.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:54 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:29:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:55 smithi122 ceph-mon[93869]: pgmap v11127: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 265 KiB/s wr, 9 op/s 2023-03-09T20:29:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:55 smithi181 ceph-mon[94189]: pgmap v11127: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 265 KiB/s wr, 9 op/s 2023-03-09T20:29:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:55 smithi038 ceph-mon[92422]: pgmap v11127: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 265 KiB/s wr, 9 op/s 2023-03-09T20:29:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19906.248041 secs 2023-03-09T20:29:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19906.248041 secs 2023-03-09T20:29:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19906.248041 secs 2023-03-09T20:29:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:29:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:29:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:57 smithi122 ceph-mon[93869]: pgmap v11128: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 480 KiB/s wr, 14 op/s 2023-03-09T20:29:57.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:29:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:29:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:57 smithi181 ceph-mon[94189]: pgmap v11128: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 480 KiB/s wr, 14 op/s 2023-03-09T20:29:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:29:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:29:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:57 smithi038 ceph-mon[92422]: pgmap v11128: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 480 KiB/s wr, 14 op/s 2023-03-09T20:29:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:29:58 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 19908.577563 secs 2023-03-09T20:29:58.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:29:58 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 19908.577563 secs 2023-03-09T20:29:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:29:58 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 19908.577563 secs 2023-03-09T20:29:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20370.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:30:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:00 smithi122 ceph-mon[93869]: pgmap v11129: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 363 KiB/s wr, 10 op/s 2023-03-09T20:30:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:30:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:30:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:00 smithi122 ceph-mon[93869]: overall HEALTH_WARN 2 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-09T20:30:00.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:00 smithi181 ceph-mon[94189]: pgmap v11129: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 363 KiB/s wr, 10 op/s 2023-03-09T20:30:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:30:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:30:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:00 smithi181 ceph-mon[94189]: overall HEALTH_WARN 2 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-09T20:30:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:00 smithi038 ceph-mon[92422]: pgmap v11129: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 363 KiB/s wr, 10 op/s 2023-03-09T20:30:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:30:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:30:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:00 smithi038 ceph-mon[92422]: overall HEALTH_WARN 2 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-09T20:30:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19911.248100 secs 2023-03-09T20:30:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19911.248100 secs 2023-03-09T20:30:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19911.248100 secs 2023-03-09T20:30:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:02 smithi122 ceph-mon[93869]: pgmap v11130: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 363 KiB/s wr, 10 op/s 2023-03-09T20:30:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:02 smithi181 ceph-mon[94189]: pgmap v11130: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 363 KiB/s wr, 10 op/s 2023-03-09T20:30:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:02 smithi038 ceph-mon[92422]: pgmap v11130: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 363 KiB/s wr, 10 op/s 2023-03-09T20:30:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20375.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:30:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:04 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 19913.577680 secs 2023-03-09T20:30:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:04 smithi122 ceph-mon[93869]: pgmap v11131: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 546 KiB/s rd, 541 KiB/s wr, 14 op/s 2023-03-09T20:30:04.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:04 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 19913.577680 secs 2023-03-09T20:30:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:04 smithi181 ceph-mon[94189]: pgmap v11131: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 546 KiB/s rd, 541 KiB/s wr, 14 op/s 2023-03-09T20:30:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:04 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 19913.577680 secs 2023-03-09T20:30:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:04 smithi038 ceph-mon[92422]: pgmap v11131: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 546 KiB/s rd, 541 KiB/s wr, 14 op/s 2023-03-09T20:30:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:06 smithi122 ceph-mon[93869]: pgmap v11132: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 394 KiB/s wr, 11 op/s 2023-03-09T20:30:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19916.248162 secs 2023-03-09T20:30:06.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:06 smithi181 ceph-mon[94189]: pgmap v11132: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 394 KiB/s wr, 11 op/s 2023-03-09T20:30:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19916.248162 secs 2023-03-09T20:30:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:06 smithi038 ceph-mon[92422]: pgmap v11132: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 394 KiB/s wr, 11 op/s 2023-03-09T20:30:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19916.248162 secs 2023-03-09T20:30:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:08 smithi122 ceph-mon[93869]: pgmap v11133: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 523 KiB/s wr, 16 op/s 2023-03-09T20:30:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:08 smithi181 ceph-mon[94189]: pgmap v11133: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 523 KiB/s wr, 16 op/s 2023-03-09T20:30:08.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:08 smithi038 ceph-mon[92422]: pgmap v11133: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 523 KiB/s wr, 16 op/s 2023-03-09T20:30:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20380.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:30:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:09 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19918.577800 secs 2023-03-09T20:30:09.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:09 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19918.577800 secs 2023-03-09T20:30:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:09 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19918.577800 secs 2023-03-09T20:30:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:10 smithi122 ceph-mon[93869]: pgmap v11134: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 400 KiB/s wr, 25 op/s 2023-03-09T20:30:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:10 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:30:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:10 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:30:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:10 smithi122 ceph-mon[93869]: osdmap e909: 12 total, 12 up, 12 in 2023-03-09T20:30:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:10 smithi181 ceph-mon[94189]: pgmap v11134: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 400 KiB/s wr, 25 op/s 2023-03-09T20:30:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:10 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:30:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:10 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:30:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:10 smithi181 ceph-mon[94189]: osdmap e909: 12 total, 12 up, 12 in 2023-03-09T20:30:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:10 smithi038 ceph-mon[92422]: pgmap v11134: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 400 KiB/s wr, 25 op/s 2023-03-09T20:30:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:10 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:30:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:10 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:30:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:10 smithi038 ceph-mon[92422]: osdmap e909: 12 total, 12 up, 12 in 2023-03-09T20:30:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19921.248219 secs 2023-03-09T20:30:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:11 smithi122 ceph-mon[93869]: pgmap v11136: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 480 KiB/s wr, 29 op/s 2023-03-09T20:30:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19921.248219 secs 2023-03-09T20:30:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:11 smithi181 ceph-mon[94189]: pgmap v11136: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 480 KiB/s wr, 29 op/s 2023-03-09T20:30:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19921.248219 secs 2023-03-09T20:30:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:11 smithi038 ceph-mon[92422]: pgmap v11136: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 480 KiB/s wr, 29 op/s 2023-03-09T20:30:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:30:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:30:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:30:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:30:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:30:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:30:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:30:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:30:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:13 smithi122 ceph-mon[93869]: pgmap v11137: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 483 KiB/s wr, 30 op/s 2023-03-09T20:30:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:30:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:30:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:13 smithi181 ceph-mon[94189]: pgmap v11137: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 483 KiB/s wr, 30 op/s 2023-03-09T20:30:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:30:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:30:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:13 smithi038 ceph-mon[92422]: pgmap v11137: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 483 KiB/s wr, 30 op/s 2023-03-09T20:30:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20385.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:30:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:14 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19923.577936 secs 2023-03-09T20:30:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:14 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19923.577936 secs 2023-03-09T20:30:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:14 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19923.577936 secs 2023-03-09T20:30:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:15 smithi122 ceph-mon[93869]: pgmap v11138: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 440 KiB/s rd, 483 KiB/s wr, 28 op/s 2023-03-09T20:30:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:15 smithi181 ceph-mon[94189]: pgmap v11138: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 440 KiB/s rd, 483 KiB/s wr, 28 op/s 2023-03-09T20:30:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:15 smithi038 ceph-mon[92422]: pgmap v11138: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 440 KiB/s rd, 483 KiB/s wr, 28 op/s 2023-03-09T20:30:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19926.248281 secs 2023-03-09T20:30:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19926.248281 secs 2023-03-09T20:30:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19926.248281 secs 2023-03-09T20:30:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:17 smithi122 ceph-mon[93869]: pgmap v11139: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 362 KiB/s wr, 25 op/s 2023-03-09T20:30:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:17 smithi181 ceph-mon[94189]: pgmap v11139: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 362 KiB/s wr, 25 op/s 2023-03-09T20:30:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:17 smithi038 ceph-mon[92422]: pgmap v11139: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 362 KiB/s wr, 25 op/s 2023-03-09T20:30:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19928.578052 secs 2023-03-09T20:30:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19928.578052 secs 2023-03-09T20:30:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19928.578052 secs 2023-03-09T20:30:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20390.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:30:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:20 smithi122 ceph-mon[93869]: pgmap v11140: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 379 KiB/s wr, 11 op/s 2023-03-09T20:30:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:20 smithi122 ceph-mon[93869]: osdmap e910: 12 total, 12 up, 12 in 2023-03-09T20:30:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:20 smithi181 ceph-mon[94189]: pgmap v11140: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 379 KiB/s wr, 11 op/s 2023-03-09T20:30:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:20 smithi181 ceph-mon[94189]: osdmap e910: 12 total, 12 up, 12 in 2023-03-09T20:30:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:20 smithi038 ceph-mon[92422]: pgmap v11140: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 379 KiB/s wr, 11 op/s 2023-03-09T20:30:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:20 smithi038 ceph-mon[92422]: osdmap e910: 12 total, 12 up, 12 in 2023-03-09T20:30:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19931.248340 secs 2023-03-09T20:30:21.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19931.248340 secs 2023-03-09T20:30:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19931.248340 secs 2023-03-09T20:30:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:22 smithi122 ceph-mon[93869]: pgmap v11142: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 379 KiB/s wr, 11 op/s 2023-03-09T20:30:22.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:22 smithi181 ceph-mon[94189]: pgmap v11142: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 379 KiB/s wr, 11 op/s 2023-03-09T20:30:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:22 smithi038 ceph-mon[92422]: pgmap v11142: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 379 KiB/s wr, 11 op/s 2023-03-09T20:30:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20395.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:30:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:24 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19933.578190 secs 2023-03-09T20:30:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:24 smithi122 ceph-mon[93869]: pgmap v11143: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 163 KiB/s wr, 9 op/s 2023-03-09T20:30:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:24 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19933.578190 secs 2023-03-09T20:30:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:24 smithi181 ceph-mon[94189]: pgmap v11143: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 163 KiB/s wr, 9 op/s 2023-03-09T20:30:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:24 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19933.578190 secs 2023-03-09T20:30:24.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:24 smithi038 ceph-mon[92422]: pgmap v11143: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 163 KiB/s wr, 9 op/s 2023-03-09T20:30:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:26 smithi122 ceph-mon[93869]: pgmap v11144: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 163 KiB/s wr, 9 op/s 2023-03-09T20:30:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19936.248401 secs 2023-03-09T20:30:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:26 smithi181 ceph-mon[94189]: pgmap v11144: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 163 KiB/s wr, 9 op/s 2023-03-09T20:30:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19936.248401 secs 2023-03-09T20:30:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:26 smithi038 ceph-mon[92422]: pgmap v11144: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 163 KiB/s wr, 9 op/s 2023-03-09T20:30:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19936.248401 secs 2023-03-09T20:30:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:28 smithi122 ceph-mon[93869]: pgmap v11145: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 179 KiB/s wr, 9 op/s 2023-03-09T20:30:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:28 smithi181 ceph-mon[94189]: pgmap v11145: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 179 KiB/s wr, 9 op/s 2023-03-09T20:30:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:28 smithi038 ceph-mon[92422]: pgmap v11145: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 179 KiB/s wr, 9 op/s 2023-03-09T20:30:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20400.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:30:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:29 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19938.578312 secs 2023-03-09T20:30:29.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:29 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19938.578312 secs 2023-03-09T20:30:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:29 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19938.578312 secs 2023-03-09T20:30:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:30 smithi122 ceph-mon[93869]: pgmap v11146: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 51 KiB/s wr, 8 op/s 2023-03-09T20:30:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:30 smithi181 ceph-mon[94189]: pgmap v11146: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 51 KiB/s wr, 8 op/s 2023-03-09T20:30:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:30 smithi038 ceph-mon[92422]: pgmap v11146: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 51 KiB/s wr, 8 op/s 2023-03-09T20:30:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19941.248461 secs 2023-03-09T20:30:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:31 smithi122 ceph-mon[93869]: pgmap v11147: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 45 KiB/s wr, 7 op/s 2023-03-09T20:30:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19941.248461 secs 2023-03-09T20:30:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:31 smithi181 ceph-mon[94189]: pgmap v11147: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 45 KiB/s wr, 7 op/s 2023-03-09T20:30:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19941.248461 secs 2023-03-09T20:30:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:31 smithi038 ceph-mon[92422]: pgmap v11147: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 45 KiB/s wr, 7 op/s 2023-03-09T20:30:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:32 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:30:32.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:32 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:30:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:32 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:30:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:32 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:30:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:32 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:30:32.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:32 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:30:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:33 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:30:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:33 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:30:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:33 smithi122 ceph-mon[93869]: pgmap v11148: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 118 KiB/s wr, 10 op/s 2023-03-09T20:30:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:33 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:30:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:33 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:30:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:33 smithi181 ceph-mon[94189]: pgmap v11148: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 118 KiB/s wr, 10 op/s 2023-03-09T20:30:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:33 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:30:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:33 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:30:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:33 smithi038 ceph-mon[92422]: pgmap v11148: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 118 KiB/s wr, 10 op/s 2023-03-09T20:30:33.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20405.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:30:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:34 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19943.578411 secs 2023-03-09T20:30:34.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:34 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19943.578411 secs 2023-03-09T20:30:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:34 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19943.578411 secs 2023-03-09T20:30:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:35 smithi122 ceph-mon[93869]: pgmap v11149: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 117 KiB/s wr, 7 op/s 2023-03-09T20:30:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:35 smithi181 ceph-mon[94189]: pgmap v11149: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 117 KiB/s wr, 7 op/s 2023-03-09T20:30:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:35 smithi038 ceph-mon[92422]: pgmap v11149: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 117 KiB/s wr, 7 op/s 2023-03-09T20:30:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19946.248522 secs 2023-03-09T20:30:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19946.248522 secs 2023-03-09T20:30:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19946.248522 secs 2023-03-09T20:30:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:37 smithi122 ceph-mon[93869]: pgmap v11150: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 117 KiB/s wr, 9 op/s 2023-03-09T20:30:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:37 smithi181 ceph-mon[94189]: pgmap v11150: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 117 KiB/s wr, 9 op/s 2023-03-09T20:30:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:37 smithi038 ceph-mon[92422]: pgmap v11150: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 117 KiB/s wr, 9 op/s 2023-03-09T20:30:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19948.578512 secs 2023-03-09T20:30:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19948.578512 secs 2023-03-09T20:30:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19948.578512 secs 2023-03-09T20:30:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20410.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:30:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:40 smithi122 ceph-mon[93869]: pgmap v11151: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 75 KiB/s wr, 6 op/s 2023-03-09T20:30:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:30:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:40 smithi181 ceph-mon[94189]: pgmap v11151: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 75 KiB/s wr, 6 op/s 2023-03-09T20:30:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:30:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:40 smithi038 ceph-mon[92422]: pgmap v11151: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 75 KiB/s wr, 6 op/s 2023-03-09T20:30:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:30:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19951.248581 secs 2023-03-09T20:30:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:30:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:30:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:30:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19951.248581 secs 2023-03-09T20:30:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:30:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:30:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:30:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19951.248581 secs 2023-03-09T20:30:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:30:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:30:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:30:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:42 smithi122 ceph-mon[93869]: pgmap v11152: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 75 KiB/s wr, 6 op/s 2023-03-09T20:30:42.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:42 smithi181 ceph-mon[94189]: pgmap v11152: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 75 KiB/s wr, 6 op/s 2023-03-09T20:30:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:42 smithi038 ceph-mon[92422]: pgmap v11152: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 75 KiB/s wr, 6 op/s 2023-03-09T20:30:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19953.578610 secs 2023-03-09T20:30:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19953.578610 secs 2023-03-09T20:30:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19953.578610 secs 2023-03-09T20:30:43.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20415.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:30:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:44 smithi122 ceph-mon[93869]: pgmap v11153: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 157 KiB/s wr, 8 op/s 2023-03-09T20:30:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:44 smithi181 ceph-mon[94189]: pgmap v11153: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 157 KiB/s wr, 8 op/s 2023-03-09T20:30:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:44 smithi038 ceph-mon[92422]: pgmap v11153: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 157 KiB/s wr, 8 op/s 2023-03-09T20:30:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:45 smithi122 ceph-mon[93869]: pgmap v11154: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 82 KiB/s wr, 6 op/s 2023-03-09T20:30:45.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:45 smithi181 ceph-mon[94189]: pgmap v11154: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 82 KiB/s wr, 6 op/s 2023-03-09T20:30:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:45 smithi038 ceph-mon[92422]: pgmap v11154: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 82 KiB/s wr, 6 op/s 2023-03-09T20:30:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19956.248643 secs 2023-03-09T20:30:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19956.248643 secs 2023-03-09T20:30:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19956.248643 secs 2023-03-09T20:30:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:47 smithi122 ceph-mon[93869]: pgmap v11155: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 311 KiB/s wr, 9 op/s 2023-03-09T20:30:47.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:47 smithi181 ceph-mon[94189]: pgmap v11155: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 311 KiB/s wr, 9 op/s 2023-03-09T20:30:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:47 smithi038 ceph-mon[92422]: pgmap v11155: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 311 KiB/s wr, 9 op/s 2023-03-09T20:30:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19958.578752 secs 2023-03-09T20:30:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19958.578752 secs 2023-03-09T20:30:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19958.578752 secs 2023-03-09T20:30:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20420.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:30:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:50 smithi122 ceph-mon[93869]: pgmap v11156: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 311 KiB/s wr, 7 op/s 2023-03-09T20:30:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:50 smithi181 ceph-mon[94189]: pgmap v11156: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 311 KiB/s wr, 7 op/s 2023-03-09T20:30:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:50 smithi038 ceph-mon[92422]: pgmap v11156: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 311 KiB/s wr, 7 op/s 2023-03-09T20:30:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19961.248704 secs 2023-03-09T20:30:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19961.248704 secs 2023-03-09T20:30:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19961.248704 secs 2023-03-09T20:30:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:52 smithi122 ceph-mon[93869]: pgmap v11157: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 402 KiB/s wr, 8 op/s 2023-03-09T20:30:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:30:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:30:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:52 smithi181 ceph-mon[94189]: pgmap v11157: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 402 KiB/s wr, 8 op/s 2023-03-09T20:30:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:30:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:30:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:52 smithi038 ceph-mon[92422]: pgmap v11157: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 402 KiB/s wr, 8 op/s 2023-03-09T20:30:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:30:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:30:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:30:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:30:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19963.578885 secs 2023-03-09T20:30:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:53 smithi122 ceph-mon[93869]: pgmap v11158: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 529 KiB/s rd, 418 KiB/s wr, 11 op/s 2023-03-09T20:30:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:30:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:30:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19963.578885 secs 2023-03-09T20:30:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:53 smithi181 ceph-mon[94189]: pgmap v11158: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 529 KiB/s rd, 418 KiB/s wr, 11 op/s 2023-03-09T20:30:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:30:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:30:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19963.578885 secs 2023-03-09T20:30:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:53 smithi038 ceph-mon[92422]: pgmap v11158: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 529 KiB/s rd, 418 KiB/s wr, 11 op/s 2023-03-09T20:30:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20425.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:30:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:56 smithi122 ceph-mon[93869]: pgmap v11159: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 446 KiB/s rd, 337 KiB/s wr, 9 op/s 2023-03-09T20:30:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19966.248766 secs 2023-03-09T20:30:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:56 smithi181 ceph-mon[94189]: pgmap v11159: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 446 KiB/s rd, 337 KiB/s wr, 9 op/s 2023-03-09T20:30:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19966.248766 secs 2023-03-09T20:30:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:56 smithi038 ceph-mon[92422]: pgmap v11159: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 446 KiB/s rd, 337 KiB/s wr, 9 op/s 2023-03-09T20:30:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19966.248766 secs 2023-03-09T20:30:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:58 smithi122 ceph-mon[93869]: pgmap v11160: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 619 KiB/s rd, 509 KiB/s wr, 13 op/s 2023-03-09T20:30:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:30:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19968.579008 secs 2023-03-09T20:30:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:58 smithi181 ceph-mon[94189]: pgmap v11160: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 619 KiB/s rd, 509 KiB/s wr, 13 op/s 2023-03-09T20:30:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:30:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19968.579008 secs 2023-03-09T20:30:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:58 smithi038 ceph-mon[92422]: pgmap v11160: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 619 KiB/s rd, 509 KiB/s wr, 13 op/s 2023-03-09T20:30:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:30:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19968.579008 secs 2023-03-09T20:30:58.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20430.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:31:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:00 smithi122 ceph-mon[93869]: pgmap v11161: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 335 KiB/s wr, 10 op/s 2023-03-09T20:31:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:31:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:31:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:00 smithi181 ceph-mon[94189]: pgmap v11161: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 335 KiB/s wr, 10 op/s 2023-03-09T20:31:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:31:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:31:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:00 smithi038 ceph-mon[92422]: pgmap v11161: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 335 KiB/s wr, 10 op/s 2023-03-09T20:31:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:31:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:31:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19971.248841 secs 2023-03-09T20:31:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:31:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:31:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:01 smithi122 ceph-mon[93869]: pgmap v11162: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 335 KiB/s wr, 10 op/s 2023-03-09T20:31:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19971.248841 secs 2023-03-09T20:31:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:31:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:31:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:01 smithi181 ceph-mon[94189]: pgmap v11162: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 335 KiB/s wr, 10 op/s 2023-03-09T20:31:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19971.248841 secs 2023-03-09T20:31:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:31:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:31:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:01 smithi038 ceph-mon[92422]: pgmap v11162: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 335 KiB/s wr, 10 op/s 2023-03-09T20:31:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19973.579135 secs 2023-03-09T20:31:02.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19973.579135 secs 2023-03-09T20:31:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19973.579135 secs 2023-03-09T20:31:03.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:03 smithi122 ceph-mon[93869]: pgmap v11163: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 244 KiB/s wr, 13 op/s 2023-03-09T20:31:03.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:03 smithi181 ceph-mon[94189]: pgmap v11163: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 244 KiB/s wr, 13 op/s 2023-03-09T20:31:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:03 smithi038 ceph-mon[92422]: pgmap v11163: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 244 KiB/s wr, 13 op/s 2023-03-09T20:31:03.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20435.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:31:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:06 smithi122 ceph-mon[93869]: pgmap v11164: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 450 KiB/s wr, 22 op/s 2023-03-09T20:31:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19976.248902 secs 2023-03-09T20:31:06.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:06 smithi181 ceph-mon[94189]: pgmap v11164: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 450 KiB/s wr, 22 op/s 2023-03-09T20:31:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19976.248902 secs 2023-03-09T20:31:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:06 smithi038 ceph-mon[92422]: pgmap v11164: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 450 KiB/s wr, 22 op/s 2023-03-09T20:31:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19976.248902 secs 2023-03-09T20:31:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:08 smithi122 ceph-mon[93869]: pgmap v11165: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 490 KiB/s rd, 503 KiB/s wr, 26 op/s 2023-03-09T20:31:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19978.579261 secs 2023-03-09T20:31:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:08 smithi181 ceph-mon[94189]: pgmap v11165: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 490 KiB/s rd, 503 KiB/s wr, 26 op/s 2023-03-09T20:31:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19978.579261 secs 2023-03-09T20:31:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:08 smithi038 ceph-mon[92422]: pgmap v11165: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 490 KiB/s rd, 503 KiB/s wr, 26 op/s 2023-03-09T20:31:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19978.579261 secs 2023-03-09T20:31:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20440.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:31:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:10 smithi122 ceph-mon[93869]: pgmap v11166: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 407 KiB/s wr, 23 op/s 2023-03-09T20:31:10.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:10 smithi181 ceph-mon[94189]: pgmap v11166: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 407 KiB/s wr, 23 op/s 2023-03-09T20:31:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:10 smithi038 ceph-mon[92422]: pgmap v11166: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 407 KiB/s wr, 23 op/s 2023-03-09T20:31:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19981.248962 secs 2023-03-09T20:31:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19981.248962 secs 2023-03-09T20:31:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19981.248962 secs 2023-03-09T20:31:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:12 smithi122 ceph-mon[93869]: pgmap v11167: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 352 KiB/s wr, 22 op/s 2023-03-09T20:31:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:31:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:31:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:12 smithi181 ceph-mon[94189]: pgmap v11167: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 352 KiB/s wr, 22 op/s 2023-03-09T20:31:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:31:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:31:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:12 smithi038 ceph-mon[92422]: pgmap v11167: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 352 KiB/s wr, 22 op/s 2023-03-09T20:31:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:31:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:31:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:31:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:31:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19983.579385 secs 2023-03-09T20:31:13.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:13 smithi122 ceph-mon[93869]: pgmap v11168: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 403 KiB/s wr, 24 op/s 2023-03-09T20:31:13.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:31:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:31:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19983.579385 secs 2023-03-09T20:31:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:13 smithi181 ceph-mon[94189]: pgmap v11168: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 403 KiB/s wr, 24 op/s 2023-03-09T20:31:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:31:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:31:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19983.579385 secs 2023-03-09T20:31:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:13 smithi038 ceph-mon[92422]: pgmap v11168: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 403 KiB/s wr, 24 op/s 2023-03-09T20:31:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20445.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:31:16.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:16 smithi122 ceph-mon[93869]: pgmap v11169: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 402 KiB/s wr, 21 op/s 2023-03-09T20:31:16.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19986.248990 secs 2023-03-09T20:31:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:16 smithi181 ceph-mon[94189]: pgmap v11169: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 402 KiB/s wr, 21 op/s 2023-03-09T20:31:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19986.248990 secs 2023-03-09T20:31:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:16 smithi038 ceph-mon[92422]: pgmap v11169: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 402 KiB/s wr, 21 op/s 2023-03-09T20:31:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19986.248990 secs 2023-03-09T20:31:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:18 smithi122 ceph-mon[93869]: pgmap v11170: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 368 KiB/s wr, 14 op/s 2023-03-09T20:31:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19988.579508 secs 2023-03-09T20:31:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:18 smithi181 ceph-mon[94189]: pgmap v11170: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 368 KiB/s wr, 14 op/s 2023-03-09T20:31:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19988.579508 secs 2023-03-09T20:31:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:18 smithi038 ceph-mon[92422]: pgmap v11170: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 368 KiB/s wr, 14 op/s 2023-03-09T20:31:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19988.579508 secs 2023-03-09T20:31:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20450.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:31:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:20 smithi122 ceph-mon[93869]: pgmap v11171: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 326 KiB/s wr, 11 op/s 2023-03-09T20:31:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:20 smithi122 ceph-mon[93869]: osdmap e911: 12 total, 12 up, 12 in 2023-03-09T20:31:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:20 smithi181 ceph-mon[94189]: pgmap v11171: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 326 KiB/s wr, 11 op/s 2023-03-09T20:31:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:20 smithi181 ceph-mon[94189]: osdmap e911: 12 total, 12 up, 12 in 2023-03-09T20:31:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:20 smithi038 ceph-mon[92422]: pgmap v11171: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 326 KiB/s wr, 11 op/s 2023-03-09T20:31:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:20 smithi038 ceph-mon[92422]: osdmap e911: 12 total, 12 up, 12 in 2023-03-09T20:31:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19991.249053 secs 2023-03-09T20:31:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:21 smithi122 ceph-mon[93869]: pgmap v11173: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 300 KiB/s wr, 12 op/s 2023-03-09T20:31:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19991.249053 secs 2023-03-09T20:31:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:21 smithi181 ceph-mon[94189]: pgmap v11173: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 300 KiB/s wr, 12 op/s 2023-03-09T20:31:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19991.249053 secs 2023-03-09T20:31:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:21 smithi038 ceph-mon[92422]: pgmap v11173: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 300 KiB/s wr, 12 op/s 2023-03-09T20:31:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19993.579597 secs 2023-03-09T20:31:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19993.579597 secs 2023-03-09T20:31:22.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19993.579597 secs 2023-03-09T20:31:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:23 smithi122 ceph-mon[93869]: pgmap v11174: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 317 KiB/s wr, 13 op/s 2023-03-09T20:31:23.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:23 smithi181 ceph-mon[94189]: pgmap v11174: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 317 KiB/s wr, 13 op/s 2023-03-09T20:31:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:23 smithi038 ceph-mon[92422]: pgmap v11174: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 317 KiB/s wr, 13 op/s 2023-03-09T20:31:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20455.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:31:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:26 smithi122 ceph-mon[93869]: pgmap v11175: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 384 KiB/s wr, 13 op/s 2023-03-09T20:31:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19996.249114 secs 2023-03-09T20:31:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:26 smithi181 ceph-mon[94189]: pgmap v11175: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 384 KiB/s wr, 13 op/s 2023-03-09T20:31:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19996.249114 secs 2023-03-09T20:31:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:26 smithi038 ceph-mon[92422]: pgmap v11175: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 384 KiB/s wr, 13 op/s 2023-03-09T20:31:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19996.249114 secs 2023-03-09T20:31:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:28 smithi122 ceph-mon[93869]: pgmap v11176: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 229 KiB/s wr, 11 op/s 2023-03-09T20:31:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:28 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:31:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:28 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:31:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 19998.579716 secs 2023-03-09T20:31:28.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:28 smithi181 ceph-mon[94189]: pgmap v11176: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 229 KiB/s wr, 11 op/s 2023-03-09T20:31:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:28 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:31:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:28 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:31:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 19998.579716 secs 2023-03-09T20:31:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:28 smithi038 ceph-mon[92422]: pgmap v11176: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 229 KiB/s wr, 11 op/s 2023-03-09T20:31:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:28 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:31:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:28 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:31:28.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 19998.579716 secs 2023-03-09T20:31:28.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20460.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:31:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:29 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:31:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:29 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:31:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:29 smithi122 ceph-mon[93869]: pgmap v11177: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 331 KiB/s wr, 10 op/s 2023-03-09T20:31:29.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:29 smithi122 ceph-mon[93869]: osdmap e912: 12 total, 12 up, 12 in 2023-03-09T20:31:29.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:29 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:31:29.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:29 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:31:29.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:29 smithi181 ceph-mon[94189]: pgmap v11177: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 331 KiB/s wr, 10 op/s 2023-03-09T20:31:29.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:29 smithi181 ceph-mon[94189]: osdmap e912: 12 total, 12 up, 12 in 2023-03-09T20:31:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:29 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:31:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:29 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:31:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:29 smithi038 ceph-mon[92422]: pgmap v11177: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 331 KiB/s wr, 10 op/s 2023-03-09T20:31:29.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:29 smithi038 ceph-mon[92422]: osdmap e912: 12 total, 12 up, 12 in 2023-03-09T20:31:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20001.249175 secs 2023-03-09T20:31:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20001.249175 secs 2023-03-09T20:31:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20001.249175 secs 2023-03-09T20:31:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:31 smithi122 ceph-mon[93869]: pgmap v11179: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 331 KiB/s wr, 10 op/s 2023-03-09T20:31:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:31 smithi181 ceph-mon[94189]: pgmap v11179: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 331 KiB/s wr, 10 op/s 2023-03-09T20:31:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:31 smithi038 ceph-mon[92422]: pgmap v11179: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 331 KiB/s wr, 10 op/s 2023-03-09T20:31:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20003.579810 secs 2023-03-09T20:31:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20003.579810 secs 2023-03-09T20:31:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20003.579810 secs 2023-03-09T20:31:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:33 smithi122 ceph-mon[93869]: pgmap v11180: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 254 KiB/s wr, 9 op/s 2023-03-09T20:31:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:33 smithi181 ceph-mon[94189]: pgmap v11180: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 254 KiB/s wr, 9 op/s 2023-03-09T20:31:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:33 smithi038 ceph-mon[92422]: pgmap v11180: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 254 KiB/s wr, 9 op/s 2023-03-09T20:31:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20465.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:31:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:36 smithi122 ceph-mon[93869]: pgmap v11181: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 250 KiB/s wr, 9 op/s 2023-03-09T20:31:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20006.249237 secs 2023-03-09T20:31:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:36 smithi181 ceph-mon[94189]: pgmap v11181: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 250 KiB/s wr, 9 op/s 2023-03-09T20:31:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20006.249237 secs 2023-03-09T20:31:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:36 smithi038 ceph-mon[92422]: pgmap v11181: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 250 KiB/s wr, 9 op/s 2023-03-09T20:31:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20006.249237 secs 2023-03-09T20:31:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:38 smithi122 ceph-mon[93869]: pgmap v11182: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 179 KiB/s wr, 9 op/s 2023-03-09T20:31:38.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:38 smithi181 ceph-mon[94189]: pgmap v11182: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 179 KiB/s wr, 9 op/s 2023-03-09T20:31:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:38 smithi038 ceph-mon[92422]: pgmap v11182: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 179 KiB/s wr, 9 op/s 2023-03-09T20:31:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20470.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:31:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:40 smithi122 ceph-mon[93869]: pgmap v11183: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 KiB/s wr, 8 op/s 2023-03-09T20:31:40.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:40 smithi181 ceph-mon[94189]: pgmap v11183: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 KiB/s wr, 8 op/s 2023-03-09T20:31:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:40 smithi038 ceph-mon[92422]: pgmap v11183: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 KiB/s wr, 8 op/s 2023-03-09T20:31:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20011.249299 secs 2023-03-09T20:31:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:31:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:41 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:31:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:31:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:31:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:31:41.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20011.249299 secs 2023-03-09T20:31:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:31:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:41 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:31:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:31:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:31:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:31:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20011.249299 secs 2023-03-09T20:31:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:31:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:41 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:31:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:31:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:31:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:31:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:42 smithi122 ceph-mon[93869]: pgmap v11184: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 81 KiB/s wr, 7 op/s 2023-03-09T20:31:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:42 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:31:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:42 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:31:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:42 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:31:42.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:42 smithi181 ceph-mon[94189]: pgmap v11184: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 81 KiB/s wr, 7 op/s 2023-03-09T20:31:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:42 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:31:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:42 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:31:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:42 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:31:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:42 smithi038 ceph-mon[92422]: pgmap v11184: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 81 KiB/s wr, 7 op/s 2023-03-09T20:31:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:42 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:31:42.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:42 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:31:42.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:42 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:31:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:43 smithi122 ceph-mon[93869]: pgmap v11185: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 170 KiB/s wr, 10 op/s 2023-03-09T20:31:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:43 smithi181 ceph-mon[94189]: pgmap v11185: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 170 KiB/s wr, 10 op/s 2023-03-09T20:31:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:43 smithi038 ceph-mon[92422]: pgmap v11185: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 170 KiB/s wr, 10 op/s 2023-03-09T20:31:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20475.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:31:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:46 smithi122 ceph-mon[93869]: pgmap v11186: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 285 KiB/s wr, 8 op/s 2023-03-09T20:31:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20016.249384 secs 2023-03-09T20:31:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:46 smithi181 ceph-mon[94189]: pgmap v11186: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 285 KiB/s wr, 8 op/s 2023-03-09T20:31:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20016.249384 secs 2023-03-09T20:31:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:46 smithi038 ceph-mon[92422]: pgmap v11186: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 285 KiB/s wr, 8 op/s 2023-03-09T20:31:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20016.249384 secs 2023-03-09T20:31:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:47 smithi122 ceph-mon[93869]: pgmap v11187: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 332 KiB/s wr, 10 op/s 2023-03-09T20:31:47.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:47 smithi181 ceph-mon[94189]: pgmap v11187: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 332 KiB/s wr, 10 op/s 2023-03-09T20:31:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:47 smithi038 ceph-mon[92422]: pgmap v11187: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 332 KiB/s wr, 10 op/s 2023-03-09T20:31:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:31:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:31:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:31:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:31:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:31:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:31:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20480.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:31:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:31:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:31:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:49 smithi122 ceph-mon[93869]: pgmap v11188: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 332 KiB/s wr, 9 op/s 2023-03-09T20:31:49.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:31:49.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:31:49.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:49 smithi181 ceph-mon[94189]: pgmap v11188: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 332 KiB/s wr, 9 op/s 2023-03-09T20:31:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:31:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:31:49.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:49 smithi038 ceph-mon[92422]: pgmap v11188: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 332 KiB/s wr, 9 op/s 2023-03-09T20:31:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20021.249442 secs 2023-03-09T20:31:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20021.249442 secs 2023-03-09T20:31:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20021.249442 secs 2023-03-09T20:31:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:51 smithi122 ceph-mon[93869]: pgmap v11189: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 328 KiB/s wr, 9 op/s 2023-03-09T20:31:51.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:51 smithi181 ceph-mon[94189]: pgmap v11189: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 328 KiB/s wr, 9 op/s 2023-03-09T20:31:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:51 smithi038 ceph-mon[92422]: pgmap v11189: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 328 KiB/s wr, 9 op/s 2023-03-09T20:31:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20485.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:31:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:53 smithi122 ceph-mon[93869]: pgmap v11190: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 577 KiB/s rd, 527 KiB/s wr, 13 op/s 2023-03-09T20:31:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:53 smithi181 ceph-mon[94189]: pgmap v11190: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 577 KiB/s rd, 527 KiB/s wr, 13 op/s 2023-03-09T20:31:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:53 smithi038 ceph-mon[92422]: pgmap v11190: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 577 KiB/s rd, 527 KiB/s wr, 13 op/s 2023-03-09T20:31:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:56 smithi122 ceph-mon[93869]: pgmap v11191: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 499 KiB/s rd, 434 KiB/s wr, 10 op/s 2023-03-09T20:31:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20026.249502 secs 2023-03-09T20:31:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:31:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:31:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:56 smithi181 ceph-mon[94189]: pgmap v11191: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 499 KiB/s rd, 434 KiB/s wr, 10 op/s 2023-03-09T20:31:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20026.249502 secs 2023-03-09T20:31:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:31:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:31:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:56 smithi038 ceph-mon[92422]: pgmap v11191: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 499 KiB/s rd, 434 KiB/s wr, 10 op/s 2023-03-09T20:31:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20026.249502 secs 2023-03-09T20:31:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:31:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:31:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:31:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:31:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:31:57 smithi122 ceph-mon[93869]: pgmap v11192: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 430 KiB/s rd, 410 KiB/s wr, 13 op/s 2023-03-09T20:31:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:31:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:31:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:31:57 smithi181 ceph-mon[94189]: pgmap v11192: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 430 KiB/s rd, 410 KiB/s wr, 13 op/s 2023-03-09T20:31:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:31:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:31:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:31:57 smithi038 ceph-mon[92422]: pgmap v11192: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 430 KiB/s rd, 410 KiB/s wr, 13 op/s 2023-03-09T20:31:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20490.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:32:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:00 smithi122 ceph-mon[93869]: pgmap v11193: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 313 KiB/s rd, 312 KiB/s wr, 10 op/s 2023-03-09T20:32:00.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:00 smithi181 ceph-mon[94189]: pgmap v11193: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 313 KiB/s rd, 312 KiB/s wr, 10 op/s 2023-03-09T20:32:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:00 smithi038 ceph-mon[92422]: pgmap v11193: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 313 KiB/s rd, 312 KiB/s wr, 10 op/s 2023-03-09T20:32:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20031.249561 secs 2023-03-09T20:32:01.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20031.249561 secs 2023-03-09T20:32:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20031.249561 secs 2023-03-09T20:32:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:02 smithi122 ceph-mon[93869]: pgmap v11194: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 505 KiB/s rd, 503 KiB/s wr, 10 op/s 2023-03-09T20:32:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:02 smithi181 ceph-mon[94189]: pgmap v11194: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 505 KiB/s rd, 503 KiB/s wr, 10 op/s 2023-03-09T20:32:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:02 smithi038 ceph-mon[92422]: pgmap v11194: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 505 KiB/s rd, 503 KiB/s wr, 10 op/s 2023-03-09T20:32:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20033.580512 secs 2023-03-09T20:32:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:03 smithi122 ceph-mon[93869]: pgmap v11195: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 519 KiB/s rd, 538 KiB/s wr, 15 op/s 2023-03-09T20:32:03.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20033.580512 secs 2023-03-09T20:32:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:03 smithi181 ceph-mon[94189]: pgmap v11195: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 519 KiB/s rd, 538 KiB/s wr, 15 op/s 2023-03-09T20:32:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20033.580512 secs 2023-03-09T20:32:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:03 smithi038 ceph-mon[92422]: pgmap v11195: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 519 KiB/s rd, 538 KiB/s wr, 15 op/s 2023-03-09T20:32:03.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20495.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:32:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:05 smithi122 ceph-mon[93869]: pgmap v11196: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 360 KiB/s wr, 24 op/s 2023-03-09T20:32:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:05 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:32:05.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:05 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:32:05.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:05 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:32:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:05 smithi181 ceph-mon[94189]: pgmap v11196: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 360 KiB/s wr, 24 op/s 2023-03-09T20:32:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:05 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:32:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:05 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:32:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:05 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:32:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:05 smithi038 ceph-mon[92422]: pgmap v11196: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 360 KiB/s wr, 24 op/s 2023-03-09T20:32:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:05 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:32:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:05 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:32:05.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:05 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:32:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20036.249622 secs 2023-03-09T20:32:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20036.249622 secs 2023-03-09T20:32:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20036.249622 secs 2023-03-09T20:32:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:07 smithi122 ceph-mon[93869]: pgmap v11197: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 399 KiB/s wr, 27 op/s 2023-03-09T20:32:07.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:07 smithi181 ceph-mon[94189]: pgmap v11197: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 399 KiB/s wr, 27 op/s 2023-03-09T20:32:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:07 smithi038 ceph-mon[92422]: pgmap v11197: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 399 KiB/s wr, 27 op/s 2023-03-09T20:32:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20038.580581 secs 2023-03-09T20:32:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20038.580581 secs 2023-03-09T20:32:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20038.580581 secs 2023-03-09T20:32:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20500.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:32:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:10 smithi122 ceph-mon[93869]: pgmap v11198: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 431 KiB/s wr, 25 op/s 2023-03-09T20:32:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:10 smithi122 ceph-mon[93869]: osdmap e913: 12 total, 12 up, 12 in 2023-03-09T20:32:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:10 smithi181 ceph-mon[94189]: pgmap v11198: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 431 KiB/s wr, 25 op/s 2023-03-09T20:32:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:10 smithi181 ceph-mon[94189]: osdmap e913: 12 total, 12 up, 12 in 2023-03-09T20:32:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:10 smithi038 ceph-mon[92422]: pgmap v11198: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 431 KiB/s wr, 25 op/s 2023-03-09T20:32:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:10 smithi038 ceph-mon[92422]: osdmap e913: 12 total, 12 up, 12 in 2023-03-09T20:32:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20041.249707 secs 2023-03-09T20:32:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20041.249707 secs 2023-03-09T20:32:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20041.249707 secs 2023-03-09T20:32:12.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:12 smithi122 ceph-mon[93869]: pgmap v11200: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 247 KiB/s rd, 287 KiB/s wr, 28 op/s 2023-03-09T20:32:12.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:32:12.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:32:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:12 smithi181 ceph-mon[94189]: pgmap v11200: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 247 KiB/s rd, 287 KiB/s wr, 28 op/s 2023-03-09T20:32:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:32:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:32:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:12 smithi038 ceph-mon[92422]: pgmap v11200: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 247 KiB/s rd, 287 KiB/s wr, 28 op/s 2023-03-09T20:32:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:32:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:32:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:32:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:32:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20043.580701 secs 2023-03-09T20:32:13.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:13 smithi122 ceph-mon[93869]: pgmap v11201: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 222 KiB/s wr, 24 op/s 2023-03-09T20:32:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:32:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:32:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20043.580701 secs 2023-03-09T20:32:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:13 smithi181 ceph-mon[94189]: pgmap v11201: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 222 KiB/s wr, 24 op/s 2023-03-09T20:32:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:32:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:32:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20043.580701 secs 2023-03-09T20:32:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:13 smithi038 ceph-mon[92422]: pgmap v11201: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 222 KiB/s wr, 24 op/s 2023-03-09T20:32:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20505.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:32:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:16 smithi122 ceph-mon[93869]: pgmap v11202: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 196 KiB/s wr, 10 op/s 2023-03-09T20:32:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20046.249770 secs 2023-03-09T20:32:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:16 smithi181 ceph-mon[94189]: pgmap v11202: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 196 KiB/s wr, 10 op/s 2023-03-09T20:32:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20046.249770 secs 2023-03-09T20:32:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:16 smithi038 ceph-mon[92422]: pgmap v11202: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 196 KiB/s wr, 10 op/s 2023-03-09T20:32:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20046.249770 secs 2023-03-09T20:32:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:18 smithi122 ceph-mon[93869]: pgmap v11203: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 169 KiB/s wr, 9 op/s 2023-03-09T20:32:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20048.580842 secs 2023-03-09T20:32:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:18 smithi181 ceph-mon[94189]: pgmap v11203: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 169 KiB/s wr, 9 op/s 2023-03-09T20:32:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20048.580842 secs 2023-03-09T20:32:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:18 smithi038 ceph-mon[92422]: pgmap v11203: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 169 KiB/s wr, 9 op/s 2023-03-09T20:32:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20048.580842 secs 2023-03-09T20:32:18.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20510.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:32:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:20 smithi122 ceph-mon[93869]: pgmap v11204: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 81 KiB/s wr, 9 op/s 2023-03-09T20:32:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:20 smithi122 ceph-mon[93869]: osdmap e914: 12 total, 12 up, 12 in 2023-03-09T20:32:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:20 smithi181 ceph-mon[94189]: pgmap v11204: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 81 KiB/s wr, 9 op/s 2023-03-09T20:32:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:20 smithi181 ceph-mon[94189]: osdmap e914: 12 total, 12 up, 12 in 2023-03-09T20:32:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:20 smithi038 ceph-mon[92422]: pgmap v11204: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 81 KiB/s wr, 9 op/s 2023-03-09T20:32:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:20 smithi038 ceph-mon[92422]: osdmap e914: 12 total, 12 up, 12 in 2023-03-09T20:32:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20051.249850 secs 2023-03-09T20:32:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:21 smithi122 ceph-mon[93869]: pgmap v11206: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 81 KiB/s wr, 9 op/s 2023-03-09T20:32:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20051.249850 secs 2023-03-09T20:32:21.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:21 smithi181 ceph-mon[94189]: pgmap v11206: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 81 KiB/s wr, 9 op/s 2023-03-09T20:32:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20051.249850 secs 2023-03-09T20:32:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:21 smithi038 ceph-mon[92422]: pgmap v11206: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 81 KiB/s wr, 9 op/s 2023-03-09T20:32:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20053.580959 secs 2023-03-09T20:32:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20053.580959 secs 2023-03-09T20:32:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20053.580959 secs 2023-03-09T20:32:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:23 smithi122 ceph-mon[93869]: pgmap v11207: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 82 KiB/s wr, 9 op/s 2023-03-09T20:32:23.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:23 smithi181 ceph-mon[94189]: pgmap v11207: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 82 KiB/s wr, 9 op/s 2023-03-09T20:32:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:23 smithi038 ceph-mon[92422]: pgmap v11207: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 82 KiB/s wr, 9 op/s 2023-03-09T20:32:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20515.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:32:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:26 smithi122 ceph-mon[93869]: pgmap v11208: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 173 KiB/s wr, 9 op/s 2023-03-09T20:32:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20056.249909 secs 2023-03-09T20:32:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:26 smithi181 ceph-mon[94189]: pgmap v11208: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 173 KiB/s wr, 9 op/s 2023-03-09T20:32:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20056.249909 secs 2023-03-09T20:32:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:26 smithi038 ceph-mon[92422]: pgmap v11208: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 173 KiB/s wr, 9 op/s 2023-03-09T20:32:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20056.249909 secs 2023-03-09T20:32:27.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:27 smithi122 ceph-mon[93869]: pgmap v11209: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 154 KiB/s wr, 8 op/s 2023-03-09T20:32:27.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:27 smithi181 ceph-mon[94189]: pgmap v11209: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 154 KiB/s wr, 8 op/s 2023-03-09T20:32:27.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:27 smithi038 ceph-mon[92422]: pgmap v11209: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 154 KiB/s wr, 8 op/s 2023-03-09T20:32:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20058.581082 secs 2023-03-09T20:32:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20058.581082 secs 2023-03-09T20:32:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20058.581082 secs 2023-03-09T20:32:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20520.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:32:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:30 smithi122 ceph-mon[93869]: pgmap v11210: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 93 KiB/s wr, 8 op/s 2023-03-09T20:32:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:30 smithi181 ceph-mon[94189]: pgmap v11210: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 93 KiB/s wr, 8 op/s 2023-03-09T20:32:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:30 smithi038 ceph-mon[92422]: pgmap v11210: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 93 KiB/s wr, 8 op/s 2023-03-09T20:32:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20061.249999 secs 2023-03-09T20:32:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:31 smithi122 ceph-mon[93869]: pgmap v11211: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 81 KiB/s wr, 7 op/s 2023-03-09T20:32:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20061.249999 secs 2023-03-09T20:32:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:31 smithi181 ceph-mon[94189]: pgmap v11211: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 81 KiB/s wr, 7 op/s 2023-03-09T20:32:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20061.249999 secs 2023-03-09T20:32:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:31 smithi038 ceph-mon[92422]: pgmap v11211: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 81 KiB/s wr, 7 op/s 2023-03-09T20:32:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20063.581176 secs 2023-03-09T20:32:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20063.581176 secs 2023-03-09T20:32:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20063.581176 secs 2023-03-09T20:32:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:33 smithi122 ceph-mon[93869]: pgmap v11212: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 78 KiB/s wr, 9 op/s 2023-03-09T20:32:33.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:33 smithi181 ceph-mon[94189]: pgmap v11212: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 78 KiB/s wr, 9 op/s 2023-03-09T20:32:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:33 smithi038 ceph-mon[92422]: pgmap v11212: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 78 KiB/s wr, 9 op/s 2023-03-09T20:32:33.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20525.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:32:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:35 smithi122 ceph-mon[93869]: pgmap v11213: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 77 KiB/s wr, 7 op/s 2023-03-09T20:32:35.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:35 smithi181 ceph-mon[94189]: pgmap v11213: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 77 KiB/s wr, 7 op/s 2023-03-09T20:32:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:35 smithi038 ceph-mon[92422]: pgmap v11213: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 77 KiB/s wr, 7 op/s 2023-03-09T20:32:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20066.250057 secs 2023-03-09T20:32:36.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20066.250057 secs 2023-03-09T20:32:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20066.250057 secs 2023-03-09T20:32:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:37 smithi122 ceph-mon[93869]: pgmap v11214: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.3 KiB/s wr, 9 op/s 2023-03-09T20:32:37.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:37 smithi181 ceph-mon[94189]: pgmap v11214: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.3 KiB/s wr, 9 op/s 2023-03-09T20:32:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:37 smithi038 ceph-mon[92422]: pgmap v11214: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.3 KiB/s wr, 9 op/s 2023-03-09T20:32:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20530.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:32:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:39 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:32:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:39 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:32:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:39 smithi122 ceph-mon[93869]: pgmap v11215: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 91 KiB/s wr, 7 op/s 2023-03-09T20:32:39.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:39 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:32:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:39 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:32:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:39 smithi181 ceph-mon[94189]: pgmap v11215: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 91 KiB/s wr, 7 op/s 2023-03-09T20:32:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:39 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:32:39.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:39 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:32:39.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:39 smithi038 ceph-mon[92422]: pgmap v11215: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 91 KiB/s wr, 7 op/s 2023-03-09T20:32:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20071.250118 secs 2023-03-09T20:32:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:40 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:32:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20071.250118 secs 2023-03-09T20:32:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:40 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:32:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20071.250118 secs 2023-03-09T20:32:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:40 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:32:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:41 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:32:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:32:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:32:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:32:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:41 smithi122 ceph-mon[93869]: pgmap v11216: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 91 KiB/s wr, 6 op/s 2023-03-09T20:32:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:32:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:41 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:32:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:32:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:32:41.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:32:41.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:41 smithi181 ceph-mon[94189]: pgmap v11216: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 91 KiB/s wr, 6 op/s 2023-03-09T20:32:41.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:32:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:41 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:32:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:32:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:32:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:32:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:41 smithi038 ceph-mon[92422]: pgmap v11216: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 91 KiB/s wr, 6 op/s 2023-03-09T20:32:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:32:42.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:32:42.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:32:42.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:32:42.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:42 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 20073.581383 secs 2023-03-09T20:32:42.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:32:42.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:32:42.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:32:42.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:42 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 20073.581383 secs 2023-03-09T20:32:42.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:32:42.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:32:42.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:32:42.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:42 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 20073.581383 secs 2023-03-09T20:32:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:43 smithi122 ceph-mon[93869]: pgmap v11217: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 205 KiB/s wr, 9 op/s 2023-03-09T20:32:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:43 smithi181 ceph-mon[94189]: pgmap v11217: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 205 KiB/s wr, 9 op/s 2023-03-09T20:32:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:43 smithi038 ceph-mon[92422]: pgmap v11217: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 205 KiB/s wr, 9 op/s 2023-03-09T20:32:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20535.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:32:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:46 smithi122 ceph-mon[93869]: pgmap v11218: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 204 KiB/s wr, 7 op/s 2023-03-09T20:32:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20076.250178 secs 2023-03-09T20:32:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:46 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:32:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:46 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:32:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:46 smithi181 ceph-mon[94189]: pgmap v11218: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 204 KiB/s wr, 7 op/s 2023-03-09T20:32:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20076.250178 secs 2023-03-09T20:32:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:46 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:32:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:46 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:32:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:46 smithi038 ceph-mon[92422]: pgmap v11218: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 204 KiB/s wr, 7 op/s 2023-03-09T20:32:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20076.250178 secs 2023-03-09T20:32:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:46 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:32:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:46 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:32:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:47 smithi122 ceph-mon[93869]: pgmap v11219: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 398 KiB/s rd, 365 KiB/s wr, 11 op/s 2023-03-09T20:32:47.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:47 smithi181 ceph-mon[94189]: pgmap v11219: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 398 KiB/s rd, 365 KiB/s wr, 11 op/s 2023-03-09T20:32:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:47 smithi038 ceph-mon[92422]: pgmap v11219: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 398 KiB/s rd, 365 KiB/s wr, 11 op/s 2023-03-09T20:32:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20078.581459 secs 2023-03-09T20:32:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20078.581459 secs 2023-03-09T20:32:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20078.581459 secs 2023-03-09T20:32:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20540.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:32:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:49 smithi122 ceph-mon[93869]: pgmap v11220: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 365 KiB/s wr, 9 op/s 2023-03-09T20:32:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:49 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:32:49.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:49 smithi181 ceph-mon[94189]: pgmap v11220: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 365 KiB/s wr, 9 op/s 2023-03-09T20:32:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:49 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:32:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:49 smithi038 ceph-mon[92422]: pgmap v11220: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 365 KiB/s wr, 9 op/s 2023-03-09T20:32:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:49 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:32:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:50 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:32:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:50 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:32:50.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:50 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:32:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:50 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:32:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:50 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:32:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:50 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:32:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20081.250238 secs 2023-03-09T20:32:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:51 smithi122 ceph-mon[93869]: pgmap v11221: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 418 KiB/s wr, 9 op/s 2023-03-09T20:32:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20081.250238 secs 2023-03-09T20:32:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:51 smithi181 ceph-mon[94189]: pgmap v11221: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 418 KiB/s wr, 9 op/s 2023-03-09T20:32:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20081.250238 secs 2023-03-09T20:32:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:51 smithi038 ceph-mon[92422]: pgmap v11221: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 418 KiB/s wr, 9 op/s 2023-03-09T20:32:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:32:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:32:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:32:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:32:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:32:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:32:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20083.581588 secs 2023-03-09T20:32:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:32:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:32:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:53 smithi122 ceph-mon[93869]: pgmap v11222: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 535 KiB/s wr, 13 op/s 2023-03-09T20:32:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20083.581588 secs 2023-03-09T20:32:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:32:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:32:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:53 smithi181 ceph-mon[94189]: pgmap v11222: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 535 KiB/s wr, 13 op/s 2023-03-09T20:32:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20083.581588 secs 2023-03-09T20:32:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:32:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:32:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:53 smithi038 ceph-mon[92422]: pgmap v11222: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 535 KiB/s wr, 13 op/s 2023-03-09T20:32:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20545.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:32:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:56 smithi122 ceph-mon[93869]: pgmap v11223: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 421 KiB/s wr, 10 op/s 2023-03-09T20:32:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20086.250298 secs 2023-03-09T20:32:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:56 smithi181 ceph-mon[94189]: pgmap v11223: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 421 KiB/s wr, 10 op/s 2023-03-09T20:32:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20086.250298 secs 2023-03-09T20:32:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:56 smithi038 ceph-mon[92422]: pgmap v11223: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 421 KiB/s wr, 10 op/s 2023-03-09T20:32:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20086.250298 secs 2023-03-09T20:32:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:57 smithi122 ceph-mon[93869]: pgmap v11224: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 530 KiB/s wr, 14 op/s 2023-03-09T20:32:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:57 smithi181 ceph-mon[94189]: pgmap v11224: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 530 KiB/s wr, 14 op/s 2023-03-09T20:32:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:57 smithi038 ceph-mon[92422]: pgmap v11224: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 530 KiB/s wr, 14 op/s 2023-03-09T20:32:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:32:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20088.581715 secs 2023-03-09T20:32:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:32:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20088.581715 secs 2023-03-09T20:32:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:32:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20088.581715 secs 2023-03-09T20:32:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20550.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:33:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:33:00 smithi122 ceph-mon[93869]: pgmap v11225: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 369 KiB/s wr, 10 op/s 2023-03-09T20:33:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:33:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:33:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:33:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:33:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:33:00 smithi181 ceph-mon[94189]: pgmap v11225: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 369 KiB/s wr, 10 op/s 2023-03-09T20:33:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:33:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:33:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:33:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:33:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:33:00 smithi038 ceph-mon[92422]: pgmap v11225: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 369 KiB/s wr, 10 op/s 2023-03-09T20:33:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:33:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:33:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:33:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:33:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:33:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20091.250369 secs 2023-03-09T20:33:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:33:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:33:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:33:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:33:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:33:01 smithi122 ceph-mon[93869]: pgmap v11226: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 511 KiB/s rd, 510 KiB/s wr, 10 op/s 2023-03-09T20:33:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:33:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20091.250369 secs 2023-03-09T20:33:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:33:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:33:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:33:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:33:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:33:01 smithi181 ceph-mon[94189]: pgmap v11226: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 511 KiB/s rd, 510 KiB/s wr, 10 op/s 2023-03-09T20:33:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:33:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20091.250369 secs 2023-03-09T20:33:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:33:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:33:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:33:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:33:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:33:01 smithi038 ceph-mon[92422]: pgmap v11226: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 511 KiB/s rd, 510 KiB/s wr, 10 op/s 2023-03-09T20:33:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:33:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20093.581779 secs 2023-03-09T20:33:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:33:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20093.581779 secs 2023-03-09T20:33:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:33:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20093.581779 secs 2023-03-09T20:33:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:33:03 smithi122 ceph-mon[93869]: pgmap v11227: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 478 KiB/s rd, 394 KiB/s wr, 14 op/s 2023-03-09T20:33:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:33:03 smithi181 ceph-mon[94189]: pgmap v11227: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 478 KiB/s rd, 394 KiB/s wr, 14 op/s 2023-03-09T20:33:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:33:03 smithi038 ceph-mon[92422]: pgmap v11227: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 478 KiB/s rd, 394 KiB/s wr, 14 op/s 2023-03-09T20:33:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20555.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:33:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:33:05 smithi122 ceph-mon[93869]: pgmap v11228: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 373 KiB/s wr, 24 op/s 2023-03-09T20:33:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:33:05 smithi181 ceph-mon[94189]: pgmap v11228: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 373 KiB/s wr, 24 op/s 2023-03-09T20:33:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:33:05 smithi038 ceph-mon[92422]: pgmap v11228: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 373 KiB/s wr, 24 op/s 2023-03-09T20:33:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:33:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20096.250429 secs 2023-03-09T20:33:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:33:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20096.250429 secs 2023-03-09T20:33:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:33:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20096.250429 secs 2023-03-09T20:33:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:33:07 smithi122 ceph-mon[93869]: pgmap v11229: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 384 KiB/s wr, 27 op/s 2023-03-09T20:33:07.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:33:07 smithi181 ceph-mon[94189]: pgmap v11229: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 384 KiB/s wr, 27 op/s 2023-03-09T20:33:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:33:07 smithi038 ceph-mon[92422]: pgmap v11229: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 384 KiB/s wr, 27 op/s 2023-03-09T20:33:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:33:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20098.581908 secs 2023-03-09T20:33:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:33:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20098.581908 secs 2023-03-09T20:33:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:33:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20098.581908 secs 2023-03-09T20:33:08.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20560.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:33:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:33:09 smithi122 ceph-mon[93869]: pgmap v11230: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 275 KiB/s wr, 24 op/s 2023-03-09T20:33:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:33:09 smithi181 ceph-mon[94189]: pgmap v11230: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 275 KiB/s wr, 24 op/s 2023-03-09T20:33:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:33:09 smithi038 ceph-mon[92422]: pgmap v11230: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 275 KiB/s wr, 24 op/s 2023-03-09T20:33:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:33:10 smithi122 ceph-mon[93869]: osdmap e915: 12 total, 12 up, 12 in 2023-03-09T20:33:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:33:10 smithi181 ceph-mon[94189]: osdmap e915: 12 total, 12 up, 12 in 2023-03-09T20:33:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:33:10 smithi038 ceph-mon[92422]: osdmap e915: 12 total, 12 up, 12 in 2023-03-09T20:33:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:33:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20101.250489 secs 2023-03-09T20:33:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:33:11 smithi122 ceph-mon[93869]: pgmap v11232: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 221 KiB/s wr, 27 op/s 2023-03-09T20:33:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:33:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20101.250489 secs 2023-03-09T20:33:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:33:11 smithi181 ceph-mon[94189]: pgmap v11232: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 221 KiB/s wr, 27 op/s 2023-03-09T20:33:11.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:33:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20101.250489 secs 2023-03-09T20:33:11.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:33:11 smithi038 ceph-mon[92422]: pgmap v11232: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 221 KiB/s wr, 27 op/s 2023-03-09T20:33:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:33:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20103.582022 secs 2023-03-09T20:33:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:33:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20103.582022 secs 2023-03-09T20:33:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:33:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20103.582022 secs 2023-03-09T20:33:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:33:13 smithi122 ceph-mon[93869]: pgmap v11233: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 188 KiB/s wr, 24 op/s 2023-03-09T20:33:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:33:13 smithi181 ceph-mon[94189]: pgmap v11233: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 188 KiB/s wr, 24 op/s 2023-03-09T20:33:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:33:13 smithi038 ceph-mon[92422]: pgmap v11233: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 188 KiB/s wr, 24 op/s 2023-03-09T20:33:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20565.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:33:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:33:15 smithi122 ceph-mon[93869]: pgmap v11234: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 158 KiB/s wr, 9 op/s 2023-03-09T20:33:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:33:15 smithi181 ceph-mon[94189]: pgmap v11234: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 158 KiB/s wr, 9 op/s 2023-03-09T20:33:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:33:15 smithi038 ceph-mon[92422]: pgmap v11234: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 158 KiB/s wr, 9 op/s 2023-03-09T20:33:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:33:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20106.250551 secs 2023-03-09T20:33:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:33:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20106.250551 secs 2023-03-09T20:33:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:33:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20106.250551 secs 2023-03-09T20:33:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:33:17 smithi122 ceph-mon[93869]: pgmap v11235: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 145 KiB/s wr, 8 op/s 2023-03-09T20:33:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:33:17 smithi181 ceph-mon[94189]: pgmap v11235: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 145 KiB/s wr, 8 op/s 2023-03-09T20:33:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:33:17 smithi038 ceph-mon[92422]: pgmap v11235: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 145 KiB/s wr, 8 op/s 2023-03-09T20:33:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:33:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20108.582143 secs 2023-03-09T20:33:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:33:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20108.582143 secs 2023-03-09T20:33:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:33:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20108.582143 secs 2023-03-09T20:33:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20570.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:33:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:33:19 smithi122 ceph-mon[93869]: pgmap v11236: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 145 KiB/s wr, 7 op/s 2023-03-09T20:33:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:33:19 smithi122 ceph-mon[93869]: osdmap e916: 12 total, 12 up, 12 in 2023-03-09T20:33:19.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:33:19 smithi181 ceph-mon[94189]: pgmap v11236: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 145 KiB/s wr, 7 op/s 2023-03-09T20:33:19.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:33:19 smithi181 ceph-mon[94189]: osdmap e916: 12 total, 12 up, 12 in 2023-03-09T20:33:19.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:33:19 smithi038 ceph-mon[92422]: pgmap v11236: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 145 KiB/s wr, 7 op/s 2023-03-09T20:33:19.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:33:19 smithi038 ceph-mon[92422]: osdmap e916: 12 total, 12 up, 12 in 2023-03-09T20:33:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:33:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20111.250609 secs 2023-03-09T20:33:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:33:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20111.250609 secs 2023-03-09T20:33:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:33:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20111.250609 secs 2023-03-09T20:33:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:33:21 smithi122 ceph-mon[93869]: pgmap v11238: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 85 KiB/s wr, 7 op/s 2023-03-09T20:33:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:33:21 smithi181 ceph-mon[94189]: pgmap v11238: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 85 KiB/s wr, 7 op/s 2023-03-09T20:33:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:33:21 smithi038 ceph-mon[92422]: pgmap v11238: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 85 KiB/s wr, 7 op/s 2023-03-09T20:33:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:33:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20113.582256 secs 2023-03-09T20:33:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:33:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20113.582256 secs 2023-03-09T20:33:22.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:33:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20113.582256 secs 2023-03-09T20:33:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:33:23 smithi122 ceph-mon[93869]: pgmap v11239: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 85 KiB/s wr, 7 op/s 2023-03-09T20:33:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:33:23 smithi181 ceph-mon[94189]: pgmap v11239: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 85 KiB/s wr, 7 op/s 2023-03-09T20:33:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:33:23 smithi038 ceph-mon[92422]: pgmap v11239: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 85 KiB/s wr, 7 op/s 2023-03-09T20:33:23.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20575.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:33:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:33:25 smithi122 ceph-mon[93869]: pgmap v11240: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 131 KiB/s wr, 7 op/s 2023-03-09T20:33:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:33:25 smithi181 ceph-mon[94189]: pgmap v11240: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 131 KiB/s wr, 7 op/s 2023-03-09T20:33:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:33:25 smithi038 ceph-mon[92422]: pgmap v11240: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 131 KiB/s wr, 7 op/s 2023-03-09T20:33:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:33:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20116.250670 secs 2023-03-09T20:33:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:33:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20116.250670 secs 2023-03-09T20:33:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:33:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20116.250670 secs 2023-03-09T20:33:27.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:33:27 smithi122 ceph-mon[93869]: pgmap v11241: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 131 KiB/s wr, 7 op/s 2023-03-09T20:33:27.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:33:27 smithi181 ceph-mon[94189]: pgmap v11241: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 131 KiB/s wr, 7 op/s 2023-03-09T20:33:27.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:33:27 smithi038 ceph-mon[92422]: pgmap v11241: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 131 KiB/s wr, 7 op/s 2023-03-09T20:33:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:33:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20118.582376 secs 2023-03-09T20:33:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:33:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20118.582376 secs 2023-03-09T20:33:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:33:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20118.582376 secs 2023-03-09T20:33:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20580.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:33:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:33:29 smithi122 ceph-mon[93869]: pgmap v11242: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 131 KiB/s wr, 8 op/s 2023-03-09T20:33:29.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:33:29 smithi181 ceph-mon[94189]: pgmap v11242: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 131 KiB/s wr, 8 op/s 2023-03-09T20:33:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:33:29 smithi038 ceph-mon[92422]: pgmap v11242: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 131 KiB/s wr, 8 op/s 2023-03-09T20:33:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:33:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20121.250730 secs 2023-03-09T20:33:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:33:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20121.250730 secs 2023-03-09T20:33:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:33:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20121.250730 secs 2023-03-09T20:33:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:33:31 smithi122 ceph-mon[93869]: pgmap v11243: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 115 KiB/s wr, 7 op/s 2023-03-09T20:33:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:33:31 smithi181 ceph-mon[94189]: pgmap v11243: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 115 KiB/s wr, 7 op/s 2023-03-09T20:33:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:33:31 smithi038 ceph-mon[92422]: pgmap v11243: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 115 KiB/s wr, 7 op/s 2023-03-09T20:33:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:33:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20123.582491 secs 2023-03-09T20:33:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:33:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20123.582491 secs 2023-03-09T20:33:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:33:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20123.582491 secs 2023-03-09T20:33:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:33:33 smithi122 ceph-mon[93869]: pgmap v11244: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 109 KiB/s wr, 9 op/s 2023-03-09T20:33:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:33:33 smithi181 ceph-mon[94189]: pgmap v11244: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 109 KiB/s wr, 9 op/s 2023-03-09T20:33:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:33:33 smithi038 ceph-mon[92422]: pgmap v11244: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 109 KiB/s wr, 9 op/s 2023-03-09T20:33:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20585.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:33:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:33:35 smithi122 ceph-mon[93869]: pgmap v11245: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 115 KiB/s wr, 7 op/s 2023-03-09T20:33:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:33:35 smithi181 ceph-mon[94189]: pgmap v11245: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 115 KiB/s wr, 7 op/s 2023-03-09T20:33:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:33:35 smithi038 ceph-mon[92422]: pgmap v11245: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 115 KiB/s wr, 7 op/s 2023-03-09T20:33:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:33:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20126.250788 secs 2023-03-09T20:33:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:33:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20126.250788 secs 2023-03-09T20:33:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:33:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20126.250788 secs 2023-03-09T20:33:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:33:37 smithi122 ceph-mon[93869]: pgmap v11246: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 77 KiB/s wr, 9 op/s 2023-03-09T20:33:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:33:37 smithi181 ceph-mon[94189]: pgmap v11246: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 77 KiB/s wr, 9 op/s 2023-03-09T20:33:37.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:33:37 smithi038 ceph-mon[92422]: pgmap v11246: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 77 KiB/s wr, 9 op/s 2023-03-09T20:33:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:33:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20128.582591 secs 2023-03-09T20:33:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:33:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20128.582591 secs 2023-03-09T20:33:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:33:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20128.582591 secs 2023-03-09T20:33:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20590.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:33:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:33:39 smithi122 ceph-mon[93869]: pgmap v11247: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 76 KiB/s wr, 6 op/s 2023-03-09T20:33:39.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:33:39 smithi181 ceph-mon[94189]: pgmap v11247: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 76 KiB/s wr, 6 op/s 2023-03-09T20:33:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:33:39 smithi038 ceph-mon[92422]: pgmap v11247: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 76 KiB/s wr, 6 op/s 2023-03-09T20:33:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:33:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20131.250848 secs 2023-03-09T20:33:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:33:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:33:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:33:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:33:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:33:41 smithi122 ceph-mon[93869]: pgmap v11248: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 159 KiB/s wr, 7 op/s 2023-03-09T20:33:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:33:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20131.250848 secs 2023-03-09T20:33:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:33:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:33:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:33:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:33:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:33:41 smithi181 ceph-mon[94189]: pgmap v11248: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 159 KiB/s wr, 7 op/s 2023-03-09T20:33:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:33:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20131.250848 secs 2023-03-09T20:33:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:33:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:33:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:33:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:33:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:33:41 smithi038 ceph-mon[92422]: pgmap v11248: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 159 KiB/s wr, 7 op/s 2023-03-09T20:33:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:33:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:33:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:33:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:33:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:33:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:33:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:33:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20133.582680 secs 2023-03-09T20:33:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:33:43 smithi122 ceph-mon[93869]: pgmap v11249: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 159 KiB/s wr, 9 op/s 2023-03-09T20:33:43.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:33:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:33:43.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:33:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:33:43.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:33:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:33:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:33:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20133.582680 secs 2023-03-09T20:33:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:33:43 smithi181 ceph-mon[94189]: pgmap v11249: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 159 KiB/s wr, 9 op/s 2023-03-09T20:33:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:33:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:33:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:33:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:33:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:33:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:33:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:33:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20133.582680 secs 2023-03-09T20:33:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:33:43 smithi038 ceph-mon[92422]: pgmap v11249: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 159 KiB/s wr, 9 op/s 2023-03-09T20:33:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:33:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:33:43.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:33:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:33:43.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:33:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:33:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20595.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:33:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:33:45 smithi122 ceph-mon[93869]: pgmap v11250: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 159 KiB/s wr, 7 op/s 2023-03-09T20:33:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:33:45 smithi181 ceph-mon[94189]: pgmap v11250: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 159 KiB/s wr, 7 op/s 2023-03-09T20:33:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:33:45 smithi038 ceph-mon[92422]: pgmap v11250: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 159 KiB/s wr, 7 op/s 2023-03-09T20:33:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:33:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20136.250911 secs 2023-03-09T20:33:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:33:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20136.250911 secs 2023-03-09T20:33:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:33:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20136.250911 secs 2023-03-09T20:33:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:33:47 smithi122 ceph-mon[93869]: pgmap v11251: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 313 KiB/s wr, 9 op/s 2023-03-09T20:33:47.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:33:47 smithi181 ceph-mon[94189]: pgmap v11251: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 313 KiB/s wr, 9 op/s 2023-03-09T20:33:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:33:47 smithi038 ceph-mon[92422]: pgmap v11251: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 313 KiB/s wr, 9 op/s 2023-03-09T20:33:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:33:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:33:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:33:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:33:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:33:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20138.582818 secs 2023-03-09T20:33:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:33:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:33:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:33:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:33:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:33:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20138.582818 secs 2023-03-09T20:33:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:33:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:33:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:33:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:33:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:33:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20138.582818 secs 2023-03-09T20:33:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20600.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:33:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:33:49 smithi122 ceph-mon[93869]: pgmap v11252: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 243 KiB/s wr, 7 op/s 2023-03-09T20:33:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:33:49 smithi181 ceph-mon[94189]: pgmap v11252: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 243 KiB/s wr, 7 op/s 2023-03-09T20:33:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:33:49 smithi038 ceph-mon[92422]: pgmap v11252: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 243 KiB/s wr, 7 op/s 2023-03-09T20:33:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:33:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20141.250975 secs 2023-03-09T20:33:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:33:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20141.250975 secs 2023-03-09T20:33:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:33:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20141.250975 secs 2023-03-09T20:33:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:33:51 smithi122 ceph-mon[93869]: pgmap v11253: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 288 KiB/s wr, 8 op/s 2023-03-09T20:33:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:33:51 smithi181 ceph-mon[94189]: pgmap v11253: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 288 KiB/s wr, 8 op/s 2023-03-09T20:33:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:33:51 smithi038 ceph-mon[92422]: pgmap v11253: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 288 KiB/s wr, 8 op/s 2023-03-09T20:33:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:33:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20143.582877 secs 2023-03-09T20:33:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:33:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20143.582877 secs 2023-03-09T20:33:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:33:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20143.582877 secs 2023-03-09T20:33:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:33:53 smithi122 ceph-mon[93869]: pgmap v11254: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 222 KiB/s wr, 10 op/s 2023-03-09T20:33:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:33:53 smithi181 ceph-mon[94189]: pgmap v11254: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 222 KiB/s wr, 10 op/s 2023-03-09T20:33:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:33:53 smithi038 ceph-mon[92422]: pgmap v11254: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 222 KiB/s wr, 10 op/s 2023-03-09T20:33:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20605.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:33:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:33:55 smithi122 ceph-mon[93869]: pgmap v11255: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 406 KiB/s rd, 484 KiB/s wr, 9 op/s 2023-03-09T20:33:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:33:55 smithi181 ceph-mon[94189]: pgmap v11255: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 406 KiB/s rd, 484 KiB/s wr, 9 op/s 2023-03-09T20:33:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:33:55 smithi038 ceph-mon[92422]: pgmap v11255: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 406 KiB/s rd, 484 KiB/s wr, 9 op/s 2023-03-09T20:33:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:33:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20146.251036 secs 2023-03-09T20:33:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:33:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:33:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:33:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:33:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:33:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20146.251036 secs 2023-03-09T20:33:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:33:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:33:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:33:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:33:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:33:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20146.251036 secs 2023-03-09T20:33:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:33:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:33:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:33:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:33:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:33:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:33:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:33:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:33:57.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:33:57 smithi122 ceph-mon[93869]: pgmap v11256: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 500 KiB/s wr, 12 op/s 2023-03-09T20:33:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:33:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:33:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:33:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:33:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:33:57 smithi181 ceph-mon[94189]: pgmap v11256: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 500 KiB/s wr, 12 op/s 2023-03-09T20:33:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:33:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:33:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:33:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:33:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:33:57 smithi038 ceph-mon[92422]: pgmap v11256: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 500 KiB/s wr, 12 op/s 2023-03-09T20:33:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20610.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:33:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:33:59 smithi122 ceph-mon[93869]: pgmap v11257: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 340 KiB/s wr, 10 op/s 2023-03-09T20:33:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:33:59 smithi181 ceph-mon[94189]: pgmap v11257: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 340 KiB/s wr, 10 op/s 2023-03-09T20:33:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:33:59 smithi038 ceph-mon[92422]: pgmap v11257: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 340 KiB/s wr, 10 op/s 2023-03-09T20:34:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20151.251095 secs 2023-03-09T20:34:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:00 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:34:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20151.251095 secs 2023-03-09T20:34:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:00 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:34:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20151.251095 secs 2023-03-09T20:34:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:00 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:34:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:01 smithi122 ceph-mon[93869]: pgmap v11258: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 420 KiB/s wr, 10 op/s 2023-03-09T20:34:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:01 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:34:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:34:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:34:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:01 smithi181 ceph-mon[94189]: pgmap v11258: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 420 KiB/s wr, 10 op/s 2023-03-09T20:34:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:01 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:34:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:34:01.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:34:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:01 smithi038 ceph-mon[92422]: pgmap v11258: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 420 KiB/s wr, 10 op/s 2023-03-09T20:34:02.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:01 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:34:02.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:34:02.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:34:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20615.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:34:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:03 smithi122 ceph-mon[93869]: pgmap v11259: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 376 KiB/s wr, 12 op/s 2023-03-09T20:34:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:03 smithi181 ceph-mon[94189]: pgmap v11259: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 376 KiB/s wr, 12 op/s 2023-03-09T20:34:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:03 smithi038 ceph-mon[92422]: pgmap v11259: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 376 KiB/s wr, 12 op/s 2023-03-09T20:34:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:05 smithi122 ceph-mon[93869]: pgmap v11260: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 556 KiB/s rd, 505 KiB/s wr, 21 op/s 2023-03-09T20:34:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:05 smithi181 ceph-mon[94189]: pgmap v11260: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 556 KiB/s rd, 505 KiB/s wr, 21 op/s 2023-03-09T20:34:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:05 smithi038 ceph-mon[92422]: pgmap v11260: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 556 KiB/s rd, 505 KiB/s wr, 21 op/s 2023-03-09T20:34:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20156.251156 secs 2023-03-09T20:34:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20156.251156 secs 2023-03-09T20:34:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20156.251156 secs 2023-03-09T20:34:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:07 smithi122 ceph-mon[93869]: pgmap v11261: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 270 KiB/s wr, 24 op/s 2023-03-09T20:34:07.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:07 smithi181 ceph-mon[94189]: pgmap v11261: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 270 KiB/s wr, 24 op/s 2023-03-09T20:34:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:07 smithi038 ceph-mon[92422]: pgmap v11261: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 270 KiB/s wr, 24 op/s 2023-03-09T20:34:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:34:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:34:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:34:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:34:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:34:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:34:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20620.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:34:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:34:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:34:09.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:09 smithi122 ceph-mon[93869]: pgmap v11262: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 425 KiB/s wr, 22 op/s 2023-03-09T20:34:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:34:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:34:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:09 smithi181 ceph-mon[94189]: pgmap v11262: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 425 KiB/s wr, 22 op/s 2023-03-09T20:34:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:34:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:34:09.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:09 smithi038 ceph-mon[92422]: pgmap v11262: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 425 KiB/s wr, 22 op/s 2023-03-09T20:34:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20161.251215 secs 2023-03-09T20:34:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20161.251215 secs 2023-03-09T20:34:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20161.251215 secs 2023-03-09T20:34:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:11 smithi038 ceph-mon[92422]: pgmap v11263: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 424 KiB/s wr, 22 op/s 2023-03-09T20:34:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:11 smithi122 ceph-mon[93869]: pgmap v11263: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 424 KiB/s wr, 22 op/s 2023-03-09T20:34:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:11 smithi181 ceph-mon[94189]: pgmap v11263: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 424 KiB/s wr, 22 op/s 2023-03-09T20:34:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20625.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:34:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:13 smithi122 ceph-mon[93869]: pgmap v11264: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 378 KiB/s wr, 24 op/s 2023-03-09T20:34:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:13 smithi181 ceph-mon[94189]: pgmap v11264: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 378 KiB/s wr, 24 op/s 2023-03-09T20:34:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:13 smithi038 ceph-mon[92422]: pgmap v11264: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 378 KiB/s wr, 24 op/s 2023-03-09T20:34:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:15 smithi122 ceph-mon[93869]: pgmap v11265: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 627 KiB/s rd, 414 KiB/s wr, 23 op/s 2023-03-09T20:34:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:15 smithi181 ceph-mon[94189]: pgmap v11265: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 627 KiB/s rd, 414 KiB/s wr, 23 op/s 2023-03-09T20:34:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:15 smithi038 ceph-mon[92422]: pgmap v11265: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 627 KiB/s rd, 414 KiB/s wr, 23 op/s 2023-03-09T20:34:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20166.251275 secs 2023-03-09T20:34:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20166.251275 secs 2023-03-09T20:34:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20166.251275 secs 2023-03-09T20:34:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:17 smithi122 ceph-mon[93869]: pgmap v11266: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 331 KiB/s wr, 13 op/s 2023-03-09T20:34:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:17 smithi181 ceph-mon[94189]: pgmap v11266: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 331 KiB/s wr, 13 op/s 2023-03-09T20:34:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:17 smithi038 ceph-mon[92422]: pgmap v11266: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 331 KiB/s wr, 13 op/s 2023-03-09T20:34:18.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20630.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:34:19.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:19 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:34:19.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:19 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:34:19.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:19 smithi122 ceph-mon[93869]: pgmap v11267: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 304 KiB/s wr, 9 op/s 2023-03-09T20:34:19.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:19 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:34:19.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:19 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:34:19.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:19 smithi181 ceph-mon[94189]: pgmap v11267: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 304 KiB/s wr, 9 op/s 2023-03-09T20:34:19.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:19 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:34:19.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:19 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:34:19.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:19 smithi038 ceph-mon[92422]: pgmap v11267: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 304 KiB/s wr, 9 op/s 2023-03-09T20:34:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20171.251358 secs 2023-03-09T20:34:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20171.251358 secs 2023-03-09T20:34:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20171.251358 secs 2023-03-09T20:34:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:34:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:34:21.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:21 smithi122 ceph-mon[93869]: pgmap v11268: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 133 KiB/s wr, 8 op/s 2023-03-09T20:34:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:34:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:34:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:21 smithi181 ceph-mon[94189]: pgmap v11268: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 133 KiB/s wr, 8 op/s 2023-03-09T20:34:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:34:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:34:21.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:21 smithi038 ceph-mon[92422]: pgmap v11268: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 133 KiB/s wr, 8 op/s 2023-03-09T20:34:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20173.583500 secs 2023-03-09T20:34:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20173.583500 secs 2023-03-09T20:34:22.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20173.583500 secs 2023-03-09T20:34:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:23 smithi122 ceph-mon[93869]: pgmap v11269: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 317 KiB/s wr, 12 op/s 2023-03-09T20:34:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:23 smithi181 ceph-mon[94189]: pgmap v11269: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 317 KiB/s wr, 12 op/s 2023-03-09T20:34:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:23 smithi038 ceph-mon[92422]: pgmap v11269: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 317 KiB/s wr, 12 op/s 2023-03-09T20:34:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20635.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:34:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:25 smithi122 ceph-mon[93869]: osdmap e917: 12 total, 12 up, 12 in 2023-03-09T20:34:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:25 smithi122 ceph-mon[93869]: pgmap v11271: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 310 KiB/s wr, 11 op/s 2023-03-09T20:34:25.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20176.251416 secs 2023-03-09T20:34:25.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:25 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:34:25.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:25 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:34:25.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:25 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:34:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:25 smithi181 ceph-mon[94189]: osdmap e917: 12 total, 12 up, 12 in 2023-03-09T20:34:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:25 smithi181 ceph-mon[94189]: pgmap v11271: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 310 KiB/s wr, 11 op/s 2023-03-09T20:34:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20176.251416 secs 2023-03-09T20:34:25.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:25 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:34:25.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:25 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:34:25.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:25 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:34:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:25 smithi038 ceph-mon[92422]: osdmap e917: 12 total, 12 up, 12 in 2023-03-09T20:34:25.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:25 smithi038 ceph-mon[92422]: pgmap v11271: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 310 KiB/s wr, 11 op/s 2023-03-09T20:34:25.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20176.251416 secs 2023-03-09T20:34:25.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:25 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:34:25.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:25 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:34:25.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:25 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:34:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:27 smithi122 ceph-mon[93869]: pgmap v11272: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 328 KiB/s wr, 11 op/s 2023-03-09T20:34:27.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20178.583620 secs 2023-03-09T20:34:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:27 smithi181 ceph-mon[94189]: pgmap v11272: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 328 KiB/s wr, 11 op/s 2023-03-09T20:34:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20178.583620 secs 2023-03-09T20:34:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:27 smithi038 ceph-mon[92422]: pgmap v11272: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 328 KiB/s wr, 11 op/s 2023-03-09T20:34:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20178.583620 secs 2023-03-09T20:34:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20640.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:34:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:29 smithi122 ceph-mon[93869]: pgmap v11273: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 427 KiB/s wr, 12 op/s 2023-03-09T20:34:29.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:29 smithi181 ceph-mon[94189]: pgmap v11273: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 427 KiB/s wr, 12 op/s 2023-03-09T20:34:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:29 smithi038 ceph-mon[92422]: pgmap v11273: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 427 KiB/s wr, 12 op/s 2023-03-09T20:34:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20181.251477 secs 2023-03-09T20:34:30.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20181.251477 secs 2023-03-09T20:34:30.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20181.251477 secs 2023-03-09T20:34:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:31 smithi122 ceph-mon[93869]: pgmap v11274: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 427 KiB/s wr, 12 op/s 2023-03-09T20:34:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:31 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:34:31.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:31 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:34:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:31 smithi181 ceph-mon[94189]: pgmap v11274: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 427 KiB/s wr, 12 op/s 2023-03-09T20:34:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:31 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:34:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:31 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:34:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:31 smithi038 ceph-mon[92422]: pgmap v11274: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 427 KiB/s wr, 12 op/s 2023-03-09T20:34:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:31 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:34:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:31 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:34:32.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:32 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:34:32.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:32 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:34:32.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20183.583751 secs 2023-03-09T20:34:32.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:32 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:34:32.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:32 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:34:32.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20183.583751 secs 2023-03-09T20:34:32.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:32 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:34:32.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:32 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:34:33.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20183.583751 secs 2023-03-09T20:34:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20645.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:34:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:33 smithi122 ceph-mon[93869]: pgmap v11275: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 245 KiB/s wr, 11 op/s 2023-03-09T20:34:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:33 smithi181 ceph-mon[94189]: pgmap v11275: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 245 KiB/s wr, 11 op/s 2023-03-09T20:34:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:33 smithi038 ceph-mon[92422]: pgmap v11275: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 245 KiB/s wr, 11 op/s 2023-03-09T20:34:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:35 smithi122 ceph-mon[93869]: osdmap e918: 12 total, 12 up, 12 in 2023-03-09T20:34:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:35 smithi122 ceph-mon[93869]: pgmap v11277: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 231 KiB/s wr, 9 op/s 2023-03-09T20:34:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:35 smithi181 ceph-mon[94189]: osdmap e918: 12 total, 12 up, 12 in 2023-03-09T20:34:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:35 smithi181 ceph-mon[94189]: pgmap v11277: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 231 KiB/s wr, 9 op/s 2023-03-09T20:34:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:35 smithi038 ceph-mon[92422]: osdmap e918: 12 total, 12 up, 12 in 2023-03-09T20:34:35.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:35 smithi038 ceph-mon[92422]: pgmap v11277: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 231 KiB/s wr, 9 op/s 2023-03-09T20:34:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20186.251537 secs 2023-03-09T20:34:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20186.251537 secs 2023-03-09T20:34:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20186.251537 secs 2023-03-09T20:34:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:37 smithi122 ceph-mon[93869]: pgmap v11278: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 139 KiB/s wr, 9 op/s 2023-03-09T20:34:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:37 smithi181 ceph-mon[94189]: pgmap v11278: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 139 KiB/s wr, 9 op/s 2023-03-09T20:34:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:37 smithi038 ceph-mon[92422]: pgmap v11278: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 139 KiB/s wr, 9 op/s 2023-03-09T20:34:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20650.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:34:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:39 smithi122 ceph-mon[93869]: pgmap v11279: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 103 KiB/s wr, 9 op/s 2023-03-09T20:34:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:39 smithi181 ceph-mon[94189]: pgmap v11279: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 103 KiB/s wr, 9 op/s 2023-03-09T20:34:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:39 smithi038 ceph-mon[92422]: pgmap v11279: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 103 KiB/s wr, 9 op/s 2023-03-09T20:34:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20191.251621 secs 2023-03-09T20:34:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:40 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:34:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20191.251621 secs 2023-03-09T20:34:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:40 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:34:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20191.251621 secs 2023-03-09T20:34:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:40 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:34:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:41 smithi122 ceph-mon[93869]: pgmap v11280: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 103 KiB/s wr, 9 op/s 2023-03-09T20:34:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:41 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:34:41.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:34:41.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:34:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:41 smithi181 ceph-mon[94189]: pgmap v11280: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 103 KiB/s wr, 9 op/s 2023-03-09T20:34:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:41 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:34:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:34:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:34:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:41 smithi038 ceph-mon[92422]: pgmap v11280: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 103 KiB/s wr, 9 op/s 2023-03-09T20:34:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:41 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:34:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:34:42.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:34:43.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20655.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:34:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:43 smithi122 ceph-mon[93869]: pgmap v11281: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 84 KiB/s wr, 8 op/s 2023-03-09T20:34:43.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:34:43.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:34:43.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:34:43.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:34:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:43 smithi181 ceph-mon[94189]: pgmap v11281: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 84 KiB/s wr, 8 op/s 2023-03-09T20:34:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:34:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:34:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:34:43.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:34:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:43 smithi038 ceph-mon[92422]: pgmap v11281: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 84 KiB/s wr, 8 op/s 2023-03-09T20:34:44.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:34:44.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:34:44.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:34:44.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:34:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:45 smithi122 ceph-mon[93869]: pgmap v11282: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 320 KiB/s wr, 9 op/s 2023-03-09T20:34:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:45 smithi181 ceph-mon[94189]: pgmap v11282: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 320 KiB/s wr, 9 op/s 2023-03-09T20:34:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:45 smithi038 ceph-mon[92422]: pgmap v11282: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 320 KiB/s wr, 9 op/s 2023-03-09T20:34:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20196.251683 secs 2023-03-09T20:34:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20196.251683 secs 2023-03-09T20:34:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20196.251683 secs 2023-03-09T20:34:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:47 smithi122 ceph-mon[93869]: pgmap v11283: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 293 KiB/s wr, 10 op/s 2023-03-09T20:34:47.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:47 smithi181 ceph-mon[94189]: pgmap v11283: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 293 KiB/s wr, 10 op/s 2023-03-09T20:34:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:47 smithi038 ceph-mon[92422]: pgmap v11283: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 293 KiB/s wr, 10 op/s 2023-03-09T20:34:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20660.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:34:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:49 smithi122 ceph-mon[93869]: pgmap v11284: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 498 KiB/s wr, 10 op/s 2023-03-09T20:34:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:49 smithi181 ceph-mon[94189]: pgmap v11284: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 498 KiB/s wr, 10 op/s 2023-03-09T20:34:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:49 smithi038 ceph-mon[92422]: pgmap v11284: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 498 KiB/s wr, 10 op/s 2023-03-09T20:34:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20201.251744 secs 2023-03-09T20:34:50.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20201.251744 secs 2023-03-09T20:34:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20201.251744 secs 2023-03-09T20:34:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:51 smithi122 ceph-mon[93869]: pgmap v11285: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 446 KiB/s wr, 9 op/s 2023-03-09T20:34:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:34:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:34:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:51 smithi181 ceph-mon[94189]: pgmap v11285: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 446 KiB/s wr, 9 op/s 2023-03-09T20:34:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:34:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:34:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:51 smithi038 ceph-mon[92422]: pgmap v11285: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 446 KiB/s wr, 9 op/s 2023-03-09T20:34:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:34:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:34:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:34:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:34:52.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:34:52.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:34:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:34:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:34:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20665.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:34:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:53 smithi122 ceph-mon[93869]: pgmap v11286: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 463 KiB/s wr, 12 op/s 2023-03-09T20:34:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:53 smithi181 ceph-mon[94189]: pgmap v11286: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 463 KiB/s wr, 12 op/s 2023-03-09T20:34:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:53 smithi038 ceph-mon[92422]: pgmap v11286: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 463 KiB/s wr, 12 op/s 2023-03-09T20:34:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:55 smithi122 ceph-mon[93869]: pgmap v11287: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 519 KiB/s wr, 11 op/s 2023-03-09T20:34:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:55 smithi181 ceph-mon[94189]: pgmap v11287: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 519 KiB/s wr, 11 op/s 2023-03-09T20:34:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:55 smithi038 ceph-mon[92422]: pgmap v11287: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 519 KiB/s wr, 11 op/s 2023-03-09T20:34:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20206.251803 secs 2023-03-09T20:34:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20206.251803 secs 2023-03-09T20:34:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20206.251803 secs 2023-03-09T20:34:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:57 smithi122 ceph-mon[93869]: pgmap v11288: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 328 KiB/s wr, 12 op/s 2023-03-09T20:34:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:57 smithi181 ceph-mon[94189]: pgmap v11288: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 328 KiB/s wr, 12 op/s 2023-03-09T20:34:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:57 smithi038 ceph-mon[92422]: pgmap v11288: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 328 KiB/s wr, 12 op/s 2023-03-09T20:34:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:58 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 20208.584351 secs 2023-03-09T20:34:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:58 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 20208.584351 secs 2023-03-09T20:34:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:58 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 20208.584351 secs 2023-03-09T20:34:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20670.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:34:59.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:34:59 smithi122 ceph-mon[93869]: pgmap v11289: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 590 KiB/s wr, 12 op/s 2023-03-09T20:34:59.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:34:59 smithi181 ceph-mon[94189]: pgmap v11289: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 590 KiB/s wr, 12 op/s 2023-03-09T20:34:59.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:34:59 smithi038 ceph-mon[92422]: pgmap v11289: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 590 KiB/s wr, 12 op/s 2023-03-09T20:35:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:35:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:35:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:35:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:35:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:35:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:35:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20211.251863 secs 2023-03-09T20:35:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:35:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:35:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:01 smithi122 ceph-mon[93869]: pgmap v11290: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 384 KiB/s wr, 9 op/s 2023-03-09T20:35:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20211.251863 secs 2023-03-09T20:35:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:35:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:35:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:01 smithi181 ceph-mon[94189]: pgmap v11290: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 384 KiB/s wr, 9 op/s 2023-03-09T20:35:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20211.251863 secs 2023-03-09T20:35:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:35:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:35:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:01 smithi038 ceph-mon[92422]: pgmap v11290: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 384 KiB/s wr, 9 op/s 2023-03-09T20:35:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:02 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:35:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:02 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:35:02.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:02 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:35:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:02 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:35:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:02 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:35:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:02 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:35:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:02 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:35:02.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:02 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:35:02.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:02 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:35:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20213.584463 secs 2023-03-09T20:35:03.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:03 smithi122 ceph-mon[93869]: pgmap v11291: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 422 KiB/s wr, 14 op/s 2023-03-09T20:35:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20213.584463 secs 2023-03-09T20:35:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:03 smithi181 ceph-mon[94189]: pgmap v11291: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 422 KiB/s wr, 14 op/s 2023-03-09T20:35:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20213.584463 secs 2023-03-09T20:35:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:03 smithi038 ceph-mon[92422]: pgmap v11291: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 422 KiB/s wr, 14 op/s 2023-03-09T20:35:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20675.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:35:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:05 smithi122 ceph-mon[93869]: pgmap v11292: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 470 KiB/s wr, 25 op/s 2023-03-09T20:35:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:05 smithi181 ceph-mon[94189]: pgmap v11292: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 470 KiB/s wr, 25 op/s 2023-03-09T20:35:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:05 smithi038 ceph-mon[92422]: pgmap v11292: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 470 KiB/s wr, 25 op/s 2023-03-09T20:35:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20216.251922 secs 2023-03-09T20:35:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:06 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:35:06.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:35:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20216.251922 secs 2023-03-09T20:35:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:06 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:35:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:35:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20216.251922 secs 2023-03-09T20:35:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:06 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:35:06.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:35:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:07 smithi122 ceph-mon[93869]: pgmap v11293: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 397 KiB/s wr, 25 op/s 2023-03-09T20:35:07.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:07 smithi181 ceph-mon[94189]: pgmap v11293: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 397 KiB/s wr, 25 op/s 2023-03-09T20:35:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:07 smithi038 ceph-mon[92422]: pgmap v11293: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 397 KiB/s wr, 25 op/s 2023-03-09T20:35:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20218.584537 secs 2023-03-09T20:35:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20218.584537 secs 2023-03-09T20:35:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20218.584537 secs 2023-03-09T20:35:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20680.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:35:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:09 smithi122 ceph-mon[93869]: pgmap v11294: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 444 KiB/s wr, 24 op/s 2023-03-09T20:35:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:09 smithi181 ceph-mon[94189]: pgmap v11294: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 444 KiB/s wr, 24 op/s 2023-03-09T20:35:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:09 smithi038 ceph-mon[92422]: pgmap v11294: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 444 KiB/s wr, 24 op/s 2023-03-09T20:35:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:10 smithi122 ceph-mon[93869]: osdmap e919: 12 total, 12 up, 12 in 2023-03-09T20:35:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:10 smithi181 ceph-mon[94189]: osdmap e919: 12 total, 12 up, 12 in 2023-03-09T20:35:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:10 smithi038 ceph-mon[92422]: osdmap e919: 12 total, 12 up, 12 in 2023-03-09T20:35:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20221.251999 secs 2023-03-09T20:35:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:11 smithi122 ceph-mon[93869]: pgmap v11296: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 200 KiB/s wr, 26 op/s 2023-03-09T20:35:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20221.251999 secs 2023-03-09T20:35:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:11 smithi181 ceph-mon[94189]: pgmap v11296: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 200 KiB/s wr, 26 op/s 2023-03-09T20:35:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20221.251999 secs 2023-03-09T20:35:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:11 smithi038 ceph-mon[92422]: pgmap v11296: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 200 KiB/s wr, 26 op/s 2023-03-09T20:35:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20223.584687 secs 2023-03-09T20:35:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20223.584687 secs 2023-03-09T20:35:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20223.584687 secs 2023-03-09T20:35:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:13 smithi038 ceph-mon[92422]: pgmap v11297: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 154 KiB/s wr, 23 op/s 2023-03-09T20:35:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:13 smithi122 ceph-mon[93869]: pgmap v11297: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 154 KiB/s wr, 23 op/s 2023-03-09T20:35:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:13 smithi181 ceph-mon[94189]: pgmap v11297: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 154 KiB/s wr, 23 op/s 2023-03-09T20:35:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20685.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:35:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:15 smithi122 ceph-mon[93869]: pgmap v11298: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 124 KiB/s wr, 8 op/s 2023-03-09T20:35:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:15 smithi181 ceph-mon[94189]: pgmap v11298: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 124 KiB/s wr, 8 op/s 2023-03-09T20:35:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:15 smithi038 ceph-mon[92422]: pgmap v11298: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 124 KiB/s wr, 8 op/s 2023-03-09T20:35:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20226.252072 secs 2023-03-09T20:35:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20226.252072 secs 2023-03-09T20:35:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20226.252072 secs 2023-03-09T20:35:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:17 smithi122 ceph-mon[93869]: pgmap v11299: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 143 KiB/s wr, 8 op/s 2023-03-09T20:35:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:17 smithi181 ceph-mon[94189]: pgmap v11299: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 143 KiB/s wr, 8 op/s 2023-03-09T20:35:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:17 smithi038 ceph-mon[92422]: pgmap v11299: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 143 KiB/s wr, 8 op/s 2023-03-09T20:35:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20228.584766 secs 2023-03-09T20:35:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20228.584766 secs 2023-03-09T20:35:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20228.584766 secs 2023-03-09T20:35:18.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20690.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:35:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:19 smithi122 ceph-mon[93869]: pgmap v11300: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 95 KiB/s wr, 8 op/s 2023-03-09T20:35:19.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:19 smithi122 ceph-mon[93869]: osdmap e920: 12 total, 12 up, 12 in 2023-03-09T20:35:19.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:19 smithi181 ceph-mon[94189]: pgmap v11300: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 95 KiB/s wr, 8 op/s 2023-03-09T20:35:19.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:19 smithi181 ceph-mon[94189]: osdmap e920: 12 total, 12 up, 12 in 2023-03-09T20:35:19.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:19 smithi038 ceph-mon[92422]: pgmap v11300: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 95 KiB/s wr, 8 op/s 2023-03-09T20:35:19.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:19 smithi038 ceph-mon[92422]: osdmap e920: 12 total, 12 up, 12 in 2023-03-09T20:35:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20231.252131 secs 2023-03-09T20:35:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20231.252131 secs 2023-03-09T20:35:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20231.252131 secs 2023-03-09T20:35:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:21 smithi122 ceph-mon[93869]: pgmap v11302: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 95 KiB/s wr, 8 op/s 2023-03-09T20:35:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:21 smithi181 ceph-mon[94189]: pgmap v11302: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 95 KiB/s wr, 8 op/s 2023-03-09T20:35:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:21 smithi038 ceph-mon[92422]: pgmap v11302: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 95 KiB/s wr, 8 op/s 2023-03-09T20:35:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20233.584884 secs 2023-03-09T20:35:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20233.584884 secs 2023-03-09T20:35:22.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20233.584884 secs 2023-03-09T20:35:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:23 smithi122 ceph-mon[93869]: pgmap v11303: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 114 KiB/s wr, 9 op/s 2023-03-09T20:35:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:23 smithi181 ceph-mon[94189]: pgmap v11303: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 114 KiB/s wr, 9 op/s 2023-03-09T20:35:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:23 smithi038 ceph-mon[92422]: pgmap v11303: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 114 KiB/s wr, 9 op/s 2023-03-09T20:35:23.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20695.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:35:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:25 smithi122 ceph-mon[93869]: pgmap v11304: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 142 KiB/s wr, 9 op/s 2023-03-09T20:35:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:25 smithi181 ceph-mon[94189]: pgmap v11304: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 142 KiB/s wr, 9 op/s 2023-03-09T20:35:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:25 smithi038 ceph-mon[92422]: pgmap v11304: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 142 KiB/s wr, 9 op/s 2023-03-09T20:35:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20236.252200 secs 2023-03-09T20:35:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20236.252200 secs 2023-03-09T20:35:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20236.252200 secs 2023-03-09T20:35:27.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:27 smithi122 ceph-mon[93869]: pgmap v11305: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 123 KiB/s wr, 9 op/s 2023-03-09T20:35:27.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:27 smithi181 ceph-mon[94189]: pgmap v11305: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 123 KiB/s wr, 9 op/s 2023-03-09T20:35:27.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:27 smithi038 ceph-mon[92422]: pgmap v11305: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 123 KiB/s wr, 9 op/s 2023-03-09T20:35:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20238.584979 secs 2023-03-09T20:35:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20238.584979 secs 2023-03-09T20:35:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20238.584979 secs 2023-03-09T20:35:28.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20700.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:35:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:29 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:35:29.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:29 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:35:29.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:29 smithi122 ceph-mon[93869]: pgmap v11306: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 96 KiB/s wr, 8 op/s 2023-03-09T20:35:29.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:29 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:35:29.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:29 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:35:29.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:29 smithi181 ceph-mon[94189]: pgmap v11306: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 96 KiB/s wr, 8 op/s 2023-03-09T20:35:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:29 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:35:29.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:29 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:35:29.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:29 smithi038 ceph-mon[92422]: pgmap v11306: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 96 KiB/s wr, 8 op/s 2023-03-09T20:35:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20241.252284 secs 2023-03-09T20:35:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20241.252284 secs 2023-03-09T20:35:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20241.252284 secs 2023-03-09T20:35:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:31 smithi122 ceph-mon[93869]: pgmap v11307: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 165 KiB/s wr, 8 op/s 2023-03-09T20:35:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:31 smithi181 ceph-mon[94189]: pgmap v11307: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 165 KiB/s wr, 8 op/s 2023-03-09T20:35:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:31 smithi038 ceph-mon[92422]: pgmap v11307: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 165 KiB/s wr, 8 op/s 2023-03-09T20:35:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:32 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:35:32.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:32 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:35:32.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20243.585117 secs 2023-03-09T20:35:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:32 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:35:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:32 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:35:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20243.585117 secs 2023-03-09T20:35:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:32 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:35:32.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:32 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:35:32.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20243.585117 secs 2023-03-09T20:35:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:33 smithi122 ceph-mon[93869]: pgmap v11308: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 158 KiB/s wr, 10 op/s 2023-03-09T20:35:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:33 smithi181 ceph-mon[94189]: pgmap v11308: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 158 KiB/s wr, 10 op/s 2023-03-09T20:35:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:33 smithi038 ceph-mon[92422]: pgmap v11308: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 158 KiB/s wr, 10 op/s 2023-03-09T20:35:33.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20705.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:35:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:35 smithi122 ceph-mon[93869]: pgmap v11309: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 142 KiB/s wr, 8 op/s 2023-03-09T20:35:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:35 smithi181 ceph-mon[94189]: pgmap v11309: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 142 KiB/s wr, 8 op/s 2023-03-09T20:35:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:35 smithi038 ceph-mon[92422]: pgmap v11309: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 142 KiB/s wr, 8 op/s 2023-03-09T20:35:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20246.252344 secs 2023-03-09T20:35:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20246.252344 secs 2023-03-09T20:35:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20246.252344 secs 2023-03-09T20:35:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:37 smithi122 ceph-mon[93869]: pgmap v11310: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 78 KiB/s wr, 9 op/s 2023-03-09T20:35:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:37 smithi181 ceph-mon[94189]: pgmap v11310: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 78 KiB/s wr, 9 op/s 2023-03-09T20:35:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:37 smithi038 ceph-mon[92422]: pgmap v11310: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 78 KiB/s wr, 9 op/s 2023-03-09T20:35:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20248.585231 secs 2023-03-09T20:35:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20248.585231 secs 2023-03-09T20:35:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20248.585231 secs 2023-03-09T20:35:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20710.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:35:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:39 smithi122 ceph-mon[93869]: pgmap v11311: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 78 KiB/s wr, 7 op/s 2023-03-09T20:35:39.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:39 smithi181 ceph-mon[94189]: pgmap v11311: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 78 KiB/s wr, 7 op/s 2023-03-09T20:35:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:39 smithi038 ceph-mon[92422]: pgmap v11311: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 78 KiB/s wr, 7 op/s 2023-03-09T20:35:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20251.252404 secs 2023-03-09T20:35:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:35:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:35:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:41 smithi122 ceph-mon[93869]: pgmap v11312: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 269 KiB/s wr, 8 op/s 2023-03-09T20:35:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20251.252404 secs 2023-03-09T20:35:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:35:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:35:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:41 smithi181 ceph-mon[94189]: pgmap v11312: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 269 KiB/s wr, 8 op/s 2023-03-09T20:35:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20251.252404 secs 2023-03-09T20:35:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:35:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:35:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:41 smithi038 ceph-mon[92422]: pgmap v11312: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 269 KiB/s wr, 8 op/s 2023-03-09T20:35:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20253.585337 secs 2023-03-09T20:35:42.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20253.585337 secs 2023-03-09T20:35:42.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20253.585337 secs 2023-03-09T20:35:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:43 smithi038 ceph-mon[92422]: pgmap v11313: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 195 KiB/s wr, 10 op/s 2023-03-09T20:35:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:43 smithi122 ceph-mon[93869]: pgmap v11313: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 195 KiB/s wr, 10 op/s 2023-03-09T20:35:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:43 smithi181 ceph-mon[94189]: pgmap v11313: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 195 KiB/s wr, 10 op/s 2023-03-09T20:35:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20715.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:35:44.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:44 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:35:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:44 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:35:44.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:44 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:35:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:45 smithi122 ceph-mon[93869]: pgmap v11314: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 208 KiB/s wr, 8 op/s 2023-03-09T20:35:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20256.252462 secs 2023-03-09T20:35:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:45 smithi181 ceph-mon[94189]: pgmap v11314: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 208 KiB/s wr, 8 op/s 2023-03-09T20:35:45.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20256.252462 secs 2023-03-09T20:35:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:45 smithi038 ceph-mon[92422]: pgmap v11314: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 208 KiB/s wr, 8 op/s 2023-03-09T20:35:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20256.252462 secs 2023-03-09T20:35:47.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:35:47.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:35:47.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:35:47.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:35:47.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:35:47.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:35:47.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:47 smithi122 ceph-mon[93869]: pgmap v11315: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 366 KiB/s wr, 11 op/s 2023-03-09T20:35:47.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:35:47.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:35:47.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:35:47.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:35:47.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:35:47.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:35:47.731 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:35:47.731 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:35:47.731 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:35:47.731 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:47 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T20:35:47.732 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:35:47.732 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:47 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi038 to 92190k 2023-03-09T20:35:47.732 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:47 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T20:35:47.733 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:47 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi038 to 94402625: error parsing value: Value '94402625' is below minimum 939524096 2023-03-09T20:35:47.733 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20258.585465 secs 2023-03-09T20:35:47.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:35:47.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:35:47.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:35:47.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:35:47.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:35:47.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:35:47.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:47 smithi181 ceph-mon[94189]: pgmap v11315: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 366 KiB/s wr, 11 op/s 2023-03-09T20:35:47.736 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:35:47.736 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:35:47.736 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:35:47.736 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:35:47.736 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:35:47.737 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:35:47.737 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:35:47.737 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:35:47.737 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:35:47.737 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:47 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T20:35:47.737 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:35:47.738 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:47 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi038 to 92190k 2023-03-09T20:35:47.738 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:47 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T20:35:47.738 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:47 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi038 to 94402625: error parsing value: Value '94402625' is below minimum 939524096 2023-03-09T20:35:47.738 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20258.585465 secs 2023-03-09T20:35:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:35:47.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:35:47.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:35:47.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:35:47.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:35:47.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:35:47.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:47 smithi038 ceph-mon[92422]: pgmap v11315: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 366 KiB/s wr, 11 op/s 2023-03-09T20:35:47.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:35:47.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:35:47.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:35:47.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:35:47.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:35:47.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:35:47.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:35:47.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:35:47.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:35:47.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:47 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T20:35:47.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:35:47.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:47 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi038 to 92190k 2023-03-09T20:35:47.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:47 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T20:35:47.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:47 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi038 to 94402625: error parsing value: Value '94402625' is below minimum 939524096 2023-03-09T20:35:47.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20258.585465 secs 2023-03-09T20:35:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:35:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:35:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:35:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:35:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:48 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T20:35:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:48 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T20:35:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:35:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:35:48.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:35:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:35:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:35:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:35:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:35:48.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:48 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T20:35:48.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:48 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T20:35:48.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:35:48.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:35:48.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:35:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:35:48.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:35:48.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:35:48.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:35:48.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:48 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T20:35:48.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:48 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T20:35:48.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:35:48.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:35:48.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:35:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20720.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:35:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:49 smithi122 ceph-mon[93869]: pgmap v11316: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 382 KiB/s wr, 10 op/s 2023-03-09T20:35:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:49 smithi181 ceph-mon[94189]: pgmap v11316: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 382 KiB/s wr, 10 op/s 2023-03-09T20:35:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:49 smithi038 ceph-mon[92422]: pgmap v11316: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 382 KiB/s wr, 10 op/s 2023-03-09T20:35:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20261.252522 secs 2023-03-09T20:35:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20261.252522 secs 2023-03-09T20:35:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20261.252522 secs 2023-03-09T20:35:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:51 smithi122 ceph-mon[93869]: pgmap v11317: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 538 KiB/s wr, 11 op/s 2023-03-09T20:35:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:51 smithi181 ceph-mon[94189]: pgmap v11317: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 538 KiB/s wr, 11 op/s 2023-03-09T20:35:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:51 smithi038 ceph-mon[92422]: pgmap v11317: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 538 KiB/s wr, 11 op/s 2023-03-09T20:35:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20263.585601 secs 2023-03-09T20:35:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20263.585601 secs 2023-03-09T20:35:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20263.585601 secs 2023-03-09T20:35:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:53 smithi122 ceph-mon[93869]: pgmap v11318: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 443 KiB/s rd, 427 KiB/s wr, 12 op/s 2023-03-09T20:35:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:53 smithi181 ceph-mon[94189]: pgmap v11318: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 443 KiB/s rd, 427 KiB/s wr, 12 op/s 2023-03-09T20:35:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:53 smithi038 ceph-mon[92422]: pgmap v11318: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 443 KiB/s rd, 427 KiB/s wr, 12 op/s 2023-03-09T20:35:53.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20725.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:35:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:55 smithi122 ceph-mon[93869]: pgmap v11319: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 455 KiB/s rd, 441 KiB/s wr, 10 op/s 2023-03-09T20:35:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:55 smithi181 ceph-mon[94189]: pgmap v11319: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 455 KiB/s rd, 441 KiB/s wr, 10 op/s 2023-03-09T20:35:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:55 smithi038 ceph-mon[92422]: pgmap v11319: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 455 KiB/s rd, 441 KiB/s wr, 10 op/s 2023-03-09T20:35:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20266.252584 secs 2023-03-09T20:35:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:35:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:35:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20266.252584 secs 2023-03-09T20:35:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:35:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:35:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20266.252584 secs 2023-03-09T20:35:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:35:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:35:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:35:57.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:35:57.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:57 smithi122 ceph-mon[93869]: pgmap v11320: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 538 KiB/s rd, 521 KiB/s wr, 13 op/s 2023-03-09T20:35:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:35:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:35:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:57 smithi181 ceph-mon[94189]: pgmap v11320: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 538 KiB/s rd, 521 KiB/s wr, 13 op/s 2023-03-09T20:35:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:35:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:35:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:57 smithi038 ceph-mon[92422]: pgmap v11320: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 538 KiB/s rd, 521 KiB/s wr, 13 op/s 2023-03-09T20:35:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20268.585742 secs 2023-03-09T20:35:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20268.585742 secs 2023-03-09T20:35:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20268.585742 secs 2023-03-09T20:35:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20730.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:35:59.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:35:59 smithi122 ceph-mon[93869]: pgmap v11321: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 395 KiB/s wr, 10 op/s 2023-03-09T20:35:59.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:35:59 smithi181 ceph-mon[94189]: pgmap v11321: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 395 KiB/s wr, 10 op/s 2023-03-09T20:35:59.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:35:59 smithi038 ceph-mon[92422]: pgmap v11321: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 395 KiB/s wr, 10 op/s 2023-03-09T20:36:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20271.252646 secs 2023-03-09T20:36:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20271.252646 secs 2023-03-09T20:36:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20271.252646 secs 2023-03-09T20:36:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:01 smithi122 ceph-mon[93869]: pgmap v11322: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 379 KiB/s wr, 9 op/s 2023-03-09T20:36:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:01 smithi181 ceph-mon[94189]: pgmap v11322: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 379 KiB/s wr, 9 op/s 2023-03-09T20:36:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:01 smithi038 ceph-mon[92422]: pgmap v11322: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 379 KiB/s wr, 9 op/s 2023-03-09T20:36:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20273.585860 secs 2023-03-09T20:36:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20273.585860 secs 2023-03-09T20:36:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20273.585860 secs 2023-03-09T20:36:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:03 smithi122 ceph-mon[93869]: pgmap v11323: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 378 KiB/s wr, 12 op/s 2023-03-09T20:36:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:03 smithi181 ceph-mon[94189]: pgmap v11323: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 378 KiB/s wr, 12 op/s 2023-03-09T20:36:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:03 smithi038 ceph-mon[92422]: pgmap v11323: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 378 KiB/s wr, 12 op/s 2023-03-09T20:36:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20735.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:36:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:05 smithi122 ceph-mon[93869]: pgmap v11324: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 319 KiB/s wr, 23 op/s 2023-03-09T20:36:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:05 smithi181 ceph-mon[94189]: pgmap v11324: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 319 KiB/s wr, 23 op/s 2023-03-09T20:36:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:05 smithi038 ceph-mon[92422]: pgmap v11324: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 319 KiB/s wr, 23 op/s 2023-03-09T20:36:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20276.252708 secs 2023-03-09T20:36:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20276.252708 secs 2023-03-09T20:36:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20276.252708 secs 2023-03-09T20:36:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:07 smithi122 ceph-mon[93869]: pgmap v11325: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 318 KiB/s rd, 338 KiB/s wr, 25 op/s 2023-03-09T20:36:07.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:07 smithi181 ceph-mon[94189]: pgmap v11325: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 318 KiB/s rd, 338 KiB/s wr, 25 op/s 2023-03-09T20:36:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:07 smithi038 ceph-mon[92422]: pgmap v11325: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 318 KiB/s rd, 338 KiB/s wr, 25 op/s 2023-03-09T20:36:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20278.585983 secs 2023-03-09T20:36:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20278.585983 secs 2023-03-09T20:36:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20278.585983 secs 2023-03-09T20:36:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20740.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:36:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:09 smithi122 ceph-mon[93869]: pgmap v11326: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 350 KiB/s wr, 24 op/s 2023-03-09T20:36:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:09 smithi181 ceph-mon[94189]: pgmap v11326: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 350 KiB/s wr, 24 op/s 2023-03-09T20:36:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:09 smithi038 ceph-mon[92422]: pgmap v11326: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 350 KiB/s wr, 24 op/s 2023-03-09T20:36:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:10 smithi122 ceph-mon[93869]: osdmap e921: 12 total, 12 up, 12 in 2023-03-09T20:36:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:10 smithi181 ceph-mon[94189]: osdmap e921: 12 total, 12 up, 12 in 2023-03-09T20:36:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:10 smithi038 ceph-mon[92422]: osdmap e921: 12 total, 12 up, 12 in 2023-03-09T20:36:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20281.252768 secs 2023-03-09T20:36:11.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:11 smithi122 ceph-mon[93869]: pgmap v11328: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 382 KiB/s wr, 27 op/s 2023-03-09T20:36:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20281.252768 secs 2023-03-09T20:36:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:11 smithi181 ceph-mon[94189]: pgmap v11328: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 382 KiB/s wr, 27 op/s 2023-03-09T20:36:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20281.252768 secs 2023-03-09T20:36:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:11 smithi038 ceph-mon[92422]: pgmap v11328: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 382 KiB/s wr, 27 op/s 2023-03-09T20:36:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:36:12.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:36:12.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20283.586121 secs 2023-03-09T20:36:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:36:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:36:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20283.586121 secs 2023-03-09T20:36:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:36:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:36:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20283.586121 secs 2023-03-09T20:36:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:36:13.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:36:13.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:13 smithi122 ceph-mon[93869]: pgmap v11329: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 196 KiB/s wr, 25 op/s 2023-03-09T20:36:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:36:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:36:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:13 smithi181 ceph-mon[94189]: pgmap v11329: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 196 KiB/s wr, 25 op/s 2023-03-09T20:36:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:36:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:36:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:13 smithi038 ceph-mon[92422]: pgmap v11329: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 196 KiB/s wr, 25 op/s 2023-03-09T20:36:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20745.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:36:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:15 smithi122 ceph-mon[93869]: pgmap v11330: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 189 KiB/s wr, 10 op/s 2023-03-09T20:36:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:15 smithi181 ceph-mon[94189]: pgmap v11330: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 189 KiB/s wr, 10 op/s 2023-03-09T20:36:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:15 smithi038 ceph-mon[92422]: pgmap v11330: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 189 KiB/s wr, 10 op/s 2023-03-09T20:36:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20286.252828 secs 2023-03-09T20:36:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20286.252828 secs 2023-03-09T20:36:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20286.252828 secs 2023-03-09T20:36:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:17 smithi122 ceph-mon[93869]: pgmap v11331: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 147 KiB/s wr, 9 op/s 2023-03-09T20:36:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:17 smithi181 ceph-mon[94189]: pgmap v11331: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 147 KiB/s wr, 9 op/s 2023-03-09T20:36:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:17 smithi038 ceph-mon[92422]: pgmap v11331: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 147 KiB/s wr, 9 op/s 2023-03-09T20:36:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20288.586246 secs 2023-03-09T20:36:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20288.586246 secs 2023-03-09T20:36:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20288.586246 secs 2023-03-09T20:36:18.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20750.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:36:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:19 smithi122 ceph-mon[93869]: pgmap v11332: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 82 KiB/s wr, 9 op/s 2023-03-09T20:36:19.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:19 smithi122 ceph-mon[93869]: osdmap e922: 12 total, 12 up, 12 in 2023-03-09T20:36:19.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:19 smithi181 ceph-mon[94189]: pgmap v11332: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 82 KiB/s wr, 9 op/s 2023-03-09T20:36:19.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:19 smithi181 ceph-mon[94189]: osdmap e922: 12 total, 12 up, 12 in 2023-03-09T20:36:19.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:19 smithi038 ceph-mon[92422]: pgmap v11332: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 82 KiB/s wr, 9 op/s 2023-03-09T20:36:19.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:19 smithi038 ceph-mon[92422]: osdmap e922: 12 total, 12 up, 12 in 2023-03-09T20:36:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20291.252890 secs 2023-03-09T20:36:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20291.252890 secs 2023-03-09T20:36:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20291.252890 secs 2023-03-09T20:36:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:21 smithi122 ceph-mon[93869]: pgmap v11334: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 82 KiB/s wr, 9 op/s 2023-03-09T20:36:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:21 smithi181 ceph-mon[94189]: pgmap v11334: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 82 KiB/s wr, 9 op/s 2023-03-09T20:36:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:21 smithi038 ceph-mon[92422]: pgmap v11334: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 82 KiB/s wr, 9 op/s 2023-03-09T20:36:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20293.586354 secs 2023-03-09T20:36:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20293.586354 secs 2023-03-09T20:36:22.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20293.586354 secs 2023-03-09T20:36:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:23 smithi122 ceph-mon[93869]: pgmap v11335: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 82 KiB/s wr, 8 op/s 2023-03-09T20:36:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:23 smithi181 ceph-mon[94189]: pgmap v11335: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 82 KiB/s wr, 8 op/s 2023-03-09T20:36:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:23 smithi038 ceph-mon[92422]: pgmap v11335: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 82 KiB/s wr, 8 op/s 2023-03-09T20:36:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20755.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:36:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:25 smithi122 ceph-mon[93869]: pgmap v11336: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 63 KiB/s wr, 8 op/s 2023-03-09T20:36:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:25 smithi181 ceph-mon[94189]: pgmap v11336: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 63 KiB/s wr, 8 op/s 2023-03-09T20:36:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:25 smithi038 ceph-mon[92422]: pgmap v11336: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 63 KiB/s wr, 8 op/s 2023-03-09T20:36:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20296.252950 secs 2023-03-09T20:36:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20296.252950 secs 2023-03-09T20:36:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20296.252950 secs 2023-03-09T20:36:27.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:27 smithi122 ceph-mon[93869]: pgmap v11337: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 63 KiB/s wr, 7 op/s 2023-03-09T20:36:27.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:27 smithi181 ceph-mon[94189]: pgmap v11337: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 63 KiB/s wr, 7 op/s 2023-03-09T20:36:27.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:27 smithi038 ceph-mon[92422]: pgmap v11337: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 63 KiB/s wr, 7 op/s 2023-03-09T20:36:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20298.586477 secs 2023-03-09T20:36:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20298.586477 secs 2023-03-09T20:36:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20298.586477 secs 2023-03-09T20:36:28.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20760.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:36:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:29 smithi122 ceph-mon[93869]: pgmap v11338: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 93 KiB/s wr, 7 op/s 2023-03-09T20:36:29.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:29 smithi181 ceph-mon[94189]: pgmap v11338: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 93 KiB/s wr, 7 op/s 2023-03-09T20:36:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:29 smithi038 ceph-mon[92422]: pgmap v11338: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 93 KiB/s wr, 7 op/s 2023-03-09T20:36:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20301.252984 secs 2023-03-09T20:36:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20301.252984 secs 2023-03-09T20:36:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20301.252984 secs 2023-03-09T20:36:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:31 smithi122 ceph-mon[93869]: pgmap v11339: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 82 KiB/s wr, 6 op/s 2023-03-09T20:36:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:31 smithi181 ceph-mon[94189]: pgmap v11339: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 82 KiB/s wr, 6 op/s 2023-03-09T20:36:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:31 smithi038 ceph-mon[92422]: pgmap v11339: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 82 KiB/s wr, 6 op/s 2023-03-09T20:36:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20303.586546 secs 2023-03-09T20:36:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20303.586546 secs 2023-03-09T20:36:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20303.586546 secs 2023-03-09T20:36:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:33 smithi122 ceph-mon[93869]: pgmap v11340: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 78 KiB/s wr, 8 op/s 2023-03-09T20:36:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:33 smithi181 ceph-mon[94189]: pgmap v11340: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 78 KiB/s wr, 8 op/s 2023-03-09T20:36:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:33 smithi038 ceph-mon[92422]: pgmap v11340: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 78 KiB/s wr, 8 op/s 2023-03-09T20:36:33.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20765.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:36:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:35 smithi122 ceph-mon[93869]: pgmap v11341: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 78 KiB/s wr, 6 op/s 2023-03-09T20:36:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:35 smithi181 ceph-mon[94189]: pgmap v11341: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 78 KiB/s wr, 6 op/s 2023-03-09T20:36:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:35 smithi038 ceph-mon[92422]: pgmap v11341: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 78 KiB/s wr, 6 op/s 2023-03-09T20:36:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20306.253069 secs 2023-03-09T20:36:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20306.253069 secs 2023-03-09T20:36:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20306.253069 secs 2023-03-09T20:36:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:37 smithi122 ceph-mon[93869]: pgmap v11342: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 78 KiB/s wr, 7 op/s 2023-03-09T20:36:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:37 smithi181 ceph-mon[94189]: pgmap v11342: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 78 KiB/s wr, 7 op/s 2023-03-09T20:36:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:37 smithi038 ceph-mon[92422]: pgmap v11342: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 78 KiB/s wr, 7 op/s 2023-03-09T20:36:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20308.586669 secs 2023-03-09T20:36:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20308.586669 secs 2023-03-09T20:36:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20308.586669 secs 2023-03-09T20:36:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20770.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:36:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:39 smithi122 ceph-mon[93869]: pgmap v11343: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 201 KiB/s wr, 7 op/s 2023-03-09T20:36:39.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:39 smithi181 ceph-mon[94189]: pgmap v11343: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 201 KiB/s wr, 7 op/s 2023-03-09T20:36:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:39 smithi038 ceph-mon[92422]: pgmap v11343: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 201 KiB/s wr, 7 op/s 2023-03-09T20:36:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20311.253128 secs 2023-03-09T20:36:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:36:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:36:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:41 smithi122 ceph-mon[93869]: pgmap v11344: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 124 KiB/s wr, 6 op/s 2023-03-09T20:36:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20311.253128 secs 2023-03-09T20:36:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:36:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:36:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:41 smithi181 ceph-mon[94189]: pgmap v11344: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 124 KiB/s wr, 6 op/s 2023-03-09T20:36:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20311.253128 secs 2023-03-09T20:36:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:36:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:36:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:41 smithi038 ceph-mon[92422]: pgmap v11344: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 124 KiB/s wr, 6 op/s 2023-03-09T20:36:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20313.586784 secs 2023-03-09T20:36:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20313.586784 secs 2023-03-09T20:36:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20313.586784 secs 2023-03-09T20:36:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:43 smithi122 ceph-mon[93869]: pgmap v11345: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 161 KiB/s wr, 9 op/s 2023-03-09T20:36:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:43 smithi181 ceph-mon[94189]: pgmap v11345: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 161 KiB/s wr, 9 op/s 2023-03-09T20:36:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:43 smithi038 ceph-mon[92422]: pgmap v11345: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 161 KiB/s wr, 9 op/s 2023-03-09T20:36:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20775.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:36:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:45 smithi122 ceph-mon[93869]: pgmap v11346: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 161 KiB/s wr, 7 op/s 2023-03-09T20:36:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:45 smithi181 ceph-mon[94189]: pgmap v11346: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 161 KiB/s wr, 7 op/s 2023-03-09T20:36:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:45 smithi038 ceph-mon[92422]: pgmap v11346: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 161 KiB/s wr, 7 op/s 2023-03-09T20:36:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20316.253187 secs 2023-03-09T20:36:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20316.253187 secs 2023-03-09T20:36:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20316.253187 secs 2023-03-09T20:36:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:47 smithi122 ceph-mon[93869]: pgmap v11347: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 308 KiB/s wr, 9 op/s 2023-03-09T20:36:47.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:47 smithi181 ceph-mon[94189]: pgmap v11347: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 308 KiB/s wr, 9 op/s 2023-03-09T20:36:47.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:47 smithi038 ceph-mon[92422]: pgmap v11347: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 308 KiB/s wr, 9 op/s 2023-03-09T20:36:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:36:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:36:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:36:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:36:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:36:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:36:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:36:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:36:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:36:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20780.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:36:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:49 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:36:49.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:36:49.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:36:49.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:36:49.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:49 smithi122 ceph-mon[93869]: pgmap v11348: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 464 KiB/s rd, 324 KiB/s wr, 8 op/s 2023-03-09T20:36:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:49 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:36:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:36:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:36:49.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:36:49.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:49 smithi181 ceph-mon[94189]: pgmap v11348: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 464 KiB/s rd, 324 KiB/s wr, 8 op/s 2023-03-09T20:36:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:49 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:36:49.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:36:49.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:36:49.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:36:49.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:49 smithi038 ceph-mon[92422]: pgmap v11348: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 464 KiB/s rd, 324 KiB/s wr, 8 op/s 2023-03-09T20:36:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:50 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:36:50.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:50 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:36:50.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:50 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:36:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:50 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:36:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:50 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:36:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:50 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:36:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:50 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:36:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:50 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:36:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:50 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:36:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20321.253282 secs 2023-03-09T20:36:51.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:51 smithi122 ceph-mon[93869]: pgmap v11349: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 263 KiB/s wr, 7 op/s 2023-03-09T20:36:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20321.253282 secs 2023-03-09T20:36:51.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:51 smithi181 ceph-mon[94189]: pgmap v11349: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 263 KiB/s wr, 7 op/s 2023-03-09T20:36:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20321.253282 secs 2023-03-09T20:36:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:51 smithi038 ceph-mon[92422]: pgmap v11349: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 263 KiB/s wr, 7 op/s 2023-03-09T20:36:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20785.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:36:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:53 smithi122 ceph-mon[93869]: pgmap v11350: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 513 KiB/s rd, 421 KiB/s wr, 11 op/s 2023-03-09T20:36:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:53 smithi181 ceph-mon[94189]: pgmap v11350: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 513 KiB/s rd, 421 KiB/s wr, 11 op/s 2023-03-09T20:36:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:53 smithi038 ceph-mon[92422]: pgmap v11350: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 513 KiB/s rd, 421 KiB/s wr, 11 op/s 2023-03-09T20:36:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:55 smithi122 ceph-mon[93869]: pgmap v11351: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 537 KiB/s rd, 617 KiB/s wr, 10 op/s 2023-03-09T20:36:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:55 smithi181 ceph-mon[94189]: pgmap v11351: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 537 KiB/s rd, 617 KiB/s wr, 10 op/s 2023-03-09T20:36:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:55 smithi038 ceph-mon[92422]: pgmap v11351: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 537 KiB/s rd, 617 KiB/s wr, 10 op/s 2023-03-09T20:36:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20326.253339 secs 2023-03-09T20:36:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:36:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:36:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20326.253339 secs 2023-03-09T20:36:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:36:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:36:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20326.253339 secs 2023-03-09T20:36:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:36:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:36:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:36:57.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:36:57.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:57 smithi122 ceph-mon[93869]: pgmap v11352: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 538 KiB/s rd, 617 KiB/s wr, 12 op/s 2023-03-09T20:36:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:36:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:36:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:57 smithi181 ceph-mon[94189]: pgmap v11352: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 538 KiB/s rd, 617 KiB/s wr, 12 op/s 2023-03-09T20:36:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:36:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:36:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:57 smithi038 ceph-mon[92422]: pgmap v11352: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 538 KiB/s rd, 617 KiB/s wr, 12 op/s 2023-03-09T20:36:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20790.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:36:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:36:59 smithi122 ceph-mon[93869]: pgmap v11353: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 486 KiB/s wr, 11 op/s 2023-03-09T20:36:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:36:59 smithi181 ceph-mon[94189]: pgmap v11353: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 486 KiB/s wr, 11 op/s 2023-03-09T20:36:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:36:59 smithi038 ceph-mon[92422]: pgmap v11353: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 486 KiB/s wr, 11 op/s 2023-03-09T20:37:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20331.253399 secs 2023-03-09T20:37:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20331.253399 secs 2023-03-09T20:37:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20331.253399 secs 2023-03-09T20:37:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:01 smithi122 ceph-mon[93869]: pgmap v11354: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 521 KiB/s rd, 532 KiB/s wr, 11 op/s 2023-03-09T20:37:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:01 smithi181 ceph-mon[94189]: pgmap v11354: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 521 KiB/s rd, 532 KiB/s wr, 11 op/s 2023-03-09T20:37:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:01 smithi038 ceph-mon[92422]: pgmap v11354: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 521 KiB/s rd, 532 KiB/s wr, 11 op/s 2023-03-09T20:37:03.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20795.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:37:03.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:03 smithi122 ceph-mon[93869]: pgmap v11355: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 471 KiB/s wr, 13 op/s 2023-03-09T20:37:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:03 smithi181 ceph-mon[94189]: pgmap v11355: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 471 KiB/s wr, 13 op/s 2023-03-09T20:37:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:03 smithi038 ceph-mon[92422]: pgmap v11355: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 471 KiB/s wr, 13 op/s 2023-03-09T20:37:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:05 smithi122 ceph-mon[93869]: pgmap v11356: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 366 KiB/s wr, 23 op/s 2023-03-09T20:37:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:05 smithi181 ceph-mon[94189]: pgmap v11356: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 366 KiB/s wr, 23 op/s 2023-03-09T20:37:05.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:05 smithi038 ceph-mon[92422]: pgmap v11356: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 366 KiB/s wr, 23 op/s 2023-03-09T20:37:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20336.253472 secs 2023-03-09T20:37:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20336.253472 secs 2023-03-09T20:37:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20336.253472 secs 2023-03-09T20:37:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:07 smithi122 ceph-mon[93869]: pgmap v11357: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 410 KiB/s wr, 26 op/s 2023-03-09T20:37:07.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:07 smithi181 ceph-mon[94189]: pgmap v11357: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 410 KiB/s wr, 26 op/s 2023-03-09T20:37:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:07 smithi038 ceph-mon[92422]: pgmap v11357: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 410 KiB/s wr, 26 op/s 2023-03-09T20:37:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20338.587313 secs 2023-03-09T20:37:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:37:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:37:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20338.587313 secs 2023-03-09T20:37:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:37:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:37:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20338.587313 secs 2023-03-09T20:37:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:37:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:37:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20800.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:37:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:37:09.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:37:09.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:09 smithi122 ceph-mon[93869]: pgmap v11358: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 426 KiB/s wr, 25 op/s 2023-03-09T20:37:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:37:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:37:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:09 smithi181 ceph-mon[94189]: pgmap v11358: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 426 KiB/s wr, 25 op/s 2023-03-09T20:37:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:37:09.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:37:09.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:09 smithi038 ceph-mon[92422]: pgmap v11358: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 426 KiB/s wr, 25 op/s 2023-03-09T20:37:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:10 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:37:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:10 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:37:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:10 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:37:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:10 smithi122 ceph-mon[93869]: osdmap e923: 12 total, 12 up, 12 in 2023-03-09T20:37:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:10 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:37:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:10 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:37:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:10 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:37:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:10 smithi181 ceph-mon[94189]: osdmap e923: 12 total, 12 up, 12 in 2023-03-09T20:37:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:10 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:37:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:10 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:37:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:10 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:37:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:10 smithi038 ceph-mon[92422]: osdmap e923: 12 total, 12 up, 12 in 2023-03-09T20:37:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20341.253533 secs 2023-03-09T20:37:11.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:11 smithi122 ceph-mon[93869]: pgmap v11360: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 490 KiB/s rd, 481 KiB/s wr, 28 op/s 2023-03-09T20:37:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20341.253533 secs 2023-03-09T20:37:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:11 smithi181 ceph-mon[94189]: pgmap v11360: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 490 KiB/s rd, 481 KiB/s wr, 28 op/s 2023-03-09T20:37:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20341.253533 secs 2023-03-09T20:37:11.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:11 smithi038 ceph-mon[92422]: pgmap v11360: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 490 KiB/s rd, 481 KiB/s wr, 28 op/s 2023-03-09T20:37:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20343.587430 secs 2023-03-09T20:37:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20343.587430 secs 2023-03-09T20:37:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20343.587430 secs 2023-03-09T20:37:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:13 smithi122 ceph-mon[93869]: pgmap v11361: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 480 KiB/s wr, 28 op/s 2023-03-09T20:37:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:13 smithi181 ceph-mon[94189]: pgmap v11361: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 480 KiB/s wr, 28 op/s 2023-03-09T20:37:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:13 smithi038 ceph-mon[92422]: pgmap v11361: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 480 KiB/s wr, 28 op/s 2023-03-09T20:37:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20805.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:37:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:15 smithi122 ceph-mon[93869]: pgmap v11362: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 416 KiB/s wr, 12 op/s 2023-03-09T20:37:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:15 smithi181 ceph-mon[94189]: pgmap v11362: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 416 KiB/s wr, 12 op/s 2023-03-09T20:37:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:15 smithi038 ceph-mon[92422]: pgmap v11362: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 416 KiB/s wr, 12 op/s 2023-03-09T20:37:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20346.253593 secs 2023-03-09T20:37:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20346.253593 secs 2023-03-09T20:37:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20346.253593 secs 2023-03-09T20:37:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:17 smithi122 ceph-mon[93869]: pgmap v11363: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 83 KiB/s wr, 8 op/s 2023-03-09T20:37:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:17 smithi181 ceph-mon[94189]: pgmap v11363: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 83 KiB/s wr, 8 op/s 2023-03-09T20:37:17.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:17 smithi038 ceph-mon[92422]: pgmap v11363: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 83 KiB/s wr, 8 op/s 2023-03-09T20:37:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20348.587552 secs 2023-03-09T20:37:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20348.587552 secs 2023-03-09T20:37:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20348.587552 secs 2023-03-09T20:37:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20810.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:37:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:19 smithi122 ceph-mon[93869]: pgmap v11364: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 83 KiB/s wr, 7 op/s 2023-03-09T20:37:19.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:19 smithi122 ceph-mon[93869]: osdmap e924: 12 total, 12 up, 12 in 2023-03-09T20:37:19.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:19 smithi181 ceph-mon[94189]: pgmap v11364: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 83 KiB/s wr, 7 op/s 2023-03-09T20:37:19.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:19 smithi181 ceph-mon[94189]: osdmap e924: 12 total, 12 up, 12 in 2023-03-09T20:37:19.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:19 smithi038 ceph-mon[92422]: pgmap v11364: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 83 KiB/s wr, 7 op/s 2023-03-09T20:37:19.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:19 smithi038 ceph-mon[92422]: osdmap e924: 12 total, 12 up, 12 in 2023-03-09T20:37:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20351.253654 secs 2023-03-09T20:37:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20351.253654 secs 2023-03-09T20:37:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20351.253654 secs 2023-03-09T20:37:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:21 smithi122 ceph-mon[93869]: pgmap v11366: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T20:37:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:21 smithi181 ceph-mon[94189]: pgmap v11366: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T20:37:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:21 smithi038 ceph-mon[92422]: pgmap v11366: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T20:37:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20353.587666 secs 2023-03-09T20:37:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20353.587666 secs 2023-03-09T20:37:22.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20353.587666 secs 2023-03-09T20:37:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:23 smithi122 ceph-mon[93869]: pgmap v11367: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T20:37:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:23 smithi181 ceph-mon[94189]: pgmap v11367: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T20:37:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:23 smithi038 ceph-mon[92422]: pgmap v11367: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T20:37:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20815.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:37:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:25 smithi122 ceph-mon[93869]: pgmap v11368: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T20:37:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:25 smithi181 ceph-mon[94189]: pgmap v11368: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T20:37:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:25 smithi038 ceph-mon[92422]: pgmap v11368: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T20:37:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20356.253715 secs 2023-03-09T20:37:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20356.253715 secs 2023-03-09T20:37:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20356.253715 secs 2023-03-09T20:37:27.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:27 smithi122 ceph-mon[93869]: pgmap v11369: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T20:37:27.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:27 smithi181 ceph-mon[94189]: pgmap v11369: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T20:37:27.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:27 smithi038 ceph-mon[92422]: pgmap v11369: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T20:37:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20358.587790 secs 2023-03-09T20:37:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20358.587790 secs 2023-03-09T20:37:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20358.587790 secs 2023-03-09T20:37:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20820.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:37:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:29 smithi122 ceph-mon[93869]: pgmap v11370: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 54 KiB/s wr, 7 op/s 2023-03-09T20:37:29.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:29 smithi181 ceph-mon[94189]: pgmap v11370: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 54 KiB/s wr, 7 op/s 2023-03-09T20:37:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:29 smithi038 ceph-mon[92422]: pgmap v11370: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 54 KiB/s wr, 7 op/s 2023-03-09T20:37:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20361.253775 secs 2023-03-09T20:37:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20361.253775 secs 2023-03-09T20:37:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20361.253775 secs 2023-03-09T20:37:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:31 smithi122 ceph-mon[93869]: pgmap v11371: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 83 KiB/s wr, 7 op/s 2023-03-09T20:37:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:31 smithi181 ceph-mon[94189]: pgmap v11371: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 83 KiB/s wr, 7 op/s 2023-03-09T20:37:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:31 smithi038 ceph-mon[92422]: pgmap v11371: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 83 KiB/s wr, 7 op/s 2023-03-09T20:37:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20363.587904 secs 2023-03-09T20:37:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20363.587904 secs 2023-03-09T20:37:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20363.587904 secs 2023-03-09T20:37:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:33 smithi122 ceph-mon[93869]: pgmap v11372: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 79 KiB/s wr, 8 op/s 2023-03-09T20:37:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:33 smithi181 ceph-mon[94189]: pgmap v11372: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 79 KiB/s wr, 8 op/s 2023-03-09T20:37:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:33 smithi038 ceph-mon[92422]: pgmap v11372: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 79 KiB/s wr, 8 op/s 2023-03-09T20:37:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20825.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:37:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:35 smithi122 ceph-mon[93869]: pgmap v11373: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 79 KiB/s wr, 7 op/s 2023-03-09T20:37:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:35 smithi181 ceph-mon[94189]: pgmap v11373: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 79 KiB/s wr, 7 op/s 2023-03-09T20:37:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:35 smithi038 ceph-mon[92422]: pgmap v11373: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 79 KiB/s wr, 7 op/s 2023-03-09T20:37:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20366.253859 secs 2023-03-09T20:37:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20366.253859 secs 2023-03-09T20:37:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20366.253859 secs 2023-03-09T20:37:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:37 smithi122 ceph-mon[93869]: pgmap v11374: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 79 KiB/s wr, 8 op/s 2023-03-09T20:37:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:37 smithi181 ceph-mon[94189]: pgmap v11374: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 79 KiB/s wr, 8 op/s 2023-03-09T20:37:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:37 smithi038 ceph-mon[92422]: pgmap v11374: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 79 KiB/s wr, 8 op/s 2023-03-09T20:37:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20830.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:37:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:39 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:37:39.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:39 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:37:39.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:39 smithi122 ceph-mon[93869]: pgmap v11375: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 240 KiB/s wr, 8 op/s 2023-03-09T20:37:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:39 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:37:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:39 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:37:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:39 smithi181 ceph-mon[94189]: pgmap v11375: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 240 KiB/s wr, 8 op/s 2023-03-09T20:37:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:39 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:37:39.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:39 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:37:39.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:39 smithi038 ceph-mon[92422]: pgmap v11375: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 240 KiB/s wr, 8 op/s 2023-03-09T20:37:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20371.253938 secs 2023-03-09T20:37:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:40 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:37:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20371.253938 secs 2023-03-09T20:37:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:40 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:37:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20371.253938 secs 2023-03-09T20:37:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:40 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:37:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:41 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:37:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:37:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:37:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:37:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:41 smithi122 ceph-mon[93869]: pgmap v11376: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 196 KiB/s wr, 7 op/s 2023-03-09T20:37:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:41 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:37:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:37:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:37:41.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:37:41.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:41 smithi181 ceph-mon[94189]: pgmap v11376: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 196 KiB/s wr, 7 op/s 2023-03-09T20:37:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:41 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:37:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:37:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:37:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:37:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:41 smithi038 ceph-mon[92422]: pgmap v11376: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 196 KiB/s wr, 7 op/s 2023-03-09T20:37:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20835.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:37:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:43 smithi122 ceph-mon[93869]: pgmap v11377: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 162 KiB/s wr, 9 op/s 2023-03-09T20:37:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:43 smithi181 ceph-mon[94189]: pgmap v11377: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 162 KiB/s wr, 9 op/s 2023-03-09T20:37:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:43 smithi038 ceph-mon[92422]: pgmap v11377: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 162 KiB/s wr, 9 op/s 2023-03-09T20:37:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:45 smithi122 ceph-mon[93869]: pgmap v11378: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 231 KiB/s wr, 9 op/s 2023-03-09T20:37:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:45 smithi181 ceph-mon[94189]: pgmap v11378: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 231 KiB/s wr, 9 op/s 2023-03-09T20:37:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:45 smithi038 ceph-mon[92422]: pgmap v11378: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 231 KiB/s wr, 9 op/s 2023-03-09T20:37:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20376.253983 secs 2023-03-09T20:37:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20376.253983 secs 2023-03-09T20:37:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20376.253983 secs 2023-03-09T20:37:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:47 smithi122 ceph-mon[93869]: pgmap v11379: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 231 KiB/s wr, 10 op/s 2023-03-09T20:37:47.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:47 smithi181 ceph-mon[94189]: pgmap v11379: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 231 KiB/s wr, 10 op/s 2023-03-09T20:37:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:47 smithi038 ceph-mon[92422]: pgmap v11379: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 231 KiB/s wr, 10 op/s 2023-03-09T20:37:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:48 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 20378.588285 secs 2023-03-09T20:37:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:48 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 20378.588285 secs 2023-03-09T20:37:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:48 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 20378.588285 secs 2023-03-09T20:37:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20840.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:37:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:37:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:49 smithi181 ceph-mon[94189]: pgmap v11380: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 335 KiB/s rd, 363 KiB/s wr, 10 op/s 2023-03-09T20:37:49.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:37:49.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:37:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:37:49.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:49 smithi038 ceph-mon[92422]: pgmap v11380: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 335 KiB/s rd, 363 KiB/s wr, 10 op/s 2023-03-09T20:37:49.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:37:49.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:37:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:37:49.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:49 smithi122 ceph-mon[93869]: pgmap v11380: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 335 KiB/s rd, 363 KiB/s wr, 10 op/s 2023-03-09T20:37:49.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:37:49.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:37:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:50 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:37:50.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:50 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:37:50.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:50 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:37:50.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:50 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:37:50.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:50 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:37:50.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:50 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:37:50.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20381.254045 secs 2023-03-09T20:37:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:50 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:37:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:50 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:37:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:50 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:37:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:50 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:37:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:50 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:37:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:50 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:37:50.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20381.254045 secs 2023-03-09T20:37:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:50 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:37:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:50 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:37:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:50 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:37:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:50 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:37:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:50 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:37:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:50 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:37:50.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20381.254045 secs 2023-03-09T20:37:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:51 smithi038 ceph-mon[92422]: pgmap v11381: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 356 KiB/s wr, 10 op/s 2023-03-09T20:37:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:51 smithi122 ceph-mon[93869]: pgmap v11381: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 356 KiB/s wr, 10 op/s 2023-03-09T20:37:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:51 smithi181 ceph-mon[94189]: pgmap v11381: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 356 KiB/s wr, 10 op/s 2023-03-09T20:37:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:37:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:37:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:52 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 20383.588420 secs 2023-03-09T20:37:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:37:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:37:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:52 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 20383.588420 secs 2023-03-09T20:37:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:37:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:37:52.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:52 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 20383.588420 secs 2023-03-09T20:37:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:37:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:37:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:53 smithi122 ceph-mon[93869]: pgmap v11382: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 356 KiB/s wr, 11 op/s 2023-03-09T20:37:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:37:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:37:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:53 smithi181 ceph-mon[94189]: pgmap v11382: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 356 KiB/s wr, 11 op/s 2023-03-09T20:37:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:37:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:37:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:53 smithi038 ceph-mon[92422]: pgmap v11382: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 356 KiB/s wr, 11 op/s 2023-03-09T20:37:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20845.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:37:55.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:55 smithi122 ceph-mon[93869]: pgmap v11383: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 528 KiB/s wr, 12 op/s 2023-03-09T20:37:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:55 smithi181 ceph-mon[94189]: pgmap v11383: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 528 KiB/s wr, 12 op/s 2023-03-09T20:37:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:55 smithi038 ceph-mon[92422]: pgmap v11383: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 528 KiB/s wr, 12 op/s 2023-03-09T20:37:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20386.275900 secs 2023-03-09T20:37:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20386.275900 secs 2023-03-09T20:37:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20386.275900 secs 2023-03-09T20:37:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:57 smithi122 ceph-mon[93869]: pgmap v11384: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 459 KiB/s wr, 12 op/s 2023-03-09T20:37:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:57 smithi181 ceph-mon[94189]: pgmap v11384: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 459 KiB/s wr, 12 op/s 2023-03-09T20:37:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:57 smithi038 ceph-mon[92422]: pgmap v11384: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 459 KiB/s wr, 12 op/s 2023-03-09T20:37:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:58 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 20388.588537 secs 2023-03-09T20:37:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:58 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 20388.588537 secs 2023-03-09T20:37:58.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:58 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 20388.588537 secs 2023-03-09T20:37:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20850.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:37:59.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:37:59 smithi122 ceph-mon[93869]: pgmap v11385: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 530 KiB/s wr, 12 op/s 2023-03-09T20:37:59.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:37:59 smithi181 ceph-mon[94189]: pgmap v11385: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 530 KiB/s wr, 12 op/s 2023-03-09T20:37:59.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:37:59 smithi038 ceph-mon[92422]: pgmap v11385: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 530 KiB/s wr, 12 op/s 2023-03-09T20:38:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:38:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:38:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:38:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:38:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:38:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:38:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20391.275986 secs 2023-03-09T20:38:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:38:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:38:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:01 smithi181 ceph-mon[94189]: pgmap v11386: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 443 KiB/s rd, 398 KiB/s wr, 10 op/s 2023-03-09T20:38:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20391.275986 secs 2023-03-09T20:38:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:38:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:38:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:01 smithi038 ceph-mon[92422]: pgmap v11386: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 443 KiB/s rd, 398 KiB/s wr, 10 op/s 2023-03-09T20:38:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20391.275986 secs 2023-03-09T20:38:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:38:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:38:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:01 smithi122 ceph-mon[93869]: pgmap v11386: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 443 KiB/s rd, 398 KiB/s wr, 10 op/s 2023-03-09T20:38:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20393.588627 secs 2023-03-09T20:38:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20393.588627 secs 2023-03-09T20:38:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20393.588627 secs 2023-03-09T20:38:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:03 smithi038 ceph-mon[92422]: pgmap v11387: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 516 KiB/s wr, 14 op/s 2023-03-09T20:38:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:03 smithi122 ceph-mon[93869]: pgmap v11387: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 516 KiB/s wr, 14 op/s 2023-03-09T20:38:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:03 smithi181 ceph-mon[94189]: pgmap v11387: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 516 KiB/s wr, 14 op/s 2023-03-09T20:38:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20855.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:38:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:05 smithi122 ceph-mon[93869]: pgmap v11388: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 540 KiB/s rd, 553 KiB/s wr, 25 op/s 2023-03-09T20:38:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:05 smithi181 ceph-mon[94189]: pgmap v11388: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 540 KiB/s rd, 553 KiB/s wr, 25 op/s 2023-03-09T20:38:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:05 smithi038 ceph-mon[92422]: pgmap v11388: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 540 KiB/s rd, 553 KiB/s wr, 25 op/s 2023-03-09T20:38:06.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20396.276044 secs 2023-03-09T20:38:06.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:06 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:38:06.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:38:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20396.276044 secs 2023-03-09T20:38:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:06 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:38:06.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:38:06.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20396.276044 secs 2023-03-09T20:38:06.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:06 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:38:06.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:38:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:07 smithi122 ceph-mon[93869]: pgmap v11389: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 541 KiB/s rd, 419 KiB/s wr, 26 op/s 2023-03-09T20:38:07.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:07 smithi181 ceph-mon[94189]: pgmap v11389: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 541 KiB/s rd, 419 KiB/s wr, 26 op/s 2023-03-09T20:38:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:07 smithi038 ceph-mon[92422]: pgmap v11389: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 541 KiB/s rd, 419 KiB/s wr, 26 op/s 2023-03-09T20:38:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20398.588745 secs 2023-03-09T20:38:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20398.588745 secs 2023-03-09T20:38:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20398.588745 secs 2023-03-09T20:38:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20860.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:38:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:09 smithi181 ceph-mon[94189]: pgmap v11390: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 592 KiB/s rd, 472 KiB/s wr, 25 op/s 2023-03-09T20:38:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:09 smithi038 ceph-mon[92422]: pgmap v11390: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 592 KiB/s rd, 472 KiB/s wr, 25 op/s 2023-03-09T20:38:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:09 smithi122 ceph-mon[93869]: pgmap v11390: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 592 KiB/s rd, 472 KiB/s wr, 25 op/s 2023-03-09T20:38:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:10 smithi122 ceph-mon[93869]: osdmap e925: 12 total, 12 up, 12 in 2023-03-09T20:38:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20401.276104 secs 2023-03-09T20:38:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:10 smithi181 ceph-mon[94189]: osdmap e925: 12 total, 12 up, 12 in 2023-03-09T20:38:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20401.276104 secs 2023-03-09T20:38:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:10 smithi038 ceph-mon[92422]: osdmap e925: 12 total, 12 up, 12 in 2023-03-09T20:38:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20401.276104 secs 2023-03-09T20:38:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:11 smithi038 ceph-mon[92422]: pgmap v11392: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 440 KiB/s rd, 480 KiB/s wr, 27 op/s 2023-03-09T20:38:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:11 smithi122 ceph-mon[93869]: pgmap v11392: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 440 KiB/s rd, 480 KiB/s wr, 27 op/s 2023-03-09T20:38:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:11 smithi181 ceph-mon[94189]: pgmap v11392: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 440 KiB/s rd, 480 KiB/s wr, 27 op/s 2023-03-09T20:38:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:38:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:38:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20403.588861 secs 2023-03-09T20:38:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:38:12.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:38:12.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20403.588861 secs 2023-03-09T20:38:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:38:12.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:38:12.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20403.588861 secs 2023-03-09T20:38:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:38:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:38:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:13 smithi038 ceph-mon[92422]: pgmap v11393: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 277 KiB/s wr, 24 op/s 2023-03-09T20:38:13.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:38:13.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:38:13.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:13 smithi122 ceph-mon[93869]: pgmap v11393: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 277 KiB/s wr, 24 op/s 2023-03-09T20:38:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:38:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:38:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:13 smithi181 ceph-mon[94189]: pgmap v11393: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 277 KiB/s wr, 24 op/s 2023-03-09T20:38:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20865.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:38:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:15 smithi122 ceph-mon[93869]: pgmap v11394: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 251 KiB/s wr, 9 op/s 2023-03-09T20:38:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:15 smithi181 ceph-mon[94189]: pgmap v11394: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 251 KiB/s wr, 9 op/s 2023-03-09T20:38:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:15 smithi038 ceph-mon[92422]: pgmap v11394: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 251 KiB/s wr, 9 op/s 2023-03-09T20:38:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20406.276181 secs 2023-03-09T20:38:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20406.276181 secs 2023-03-09T20:38:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20406.276181 secs 2023-03-09T20:38:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:17 smithi122 ceph-mon[93869]: pgmap v11395: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 270 KiB/s wr, 10 op/s 2023-03-09T20:38:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:17 smithi181 ceph-mon[94189]: pgmap v11395: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 270 KiB/s wr, 10 op/s 2023-03-09T20:38:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:17 smithi038 ceph-mon[92422]: pgmap v11395: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 270 KiB/s wr, 10 op/s 2023-03-09T20:38:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20408.588972 secs 2023-03-09T20:38:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20408.588972 secs 2023-03-09T20:38:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20408.588972 secs 2023-03-09T20:38:18.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20870.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:38:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:19 smithi122 ceph-mon[93869]: pgmap v11396: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 207 KiB/s wr, 9 op/s 2023-03-09T20:38:19.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:19 smithi122 ceph-mon[93869]: osdmap e926: 12 total, 12 up, 12 in 2023-03-09T20:38:19.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:19 smithi181 ceph-mon[94189]: pgmap v11396: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 207 KiB/s wr, 9 op/s 2023-03-09T20:38:19.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:19 smithi181 ceph-mon[94189]: osdmap e926: 12 total, 12 up, 12 in 2023-03-09T20:38:19.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:19 smithi038 ceph-mon[92422]: pgmap v11396: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 207 KiB/s wr, 9 op/s 2023-03-09T20:38:19.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:19 smithi038 ceph-mon[92422]: osdmap e926: 12 total, 12 up, 12 in 2023-03-09T20:38:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20411.276241 secs 2023-03-09T20:38:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20411.276241 secs 2023-03-09T20:38:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20411.276241 secs 2023-03-09T20:38:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:21 smithi122 ceph-mon[93869]: pgmap v11398: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 207 KiB/s wr, 9 op/s 2023-03-09T20:38:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:21 smithi181 ceph-mon[94189]: pgmap v11398: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 207 KiB/s wr, 9 op/s 2023-03-09T20:38:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:21 smithi038 ceph-mon[92422]: pgmap v11398: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 207 KiB/s wr, 9 op/s 2023-03-09T20:38:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20413.589087 secs 2023-03-09T20:38:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20413.589087 secs 2023-03-09T20:38:22.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20413.589087 secs 2023-03-09T20:38:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:23 smithi122 ceph-mon[93869]: pgmap v11399: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 85 KiB/s wr, 8 op/s 2023-03-09T20:38:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:23 smithi181 ceph-mon[94189]: pgmap v11399: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 85 KiB/s wr, 8 op/s 2023-03-09T20:38:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:23 smithi038 ceph-mon[92422]: pgmap v11399: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 85 KiB/s wr, 8 op/s 2023-03-09T20:38:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20875.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:38:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:25 smithi122 ceph-mon[93869]: pgmap v11400: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 65 KiB/s wr, 8 op/s 2023-03-09T20:38:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:25 smithi181 ceph-mon[94189]: pgmap v11400: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 65 KiB/s wr, 8 op/s 2023-03-09T20:38:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:25 smithi038 ceph-mon[92422]: pgmap v11400: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 65 KiB/s wr, 8 op/s 2023-03-09T20:38:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20416.276299 secs 2023-03-09T20:38:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20416.276299 secs 2023-03-09T20:38:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20416.276299 secs 2023-03-09T20:38:27.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:27 smithi181 ceph-mon[94189]: pgmap v11401: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 146 KiB/s wr, 7 op/s 2023-03-09T20:38:27.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20418.589224 secs 2023-03-09T20:38:27.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:27 smithi038 ceph-mon[92422]: pgmap v11401: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 146 KiB/s wr, 7 op/s 2023-03-09T20:38:27.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20418.589224 secs 2023-03-09T20:38:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:27 smithi122 ceph-mon[93869]: pgmap v11401: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 146 KiB/s wr, 7 op/s 2023-03-09T20:38:27.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20418.589224 secs 2023-03-09T20:38:28.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20880.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:38:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:29 smithi122 ceph-mon[93869]: pgmap v11402: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 145 KiB/s wr, 7 op/s 2023-03-09T20:38:29.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:29 smithi181 ceph-mon[94189]: pgmap v11402: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 145 KiB/s wr, 7 op/s 2023-03-09T20:38:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:29 smithi038 ceph-mon[92422]: pgmap v11402: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 145 KiB/s wr, 7 op/s 2023-03-09T20:38:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20421.276359 secs 2023-03-09T20:38:30.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20421.276359 secs 2023-03-09T20:38:30.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20421.276359 secs 2023-03-09T20:38:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:31 smithi122 ceph-mon[93869]: pgmap v11403: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 128 KiB/s wr, 6 op/s 2023-03-09T20:38:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:31 smithi181 ceph-mon[94189]: pgmap v11403: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 128 KiB/s wr, 6 op/s 2023-03-09T20:38:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:31 smithi038 ceph-mon[92422]: pgmap v11403: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 128 KiB/s wr, 6 op/s 2023-03-09T20:38:32.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20423.589335 secs 2023-03-09T20:38:32.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20423.589335 secs 2023-03-09T20:38:32.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20423.589335 secs 2023-03-09T20:38:33.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20885.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:38:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:33 smithi122 ceph-mon[93869]: pgmap v11404: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 122 KiB/s wr, 8 op/s 2023-03-09T20:38:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:33 smithi181 ceph-mon[94189]: pgmap v11404: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 122 KiB/s wr, 8 op/s 2023-03-09T20:38:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:33 smithi038 ceph-mon[92422]: pgmap v11404: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 122 KiB/s wr, 8 op/s 2023-03-09T20:38:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:35 smithi122 ceph-mon[93869]: pgmap v11405: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 121 KiB/s wr, 7 op/s 2023-03-09T20:38:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:35 smithi181 ceph-mon[94189]: pgmap v11405: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 121 KiB/s wr, 7 op/s 2023-03-09T20:38:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:35 smithi038 ceph-mon[92422]: pgmap v11405: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 121 KiB/s wr, 7 op/s 2023-03-09T20:38:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20426.276420 secs 2023-03-09T20:38:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20426.276420 secs 2023-03-09T20:38:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20426.276420 secs 2023-03-09T20:38:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:37 smithi122 ceph-mon[93869]: pgmap v11406: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 194 KiB/s wr, 9 op/s 2023-03-09T20:38:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:37 smithi181 ceph-mon[94189]: pgmap v11406: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 194 KiB/s wr, 9 op/s 2023-03-09T20:38:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:37 smithi038 ceph-mon[92422]: pgmap v11406: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 194 KiB/s wr, 9 op/s 2023-03-09T20:38:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20428.589474 secs 2023-03-09T20:38:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20428.589474 secs 2023-03-09T20:38:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20428.589474 secs 2023-03-09T20:38:38.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20890.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:38:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:39 smithi122 ceph-mon[93869]: pgmap v11407: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 75 KiB/s wr, 7 op/s 2023-03-09T20:38:39.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:39 smithi181 ceph-mon[94189]: pgmap v11407: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 75 KiB/s wr, 7 op/s 2023-03-09T20:38:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:39 smithi038 ceph-mon[92422]: pgmap v11407: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 75 KiB/s wr, 7 op/s 2023-03-09T20:38:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20431.276488 secs 2023-03-09T20:38:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:38:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:38:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20431.276488 secs 2023-03-09T20:38:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:38:40.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:38:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20431.276488 secs 2023-03-09T20:38:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:38:40.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:38:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:41 smithi122 ceph-mon[93869]: pgmap v11408: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 74 KiB/s wr, 6 op/s 2023-03-09T20:38:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:41 smithi181 ceph-mon[94189]: pgmap v11408: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 74 KiB/s wr, 6 op/s 2023-03-09T20:38:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:41 smithi038 ceph-mon[92422]: pgmap v11408: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 74 KiB/s wr, 6 op/s 2023-03-09T20:38:42.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20433.589537 secs 2023-03-09T20:38:42.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20433.589537 secs 2023-03-09T20:38:42.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20433.589537 secs 2023-03-09T20:38:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:43 smithi122 ceph-mon[93869]: pgmap v11409: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 96 KiB/s wr, 9 op/s 2023-03-09T20:38:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:43 smithi181 ceph-mon[94189]: pgmap v11409: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 96 KiB/s wr, 9 op/s 2023-03-09T20:38:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:43 smithi038 ceph-mon[92422]: pgmap v11409: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 96 KiB/s wr, 9 op/s 2023-03-09T20:38:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20895.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:38:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:45 smithi122 ceph-mon[93869]: pgmap v11410: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 105 KiB/s wr, 7 op/s 2023-03-09T20:38:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:45 smithi181 ceph-mon[94189]: pgmap v11410: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 105 KiB/s wr, 7 op/s 2023-03-09T20:38:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:45 smithi038 ceph-mon[92422]: pgmap v11410: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 105 KiB/s wr, 7 op/s 2023-03-09T20:38:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20436.276547 secs 2023-03-09T20:38:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20436.276547 secs 2023-03-09T20:38:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20436.276547 secs 2023-03-09T20:38:47.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:47 smithi181 ceph-mon[94189]: pgmap v11411: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 404 KiB/s wr, 11 op/s 2023-03-09T20:38:47.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20438.589658 secs 2023-03-09T20:38:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:47 smithi038 ceph-mon[92422]: pgmap v11411: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 404 KiB/s wr, 11 op/s 2023-03-09T20:38:47.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20438.589658 secs 2023-03-09T20:38:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:47 smithi122 ceph-mon[93869]: pgmap v11411: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 404 KiB/s wr, 11 op/s 2023-03-09T20:38:47.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20438.589658 secs 2023-03-09T20:38:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:38:48.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:38:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:38:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:38:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:38:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:38:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20900.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:38:49.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:49 smithi038 ceph-mon[92422]: pgmap v11412: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 330 KiB/s wr, 9 op/s 2023-03-09T20:38:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:49 smithi122 ceph-mon[93869]: pgmap v11412: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 330 KiB/s wr, 9 op/s 2023-03-09T20:38:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:49 smithi181 ceph-mon[94189]: pgmap v11412: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 330 KiB/s wr, 9 op/s 2023-03-09T20:38:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20441.276607 secs 2023-03-09T20:38:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:50 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:38:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20441.276607 secs 2023-03-09T20:38:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:50 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:38:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20441.276607 secs 2023-03-09T20:38:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:50 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:38:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:38:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:38:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:38:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:38:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:51 smithi122 ceph-mon[93869]: pgmap v11413: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 330 KiB/s wr, 9 op/s 2023-03-09T20:38:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:38:51.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:38:51.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:38:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:38:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:38:51.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:38:51.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:38:51.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:51 smithi181 ceph-mon[94189]: pgmap v11413: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 330 KiB/s wr, 9 op/s 2023-03-09T20:38:51.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:38:51.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:38:51.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:38:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:38:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:38:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:38:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:38:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:51 smithi038 ceph-mon[92422]: pgmap v11413: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 330 KiB/s wr, 9 op/s 2023-03-09T20:38:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:38:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:38:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:38:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20443.589772 secs 2023-03-09T20:38:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20443.589772 secs 2023-03-09T20:38:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20443.589772 secs 2023-03-09T20:38:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:53 smithi122 ceph-mon[93869]: pgmap v11414: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 464 KiB/s wr, 14 op/s 2023-03-09T20:38:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:53 smithi181 ceph-mon[94189]: pgmap v11414: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 464 KiB/s wr, 14 op/s 2023-03-09T20:38:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:53 smithi038 ceph-mon[92422]: pgmap v11414: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 464 KiB/s wr, 14 op/s 2023-03-09T20:38:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20905.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:38:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:55 smithi122 ceph-mon[93869]: pgmap v11415: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 443 KiB/s wr, 12 op/s 2023-03-09T20:38:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:55 smithi181 ceph-mon[94189]: pgmap v11415: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 443 KiB/s wr, 12 op/s 2023-03-09T20:38:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:55 smithi038 ceph-mon[92422]: pgmap v11415: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 443 KiB/s wr, 12 op/s 2023-03-09T20:38:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20446.276669 secs 2023-03-09T20:38:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:38:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:38:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20446.276669 secs 2023-03-09T20:38:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:38:56.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:38:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20446.276669 secs 2023-03-09T20:38:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:38:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:38:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:38:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:38:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:57 smithi038 ceph-mon[92422]: pgmap v11416: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 673 KiB/s wr, 15 op/s 2023-03-09T20:38:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20448.589907 secs 2023-03-09T20:38:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:38:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:38:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:57 smithi122 ceph-mon[93869]: pgmap v11416: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 673 KiB/s wr, 15 op/s 2023-03-09T20:38:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20448.589907 secs 2023-03-09T20:38:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:38:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:38:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:57 smithi181 ceph-mon[94189]: pgmap v11416: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 673 KiB/s wr, 15 op/s 2023-03-09T20:38:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20448.589907 secs 2023-03-09T20:38:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20910.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:38:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:38:59 smithi122 ceph-mon[93869]: pgmap v11417: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 430 KiB/s rd, 375 KiB/s wr, 12 op/s 2023-03-09T20:38:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:38:59 smithi181 ceph-mon[94189]: pgmap v11417: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 430 KiB/s rd, 375 KiB/s wr, 12 op/s 2023-03-09T20:38:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:38:59 smithi038 ceph-mon[92422]: pgmap v11417: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 430 KiB/s rd, 375 KiB/s wr, 12 op/s 2023-03-09T20:39:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20451.276731 secs 2023-03-09T20:39:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:39:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:39:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:01 smithi122 ceph-mon[93869]: pgmap v11418: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 430 KiB/s rd, 375 KiB/s wr, 11 op/s 2023-03-09T20:39:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20451.276731 secs 2023-03-09T20:39:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:39:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:39:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:01 smithi181 ceph-mon[94189]: pgmap v11418: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 430 KiB/s rd, 375 KiB/s wr, 11 op/s 2023-03-09T20:39:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20451.276731 secs 2023-03-09T20:39:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:39:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:39:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:01 smithi038 ceph-mon[92422]: pgmap v11418: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 430 KiB/s rd, 375 KiB/s wr, 11 op/s 2023-03-09T20:39:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20453.590048 secs 2023-03-09T20:39:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20453.590048 secs 2023-03-09T20:39:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20453.590048 secs 2023-03-09T20:39:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:03 smithi038 ceph-mon[92422]: pgmap v11419: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 734 KiB/s rd, 625 KiB/s wr, 17 op/s 2023-03-09T20:39:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:03 smithi122 ceph-mon[93869]: pgmap v11419: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 734 KiB/s rd, 625 KiB/s wr, 17 op/s 2023-03-09T20:39:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:03 smithi181 ceph-mon[94189]: pgmap v11419: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 734 KiB/s rd, 625 KiB/s wr, 17 op/s 2023-03-09T20:39:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20915.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:39:05.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:05 smithi038 ceph-mon[92422]: osdmap e927: 12 total, 12 up, 12 in 2023-03-09T20:39:05.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:05 smithi038 ceph-mon[92422]: pgmap v11421: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 815 KiB/s rd, 614 KiB/s wr, 29 op/s 2023-03-09T20:39:05.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20456.276793 secs 2023-03-09T20:39:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:05 smithi122 ceph-mon[93869]: osdmap e927: 12 total, 12 up, 12 in 2023-03-09T20:39:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:05 smithi122 ceph-mon[93869]: pgmap v11421: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 815 KiB/s rd, 614 KiB/s wr, 29 op/s 2023-03-09T20:39:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20456.276793 secs 2023-03-09T20:39:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:05 smithi181 ceph-mon[94189]: osdmap e927: 12 total, 12 up, 12 in 2023-03-09T20:39:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:05 smithi181 ceph-mon[94189]: pgmap v11421: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 815 KiB/s rd, 614 KiB/s wr, 29 op/s 2023-03-09T20:39:05.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20456.276793 secs 2023-03-09T20:39:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:39:07.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:39:07.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:07 smithi122 ceph-mon[93869]: pgmap v11422: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 452 KiB/s rd, 421 KiB/s wr, 28 op/s 2023-03-09T20:39:07.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20458.590186 secs 2023-03-09T20:39:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:39:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:39:07.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:07 smithi181 ceph-mon[94189]: pgmap v11422: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 452 KiB/s rd, 421 KiB/s wr, 28 op/s 2023-03-09T20:39:07.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20458.590186 secs 2023-03-09T20:39:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:39:07.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:39:07.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:07 smithi038 ceph-mon[92422]: pgmap v11422: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 452 KiB/s rd, 421 KiB/s wr, 28 op/s 2023-03-09T20:39:07.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20458.590186 secs 2023-03-09T20:39:08.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20920.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:39:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:09 smithi122 ceph-mon[93869]: pgmap v11423: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 601 KiB/s rd, 570 KiB/s wr, 28 op/s 2023-03-09T20:39:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:09 smithi181 ceph-mon[94189]: pgmap v11423: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 601 KiB/s rd, 570 KiB/s wr, 28 op/s 2023-03-09T20:39:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:09 smithi038 ceph-mon[92422]: pgmap v11423: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 601 KiB/s rd, 570 KiB/s wr, 28 op/s 2023-03-09T20:39:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20461.276851 secs 2023-03-09T20:39:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20461.276851 secs 2023-03-09T20:39:11.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20461.276851 secs 2023-03-09T20:39:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:11 smithi122 ceph-mon[93869]: pgmap v11424: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 601 KiB/s rd, 570 KiB/s wr, 28 op/s 2023-03-09T20:39:11.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:11 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:39:11.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:11 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:39:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:11 smithi181 ceph-mon[94189]: pgmap v11424: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 601 KiB/s rd, 570 KiB/s wr, 28 op/s 2023-03-09T20:39:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:11 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:39:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:11 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:39:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:11 smithi038 ceph-mon[92422]: pgmap v11424: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 601 KiB/s rd, 570 KiB/s wr, 28 op/s 2023-03-09T20:39:12.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:11 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:39:12.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:11 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:39:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:39:12.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:39:12.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20463.590301 secs 2023-03-09T20:39:12.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:39:12.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:39:12.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20463.590301 secs 2023-03-09T20:39:13.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:39:13.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:39:13.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20463.590301 secs 2023-03-09T20:39:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20925.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:39:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:13 smithi122 ceph-mon[93869]: pgmap v11425: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 353 KiB/s wr, 26 op/s 2023-03-09T20:39:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:13 smithi181 ceph-mon[94189]: pgmap v11425: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 353 KiB/s wr, 26 op/s 2023-03-09T20:39:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:13 smithi038 ceph-mon[92422]: pgmap v11425: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 353 KiB/s wr, 26 op/s 2023-03-09T20:39:15.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:15 smithi122 ceph-mon[93869]: osdmap e928: 12 total, 12 up, 12 in 2023-03-09T20:39:15.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:15 smithi122 ceph-mon[93869]: pgmap v11427: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 327 KiB/s wr, 11 op/s 2023-03-09T20:39:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:15 smithi181 ceph-mon[94189]: osdmap e928: 12 total, 12 up, 12 in 2023-03-09T20:39:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:15 smithi181 ceph-mon[94189]: pgmap v11427: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 327 KiB/s wr, 11 op/s 2023-03-09T20:39:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:15 smithi038 ceph-mon[92422]: osdmap e928: 12 total, 12 up, 12 in 2023-03-09T20:39:15.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:15 smithi038 ceph-mon[92422]: pgmap v11427: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 327 KiB/s wr, 11 op/s 2023-03-09T20:39:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20466.276911 secs 2023-03-09T20:39:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20466.276911 secs 2023-03-09T20:39:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20466.276911 secs 2023-03-09T20:39:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:17 smithi181 ceph-mon[94189]: pgmap v11428: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 233 KiB/s wr, 10 op/s 2023-03-09T20:39:17.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20468.590425 secs 2023-03-09T20:39:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:17 smithi038 ceph-mon[92422]: pgmap v11428: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 233 KiB/s wr, 10 op/s 2023-03-09T20:39:17.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20468.590425 secs 2023-03-09T20:39:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:17 smithi122 ceph-mon[93869]: pgmap v11428: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 233 KiB/s wr, 10 op/s 2023-03-09T20:39:17.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20468.590425 secs 2023-03-09T20:39:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20930.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:39:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:19 smithi122 ceph-mon[93869]: pgmap v11429: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 83 KiB/s wr, 8 op/s 2023-03-09T20:39:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:19 smithi181 ceph-mon[94189]: pgmap v11429: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 83 KiB/s wr, 8 op/s 2023-03-09T20:39:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:19 smithi038 ceph-mon[92422]: pgmap v11429: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 83 KiB/s wr, 8 op/s 2023-03-09T20:39:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20471.276981 secs 2023-03-09T20:39:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20471.276981 secs 2023-03-09T20:39:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20471.276981 secs 2023-03-09T20:39:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:21 smithi122 ceph-mon[93869]: pgmap v11430: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 83 KiB/s wr, 8 op/s 2023-03-09T20:39:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:21 smithi181 ceph-mon[94189]: pgmap v11430: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 83 KiB/s wr, 8 op/s 2023-03-09T20:39:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:21 smithi038 ceph-mon[92422]: pgmap v11430: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 83 KiB/s wr, 8 op/s 2023-03-09T20:39:22.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20473.590541 secs 2023-03-09T20:39:22.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20473.590541 secs 2023-03-09T20:39:22.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20473.590541 secs 2023-03-09T20:39:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20935.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:39:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:23 smithi122 ceph-mon[93869]: pgmap v11431: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T20:39:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:23 smithi181 ceph-mon[94189]: pgmap v11431: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T20:39:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:23 smithi038 ceph-mon[92422]: pgmap v11431: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T20:39:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:25 smithi122 ceph-mon[93869]: pgmap v11432: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T20:39:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:25 smithi181 ceph-mon[94189]: pgmap v11432: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T20:39:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:25 smithi038 ceph-mon[92422]: pgmap v11432: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 19 KiB/s wr, 7 op/s 2023-03-09T20:39:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20476.277043 secs 2023-03-09T20:39:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20476.277043 secs 2023-03-09T20:39:26.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20476.277043 secs 2023-03-09T20:39:27.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:27 smithi038 ceph-mon[92422]: pgmap v11433: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 95 KiB/s wr, 9 op/s 2023-03-09T20:39:27.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20478.590613 secs 2023-03-09T20:39:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:27 smithi122 ceph-mon[93869]: pgmap v11433: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 95 KiB/s wr, 9 op/s 2023-03-09T20:39:27.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20478.590613 secs 2023-03-09T20:39:27.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:27 smithi181 ceph-mon[94189]: pgmap v11433: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 95 KiB/s wr, 9 op/s 2023-03-09T20:39:27.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20478.590613 secs 2023-03-09T20:39:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20940.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:39:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:29 smithi122 ceph-mon[93869]: pgmap v11434: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 95 KiB/s wr, 6 op/s 2023-03-09T20:39:29.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:29 smithi181 ceph-mon[94189]: pgmap v11434: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 95 KiB/s wr, 6 op/s 2023-03-09T20:39:30.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:29 smithi038 ceph-mon[92422]: pgmap v11434: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 95 KiB/s wr, 6 op/s 2023-03-09T20:39:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20481.277117 secs 2023-03-09T20:39:30.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20481.277117 secs 2023-03-09T20:39:30.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20481.277117 secs 2023-03-09T20:39:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:31 smithi122 ceph-mon[93869]: pgmap v11435: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 95 KiB/s wr, 6 op/s 2023-03-09T20:39:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:31 smithi181 ceph-mon[94189]: pgmap v11435: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 95 KiB/s wr, 6 op/s 2023-03-09T20:39:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:31 smithi038 ceph-mon[92422]: pgmap v11435: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 95 KiB/s wr, 6 op/s 2023-03-09T20:39:32.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20483.590701 secs 2023-03-09T20:39:32.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20483.590701 secs 2023-03-09T20:39:32.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20483.590701 secs 2023-03-09T20:39:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20945.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:39:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:33 smithi122 ceph-mon[93869]: pgmap v11436: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 95 KiB/s wr, 9 op/s 2023-03-09T20:39:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:33 smithi181 ceph-mon[94189]: pgmap v11436: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 95 KiB/s wr, 9 op/s 2023-03-09T20:39:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:33 smithi038 ceph-mon[92422]: pgmap v11436: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 95 KiB/s wr, 9 op/s 2023-03-09T20:39:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:35 smithi122 ceph-mon[93869]: pgmap v11437: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 79 KiB/s wr, 6 op/s 2023-03-09T20:39:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:35 smithi181 ceph-mon[94189]: pgmap v11437: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 79 KiB/s wr, 6 op/s 2023-03-09T20:39:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:35 smithi038 ceph-mon[92422]: pgmap v11437: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 79 KiB/s wr, 6 op/s 2023-03-09T20:39:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20486.277184 secs 2023-03-09T20:39:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20486.277184 secs 2023-03-09T20:39:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20486.277184 secs 2023-03-09T20:39:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:37 smithi038 ceph-mon[92422]: pgmap v11438: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 216 KiB/s wr, 9 op/s 2023-03-09T20:39:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:37 smithi122 ceph-mon[93869]: pgmap v11438: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 216 KiB/s wr, 9 op/s 2023-03-09T20:39:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:37 smithi181 ceph-mon[94189]: pgmap v11438: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 216 KiB/s wr, 9 op/s 2023-03-09T20:39:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20950.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:39:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:39 smithi122 ceph-mon[93869]: pgmap v11439: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 138 KiB/s wr, 6 op/s 2023-03-09T20:39:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:39 smithi181 ceph-mon[94189]: pgmap v11439: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 138 KiB/s wr, 6 op/s 2023-03-09T20:39:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:39 smithi038 ceph-mon[92422]: pgmap v11439: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 138 KiB/s wr, 6 op/s 2023-03-09T20:39:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20491.277265 secs 2023-03-09T20:39:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:39:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:39:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:41 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:39:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:41 smithi122 ceph-mon[93869]: pgmap v11440: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 163 KiB/s wr, 6 op/s 2023-03-09T20:39:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20491.277265 secs 2023-03-09T20:39:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:39:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:39:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:41 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:39:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:41 smithi181 ceph-mon[94189]: pgmap v11440: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 163 KiB/s wr, 6 op/s 2023-03-09T20:39:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20491.277265 secs 2023-03-09T20:39:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:39:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:39:41.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:41 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:39:41.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:41 smithi038 ceph-mon[92422]: pgmap v11440: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 163 KiB/s wr, 6 op/s 2023-03-09T20:39:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:42 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:39:42.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:42 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:39:42.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:42 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:39:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:42 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:39:42.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:42 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:39:42.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:42 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:39:42.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:42 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:39:42.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:42 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:39:42.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:42 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:39:43.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:43 smithi038 ceph-mon[92422]: pgmap v11441: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 163 KiB/s wr, 9 op/s 2023-03-09T20:39:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:43 smithi122 ceph-mon[93869]: pgmap v11441: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 163 KiB/s wr, 9 op/s 2023-03-09T20:39:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:43 smithi181 ceph-mon[94189]: pgmap v11441: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 163 KiB/s wr, 9 op/s 2023-03-09T20:39:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20955.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:39:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:45 smithi122 ceph-mon[93869]: pgmap v11442: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 163 KiB/s wr, 6 op/s 2023-03-09T20:39:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:45 smithi181 ceph-mon[94189]: pgmap v11442: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 163 KiB/s wr, 6 op/s 2023-03-09T20:39:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:45 smithi038 ceph-mon[92422]: pgmap v11442: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 163 KiB/s wr, 6 op/s 2023-03-09T20:39:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20496.277325 secs 2023-03-09T20:39:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20496.277325 secs 2023-03-09T20:39:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20496.277325 secs 2023-03-09T20:39:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:47 smithi038 ceph-mon[92422]: pgmap v11443: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 327 KiB/s wr, 10 op/s 2023-03-09T20:39:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:47 smithi122 ceph-mon[93869]: pgmap v11443: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 327 KiB/s wr, 10 op/s 2023-03-09T20:39:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:47 smithi181 ceph-mon[94189]: pgmap v11443: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 327 KiB/s wr, 10 op/s 2023-03-09T20:39:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:39:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:39:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:39:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:39:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:39:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:39:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20960.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:39:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:49 smithi038 ceph-mon[92422]: pgmap v11444: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 190 KiB/s wr, 7 op/s 2023-03-09T20:39:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:49 smithi122 ceph-mon[93869]: pgmap v11444: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 190 KiB/s wr, 7 op/s 2023-03-09T20:39:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:49 smithi181 ceph-mon[94189]: pgmap v11444: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 190 KiB/s wr, 7 op/s 2023-03-09T20:39:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20501.277411 secs 2023-03-09T20:39:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20501.277411 secs 2023-03-09T20:39:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20501.277411 secs 2023-03-09T20:39:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:51 smithi038 ceph-mon[92422]: pgmap v11445: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 521 KiB/s rd, 222 KiB/s wr, 8 op/s 2023-03-09T20:39:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:39:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:51 smithi122 ceph-mon[93869]: pgmap v11445: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 521 KiB/s rd, 222 KiB/s wr, 8 op/s 2023-03-09T20:39:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:39:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:51 smithi181 ceph-mon[94189]: pgmap v11445: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 521 KiB/s rd, 222 KiB/s wr, 8 op/s 2023-03-09T20:39:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:39:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:39:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:39:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:39:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:39:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:39:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:39:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:39:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:39:52.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:39:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:53 smithi122 ceph-mon[93869]: pgmap v11446: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 514 KiB/s rd, 213 KiB/s wr, 11 op/s 2023-03-09T20:39:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:53 smithi181 ceph-mon[94189]: pgmap v11446: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 514 KiB/s rd, 213 KiB/s wr, 11 op/s 2023-03-09T20:39:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:53 smithi038 ceph-mon[92422]: pgmap v11446: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 514 KiB/s rd, 213 KiB/s wr, 11 op/s 2023-03-09T20:39:53.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20965.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:39:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:55 smithi181 ceph-mon[94189]: pgmap v11447: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 212 KiB/s wr, 8 op/s 2023-03-09T20:39:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:55 smithi038 ceph-mon[92422]: pgmap v11447: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 212 KiB/s wr, 8 op/s 2023-03-09T20:39:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:55 smithi122 ceph-mon[93869]: pgmap v11447: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 212 KiB/s wr, 8 op/s 2023-03-09T20:39:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20506.277471 secs 2023-03-09T20:39:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:39:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:39:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20506.277471 secs 2023-03-09T20:39:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:39:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:39:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20506.277471 secs 2023-03-09T20:39:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:39:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:39:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:39:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:39:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:57 smithi038 ceph-mon[92422]: pgmap v11448: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 571 KiB/s wr, 14 op/s 2023-03-09T20:39:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:39:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:39:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:57 smithi122 ceph-mon[93869]: pgmap v11448: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 571 KiB/s wr, 14 op/s 2023-03-09T20:39:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:39:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:39:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:57 smithi181 ceph-mon[94189]: pgmap v11448: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 571 KiB/s wr, 14 op/s 2023-03-09T20:39:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20970.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:39:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:39:59 smithi122 ceph-mon[93869]: pgmap v11449: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 407 KiB/s wr, 11 op/s 2023-03-09T20:39:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:39:59 smithi181 ceph-mon[94189]: pgmap v11449: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 407 KiB/s wr, 11 op/s 2023-03-09T20:39:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:39:59 smithi038 ceph-mon[92422]: pgmap v11449: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 407 KiB/s wr, 11 op/s 2023-03-09T20:40:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20511.277531 secs 2023-03-09T20:40:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:00 smithi122 ceph-mon[93869]: Health detail: HEALTH_WARN 1 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-09T20:40:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:00 smithi122 ceph-mon[93869]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2023-03-09T20:40:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:00 smithi122 ceph-mon[93869]: mds.h(mds.1): 2 slow requests are blocked > 180 secs 2023-03-09T20:40:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:00 smithi122 ceph-mon[93869]: [WRN] MDS_TRIM: 2 MDSs behind on trimming 2023-03-09T20:40:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:00 smithi122 ceph-mon[93869]: mds.e(mds.0): Behind on trimming (423/128) max_segments: 128, num_segments: 423 2023-03-09T20:40:00.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:00 smithi122 ceph-mon[93869]: mds.b(mds.0): Behind on trimming (423/128) max_segments: 128, num_segments: 423 2023-03-09T20:40:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20511.277531 secs 2023-03-09T20:40:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:00 smithi181 ceph-mon[94189]: Health detail: HEALTH_WARN 1 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-09T20:40:00.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:00 smithi181 ceph-mon[94189]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2023-03-09T20:40:00.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:00 smithi181 ceph-mon[94189]: mds.h(mds.1): 2 slow requests are blocked > 180 secs 2023-03-09T20:40:00.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:00 smithi181 ceph-mon[94189]: [WRN] MDS_TRIM: 2 MDSs behind on trimming 2023-03-09T20:40:00.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:00 smithi181 ceph-mon[94189]: mds.e(mds.0): Behind on trimming (423/128) max_segments: 128, num_segments: 423 2023-03-09T20:40:00.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:00 smithi181 ceph-mon[94189]: mds.b(mds.0): Behind on trimming (423/128) max_segments: 128, num_segments: 423 2023-03-09T20:40:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20511.277531 secs 2023-03-09T20:40:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:00 smithi038 ceph-mon[92422]: Health detail: HEALTH_WARN 1 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-09T20:40:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:00 smithi038 ceph-mon[92422]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2023-03-09T20:40:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:00 smithi038 ceph-mon[92422]: mds.h(mds.1): 2 slow requests are blocked > 180 secs 2023-03-09T20:40:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:00 smithi038 ceph-mon[92422]: [WRN] MDS_TRIM: 2 MDSs behind on trimming 2023-03-09T20:40:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:00 smithi038 ceph-mon[92422]: mds.e(mds.0): Behind on trimming (423/128) max_segments: 128, num_segments: 423 2023-03-09T20:40:01.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:00 smithi038 ceph-mon[92422]: mds.b(mds.0): Behind on trimming (423/128) max_segments: 128, num_segments: 423 2023-03-09T20:40:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:01 smithi122 ceph-mon[93869]: pgmap v11450: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 422 KiB/s rd, 407 KiB/s wr, 12 op/s 2023-03-09T20:40:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:01 smithi181 ceph-mon[94189]: pgmap v11450: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 422 KiB/s rd, 407 KiB/s wr, 12 op/s 2023-03-09T20:40:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:01 smithi038 ceph-mon[92422]: pgmap v11450: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 422 KiB/s rd, 407 KiB/s wr, 12 op/s 2023-03-09T20:40:02.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20513.591353 secs 2023-03-09T20:40:02.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20513.591353 secs 2023-03-09T20:40:02.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20513.591353 secs 2023-03-09T20:40:03.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20975.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:40:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:03 smithi122 ceph-mon[93869]: pgmap v11451: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 440 KiB/s wr, 14 op/s 2023-03-09T20:40:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:03 smithi181 ceph-mon[94189]: pgmap v11451: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 440 KiB/s wr, 14 op/s 2023-03-09T20:40:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:03 smithi038 ceph-mon[92422]: pgmap v11451: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 440 KiB/s wr, 14 op/s 2023-03-09T20:40:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:05 smithi122 ceph-mon[93869]: pgmap v11452: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 461 KiB/s wr, 23 op/s 2023-03-09T20:40:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:05 smithi122 ceph-mon[93869]: osdmap e929: 12 total, 12 up, 12 in 2023-03-09T20:40:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20516.277593 secs 2023-03-09T20:40:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:05 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:40:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:05 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:40:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:05 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:40:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:05 smithi181 ceph-mon[94189]: pgmap v11452: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 461 KiB/s wr, 23 op/s 2023-03-09T20:40:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:05 smithi181 ceph-mon[94189]: osdmap e929: 12 total, 12 up, 12 in 2023-03-09T20:40:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20516.277593 secs 2023-03-09T20:40:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:05 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:40:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:05 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:40:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:05 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:40:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:05 smithi038 ceph-mon[92422]: pgmap v11452: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 461 KiB/s wr, 23 op/s 2023-03-09T20:40:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:05 smithi038 ceph-mon[92422]: osdmap e929: 12 total, 12 up, 12 in 2023-03-09T20:40:06.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20516.277593 secs 2023-03-09T20:40:06.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:05 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:40:06.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:05 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:40:06.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:05 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:40:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:07 smithi181 ceph-mon[94189]: pgmap v11454: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 274 KiB/s wr, 28 op/s 2023-03-09T20:40:07.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20518.591468 secs 2023-03-09T20:40:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:07 smithi038 ceph-mon[92422]: pgmap v11454: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 274 KiB/s wr, 28 op/s 2023-03-09T20:40:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20518.591468 secs 2023-03-09T20:40:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:07 smithi122 ceph-mon[93869]: pgmap v11454: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 274 KiB/s wr, 28 op/s 2023-03-09T20:40:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20518.591468 secs 2023-03-09T20:40:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20980.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:40:10.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:09 smithi038 ceph-mon[92422]: pgmap v11455: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 274 KiB/s wr, 28 op/s 2023-03-09T20:40:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:09 smithi122 ceph-mon[93869]: pgmap v11455: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 274 KiB/s wr, 28 op/s 2023-03-09T20:40:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:09 smithi181 ceph-mon[94189]: pgmap v11455: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 274 KiB/s wr, 28 op/s 2023-03-09T20:40:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20521.277652 secs 2023-03-09T20:40:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20521.277652 secs 2023-03-09T20:40:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20521.277652 secs 2023-03-09T20:40:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:11 smithi122 ceph-mon[93869]: pgmap v11456: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 402 KiB/s wr, 28 op/s 2023-03-09T20:40:12.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:11 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:40:12.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:11 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:40:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:11 smithi181 ceph-mon[94189]: pgmap v11456: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 402 KiB/s wr, 28 op/s 2023-03-09T20:40:12.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:11 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:40:12.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:11 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:40:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:11 smithi038 ceph-mon[92422]: pgmap v11456: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 402 KiB/s wr, 28 op/s 2023-03-09T20:40:12.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:11 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:40:12.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:11 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:40:13.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:40:13.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:40:13.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20523.591565 secs 2023-03-09T20:40:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:40:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:40:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20523.591565 secs 2023-03-09T20:40:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:40:13.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:40:13.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20523.591565 secs 2023-03-09T20:40:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20985.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:40:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:13 smithi122 ceph-mon[93869]: pgmap v11457: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 325 KiB/s wr, 26 op/s 2023-03-09T20:40:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:13 smithi181 ceph-mon[94189]: pgmap v11457: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 325 KiB/s wr, 26 op/s 2023-03-09T20:40:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:13 smithi038 ceph-mon[92422]: pgmap v11457: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 325 KiB/s wr, 26 op/s 2023-03-09T20:40:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:15 smithi181 ceph-mon[94189]: osdmap e930: 12 total, 12 up, 12 in 2023-03-09T20:40:15.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:15 smithi181 ceph-mon[94189]: pgmap v11459: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 283 KiB/s wr, 12 op/s 2023-03-09T20:40:15.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:15 smithi038 ceph-mon[92422]: osdmap e930: 12 total, 12 up, 12 in 2023-03-09T20:40:15.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:15 smithi038 ceph-mon[92422]: pgmap v11459: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 283 KiB/s wr, 12 op/s 2023-03-09T20:40:15.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:15 smithi122 ceph-mon[93869]: osdmap e930: 12 total, 12 up, 12 in 2023-03-09T20:40:15.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:15 smithi122 ceph-mon[93869]: pgmap v11459: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 283 KiB/s wr, 12 op/s 2023-03-09T20:40:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20526.277713 secs 2023-03-09T20:40:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20526.277713 secs 2023-03-09T20:40:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20526.277713 secs 2023-03-09T20:40:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:17 smithi038 ceph-mon[92422]: pgmap v11460: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 129 KiB/s wr, 8 op/s 2023-03-09T20:40:17.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20528.591710 secs 2023-03-09T20:40:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:17 smithi122 ceph-mon[93869]: pgmap v11460: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 129 KiB/s wr, 8 op/s 2023-03-09T20:40:17.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20528.591710 secs 2023-03-09T20:40:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:17 smithi181 ceph-mon[94189]: pgmap v11460: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 129 KiB/s wr, 8 op/s 2023-03-09T20:40:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20528.591710 secs 2023-03-09T20:40:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20990.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:40:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:19 smithi122 ceph-mon[93869]: pgmap v11461: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 128 KiB/s wr, 8 op/s 2023-03-09T20:40:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:19 smithi181 ceph-mon[94189]: pgmap v11461: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 128 KiB/s wr, 8 op/s 2023-03-09T20:40:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:19 smithi038 ceph-mon[92422]: pgmap v11461: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 128 KiB/s wr, 8 op/s 2023-03-09T20:40:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20531.277806 secs 2023-03-09T20:40:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20531.277806 secs 2023-03-09T20:40:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20531.277806 secs 2023-03-09T20:40:21.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:21 smithi122 ceph-mon[93869]: pgmap v11462: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 614 B/s wr, 7 op/s 2023-03-09T20:40:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:21 smithi181 ceph-mon[94189]: pgmap v11462: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 614 B/s wr, 7 op/s 2023-03-09T20:40:22.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:21 smithi038 ceph-mon[92422]: pgmap v11462: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 614 B/s wr, 7 op/s 2023-03-09T20:40:22.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20533.591781 secs 2023-03-09T20:40:22.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20533.591781 secs 2023-03-09T20:40:22.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20533.591781 secs 2023-03-09T20:40:23.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 20995.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:40:23.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:23 smithi122 ceph-mon[93869]: pgmap v11463: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T20:40:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:23 smithi181 ceph-mon[94189]: pgmap v11463: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T20:40:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:23 smithi038 ceph-mon[92422]: pgmap v11463: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T20:40:25.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:25 smithi038 ceph-mon[92422]: pgmap v11464: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T20:40:25.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20536.277866 secs 2023-03-09T20:40:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:25 smithi122 ceph-mon[93869]: pgmap v11464: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T20:40:25.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20536.277866 secs 2023-03-09T20:40:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:25 smithi181 ceph-mon[94189]: pgmap v11464: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T20:40:25.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20536.277866 secs 2023-03-09T20:40:27.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:27 smithi122 ceph-mon[93869]: pgmap v11465: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 8 op/s 2023-03-09T20:40:27.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20538.591919 secs 2023-03-09T20:40:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:27 smithi181 ceph-mon[94189]: pgmap v11465: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 8 op/s 2023-03-09T20:40:27.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20538.591919 secs 2023-03-09T20:40:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:27 smithi038 ceph-mon[92422]: pgmap v11465: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 8 op/s 2023-03-09T20:40:28.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20538.591919 secs 2023-03-09T20:40:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21000.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:40:29.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:29 smithi122 ceph-mon[93869]: pgmap v11466: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 170 B/s wr, 6 op/s 2023-03-09T20:40:29.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:29 smithi181 ceph-mon[94189]: pgmap v11466: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 170 B/s wr, 6 op/s 2023-03-09T20:40:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:29 smithi038 ceph-mon[92422]: pgmap v11466: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 170 B/s wr, 6 op/s 2023-03-09T20:40:30.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20541.277927 secs 2023-03-09T20:40:30.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20541.277927 secs 2023-03-09T20:40:30.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20541.277927 secs 2023-03-09T20:40:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:31 smithi122 ceph-mon[93869]: pgmap v11467: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 64 KiB/s wr, 7 op/s 2023-03-09T20:40:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:31 smithi181 ceph-mon[94189]: pgmap v11467: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 64 KiB/s wr, 7 op/s 2023-03-09T20:40:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:31 smithi038 ceph-mon[92422]: pgmap v11467: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 64 KiB/s wr, 7 op/s 2023-03-09T20:40:32.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20543.592032 secs 2023-03-09T20:40:32.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20543.592032 secs 2023-03-09T20:40:32.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20543.592032 secs 2023-03-09T20:40:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21005.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:40:33.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:33 smithi122 ceph-mon[93869]: pgmap v11468: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 65 KiB/s wr, 8 op/s 2023-03-09T20:40:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:33 smithi181 ceph-mon[94189]: pgmap v11468: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 65 KiB/s wr, 8 op/s 2023-03-09T20:40:34.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:33 smithi038 ceph-mon[92422]: pgmap v11468: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 65 KiB/s wr, 8 op/s 2023-03-09T20:40:35.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:35 smithi038 ceph-mon[92422]: pgmap v11469: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 143 KiB/s wr, 7 op/s 2023-03-09T20:40:35.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20546.277983 secs 2023-03-09T20:40:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:35 smithi122 ceph-mon[93869]: pgmap v11469: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 143 KiB/s wr, 7 op/s 2023-03-09T20:40:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20546.277983 secs 2023-03-09T20:40:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:35 smithi181 ceph-mon[94189]: pgmap v11469: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 143 KiB/s wr, 7 op/s 2023-03-09T20:40:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20546.277983 secs 2023-03-09T20:40:37.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:37 smithi122 ceph-mon[93869]: pgmap v11470: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 143 KiB/s wr, 9 op/s 2023-03-09T20:40:37.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:37 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 20548.592155 secs 2023-03-09T20:40:37.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:37 smithi181 ceph-mon[94189]: pgmap v11470: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 143 KiB/s wr, 9 op/s 2023-03-09T20:40:37.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:37 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 20548.592155 secs 2023-03-09T20:40:38.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:37 smithi038 ceph-mon[92422]: pgmap v11470: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 143 KiB/s wr, 9 op/s 2023-03-09T20:40:38.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:37 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 20548.592155 secs 2023-03-09T20:40:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21010.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:40:39.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:39 smithi122 ceph-mon[93869]: pgmap v11471: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 143 KiB/s wr, 6 op/s 2023-03-09T20:40:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:39 smithi181 ceph-mon[94189]: pgmap v11471: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 143 KiB/s wr, 6 op/s 2023-03-09T20:40:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:39 smithi038 ceph-mon[92422]: pgmap v11471: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 143 KiB/s wr, 6 op/s 2023-03-09T20:40:40.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20551.278044 secs 2023-03-09T20:40:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20551.278044 secs 2023-03-09T20:40:41.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20551.278044 secs 2023-03-09T20:40:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:41 smithi038 ceph-mon[92422]: pgmap v11472: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 220 KiB/s wr, 8 op/s 2023-03-09T20:40:42.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:40:42.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:40:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:41 smithi122 ceph-mon[93869]: pgmap v11472: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 220 KiB/s wr, 8 op/s 2023-03-09T20:40:42.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:40:42.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:40:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:41 smithi181 ceph-mon[94189]: pgmap v11472: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 220 KiB/s wr, 8 op/s 2023-03-09T20:40:42.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:40:42.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:40:43.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:42 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 20553.592276 secs 2023-03-09T20:40:43.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:42 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 20553.592276 secs 2023-03-09T20:40:43.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:42 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 20553.592276 secs 2023-03-09T20:40:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21015.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:40:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:43 smithi122 ceph-mon[93869]: pgmap v11473: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 164 KiB/s wr, 9 op/s 2023-03-09T20:40:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:43 smithi181 ceph-mon[94189]: pgmap v11473: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 164 KiB/s wr, 9 op/s 2023-03-09T20:40:44.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:43 smithi038 ceph-mon[92422]: pgmap v11473: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 164 KiB/s wr, 9 op/s 2023-03-09T20:40:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:45 smithi038 ceph-mon[92422]: pgmap v11474: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 164 KiB/s wr, 7 op/s 2023-03-09T20:40:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20556.278106 secs 2023-03-09T20:40:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:45 smithi122 ceph-mon[93869]: pgmap v11474: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 164 KiB/s wr, 7 op/s 2023-03-09T20:40:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20556.278106 secs 2023-03-09T20:40:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:45 smithi181 ceph-mon[94189]: pgmap v11474: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 164 KiB/s wr, 7 op/s 2023-03-09T20:40:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20556.278106 secs 2023-03-09T20:40:47.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:47 smithi122 ceph-mon[93869]: pgmap v11475: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 101 KiB/s wr, 9 op/s 2023-03-09T20:40:47.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:47 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 20558.592341 secs 2023-03-09T20:40:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:47 smithi181 ceph-mon[94189]: pgmap v11475: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 101 KiB/s wr, 9 op/s 2023-03-09T20:40:47.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:47 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 20558.592341 secs 2023-03-09T20:40:48.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:47 smithi038 ceph-mon[92422]: pgmap v11475: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 101 KiB/s wr, 9 op/s 2023-03-09T20:40:48.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:47 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 20558.592341 secs 2023-03-09T20:40:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21020.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:40:48.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:40:48.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:40:48.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:40:48.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:40:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:40:49.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:40:49.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:49 smithi122 ceph-mon[93869]: pgmap v11476: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 250 KiB/s wr, 8 op/s 2023-03-09T20:40:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:49 smithi181 ceph-mon[94189]: pgmap v11476: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 250 KiB/s wr, 8 op/s 2023-03-09T20:40:50.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:49 smithi038 ceph-mon[92422]: pgmap v11476: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 250 KiB/s wr, 8 op/s 2023-03-09T20:40:50.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20561.278165 secs 2023-03-09T20:40:51.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20561.278165 secs 2023-03-09T20:40:51.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20561.278165 secs 2023-03-09T20:40:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:51 smithi181 ceph-mon[94189]: pgmap v11477: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 250 KiB/s wr, 8 op/s 2023-03-09T20:40:51.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:40:51.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:40:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:51 smithi038 ceph-mon[92422]: pgmap v11477: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 250 KiB/s wr, 8 op/s 2023-03-09T20:40:52.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:40:52.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:40:52.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:51 smithi122 ceph-mon[93869]: pgmap v11477: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 250 KiB/s wr, 8 op/s 2023-03-09T20:40:52.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:40:52.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:40:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:40:53.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20563.592407 secs 2023-03-09T20:40:53.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:40:53.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:40:53.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:40:53.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:40:53.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20563.592407 secs 2023-03-09T20:40:53.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:40:53.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:40:53.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:40:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:40:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20563.592407 secs 2023-03-09T20:40:53.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:40:53.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:40:53.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:40:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21025.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:40:54.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:53 smithi038 ceph-mon[92422]: pgmap v11478: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 221 KiB/s wr, 10 op/s 2023-03-09T20:40:54.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:40:54.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:40:54.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:53 smithi122 ceph-mon[93869]: pgmap v11478: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 221 KiB/s wr, 10 op/s 2023-03-09T20:40:54.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:40:54.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:40:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:53 smithi181 ceph-mon[94189]: pgmap v11478: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 221 KiB/s wr, 10 op/s 2023-03-09T20:40:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:40:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:40:55.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:55 smithi038 ceph-mon[92422]: pgmap v11479: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 214 KiB/s wr, 9 op/s 2023-03-09T20:40:55.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20566.278230 secs 2023-03-09T20:40:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:55 smithi122 ceph-mon[93869]: pgmap v11479: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 214 KiB/s wr, 9 op/s 2023-03-09T20:40:55.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20566.278230 secs 2023-03-09T20:40:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:55 smithi181 ceph-mon[94189]: pgmap v11479: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 214 KiB/s wr, 9 op/s 2023-03-09T20:40:55.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20566.278230 secs 2023-03-09T20:40:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:40:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:40:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:40:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:40:56.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:40:56.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:40:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:40:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:40:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:57 smithi122 ceph-mon[93869]: pgmap v11480: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 230 KiB/s wr, 11 op/s 2023-03-09T20:40:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20568.592531 secs 2023-03-09T20:40:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:40:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:40:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:57 smithi181 ceph-mon[94189]: pgmap v11480: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 230 KiB/s wr, 11 op/s 2023-03-09T20:40:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20568.592531 secs 2023-03-09T20:40:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:40:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:40:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:57 smithi038 ceph-mon[92422]: pgmap v11480: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 230 KiB/s wr, 11 op/s 2023-03-09T20:40:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20568.592531 secs 2023-03-09T20:40:58.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21030.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:40:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:40:59 smithi122 ceph-mon[93869]: pgmap v11481: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 513 KiB/s rd, 498 KiB/s wr, 10 op/s 2023-03-09T20:40:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:40:59 smithi181 ceph-mon[94189]: pgmap v11481: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 513 KiB/s rd, 498 KiB/s wr, 10 op/s 2023-03-09T20:41:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:40:59 smithi038 ceph-mon[92422]: pgmap v11481: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 513 KiB/s rd, 498 KiB/s wr, 10 op/s 2023-03-09T20:41:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20571.278288 secs 2023-03-09T20:41:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20571.278288 secs 2023-03-09T20:41:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20571.278288 secs 2023-03-09T20:41:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:01 smithi122 ceph-mon[93869]: pgmap v11482: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 350 KiB/s wr, 10 op/s 2023-03-09T20:41:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:01 smithi181 ceph-mon[94189]: pgmap v11482: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 350 KiB/s wr, 10 op/s 2023-03-09T20:41:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:01 smithi038 ceph-mon[92422]: pgmap v11482: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 350 KiB/s wr, 10 op/s 2023-03-09T20:41:02.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20573.592652 secs 2023-03-09T20:41:02.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20573.592652 secs 2023-03-09T20:41:02.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20573.592652 secs 2023-03-09T20:41:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21035.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:41:03.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:03 smithi122 ceph-mon[93869]: pgmap v11483: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 382 KiB/s wr, 12 op/s 2023-03-09T20:41:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:03 smithi181 ceph-mon[94189]: pgmap v11483: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 382 KiB/s wr, 12 op/s 2023-03-09T20:41:04.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:03 smithi038 ceph-mon[92422]: pgmap v11483: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 382 KiB/s wr, 12 op/s 2023-03-09T20:41:05.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:05 smithi038 ceph-mon[92422]: pgmap v11484: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 363 KiB/s rd, 355 KiB/s wr, 20 op/s 2023-03-09T20:41:05.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20576.278372 secs 2023-03-09T20:41:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:05 smithi122 ceph-mon[93869]: pgmap v11484: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 363 KiB/s rd, 355 KiB/s wr, 20 op/s 2023-03-09T20:41:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20576.278372 secs 2023-03-09T20:41:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:05 smithi181 ceph-mon[94189]: pgmap v11484: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 363 KiB/s rd, 355 KiB/s wr, 20 op/s 2023-03-09T20:41:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20576.278372 secs 2023-03-09T20:41:07.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:41:07.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:41:07.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:07 smithi122 ceph-mon[93869]: pgmap v11485: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 382 KiB/s wr, 23 op/s 2023-03-09T20:41:07.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20578.592792 secs 2023-03-09T20:41:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:41:07.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:41:07.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:07 smithi181 ceph-mon[94189]: pgmap v11485: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 382 KiB/s wr, 23 op/s 2023-03-09T20:41:07.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20578.592792 secs 2023-03-09T20:41:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:41:07.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:41:07.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:07 smithi038 ceph-mon[92422]: pgmap v11485: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 382 KiB/s wr, 23 op/s 2023-03-09T20:41:07.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20578.592792 secs 2023-03-09T20:41:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21040.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:41:09.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:41:09.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:41:09.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:09 smithi122 ceph-mon[93869]: pgmap v11486: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 531 KiB/s rd, 544 KiB/s wr, 22 op/s 2023-03-09T20:41:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:41:09.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:41:09.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:09 smithi181 ceph-mon[94189]: pgmap v11486: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 531 KiB/s rd, 544 KiB/s wr, 22 op/s 2023-03-09T20:41:09.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:41:09.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:41:09.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:09 smithi038 ceph-mon[92422]: pgmap v11486: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 531 KiB/s rd, 544 KiB/s wr, 22 op/s 2023-03-09T20:41:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20581.278432 secs 2023-03-09T20:41:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20581.278432 secs 2023-03-09T20:41:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20581.278432 secs 2023-03-09T20:41:11.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:11 smithi122 ceph-mon[93869]: pgmap v11487: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 366 KiB/s wr, 22 op/s 2023-03-09T20:41:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:11 smithi181 ceph-mon[94189]: pgmap v11487: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 366 KiB/s wr, 22 op/s 2023-03-09T20:41:11.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:11 smithi038 ceph-mon[92422]: pgmap v11487: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 366 KiB/s wr, 22 op/s 2023-03-09T20:41:12.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20583.592879 secs 2023-03-09T20:41:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20583.592879 secs 2023-03-09T20:41:12.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20583.592879 secs 2023-03-09T20:41:13.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:13 smithi122 ceph-mon[93869]: pgmap v11488: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 382 KiB/s wr, 23 op/s 2023-03-09T20:41:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:13 smithi181 ceph-mon[94189]: pgmap v11488: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 382 KiB/s wr, 23 op/s 2023-03-09T20:41:13.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:13 smithi038 ceph-mon[92422]: pgmap v11488: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 382 KiB/s wr, 23 op/s 2023-03-09T20:41:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21045.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:41:15.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:15 smithi122 ceph-mon[93869]: pgmap v11489: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 403 KiB/s wr, 21 op/s 2023-03-09T20:41:15.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20586.278493 secs 2023-03-09T20:41:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:15 smithi181 ceph-mon[94189]: pgmap v11489: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 403 KiB/s wr, 21 op/s 2023-03-09T20:41:15.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20586.278493 secs 2023-03-09T20:41:15.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:15 smithi038 ceph-mon[92422]: pgmap v11489: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 403 KiB/s wr, 21 op/s 2023-03-09T20:41:15.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20586.278493 secs 2023-03-09T20:41:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:17 smithi122 ceph-mon[93869]: pgmap v11490: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 382 KiB/s wr, 12 op/s 2023-03-09T20:41:17.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20588.592975 secs 2023-03-09T20:41:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:17 smithi181 ceph-mon[94189]: pgmap v11490: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 382 KiB/s wr, 12 op/s 2023-03-09T20:41:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20588.592975 secs 2023-03-09T20:41:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:17 smithi038 ceph-mon[92422]: pgmap v11490: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 382 KiB/s wr, 12 op/s 2023-03-09T20:41:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20588.592975 secs 2023-03-09T20:41:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21050.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:41:19.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:19 smithi122 ceph-mon[93869]: pgmap v11491: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 355 KiB/s wr, 9 op/s 2023-03-09T20:41:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:19 smithi181 ceph-mon[94189]: pgmap v11491: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 355 KiB/s wr, 9 op/s 2023-03-09T20:41:20.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:19 smithi038 ceph-mon[92422]: pgmap v11491: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 355 KiB/s wr, 9 op/s 2023-03-09T20:41:20.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20591.278553 secs 2023-03-09T20:41:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20591.278553 secs 2023-03-09T20:41:21.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20591.278553 secs 2023-03-09T20:41:21.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:21 smithi122 ceph-mon[93869]: pgmap v11492: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 230 KiB/s wr, 9 op/s 2023-03-09T20:41:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:21 smithi181 ceph-mon[94189]: pgmap v11492: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 230 KiB/s wr, 9 op/s 2023-03-09T20:41:22.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:21 smithi038 ceph-mon[92422]: pgmap v11492: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 230 KiB/s wr, 9 op/s 2023-03-09T20:41:23.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20593.593118 secs 2023-03-09T20:41:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20593.593118 secs 2023-03-09T20:41:23.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20593.593118 secs 2023-03-09T20:41:23.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21055.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:41:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:23 smithi038 ceph-mon[92422]: pgmap v11493: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 150 KiB/s wr, 9 op/s 2023-03-09T20:41:24.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:23 smithi122 ceph-mon[93869]: pgmap v11493: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 150 KiB/s wr, 9 op/s 2023-03-09T20:41:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:23 smithi181 ceph-mon[94189]: pgmap v11493: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 150 KiB/s wr, 9 op/s 2023-03-09T20:41:26.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:25 smithi038 ceph-mon[92422]: pgmap v11494: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 318 KiB/s wr, 10 op/s 2023-03-09T20:41:26.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:25 smithi038 ceph-mon[92422]: osdmap e931: 12 total, 12 up, 12 in 2023-03-09T20:41:26.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20596.278619 secs 2023-03-09T20:41:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:25 smithi122 ceph-mon[93869]: pgmap v11494: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 318 KiB/s wr, 10 op/s 2023-03-09T20:41:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:25 smithi122 ceph-mon[93869]: osdmap e931: 12 total, 12 up, 12 in 2023-03-09T20:41:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20596.278619 secs 2023-03-09T20:41:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:25 smithi181 ceph-mon[94189]: pgmap v11494: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 318 KiB/s wr, 10 op/s 2023-03-09T20:41:26.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:25 smithi181 ceph-mon[94189]: osdmap e931: 12 total, 12 up, 12 in 2023-03-09T20:41:26.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20596.278619 secs 2023-03-09T20:41:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:27 smithi122 ceph-mon[93869]: pgmap v11496: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 394 KiB/s wr, 12 op/s 2023-03-09T20:41:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20598.593211 secs 2023-03-09T20:41:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:27 smithi181 ceph-mon[94189]: pgmap v11496: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 394 KiB/s wr, 12 op/s 2023-03-09T20:41:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20598.593211 secs 2023-03-09T20:41:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:27 smithi038 ceph-mon[92422]: pgmap v11496: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 394 KiB/s wr, 12 op/s 2023-03-09T20:41:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20598.593211 secs 2023-03-09T20:41:28.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21060.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:41:29.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:28 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:41:29.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:28 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:41:29.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:28 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:41:29.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:28 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:41:29.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:28 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:41:29.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:28 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:41:30.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:29 smithi122 ceph-mon[93869]: pgmap v11497: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 395 KiB/s wr, 12 op/s 2023-03-09T20:41:30.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:29 smithi181 ceph-mon[94189]: pgmap v11497: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 395 KiB/s wr, 12 op/s 2023-03-09T20:41:30.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:29 smithi038 ceph-mon[92422]: pgmap v11497: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 395 KiB/s wr, 12 op/s 2023-03-09T20:41:31.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20601.278700 secs 2023-03-09T20:41:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20601.278700 secs 2023-03-09T20:41:31.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20601.278700 secs 2023-03-09T20:41:32.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:31 smithi122 ceph-mon[93869]: pgmap v11498: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 456 KiB/s wr, 12 op/s 2023-03-09T20:41:32.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:31 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:41:32.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:31 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:41:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:31 smithi181 ceph-mon[94189]: pgmap v11498: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 456 KiB/s wr, 12 op/s 2023-03-09T20:41:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:31 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:41:32.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:31 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:41:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:31 smithi038 ceph-mon[92422]: pgmap v11498: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 456 KiB/s wr, 12 op/s 2023-03-09T20:41:32.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:31 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:41:32.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:31 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:41:33.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20603.593316 secs 2023-03-09T20:41:33.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20603.593316 secs 2023-03-09T20:41:33.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20603.593316 secs 2023-03-09T20:41:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21065.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:41:34.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:33 smithi122 ceph-mon[93869]: pgmap v11499: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 423 KiB/s wr, 12 op/s 2023-03-09T20:41:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:33 smithi181 ceph-mon[94189]: pgmap v11499: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 423 KiB/s wr, 12 op/s 2023-03-09T20:41:34.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:33 smithi038 ceph-mon[92422]: pgmap v11499: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 423 KiB/s wr, 12 op/s 2023-03-09T20:41:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:35 smithi122 ceph-mon[93869]: osdmap e932: 12 total, 12 up, 12 in 2023-03-09T20:41:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:35 smithi122 ceph-mon[93869]: pgmap v11501: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 205 KiB/s wr, 9 op/s 2023-03-09T20:41:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20606.278761 secs 2023-03-09T20:41:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:35 smithi181 ceph-mon[94189]: osdmap e932: 12 total, 12 up, 12 in 2023-03-09T20:41:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:35 smithi181 ceph-mon[94189]: pgmap v11501: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 205 KiB/s wr, 9 op/s 2023-03-09T20:41:35.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20606.278761 secs 2023-03-09T20:41:35.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:35 smithi038 ceph-mon[92422]: osdmap e932: 12 total, 12 up, 12 in 2023-03-09T20:41:35.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:35 smithi038 ceph-mon[92422]: pgmap v11501: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 205 KiB/s wr, 9 op/s 2023-03-09T20:41:35.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20606.278761 secs 2023-03-09T20:41:37.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:37 smithi122 ceph-mon[93869]: pgmap v11502: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 191 KiB/s wr, 10 op/s 2023-03-09T20:41:37.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20608.593442 secs 2023-03-09T20:41:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:37 smithi181 ceph-mon[94189]: pgmap v11502: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 191 KiB/s wr, 10 op/s 2023-03-09T20:41:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20608.593442 secs 2023-03-09T20:41:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:37 smithi038 ceph-mon[92422]: pgmap v11502: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 191 KiB/s wr, 10 op/s 2023-03-09T20:41:38.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20608.593442 secs 2023-03-09T20:41:38.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21070.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:41:39.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:39 smithi122 ceph-mon[93869]: pgmap v11503: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 191 KiB/s wr, 9 op/s 2023-03-09T20:41:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:39 smithi181 ceph-mon[94189]: pgmap v11503: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 191 KiB/s wr, 9 op/s 2023-03-09T20:41:40.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:39 smithi038 ceph-mon[92422]: pgmap v11503: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 191 KiB/s wr, 9 op/s 2023-03-09T20:41:40.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20611.278821 secs 2023-03-09T20:41:40.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:41:40.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:41:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20611.278821 secs 2023-03-09T20:41:40.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:41:40.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:41:41.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20611.278821 secs 2023-03-09T20:41:41.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:41:41.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:41:41.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:41 smithi122 ceph-mon[93869]: pgmap v11504: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 426 KiB/s wr, 10 op/s 2023-03-09T20:41:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:41 smithi181 ceph-mon[94189]: pgmap v11504: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 426 KiB/s wr, 10 op/s 2023-03-09T20:41:42.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:41 smithi038 ceph-mon[92422]: pgmap v11504: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 426 KiB/s wr, 10 op/s 2023-03-09T20:41:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21075.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:41:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:43 smithi181 ceph-mon[94189]: pgmap v11505: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 464 KiB/s wr, 10 op/s 2023-03-09T20:41:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:43 smithi038 ceph-mon[92422]: pgmap v11505: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 464 KiB/s wr, 10 op/s 2023-03-09T20:41:44.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:43 smithi122 ceph-mon[93869]: pgmap v11505: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 464 KiB/s wr, 10 op/s 2023-03-09T20:41:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:45 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:41:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:45 smithi122 ceph-mon[93869]: pgmap v11506: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 446 KiB/s wr, 10 op/s 2023-03-09T20:41:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20616.278881 secs 2023-03-09T20:41:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:45 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:41:45.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:45 smithi181 ceph-mon[94189]: pgmap v11506: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 446 KiB/s wr, 10 op/s 2023-03-09T20:41:45.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20616.278881 secs 2023-03-09T20:41:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:45 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:41:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:45 smithi038 ceph-mon[92422]: pgmap v11506: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 446 KiB/s wr, 10 op/s 2023-03-09T20:41:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20616.278881 secs 2023-03-09T20:41:46.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:46 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:41:46.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:46 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:41:46.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:46 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:41:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:46 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:41:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:46 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:41:46.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:46 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:41:46.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:46 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:41:46.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:46 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:41:46.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:46 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:41:47.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:47 smithi122 ceph-mon[93869]: pgmap v11507: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 637 KiB/s rd, 489 KiB/s wr, 13 op/s 2023-03-09T20:41:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:47 smithi181 ceph-mon[94189]: pgmap v11507: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 637 KiB/s rd, 489 KiB/s wr, 13 op/s 2023-03-09T20:41:47.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:47 smithi038 ceph-mon[92422]: pgmap v11507: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 637 KiB/s rd, 489 KiB/s wr, 13 op/s 2023-03-09T20:41:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:41:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:41:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:41:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:41:48.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:41:48.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:41:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21080.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:41:49.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:49 smithi122 ceph-mon[93869]: pgmap v11508: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 684 KiB/s rd, 538 KiB/s wr, 10 op/s 2023-03-09T20:41:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:49 smithi181 ceph-mon[94189]: pgmap v11508: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 684 KiB/s rd, 538 KiB/s wr, 10 op/s 2023-03-09T20:41:49.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:49 smithi038 ceph-mon[92422]: pgmap v11508: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 684 KiB/s rd, 538 KiB/s wr, 10 op/s 2023-03-09T20:41:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20621.278942 secs 2023-03-09T20:41:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20621.278942 secs 2023-03-09T20:41:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20621.278942 secs 2023-03-09T20:41:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:51 smithi122 ceph-mon[93869]: pgmap v11509: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 707 KiB/s rd, 560 KiB/s wr, 12 op/s 2023-03-09T20:41:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:51 smithi181 ceph-mon[94189]: pgmap v11509: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 707 KiB/s rd, 560 KiB/s wr, 12 op/s 2023-03-09T20:41:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:51 smithi038 ceph-mon[92422]: pgmap v11509: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 707 KiB/s rd, 560 KiB/s wr, 12 op/s 2023-03-09T20:41:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:41:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:41:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:41:52.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:41:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:41:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:41:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:41:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:41:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:53 smithi122 ceph-mon[93869]: pgmap v11510: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 277 KiB/s wr, 11 op/s 2023-03-09T20:41:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:41:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:41:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:41:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:53 smithi181 ceph-mon[94189]: pgmap v11510: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 277 KiB/s wr, 11 op/s 2023-03-09T20:41:53.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:41:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:41:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:41:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:53 smithi038 ceph-mon[92422]: pgmap v11510: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 277 KiB/s wr, 11 op/s 2023-03-09T20:41:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:41:53.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21085.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:41:54.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:54 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:41:54.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:54 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:41:54.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:54 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:41:54.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:54 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:41:54.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:54 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:41:54.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:54 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:41:54.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:54 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:41:54.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:54 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:41:54.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:54 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:41:55.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:55 smithi122 ceph-mon[93869]: pgmap v11511: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 436 KiB/s rd, 320 KiB/s wr, 10 op/s 2023-03-09T20:41:55.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20626.279022 secs 2023-03-09T20:41:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:55 smithi181 ceph-mon[94189]: pgmap v11511: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 436 KiB/s rd, 320 KiB/s wr, 10 op/s 2023-03-09T20:41:55.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20626.279022 secs 2023-03-09T20:41:55.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:55 smithi038 ceph-mon[92422]: pgmap v11511: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 436 KiB/s rd, 320 KiB/s wr, 10 op/s 2023-03-09T20:41:55.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20626.279022 secs 2023-03-09T20:41:57.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:57 smithi122 ceph-mon[93869]: pgmap v11512: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 432 KiB/s rd, 322 KiB/s wr, 12 op/s 2023-03-09T20:41:57.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20628.593919 secs 2023-03-09T20:41:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:57 smithi181 ceph-mon[94189]: pgmap v11512: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 432 KiB/s rd, 322 KiB/s wr, 12 op/s 2023-03-09T20:41:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20628.593919 secs 2023-03-09T20:41:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:57 smithi038 ceph-mon[92422]: pgmap v11512: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 432 KiB/s rd, 322 KiB/s wr, 12 op/s 2023-03-09T20:41:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20628.593919 secs 2023-03-09T20:41:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21090.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:42:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:59 smithi038 ceph-mon[92422]: pgmap v11513: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 398 KiB/s rd, 397 KiB/s wr, 10 op/s 2023-03-09T20:42:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:42:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:41:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:42:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:59 smithi122 ceph-mon[93869]: pgmap v11513: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 398 KiB/s rd, 397 KiB/s wr, 10 op/s 2023-03-09T20:42:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:42:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:41:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:42:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:59 smithi181 ceph-mon[94189]: pgmap v11513: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 398 KiB/s rd, 397 KiB/s wr, 10 op/s 2023-03-09T20:42:00.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:42:00.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:41:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:42:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20631.279090 secs 2023-03-09T20:42:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:42:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:42:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20631.279090 secs 2023-03-09T20:42:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:42:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:42:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20631.279090 secs 2023-03-09T20:42:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:42:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:42:02.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:01 smithi038 ceph-mon[92422]: pgmap v11514: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 294 KiB/s wr, 11 op/s 2023-03-09T20:42:02.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:01 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:42:02.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:42:02.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:42:02.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:01 smithi122 ceph-mon[93869]: pgmap v11514: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 294 KiB/s wr, 11 op/s 2023-03-09T20:42:02.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:01 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:42:02.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:42:02.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:42:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:01 smithi181 ceph-mon[94189]: pgmap v11514: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 294 KiB/s wr, 11 op/s 2023-03-09T20:42:02.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:01 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:42:02.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:42:02.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:42:03.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20633.594038 secs 2023-03-09T20:42:03.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20633.594038 secs 2023-03-09T20:42:03.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20633.594038 secs 2023-03-09T20:42:03.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21095.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:42:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:03 smithi038 ceph-mon[92422]: pgmap v11515: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 386 KiB/s rd, 407 KiB/s wr, 13 op/s 2023-03-09T20:42:04.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:03 smithi122 ceph-mon[93869]: pgmap v11515: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 386 KiB/s rd, 407 KiB/s wr, 13 op/s 2023-03-09T20:42:04.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:03 smithi181 ceph-mon[94189]: pgmap v11515: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 386 KiB/s rd, 407 KiB/s wr, 13 op/s 2023-03-09T20:42:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:05 smithi122 ceph-mon[93869]: pgmap v11516: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 412 KiB/s wr, 24 op/s 2023-03-09T20:42:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20636.279154 secs 2023-03-09T20:42:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:05 smithi181 ceph-mon[94189]: pgmap v11516: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 412 KiB/s wr, 24 op/s 2023-03-09T20:42:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20636.279154 secs 2023-03-09T20:42:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:05 smithi038 ceph-mon[92422]: pgmap v11516: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 412 KiB/s wr, 24 op/s 2023-03-09T20:42:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20636.279154 secs 2023-03-09T20:42:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:07 smithi038 ceph-mon[92422]: pgmap v11517: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 413 KiB/s wr, 27 op/s 2023-03-09T20:42:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20638.594162 secs 2023-03-09T20:42:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:07 smithi122 ceph-mon[93869]: pgmap v11517: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 413 KiB/s wr, 27 op/s 2023-03-09T20:42:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20638.594162 secs 2023-03-09T20:42:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:07 smithi181 ceph-mon[94189]: pgmap v11517: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 413 KiB/s wr, 27 op/s 2023-03-09T20:42:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20638.594162 secs 2023-03-09T20:42:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21100.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:42:10.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:09 smithi038 ceph-mon[92422]: pgmap v11518: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 410 KiB/s wr, 25 op/s 2023-03-09T20:42:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:09 smithi122 ceph-mon[93869]: pgmap v11518: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 410 KiB/s wr, 25 op/s 2023-03-09T20:42:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:09 smithi181 ceph-mon[94189]: pgmap v11518: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 410 KiB/s wr, 25 op/s 2023-03-09T20:42:11.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:10 smithi038 ceph-mon[92422]: osdmap e933: 12 total, 12 up, 12 in 2023-03-09T20:42:11.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20641.279250 secs 2023-03-09T20:42:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:10 smithi122 ceph-mon[93869]: osdmap e933: 12 total, 12 up, 12 in 2023-03-09T20:42:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20641.279250 secs 2023-03-09T20:42:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:10 smithi181 ceph-mon[94189]: osdmap e933: 12 total, 12 up, 12 in 2023-03-09T20:42:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20641.279250 secs 2023-03-09T20:42:12.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:11 smithi122 ceph-mon[93869]: pgmap v11520: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 280 KiB/s wr, 27 op/s 2023-03-09T20:42:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:11 smithi181 ceph-mon[94189]: pgmap v11520: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 280 KiB/s wr, 27 op/s 2023-03-09T20:42:12.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:11 smithi038 ceph-mon[92422]: pgmap v11520: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 280 KiB/s wr, 27 op/s 2023-03-09T20:42:13.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20643.594236 secs 2023-03-09T20:42:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20643.594236 secs 2023-03-09T20:42:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20643.594236 secs 2023-03-09T20:42:13.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21105.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:42:14.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:13 smithi122 ceph-mon[93869]: pgmap v11521: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 207 KiB/s wr, 24 op/s 2023-03-09T20:42:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:13 smithi181 ceph-mon[94189]: pgmap v11521: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 207 KiB/s wr, 24 op/s 2023-03-09T20:42:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:13 smithi038 ceph-mon[92422]: pgmap v11521: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 207 KiB/s wr, 24 op/s 2023-03-09T20:42:15.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:15 smithi122 ceph-mon[93869]: pgmap v11522: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 200 KiB/s wr, 9 op/s 2023-03-09T20:42:15.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20646.279308 secs 2023-03-09T20:42:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:15 smithi181 ceph-mon[94189]: pgmap v11522: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 200 KiB/s wr, 9 op/s 2023-03-09T20:42:15.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20646.279308 secs 2023-03-09T20:42:15.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:15 smithi038 ceph-mon[92422]: pgmap v11522: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 200 KiB/s wr, 9 op/s 2023-03-09T20:42:15.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20646.279308 secs 2023-03-09T20:42:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:17 smithi038 ceph-mon[92422]: pgmap v11523: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 109 KiB/s wr, 8 op/s 2023-03-09T20:42:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20648.594318 secs 2023-03-09T20:42:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:17 smithi122 ceph-mon[93869]: pgmap v11523: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 109 KiB/s wr, 8 op/s 2023-03-09T20:42:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20648.594318 secs 2023-03-09T20:42:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:17 smithi181 ceph-mon[94189]: pgmap v11523: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 109 KiB/s wr, 8 op/s 2023-03-09T20:42:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20648.594318 secs 2023-03-09T20:42:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21110.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:42:20.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:19 smithi038 ceph-mon[92422]: pgmap v11524: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 110 KiB/s wr, 8 op/s 2023-03-09T20:42:20.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:19 smithi038 ceph-mon[92422]: osdmap e934: 12 total, 12 up, 12 in 2023-03-09T20:42:20.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:19 smithi122 ceph-mon[93869]: pgmap v11524: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 110 KiB/s wr, 8 op/s 2023-03-09T20:42:20.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:19 smithi122 ceph-mon[93869]: osdmap e934: 12 total, 12 up, 12 in 2023-03-09T20:42:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:19 smithi181 ceph-mon[94189]: pgmap v11524: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 110 KiB/s wr, 8 op/s 2023-03-09T20:42:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:19 smithi181 ceph-mon[94189]: osdmap e934: 12 total, 12 up, 12 in 2023-03-09T20:42:21.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20651.279368 secs 2023-03-09T20:42:21.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20651.279368 secs 2023-03-09T20:42:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20651.279368 secs 2023-03-09T20:42:22.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:21 smithi038 ceph-mon[92422]: pgmap v11526: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 111 KiB/s wr, 8 op/s 2023-03-09T20:42:22.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:21 smithi122 ceph-mon[93869]: pgmap v11526: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 111 KiB/s wr, 8 op/s 2023-03-09T20:42:22.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:21 smithi181 ceph-mon[94189]: pgmap v11526: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 111 KiB/s wr, 8 op/s 2023-03-09T20:42:23.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20653.594450 secs 2023-03-09T20:42:23.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20653.594450 secs 2023-03-09T20:42:23.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20653.594450 secs 2023-03-09T20:42:23.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21115.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:42:24.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:23 smithi122 ceph-mon[93869]: pgmap v11527: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 21 KiB/s wr, 7 op/s 2023-03-09T20:42:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:23 smithi181 ceph-mon[94189]: pgmap v11527: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 21 KiB/s wr, 7 op/s 2023-03-09T20:42:24.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:23 smithi038 ceph-mon[92422]: pgmap v11527: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 21 KiB/s wr, 7 op/s 2023-03-09T20:42:25.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:25 smithi122 ceph-mon[93869]: pgmap v11528: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 2.3 KiB/s wr, 8 op/s 2023-03-09T20:42:25.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20656.279437 secs 2023-03-09T20:42:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:25 smithi181 ceph-mon[94189]: pgmap v11528: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 2.3 KiB/s wr, 8 op/s 2023-03-09T20:42:25.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20656.279437 secs 2023-03-09T20:42:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:25 smithi038 ceph-mon[92422]: pgmap v11528: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 2.3 KiB/s wr, 8 op/s 2023-03-09T20:42:25.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20656.279437 secs 2023-03-09T20:42:28.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:27 smithi038 ceph-mon[92422]: pgmap v11529: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 110 KiB/s wr, 8 op/s 2023-03-09T20:42:28.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20658.594609 secs 2023-03-09T20:42:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:27 smithi122 ceph-mon[93869]: pgmap v11529: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 110 KiB/s wr, 8 op/s 2023-03-09T20:42:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20658.594609 secs 2023-03-09T20:42:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:27 smithi181 ceph-mon[94189]: pgmap v11529: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 110 KiB/s wr, 8 op/s 2023-03-09T20:42:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20658.594609 secs 2023-03-09T20:42:28.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21120.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:42:29.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:28 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:42:29.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:28 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:42:29.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:28 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:42:29.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:28 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:42:29.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:28 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:42:29.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:28 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:42:30.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:29 smithi122 ceph-mon[93869]: pgmap v11530: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 109 KiB/s wr, 8 op/s 2023-03-09T20:42:30.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:29 smithi181 ceph-mon[94189]: pgmap v11530: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 109 KiB/s wr, 8 op/s 2023-03-09T20:42:30.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:29 smithi038 ceph-mon[92422]: pgmap v11530: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 109 KiB/s wr, 8 op/s 2023-03-09T20:42:31.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20661.279523 secs 2023-03-09T20:42:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20661.279523 secs 2023-03-09T20:42:31.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20661.279523 secs 2023-03-09T20:42:32.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:31 smithi122 ceph-mon[93869]: pgmap v11531: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 152 KiB/s wr, 9 op/s 2023-03-09T20:42:32.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:31 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:42:32.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:31 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:42:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:31 smithi181 ceph-mon[94189]: pgmap v11531: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 152 KiB/s wr, 9 op/s 2023-03-09T20:42:32.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:31 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:42:32.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:31 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:42:32.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:31 smithi038 ceph-mon[92422]: pgmap v11531: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 152 KiB/s wr, 9 op/s 2023-03-09T20:42:32.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:31 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:42:32.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:31 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:42:33.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20663.594690 secs 2023-03-09T20:42:33.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20663.594690 secs 2023-03-09T20:42:33.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20663.594690 secs 2023-03-09T20:42:33.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21125.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:42:34.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:33 smithi122 ceph-mon[93869]: pgmap v11532: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 145 KiB/s wr, 9 op/s 2023-03-09T20:42:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:33 smithi181 ceph-mon[94189]: pgmap v11532: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 145 KiB/s wr, 9 op/s 2023-03-09T20:42:34.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:33 smithi038 ceph-mon[92422]: pgmap v11532: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 145 KiB/s wr, 9 op/s 2023-03-09T20:42:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:35 smithi122 ceph-mon[93869]: pgmap v11533: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 145 KiB/s wr, 8 op/s 2023-03-09T20:42:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20666.279581 secs 2023-03-09T20:42:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:35 smithi181 ceph-mon[94189]: pgmap v11533: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 145 KiB/s wr, 8 op/s 2023-03-09T20:42:35.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20666.279581 secs 2023-03-09T20:42:35.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:35 smithi038 ceph-mon[92422]: pgmap v11533: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 145 KiB/s wr, 8 op/s 2023-03-09T20:42:35.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20666.279581 secs 2023-03-09T20:42:38.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:37 smithi038 ceph-mon[92422]: pgmap v11534: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 144 KiB/s wr, 9 op/s 2023-03-09T20:42:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:37 smithi122 ceph-mon[93869]: pgmap v11534: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 144 KiB/s wr, 9 op/s 2023-03-09T20:42:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:37 smithi181 ceph-mon[94189]: pgmap v11534: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 144 KiB/s wr, 9 op/s 2023-03-09T20:42:38.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21130.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:42:40.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:39 smithi038 ceph-mon[92422]: pgmap v11535: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 54 KiB/s wr, 7 op/s 2023-03-09T20:42:40.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:39 smithi122 ceph-mon[93869]: pgmap v11535: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 54 KiB/s wr, 7 op/s 2023-03-09T20:42:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:39 smithi181 ceph-mon[94189]: pgmap v11535: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 54 KiB/s wr, 7 op/s 2023-03-09T20:42:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20671.279642 secs 2023-03-09T20:42:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:40 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:42:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20671.279642 secs 2023-03-09T20:42:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:40 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:42:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20671.279642 secs 2023-03-09T20:42:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:40 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:42:42.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:41 smithi122 ceph-mon[93869]: pgmap v11536: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 140 KiB/s wr, 8 op/s 2023-03-09T20:42:42.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:41 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:42:42.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:42:42.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:42:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:41 smithi181 ceph-mon[94189]: pgmap v11536: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 140 KiB/s wr, 8 op/s 2023-03-09T20:42:42.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:41 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:42:42.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:42:42.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:42:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:41 smithi038 ceph-mon[92422]: pgmap v11536: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 140 KiB/s wr, 8 op/s 2023-03-09T20:42:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:41 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:42:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:42:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:42:43.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21135.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:42:44.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:43 smithi122 ceph-mon[93869]: pgmap v11537: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 86 KiB/s wr, 7 op/s 2023-03-09T20:42:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:43 smithi181 ceph-mon[94189]: pgmap v11537: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 86 KiB/s wr, 7 op/s 2023-03-09T20:42:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:43 smithi038 ceph-mon[92422]: pgmap v11537: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 86 KiB/s wr, 7 op/s 2023-03-09T20:42:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:45 smithi122 ceph-mon[93869]: pgmap v11538: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 86 KiB/s wr, 6 op/s 2023-03-09T20:42:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20676.279703 secs 2023-03-09T20:42:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:45 smithi181 ceph-mon[94189]: pgmap v11538: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 86 KiB/s wr, 6 op/s 2023-03-09T20:42:45.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20676.279703 secs 2023-03-09T20:42:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:45 smithi038 ceph-mon[92422]: pgmap v11538: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 86 KiB/s wr, 6 op/s 2023-03-09T20:42:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20676.279703 secs 2023-03-09T20:42:48.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:47 smithi038 ceph-mon[92422]: pgmap v11539: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 237 KiB/s wr, 9 op/s 2023-03-09T20:42:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:47 smithi122 ceph-mon[93869]: pgmap v11539: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 237 KiB/s wr, 9 op/s 2023-03-09T20:42:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:47 smithi181 ceph-mon[94189]: pgmap v11539: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 237 KiB/s wr, 9 op/s 2023-03-09T20:42:48.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21140.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:42:50.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:49 smithi038 ceph-mon[92422]: pgmap v11540: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 254 KiB/s wr, 7 op/s 2023-03-09T20:42:50.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:49 smithi122 ceph-mon[93869]: pgmap v11540: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 254 KiB/s wr, 7 op/s 2023-03-09T20:42:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:49 smithi181 ceph-mon[94189]: pgmap v11540: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 254 KiB/s wr, 7 op/s 2023-03-09T20:42:51.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20681.279778 secs 2023-03-09T20:42:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20681.279778 secs 2023-03-09T20:42:51.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20681.279778 secs 2023-03-09T20:42:52.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:51 smithi122 ceph-mon[93869]: pgmap v11541: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 351 KiB/s wr, 9 op/s 2023-03-09T20:42:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:51 smithi181 ceph-mon[94189]: pgmap v11541: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 351 KiB/s wr, 9 op/s 2023-03-09T20:42:52.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:51 smithi038 ceph-mon[92422]: pgmap v11541: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 351 KiB/s wr, 9 op/s 2023-03-09T20:42:53.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:52 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 20683.595176 secs 2023-03-09T20:42:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:52 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 20683.595176 secs 2023-03-09T20:42:53.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:52 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 20683.595176 secs 2023-03-09T20:42:53.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21145.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:42:54.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:53 smithi122 ceph-mon[93869]: pgmap v11542: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 265 KiB/s wr, 9 op/s 2023-03-09T20:42:54.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:53 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:42:54.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:42:54.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:42:54.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:42:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:53 smithi181 ceph-mon[94189]: pgmap v11542: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 265 KiB/s wr, 9 op/s 2023-03-09T20:42:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:53 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:42:54.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:42:54.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:42:54.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:42:54.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:53 smithi038 ceph-mon[92422]: pgmap v11542: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 265 KiB/s wr, 9 op/s 2023-03-09T20:42:54.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:53 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:42:54.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:42:54.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:42:54.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:42:55.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:55 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:42:55.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:55 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:42:55.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:55 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:42:55.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:55 smithi122 ceph-mon[93869]: pgmap v11543: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 281 KiB/s wr, 9 op/s 2023-03-09T20:42:55.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20686.279880 secs 2023-03-09T20:42:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:55 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:42:55.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:55 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:42:55.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:55 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:42:55.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:55 smithi181 ceph-mon[94189]: pgmap v11543: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 281 KiB/s wr, 9 op/s 2023-03-09T20:42:55.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20686.279880 secs 2023-03-09T20:42:55.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:55 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:42:55.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:55 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:42:55.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:55 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:42:55.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:55 smithi038 ceph-mon[92422]: pgmap v11543: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 281 KiB/s wr, 9 op/s 2023-03-09T20:42:55.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20686.279880 secs 2023-03-09T20:42:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:42:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:42:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:42:56.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:42:56.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:42:56.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:42:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:42:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:42:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:57 smithi122 ceph-mon[93869]: pgmap v11544: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 520 KiB/s rd, 504 KiB/s wr, 12 op/s 2023-03-09T20:42:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:57 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 20688.595291 secs 2023-03-09T20:42:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:42:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:42:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:57 smithi181 ceph-mon[94189]: pgmap v11544: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 520 KiB/s rd, 504 KiB/s wr, 12 op/s 2023-03-09T20:42:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:57 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 20688.595291 secs 2023-03-09T20:42:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:42:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:42:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:57 smithi038 ceph-mon[92422]: pgmap v11544: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 520 KiB/s rd, 504 KiB/s wr, 12 op/s 2023-03-09T20:42:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:57 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 20688.595291 secs 2023-03-09T20:42:58.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21150.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:43:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:42:59 smithi122 ceph-mon[93869]: pgmap v11545: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 370 KiB/s wr, 10 op/s 2023-03-09T20:43:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:42:59 smithi181 ceph-mon[94189]: pgmap v11545: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 370 KiB/s wr, 10 op/s 2023-03-09T20:43:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:42:59 smithi038 ceph-mon[92422]: pgmap v11545: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 370 KiB/s wr, 10 op/s 2023-03-09T20:43:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20691.279943 secs 2023-03-09T20:43:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20691.279943 secs 2023-03-09T20:43:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20691.279943 secs 2023-03-09T20:43:02.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:01 smithi122 ceph-mon[93869]: pgmap v11546: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 416 KiB/s wr, 11 op/s 2023-03-09T20:43:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:01 smithi181 ceph-mon[94189]: pgmap v11546: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 416 KiB/s wr, 11 op/s 2023-03-09T20:43:02.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:01 smithi038 ceph-mon[92422]: pgmap v11546: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 416 KiB/s wr, 11 op/s 2023-03-09T20:43:03.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:02 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 20693.595385 secs 2023-03-09T20:43:03.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:02 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 20693.595385 secs 2023-03-09T20:43:03.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:02 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 20693.595385 secs 2023-03-09T20:43:03.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21155.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:43:04.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:03 smithi122 ceph-mon[93869]: pgmap v11547: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 319 KiB/s wr, 10 op/s 2023-03-09T20:43:04.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:03 smithi181 ceph-mon[94189]: pgmap v11547: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 319 KiB/s wr, 10 op/s 2023-03-09T20:43:04.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:03 smithi038 ceph-mon[92422]: pgmap v11547: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 319 KiB/s wr, 10 op/s 2023-03-09T20:43:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:05 smithi122 ceph-mon[93869]: pgmap v11548: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 507 KiB/s wr, 21 op/s 2023-03-09T20:43:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20696.279986 secs 2023-03-09T20:43:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:05 smithi181 ceph-mon[94189]: pgmap v11548: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 507 KiB/s wr, 21 op/s 2023-03-09T20:43:05.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20696.279986 secs 2023-03-09T20:43:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:05 smithi038 ceph-mon[92422]: pgmap v11548: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 507 KiB/s wr, 21 op/s 2023-03-09T20:43:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20696.279986 secs 2023-03-09T20:43:07.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:43:07.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:43:07.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:07 smithi122 ceph-mon[93869]: pgmap v11549: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 562 KiB/s rd, 502 KiB/s wr, 24 op/s 2023-03-09T20:43:07.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20698.595500 secs 2023-03-09T20:43:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:43:07.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:43:07.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:07 smithi181 ceph-mon[94189]: pgmap v11549: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 562 KiB/s rd, 502 KiB/s wr, 24 op/s 2023-03-09T20:43:07.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20698.595500 secs 2023-03-09T20:43:07.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:43:07.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:43:07.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:07 smithi038 ceph-mon[92422]: pgmap v11549: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 562 KiB/s rd, 502 KiB/s wr, 24 op/s 2023-03-09T20:43:07.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20698.595500 secs 2023-03-09T20:43:08.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21160.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:43:09.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:43:09.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:43:09.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:43:09.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:09 smithi122 ceph-mon[93869]: pgmap v11550: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 403 KiB/s wr, 22 op/s 2023-03-09T20:43:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:43:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:43:09.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:43:09.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:09 smithi181 ceph-mon[94189]: pgmap v11550: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 403 KiB/s wr, 22 op/s 2023-03-09T20:43:09.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:43:09.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:43:09.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:43:09.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:09 smithi038 ceph-mon[92422]: pgmap v11550: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 403 KiB/s wr, 22 op/s 2023-03-09T20:43:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20701.280048 secs 2023-03-09T20:43:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20701.280048 secs 2023-03-09T20:43:10.769 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20701.280048 secs 2023-03-09T20:43:11.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:11 smithi122 ceph-mon[93869]: pgmap v11551: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 446 KiB/s rd, 386 KiB/s wr, 22 op/s 2023-03-09T20:43:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:11 smithi181 ceph-mon[94189]: pgmap v11551: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 446 KiB/s rd, 386 KiB/s wr, 22 op/s 2023-03-09T20:43:11.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:11 smithi038 ceph-mon[92422]: pgmap v11551: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 446 KiB/s rd, 386 KiB/s wr, 22 op/s 2023-03-09T20:43:12.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20703.595630 secs 2023-03-09T20:43:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20703.595630 secs 2023-03-09T20:43:12.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20703.595630 secs 2023-03-09T20:43:13.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:13 smithi122 ceph-mon[93869]: pgmap v11552: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 384 KiB/s rd, 325 KiB/s wr, 22 op/s 2023-03-09T20:43:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:13 smithi181 ceph-mon[94189]: pgmap v11552: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 384 KiB/s rd, 325 KiB/s wr, 22 op/s 2023-03-09T20:43:13.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:13 smithi038 ceph-mon[92422]: pgmap v11552: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 384 KiB/s rd, 325 KiB/s wr, 22 op/s 2023-03-09T20:43:13.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21165.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:43:15.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:15 smithi122 ceph-mon[93869]: pgmap v11553: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 453 KiB/s rd, 379 KiB/s wr, 22 op/s 2023-03-09T20:43:15.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20706.280108 secs 2023-03-09T20:43:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:15 smithi181 ceph-mon[94189]: pgmap v11553: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 453 KiB/s rd, 379 KiB/s wr, 22 op/s 2023-03-09T20:43:15.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20706.280108 secs 2023-03-09T20:43:15.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:15 smithi038 ceph-mon[92422]: pgmap v11553: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 453 KiB/s rd, 379 KiB/s wr, 22 op/s 2023-03-09T20:43:15.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20706.280108 secs 2023-03-09T20:43:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:17 smithi122 ceph-mon[93869]: pgmap v11554: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 190 KiB/s wr, 11 op/s 2023-03-09T20:43:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20708.595756 secs 2023-03-09T20:43:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:17 smithi181 ceph-mon[94189]: pgmap v11554: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 190 KiB/s wr, 11 op/s 2023-03-09T20:43:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20708.595756 secs 2023-03-09T20:43:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:17 smithi038 ceph-mon[92422]: pgmap v11554: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 190 KiB/s wr, 11 op/s 2023-03-09T20:43:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20708.595756 secs 2023-03-09T20:43:18.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21170.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:43:20.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:19 smithi122 ceph-mon[93869]: pgmap v11555: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 197 KiB/s wr, 9 op/s 2023-03-09T20:43:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:19 smithi181 ceph-mon[94189]: pgmap v11555: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 197 KiB/s wr, 9 op/s 2023-03-09T20:43:20.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:19 smithi038 ceph-mon[92422]: pgmap v11555: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 197 KiB/s wr, 9 op/s 2023-03-09T20:43:21.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20711.280171 secs 2023-03-09T20:43:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20711.280171 secs 2023-03-09T20:43:21.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20711.280171 secs 2023-03-09T20:43:22.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:21 smithi122 ceph-mon[93869]: pgmap v11556: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 217 KiB/s wr, 8 op/s 2023-03-09T20:43:22.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:21 smithi181 ceph-mon[94189]: pgmap v11556: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 217 KiB/s wr, 8 op/s 2023-03-09T20:43:22.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:21 smithi038 ceph-mon[92422]: pgmap v11556: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 217 KiB/s wr, 8 op/s 2023-03-09T20:43:23.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20713.595869 secs 2023-03-09T20:43:23.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20713.595869 secs 2023-03-09T20:43:23.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20713.595869 secs 2023-03-09T20:43:23.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21175.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:43:24.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:23 smithi122 ceph-mon[93869]: pgmap v11557: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 228 KiB/s wr, 9 op/s 2023-03-09T20:43:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:23 smithi181 ceph-mon[94189]: pgmap v11557: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 228 KiB/s wr, 9 op/s 2023-03-09T20:43:24.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:23 smithi038 ceph-mon[92422]: pgmap v11557: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 228 KiB/s wr, 9 op/s 2023-03-09T20:43:25.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:25 smithi122 ceph-mon[93869]: pgmap v11558: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 255 KiB/s wr, 10 op/s 2023-03-09T20:43:25.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20716.280230 secs 2023-03-09T20:43:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:25 smithi181 ceph-mon[94189]: pgmap v11558: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 255 KiB/s wr, 10 op/s 2023-03-09T20:43:25.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20716.280230 secs 2023-03-09T20:43:25.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:25 smithi038 ceph-mon[92422]: pgmap v11558: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 255 KiB/s wr, 10 op/s 2023-03-09T20:43:25.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20716.280230 secs 2023-03-09T20:43:27.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:27 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:43:27.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:27 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:43:27.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:27 smithi122 ceph-mon[93869]: pgmap v11559: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 267 KiB/s wr, 12 op/s 2023-03-09T20:43:27.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20718.596001 secs 2023-03-09T20:43:27.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:27 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:43:27.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:27 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:43:27.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:27 smithi181 ceph-mon[94189]: pgmap v11559: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 267 KiB/s wr, 12 op/s 2023-03-09T20:43:27.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20718.596001 secs 2023-03-09T20:43:27.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:27 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:43:27.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:27 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:43:27.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:27 smithi038 ceph-mon[92422]: pgmap v11559: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 267 KiB/s wr, 12 op/s 2023-03-09T20:43:27.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20718.596001 secs 2023-03-09T20:43:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21180.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:43:29.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:29 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:43:29.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:29 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:43:29.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:29 smithi122 ceph-mon[93869]: pgmap v11560: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 268 KiB/s wr, 11 op/s 2023-03-09T20:43:29.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:29 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:43:29.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:29 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:43:29.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:29 smithi181 ceph-mon[94189]: pgmap v11560: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 268 KiB/s wr, 11 op/s 2023-03-09T20:43:29.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:29 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:43:29.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:29 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:43:29.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:29 smithi038 ceph-mon[92422]: pgmap v11560: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 268 KiB/s wr, 11 op/s 2023-03-09T20:43:30.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:30 smithi122 ceph-mon[93869]: osdmap e935: 12 total, 12 up, 12 in 2023-03-09T20:43:30.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20721.282785 secs 2023-03-09T20:43:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:30 smithi181 ceph-mon[94189]: osdmap e935: 12 total, 12 up, 12 in 2023-03-09T20:43:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20721.282785 secs 2023-03-09T20:43:30.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:30 smithi038 ceph-mon[92422]: osdmap e935: 12 total, 12 up, 12 in 2023-03-09T20:43:30.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20721.282785 secs 2023-03-09T20:43:31.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:31 smithi122 ceph-mon[93869]: pgmap v11562: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 252 KiB/s wr, 11 op/s 2023-03-09T20:43:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:31 smithi181 ceph-mon[94189]: pgmap v11562: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 252 KiB/s wr, 11 op/s 2023-03-09T20:43:31.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:31 smithi038 ceph-mon[92422]: pgmap v11562: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 252 KiB/s wr, 11 op/s 2023-03-09T20:43:32.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20723.596095 secs 2023-03-09T20:43:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20723.596095 secs 2023-03-09T20:43:32.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20723.596095 secs 2023-03-09T20:43:33.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:33 smithi122 ceph-mon[93869]: pgmap v11563: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 239 KiB/s wr, 11 op/s 2023-03-09T20:43:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:33 smithi181 ceph-mon[94189]: pgmap v11563: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 239 KiB/s wr, 11 op/s 2023-03-09T20:43:33.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:33 smithi038 ceph-mon[92422]: pgmap v11563: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 239 KiB/s wr, 11 op/s 2023-03-09T20:43:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21185.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:43:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:35 smithi122 ceph-mon[93869]: pgmap v11564: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 207 KiB/s wr, 9 op/s 2023-03-09T20:43:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20726.282846 secs 2023-03-09T20:43:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:35 smithi181 ceph-mon[94189]: pgmap v11564: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 207 KiB/s wr, 9 op/s 2023-03-09T20:43:35.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20726.282846 secs 2023-03-09T20:43:35.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:35 smithi038 ceph-mon[92422]: pgmap v11564: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 207 KiB/s wr, 9 op/s 2023-03-09T20:43:35.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20726.282846 secs 2023-03-09T20:43:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:37 smithi122 ceph-mon[93869]: pgmap v11565: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 193 KiB/s wr, 9 op/s 2023-03-09T20:43:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20728.596215 secs 2023-03-09T20:43:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:37 smithi181 ceph-mon[94189]: pgmap v11565: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 193 KiB/s wr, 9 op/s 2023-03-09T20:43:38.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20728.596215 secs 2023-03-09T20:43:38.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:37 smithi038 ceph-mon[92422]: pgmap v11565: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 193 KiB/s wr, 9 op/s 2023-03-09T20:43:38.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20728.596215 secs 2023-03-09T20:43:38.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21190.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:43:40.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:39 smithi122 ceph-mon[93869]: pgmap v11566: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 193 KiB/s wr, 8 op/s 2023-03-09T20:43:40.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:39 smithi122 ceph-mon[93869]: osdmap e936: 12 total, 12 up, 12 in 2023-03-09T20:43:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:39 smithi181 ceph-mon[94189]: pgmap v11566: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 193 KiB/s wr, 8 op/s 2023-03-09T20:43:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:39 smithi181 ceph-mon[94189]: osdmap e936: 12 total, 12 up, 12 in 2023-03-09T20:43:40.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:39 smithi038 ceph-mon[92422]: pgmap v11566: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 193 KiB/s wr, 8 op/s 2023-03-09T20:43:40.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:39 smithi038 ceph-mon[92422]: osdmap e936: 12 total, 12 up, 12 in 2023-03-09T20:43:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20731.282918 secs 2023-03-09T20:43:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:43:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:43:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20731.282918 secs 2023-03-09T20:43:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:43:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:43:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20731.282918 secs 2023-03-09T20:43:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:43:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:43:42.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:41 smithi122 ceph-mon[93869]: pgmap v11568: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 314 KiB/s wr, 9 op/s 2023-03-09T20:43:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:41 smithi181 ceph-mon[94189]: pgmap v11568: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 314 KiB/s wr, 9 op/s 2023-03-09T20:43:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:41 smithi038 ceph-mon[92422]: pgmap v11568: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 314 KiB/s wr, 9 op/s 2023-03-09T20:43:43.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20733.596308 secs 2023-03-09T20:43:43.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20733.596308 secs 2023-03-09T20:43:43.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20733.596308 secs 2023-03-09T20:43:43.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21195.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:43:44.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:43 smithi122 ceph-mon[93869]: pgmap v11569: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 313 KiB/s wr, 9 op/s 2023-03-09T20:43:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:43 smithi181 ceph-mon[94189]: pgmap v11569: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 313 KiB/s wr, 9 op/s 2023-03-09T20:43:44.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:43 smithi038 ceph-mon[92422]: pgmap v11569: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 313 KiB/s wr, 9 op/s 2023-03-09T20:43:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:45 smithi122 ceph-mon[93869]: pgmap v11570: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 333 KiB/s wr, 9 op/s 2023-03-09T20:43:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20736.282982 secs 2023-03-09T20:43:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:45 smithi181 ceph-mon[94189]: pgmap v11570: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 333 KiB/s wr, 9 op/s 2023-03-09T20:43:45.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20736.282982 secs 2023-03-09T20:43:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:45 smithi038 ceph-mon[92422]: pgmap v11570: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 333 KiB/s wr, 9 op/s 2023-03-09T20:43:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20736.282982 secs 2023-03-09T20:43:47.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:43:47.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:43:47.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:47 smithi122 ceph-mon[93869]: pgmap v11571: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 398 KiB/s wr, 9 op/s 2023-03-09T20:43:47.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20738.596410 secs 2023-03-09T20:43:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:43:47.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:43:47.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:47 smithi181 ceph-mon[94189]: pgmap v11571: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 398 KiB/s wr, 9 op/s 2023-03-09T20:43:47.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20738.596410 secs 2023-03-09T20:43:47.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:43:47.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:43:47.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:47 smithi038 ceph-mon[92422]: pgmap v11571: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 398 KiB/s wr, 9 op/s 2023-03-09T20:43:47.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20738.596410 secs 2023-03-09T20:43:48.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21200.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:43:50.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:49 smithi122 ceph-mon[93869]: pgmap v11572: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 418 KiB/s wr, 10 op/s 2023-03-09T20:43:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:49 smithi181 ceph-mon[94189]: pgmap v11572: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 418 KiB/s wr, 10 op/s 2023-03-09T20:43:50.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:49 smithi038 ceph-mon[92422]: pgmap v11572: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 418 KiB/s wr, 10 op/s 2023-03-09T20:43:51.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20741.283042 secs 2023-03-09T20:43:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20741.283042 secs 2023-03-09T20:43:51.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20741.283042 secs 2023-03-09T20:43:52.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:51 smithi122 ceph-mon[93869]: pgmap v11573: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 384 KiB/s rd, 366 KiB/s wr, 11 op/s 2023-03-09T20:43:52.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:43:52.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:43:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:51 smithi181 ceph-mon[94189]: pgmap v11573: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 384 KiB/s rd, 366 KiB/s wr, 11 op/s 2023-03-09T20:43:52.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:43:52.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:43:52.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:51 smithi038 ceph-mon[92422]: pgmap v11573: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 384 KiB/s rd, 366 KiB/s wr, 11 op/s 2023-03-09T20:43:52.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:43:52.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:43:53.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:43:53.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:43:53.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20743.596500 secs 2023-03-09T20:43:53.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:43:53.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:43:53.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20743.596500 secs 2023-03-09T20:43:53.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:43:53.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:43:53.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20743.596500 secs 2023-03-09T20:43:53.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21205.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:43:54.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:53 smithi122 ceph-mon[93869]: pgmap v11574: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 335 KiB/s wr, 10 op/s 2023-03-09T20:43:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:53 smithi181 ceph-mon[94189]: pgmap v11574: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 335 KiB/s wr, 10 op/s 2023-03-09T20:43:54.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:53 smithi038 ceph-mon[92422]: pgmap v11574: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 335 KiB/s wr, 10 op/s 2023-03-09T20:43:55.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:55 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:43:55.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:55 smithi122 ceph-mon[93869]: pgmap v11575: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 352 KiB/s wr, 11 op/s 2023-03-09T20:43:55.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20746.283101 secs 2023-03-09T20:43:55.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:55 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:43:55.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:55 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:43:55.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:55 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:43:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:55 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:43:55.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:55 smithi181 ceph-mon[94189]: pgmap v11575: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 352 KiB/s wr, 11 op/s 2023-03-09T20:43:55.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20746.283101 secs 2023-03-09T20:43:55.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:55 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:43:55.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:55 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:43:55.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:55 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:43:55.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:55 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:43:55.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:55 smithi038 ceph-mon[92422]: pgmap v11575: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 352 KiB/s wr, 11 op/s 2023-03-09T20:43:55.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20746.283101 secs 2023-03-09T20:43:55.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:55 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:43:55.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:55 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:43:55.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:55 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:43:58.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:57 smithi122 ceph-mon[93869]: pgmap v11576: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 498 KiB/s wr, 13 op/s 2023-03-09T20:43:58.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20748.596604 secs 2023-03-09T20:43:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:57 smithi181 ceph-mon[94189]: pgmap v11576: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 498 KiB/s wr, 13 op/s 2023-03-09T20:43:58.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20748.596604 secs 2023-03-09T20:43:58.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:57 smithi038 ceph-mon[92422]: pgmap v11576: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 498 KiB/s wr, 13 op/s 2023-03-09T20:43:58.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20748.596604 secs 2023-03-09T20:43:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21210.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:44:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:59 smithi122 ceph-mon[93869]: pgmap v11577: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 421 KiB/s wr, 11 op/s 2023-03-09T20:44:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:44:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:43:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:44:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:59 smithi181 ceph-mon[94189]: pgmap v11577: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 421 KiB/s wr, 11 op/s 2023-03-09T20:44:00.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:44:00.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:43:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:44:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:59 smithi038 ceph-mon[92422]: pgmap v11577: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 421 KiB/s wr, 11 op/s 2023-03-09T20:44:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:44:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:43:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:44:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:44:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20751.283159 secs 2023-03-09T20:44:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:44:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:44:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:44:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:44:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:44:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20751.283159 secs 2023-03-09T20:44:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:44:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:44:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:44:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:44:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:44:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20751.283159 secs 2023-03-09T20:44:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:44:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:44:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:44:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:44:02.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:44:01 smithi122 ceph-mon[93869]: pgmap v11578: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 570 KiB/s rd, 575 KiB/s wr, 12 op/s 2023-03-09T20:44:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:44:01 smithi181 ceph-mon[94189]: pgmap v11578: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 570 KiB/s rd, 575 KiB/s wr, 12 op/s 2023-03-09T20:44:02.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:44:01 smithi038 ceph-mon[92422]: pgmap v11578: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 570 KiB/s rd, 575 KiB/s wr, 12 op/s 2023-03-09T20:44:03.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:44:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20753.596711 secs 2023-03-09T20:44:03.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:44:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20753.596711 secs 2023-03-09T20:44:03.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:44:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20753.596711 secs 2023-03-09T20:44:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21215.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:44:04.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:44:03 smithi122 ceph-mon[93869]: pgmap v11579: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 561 KiB/s rd, 597 KiB/s wr, 14 op/s 2023-03-09T20:44:04.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:44:03 smithi181 ceph-mon[94189]: pgmap v11579: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 561 KiB/s rd, 597 KiB/s wr, 14 op/s 2023-03-09T20:44:04.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:44:03 smithi038 ceph-mon[92422]: pgmap v11579: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 561 KiB/s rd, 597 KiB/s wr, 14 op/s 2023-03-09T20:44:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:44:05 smithi122 ceph-mon[93869]: pgmap v11580: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 427 KiB/s wr, 24 op/s 2023-03-09T20:44:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:44:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20756.283220 secs 2023-03-09T20:44:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:44:05 smithi181 ceph-mon[94189]: pgmap v11580: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 427 KiB/s wr, 24 op/s 2023-03-09T20:44:05.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:44:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20756.283220 secs 2023-03-09T20:44:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:44:05 smithi038 ceph-mon[92422]: pgmap v11580: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 427 KiB/s wr, 24 op/s 2023-03-09T20:44:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:44:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20756.283220 secs 2023-03-09T20:44:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:44:07 smithi122 ceph-mon[93869]: pgmap v11581: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 443 KiB/s rd, 477 KiB/s wr, 26 op/s 2023-03-09T20:44:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:44:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20758.596845 secs 2023-03-09T20:44:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:44:07 smithi181 ceph-mon[94189]: pgmap v11581: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 443 KiB/s rd, 477 KiB/s wr, 26 op/s 2023-03-09T20:44:08.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:44:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20758.596845 secs 2023-03-09T20:44:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:44:07 smithi038 ceph-mon[92422]: pgmap v11581: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 443 KiB/s rd, 477 KiB/s wr, 26 op/s 2023-03-09T20:44:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:44:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20758.596845 secs 2023-03-09T20:44:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21220.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:44:10.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:44:09 smithi122 ceph-mon[93869]: pgmap v11582: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 331 KiB/s wr, 23 op/s 2023-03-09T20:44:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:44:09 smithi181 ceph-mon[94189]: pgmap v11582: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 331 KiB/s wr, 23 op/s 2023-03-09T20:44:10.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:44:09 smithi038 ceph-mon[92422]: pgmap v11582: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 331 KiB/s wr, 23 op/s 2023-03-09T20:44:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:44:10 smithi122 ceph-mon[93869]: osdmap e937: 12 total, 12 up, 12 in 2023-03-09T20:44:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:44:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20761.283296 secs 2023-03-09T20:44:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:44:10 smithi181 ceph-mon[94189]: osdmap e937: 12 total, 12 up, 12 in 2023-03-09T20:44:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:44:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20761.283296 secs 2023-03-09T20:44:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:44:10 smithi038 ceph-mon[92422]: osdmap e937: 12 total, 12 up, 12 in 2023-03-09T20:44:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:44:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20761.283296 secs 2023-03-09T20:44:12.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:44:11 smithi122 ceph-mon[93869]: pgmap v11584: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 153 KiB/s wr, 26 op/s 2023-03-09T20:44:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:44:11 smithi181 ceph-mon[94189]: pgmap v11584: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 153 KiB/s wr, 26 op/s 2023-03-09T20:44:12.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:44:11 smithi038 ceph-mon[92422]: pgmap v11584: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 153 KiB/s wr, 26 op/s 2023-03-09T20:44:13.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:44:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20763.596979 secs 2023-03-09T20:44:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:44:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20763.596979 secs 2023-03-09T20:44:13.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:44:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20763.596979 secs 2023-03-09T20:44:13.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21225.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:44:14.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:44:13 smithi122 ceph-mon[93869]: pgmap v11585: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 126 KiB/s wr, 24 op/s 2023-03-09T20:44:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:44:13 smithi181 ceph-mon[94189]: pgmap v11585: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 126 KiB/s wr, 24 op/s 2023-03-09T20:44:14.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:44:13 smithi038 ceph-mon[92422]: pgmap v11585: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 126 KiB/s wr, 24 op/s 2023-03-09T20:44:15.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:44:15 smithi122 ceph-mon[93869]: pgmap v11586: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 120 KiB/s wr, 9 op/s 2023-03-09T20:44:15.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:44:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20766.283359 secs 2023-03-09T20:44:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:44:15 smithi181 ceph-mon[94189]: pgmap v11586: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 120 KiB/s wr, 9 op/s 2023-03-09T20:44:15.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:44:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20766.283359 secs 2023-03-09T20:44:15.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:44:15 smithi038 ceph-mon[92422]: pgmap v11586: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 120 KiB/s wr, 9 op/s 2023-03-09T20:44:15.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:44:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20766.283359 secs 2023-03-09T20:44:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:44:17 smithi122 ceph-mon[93869]: pgmap v11587: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 118 KiB/s wr, 9 op/s 2023-03-09T20:44:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:44:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20768.597111 secs 2023-03-09T20:44:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:44:17 smithi181 ceph-mon[94189]: pgmap v11587: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 118 KiB/s wr, 9 op/s 2023-03-09T20:44:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:44:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20768.597111 secs 2023-03-09T20:44:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:44:17 smithi038 ceph-mon[92422]: pgmap v11587: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 118 KiB/s wr, 9 op/s 2023-03-09T20:44:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:44:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20768.597111 secs 2023-03-09T20:44:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21230.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:44:20.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:44:19 smithi122 ceph-mon[93869]: pgmap v11588: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 99 KiB/s wr, 9 op/s 2023-03-09T20:44:20.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:44:19 smithi122 ceph-mon[93869]: osdmap e938: 12 total, 12 up, 12 in 2023-03-09T20:44:20.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:44:19 smithi181 ceph-mon[94189]: pgmap v11588: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 99 KiB/s wr, 9 op/s 2023-03-09T20:44:20.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:44:19 smithi181 ceph-mon[94189]: osdmap e938: 12 total, 12 up, 12 in 2023-03-09T20:44:20.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:44:19 smithi038 ceph-mon[92422]: pgmap v11588: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 99 KiB/s wr, 9 op/s 2023-03-09T20:44:20.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:44:19 smithi038 ceph-mon[92422]: osdmap e938: 12 total, 12 up, 12 in 2023-03-09T20:44:21.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:44:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20771.283433 secs 2023-03-09T20:44:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:44:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20771.283433 secs 2023-03-09T20:44:21.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:44:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20771.283433 secs 2023-03-09T20:44:22.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:44:21 smithi122 ceph-mon[93869]: pgmap v11590: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 100 KiB/s wr, 8 op/s 2023-03-09T20:44:22.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:44:21 smithi181 ceph-mon[94189]: pgmap v11590: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 100 KiB/s wr, 8 op/s 2023-03-09T20:44:22.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:44:21 smithi038 ceph-mon[92422]: pgmap v11590: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 100 KiB/s wr, 8 op/s 2023-03-09T20:44:23.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:44:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20773.597218 secs 2023-03-09T20:44:23.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:44:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20773.597218 secs 2023-03-09T20:44:23.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:44:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20773.597218 secs 2023-03-09T20:44:23.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21235.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:44:24.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:44:23 smithi122 ceph-mon[93869]: pgmap v11591: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 99 KiB/s wr, 8 op/s 2023-03-09T20:44:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:44:23 smithi181 ceph-mon[94189]: pgmap v11591: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 99 KiB/s wr, 8 op/s 2023-03-09T20:44:24.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:44:23 smithi038 ceph-mon[92422]: pgmap v11591: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 99 KiB/s wr, 8 op/s 2023-03-09T20:44:25.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:44:25 smithi122 ceph-mon[93869]: pgmap v11592: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 80 KiB/s wr, 8 op/s 2023-03-09T20:44:25.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:44:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20776.283494 secs 2023-03-09T20:44:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:44:25 smithi181 ceph-mon[94189]: pgmap v11592: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 80 KiB/s wr, 8 op/s 2023-03-09T20:44:25.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:44:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20776.283494 secs 2023-03-09T20:44:25.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:44:25 smithi038 ceph-mon[92422]: pgmap v11592: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 80 KiB/s wr, 8 op/s 2023-03-09T20:44:25.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:44:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20776.283494 secs 2023-03-09T20:44:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:44:27 smithi122 ceph-mon[93869]: pgmap v11593: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 35 KiB/s wr, 7 op/s 2023-03-09T20:44:28.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:44:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20778.597332 secs 2023-03-09T20:44:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:44:27 smithi181 ceph-mon[94189]: pgmap v11593: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 35 KiB/s wr, 7 op/s 2023-03-09T20:44:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:44:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20778.597332 secs 2023-03-09T20:44:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:44:27 smithi038 ceph-mon[92422]: pgmap v11593: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 35 KiB/s wr, 7 op/s 2023-03-09T20:44:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:44:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20778.597332 secs 2023-03-09T20:44:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21240.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:44:30.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:44:29 smithi122 ceph-mon[93869]: pgmap v11594: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 98 KiB/s wr, 8 op/s 2023-03-09T20:44:30.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:44:29 smithi181 ceph-mon[94189]: pgmap v11594: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 98 KiB/s wr, 8 op/s 2023-03-09T20:44:30.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:44:29 smithi038 ceph-mon[92422]: pgmap v11594: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 98 KiB/s wr, 8 op/s 2023-03-09T20:44:31.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:44:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20781.283578 secs 2023-03-09T20:44:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:44:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20781.283578 secs 2023-03-09T20:44:31.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:44:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20781.283578 secs 2023-03-09T20:44:32.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:44:31 smithi122 ceph-mon[93869]: pgmap v11595: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 86 KiB/s wr, 8 op/s 2023-03-09T20:44:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:44:31 smithi181 ceph-mon[94189]: pgmap v11595: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 86 KiB/s wr, 8 op/s 2023-03-09T20:44:32.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:44:31 smithi038 ceph-mon[92422]: pgmap v11595: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 86 KiB/s wr, 8 op/s 2023-03-09T20:44:33.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:44:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20783.597413 secs 2023-03-09T20:44:33.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:44:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20783.597413 secs 2023-03-09T20:44:33.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:44:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20783.597413 secs 2023-03-09T20:44:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21245.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:44:34.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:44:33 smithi122 ceph-mon[93869]: pgmap v11596: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 81 KiB/s wr, 8 op/s 2023-03-09T20:44:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:44:33 smithi181 ceph-mon[94189]: pgmap v11596: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 81 KiB/s wr, 8 op/s 2023-03-09T20:44:34.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:44:33 smithi038 ceph-mon[92422]: pgmap v11596: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 81 KiB/s wr, 8 op/s 2023-03-09T20:44:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:44:35 smithi122 ceph-mon[93869]: pgmap v11597: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 81 KiB/s wr, 8 op/s 2023-03-09T20:44:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:44:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20786.283640 secs 2023-03-09T20:44:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:44:35 smithi181 ceph-mon[94189]: pgmap v11597: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 81 KiB/s wr, 8 op/s 2023-03-09T20:44:35.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:44:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20786.283640 secs 2023-03-09T20:44:35.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:44:35 smithi038 ceph-mon[92422]: pgmap v11597: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 81 KiB/s wr, 8 op/s 2023-03-09T20:44:35.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:44:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20786.283640 secs 2023-03-09T20:44:36.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:44:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:44:36.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:44:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:44:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:44:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:44:36.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:44:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:44:36.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:44:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:44:36.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:44:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:44:37.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:44:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:44:37.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:44:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:44:37.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:44:37 smithi122 ceph-mon[93869]: pgmap v11598: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 81 KiB/s wr, 9 op/s 2023-03-09T20:44:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:44:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:44:37.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:44:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:44:37.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:44:37 smithi181 ceph-mon[94189]: pgmap v11598: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 81 KiB/s wr, 9 op/s 2023-03-09T20:44:37.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:44:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:44:37.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:44:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:44:37.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:44:37 smithi038 ceph-mon[92422]: pgmap v11598: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 81 KiB/s wr, 9 op/s 2023-03-09T20:44:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21250.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:44:40.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:44:39 smithi122 ceph-mon[93869]: pgmap v11599: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 209 KiB/s wr, 8 op/s 2023-03-09T20:44:40.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:44:39 smithi181 ceph-mon[94189]: pgmap v11599: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 209 KiB/s wr, 8 op/s 2023-03-09T20:44:40.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:44:39 smithi038 ceph-mon[92422]: pgmap v11599: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 209 KiB/s wr, 8 op/s 2023-03-09T20:44:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:44:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20791.283724 secs 2023-03-09T20:44:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:44:40 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:44:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:44:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20791.283724 secs 2023-03-09T20:44:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:44:40 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:44:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:44:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20791.283724 secs 2023-03-09T20:44:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:44:40 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:44:42.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:44:41 smithi122 ceph-mon[93869]: pgmap v11600: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 156 KiB/s wr, 8 op/s 2023-03-09T20:44:42.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:44:41 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:44:42.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:44:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:44:42.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:44:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:44:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:44:41 smithi181 ceph-mon[94189]: pgmap v11600: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 156 KiB/s wr, 8 op/s 2023-03-09T20:44:42.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:44:41 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:44:42.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:44:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:44:42.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:44:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:44:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:44:41 smithi038 ceph-mon[92422]: pgmap v11600: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 156 KiB/s wr, 8 op/s 2023-03-09T20:44:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:44:41 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:44:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:44:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:44:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:44:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:44:43.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21255.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:44:44.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:44:43 smithi122 ceph-mon[93869]: pgmap v11601: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 178 KiB/s wr, 8 op/s 2023-03-09T20:44:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:44:43 smithi181 ceph-mon[94189]: pgmap v11601: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 178 KiB/s wr, 8 op/s 2023-03-09T20:44:44.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:44:43 smithi038 ceph-mon[92422]: pgmap v11601: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 178 KiB/s wr, 8 op/s 2023-03-09T20:44:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:44:45 smithi122 ceph-mon[93869]: pgmap v11602: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 178 KiB/s wr, 7 op/s 2023-03-09T20:44:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:44:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20796.283784 secs 2023-03-09T20:44:45.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:44:45 smithi181 ceph-mon[94189]: pgmap v11602: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 178 KiB/s wr, 7 op/s 2023-03-09T20:44:45.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:44:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20796.283784 secs 2023-03-09T20:44:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:44:45 smithi038 ceph-mon[92422]: pgmap v11602: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 178 KiB/s wr, 7 op/s 2023-03-09T20:44:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:44:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20796.283784 secs 2023-03-09T20:44:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:44:47 smithi122 ceph-mon[93869]: pgmap v11603: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 178 KiB/s wr, 8 op/s 2023-03-09T20:44:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:44:47 smithi181 ceph-mon[94189]: pgmap v11603: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 178 KiB/s wr, 8 op/s 2023-03-09T20:44:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:44:47 smithi038 ceph-mon[92422]: pgmap v11603: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 178 KiB/s wr, 8 op/s 2023-03-09T20:44:48.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21260.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:44:50.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:44:49 smithi122 ceph-mon[93869]: pgmap v11604: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 307 KiB/s wr, 8 op/s 2023-03-09T20:44:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:44:49 smithi181 ceph-mon[94189]: pgmap v11604: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 307 KiB/s wr, 8 op/s 2023-03-09T20:44:50.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:44:49 smithi038 ceph-mon[92422]: pgmap v11604: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 307 KiB/s wr, 8 op/s 2023-03-09T20:44:51.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:44:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20801.283855 secs 2023-03-09T20:44:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:44:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20801.283855 secs 2023-03-09T20:44:51.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:44:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20801.283855 secs 2023-03-09T20:44:52.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:44:51 smithi122 ceph-mon[93869]: pgmap v11605: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 192 KiB/s wr, 8 op/s 2023-03-09T20:44:52.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:44:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:44:52.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:44:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:44:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:44:51 smithi181 ceph-mon[94189]: pgmap v11605: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 192 KiB/s wr, 8 op/s 2023-03-09T20:44:52.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:44:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:44:52.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:44:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:44:52.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:44:51 smithi038 ceph-mon[92422]: pgmap v11605: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 192 KiB/s wr, 8 op/s 2023-03-09T20:44:52.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:44:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:44:52.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:44:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:44:53.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:44:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:44:53.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:44:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:44:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:44:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:44:53.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:44:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:44:53.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:44:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:44:53.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:44:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:44:53.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21265.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:44:54.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:44:53 smithi122 ceph-mon[93869]: pgmap v11606: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 192 KiB/s wr, 9 op/s 2023-03-09T20:44:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:44:53 smithi181 ceph-mon[94189]: pgmap v11606: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 192 KiB/s wr, 9 op/s 2023-03-09T20:44:54.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:44:53 smithi038 ceph-mon[92422]: pgmap v11606: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 192 KiB/s wr, 9 op/s 2023-03-09T20:44:55.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:44:55 smithi122 ceph-mon[93869]: pgmap v11607: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 486 KiB/s rd, 471 KiB/s wr, 10 op/s 2023-03-09T20:44:55.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:44:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20806.283914 secs 2023-03-09T20:44:55.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:44:55 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:44:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:44:55 smithi181 ceph-mon[94189]: pgmap v11607: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 486 KiB/s rd, 471 KiB/s wr, 10 op/s 2023-03-09T20:44:55.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:44:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20806.283914 secs 2023-03-09T20:44:55.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:44:55 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:44:55.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:44:55 smithi038 ceph-mon[92422]: pgmap v11607: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 486 KiB/s rd, 471 KiB/s wr, 10 op/s 2023-03-09T20:44:55.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:44:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20806.283914 secs 2023-03-09T20:44:55.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:44:55 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:44:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:44:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:44:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:44:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:44:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:44:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:44:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:44:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:44:56.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:44:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:44:56.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:44:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:44:56.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:44:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:44:56.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:44:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:44:56.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:44:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:44:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:44:57 smithi122 ceph-mon[93869]: pgmap v11608: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 471 KiB/s wr, 12 op/s 2023-03-09T20:44:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:44:57 smithi181 ceph-mon[94189]: pgmap v11608: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 471 KiB/s wr, 12 op/s 2023-03-09T20:44:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:44:57 smithi038 ceph-mon[92422]: pgmap v11608: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 471 KiB/s wr, 12 op/s 2023-03-09T20:44:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21270.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:45:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:44:59 smithi122 ceph-mon[93869]: pgmap v11609: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 540 KiB/s rd, 525 KiB/s wr, 11 op/s 2023-03-09T20:45:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:44:59 smithi181 ceph-mon[94189]: pgmap v11609: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 540 KiB/s rd, 525 KiB/s wr, 11 op/s 2023-03-09T20:45:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:44:59 smithi038 ceph-mon[92422]: pgmap v11609: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 540 KiB/s rd, 525 KiB/s wr, 11 op/s 2023-03-09T20:45:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20811.283993 secs 2023-03-09T20:45:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:45:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:45:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20811.283993 secs 2023-03-09T20:45:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:45:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:45:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20811.283993 secs 2023-03-09T20:45:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:45:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:45:02.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:01 smithi122 ceph-mon[93869]: pgmap v11610: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 445 KiB/s rd, 430 KiB/s wr, 12 op/s 2023-03-09T20:45:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:01 smithi181 ceph-mon[94189]: pgmap v11610: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 445 KiB/s rd, 430 KiB/s wr, 12 op/s 2023-03-09T20:45:02.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:01 smithi038 ceph-mon[92422]: pgmap v11610: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 445 KiB/s rd, 430 KiB/s wr, 12 op/s 2023-03-09T20:45:03.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21275.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:45:04.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:03 smithi122 ceph-mon[93869]: pgmap v11611: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 564 KiB/s wr, 12 op/s 2023-03-09T20:45:04.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:03 smithi181 ceph-mon[94189]: pgmap v11611: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 564 KiB/s wr, 12 op/s 2023-03-09T20:45:04.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:03 smithi038 ceph-mon[92422]: pgmap v11611: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 564 KiB/s wr, 12 op/s 2023-03-09T20:45:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:05 smithi122 ceph-mon[93869]: pgmap v11612: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 583 KiB/s rd, 630 KiB/s wr, 13 op/s 2023-03-09T20:45:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20816.284054 secs 2023-03-09T20:45:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:05 smithi181 ceph-mon[94189]: pgmap v11612: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 583 KiB/s rd, 630 KiB/s wr, 13 op/s 2023-03-09T20:45:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20816.284054 secs 2023-03-09T20:45:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:05 smithi038 ceph-mon[92422]: pgmap v11612: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 583 KiB/s rd, 630 KiB/s wr, 13 op/s 2023-03-09T20:45:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20816.284054 secs 2023-03-09T20:45:07.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:45:07.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:45:07.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:07 smithi122 ceph-mon[93869]: pgmap v11613: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 351 KiB/s wr, 14 op/s 2023-03-09T20:45:07.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20818.598179 secs 2023-03-09T20:45:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:45:07.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:45:07.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:07 smithi181 ceph-mon[94189]: pgmap v11613: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 351 KiB/s wr, 14 op/s 2023-03-09T20:45:07.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20818.598179 secs 2023-03-09T20:45:07.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:45:07.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:45:07.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:07 smithi038 ceph-mon[92422]: pgmap v11613: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 351 KiB/s wr, 14 op/s 2023-03-09T20:45:07.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20818.598179 secs 2023-03-09T20:45:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:08 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:45:08.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:45:08.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:45:08.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:08 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:45:08.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:45:08.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:45:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:08 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:45:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:45:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:45:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21280.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:45:09.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:09 smithi038 ceph-mon[92422]: pgmap v11614: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 435 KiB/s rd, 518 KiB/s wr, 25 op/s 2023-03-09T20:45:09.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:09 smithi122 ceph-mon[93869]: pgmap v11614: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 435 KiB/s rd, 518 KiB/s wr, 25 op/s 2023-03-09T20:45:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:09 smithi181 ceph-mon[94189]: pgmap v11614: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 435 KiB/s rd, 518 KiB/s wr, 25 op/s 2023-03-09T20:45:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:10 smithi038 ceph-mon[92422]: osdmap e939: 12 total, 12 up, 12 in 2023-03-09T20:45:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20821.284115 secs 2023-03-09T20:45:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:10 smithi122 ceph-mon[93869]: osdmap e939: 12 total, 12 up, 12 in 2023-03-09T20:45:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20821.284115 secs 2023-03-09T20:45:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:10 smithi181 ceph-mon[94189]: osdmap e939: 12 total, 12 up, 12 in 2023-03-09T20:45:10.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20821.284115 secs 2023-03-09T20:45:11.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:11 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:45:11.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:11 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:45:11.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:11 smithi038 ceph-mon[92422]: pgmap v11616: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 629 KiB/s rd, 670 KiB/s wr, 28 op/s 2023-03-09T20:45:11.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:11 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:45:11.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:11 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:45:11.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:11 smithi122 ceph-mon[93869]: pgmap v11616: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 629 KiB/s rd, 670 KiB/s wr, 28 op/s 2023-03-09T20:45:11.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:11 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:45:11.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:11 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:45:11.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:11 smithi181 ceph-mon[94189]: pgmap v11616: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 629 KiB/s rd, 670 KiB/s wr, 28 op/s 2023-03-09T20:45:12.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20823.598240 secs 2023-03-09T20:45:12.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20823.598240 secs 2023-03-09T20:45:13.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20823.598240 secs 2023-03-09T20:45:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21285.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:45:13.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:45:13.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:45:13.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:13 smithi122 ceph-mon[93869]: pgmap v11617: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 461 KiB/s wr, 28 op/s 2023-03-09T20:45:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:45:13.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:45:13.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:13 smithi181 ceph-mon[94189]: pgmap v11617: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 461 KiB/s wr, 28 op/s 2023-03-09T20:45:14.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:45:14.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:45:14.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:13 smithi038 ceph-mon[92422]: pgmap v11617: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 461 KiB/s wr, 28 op/s 2023-03-09T20:45:15.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:15 smithi122 ceph-mon[93869]: pgmap v11618: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 475 KiB/s wr, 26 op/s 2023-03-09T20:45:15.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20826.284177 secs 2023-03-09T20:45:15.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:15 smithi181 ceph-mon[94189]: pgmap v11618: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 475 KiB/s wr, 26 op/s 2023-03-09T20:45:15.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20826.284177 secs 2023-03-09T20:45:15.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:15 smithi038 ceph-mon[92422]: pgmap v11618: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 475 KiB/s wr, 26 op/s 2023-03-09T20:45:15.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20826.284177 secs 2023-03-09T20:45:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:17 smithi122 ceph-mon[93869]: pgmap v11619: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 399 KiB/s rd, 448 KiB/s wr, 24 op/s 2023-03-09T20:45:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20828.598324 secs 2023-03-09T20:45:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:17 smithi181 ceph-mon[94189]: pgmap v11619: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 399 KiB/s rd, 448 KiB/s wr, 24 op/s 2023-03-09T20:45:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20828.598324 secs 2023-03-09T20:45:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:17 smithi038 ceph-mon[92422]: pgmap v11619: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 399 KiB/s rd, 448 KiB/s wr, 24 op/s 2023-03-09T20:45:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20828.598324 secs 2023-03-09T20:45:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21290.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:45:20.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:19 smithi122 ceph-mon[93869]: pgmap v11620: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 247 KiB/s wr, 10 op/s 2023-03-09T20:45:20.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:19 smithi122 ceph-mon[93869]: osdmap e940: 12 total, 12 up, 12 in 2023-03-09T20:45:20.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:19 smithi181 ceph-mon[94189]: pgmap v11620: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 247 KiB/s wr, 10 op/s 2023-03-09T20:45:20.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:19 smithi181 ceph-mon[94189]: osdmap e940: 12 total, 12 up, 12 in 2023-03-09T20:45:20.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:19 smithi038 ceph-mon[92422]: pgmap v11620: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 247 KiB/s wr, 10 op/s 2023-03-09T20:45:20.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:19 smithi038 ceph-mon[92422]: osdmap e940: 12 total, 12 up, 12 in 2023-03-09T20:45:21.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20831.284239 secs 2023-03-09T20:45:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20831.284239 secs 2023-03-09T20:45:21.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20831.284239 secs 2023-03-09T20:45:22.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:21 smithi122 ceph-mon[93869]: pgmap v11622: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 94 KiB/s wr, 9 op/s 2023-03-09T20:45:22.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:21 smithi181 ceph-mon[94189]: pgmap v11622: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 94 KiB/s wr, 9 op/s 2023-03-09T20:45:22.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:21 smithi038 ceph-mon[92422]: pgmap v11622: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 94 KiB/s wr, 9 op/s 2023-03-09T20:45:23.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20833.598442 secs 2023-03-09T20:45:23.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20833.598442 secs 2023-03-09T20:45:23.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20833.598442 secs 2023-03-09T20:45:23.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21295.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:45:24.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:23 smithi122 ceph-mon[93869]: pgmap v11623: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 113 KiB/s wr, 9 op/s 2023-03-09T20:45:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:23 smithi181 ceph-mon[94189]: pgmap v11623: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 113 KiB/s wr, 9 op/s 2023-03-09T20:45:24.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:23 smithi038 ceph-mon[92422]: pgmap v11623: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 113 KiB/s wr, 9 op/s 2023-03-09T20:45:25.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:25 smithi122 ceph-mon[93869]: pgmap v11624: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 72 KiB/s wr, 8 op/s 2023-03-09T20:45:25.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20836.284299 secs 2023-03-09T20:45:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:25 smithi181 ceph-mon[94189]: pgmap v11624: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 72 KiB/s wr, 8 op/s 2023-03-09T20:45:25.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20836.284299 secs 2023-03-09T20:45:25.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:25 smithi038 ceph-mon[92422]: pgmap v11624: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 72 KiB/s wr, 8 op/s 2023-03-09T20:45:25.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20836.284299 secs 2023-03-09T20:45:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:27 smithi122 ceph-mon[93869]: pgmap v11625: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 74 KiB/s wr, 9 op/s 2023-03-09T20:45:28.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20838.598562 secs 2023-03-09T20:45:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:27 smithi181 ceph-mon[94189]: pgmap v11625: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 74 KiB/s wr, 9 op/s 2023-03-09T20:45:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20838.598562 secs 2023-03-09T20:45:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:27 smithi038 ceph-mon[92422]: pgmap v11625: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 74 KiB/s wr, 9 op/s 2023-03-09T20:45:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20838.598562 secs 2023-03-09T20:45:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21300.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:45:30.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:29 smithi122 ceph-mon[93869]: pgmap v11626: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 74 KiB/s wr, 8 op/s 2023-03-09T20:45:30.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:29 smithi181 ceph-mon[94189]: pgmap v11626: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 74 KiB/s wr, 8 op/s 2023-03-09T20:45:30.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:29 smithi038 ceph-mon[92422]: pgmap v11626: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 74 KiB/s wr, 8 op/s 2023-03-09T20:45:31.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20841.284375 secs 2023-03-09T20:45:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20841.284375 secs 2023-03-09T20:45:31.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20841.284375 secs 2023-03-09T20:45:32.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:31 smithi122 ceph-mon[93869]: pgmap v11627: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 111 KiB/s wr, 8 op/s 2023-03-09T20:45:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:31 smithi181 ceph-mon[94189]: pgmap v11627: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 111 KiB/s wr, 8 op/s 2023-03-09T20:45:32.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:31 smithi038 ceph-mon[92422]: pgmap v11627: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 111 KiB/s wr, 8 op/s 2023-03-09T20:45:33.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20843.598701 secs 2023-03-09T20:45:33.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20843.598701 secs 2023-03-09T20:45:33.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20843.598701 secs 2023-03-09T20:45:33.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21305.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:45:34.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:33 smithi122 ceph-mon[93869]: pgmap v11628: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 142 KiB/s wr, 9 op/s 2023-03-09T20:45:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:33 smithi181 ceph-mon[94189]: pgmap v11628: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 142 KiB/s wr, 9 op/s 2023-03-09T20:45:34.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:33 smithi038 ceph-mon[92422]: pgmap v11628: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 142 KiB/s wr, 9 op/s 2023-03-09T20:45:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:35 smithi122 ceph-mon[93869]: pgmap v11629: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 126 KiB/s wr, 7 op/s 2023-03-09T20:45:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20846.284433 secs 2023-03-09T20:45:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:35 smithi181 ceph-mon[94189]: pgmap v11629: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 126 KiB/s wr, 7 op/s 2023-03-09T20:45:35.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20846.284433 secs 2023-03-09T20:45:35.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:35 smithi038 ceph-mon[92422]: pgmap v11629: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 126 KiB/s wr, 7 op/s 2023-03-09T20:45:35.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20846.284433 secs 2023-03-09T20:45:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:37 smithi122 ceph-mon[93869]: pgmap v11630: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 82 KiB/s wr, 9 op/s 2023-03-09T20:45:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:37 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 20848.598841 secs 2023-03-09T20:45:38.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:37 smithi181 ceph-mon[94189]: pgmap v11630: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 82 KiB/s wr, 9 op/s 2023-03-09T20:45:38.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:37 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 20848.598841 secs 2023-03-09T20:45:38.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:37 smithi038 ceph-mon[92422]: pgmap v11630: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 82 KiB/s wr, 9 op/s 2023-03-09T20:45:38.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:37 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 20848.598841 secs 2023-03-09T20:45:38.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21310.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:45:40.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:39 smithi122 ceph-mon[93869]: pgmap v11631: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 81 KiB/s wr, 7 op/s 2023-03-09T20:45:40.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:39 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:45:40.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:39 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:45:40.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:39 smithi181 ceph-mon[94189]: pgmap v11631: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 81 KiB/s wr, 7 op/s 2023-03-09T20:45:40.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:39 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:45:40.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:39 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:45:40.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:39 smithi038 ceph-mon[92422]: pgmap v11631: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 81 KiB/s wr, 7 op/s 2023-03-09T20:45:40.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:39 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:45:40.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:39 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:45:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20851.284495 secs 2023-03-09T20:45:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:45:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:45:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20851.284495 secs 2023-03-09T20:45:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:45:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:45:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20851.284495 secs 2023-03-09T20:45:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:45:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:45:42.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:41 smithi122 ceph-mon[93869]: pgmap v11632: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 81 KiB/s wr, 8 op/s 2023-03-09T20:45:42.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:45:42.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:45:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:41 smithi181 ceph-mon[94189]: pgmap v11632: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 81 KiB/s wr, 8 op/s 2023-03-09T20:45:42.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:45:42.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:45:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:41 smithi038 ceph-mon[92422]: pgmap v11632: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 81 KiB/s wr, 8 op/s 2023-03-09T20:45:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:45:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:45:43.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:42 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 20853.598957 secs 2023-03-09T20:45:43.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:42 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 20853.598957 secs 2023-03-09T20:45:43.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:42 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 20853.598957 secs 2023-03-09T20:45:43.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21315.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:45:44.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:43 smithi122 ceph-mon[93869]: pgmap v11633: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 243 KiB/s wr, 10 op/s 2023-03-09T20:45:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:43 smithi181 ceph-mon[94189]: pgmap v11633: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 243 KiB/s wr, 10 op/s 2023-03-09T20:45:44.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:43 smithi038 ceph-mon[92422]: pgmap v11633: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 243 KiB/s wr, 10 op/s 2023-03-09T20:45:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:45 smithi122 ceph-mon[93869]: pgmap v11634: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 207 KiB/s wr, 8 op/s 2023-03-09T20:45:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20856.284575 secs 2023-03-09T20:45:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:45 smithi181 ceph-mon[94189]: pgmap v11634: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 207 KiB/s wr, 8 op/s 2023-03-09T20:45:45.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20856.284575 secs 2023-03-09T20:45:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:45 smithi038 ceph-mon[92422]: pgmap v11634: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 207 KiB/s wr, 8 op/s 2023-03-09T20:45:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20856.284575 secs 2023-03-09T20:45:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:47 smithi122 ceph-mon[93869]: pgmap v11635: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 261 KiB/s wr, 11 op/s 2023-03-09T20:45:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20858.599078 secs 2023-03-09T20:45:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:47 smithi181 ceph-mon[94189]: pgmap v11635: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 261 KiB/s wr, 11 op/s 2023-03-09T20:45:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20858.599078 secs 2023-03-09T20:45:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:47 smithi038 ceph-mon[92422]: pgmap v11635: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 261 KiB/s wr, 11 op/s 2023-03-09T20:45:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20858.599078 secs 2023-03-09T20:45:48.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21320.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:45:50.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:49 smithi122 ceph-mon[93869]: pgmap v11636: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 377 KiB/s wr, 10 op/s 2023-03-09T20:45:50.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:45:50.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:45:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:49 smithi181 ceph-mon[94189]: pgmap v11636: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 377 KiB/s wr, 10 op/s 2023-03-09T20:45:50.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:45:50.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:45:50.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:49 smithi038 ceph-mon[92422]: pgmap v11636: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 377 KiB/s wr, 10 op/s 2023-03-09T20:45:50.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:45:50.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:45:51.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20861.284634 secs 2023-03-09T20:45:51.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20861.284634 secs 2023-03-09T20:45:51.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20861.284634 secs 2023-03-09T20:45:52.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:51 smithi122 ceph-mon[93869]: pgmap v11637: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 377 KiB/s wr, 10 op/s 2023-03-09T20:45:52.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:45:52.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:45:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:51 smithi181 ceph-mon[94189]: pgmap v11637: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 377 KiB/s wr, 10 op/s 2023-03-09T20:45:52.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:45:52.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:45:52.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:51 smithi038 ceph-mon[92422]: pgmap v11637: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 377 KiB/s wr, 10 op/s 2023-03-09T20:45:52.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:45:52.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:45:53.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:45:53.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:45:53.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20863.599145 secs 2023-03-09T20:45:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:45:53.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:45:53.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20863.599145 secs 2023-03-09T20:45:53.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:45:53.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:45:53.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20863.599145 secs 2023-03-09T20:45:53.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21325.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:45:54.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:53 smithi122 ceph-mon[93869]: pgmap v11638: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 569 KiB/s rd, 554 KiB/s wr, 12 op/s 2023-03-09T20:45:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:53 smithi181 ceph-mon[94189]: pgmap v11638: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 569 KiB/s rd, 554 KiB/s wr, 12 op/s 2023-03-09T20:45:54.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:53 smithi038 ceph-mon[92422]: pgmap v11638: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 569 KiB/s rd, 554 KiB/s wr, 12 op/s 2023-03-09T20:45:55.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:55 smithi122 ceph-mon[93869]: pgmap v11639: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 363 KiB/s rd, 348 KiB/s wr, 9 op/s 2023-03-09T20:45:55.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20866.284718 secs 2023-03-09T20:45:55.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:55 smithi181 ceph-mon[94189]: pgmap v11639: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 363 KiB/s rd, 348 KiB/s wr, 9 op/s 2023-03-09T20:45:55.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20866.284718 secs 2023-03-09T20:45:55.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:55 smithi038 ceph-mon[92422]: pgmap v11639: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 363 KiB/s rd, 348 KiB/s wr, 9 op/s 2023-03-09T20:45:55.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20866.284718 secs 2023-03-09T20:45:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:45:56.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:45:56.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:45:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:57 smithi122 ceph-mon[93869]: pgmap v11640: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 497 KiB/s rd, 492 KiB/s wr, 13 op/s 2023-03-09T20:45:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20868.599240 secs 2023-03-09T20:45:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:57 smithi181 ceph-mon[94189]: pgmap v11640: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 497 KiB/s rd, 492 KiB/s wr, 13 op/s 2023-03-09T20:45:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20868.599240 secs 2023-03-09T20:45:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:57 smithi038 ceph-mon[92422]: pgmap v11640: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 497 KiB/s rd, 492 KiB/s wr, 13 op/s 2023-03-09T20:45:57.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20868.599240 secs 2023-03-09T20:45:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21330.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:45:59.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:45:59.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:45:59.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:59 smithi122 ceph-mon[93869]: pgmap v11641: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 598 KiB/s wr, 11 op/s 2023-03-09T20:45:59.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:45:59.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:45:59.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:45:59.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:45:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:45:59.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:45:59.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:45:59.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:59 smithi181 ceph-mon[94189]: pgmap v11641: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 598 KiB/s wr, 11 op/s 2023-03-09T20:45:59.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:45:59.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:45:59.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:45:59.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:45:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:45:59.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:45:59.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:45:59.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:59 smithi038 ceph-mon[92422]: pgmap v11641: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 598 KiB/s wr, 11 op/s 2023-03-09T20:45:59.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:45:59.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:45:59.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:45:59.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:45:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:46:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:46:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:46:00.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:46:00.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:46:00.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:46:00.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:46:00.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:46:00.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:46:00.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:46:00.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:46:00.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:00 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T20:46:00.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:00 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi038 to 84063k 2023-03-09T20:46:00.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:00 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T20:46:00.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:00 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi038 to 86081126: error parsing value: Value '86081126' is below minimum 939524096 2023-03-09T20:46:00.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20871.284785 secs 2023-03-09T20:46:00.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:46:00.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:46:00.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:46:00.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:46:00.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:46:00.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:46:00.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:46:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:46:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:46:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:46:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:46:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:46:00.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:46:00.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:46:00.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:46:00.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:46:00.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:46:00.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:00 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T20:46:00.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:00 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi038 to 84063k 2023-03-09T20:46:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:00 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T20:46:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:00 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi038 to 86081126: error parsing value: Value '86081126' is below minimum 939524096 2023-03-09T20:46:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20871.284785 secs 2023-03-09T20:46:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:46:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:46:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:46:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:46:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:46:00.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:46:00.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:46:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:46:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:46:00.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:46:00.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:46:00.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:46:00.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:46:00.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:46:00.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:46:00.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:46:00.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:46:00.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:00 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T20:46:00.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:00 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi038 to 84063k 2023-03-09T20:46:00.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:00 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T20:46:00.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:00 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi038 to 86081126: error parsing value: Value '86081126' is below minimum 939524096 2023-03-09T20:46:00.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20871.284785 secs 2023-03-09T20:46:00.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:46:00.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:46:00.753 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:46:00.753 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:46:00.753 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:46:00.753 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:46:00.753 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:46:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:01 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T20:46:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:01 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T20:46:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:46:01.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:46:01.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:01 smithi122 ceph-mon[93869]: pgmap v11642: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 335 KiB/s rd, 482 KiB/s wr, 11 op/s 2023-03-09T20:46:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:01 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T20:46:01.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:01 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T20:46:01.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:46:01.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:46:01.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:01 smithi181 ceph-mon[94189]: pgmap v11642: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 335 KiB/s rd, 482 KiB/s wr, 11 op/s 2023-03-09T20:46:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:01 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T20:46:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:01 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T20:46:01.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:46:01.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:46:01.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:01 smithi038 ceph-mon[92422]: pgmap v11642: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 335 KiB/s rd, 482 KiB/s wr, 11 op/s 2023-03-09T20:46:02.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20873.599363 secs 2023-03-09T20:46:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20873.599363 secs 2023-03-09T20:46:02.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20873.599363 secs 2023-03-09T20:46:03.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:03 smithi122 ceph-mon[93869]: pgmap v11643: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 537 KiB/s wr, 14 op/s 2023-03-09T20:46:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:03 smithi181 ceph-mon[94189]: pgmap v11643: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 537 KiB/s wr, 14 op/s 2023-03-09T20:46:03.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:03 smithi038 ceph-mon[92422]: pgmap v11643: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 537 KiB/s wr, 14 op/s 2023-03-09T20:46:03.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21335.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:46:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:05 smithi122 ceph-mon[93869]: pgmap v11644: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 446 KiB/s rd, 480 KiB/s wr, 24 op/s 2023-03-09T20:46:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20876.284842 secs 2023-03-09T20:46:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:05 smithi181 ceph-mon[94189]: pgmap v11644: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 446 KiB/s rd, 480 KiB/s wr, 24 op/s 2023-03-09T20:46:05.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20876.284842 secs 2023-03-09T20:46:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:05 smithi038 ceph-mon[92422]: pgmap v11644: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 446 KiB/s rd, 480 KiB/s wr, 24 op/s 2023-03-09T20:46:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20876.284842 secs 2023-03-09T20:46:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:07 smithi122 ceph-mon[93869]: pgmap v11645: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 480 KiB/s wr, 27 op/s 2023-03-09T20:46:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20878.599477 secs 2023-03-09T20:46:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:07 smithi181 ceph-mon[94189]: pgmap v11645: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 480 KiB/s wr, 27 op/s 2023-03-09T20:46:08.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20878.599477 secs 2023-03-09T20:46:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:07 smithi038 ceph-mon[92422]: pgmap v11645: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 480 KiB/s wr, 27 op/s 2023-03-09T20:46:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20878.599477 secs 2023-03-09T20:46:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21340.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:46:10.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:09 smithi122 ceph-mon[93869]: pgmap v11646: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 419 KiB/s wr, 24 op/s 2023-03-09T20:46:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:09 smithi181 ceph-mon[94189]: pgmap v11646: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 419 KiB/s wr, 24 op/s 2023-03-09T20:46:10.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:09 smithi038 ceph-mon[92422]: pgmap v11646: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 419 KiB/s wr, 24 op/s 2023-03-09T20:46:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:10 smithi122 ceph-mon[93869]: osdmap e941: 12 total, 12 up, 12 in 2023-03-09T20:46:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20881.284902 secs 2023-03-09T20:46:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:10 smithi181 ceph-mon[94189]: osdmap e941: 12 total, 12 up, 12 in 2023-03-09T20:46:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20881.284902 secs 2023-03-09T20:46:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:10 smithi038 ceph-mon[92422]: osdmap e941: 12 total, 12 up, 12 in 2023-03-09T20:46:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20881.284902 secs 2023-03-09T20:46:12.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:11 smithi122 ceph-mon[93869]: pgmap v11648: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 577 KiB/s rd, 445 KiB/s wr, 28 op/s 2023-03-09T20:46:12.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:11 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:46:12.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:11 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:46:12.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:11 smithi181 ceph-mon[94189]: pgmap v11648: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 577 KiB/s rd, 445 KiB/s wr, 28 op/s 2023-03-09T20:46:12.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:11 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:46:12.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:11 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:46:12.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:11 smithi038 ceph-mon[92422]: pgmap v11648: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 577 KiB/s rd, 445 KiB/s wr, 28 op/s 2023-03-09T20:46:12.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:11 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:46:12.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:11 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:46:13.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:46:13.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:46:13.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20883.599553 secs 2023-03-09T20:46:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:46:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:46:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20883.599553 secs 2023-03-09T20:46:13.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:46:13.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:46:13.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20883.599553 secs 2023-03-09T20:46:13.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21345.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:46:14.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:13 smithi122 ceph-mon[93869]: pgmap v11649: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 536 KiB/s rd, 379 KiB/s wr, 25 op/s 2023-03-09T20:46:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:13 smithi181 ceph-mon[94189]: pgmap v11649: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 536 KiB/s rd, 379 KiB/s wr, 25 op/s 2023-03-09T20:46:14.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:13 smithi038 ceph-mon[92422]: pgmap v11649: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 536 KiB/s rd, 379 KiB/s wr, 25 op/s 2023-03-09T20:46:15.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:15 smithi122 ceph-mon[93869]: pgmap v11650: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 234 KiB/s wr, 9 op/s 2023-03-09T20:46:15.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20886.284963 secs 2023-03-09T20:46:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:15 smithi181 ceph-mon[94189]: pgmap v11650: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 234 KiB/s wr, 9 op/s 2023-03-09T20:46:15.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20886.284963 secs 2023-03-09T20:46:15.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:15 smithi038 ceph-mon[92422]: pgmap v11650: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 234 KiB/s wr, 9 op/s 2023-03-09T20:46:15.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20886.284963 secs 2023-03-09T20:46:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:17 smithi122 ceph-mon[93869]: pgmap v11651: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 320 KiB/s wr, 10 op/s 2023-03-09T20:46:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20888.599696 secs 2023-03-09T20:46:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:17 smithi181 ceph-mon[94189]: pgmap v11651: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 320 KiB/s wr, 10 op/s 2023-03-09T20:46:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20888.599696 secs 2023-03-09T20:46:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:17 smithi038 ceph-mon[92422]: pgmap v11651: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 320 KiB/s wr, 10 op/s 2023-03-09T20:46:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20888.599696 secs 2023-03-09T20:46:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21350.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:46:20.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:19 smithi122 ceph-mon[93869]: pgmap v11652: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 220 KiB/s wr, 9 op/s 2023-03-09T20:46:20.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:19 smithi122 ceph-mon[93869]: osdmap e942: 12 total, 12 up, 12 in 2023-03-09T20:46:20.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:19 smithi181 ceph-mon[94189]: pgmap v11652: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 220 KiB/s wr, 9 op/s 2023-03-09T20:46:20.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:19 smithi181 ceph-mon[94189]: osdmap e942: 12 total, 12 up, 12 in 2023-03-09T20:46:20.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:19 smithi038 ceph-mon[92422]: pgmap v11652: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 220 KiB/s wr, 9 op/s 2023-03-09T20:46:20.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:19 smithi038 ceph-mon[92422]: osdmap e942: 12 total, 12 up, 12 in 2023-03-09T20:46:21.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20891.284985 secs 2023-03-09T20:46:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20891.284985 secs 2023-03-09T20:46:21.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20891.284985 secs 2023-03-09T20:46:22.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:21 smithi122 ceph-mon[93869]: pgmap v11654: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 107 KiB/s wr, 11 op/s 2023-03-09T20:46:22.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:21 smithi181 ceph-mon[94189]: pgmap v11654: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 107 KiB/s wr, 11 op/s 2023-03-09T20:46:22.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:21 smithi038 ceph-mon[92422]: pgmap v11654: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 107 KiB/s wr, 11 op/s 2023-03-09T20:46:23.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20893.599828 secs 2023-03-09T20:46:23.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20893.599828 secs 2023-03-09T20:46:23.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20893.599828 secs 2023-03-09T20:46:23.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21355.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:46:24.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:23 smithi122 ceph-mon[93869]: pgmap v11655: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 107 KiB/s wr, 9 op/s 2023-03-09T20:46:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:23 smithi181 ceph-mon[94189]: pgmap v11655: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 107 KiB/s wr, 9 op/s 2023-03-09T20:46:24.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:23 smithi038 ceph-mon[92422]: pgmap v11655: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 107 KiB/s wr, 9 op/s 2023-03-09T20:46:25.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:25 smithi122 ceph-mon[93869]: pgmap v11656: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 107 KiB/s wr, 8 op/s 2023-03-09T20:46:25.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20896.285045 secs 2023-03-09T20:46:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:25 smithi181 ceph-mon[94189]: pgmap v11656: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 107 KiB/s wr, 8 op/s 2023-03-09T20:46:25.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20896.285045 secs 2023-03-09T20:46:25.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:25 smithi038 ceph-mon[92422]: pgmap v11656: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 107 KiB/s wr, 8 op/s 2023-03-09T20:46:25.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20896.285045 secs 2023-03-09T20:46:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:27 smithi122 ceph-mon[93869]: pgmap v11657: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 87 KiB/s wr, 7 op/s 2023-03-09T20:46:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20898.599953 secs 2023-03-09T20:46:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:27 smithi181 ceph-mon[94189]: pgmap v11657: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 87 KiB/s wr, 7 op/s 2023-03-09T20:46:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20898.599953 secs 2023-03-09T20:46:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:27 smithi038 ceph-mon[92422]: pgmap v11657: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 87 KiB/s wr, 7 op/s 2023-03-09T20:46:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20898.599953 secs 2023-03-09T20:46:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21360.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:46:30.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:29 smithi122 ceph-mon[93869]: pgmap v11658: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 87 KiB/s wr, 7 op/s 2023-03-09T20:46:30.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:29 smithi181 ceph-mon[94189]: pgmap v11658: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 87 KiB/s wr, 7 op/s 2023-03-09T20:46:30.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:29 smithi038 ceph-mon[92422]: pgmap v11658: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 87 KiB/s wr, 7 op/s 2023-03-09T20:46:31.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20901.285106 secs 2023-03-09T20:46:31.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20901.285106 secs 2023-03-09T20:46:31.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20901.285106 secs 2023-03-09T20:46:32.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:31 smithi122 ceph-mon[93869]: pgmap v11659: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 161 KiB/s wr, 10 op/s 2023-03-09T20:46:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:31 smithi181 ceph-mon[94189]: pgmap v11659: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 161 KiB/s wr, 10 op/s 2023-03-09T20:46:32.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:31 smithi038 ceph-mon[92422]: pgmap v11659: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 161 KiB/s wr, 10 op/s 2023-03-09T20:46:33.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20903.600021 secs 2023-03-09T20:46:33.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20903.600021 secs 2023-03-09T20:46:33.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20903.600021 secs 2023-03-09T20:46:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21365.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:46:34.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:33 smithi122 ceph-mon[93869]: pgmap v11660: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 137 KiB/s wr, 6 op/s 2023-03-09T20:46:34.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:33 smithi181 ceph-mon[94189]: pgmap v11660: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 137 KiB/s wr, 6 op/s 2023-03-09T20:46:34.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:33 smithi038 ceph-mon[92422]: pgmap v11660: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 137 KiB/s wr, 6 op/s 2023-03-09T20:46:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:35 smithi122 ceph-mon[93869]: pgmap v11661: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 137 KiB/s wr, 6 op/s 2023-03-09T20:46:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20906.285174 secs 2023-03-09T20:46:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:35 smithi181 ceph-mon[94189]: pgmap v11661: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 137 KiB/s wr, 6 op/s 2023-03-09T20:46:35.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20906.285174 secs 2023-03-09T20:46:35.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:35 smithi038 ceph-mon[92422]: pgmap v11661: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 137 KiB/s wr, 6 op/s 2023-03-09T20:46:35.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20906.285174 secs 2023-03-09T20:46:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:37 smithi122 ceph-mon[93869]: pgmap v11662: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 177 KiB/s wr, 9 op/s 2023-03-09T20:46:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20908.600142 secs 2023-03-09T20:46:38.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:37 smithi181 ceph-mon[94189]: pgmap v11662: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 177 KiB/s wr, 9 op/s 2023-03-09T20:46:38.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20908.600142 secs 2023-03-09T20:46:38.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:37 smithi038 ceph-mon[92422]: pgmap v11662: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 177 KiB/s wr, 9 op/s 2023-03-09T20:46:38.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20908.600142 secs 2023-03-09T20:46:38.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21370.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:46:40.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:39 smithi122 ceph-mon[93869]: pgmap v11663: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 158 KiB/s wr, 7 op/s 2023-03-09T20:46:40.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:39 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:46:40.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:39 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:46:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:39 smithi181 ceph-mon[94189]: pgmap v11663: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 158 KiB/s wr, 7 op/s 2023-03-09T20:46:40.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:39 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:46:40.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:39 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:46:40.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:39 smithi038 ceph-mon[92422]: pgmap v11663: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 158 KiB/s wr, 7 op/s 2023-03-09T20:46:40.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:39 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:46:40.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:39 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:46:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20911.285236 secs 2023-03-09T20:46:41.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:46:41.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:46:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20911.285236 secs 2023-03-09T20:46:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:46:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:46:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20911.285236 secs 2023-03-09T20:46:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:46:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:46:42.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:41 smithi122 ceph-mon[93869]: pgmap v11664: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 175 KiB/s wr, 10 op/s 2023-03-09T20:46:42.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:41 smithi181 ceph-mon[94189]: pgmap v11664: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 175 KiB/s wr, 10 op/s 2023-03-09T20:46:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:41 smithi038 ceph-mon[92422]: pgmap v11664: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 175 KiB/s wr, 10 op/s 2023-03-09T20:46:43.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20913.600264 secs 2023-03-09T20:46:43.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20913.600264 secs 2023-03-09T20:46:43.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20913.600264 secs 2023-03-09T20:46:43.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21375.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:46:44.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:43 smithi122 ceph-mon[93869]: pgmap v11665: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 218 KiB/s wr, 7 op/s 2023-03-09T20:46:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:43 smithi181 ceph-mon[94189]: pgmap v11665: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 218 KiB/s wr, 7 op/s 2023-03-09T20:46:44.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:43 smithi038 ceph-mon[92422]: pgmap v11665: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 218 KiB/s wr, 7 op/s 2023-03-09T20:46:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:45 smithi122 ceph-mon[93869]: pgmap v11666: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 218 KiB/s wr, 8 op/s 2023-03-09T20:46:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20916.285298 secs 2023-03-09T20:46:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:45 smithi181 ceph-mon[94189]: pgmap v11666: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 218 KiB/s wr, 8 op/s 2023-03-09T20:46:45.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20916.285298 secs 2023-03-09T20:46:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:45 smithi038 ceph-mon[92422]: pgmap v11666: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 218 KiB/s wr, 8 op/s 2023-03-09T20:46:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20916.285298 secs 2023-03-09T20:46:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:47 smithi122 ceph-mon[93869]: pgmap v11667: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 235 KiB/s wr, 11 op/s 2023-03-09T20:46:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:47 smithi181 ceph-mon[94189]: pgmap v11667: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 235 KiB/s wr, 11 op/s 2023-03-09T20:46:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:47 smithi038 ceph-mon[92422]: pgmap v11667: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 235 KiB/s wr, 11 op/s 2023-03-09T20:46:48.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21380.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:46:49.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:48 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:46:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:48 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:46:49.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:48 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:46:50.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:49 smithi122 ceph-mon[93869]: pgmap v11668: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 376 KiB/s wr, 10 op/s 2023-03-09T20:46:50.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:49 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:46:50.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:46:50.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:46:50.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:49 smithi181 ceph-mon[94189]: pgmap v11668: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 376 KiB/s wr, 10 op/s 2023-03-09T20:46:50.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:49 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:46:50.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:46:50.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:46:50.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:49 smithi038 ceph-mon[92422]: pgmap v11668: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 376 KiB/s wr, 10 op/s 2023-03-09T20:46:50.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:49 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:46:50.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:46:50.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:46:51.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20921.285361 secs 2023-03-09T20:46:51.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20921.285361 secs 2023-03-09T20:46:51.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20921.285361 secs 2023-03-09T20:46:52.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:51 smithi122 ceph-mon[93869]: pgmap v11669: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 356 KiB/s wr, 12 op/s 2023-03-09T20:46:52.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:46:52.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:46:52.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:51 smithi181 ceph-mon[94189]: pgmap v11669: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 356 KiB/s wr, 12 op/s 2023-03-09T20:46:52.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:46:52.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:46:52.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:51 smithi038 ceph-mon[92422]: pgmap v11669: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 356 KiB/s wr, 12 op/s 2023-03-09T20:46:52.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:46:52.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:46:53.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:46:53.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:46:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:46:53.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:46:53.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:46:53.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:46:53.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21385.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:46:54.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:53 smithi122 ceph-mon[93869]: pgmap v11670: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 601 KiB/s rd, 573 KiB/s wr, 12 op/s 2023-03-09T20:46:54.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:53 smithi181 ceph-mon[94189]: pgmap v11670: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 601 KiB/s rd, 573 KiB/s wr, 12 op/s 2023-03-09T20:46:54.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:53 smithi038 ceph-mon[92422]: pgmap v11670: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 601 KiB/s rd, 573 KiB/s wr, 12 op/s 2023-03-09T20:46:55.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:55 smithi122 ceph-mon[93869]: pgmap v11671: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 448 KiB/s wr, 11 op/s 2023-03-09T20:46:55.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20926.285434 secs 2023-03-09T20:46:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:55 smithi181 ceph-mon[94189]: pgmap v11671: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 448 KiB/s wr, 11 op/s 2023-03-09T20:46:55.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20926.285434 secs 2023-03-09T20:46:55.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:55 smithi038 ceph-mon[92422]: pgmap v11671: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 448 KiB/s wr, 11 op/s 2023-03-09T20:46:55.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20926.285434 secs 2023-03-09T20:46:58.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:57 smithi122 ceph-mon[93869]: pgmap v11672: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 483 KiB/s rd, 465 KiB/s wr, 14 op/s 2023-03-09T20:46:58.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:57 smithi181 ceph-mon[94189]: pgmap v11672: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 483 KiB/s rd, 465 KiB/s wr, 14 op/s 2023-03-09T20:46:58.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:57 smithi038 ceph-mon[92422]: pgmap v11672: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 483 KiB/s rd, 465 KiB/s wr, 14 op/s 2023-03-09T20:46:58.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21390.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:47:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:59 smithi122 ceph-mon[93869]: pgmap v11673: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 592 KiB/s rd, 557 KiB/s wr, 12 op/s 2023-03-09T20:47:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:47:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:46:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:47:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:59 smithi181 ceph-mon[94189]: pgmap v11673: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 592 KiB/s rd, 557 KiB/s wr, 12 op/s 2023-03-09T20:47:00.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:47:00.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:46:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:47:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:59 smithi038 ceph-mon[92422]: pgmap v11673: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 592 KiB/s rd, 557 KiB/s wr, 12 op/s 2023-03-09T20:47:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:47:00.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:46:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:47:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20931.285517 secs 2023-03-09T20:47:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:47:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:47:01.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:47:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20931.285517 secs 2023-03-09T20:47:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:47:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:47:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:47:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20931.285517 secs 2023-03-09T20:47:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:47:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:47:01.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:47:02.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:01 smithi122 ceph-mon[93869]: pgmap v11674: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 415 KiB/s wr, 15 op/s 2023-03-09T20:47:02.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:01 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:47:02.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:01 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:47:02.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:01 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:47:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:01 smithi181 ceph-mon[94189]: pgmap v11674: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 415 KiB/s wr, 15 op/s 2023-03-09T20:47:02.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:01 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:47:02.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:01 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:47:02.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:01 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:47:02.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:01 smithi038 ceph-mon[92422]: pgmap v11674: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 415 KiB/s wr, 15 op/s 2023-03-09T20:47:02.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:01 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:47:02.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:01 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:47:02.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:01 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:47:03.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20933.600688 secs 2023-03-09T20:47:03.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20933.600688 secs 2023-03-09T20:47:03.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20933.600688 secs 2023-03-09T20:47:03.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21395.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:47:04.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:03 smithi122 ceph-mon[93869]: pgmap v11675: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 553 KiB/s rd, 540 KiB/s wr, 14 op/s 2023-03-09T20:47:04.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:03 smithi181 ceph-mon[94189]: pgmap v11675: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 553 KiB/s rd, 540 KiB/s wr, 14 op/s 2023-03-09T20:47:04.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:03 smithi038 ceph-mon[92422]: pgmap v11675: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 553 KiB/s rd, 540 KiB/s wr, 14 op/s 2023-03-09T20:47:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:05 smithi122 ceph-mon[93869]: pgmap v11676: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 329 KiB/s wr, 25 op/s 2023-03-09T20:47:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20936.285606 secs 2023-03-09T20:47:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:05 smithi181 ceph-mon[94189]: pgmap v11676: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 329 KiB/s wr, 25 op/s 2023-03-09T20:47:05.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20936.285606 secs 2023-03-09T20:47:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:05 smithi038 ceph-mon[92422]: pgmap v11676: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 329 KiB/s wr, 25 op/s 2023-03-09T20:47:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20936.285606 secs 2023-03-09T20:47:06.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:06 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:47:06.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:06 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:47:06.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:47:06.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:06 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:47:06.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:06 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:47:06.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:47:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:06 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:47:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:06 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:47:06.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:47:07.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:07 smithi122 ceph-mon[93869]: pgmap v11677: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 348 KiB/s wr, 27 op/s 2023-03-09T20:47:07.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20938.600777 secs 2023-03-09T20:47:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:07 smithi181 ceph-mon[94189]: pgmap v11677: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 348 KiB/s wr, 27 op/s 2023-03-09T20:47:07.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20938.600777 secs 2023-03-09T20:47:07.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:07 smithi038 ceph-mon[92422]: pgmap v11677: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 348 KiB/s wr, 27 op/s 2023-03-09T20:47:07.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20938.600777 secs 2023-03-09T20:47:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21400.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:47:10.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:09 smithi122 ceph-mon[93869]: pgmap v11678: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 444 KiB/s rd, 451 KiB/s wr, 26 op/s 2023-03-09T20:47:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:09 smithi181 ceph-mon[94189]: pgmap v11678: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 444 KiB/s rd, 451 KiB/s wr, 26 op/s 2023-03-09T20:47:10.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:09 smithi038 ceph-mon[92422]: pgmap v11678: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 444 KiB/s rd, 451 KiB/s wr, 26 op/s 2023-03-09T20:47:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:10 smithi122 ceph-mon[93869]: osdmap e943: 12 total, 12 up, 12 in 2023-03-09T20:47:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20941.285669 secs 2023-03-09T20:47:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:10 smithi181 ceph-mon[94189]: osdmap e943: 12 total, 12 up, 12 in 2023-03-09T20:47:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20941.285669 secs 2023-03-09T20:47:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:10 smithi038 ceph-mon[92422]: osdmap e943: 12 total, 12 up, 12 in 2023-03-09T20:47:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20941.285669 secs 2023-03-09T20:47:12.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:11 smithi122 ceph-mon[93869]: pgmap v11680: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 381 KiB/s wr, 26 op/s 2023-03-09T20:47:12.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:11 smithi181 ceph-mon[94189]: pgmap v11680: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 381 KiB/s wr, 26 op/s 2023-03-09T20:47:12.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:11 smithi038 ceph-mon[92422]: pgmap v11680: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 381 KiB/s wr, 26 op/s 2023-03-09T20:47:13.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20943.600842 secs 2023-03-09T20:47:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20943.600842 secs 2023-03-09T20:47:13.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20943.600842 secs 2023-03-09T20:47:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21405.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:47:14.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:13 smithi122 ceph-mon[93869]: pgmap v11681: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 211 KiB/s wr, 24 op/s 2023-03-09T20:47:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:13 smithi181 ceph-mon[94189]: pgmap v11681: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 211 KiB/s wr, 24 op/s 2023-03-09T20:47:14.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:13 smithi038 ceph-mon[92422]: pgmap v11681: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 211 KiB/s wr, 24 op/s 2023-03-09T20:47:15.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:15 smithi122 ceph-mon[93869]: pgmap v11682: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 185 KiB/s wr, 9 op/s 2023-03-09T20:47:15.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20946.285727 secs 2023-03-09T20:47:15.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:15 smithi181 ceph-mon[94189]: pgmap v11682: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 185 KiB/s wr, 9 op/s 2023-03-09T20:47:15.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20946.285727 secs 2023-03-09T20:47:15.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:15 smithi038 ceph-mon[92422]: pgmap v11682: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 185 KiB/s wr, 9 op/s 2023-03-09T20:47:15.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20946.285727 secs 2023-03-09T20:47:18.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:17 smithi122 ceph-mon[93869]: pgmap v11683: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 162 KiB/s wr, 8 op/s 2023-03-09T20:47:18.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20948.600965 secs 2023-03-09T20:47:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:17 smithi181 ceph-mon[94189]: pgmap v11683: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 162 KiB/s wr, 8 op/s 2023-03-09T20:47:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20948.600965 secs 2023-03-09T20:47:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:17 smithi038 ceph-mon[92422]: pgmap v11683: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 162 KiB/s wr, 8 op/s 2023-03-09T20:47:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20948.600965 secs 2023-03-09T20:47:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21410.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:47:20.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:19 smithi122 ceph-mon[93869]: pgmap v11684: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 83 KiB/s wr, 7 op/s 2023-03-09T20:47:20.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:19 smithi122 ceph-mon[93869]: osdmap e944: 12 total, 12 up, 12 in 2023-03-09T20:47:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:19 smithi181 ceph-mon[94189]: pgmap v11684: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 83 KiB/s wr, 7 op/s 2023-03-09T20:47:20.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:19 smithi181 ceph-mon[94189]: osdmap e944: 12 total, 12 up, 12 in 2023-03-09T20:47:20.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:19 smithi038 ceph-mon[92422]: pgmap v11684: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 83 KiB/s wr, 7 op/s 2023-03-09T20:47:20.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:19 smithi038 ceph-mon[92422]: osdmap e944: 12 total, 12 up, 12 in 2023-03-09T20:47:20.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20951.285789 secs 2023-03-09T20:47:21.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20951.285789 secs 2023-03-09T20:47:21.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20951.285789 secs 2023-03-09T20:47:21.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:21 smithi122 ceph-mon[93869]: pgmap v11686: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 65 KiB/s wr, 7 op/s 2023-03-09T20:47:22.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:21 smithi181 ceph-mon[94189]: pgmap v11686: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 65 KiB/s wr, 7 op/s 2023-03-09T20:47:22.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:21 smithi038 ceph-mon[92422]: pgmap v11686: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 65 KiB/s wr, 7 op/s 2023-03-09T20:47:23.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20953.601081 secs 2023-03-09T20:47:23.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20953.601081 secs 2023-03-09T20:47:23.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20953.601081 secs 2023-03-09T20:47:23.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21415.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:47:24.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:23 smithi122 ceph-mon[93869]: pgmap v11687: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 65 KiB/s wr, 6 op/s 2023-03-09T20:47:24.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:23 smithi181 ceph-mon[94189]: pgmap v11687: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 65 KiB/s wr, 6 op/s 2023-03-09T20:47:24.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:23 smithi038 ceph-mon[92422]: pgmap v11687: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 65 KiB/s wr, 6 op/s 2023-03-09T20:47:25.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:25 smithi122 ceph-mon[93869]: pgmap v11688: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 65 KiB/s wr, 6 op/s 2023-03-09T20:47:25.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20956.285851 secs 2023-03-09T20:47:25.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:25 smithi181 ceph-mon[94189]: pgmap v11688: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 65 KiB/s wr, 6 op/s 2023-03-09T20:47:25.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20956.285851 secs 2023-03-09T20:47:25.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:25 smithi038 ceph-mon[92422]: pgmap v11688: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 65 KiB/s wr, 6 op/s 2023-03-09T20:47:25.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20956.285851 secs 2023-03-09T20:47:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:27 smithi122 ceph-mon[93869]: pgmap v11689: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 65 KiB/s wr, 6 op/s 2023-03-09T20:47:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20958.601199 secs 2023-03-09T20:47:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:27 smithi181 ceph-mon[94189]: pgmap v11689: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 65 KiB/s wr, 6 op/s 2023-03-09T20:47:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20958.601199 secs 2023-03-09T20:47:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:27 smithi038 ceph-mon[92422]: pgmap v11689: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 65 KiB/s wr, 6 op/s 2023-03-09T20:47:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20958.601199 secs 2023-03-09T20:47:28.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21420.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:47:29.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:29 smithi122 ceph-mon[93869]: pgmap v11690: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T20:47:30.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:29 smithi181 ceph-mon[94189]: pgmap v11690: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T20:47:30.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:29 smithi038 ceph-mon[92422]: pgmap v11690: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T20:47:31.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20961.285912 secs 2023-03-09T20:47:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20961.285912 secs 2023-03-09T20:47:31.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20961.285912 secs 2023-03-09T20:47:32.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:31 smithi122 ceph-mon[93869]: pgmap v11691: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 8 op/s 2023-03-09T20:47:32.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:31 smithi181 ceph-mon[94189]: pgmap v11691: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 8 op/s 2023-03-09T20:47:32.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:31 smithi038 ceph-mon[92422]: pgmap v11691: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 8 op/s 2023-03-09T20:47:33.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20963.601307 secs 2023-03-09T20:47:33.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20963.601307 secs 2023-03-09T20:47:33.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20963.601307 secs 2023-03-09T20:47:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21425.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:47:34.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:33 smithi122 ceph-mon[93869]: pgmap v11692: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 82 KiB/s wr, 7 op/s 2023-03-09T20:47:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:33 smithi181 ceph-mon[94189]: pgmap v11692: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 82 KiB/s wr, 7 op/s 2023-03-09T20:47:34.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:33 smithi038 ceph-mon[92422]: pgmap v11692: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 82 KiB/s wr, 7 op/s 2023-03-09T20:47:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:35 smithi122 ceph-mon[93869]: pgmap v11693: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 82 KiB/s wr, 6 op/s 2023-03-09T20:47:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20966.285978 secs 2023-03-09T20:47:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:35 smithi181 ceph-mon[94189]: pgmap v11693: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 82 KiB/s wr, 6 op/s 2023-03-09T20:47:35.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20966.285978 secs 2023-03-09T20:47:35.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:35 smithi038 ceph-mon[92422]: pgmap v11693: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 82 KiB/s wr, 6 op/s 2023-03-09T20:47:35.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20966.285978 secs 2023-03-09T20:47:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:37 smithi122 ceph-mon[93869]: pgmap v11694: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 82 KiB/s wr, 9 op/s 2023-03-09T20:47:38.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:37 smithi181 ceph-mon[94189]: pgmap v11694: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 82 KiB/s wr, 9 op/s 2023-03-09T20:47:38.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:37 smithi038 ceph-mon[92422]: pgmap v11694: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 82 KiB/s wr, 9 op/s 2023-03-09T20:47:38.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21430.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:47:40.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:39 smithi122 ceph-mon[93869]: pgmap v11695: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 82 KiB/s wr, 7 op/s 2023-03-09T20:47:40.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:39 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:47:40.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:39 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:47:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:39 smithi181 ceph-mon[94189]: pgmap v11695: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 82 KiB/s wr, 7 op/s 2023-03-09T20:47:40.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:39 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:47:40.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:39 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:47:40.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:39 smithi038 ceph-mon[92422]: pgmap v11695: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 82 KiB/s wr, 7 op/s 2023-03-09T20:47:40.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:39 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:47:40.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:39 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:47:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20971.286036 secs 2023-03-09T20:47:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:47:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:47:41.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:40 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:47:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20971.286036 secs 2023-03-09T20:47:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:47:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:47:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:40 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:47:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20971.286036 secs 2023-03-09T20:47:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:47:41.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:47:41.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:40 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:47:41.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:41 smithi122 ceph-mon[93869]: pgmap v11696: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 293 KiB/s rd, 289 KiB/s wr, 10 op/s 2023-03-09T20:47:41.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:41 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:47:41.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:47:41.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:47:42.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:41 smithi181 ceph-mon[94189]: pgmap v11696: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 293 KiB/s rd, 289 KiB/s wr, 10 op/s 2023-03-09T20:47:42.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:41 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:47:42.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:47:42.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:47:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:41 smithi038 ceph-mon[92422]: pgmap v11696: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 293 KiB/s rd, 289 KiB/s wr, 10 op/s 2023-03-09T20:47:42.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:41 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:47:42.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:47:42.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:47:43.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21435.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:47:44.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:43 smithi122 ceph-mon[93869]: pgmap v11697: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 235 KiB/s wr, 7 op/s 2023-03-09T20:47:44.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:43 smithi181 ceph-mon[94189]: pgmap v11697: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 235 KiB/s wr, 7 op/s 2023-03-09T20:47:44.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:43 smithi038 ceph-mon[92422]: pgmap v11697: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 235 KiB/s wr, 7 op/s 2023-03-09T20:47:45.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:45 smithi122 ceph-mon[93869]: pgmap v11698: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 208 KiB/s wr, 7 op/s 2023-03-09T20:47:45.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20976.286098 secs 2023-03-09T20:47:45.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:45 smithi181 ceph-mon[94189]: pgmap v11698: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 208 KiB/s wr, 7 op/s 2023-03-09T20:47:45.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20976.286098 secs 2023-03-09T20:47:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:45 smithi038 ceph-mon[92422]: pgmap v11698: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 208 KiB/s wr, 7 op/s 2023-03-09T20:47:45.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20976.286098 secs 2023-03-09T20:47:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:47 smithi122 ceph-mon[93869]: pgmap v11699: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 321 KiB/s wr, 10 op/s 2023-03-09T20:47:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:47 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 20978.601687 secs 2023-03-09T20:47:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:47 smithi181 ceph-mon[94189]: pgmap v11699: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 321 KiB/s wr, 10 op/s 2023-03-09T20:47:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:47 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 20978.601687 secs 2023-03-09T20:47:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:47 smithi038 ceph-mon[92422]: pgmap v11699: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 321 KiB/s wr, 10 op/s 2023-03-09T20:47:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:47 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 20978.601687 secs 2023-03-09T20:47:48.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21440.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:47:49.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:48 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:47:49.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:47:49.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:47:49.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:48 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:47:49.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:47:49.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:47:49.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:48 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:47:49.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:47:49.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:47:50.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:49 smithi122 ceph-mon[93869]: pgmap v11700: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 379 KiB/s wr, 9 op/s 2023-03-09T20:47:50.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:49 smithi181 ceph-mon[94189]: pgmap v11700: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 379 KiB/s wr, 9 op/s 2023-03-09T20:47:50.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:49 smithi038 ceph-mon[92422]: pgmap v11700: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 379 KiB/s wr, 9 op/s 2023-03-09T20:47:51.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20981.286170 secs 2023-03-09T20:47:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20981.286170 secs 2023-03-09T20:47:51.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20981.286170 secs 2023-03-09T20:47:52.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:51 smithi122 ceph-mon[93869]: pgmap v11701: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 396 KiB/s wr, 11 op/s 2023-03-09T20:47:52.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:47:52.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:47:52.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:51 smithi181 ceph-mon[94189]: pgmap v11701: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 396 KiB/s wr, 11 op/s 2023-03-09T20:47:52.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:47:52.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:47:52.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:51 smithi038 ceph-mon[92422]: pgmap v11701: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 396 KiB/s wr, 11 op/s 2023-03-09T20:47:52.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:47:52.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:47:53.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:47:53.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:47:53.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:52 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 20983.601802 secs 2023-03-09T20:47:53.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:47:53.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:47:53.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:52 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 20983.601802 secs 2023-03-09T20:47:53.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:47:53.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:47:53.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:52 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 20983.601802 secs 2023-03-09T20:47:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21445.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:47:54.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:53 smithi122 ceph-mon[93869]: pgmap v11702: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 351 KiB/s wr, 9 op/s 2023-03-09T20:47:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:53 smithi181 ceph-mon[94189]: pgmap v11702: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 351 KiB/s wr, 9 op/s 2023-03-09T20:47:54.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:53 smithi038 ceph-mon[92422]: pgmap v11702: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 351 KiB/s wr, 9 op/s 2023-03-09T20:47:55.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:55 smithi122 ceph-mon[93869]: pgmap v11703: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 351 KiB/s wr, 9 op/s 2023-03-09T20:47:55.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20986.286232 secs 2023-03-09T20:47:55.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:55 smithi181 ceph-mon[94189]: pgmap v11703: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 351 KiB/s wr, 9 op/s 2023-03-09T20:47:55.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20986.286232 secs 2023-03-09T20:47:55.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:55 smithi038 ceph-mon[92422]: pgmap v11703: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 351 KiB/s wr, 9 op/s 2023-03-09T20:47:55.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20986.286232 secs 2023-03-09T20:47:58.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:57 smithi122 ceph-mon[93869]: pgmap v11704: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 351 KiB/s wr, 12 op/s 2023-03-09T20:47:58.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:57 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 20988.601897 secs 2023-03-09T20:47:58.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:57 smithi181 ceph-mon[94189]: pgmap v11704: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 351 KiB/s wr, 12 op/s 2023-03-09T20:47:58.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:57 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 20988.601897 secs 2023-03-09T20:47:58.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:57 smithi038 ceph-mon[92422]: pgmap v11704: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 351 KiB/s wr, 12 op/s 2023-03-09T20:47:58.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:57 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 20988.601897 secs 2023-03-09T20:47:58.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21450.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:48:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:59 smithi122 ceph-mon[93869]: pgmap v11705: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 533 KiB/s wr, 11 op/s 2023-03-09T20:48:00.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:48:00.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:47:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:48:00.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:59 smithi181 ceph-mon[94189]: pgmap v11705: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 533 KiB/s wr, 11 op/s 2023-03-09T20:48:00.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:48:00.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:47:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:48:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:59 smithi038 ceph-mon[92422]: pgmap v11705: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 533 KiB/s wr, 11 op/s 2023-03-09T20:48:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:48:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:47:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:48:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20991.286293 secs 2023-03-09T20:48:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:48:01.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:48:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20991.286293 secs 2023-03-09T20:48:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:48:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:48:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20991.286293 secs 2023-03-09T20:48:01.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:48:01.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:48:02.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:01 smithi122 ceph-mon[93869]: pgmap v11706: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 519 KiB/s wr, 14 op/s 2023-03-09T20:48:02.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:01 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:48:02.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:01 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:48:02.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:01 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:48:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:01 smithi181 ceph-mon[94189]: pgmap v11706: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 519 KiB/s wr, 14 op/s 2023-03-09T20:48:02.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:01 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:48:02.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:01 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:48:02.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:01 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:48:02.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:01 smithi038 ceph-mon[92422]: pgmap v11706: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 519 KiB/s wr, 14 op/s 2023-03-09T20:48:02.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:01 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:48:02.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:01 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:48:02.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:01 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:48:03.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20993.602012 secs 2023-03-09T20:48:03.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:48:03.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:48:03.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:48:03.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20993.602012 secs 2023-03-09T20:48:03.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:48:03.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:48:03.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:48:03.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20993.602012 secs 2023-03-09T20:48:03.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:48:03.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:48:03.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:48:03.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21455.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:48:04.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:03 smithi122 ceph-mon[93869]: pgmap v11707: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 502 KiB/s wr, 11 op/s 2023-03-09T20:48:04.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:03 smithi181 ceph-mon[94189]: pgmap v11707: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 502 KiB/s wr, 11 op/s 2023-03-09T20:48:04.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:03 smithi038 ceph-mon[92422]: pgmap v11707: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 502 KiB/s wr, 11 op/s 2023-03-09T20:48:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:05 smithi122 ceph-mon[93869]: pgmap v11708: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 461 KiB/s wr, 23 op/s 2023-03-09T20:48:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20996.286353 secs 2023-03-09T20:48:05.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:05 smithi181 ceph-mon[94189]: pgmap v11708: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 461 KiB/s wr, 23 op/s 2023-03-09T20:48:05.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20996.286353 secs 2023-03-09T20:48:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:05 smithi038 ceph-mon[92422]: pgmap v11708: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 461 KiB/s wr, 23 op/s 2023-03-09T20:48:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20996.286353 secs 2023-03-09T20:48:06.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:06 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:48:06.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:48:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:06 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:48:06.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:48:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:06 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:48:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:48:07.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:07 smithi122 ceph-mon[93869]: pgmap v11709: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 472 KiB/s wr, 26 op/s 2023-03-09T20:48:07.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 20998.602129 secs 2023-03-09T20:48:07.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:07 smithi181 ceph-mon[94189]: pgmap v11709: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 472 KiB/s wr, 26 op/s 2023-03-09T20:48:07.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 20998.602129 secs 2023-03-09T20:48:07.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:07 smithi038 ceph-mon[92422]: pgmap v11709: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 472 KiB/s wr, 26 op/s 2023-03-09T20:48:07.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 20998.602129 secs 2023-03-09T20:48:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21460.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:48:10.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:09 smithi122 ceph-mon[93869]: pgmap v11710: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 507 KiB/s rd, 541 KiB/s wr, 25 op/s 2023-03-09T20:48:10.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:09 smithi181 ceph-mon[94189]: pgmap v11710: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 507 KiB/s rd, 541 KiB/s wr, 25 op/s 2023-03-09T20:48:10.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:09 smithi038 ceph-mon[92422]: pgmap v11710: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 507 KiB/s rd, 541 KiB/s wr, 25 op/s 2023-03-09T20:48:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21001.286415 secs 2023-03-09T20:48:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:10 smithi122 ceph-mon[93869]: osdmap e945: 12 total, 12 up, 12 in 2023-03-09T20:48:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21001.286415 secs 2023-03-09T20:48:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:10 smithi181 ceph-mon[94189]: osdmap e945: 12 total, 12 up, 12 in 2023-03-09T20:48:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21001.286415 secs 2023-03-09T20:48:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:10 smithi038 ceph-mon[92422]: osdmap e945: 12 total, 12 up, 12 in 2023-03-09T20:48:12.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:11 smithi122 ceph-mon[93869]: pgmap v11712: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 388 KiB/s wr, 27 op/s 2023-03-09T20:48:12.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:11 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:48:12.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:11 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:48:12.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:11 smithi181 ceph-mon[94189]: pgmap v11712: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 388 KiB/s wr, 27 op/s 2023-03-09T20:48:12.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:11 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:48:12.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:11 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:48:12.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:11 smithi038 ceph-mon[92422]: pgmap v11712: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 388 KiB/s wr, 27 op/s 2023-03-09T20:48:12.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:11 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:48:12.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:11 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:48:13.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:48:13.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:48:13.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21003.602222 secs 2023-03-09T20:48:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:48:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:48:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21003.602222 secs 2023-03-09T20:48:13.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:48:13.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:48:13.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21003.602222 secs 2023-03-09T20:48:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21465.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:48:14.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:13 smithi122 ceph-mon[93869]: pgmap v11713: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 388 KiB/s wr, 27 op/s 2023-03-09T20:48:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:13 smithi181 ceph-mon[94189]: pgmap v11713: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 388 KiB/s wr, 27 op/s 2023-03-09T20:48:14.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:13 smithi038 ceph-mon[92422]: pgmap v11713: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 388 KiB/s wr, 27 op/s 2023-03-09T20:48:15.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:15 smithi122 ceph-mon[93869]: pgmap v11714: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 309 KiB/s wr, 12 op/s 2023-03-09T20:48:15.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21006.286474 secs 2023-03-09T20:48:15.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:15 smithi181 ceph-mon[94189]: pgmap v11714: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 309 KiB/s wr, 12 op/s 2023-03-09T20:48:15.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21006.286474 secs 2023-03-09T20:48:15.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:15 smithi038 ceph-mon[92422]: pgmap v11714: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 309 KiB/s wr, 12 op/s 2023-03-09T20:48:15.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21006.286474 secs 2023-03-09T20:48:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:17 smithi122 ceph-mon[93869]: pgmap v11715: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 315 KiB/s wr, 11 op/s 2023-03-09T20:48:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21008.602339 secs 2023-03-09T20:48:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:17 smithi181 ceph-mon[94189]: pgmap v11715: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 315 KiB/s wr, 11 op/s 2023-03-09T20:48:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21008.602339 secs 2023-03-09T20:48:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:17 smithi038 ceph-mon[92422]: pgmap v11715: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 315 KiB/s wr, 11 op/s 2023-03-09T20:48:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21008.602339 secs 2023-03-09T20:48:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21470.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:48:20.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:19 smithi122 ceph-mon[93869]: pgmap v11716: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 232 KiB/s wr, 10 op/s 2023-03-09T20:48:20.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:19 smithi122 ceph-mon[93869]: osdmap e946: 12 total, 12 up, 12 in 2023-03-09T20:48:20.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:19 smithi181 ceph-mon[94189]: pgmap v11716: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 232 KiB/s wr, 10 op/s 2023-03-09T20:48:20.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:19 smithi181 ceph-mon[94189]: osdmap e946: 12 total, 12 up, 12 in 2023-03-09T20:48:20.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:19 smithi038 ceph-mon[92422]: pgmap v11716: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 232 KiB/s wr, 10 op/s 2023-03-09T20:48:20.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:19 smithi038 ceph-mon[92422]: osdmap e946: 12 total, 12 up, 12 in 2023-03-09T20:48:21.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21011.286537 secs 2023-03-09T20:48:21.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21011.286537 secs 2023-03-09T20:48:21.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21011.286537 secs 2023-03-09T20:48:22.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:21 smithi122 ceph-mon[93869]: pgmap v11718: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 88 KiB/s wr, 9 op/s 2023-03-09T20:48:22.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:21 smithi181 ceph-mon[94189]: pgmap v11718: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 88 KiB/s wr, 9 op/s 2023-03-09T20:48:22.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:21 smithi038 ceph-mon[92422]: pgmap v11718: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 88 KiB/s wr, 9 op/s 2023-03-09T20:48:23.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21013.602454 secs 2023-03-09T20:48:23.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21013.602454 secs 2023-03-09T20:48:23.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21013.602454 secs 2023-03-09T20:48:23.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21475.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:48:24.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:23 smithi122 ceph-mon[93869]: pgmap v11719: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 88 KiB/s wr, 8 op/s 2023-03-09T20:48:24.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:23 smithi181 ceph-mon[94189]: pgmap v11719: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 88 KiB/s wr, 8 op/s 2023-03-09T20:48:24.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:23 smithi038 ceph-mon[92422]: pgmap v11719: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 88 KiB/s wr, 8 op/s 2023-03-09T20:48:25.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:25 smithi122 ceph-mon[93869]: pgmap v11720: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 119 KiB/s wr, 8 op/s 2023-03-09T20:48:25.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21016.286597 secs 2023-03-09T20:48:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:25 smithi181 ceph-mon[94189]: pgmap v11720: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 119 KiB/s wr, 8 op/s 2023-03-09T20:48:25.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21016.286597 secs 2023-03-09T20:48:25.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:25 smithi038 ceph-mon[92422]: pgmap v11720: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 119 KiB/s wr, 8 op/s 2023-03-09T20:48:25.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21016.286597 secs 2023-03-09T20:48:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:27 smithi122 ceph-mon[93869]: pgmap v11721: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 99 KiB/s wr, 7 op/s 2023-03-09T20:48:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21018.602583 secs 2023-03-09T20:48:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:27 smithi181 ceph-mon[94189]: pgmap v11721: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 99 KiB/s wr, 7 op/s 2023-03-09T20:48:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21018.602583 secs 2023-03-09T20:48:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:27 smithi038 ceph-mon[92422]: pgmap v11721: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 99 KiB/s wr, 7 op/s 2023-03-09T20:48:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21018.602583 secs 2023-03-09T20:48:28.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21480.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:48:30.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:29 smithi122 ceph-mon[93869]: pgmap v11722: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 99 KiB/s wr, 7 op/s 2023-03-09T20:48:30.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:29 smithi181 ceph-mon[94189]: pgmap v11722: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 99 KiB/s wr, 7 op/s 2023-03-09T20:48:30.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:29 smithi038 ceph-mon[92422]: pgmap v11722: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 99 KiB/s wr, 7 op/s 2023-03-09T20:48:31.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21021.286655 secs 2023-03-09T20:48:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21021.286655 secs 2023-03-09T20:48:31.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21021.286655 secs 2023-03-09T20:48:32.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:31 smithi122 ceph-mon[93869]: pgmap v11723: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 87 KiB/s wr, 9 op/s 2023-03-09T20:48:32.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:31 smithi181 ceph-mon[94189]: pgmap v11723: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 87 KiB/s wr, 9 op/s 2023-03-09T20:48:32.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:31 smithi038 ceph-mon[92422]: pgmap v11723: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 87 KiB/s wr, 9 op/s 2023-03-09T20:48:33.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21023.602701 secs 2023-03-09T20:48:33.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21023.602701 secs 2023-03-09T20:48:33.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21023.602701 secs 2023-03-09T20:48:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21485.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:48:34.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:33 smithi122 ceph-mon[93869]: pgmap v11724: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 82 KiB/s wr, 6 op/s 2023-03-09T20:48:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:33 smithi181 ceph-mon[94189]: pgmap v11724: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 82 KiB/s wr, 6 op/s 2023-03-09T20:48:34.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:33 smithi038 ceph-mon[92422]: pgmap v11724: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 82 KiB/s wr, 6 op/s 2023-03-09T20:48:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:35 smithi122 ceph-mon[93869]: pgmap v11725: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 82 KiB/s wr, 7 op/s 2023-03-09T20:48:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21026.286717 secs 2023-03-09T20:48:35.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:35 smithi181 ceph-mon[94189]: pgmap v11725: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 82 KiB/s wr, 7 op/s 2023-03-09T20:48:35.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21026.286717 secs 2023-03-09T20:48:35.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:35 smithi038 ceph-mon[92422]: pgmap v11725: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 82 KiB/s wr, 7 op/s 2023-03-09T20:48:35.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21026.286717 secs 2023-03-09T20:48:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:37 smithi122 ceph-mon[93869]: pgmap v11726: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.5 KiB/s wr, 8 op/s 2023-03-09T20:48:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21028.602823 secs 2023-03-09T20:48:38.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:37 smithi181 ceph-mon[94189]: pgmap v11726: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.5 KiB/s wr, 8 op/s 2023-03-09T20:48:38.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21028.602823 secs 2023-03-09T20:48:38.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:37 smithi038 ceph-mon[92422]: pgmap v11726: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.5 KiB/s wr, 8 op/s 2023-03-09T20:48:38.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21028.602823 secs 2023-03-09T20:48:38.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21490.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:48:40.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:39 smithi122 ceph-mon[93869]: pgmap v11727: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2023-03-09T20:48:40.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:39 smithi181 ceph-mon[94189]: pgmap v11727: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2023-03-09T20:48:40.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:39 smithi038 ceph-mon[92422]: pgmap v11727: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2023-03-09T20:48:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21031.286777 secs 2023-03-09T20:48:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:48:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:48:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21031.286777 secs 2023-03-09T20:48:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:48:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:48:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21031.286777 secs 2023-03-09T20:48:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:48:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:48:42.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:41 smithi122 ceph-mon[93869]: pgmap v11728: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 102 KiB/s wr, 9 op/s 2023-03-09T20:48:42.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:41 smithi181 ceph-mon[94189]: pgmap v11728: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 102 KiB/s wr, 9 op/s 2023-03-09T20:48:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:41 smithi038 ceph-mon[92422]: pgmap v11728: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 102 KiB/s wr, 9 op/s 2023-03-09T20:48:43.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21033.602956 secs 2023-03-09T20:48:43.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21033.602956 secs 2023-03-09T20:48:43.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21033.602956 secs 2023-03-09T20:48:43.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21495.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:48:44.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:43 smithi122 ceph-mon[93869]: pgmap v11729: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 102 KiB/s wr, 6 op/s 2023-03-09T20:48:44.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:43 smithi181 ceph-mon[94189]: pgmap v11729: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 102 KiB/s wr, 6 op/s 2023-03-09T20:48:44.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:43 smithi038 ceph-mon[92422]: pgmap v11729: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 102 KiB/s wr, 6 op/s 2023-03-09T20:48:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:45 smithi122 ceph-mon[93869]: pgmap v11730: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 422 KiB/s rd, 336 KiB/s wr, 8 op/s 2023-03-09T20:48:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21036.286836 secs 2023-03-09T20:48:45.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:45 smithi181 ceph-mon[94189]: pgmap v11730: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 422 KiB/s rd, 336 KiB/s wr, 8 op/s 2023-03-09T20:48:45.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21036.286836 secs 2023-03-09T20:48:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:45 smithi038 ceph-mon[92422]: pgmap v11730: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 422 KiB/s rd, 336 KiB/s wr, 8 op/s 2023-03-09T20:48:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21036.286836 secs 2023-03-09T20:48:47.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:48:47.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:48:47.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:47 smithi122 ceph-mon[93869]: pgmap v11731: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 369 KiB/s wr, 10 op/s 2023-03-09T20:48:47.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21038.603085 secs 2023-03-09T20:48:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:48:47.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:48:47.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:47 smithi181 ceph-mon[94189]: pgmap v11731: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 369 KiB/s wr, 10 op/s 2023-03-09T20:48:47.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21038.603085 secs 2023-03-09T20:48:47.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:48:47.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:48:47.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:47 smithi038 ceph-mon[92422]: pgmap v11731: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 369 KiB/s wr, 10 op/s 2023-03-09T20:48:47.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21038.603085 secs 2023-03-09T20:48:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21500.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:48:50.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:49 smithi122 ceph-mon[93869]: pgmap v11732: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 457 KiB/s rd, 369 KiB/s wr, 9 op/s 2023-03-09T20:48:50.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:49 smithi181 ceph-mon[94189]: pgmap v11732: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 457 KiB/s rd, 369 KiB/s wr, 9 op/s 2023-03-09T20:48:50.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:49 smithi038 ceph-mon[92422]: pgmap v11732: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 457 KiB/s rd, 369 KiB/s wr, 9 op/s 2023-03-09T20:48:51.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21041.286898 secs 2023-03-09T20:48:51.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21041.286898 secs 2023-03-09T20:48:51.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21041.286898 secs 2023-03-09T20:48:52.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:51 smithi122 ceph-mon[93869]: pgmap v11733: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 778 KiB/s rd, 629 KiB/s wr, 14 op/s 2023-03-09T20:48:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:51 smithi181 ceph-mon[94189]: pgmap v11733: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 778 KiB/s rd, 629 KiB/s wr, 14 op/s 2023-03-09T20:48:52.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:51 smithi038 ceph-mon[92422]: pgmap v11733: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 778 KiB/s rd, 629 KiB/s wr, 14 op/s 2023-03-09T20:48:53.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:48:53.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:48:53.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21043.603202 secs 2023-03-09T20:48:53.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:48:53.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:48:53.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21043.603202 secs 2023-03-09T20:48:53.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:48:53.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:48:53.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21043.603202 secs 2023-03-09T20:48:53.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21505.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:48:54.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:53 smithi122 ceph-mon[93869]: pgmap v11734: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 627 KiB/s rd, 528 KiB/s wr, 11 op/s 2023-03-09T20:48:54.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:53 smithi181 ceph-mon[94189]: pgmap v11734: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 627 KiB/s rd, 528 KiB/s wr, 11 op/s 2023-03-09T20:48:54.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:53 smithi038 ceph-mon[92422]: pgmap v11734: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 627 KiB/s rd, 528 KiB/s wr, 11 op/s 2023-03-09T20:48:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:55 smithi122 ceph-mon[93869]: pgmap v11735: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 646 KiB/s rd, 528 KiB/s wr, 11 op/s 2023-03-09T20:48:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21046.286956 secs 2023-03-09T20:48:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:48:56.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:48:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:55 smithi181 ceph-mon[94189]: pgmap v11735: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 646 KiB/s rd, 528 KiB/s wr, 11 op/s 2023-03-09T20:48:56.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21046.286956 secs 2023-03-09T20:48:56.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:48:56.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:48:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:55 smithi038 ceph-mon[92422]: pgmap v11735: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 646 KiB/s rd, 528 KiB/s wr, 11 op/s 2023-03-09T20:48:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21046.286956 secs 2023-03-09T20:48:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:48:56.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:48:58.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:57 smithi122 ceph-mon[93869]: pgmap v11736: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 617 KiB/s rd, 517 KiB/s wr, 13 op/s 2023-03-09T20:48:58.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21048.603321 secs 2023-03-09T20:48:58.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:57 smithi181 ceph-mon[94189]: pgmap v11736: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 617 KiB/s rd, 517 KiB/s wr, 13 op/s 2023-03-09T20:48:58.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21048.603321 secs 2023-03-09T20:48:58.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:57 smithi038 ceph-mon[92422]: pgmap v11736: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 617 KiB/s rd, 517 KiB/s wr, 13 op/s 2023-03-09T20:48:58.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21048.603321 secs 2023-03-09T20:48:58.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21510.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:49:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:59 smithi122 ceph-mon[93869]: pgmap v11737: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 580 KiB/s rd, 484 KiB/s wr, 11 op/s 2023-03-09T20:49:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:49:00.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:48:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:49:00.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:59 smithi181 ceph-mon[94189]: pgmap v11737: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 580 KiB/s rd, 484 KiB/s wr, 11 op/s 2023-03-09T20:49:00.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:49:00.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:48:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:49:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:59 smithi038 ceph-mon[92422]: pgmap v11737: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 580 KiB/s rd, 484 KiB/s wr, 11 op/s 2023-03-09T20:49:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:49:00.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:48:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:49:01.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21051.287016 secs 2023-03-09T20:49:01.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:49:01.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:49:01.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21051.287016 secs 2023-03-09T20:49:01.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:49:01.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:49:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21051.287016 secs 2023-03-09T20:49:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:49:01.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:49:02.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:01 smithi122 ceph-mon[93869]: pgmap v11738: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 610 KiB/s rd, 513 KiB/s wr, 14 op/s 2023-03-09T20:49:02.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:01 smithi181 ceph-mon[94189]: pgmap v11738: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 610 KiB/s rd, 513 KiB/s wr, 14 op/s 2023-03-09T20:49:02.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:01 smithi038 ceph-mon[92422]: pgmap v11738: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 610 KiB/s rd, 513 KiB/s wr, 14 op/s 2023-03-09T20:49:03.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21053.603386 secs 2023-03-09T20:49:03.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:49:03.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21053.603386 secs 2023-03-09T20:49:03.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:49:03.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21053.603386 secs 2023-03-09T20:49:03.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:49:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21515.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:49:04.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:04 smithi122 ceph-mon[93869]: pgmap v11739: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 253 KiB/s wr, 9 op/s 2023-03-09T20:49:04.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:49:04.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:49:04.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:49:04.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:49:04.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:49:04.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:49:04.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:49:04.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:04 smithi181 ceph-mon[94189]: pgmap v11739: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 253 KiB/s wr, 9 op/s 2023-03-09T20:49:04.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:49:04.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:49:04.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:49:04.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:49:04.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:49:04.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:49:04.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:49:04.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:04 smithi038 ceph-mon[92422]: pgmap v11739: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 253 KiB/s wr, 9 op/s 2023-03-09T20:49:04.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:49:04.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:49:04.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:49:04.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:49:04.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:49:04.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:49:04.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:49:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:05 smithi122 ceph-mon[93869]: osdmap e947: 12 total, 12 up, 12 in 2023-03-09T20:49:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:05 smithi122 ceph-mon[93869]: pgmap v11741: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 676 KiB/s rd, 671 KiB/s wr, 26 op/s 2023-03-09T20:49:06.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21056.287079 secs 2023-03-09T20:49:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:05 smithi181 ceph-mon[94189]: osdmap e947: 12 total, 12 up, 12 in 2023-03-09T20:49:06.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:05 smithi181 ceph-mon[94189]: pgmap v11741: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 676 KiB/s rd, 671 KiB/s wr, 26 op/s 2023-03-09T20:49:06.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21056.287079 secs 2023-03-09T20:49:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:05 smithi038 ceph-mon[92422]: osdmap e947: 12 total, 12 up, 12 in 2023-03-09T20:49:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:05 smithi038 ceph-mon[92422]: pgmap v11741: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 676 KiB/s rd, 671 KiB/s wr, 26 op/s 2023-03-09T20:49:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21056.287079 secs 2023-03-09T20:49:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:07 smithi122 ceph-mon[93869]: pgmap v11742: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 502 KiB/s wr, 28 op/s 2023-03-09T20:49:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21058.603517 secs 2023-03-09T20:49:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:07 smithi181 ceph-mon[94189]: pgmap v11742: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 502 KiB/s wr, 28 op/s 2023-03-09T20:49:08.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21058.603517 secs 2023-03-09T20:49:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:07 smithi038 ceph-mon[92422]: pgmap v11742: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 502 KiB/s wr, 28 op/s 2023-03-09T20:49:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21058.603517 secs 2023-03-09T20:49:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21520.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:49:10.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:09 smithi122 ceph-mon[93869]: pgmap v11743: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 502 KiB/s wr, 28 op/s 2023-03-09T20:49:10.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:09 smithi181 ceph-mon[94189]: pgmap v11743: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 502 KiB/s wr, 28 op/s 2023-03-09T20:49:10.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:09 smithi038 ceph-mon[92422]: pgmap v11743: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 502 KiB/s wr, 28 op/s 2023-03-09T20:49:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21061.287138 secs 2023-03-09T20:49:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21061.287138 secs 2023-03-09T20:49:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21061.287138 secs 2023-03-09T20:49:12.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:11 smithi122 ceph-mon[93869]: pgmap v11744: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 650 KiB/s rd, 691 KiB/s wr, 28 op/s 2023-03-09T20:49:12.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:11 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:49:12.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:11 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:49:12.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:11 smithi181 ceph-mon[94189]: pgmap v11744: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 650 KiB/s rd, 691 KiB/s wr, 28 op/s 2023-03-09T20:49:12.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:11 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:49:12.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:11 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:49:12.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:11 smithi038 ceph-mon[92422]: pgmap v11744: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 650 KiB/s rd, 691 KiB/s wr, 28 op/s 2023-03-09T20:49:12.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:11 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:49:12.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:11 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:49:13.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:49:13.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:49:13.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21063.603639 secs 2023-03-09T20:49:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:49:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:49:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21063.603639 secs 2023-03-09T20:49:13.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:49:13.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:49:13.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21063.603639 secs 2023-03-09T20:49:13.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21525.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:49:14.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:13 smithi122 ceph-mon[93869]: pgmap v11745: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 650 KiB/s rd, 691 KiB/s wr, 29 op/s 2023-03-09T20:49:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:13 smithi181 ceph-mon[94189]: pgmap v11745: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 650 KiB/s rd, 691 KiB/s wr, 29 op/s 2023-03-09T20:49:14.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:13 smithi038 ceph-mon[92422]: pgmap v11745: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 650 KiB/s rd, 691 KiB/s wr, 29 op/s 2023-03-09T20:49:15.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:15 smithi122 ceph-mon[93869]: osdmap e948: 12 total, 12 up, 12 in 2023-03-09T20:49:15.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:15 smithi122 ceph-mon[93869]: pgmap v11747: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 323 KiB/s wr, 12 op/s 2023-03-09T20:49:15.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21066.287202 secs 2023-03-09T20:49:15.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:15 smithi181 ceph-mon[94189]: osdmap e948: 12 total, 12 up, 12 in 2023-03-09T20:49:15.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:15 smithi181 ceph-mon[94189]: pgmap v11747: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 323 KiB/s wr, 12 op/s 2023-03-09T20:49:15.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21066.287202 secs 2023-03-09T20:49:15.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:15 smithi038 ceph-mon[92422]: osdmap e948: 12 total, 12 up, 12 in 2023-03-09T20:49:15.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:15 smithi038 ceph-mon[92422]: pgmap v11747: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 323 KiB/s wr, 12 op/s 2023-03-09T20:49:15.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21066.287202 secs 2023-03-09T20:49:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:17 smithi122 ceph-mon[93869]: pgmap v11748: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 305 KiB/s wr, 9 op/s 2023-03-09T20:49:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21068.603758 secs 2023-03-09T20:49:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:17 smithi181 ceph-mon[94189]: pgmap v11748: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 305 KiB/s wr, 9 op/s 2023-03-09T20:49:18.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21068.603758 secs 2023-03-09T20:49:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:17 smithi038 ceph-mon[92422]: pgmap v11748: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 305 KiB/s wr, 9 op/s 2023-03-09T20:49:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21068.603758 secs 2023-03-09T20:49:18.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21530.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:49:20.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:19 smithi122 ceph-mon[93869]: pgmap v11749: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 304 KiB/s wr, 8 op/s 2023-03-09T20:49:20.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:19 smithi181 ceph-mon[94189]: pgmap v11749: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 304 KiB/s wr, 8 op/s 2023-03-09T20:49:20.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:19 smithi038 ceph-mon[92422]: pgmap v11749: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 304 KiB/s wr, 8 op/s 2023-03-09T20:49:21.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21071.287262 secs 2023-03-09T20:49:21.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21071.287262 secs 2023-03-09T20:49:21.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21071.287262 secs 2023-03-09T20:49:22.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:21 smithi122 ceph-mon[93869]: pgmap v11750: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 102 KiB/s wr, 7 op/s 2023-03-09T20:49:22.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:21 smithi181 ceph-mon[94189]: pgmap v11750: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 102 KiB/s wr, 7 op/s 2023-03-09T20:49:22.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:21 smithi038 ceph-mon[92422]: pgmap v11750: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 102 KiB/s wr, 7 op/s 2023-03-09T20:49:23.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21073.603877 secs 2023-03-09T20:49:23.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21073.603877 secs 2023-03-09T20:49:23.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21073.603877 secs 2023-03-09T20:49:23.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21535.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:49:24.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:23 smithi122 ceph-mon[93869]: pgmap v11751: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 102 KiB/s wr, 7 op/s 2023-03-09T20:49:24.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:23 smithi181 ceph-mon[94189]: pgmap v11751: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 102 KiB/s wr, 7 op/s 2023-03-09T20:49:24.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:23 smithi038 ceph-mon[92422]: pgmap v11751: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 102 KiB/s wr, 7 op/s 2023-03-09T20:49:25.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:25 smithi122 ceph-mon[93869]: pgmap v11752: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 97 KiB/s wr, 7 op/s 2023-03-09T20:49:25.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21076.287322 secs 2023-03-09T20:49:25.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:25 smithi181 ceph-mon[94189]: pgmap v11752: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 97 KiB/s wr, 7 op/s 2023-03-09T20:49:25.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21076.287322 secs 2023-03-09T20:49:25.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:25 smithi038 ceph-mon[92422]: pgmap v11752: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 97 KiB/s wr, 7 op/s 2023-03-09T20:49:25.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21076.287322 secs 2023-03-09T20:49:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:27 smithi122 ceph-mon[93869]: pgmap v11753: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 166 KiB/s wr, 9 op/s 2023-03-09T20:49:28.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21078.603995 secs 2023-03-09T20:49:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:27 smithi181 ceph-mon[94189]: pgmap v11753: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 166 KiB/s wr, 9 op/s 2023-03-09T20:49:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21078.603995 secs 2023-03-09T20:49:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:27 smithi038 ceph-mon[92422]: pgmap v11753: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 166 KiB/s wr, 9 op/s 2023-03-09T20:49:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21078.603995 secs 2023-03-09T20:49:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21540.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:49:30.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:29 smithi122 ceph-mon[93869]: pgmap v11754: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 99 KiB/s wr, 6 op/s 2023-03-09T20:49:30.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:29 smithi181 ceph-mon[94189]: pgmap v11754: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 99 KiB/s wr, 6 op/s 2023-03-09T20:49:30.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:29 smithi038 ceph-mon[92422]: pgmap v11754: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 99 KiB/s wr, 6 op/s 2023-03-09T20:49:31.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21081.287383 secs 2023-03-09T20:49:31.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21081.287383 secs 2023-03-09T20:49:31.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21081.287383 secs 2023-03-09T20:49:32.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:31 smithi122 ceph-mon[93869]: pgmap v11755: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 100 KiB/s wr, 9 op/s 2023-03-09T20:49:32.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:31 smithi181 ceph-mon[94189]: pgmap v11755: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 100 KiB/s wr, 9 op/s 2023-03-09T20:49:32.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:31 smithi038 ceph-mon[92422]: pgmap v11755: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 100 KiB/s wr, 9 op/s 2023-03-09T20:49:33.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21083.604135 secs 2023-03-09T20:49:33.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21083.604135 secs 2023-03-09T20:49:33.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21083.604135 secs 2023-03-09T20:49:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21545.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:49:34.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:33 smithi122 ceph-mon[93869]: pgmap v11756: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 82 KiB/s wr, 6 op/s 2023-03-09T20:49:34.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:33 smithi181 ceph-mon[94189]: pgmap v11756: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 82 KiB/s wr, 6 op/s 2023-03-09T20:49:34.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:33 smithi038 ceph-mon[92422]: pgmap v11756: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 82 KiB/s wr, 6 op/s 2023-03-09T20:49:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:35 smithi122 ceph-mon[93869]: pgmap v11757: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 82 KiB/s wr, 6 op/s 2023-03-09T20:49:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21086.287445 secs 2023-03-09T20:49:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:35 smithi181 ceph-mon[94189]: pgmap v11757: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 82 KiB/s wr, 6 op/s 2023-03-09T20:49:35.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21086.287445 secs 2023-03-09T20:49:35.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:35 smithi038 ceph-mon[92422]: pgmap v11757: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 82 KiB/s wr, 6 op/s 2023-03-09T20:49:35.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21086.287445 secs 2023-03-09T20:49:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:37 smithi122 ceph-mon[93869]: pgmap v11758: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 164 KiB/s wr, 9 op/s 2023-03-09T20:49:38.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:37 smithi181 ceph-mon[94189]: pgmap v11758: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 164 KiB/s wr, 9 op/s 2023-03-09T20:49:38.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:37 smithi038 ceph-mon[92422]: pgmap v11758: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 164 KiB/s wr, 9 op/s 2023-03-09T20:49:38.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21550.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:49:40.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:39 smithi122 ceph-mon[93869]: pgmap v11759: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 82 KiB/s wr, 6 op/s 2023-03-09T20:49:40.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:39 smithi181 ceph-mon[94189]: pgmap v11759: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 82 KiB/s wr, 6 op/s 2023-03-09T20:49:40.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:39 smithi038 ceph-mon[92422]: pgmap v11759: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 82 KiB/s wr, 6 op/s 2023-03-09T20:49:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21091.287532 secs 2023-03-09T20:49:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:40 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:49:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:40 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:49:41.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:49:41.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:49:41.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:49:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21091.287532 secs 2023-03-09T20:49:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:40 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:49:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:40 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:49:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:49:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:49:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:49:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21091.287532 secs 2023-03-09T20:49:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:40 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:49:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:40 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:49:41.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:49:41.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:49:41.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:49:42.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:41 smithi122 ceph-mon[93869]: pgmap v11760: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 171 KiB/s wr, 9 op/s 2023-03-09T20:49:42.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:41 smithi181 ceph-mon[94189]: pgmap v11760: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 171 KiB/s wr, 9 op/s 2023-03-09T20:49:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:41 smithi038 ceph-mon[92422]: pgmap v11760: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 171 KiB/s wr, 9 op/s 2023-03-09T20:49:43.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21555.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:49:44.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:43 smithi122 ceph-mon[93869]: pgmap v11761: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 171 KiB/s wr, 6 op/s 2023-03-09T20:49:44.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:43 smithi181 ceph-mon[94189]: pgmap v11761: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 171 KiB/s wr, 6 op/s 2023-03-09T20:49:44.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:43 smithi038 ceph-mon[92422]: pgmap v11761: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 171 KiB/s wr, 6 op/s 2023-03-09T20:49:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:45 smithi122 ceph-mon[93869]: pgmap v11762: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 171 KiB/s wr, 6 op/s 2023-03-09T20:49:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21096.287595 secs 2023-03-09T20:49:45.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:45 smithi181 ceph-mon[94189]: pgmap v11762: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 171 KiB/s wr, 6 op/s 2023-03-09T20:49:45.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21096.287595 secs 2023-03-09T20:49:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:45 smithi038 ceph-mon[92422]: pgmap v11762: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 171 KiB/s wr, 6 op/s 2023-03-09T20:49:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21096.287595 secs 2023-03-09T20:49:47.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:49:47.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:49:47.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:47 smithi122 ceph-mon[93869]: pgmap v11763: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 343 KiB/s wr, 9 op/s 2023-03-09T20:49:47.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:49:47.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:49:47.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:47 smithi181 ceph-mon[94189]: pgmap v11763: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 343 KiB/s wr, 9 op/s 2023-03-09T20:49:47.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:49:47.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:49:47.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:47 smithi038 ceph-mon[92422]: pgmap v11763: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 343 KiB/s wr, 9 op/s 2023-03-09T20:49:48.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21560.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:49:50.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:49 smithi122 ceph-mon[93869]: pgmap v11764: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 261 KiB/s wr, 6 op/s 2023-03-09T20:49:50.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:49 smithi181 ceph-mon[94189]: pgmap v11764: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 261 KiB/s wr, 6 op/s 2023-03-09T20:49:50.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:49 smithi038 ceph-mon[92422]: pgmap v11764: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 261 KiB/s wr, 6 op/s 2023-03-09T20:49:51.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21101.287657 secs 2023-03-09T20:49:51.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21101.287657 secs 2023-03-09T20:49:51.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21101.287657 secs 2023-03-09T20:49:52.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:51 smithi122 ceph-mon[93869]: pgmap v11765: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 321 KiB/s rd, 304 KiB/s wr, 10 op/s 2023-03-09T20:49:52.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:51 smithi181 ceph-mon[94189]: pgmap v11765: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 321 KiB/s rd, 304 KiB/s wr, 10 op/s 2023-03-09T20:49:52.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:51 smithi038 ceph-mon[92422]: pgmap v11765: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 321 KiB/s rd, 304 KiB/s wr, 10 op/s 2023-03-09T20:49:53.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21565.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:49:54.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:53 smithi122 ceph-mon[93869]: pgmap v11766: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 215 KiB/s wr, 7 op/s 2023-03-09T20:49:54.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:53 smithi181 ceph-mon[94189]: pgmap v11766: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 215 KiB/s wr, 7 op/s 2023-03-09T20:49:54.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:53 smithi038 ceph-mon[92422]: pgmap v11766: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 215 KiB/s wr, 7 op/s 2023-03-09T20:49:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:55 smithi122 ceph-mon[93869]: pgmap v11767: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 419 KiB/s wr, 9 op/s 2023-03-09T20:49:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21106.287717 secs 2023-03-09T20:49:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:49:56.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:49:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:55 smithi181 ceph-mon[94189]: pgmap v11767: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 419 KiB/s wr, 9 op/s 2023-03-09T20:49:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21106.287717 secs 2023-03-09T20:49:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:49:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:49:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:55 smithi038 ceph-mon[92422]: pgmap v11767: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 419 KiB/s wr, 9 op/s 2023-03-09T20:49:56.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21106.287717 secs 2023-03-09T20:49:56.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:49:56.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:49:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:49:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:49:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:57 smithi122 ceph-mon[93869]: pgmap v11768: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 452 KiB/s rd, 436 KiB/s wr, 12 op/s 2023-03-09T20:49:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:49:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:49:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:57 smithi181 ceph-mon[94189]: pgmap v11768: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 452 KiB/s rd, 436 KiB/s wr, 12 op/s 2023-03-09T20:49:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:49:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:49:57.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:57 smithi038 ceph-mon[92422]: pgmap v11768: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 452 KiB/s rd, 436 KiB/s wr, 12 op/s 2023-03-09T20:49:58.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21570.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:50:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:49:59 smithi122 ceph-mon[93869]: pgmap v11769: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 365 KiB/s wr, 10 op/s 2023-03-09T20:50:00.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:49:59 smithi181 ceph-mon[94189]: pgmap v11769: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 365 KiB/s wr, 10 op/s 2023-03-09T20:50:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:49:59 smithi038 ceph-mon[92422]: pgmap v11769: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 365 KiB/s wr, 10 op/s 2023-03-09T20:50:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21111.287791 secs 2023-03-09T20:50:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:00 smithi122 ceph-mon[93869]: overall HEALTH_WARN 1 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-09T20:50:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21111.287791 secs 2023-03-09T20:50:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:00 smithi181 ceph-mon[94189]: overall HEALTH_WARN 1 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-09T20:50:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21111.287791 secs 2023-03-09T20:50:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:00 smithi038 ceph-mon[92422]: overall HEALTH_WARN 1 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-09T20:50:02.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:01 smithi122 ceph-mon[93869]: pgmap v11770: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 365 KiB/s wr, 12 op/s 2023-03-09T20:50:02.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:01 smithi181 ceph-mon[94189]: pgmap v11770: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 365 KiB/s wr, 12 op/s 2023-03-09T20:50:02.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:01 smithi038 ceph-mon[92422]: pgmap v11770: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 365 KiB/s wr, 12 op/s 2023-03-09T20:50:03.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21575.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:50:04.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:03 smithi122 ceph-mon[93869]: pgmap v11771: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 322 KiB/s wr, 9 op/s 2023-03-09T20:50:04.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:03 smithi181 ceph-mon[94189]: pgmap v11771: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 322 KiB/s wr, 9 op/s 2023-03-09T20:50:04.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:03 smithi038 ceph-mon[92422]: pgmap v11771: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 322 KiB/s wr, 9 op/s 2023-03-09T20:50:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21116.287852 secs 2023-03-09T20:50:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:05 smithi122 ceph-mon[93869]: pgmap v11772: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 371 KiB/s wr, 21 op/s 2023-03-09T20:50:05.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21116.287852 secs 2023-03-09T20:50:05.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:05 smithi181 ceph-mon[94189]: pgmap v11772: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 371 KiB/s wr, 21 op/s 2023-03-09T20:50:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21116.287852 secs 2023-03-09T20:50:05.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:05 smithi038 ceph-mon[92422]: pgmap v11772: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 371 KiB/s wr, 21 op/s 2023-03-09T20:50:07.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:50:07.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:50:07.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:07 smithi122 ceph-mon[93869]: pgmap v11773: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 478 KiB/s rd, 503 KiB/s wr, 24 op/s 2023-03-09T20:50:07.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:50:07.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:50:07.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:07 smithi181 ceph-mon[94189]: pgmap v11773: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 478 KiB/s rd, 503 KiB/s wr, 24 op/s 2023-03-09T20:50:07.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:50:07.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:50:07.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:07 smithi038 ceph-mon[92422]: pgmap v11773: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 478 KiB/s rd, 503 KiB/s wr, 24 op/s 2023-03-09T20:50:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21580.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:50:09.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:50:09.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:50:09.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:09 smithi122 ceph-mon[93869]: pgmap v11774: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 486 KiB/s wr, 22 op/s 2023-03-09T20:50:09.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:50:09.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:50:09.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:09 smithi181 ceph-mon[94189]: pgmap v11774: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 486 KiB/s wr, 22 op/s 2023-03-09T20:50:09.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:50:09.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:50:09.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:09 smithi038 ceph-mon[92422]: pgmap v11774: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 486 KiB/s wr, 22 op/s 2023-03-09T20:50:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21121.287912 secs 2023-03-09T20:50:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21121.287912 secs 2023-03-09T20:50:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21121.287912 secs 2023-03-09T20:50:11.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:11 smithi122 ceph-mon[93869]: pgmap v11775: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 528 KiB/s wr, 25 op/s 2023-03-09T20:50:11.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:11 smithi181 ceph-mon[94189]: pgmap v11775: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 528 KiB/s wr, 25 op/s 2023-03-09T20:50:11.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:11 smithi038 ceph-mon[92422]: pgmap v11775: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 528 KiB/s wr, 25 op/s 2023-03-09T20:50:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21585.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:50:14.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:13 smithi122 ceph-mon[93869]: pgmap v11776: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 472 KiB/s rd, 528 KiB/s wr, 22 op/s 2023-03-09T20:50:14.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:13 smithi181 ceph-mon[94189]: pgmap v11776: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 472 KiB/s rd, 528 KiB/s wr, 22 op/s 2023-03-09T20:50:14.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:13 smithi038 ceph-mon[92422]: pgmap v11776: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 472 KiB/s rd, 528 KiB/s wr, 22 op/s 2023-03-09T20:50:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21126.287975 secs 2023-03-09T20:50:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:15 smithi122 ceph-mon[93869]: pgmap v11777: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 595 KiB/s wr, 23 op/s 2023-03-09T20:50:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:15 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:50:16.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:15 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:50:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21126.287975 secs 2023-03-09T20:50:16.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:15 smithi181 ceph-mon[94189]: pgmap v11777: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 595 KiB/s wr, 23 op/s 2023-03-09T20:50:16.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:15 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:50:16.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:15 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:50:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21126.287975 secs 2023-03-09T20:50:16.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:15 smithi038 ceph-mon[92422]: pgmap v11777: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 595 KiB/s wr, 23 op/s 2023-03-09T20:50:16.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:15 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:50:16.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:15 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:50:17.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:50:17.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:50:17.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:17 smithi122 ceph-mon[93869]: pgmap v11778: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 472 KiB/s rd, 548 KiB/s wr, 13 op/s 2023-03-09T20:50:17.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:50:17.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:50:17.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:17 smithi181 ceph-mon[94189]: pgmap v11778: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 472 KiB/s rd, 548 KiB/s wr, 13 op/s 2023-03-09T20:50:17.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:50:17.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:50:17.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:17 smithi038 ceph-mon[92422]: pgmap v11778: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 472 KiB/s rd, 548 KiB/s wr, 13 op/s 2023-03-09T20:50:18.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21590.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:50:20.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:19 smithi122 ceph-mon[93869]: pgmap v11779: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 212 KiB/s wr, 8 op/s 2023-03-09T20:50:20.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:50:20.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:50:20.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:50:20.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:50:20.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:19 smithi181 ceph-mon[94189]: pgmap v11779: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 212 KiB/s wr, 8 op/s 2023-03-09T20:50:20.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:50:20.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:50:20.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:50:20.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:50:20.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:19 smithi038 ceph-mon[92422]: pgmap v11779: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 212 KiB/s wr, 8 op/s 2023-03-09T20:50:20.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:50:20.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:50:20.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:50:20.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:50:21.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21131.288038 secs 2023-03-09T20:50:21.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21131.288038 secs 2023-03-09T20:50:21.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21131.288038 secs 2023-03-09T20:50:22.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:21 smithi122 ceph-mon[93869]: pgmap v11780: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 341 KiB/s wr, 12 op/s 2023-03-09T20:50:22.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:21 smithi181 ceph-mon[94189]: pgmap v11780: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 341 KiB/s wr, 12 op/s 2023-03-09T20:50:22.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:21 smithi038 ceph-mon[92422]: pgmap v11780: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 341 KiB/s wr, 12 op/s 2023-03-09T20:50:23.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21133.605354 secs 2023-03-09T20:50:23.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21133.605354 secs 2023-03-09T20:50:23.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21133.605354 secs 2023-03-09T20:50:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21595.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:50:24.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:23 smithi122 ceph-mon[93869]: pgmap v11781: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 199 KiB/s wr, 8 op/s 2023-03-09T20:50:24.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:23 smithi181 ceph-mon[94189]: pgmap v11781: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 199 KiB/s wr, 8 op/s 2023-03-09T20:50:24.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:23 smithi038 ceph-mon[92422]: pgmap v11781: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 199 KiB/s wr, 8 op/s 2023-03-09T20:50:25.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21136.288096 secs 2023-03-09T20:50:25.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:25 smithi122 ceph-mon[93869]: pgmap v11782: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 436 KiB/s wr, 11 op/s 2023-03-09T20:50:25.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:25 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:50:25.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:25 smithi122 ceph-mon[93869]: osdmap e949: 12 total, 12 up, 12 in 2023-03-09T20:50:25.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:25 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:50:25.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:25 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:50:25.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21136.288096 secs 2023-03-09T20:50:25.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:25 smithi181 ceph-mon[94189]: pgmap v11782: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 436 KiB/s wr, 11 op/s 2023-03-09T20:50:25.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:25 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:50:25.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:25 smithi181 ceph-mon[94189]: osdmap e949: 12 total, 12 up, 12 in 2023-03-09T20:50:25.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:25 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:50:25.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:25 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:50:25.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21136.288096 secs 2023-03-09T20:50:25.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:25 smithi038 ceph-mon[92422]: pgmap v11782: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 436 KiB/s wr, 11 op/s 2023-03-09T20:50:25.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:25 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:50:25.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:25 smithi038 ceph-mon[92422]: osdmap e949: 12 total, 12 up, 12 in 2023-03-09T20:50:25.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:25 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:50:25.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:25 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:50:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:27 smithi122 ceph-mon[93869]: pgmap v11784: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 441 KiB/s wr, 13 op/s 2023-03-09T20:50:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21138.605420 secs 2023-03-09T20:50:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:27 smithi181 ceph-mon[94189]: pgmap v11784: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 441 KiB/s wr, 13 op/s 2023-03-09T20:50:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21138.605420 secs 2023-03-09T20:50:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:27 smithi038 ceph-mon[92422]: pgmap v11784: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 441 KiB/s wr, 13 op/s 2023-03-09T20:50:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21138.605420 secs 2023-03-09T20:50:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21600.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:50:30.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:29 smithi122 ceph-mon[93869]: pgmap v11785: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 442 KiB/s wr, 13 op/s 2023-03-09T20:50:30.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:29 smithi181 ceph-mon[94189]: pgmap v11785: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 442 KiB/s wr, 13 op/s 2023-03-09T20:50:30.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:29 smithi038 ceph-mon[92422]: pgmap v11785: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 442 KiB/s wr, 13 op/s 2023-03-09T20:50:31.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21141.288222 secs 2023-03-09T20:50:31.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21141.288222 secs 2023-03-09T20:50:31.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21141.288222 secs 2023-03-09T20:50:32.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:31 smithi122 ceph-mon[93869]: pgmap v11786: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 532 KiB/s rd, 421 KiB/s wr, 11 op/s 2023-03-09T20:50:32.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:31 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:50:32.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:31 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:50:32.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:31 smithi181 ceph-mon[94189]: pgmap v11786: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 532 KiB/s rd, 421 KiB/s wr, 11 op/s 2023-03-09T20:50:32.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:31 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:50:32.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:31 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:50:32.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:31 smithi038 ceph-mon[92422]: pgmap v11786: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 532 KiB/s rd, 421 KiB/s wr, 11 op/s 2023-03-09T20:50:32.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:31 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:50:32.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:31 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:50:33.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:32 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:50:33.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:32 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:50:33.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21143.605545 secs 2023-03-09T20:50:33.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:32 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:50:33.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:32 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:50:33.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21143.605545 secs 2023-03-09T20:50:33.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:32 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:50:33.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:32 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:50:33.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21143.605545 secs 2023-03-09T20:50:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21605.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:50:34.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:33 smithi122 ceph-mon[93869]: pgmap v11787: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 532 KiB/s rd, 421 KiB/s wr, 11 op/s 2023-03-09T20:50:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:33 smithi181 ceph-mon[94189]: pgmap v11787: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 532 KiB/s rd, 421 KiB/s wr, 11 op/s 2023-03-09T20:50:34.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:33 smithi038 ceph-mon[92422]: pgmap v11787: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 532 KiB/s rd, 421 KiB/s wr, 11 op/s 2023-03-09T20:50:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:35 smithi122 ceph-mon[93869]: osdmap e950: 12 total, 12 up, 12 in 2023-03-09T20:50:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21146.288282 secs 2023-03-09T20:50:35.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:35 smithi122 ceph-mon[93869]: pgmap v11789: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 205 KiB/s wr, 9 op/s 2023-03-09T20:50:35.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:35 smithi181 ceph-mon[94189]: osdmap e950: 12 total, 12 up, 12 in 2023-03-09T20:50:35.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21146.288282 secs 2023-03-09T20:50:35.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:35 smithi181 ceph-mon[94189]: pgmap v11789: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 205 KiB/s wr, 9 op/s 2023-03-09T20:50:35.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:35 smithi038 ceph-mon[92422]: osdmap e950: 12 total, 12 up, 12 in 2023-03-09T20:50:35.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21146.288282 secs 2023-03-09T20:50:35.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:35 smithi038 ceph-mon[92422]: pgmap v11789: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 205 KiB/s wr, 9 op/s 2023-03-09T20:50:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:37 smithi122 ceph-mon[93869]: pgmap v11790: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 203 KiB/s wr, 8 op/s 2023-03-09T20:50:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:37 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 21148.605666 secs 2023-03-09T20:50:38.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:37 smithi181 ceph-mon[94189]: pgmap v11790: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 203 KiB/s wr, 8 op/s 2023-03-09T20:50:38.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:37 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 21148.605666 secs 2023-03-09T20:50:38.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:37 smithi038 ceph-mon[92422]: pgmap v11790: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 203 KiB/s wr, 8 op/s 2023-03-09T20:50:38.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:37 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 21148.605666 secs 2023-03-09T20:50:38.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21610.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:50:40.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:39 smithi122 ceph-mon[93869]: pgmap v11791: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 203 KiB/s wr, 8 op/s 2023-03-09T20:50:40.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:39 smithi181 ceph-mon[94189]: pgmap v11791: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 203 KiB/s wr, 8 op/s 2023-03-09T20:50:40.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:39 smithi038 ceph-mon[92422]: pgmap v11791: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 203 KiB/s wr, 8 op/s 2023-03-09T20:50:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21151.288365 secs 2023-03-09T20:50:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:50:41.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:50:41.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:50:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21151.288365 secs 2023-03-09T20:50:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:50:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:50:41.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:50:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21151.288365 secs 2023-03-09T20:50:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:50:41.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:50:41.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:50:42.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:41 smithi122 ceph-mon[93869]: pgmap v11792: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 105 KiB/s wr, 7 op/s 2023-03-09T20:50:42.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:41 smithi181 ceph-mon[94189]: pgmap v11792: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 105 KiB/s wr, 7 op/s 2023-03-09T20:50:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:41 smithi038 ceph-mon[92422]: pgmap v11792: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 105 KiB/s wr, 7 op/s 2023-03-09T20:50:43.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:42 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 21153.605789 secs 2023-03-09T20:50:43.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:42 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 21153.605789 secs 2023-03-09T20:50:43.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:42 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 21153.605789 secs 2023-03-09T20:50:43.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21615.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:50:44.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:43 smithi122 ceph-mon[93869]: pgmap v11793: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 105 KiB/s wr, 7 op/s 2023-03-09T20:50:44.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:43 smithi181 ceph-mon[94189]: pgmap v11793: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 105 KiB/s wr, 7 op/s 2023-03-09T20:50:44.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:43 smithi038 ceph-mon[92422]: pgmap v11793: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 105 KiB/s wr, 7 op/s 2023-03-09T20:50:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21156.288451 secs 2023-03-09T20:50:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:45 smithi122 ceph-mon[93869]: pgmap v11794: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 104 KiB/s wr, 8 op/s 2023-03-09T20:50:45.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21156.288451 secs 2023-03-09T20:50:45.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:45 smithi181 ceph-mon[94189]: pgmap v11794: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 104 KiB/s wr, 8 op/s 2023-03-09T20:50:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21156.288451 secs 2023-03-09T20:50:45.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:45 smithi038 ceph-mon[92422]: pgmap v11794: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 104 KiB/s wr, 8 op/s 2023-03-09T20:50:47.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:50:47.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:50:47.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:47 smithi122 ceph-mon[93869]: pgmap v11795: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 107 KiB/s wr, 9 op/s 2023-03-09T20:50:47.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:47 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 21158.605908 secs 2023-03-09T20:50:47.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:50:47.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:50:47.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:47 smithi181 ceph-mon[94189]: pgmap v11795: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 107 KiB/s wr, 9 op/s 2023-03-09T20:50:47.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:47 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 21158.605908 secs 2023-03-09T20:50:47.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:50:47.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:50:47.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:47 smithi038 ceph-mon[92422]: pgmap v11795: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 107 KiB/s wr, 9 op/s 2023-03-09T20:50:47.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:47 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 21158.605908 secs 2023-03-09T20:50:48.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21620.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:50:50.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:49 smithi122 ceph-mon[93869]: pgmap v11796: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 263 KiB/s wr, 8 op/s 2023-03-09T20:50:50.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:49 smithi181 ceph-mon[94189]: pgmap v11796: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 263 KiB/s wr, 8 op/s 2023-03-09T20:50:50.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:49 smithi038 ceph-mon[92422]: pgmap v11796: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 263 KiB/s wr, 8 op/s 2023-03-09T20:50:51.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21161.288511 secs 2023-03-09T20:50:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21161.288511 secs 2023-03-09T20:50:51.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21161.288511 secs 2023-03-09T20:50:52.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:51 smithi122 ceph-mon[93869]: pgmap v11797: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 445 KiB/s rd, 280 KiB/s wr, 10 op/s 2023-03-09T20:50:52.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:51 smithi181 ceph-mon[94189]: pgmap v11797: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 445 KiB/s rd, 280 KiB/s wr, 10 op/s 2023-03-09T20:50:52.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:51 smithi038 ceph-mon[92422]: pgmap v11797: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 445 KiB/s rd, 280 KiB/s wr, 10 op/s 2023-03-09T20:50:53.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21163.606028 secs 2023-03-09T20:50:53.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21163.606028 secs 2023-03-09T20:50:53.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21163.606028 secs 2023-03-09T20:50:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21625.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:50:54.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:53 smithi122 ceph-mon[93869]: pgmap v11798: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 249 KiB/s wr, 8 op/s 2023-03-09T20:50:54.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:50:54.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:50:54.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:53 smithi181 ceph-mon[94189]: pgmap v11798: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 249 KiB/s wr, 8 op/s 2023-03-09T20:50:54.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:50:54.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:50:54.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:53 smithi038 ceph-mon[92422]: pgmap v11798: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 249 KiB/s wr, 8 op/s 2023-03-09T20:50:54.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:50:54.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:50:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21166.288574 secs 2023-03-09T20:50:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:55 smithi122 ceph-mon[93869]: pgmap v11799: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 376 KiB/s wr, 9 op/s 2023-03-09T20:50:56.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:50:56.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:50:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21166.288574 secs 2023-03-09T20:50:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:55 smithi181 ceph-mon[94189]: pgmap v11799: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 376 KiB/s wr, 9 op/s 2023-03-09T20:50:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:50:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:50:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21166.288574 secs 2023-03-09T20:50:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:55 smithi038 ceph-mon[92422]: pgmap v11799: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 376 KiB/s wr, 9 op/s 2023-03-09T20:50:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:50:56.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:50:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:50:57.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:50:57.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:57 smithi122 ceph-mon[93869]: pgmap v11800: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 556 KiB/s rd, 541 KiB/s wr, 11 op/s 2023-03-09T20:50:57.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21168.606138 secs 2023-03-09T20:50:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:50:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:50:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:57 smithi181 ceph-mon[94189]: pgmap v11800: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 556 KiB/s rd, 541 KiB/s wr, 11 op/s 2023-03-09T20:50:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21168.606138 secs 2023-03-09T20:50:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:50:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:50:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:57 smithi038 ceph-mon[92422]: pgmap v11800: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 556 KiB/s rd, 541 KiB/s wr, 11 op/s 2023-03-09T20:50:57.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21168.606138 secs 2023-03-09T20:50:58.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21630.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:51:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:50:59 smithi122 ceph-mon[93869]: pgmap v11801: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 538 KiB/s rd, 524 KiB/s wr, 10 op/s 2023-03-09T20:51:00.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:50:59 smithi181 ceph-mon[94189]: pgmap v11801: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 538 KiB/s rd, 524 KiB/s wr, 10 op/s 2023-03-09T20:51:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:50:59 smithi038 ceph-mon[92422]: pgmap v11801: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 538 KiB/s rd, 524 KiB/s wr, 10 op/s 2023-03-09T20:51:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21171.288633 secs 2023-03-09T20:51:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21171.288633 secs 2023-03-09T20:51:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21171.288633 secs 2023-03-09T20:51:02.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:01 smithi122 ceph-mon[93869]: pgmap v11802: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 401 KiB/s wr, 12 op/s 2023-03-09T20:51:02.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:01 smithi181 ceph-mon[94189]: pgmap v11802: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 401 KiB/s wr, 12 op/s 2023-03-09T20:51:02.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:01 smithi038 ceph-mon[92422]: pgmap v11802: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 401 KiB/s wr, 12 op/s 2023-03-09T20:51:03.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21173.606260 secs 2023-03-09T20:51:03.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21173.606260 secs 2023-03-09T20:51:03.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21173.606260 secs 2023-03-09T20:51:03.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21635.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:51:04.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:03 smithi122 ceph-mon[93869]: pgmap v11803: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 508 KiB/s rd, 507 KiB/s wr, 10 op/s 2023-03-09T20:51:04.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:03 smithi181 ceph-mon[94189]: pgmap v11803: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 508 KiB/s rd, 507 KiB/s wr, 10 op/s 2023-03-09T20:51:04.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:03 smithi038 ceph-mon[92422]: pgmap v11803: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 508 KiB/s rd, 507 KiB/s wr, 10 op/s 2023-03-09T20:51:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21176.288695 secs 2023-03-09T20:51:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:05 smithi122 ceph-mon[93869]: pgmap v11804: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 515 KiB/s rd, 518 KiB/s wr, 21 op/s 2023-03-09T20:51:05.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21176.288695 secs 2023-03-09T20:51:05.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:05 smithi181 ceph-mon[94189]: pgmap v11804: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 515 KiB/s rd, 518 KiB/s wr, 21 op/s 2023-03-09T20:51:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21176.288695 secs 2023-03-09T20:51:05.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:05 smithi038 ceph-mon[92422]: pgmap v11804: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 515 KiB/s rd, 518 KiB/s wr, 21 op/s 2023-03-09T20:51:07.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:51:07.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:51:07.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:07 smithi122 ceph-mon[93869]: pgmap v11805: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 509 KiB/s rd, 521 KiB/s wr, 24 op/s 2023-03-09T20:51:07.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21178.606375 secs 2023-03-09T20:51:07.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:51:07.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:51:07.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:07 smithi181 ceph-mon[94189]: pgmap v11805: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 509 KiB/s rd, 521 KiB/s wr, 24 op/s 2023-03-09T20:51:07.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21178.606375 secs 2023-03-09T20:51:07.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:51:07.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:51:07.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:07 smithi038 ceph-mon[92422]: pgmap v11805: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 509 KiB/s rd, 521 KiB/s wr, 24 op/s 2023-03-09T20:51:07.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21178.606375 secs 2023-03-09T20:51:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21640.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:51:09.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:51:09.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:51:09.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:09 smithi122 ceph-mon[93869]: pgmap v11806: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 442 KiB/s rd, 455 KiB/s wr, 21 op/s 2023-03-09T20:51:09.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:51:09.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:51:09.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:09 smithi181 ceph-mon[94189]: pgmap v11806: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 442 KiB/s rd, 455 KiB/s wr, 21 op/s 2023-03-09T20:51:09.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:51:09.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:51:09.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:09 smithi038 ceph-mon[92422]: pgmap v11806: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 442 KiB/s rd, 455 KiB/s wr, 21 op/s 2023-03-09T20:51:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21181.288779 secs 2023-03-09T20:51:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21181.288779 secs 2023-03-09T20:51:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21181.288779 secs 2023-03-09T20:51:11.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:11 smithi122 ceph-mon[93869]: pgmap v11807: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 443 KiB/s rd, 455 KiB/s wr, 23 op/s 2023-03-09T20:51:11.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:11 smithi181 ceph-mon[94189]: pgmap v11807: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 443 KiB/s rd, 455 KiB/s wr, 23 op/s 2023-03-09T20:51:11.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:11 smithi038 ceph-mon[92422]: pgmap v11807: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 443 KiB/s rd, 455 KiB/s wr, 23 op/s 2023-03-09T20:51:12.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21183.606497 secs 2023-03-09T20:51:12.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21183.606497 secs 2023-03-09T20:51:12.755 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21183.606497 secs 2023-03-09T20:51:13.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:13 smithi122 ceph-mon[93869]: pgmap v11808: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 421 KiB/s wr, 20 op/s 2023-03-09T20:51:13.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:13 smithi181 ceph-mon[94189]: pgmap v11808: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 421 KiB/s wr, 20 op/s 2023-03-09T20:51:13.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:13 smithi038 ceph-mon[92422]: pgmap v11808: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 421 KiB/s wr, 20 op/s 2023-03-09T20:51:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21645.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:51:15.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21186.288837 secs 2023-03-09T20:51:15.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:15 smithi122 ceph-mon[93869]: pgmap v11809: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 298 KiB/s wr, 19 op/s 2023-03-09T20:51:15.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21186.288837 secs 2023-03-09T20:51:15.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:15 smithi181 ceph-mon[94189]: pgmap v11809: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 298 KiB/s wr, 19 op/s 2023-03-09T20:51:15.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21186.288837 secs 2023-03-09T20:51:15.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:15 smithi038 ceph-mon[92422]: pgmap v11809: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 298 KiB/s wr, 19 op/s 2023-03-09T20:51:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:17 smithi122 ceph-mon[93869]: pgmap v11810: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 304 KiB/s wr, 11 op/s 2023-03-09T20:51:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21188.606617 secs 2023-03-09T20:51:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:17 smithi181 ceph-mon[94189]: pgmap v11810: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 304 KiB/s wr, 11 op/s 2023-03-09T20:51:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21188.606617 secs 2023-03-09T20:51:18.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:17 smithi038 ceph-mon[92422]: pgmap v11810: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 304 KiB/s wr, 11 op/s 2023-03-09T20:51:18.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21188.606617 secs 2023-03-09T20:51:18.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21650.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:51:20.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:19 smithi122 ceph-mon[93869]: pgmap v11811: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 185 KiB/s wr, 9 op/s 2023-03-09T20:51:20.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:19 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:51:20.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:19 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:51:20.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:51:20.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:19 smithi181 ceph-mon[94189]: pgmap v11811: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 185 KiB/s wr, 9 op/s 2023-03-09T20:51:20.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:19 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:51:20.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:19 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:51:20.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:51:20.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:19 smithi038 ceph-mon[92422]: pgmap v11811: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 185 KiB/s wr, 9 op/s 2023-03-09T20:51:20.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:19 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:51:20.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:19 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:51:20.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:51:21.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21191.288898 secs 2023-03-09T20:51:21.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:51:21.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:51:21.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:51:21.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:51:21.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:51:21.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21191.288898 secs 2023-03-09T20:51:21.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:51:21.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:51:21.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:51:21.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:51:21.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:51:21.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21191.288898 secs 2023-03-09T20:51:21.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:51:21.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:51:21.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:51:21.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:51:21.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:51:22.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:21 smithi122 ceph-mon[93869]: pgmap v11812: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 115 KiB/s wr, 12 op/s 2023-03-09T20:51:22.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:21 smithi181 ceph-mon[94189]: pgmap v11812: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 115 KiB/s wr, 12 op/s 2023-03-09T20:51:22.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:21 smithi038 ceph-mon[92422]: pgmap v11812: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 115 KiB/s wr, 12 op/s 2023-03-09T20:51:23.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21193.606754 secs 2023-03-09T20:51:23.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21193.606754 secs 2023-03-09T20:51:23.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21193.606754 secs 2023-03-09T20:51:23.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21655.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:51:24.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:23 smithi122 ceph-mon[93869]: pgmap v11813: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 177 KiB/s wr, 10 op/s 2023-03-09T20:51:24.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:23 smithi181 ceph-mon[94189]: pgmap v11813: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 177 KiB/s wr, 10 op/s 2023-03-09T20:51:24.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:23 smithi038 ceph-mon[92422]: pgmap v11813: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 177 KiB/s wr, 10 op/s 2023-03-09T20:51:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21196.288956 secs 2023-03-09T20:51:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:25 smithi122 ceph-mon[93869]: pgmap v11814: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 178 KiB/s wr, 10 op/s 2023-03-09T20:51:26.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:25 smithi122 ceph-mon[93869]: osdmap e951: 12 total, 12 up, 12 in 2023-03-09T20:51:26.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21196.288956 secs 2023-03-09T20:51:26.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:25 smithi181 ceph-mon[94189]: pgmap v11814: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 178 KiB/s wr, 10 op/s 2023-03-09T20:51:26.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:25 smithi181 ceph-mon[94189]: osdmap e951: 12 total, 12 up, 12 in 2023-03-09T20:51:26.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21196.288956 secs 2023-03-09T20:51:26.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:25 smithi038 ceph-mon[92422]: pgmap v11814: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 178 KiB/s wr, 10 op/s 2023-03-09T20:51:26.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:25 smithi038 ceph-mon[92422]: osdmap e951: 12 total, 12 up, 12 in 2023-03-09T20:51:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:27 smithi122 ceph-mon[93869]: pgmap v11816: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 289 KiB/s wr, 12 op/s 2023-03-09T20:51:28.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21198.606904 secs 2023-03-09T20:51:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:27 smithi181 ceph-mon[94189]: pgmap v11816: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 289 KiB/s wr, 12 op/s 2023-03-09T20:51:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21198.606904 secs 2023-03-09T20:51:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:27 smithi038 ceph-mon[92422]: pgmap v11816: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 289 KiB/s wr, 12 op/s 2023-03-09T20:51:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21198.606904 secs 2023-03-09T20:51:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21660.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:51:30.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:29 smithi181 ceph-mon[94189]: pgmap v11817: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 276 KiB/s wr, 11 op/s 2023-03-09T20:51:30.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:29 smithi038 ceph-mon[92422]: pgmap v11817: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 276 KiB/s wr, 11 op/s 2023-03-09T20:51:30.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:29 smithi122 ceph-mon[93869]: pgmap v11817: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 276 KiB/s wr, 11 op/s 2023-03-09T20:51:31.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21201.289025 secs 2023-03-09T20:51:31.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21201.289025 secs 2023-03-09T20:51:31.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21201.289025 secs 2023-03-09T20:51:32.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:31 smithi038 ceph-mon[92422]: pgmap v11818: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 271 KiB/s wr, 9 op/s 2023-03-09T20:51:32.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:31 smithi122 ceph-mon[93869]: pgmap v11818: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 271 KiB/s wr, 9 op/s 2023-03-09T20:51:32.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:31 smithi181 ceph-mon[94189]: pgmap v11818: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 271 KiB/s wr, 9 op/s 2023-03-09T20:51:33.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21203.607033 secs 2023-03-09T20:51:33.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21203.607033 secs 2023-03-09T20:51:33.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21203.607033 secs 2023-03-09T20:51:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21665.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:51:34.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:34 smithi122 ceph-mon[93869]: pgmap v11819: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 196 KiB/s wr, 9 op/s 2023-03-09T20:51:34.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:34 smithi181 ceph-mon[94189]: pgmap v11819: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 196 KiB/s wr, 9 op/s 2023-03-09T20:51:34.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:34 smithi038 ceph-mon[92422]: pgmap v11819: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 196 KiB/s wr, 9 op/s 2023-03-09T20:51:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:35 smithi122 ceph-mon[93869]: osdmap e952: 12 total, 12 up, 12 in 2023-03-09T20:51:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21206.289085 secs 2023-03-09T20:51:35.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:35 smithi122 ceph-mon[93869]: pgmap v11821: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 197 KiB/s wr, 9 op/s 2023-03-09T20:51:35.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:35 smithi181 ceph-mon[94189]: osdmap e952: 12 total, 12 up, 12 in 2023-03-09T20:51:35.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21206.289085 secs 2023-03-09T20:51:35.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:35 smithi181 ceph-mon[94189]: pgmap v11821: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 197 KiB/s wr, 9 op/s 2023-03-09T20:51:35.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:35 smithi038 ceph-mon[92422]: osdmap e952: 12 total, 12 up, 12 in 2023-03-09T20:51:35.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21206.289085 secs 2023-03-09T20:51:35.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:35 smithi038 ceph-mon[92422]: pgmap v11821: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 197 KiB/s wr, 9 op/s 2023-03-09T20:51:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:37 smithi122 ceph-mon[93869]: pgmap v11822: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 101 KiB/s wr, 9 op/s 2023-03-09T20:51:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21208.607156 secs 2023-03-09T20:51:38.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:37 smithi181 ceph-mon[94189]: pgmap v11822: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 101 KiB/s wr, 9 op/s 2023-03-09T20:51:38.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21208.607156 secs 2023-03-09T20:51:38.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:37 smithi038 ceph-mon[92422]: pgmap v11822: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 101 KiB/s wr, 9 op/s 2023-03-09T20:51:38.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21208.607156 secs 2023-03-09T20:51:38.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21670.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:51:40.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:39 smithi122 ceph-mon[93869]: pgmap v11823: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 101 KiB/s wr, 9 op/s 2023-03-09T20:51:40.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:39 smithi181 ceph-mon[94189]: pgmap v11823: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 101 KiB/s wr, 9 op/s 2023-03-09T20:51:40.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:39 smithi038 ceph-mon[92422]: pgmap v11823: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 101 KiB/s wr, 9 op/s 2023-03-09T20:51:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21211.289147 secs 2023-03-09T20:51:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21211.289147 secs 2023-03-09T20:51:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21211.289147 secs 2023-03-09T20:51:42.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:41 smithi122 ceph-mon[93869]: pgmap v11824: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 259 KiB/s wr, 9 op/s 2023-03-09T20:51:42.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:41 smithi181 ceph-mon[94189]: pgmap v11824: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 259 KiB/s wr, 9 op/s 2023-03-09T20:51:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:41 smithi038 ceph-mon[92422]: pgmap v11824: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 259 KiB/s wr, 9 op/s 2023-03-09T20:51:43.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21213.607241 secs 2023-03-09T20:51:43.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21213.607241 secs 2023-03-09T20:51:43.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21213.607241 secs 2023-03-09T20:51:43.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21675.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:51:44.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:43 smithi122 ceph-mon[93869]: pgmap v11825: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 259 KiB/s wr, 9 op/s 2023-03-09T20:51:44.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:43 smithi181 ceph-mon[94189]: pgmap v11825: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 259 KiB/s wr, 9 op/s 2023-03-09T20:51:44.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:43 smithi038 ceph-mon[92422]: pgmap v11825: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 259 KiB/s wr, 9 op/s 2023-03-09T20:51:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21216.289207 secs 2023-03-09T20:51:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:45 smithi122 ceph-mon[93869]: pgmap v11826: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 266 KiB/s wr, 9 op/s 2023-03-09T20:51:45.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21216.289207 secs 2023-03-09T20:51:45.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:45 smithi181 ceph-mon[94189]: pgmap v11826: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 266 KiB/s wr, 9 op/s 2023-03-09T20:51:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21216.289207 secs 2023-03-09T20:51:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:45 smithi038 ceph-mon[92422]: pgmap v11826: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 266 KiB/s wr, 9 op/s 2023-03-09T20:51:47.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:51:47.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:51:47.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:47 smithi122 ceph-mon[93869]: pgmap v11827: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 322 KiB/s wr, 12 op/s 2023-03-09T20:51:47.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:51:47.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:51:47.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:47 smithi181 ceph-mon[94189]: pgmap v11827: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 322 KiB/s wr, 12 op/s 2023-03-09T20:51:47.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:51:47.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:51:47.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:47 smithi038 ceph-mon[92422]: pgmap v11827: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 322 KiB/s wr, 12 op/s 2023-03-09T20:51:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:48 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:51:48.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:48 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:51:48.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:48 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:51:48.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21680.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:51:49.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:49 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:51:49.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:51:49.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:51:49.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:51:49.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:49 smithi038 ceph-mon[92422]: pgmap v11828: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 384 KiB/s rd, 339 KiB/s wr, 10 op/s 2023-03-09T20:51:49.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:49 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:51:49.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:51:49.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:51:49.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:51:49.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:49 smithi122 ceph-mon[93869]: pgmap v11828: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 384 KiB/s rd, 339 KiB/s wr, 10 op/s 2023-03-09T20:51:49.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:49 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:51:49.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:51:49.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:51:49.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:51:49.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:49 smithi181 ceph-mon[94189]: pgmap v11828: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 384 KiB/s rd, 339 KiB/s wr, 10 op/s 2023-03-09T20:51:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21221.289282 secs 2023-03-09T20:51:50.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21221.289282 secs 2023-03-09T20:51:50.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21221.289282 secs 2023-03-09T20:51:51.753 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:51 smithi038 ceph-mon[92422]: pgmap v11829: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 607 KiB/s rd, 561 KiB/s wr, 14 op/s 2023-03-09T20:51:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:51 smithi122 ceph-mon[93869]: pgmap v11829: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 607 KiB/s rd, 561 KiB/s wr, 14 op/s 2023-03-09T20:51:51.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:51 smithi181 ceph-mon[94189]: pgmap v11829: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 607 KiB/s rd, 561 KiB/s wr, 14 op/s 2023-03-09T20:51:53.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21685.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:51:54.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:53 smithi122 ceph-mon[93869]: pgmap v11830: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 346 KiB/s wr, 10 op/s 2023-03-09T20:51:54.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:53 smithi181 ceph-mon[94189]: pgmap v11830: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 346 KiB/s wr, 10 op/s 2023-03-09T20:51:54.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:53 smithi038 ceph-mon[92422]: pgmap v11830: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 346 KiB/s wr, 10 op/s 2023-03-09T20:51:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21226.289344 secs 2023-03-09T20:51:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:55 smithi122 ceph-mon[93869]: pgmap v11831: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 364 KiB/s wr, 11 op/s 2023-03-09T20:51:56.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:51:56.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:51:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21226.289344 secs 2023-03-09T20:51:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:55 smithi181 ceph-mon[94189]: pgmap v11831: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 364 KiB/s wr, 11 op/s 2023-03-09T20:51:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:51:56.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:51:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21226.289344 secs 2023-03-09T20:51:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:55 smithi038 ceph-mon[92422]: pgmap v11831: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 364 KiB/s wr, 11 op/s 2023-03-09T20:51:56.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:51:56.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:51:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:51:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:51:57.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:57 smithi122 ceph-mon[93869]: pgmap v11832: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 532 KiB/s rd, 528 KiB/s wr, 14 op/s 2023-03-09T20:51:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:51:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:51:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:57 smithi181 ceph-mon[94189]: pgmap v11832: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 532 KiB/s rd, 528 KiB/s wr, 14 op/s 2023-03-09T20:51:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:51:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:51:57.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:57 smithi038 ceph-mon[92422]: pgmap v11832: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 532 KiB/s rd, 528 KiB/s wr, 14 op/s 2023-03-09T20:51:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21690.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:51:59.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:51:59.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:51:59.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:51:59 smithi122 ceph-mon[93869]: pgmap v11833: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 455 KiB/s rd, 471 KiB/s wr, 11 op/s 2023-03-09T20:51:59.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:51:59.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:51:59.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:51:59 smithi181 ceph-mon[94189]: pgmap v11833: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 455 KiB/s rd, 471 KiB/s wr, 11 op/s 2023-03-09T20:51:59.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:51:59.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:51:59.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:51:59 smithi038 ceph-mon[92422]: pgmap v11833: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 455 KiB/s rd, 471 KiB/s wr, 11 op/s 2023-03-09T20:52:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21231.289402 secs 2023-03-09T20:52:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21231.289402 secs 2023-03-09T20:52:00.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21231.289402 secs 2023-03-09T20:52:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:52:01.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:52:01.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:01 smithi038 ceph-mon[92422]: pgmap v11834: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 617 KiB/s rd, 631 KiB/s wr, 13 op/s 2023-03-09T20:52:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:52:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:52:01.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:01 smithi122 ceph-mon[93869]: pgmap v11834: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 617 KiB/s rd, 631 KiB/s wr, 13 op/s 2023-03-09T20:52:01.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:52:01.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:52:01.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:01 smithi181 ceph-mon[94189]: pgmap v11834: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 617 KiB/s rd, 631 KiB/s wr, 13 op/s 2023-03-09T20:52:02.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21233.607753 secs 2023-03-09T20:52:02.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21233.607753 secs 2023-03-09T20:52:02.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21233.607753 secs 2023-03-09T20:52:03.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21695.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:52:03.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:03 smithi122 ceph-mon[93869]: pgmap v11835: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 408 KiB/s wr, 9 op/s 2023-03-09T20:52:03.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:03 smithi181 ceph-mon[94189]: pgmap v11835: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 408 KiB/s wr, 9 op/s 2023-03-09T20:52:04.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:03 smithi038 ceph-mon[92422]: pgmap v11835: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 408 KiB/s wr, 9 op/s 2023-03-09T20:52:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:05 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:52:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:05 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:52:05.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:05 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:52:05.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21236.289464 secs 2023-03-09T20:52:05.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:05 smithi122 ceph-mon[93869]: pgmap v11836: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 430 KiB/s wr, 21 op/s 2023-03-09T20:52:05.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:05 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:52:05.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:05 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:52:05.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:05 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:52:05.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21236.289464 secs 2023-03-09T20:52:05.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:05 smithi181 ceph-mon[94189]: pgmap v11836: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 430 KiB/s wr, 21 op/s 2023-03-09T20:52:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:05 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:52:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:05 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:52:05.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:05 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:52:05.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21236.289464 secs 2023-03-09T20:52:05.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:05 smithi038 ceph-mon[92422]: pgmap v11836: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 430 KiB/s wr, 21 op/s 2023-03-09T20:52:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:07 smithi122 ceph-mon[93869]: pgmap v11837: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 471 KiB/s rd, 504 KiB/s wr, 26 op/s 2023-03-09T20:52:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21238.607890 secs 2023-03-09T20:52:08.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:07 smithi181 ceph-mon[94189]: pgmap v11837: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 471 KiB/s rd, 504 KiB/s wr, 26 op/s 2023-03-09T20:52:08.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21238.607890 secs 2023-03-09T20:52:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:07 smithi038 ceph-mon[92422]: pgmap v11837: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 471 KiB/s rd, 504 KiB/s wr, 26 op/s 2023-03-09T20:52:08.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21238.607890 secs 2023-03-09T20:52:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21700.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:52:10.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:09 smithi122 ceph-mon[93869]: pgmap v11838: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 341 KiB/s wr, 23 op/s 2023-03-09T20:52:10.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:09 smithi181 ceph-mon[94189]: pgmap v11838: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 341 KiB/s wr, 23 op/s 2023-03-09T20:52:10.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:09 smithi038 ceph-mon[92422]: pgmap v11838: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 341 KiB/s wr, 23 op/s 2023-03-09T20:52:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21241.289524 secs 2023-03-09T20:52:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:10 smithi181 ceph-mon[94189]: osdmap e953: 12 total, 12 up, 12 in 2023-03-09T20:52:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21241.289524 secs 2023-03-09T20:52:11.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:10 smithi038 ceph-mon[92422]: osdmap e953: 12 total, 12 up, 12 in 2023-03-09T20:52:11.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21241.289524 secs 2023-03-09T20:52:11.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:10 smithi122 ceph-mon[93869]: osdmap e953: 12 total, 12 up, 12 in 2023-03-09T20:52:12.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:11 smithi181 ceph-mon[94189]: pgmap v11840: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 157 KiB/s wr, 26 op/s 2023-03-09T20:52:12.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:11 smithi038 ceph-mon[92422]: pgmap v11840: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 157 KiB/s wr, 26 op/s 2023-03-09T20:52:12.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:11 smithi122 ceph-mon[93869]: pgmap v11840: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 157 KiB/s wr, 26 op/s 2023-03-09T20:52:13.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21243.607957 secs 2023-03-09T20:52:13.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21243.607957 secs 2023-03-09T20:52:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21243.607957 secs 2023-03-09T20:52:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21705.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:52:14.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:13 smithi038 ceph-mon[92422]: pgmap v11841: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 157 KiB/s wr, 26 op/s 2023-03-09T20:52:14.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:13 smithi122 ceph-mon[93869]: pgmap v11841: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 157 KiB/s wr, 26 op/s 2023-03-09T20:52:14.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:13 smithi181 ceph-mon[94189]: pgmap v11841: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 157 KiB/s wr, 26 op/s 2023-03-09T20:52:15.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21246.289584 secs 2023-03-09T20:52:15.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:15 smithi122 ceph-mon[93869]: pgmap v11842: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 151 KiB/s wr, 12 op/s 2023-03-09T20:52:15.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21246.289584 secs 2023-03-09T20:52:15.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:15 smithi181 ceph-mon[94189]: pgmap v11842: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 151 KiB/s wr, 12 op/s 2023-03-09T20:52:15.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21246.289584 secs 2023-03-09T20:52:15.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:15 smithi038 ceph-mon[92422]: pgmap v11842: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 151 KiB/s wr, 12 op/s 2023-03-09T20:52:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:17 smithi122 ceph-mon[93869]: pgmap v11843: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 122 KiB/s wr, 8 op/s 2023-03-09T20:52:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21248.608072 secs 2023-03-09T20:52:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:17 smithi181 ceph-mon[94189]: pgmap v11843: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 122 KiB/s wr, 8 op/s 2023-03-09T20:52:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21248.608072 secs 2023-03-09T20:52:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:17 smithi038 ceph-mon[92422]: pgmap v11843: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 122 KiB/s wr, 8 op/s 2023-03-09T20:52:18.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21248.608072 secs 2023-03-09T20:52:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21710.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:52:20.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:19 smithi122 ceph-mon[93869]: pgmap v11844: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 103 KiB/s wr, 8 op/s 2023-03-09T20:52:20.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:19 smithi122 ceph-mon[93869]: osdmap e954: 12 total, 12 up, 12 in 2023-03-09T20:52:20.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:19 smithi181 ceph-mon[94189]: pgmap v11844: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 103 KiB/s wr, 8 op/s 2023-03-09T20:52:20.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:19 smithi181 ceph-mon[94189]: osdmap e954: 12 total, 12 up, 12 in 2023-03-09T20:52:20.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:19 smithi038 ceph-mon[92422]: pgmap v11844: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 103 KiB/s wr, 8 op/s 2023-03-09T20:52:20.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:19 smithi038 ceph-mon[92422]: osdmap e954: 12 total, 12 up, 12 in 2023-03-09T20:52:21.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21251.289668 secs 2023-03-09T20:52:21.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:52:21.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21251.289668 secs 2023-03-09T20:52:21.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:52:21.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21251.289668 secs 2023-03-09T20:52:21.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:52:22.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:21 smithi181 ceph-mon[94189]: pgmap v11846: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 103 KiB/s wr, 7 op/s 2023-03-09T20:52:22.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:21 smithi038 ceph-mon[92422]: pgmap v11846: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 103 KiB/s wr, 7 op/s 2023-03-09T20:52:22.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:21 smithi122 ceph-mon[93869]: pgmap v11846: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 103 KiB/s wr, 7 op/s 2023-03-09T20:52:23.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21253.608209 secs 2023-03-09T20:52:23.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21253.608209 secs 2023-03-09T20:52:23.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21253.608209 secs 2023-03-09T20:52:23.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21715.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:52:24.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:23 smithi038 ceph-mon[92422]: pgmap v11847: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 103 KiB/s wr, 7 op/s 2023-03-09T20:52:24.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:23 smithi122 ceph-mon[93869]: pgmap v11847: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 103 KiB/s wr, 7 op/s 2023-03-09T20:52:24.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:23 smithi181 ceph-mon[94189]: pgmap v11847: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 103 KiB/s wr, 7 op/s 2023-03-09T20:52:25.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:52:25.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:52:25.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:52:25.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:52:25.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:52:25.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21256.289731 secs 2023-03-09T20:52:25.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:25 smithi122 ceph-mon[93869]: pgmap v11848: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 83 KiB/s wr, 7 op/s 2023-03-09T20:52:25.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:52:25.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:52:25.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:52:25.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:52:25.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:52:25.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21256.289731 secs 2023-03-09T20:52:25.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:25 smithi181 ceph-mon[94189]: pgmap v11848: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 83 KiB/s wr, 7 op/s 2023-03-09T20:52:26.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:52:26.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:52:26.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:52:26.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:52:26.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:52:26.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21256.289731 secs 2023-03-09T20:52:26.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:25 smithi038 ceph-mon[92422]: pgmap v11848: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 83 KiB/s wr, 7 op/s 2023-03-09T20:52:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:27 smithi122 ceph-mon[93869]: pgmap v11849: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 101 KiB/s wr, 7 op/s 2023-03-09T20:52:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21258.608327 secs 2023-03-09T20:52:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:27 smithi181 ceph-mon[94189]: pgmap v11849: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 101 KiB/s wr, 7 op/s 2023-03-09T20:52:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21258.608327 secs 2023-03-09T20:52:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:27 smithi038 ceph-mon[92422]: pgmap v11849: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 101 KiB/s wr, 7 op/s 2023-03-09T20:52:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21258.608327 secs 2023-03-09T20:52:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21720.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:52:30.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:29 smithi122 ceph-mon[93869]: pgmap v11850: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 101 KiB/s wr, 7 op/s 2023-03-09T20:52:30.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:29 smithi181 ceph-mon[94189]: pgmap v11850: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 101 KiB/s wr, 7 op/s 2023-03-09T20:52:30.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:29 smithi038 ceph-mon[92422]: pgmap v11850: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 101 KiB/s wr, 7 op/s 2023-03-09T20:52:31.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21261.289791 secs 2023-03-09T20:52:31.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21261.289791 secs 2023-03-09T20:52:31.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21261.289791 secs 2023-03-09T20:52:32.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:31 smithi038 ceph-mon[92422]: pgmap v11851: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 89 KiB/s wr, 9 op/s 2023-03-09T20:52:32.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:31 smithi122 ceph-mon[93869]: pgmap v11851: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 89 KiB/s wr, 9 op/s 2023-03-09T20:52:32.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:31 smithi181 ceph-mon[94189]: pgmap v11851: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 89 KiB/s wr, 9 op/s 2023-03-09T20:52:33.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21263.608458 secs 2023-03-09T20:52:33.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21263.608458 secs 2023-03-09T20:52:33.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21263.608458 secs 2023-03-09T20:52:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21725.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:52:34.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:34 smithi122 ceph-mon[93869]: pgmap v11852: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 84 KiB/s wr, 6 op/s 2023-03-09T20:52:34.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:34 smithi181 ceph-mon[94189]: pgmap v11852: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 84 KiB/s wr, 6 op/s 2023-03-09T20:52:34.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:34 smithi038 ceph-mon[92422]: pgmap v11852: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 84 KiB/s wr, 6 op/s 2023-03-09T20:52:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21266.289848 secs 2023-03-09T20:52:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:35 smithi122 ceph-mon[93869]: pgmap v11853: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 84 KiB/s wr, 7 op/s 2023-03-09T20:52:35.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21266.289848 secs 2023-03-09T20:52:35.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:35 smithi181 ceph-mon[94189]: pgmap v11853: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 84 KiB/s wr, 7 op/s 2023-03-09T20:52:35.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21266.289848 secs 2023-03-09T20:52:35.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:35 smithi038 ceph-mon[92422]: pgmap v11853: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 84 KiB/s wr, 7 op/s 2023-03-09T20:52:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:37 smithi122 ceph-mon[93869]: pgmap v11854: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 162 KiB/s wr, 9 op/s 2023-03-09T20:52:38.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:37 smithi181 ceph-mon[94189]: pgmap v11854: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 162 KiB/s wr, 9 op/s 2023-03-09T20:52:38.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:37 smithi038 ceph-mon[92422]: pgmap v11854: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 162 KiB/s wr, 9 op/s 2023-03-09T20:52:38.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21730.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:52:40.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:39 smithi038 ceph-mon[92422]: pgmap v11855: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 80 KiB/s wr, 7 op/s 2023-03-09T20:52:40.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:39 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:52:40.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:39 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:52:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:39 smithi122 ceph-mon[93869]: pgmap v11855: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 80 KiB/s wr, 7 op/s 2023-03-09T20:52:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:39 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:52:40.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:39 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:52:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:39 smithi181 ceph-mon[94189]: pgmap v11855: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 80 KiB/s wr, 7 op/s 2023-03-09T20:52:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:39 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:52:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:39 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:52:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21271.289908 secs 2023-03-09T20:52:41.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:40 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:52:41.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:40 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:52:41.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:52:41.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:52:41.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:52:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21271.289908 secs 2023-03-09T20:52:41.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:40 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:52:41.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:40 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:52:41.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:52:41.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:52:41.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:52:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21271.289908 secs 2023-03-09T20:52:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:40 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:52:41.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:40 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:52:41.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:52:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:52:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:52:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:41 smithi038 ceph-mon[92422]: pgmap v11856: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 276 KiB/s wr, 9 op/s 2023-03-09T20:52:42.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:41 smithi122 ceph-mon[93869]: pgmap v11856: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 276 KiB/s wr, 9 op/s 2023-03-09T20:52:42.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:41 smithi181 ceph-mon[94189]: pgmap v11856: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 276 KiB/s wr, 9 op/s 2023-03-09T20:52:43.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21735.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:52:44.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:43 smithi038 ceph-mon[92422]: pgmap v11857: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 275 KiB/s wr, 7 op/s 2023-03-09T20:52:44.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:43 smithi122 ceph-mon[93869]: pgmap v11857: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 275 KiB/s wr, 7 op/s 2023-03-09T20:52:44.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:43 smithi181 ceph-mon[94189]: pgmap v11857: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 275 KiB/s wr, 7 op/s 2023-03-09T20:52:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21276.289967 secs 2023-03-09T20:52:45.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:45 smithi122 ceph-mon[93869]: pgmap v11858: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 292 KiB/s wr, 8 op/s 2023-03-09T20:52:45.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21276.289967 secs 2023-03-09T20:52:45.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:45 smithi181 ceph-mon[94189]: pgmap v11858: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 292 KiB/s wr, 8 op/s 2023-03-09T20:52:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21276.289967 secs 2023-03-09T20:52:45.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:45 smithi038 ceph-mon[92422]: pgmap v11858: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 292 KiB/s wr, 8 op/s 2023-03-09T20:52:48.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:47 smithi122 ceph-mon[93869]: pgmap v11859: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 403 KiB/s wr, 11 op/s 2023-03-09T20:52:48.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:47 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 21278.608810 secs 2023-03-09T20:52:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:47 smithi181 ceph-mon[94189]: pgmap v11859: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 403 KiB/s wr, 11 op/s 2023-03-09T20:52:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:47 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 21278.608810 secs 2023-03-09T20:52:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:47 smithi038 ceph-mon[92422]: pgmap v11859: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 403 KiB/s wr, 11 op/s 2023-03-09T20:52:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:47 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 21278.608810 secs 2023-03-09T20:52:48.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21740.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:52:49.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:49 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:52:49.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:52:49.481 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:52:49.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:49 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:52:49.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:52:49.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:52:49.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:49 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:52:49.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:52:49.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:52:50.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:50 smithi122 ceph-mon[93869]: pgmap v11860: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 342 KiB/s wr, 10 op/s 2023-03-09T20:52:50.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:50 smithi181 ceph-mon[94189]: pgmap v11860: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 342 KiB/s wr, 10 op/s 2023-03-09T20:52:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:50 smithi038 ceph-mon[92422]: pgmap v11860: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 342 KiB/s wr, 10 op/s 2023-03-09T20:52:51.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21281.289984 secs 2023-03-09T20:52:51.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21281.289984 secs 2023-03-09T20:52:51.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21281.289984 secs 2023-03-09T20:52:52.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:52 smithi122 ceph-mon[93869]: pgmap v11861: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 584 KiB/s rd, 533 KiB/s wr, 13 op/s 2023-03-09T20:52:52.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:52:52.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:52:52.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:52 smithi181 ceph-mon[94189]: pgmap v11861: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 584 KiB/s rd, 533 KiB/s wr, 13 op/s 2023-03-09T20:52:52.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:52:52.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:52:52.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:52 smithi038 ceph-mon[92422]: pgmap v11861: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 584 KiB/s rd, 533 KiB/s wr, 13 op/s 2023-03-09T20:52:52.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:52:52.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:52:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:52:53.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:52:53.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:53 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 21283.608933 secs 2023-03-09T20:52:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:52:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:52:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:53 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 21283.608933 secs 2023-03-09T20:52:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:52:53.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:52:53.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:53 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 21283.608933 secs 2023-03-09T20:52:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21745.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:52:54.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:54 smithi122 ceph-mon[93869]: pgmap v11862: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 337 KiB/s wr, 10 op/s 2023-03-09T20:52:54.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:54 smithi181 ceph-mon[94189]: pgmap v11862: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 337 KiB/s wr, 10 op/s 2023-03-09T20:52:54.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:54 smithi038 ceph-mon[92422]: pgmap v11862: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 337 KiB/s wr, 10 op/s 2023-03-09T20:52:55.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21286.290043 secs 2023-03-09T20:52:55.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:55 smithi122 ceph-mon[93869]: pgmap v11863: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 493 KiB/s rd, 478 KiB/s wr, 12 op/s 2023-03-09T20:52:55.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21286.290043 secs 2023-03-09T20:52:55.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:55 smithi181 ceph-mon[94189]: pgmap v11863: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 493 KiB/s rd, 478 KiB/s wr, 12 op/s 2023-03-09T20:52:55.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21286.290043 secs 2023-03-09T20:52:55.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:55 smithi038 ceph-mon[92422]: pgmap v11863: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 493 KiB/s rd, 478 KiB/s wr, 12 op/s 2023-03-09T20:52:58.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:57 smithi122 ceph-mon[93869]: pgmap v11864: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 537 KiB/s wr, 14 op/s 2023-03-09T20:52:58.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:57 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 21288.609072 secs 2023-03-09T20:52:58.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:57 smithi181 ceph-mon[94189]: pgmap v11864: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 537 KiB/s wr, 14 op/s 2023-03-09T20:52:58.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:57 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 21288.609072 secs 2023-03-09T20:52:58.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:57 smithi038 ceph-mon[92422]: pgmap v11864: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 537 KiB/s wr, 14 op/s 2023-03-09T20:52:58.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:57 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 21288.609072 secs 2023-03-09T20:52:58.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21750.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:53:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:59 smithi038 ceph-mon[92422]: pgmap v11865: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 472 KiB/s rd, 459 KiB/s wr, 12 op/s 2023-03-09T20:53:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:53:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:52:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:53:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:59 smithi122 ceph-mon[93869]: pgmap v11865: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 472 KiB/s rd, 459 KiB/s wr, 12 op/s 2023-03-09T20:53:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:53:00.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:52:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:53:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:59 smithi181 ceph-mon[94189]: pgmap v11865: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 472 KiB/s rd, 459 KiB/s wr, 12 op/s 2023-03-09T20:53:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:53:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:52:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:53:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21291.290101 secs 2023-03-09T20:53:01.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:53:01.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:53:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21291.290101 secs 2023-03-09T20:53:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:53:01.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:53:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21291.290101 secs 2023-03-09T20:53:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:53:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:53:02.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:01 smithi038 ceph-mon[92422]: pgmap v11866: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 750 KiB/s rd, 758 KiB/s wr, 16 op/s 2023-03-09T20:53:02.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:01 smithi122 ceph-mon[93869]: pgmap v11866: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 750 KiB/s rd, 758 KiB/s wr, 16 op/s 2023-03-09T20:53:02.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:01 smithi181 ceph-mon[94189]: pgmap v11866: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 750 KiB/s rd, 758 KiB/s wr, 16 op/s 2023-03-09T20:53:03.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21293.609174 secs 2023-03-09T20:53:03.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21293.609174 secs 2023-03-09T20:53:03.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21293.609174 secs 2023-03-09T20:53:03.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21755.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:53:04.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:04 smithi122 ceph-mon[93869]: pgmap v11867: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 582 KiB/s rd, 603 KiB/s wr, 13 op/s 2023-03-09T20:53:04.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:04 smithi181 ceph-mon[94189]: pgmap v11867: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 582 KiB/s rd, 603 KiB/s wr, 13 op/s 2023-03-09T20:53:04.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:04 smithi038 ceph-mon[92422]: pgmap v11867: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 582 KiB/s rd, 603 KiB/s wr, 13 op/s 2023-03-09T20:53:06.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21296.290161 secs 2023-03-09T20:53:06.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:06 smithi122 ceph-mon[93869]: pgmap v11868: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 590 KiB/s rd, 625 KiB/s wr, 26 op/s 2023-03-09T20:53:06.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:06 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:53:06.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:53:06.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21296.290161 secs 2023-03-09T20:53:06.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:06 smithi181 ceph-mon[94189]: pgmap v11868: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 590 KiB/s rd, 625 KiB/s wr, 26 op/s 2023-03-09T20:53:06.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:06 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:53:06.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:53:06.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21296.290161 secs 2023-03-09T20:53:06.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:06 smithi038 ceph-mon[92422]: pgmap v11868: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 590 KiB/s rd, 625 KiB/s wr, 26 op/s 2023-03-09T20:53:06.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:06 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:53:06.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:53:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:08 smithi122 ceph-mon[93869]: pgmap v11869: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 483 KiB/s wr, 26 op/s 2023-03-09T20:53:08.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:53:08.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:53:08.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21298.609291 secs 2023-03-09T20:53:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:08 smithi181 ceph-mon[94189]: pgmap v11869: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 483 KiB/s wr, 26 op/s 2023-03-09T20:53:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:53:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:53:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21298.609291 secs 2023-03-09T20:53:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:08 smithi038 ceph-mon[92422]: pgmap v11869: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 483 KiB/s wr, 26 op/s 2023-03-09T20:53:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:53:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:53:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21298.609291 secs 2023-03-09T20:53:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21760.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:53:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:10 smithi122 ceph-mon[93869]: pgmap v11870: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 514 KiB/s rd, 548 KiB/s wr, 25 op/s 2023-03-09T20:53:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:10 smithi122 ceph-mon[93869]: osdmap e955: 12 total, 12 up, 12 in 2023-03-09T20:53:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:10 smithi181 ceph-mon[94189]: pgmap v11870: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 514 KiB/s rd, 548 KiB/s wr, 25 op/s 2023-03-09T20:53:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:10 smithi181 ceph-mon[94189]: osdmap e955: 12 total, 12 up, 12 in 2023-03-09T20:53:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:10 smithi038 ceph-mon[92422]: pgmap v11870: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 514 KiB/s rd, 548 KiB/s wr, 25 op/s 2023-03-09T20:53:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:10 smithi038 ceph-mon[92422]: osdmap e955: 12 total, 12 up, 12 in 2023-03-09T20:53:11.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21301.290222 secs 2023-03-09T20:53:11.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21301.290222 secs 2023-03-09T20:53:11.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21301.290222 secs 2023-03-09T20:53:12.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:12 smithi122 ceph-mon[93869]: pgmap v11872: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 239 KiB/s wr, 25 op/s 2023-03-09T20:53:12.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:12 smithi181 ceph-mon[94189]: pgmap v11872: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 239 KiB/s wr, 25 op/s 2023-03-09T20:53:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:12 smithi038 ceph-mon[92422]: pgmap v11872: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 239 KiB/s wr, 25 op/s 2023-03-09T20:53:13.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21303.609430 secs 2023-03-09T20:53:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21303.609430 secs 2023-03-09T20:53:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21303.609430 secs 2023-03-09T20:53:13.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21765.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:53:14.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:14 smithi122 ceph-mon[93869]: pgmap v11873: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 195 KiB/s wr, 25 op/s 2023-03-09T20:53:14.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:14 smithi181 ceph-mon[94189]: pgmap v11873: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 195 KiB/s wr, 25 op/s 2023-03-09T20:53:14.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:14 smithi038 ceph-mon[92422]: pgmap v11873: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 195 KiB/s wr, 25 op/s 2023-03-09T20:53:15.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21306.290282 secs 2023-03-09T20:53:15.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:15 smithi122 ceph-mon[93869]: pgmap v11874: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 189 KiB/s wr, 10 op/s 2023-03-09T20:53:15.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21306.290282 secs 2023-03-09T20:53:15.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:15 smithi181 ceph-mon[94189]: pgmap v11874: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 189 KiB/s wr, 10 op/s 2023-03-09T20:53:15.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21306.290282 secs 2023-03-09T20:53:15.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:15 smithi038 ceph-mon[92422]: pgmap v11874: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 189 KiB/s wr, 10 op/s 2023-03-09T20:53:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:17 smithi038 ceph-mon[92422]: pgmap v11875: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 256 KiB/s wr, 11 op/s 2023-03-09T20:53:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21308.609555 secs 2023-03-09T20:53:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:17 smithi122 ceph-mon[93869]: pgmap v11875: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 256 KiB/s wr, 11 op/s 2023-03-09T20:53:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21308.609555 secs 2023-03-09T20:53:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:17 smithi181 ceph-mon[94189]: pgmap v11875: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 256 KiB/s wr, 11 op/s 2023-03-09T20:53:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21308.609555 secs 2023-03-09T20:53:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21770.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:53:20.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:19 smithi038 ceph-mon[92422]: pgmap v11876: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 89 KiB/s wr, 10 op/s 2023-03-09T20:53:20.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:19 smithi038 ceph-mon[92422]: osdmap e956: 12 total, 12 up, 12 in 2023-03-09T20:53:20.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:19 smithi122 ceph-mon[93869]: pgmap v11876: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 89 KiB/s wr, 10 op/s 2023-03-09T20:53:20.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:19 smithi122 ceph-mon[93869]: osdmap e956: 12 total, 12 up, 12 in 2023-03-09T20:53:20.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:19 smithi181 ceph-mon[94189]: pgmap v11876: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 89 KiB/s wr, 10 op/s 2023-03-09T20:53:20.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:19 smithi181 ceph-mon[94189]: osdmap e956: 12 total, 12 up, 12 in 2023-03-09T20:53:21.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21311.290342 secs 2023-03-09T20:53:21.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21311.290342 secs 2023-03-09T20:53:21.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21311.290342 secs 2023-03-09T20:53:22.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:21 smithi038 ceph-mon[92422]: pgmap v11878: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 89 KiB/s wr, 9 op/s 2023-03-09T20:53:22.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:21 smithi122 ceph-mon[93869]: pgmap v11878: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 89 KiB/s wr, 9 op/s 2023-03-09T20:53:22.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:21 smithi181 ceph-mon[94189]: pgmap v11878: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 89 KiB/s wr, 9 op/s 2023-03-09T20:53:23.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21313.609678 secs 2023-03-09T20:53:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21313.609678 secs 2023-03-09T20:53:23.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21313.609678 secs 2023-03-09T20:53:23.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21775.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:53:24.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:24 smithi122 ceph-mon[93869]: pgmap v11879: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 89 KiB/s wr, 9 op/s 2023-03-09T20:53:24.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:24 smithi181 ceph-mon[94189]: pgmap v11879: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 89 KiB/s wr, 9 op/s 2023-03-09T20:53:24.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:24 smithi038 ceph-mon[92422]: pgmap v11879: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 89 KiB/s wr, 9 op/s 2023-03-09T20:53:25.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:53:25.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21316.290405 secs 2023-03-09T20:53:25.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:25 smithi038 ceph-mon[92422]: pgmap v11880: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 69 KiB/s wr, 8 op/s 2023-03-09T20:53:25.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:53:25.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21316.290405 secs 2023-03-09T20:53:25.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:25 smithi122 ceph-mon[93869]: pgmap v11880: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 69 KiB/s wr, 8 op/s 2023-03-09T20:53:25.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:53:25.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21316.290405 secs 2023-03-09T20:53:25.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:25 smithi181 ceph-mon[94189]: pgmap v11880: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 69 KiB/s wr, 8 op/s 2023-03-09T20:53:28.221 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:27 smithi181 ceph-mon[94189]: pgmap v11881: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 101 KiB/s wr, 7 op/s 2023-03-09T20:53:28.221 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21318.609755 secs 2023-03-09T20:53:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:27 smithi038 ceph-mon[92422]: pgmap v11881: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 101 KiB/s wr, 7 op/s 2023-03-09T20:53:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21318.609755 secs 2023-03-09T20:53:28.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:27 smithi122 ceph-mon[93869]: pgmap v11881: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 101 KiB/s wr, 7 op/s 2023-03-09T20:53:28.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21318.609755 secs 2023-03-09T20:53:28.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21780.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:53:29.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:53:29.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:53:29.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:53:29.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:53:29.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:53:29.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:53:29.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:53:29.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:53:29.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:53:29.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:53:29.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:53:29.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:53:29.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:53:29.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:53:29.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:53:29.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:53:29.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:53:29.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:53:29.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:53:29.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:53:29.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:53:30.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:30 smithi122 ceph-mon[93869]: pgmap v11882: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 100 KiB/s wr, 7 op/s 2023-03-09T20:53:30.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:30 smithi181 ceph-mon[94189]: pgmap v11882: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 100 KiB/s wr, 7 op/s 2023-03-09T20:53:30.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:30 smithi038 ceph-mon[92422]: pgmap v11882: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 100 KiB/s wr, 7 op/s 2023-03-09T20:53:31.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21321.290463 secs 2023-03-09T20:53:31.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:31 smithi122 ceph-mon[93869]: pgmap v11883: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 88 KiB/s wr, 8 op/s 2023-03-09T20:53:31.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21321.290463 secs 2023-03-09T20:53:31.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:31 smithi181 ceph-mon[94189]: pgmap v11883: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 88 KiB/s wr, 8 op/s 2023-03-09T20:53:31.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21321.290463 secs 2023-03-09T20:53:31.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:31 smithi038 ceph-mon[92422]: pgmap v11883: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 88 KiB/s wr, 8 op/s 2023-03-09T20:53:32.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21323.609853 secs 2023-03-09T20:53:32.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21323.609853 secs 2023-03-09T20:53:32.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21323.609853 secs 2023-03-09T20:53:33.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:33 smithi122 ceph-mon[93869]: pgmap v11884: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 84 KiB/s wr, 6 op/s 2023-03-09T20:53:33.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:33 smithi181 ceph-mon[94189]: pgmap v11884: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 84 KiB/s wr, 6 op/s 2023-03-09T20:53:33.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:33 smithi038 ceph-mon[92422]: pgmap v11884: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 84 KiB/s wr, 6 op/s 2023-03-09T20:53:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21785.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:53:35.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21326.290524 secs 2023-03-09T20:53:35.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:35 smithi038 ceph-mon[92422]: pgmap v11885: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 84 KiB/s wr, 6 op/s 2023-03-09T20:53:35.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21326.290524 secs 2023-03-09T20:53:35.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:35 smithi122 ceph-mon[93869]: pgmap v11885: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 84 KiB/s wr, 6 op/s 2023-03-09T20:53:35.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21326.290524 secs 2023-03-09T20:53:35.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:35 smithi181 ceph-mon[94189]: pgmap v11885: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 84 KiB/s wr, 6 op/s 2023-03-09T20:53:38.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:37 smithi038 ceph-mon[92422]: pgmap v11886: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 127 KiB/s wr, 8 op/s 2023-03-09T20:53:38.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21328.609971 secs 2023-03-09T20:53:38.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:37 smithi122 ceph-mon[93869]: pgmap v11886: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 127 KiB/s wr, 8 op/s 2023-03-09T20:53:38.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21328.609971 secs 2023-03-09T20:53:38.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:37 smithi181 ceph-mon[94189]: pgmap v11886: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 127 KiB/s wr, 8 op/s 2023-03-09T20:53:38.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21328.609971 secs 2023-03-09T20:53:38.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21790.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:53:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:40 smithi122 ceph-mon[93869]: pgmap v11887: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 6 op/s 2023-03-09T20:53:40.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:53:40.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:53:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:40 smithi181 ceph-mon[94189]: pgmap v11887: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 6 op/s 2023-03-09T20:53:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:53:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:53:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:40 smithi038 ceph-mon[92422]: pgmap v11887: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 44 KiB/s wr, 6 op/s 2023-03-09T20:53:40.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:53:40.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:53:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21331.290584 secs 2023-03-09T20:53:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:53:41.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:53:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21331.290584 secs 2023-03-09T20:53:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:53:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:53:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21331.290584 secs 2023-03-09T20:53:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:53:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:53:42.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:42 smithi122 ceph-mon[93869]: pgmap v11888: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T20:53:42.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:42 smithi181 ceph-mon[94189]: pgmap v11888: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T20:53:42.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:42 smithi038 ceph-mon[92422]: pgmap v11888: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T20:53:43.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21333.610088 secs 2023-03-09T20:53:43.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21333.610088 secs 2023-03-09T20:53:43.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21333.610088 secs 2023-03-09T20:53:43.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21795.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:53:44.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:44 smithi122 ceph-mon[93869]: pgmap v11889: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 198 KiB/s wr, 8 op/s 2023-03-09T20:53:44.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:44 smithi181 ceph-mon[94189]: pgmap v11889: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 198 KiB/s wr, 8 op/s 2023-03-09T20:53:44.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:44 smithi038 ceph-mon[92422]: pgmap v11889: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 198 KiB/s wr, 8 op/s 2023-03-09T20:53:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21336.290644 secs 2023-03-09T20:53:45.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:45 smithi038 ceph-mon[92422]: pgmap v11890: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 215 KiB/s wr, 8 op/s 2023-03-09T20:53:45.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21336.290644 secs 2023-03-09T20:53:45.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:45 smithi122 ceph-mon[93869]: pgmap v11890: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 215 KiB/s wr, 8 op/s 2023-03-09T20:53:45.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21336.290644 secs 2023-03-09T20:53:45.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:45 smithi181 ceph-mon[94189]: pgmap v11890: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 215 KiB/s wr, 8 op/s 2023-03-09T20:53:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:47 smithi038 ceph-mon[92422]: pgmap v11891: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 464 KiB/s rd, 403 KiB/s wr, 11 op/s 2023-03-09T20:53:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21338.610210 secs 2023-03-09T20:53:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:47 smithi122 ceph-mon[93869]: pgmap v11891: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 464 KiB/s rd, 403 KiB/s wr, 11 op/s 2023-03-09T20:53:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21338.610210 secs 2023-03-09T20:53:48.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:47 smithi181 ceph-mon[94189]: pgmap v11891: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 464 KiB/s rd, 403 KiB/s wr, 11 op/s 2023-03-09T20:53:48.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21338.610210 secs 2023-03-09T20:53:48.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21800.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:53:50.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:49 smithi038 ceph-mon[92422]: pgmap v11892: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 377 KiB/s wr, 10 op/s 2023-03-09T20:53:50.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:49 smithi122 ceph-mon[93869]: pgmap v11892: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 377 KiB/s wr, 10 op/s 2023-03-09T20:53:50.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:49 smithi181 ceph-mon[94189]: pgmap v11892: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 377 KiB/s wr, 10 op/s 2023-03-09T20:53:51.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21341.290707 secs 2023-03-09T20:53:51.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21341.290707 secs 2023-03-09T20:53:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21341.290707 secs 2023-03-09T20:53:52.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:52 smithi122 ceph-mon[93869]: pgmap v11893: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 514 KiB/s wr, 12 op/s 2023-03-09T20:53:52.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:53:52.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:53:52.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:52 smithi181 ceph-mon[94189]: pgmap v11893: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 514 KiB/s wr, 12 op/s 2023-03-09T20:53:52.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:53:52.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:53:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:52 smithi038 ceph-mon[92422]: pgmap v11893: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 514 KiB/s wr, 12 op/s 2023-03-09T20:53:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:53:52.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:53:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:53:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:53:53.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21343.610330 secs 2023-03-09T20:53:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:53:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:53:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21343.610330 secs 2023-03-09T20:53:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:53:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:53:53.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21343.610330 secs 2023-03-09T20:53:53.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21805.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:53:54.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:54 smithi122 ceph-mon[93869]: pgmap v11894: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 592 KiB/s rd, 514 KiB/s wr, 12 op/s 2023-03-09T20:53:54.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:54 smithi181 ceph-mon[94189]: pgmap v11894: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 592 KiB/s rd, 514 KiB/s wr, 12 op/s 2023-03-09T20:53:54.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:54 smithi038 ceph-mon[92422]: pgmap v11894: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 592 KiB/s rd, 514 KiB/s wr, 12 op/s 2023-03-09T20:53:55.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21346.290766 secs 2023-03-09T20:53:55.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:55 smithi038 ceph-mon[92422]: pgmap v11895: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 378 KiB/s wr, 11 op/s 2023-03-09T20:53:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21346.290766 secs 2023-03-09T20:53:55.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:55 smithi122 ceph-mon[93869]: pgmap v11895: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 378 KiB/s wr, 11 op/s 2023-03-09T20:53:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21346.290766 secs 2023-03-09T20:53:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:55 smithi181 ceph-mon[94189]: pgmap v11895: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 378 KiB/s wr, 11 op/s 2023-03-09T20:53:58.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:57 smithi038 ceph-mon[92422]: pgmap v11896: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 556 KiB/s rd, 520 KiB/s wr, 14 op/s 2023-03-09T20:53:58.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:53:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21348.610455 secs 2023-03-09T20:53:58.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:57 smithi122 ceph-mon[93869]: pgmap v11896: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 556 KiB/s rd, 520 KiB/s wr, 14 op/s 2023-03-09T20:53:58.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:53:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21348.610455 secs 2023-03-09T20:53:58.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:57 smithi181 ceph-mon[94189]: pgmap v11896: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 556 KiB/s rd, 520 KiB/s wr, 14 op/s 2023-03-09T20:53:58.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:53:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21348.610455 secs 2023-03-09T20:53:58.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21810.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:54:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:00 smithi122 ceph-mon[93869]: pgmap v11897: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 478 KiB/s rd, 459 KiB/s wr, 11 op/s 2023-03-09T20:54:00.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:54:00.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:54:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:00 smithi181 ceph-mon[94189]: pgmap v11897: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 478 KiB/s rd, 459 KiB/s wr, 11 op/s 2023-03-09T20:54:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:54:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:54:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:00 smithi038 ceph-mon[92422]: pgmap v11897: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 478 KiB/s rd, 459 KiB/s wr, 11 op/s 2023-03-09T20:54:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:54:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:54:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21351.290829 secs 2023-03-09T20:54:01.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:54:01.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:54:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21351.290829 secs 2023-03-09T20:54:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:54:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:54:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21351.290829 secs 2023-03-09T20:54:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:54:01.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:54:02.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:02 smithi122 ceph-mon[93869]: pgmap v11898: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 508 KiB/s rd, 509 KiB/s wr, 14 op/s 2023-03-09T20:54:02.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:02 smithi181 ceph-mon[94189]: pgmap v11898: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 508 KiB/s rd, 509 KiB/s wr, 14 op/s 2023-03-09T20:54:02.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:02 smithi038 ceph-mon[92422]: pgmap v11898: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 508 KiB/s rd, 509 KiB/s wr, 14 op/s 2023-03-09T20:54:03.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21353.610573 secs 2023-03-09T20:54:03.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21353.610573 secs 2023-03-09T20:54:03.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21353.610573 secs 2023-03-09T20:54:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21815.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:54:04.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:04 smithi122 ceph-mon[93869]: pgmap v11899: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 474 KiB/s wr, 13 op/s 2023-03-09T20:54:04.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:04 smithi181 ceph-mon[94189]: pgmap v11899: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 474 KiB/s wr, 13 op/s 2023-03-09T20:54:04.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:04 smithi038 ceph-mon[92422]: pgmap v11899: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 474 KiB/s wr, 13 op/s 2023-03-09T20:54:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21356.290907 secs 2023-03-09T20:54:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:05 smithi038 ceph-mon[92422]: pgmap v11900: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 513 KiB/s wr, 24 op/s 2023-03-09T20:54:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21356.290907 secs 2023-03-09T20:54:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:05 smithi122 ceph-mon[93869]: pgmap v11900: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 513 KiB/s wr, 24 op/s 2023-03-09T20:54:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21356.290907 secs 2023-03-09T20:54:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:05 smithi181 ceph-mon[94189]: pgmap v11900: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 513 KiB/s wr, 24 op/s 2023-03-09T20:54:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:08 smithi122 ceph-mon[93869]: pgmap v11901: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 496 KiB/s wr, 26 op/s 2023-03-09T20:54:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21358.610696 secs 2023-03-09T20:54:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:08 smithi181 ceph-mon[94189]: pgmap v11901: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 496 KiB/s wr, 26 op/s 2023-03-09T20:54:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21358.610696 secs 2023-03-09T20:54:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:08 smithi038 ceph-mon[92422]: pgmap v11901: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 496 KiB/s wr, 26 op/s 2023-03-09T20:54:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21358.610696 secs 2023-03-09T20:54:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21820.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:54:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:10 smithi122 ceph-mon[93869]: pgmap v11902: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 586 KiB/s wr, 24 op/s 2023-03-09T20:54:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:10 smithi122 ceph-mon[93869]: osdmap e957: 12 total, 12 up, 12 in 2023-03-09T20:54:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:10 smithi181 ceph-mon[94189]: pgmap v11902: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 586 KiB/s wr, 24 op/s 2023-03-09T20:54:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:10 smithi181 ceph-mon[94189]: osdmap e957: 12 total, 12 up, 12 in 2023-03-09T20:54:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:10 smithi038 ceph-mon[92422]: pgmap v11902: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 586 KiB/s wr, 24 op/s 2023-03-09T20:54:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:10 smithi038 ceph-mon[92422]: osdmap e957: 12 total, 12 up, 12 in 2023-03-09T20:54:11.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21361.290969 secs 2023-03-09T20:54:11.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21361.290969 secs 2023-03-09T20:54:11.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21361.290969 secs 2023-03-09T20:54:12.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:12 smithi122 ceph-mon[93869]: pgmap v11904: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 455 KiB/s rd, 470 KiB/s wr, 25 op/s 2023-03-09T20:54:12.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:54:12.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:54:12.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:12 smithi181 ceph-mon[94189]: pgmap v11904: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 455 KiB/s rd, 470 KiB/s wr, 25 op/s 2023-03-09T20:54:12.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:54:12.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:54:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:12 smithi038 ceph-mon[92422]: pgmap v11904: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 455 KiB/s rd, 470 KiB/s wr, 25 op/s 2023-03-09T20:54:12.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:54:12.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:54:13.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:54:13.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:54:13.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21363.610823 secs 2023-03-09T20:54:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:54:13.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:54:13.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21363.610823 secs 2023-03-09T20:54:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:54:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:54:13.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21363.610823 secs 2023-03-09T20:54:13.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21825.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:54:14.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:14 smithi122 ceph-mon[93869]: pgmap v11905: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 347 KiB/s wr, 25 op/s 2023-03-09T20:54:14.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:14 smithi181 ceph-mon[94189]: pgmap v11905: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 347 KiB/s wr, 25 op/s 2023-03-09T20:54:14.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:14 smithi038 ceph-mon[92422]: pgmap v11905: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 347 KiB/s wr, 25 op/s 2023-03-09T20:54:15.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21366.290986 secs 2023-03-09T20:54:15.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:15 smithi038 ceph-mon[92422]: pgmap v11906: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 307 KiB/s wr, 10 op/s 2023-03-09T20:54:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21366.290986 secs 2023-03-09T20:54:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:15 smithi122 ceph-mon[93869]: pgmap v11906: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 307 KiB/s wr, 10 op/s 2023-03-09T20:54:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21366.290986 secs 2023-03-09T20:54:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:15 smithi181 ceph-mon[94189]: pgmap v11906: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 307 KiB/s wr, 10 op/s 2023-03-09T20:54:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:18 smithi122 ceph-mon[93869]: pgmap v11907: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 306 KiB/s wr, 10 op/s 2023-03-09T20:54:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21368.610962 secs 2023-03-09T20:54:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:18 smithi181 ceph-mon[94189]: pgmap v11907: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 306 KiB/s wr, 10 op/s 2023-03-09T20:54:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21368.610962 secs 2023-03-09T20:54:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:18 smithi038 ceph-mon[92422]: pgmap v11907: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 306 KiB/s wr, 10 op/s 2023-03-09T20:54:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21368.610962 secs 2023-03-09T20:54:18.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21830.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:54:20.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:20 smithi122 ceph-mon[93869]: pgmap v11908: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 75 KiB/s wr, 9 op/s 2023-03-09T20:54:20.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:20 smithi122 ceph-mon[93869]: osdmap e958: 12 total, 12 up, 12 in 2023-03-09T20:54:20.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:20 smithi181 ceph-mon[94189]: pgmap v11908: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 75 KiB/s wr, 9 op/s 2023-03-09T20:54:20.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:20 smithi181 ceph-mon[94189]: osdmap e958: 12 total, 12 up, 12 in 2023-03-09T20:54:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:20 smithi038 ceph-mon[92422]: pgmap v11908: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 75 KiB/s wr, 9 op/s 2023-03-09T20:54:20.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:20 smithi038 ceph-mon[92422]: osdmap e958: 12 total, 12 up, 12 in 2023-03-09T20:54:21.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21371.291048 secs 2023-03-09T20:54:21.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21371.291048 secs 2023-03-09T20:54:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21371.291048 secs 2023-03-09T20:54:22.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:22 smithi122 ceph-mon[93869]: pgmap v11910: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 76 KiB/s wr, 9 op/s 2023-03-09T20:54:22.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:22 smithi181 ceph-mon[94189]: pgmap v11910: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 76 KiB/s wr, 9 op/s 2023-03-09T20:54:22.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:22 smithi038 ceph-mon[92422]: pgmap v11910: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 76 KiB/s wr, 9 op/s 2023-03-09T20:54:23.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21373.611078 secs 2023-03-09T20:54:23.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21373.611078 secs 2023-03-09T20:54:23.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21373.611078 secs 2023-03-09T20:54:23.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21835.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:54:24.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:24 smithi122 ceph-mon[93869]: pgmap v11911: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 91 KiB/s wr, 9 op/s 2023-03-09T20:54:24.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:24 smithi181 ceph-mon[94189]: pgmap v11911: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 91 KiB/s wr, 9 op/s 2023-03-09T20:54:24.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:24 smithi038 ceph-mon[92422]: pgmap v11911: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 91 KiB/s wr, 9 op/s 2023-03-09T20:54:25.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21376.291107 secs 2023-03-09T20:54:25.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:25 smithi122 ceph-mon[93869]: pgmap v11912: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 85 KiB/s wr, 8 op/s 2023-03-09T20:54:25.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21376.291107 secs 2023-03-09T20:54:25.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:25 smithi181 ceph-mon[94189]: pgmap v11912: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 85 KiB/s wr, 8 op/s 2023-03-09T20:54:26.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21376.291107 secs 2023-03-09T20:54:26.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:25 smithi038 ceph-mon[92422]: pgmap v11912: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 85 KiB/s wr, 8 op/s 2023-03-09T20:54:28.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:28 smithi122 ceph-mon[93869]: pgmap v11913: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 85 KiB/s wr, 9 op/s 2023-03-09T20:54:28.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21378.611194 secs 2023-03-09T20:54:28.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:28 smithi181 ceph-mon[94189]: pgmap v11913: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 85 KiB/s wr, 9 op/s 2023-03-09T20:54:28.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21378.611194 secs 2023-03-09T20:54:28.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:28 smithi038 ceph-mon[92422]: pgmap v11913: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 85 KiB/s wr, 9 op/s 2023-03-09T20:54:28.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21378.611194 secs 2023-03-09T20:54:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21840.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:54:30.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:30 smithi122 ceph-mon[93869]: pgmap v11914: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 197 KiB/s wr, 9 op/s 2023-03-09T20:54:30.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:30 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:54:30.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:30 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:54:30.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:30 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:54:30.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:30 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:54:30.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:30 smithi181 ceph-mon[94189]: pgmap v11914: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 197 KiB/s wr, 9 op/s 2023-03-09T20:54:30.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:30 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:54:30.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:30 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:54:30.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:30 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:54:30.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:30 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:54:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:30 smithi038 ceph-mon[92422]: pgmap v11914: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 197 KiB/s wr, 9 op/s 2023-03-09T20:54:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:30 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:54:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:30 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:54:30.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:30 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:54:30.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:30 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:54:31.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21381.291169 secs 2023-03-09T20:54:31.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21381.291169 secs 2023-03-09T20:54:31.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21381.291169 secs 2023-03-09T20:54:32.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:32 smithi122 ceph-mon[93869]: pgmap v11915: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 172 KiB/s wr, 10 op/s 2023-03-09T20:54:32.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:32 smithi181 ceph-mon[94189]: pgmap v11915: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 172 KiB/s wr, 10 op/s 2023-03-09T20:54:32.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:32 smithi038 ceph-mon[92422]: pgmap v11915: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 172 KiB/s wr, 10 op/s 2023-03-09T20:54:33.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21383.611315 secs 2023-03-09T20:54:33.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21383.611315 secs 2023-03-09T20:54:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21383.611315 secs 2023-03-09T20:54:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21845.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:54:34.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:34 smithi122 ceph-mon[93869]: pgmap v11916: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 164 KiB/s wr, 8 op/s 2023-03-09T20:54:34.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:34 smithi181 ceph-mon[94189]: pgmap v11916: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 164 KiB/s wr, 8 op/s 2023-03-09T20:54:34.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:34 smithi038 ceph-mon[92422]: pgmap v11916: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 164 KiB/s wr, 8 op/s 2023-03-09T20:54:35.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21386.291229 secs 2023-03-09T20:54:35.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:35 smithi038 ceph-mon[92422]: pgmap v11917: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 152 KiB/s wr, 8 op/s 2023-03-09T20:54:35.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21386.291229 secs 2023-03-09T20:54:35.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:35 smithi122 ceph-mon[93869]: pgmap v11917: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 152 KiB/s wr, 8 op/s 2023-03-09T20:54:35.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21386.291229 secs 2023-03-09T20:54:35.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:35 smithi181 ceph-mon[94189]: pgmap v11917: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 152 KiB/s wr, 8 op/s 2023-03-09T20:54:38.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:38 smithi122 ceph-mon[93869]: pgmap v11918: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 210 KiB/s wr, 10 op/s 2023-03-09T20:54:38.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:38 smithi181 ceph-mon[94189]: pgmap v11918: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 210 KiB/s wr, 10 op/s 2023-03-09T20:54:38.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:38 smithi038 ceph-mon[92422]: pgmap v11918: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 210 KiB/s wr, 10 op/s 2023-03-09T20:54:38.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21850.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:54:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:40 smithi122 ceph-mon[93869]: pgmap v11919: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 241 KiB/s wr, 8 op/s 2023-03-09T20:54:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:54:40.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:54:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:40 smithi181 ceph-mon[94189]: pgmap v11919: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 241 KiB/s wr, 8 op/s 2023-03-09T20:54:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:54:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:54:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:40 smithi038 ceph-mon[92422]: pgmap v11919: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 241 KiB/s wr, 8 op/s 2023-03-09T20:54:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:54:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:54:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21391.291288 secs 2023-03-09T20:54:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:41 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:54:41.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:41 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:54:41.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:54:41.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:54:41.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:54:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21391.291288 secs 2023-03-09T20:54:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:41 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:54:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:41 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:54:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:54:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:54:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:54:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21391.291288 secs 2023-03-09T20:54:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:41 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:54:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:41 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:54:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:54:41.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:54:41.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:54:42.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:42 smithi122 ceph-mon[93869]: pgmap v11920: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 288 KiB/s wr, 10 op/s 2023-03-09T20:54:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:42 smithi181 ceph-mon[94189]: pgmap v11920: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 288 KiB/s wr, 10 op/s 2023-03-09T20:54:42.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:42 smithi038 ceph-mon[92422]: pgmap v11920: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 288 KiB/s wr, 10 op/s 2023-03-09T20:54:43.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21855.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:54:44.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:44 smithi122 ceph-mon[93869]: pgmap v11921: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 305 KiB/s wr, 9 op/s 2023-03-09T20:54:44.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:44 smithi181 ceph-mon[94189]: pgmap v11921: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 305 KiB/s wr, 9 op/s 2023-03-09T20:54:44.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:44 smithi038 ceph-mon[92422]: pgmap v11921: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 305 KiB/s wr, 9 op/s 2023-03-09T20:54:45.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21396.291348 secs 2023-03-09T20:54:45.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:45 smithi122 ceph-mon[93869]: pgmap v11922: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 305 KiB/s wr, 8 op/s 2023-03-09T20:54:45.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21396.291348 secs 2023-03-09T20:54:45.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:45 smithi181 ceph-mon[94189]: pgmap v11922: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 305 KiB/s wr, 8 op/s 2023-03-09T20:54:46.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21396.291348 secs 2023-03-09T20:54:46.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:45 smithi038 ceph-mon[92422]: pgmap v11922: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 305 KiB/s wr, 8 op/s 2023-03-09T20:54:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:48 smithi122 ceph-mon[93869]: pgmap v11923: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 408 KiB/s wr, 11 op/s 2023-03-09T20:54:48.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:48 smithi181 ceph-mon[94189]: pgmap v11923: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 408 KiB/s wr, 11 op/s 2023-03-09T20:54:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:48 smithi038 ceph-mon[92422]: pgmap v11923: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 408 KiB/s wr, 11 op/s 2023-03-09T20:54:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21860.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:54:50.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:50 smithi122 ceph-mon[93869]: pgmap v11924: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 568 KiB/s wr, 11 op/s 2023-03-09T20:54:50.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:50 smithi181 ceph-mon[94189]: pgmap v11924: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 568 KiB/s wr, 11 op/s 2023-03-09T20:54:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:50 smithi038 ceph-mon[92422]: pgmap v11924: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 568 KiB/s wr, 11 op/s 2023-03-09T20:54:51.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21401.291409 secs 2023-03-09T20:54:51.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21401.291409 secs 2023-03-09T20:54:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21401.291409 secs 2023-03-09T20:54:52.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:52 smithi122 ceph-mon[93869]: pgmap v11925: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 555 KiB/s rd, 539 KiB/s wr, 12 op/s 2023-03-09T20:54:52.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:54:52.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:54:52.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:52 smithi181 ceph-mon[94189]: pgmap v11925: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 555 KiB/s rd, 539 KiB/s wr, 12 op/s 2023-03-09T20:54:52.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:54:52.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:54:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:52 smithi038 ceph-mon[92422]: pgmap v11925: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 555 KiB/s rd, 539 KiB/s wr, 12 op/s 2023-03-09T20:54:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:54:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:54:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:54:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:54:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:54:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:54:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:54:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:54:53.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21865.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:54:54.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:54 smithi122 ceph-mon[93869]: pgmap v11926: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 360 KiB/s wr, 11 op/s 2023-03-09T20:54:54.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:54 smithi181 ceph-mon[94189]: pgmap v11926: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 360 KiB/s wr, 11 op/s 2023-03-09T20:54:54.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:54 smithi038 ceph-mon[92422]: pgmap v11926: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 360 KiB/s wr, 11 op/s 2023-03-09T20:54:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21406.291471 secs 2023-03-09T20:54:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:55 smithi122 ceph-mon[93869]: pgmap v11927: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 466 KiB/s wr, 11 op/s 2023-03-09T20:54:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21406.291471 secs 2023-03-09T20:54:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:55 smithi181 ceph-mon[94189]: pgmap v11927: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 466 KiB/s wr, 11 op/s 2023-03-09T20:54:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21406.291471 secs 2023-03-09T20:54:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:55 smithi038 ceph-mon[92422]: pgmap v11927: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 466 KiB/s wr, 11 op/s 2023-03-09T20:54:57.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:54:57.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:54:57.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:57 smithi122 ceph-mon[93869]: pgmap v11928: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 601 KiB/s rd, 586 KiB/s wr, 15 op/s 2023-03-09T20:54:57.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:54:57.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:54:57.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:57 smithi181 ceph-mon[94189]: pgmap v11928: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 601 KiB/s rd, 586 KiB/s wr, 15 op/s 2023-03-09T20:54:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:54:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:54:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:57 smithi038 ceph-mon[92422]: pgmap v11928: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 601 KiB/s rd, 586 KiB/s wr, 15 op/s 2023-03-09T20:54:58.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21870.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:54:59.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:54:59.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:54:59.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:54:59 smithi122 ceph-mon[93869]: pgmap v11929: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 483 KiB/s wr, 11 op/s 2023-03-09T20:54:59.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:54:59.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:54:59.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:54:59 smithi181 ceph-mon[94189]: pgmap v11929: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 483 KiB/s wr, 11 op/s 2023-03-09T20:55:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:55:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:55:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:54:59 smithi038 ceph-mon[92422]: pgmap v11929: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 483 KiB/s wr, 11 op/s 2023-03-09T20:55:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21411.291533 secs 2023-03-09T20:55:00.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21411.291533 secs 2023-03-09T20:55:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21411.291533 secs 2023-03-09T20:55:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:01 smithi122 ceph-mon[93869]: pgmap v11930: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 446 KiB/s rd, 455 KiB/s wr, 15 op/s 2023-03-09T20:55:01.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:01 smithi181 ceph-mon[94189]: pgmap v11930: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 446 KiB/s rd, 455 KiB/s wr, 15 op/s 2023-03-09T20:55:02.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:01 smithi038 ceph-mon[92422]: pgmap v11930: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 446 KiB/s rd, 455 KiB/s wr, 15 op/s 2023-03-09T20:55:02.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21413.612036 secs 2023-03-09T20:55:02.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21413.612036 secs 2023-03-09T20:55:03.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21413.612036 secs 2023-03-09T20:55:03.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21875.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:55:03.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:03 smithi122 ceph-mon[93869]: pgmap v11931: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 471 KiB/s wr, 13 op/s 2023-03-09T20:55:03.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:03 smithi181 ceph-mon[94189]: pgmap v11931: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 471 KiB/s wr, 13 op/s 2023-03-09T20:55:04.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:03 smithi038 ceph-mon[92422]: pgmap v11931: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 471 KiB/s wr, 13 op/s 2023-03-09T20:55:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:05 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:55:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:05 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:55:05.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:05 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:55:05.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21416.291594 secs 2023-03-09T20:55:05.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:05 smithi122 ceph-mon[93869]: pgmap v11932: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 475 KiB/s wr, 24 op/s 2023-03-09T20:55:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:05 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:55:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:05 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:55:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:05 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:55:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21416.291594 secs 2023-03-09T20:55:05.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:05 smithi181 ceph-mon[94189]: pgmap v11932: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 475 KiB/s wr, 24 op/s 2023-03-09T20:55:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:05 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:55:06.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:05 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:55:06.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:05 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:55:06.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21416.291594 secs 2023-03-09T20:55:06.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:05 smithi038 ceph-mon[92422]: pgmap v11932: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 475 KiB/s wr, 24 op/s 2023-03-09T20:55:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:08 smithi122 ceph-mon[93869]: pgmap v11933: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 399 KiB/s rd, 476 KiB/s wr, 27 op/s 2023-03-09T20:55:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21418.612176 secs 2023-03-09T20:55:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:08 smithi181 ceph-mon[94189]: pgmap v11933: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 399 KiB/s rd, 476 KiB/s wr, 27 op/s 2023-03-09T20:55:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21418.612176 secs 2023-03-09T20:55:08.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:08 smithi038 ceph-mon[92422]: pgmap v11933: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 399 KiB/s rd, 476 KiB/s wr, 27 op/s 2023-03-09T20:55:08.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21418.612176 secs 2023-03-09T20:55:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21880.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:55:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:10 smithi122 ceph-mon[93869]: pgmap v11934: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 355 KiB/s wr, 23 op/s 2023-03-09T20:55:10.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:10 smithi122 ceph-mon[93869]: osdmap e959: 12 total, 12 up, 12 in 2023-03-09T20:55:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:10 smithi181 ceph-mon[94189]: pgmap v11934: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 355 KiB/s wr, 23 op/s 2023-03-09T20:55:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:10 smithi181 ceph-mon[94189]: osdmap e959: 12 total, 12 up, 12 in 2023-03-09T20:55:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:10 smithi038 ceph-mon[92422]: pgmap v11934: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 355 KiB/s wr, 23 op/s 2023-03-09T20:55:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:10 smithi038 ceph-mon[92422]: osdmap e959: 12 total, 12 up, 12 in 2023-03-09T20:55:11.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21421.291657 secs 2023-03-09T20:55:11.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21421.291657 secs 2023-03-09T20:55:11.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21421.291657 secs 2023-03-09T20:55:12.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:12 smithi122 ceph-mon[93869]: pgmap v11936: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 197 KiB/s wr, 24 op/s 2023-03-09T20:55:12.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:55:12.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:55:12.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:12 smithi181 ceph-mon[94189]: pgmap v11936: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 197 KiB/s wr, 24 op/s 2023-03-09T20:55:12.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:55:12.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:55:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:12 smithi038 ceph-mon[92422]: pgmap v11936: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 197 KiB/s wr, 24 op/s 2023-03-09T20:55:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:55:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:55:13.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:55:13.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:55:13.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21423.612268 secs 2023-03-09T20:55:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:55:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:55:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21423.612268 secs 2023-03-09T20:55:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:55:13.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:55:13.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21423.612268 secs 2023-03-09T20:55:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21885.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:55:14.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:14 smithi122 ceph-mon[93869]: pgmap v11937: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 197 KiB/s wr, 24 op/s 2023-03-09T20:55:14.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:14 smithi181 ceph-mon[94189]: pgmap v11937: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 197 KiB/s wr, 24 op/s 2023-03-09T20:55:14.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:14 smithi038 ceph-mon[92422]: pgmap v11937: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 197 KiB/s wr, 24 op/s 2023-03-09T20:55:15.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21426.291719 secs 2023-03-09T20:55:15.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:15 smithi122 ceph-mon[93869]: pgmap v11938: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 171 KiB/s wr, 9 op/s 2023-03-09T20:55:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21426.291719 secs 2023-03-09T20:55:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:15 smithi181 ceph-mon[94189]: pgmap v11938: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 171 KiB/s wr, 9 op/s 2023-03-09T20:55:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21426.291719 secs 2023-03-09T20:55:16.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:15 smithi038 ceph-mon[92422]: pgmap v11938: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 171 KiB/s wr, 9 op/s 2023-03-09T20:55:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:18 smithi122 ceph-mon[93869]: pgmap v11939: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 121 KiB/s wr, 8 op/s 2023-03-09T20:55:18.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21428.612381 secs 2023-03-09T20:55:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:18 smithi181 ceph-mon[94189]: pgmap v11939: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 121 KiB/s wr, 8 op/s 2023-03-09T20:55:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21428.612381 secs 2023-03-09T20:55:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:18 smithi038 ceph-mon[92422]: pgmap v11939: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 121 KiB/s wr, 8 op/s 2023-03-09T20:55:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21428.612381 secs 2023-03-09T20:55:18.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21890.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:55:20.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:20 smithi122 ceph-mon[93869]: pgmap v11940: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 121 KiB/s wr, 8 op/s 2023-03-09T20:55:20.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:20 smithi122 ceph-mon[93869]: osdmap e960: 12 total, 12 up, 12 in 2023-03-09T20:55:20.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:20 smithi181 ceph-mon[94189]: pgmap v11940: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 121 KiB/s wr, 8 op/s 2023-03-09T20:55:20.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:20 smithi181 ceph-mon[94189]: osdmap e960: 12 total, 12 up, 12 in 2023-03-09T20:55:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:20 smithi038 ceph-mon[92422]: pgmap v11940: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 121 KiB/s wr, 8 op/s 2023-03-09T20:55:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:20 smithi038 ceph-mon[92422]: osdmap e960: 12 total, 12 up, 12 in 2023-03-09T20:55:21.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21431.291779 secs 2023-03-09T20:55:21.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21431.291779 secs 2023-03-09T20:55:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21431.291779 secs 2023-03-09T20:55:22.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:22 smithi122 ceph-mon[93869]: pgmap v11942: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 122 KiB/s wr, 8 op/s 2023-03-09T20:55:22.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:22 smithi181 ceph-mon[94189]: pgmap v11942: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 122 KiB/s wr, 8 op/s 2023-03-09T20:55:22.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:22 smithi038 ceph-mon[92422]: pgmap v11942: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 122 KiB/s wr, 8 op/s 2023-03-09T20:55:23.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21433.612500 secs 2023-03-09T20:55:23.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21433.612500 secs 2023-03-09T20:55:23.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21433.612500 secs 2023-03-09T20:55:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21895.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:55:24.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:24 smithi122 ceph-mon[93869]: pgmap v11943: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 102 KiB/s wr, 8 op/s 2023-03-09T20:55:24.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:24 smithi181 ceph-mon[94189]: pgmap v11943: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 102 KiB/s wr, 8 op/s 2023-03-09T20:55:24.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:24 smithi038 ceph-mon[92422]: pgmap v11943: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 102 KiB/s wr, 8 op/s 2023-03-09T20:55:25.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21436.291869 secs 2023-03-09T20:55:25.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:25 smithi122 ceph-mon[93869]: pgmap v11944: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 102 KiB/s wr, 8 op/s 2023-03-09T20:55:25.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21436.291869 secs 2023-03-09T20:55:25.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:25 smithi181 ceph-mon[94189]: pgmap v11944: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 102 KiB/s wr, 8 op/s 2023-03-09T20:55:26.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21436.291869 secs 2023-03-09T20:55:26.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:25 smithi038 ceph-mon[92422]: pgmap v11944: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 102 KiB/s wr, 8 op/s 2023-03-09T20:55:28.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:28 smithi122 ceph-mon[93869]: pgmap v11945: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T20:55:28.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21438.612639 secs 2023-03-09T20:55:28.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:28 smithi181 ceph-mon[94189]: pgmap v11945: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T20:55:28.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21438.612639 secs 2023-03-09T20:55:28.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:28 smithi038 ceph-mon[92422]: pgmap v11945: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T20:55:28.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21438.612639 secs 2023-03-09T20:55:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21900.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:55:30.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:30 smithi122 ceph-mon[93869]: pgmap v11946: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T20:55:30.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:30 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:55:30.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:30 smithi181 ceph-mon[94189]: pgmap v11946: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T20:55:30.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:30 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:55:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:30 smithi038 ceph-mon[92422]: pgmap v11946: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 32 KiB/s wr, 7 op/s 2023-03-09T20:55:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:30 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:55:31.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21441.291928 secs 2023-03-09T20:55:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21441.291928 secs 2023-03-09T20:55:31.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21441.291928 secs 2023-03-09T20:55:32.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:32 smithi122 ceph-mon[93869]: pgmap v11947: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 115 KiB/s wr, 9 op/s 2023-03-09T20:55:32.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:55:32.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:55:32.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:55:32.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:55:32.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:55:32.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:55:32.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:55:32.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:55:32.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:55:32.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:32 smithi181 ceph-mon[94189]: pgmap v11947: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 115 KiB/s wr, 9 op/s 2023-03-09T20:55:32.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:55:32.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:55:32.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:55:32.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:55:32.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:55:32.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:55:32.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:55:32.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:55:32.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:55:32.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:32 smithi038 ceph-mon[92422]: pgmap v11947: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 115 KiB/s wr, 9 op/s 2023-03-09T20:55:32.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:55:32.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:55:32.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:55:32.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:55:32.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:55:32.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:55:32.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:55:32.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:55:32.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:55:33.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21443.612778 secs 2023-03-09T20:55:33.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21443.612778 secs 2023-03-09T20:55:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21443.612778 secs 2023-03-09T20:55:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21905.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:55:34.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:34 smithi122 ceph-mon[93869]: pgmap v11948: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 110 KiB/s wr, 7 op/s 2023-03-09T20:55:34.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:34 smithi181 ceph-mon[94189]: pgmap v11948: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 110 KiB/s wr, 7 op/s 2023-03-09T20:55:34.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:34 smithi038 ceph-mon[92422]: pgmap v11948: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 110 KiB/s wr, 7 op/s 2023-03-09T20:55:35.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21446.291983 secs 2023-03-09T20:55:35.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:35 smithi122 ceph-mon[93869]: pgmap v11949: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 110 KiB/s wr, 6 op/s 2023-03-09T20:55:35.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21446.291983 secs 2023-03-09T20:55:35.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:35 smithi181 ceph-mon[94189]: pgmap v11949: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 110 KiB/s wr, 6 op/s 2023-03-09T20:55:36.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21446.291983 secs 2023-03-09T20:55:36.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:35 smithi038 ceph-mon[92422]: pgmap v11949: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 110 KiB/s wr, 6 op/s 2023-03-09T20:55:38.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:38 smithi122 ceph-mon[93869]: pgmap v11950: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 110 KiB/s wr, 9 op/s 2023-03-09T20:55:38.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:38 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 21448.612888 secs 2023-03-09T20:55:38.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:38 smithi181 ceph-mon[94189]: pgmap v11950: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 110 KiB/s wr, 9 op/s 2023-03-09T20:55:38.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:38 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 21448.612888 secs 2023-03-09T20:55:38.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:38 smithi038 ceph-mon[92422]: pgmap v11950: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 110 KiB/s wr, 9 op/s 2023-03-09T20:55:38.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:38 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 21448.612888 secs 2023-03-09T20:55:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21910.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:55:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:40 smithi122 ceph-mon[93869]: pgmap v11951: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 84 KiB/s wr, 6 op/s 2023-03-09T20:55:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:40 smithi181 ceph-mon[94189]: pgmap v11951: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 84 KiB/s wr, 6 op/s 2023-03-09T20:55:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:40 smithi038 ceph-mon[92422]: pgmap v11951: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 84 KiB/s wr, 6 op/s 2023-03-09T20:55:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21451.292049 secs 2023-03-09T20:55:41.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:55:41.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:55:41.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:55:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21451.292049 secs 2023-03-09T20:55:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:55:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:55:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:55:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21451.292049 secs 2023-03-09T20:55:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:55:41.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:55:41.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:55:42.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:42 smithi122 ceph-mon[93869]: pgmap v11952: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 260 KiB/s wr, 8 op/s 2023-03-09T20:55:42.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:42 smithi181 ceph-mon[94189]: pgmap v11952: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 260 KiB/s wr, 8 op/s 2023-03-09T20:55:42.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:42 smithi038 ceph-mon[92422]: pgmap v11952: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 260 KiB/s wr, 8 op/s 2023-03-09T20:55:43.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:43 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 21453.613004 secs 2023-03-09T20:55:43.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:43 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 21453.613004 secs 2023-03-09T20:55:43.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:43 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 21453.613004 secs 2023-03-09T20:55:43.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21915.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:55:44.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:44 smithi122 ceph-mon[93869]: pgmap v11953: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 194 KiB/s wr, 7 op/s 2023-03-09T20:55:44.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:44 smithi181 ceph-mon[94189]: pgmap v11953: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 194 KiB/s wr, 7 op/s 2023-03-09T20:55:44.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:44 smithi038 ceph-mon[92422]: pgmap v11953: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 194 KiB/s wr, 7 op/s 2023-03-09T20:55:45.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21456.292127 secs 2023-03-09T20:55:45.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:45 smithi122 ceph-mon[93869]: pgmap v11954: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 217 KiB/s wr, 7 op/s 2023-03-09T20:55:45.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21456.292127 secs 2023-03-09T20:55:45.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:45 smithi181 ceph-mon[94189]: pgmap v11954: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 217 KiB/s wr, 7 op/s 2023-03-09T20:55:46.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21456.292127 secs 2023-03-09T20:55:46.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:45 smithi038 ceph-mon[92422]: pgmap v11954: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 217 KiB/s wr, 7 op/s 2023-03-09T20:55:47.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:55:47.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:55:47.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:47 smithi122 ceph-mon[93869]: pgmap v11955: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 234 KiB/s wr, 10 op/s 2023-03-09T20:55:47.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21458.613139 secs 2023-03-09T20:55:47.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:55:47.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:55:47.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:47 smithi181 ceph-mon[94189]: pgmap v11955: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 234 KiB/s wr, 10 op/s 2023-03-09T20:55:47.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21458.613139 secs 2023-03-09T20:55:48.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:55:48.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:55:48.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:47 smithi038 ceph-mon[92422]: pgmap v11955: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 234 KiB/s wr, 10 op/s 2023-03-09T20:55:48.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21458.613139 secs 2023-03-09T20:55:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21920.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:55:49.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:55:49.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:55:49.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:49 smithi122 ceph-mon[93869]: pgmap v11956: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 284 KiB/s wr, 8 op/s 2023-03-09T20:55:49.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:55:49.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:55:49.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:49 smithi181 ceph-mon[94189]: pgmap v11956: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 284 KiB/s wr, 8 op/s 2023-03-09T20:55:50.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:55:50.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:55:50.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:49 smithi038 ceph-mon[92422]: pgmap v11956: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 284 KiB/s wr, 8 op/s 2023-03-09T20:55:50.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21461.292189 secs 2023-03-09T20:55:50.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21461.292189 secs 2023-03-09T20:55:51.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21461.292189 secs 2023-03-09T20:55:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:55:51.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:55:51.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:51 smithi122 ceph-mon[93869]: pgmap v11957: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 555 KiB/s rd, 539 KiB/s wr, 11 op/s 2023-03-09T20:55:51.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:55:51.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:55:51.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:51 smithi181 ceph-mon[94189]: pgmap v11957: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 555 KiB/s rd, 539 KiB/s wr, 11 op/s 2023-03-09T20:55:52.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:55:52.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:55:52.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:51 smithi038 ceph-mon[92422]: pgmap v11957: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 555 KiB/s rd, 539 KiB/s wr, 11 op/s 2023-03-09T20:55:52.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21463.613201 secs 2023-03-09T20:55:52.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21463.613201 secs 2023-03-09T20:55:53.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21463.613201 secs 2023-03-09T20:55:53.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21925.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:55:53.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:55:53.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:55:53.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:53 smithi122 ceph-mon[93869]: pgmap v11958: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 405 KiB/s wr, 11 op/s 2023-03-09T20:55:53.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:55:53.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:55:53.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:53 smithi181 ceph-mon[94189]: pgmap v11958: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 405 KiB/s wr, 11 op/s 2023-03-09T20:55:54.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:55:54.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:55:54.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:53 smithi038 ceph-mon[92422]: pgmap v11958: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 405 KiB/s wr, 11 op/s 2023-03-09T20:55:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21466.292275 secs 2023-03-09T20:55:55.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:55 smithi122 ceph-mon[93869]: pgmap v11959: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 402 KiB/s rd, 387 KiB/s wr, 10 op/s 2023-03-09T20:55:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21466.292275 secs 2023-03-09T20:55:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:55 smithi181 ceph-mon[94189]: pgmap v11959: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 402 KiB/s rd, 387 KiB/s wr, 10 op/s 2023-03-09T20:55:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21466.292275 secs 2023-03-09T20:55:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:55 smithi038 ceph-mon[92422]: pgmap v11959: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 402 KiB/s rd, 387 KiB/s wr, 10 op/s 2023-03-09T20:55:58.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:58 smithi122 ceph-mon[93869]: pgmap v11960: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 365 KiB/s wr, 12 op/s 2023-03-09T20:55:58.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:55:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21468.613292 secs 2023-03-09T20:55:58.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:58 smithi181 ceph-mon[94189]: pgmap v11960: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 365 KiB/s wr, 12 op/s 2023-03-09T20:55:58.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:55:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21468.613292 secs 2023-03-09T20:55:58.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:58 smithi038 ceph-mon[92422]: pgmap v11960: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 365 KiB/s wr, 12 op/s 2023-03-09T20:55:58.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:55:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21468.613292 secs 2023-03-09T20:55:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21930.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:56:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:00 smithi122 ceph-mon[93869]: pgmap v11961: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 676 KiB/s rd, 664 KiB/s wr, 11 op/s 2023-03-09T20:56:00.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:56:00.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:56:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:00 smithi181 ceph-mon[94189]: pgmap v11961: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 676 KiB/s rd, 664 KiB/s wr, 11 op/s 2023-03-09T20:56:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:56:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:56:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:00 smithi038 ceph-mon[92422]: pgmap v11961: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 676 KiB/s rd, 664 KiB/s wr, 11 op/s 2023-03-09T20:56:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:56:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:56:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21471.292353 secs 2023-03-09T20:56:01.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:56:01.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:56:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21471.292353 secs 2023-03-09T20:56:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:56:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:56:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21471.292353 secs 2023-03-09T20:56:01.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:56:01.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:56:02.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:02 smithi122 ceph-mon[93869]: pgmap v11962: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 630 KiB/s wr, 14 op/s 2023-03-09T20:56:02.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:02 smithi181 ceph-mon[94189]: pgmap v11962: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 630 KiB/s wr, 14 op/s 2023-03-09T20:56:02.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:02 smithi038 ceph-mon[92422]: pgmap v11962: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 630 KiB/s wr, 14 op/s 2023-03-09T20:56:03.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21473.613411 secs 2023-03-09T20:56:03.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21473.613411 secs 2023-03-09T20:56:03.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21473.613411 secs 2023-03-09T20:56:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21935.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:56:04.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:04 smithi122 ceph-mon[93869]: pgmap v11963: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 432 KiB/s wr, 24 op/s 2023-03-09T20:56:04.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:04 smithi181 ceph-mon[94189]: pgmap v11963: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 432 KiB/s wr, 24 op/s 2023-03-09T20:56:04.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:04 smithi038 ceph-mon[92422]: pgmap v11963: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 432 KiB/s wr, 24 op/s 2023-03-09T20:56:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21476.292411 secs 2023-03-09T20:56:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:05 smithi122 ceph-mon[93869]: pgmap v11964: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 466 KiB/s rd, 494 KiB/s wr, 23 op/s 2023-03-09T20:56:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21476.292411 secs 2023-03-09T20:56:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:05 smithi181 ceph-mon[94189]: pgmap v11964: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 466 KiB/s rd, 494 KiB/s wr, 23 op/s 2023-03-09T20:56:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21476.292411 secs 2023-03-09T20:56:06.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:05 smithi038 ceph-mon[92422]: pgmap v11964: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 466 KiB/s rd, 494 KiB/s wr, 23 op/s 2023-03-09T20:56:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:08 smithi122 ceph-mon[93869]: pgmap v11965: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 467 KiB/s rd, 500 KiB/s wr, 26 op/s 2023-03-09T20:56:08.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21478.613540 secs 2023-03-09T20:56:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:08 smithi181 ceph-mon[94189]: pgmap v11965: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 467 KiB/s rd, 500 KiB/s wr, 26 op/s 2023-03-09T20:56:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21478.613540 secs 2023-03-09T20:56:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:08 smithi038 ceph-mon[92422]: pgmap v11965: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 467 KiB/s rd, 500 KiB/s wr, 26 op/s 2023-03-09T20:56:08.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21478.613540 secs 2023-03-09T20:56:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21940.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:56:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:10 smithi122 ceph-mon[93869]: pgmap v11966: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 690 KiB/s wr, 25 op/s 2023-03-09T20:56:10.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:10 smithi122 ceph-mon[93869]: osdmap e961: 12 total, 12 up, 12 in 2023-03-09T20:56:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:10 smithi181 ceph-mon[94189]: pgmap v11966: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 690 KiB/s wr, 25 op/s 2023-03-09T20:56:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:10 smithi181 ceph-mon[94189]: osdmap e961: 12 total, 12 up, 12 in 2023-03-09T20:56:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:10 smithi038 ceph-mon[92422]: pgmap v11966: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 690 KiB/s wr, 25 op/s 2023-03-09T20:56:10.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:10 smithi038 ceph-mon[92422]: osdmap e961: 12 total, 12 up, 12 in 2023-03-09T20:56:11.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21481.292470 secs 2023-03-09T20:56:11.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21481.292470 secs 2023-03-09T20:56:11.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21481.292470 secs 2023-03-09T20:56:12.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:12 smithi122 ceph-mon[93869]: pgmap v11968: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 429 KiB/s wr, 26 op/s 2023-03-09T20:56:12.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:12 smithi181 ceph-mon[94189]: pgmap v11968: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 429 KiB/s wr, 26 op/s 2023-03-09T20:56:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:12 smithi038 ceph-mon[92422]: pgmap v11968: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 429 KiB/s wr, 26 op/s 2023-03-09T20:56:13.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21483.613657 secs 2023-03-09T20:56:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21483.613657 secs 2023-03-09T20:56:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21483.613657 secs 2023-03-09T20:56:13.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21945.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:56:14.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:14 smithi122 ceph-mon[93869]: pgmap v11969: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 382 KiB/s wr, 11 op/s 2023-03-09T20:56:14.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:14 smithi181 ceph-mon[94189]: pgmap v11969: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 382 KiB/s wr, 11 op/s 2023-03-09T20:56:14.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:14 smithi038 ceph-mon[92422]: pgmap v11969: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 382 KiB/s wr, 11 op/s 2023-03-09T20:56:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21486.292530 secs 2023-03-09T20:56:15.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:15 smithi122 ceph-mon[93869]: pgmap v11970: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 258 KiB/s wr, 11 op/s 2023-03-09T20:56:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21486.292530 secs 2023-03-09T20:56:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:15 smithi181 ceph-mon[94189]: pgmap v11970: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 258 KiB/s wr, 11 op/s 2023-03-09T20:56:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21486.292530 secs 2023-03-09T20:56:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:15 smithi038 ceph-mon[92422]: pgmap v11970: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 258 KiB/s wr, 11 op/s 2023-03-09T20:56:17.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:56:17.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:56:17.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:17 smithi122 ceph-mon[93869]: pgmap v11971: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 273 KiB/s wr, 10 op/s 2023-03-09T20:56:17.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21488.613793 secs 2023-03-09T20:56:17.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:56:17.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:56:17.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:17 smithi181 ceph-mon[94189]: pgmap v11971: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 273 KiB/s wr, 10 op/s 2023-03-09T20:56:17.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21488.613793 secs 2023-03-09T20:56:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:56:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:56:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:17 smithi038 ceph-mon[92422]: pgmap v11971: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 273 KiB/s wr, 10 op/s 2023-03-09T20:56:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21488.613793 secs 2023-03-09T20:56:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21950.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:56:19.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:19 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:56:19.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:19 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:56:19.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:19 smithi122 ceph-mon[93869]: pgmap v11972: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 113 KiB/s wr, 9 op/s 2023-03-09T20:56:19.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:19 smithi122 ceph-mon[93869]: osdmap e962: 12 total, 12 up, 12 in 2023-03-09T20:56:19.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:19 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:56:19.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:19 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:56:19.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:19 smithi181 ceph-mon[94189]: pgmap v11972: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 113 KiB/s wr, 9 op/s 2023-03-09T20:56:19.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:19 smithi181 ceph-mon[94189]: osdmap e962: 12 total, 12 up, 12 in 2023-03-09T20:56:20.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:19 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:56:20.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:19 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:56:20.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:19 smithi038 ceph-mon[92422]: pgmap v11972: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 113 KiB/s wr, 9 op/s 2023-03-09T20:56:20.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:19 smithi038 ceph-mon[92422]: osdmap e962: 12 total, 12 up, 12 in 2023-03-09T20:56:20.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21491.292591 secs 2023-03-09T20:56:20.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21491.292591 secs 2023-03-09T20:56:21.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21491.292591 secs 2023-03-09T20:56:21.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:21 smithi122 ceph-mon[93869]: pgmap v11974: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 113 KiB/s wr, 9 op/s 2023-03-09T20:56:21.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:21 smithi181 ceph-mon[94189]: pgmap v11974: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 113 KiB/s wr, 9 op/s 2023-03-09T20:56:22.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:21 smithi038 ceph-mon[92422]: pgmap v11974: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 113 KiB/s wr, 9 op/s 2023-03-09T20:56:22.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21493.613922 secs 2023-03-09T20:56:22.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21493.613922 secs 2023-03-09T20:56:23.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21493.613922 secs 2023-03-09T20:56:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21955.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:56:23.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:23 smithi122 ceph-mon[93869]: pgmap v11975: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 193 KiB/s wr, 9 op/s 2023-03-09T20:56:23.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:23 smithi181 ceph-mon[94189]: pgmap v11975: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 193 KiB/s wr, 9 op/s 2023-03-09T20:56:24.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:23 smithi038 ceph-mon[92422]: pgmap v11975: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 193 KiB/s wr, 9 op/s 2023-03-09T20:56:25.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21496.292653 secs 2023-03-09T20:56:25.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:25 smithi122 ceph-mon[93869]: pgmap v11976: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 193 KiB/s wr, 8 op/s 2023-03-09T20:56:25.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21496.292653 secs 2023-03-09T20:56:25.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:25 smithi181 ceph-mon[94189]: pgmap v11976: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 193 KiB/s wr, 8 op/s 2023-03-09T20:56:26.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21496.292653 secs 2023-03-09T20:56:26.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:25 smithi038 ceph-mon[92422]: pgmap v11976: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 193 KiB/s wr, 8 op/s 2023-03-09T20:56:28.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:28 smithi122 ceph-mon[93869]: pgmap v11977: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 172 KiB/s wr, 8 op/s 2023-03-09T20:56:28.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21498.613990 secs 2023-03-09T20:56:28.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:28 smithi181 ceph-mon[94189]: pgmap v11977: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 172 KiB/s wr, 8 op/s 2023-03-09T20:56:28.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21498.613990 secs 2023-03-09T20:56:28.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:28 smithi038 ceph-mon[92422]: pgmap v11977: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 172 KiB/s wr, 8 op/s 2023-03-09T20:56:28.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21498.613990 secs 2023-03-09T20:56:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21960.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:56:30.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:30 smithi122 ceph-mon[93869]: pgmap v11978: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 157 KiB/s wr, 8 op/s 2023-03-09T20:56:30.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:30 smithi181 ceph-mon[94189]: pgmap v11978: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 157 KiB/s wr, 8 op/s 2023-03-09T20:56:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:30 smithi038 ceph-mon[92422]: pgmap v11978: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 157 KiB/s wr, 8 op/s 2023-03-09T20:56:31.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21501.292714 secs 2023-03-09T20:56:31.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21501.292714 secs 2023-03-09T20:56:31.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21501.292714 secs 2023-03-09T20:56:32.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:32 smithi122 ceph-mon[93869]: pgmap v11979: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 137 KiB/s wr, 8 op/s 2023-03-09T20:56:32.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:56:32.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:32 smithi181 ceph-mon[94189]: pgmap v11979: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 137 KiB/s wr, 8 op/s 2023-03-09T20:56:32.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:56:32.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:32 smithi038 ceph-mon[92422]: pgmap v11979: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 137 KiB/s wr, 8 op/s 2023-03-09T20:56:32.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:56:33.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21503.614140 secs 2023-03-09T20:56:33.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21503.614140 secs 2023-03-09T20:56:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21503.614140 secs 2023-03-09T20:56:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21965.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:56:34.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:34 smithi122 ceph-mon[93869]: pgmap v11980: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 295 KiB/s wr, 8 op/s 2023-03-09T20:56:34.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:34 smithi181 ceph-mon[94189]: pgmap v11980: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 295 KiB/s wr, 8 op/s 2023-03-09T20:56:34.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:34 smithi038 ceph-mon[92422]: pgmap v11980: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 295 KiB/s wr, 8 op/s 2023-03-09T20:56:35.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:56:35.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:56:35.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:56:35.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:56:35.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:56:35.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:56:35.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:56:35.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:56:35.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:56:35.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:56:35.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:56:35.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:56:35.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:56:35.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:56:35.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:56:35.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:56:35.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:56:35.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:56:35.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:56:35.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:56:35.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:56:35.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:56:35.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:56:35.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:56:35.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:56:35.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:56:35.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:56:35.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:56:35.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:56:35.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:56:35.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:56:35.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:56:35.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:56:35.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:56:35.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:56:35.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:56:35.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:56:35.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:56:35.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:56:35.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:56:35.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:56:35.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:56:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:36 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T20:56:36.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:36 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T20:56:36.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:36 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi038 to 75150k 2023-03-09T20:56:36.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:36 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi038 to 76954321: error parsing value: Value '76954321' is below minimum 939524096 2023-03-09T20:56:36.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21506.292798 secs 2023-03-09T20:56:36.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:36 smithi122 ceph-mon[93869]: pgmap v11981: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 211 KiB/s wr, 7 op/s 2023-03-09T20:56:36.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:56:36.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:56:36.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:56:36.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:56:36.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:56:36.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:56:36.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:56:36.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:36 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T20:56:36.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:36 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T20:56:36.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:36 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi038 to 75150k 2023-03-09T20:56:36.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:36 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi038 to 76954321: error parsing value: Value '76954321' is below minimum 939524096 2023-03-09T20:56:36.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21506.292798 secs 2023-03-09T20:56:36.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:36 smithi181 ceph-mon[94189]: pgmap v11981: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 211 KiB/s wr, 7 op/s 2023-03-09T20:56:36.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:56:36.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:56:36.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:56:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:56:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:56:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:56:36.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:56:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:36 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T20:56:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:36 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T20:56:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:36 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi038 to 75150k 2023-03-09T20:56:36.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:36 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi038 to 76954321: error parsing value: Value '76954321' is below minimum 939524096 2023-03-09T20:56:36.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21506.292798 secs 2023-03-09T20:56:36.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:36 smithi038 ceph-mon[92422]: pgmap v11981: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 211 KiB/s wr, 7 op/s 2023-03-09T20:56:36.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:56:36.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:56:36.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:56:36.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T20:56:36.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:56:36.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:56:36.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:56:37.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:37 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T20:56:37.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:37 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T20:56:37.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:37 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T20:56:37.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:37 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T20:56:37.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:37 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T20:56:37.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:37 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T20:56:38.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:38 smithi122 ceph-mon[93869]: pgmap v11982: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 211 KiB/s wr, 9 op/s 2023-03-09T20:56:38.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21508.614269 secs 2023-03-09T20:56:38.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:38 smithi181 ceph-mon[94189]: pgmap v11982: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 211 KiB/s wr, 9 op/s 2023-03-09T20:56:38.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21508.614269 secs 2023-03-09T20:56:38.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:38 smithi038 ceph-mon[92422]: pgmap v11982: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 211 KiB/s wr, 9 op/s 2023-03-09T20:56:38.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21508.614269 secs 2023-03-09T20:56:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21970.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:56:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:40 smithi122 ceph-mon[93869]: pgmap v11983: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 210 KiB/s wr, 7 op/s 2023-03-09T20:56:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:40 smithi181 ceph-mon[94189]: pgmap v11983: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 210 KiB/s wr, 7 op/s 2023-03-09T20:56:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:40 smithi038 ceph-mon[92422]: pgmap v11983: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 210 KiB/s wr, 7 op/s 2023-03-09T20:56:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21511.292858 secs 2023-03-09T20:56:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:56:41.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:56:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21511.292858 secs 2023-03-09T20:56:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:56:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:56:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21511.292858 secs 2023-03-09T20:56:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:56:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:56:42.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:42 smithi122 ceph-mon[93869]: pgmap v11984: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 165 KiB/s wr, 8 op/s 2023-03-09T20:56:42.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:42 smithi181 ceph-mon[94189]: pgmap v11984: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 165 KiB/s wr, 8 op/s 2023-03-09T20:56:42.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:42 smithi038 ceph-mon[92422]: pgmap v11984: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 165 KiB/s wr, 8 op/s 2023-03-09T20:56:43.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21513.614393 secs 2023-03-09T20:56:43.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21513.614393 secs 2023-03-09T20:56:43.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21513.614393 secs 2023-03-09T20:56:43.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21975.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:56:44.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:44 smithi122 ceph-mon[93869]: pgmap v11985: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 257 KiB/s wr, 8 op/s 2023-03-09T20:56:44.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:44 smithi181 ceph-mon[94189]: pgmap v11985: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 257 KiB/s wr, 8 op/s 2023-03-09T20:56:44.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:44 smithi038 ceph-mon[92422]: pgmap v11985: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 257 KiB/s wr, 8 op/s 2023-03-09T20:56:45.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21516.292941 secs 2023-03-09T20:56:45.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:45 smithi122 ceph-mon[93869]: pgmap v11986: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 92 KiB/s wr, 7 op/s 2023-03-09T20:56:45.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21516.292941 secs 2023-03-09T20:56:45.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:45 smithi181 ceph-mon[94189]: pgmap v11986: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 92 KiB/s wr, 7 op/s 2023-03-09T20:56:46.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21516.292941 secs 2023-03-09T20:56:46.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:45 smithi038 ceph-mon[92422]: pgmap v11986: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 92 KiB/s wr, 7 op/s 2023-03-09T20:56:47.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:56:47.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:56:47.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:47 smithi122 ceph-mon[93869]: pgmap v11987: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 182 KiB/s wr, 10 op/s 2023-03-09T20:56:47.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21518.614505 secs 2023-03-09T20:56:47.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:56:47.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:56:47.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:47 smithi181 ceph-mon[94189]: pgmap v11987: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 182 KiB/s wr, 10 op/s 2023-03-09T20:56:47.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21518.614505 secs 2023-03-09T20:56:48.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:56:48.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:56:48.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:47 smithi038 ceph-mon[92422]: pgmap v11987: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 182 KiB/s wr, 10 op/s 2023-03-09T20:56:48.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21518.614505 secs 2023-03-09T20:56:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21980.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:56:50.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:50 smithi122 ceph-mon[93869]: pgmap v11988: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 435 KiB/s rd, 422 KiB/s wr, 9 op/s 2023-03-09T20:56:50.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:50 smithi181 ceph-mon[94189]: pgmap v11988: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 435 KiB/s rd, 422 KiB/s wr, 9 op/s 2023-03-09T20:56:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:50 smithi038 ceph-mon[92422]: pgmap v11988: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 435 KiB/s rd, 422 KiB/s wr, 9 op/s 2023-03-09T20:56:51.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21521.292989 secs 2023-03-09T20:56:51.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21521.292989 secs 2023-03-09T20:56:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21521.292989 secs 2023-03-09T20:56:52.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:52 smithi122 ceph-mon[93869]: pgmap v11989: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 447 KiB/s wr, 11 op/s 2023-03-09T20:56:52.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:56:52.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:56:52.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:52 smithi181 ceph-mon[94189]: pgmap v11989: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 447 KiB/s wr, 11 op/s 2023-03-09T20:56:52.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:56:52.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:56:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:52 smithi038 ceph-mon[92422]: pgmap v11989: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 447 KiB/s wr, 11 op/s 2023-03-09T20:56:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:56:52.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:56:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:56:53.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:56:53.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21523.614644 secs 2023-03-09T20:56:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:56:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:56:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21523.614644 secs 2023-03-09T20:56:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:56:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:56:53.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21523.614644 secs 2023-03-09T20:56:53.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21985.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:56:54.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:54 smithi122 ceph-mon[93869]: pgmap v11990: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 447 KiB/s wr, 10 op/s 2023-03-09T20:56:54.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:54 smithi181 ceph-mon[94189]: pgmap v11990: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 447 KiB/s wr, 10 op/s 2023-03-09T20:56:54.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:54 smithi038 ceph-mon[92422]: pgmap v11990: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 447 KiB/s wr, 10 op/s 2023-03-09T20:56:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21526.293051 secs 2023-03-09T20:56:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:55 smithi122 ceph-mon[93869]: pgmap v11991: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 400 KiB/s wr, 9 op/s 2023-03-09T20:56:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21526.293051 secs 2023-03-09T20:56:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:55 smithi181 ceph-mon[94189]: pgmap v11991: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 400 KiB/s wr, 9 op/s 2023-03-09T20:56:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21526.293051 secs 2023-03-09T20:56:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:55 smithi038 ceph-mon[92422]: pgmap v11991: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 400 KiB/s wr, 9 op/s 2023-03-09T20:56:58.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:56:58 smithi122 ceph-mon[93869]: pgmap v11992: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 652 KiB/s rd, 683 KiB/s wr, 13 op/s 2023-03-09T20:56:58.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:56:58 smithi181 ceph-mon[94189]: pgmap v11992: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 652 KiB/s rd, 683 KiB/s wr, 13 op/s 2023-03-09T20:56:58.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:56:58 smithi038 ceph-mon[92422]: pgmap v11992: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 652 KiB/s rd, 683 KiB/s wr, 13 op/s 2023-03-09T20:56:58.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21990.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:57:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:00 smithi122 ceph-mon[93869]: pgmap v11993: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 560 KiB/s rd, 593 KiB/s wr, 10 op/s 2023-03-09T20:57:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:00 smithi181 ceph-mon[94189]: pgmap v11993: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 560 KiB/s rd, 593 KiB/s wr, 10 op/s 2023-03-09T20:57:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:00 smithi038 ceph-mon[92422]: pgmap v11993: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 560 KiB/s rd, 593 KiB/s wr, 10 op/s 2023-03-09T20:57:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21531.293112 secs 2023-03-09T20:57:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:01 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:57:01.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:01 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:57:01.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:57:01.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:57:01.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:57:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21531.293112 secs 2023-03-09T20:57:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:01 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:57:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:01 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:57:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:57:01.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:57:01.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:57:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21531.293112 secs 2023-03-09T20:57:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:01 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:57:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:01 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:57:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:57:01.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:57:01.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:57:02.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:02 smithi122 ceph-mon[93869]: pgmap v11994: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 470 KiB/s rd, 457 KiB/s wr, 12 op/s 2023-03-09T20:57:02.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:02 smithi181 ceph-mon[94189]: pgmap v11994: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 470 KiB/s rd, 457 KiB/s wr, 12 op/s 2023-03-09T20:57:02.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:02 smithi038 ceph-mon[92422]: pgmap v11994: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 470 KiB/s rd, 457 KiB/s wr, 12 op/s 2023-03-09T20:57:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 21995.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:57:04.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:04 smithi122 ceph-mon[93869]: pgmap v11995: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 464 KiB/s rd, 541 KiB/s wr, 23 op/s 2023-03-09T20:57:04.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:04 smithi181 ceph-mon[94189]: pgmap v11995: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 464 KiB/s rd, 541 KiB/s wr, 23 op/s 2023-03-09T20:57:04.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:04 smithi038 ceph-mon[92422]: pgmap v11995: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 464 KiB/s rd, 541 KiB/s wr, 23 op/s 2023-03-09T20:57:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21536.293172 secs 2023-03-09T20:57:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:05 smithi122 ceph-mon[93869]: pgmap v11996: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 734 KiB/s wr, 23 op/s 2023-03-09T20:57:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21536.293172 secs 2023-03-09T20:57:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:05 smithi181 ceph-mon[94189]: pgmap v11996: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 734 KiB/s wr, 23 op/s 2023-03-09T20:57:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21536.293172 secs 2023-03-09T20:57:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:05 smithi038 ceph-mon[92422]: pgmap v11996: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 734 KiB/s wr, 23 op/s 2023-03-09T20:57:07.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:57:07.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:57:07.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:07 smithi122 ceph-mon[93869]: pgmap v11997: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 672 KiB/s rd, 724 KiB/s wr, 28 op/s 2023-03-09T20:57:07.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21538.614981 secs 2023-03-09T20:57:07.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:57:07.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:57:07.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:07 smithi181 ceph-mon[94189]: pgmap v11997: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 672 KiB/s rd, 724 KiB/s wr, 28 op/s 2023-03-09T20:57:07.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21538.614981 secs 2023-03-09T20:57:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:57:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:57:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:07 smithi038 ceph-mon[92422]: pgmap v11997: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 672 KiB/s rd, 724 KiB/s wr, 28 op/s 2023-03-09T20:57:08.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21538.614981 secs 2023-03-09T20:57:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22000.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:57:08.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:08 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:57:08.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:57:08.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:57:08.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:57:08.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:08 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:57:08.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:57:08.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:57:08.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:57:09.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:08 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:57:09.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:57:09.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:57:09.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:57:09.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:09 smithi122 ceph-mon[93869]: pgmap v11998: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 543 KiB/s rd, 617 KiB/s wr, 25 op/s 2023-03-09T20:57:09.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:09 smithi181 ceph-mon[94189]: pgmap v11998: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 543 KiB/s rd, 617 KiB/s wr, 25 op/s 2023-03-09T20:57:10.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:09 smithi038 ceph-mon[92422]: pgmap v11998: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 543 KiB/s rd, 617 KiB/s wr, 25 op/s 2023-03-09T20:57:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:10 smithi122 ceph-mon[93869]: osdmap e963: 12 total, 12 up, 12 in 2023-03-09T20:57:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21541.293235 secs 2023-03-09T20:57:10.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:10 smithi181 ceph-mon[94189]: osdmap e963: 12 total, 12 up, 12 in 2023-03-09T20:57:10.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21541.293235 secs 2023-03-09T20:57:11.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:10 smithi038 ceph-mon[92422]: osdmap e963: 12 total, 12 up, 12 in 2023-03-09T20:57:11.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21541.293235 secs 2023-03-09T20:57:11.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:11 smithi122 ceph-mon[93869]: pgmap v12000: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 474 KiB/s rd, 616 KiB/s wr, 29 op/s 2023-03-09T20:57:11.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:11 smithi181 ceph-mon[94189]: pgmap v12000: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 474 KiB/s rd, 616 KiB/s wr, 29 op/s 2023-03-09T20:57:12.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:11 smithi038 ceph-mon[92422]: pgmap v12000: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 474 KiB/s rd, 616 KiB/s wr, 29 op/s 2023-03-09T20:57:12.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21543.615091 secs 2023-03-09T20:57:12.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21543.615091 secs 2023-03-09T20:57:13.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21543.615091 secs 2023-03-09T20:57:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22005.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:57:13.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:13 smithi122 ceph-mon[93869]: pgmap v12001: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 621 KiB/s rd, 670 KiB/s wr, 14 op/s 2023-03-09T20:57:13.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:13 smithi181 ceph-mon[94189]: pgmap v12001: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 621 KiB/s rd, 670 KiB/s wr, 14 op/s 2023-03-09T20:57:14.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:13 smithi038 ceph-mon[92422]: pgmap v12001: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 621 KiB/s rd, 670 KiB/s wr, 14 op/s 2023-03-09T20:57:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21546.293294 secs 2023-03-09T20:57:15.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:15 smithi122 ceph-mon[93869]: pgmap v12002: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 438 KiB/s wr, 13 op/s 2023-03-09T20:57:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21546.293294 secs 2023-03-09T20:57:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:15 smithi181 ceph-mon[94189]: pgmap v12002: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 438 KiB/s wr, 13 op/s 2023-03-09T20:57:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21546.293294 secs 2023-03-09T20:57:16.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:15 smithi038 ceph-mon[92422]: pgmap v12002: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 438 KiB/s wr, 13 op/s 2023-03-09T20:57:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:18 smithi122 ceph-mon[93869]: pgmap v12003: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 396 KiB/s wr, 9 op/s 2023-03-09T20:57:18.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21548.615122 secs 2023-03-09T20:57:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:18 smithi181 ceph-mon[94189]: pgmap v12003: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 396 KiB/s wr, 9 op/s 2023-03-09T20:57:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21548.615122 secs 2023-03-09T20:57:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:18 smithi038 ceph-mon[92422]: pgmap v12003: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 396 KiB/s wr, 9 op/s 2023-03-09T20:57:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21548.615122 secs 2023-03-09T20:57:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22010.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:57:20.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:20 smithi122 ceph-mon[93869]: pgmap v12004: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 240 KiB/s wr, 9 op/s 2023-03-09T20:57:20.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:57:20.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:57:20.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:20 smithi122 ceph-mon[93869]: osdmap e964: 12 total, 12 up, 12 in 2023-03-09T20:57:20.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:20 smithi181 ceph-mon[94189]: pgmap v12004: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 240 KiB/s wr, 9 op/s 2023-03-09T20:57:20.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:57:20.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:57:20.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:20 smithi181 ceph-mon[94189]: osdmap e964: 12 total, 12 up, 12 in 2023-03-09T20:57:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:20 smithi038 ceph-mon[92422]: pgmap v12004: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 240 KiB/s wr, 9 op/s 2023-03-09T20:57:20.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:57:20.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:57:20.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:20 smithi038 ceph-mon[92422]: osdmap e964: 12 total, 12 up, 12 in 2023-03-09T20:57:21.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21551.293354 secs 2023-03-09T20:57:21.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:57:21.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:57:21.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21551.293354 secs 2023-03-09T20:57:21.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:57:21.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:57:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21551.293354 secs 2023-03-09T20:57:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:57:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:57:22.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:22 smithi122 ceph-mon[93869]: pgmap v12006: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 240 KiB/s wr, 9 op/s 2023-03-09T20:57:22.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:22 smithi181 ceph-mon[94189]: pgmap v12006: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 240 KiB/s wr, 9 op/s 2023-03-09T20:57:22.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:22 smithi038 ceph-mon[92422]: pgmap v12006: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 240 KiB/s wr, 9 op/s 2023-03-09T20:57:23.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21553.615239 secs 2023-03-09T20:57:23.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21553.615239 secs 2023-03-09T20:57:23.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21553.615239 secs 2023-03-09T20:57:23.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22015.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:57:24.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:24 smithi122 ceph-mon[93869]: pgmap v12007: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 56 KiB/s wr, 9 op/s 2023-03-09T20:57:24.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:24 smithi181 ceph-mon[94189]: pgmap v12007: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 56 KiB/s wr, 9 op/s 2023-03-09T20:57:24.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:24 smithi038 ceph-mon[92422]: pgmap v12007: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 56 KiB/s wr, 9 op/s 2023-03-09T20:57:25.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21556.293414 secs 2023-03-09T20:57:25.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:25 smithi122 ceph-mon[93869]: pgmap v12008: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 56 KiB/s wr, 9 op/s 2023-03-09T20:57:25.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21556.293414 secs 2023-03-09T20:57:25.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:25 smithi181 ceph-mon[94189]: pgmap v12008: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 56 KiB/s wr, 9 op/s 2023-03-09T20:57:26.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21556.293414 secs 2023-03-09T20:57:26.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:25 smithi038 ceph-mon[92422]: pgmap v12008: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 56 KiB/s wr, 9 op/s 2023-03-09T20:57:28.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:28 smithi122 ceph-mon[93869]: pgmap v12009: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 56 KiB/s wr, 9 op/s 2023-03-09T20:57:28.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21558.615374 secs 2023-03-09T20:57:28.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:28 smithi181 ceph-mon[94189]: pgmap v12009: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 56 KiB/s wr, 9 op/s 2023-03-09T20:57:28.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21558.615374 secs 2023-03-09T20:57:28.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:28 smithi038 ceph-mon[92422]: pgmap v12009: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 56 KiB/s wr, 9 op/s 2023-03-09T20:57:28.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21558.615374 secs 2023-03-09T20:57:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22020.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:57:30.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:30 smithi122 ceph-mon[93869]: pgmap v12010: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 102 KiB/s wr, 9 op/s 2023-03-09T20:57:30.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:30 smithi181 ceph-mon[94189]: pgmap v12010: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 102 KiB/s wr, 9 op/s 2023-03-09T20:57:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:30 smithi038 ceph-mon[92422]: pgmap v12010: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 102 KiB/s wr, 9 op/s 2023-03-09T20:57:31.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21561.293497 secs 2023-03-09T20:57:31.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21561.293497 secs 2023-03-09T20:57:31.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21561.293497 secs 2023-03-09T20:57:32.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:32 smithi122 ceph-mon[93869]: pgmap v12011: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 89 KiB/s wr, 9 op/s 2023-03-09T20:57:32.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:32 smithi181 ceph-mon[94189]: pgmap v12011: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 89 KiB/s wr, 9 op/s 2023-03-09T20:57:32.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:32 smithi038 ceph-mon[92422]: pgmap v12011: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 89 KiB/s wr, 9 op/s 2023-03-09T20:57:33.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21563.615485 secs 2023-03-09T20:57:33.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21563.615485 secs 2023-03-09T20:57:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21563.615485 secs 2023-03-09T20:57:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22025.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:57:34.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:34 smithi122 ceph-mon[93869]: pgmap v12012: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 85 KiB/s wr, 8 op/s 2023-03-09T20:57:34.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:34 smithi181 ceph-mon[94189]: pgmap v12012: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 85 KiB/s wr, 8 op/s 2023-03-09T20:57:34.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:34 smithi038 ceph-mon[92422]: pgmap v12012: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 85 KiB/s wr, 8 op/s 2023-03-09T20:57:35.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21566.293567 secs 2023-03-09T20:57:35.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:35 smithi122 ceph-mon[93869]: pgmap v12013: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 85 KiB/s wr, 7 op/s 2023-03-09T20:57:35.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21566.293567 secs 2023-03-09T20:57:35.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:35 smithi181 ceph-mon[94189]: pgmap v12013: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 85 KiB/s wr, 7 op/s 2023-03-09T20:57:36.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21566.293567 secs 2023-03-09T20:57:36.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:35 smithi038 ceph-mon[92422]: pgmap v12013: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 85 KiB/s wr, 7 op/s 2023-03-09T20:57:36.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:57:36.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:57:36.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:57:36.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:57:36.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:57:36.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:57:36.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:57:36.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:57:37.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:57:37.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:57:37.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:57:37.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:57:37.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:37 smithi122 ceph-mon[93869]: pgmap v12014: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 85 KiB/s wr, 8 op/s 2023-03-09T20:57:37.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:37 smithi181 ceph-mon[94189]: pgmap v12014: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 85 KiB/s wr, 8 op/s 2023-03-09T20:57:38.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:37 smithi038 ceph-mon[92422]: pgmap v12014: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 85 KiB/s wr, 8 op/s 2023-03-09T20:57:38.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22030.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:57:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:40 smithi122 ceph-mon[93869]: pgmap v12015: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 245 KiB/s wr, 7 op/s 2023-03-09T20:57:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:40 smithi181 ceph-mon[94189]: pgmap v12015: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 245 KiB/s wr, 7 op/s 2023-03-09T20:57:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:40 smithi038 ceph-mon[92422]: pgmap v12015: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 245 KiB/s wr, 7 op/s 2023-03-09T20:57:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21571.293626 secs 2023-03-09T20:57:41.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:41 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:57:41.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:41 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:57:41.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:57:41.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:57:41.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:57:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21571.293626 secs 2023-03-09T20:57:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:41 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:57:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:41 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:57:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:57:41.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:57:41.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:57:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21571.293626 secs 2023-03-09T20:57:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:41 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:57:41.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:41 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:57:41.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:57:41.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:57:41.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:57:42.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:42 smithi122 ceph-mon[93869]: pgmap v12016: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 200 KiB/s wr, 8 op/s 2023-03-09T20:57:42.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:42 smithi181 ceph-mon[94189]: pgmap v12016: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 200 KiB/s wr, 8 op/s 2023-03-09T20:57:42.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:42 smithi038 ceph-mon[92422]: pgmap v12016: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 200 KiB/s wr, 8 op/s 2023-03-09T20:57:43.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22035.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:57:44.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:44 smithi122 ceph-mon[93869]: pgmap v12017: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 218 KiB/s wr, 8 op/s 2023-03-09T20:57:44.490 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:44 smithi181 ceph-mon[94189]: pgmap v12017: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 218 KiB/s wr, 8 op/s 2023-03-09T20:57:44.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:44 smithi038 ceph-mon[92422]: pgmap v12017: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 218 KiB/s wr, 8 op/s 2023-03-09T20:57:45.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21576.293689 secs 2023-03-09T20:57:45.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:45 smithi122 ceph-mon[93869]: pgmap v12018: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 218 KiB/s wr, 7 op/s 2023-03-09T20:57:45.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21576.293689 secs 2023-03-09T20:57:45.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:45 smithi181 ceph-mon[94189]: pgmap v12018: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 218 KiB/s wr, 7 op/s 2023-03-09T20:57:46.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21576.293689 secs 2023-03-09T20:57:46.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:45 smithi038 ceph-mon[92422]: pgmap v12018: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 218 KiB/s wr, 7 op/s 2023-03-09T20:57:47.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:57:47.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:57:47.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:47 smithi122 ceph-mon[93869]: pgmap v12019: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 384 KiB/s rd, 379 KiB/s wr, 10 op/s 2023-03-09T20:57:47.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:57:47.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:57:47.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:47 smithi181 ceph-mon[94189]: pgmap v12019: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 384 KiB/s rd, 379 KiB/s wr, 10 op/s 2023-03-09T20:57:48.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:57:48.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:57:48.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:47 smithi038 ceph-mon[92422]: pgmap v12019: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 384 KiB/s rd, 379 KiB/s wr, 10 op/s 2023-03-09T20:57:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22040.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:57:50.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:50 smithi122 ceph-mon[93869]: pgmap v12020: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 397 KiB/s wr, 10 op/s 2023-03-09T20:57:50.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:50 smithi181 ceph-mon[94189]: pgmap v12020: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 397 KiB/s wr, 10 op/s 2023-03-09T20:57:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:50 smithi038 ceph-mon[92422]: pgmap v12020: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 397 KiB/s wr, 10 op/s 2023-03-09T20:57:51.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21581.293749 secs 2023-03-09T20:57:51.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21581.293749 secs 2023-03-09T20:57:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21581.293749 secs 2023-03-09T20:57:52.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:52 smithi122 ceph-mon[93869]: pgmap v12021: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 686 KiB/s rd, 511 KiB/s wr, 13 op/s 2023-03-09T20:57:52.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:52 smithi181 ceph-mon[94189]: pgmap v12021: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 686 KiB/s rd, 511 KiB/s wr, 13 op/s 2023-03-09T20:57:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:52 smithi038 ceph-mon[92422]: pgmap v12021: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 686 KiB/s rd, 511 KiB/s wr, 13 op/s 2023-03-09T20:57:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:57:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:57:53.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:53 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 21583.615993 secs 2023-03-09T20:57:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:57:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:57:53.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:53 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 21583.615993 secs 2023-03-09T20:57:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:57:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:57:53.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:53 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 21583.615993 secs 2023-03-09T20:57:53.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22045.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:57:54.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:54 smithi122 ceph-mon[93869]: pgmap v12022: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 665 KiB/s rd, 492 KiB/s wr, 11 op/s 2023-03-09T20:57:54.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:54 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:57:54.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:54 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:57:54.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:54 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:57:54.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:54 smithi181 ceph-mon[94189]: pgmap v12022: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 665 KiB/s rd, 492 KiB/s wr, 11 op/s 2023-03-09T20:57:54.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:54 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:57:54.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:54 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:57:54.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:54 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:57:54.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:54 smithi038 ceph-mon[92422]: pgmap v12022: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 665 KiB/s rd, 492 KiB/s wr, 11 op/s 2023-03-09T20:57:54.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:54 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:57:54.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:54 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:57:54.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:54 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:57:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21586.293811 secs 2023-03-09T20:57:56.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:56 smithi122 ceph-mon[93869]: pgmap v12023: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 474 KiB/s wr, 10 op/s 2023-03-09T20:57:56.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:57:56.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:57:56.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21586.293811 secs 2023-03-09T20:57:56.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:56 smithi181 ceph-mon[94189]: pgmap v12023: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 474 KiB/s wr, 10 op/s 2023-03-09T20:57:56.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:57:56.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:57:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21586.293811 secs 2023-03-09T20:57:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:56 smithi038 ceph-mon[92422]: pgmap v12023: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 474 KiB/s wr, 10 op/s 2023-03-09T20:57:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:57:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:57:58.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:58 smithi122 ceph-mon[93869]: pgmap v12024: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 593 KiB/s rd, 579 KiB/s wr, 14 op/s 2023-03-09T20:57:58.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:57:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21588.616058 secs 2023-03-09T20:57:58.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:58 smithi181 ceph-mon[94189]: pgmap v12024: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 593 KiB/s rd, 579 KiB/s wr, 14 op/s 2023-03-09T20:57:58.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:57:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21588.616058 secs 2023-03-09T20:57:58.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:58 smithi038 ceph-mon[92422]: pgmap v12024: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 593 KiB/s rd, 579 KiB/s wr, 14 op/s 2023-03-09T20:57:58.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:57:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21588.616058 secs 2023-03-09T20:57:58.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22050.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:58:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:00 smithi122 ceph-mon[93869]: pgmap v12025: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 454 KiB/s wr, 12 op/s 2023-03-09T20:58:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:00 smithi181 ceph-mon[94189]: pgmap v12025: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 454 KiB/s wr, 12 op/s 2023-03-09T20:58:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:00 smithi038 ceph-mon[92422]: pgmap v12025: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 454 KiB/s wr, 12 op/s 2023-03-09T20:58:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21591.293872 secs 2023-03-09T20:58:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21591.293872 secs 2023-03-09T20:58:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21591.293872 secs 2023-03-09T20:58:02.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:02 smithi122 ceph-mon[93869]: pgmap v12026: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 559 KiB/s wr, 15 op/s 2023-03-09T20:58:02.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:02 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:58:02.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:02 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:58:02.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:02 smithi181 ceph-mon[94189]: pgmap v12026: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 559 KiB/s wr, 15 op/s 2023-03-09T20:58:02.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:02 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:58:02.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:02 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:58:02.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:02 smithi038 ceph-mon[92422]: pgmap v12026: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 559 KiB/s wr, 15 op/s 2023-03-09T20:58:02.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:02 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:58:02.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:02 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:58:03.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21593.616158 secs 2023-03-09T20:58:03.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21593.616158 secs 2023-03-09T20:58:03.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21593.616158 secs 2023-03-09T20:58:03.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22055.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:58:04.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:04 smithi122 ceph-mon[93869]: pgmap v12027: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 308 KiB/s wr, 25 op/s 2023-03-09T20:58:04.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:04 smithi181 ceph-mon[94189]: pgmap v12027: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 308 KiB/s wr, 25 op/s 2023-03-09T20:58:04.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:04 smithi038 ceph-mon[92422]: pgmap v12027: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 308 KiB/s wr, 25 op/s 2023-03-09T20:58:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21596.293932 secs 2023-03-09T20:58:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:05 smithi122 ceph-mon[93869]: pgmap v12028: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 288 KiB/s wr, 24 op/s 2023-03-09T20:58:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21596.293932 secs 2023-03-09T20:58:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:05 smithi181 ceph-mon[94189]: pgmap v12028: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 288 KiB/s wr, 24 op/s 2023-03-09T20:58:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21596.293932 secs 2023-03-09T20:58:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:05 smithi038 ceph-mon[92422]: pgmap v12028: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 288 KiB/s wr, 24 op/s 2023-03-09T20:58:07.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:58:07.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:58:07.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:07 smithi122 ceph-mon[93869]: pgmap v12029: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 294 KiB/s wr, 26 op/s 2023-03-09T20:58:07.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21598.616293 secs 2023-03-09T20:58:07.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:58:07.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:58:07.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:07 smithi181 ceph-mon[94189]: pgmap v12029: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 294 KiB/s wr, 26 op/s 2023-03-09T20:58:07.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21598.616293 secs 2023-03-09T20:58:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:58:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:58:08.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:07 smithi038 ceph-mon[92422]: pgmap v12029: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 294 KiB/s wr, 26 op/s 2023-03-09T20:58:08.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21598.616293 secs 2023-03-09T20:58:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22060.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:58:09.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:58:09.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:58:09.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:09 smithi122 ceph-mon[93869]: pgmap v12030: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 270 KiB/s wr, 23 op/s 2023-03-09T20:58:09.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:58:09.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:58:09.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:09 smithi181 ceph-mon[94189]: pgmap v12030: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 270 KiB/s wr, 23 op/s 2023-03-09T20:58:10.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:58:10.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:58:10.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:09 smithi038 ceph-mon[92422]: pgmap v12030: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 270 KiB/s wr, 23 op/s 2023-03-09T20:58:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:10 smithi122 ceph-mon[93869]: osdmap e965: 12 total, 12 up, 12 in 2023-03-09T20:58:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21601.293985 secs 2023-03-09T20:58:10.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:10 smithi181 ceph-mon[94189]: osdmap e965: 12 total, 12 up, 12 in 2023-03-09T20:58:10.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21601.293985 secs 2023-03-09T20:58:11.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:10 smithi038 ceph-mon[92422]: osdmap e965: 12 total, 12 up, 12 in 2023-03-09T20:58:11.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21601.293985 secs 2023-03-09T20:58:11.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:11 smithi122 ceph-mon[93869]: pgmap v12032: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 132 KiB/s wr, 22 op/s 2023-03-09T20:58:11.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:11 smithi181 ceph-mon[94189]: pgmap v12032: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 132 KiB/s wr, 22 op/s 2023-03-09T20:58:12.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:11 smithi038 ceph-mon[92422]: pgmap v12032: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 132 KiB/s wr, 22 op/s 2023-03-09T20:58:12.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21603.616410 secs 2023-03-09T20:58:12.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21603.616410 secs 2023-03-09T20:58:13.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21603.616410 secs 2023-03-09T20:58:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22065.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:58:13.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:13 smithi122 ceph-mon[93869]: pgmap v12033: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 104 KiB/s wr, 7 op/s 2023-03-09T20:58:13.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:13 smithi181 ceph-mon[94189]: pgmap v12033: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 104 KiB/s wr, 7 op/s 2023-03-09T20:58:14.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:13 smithi038 ceph-mon[92422]: pgmap v12033: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 104 KiB/s wr, 7 op/s 2023-03-09T20:58:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21606.294045 secs 2023-03-09T20:58:15.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:15 smithi122 ceph-mon[93869]: pgmap v12034: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 104 KiB/s wr, 7 op/s 2023-03-09T20:58:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21606.294045 secs 2023-03-09T20:58:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:15 smithi181 ceph-mon[94189]: pgmap v12034: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 104 KiB/s wr, 7 op/s 2023-03-09T20:58:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21606.294045 secs 2023-03-09T20:58:16.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:15 smithi038 ceph-mon[92422]: pgmap v12034: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 104 KiB/s wr, 7 op/s 2023-03-09T20:58:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:18 smithi122 ceph-mon[93869]: pgmap v12035: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 97 KiB/s wr, 6 op/s 2023-03-09T20:58:18.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21608.616550 secs 2023-03-09T20:58:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:18 smithi181 ceph-mon[94189]: pgmap v12035: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 97 KiB/s wr, 6 op/s 2023-03-09T20:58:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21608.616550 secs 2023-03-09T20:58:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:18 smithi038 ceph-mon[92422]: pgmap v12035: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 97 KiB/s wr, 6 op/s 2023-03-09T20:58:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21608.616550 secs 2023-03-09T20:58:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22070.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:58:20.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:20 smithi122 ceph-mon[93869]: pgmap v12036: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T20:58:20.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:20 smithi122 ceph-mon[93869]: osdmap e966: 12 total, 12 up, 12 in 2023-03-09T20:58:20.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:20 smithi181 ceph-mon[94189]: pgmap v12036: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T20:58:20.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:20 smithi181 ceph-mon[94189]: osdmap e966: 12 total, 12 up, 12 in 2023-03-09T20:58:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:20 smithi038 ceph-mon[92422]: pgmap v12036: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T20:58:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:20 smithi038 ceph-mon[92422]: osdmap e966: 12 total, 12 up, 12 in 2023-03-09T20:58:21.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21611.294109 secs 2023-03-09T20:58:21.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21611.294109 secs 2023-03-09T20:58:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21611.294109 secs 2023-03-09T20:58:22.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:22 smithi122 ceph-mon[93869]: pgmap v12038: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T20:58:22.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:22 smithi181 ceph-mon[94189]: pgmap v12038: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T20:58:22.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:22 smithi038 ceph-mon[92422]: pgmap v12038: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T20:58:23.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21613.616676 secs 2023-03-09T20:58:23.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21613.616676 secs 2023-03-09T20:58:23.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21613.616676 secs 2023-03-09T20:58:23.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22075.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:58:24.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:24 smithi122 ceph-mon[93869]: pgmap v12039: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T20:58:24.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:24 smithi181 ceph-mon[94189]: pgmap v12039: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T20:58:24.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:24 smithi038 ceph-mon[92422]: pgmap v12039: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T20:58:25.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21616.294193 secs 2023-03-09T20:58:25.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:25 smithi122 ceph-mon[93869]: pgmap v12040: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T20:58:25.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21616.294193 secs 2023-03-09T20:58:25.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:25 smithi181 ceph-mon[94189]: pgmap v12040: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T20:58:26.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21616.294193 secs 2023-03-09T20:58:26.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:25 smithi038 ceph-mon[92422]: pgmap v12040: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T20:58:28.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:28 smithi122 ceph-mon[93869]: pgmap v12041: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T20:58:28.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21618.616812 secs 2023-03-09T20:58:28.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:28 smithi181 ceph-mon[94189]: pgmap v12041: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T20:58:28.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21618.616812 secs 2023-03-09T20:58:28.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:28 smithi038 ceph-mon[92422]: pgmap v12041: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T20:58:28.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21618.616812 secs 2023-03-09T20:58:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22080.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:58:30.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:30 smithi122 ceph-mon[93869]: pgmap v12042: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T20:58:30.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:30 smithi181 ceph-mon[94189]: pgmap v12042: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T20:58:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:30 smithi038 ceph-mon[92422]: pgmap v12042: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T20:58:31.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21621.294268 secs 2023-03-09T20:58:31.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21621.294268 secs 2023-03-09T20:58:31.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21621.294268 secs 2023-03-09T20:58:32.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:32 smithi122 ceph-mon[93869]: pgmap v12043: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 356 B/s wr, 7 op/s 2023-03-09T20:58:32.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:32 smithi181 ceph-mon[94189]: pgmap v12043: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 356 B/s wr, 7 op/s 2023-03-09T20:58:32.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:32 smithi038 ceph-mon[92422]: pgmap v12043: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 356 B/s wr, 7 op/s 2023-03-09T20:58:33.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21623.616956 secs 2023-03-09T20:58:33.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21623.616956 secs 2023-03-09T20:58:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21623.616956 secs 2023-03-09T20:58:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22085.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:58:34.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:34 smithi122 ceph-mon[93869]: pgmap v12044: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 70 KiB/s wr, 7 op/s 2023-03-09T20:58:34.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:34 smithi181 ceph-mon[94189]: pgmap v12044: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 70 KiB/s wr, 7 op/s 2023-03-09T20:58:34.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:34 smithi038 ceph-mon[92422]: pgmap v12044: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 70 KiB/s wr, 7 op/s 2023-03-09T20:58:35.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21626.294326 secs 2023-03-09T20:58:35.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:35 smithi122 ceph-mon[93869]: pgmap v12045: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 70 KiB/s wr, 6 op/s 2023-03-09T20:58:35.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21626.294326 secs 2023-03-09T20:58:35.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:35 smithi181 ceph-mon[94189]: pgmap v12045: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 70 KiB/s wr, 6 op/s 2023-03-09T20:58:36.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21626.294326 secs 2023-03-09T20:58:36.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:35 smithi038 ceph-mon[92422]: pgmap v12045: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 70 KiB/s wr, 6 op/s 2023-03-09T20:58:36.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:58:36.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:58:37.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:58:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:58:38.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:58:38.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:37 smithi122 ceph-mon[93869]: pgmap v12046: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 70 KiB/s wr, 8 op/s 2023-03-09T20:58:38.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21628.617041 secs 2023-03-09T20:58:38.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:58:38.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:58:38.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:58:38.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:58:38.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:58:38.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:37 smithi181 ceph-mon[94189]: pgmap v12046: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 70 KiB/s wr, 8 op/s 2023-03-09T20:58:38.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21628.617041 secs 2023-03-09T20:58:38.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:58:38.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:58:38.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:58:38.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:58:38.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:58:38.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:37 smithi038 ceph-mon[92422]: pgmap v12046: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 70 KiB/s wr, 8 op/s 2023-03-09T20:58:38.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21628.617041 secs 2023-03-09T20:58:38.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:58:38.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:58:38.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:58:38.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22090.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:58:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:40 smithi122 ceph-mon[93869]: pgmap v12047: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 327 KiB/s wr, 8 op/s 2023-03-09T20:58:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:40 smithi181 ceph-mon[94189]: pgmap v12047: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 327 KiB/s wr, 8 op/s 2023-03-09T20:58:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:40 smithi038 ceph-mon[92422]: pgmap v12047: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 327 KiB/s wr, 8 op/s 2023-03-09T20:58:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21631.294410 secs 2023-03-09T20:58:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:58:41.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:58:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21631.294410 secs 2023-03-09T20:58:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:58:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:58:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21631.294410 secs 2023-03-09T20:58:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:58:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:58:42.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:42 smithi122 ceph-mon[93869]: pgmap v12048: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 318 KiB/s rd, 327 KiB/s wr, 9 op/s 2023-03-09T20:58:42.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:42 smithi181 ceph-mon[94189]: pgmap v12048: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 318 KiB/s rd, 327 KiB/s wr, 9 op/s 2023-03-09T20:58:42.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:42 smithi038 ceph-mon[92422]: pgmap v12048: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 318 KiB/s rd, 327 KiB/s wr, 9 op/s 2023-03-09T20:58:43.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21633.617223 secs 2023-03-09T20:58:43.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21633.617223 secs 2023-03-09T20:58:43.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21633.617223 secs 2023-03-09T20:58:43.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22095.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:58:44.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:44 smithi122 ceph-mon[93869]: pgmap v12049: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 345 KiB/s wr, 8 op/s 2023-03-09T20:58:44.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:44 smithi181 ceph-mon[94189]: pgmap v12049: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 345 KiB/s wr, 8 op/s 2023-03-09T20:58:44.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:44 smithi038 ceph-mon[92422]: pgmap v12049: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 345 KiB/s wr, 8 op/s 2023-03-09T20:58:45.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21636.294469 secs 2023-03-09T20:58:45.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:45 smithi122 ceph-mon[93869]: pgmap v12050: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 298 KiB/s wr, 7 op/s 2023-03-09T20:58:45.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21636.294469 secs 2023-03-09T20:58:45.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:45 smithi181 ceph-mon[94189]: pgmap v12050: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 298 KiB/s wr, 7 op/s 2023-03-09T20:58:46.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21636.294469 secs 2023-03-09T20:58:46.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:45 smithi038 ceph-mon[92422]: pgmap v12050: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 298 KiB/s wr, 7 op/s 2023-03-09T20:58:47.967 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:58:47.967 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:58:47.968 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:47 smithi122 ceph-mon[93869]: pgmap v12051: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 299 KiB/s wr, 10 op/s 2023-03-09T20:58:47.968 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21638.617377 secs 2023-03-09T20:58:47.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:58:47.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:58:47.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:47 smithi181 ceph-mon[94189]: pgmap v12051: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 299 KiB/s wr, 10 op/s 2023-03-09T20:58:47.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21638.617377 secs 2023-03-09T20:58:48.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:58:48.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:58:48.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:47 smithi038 ceph-mon[92422]: pgmap v12051: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 299 KiB/s wr, 10 op/s 2023-03-09T20:58:48.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21638.617377 secs 2023-03-09T20:58:48.227 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 09 20:58:47 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[130534]: problem writing to /var/log/ceph/ceph-osd.1.log: (28) No space left on device 2023-03-09T20:58:48.228 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 09 20:58:47 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[132036]: problem writing to /var/log/ceph/ceph-osd.7.log: (28) No space left on device 2023-03-09T20:58:48.228 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 09 20:58:47 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-10[115379]: problem writing to /var/log/ceph/ceph-osd.10.log: (28) No space left on device 2023-03-09T20:58:48.229 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 09 20:58:48 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[124955]: problem writing to /var/log/ceph/ceph-osd.4.log: (28) No space left on device 2023-03-09T20:58:48.229 INFO:journalctl@ceph.mds.b.smithi122.stdout:Mar 09 20:58:47 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mds-b[119145]: problem writing to /var/log/ceph/ceph-mds.b.log: (28) No space left on device 2023-03-09T20:58:48.229 INFO:journalctl@ceph.mds.h.smithi122.stdout:Mar 09 20:58:47 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mds-h[118806]: problem writing to /var/log/ceph/ceph-mds.h.log: (28) No space left on device 2023-03-09T20:58:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:48 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mon-b[93845]: problem writing to /var/log/ceph/ceph-mon.b.log: (28) No space left on device 2023-03-09T20:58:48.728 INFO:journalctl@ceph.mds.e.smithi122.stdout:Mar 09 20:58:48 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mds-e[118464]: problem writing to /var/log/ceph/ceph-mds.e.log: (28) No space left on device 2023-03-09T20:58:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22100.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:58:50.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:50 smithi122 ceph-mon[93869]: pgmap v12052: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 640 KiB/s rd, 625 KiB/s wr, 11 op/s 2023-03-09T20:58:50.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:50 smithi181 ceph-mon[94189]: pgmap v12052: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 640 KiB/s rd, 625 KiB/s wr, 11 op/s 2023-03-09T20:58:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:50 smithi038 ceph-mon[92422]: pgmap v12052: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 640 KiB/s rd, 625 KiB/s wr, 11 op/s 2023-03-09T20:58:51.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21641.294529 secs 2023-03-09T20:58:51.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21641.294529 secs 2023-03-09T20:58:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21641.294529 secs 2023-03-09T20:58:52.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:52 smithi122 ceph-mon[93869]: pgmap v12053: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 369 KiB/s wr, 12 op/s 2023-03-09T20:58:52.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:52 smithi181 ceph-mon[94189]: pgmap v12053: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 369 KiB/s wr, 12 op/s 2023-03-09T20:58:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:52 smithi038 ceph-mon[92422]: pgmap v12053: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 369 KiB/s wr, 12 op/s 2023-03-09T20:58:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:58:53.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:58:53.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21643.617505 secs 2023-03-09T20:58:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:58:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:58:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21643.617505 secs 2023-03-09T20:58:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:58:53.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:58:53.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21643.617505 secs 2023-03-09T20:58:53.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22105.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:58:54.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:54 smithi122 ceph-mon[93869]: pgmap v12054: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 416 KiB/s rd, 390 KiB/s wr, 12 op/s 2023-03-09T20:58:54.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:54 smithi181 ceph-mon[94189]: pgmap v12054: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 416 KiB/s rd, 390 KiB/s wr, 12 op/s 2023-03-09T20:58:54.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:54 smithi038 ceph-mon[92422]: pgmap v12054: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 416 KiB/s rd, 390 KiB/s wr, 12 op/s 2023-03-09T20:58:55.977 INFO:journalctl@ceph.mgr.y.smithi122.stdout:Mar 09 20:58:55 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mgr-y[95318]: problem writing to /var/log/ceph/ceph-mgr.y.log: (28) No space left on device 2023-03-09T20:58:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21646.294590 secs 2023-03-09T20:58:56.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:56 smithi122 ceph-mon[93869]: pgmap v12055: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 513 KiB/s rd, 500 KiB/s wr, 12 op/s 2023-03-09T20:58:56.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:58:56.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:58:56.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21646.294590 secs 2023-03-09T20:58:56.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:56 smithi181 ceph-mon[94189]: pgmap v12055: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 513 KiB/s rd, 500 KiB/s wr, 12 op/s 2023-03-09T20:58:56.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:58:56.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:58:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21646.294590 secs 2023-03-09T20:58:56.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:56 smithi038 ceph-mon[92422]: pgmap v12055: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 513 KiB/s rd, 500 KiB/s wr, 12 op/s 2023-03-09T20:58:56.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:58:56.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:58:58.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:58 smithi122 ceph-mon[93869]: pgmap v12056: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 477 KiB/s wr, 13 op/s 2023-03-09T20:58:58.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:58:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21648.617641 secs 2023-03-09T20:58:58.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:58 smithi181 ceph-mon[94189]: pgmap v12056: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 477 KiB/s wr, 13 op/s 2023-03-09T20:58:58.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:58:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21648.617641 secs 2023-03-09T20:58:58.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:58 smithi038 ceph-mon[92422]: pgmap v12056: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 477 KiB/s wr, 13 op/s 2023-03-09T20:58:58.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:58:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21648.617641 secs 2023-03-09T20:58:58.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22110.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:59:00.158 INFO:journalctl@ceph.mds.k.smithi122.stdout:Mar 09 20:58:59 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mds-k[118124]: problem writing to /var/log/ceph/ceph-mds.k.log: (28) No space left on device 2023-03-09T20:59:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:00 smithi122 ceph-mon[93869]: pgmap v12057: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 735 KiB/s rd, 724 KiB/s wr, 13 op/s 2023-03-09T20:59:00.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:59:00.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:59:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:00 smithi181 ceph-mon[94189]: pgmap v12057: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 735 KiB/s rd, 724 KiB/s wr, 13 op/s 2023-03-09T20:59:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:59:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:59:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:00 smithi038 ceph-mon[92422]: pgmap v12057: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 735 KiB/s rd, 724 KiB/s wr, 13 op/s 2023-03-09T20:59:00.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:59:00.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:59:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21651.294675 secs 2023-03-09T20:59:01.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:59:01.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:59:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21651.294675 secs 2023-03-09T20:59:01.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:59:01.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:59:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21651.294675 secs 2023-03-09T20:59:01.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:59:01.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:59:02.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:02 smithi122 ceph-mon[93869]: pgmap v12058: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 404 KiB/s wr, 12 op/s 2023-03-09T20:59:02.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:02 smithi181 ceph-mon[94189]: pgmap v12058: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 404 KiB/s wr, 12 op/s 2023-03-09T20:59:02.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:02 smithi038 ceph-mon[92422]: pgmap v12058: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 404 KiB/s wr, 12 op/s 2023-03-09T20:59:03.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21653.617785 secs 2023-03-09T20:59:03.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21653.617785 secs 2023-03-09T20:59:03.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21653.617785 secs 2023-03-09T20:59:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22115.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:59:04.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:04 smithi122 ceph-mon[93869]: pgmap v12059: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 406 KiB/s rd, 425 KiB/s wr, 23 op/s 2023-03-09T20:59:04.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:04 smithi181 ceph-mon[94189]: pgmap v12059: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 406 KiB/s rd, 425 KiB/s wr, 23 op/s 2023-03-09T20:59:04.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:04 smithi038 ceph-mon[92422]: pgmap v12059: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 406 KiB/s rd, 425 KiB/s wr, 23 op/s 2023-03-09T20:59:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21656.294737 secs 2023-03-09T20:59:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:05 smithi122 ceph-mon[93869]: pgmap v12060: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 486 KiB/s wr, 23 op/s 2023-03-09T20:59:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21656.294737 secs 2023-03-09T20:59:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:05 smithi181 ceph-mon[94189]: pgmap v12060: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 486 KiB/s wr, 23 op/s 2023-03-09T20:59:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21656.294737 secs 2023-03-09T20:59:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:05 smithi038 ceph-mon[92422]: pgmap v12060: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 486 KiB/s wr, 23 op/s 2023-03-09T20:59:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:08 smithi122 ceph-mon[93869]: pgmap v12061: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 375 KiB/s wr, 25 op/s 2023-03-09T20:59:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21658.617917 secs 2023-03-09T20:59:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:08 smithi181 ceph-mon[94189]: pgmap v12061: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 375 KiB/s wr, 25 op/s 2023-03-09T20:59:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21658.617917 secs 2023-03-09T20:59:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:08 smithi038 ceph-mon[92422]: pgmap v12061: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 375 KiB/s wr, 25 op/s 2023-03-09T20:59:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21658.617917 secs 2023-03-09T20:59:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22120.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:59:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:10 smithi122 ceph-mon[93869]: pgmap v12062: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 375 KiB/s wr, 24 op/s 2023-03-09T20:59:10.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:10 smithi122 ceph-mon[93869]: osdmap e967: 12 total, 12 up, 12 in 2023-03-09T20:59:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:10 smithi181 ceph-mon[94189]: pgmap v12062: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 375 KiB/s wr, 24 op/s 2023-03-09T20:59:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:10 smithi181 ceph-mon[94189]: osdmap e967: 12 total, 12 up, 12 in 2023-03-09T20:59:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:10 smithi038 ceph-mon[92422]: pgmap v12062: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 375 KiB/s wr, 24 op/s 2023-03-09T20:59:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:10 smithi038 ceph-mon[92422]: osdmap e967: 12 total, 12 up, 12 in 2023-03-09T20:59:11.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21661.294799 secs 2023-03-09T20:59:11.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21661.294799 secs 2023-03-09T20:59:11.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21661.294799 secs 2023-03-09T20:59:12.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:12 smithi122 ceph-mon[93869]: pgmap v12064: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 146 KiB/s wr, 24 op/s 2023-03-09T20:59:12.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:12 smithi181 ceph-mon[94189]: pgmap v12064: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 146 KiB/s wr, 24 op/s 2023-03-09T20:59:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:12 smithi038 ceph-mon[92422]: pgmap v12064: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 146 KiB/s wr, 24 op/s 2023-03-09T20:59:13.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21663.618054 secs 2023-03-09T20:59:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21663.618054 secs 2023-03-09T20:59:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21663.618054 secs 2023-03-09T20:59:13.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22125.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:59:14.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:14 smithi122 ceph-mon[93869]: pgmap v12065: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 119 KiB/s wr, 9 op/s 2023-03-09T20:59:14.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:14 smithi181 ceph-mon[94189]: pgmap v12065: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 119 KiB/s wr, 9 op/s 2023-03-09T20:59:14.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:14 smithi038 ceph-mon[92422]: pgmap v12065: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 119 KiB/s wr, 9 op/s 2023-03-09T20:59:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21666.294868 secs 2023-03-09T20:59:15.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:15 smithi122 ceph-mon[93869]: pgmap v12066: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 20 KiB/s wr, 8 op/s 2023-03-09T20:59:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21666.294868 secs 2023-03-09T20:59:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:15 smithi181 ceph-mon[94189]: pgmap v12066: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 20 KiB/s wr, 8 op/s 2023-03-09T20:59:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21666.294868 secs 2023-03-09T20:59:16.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:15 smithi038 ceph-mon[92422]: pgmap v12066: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 20 KiB/s wr, 8 op/s 2023-03-09T20:59:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:18 smithi122 ceph-mon[93869]: pgmap v12067: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T20:59:18.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21668.618125 secs 2023-03-09T20:59:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:18 smithi181 ceph-mon[94189]: pgmap v12067: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T20:59:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21668.618125 secs 2023-03-09T20:59:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:18 smithi038 ceph-mon[92422]: pgmap v12067: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T20:59:18.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21668.618125 secs 2023-03-09T20:59:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22130.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:59:20.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:20 smithi122 ceph-mon[93869]: pgmap v12068: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T20:59:20.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:20 smithi122 ceph-mon[93869]: osdmap e968: 12 total, 12 up, 12 in 2023-03-09T20:59:20.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:20 smithi181 ceph-mon[94189]: pgmap v12068: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T20:59:20.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:20 smithi181 ceph-mon[94189]: osdmap e968: 12 total, 12 up, 12 in 2023-03-09T20:59:20.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:20 smithi038 ceph-mon[92422]: pgmap v12068: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T20:59:20.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:20 smithi038 ceph-mon[92422]: osdmap e968: 12 total, 12 up, 12 in 2023-03-09T20:59:21.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21671.294949 secs 2023-03-09T20:59:21.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21671.294949 secs 2023-03-09T20:59:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21671.294949 secs 2023-03-09T20:59:22.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:22 smithi122 ceph-mon[93869]: pgmap v12070: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T20:59:22.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:22 smithi181 ceph-mon[94189]: pgmap v12070: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T20:59:22.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:22 smithi038 ceph-mon[92422]: pgmap v12070: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T20:59:23.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21673.618195 secs 2023-03-09T20:59:23.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21673.618195 secs 2023-03-09T20:59:23.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21673.618195 secs 2023-03-09T20:59:23.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22135.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:59:24.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:24 smithi122 ceph-mon[93869]: pgmap v12071: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T20:59:24.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:24 smithi181 ceph-mon[94189]: pgmap v12071: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T20:59:24.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:24 smithi038 ceph-mon[92422]: pgmap v12071: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T20:59:25.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21676.294986 secs 2023-03-09T20:59:25.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:25 smithi122 ceph-mon[93869]: pgmap v12072: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2023-03-09T20:59:25.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21676.294986 secs 2023-03-09T20:59:25.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:25 smithi181 ceph-mon[94189]: pgmap v12072: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2023-03-09T20:59:26.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21676.294986 secs 2023-03-09T20:59:26.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:25 smithi038 ceph-mon[92422]: pgmap v12072: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2023-03-09T20:59:28.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:28 smithi122 ceph-mon[93869]: pgmap v12073: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2023-03-09T20:59:28.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21678.618309 secs 2023-03-09T20:59:28.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:28 smithi181 ceph-mon[94189]: pgmap v12073: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2023-03-09T20:59:28.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21678.618309 secs 2023-03-09T20:59:28.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:28 smithi038 ceph-mon[92422]: pgmap v12073: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2023-03-09T20:59:28.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21678.618309 secs 2023-03-09T20:59:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22140.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:59:30.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:30 smithi122 ceph-mon[93869]: pgmap v12074: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T20:59:30.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:30 smithi181 ceph-mon[94189]: pgmap v12074: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T20:59:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:30 smithi038 ceph-mon[92422]: pgmap v12074: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T20:59:31.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21681.295047 secs 2023-03-09T20:59:31.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21681.295047 secs 2023-03-09T20:59:31.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21681.295047 secs 2023-03-09T20:59:32.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:32 smithi122 ceph-mon[93869]: pgmap v12075: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T20:59:32.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:32 smithi181 ceph-mon[94189]: pgmap v12075: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T20:59:32.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:32 smithi038 ceph-mon[92422]: pgmap v12075: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T20:59:33.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21683.618382 secs 2023-03-09T20:59:33.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21683.618382 secs 2023-03-09T20:59:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21683.618382 secs 2023-03-09T20:59:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22145.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:59:34.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:34 smithi122 ceph-mon[93869]: pgmap v12076: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T20:59:34.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:34 smithi181 ceph-mon[94189]: pgmap v12076: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T20:59:34.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:34 smithi038 ceph-mon[92422]: pgmap v12076: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T20:59:35.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21686.295104 secs 2023-03-09T20:59:35.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:35 smithi122 ceph-mon[93869]: pgmap v12077: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T20:59:35.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21686.295104 secs 2023-03-09T20:59:35.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:35 smithi181 ceph-mon[94189]: pgmap v12077: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T20:59:36.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21686.295104 secs 2023-03-09T20:59:36.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:35 smithi038 ceph-mon[92422]: pgmap v12077: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T20:59:38.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:38 smithi122 ceph-mon[93869]: pgmap v12078: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 682 B/s wr, 7 op/s 2023-03-09T20:59:38.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21688.618490 secs 2023-03-09T20:59:38.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:38 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:59:38.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:38 smithi181 ceph-mon[94189]: pgmap v12078: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 682 B/s wr, 7 op/s 2023-03-09T20:59:38.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21688.618490 secs 2023-03-09T20:59:38.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:38 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:59:38.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:38 smithi038 ceph-mon[92422]: pgmap v12078: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 682 B/s wr, 7 op/s 2023-03-09T20:59:38.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21688.618490 secs 2023-03-09T20:59:38.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:38 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T20:59:38.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22150.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:59:39.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:59:39.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:59:39.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:59:39.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:59:39.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:59:39.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:59:39.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:59:39.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:59:39.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:59:39.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:59:39.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:59:39.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:59:39.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:59:39.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:59:39.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:59:39.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:59:39.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:59:39.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:59:39.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T20:59:39.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T20:59:39.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T20:59:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:40 smithi038 ceph-mon[92422]: pgmap v12079: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 197 KiB/s wr, 7 op/s 2023-03-09T20:59:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:40 smithi122 ceph-mon[93869]: pgmap v12079: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 197 KiB/s wr, 7 op/s 2023-03-09T20:59:40.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:40 smithi181 ceph-mon[94189]: pgmap v12079: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 197 KiB/s wr, 7 op/s 2023-03-09T20:59:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21691.295165 secs 2023-03-09T20:59:41.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21691.295165 secs 2023-03-09T20:59:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21691.295165 secs 2023-03-09T20:59:42.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:42 smithi122 ceph-mon[93869]: pgmap v12080: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 213 KiB/s wr, 8 op/s 2023-03-09T20:59:42.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:42 smithi181 ceph-mon[94189]: pgmap v12080: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 213 KiB/s wr, 8 op/s 2023-03-09T20:59:42.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:42 smithi038 ceph-mon[92422]: pgmap v12080: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 213 KiB/s wr, 8 op/s 2023-03-09T20:59:43.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21693.618614 secs 2023-03-09T20:59:43.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21693.618614 secs 2023-03-09T20:59:43.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21693.618614 secs 2023-03-09T20:59:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22155.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:59:44.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:44 smithi122 ceph-mon[93869]: pgmap v12081: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 231 KiB/s wr, 7 op/s 2023-03-09T20:59:44.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:44 smithi181 ceph-mon[94189]: pgmap v12081: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 231 KiB/s wr, 7 op/s 2023-03-09T20:59:44.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:44 smithi038 ceph-mon[92422]: pgmap v12081: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 231 KiB/s wr, 7 op/s 2023-03-09T20:59:45.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21696.295223 secs 2023-03-09T20:59:45.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:45 smithi122 ceph-mon[93869]: pgmap v12082: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 231 KiB/s wr, 7 op/s 2023-03-09T20:59:45.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21696.295223 secs 2023-03-09T20:59:45.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:45 smithi181 ceph-mon[94189]: pgmap v12082: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 231 KiB/s wr, 7 op/s 2023-03-09T20:59:46.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21696.295223 secs 2023-03-09T20:59:46.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:45 smithi038 ceph-mon[92422]: pgmap v12082: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 231 KiB/s wr, 7 op/s 2023-03-09T20:59:47.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:59:47.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:59:47.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:47 smithi122 ceph-mon[93869]: pgmap v12083: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 393 KiB/s wr, 10 op/s 2023-03-09T20:59:47.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:59:47.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:59:47.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:47 smithi181 ceph-mon[94189]: pgmap v12083: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 393 KiB/s wr, 10 op/s 2023-03-09T20:59:48.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:59:48.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:59:48.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:47 smithi038 ceph-mon[92422]: pgmap v12083: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 393 KiB/s wr, 10 op/s 2023-03-09T20:59:48.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22160.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:59:48.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:59:48.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:59:48.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:48 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:59:48.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:59:48.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:59:48.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:48 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:59:49.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:59:49.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:59:49.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:48 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T20:59:49.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:49 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:59:49.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:59:49.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:59:49.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:49 smithi122 ceph-mon[93869]: pgmap v12084: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 411 KiB/s wr, 10 op/s 2023-03-09T20:59:49.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:49 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:59:49.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:59:49.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:59:49.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:49 smithi181 ceph-mon[94189]: pgmap v12084: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 411 KiB/s wr, 10 op/s 2023-03-09T20:59:50.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:49 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T20:59:50.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T20:59:50.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:59:50.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:49 smithi038 ceph-mon[92422]: pgmap v12084: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 411 KiB/s wr, 10 op/s 2023-03-09T20:59:50.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21701.295287 secs 2023-03-09T20:59:50.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21701.295287 secs 2023-03-09T20:59:51.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21701.295287 secs 2023-03-09T20:59:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:51 smithi122 ceph-mon[93869]: pgmap v12085: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 513 KiB/s rd, 490 KiB/s wr, 12 op/s 2023-03-09T20:59:51.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:51 smithi181 ceph-mon[94189]: pgmap v12085: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 513 KiB/s rd, 490 KiB/s wr, 12 op/s 2023-03-09T20:59:52.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:51 smithi038 ceph-mon[92422]: pgmap v12085: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 513 KiB/s rd, 490 KiB/s wr, 12 op/s 2023-03-09T20:59:53.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22165.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T20:59:54.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:54 smithi122 ceph-mon[93869]: pgmap v12086: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 495 KiB/s wr, 12 op/s 2023-03-09T20:59:54.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:54 smithi181 ceph-mon[94189]: pgmap v12086: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 495 KiB/s wr, 12 op/s 2023-03-09T20:59:54.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:54 smithi038 ceph-mon[92422]: pgmap v12086: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 495 KiB/s wr, 12 op/s 2023-03-09T20:59:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21706.295372 secs 2023-03-09T20:59:56.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:56 smithi122 ceph-mon[93869]: pgmap v12087: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 478 KiB/s wr, 11 op/s 2023-03-09T20:59:56.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:59:56.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:59:56.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21706.295372 secs 2023-03-09T20:59:56.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:56 smithi181 ceph-mon[94189]: pgmap v12087: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 478 KiB/s wr, 11 op/s 2023-03-09T20:59:56.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:59:56.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:59:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21706.295372 secs 2023-03-09T20:59:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:56 smithi038 ceph-mon[92422]: pgmap v12087: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 478 KiB/s wr, 11 op/s 2023-03-09T20:59:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T20:59:56.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:59:57.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:59:57.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:59:57.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:57 smithi122 ceph-mon[93869]: pgmap v12088: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 594 KiB/s rd, 583 KiB/s wr, 14 op/s 2023-03-09T20:59:57.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 20:59:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21708.618895 secs 2023-03-09T20:59:57.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:59:57.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:59:57.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:57 smithi181 ceph-mon[94189]: pgmap v12088: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 594 KiB/s rd, 583 KiB/s wr, 14 op/s 2023-03-09T20:59:57.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 20:59:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21708.618895 secs 2023-03-09T20:59:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T20:59:58.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T20:59:58.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:57 smithi038 ceph-mon[92422]: pgmap v12088: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 594 KiB/s rd, 583 KiB/s wr, 14 op/s 2023-03-09T20:59:58.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 20:59:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21708.618895 secs 2023-03-09T20:59:58.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22170.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:00:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:00 smithi122 ceph-mon[93869]: pgmap v12089: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 446 KiB/s rd, 453 KiB/s wr, 12 op/s 2023-03-09T21:00:00.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:00 smithi122 ceph-mon[93869]: overall HEALTH_WARN 1 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-09T21:00:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:00 smithi181 ceph-mon[94189]: pgmap v12089: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 446 KiB/s rd, 453 KiB/s wr, 12 op/s 2023-03-09T21:00:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:00 smithi181 ceph-mon[94189]: overall HEALTH_WARN 1 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-09T21:00:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:00 smithi038 ceph-mon[92422]: pgmap v12089: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 446 KiB/s rd, 453 KiB/s wr, 12 op/s 2023-03-09T21:00:00.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:00 smithi038 ceph-mon[92422]: overall HEALTH_WARN 1 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-09T21:00:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21711.295431 secs 2023-03-09T21:00:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21711.295431 secs 2023-03-09T21:00:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21711.295431 secs 2023-03-09T21:00:02.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:02 smithi122 ceph-mon[93869]: pgmap v12090: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 533 KiB/s rd, 547 KiB/s wr, 13 op/s 2023-03-09T21:00:02.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:02 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:00:02.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:02 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:00:02.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:02 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:00:02.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:02 smithi181 ceph-mon[94189]: pgmap v12090: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 533 KiB/s rd, 547 KiB/s wr, 13 op/s 2023-03-09T21:00:02.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:02 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:00:02.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:02 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:00:02.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:02 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:00:02.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:02 smithi038 ceph-mon[92422]: pgmap v12090: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 533 KiB/s rd, 547 KiB/s wr, 13 op/s 2023-03-09T21:00:02.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:02 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:00:02.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:02 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:00:02.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:02 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:00:03.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21713.619035 secs 2023-03-09T21:00:03.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21713.619035 secs 2023-03-09T21:00:03.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21713.619035 secs 2023-03-09T21:00:03.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22175.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:00:04.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:04 smithi122 ceph-mon[93869]: pgmap v12091: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 293 KiB/s wr, 22 op/s 2023-03-09T21:00:04.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:04 smithi181 ceph-mon[94189]: pgmap v12091: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 293 KiB/s wr, 22 op/s 2023-03-09T21:00:04.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:04 smithi038 ceph-mon[92422]: pgmap v12091: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 293 KiB/s wr, 22 op/s 2023-03-09T21:00:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21716.295489 secs 2023-03-09T21:00:05.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:05 smithi122 ceph-mon[93869]: pgmap v12092: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 272 KiB/s wr, 21 op/s 2023-03-09T21:00:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21716.295489 secs 2023-03-09T21:00:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:05 smithi181 ceph-mon[94189]: pgmap v12092: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 272 KiB/s wr, 21 op/s 2023-03-09T21:00:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21716.295489 secs 2023-03-09T21:00:06.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:05 smithi038 ceph-mon[92422]: pgmap v12092: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 272 KiB/s wr, 21 op/s 2023-03-09T21:00:08.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:08 smithi122 ceph-mon[93869]: pgmap v12093: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 377 KiB/s wr, 26 op/s 2023-03-09T21:00:08.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21718.619157 secs 2023-03-09T21:00:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:08 smithi181 ceph-mon[94189]: pgmap v12093: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 377 KiB/s wr, 26 op/s 2023-03-09T21:00:08.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21718.619157 secs 2023-03-09T21:00:08.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:08 smithi038 ceph-mon[92422]: pgmap v12093: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 377 KiB/s wr, 26 op/s 2023-03-09T21:00:08.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21718.619157 secs 2023-03-09T21:00:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22180.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:00:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:10 smithi122 ceph-mon[93869]: pgmap v12094: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 272 KiB/s wr, 22 op/s 2023-03-09T21:00:10.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:10 smithi122 ceph-mon[93869]: osdmap e969: 12 total, 12 up, 12 in 2023-03-09T21:00:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:10 smithi181 ceph-mon[94189]: pgmap v12094: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 272 KiB/s wr, 22 op/s 2023-03-09T21:00:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:10 smithi181 ceph-mon[94189]: osdmap e969: 12 total, 12 up, 12 in 2023-03-09T21:00:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:10 smithi038 ceph-mon[92422]: pgmap v12094: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 272 KiB/s wr, 22 op/s 2023-03-09T21:00:10.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:10 smithi038 ceph-mon[92422]: osdmap e969: 12 total, 12 up, 12 in 2023-03-09T21:00:11.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21721.295574 secs 2023-03-09T21:00:11.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21721.295574 secs 2023-03-09T21:00:11.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21721.295574 secs 2023-03-09T21:00:12.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:12 smithi122 ceph-mon[93869]: pgmap v12096: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 153 KiB/s wr, 25 op/s 2023-03-09T21:00:12.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:12 smithi181 ceph-mon[94189]: pgmap v12096: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 153 KiB/s wr, 25 op/s 2023-03-09T21:00:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:12 smithi038 ceph-mon[92422]: pgmap v12096: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 153 KiB/s wr, 25 op/s 2023-03-09T21:00:13.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21723.619229 secs 2023-03-09T21:00:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21723.619229 secs 2023-03-09T21:00:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21723.619229 secs 2023-03-09T21:00:13.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22185.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:00:14.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:14 smithi122 ceph-mon[93869]: pgmap v12097: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 126 KiB/s wr, 10 op/s 2023-03-09T21:00:14.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:14 smithi181 ceph-mon[94189]: pgmap v12097: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 126 KiB/s wr, 10 op/s 2023-03-09T21:00:14.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:14 smithi038 ceph-mon[92422]: pgmap v12097: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 126 KiB/s wr, 10 op/s 2023-03-09T21:00:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21726.295634 secs 2023-03-09T21:00:15.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:15 smithi122 ceph-mon[93869]: pgmap v12098: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 126 KiB/s wr, 10 op/s 2023-03-09T21:00:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21726.295634 secs 2023-03-09T21:00:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:15 smithi181 ceph-mon[94189]: pgmap v12098: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 126 KiB/s wr, 10 op/s 2023-03-09T21:00:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21726.295634 secs 2023-03-09T21:00:16.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:15 smithi038 ceph-mon[92422]: pgmap v12098: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 126 KiB/s wr, 10 op/s 2023-03-09T21:00:16.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:00:16.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:00:16.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:00:16.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:00:17.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:00:17.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:00:17.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:17 smithi122 ceph-mon[93869]: pgmap v12099: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 3.0 KiB/s wr, 7 op/s 2023-03-09T21:00:17.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21728.619317 secs 2023-03-09T21:00:17.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:17 smithi181 ceph-mon[94189]: pgmap v12099: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 3.0 KiB/s wr, 7 op/s 2023-03-09T21:00:17.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21728.619317 secs 2023-03-09T21:00:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:17 smithi038 ceph-mon[92422]: pgmap v12099: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 3.0 KiB/s wr, 7 op/s 2023-03-09T21:00:18.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21728.619317 secs 2023-03-09T21:00:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22190.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:00:19.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:19 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:00:19.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:19 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:00:19.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:19 smithi122 ceph-mon[93869]: pgmap v12100: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 3.0 KiB/s wr, 7 op/s 2023-03-09T21:00:19.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:19 smithi122 ceph-mon[93869]: osdmap e970: 12 total, 12 up, 12 in 2023-03-09T21:00:19.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:19 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:00:19.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:19 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:00:19.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:19 smithi181 ceph-mon[94189]: pgmap v12100: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 3.0 KiB/s wr, 7 op/s 2023-03-09T21:00:19.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:19 smithi181 ceph-mon[94189]: osdmap e970: 12 total, 12 up, 12 in 2023-03-09T21:00:20.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:19 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:00:20.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:19 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:00:20.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:19 smithi038 ceph-mon[92422]: pgmap v12100: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 3.0 KiB/s wr, 7 op/s 2023-03-09T21:00:20.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:19 smithi038 ceph-mon[92422]: osdmap e970: 12 total, 12 up, 12 in 2023-03-09T21:00:20.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21731.295730 secs 2023-03-09T21:00:20.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21731.295730 secs 2023-03-09T21:00:21.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21731.295730 secs 2023-03-09T21:00:21.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:21 smithi122 ceph-mon[93869]: pgmap v12102: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 2.9 KiB/s wr, 6 op/s 2023-03-09T21:00:21.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:21 smithi181 ceph-mon[94189]: pgmap v12102: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 2.9 KiB/s wr, 6 op/s 2023-03-09T21:00:22.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:21 smithi038 ceph-mon[92422]: pgmap v12102: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 2.9 KiB/s wr, 6 op/s 2023-03-09T21:00:22.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21733.619435 secs 2023-03-09T21:00:22.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21733.619435 secs 2023-03-09T21:00:23.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21733.619435 secs 2023-03-09T21:00:23.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22195.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:00:23.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:23 smithi122 ceph-mon[93869]: pgmap v12103: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 2.8 KiB/s wr, 6 op/s 2023-03-09T21:00:23.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:23 smithi181 ceph-mon[94189]: pgmap v12103: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 2.8 KiB/s wr, 6 op/s 2023-03-09T21:00:24.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:23 smithi038 ceph-mon[92422]: pgmap v12103: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 2.8 KiB/s wr, 6 op/s 2023-03-09T21:00:25.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21736.295793 secs 2023-03-09T21:00:25.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:25 smithi122 ceph-mon[93869]: pgmap v12104: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 2.8 KiB/s wr, 6 op/s 2023-03-09T21:00:25.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21736.295793 secs 2023-03-09T21:00:25.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:25 smithi181 ceph-mon[94189]: pgmap v12104: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 2.8 KiB/s wr, 6 op/s 2023-03-09T21:00:26.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21736.295793 secs 2023-03-09T21:00:26.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:25 smithi038 ceph-mon[92422]: pgmap v12104: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 2.8 KiB/s wr, 6 op/s 2023-03-09T21:00:28.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:28 smithi122 ceph-mon[93869]: pgmap v12105: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T21:00:28.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21738.619580 secs 2023-03-09T21:00:28.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:28 smithi181 ceph-mon[94189]: pgmap v12105: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T21:00:28.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21738.619580 secs 2023-03-09T21:00:28.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:28 smithi038 ceph-mon[92422]: pgmap v12105: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T21:00:28.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21738.619580 secs 2023-03-09T21:00:28.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22200.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:00:30.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:30 smithi122 ceph-mon[93869]: pgmap v12106: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T21:00:30.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:30 smithi181 ceph-mon[94189]: pgmap v12106: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T21:00:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:30 smithi038 ceph-mon[92422]: pgmap v12106: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T21:00:31.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21741.295856 secs 2023-03-09T21:00:31.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21741.295856 secs 2023-03-09T21:00:31.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21741.295856 secs 2023-03-09T21:00:32.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:32 smithi122 ceph-mon[93869]: pgmap v12107: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T21:00:32.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:32 smithi181 ceph-mon[94189]: pgmap v12107: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T21:00:32.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:32 smithi038 ceph-mon[92422]: pgmap v12107: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T21:00:33.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21743.619707 secs 2023-03-09T21:00:33.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21743.619707 secs 2023-03-09T21:00:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21743.619707 secs 2023-03-09T21:00:33.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22205.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:00:34.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:34 smithi122 ceph-mon[93869]: pgmap v12108: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T21:00:34.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:34 smithi181 ceph-mon[94189]: pgmap v12108: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T21:00:34.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:34 smithi038 ceph-mon[92422]: pgmap v12108: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T21:00:35.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21746.295955 secs 2023-03-09T21:00:35.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:35 smithi122 ceph-mon[93869]: pgmap v12109: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 511 B/s wr, 5 op/s 2023-03-09T21:00:35.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21746.295955 secs 2023-03-09T21:00:35.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:35 smithi181 ceph-mon[94189]: pgmap v12109: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 511 B/s wr, 5 op/s 2023-03-09T21:00:36.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21746.295955 secs 2023-03-09T21:00:36.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:35 smithi038 ceph-mon[92422]: pgmap v12109: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 511 B/s wr, 5 op/s 2023-03-09T21:00:38.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:38 smithi122 ceph-mon[93869]: pgmap v12110: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 71 KiB/s wr, 8 op/s 2023-03-09T21:00:38.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:38 smithi181 ceph-mon[94189]: pgmap v12110: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 71 KiB/s wr, 8 op/s 2023-03-09T21:00:38.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:38 smithi038 ceph-mon[92422]: pgmap v12110: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 71 KiB/s wr, 8 op/s 2023-03-09T21:00:38.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22210.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:00:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:40 smithi122 ceph-mon[93869]: pgmap v12111: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 71 KiB/s wr, 6 op/s 2023-03-09T21:00:40.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:00:40.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:00:40.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:00:40.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:00:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:40 smithi181 ceph-mon[94189]: pgmap v12111: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 71 KiB/s wr, 6 op/s 2023-03-09T21:00:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:00:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:00:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:00:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:00:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:40 smithi038 ceph-mon[92422]: pgmap v12111: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 71 KiB/s wr, 6 op/s 2023-03-09T21:00:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:00:40.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:00:40.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:00:40.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:00:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21751.295983 secs 2023-03-09T21:00:41.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:41 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:00:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21751.295983 secs 2023-03-09T21:00:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:41 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:00:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21751.295983 secs 2023-03-09T21:00:41.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:41 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:00:42.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:42 smithi122 ceph-mon[93869]: pgmap v12112: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 71 KiB/s wr, 8 op/s 2023-03-09T21:00:42.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:42 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:00:42.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:42 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:00:42.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:42 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:00:42.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:42 smithi181 ceph-mon[94189]: pgmap v12112: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 71 KiB/s wr, 8 op/s 2023-03-09T21:00:42.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:42 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:00:42.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:42 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:00:42.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:42 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:00:42.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:42 smithi038 ceph-mon[92422]: pgmap v12112: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 71 KiB/s wr, 8 op/s 2023-03-09T21:00:42.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:42 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:00:42.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:42 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:00:42.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:42 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:00:43.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22215.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:00:44.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:44 smithi122 ceph-mon[93869]: pgmap v12113: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 292 KiB/s wr, 9 op/s 2023-03-09T21:00:44.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:44 smithi181 ceph-mon[94189]: pgmap v12113: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 292 KiB/s wr, 9 op/s 2023-03-09T21:00:44.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:44 smithi038 ceph-mon[92422]: pgmap v12113: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 292 KiB/s wr, 9 op/s 2023-03-09T21:00:45.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21756.296045 secs 2023-03-09T21:00:45.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:45 smithi122 ceph-mon[93869]: pgmap v12114: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 291 KiB/s wr, 7 op/s 2023-03-09T21:00:45.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21756.296045 secs 2023-03-09T21:00:45.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:45 smithi181 ceph-mon[94189]: pgmap v12114: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 291 KiB/s wr, 7 op/s 2023-03-09T21:00:46.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21756.296045 secs 2023-03-09T21:00:46.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:45 smithi038 ceph-mon[92422]: pgmap v12114: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 291 KiB/s wr, 7 op/s 2023-03-09T21:00:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:48 smithi122 ceph-mon[93869]: pgmap v12115: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 476 KiB/s rd, 465 KiB/s wr, 11 op/s 2023-03-09T21:00:48.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:48 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 21758.620121 secs 2023-03-09T21:00:48.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:48 smithi181 ceph-mon[94189]: pgmap v12115: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 476 KiB/s rd, 465 KiB/s wr, 11 op/s 2023-03-09T21:00:48.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:48 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 21758.620121 secs 2023-03-09T21:00:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:48 smithi038 ceph-mon[92422]: pgmap v12115: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 476 KiB/s rd, 465 KiB/s wr, 11 op/s 2023-03-09T21:00:48.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:48 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 21758.620121 secs 2023-03-09T21:00:48.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22220.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:00:49.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:49 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:00:49.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:00:49.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:00:49.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:49 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:00:49.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:00:49.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:00:49.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:49 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:00:49.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:00:49.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:00:50.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:50 smithi122 ceph-mon[93869]: pgmap v12116: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 413 KiB/s wr, 10 op/s 2023-03-09T21:00:50.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:50 smithi181 ceph-mon[94189]: pgmap v12116: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 413 KiB/s wr, 10 op/s 2023-03-09T21:00:50.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:50 smithi038 ceph-mon[92422]: pgmap v12116: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 413 KiB/s wr, 10 op/s 2023-03-09T21:00:51.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21761.296105 secs 2023-03-09T21:00:51.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21761.296105 secs 2023-03-09T21:00:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21761.296105 secs 2023-03-09T21:00:52.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:52 smithi122 ceph-mon[93869]: pgmap v12117: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 603 KiB/s rd, 582 KiB/s wr, 12 op/s 2023-03-09T21:00:52.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:00:52.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:00:52.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:52 smithi181 ceph-mon[94189]: pgmap v12117: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 603 KiB/s rd, 582 KiB/s wr, 12 op/s 2023-03-09T21:00:52.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:00:52.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:00:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:52 smithi038 ceph-mon[92422]: pgmap v12117: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 603 KiB/s rd, 582 KiB/s wr, 12 op/s 2023-03-09T21:00:52.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:00:52.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:00:53.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21763.620185 secs 2023-03-09T21:00:53.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:00:53.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:00:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21763.620185 secs 2023-03-09T21:00:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:00:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:00:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21763.620185 secs 2023-03-09T21:00:53.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:00:53.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:00:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22225.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:00:54.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:54 smithi122 ceph-mon[93869]: pgmap v12118: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 698 KiB/s wr, 12 op/s 2023-03-09T21:00:54.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:54 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:00:54.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:54 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:00:54.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:54 smithi181 ceph-mon[94189]: pgmap v12118: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 698 KiB/s wr, 12 op/s 2023-03-09T21:00:54.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:54 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:00:54.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:54 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:00:54.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:54 smithi038 ceph-mon[92422]: pgmap v12118: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 698 KiB/s wr, 12 op/s 2023-03-09T21:00:54.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:54 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:00:54.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:54 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:00:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21766.296162 secs 2023-03-09T21:00:56.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:56 smithi038 ceph-mon[92422]: pgmap v12119: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 481 KiB/s wr, 10 op/s 2023-03-09T21:00:56.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:00:56.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:00:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21766.296162 secs 2023-03-09T21:00:56.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:56 smithi122 ceph-mon[93869]: pgmap v12119: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 481 KiB/s wr, 10 op/s 2023-03-09T21:00:56.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:00:56.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:00:56.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21766.296162 secs 2023-03-09T21:00:56.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:56 smithi181 ceph-mon[94189]: pgmap v12119: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 481 KiB/s wr, 10 op/s 2023-03-09T21:00:56.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:00:56.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:00:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:00:57.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:00:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:00:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:00:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:00:57.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:00:58.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:58 smithi122 ceph-mon[93869]: pgmap v12120: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 497 KiB/s rd, 481 KiB/s wr, 13 op/s 2023-03-09T21:00:58.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:00:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21768.620305 secs 2023-03-09T21:00:58.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:58 smithi181 ceph-mon[94189]: pgmap v12120: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 497 KiB/s rd, 481 KiB/s wr, 13 op/s 2023-03-09T21:00:58.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:00:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21768.620305 secs 2023-03-09T21:00:58.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:58 smithi038 ceph-mon[92422]: pgmap v12120: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 497 KiB/s rd, 481 KiB/s wr, 13 op/s 2023-03-09T21:00:58.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:00:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21768.620305 secs 2023-03-09T21:00:58.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22230.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:01:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:01:00 smithi122 ceph-mon[93869]: pgmap v12121: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 566 KiB/s rd, 554 KiB/s wr, 13 op/s 2023-03-09T21:01:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:01:00 smithi181 ceph-mon[94189]: pgmap v12121: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 566 KiB/s rd, 554 KiB/s wr, 13 op/s 2023-03-09T21:01:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:01:00 smithi038 ceph-mon[92422]: pgmap v12121: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 566 KiB/s rd, 554 KiB/s wr, 13 op/s 2023-03-09T21:01:01.553 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:01:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21771.296222 secs 2023-03-09T21:01:01.707 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:01:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21771.296222 secs 2023-03-09T21:01:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:01:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21771.296222 secs 2023-03-09T21:01:02.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:01:02 smithi122 ceph-mon[93869]: pgmap v12122: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 538 KiB/s rd, 537 KiB/s wr, 13 op/s 2023-03-09T21:01:02.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:01:02 smithi181 ceph-mon[94189]: pgmap v12122: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 538 KiB/s rd, 537 KiB/s wr, 13 op/s 2023-03-09T21:01:02.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:01:02 smithi038 ceph-mon[92422]: pgmap v12122: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 538 KiB/s rd, 537 KiB/s wr, 13 op/s 2023-03-09T21:01:03.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:01:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21773.620421 secs 2023-03-09T21:01:03.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:01:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21773.620421 secs 2023-03-09T21:01:03.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:01:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21773.620421 secs 2023-03-09T21:01:03.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22235.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:01:04.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:01:04 smithi122 ceph-mon[93869]: pgmap v12123: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 390 KiB/s wr, 24 op/s 2023-03-09T21:01:04.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:01:04 smithi181 ceph-mon[94189]: pgmap v12123: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 390 KiB/s wr, 24 op/s 2023-03-09T21:01:04.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:01:04 smithi038 ceph-mon[92422]: pgmap v12123: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 390 KiB/s wr, 24 op/s 2023-03-09T21:01:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:01:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21776.296283 secs 2023-03-09T21:01:05.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:01:05 smithi122 ceph-mon[93869]: pgmap v12124: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 274 KiB/s wr, 22 op/s 2023-03-09T21:01:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:01:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21776.296283 secs 2023-03-09T21:01:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:01:05 smithi181 ceph-mon[94189]: pgmap v12124: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 274 KiB/s wr, 22 op/s 2023-03-09T21:01:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:01:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21776.296283 secs 2023-03-09T21:01:06.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:01:05 smithi038 ceph-mon[92422]: pgmap v12124: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 274 KiB/s wr, 22 op/s 2023-03-09T21:01:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:01:08 smithi122 ceph-mon[93869]: pgmap v12125: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 347 KiB/s wr, 27 op/s 2023-03-09T21:01:08.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:01:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21778.620538 secs 2023-03-09T21:01:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:01:08 smithi181 ceph-mon[94189]: pgmap v12125: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 347 KiB/s wr, 27 op/s 2023-03-09T21:01:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:01:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21778.620538 secs 2023-03-09T21:01:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:01:08 smithi038 ceph-mon[92422]: pgmap v12125: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 347 KiB/s wr, 27 op/s 2023-03-09T21:01:08.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:01:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21778.620538 secs 2023-03-09T21:01:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22240.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:01:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:01:10 smithi122 ceph-mon[93869]: pgmap v12126: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 374 KiB/s wr, 25 op/s 2023-03-09T21:01:10.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:01:10 smithi122 ceph-mon[93869]: osdmap e971: 12 total, 12 up, 12 in 2023-03-09T21:01:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:01:10 smithi181 ceph-mon[94189]: pgmap v12126: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 374 KiB/s wr, 25 op/s 2023-03-09T21:01:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:01:10 smithi181 ceph-mon[94189]: osdmap e971: 12 total, 12 up, 12 in 2023-03-09T21:01:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:01:10 smithi038 ceph-mon[92422]: pgmap v12126: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 374 KiB/s wr, 25 op/s 2023-03-09T21:01:10.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:01:10 smithi038 ceph-mon[92422]: osdmap e971: 12 total, 12 up, 12 in 2023-03-09T21:01:11.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:01:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21781.296342 secs 2023-03-09T21:01:11.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:01:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21781.296342 secs 2023-03-09T21:01:11.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:01:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21781.296342 secs 2023-03-09T21:01:12.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:01:12 smithi122 ceph-mon[93869]: pgmap v12128: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 153 KiB/s wr, 26 op/s 2023-03-09T21:01:12.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:01:12 smithi181 ceph-mon[94189]: pgmap v12128: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 153 KiB/s wr, 26 op/s 2023-03-09T21:01:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:01:12 smithi038 ceph-mon[92422]: pgmap v12128: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 153 KiB/s wr, 26 op/s 2023-03-09T21:01:13.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:01:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21783.620634 secs 2023-03-09T21:01:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:01:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21783.620634 secs 2023-03-09T21:01:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:01:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21783.620634 secs 2023-03-09T21:01:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22245.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:01:14.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:01:14 smithi122 ceph-mon[93869]: pgmap v12129: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 126 KiB/s wr, 11 op/s 2023-03-09T21:01:14.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:01:14 smithi181 ceph-mon[94189]: pgmap v12129: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 126 KiB/s wr, 11 op/s 2023-03-09T21:01:14.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:01:14 smithi038 ceph-mon[92422]: pgmap v12129: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 126 KiB/s wr, 11 op/s 2023-03-09T21:01:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:01:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21786.296433 secs 2023-03-09T21:01:15.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:01:15 smithi122 ceph-mon[93869]: pgmap v12130: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 125 KiB/s wr, 10 op/s 2023-03-09T21:01:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:01:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21786.296433 secs 2023-03-09T21:01:15.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:01:15 smithi181 ceph-mon[94189]: pgmap v12130: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 125 KiB/s wr, 10 op/s 2023-03-09T21:01:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:01:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21786.296433 secs 2023-03-09T21:01:16.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:01:15 smithi038 ceph-mon[92422]: pgmap v12130: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 125 KiB/s wr, 10 op/s 2023-03-09T21:01:16.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:01:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:01:16.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:01:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:01:16.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:01:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:01:16.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:01:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:01:17.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:01:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:01:17.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:01:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:01:17.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:01:17 smithi122 ceph-mon[93869]: pgmap v12131: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 34 KiB/s wr, 8 op/s 2023-03-09T21:01:17.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:01:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21788.620757 secs 2023-03-09T21:01:17.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:01:17 smithi181 ceph-mon[94189]: pgmap v12131: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 34 KiB/s wr, 8 op/s 2023-03-09T21:01:17.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:01:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21788.620757 secs 2023-03-09T21:01:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:01:17 smithi038 ceph-mon[92422]: pgmap v12131: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 34 KiB/s wr, 8 op/s 2023-03-09T21:01:18.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:01:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21788.620757 secs 2023-03-09T21:01:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22250.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:01:19.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:01:19 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:01:19.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:01:19 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:01:19.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:01:19 smithi122 ceph-mon[93869]: pgmap v12132: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 818 B/s wr, 7 op/s 2023-03-09T21:01:19.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:01:19 smithi122 ceph-mon[93869]: osdmap e972: 12 total, 12 up, 12 in 2023-03-09T21:01:19.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:01:19 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:01:19.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:01:19 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:01:19.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:01:19 smithi181 ceph-mon[94189]: pgmap v12132: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 818 B/s wr, 7 op/s 2023-03-09T21:01:19.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:01:19 smithi181 ceph-mon[94189]: osdmap e972: 12 total, 12 up, 12 in 2023-03-09T21:01:20.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:01:19 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:01:20.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:01:19 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:01:20.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:01:19 smithi038 ceph-mon[92422]: pgmap v12132: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 818 B/s wr, 7 op/s 2023-03-09T21:01:20.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:01:19 smithi038 ceph-mon[92422]: osdmap e972: 12 total, 12 up, 12 in 2023-03-09T21:01:20.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:01:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21791.296517 secs 2023-03-09T21:01:20.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:01:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21791.296517 secs 2023-03-09T21:01:21.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:01:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21791.296517 secs 2023-03-09T21:01:21.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:01:21 smithi122 ceph-mon[93869]: pgmap v12134: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 716 B/s wr, 7 op/s 2023-03-09T21:01:21.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:01:21 smithi181 ceph-mon[94189]: pgmap v12134: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 716 B/s wr, 7 op/s 2023-03-09T21:01:22.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:01:21 smithi038 ceph-mon[92422]: pgmap v12134: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 716 B/s wr, 7 op/s 2023-03-09T21:01:22.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:01:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21793.620894 secs 2023-03-09T21:01:22.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:01:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21793.620894 secs 2023-03-09T21:01:23.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:01:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21793.620894 secs 2023-03-09T21:01:23.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22255.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:01:23.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:01:23 smithi122 ceph-mon[93869]: pgmap v12135: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 716 B/s wr, 7 op/s 2023-03-09T21:01:23.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:01:23 smithi181 ceph-mon[94189]: pgmap v12135: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 716 B/s wr, 7 op/s 2023-03-09T21:01:24.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:01:23 smithi038 ceph-mon[92422]: pgmap v12135: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 716 B/s wr, 7 op/s 2023-03-09T21:01:25.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:01:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21796.296577 secs 2023-03-09T21:01:25.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:01:25 smithi122 ceph-mon[93869]: pgmap v12136: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 716 B/s wr, 7 op/s 2023-03-09T21:01:25.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:01:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21796.296577 secs 2023-03-09T21:01:25.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:01:25 smithi181 ceph-mon[94189]: pgmap v12136: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 716 B/s wr, 7 op/s 2023-03-09T21:01:26.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:01:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21796.296577 secs 2023-03-09T21:01:26.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:01:25 smithi038 ceph-mon[92422]: pgmap v12136: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 716 B/s wr, 7 op/s 2023-03-09T21:01:28.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:01:28 smithi122 ceph-mon[93869]: pgmap v12137: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T21:01:28.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:01:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21798.621013 secs 2023-03-09T21:01:28.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:01:28 smithi181 ceph-mon[94189]: pgmap v12137: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T21:01:28.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:01:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21798.621013 secs 2023-03-09T21:01:28.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:01:28 smithi038 ceph-mon[92422]: pgmap v12137: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T21:01:28.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:01:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21798.621013 secs 2023-03-09T21:01:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22260.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:01:30.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:01:30 smithi122 ceph-mon[93869]: pgmap v12138: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T21:01:30.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:01:30 smithi181 ceph-mon[94189]: pgmap v12138: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T21:01:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:01:30 smithi038 ceph-mon[92422]: pgmap v12138: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T21:01:31.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:01:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21801.296639 secs 2023-03-09T21:01:31.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:01:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21801.296639 secs 2023-03-09T21:01:31.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:01:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21801.296639 secs 2023-03-09T21:01:32.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:01:32 smithi122 ceph-mon[93869]: pgmap v12139: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 0 B/s wr, 8 op/s 2023-03-09T21:01:32.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:01:32 smithi181 ceph-mon[94189]: pgmap v12139: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 0 B/s wr, 8 op/s 2023-03-09T21:01:32.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:01:32 smithi038 ceph-mon[92422]: pgmap v12139: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 0 B/s wr, 8 op/s 2023-03-09T21:01:33.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:01:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21803.621081 secs 2023-03-09T21:01:33.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:01:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21803.621081 secs 2023-03-09T21:01:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:01:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21803.621081 secs 2023-03-09T21:01:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22265.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:01:34.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:01:34 smithi122 ceph-mon[93869]: pgmap v12140: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-09T21:01:34.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:01:34 smithi181 ceph-mon[94189]: pgmap v12140: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-09T21:01:34.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:01:34 smithi038 ceph-mon[92422]: pgmap v12140: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-09T21:01:35.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:01:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21806.296709 secs 2023-03-09T21:01:35.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:01:35 smithi122 ceph-mon[93869]: pgmap v12141: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T21:01:35.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:01:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21806.296709 secs 2023-03-09T21:01:35.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:01:35 smithi181 ceph-mon[94189]: pgmap v12141: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T21:01:36.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:01:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21806.296709 secs 2023-03-09T21:01:36.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:01:35 smithi038 ceph-mon[92422]: pgmap v12141: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T21:01:38.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:01:38 smithi122 ceph-mon[93869]: pgmap v12142: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 8 op/s 2023-03-09T21:01:38.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:01:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21808.621231 secs 2023-03-09T21:01:38.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:01:38 smithi181 ceph-mon[94189]: pgmap v12142: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 8 op/s 2023-03-09T21:01:38.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:01:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21808.621231 secs 2023-03-09T21:01:38.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:01:38 smithi038 ceph-mon[92422]: pgmap v12142: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 8 op/s 2023-03-09T21:01:38.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:01:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21808.621231 secs 2023-03-09T21:01:38.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22270.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:01:40.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:01:40 smithi122 ceph-mon[93869]: pgmap v12143: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 18 KiB/s wr, 7 op/s 2023-03-09T21:01:40.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:01:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:01:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:01:40 smithi181 ceph-mon[94189]: pgmap v12143: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 18 KiB/s wr, 7 op/s 2023-03-09T21:01:40.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:01:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:01:40.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:01:40 smithi038 ceph-mon[92422]: pgmap v12143: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 18 KiB/s wr, 7 op/s 2023-03-09T21:01:40.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:01:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:01:41.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:01:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21811.296769 secs 2023-03-09T21:01:41.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:01:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:01:41.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:01:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:01:41.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:01:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:01:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:01:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21811.296769 secs 2023-03-09T21:01:41.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:01:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:01:41.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:01:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:01:41.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:01:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:01:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:01:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21811.296769 secs 2023-03-09T21:01:41.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:01:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:01:41.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:01:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:01:41.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:01:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:01:42.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:01:42 smithi122 ceph-mon[93869]: pgmap v12144: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 306 KiB/s wr, 9 op/s 2023-03-09T21:01:42.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:01:42 smithi181 ceph-mon[94189]: pgmap v12144: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 306 KiB/s wr, 9 op/s 2023-03-09T21:01:42.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:01:42 smithi038 ceph-mon[92422]: pgmap v12144: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 306 KiB/s wr, 9 op/s 2023-03-09T21:01:43.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:01:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21813.621300 secs 2023-03-09T21:01:43.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:01:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21813.621300 secs 2023-03-09T21:01:43.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:01:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21813.621300 secs 2023-03-09T21:01:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22275.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:01:44.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:01:44 smithi122 ceph-mon[93869]: pgmap v12145: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 306 KiB/s wr, 7 op/s 2023-03-09T21:01:44.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:01:44 smithi181 ceph-mon[94189]: pgmap v12145: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 306 KiB/s wr, 7 op/s 2023-03-09T21:01:44.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:01:44 smithi038 ceph-mon[92422]: pgmap v12145: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 306 KiB/s wr, 7 op/s 2023-03-09T21:01:45.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:01:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21816.296828 secs 2023-03-09T21:01:45.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:01:45 smithi122 ceph-mon[93869]: pgmap v12146: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 323 KiB/s wr, 8 op/s 2023-03-09T21:01:45.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:01:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21816.296828 secs 2023-03-09T21:01:45.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:01:45 smithi181 ceph-mon[94189]: pgmap v12146: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 323 KiB/s wr, 8 op/s 2023-03-09T21:01:46.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:01:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21816.296828 secs 2023-03-09T21:01:46.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:01:45 smithi038 ceph-mon[92422]: pgmap v12146: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 323 KiB/s wr, 8 op/s 2023-03-09T21:01:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:01:48 smithi122 ceph-mon[93869]: pgmap v12147: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 779 KiB/s rd, 413 KiB/s wr, 11 op/s 2023-03-09T21:01:48.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:01:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21818.621451 secs 2023-03-09T21:01:48.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:01:48 smithi181 ceph-mon[94189]: pgmap v12147: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 779 KiB/s rd, 413 KiB/s wr, 11 op/s 2023-03-09T21:01:48.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:01:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21818.621451 secs 2023-03-09T21:01:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:01:48 smithi038 ceph-mon[92422]: pgmap v12147: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 779 KiB/s rd, 413 KiB/s wr, 11 op/s 2023-03-09T21:01:48.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:01:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21818.621451 secs 2023-03-09T21:01:48.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22280.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:01:50.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:01:50 smithi122 ceph-mon[93869]: pgmap v12148: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 796 KiB/s rd, 431 KiB/s wr, 10 op/s 2023-03-09T21:01:50.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:01:50 smithi181 ceph-mon[94189]: pgmap v12148: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 796 KiB/s rd, 431 KiB/s wr, 10 op/s 2023-03-09T21:01:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:01:50 smithi038 ceph-mon[92422]: pgmap v12148: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 796 KiB/s rd, 431 KiB/s wr, 10 op/s 2023-03-09T21:01:51.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:01:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21821.296889 secs 2023-03-09T21:01:51.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:01:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21821.296889 secs 2023-03-09T21:01:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:01:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21821.296889 secs 2023-03-09T21:01:52.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:01:52 smithi122 ceph-mon[93869]: pgmap v12149: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 666 KiB/s wr, 13 op/s 2023-03-09T21:01:52.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:01:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:01:52.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:01:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:01:52.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:01:52 smithi181 ceph-mon[94189]: pgmap v12149: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 666 KiB/s wr, 13 op/s 2023-03-09T21:01:52.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:01:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:01:52.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:01:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:01:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:01:52 smithi038 ceph-mon[92422]: pgmap v12149: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 666 KiB/s wr, 13 op/s 2023-03-09T21:01:52.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:01:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:01:52.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:01:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:01:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:01:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21823.621568 secs 2023-03-09T21:01:53.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:01:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:01:53.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:01:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:01:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:01:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21823.621568 secs 2023-03-09T21:01:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:01:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:01:53.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:01:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:01:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:01:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21823.621568 secs 2023-03-09T21:01:53.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:01:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:01:53.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:01:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:01:53.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22285.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:01:54.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:01:54 smithi122 ceph-mon[93869]: pgmap v12150: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 404 KiB/s wr, 11 op/s 2023-03-09T21:01:54.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:01:54 smithi181 ceph-mon[94189]: pgmap v12150: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 404 KiB/s wr, 11 op/s 2023-03-09T21:01:54.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:01:54 smithi038 ceph-mon[92422]: pgmap v12150: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 404 KiB/s wr, 11 op/s 2023-03-09T21:01:56.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:01:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21826.296951 secs 2023-03-09T21:01:56.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:01:56 smithi181 ceph-mon[94189]: pgmap v12151: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 805 KiB/s rd, 424 KiB/s wr, 11 op/s 2023-03-09T21:01:56.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:01:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:01:56.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:01:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:01:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:01:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21826.296951 secs 2023-03-09T21:01:56.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:01:56 smithi038 ceph-mon[92422]: pgmap v12151: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 805 KiB/s rd, 424 KiB/s wr, 11 op/s 2023-03-09T21:01:56.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:01:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:01:56.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:01:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:01:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:01:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21826.296951 secs 2023-03-09T21:01:56.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:01:56 smithi122 ceph-mon[93869]: pgmap v12151: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 805 KiB/s rd, 424 KiB/s wr, 11 op/s 2023-03-09T21:01:56.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:01:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:01:56.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:01:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:01:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:01:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:01:57.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:01:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:01:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:01:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:01:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:01:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:01:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:01:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:01:57.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:01:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:01:58.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:01:58 smithi122 ceph-mon[93869]: pgmap v12152: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 883 KiB/s rd, 512 KiB/s wr, 14 op/s 2023-03-09T21:01:58.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:01:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21828.621696 secs 2023-03-09T21:01:58.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:01:58 smithi181 ceph-mon[94189]: pgmap v12152: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 883 KiB/s rd, 512 KiB/s wr, 14 op/s 2023-03-09T21:01:58.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:01:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21828.621696 secs 2023-03-09T21:01:58.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:01:58 smithi038 ceph-mon[92422]: pgmap v12152: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 883 KiB/s rd, 512 KiB/s wr, 14 op/s 2023-03-09T21:01:58.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:01:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21828.621696 secs 2023-03-09T21:01:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22290.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:02:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:00 smithi122 ceph-mon[93869]: pgmap v12153: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 455 KiB/s rd, 437 KiB/s wr, 12 op/s 2023-03-09T21:02:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:00 smithi181 ceph-mon[94189]: pgmap v12153: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 455 KiB/s rd, 437 KiB/s wr, 12 op/s 2023-03-09T21:02:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:00 smithi038 ceph-mon[92422]: pgmap v12153: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 455 KiB/s rd, 437 KiB/s wr, 12 op/s 2023-03-09T21:02:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21831.297011 secs 2023-03-09T21:02:01.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21831.297011 secs 2023-03-09T21:02:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21831.297011 secs 2023-03-09T21:02:02.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:02 smithi122 ceph-mon[93869]: pgmap v12154: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 442 KiB/s wr, 14 op/s 2023-03-09T21:02:02.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:02 smithi181 ceph-mon[94189]: pgmap v12154: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 442 KiB/s wr, 14 op/s 2023-03-09T21:02:02.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:02 smithi038 ceph-mon[92422]: pgmap v12154: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 442 KiB/s wr, 14 op/s 2023-03-09T21:02:03.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21833.621813 secs 2023-03-09T21:02:03.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21833.621813 secs 2023-03-09T21:02:03.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21833.621813 secs 2023-03-09T21:02:03.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22295.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:02:04.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:04 smithi122 ceph-mon[93869]: pgmap v12155: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 325 KiB/s wr, 24 op/s 2023-03-09T21:02:04.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:04 smithi181 ceph-mon[94189]: pgmap v12155: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 325 KiB/s wr, 24 op/s 2023-03-09T21:02:04.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:04 smithi038 ceph-mon[92422]: pgmap v12155: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 325 KiB/s wr, 24 op/s 2023-03-09T21:02:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21836.297074 secs 2023-03-09T21:02:05.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:05 smithi122 ceph-mon[93869]: pgmap v12156: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 283 KiB/s rd, 299 KiB/s wr, 24 op/s 2023-03-09T21:02:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21836.297074 secs 2023-03-09T21:02:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:05 smithi181 ceph-mon[94189]: pgmap v12156: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 283 KiB/s rd, 299 KiB/s wr, 24 op/s 2023-03-09T21:02:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21836.297074 secs 2023-03-09T21:02:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:05 smithi038 ceph-mon[92422]: pgmap v12156: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 283 KiB/s rd, 299 KiB/s wr, 24 op/s 2023-03-09T21:02:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:08 smithi122 ceph-mon[93869]: pgmap v12157: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 362 KiB/s wr, 26 op/s 2023-03-09T21:02:08.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21838.621935 secs 2023-03-09T21:02:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:08 smithi181 ceph-mon[94189]: pgmap v12157: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 362 KiB/s wr, 26 op/s 2023-03-09T21:02:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21838.621935 secs 2023-03-09T21:02:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:08 smithi038 ceph-mon[92422]: pgmap v12157: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 362 KiB/s wr, 26 op/s 2023-03-09T21:02:08.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21838.621935 secs 2023-03-09T21:02:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22300.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:02:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:10 smithi122 ceph-mon[93869]: pgmap v12158: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 256 KiB/s wr, 22 op/s 2023-03-09T21:02:10.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:10 smithi122 ceph-mon[93869]: osdmap e973: 12 total, 12 up, 12 in 2023-03-09T21:02:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:10 smithi181 ceph-mon[94189]: pgmap v12158: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 256 KiB/s wr, 22 op/s 2023-03-09T21:02:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:10 smithi181 ceph-mon[94189]: osdmap e973: 12 total, 12 up, 12 in 2023-03-09T21:02:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:10 smithi038 ceph-mon[92422]: pgmap v12158: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 256 KiB/s wr, 22 op/s 2023-03-09T21:02:10.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:10 smithi038 ceph-mon[92422]: osdmap e973: 12 total, 12 up, 12 in 2023-03-09T21:02:11.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21841.297134 secs 2023-03-09T21:02:11.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21841.297134 secs 2023-03-09T21:02:11.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21841.297134 secs 2023-03-09T21:02:12.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:12 smithi122 ceph-mon[93869]: pgmap v12160: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 247 KiB/s rd, 262 KiB/s wr, 22 op/s 2023-03-09T21:02:12.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:12 smithi181 ceph-mon[94189]: pgmap v12160: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 247 KiB/s rd, 262 KiB/s wr, 22 op/s 2023-03-09T21:02:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:12 smithi038 ceph-mon[92422]: pgmap v12160: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 247 KiB/s rd, 262 KiB/s wr, 22 op/s 2023-03-09T21:02:13.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21843.622033 secs 2023-03-09T21:02:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21843.622033 secs 2023-03-09T21:02:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21843.622033 secs 2023-03-09T21:02:13.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22305.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:02:14.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:14 smithi038 ceph-mon[92422]: pgmap v12161: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 99 KiB/s wr, 7 op/s 2023-03-09T21:02:14.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:14 smithi122 ceph-mon[93869]: pgmap v12161: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 99 KiB/s wr, 7 op/s 2023-03-09T21:02:14.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:14 smithi181 ceph-mon[94189]: pgmap v12161: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 99 KiB/s wr, 7 op/s 2023-03-09T21:02:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21846.297193 secs 2023-03-09T21:02:15.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:15 smithi122 ceph-mon[93869]: pgmap v12162: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 99 KiB/s wr, 6 op/s 2023-03-09T21:02:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21846.297193 secs 2023-03-09T21:02:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:15 smithi181 ceph-mon[94189]: pgmap v12162: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 99 KiB/s wr, 6 op/s 2023-03-09T21:02:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21846.297193 secs 2023-03-09T21:02:16.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:15 smithi038 ceph-mon[92422]: pgmap v12162: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 99 KiB/s wr, 6 op/s 2023-03-09T21:02:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:18 smithi122 ceph-mon[93869]: pgmap v12163: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2023-03-09T21:02:18.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21848.622173 secs 2023-03-09T21:02:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:18 smithi181 ceph-mon[94189]: pgmap v12163: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2023-03-09T21:02:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21848.622173 secs 2023-03-09T21:02:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:18 smithi038 ceph-mon[92422]: pgmap v12163: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2023-03-09T21:02:18.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21848.622173 secs 2023-03-09T21:02:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22310.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:02:20.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:20 smithi122 ceph-mon[93869]: pgmap v12164: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2023-03-09T21:02:20.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:20 smithi122 ceph-mon[93869]: osdmap e974: 12 total, 12 up, 12 in 2023-03-09T21:02:20.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:20 smithi181 ceph-mon[94189]: pgmap v12164: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2023-03-09T21:02:20.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:20 smithi181 ceph-mon[94189]: osdmap e974: 12 total, 12 up, 12 in 2023-03-09T21:02:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:20 smithi038 ceph-mon[92422]: pgmap v12164: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2023-03-09T21:02:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:20 smithi038 ceph-mon[92422]: osdmap e974: 12 total, 12 up, 12 in 2023-03-09T21:02:21.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21851.297254 secs 2023-03-09T21:02:21.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21851.297254 secs 2023-03-09T21:02:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21851.297254 secs 2023-03-09T21:02:22.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:22 smithi122 ceph-mon[93869]: pgmap v12166: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T21:02:22.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:22 smithi181 ceph-mon[94189]: pgmap v12166: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T21:02:22.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:22 smithi038 ceph-mon[92422]: pgmap v12166: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T21:02:23.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21853.622290 secs 2023-03-09T21:02:23.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21853.622290 secs 2023-03-09T21:02:23.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21853.622290 secs 2023-03-09T21:02:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22315.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:02:24.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:24 smithi122 ceph-mon[93869]: pgmap v12167: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T21:02:24.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:24 smithi181 ceph-mon[94189]: pgmap v12167: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T21:02:24.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:24 smithi038 ceph-mon[92422]: pgmap v12167: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T21:02:25.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21856.297313 secs 2023-03-09T21:02:25.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:25 smithi122 ceph-mon[93869]: pgmap v12168: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T21:02:25.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21856.297313 secs 2023-03-09T21:02:25.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:25 smithi181 ceph-mon[94189]: pgmap v12168: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T21:02:26.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21856.297313 secs 2023-03-09T21:02:26.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:25 smithi038 ceph-mon[92422]: pgmap v12168: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T21:02:28.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:28 smithi122 ceph-mon[93869]: pgmap v12169: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-09T21:02:28.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21858.622404 secs 2023-03-09T21:02:28.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:28 smithi181 ceph-mon[94189]: pgmap v12169: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-09T21:02:28.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21858.622404 secs 2023-03-09T21:02:28.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:28 smithi038 ceph-mon[92422]: pgmap v12169: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-09T21:02:28.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21858.622404 secs 2023-03-09T21:02:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22320.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:02:30.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:30 smithi122 ceph-mon[93869]: pgmap v12170: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T21:02:30.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:30 smithi181 ceph-mon[94189]: pgmap v12170: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T21:02:30.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:30 smithi038 ceph-mon[92422]: pgmap v12170: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T21:02:31.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21861.297374 secs 2023-03-09T21:02:31.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21861.297374 secs 2023-03-09T21:02:31.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21861.297374 secs 2023-03-09T21:02:32.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:32 smithi122 ceph-mon[93869]: pgmap v12171: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 534 B/s wr, 7 op/s 2023-03-09T21:02:32.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:32 smithi181 ceph-mon[94189]: pgmap v12171: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 534 B/s wr, 7 op/s 2023-03-09T21:02:32.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:32 smithi038 ceph-mon[92422]: pgmap v12171: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 534 B/s wr, 7 op/s 2023-03-09T21:02:33.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21863.622528 secs 2023-03-09T21:02:33.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21863.622528 secs 2023-03-09T21:02:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21863.622528 secs 2023-03-09T21:02:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22325.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:02:34.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:34 smithi122 ceph-mon[93869]: pgmap v12172: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 597 B/s wr, 6 op/s 2023-03-09T21:02:34.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:34 smithi181 ceph-mon[94189]: pgmap v12172: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 597 B/s wr, 6 op/s 2023-03-09T21:02:34.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:34 smithi038 ceph-mon[92422]: pgmap v12172: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 597 B/s wr, 6 op/s 2023-03-09T21:02:35.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21866.297434 secs 2023-03-09T21:02:35.980 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:35 smithi122 ceph-mon[93869]: pgmap v12173: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 597 B/s wr, 6 op/s 2023-03-09T21:02:35.987 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21866.297434 secs 2023-03-09T21:02:35.987 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:35 smithi181 ceph-mon[94189]: pgmap v12173: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 597 B/s wr, 6 op/s 2023-03-09T21:02:36.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21866.297434 secs 2023-03-09T21:02:36.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:35 smithi038 ceph-mon[92422]: pgmap v12173: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 597 B/s wr, 6 op/s 2023-03-09T21:02:38.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:38 smithi122 ceph-mon[93869]: pgmap v12174: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 86 KiB/s wr, 8 op/s 2023-03-09T21:02:38.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21868.622636 secs 2023-03-09T21:02:38.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:38 smithi181 ceph-mon[94189]: pgmap v12174: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 86 KiB/s wr, 8 op/s 2023-03-09T21:02:38.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21868.622636 secs 2023-03-09T21:02:38.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:38 smithi038 ceph-mon[92422]: pgmap v12174: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 86 KiB/s wr, 8 op/s 2023-03-09T21:02:38.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21868.622636 secs 2023-03-09T21:02:38.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22330.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:02:40.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:40 smithi122 ceph-mon[93869]: pgmap v12175: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 87 KiB/s wr, 7 op/s 2023-03-09T21:02:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:40 smithi181 ceph-mon[94189]: pgmap v12175: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 87 KiB/s wr, 7 op/s 2023-03-09T21:02:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:40 smithi038 ceph-mon[92422]: pgmap v12175: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 87 KiB/s wr, 7 op/s 2023-03-09T21:02:41.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21871.297495 secs 2023-03-09T21:02:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:02:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:02:41.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:02:41.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21871.297495 secs 2023-03-09T21:02:41.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:02:41.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:02:41.503 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:02:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21871.297495 secs 2023-03-09T21:02:41.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:02:41.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:02:41.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:02:42.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:42 smithi181 ceph-mon[94189]: pgmap v12176: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 180 KiB/s wr, 8 op/s 2023-03-09T21:02:42.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:02:42.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:02:42.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:02:42.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:42 smithi038 ceph-mon[92422]: pgmap v12176: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 180 KiB/s wr, 8 op/s 2023-03-09T21:02:42.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:02:42.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:02:42.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:02:42.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:42 smithi122 ceph-mon[93869]: pgmap v12176: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 180 KiB/s wr, 8 op/s 2023-03-09T21:02:42.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:02:42.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:02:42.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:02:43.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21873.622702 secs 2023-03-09T21:02:43.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21873.622702 secs 2023-03-09T21:02:43.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21873.622702 secs 2023-03-09T21:02:43.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22335.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:02:44.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:44 smithi038 ceph-mon[92422]: pgmap v12177: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 180 KiB/s wr, 7 op/s 2023-03-09T21:02:44.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:44 smithi122 ceph-mon[93869]: pgmap v12177: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 180 KiB/s wr, 7 op/s 2023-03-09T21:02:44.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:44 smithi181 ceph-mon[94189]: pgmap v12177: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 180 KiB/s wr, 7 op/s 2023-03-09T21:02:45.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21876.297554 secs 2023-03-09T21:02:45.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:45 smithi122 ceph-mon[93869]: pgmap v12178: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 265 KiB/s wr, 7 op/s 2023-03-09T21:02:45.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21876.297554 secs 2023-03-09T21:02:45.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:45 smithi181 ceph-mon[94189]: pgmap v12178: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 265 KiB/s wr, 7 op/s 2023-03-09T21:02:46.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21876.297554 secs 2023-03-09T21:02:46.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:45 smithi038 ceph-mon[92422]: pgmap v12178: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 265 KiB/s wr, 7 op/s 2023-03-09T21:02:47.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:02:47.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:02:47.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:47 smithi122 ceph-mon[93869]: pgmap v12179: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 443 KiB/s wr, 10 op/s 2023-03-09T21:02:47.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21878.622820 secs 2023-03-09T21:02:47.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:02:47.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:02:47.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:47 smithi181 ceph-mon[94189]: pgmap v12179: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 443 KiB/s wr, 10 op/s 2023-03-09T21:02:47.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21878.622820 secs 2023-03-09T21:02:48.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:02:48.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:02:48.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:47 smithi038 ceph-mon[92422]: pgmap v12179: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 443 KiB/s wr, 10 op/s 2023-03-09T21:02:48.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21878.622820 secs 2023-03-09T21:02:48.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22340.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:02:50.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:50 smithi122 ceph-mon[93869]: pgmap v12180: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 621 KiB/s wr, 9 op/s 2023-03-09T21:02:50.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:50 smithi181 ceph-mon[94189]: pgmap v12180: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 621 KiB/s wr, 9 op/s 2023-03-09T21:02:50.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:50 smithi038 ceph-mon[92422]: pgmap v12180: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 621 KiB/s wr, 9 op/s 2023-03-09T21:02:51.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21881.297614 secs 2023-03-09T21:02:51.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21881.297614 secs 2023-03-09T21:02:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21881.297614 secs 2023-03-09T21:02:52.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:52 smithi122 ceph-mon[93869]: pgmap v12181: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 622 KiB/s wr, 11 op/s 2023-03-09T21:02:52.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:02:52.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:02:52.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:52 smithi181 ceph-mon[94189]: pgmap v12181: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 622 KiB/s wr, 11 op/s 2023-03-09T21:02:52.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:02:52.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:02:52.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:52 smithi038 ceph-mon[92422]: pgmap v12181: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 622 KiB/s wr, 11 op/s 2023-03-09T21:02:52.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:02:52.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:02:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:02:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:02:53.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:53 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:02:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:02:53.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:02:53.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:53 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:02:53.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:02:53.762 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:02:53.763 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:53 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:02:53.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22345.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:02:54.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:54 smithi181 ceph-mon[94189]: pgmap v12182: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 549 KiB/s wr, 11 op/s 2023-03-09T21:02:54.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:54 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:02:54.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:54 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:02:54.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:54 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:02:54.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:54 smithi038 ceph-mon[92422]: pgmap v12182: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 549 KiB/s wr, 11 op/s 2023-03-09T21:02:54.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:54 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:02:54.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:54 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:02:54.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:54 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:02:54.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:54 smithi122 ceph-mon[93869]: pgmap v12182: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 549 KiB/s wr, 11 op/s 2023-03-09T21:02:54.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:54 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:02:54.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:54 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:02:54.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:54 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:02:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21886.297673 secs 2023-03-09T21:02:56.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:56 smithi038 ceph-mon[92422]: pgmap v12183: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 592 KiB/s rd, 597 KiB/s wr, 11 op/s 2023-03-09T21:02:56.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:02:56.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:02:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21886.297673 secs 2023-03-09T21:02:56.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:56 smithi122 ceph-mon[93869]: pgmap v12183: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 592 KiB/s rd, 597 KiB/s wr, 11 op/s 2023-03-09T21:02:56.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:02:56.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:02:56.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21886.297673 secs 2023-03-09T21:02:56.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:56 smithi181 ceph-mon[94189]: pgmap v12183: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 592 KiB/s rd, 597 KiB/s wr, 11 op/s 2023-03-09T21:02:56.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:02:56.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:02:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:02:57.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:02:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:02:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:02:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:02:57.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:02:58.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:58 smithi122 ceph-mon[93869]: pgmap v12184: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 876 KiB/s rd, 794 KiB/s wr, 15 op/s 2023-03-09T21:02:58.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:02:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21888.623066 secs 2023-03-09T21:02:58.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:58 smithi181 ceph-mon[94189]: pgmap v12184: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 876 KiB/s rd, 794 KiB/s wr, 15 op/s 2023-03-09T21:02:58.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:02:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21888.623066 secs 2023-03-09T21:02:58.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:58 smithi038 ceph-mon[92422]: pgmap v12184: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 876 KiB/s rd, 794 KiB/s wr, 15 op/s 2023-03-09T21:02:58.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:02:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21888.623066 secs 2023-03-09T21:02:58.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22350.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:03:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:00 smithi122 ceph-mon[93869]: pgmap v12185: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 634 KiB/s wr, 14 op/s 2023-03-09T21:03:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:00 smithi181 ceph-mon[94189]: pgmap v12185: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 634 KiB/s wr, 14 op/s 2023-03-09T21:03:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:00 smithi038 ceph-mon[92422]: pgmap v12185: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 634 KiB/s wr, 14 op/s 2023-03-09T21:03:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21891.297734 secs 2023-03-09T21:03:01.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21891.297734 secs 2023-03-09T21:03:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21891.297734 secs 2023-03-09T21:03:02.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:02 smithi122 ceph-mon[93869]: pgmap v12186: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 638 KiB/s rd, 394 KiB/s wr, 16 op/s 2023-03-09T21:03:02.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:02 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:03:02.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:02 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:03:02.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:02 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:03:02.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:02 smithi181 ceph-mon[94189]: pgmap v12186: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 638 KiB/s rd, 394 KiB/s wr, 16 op/s 2023-03-09T21:03:02.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:02 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:03:02.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:02 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:03:02.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:02 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:03:02.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:02 smithi038 ceph-mon[92422]: pgmap v12186: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 638 KiB/s rd, 394 KiB/s wr, 16 op/s 2023-03-09T21:03:02.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:02 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:03:02.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:02 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:03:02.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:02 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:03:03.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21893.623190 secs 2023-03-09T21:03:03.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21893.623190 secs 2023-03-09T21:03:03.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21893.623190 secs 2023-03-09T21:03:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22355.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:03:04.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:04 smithi122 ceph-mon[93869]: pgmap v12187: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 643 KiB/s rd, 416 KiB/s wr, 26 op/s 2023-03-09T21:03:04.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:04 smithi181 ceph-mon[94189]: pgmap v12187: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 643 KiB/s rd, 416 KiB/s wr, 26 op/s 2023-03-09T21:03:04.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:04 smithi038 ceph-mon[92422]: pgmap v12187: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 643 KiB/s rd, 416 KiB/s wr, 26 op/s 2023-03-09T21:03:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:05 smithi122 ceph-mon[93869]: osdmap e975: 12 total, 12 up, 12 in 2023-03-09T21:03:05.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:05 smithi181 ceph-mon[94189]: osdmap e975: 12 total, 12 up, 12 in 2023-03-09T21:03:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:05 smithi038 ceph-mon[92422]: osdmap e975: 12 total, 12 up, 12 in 2023-03-09T21:03:06.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21896.297796 secs 2023-03-09T21:03:06.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:06 smithi122 ceph-mon[93869]: pgmap v12189: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 486 KiB/s rd, 606 KiB/s wr, 30 op/s 2023-03-09T21:03:06.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21896.297796 secs 2023-03-09T21:03:06.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:06 smithi181 ceph-mon[94189]: pgmap v12189: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 486 KiB/s rd, 606 KiB/s wr, 30 op/s 2023-03-09T21:03:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21896.297796 secs 2023-03-09T21:03:06.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:06 smithi038 ceph-mon[92422]: pgmap v12189: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 486 KiB/s rd, 606 KiB/s wr, 30 op/s 2023-03-09T21:03:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:08 smithi122 ceph-mon[93869]: pgmap v12190: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 312 KiB/s wr, 28 op/s 2023-03-09T21:03:08.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21898.623286 secs 2023-03-09T21:03:08.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:03:08.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:03:08.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:08 smithi181 ceph-mon[94189]: pgmap v12190: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 312 KiB/s wr, 28 op/s 2023-03-09T21:03:08.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21898.623286 secs 2023-03-09T21:03:08.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:03:08.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:03:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:08 smithi038 ceph-mon[92422]: pgmap v12190: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 312 KiB/s wr, 28 op/s 2023-03-09T21:03:08.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21898.623286 secs 2023-03-09T21:03:08.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:03:08.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:03:08.801 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22360.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:03:09.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:03:09.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:03:09.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:03:09.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:03:09.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:03:09.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:03:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:10 smithi122 ceph-mon[93869]: pgmap v12191: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 362 KiB/s wr, 27 op/s 2023-03-09T21:03:10.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:10 smithi122 ceph-mon[93869]: osdmap e976: 12 total, 12 up, 12 in 2023-03-09T21:03:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:10 smithi181 ceph-mon[94189]: pgmap v12191: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 362 KiB/s wr, 27 op/s 2023-03-09T21:03:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:10 smithi181 ceph-mon[94189]: osdmap e976: 12 total, 12 up, 12 in 2023-03-09T21:03:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:10 smithi038 ceph-mon[92422]: pgmap v12191: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 362 KiB/s wr, 27 op/s 2023-03-09T21:03:10.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:10 smithi038 ceph-mon[92422]: osdmap e976: 12 total, 12 up, 12 in 2023-03-09T21:03:11.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21901.297858 secs 2023-03-09T21:03:11.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21901.297858 secs 2023-03-09T21:03:11.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21901.297858 secs 2023-03-09T21:03:12.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:12 smithi122 ceph-mon[93869]: pgmap v12193: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 383 KiB/s wr, 10 op/s 2023-03-09T21:03:12.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:12 smithi181 ceph-mon[94189]: pgmap v12193: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 383 KiB/s wr, 10 op/s 2023-03-09T21:03:12.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:12 smithi038 ceph-mon[92422]: pgmap v12193: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 383 KiB/s wr, 10 op/s 2023-03-09T21:03:13.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21903.623406 secs 2023-03-09T21:03:13.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21903.623406 secs 2023-03-09T21:03:13.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21903.623406 secs 2023-03-09T21:03:13.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22365.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:03:14.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:14 smithi122 ceph-mon[93869]: pgmap v12194: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 132 KiB/s wr, 7 op/s 2023-03-09T21:03:14.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:14 smithi181 ceph-mon[94189]: pgmap v12194: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 132 KiB/s wr, 7 op/s 2023-03-09T21:03:14.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:14 smithi038 ceph-mon[92422]: pgmap v12194: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 132 KiB/s wr, 7 op/s 2023-03-09T21:03:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21906.297942 secs 2023-03-09T21:03:15.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:15 smithi122 ceph-mon[93869]: pgmap v12195: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 117 KiB/s wr, 7 op/s 2023-03-09T21:03:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21906.297942 secs 2023-03-09T21:03:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:15 smithi181 ceph-mon[94189]: pgmap v12195: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 117 KiB/s wr, 7 op/s 2023-03-09T21:03:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21906.297942 secs 2023-03-09T21:03:16.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:15 smithi038 ceph-mon[92422]: pgmap v12195: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 117 KiB/s wr, 7 op/s 2023-03-09T21:03:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:18 smithi122 ceph-mon[93869]: pgmap v12196: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 71 KiB/s wr, 6 op/s 2023-03-09T21:03:18.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21908.623501 secs 2023-03-09T21:03:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:18 smithi181 ceph-mon[94189]: pgmap v12196: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 71 KiB/s wr, 6 op/s 2023-03-09T21:03:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21908.623501 secs 2023-03-09T21:03:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:18 smithi038 ceph-mon[92422]: pgmap v12196: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 71 KiB/s wr, 6 op/s 2023-03-09T21:03:18.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21908.623501 secs 2023-03-09T21:03:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22370.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:03:20.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:20 smithi122 ceph-mon[93869]: pgmap v12197: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 6 op/s 2023-03-09T21:03:20.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:20 smithi181 ceph-mon[94189]: pgmap v12197: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 6 op/s 2023-03-09T21:03:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:20 smithi038 ceph-mon[92422]: pgmap v12197: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 6 op/s 2023-03-09T21:03:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21911.297986 secs 2023-03-09T21:03:21.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21911.297986 secs 2023-03-09T21:03:21.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21911.297986 secs 2023-03-09T21:03:22.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:22 smithi038 ceph-mon[92422]: pgmap v12198: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 7 op/s 2023-03-09T21:03:22.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:22 smithi122 ceph-mon[93869]: pgmap v12198: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 7 op/s 2023-03-09T21:03:22.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:22 smithi181 ceph-mon[94189]: pgmap v12198: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 7 op/s 2023-03-09T21:03:23.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21913.623621 secs 2023-03-09T21:03:23.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21913.623621 secs 2023-03-09T21:03:23.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21913.623621 secs 2023-03-09T21:03:23.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22375.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:03:24.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:24 smithi122 ceph-mon[93869]: pgmap v12199: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 5 op/s 2023-03-09T21:03:24.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:24 smithi181 ceph-mon[94189]: pgmap v12199: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 5 op/s 2023-03-09T21:03:24.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:24 smithi038 ceph-mon[92422]: pgmap v12199: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 5 op/s 2023-03-09T21:03:25.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21916.298071 secs 2023-03-09T21:03:25.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:25 smithi122 ceph-mon[93869]: pgmap v12200: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 5 op/s 2023-03-09T21:03:25.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21916.298071 secs 2023-03-09T21:03:25.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:25 smithi181 ceph-mon[94189]: pgmap v12200: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 5 op/s 2023-03-09T21:03:26.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21916.298071 secs 2023-03-09T21:03:26.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:25 smithi038 ceph-mon[92422]: pgmap v12200: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 5 op/s 2023-03-09T21:03:28.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:28 smithi122 ceph-mon[93869]: pgmap v12201: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-03-09T21:03:28.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21918.623758 secs 2023-03-09T21:03:28.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:28 smithi181 ceph-mon[94189]: pgmap v12201: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-03-09T21:03:28.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21918.623758 secs 2023-03-09T21:03:28.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:28 smithi038 ceph-mon[92422]: pgmap v12201: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-03-09T21:03:28.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21918.623758 secs 2023-03-09T21:03:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22380.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:03:30.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:30 smithi181 ceph-mon[94189]: pgmap v12202: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2023-03-09T21:03:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:30 smithi038 ceph-mon[92422]: pgmap v12202: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2023-03-09T21:03:30.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:30 smithi122 ceph-mon[93869]: pgmap v12202: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2023-03-09T21:03:31.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21921.298132 secs 2023-03-09T21:03:31.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21921.298132 secs 2023-03-09T21:03:31.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21921.298132 secs 2023-03-09T21:03:32.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:32 smithi038 ceph-mon[92422]: pgmap v12203: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 341 B/s wr, 6 op/s 2023-03-09T21:03:32.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:32 smithi122 ceph-mon[93869]: pgmap v12203: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 341 B/s wr, 6 op/s 2023-03-09T21:03:32.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:32 smithi181 ceph-mon[94189]: pgmap v12203: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 341 B/s wr, 6 op/s 2023-03-09T21:03:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21923.623850 secs 2023-03-09T21:03:33.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21923.623850 secs 2023-03-09T21:03:33.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21923.623850 secs 2023-03-09T21:03:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22385.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:03:34.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:34 smithi038 ceph-mon[92422]: pgmap v12204: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2023-03-09T21:03:34.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:34 smithi122 ceph-mon[93869]: pgmap v12204: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2023-03-09T21:03:34.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:34 smithi181 ceph-mon[94189]: pgmap v12204: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2023-03-09T21:03:35.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21926.298191 secs 2023-03-09T21:03:35.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:35 smithi122 ceph-mon[93869]: pgmap v12205: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 87 KiB/s wr, 6 op/s 2023-03-09T21:03:35.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21926.298191 secs 2023-03-09T21:03:35.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:35 smithi181 ceph-mon[94189]: pgmap v12205: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 87 KiB/s wr, 6 op/s 2023-03-09T21:03:36.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21926.298191 secs 2023-03-09T21:03:36.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:35 smithi038 ceph-mon[92422]: pgmap v12205: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 87 KiB/s wr, 6 op/s 2023-03-09T21:03:38.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:38 smithi122 ceph-mon[93869]: pgmap v12206: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 87 KiB/s wr, 8 op/s 2023-03-09T21:03:38.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:38 smithi181 ceph-mon[94189]: pgmap v12206: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 87 KiB/s wr, 8 op/s 2023-03-09T21:03:38.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:38 smithi038 ceph-mon[92422]: pgmap v12206: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 87 KiB/s wr, 8 op/s 2023-03-09T21:03:38.801 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22390.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:03:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:40 smithi122 ceph-mon[93869]: pgmap v12207: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 127 KiB/s wr, 8 op/s 2023-03-09T21:03:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:40 smithi181 ceph-mon[94189]: pgmap v12207: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 127 KiB/s wr, 8 op/s 2023-03-09T21:03:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:40 smithi038 ceph-mon[92422]: pgmap v12207: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 127 KiB/s wr, 8 op/s 2023-03-09T21:03:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21931.298252 secs 2023-03-09T21:03:41.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:03:41.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:03:41.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:41 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:03:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21931.298252 secs 2023-03-09T21:03:41.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:03:41.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:03:41.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:41 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:03:41.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21931.298252 secs 2023-03-09T21:03:41.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:03:41.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:03:41.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:41 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:03:42.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:42 smithi038 ceph-mon[92422]: pgmap v12208: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 126 KiB/s wr, 8 op/s 2023-03-09T21:03:42.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:42 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:03:42.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:42 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:03:42.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:42 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:03:42.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:03:42.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:42 smithi122 ceph-mon[93869]: pgmap v12208: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 126 KiB/s wr, 8 op/s 2023-03-09T21:03:42.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:42 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:03:42.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:42 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:03:42.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:42 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:03:42.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:03:42.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:42 smithi181 ceph-mon[94189]: pgmap v12208: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 126 KiB/s wr, 8 op/s 2023-03-09T21:03:42.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:42 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:03:42.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:42 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:03:42.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:42 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:03:42.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:03:43.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:03:43.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:03:43.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:03:43.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:03:43.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:03:43.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:03:43.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:03:43.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:03:43.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:03:43.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22395.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:03:44.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:44 smithi038 ceph-mon[92422]: pgmap v12209: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 181 KiB/s wr, 7 op/s 2023-03-09T21:03:44.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:44 smithi122 ceph-mon[93869]: pgmap v12209: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 181 KiB/s wr, 7 op/s 2023-03-09T21:03:44.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:44 smithi181 ceph-mon[94189]: pgmap v12209: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 181 KiB/s wr, 7 op/s 2023-03-09T21:03:45.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21936.298309 secs 2023-03-09T21:03:45.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:45 smithi122 ceph-mon[93869]: pgmap v12210: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 199 KiB/s wr, 8 op/s 2023-03-09T21:03:45.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21936.298309 secs 2023-03-09T21:03:45.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:45 smithi181 ceph-mon[94189]: pgmap v12210: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 199 KiB/s wr, 8 op/s 2023-03-09T21:03:46.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21936.298309 secs 2023-03-09T21:03:46.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:45 smithi038 ceph-mon[92422]: pgmap v12210: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 199 KiB/s wr, 8 op/s 2023-03-09T21:03:47.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:03:47.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:03:47.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:47 smithi122 ceph-mon[93869]: pgmap v12211: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 359 KiB/s wr, 9 op/s 2023-03-09T21:03:47.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:03:47.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:03:47.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:47 smithi181 ceph-mon[94189]: pgmap v12211: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 359 KiB/s wr, 9 op/s 2023-03-09T21:03:48.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:03:48.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:03:48.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:47 smithi038 ceph-mon[92422]: pgmap v12211: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 359 KiB/s wr, 9 op/s 2023-03-09T21:03:48.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22400.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:03:50.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:50 smithi122 ceph-mon[93869]: pgmap v12212: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 377 KiB/s wr, 9 op/s 2023-03-09T21:03:50.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:50 smithi181 ceph-mon[94189]: pgmap v12212: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 377 KiB/s wr, 9 op/s 2023-03-09T21:03:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:50 smithi038 ceph-mon[92422]: pgmap v12212: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 377 KiB/s wr, 9 op/s 2023-03-09T21:03:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21941.298384 secs 2023-03-09T21:03:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21941.298384 secs 2023-03-09T21:03:51.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21941.298384 secs 2023-03-09T21:03:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:52 smithi038 ceph-mon[92422]: pgmap v12213: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 451 KiB/s wr, 10 op/s 2023-03-09T21:03:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:52 smithi122 ceph-mon[93869]: pgmap v12213: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 451 KiB/s wr, 10 op/s 2023-03-09T21:03:52.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:52 smithi181 ceph-mon[94189]: pgmap v12213: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 451 KiB/s wr, 10 op/s 2023-03-09T21:03:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21943.624239 secs 2023-03-09T21:03:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21943.624239 secs 2023-03-09T21:03:53.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21943.624239 secs 2023-03-09T21:03:53.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22405.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:03:54.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:54 smithi038 ceph-mon[92422]: pgmap v12214: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 492 KiB/s wr, 10 op/s 2023-03-09T21:03:54.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:54 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:03:54.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:54 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:03:54.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:54 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:03:54.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:54 smithi122 ceph-mon[93869]: pgmap v12214: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 492 KiB/s wr, 10 op/s 2023-03-09T21:03:54.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:54 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:03:54.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:54 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:03:54.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:54 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:03:54.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:54 smithi181 ceph-mon[94189]: pgmap v12214: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 492 KiB/s wr, 10 op/s 2023-03-09T21:03:54.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:54 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:03:54.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:54 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:03:54.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:54 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:03:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21946.298454 secs 2023-03-09T21:03:56.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:56 smithi038 ceph-mon[92422]: pgmap v12215: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 457 KiB/s wr, 10 op/s 2023-03-09T21:03:56.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:03:56.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:03:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21946.298454 secs 2023-03-09T21:03:56.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:56 smithi122 ceph-mon[93869]: pgmap v12215: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 457 KiB/s wr, 10 op/s 2023-03-09T21:03:56.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:03:56.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:03:56.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21946.298454 secs 2023-03-09T21:03:56.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:56 smithi181 ceph-mon[94189]: pgmap v12215: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 457 KiB/s wr, 10 op/s 2023-03-09T21:03:56.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:03:56.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:03:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:03:57.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:03:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:03:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:03:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:03:57.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:03:58.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:58 smithi122 ceph-mon[93869]: pgmap v12216: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 688 KiB/s rd, 668 KiB/s wr, 13 op/s 2023-03-09T21:03:58.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:03:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21948.624346 secs 2023-03-09T21:03:58.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:58 smithi181 ceph-mon[94189]: pgmap v12216: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 688 KiB/s rd, 668 KiB/s wr, 13 op/s 2023-03-09T21:03:58.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:03:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21948.624346 secs 2023-03-09T21:03:58.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:58 smithi038 ceph-mon[92422]: pgmap v12216: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 688 KiB/s rd, 668 KiB/s wr, 13 op/s 2023-03-09T21:03:58.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:03:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21948.624346 secs 2023-03-09T21:03:58.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22410.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:04:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:00 smithi122 ceph-mon[93869]: pgmap v12217: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 476 KiB/s rd, 457 KiB/s wr, 12 op/s 2023-03-09T21:04:00.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:04:00.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:04:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:00 smithi181 ceph-mon[94189]: pgmap v12217: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 476 KiB/s rd, 457 KiB/s wr, 12 op/s 2023-03-09T21:04:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:04:00.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:04:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:00 smithi038 ceph-mon[92422]: pgmap v12217: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 476 KiB/s rd, 457 KiB/s wr, 12 op/s 2023-03-09T21:04:00.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:04:00.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:04:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21951.298513 secs 2023-03-09T21:04:01.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:04:01.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:04:01.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21951.298513 secs 2023-03-09T21:04:01.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:04:01.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:04:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21951.298513 secs 2023-03-09T21:04:01.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:04:01.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:04:02.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:02 smithi122 ceph-mon[93869]: pgmap v12218: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 462 KiB/s wr, 15 op/s 2023-03-09T21:04:02.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:02 smithi181 ceph-mon[94189]: pgmap v12218: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 462 KiB/s wr, 15 op/s 2023-03-09T21:04:02.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:02 smithi038 ceph-mon[92422]: pgmap v12218: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 462 KiB/s wr, 15 op/s 2023-03-09T21:04:03.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21953.624448 secs 2023-03-09T21:04:03.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21953.624448 secs 2023-03-09T21:04:03.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21953.624448 secs 2023-03-09T21:04:03.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22415.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:04:04.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:04 smithi122 ceph-mon[93869]: pgmap v12219: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 370 KiB/s wr, 25 op/s 2023-03-09T21:04:04.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:04 smithi181 ceph-mon[94189]: pgmap v12219: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 370 KiB/s wr, 25 op/s 2023-03-09T21:04:04.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:04 smithi038 ceph-mon[92422]: pgmap v12219: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 370 KiB/s wr, 25 op/s 2023-03-09T21:04:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:05 smithi122 ceph-mon[93869]: osdmap e977: 12 total, 12 up, 12 in 2023-03-09T21:04:05.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:05 smithi181 ceph-mon[94189]: osdmap e977: 12 total, 12 up, 12 in 2023-03-09T21:04:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:05 smithi038 ceph-mon[92422]: osdmap e977: 12 total, 12 up, 12 in 2023-03-09T21:04:06.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21956.298576 secs 2023-03-09T21:04:06.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:06 smithi122 ceph-mon[93869]: pgmap v12221: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 546 KiB/s rd, 565 KiB/s wr, 30 op/s 2023-03-09T21:04:06.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21956.298576 secs 2023-03-09T21:04:06.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:06 smithi181 ceph-mon[94189]: pgmap v12221: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 546 KiB/s rd, 565 KiB/s wr, 30 op/s 2023-03-09T21:04:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21956.298576 secs 2023-03-09T21:04:06.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:06 smithi038 ceph-mon[92422]: pgmap v12221: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 546 KiB/s rd, 565 KiB/s wr, 30 op/s 2023-03-09T21:04:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:08 smithi122 ceph-mon[93869]: pgmap v12222: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 291 KiB/s wr, 29 op/s 2023-03-09T21:04:08.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21958.624563 secs 2023-03-09T21:04:08.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:08 smithi181 ceph-mon[94189]: pgmap v12222: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 291 KiB/s wr, 29 op/s 2023-03-09T21:04:08.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21958.624563 secs 2023-03-09T21:04:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:08 smithi038 ceph-mon[92422]: pgmap v12222: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 291 KiB/s wr, 29 op/s 2023-03-09T21:04:08.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21958.624563 secs 2023-03-09T21:04:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22420.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:04:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:10 smithi122 ceph-mon[93869]: pgmap v12223: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 328 KiB/s rd, 348 KiB/s wr, 28 op/s 2023-03-09T21:04:10.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:10 smithi122 ceph-mon[93869]: osdmap e978: 12 total, 12 up, 12 in 2023-03-09T21:04:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:10 smithi181 ceph-mon[94189]: pgmap v12223: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 328 KiB/s rd, 348 KiB/s wr, 28 op/s 2023-03-09T21:04:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:10 smithi181 ceph-mon[94189]: osdmap e978: 12 total, 12 up, 12 in 2023-03-09T21:04:10.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:10 smithi038 ceph-mon[92422]: pgmap v12223: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 328 KiB/s rd, 348 KiB/s wr, 28 op/s 2023-03-09T21:04:10.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:10 smithi038 ceph-mon[92422]: osdmap e978: 12 total, 12 up, 12 in 2023-03-09T21:04:11.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21961.298637 secs 2023-03-09T21:04:11.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21961.298637 secs 2023-03-09T21:04:11.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21961.298637 secs 2023-03-09T21:04:12.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:12 smithi122 ceph-mon[93869]: pgmap v12225: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 369 KiB/s wr, 12 op/s 2023-03-09T21:04:12.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:12 smithi181 ceph-mon[94189]: pgmap v12225: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 369 KiB/s wr, 12 op/s 2023-03-09T21:04:12.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:12 smithi038 ceph-mon[92422]: pgmap v12225: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 369 KiB/s wr, 12 op/s 2023-03-09T21:04:13.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21963.624666 secs 2023-03-09T21:04:13.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21963.624666 secs 2023-03-09T21:04:13.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21963.624666 secs 2023-03-09T21:04:13.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22425.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:04:14.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:14 smithi122 ceph-mon[93869]: pgmap v12226: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 113 KiB/s wr, 8 op/s 2023-03-09T21:04:14.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:14 smithi181 ceph-mon[94189]: pgmap v12226: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 113 KiB/s wr, 8 op/s 2023-03-09T21:04:14.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:14 smithi038 ceph-mon[92422]: pgmap v12226: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 113 KiB/s wr, 8 op/s 2023-03-09T21:04:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21966.298699 secs 2023-03-09T21:04:15.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:15 smithi122 ceph-mon[93869]: pgmap v12227: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 171 KiB/s wr, 8 op/s 2023-03-09T21:04:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21966.298699 secs 2023-03-09T21:04:15.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:15 smithi181 ceph-mon[94189]: pgmap v12227: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 171 KiB/s wr, 8 op/s 2023-03-09T21:04:16.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21966.298699 secs 2023-03-09T21:04:16.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:15 smithi038 ceph-mon[92422]: pgmap v12227: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 171 KiB/s wr, 8 op/s 2023-03-09T21:04:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:18 smithi038 ceph-mon[92422]: pgmap v12228: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 170 KiB/s wr, 7 op/s 2023-03-09T21:04:18.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21968.624729 secs 2023-03-09T21:04:18.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:18 smithi122 ceph-mon[93869]: pgmap v12228: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 170 KiB/s wr, 7 op/s 2023-03-09T21:04:18.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21968.624729 secs 2023-03-09T21:04:18.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:18 smithi181 ceph-mon[94189]: pgmap v12228: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 170 KiB/s wr, 7 op/s 2023-03-09T21:04:18.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21968.624729 secs 2023-03-09T21:04:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22430.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:04:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:20 smithi038 ceph-mon[92422]: pgmap v12229: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 71 KiB/s wr, 6 op/s 2023-03-09T21:04:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:20 smithi122 ceph-mon[93869]: pgmap v12229: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 71 KiB/s wr, 6 op/s 2023-03-09T21:04:20.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:20 smithi181 ceph-mon[94189]: pgmap v12229: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 71 KiB/s wr, 6 op/s 2023-03-09T21:04:21.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21971.298760 secs 2023-03-09T21:04:21.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21971.298760 secs 2023-03-09T21:04:21.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21971.298760 secs 2023-03-09T21:04:22.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:22 smithi122 ceph-mon[93869]: pgmap v12230: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 62 KiB/s wr, 7 op/s 2023-03-09T21:04:22.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:22 smithi181 ceph-mon[94189]: pgmap v12230: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 62 KiB/s wr, 7 op/s 2023-03-09T21:04:22.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:22 smithi038 ceph-mon[92422]: pgmap v12230: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 62 KiB/s wr, 7 op/s 2023-03-09T21:04:23.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21973.624848 secs 2023-03-09T21:04:23.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21973.624848 secs 2023-03-09T21:04:23.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21973.624848 secs 2023-03-09T21:04:23.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22435.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:04:24.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:24 smithi122 ceph-mon[93869]: pgmap v12231: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 5 op/s 2023-03-09T21:04:24.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:24 smithi181 ceph-mon[94189]: pgmap v12231: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 5 op/s 2023-03-09T21:04:24.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:24 smithi038 ceph-mon[92422]: pgmap v12231: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 5 op/s 2023-03-09T21:04:25.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21976.298818 secs 2023-03-09T21:04:25.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:25 smithi122 ceph-mon[93869]: pgmap v12232: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 6 op/s 2023-03-09T21:04:25.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21976.298818 secs 2023-03-09T21:04:25.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:25 smithi181 ceph-mon[94189]: pgmap v12232: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 6 op/s 2023-03-09T21:04:26.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21976.298818 secs 2023-03-09T21:04:26.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:25 smithi038 ceph-mon[92422]: pgmap v12232: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 6 op/s 2023-03-09T21:04:28.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:28 smithi038 ceph-mon[92422]: pgmap v12233: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T21:04:28.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21978.624967 secs 2023-03-09T21:04:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:28 smithi122 ceph-mon[93869]: pgmap v12233: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T21:04:28.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21978.624967 secs 2023-03-09T21:04:28.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:28 smithi181 ceph-mon[94189]: pgmap v12233: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T21:04:28.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21978.624967 secs 2023-03-09T21:04:28.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22440.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:04:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:30 smithi038 ceph-mon[92422]: pgmap v12234: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 36 KiB/s wr, 6 op/s 2023-03-09T21:04:30.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:30 smithi122 ceph-mon[93869]: pgmap v12234: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 36 KiB/s wr, 6 op/s 2023-03-09T21:04:30.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:30 smithi181 ceph-mon[94189]: pgmap v12234: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 36 KiB/s wr, 6 op/s 2023-03-09T21:04:31.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21981.298903 secs 2023-03-09T21:04:31.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21981.298903 secs 2023-03-09T21:04:31.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21981.298903 secs 2023-03-09T21:04:32.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:32 smithi038 ceph-mon[92422]: pgmap v12235: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 36 KiB/s wr, 7 op/s 2023-03-09T21:04:32.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:32 smithi122 ceph-mon[93869]: pgmap v12235: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 36 KiB/s wr, 7 op/s 2023-03-09T21:04:32.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:32 smithi181 ceph-mon[94189]: pgmap v12235: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 36 KiB/s wr, 7 op/s 2023-03-09T21:04:33.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21983.625115 secs 2023-03-09T21:04:33.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21983.625115 secs 2023-03-09T21:04:33.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21983.625115 secs 2023-03-09T21:04:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22445.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:04:34.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:34 smithi122 ceph-mon[93869]: pgmap v12236: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 36 KiB/s wr, 6 op/s 2023-03-09T21:04:34.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:34 smithi181 ceph-mon[94189]: pgmap v12236: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 36 KiB/s wr, 6 op/s 2023-03-09T21:04:34.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:34 smithi038 ceph-mon[92422]: pgmap v12236: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 36 KiB/s wr, 6 op/s 2023-03-09T21:04:36.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21986.298987 secs 2023-03-09T21:04:36.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:36 smithi122 ceph-mon[93869]: pgmap v12237: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 158 KiB/s wr, 7 op/s 2023-03-09T21:04:36.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:04:36.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:04:36.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21986.298987 secs 2023-03-09T21:04:36.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:36 smithi181 ceph-mon[94189]: pgmap v12237: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 158 KiB/s wr, 7 op/s 2023-03-09T21:04:36.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:04:36.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:04:36.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21986.298987 secs 2023-03-09T21:04:36.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:36 smithi038 ceph-mon[92422]: pgmap v12237: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 158 KiB/s wr, 7 op/s 2023-03-09T21:04:36.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:04:36.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:04:37.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:04:37.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:04:37.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:04:37.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:04:37.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:04:37.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:04:38.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:38 smithi122 ceph-mon[93869]: pgmap v12238: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 158 KiB/s wr, 9 op/s 2023-03-09T21:04:38.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21988.625253 secs 2023-03-09T21:04:38.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:38 smithi181 ceph-mon[94189]: pgmap v12238: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 158 KiB/s wr, 9 op/s 2023-03-09T21:04:38.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21988.625253 secs 2023-03-09T21:04:38.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:38 smithi038 ceph-mon[92422]: pgmap v12238: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 158 KiB/s wr, 9 op/s 2023-03-09T21:04:38.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21988.625253 secs 2023-03-09T21:04:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22450.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:04:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:40 smithi122 ceph-mon[93869]: pgmap v12239: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 158 KiB/s wr, 9 op/s 2023-03-09T21:04:40.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:40 smithi181 ceph-mon[94189]: pgmap v12239: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 158 KiB/s wr, 9 op/s 2023-03-09T21:04:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:40 smithi038 ceph-mon[92422]: pgmap v12239: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 158 KiB/s wr, 9 op/s 2023-03-09T21:04:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21991.299047 secs 2023-03-09T21:04:41.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21991.299047 secs 2023-03-09T21:04:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21991.299047 secs 2023-03-09T21:04:42.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:42 smithi122 ceph-mon[93869]: pgmap v12240: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 122 KiB/s wr, 9 op/s 2023-03-09T21:04:42.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:42 smithi181 ceph-mon[94189]: pgmap v12240: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 122 KiB/s wr, 9 op/s 2023-03-09T21:04:42.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:42 smithi038 ceph-mon[92422]: pgmap v12240: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 122 KiB/s wr, 9 op/s 2023-03-09T21:04:43.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21993.625368 secs 2023-03-09T21:04:43.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:04:43.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:04:43.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:04:43.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:04:43.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21993.625368 secs 2023-03-09T21:04:43.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:04:43.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:04:43.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:04:43.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:04:43.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21993.625368 secs 2023-03-09T21:04:43.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:04:43.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:04:43.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:04:43.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:04:43.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22455.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:04:44.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:44 smithi122 ceph-mon[93869]: pgmap v12241: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 122 KiB/s wr, 7 op/s 2023-03-09T21:04:44.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:44 smithi181 ceph-mon[94189]: pgmap v12241: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 122 KiB/s wr, 7 op/s 2023-03-09T21:04:44.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:44 smithi038 ceph-mon[92422]: pgmap v12241: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 122 KiB/s wr, 7 op/s 2023-03-09T21:04:45.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21996.299105 secs 2023-03-09T21:04:45.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:45 smithi122 ceph-mon[93869]: pgmap v12242: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 482 KiB/s wr, 9 op/s 2023-03-09T21:04:45.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21996.299105 secs 2023-03-09T21:04:45.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:45 smithi181 ceph-mon[94189]: pgmap v12242: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 482 KiB/s wr, 9 op/s 2023-03-09T21:04:46.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21996.299105 secs 2023-03-09T21:04:46.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:45 smithi038 ceph-mon[92422]: pgmap v12242: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 482 KiB/s wr, 9 op/s 2023-03-09T21:04:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:48 smithi038 ceph-mon[92422]: pgmap v12243: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 360 KiB/s wr, 9 op/s 2023-03-09T21:04:48.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 21998.625487 secs 2023-03-09T21:04:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:48 smithi122 ceph-mon[93869]: pgmap v12243: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 360 KiB/s wr, 9 op/s 2023-03-09T21:04:48.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 21998.625487 secs 2023-03-09T21:04:48.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:48 smithi181 ceph-mon[94189]: pgmap v12243: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 360 KiB/s wr, 9 op/s 2023-03-09T21:04:48.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 21998.625487 secs 2023-03-09T21:04:48.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22460.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:04:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:50 smithi122 ceph-mon[93869]: pgmap v12244: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 484 KiB/s wr, 9 op/s 2023-03-09T21:04:50.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:50 smithi181 ceph-mon[94189]: pgmap v12244: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 484 KiB/s wr, 9 op/s 2023-03-09T21:04:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:50 smithi038 ceph-mon[92422]: pgmap v12244: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 484 KiB/s wr, 9 op/s 2023-03-09T21:04:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22001.299166 secs 2023-03-09T21:04:51.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22001.299166 secs 2023-03-09T21:04:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22001.299166 secs 2023-03-09T21:04:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:52 smithi122 ceph-mon[93869]: pgmap v12245: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 484 KiB/s wr, 10 op/s 2023-03-09T21:04:52.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:04:52.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:04:52.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:52 smithi181 ceph-mon[94189]: pgmap v12245: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 484 KiB/s wr, 10 op/s 2023-03-09T21:04:52.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:04:52.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:04:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:52 smithi038 ceph-mon[92422]: pgmap v12245: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 484 KiB/s wr, 10 op/s 2023-03-09T21:04:52.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:04:52.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:04:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:04:53.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:04:53.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:53 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:04:53.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:04:53.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:04:53.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:53 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:04:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:04:53.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:04:53.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:53 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:04:53.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22465.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:04:54.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:54 smithi122 ceph-mon[93869]: pgmap v12246: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 485 KiB/s wr, 9 op/s 2023-03-09T21:04:54.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:54 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:04:54.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:54 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:04:54.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:54 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:04:54.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:54 smithi181 ceph-mon[94189]: pgmap v12246: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 485 KiB/s wr, 9 op/s 2023-03-09T21:04:54.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:54 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:04:54.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:54 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:04:54.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:54 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:04:54.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:54 smithi038 ceph-mon[92422]: pgmap v12246: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 485 KiB/s wr, 9 op/s 2023-03-09T21:04:54.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:54 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:04:54.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:54 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:04:54.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:54 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:04:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22006.299228 secs 2023-03-09T21:04:56.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:56 smithi122 ceph-mon[93869]: pgmap v12247: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 701 KiB/s rd, 619 KiB/s wr, 11 op/s 2023-03-09T21:04:56.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:04:56.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:04:56.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22006.299228 secs 2023-03-09T21:04:56.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:56 smithi181 ceph-mon[94189]: pgmap v12247: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 701 KiB/s rd, 619 KiB/s wr, 11 op/s 2023-03-09T21:04:56.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:04:56.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:04:56.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22006.299228 secs 2023-03-09T21:04:56.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:56 smithi038 ceph-mon[92422]: pgmap v12247: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 701 KiB/s rd, 619 KiB/s wr, 11 op/s 2023-03-09T21:04:56.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:04:56.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:04:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:04:57.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:04:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:04:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:04:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:04:57.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:04:58.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:04:58 smithi122 ceph-mon[93869]: pgmap v12248: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 606 KiB/s rd, 426 KiB/s wr, 13 op/s 2023-03-09T21:04:58.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:04:58 smithi181 ceph-mon[94189]: pgmap v12248: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 606 KiB/s rd, 426 KiB/s wr, 13 op/s 2023-03-09T21:04:58.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:04:58 smithi038 ceph-mon[92422]: pgmap v12248: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 606 KiB/s rd, 426 KiB/s wr, 13 op/s 2023-03-09T21:04:58.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22470.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:05:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:00 smithi122 ceph-mon[93869]: pgmap v12249: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 638 KiB/s rd, 444 KiB/s wr, 13 op/s 2023-03-09T21:05:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:00 smithi181 ceph-mon[94189]: pgmap v12249: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 638 KiB/s rd, 444 KiB/s wr, 13 op/s 2023-03-09T21:05:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:00 smithi038 ceph-mon[92422]: pgmap v12249: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 638 KiB/s rd, 444 KiB/s wr, 13 op/s 2023-03-09T21:05:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22011.299288 secs 2023-03-09T21:05:01.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22011.299288 secs 2023-03-09T21:05:01.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22011.299288 secs 2023-03-09T21:05:02.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:02 smithi122 ceph-mon[93869]: pgmap v12250: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 493 KiB/s rd, 390 KiB/s wr, 15 op/s 2023-03-09T21:05:02.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:02 smithi181 ceph-mon[94189]: pgmap v12250: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 493 KiB/s rd, 390 KiB/s wr, 15 op/s 2023-03-09T21:05:02.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:02 smithi038 ceph-mon[92422]: pgmap v12250: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 493 KiB/s rd, 390 KiB/s wr, 15 op/s 2023-03-09T21:05:03.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22013.625818 secs 2023-03-09T21:05:03.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22013.625818 secs 2023-03-09T21:05:03.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22013.625818 secs 2023-03-09T21:05:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22475.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:05:04.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:04 smithi122 ceph-mon[93869]: pgmap v12251: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 513 KiB/s rd, 427 KiB/s wr, 26 op/s 2023-03-09T21:05:04.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:04 smithi181 ceph-mon[94189]: pgmap v12251: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 513 KiB/s rd, 427 KiB/s wr, 26 op/s 2023-03-09T21:05:04.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:04 smithi038 ceph-mon[92422]: pgmap v12251: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 513 KiB/s rd, 427 KiB/s wr, 26 op/s 2023-03-09T21:05:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:05 smithi122 ceph-mon[93869]: osdmap e979: 12 total, 12 up, 12 in 2023-03-09T21:05:05.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:05 smithi181 ceph-mon[94189]: osdmap e979: 12 total, 12 up, 12 in 2023-03-09T21:05:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:05 smithi038 ceph-mon[92422]: osdmap e979: 12 total, 12 up, 12 in 2023-03-09T21:05:06.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22016.299350 secs 2023-03-09T21:05:06.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:06 smithi122 ceph-mon[93869]: pgmap v12253: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 351 KiB/s wr, 28 op/s 2023-03-09T21:05:06.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:06 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:05:06.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:06 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:05:06.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:05:06.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22016.299350 secs 2023-03-09T21:05:06.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:06 smithi181 ceph-mon[94189]: pgmap v12253: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 351 KiB/s wr, 28 op/s 2023-03-09T21:05:06.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:06 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:05:06.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:06 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:05:06.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:05:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22016.299350 secs 2023-03-09T21:05:06.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:06 smithi038 ceph-mon[92422]: pgmap v12253: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 351 KiB/s wr, 28 op/s 2023-03-09T21:05:06.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:06 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:05:06.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:06 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:05:06.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:05:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22018.625936 secs 2023-03-09T21:05:08.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:08 smithi122 ceph-mon[93869]: pgmap v12254: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 388 KiB/s wr, 29 op/s 2023-03-09T21:05:08.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22018.625936 secs 2023-03-09T21:05:08.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:08 smithi181 ceph-mon[94189]: pgmap v12254: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 388 KiB/s wr, 29 op/s 2023-03-09T21:05:08.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22018.625936 secs 2023-03-09T21:05:08.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:08 smithi038 ceph-mon[92422]: pgmap v12254: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 388 KiB/s wr, 29 op/s 2023-03-09T21:05:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22480.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:05:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:10 smithi122 ceph-mon[93869]: pgmap v12255: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 329 KiB/s rd, 368 KiB/s wr, 28 op/s 2023-03-09T21:05:10.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:10 smithi122 ceph-mon[93869]: osdmap e980: 12 total, 12 up, 12 in 2023-03-09T21:05:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:10 smithi181 ceph-mon[94189]: pgmap v12255: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 329 KiB/s rd, 368 KiB/s wr, 28 op/s 2023-03-09T21:05:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:10 smithi181 ceph-mon[94189]: osdmap e980: 12 total, 12 up, 12 in 2023-03-09T21:05:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:10 smithi038 ceph-mon[92422]: pgmap v12255: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 329 KiB/s rd, 368 KiB/s wr, 28 op/s 2023-03-09T21:05:10.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:10 smithi038 ceph-mon[92422]: osdmap e980: 12 total, 12 up, 12 in 2023-03-09T21:05:11.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22021.299426 secs 2023-03-09T21:05:11.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22021.299426 secs 2023-03-09T21:05:11.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22021.299426 secs 2023-03-09T21:05:12.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:12 smithi122 ceph-mon[93869]: pgmap v12257: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 386 KiB/s wr, 11 op/s 2023-03-09T21:05:12.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:05:12.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:05:12.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:12 smithi181 ceph-mon[94189]: pgmap v12257: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 386 KiB/s wr, 11 op/s 2023-03-09T21:05:12.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:05:12.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:05:12.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:12 smithi038 ceph-mon[92422]: pgmap v12257: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 386 KiB/s wr, 11 op/s 2023-03-09T21:05:12.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:05:12.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:05:13.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22023.626051 secs 2023-03-09T21:05:13.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:05:13.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:05:13.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22023.626051 secs 2023-03-09T21:05:13.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:05:13.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:05:13.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22023.626051 secs 2023-03-09T21:05:13.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:05:13.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:05:13.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22485.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:05:14.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:14 smithi122 ceph-mon[93869]: pgmap v12258: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 350 KiB/s wr, 11 op/s 2023-03-09T21:05:14.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:14 smithi181 ceph-mon[94189]: pgmap v12258: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 350 KiB/s wr, 11 op/s 2023-03-09T21:05:14.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:14 smithi038 ceph-mon[92422]: pgmap v12258: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 350 KiB/s wr, 11 op/s 2023-03-09T21:05:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22026.299486 secs 2023-03-09T21:05:15.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:15 smithi122 ceph-mon[93869]: pgmap v12259: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 309 KiB/s wr, 9 op/s 2023-03-09T21:05:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22026.299486 secs 2023-03-09T21:05:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:15 smithi181 ceph-mon[94189]: pgmap v12259: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 309 KiB/s wr, 9 op/s 2023-03-09T21:05:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22026.299486 secs 2023-03-09T21:05:16.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:15 smithi038 ceph-mon[92422]: pgmap v12259: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 309 KiB/s wr, 9 op/s 2023-03-09T21:05:18.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22028.626168 secs 2023-03-09T21:05:18.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:18 smithi122 ceph-mon[93869]: pgmap v12260: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 8 op/s 2023-03-09T21:05:18.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22028.626168 secs 2023-03-09T21:05:18.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:18 smithi181 ceph-mon[94189]: pgmap v12260: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 8 op/s 2023-03-09T21:05:18.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22028.626168 secs 2023-03-09T21:05:18.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:18 smithi038 ceph-mon[92422]: pgmap v12260: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 8 op/s 2023-03-09T21:05:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22490.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:05:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:20 smithi122 ceph-mon[93869]: pgmap v12261: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 71 KiB/s wr, 7 op/s 2023-03-09T21:05:20.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:20 smithi181 ceph-mon[94189]: pgmap v12261: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 71 KiB/s wr, 7 op/s 2023-03-09T21:05:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:20 smithi038 ceph-mon[92422]: pgmap v12261: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 71 KiB/s wr, 7 op/s 2023-03-09T21:05:21.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22031.299549 secs 2023-03-09T21:05:21.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22031.299549 secs 2023-03-09T21:05:21.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22031.299549 secs 2023-03-09T21:05:22.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:22 smithi122 ceph-mon[93869]: pgmap v12262: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 62 KiB/s wr, 9 op/s 2023-03-09T21:05:22.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:22 smithi181 ceph-mon[94189]: pgmap v12262: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 62 KiB/s wr, 9 op/s 2023-03-09T21:05:22.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:22 smithi038 ceph-mon[92422]: pgmap v12262: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 62 KiB/s wr, 9 op/s 2023-03-09T21:05:23.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22033.626292 secs 2023-03-09T21:05:23.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22033.626292 secs 2023-03-09T21:05:23.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22033.626292 secs 2023-03-09T21:05:23.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22495.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:05:24.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:24 smithi122 ceph-mon[93869]: pgmap v12263: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 170 B/s wr, 6 op/s 2023-03-09T21:05:24.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:24 smithi181 ceph-mon[94189]: pgmap v12263: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 170 B/s wr, 6 op/s 2023-03-09T21:05:24.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:24 smithi038 ceph-mon[92422]: pgmap v12263: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 170 B/s wr, 6 op/s 2023-03-09T21:05:25.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22036.299609 secs 2023-03-09T21:05:25.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:25 smithi122 ceph-mon[93869]: pgmap v12264: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T21:05:25.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22036.299609 secs 2023-03-09T21:05:25.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:25 smithi181 ceph-mon[94189]: pgmap v12264: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T21:05:26.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22036.299609 secs 2023-03-09T21:05:26.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:25 smithi038 ceph-mon[92422]: pgmap v12264: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T21:05:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22038.626416 secs 2023-03-09T21:05:28.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:28 smithi122 ceph-mon[93869]: pgmap v12265: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 8 op/s 2023-03-09T21:05:28.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22038.626416 secs 2023-03-09T21:05:28.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:28 smithi181 ceph-mon[94189]: pgmap v12265: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 8 op/s 2023-03-09T21:05:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22038.626416 secs 2023-03-09T21:05:28.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:28 smithi038 ceph-mon[92422]: pgmap v12265: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 8 op/s 2023-03-09T21:05:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22500.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:05:30.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:30 smithi122 ceph-mon[93869]: pgmap v12266: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 6 op/s 2023-03-09T21:05:30.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:30 smithi181 ceph-mon[94189]: pgmap v12266: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 6 op/s 2023-03-09T21:05:30.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:30 smithi038 ceph-mon[92422]: pgmap v12266: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 6 op/s 2023-03-09T21:05:31.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22041.299668 secs 2023-03-09T21:05:31.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22041.299668 secs 2023-03-09T21:05:31.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22041.299668 secs 2023-03-09T21:05:32.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:32 smithi122 ceph-mon[93869]: pgmap v12267: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 341 B/s wr, 8 op/s 2023-03-09T21:05:32.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:32 smithi181 ceph-mon[94189]: pgmap v12267: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 341 B/s wr, 8 op/s 2023-03-09T21:05:32.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:32 smithi038 ceph-mon[92422]: pgmap v12267: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 341 B/s wr, 8 op/s 2023-03-09T21:05:33.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22043.626484 secs 2023-03-09T21:05:33.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22043.626484 secs 2023-03-09T21:05:33.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22043.626484 secs 2023-03-09T21:05:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22505.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:05:34.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:34 smithi122 ceph-mon[93869]: pgmap v12268: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 7 op/s 2023-03-09T21:05:34.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:34 smithi181 ceph-mon[94189]: pgmap v12268: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 7 op/s 2023-03-09T21:05:34.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:34 smithi038 ceph-mon[92422]: pgmap v12268: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 7 op/s 2023-03-09T21:05:35.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22046.299732 secs 2023-03-09T21:05:35.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:35 smithi122 ceph-mon[93869]: pgmap v12269: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-09T21:05:35.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22046.299732 secs 2023-03-09T21:05:35.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:35 smithi181 ceph-mon[94189]: pgmap v12269: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-09T21:05:36.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22046.299732 secs 2023-03-09T21:05:36.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:35 smithi038 ceph-mon[92422]: pgmap v12269: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-09T21:05:38.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:38 smithi122 ceph-mon[93869]: pgmap v12270: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 87 KiB/s wr, 9 op/s 2023-03-09T21:05:38.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:38 smithi181 ceph-mon[94189]: pgmap v12270: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 87 KiB/s wr, 9 op/s 2023-03-09T21:05:38.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:38 smithi038 ceph-mon[92422]: pgmap v12270: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 87 KiB/s wr, 9 op/s 2023-03-09T21:05:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22510.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:05:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:40 smithi122 ceph-mon[93869]: pgmap v12271: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 87 KiB/s wr, 7 op/s 2023-03-09T21:05:40.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:40 smithi181 ceph-mon[94189]: pgmap v12271: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 87 KiB/s wr, 7 op/s 2023-03-09T21:05:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:40 smithi038 ceph-mon[92422]: pgmap v12271: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 87 KiB/s wr, 7 op/s 2023-03-09T21:05:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22051.299793 secs 2023-03-09T21:05:41.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:41 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:05:41.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22051.299793 secs 2023-03-09T21:05:41.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:41 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:05:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22051.299793 secs 2023-03-09T21:05:41.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:41 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:05:42.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:42 smithi122 ceph-mon[93869]: pgmap v12272: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 247 KiB/s rd, 213 KiB/s wr, 9 op/s 2023-03-09T21:05:42.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:42 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:05:42.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:42 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:05:42.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:42 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:05:42.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:42 smithi181 ceph-mon[94189]: pgmap v12272: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 247 KiB/s rd, 213 KiB/s wr, 9 op/s 2023-03-09T21:05:42.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:42 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:05:42.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:42 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:05:42.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:42 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:05:42.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:42 smithi038 ceph-mon[92422]: pgmap v12272: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 247 KiB/s rd, 213 KiB/s wr, 9 op/s 2023-03-09T21:05:42.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:42 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:05:42.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:42 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:05:42.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:42 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:05:43.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22515.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:05:44.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:44 smithi122 ceph-mon[93869]: pgmap v12273: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 230 KiB/s wr, 7 op/s 2023-03-09T21:05:44.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:44 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:05:44.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:44 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:05:44.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:44 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:05:44.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:44 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:05:44.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:44 smithi181 ceph-mon[94189]: pgmap v12273: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 230 KiB/s wr, 7 op/s 2023-03-09T21:05:44.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:44 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:05:44.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:44 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:05:44.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:44 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:05:44.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:44 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:05:44.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:44 smithi038 ceph-mon[92422]: pgmap v12273: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 230 KiB/s wr, 7 op/s 2023-03-09T21:05:44.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:44 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:05:44.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:44 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:05:44.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:44 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:05:44.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:44 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:05:45.703 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22056.299854 secs 2023-03-09T21:05:45.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22056.299854 secs 2023-03-09T21:05:45.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:45 smithi122 ceph-mon[93869]: pgmap v12274: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 578 KiB/s rd, 534 KiB/s wr, 7 op/s 2023-03-09T21:05:45.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:45 smithi181 ceph-mon[94189]: pgmap v12274: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 578 KiB/s rd, 534 KiB/s wr, 7 op/s 2023-03-09T21:05:46.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22056.299854 secs 2023-03-09T21:05:46.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:45 smithi038 ceph-mon[92422]: pgmap v12274: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 578 KiB/s rd, 534 KiB/s wr, 7 op/s 2023-03-09T21:05:47.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:05:47.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:05:47.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:47 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 22058.626839 secs 2023-03-09T21:05:47.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:47 smithi122 ceph-mon[93869]: pgmap v12275: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 644 KiB/s rd, 534 KiB/s wr, 10 op/s 2023-03-09T21:05:47.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:05:47.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:05:47.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:47 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 22058.626839 secs 2023-03-09T21:05:47.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:47 smithi181 ceph-mon[94189]: pgmap v12275: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 644 KiB/s rd, 534 KiB/s wr, 10 op/s 2023-03-09T21:05:48.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:05:48.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:05:48.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:47 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 22058.626839 secs 2023-03-09T21:05:48.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:47 smithi038 ceph-mon[92422]: pgmap v12275: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 644 KiB/s rd, 534 KiB/s wr, 10 op/s 2023-03-09T21:05:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22520.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:05:48.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:05:48.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:05:48.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:05:48.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:05:49.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:05:49.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:05:49.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:49 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:05:49.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:05:49.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:05:49.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:49 smithi122 ceph-mon[93869]: pgmap v12276: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 686 KiB/s rd, 710 KiB/s wr, 10 op/s 2023-03-09T21:05:49.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:49 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:05:49.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:05:49.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:05:49.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:49 smithi181 ceph-mon[94189]: pgmap v12276: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 686 KiB/s rd, 710 KiB/s wr, 10 op/s 2023-03-09T21:05:50.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:49 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:05:50.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:05:50.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:05:50.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:49 smithi038 ceph-mon[92422]: pgmap v12276: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 686 KiB/s rd, 710 KiB/s wr, 10 op/s 2023-03-09T21:05:51.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22061.299915 secs 2023-03-09T21:05:51.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22061.299915 secs 2023-03-09T21:05:51.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22061.299915 secs 2023-03-09T21:05:52.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:05:52.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:05:52.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:51 smithi038 ceph-mon[92422]: pgmap v12277: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 710 KiB/s wr, 12 op/s 2023-03-09T21:05:52.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:05:52.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:05:52.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:51 smithi122 ceph-mon[93869]: pgmap v12277: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 710 KiB/s wr, 12 op/s 2023-03-09T21:05:52.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:05:52.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:05:52.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:51 smithi181 ceph-mon[94189]: pgmap v12277: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 710 KiB/s wr, 12 op/s 2023-03-09T21:05:53.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:05:53.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:05:53.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:52 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 22063.626958 secs 2023-03-09T21:05:53.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:05:53.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:05:53.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:52 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 22063.626958 secs 2023-03-09T21:05:53.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:05:53.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:05:53.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:52 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 22063.626958 secs 2023-03-09T21:05:53.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22525.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:05:54.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:53 smithi122 ceph-mon[93869]: pgmap v12278: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 813 KiB/s rd, 756 KiB/s wr, 12 op/s 2023-03-09T21:05:54.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:53 smithi181 ceph-mon[94189]: pgmap v12278: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 813 KiB/s rd, 756 KiB/s wr, 12 op/s 2023-03-09T21:05:54.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:53 smithi038 ceph-mon[92422]: pgmap v12278: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 813 KiB/s rd, 756 KiB/s wr, 12 op/s 2023-03-09T21:05:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:05:56.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:05:56.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22066.299999 secs 2023-03-09T21:05:56.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:55 smithi122 ceph-mon[93869]: pgmap v12279: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 796 KiB/s rd, 738 KiB/s wr, 11 op/s 2023-03-09T21:05:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:05:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:05:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22066.299999 secs 2023-03-09T21:05:56.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:55 smithi181 ceph-mon[94189]: pgmap v12279: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 796 KiB/s rd, 738 KiB/s wr, 11 op/s 2023-03-09T21:05:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:05:56.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:05:56.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22066.299999 secs 2023-03-09T21:05:56.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:55 smithi038 ceph-mon[92422]: pgmap v12279: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 796 KiB/s rd, 738 KiB/s wr, 11 op/s 2023-03-09T21:05:57.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:05:57.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:05:57.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:05:57.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:05:57.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:05:57.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:05:58.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22068.627078 secs 2023-03-09T21:05:58.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:05:57 smithi122 ceph-mon[93869]: pgmap v12280: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 656 KiB/s rd, 609 KiB/s wr, 14 op/s 2023-03-09T21:05:58.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22068.627078 secs 2023-03-09T21:05:58.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:05:57 smithi181 ceph-mon[94189]: pgmap v12280: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 656 KiB/s rd, 609 KiB/s wr, 14 op/s 2023-03-09T21:05:58.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22068.627078 secs 2023-03-09T21:05:58.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:05:57 smithi038 ceph-mon[92422]: pgmap v12280: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 656 KiB/s rd, 609 KiB/s wr, 14 op/s 2023-03-09T21:05:58.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22530.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:06:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:00 smithi122 ceph-mon[93869]: pgmap v12281: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 626 KiB/s rd, 626 KiB/s wr, 12 op/s 2023-03-09T21:06:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:00 smithi181 ceph-mon[94189]: pgmap v12281: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 626 KiB/s rd, 626 KiB/s wr, 12 op/s 2023-03-09T21:06:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:00 smithi038 ceph-mon[92422]: pgmap v12281: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 626 KiB/s rd, 626 KiB/s wr, 12 op/s 2023-03-09T21:06:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22071.300059 secs 2023-03-09T21:06:01.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22071.300059 secs 2023-03-09T21:06:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22071.300059 secs 2023-03-09T21:06:02.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:02 smithi122 ceph-mon[93869]: pgmap v12282: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 376 KiB/s wr, 12 op/s 2023-03-09T21:06:02.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:02 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:06:02.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:02 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:06:02.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:02 smithi181 ceph-mon[94189]: pgmap v12282: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 376 KiB/s wr, 12 op/s 2023-03-09T21:06:02.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:02 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:06:02.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:02 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:06:02.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:02 smithi038 ceph-mon[92422]: pgmap v12282: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 376 KiB/s wr, 12 op/s 2023-03-09T21:06:02.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:02 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:06:02.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:02 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:06:03.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22073.627219 secs 2023-03-09T21:06:03.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22073.627219 secs 2023-03-09T21:06:03.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22073.627219 secs 2023-03-09T21:06:03.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22535.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:06:04.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:04 smithi122 ceph-mon[93869]: pgmap v12283: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 505 KiB/s rd, 398 KiB/s wr, 24 op/s 2023-03-09T21:06:04.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:04 smithi181 ceph-mon[94189]: pgmap v12283: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 505 KiB/s rd, 398 KiB/s wr, 24 op/s 2023-03-09T21:06:04.753 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:04 smithi038 ceph-mon[92422]: pgmap v12283: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 505 KiB/s rd, 398 KiB/s wr, 24 op/s 2023-03-09T21:06:05.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:05 smithi122 ceph-mon[93869]: osdmap e981: 12 total, 12 up, 12 in 2023-03-09T21:06:05.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:05 smithi181 ceph-mon[94189]: osdmap e981: 12 total, 12 up, 12 in 2023-03-09T21:06:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:05 smithi038 ceph-mon[92422]: osdmap e981: 12 total, 12 up, 12 in 2023-03-09T21:06:06.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22076.300120 secs 2023-03-09T21:06:06.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:06 smithi122 ceph-mon[93869]: pgmap v12285: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 272 KiB/s wr, 26 op/s 2023-03-09T21:06:06.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22076.300120 secs 2023-03-09T21:06:06.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:06 smithi181 ceph-mon[94189]: pgmap v12285: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 272 KiB/s wr, 26 op/s 2023-03-09T21:06:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22076.300120 secs 2023-03-09T21:06:06.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:06 smithi038 ceph-mon[92422]: pgmap v12285: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 272 KiB/s wr, 26 op/s 2023-03-09T21:06:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22078.627342 secs 2023-03-09T21:06:08.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:08 smithi122 ceph-mon[93869]: pgmap v12286: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 276 KiB/s wr, 26 op/s 2023-03-09T21:06:08.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22078.627342 secs 2023-03-09T21:06:08.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:08 smithi181 ceph-mon[94189]: pgmap v12286: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 276 KiB/s wr, 26 op/s 2023-03-09T21:06:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22078.627342 secs 2023-03-09T21:06:08.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:08 smithi038 ceph-mon[92422]: pgmap v12286: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 276 KiB/s wr, 26 op/s 2023-03-09T21:06:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22540.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:06:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:10 smithi122 ceph-mon[93869]: pgmap v12287: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 277 KiB/s wr, 26 op/s 2023-03-09T21:06:10.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:10 smithi122 ceph-mon[93869]: osdmap e982: 12 total, 12 up, 12 in 2023-03-09T21:06:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:10 smithi181 ceph-mon[94189]: pgmap v12287: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 277 KiB/s wr, 26 op/s 2023-03-09T21:06:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:10 smithi181 ceph-mon[94189]: osdmap e982: 12 total, 12 up, 12 in 2023-03-09T21:06:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:10 smithi038 ceph-mon[92422]: pgmap v12287: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 277 KiB/s wr, 26 op/s 2023-03-09T21:06:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:10 smithi038 ceph-mon[92422]: osdmap e982: 12 total, 12 up, 12 in 2023-03-09T21:06:11.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22081.300180 secs 2023-03-09T21:06:11.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22081.300180 secs 2023-03-09T21:06:11.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22081.300180 secs 2023-03-09T21:06:12.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:12 smithi122 ceph-mon[93869]: pgmap v12289: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 283 KiB/s rd, 296 KiB/s wr, 10 op/s 2023-03-09T21:06:12.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:12 smithi181 ceph-mon[94189]: pgmap v12289: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 283 KiB/s rd, 296 KiB/s wr, 10 op/s 2023-03-09T21:06:12.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:12 smithi038 ceph-mon[92422]: pgmap v12289: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 283 KiB/s rd, 296 KiB/s wr, 10 op/s 2023-03-09T21:06:13.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22083.627464 secs 2023-03-09T21:06:13.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:06:13.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:06:13.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22083.627464 secs 2023-03-09T21:06:13.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:06:13.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:06:13.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22083.627464 secs 2023-03-09T21:06:13.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:06:13.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:06:13.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22545.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:06:14.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:14 smithi122 ceph-mon[93869]: pgmap v12290: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 346 KiB/s wr, 11 op/s 2023-03-09T21:06:14.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:14 smithi181 ceph-mon[94189]: pgmap v12290: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 346 KiB/s wr, 11 op/s 2023-03-09T21:06:14.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:14 smithi038 ceph-mon[92422]: pgmap v12290: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 346 KiB/s wr, 11 op/s 2023-03-09T21:06:16.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22086.300245 secs 2023-03-09T21:06:16.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:16 smithi122 ceph-mon[93869]: pgmap v12291: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 308 KiB/s wr, 10 op/s 2023-03-09T21:06:16.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:06:16.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:06:16.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22086.300245 secs 2023-03-09T21:06:16.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:16 smithi181 ceph-mon[94189]: pgmap v12291: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 308 KiB/s wr, 10 op/s 2023-03-09T21:06:16.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:06:16.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:06:16.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22086.300245 secs 2023-03-09T21:06:16.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:16 smithi038 ceph-mon[92422]: pgmap v12291: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 308 KiB/s wr, 10 op/s 2023-03-09T21:06:16.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:06:16.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:06:18.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22088.627578 secs 2023-03-09T21:06:18.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:18 smithi122 ceph-mon[93869]: pgmap v12292: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 94 KiB/s wr, 7 op/s 2023-03-09T21:06:18.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22088.627578 secs 2023-03-09T21:06:18.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:18 smithi181 ceph-mon[94189]: pgmap v12292: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 94 KiB/s wr, 7 op/s 2023-03-09T21:06:18.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22088.627578 secs 2023-03-09T21:06:18.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:18 smithi038 ceph-mon[92422]: pgmap v12292: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 94 KiB/s wr, 7 op/s 2023-03-09T21:06:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22550.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:06:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:20 smithi122 ceph-mon[93869]: pgmap v12293: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 71 KiB/s wr, 7 op/s 2023-03-09T21:06:20.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:20 smithi181 ceph-mon[94189]: pgmap v12293: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 71 KiB/s wr, 7 op/s 2023-03-09T21:06:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:20 smithi038 ceph-mon[92422]: pgmap v12293: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 71 KiB/s wr, 7 op/s 2023-03-09T21:06:21.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22091.300305 secs 2023-03-09T21:06:21.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22091.300305 secs 2023-03-09T21:06:21.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22091.300305 secs 2023-03-09T21:06:22.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:22 smithi122 ceph-mon[93869]: pgmap v12294: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 62 KiB/s wr, 8 op/s 2023-03-09T21:06:22.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:22 smithi181 ceph-mon[94189]: pgmap v12294: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 62 KiB/s wr, 8 op/s 2023-03-09T21:06:22.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:22 smithi038 ceph-mon[92422]: pgmap v12294: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 62 KiB/s wr, 8 op/s 2023-03-09T21:06:23.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22093.627729 secs 2023-03-09T21:06:23.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22093.627729 secs 2023-03-09T21:06:23.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22093.627729 secs 2023-03-09T21:06:23.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22555.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:06:24.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:24 smithi122 ceph-mon[93869]: pgmap v12295: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 60 KiB/s wr, 7 op/s 2023-03-09T21:06:24.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:24 smithi181 ceph-mon[94189]: pgmap v12295: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 60 KiB/s wr, 7 op/s 2023-03-09T21:06:24.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:24 smithi038 ceph-mon[92422]: pgmap v12295: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 60 KiB/s wr, 7 op/s 2023-03-09T21:06:25.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22096.300380 secs 2023-03-09T21:06:25.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:25 smithi122 ceph-mon[93869]: pgmap v12296: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 5 op/s 2023-03-09T21:06:25.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22096.300380 secs 2023-03-09T21:06:25.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:25 smithi181 ceph-mon[94189]: pgmap v12296: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 5 op/s 2023-03-09T21:06:26.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22096.300380 secs 2023-03-09T21:06:26.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:25 smithi038 ceph-mon[92422]: pgmap v12296: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 5 op/s 2023-03-09T21:06:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22098.627841 secs 2023-03-09T21:06:28.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:28 smithi122 ceph-mon[93869]: pgmap v12297: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 511 B/s wr, 8 op/s 2023-03-09T21:06:28.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22098.627841 secs 2023-03-09T21:06:28.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:28 smithi181 ceph-mon[94189]: pgmap v12297: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 511 B/s wr, 8 op/s 2023-03-09T21:06:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22098.627841 secs 2023-03-09T21:06:28.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:28 smithi038 ceph-mon[92422]: pgmap v12297: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 511 B/s wr, 8 op/s 2023-03-09T21:06:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22560.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:06:30.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:30 smithi122 ceph-mon[93869]: pgmap v12298: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 159 KiB/s wr, 7 op/s 2023-03-09T21:06:30.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:30 smithi181 ceph-mon[94189]: pgmap v12298: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 159 KiB/s wr, 7 op/s 2023-03-09T21:06:30.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:30 smithi038 ceph-mon[92422]: pgmap v12298: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 159 KiB/s wr, 7 op/s 2023-03-09T21:06:31.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22101.300440 secs 2023-03-09T21:06:31.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22101.300440 secs 2023-03-09T21:06:31.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22101.300440 secs 2023-03-09T21:06:32.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:32 smithi122 ceph-mon[93869]: pgmap v12299: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 159 KiB/s wr, 8 op/s 2023-03-09T21:06:32.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:32 smithi181 ceph-mon[94189]: pgmap v12299: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 159 KiB/s wr, 8 op/s 2023-03-09T21:06:32.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:32 smithi038 ceph-mon[92422]: pgmap v12299: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 159 KiB/s wr, 8 op/s 2023-03-09T21:06:33.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22103.627976 secs 2023-03-09T21:06:33.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22103.627976 secs 2023-03-09T21:06:33.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22103.627976 secs 2023-03-09T21:06:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22565.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:06:34.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:34 smithi122 ceph-mon[93869]: pgmap v12300: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 159 KiB/s wr, 7 op/s 2023-03-09T21:06:34.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:34 smithi181 ceph-mon[94189]: pgmap v12300: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 159 KiB/s wr, 7 op/s 2023-03-09T21:06:34.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:34 smithi038 ceph-mon[92422]: pgmap v12300: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 159 KiB/s wr, 7 op/s 2023-03-09T21:06:35.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22106.300502 secs 2023-03-09T21:06:35.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:35 smithi122 ceph-mon[93869]: pgmap v12301: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 159 KiB/s wr, 6 op/s 2023-03-09T21:06:35.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22106.300502 secs 2023-03-09T21:06:35.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:35 smithi181 ceph-mon[94189]: pgmap v12301: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 159 KiB/s wr, 6 op/s 2023-03-09T21:06:36.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22106.300502 secs 2023-03-09T21:06:36.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:35 smithi038 ceph-mon[92422]: pgmap v12301: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 159 KiB/s wr, 6 op/s 2023-03-09T21:06:38.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22108.628152 secs 2023-03-09T21:06:38.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:38 smithi122 ceph-mon[93869]: pgmap v12302: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 159 KiB/s wr, 8 op/s 2023-03-09T21:06:38.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22108.628152 secs 2023-03-09T21:06:38.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:38 smithi181 ceph-mon[94189]: pgmap v12302: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 159 KiB/s wr, 8 op/s 2023-03-09T21:06:38.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22108.628152 secs 2023-03-09T21:06:38.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:38 smithi038 ceph-mon[92422]: pgmap v12302: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 159 KiB/s wr, 8 op/s 2023-03-09T21:06:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22570.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:06:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:40 smithi122 ceph-mon[93869]: pgmap v12303: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 264 KiB/s wr, 7 op/s 2023-03-09T21:06:40.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:40 smithi181 ceph-mon[94189]: pgmap v12303: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 264 KiB/s wr, 7 op/s 2023-03-09T21:06:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:40 smithi038 ceph-mon[92422]: pgmap v12303: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 264 KiB/s wr, 7 op/s 2023-03-09T21:06:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22111.300587 secs 2023-03-09T21:06:41.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:06:41.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:06:41.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22111.300587 secs 2023-03-09T21:06:41.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:06:41.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:06:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22111.300587 secs 2023-03-09T21:06:41.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:06:41.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:06:42.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:42 smithi122 ceph-mon[93869]: pgmap v12304: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 106 KiB/s wr, 7 op/s 2023-03-09T21:06:42.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:42 smithi181 ceph-mon[94189]: pgmap v12304: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 106 KiB/s wr, 7 op/s 2023-03-09T21:06:42.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:42 smithi038 ceph-mon[92422]: pgmap v12304: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 106 KiB/s wr, 7 op/s 2023-03-09T21:06:43.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22113.628272 secs 2023-03-09T21:06:43.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22113.628272 secs 2023-03-09T21:06:43.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22113.628272 secs 2023-03-09T21:06:43.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22575.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:06:44.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:44 smithi122 ceph-mon[93869]: pgmap v12305: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 165 KiB/s wr, 7 op/s 2023-03-09T21:06:44.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:44 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:06:44.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:44 smithi181 ceph-mon[94189]: pgmap v12305: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 165 KiB/s wr, 7 op/s 2023-03-09T21:06:44.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:44 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:06:44.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:44 smithi038 ceph-mon[92422]: pgmap v12305: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 165 KiB/s wr, 7 op/s 2023-03-09T21:06:44.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:44 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:06:45.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22116.300647 secs 2023-03-09T21:06:45.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:45 smithi122 ceph-mon[93869]: pgmap v12306: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 165 KiB/s wr, 7 op/s 2023-03-09T21:06:45.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22116.300647 secs 2023-03-09T21:06:45.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:45 smithi181 ceph-mon[94189]: pgmap v12306: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 165 KiB/s wr, 7 op/s 2023-03-09T21:06:46.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22116.300647 secs 2023-03-09T21:06:46.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:45 smithi038 ceph-mon[92422]: pgmap v12306: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 165 KiB/s wr, 7 op/s 2023-03-09T21:06:47.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:06:47.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:06:47.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:06:47.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:06:47.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:06:47.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:06:47.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22118.628369 secs 2023-03-09T21:06:47.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:47 smithi122 ceph-mon[93869]: pgmap v12307: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 344 KiB/s wr, 10 op/s 2023-03-09T21:06:47.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:06:47.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:06:47.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:06:47.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:06:47.980 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:06:47.980 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:06:47.980 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:06:47.980 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:06:47.980 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:06:47.980 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:06:47.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:06:47.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:06:47.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:06:47.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:06:47.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:06:47.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:06:47.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22118.628369 secs 2023-03-09T21:06:47.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:47 smithi181 ceph-mon[94189]: pgmap v12307: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 344 KiB/s wr, 10 op/s 2023-03-09T21:06:47.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:06:47.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:06:47.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:06:47.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:06:47.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:06:47.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:06:47.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:06:47.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:06:47.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:06:47.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:06:48.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:06:48.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:06:48.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:06:48.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:06:48.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:06:48.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:06:48.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22118.628369 secs 2023-03-09T21:06:48.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:47 smithi038 ceph-mon[92422]: pgmap v12307: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 344 KiB/s wr, 10 op/s 2023-03-09T21:06:48.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:06:48.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:06:48.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:06:48.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:06:48.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:06:48.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:06:48.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:06:48.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:06:48.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:06:48.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:06:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22580.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:06:48.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:48 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi038 to 68072k 2023-03-09T21:06:48.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:48 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T21:06:48.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:48 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi038 to 69706563: error parsing value: Value '69706563' is below minimum 939524096 2023-03-09T21:06:48.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:48 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T21:06:48.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:06:48.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:06:48.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:06:48.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:06:48.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:48 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T21:06:48.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:48 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T21:06:48.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:06:48.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:06:48.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:06:48.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:48 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi038 to 68072k 2023-03-09T21:06:48.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:48 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T21:06:48.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:48 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi038 to 69706563: error parsing value: Value '69706563' is below minimum 939524096 2023-03-09T21:06:48.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:48 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T21:06:48.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:06:48.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:06:48.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:06:48.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:06:48.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:48 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T21:06:48.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:48 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T21:06:48.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:06:48.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:06:48.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:06:49.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:48 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi038 to 68072k 2023-03-09T21:06:49.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:48 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T21:06:49.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:48 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi038 to 69706563: error parsing value: Value '69706563' is below minimum 939524096 2023-03-09T21:06:49.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:48 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T21:06:49.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:06:49.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:06:49.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:06:49.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:06:49.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:48 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T21:06:49.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:48 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T21:06:49.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:06:49.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:06:49.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:06:50.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:49 smithi038 ceph-mon[92422]: pgmap v12308: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 362 KiB/s wr, 9 op/s 2023-03-09T21:06:50.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:49 smithi122 ceph-mon[93869]: pgmap v12308: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 362 KiB/s wr, 9 op/s 2023-03-09T21:06:50.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:49 smithi181 ceph-mon[94189]: pgmap v12308: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 362 KiB/s wr, 9 op/s 2023-03-09T21:06:51.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22121.300708 secs 2023-03-09T21:06:51.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22121.300708 secs 2023-03-09T21:06:51.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22121.300708 secs 2023-03-09T21:06:52.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:06:52.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:06:52.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:51 smithi038 ceph-mon[92422]: pgmap v12309: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 443 KiB/s rd, 428 KiB/s wr, 12 op/s 2023-03-09T21:06:52.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:06:52.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:06:52.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:51 smithi122 ceph-mon[93869]: pgmap v12309: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 443 KiB/s rd, 428 KiB/s wr, 12 op/s 2023-03-09T21:06:52.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:06:52.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:06:52.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:51 smithi181 ceph-mon[94189]: pgmap v12309: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 443 KiB/s rd, 428 KiB/s wr, 12 op/s 2023-03-09T21:06:53.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:06:53.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:06:53.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22123.628502 secs 2023-03-09T21:06:53.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:06:53.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:06:53.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22123.628502 secs 2023-03-09T21:06:53.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:06:53.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:06:53.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22123.628502 secs 2023-03-09T21:06:53.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22585.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:06:54.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:53 smithi122 ceph-mon[93869]: pgmap v12310: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 584 KiB/s rd, 572 KiB/s wr, 12 op/s 2023-03-09T21:06:54.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:53 smithi181 ceph-mon[94189]: pgmap v12310: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 584 KiB/s rd, 572 KiB/s wr, 12 op/s 2023-03-09T21:06:54.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:53 smithi038 ceph-mon[92422]: pgmap v12310: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 584 KiB/s rd, 572 KiB/s wr, 12 op/s 2023-03-09T21:06:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22126.300766 secs 2023-03-09T21:06:55.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:55 smithi122 ceph-mon[93869]: pgmap v12311: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 525 KiB/s rd, 512 KiB/s wr, 10 op/s 2023-03-09T21:06:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22126.300766 secs 2023-03-09T21:06:55.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:55 smithi181 ceph-mon[94189]: pgmap v12311: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 525 KiB/s rd, 512 KiB/s wr, 10 op/s 2023-03-09T21:06:56.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22126.300766 secs 2023-03-09T21:06:56.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:55 smithi038 ceph-mon[92422]: pgmap v12311: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 525 KiB/s rd, 512 KiB/s wr, 10 op/s 2023-03-09T21:06:58.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22128.628606 secs 2023-03-09T21:06:58.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:06:58 smithi122 ceph-mon[93869]: pgmap v12312: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 655 KiB/s wr, 13 op/s 2023-03-09T21:06:58.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22128.628606 secs 2023-03-09T21:06:58.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:06:58 smithi181 ceph-mon[94189]: pgmap v12312: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 655 KiB/s wr, 13 op/s 2023-03-09T21:06:58.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22128.628606 secs 2023-03-09T21:06:58.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:06:58 smithi038 ceph-mon[92422]: pgmap v12312: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 655 KiB/s wr, 13 op/s 2023-03-09T21:06:58.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22590.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:07:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:00 smithi122 ceph-mon[93869]: pgmap v12313: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 493 KiB/s rd, 509 KiB/s wr, 11 op/s 2023-03-09T21:07:00.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:07:00.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:07:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:00 smithi181 ceph-mon[94189]: pgmap v12313: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 493 KiB/s rd, 509 KiB/s wr, 11 op/s 2023-03-09T21:07:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:07:00.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:07:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:00 smithi038 ceph-mon[92422]: pgmap v12313: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 493 KiB/s rd, 509 KiB/s wr, 11 op/s 2023-03-09T21:07:00.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:07:00.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:07:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22131.300835 secs 2023-03-09T21:07:01.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:07:01.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:07:01.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22131.300835 secs 2023-03-09T21:07:01.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:07:01.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:07:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22131.300835 secs 2023-03-09T21:07:01.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:07:01.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:07:02.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:02 smithi122 ceph-mon[93869]: pgmap v12314: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 591 KiB/s rd, 606 KiB/s wr, 13 op/s 2023-03-09T21:07:02.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:02 smithi181 ceph-mon[94189]: pgmap v12314: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 591 KiB/s rd, 606 KiB/s wr, 13 op/s 2023-03-09T21:07:02.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:02 smithi038 ceph-mon[92422]: pgmap v12314: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 591 KiB/s rd, 606 KiB/s wr, 13 op/s 2023-03-09T21:07:03.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22133.628749 secs 2023-03-09T21:07:03.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22133.628749 secs 2023-03-09T21:07:03.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22133.628749 secs 2023-03-09T21:07:03.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22595.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:07:04.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:04 smithi122 ceph-mon[93869]: pgmap v12315: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 442 KiB/s rd, 455 KiB/s wr, 22 op/s 2023-03-09T21:07:04.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:04 smithi181 ceph-mon[94189]: pgmap v12315: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 442 KiB/s rd, 455 KiB/s wr, 22 op/s 2023-03-09T21:07:04.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:04 smithi038 ceph-mon[92422]: pgmap v12315: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 442 KiB/s rd, 455 KiB/s wr, 22 op/s 2023-03-09T21:07:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22136.300898 secs 2023-03-09T21:07:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:05 smithi181 ceph-mon[94189]: pgmap v12316: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 312 KiB/s wr, 20 op/s 2023-03-09T21:07:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22136.300898 secs 2023-03-09T21:07:06.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:05 smithi038 ceph-mon[92422]: pgmap v12316: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 312 KiB/s wr, 20 op/s 2023-03-09T21:07:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22136.300898 secs 2023-03-09T21:07:06.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:05 smithi122 ceph-mon[93869]: pgmap v12316: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 312 KiB/s wr, 20 op/s 2023-03-09T21:07:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22138.628867 secs 2023-03-09T21:07:08.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:08 smithi122 ceph-mon[93869]: pgmap v12317: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 440 KiB/s wr, 25 op/s 2023-03-09T21:07:08.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22138.628867 secs 2023-03-09T21:07:08.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:08 smithi181 ceph-mon[94189]: pgmap v12317: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 440 KiB/s wr, 25 op/s 2023-03-09T21:07:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22138.628867 secs 2023-03-09T21:07:08.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:08 smithi038 ceph-mon[92422]: pgmap v12317: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 440 KiB/s wr, 25 op/s 2023-03-09T21:07:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22600.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:07:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:10 smithi122 ceph-mon[93869]: pgmap v12318: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 298 KiB/s wr, 23 op/s 2023-03-09T21:07:10.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:10 smithi122 ceph-mon[93869]: osdmap e983: 12 total, 12 up, 12 in 2023-03-09T21:07:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:10 smithi181 ceph-mon[94189]: pgmap v12318: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 298 KiB/s wr, 23 op/s 2023-03-09T21:07:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:10 smithi181 ceph-mon[94189]: osdmap e983: 12 total, 12 up, 12 in 2023-03-09T21:07:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:10 smithi038 ceph-mon[92422]: pgmap v12318: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 298 KiB/s wr, 23 op/s 2023-03-09T21:07:10.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:10 smithi038 ceph-mon[92422]: osdmap e983: 12 total, 12 up, 12 in 2023-03-09T21:07:11.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22141.300957 secs 2023-03-09T21:07:11.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22141.300957 secs 2023-03-09T21:07:11.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22141.300957 secs 2023-03-09T21:07:12.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:12 smithi122 ceph-mon[93869]: pgmap v12320: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 180 KiB/s wr, 25 op/s 2023-03-09T21:07:12.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:12 smithi181 ceph-mon[94189]: pgmap v12320: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 180 KiB/s wr, 25 op/s 2023-03-09T21:07:12.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:12 smithi038 ceph-mon[92422]: pgmap v12320: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 180 KiB/s wr, 25 op/s 2023-03-09T21:07:13.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22143.628985 secs 2023-03-09T21:07:13.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22143.628985 secs 2023-03-09T21:07:13.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22143.628985 secs 2023-03-09T21:07:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22605.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:07:14.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:14 smithi122 ceph-mon[93869]: pgmap v12321: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 154 KiB/s wr, 10 op/s 2023-03-09T21:07:14.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:14 smithi181 ceph-mon[94189]: pgmap v12321: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 154 KiB/s wr, 10 op/s 2023-03-09T21:07:14.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:14 smithi038 ceph-mon[92422]: pgmap v12321: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 154 KiB/s wr, 10 op/s 2023-03-09T21:07:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22146.301019 secs 2023-03-09T21:07:15.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:15 smithi122 ceph-mon[93869]: pgmap v12322: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 154 KiB/s wr, 10 op/s 2023-03-09T21:07:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22146.301019 secs 2023-03-09T21:07:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:15 smithi181 ceph-mon[94189]: pgmap v12322: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 154 KiB/s wr, 10 op/s 2023-03-09T21:07:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22146.301019 secs 2023-03-09T21:07:16.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:15 smithi038 ceph-mon[92422]: pgmap v12322: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 154 KiB/s wr, 10 op/s 2023-03-09T21:07:18.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22148.629101 secs 2023-03-09T21:07:18.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:18 smithi122 ceph-mon[93869]: pgmap v12323: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 7 op/s 2023-03-09T21:07:18.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22148.629101 secs 2023-03-09T21:07:18.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:18 smithi181 ceph-mon[94189]: pgmap v12323: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 7 op/s 2023-03-09T21:07:18.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22148.629101 secs 2023-03-09T21:07:18.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:18 smithi038 ceph-mon[92422]: pgmap v12323: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 7 op/s 2023-03-09T21:07:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22610.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:07:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:20 smithi122 ceph-mon[93869]: pgmap v12324: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 7 op/s 2023-03-09T21:07:20.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:20 smithi122 ceph-mon[93869]: osdmap e984: 12 total, 12 up, 12 in 2023-03-09T21:07:20.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:20 smithi181 ceph-mon[94189]: pgmap v12324: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 7 op/s 2023-03-09T21:07:20.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:20 smithi181 ceph-mon[94189]: osdmap e984: 12 total, 12 up, 12 in 2023-03-09T21:07:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:20 smithi038 ceph-mon[92422]: pgmap v12324: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 7 op/s 2023-03-09T21:07:20.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:20 smithi038 ceph-mon[92422]: osdmap e984: 12 total, 12 up, 12 in 2023-03-09T21:07:21.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22151.301080 secs 2023-03-09T21:07:21.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22151.301080 secs 2023-03-09T21:07:21.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22151.301080 secs 2023-03-09T21:07:22.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:22 smithi122 ceph-mon[93869]: pgmap v12326: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 6 op/s 2023-03-09T21:07:22.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:22 smithi181 ceph-mon[94189]: pgmap v12326: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 6 op/s 2023-03-09T21:07:22.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:22 smithi038 ceph-mon[92422]: pgmap v12326: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 6 op/s 2023-03-09T21:07:23.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22153.629201 secs 2023-03-09T21:07:23.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22153.629201 secs 2023-03-09T21:07:23.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22153.629201 secs 2023-03-09T21:07:23.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22615.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:07:24.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:24 smithi122 ceph-mon[93869]: pgmap v12327: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 6 op/s 2023-03-09T21:07:24.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:24 smithi181 ceph-mon[94189]: pgmap v12327: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 6 op/s 2023-03-09T21:07:24.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:24 smithi038 ceph-mon[92422]: pgmap v12327: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 6 op/s 2023-03-09T21:07:26.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22156.301139 secs 2023-03-09T21:07:26.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:25 smithi038 ceph-mon[92422]: pgmap v12328: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 6 op/s 2023-03-09T21:07:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22156.301139 secs 2023-03-09T21:07:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:25 smithi122 ceph-mon[93869]: pgmap v12328: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 6 op/s 2023-03-09T21:07:26.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22156.301139 secs 2023-03-09T21:07:26.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:25 smithi181 ceph-mon[94189]: pgmap v12328: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 6 op/s 2023-03-09T21:07:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22158.629319 secs 2023-03-09T21:07:28.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:28 smithi122 ceph-mon[93869]: pgmap v12329: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-09T21:07:28.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22158.629319 secs 2023-03-09T21:07:28.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:28 smithi181 ceph-mon[94189]: pgmap v12329: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-09T21:07:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22158.629319 secs 2023-03-09T21:07:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:28 smithi038 ceph-mon[92422]: pgmap v12329: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-09T21:07:28.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22620.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:07:30.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:30 smithi122 ceph-mon[93869]: pgmap v12330: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T21:07:30.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:30 smithi181 ceph-mon[94189]: pgmap v12330: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T21:07:30.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:30 smithi038 ceph-mon[92422]: pgmap v12330: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T21:07:31.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22161.301198 secs 2023-03-09T21:07:31.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22161.301198 secs 2023-03-09T21:07:31.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22161.301198 secs 2023-03-09T21:07:32.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:32 smithi122 ceph-mon[93869]: pgmap v12331: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T21:07:32.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:32 smithi181 ceph-mon[94189]: pgmap v12331: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T21:07:32.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:32 smithi038 ceph-mon[92422]: pgmap v12331: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T21:07:33.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22163.629444 secs 2023-03-09T21:07:33.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22163.629444 secs 2023-03-09T21:07:33.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22163.629444 secs 2023-03-09T21:07:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22625.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:07:34.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:34 smithi122 ceph-mon[93869]: pgmap v12332: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T21:07:34.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:34 smithi181 ceph-mon[94189]: pgmap v12332: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T21:07:34.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:34 smithi038 ceph-mon[92422]: pgmap v12332: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T21:07:36.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22166.301257 secs 2023-03-09T21:07:36.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:35 smithi038 ceph-mon[92422]: pgmap v12333: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 511 B/s wr, 5 op/s 2023-03-09T21:07:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22166.301257 secs 2023-03-09T21:07:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:35 smithi122 ceph-mon[93869]: pgmap v12333: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 511 B/s wr, 5 op/s 2023-03-09T21:07:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22166.301257 secs 2023-03-09T21:07:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:35 smithi181 ceph-mon[94189]: pgmap v12333: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 511 B/s wr, 5 op/s 2023-03-09T21:07:38.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22168.629545 secs 2023-03-09T21:07:38.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:38 smithi122 ceph-mon[93869]: pgmap v12334: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 682 B/s wr, 8 op/s 2023-03-09T21:07:38.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22168.629545 secs 2023-03-09T21:07:38.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:38 smithi181 ceph-mon[94189]: pgmap v12334: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 682 B/s wr, 8 op/s 2023-03-09T21:07:38.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22168.629545 secs 2023-03-09T21:07:38.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:38 smithi038 ceph-mon[92422]: pgmap v12334: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 682 B/s wr, 8 op/s 2023-03-09T21:07:38.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22630.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:07:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:40 smithi122 ceph-mon[93869]: pgmap v12335: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 682 B/s wr, 6 op/s 2023-03-09T21:07:40.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:07:40.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:07:40.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:40 smithi181 ceph-mon[94189]: pgmap v12335: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 682 B/s wr, 6 op/s 2023-03-09T21:07:40.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:07:40.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:07:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:40 smithi038 ceph-mon[92422]: pgmap v12335: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 682 B/s wr, 6 op/s 2023-03-09T21:07:40.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:07:40.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:07:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22171.301316 secs 2023-03-09T21:07:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:07:41.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:07:41.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22171.301316 secs 2023-03-09T21:07:41.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:07:41.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:07:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22171.301316 secs 2023-03-09T21:07:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:07:41.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:07:42.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:42 smithi122 ceph-mon[93869]: pgmap v12336: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 160 KiB/s wr, 9 op/s 2023-03-09T21:07:42.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:42 smithi181 ceph-mon[94189]: pgmap v12336: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 160 KiB/s wr, 9 op/s 2023-03-09T21:07:42.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:42 smithi038 ceph-mon[92422]: pgmap v12336: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 160 KiB/s wr, 9 op/s 2023-03-09T21:07:43.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22635.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:07:44.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:44 smithi122 ceph-mon[93869]: pgmap v12337: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 178 KiB/s wr, 8 op/s 2023-03-09T21:07:44.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:44 smithi181 ceph-mon[94189]: pgmap v12337: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 178 KiB/s wr, 8 op/s 2023-03-09T21:07:44.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:44 smithi038 ceph-mon[92422]: pgmap v12337: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 178 KiB/s wr, 8 op/s 2023-03-09T21:07:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:45 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:07:45.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:45 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:07:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:45 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:07:46.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22176.301378 secs 2023-03-09T21:07:46.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:46 smithi122 ceph-mon[93869]: pgmap v12338: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 178 KiB/s wr, 7 op/s 2023-03-09T21:07:46.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:46 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:07:46.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:46 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:07:46.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:46 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:07:46.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22176.301378 secs 2023-03-09T21:07:46.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:46 smithi181 ceph-mon[94189]: pgmap v12338: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 178 KiB/s wr, 7 op/s 2023-03-09T21:07:46.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:46 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:07:46.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:46 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:07:46.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:46 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:07:46.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22176.301378 secs 2023-03-09T21:07:46.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:46 smithi038 ceph-mon[92422]: pgmap v12338: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 178 KiB/s wr, 7 op/s 2023-03-09T21:07:46.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:46 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:07:46.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:46 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:07:46.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:46 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:07:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:48 smithi122 ceph-mon[93869]: pgmap v12339: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 357 KiB/s wr, 11 op/s 2023-03-09T21:07:48.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:07:48.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:48 smithi181 ceph-mon[94189]: pgmap v12339: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 357 KiB/s wr, 11 op/s 2023-03-09T21:07:48.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:07:48.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:48 smithi038 ceph-mon[92422]: pgmap v12339: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 357 KiB/s wr, 11 op/s 2023-03-09T21:07:48.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:07:48.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22640.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:07:49.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:07:49.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:07:49.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:07:49.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:07:49.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:07:49.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:07:49.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:07:49.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:07:49.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:07:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:50 smithi122 ceph-mon[93869]: pgmap v12340: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 454 KiB/s rd, 432 KiB/s wr, 11 op/s 2023-03-09T21:07:50.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:50 smithi181 ceph-mon[94189]: pgmap v12340: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 454 KiB/s rd, 432 KiB/s wr, 11 op/s 2023-03-09T21:07:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:50 smithi038 ceph-mon[92422]: pgmap v12340: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 454 KiB/s rd, 432 KiB/s wr, 11 op/s 2023-03-09T21:07:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22181.301437 secs 2023-03-09T21:07:51.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22181.301437 secs 2023-03-09T21:07:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22181.301437 secs 2023-03-09T21:07:52.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:52 smithi122 ceph-mon[93869]: pgmap v12341: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 561 KiB/s rd, 539 KiB/s wr, 13 op/s 2023-03-09T21:07:52.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:07:52.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:07:52.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:52 smithi181 ceph-mon[94189]: pgmap v12341: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 561 KiB/s rd, 539 KiB/s wr, 13 op/s 2023-03-09T21:07:52.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:07:52.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:07:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:52 smithi038 ceph-mon[92422]: pgmap v12341: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 561 KiB/s rd, 539 KiB/s wr, 13 op/s 2023-03-09T21:07:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:07:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:07:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:07:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:07:53.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:07:53.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:07:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:07:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:07:53.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22645.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:07:54.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:54 smithi122 ceph-mon[93869]: pgmap v12342: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 401 KiB/s wr, 11 op/s 2023-03-09T21:07:54.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:54 smithi181 ceph-mon[94189]: pgmap v12342: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 401 KiB/s wr, 11 op/s 2023-03-09T21:07:54.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:54 smithi038 ceph-mon[92422]: pgmap v12342: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 401 KiB/s wr, 11 op/s 2023-03-09T21:07:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22186.301498 secs 2023-03-09T21:07:56.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:55 smithi038 ceph-mon[92422]: pgmap v12343: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 527 KiB/s rd, 514 KiB/s wr, 11 op/s 2023-03-09T21:07:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22186.301498 secs 2023-03-09T21:07:56.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:55 smithi122 ceph-mon[93869]: pgmap v12343: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 527 KiB/s rd, 514 KiB/s wr, 11 op/s 2023-03-09T21:07:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22186.301498 secs 2023-03-09T21:07:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:55 smithi181 ceph-mon[94189]: pgmap v12343: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 527 KiB/s rd, 514 KiB/s wr, 11 op/s 2023-03-09T21:07:58.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22188.630057 secs 2023-03-09T21:07:58.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:07:58 smithi122 ceph-mon[93869]: pgmap v12344: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 529 KiB/s rd, 514 KiB/s wr, 14 op/s 2023-03-09T21:07:58.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22188.630057 secs 2023-03-09T21:07:58.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:07:58 smithi181 ceph-mon[94189]: pgmap v12344: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 529 KiB/s rd, 514 KiB/s wr, 14 op/s 2023-03-09T21:07:58.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22188.630057 secs 2023-03-09T21:07:58.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:07:58 smithi038 ceph-mon[92422]: pgmap v12344: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 529 KiB/s rd, 514 KiB/s wr, 14 op/s 2023-03-09T21:07:58.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22650.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:08:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:00 smithi122 ceph-mon[93869]: pgmap v12345: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 522 KiB/s wr, 12 op/s 2023-03-09T21:08:00.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:08:00.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:08:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:00 smithi181 ceph-mon[94189]: pgmap v12345: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 522 KiB/s wr, 12 op/s 2023-03-09T21:08:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:08:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:08:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:00 smithi038 ceph-mon[92422]: pgmap v12345: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 522 KiB/s wr, 12 op/s 2023-03-09T21:08:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:08:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:08:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22191.301558 secs 2023-03-09T21:08:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:08:01.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:08:01.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22191.301558 secs 2023-03-09T21:08:01.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:08:01.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:08:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22191.301558 secs 2023-03-09T21:08:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:08:01.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:08:02.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:02 smithi122 ceph-mon[93869]: pgmap v12346: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 516 KiB/s wr, 13 op/s 2023-03-09T21:08:02.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:02 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:08:02.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:02 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:08:02.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:02 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:08:02.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:02 smithi181 ceph-mon[94189]: pgmap v12346: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 516 KiB/s wr, 13 op/s 2023-03-09T21:08:02.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:02 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:08:02.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:02 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:08:02.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:02 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:08:02.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:02 smithi038 ceph-mon[92422]: pgmap v12346: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 516 KiB/s wr, 13 op/s 2023-03-09T21:08:02.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:02 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:08:02.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:02 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:08:02.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:02 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:08:03.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22193.630197 secs 2023-03-09T21:08:03.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22193.630197 secs 2023-03-09T21:08:03.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22193.630197 secs 2023-03-09T21:08:03.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22655.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:08:04.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:04 smithi122 ceph-mon[93869]: pgmap v12347: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 430 KiB/s wr, 23 op/s 2023-03-09T21:08:04.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:04 smithi181 ceph-mon[94189]: pgmap v12347: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 430 KiB/s wr, 23 op/s 2023-03-09T21:08:04.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:04 smithi038 ceph-mon[92422]: pgmap v12347: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 430 KiB/s wr, 23 op/s 2023-03-09T21:08:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22196.301639 secs 2023-03-09T21:08:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:05 smithi038 ceph-mon[92422]: pgmap v12348: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 495 KiB/s wr, 23 op/s 2023-03-09T21:08:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22196.301639 secs 2023-03-09T21:08:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:05 smithi122 ceph-mon[93869]: pgmap v12348: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 495 KiB/s wr, 23 op/s 2023-03-09T21:08:06.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22196.301639 secs 2023-03-09T21:08:06.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:05 smithi181 ceph-mon[94189]: pgmap v12348: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 495 KiB/s wr, 23 op/s 2023-03-09T21:08:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22198.630333 secs 2023-03-09T21:08:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:08 smithi122 ceph-mon[93869]: pgmap v12349: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 385 KiB/s wr, 25 op/s 2023-03-09T21:08:08.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22198.630333 secs 2023-03-09T21:08:08.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:08 smithi181 ceph-mon[94189]: pgmap v12349: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 385 KiB/s wr, 25 op/s 2023-03-09T21:08:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22198.630333 secs 2023-03-09T21:08:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:08 smithi038 ceph-mon[92422]: pgmap v12349: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 385 KiB/s wr, 25 op/s 2023-03-09T21:08:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22660.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:08:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:10 smithi122 ceph-mon[93869]: pgmap v12350: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 385 KiB/s wr, 24 op/s 2023-03-09T21:08:10.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:10 smithi122 ceph-mon[93869]: osdmap e985: 12 total, 12 up, 12 in 2023-03-09T21:08:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:10 smithi181 ceph-mon[94189]: pgmap v12350: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 385 KiB/s wr, 24 op/s 2023-03-09T21:08:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:10 smithi181 ceph-mon[94189]: osdmap e985: 12 total, 12 up, 12 in 2023-03-09T21:08:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:10 smithi038 ceph-mon[92422]: pgmap v12350: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 385 KiB/s wr, 24 op/s 2023-03-09T21:08:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:10 smithi038 ceph-mon[92422]: osdmap e985: 12 total, 12 up, 12 in 2023-03-09T21:08:11.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22201.301698 secs 2023-03-09T21:08:11.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22201.301698 secs 2023-03-09T21:08:11.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22201.301698 secs 2023-03-09T21:08:12.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:12 smithi122 ceph-mon[93869]: pgmap v12352: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 155 KiB/s wr, 25 op/s 2023-03-09T21:08:12.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:12 smithi181 ceph-mon[94189]: pgmap v12352: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 155 KiB/s wr, 25 op/s 2023-03-09T21:08:12.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:12 smithi038 ceph-mon[92422]: pgmap v12352: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 155 KiB/s wr, 25 op/s 2023-03-09T21:08:13.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22203.630450 secs 2023-03-09T21:08:13.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22203.630450 secs 2023-03-09T21:08:13.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22203.630450 secs 2023-03-09T21:08:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22665.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:08:14.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:14 smithi122 ceph-mon[93869]: pgmap v12353: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 129 KiB/s wr, 10 op/s 2023-03-09T21:08:14.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:14 smithi181 ceph-mon[94189]: pgmap v12353: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 129 KiB/s wr, 10 op/s 2023-03-09T21:08:14.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:14 smithi038 ceph-mon[92422]: pgmap v12353: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 129 KiB/s wr, 10 op/s 2023-03-09T21:08:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22206.301759 secs 2023-03-09T21:08:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:15 smithi122 ceph-mon[93869]: pgmap v12354: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 98 KiB/s wr, 10 op/s 2023-03-09T21:08:16.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22206.301759 secs 2023-03-09T21:08:16.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:15 smithi181 ceph-mon[94189]: pgmap v12354: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 98 KiB/s wr, 10 op/s 2023-03-09T21:08:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22206.301759 secs 2023-03-09T21:08:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:15 smithi038 ceph-mon[92422]: pgmap v12354: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 98 KiB/s wr, 10 op/s 2023-03-09T21:08:18.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22208.630578 secs 2023-03-09T21:08:18.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:18 smithi122 ceph-mon[93869]: pgmap v12355: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 72 KiB/s wr, 7 op/s 2023-03-09T21:08:18.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22208.630578 secs 2023-03-09T21:08:18.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:18 smithi181 ceph-mon[94189]: pgmap v12355: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 72 KiB/s wr, 7 op/s 2023-03-09T21:08:18.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22208.630578 secs 2023-03-09T21:08:18.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:18 smithi038 ceph-mon[92422]: pgmap v12355: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 72 KiB/s wr, 7 op/s 2023-03-09T21:08:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22670.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:08:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:20 smithi122 ceph-mon[93869]: pgmap v12356: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 6 op/s 2023-03-09T21:08:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:20 smithi122 ceph-mon[93869]: osdmap e986: 12 total, 12 up, 12 in 2023-03-09T21:08:20.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:20 smithi181 ceph-mon[94189]: pgmap v12356: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 6 op/s 2023-03-09T21:08:20.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:20 smithi181 ceph-mon[94189]: osdmap e986: 12 total, 12 up, 12 in 2023-03-09T21:08:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:20 smithi038 ceph-mon[92422]: pgmap v12356: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 6 op/s 2023-03-09T21:08:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:20 smithi038 ceph-mon[92422]: osdmap e986: 12 total, 12 up, 12 in 2023-03-09T21:08:21.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22211.301821 secs 2023-03-09T21:08:21.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22211.301821 secs 2023-03-09T21:08:21.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22211.301821 secs 2023-03-09T21:08:22.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:22 smithi122 ceph-mon[93869]: pgmap v12358: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 6 op/s 2023-03-09T21:08:22.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:22 smithi181 ceph-mon[94189]: pgmap v12358: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 6 op/s 2023-03-09T21:08:22.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:22 smithi038 ceph-mon[92422]: pgmap v12358: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 6 op/s 2023-03-09T21:08:23.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22213.630719 secs 2023-03-09T21:08:23.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22213.630719 secs 2023-03-09T21:08:23.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22213.630719 secs 2023-03-09T21:08:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22675.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:08:24.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:24 smithi122 ceph-mon[93869]: pgmap v12359: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 6 op/s 2023-03-09T21:08:24.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:24 smithi181 ceph-mon[94189]: pgmap v12359: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 6 op/s 2023-03-09T21:08:24.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:24 smithi038 ceph-mon[92422]: pgmap v12359: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 6 op/s 2023-03-09T21:08:26.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22216.301881 secs 2023-03-09T21:08:26.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:25 smithi038 ceph-mon[92422]: pgmap v12360: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T21:08:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22216.301881 secs 2023-03-09T21:08:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:25 smithi122 ceph-mon[93869]: pgmap v12360: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T21:08:26.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22216.301881 secs 2023-03-09T21:08:26.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:25 smithi181 ceph-mon[94189]: pgmap v12360: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T21:08:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22218.630842 secs 2023-03-09T21:08:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:28 smithi122 ceph-mon[93869]: pgmap v12361: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-09T21:08:28.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22218.630842 secs 2023-03-09T21:08:28.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:28 smithi181 ceph-mon[94189]: pgmap v12361: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-09T21:08:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22218.630842 secs 2023-03-09T21:08:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:28 smithi038 ceph-mon[92422]: pgmap v12361: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-09T21:08:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22680.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:08:30.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:30 smithi122 ceph-mon[93869]: pgmap v12362: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T21:08:30.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:30 smithi181 ceph-mon[94189]: pgmap v12362: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T21:08:30.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:30 smithi038 ceph-mon[92422]: pgmap v12362: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T21:08:31.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22221.301960 secs 2023-03-09T21:08:31.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22221.301960 secs 2023-03-09T21:08:31.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22221.301960 secs 2023-03-09T21:08:32.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:32 smithi122 ceph-mon[93869]: pgmap v12363: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 75 KiB/s wr, 7 op/s 2023-03-09T21:08:32.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:32 smithi181 ceph-mon[94189]: pgmap v12363: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 75 KiB/s wr, 7 op/s 2023-03-09T21:08:32.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:32 smithi038 ceph-mon[92422]: pgmap v12363: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 75 KiB/s wr, 7 op/s 2023-03-09T21:08:33.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22223.630955 secs 2023-03-09T21:08:33.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22223.630955 secs 2023-03-09T21:08:33.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22223.630955 secs 2023-03-09T21:08:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22685.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:08:34.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:34 smithi122 ceph-mon[93869]: pgmap v12364: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 6 op/s 2023-03-09T21:08:34.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:34 smithi181 ceph-mon[94189]: pgmap v12364: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 6 op/s 2023-03-09T21:08:34.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:34 smithi038 ceph-mon[92422]: pgmap v12364: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 6 op/s 2023-03-09T21:08:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22226.302021 secs 2023-03-09T21:08:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:35 smithi122 ceph-mon[93869]: pgmap v12365: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 6 op/s 2023-03-09T21:08:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22226.302021 secs 2023-03-09T21:08:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:35 smithi181 ceph-mon[94189]: pgmap v12365: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 6 op/s 2023-03-09T21:08:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22226.302021 secs 2023-03-09T21:08:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:35 smithi038 ceph-mon[92422]: pgmap v12365: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 6 op/s 2023-03-09T21:08:38.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:38 smithi122 ceph-mon[93869]: pgmap v12366: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 73 KiB/s wr, 8 op/s 2023-03-09T21:08:38.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:38 smithi181 ceph-mon[94189]: pgmap v12366: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 73 KiB/s wr, 8 op/s 2023-03-09T21:08:38.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:38 smithi038 ceph-mon[92422]: pgmap v12366: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 73 KiB/s wr, 8 op/s 2023-03-09T21:08:38.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22690.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:08:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:40 smithi122 ceph-mon[93869]: pgmap v12367: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 73 KiB/s wr, 7 op/s 2023-03-09T21:08:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:08:40.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:08:40.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:40 smithi181 ceph-mon[94189]: pgmap v12367: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 73 KiB/s wr, 7 op/s 2023-03-09T21:08:40.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:08:40.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:08:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:40 smithi038 ceph-mon[92422]: pgmap v12367: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 73 KiB/s wr, 7 op/s 2023-03-09T21:08:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:08:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:08:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22231.302094 secs 2023-03-09T21:08:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:08:41.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:08:41.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:41 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:08:41.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22231.302094 secs 2023-03-09T21:08:41.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:08:41.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:08:41.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:41 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:08:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22231.302094 secs 2023-03-09T21:08:41.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:08:41.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:08:41.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:41 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:08:42.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:42 smithi122 ceph-mon[93869]: pgmap v12368: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 220 KiB/s wr, 9 op/s 2023-03-09T21:08:42.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:42 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:08:42.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:42 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:08:42.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:42 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:08:42.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:42 smithi181 ceph-mon[94189]: pgmap v12368: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 220 KiB/s wr, 9 op/s 2023-03-09T21:08:42.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:42 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:08:42.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:42 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:08:42.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:42 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:08:42.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:42 smithi038 ceph-mon[92422]: pgmap v12368: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 220 KiB/s wr, 9 op/s 2023-03-09T21:08:42.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:42 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:08:42.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:42 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:08:42.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:42 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:08:43.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:43 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 22233.631200 secs 2023-03-09T21:08:43.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:43 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 22233.631200 secs 2023-03-09T21:08:43.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:43 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 22233.631200 secs 2023-03-09T21:08:43.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22695.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:08:44.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:44 smithi122 ceph-mon[93869]: pgmap v12369: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 167 KiB/s wr, 8 op/s 2023-03-09T21:08:44.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:44 smithi181 ceph-mon[94189]: pgmap v12369: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 167 KiB/s wr, 8 op/s 2023-03-09T21:08:44.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:44 smithi038 ceph-mon[92422]: pgmap v12369: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 167 KiB/s wr, 8 op/s 2023-03-09T21:08:46.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22236.302154 secs 2023-03-09T21:08:46.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:46 smithi122 ceph-mon[93869]: pgmap v12370: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 166 KiB/s wr, 7 op/s 2023-03-09T21:08:46.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:46 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:08:46.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:46 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:08:46.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22236.302154 secs 2023-03-09T21:08:46.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:46 smithi181 ceph-mon[94189]: pgmap v12370: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 166 KiB/s wr, 7 op/s 2023-03-09T21:08:46.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:46 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:08:46.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:46 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:08:46.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22236.302154 secs 2023-03-09T21:08:46.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:46 smithi038 ceph-mon[92422]: pgmap v12370: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 166 KiB/s wr, 7 op/s 2023-03-09T21:08:46.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:46 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:08:46.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:46 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:08:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:48 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 22238.631319 secs 2023-03-09T21:08:48.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:48 smithi122 ceph-mon[93869]: pgmap v12371: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 346 KiB/s wr, 10 op/s 2023-03-09T21:08:48.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:48 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 22238.631319 secs 2023-03-09T21:08:48.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:48 smithi181 ceph-mon[94189]: pgmap v12371: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 346 KiB/s wr, 10 op/s 2023-03-09T21:08:48.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:48 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 22238.631319 secs 2023-03-09T21:08:48.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:48 smithi038 ceph-mon[92422]: pgmap v12371: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 346 KiB/s wr, 10 op/s 2023-03-09T21:08:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22700.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:08:49.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:08:49.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:08:49.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:08:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:50 smithi122 ceph-mon[93869]: pgmap v12372: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 435 KiB/s rd, 364 KiB/s wr, 10 op/s 2023-03-09T21:08:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:50 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:08:50.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:50 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:08:50.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:50 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:08:50.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:50 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:08:50.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:50 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:08:50.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:50 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:08:50.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:50 smithi181 ceph-mon[94189]: pgmap v12372: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 435 KiB/s rd, 364 KiB/s wr, 10 op/s 2023-03-09T21:08:50.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:50 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:08:50.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:50 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:08:50.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:50 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:08:50.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:50 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:08:50.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:50 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:08:50.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:50 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:08:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:50 smithi038 ceph-mon[92422]: pgmap v12372: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 435 KiB/s rd, 364 KiB/s wr, 10 op/s 2023-03-09T21:08:50.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:50 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:08:50.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:50 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:08:50.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:50 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:08:50.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:50 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:08:50.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:50 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:08:50.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:50 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:08:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22241.302215 secs 2023-03-09T21:08:51.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22241.302215 secs 2023-03-09T21:08:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22241.302215 secs 2023-03-09T21:08:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:52 smithi122 ceph-mon[93869]: pgmap v12373: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 731 KiB/s rd, 660 KiB/s wr, 13 op/s 2023-03-09T21:08:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:08:52.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:08:52.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:52 smithi181 ceph-mon[94189]: pgmap v12373: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 731 KiB/s rd, 660 KiB/s wr, 13 op/s 2023-03-09T21:08:52.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:08:52.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:08:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:52 smithi038 ceph-mon[92422]: pgmap v12373: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 731 KiB/s rd, 660 KiB/s wr, 13 op/s 2023-03-09T21:08:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:08:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:08:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22243.631389 secs 2023-03-09T21:08:53.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:08:53.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:08:53.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22243.631389 secs 2023-03-09T21:08:53.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:08:53.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:08:53.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22243.631389 secs 2023-03-09T21:08:53.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:08:53.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:08:53.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22705.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:08:54.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:54 smithi122 ceph-mon[93869]: pgmap v12374: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 547 KiB/s rd, 534 KiB/s wr, 12 op/s 2023-03-09T21:08:54.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:54 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:08:54.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:54 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:08:54.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:54 smithi181 ceph-mon[94189]: pgmap v12374: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 547 KiB/s rd, 534 KiB/s wr, 12 op/s 2023-03-09T21:08:54.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:54 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:08:54.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:54 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:08:54.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:54 smithi038 ceph-mon[92422]: pgmap v12374: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 547 KiB/s rd, 534 KiB/s wr, 12 op/s 2023-03-09T21:08:54.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:54 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:08:54.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:54 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:08:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22246.302278 secs 2023-03-09T21:08:56.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:55 smithi122 ceph-mon[93869]: pgmap v12375: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 529 KiB/s rd, 516 KiB/s wr, 10 op/s 2023-03-09T21:08:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22246.302278 secs 2023-03-09T21:08:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:55 smithi181 ceph-mon[94189]: pgmap v12375: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 529 KiB/s rd, 516 KiB/s wr, 10 op/s 2023-03-09T21:08:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22246.302278 secs 2023-03-09T21:08:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:55 smithi038 ceph-mon[92422]: pgmap v12375: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 529 KiB/s rd, 516 KiB/s wr, 10 op/s 2023-03-09T21:08:58.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22248.631528 secs 2023-03-09T21:08:58.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:08:58 smithi122 ceph-mon[93869]: pgmap v12376: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 664 KiB/s rd, 650 KiB/s wr, 14 op/s 2023-03-09T21:08:58.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22248.631528 secs 2023-03-09T21:08:58.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:08:58 smithi181 ceph-mon[94189]: pgmap v12376: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 664 KiB/s rd, 650 KiB/s wr, 14 op/s 2023-03-09T21:08:58.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22248.631528 secs 2023-03-09T21:08:58.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:08:58 smithi038 ceph-mon[92422]: pgmap v12376: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 664 KiB/s rd, 650 KiB/s wr, 14 op/s 2023-03-09T21:08:58.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22710.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:09:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:00 smithi122 ceph-mon[93869]: pgmap v12377: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 623 KiB/s rd, 612 KiB/s wr, 12 op/s 2023-03-09T21:09:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:09:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:09:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:00 smithi181 ceph-mon[94189]: pgmap v12377: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 623 KiB/s rd, 612 KiB/s wr, 12 op/s 2023-03-09T21:09:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:09:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:09:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:00 smithi038 ceph-mon[92422]: pgmap v12377: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 623 KiB/s rd, 612 KiB/s wr, 12 op/s 2023-03-09T21:09:00.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:09:00.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:09:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22251.302338 secs 2023-03-09T21:09:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:09:01.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:09:01.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22251.302338 secs 2023-03-09T21:09:01.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:09:01.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:09:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22251.302338 secs 2023-03-09T21:09:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:09:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:09:02.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:02 smithi122 ceph-mon[93869]: pgmap v12378: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 612 KiB/s rd, 632 KiB/s wr, 17 op/s 2023-03-09T21:09:02.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:02 smithi181 ceph-mon[94189]: pgmap v12378: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 612 KiB/s rd, 632 KiB/s wr, 17 op/s 2023-03-09T21:09:02.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:02 smithi038 ceph-mon[92422]: pgmap v12378: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 612 KiB/s rd, 632 KiB/s wr, 17 op/s 2023-03-09T21:09:03.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22253.631667 secs 2023-03-09T21:09:03.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22253.631667 secs 2023-03-09T21:09:03.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22253.631667 secs 2023-03-09T21:09:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22715.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:09:04.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:04 smithi122 ceph-mon[93869]: pgmap v12379: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 358 KiB/s wr, 26 op/s 2023-03-09T21:09:04.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:04 smithi181 ceph-mon[94189]: pgmap v12379: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 358 KiB/s wr, 26 op/s 2023-03-09T21:09:04.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:04 smithi038 ceph-mon[92422]: pgmap v12379: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 358 KiB/s wr, 26 op/s 2023-03-09T21:09:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22256.302398 secs 2023-03-09T21:09:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:05 smithi122 ceph-mon[93869]: pgmap v12380: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 337 KiB/s wr, 25 op/s 2023-03-09T21:09:06.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22256.302398 secs 2023-03-09T21:09:06.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:05 smithi181 ceph-mon[94189]: pgmap v12380: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 337 KiB/s wr, 25 op/s 2023-03-09T21:09:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22256.302398 secs 2023-03-09T21:09:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:05 smithi038 ceph-mon[92422]: pgmap v12380: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 337 KiB/s wr, 25 op/s 2023-03-09T21:09:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22258.631785 secs 2023-03-09T21:09:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:08 smithi122 ceph-mon[93869]: pgmap v12381: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 337 KiB/s wr, 27 op/s 2023-03-09T21:09:08.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22258.631785 secs 2023-03-09T21:09:08.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:08 smithi181 ceph-mon[94189]: pgmap v12381: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 337 KiB/s wr, 27 op/s 2023-03-09T21:09:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22258.631785 secs 2023-03-09T21:09:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:08 smithi038 ceph-mon[92422]: pgmap v12381: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 337 KiB/s wr, 27 op/s 2023-03-09T21:09:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22720.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:09:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:10 smithi122 ceph-mon[93869]: pgmap v12382: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 309 KiB/s wr, 24 op/s 2023-03-09T21:09:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:10 smithi122 ceph-mon[93869]: osdmap e987: 12 total, 12 up, 12 in 2023-03-09T21:09:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:10 smithi181 ceph-mon[94189]: pgmap v12382: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 309 KiB/s wr, 24 op/s 2023-03-09T21:09:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:10 smithi181 ceph-mon[94189]: osdmap e987: 12 total, 12 up, 12 in 2023-03-09T21:09:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:10 smithi038 ceph-mon[92422]: pgmap v12382: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 309 KiB/s wr, 24 op/s 2023-03-09T21:09:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:10 smithi038 ceph-mon[92422]: osdmap e987: 12 total, 12 up, 12 in 2023-03-09T21:09:11.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22261.302457 secs 2023-03-09T21:09:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22261.302457 secs 2023-03-09T21:09:11.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22261.302457 secs 2023-03-09T21:09:12.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:12 smithi122 ceph-mon[93869]: pgmap v12384: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 154 KiB/s wr, 22 op/s 2023-03-09T21:09:12.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:12 smithi181 ceph-mon[94189]: pgmap v12384: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 154 KiB/s wr, 22 op/s 2023-03-09T21:09:12.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:12 smithi038 ceph-mon[92422]: pgmap v12384: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 154 KiB/s wr, 22 op/s 2023-03-09T21:09:13.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22263.631904 secs 2023-03-09T21:09:13.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22263.631904 secs 2023-03-09T21:09:13.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22263.631904 secs 2023-03-09T21:09:13.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22725.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:09:14.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:14 smithi122 ceph-mon[93869]: pgmap v12385: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 129 KiB/s wr, 7 op/s 2023-03-09T21:09:14.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:14 smithi181 ceph-mon[94189]: pgmap v12385: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 129 KiB/s wr, 7 op/s 2023-03-09T21:09:14.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:14 smithi038 ceph-mon[92422]: pgmap v12385: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 129 KiB/s wr, 7 op/s 2023-03-09T21:09:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22266.302518 secs 2023-03-09T21:09:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:15 smithi122 ceph-mon[93869]: pgmap v12386: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 127 KiB/s wr, 6 op/s 2023-03-09T21:09:16.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22266.302518 secs 2023-03-09T21:09:16.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:15 smithi181 ceph-mon[94189]: pgmap v12386: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 127 KiB/s wr, 6 op/s 2023-03-09T21:09:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22266.302518 secs 2023-03-09T21:09:16.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:15 smithi038 ceph-mon[92422]: pgmap v12386: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 127 KiB/s wr, 6 op/s 2023-03-09T21:09:18.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22268.632027 secs 2023-03-09T21:09:18.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:18 smithi122 ceph-mon[93869]: pgmap v12387: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 127 KiB/s wr, 6 op/s 2023-03-09T21:09:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22268.632027 secs 2023-03-09T21:09:18.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:18 smithi181 ceph-mon[94189]: pgmap v12387: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 127 KiB/s wr, 6 op/s 2023-03-09T21:09:18.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22268.632027 secs 2023-03-09T21:09:18.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:18 smithi038 ceph-mon[92422]: pgmap v12387: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 127 KiB/s wr, 6 op/s 2023-03-09T21:09:18.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22730.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:09:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:20 smithi122 ceph-mon[93869]: pgmap v12388: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 6 op/s 2023-03-09T21:09:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:20 smithi122 ceph-mon[93869]: osdmap e988: 12 total, 12 up, 12 in 2023-03-09T21:09:20.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:20 smithi181 ceph-mon[94189]: pgmap v12388: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 6 op/s 2023-03-09T21:09:20.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:20 smithi181 ceph-mon[94189]: osdmap e988: 12 total, 12 up, 12 in 2023-03-09T21:09:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:20 smithi038 ceph-mon[92422]: pgmap v12388: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 6 op/s 2023-03-09T21:09:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:20 smithi038 ceph-mon[92422]: osdmap e988: 12 total, 12 up, 12 in 2023-03-09T21:09:21.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22271.302578 secs 2023-03-09T21:09:21.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22271.302578 secs 2023-03-09T21:09:21.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22271.302578 secs 2023-03-09T21:09:22.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:22 smithi122 ceph-mon[93869]: pgmap v12390: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 6 op/s 2023-03-09T21:09:22.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:22 smithi181 ceph-mon[94189]: pgmap v12390: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 6 op/s 2023-03-09T21:09:22.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:22 smithi038 ceph-mon[92422]: pgmap v12390: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 6 op/s 2023-03-09T21:09:23.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22273.632151 secs 2023-03-09T21:09:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22273.632151 secs 2023-03-09T21:09:23.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22273.632151 secs 2023-03-09T21:09:23.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22735.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:09:24.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:24 smithi122 ceph-mon[93869]: pgmap v12391: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 6 op/s 2023-03-09T21:09:24.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:24 smithi181 ceph-mon[94189]: pgmap v12391: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 6 op/s 2023-03-09T21:09:24.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:24 smithi038 ceph-mon[92422]: pgmap v12391: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 6 op/s 2023-03-09T21:09:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22276.302638 secs 2023-03-09T21:09:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:25 smithi122 ceph-mon[93869]: pgmap v12392: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 6 op/s 2023-03-09T21:09:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22276.302638 secs 2023-03-09T21:09:26.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:25 smithi181 ceph-mon[94189]: pgmap v12392: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 6 op/s 2023-03-09T21:09:26.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22276.302638 secs 2023-03-09T21:09:26.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:25 smithi038 ceph-mon[92422]: pgmap v12392: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 6 op/s 2023-03-09T21:09:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22278.632235 secs 2023-03-09T21:09:28.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:28 smithi122 ceph-mon[93869]: pgmap v12393: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 6 op/s 2023-03-09T21:09:28.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22278.632235 secs 2023-03-09T21:09:28.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:28 smithi181 ceph-mon[94189]: pgmap v12393: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 6 op/s 2023-03-09T21:09:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22278.632235 secs 2023-03-09T21:09:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:28 smithi038 ceph-mon[92422]: pgmap v12393: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 6 op/s 2023-03-09T21:09:28.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22740.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:09:30.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:30 smithi122 ceph-mon[93869]: pgmap v12394: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T21:09:30.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:30 smithi181 ceph-mon[94189]: pgmap v12394: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T21:09:30.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:30 smithi038 ceph-mon[92422]: pgmap v12394: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T21:09:31.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22281.302715 secs 2023-03-09T21:09:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22281.302715 secs 2023-03-09T21:09:31.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22281.302715 secs 2023-03-09T21:09:32.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:32 smithi122 ceph-mon[93869]: pgmap v12395: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T21:09:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:32 smithi181 ceph-mon[94189]: pgmap v12395: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T21:09:32.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:32 smithi038 ceph-mon[92422]: pgmap v12395: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T21:09:33.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22283.632359 secs 2023-03-09T21:09:33.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22283.632359 secs 2023-03-09T21:09:33.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22283.632359 secs 2023-03-09T21:09:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22745.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:09:34.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:34 smithi122 ceph-mon[93869]: pgmap v12396: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-09T21:09:34.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:34 smithi181 ceph-mon[94189]: pgmap v12396: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-09T21:09:34.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:34 smithi038 ceph-mon[92422]: pgmap v12396: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-09T21:09:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22286.302774 secs 2023-03-09T21:09:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:35 smithi122 ceph-mon[93869]: pgmap v12397: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-09T21:09:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22286.302774 secs 2023-03-09T21:09:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:35 smithi181 ceph-mon[94189]: pgmap v12397: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-09T21:09:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22286.302774 secs 2023-03-09T21:09:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:35 smithi038 ceph-mon[92422]: pgmap v12397: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-09T21:09:38.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22288.632478 secs 2023-03-09T21:09:38.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:38 smithi122 ceph-mon[93869]: pgmap v12398: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 8 op/s 2023-03-09T21:09:38.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22288.632478 secs 2023-03-09T21:09:38.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:38 smithi181 ceph-mon[94189]: pgmap v12398: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 8 op/s 2023-03-09T21:09:38.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22288.632478 secs 2023-03-09T21:09:38.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:38 smithi038 ceph-mon[92422]: pgmap v12398: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 8 op/s 2023-03-09T21:09:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22750.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:09:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:40 smithi122 ceph-mon[93869]: pgmap v12399: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 47 KiB/s wr, 6 op/s 2023-03-09T21:09:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:09:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:09:40.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:40 smithi181 ceph-mon[94189]: pgmap v12399: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 47 KiB/s wr, 6 op/s 2023-03-09T21:09:40.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:09:40.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:09:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:40 smithi038 ceph-mon[92422]: pgmap v12399: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 47 KiB/s wr, 6 op/s 2023-03-09T21:09:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:09:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:09:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22291.302834 secs 2023-03-09T21:09:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:09:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:09:41.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22291.302834 secs 2023-03-09T21:09:41.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:09:41.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:09:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22291.302834 secs 2023-03-09T21:09:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:09:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:09:42.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:42 smithi122 ceph-mon[93869]: pgmap v12400: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 166 KiB/s wr, 9 op/s 2023-03-09T21:09:42.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:42 smithi181 ceph-mon[94189]: pgmap v12400: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 166 KiB/s wr, 9 op/s 2023-03-09T21:09:42.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:42 smithi038 ceph-mon[92422]: pgmap v12400: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 166 KiB/s wr, 9 op/s 2023-03-09T21:09:43.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22293.632613 secs 2023-03-09T21:09:43.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22293.632613 secs 2023-03-09T21:09:43.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22293.632613 secs 2023-03-09T21:09:43.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22755.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:09:44.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:44 smithi181 ceph-mon[94189]: pgmap v12401: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 167 KiB/s wr, 7 op/s 2023-03-09T21:09:44.731 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:44 smithi122 ceph-mon[93869]: pgmap v12401: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 167 KiB/s wr, 7 op/s 2023-03-09T21:09:44.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:44 smithi038 ceph-mon[92422]: pgmap v12401: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 167 KiB/s wr, 7 op/s 2023-03-09T21:09:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22296.302898 secs 2023-03-09T21:09:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:45 smithi122 ceph-mon[93869]: pgmap v12402: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 395 KiB/s rd, 347 KiB/s wr, 8 op/s 2023-03-09T21:09:46.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22296.302898 secs 2023-03-09T21:09:46.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:45 smithi181 ceph-mon[94189]: pgmap v12402: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 395 KiB/s rd, 347 KiB/s wr, 8 op/s 2023-03-09T21:09:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22296.302898 secs 2023-03-09T21:09:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:45 smithi038 ceph-mon[92422]: pgmap v12402: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 395 KiB/s rd, 347 KiB/s wr, 8 op/s 2023-03-09T21:09:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22298.632730 secs 2023-03-09T21:09:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:48 smithi122 ceph-mon[93869]: pgmap v12403: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 416 KiB/s rd, 365 KiB/s wr, 11 op/s 2023-03-09T21:09:48.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22298.632730 secs 2023-03-09T21:09:48.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:48 smithi181 ceph-mon[94189]: pgmap v12403: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 416 KiB/s rd, 365 KiB/s wr, 11 op/s 2023-03-09T21:09:48.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22298.632730 secs 2023-03-09T21:09:48.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:48 smithi038 ceph-mon[92422]: pgmap v12403: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 416 KiB/s rd, 365 KiB/s wr, 11 op/s 2023-03-09T21:09:48.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22760.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:09:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:50 smithi122 ceph-mon[93869]: pgmap v12404: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 364 KiB/s wr, 10 op/s 2023-03-09T21:09:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:50 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:09:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:50 smithi181 ceph-mon[94189]: pgmap v12404: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 364 KiB/s wr, 10 op/s 2023-03-09T21:09:50.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:50 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:09:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:50 smithi038 ceph-mon[92422]: pgmap v12404: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 364 KiB/s wr, 10 op/s 2023-03-09T21:09:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:50 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:09:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22301.302958 secs 2023-03-09T21:09:51.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:09:51.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:09:51.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:09:51.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:09:51.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22301.302958 secs 2023-03-09T21:09:51.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:09:51.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:09:51.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:09:51.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:09:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22301.302958 secs 2023-03-09T21:09:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:09:51.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:09:51.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:09:51.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:09:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:52 smithi122 ceph-mon[93869]: pgmap v12405: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 650 KiB/s rd, 635 KiB/s wr, 14 op/s 2023-03-09T21:09:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:09:52.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:09:52.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:09:52.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:09:52.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:09:52.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:52 smithi181 ceph-mon[94189]: pgmap v12405: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 650 KiB/s rd, 635 KiB/s wr, 14 op/s 2023-03-09T21:09:52.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:09:52.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:09:52.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:09:52.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:09:52.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:09:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:52 smithi038 ceph-mon[92422]: pgmap v12405: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 650 KiB/s rd, 635 KiB/s wr, 14 op/s 2023-03-09T21:09:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:09:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:09:52.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:09:52.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:09:52.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:09:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22303.632849 secs 2023-03-09T21:09:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:09:53.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:09:53.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22303.632849 secs 2023-03-09T21:09:53.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:09:53.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:09:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22303.632849 secs 2023-03-09T21:09:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:09:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:09:53.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22765.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:09:54.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:54 smithi122 ceph-mon[93869]: pgmap v12406: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 530 KiB/s rd, 517 KiB/s wr, 11 op/s 2023-03-09T21:09:54.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:54 smithi181 ceph-mon[94189]: pgmap v12406: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 530 KiB/s rd, 517 KiB/s wr, 11 op/s 2023-03-09T21:09:54.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:54 smithi038 ceph-mon[92422]: pgmap v12406: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 530 KiB/s rd, 517 KiB/s wr, 11 op/s 2023-03-09T21:09:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22306.302984 secs 2023-03-09T21:09:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:55 smithi122 ceph-mon[93869]: pgmap v12407: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 530 KiB/s rd, 516 KiB/s wr, 12 op/s 2023-03-09T21:09:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22306.302984 secs 2023-03-09T21:09:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:55 smithi181 ceph-mon[94189]: pgmap v12407: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 530 KiB/s rd, 516 KiB/s wr, 12 op/s 2023-03-09T21:09:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22306.302984 secs 2023-03-09T21:09:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:55 smithi038 ceph-mon[92422]: pgmap v12407: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 530 KiB/s rd, 516 KiB/s wr, 12 op/s 2023-03-09T21:09:58.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22308.632970 secs 2023-03-09T21:09:58.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:09:58 smithi122 ceph-mon[93869]: pgmap v12408: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 459 KiB/s wr, 14 op/s 2023-03-09T21:09:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22308.632970 secs 2023-03-09T21:09:58.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:09:58 smithi181 ceph-mon[94189]: pgmap v12408: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 459 KiB/s wr, 14 op/s 2023-03-09T21:09:58.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22308.632970 secs 2023-03-09T21:09:58.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:09:58 smithi038 ceph-mon[92422]: pgmap v12408: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 459 KiB/s wr, 14 op/s 2023-03-09T21:09:58.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22770.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:10:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:00 smithi122 ceph-mon[93869]: pgmap v12409: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 456 KiB/s rd, 560 KiB/s wr, 12 op/s 2023-03-09T21:10:00.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:10:00.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:10:00.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:00 smithi122 ceph-mon[93869]: Health detail: HEALTH_WARN 2 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-09T21:10:00.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:00 smithi122 ceph-mon[93869]: [WRN] MDS_SLOW_REQUEST: 2 MDSs report slow requests 2023-03-09T21:10:00.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:00 smithi122 ceph-mon[93869]: mds.i(mds.2): 2 slow requests are blocked > 180 secs 2023-03-09T21:10:00.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:00 smithi122 ceph-mon[93869]: mds.h(mds.1): 2 slow requests are blocked > 180 secs 2023-03-09T21:10:00.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:00 smithi122 ceph-mon[93869]: [WRN] MDS_TRIM: 2 MDSs behind on trimming 2023-03-09T21:10:00.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:00 smithi122 ceph-mon[93869]: mds.e(mds.0): Behind on trimming (531/128) max_segments: 128, num_segments: 531 2023-03-09T21:10:00.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:00 smithi122 ceph-mon[93869]: mds.b(mds.0): Behind on trimming (530/128) max_segments: 128, num_segments: 530 2023-03-09T21:10:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:00 smithi181 ceph-mon[94189]: pgmap v12409: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 456 KiB/s rd, 560 KiB/s wr, 12 op/s 2023-03-09T21:10:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:10:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:10:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:00 smithi181 ceph-mon[94189]: Health detail: HEALTH_WARN 2 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-09T21:10:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:00 smithi181 ceph-mon[94189]: [WRN] MDS_SLOW_REQUEST: 2 MDSs report slow requests 2023-03-09T21:10:00.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:00 smithi181 ceph-mon[94189]: mds.i(mds.2): 2 slow requests are blocked > 180 secs 2023-03-09T21:10:00.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:00 smithi181 ceph-mon[94189]: mds.h(mds.1): 2 slow requests are blocked > 180 secs 2023-03-09T21:10:00.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:00 smithi181 ceph-mon[94189]: [WRN] MDS_TRIM: 2 MDSs behind on trimming 2023-03-09T21:10:00.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:00 smithi181 ceph-mon[94189]: mds.e(mds.0): Behind on trimming (531/128) max_segments: 128, num_segments: 531 2023-03-09T21:10:00.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:00 smithi181 ceph-mon[94189]: mds.b(mds.0): Behind on trimming (530/128) max_segments: 128, num_segments: 530 2023-03-09T21:10:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:00 smithi038 ceph-mon[92422]: pgmap v12409: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 456 KiB/s rd, 560 KiB/s wr, 12 op/s 2023-03-09T21:10:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:10:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:10:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:00 smithi038 ceph-mon[92422]: Health detail: HEALTH_WARN 2 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-09T21:10:00.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:00 smithi038 ceph-mon[92422]: [WRN] MDS_SLOW_REQUEST: 2 MDSs report slow requests 2023-03-09T21:10:00.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:00 smithi038 ceph-mon[92422]: mds.i(mds.2): 2 slow requests are blocked > 180 secs 2023-03-09T21:10:00.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:00 smithi038 ceph-mon[92422]: mds.h(mds.1): 2 slow requests are blocked > 180 secs 2023-03-09T21:10:00.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:00 smithi038 ceph-mon[92422]: [WRN] MDS_TRIM: 2 MDSs behind on trimming 2023-03-09T21:10:00.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:00 smithi038 ceph-mon[92422]: mds.e(mds.0): Behind on trimming (531/128) max_segments: 128, num_segments: 531 2023-03-09T21:10:00.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:00 smithi038 ceph-mon[92422]: mds.b(mds.0): Behind on trimming (530/128) max_segments: 128, num_segments: 530 2023-03-09T21:10:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22311.303046 secs 2023-03-09T21:10:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:10:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:10:01.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22311.303046 secs 2023-03-09T21:10:01.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:10:01.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:10:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22311.303046 secs 2023-03-09T21:10:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:10:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:10:02.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:02 smithi122 ceph-mon[93869]: pgmap v12410: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 598 KiB/s wr, 16 op/s 2023-03-09T21:10:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:02 smithi181 ceph-mon[94189]: pgmap v12410: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 598 KiB/s wr, 16 op/s 2023-03-09T21:10:02.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:02 smithi038 ceph-mon[92422]: pgmap v12410: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 598 KiB/s wr, 16 op/s 2023-03-09T21:10:03.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22313.633092 secs 2023-03-09T21:10:03.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22313.633092 secs 2023-03-09T21:10:03.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22313.633092 secs 2023-03-09T21:10:03.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22775.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:10:04.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:04 smithi122 ceph-mon[93869]: pgmap v12411: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 303 KiB/s wr, 25 op/s 2023-03-09T21:10:04.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:04 smithi181 ceph-mon[94189]: pgmap v12411: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 303 KiB/s wr, 25 op/s 2023-03-09T21:10:04.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:04 smithi038 ceph-mon[92422]: pgmap v12411: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 303 KiB/s wr, 25 op/s 2023-03-09T21:10:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22316.303132 secs 2023-03-09T21:10:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:05 smithi122 ceph-mon[93869]: pgmap v12412: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 302 KiB/s wr, 24 op/s 2023-03-09T21:10:06.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22316.303132 secs 2023-03-09T21:10:06.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:05 smithi181 ceph-mon[94189]: pgmap v12412: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 302 KiB/s wr, 24 op/s 2023-03-09T21:10:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22316.303132 secs 2023-03-09T21:10:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:05 smithi038 ceph-mon[92422]: pgmap v12412: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 302 KiB/s wr, 24 op/s 2023-03-09T21:10:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22318.633157 secs 2023-03-09T21:10:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:08 smithi122 ceph-mon[93869]: pgmap v12413: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 303 KiB/s wr, 26 op/s 2023-03-09T21:10:08.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22318.633157 secs 2023-03-09T21:10:08.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:08 smithi181 ceph-mon[94189]: pgmap v12413: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 303 KiB/s wr, 26 op/s 2023-03-09T21:10:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22318.633157 secs 2023-03-09T21:10:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:08 smithi038 ceph-mon[92422]: pgmap v12413: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 303 KiB/s wr, 26 op/s 2023-03-09T21:10:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22780.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:10:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:10 smithi122 ceph-mon[93869]: pgmap v12414: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 265 KiB/s wr, 23 op/s 2023-03-09T21:10:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:10 smithi122 ceph-mon[93869]: osdmap e989: 12 total, 12 up, 12 in 2023-03-09T21:10:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:10 smithi181 ceph-mon[94189]: pgmap v12414: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 265 KiB/s wr, 23 op/s 2023-03-09T21:10:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:10 smithi181 ceph-mon[94189]: osdmap e989: 12 total, 12 up, 12 in 2023-03-09T21:10:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:10 smithi038 ceph-mon[92422]: pgmap v12414: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 265 KiB/s wr, 23 op/s 2023-03-09T21:10:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:10 smithi038 ceph-mon[92422]: osdmap e989: 12 total, 12 up, 12 in 2023-03-09T21:10:11.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22321.303194 secs 2023-03-09T21:10:11.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22321.303194 secs 2023-03-09T21:10:11.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22321.303194 secs 2023-03-09T21:10:12.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:12 smithi122 ceph-mon[93869]: pgmap v12416: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 129 KiB/s wr, 23 op/s 2023-03-09T21:10:12.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:12 smithi181 ceph-mon[94189]: pgmap v12416: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 129 KiB/s wr, 23 op/s 2023-03-09T21:10:12.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:12 smithi038 ceph-mon[92422]: pgmap v12416: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 129 KiB/s wr, 23 op/s 2023-03-09T21:10:13.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22323.633293 secs 2023-03-09T21:10:13.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22323.633293 secs 2023-03-09T21:10:13.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22323.633293 secs 2023-03-09T21:10:13.801 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22785.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:10:14.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:14 smithi122 ceph-mon[93869]: pgmap v12417: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 102 KiB/s wr, 7 op/s 2023-03-09T21:10:14.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:14 smithi181 ceph-mon[94189]: pgmap v12417: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 102 KiB/s wr, 7 op/s 2023-03-09T21:10:14.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:14 smithi038 ceph-mon[92422]: pgmap v12417: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 102 KiB/s wr, 7 op/s 2023-03-09T21:10:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22326.303256 secs 2023-03-09T21:10:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:15 smithi122 ceph-mon[93869]: pgmap v12418: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 102 KiB/s wr, 7 op/s 2023-03-09T21:10:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22326.303256 secs 2023-03-09T21:10:16.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:15 smithi181 ceph-mon[94189]: pgmap v12418: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 102 KiB/s wr, 7 op/s 2023-03-09T21:10:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22326.303256 secs 2023-03-09T21:10:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:15 smithi038 ceph-mon[92422]: pgmap v12418: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 102 KiB/s wr, 7 op/s 2023-03-09T21:10:18.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22328.633414 secs 2023-03-09T21:10:18.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:18 smithi122 ceph-mon[93869]: pgmap v12419: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 101 KiB/s wr, 7 op/s 2023-03-09T21:10:18.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22328.633414 secs 2023-03-09T21:10:18.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:18 smithi181 ceph-mon[94189]: pgmap v12419: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 101 KiB/s wr, 7 op/s 2023-03-09T21:10:18.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22328.633414 secs 2023-03-09T21:10:18.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:18 smithi038 ceph-mon[92422]: pgmap v12419: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 101 KiB/s wr, 7 op/s 2023-03-09T21:10:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22790.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:10:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:20 smithi122 ceph-mon[93869]: pgmap v12420: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 72 KiB/s wr, 7 op/s 2023-03-09T21:10:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:20 smithi122 ceph-mon[93869]: osdmap e990: 12 total, 12 up, 12 in 2023-03-09T21:10:20.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:20 smithi181 ceph-mon[94189]: pgmap v12420: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 72 KiB/s wr, 7 op/s 2023-03-09T21:10:20.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:20 smithi181 ceph-mon[94189]: osdmap e990: 12 total, 12 up, 12 in 2023-03-09T21:10:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:20 smithi038 ceph-mon[92422]: pgmap v12420: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 72 KiB/s wr, 7 op/s 2023-03-09T21:10:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:20 smithi038 ceph-mon[92422]: osdmap e990: 12 total, 12 up, 12 in 2023-03-09T21:10:21.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22331.303316 secs 2023-03-09T21:10:21.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22331.303316 secs 2023-03-09T21:10:21.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22331.303316 secs 2023-03-09T21:10:22.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:22 smithi122 ceph-mon[93869]: pgmap v12422: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 72 KiB/s wr, 6 op/s 2023-03-09T21:10:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:22 smithi181 ceph-mon[94189]: pgmap v12422: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 72 KiB/s wr, 6 op/s 2023-03-09T21:10:22.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:22 smithi038 ceph-mon[92422]: pgmap v12422: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 72 KiB/s wr, 6 op/s 2023-03-09T21:10:23.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22333.633546 secs 2023-03-09T21:10:23.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22333.633546 secs 2023-03-09T21:10:23.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22333.633546 secs 2023-03-09T21:10:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22795.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:10:24.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:24 smithi122 ceph-mon[93869]: pgmap v12423: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 72 KiB/s wr, 6 op/s 2023-03-09T21:10:24.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:24 smithi181 ceph-mon[94189]: pgmap v12423: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 72 KiB/s wr, 6 op/s 2023-03-09T21:10:24.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:24 smithi038 ceph-mon[92422]: pgmap v12423: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 72 KiB/s wr, 6 op/s 2023-03-09T21:10:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22336.303375 secs 2023-03-09T21:10:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:25 smithi122 ceph-mon[93869]: pgmap v12424: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 72 KiB/s wr, 6 op/s 2023-03-09T21:10:26.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22336.303375 secs 2023-03-09T21:10:26.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:25 smithi181 ceph-mon[94189]: pgmap v12424: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 72 KiB/s wr, 6 op/s 2023-03-09T21:10:26.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22336.303375 secs 2023-03-09T21:10:26.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:25 smithi038 ceph-mon[92422]: pgmap v12424: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 72 KiB/s wr, 6 op/s 2023-03-09T21:10:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22338.633640 secs 2023-03-09T21:10:28.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:28 smithi122 ceph-mon[93869]: pgmap v12425: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 72 KiB/s wr, 6 op/s 2023-03-09T21:10:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22338.633640 secs 2023-03-09T21:10:28.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:28 smithi181 ceph-mon[94189]: pgmap v12425: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 72 KiB/s wr, 6 op/s 2023-03-09T21:10:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22338.633640 secs 2023-03-09T21:10:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:28 smithi038 ceph-mon[92422]: pgmap v12425: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 72 KiB/s wr, 6 op/s 2023-03-09T21:10:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22800.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:10:30.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:30 smithi122 ceph-mon[93869]: pgmap v12426: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-09T21:10:30.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:30 smithi181 ceph-mon[94189]: pgmap v12426: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-09T21:10:30.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:30 smithi038 ceph-mon[92422]: pgmap v12426: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-09T21:10:31.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22341.303450 secs 2023-03-09T21:10:31.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22341.303450 secs 2023-03-09T21:10:31.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22341.303450 secs 2023-03-09T21:10:32.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:32 smithi122 ceph-mon[93869]: pgmap v12427: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2023-03-09T21:10:32.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:32 smithi181 ceph-mon[94189]: pgmap v12427: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2023-03-09T21:10:32.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:32 smithi038 ceph-mon[92422]: pgmap v12427: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2023-03-09T21:10:33.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22343.633766 secs 2023-03-09T21:10:33.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22343.633766 secs 2023-03-09T21:10:33.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22343.633766 secs 2023-03-09T21:10:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22805.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:10:34.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:34 smithi122 ceph-mon[93869]: pgmap v12428: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-09T21:10:34.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:34 smithi181 ceph-mon[94189]: pgmap v12428: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-09T21:10:34.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:34 smithi038 ceph-mon[92422]: pgmap v12428: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-09T21:10:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22346.303508 secs 2023-03-09T21:10:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:35 smithi122 ceph-mon[93869]: pgmap v12429: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-09T21:10:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22346.303508 secs 2023-03-09T21:10:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:35 smithi181 ceph-mon[94189]: pgmap v12429: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-09T21:10:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22346.303508 secs 2023-03-09T21:10:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:35 smithi038 ceph-mon[92422]: pgmap v12429: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-09T21:10:38.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:38 smithi122 ceph-mon[93869]: pgmap v12430: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 767 B/s wr, 8 op/s 2023-03-09T21:10:38.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:38 smithi181 ceph-mon[94189]: pgmap v12430: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 767 B/s wr, 8 op/s 2023-03-09T21:10:38.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:38 smithi038 ceph-mon[92422]: pgmap v12430: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 767 B/s wr, 8 op/s 2023-03-09T21:10:38.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22810.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:10:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:40 smithi122 ceph-mon[93869]: pgmap v12431: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 197 KiB/s wr, 7 op/s 2023-03-09T21:10:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:10:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:10:40.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:40 smithi181 ceph-mon[94189]: pgmap v12431: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 197 KiB/s wr, 7 op/s 2023-03-09T21:10:40.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:10:40.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:10:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:40 smithi038 ceph-mon[92422]: pgmap v12431: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 197 KiB/s wr, 7 op/s 2023-03-09T21:10:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:10:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:10:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22351.303577 secs 2023-03-09T21:10:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:10:41.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:10:41.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:41 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:10:41.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22351.303577 secs 2023-03-09T21:10:41.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:10:41.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:10:41.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:41 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:10:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22351.303577 secs 2023-03-09T21:10:41.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:10:41.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:10:41.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:41 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:10:42.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:42 smithi122 ceph-mon[93869]: pgmap v12432: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 215 KiB/s wr, 9 op/s 2023-03-09T21:10:42.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:42 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:10:42.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:42 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:10:42.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:42 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:10:42.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:42 smithi181 ceph-mon[94189]: pgmap v12432: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 215 KiB/s wr, 9 op/s 2023-03-09T21:10:42.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:42 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:10:42.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:42 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:10:42.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:42 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:10:42.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:42 smithi038 ceph-mon[92422]: pgmap v12432: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 215 KiB/s wr, 9 op/s 2023-03-09T21:10:42.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:42 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:10:42.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:42 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:10:42.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:42 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:10:43.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22815.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:10:44.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:44 smithi122 ceph-mon[93869]: pgmap v12433: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 215 KiB/s wr, 8 op/s 2023-03-09T21:10:44.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:44 smithi181 ceph-mon[94189]: pgmap v12433: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 215 KiB/s wr, 8 op/s 2023-03-09T21:10:44.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:44 smithi038 ceph-mon[92422]: pgmap v12433: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 215 KiB/s wr, 8 op/s 2023-03-09T21:10:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22356.303638 secs 2023-03-09T21:10:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:45 smithi122 ceph-mon[93869]: pgmap v12434: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 239 KiB/s wr, 8 op/s 2023-03-09T21:10:46.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22356.303638 secs 2023-03-09T21:10:46.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:45 smithi181 ceph-mon[94189]: pgmap v12434: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 239 KiB/s wr, 8 op/s 2023-03-09T21:10:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22356.303638 secs 2023-03-09T21:10:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:45 smithi038 ceph-mon[92422]: pgmap v12434: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 239 KiB/s wr, 8 op/s 2023-03-09T21:10:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:48 smithi122 ceph-mon[93869]: pgmap v12435: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 257 KiB/s wr, 11 op/s 2023-03-09T21:10:48.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:48 smithi181 ceph-mon[94189]: pgmap v12435: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 257 KiB/s wr, 11 op/s 2023-03-09T21:10:48.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:48 smithi038 ceph-mon[92422]: pgmap v12435: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 257 KiB/s wr, 11 op/s 2023-03-09T21:10:48.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22820.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:10:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:50 smithi122 ceph-mon[93869]: pgmap v12436: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 543 KiB/s rd, 455 KiB/s wr, 11 op/s 2023-03-09T21:10:50.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:50 smithi181 ceph-mon[94189]: pgmap v12436: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 543 KiB/s rd, 455 KiB/s wr, 11 op/s 2023-03-09T21:10:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:50 smithi038 ceph-mon[92422]: pgmap v12436: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 543 KiB/s rd, 455 KiB/s wr, 11 op/s 2023-03-09T21:10:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22361.303724 secs 2023-03-09T21:10:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22361.303724 secs 2023-03-09T21:10:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22361.303724 secs 2023-03-09T21:10:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:52 smithi122 ceph-mon[93869]: pgmap v12437: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 560 KiB/s wr, 13 op/s 2023-03-09T21:10:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:10:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:10:52.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:10:52.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:10:52.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:10:52.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:10:52.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:52 smithi181 ceph-mon[94189]: pgmap v12437: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 560 KiB/s wr, 13 op/s 2023-03-09T21:10:52.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:10:52.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:10:52.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:10:52.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:10:52.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:10:52.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:10:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:52 smithi038 ceph-mon[92422]: pgmap v12437: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 560 KiB/s wr, 13 op/s 2023-03-09T21:10:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:10:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:10:52.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:10:52.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:10:52.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:10:52.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:10:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:10:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:10:53.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:10:53.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:10:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:10:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:10:53.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22825.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:10:54.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:54 smithi122 ceph-mon[93869]: pgmap v12438: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 629 KiB/s rd, 542 KiB/s wr, 12 op/s 2023-03-09T21:10:54.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:54 smithi181 ceph-mon[94189]: pgmap v12438: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 629 KiB/s rd, 542 KiB/s wr, 12 op/s 2023-03-09T21:10:54.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:54 smithi038 ceph-mon[92422]: pgmap v12438: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 629 KiB/s rd, 542 KiB/s wr, 12 op/s 2023-03-09T21:10:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22366.303808 secs 2023-03-09T21:10:56.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:55 smithi122 ceph-mon[93869]: pgmap v12439: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 706 KiB/s rd, 625 KiB/s wr, 11 op/s 2023-03-09T21:10:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22366.303808 secs 2023-03-09T21:10:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:55 smithi181 ceph-mon[94189]: pgmap v12439: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 706 KiB/s rd, 625 KiB/s wr, 11 op/s 2023-03-09T21:10:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22366.303808 secs 2023-03-09T21:10:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:55 smithi038 ceph-mon[92422]: pgmap v12439: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 706 KiB/s rd, 625 KiB/s wr, 11 op/s 2023-03-09T21:10:58.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22368.634379 secs 2023-03-09T21:10:58.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:10:58 smithi122 ceph-mon[93869]: pgmap v12440: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 762 KiB/s rd, 680 KiB/s wr, 14 op/s 2023-03-09T21:10:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22368.634379 secs 2023-03-09T21:10:58.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:10:58 smithi181 ceph-mon[94189]: pgmap v12440: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 762 KiB/s rd, 680 KiB/s wr, 14 op/s 2023-03-09T21:10:58.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22368.634379 secs 2023-03-09T21:10:58.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:10:58 smithi038 ceph-mon[92422]: pgmap v12440: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 762 KiB/s rd, 680 KiB/s wr, 14 op/s 2023-03-09T21:10:58.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22830.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:11:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:00 smithi122 ceph-mon[93869]: pgmap v12441: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 741 KiB/s rd, 662 KiB/s wr, 12 op/s 2023-03-09T21:11:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:11:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:11:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:00 smithi181 ceph-mon[94189]: pgmap v12441: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 741 KiB/s rd, 662 KiB/s wr, 12 op/s 2023-03-09T21:11:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:11:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:11:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:00 smithi038 ceph-mon[92422]: pgmap v12441: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 741 KiB/s rd, 662 KiB/s wr, 12 op/s 2023-03-09T21:11:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:11:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:11:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22371.303890 secs 2023-03-09T21:11:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:11:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:11:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22371.303890 secs 2023-03-09T21:11:01.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:11:01.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:11:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22371.303890 secs 2023-03-09T21:11:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:11:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:11:02.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:02 smithi122 ceph-mon[93869]: pgmap v12442: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 597 KiB/s rd, 618 KiB/s wr, 15 op/s 2023-03-09T21:11:02.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:02 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:11:02.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:02 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:11:02.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:02 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:11:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:02 smithi181 ceph-mon[94189]: pgmap v12442: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 597 KiB/s rd, 618 KiB/s wr, 15 op/s 2023-03-09T21:11:02.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:02 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:11:02.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:02 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:11:02.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:02 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:11:02.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:02 smithi038 ceph-mon[92422]: pgmap v12442: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 597 KiB/s rd, 618 KiB/s wr, 15 op/s 2023-03-09T21:11:02.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:02 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:11:02.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:02 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:11:02.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:02 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:11:03.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22373.634499 secs 2023-03-09T21:11:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22373.634499 secs 2023-03-09T21:11:03.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22373.634499 secs 2023-03-09T21:11:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22835.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:11:04.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:04 smithi122 ceph-mon[93869]: pgmap v12443: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 338 KiB/s wr, 23 op/s 2023-03-09T21:11:04.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:04 smithi181 ceph-mon[94189]: pgmap v12443: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 338 KiB/s wr, 23 op/s 2023-03-09T21:11:04.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:04 smithi038 ceph-mon[92422]: pgmap v12443: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 338 KiB/s wr, 23 op/s 2023-03-09T21:11:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22376.303952 secs 2023-03-09T21:11:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:05 smithi122 ceph-mon[93869]: pgmap v12444: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 445 KiB/s wr, 24 op/s 2023-03-09T21:11:06.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22376.303952 secs 2023-03-09T21:11:06.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:05 smithi181 ceph-mon[94189]: pgmap v12444: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 445 KiB/s wr, 24 op/s 2023-03-09T21:11:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22376.303952 secs 2023-03-09T21:11:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:05 smithi038 ceph-mon[92422]: pgmap v12444: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 445 KiB/s wr, 24 op/s 2023-03-09T21:11:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22378.634620 secs 2023-03-09T21:11:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:08 smithi122 ceph-mon[93869]: pgmap v12445: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 363 KiB/s wr, 26 op/s 2023-03-09T21:11:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22378.634620 secs 2023-03-09T21:11:08.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:08 smithi181 ceph-mon[94189]: pgmap v12445: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 363 KiB/s wr, 26 op/s 2023-03-09T21:11:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22378.634620 secs 2023-03-09T21:11:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:08 smithi038 ceph-mon[92422]: pgmap v12445: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 363 KiB/s wr, 26 op/s 2023-03-09T21:11:08.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22840.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:11:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:10 smithi122 ceph-mon[93869]: pgmap v12446: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 284 KiB/s wr, 22 op/s 2023-03-09T21:11:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:10 smithi122 ceph-mon[93869]: osdmap e991: 12 total, 12 up, 12 in 2023-03-09T21:11:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:10 smithi181 ceph-mon[94189]: pgmap v12446: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 284 KiB/s wr, 22 op/s 2023-03-09T21:11:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:10 smithi181 ceph-mon[94189]: osdmap e991: 12 total, 12 up, 12 in 2023-03-09T21:11:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:10 smithi038 ceph-mon[92422]: pgmap v12446: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 284 KiB/s wr, 22 op/s 2023-03-09T21:11:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:10 smithi038 ceph-mon[92422]: osdmap e991: 12 total, 12 up, 12 in 2023-03-09T21:11:11.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22381.303984 secs 2023-03-09T21:11:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22381.303984 secs 2023-03-09T21:11:11.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22381.303984 secs 2023-03-09T21:11:12.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:12 smithi122 ceph-mon[93869]: pgmap v12448: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 156 KiB/s wr, 23 op/s 2023-03-09T21:11:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:12 smithi181 ceph-mon[94189]: pgmap v12448: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 156 KiB/s wr, 23 op/s 2023-03-09T21:11:12.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:12 smithi038 ceph-mon[92422]: pgmap v12448: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 156 KiB/s wr, 23 op/s 2023-03-09T21:11:13.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22383.634685 secs 2023-03-09T21:11:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22383.634685 secs 2023-03-09T21:11:13.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22383.634685 secs 2023-03-09T21:11:13.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22845.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:11:14.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:14 smithi122 ceph-mon[93869]: pgmap v12449: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 130 KiB/s wr, 7 op/s 2023-03-09T21:11:14.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:14 smithi181 ceph-mon[94189]: pgmap v12449: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 130 KiB/s wr, 7 op/s 2023-03-09T21:11:14.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:14 smithi038 ceph-mon[92422]: pgmap v12449: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 130 KiB/s wr, 7 op/s 2023-03-09T21:11:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22386.304042 secs 2023-03-09T21:11:16.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:15 smithi122 ceph-mon[93869]: pgmap v12450: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T21:11:16.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22386.304042 secs 2023-03-09T21:11:16.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:15 smithi181 ceph-mon[94189]: pgmap v12450: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T21:11:16.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22386.304042 secs 2023-03-09T21:11:16.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:15 smithi038 ceph-mon[92422]: pgmap v12450: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 44 KiB/s wr, 7 op/s 2023-03-09T21:11:18.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22388.634810 secs 2023-03-09T21:11:18.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:18 smithi122 ceph-mon[93869]: pgmap v12451: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 6 op/s 2023-03-09T21:11:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22388.634810 secs 2023-03-09T21:11:18.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:18 smithi181 ceph-mon[94189]: pgmap v12451: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 6 op/s 2023-03-09T21:11:18.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22388.634810 secs 2023-03-09T21:11:18.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:18 smithi038 ceph-mon[92422]: pgmap v12451: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 6 op/s 2023-03-09T21:11:18.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22850.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:11:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:20 smithi122 ceph-mon[93869]: pgmap v12452: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 6 op/s 2023-03-09T21:11:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:20 smithi122 ceph-mon[93869]: osdmap e992: 12 total, 12 up, 12 in 2023-03-09T21:11:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:20 smithi181 ceph-mon[94189]: pgmap v12452: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 6 op/s 2023-03-09T21:11:20.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:20 smithi181 ceph-mon[94189]: osdmap e992: 12 total, 12 up, 12 in 2023-03-09T21:11:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:20 smithi038 ceph-mon[92422]: pgmap v12452: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 6 op/s 2023-03-09T21:11:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:20 smithi038 ceph-mon[92422]: osdmap e992: 12 total, 12 up, 12 in 2023-03-09T21:11:21.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22391.304102 secs 2023-03-09T21:11:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22391.304102 secs 2023-03-09T21:11:21.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22391.304102 secs 2023-03-09T21:11:22.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:22 smithi122 ceph-mon[93869]: pgmap v12454: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 7 op/s 2023-03-09T21:11:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:22 smithi181 ceph-mon[94189]: pgmap v12454: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 7 op/s 2023-03-09T21:11:22.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:22 smithi038 ceph-mon[92422]: pgmap v12454: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 7 op/s 2023-03-09T21:11:23.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22393.634931 secs 2023-03-09T21:11:23.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22393.634931 secs 2023-03-09T21:11:23.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22393.634931 secs 2023-03-09T21:11:23.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22855.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:11:24.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:24 smithi122 ceph-mon[93869]: pgmap v12455: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 7 op/s 2023-03-09T21:11:24.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:24 smithi181 ceph-mon[94189]: pgmap v12455: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 7 op/s 2023-03-09T21:11:24.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:24 smithi038 ceph-mon[92422]: pgmap v12455: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 58 KiB/s wr, 7 op/s 2023-03-09T21:11:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22396.304162 secs 2023-03-09T21:11:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:25 smithi122 ceph-mon[93869]: pgmap v12456: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T21:11:26.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22396.304162 secs 2023-03-09T21:11:26.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:25 smithi181 ceph-mon[94189]: pgmap v12456: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T21:11:26.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22396.304162 secs 2023-03-09T21:11:26.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:25 smithi038 ceph-mon[92422]: pgmap v12456: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-09T21:11:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22398.635053 secs 2023-03-09T21:11:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:28 smithi122 ceph-mon[93869]: pgmap v12457: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 7 op/s 2023-03-09T21:11:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22398.635053 secs 2023-03-09T21:11:28.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:28 smithi181 ceph-mon[94189]: pgmap v12457: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 7 op/s 2023-03-09T21:11:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22398.635053 secs 2023-03-09T21:11:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:28 smithi038 ceph-mon[92422]: pgmap v12457: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 7 op/s 2023-03-09T21:11:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22860.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:11:30.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:30 smithi122 ceph-mon[93869]: pgmap v12458: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 7 op/s 2023-03-09T21:11:30.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:30 smithi181 ceph-mon[94189]: pgmap v12458: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 7 op/s 2023-03-09T21:11:30.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:30 smithi038 ceph-mon[92422]: pgmap v12458: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 7 op/s 2023-03-09T21:11:31.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22401.304224 secs 2023-03-09T21:11:31.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22401.304224 secs 2023-03-09T21:11:31.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22401.304224 secs 2023-03-09T21:11:32.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:32 smithi122 ceph-mon[93869]: pgmap v12459: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 621 B/s wr, 9 op/s 2023-03-09T21:11:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:32 smithi181 ceph-mon[94189]: pgmap v12459: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 621 B/s wr, 9 op/s 2023-03-09T21:11:32.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:32 smithi038 ceph-mon[92422]: pgmap v12459: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 621 B/s wr, 9 op/s 2023-03-09T21:11:33.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22403.635173 secs 2023-03-09T21:11:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22403.635173 secs 2023-03-09T21:11:33.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22403.635173 secs 2023-03-09T21:11:33.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22865.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:11:34.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:34 smithi122 ceph-mon[93869]: pgmap v12460: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 682 B/s wr, 6 op/s 2023-03-09T21:11:34.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:34 smithi181 ceph-mon[94189]: pgmap v12460: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 682 B/s wr, 6 op/s 2023-03-09T21:11:34.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:34 smithi038 ceph-mon[92422]: pgmap v12460: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 682 B/s wr, 6 op/s 2023-03-09T21:11:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:35 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:11:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:35 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:11:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:35 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:11:35.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:35 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:11:35.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:35 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:11:35.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:35 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:11:36.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22406.304284 secs 2023-03-09T21:11:36.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:36 smithi122 ceph-mon[93869]: pgmap v12461: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 682 B/s wr, 6 op/s 2023-03-09T21:11:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22406.304284 secs 2023-03-09T21:11:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:36 smithi181 ceph-mon[94189]: pgmap v12461: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 682 B/s wr, 6 op/s 2023-03-09T21:11:36.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22406.304284 secs 2023-03-09T21:11:36.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:36 smithi038 ceph-mon[92422]: pgmap v12461: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 682 B/s wr, 6 op/s 2023-03-09T21:11:37.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:11:37.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:11:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:11:37.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:11:37.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:11:37.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:11:38.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:38 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 22408.635296 secs 2023-03-09T21:11:38.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:38 smithi122 ceph-mon[93869]: pgmap v12462: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 89 KiB/s wr, 9 op/s 2023-03-09T21:11:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:38 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 22408.635296 secs 2023-03-09T21:11:38.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:38 smithi181 ceph-mon[94189]: pgmap v12462: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 89 KiB/s wr, 9 op/s 2023-03-09T21:11:38.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:38 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 22408.635296 secs 2023-03-09T21:11:38.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:38 smithi038 ceph-mon[92422]: pgmap v12462: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 89 KiB/s wr, 9 op/s 2023-03-09T21:11:38.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22870.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:11:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:40 smithi122 ceph-mon[93869]: pgmap v12463: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 89 KiB/s wr, 6 op/s 2023-03-09T21:11:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:40 smithi181 ceph-mon[94189]: pgmap v12463: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 89 KiB/s wr, 6 op/s 2023-03-09T21:11:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:40 smithi038 ceph-mon[92422]: pgmap v12463: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 89 KiB/s wr, 6 op/s 2023-03-09T21:11:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22411.304370 secs 2023-03-09T21:11:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22411.304370 secs 2023-03-09T21:11:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22411.304370 secs 2023-03-09T21:11:42.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:42 smithi122 ceph-mon[93869]: pgmap v12464: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 198 KiB/s wr, 9 op/s 2023-03-09T21:11:42.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:42 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:11:42.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:42 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:11:42.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:42 smithi181 ceph-mon[94189]: pgmap v12464: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 198 KiB/s wr, 9 op/s 2023-03-09T21:11:42.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:42 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:11:42.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:42 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:11:42.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:42 smithi038 ceph-mon[92422]: pgmap v12464: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 198 KiB/s wr, 9 op/s 2023-03-09T21:11:42.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:42 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:11:42.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:42 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:11:43.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:43 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 22413.635414 secs 2023-03-09T21:11:43.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:43 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 22413.635414 secs 2023-03-09T21:11:43.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22875.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:11:43.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:43 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 22413.635414 secs 2023-03-09T21:11:44.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:44 smithi038 ceph-mon[92422]: pgmap v12465: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 197 KiB/s wr, 6 op/s 2023-03-09T21:11:44.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:44 smithi122 ceph-mon[93869]: pgmap v12465: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 197 KiB/s wr, 6 op/s 2023-03-09T21:11:44.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:44 smithi181 ceph-mon[94189]: pgmap v12465: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 197 KiB/s wr, 6 op/s 2023-03-09T21:11:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22416.304449 secs 2023-03-09T21:11:46.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:45 smithi122 ceph-mon[93869]: pgmap v12466: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 197 KiB/s wr, 6 op/s 2023-03-09T21:11:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22416.304449 secs 2023-03-09T21:11:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:45 smithi181 ceph-mon[94189]: pgmap v12466: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 197 KiB/s wr, 6 op/s 2023-03-09T21:11:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22416.304449 secs 2023-03-09T21:11:46.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:45 smithi038 ceph-mon[92422]: pgmap v12466: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 197 KiB/s wr, 6 op/s 2023-03-09T21:11:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:48 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 22418.635535 secs 2023-03-09T21:11:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:48 smithi122 ceph-mon[93869]: pgmap v12467: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 468 KiB/s wr, 9 op/s 2023-03-09T21:11:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:48 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 22418.635535 secs 2023-03-09T21:11:48.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:48 smithi181 ceph-mon[94189]: pgmap v12467: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 468 KiB/s wr, 9 op/s 2023-03-09T21:11:48.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:48 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 22418.635535 secs 2023-03-09T21:11:48.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:48 smithi038 ceph-mon[92422]: pgmap v12467: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 468 KiB/s wr, 9 op/s 2023-03-09T21:11:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22880.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:11:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:50 smithi122 ceph-mon[93869]: pgmap v12468: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 478 KiB/s rd, 380 KiB/s wr, 7 op/s 2023-03-09T21:11:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:50 smithi181 ceph-mon[94189]: pgmap v12468: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 478 KiB/s rd, 380 KiB/s wr, 7 op/s 2023-03-09T21:11:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:50 smithi038 ceph-mon[92422]: pgmap v12468: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 478 KiB/s rd, 380 KiB/s wr, 7 op/s 2023-03-09T21:11:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22421.304510 secs 2023-03-09T21:11:51.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:11:51.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:11:51.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22421.304510 secs 2023-03-09T21:11:51.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:11:51.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:11:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22421.304510 secs 2023-03-09T21:11:51.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:11:51.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:11:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:52 smithi122 ceph-mon[93869]: pgmap v12469: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 499 KiB/s rd, 398 KiB/s wr, 10 op/s 2023-03-09T21:11:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:11:52.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:52 smithi181 ceph-mon[94189]: pgmap v12469: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 499 KiB/s rd, 398 KiB/s wr, 10 op/s 2023-03-09T21:11:52.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:11:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:52 smithi038 ceph-mon[92422]: pgmap v12469: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 499 KiB/s rd, 398 KiB/s wr, 10 op/s 2023-03-09T21:11:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:11:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22423.635673 secs 2023-03-09T21:11:53.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:11:53.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:11:53.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:11:53.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:11:53.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:11:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22423.635673 secs 2023-03-09T21:11:53.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:11:53.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:11:53.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:11:53.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:11:53.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:11:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22423.635673 secs 2023-03-09T21:11:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:11:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:11:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:11:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:11:53.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:11:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22885.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:11:54.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:54 smithi122 ceph-mon[93869]: pgmap v12470: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 326 KiB/s wr, 9 op/s 2023-03-09T21:11:54.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:54 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:11:54.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:54 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:11:54.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:54 smithi181 ceph-mon[94189]: pgmap v12470: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 326 KiB/s wr, 9 op/s 2023-03-09T21:11:54.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:54 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:11:54.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:54 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:11:54.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:54 smithi038 ceph-mon[92422]: pgmap v12470: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 326 KiB/s wr, 9 op/s 2023-03-09T21:11:54.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:54 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:11:54.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:54 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:11:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22426.304564 secs 2023-03-09T21:11:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:55 smithi122 ceph-mon[93869]: pgmap v12471: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 326 KiB/s wr, 9 op/s 2023-03-09T21:11:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22426.304564 secs 2023-03-09T21:11:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:55 smithi181 ceph-mon[94189]: pgmap v12471: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 326 KiB/s wr, 9 op/s 2023-03-09T21:11:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22426.304564 secs 2023-03-09T21:11:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:55 smithi038 ceph-mon[92422]: pgmap v12471: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 326 KiB/s wr, 9 op/s 2023-03-09T21:11:58.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22428.635792 secs 2023-03-09T21:11:58.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:58 smithi122 ceph-mon[93869]: pgmap v12472: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 735 KiB/s rd, 624 KiB/s wr, 14 op/s 2023-03-09T21:11:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22428.635792 secs 2023-03-09T21:11:58.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:58 smithi181 ceph-mon[94189]: pgmap v12472: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 735 KiB/s rd, 624 KiB/s wr, 14 op/s 2023-03-09T21:11:58.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22428.635792 secs 2023-03-09T21:11:58.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:58 smithi038 ceph-mon[92422]: pgmap v12472: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 735 KiB/s rd, 624 KiB/s wr, 14 op/s 2023-03-09T21:11:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22890.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:11:59.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:11:59.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:11:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:11:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:11:59.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:11:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:11:59.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:11:59.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:11:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:12:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:00 smithi122 ceph-mon[93869]: pgmap v12473: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 353 KiB/s wr, 10 op/s 2023-03-09T21:12:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:00 smithi181 ceph-mon[94189]: pgmap v12473: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 353 KiB/s wr, 10 op/s 2023-03-09T21:12:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:00 smithi038 ceph-mon[92422]: pgmap v12473: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 353 KiB/s wr, 10 op/s 2023-03-09T21:12:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22431.304624 secs 2023-03-09T21:12:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:12:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:12:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22431.304624 secs 2023-03-09T21:12:01.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:12:01.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:12:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22431.304624 secs 2023-03-09T21:12:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:12:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:12:02.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:02 smithi122 ceph-mon[93869]: pgmap v12474: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 406 KiB/s rd, 413 KiB/s wr, 15 op/s 2023-03-09T21:12:02.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:02 smithi181 ceph-mon[94189]: pgmap v12474: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 406 KiB/s rd, 413 KiB/s wr, 15 op/s 2023-03-09T21:12:02.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:02 smithi038 ceph-mon[92422]: pgmap v12474: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 406 KiB/s rd, 413 KiB/s wr, 15 op/s 2023-03-09T21:12:03.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22433.635895 secs 2023-03-09T21:12:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22433.635895 secs 2023-03-09T21:12:03.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22433.635895 secs 2023-03-09T21:12:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22895.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:12:04.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:04 smithi122 ceph-mon[93869]: pgmap v12475: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 526 KiB/s rd, 551 KiB/s wr, 26 op/s 2023-03-09T21:12:04.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:04 smithi181 ceph-mon[94189]: pgmap v12475: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 526 KiB/s rd, 551 KiB/s wr, 26 op/s 2023-03-09T21:12:04.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:04 smithi038 ceph-mon[92422]: pgmap v12475: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 526 KiB/s rd, 551 KiB/s wr, 26 op/s 2023-03-09T21:12:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:05 smithi122 ceph-mon[93869]: osdmap e993: 12 total, 12 up, 12 in 2023-03-09T21:12:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:05 smithi181 ceph-mon[94189]: osdmap e993: 12 total, 12 up, 12 in 2023-03-09T21:12:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:05 smithi038 ceph-mon[92422]: osdmap e993: 12 total, 12 up, 12 in 2023-03-09T21:12:06.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22436.304684 secs 2023-03-09T21:12:06.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:06 smithi181 ceph-mon[94189]: pgmap v12477: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 579 KiB/s rd, 622 KiB/s wr, 30 op/s 2023-03-09T21:12:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22436.304684 secs 2023-03-09T21:12:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:06 smithi038 ceph-mon[92422]: pgmap v12477: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 579 KiB/s rd, 622 KiB/s wr, 30 op/s 2023-03-09T21:12:06.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22436.304684 secs 2023-03-09T21:12:06.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:06 smithi122 ceph-mon[93869]: pgmap v12477: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 579 KiB/s rd, 622 KiB/s wr, 30 op/s 2023-03-09T21:12:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22438.636041 secs 2023-03-09T21:12:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:08 smithi038 ceph-mon[92422]: pgmap v12478: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 400 KiB/s wr, 29 op/s 2023-03-09T21:12:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22900.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:12:08.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22438.636041 secs 2023-03-09T21:12:08.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:08 smithi122 ceph-mon[93869]: pgmap v12478: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 400 KiB/s wr, 29 op/s 2023-03-09T21:12:08.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22438.636041 secs 2023-03-09T21:12:08.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:08 smithi181 ceph-mon[94189]: pgmap v12478: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 400 KiB/s wr, 29 op/s 2023-03-09T21:12:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:10 smithi038 ceph-mon[92422]: pgmap v12479: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 445 KiB/s rd, 487 KiB/s wr, 30 op/s 2023-03-09T21:12:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:10 smithi038 ceph-mon[92422]: osdmap e994: 12 total, 12 up, 12 in 2023-03-09T21:12:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:10 smithi122 ceph-mon[93869]: pgmap v12479: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 445 KiB/s rd, 487 KiB/s wr, 30 op/s 2023-03-09T21:12:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:10 smithi122 ceph-mon[93869]: osdmap e994: 12 total, 12 up, 12 in 2023-03-09T21:12:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:10 smithi181 ceph-mon[94189]: pgmap v12479: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 445 KiB/s rd, 487 KiB/s wr, 30 op/s 2023-03-09T21:12:10.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:10 smithi181 ceph-mon[94189]: osdmap e994: 12 total, 12 up, 12 in 2023-03-09T21:12:11.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22441.304746 secs 2023-03-09T21:12:11.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:11 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:12:11.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:11 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:12:11.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22441.304746 secs 2023-03-09T21:12:11.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:11 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:12:11.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:11 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:12:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22441.304746 secs 2023-03-09T21:12:11.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:11 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:12:11.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:11 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:12:12.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:12 smithi122 ceph-mon[93869]: pgmap v12481: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 286 KiB/s wr, 10 op/s 2023-03-09T21:12:12.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:12 smithi181 ceph-mon[94189]: pgmap v12481: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 286 KiB/s wr, 10 op/s 2023-03-09T21:12:13.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:12 smithi038 ceph-mon[92422]: pgmap v12481: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 286 KiB/s wr, 10 op/s 2023-03-09T21:12:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22905.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:12:13.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22443.636201 secs 2023-03-09T21:12:13.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:12:13.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:12:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22443.636201 secs 2023-03-09T21:12:13.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:12:13.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:12:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22443.636201 secs 2023-03-09T21:12:14.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:12:14.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:12:14.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:14 smithi122 ceph-mon[93869]: pgmap v12482: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 252 KiB/s wr, 9 op/s 2023-03-09T21:12:14.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:14 smithi181 ceph-mon[94189]: pgmap v12482: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 252 KiB/s wr, 9 op/s 2023-03-09T21:12:15.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:14 smithi038 ceph-mon[92422]: pgmap v12482: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 252 KiB/s wr, 9 op/s 2023-03-09T21:12:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22446.304806 secs 2023-03-09T21:12:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:15 smithi122 ceph-mon[93869]: pgmap v12483: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 296 KiB/s wr, 9 op/s 2023-03-09T21:12:16.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22446.304806 secs 2023-03-09T21:12:16.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:15 smithi181 ceph-mon[94189]: pgmap v12483: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 296 KiB/s wr, 9 op/s 2023-03-09T21:12:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22446.304806 secs 2023-03-09T21:12:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:15 smithi038 ceph-mon[92422]: pgmap v12483: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 296 KiB/s wr, 9 op/s 2023-03-09T21:12:18.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22448.636325 secs 2023-03-09T21:12:18.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:18 smithi122 ceph-mon[93869]: pgmap v12484: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 160 KiB/s wr, 7 op/s 2023-03-09T21:12:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22448.636325 secs 2023-03-09T21:12:18.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:18 smithi181 ceph-mon[94189]: pgmap v12484: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 160 KiB/s wr, 7 op/s 2023-03-09T21:12:18.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22448.636325 secs 2023-03-09T21:12:18.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:18 smithi038 ceph-mon[92422]: pgmap v12484: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 160 KiB/s wr, 7 op/s 2023-03-09T21:12:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22910.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:12:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:20 smithi122 ceph-mon[93869]: pgmap v12485: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 131 KiB/s wr, 6 op/s 2023-03-09T21:12:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:20 smithi181 ceph-mon[94189]: pgmap v12485: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 131 KiB/s wr, 6 op/s 2023-03-09T21:12:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:20 smithi038 ceph-mon[92422]: pgmap v12485: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 131 KiB/s wr, 6 op/s 2023-03-09T21:12:21.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22451.304866 secs 2023-03-09T21:12:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22451.304866 secs 2023-03-09T21:12:21.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22451.304866 secs 2023-03-09T21:12:22.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:22 smithi122 ceph-mon[93869]: pgmap v12486: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 113 KiB/s wr, 8 op/s 2023-03-09T21:12:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:22 smithi181 ceph-mon[94189]: pgmap v12486: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 113 KiB/s wr, 8 op/s 2023-03-09T21:12:22.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:22 smithi038 ceph-mon[92422]: pgmap v12486: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 113 KiB/s wr, 8 op/s 2023-03-09T21:12:23.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22453.636442 secs 2023-03-09T21:12:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22453.636442 secs 2023-03-09T21:12:23.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22453.636442 secs 2023-03-09T21:12:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22915.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:12:24.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:24 smithi122 ceph-mon[93869]: pgmap v12487: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 109 KiB/s wr, 5 op/s 2023-03-09T21:12:24.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:24 smithi181 ceph-mon[94189]: pgmap v12487: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 109 KiB/s wr, 5 op/s 2023-03-09T21:12:24.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:24 smithi038 ceph-mon[92422]: pgmap v12487: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 109 KiB/s wr, 5 op/s 2023-03-09T21:12:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22456.304924 secs 2023-03-09T21:12:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:25 smithi122 ceph-mon[93869]: pgmap v12488: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 109 KiB/s wr, 5 op/s 2023-03-09T21:12:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22456.304924 secs 2023-03-09T21:12:26.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:25 smithi181 ceph-mon[94189]: pgmap v12488: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 109 KiB/s wr, 5 op/s 2023-03-09T21:12:26.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22456.304924 secs 2023-03-09T21:12:26.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:25 smithi038 ceph-mon[92422]: pgmap v12488: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 109 KiB/s wr, 5 op/s 2023-03-09T21:12:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22458.636559 secs 2023-03-09T21:12:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:28 smithi122 ceph-mon[93869]: pgmap v12489: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 48 KiB/s wr, 7 op/s 2023-03-09T21:12:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22458.636559 secs 2023-03-09T21:12:28.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:28 smithi181 ceph-mon[94189]: pgmap v12489: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 48 KiB/s wr, 7 op/s 2023-03-09T21:12:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22458.636559 secs 2023-03-09T21:12:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:28 smithi038 ceph-mon[92422]: pgmap v12489: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 48 KiB/s wr, 7 op/s 2023-03-09T21:12:28.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22920.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:12:30.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:30 smithi122 ceph-mon[93869]: pgmap v12490: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 48 KiB/s wr, 5 op/s 2023-03-09T21:12:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:30 smithi181 ceph-mon[94189]: pgmap v12490: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 48 KiB/s wr, 5 op/s 2023-03-09T21:12:30.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:30 smithi038 ceph-mon[92422]: pgmap v12490: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 48 KiB/s wr, 5 op/s 2023-03-09T21:12:31.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22461.304982 secs 2023-03-09T21:12:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22461.304982 secs 2023-03-09T21:12:31.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22461.304982 secs 2023-03-09T21:12:32.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:32 smithi122 ceph-mon[93869]: pgmap v12491: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T21:12:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:32 smithi181 ceph-mon[94189]: pgmap v12491: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T21:12:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:32 smithi038 ceph-mon[92422]: pgmap v12491: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T21:12:33.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22463.636676 secs 2023-03-09T21:12:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22463.636676 secs 2023-03-09T21:12:33.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22463.636676 secs 2023-03-09T21:12:33.805 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22925.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:12:34.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:34 smithi122 ceph-mon[93869]: pgmap v12492: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-09T21:12:34.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:34 smithi181 ceph-mon[94189]: pgmap v12492: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-09T21:12:34.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:34 smithi038 ceph-mon[92422]: pgmap v12492: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-09T21:12:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22466.305044 secs 2023-03-09T21:12:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:35 smithi122 ceph-mon[93869]: pgmap v12493: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-09T21:12:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22466.305044 secs 2023-03-09T21:12:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:35 smithi181 ceph-mon[94189]: pgmap v12493: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-09T21:12:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22466.305044 secs 2023-03-09T21:12:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:35 smithi038 ceph-mon[92422]: pgmap v12493: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-09T21:12:38.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22468.636797 secs 2023-03-09T21:12:38.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:38 smithi122 ceph-mon[93869]: pgmap v12494: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 597 B/s wr, 8 op/s 2023-03-09T21:12:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22468.636797 secs 2023-03-09T21:12:38.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:38 smithi181 ceph-mon[94189]: pgmap v12494: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 597 B/s wr, 8 op/s 2023-03-09T21:12:38.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22468.636797 secs 2023-03-09T21:12:38.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:38 smithi038 ceph-mon[92422]: pgmap v12494: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 597 B/s wr, 8 op/s 2023-03-09T21:12:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22930.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:12:39.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:39 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:12:39.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:39 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:12:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:39 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:12:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:39 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:12:39.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:39 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:12:39.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:39 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:12:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:40 smithi122 ceph-mon[93869]: pgmap v12495: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 211 KiB/s wr, 7 op/s 2023-03-09T21:12:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:40 smithi181 ceph-mon[94189]: pgmap v12495: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 211 KiB/s wr, 7 op/s 2023-03-09T21:12:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:40 smithi038 ceph-mon[92422]: pgmap v12495: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 211 KiB/s wr, 7 op/s 2023-03-09T21:12:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22471.305106 secs 2023-03-09T21:12:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:12:41.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:12:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22471.305106 secs 2023-03-09T21:12:41.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:12:41.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:12:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22471.305106 secs 2023-03-09T21:12:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:12:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:12:42.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:42 smithi122 ceph-mon[93869]: pgmap v12496: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 228 KiB/s wr, 9 op/s 2023-03-09T21:12:42.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:42 smithi181 ceph-mon[94189]: pgmap v12496: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 228 KiB/s wr, 9 op/s 2023-03-09T21:12:42.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:42 smithi038 ceph-mon[92422]: pgmap v12496: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 228 KiB/s wr, 9 op/s 2023-03-09T21:12:43.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22935.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:12:44.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:44 smithi122 ceph-mon[93869]: pgmap v12497: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 229 KiB/s wr, 7 op/s 2023-03-09T21:12:44.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:44 smithi181 ceph-mon[94189]: pgmap v12497: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 229 KiB/s wr, 7 op/s 2023-03-09T21:12:44.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:44 smithi038 ceph-mon[92422]: pgmap v12497: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 229 KiB/s wr, 7 op/s 2023-03-09T21:12:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:45 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:12:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:45 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:12:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:45 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:12:46.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22476.305189 secs 2023-03-09T21:12:46.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:46 smithi122 ceph-mon[93869]: pgmap v12498: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 253 KiB/s wr, 8 op/s 2023-03-09T21:12:46.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:46 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:12:46.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:46 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:12:46.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:46 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:12:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22476.305189 secs 2023-03-09T21:12:46.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:46 smithi181 ceph-mon[94189]: pgmap v12498: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 253 KiB/s wr, 8 op/s 2023-03-09T21:12:46.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:46 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:12:46.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:46 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:12:46.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:46 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:12:46.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22476.305189 secs 2023-03-09T21:12:46.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:46 smithi038 ceph-mon[92422]: pgmap v12498: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 253 KiB/s wr, 8 op/s 2023-03-09T21:12:46.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:46 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:12:46.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:46 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:12:46.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:46 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:12:48.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:48 smithi181 ceph-mon[94189]: pgmap v12499: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 271 KiB/s wr, 11 op/s 2023-03-09T21:12:48.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:48 smithi038 ceph-mon[92422]: pgmap v12499: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 271 KiB/s wr, 11 op/s 2023-03-09T21:12:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22940.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:12:48.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:48 smithi122 ceph-mon[93869]: pgmap v12499: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 271 KiB/s wr, 11 op/s 2023-03-09T21:12:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:50 smithi038 ceph-mon[92422]: pgmap v12500: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 549 KiB/s rd, 534 KiB/s wr, 10 op/s 2023-03-09T21:12:50.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:50 smithi122 ceph-mon[93869]: pgmap v12500: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 549 KiB/s rd, 534 KiB/s wr, 10 op/s 2023-03-09T21:12:50.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:50 smithi181 ceph-mon[94189]: pgmap v12500: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 549 KiB/s rd, 534 KiB/s wr, 10 op/s 2023-03-09T21:12:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22481.305250 secs 2023-03-09T21:12:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:12:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:12:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22481.305250 secs 2023-03-09T21:12:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:12:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:12:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22481.305250 secs 2023-03-09T21:12:51.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:12:51.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:12:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:52 smithi038 ceph-mon[92422]: pgmap v12501: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 344 KiB/s wr, 12 op/s 2023-03-09T21:12:52.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:52 smithi122 ceph-mon[93869]: pgmap v12501: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 344 KiB/s wr, 12 op/s 2023-03-09T21:12:52.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:52 smithi181 ceph-mon[94189]: pgmap v12501: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 344 KiB/s wr, 12 op/s 2023-03-09T21:12:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22945.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:12:53.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:12:53.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:12:53.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:12:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:12:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:12:53.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:12:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:12:54.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:12:54.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:12:54.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:54 smithi122 ceph-mon[93869]: pgmap v12502: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 559 KiB/s rd, 546 KiB/s wr, 12 op/s 2023-03-09T21:12:54.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:54 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:12:54.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:54 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:12:54.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:54 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:12:54.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:54 smithi181 ceph-mon[94189]: pgmap v12502: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 559 KiB/s rd, 546 KiB/s wr, 12 op/s 2023-03-09T21:12:54.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:54 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:12:54.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:54 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:12:54.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:54 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:12:55.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:54 smithi038 ceph-mon[92422]: pgmap v12502: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 559 KiB/s rd, 546 KiB/s wr, 12 op/s 2023-03-09T21:12:55.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:54 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:12:55.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:54 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:12:55.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:54 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:12:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22486.305309 secs 2023-03-09T21:12:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:55 smithi122 ceph-mon[93869]: pgmap v12503: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 559 KiB/s rd, 546 KiB/s wr, 12 op/s 2023-03-09T21:12:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22486.305309 secs 2023-03-09T21:12:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:55 smithi181 ceph-mon[94189]: pgmap v12503: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 559 KiB/s rd, 546 KiB/s wr, 12 op/s 2023-03-09T21:12:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22486.305309 secs 2023-03-09T21:12:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:55 smithi038 ceph-mon[92422]: pgmap v12503: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 559 KiB/s rd, 546 KiB/s wr, 12 op/s 2023-03-09T21:12:58.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22488.637280 secs 2023-03-09T21:12:58.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:12:58 smithi122 ceph-mon[93869]: pgmap v12504: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 662 KiB/s rd, 648 KiB/s wr, 15 op/s 2023-03-09T21:12:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22488.637280 secs 2023-03-09T21:12:58.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:12:58 smithi181 ceph-mon[94189]: pgmap v12504: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 662 KiB/s rd, 648 KiB/s wr, 15 op/s 2023-03-09T21:12:58.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22488.637280 secs 2023-03-09T21:12:58.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:12:58 smithi038 ceph-mon[92422]: pgmap v12504: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 662 KiB/s rd, 648 KiB/s wr, 15 op/s 2023-03-09T21:12:58.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22950.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:13:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:00 smithi122 ceph-mon[93869]: pgmap v12505: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 733 KiB/s wr, 12 op/s 2023-03-09T21:13:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:00 smithi181 ceph-mon[94189]: pgmap v12505: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 733 KiB/s wr, 12 op/s 2023-03-09T21:13:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:00 smithi038 ceph-mon[92422]: pgmap v12505: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 733 KiB/s wr, 12 op/s 2023-03-09T21:13:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22491.305394 secs 2023-03-09T21:13:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22491.305394 secs 2023-03-09T21:13:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22491.305394 secs 2023-03-09T21:13:02.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:02 smithi122 ceph-mon[93869]: pgmap v12506: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 497 KiB/s wr, 14 op/s 2023-03-09T21:13:02.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:02 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:13:02.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:02 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:13:02.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:02 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:13:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:02 smithi181 ceph-mon[94189]: pgmap v12506: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 497 KiB/s wr, 14 op/s 2023-03-09T21:13:02.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:02 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:13:02.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:02 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:13:02.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:02 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:13:02.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:02 smithi038 ceph-mon[92422]: pgmap v12506: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 497 KiB/s wr, 14 op/s 2023-03-09T21:13:02.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:02 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:13:02.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:02 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:13:02.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:02 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:13:03.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22493.637363 secs 2023-03-09T21:13:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22493.637363 secs 2023-03-09T21:13:03.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22493.637363 secs 2023-03-09T21:13:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22955.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:13:04.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:04 smithi122 ceph-mon[93869]: pgmap v12507: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 512 KiB/s wr, 24 op/s 2023-03-09T21:13:04.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:04 smithi181 ceph-mon[94189]: pgmap v12507: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 512 KiB/s wr, 24 op/s 2023-03-09T21:13:04.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:04 smithi038 ceph-mon[92422]: pgmap v12507: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 512 KiB/s wr, 24 op/s 2023-03-09T21:13:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22496.305461 secs 2023-03-09T21:13:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:05 smithi122 ceph-mon[93869]: pgmap v12508: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 379 KiB/s wr, 23 op/s 2023-03-09T21:13:06.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22496.305461 secs 2023-03-09T21:13:06.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:05 smithi181 ceph-mon[94189]: pgmap v12508: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 379 KiB/s wr, 23 op/s 2023-03-09T21:13:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22496.305461 secs 2023-03-09T21:13:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:05 smithi038 ceph-mon[92422]: pgmap v12508: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 379 KiB/s wr, 23 op/s 2023-03-09T21:13:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:13:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:13:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22498.637478 secs 2023-03-09T21:13:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:07 smithi122 ceph-mon[93869]: pgmap v12509: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 391 KiB/s wr, 27 op/s 2023-03-09T21:13:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:13:08.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:13:08.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22498.637478 secs 2023-03-09T21:13:08.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:07 smithi181 ceph-mon[94189]: pgmap v12509: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 391 KiB/s wr, 27 op/s 2023-03-09T21:13:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:13:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:13:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22498.637478 secs 2023-03-09T21:13:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:07 smithi038 ceph-mon[92422]: pgmap v12509: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 391 KiB/s wr, 27 op/s 2023-03-09T21:13:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22960.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:13:09.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:13:09.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:13:09.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:13:09.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:13:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:13:09.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:13:10.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:09 smithi122 ceph-mon[93869]: pgmap v12510: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 264 KiB/s wr, 23 op/s 2023-03-09T21:13:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:09 smithi181 ceph-mon[94189]: pgmap v12510: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 264 KiB/s wr, 23 op/s 2023-03-09T21:13:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:09 smithi038 ceph-mon[92422]: pgmap v12510: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 264 KiB/s wr, 23 op/s 2023-03-09T21:13:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:10 smithi122 ceph-mon[93869]: osdmap e995: 12 total, 12 up, 12 in 2023-03-09T21:13:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22501.305525 secs 2023-03-09T21:13:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:10 smithi181 ceph-mon[94189]: osdmap e995: 12 total, 12 up, 12 in 2023-03-09T21:13:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22501.305525 secs 2023-03-09T21:13:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:10 smithi038 ceph-mon[92422]: osdmap e995: 12 total, 12 up, 12 in 2023-03-09T21:13:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22501.305525 secs 2023-03-09T21:13:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:11 smithi122 ceph-mon[93869]: pgmap v12512: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 161 KiB/s wr, 25 op/s 2023-03-09T21:13:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:11 smithi181 ceph-mon[94189]: pgmap v12512: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 161 KiB/s wr, 25 op/s 2023-03-09T21:13:12.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:11 smithi038 ceph-mon[92422]: pgmap v12512: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 161 KiB/s wr, 25 op/s 2023-03-09T21:13:13.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22503.637598 secs 2023-03-09T21:13:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22503.637598 secs 2023-03-09T21:13:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22503.637598 secs 2023-03-09T21:13:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22965.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:13:14.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:13 smithi122 ceph-mon[93869]: pgmap v12513: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 118 KiB/s wr, 9 op/s 2023-03-09T21:13:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:13 smithi181 ceph-mon[94189]: pgmap v12513: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 118 KiB/s wr, 9 op/s 2023-03-09T21:13:14.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:13 smithi038 ceph-mon[92422]: pgmap v12513: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 118 KiB/s wr, 9 op/s 2023-03-09T21:13:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22506.305580 secs 2023-03-09T21:13:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:15 smithi122 ceph-mon[93869]: pgmap v12514: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 87 KiB/s wr, 9 op/s 2023-03-09T21:13:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22506.305580 secs 2023-03-09T21:13:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:15 smithi181 ceph-mon[94189]: pgmap v12514: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 87 KiB/s wr, 9 op/s 2023-03-09T21:13:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22506.305580 secs 2023-03-09T21:13:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:15 smithi038 ceph-mon[92422]: pgmap v12514: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 87 KiB/s wr, 9 op/s 2023-03-09T21:13:18.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22508.637715 secs 2023-03-09T21:13:18.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:18 smithi122 ceph-mon[93869]: pgmap v12515: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 73 KiB/s wr, 6 op/s 2023-03-09T21:13:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22508.637715 secs 2023-03-09T21:13:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:18 smithi181 ceph-mon[94189]: pgmap v12515: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 73 KiB/s wr, 6 op/s 2023-03-09T21:13:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22508.637715 secs 2023-03-09T21:13:18.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:18 smithi038 ceph-mon[92422]: pgmap v12515: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 73 KiB/s wr, 6 op/s 2023-03-09T21:13:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22970.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:13:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:20 smithi122 ceph-mon[93869]: pgmap v12516: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 73 KiB/s wr, 6 op/s 2023-03-09T21:13:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:20 smithi122 ceph-mon[93869]: osdmap e996: 12 total, 12 up, 12 in 2023-03-09T21:13:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:20 smithi181 ceph-mon[94189]: pgmap v12516: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 73 KiB/s wr, 6 op/s 2023-03-09T21:13:20.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:20 smithi181 ceph-mon[94189]: osdmap e996: 12 total, 12 up, 12 in 2023-03-09T21:13:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:20 smithi038 ceph-mon[92422]: pgmap v12516: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 73 KiB/s wr, 6 op/s 2023-03-09T21:13:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:20 smithi038 ceph-mon[92422]: osdmap e996: 12 total, 12 up, 12 in 2023-03-09T21:13:21.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22511.305642 secs 2023-03-09T21:13:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22511.305642 secs 2023-03-09T21:13:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22511.305642 secs 2023-03-09T21:13:22.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:22 smithi122 ceph-mon[93869]: pgmap v12518: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 73 KiB/s wr, 6 op/s 2023-03-09T21:13:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:22 smithi181 ceph-mon[94189]: pgmap v12518: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 73 KiB/s wr, 6 op/s 2023-03-09T21:13:22.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:22 smithi038 ceph-mon[92422]: pgmap v12518: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 73 KiB/s wr, 6 op/s 2023-03-09T21:13:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22513.637836 secs 2023-03-09T21:13:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22513.637836 secs 2023-03-09T21:13:23.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22513.637836 secs 2023-03-09T21:13:23.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22975.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:13:24.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:24 smithi122 ceph-mon[93869]: pgmap v12519: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 73 KiB/s wr, 6 op/s 2023-03-09T21:13:24.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:24 smithi181 ceph-mon[94189]: pgmap v12519: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 73 KiB/s wr, 6 op/s 2023-03-09T21:13:24.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:24 smithi038 ceph-mon[92422]: pgmap v12519: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 73 KiB/s wr, 6 op/s 2023-03-09T21:13:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22516.305701 secs 2023-03-09T21:13:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:25 smithi122 ceph-mon[93869]: pgmap v12520: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 6 op/s 2023-03-09T21:13:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22516.305701 secs 2023-03-09T21:13:26.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:25 smithi181 ceph-mon[94189]: pgmap v12520: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 6 op/s 2023-03-09T21:13:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22516.305701 secs 2023-03-09T21:13:26.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:25 smithi038 ceph-mon[92422]: pgmap v12520: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 6 op/s 2023-03-09T21:13:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22518.637973 secs 2023-03-09T21:13:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:28 smithi122 ceph-mon[93869]: pgmap v12521: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T21:13:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22518.637973 secs 2023-03-09T21:13:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:28 smithi181 ceph-mon[94189]: pgmap v12521: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T21:13:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22518.637973 secs 2023-03-09T21:13:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:28 smithi038 ceph-mon[92422]: pgmap v12521: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T21:13:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22980.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:13:30.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:30 smithi122 ceph-mon[93869]: pgmap v12522: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T21:13:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:30 smithi181 ceph-mon[94189]: pgmap v12522: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T21:13:30.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:30 smithi038 ceph-mon[92422]: pgmap v12522: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T21:13:31.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22521.305762 secs 2023-03-09T21:13:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22521.305762 secs 2023-03-09T21:13:31.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22521.305762 secs 2023-03-09T21:13:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:32 smithi122 ceph-mon[93869]: pgmap v12523: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 620 B/s wr, 8 op/s 2023-03-09T21:13:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:32 smithi181 ceph-mon[94189]: pgmap v12523: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 620 B/s wr, 8 op/s 2023-03-09T21:13:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:32 smithi038 ceph-mon[92422]: pgmap v12523: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 620 B/s wr, 8 op/s 2023-03-09T21:13:33.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22523.638091 secs 2023-03-09T21:13:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22523.638091 secs 2023-03-09T21:13:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22523.638091 secs 2023-03-09T21:13:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22985.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:13:34.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:34 smithi122 ceph-mon[93869]: pgmap v12524: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 682 B/s wr, 6 op/s 2023-03-09T21:13:34.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:34 smithi181 ceph-mon[94189]: pgmap v12524: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 682 B/s wr, 6 op/s 2023-03-09T21:13:34.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:34 smithi038 ceph-mon[92422]: pgmap v12524: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 682 B/s wr, 6 op/s 2023-03-09T21:13:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22526.305843 secs 2023-03-09T21:13:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:35 smithi122 ceph-mon[93869]: pgmap v12525: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 89 KiB/s wr, 6 op/s 2023-03-09T21:13:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22526.305843 secs 2023-03-09T21:13:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:35 smithi181 ceph-mon[94189]: pgmap v12525: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 89 KiB/s wr, 6 op/s 2023-03-09T21:13:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22526.305843 secs 2023-03-09T21:13:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:35 smithi038 ceph-mon[92422]: pgmap v12525: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 89 KiB/s wr, 6 op/s 2023-03-09T21:13:38.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:38 smithi122 ceph-mon[93869]: pgmap v12526: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 90 KiB/s wr, 8 op/s 2023-03-09T21:13:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:38 smithi181 ceph-mon[94189]: pgmap v12526: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 90 KiB/s wr, 8 op/s 2023-03-09T21:13:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:38 smithi038 ceph-mon[92422]: pgmap v12526: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 90 KiB/s wr, 8 op/s 2023-03-09T21:13:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22990.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:13:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:40 smithi122 ceph-mon[93869]: pgmap v12527: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 93 KiB/s wr, 6 op/s 2023-03-09T21:13:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:40 smithi181 ceph-mon[94189]: pgmap v12527: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 93 KiB/s wr, 6 op/s 2023-03-09T21:13:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:40 smithi038 ceph-mon[92422]: pgmap v12527: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 93 KiB/s wr, 6 op/s 2023-03-09T21:13:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22531.305922 secs 2023-03-09T21:13:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:13:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:13:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:41 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:13:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22531.305922 secs 2023-03-09T21:13:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:13:41.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:13:41.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:41 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:13:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22531.305922 secs 2023-03-09T21:13:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:13:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:13:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:41 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:13:42.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:42 smithi122 ceph-mon[93869]: pgmap v12528: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 93 KiB/s wr, 8 op/s 2023-03-09T21:13:42.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:42 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:13:42.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:42 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:13:42.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:42 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:13:42.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:42 smithi181 ceph-mon[94189]: pgmap v12528: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 93 KiB/s wr, 8 op/s 2023-03-09T21:13:42.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:42 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:13:42.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:42 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:13:42.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:42 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:13:42.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:42 smithi038 ceph-mon[92422]: pgmap v12528: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 93 KiB/s wr, 8 op/s 2023-03-09T21:13:42.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:42 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:13:42.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:42 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:13:42.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:42 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:13:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 22995.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:13:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:44 smithi122 ceph-mon[93869]: pgmap v12529: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 92 KiB/s wr, 6 op/s 2023-03-09T21:13:44.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:44 smithi181 ceph-mon[94189]: pgmap v12529: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 92 KiB/s wr, 6 op/s 2023-03-09T21:13:44.753 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:44 smithi038 ceph-mon[92422]: pgmap v12529: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 92 KiB/s wr, 6 op/s 2023-03-09T21:13:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22536.305980 secs 2023-03-09T21:13:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:45 smithi122 ceph-mon[93869]: pgmap v12530: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 187 KiB/s wr, 7 op/s 2023-03-09T21:13:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22536.305980 secs 2023-03-09T21:13:46.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:45 smithi181 ceph-mon[94189]: pgmap v12530: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 187 KiB/s wr, 7 op/s 2023-03-09T21:13:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22536.305980 secs 2023-03-09T21:13:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:45 smithi038 ceph-mon[92422]: pgmap v12530: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 187 KiB/s wr, 7 op/s 2023-03-09T21:13:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:13:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:13:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:47 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 22538.638475 secs 2023-03-09T21:13:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:47 smithi122 ceph-mon[93869]: pgmap v12531: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 123 KiB/s wr, 9 op/s 2023-03-09T21:13:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:13:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:13:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:47 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 22538.638475 secs 2023-03-09T21:13:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:47 smithi181 ceph-mon[94189]: pgmap v12531: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 123 KiB/s wr, 9 op/s 2023-03-09T21:13:48.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:13:48.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:13:48.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:47 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 22538.638475 secs 2023-03-09T21:13:48.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:47 smithi038 ceph-mon[92422]: pgmap v12531: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 123 KiB/s wr, 9 op/s 2023-03-09T21:13:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23000.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:13:49.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:48 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:13:49.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:13:49.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:13:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:48 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:13:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:13:49.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:13:49.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:48 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:13:49.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:13:49.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:13:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:49 smithi122 ceph-mon[93869]: pgmap v12532: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 371 KiB/s wr, 8 op/s 2023-03-09T21:13:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:49 smithi181 ceph-mon[94189]: pgmap v12532: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 371 KiB/s wr, 8 op/s 2023-03-09T21:13:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:49 smithi038 ceph-mon[92422]: pgmap v12532: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 371 KiB/s wr, 8 op/s 2023-03-09T21:13:51.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22541.306040 secs 2023-03-09T21:13:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22541.306040 secs 2023-03-09T21:13:51.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22541.306040 secs 2023-03-09T21:13:52.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:13:52.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:13:52.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:51 smithi122 ceph-mon[93869]: pgmap v12533: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 554 KiB/s rd, 546 KiB/s wr, 12 op/s 2023-03-09T21:13:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:13:52.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:13:52.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:51 smithi181 ceph-mon[94189]: pgmap v12533: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 554 KiB/s rd, 546 KiB/s wr, 12 op/s 2023-03-09T21:13:52.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:13:52.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:13:52.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:51 smithi038 ceph-mon[92422]: pgmap v12533: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 554 KiB/s rd, 546 KiB/s wr, 12 op/s 2023-03-09T21:13:53.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:13:53.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:13:53.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22543.638617 secs 2023-03-09T21:13:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:13:53.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:13:53.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22543.638617 secs 2023-03-09T21:13:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:13:53.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:13:53.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22543.638617 secs 2023-03-09T21:13:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23005.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:13:54.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:13:54.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:13:54.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:53 smithi122 ceph-mon[93869]: pgmap v12534: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 547 KiB/s wr, 10 op/s 2023-03-09T21:13:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:13:54.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:13:54.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:53 smithi181 ceph-mon[94189]: pgmap v12534: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 547 KiB/s wr, 10 op/s 2023-03-09T21:13:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:13:54.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:13:54.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:53 smithi038 ceph-mon[92422]: pgmap v12534: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 547 KiB/s wr, 10 op/s 2023-03-09T21:13:55.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:54 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:13:55.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:54 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:13:55.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:54 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:13:55.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:54 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:13:55.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:54 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:13:55.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:54 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:13:55.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:54 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:13:55.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:54 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:13:55.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:54 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:13:55.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:54 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:13:55.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:54 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:13:55.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:54 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:13:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22546.306100 secs 2023-03-09T21:13:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:55 smithi122 ceph-mon[93869]: pgmap v12535: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 547 KiB/s wr, 10 op/s 2023-03-09T21:13:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22546.306100 secs 2023-03-09T21:13:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:55 smithi181 ceph-mon[94189]: pgmap v12535: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 547 KiB/s wr, 10 op/s 2023-03-09T21:13:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22546.306100 secs 2023-03-09T21:13:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:55 smithi038 ceph-mon[92422]: pgmap v12535: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 547 KiB/s wr, 10 op/s 2023-03-09T21:13:58.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22548.638756 secs 2023-03-09T21:13:58.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:58 smithi122 ceph-mon[93869]: pgmap v12536: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 724 KiB/s rd, 711 KiB/s wr, 14 op/s 2023-03-09T21:13:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22548.638756 secs 2023-03-09T21:13:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:58 smithi181 ceph-mon[94189]: pgmap v12536: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 724 KiB/s rd, 711 KiB/s wr, 14 op/s 2023-03-09T21:13:58.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22548.638756 secs 2023-03-09T21:13:58.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:58 smithi038 ceph-mon[92422]: pgmap v12536: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 724 KiB/s rd, 711 KiB/s wr, 14 op/s 2023-03-09T21:13:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23010.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:13:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:13:59.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:13:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:13:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:13:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:13:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:13:59.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:13:59.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:13:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:14:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:14:00 smithi038 ceph-mon[92422]: pgmap v12537: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 685 KiB/s wr, 11 op/s 2023-03-09T21:14:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:14:00 smithi122 ceph-mon[93869]: pgmap v12537: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 685 KiB/s wr, 11 op/s 2023-03-09T21:14:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:14:00 smithi181 ceph-mon[94189]: pgmap v12537: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 685 KiB/s wr, 11 op/s 2023-03-09T21:14:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:14:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22551.306159 secs 2023-03-09T21:14:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:14:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:14:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:14:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:14:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:14:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22551.306159 secs 2023-03-09T21:14:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:14:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:14:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:14:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:14:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:14:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22551.306159 secs 2023-03-09T21:14:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:14:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:14:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:14:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:14:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:14:02 smithi038 ceph-mon[92422]: pgmap v12538: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 520 KiB/s wr, 14 op/s 2023-03-09T21:14:02.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:14:02 smithi122 ceph-mon[93869]: pgmap v12538: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 520 KiB/s wr, 14 op/s 2023-03-09T21:14:02.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:14:02 smithi181 ceph-mon[94189]: pgmap v12538: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 520 KiB/s wr, 14 op/s 2023-03-09T21:14:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:14:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22553.638875 secs 2023-03-09T21:14:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23015.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:14:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:14:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22553.638875 secs 2023-03-09T21:14:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:14:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22553.638875 secs 2023-03-09T21:14:04.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:14:04 smithi122 ceph-mon[93869]: pgmap v12539: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 363 KiB/s wr, 22 op/s 2023-03-09T21:14:04.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:14:04 smithi181 ceph-mon[94189]: pgmap v12539: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 363 KiB/s wr, 22 op/s 2023-03-09T21:14:05.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:14:04 smithi038 ceph-mon[92422]: pgmap v12539: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 363 KiB/s wr, 22 op/s 2023-03-09T21:14:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:14:05 smithi122 ceph-mon[93869]: osdmap e997: 12 total, 12 up, 12 in 2023-03-09T21:14:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:14:05 smithi181 ceph-mon[94189]: osdmap e997: 12 total, 12 up, 12 in 2023-03-09T21:14:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:14:05 smithi038 ceph-mon[92422]: osdmap e997: 12 total, 12 up, 12 in 2023-03-09T21:14:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:14:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22556.306219 secs 2023-03-09T21:14:06.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:14:06 smithi122 ceph-mon[93869]: pgmap v12541: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 434 KiB/s wr, 26 op/s 2023-03-09T21:14:06.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:14:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22556.306219 secs 2023-03-09T21:14:06.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:14:06 smithi181 ceph-mon[94189]: pgmap v12541: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 434 KiB/s wr, 26 op/s 2023-03-09T21:14:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:14:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22556.306219 secs 2023-03-09T21:14:07.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:14:06 smithi038 ceph-mon[92422]: pgmap v12541: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 434 KiB/s wr, 26 op/s 2023-03-09T21:14:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23020.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:14:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:14:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22558.638956 secs 2023-03-09T21:14:08.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:14:08 smithi122 ceph-mon[93869]: pgmap v12542: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 397 KiB/s wr, 27 op/s 2023-03-09T21:14:08.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:14:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22558.638956 secs 2023-03-09T21:14:08.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:14:08 smithi181 ceph-mon[94189]: pgmap v12542: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 397 KiB/s wr, 27 op/s 2023-03-09T21:14:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:14:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22558.638956 secs 2023-03-09T21:14:09.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:14:08 smithi038 ceph-mon[92422]: pgmap v12542: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 397 KiB/s wr, 27 op/s 2023-03-09T21:14:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:14:10 smithi122 ceph-mon[93869]: pgmap v12543: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 397 KiB/s wr, 27 op/s 2023-03-09T21:14:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:14:10 smithi122 ceph-mon[93869]: osdmap e998: 12 total, 12 up, 12 in 2023-03-09T21:14:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:14:10 smithi181 ceph-mon[94189]: pgmap v12543: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 397 KiB/s wr, 27 op/s 2023-03-09T21:14:10.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:14:10 smithi181 ceph-mon[94189]: osdmap e998: 12 total, 12 up, 12 in 2023-03-09T21:14:11.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:14:10 smithi038 ceph-mon[92422]: pgmap v12543: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 397 KiB/s wr, 27 op/s 2023-03-09T21:14:11.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:14:10 smithi038 ceph-mon[92422]: osdmap e998: 12 total, 12 up, 12 in 2023-03-09T21:14:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:14:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22561.306281 secs 2023-03-09T21:14:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:14:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22561.306281 secs 2023-03-09T21:14:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:14:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22561.306281 secs 2023-03-09T21:14:12.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:14:12 smithi122 ceph-mon[93869]: pgmap v12545: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 402 KiB/s rd, 432 KiB/s wr, 15 op/s 2023-03-09T21:14:12.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:14:12 smithi181 ceph-mon[94189]: pgmap v12545: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 402 KiB/s rd, 432 KiB/s wr, 15 op/s 2023-03-09T21:14:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:14:12 smithi038 ceph-mon[92422]: pgmap v12545: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 402 KiB/s rd, 432 KiB/s wr, 15 op/s 2023-03-09T21:14:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23025.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:14:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:14:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22563.639077 secs 2023-03-09T21:14:13.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:14:13 smithi122 ceph-mon[93869]: pgmap v12546: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 362 KiB/s rd, 389 KiB/s wr, 13 op/s 2023-03-09T21:14:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:14:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22563.639077 secs 2023-03-09T21:14:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:14:13 smithi181 ceph-mon[94189]: pgmap v12546: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 362 KiB/s rd, 389 KiB/s wr, 13 op/s 2023-03-09T21:14:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:14:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22563.639077 secs 2023-03-09T21:14:14.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:14:13 smithi038 ceph-mon[92422]: pgmap v12546: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 362 KiB/s rd, 389 KiB/s wr, 13 op/s 2023-03-09T21:14:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:14:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22566.306342 secs 2023-03-09T21:14:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:14:15 smithi122 ceph-mon[93869]: pgmap v12547: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 346 KiB/s wr, 12 op/s 2023-03-09T21:14:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:14:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22566.306342 secs 2023-03-09T21:14:16.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:14:15 smithi181 ceph-mon[94189]: pgmap v12547: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 346 KiB/s wr, 12 op/s 2023-03-09T21:14:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:14:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22566.306342 secs 2023-03-09T21:14:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:14:15 smithi038 ceph-mon[92422]: pgmap v12547: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 346 KiB/s wr, 12 op/s 2023-03-09T21:14:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:14:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22568.639190 secs 2023-03-09T21:14:18.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:14:18 smithi122 ceph-mon[93869]: pgmap v12548: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 74 KiB/s wr, 7 op/s 2023-03-09T21:14:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:14:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22568.639190 secs 2023-03-09T21:14:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:14:18 smithi181 ceph-mon[94189]: pgmap v12548: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 74 KiB/s wr, 7 op/s 2023-03-09T21:14:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:14:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22568.639190 secs 2023-03-09T21:14:18.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:14:18 smithi038 ceph-mon[92422]: pgmap v12548: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 74 KiB/s wr, 7 op/s 2023-03-09T21:14:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23030.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:14:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:14:20 smithi122 ceph-mon[93869]: pgmap v12549: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 73 KiB/s wr, 7 op/s 2023-03-09T21:14:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:14:20 smithi181 ceph-mon[94189]: pgmap v12549: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 73 KiB/s wr, 7 op/s 2023-03-09T21:14:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:14:20 smithi038 ceph-mon[92422]: pgmap v12549: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 73 KiB/s wr, 7 op/s 2023-03-09T21:14:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:14:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22571.306401 secs 2023-03-09T21:14:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:14:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22571.306401 secs 2023-03-09T21:14:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:14:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22571.306401 secs 2023-03-09T21:14:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:14:22 smithi122 ceph-mon[93869]: pgmap v12550: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 64 KiB/s wr, 9 op/s 2023-03-09T21:14:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:14:22 smithi181 ceph-mon[94189]: pgmap v12550: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 64 KiB/s wr, 9 op/s 2023-03-09T21:14:22.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:14:22 smithi038 ceph-mon[92422]: pgmap v12550: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 64 KiB/s wr, 9 op/s 2023-03-09T21:14:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:14:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22573.639264 secs 2023-03-09T21:14:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:14:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22573.639264 secs 2023-03-09T21:14:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:14:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22573.639264 secs 2023-03-09T21:14:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23035.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:14:24.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:14:24 smithi181 ceph-mon[94189]: pgmap v12551: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T21:14:24.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:14:24 smithi038 ceph-mon[92422]: pgmap v12551: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T21:14:24.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:14:24 smithi122 ceph-mon[93869]: pgmap v12551: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T21:14:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:14:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22576.306465 secs 2023-03-09T21:14:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:14:25 smithi122 ceph-mon[93869]: pgmap v12552: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T21:14:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:14:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22576.306465 secs 2023-03-09T21:14:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:14:25 smithi181 ceph-mon[94189]: pgmap v12552: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T21:14:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:14:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22576.306465 secs 2023-03-09T21:14:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:14:25 smithi038 ceph-mon[92422]: pgmap v12552: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T21:14:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:14:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22578.639375 secs 2023-03-09T21:14:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:14:28 smithi122 ceph-mon[93869]: pgmap v12553: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T21:14:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:14:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22578.639375 secs 2023-03-09T21:14:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:14:28 smithi181 ceph-mon[94189]: pgmap v12553: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T21:14:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:14:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22578.639375 secs 2023-03-09T21:14:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:14:28 smithi038 ceph-mon[92422]: pgmap v12553: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T21:14:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23040.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:14:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:14:30 smithi122 ceph-mon[93869]: pgmap v12554: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2023-03-09T21:14:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:14:30 smithi181 ceph-mon[94189]: pgmap v12554: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2023-03-09T21:14:30.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:14:30 smithi038 ceph-mon[92422]: pgmap v12554: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2023-03-09T21:14:31.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:14:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22581.306544 secs 2023-03-09T21:14:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:14:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22581.306544 secs 2023-03-09T21:14:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:14:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22581.306544 secs 2023-03-09T21:14:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:14:32 smithi122 ceph-mon[93869]: pgmap v12555: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T21:14:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:14:32 smithi181 ceph-mon[94189]: pgmap v12555: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T21:14:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:14:32 smithi038 ceph-mon[92422]: pgmap v12555: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T21:14:33.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:14:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22583.639497 secs 2023-03-09T21:14:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23045.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:14:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:14:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22583.639497 secs 2023-03-09T21:14:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:14:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22583.639497 secs 2023-03-09T21:14:34.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:14:34 smithi038 ceph-mon[92422]: pgmap v12556: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T21:14:34.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:14:34 smithi122 ceph-mon[93869]: pgmap v12556: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T21:14:34.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:14:34 smithi181 ceph-mon[94189]: pgmap v12556: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T21:14:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:14:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22586.306613 secs 2023-03-09T21:14:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:14:35 smithi122 ceph-mon[93869]: pgmap v12557: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T21:14:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:14:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22586.306613 secs 2023-03-09T21:14:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:14:35 smithi181 ceph-mon[94189]: pgmap v12557: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T21:14:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:14:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22586.306613 secs 2023-03-09T21:14:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:14:35 smithi038 ceph-mon[92422]: pgmap v12557: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T21:14:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:14:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22588.639631 secs 2023-03-09T21:14:38.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:14:38 smithi122 ceph-mon[93869]: pgmap v12558: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 9 op/s 2023-03-09T21:14:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:14:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22588.639631 secs 2023-03-09T21:14:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:14:38 smithi181 ceph-mon[94189]: pgmap v12558: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 9 op/s 2023-03-09T21:14:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:14:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22588.639631 secs 2023-03-09T21:14:38.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:14:38 smithi038 ceph-mon[92422]: pgmap v12558: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 9 op/s 2023-03-09T21:14:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23050.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:14:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:14:39 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:14:39.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:14:39 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:14:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:14:39 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:14:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:14:39 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:14:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:14:39 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:14:39.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:14:39 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:14:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:14:40 smithi122 ceph-mon[93869]: pgmap v12559: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-09T21:14:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:14:40 smithi181 ceph-mon[94189]: pgmap v12559: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-09T21:14:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:14:40 smithi038 ceph-mon[92422]: pgmap v12559: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-09T21:14:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:14:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22591.306681 secs 2023-03-09T21:14:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:14:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:14:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:14:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:14:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:14:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22591.306681 secs 2023-03-09T21:14:41.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:14:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:14:41.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:14:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:14:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:14:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22591.306681 secs 2023-03-09T21:14:41.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:14:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:14:41.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:14:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:14:42.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:14:42 smithi038 ceph-mon[92422]: pgmap v12560: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 169 KiB/s wr, 10 op/s 2023-03-09T21:14:42.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:14:42 smithi122 ceph-mon[93869]: pgmap v12560: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 169 KiB/s wr, 10 op/s 2023-03-09T21:14:42.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:14:42 smithi181 ceph-mon[94189]: pgmap v12560: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 169 KiB/s wr, 10 op/s 2023-03-09T21:14:43.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:14:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22593.639766 secs 2023-03-09T21:14:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23055.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:14:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:14:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22593.639766 secs 2023-03-09T21:14:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:14:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22593.639766 secs 2023-03-09T21:14:44.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:14:44 smithi038 ceph-mon[92422]: pgmap v12561: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 169 KiB/s wr, 7 op/s 2023-03-09T21:14:44.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:14:44 smithi122 ceph-mon[93869]: pgmap v12561: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 169 KiB/s wr, 7 op/s 2023-03-09T21:14:44.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:14:44 smithi181 ceph-mon[94189]: pgmap v12561: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 169 KiB/s wr, 7 op/s 2023-03-09T21:14:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:14:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22596.306742 secs 2023-03-09T21:14:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:14:45 smithi122 ceph-mon[93869]: pgmap v12562: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 169 KiB/s wr, 7 op/s 2023-03-09T21:14:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:14:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22596.306742 secs 2023-03-09T21:14:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:14:45 smithi181 ceph-mon[94189]: pgmap v12562: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 169 KiB/s wr, 7 op/s 2023-03-09T21:14:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:14:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22596.306742 secs 2023-03-09T21:14:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:14:45 smithi038 ceph-mon[92422]: pgmap v12562: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 169 KiB/s wr, 7 op/s 2023-03-09T21:14:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:14:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22598.639883 secs 2023-03-09T21:14:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:14:48 smithi122 ceph-mon[93869]: pgmap v12563: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 371 KiB/s wr, 12 op/s 2023-03-09T21:14:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:14:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22598.639883 secs 2023-03-09T21:14:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:14:48 smithi181 ceph-mon[94189]: pgmap v12563: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 371 KiB/s wr, 12 op/s 2023-03-09T21:14:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:14:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22598.639883 secs 2023-03-09T21:14:48.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:14:48 smithi038 ceph-mon[92422]: pgmap v12563: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 371 KiB/s wr, 12 op/s 2023-03-09T21:14:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23060.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:14:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:14:50 smithi122 ceph-mon[93869]: pgmap v12564: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 371 KiB/s wr, 9 op/s 2023-03-09T21:14:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:14:50 smithi181 ceph-mon[94189]: pgmap v12564: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 371 KiB/s wr, 9 op/s 2023-03-09T21:14:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:14:50 smithi038 ceph-mon[92422]: pgmap v12564: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 371 KiB/s wr, 9 op/s 2023-03-09T21:14:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:14:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22601.306815 secs 2023-03-09T21:14:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:14:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:14:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:14:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:14:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:14:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22601.306815 secs 2023-03-09T21:14:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:14:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:14:51.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:14:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:14:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:14:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22601.306815 secs 2023-03-09T21:14:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:14:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:14:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:14:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:14:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:14:52 smithi181 ceph-mon[94189]: pgmap v12565: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 406 KiB/s rd, 391 KiB/s wr, 12 op/s 2023-03-09T21:14:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:14:52 smithi038 ceph-mon[92422]: pgmap v12565: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 406 KiB/s rd, 391 KiB/s wr, 12 op/s 2023-03-09T21:14:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:14:52 smithi122 ceph-mon[93869]: pgmap v12565: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 406 KiB/s rd, 391 KiB/s wr, 12 op/s 2023-03-09T21:14:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:14:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22603.640023 secs 2023-03-09T21:14:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:14:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:14:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:14:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:14:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23065.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:14:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:14:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22603.640023 secs 2023-03-09T21:14:53.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:14:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:14:53.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:14:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:14:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:14:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22603.640023 secs 2023-03-09T21:14:53.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:14:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:14:53.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:14:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:14:54.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:14:54 smithi038 ceph-mon[92422]: pgmap v12566: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 561 KiB/s rd, 536 KiB/s wr, 11 op/s 2023-03-09T21:14:54.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:14:54 smithi122 ceph-mon[93869]: pgmap v12566: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 561 KiB/s rd, 536 KiB/s wr, 11 op/s 2023-03-09T21:14:54.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:14:54 smithi181 ceph-mon[94189]: pgmap v12566: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 561 KiB/s rd, 536 KiB/s wr, 11 op/s 2023-03-09T21:14:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:14:55 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:14:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:14:55 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:14:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:14:55 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:14:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:14:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22606.306877 secs 2023-03-09T21:14:56.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:14:56 smithi122 ceph-mon[93869]: pgmap v12567: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 561 KiB/s rd, 536 KiB/s wr, 11 op/s 2023-03-09T21:14:56.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:14:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:14:56.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:14:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:14:56.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:14:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:14:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:14:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22606.306877 secs 2023-03-09T21:14:56.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:14:56 smithi181 ceph-mon[94189]: pgmap v12567: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 561 KiB/s rd, 536 KiB/s wr, 11 op/s 2023-03-09T21:14:56.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:14:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:14:56.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:14:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:14:56.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:14:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:14:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:14:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22606.306877 secs 2023-03-09T21:14:57.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:14:56 smithi038 ceph-mon[92422]: pgmap v12567: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 561 KiB/s rd, 536 KiB/s wr, 11 op/s 2023-03-09T21:14:57.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:14:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:14:57.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:14:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:14:57.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:14:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:14:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23070.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:14:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:14:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22608.640165 secs 2023-03-09T21:14:58.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:14:58 smithi122 ceph-mon[93869]: pgmap v12568: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 554 KiB/s wr, 15 op/s 2023-03-09T21:14:58.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:14:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22608.640165 secs 2023-03-09T21:14:58.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:14:58 smithi181 ceph-mon[94189]: pgmap v12568: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 554 KiB/s wr, 15 op/s 2023-03-09T21:14:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:14:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22608.640165 secs 2023-03-09T21:14:59.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:14:58 smithi038 ceph-mon[92422]: pgmap v12568: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 554 KiB/s wr, 15 op/s 2023-03-09T21:14:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:14:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:14:59.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:14:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:14:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:14:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:14:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:14:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:15:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:14:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:15:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:14:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:15:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:00 smithi122 ceph-mon[93869]: pgmap v12569: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 637 KiB/s rd, 569 KiB/s wr, 11 op/s 2023-03-09T21:15:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:00 smithi181 ceph-mon[94189]: pgmap v12569: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 637 KiB/s rd, 569 KiB/s wr, 11 op/s 2023-03-09T21:15:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:00 smithi038 ceph-mon[92422]: pgmap v12569: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 637 KiB/s rd, 569 KiB/s wr, 11 op/s 2023-03-09T21:15:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22611.306958 secs 2023-03-09T21:15:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:15:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:15:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22611.306958 secs 2023-03-09T21:15:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:15:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:15:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22611.306958 secs 2023-03-09T21:15:02.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:15:02.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:15:02.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:02 smithi122 ceph-mon[93869]: pgmap v12570: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 643 KiB/s rd, 584 KiB/s wr, 13 op/s 2023-03-09T21:15:02.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:02 smithi181 ceph-mon[94189]: pgmap v12570: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 643 KiB/s rd, 584 KiB/s wr, 13 op/s 2023-03-09T21:15:02.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:02 smithi038 ceph-mon[92422]: pgmap v12570: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 643 KiB/s rd, 584 KiB/s wr, 13 op/s 2023-03-09T21:15:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23075.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:15:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22613.640291 secs 2023-03-09T21:15:03.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:03 smithi122 ceph-mon[93869]: pgmap v12571: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 630 KiB/s rd, 587 KiB/s wr, 24 op/s 2023-03-09T21:15:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22613.640291 secs 2023-03-09T21:15:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:03 smithi181 ceph-mon[94189]: pgmap v12571: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 630 KiB/s rd, 587 KiB/s wr, 24 op/s 2023-03-09T21:15:04.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22613.640291 secs 2023-03-09T21:15:04.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:03 smithi038 ceph-mon[92422]: pgmap v12571: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 630 KiB/s rd, 587 KiB/s wr, 24 op/s 2023-03-09T21:15:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22616.307020 secs 2023-03-09T21:15:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:05 smithi122 ceph-mon[93869]: pgmap v12572: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 273 KiB/s wr, 21 op/s 2023-03-09T21:15:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22616.307020 secs 2023-03-09T21:15:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:05 smithi181 ceph-mon[94189]: pgmap v12572: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 273 KiB/s wr, 21 op/s 2023-03-09T21:15:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22616.307020 secs 2023-03-09T21:15:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:05 smithi038 ceph-mon[92422]: pgmap v12572: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 273 KiB/s wr, 21 op/s 2023-03-09T21:15:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22618.640409 secs 2023-03-09T21:15:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:08 smithi122 ceph-mon[93869]: pgmap v12573: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 296 KiB/s wr, 26 op/s 2023-03-09T21:15:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22618.640409 secs 2023-03-09T21:15:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:08 smithi181 ceph-mon[94189]: pgmap v12573: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 296 KiB/s wr, 26 op/s 2023-03-09T21:15:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22618.640409 secs 2023-03-09T21:15:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:08 smithi038 ceph-mon[92422]: pgmap v12573: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 296 KiB/s wr, 26 op/s 2023-03-09T21:15:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23080.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:15:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:10 smithi038 ceph-mon[92422]: pgmap v12574: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 364 KiB/s wr, 23 op/s 2023-03-09T21:15:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:10 smithi038 ceph-mon[92422]: osdmap e999: 12 total, 12 up, 12 in 2023-03-09T21:15:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:10 smithi122 ceph-mon[93869]: pgmap v12574: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 364 KiB/s wr, 23 op/s 2023-03-09T21:15:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:10 smithi122 ceph-mon[93869]: osdmap e999: 12 total, 12 up, 12 in 2023-03-09T21:15:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:10 smithi181 ceph-mon[94189]: pgmap v12574: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 364 KiB/s wr, 23 op/s 2023-03-09T21:15:10.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:10 smithi181 ceph-mon[94189]: osdmap e999: 12 total, 12 up, 12 in 2023-03-09T21:15:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22621.307082 secs 2023-03-09T21:15:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22621.307082 secs 2023-03-09T21:15:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22621.307082 secs 2023-03-09T21:15:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:12 smithi038 ceph-mon[92422]: pgmap v12576: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 158 KiB/s wr, 25 op/s 2023-03-09T21:15:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:12 smithi122 ceph-mon[93869]: pgmap v12576: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 158 KiB/s wr, 25 op/s 2023-03-09T21:15:12.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:12 smithi181 ceph-mon[94189]: pgmap v12576: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 158 KiB/s wr, 25 op/s 2023-03-09T21:15:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22623.640521 secs 2023-03-09T21:15:13.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23085.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:15:13.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22623.640521 secs 2023-03-09T21:15:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22623.640521 secs 2023-03-09T21:15:14.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:14 smithi122 ceph-mon[93869]: pgmap v12577: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 130 KiB/s wr, 10 op/s 2023-03-09T21:15:14.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:14 smithi181 ceph-mon[94189]: pgmap v12577: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 130 KiB/s wr, 10 op/s 2023-03-09T21:15:14.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:14 smithi038 ceph-mon[92422]: pgmap v12577: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 130 KiB/s wr, 10 op/s 2023-03-09T21:15:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22626.307144 secs 2023-03-09T21:15:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:15 smithi122 ceph-mon[93869]: pgmap v12578: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 130 KiB/s wr, 10 op/s 2023-03-09T21:15:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22626.307144 secs 2023-03-09T21:15:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:15 smithi181 ceph-mon[94189]: pgmap v12578: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 130 KiB/s wr, 10 op/s 2023-03-09T21:15:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22626.307144 secs 2023-03-09T21:15:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:15 smithi038 ceph-mon[92422]: pgmap v12578: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 130 KiB/s wr, 10 op/s 2023-03-09T21:15:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22628.640614 secs 2023-03-09T21:15:18.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:18 smithi038 ceph-mon[92422]: pgmap v12579: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 103 KiB/s wr, 7 op/s 2023-03-09T21:15:18.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23090.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:15:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22628.640614 secs 2023-03-09T21:15:18.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:18 smithi122 ceph-mon[93869]: pgmap v12579: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 103 KiB/s wr, 7 op/s 2023-03-09T21:15:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22628.640614 secs 2023-03-09T21:15:18.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:18 smithi181 ceph-mon[94189]: pgmap v12579: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 103 KiB/s wr, 7 op/s 2023-03-09T21:15:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:20 smithi038 ceph-mon[92422]: pgmap v12580: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 73 KiB/s wr, 7 op/s 2023-03-09T21:15:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:20 smithi038 ceph-mon[92422]: osdmap e1000: 12 total, 12 up, 12 in 2023-03-09T21:15:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:20 smithi122 ceph-mon[93869]: pgmap v12580: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 73 KiB/s wr, 7 op/s 2023-03-09T21:15:20.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:20 smithi122 ceph-mon[93869]: osdmap e1000: 12 total, 12 up, 12 in 2023-03-09T21:15:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:20 smithi181 ceph-mon[94189]: pgmap v12580: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 73 KiB/s wr, 7 op/s 2023-03-09T21:15:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:20 smithi181 ceph-mon[94189]: osdmap e1000: 12 total, 12 up, 12 in 2023-03-09T21:15:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22631.307207 secs 2023-03-09T21:15:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22631.307207 secs 2023-03-09T21:15:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22631.307207 secs 2023-03-09T21:15:22.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:22 smithi038 ceph-mon[92422]: pgmap v12582: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 73 KiB/s wr, 7 op/s 2023-03-09T21:15:22.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:22 smithi122 ceph-mon[93869]: pgmap v12582: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 73 KiB/s wr, 7 op/s 2023-03-09T21:15:22.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:22 smithi181 ceph-mon[94189]: pgmap v12582: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 73 KiB/s wr, 7 op/s 2023-03-09T21:15:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23095.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:15:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22633.640741 secs 2023-03-09T21:15:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22633.640741 secs 2023-03-09T21:15:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22633.640741 secs 2023-03-09T21:15:24.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:24 smithi122 ceph-mon[93869]: pgmap v12583: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 73 KiB/s wr, 7 op/s 2023-03-09T21:15:24.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:24 smithi181 ceph-mon[94189]: pgmap v12583: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 73 KiB/s wr, 7 op/s 2023-03-09T21:15:24.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:24 smithi038 ceph-mon[92422]: pgmap v12583: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 73 KiB/s wr, 7 op/s 2023-03-09T21:15:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22636.307267 secs 2023-03-09T21:15:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:25 smithi122 ceph-mon[93869]: pgmap v12584: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 73 KiB/s wr, 7 op/s 2023-03-09T21:15:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22636.307267 secs 2023-03-09T21:15:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:25 smithi181 ceph-mon[94189]: pgmap v12584: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 73 KiB/s wr, 7 op/s 2023-03-09T21:15:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22636.307267 secs 2023-03-09T21:15:26.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:25 smithi038 ceph-mon[92422]: pgmap v12584: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 73 KiB/s wr, 7 op/s 2023-03-09T21:15:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22638.640851 secs 2023-03-09T21:15:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:28 smithi038 ceph-mon[92422]: pgmap v12585: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 73 KiB/s wr, 7 op/s 2023-03-09T21:15:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23100.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:15:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22638.640851 secs 2023-03-09T21:15:28.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:28 smithi122 ceph-mon[93869]: pgmap v12585: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 73 KiB/s wr, 7 op/s 2023-03-09T21:15:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22638.640851 secs 2023-03-09T21:15:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:28 smithi181 ceph-mon[94189]: pgmap v12585: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 73 KiB/s wr, 7 op/s 2023-03-09T21:15:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:30 smithi038 ceph-mon[92422]: pgmap v12586: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T21:15:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:30 smithi122 ceph-mon[93869]: pgmap v12586: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T21:15:30.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:30 smithi181 ceph-mon[94189]: pgmap v12586: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T21:15:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22641.307330 secs 2023-03-09T21:15:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22641.307330 secs 2023-03-09T21:15:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22641.307330 secs 2023-03-09T21:15:32.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:32 smithi122 ceph-mon[93869]: pgmap v12587: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 88 B/s wr, 8 op/s 2023-03-09T21:15:32.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:32 smithi181 ceph-mon[94189]: pgmap v12587: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 88 B/s wr, 8 op/s 2023-03-09T21:15:32.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:32 smithi038 ceph-mon[92422]: pgmap v12587: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 88 B/s wr, 8 op/s 2023-03-09T21:15:33.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23105.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:15:33.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22643.640971 secs 2023-03-09T21:15:33.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:33 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:15:33.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:33 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:15:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22643.640971 secs 2023-03-09T21:15:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:33 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:15:33.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:33 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:15:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22643.640971 secs 2023-03-09T21:15:34.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:33 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:15:34.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:33 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:15:34.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:34 smithi122 ceph-mon[93869]: pgmap v12588: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 341 B/s wr, 6 op/s 2023-03-09T21:15:34.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:34 smithi181 ceph-mon[94189]: pgmap v12588: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 341 B/s wr, 6 op/s 2023-03-09T21:15:34.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:34 smithi038 ceph-mon[92422]: pgmap v12588: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 341 B/s wr, 6 op/s 2023-03-09T21:15:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:35 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:15:35.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:35 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:15:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:35 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:15:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:35 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:15:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:35 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:15:36.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:35 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:15:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22646.307389 secs 2023-03-09T21:15:36.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:36 smithi122 ceph-mon[93869]: pgmap v12589: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 74 KiB/s wr, 7 op/s 2023-03-09T21:15:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22646.307389 secs 2023-03-09T21:15:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:36 smithi181 ceph-mon[94189]: pgmap v12589: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 74 KiB/s wr, 7 op/s 2023-03-09T21:15:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22646.307389 secs 2023-03-09T21:15:37.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:36 smithi038 ceph-mon[92422]: pgmap v12589: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 74 KiB/s wr, 7 op/s 2023-03-09T21:15:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23110.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:15:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:38 smithi122 ceph-mon[93869]: pgmap v12590: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 74 KiB/s wr, 9 op/s 2023-03-09T21:15:38.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:38 smithi181 ceph-mon[94189]: pgmap v12590: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 74 KiB/s wr, 9 op/s 2023-03-09T21:15:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:38 smithi038 ceph-mon[92422]: pgmap v12590: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 74 KiB/s wr, 9 op/s 2023-03-09T21:15:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:40 smithi122 ceph-mon[93869]: pgmap v12591: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 201 KiB/s wr, 7 op/s 2023-03-09T21:15:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:40 smithi181 ceph-mon[94189]: pgmap v12591: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 201 KiB/s wr, 7 op/s 2023-03-09T21:15:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:40 smithi038 ceph-mon[92422]: pgmap v12591: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 201 KiB/s wr, 7 op/s 2023-03-09T21:15:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22651.307474 secs 2023-03-09T21:15:41.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:41 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:15:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22651.307474 secs 2023-03-09T21:15:41.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:41 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:15:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22651.307474 secs 2023-03-09T21:15:42.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:41 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:15:42.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:42 smithi122 ceph-mon[93869]: pgmap v12592: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 218 KiB/s wr, 9 op/s 2023-03-09T21:15:42.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:42 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:15:42.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:42 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:15:42.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:42 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:15:42.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:42 smithi181 ceph-mon[94189]: pgmap v12592: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 218 KiB/s wr, 9 op/s 2023-03-09T21:15:42.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:42 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:15:42.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:42 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:15:42.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:42 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:15:42.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:42 smithi038 ceph-mon[92422]: pgmap v12592: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 218 KiB/s wr, 9 op/s 2023-03-09T21:15:43.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:42 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:15:43.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:42 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:15:43.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:42 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:15:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23115.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:15:43.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:43 smithi122 ceph-mon[93869]: pgmap v12593: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 218 KiB/s wr, 7 op/s 2023-03-09T21:15:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:43 smithi181 ceph-mon[94189]: pgmap v12593: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 218 KiB/s wr, 7 op/s 2023-03-09T21:15:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:43 smithi038 ceph-mon[92422]: pgmap v12593: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 218 KiB/s wr, 7 op/s 2023-03-09T21:15:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22656.307545 secs 2023-03-09T21:15:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:45 smithi122 ceph-mon[93869]: pgmap v12594: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 243 KiB/s wr, 7 op/s 2023-03-09T21:15:46.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22656.307545 secs 2023-03-09T21:15:46.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:45 smithi181 ceph-mon[94189]: pgmap v12594: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 243 KiB/s wr, 7 op/s 2023-03-09T21:15:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22656.307545 secs 2023-03-09T21:15:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:45 smithi038 ceph-mon[92422]: pgmap v12594: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 243 KiB/s wr, 7 op/s 2023-03-09T21:15:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:48 smithi038 ceph-mon[92422]: pgmap v12595: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 277 KiB/s wr, 10 op/s 2023-03-09T21:15:48.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23120.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:15:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:48 smithi122 ceph-mon[93869]: pgmap v12595: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 277 KiB/s wr, 10 op/s 2023-03-09T21:15:48.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:48 smithi181 ceph-mon[94189]: pgmap v12595: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 277 KiB/s wr, 10 op/s 2023-03-09T21:15:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:50 smithi038 ceph-mon[92422]: pgmap v12596: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 386 KiB/s rd, 371 KiB/s wr, 9 op/s 2023-03-09T21:15:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:50 smithi122 ceph-mon[93869]: pgmap v12596: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 386 KiB/s rd, 371 KiB/s wr, 9 op/s 2023-03-09T21:15:50.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:50 smithi181 ceph-mon[94189]: pgmap v12596: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 386 KiB/s rd, 371 KiB/s wr, 9 op/s 2023-03-09T21:15:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22661.307603 secs 2023-03-09T21:15:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:15:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:15:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22661.307603 secs 2023-03-09T21:15:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:15:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:15:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22661.307603 secs 2023-03-09T21:15:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:15:51.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:15:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:52 smithi122 ceph-mon[93869]: pgmap v12597: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 583 KiB/s rd, 569 KiB/s wr, 13 op/s 2023-03-09T21:15:52.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:52 smithi181 ceph-mon[94189]: pgmap v12597: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 583 KiB/s rd, 569 KiB/s wr, 13 op/s 2023-03-09T21:15:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:52 smithi038 ceph-mon[92422]: pgmap v12597: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 583 KiB/s rd, 569 KiB/s wr, 13 op/s 2023-03-09T21:15:53.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23125.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:15:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:15:53.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:15:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:15:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:15:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:15:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:15:54.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:54 smithi122 ceph-mon[93869]: pgmap v12598: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 551 KiB/s wr, 11 op/s 2023-03-09T21:15:54.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:54 smithi181 ceph-mon[94189]: pgmap v12598: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 551 KiB/s wr, 11 op/s 2023-03-09T21:15:54.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:54 smithi038 ceph-mon[92422]: pgmap v12598: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 551 KiB/s wr, 11 op/s 2023-03-09T21:15:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:15:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:15:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:15:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:15:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:15:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:15:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22666.307664 secs 2023-03-09T21:15:56.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:56 smithi122 ceph-mon[93869]: pgmap v12599: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 551 KiB/s wr, 11 op/s 2023-03-09T21:15:56.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:15:56.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:15:56.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:15:56.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:15:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22666.307664 secs 2023-03-09T21:15:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:56 smithi181 ceph-mon[94189]: pgmap v12599: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 551 KiB/s wr, 11 op/s 2023-03-09T21:15:56.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:15:56.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:15:56.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:15:56.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:15:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22666.307664 secs 2023-03-09T21:15:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:56 smithi038 ceph-mon[92422]: pgmap v12599: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 551 KiB/s wr, 11 op/s 2023-03-09T21:15:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:15:57.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:15:57.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:15:57.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:15:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:15:57.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:15:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:15:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:15:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:15:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:15:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23130.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:15:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22668.641567 secs 2023-03-09T21:15:58.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:58 smithi122 ceph-mon[93869]: pgmap v12600: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 704 KiB/s rd, 691 KiB/s wr, 15 op/s 2023-03-09T21:15:58.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22668.641567 secs 2023-03-09T21:15:58.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:58 smithi181 ceph-mon[94189]: pgmap v12600: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 704 KiB/s rd, 691 KiB/s wr, 15 op/s 2023-03-09T21:15:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22668.641567 secs 2023-03-09T21:15:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:58 smithi038 ceph-mon[92422]: pgmap v12600: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 704 KiB/s rd, 691 KiB/s wr, 15 op/s 2023-03-09T21:15:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:15:59 smithi122 ceph-mon[93869]: pgmap v12601: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 594 KiB/s rd, 583 KiB/s wr, 12 op/s 2023-03-09T21:15:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:15:59 smithi181 ceph-mon[94189]: pgmap v12601: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 594 KiB/s rd, 583 KiB/s wr, 12 op/s 2023-03-09T21:15:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:15:59 smithi038 ceph-mon[92422]: pgmap v12601: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 594 KiB/s rd, 583 KiB/s wr, 12 op/s 2023-03-09T21:16:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22671.307722 secs 2023-03-09T21:16:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22671.307722 secs 2023-03-09T21:16:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22671.307722 secs 2023-03-09T21:16:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:01 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:16:02.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:16:02.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:16:02.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:01 smithi122 ceph-mon[93869]: pgmap v12602: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 646 KiB/s wr, 17 op/s 2023-03-09T21:16:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:01 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:16:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:16:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:16:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:01 smithi181 ceph-mon[94189]: pgmap v12602: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 646 KiB/s wr, 17 op/s 2023-03-09T21:16:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:01 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:16:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:16:02.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:16:02.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:01 smithi038 ceph-mon[92422]: pgmap v12602: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 646 KiB/s wr, 17 op/s 2023-03-09T21:16:03.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22673.641687 secs 2023-03-09T21:16:03.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22673.641687 secs 2023-03-09T21:16:03.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22673.641687 secs 2023-03-09T21:16:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23135.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:16:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:03 smithi122 ceph-mon[93869]: pgmap v12603: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 344 KiB/s wr, 25 op/s 2023-03-09T21:16:04.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:03 smithi181 ceph-mon[94189]: pgmap v12603: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 344 KiB/s wr, 25 op/s 2023-03-09T21:16:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:03 smithi038 ceph-mon[92422]: pgmap v12603: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 344 KiB/s wr, 25 op/s 2023-03-09T21:16:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22676.307796 secs 2023-03-09T21:16:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:05 smithi122 ceph-mon[93869]: pgmap v12604: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 343 KiB/s wr, 24 op/s 2023-03-09T21:16:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22676.307796 secs 2023-03-09T21:16:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:05 smithi181 ceph-mon[94189]: pgmap v12604: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 343 KiB/s wr, 24 op/s 2023-03-09T21:16:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22676.307796 secs 2023-03-09T21:16:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:05 smithi038 ceph-mon[92422]: pgmap v12604: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 343 KiB/s wr, 24 op/s 2023-03-09T21:16:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22678.641813 secs 2023-03-09T21:16:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:08 smithi038 ceph-mon[92422]: pgmap v12605: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 337 KiB/s rd, 370 KiB/s wr, 27 op/s 2023-03-09T21:16:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23140.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:16:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22678.641813 secs 2023-03-09T21:16:08.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:08 smithi122 ceph-mon[93869]: pgmap v12605: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 337 KiB/s rd, 370 KiB/s wr, 27 op/s 2023-03-09T21:16:08.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22678.641813 secs 2023-03-09T21:16:08.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:08 smithi181 ceph-mon[94189]: pgmap v12605: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 337 KiB/s rd, 370 KiB/s wr, 27 op/s 2023-03-09T21:16:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:10 smithi122 ceph-mon[93869]: pgmap v12606: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 287 KiB/s wr, 23 op/s 2023-03-09T21:16:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:10 smithi122 ceph-mon[93869]: osdmap e1001: 12 total, 12 up, 12 in 2023-03-09T21:16:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:10 smithi181 ceph-mon[94189]: pgmap v12606: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 287 KiB/s wr, 23 op/s 2023-03-09T21:16:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:10 smithi181 ceph-mon[94189]: osdmap e1001: 12 total, 12 up, 12 in 2023-03-09T21:16:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:10 smithi038 ceph-mon[92422]: pgmap v12606: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 287 KiB/s wr, 23 op/s 2023-03-09T21:16:11.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:10 smithi038 ceph-mon[92422]: osdmap e1001: 12 total, 12 up, 12 in 2023-03-09T21:16:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22681.307858 secs 2023-03-09T21:16:11.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:11 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:16:11.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:11 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:16:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22681.307858 secs 2023-03-09T21:16:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:11 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:16:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:11 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:16:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22681.307858 secs 2023-03-09T21:16:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:11 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:16:12.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:11 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:16:12.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:12 smithi122 ceph-mon[93869]: pgmap v12608: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 156 KiB/s wr, 22 op/s 2023-03-09T21:16:12.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:12 smithi181 ceph-mon[94189]: pgmap v12608: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 156 KiB/s wr, 22 op/s 2023-03-09T21:16:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:12 smithi038 ceph-mon[92422]: pgmap v12608: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 156 KiB/s wr, 22 op/s 2023-03-09T21:16:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23145.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:16:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22683.641936 secs 2023-03-09T21:16:13.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:16:13.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:16:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22683.641936 secs 2023-03-09T21:16:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:16:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:16:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22683.641936 secs 2023-03-09T21:16:14.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:16:14.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:16:14.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:14 smithi122 ceph-mon[93869]: pgmap v12609: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 130 KiB/s wr, 8 op/s 2023-03-09T21:16:14.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:14 smithi181 ceph-mon[94189]: pgmap v12609: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 130 KiB/s wr, 8 op/s 2023-03-09T21:16:14.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:14 smithi038 ceph-mon[92422]: pgmap v12609: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 130 KiB/s wr, 8 op/s 2023-03-09T21:16:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22686.307929 secs 2023-03-09T21:16:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:15 smithi122 ceph-mon[93869]: pgmap v12610: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 130 KiB/s wr, 8 op/s 2023-03-09T21:16:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22686.307929 secs 2023-03-09T21:16:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:15 smithi181 ceph-mon[94189]: pgmap v12610: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 130 KiB/s wr, 8 op/s 2023-03-09T21:16:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22686.307929 secs 2023-03-09T21:16:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:15 smithi038 ceph-mon[92422]: pgmap v12610: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 130 KiB/s wr, 8 op/s 2023-03-09T21:16:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23150.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:16:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22688.642054 secs 2023-03-09T21:16:18.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:18 smithi122 ceph-mon[93869]: pgmap v12611: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 98 KiB/s wr, 7 op/s 2023-03-09T21:16:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22688.642054 secs 2023-03-09T21:16:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:18 smithi181 ceph-mon[94189]: pgmap v12611: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 98 KiB/s wr, 7 op/s 2023-03-09T21:16:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22688.642054 secs 2023-03-09T21:16:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:18 smithi038 ceph-mon[92422]: pgmap v12611: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 98 KiB/s wr, 7 op/s 2023-03-09T21:16:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:20 smithi122 ceph-mon[93869]: pgmap v12612: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 59 KiB/s wr, 7 op/s 2023-03-09T21:16:20.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:20 smithi122 ceph-mon[93869]: osdmap e1002: 12 total, 12 up, 12 in 2023-03-09T21:16:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:20 smithi181 ceph-mon[94189]: pgmap v12612: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 59 KiB/s wr, 7 op/s 2023-03-09T21:16:20.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:20 smithi181 ceph-mon[94189]: osdmap e1002: 12 total, 12 up, 12 in 2023-03-09T21:16:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:20 smithi038 ceph-mon[92422]: pgmap v12612: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 59 KiB/s wr, 7 op/s 2023-03-09T21:16:21.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:20 smithi038 ceph-mon[92422]: osdmap e1002: 12 total, 12 up, 12 in 2023-03-09T21:16:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22691.307984 secs 2023-03-09T21:16:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22691.307984 secs 2023-03-09T21:16:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22691.307984 secs 2023-03-09T21:16:22.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:22 smithi122 ceph-mon[93869]: pgmap v12614: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 59 KiB/s wr, 7 op/s 2023-03-09T21:16:22.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:22 smithi181 ceph-mon[94189]: pgmap v12614: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 59 KiB/s wr, 7 op/s 2023-03-09T21:16:22.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:22 smithi038 ceph-mon[92422]: pgmap v12614: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 59 KiB/s wr, 7 op/s 2023-03-09T21:16:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23155.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:16:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22693.642168 secs 2023-03-09T21:16:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22693.642168 secs 2023-03-09T21:16:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22693.642168 secs 2023-03-09T21:16:24.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:24 smithi122 ceph-mon[93869]: pgmap v12615: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 59 KiB/s wr, 7 op/s 2023-03-09T21:16:24.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:24 smithi181 ceph-mon[94189]: pgmap v12615: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 59 KiB/s wr, 7 op/s 2023-03-09T21:16:24.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:24 smithi038 ceph-mon[92422]: pgmap v12615: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 59 KiB/s wr, 7 op/s 2023-03-09T21:16:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22696.308046 secs 2023-03-09T21:16:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:25 smithi122 ceph-mon[93869]: pgmap v12616: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 59 KiB/s wr, 7 op/s 2023-03-09T21:16:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22696.308046 secs 2023-03-09T21:16:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:25 smithi181 ceph-mon[94189]: pgmap v12616: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 59 KiB/s wr, 7 op/s 2023-03-09T21:16:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22696.308046 secs 2023-03-09T21:16:26.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:25 smithi038 ceph-mon[92422]: pgmap v12616: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 59 KiB/s wr, 7 op/s 2023-03-09T21:16:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23160.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:16:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22698.642284 secs 2023-03-09T21:16:28.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:28 smithi122 ceph-mon[93869]: pgmap v12617: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 59 KiB/s wr, 7 op/s 2023-03-09T21:16:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22698.642284 secs 2023-03-09T21:16:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:28 smithi181 ceph-mon[94189]: pgmap v12617: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 59 KiB/s wr, 7 op/s 2023-03-09T21:16:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22698.642284 secs 2023-03-09T21:16:29.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:28 smithi038 ceph-mon[92422]: pgmap v12617: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 59 KiB/s wr, 7 op/s 2023-03-09T21:16:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:30 smithi122 ceph-mon[93869]: pgmap v12618: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T21:16:30.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:30 smithi181 ceph-mon[94189]: pgmap v12618: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T21:16:30.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:30 smithi038 ceph-mon[92422]: pgmap v12618: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T21:16:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22701.308103 secs 2023-03-09T21:16:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22701.308103 secs 2023-03-09T21:16:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22701.308103 secs 2023-03-09T21:16:32.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:32 smithi122 ceph-mon[93869]: pgmap v12619: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 0 B/s wr, 9 op/s 2023-03-09T21:16:32.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:32 smithi181 ceph-mon[94189]: pgmap v12619: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 0 B/s wr, 9 op/s 2023-03-09T21:16:32.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:32 smithi038 ceph-mon[92422]: pgmap v12619: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 0 B/s wr, 9 op/s 2023-03-09T21:16:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23165.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:16:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22703.642404 secs 2023-03-09T21:16:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22703.642404 secs 2023-03-09T21:16:34.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22703.642404 secs 2023-03-09T21:16:34.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:34 smithi122 ceph-mon[93869]: pgmap v12620: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-09T21:16:34.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:34 smithi181 ceph-mon[94189]: pgmap v12620: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-09T21:16:34.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:34 smithi038 ceph-mon[92422]: pgmap v12620: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-09T21:16:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22706.308164 secs 2023-03-09T21:16:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:35 smithi122 ceph-mon[93869]: pgmap v12621: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-09T21:16:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22706.308164 secs 2023-03-09T21:16:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:35 smithi181 ceph-mon[94189]: pgmap v12621: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-09T21:16:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22706.308164 secs 2023-03-09T21:16:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:35 smithi038 ceph-mon[92422]: pgmap v12621: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-09T21:16:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23170.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:16:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:38 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 22708.642501 secs 2023-03-09T21:16:38.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:38 smithi122 ceph-mon[93869]: pgmap v12622: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 9 op/s 2023-03-09T21:16:38.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:38 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 22708.642501 secs 2023-03-09T21:16:38.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:38 smithi181 ceph-mon[94189]: pgmap v12622: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 9 op/s 2023-03-09T21:16:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:38 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 22708.642501 secs 2023-03-09T21:16:39.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:38 smithi038 ceph-mon[92422]: pgmap v12622: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 9 op/s 2023-03-09T21:16:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:40 smithi122 ceph-mon[93869]: pgmap v12623: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 140 KiB/s wr, 7 op/s 2023-03-09T21:16:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:40 smithi181 ceph-mon[94189]: pgmap v12623: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 140 KiB/s wr, 7 op/s 2023-03-09T21:16:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:40 smithi038 ceph-mon[92422]: pgmap v12623: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 140 KiB/s wr, 7 op/s 2023-03-09T21:16:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22711.308224 secs 2023-03-09T21:16:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22711.308224 secs 2023-03-09T21:16:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22711.308224 secs 2023-03-09T21:16:42.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:42 smithi122 ceph-mon[93869]: pgmap v12624: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 157 KiB/s wr, 9 op/s 2023-03-09T21:16:42.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:42 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:16:42.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:42 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:16:42.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:42 smithi181 ceph-mon[94189]: pgmap v12624: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 157 KiB/s wr, 9 op/s 2023-03-09T21:16:42.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:42 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:16:42.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:42 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:16:42.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:42 smithi038 ceph-mon[92422]: pgmap v12624: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 157 KiB/s wr, 9 op/s 2023-03-09T21:16:42.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:42 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:16:43.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:42 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:16:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23175.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:16:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:43 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 22713.642646 secs 2023-03-09T21:16:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:43 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 22713.642646 secs 2023-03-09T21:16:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:43 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 22713.642646 secs 2023-03-09T21:16:44.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:44 smithi122 ceph-mon[93869]: pgmap v12625: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 230 KiB/s wr, 8 op/s 2023-03-09T21:16:44.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:44 smithi181 ceph-mon[94189]: pgmap v12625: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 230 KiB/s wr, 8 op/s 2023-03-09T21:16:44.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:44 smithi038 ceph-mon[92422]: pgmap v12625: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 230 KiB/s wr, 8 op/s 2023-03-09T21:16:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22716.308311 secs 2023-03-09T21:16:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:45 smithi122 ceph-mon[93869]: pgmap v12626: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 230 KiB/s wr, 7 op/s 2023-03-09T21:16:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22716.308311 secs 2023-03-09T21:16:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:45 smithi181 ceph-mon[94189]: pgmap v12626: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 230 KiB/s wr, 7 op/s 2023-03-09T21:16:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22716.308311 secs 2023-03-09T21:16:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:45 smithi038 ceph-mon[92422]: pgmap v12626: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 230 KiB/s wr, 7 op/s 2023-03-09T21:16:48.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23180.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:16:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22718.642740 secs 2023-03-09T21:16:48.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:48 smithi122 ceph-mon[93869]: pgmap v12627: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 741 KiB/s rd, 590 KiB/s wr, 12 op/s 2023-03-09T21:16:48.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22718.642740 secs 2023-03-09T21:16:48.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:48 smithi181 ceph-mon[94189]: pgmap v12627: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 741 KiB/s rd, 590 KiB/s wr, 12 op/s 2023-03-09T21:16:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22718.642740 secs 2023-03-09T21:16:49.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:48 smithi038 ceph-mon[92422]: pgmap v12627: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 741 KiB/s rd, 590 KiB/s wr, 12 op/s 2023-03-09T21:16:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:16:49.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:16:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:16:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:16:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:16:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:16:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:50 smithi122 ceph-mon[93869]: pgmap v12628: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 846 KiB/s rd, 752 KiB/s wr, 10 op/s 2023-03-09T21:16:50.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:50 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:16:50.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:50 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:16:50.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:50 smithi181 ceph-mon[94189]: pgmap v12628: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 846 KiB/s rd, 752 KiB/s wr, 10 op/s 2023-03-09T21:16:50.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:50 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:16:50.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:50 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:16:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:50 smithi038 ceph-mon[92422]: pgmap v12628: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 846 KiB/s rd, 752 KiB/s wr, 10 op/s 2023-03-09T21:16:51.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:50 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:16:51.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:50 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:16:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22721.308374 secs 2023-03-09T21:16:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:16:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:16:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22721.308374 secs 2023-03-09T21:16:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:16:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:16:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22721.308374 secs 2023-03-09T21:16:52.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:16:52.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:16:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:52 smithi122 ceph-mon[93869]: pgmap v12629: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 862 KiB/s rd, 614 KiB/s wr, 13 op/s 2023-03-09T21:16:52.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:52 smithi181 ceph-mon[94189]: pgmap v12629: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 862 KiB/s rd, 614 KiB/s wr, 13 op/s 2023-03-09T21:16:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:52 smithi038 ceph-mon[92422]: pgmap v12629: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 862 KiB/s rd, 614 KiB/s wr, 13 op/s 2023-03-09T21:16:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23185.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:16:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22723.642853 secs 2023-03-09T21:16:53.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:16:53.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:16:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22723.642853 secs 2023-03-09T21:16:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:16:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:16:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22723.642853 secs 2023-03-09T21:16:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:16:54.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:16:54.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:54 smithi122 ceph-mon[93869]: pgmap v12630: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 753 KiB/s wr, 13 op/s 2023-03-09T21:16:54.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:54 smithi181 ceph-mon[94189]: pgmap v12630: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 753 KiB/s wr, 13 op/s 2023-03-09T21:16:54.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:54 smithi038 ceph-mon[92422]: pgmap v12630: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 753 KiB/s wr, 13 op/s 2023-03-09T21:16:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22726.308451 secs 2023-03-09T21:16:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:55 smithi122 ceph-mon[93869]: pgmap v12631: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 808 KiB/s rd, 680 KiB/s wr, 12 op/s 2023-03-09T21:16:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22726.308451 secs 2023-03-09T21:16:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:55 smithi181 ceph-mon[94189]: pgmap v12631: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 808 KiB/s rd, 680 KiB/s wr, 12 op/s 2023-03-09T21:16:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22726.308451 secs 2023-03-09T21:16:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:55 smithi038 ceph-mon[92422]: pgmap v12631: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 808 KiB/s rd, 680 KiB/s wr, 12 op/s 2023-03-09T21:16:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:16:57.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:16:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:16:58.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22728.642988 secs 2023-03-09T21:16:58.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:57 smithi122 ceph-mon[93869]: pgmap v12632: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 827 KiB/s rd, 699 KiB/s wr, 15 op/s 2023-03-09T21:16:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22728.642988 secs 2023-03-09T21:16:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:57 smithi181 ceph-mon[94189]: pgmap v12632: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 827 KiB/s rd, 699 KiB/s wr, 15 op/s 2023-03-09T21:16:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22728.642988 secs 2023-03-09T21:16:58.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:57 smithi038 ceph-mon[92422]: pgmap v12632: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 827 KiB/s rd, 699 KiB/s wr, 15 op/s 2023-03-09T21:16:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23190.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:17:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:17:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:17:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:17:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:17:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:17:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:17:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:59 smithi122 ceph-mon[93869]: pgmap v12633: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 571 KiB/s rd, 568 KiB/s wr, 13 op/s 2023-03-09T21:17:00.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:17:00.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:17:00.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:17:00.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:16:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:17:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:17:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:17:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:17:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:17:00.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:17:00.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:17:00.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:59 smithi181 ceph-mon[94189]: pgmap v12633: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 571 KiB/s rd, 568 KiB/s wr, 13 op/s 2023-03-09T21:17:00.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:17:00.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:17:00.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:17:00.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:16:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:17:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:17:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:17:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:17:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:17:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:17:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:17:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:59 smithi038 ceph-mon[92422]: pgmap v12633: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 571 KiB/s rd, 568 KiB/s wr, 13 op/s 2023-03-09T21:17:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:17:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:17:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:17:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:16:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:17:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22731.308511 secs 2023-03-09T21:17:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:17:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:17:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:00 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T21:17:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:00 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T21:17:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:17:01.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:17:01.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:17:01.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:17:01.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:00 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi038 to 59422k 2023-03-09T21:17:01.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:00 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi038 to 60848193: error parsing value: Value '60848193' is below minimum 939524096 2023-03-09T21:17:01.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:17:01.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:17:01.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:17:01.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:17:01.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:00 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T21:17:01.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:00 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T21:17:01.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:17:01.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:17:01.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:17:01.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:17:01.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:17:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22731.308511 secs 2023-03-09T21:17:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:17:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:17:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:00 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T21:17:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:00 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T21:17:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:17:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:17:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:17:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:17:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:00 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi038 to 59422k 2023-03-09T21:17:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:00 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi038 to 60848193: error parsing value: Value '60848193' is below minimum 939524096 2023-03-09T21:17:01.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:17:01.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:17:01.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:17:01.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:17:01.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:00 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T21:17:01.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:00 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T21:17:01.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:17:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:17:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:17:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:17:01.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:17:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22731.308511 secs 2023-03-09T21:17:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:17:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:17:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:00 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T21:17:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:00 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T21:17:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:17:01.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:17:01.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:17:01.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:17:01.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:00 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi038 to 59422k 2023-03-09T21:17:01.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:00 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi038 to 60848193: error parsing value: Value '60848193' is below minimum 939524096 2023-03-09T21:17:01.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:17:01.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:17:01.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:17:01.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:17:01.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:00 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T21:17:01.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:00 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T21:17:01.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:17:01.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:17:01.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:17:01.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:17:01.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:17:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:01 smithi122 ceph-mon[93869]: pgmap v12634: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 467 KiB/s rd, 428 KiB/s wr, 15 op/s 2023-03-09T21:17:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:01 smithi181 ceph-mon[94189]: pgmap v12634: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 467 KiB/s rd, 428 KiB/s wr, 15 op/s 2023-03-09T21:17:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:01 smithi038 ceph-mon[92422]: pgmap v12634: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 467 KiB/s rd, 428 KiB/s wr, 15 op/s 2023-03-09T21:17:03.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22733.643136 secs 2023-03-09T21:17:03.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22733.643136 secs 2023-03-09T21:17:03.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22733.643136 secs 2023-03-09T21:17:03.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23195.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:17:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:03 smithi122 ceph-mon[93869]: pgmap v12635: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 593 KiB/s rd, 465 KiB/s wr, 27 op/s 2023-03-09T21:17:04.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:03 smithi181 ceph-mon[94189]: pgmap v12635: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 593 KiB/s rd, 465 KiB/s wr, 27 op/s 2023-03-09T21:17:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:03 smithi038 ceph-mon[92422]: pgmap v12635: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 593 KiB/s rd, 465 KiB/s wr, 27 op/s 2023-03-09T21:17:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22736.308596 secs 2023-03-09T21:17:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:05 smithi122 ceph-mon[93869]: pgmap v12636: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 309 KiB/s wr, 24 op/s 2023-03-09T21:17:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22736.308596 secs 2023-03-09T21:17:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:05 smithi181 ceph-mon[94189]: pgmap v12636: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 309 KiB/s wr, 24 op/s 2023-03-09T21:17:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22736.308596 secs 2023-03-09T21:17:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:05 smithi038 ceph-mon[92422]: pgmap v12636: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 309 KiB/s wr, 24 op/s 2023-03-09T21:17:08.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23200.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:17:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22738.643245 secs 2023-03-09T21:17:08.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:08 smithi122 ceph-mon[93869]: pgmap v12637: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 309 KiB/s wr, 26 op/s 2023-03-09T21:17:08.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22738.643245 secs 2023-03-09T21:17:08.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:08 smithi181 ceph-mon[94189]: pgmap v12637: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 309 KiB/s wr, 26 op/s 2023-03-09T21:17:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22738.643245 secs 2023-03-09T21:17:09.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:08 smithi038 ceph-mon[92422]: pgmap v12637: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 309 KiB/s wr, 26 op/s 2023-03-09T21:17:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:10 smithi122 ceph-mon[93869]: pgmap v12638: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 377 KiB/s wr, 24 op/s 2023-03-09T21:17:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:10 smithi122 ceph-mon[93869]: osdmap e1003: 12 total, 12 up, 12 in 2023-03-09T21:17:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:10 smithi181 ceph-mon[94189]: pgmap v12638: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 377 KiB/s wr, 24 op/s 2023-03-09T21:17:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:10 smithi181 ceph-mon[94189]: osdmap e1003: 12 total, 12 up, 12 in 2023-03-09T21:17:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:10 smithi038 ceph-mon[92422]: pgmap v12638: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 377 KiB/s wr, 24 op/s 2023-03-09T21:17:11.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:10 smithi038 ceph-mon[92422]: osdmap e1003: 12 total, 12 up, 12 in 2023-03-09T21:17:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22741.308656 secs 2023-03-09T21:17:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22741.308656 secs 2023-03-09T21:17:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22741.308656 secs 2023-03-09T21:17:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:12 smithi122 ceph-mon[93869]: pgmap v12640: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 151 KiB/s wr, 23 op/s 2023-03-09T21:17:12.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:12 smithi181 ceph-mon[94189]: pgmap v12640: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 151 KiB/s wr, 23 op/s 2023-03-09T21:17:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:12 smithi038 ceph-mon[92422]: pgmap v12640: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 151 KiB/s wr, 23 op/s 2023-03-09T21:17:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23205.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:17:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22743.643363 secs 2023-03-09T21:17:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22743.643363 secs 2023-03-09T21:17:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22743.643363 secs 2023-03-09T21:17:14.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:14 smithi122 ceph-mon[93869]: pgmap v12641: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 104 KiB/s wr, 7 op/s 2023-03-09T21:17:14.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:14 smithi181 ceph-mon[94189]: pgmap v12641: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 104 KiB/s wr, 7 op/s 2023-03-09T21:17:14.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:14 smithi038 ceph-mon[92422]: pgmap v12641: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 104 KiB/s wr, 7 op/s 2023-03-09T21:17:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22746.308719 secs 2023-03-09T21:17:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:15 smithi122 ceph-mon[93869]: pgmap v12642: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 104 KiB/s wr, 7 op/s 2023-03-09T21:17:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22746.308719 secs 2023-03-09T21:17:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:15 smithi181 ceph-mon[94189]: pgmap v12642: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 104 KiB/s wr, 7 op/s 2023-03-09T21:17:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22746.308719 secs 2023-03-09T21:17:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:15 smithi038 ceph-mon[92422]: pgmap v12642: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 104 KiB/s wr, 7 op/s 2023-03-09T21:17:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23210.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:17:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22748.643478 secs 2023-03-09T21:17:18.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:18 smithi122 ceph-mon[93869]: pgmap v12643: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 104 KiB/s wr, 7 op/s 2023-03-09T21:17:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22748.643478 secs 2023-03-09T21:17:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:18 smithi181 ceph-mon[94189]: pgmap v12643: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 104 KiB/s wr, 7 op/s 2023-03-09T21:17:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22748.643478 secs 2023-03-09T21:17:19.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:18 smithi038 ceph-mon[92422]: pgmap v12643: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 104 KiB/s wr, 7 op/s 2023-03-09T21:17:20.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:20 smithi122 ceph-mon[93869]: pgmap v12644: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 74 KiB/s wr, 6 op/s 2023-03-09T21:17:20.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:20 smithi122 ceph-mon[93869]: osdmap e1004: 12 total, 12 up, 12 in 2023-03-09T21:17:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:20 smithi181 ceph-mon[94189]: pgmap v12644: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 74 KiB/s wr, 6 op/s 2023-03-09T21:17:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:20 smithi181 ceph-mon[94189]: osdmap e1004: 12 total, 12 up, 12 in 2023-03-09T21:17:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:20 smithi038 ceph-mon[92422]: pgmap v12644: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 74 KiB/s wr, 6 op/s 2023-03-09T21:17:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:20 smithi038 ceph-mon[92422]: osdmap e1004: 12 total, 12 up, 12 in 2023-03-09T21:17:21.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22751.308779 secs 2023-03-09T21:17:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22751.308779 secs 2023-03-09T21:17:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22751.308779 secs 2023-03-09T21:17:22.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:22 smithi122 ceph-mon[93869]: pgmap v12646: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 74 KiB/s wr, 6 op/s 2023-03-09T21:17:22.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:22 smithi181 ceph-mon[94189]: pgmap v12646: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 74 KiB/s wr, 6 op/s 2023-03-09T21:17:22.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:22 smithi038 ceph-mon[92422]: pgmap v12646: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 74 KiB/s wr, 6 op/s 2023-03-09T21:17:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23215.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:17:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22753.643611 secs 2023-03-09T21:17:23.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22753.643611 secs 2023-03-09T21:17:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22753.643611 secs 2023-03-09T21:17:24.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:24 smithi122 ceph-mon[93869]: pgmap v12647: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 74 KiB/s wr, 6 op/s 2023-03-09T21:17:24.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:24 smithi181 ceph-mon[94189]: pgmap v12647: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 74 KiB/s wr, 6 op/s 2023-03-09T21:17:24.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:24 smithi038 ceph-mon[92422]: pgmap v12647: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 74 KiB/s wr, 6 op/s 2023-03-09T21:17:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22756.308836 secs 2023-03-09T21:17:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:25 smithi122 ceph-mon[93869]: pgmap v12648: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 74 KiB/s wr, 6 op/s 2023-03-09T21:17:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22756.308836 secs 2023-03-09T21:17:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:25 smithi181 ceph-mon[94189]: pgmap v12648: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 74 KiB/s wr, 6 op/s 2023-03-09T21:17:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22756.308836 secs 2023-03-09T21:17:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:25 smithi038 ceph-mon[92422]: pgmap v12648: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 74 KiB/s wr, 6 op/s 2023-03-09T21:17:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23220.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:17:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22758.643741 secs 2023-03-09T21:17:28.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:28 smithi122 ceph-mon[93869]: pgmap v12649: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 74 KiB/s wr, 6 op/s 2023-03-09T21:17:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22758.643741 secs 2023-03-09T21:17:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:28 smithi181 ceph-mon[94189]: pgmap v12649: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 74 KiB/s wr, 6 op/s 2023-03-09T21:17:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22758.643741 secs 2023-03-09T21:17:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:28 smithi038 ceph-mon[92422]: pgmap v12649: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 74 KiB/s wr, 6 op/s 2023-03-09T21:17:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:30 smithi122 ceph-mon[93869]: pgmap v12650: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T21:17:30.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:30 smithi181 ceph-mon[94189]: pgmap v12650: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T21:17:30.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:30 smithi038 ceph-mon[92422]: pgmap v12650: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T21:17:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22761.308921 secs 2023-03-09T21:17:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22761.308921 secs 2023-03-09T21:17:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22761.308921 secs 2023-03-09T21:17:32.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:32 smithi122 ceph-mon[93869]: pgmap v12651: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T21:17:32.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:32 smithi181 ceph-mon[94189]: pgmap v12651: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T21:17:32.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:32 smithi038 ceph-mon[92422]: pgmap v12651: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T21:17:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23225.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:17:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22763.643822 secs 2023-03-09T21:17:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22763.643822 secs 2023-03-09T21:17:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22763.643822 secs 2023-03-09T21:17:34.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:34 smithi122 ceph-mon[93869]: pgmap v12652: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 7.1 KiB/s wr, 6 op/s 2023-03-09T21:17:34.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:34 smithi181 ceph-mon[94189]: pgmap v12652: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 7.1 KiB/s wr, 6 op/s 2023-03-09T21:17:34.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:34 smithi038 ceph-mon[92422]: pgmap v12652: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 7.1 KiB/s wr, 6 op/s 2023-03-09T21:17:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22766.308985 secs 2023-03-09T21:17:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:35 smithi122 ceph-mon[93869]: pgmap v12653: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 7.1 KiB/s wr, 5 op/s 2023-03-09T21:17:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22766.308985 secs 2023-03-09T21:17:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:35 smithi181 ceph-mon[94189]: pgmap v12653: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 7.1 KiB/s wr, 5 op/s 2023-03-09T21:17:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22766.308985 secs 2023-03-09T21:17:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:35 smithi038 ceph-mon[92422]: pgmap v12653: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 7.1 KiB/s wr, 5 op/s 2023-03-09T21:17:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23230.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:17:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22768.643936 secs 2023-03-09T21:17:38.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:38 smithi122 ceph-mon[93869]: pgmap v12654: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 7.7 KiB/s wr, 8 op/s 2023-03-09T21:17:38.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22768.643936 secs 2023-03-09T21:17:38.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:38 smithi181 ceph-mon[94189]: pgmap v12654: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 7.7 KiB/s wr, 8 op/s 2023-03-09T21:17:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22768.643936 secs 2023-03-09T21:17:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:38 smithi038 ceph-mon[92422]: pgmap v12654: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 7.7 KiB/s wr, 8 op/s 2023-03-09T21:17:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:39 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:17:39.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:39 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:17:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:39 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:17:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:39 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:17:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:39 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:17:40.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:39 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:17:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:40 smithi122 ceph-mon[93869]: pgmap v12655: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 202 KiB/s wr, 7 op/s 2023-03-09T21:17:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:40 smithi181 ceph-mon[94189]: pgmap v12655: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 202 KiB/s wr, 7 op/s 2023-03-09T21:17:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:40 smithi038 ceph-mon[92422]: pgmap v12655: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 202 KiB/s wr, 7 op/s 2023-03-09T21:17:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22771.309045 secs 2023-03-09T21:17:41.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:17:41.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:17:41.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:41 smithi122 ceph-mon[93869]: pgmap v12656: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 202 KiB/s wr, 8 op/s 2023-03-09T21:17:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22771.309045 secs 2023-03-09T21:17:41.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:17:41.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:17:41.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:41 smithi181 ceph-mon[94189]: pgmap v12656: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 202 KiB/s wr, 8 op/s 2023-03-09T21:17:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22771.309045 secs 2023-03-09T21:17:42.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:17:42.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:17:42.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:41 smithi038 ceph-mon[92422]: pgmap v12656: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 202 KiB/s wr, 8 op/s 2023-03-09T21:17:42.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22773.644071 secs 2023-03-09T21:17:42.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22773.644071 secs 2023-03-09T21:17:42.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22773.644071 secs 2023-03-09T21:17:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23235.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:17:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:43 smithi122 ceph-mon[93869]: pgmap v12657: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 245 KiB/s wr, 9 op/s 2023-03-09T21:17:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:43 smithi181 ceph-mon[94189]: pgmap v12657: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 245 KiB/s wr, 9 op/s 2023-03-09T21:17:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:43 smithi038 ceph-mon[92422]: pgmap v12657: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 245 KiB/s wr, 9 op/s 2023-03-09T21:17:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22776.309117 secs 2023-03-09T21:17:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:45 smithi122 ceph-mon[93869]: pgmap v12658: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 238 KiB/s wr, 8 op/s 2023-03-09T21:17:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22776.309117 secs 2023-03-09T21:17:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:45 smithi181 ceph-mon[94189]: pgmap v12658: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 238 KiB/s wr, 8 op/s 2023-03-09T21:17:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22776.309117 secs 2023-03-09T21:17:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:45 smithi038 ceph-mon[92422]: pgmap v12658: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 238 KiB/s wr, 8 op/s 2023-03-09T21:17:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23240.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:17:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:48 smithi122 ceph-mon[93869]: pgmap v12659: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 319 KiB/s wr, 12 op/s 2023-03-09T21:17:48.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:48 smithi181 ceph-mon[94189]: pgmap v12659: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 319 KiB/s wr, 12 op/s 2023-03-09T21:17:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:48 smithi038 ceph-mon[92422]: pgmap v12659: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 319 KiB/s wr, 12 op/s 2023-03-09T21:17:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:49 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:17:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:49 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:17:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:49 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:17:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:50 smithi122 ceph-mon[93869]: pgmap v12660: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 470 KiB/s rd, 441 KiB/s wr, 11 op/s 2023-03-09T21:17:50.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:50 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:17:50.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:50 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:17:50.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:50 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:17:50.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:50 smithi181 ceph-mon[94189]: pgmap v12660: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 470 KiB/s rd, 441 KiB/s wr, 11 op/s 2023-03-09T21:17:50.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:50 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:17:50.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:50 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:17:50.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:50 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:17:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:50 smithi038 ceph-mon[92422]: pgmap v12660: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 470 KiB/s rd, 441 KiB/s wr, 11 op/s 2023-03-09T21:17:51.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:50 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:17:51.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:50 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:17:51.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:50 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:17:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22781.309202 secs 2023-03-09T21:17:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:17:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:17:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22781.309202 secs 2023-03-09T21:17:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:17:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:17:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22781.309202 secs 2023-03-09T21:17:52.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:17:52.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:17:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:52 smithi122 ceph-mon[93869]: pgmap v12661: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 656 KiB/s rd, 553 KiB/s wr, 13 op/s 2023-03-09T21:17:52.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:52 smithi181 ceph-mon[94189]: pgmap v12661: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 656 KiB/s rd, 553 KiB/s wr, 13 op/s 2023-03-09T21:17:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:52 smithi038 ceph-mon[92422]: pgmap v12661: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 656 KiB/s rd, 553 KiB/s wr, 13 op/s 2023-03-09T21:17:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23245.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:17:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:17:53.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:17:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:17:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:17:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:17:54.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:17:54.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:54 smithi122 ceph-mon[93869]: pgmap v12662: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 676 KiB/s rd, 573 KiB/s wr, 13 op/s 2023-03-09T21:17:54.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:54 smithi181 ceph-mon[94189]: pgmap v12662: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 676 KiB/s rd, 573 KiB/s wr, 13 op/s 2023-03-09T21:17:54.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:54 smithi038 ceph-mon[92422]: pgmap v12662: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 676 KiB/s rd, 573 KiB/s wr, 13 op/s 2023-03-09T21:17:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:17:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:17:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:55 smithi122 ceph-mon[93869]: pgmap v12663: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 776 KiB/s rd, 678 KiB/s wr, 12 op/s 2023-03-09T21:17:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:17:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:17:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:55 smithi181 ceph-mon[94189]: pgmap v12663: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 776 KiB/s rd, 678 KiB/s wr, 12 op/s 2023-03-09T21:17:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:17:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:17:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:55 smithi038 ceph-mon[92422]: pgmap v12663: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 776 KiB/s rd, 678 KiB/s wr, 12 op/s 2023-03-09T21:17:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22786.309288 secs 2023-03-09T21:17:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22786.309288 secs 2023-03-09T21:17:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22786.309288 secs 2023-03-09T21:17:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:17:57.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:17:57.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22788.644425 secs 2023-03-09T21:17:57.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:17:57 smithi122 ceph-mon[93869]: pgmap v12664: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 796 KiB/s rd, 696 KiB/s wr, 15 op/s 2023-03-09T21:17:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:17:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:17:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22788.644425 secs 2023-03-09T21:17:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:17:57 smithi181 ceph-mon[94189]: pgmap v12664: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 796 KiB/s rd, 696 KiB/s wr, 15 op/s 2023-03-09T21:17:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:17:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:17:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22788.644425 secs 2023-03-09T21:17:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:17:57 smithi038 ceph-mon[92422]: pgmap v12664: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 796 KiB/s rd, 696 KiB/s wr, 15 op/s 2023-03-09T21:17:58.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23250.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:18:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:00 smithi122 ceph-mon[93869]: pgmap v12665: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 712 KiB/s rd, 615 KiB/s wr, 12 op/s 2023-03-09T21:18:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:18:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:00 smithi181 ceph-mon[94189]: pgmap v12665: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 712 KiB/s rd, 615 KiB/s wr, 12 op/s 2023-03-09T21:18:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:18:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:00 smithi038 ceph-mon[92422]: pgmap v12665: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 712 KiB/s rd, 615 KiB/s wr, 12 op/s 2023-03-09T21:18:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:18:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22791.309369 secs 2023-03-09T21:18:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:01 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:18:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:01 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:18:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:01 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:18:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22791.309369 secs 2023-03-09T21:18:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:01 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:18:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:01 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:18:01.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:01 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:18:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22791.309369 secs 2023-03-09T21:18:02.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:01 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:18:02.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:01 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:18:02.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:01 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:18:02.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:02 smithi122 ceph-mon[93869]: pgmap v12666: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 699 KiB/s rd, 633 KiB/s wr, 15 op/s 2023-03-09T21:18:02.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:02 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:18:02.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:02 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:18:02.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:02 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:18:02.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:02 smithi181 ceph-mon[94189]: pgmap v12666: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 699 KiB/s rd, 633 KiB/s wr, 15 op/s 2023-03-09T21:18:02.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:02 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:18:02.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:02 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:18:02.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:02 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:18:02.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:02 smithi038 ceph-mon[92422]: pgmap v12666: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 699 KiB/s rd, 633 KiB/s wr, 15 op/s 2023-03-09T21:18:03.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:02 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:18:03.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:02 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:18:03.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:02 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:18:03.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23255.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:18:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22793.644553 secs 2023-03-09T21:18:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22793.644553 secs 2023-03-09T21:18:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22793.644553 secs 2023-03-09T21:18:04.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:04 smithi122 ceph-mon[93869]: pgmap v12667: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 365 KiB/s wr, 24 op/s 2023-03-09T21:18:04.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:04 smithi181 ceph-mon[94189]: pgmap v12667: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 365 KiB/s wr, 24 op/s 2023-03-09T21:18:04.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:04 smithi038 ceph-mon[92422]: pgmap v12667: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 365 KiB/s wr, 24 op/s 2023-03-09T21:18:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22796.309427 secs 2023-03-09T21:18:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:05 smithi122 ceph-mon[93869]: pgmap v12668: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 386 KiB/s rd, 395 KiB/s wr, 24 op/s 2023-03-09T21:18:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22796.309427 secs 2023-03-09T21:18:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:05 smithi181 ceph-mon[94189]: pgmap v12668: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 386 KiB/s rd, 395 KiB/s wr, 24 op/s 2023-03-09T21:18:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22796.309427 secs 2023-03-09T21:18:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:05 smithi038 ceph-mon[92422]: pgmap v12668: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 386 KiB/s rd, 395 KiB/s wr, 24 op/s 2023-03-09T21:18:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23260.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:18:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22798.644682 secs 2023-03-09T21:18:08.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:08 smithi122 ceph-mon[93869]: pgmap v12669: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 307 KiB/s wr, 26 op/s 2023-03-09T21:18:08.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22798.644682 secs 2023-03-09T21:18:08.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:08 smithi181 ceph-mon[94189]: pgmap v12669: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 307 KiB/s wr, 26 op/s 2023-03-09T21:18:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22798.644682 secs 2023-03-09T21:18:09.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:08 smithi038 ceph-mon[92422]: pgmap v12669: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 307 KiB/s wr, 26 op/s 2023-03-09T21:18:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:10 smithi122 ceph-mon[93869]: pgmap v12670: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 289 KiB/s wr, 23 op/s 2023-03-09T21:18:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:10 smithi122 ceph-mon[93869]: osdmap e1005: 12 total, 12 up, 12 in 2023-03-09T21:18:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:10 smithi181 ceph-mon[94189]: pgmap v12670: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 289 KiB/s wr, 23 op/s 2023-03-09T21:18:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:10 smithi181 ceph-mon[94189]: osdmap e1005: 12 total, 12 up, 12 in 2023-03-09T21:18:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:10 smithi038 ceph-mon[92422]: pgmap v12670: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 289 KiB/s wr, 23 op/s 2023-03-09T21:18:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:10 smithi038 ceph-mon[92422]: osdmap e1005: 12 total, 12 up, 12 in 2023-03-09T21:18:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22801.309680 secs 2023-03-09T21:18:11.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:11 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:18:11.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:11 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:18:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22801.309680 secs 2023-03-09T21:18:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:11 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:18:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:11 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:18:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22801.309680 secs 2023-03-09T21:18:12.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:11 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:18:12.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:11 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:18:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:12 smithi122 ceph-mon[93869]: pgmap v12672: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 178 KiB/s wr, 24 op/s 2023-03-09T21:18:12.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:12 smithi181 ceph-mon[94189]: pgmap v12672: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 178 KiB/s wr, 24 op/s 2023-03-09T21:18:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:12 smithi038 ceph-mon[92422]: pgmap v12672: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 178 KiB/s wr, 24 op/s 2023-03-09T21:18:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23265.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:18:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22803.644795 secs 2023-03-09T21:18:13.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:18:13.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:18:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22803.644795 secs 2023-03-09T21:18:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:18:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:18:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22803.644795 secs 2023-03-09T21:18:14.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:18:14.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:18:14.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:14 smithi122 ceph-mon[93869]: pgmap v12673: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 133 KiB/s wr, 9 op/s 2023-03-09T21:18:14.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:14 smithi181 ceph-mon[94189]: pgmap v12673: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 133 KiB/s wr, 9 op/s 2023-03-09T21:18:14.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:14 smithi038 ceph-mon[92422]: pgmap v12673: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 133 KiB/s wr, 9 op/s 2023-03-09T21:18:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22806.309782 secs 2023-03-09T21:18:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:15 smithi122 ceph-mon[93869]: pgmap v12674: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 8 op/s 2023-03-09T21:18:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22806.309782 secs 2023-03-09T21:18:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:15 smithi181 ceph-mon[94189]: pgmap v12674: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 8 op/s 2023-03-09T21:18:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22806.309782 secs 2023-03-09T21:18:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:15 smithi038 ceph-mon[92422]: pgmap v12674: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 8 op/s 2023-03-09T21:18:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23270.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:18:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22808.644888 secs 2023-03-09T21:18:18.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:18 smithi122 ceph-mon[93869]: pgmap v12675: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 59 KiB/s wr, 7 op/s 2023-03-09T21:18:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22808.644888 secs 2023-03-09T21:18:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:18 smithi181 ceph-mon[94189]: pgmap v12675: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 59 KiB/s wr, 7 op/s 2023-03-09T21:18:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22808.644888 secs 2023-03-09T21:18:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:18 smithi038 ceph-mon[92422]: pgmap v12675: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 59 KiB/s wr, 7 op/s 2023-03-09T21:18:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:20 smithi122 ceph-mon[93869]: pgmap v12676: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 59 KiB/s wr, 7 op/s 2023-03-09T21:18:20.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:20 smithi122 ceph-mon[93869]: osdmap e1006: 12 total, 12 up, 12 in 2023-03-09T21:18:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:20 smithi181 ceph-mon[94189]: pgmap v12676: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 59 KiB/s wr, 7 op/s 2023-03-09T21:18:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:20 smithi181 ceph-mon[94189]: osdmap e1006: 12 total, 12 up, 12 in 2023-03-09T21:18:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:20 smithi038 ceph-mon[92422]: pgmap v12676: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 59 KiB/s wr, 7 op/s 2023-03-09T21:18:21.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:20 smithi038 ceph-mon[92422]: osdmap e1006: 12 total, 12 up, 12 in 2023-03-09T21:18:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22811.309841 secs 2023-03-09T21:18:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22811.309841 secs 2023-03-09T21:18:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22811.309841 secs 2023-03-09T21:18:22.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:22 smithi122 ceph-mon[93869]: pgmap v12678: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 59 KiB/s wr, 7 op/s 2023-03-09T21:18:22.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:22 smithi181 ceph-mon[94189]: pgmap v12678: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 59 KiB/s wr, 7 op/s 2023-03-09T21:18:22.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:22 smithi038 ceph-mon[92422]: pgmap v12678: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 59 KiB/s wr, 7 op/s 2023-03-09T21:18:23.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23275.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:18:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22813.645012 secs 2023-03-09T21:18:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22813.645012 secs 2023-03-09T21:18:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22813.645012 secs 2023-03-09T21:18:24.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:24 smithi122 ceph-mon[93869]: pgmap v12679: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 59 KiB/s wr, 7 op/s 2023-03-09T21:18:24.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:24 smithi181 ceph-mon[94189]: pgmap v12679: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 59 KiB/s wr, 7 op/s 2023-03-09T21:18:24.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:24 smithi038 ceph-mon[92422]: pgmap v12679: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 59 KiB/s wr, 7 op/s 2023-03-09T21:18:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22816.309903 secs 2023-03-09T21:18:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:25 smithi122 ceph-mon[93869]: pgmap v12680: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 59 KiB/s wr, 7 op/s 2023-03-09T21:18:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22816.309903 secs 2023-03-09T21:18:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:25 smithi181 ceph-mon[94189]: pgmap v12680: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 59 KiB/s wr, 7 op/s 2023-03-09T21:18:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22816.309903 secs 2023-03-09T21:18:26.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:25 smithi038 ceph-mon[92422]: pgmap v12680: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 59 KiB/s wr, 7 op/s 2023-03-09T21:18:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23280.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:18:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22818.645135 secs 2023-03-09T21:18:28.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:28 smithi122 ceph-mon[93869]: pgmap v12681: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-03-09T21:18:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22818.645135 secs 2023-03-09T21:18:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:28 smithi181 ceph-mon[94189]: pgmap v12681: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-03-09T21:18:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22818.645135 secs 2023-03-09T21:18:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:28 smithi038 ceph-mon[92422]: pgmap v12681: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-03-09T21:18:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:30 smithi122 ceph-mon[93869]: pgmap v12682: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T21:18:30.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:30 smithi181 ceph-mon[94189]: pgmap v12682: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T21:18:30.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:30 smithi038 ceph-mon[92422]: pgmap v12682: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T21:18:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22821.309964 secs 2023-03-09T21:18:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22821.309964 secs 2023-03-09T21:18:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22821.309964 secs 2023-03-09T21:18:32.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:32 smithi122 ceph-mon[93869]: pgmap v12683: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 88 B/s wr, 8 op/s 2023-03-09T21:18:32.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:32 smithi181 ceph-mon[94189]: pgmap v12683: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 88 B/s wr, 8 op/s 2023-03-09T21:18:32.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:32 smithi038 ceph-mon[92422]: pgmap v12683: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 88 B/s wr, 8 op/s 2023-03-09T21:18:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23285.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:18:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22823.645184 secs 2023-03-09T21:18:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22823.645184 secs 2023-03-09T21:18:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22823.645184 secs 2023-03-09T21:18:34.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:34 smithi122 ceph-mon[93869]: pgmap v12684: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 597 B/s wr, 7 op/s 2023-03-09T21:18:34.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:34 smithi181 ceph-mon[94189]: pgmap v12684: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 597 B/s wr, 7 op/s 2023-03-09T21:18:34.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:34 smithi038 ceph-mon[92422]: pgmap v12684: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 597 B/s wr, 7 op/s 2023-03-09T21:18:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22826.309986 secs 2023-03-09T21:18:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:35 smithi122 ceph-mon[93869]: pgmap v12685: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 597 B/s wr, 6 op/s 2023-03-09T21:18:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22826.309986 secs 2023-03-09T21:18:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:35 smithi181 ceph-mon[94189]: pgmap v12685: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 597 B/s wr, 6 op/s 2023-03-09T21:18:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22826.309986 secs 2023-03-09T21:18:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:35 smithi038 ceph-mon[92422]: pgmap v12685: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 597 B/s wr, 6 op/s 2023-03-09T21:18:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23290.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:18:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:38 smithi122 ceph-mon[93869]: pgmap v12686: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 91 KiB/s wr, 9 op/s 2023-03-09T21:18:38.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:38 smithi181 ceph-mon[94189]: pgmap v12686: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 91 KiB/s wr, 9 op/s 2023-03-09T21:18:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:38 smithi038 ceph-mon[92422]: pgmap v12686: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 91 KiB/s wr, 9 op/s 2023-03-09T21:18:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:40 smithi122 ceph-mon[93869]: pgmap v12687: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 91 KiB/s wr, 6 op/s 2023-03-09T21:18:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:40 smithi181 ceph-mon[94189]: pgmap v12687: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 91 KiB/s wr, 6 op/s 2023-03-09T21:18:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:40 smithi038 ceph-mon[92422]: pgmap v12687: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 91 KiB/s wr, 6 op/s 2023-03-09T21:18:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22831.310062 secs 2023-03-09T21:18:41.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:41 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:18:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22831.310062 secs 2023-03-09T21:18:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:41 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:18:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22831.310062 secs 2023-03-09T21:18:42.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:41 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:18:42.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:42 smithi122 ceph-mon[93869]: pgmap v12688: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 234 KiB/s wr, 9 op/s 2023-03-09T21:18:42.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:42 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:18:42.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:42 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:18:42.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:42 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:18:42.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:42 smithi181 ceph-mon[94189]: pgmap v12688: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 234 KiB/s wr, 9 op/s 2023-03-09T21:18:42.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:42 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:18:42.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:42 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:18:42.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:42 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:18:42.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:42 smithi038 ceph-mon[92422]: pgmap v12688: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 234 KiB/s wr, 9 op/s 2023-03-09T21:18:42.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:42 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:18:43.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:42 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:18:43.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:42 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:18:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23295.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:18:44.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:44 smithi122 ceph-mon[93869]: pgmap v12689: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 234 KiB/s wr, 7 op/s 2023-03-09T21:18:44.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:44 smithi181 ceph-mon[94189]: pgmap v12689: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 234 KiB/s wr, 7 op/s 2023-03-09T21:18:44.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:44 smithi038 ceph-mon[92422]: pgmap v12689: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 234 KiB/s wr, 7 op/s 2023-03-09T21:18:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22836.310123 secs 2023-03-09T21:18:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:45 smithi122 ceph-mon[93869]: pgmap v12690: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 449 KiB/s wr, 6 op/s 2023-03-09T21:18:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22836.310123 secs 2023-03-09T21:18:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:45 smithi181 ceph-mon[94189]: pgmap v12690: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 449 KiB/s wr, 6 op/s 2023-03-09T21:18:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22836.310123 secs 2023-03-09T21:18:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:45 smithi038 ceph-mon[92422]: pgmap v12690: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 449 KiB/s wr, 6 op/s 2023-03-09T21:18:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23300.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:18:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:48 smithi122 ceph-mon[93869]: pgmap v12691: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 690 KiB/s rd, 467 KiB/s wr, 9 op/s 2023-03-09T21:18:48.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:48 smithi181 ceph-mon[94189]: pgmap v12691: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 690 KiB/s rd, 467 KiB/s wr, 9 op/s 2023-03-09T21:18:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:48 smithi038 ceph-mon[92422]: pgmap v12691: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 690 KiB/s rd, 467 KiB/s wr, 9 op/s 2023-03-09T21:18:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:18:49.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:18:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:18:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:18:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:18:50.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:18:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:50 smithi122 ceph-mon[93869]: pgmap v12692: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 610 KiB/s rd, 377 KiB/s wr, 7 op/s 2023-03-09T21:18:50.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:50 smithi181 ceph-mon[94189]: pgmap v12692: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 610 KiB/s rd, 377 KiB/s wr, 7 op/s 2023-03-09T21:18:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:50 smithi038 ceph-mon[92422]: pgmap v12692: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 610 KiB/s rd, 377 KiB/s wr, 7 op/s 2023-03-09T21:18:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22841.310198 secs 2023-03-09T21:18:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:18:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:18:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22841.310198 secs 2023-03-09T21:18:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:18:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:18:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22841.310198 secs 2023-03-09T21:18:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:18:52.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:18:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:52 smithi122 ceph-mon[93869]: pgmap v12693: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 739 KiB/s rd, 504 KiB/s wr, 11 op/s 2023-03-09T21:18:52.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:52 smithi181 ceph-mon[94189]: pgmap v12693: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 739 KiB/s rd, 504 KiB/s wr, 11 op/s 2023-03-09T21:18:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:52 smithi038 ceph-mon[92422]: pgmap v12693: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 739 KiB/s rd, 504 KiB/s wr, 11 op/s 2023-03-09T21:18:53.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23305.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:18:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:53 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 22843.645544 secs 2023-03-09T21:18:53.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:53 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:18:53.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:18:53.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:18:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:53 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 22843.645544 secs 2023-03-09T21:18:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:53 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:18:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:18:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:18:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:53 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 22843.645544 secs 2023-03-09T21:18:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:53 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:18:54.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:18:54.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:18:54.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:54 smithi122 ceph-mon[93869]: pgmap v12694: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 738 KiB/s rd, 549 KiB/s wr, 11 op/s 2023-03-09T21:18:54.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:54 smithi181 ceph-mon[94189]: pgmap v12694: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 738 KiB/s rd, 549 KiB/s wr, 11 op/s 2023-03-09T21:18:54.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:54 smithi038 ceph-mon[92422]: pgmap v12694: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 738 KiB/s rd, 549 KiB/s wr, 11 op/s 2023-03-09T21:18:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22846.310256 secs 2023-03-09T21:18:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:18:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:18:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:55 smithi122 ceph-mon[93869]: pgmap v12695: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 738 KiB/s rd, 549 KiB/s wr, 10 op/s 2023-03-09T21:18:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22846.310256 secs 2023-03-09T21:18:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:18:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:18:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:55 smithi181 ceph-mon[94189]: pgmap v12695: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 738 KiB/s rd, 549 KiB/s wr, 10 op/s 2023-03-09T21:18:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22846.310256 secs 2023-03-09T21:18:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:18:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:18:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:55 smithi038 ceph-mon[92422]: pgmap v12695: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 738 KiB/s rd, 549 KiB/s wr, 10 op/s 2023-03-09T21:18:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:18:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:18:57.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:18:57.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:18:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:18:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:18:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22848.645630 secs 2023-03-09T21:18:58.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:18:57 smithi122 ceph-mon[93869]: pgmap v12696: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 545 KiB/s rd, 355 KiB/s wr, 13 op/s 2023-03-09T21:18:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22848.645630 secs 2023-03-09T21:18:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:18:57 smithi181 ceph-mon[94189]: pgmap v12696: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 545 KiB/s rd, 355 KiB/s wr, 13 op/s 2023-03-09T21:18:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22848.645630 secs 2023-03-09T21:18:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:18:57 smithi038 ceph-mon[92422]: pgmap v12696: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 545 KiB/s rd, 355 KiB/s wr, 13 op/s 2023-03-09T21:18:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23310.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:19:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:00 smithi122 ceph-mon[93869]: pgmap v12697: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 503 KiB/s wr, 12 op/s 2023-03-09T21:19:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:00 smithi181 ceph-mon[94189]: pgmap v12697: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 503 KiB/s wr, 12 op/s 2023-03-09T21:19:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:00 smithi038 ceph-mon[92422]: pgmap v12697: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 503 KiB/s wr, 12 op/s 2023-03-09T21:19:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22851.310318 secs 2023-03-09T21:19:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:19:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:19:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:01 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:19:01.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22851.310318 secs 2023-03-09T21:19:01.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:19:01.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:19:01.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:01 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:19:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22851.310318 secs 2023-03-09T21:19:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:19:02.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:19:02.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:01 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:19:02.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:02 smithi122 ceph-mon[93869]: pgmap v12698: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 508 KiB/s rd, 516 KiB/s wr, 15 op/s 2023-03-09T21:19:02.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:19:02.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:19:02.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:19:02.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:19:02.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:19:02.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:02 smithi181 ceph-mon[94189]: pgmap v12698: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 508 KiB/s rd, 516 KiB/s wr, 15 op/s 2023-03-09T21:19:02.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:19:02.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:19:02.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:19:02.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:19:02.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:19:03.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:02 smithi038 ceph-mon[92422]: pgmap v12698: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 508 KiB/s rd, 516 KiB/s wr, 15 op/s 2023-03-09T21:19:03.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:19:03.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:19:03.004 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:19:03.004 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:19:03.004 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:19:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23315.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:19:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22853.645740 secs 2023-03-09T21:19:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22853.645740 secs 2023-03-09T21:19:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22853.645740 secs 2023-03-09T21:19:04.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:04 smithi122 ceph-mon[93869]: pgmap v12699: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 435 KiB/s wr, 26 op/s 2023-03-09T21:19:04.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:04 smithi181 ceph-mon[94189]: pgmap v12699: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 435 KiB/s wr, 26 op/s 2023-03-09T21:19:04.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:04 smithi038 ceph-mon[92422]: pgmap v12699: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 435 KiB/s wr, 26 op/s 2023-03-09T21:19:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:05 smithi122 ceph-mon[93869]: osdmap e1007: 12 total, 12 up, 12 in 2023-03-09T21:19:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:05 smithi122 ceph-mon[93869]: pgmap v12701: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 296 KiB/s wr, 27 op/s 2023-03-09T21:19:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:05 smithi181 ceph-mon[94189]: osdmap e1007: 12 total, 12 up, 12 in 2023-03-09T21:19:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:05 smithi181 ceph-mon[94189]: pgmap v12701: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 296 KiB/s wr, 27 op/s 2023-03-09T21:19:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:05 smithi038 ceph-mon[92422]: osdmap e1007: 12 total, 12 up, 12 in 2023-03-09T21:19:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:05 smithi038 ceph-mon[92422]: pgmap v12701: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 296 KiB/s wr, 27 op/s 2023-03-09T21:19:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22856.310378 secs 2023-03-09T21:19:06.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22856.310378 secs 2023-03-09T21:19:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22856.310378 secs 2023-03-09T21:19:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:19:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:19:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22858.645862 secs 2023-03-09T21:19:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:07 smithi122 ceph-mon[93869]: pgmap v12702: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 524 KiB/s wr, 29 op/s 2023-03-09T21:19:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:19:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:19:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22858.645862 secs 2023-03-09T21:19:08.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:07 smithi181 ceph-mon[94189]: pgmap v12702: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 524 KiB/s wr, 29 op/s 2023-03-09T21:19:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:19:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:19:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22858.645862 secs 2023-03-09T21:19:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:07 smithi038 ceph-mon[92422]: pgmap v12702: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 524 KiB/s wr, 29 op/s 2023-03-09T21:19:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23320.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:19:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:19:09.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:19:09.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:19:09.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:19:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:19:09.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:19:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:09 smithi038 ceph-mon[92422]: pgmap v12703: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 323 KiB/s wr, 27 op/s 2023-03-09T21:19:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:09 smithi122 ceph-mon[93869]: pgmap v12703: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 323 KiB/s wr, 27 op/s 2023-03-09T21:19:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:09 smithi181 ceph-mon[94189]: pgmap v12703: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 323 KiB/s wr, 27 op/s 2023-03-09T21:19:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22861.310437 secs 2023-03-09T21:19:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22861.310437 secs 2023-03-09T21:19:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22861.310437 secs 2023-03-09T21:19:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:11 smithi038 ceph-mon[92422]: pgmap v12704: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 308 KiB/s wr, 25 op/s 2023-03-09T21:19:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:11 smithi122 ceph-mon[93869]: pgmap v12704: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 308 KiB/s wr, 25 op/s 2023-03-09T21:19:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:11 smithi181 ceph-mon[94189]: pgmap v12704: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 308 KiB/s wr, 25 op/s 2023-03-09T21:19:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22863.645980 secs 2023-03-09T21:19:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22863.645980 secs 2023-03-09T21:19:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22863.645980 secs 2023-03-09T21:19:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23325.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:19:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:13 smithi038 ceph-mon[92422]: pgmap v12705: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 328 KiB/s wr, 10 op/s 2023-03-09T21:19:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:13 smithi122 ceph-mon[93869]: pgmap v12705: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 328 KiB/s wr, 10 op/s 2023-03-09T21:19:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:13 smithi181 ceph-mon[94189]: pgmap v12705: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 328 KiB/s wr, 10 op/s 2023-03-09T21:19:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22866.310497 secs 2023-03-09T21:19:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:15 smithi122 ceph-mon[93869]: osdmap e1008: 12 total, 12 up, 12 in 2023-03-09T21:19:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:15 smithi122 ceph-mon[93869]: pgmap v12707: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 328 KiB/s wr, 9 op/s 2023-03-09T21:19:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22866.310497 secs 2023-03-09T21:19:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:15 smithi181 ceph-mon[94189]: osdmap e1008: 12 total, 12 up, 12 in 2023-03-09T21:19:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:15 smithi181 ceph-mon[94189]: pgmap v12707: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 328 KiB/s wr, 9 op/s 2023-03-09T21:19:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22866.310497 secs 2023-03-09T21:19:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:15 smithi038 ceph-mon[92422]: osdmap e1008: 12 total, 12 up, 12 in 2023-03-09T21:19:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:15 smithi038 ceph-mon[92422]: pgmap v12707: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 328 KiB/s wr, 9 op/s 2023-03-09T21:19:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23330.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:19:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22868.646096 secs 2023-03-09T21:19:18.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:18 smithi122 ceph-mon[93869]: pgmap v12708: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 75 KiB/s wr, 6 op/s 2023-03-09T21:19:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22868.646096 secs 2023-03-09T21:19:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:18 smithi181 ceph-mon[94189]: pgmap v12708: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 75 KiB/s wr, 6 op/s 2023-03-09T21:19:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22868.646096 secs 2023-03-09T21:19:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:18 smithi038 ceph-mon[92422]: pgmap v12708: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 75 KiB/s wr, 6 op/s 2023-03-09T21:19:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:20 smithi122 ceph-mon[93869]: pgmap v12709: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 75 KiB/s wr, 6 op/s 2023-03-09T21:19:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:20 smithi181 ceph-mon[94189]: pgmap v12709: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 75 KiB/s wr, 6 op/s 2023-03-09T21:19:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:20 smithi038 ceph-mon[92422]: pgmap v12709: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 75 KiB/s wr, 6 op/s 2023-03-09T21:19:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22871.310556 secs 2023-03-09T21:19:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22871.310556 secs 2023-03-09T21:19:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22871.310556 secs 2023-03-09T21:19:22.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:22 smithi122 ceph-mon[93869]: pgmap v12710: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 75 KiB/s wr, 6 op/s 2023-03-09T21:19:22.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:22 smithi181 ceph-mon[94189]: pgmap v12710: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 75 KiB/s wr, 6 op/s 2023-03-09T21:19:22.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:22 smithi038 ceph-mon[92422]: pgmap v12710: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 75 KiB/s wr, 6 op/s 2023-03-09T21:19:23.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23335.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:19:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22873.646216 secs 2023-03-09T21:19:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22873.646216 secs 2023-03-09T21:19:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22873.646216 secs 2023-03-09T21:19:24.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:24 smithi122 ceph-mon[93869]: pgmap v12711: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-09T21:19:24.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:24 smithi181 ceph-mon[94189]: pgmap v12711: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-09T21:19:24.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:24 smithi038 ceph-mon[92422]: pgmap v12711: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-09T21:19:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22876.310618 secs 2023-03-09T21:19:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:25 smithi122 ceph-mon[93869]: pgmap v12712: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2023-03-09T21:19:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22876.310618 secs 2023-03-09T21:19:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:25 smithi181 ceph-mon[94189]: pgmap v12712: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2023-03-09T21:19:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22876.310618 secs 2023-03-09T21:19:26.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:25 smithi038 ceph-mon[92422]: pgmap v12712: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2023-03-09T21:19:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23340.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:19:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22878.646312 secs 2023-03-09T21:19:28.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:28 smithi122 ceph-mon[93869]: pgmap v12713: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T21:19:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22878.646312 secs 2023-03-09T21:19:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:28 smithi181 ceph-mon[94189]: pgmap v12713: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T21:19:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22878.646312 secs 2023-03-09T21:19:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:28 smithi038 ceph-mon[92422]: pgmap v12713: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T21:19:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:30 smithi122 ceph-mon[93869]: pgmap v12714: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T21:19:30.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:30 smithi181 ceph-mon[94189]: pgmap v12714: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T21:19:30.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:30 smithi038 ceph-mon[92422]: pgmap v12714: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T21:19:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22881.310677 secs 2023-03-09T21:19:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22881.310677 secs 2023-03-09T21:19:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22881.310677 secs 2023-03-09T21:19:32.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:32 smithi122 ceph-mon[93869]: pgmap v12715: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 6 op/s 2023-03-09T21:19:32.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:32 smithi181 ceph-mon[94189]: pgmap v12715: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 6 op/s 2023-03-09T21:19:32.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:32 smithi038 ceph-mon[92422]: pgmap v12715: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 6 op/s 2023-03-09T21:19:33.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23345.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:19:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22883.646391 secs 2023-03-09T21:19:33.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:33 smithi122 ceph-mon[93869]: pgmap v12716: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 75 KiB/s wr, 6 op/s 2023-03-09T21:19:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22883.646391 secs 2023-03-09T21:19:33.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:33 smithi181 ceph-mon[94189]: pgmap v12716: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 75 KiB/s wr, 6 op/s 2023-03-09T21:19:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22883.646391 secs 2023-03-09T21:19:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:33 smithi038 ceph-mon[92422]: pgmap v12716: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 75 KiB/s wr, 6 op/s 2023-03-09T21:19:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22886.310736 secs 2023-03-09T21:19:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:35 smithi122 ceph-mon[93869]: pgmap v12717: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 75 KiB/s wr, 6 op/s 2023-03-09T21:19:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22886.310736 secs 2023-03-09T21:19:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:35 smithi181 ceph-mon[94189]: pgmap v12717: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 75 KiB/s wr, 6 op/s 2023-03-09T21:19:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22886.310736 secs 2023-03-09T21:19:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:35 smithi038 ceph-mon[92422]: pgmap v12717: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 75 KiB/s wr, 6 op/s 2023-03-09T21:19:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23350.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:19:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22888.646505 secs 2023-03-09T21:19:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:38 smithi122 ceph-mon[93869]: pgmap v12718: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 166 KiB/s wr, 9 op/s 2023-03-09T21:19:38.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22888.646505 secs 2023-03-09T21:19:38.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:38 smithi181 ceph-mon[94189]: pgmap v12718: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 166 KiB/s wr, 9 op/s 2023-03-09T21:19:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22888.646505 secs 2023-03-09T21:19:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:38 smithi038 ceph-mon[92422]: pgmap v12718: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 166 KiB/s wr, 9 op/s 2023-03-09T21:19:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:40 smithi122 ceph-mon[93869]: pgmap v12719: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 166 KiB/s wr, 7 op/s 2023-03-09T21:19:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:40 smithi181 ceph-mon[94189]: pgmap v12719: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 166 KiB/s wr, 7 op/s 2023-03-09T21:19:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:40 smithi038 ceph-mon[92422]: pgmap v12719: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 166 KiB/s wr, 7 op/s 2023-03-09T21:19:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22891.310797 secs 2023-03-09T21:19:41.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:19:41.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:19:41.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:41 smithi122 ceph-mon[93869]: pgmap v12720: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 265 KiB/s wr, 9 op/s 2023-03-09T21:19:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22891.310797 secs 2023-03-09T21:19:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:19:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:19:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:41 smithi181 ceph-mon[94189]: pgmap v12720: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 265 KiB/s wr, 9 op/s 2023-03-09T21:19:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22891.310797 secs 2023-03-09T21:19:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:19:42.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:19:42.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:41 smithi038 ceph-mon[92422]: pgmap v12720: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 265 KiB/s wr, 9 op/s 2023-03-09T21:19:42.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22893.646596 secs 2023-03-09T21:19:42.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22893.646596 secs 2023-03-09T21:19:42.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22893.646596 secs 2023-03-09T21:19:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23355.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:19:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:43 smithi122 ceph-mon[93869]: pgmap v12721: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 265 KiB/s wr, 8 op/s 2023-03-09T21:19:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:43 smithi181 ceph-mon[94189]: pgmap v12721: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 265 KiB/s wr, 8 op/s 2023-03-09T21:19:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:43 smithi038 ceph-mon[92422]: pgmap v12721: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 265 KiB/s wr, 8 op/s 2023-03-09T21:19:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22896.310858 secs 2023-03-09T21:19:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:45 smithi122 ceph-mon[93869]: pgmap v12722: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 190 KiB/s wr, 7 op/s 2023-03-09T21:19:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22896.310858 secs 2023-03-09T21:19:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:45 smithi181 ceph-mon[94189]: pgmap v12722: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 190 KiB/s wr, 7 op/s 2023-03-09T21:19:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22896.310858 secs 2023-03-09T21:19:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:45 smithi038 ceph-mon[92422]: pgmap v12722: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 190 KiB/s wr, 7 op/s 2023-03-09T21:19:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:19:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:19:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22898.646732 secs 2023-03-09T21:19:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:47 smithi181 ceph-mon[94189]: pgmap v12723: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 367 KiB/s wr, 10 op/s 2023-03-09T21:19:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:19:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:19:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22898.646732 secs 2023-03-09T21:19:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:47 smithi038 ceph-mon[92422]: pgmap v12723: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 367 KiB/s wr, 10 op/s 2023-03-09T21:19:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:19:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:19:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22898.646732 secs 2023-03-09T21:19:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:47 smithi122 ceph-mon[93869]: pgmap v12723: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 367 KiB/s wr, 10 op/s 2023-03-09T21:19:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23360.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:19:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:50 smithi122 ceph-mon[93869]: pgmap v12724: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 379 KiB/s wr, 8 op/s 2023-03-09T21:19:50.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:50 smithi181 ceph-mon[94189]: pgmap v12724: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 379 KiB/s wr, 8 op/s 2023-03-09T21:19:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:50 smithi038 ceph-mon[92422]: pgmap v12724: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 379 KiB/s wr, 8 op/s 2023-03-09T21:19:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22901.310921 secs 2023-03-09T21:19:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:19:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:19:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22901.310921 secs 2023-03-09T21:19:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:19:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:19:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22901.310921 secs 2023-03-09T21:19:52.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:19:52.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:19:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:52 smithi122 ceph-mon[93869]: pgmap v12725: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 537 KiB/s rd, 379 KiB/s wr, 10 op/s 2023-03-09T21:19:52.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:52 smithi181 ceph-mon[94189]: pgmap v12725: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 537 KiB/s rd, 379 KiB/s wr, 10 op/s 2023-03-09T21:19:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:52 smithi038 ceph-mon[92422]: pgmap v12725: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 537 KiB/s rd, 379 KiB/s wr, 10 op/s 2023-03-09T21:19:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23365.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:19:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22903.646852 secs 2023-03-09T21:19:53.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:19:53.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:19:53.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:53 smithi122 ceph-mon[93869]: pgmap v12726: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 356 KiB/s wr, 10 op/s 2023-03-09T21:19:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22903.646852 secs 2023-03-09T21:19:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:19:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:19:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:53 smithi181 ceph-mon[94189]: pgmap v12726: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 356 KiB/s wr, 10 op/s 2023-03-09T21:19:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22903.646852 secs 2023-03-09T21:19:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:19:54.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:19:54.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:53 smithi038 ceph-mon[92422]: pgmap v12726: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 356 KiB/s wr, 10 op/s 2023-03-09T21:19:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:19:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:19:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:55 smithi122 ceph-mon[93869]: pgmap v12727: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 750 KiB/s rd, 595 KiB/s wr, 10 op/s 2023-03-09T21:19:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:19:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:19:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:55 smithi181 ceph-mon[94189]: pgmap v12727: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 750 KiB/s rd, 595 KiB/s wr, 10 op/s 2023-03-09T21:19:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:19:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:19:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:55 smithi038 ceph-mon[92422]: pgmap v12727: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 750 KiB/s rd, 595 KiB/s wr, 10 op/s 2023-03-09T21:19:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22906.310993 secs 2023-03-09T21:19:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22906.310993 secs 2023-03-09T21:19:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22906.310993 secs 2023-03-09T21:19:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:19:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:19:57.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22908.646965 secs 2023-03-09T21:19:57.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:19:57 smithi122 ceph-mon[93869]: pgmap v12728: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 770 KiB/s rd, 614 KiB/s wr, 14 op/s 2023-03-09T21:19:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:19:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:19:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22908.646965 secs 2023-03-09T21:19:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:19:57 smithi181 ceph-mon[94189]: pgmap v12728: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 770 KiB/s rd, 614 KiB/s wr, 14 op/s 2023-03-09T21:19:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:19:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:19:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22908.646965 secs 2023-03-09T21:19:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:19:57 smithi038 ceph-mon[92422]: pgmap v12728: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 770 KiB/s rd, 614 KiB/s wr, 14 op/s 2023-03-09T21:19:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23370.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:20:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:00 smithi122 ceph-mon[93869]: pgmap v12729: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 549 KiB/s wr, 12 op/s 2023-03-09T21:20:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:00 smithi122 ceph-mon[93869]: overall HEALTH_WARN 2 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-09T21:20:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:00 smithi181 ceph-mon[94189]: pgmap v12729: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 549 KiB/s wr, 12 op/s 2023-03-09T21:20:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:00 smithi181 ceph-mon[94189]: overall HEALTH_WARN 2 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-09T21:20:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:00 smithi038 ceph-mon[92422]: pgmap v12729: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 549 KiB/s wr, 12 op/s 2023-03-09T21:20:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:00 smithi038 ceph-mon[92422]: overall HEALTH_WARN 2 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-09T21:20:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22911.311054 secs 2023-03-09T21:20:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22911.311054 secs 2023-03-09T21:20:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22911.311054 secs 2023-03-09T21:20:02.869 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:02 smithi122 ceph-mon[93869]: pgmap v12730: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 447 KiB/s wr, 13 op/s 2023-03-09T21:20:02.869 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:20:02.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:02 smithi181 ceph-mon[94189]: pgmap v12730: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 447 KiB/s wr, 13 op/s 2023-03-09T21:20:02.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:20:02.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:02 smithi038 ceph-mon[92422]: pgmap v12730: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 447 KiB/s wr, 13 op/s 2023-03-09T21:20:02.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:20:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23375.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:20:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22913.647065 secs 2023-03-09T21:20:03.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:03 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:20:03.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:03 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:20:03.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:03 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:20:03.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:03 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:20:03.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:03 smithi122 ceph-mon[93869]: pgmap v12731: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 663 KiB/s rd, 677 KiB/s wr, 25 op/s 2023-03-09T21:20:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22913.647065 secs 2023-03-09T21:20:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:03 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:20:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:03 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:20:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:03 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:20:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:03 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:20:03.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:03 smithi181 ceph-mon[94189]: pgmap v12731: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 663 KiB/s rd, 677 KiB/s wr, 25 op/s 2023-03-09T21:20:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22913.647065 secs 2023-03-09T21:20:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:03 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:20:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:03 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:20:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:03 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:20:04.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:03 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:20:04.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:03 smithi038 ceph-mon[92422]: pgmap v12731: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 663 KiB/s rd, 677 KiB/s wr, 25 op/s 2023-03-09T21:20:04.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:20:04.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:20:04.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:20:04.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:20:04.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:20:04.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:20:05.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:20:05.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:20:05.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:20:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22916.311119 secs 2023-03-09T21:20:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:05 smithi038 ceph-mon[92422]: osdmap e1009: 12 total, 12 up, 12 in 2023-03-09T21:20:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:05 smithi038 ceph-mon[92422]: pgmap v12733: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 623 KiB/s wr, 27 op/s 2023-03-09T21:20:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22916.311119 secs 2023-03-09T21:20:06.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:05 smithi122 ceph-mon[93869]: osdmap e1009: 12 total, 12 up, 12 in 2023-03-09T21:20:06.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:05 smithi122 ceph-mon[93869]: pgmap v12733: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 623 KiB/s wr, 27 op/s 2023-03-09T21:20:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22916.311119 secs 2023-03-09T21:20:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:05 smithi181 ceph-mon[94189]: osdmap e1009: 12 total, 12 up, 12 in 2023-03-09T21:20:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:05 smithi181 ceph-mon[94189]: pgmap v12733: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 623 KiB/s wr, 27 op/s 2023-03-09T21:20:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:20:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:20:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22918.647186 secs 2023-03-09T21:20:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:07 smithi038 ceph-mon[92422]: pgmap v12734: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 628 KiB/s wr, 29 op/s 2023-03-09T21:20:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:20:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:20:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22918.647186 secs 2023-03-09T21:20:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:07 smithi122 ceph-mon[93869]: pgmap v12734: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 628 KiB/s wr, 29 op/s 2023-03-09T21:20:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:20:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:20:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22918.647186 secs 2023-03-09T21:20:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:07 smithi181 ceph-mon[94189]: pgmap v12734: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 628 KiB/s wr, 29 op/s 2023-03-09T21:20:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23380.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:20:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:20:09.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:20:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:20:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:20:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:20:09.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:20:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:10 smithi122 ceph-mon[93869]: pgmap v12735: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 453 KiB/s rd, 568 KiB/s wr, 28 op/s 2023-03-09T21:20:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:10 smithi181 ceph-mon[94189]: pgmap v12735: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 453 KiB/s rd, 568 KiB/s wr, 28 op/s 2023-03-09T21:20:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:10 smithi038 ceph-mon[92422]: pgmap v12735: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 453 KiB/s rd, 568 KiB/s wr, 28 op/s 2023-03-09T21:20:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22921.311180 secs 2023-03-09T21:20:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22921.311180 secs 2023-03-09T21:20:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22921.311180 secs 2023-03-09T21:20:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:12 smithi122 ceph-mon[93869]: pgmap v12736: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 452 KiB/s rd, 568 KiB/s wr, 28 op/s 2023-03-09T21:20:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:12 smithi181 ceph-mon[94189]: pgmap v12736: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 452 KiB/s rd, 568 KiB/s wr, 28 op/s 2023-03-09T21:20:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:12 smithi038 ceph-mon[92422]: pgmap v12736: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 452 KiB/s rd, 568 KiB/s wr, 28 op/s 2023-03-09T21:20:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22923.647272 secs 2023-03-09T21:20:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22923.647272 secs 2023-03-09T21:20:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22923.647272 secs 2023-03-09T21:20:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23385.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:20:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:14 smithi122 ceph-mon[93869]: pgmap v12737: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 291 KiB/s wr, 11 op/s 2023-03-09T21:20:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:14 smithi181 ceph-mon[94189]: pgmap v12737: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 291 KiB/s wr, 11 op/s 2023-03-09T21:20:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:14 smithi038 ceph-mon[92422]: pgmap v12737: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 291 KiB/s wr, 11 op/s 2023-03-09T21:20:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22926.311290 secs 2023-03-09T21:20:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:15 smithi122 ceph-mon[93869]: osdmap e1010: 12 total, 12 up, 12 in 2023-03-09T21:20:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:15 smithi122 ceph-mon[93869]: pgmap v12739: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 102 KiB/s wr, 11 op/s 2023-03-09T21:20:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22926.311290 secs 2023-03-09T21:20:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:15 smithi181 ceph-mon[94189]: osdmap e1010: 12 total, 12 up, 12 in 2023-03-09T21:20:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:15 smithi181 ceph-mon[94189]: pgmap v12739: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 102 KiB/s wr, 11 op/s 2023-03-09T21:20:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22926.311290 secs 2023-03-09T21:20:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:15 smithi038 ceph-mon[92422]: osdmap e1010: 12 total, 12 up, 12 in 2023-03-09T21:20:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:15 smithi038 ceph-mon[92422]: pgmap v12739: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 102 KiB/s wr, 11 op/s 2023-03-09T21:20:18.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23390.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:20:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22928.647335 secs 2023-03-09T21:20:18.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:18 smithi122 ceph-mon[93869]: pgmap v12740: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 75 KiB/s wr, 8 op/s 2023-03-09T21:20:18.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22928.647335 secs 2023-03-09T21:20:18.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:18 smithi181 ceph-mon[94189]: pgmap v12740: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 75 KiB/s wr, 8 op/s 2023-03-09T21:20:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22928.647335 secs 2023-03-09T21:20:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:18 smithi038 ceph-mon[92422]: pgmap v12740: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 75 KiB/s wr, 8 op/s 2023-03-09T21:20:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:19 smithi122 ceph-mon[93869]: pgmap v12741: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 511 B/s wr, 7 op/s 2023-03-09T21:20:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:19 smithi181 ceph-mon[94189]: pgmap v12741: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 511 B/s wr, 7 op/s 2023-03-09T21:20:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:19 smithi038 ceph-mon[92422]: pgmap v12741: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 511 B/s wr, 7 op/s 2023-03-09T21:20:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22931.311348 secs 2023-03-09T21:20:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22931.311348 secs 2023-03-09T21:20:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22931.311348 secs 2023-03-09T21:20:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:21 smithi038 ceph-mon[92422]: pgmap v12742: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 511 B/s wr, 7 op/s 2023-03-09T21:20:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:21 smithi122 ceph-mon[93869]: pgmap v12742: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 511 B/s wr, 7 op/s 2023-03-09T21:20:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:21 smithi181 ceph-mon[94189]: pgmap v12742: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 511 B/s wr, 7 op/s 2023-03-09T21:20:23.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22933.647419 secs 2023-03-09T21:20:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22933.647419 secs 2023-03-09T21:20:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22933.647419 secs 2023-03-09T21:20:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23395.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:20:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:24 smithi122 ceph-mon[93869]: pgmap v12743: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T21:20:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:24 smithi181 ceph-mon[94189]: pgmap v12743: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T21:20:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:24 smithi038 ceph-mon[92422]: pgmap v12743: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T21:20:26.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22936.311418 secs 2023-03-09T21:20:26.254 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:25 smithi038 ceph-mon[92422]: pgmap v12744: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T21:20:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22936.311418 secs 2023-03-09T21:20:26.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:25 smithi122 ceph-mon[93869]: pgmap v12744: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T21:20:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22936.311418 secs 2023-03-09T21:20:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:25 smithi181 ceph-mon[94189]: pgmap v12744: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T21:20:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22938.647541 secs 2023-03-09T21:20:27.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:27 smithi122 ceph-mon[93869]: pgmap v12745: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T21:20:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22938.647541 secs 2023-03-09T21:20:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:27 smithi181 ceph-mon[94189]: pgmap v12745: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T21:20:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22938.647541 secs 2023-03-09T21:20:28.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:27 smithi038 ceph-mon[92422]: pgmap v12745: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T21:20:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23400.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:20:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:29 smithi122 ceph-mon[93869]: pgmap v12746: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 6 op/s 2023-03-09T21:20:29.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:29 smithi181 ceph-mon[94189]: pgmap v12746: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 6 op/s 2023-03-09T21:20:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:29 smithi038 ceph-mon[92422]: pgmap v12746: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 6 op/s 2023-03-09T21:20:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22941.311480 secs 2023-03-09T21:20:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22941.311480 secs 2023-03-09T21:20:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22941.311480 secs 2023-03-09T21:20:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:31 smithi038 ceph-mon[92422]: pgmap v12747: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2023-03-09T21:20:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:31 smithi122 ceph-mon[93869]: pgmap v12747: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2023-03-09T21:20:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:31 smithi181 ceph-mon[94189]: pgmap v12747: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2023-03-09T21:20:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22943.647667 secs 2023-03-09T21:20:33.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22943.647667 secs 2023-03-09T21:20:33.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22943.647667 secs 2023-03-09T21:20:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23405.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:20:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:33 smithi038 ceph-mon[92422]: pgmap v12748: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 7 op/s 2023-03-09T21:20:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:33 smithi122 ceph-mon[93869]: pgmap v12748: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 7 op/s 2023-03-09T21:20:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:33 smithi181 ceph-mon[94189]: pgmap v12748: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 7 op/s 2023-03-09T21:20:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22946.311541 secs 2023-03-09T21:20:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:35 smithi038 ceph-mon[92422]: pgmap v12749: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 91 KiB/s wr, 7 op/s 2023-03-09T21:20:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22946.311541 secs 2023-03-09T21:20:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:35 smithi122 ceph-mon[93869]: pgmap v12749: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 91 KiB/s wr, 7 op/s 2023-03-09T21:20:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22946.311541 secs 2023-03-09T21:20:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:35 smithi181 ceph-mon[94189]: pgmap v12749: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 91 KiB/s wr, 7 op/s 2023-03-09T21:20:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23410.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:20:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:38 smithi122 ceph-mon[93869]: pgmap v12750: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 KiB/s wr, 9 op/s 2023-03-09T21:20:38.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:38 smithi181 ceph-mon[94189]: pgmap v12750: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 KiB/s wr, 9 op/s 2023-03-09T21:20:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:38 smithi038 ceph-mon[92422]: pgmap v12750: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 KiB/s wr, 9 op/s 2023-03-09T21:20:39.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:39 smithi122 ceph-mon[93869]: pgmap v12751: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 191 KiB/s wr, 8 op/s 2023-03-09T21:20:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:39 smithi181 ceph-mon[94189]: pgmap v12751: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 191 KiB/s wr, 8 op/s 2023-03-09T21:20:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:39 smithi038 ceph-mon[92422]: pgmap v12751: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 191 KiB/s wr, 8 op/s 2023-03-09T21:20:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22951.311601 secs 2023-03-09T21:20:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:40 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:20:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22951.311601 secs 2023-03-09T21:20:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:40 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:20:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22951.311601 secs 2023-03-09T21:20:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:40 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:20:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:41 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:20:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:20:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:20:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:41 smithi038 ceph-mon[92422]: pgmap v12752: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 191 KiB/s wr, 8 op/s 2023-03-09T21:20:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:41 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:20:42.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:20:42.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:20:42.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:41 smithi122 ceph-mon[93869]: pgmap v12752: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 191 KiB/s wr, 8 op/s 2023-03-09T21:20:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:41 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:20:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:20:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:20:42.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:41 smithi181 ceph-mon[94189]: pgmap v12752: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 191 KiB/s wr, 8 op/s 2023-03-09T21:20:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23415.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:20:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:43 smithi122 ceph-mon[93869]: pgmap v12753: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 192 KiB/s wr, 8 op/s 2023-03-09T21:20:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:43 smithi181 ceph-mon[94189]: pgmap v12753: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 192 KiB/s wr, 8 op/s 2023-03-09T21:20:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:43 smithi038 ceph-mon[92422]: pgmap v12753: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 192 KiB/s wr, 8 op/s 2023-03-09T21:20:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22956.311665 secs 2023-03-09T21:20:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:45 smithi038 ceph-mon[92422]: pgmap v12754: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 465 KiB/s wr, 8 op/s 2023-03-09T21:20:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22956.311665 secs 2023-03-09T21:20:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:45 smithi122 ceph-mon[93869]: pgmap v12754: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 465 KiB/s wr, 8 op/s 2023-03-09T21:20:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22956.311665 secs 2023-03-09T21:20:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:45 smithi181 ceph-mon[94189]: pgmap v12754: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 465 KiB/s wr, 8 op/s 2023-03-09T21:20:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:47 smithi122 ceph-mon[93869]: pgmap v12755: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 392 KiB/s wr, 9 op/s 2023-03-09T21:20:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:47 smithi181 ceph-mon[94189]: pgmap v12755: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 392 KiB/s wr, 9 op/s 2023-03-09T21:20:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:47 smithi038 ceph-mon[92422]: pgmap v12755: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 392 KiB/s wr, 9 op/s 2023-03-09T21:20:48.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23420.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:20:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:20:49.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:20:49.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:49 smithi122 ceph-mon[93869]: pgmap v12756: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 515 KiB/s rd, 392 KiB/s wr, 9 op/s 2023-03-09T21:20:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:20:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:20:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:49 smithi181 ceph-mon[94189]: pgmap v12756: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 515 KiB/s rd, 392 KiB/s wr, 9 op/s 2023-03-09T21:20:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:20:50.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:20:50.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:49 smithi038 ceph-mon[92422]: pgmap v12756: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 515 KiB/s rd, 392 KiB/s wr, 9 op/s 2023-03-09T21:20:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22961.311742 secs 2023-03-09T21:20:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22961.311742 secs 2023-03-09T21:20:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22961.311742 secs 2023-03-09T21:20:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:20:52.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:20:52.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:52 smithi122 ceph-mon[93869]: pgmap v12757: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 467 KiB/s rd, 304 KiB/s wr, 11 op/s 2023-03-09T21:20:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:20:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:20:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:52 smithi181 ceph-mon[94189]: pgmap v12757: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 467 KiB/s rd, 304 KiB/s wr, 11 op/s 2023-03-09T21:20:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:20:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:20:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:52 smithi038 ceph-mon[92422]: pgmap v12757: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 467 KiB/s rd, 304 KiB/s wr, 11 op/s 2023-03-09T21:20:53.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23425.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:20:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:53 smithi122 ceph-mon[93869]: pgmap v12758: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 464 KiB/s wr, 11 op/s 2023-03-09T21:20:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:53 smithi181 ceph-mon[94189]: pgmap v12758: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 464 KiB/s wr, 11 op/s 2023-03-09T21:20:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:53 smithi038 ceph-mon[92422]: pgmap v12758: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 464 KiB/s wr, 11 op/s 2023-03-09T21:20:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:20:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:20:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:55 smithi122 ceph-mon[93869]: pgmap v12759: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 637 KiB/s rd, 464 KiB/s wr, 11 op/s 2023-03-09T21:20:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:20:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:20:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:55 smithi181 ceph-mon[94189]: pgmap v12759: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 637 KiB/s rd, 464 KiB/s wr, 11 op/s 2023-03-09T21:20:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:20:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:20:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:55 smithi038 ceph-mon[92422]: pgmap v12759: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 637 KiB/s rd, 464 KiB/s wr, 11 op/s 2023-03-09T21:20:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22966.311801 secs 2023-03-09T21:20:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:20:56.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:20:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22966.311801 secs 2023-03-09T21:20:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:20:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:20:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22966.311801 secs 2023-03-09T21:20:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:20:57.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:20:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:57 smithi122 ceph-mon[93869]: pgmap v12760: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 802 KiB/s rd, 454 KiB/s wr, 14 op/s 2023-03-09T21:20:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:57 smithi181 ceph-mon[94189]: pgmap v12760: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 802 KiB/s rd, 454 KiB/s wr, 14 op/s 2023-03-09T21:20:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:57 smithi038 ceph-mon[92422]: pgmap v12760: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 802 KiB/s rd, 454 KiB/s wr, 14 op/s 2023-03-09T21:20:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23430.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:20:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:20:59.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:20:59.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:20:59 smithi122 ceph-mon[93869]: pgmap v12761: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 782 KiB/s rd, 436 KiB/s wr, 13 op/s 2023-03-09T21:20:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:20:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:20:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:20:59 smithi181 ceph-mon[94189]: pgmap v12761: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 782 KiB/s rd, 436 KiB/s wr, 13 op/s 2023-03-09T21:20:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:21:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:21:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:20:59 smithi038 ceph-mon[92422]: pgmap v12761: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 782 KiB/s rd, 436 KiB/s wr, 13 op/s 2023-03-09T21:21:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:21:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:21:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22971.311861 secs 2023-03-09T21:21:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:21:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:21:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22971.311861 secs 2023-03-09T21:21:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:21:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:21:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22971.311861 secs 2023-03-09T21:21:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:01 smithi122 ceph-mon[93869]: pgmap v12762: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 535 KiB/s rd, 520 KiB/s wr, 14 op/s 2023-03-09T21:21:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:01 smithi181 ceph-mon[94189]: pgmap v12762: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 535 KiB/s rd, 520 KiB/s wr, 14 op/s 2023-03-09T21:21:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:01 smithi038 ceph-mon[92422]: pgmap v12762: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 535 KiB/s rd, 520 KiB/s wr, 14 op/s 2023-03-09T21:21:02.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22973.648292 secs 2023-03-09T21:21:02.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22973.648292 secs 2023-03-09T21:21:02.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22973.648292 secs 2023-03-09T21:21:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23435.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:21:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:03 smithi122 ceph-mon[93869]: pgmap v12763: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 529 KiB/s rd, 530 KiB/s wr, 25 op/s 2023-03-09T21:21:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:03 smithi181 ceph-mon[94189]: pgmap v12763: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 529 KiB/s rd, 530 KiB/s wr, 25 op/s 2023-03-09T21:21:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:03 smithi038 ceph-mon[92422]: pgmap v12763: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 529 KiB/s rd, 530 KiB/s wr, 25 op/s 2023-03-09T21:21:04.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:21:04.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:21:04.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:21:04.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:21:04.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:21:04.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:21:04.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:21:04.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:21:04.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:21:04.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:21:04.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:21:05.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:21:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:05 smithi038 ceph-mon[92422]: osdmap e1011: 12 total, 12 up, 12 in 2023-03-09T21:21:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22976.311925 secs 2023-03-09T21:21:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:05 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:21:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:05 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:21:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:05 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:21:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:05 smithi038 ceph-mon[92422]: pgmap v12765: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 444 KiB/s wr, 27 op/s 2023-03-09T21:21:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:05 smithi122 ceph-mon[93869]: osdmap e1011: 12 total, 12 up, 12 in 2023-03-09T21:21:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22976.311925 secs 2023-03-09T21:21:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:05 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:21:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:05 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:21:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:05 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:21:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:05 smithi122 ceph-mon[93869]: pgmap v12765: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 444 KiB/s wr, 27 op/s 2023-03-09T21:21:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:05 smithi181 ceph-mon[94189]: osdmap e1011: 12 total, 12 up, 12 in 2023-03-09T21:21:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22976.311925 secs 2023-03-09T21:21:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:05 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:21:06.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:05 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:21:06.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:05 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:21:06.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:05 smithi181 ceph-mon[94189]: pgmap v12765: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 444 KiB/s wr, 27 op/s 2023-03-09T21:21:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22978.648407 secs 2023-03-09T21:21:07.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:07 smithi122 ceph-mon[93869]: pgmap v12766: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 405 KiB/s wr, 29 op/s 2023-03-09T21:21:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22978.648407 secs 2023-03-09T21:21:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:07 smithi181 ceph-mon[94189]: pgmap v12766: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 405 KiB/s wr, 29 op/s 2023-03-09T21:21:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22978.648407 secs 2023-03-09T21:21:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:07 smithi038 ceph-mon[92422]: pgmap v12766: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 405 KiB/s wr, 29 op/s 2023-03-09T21:21:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23440.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:21:09.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:09 smithi122 ceph-mon[93869]: pgmap v12767: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 405 KiB/s wr, 28 op/s 2023-03-09T21:21:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:09 smithi181 ceph-mon[94189]: pgmap v12767: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 405 KiB/s wr, 28 op/s 2023-03-09T21:21:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:09 smithi038 ceph-mon[92422]: pgmap v12767: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 405 KiB/s wr, 28 op/s 2023-03-09T21:21:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22981.311983 secs 2023-03-09T21:21:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:10 smithi038 ceph-mon[92422]: osdmap e1012: 12 total, 12 up, 12 in 2023-03-09T21:21:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22981.311983 secs 2023-03-09T21:21:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:10 smithi122 ceph-mon[93869]: osdmap e1012: 12 total, 12 up, 12 in 2023-03-09T21:21:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22981.311983 secs 2023-03-09T21:21:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:10 smithi181 ceph-mon[94189]: osdmap e1012: 12 total, 12 up, 12 in 2023-03-09T21:21:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:12 smithi122 ceph-mon[93869]: pgmap v12769: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 362 KiB/s rd, 392 KiB/s wr, 15 op/s 2023-03-09T21:21:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:12 smithi181 ceph-mon[94189]: pgmap v12769: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 362 KiB/s rd, 392 KiB/s wr, 15 op/s 2023-03-09T21:21:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:12 smithi038 ceph-mon[92422]: pgmap v12769: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 362 KiB/s rd, 392 KiB/s wr, 15 op/s 2023-03-09T21:21:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22983.648529 secs 2023-03-09T21:21:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22983.648529 secs 2023-03-09T21:21:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22983.648529 secs 2023-03-09T21:21:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23445.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:21:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:14 smithi122 ceph-mon[93869]: pgmap v12770: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 400 KiB/s wr, 14 op/s 2023-03-09T21:21:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:14 smithi181 ceph-mon[94189]: pgmap v12770: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 400 KiB/s wr, 14 op/s 2023-03-09T21:21:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:14 smithi038 ceph-mon[92422]: pgmap v12770: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 400 KiB/s wr, 14 op/s 2023-03-09T21:21:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22986.312044 secs 2023-03-09T21:21:16.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:16 smithi122 ceph-mon[93869]: pgmap v12771: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 329 KiB/s rd, 353 KiB/s wr, 12 op/s 2023-03-09T21:21:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22986.312044 secs 2023-03-09T21:21:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:16 smithi181 ceph-mon[94189]: pgmap v12771: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 329 KiB/s rd, 353 KiB/s wr, 12 op/s 2023-03-09T21:21:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22986.312044 secs 2023-03-09T21:21:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:16 smithi038 ceph-mon[92422]: pgmap v12771: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 329 KiB/s rd, 353 KiB/s wr, 12 op/s 2023-03-09T21:21:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22988.648659 secs 2023-03-09T21:21:17.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:17 smithi122 ceph-mon[93869]: pgmap v12772: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 75 KiB/s wr, 8 op/s 2023-03-09T21:21:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22988.648659 secs 2023-03-09T21:21:17.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:17 smithi181 ceph-mon[94189]: pgmap v12772: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 75 KiB/s wr, 8 op/s 2023-03-09T21:21:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22988.648659 secs 2023-03-09T21:21:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:17 smithi038 ceph-mon[92422]: pgmap v12772: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 75 KiB/s wr, 8 op/s 2023-03-09T21:21:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23450.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:21:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:19 smithi122 ceph-mon[93869]: pgmap v12773: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 75 KiB/s wr, 8 op/s 2023-03-09T21:21:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:19 smithi181 ceph-mon[94189]: pgmap v12773: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 75 KiB/s wr, 8 op/s 2023-03-09T21:21:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:19 smithi038 ceph-mon[92422]: pgmap v12773: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 75 KiB/s wr, 8 op/s 2023-03-09T21:21:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22991.312105 secs 2023-03-09T21:21:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22991.312105 secs 2023-03-09T21:21:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22991.312105 secs 2023-03-09T21:21:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:21 smithi038 ceph-mon[92422]: pgmap v12774: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 65 KiB/s wr, 9 op/s 2023-03-09T21:21:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:21 smithi122 ceph-mon[93869]: pgmap v12774: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 65 KiB/s wr, 9 op/s 2023-03-09T21:21:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:21 smithi181 ceph-mon[94189]: pgmap v12774: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 65 KiB/s wr, 9 op/s 2023-03-09T21:21:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22993.648774 secs 2023-03-09T21:21:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22993.648774 secs 2023-03-09T21:21:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22993.648774 secs 2023-03-09T21:21:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23455.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:21:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:24 smithi122 ceph-mon[93869]: pgmap v12775: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 33 KiB/s wr, 6 op/s 2023-03-09T21:21:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:24 smithi181 ceph-mon[94189]: pgmap v12775: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 33 KiB/s wr, 6 op/s 2023-03-09T21:21:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:24 smithi038 ceph-mon[92422]: pgmap v12775: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 33 KiB/s wr, 6 op/s 2023-03-09T21:21:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22996.312168 secs 2023-03-09T21:21:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:26 smithi122 ceph-mon[93869]: pgmap v12776: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T21:21:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22996.312168 secs 2023-03-09T21:21:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:26 smithi181 ceph-mon[94189]: pgmap v12776: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T21:21:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22996.312168 secs 2023-03-09T21:21:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:26 smithi038 ceph-mon[92422]: pgmap v12776: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T21:21:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 22998.648884 secs 2023-03-09T21:21:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:27 smithi122 ceph-mon[93869]: pgmap v12777: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 8 op/s 2023-03-09T21:21:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 22998.648884 secs 2023-03-09T21:21:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:27 smithi181 ceph-mon[94189]: pgmap v12777: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 8 op/s 2023-03-09T21:21:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 22998.648884 secs 2023-03-09T21:21:28.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:27 smithi038 ceph-mon[92422]: pgmap v12777: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 8 op/s 2023-03-09T21:21:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23460.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:21:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:29 smithi122 ceph-mon[93869]: pgmap v12778: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T21:21:29.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:29 smithi181 ceph-mon[94189]: pgmap v12778: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T21:21:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:29 smithi038 ceph-mon[92422]: pgmap v12778: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T21:21:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23001.312230 secs 2023-03-09T21:21:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23001.312230 secs 2023-03-09T21:21:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23001.312230 secs 2023-03-09T21:21:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:32 smithi122 ceph-mon[93869]: pgmap v12779: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 426 B/s wr, 8 op/s 2023-03-09T21:21:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:32 smithi181 ceph-mon[94189]: pgmap v12779: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 426 B/s wr, 8 op/s 2023-03-09T21:21:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:32 smithi038 ceph-mon[92422]: pgmap v12779: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 426 B/s wr, 8 op/s 2023-03-09T21:21:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:33 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:21:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:33 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:21:33.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23003.648983 secs 2023-03-09T21:21:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:33 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:21:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:33 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:21:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23003.648983 secs 2023-03-09T21:21:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:33 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:21:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:33 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:21:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23003.648983 secs 2023-03-09T21:21:33.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23465.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:21:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:34 smithi122 ceph-mon[93869]: pgmap v12780: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 75 KiB/s wr, 7 op/s 2023-03-09T21:21:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:34 smithi181 ceph-mon[94189]: pgmap v12780: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 75 KiB/s wr, 7 op/s 2023-03-09T21:21:34.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:34 smithi038 ceph-mon[92422]: pgmap v12780: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 75 KiB/s wr, 7 op/s 2023-03-09T21:21:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23006.312290 secs 2023-03-09T21:21:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:21:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:21:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:36 smithi122 ceph-mon[93869]: pgmap v12781: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 75 KiB/s wr, 6 op/s 2023-03-09T21:21:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23006.312290 secs 2023-03-09T21:21:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:21:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:21:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:36 smithi181 ceph-mon[94189]: pgmap v12781: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 75 KiB/s wr, 6 op/s 2023-03-09T21:21:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23006.312290 secs 2023-03-09T21:21:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:21:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:21:36.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:36 smithi038 ceph-mon[92422]: pgmap v12781: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 75 KiB/s wr, 6 op/s 2023-03-09T21:21:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:37 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 23008.649133 secs 2023-03-09T21:21:37.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:37 smithi122 ceph-mon[93869]: pgmap v12782: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 166 KiB/s wr, 9 op/s 2023-03-09T21:21:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:37 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 23008.649133 secs 2023-03-09T21:21:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:37 smithi181 ceph-mon[94189]: pgmap v12782: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 166 KiB/s wr, 9 op/s 2023-03-09T21:21:38.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:37 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 23008.649133 secs 2023-03-09T21:21:38.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:37 smithi038 ceph-mon[92422]: pgmap v12782: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 166 KiB/s wr, 9 op/s 2023-03-09T21:21:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23470.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:21:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:39 smithi122 ceph-mon[93869]: pgmap v12783: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 166 KiB/s wr, 6 op/s 2023-03-09T21:21:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:39 smithi181 ceph-mon[94189]: pgmap v12783: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 166 KiB/s wr, 6 op/s 2023-03-09T21:21:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:39 smithi038 ceph-mon[92422]: pgmap v12783: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 166 KiB/s wr, 6 op/s 2023-03-09T21:21:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23011.312352 secs 2023-03-09T21:21:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23011.312352 secs 2023-03-09T21:21:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23011.312352 secs 2023-03-09T21:21:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:42 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:21:42.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:42 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:21:42.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:42 smithi122 ceph-mon[93869]: pgmap v12784: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 167 KiB/s wr, 8 op/s 2023-03-09T21:21:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:42 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:21:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:42 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:21:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:42 smithi181 ceph-mon[94189]: pgmap v12784: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 167 KiB/s wr, 8 op/s 2023-03-09T21:21:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:42 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:21:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:42 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:21:42.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:42 smithi038 ceph-mon[92422]: pgmap v12784: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 167 KiB/s wr, 8 op/s 2023-03-09T21:21:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:43 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 23013.649227 secs 2023-03-09T21:21:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:43 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 23013.649227 secs 2023-03-09T21:21:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:43 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 23013.649227 secs 2023-03-09T21:21:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23475.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:21:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:44 smithi122 ceph-mon[93869]: pgmap v12785: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 309 KiB/s wr, 7 op/s 2023-03-09T21:21:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:44 smithi181 ceph-mon[94189]: pgmap v12785: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 309 KiB/s wr, 7 op/s 2023-03-09T21:21:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:44 smithi038 ceph-mon[92422]: pgmap v12785: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 309 KiB/s wr, 7 op/s 2023-03-09T21:21:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23016.312413 secs 2023-03-09T21:21:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:46 smithi122 ceph-mon[93869]: pgmap v12786: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 234 KiB/s wr, 6 op/s 2023-03-09T21:21:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23016.312413 secs 2023-03-09T21:21:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:46 smithi181 ceph-mon[94189]: pgmap v12786: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 234 KiB/s wr, 6 op/s 2023-03-09T21:21:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23016.312413 secs 2023-03-09T21:21:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:46 smithi038 ceph-mon[92422]: pgmap v12786: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 234 KiB/s wr, 6 op/s 2023-03-09T21:21:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:21:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:21:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23018.649350 secs 2023-03-09T21:21:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:48 smithi122 ceph-mon[93869]: pgmap v12787: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 673 KiB/s rd, 529 KiB/s wr, 11 op/s 2023-03-09T21:21:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:21:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:21:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23018.649350 secs 2023-03-09T21:21:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:48 smithi181 ceph-mon[94189]: pgmap v12787: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 673 KiB/s rd, 529 KiB/s wr, 11 op/s 2023-03-09T21:21:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:21:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:21:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23018.649350 secs 2023-03-09T21:21:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:48 smithi038 ceph-mon[92422]: pgmap v12787: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 673 KiB/s rd, 529 KiB/s wr, 11 op/s 2023-03-09T21:21:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23480.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:21:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:21:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:21:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:21:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:21:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:21:49.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:21:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:50 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:21:50.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:50 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:21:50.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:50 smithi122 ceph-mon[93869]: pgmap v12788: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 581 KiB/s rd, 438 KiB/s wr, 9 op/s 2023-03-09T21:21:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:50 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:21:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:50 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:21:50.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:50 smithi181 ceph-mon[94189]: pgmap v12788: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 581 KiB/s rd, 438 KiB/s wr, 9 op/s 2023-03-09T21:21:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:50 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:21:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:50 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:21:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:50 smithi038 ceph-mon[92422]: pgmap v12788: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 581 KiB/s rd, 438 KiB/s wr, 9 op/s 2023-03-09T21:21:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23021.312475 secs 2023-03-09T21:21:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23021.312475 secs 2023-03-09T21:21:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23021.312475 secs 2023-03-09T21:21:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:52 smithi122 ceph-mon[93869]: pgmap v12789: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 753 KiB/s rd, 525 KiB/s wr, 12 op/s 2023-03-09T21:21:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:52 smithi181 ceph-mon[94189]: pgmap v12789: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 753 KiB/s rd, 525 KiB/s wr, 12 op/s 2023-03-09T21:21:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:52 smithi038 ceph-mon[92422]: pgmap v12789: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 753 KiB/s rd, 525 KiB/s wr, 12 op/s 2023-03-09T21:21:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23023.649451 secs 2023-03-09T21:21:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23023.649451 secs 2023-03-09T21:21:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23023.649451 secs 2023-03-09T21:21:53.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23485.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:21:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:54 smithi122 ceph-mon[93869]: pgmap v12790: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 884 KiB/s rd, 866 KiB/s wr, 11 op/s 2023-03-09T21:21:54.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:54 smithi181 ceph-mon[94189]: pgmap v12790: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 884 KiB/s rd, 866 KiB/s wr, 11 op/s 2023-03-09T21:21:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:54 smithi038 ceph-mon[92422]: pgmap v12790: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 884 KiB/s rd, 866 KiB/s wr, 11 op/s 2023-03-09T21:21:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23026.312556 secs 2023-03-09T21:21:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:21:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:21:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:56 smithi122 ceph-mon[93869]: pgmap v12791: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 724 KiB/s wr, 10 op/s 2023-03-09T21:21:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23026.312556 secs 2023-03-09T21:21:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:21:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:21:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:56 smithi181 ceph-mon[94189]: pgmap v12791: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 724 KiB/s wr, 10 op/s 2023-03-09T21:21:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23026.312556 secs 2023-03-09T21:21:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:21:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:21:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:56 smithi038 ceph-mon[92422]: pgmap v12791: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 724 KiB/s wr, 10 op/s 2023-03-09T21:21:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:21:57.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:21:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:21:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:21:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:21:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:21:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23028.649579 secs 2023-03-09T21:21:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:21:58 smithi122 ceph-mon[93869]: pgmap v12792: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 808 KiB/s rd, 844 KiB/s wr, 14 op/s 2023-03-09T21:21:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23028.649579 secs 2023-03-09T21:21:58.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:21:58 smithi181 ceph-mon[94189]: pgmap v12792: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 808 KiB/s rd, 844 KiB/s wr, 14 op/s 2023-03-09T21:21:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23028.649579 secs 2023-03-09T21:21:58.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:21:58 smithi038 ceph-mon[92422]: pgmap v12792: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 808 KiB/s rd, 844 KiB/s wr, 14 op/s 2023-03-09T21:21:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23490.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:22:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:22:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:22:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:00 smithi122 ceph-mon[93869]: pgmap v12793: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 639 KiB/s rd, 550 KiB/s wr, 11 op/s 2023-03-09T21:22:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:22:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:22:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:00 smithi181 ceph-mon[94189]: pgmap v12793: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 639 KiB/s rd, 550 KiB/s wr, 11 op/s 2023-03-09T21:22:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:22:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:22:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:00 smithi038 ceph-mon[92422]: pgmap v12793: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 639 KiB/s rd, 550 KiB/s wr, 11 op/s 2023-03-09T21:22:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23031.312617 secs 2023-03-09T21:22:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:22:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:22:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23031.312617 secs 2023-03-09T21:22:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:22:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:22:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23031.312617 secs 2023-03-09T21:22:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:22:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:22:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:02 smithi122 ceph-mon[93869]: pgmap v12794: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 807 KiB/s rd, 717 KiB/s wr, 14 op/s 2023-03-09T21:22:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:02 smithi181 ceph-mon[94189]: pgmap v12794: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 807 KiB/s rd, 717 KiB/s wr, 14 op/s 2023-03-09T21:22:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:02 smithi038 ceph-mon[92422]: pgmap v12794: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 807 KiB/s rd, 717 KiB/s wr, 14 op/s 2023-03-09T21:22:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23033.649694 secs 2023-03-09T21:22:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23033.649694 secs 2023-03-09T21:22:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23033.649694 secs 2023-03-09T21:22:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23495.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:22:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:04 smithi122 ceph-mon[93869]: pgmap v12795: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 642 KiB/s rd, 652 KiB/s wr, 23 op/s 2023-03-09T21:22:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:04 smithi181 ceph-mon[94189]: pgmap v12795: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 642 KiB/s rd, 652 KiB/s wr, 23 op/s 2023-03-09T21:22:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:04 smithi038 ceph-mon[92422]: pgmap v12795: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 642 KiB/s rd, 652 KiB/s wr, 23 op/s 2023-03-09T21:22:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:05 smithi122 ceph-mon[93869]: osdmap e1013: 12 total, 12 up, 12 in 2023-03-09T21:22:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:05 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:22:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:05 smithi181 ceph-mon[94189]: osdmap e1013: 12 total, 12 up, 12 in 2023-03-09T21:22:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:05 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:22:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:05 smithi038 ceph-mon[92422]: osdmap e1013: 12 total, 12 up, 12 in 2023-03-09T21:22:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:05 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:22:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23036.312677 secs 2023-03-09T21:22:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:06 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:22:06.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:06 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:22:06.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:06 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:22:06.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:06 smithi122 ceph-mon[93869]: pgmap v12797: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 612 KiB/s rd, 371 KiB/s wr, 26 op/s 2023-03-09T21:22:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23036.312677 secs 2023-03-09T21:22:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:06 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:22:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:06 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:22:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:06 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:22:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:06 smithi181 ceph-mon[94189]: pgmap v12797: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 612 KiB/s rd, 371 KiB/s wr, 26 op/s 2023-03-09T21:22:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23036.312677 secs 2023-03-09T21:22:06.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:06 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:22:06.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:06 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:22:06.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:06 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:22:06.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:06 smithi038 ceph-mon[92422]: pgmap v12797: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 612 KiB/s rd, 371 KiB/s wr, 26 op/s 2023-03-09T21:22:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23038.649807 secs 2023-03-09T21:22:07.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:07 smithi122 ceph-mon[93869]: pgmap v12798: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 675 KiB/s rd, 460 KiB/s wr, 29 op/s 2023-03-09T21:22:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23038.649807 secs 2023-03-09T21:22:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:07 smithi181 ceph-mon[94189]: pgmap v12798: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 675 KiB/s rd, 460 KiB/s wr, 29 op/s 2023-03-09T21:22:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23038.649807 secs 2023-03-09T21:22:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:07 smithi038 ceph-mon[92422]: pgmap v12798: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 675 KiB/s rd, 460 KiB/s wr, 29 op/s 2023-03-09T21:22:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23500.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:22:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:09 smithi122 ceph-mon[93869]: pgmap v12799: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 460 KiB/s wr, 28 op/s 2023-03-09T21:22:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:09 smithi181 ceph-mon[94189]: pgmap v12799: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 460 KiB/s wr, 28 op/s 2023-03-09T21:22:10.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:09 smithi038 ceph-mon[92422]: pgmap v12799: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 460 KiB/s wr, 28 op/s 2023-03-09T21:22:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:11 smithi122 ceph-mon[93869]: 2 slow requests, 2 included below; oldest blocked for > 23041.312739 secs 2023-03-09T21:22:11.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:11 smithi122 ceph-mon[93869]: slow request 23041.312738 seconds old, received at 2023-03-09T14:58:08.564667+0000: peer_request:mds.2:599 currently acquired locks 2023-03-09T21:22:11.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:11 smithi122 ceph-mon[93869]: slow request 23041.302038 seconds old, received at 2023-03-09T14:58:08.575367+0000: peer_request(client.15375:6773.0 authpin) currently dispatched 2023-03-09T21:22:11.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:11 smithi122 ceph-mon[93869]: osdmap e1014: 12 total, 12 up, 12 in 2023-03-09T21:22:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:11 smithi181 ceph-mon[94189]: 2 slow requests, 2 included below; oldest blocked for > 23041.312739 secs 2023-03-09T21:22:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:11 smithi181 ceph-mon[94189]: slow request 23041.312738 seconds old, received at 2023-03-09T14:58:08.564667+0000: peer_request:mds.2:599 currently acquired locks 2023-03-09T21:22:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:11 smithi181 ceph-mon[94189]: slow request 23041.302038 seconds old, received at 2023-03-09T14:58:08.575367+0000: peer_request(client.15375:6773.0 authpin) currently dispatched 2023-03-09T21:22:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:11 smithi181 ceph-mon[94189]: osdmap e1014: 12 total, 12 up, 12 in 2023-03-09T21:22:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:11 smithi038 ceph-mon[92422]: 2 slow requests, 2 included below; oldest blocked for > 23041.312739 secs 2023-03-09T21:22:11.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:11 smithi038 ceph-mon[92422]: slow request 23041.312738 seconds old, received at 2023-03-09T14:58:08.564667+0000: peer_request:mds.2:599 currently acquired locks 2023-03-09T21:22:11.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:11 smithi038 ceph-mon[92422]: slow request 23041.302038 seconds old, received at 2023-03-09T14:58:08.575367+0000: peer_request(client.15375:6773.0 authpin) currently dispatched 2023-03-09T21:22:11.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:11 smithi038 ceph-mon[92422]: osdmap e1014: 12 total, 12 up, 12 in 2023-03-09T21:22:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:22:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:22:12.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:12 smithi122 ceph-mon[93869]: pgmap v12801: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 395 KiB/s rd, 425 KiB/s wr, 16 op/s 2023-03-09T21:22:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:22:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:22:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:12 smithi181 ceph-mon[94189]: pgmap v12801: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 395 KiB/s rd, 425 KiB/s wr, 16 op/s 2023-03-09T21:22:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:22:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:22:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:12 smithi038 ceph-mon[92422]: pgmap v12801: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 395 KiB/s rd, 425 KiB/s wr, 16 op/s 2023-03-09T21:22:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:22:13.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:22:13.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:13 smithi122 ceph-mon[93869]: 2 slow requests, 2 included below; oldest blocked for > 23043.649927 secs 2023-03-09T21:22:13.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:13 smithi122 ceph-mon[93869]: slow request 23043.649926 seconds old, received at 2023-03-09T14:58:08.560137+0000: client_request(mds.2:599 rename #0x10000000007/0000000100000001000000EB #0x60f/1000000062a caller_uid=0, caller_gid=0{}) currently acquired locks 2023-03-09T21:22:13.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:13 smithi122 ceph-mon[93869]: slow request 23043.635546 seconds old, received at 2023-03-09T14:58:08.574517+0000: client_request(client.15375:6773 unlink #0x10000000007/0000000100000001000000EB 2023-03-09T14:58:08.569481+0000 caller_uid=26, caller_gid=26{26,}) currently requesting remote authpins 2023-03-09T21:22:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:22:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:22:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:13 smithi181 ceph-mon[94189]: 2 slow requests, 2 included below; oldest blocked for > 23043.649927 secs 2023-03-09T21:22:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:13 smithi181 ceph-mon[94189]: slow request 23043.649926 seconds old, received at 2023-03-09T14:58:08.560137+0000: client_request(mds.2:599 rename #0x10000000007/0000000100000001000000EB #0x60f/1000000062a caller_uid=0, caller_gid=0{}) currently acquired locks 2023-03-09T21:22:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:13 smithi181 ceph-mon[94189]: slow request 23043.635546 seconds old, received at 2023-03-09T14:58:08.574517+0000: client_request(client.15375:6773 unlink #0x10000000007/0000000100000001000000EB 2023-03-09T14:58:08.569481+0000 caller_uid=26, caller_gid=26{26,}) currently requesting remote authpins 2023-03-09T21:22:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:22:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:22:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:13 smithi038 ceph-mon[92422]: 2 slow requests, 2 included below; oldest blocked for > 23043.649927 secs 2023-03-09T21:22:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:13 smithi038 ceph-mon[92422]: slow request 23043.649926 seconds old, received at 2023-03-09T14:58:08.560137+0000: client_request(mds.2:599 rename #0x10000000007/0000000100000001000000EB #0x60f/1000000062a caller_uid=0, caller_gid=0{}) currently acquired locks 2023-03-09T21:22:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:13 smithi038 ceph-mon[92422]: slow request 23043.635546 seconds old, received at 2023-03-09T14:58:08.574517+0000: client_request(client.15375:6773 unlink #0x10000000007/0000000100000001000000EB 2023-03-09T14:58:08.569481+0000 caller_uid=26, caller_gid=26{26,}) currently requesting remote authpins 2023-03-09T21:22:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23505.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:22:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:14 smithi122 ceph-mon[93869]: pgmap v12802: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 335 KiB/s rd, 360 KiB/s wr, 14 op/s 2023-03-09T21:22:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:14 smithi181 ceph-mon[94189]: pgmap v12802: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 335 KiB/s rd, 360 KiB/s wr, 14 op/s 2023-03-09T21:22:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:14 smithi038 ceph-mon[92422]: pgmap v12802: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 335 KiB/s rd, 360 KiB/s wr, 14 op/s 2023-03-09T21:22:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23046.312801 secs 2023-03-09T21:22:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:16 smithi122 ceph-mon[93869]: pgmap v12803: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 340 KiB/s wr, 13 op/s 2023-03-09T21:22:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23046.312801 secs 2023-03-09T21:22:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:16 smithi181 ceph-mon[94189]: pgmap v12803: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 340 KiB/s wr, 13 op/s 2023-03-09T21:22:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23046.312801 secs 2023-03-09T21:22:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:16 smithi038 ceph-mon[92422]: pgmap v12803: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 340 KiB/s wr, 13 op/s 2023-03-09T21:22:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23048.650039 secs 2023-03-09T21:22:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:17 smithi122 ceph-mon[93869]: pgmap v12804: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 210 KiB/s wr, 8 op/s 2023-03-09T21:22:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23048.650039 secs 2023-03-09T21:22:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:17 smithi181 ceph-mon[94189]: pgmap v12804: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 210 KiB/s wr, 8 op/s 2023-03-09T21:22:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23048.650039 secs 2023-03-09T21:22:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:17 smithi038 ceph-mon[92422]: pgmap v12804: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 210 KiB/s wr, 8 op/s 2023-03-09T21:22:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23510.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:22:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:19 smithi122 ceph-mon[93869]: pgmap v12805: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 210 KiB/s wr, 8 op/s 2023-03-09T21:22:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:19 smithi181 ceph-mon[94189]: pgmap v12805: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 210 KiB/s wr, 8 op/s 2023-03-09T21:22:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:19 smithi038 ceph-mon[92422]: pgmap v12805: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 210 KiB/s wr, 8 op/s 2023-03-09T21:22:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23051.312863 secs 2023-03-09T21:22:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23051.312863 secs 2023-03-09T21:22:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23051.312863 secs 2023-03-09T21:22:22.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:22 smithi122 ceph-mon[93869]: pgmap v12806: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 182 KiB/s wr, 10 op/s 2023-03-09T21:22:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:22 smithi181 ceph-mon[94189]: pgmap v12806: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 182 KiB/s wr, 10 op/s 2023-03-09T21:22:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:22 smithi038 ceph-mon[92422]: pgmap v12806: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 182 KiB/s wr, 10 op/s 2023-03-09T21:22:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23053.650158 secs 2023-03-09T21:22:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23053.650158 secs 2023-03-09T21:22:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23053.650158 secs 2023-03-09T21:22:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23515.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:22:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:24 smithi122 ceph-mon[93869]: pgmap v12807: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 86 KiB/s wr, 5 op/s 2023-03-09T21:22:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:24 smithi181 ceph-mon[94189]: pgmap v12807: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 86 KiB/s wr, 5 op/s 2023-03-09T21:22:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:24 smithi038 ceph-mon[92422]: pgmap v12807: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 86 KiB/s wr, 5 op/s 2023-03-09T21:22:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23056.312924 secs 2023-03-09T21:22:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:26 smithi122 ceph-mon[93869]: pgmap v12808: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 86 KiB/s wr, 5 op/s 2023-03-09T21:22:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23056.312924 secs 2023-03-09T21:22:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:26 smithi181 ceph-mon[94189]: pgmap v12808: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 86 KiB/s wr, 5 op/s 2023-03-09T21:22:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23056.312924 secs 2023-03-09T21:22:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:26 smithi038 ceph-mon[92422]: pgmap v12808: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 86 KiB/s wr, 5 op/s 2023-03-09T21:22:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23058.650254 secs 2023-03-09T21:22:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:27 smithi122 ceph-mon[93869]: pgmap v12809: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 86 KiB/s wr, 8 op/s 2023-03-09T21:22:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23058.650254 secs 2023-03-09T21:22:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:27 smithi181 ceph-mon[94189]: pgmap v12809: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 86 KiB/s wr, 8 op/s 2023-03-09T21:22:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23058.650254 secs 2023-03-09T21:22:28.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:27 smithi038 ceph-mon[92422]: pgmap v12809: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 86 KiB/s wr, 8 op/s 2023-03-09T21:22:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23520.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:22:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:29 smithi122 ceph-mon[93869]: pgmap v12810: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2023-03-09T21:22:29.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:29 smithi181 ceph-mon[94189]: pgmap v12810: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2023-03-09T21:22:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:29 smithi038 ceph-mon[92422]: pgmap v12810: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2023-03-09T21:22:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23061.312999 secs 2023-03-09T21:22:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23061.312999 secs 2023-03-09T21:22:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23061.312999 secs 2023-03-09T21:22:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:32 smithi122 ceph-mon[93869]: pgmap v12811: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 170 B/s wr, 8 op/s 2023-03-09T21:22:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:32 smithi181 ceph-mon[94189]: pgmap v12811: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 170 B/s wr, 8 op/s 2023-03-09T21:22:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:32 smithi038 ceph-mon[92422]: pgmap v12811: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 170 B/s wr, 8 op/s 2023-03-09T21:22:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23063.650371 secs 2023-03-09T21:22:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23063.650371 secs 2023-03-09T21:22:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23063.650371 secs 2023-03-09T21:22:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23525.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:22:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:34 smithi122 ceph-mon[93869]: pgmap v12812: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-09T21:22:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:34 smithi181 ceph-mon[94189]: pgmap v12812: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-09T21:22:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:34 smithi038 ceph-mon[92422]: pgmap v12812: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-09T21:22:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23066.313058 secs 2023-03-09T21:22:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:36 smithi122 ceph-mon[93869]: pgmap v12813: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-09T21:22:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23066.313058 secs 2023-03-09T21:22:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:36 smithi181 ceph-mon[94189]: pgmap v12813: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-09T21:22:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23066.313058 secs 2023-03-09T21:22:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:36 smithi038 ceph-mon[92422]: pgmap v12813: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-09T21:22:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23068.650486 secs 2023-03-09T21:22:37.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:37 smithi122 ceph-mon[93869]: pgmap v12814: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 511 B/s wr, 8 op/s 2023-03-09T21:22:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23068.650486 secs 2023-03-09T21:22:37.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:37 smithi181 ceph-mon[94189]: pgmap v12814: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 511 B/s wr, 8 op/s 2023-03-09T21:22:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23068.650486 secs 2023-03-09T21:22:38.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:37 smithi038 ceph-mon[92422]: pgmap v12814: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 511 B/s wr, 8 op/s 2023-03-09T21:22:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23530.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:22:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:39 smithi122 ceph-mon[93869]: pgmap v12815: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2023-03-09T21:22:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:39 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:22:39.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:39 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:22:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:39 smithi181 ceph-mon[94189]: pgmap v12815: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2023-03-09T21:22:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:39 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:22:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:39 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:22:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:39 smithi038 ceph-mon[92422]: pgmap v12815: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2023-03-09T21:22:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:39 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:22:40.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:39 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:22:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23071.313117 secs 2023-03-09T21:22:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23071.313117 secs 2023-03-09T21:22:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23071.313117 secs 2023-03-09T21:22:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:42 smithi122 ceph-mon[93869]: pgmap v12816: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 123 KiB/s wr, 8 op/s 2023-03-09T21:22:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:42 smithi181 ceph-mon[94189]: pgmap v12816: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 123 KiB/s wr, 8 op/s 2023-03-09T21:22:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:42 smithi038 ceph-mon[92422]: pgmap v12816: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 123 KiB/s wr, 8 op/s 2023-03-09T21:22:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23073.650587 secs 2023-03-09T21:22:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23073.650587 secs 2023-03-09T21:22:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23073.650587 secs 2023-03-09T21:22:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23535.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:22:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:44 smithi122 ceph-mon[93869]: pgmap v12817: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 174 KiB/s wr, 7 op/s 2023-03-09T21:22:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:44 smithi181 ceph-mon[94189]: pgmap v12817: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 174 KiB/s wr, 7 op/s 2023-03-09T21:22:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:44 smithi038 ceph-mon[92422]: pgmap v12817: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 174 KiB/s wr, 7 op/s 2023-03-09T21:22:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23076.313176 secs 2023-03-09T21:22:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:46 smithi122 ceph-mon[93869]: pgmap v12818: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 174 KiB/s wr, 7 op/s 2023-03-09T21:22:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23076.313176 secs 2023-03-09T21:22:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:46 smithi181 ceph-mon[94189]: pgmap v12818: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 174 KiB/s wr, 7 op/s 2023-03-09T21:22:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23076.313176 secs 2023-03-09T21:22:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:46 smithi038 ceph-mon[92422]: pgmap v12818: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 174 KiB/s wr, 7 op/s 2023-03-09T21:22:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:22:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:22:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:48 smithi122 ceph-mon[93869]: pgmap v12819: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 422 KiB/s rd, 406 KiB/s wr, 12 op/s 2023-03-09T21:22:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:22:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:22:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:48 smithi181 ceph-mon[94189]: pgmap v12819: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 422 KiB/s rd, 406 KiB/s wr, 12 op/s 2023-03-09T21:22:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:22:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:22:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:48 smithi038 ceph-mon[92422]: pgmap v12819: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 422 KiB/s rd, 406 KiB/s wr, 12 op/s 2023-03-09T21:22:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23540.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:22:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:49 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:22:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:49 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:22:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:49 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:22:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:50 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:22:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:50 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:22:50.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:50 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:22:50.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:50 smithi122 ceph-mon[93869]: pgmap v12820: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 406 KiB/s wr, 9 op/s 2023-03-09T21:22:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:50 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:22:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:50 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:22:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:50 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:22:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:50 smithi181 ceph-mon[94189]: pgmap v12820: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 406 KiB/s wr, 9 op/s 2023-03-09T21:22:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:50 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:22:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:50 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:22:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:50 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:22:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:50 smithi038 ceph-mon[92422]: pgmap v12820: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 406 KiB/s wr, 9 op/s 2023-03-09T21:22:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23081.313238 secs 2023-03-09T21:22:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23081.313238 secs 2023-03-09T21:22:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23081.313238 secs 2023-03-09T21:22:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:22:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:22:52.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:52 smithi122 ceph-mon[93869]: pgmap v12821: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 442 KiB/s rd, 427 KiB/s wr, 12 op/s 2023-03-09T21:22:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:22:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:22:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:52 smithi181 ceph-mon[94189]: pgmap v12821: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 442 KiB/s rd, 427 KiB/s wr, 12 op/s 2023-03-09T21:22:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:22:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:22:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:52 smithi038 ceph-mon[92422]: pgmap v12821: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 442 KiB/s rd, 427 KiB/s wr, 12 op/s 2023-03-09T21:22:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:22:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:22:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:22:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:22:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:22:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:22:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23545.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:22:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:54 smithi122 ceph-mon[93869]: pgmap v12822: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 603 KiB/s rd, 589 KiB/s wr, 12 op/s 2023-03-09T21:22:54.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:54 smithi181 ceph-mon[94189]: pgmap v12822: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 603 KiB/s rd, 589 KiB/s wr, 12 op/s 2023-03-09T21:22:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:54 smithi038 ceph-mon[92422]: pgmap v12822: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 603 KiB/s rd, 589 KiB/s wr, 12 op/s 2023-03-09T21:22:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23086.313300 secs 2023-03-09T21:22:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:56 smithi122 ceph-mon[93869]: pgmap v12823: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 538 KiB/s wr, 11 op/s 2023-03-09T21:22:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23086.313300 secs 2023-03-09T21:22:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:56 smithi181 ceph-mon[94189]: pgmap v12823: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 538 KiB/s wr, 11 op/s 2023-03-09T21:22:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23086.313300 secs 2023-03-09T21:22:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:56 smithi038 ceph-mon[92422]: pgmap v12823: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 538 KiB/s wr, 11 op/s 2023-03-09T21:22:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23088.650851 secs 2023-03-09T21:22:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:57 smithi122 ceph-mon[93869]: pgmap v12824: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 673 KiB/s rd, 670 KiB/s wr, 15 op/s 2023-03-09T21:22:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23088.650851 secs 2023-03-09T21:22:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:57 smithi181 ceph-mon[94189]: pgmap v12824: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 673 KiB/s rd, 670 KiB/s wr, 15 op/s 2023-03-09T21:22:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23088.650851 secs 2023-03-09T21:22:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:57 smithi038 ceph-mon[92422]: pgmap v12824: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 673 KiB/s rd, 670 KiB/s wr, 15 op/s 2023-03-09T21:22:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23550.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:22:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:22:59.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:22:59.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:22:59 smithi122 ceph-mon[93869]: pgmap v12825: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 537 KiB/s wr, 11 op/s 2023-03-09T21:22:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:22:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:22:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:22:59 smithi181 ceph-mon[94189]: pgmap v12825: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 537 KiB/s wr, 11 op/s 2023-03-09T21:22:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:22:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:22:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:22:59 smithi038 ceph-mon[92422]: pgmap v12825: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 537 KiB/s wr, 11 op/s 2023-03-09T21:23:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:23:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:23:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23091.313360 secs 2023-03-09T21:23:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:23:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:23:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23091.313360 secs 2023-03-09T21:23:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:23:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:23:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23091.313360 secs 2023-03-09T21:23:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:01 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:23:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:23:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:23:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:01 smithi122 ceph-mon[93869]: pgmap v12826: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 456 KiB/s rd, 575 KiB/s wr, 15 op/s 2023-03-09T21:23:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:01 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:23:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:23:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:23:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:01 smithi181 ceph-mon[94189]: pgmap v12826: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 456 KiB/s rd, 575 KiB/s wr, 15 op/s 2023-03-09T21:23:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:01 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:23:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:23:02.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:23:02.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:01 smithi038 ceph-mon[92422]: pgmap v12826: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 456 KiB/s rd, 575 KiB/s wr, 15 op/s 2023-03-09T21:23:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23093.650972 secs 2023-03-09T21:23:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23093.650972 secs 2023-03-09T21:23:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23093.650972 secs 2023-03-09T21:23:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23555.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:23:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:04 smithi122 ceph-mon[93869]: pgmap v12827: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 598 KiB/s wr, 26 op/s 2023-03-09T21:23:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:04 smithi181 ceph-mon[94189]: pgmap v12827: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 598 KiB/s wr, 26 op/s 2023-03-09T21:23:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:04 smithi038 ceph-mon[92422]: pgmap v12827: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 598 KiB/s wr, 26 op/s 2023-03-09T21:23:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23096.313420 secs 2023-03-09T21:23:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:06 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:23:06.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:06 smithi122 ceph-mon[93869]: pgmap v12828: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 313 KiB/s wr, 23 op/s 2023-03-09T21:23:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23096.313420 secs 2023-03-09T21:23:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:06 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:23:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:06 smithi181 ceph-mon[94189]: pgmap v12828: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 313 KiB/s wr, 23 op/s 2023-03-09T21:23:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23096.313420 secs 2023-03-09T21:23:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:06 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:23:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:06 smithi038 ceph-mon[92422]: pgmap v12828: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 313 KiB/s wr, 23 op/s 2023-03-09T21:23:07.969 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23098.651122 secs 2023-03-09T21:23:07.969 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:07 smithi038 ceph-mon[92422]: pgmap v12829: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 314 KiB/s wr, 26 op/s 2023-03-09T21:23:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23098.651122 secs 2023-03-09T21:23:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:07 smithi122 ceph-mon[93869]: pgmap v12829: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 314 KiB/s wr, 26 op/s 2023-03-09T21:23:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23098.651122 secs 2023-03-09T21:23:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:07 smithi181 ceph-mon[94189]: pgmap v12829: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 314 KiB/s wr, 26 op/s 2023-03-09T21:23:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23560.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:23:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:10 smithi122 ceph-mon[93869]: osdmap e1015: 12 total, 12 up, 12 in 2023-03-09T21:23:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:23:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:23:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:23:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:23:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:23:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:10 smithi122 ceph-mon[93869]: pgmap v12831: 129 pgs: 10 active+clean+snaptrim_wait, 119 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 206 KiB/s wr, 26 op/s 2023-03-09T21:23:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:10 smithi181 ceph-mon[94189]: osdmap e1015: 12 total, 12 up, 12 in 2023-03-09T21:23:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:23:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:23:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:23:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:23:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:23:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:10 smithi181 ceph-mon[94189]: pgmap v12831: 129 pgs: 10 active+clean+snaptrim_wait, 119 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 206 KiB/s wr, 26 op/s 2023-03-09T21:23:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:10 smithi038 ceph-mon[92422]: osdmap e1015: 12 total, 12 up, 12 in 2023-03-09T21:23:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:23:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:23:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:23:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:23:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:23:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:10 smithi038 ceph-mon[92422]: pgmap v12831: 129 pgs: 10 active+clean+snaptrim_wait, 119 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 206 KiB/s wr, 26 op/s 2023-03-09T21:23:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23101.313483 secs 2023-03-09T21:23:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23101.313483 secs 2023-03-09T21:23:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23101.313483 secs 2023-03-09T21:23:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:12 smithi038 ceph-mon[92422]: pgmap v12832: 129 pgs: 10 active+clean+snaptrim_wait, 119 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 160 KiB/s wr, 23 op/s 2023-03-09T21:23:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:12 smithi122 ceph-mon[93869]: pgmap v12832: 129 pgs: 10 active+clean+snaptrim_wait, 119 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 160 KiB/s wr, 23 op/s 2023-03-09T21:23:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:12 smithi181 ceph-mon[94189]: pgmap v12832: 129 pgs: 10 active+clean+snaptrim_wait, 119 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 160 KiB/s wr, 23 op/s 2023-03-09T21:23:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23103.651197 secs 2023-03-09T21:23:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23103.651197 secs 2023-03-09T21:23:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23103.651197 secs 2023-03-09T21:23:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23565.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:23:14.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:14 smithi122 ceph-mon[93869]: pgmap v12833: 129 pgs: 10 active+clean+snaptrim_wait, 119 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 106 KiB/s wr, 7 op/s 2023-03-09T21:23:14.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:14 smithi181 ceph-mon[94189]: pgmap v12833: 129 pgs: 10 active+clean+snaptrim_wait, 119 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 106 KiB/s wr, 7 op/s 2023-03-09T21:23:14.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:14 smithi038 ceph-mon[92422]: pgmap v12833: 129 pgs: 10 active+clean+snaptrim_wait, 119 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 106 KiB/s wr, 7 op/s 2023-03-09T21:23:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23106.313543 secs 2023-03-09T21:23:16.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:16 smithi122 ceph-mon[93869]: pgmap v12834: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 106 KiB/s wr, 7 op/s 2023-03-09T21:23:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23106.313543 secs 2023-03-09T21:23:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:16 smithi181 ceph-mon[94189]: pgmap v12834: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 106 KiB/s wr, 7 op/s 2023-03-09T21:23:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23106.313543 secs 2023-03-09T21:23:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:16 smithi038 ceph-mon[92422]: pgmap v12834: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 106 KiB/s wr, 7 op/s 2023-03-09T21:23:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23108.651314 secs 2023-03-09T21:23:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:17 smithi122 ceph-mon[93869]: pgmap v12835: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 106 KiB/s wr, 7 op/s 2023-03-09T21:23:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23108.651314 secs 2023-03-09T21:23:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:17 smithi181 ceph-mon[94189]: pgmap v12835: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 106 KiB/s wr, 7 op/s 2023-03-09T21:23:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23108.651314 secs 2023-03-09T21:23:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:17 smithi038 ceph-mon[92422]: pgmap v12835: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 106 KiB/s wr, 7 op/s 2023-03-09T21:23:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23570.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:23:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:19 smithi122 ceph-mon[93869]: pgmap v12836: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 6 op/s 2023-03-09T21:23:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:19 smithi181 ceph-mon[94189]: pgmap v12836: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 6 op/s 2023-03-09T21:23:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:19 smithi038 ceph-mon[92422]: pgmap v12836: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 6 op/s 2023-03-09T21:23:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23111.313604 secs 2023-03-09T21:23:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:21 smithi122 ceph-mon[93869]: osdmap e1016: 12 total, 12 up, 12 in 2023-03-09T21:23:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23111.313604 secs 2023-03-09T21:23:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:21 smithi181 ceph-mon[94189]: osdmap e1016: 12 total, 12 up, 12 in 2023-03-09T21:23:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23111.313604 secs 2023-03-09T21:23:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:21 smithi038 ceph-mon[92422]: osdmap e1016: 12 total, 12 up, 12 in 2023-03-09T21:23:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:22 smithi122 ceph-mon[93869]: pgmap v12838: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 75 KiB/s wr, 7 op/s 2023-03-09T21:23:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:22 smithi181 ceph-mon[94189]: pgmap v12838: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 75 KiB/s wr, 7 op/s 2023-03-09T21:23:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:22 smithi038 ceph-mon[92422]: pgmap v12838: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 75 KiB/s wr, 7 op/s 2023-03-09T21:23:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23113.651447 secs 2023-03-09T21:23:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23113.651447 secs 2023-03-09T21:23:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23113.651447 secs 2023-03-09T21:23:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23575.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:23:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:24 smithi122 ceph-mon[93869]: pgmap v12839: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 75 KiB/s wr, 6 op/s 2023-03-09T21:23:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:24 smithi181 ceph-mon[94189]: pgmap v12839: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 75 KiB/s wr, 6 op/s 2023-03-09T21:23:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:24 smithi038 ceph-mon[92422]: pgmap v12839: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 75 KiB/s wr, 6 op/s 2023-03-09T21:23:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23116.313665 secs 2023-03-09T21:23:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:26 smithi122 ceph-mon[93869]: pgmap v12840: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 75 KiB/s wr, 6 op/s 2023-03-09T21:23:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23116.313665 secs 2023-03-09T21:23:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:26 smithi181 ceph-mon[94189]: pgmap v12840: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 75 KiB/s wr, 6 op/s 2023-03-09T21:23:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23116.313665 secs 2023-03-09T21:23:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:26 smithi038 ceph-mon[92422]: pgmap v12840: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 75 KiB/s wr, 6 op/s 2023-03-09T21:23:27.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23118.651593 secs 2023-03-09T21:23:27.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:27 smithi122 ceph-mon[93869]: pgmap v12841: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 75 KiB/s wr, 6 op/s 2023-03-09T21:23:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23118.651593 secs 2023-03-09T21:23:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:27 smithi181 ceph-mon[94189]: pgmap v12841: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 75 KiB/s wr, 6 op/s 2023-03-09T21:23:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23118.651593 secs 2023-03-09T21:23:28.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:27 smithi038 ceph-mon[92422]: pgmap v12841: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 75 KiB/s wr, 6 op/s 2023-03-09T21:23:28.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23580.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:23:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:29 smithi122 ceph-mon[93869]: pgmap v12842: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T21:23:29.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:29 smithi181 ceph-mon[94189]: pgmap v12842: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T21:23:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:29 smithi038 ceph-mon[92422]: pgmap v12842: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T21:23:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23121.313746 secs 2023-03-09T21:23:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23121.313746 secs 2023-03-09T21:23:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23121.313746 secs 2023-03-09T21:23:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:32 smithi122 ceph-mon[93869]: pgmap v12843: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 265 B/s wr, 7 op/s 2023-03-09T21:23:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:32 smithi181 ceph-mon[94189]: pgmap v12843: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 265 B/s wr, 7 op/s 2023-03-09T21:23:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:32 smithi038 ceph-mon[92422]: pgmap v12843: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 265 B/s wr, 7 op/s 2023-03-09T21:23:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23123.651718 secs 2023-03-09T21:23:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23123.651718 secs 2023-03-09T21:23:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23123.651718 secs 2023-03-09T21:23:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23585.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:23:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:34 smithi122 ceph-mon[93869]: pgmap v12844: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 341 B/s wr, 6 op/s 2023-03-09T21:23:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:34 smithi181 ceph-mon[94189]: pgmap v12844: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 341 B/s wr, 6 op/s 2023-03-09T21:23:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:34 smithi038 ceph-mon[92422]: pgmap v12844: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 341 B/s wr, 6 op/s 2023-03-09T21:23:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23126.313806 secs 2023-03-09T21:23:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:36 smithi122 ceph-mon[93869]: pgmap v12845: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 341 B/s wr, 5 op/s 2023-03-09T21:23:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23126.313806 secs 2023-03-09T21:23:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:36 smithi181 ceph-mon[94189]: pgmap v12845: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 341 B/s wr, 5 op/s 2023-03-09T21:23:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23126.313806 secs 2023-03-09T21:23:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:36 smithi038 ceph-mon[92422]: pgmap v12845: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 341 B/s wr, 5 op/s 2023-03-09T21:23:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:37 smithi122 ceph-mon[93869]: pgmap v12846: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1023 B/s wr, 8 op/s 2023-03-09T21:23:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:37 smithi181 ceph-mon[94189]: pgmap v12846: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1023 B/s wr, 8 op/s 2023-03-09T21:23:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:37 smithi038 ceph-mon[92422]: pgmap v12846: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1023 B/s wr, 8 op/s 2023-03-09T21:23:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23590.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:23:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:39 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:23:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:39 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:23:39.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:39 smithi122 ceph-mon[93869]: pgmap v12847: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 92 KiB/s wr, 7 op/s 2023-03-09T21:23:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:39 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:23:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:39 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:23:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:39 smithi181 ceph-mon[94189]: pgmap v12847: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 92 KiB/s wr, 7 op/s 2023-03-09T21:23:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:39 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:23:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:39 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:23:40.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:39 smithi038 ceph-mon[92422]: pgmap v12847: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 92 KiB/s wr, 7 op/s 2023-03-09T21:23:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:23:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:23:40.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23131.313866 secs 2023-03-09T21:23:40.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:40 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:23:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:23:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:23:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23131.313866 secs 2023-03-09T21:23:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:40 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:23:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:23:41.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:23:41.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23131.313866 secs 2023-03-09T21:23:41.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:40 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:23:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:41 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:23:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:23:41.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:23:41.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:41 smithi122 ceph-mon[93869]: pgmap v12848: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 273 KiB/s wr, 10 op/s 2023-03-09T21:23:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:41 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:23:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:23:41.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:23:41.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:41 smithi181 ceph-mon[94189]: pgmap v12848: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 273 KiB/s wr, 10 op/s 2023-03-09T21:23:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:41 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:23:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:23:42.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:23:42.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:41 smithi038 ceph-mon[92422]: pgmap v12848: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 273 KiB/s wr, 10 op/s 2023-03-09T21:23:43.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23595.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:23:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:43 smithi122 ceph-mon[93869]: pgmap v12849: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 273 KiB/s wr, 8 op/s 2023-03-09T21:23:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:43 smithi181 ceph-mon[94189]: pgmap v12849: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 273 KiB/s wr, 8 op/s 2023-03-09T21:23:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:43 smithi038 ceph-mon[92422]: pgmap v12849: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 273 KiB/s wr, 8 op/s 2023-03-09T21:23:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23136.313950 secs 2023-03-09T21:23:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:46 smithi122 ceph-mon[93869]: pgmap v12850: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 452 KiB/s wr, 9 op/s 2023-03-09T21:23:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23136.313950 secs 2023-03-09T21:23:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:46 smithi181 ceph-mon[94189]: pgmap v12850: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 452 KiB/s wr, 9 op/s 2023-03-09T21:23:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23136.313950 secs 2023-03-09T21:23:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:46 smithi038 ceph-mon[92422]: pgmap v12850: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 452 KiB/s wr, 9 op/s 2023-03-09T21:23:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:47 smithi122 ceph-mon[93869]: pgmap v12851: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 497 KiB/s rd, 481 KiB/s wr, 12 op/s 2023-03-09T21:23:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:47 smithi181 ceph-mon[94189]: pgmap v12851: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 497 KiB/s rd, 481 KiB/s wr, 12 op/s 2023-03-09T21:23:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:47 smithi038 ceph-mon[92422]: pgmap v12851: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 497 KiB/s rd, 481 KiB/s wr, 12 op/s 2023-03-09T21:23:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23600.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:23:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:49 smithi122 ceph-mon[93869]: pgmap v12852: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 480 KiB/s wr, 10 op/s 2023-03-09T21:23:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:49 smithi181 ceph-mon[94189]: pgmap v12852: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 480 KiB/s wr, 10 op/s 2023-03-09T21:23:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:49 smithi038 ceph-mon[92422]: pgmap v12852: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 480 KiB/s wr, 10 op/s 2023-03-09T21:23:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23141.313982 secs 2023-03-09T21:23:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23141.313982 secs 2023-03-09T21:23:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23141.313982 secs 2023-03-09T21:23:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:23:52.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:23:52.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:52 smithi122 ceph-mon[93869]: pgmap v12853: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 411 KiB/s wr, 13 op/s 2023-03-09T21:23:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:23:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:23:52.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:52 smithi181 ceph-mon[94189]: pgmap v12853: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 411 KiB/s wr, 13 op/s 2023-03-09T21:23:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:23:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:23:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:52 smithi038 ceph-mon[92422]: pgmap v12853: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 411 KiB/s wr, 13 op/s 2023-03-09T21:23:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:23:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:23:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:53 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 23143.652217 secs 2023-03-09T21:23:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:23:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:23:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:53 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 23143.652217 secs 2023-03-09T21:23:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:23:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:23:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:53 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 23143.652217 secs 2023-03-09T21:23:53.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23605.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:23:54.065 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:54 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:23:54.066 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:54 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:23:54.066 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:54 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:23:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:54 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:23:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:54 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:23:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:54 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:23:54.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:54 smithi122 ceph-mon[93869]: pgmap v12854: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 554 KiB/s rd, 541 KiB/s wr, 12 op/s 2023-03-09T21:23:54.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:54 smithi181 ceph-mon[94189]: pgmap v12854: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 554 KiB/s rd, 541 KiB/s wr, 12 op/s 2023-03-09T21:23:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:54 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:23:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:54 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:23:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:54 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:23:54.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:54 smithi038 ceph-mon[92422]: pgmap v12854: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 554 KiB/s rd, 541 KiB/s wr, 12 op/s 2023-03-09T21:23:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23146.314043 secs 2023-03-09T21:23:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:56 smithi122 ceph-mon[93869]: pgmap v12855: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 646 KiB/s rd, 633 KiB/s wr, 12 op/s 2023-03-09T21:23:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23146.314043 secs 2023-03-09T21:23:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:56 smithi181 ceph-mon[94189]: pgmap v12855: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 646 KiB/s rd, 633 KiB/s wr, 12 op/s 2023-03-09T21:23:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23146.314043 secs 2023-03-09T21:23:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:56 smithi038 ceph-mon[92422]: pgmap v12855: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 646 KiB/s rd, 633 KiB/s wr, 12 op/s 2023-03-09T21:23:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23148.652336 secs 2023-03-09T21:23:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:57 smithi122 ceph-mon[93869]: pgmap v12856: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 497 KiB/s rd, 494 KiB/s wr, 15 op/s 2023-03-09T21:23:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23148.652336 secs 2023-03-09T21:23:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:57 smithi181 ceph-mon[94189]: pgmap v12856: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 497 KiB/s rd, 494 KiB/s wr, 15 op/s 2023-03-09T21:23:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23148.652336 secs 2023-03-09T21:23:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:57 smithi038 ceph-mon[92422]: pgmap v12856: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 497 KiB/s rd, 494 KiB/s wr, 15 op/s 2023-03-09T21:23:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23610.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:23:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:23:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:23:59.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:23:59 smithi122 ceph-mon[93869]: pgmap v12857: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 465 KiB/s wr, 11 op/s 2023-03-09T21:23:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:23:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:23:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:23:59 smithi181 ceph-mon[94189]: pgmap v12857: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 465 KiB/s wr, 11 op/s 2023-03-09T21:23:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:23:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:24:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:23:59 smithi038 ceph-mon[92422]: pgmap v12857: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 465 KiB/s wr, 11 op/s 2023-03-09T21:24:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:24:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:24:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23151.314103 secs 2023-03-09T21:24:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:24:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:24:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23151.314103 secs 2023-03-09T21:24:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:24:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:24:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23151.314103 secs 2023-03-09T21:24:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:02 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:24:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:02 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:24:02.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:02 smithi122 ceph-mon[93869]: pgmap v12858: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 604 KiB/s rd, 624 KiB/s wr, 16 op/s 2023-03-09T21:24:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:02 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:24:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:02 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:24:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:02 smithi181 ceph-mon[94189]: pgmap v12858: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 604 KiB/s rd, 624 KiB/s wr, 16 op/s 2023-03-09T21:24:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:02 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:24:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:02 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:24:02.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:02 smithi038 ceph-mon[92422]: pgmap v12858: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 604 KiB/s rd, 624 KiB/s wr, 16 op/s 2023-03-09T21:24:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23153.652453 secs 2023-03-09T21:24:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23153.652453 secs 2023-03-09T21:24:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23153.652453 secs 2023-03-09T21:24:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23615.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:24:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:04 smithi122 ceph-mon[93869]: pgmap v12859: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 590 KiB/s rd, 624 KiB/s wr, 25 op/s 2023-03-09T21:24:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:04 smithi181 ceph-mon[94189]: pgmap v12859: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 590 KiB/s rd, 624 KiB/s wr, 25 op/s 2023-03-09T21:24:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:04 smithi038 ceph-mon[92422]: pgmap v12859: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 590 KiB/s rd, 624 KiB/s wr, 25 op/s 2023-03-09T21:24:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23156.314166 secs 2023-03-09T21:24:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:06 smithi122 ceph-mon[93869]: pgmap v12860: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 403 KiB/s wr, 24 op/s 2023-03-09T21:24:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23156.314166 secs 2023-03-09T21:24:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:06 smithi181 ceph-mon[94189]: pgmap v12860: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 403 KiB/s wr, 24 op/s 2023-03-09T21:24:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23156.314166 secs 2023-03-09T21:24:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:06 smithi038 ceph-mon[92422]: pgmap v12860: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 403 KiB/s wr, 24 op/s 2023-03-09T21:24:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23158.652574 secs 2023-03-09T21:24:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:07 smithi122 ceph-mon[93869]: pgmap v12861: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 311 KiB/s wr, 26 op/s 2023-03-09T21:24:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23158.652574 secs 2023-03-09T21:24:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:07 smithi181 ceph-mon[94189]: pgmap v12861: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 311 KiB/s wr, 26 op/s 2023-03-09T21:24:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23158.652574 secs 2023-03-09T21:24:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:07 smithi038 ceph-mon[92422]: pgmap v12861: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 311 KiB/s wr, 26 op/s 2023-03-09T21:24:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23620.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:24:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:10 smithi122 ceph-mon[93869]: osdmap e1017: 12 total, 12 up, 12 in 2023-03-09T21:24:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:24:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:10 smithi122 ceph-mon[93869]: pgmap v12863: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 325 KiB/s wr, 27 op/s 2023-03-09T21:24:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:10 smithi181 ceph-mon[94189]: osdmap e1017: 12 total, 12 up, 12 in 2023-03-09T21:24:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:24:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:10 smithi181 ceph-mon[94189]: pgmap v12863: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 325 KiB/s wr, 27 op/s 2023-03-09T21:24:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:10 smithi038 ceph-mon[92422]: osdmap e1017: 12 total, 12 up, 12 in 2023-03-09T21:24:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:24:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:10 smithi038 ceph-mon[92422]: pgmap v12863: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 325 KiB/s wr, 27 op/s 2023-03-09T21:24:11.324 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23161.314226 secs 2023-03-09T21:24:11.347 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23161.314226 secs 2023-03-09T21:24:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23161.314226 secs 2023-03-09T21:24:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:12 smithi122 ceph-mon[93869]: pgmap v12864: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 134 KiB/s wr, 23 op/s 2023-03-09T21:24:12.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:12 smithi181 ceph-mon[94189]: pgmap v12864: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 134 KiB/s wr, 23 op/s 2023-03-09T21:24:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:12 smithi038 ceph-mon[92422]: pgmap v12864: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 134 KiB/s wr, 23 op/s 2023-03-09T21:24:13.169 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23163.652692 secs 2023-03-09T21:24:13.169 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:13 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:24:13.169 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:13 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:24:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23163.652692 secs 2023-03-09T21:24:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:13 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:24:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:13 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:24:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23163.652692 secs 2023-03-09T21:24:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:13 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:24:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:13 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:24:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23625.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:24:14.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:14 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:24:14.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:14 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:24:14.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:14 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:24:14.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:14 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:24:14.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:14 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:24:14.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:14 smithi122 ceph-mon[93869]: pgmap v12865: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 108 KiB/s wr, 8 op/s 2023-03-09T21:24:14.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:14 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:24:14.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:14 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:24:14.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:14 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:24:14.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:14 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:24:14.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:14 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:24:14.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:14 smithi181 ceph-mon[94189]: pgmap v12865: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 108 KiB/s wr, 8 op/s 2023-03-09T21:24:14.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:14 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:24:14.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:14 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:24:14.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:14 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:24:14.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:14 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:24:14.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:14 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:24:14.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:14 smithi038 ceph-mon[92422]: pgmap v12865: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 108 KiB/s wr, 8 op/s 2023-03-09T21:24:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23166.314287 secs 2023-03-09T21:24:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:16 smithi122 ceph-mon[93869]: pgmap v12866: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 47 KiB/s wr, 7 op/s 2023-03-09T21:24:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23166.314287 secs 2023-03-09T21:24:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:16 smithi181 ceph-mon[94189]: pgmap v12866: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 47 KiB/s wr, 7 op/s 2023-03-09T21:24:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23166.314287 secs 2023-03-09T21:24:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:16 smithi038 ceph-mon[92422]: pgmap v12866: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 47 KiB/s wr, 7 op/s 2023-03-09T21:24:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23168.652835 secs 2023-03-09T21:24:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:17 smithi122 ceph-mon[93869]: pgmap v12867: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 47 KiB/s wr, 7 op/s 2023-03-09T21:24:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23168.652835 secs 2023-03-09T21:24:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:17 smithi181 ceph-mon[94189]: pgmap v12867: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 47 KiB/s wr, 7 op/s 2023-03-09T21:24:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23168.652835 secs 2023-03-09T21:24:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:17 smithi038 ceph-mon[92422]: pgmap v12867: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 47 KiB/s wr, 7 op/s 2023-03-09T21:24:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23630.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:24:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:19 smithi122 ceph-mon[93869]: pgmap v12868: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 7 op/s 2023-03-09T21:24:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:19 smithi181 ceph-mon[94189]: pgmap v12868: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 7 op/s 2023-03-09T21:24:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:19 smithi038 ceph-mon[92422]: pgmap v12868: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 7 op/s 2023-03-09T21:24:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23171.314346 secs 2023-03-09T21:24:21.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:21 smithi122 ceph-mon[93869]: osdmap e1018: 12 total, 12 up, 12 in 2023-03-09T21:24:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23171.314346 secs 2023-03-09T21:24:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:21 smithi181 ceph-mon[94189]: osdmap e1018: 12 total, 12 up, 12 in 2023-03-09T21:24:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23171.314346 secs 2023-03-09T21:24:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:21 smithi038 ceph-mon[92422]: osdmap e1018: 12 total, 12 up, 12 in 2023-03-09T21:24:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:22 smithi122 ceph-mon[93869]: pgmap v12870: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 75 KiB/s wr, 7 op/s 2023-03-09T21:24:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:22 smithi181 ceph-mon[94189]: pgmap v12870: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 75 KiB/s wr, 7 op/s 2023-03-09T21:24:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:22 smithi038 ceph-mon[92422]: pgmap v12870: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 75 KiB/s wr, 7 op/s 2023-03-09T21:24:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23173.652976 secs 2023-03-09T21:24:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23173.652976 secs 2023-03-09T21:24:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23173.652976 secs 2023-03-09T21:24:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23635.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:24:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:24 smithi122 ceph-mon[93869]: pgmap v12871: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 76 KiB/s wr, 7 op/s 2023-03-09T21:24:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:24 smithi181 ceph-mon[94189]: pgmap v12871: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 76 KiB/s wr, 7 op/s 2023-03-09T21:24:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:24 smithi038 ceph-mon[92422]: pgmap v12871: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 76 KiB/s wr, 7 op/s 2023-03-09T21:24:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23176.314408 secs 2023-03-09T21:24:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:26 smithi122 ceph-mon[93869]: pgmap v12872: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 29 KiB/s wr, 7 op/s 2023-03-09T21:24:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23176.314408 secs 2023-03-09T21:24:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:26 smithi181 ceph-mon[94189]: pgmap v12872: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 29 KiB/s wr, 7 op/s 2023-03-09T21:24:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23176.314408 secs 2023-03-09T21:24:26.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:26 smithi038 ceph-mon[92422]: pgmap v12872: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 29 KiB/s wr, 7 op/s 2023-03-09T21:24:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23178.653073 secs 2023-03-09T21:24:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:27 smithi122 ceph-mon[93869]: pgmap v12873: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 29 KiB/s wr, 7 op/s 2023-03-09T21:24:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23178.653073 secs 2023-03-09T21:24:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:27 smithi181 ceph-mon[94189]: pgmap v12873: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 29 KiB/s wr, 7 op/s 2023-03-09T21:24:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23178.653073 secs 2023-03-09T21:24:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:27 smithi038 ceph-mon[92422]: pgmap v12873: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 29 KiB/s wr, 7 op/s 2023-03-09T21:24:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23640.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:24:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:29 smithi122 ceph-mon[93869]: pgmap v12874: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 111 KiB/s wr, 8 op/s 2023-03-09T21:24:29.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:29 smithi181 ceph-mon[94189]: pgmap v12874: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 111 KiB/s wr, 8 op/s 2023-03-09T21:24:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:29 smithi038 ceph-mon[92422]: pgmap v12874: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 111 KiB/s wr, 8 op/s 2023-03-09T21:24:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23181.314468 secs 2023-03-09T21:24:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23181.314468 secs 2023-03-09T21:24:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23181.314468 secs 2023-03-09T21:24:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:32 smithi122 ceph-mon[93869]: pgmap v12875: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 96 KiB/s wr, 9 op/s 2023-03-09T21:24:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:32 smithi181 ceph-mon[94189]: pgmap v12875: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 96 KiB/s wr, 9 op/s 2023-03-09T21:24:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:32 smithi038 ceph-mon[92422]: pgmap v12875: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 96 KiB/s wr, 9 op/s 2023-03-09T21:24:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:33 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:24:33.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:33 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:24:33.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23183.653180 secs 2023-03-09T21:24:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:33 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:24:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:33 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:24:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23183.653180 secs 2023-03-09T21:24:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:33 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:24:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:33 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:24:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23183.653180 secs 2023-03-09T21:24:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23645.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:24:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:34 smithi122 ceph-mon[93869]: pgmap v12876: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 KiB/s wr, 7 op/s 2023-03-09T21:24:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:34 smithi181 ceph-mon[94189]: pgmap v12876: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 KiB/s wr, 7 op/s 2023-03-09T21:24:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:34 smithi038 ceph-mon[92422]: pgmap v12876: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 KiB/s wr, 7 op/s 2023-03-09T21:24:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23186.314539 secs 2023-03-09T21:24:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:24:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:24:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:36 smithi122 ceph-mon[93869]: pgmap v12877: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 KiB/s wr, 7 op/s 2023-03-09T21:24:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23186.314539 secs 2023-03-09T21:24:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:24:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:24:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:36 smithi181 ceph-mon[94189]: pgmap v12877: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 KiB/s wr, 7 op/s 2023-03-09T21:24:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23186.314539 secs 2023-03-09T21:24:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:24:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:24:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:36 smithi038 ceph-mon[92422]: pgmap v12877: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 KiB/s wr, 7 op/s 2023-03-09T21:24:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:37 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 23188.653294 secs 2023-03-09T21:24:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:37 smithi122 ceph-mon[93869]: pgmap v12878: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 93 KiB/s wr, 9 op/s 2023-03-09T21:24:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:37 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 23188.653294 secs 2023-03-09T21:24:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:37 smithi181 ceph-mon[94189]: pgmap v12878: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 93 KiB/s wr, 9 op/s 2023-03-09T21:24:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:37 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 23188.653294 secs 2023-03-09T21:24:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:37 smithi038 ceph-mon[92422]: pgmap v12878: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 93 KiB/s wr, 9 op/s 2023-03-09T21:24:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23650.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:24:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:39 smithi122 ceph-mon[93869]: pgmap v12879: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 235 KiB/s wr, 7 op/s 2023-03-09T21:24:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:39 smithi181 ceph-mon[94189]: pgmap v12879: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 235 KiB/s wr, 7 op/s 2023-03-09T21:24:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:39 smithi038 ceph-mon[92422]: pgmap v12879: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 235 KiB/s wr, 7 op/s 2023-03-09T21:24:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23191.314625 secs 2023-03-09T21:24:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23191.314625 secs 2023-03-09T21:24:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23191.314625 secs 2023-03-09T21:24:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:42 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:24:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:42 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:24:42.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:42 smithi122 ceph-mon[93869]: pgmap v12880: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 147 KiB/s wr, 9 op/s 2023-03-09T21:24:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:42 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:24:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:42 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:24:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:42 smithi181 ceph-mon[94189]: pgmap v12880: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 147 KiB/s wr, 9 op/s 2023-03-09T21:24:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:42 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:24:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:42 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:24:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:42 smithi038 ceph-mon[92422]: pgmap v12880: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 147 KiB/s wr, 9 op/s 2023-03-09T21:24:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:43 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 23193.653374 secs 2023-03-09T21:24:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:43 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 23193.653374 secs 2023-03-09T21:24:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:43 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 23193.653374 secs 2023-03-09T21:24:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23655.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:24:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:44 smithi122 ceph-mon[93869]: pgmap v12881: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 162 KiB/s wr, 7 op/s 2023-03-09T21:24:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:44 smithi181 ceph-mon[94189]: pgmap v12881: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 162 KiB/s wr, 7 op/s 2023-03-09T21:24:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:44 smithi038 ceph-mon[92422]: pgmap v12881: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 162 KiB/s wr, 7 op/s 2023-03-09T21:24:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23196.314708 secs 2023-03-09T21:24:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:46 smithi122 ceph-mon[93869]: pgmap v12882: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 245 KiB/s wr, 9 op/s 2023-03-09T21:24:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23196.314708 secs 2023-03-09T21:24:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:46 smithi181 ceph-mon[94189]: pgmap v12882: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 245 KiB/s wr, 9 op/s 2023-03-09T21:24:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23196.314708 secs 2023-03-09T21:24:46.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:46 smithi038 ceph-mon[92422]: pgmap v12882: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 245 KiB/s wr, 9 op/s 2023-03-09T21:24:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23198.653485 secs 2023-03-09T21:24:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:47 smithi122 ceph-mon[93869]: pgmap v12883: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 377 KiB/s wr, 11 op/s 2023-03-09T21:24:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23198.653485 secs 2023-03-09T21:24:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:47 smithi181 ceph-mon[94189]: pgmap v12883: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 377 KiB/s wr, 11 op/s 2023-03-09T21:24:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23198.653485 secs 2023-03-09T21:24:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:47 smithi038 ceph-mon[92422]: pgmap v12883: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 377 KiB/s wr, 11 op/s 2023-03-09T21:24:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23660.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:24:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:24:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:24:49.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:49 smithi122 ceph-mon[93869]: pgmap v12884: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 395 KiB/s wr, 11 op/s 2023-03-09T21:24:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:24:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:24:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:49 smithi181 ceph-mon[94189]: pgmap v12884: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 395 KiB/s wr, 11 op/s 2023-03-09T21:24:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:24:50.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:24:50.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:49 smithi038 ceph-mon[92422]: pgmap v12884: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 395 KiB/s wr, 11 op/s 2023-03-09T21:24:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23201.314779 secs 2023-03-09T21:24:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23201.314779 secs 2023-03-09T21:24:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23201.314779 secs 2023-03-09T21:24:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:24:52.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:24:52.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:52 smithi122 ceph-mon[93869]: pgmap v12885: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 557 KiB/s rd, 253 KiB/s wr, 12 op/s 2023-03-09T21:24:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:24:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:24:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:52 smithi181 ceph-mon[94189]: pgmap v12885: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 557 KiB/s rd, 253 KiB/s wr, 12 op/s 2023-03-09T21:24:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:24:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:24:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:52 smithi038 ceph-mon[92422]: pgmap v12885: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 557 KiB/s rd, 253 KiB/s wr, 12 op/s 2023-03-09T21:24:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:24:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:24:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23203.653582 secs 2023-03-09T21:24:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:24:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:24:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23203.653582 secs 2023-03-09T21:24:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:24:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:24:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23203.653582 secs 2023-03-09T21:24:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23665.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:24:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:55 smithi122 ceph-mon[93869]: pgmap v12886: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 805 KiB/s rd, 478 KiB/s wr, 12 op/s 2023-03-09T21:24:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:55 smithi181 ceph-mon[94189]: pgmap v12886: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 805 KiB/s rd, 478 KiB/s wr, 12 op/s 2023-03-09T21:24:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:55 smithi038 ceph-mon[92422]: pgmap v12886: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 805 KiB/s rd, 478 KiB/s wr, 12 op/s 2023-03-09T21:24:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23206.314840 secs 2023-03-09T21:24:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23206.314840 secs 2023-03-09T21:24:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23206.314840 secs 2023-03-09T21:24:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:57 smithi122 ceph-mon[93869]: pgmap v12887: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 790 KiB/s rd, 463 KiB/s wr, 12 op/s 2023-03-09T21:24:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:57 smithi181 ceph-mon[94189]: pgmap v12887: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 790 KiB/s rd, 463 KiB/s wr, 12 op/s 2023-03-09T21:24:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:57 smithi038 ceph-mon[92422]: pgmap v12887: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 790 KiB/s rd, 463 KiB/s wr, 12 op/s 2023-03-09T21:24:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23208.653702 secs 2023-03-09T21:24:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23208.653702 secs 2023-03-09T21:24:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23208.653702 secs 2023-03-09T21:24:58.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23670.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:24:59.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:24:59 smithi122 ceph-mon[93869]: pgmap v12888: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 503 KiB/s rd, 380 KiB/s wr, 14 op/s 2023-03-09T21:24:59.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:24:59 smithi181 ceph-mon[94189]: pgmap v12888: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 503 KiB/s rd, 380 KiB/s wr, 14 op/s 2023-03-09T21:24:59.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:24:59 smithi038 ceph-mon[92422]: pgmap v12888: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 503 KiB/s rd, 380 KiB/s wr, 14 op/s 2023-03-09T21:25:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:25:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:25:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:25:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:25:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:25:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:25:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:01 smithi122 ceph-mon[93869]: pgmap v12889: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 608 KiB/s wr, 13 op/s 2023-03-09T21:25:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23211.314901 secs 2023-03-09T21:25:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:25:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:25:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:01 smithi181 ceph-mon[94189]: pgmap v12889: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 608 KiB/s wr, 13 op/s 2023-03-09T21:25:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23211.314901 secs 2023-03-09T21:25:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:25:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:25:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:01 smithi038 ceph-mon[92422]: pgmap v12889: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 608 KiB/s wr, 13 op/s 2023-03-09T21:25:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23211.314901 secs 2023-03-09T21:25:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:25:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:25:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:03 smithi122 ceph-mon[93869]: pgmap v12890: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 636 KiB/s rd, 612 KiB/s wr, 15 op/s 2023-03-09T21:25:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23213.653832 secs 2023-03-09T21:25:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:03 smithi181 ceph-mon[94189]: pgmap v12890: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 636 KiB/s rd, 612 KiB/s wr, 15 op/s 2023-03-09T21:25:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23213.653832 secs 2023-03-09T21:25:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:03 smithi038 ceph-mon[92422]: pgmap v12890: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 636 KiB/s rd, 612 KiB/s wr, 15 op/s 2023-03-09T21:25:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23213.653832 secs 2023-03-09T21:25:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23675.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:25:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:05 smithi122 ceph-mon[93869]: pgmap v12891: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 771 KiB/s wr, 28 op/s 2023-03-09T21:25:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:05 smithi122 ceph-mon[93869]: osdmap e1019: 12 total, 12 up, 12 in 2023-03-09T21:25:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:05 smithi181 ceph-mon[94189]: pgmap v12891: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 771 KiB/s wr, 28 op/s 2023-03-09T21:25:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:05 smithi181 ceph-mon[94189]: osdmap e1019: 12 total, 12 up, 12 in 2023-03-09T21:25:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:05 smithi038 ceph-mon[92422]: pgmap v12891: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 771 KiB/s wr, 28 op/s 2023-03-09T21:25:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:05 smithi038 ceph-mon[92422]: osdmap e1019: 12 total, 12 up, 12 in 2023-03-09T21:25:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23216.314982 secs 2023-03-09T21:25:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23216.314982 secs 2023-03-09T21:25:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23216.314982 secs 2023-03-09T21:25:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:07 smithi122 ceph-mon[93869]: pgmap v12893: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 654 KiB/s rd, 652 KiB/s wr, 31 op/s 2023-03-09T21:25:07.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:07 smithi181 ceph-mon[94189]: pgmap v12893: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 654 KiB/s rd, 652 KiB/s wr, 31 op/s 2023-03-09T21:25:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:07 smithi038 ceph-mon[92422]: pgmap v12893: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 654 KiB/s rd, 652 KiB/s wr, 31 op/s 2023-03-09T21:25:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:25:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:25:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23218.653947 secs 2023-03-09T21:25:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:25:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:25:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23218.653947 secs 2023-03-09T21:25:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:25:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:25:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23218.653947 secs 2023-03-09T21:25:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23680.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:25:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:09 smithi122 ceph-mon[93869]: pgmap v12894: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 874 KiB/s rd, 917 KiB/s wr, 32 op/s 2023-03-09T21:25:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:09 smithi181 ceph-mon[94189]: pgmap v12894: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 874 KiB/s rd, 917 KiB/s wr, 32 op/s 2023-03-09T21:25:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:09 smithi038 ceph-mon[92422]: pgmap v12894: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 874 KiB/s rd, 917 KiB/s wr, 32 op/s 2023-03-09T21:25:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:11 smithi122 ceph-mon[93869]: pgmap v12895: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 606 KiB/s rd, 649 KiB/s wr, 31 op/s 2023-03-09T21:25:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23221.315041 secs 2023-03-09T21:25:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:11 smithi181 ceph-mon[94189]: pgmap v12895: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 606 KiB/s rd, 649 KiB/s wr, 31 op/s 2023-03-09T21:25:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23221.315041 secs 2023-03-09T21:25:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:11 smithi038 ceph-mon[92422]: pgmap v12895: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 606 KiB/s rd, 649 KiB/s wr, 31 op/s 2023-03-09T21:25:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23221.315041 secs 2023-03-09T21:25:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:25:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:25:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:25:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:25:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:25:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:25:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:13 smithi122 ceph-mon[93869]: pgmap v12896: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 607 KiB/s rd, 624 KiB/s wr, 28 op/s 2023-03-09T21:25:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:25:13.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:25:13.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23223.654071 secs 2023-03-09T21:25:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:13 smithi181 ceph-mon[94189]: pgmap v12896: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 607 KiB/s rd, 624 KiB/s wr, 28 op/s 2023-03-09T21:25:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:25:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:25:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23223.654071 secs 2023-03-09T21:25:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:13 smithi038 ceph-mon[92422]: pgmap v12896: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 607 KiB/s rd, 624 KiB/s wr, 28 op/s 2023-03-09T21:25:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:25:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:25:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23223.654071 secs 2023-03-09T21:25:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23685.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:25:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:14 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:25:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:14 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:25:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:14 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:25:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:15 smithi122 ceph-mon[93869]: pgmap v12897: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 541 KiB/s rd, 541 KiB/s wr, 13 op/s 2023-03-09T21:25:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:25:15.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:25:15.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:25:15.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:15 smithi122 ceph-mon[93869]: osdmap e1020: 12 total, 12 up, 12 in 2023-03-09T21:25:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:15 smithi181 ceph-mon[94189]: pgmap v12897: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 541 KiB/s rd, 541 KiB/s wr, 13 op/s 2023-03-09T21:25:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:25:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:25:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:25:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:15 smithi181 ceph-mon[94189]: osdmap e1020: 12 total, 12 up, 12 in 2023-03-09T21:25:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:15 smithi038 ceph-mon[92422]: pgmap v12897: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 541 KiB/s rd, 541 KiB/s wr, 13 op/s 2023-03-09T21:25:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:25:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:25:15.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:25:15.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:15 smithi038 ceph-mon[92422]: osdmap e1020: 12 total, 12 up, 12 in 2023-03-09T21:25:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23226.315100 secs 2023-03-09T21:25:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23226.315100 secs 2023-03-09T21:25:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23226.315100 secs 2023-03-09T21:25:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:17 smithi122 ceph-mon[93869]: pgmap v12899: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 541 KiB/s rd, 541 KiB/s wr, 12 op/s 2023-03-09T21:25:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:17 smithi181 ceph-mon[94189]: pgmap v12899: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 541 KiB/s rd, 541 KiB/s wr, 12 op/s 2023-03-09T21:25:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:17 smithi038 ceph-mon[92422]: pgmap v12899: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 541 KiB/s rd, 541 KiB/s wr, 12 op/s 2023-03-09T21:25:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23228.654195 secs 2023-03-09T21:25:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23228.654195 secs 2023-03-09T21:25:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23228.654195 secs 2023-03-09T21:25:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23690.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:25:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:19 smithi122 ceph-mon[93869]: pgmap v12900: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 275 KiB/s wr, 9 op/s 2023-03-09T21:25:19.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:19 smithi181 ceph-mon[94189]: pgmap v12900: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 275 KiB/s wr, 9 op/s 2023-03-09T21:25:19.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:19 smithi038 ceph-mon[92422]: pgmap v12900: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 275 KiB/s wr, 9 op/s 2023-03-09T21:25:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:21 smithi122 ceph-mon[93869]: pgmap v12901: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 186 KiB/s wr, 9 op/s 2023-03-09T21:25:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23231.315159 secs 2023-03-09T21:25:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:21 smithi181 ceph-mon[94189]: pgmap v12901: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 186 KiB/s wr, 9 op/s 2023-03-09T21:25:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23231.315159 secs 2023-03-09T21:25:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:21 smithi038 ceph-mon[92422]: pgmap v12901: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 186 KiB/s wr, 9 op/s 2023-03-09T21:25:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23231.315159 secs 2023-03-09T21:25:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:23 smithi181 ceph-mon[94189]: pgmap v12902: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 185 KiB/s wr, 8 op/s 2023-03-09T21:25:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23233.654300 secs 2023-03-09T21:25:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:23 smithi038 ceph-mon[92422]: pgmap v12902: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 185 KiB/s wr, 8 op/s 2023-03-09T21:25:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23233.654300 secs 2023-03-09T21:25:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:23 smithi122 ceph-mon[93869]: pgmap v12902: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 185 KiB/s wr, 8 op/s 2023-03-09T21:25:23.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23233.654300 secs 2023-03-09T21:25:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23695.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:25:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:25 smithi181 ceph-mon[94189]: pgmap v12903: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 76 KiB/s wr, 7 op/s 2023-03-09T21:25:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:25 smithi038 ceph-mon[92422]: pgmap v12903: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 76 KiB/s wr, 7 op/s 2023-03-09T21:25:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:25 smithi122 ceph-mon[93869]: pgmap v12903: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 76 KiB/s wr, 7 op/s 2023-03-09T21:25:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23236.315220 secs 2023-03-09T21:25:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23236.315220 secs 2023-03-09T21:25:26.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23236.315220 secs 2023-03-09T21:25:27.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:27 smithi038 ceph-mon[92422]: pgmap v12904: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 140 KiB/s wr, 6 op/s 2023-03-09T21:25:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:27 smithi122 ceph-mon[93869]: pgmap v12904: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 140 KiB/s wr, 6 op/s 2023-03-09T21:25:27.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:27 smithi181 ceph-mon[94189]: pgmap v12904: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 140 KiB/s wr, 6 op/s 2023-03-09T21:25:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23238.654415 secs 2023-03-09T21:25:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23238.654415 secs 2023-03-09T21:25:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23238.654415 secs 2023-03-09T21:25:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23700.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:25:29.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:29 smithi122 ceph-mon[93869]: pgmap v12905: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 124 KiB/s wr, 7 op/s 2023-03-09T21:25:29.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:29 smithi181 ceph-mon[94189]: pgmap v12905: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 124 KiB/s wr, 7 op/s 2023-03-09T21:25:29.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:29 smithi038 ceph-mon[92422]: pgmap v12905: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 124 KiB/s wr, 7 op/s 2023-03-09T21:25:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:31 smithi122 ceph-mon[93869]: pgmap v12906: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 197 KiB/s wr, 7 op/s 2023-03-09T21:25:31.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23241.315280 secs 2023-03-09T21:25:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:31 smithi181 ceph-mon[94189]: pgmap v12906: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 197 KiB/s wr, 7 op/s 2023-03-09T21:25:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23241.315280 secs 2023-03-09T21:25:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:31 smithi038 ceph-mon[92422]: pgmap v12906: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 197 KiB/s wr, 7 op/s 2023-03-09T21:25:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23241.315280 secs 2023-03-09T21:25:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:32 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:25:32.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:32 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:25:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:32 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:25:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:32 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:25:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:32 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:25:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:32 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:25:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:33 smithi122 ceph-mon[93869]: pgmap v12907: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 134 KiB/s wr, 8 op/s 2023-03-09T21:25:33.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23243.654532 secs 2023-03-09T21:25:33.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:33 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:25:33.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:33 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:25:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:33 smithi181 ceph-mon[94189]: pgmap v12907: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 134 KiB/s wr, 8 op/s 2023-03-09T21:25:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23243.654532 secs 2023-03-09T21:25:33.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:33 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:25:33.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:33 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:25:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:33 smithi038 ceph-mon[92422]: pgmap v12907: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 134 KiB/s wr, 8 op/s 2023-03-09T21:25:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23243.654532 secs 2023-03-09T21:25:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:33 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:25:33.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:33 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:25:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23705.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:25:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:35 smithi122 ceph-mon[93869]: pgmap v12908: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 134 KiB/s wr, 7 op/s 2023-03-09T21:25:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:35 smithi181 ceph-mon[94189]: pgmap v12908: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 134 KiB/s wr, 7 op/s 2023-03-09T21:25:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:35 smithi038 ceph-mon[92422]: pgmap v12908: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 134 KiB/s wr, 7 op/s 2023-03-09T21:25:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23246.315371 secs 2023-03-09T21:25:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23246.315371 secs 2023-03-09T21:25:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23246.315371 secs 2023-03-09T21:25:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:37 smithi122 ceph-mon[93869]: pgmap v12909: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 184 KiB/s wr, 7 op/s 2023-03-09T21:25:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:37 smithi181 ceph-mon[94189]: pgmap v12909: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 184 KiB/s wr, 7 op/s 2023-03-09T21:25:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:37 smithi038 ceph-mon[92422]: pgmap v12909: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 184 KiB/s wr, 7 op/s 2023-03-09T21:25:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23248.654649 secs 2023-03-09T21:25:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23248.654649 secs 2023-03-09T21:25:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23248.654649 secs 2023-03-09T21:25:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23710.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:25:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:39 smithi122 ceph-mon[93869]: pgmap v12910: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 124 KiB/s wr, 9 op/s 2023-03-09T21:25:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:39 smithi181 ceph-mon[94189]: pgmap v12910: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 124 KiB/s wr, 9 op/s 2023-03-09T21:25:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:39 smithi038 ceph-mon[92422]: pgmap v12910: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 124 KiB/s wr, 9 op/s 2023-03-09T21:25:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:41 smithi122 ceph-mon[93869]: pgmap v12911: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 124 KiB/s wr, 7 op/s 2023-03-09T21:25:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23251.315430 secs 2023-03-09T21:25:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:41 smithi181 ceph-mon[94189]: pgmap v12911: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 124 KiB/s wr, 7 op/s 2023-03-09T21:25:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23251.315430 secs 2023-03-09T21:25:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:41 smithi038 ceph-mon[92422]: pgmap v12911: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 124 KiB/s wr, 7 op/s 2023-03-09T21:25:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23251.315430 secs 2023-03-09T21:25:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:43 smithi122 ceph-mon[93869]: pgmap v12912: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 152 KiB/s wr, 9 op/s 2023-03-09T21:25:43.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23253.654735 secs 2023-03-09T21:25:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:43 smithi181 ceph-mon[94189]: pgmap v12912: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 152 KiB/s wr, 9 op/s 2023-03-09T21:25:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23253.654735 secs 2023-03-09T21:25:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:43 smithi038 ceph-mon[92422]: pgmap v12912: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 152 KiB/s wr, 9 op/s 2023-03-09T21:25:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23253.654735 secs 2023-03-09T21:25:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23715.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:25:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:45 smithi122 ceph-mon[93869]: pgmap v12913: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 152 KiB/s wr, 7 op/s 2023-03-09T21:25:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:45 smithi181 ceph-mon[94189]: pgmap v12913: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 152 KiB/s wr, 7 op/s 2023-03-09T21:25:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:45 smithi038 ceph-mon[92422]: pgmap v12913: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 152 KiB/s wr, 7 op/s 2023-03-09T21:25:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23256.315489 secs 2023-03-09T21:25:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23256.315489 secs 2023-03-09T21:25:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23256.315489 secs 2023-03-09T21:25:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:47 smithi122 ceph-mon[93869]: pgmap v12914: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 282 KiB/s wr, 7 op/s 2023-03-09T21:25:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:47 smithi181 ceph-mon[94189]: pgmap v12914: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 282 KiB/s wr, 7 op/s 2023-03-09T21:25:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:47 smithi038 ceph-mon[92422]: pgmap v12914: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 282 KiB/s wr, 7 op/s 2023-03-09T21:25:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23258.654853 secs 2023-03-09T21:25:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23258.654853 secs 2023-03-09T21:25:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23258.654853 secs 2023-03-09T21:25:48.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23720.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:25:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:49 smithi122 ceph-mon[93869]: pgmap v12915: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 279 KiB/s wr, 8 op/s 2023-03-09T21:25:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:49 smithi181 ceph-mon[94189]: pgmap v12915: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 279 KiB/s wr, 8 op/s 2023-03-09T21:25:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:49 smithi038 ceph-mon[92422]: pgmap v12915: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 279 KiB/s wr, 8 op/s 2023-03-09T21:25:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:51 smithi122 ceph-mon[93869]: pgmap v12916: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 310 KiB/s rd, 297 KiB/s wr, 7 op/s 2023-03-09T21:25:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23261.315551 secs 2023-03-09T21:25:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:51 smithi181 ceph-mon[94189]: pgmap v12916: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 310 KiB/s rd, 297 KiB/s wr, 7 op/s 2023-03-09T21:25:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23261.315551 secs 2023-03-09T21:25:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:51 smithi038 ceph-mon[92422]: pgmap v12916: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 310 KiB/s rd, 297 KiB/s wr, 7 op/s 2023-03-09T21:25:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23261.315551 secs 2023-03-09T21:25:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:25:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:25:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:25:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:25:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:25:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:25:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:53 smithi122 ceph-mon[93869]: pgmap v12917: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 297 KiB/s wr, 9 op/s 2023-03-09T21:25:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:25:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:25:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:53 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:25:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:53 smithi181 ceph-mon[94189]: pgmap v12917: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 297 KiB/s wr, 9 op/s 2023-03-09T21:25:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:25:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:25:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:53 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:25:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:53 smithi038 ceph-mon[92422]: pgmap v12917: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 297 KiB/s wr, 9 op/s 2023-03-09T21:25:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:25:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:25:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:53 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:25:53.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23725.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:25:54.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:54 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:25:54.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:54 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:25:54.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:54 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:25:54.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:54 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:25:54.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:54 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:25:54.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:54 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:25:54.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:54 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:25:54.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:54 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:25:54.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:54 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:25:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:55 smithi122 ceph-mon[93869]: pgmap v12918: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 474 KiB/s wr, 10 op/s 2023-03-09T21:25:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:55 smithi181 ceph-mon[94189]: pgmap v12918: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 474 KiB/s wr, 10 op/s 2023-03-09T21:25:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:55 smithi038 ceph-mon[92422]: pgmap v12918: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 474 KiB/s wr, 10 op/s 2023-03-09T21:25:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23266.315610 secs 2023-03-09T21:25:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23266.315610 secs 2023-03-09T21:25:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23266.315610 secs 2023-03-09T21:25:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:57 smithi122 ceph-mon[93869]: pgmap v12919: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 474 KiB/s wr, 10 op/s 2023-03-09T21:25:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:25:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:25:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:57 smithi181 ceph-mon[94189]: pgmap v12919: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 474 KiB/s wr, 10 op/s 2023-03-09T21:25:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:25:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:25:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:57 smithi038 ceph-mon[92422]: pgmap v12919: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 474 KiB/s wr, 10 op/s 2023-03-09T21:25:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:25:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:25:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23730.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:25:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:59 smithi122 ceph-mon[93869]: pgmap v12920: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 545 KiB/s rd, 531 KiB/s wr, 12 op/s 2023-03-09T21:25:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:25:59.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:25:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:25:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:59 smithi181 ceph-mon[94189]: pgmap v12920: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 545 KiB/s rd, 531 KiB/s wr, 12 op/s 2023-03-09T21:25:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:25:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:25:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:25:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:59 smithi038 ceph-mon[92422]: pgmap v12920: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 545 KiB/s rd, 531 KiB/s wr, 12 op/s 2023-03-09T21:25:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:25:59.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:25:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:26:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:01 smithi122 ceph-mon[93869]: pgmap v12921: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 502 KiB/s wr, 11 op/s 2023-03-09T21:26:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23271.315673 secs 2023-03-09T21:26:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:01 smithi181 ceph-mon[94189]: pgmap v12921: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 502 KiB/s wr, 11 op/s 2023-03-09T21:26:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23271.315673 secs 2023-03-09T21:26:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:01 smithi038 ceph-mon[92422]: pgmap v12921: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 502 KiB/s wr, 11 op/s 2023-03-09T21:26:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23271.315673 secs 2023-03-09T21:26:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:03 smithi122 ceph-mon[93869]: pgmap v12922: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 539 KiB/s rd, 547 KiB/s wr, 14 op/s 2023-03-09T21:26:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23273.655193 secs 2023-03-09T21:26:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:03 smithi181 ceph-mon[94189]: pgmap v12922: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 539 KiB/s rd, 547 KiB/s wr, 14 op/s 2023-03-09T21:26:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23273.655193 secs 2023-03-09T21:26:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:03 smithi038 ceph-mon[92422]: pgmap v12922: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 539 KiB/s rd, 547 KiB/s wr, 14 op/s 2023-03-09T21:26:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23273.655193 secs 2023-03-09T21:26:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23735.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:26:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:05 smithi122 ceph-mon[93869]: pgmap v12923: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 561 KiB/s rd, 587 KiB/s wr, 25 op/s 2023-03-09T21:26:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:05 smithi122 ceph-mon[93869]: osdmap e1021: 12 total, 12 up, 12 in 2023-03-09T21:26:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:05 smithi181 ceph-mon[94189]: pgmap v12923: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 561 KiB/s rd, 587 KiB/s wr, 25 op/s 2023-03-09T21:26:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:05 smithi181 ceph-mon[94189]: osdmap e1021: 12 total, 12 up, 12 in 2023-03-09T21:26:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:05 smithi038 ceph-mon[92422]: pgmap v12923: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 561 KiB/s rd, 587 KiB/s wr, 25 op/s 2023-03-09T21:26:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:05 smithi038 ceph-mon[92422]: osdmap e1021: 12 total, 12 up, 12 in 2023-03-09T21:26:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23276.315735 secs 2023-03-09T21:26:06.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:06 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:26:06.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:06 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:26:06.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:26:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23276.315735 secs 2023-03-09T21:26:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:06 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:26:06.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:06 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:26:06.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:26:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23276.315735 secs 2023-03-09T21:26:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:06 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:26:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:06 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:26:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:26:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:07 smithi122 ceph-mon[93869]: pgmap v12925: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 595 KiB/s rd, 638 KiB/s wr, 31 op/s 2023-03-09T21:26:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:07 smithi181 ceph-mon[94189]: pgmap v12925: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 595 KiB/s rd, 638 KiB/s wr, 31 op/s 2023-03-09T21:26:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:07 smithi038 ceph-mon[92422]: pgmap v12925: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 595 KiB/s rd, 638 KiB/s wr, 31 op/s 2023-03-09T21:26:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23278.655285 secs 2023-03-09T21:26:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23278.655285 secs 2023-03-09T21:26:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23278.655285 secs 2023-03-09T21:26:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23740.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:26:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:09 smithi122 ceph-mon[93869]: pgmap v12926: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 413 KiB/s wr, 28 op/s 2023-03-09T21:26:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:09 smithi181 ceph-mon[94189]: pgmap v12926: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 413 KiB/s wr, 28 op/s 2023-03-09T21:26:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:09 smithi038 ceph-mon[92422]: pgmap v12926: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 413 KiB/s wr, 28 op/s 2023-03-09T21:26:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:11 smithi122 ceph-mon[93869]: pgmap v12927: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 391 KiB/s wr, 28 op/s 2023-03-09T21:26:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23281.315793 secs 2023-03-09T21:26:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:11 smithi181 ceph-mon[94189]: pgmap v12927: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 391 KiB/s wr, 28 op/s 2023-03-09T21:26:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23281.315793 secs 2023-03-09T21:26:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:11 smithi038 ceph-mon[92422]: pgmap v12927: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 391 KiB/s wr, 28 op/s 2023-03-09T21:26:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23281.315793 secs 2023-03-09T21:26:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:13 smithi122 ceph-mon[93869]: pgmap v12928: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 392 KiB/s wr, 26 op/s 2023-03-09T21:26:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23283.655398 secs 2023-03-09T21:26:13.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:26:13.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:26:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:13 smithi181 ceph-mon[94189]: pgmap v12928: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 392 KiB/s wr, 26 op/s 2023-03-09T21:26:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23283.655398 secs 2023-03-09T21:26:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:26:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:26:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:13 smithi038 ceph-mon[92422]: pgmap v12928: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 392 KiB/s wr, 26 op/s 2023-03-09T21:26:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23283.655398 secs 2023-03-09T21:26:13.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:26:13.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:26:13.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23745.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:26:14.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:14 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:26:14.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:14 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:26:14.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:14 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:26:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:15 smithi038 ceph-mon[92422]: pgmap v12929: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 344 KiB/s wr, 11 op/s 2023-03-09T21:26:15.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:15 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:26:15.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:15 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:26:15.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:15 smithi038 ceph-mon[92422]: osdmap e1022: 12 total, 12 up, 12 in 2023-03-09T21:26:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:15 smithi122 ceph-mon[93869]: pgmap v12929: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 344 KiB/s wr, 11 op/s 2023-03-09T21:26:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:15 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:26:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:15 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:26:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:15 smithi122 ceph-mon[93869]: osdmap e1022: 12 total, 12 up, 12 in 2023-03-09T21:26:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:15 smithi181 ceph-mon[94189]: pgmap v12929: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 344 KiB/s wr, 11 op/s 2023-03-09T21:26:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:15 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:26:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:15 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:26:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:15 smithi181 ceph-mon[94189]: osdmap e1022: 12 total, 12 up, 12 in 2023-03-09T21:26:16.622 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23286.315856 secs 2023-03-09T21:26:16.623 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:16 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:26:16.623 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:16 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:26:16.623 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:16 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:26:16.623 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:16 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:26:16.623 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:16 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:26:16.623 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:16 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:26:16.624 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:16 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:26:16.624 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:16 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:26:16.624 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:16 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:26:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23286.315856 secs 2023-03-09T21:26:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:16 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:26:16.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:16 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:26:16.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:16 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:26:16.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:16 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:26:16.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:16 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:26:16.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:16 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:26:16.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:16 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:26:16.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:16 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:26:16.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:16 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:26:16.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23286.315856 secs 2023-03-09T21:26:16.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:16 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:26:16.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:16 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:26:16.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:16 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:26:16.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:16 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:26:16.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:16 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:26:16.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:16 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:26:16.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:16 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:26:16.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:16 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:26:16.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:16 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:26:17.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:17 smithi122 ceph-mon[93869]: pgmap v12931: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 77 KiB/s wr, 9 op/s 2023-03-09T21:26:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:17 smithi181 ceph-mon[94189]: pgmap v12931: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 77 KiB/s wr, 9 op/s 2023-03-09T21:26:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:17 smithi038 ceph-mon[92422]: pgmap v12931: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 77 KiB/s wr, 9 op/s 2023-03-09T21:26:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23750.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:26:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23288.655478 secs 2023-03-09T21:26:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23288.655478 secs 2023-03-09T21:26:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23288.655478 secs 2023-03-09T21:26:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:19 smithi122 ceph-mon[93869]: pgmap v12932: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 77 KiB/s wr, 8 op/s 2023-03-09T21:26:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:19 smithi181 ceph-mon[94189]: pgmap v12932: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 77 KiB/s wr, 8 op/s 2023-03-09T21:26:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:19 smithi038 ceph-mon[92422]: pgmap v12932: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 77 KiB/s wr, 8 op/s 2023-03-09T21:26:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:21 smithi122 ceph-mon[93869]: pgmap v12933: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 77 KiB/s wr, 8 op/s 2023-03-09T21:26:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23291.315918 secs 2023-03-09T21:26:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:21 smithi181 ceph-mon[94189]: pgmap v12933: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 77 KiB/s wr, 8 op/s 2023-03-09T21:26:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23291.315918 secs 2023-03-09T21:26:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:21 smithi038 ceph-mon[92422]: pgmap v12933: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 77 KiB/s wr, 8 op/s 2023-03-09T21:26:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23291.315918 secs 2023-03-09T21:26:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:23 smithi122 ceph-mon[93869]: pgmap v12934: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 921 B/s wr, 8 op/s 2023-03-09T21:26:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23293.655543 secs 2023-03-09T21:26:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:23 smithi181 ceph-mon[94189]: pgmap v12934: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 921 B/s wr, 8 op/s 2023-03-09T21:26:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23293.655543 secs 2023-03-09T21:26:23.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:23 smithi038 ceph-mon[92422]: pgmap v12934: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 921 B/s wr, 8 op/s 2023-03-09T21:26:23.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23293.655543 secs 2023-03-09T21:26:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23755.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:26:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:25 smithi122 ceph-mon[93869]: pgmap v12935: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 716 B/s wr, 7 op/s 2023-03-09T21:26:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:25 smithi181 ceph-mon[94189]: pgmap v12935: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 716 B/s wr, 7 op/s 2023-03-09T21:26:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:25 smithi038 ceph-mon[92422]: pgmap v12935: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 716 B/s wr, 7 op/s 2023-03-09T21:26:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23296.315981 secs 2023-03-09T21:26:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23296.315981 secs 2023-03-09T21:26:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23296.315981 secs 2023-03-09T21:26:27.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:27 smithi122 ceph-mon[93869]: pgmap v12936: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 105 KiB/s wr, 9 op/s 2023-03-09T21:26:27.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:27 smithi181 ceph-mon[94189]: pgmap v12936: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 105 KiB/s wr, 9 op/s 2023-03-09T21:26:27.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:27 smithi038 ceph-mon[92422]: pgmap v12936: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 105 KiB/s wr, 9 op/s 2023-03-09T21:26:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23298.655663 secs 2023-03-09T21:26:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23298.655663 secs 2023-03-09T21:26:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23298.655663 secs 2023-03-09T21:26:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23760.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:26:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:29 smithi122 ceph-mon[93869]: pgmap v12937: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 93 KiB/s wr, 7 op/s 2023-03-09T21:26:29.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:29 smithi181 ceph-mon[94189]: pgmap v12937: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 93 KiB/s wr, 7 op/s 2023-03-09T21:26:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:29 smithi038 ceph-mon[92422]: pgmap v12937: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 93 KiB/s wr, 7 op/s 2023-03-09T21:26:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:31 smithi122 ceph-mon[93869]: pgmap v12938: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 93 KiB/s wr, 6 op/s 2023-03-09T21:26:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23301.316040 secs 2023-03-09T21:26:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:31 smithi181 ceph-mon[94189]: pgmap v12938: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 93 KiB/s wr, 6 op/s 2023-03-09T21:26:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23301.316040 secs 2023-03-09T21:26:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:31 smithi038 ceph-mon[92422]: pgmap v12938: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 93 KiB/s wr, 6 op/s 2023-03-09T21:26:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23301.316040 secs 2023-03-09T21:26:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:33 smithi122 ceph-mon[93869]: pgmap v12939: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 93 KiB/s wr, 8 op/s 2023-03-09T21:26:33.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23303.655784 secs 2023-03-09T21:26:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:33 smithi181 ceph-mon[94189]: pgmap v12939: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 93 KiB/s wr, 8 op/s 2023-03-09T21:26:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23303.655784 secs 2023-03-09T21:26:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:33 smithi038 ceph-mon[92422]: pgmap v12939: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 93 KiB/s wr, 8 op/s 2023-03-09T21:26:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23303.655784 secs 2023-03-09T21:26:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23765.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:26:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:35 smithi122 ceph-mon[93869]: pgmap v12940: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 93 KiB/s wr, 6 op/s 2023-03-09T21:26:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:35 smithi181 ceph-mon[94189]: pgmap v12940: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 93 KiB/s wr, 6 op/s 2023-03-09T21:26:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:35 smithi038 ceph-mon[92422]: pgmap v12940: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 93 KiB/s wr, 6 op/s 2023-03-09T21:26:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23306.316103 secs 2023-03-09T21:26:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23306.316103 secs 2023-03-09T21:26:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23306.316103 secs 2023-03-09T21:26:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:37 smithi122 ceph-mon[93869]: pgmap v12941: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 169 KiB/s wr, 7 op/s 2023-03-09T21:26:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:37 smithi181 ceph-mon[94189]: pgmap v12941: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 169 KiB/s wr, 7 op/s 2023-03-09T21:26:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:37 smithi038 ceph-mon[92422]: pgmap v12941: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 169 KiB/s wr, 7 op/s 2023-03-09T21:26:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23770.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:26:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:39 smithi122 ceph-mon[93869]: pgmap v12942: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 77 KiB/s wr, 6 op/s 2023-03-09T21:26:39.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:39 smithi181 ceph-mon[94189]: pgmap v12942: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 77 KiB/s wr, 6 op/s 2023-03-09T21:26:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:39 smithi038 ceph-mon[92422]: pgmap v12942: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 77 KiB/s wr, 6 op/s 2023-03-09T21:26:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:26:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:26:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:26:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:26:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:26:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:26:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:41 smithi122 ceph-mon[93869]: pgmap v12943: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 152 KiB/s wr, 7 op/s 2023-03-09T21:26:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23311.316164 secs 2023-03-09T21:26:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:26:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:26:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:41 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:26:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:41 smithi181 ceph-mon[94189]: pgmap v12943: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 152 KiB/s wr, 7 op/s 2023-03-09T21:26:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23311.316164 secs 2023-03-09T21:26:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:26:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:26:41.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:41 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:26:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:41 smithi038 ceph-mon[92422]: pgmap v12943: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 152 KiB/s wr, 7 op/s 2023-03-09T21:26:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23311.316164 secs 2023-03-09T21:26:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:26:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:26:41.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:41 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:26:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:42 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:26:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:42 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:26:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:42 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:26:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:42 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:26:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:42 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:26:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:42 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:26:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:42 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:26:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:42 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:26:42.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:42 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:26:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:43 smithi122 ceph-mon[93869]: pgmap v12944: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 208 KiB/s wr, 9 op/s 2023-03-09T21:26:43.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:43 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 23313.656048 secs 2023-03-09T21:26:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:43 smithi181 ceph-mon[94189]: pgmap v12944: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 208 KiB/s wr, 9 op/s 2023-03-09T21:26:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:43 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 23313.656048 secs 2023-03-09T21:26:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:43 smithi038 ceph-mon[92422]: pgmap v12944: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 208 KiB/s wr, 9 op/s 2023-03-09T21:26:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:43 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 23313.656048 secs 2023-03-09T21:26:43.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23775.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:26:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:45 smithi122 ceph-mon[93869]: pgmap v12945: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 393 KiB/s wr, 9 op/s 2023-03-09T21:26:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:45 smithi181 ceph-mon[94189]: pgmap v12945: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 393 KiB/s wr, 9 op/s 2023-03-09T21:26:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:45 smithi038 ceph-mon[92422]: pgmap v12945: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 393 KiB/s wr, 9 op/s 2023-03-09T21:26:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23316.316225 secs 2023-03-09T21:26:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:46 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:26:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:46 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:26:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23316.316225 secs 2023-03-09T21:26:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:46 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:26:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:46 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:26:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23316.316225 secs 2023-03-09T21:26:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:46 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:26:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:46 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:26:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:47 smithi122 ceph-mon[93869]: pgmap v12946: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 405 KiB/s wr, 10 op/s 2023-03-09T21:26:47.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:47 smithi181 ceph-mon[94189]: pgmap v12946: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 405 KiB/s wr, 10 op/s 2023-03-09T21:26:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:47 smithi038 ceph-mon[92422]: pgmap v12946: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 405 KiB/s wr, 10 op/s 2023-03-09T21:26:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:48 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 23318.656142 secs 2023-03-09T21:26:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:48 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 23318.656142 secs 2023-03-09T21:26:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:48 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 23318.656142 secs 2023-03-09T21:26:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23780.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:26:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:49 smithi038 ceph-mon[92422]: pgmap v12947: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 329 KiB/s wr, 10 op/s 2023-03-09T21:26:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:49 smithi122 ceph-mon[93869]: pgmap v12947: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 329 KiB/s wr, 10 op/s 2023-03-09T21:26:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:49 smithi181 ceph-mon[94189]: pgmap v12947: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 329 KiB/s wr, 10 op/s 2023-03-09T21:26:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:50 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:26:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:50 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:26:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:50 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:26:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:51 smithi038 ceph-mon[92422]: pgmap v12948: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 466 KiB/s wr, 11 op/s 2023-03-09T21:26:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23321.316280 secs 2023-03-09T21:26:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:51 smithi122 ceph-mon[93869]: pgmap v12948: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 466 KiB/s wr, 11 op/s 2023-03-09T21:26:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23321.316280 secs 2023-03-09T21:26:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:51 smithi181 ceph-mon[94189]: pgmap v12948: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 466 KiB/s wr, 11 op/s 2023-03-09T21:26:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23321.316280 secs 2023-03-09T21:26:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:26:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:26:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:26:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:26:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:26:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:26:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:53 smithi122 ceph-mon[93869]: pgmap v12949: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 559 KiB/s rd, 577 KiB/s wr, 12 op/s 2023-03-09T21:26:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:53 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 23323.656254 secs 2023-03-09T21:26:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:26:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:26:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:53 smithi181 ceph-mon[94189]: pgmap v12949: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 559 KiB/s rd, 577 KiB/s wr, 12 op/s 2023-03-09T21:26:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:53 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 23323.656254 secs 2023-03-09T21:26:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:26:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:26:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:53 smithi038 ceph-mon[92422]: pgmap v12949: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 559 KiB/s rd, 577 KiB/s wr, 12 op/s 2023-03-09T21:26:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:53 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 23323.656254 secs 2023-03-09T21:26:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:26:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:26:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23785.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:26:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:55 smithi122 ceph-mon[93869]: pgmap v12950: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 522 KiB/s wr, 11 op/s 2023-03-09T21:26:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:55 smithi181 ceph-mon[94189]: pgmap v12950: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 522 KiB/s wr, 11 op/s 2023-03-09T21:26:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:55 smithi038 ceph-mon[92422]: pgmap v12950: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 522 KiB/s wr, 11 op/s 2023-03-09T21:26:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23326.316341 secs 2023-03-09T21:26:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23326.316341 secs 2023-03-09T21:26:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23326.316341 secs 2023-03-09T21:26:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:57 smithi122 ceph-mon[93869]: pgmap v12951: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 493 KiB/s rd, 480 KiB/s wr, 10 op/s 2023-03-09T21:26:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:57 smithi181 ceph-mon[94189]: pgmap v12951: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 493 KiB/s rd, 480 KiB/s wr, 10 op/s 2023-03-09T21:26:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:57 smithi038 ceph-mon[92422]: pgmap v12951: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 493 KiB/s rd, 480 KiB/s wr, 10 op/s 2023-03-09T21:26:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23328.656379 secs 2023-03-09T21:26:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23328.656379 secs 2023-03-09T21:26:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23328.656379 secs 2023-03-09T21:26:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23790.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:26:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:26:59 smithi122 ceph-mon[93869]: pgmap v12952: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 554 KiB/s rd, 552 KiB/s wr, 11 op/s 2023-03-09T21:26:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:26:59 smithi181 ceph-mon[94189]: pgmap v12952: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 554 KiB/s rd, 552 KiB/s wr, 11 op/s 2023-03-09T21:26:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:26:59 smithi038 ceph-mon[92422]: pgmap v12952: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 554 KiB/s rd, 552 KiB/s wr, 11 op/s 2023-03-09T21:27:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:27:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:27:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:27:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:27:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:27:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:27:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:01 smithi122 ceph-mon[93869]: pgmap v12953: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 570 KiB/s wr, 10 op/s 2023-03-09T21:27:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23331.316414 secs 2023-03-09T21:27:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:27:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:27:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:01 smithi181 ceph-mon[94189]: pgmap v12953: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 570 KiB/s wr, 10 op/s 2023-03-09T21:27:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23331.316414 secs 2023-03-09T21:27:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:27:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:27:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:01 smithi038 ceph-mon[92422]: pgmap v12953: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 570 KiB/s wr, 10 op/s 2023-03-09T21:27:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23331.316414 secs 2023-03-09T21:27:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:27:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:27:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:02 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:27:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:02 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:27:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:02 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:27:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:02 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:27:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:02 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:27:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:02 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:27:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:03 smithi122 ceph-mon[93869]: pgmap v12954: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 546 KiB/s wr, 12 op/s 2023-03-09T21:27:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23333.656497 secs 2023-03-09T21:27:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:03 smithi181 ceph-mon[94189]: pgmap v12954: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 546 KiB/s wr, 12 op/s 2023-03-09T21:27:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23333.656497 secs 2023-03-09T21:27:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:03 smithi038 ceph-mon[92422]: pgmap v12954: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 546 KiB/s wr, 12 op/s 2023-03-09T21:27:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23333.656497 secs 2023-03-09T21:27:03.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23795.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:27:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:05 smithi122 ceph-mon[93869]: pgmap v12955: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 335 KiB/s rd, 400 KiB/s wr, 22 op/s 2023-03-09T21:27:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:05 smithi122 ceph-mon[93869]: osdmap e1023: 12 total, 12 up, 12 in 2023-03-09T21:27:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:05 smithi181 ceph-mon[94189]: pgmap v12955: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 335 KiB/s rd, 400 KiB/s wr, 22 op/s 2023-03-09T21:27:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:05 smithi181 ceph-mon[94189]: osdmap e1023: 12 total, 12 up, 12 in 2023-03-09T21:27:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:05 smithi038 ceph-mon[92422]: pgmap v12955: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 335 KiB/s rd, 400 KiB/s wr, 22 op/s 2023-03-09T21:27:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:05 smithi038 ceph-mon[92422]: osdmap e1023: 12 total, 12 up, 12 in 2023-03-09T21:27:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23336.316476 secs 2023-03-09T21:27:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23336.316476 secs 2023-03-09T21:27:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23336.316476 secs 2023-03-09T21:27:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:07 smithi122 ceph-mon[93869]: pgmap v12957: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 334 KiB/s wr, 27 op/s 2023-03-09T21:27:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:07 smithi181 ceph-mon[94189]: pgmap v12957: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 334 KiB/s wr, 27 op/s 2023-03-09T21:27:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:07 smithi038 ceph-mon[92422]: pgmap v12957: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 334 KiB/s wr, 27 op/s 2023-03-09T21:27:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23338.656620 secs 2023-03-09T21:27:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23338.656620 secs 2023-03-09T21:27:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23338.656620 secs 2023-03-09T21:27:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23800.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:27:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:09 smithi122 ceph-mon[93869]: pgmap v12958: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 445 KiB/s rd, 486 KiB/s wr, 28 op/s 2023-03-09T21:27:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:09 smithi181 ceph-mon[94189]: pgmap v12958: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 445 KiB/s rd, 486 KiB/s wr, 28 op/s 2023-03-09T21:27:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:09 smithi038 ceph-mon[92422]: pgmap v12958: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 445 KiB/s rd, 486 KiB/s wr, 28 op/s 2023-03-09T21:27:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:11 smithi122 ceph-mon[93869]: pgmap v12959: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 464 KiB/s wr, 28 op/s 2023-03-09T21:27:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23341.316557 secs 2023-03-09T21:27:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:11 smithi181 ceph-mon[94189]: pgmap v12959: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 464 KiB/s wr, 28 op/s 2023-03-09T21:27:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23341.316557 secs 2023-03-09T21:27:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:11 smithi038 ceph-mon[92422]: pgmap v12959: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 464 KiB/s wr, 28 op/s 2023-03-09T21:27:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23341.316557 secs 2023-03-09T21:27:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:27:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:27:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:27:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:27:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:27:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:27:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:13 smithi122 ceph-mon[93869]: pgmap v12960: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 329 KiB/s wr, 27 op/s 2023-03-09T21:27:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23343.656739 secs 2023-03-09T21:27:13.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:27:13.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:27:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:13 smithi181 ceph-mon[94189]: pgmap v12960: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 329 KiB/s wr, 27 op/s 2023-03-09T21:27:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23343.656739 secs 2023-03-09T21:27:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:27:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:27:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:13 smithi038 ceph-mon[92422]: pgmap v12960: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 329 KiB/s wr, 27 op/s 2023-03-09T21:27:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23343.656739 secs 2023-03-09T21:27:13.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:27:13.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:27:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23805.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:27:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:15 smithi122 ceph-mon[93869]: pgmap v12961: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 379 KiB/s wr, 13 op/s 2023-03-09T21:27:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:15 smithi122 ceph-mon[93869]: osdmap e1024: 12 total, 12 up, 12 in 2023-03-09T21:27:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:15 smithi181 ceph-mon[94189]: pgmap v12961: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 379 KiB/s wr, 13 op/s 2023-03-09T21:27:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:15 smithi181 ceph-mon[94189]: osdmap e1024: 12 total, 12 up, 12 in 2023-03-09T21:27:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:15 smithi038 ceph-mon[92422]: pgmap v12961: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 379 KiB/s wr, 13 op/s 2023-03-09T21:27:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:15 smithi038 ceph-mon[92422]: osdmap e1024: 12 total, 12 up, 12 in 2023-03-09T21:27:16.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23346.316626 secs 2023-03-09T21:27:16.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:16 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:27:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23346.316626 secs 2023-03-09T21:27:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:16 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:27:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23346.316626 secs 2023-03-09T21:27:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:16 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:27:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:17 smithi122 ceph-mon[93869]: pgmap v12963: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 353 KiB/s wr, 10 op/s 2023-03-09T21:27:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:17 smithi181 ceph-mon[94189]: pgmap v12963: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 353 KiB/s wr, 10 op/s 2023-03-09T21:27:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:17 smithi038 ceph-mon[92422]: pgmap v12963: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 353 KiB/s wr, 10 op/s 2023-03-09T21:27:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23348.656836 secs 2023-03-09T21:27:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23348.656836 secs 2023-03-09T21:27:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23348.656836 secs 2023-03-09T21:27:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23810.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:27:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:19 smithi122 ceph-mon[93869]: pgmap v12964: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 161 KiB/s wr, 9 op/s 2023-03-09T21:27:19.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:27:19.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:27:19.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:27:19.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:27:19.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:27:19.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:27:19.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:27:19.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:27:19.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:27:19.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:27:19.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:27:19.980 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:27:19.980 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:27:19.980 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:27:19.980 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:27:19.981 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:27:19.981 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:27:19.981 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:27:19.982 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:27:19.982 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:27:19.982 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:27:19.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:19 smithi181 ceph-mon[94189]: pgmap v12964: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 161 KiB/s wr, 9 op/s 2023-03-09T21:27:19.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:27:19.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:27:19.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:27:19.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:27:19.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:27:19.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:27:19.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:27:19.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:27:19.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:27:19.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:27:19.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:27:19.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:27:19.986 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:27:19.986 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:27:19.986 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:27:19.986 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:27:19.986 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:27:19.986 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:27:19.986 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:27:19.987 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:27:19.987 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:27:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:19 smithi038 ceph-mon[92422]: pgmap v12964: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 161 KiB/s wr, 9 op/s 2023-03-09T21:27:20.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:27:20.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:27:20.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:27:20.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:27:20.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:27:20.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:27:20.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:27:20.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:27:20.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:27:20.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:27:20.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:27:20.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:27:20.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:27:20.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:27:20.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:27:20.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:27:20.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:27:20.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:27:20.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:27:20.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:27:20.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:27:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:20 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T21:27:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:20 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T21:27:20.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:20 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi038 to 51557k 2023-03-09T21:27:20.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:20 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi038 to 52795130: error parsing value: Value '52795130' is below minimum 939524096 2023-03-09T21:27:20.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:20 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T21:27:20.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:20 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T21:27:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:20 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T21:27:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:20 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T21:27:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:20 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi038 to 51557k 2023-03-09T21:27:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:20 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi038 to 52795130: error parsing value: Value '52795130' is below minimum 939524096 2023-03-09T21:27:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:20 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T21:27:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:20 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T21:27:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:20 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T21:27:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:20 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T21:27:21.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:20 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi038 to 51557k 2023-03-09T21:27:21.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:20 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi038 to 52795130: error parsing value: Value '52795130' is below minimum 939524096 2023-03-09T21:27:21.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:20 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T21:27:21.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:20 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T21:27:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:21 smithi122 ceph-mon[93869]: pgmap v12965: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 182 KiB/s wr, 8 op/s 2023-03-09T21:27:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23351.316686 secs 2023-03-09T21:27:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:21 smithi181 ceph-mon[94189]: pgmap v12965: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 182 KiB/s wr, 8 op/s 2023-03-09T21:27:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23351.316686 secs 2023-03-09T21:27:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:21 smithi038 ceph-mon[92422]: pgmap v12965: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 182 KiB/s wr, 8 op/s 2023-03-09T21:27:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23351.316686 secs 2023-03-09T21:27:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23815.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:27:23.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:23 smithi122 ceph-mon[93869]: pgmap v12966: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 182 KiB/s wr, 8 op/s 2023-03-09T21:27:23.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23353.656938 secs 2023-03-09T21:27:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:23 smithi181 ceph-mon[94189]: pgmap v12966: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 182 KiB/s wr, 8 op/s 2023-03-09T21:27:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23353.656938 secs 2023-03-09T21:27:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:23 smithi038 ceph-mon[92422]: pgmap v12966: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 182 KiB/s wr, 8 op/s 2023-03-09T21:27:24.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23353.656938 secs 2023-03-09T21:27:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:25 smithi122 ceph-mon[93869]: pgmap v12967: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 85 KiB/s wr, 8 op/s 2023-03-09T21:27:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:25 smithi181 ceph-mon[94189]: pgmap v12967: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 85 KiB/s wr, 8 op/s 2023-03-09T21:27:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:25 smithi038 ceph-mon[92422]: pgmap v12967: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 85 KiB/s wr, 8 op/s 2023-03-09T21:27:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23356.316748 secs 2023-03-09T21:27:26.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23356.316748 secs 2023-03-09T21:27:26.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23356.316748 secs 2023-03-09T21:27:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:27 smithi122 ceph-mon[93869]: pgmap v12968: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 81 KiB/s wr, 9 op/s 2023-03-09T21:27:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:27 smithi181 ceph-mon[94189]: pgmap v12968: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 81 KiB/s wr, 9 op/s 2023-03-09T21:27:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:27 smithi038 ceph-mon[92422]: pgmap v12968: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 81 KiB/s wr, 9 op/s 2023-03-09T21:27:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23820.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:27:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23358.657058 secs 2023-03-09T21:27:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23358.657058 secs 2023-03-09T21:27:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23358.657058 secs 2023-03-09T21:27:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:29 smithi122 ceph-mon[93869]: pgmap v12969: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 164 KiB/s wr, 8 op/s 2023-03-09T21:27:29.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:29 smithi181 ceph-mon[94189]: pgmap v12969: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 164 KiB/s wr, 8 op/s 2023-03-09T21:27:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:29 smithi038 ceph-mon[92422]: pgmap v12969: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 164 KiB/s wr, 8 op/s 2023-03-09T21:27:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:31 smithi122 ceph-mon[93869]: pgmap v12970: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 113 KiB/s wr, 7 op/s 2023-03-09T21:27:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23361.316807 secs 2023-03-09T21:27:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:31 smithi181 ceph-mon[94189]: pgmap v12970: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 113 KiB/s wr, 7 op/s 2023-03-09T21:27:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23361.316807 secs 2023-03-09T21:27:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:31 smithi038 ceph-mon[92422]: pgmap v12970: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 113 KiB/s wr, 7 op/s 2023-03-09T21:27:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23361.316807 secs 2023-03-09T21:27:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23825.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:27:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:33 smithi122 ceph-mon[93869]: pgmap v12971: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 94 KiB/s wr, 8 op/s 2023-03-09T21:27:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23363.657165 secs 2023-03-09T21:27:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:33 smithi181 ceph-mon[94189]: pgmap v12971: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 94 KiB/s wr, 8 op/s 2023-03-09T21:27:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23363.657165 secs 2023-03-09T21:27:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:33 smithi038 ceph-mon[92422]: pgmap v12971: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 94 KiB/s wr, 8 op/s 2023-03-09T21:27:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23363.657165 secs 2023-03-09T21:27:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:35 smithi122 ceph-mon[93869]: pgmap v12972: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 182 KiB/s wr, 8 op/s 2023-03-09T21:27:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:35 smithi181 ceph-mon[94189]: pgmap v12972: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 182 KiB/s wr, 8 op/s 2023-03-09T21:27:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:35 smithi038 ceph-mon[92422]: pgmap v12972: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 182 KiB/s wr, 8 op/s 2023-03-09T21:27:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23366.316867 secs 2023-03-09T21:27:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23366.316867 secs 2023-03-09T21:27:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23366.316867 secs 2023-03-09T21:27:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:37 smithi122 ceph-mon[93869]: pgmap v12973: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 182 KiB/s wr, 8 op/s 2023-03-09T21:27:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:37 smithi181 ceph-mon[94189]: pgmap v12973: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 182 KiB/s wr, 8 op/s 2023-03-09T21:27:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:37 smithi038 ceph-mon[92422]: pgmap v12973: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 182 KiB/s wr, 8 op/s 2023-03-09T21:27:38.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23830.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:27:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23368.657226 secs 2023-03-09T21:27:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23368.657226 secs 2023-03-09T21:27:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23368.657226 secs 2023-03-09T21:27:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:39 smithi122 ceph-mon[93869]: pgmap v12974: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 181 KiB/s wr, 8 op/s 2023-03-09T21:27:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:39 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:27:39.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:39 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:27:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:39 smithi181 ceph-mon[94189]: pgmap v12974: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 181 KiB/s wr, 8 op/s 2023-03-09T21:27:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:39 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:27:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:39 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:27:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:39 smithi038 ceph-mon[92422]: pgmap v12974: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 181 KiB/s wr, 8 op/s 2023-03-09T21:27:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:39 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:27:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:39 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:27:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:41 smithi122 ceph-mon[93869]: pgmap v12975: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 88 KiB/s wr, 7 op/s 2023-03-09T21:27:41.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23371.316928 secs 2023-03-09T21:27:41.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:27:41.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:27:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:41 smithi181 ceph-mon[94189]: pgmap v12975: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 88 KiB/s wr, 7 op/s 2023-03-09T21:27:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23371.316928 secs 2023-03-09T21:27:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:27:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:27:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:41 smithi038 ceph-mon[92422]: pgmap v12975: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 88 KiB/s wr, 7 op/s 2023-03-09T21:27:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23371.316928 secs 2023-03-09T21:27:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:27:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:27:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23835.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:27:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:43 smithi122 ceph-mon[93869]: pgmap v12976: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 93 KiB/s wr, 9 op/s 2023-03-09T21:27:43.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23373.657338 secs 2023-03-09T21:27:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:43 smithi181 ceph-mon[94189]: pgmap v12976: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 93 KiB/s wr, 9 op/s 2023-03-09T21:27:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23373.657338 secs 2023-03-09T21:27:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:43 smithi038 ceph-mon[92422]: pgmap v12976: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 93 KiB/s wr, 9 op/s 2023-03-09T21:27:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23373.657338 secs 2023-03-09T21:27:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:45 smithi122 ceph-mon[93869]: pgmap v12977: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 264 KiB/s wr, 9 op/s 2023-03-09T21:27:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:45 smithi181 ceph-mon[94189]: pgmap v12977: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 264 KiB/s wr, 9 op/s 2023-03-09T21:27:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:45 smithi038 ceph-mon[92422]: pgmap v12977: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 264 KiB/s wr, 9 op/s 2023-03-09T21:27:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23376.317008 secs 2023-03-09T21:27:46.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23376.317008 secs 2023-03-09T21:27:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23376.317008 secs 2023-03-09T21:27:47.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:47 smithi122 ceph-mon[93869]: pgmap v12978: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 296 KiB/s wr, 10 op/s 2023-03-09T21:27:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:47 smithi181 ceph-mon[94189]: pgmap v12978: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 296 KiB/s wr, 10 op/s 2023-03-09T21:27:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:47 smithi038 ceph-mon[92422]: pgmap v12978: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 296 KiB/s wr, 10 op/s 2023-03-09T21:27:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23840.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:27:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23378.657410 secs 2023-03-09T21:27:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23378.657410 secs 2023-03-09T21:27:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23378.657410 secs 2023-03-09T21:27:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:49 smithi122 ceph-mon[93869]: pgmap v12979: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 462 KiB/s wr, 12 op/s 2023-03-09T21:27:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:49 smithi181 ceph-mon[94189]: pgmap v12979: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 462 KiB/s wr, 12 op/s 2023-03-09T21:27:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:49 smithi038 ceph-mon[92422]: pgmap v12979: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 462 KiB/s wr, 12 op/s 2023-03-09T21:27:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:51 smithi122 ceph-mon[93869]: pgmap v12980: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 462 KiB/s wr, 11 op/s 2023-03-09T21:27:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23381.317068 secs 2023-03-09T21:27:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:27:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:27:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:51 smithi181 ceph-mon[94189]: pgmap v12980: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 462 KiB/s wr, 11 op/s 2023-03-09T21:27:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23381.317068 secs 2023-03-09T21:27:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:27:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:27:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:51 smithi038 ceph-mon[92422]: pgmap v12980: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 462 KiB/s wr, 11 op/s 2023-03-09T21:27:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23381.317068 secs 2023-03-09T21:27:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:27:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:27:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:53 smithi122 ceph-mon[93869]: pgmap v12981: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 736 KiB/s rd, 721 KiB/s wr, 14 op/s 2023-03-09T21:27:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:27:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:27:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23383.657518 secs 2023-03-09T21:27:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:53 smithi181 ceph-mon[94189]: pgmap v12981: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 736 KiB/s rd, 721 KiB/s wr, 14 op/s 2023-03-09T21:27:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:27:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:27:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23383.657518 secs 2023-03-09T21:27:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:53 smithi038 ceph-mon[92422]: pgmap v12981: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 736 KiB/s rd, 721 KiB/s wr, 14 op/s 2023-03-09T21:27:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:27:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:27:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23383.657518 secs 2023-03-09T21:27:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23845.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:27:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:55 smithi122 ceph-mon[93869]: pgmap v12982: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 731 KiB/s rd, 717 KiB/s wr, 13 op/s 2023-03-09T21:27:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:55 smithi181 ceph-mon[94189]: pgmap v12982: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 731 KiB/s rd, 717 KiB/s wr, 13 op/s 2023-03-09T21:27:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:55 smithi038 ceph-mon[92422]: pgmap v12982: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 731 KiB/s rd, 717 KiB/s wr, 13 op/s 2023-03-09T21:27:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23386.317128 secs 2023-03-09T21:27:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23386.317128 secs 2023-03-09T21:27:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23386.317128 secs 2023-03-09T21:27:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:57 smithi122 ceph-mon[93869]: pgmap v12983: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 896 KiB/s rd, 564 KiB/s wr, 14 op/s 2023-03-09T21:27:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:57 smithi181 ceph-mon[94189]: pgmap v12983: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 896 KiB/s rd, 564 KiB/s wr, 14 op/s 2023-03-09T21:27:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:57 smithi038 ceph-mon[92422]: pgmap v12983: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 896 KiB/s rd, 564 KiB/s wr, 14 op/s 2023-03-09T21:27:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23388.657618 secs 2023-03-09T21:27:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23388.657618 secs 2023-03-09T21:27:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23388.657618 secs 2023-03-09T21:27:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23850.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:27:59.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:27:59 smithi122 ceph-mon[93869]: pgmap v12984: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 875 KiB/s rd, 556 KiB/s wr, 13 op/s 2023-03-09T21:27:59.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:27:59 smithi181 ceph-mon[94189]: pgmap v12984: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 875 KiB/s rd, 556 KiB/s wr, 13 op/s 2023-03-09T21:27:59.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:27:59 smithi038 ceph-mon[92422]: pgmap v12984: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 875 KiB/s rd, 556 KiB/s wr, 13 op/s 2023-03-09T21:28:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:28:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:28:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:28:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:28:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:28:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:28:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:01 smithi122 ceph-mon[93869]: pgmap v12985: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 709 KiB/s rd, 391 KiB/s wr, 11 op/s 2023-03-09T21:28:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23391.317192 secs 2023-03-09T21:28:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:28:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:28:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:01 smithi181 ceph-mon[94189]: pgmap v12985: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 709 KiB/s rd, 391 KiB/s wr, 11 op/s 2023-03-09T21:28:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23391.317192 secs 2023-03-09T21:28:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:28:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:28:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:01 smithi038 ceph-mon[92422]: pgmap v12985: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 709 KiB/s rd, 391 KiB/s wr, 11 op/s 2023-03-09T21:28:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23391.317192 secs 2023-03-09T21:28:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:28:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:28:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:03 smithi122 ceph-mon[93869]: pgmap v12986: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 958 KiB/s rd, 638 KiB/s wr, 14 op/s 2023-03-09T21:28:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23393.657745 secs 2023-03-09T21:28:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:03 smithi181 ceph-mon[94189]: pgmap v12986: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 958 KiB/s rd, 638 KiB/s wr, 14 op/s 2023-03-09T21:28:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23393.657745 secs 2023-03-09T21:28:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:03 smithi038 ceph-mon[92422]: pgmap v12986: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 958 KiB/s rd, 638 KiB/s wr, 14 op/s 2023-03-09T21:28:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23393.657745 secs 2023-03-09T21:28:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23855.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:28:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:05 smithi122 ceph-mon[93869]: pgmap v12987: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 792 KiB/s rd, 611 KiB/s wr, 23 op/s 2023-03-09T21:28:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:05 smithi122 ceph-mon[93869]: osdmap e1025: 12 total, 12 up, 12 in 2023-03-09T21:28:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:05 smithi181 ceph-mon[94189]: pgmap v12987: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 792 KiB/s rd, 611 KiB/s wr, 23 op/s 2023-03-09T21:28:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:05 smithi181 ceph-mon[94189]: osdmap e1025: 12 total, 12 up, 12 in 2023-03-09T21:28:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:05 smithi038 ceph-mon[92422]: pgmap v12987: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 792 KiB/s rd, 611 KiB/s wr, 23 op/s 2023-03-09T21:28:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:05 smithi038 ceph-mon[92422]: osdmap e1025: 12 total, 12 up, 12 in 2023-03-09T21:28:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23396.317255 secs 2023-03-09T21:28:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23396.317255 secs 2023-03-09T21:28:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23396.317255 secs 2023-03-09T21:28:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:07 smithi122 ceph-mon[93869]: pgmap v12989: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 549 KiB/s rd, 737 KiB/s wr, 28 op/s 2023-03-09T21:28:07.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:07 smithi181 ceph-mon[94189]: pgmap v12989: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 549 KiB/s rd, 737 KiB/s wr, 28 op/s 2023-03-09T21:28:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:07 smithi038 ceph-mon[92422]: pgmap v12989: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 549 KiB/s rd, 737 KiB/s wr, 28 op/s 2023-03-09T21:28:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:28:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:28:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23398.657836 secs 2023-03-09T21:28:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:28:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:28:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23398.657836 secs 2023-03-09T21:28:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:28:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:28:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23398.657836 secs 2023-03-09T21:28:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23860.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:28:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:09 smithi122 ceph-mon[93869]: pgmap v12990: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 603 KiB/s wr, 27 op/s 2023-03-09T21:28:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:28:09.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:28:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:09 smithi181 ceph-mon[94189]: pgmap v12990: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 603 KiB/s wr, 27 op/s 2023-03-09T21:28:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:28:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:28:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:09 smithi038 ceph-mon[92422]: pgmap v12990: 129 pgs: 1 active+clean+snaptrim, 128 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 603 KiB/s wr, 27 op/s 2023-03-09T21:28:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:28:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:28:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:11 smithi122 ceph-mon[93869]: pgmap v12991: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 668 KiB/s rd, 709 KiB/s wr, 28 op/s 2023-03-09T21:28:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23401.317316 secs 2023-03-09T21:28:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:11 smithi181 ceph-mon[94189]: pgmap v12991: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 668 KiB/s rd, 709 KiB/s wr, 28 op/s 2023-03-09T21:28:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23401.317316 secs 2023-03-09T21:28:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:11 smithi038 ceph-mon[92422]: pgmap v12991: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 668 KiB/s rd, 709 KiB/s wr, 28 op/s 2023-03-09T21:28:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23401.317316 secs 2023-03-09T21:28:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:13 smithi122 ceph-mon[93869]: pgmap v12992: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 435 KiB/s wr, 27 op/s 2023-03-09T21:28:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23403.657955 secs 2023-03-09T21:28:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:13 smithi181 ceph-mon[94189]: pgmap v12992: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 435 KiB/s wr, 27 op/s 2023-03-09T21:28:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23403.657955 secs 2023-03-09T21:28:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:13 smithi038 ceph-mon[92422]: pgmap v12992: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 435 KiB/s wr, 27 op/s 2023-03-09T21:28:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23403.657955 secs 2023-03-09T21:28:13.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23865.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:28:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:15 smithi122 ceph-mon[93869]: pgmap v12993: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 210 KiB/s wr, 12 op/s 2023-03-09T21:28:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:15 smithi122 ceph-mon[93869]: osdmap e1026: 12 total, 12 up, 12 in 2023-03-09T21:28:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:15 smithi181 ceph-mon[94189]: pgmap v12993: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 210 KiB/s wr, 12 op/s 2023-03-09T21:28:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:15 smithi181 ceph-mon[94189]: osdmap e1026: 12 total, 12 up, 12 in 2023-03-09T21:28:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:15 smithi038 ceph-mon[92422]: pgmap v12993: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 210 KiB/s wr, 12 op/s 2023-03-09T21:28:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:15 smithi038 ceph-mon[92422]: osdmap e1026: 12 total, 12 up, 12 in 2023-03-09T21:28:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23406.317376 secs 2023-03-09T21:28:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23406.317376 secs 2023-03-09T21:28:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23406.317376 secs 2023-03-09T21:28:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:17 smithi038 ceph-mon[92422]: pgmap v12995: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 266 KiB/s wr, 9 op/s 2023-03-09T21:28:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:17 smithi122 ceph-mon[93869]: pgmap v12995: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 266 KiB/s wr, 9 op/s 2023-03-09T21:28:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:17 smithi181 ceph-mon[94189]: pgmap v12995: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 266 KiB/s wr, 9 op/s 2023-03-09T21:28:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23408.658074 secs 2023-03-09T21:28:18.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23408.658074 secs 2023-03-09T21:28:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23408.658074 secs 2023-03-09T21:28:18.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23870.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:28:19.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:19 smithi038 ceph-mon[92422]: pgmap v12996: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 266 KiB/s wr, 9 op/s 2023-03-09T21:28:19.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:19 smithi122 ceph-mon[93869]: pgmap v12996: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 266 KiB/s wr, 9 op/s 2023-03-09T21:28:19.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:19 smithi181 ceph-mon[94189]: pgmap v12996: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 266 KiB/s wr, 9 op/s 2023-03-09T21:28:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:28:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:28:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:28:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:28:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:28:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:28:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:28:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:28:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:28:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:28:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:28:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:28:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:28:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:28:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:28:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:28:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:28:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:28:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:21 smithi122 ceph-mon[93869]: pgmap v12997: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 160 KiB/s wr, 8 op/s 2023-03-09T21:28:21.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23411.317439 secs 2023-03-09T21:28:21.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:28:21.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:28:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:21 smithi181 ceph-mon[94189]: pgmap v12997: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 160 KiB/s wr, 8 op/s 2023-03-09T21:28:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23411.317439 secs 2023-03-09T21:28:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:28:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:28:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:21 smithi038 ceph-mon[92422]: pgmap v12997: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 160 KiB/s wr, 8 op/s 2023-03-09T21:28:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23411.317439 secs 2023-03-09T21:28:21.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:28:21.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:28:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:23 smithi122 ceph-mon[93869]: pgmap v12998: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 137 KiB/s wr, 8 op/s 2023-03-09T21:28:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23413.658168 secs 2023-03-09T21:28:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:23 smithi181 ceph-mon[94189]: pgmap v12998: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 137 KiB/s wr, 8 op/s 2023-03-09T21:28:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23413.658168 secs 2023-03-09T21:28:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:23 smithi038 ceph-mon[92422]: pgmap v12998: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 137 KiB/s wr, 8 op/s 2023-03-09T21:28:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23413.658168 secs 2023-03-09T21:28:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23875.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:28:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:25 smithi122 ceph-mon[93869]: pgmap v12999: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 84 KiB/s wr, 8 op/s 2023-03-09T21:28:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:25 smithi181 ceph-mon[94189]: pgmap v12999: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 84 KiB/s wr, 8 op/s 2023-03-09T21:28:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:25 smithi038 ceph-mon[92422]: pgmap v12999: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 84 KiB/s wr, 8 op/s 2023-03-09T21:28:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23416.317500 secs 2023-03-09T21:28:26.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23416.317500 secs 2023-03-09T21:28:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23416.317500 secs 2023-03-09T21:28:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:27 smithi122 ceph-mon[93869]: pgmap v13000: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 80 KiB/s wr, 11 op/s 2023-03-09T21:28:27.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:27 smithi181 ceph-mon[94189]: pgmap v13000: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 80 KiB/s wr, 11 op/s 2023-03-09T21:28:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:27 smithi038 ceph-mon[92422]: pgmap v13000: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 80 KiB/s wr, 11 op/s 2023-03-09T21:28:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23418.658323 secs 2023-03-09T21:28:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23418.658323 secs 2023-03-09T21:28:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23418.658323 secs 2023-03-09T21:28:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23880.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:28:29.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:29 smithi122 ceph-mon[93869]: pgmap v13001: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2023-03-09T21:28:29.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:29 smithi181 ceph-mon[94189]: pgmap v13001: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2023-03-09T21:28:29.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:29 smithi038 ceph-mon[92422]: pgmap v13001: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2023-03-09T21:28:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:31 smithi122 ceph-mon[93869]: pgmap v13002: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2023-03-09T21:28:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23421.317562 secs 2023-03-09T21:28:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:31 smithi181 ceph-mon[94189]: pgmap v13002: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2023-03-09T21:28:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23421.317562 secs 2023-03-09T21:28:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:31 smithi038 ceph-mon[92422]: pgmap v13002: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2023-03-09T21:28:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23421.317562 secs 2023-03-09T21:28:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:33 smithi122 ceph-mon[93869]: pgmap v13003: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 95 KiB/s wr, 9 op/s 2023-03-09T21:28:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23423.658447 secs 2023-03-09T21:28:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:33 smithi181 ceph-mon[94189]: pgmap v13003: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 95 KiB/s wr, 9 op/s 2023-03-09T21:28:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23423.658447 secs 2023-03-09T21:28:33.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:33 smithi038 ceph-mon[92422]: pgmap v13003: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 95 KiB/s wr, 9 op/s 2023-03-09T21:28:33.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23423.658447 secs 2023-03-09T21:28:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23885.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:28:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:35 smithi122 ceph-mon[93869]: pgmap v13004: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 94 KiB/s wr, 7 op/s 2023-03-09T21:28:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:35 smithi181 ceph-mon[94189]: pgmap v13004: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 94 KiB/s wr, 7 op/s 2023-03-09T21:28:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:35 smithi038 ceph-mon[92422]: pgmap v13004: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 94 KiB/s wr, 7 op/s 2023-03-09T21:28:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23426.317621 secs 2023-03-09T21:28:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23426.317621 secs 2023-03-09T21:28:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23426.317621 secs 2023-03-09T21:28:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:37 smithi122 ceph-mon[93869]: pgmap v13005: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 94 KiB/s wr, 9 op/s 2023-03-09T21:28:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:37 smithi181 ceph-mon[94189]: pgmap v13005: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 94 KiB/s wr, 9 op/s 2023-03-09T21:28:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:37 smithi038 ceph-mon[92422]: pgmap v13005: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 94 KiB/s wr, 9 op/s 2023-03-09T21:28:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23428.658563 secs 2023-03-09T21:28:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23428.658563 secs 2023-03-09T21:28:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23428.658563 secs 2023-03-09T21:28:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23890.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:28:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:39 smithi122 ceph-mon[93869]: pgmap v13006: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 94 KiB/s wr, 6 op/s 2023-03-09T21:28:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:39 smithi181 ceph-mon[94189]: pgmap v13006: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 94 KiB/s wr, 6 op/s 2023-03-09T21:28:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:39 smithi038 ceph-mon[92422]: pgmap v13006: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 94 KiB/s wr, 6 op/s 2023-03-09T21:28:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:41 smithi122 ceph-mon[93869]: pgmap v13007: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 94 KiB/s wr, 6 op/s 2023-03-09T21:28:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23431.317679 secs 2023-03-09T21:28:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:41 smithi181 ceph-mon[94189]: pgmap v13007: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 94 KiB/s wr, 6 op/s 2023-03-09T21:28:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23431.317679 secs 2023-03-09T21:28:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:41 smithi038 ceph-mon[92422]: pgmap v13007: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 94 KiB/s wr, 6 op/s 2023-03-09T21:28:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23431.317679 secs 2023-03-09T21:28:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:43 smithi122 ceph-mon[93869]: pgmap v13008: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 321 KiB/s rd, 272 KiB/s wr, 9 op/s 2023-03-09T21:28:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23433.658681 secs 2023-03-09T21:28:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:43 smithi181 ceph-mon[94189]: pgmap v13008: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 321 KiB/s rd, 272 KiB/s wr, 9 op/s 2023-03-09T21:28:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23433.658681 secs 2023-03-09T21:28:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:43 smithi038 ceph-mon[92422]: pgmap v13008: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 321 KiB/s rd, 272 KiB/s wr, 9 op/s 2023-03-09T21:28:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23433.658681 secs 2023-03-09T21:28:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23895.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:28:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:45 smithi122 ceph-mon[93869]: pgmap v13009: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 179 KiB/s wr, 6 op/s 2023-03-09T21:28:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:45 smithi181 ceph-mon[94189]: pgmap v13009: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 179 KiB/s wr, 6 op/s 2023-03-09T21:28:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:45 smithi038 ceph-mon[92422]: pgmap v13009: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 179 KiB/s wr, 6 op/s 2023-03-09T21:28:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23436.317739 secs 2023-03-09T21:28:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23436.317739 secs 2023-03-09T21:28:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23436.317739 secs 2023-03-09T21:28:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:47 smithi122 ceph-mon[93869]: pgmap v13010: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 330 KiB/s wr, 10 op/s 2023-03-09T21:28:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:47 smithi181 ceph-mon[94189]: pgmap v13010: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 330 KiB/s wr, 10 op/s 2023-03-09T21:28:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:47 smithi038 ceph-mon[92422]: pgmap v13010: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 330 KiB/s wr, 10 op/s 2023-03-09T21:28:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23900.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:28:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:49 smithi122 ceph-mon[93869]: pgmap v13011: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 330 KiB/s wr, 7 op/s 2023-03-09T21:28:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:49 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:28:49.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:49 smithi181 ceph-mon[94189]: pgmap v13011: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 330 KiB/s wr, 7 op/s 2023-03-09T21:28:49.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:49 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:28:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:49 smithi038 ceph-mon[92422]: pgmap v13011: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 330 KiB/s wr, 7 op/s 2023-03-09T21:28:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:49 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:28:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:50 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:28:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:50 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:28:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:50 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:28:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:50 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:28:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:50 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:28:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:50 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:28:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:50 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:28:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:50 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:28:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:50 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:28:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:51 smithi122 ceph-mon[93869]: pgmap v13012: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 574 KiB/s rd, 330 KiB/s wr, 7 op/s 2023-03-09T21:28:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23441.317801 secs 2023-03-09T21:28:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:28:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:28:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:51 smithi181 ceph-mon[94189]: pgmap v13012: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 574 KiB/s rd, 330 KiB/s wr, 7 op/s 2023-03-09T21:28:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23441.317801 secs 2023-03-09T21:28:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:28:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:28:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:51 smithi038 ceph-mon[92422]: pgmap v13012: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 574 KiB/s rd, 330 KiB/s wr, 7 op/s 2023-03-09T21:28:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23441.317801 secs 2023-03-09T21:28:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:28:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:28:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:53 smithi122 ceph-mon[93869]: pgmap v13013: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 400 KiB/s wr, 12 op/s 2023-03-09T21:28:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:28:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:28:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:53 smithi181 ceph-mon[94189]: pgmap v13013: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 400 KiB/s wr, 12 op/s 2023-03-09T21:28:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:28:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:28:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:53 smithi038 ceph-mon[92422]: pgmap v13013: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 400 KiB/s wr, 12 op/s 2023-03-09T21:28:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:28:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:28:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23905.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:28:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:55 smithi122 ceph-mon[93869]: pgmap v13014: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 422 KiB/s rd, 223 KiB/s wr, 10 op/s 2023-03-09T21:28:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:28:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:28:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:55 smithi181 ceph-mon[94189]: pgmap v13014: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 422 KiB/s rd, 223 KiB/s wr, 10 op/s 2023-03-09T21:28:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:28:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:28:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:55 smithi038 ceph-mon[92422]: pgmap v13014: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 422 KiB/s rd, 223 KiB/s wr, 10 op/s 2023-03-09T21:28:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:28:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:28:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23446.317863 secs 2023-03-09T21:28:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23446.317863 secs 2023-03-09T21:28:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23446.317863 secs 2023-03-09T21:28:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:57 smithi122 ceph-mon[93869]: pgmap v13015: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 838 KiB/s rd, 637 KiB/s wr, 14 op/s 2023-03-09T21:28:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:28:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:28:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:57 smithi181 ceph-mon[94189]: pgmap v13015: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 838 KiB/s rd, 637 KiB/s wr, 14 op/s 2023-03-09T21:28:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:28:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:28:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:57 smithi038 ceph-mon[92422]: pgmap v13015: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 838 KiB/s rd, 637 KiB/s wr, 14 op/s 2023-03-09T21:28:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:28:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:28:58.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23910.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:28:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:28:59 smithi122 ceph-mon[93869]: pgmap v13016: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 486 KiB/s wr, 10 op/s 2023-03-09T21:28:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:28:59 smithi181 ceph-mon[94189]: pgmap v13016: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 486 KiB/s wr, 10 op/s 2023-03-09T21:28:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:28:59 smithi038 ceph-mon[92422]: pgmap v13016: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 486 KiB/s wr, 10 op/s 2023-03-09T21:29:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:01 smithi122 ceph-mon[93869]: pgmap v13017: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 486 KiB/s wr, 11 op/s 2023-03-09T21:29:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23451.317925 secs 2023-03-09T21:29:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:01 smithi181 ceph-mon[94189]: pgmap v13017: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 486 KiB/s wr, 11 op/s 2023-03-09T21:29:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23451.317925 secs 2023-03-09T21:29:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:01 smithi038 ceph-mon[92422]: pgmap v13017: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 486 KiB/s wr, 11 op/s 2023-03-09T21:29:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23451.317925 secs 2023-03-09T21:29:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:03 smithi122 ceph-mon[93869]: pgmap v13018: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 691 KiB/s rd, 609 KiB/s wr, 15 op/s 2023-03-09T21:29:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:03 smithi181 ceph-mon[94189]: pgmap v13018: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 691 KiB/s rd, 609 KiB/s wr, 15 op/s 2023-03-09T21:29:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:03 smithi038 ceph-mon[92422]: pgmap v13018: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 691 KiB/s rd, 609 KiB/s wr, 15 op/s 2023-03-09T21:29:03.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23915.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:29:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:05 smithi122 ceph-mon[93869]: pgmap v13019: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 675 KiB/s rd, 748 KiB/s wr, 24 op/s 2023-03-09T21:29:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:05 smithi181 ceph-mon[94189]: pgmap v13019: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 675 KiB/s rd, 748 KiB/s wr, 24 op/s 2023-03-09T21:29:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:05 smithi038 ceph-mon[92422]: pgmap v13019: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 675 KiB/s rd, 748 KiB/s wr, 24 op/s 2023-03-09T21:29:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23456.317985 secs 2023-03-09T21:29:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23456.317985 secs 2023-03-09T21:29:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23456.317985 secs 2023-03-09T21:29:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:07 smithi122 ceph-mon[93869]: pgmap v13020: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 851 KiB/s rd, 789 KiB/s wr, 29 op/s 2023-03-09T21:29:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:29:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:29:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:07 smithi181 ceph-mon[94189]: pgmap v13020: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 851 KiB/s rd, 789 KiB/s wr, 29 op/s 2023-03-09T21:29:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:29:07.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:29:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:07 smithi038 ceph-mon[92422]: pgmap v13020: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 851 KiB/s rd, 789 KiB/s wr, 29 op/s 2023-03-09T21:29:07.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:29:07.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:29:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23458.659252 secs 2023-03-09T21:29:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23458.659252 secs 2023-03-09T21:29:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23458.659252 secs 2023-03-09T21:29:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23920.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:29:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:09 smithi122 ceph-mon[93869]: pgmap v13021: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 436 KiB/s rd, 375 KiB/s wr, 24 op/s 2023-03-09T21:29:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:29:09.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:29:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:09 smithi181 ceph-mon[94189]: pgmap v13021: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 436 KiB/s rd, 375 KiB/s wr, 24 op/s 2023-03-09T21:29:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:29:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:29:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:09 smithi038 ceph-mon[92422]: pgmap v13021: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 436 KiB/s rd, 375 KiB/s wr, 24 op/s 2023-03-09T21:29:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:29:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:29:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:10 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:29:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:10 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:29:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:10 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:29:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:10 smithi181 ceph-mon[94189]: osdmap e1027: 12 total, 12 up, 12 in 2023-03-09T21:29:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:10 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:29:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:10 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:29:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:10 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:29:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:10 smithi038 ceph-mon[92422]: osdmap e1027: 12 total, 12 up, 12 in 2023-03-09T21:29:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:10 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:29:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:10 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:29:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:10 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:29:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:10 smithi122 ceph-mon[93869]: osdmap e1027: 12 total, 12 up, 12 in 2023-03-09T21:29:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:11 smithi038 ceph-mon[92422]: pgmap v13023: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 925 KiB/s rd, 687 KiB/s wr, 31 op/s 2023-03-09T21:29:11.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23461.318048 secs 2023-03-09T21:29:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:11 smithi122 ceph-mon[93869]: pgmap v13023: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 925 KiB/s rd, 687 KiB/s wr, 31 op/s 2023-03-09T21:29:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23461.318048 secs 2023-03-09T21:29:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:11 smithi181 ceph-mon[94189]: pgmap v13023: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 925 KiB/s rd, 687 KiB/s wr, 31 op/s 2023-03-09T21:29:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23461.318048 secs 2023-03-09T21:29:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:13 smithi038 ceph-mon[92422]: pgmap v13024: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 686 KiB/s rd, 541 KiB/s wr, 29 op/s 2023-03-09T21:29:13.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23463.659372 secs 2023-03-09T21:29:13.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23925.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:29:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:13 smithi122 ceph-mon[93869]: pgmap v13024: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 686 KiB/s rd, 541 KiB/s wr, 29 op/s 2023-03-09T21:29:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23463.659372 secs 2023-03-09T21:29:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:13 smithi181 ceph-mon[94189]: pgmap v13024: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 686 KiB/s rd, 541 KiB/s wr, 29 op/s 2023-03-09T21:29:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23463.659372 secs 2023-03-09T21:29:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:15 smithi038 ceph-mon[92422]: pgmap v13025: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 616 KiB/s rd, 290 KiB/s wr, 14 op/s 2023-03-09T21:29:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:15 smithi122 ceph-mon[93869]: pgmap v13025: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 616 KiB/s rd, 290 KiB/s wr, 14 op/s 2023-03-09T21:29:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:15 smithi181 ceph-mon[94189]: pgmap v13025: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 616 KiB/s rd, 290 KiB/s wr, 14 op/s 2023-03-09T21:29:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23466.318110 secs 2023-03-09T21:29:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23466.318110 secs 2023-03-09T21:29:16.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23466.318110 secs 2023-03-09T21:29:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:17 smithi122 ceph-mon[93869]: pgmap v13026: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 263 KiB/s wr, 11 op/s 2023-03-09T21:29:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:29:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:29:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:17 smithi181 ceph-mon[94189]: pgmap v13026: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 263 KiB/s wr, 11 op/s 2023-03-09T21:29:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:29:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:29:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:17 smithi038 ceph-mon[92422]: pgmap v13026: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 263 KiB/s wr, 11 op/s 2023-03-09T21:29:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:29:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:29:18.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23930.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:29:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23468.659485 secs 2023-03-09T21:29:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23468.659485 secs 2023-03-09T21:29:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23468.659485 secs 2023-03-09T21:29:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:19 smithi122 ceph-mon[93869]: pgmap v13027: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 263 KiB/s wr, 11 op/s 2023-03-09T21:29:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:19 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:29:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:19 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:29:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:19 smithi181 ceph-mon[94189]: pgmap v13027: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 263 KiB/s wr, 11 op/s 2023-03-09T21:29:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:19 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:29:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:19 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:29:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:19 smithi038 ceph-mon[92422]: pgmap v13027: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 263 KiB/s wr, 11 op/s 2023-03-09T21:29:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:19 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:29:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:19 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:29:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:21 smithi122 ceph-mon[93869]: pgmap v13028: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 114 KiB/s wr, 9 op/s 2023-03-09T21:29:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23471.318171 secs 2023-03-09T21:29:21.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:21 smithi122 ceph-mon[93869]: osdmap e1028: 12 total, 12 up, 12 in 2023-03-09T21:29:21.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:29:21.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:29:21.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:29:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:21 smithi181 ceph-mon[94189]: pgmap v13028: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 114 KiB/s wr, 9 op/s 2023-03-09T21:29:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23471.318171 secs 2023-03-09T21:29:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:21 smithi181 ceph-mon[94189]: osdmap e1028: 12 total, 12 up, 12 in 2023-03-09T21:29:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:29:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:29:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:29:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:21 smithi038 ceph-mon[92422]: pgmap v13028: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 114 KiB/s wr, 9 op/s 2023-03-09T21:29:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23471.318171 secs 2023-03-09T21:29:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:21 smithi038 ceph-mon[92422]: osdmap e1028: 12 total, 12 up, 12 in 2023-03-09T21:29:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:29:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:29:21.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:29:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:22 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:29:22.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:22 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:29:22.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:22 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:29:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:22 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:29:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:22 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:29:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:22 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:29:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:22 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:29:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:22 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:29:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:22 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:29:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23935.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:29:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:23 smithi122 ceph-mon[93869]: pgmap v13030: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 115 KiB/s wr, 9 op/s 2023-03-09T21:29:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23473.659560 secs 2023-03-09T21:29:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:23 smithi181 ceph-mon[94189]: pgmap v13030: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 115 KiB/s wr, 9 op/s 2023-03-09T21:29:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23473.659560 secs 2023-03-09T21:29:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:23 smithi038 ceph-mon[92422]: pgmap v13030: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 115 KiB/s wr, 9 op/s 2023-03-09T21:29:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23473.659560 secs 2023-03-09T21:29:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:25 smithi122 ceph-mon[93869]: pgmap v13031: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 115 KiB/s wr, 8 op/s 2023-03-09T21:29:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:25 smithi181 ceph-mon[94189]: pgmap v13031: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 115 KiB/s wr, 8 op/s 2023-03-09T21:29:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:25 smithi038 ceph-mon[92422]: pgmap v13031: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 115 KiB/s wr, 8 op/s 2023-03-09T21:29:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23476.318231 secs 2023-03-09T21:29:26.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23476.318231 secs 2023-03-09T21:29:26.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23476.318231 secs 2023-03-09T21:29:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:27 smithi122 ceph-mon[93869]: pgmap v13032: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 KiB/s wr, 7 op/s 2023-03-09T21:29:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:27 smithi181 ceph-mon[94189]: pgmap v13032: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 KiB/s wr, 7 op/s 2023-03-09T21:29:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:27 smithi038 ceph-mon[92422]: pgmap v13032: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 KiB/s wr, 7 op/s 2023-03-09T21:29:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23940.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:29:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23478.659676 secs 2023-03-09T21:29:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23478.659676 secs 2023-03-09T21:29:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23478.659676 secs 2023-03-09T21:29:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:29 smithi122 ceph-mon[93869]: pgmap v13033: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 KiB/s wr, 6 op/s 2023-03-09T21:29:29.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:29 smithi181 ceph-mon[94189]: pgmap v13033: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 KiB/s wr, 6 op/s 2023-03-09T21:29:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:29 smithi038 ceph-mon[92422]: pgmap v13033: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 KiB/s wr, 6 op/s 2023-03-09T21:29:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:31 smithi122 ceph-mon[93869]: pgmap v13034: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2023-03-09T21:29:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23481.318292 secs 2023-03-09T21:29:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:31 smithi181 ceph-mon[94189]: pgmap v13034: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2023-03-09T21:29:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23481.318292 secs 2023-03-09T21:29:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:31 smithi038 ceph-mon[92422]: pgmap v13034: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2023-03-09T21:29:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23481.318292 secs 2023-03-09T21:29:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23945.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:29:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:33 smithi122 ceph-mon[93869]: pgmap v13035: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.4 KiB/s wr, 7 op/s 2023-03-09T21:29:33.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23483.659792 secs 2023-03-09T21:29:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:33 smithi181 ceph-mon[94189]: pgmap v13035: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.4 KiB/s wr, 7 op/s 2023-03-09T21:29:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23483.659792 secs 2023-03-09T21:29:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:33 smithi038 ceph-mon[92422]: pgmap v13035: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.4 KiB/s wr, 7 op/s 2023-03-09T21:29:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23483.659792 secs 2023-03-09T21:29:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:35 smithi122 ceph-mon[93869]: pgmap v13036: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 95 KiB/s wr, 6 op/s 2023-03-09T21:29:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:35 smithi181 ceph-mon[94189]: pgmap v13036: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 95 KiB/s wr, 6 op/s 2023-03-09T21:29:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:35 smithi038 ceph-mon[92422]: pgmap v13036: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 95 KiB/s wr, 6 op/s 2023-03-09T21:29:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23486.318352 secs 2023-03-09T21:29:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23486.318352 secs 2023-03-09T21:29:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23486.318352 secs 2023-03-09T21:29:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:37 smithi122 ceph-mon[93869]: pgmap v13037: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 95 KiB/s wr, 8 op/s 2023-03-09T21:29:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:37 smithi181 ceph-mon[94189]: pgmap v13037: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 95 KiB/s wr, 8 op/s 2023-03-09T21:29:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:37 smithi038 ceph-mon[92422]: pgmap v13037: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 95 KiB/s wr, 8 op/s 2023-03-09T21:29:38.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23950.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:29:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23488.659912 secs 2023-03-09T21:29:38.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23488.659912 secs 2023-03-09T21:29:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23488.659912 secs 2023-03-09T21:29:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:39 smithi122 ceph-mon[93869]: pgmap v13038: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 95 KiB/s wr, 6 op/s 2023-03-09T21:29:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:39 smithi181 ceph-mon[94189]: pgmap v13038: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 95 KiB/s wr, 6 op/s 2023-03-09T21:29:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:39 smithi038 ceph-mon[92422]: pgmap v13038: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 95 KiB/s wr, 6 op/s 2023-03-09T21:29:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:41 smithi122 ceph-mon[93869]: pgmap v13039: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 95 KiB/s wr, 7 op/s 2023-03-09T21:29:41.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23491.318415 secs 2023-03-09T21:29:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:41 smithi181 ceph-mon[94189]: pgmap v13039: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 95 KiB/s wr, 7 op/s 2023-03-09T21:29:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23491.318415 secs 2023-03-09T21:29:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:41 smithi038 ceph-mon[92422]: pgmap v13039: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 95 KiB/s wr, 7 op/s 2023-03-09T21:29:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23491.318415 secs 2023-03-09T21:29:43.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23955.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:29:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:43 smithi122 ceph-mon[93869]: pgmap v13040: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 95 KiB/s wr, 8 op/s 2023-03-09T21:29:43.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:43 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 23493.660045 secs 2023-03-09T21:29:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:43 smithi181 ceph-mon[94189]: pgmap v13040: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 95 KiB/s wr, 8 op/s 2023-03-09T21:29:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:43 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 23493.660045 secs 2023-03-09T21:29:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:43 smithi038 ceph-mon[92422]: pgmap v13040: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 95 KiB/s wr, 8 op/s 2023-03-09T21:29:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:43 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 23493.660045 secs 2023-03-09T21:29:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:45 smithi122 ceph-mon[93869]: pgmap v13041: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 222 KiB/s wr, 7 op/s 2023-03-09T21:29:45.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:45 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:29:45.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:45 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:29:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:45 smithi181 ceph-mon[94189]: pgmap v13041: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 222 KiB/s wr, 7 op/s 2023-03-09T21:29:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:45 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:29:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:45 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:29:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:45 smithi038 ceph-mon[92422]: pgmap v13041: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 222 KiB/s wr, 7 op/s 2023-03-09T21:29:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:45 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:29:46.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:45 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:29:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23496.318487 secs 2023-03-09T21:29:46.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23496.318487 secs 2023-03-09T21:29:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23496.318487 secs 2023-03-09T21:29:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:47 smithi122 ceph-mon[93869]: pgmap v13042: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 301 KiB/s wr, 9 op/s 2023-03-09T21:29:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:47 smithi181 ceph-mon[94189]: pgmap v13042: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 301 KiB/s wr, 9 op/s 2023-03-09T21:29:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:47 smithi038 ceph-mon[92422]: pgmap v13042: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 301 KiB/s wr, 9 op/s 2023-03-09T21:29:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23960.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:29:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:48 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 23498.660163 secs 2023-03-09T21:29:48.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:48 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 23498.660163 secs 2023-03-09T21:29:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:48 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 23498.660163 secs 2023-03-09T21:29:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:49 smithi122 ceph-mon[93869]: pgmap v13043: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 471 KiB/s rd, 301 KiB/s wr, 7 op/s 2023-03-09T21:29:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:49 smithi181 ceph-mon[94189]: pgmap v13043: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 471 KiB/s rd, 301 KiB/s wr, 7 op/s 2023-03-09T21:29:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:49 smithi038 ceph-mon[92422]: pgmap v13043: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 471 KiB/s rd, 301 KiB/s wr, 7 op/s 2023-03-09T21:29:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:51 smithi122 ceph-mon[93869]: pgmap v13044: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 532 KiB/s rd, 301 KiB/s wr, 8 op/s 2023-03-09T21:29:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23501.318549 secs 2023-03-09T21:29:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:29:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:29:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:51 smithi181 ceph-mon[94189]: pgmap v13044: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 532 KiB/s rd, 301 KiB/s wr, 8 op/s 2023-03-09T21:29:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23501.318549 secs 2023-03-09T21:29:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:29:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:29:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:51 smithi038 ceph-mon[92422]: pgmap v13044: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 532 KiB/s rd, 301 KiB/s wr, 8 op/s 2023-03-09T21:29:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23501.318549 secs 2023-03-09T21:29:52.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:29:52.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:29:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:53 smithi122 ceph-mon[93869]: pgmap v13045: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 676 KiB/s rd, 507 KiB/s wr, 12 op/s 2023-03-09T21:29:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:29:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:29:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23503.660242 secs 2023-03-09T21:29:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:53 smithi181 ceph-mon[94189]: pgmap v13045: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 676 KiB/s rd, 507 KiB/s wr, 12 op/s 2023-03-09T21:29:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:29:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:29:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23503.660242 secs 2023-03-09T21:29:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:53 smithi038 ceph-mon[92422]: pgmap v13045: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 676 KiB/s rd, 507 KiB/s wr, 12 op/s 2023-03-09T21:29:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:29:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:29:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23503.660242 secs 2023-03-09T21:29:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23965.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:29:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:54 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:29:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:54 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:29:54.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:54 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:29:54.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:54 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:29:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:54 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:29:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:54 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:29:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:55 smithi122 ceph-mon[93869]: pgmap v13046: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 674 KiB/s rd, 507 KiB/s wr, 10 op/s 2023-03-09T21:29:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:55 smithi181 ceph-mon[94189]: pgmap v13046: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 674 KiB/s rd, 507 KiB/s wr, 10 op/s 2023-03-09T21:29:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:55 smithi038 ceph-mon[92422]: pgmap v13046: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 674 KiB/s rd, 507 KiB/s wr, 10 op/s 2023-03-09T21:29:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23506.318610 secs 2023-03-09T21:29:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:29:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:29:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23506.318610 secs 2023-03-09T21:29:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:29:56.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:29:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23506.318610 secs 2023-03-09T21:29:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:29:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:29:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:57 smithi122 ceph-mon[93869]: pgmap v13047: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 669 KiB/s wr, 13 op/s 2023-03-09T21:29:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:29:57.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:29:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:57 smithi181 ceph-mon[94189]: pgmap v13047: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 669 KiB/s wr, 13 op/s 2023-03-09T21:29:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:29:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:29:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:57 smithi038 ceph-mon[92422]: pgmap v13047: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 669 KiB/s wr, 13 op/s 2023-03-09T21:29:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:29:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:29:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23508.660356 secs 2023-03-09T21:29:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23508.660356 secs 2023-03-09T21:29:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23508.660356 secs 2023-03-09T21:29:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23970.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:29:59.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:29:59 smithi122 ceph-mon[93869]: pgmap v13048: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 645 KiB/s rd, 496 KiB/s wr, 10 op/s 2023-03-09T21:29:59.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:29:59 smithi181 ceph-mon[94189]: pgmap v13048: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 645 KiB/s rd, 496 KiB/s wr, 10 op/s 2023-03-09T21:29:59.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:29:59 smithi038 ceph-mon[92422]: pgmap v13048: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 645 KiB/s rd, 496 KiB/s wr, 10 op/s 2023-03-09T21:30:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:00 smithi122 ceph-mon[93869]: overall HEALTH_WARN 2 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-09T21:30:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:00 smithi181 ceph-mon[94189]: overall HEALTH_WARN 2 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-09T21:30:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:00 smithi038 ceph-mon[92422]: overall HEALTH_WARN 2 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-09T21:30:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:01 smithi122 ceph-mon[93869]: pgmap v13049: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 645 KiB/s rd, 496 KiB/s wr, 10 op/s 2023-03-09T21:30:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23511.318668 secs 2023-03-09T21:30:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:01 smithi181 ceph-mon[94189]: pgmap v13049: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 645 KiB/s rd, 496 KiB/s wr, 10 op/s 2023-03-09T21:30:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23511.318668 secs 2023-03-09T21:30:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:01 smithi038 ceph-mon[92422]: pgmap v13049: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 645 KiB/s rd, 496 KiB/s wr, 10 op/s 2023-03-09T21:30:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23511.318668 secs 2023-03-09T21:30:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:03 smithi122 ceph-mon[93869]: pgmap v13050: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 695 KiB/s rd, 586 KiB/s wr, 14 op/s 2023-03-09T21:30:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23513.660507 secs 2023-03-09T21:30:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:03 smithi181 ceph-mon[94189]: pgmap v13050: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 695 KiB/s rd, 586 KiB/s wr, 14 op/s 2023-03-09T21:30:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23513.660507 secs 2023-03-09T21:30:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:03 smithi038 ceph-mon[92422]: pgmap v13050: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 695 KiB/s rd, 586 KiB/s wr, 14 op/s 2023-03-09T21:30:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23513.660507 secs 2023-03-09T21:30:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23975.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:30:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:05 smithi122 ceph-mon[93869]: pgmap v13051: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 643 KiB/s rd, 580 KiB/s wr, 23 op/s 2023-03-09T21:30:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:05 smithi181 ceph-mon[94189]: pgmap v13051: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 643 KiB/s rd, 580 KiB/s wr, 23 op/s 2023-03-09T21:30:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:05 smithi038 ceph-mon[92422]: pgmap v13051: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 643 KiB/s rd, 580 KiB/s wr, 23 op/s 2023-03-09T21:30:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23516.318728 secs 2023-03-09T21:30:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23516.318728 secs 2023-03-09T21:30:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23516.318728 secs 2023-03-09T21:30:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:07 smithi038 ceph-mon[92422]: pgmap v13052: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 702 KiB/s rd, 620 KiB/s wr, 27 op/s 2023-03-09T21:30:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:07 smithi122 ceph-mon[93869]: pgmap v13052: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 702 KiB/s rd, 620 KiB/s wr, 27 op/s 2023-03-09T21:30:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:07 smithi181 ceph-mon[94189]: pgmap v13052: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 702 KiB/s rd, 620 KiB/s wr, 27 op/s 2023-03-09T21:30:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23518.660624 secs 2023-03-09T21:30:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:30:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:30:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23518.660624 secs 2023-03-09T21:30:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:30:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:30:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23518.660624 secs 2023-03-09T21:30:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:30:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:30:08.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23980.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:30:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:09 smithi038 ceph-mon[92422]: pgmap v13053: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 331 KiB/s wr, 23 op/s 2023-03-09T21:30:09.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:30:09.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:30:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:09 smithi122 ceph-mon[93869]: pgmap v13053: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 331 KiB/s wr, 23 op/s 2023-03-09T21:30:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:30:09.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:30:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:09 smithi181 ceph-mon[94189]: pgmap v13053: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 331 KiB/s wr, 23 op/s 2023-03-09T21:30:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:30:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:30:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:10 smithi122 ceph-mon[93869]: osdmap e1029: 12 total, 12 up, 12 in 2023-03-09T21:30:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:10 smithi181 ceph-mon[94189]: osdmap e1029: 12 total, 12 up, 12 in 2023-03-09T21:30:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:10 smithi038 ceph-mon[92422]: osdmap e1029: 12 total, 12 up, 12 in 2023-03-09T21:30:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:11 smithi122 ceph-mon[93869]: pgmap v13055: 129 pgs: 1 active+clean+snaptrim_wait, 128 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 674 KiB/s rd, 737 KiB/s wr, 29 op/s 2023-03-09T21:30:11.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23521.318787 secs 2023-03-09T21:30:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:11 smithi181 ceph-mon[94189]: pgmap v13055: 129 pgs: 1 active+clean+snaptrim_wait, 128 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 674 KiB/s rd, 737 KiB/s wr, 29 op/s 2023-03-09T21:30:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23521.318787 secs 2023-03-09T21:30:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:11 smithi038 ceph-mon[92422]: pgmap v13055: 129 pgs: 1 active+clean+snaptrim_wait, 128 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 674 KiB/s rd, 737 KiB/s wr, 29 op/s 2023-03-09T21:30:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23521.318787 secs 2023-03-09T21:30:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:13 smithi122 ceph-mon[93869]: pgmap v13056: 129 pgs: 1 active+clean+snaptrim_wait, 128 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 543 KiB/s rd, 630 KiB/s wr, 27 op/s 2023-03-09T21:30:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23523.660739 secs 2023-03-09T21:30:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:13 smithi181 ceph-mon[94189]: pgmap v13056: 129 pgs: 1 active+clean+snaptrim_wait, 128 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 543 KiB/s rd, 630 KiB/s wr, 27 op/s 2023-03-09T21:30:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23523.660739 secs 2023-03-09T21:30:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:13 smithi038 ceph-mon[92422]: pgmap v13056: 129 pgs: 1 active+clean+snaptrim_wait, 128 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 543 KiB/s rd, 630 KiB/s wr, 27 op/s 2023-03-09T21:30:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23523.660739 secs 2023-03-09T21:30:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23985.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:30:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:15 smithi122 ceph-mon[93869]: pgmap v13057: 129 pgs: 1 active+clean+snaptrim_wait, 128 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 321 KiB/s rd, 390 KiB/s wr, 12 op/s 2023-03-09T21:30:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:15 smithi181 ceph-mon[94189]: pgmap v13057: 129 pgs: 1 active+clean+snaptrim_wait, 128 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 321 KiB/s rd, 390 KiB/s wr, 12 op/s 2023-03-09T21:30:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:15 smithi038 ceph-mon[92422]: pgmap v13057: 129 pgs: 1 active+clean+snaptrim_wait, 128 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 321 KiB/s rd, 390 KiB/s wr, 12 op/s 2023-03-09T21:30:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23526.318850 secs 2023-03-09T21:30:16.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:30:16.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:30:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23526.318850 secs 2023-03-09T21:30:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:30:16.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:30:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23526.318850 secs 2023-03-09T21:30:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:30:16.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:30:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:17 smithi122 ceph-mon[93869]: pgmap v13058: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 441 KiB/s wr, 10 op/s 2023-03-09T21:30:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:30:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:30:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:17 smithi181 ceph-mon[94189]: pgmap v13058: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 441 KiB/s wr, 10 op/s 2023-03-09T21:30:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:30:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:30:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:17 smithi038 ceph-mon[92422]: pgmap v13058: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 441 KiB/s wr, 10 op/s 2023-03-09T21:30:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:30:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:30:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23528.660889 secs 2023-03-09T21:30:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23528.660889 secs 2023-03-09T21:30:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23528.660889 secs 2023-03-09T21:30:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23990.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:30:19.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:19 smithi122 ceph-mon[93869]: pgmap v13059: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 440 KiB/s wr, 10 op/s 2023-03-09T21:30:19.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:19 smithi181 ceph-mon[94189]: pgmap v13059: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 440 KiB/s wr, 10 op/s 2023-03-09T21:30:19.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:19 smithi038 ceph-mon[92422]: pgmap v13059: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 440 KiB/s wr, 10 op/s 2023-03-09T21:30:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:21 smithi122 ceph-mon[93869]: pgmap v13060: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 232 KiB/s wr, 9 op/s 2023-03-09T21:30:21.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23531.318927 secs 2023-03-09T21:30:21.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:21 smithi122 ceph-mon[93869]: osdmap e1030: 12 total, 12 up, 12 in 2023-03-09T21:30:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:21 smithi181 ceph-mon[94189]: pgmap v13060: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 232 KiB/s wr, 9 op/s 2023-03-09T21:30:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23531.318927 secs 2023-03-09T21:30:21.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:21 smithi181 ceph-mon[94189]: osdmap e1030: 12 total, 12 up, 12 in 2023-03-09T21:30:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:21 smithi038 ceph-mon[92422]: pgmap v13060: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 232 KiB/s wr, 9 op/s 2023-03-09T21:30:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23531.318927 secs 2023-03-09T21:30:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:21 smithi038 ceph-mon[92422]: osdmap e1030: 12 total, 12 up, 12 in 2023-03-09T21:30:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:22 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:30:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:22 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:30:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:22 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:30:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:23 smithi122 ceph-mon[93869]: pgmap v13062: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 100 KiB/s wr, 9 op/s 2023-03-09T21:30:23.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:23 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:30:23.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:23 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:30:23.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23533.661002 secs 2023-03-09T21:30:23.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:23 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:30:23.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:23 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:30:23.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:23 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:30:23.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:23 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:30:23.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:23 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:30:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:23 smithi181 ceph-mon[94189]: pgmap v13062: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 100 KiB/s wr, 9 op/s 2023-03-09T21:30:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:23 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:30:23.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:23 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:30:23.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23533.661002 secs 2023-03-09T21:30:23.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:23 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:30:23.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:23 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:30:23.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:23 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:30:23.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:23 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:30:23.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:23 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:30:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:23 smithi038 ceph-mon[92422]: pgmap v13062: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 100 KiB/s wr, 9 op/s 2023-03-09T21:30:23.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:23 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:30:23.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:23 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:30:23.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23533.661002 secs 2023-03-09T21:30:23.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:23 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:30:23.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:23 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:30:23.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:23 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:30:23.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:23 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:30:23.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:23 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:30:23.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 23995.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:30:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:25 smithi122 ceph-mon[93869]: pgmap v13063: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 101 KiB/s wr, 8 op/s 2023-03-09T21:30:25.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:25 smithi181 ceph-mon[94189]: pgmap v13063: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 101 KiB/s wr, 8 op/s 2023-03-09T21:30:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:25 smithi038 ceph-mon[92422]: pgmap v13063: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 101 KiB/s wr, 8 op/s 2023-03-09T21:30:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23536.318983 secs 2023-03-09T21:30:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23536.318983 secs 2023-03-09T21:30:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23536.318983 secs 2023-03-09T21:30:27.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:27 smithi122 ceph-mon[93869]: pgmap v13064: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 86 KiB/s wr, 9 op/s 2023-03-09T21:30:27.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:27 smithi181 ceph-mon[94189]: pgmap v13064: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 86 KiB/s wr, 9 op/s 2023-03-09T21:30:27.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:27 smithi038 ceph-mon[92422]: pgmap v13064: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 86 KiB/s wr, 9 op/s 2023-03-09T21:30:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23538.661135 secs 2023-03-09T21:30:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23538.661135 secs 2023-03-09T21:30:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23538.661135 secs 2023-03-09T21:30:28.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24000.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:30:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:29 smithi038 ceph-mon[92422]: pgmap v13065: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 86 KiB/s wr, 9 op/s 2023-03-09T21:30:29.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:29 smithi122 ceph-mon[93869]: pgmap v13065: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 86 KiB/s wr, 9 op/s 2023-03-09T21:30:29.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:29 smithi181 ceph-mon[94189]: pgmap v13065: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 86 KiB/s wr, 9 op/s 2023-03-09T21:30:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:31 smithi038 ceph-mon[92422]: pgmap v13066: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 199 KiB/s wr, 8 op/s 2023-03-09T21:30:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23541.319040 secs 2023-03-09T21:30:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:31 smithi122 ceph-mon[93869]: pgmap v13066: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 199 KiB/s wr, 8 op/s 2023-03-09T21:30:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23541.319040 secs 2023-03-09T21:30:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:31 smithi181 ceph-mon[94189]: pgmap v13066: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 199 KiB/s wr, 8 op/s 2023-03-09T21:30:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23541.319040 secs 2023-03-09T21:30:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:33 smithi038 ceph-mon[92422]: pgmap v13067: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 249 KiB/s wr, 10 op/s 2023-03-09T21:30:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23543.661253 secs 2023-03-09T21:30:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:33 smithi122 ceph-mon[93869]: pgmap v13067: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 249 KiB/s wr, 10 op/s 2023-03-09T21:30:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23543.661253 secs 2023-03-09T21:30:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:33 smithi181 ceph-mon[94189]: pgmap v13067: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 249 KiB/s wr, 10 op/s 2023-03-09T21:30:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23543.661253 secs 2023-03-09T21:30:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24005.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:30:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:35 smithi038 ceph-mon[92422]: pgmap v13068: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 240 KiB/s wr, 7 op/s 2023-03-09T21:30:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:35 smithi122 ceph-mon[93869]: pgmap v13068: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 240 KiB/s wr, 7 op/s 2023-03-09T21:30:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:35 smithi181 ceph-mon[94189]: pgmap v13068: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 240 KiB/s wr, 7 op/s 2023-03-09T21:30:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23546.319099 secs 2023-03-09T21:30:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23546.319099 secs 2023-03-09T21:30:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23546.319099 secs 2023-03-09T21:30:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:37 smithi122 ceph-mon[93869]: pgmap v13069: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 253 KiB/s wr, 10 op/s 2023-03-09T21:30:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:37 smithi181 ceph-mon[94189]: pgmap v13069: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 253 KiB/s wr, 10 op/s 2023-03-09T21:30:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:37 smithi038 ceph-mon[92422]: pgmap v13069: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 253 KiB/s wr, 10 op/s 2023-03-09T21:30:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23548.661387 secs 2023-03-09T21:30:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23548.661387 secs 2023-03-09T21:30:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23548.661387 secs 2023-03-09T21:30:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24010.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:30:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:39 smithi122 ceph-mon[93869]: pgmap v13070: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 182 KiB/s wr, 7 op/s 2023-03-09T21:30:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:39 smithi181 ceph-mon[94189]: pgmap v13070: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 182 KiB/s wr, 7 op/s 2023-03-09T21:30:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:39 smithi038 ceph-mon[92422]: pgmap v13070: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 182 KiB/s wr, 7 op/s 2023-03-09T21:30:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:41 smithi122 ceph-mon[93869]: pgmap v13071: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 182 KiB/s wr, 7 op/s 2023-03-09T21:30:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23551.319160 secs 2023-03-09T21:30:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:30:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:30:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:41 smithi181 ceph-mon[94189]: pgmap v13071: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 182 KiB/s wr, 7 op/s 2023-03-09T21:30:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23551.319160 secs 2023-03-09T21:30:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:30:41.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:30:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:41 smithi038 ceph-mon[92422]: pgmap v13071: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 182 KiB/s wr, 7 op/s 2023-03-09T21:30:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23551.319160 secs 2023-03-09T21:30:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:30:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:30:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:43 smithi122 ceph-mon[93869]: pgmap v13072: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 192 KiB/s wr, 9 op/s 2023-03-09T21:30:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23553.661498 secs 2023-03-09T21:30:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:43 smithi181 ceph-mon[94189]: pgmap v13072: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 192 KiB/s wr, 9 op/s 2023-03-09T21:30:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23553.661498 secs 2023-03-09T21:30:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:43 smithi038 ceph-mon[92422]: pgmap v13072: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 192 KiB/s wr, 9 op/s 2023-03-09T21:30:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23553.661498 secs 2023-03-09T21:30:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24015.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:30:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:45 smithi122 ceph-mon[93869]: pgmap v13073: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 117 KiB/s wr, 7 op/s 2023-03-09T21:30:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:45 smithi181 ceph-mon[94189]: pgmap v13073: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 117 KiB/s wr, 7 op/s 2023-03-09T21:30:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:45 smithi038 ceph-mon[92422]: pgmap v13073: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 117 KiB/s wr, 7 op/s 2023-03-09T21:30:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23556.319219 secs 2023-03-09T21:30:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23556.319219 secs 2023-03-09T21:30:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23556.319219 secs 2023-03-09T21:30:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:47 smithi122 ceph-mon[93869]: pgmap v13074: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 408 KiB/s wr, 12 op/s 2023-03-09T21:30:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:47 smithi181 ceph-mon[94189]: pgmap v13074: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 408 KiB/s wr, 12 op/s 2023-03-09T21:30:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:47 smithi038 ceph-mon[92422]: pgmap v13074: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 408 KiB/s wr, 12 op/s 2023-03-09T21:30:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:30:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:30:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:30:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:30:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:30:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:30:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24020.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:30:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:49 smithi122 ceph-mon[93869]: pgmap v13075: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 395 KiB/s wr, 9 op/s 2023-03-09T21:30:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:49 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:30:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:49 smithi181 ceph-mon[94189]: pgmap v13075: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 395 KiB/s wr, 9 op/s 2023-03-09T21:30:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:49 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:30:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:49 smithi038 ceph-mon[92422]: pgmap v13075: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 395 KiB/s wr, 9 op/s 2023-03-09T21:30:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:49 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:30:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:50 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:30:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:50 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:30:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:50 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:30:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:50 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:30:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:50 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:30:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:50 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:30:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:50 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:30:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:50 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:30:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:50 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:30:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:51 smithi122 ceph-mon[93869]: pgmap v13076: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 432 KiB/s rd, 395 KiB/s wr, 34 op/s 2023-03-09T21:30:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23561.319281 secs 2023-03-09T21:30:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:51 smithi181 ceph-mon[94189]: pgmap v13076: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 432 KiB/s rd, 395 KiB/s wr, 34 op/s 2023-03-09T21:30:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23561.319281 secs 2023-03-09T21:30:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:51 smithi038 ceph-mon[92422]: pgmap v13076: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 432 KiB/s rd, 395 KiB/s wr, 34 op/s 2023-03-09T21:30:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23561.319281 secs 2023-03-09T21:30:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:30:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:30:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:30:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:30:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:30:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:30:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:53 smithi122 ceph-mon[93869]: pgmap v13077: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 421 KiB/s wr, 37 op/s 2023-03-09T21:30:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:30:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:30:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:53 smithi181 ceph-mon[94189]: pgmap v13077: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 421 KiB/s wr, 37 op/s 2023-03-09T21:30:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:30:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:30:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:53 smithi038 ceph-mon[92422]: pgmap v13077: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 421 KiB/s wr, 37 op/s 2023-03-09T21:30:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:30:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:30:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24025.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:30:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:55 smithi122 ceph-mon[93869]: pgmap v13078: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 319 KiB/s wr, 45 op/s 2023-03-09T21:30:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:55 smithi181 ceph-mon[94189]: pgmap v13078: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 319 KiB/s wr, 45 op/s 2023-03-09T21:30:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:55 smithi038 ceph-mon[92422]: pgmap v13078: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 319 KiB/s wr, 45 op/s 2023-03-09T21:30:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23566.319369 secs 2023-03-09T21:30:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23566.319369 secs 2023-03-09T21:30:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23566.319369 secs 2023-03-09T21:30:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:57 smithi122 ceph-mon[93869]: pgmap v13079: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 733 KiB/s rd, 683 KiB/s wr, 49 op/s 2023-03-09T21:30:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:30:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:30:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:57 smithi181 ceph-mon[94189]: pgmap v13079: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 733 KiB/s rd, 683 KiB/s wr, 49 op/s 2023-03-09T21:30:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:30:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:30:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:57 smithi038 ceph-mon[92422]: pgmap v13079: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 733 KiB/s rd, 683 KiB/s wr, 49 op/s 2023-03-09T21:30:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:30:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:30:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24030.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:30:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:30:59 smithi122 ceph-mon[93869]: pgmap v13080: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 442 KiB/s rd, 392 KiB/s wr, 44 op/s 2023-03-09T21:30:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:30:59 smithi181 ceph-mon[94189]: pgmap v13080: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 442 KiB/s rd, 392 KiB/s wr, 44 op/s 2023-03-09T21:30:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:30:59 smithi038 ceph-mon[92422]: pgmap v13080: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 442 KiB/s rd, 392 KiB/s wr, 44 op/s 2023-03-09T21:31:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:31:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:31:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:31:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:31:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:31:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:31:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:01 smithi122 ceph-mon[93869]: pgmap v13081: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 442 KiB/s rd, 392 KiB/s wr, 44 op/s 2023-03-09T21:31:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23571.319430 secs 2023-03-09T21:31:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:01 smithi181 ceph-mon[94189]: pgmap v13081: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 442 KiB/s rd, 392 KiB/s wr, 44 op/s 2023-03-09T21:31:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23571.319430 secs 2023-03-09T21:31:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:01 smithi038 ceph-mon[92422]: pgmap v13081: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 442 KiB/s rd, 392 KiB/s wr, 44 op/s 2023-03-09T21:31:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23571.319430 secs 2023-03-09T21:31:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:03 smithi122 ceph-mon[93869]: pgmap v13082: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 478 KiB/s rd, 463 KiB/s wr, 23 op/s 2023-03-09T21:31:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:03 smithi181 ceph-mon[94189]: pgmap v13082: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 478 KiB/s rd, 463 KiB/s wr, 23 op/s 2023-03-09T21:31:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:03 smithi038 ceph-mon[92422]: pgmap v13082: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 478 KiB/s rd, 463 KiB/s wr, 23 op/s 2023-03-09T21:31:03.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24035.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:31:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:05 smithi122 ceph-mon[93869]: pgmap v13083: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 437 KiB/s wr, 20 op/s 2023-03-09T21:31:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:05 smithi181 ceph-mon[94189]: pgmap v13083: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 437 KiB/s wr, 20 op/s 2023-03-09T21:31:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:05 smithi038 ceph-mon[92422]: pgmap v13083: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 437 KiB/s wr, 20 op/s 2023-03-09T21:31:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23576.319490 secs 2023-03-09T21:31:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23576.319490 secs 2023-03-09T21:31:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23576.319490 secs 2023-03-09T21:31:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:07 smithi122 ceph-mon[93869]: pgmap v13084: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 615 KiB/s rd, 633 KiB/s wr, 16 op/s 2023-03-09T21:31:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:07 smithi181 ceph-mon[94189]: pgmap v13084: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 615 KiB/s rd, 633 KiB/s wr, 16 op/s 2023-03-09T21:31:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:07 smithi038 ceph-mon[92422]: pgmap v13084: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 615 KiB/s rd, 633 KiB/s wr, 16 op/s 2023-03-09T21:31:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23578.662120 secs 2023-03-09T21:31:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23578.662120 secs 2023-03-09T21:31:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23578.662120 secs 2023-03-09T21:31:08.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24040.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:31:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:09 smithi122 ceph-mon[93869]: pgmap v13085: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 268 KiB/s wr, 12 op/s 2023-03-09T21:31:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:09 smithi181 ceph-mon[94189]: pgmap v13085: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 268 KiB/s wr, 12 op/s 2023-03-09T21:31:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:09 smithi038 ceph-mon[92422]: pgmap v13085: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 268 KiB/s wr, 12 op/s 2023-03-09T21:31:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:10 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:31:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:10 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:31:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:10 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:31:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:10 smithi122 ceph-mon[93869]: osdmap e1031: 12 total, 12 up, 12 in 2023-03-09T21:31:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:10 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:31:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:10 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:31:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:10 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:31:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:10 smithi181 ceph-mon[94189]: osdmap e1031: 12 total, 12 up, 12 in 2023-03-09T21:31:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:10 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:31:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:10 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:31:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:10 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:31:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:10 smithi038 ceph-mon[92422]: osdmap e1031: 12 total, 12 up, 12 in 2023-03-09T21:31:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:11 smithi122 ceph-mon[93869]: pgmap v13087: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 348 KiB/s wr, 27 op/s 2023-03-09T21:31:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23581.319549 secs 2023-03-09T21:31:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:11 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:31:11.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:11 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:31:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:11 smithi181 ceph-mon[94189]: pgmap v13087: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 348 KiB/s wr, 27 op/s 2023-03-09T21:31:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23581.319549 secs 2023-03-09T21:31:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:11 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:31:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:11 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:31:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:11 smithi038 ceph-mon[92422]: pgmap v13087: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 348 KiB/s wr, 27 op/s 2023-03-09T21:31:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23581.319549 secs 2023-03-09T21:31:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:11 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:31:11.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:11 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:31:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:13 smithi122 ceph-mon[93869]: pgmap v13088: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 279 KiB/s wr, 26 op/s 2023-03-09T21:31:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23583.662235 secs 2023-03-09T21:31:13.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:31:13.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:31:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:13 smithi181 ceph-mon[94189]: pgmap v13088: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 279 KiB/s wr, 26 op/s 2023-03-09T21:31:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23583.662235 secs 2023-03-09T21:31:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:31:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:31:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:13 smithi038 ceph-mon[92422]: pgmap v13088: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 279 KiB/s wr, 26 op/s 2023-03-09T21:31:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23583.662235 secs 2023-03-09T21:31:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:31:13.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:31:13.802 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24045.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:31:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:15 smithi122 ceph-mon[93869]: pgmap v13089: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 504 KiB/s wr, 26 op/s 2023-03-09T21:31:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:15 smithi181 ceph-mon[94189]: pgmap v13089: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 504 KiB/s wr, 26 op/s 2023-03-09T21:31:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:15 smithi038 ceph-mon[92422]: pgmap v13089: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 504 KiB/s wr, 26 op/s 2023-03-09T21:31:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23586.319610 secs 2023-03-09T21:31:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23586.319610 secs 2023-03-09T21:31:16.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23586.319610 secs 2023-03-09T21:31:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:17 smithi038 ceph-mon[92422]: pgmap v13090: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 291 KiB/s wr, 22 op/s 2023-03-09T21:31:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:17 smithi122 ceph-mon[93869]: pgmap v13090: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 291 KiB/s wr, 22 op/s 2023-03-09T21:31:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:17 smithi181 ceph-mon[94189]: pgmap v13090: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 291 KiB/s wr, 22 op/s 2023-03-09T21:31:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23588.662351 secs 2023-03-09T21:31:18.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24050.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:31:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23588.662351 secs 2023-03-09T21:31:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23588.662351 secs 2023-03-09T21:31:19.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:19 smithi038 ceph-mon[92422]: pgmap v13091: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 368 KiB/s wr, 23 op/s 2023-03-09T21:31:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:19 smithi122 ceph-mon[93869]: pgmap v13091: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 368 KiB/s wr, 23 op/s 2023-03-09T21:31:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:19 smithi181 ceph-mon[94189]: pgmap v13091: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 368 KiB/s wr, 23 op/s 2023-03-09T21:31:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:21 smithi122 ceph-mon[93869]: pgmap v13092: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 331 KiB/s wr, 9 op/s 2023-03-09T21:31:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23591.319671 secs 2023-03-09T21:31:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:21 smithi122 ceph-mon[93869]: osdmap e1032: 12 total, 12 up, 12 in 2023-03-09T21:31:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:21 smithi181 ceph-mon[94189]: pgmap v13092: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 331 KiB/s wr, 9 op/s 2023-03-09T21:31:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23591.319671 secs 2023-03-09T21:31:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:21 smithi181 ceph-mon[94189]: osdmap e1032: 12 total, 12 up, 12 in 2023-03-09T21:31:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:21 smithi038 ceph-mon[92422]: pgmap v13092: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 331 KiB/s wr, 9 op/s 2023-03-09T21:31:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23591.319671 secs 2023-03-09T21:31:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:21 smithi038 ceph-mon[92422]: osdmap e1032: 12 total, 12 up, 12 in 2023-03-09T21:31:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:23 smithi122 ceph-mon[93869]: pgmap v13094: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 326 KiB/s wr, 8 op/s 2023-03-09T21:31:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23593.662483 secs 2023-03-09T21:31:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:23 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:31:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:23 smithi181 ceph-mon[94189]: pgmap v13094: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 326 KiB/s wr, 8 op/s 2023-03-09T21:31:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23593.662483 secs 2023-03-09T21:31:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:23 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:31:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:23 smithi038 ceph-mon[92422]: pgmap v13094: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 326 KiB/s wr, 8 op/s 2023-03-09T21:31:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23593.662483 secs 2023-03-09T21:31:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:23 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:31:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24055.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:31:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:31:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:31:24.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:31:24.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:31:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:31:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:31:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:31:24.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:31:24.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:31:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:25 smithi122 ceph-mon[93869]: pgmap v13095: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 101 KiB/s wr, 7 op/s 2023-03-09T21:31:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:25 smithi181 ceph-mon[94189]: pgmap v13095: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 101 KiB/s wr, 7 op/s 2023-03-09T21:31:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:25 smithi038 ceph-mon[92422]: pgmap v13095: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 101 KiB/s wr, 7 op/s 2023-03-09T21:31:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23596.319730 secs 2023-03-09T21:31:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23596.319730 secs 2023-03-09T21:31:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23596.319730 secs 2023-03-09T21:31:27.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:27 smithi122 ceph-mon[93869]: pgmap v13096: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 79 KiB/s wr, 7 op/s 2023-03-09T21:31:27.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:27 smithi181 ceph-mon[94189]: pgmap v13096: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 79 KiB/s wr, 7 op/s 2023-03-09T21:31:27.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:27 smithi038 ceph-mon[92422]: pgmap v13096: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 79 KiB/s wr, 7 op/s 2023-03-09T21:31:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23598.662595 secs 2023-03-09T21:31:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23598.662595 secs 2023-03-09T21:31:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23598.662595 secs 2023-03-09T21:31:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24060.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:31:29.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:29 smithi181 ceph-mon[94189]: pgmap v13097: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.6 KiB/s wr, 6 op/s 2023-03-09T21:31:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:29 smithi038 ceph-mon[92422]: pgmap v13097: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.6 KiB/s wr, 6 op/s 2023-03-09T21:31:29.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:29 smithi122 ceph-mon[93869]: pgmap v13097: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.6 KiB/s wr, 6 op/s 2023-03-09T21:31:31.487 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:31 smithi181 ceph-mon[94189]: pgmap v13098: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2023-03-09T21:31:31.488 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23601.319793 secs 2023-03-09T21:31:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:31 smithi038 ceph-mon[92422]: pgmap v13098: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2023-03-09T21:31:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23601.319793 secs 2023-03-09T21:31:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:31 smithi122 ceph-mon[93869]: pgmap v13098: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2023-03-09T21:31:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23601.319793 secs 2023-03-09T21:31:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:33 smithi038 ceph-mon[92422]: pgmap v13099: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.2 KiB/s wr, 8 op/s 2023-03-09T21:31:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23603.662711 secs 2023-03-09T21:31:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:33 smithi122 ceph-mon[93869]: pgmap v13099: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.2 KiB/s wr, 8 op/s 2023-03-09T21:31:33.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23603.662711 secs 2023-03-09T21:31:33.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:33 smithi181 ceph-mon[94189]: pgmap v13099: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 2.2 KiB/s wr, 8 op/s 2023-03-09T21:31:33.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23603.662711 secs 2023-03-09T21:31:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24065.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:31:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:35 smithi038 ceph-mon[92422]: pgmap v13100: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 95 KiB/s wr, 7 op/s 2023-03-09T21:31:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:35 smithi122 ceph-mon[93869]: pgmap v13100: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 95 KiB/s wr, 7 op/s 2023-03-09T21:31:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:35 smithi181 ceph-mon[94189]: pgmap v13100: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 95 KiB/s wr, 7 op/s 2023-03-09T21:31:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23606.319870 secs 2023-03-09T21:31:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23606.319870 secs 2023-03-09T21:31:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23606.319870 secs 2023-03-09T21:31:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:37 smithi038 ceph-mon[92422]: pgmap v13101: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 96 KiB/s wr, 9 op/s 2023-03-09T21:31:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:37 smithi122 ceph-mon[93869]: pgmap v13101: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 96 KiB/s wr, 9 op/s 2023-03-09T21:31:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:37 smithi181 ceph-mon[94189]: pgmap v13101: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 96 KiB/s wr, 9 op/s 2023-03-09T21:31:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23608.662825 secs 2023-03-09T21:31:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23608.662825 secs 2023-03-09T21:31:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23608.662825 secs 2023-03-09T21:31:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24070.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:31:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:39 smithi122 ceph-mon[93869]: pgmap v13102: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 184 KiB/s wr, 8 op/s 2023-03-09T21:31:39.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:39 smithi181 ceph-mon[94189]: pgmap v13102: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 184 KiB/s wr, 8 op/s 2023-03-09T21:31:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:39 smithi038 ceph-mon[92422]: pgmap v13102: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 184 KiB/s wr, 8 op/s 2023-03-09T21:31:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:31:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:31:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:31:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:31:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:31:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:31:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:41 smithi122 ceph-mon[93869]: pgmap v13103: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 184 KiB/s wr, 7 op/s 2023-03-09T21:31:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23611.319955 secs 2023-03-09T21:31:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:31:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:31:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:41 smithi181 ceph-mon[94189]: pgmap v13103: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 184 KiB/s wr, 7 op/s 2023-03-09T21:31:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23611.319955 secs 2023-03-09T21:31:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:31:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:31:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:41 smithi038 ceph-mon[92422]: pgmap v13103: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 184 KiB/s wr, 7 op/s 2023-03-09T21:31:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23611.319955 secs 2023-03-09T21:31:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:31:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:31:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:43 smithi122 ceph-mon[93869]: pgmap v13104: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 184 KiB/s wr, 9 op/s 2023-03-09T21:31:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:43 smithi181 ceph-mon[94189]: pgmap v13104: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 184 KiB/s wr, 9 op/s 2023-03-09T21:31:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:43 smithi038 ceph-mon[92422]: pgmap v13104: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 184 KiB/s wr, 9 op/s 2023-03-09T21:31:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24075.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:31:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:45 smithi122 ceph-mon[93869]: pgmap v13105: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 287 KiB/s wr, 8 op/s 2023-03-09T21:31:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:45 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:31:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:45 smithi181 ceph-mon[94189]: pgmap v13105: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 287 KiB/s wr, 8 op/s 2023-03-09T21:31:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:45 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:31:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:45 smithi038 ceph-mon[92422]: pgmap v13105: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 287 KiB/s wr, 8 op/s 2023-03-09T21:31:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:45 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:31:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23616.319983 secs 2023-03-09T21:31:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:46 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:31:46.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:46 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:31:46.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:46 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:31:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23616.319983 secs 2023-03-09T21:31:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:46 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:31:46.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:46 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:31:46.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:46 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:31:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23616.319983 secs 2023-03-09T21:31:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:46 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:31:46.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:46 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:31:46.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:46 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:31:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:47 smithi122 ceph-mon[93869]: pgmap v13106: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 402 KiB/s rd, 385 KiB/s wr, 10 op/s 2023-03-09T21:31:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:47 smithi181 ceph-mon[94189]: pgmap v13106: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 402 KiB/s rd, 385 KiB/s wr, 10 op/s 2023-03-09T21:31:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:47 smithi038 ceph-mon[92422]: pgmap v13106: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 402 KiB/s rd, 385 KiB/s wr, 10 op/s 2023-03-09T21:31:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:48 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 23618.663082 secs 2023-03-09T21:31:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:48 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 23618.663082 secs 2023-03-09T21:31:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:48 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 23618.663082 secs 2023-03-09T21:31:48.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24080.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:31:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:49 smithi122 ceph-mon[93869]: pgmap v13107: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 546 KiB/s rd, 531 KiB/s wr, 9 op/s 2023-03-09T21:31:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:31:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:31:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:49 smithi181 ceph-mon[94189]: pgmap v13107: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 546 KiB/s rd, 531 KiB/s wr, 9 op/s 2023-03-09T21:31:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:31:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:31:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:49 smithi038 ceph-mon[92422]: pgmap v13107: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 546 KiB/s rd, 531 KiB/s wr, 9 op/s 2023-03-09T21:31:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:31:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:31:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:50 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:31:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:50 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:31:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:50 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:31:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:50 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:31:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:50 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:31:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:50 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:31:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:51 smithi122 ceph-mon[93869]: pgmap v13108: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 443 KiB/s wr, 8 op/s 2023-03-09T21:31:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23621.320045 secs 2023-03-09T21:31:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:51 smithi181 ceph-mon[94189]: pgmap v13108: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 443 KiB/s wr, 8 op/s 2023-03-09T21:31:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23621.320045 secs 2023-03-09T21:31:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:51 smithi038 ceph-mon[92422]: pgmap v13108: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 443 KiB/s wr, 8 op/s 2023-03-09T21:31:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23621.320045 secs 2023-03-09T21:31:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:31:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:31:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:31:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:31:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:31:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:31:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:53 smithi122 ceph-mon[93869]: pgmap v13109: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 499 KiB/s rd, 650 KiB/s wr, 12 op/s 2023-03-09T21:31:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:53 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 23623.663200 secs 2023-03-09T21:31:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:53 smithi181 ceph-mon[94189]: pgmap v13109: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 499 KiB/s rd, 650 KiB/s wr, 12 op/s 2023-03-09T21:31:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:53 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 23623.663200 secs 2023-03-09T21:31:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:53 smithi038 ceph-mon[92422]: pgmap v13109: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 499 KiB/s rd, 650 KiB/s wr, 12 op/s 2023-03-09T21:31:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:53 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 23623.663200 secs 2023-03-09T21:31:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24085.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:31:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:55 smithi122 ceph-mon[93869]: pgmap v13110: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 497 KiB/s rd, 650 KiB/s wr, 11 op/s 2023-03-09T21:31:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:55 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:31:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:55 smithi181 ceph-mon[94189]: pgmap v13110: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 497 KiB/s rd, 650 KiB/s wr, 11 op/s 2023-03-09T21:31:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:55 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:31:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:55 smithi038 ceph-mon[92422]: pgmap v13110: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 497 KiB/s rd, 650 KiB/s wr, 11 op/s 2023-03-09T21:31:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:55 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:31:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23626.320105 secs 2023-03-09T21:31:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:31:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:31:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23626.320105 secs 2023-03-09T21:31:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:31:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:31:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23626.320105 secs 2023-03-09T21:31:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:31:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:31:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:57 smithi122 ceph-mon[93869]: pgmap v13111: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 842 KiB/s rd, 813 KiB/s wr, 15 op/s 2023-03-09T21:31:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:31:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:31:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:57 smithi181 ceph-mon[94189]: pgmap v13111: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 842 KiB/s rd, 813 KiB/s wr, 15 op/s 2023-03-09T21:31:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:31:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:31:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:57 smithi038 ceph-mon[92422]: pgmap v13111: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 842 KiB/s rd, 813 KiB/s wr, 15 op/s 2023-03-09T21:31:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:31:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:31:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:58 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 23628.663312 secs 2023-03-09T21:31:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:58 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 23628.663312 secs 2023-03-09T21:31:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:58 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 23628.663312 secs 2023-03-09T21:31:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24090.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:31:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:31:59 smithi122 ceph-mon[93869]: pgmap v13112: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 620 KiB/s wr, 11 op/s 2023-03-09T21:31:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:31:59 smithi181 ceph-mon[94189]: pgmap v13112: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 620 KiB/s wr, 11 op/s 2023-03-09T21:31:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:31:59 smithi038 ceph-mon[92422]: pgmap v13112: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 620 KiB/s wr, 11 op/s 2023-03-09T21:32:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:01 smithi122 ceph-mon[93869]: pgmap v13113: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 473 KiB/s wr, 10 op/s 2023-03-09T21:32:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23631.320167 secs 2023-03-09T21:32:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:01 smithi181 ceph-mon[94189]: pgmap v13113: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 473 KiB/s wr, 10 op/s 2023-03-09T21:32:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23631.320167 secs 2023-03-09T21:32:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:01 smithi038 ceph-mon[92422]: pgmap v13113: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 473 KiB/s wr, 10 op/s 2023-03-09T21:32:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23631.320167 secs 2023-03-09T21:32:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:03 smithi122 ceph-mon[93869]: pgmap v13114: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 562 KiB/s rd, 568 KiB/s wr, 15 op/s 2023-03-09T21:32:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23633.663431 secs 2023-03-09T21:32:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:03 smithi181 ceph-mon[94189]: pgmap v13114: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 562 KiB/s rd, 568 KiB/s wr, 15 op/s 2023-03-09T21:32:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23633.663431 secs 2023-03-09T21:32:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:03 smithi038 ceph-mon[92422]: pgmap v13114: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 562 KiB/s rd, 568 KiB/s wr, 15 op/s 2023-03-09T21:32:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23633.663431 secs 2023-03-09T21:32:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24095.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:32:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:05 smithi122 ceph-mon[93869]: pgmap v13115: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 383 KiB/s wr, 24 op/s 2023-03-09T21:32:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:05 smithi122 ceph-mon[93869]: osdmap e1033: 12 total, 12 up, 12 in 2023-03-09T21:32:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:05 smithi181 ceph-mon[94189]: pgmap v13115: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 383 KiB/s wr, 24 op/s 2023-03-09T21:32:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:05 smithi181 ceph-mon[94189]: osdmap e1033: 12 total, 12 up, 12 in 2023-03-09T21:32:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:05 smithi038 ceph-mon[92422]: pgmap v13115: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 383 KiB/s wr, 24 op/s 2023-03-09T21:32:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:05 smithi038 ceph-mon[92422]: osdmap e1033: 12 total, 12 up, 12 in 2023-03-09T21:32:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23636.320229 secs 2023-03-09T21:32:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:06 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:32:06.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:32:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23636.320229 secs 2023-03-09T21:32:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:06 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:32:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:32:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23636.320229 secs 2023-03-09T21:32:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:06 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:32:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:32:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:07 smithi122 ceph-mon[93869]: pgmap v13117: 129 pgs: 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 125 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 382 KiB/s wr, 27 op/s 2023-03-09T21:32:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:32:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:32:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:07 smithi181 ceph-mon[94189]: pgmap v13117: 129 pgs: 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 125 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 382 KiB/s wr, 27 op/s 2023-03-09T21:32:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:32:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:32:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:07 smithi038 ceph-mon[92422]: pgmap v13117: 129 pgs: 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 125 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 382 KiB/s wr, 27 op/s 2023-03-09T21:32:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:32:07.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:32:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23638.663545 secs 2023-03-09T21:32:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23638.663545 secs 2023-03-09T21:32:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23638.663545 secs 2023-03-09T21:32:08.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24100.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:32:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:09 smithi122 ceph-mon[93869]: pgmap v13118: 129 pgs: 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 125 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 382 KiB/s wr, 27 op/s 2023-03-09T21:32:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:32:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:32:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:09 smithi181 ceph-mon[94189]: pgmap v13118: 129 pgs: 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 125 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 382 KiB/s wr, 27 op/s 2023-03-09T21:32:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:32:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:32:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:09 smithi038 ceph-mon[92422]: pgmap v13118: 129 pgs: 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 125 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 382 KiB/s wr, 27 op/s 2023-03-09T21:32:09.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:32:09.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:32:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:11 smithi122 ceph-mon[93869]: pgmap v13119: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 491 KiB/s wr, 28 op/s 2023-03-09T21:32:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23641.320308 secs 2023-03-09T21:32:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:11 smithi181 ceph-mon[94189]: pgmap v13119: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 491 KiB/s wr, 28 op/s 2023-03-09T21:32:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23641.320308 secs 2023-03-09T21:32:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:11 smithi038 ceph-mon[92422]: pgmap v13119: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 491 KiB/s wr, 28 op/s 2023-03-09T21:32:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23641.320308 secs 2023-03-09T21:32:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:13 smithi122 ceph-mon[93869]: pgmap v13120: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 378 KiB/s wr, 25 op/s 2023-03-09T21:32:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23643.663659 secs 2023-03-09T21:32:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:13 smithi181 ceph-mon[94189]: pgmap v13120: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 378 KiB/s wr, 25 op/s 2023-03-09T21:32:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23643.663659 secs 2023-03-09T21:32:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:13 smithi038 ceph-mon[92422]: pgmap v13120: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 378 KiB/s wr, 25 op/s 2023-03-09T21:32:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23643.663659 secs 2023-03-09T21:32:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24105.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:32:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:15 smithi122 ceph-mon[93869]: pgmap v13121: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 365 KiB/s wr, 10 op/s 2023-03-09T21:32:15.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:15 smithi122 ceph-mon[93869]: osdmap e1034: 12 total, 12 up, 12 in 2023-03-09T21:32:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:15 smithi181 ceph-mon[94189]: pgmap v13121: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 365 KiB/s wr, 10 op/s 2023-03-09T21:32:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:15 smithi181 ceph-mon[94189]: osdmap e1034: 12 total, 12 up, 12 in 2023-03-09T21:32:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:15 smithi038 ceph-mon[92422]: pgmap v13121: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 365 KiB/s wr, 10 op/s 2023-03-09T21:32:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:15 smithi038 ceph-mon[92422]: osdmap e1034: 12 total, 12 up, 12 in 2023-03-09T21:32:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23646.320367 secs 2023-03-09T21:32:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23646.320367 secs 2023-03-09T21:32:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23646.320367 secs 2023-03-09T21:32:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:17 smithi122 ceph-mon[93869]: pgmap v13123: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 255 KiB/s wr, 9 op/s 2023-03-09T21:32:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:17 smithi181 ceph-mon[94189]: pgmap v13123: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 255 KiB/s wr, 9 op/s 2023-03-09T21:32:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:17 smithi038 ceph-mon[92422]: pgmap v13123: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 255 KiB/s wr, 9 op/s 2023-03-09T21:32:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23648.663766 secs 2023-03-09T21:32:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23648.663766 secs 2023-03-09T21:32:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23648.663766 secs 2023-03-09T21:32:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24110.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:32:19.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:19 smithi122 ceph-mon[93869]: pgmap v13124: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 255 KiB/s wr, 10 op/s 2023-03-09T21:32:19.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:19 smithi181 ceph-mon[94189]: pgmap v13124: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 255 KiB/s wr, 10 op/s 2023-03-09T21:32:19.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:19 smithi038 ceph-mon[92422]: pgmap v13124: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 255 KiB/s wr, 10 op/s 2023-03-09T21:32:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:21 smithi122 ceph-mon[93869]: pgmap v13125: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 146 KiB/s wr, 9 op/s 2023-03-09T21:32:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23651.320426 secs 2023-03-09T21:32:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:21 smithi181 ceph-mon[94189]: pgmap v13125: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 146 KiB/s wr, 9 op/s 2023-03-09T21:32:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23651.320426 secs 2023-03-09T21:32:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:21 smithi038 ceph-mon[92422]: pgmap v13125: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 146 KiB/s wr, 9 op/s 2023-03-09T21:32:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23651.320426 secs 2023-03-09T21:32:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:23 smithi122 ceph-mon[93869]: pgmap v13126: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 169 KiB/s wr, 9 op/s 2023-03-09T21:32:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23653.663881 secs 2023-03-09T21:32:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:23 smithi181 ceph-mon[94189]: pgmap v13126: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 169 KiB/s wr, 9 op/s 2023-03-09T21:32:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23653.663881 secs 2023-03-09T21:32:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:23 smithi038 ceph-mon[92422]: pgmap v13126: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 169 KiB/s wr, 9 op/s 2023-03-09T21:32:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23653.663881 secs 2023-03-09T21:32:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24115.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:32:24.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:32:24.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:32:24.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:32:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:25 smithi181 ceph-mon[94189]: pgmap v13127: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 218 KiB/s wr, 9 op/s 2023-03-09T21:32:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:32:25.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:32:25.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:32:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:25 smithi038 ceph-mon[92422]: pgmap v13127: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 218 KiB/s wr, 9 op/s 2023-03-09T21:32:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:32:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:32:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:32:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:25 smithi122 ceph-mon[93869]: pgmap v13127: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 218 KiB/s wr, 9 op/s 2023-03-09T21:32:25.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:32:25.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:32:25.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:32:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23656.320489 secs 2023-03-09T21:32:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23656.320489 secs 2023-03-09T21:32:26.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23656.320489 secs 2023-03-09T21:32:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:27 smithi038 ceph-mon[92422]: pgmap v13128: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 102 KiB/s wr, 11 op/s 2023-03-09T21:32:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:27 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:32:27.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:27 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:32:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:27 smithi122 ceph-mon[93869]: pgmap v13128: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 102 KiB/s wr, 11 op/s 2023-03-09T21:32:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:27 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:32:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:27 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:32:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:27 smithi181 ceph-mon[94189]: pgmap v13128: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 102 KiB/s wr, 11 op/s 2023-03-09T21:32:27.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:27 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:32:27.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:27 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:32:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23658.664015 secs 2023-03-09T21:32:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24120.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:32:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23658.664015 secs 2023-03-09T21:32:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23658.664015 secs 2023-03-09T21:32:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:29 smithi122 ceph-mon[93869]: pgmap v13129: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 72 KiB/s wr, 7 op/s 2023-03-09T21:32:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:29 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:32:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:29 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:32:29.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:29 smithi181 ceph-mon[94189]: pgmap v13129: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 72 KiB/s wr, 7 op/s 2023-03-09T21:32:29.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:29 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:32:29.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:29 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:32:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:29 smithi038 ceph-mon[92422]: pgmap v13129: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 72 KiB/s wr, 7 op/s 2023-03-09T21:32:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:29 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:32:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:29 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:32:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:31 smithi122 ceph-mon[93869]: pgmap v13130: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 72 KiB/s wr, 7 op/s 2023-03-09T21:32:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23661.320551 secs 2023-03-09T21:32:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:31 smithi181 ceph-mon[94189]: pgmap v13130: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 72 KiB/s wr, 7 op/s 2023-03-09T21:32:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23661.320551 secs 2023-03-09T21:32:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:31 smithi038 ceph-mon[92422]: pgmap v13130: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 72 KiB/s wr, 7 op/s 2023-03-09T21:32:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23661.320551 secs 2023-03-09T21:32:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24125.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:32:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:33 smithi122 ceph-mon[93869]: pgmap v13131: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 9 op/s 2023-03-09T21:32:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23663.664140 secs 2023-03-09T21:32:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:33 smithi181 ceph-mon[94189]: pgmap v13131: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 9 op/s 2023-03-09T21:32:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23663.664140 secs 2023-03-09T21:32:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:33 smithi038 ceph-mon[92422]: pgmap v13131: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 72 KiB/s wr, 9 op/s 2023-03-09T21:32:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23663.664140 secs 2023-03-09T21:32:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:35 smithi122 ceph-mon[93869]: pgmap v13132: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 146 KiB/s wr, 8 op/s 2023-03-09T21:32:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:35 smithi181 ceph-mon[94189]: pgmap v13132: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 146 KiB/s wr, 8 op/s 2023-03-09T21:32:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:35 smithi038 ceph-mon[92422]: pgmap v13132: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 146 KiB/s wr, 8 op/s 2023-03-09T21:32:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23666.320611 secs 2023-03-09T21:32:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23666.320611 secs 2023-03-09T21:32:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23666.320611 secs 2023-03-09T21:32:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:37 smithi122 ceph-mon[93869]: pgmap v13133: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 95 KiB/s wr, 9 op/s 2023-03-09T21:32:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:37 smithi181 ceph-mon[94189]: pgmap v13133: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 95 KiB/s wr, 9 op/s 2023-03-09T21:32:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:37 smithi038 ceph-mon[92422]: pgmap v13133: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 95 KiB/s wr, 9 op/s 2023-03-09T21:32:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24130.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:32:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23668.664253 secs 2023-03-09T21:32:38.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23668.664253 secs 2023-03-09T21:32:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23668.664253 secs 2023-03-09T21:32:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:39 smithi122 ceph-mon[93869]: pgmap v13134: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 95 KiB/s wr, 7 op/s 2023-03-09T21:32:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:39 smithi181 ceph-mon[94189]: pgmap v13134: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 95 KiB/s wr, 7 op/s 2023-03-09T21:32:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:39 smithi038 ceph-mon[92422]: pgmap v13134: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 95 KiB/s wr, 7 op/s 2023-03-09T21:32:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:41 smithi122 ceph-mon[93869]: pgmap v13135: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 95 KiB/s wr, 6 op/s 2023-03-09T21:32:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23671.320671 secs 2023-03-09T21:32:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:41 smithi181 ceph-mon[94189]: pgmap v13135: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 95 KiB/s wr, 6 op/s 2023-03-09T21:32:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23671.320671 secs 2023-03-09T21:32:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:41 smithi038 ceph-mon[92422]: pgmap v13135: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 95 KiB/s wr, 6 op/s 2023-03-09T21:32:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23671.320671 secs 2023-03-09T21:32:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24135.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:32:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:43 smithi122 ceph-mon[93869]: pgmap v13136: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 95 KiB/s wr, 8 op/s 2023-03-09T21:32:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23673.664381 secs 2023-03-09T21:32:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:43 smithi181 ceph-mon[94189]: pgmap v13136: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 95 KiB/s wr, 8 op/s 2023-03-09T21:32:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23673.664381 secs 2023-03-09T21:32:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:43 smithi038 ceph-mon[92422]: pgmap v13136: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 95 KiB/s wr, 8 op/s 2023-03-09T21:32:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23673.664381 secs 2023-03-09T21:32:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:45 smithi122 ceph-mon[93869]: pgmap v13137: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 280 KiB/s wr, 7 op/s 2023-03-09T21:32:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:45 smithi181 ceph-mon[94189]: pgmap v13137: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 280 KiB/s wr, 7 op/s 2023-03-09T21:32:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:45 smithi038 ceph-mon[92422]: pgmap v13137: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 280 KiB/s wr, 7 op/s 2023-03-09T21:32:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23676.320740 secs 2023-03-09T21:32:46.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23676.320740 secs 2023-03-09T21:32:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23676.320740 secs 2023-03-09T21:32:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:47 smithi122 ceph-mon[93869]: pgmap v13138: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 454 KiB/s rd, 321 KiB/s wr, 9 op/s 2023-03-09T21:32:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:47 smithi181 ceph-mon[94189]: pgmap v13138: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 454 KiB/s rd, 321 KiB/s wr, 9 op/s 2023-03-09T21:32:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:47 smithi038 ceph-mon[92422]: pgmap v13138: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 454 KiB/s rd, 321 KiB/s wr, 9 op/s 2023-03-09T21:32:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24140.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:32:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23678.664498 secs 2023-03-09T21:32:48.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23678.664498 secs 2023-03-09T21:32:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23678.664498 secs 2023-03-09T21:32:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:49 smithi122 ceph-mon[93869]: pgmap v13139: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 647 KiB/s rd, 321 KiB/s wr, 8 op/s 2023-03-09T21:32:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:49 smithi181 ceph-mon[94189]: pgmap v13139: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 647 KiB/s rd, 321 KiB/s wr, 8 op/s 2023-03-09T21:32:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:49 smithi038 ceph-mon[92422]: pgmap v13139: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 647 KiB/s rd, 321 KiB/s wr, 8 op/s 2023-03-09T21:32:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:51 smithi122 ceph-mon[93869]: pgmap v13140: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 658 KiB/s rd, 321 KiB/s wr, 7 op/s 2023-03-09T21:32:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23681.320800 secs 2023-03-09T21:32:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:32:51.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:32:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:51 smithi181 ceph-mon[94189]: pgmap v13140: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 658 KiB/s rd, 321 KiB/s wr, 7 op/s 2023-03-09T21:32:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23681.320800 secs 2023-03-09T21:32:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:32:51.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:32:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:51 smithi038 ceph-mon[92422]: pgmap v13140: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 658 KiB/s rd, 321 KiB/s wr, 7 op/s 2023-03-09T21:32:52.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23681.320800 secs 2023-03-09T21:32:52.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:32:52.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:32:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24145.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:32:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:53 smithi122 ceph-mon[93869]: pgmap v13141: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 458 KiB/s wr, 11 op/s 2023-03-09T21:32:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23683.664620 secs 2023-03-09T21:32:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:32:53.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:32:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:53 smithi181 ceph-mon[94189]: pgmap v13141: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 458 KiB/s wr, 11 op/s 2023-03-09T21:32:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23683.664620 secs 2023-03-09T21:32:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:32:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:32:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:53 smithi038 ceph-mon[92422]: pgmap v13141: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 458 KiB/s wr, 11 op/s 2023-03-09T21:32:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23683.664620 secs 2023-03-09T21:32:54.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:32:54.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:32:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:55 smithi122 ceph-mon[93869]: pgmap v13142: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 796 KiB/s rd, 458 KiB/s wr, 10 op/s 2023-03-09T21:32:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:32:55.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:32:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:55 smithi181 ceph-mon[94189]: pgmap v13142: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 796 KiB/s rd, 458 KiB/s wr, 10 op/s 2023-03-09T21:32:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:32:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:32:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:55 smithi038 ceph-mon[92422]: pgmap v13142: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 796 KiB/s rd, 458 KiB/s wr, 10 op/s 2023-03-09T21:32:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:32:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:32:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23686.320875 secs 2023-03-09T21:32:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23686.320875 secs 2023-03-09T21:32:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23686.320875 secs 2023-03-09T21:32:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:57 smithi122 ceph-mon[93869]: pgmap v13143: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 653 KiB/s rd, 443 KiB/s wr, 12 op/s 2023-03-09T21:32:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:32:57.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:32:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:57 smithi181 ceph-mon[94189]: pgmap v13143: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 653 KiB/s rd, 443 KiB/s wr, 12 op/s 2023-03-09T21:32:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:32:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:32:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:57 smithi038 ceph-mon[92422]: pgmap v13143: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 653 KiB/s rd, 443 KiB/s wr, 12 op/s 2023-03-09T21:32:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:32:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:32:58.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24150.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:32:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23688.664709 secs 2023-03-09T21:32:58.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23688.664709 secs 2023-03-09T21:32:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23688.664709 secs 2023-03-09T21:32:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:32:59 smithi122 ceph-mon[93869]: pgmap v13144: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 307 KiB/s wr, 10 op/s 2023-03-09T21:32:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:32:59 smithi181 ceph-mon[94189]: pgmap v13144: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 307 KiB/s wr, 10 op/s 2023-03-09T21:32:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:32:59 smithi038 ceph-mon[92422]: pgmap v13144: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 307 KiB/s wr, 10 op/s 2023-03-09T21:33:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:01 smithi122 ceph-mon[93869]: pgmap v13145: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 424 KiB/s wr, 10 op/s 2023-03-09T21:33:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23691.320936 secs 2023-03-09T21:33:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:01 smithi181 ceph-mon[94189]: pgmap v13145: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 424 KiB/s wr, 10 op/s 2023-03-09T21:33:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23691.320936 secs 2023-03-09T21:33:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:01 smithi038 ceph-mon[92422]: pgmap v13145: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 424 KiB/s wr, 10 op/s 2023-03-09T21:33:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23691.320936 secs 2023-03-09T21:33:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24155.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:33:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:03 smithi122 ceph-mon[93869]: pgmap v13146: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 424 KiB/s wr, 13 op/s 2023-03-09T21:33:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23693.664791 secs 2023-03-09T21:33:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:03 smithi181 ceph-mon[94189]: pgmap v13146: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 424 KiB/s wr, 13 op/s 2023-03-09T21:33:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23693.664791 secs 2023-03-09T21:33:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:03 smithi038 ceph-mon[92422]: pgmap v13146: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 424 KiB/s wr, 13 op/s 2023-03-09T21:33:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23693.664791 secs 2023-03-09T21:33:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:05 smithi122 ceph-mon[93869]: pgmap v13147: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 299 KiB/s wr, 21 op/s 2023-03-09T21:33:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:05 smithi181 ceph-mon[94189]: pgmap v13147: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 299 KiB/s wr, 21 op/s 2023-03-09T21:33:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:05 smithi038 ceph-mon[92422]: pgmap v13147: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 299 KiB/s wr, 21 op/s 2023-03-09T21:33:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23696.320991 secs 2023-03-09T21:33:06.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23696.320991 secs 2023-03-09T21:33:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23696.320991 secs 2023-03-09T21:33:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:07 smithi122 ceph-mon[93869]: pgmap v13148: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 714 KiB/s rd, 726 KiB/s wr, 26 op/s 2023-03-09T21:33:08.040 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:33:08.041 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:33:08.042 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:07 smithi038 ceph-mon[92422]: pgmap v13148: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 714 KiB/s rd, 726 KiB/s wr, 26 op/s 2023-03-09T21:33:08.042 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:33:08.042 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:33:08.042 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:07 smithi181 ceph-mon[94189]: pgmap v13148: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 714 KiB/s rd, 726 KiB/s wr, 26 op/s 2023-03-09T21:33:08.043 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:33:08.043 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:33:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24160.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:33:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23698.664909 secs 2023-03-09T21:33:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:33:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:33:08.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23698.664909 secs 2023-03-09T21:33:08.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:33:08.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:33:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23698.664909 secs 2023-03-09T21:33:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:33:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:33:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:09 smithi122 ceph-mon[93869]: pgmap v13149: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 542 KiB/s rd, 556 KiB/s wr, 23 op/s 2023-03-09T21:33:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:09 smithi181 ceph-mon[94189]: pgmap v13149: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 542 KiB/s rd, 556 KiB/s wr, 23 op/s 2023-03-09T21:33:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:09 smithi038 ceph-mon[92422]: pgmap v13149: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 542 KiB/s rd, 556 KiB/s wr, 23 op/s 2023-03-09T21:33:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:11 smithi122 ceph-mon[93869]: pgmap v13150: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 542 KiB/s rd, 556 KiB/s wr, 23 op/s 2023-03-09T21:33:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23701.321052 secs 2023-03-09T21:33:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:11 smithi181 ceph-mon[94189]: pgmap v13150: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 542 KiB/s rd, 556 KiB/s wr, 23 op/s 2023-03-09T21:33:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23701.321052 secs 2023-03-09T21:33:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:11 smithi038 ceph-mon[92422]: pgmap v13150: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 542 KiB/s rd, 556 KiB/s wr, 23 op/s 2023-03-09T21:33:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23701.321052 secs 2023-03-09T21:33:13.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24165.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:33:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:13 smithi122 ceph-mon[93869]: pgmap v13151: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 555 KiB/s rd, 567 KiB/s wr, 25 op/s 2023-03-09T21:33:13.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23703.665004 secs 2023-03-09T21:33:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:13 smithi181 ceph-mon[94189]: pgmap v13151: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 555 KiB/s rd, 567 KiB/s wr, 25 op/s 2023-03-09T21:33:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23703.665004 secs 2023-03-09T21:33:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:13 smithi038 ceph-mon[92422]: pgmap v13151: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 555 KiB/s rd, 567 KiB/s wr, 25 op/s 2023-03-09T21:33:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23703.665004 secs 2023-03-09T21:33:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:15 smithi122 ceph-mon[93869]: pgmap v13152: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 601 KiB/s rd, 615 KiB/s wr, 22 op/s 2023-03-09T21:33:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:15 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:33:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:15 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:33:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:15 smithi181 ceph-mon[94189]: pgmap v13152: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 601 KiB/s rd, 615 KiB/s wr, 22 op/s 2023-03-09T21:33:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:15 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:33:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:15 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:33:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:15 smithi038 ceph-mon[92422]: pgmap v13152: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 601 KiB/s rd, 615 KiB/s wr, 22 op/s 2023-03-09T21:33:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:15 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:33:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:15 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:33:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23706.321113 secs 2023-03-09T21:33:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:33:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:33:16.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23706.321113 secs 2023-03-09T21:33:16.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:33:16.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:33:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23706.321113 secs 2023-03-09T21:33:17.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:33:17.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:33:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:17 smithi122 ceph-mon[93869]: pgmap v13153: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 615 KiB/s rd, 624 KiB/s wr, 14 op/s 2023-03-09T21:33:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:17 smithi181 ceph-mon[94189]: pgmap v13153: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 615 KiB/s rd, 624 KiB/s wr, 14 op/s 2023-03-09T21:33:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:17 smithi038 ceph-mon[92422]: pgmap v13153: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 615 KiB/s rd, 624 KiB/s wr, 14 op/s 2023-03-09T21:33:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24170.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:33:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23708.665137 secs 2023-03-09T21:33:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23708.665137 secs 2023-03-09T21:33:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23708.665137 secs 2023-03-09T21:33:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:19 smithi122 ceph-mon[93869]: pgmap v13154: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 197 KiB/s wr, 8 op/s 2023-03-09T21:33:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:19 smithi181 ceph-mon[94189]: pgmap v13154: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 197 KiB/s wr, 8 op/s 2023-03-09T21:33:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:19 smithi038 ceph-mon[92422]: pgmap v13154: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 197 KiB/s wr, 8 op/s 2023-03-09T21:33:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:21 smithi122 ceph-mon[93869]: pgmap v13155: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 197 KiB/s wr, 9 op/s 2023-03-09T21:33:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23711.321175 secs 2023-03-09T21:33:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:21 smithi181 ceph-mon[94189]: pgmap v13155: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 197 KiB/s wr, 9 op/s 2023-03-09T21:33:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23711.321175 secs 2023-03-09T21:33:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:21 smithi038 ceph-mon[92422]: pgmap v13155: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 197 KiB/s wr, 9 op/s 2023-03-09T21:33:22.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23711.321175 secs 2023-03-09T21:33:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24175.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:33:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:23 smithi122 ceph-mon[93869]: pgmap v13156: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 228 KiB/s wr, 12 op/s 2023-03-09T21:33:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23713.665255 secs 2023-03-09T21:33:23.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:23 smithi181 ceph-mon[94189]: pgmap v13156: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 228 KiB/s wr, 12 op/s 2023-03-09T21:33:23.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23713.665255 secs 2023-03-09T21:33:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:23 smithi038 ceph-mon[92422]: pgmap v13156: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 228 KiB/s wr, 12 op/s 2023-03-09T21:33:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23713.665255 secs 2023-03-09T21:33:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:25 smithi122 ceph-mon[93869]: pgmap v13157: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 358 KiB/s wr, 12 op/s 2023-03-09T21:33:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:33:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23716.321238 secs 2023-03-09T21:33:25.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:33:25.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:33:25.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:33:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:25 smithi181 ceph-mon[94189]: pgmap v13157: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 358 KiB/s wr, 12 op/s 2023-03-09T21:33:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:33:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23716.321238 secs 2023-03-09T21:33:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:33:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:33:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:33:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:25 smithi038 ceph-mon[92422]: pgmap v13157: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 358 KiB/s wr, 12 op/s 2023-03-09T21:33:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:33:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23716.321238 secs 2023-03-09T21:33:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:33:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:33:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:33:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:27 smithi122 ceph-mon[93869]: pgmap v13158: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 360 KiB/s wr, 15 op/s 2023-03-09T21:33:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:27 smithi181 ceph-mon[94189]: pgmap v13158: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 360 KiB/s wr, 15 op/s 2023-03-09T21:33:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:27 smithi038 ceph-mon[92422]: pgmap v13158: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 360 KiB/s wr, 15 op/s 2023-03-09T21:33:28.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24180.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:33:29.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23718.665373 secs 2023-03-09T21:33:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23718.665373 secs 2023-03-09T21:33:29.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23718.665373 secs 2023-03-09T21:33:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:29 smithi038 ceph-mon[92422]: pgmap v13159: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 321 KiB/s rd, 340 KiB/s wr, 11 op/s 2023-03-09T21:33:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:29 smithi122 ceph-mon[93869]: pgmap v13159: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 321 KiB/s rd, 340 KiB/s wr, 11 op/s 2023-03-09T21:33:30.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:29 smithi181 ceph-mon[94189]: pgmap v13159: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 321 KiB/s rd, 340 KiB/s wr, 11 op/s 2023-03-09T21:33:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:30 smithi122 ceph-mon[93869]: osdmap e1035: 12 total, 12 up, 12 in 2023-03-09T21:33:31.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:30 smithi122 ceph-mon[93869]: pgmap v13161: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 464 KiB/s wr, 13 op/s 2023-03-09T21:33:31.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23721.322672 secs 2023-03-09T21:33:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:30 smithi181 ceph-mon[94189]: osdmap e1035: 12 total, 12 up, 12 in 2023-03-09T21:33:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:30 smithi181 ceph-mon[94189]: pgmap v13161: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 464 KiB/s wr, 13 op/s 2023-03-09T21:33:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23721.322672 secs 2023-03-09T21:33:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:30 smithi038 ceph-mon[92422]: osdmap e1035: 12 total, 12 up, 12 in 2023-03-09T21:33:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:30 smithi038 ceph-mon[92422]: pgmap v13161: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 464 KiB/s wr, 13 op/s 2023-03-09T21:33:31.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23721.322672 secs 2023-03-09T21:33:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:32 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:33:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:32 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:33:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:32 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:33:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:32 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:33:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:32 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:33:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:32 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:33:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:33 smithi038 ceph-mon[92422]: pgmap v13162: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 584 KiB/s rd, 597 KiB/s wr, 13 op/s 2023-03-09T21:33:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23723.665440 secs 2023-03-09T21:33:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:33 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:33:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:33 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:33:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:33 smithi122 ceph-mon[93869]: pgmap v13162: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 584 KiB/s rd, 597 KiB/s wr, 13 op/s 2023-03-09T21:33:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23723.665440 secs 2023-03-09T21:33:33.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:33 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:33:33.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:33 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:33:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:33 smithi181 ceph-mon[94189]: pgmap v13162: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 584 KiB/s rd, 597 KiB/s wr, 13 op/s 2023-03-09T21:33:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23723.665440 secs 2023-03-09T21:33:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:33 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:33:33.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:33 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:33:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24185.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:33:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:35 smithi038 ceph-mon[92422]: pgmap v13163: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 288 KiB/s wr, 10 op/s 2023-03-09T21:33:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:35 smithi122 ceph-mon[93869]: pgmap v13163: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 288 KiB/s wr, 10 op/s 2023-03-09T21:33:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:35 smithi181 ceph-mon[94189]: pgmap v13163: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 288 KiB/s wr, 10 op/s 2023-03-09T21:33:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23726.322731 secs 2023-03-09T21:33:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23726.322731 secs 2023-03-09T21:33:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23726.322731 secs 2023-03-09T21:33:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:37 smithi122 ceph-mon[93869]: pgmap v13164: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 307 KiB/s wr, 10 op/s 2023-03-09T21:33:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:37 smithi181 ceph-mon[94189]: pgmap v13164: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 307 KiB/s wr, 10 op/s 2023-03-09T21:33:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:37 smithi038 ceph-mon[92422]: pgmap v13164: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 307 KiB/s wr, 10 op/s 2023-03-09T21:33:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23728.665567 secs 2023-03-09T21:33:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23728.665567 secs 2023-03-09T21:33:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23728.665567 secs 2023-03-09T21:33:38.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24190.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:33:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:39 smithi122 ceph-mon[93869]: pgmap v13165: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 306 KiB/s wr, 10 op/s 2023-03-09T21:33:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:39 smithi181 ceph-mon[94189]: pgmap v13165: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 306 KiB/s wr, 10 op/s 2023-03-09T21:33:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:39 smithi038 ceph-mon[92422]: pgmap v13165: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 306 KiB/s wr, 10 op/s 2023-03-09T21:33:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:41 smithi122 ceph-mon[93869]: pgmap v13166: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 248 KiB/s wr, 9 op/s 2023-03-09T21:33:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23731.322791 secs 2023-03-09T21:33:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:41 smithi122 ceph-mon[93869]: osdmap e1036: 12 total, 12 up, 12 in 2023-03-09T21:33:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:33:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:33:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:41 smithi181 ceph-mon[94189]: pgmap v13166: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 248 KiB/s wr, 9 op/s 2023-03-09T21:33:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23731.322791 secs 2023-03-09T21:33:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:41 smithi181 ceph-mon[94189]: osdmap e1036: 12 total, 12 up, 12 in 2023-03-09T21:33:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:33:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:33:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:41 smithi038 ceph-mon[92422]: pgmap v13166: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 248 KiB/s wr, 9 op/s 2023-03-09T21:33:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23731.322791 secs 2023-03-09T21:33:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:41 smithi038 ceph-mon[92422]: osdmap e1036: 12 total, 12 up, 12 in 2023-03-09T21:33:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:33:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:33:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:43 smithi122 ceph-mon[93869]: pgmap v13168: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 419 KiB/s wr, 10 op/s 2023-03-09T21:33:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:43 smithi181 ceph-mon[94189]: pgmap v13168: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 419 KiB/s wr, 10 op/s 2023-03-09T21:33:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:43 smithi038 ceph-mon[92422]: pgmap v13168: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 419 KiB/s wr, 10 op/s 2023-03-09T21:33:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24195.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:33:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:45 smithi122 ceph-mon[93869]: pgmap v13169: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 419 KiB/s wr, 9 op/s 2023-03-09T21:33:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:45 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:33:45.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:45 smithi181 ceph-mon[94189]: pgmap v13169: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 419 KiB/s wr, 9 op/s 2023-03-09T21:33:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:45 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:33:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:45 smithi038 ceph-mon[92422]: pgmap v13169: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 419 KiB/s wr, 9 op/s 2023-03-09T21:33:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:45 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:33:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23736.322853 secs 2023-03-09T21:33:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:46 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:33:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:46 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:33:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:46 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:33:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23736.322853 secs 2023-03-09T21:33:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:46 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:33:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:46 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:33:46.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:46 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:33:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23736.322853 secs 2023-03-09T21:33:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:46 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:33:46.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:46 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:33:46.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:46 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:33:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:47 smithi038 ceph-mon[92422]: pgmap v13170: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 488 KiB/s wr, 11 op/s 2023-03-09T21:33:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:47 smithi122 ceph-mon[93869]: pgmap v13170: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 488 KiB/s wr, 11 op/s 2023-03-09T21:33:47.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:47 smithi181 ceph-mon[94189]: pgmap v13170: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 488 KiB/s wr, 11 op/s 2023-03-09T21:33:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:33:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:33:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:33:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:33:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:33:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:33:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24200.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:33:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:49 smithi038 ceph-mon[92422]: pgmap v13171: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 488 KiB/s wr, 11 op/s 2023-03-09T21:33:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:49 smithi122 ceph-mon[93869]: pgmap v13171: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 488 KiB/s wr, 11 op/s 2023-03-09T21:33:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:49 smithi181 ceph-mon[94189]: pgmap v13171: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 488 KiB/s wr, 11 op/s 2023-03-09T21:33:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:51 smithi122 ceph-mon[93869]: pgmap v13172: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 488 KiB/s wr, 11 op/s 2023-03-09T21:33:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23741.322911 secs 2023-03-09T21:33:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:51 smithi181 ceph-mon[94189]: pgmap v13172: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 488 KiB/s wr, 11 op/s 2023-03-09T21:33:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23741.322911 secs 2023-03-09T21:33:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:51 smithi038 ceph-mon[92422]: pgmap v13172: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 488 KiB/s wr, 11 op/s 2023-03-09T21:33:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23741.322911 secs 2023-03-09T21:33:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:33:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:33:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:33:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:33:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:33:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:33:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:53 smithi122 ceph-mon[93869]: pgmap v13173: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 721 KiB/s rd, 705 KiB/s wr, 14 op/s 2023-03-09T21:33:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:33:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:33:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:53 smithi181 ceph-mon[94189]: pgmap v13173: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 721 KiB/s rd, 705 KiB/s wr, 14 op/s 2023-03-09T21:33:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:33:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:33:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:53 smithi038 ceph-mon[92422]: pgmap v13173: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 721 KiB/s rd, 705 KiB/s wr, 14 op/s 2023-03-09T21:33:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:33:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:33:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24205.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:33:55.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:55 smithi122 ceph-mon[93869]: pgmap v13174: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 400 KiB/s wr, 10 op/s 2023-03-09T21:33:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:55 smithi181 ceph-mon[94189]: pgmap v13174: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 400 KiB/s wr, 10 op/s 2023-03-09T21:33:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:55 smithi038 ceph-mon[92422]: pgmap v13174: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 400 KiB/s wr, 10 op/s 2023-03-09T21:33:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23746.322977 secs 2023-03-09T21:33:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23746.322977 secs 2023-03-09T21:33:56.753 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23746.322977 secs 2023-03-09T21:33:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:57 smithi122 ceph-mon[93869]: pgmap v13175: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 617 KiB/s rd, 601 KiB/s wr, 14 op/s 2023-03-09T21:33:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:57 smithi181 ceph-mon[94189]: pgmap v13175: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 617 KiB/s rd, 601 KiB/s wr, 14 op/s 2023-03-09T21:33:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:57 smithi038 ceph-mon[92422]: pgmap v13175: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 617 KiB/s rd, 601 KiB/s wr, 14 op/s 2023-03-09T21:33:58.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24210.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:33:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:33:59 smithi122 ceph-mon[93869]: pgmap v13176: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 477 KiB/s wr, 10 op/s 2023-03-09T21:33:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:33:59 smithi181 ceph-mon[94189]: pgmap v13176: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 477 KiB/s wr, 10 op/s 2023-03-09T21:33:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:33:59 smithi038 ceph-mon[92422]: pgmap v13176: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 477 KiB/s wr, 10 op/s 2023-03-09T21:34:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:34:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:34:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:34:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:34:00.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:34:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:34:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:01 smithi122 ceph-mon[93869]: pgmap v13177: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 477 KiB/s wr, 10 op/s 2023-03-09T21:34:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23751.323007 secs 2023-03-09T21:34:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:34:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:34:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:01 smithi181 ceph-mon[94189]: pgmap v13177: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 477 KiB/s wr, 10 op/s 2023-03-09T21:34:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23751.323007 secs 2023-03-09T21:34:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:34:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:34:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:01 smithi038 ceph-mon[92422]: pgmap v13177: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 477 KiB/s wr, 10 op/s 2023-03-09T21:34:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23751.323007 secs 2023-03-09T21:34:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:34:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:34:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:03 smithi122 ceph-mon[93869]: pgmap v13178: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 703 KiB/s rd, 717 KiB/s wr, 15 op/s 2023-03-09T21:34:03.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23753.666187 secs 2023-03-09T21:34:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:03 smithi181 ceph-mon[94189]: pgmap v13178: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 703 KiB/s rd, 717 KiB/s wr, 15 op/s 2023-03-09T21:34:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23753.666187 secs 2023-03-09T21:34:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:03 smithi038 ceph-mon[92422]: pgmap v13178: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 703 KiB/s rd, 717 KiB/s wr, 15 op/s 2023-03-09T21:34:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23753.666187 secs 2023-03-09T21:34:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24215.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:34:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:05 smithi122 ceph-mon[93869]: pgmap v13179: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 463 KiB/s wr, 23 op/s 2023-03-09T21:34:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:05 smithi181 ceph-mon[94189]: pgmap v13179: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 463 KiB/s wr, 23 op/s 2023-03-09T21:34:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:05 smithi038 ceph-mon[92422]: pgmap v13179: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 463 KiB/s wr, 23 op/s 2023-03-09T21:34:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23756.323065 secs 2023-03-09T21:34:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:06 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:34:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:06 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:34:06.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:34:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23756.323065 secs 2023-03-09T21:34:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:06 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:34:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:06 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:34:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:34:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23756.323065 secs 2023-03-09T21:34:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:06 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:34:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:06 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:34:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:34:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:07 smithi122 ceph-mon[93869]: pgmap v13180: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 560 KiB/s wr, 27 op/s 2023-03-09T21:34:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:07 smithi181 ceph-mon[94189]: pgmap v13180: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 560 KiB/s wr, 27 op/s 2023-03-09T21:34:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:07 smithi038 ceph-mon[92422]: pgmap v13180: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 560 KiB/s wr, 27 op/s 2023-03-09T21:34:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23758.666609 secs 2023-03-09T21:34:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23758.666609 secs 2023-03-09T21:34:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23758.666609 secs 2023-03-09T21:34:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24220.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:34:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:09 smithi122 ceph-mon[93869]: pgmap v13181: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 326 KiB/s rd, 360 KiB/s wr, 23 op/s 2023-03-09T21:34:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:09 smithi181 ceph-mon[94189]: pgmap v13181: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 326 KiB/s rd, 360 KiB/s wr, 23 op/s 2023-03-09T21:34:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:09 smithi038 ceph-mon[92422]: pgmap v13181: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 326 KiB/s rd, 360 KiB/s wr, 23 op/s 2023-03-09T21:34:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:10 smithi122 ceph-mon[93869]: osdmap e1037: 12 total, 12 up, 12 in 2023-03-09T21:34:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:10 smithi181 ceph-mon[94189]: osdmap e1037: 12 total, 12 up, 12 in 2023-03-09T21:34:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:10 smithi038 ceph-mon[92422]: osdmap e1037: 12 total, 12 up, 12 in 2023-03-09T21:34:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:11 smithi122 ceph-mon[93869]: pgmap v13183: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 431 KiB/s wr, 27 op/s 2023-03-09T21:34:11.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23761.323126 secs 2023-03-09T21:34:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:11 smithi181 ceph-mon[94189]: pgmap v13183: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 431 KiB/s wr, 27 op/s 2023-03-09T21:34:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23761.323126 secs 2023-03-09T21:34:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:11 smithi038 ceph-mon[92422]: pgmap v13183: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 431 KiB/s wr, 27 op/s 2023-03-09T21:34:11.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23761.323126 secs 2023-03-09T21:34:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:13 smithi122 ceph-mon[93869]: pgmap v13184: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 143 KiB/s wr, 22 op/s 2023-03-09T21:34:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23763.666741 secs 2023-03-09T21:34:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:13 smithi181 ceph-mon[94189]: pgmap v13184: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 143 KiB/s wr, 22 op/s 2023-03-09T21:34:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23763.666741 secs 2023-03-09T21:34:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:13 smithi038 ceph-mon[92422]: pgmap v13184: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 143 KiB/s wr, 22 op/s 2023-03-09T21:34:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23763.666741 secs 2023-03-09T21:34:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24225.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:34:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:15 smithi122 ceph-mon[93869]: pgmap v13185: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 117 KiB/s wr, 8 op/s 2023-03-09T21:34:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:15 smithi181 ceph-mon[94189]: pgmap v13185: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 117 KiB/s wr, 8 op/s 2023-03-09T21:34:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:15 smithi038 ceph-mon[92422]: pgmap v13185: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 117 KiB/s wr, 8 op/s 2023-03-09T21:34:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23766.323186 secs 2023-03-09T21:34:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23766.323186 secs 2023-03-09T21:34:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23766.323186 secs 2023-03-09T21:34:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:17 smithi122 ceph-mon[93869]: pgmap v13186: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T21:34:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:17 smithi181 ceph-mon[94189]: pgmap v13186: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T21:34:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:17 smithi038 ceph-mon[92422]: pgmap v13186: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T21:34:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23768.666852 secs 2023-03-09T21:34:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23768.666852 secs 2023-03-09T21:34:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23768.666852 secs 2023-03-09T21:34:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24230.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:34:19.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:19 smithi122 ceph-mon[93869]: pgmap v13187: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T21:34:19.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:19 smithi181 ceph-mon[94189]: pgmap v13187: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T21:34:19.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:19 smithi038 ceph-mon[92422]: pgmap v13187: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 7 op/s 2023-03-09T21:34:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:21 smithi038 ceph-mon[92422]: pgmap v13188: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 7 op/s 2023-03-09T21:34:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23771.323245 secs 2023-03-09T21:34:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:21 smithi038 ceph-mon[92422]: osdmap e1038: 12 total, 12 up, 12 in 2023-03-09T21:34:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:21 smithi122 ceph-mon[93869]: pgmap v13188: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 7 op/s 2023-03-09T21:34:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23771.323245 secs 2023-03-09T21:34:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:21 smithi122 ceph-mon[93869]: osdmap e1038: 12 total, 12 up, 12 in 2023-03-09T21:34:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:21 smithi181 ceph-mon[94189]: pgmap v13188: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 22 KiB/s wr, 7 op/s 2023-03-09T21:34:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23771.323245 secs 2023-03-09T21:34:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:21 smithi181 ceph-mon[94189]: osdmap e1038: 12 total, 12 up, 12 in 2023-03-09T21:34:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:23 smithi038 ceph-mon[92422]: pgmap v13190: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 47 KiB/s wr, 7 op/s 2023-03-09T21:34:23.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23773.666913 secs 2023-03-09T21:34:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:23 smithi122 ceph-mon[93869]: pgmap v13190: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 47 KiB/s wr, 7 op/s 2023-03-09T21:34:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23773.666913 secs 2023-03-09T21:34:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:23 smithi181 ceph-mon[94189]: pgmap v13190: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 47 KiB/s wr, 7 op/s 2023-03-09T21:34:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23773.666913 secs 2023-03-09T21:34:23.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24235.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:34:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:25 smithi122 ceph-mon[93869]: pgmap v13191: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 47 KiB/s wr, 8 op/s 2023-03-09T21:34:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:25 smithi181 ceph-mon[94189]: pgmap v13191: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 47 KiB/s wr, 8 op/s 2023-03-09T21:34:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:25 smithi038 ceph-mon[92422]: pgmap v13191: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 47 KiB/s wr, 8 op/s 2023-03-09T21:34:26.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23776.323305 secs 2023-03-09T21:34:26.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:26 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:34:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23776.323305 secs 2023-03-09T21:34:26.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:26 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:34:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23776.323305 secs 2023-03-09T21:34:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:26 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:34:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:27 smithi122 ceph-mon[93869]: pgmap v13192: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T21:34:27.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:34:27.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:34:27.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:34:27.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:27 smithi181 ceph-mon[94189]: pgmap v13192: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T21:34:27.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:34:27.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:34:27.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:34:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:27 smithi038 ceph-mon[92422]: pgmap v13192: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T21:34:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:34:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:34:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:34:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23778.667035 secs 2023-03-09T21:34:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23778.667035 secs 2023-03-09T21:34:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23778.667035 secs 2023-03-09T21:34:28.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24240.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:34:29.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:29 smithi122 ceph-mon[93869]: pgmap v13193: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T21:34:29.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:29 smithi181 ceph-mon[94189]: pgmap v13193: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T21:34:29.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:29 smithi038 ceph-mon[92422]: pgmap v13193: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T21:34:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:31 smithi122 ceph-mon[93869]: pgmap v13194: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 126 KiB/s wr, 7 op/s 2023-03-09T21:34:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23781.323367 secs 2023-03-09T21:34:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:31 smithi181 ceph-mon[94189]: pgmap v13194: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 126 KiB/s wr, 7 op/s 2023-03-09T21:34:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23781.323367 secs 2023-03-09T21:34:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:31 smithi038 ceph-mon[92422]: pgmap v13194: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 126 KiB/s wr, 7 op/s 2023-03-09T21:34:31.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23781.323367 secs 2023-03-09T21:34:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:33 smithi122 ceph-mon[93869]: pgmap v13195: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 120 KiB/s wr, 10 op/s 2023-03-09T21:34:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23783.667147 secs 2023-03-09T21:34:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:33 smithi181 ceph-mon[94189]: pgmap v13195: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 120 KiB/s wr, 10 op/s 2023-03-09T21:34:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23783.667147 secs 2023-03-09T21:34:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:33 smithi038 ceph-mon[92422]: pgmap v13195: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 120 KiB/s wr, 10 op/s 2023-03-09T21:34:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23783.667147 secs 2023-03-09T21:34:33.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24245.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:34:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:35 smithi122 ceph-mon[93869]: pgmap v13196: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 97 KiB/s wr, 7 op/s 2023-03-09T21:34:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:35 smithi181 ceph-mon[94189]: pgmap v13196: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 97 KiB/s wr, 7 op/s 2023-03-09T21:34:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:35 smithi038 ceph-mon[92422]: pgmap v13196: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 97 KiB/s wr, 7 op/s 2023-03-09T21:34:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23786.323427 secs 2023-03-09T21:34:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23786.323427 secs 2023-03-09T21:34:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23786.323427 secs 2023-03-09T21:34:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:37 smithi122 ceph-mon[93869]: pgmap v13197: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 97 KiB/s wr, 10 op/s 2023-03-09T21:34:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:37 smithi181 ceph-mon[94189]: pgmap v13197: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 97 KiB/s wr, 10 op/s 2023-03-09T21:34:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:37 smithi038 ceph-mon[92422]: pgmap v13197: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 97 KiB/s wr, 10 op/s 2023-03-09T21:34:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23788.667267 secs 2023-03-09T21:34:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23788.667267 secs 2023-03-09T21:34:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23788.667267 secs 2023-03-09T21:34:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24250.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:34:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:39 smithi122 ceph-mon[93869]: pgmap v13198: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 97 KiB/s wr, 7 op/s 2023-03-09T21:34:39.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:39 smithi181 ceph-mon[94189]: pgmap v13198: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 97 KiB/s wr, 7 op/s 2023-03-09T21:34:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:39 smithi038 ceph-mon[92422]: pgmap v13198: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 97 KiB/s wr, 7 op/s 2023-03-09T21:34:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:34:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:34:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:34:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:34:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:34:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:34:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:41 smithi122 ceph-mon[93869]: pgmap v13199: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 281 KiB/s wr, 8 op/s 2023-03-09T21:34:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23791.323486 secs 2023-03-09T21:34:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:34:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:34:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:41 smithi181 ceph-mon[94189]: pgmap v13199: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 281 KiB/s wr, 8 op/s 2023-03-09T21:34:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23791.323486 secs 2023-03-09T21:34:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:34:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:34:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:41 smithi038 ceph-mon[92422]: pgmap v13199: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 281 KiB/s wr, 8 op/s 2023-03-09T21:34:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23791.323486 secs 2023-03-09T21:34:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:34:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:34:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:43 smithi122 ceph-mon[93869]: pgmap v13200: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 196 KiB/s wr, 10 op/s 2023-03-09T21:34:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:43 smithi181 ceph-mon[94189]: pgmap v13200: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 196 KiB/s wr, 10 op/s 2023-03-09T21:34:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:43 smithi038 ceph-mon[92422]: pgmap v13200: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 196 KiB/s wr, 10 op/s 2023-03-09T21:34:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24255.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:34:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:45 smithi122 ceph-mon[93869]: pgmap v13201: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 257 KiB/s wr, 8 op/s 2023-03-09T21:34:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:45 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:34:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:45 smithi181 ceph-mon[94189]: pgmap v13201: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 257 KiB/s wr, 8 op/s 2023-03-09T21:34:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:45 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:34:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:45 smithi038 ceph-mon[92422]: pgmap v13201: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 257 KiB/s wr, 8 op/s 2023-03-09T21:34:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:45 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:34:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23796.323565 secs 2023-03-09T21:34:46.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:46 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:34:46.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:46 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:34:46.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:46 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:34:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23796.323565 secs 2023-03-09T21:34:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:46 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:34:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:46 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:34:46.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:46 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:34:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23796.323565 secs 2023-03-09T21:34:46.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:46 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:34:46.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:46 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:34:46.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:46 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:34:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:47 smithi122 ceph-mon[93869]: pgmap v13202: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 553 KiB/s rd, 414 KiB/s wr, 11 op/s 2023-03-09T21:34:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:47 smithi181 ceph-mon[94189]: pgmap v13202: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 553 KiB/s rd, 414 KiB/s wr, 11 op/s 2023-03-09T21:34:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:47 smithi038 ceph-mon[92422]: pgmap v13202: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 553 KiB/s rd, 414 KiB/s wr, 11 op/s 2023-03-09T21:34:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24260.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:34:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:49 smithi122 ceph-mon[93869]: pgmap v13203: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 414 KiB/s wr, 8 op/s 2023-03-09T21:34:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:49 smithi181 ceph-mon[94189]: pgmap v13203: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 414 KiB/s wr, 8 op/s 2023-03-09T21:34:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:49 smithi038 ceph-mon[92422]: pgmap v13203: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 414 KiB/s wr, 8 op/s 2023-03-09T21:34:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:51 smithi122 ceph-mon[93869]: pgmap v13204: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 653 KiB/s rd, 414 KiB/s wr, 9 op/s 2023-03-09T21:34:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23801.323624 secs 2023-03-09T21:34:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:51 smithi181 ceph-mon[94189]: pgmap v13204: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 653 KiB/s rd, 414 KiB/s wr, 9 op/s 2023-03-09T21:34:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23801.323624 secs 2023-03-09T21:34:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:51 smithi038 ceph-mon[92422]: pgmap v13204: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 653 KiB/s rd, 414 KiB/s wr, 9 op/s 2023-03-09T21:34:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23801.323624 secs 2023-03-09T21:34:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:34:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:34:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:34:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:34:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:34:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:34:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:53 smithi122 ceph-mon[93869]: pgmap v13205: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 755 KiB/s rd, 418 KiB/s wr, 12 op/s 2023-03-09T21:34:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23803.667638 secs 2023-03-09T21:34:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:34:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:34:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:53 smithi181 ceph-mon[94189]: pgmap v13205: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 755 KiB/s rd, 418 KiB/s wr, 12 op/s 2023-03-09T21:34:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23803.667638 secs 2023-03-09T21:34:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:34:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:34:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:53 smithi038 ceph-mon[92422]: pgmap v13205: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 755 KiB/s rd, 418 KiB/s wr, 12 op/s 2023-03-09T21:34:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23803.667638 secs 2023-03-09T21:34:53.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:34:53.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:34:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24265.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:34:54.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:54 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:34:54.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:54 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:34:54.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:54 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:34:54.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:54 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:34:54.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:54 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:34:54.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:54 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:34:54.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:54 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:34:54.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:54 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:34:54.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:54 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:34:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:55 smithi122 ceph-mon[93869]: pgmap v13206: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 754 KiB/s rd, 418 KiB/s wr, 11 op/s 2023-03-09T21:34:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:55 smithi181 ceph-mon[94189]: pgmap v13206: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 754 KiB/s rd, 418 KiB/s wr, 11 op/s 2023-03-09T21:34:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:55 smithi038 ceph-mon[92422]: pgmap v13206: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 754 KiB/s rd, 418 KiB/s wr, 11 op/s 2023-03-09T21:34:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23806.323685 secs 2023-03-09T21:34:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23806.323685 secs 2023-03-09T21:34:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23806.323685 secs 2023-03-09T21:34:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:57 smithi122 ceph-mon[93869]: pgmap v13207: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 507 KiB/s rd, 401 KiB/s wr, 13 op/s 2023-03-09T21:34:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:34:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:34:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:57 smithi181 ceph-mon[94189]: pgmap v13207: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 507 KiB/s rd, 401 KiB/s wr, 13 op/s 2023-03-09T21:34:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:34:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:34:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:57 smithi038 ceph-mon[92422]: pgmap v13207: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 507 KiB/s rd, 401 KiB/s wr, 13 op/s 2023-03-09T21:34:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:34:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:34:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23808.667770 secs 2023-03-09T21:34:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23808.667770 secs 2023-03-09T21:34:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23808.667770 secs 2023-03-09T21:34:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24270.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:34:59.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:34:59 smithi122 ceph-mon[93869]: pgmap v13208: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 570 KiB/s rd, 465 KiB/s wr, 11 op/s 2023-03-09T21:34:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:34:59 smithi181 ceph-mon[94189]: pgmap v13208: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 570 KiB/s rd, 465 KiB/s wr, 11 op/s 2023-03-09T21:34:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:34:59 smithi038 ceph-mon[92422]: pgmap v13208: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 570 KiB/s rd, 465 KiB/s wr, 11 op/s 2023-03-09T21:35:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:35:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:35:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:35:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:35:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:35:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:35:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:01 smithi122 ceph-mon[93869]: pgmap v13209: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 570 KiB/s rd, 465 KiB/s wr, 11 op/s 2023-03-09T21:35:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23811.323747 secs 2023-03-09T21:35:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:01 smithi181 ceph-mon[94189]: pgmap v13209: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 570 KiB/s rd, 465 KiB/s wr, 11 op/s 2023-03-09T21:35:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23811.323747 secs 2023-03-09T21:35:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:01 smithi038 ceph-mon[92422]: pgmap v13209: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 570 KiB/s rd, 465 KiB/s wr, 11 op/s 2023-03-09T21:35:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23811.323747 secs 2023-03-09T21:35:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:03 smithi122 ceph-mon[93869]: pgmap v13210: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 566 KiB/s rd, 562 KiB/s wr, 14 op/s 2023-03-09T21:35:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23813.667883 secs 2023-03-09T21:35:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:03 smithi181 ceph-mon[94189]: pgmap v13210: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 566 KiB/s rd, 562 KiB/s wr, 14 op/s 2023-03-09T21:35:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23813.667883 secs 2023-03-09T21:35:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:03 smithi038 ceph-mon[92422]: pgmap v13210: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 566 KiB/s rd, 562 KiB/s wr, 14 op/s 2023-03-09T21:35:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23813.667883 secs 2023-03-09T21:35:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24275.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:35:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:05 smithi122 ceph-mon[93869]: pgmap v13211: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 493 KiB/s wr, 13 op/s 2023-03-09T21:35:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:05 smithi181 ceph-mon[94189]: pgmap v13211: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 493 KiB/s wr, 13 op/s 2023-03-09T21:35:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:05 smithi038 ceph-mon[92422]: pgmap v13211: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 493 KiB/s wr, 13 op/s 2023-03-09T21:35:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23816.323809 secs 2023-03-09T21:35:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23816.323809 secs 2023-03-09T21:35:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23816.323809 secs 2023-03-09T21:35:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:07 smithi122 ceph-mon[93869]: pgmap v13212: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 738 KiB/s rd, 756 KiB/s wr, 18 op/s 2023-03-09T21:35:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:35:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:35:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:07 smithi181 ceph-mon[94189]: pgmap v13212: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 738 KiB/s rd, 756 KiB/s wr, 18 op/s 2023-03-09T21:35:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:35:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:35:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:07 smithi038 ceph-mon[92422]: pgmap v13212: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 738 KiB/s rd, 756 KiB/s wr, 18 op/s 2023-03-09T21:35:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:35:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:35:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23818.668001 secs 2023-03-09T21:35:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23818.668001 secs 2023-03-09T21:35:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23818.668001 secs 2023-03-09T21:35:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24280.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:35:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:09 smithi122 ceph-mon[93869]: pgmap v13213: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 682 KiB/s rd, 702 KiB/s wr, 15 op/s 2023-03-09T21:35:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:35:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:35:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:09 smithi181 ceph-mon[94189]: pgmap v13213: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 682 KiB/s rd, 702 KiB/s wr, 15 op/s 2023-03-09T21:35:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:35:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:35:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:09 smithi038 ceph-mon[92422]: pgmap v13213: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 682 KiB/s rd, 702 KiB/s wr, 15 op/s 2023-03-09T21:35:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:35:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:35:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:10 smithi122 ceph-mon[93869]: osdmap e1039: 12 total, 12 up, 12 in 2023-03-09T21:35:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:10 smithi181 ceph-mon[94189]: osdmap e1039: 12 total, 12 up, 12 in 2023-03-09T21:35:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:10 smithi038 ceph-mon[92422]: osdmap e1039: 12 total, 12 up, 12 in 2023-03-09T21:35:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:11 smithi122 ceph-mon[93869]: pgmap v13215: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 660 KiB/s rd, 701 KiB/s wr, 29 op/s 2023-03-09T21:35:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23821.323868 secs 2023-03-09T21:35:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:11 smithi181 ceph-mon[94189]: pgmap v13215: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 660 KiB/s rd, 701 KiB/s wr, 29 op/s 2023-03-09T21:35:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23821.323868 secs 2023-03-09T21:35:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:11 smithi038 ceph-mon[92422]: pgmap v13215: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 660 KiB/s rd, 701 KiB/s wr, 29 op/s 2023-03-09T21:35:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23821.323868 secs 2023-03-09T21:35:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:13 smithi122 ceph-mon[93869]: pgmap v13216: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 597 KiB/s rd, 638 KiB/s wr, 29 op/s 2023-03-09T21:35:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23823.668137 secs 2023-03-09T21:35:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:13 smithi181 ceph-mon[94189]: pgmap v13216: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 597 KiB/s rd, 638 KiB/s wr, 29 op/s 2023-03-09T21:35:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23823.668137 secs 2023-03-09T21:35:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:13 smithi038 ceph-mon[92422]: pgmap v13216: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 597 KiB/s rd, 638 KiB/s wr, 29 op/s 2023-03-09T21:35:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23823.668137 secs 2023-03-09T21:35:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24285.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:35:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:15 smithi122 ceph-mon[93869]: pgmap v13217: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 452 KiB/s rd, 494 KiB/s wr, 27 op/s 2023-03-09T21:35:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:15 smithi181 ceph-mon[94189]: pgmap v13217: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 452 KiB/s rd, 494 KiB/s wr, 27 op/s 2023-03-09T21:35:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:15 smithi038 ceph-mon[92422]: pgmap v13217: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 452 KiB/s rd, 494 KiB/s wr, 27 op/s 2023-03-09T21:35:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23826.323953 secs 2023-03-09T21:35:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23826.323953 secs 2023-03-09T21:35:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23826.323953 secs 2023-03-09T21:35:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:17 smithi122 ceph-mon[93869]: pgmap v13218: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 312 KiB/s wr, 24 op/s 2023-03-09T21:35:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:35:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:35:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:17 smithi181 ceph-mon[94189]: pgmap v13218: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 312 KiB/s wr, 24 op/s 2023-03-09T21:35:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:35:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:35:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:17 smithi038 ceph-mon[92422]: pgmap v13218: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 312 KiB/s wr, 24 op/s 2023-03-09T21:35:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:35:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:35:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23828.668262 secs 2023-03-09T21:35:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23828.668262 secs 2023-03-09T21:35:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23828.668262 secs 2023-03-09T21:35:18.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24290.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:35:19.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:19 smithi122 ceph-mon[93869]: pgmap v13219: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 315 KiB/s wr, 24 op/s 2023-03-09T21:35:19.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:19 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:35:19.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:19 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:35:19.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:19 smithi181 ceph-mon[94189]: pgmap v13219: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 315 KiB/s wr, 24 op/s 2023-03-09T21:35:19.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:19 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:35:19.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:19 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:35:19.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:19 smithi038 ceph-mon[92422]: pgmap v13219: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 315 KiB/s wr, 24 op/s 2023-03-09T21:35:19.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:19 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:35:19.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:19 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:35:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:21 smithi122 ceph-mon[93869]: pgmap v13220: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 277 KiB/s wr, 11 op/s 2023-03-09T21:35:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23831.324015 secs 2023-03-09T21:35:21.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:21 smithi122 ceph-mon[93869]: osdmap e1040: 12 total, 12 up, 12 in 2023-03-09T21:35:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:21 smithi181 ceph-mon[94189]: pgmap v13220: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 277 KiB/s wr, 11 op/s 2023-03-09T21:35:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23831.324015 secs 2023-03-09T21:35:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:21 smithi181 ceph-mon[94189]: osdmap e1040: 12 total, 12 up, 12 in 2023-03-09T21:35:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:21 smithi038 ceph-mon[92422]: pgmap v13220: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 277 KiB/s wr, 11 op/s 2023-03-09T21:35:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23831.324015 secs 2023-03-09T21:35:21.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:21 smithi038 ceph-mon[92422]: osdmap e1040: 12 total, 12 up, 12 in 2023-03-09T21:35:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:23 smithi122 ceph-mon[93869]: pgmap v13222: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 137 KiB/s wr, 9 op/s 2023-03-09T21:35:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23833.668383 secs 2023-03-09T21:35:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:23 smithi181 ceph-mon[94189]: pgmap v13222: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 137 KiB/s wr, 9 op/s 2023-03-09T21:35:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23833.668383 secs 2023-03-09T21:35:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:23 smithi038 ceph-mon[92422]: pgmap v13222: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 137 KiB/s wr, 9 op/s 2023-03-09T21:35:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23833.668383 secs 2023-03-09T21:35:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24295.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:35:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:25 smithi122 ceph-mon[93869]: pgmap v13223: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 137 KiB/s wr, 8 op/s 2023-03-09T21:35:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:25 smithi181 ceph-mon[94189]: pgmap v13223: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 137 KiB/s wr, 8 op/s 2023-03-09T21:35:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:25 smithi038 ceph-mon[92422]: pgmap v13223: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 137 KiB/s wr, 8 op/s 2023-03-09T21:35:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23836.324084 secs 2023-03-09T21:35:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23836.324084 secs 2023-03-09T21:35:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23836.324084 secs 2023-03-09T21:35:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:27 smithi122 ceph-mon[93869]: pgmap v13224: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 83 KiB/s wr, 7 op/s 2023-03-09T21:35:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:35:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:35:27.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:35:27.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:35:27.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:27 smithi181 ceph-mon[94189]: pgmap v13224: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 83 KiB/s wr, 7 op/s 2023-03-09T21:35:27.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:35:27.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:35:27.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:35:27.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:35:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:27 smithi038 ceph-mon[92422]: pgmap v13224: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 83 KiB/s wr, 7 op/s 2023-03-09T21:35:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:35:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:35:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:35:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:35:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23838.668448 secs 2023-03-09T21:35:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23838.668448 secs 2023-03-09T21:35:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23838.668448 secs 2023-03-09T21:35:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24300.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:35:29.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:29 smithi122 ceph-mon[93869]: pgmap v13225: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 79 KiB/s wr, 6 op/s 2023-03-09T21:35:29.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:29 smithi181 ceph-mon[94189]: pgmap v13225: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 79 KiB/s wr, 6 op/s 2023-03-09T21:35:29.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:29 smithi038 ceph-mon[92422]: pgmap v13225: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 79 KiB/s wr, 6 op/s 2023-03-09T21:35:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:31 smithi122 ceph-mon[93869]: pgmap v13226: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 79 KiB/s wr, 6 op/s 2023-03-09T21:35:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23841.324144 secs 2023-03-09T21:35:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:31 smithi181 ceph-mon[94189]: pgmap v13226: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 79 KiB/s wr, 6 op/s 2023-03-09T21:35:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23841.324144 secs 2023-03-09T21:35:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:31 smithi038 ceph-mon[92422]: pgmap v13226: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 79 KiB/s wr, 6 op/s 2023-03-09T21:35:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23841.324144 secs 2023-03-09T21:35:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:33 smithi122 ceph-mon[93869]: pgmap v13227: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 69 KiB/s wr, 8 op/s 2023-03-09T21:35:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23843.668563 secs 2023-03-09T21:35:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:33 smithi181 ceph-mon[94189]: pgmap v13227: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 69 KiB/s wr, 8 op/s 2023-03-09T21:35:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23843.668563 secs 2023-03-09T21:35:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:33 smithi038 ceph-mon[92422]: pgmap v13227: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 69 KiB/s wr, 8 op/s 2023-03-09T21:35:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23843.668563 secs 2023-03-09T21:35:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24305.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:35:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:35 smithi122 ceph-mon[93869]: pgmap v13228: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 67 KiB/s wr, 6 op/s 2023-03-09T21:35:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:35 smithi181 ceph-mon[94189]: pgmap v13228: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 67 KiB/s wr, 6 op/s 2023-03-09T21:35:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:35 smithi038 ceph-mon[92422]: pgmap v13228: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 67 KiB/s wr, 6 op/s 2023-03-09T21:35:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23846.324203 secs 2023-03-09T21:35:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23846.324203 secs 2023-03-09T21:35:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23846.324203 secs 2023-03-09T21:35:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:37 smithi122 ceph-mon[93869]: pgmap v13229: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 162 KiB/s wr, 8 op/s 2023-03-09T21:35:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:37 smithi181 ceph-mon[94189]: pgmap v13229: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 162 KiB/s wr, 8 op/s 2023-03-09T21:35:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:37 smithi038 ceph-mon[92422]: pgmap v13229: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 162 KiB/s wr, 8 op/s 2023-03-09T21:35:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23848.668667 secs 2023-03-09T21:35:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23848.668667 secs 2023-03-09T21:35:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23848.668667 secs 2023-03-09T21:35:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24310.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:35:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:39 smithi122 ceph-mon[93869]: pgmap v13230: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 96 KiB/s wr, 6 op/s 2023-03-09T21:35:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:39 smithi181 ceph-mon[94189]: pgmap v13230: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 96 KiB/s wr, 6 op/s 2023-03-09T21:35:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:39 smithi038 ceph-mon[92422]: pgmap v13230: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 96 KiB/s wr, 6 op/s 2023-03-09T21:35:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:41 smithi122 ceph-mon[93869]: pgmap v13231: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 102 KiB/s wr, 7 op/s 2023-03-09T21:35:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23851.324262 secs 2023-03-09T21:35:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:35:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:35:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:41 smithi181 ceph-mon[94189]: pgmap v13231: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 102 KiB/s wr, 7 op/s 2023-03-09T21:35:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23851.324262 secs 2023-03-09T21:35:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:35:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:35:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:41 smithi038 ceph-mon[92422]: pgmap v13231: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 102 KiB/s wr, 7 op/s 2023-03-09T21:35:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23851.324262 secs 2023-03-09T21:35:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:35:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:35:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:43 smithi122 ceph-mon[93869]: pgmap v13232: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 102 KiB/s wr, 8 op/s 2023-03-09T21:35:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23853.668786 secs 2023-03-09T21:35:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:43 smithi181 ceph-mon[94189]: pgmap v13232: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 102 KiB/s wr, 8 op/s 2023-03-09T21:35:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23853.668786 secs 2023-03-09T21:35:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:43 smithi038 ceph-mon[92422]: pgmap v13232: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 102 KiB/s wr, 8 op/s 2023-03-09T21:35:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23853.668786 secs 2023-03-09T21:35:43.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24315.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:35:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:45 smithi122 ceph-mon[93869]: pgmap v13233: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 201 KiB/s wr, 7 op/s 2023-03-09T21:35:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:45 smithi181 ceph-mon[94189]: pgmap v13233: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 201 KiB/s wr, 7 op/s 2023-03-09T21:35:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:45 smithi038 ceph-mon[92422]: pgmap v13233: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 201 KiB/s wr, 7 op/s 2023-03-09T21:35:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23856.324324 secs 2023-03-09T21:35:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23856.324324 secs 2023-03-09T21:35:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23856.324324 secs 2023-03-09T21:35:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:47 smithi122 ceph-mon[93869]: pgmap v13234: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 240 KiB/s wr, 10 op/s 2023-03-09T21:35:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:47 smithi181 ceph-mon[94189]: pgmap v13234: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 240 KiB/s wr, 10 op/s 2023-03-09T21:35:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:47 smithi038 ceph-mon[92422]: pgmap v13234: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 240 KiB/s wr, 10 op/s 2023-03-09T21:35:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23858.668920 secs 2023-03-09T21:35:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:35:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:35:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23858.668920 secs 2023-03-09T21:35:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:35:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:35:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23858.668920 secs 2023-03-09T21:35:48.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:35:48.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:35:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24320.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:35:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:49 smithi122 ceph-mon[93869]: pgmap v13235: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 450 KiB/s wr, 9 op/s 2023-03-09T21:35:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:49 smithi181 ceph-mon[94189]: pgmap v13235: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 450 KiB/s wr, 9 op/s 2023-03-09T21:35:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:49 smithi038 ceph-mon[92422]: pgmap v13235: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 450 KiB/s wr, 9 op/s 2023-03-09T21:35:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:51 smithi122 ceph-mon[93869]: pgmap v13236: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 450 KiB/s wr, 10 op/s 2023-03-09T21:35:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23861.324398 secs 2023-03-09T21:35:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:51 smithi181 ceph-mon[94189]: pgmap v13236: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 450 KiB/s wr, 10 op/s 2023-03-09T21:35:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23861.324398 secs 2023-03-09T21:35:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:51 smithi038 ceph-mon[92422]: pgmap v13236: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 450 KiB/s wr, 10 op/s 2023-03-09T21:35:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23861.324398 secs 2023-03-09T21:35:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:35:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:35:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:35:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:35:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:35:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:35:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:53 smithi122 ceph-mon[93869]: pgmap v13237: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 444 KiB/s wr, 12 op/s 2023-03-09T21:35:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23863.669045 secs 2023-03-09T21:35:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:35:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:35:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:53 smithi181 ceph-mon[94189]: pgmap v13237: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 444 KiB/s wr, 12 op/s 2023-03-09T21:35:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23863.669045 secs 2023-03-09T21:35:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:35:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:35:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:53 smithi038 ceph-mon[92422]: pgmap v13237: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 444 KiB/s wr, 12 op/s 2023-03-09T21:35:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23863.669045 secs 2023-03-09T21:35:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:35:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:35:53.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24325.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:35:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:55 smithi122 ceph-mon[93869]: pgmap v13238: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 499 KiB/s rd, 484 KiB/s wr, 11 op/s 2023-03-09T21:35:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:55 smithi181 ceph-mon[94189]: pgmap v13238: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 499 KiB/s rd, 484 KiB/s wr, 11 op/s 2023-03-09T21:35:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:55 smithi038 ceph-mon[92422]: pgmap v13238: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 499 KiB/s rd, 484 KiB/s wr, 11 op/s 2023-03-09T21:35:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23866.324476 secs 2023-03-09T21:35:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23866.324476 secs 2023-03-09T21:35:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23866.324476 secs 2023-03-09T21:35:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:57 smithi122 ceph-mon[93869]: pgmap v13239: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 405 KiB/s wr, 13 op/s 2023-03-09T21:35:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:57 smithi181 ceph-mon[94189]: pgmap v13239: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 405 KiB/s wr, 13 op/s 2023-03-09T21:35:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:57 smithi038 ceph-mon[92422]: pgmap v13239: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 405 KiB/s wr, 13 op/s 2023-03-09T21:35:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24330.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:35:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:35:59 smithi122 ceph-mon[93869]: pgmap v13240: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 719 KiB/s rd, 703 KiB/s wr, 11 op/s 2023-03-09T21:35:59.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:35:59 smithi181 ceph-mon[94189]: pgmap v13240: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 719 KiB/s rd, 703 KiB/s wr, 11 op/s 2023-03-09T21:35:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:35:59 smithi038 ceph-mon[92422]: pgmap v13240: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 719 KiB/s rd, 703 KiB/s wr, 11 op/s 2023-03-09T21:36:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:36:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:36:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:36:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:36:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:36:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:36:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23871.324538 secs 2023-03-09T21:36:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:01 smithi122 ceph-mon[93869]: pgmap v13241: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 399 KiB/s wr, 10 op/s 2023-03-09T21:36:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:36:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:36:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:01 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:36:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23871.324538 secs 2023-03-09T21:36:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:01 smithi181 ceph-mon[94189]: pgmap v13241: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 399 KiB/s wr, 10 op/s 2023-03-09T21:36:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:36:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:36:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:01 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:36:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23871.324538 secs 2023-03-09T21:36:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:01 smithi038 ceph-mon[92422]: pgmap v13241: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 399 KiB/s wr, 10 op/s 2023-03-09T21:36:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:36:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:36:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:01 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:36:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:02 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:36:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:02 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:36:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:02 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:36:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:02 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:36:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:02 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:36:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:02 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:36:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:02 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:36:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:02 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:36:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:02 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:36:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:03 smithi122 ceph-mon[93869]: pgmap v13242: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 438 KiB/s wr, 13 op/s 2023-03-09T21:36:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23873.669290 secs 2023-03-09T21:36:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:03 smithi181 ceph-mon[94189]: pgmap v13242: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 438 KiB/s wr, 13 op/s 2023-03-09T21:36:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23873.669290 secs 2023-03-09T21:36:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:03 smithi038 ceph-mon[92422]: pgmap v13242: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 438 KiB/s wr, 13 op/s 2023-03-09T21:36:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23873.669290 secs 2023-03-09T21:36:03.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24335.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:36:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:05 smithi122 ceph-mon[93869]: pgmap v13243: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 644 KiB/s wr, 25 op/s 2023-03-09T21:36:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:05 smithi181 ceph-mon[94189]: pgmap v13243: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 644 KiB/s wr, 25 op/s 2023-03-09T21:36:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:05 smithi038 ceph-mon[92422]: pgmap v13243: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 644 KiB/s wr, 25 op/s 2023-03-09T21:36:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23876.324622 secs 2023-03-09T21:36:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:06 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:36:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:36:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23876.324622 secs 2023-03-09T21:36:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:06 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:36:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:36:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23876.324622 secs 2023-03-09T21:36:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:06 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:36:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:36:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:07 smithi122 ceph-mon[93869]: pgmap v13244: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 588 KiB/s rd, 606 KiB/s wr, 27 op/s 2023-03-09T21:36:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:07 smithi181 ceph-mon[94189]: pgmap v13244: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 588 KiB/s rd, 606 KiB/s wr, 27 op/s 2023-03-09T21:36:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:07 smithi038 ceph-mon[92422]: pgmap v13244: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 588 KiB/s rd, 606 KiB/s wr, 27 op/s 2023-03-09T21:36:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23878.669368 secs 2023-03-09T21:36:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23878.669368 secs 2023-03-09T21:36:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23878.669368 secs 2023-03-09T21:36:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24340.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:36:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:09 smithi122 ceph-mon[93869]: pgmap v13245: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 786 KiB/s rd, 805 KiB/s wr, 26 op/s 2023-03-09T21:36:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:09 smithi181 ceph-mon[94189]: pgmap v13245: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 786 KiB/s rd, 805 KiB/s wr, 26 op/s 2023-03-09T21:36:09.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:09 smithi038 ceph-mon[92422]: pgmap v13245: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 786 KiB/s rd, 805 KiB/s wr, 26 op/s 2023-03-09T21:36:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:10 smithi122 ceph-mon[93869]: osdmap e1041: 12 total, 12 up, 12 in 2023-03-09T21:36:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:10 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:36:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:10 smithi181 ceph-mon[94189]: osdmap e1041: 12 total, 12 up, 12 in 2023-03-09T21:36:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:10 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:36:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:10 smithi038 ceph-mon[92422]: osdmap e1041: 12 total, 12 up, 12 in 2023-03-09T21:36:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:10 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:36:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23881.324683 secs 2023-03-09T21:36:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:11 smithi122 ceph-mon[93869]: pgmap v13247: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 539 KiB/s rd, 580 KiB/s wr, 29 op/s 2023-03-09T21:36:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:11 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:36:11.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:11 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:36:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23881.324683 secs 2023-03-09T21:36:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:11 smithi181 ceph-mon[94189]: pgmap v13247: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 539 KiB/s rd, 580 KiB/s wr, 29 op/s 2023-03-09T21:36:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:11 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:36:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:11 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:36:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23881.324683 secs 2023-03-09T21:36:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:11 smithi038 ceph-mon[92422]: pgmap v13247: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 539 KiB/s rd, 580 KiB/s wr, 29 op/s 2023-03-09T21:36:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:11 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:36:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:11 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:36:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:13 smithi122 ceph-mon[93869]: pgmap v13248: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 533 KiB/s wr, 26 op/s 2023-03-09T21:36:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23883.669516 secs 2023-03-09T21:36:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:36:13.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:36:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:13 smithi181 ceph-mon[94189]: pgmap v13248: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 533 KiB/s wr, 26 op/s 2023-03-09T21:36:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23883.669516 secs 2023-03-09T21:36:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:36:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:36:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:13 smithi038 ceph-mon[92422]: pgmap v13248: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 533 KiB/s wr, 26 op/s 2023-03-09T21:36:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23883.669516 secs 2023-03-09T21:36:13.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:36:13.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:36:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24345.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:36:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:15 smithi038 ceph-mon[92422]: pgmap v13249: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 380 KiB/s wr, 10 op/s 2023-03-09T21:36:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:15 smithi122 ceph-mon[93869]: pgmap v13249: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 380 KiB/s wr, 10 op/s 2023-03-09T21:36:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:15 smithi181 ceph-mon[94189]: pgmap v13249: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 380 KiB/s wr, 10 op/s 2023-03-09T21:36:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23886.324761 secs 2023-03-09T21:36:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23886.324761 secs 2023-03-09T21:36:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23886.324761 secs 2023-03-09T21:36:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:17 smithi038 ceph-mon[92422]: pgmap v13250: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 401 KiB/s wr, 10 op/s 2023-03-09T21:36:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:17 smithi122 ceph-mon[93869]: pgmap v13250: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 401 KiB/s wr, 10 op/s 2023-03-09T21:36:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:17 smithi181 ceph-mon[94189]: pgmap v13250: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 401 KiB/s wr, 10 op/s 2023-03-09T21:36:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24350.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:36:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23888.669651 secs 2023-03-09T21:36:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23888.669651 secs 2023-03-09T21:36:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23888.669651 secs 2023-03-09T21:36:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:19 smithi122 ceph-mon[93869]: pgmap v13251: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 218 KiB/s wr, 10 op/s 2023-03-09T21:36:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:19 smithi181 ceph-mon[94189]: pgmap v13251: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 218 KiB/s wr, 10 op/s 2023-03-09T21:36:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:19 smithi038 ceph-mon[92422]: pgmap v13251: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 218 KiB/s wr, 10 op/s 2023-03-09T21:36:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23891.324836 secs 2023-03-09T21:36:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:21 smithi122 ceph-mon[93869]: pgmap v13252: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 230 KiB/s wr, 10 op/s 2023-03-09T21:36:21.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:21 smithi122 ceph-mon[93869]: osdmap e1042: 12 total, 12 up, 12 in 2023-03-09T21:36:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23891.324836 secs 2023-03-09T21:36:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:21 smithi181 ceph-mon[94189]: pgmap v13252: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 230 KiB/s wr, 10 op/s 2023-03-09T21:36:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:21 smithi181 ceph-mon[94189]: osdmap e1042: 12 total, 12 up, 12 in 2023-03-09T21:36:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23891.324836 secs 2023-03-09T21:36:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:21 smithi038 ceph-mon[92422]: pgmap v13252: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 230 KiB/s wr, 10 op/s 2023-03-09T21:36:21.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:21 smithi038 ceph-mon[92422]: osdmap e1042: 12 total, 12 up, 12 in 2023-03-09T21:36:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:23 smithi122 ceph-mon[93869]: pgmap v13254: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 220 KiB/s wr, 9 op/s 2023-03-09T21:36:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23893.669777 secs 2023-03-09T21:36:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:23 smithi181 ceph-mon[94189]: pgmap v13254: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 220 KiB/s wr, 9 op/s 2023-03-09T21:36:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23893.669777 secs 2023-03-09T21:36:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:23 smithi038 ceph-mon[92422]: pgmap v13254: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 220 KiB/s wr, 9 op/s 2023-03-09T21:36:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23893.669777 secs 2023-03-09T21:36:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24355.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:36:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:25 smithi122 ceph-mon[93869]: pgmap v13255: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 172 KiB/s wr, 10 op/s 2023-03-09T21:36:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:25 smithi181 ceph-mon[94189]: pgmap v13255: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 172 KiB/s wr, 10 op/s 2023-03-09T21:36:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:25 smithi038 ceph-mon[92422]: pgmap v13255: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 172 KiB/s wr, 10 op/s 2023-03-09T21:36:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23896.324898 secs 2023-03-09T21:36:26.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23896.324898 secs 2023-03-09T21:36:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23896.324898 secs 2023-03-09T21:36:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:27 smithi122 ceph-mon[93869]: pgmap v13256: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 149 KiB/s wr, 9 op/s 2023-03-09T21:36:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:36:27.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:27 smithi181 ceph-mon[94189]: pgmap v13256: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 149 KiB/s wr, 9 op/s 2023-03-09T21:36:27.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:36:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:27 smithi038 ceph-mon[92422]: pgmap v13256: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 149 KiB/s wr, 9 op/s 2023-03-09T21:36:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:36:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23898.669876 secs 2023-03-09T21:36:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:36:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:36:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:36:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23898.669876 secs 2023-03-09T21:36:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:36:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:36:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:36:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23898.669876 secs 2023-03-09T21:36:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:36:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:36:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:36:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24360.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:36:29.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:29 smithi122 ceph-mon[93869]: pgmap v13257: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 202 KiB/s wr, 9 op/s 2023-03-09T21:36:29.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:29 smithi181 ceph-mon[94189]: pgmap v13257: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 202 KiB/s wr, 9 op/s 2023-03-09T21:36:29.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:29 smithi038 ceph-mon[92422]: pgmap v13257: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 202 KiB/s wr, 9 op/s 2023-03-09T21:36:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23901.324959 secs 2023-03-09T21:36:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:31 smithi122 ceph-mon[93869]: pgmap v13258: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 179 KiB/s wr, 9 op/s 2023-03-09T21:36:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23901.324959 secs 2023-03-09T21:36:31.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:31 smithi181 ceph-mon[94189]: pgmap v13258: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 179 KiB/s wr, 9 op/s 2023-03-09T21:36:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23901.324959 secs 2023-03-09T21:36:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:31 smithi038 ceph-mon[92422]: pgmap v13258: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 179 KiB/s wr, 9 op/s 2023-03-09T21:36:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:32 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:36:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:32 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:36:32.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:32 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:36:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:32 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:36:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:32 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:36:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:32 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:36:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:33 smithi122 ceph-mon[93869]: pgmap v13259: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 154 KiB/s wr, 10 op/s 2023-03-09T21:36:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23903.670021 secs 2023-03-09T21:36:33.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:33 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:36:33.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:33 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:36:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:33 smithi181 ceph-mon[94189]: pgmap v13259: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 154 KiB/s wr, 10 op/s 2023-03-09T21:36:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23903.670021 secs 2023-03-09T21:36:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:33 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:36:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:33 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:36:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:33 smithi038 ceph-mon[92422]: pgmap v13259: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 154 KiB/s wr, 10 op/s 2023-03-09T21:36:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23903.670021 secs 2023-03-09T21:36:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:33 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:36:33.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:33 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:36:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24365.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:36:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:35 smithi122 ceph-mon[93869]: pgmap v13260: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 149 KiB/s wr, 9 op/s 2023-03-09T21:36:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:35 smithi181 ceph-mon[94189]: pgmap v13260: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 149 KiB/s wr, 9 op/s 2023-03-09T21:36:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:35 smithi038 ceph-mon[92422]: pgmap v13260: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 149 KiB/s wr, 9 op/s 2023-03-09T21:36:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23906.325018 secs 2023-03-09T21:36:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23906.325018 secs 2023-03-09T21:36:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23906.325018 secs 2023-03-09T21:36:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:37 smithi122 ceph-mon[93869]: pgmap v13261: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 111 KiB/s wr, 10 op/s 2023-03-09T21:36:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:37 smithi181 ceph-mon[94189]: pgmap v13261: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 111 KiB/s wr, 10 op/s 2023-03-09T21:36:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:37 smithi038 ceph-mon[92422]: pgmap v13261: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 111 KiB/s wr, 10 op/s 2023-03-09T21:36:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23908.670140 secs 2023-03-09T21:36:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23908.670140 secs 2023-03-09T21:36:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23908.670140 secs 2023-03-09T21:36:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24370.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:36:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:39 smithi122 ceph-mon[93869]: pgmap v13262: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 296 KiB/s wr, 9 op/s 2023-03-09T21:36:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:39 smithi181 ceph-mon[94189]: pgmap v13262: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 296 KiB/s wr, 9 op/s 2023-03-09T21:36:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:39 smithi038 ceph-mon[92422]: pgmap v13262: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 296 KiB/s wr, 9 op/s 2023-03-09T21:36:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23911.325087 secs 2023-03-09T21:36:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:41 smithi122 ceph-mon[93869]: pgmap v13263: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 187 KiB/s wr, 8 op/s 2023-03-09T21:36:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:36:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:36:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23911.325087 secs 2023-03-09T21:36:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:41 smithi181 ceph-mon[94189]: pgmap v13263: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 187 KiB/s wr, 8 op/s 2023-03-09T21:36:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:36:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:36:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23911.325087 secs 2023-03-09T21:36:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:41 smithi038 ceph-mon[92422]: pgmap v13263: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 187 KiB/s wr, 8 op/s 2023-03-09T21:36:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:36:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:36:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:43 smithi122 ceph-mon[93869]: pgmap v13264: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 187 KiB/s wr, 9 op/s 2023-03-09T21:36:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:43 smithi181 ceph-mon[94189]: pgmap v13264: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 187 KiB/s wr, 9 op/s 2023-03-09T21:36:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:43 smithi038 ceph-mon[92422]: pgmap v13264: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 187 KiB/s wr, 9 op/s 2023-03-09T21:36:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24375.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:36:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:45 smithi122 ceph-mon[93869]: pgmap v13265: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 292 KiB/s wr, 9 op/s 2023-03-09T21:36:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:45 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:36:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:45 smithi181 ceph-mon[94189]: pgmap v13265: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 292 KiB/s wr, 9 op/s 2023-03-09T21:36:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:45 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:36:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:45 smithi038 ceph-mon[92422]: pgmap v13265: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 292 KiB/s wr, 9 op/s 2023-03-09T21:36:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:45 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:36:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23916.325147 secs 2023-03-09T21:36:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:46 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:36:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:46 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:36:46.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:46 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:36:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23916.325147 secs 2023-03-09T21:36:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:46 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:36:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:46 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:36:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:46 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:36:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23916.325147 secs 2023-03-09T21:36:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:46 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:36:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:46 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:36:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:46 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:36:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:47 smithi122 ceph-mon[93869]: pgmap v13266: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 311 KiB/s wr, 10 op/s 2023-03-09T21:36:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:47 smithi181 ceph-mon[94189]: pgmap v13266: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 311 KiB/s wr, 10 op/s 2023-03-09T21:36:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:47 smithi038 ceph-mon[92422]: pgmap v13266: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 311 KiB/s wr, 10 op/s 2023-03-09T21:36:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:36:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:36:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:36:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:36:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:36:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:36:48.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24380.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:36:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:49 smithi122 ceph-mon[93869]: pgmap v13267: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 589 KiB/s rd, 495 KiB/s wr, 9 op/s 2023-03-09T21:36:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:49 smithi181 ceph-mon[94189]: pgmap v13267: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 589 KiB/s rd, 495 KiB/s wr, 9 op/s 2023-03-09T21:36:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:49 smithi038 ceph-mon[92422]: pgmap v13267: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 589 KiB/s rd, 495 KiB/s wr, 9 op/s 2023-03-09T21:36:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23921.325209 secs 2023-03-09T21:36:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:51 smithi122 ceph-mon[93869]: pgmap v13268: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 328 KiB/s wr, 9 op/s 2023-03-09T21:36:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23921.325209 secs 2023-03-09T21:36:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:51 smithi181 ceph-mon[94189]: pgmap v13268: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 328 KiB/s wr, 9 op/s 2023-03-09T21:36:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23921.325209 secs 2023-03-09T21:36:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:51 smithi038 ceph-mon[92422]: pgmap v13268: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 328 KiB/s wr, 9 op/s 2023-03-09T21:36:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:53 smithi122 ceph-mon[93869]: pgmap v13269: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 401 KiB/s wr, 11 op/s 2023-03-09T21:36:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:53 smithi181 ceph-mon[94189]: pgmap v13269: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 401 KiB/s wr, 11 op/s 2023-03-09T21:36:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:53 smithi038 ceph-mon[92422]: pgmap v13269: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 401 KiB/s wr, 11 op/s 2023-03-09T21:36:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24385.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:36:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:55 smithi122 ceph-mon[93869]: pgmap v13270: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 555 KiB/s rd, 499 KiB/s wr, 11 op/s 2023-03-09T21:36:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:36:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:36:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:55 smithi181 ceph-mon[94189]: pgmap v13270: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 555 KiB/s rd, 499 KiB/s wr, 11 op/s 2023-03-09T21:36:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:36:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:36:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:55 smithi038 ceph-mon[92422]: pgmap v13270: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 555 KiB/s rd, 499 KiB/s wr, 11 op/s 2023-03-09T21:36:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:36:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:36:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23926.325270 secs 2023-03-09T21:36:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23926.325270 secs 2023-03-09T21:36:56.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23926.325270 secs 2023-03-09T21:36:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:57 smithi122 ceph-mon[93869]: pgmap v13271: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 683 KiB/s wr, 14 op/s 2023-03-09T21:36:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:36:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:36:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:57 smithi181 ceph-mon[94189]: pgmap v13271: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 683 KiB/s wr, 14 op/s 2023-03-09T21:36:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:36:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:36:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:57 smithi038 ceph-mon[92422]: pgmap v13271: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 683 KiB/s wr, 14 op/s 2023-03-09T21:36:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:36:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:36:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24390.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:36:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:36:59 smithi122 ceph-mon[93869]: pgmap v13272: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 678 KiB/s rd, 664 KiB/s wr, 12 op/s 2023-03-09T21:36:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:36:59 smithi181 ceph-mon[94189]: pgmap v13272: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 678 KiB/s rd, 664 KiB/s wr, 12 op/s 2023-03-09T21:36:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:36:59 smithi038 ceph-mon[92422]: pgmap v13272: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 678 KiB/s rd, 664 KiB/s wr, 12 op/s 2023-03-09T21:37:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23931.325347 secs 2023-03-09T21:37:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:01 smithi122 ceph-mon[93869]: pgmap v13273: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 513 KiB/s rd, 499 KiB/s wr, 11 op/s 2023-03-09T21:37:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23931.325347 secs 2023-03-09T21:37:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:01 smithi181 ceph-mon[94189]: pgmap v13273: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 513 KiB/s rd, 499 KiB/s wr, 11 op/s 2023-03-09T21:37:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23931.325347 secs 2023-03-09T21:37:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:01 smithi038 ceph-mon[92422]: pgmap v13273: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 513 KiB/s rd, 499 KiB/s wr, 11 op/s 2023-03-09T21:37:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:03 smithi122 ceph-mon[93869]: pgmap v13274: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 509 KiB/s wr, 14 op/s 2023-03-09T21:37:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:03 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 23933.670683 secs 2023-03-09T21:37:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:03 smithi181 ceph-mon[94189]: pgmap v13274: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 509 KiB/s wr, 14 op/s 2023-03-09T21:37:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:03 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 23933.670683 secs 2023-03-09T21:37:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:03 smithi038 ceph-mon[92422]: pgmap v13274: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 509 KiB/s wr, 14 op/s 2023-03-09T21:37:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:03 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 23933.670683 secs 2023-03-09T21:37:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24395.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:37:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:05 smithi122 ceph-mon[93869]: pgmap v13275: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 437 KiB/s wr, 13 op/s 2023-03-09T21:37:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:05 smithi181 ceph-mon[94189]: pgmap v13275: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 437 KiB/s wr, 13 op/s 2023-03-09T21:37:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:05 smithi038 ceph-mon[92422]: pgmap v13275: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 437 KiB/s wr, 13 op/s 2023-03-09T21:37:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23936.325407 secs 2023-03-09T21:37:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:06 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:37:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:06 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:37:06.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:37:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23936.325407 secs 2023-03-09T21:37:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:06 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:37:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:06 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:37:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:37:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23936.325407 secs 2023-03-09T21:37:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:06 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:37:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:06 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:37:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:37:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:07 smithi122 ceph-mon[93869]: pgmap v13276: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 639 KiB/s rd, 590 KiB/s wr, 17 op/s 2023-03-09T21:37:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:37:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:37:07.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:07 smithi181 ceph-mon[94189]: pgmap v13276: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 639 KiB/s rd, 590 KiB/s wr, 17 op/s 2023-03-09T21:37:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:37:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:37:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:07 smithi038 ceph-mon[92422]: pgmap v13276: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 639 KiB/s rd, 590 KiB/s wr, 17 op/s 2023-03-09T21:37:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:37:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:37:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23938.670817 secs 2023-03-09T21:37:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23938.670817 secs 2023-03-09T21:37:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23938.670817 secs 2023-03-09T21:37:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24400.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:37:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:09 smithi038 ceph-mon[92422]: pgmap v13277: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 301 KiB/s wr, 13 op/s 2023-03-09T21:37:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:37:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:37:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:09 smithi122 ceph-mon[93869]: pgmap v13277: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 301 KiB/s wr, 13 op/s 2023-03-09T21:37:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:37:09.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:37:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:09 smithi181 ceph-mon[94189]: pgmap v13277: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 301 KiB/s wr, 13 op/s 2023-03-09T21:37:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:37:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:37:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:10 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:37:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:10 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:37:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:10 smithi038 ceph-mon[92422]: osdmap e1043: 12 total, 12 up, 12 in 2023-03-09T21:37:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:10 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:37:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:10 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:37:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:10 smithi122 ceph-mon[93869]: osdmap e1043: 12 total, 12 up, 12 in 2023-03-09T21:37:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:10 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:37:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:10 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:37:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:10 smithi181 ceph-mon[94189]: osdmap e1043: 12 total, 12 up, 12 in 2023-03-09T21:37:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23941.325470 secs 2023-03-09T21:37:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:11 smithi038 ceph-mon[92422]: pgmap v13279: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 393 KiB/s wr, 27 op/s 2023-03-09T21:37:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23941.325470 secs 2023-03-09T21:37:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:11 smithi122 ceph-mon[93869]: pgmap v13279: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 393 KiB/s wr, 27 op/s 2023-03-09T21:37:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23941.325470 secs 2023-03-09T21:37:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:11 smithi181 ceph-mon[94189]: pgmap v13279: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 393 KiB/s wr, 27 op/s 2023-03-09T21:37:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24405.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:37:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:13 smithi122 ceph-mon[93869]: pgmap v13280: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 440 KiB/s rd, 481 KiB/s wr, 27 op/s 2023-03-09T21:37:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23943.670931 secs 2023-03-09T21:37:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:13 smithi181 ceph-mon[94189]: pgmap v13280: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 440 KiB/s rd, 481 KiB/s wr, 27 op/s 2023-03-09T21:37:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23943.670931 secs 2023-03-09T21:37:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:13 smithi038 ceph-mon[92422]: pgmap v13280: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 440 KiB/s rd, 481 KiB/s wr, 27 op/s 2023-03-09T21:37:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23943.670931 secs 2023-03-09T21:37:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:15 smithi122 ceph-mon[93869]: pgmap v13281: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 533 KiB/s rd, 575 KiB/s wr, 26 op/s 2023-03-09T21:37:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:15 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:37:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:15 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:37:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:15 smithi181 ceph-mon[94189]: pgmap v13281: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 533 KiB/s rd, 575 KiB/s wr, 26 op/s 2023-03-09T21:37:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:15 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:37:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:15 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:37:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:15 smithi038 ceph-mon[92422]: pgmap v13281: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 533 KiB/s rd, 575 KiB/s wr, 26 op/s 2023-03-09T21:37:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:15 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:37:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:15 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:37:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23946.325529 secs 2023-03-09T21:37:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23946.325529 secs 2023-03-09T21:37:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23946.325529 secs 2023-03-09T21:37:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:17 smithi122 ceph-mon[93869]: pgmap v13282: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 296 KiB/s wr, 22 op/s 2023-03-09T21:37:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:37:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:37:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:17 smithi181 ceph-mon[94189]: pgmap v13282: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 296 KiB/s wr, 22 op/s 2023-03-09T21:37:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:37:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:37:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:17 smithi038 ceph-mon[92422]: pgmap v13282: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 296 KiB/s wr, 22 op/s 2023-03-09T21:37:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:37:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:37:18.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24410.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:37:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23948.671051 secs 2023-03-09T21:37:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23948.671051 secs 2023-03-09T21:37:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23948.671051 secs 2023-03-09T21:37:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:19 smithi122 ceph-mon[93869]: pgmap v13283: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 296 KiB/s wr, 22 op/s 2023-03-09T21:37:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:19 smithi181 ceph-mon[94189]: pgmap v13283: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 296 KiB/s wr, 22 op/s 2023-03-09T21:37:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:19 smithi038 ceph-mon[92422]: pgmap v13283: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 296 KiB/s wr, 22 op/s 2023-03-09T21:37:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23951.325615 secs 2023-03-09T21:37:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:21 smithi122 ceph-mon[93869]: pgmap v13284: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 307 KiB/s wr, 9 op/s 2023-03-09T21:37:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:21 smithi122 ceph-mon[93869]: osdmap e1044: 12 total, 12 up, 12 in 2023-03-09T21:37:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23951.325615 secs 2023-03-09T21:37:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:21 smithi181 ceph-mon[94189]: pgmap v13284: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 307 KiB/s wr, 9 op/s 2023-03-09T21:37:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:21 smithi181 ceph-mon[94189]: osdmap e1044: 12 total, 12 up, 12 in 2023-03-09T21:37:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23951.325615 secs 2023-03-09T21:37:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:21 smithi038 ceph-mon[92422]: pgmap v13284: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 307 KiB/s wr, 9 op/s 2023-03-09T21:37:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:21 smithi038 ceph-mon[92422]: osdmap e1044: 12 total, 12 up, 12 in 2023-03-09T21:37:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:23 smithi122 ceph-mon[93869]: pgmap v13286: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 197 KiB/s wr, 8 op/s 2023-03-09T21:37:23.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23953.671171 secs 2023-03-09T21:37:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:23 smithi181 ceph-mon[94189]: pgmap v13286: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 197 KiB/s wr, 8 op/s 2023-03-09T21:37:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23953.671171 secs 2023-03-09T21:37:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:23 smithi038 ceph-mon[92422]: pgmap v13286: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 197 KiB/s wr, 8 op/s 2023-03-09T21:37:23.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23953.671171 secs 2023-03-09T21:37:23.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24415.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:37:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:25 smithi122 ceph-mon[93869]: pgmap v13287: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 102 KiB/s wr, 7 op/s 2023-03-09T21:37:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:25 smithi181 ceph-mon[94189]: pgmap v13287: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 102 KiB/s wr, 7 op/s 2023-03-09T21:37:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:25 smithi038 ceph-mon[92422]: pgmap v13287: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 102 KiB/s wr, 7 op/s 2023-03-09T21:37:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23956.325676 secs 2023-03-09T21:37:26.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23956.325676 secs 2023-03-09T21:37:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23956.325676 secs 2023-03-09T21:37:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:27 smithi122 ceph-mon[93869]: pgmap v13288: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 103 KiB/s wr, 7 op/s 2023-03-09T21:37:27.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:27 smithi181 ceph-mon[94189]: pgmap v13288: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 103 KiB/s wr, 7 op/s 2023-03-09T21:37:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:27 smithi038 ceph-mon[92422]: pgmap v13288: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 103 KiB/s wr, 7 op/s 2023-03-09T21:37:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23958.671293 secs 2023-03-09T21:37:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:37:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23958.671293 secs 2023-03-09T21:37:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:37:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23958.671293 secs 2023-03-09T21:37:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:37:28.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24420.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:37:29.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:29 smithi122 ceph-mon[93869]: pgmap v13289: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 103 KiB/s wr, 7 op/s 2023-03-09T21:37:29.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:29 smithi181 ceph-mon[94189]: pgmap v13289: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 103 KiB/s wr, 7 op/s 2023-03-09T21:37:29.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:29 smithi038 ceph-mon[92422]: pgmap v13289: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 103 KiB/s wr, 7 op/s 2023-03-09T21:37:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23961.325738 secs 2023-03-09T21:37:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:31 smithi122 ceph-mon[93869]: pgmap v13290: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 103 KiB/s wr, 7 op/s 2023-03-09T21:37:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:37:31.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:37:31.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:37:31.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:37:31.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:37:31.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:37:31.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23961.325738 secs 2023-03-09T21:37:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:31 smithi181 ceph-mon[94189]: pgmap v13290: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 103 KiB/s wr, 7 op/s 2023-03-09T21:37:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:37:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:37:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:37:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:37:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:37:31.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:37:31.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23961.325738 secs 2023-03-09T21:37:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:31 smithi038 ceph-mon[92422]: pgmap v13290: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 103 KiB/s wr, 7 op/s 2023-03-09T21:37:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:37:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:37:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:37:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:37:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:37:31.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:37:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:37:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:37:32.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:37:32.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:37:32.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:32 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T21:37:32.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:32 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T21:37:32.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:37:32.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:37:32.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:37:32.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:37:32.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:32 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi038 to 44479k 2023-03-09T21:37:32.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:32 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi038 to 45547372: error parsing value: Value '45547372' is below minimum 939524096 2023-03-09T21:37:32.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:37:32.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:37:32.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:37:32.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:37:32.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:32 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T21:37:32.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:32 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T21:37:32.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:37:32.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:37:32.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:37:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:37:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:37:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:37:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:37:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:32 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T21:37:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:32 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T21:37:32.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:37:32.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:37:32.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:37:32.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:37:32.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:32 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi038 to 44479k 2023-03-09T21:37:32.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:32 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi038 to 45547372: error parsing value: Value '45547372' is below minimum 939524096 2023-03-09T21:37:32.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:37:32.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:37:32.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:37:32.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:37:32.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:32 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T21:37:32.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:32 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T21:37:32.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:37:32.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:37:32.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:37:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:37:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:37:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:37:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:37:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:32 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T21:37:32.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:32 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T21:37:32.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:37:32.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:37:32.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:37:32.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:37:32.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:32 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi038 to 44479k 2023-03-09T21:37:32.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:32 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi038 to 45547372: error parsing value: Value '45547372' is below minimum 939524096 2023-03-09T21:37:32.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:37:32.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:37:32.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:37:32.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:37:32.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:32 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T21:37:32.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:32 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T21:37:32.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:37:32.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:37:32.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:37:33.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24425.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:37:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:33 smithi122 ceph-mon[93869]: pgmap v13291: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 89 KiB/s wr, 8 op/s 2023-03-09T21:37:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23963.671356 secs 2023-03-09T21:37:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:33 smithi181 ceph-mon[94189]: pgmap v13291: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 89 KiB/s wr, 8 op/s 2023-03-09T21:37:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23963.671356 secs 2023-03-09T21:37:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:33 smithi038 ceph-mon[92422]: pgmap v13291: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 89 KiB/s wr, 8 op/s 2023-03-09T21:37:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23963.671356 secs 2023-03-09T21:37:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:35 smithi122 ceph-mon[93869]: pgmap v13292: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 182 KiB/s wr, 7 op/s 2023-03-09T21:37:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:35 smithi181 ceph-mon[94189]: pgmap v13292: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 182 KiB/s wr, 7 op/s 2023-03-09T21:37:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:35 smithi038 ceph-mon[92422]: pgmap v13292: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 182 KiB/s wr, 7 op/s 2023-03-09T21:37:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23966.325801 secs 2023-03-09T21:37:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23966.325801 secs 2023-03-09T21:37:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23966.325801 secs 2023-03-09T21:37:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:37 smithi122 ceph-mon[93869]: pgmap v13293: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 182 KiB/s wr, 9 op/s 2023-03-09T21:37:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:37 smithi181 ceph-mon[94189]: pgmap v13293: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 182 KiB/s wr, 9 op/s 2023-03-09T21:37:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:37 smithi038 ceph-mon[92422]: pgmap v13293: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 182 KiB/s wr, 9 op/s 2023-03-09T21:37:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24430.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:37:38.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23968.671465 secs 2023-03-09T21:37:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23968.671465 secs 2023-03-09T21:37:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23968.671465 secs 2023-03-09T21:37:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:39 smithi038 ceph-mon[92422]: pgmap v13294: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 162 KiB/s wr, 6 op/s 2023-03-09T21:37:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:39 smithi122 ceph-mon[93869]: pgmap v13294: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 162 KiB/s wr, 6 op/s 2023-03-09T21:37:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:39 smithi181 ceph-mon[94189]: pgmap v13294: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 162 KiB/s wr, 6 op/s 2023-03-09T21:37:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:37:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:37:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23971.325860 secs 2023-03-09T21:37:41.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:37:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:37:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23971.325860 secs 2023-03-09T21:37:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:37:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:37:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23971.325860 secs 2023-03-09T21:37:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:41 smithi122 ceph-mon[93869]: pgmap v13295: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 173 KiB/s wr, 7 op/s 2023-03-09T21:37:42.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:41 smithi181 ceph-mon[94189]: pgmap v13295: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 173 KiB/s wr, 7 op/s 2023-03-09T21:37:42.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:41 smithi038 ceph-mon[92422]: pgmap v13295: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 173 KiB/s wr, 7 op/s 2023-03-09T21:37:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24435.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:37:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:43 smithi122 ceph-mon[93869]: pgmap v13296: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 202 KiB/s wr, 8 op/s 2023-03-09T21:37:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:43 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 23973.671550 secs 2023-03-09T21:37:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:43 smithi181 ceph-mon[94189]: pgmap v13296: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 202 KiB/s wr, 8 op/s 2023-03-09T21:37:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:43 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 23973.671550 secs 2023-03-09T21:37:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:43 smithi038 ceph-mon[92422]: pgmap v13296: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 202 KiB/s wr, 8 op/s 2023-03-09T21:37:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:43 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 23973.671550 secs 2023-03-09T21:37:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:45 smithi122 ceph-mon[93869]: pgmap v13297: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 202 KiB/s wr, 7 op/s 2023-03-09T21:37:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:45 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:37:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:45 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:37:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23976.325919 secs 2023-03-09T21:37:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:45 smithi181 ceph-mon[94189]: pgmap v13297: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 202 KiB/s wr, 7 op/s 2023-03-09T21:37:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:45 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:37:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:45 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:37:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23976.325919 secs 2023-03-09T21:37:46.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:45 smithi038 ceph-mon[92422]: pgmap v13297: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 202 KiB/s wr, 7 op/s 2023-03-09T21:37:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:45 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:37:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:45 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:37:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23976.325919 secs 2023-03-09T21:37:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:47 smithi122 ceph-mon[93869]: pgmap v13298: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 433 KiB/s wr, 10 op/s 2023-03-09T21:37:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:37:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:37:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23978.671677 secs 2023-03-09T21:37:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:47 smithi181 ceph-mon[94189]: pgmap v13298: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 433 KiB/s wr, 10 op/s 2023-03-09T21:37:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:37:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:37:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23978.671677 secs 2023-03-09T21:37:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:47 smithi038 ceph-mon[92422]: pgmap v13298: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 433 KiB/s wr, 10 op/s 2023-03-09T21:37:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:37:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:37:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23978.671677 secs 2023-03-09T21:37:48.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24440.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:37:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:48 smithi122 ceph-mon[93869]: pgmap v13299: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 433 KiB/s wr, 9 op/s 2023-03-09T21:37:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:48 smithi181 ceph-mon[94189]: pgmap v13299: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 433 KiB/s wr, 9 op/s 2023-03-09T21:37:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:48 smithi038 ceph-mon[92422]: pgmap v13299: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 433 KiB/s wr, 9 op/s 2023-03-09T21:37:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:37:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:37:50.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:37:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:37:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:37:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:37:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23981.325984 secs 2023-03-09T21:37:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:51 smithi122 ceph-mon[93869]: pgmap v13300: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 452 KiB/s wr, 10 op/s 2023-03-09T21:37:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23981.325984 secs 2023-03-09T21:37:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:51 smithi181 ceph-mon[94189]: pgmap v13300: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 452 KiB/s wr, 10 op/s 2023-03-09T21:37:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23981.325984 secs 2023-03-09T21:37:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:51 smithi038 ceph-mon[92422]: pgmap v13300: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 452 KiB/s wr, 10 op/s 2023-03-09T21:37:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:37:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:37:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:37:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:37:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:37:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:37:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:53 smithi122 ceph-mon[93869]: pgmap v13301: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 520 KiB/s rd, 482 KiB/s wr, 12 op/s 2023-03-09T21:37:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23983.671786 secs 2023-03-09T21:37:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:37:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:37:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:53 smithi181 ceph-mon[94189]: pgmap v13301: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 520 KiB/s rd, 482 KiB/s wr, 12 op/s 2023-03-09T21:37:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23983.671786 secs 2023-03-09T21:37:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:37:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:37:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:53 smithi038 ceph-mon[92422]: pgmap v13301: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 520 KiB/s rd, 482 KiB/s wr, 12 op/s 2023-03-09T21:37:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23983.671786 secs 2023-03-09T21:37:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:37:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:37:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24445.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:37:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:55 smithi122 ceph-mon[93869]: pgmap v13302: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 387 KiB/s wr, 10 op/s 2023-03-09T21:37:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:55 smithi181 ceph-mon[94189]: pgmap v13302: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 387 KiB/s wr, 10 op/s 2023-03-09T21:37:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:55 smithi038 ceph-mon[92422]: pgmap v13302: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 387 KiB/s wr, 10 op/s 2023-03-09T21:37:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23986.326061 secs 2023-03-09T21:37:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23986.326061 secs 2023-03-09T21:37:56.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23986.326061 secs 2023-03-09T21:37:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:57 smithi122 ceph-mon[93869]: pgmap v13303: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 777 KiB/s rd, 745 KiB/s wr, 14 op/s 2023-03-09T21:37:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:57 smithi181 ceph-mon[94189]: pgmap v13303: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 777 KiB/s rd, 745 KiB/s wr, 14 op/s 2023-03-09T21:37:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:57 smithi038 ceph-mon[92422]: pgmap v13303: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 777 KiB/s rd, 745 KiB/s wr, 14 op/s 2023-03-09T21:37:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23988.671928 secs 2023-03-09T21:37:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23988.671928 secs 2023-03-09T21:37:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23988.671928 secs 2023-03-09T21:37:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24450.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:37:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:37:59 smithi122 ceph-mon[93869]: pgmap v13304: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 418 KiB/s wr, 10 op/s 2023-03-09T21:37:59.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:37:59 smithi181 ceph-mon[94189]: pgmap v13304: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 418 KiB/s wr, 10 op/s 2023-03-09T21:37:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:37:59 smithi038 ceph-mon[92422]: pgmap v13304: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 418 KiB/s wr, 10 op/s 2023-03-09T21:38:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:38:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:38:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:38:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:38:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:38:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:38:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23991.326124 secs 2023-03-09T21:38:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:01 smithi122 ceph-mon[93869]: pgmap v13305: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 418 KiB/s wr, 10 op/s 2023-03-09T21:38:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:38:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:38:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23991.326124 secs 2023-03-09T21:38:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:01 smithi181 ceph-mon[94189]: pgmap v13305: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 418 KiB/s wr, 10 op/s 2023-03-09T21:38:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:38:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:38:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23991.326124 secs 2023-03-09T21:38:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:01 smithi038 ceph-mon[92422]: pgmap v13305: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 418 KiB/s wr, 10 op/s 2023-03-09T21:38:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:38:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:38:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:03 smithi122 ceph-mon[93869]: pgmap v13306: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 644 KiB/s rd, 648 KiB/s wr, 15 op/s 2023-03-09T21:38:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23993.672069 secs 2023-03-09T21:38:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:03 smithi181 ceph-mon[94189]: pgmap v13306: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 644 KiB/s rd, 648 KiB/s wr, 15 op/s 2023-03-09T21:38:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23993.672069 secs 2023-03-09T21:38:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:03 smithi038 ceph-mon[92422]: pgmap v13306: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 644 KiB/s rd, 648 KiB/s wr, 15 op/s 2023-03-09T21:38:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23993.672069 secs 2023-03-09T21:38:03.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24455.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:38:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:05 smithi122 ceph-mon[93869]: pgmap v13307: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 610 KiB/s rd, 629 KiB/s wr, 24 op/s 2023-03-09T21:38:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:05 smithi181 ceph-mon[94189]: pgmap v13307: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 610 KiB/s rd, 629 KiB/s wr, 24 op/s 2023-03-09T21:38:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:05 smithi038 ceph-mon[92422]: pgmap v13307: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 610 KiB/s rd, 629 KiB/s wr, 24 op/s 2023-03-09T21:38:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23996.326182 secs 2023-03-09T21:38:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23996.326182 secs 2023-03-09T21:38:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23996.326182 secs 2023-03-09T21:38:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:07 smithi122 ceph-mon[93869]: pgmap v13308: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 699 KiB/s rd, 716 KiB/s wr, 28 op/s 2023-03-09T21:38:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:07 smithi181 ceph-mon[94189]: pgmap v13308: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 699 KiB/s rd, 716 KiB/s wr, 28 op/s 2023-03-09T21:38:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:07 smithi038 ceph-mon[92422]: pgmap v13308: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 699 KiB/s rd, 716 KiB/s wr, 28 op/s 2023-03-09T21:38:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 23998.672177 secs 2023-03-09T21:38:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 23998.672177 secs 2023-03-09T21:38:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 23998.672177 secs 2023-03-09T21:38:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24460.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:38:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:09 smithi122 ceph-mon[93869]: pgmap v13309: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 358 KiB/s wr, 24 op/s 2023-03-09T21:38:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:09 smithi181 ceph-mon[94189]: pgmap v13309: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 358 KiB/s wr, 24 op/s 2023-03-09T21:38:09.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:09 smithi038 ceph-mon[92422]: pgmap v13309: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 358 KiB/s wr, 24 op/s 2023-03-09T21:38:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:10 smithi122 ceph-mon[93869]: osdmap e1045: 12 total, 12 up, 12 in 2023-03-09T21:38:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:10 smithi181 ceph-mon[94189]: osdmap e1045: 12 total, 12 up, 12 in 2023-03-09T21:38:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:10 smithi038 ceph-mon[92422]: osdmap e1045: 12 total, 12 up, 12 in 2023-03-09T21:38:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24001.326244 secs 2023-03-09T21:38:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:11 smithi122 ceph-mon[93869]: pgmap v13311: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 449 KiB/s wr, 28 op/s 2023-03-09T21:38:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24001.326244 secs 2023-03-09T21:38:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:11 smithi181 ceph-mon[94189]: pgmap v13311: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 449 KiB/s wr, 28 op/s 2023-03-09T21:38:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24001.326244 secs 2023-03-09T21:38:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:11 smithi038 ceph-mon[92422]: pgmap v13311: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 449 KiB/s wr, 28 op/s 2023-03-09T21:38:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:38:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:38:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:38:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:38:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:38:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:38:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:13 smithi122 ceph-mon[93869]: pgmap v13312: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 399 KiB/s wr, 24 op/s 2023-03-09T21:38:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24003.672308 secs 2023-03-09T21:38:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:38:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:38:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:13 smithi181 ceph-mon[94189]: pgmap v13312: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 399 KiB/s wr, 24 op/s 2023-03-09T21:38:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24003.672308 secs 2023-03-09T21:38:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:38:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:38:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:13 smithi038 ceph-mon[92422]: pgmap v13312: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 399 KiB/s wr, 24 op/s 2023-03-09T21:38:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24003.672308 secs 2023-03-09T21:38:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:38:13.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:38:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24465.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:38:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:15 smithi122 ceph-mon[93869]: pgmap v13313: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 373 KiB/s wr, 10 op/s 2023-03-09T21:38:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:15 smithi181 ceph-mon[94189]: pgmap v13313: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 373 KiB/s wr, 10 op/s 2023-03-09T21:38:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:15 smithi038 ceph-mon[92422]: pgmap v13313: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 373 KiB/s wr, 10 op/s 2023-03-09T21:38:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24006.326307 secs 2023-03-09T21:38:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24006.326307 secs 2023-03-09T21:38:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24006.326307 secs 2023-03-09T21:38:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:17 smithi122 ceph-mon[93869]: pgmap v13314: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 370 KiB/s wr, 9 op/s 2023-03-09T21:38:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:17 smithi181 ceph-mon[94189]: pgmap v13314: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 370 KiB/s wr, 9 op/s 2023-03-09T21:38:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:17 smithi038 ceph-mon[92422]: pgmap v13314: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 370 KiB/s wr, 9 op/s 2023-03-09T21:38:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24008.672410 secs 2023-03-09T21:38:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24008.672410 secs 2023-03-09T21:38:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24008.672410 secs 2023-03-09T21:38:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24470.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:38:19.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:19 smithi122 ceph-mon[93869]: pgmap v13315: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 370 KiB/s wr, 9 op/s 2023-03-09T21:38:19.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:19 smithi181 ceph-mon[94189]: pgmap v13315: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 370 KiB/s wr, 9 op/s 2023-03-09T21:38:19.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:19 smithi038 ceph-mon[92422]: pgmap v13315: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 370 KiB/s wr, 9 op/s 2023-03-09T21:38:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24011.326369 secs 2023-03-09T21:38:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:21 smithi122 ceph-mon[93869]: pgmap v13316: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 355 KiB/s rd, 353 KiB/s wr, 8 op/s 2023-03-09T21:38:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:21 smithi122 ceph-mon[93869]: osdmap e1046: 12 total, 12 up, 12 in 2023-03-09T21:38:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24011.326369 secs 2023-03-09T21:38:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:21 smithi181 ceph-mon[94189]: pgmap v13316: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 355 KiB/s rd, 353 KiB/s wr, 8 op/s 2023-03-09T21:38:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:21 smithi181 ceph-mon[94189]: osdmap e1046: 12 total, 12 up, 12 in 2023-03-09T21:38:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24011.326369 secs 2023-03-09T21:38:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:21 smithi038 ceph-mon[92422]: pgmap v13316: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 355 KiB/s rd, 353 KiB/s wr, 8 op/s 2023-03-09T21:38:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:21 smithi038 ceph-mon[92422]: osdmap e1046: 12 total, 12 up, 12 in 2023-03-09T21:38:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:23 smithi122 ceph-mon[93869]: pgmap v13318: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 127 KiB/s wr, 9 op/s 2023-03-09T21:38:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24013.672529 secs 2023-03-09T21:38:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:23 smithi181 ceph-mon[94189]: pgmap v13318: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 127 KiB/s wr, 9 op/s 2023-03-09T21:38:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24013.672529 secs 2023-03-09T21:38:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:23 smithi038 ceph-mon[92422]: pgmap v13318: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 127 KiB/s wr, 9 op/s 2023-03-09T21:38:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24013.672529 secs 2023-03-09T21:38:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24475.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:38:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:25 smithi122 ceph-mon[93869]: pgmap v13319: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 126 KiB/s wr, 9 op/s 2023-03-09T21:38:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:25 smithi181 ceph-mon[94189]: pgmap v13319: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 126 KiB/s wr, 9 op/s 2023-03-09T21:38:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:25 smithi038 ceph-mon[92422]: pgmap v13319: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 126 KiB/s wr, 9 op/s 2023-03-09T21:38:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24016.326431 secs 2023-03-09T21:38:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24016.326431 secs 2023-03-09T21:38:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24016.326431 secs 2023-03-09T21:38:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:27 smithi122 ceph-mon[93869]: pgmap v13320: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 51 KiB/s wr, 8 op/s 2023-03-09T21:38:27.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:27 smithi181 ceph-mon[94189]: pgmap v13320: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 51 KiB/s wr, 8 op/s 2023-03-09T21:38:27.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:27 smithi038 ceph-mon[92422]: pgmap v13320: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 51 KiB/s wr, 8 op/s 2023-03-09T21:38:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24018.672666 secs 2023-03-09T21:38:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24018.672666 secs 2023-03-09T21:38:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24018.672666 secs 2023-03-09T21:38:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24480.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:38:29.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:29 smithi122 ceph-mon[93869]: pgmap v13321: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 103 KiB/s wr, 9 op/s 2023-03-09T21:38:29.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:29 smithi181 ceph-mon[94189]: pgmap v13321: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 103 KiB/s wr, 9 op/s 2023-03-09T21:38:29.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:29 smithi038 ceph-mon[92422]: pgmap v13321: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 103 KiB/s wr, 9 op/s 2023-03-09T21:38:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24021.326494 secs 2023-03-09T21:38:31.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:31 smithi122 ceph-mon[93869]: pgmap v13322: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 81 KiB/s wr, 9 op/s 2023-03-09T21:38:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24021.326494 secs 2023-03-09T21:38:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:31 smithi181 ceph-mon[94189]: pgmap v13322: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 81 KiB/s wr, 9 op/s 2023-03-09T21:38:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24021.326494 secs 2023-03-09T21:38:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:31 smithi038 ceph-mon[92422]: pgmap v13322: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 81 KiB/s wr, 9 op/s 2023-03-09T21:38:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:32 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:38:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:32 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:38:32.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:38:32.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:32 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:38:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:32 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:38:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:38:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:32 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:38:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:32 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:38:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:38:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:33 smithi122 ceph-mon[93869]: pgmap v13323: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 169 KiB/s wr, 11 op/s 2023-03-09T21:38:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24023.672770 secs 2023-03-09T21:38:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:33 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:38:33.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:33 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:38:33.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:38:33.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:38:33.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:38:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:33 smithi181 ceph-mon[94189]: pgmap v13323: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 169 KiB/s wr, 11 op/s 2023-03-09T21:38:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24023.672770 secs 2023-03-09T21:38:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:33 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:38:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:33 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:38:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:38:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:38:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:38:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:33 smithi038 ceph-mon[92422]: pgmap v13323: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 169 KiB/s wr, 11 op/s 2023-03-09T21:38:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24023.672770 secs 2023-03-09T21:38:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:33 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:38:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:33 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:38:33.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:38:33.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:38:33.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:38:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24485.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:38:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:35 smithi122 ceph-mon[93869]: pgmap v13324: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 164 KiB/s wr, 8 op/s 2023-03-09T21:38:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:35 smithi181 ceph-mon[94189]: pgmap v13324: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 164 KiB/s wr, 8 op/s 2023-03-09T21:38:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:35 smithi038 ceph-mon[92422]: pgmap v13324: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 164 KiB/s wr, 8 op/s 2023-03-09T21:38:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24026.326555 secs 2023-03-09T21:38:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24026.326555 secs 2023-03-09T21:38:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24026.326555 secs 2023-03-09T21:38:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:37 smithi122 ceph-mon[93869]: pgmap v13325: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 164 KiB/s wr, 10 op/s 2023-03-09T21:38:37.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:37 smithi181 ceph-mon[94189]: pgmap v13325: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 164 KiB/s wr, 10 op/s 2023-03-09T21:38:37.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:37 smithi038 ceph-mon[92422]: pgmap v13325: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 164 KiB/s wr, 10 op/s 2023-03-09T21:38:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24028.672892 secs 2023-03-09T21:38:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24028.672892 secs 2023-03-09T21:38:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24028.672892 secs 2023-03-09T21:38:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24490.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:38:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:39 smithi122 ceph-mon[93869]: pgmap v13326: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 232 KiB/s wr, 8 op/s 2023-03-09T21:38:39.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:39 smithi181 ceph-mon[94189]: pgmap v13326: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 232 KiB/s wr, 8 op/s 2023-03-09T21:38:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:39 smithi038 ceph-mon[92422]: pgmap v13326: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 232 KiB/s wr, 8 op/s 2023-03-09T21:38:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24031.326617 secs 2023-03-09T21:38:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:41 smithi122 ceph-mon[93869]: pgmap v13327: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 189 KiB/s wr, 7 op/s 2023-03-09T21:38:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24031.326617 secs 2023-03-09T21:38:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:41 smithi181 ceph-mon[94189]: pgmap v13327: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 189 KiB/s wr, 7 op/s 2023-03-09T21:38:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24031.326617 secs 2023-03-09T21:38:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:41 smithi038 ceph-mon[92422]: pgmap v13327: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 189 KiB/s wr, 7 op/s 2023-03-09T21:38:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:43 smithi122 ceph-mon[93869]: pgmap v13328: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 299 KiB/s wr, 10 op/s 2023-03-09T21:38:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24033.673013 secs 2023-03-09T21:38:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:43 smithi181 ceph-mon[94189]: pgmap v13328: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 299 KiB/s wr, 10 op/s 2023-03-09T21:38:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24033.673013 secs 2023-03-09T21:38:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:43 smithi038 ceph-mon[92422]: pgmap v13328: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 299 KiB/s wr, 10 op/s 2023-03-09T21:38:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24033.673013 secs 2023-03-09T21:38:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24495.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:38:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:45 smithi122 ceph-mon[93869]: pgmap v13329: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 400 KiB/s wr, 7 op/s 2023-03-09T21:38:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:45 smithi181 ceph-mon[94189]: pgmap v13329: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 400 KiB/s wr, 7 op/s 2023-03-09T21:38:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:45 smithi038 ceph-mon[92422]: pgmap v13329: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 400 KiB/s wr, 7 op/s 2023-03-09T21:38:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24036.326680 secs 2023-03-09T21:38:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24036.326680 secs 2023-03-09T21:38:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24036.326680 secs 2023-03-09T21:38:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:47 smithi122 ceph-mon[93869]: pgmap v13330: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 438 KiB/s wr, 11 op/s 2023-03-09T21:38:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:38:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:38:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:47 smithi181 ceph-mon[94189]: pgmap v13330: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 438 KiB/s wr, 11 op/s 2023-03-09T21:38:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:38:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:38:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:47 smithi038 ceph-mon[92422]: pgmap v13330: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 438 KiB/s wr, 11 op/s 2023-03-09T21:38:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:38:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:38:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24500.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:38:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:49 smithi122 ceph-mon[93869]: pgmap v13331: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 582 KiB/s wr, 11 op/s 2023-03-09T21:38:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:38:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:38:49.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:49 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:38:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:49 smithi181 ceph-mon[94189]: pgmap v13331: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 582 KiB/s wr, 11 op/s 2023-03-09T21:38:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:38:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:38:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:49 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:38:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:49 smithi038 ceph-mon[92422]: pgmap v13331: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 582 KiB/s wr, 11 op/s 2023-03-09T21:38:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:38:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:38:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:49 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:38:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:50 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:38:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:50 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:38:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:50 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:38:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:50 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:38:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:50 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:38:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:50 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:38:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:50 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:38:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:50 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:38:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:50 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:38:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24041.326764 secs 2023-03-09T21:38:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:51 smithi122 ceph-mon[93869]: pgmap v13332: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 555 KiB/s rd, 491 KiB/s wr, 11 op/s 2023-03-09T21:38:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24041.326764 secs 2023-03-09T21:38:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:51 smithi181 ceph-mon[94189]: pgmap v13332: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 555 KiB/s rd, 491 KiB/s wr, 11 op/s 2023-03-09T21:38:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24041.326764 secs 2023-03-09T21:38:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:51 smithi038 ceph-mon[92422]: pgmap v13332: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 555 KiB/s rd, 491 KiB/s wr, 11 op/s 2023-03-09T21:38:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:53 smithi122 ceph-mon[93869]: pgmap v13333: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 511 KiB/s wr, 14 op/s 2023-03-09T21:38:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:53 smithi181 ceph-mon[94189]: pgmap v13333: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 511 KiB/s wr, 14 op/s 2023-03-09T21:38:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:53 smithi038 ceph-mon[92422]: pgmap v13333: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 511 KiB/s wr, 14 op/s 2023-03-09T21:38:53.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24505.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:38:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:55 smithi122 ceph-mon[93869]: pgmap v13334: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 623 KiB/s rd, 688 KiB/s wr, 13 op/s 2023-03-09T21:38:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:38:55.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:38:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:55 smithi181 ceph-mon[94189]: pgmap v13334: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 623 KiB/s rd, 688 KiB/s wr, 13 op/s 2023-03-09T21:38:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:38:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:38:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:55 smithi038 ceph-mon[92422]: pgmap v13334: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 623 KiB/s rd, 688 KiB/s wr, 13 op/s 2023-03-09T21:38:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:38:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:38:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24046.326849 secs 2023-03-09T21:38:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24046.326849 secs 2023-03-09T21:38:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24046.326849 secs 2023-03-09T21:38:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:57 smithi122 ceph-mon[93869]: pgmap v13335: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 576 KiB/s wr, 16 op/s 2023-03-09T21:38:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:38:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:38:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:57 smithi181 ceph-mon[94189]: pgmap v13335: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 576 KiB/s wr, 16 op/s 2023-03-09T21:38:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:38:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:38:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:57 smithi038 ceph-mon[92422]: pgmap v13335: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 576 KiB/s wr, 16 op/s 2023-03-09T21:38:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:38:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:38:58.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24510.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:38:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:59 smithi122 ceph-mon[93869]: pgmap v13336: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 862 KiB/s rd, 591 KiB/s wr, 14 op/s 2023-03-09T21:38:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:38:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:38:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:38:59.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:59 smithi181 ceph-mon[94189]: pgmap v13336: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 862 KiB/s rd, 591 KiB/s wr, 14 op/s 2023-03-09T21:38:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:38:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:38:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:38:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:59 smithi038 ceph-mon[92422]: pgmap v13336: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 862 KiB/s rd, 591 KiB/s wr, 14 op/s 2023-03-09T21:38:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:38:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:38:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:39:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24051.326911 secs 2023-03-09T21:39:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:01 smithi038 ceph-mon[92422]: pgmap v13337: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 481 KiB/s rd, 447 KiB/s wr, 12 op/s 2023-03-09T21:39:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:39:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:39:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24051.326911 secs 2023-03-09T21:39:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:01 smithi122 ceph-mon[93869]: pgmap v13337: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 481 KiB/s rd, 447 KiB/s wr, 12 op/s 2023-03-09T21:39:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:39:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:39:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24051.326911 secs 2023-03-09T21:39:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:01 smithi181 ceph-mon[94189]: pgmap v13337: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 481 KiB/s rd, 447 KiB/s wr, 12 op/s 2023-03-09T21:39:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:39:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:39:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:03 smithi038 ceph-mon[92422]: pgmap v13338: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 544 KiB/s wr, 18 op/s 2023-03-09T21:39:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24053.673512 secs 2023-03-09T21:39:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24515.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:39:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:03 smithi122 ceph-mon[93869]: pgmap v13338: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 544 KiB/s wr, 18 op/s 2023-03-09T21:39:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24053.673512 secs 2023-03-09T21:39:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:03 smithi181 ceph-mon[94189]: pgmap v13338: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 544 KiB/s wr, 18 op/s 2023-03-09T21:39:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24053.673512 secs 2023-03-09T21:39:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:05 smithi122 ceph-mon[93869]: pgmap v13339: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 531 KiB/s rd, 546 KiB/s wr, 27 op/s 2023-03-09T21:39:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:05 smithi122 ceph-mon[93869]: osdmap e1047: 12 total, 12 up, 12 in 2023-03-09T21:39:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:05 smithi181 ceph-mon[94189]: pgmap v13339: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 531 KiB/s rd, 546 KiB/s wr, 27 op/s 2023-03-09T21:39:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:05 smithi181 ceph-mon[94189]: osdmap e1047: 12 total, 12 up, 12 in 2023-03-09T21:39:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:05 smithi038 ceph-mon[92422]: pgmap v13339: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 531 KiB/s rd, 546 KiB/s wr, 27 op/s 2023-03-09T21:39:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:05 smithi038 ceph-mon[92422]: osdmap e1047: 12 total, 12 up, 12 in 2023-03-09T21:39:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24056.326993 secs 2023-03-09T21:39:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:06 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:39:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:06 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:39:06.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:39:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24056.326993 secs 2023-03-09T21:39:06.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:06 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:39:06.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:06 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:39:06.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:39:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24056.326993 secs 2023-03-09T21:39:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:06 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:39:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:06 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:39:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:39:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:07 smithi122 ceph-mon[93869]: pgmap v13341: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 472 KiB/s wr, 30 op/s 2023-03-09T21:39:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:07 smithi181 ceph-mon[94189]: pgmap v13341: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 472 KiB/s wr, 30 op/s 2023-03-09T21:39:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:07 smithi038 ceph-mon[92422]: pgmap v13341: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 472 KiB/s wr, 30 op/s 2023-03-09T21:39:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24520.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:39:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24058.673634 secs 2023-03-09T21:39:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24058.673634 secs 2023-03-09T21:39:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24058.673634 secs 2023-03-09T21:39:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:09 smithi122 ceph-mon[93869]: pgmap v13342: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 437 KiB/s wr, 28 op/s 2023-03-09T21:39:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:09 smithi181 ceph-mon[94189]: pgmap v13342: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 437 KiB/s wr, 28 op/s 2023-03-09T21:39:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:09 smithi038 ceph-mon[92422]: pgmap v13342: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 437 KiB/s wr, 28 op/s 2023-03-09T21:39:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24061.327052 secs 2023-03-09T21:39:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:11 smithi122 ceph-mon[93869]: pgmap v13343: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 437 KiB/s wr, 28 op/s 2023-03-09T21:39:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24061.327052 secs 2023-03-09T21:39:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:11 smithi181 ceph-mon[94189]: pgmap v13343: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 437 KiB/s wr, 28 op/s 2023-03-09T21:39:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24061.327052 secs 2023-03-09T21:39:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:11 smithi038 ceph-mon[92422]: pgmap v13343: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 437 KiB/s wr, 28 op/s 2023-03-09T21:39:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24525.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:39:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:13 smithi122 ceph-mon[93869]: pgmap v13344: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 401 KiB/s wr, 24 op/s 2023-03-09T21:39:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24063.673776 secs 2023-03-09T21:39:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:13 smithi181 ceph-mon[94189]: pgmap v13344: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 401 KiB/s wr, 24 op/s 2023-03-09T21:39:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24063.673776 secs 2023-03-09T21:39:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:13 smithi038 ceph-mon[92422]: pgmap v13344: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 401 KiB/s wr, 24 op/s 2023-03-09T21:39:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24063.673776 secs 2023-03-09T21:39:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:15 smithi122 ceph-mon[93869]: pgmap v13345: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 375 KiB/s wr, 9 op/s 2023-03-09T21:39:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:15 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:39:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:15 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:39:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:15 smithi122 ceph-mon[93869]: osdmap e1048: 12 total, 12 up, 12 in 2023-03-09T21:39:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:15 smithi181 ceph-mon[94189]: pgmap v13345: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 375 KiB/s wr, 9 op/s 2023-03-09T21:39:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:15 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:39:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:15 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:39:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:15 smithi181 ceph-mon[94189]: osdmap e1048: 12 total, 12 up, 12 in 2023-03-09T21:39:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:15 smithi038 ceph-mon[92422]: pgmap v13345: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 375 KiB/s wr, 9 op/s 2023-03-09T21:39:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:15 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:39:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:15 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:39:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:15 smithi038 ceph-mon[92422]: osdmap e1048: 12 total, 12 up, 12 in 2023-03-09T21:39:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24066.327110 secs 2023-03-09T21:39:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24066.327110 secs 2023-03-09T21:39:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24066.327110 secs 2023-03-09T21:39:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:17 smithi122 ceph-mon[93869]: pgmap v13347: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 132 KiB/s wr, 8 op/s 2023-03-09T21:39:17.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:39:17.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:39:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:17 smithi181 ceph-mon[94189]: pgmap v13347: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 132 KiB/s wr, 8 op/s 2023-03-09T21:39:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:39:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:39:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:17 smithi038 ceph-mon[92422]: pgmap v13347: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 132 KiB/s wr, 8 op/s 2023-03-09T21:39:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:39:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:39:18.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24530.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:39:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24068.673897 secs 2023-03-09T21:39:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24068.673897 secs 2023-03-09T21:39:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24068.673897 secs 2023-03-09T21:39:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:19 smithi122 ceph-mon[93869]: pgmap v13348: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 103 KiB/s wr, 8 op/s 2023-03-09T21:39:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:19 smithi181 ceph-mon[94189]: pgmap v13348: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 103 KiB/s wr, 8 op/s 2023-03-09T21:39:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:19 smithi038 ceph-mon[92422]: pgmap v13348: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 103 KiB/s wr, 8 op/s 2023-03-09T21:39:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24071.327171 secs 2023-03-09T21:39:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:21 smithi122 ceph-mon[93869]: pgmap v13349: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 103 KiB/s wr, 8 op/s 2023-03-09T21:39:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24071.327171 secs 2023-03-09T21:39:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:21 smithi181 ceph-mon[94189]: pgmap v13349: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 103 KiB/s wr, 8 op/s 2023-03-09T21:39:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24071.327171 secs 2023-03-09T21:39:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:21 smithi038 ceph-mon[92422]: pgmap v13349: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 103 KiB/s wr, 8 op/s 2023-03-09T21:39:23.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24535.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:39:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:23 smithi122 ceph-mon[93869]: pgmap v13350: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 47 KiB/s wr, 8 op/s 2023-03-09T21:39:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24073.674040 secs 2023-03-09T21:39:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:23 smithi181 ceph-mon[94189]: pgmap v13350: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 47 KiB/s wr, 8 op/s 2023-03-09T21:39:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24073.674040 secs 2023-03-09T21:39:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:23 smithi038 ceph-mon[92422]: pgmap v13350: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 47 KiB/s wr, 8 op/s 2023-03-09T21:39:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24073.674040 secs 2023-03-09T21:39:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:25 smithi122 ceph-mon[93869]: pgmap v13351: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 47 KiB/s wr, 8 op/s 2023-03-09T21:39:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:25 smithi181 ceph-mon[94189]: pgmap v13351: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 47 KiB/s wr, 8 op/s 2023-03-09T21:39:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:25 smithi038 ceph-mon[92422]: pgmap v13351: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 47 KiB/s wr, 8 op/s 2023-03-09T21:39:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24076.327247 secs 2023-03-09T21:39:26.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24076.327247 secs 2023-03-09T21:39:26.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24076.327247 secs 2023-03-09T21:39:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:27 smithi122 ceph-mon[93869]: pgmap v13352: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 44 KiB/s wr, 11 op/s 2023-03-09T21:39:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:27 smithi181 ceph-mon[94189]: pgmap v13352: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 44 KiB/s wr, 11 op/s 2023-03-09T21:39:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:27 smithi038 ceph-mon[92422]: pgmap v13352: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 44 KiB/s wr, 11 op/s 2023-03-09T21:39:28.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24540.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:39:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24078.674166 secs 2023-03-09T21:39:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24078.674166 secs 2023-03-09T21:39:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24078.674166 secs 2023-03-09T21:39:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:29 smithi122 ceph-mon[93869]: pgmap v13353: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T21:39:29.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:29 smithi181 ceph-mon[94189]: pgmap v13353: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T21:39:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:29 smithi038 ceph-mon[92422]: pgmap v13353: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T21:39:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24081.327311 secs 2023-03-09T21:39:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:31 smithi122 ceph-mon[93869]: pgmap v13354: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T21:39:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24081.327311 secs 2023-03-09T21:39:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:31 smithi181 ceph-mon[94189]: pgmap v13354: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T21:39:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24081.327311 secs 2023-03-09T21:39:32.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:31 smithi038 ceph-mon[92422]: pgmap v13354: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T21:39:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24545.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:39:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:33 smithi122 ceph-mon[93869]: pgmap v13355: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 117 KiB/s wr, 10 op/s 2023-03-09T21:39:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24083.674249 secs 2023-03-09T21:39:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:39:33.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:39:33.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:39:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:33 smithi181 ceph-mon[94189]: pgmap v13355: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 117 KiB/s wr, 10 op/s 2023-03-09T21:39:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24083.674249 secs 2023-03-09T21:39:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:39:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:39:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:39:33.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:33 smithi038 ceph-mon[92422]: pgmap v13355: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 117 KiB/s wr, 10 op/s 2023-03-09T21:39:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24083.674249 secs 2023-03-09T21:39:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:39:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:39:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:39:34.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:34 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:39:34.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:34 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:39:34.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:34 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:39:34.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:34 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:39:34.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:34 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:39:34.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:34 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:39:34.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:34 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:39:34.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:34 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:39:34.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:34 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:39:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:35 smithi122 ceph-mon[93869]: pgmap v13356: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 97 KiB/s wr, 7 op/s 2023-03-09T21:39:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:35 smithi181 ceph-mon[94189]: pgmap v13356: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 97 KiB/s wr, 7 op/s 2023-03-09T21:39:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:35 smithi038 ceph-mon[92422]: pgmap v13356: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 97 KiB/s wr, 7 op/s 2023-03-09T21:39:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24086.327369 secs 2023-03-09T21:39:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24086.327369 secs 2023-03-09T21:39:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24086.327369 secs 2023-03-09T21:39:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:37 smithi122 ceph-mon[93869]: pgmap v13357: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 98 KiB/s wr, 9 op/s 2023-03-09T21:39:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:37 smithi181 ceph-mon[94189]: pgmap v13357: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 98 KiB/s wr, 9 op/s 2023-03-09T21:39:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:37 smithi038 ceph-mon[92422]: pgmap v13357: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 98 KiB/s wr, 9 op/s 2023-03-09T21:39:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24550.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:39:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24088.674367 secs 2023-03-09T21:39:38.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24088.674367 secs 2023-03-09T21:39:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24088.674367 secs 2023-03-09T21:39:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:39 smithi122 ceph-mon[93869]: pgmap v13358: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 97 KiB/s wr, 7 op/s 2023-03-09T21:39:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:39 smithi181 ceph-mon[94189]: pgmap v13358: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 97 KiB/s wr, 7 op/s 2023-03-09T21:39:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:39 smithi038 ceph-mon[92422]: pgmap v13358: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 97 KiB/s wr, 7 op/s 2023-03-09T21:39:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24091.327430 secs 2023-03-09T21:39:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:41 smithi122 ceph-mon[93869]: pgmap v13359: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 97 KiB/s wr, 6 op/s 2023-03-09T21:39:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24091.327430 secs 2023-03-09T21:39:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:41 smithi181 ceph-mon[94189]: pgmap v13359: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 97 KiB/s wr, 6 op/s 2023-03-09T21:39:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24091.327430 secs 2023-03-09T21:39:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:41 smithi038 ceph-mon[92422]: pgmap v13359: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 97 KiB/s wr, 6 op/s 2023-03-09T21:39:43.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24555.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:39:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:43 smithi122 ceph-mon[93869]: pgmap v13360: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 252 KiB/s wr, 9 op/s 2023-03-09T21:39:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:43 smithi181 ceph-mon[94189]: pgmap v13360: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 252 KiB/s wr, 9 op/s 2023-03-09T21:39:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:43 smithi038 ceph-mon[92422]: pgmap v13360: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 252 KiB/s wr, 9 op/s 2023-03-09T21:39:44.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:44 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:39:44.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:44 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:39:44.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:44 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:39:44.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:44 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:39:44.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:44 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:39:44.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:44 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:39:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:45 smithi038 ceph-mon[92422]: pgmap v13361: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 155 KiB/s wr, 6 op/s 2023-03-09T21:39:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:45 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:39:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:45 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:39:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:45 smithi122 ceph-mon[93869]: pgmap v13361: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 155 KiB/s wr, 6 op/s 2023-03-09T21:39:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:45 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:39:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:45 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:39:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:45 smithi181 ceph-mon[94189]: pgmap v13361: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 155 KiB/s wr, 6 op/s 2023-03-09T21:39:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:45 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:39:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:45 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:39:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24096.327491 secs 2023-03-09T21:39:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24096.327491 secs 2023-03-09T21:39:47.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24096.327491 secs 2023-03-09T21:39:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:47 smithi038 ceph-mon[92422]: pgmap v13362: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 329 KiB/s wr, 9 op/s 2023-03-09T21:39:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:47 smithi122 ceph-mon[93869]: pgmap v13362: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 329 KiB/s wr, 9 op/s 2023-03-09T21:39:48.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:47 smithi181 ceph-mon[94189]: pgmap v13362: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 329 KiB/s wr, 9 op/s 2023-03-09T21:39:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24560.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:39:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:49 smithi038 ceph-mon[92422]: pgmap v13363: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 485 KiB/s rd, 329 KiB/s wr, 8 op/s 2023-03-09T21:39:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:49 smithi122 ceph-mon[93869]: pgmap v13363: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 485 KiB/s rd, 329 KiB/s wr, 8 op/s 2023-03-09T21:39:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:49 smithi181 ceph-mon[94189]: pgmap v13363: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 485 KiB/s rd, 329 KiB/s wr, 8 op/s 2023-03-09T21:39:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24101.327552 secs 2023-03-09T21:39:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:51 smithi122 ceph-mon[93869]: pgmap v13364: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 485 KiB/s rd, 329 KiB/s wr, 7 op/s 2023-03-09T21:39:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24101.327552 secs 2023-03-09T21:39:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:51 smithi181 ceph-mon[94189]: pgmap v13364: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 485 KiB/s rd, 329 KiB/s wr, 7 op/s 2023-03-09T21:39:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24101.327552 secs 2023-03-09T21:39:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:51 smithi038 ceph-mon[92422]: pgmap v13364: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 485 KiB/s rd, 329 KiB/s wr, 7 op/s 2023-03-09T21:39:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:39:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:39:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:39:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:39:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:39:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:39:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:53 smithi122 ceph-mon[93869]: pgmap v13365: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 389 KiB/s wr, 12 op/s 2023-03-09T21:39:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:39:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:39:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:53 smithi181 ceph-mon[94189]: pgmap v13365: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 389 KiB/s wr, 12 op/s 2023-03-09T21:39:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:39:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:39:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:53 smithi038 ceph-mon[92422]: pgmap v13365: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 389 KiB/s wr, 12 op/s 2023-03-09T21:39:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:39:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:39:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24565.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:39:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:55 smithi122 ceph-mon[93869]: pgmap v13366: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 235 KiB/s wr, 9 op/s 2023-03-09T21:39:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:55 smithi181 ceph-mon[94189]: pgmap v13366: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 235 KiB/s wr, 9 op/s 2023-03-09T21:39:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:55 smithi038 ceph-mon[92422]: pgmap v13366: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 235 KiB/s wr, 9 op/s 2023-03-09T21:39:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24106.327611 secs 2023-03-09T21:39:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24106.327611 secs 2023-03-09T21:39:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24106.327611 secs 2023-03-09T21:39:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:57 smithi122 ceph-mon[93869]: pgmap v13367: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 608 KiB/s wr, 13 op/s 2023-03-09T21:39:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:39:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:39:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:57 smithi181 ceph-mon[94189]: pgmap v13367: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 608 KiB/s wr, 13 op/s 2023-03-09T21:39:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:39:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:39:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:57 smithi038 ceph-mon[92422]: pgmap v13367: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 608 KiB/s wr, 13 op/s 2023-03-09T21:39:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:39:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:39:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24570.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:39:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:39:59 smithi122 ceph-mon[93869]: pgmap v13368: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 541 KiB/s rd, 434 KiB/s wr, 11 op/s 2023-03-09T21:39:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:39:59 smithi181 ceph-mon[94189]: pgmap v13368: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 541 KiB/s rd, 434 KiB/s wr, 11 op/s 2023-03-09T21:39:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:39:59 smithi038 ceph-mon[92422]: pgmap v13368: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 541 KiB/s rd, 434 KiB/s wr, 11 op/s 2023-03-09T21:40:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:40:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:40:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:00 smithi122 ceph-mon[93869]: Health detail: HEALTH_WARN 1 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-09T21:40:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:00 smithi122 ceph-mon[93869]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2023-03-09T21:40:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:00 smithi122 ceph-mon[93869]: mds.h(mds.1): 2 slow requests are blocked > 180 secs 2023-03-09T21:40:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:00 smithi122 ceph-mon[93869]: [WRN] MDS_TRIM: 2 MDSs behind on trimming 2023-03-09T21:40:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:00 smithi122 ceph-mon[93869]: mds.e(mds.0): Behind on trimming (647/128) max_segments: 128, num_segments: 647 2023-03-09T21:40:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:00 smithi122 ceph-mon[93869]: mds.b(mds.0): Behind on trimming (647/128) max_segments: 128, num_segments: 647 2023-03-09T21:40:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:40:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:40:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:00 smithi181 ceph-mon[94189]: Health detail: HEALTH_WARN 1 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-09T21:40:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:00 smithi181 ceph-mon[94189]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2023-03-09T21:40:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:00 smithi181 ceph-mon[94189]: mds.h(mds.1): 2 slow requests are blocked > 180 secs 2023-03-09T21:40:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:00 smithi181 ceph-mon[94189]: [WRN] MDS_TRIM: 2 MDSs behind on trimming 2023-03-09T21:40:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:00 smithi181 ceph-mon[94189]: mds.e(mds.0): Behind on trimming (647/128) max_segments: 128, num_segments: 647 2023-03-09T21:40:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:00 smithi181 ceph-mon[94189]: mds.b(mds.0): Behind on trimming (647/128) max_segments: 128, num_segments: 647 2023-03-09T21:40:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:40:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:40:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:00 smithi038 ceph-mon[92422]: Health detail: HEALTH_WARN 1 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-09T21:40:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:00 smithi038 ceph-mon[92422]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2023-03-09T21:40:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:00 smithi038 ceph-mon[92422]: mds.h(mds.1): 2 slow requests are blocked > 180 secs 2023-03-09T21:40:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:00 smithi038 ceph-mon[92422]: [WRN] MDS_TRIM: 2 MDSs behind on trimming 2023-03-09T21:40:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:00 smithi038 ceph-mon[92422]: mds.e(mds.0): Behind on trimming (647/128) max_segments: 128, num_segments: 647 2023-03-09T21:40:00.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:00 smithi038 ceph-mon[92422]: mds.b(mds.0): Behind on trimming (647/128) max_segments: 128, num_segments: 647 2023-03-09T21:40:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24111.327674 secs 2023-03-09T21:40:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:01 smithi122 ceph-mon[93869]: pgmap v13369: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 434 KiB/s wr, 10 op/s 2023-03-09T21:40:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24111.327674 secs 2023-03-09T21:40:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:01 smithi181 ceph-mon[94189]: pgmap v13369: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 434 KiB/s wr, 10 op/s 2023-03-09T21:40:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24111.327674 secs 2023-03-09T21:40:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:01 smithi038 ceph-mon[92422]: pgmap v13369: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 434 KiB/s wr, 10 op/s 2023-03-09T21:40:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:03 smithi122 ceph-mon[93869]: pgmap v13370: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 490 KiB/s rd, 475 KiB/s wr, 14 op/s 2023-03-09T21:40:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:03 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 24113.675006 secs 2023-03-09T21:40:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:03 smithi181 ceph-mon[94189]: pgmap v13370: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 490 KiB/s rd, 475 KiB/s wr, 14 op/s 2023-03-09T21:40:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:03 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 24113.675006 secs 2023-03-09T21:40:03.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:03 smithi038 ceph-mon[92422]: pgmap v13370: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 490 KiB/s rd, 475 KiB/s wr, 14 op/s 2023-03-09T21:40:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:03 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 24113.675006 secs 2023-03-09T21:40:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24575.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:40:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:05 smithi122 ceph-mon[93869]: pgmap v13371: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 700 KiB/s rd, 691 KiB/s wr, 21 op/s 2023-03-09T21:40:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:05 smithi181 ceph-mon[94189]: pgmap v13371: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 700 KiB/s rd, 691 KiB/s wr, 21 op/s 2023-03-09T21:40:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:05 smithi038 ceph-mon[92422]: pgmap v13371: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 700 KiB/s rd, 691 KiB/s wr, 21 op/s 2023-03-09T21:40:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24116.327733 secs 2023-03-09T21:40:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:06 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:40:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:06 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:40:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:40:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24116.327733 secs 2023-03-09T21:40:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:06 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:40:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:06 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:40:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:40:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24116.327733 secs 2023-03-09T21:40:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:06 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:40:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:06 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:40:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:40:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:07 smithi122 ceph-mon[93869]: pgmap v13372: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 814 KiB/s rd, 815 KiB/s wr, 25 op/s 2023-03-09T21:40:07.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:07 smithi181 ceph-mon[94189]: pgmap v13372: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 814 KiB/s rd, 815 KiB/s wr, 25 op/s 2023-03-09T21:40:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:07 smithi038 ceph-mon[92422]: pgmap v13372: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 814 KiB/s rd, 815 KiB/s wr, 25 op/s 2023-03-09T21:40:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:08 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 24118.675153 secs 2023-03-09T21:40:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:40:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:40:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:08 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 24118.675153 secs 2023-03-09T21:40:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:40:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:40:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:08 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 24118.675153 secs 2023-03-09T21:40:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:40:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:40:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24580.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:40:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:09 smithi122 ceph-mon[93869]: pgmap v13373: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 561 KiB/s wr, 23 op/s 2023-03-09T21:40:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:40:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:40:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:09 smithi181 ceph-mon[94189]: pgmap v13373: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 561 KiB/s wr, 23 op/s 2023-03-09T21:40:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:40:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:40:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:09 smithi038 ceph-mon[92422]: pgmap v13373: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 561 KiB/s wr, 23 op/s 2023-03-09T21:40:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:40:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:40:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24121.327795 secs 2023-03-09T21:40:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:11 smithi122 ceph-mon[93869]: pgmap v13374: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 560 KiB/s wr, 22 op/s 2023-03-09T21:40:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:11 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:40:11.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:11 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:40:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24121.327795 secs 2023-03-09T21:40:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:11 smithi181 ceph-mon[94189]: pgmap v13374: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 560 KiB/s wr, 22 op/s 2023-03-09T21:40:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:11 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:40:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:11 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:40:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24121.327795 secs 2023-03-09T21:40:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:11 smithi038 ceph-mon[92422]: pgmap v13374: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 560 KiB/s wr, 22 op/s 2023-03-09T21:40:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:11 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:40:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:11 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:40:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:13 smithi122 ceph-mon[93869]: pgmap v13375: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 728 KiB/s wr, 25 op/s 2023-03-09T21:40:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:13 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 24123.675277 secs 2023-03-09T21:40:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:40:13.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:40:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:13 smithi181 ceph-mon[94189]: pgmap v13375: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 728 KiB/s wr, 25 op/s 2023-03-09T21:40:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:13 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 24123.675277 secs 2023-03-09T21:40:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:40:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:40:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:13 smithi038 ceph-mon[92422]: pgmap v13375: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 728 KiB/s wr, 25 op/s 2023-03-09T21:40:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:13 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 24123.675277 secs 2023-03-09T21:40:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:40:13.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:40:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24585.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:40:15.723 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:15 smithi122 ceph-mon[93869]: pgmap v13376: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 674 KiB/s rd, 687 KiB/s wr, 22 op/s 2023-03-09T21:40:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:15 smithi181 ceph-mon[94189]: pgmap v13376: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 674 KiB/s rd, 687 KiB/s wr, 22 op/s 2023-03-09T21:40:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:15 smithi038 ceph-mon[92422]: pgmap v13376: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 674 KiB/s rd, 687 KiB/s wr, 22 op/s 2023-03-09T21:40:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24126.327872 secs 2023-03-09T21:40:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24126.327872 secs 2023-03-09T21:40:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24126.327872 secs 2023-03-09T21:40:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:17 smithi122 ceph-mon[93869]: pgmap v13377: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 431 KiB/s wr, 13 op/s 2023-03-09T21:40:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:17 smithi181 ceph-mon[94189]: pgmap v13377: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 431 KiB/s wr, 13 op/s 2023-03-09T21:40:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:17 smithi038 ceph-mon[92422]: pgmap v13377: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 431 KiB/s wr, 13 op/s 2023-03-09T21:40:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:18 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 24128.675392 secs 2023-03-09T21:40:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:18 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 24128.675392 secs 2023-03-09T21:40:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:18 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 24128.675392 secs 2023-03-09T21:40:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24590.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:40:19.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:19 smithi122 ceph-mon[93869]: pgmap v13378: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 384 KiB/s wr, 11 op/s 2023-03-09T21:40:19.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:19 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:40:19.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:19 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:40:19.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:19 smithi181 ceph-mon[94189]: pgmap v13378: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 384 KiB/s wr, 11 op/s 2023-03-09T21:40:19.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:19 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:40:19.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:19 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:40:19.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:19 smithi038 ceph-mon[92422]: pgmap v13378: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 384 KiB/s wr, 11 op/s 2023-03-09T21:40:19.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:19 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:40:19.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:19 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:40:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:20 smithi122 ceph-mon[93869]: osdmap e1049: 12 total, 12 up, 12 in 2023-03-09T21:40:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:20 smithi181 ceph-mon[94189]: osdmap e1049: 12 total, 12 up, 12 in 2023-03-09T21:40:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:20 smithi038 ceph-mon[92422]: osdmap e1049: 12 total, 12 up, 12 in 2023-03-09T21:40:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24131.327933 secs 2023-03-09T21:40:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:21 smithi122 ceph-mon[93869]: pgmap v13380: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 318 KiB/s wr, 11 op/s 2023-03-09T21:40:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:40:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:40:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24131.327933 secs 2023-03-09T21:40:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:21 smithi181 ceph-mon[94189]: pgmap v13380: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 318 KiB/s wr, 11 op/s 2023-03-09T21:40:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:40:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:40:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24131.327933 secs 2023-03-09T21:40:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:21 smithi038 ceph-mon[92422]: pgmap v13380: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 318 KiB/s wr, 11 op/s 2023-03-09T21:40:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:40:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:40:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:23 smithi122 ceph-mon[93869]: pgmap v13381: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 439 KiB/s wr, 14 op/s 2023-03-09T21:40:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24133.675516 secs 2023-03-09T21:40:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:23 smithi181 ceph-mon[94189]: pgmap v13381: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 439 KiB/s wr, 14 op/s 2023-03-09T21:40:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24133.675516 secs 2023-03-09T21:40:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:23 smithi038 ceph-mon[92422]: pgmap v13381: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 439 KiB/s wr, 14 op/s 2023-03-09T21:40:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24133.675516 secs 2023-03-09T21:40:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24595.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:40:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:25 smithi122 ceph-mon[93869]: pgmap v13382: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 440 KiB/s wr, 14 op/s 2023-03-09T21:40:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:25 smithi181 ceph-mon[94189]: pgmap v13382: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 440 KiB/s wr, 14 op/s 2023-03-09T21:40:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:25 smithi038 ceph-mon[92422]: pgmap v13382: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 440 KiB/s wr, 14 op/s 2023-03-09T21:40:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24136.327982 secs 2023-03-09T21:40:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:26 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:40:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:26 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:40:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24136.327982 secs 2023-03-09T21:40:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:26 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:40:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:26 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:40:26.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24136.327982 secs 2023-03-09T21:40:26.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:26 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:40:26.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:26 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:40:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:27 smithi038 ceph-mon[92422]: pgmap v13383: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 546 KiB/s rd, 571 KiB/s wr, 14 op/s 2023-03-09T21:40:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:27 smithi122 ceph-mon[93869]: pgmap v13383: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 546 KiB/s rd, 571 KiB/s wr, 14 op/s 2023-03-09T21:40:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:27 smithi181 ceph-mon[94189]: pgmap v13383: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 546 KiB/s rd, 571 KiB/s wr, 14 op/s 2023-03-09T21:40:28.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24600.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:40:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24138.675637 secs 2023-03-09T21:40:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24138.675637 secs 2023-03-09T21:40:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24138.675637 secs 2023-03-09T21:40:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:29 smithi122 ceph-mon[93869]: pgmap v13384: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 466 KiB/s rd, 479 KiB/s wr, 12 op/s 2023-03-09T21:40:29.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:29 smithi181 ceph-mon[94189]: pgmap v13384: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 466 KiB/s rd, 479 KiB/s wr, 12 op/s 2023-03-09T21:40:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:29 smithi038 ceph-mon[92422]: pgmap v13384: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 466 KiB/s rd, 479 KiB/s wr, 12 op/s 2023-03-09T21:40:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24141.328043 secs 2023-03-09T21:40:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:31 smithi122 ceph-mon[93869]: pgmap v13385: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 515 KiB/s wr, 12 op/s 2023-03-09T21:40:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:31 smithi122 ceph-mon[93869]: osdmap e1050: 12 total, 12 up, 12 in 2023-03-09T21:40:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24141.328043 secs 2023-03-09T21:40:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:31 smithi181 ceph-mon[94189]: pgmap v13385: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 515 KiB/s wr, 12 op/s 2023-03-09T21:40:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:31 smithi181 ceph-mon[94189]: osdmap e1050: 12 total, 12 up, 12 in 2023-03-09T21:40:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24141.328043 secs 2023-03-09T21:40:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:31 smithi038 ceph-mon[92422]: pgmap v13385: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 515 KiB/s wr, 12 op/s 2023-03-09T21:40:31.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:31 smithi038 ceph-mon[92422]: osdmap e1050: 12 total, 12 up, 12 in 2023-03-09T21:40:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:32 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:40:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:32 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:40:32.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:32 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:40:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:32 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:40:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:32 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:40:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:32 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:40:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:33 smithi122 ceph-mon[93869]: pgmap v13387: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 295 KiB/s wr, 10 op/s 2023-03-09T21:40:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24143.675778 secs 2023-03-09T21:40:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:33 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:40:33.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:33 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:40:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:33 smithi181 ceph-mon[94189]: pgmap v13387: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 295 KiB/s wr, 10 op/s 2023-03-09T21:40:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24143.675778 secs 2023-03-09T21:40:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:33 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:40:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:33 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:40:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:33 smithi038 ceph-mon[92422]: pgmap v13387: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 295 KiB/s wr, 10 op/s 2023-03-09T21:40:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24143.675778 secs 2023-03-09T21:40:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:33 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:40:33.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:33 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:40:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24605.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:40:34.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:34 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:40:34.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:34 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:40:34.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:34 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:40:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:35 smithi122 ceph-mon[93869]: pgmap v13388: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 294 KiB/s wr, 10 op/s 2023-03-09T21:40:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:40:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:40:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:40:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:40:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:40:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:40:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:40:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:35 smithi181 ceph-mon[94189]: pgmap v13388: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 294 KiB/s wr, 10 op/s 2023-03-09T21:40:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:40:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:40:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:40:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:40:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:40:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:40:35.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:40:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:35 smithi038 ceph-mon[92422]: pgmap v13388: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 294 KiB/s wr, 10 op/s 2023-03-09T21:40:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:40:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:40:35.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:40:35.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:40:35.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:40:35.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:40:35.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:40:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24146.328103 secs 2023-03-09T21:40:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24146.328103 secs 2023-03-09T21:40:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24146.328103 secs 2023-03-09T21:40:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:37 smithi122 ceph-mon[93869]: pgmap v13389: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 139 KiB/s wr, 10 op/s 2023-03-09T21:40:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:37 smithi181 ceph-mon[94189]: pgmap v13389: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 139 KiB/s wr, 10 op/s 2023-03-09T21:40:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:37 smithi038 ceph-mon[92422]: pgmap v13389: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 139 KiB/s wr, 10 op/s 2023-03-09T21:40:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24148.675886 secs 2023-03-09T21:40:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24148.675886 secs 2023-03-09T21:40:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24148.675886 secs 2023-03-09T21:40:38.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24610.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:40:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:39 smithi181 ceph-mon[94189]: pgmap v13390: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 139 KiB/s wr, 9 op/s 2023-03-09T21:40:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:39 smithi038 ceph-mon[92422]: pgmap v13390: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 139 KiB/s wr, 9 op/s 2023-03-09T21:40:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:39 smithi122 ceph-mon[93869]: pgmap v13390: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 139 KiB/s wr, 9 op/s 2023-03-09T21:40:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24151.328182 secs 2023-03-09T21:40:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:41 smithi038 ceph-mon[92422]: pgmap v13391: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 188 KiB/s wr, 10 op/s 2023-03-09T21:40:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24151.328182 secs 2023-03-09T21:40:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:41 smithi122 ceph-mon[93869]: pgmap v13391: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 188 KiB/s wr, 10 op/s 2023-03-09T21:40:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24151.328182 secs 2023-03-09T21:40:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:41 smithi181 ceph-mon[94189]: pgmap v13391: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 188 KiB/s wr, 10 op/s 2023-03-09T21:40:43.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:43 smithi038 ceph-mon[92422]: pgmap v13392: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 241 KiB/s wr, 11 op/s 2023-03-09T21:40:43.896 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24153.676005 secs 2023-03-09T21:40:43.897 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24615.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:40:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:43 smithi122 ceph-mon[93869]: pgmap v13392: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 241 KiB/s wr, 11 op/s 2023-03-09T21:40:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24153.676005 secs 2023-03-09T21:40:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:43 smithi181 ceph-mon[94189]: pgmap v13392: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 241 KiB/s wr, 11 op/s 2023-03-09T21:40:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24153.676005 secs 2023-03-09T21:40:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:45 smithi038 ceph-mon[92422]: pgmap v13393: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 367 KiB/s wr, 8 op/s 2023-03-09T21:40:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:45 smithi122 ceph-mon[93869]: pgmap v13393: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 367 KiB/s wr, 8 op/s 2023-03-09T21:40:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:45 smithi181 ceph-mon[94189]: pgmap v13393: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 367 KiB/s wr, 8 op/s 2023-03-09T21:40:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24156.328242 secs 2023-03-09T21:40:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24156.328242 secs 2023-03-09T21:40:46.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24156.328242 secs 2023-03-09T21:40:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:47 smithi122 ceph-mon[93869]: pgmap v13394: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 494 KiB/s wr, 11 op/s 2023-03-09T21:40:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:40:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:40:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:47 smithi181 ceph-mon[94189]: pgmap v13394: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 494 KiB/s wr, 11 op/s 2023-03-09T21:40:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:40:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:40:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:47 smithi038 ceph-mon[92422]: pgmap v13394: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 494 KiB/s wr, 11 op/s 2023-03-09T21:40:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:40:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:40:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24620.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:40:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24158.676120 secs 2023-03-09T21:40:48.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24158.676120 secs 2023-03-09T21:40:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24158.676120 secs 2023-03-09T21:40:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:49 smithi122 ceph-mon[93869]: pgmap v13395: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 585 KiB/s rd, 493 KiB/s wr, 9 op/s 2023-03-09T21:40:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:40:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:40:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:49 smithi181 ceph-mon[94189]: pgmap v13395: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 585 KiB/s rd, 493 KiB/s wr, 9 op/s 2023-03-09T21:40:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:40:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:40:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:49 smithi038 ceph-mon[92422]: pgmap v13395: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 585 KiB/s rd, 493 KiB/s wr, 9 op/s 2023-03-09T21:40:50.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:40:50.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:40:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24161.328303 secs 2023-03-09T21:40:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:51 smithi122 ceph-mon[93869]: pgmap v13396: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 596 KiB/s rd, 493 KiB/s wr, 10 op/s 2023-03-09T21:40:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24161.328303 secs 2023-03-09T21:40:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:51 smithi181 ceph-mon[94189]: pgmap v13396: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 596 KiB/s rd, 493 KiB/s wr, 10 op/s 2023-03-09T21:40:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24161.328303 secs 2023-03-09T21:40:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:51 smithi038 ceph-mon[92422]: pgmap v13396: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 596 KiB/s rd, 493 KiB/s wr, 10 op/s 2023-03-09T21:40:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24625.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:40:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:53 smithi122 ceph-mon[93869]: pgmap v13397: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 802 KiB/s rd, 682 KiB/s wr, 13 op/s 2023-03-09T21:40:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24163.676248 secs 2023-03-09T21:40:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:53 smithi181 ceph-mon[94189]: pgmap v13397: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 802 KiB/s rd, 682 KiB/s wr, 13 op/s 2023-03-09T21:40:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24163.676248 secs 2023-03-09T21:40:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:53 smithi038 ceph-mon[92422]: pgmap v13397: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 802 KiB/s rd, 682 KiB/s wr, 13 op/s 2023-03-09T21:40:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24163.676248 secs 2023-03-09T21:40:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:55 smithi122 ceph-mon[93869]: pgmap v13398: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 743 KiB/s rd, 605 KiB/s wr, 11 op/s 2023-03-09T21:40:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:40:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:40:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:55 smithi181 ceph-mon[94189]: pgmap v13398: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 743 KiB/s rd, 605 KiB/s wr, 11 op/s 2023-03-09T21:40:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:40:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:40:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:55 smithi038 ceph-mon[92422]: pgmap v13398: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 743 KiB/s rd, 605 KiB/s wr, 11 op/s 2023-03-09T21:40:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:40:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:40:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24166.328362 secs 2023-03-09T21:40:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24166.328362 secs 2023-03-09T21:40:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24166.328362 secs 2023-03-09T21:40:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:57 smithi122 ceph-mon[93869]: pgmap v13399: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 581 KiB/s rd, 531 KiB/s wr, 13 op/s 2023-03-09T21:40:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:40:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:40:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:57 smithi181 ceph-mon[94189]: pgmap v13399: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 581 KiB/s rd, 531 KiB/s wr, 13 op/s 2023-03-09T21:40:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:40:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:40:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:57 smithi038 ceph-mon[92422]: pgmap v13399: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 581 KiB/s rd, 531 KiB/s wr, 13 op/s 2023-03-09T21:40:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:40:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:40:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24630.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:40:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24168.676370 secs 2023-03-09T21:40:58.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24168.676370 secs 2023-03-09T21:40:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24168.676370 secs 2023-03-09T21:40:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:59 smithi122 ceph-mon[93869]: pgmap v13400: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 432 KiB/s rd, 404 KiB/s wr, 9 op/s 2023-03-09T21:40:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:40:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:40:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:40:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:59 smithi181 ceph-mon[94189]: pgmap v13400: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 432 KiB/s rd, 404 KiB/s wr, 9 op/s 2023-03-09T21:40:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:40:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:40:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:40:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:59 smithi038 ceph-mon[92422]: pgmap v13400: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 432 KiB/s rd, 404 KiB/s wr, 9 op/s 2023-03-09T21:41:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:41:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:40:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:41:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24171.328425 secs 2023-03-09T21:41:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:01 smithi122 ceph-mon[93869]: pgmap v13401: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 432 KiB/s rd, 404 KiB/s wr, 8 op/s 2023-03-09T21:41:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:41:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:41:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24171.328425 secs 2023-03-09T21:41:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:01 smithi181 ceph-mon[94189]: pgmap v13401: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 432 KiB/s rd, 404 KiB/s wr, 8 op/s 2023-03-09T21:41:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:41:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:41:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24171.328425 secs 2023-03-09T21:41:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:01 smithi038 ceph-mon[92422]: pgmap v13401: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 432 KiB/s rd, 404 KiB/s wr, 8 op/s 2023-03-09T21:41:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:41:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:41:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24635.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:41:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:03 smithi122 ceph-mon[93869]: pgmap v13402: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 630 KiB/s wr, 12 op/s 2023-03-09T21:41:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24173.676487 secs 2023-03-09T21:41:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:03 smithi181 ceph-mon[94189]: pgmap v13402: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 630 KiB/s wr, 12 op/s 2023-03-09T21:41:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24173.676487 secs 2023-03-09T21:41:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:03 smithi038 ceph-mon[92422]: pgmap v13402: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 630 KiB/s wr, 12 op/s 2023-03-09T21:41:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24173.676487 secs 2023-03-09T21:41:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:05 smithi122 ceph-mon[93869]: pgmap v13403: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 377 KiB/s wr, 20 op/s 2023-03-09T21:41:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:05 smithi181 ceph-mon[94189]: pgmap v13403: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 377 KiB/s wr, 20 op/s 2023-03-09T21:41:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:05 smithi038 ceph-mon[92422]: pgmap v13403: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 377 KiB/s wr, 20 op/s 2023-03-09T21:41:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24176.328487 secs 2023-03-09T21:41:06.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24176.328487 secs 2023-03-09T21:41:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24176.328487 secs 2023-03-09T21:41:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:07 smithi122 ceph-mon[93869]: pgmap v13404: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 482 KiB/s wr, 26 op/s 2023-03-09T21:41:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:07 smithi181 ceph-mon[94189]: pgmap v13404: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 482 KiB/s wr, 26 op/s 2023-03-09T21:41:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:07 smithi038 ceph-mon[92422]: pgmap v13404: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 482 KiB/s wr, 26 op/s 2023-03-09T21:41:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24640.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:41:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24178.676603 secs 2023-03-09T21:41:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24178.676603 secs 2023-03-09T21:41:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24178.676603 secs 2023-03-09T21:41:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:09 smithi122 ceph-mon[93869]: pgmap v13405: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 361 KiB/s wr, 23 op/s 2023-03-09T21:41:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:09 smithi181 ceph-mon[94189]: pgmap v13405: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 361 KiB/s wr, 23 op/s 2023-03-09T21:41:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:09 smithi038 ceph-mon[92422]: pgmap v13405: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 361 KiB/s wr, 23 op/s 2023-03-09T21:41:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:10 smithi122 ceph-mon[93869]: osdmap e1051: 12 total, 12 up, 12 in 2023-03-09T21:41:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:10 smithi181 ceph-mon[94189]: osdmap e1051: 12 total, 12 up, 12 in 2023-03-09T21:41:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:10 smithi038 ceph-mon[92422]: osdmap e1051: 12 total, 12 up, 12 in 2023-03-09T21:41:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24181.328558 secs 2023-03-09T21:41:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:11 smithi122 ceph-mon[93869]: pgmap v13407: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 433 KiB/s wr, 28 op/s 2023-03-09T21:41:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24181.328558 secs 2023-03-09T21:41:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:11 smithi181 ceph-mon[94189]: pgmap v13407: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 433 KiB/s wr, 28 op/s 2023-03-09T21:41:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24181.328558 secs 2023-03-09T21:41:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:11 smithi038 ceph-mon[92422]: pgmap v13407: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 433 KiB/s wr, 28 op/s 2023-03-09T21:41:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24645.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:41:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:13 smithi122 ceph-mon[93869]: pgmap v13408: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 315 KiB/s wr, 26 op/s 2023-03-09T21:41:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24183.676724 secs 2023-03-09T21:41:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:13 smithi181 ceph-mon[94189]: pgmap v13408: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 315 KiB/s wr, 26 op/s 2023-03-09T21:41:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24183.676724 secs 2023-03-09T21:41:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:13 smithi038 ceph-mon[92422]: pgmap v13408: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 315 KiB/s wr, 26 op/s 2023-03-09T21:41:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24183.676724 secs 2023-03-09T21:41:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:15 smithi122 ceph-mon[93869]: pgmap v13409: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 303 KiB/s wr, 12 op/s 2023-03-09T21:41:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:15 smithi181 ceph-mon[94189]: pgmap v13409: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 303 KiB/s wr, 12 op/s 2023-03-09T21:41:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:15 smithi038 ceph-mon[92422]: pgmap v13409: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 303 KiB/s wr, 12 op/s 2023-03-09T21:41:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24186.328619 secs 2023-03-09T21:41:16.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24186.328619 secs 2023-03-09T21:41:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24186.328619 secs 2023-03-09T21:41:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:17 smithi122 ceph-mon[93869]: pgmap v13410: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 258 KiB/s wr, 8 op/s 2023-03-09T21:41:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:17 smithi181 ceph-mon[94189]: pgmap v13410: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 258 KiB/s wr, 8 op/s 2023-03-09T21:41:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:17 smithi038 ceph-mon[92422]: pgmap v13410: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 258 KiB/s wr, 8 op/s 2023-03-09T21:41:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24650.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:41:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24188.676822 secs 2023-03-09T21:41:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24188.676822 secs 2023-03-09T21:41:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24188.676822 secs 2023-03-09T21:41:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:19 smithi122 ceph-mon[93869]: pgmap v13411: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 258 KiB/s wr, 8 op/s 2023-03-09T21:41:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:19 smithi181 ceph-mon[94189]: pgmap v13411: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 258 KiB/s wr, 8 op/s 2023-03-09T21:41:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:19 smithi038 ceph-mon[92422]: pgmap v13411: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 258 KiB/s wr, 8 op/s 2023-03-09T21:41:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24191.328682 secs 2023-03-09T21:41:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:21 smithi122 ceph-mon[93869]: pgmap v13412: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 270 KiB/s wr, 8 op/s 2023-03-09T21:41:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:21 smithi122 ceph-mon[93869]: osdmap e1052: 12 total, 12 up, 12 in 2023-03-09T21:41:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24191.328682 secs 2023-03-09T21:41:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:21 smithi181 ceph-mon[94189]: pgmap v13412: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 270 KiB/s wr, 8 op/s 2023-03-09T21:41:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:21 smithi181 ceph-mon[94189]: osdmap e1052: 12 total, 12 up, 12 in 2023-03-09T21:41:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24191.328682 secs 2023-03-09T21:41:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:21 smithi038 ceph-mon[92422]: pgmap v13412: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 270 KiB/s wr, 8 op/s 2023-03-09T21:41:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:21 smithi038 ceph-mon[92422]: osdmap e1052: 12 total, 12 up, 12 in 2023-03-09T21:41:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:23 smithi122 ceph-mon[93869]: pgmap v13414: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 224 KiB/s wr, 9 op/s 2023-03-09T21:41:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24193.676934 secs 2023-03-09T21:41:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:23 smithi181 ceph-mon[94189]: pgmap v13414: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 224 KiB/s wr, 9 op/s 2023-03-09T21:41:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24193.676934 secs 2023-03-09T21:41:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:23 smithi038 ceph-mon[92422]: pgmap v13414: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 224 KiB/s wr, 9 op/s 2023-03-09T21:41:23.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24193.676934 secs 2023-03-09T21:41:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24655.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:41:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:25 smithi122 ceph-mon[93869]: pgmap v13415: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 201 KiB/s wr, 9 op/s 2023-03-09T21:41:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:25 smithi181 ceph-mon[94189]: pgmap v13415: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 201 KiB/s wr, 9 op/s 2023-03-09T21:41:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:25 smithi038 ceph-mon[92422]: pgmap v13415: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 201 KiB/s wr, 9 op/s 2023-03-09T21:41:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24196.328768 secs 2023-03-09T21:41:26.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24196.328768 secs 2023-03-09T21:41:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24196.328768 secs 2023-03-09T21:41:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:27 smithi122 ceph-mon[93869]: pgmap v13416: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 185 KiB/s wr, 9 op/s 2023-03-09T21:41:27.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:27 smithi181 ceph-mon[94189]: pgmap v13416: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 185 KiB/s wr, 9 op/s 2023-03-09T21:41:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:27 smithi038 ceph-mon[92422]: pgmap v13416: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 185 KiB/s wr, 9 op/s 2023-03-09T21:41:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24198.677051 secs 2023-03-09T21:41:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24198.677051 secs 2023-03-09T21:41:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24198.677051 secs 2023-03-09T21:41:28.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24660.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:41:29.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:29 smithi122 ceph-mon[93869]: pgmap v13417: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 185 KiB/s wr, 9 op/s 2023-03-09T21:41:29.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:29 smithi181 ceph-mon[94189]: pgmap v13417: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 185 KiB/s wr, 9 op/s 2023-03-09T21:41:29.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:29 smithi038 ceph-mon[92422]: pgmap v13417: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 185 KiB/s wr, 9 op/s 2023-03-09T21:41:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24201.328828 secs 2023-03-09T21:41:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:31 smithi122 ceph-mon[93869]: pgmap v13418: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 162 KiB/s wr, 9 op/s 2023-03-09T21:41:31.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24201.328828 secs 2023-03-09T21:41:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:31 smithi181 ceph-mon[94189]: pgmap v13418: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 162 KiB/s wr, 9 op/s 2023-03-09T21:41:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24201.328828 secs 2023-03-09T21:41:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:31 smithi038 ceph-mon[92422]: pgmap v13418: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 162 KiB/s wr, 9 op/s 2023-03-09T21:41:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:32 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:41:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:32 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:41:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:32 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:41:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:32 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:41:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:32 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:41:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:32 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:41:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:33 smithi122 ceph-mon[93869]: pgmap v13419: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 239 KiB/s wr, 10 op/s 2023-03-09T21:41:33.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24203.677151 secs 2023-03-09T21:41:33.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:33 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:41:33.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:33 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:41:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:33 smithi181 ceph-mon[94189]: pgmap v13419: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 239 KiB/s wr, 10 op/s 2023-03-09T21:41:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24203.677151 secs 2023-03-09T21:41:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:33 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:41:33.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:33 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:41:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:33 smithi038 ceph-mon[92422]: pgmap v13419: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 239 KiB/s wr, 10 op/s 2023-03-09T21:41:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24203.677151 secs 2023-03-09T21:41:33.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:33 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:41:33.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:33 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:41:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24665.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:41:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:35 smithi122 ceph-mon[93869]: pgmap v13420: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 152 KiB/s wr, 7 op/s 2023-03-09T21:41:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:41:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:35 smithi181 ceph-mon[94189]: pgmap v13420: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 152 KiB/s wr, 7 op/s 2023-03-09T21:41:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:41:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:35 smithi038 ceph-mon[92422]: pgmap v13420: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 152 KiB/s wr, 7 op/s 2023-03-09T21:41:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:41:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24206.328890 secs 2023-03-09T21:41:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:41:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:41:36.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:41:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24206.328890 secs 2023-03-09T21:41:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:41:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:41:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:41:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24206.328890 secs 2023-03-09T21:41:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:41:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:41:36.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:41:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:37 smithi122 ceph-mon[93869]: pgmap v13421: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 247 KiB/s rd, 244 KiB/s wr, 9 op/s 2023-03-09T21:41:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:37 smithi181 ceph-mon[94189]: pgmap v13421: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 247 KiB/s rd, 244 KiB/s wr, 9 op/s 2023-03-09T21:41:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:37 smithi038 ceph-mon[92422]: pgmap v13421: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 247 KiB/s rd, 244 KiB/s wr, 9 op/s 2023-03-09T21:41:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24208.677290 secs 2023-03-09T21:41:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24208.677290 secs 2023-03-09T21:41:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24208.677290 secs 2023-03-09T21:41:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24670.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:41:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:39 smithi122 ceph-mon[93869]: pgmap v13422: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 191 KiB/s wr, 7 op/s 2023-03-09T21:41:39.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:39 smithi181 ceph-mon[94189]: pgmap v13422: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 191 KiB/s wr, 7 op/s 2023-03-09T21:41:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:39 smithi038 ceph-mon[92422]: pgmap v13422: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 191 KiB/s wr, 7 op/s 2023-03-09T21:41:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24211.328950 secs 2023-03-09T21:41:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:41 smithi122 ceph-mon[93869]: pgmap v13423: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 191 KiB/s wr, 7 op/s 2023-03-09T21:41:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:41:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:41:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24211.328950 secs 2023-03-09T21:41:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:41 smithi181 ceph-mon[94189]: pgmap v13423: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 191 KiB/s wr, 7 op/s 2023-03-09T21:41:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:41:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:41:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24211.328950 secs 2023-03-09T21:41:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:41 smithi038 ceph-mon[92422]: pgmap v13423: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 191 KiB/s wr, 7 op/s 2023-03-09T21:41:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:41:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:41:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:43 smithi122 ceph-mon[93869]: pgmap v13424: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 318 KiB/s rd, 315 KiB/s wr, 9 op/s 2023-03-09T21:41:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24213.677402 secs 2023-03-09T21:41:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:43 smithi181 ceph-mon[94189]: pgmap v13424: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 318 KiB/s rd, 315 KiB/s wr, 9 op/s 2023-03-09T21:41:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24213.677402 secs 2023-03-09T21:41:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:43 smithi038 ceph-mon[92422]: pgmap v13424: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 318 KiB/s rd, 315 KiB/s wr, 9 op/s 2023-03-09T21:41:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24213.677402 secs 2023-03-09T21:41:43.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24675.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:41:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:45 smithi122 ceph-mon[93869]: pgmap v13425: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 217 KiB/s wr, 7 op/s 2023-03-09T21:41:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:45 smithi181 ceph-mon[94189]: pgmap v13425: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 217 KiB/s wr, 7 op/s 2023-03-09T21:41:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:45 smithi038 ceph-mon[92422]: pgmap v13425: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 217 KiB/s wr, 7 op/s 2023-03-09T21:41:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24216.329034 secs 2023-03-09T21:41:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24216.329034 secs 2023-03-09T21:41:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24216.329034 secs 2023-03-09T21:41:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:47 smithi122 ceph-mon[93869]: pgmap v13426: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 405 KiB/s wr, 10 op/s 2023-03-09T21:41:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:41:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:41:47.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:47 smithi181 ceph-mon[94189]: pgmap v13426: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 405 KiB/s wr, 10 op/s 2023-03-09T21:41:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:41:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:41:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:47 smithi038 ceph-mon[92422]: pgmap v13426: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 405 KiB/s wr, 10 op/s 2023-03-09T21:41:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:41:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:41:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24680.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:41:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:49 smithi122 ceph-mon[93869]: pgmap v13427: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 313 KiB/s wr, 8 op/s 2023-03-09T21:41:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:49 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:41:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:49 smithi181 ceph-mon[94189]: pgmap v13427: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 313 KiB/s wr, 8 op/s 2023-03-09T21:41:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:49 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:41:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:49 smithi038 ceph-mon[92422]: pgmap v13427: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 313 KiB/s wr, 8 op/s 2023-03-09T21:41:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:49 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:41:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:50 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:41:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:50 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:41:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:50 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:41:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:50 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:41:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:50 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:41:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:50 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:41:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:50 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:41:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:50 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:41:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:50 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:41:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24221.329094 secs 2023-03-09T21:41:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:51 smithi122 ceph-mon[93869]: pgmap v13428: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 332 KiB/s wr, 8 op/s 2023-03-09T21:41:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24221.329094 secs 2023-03-09T21:41:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:51 smithi181 ceph-mon[94189]: pgmap v13428: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 332 KiB/s wr, 8 op/s 2023-03-09T21:41:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24221.329094 secs 2023-03-09T21:41:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:51 smithi038 ceph-mon[92422]: pgmap v13428: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 332 KiB/s wr, 8 op/s 2023-03-09T21:41:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:53 smithi038 ceph-mon[92422]: pgmap v13429: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 385 KiB/s wr, 11 op/s 2023-03-09T21:41:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24685.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:41:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:53 smithi122 ceph-mon[93869]: pgmap v13429: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 385 KiB/s wr, 11 op/s 2023-03-09T21:41:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:53 smithi181 ceph-mon[94189]: pgmap v13429: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 385 KiB/s wr, 11 op/s 2023-03-09T21:41:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:55 smithi038 ceph-mon[92422]: pgmap v13430: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 589 KiB/s rd, 575 KiB/s wr, 11 op/s 2023-03-09T21:41:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:41:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:41:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:55 smithi122 ceph-mon[93869]: pgmap v13430: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 589 KiB/s rd, 575 KiB/s wr, 11 op/s 2023-03-09T21:41:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:41:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:41:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:55 smithi181 ceph-mon[94189]: pgmap v13430: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 589 KiB/s rd, 575 KiB/s wr, 11 op/s 2023-03-09T21:41:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:41:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:41:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24226.329156 secs 2023-03-09T21:41:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24226.329156 secs 2023-03-09T21:41:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24226.329156 secs 2023-03-09T21:41:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:57 smithi122 ceph-mon[93869]: pgmap v13431: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 591 KiB/s rd, 575 KiB/s wr, 12 op/s 2023-03-09T21:41:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:41:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:41:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:57 smithi181 ceph-mon[94189]: pgmap v13431: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 591 KiB/s rd, 575 KiB/s wr, 12 op/s 2023-03-09T21:41:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:41:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:41:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:57 smithi038 ceph-mon[92422]: pgmap v13431: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 591 KiB/s rd, 575 KiB/s wr, 12 op/s 2023-03-09T21:41:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:41:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:41:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24690.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:41:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:41:59 smithi122 ceph-mon[93869]: pgmap v13432: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 467 KiB/s rd, 452 KiB/s wr, 11 op/s 2023-03-09T21:41:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:41:59 smithi181 ceph-mon[94189]: pgmap v13432: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 467 KiB/s rd, 452 KiB/s wr, 11 op/s 2023-03-09T21:41:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:41:59 smithi038 ceph-mon[92422]: pgmap v13432: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 467 KiB/s rd, 452 KiB/s wr, 11 op/s 2023-03-09T21:42:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24231.329214 secs 2023-03-09T21:42:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:01 smithi122 ceph-mon[93869]: pgmap v13433: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 495 KiB/s wr, 11 op/s 2023-03-09T21:42:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:42:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:42:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24231.329214 secs 2023-03-09T21:42:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:01 smithi181 ceph-mon[94189]: pgmap v13433: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 495 KiB/s wr, 11 op/s 2023-03-09T21:42:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:42:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:42:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24231.329214 secs 2023-03-09T21:42:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:01 smithi038 ceph-mon[92422]: pgmap v13433: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 495 KiB/s wr, 11 op/s 2023-03-09T21:42:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:42:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:42:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24695.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:42:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:03 smithi122 ceph-mon[93869]: pgmap v13434: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 477 KiB/s wr, 13 op/s 2023-03-09T21:42:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:03 smithi181 ceph-mon[94189]: pgmap v13434: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 477 KiB/s wr, 13 op/s 2023-03-09T21:42:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:03 smithi038 ceph-mon[92422]: pgmap v13434: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 477 KiB/s wr, 13 op/s 2023-03-09T21:42:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:05 smithi122 ceph-mon[93869]: pgmap v13435: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 497 KiB/s wr, 13 op/s 2023-03-09T21:42:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:05 smithi181 ceph-mon[94189]: pgmap v13435: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 497 KiB/s wr, 13 op/s 2023-03-09T21:42:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:05 smithi038 ceph-mon[92422]: pgmap v13435: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 497 KiB/s wr, 13 op/s 2023-03-09T21:42:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24236.329274 secs 2023-03-09T21:42:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24236.329274 secs 2023-03-09T21:42:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24236.329274 secs 2023-03-09T21:42:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:07 smithi122 ceph-mon[93869]: pgmap v13436: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 205 KiB/s wr, 14 op/s 2023-03-09T21:42:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:42:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:42:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:07 smithi181 ceph-mon[94189]: pgmap v13436: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 205 KiB/s wr, 14 op/s 2023-03-09T21:42:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:42:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:42:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:07 smithi038 ceph-mon[92422]: pgmap v13436: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 205 KiB/s wr, 14 op/s 2023-03-09T21:42:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:42:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:42:08.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24700.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:42:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24238.678009 secs 2023-03-09T21:42:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24238.678009 secs 2023-03-09T21:42:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24238.678009 secs 2023-03-09T21:42:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:09 smithi122 ceph-mon[93869]: pgmap v13437: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 226 KiB/s wr, 23 op/s 2023-03-09T21:42:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:09 smithi181 ceph-mon[94189]: pgmap v13437: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 226 KiB/s wr, 23 op/s 2023-03-09T21:42:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:09 smithi038 ceph-mon[92422]: pgmap v13437: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 226 KiB/s wr, 23 op/s 2023-03-09T21:42:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:10 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:42:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:10 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:42:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:10 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:42:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:10 smithi122 ceph-mon[93869]: osdmap e1053: 12 total, 12 up, 12 in 2023-03-09T21:42:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:10 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:42:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:10 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:42:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:10 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:42:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:10 smithi181 ceph-mon[94189]: osdmap e1053: 12 total, 12 up, 12 in 2023-03-09T21:42:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:10 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:42:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:10 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:42:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:10 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:42:11.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:10 smithi038 ceph-mon[92422]: osdmap e1053: 12 total, 12 up, 12 in 2023-03-09T21:42:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24241.329334 secs 2023-03-09T21:42:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:11 smithi122 ceph-mon[93869]: pgmap v13439: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 458 KiB/s wr, 28 op/s 2023-03-09T21:42:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:11 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:42:11.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:11 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:42:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24241.329334 secs 2023-03-09T21:42:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:11 smithi181 ceph-mon[94189]: pgmap v13439: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 458 KiB/s wr, 28 op/s 2023-03-09T21:42:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:11 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:42:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:11 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:42:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24241.329334 secs 2023-03-09T21:42:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:11 smithi038 ceph-mon[92422]: pgmap v13439: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 458 KiB/s wr, 28 op/s 2023-03-09T21:42:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:11 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:42:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:11 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:42:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24705.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:42:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:13 smithi122 ceph-mon[93869]: pgmap v13440: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 629 KiB/s rd, 670 KiB/s wr, 28 op/s 2023-03-09T21:42:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24243.678136 secs 2023-03-09T21:42:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:42:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:42:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:13 smithi181 ceph-mon[94189]: pgmap v13440: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 629 KiB/s rd, 670 KiB/s wr, 28 op/s 2023-03-09T21:42:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24243.678136 secs 2023-03-09T21:42:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:42:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:42:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:13 smithi038 ceph-mon[92422]: pgmap v13440: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 629 KiB/s rd, 670 KiB/s wr, 28 op/s 2023-03-09T21:42:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24243.678136 secs 2023-03-09T21:42:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:42:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:42:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:15 smithi122 ceph-mon[93869]: pgmap v13441: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 605 KiB/s wr, 26 op/s 2023-03-09T21:42:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:15 smithi122 ceph-mon[93869]: osdmap e1054: 12 total, 12 up, 12 in 2023-03-09T21:42:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:15 smithi181 ceph-mon[94189]: pgmap v13441: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 605 KiB/s wr, 26 op/s 2023-03-09T21:42:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:15 smithi181 ceph-mon[94189]: osdmap e1054: 12 total, 12 up, 12 in 2023-03-09T21:42:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:15 smithi038 ceph-mon[92422]: pgmap v13441: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 605 KiB/s wr, 26 op/s 2023-03-09T21:42:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:15 smithi038 ceph-mon[92422]: osdmap e1054: 12 total, 12 up, 12 in 2023-03-09T21:42:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24246.329392 secs 2023-03-09T21:42:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24246.329392 secs 2023-03-09T21:42:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24246.329392 secs 2023-03-09T21:42:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:17 smithi122 ceph-mon[93869]: pgmap v13443: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 693 KiB/s rd, 691 KiB/s wr, 12 op/s 2023-03-09T21:42:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:17 smithi181 ceph-mon[94189]: pgmap v13443: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 693 KiB/s rd, 691 KiB/s wr, 12 op/s 2023-03-09T21:42:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:17 smithi038 ceph-mon[92422]: pgmap v13443: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 693 KiB/s rd, 691 KiB/s wr, 12 op/s 2023-03-09T21:42:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24710.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:42:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24248.678232 secs 2023-03-09T21:42:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24248.678232 secs 2023-03-09T21:42:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24248.678232 secs 2023-03-09T21:42:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:19 smithi122 ceph-mon[93869]: pgmap v13444: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 375 KiB/s wr, 10 op/s 2023-03-09T21:42:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:19 smithi181 ceph-mon[94189]: pgmap v13444: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 375 KiB/s wr, 10 op/s 2023-03-09T21:42:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:19 smithi038 ceph-mon[92422]: pgmap v13444: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 375 KiB/s wr, 10 op/s 2023-03-09T21:42:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24251.329452 secs 2023-03-09T21:42:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:21 smithi122 ceph-mon[93869]: pgmap v13445: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 317 KiB/s wr, 9 op/s 2023-03-09T21:42:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24251.329452 secs 2023-03-09T21:42:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:21 smithi181 ceph-mon[94189]: pgmap v13445: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 317 KiB/s wr, 9 op/s 2023-03-09T21:42:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24251.329452 secs 2023-03-09T21:42:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:21 smithi038 ceph-mon[92422]: pgmap v13445: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 317 KiB/s wr, 9 op/s 2023-03-09T21:42:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24715.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:42:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:23 smithi122 ceph-mon[93869]: pgmap v13446: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 103 KiB/s wr, 8 op/s 2023-03-09T21:42:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24253.678350 secs 2023-03-09T21:42:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:23 smithi181 ceph-mon[94189]: pgmap v13446: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 103 KiB/s wr, 8 op/s 2023-03-09T21:42:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24253.678350 secs 2023-03-09T21:42:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:23 smithi038 ceph-mon[92422]: pgmap v13446: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 103 KiB/s wr, 8 op/s 2023-03-09T21:42:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24253.678350 secs 2023-03-09T21:42:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:25 smithi122 ceph-mon[93869]: pgmap v13447: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 103 KiB/s wr, 7 op/s 2023-03-09T21:42:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:25 smithi181 ceph-mon[94189]: pgmap v13447: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 103 KiB/s wr, 7 op/s 2023-03-09T21:42:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:25 smithi038 ceph-mon[92422]: pgmap v13447: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 103 KiB/s wr, 7 op/s 2023-03-09T21:42:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24256.329512 secs 2023-03-09T21:42:26.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24256.329512 secs 2023-03-09T21:42:26.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24256.329512 secs 2023-03-09T21:42:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:27 smithi122 ceph-mon[93869]: pgmap v13448: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 98 KiB/s wr, 9 op/s 2023-03-09T21:42:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:27 smithi181 ceph-mon[94189]: pgmap v13448: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 98 KiB/s wr, 9 op/s 2023-03-09T21:42:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:27 smithi038 ceph-mon[92422]: pgmap v13448: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 98 KiB/s wr, 9 op/s 2023-03-09T21:42:28.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24720.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:42:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24258.678493 secs 2023-03-09T21:42:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24258.678493 secs 2023-03-09T21:42:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24258.678493 secs 2023-03-09T21:42:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:29 smithi122 ceph-mon[93869]: pgmap v13449: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 140 KiB/s wr, 7 op/s 2023-03-09T21:42:29.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:29 smithi181 ceph-mon[94189]: pgmap v13449: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 140 KiB/s wr, 7 op/s 2023-03-09T21:42:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:29 smithi038 ceph-mon[92422]: pgmap v13449: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 140 KiB/s wr, 7 op/s 2023-03-09T21:42:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24261.329573 secs 2023-03-09T21:42:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:31 smithi122 ceph-mon[93869]: pgmap v13450: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 73 KiB/s wr, 6 op/s 2023-03-09T21:42:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24261.329573 secs 2023-03-09T21:42:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:31 smithi181 ceph-mon[94189]: pgmap v13450: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 73 KiB/s wr, 6 op/s 2023-03-09T21:42:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24261.329573 secs 2023-03-09T21:42:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:31 smithi038 ceph-mon[92422]: pgmap v13450: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 73 KiB/s wr, 6 op/s 2023-03-09T21:42:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24725.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:42:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:33 smithi122 ceph-mon[93869]: pgmap v13451: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 73 KiB/s wr, 8 op/s 2023-03-09T21:42:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24263.678586 secs 2023-03-09T21:42:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:33 smithi181 ceph-mon[94189]: pgmap v13451: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 73 KiB/s wr, 8 op/s 2023-03-09T21:42:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24263.678586 secs 2023-03-09T21:42:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:33 smithi038 ceph-mon[92422]: pgmap v13451: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 73 KiB/s wr, 8 op/s 2023-03-09T21:42:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24263.678586 secs 2023-03-09T21:42:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:35 smithi122 ceph-mon[93869]: pgmap v13452: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 73 KiB/s wr, 6 op/s 2023-03-09T21:42:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:35 smithi181 ceph-mon[94189]: pgmap v13452: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 73 KiB/s wr, 6 op/s 2023-03-09T21:42:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:35 smithi038 ceph-mon[92422]: pgmap v13452: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 73 KiB/s wr, 6 op/s 2023-03-09T21:42:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24266.329633 secs 2023-03-09T21:42:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:42:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24266.329633 secs 2023-03-09T21:42:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:42:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24266.329633 secs 2023-03-09T21:42:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:42:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:37 smithi122 ceph-mon[93869]: pgmap v13453: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 55 KiB/s wr, 7 op/s 2023-03-09T21:42:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:42:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:42:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:42:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:37 smithi181 ceph-mon[94189]: pgmap v13453: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 55 KiB/s wr, 7 op/s 2023-03-09T21:42:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:42:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:42:37.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:42:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:37 smithi038 ceph-mon[92422]: pgmap v13453: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 55 KiB/s wr, 7 op/s 2023-03-09T21:42:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:42:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:42:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:42:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24730.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:42:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24268.678688 secs 2023-03-09T21:42:38.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24268.678688 secs 2023-03-09T21:42:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24268.678688 secs 2023-03-09T21:42:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:39 smithi038 ceph-mon[92422]: pgmap v13454: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 152 KiB/s wr, 7 op/s 2023-03-09T21:42:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:39 smithi122 ceph-mon[93869]: pgmap v13454: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 152 KiB/s wr, 7 op/s 2023-03-09T21:42:40.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:39 smithi181 ceph-mon[94189]: pgmap v13454: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 152 KiB/s wr, 7 op/s 2023-03-09T21:42:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:42:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:42:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:42:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:42:41.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:42:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:42:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24271.329692 secs 2023-03-09T21:42:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:41 smithi038 ceph-mon[92422]: pgmap v13455: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 99 KiB/s wr, 7 op/s 2023-03-09T21:42:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24271.329692 secs 2023-03-09T21:42:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:41 smithi122 ceph-mon[93869]: pgmap v13455: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 99 KiB/s wr, 7 op/s 2023-03-09T21:42:42.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24271.329692 secs 2023-03-09T21:42:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:41 smithi181 ceph-mon[94189]: pgmap v13455: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 99 KiB/s wr, 7 op/s 2023-03-09T21:42:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24735.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:42:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:43 smithi038 ceph-mon[92422]: pgmap v13456: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 206 KiB/s wr, 9 op/s 2023-03-09T21:42:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:43 smithi122 ceph-mon[93869]: pgmap v13456: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 206 KiB/s wr, 9 op/s 2023-03-09T21:42:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:43 smithi181 ceph-mon[94189]: pgmap v13456: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 206 KiB/s wr, 9 op/s 2023-03-09T21:42:45.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:44 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:42:45.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:44 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:42:45.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:44 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:42:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:45 smithi122 ceph-mon[93869]: pgmap v13457: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 225 KiB/s wr, 8 op/s 2023-03-09T21:42:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:45 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:42:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:45 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:42:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:45 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:42:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:45 smithi181 ceph-mon[94189]: pgmap v13457: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 225 KiB/s wr, 8 op/s 2023-03-09T21:42:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:45 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:42:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:45 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:42:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:45 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:42:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:45 smithi038 ceph-mon[92422]: pgmap v13457: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 225 KiB/s wr, 8 op/s 2023-03-09T21:42:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:45 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:42:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:45 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:42:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:45 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:42:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24276.329751 secs 2023-03-09T21:42:47.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24276.329751 secs 2023-03-09T21:42:47.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24276.329751 secs 2023-03-09T21:42:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:47 smithi122 ceph-mon[93869]: pgmap v13458: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 363 KiB/s rd, 358 KiB/s wr, 10 op/s 2023-03-09T21:42:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:42:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:42:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:47 smithi181 ceph-mon[94189]: pgmap v13458: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 363 KiB/s rd, 358 KiB/s wr, 10 op/s 2023-03-09T21:42:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:42:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:42:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:47 smithi038 ceph-mon[92422]: pgmap v13458: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 363 KiB/s rd, 358 KiB/s wr, 10 op/s 2023-03-09T21:42:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:42:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:42:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24740.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:42:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24278.678947 secs 2023-03-09T21:42:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:42:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:42:49.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24278.678947 secs 2023-03-09T21:42:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:42:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:42:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24278.678947 secs 2023-03-09T21:42:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:42:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:42:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:49 smithi122 ceph-mon[93869]: pgmap v13459: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 542 KiB/s rd, 538 KiB/s wr, 10 op/s 2023-03-09T21:42:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:42:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:42:50.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:49 smithi181 ceph-mon[94189]: pgmap v13459: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 542 KiB/s rd, 538 KiB/s wr, 10 op/s 2023-03-09T21:42:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:42:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:42:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:49 smithi038 ceph-mon[92422]: pgmap v13459: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 542 KiB/s rd, 538 KiB/s wr, 10 op/s 2023-03-09T21:42:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:42:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:42:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24281.329812 secs 2023-03-09T21:42:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:50 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:42:51.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24281.329812 secs 2023-03-09T21:42:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:50 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:42:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24281.329812 secs 2023-03-09T21:42:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:50 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:42:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:51 smithi122 ceph-mon[93869]: pgmap v13460: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 474 KiB/s rd, 459 KiB/s wr, 9 op/s 2023-03-09T21:42:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:42:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:42:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:51 smithi181 ceph-mon[94189]: pgmap v13460: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 474 KiB/s rd, 459 KiB/s wr, 9 op/s 2023-03-09T21:42:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:42:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:42:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:51 smithi038 ceph-mon[92422]: pgmap v13460: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 474 KiB/s rd, 459 KiB/s wr, 9 op/s 2023-03-09T21:42:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:42:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:42:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24745.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:42:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:53 smithi122 ceph-mon[93869]: pgmap v13461: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 517 KiB/s rd, 501 KiB/s wr, 13 op/s 2023-03-09T21:42:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24283.679095 secs 2023-03-09T21:42:54.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:53 smithi181 ceph-mon[94189]: pgmap v13461: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 517 KiB/s rd, 501 KiB/s wr, 13 op/s 2023-03-09T21:42:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24283.679095 secs 2023-03-09T21:42:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:53 smithi038 ceph-mon[92422]: pgmap v13461: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 517 KiB/s rd, 501 KiB/s wr, 13 op/s 2023-03-09T21:42:54.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24283.679095 secs 2023-03-09T21:42:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:55 smithi122 ceph-mon[93869]: pgmap v13462: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 481 KiB/s wr, 11 op/s 2023-03-09T21:42:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:42:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:42:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24286.329872 secs 2023-03-09T21:42:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:55 smithi181 ceph-mon[94189]: pgmap v13462: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 481 KiB/s wr, 11 op/s 2023-03-09T21:42:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:42:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:42:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24286.329872 secs 2023-03-09T21:42:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:55 smithi038 ceph-mon[92422]: pgmap v13462: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 481 KiB/s wr, 11 op/s 2023-03-09T21:42:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:42:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:42:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24286.329872 secs 2023-03-09T21:42:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:57 smithi122 ceph-mon[93869]: pgmap v13463: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 646 KiB/s rd, 630 KiB/s wr, 13 op/s 2023-03-09T21:42:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:42:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:42:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:57 smithi181 ceph-mon[94189]: pgmap v13463: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 646 KiB/s rd, 630 KiB/s wr, 13 op/s 2023-03-09T21:42:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:42:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:42:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:57 smithi038 ceph-mon[92422]: pgmap v13463: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 646 KiB/s rd, 630 KiB/s wr, 13 op/s 2023-03-09T21:42:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:42:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:42:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24288.679213 secs 2023-03-09T21:42:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24288.679213 secs 2023-03-09T21:42:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24288.679213 secs 2023-03-09T21:42:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24750.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:42:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:42:59 smithi122 ceph-mon[93869]: pgmap v13464: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 497 KiB/s wr, 11 op/s 2023-03-09T21:42:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:42:59 smithi181 ceph-mon[94189]: pgmap v13464: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 497 KiB/s wr, 11 op/s 2023-03-09T21:42:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:42:59 smithi038 ceph-mon[92422]: pgmap v13464: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 497 KiB/s wr, 11 op/s 2023-03-09T21:43:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24291.329934 secs 2023-03-09T21:43:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:01 smithi122 ceph-mon[93869]: pgmap v13465: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 439 KiB/s wr, 10 op/s 2023-03-09T21:43:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24291.329934 secs 2023-03-09T21:43:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:01 smithi181 ceph-mon[94189]: pgmap v13465: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 439 KiB/s wr, 10 op/s 2023-03-09T21:43:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24291.329934 secs 2023-03-09T21:43:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:01 smithi038 ceph-mon[92422]: pgmap v13465: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 439 KiB/s wr, 10 op/s 2023-03-09T21:43:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:03 smithi122 ceph-mon[93869]: pgmap v13466: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 442 KiB/s wr, 14 op/s 2023-03-09T21:43:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24293.679278 secs 2023-03-09T21:43:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:03 smithi181 ceph-mon[94189]: pgmap v13466: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 442 KiB/s wr, 14 op/s 2023-03-09T21:43:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24293.679278 secs 2023-03-09T21:43:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:03 smithi038 ceph-mon[92422]: pgmap v13466: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 442 KiB/s wr, 14 op/s 2023-03-09T21:43:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24293.679278 secs 2023-03-09T21:43:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24755.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:43:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:05 smithi122 ceph-mon[93869]: pgmap v13467: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 612 KiB/s rd, 646 KiB/s wr, 25 op/s 2023-03-09T21:43:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:05 smithi181 ceph-mon[94189]: pgmap v13467: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 612 KiB/s rd, 646 KiB/s wr, 25 op/s 2023-03-09T21:43:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:05 smithi038 ceph-mon[92422]: pgmap v13467: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 612 KiB/s rd, 646 KiB/s wr, 25 op/s 2023-03-09T21:43:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24296.329984 secs 2023-03-09T21:43:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24296.329984 secs 2023-03-09T21:43:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24296.329984 secs 2023-03-09T21:43:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:07 smithi122 ceph-mon[93869]: pgmap v13468: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 525 KiB/s rd, 558 KiB/s wr, 26 op/s 2023-03-09T21:43:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:43:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:43:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:07 smithi181 ceph-mon[94189]: pgmap v13468: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 525 KiB/s rd, 558 KiB/s wr, 26 op/s 2023-03-09T21:43:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:43:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:43:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:07 smithi038 ceph-mon[92422]: pgmap v13468: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 525 KiB/s rd, 558 KiB/s wr, 26 op/s 2023-03-09T21:43:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:43:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:43:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24298.679380 secs 2023-03-09T21:43:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24298.679380 secs 2023-03-09T21:43:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24298.679380 secs 2023-03-09T21:43:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24760.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:43:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:09 smithi122 ceph-mon[93869]: pgmap v13469: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 390 KiB/s wr, 23 op/s 2023-03-09T21:43:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:43:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:43:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:09 smithi181 ceph-mon[94189]: pgmap v13469: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 390 KiB/s wr, 23 op/s 2023-03-09T21:43:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:43:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:43:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:09 smithi038 ceph-mon[92422]: pgmap v13469: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 390 KiB/s wr, 23 op/s 2023-03-09T21:43:09.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:43:09.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:43:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:10 smithi122 ceph-mon[93869]: osdmap e1055: 12 total, 12 up, 12 in 2023-03-09T21:43:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:10 smithi181 ceph-mon[94189]: osdmap e1055: 12 total, 12 up, 12 in 2023-03-09T21:43:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:10 smithi038 ceph-mon[92422]: osdmap e1055: 12 total, 12 up, 12 in 2023-03-09T21:43:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24301.330043 secs 2023-03-09T21:43:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:11 smithi122 ceph-mon[93869]: pgmap v13471: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 520 KiB/s rd, 561 KiB/s wr, 28 op/s 2023-03-09T21:43:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24301.330043 secs 2023-03-09T21:43:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:11 smithi181 ceph-mon[94189]: pgmap v13471: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 520 KiB/s rd, 561 KiB/s wr, 28 op/s 2023-03-09T21:43:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24301.330043 secs 2023-03-09T21:43:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:11 smithi038 ceph-mon[92422]: pgmap v13471: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 520 KiB/s rd, 561 KiB/s wr, 28 op/s 2023-03-09T21:43:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:13 smithi122 ceph-mon[93869]: pgmap v13472: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 519 KiB/s rd, 534 KiB/s wr, 25 op/s 2023-03-09T21:43:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24303.679462 secs 2023-03-09T21:43:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:13 smithi181 ceph-mon[94189]: pgmap v13472: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 519 KiB/s rd, 534 KiB/s wr, 25 op/s 2023-03-09T21:43:13.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24303.679462 secs 2023-03-09T21:43:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:13 smithi038 ceph-mon[92422]: pgmap v13472: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 519 KiB/s rd, 534 KiB/s wr, 25 op/s 2023-03-09T21:43:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24303.679462 secs 2023-03-09T21:43:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24765.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:43:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:15 smithi122 ceph-mon[93869]: pgmap v13473: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 376 KiB/s wr, 8 op/s 2023-03-09T21:43:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:15 smithi122 ceph-mon[93869]: osdmap e1056: 12 total, 12 up, 12 in 2023-03-09T21:43:15.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:15 smithi181 ceph-mon[94189]: pgmap v13473: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 376 KiB/s wr, 8 op/s 2023-03-09T21:43:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:15 smithi181 ceph-mon[94189]: osdmap e1056: 12 total, 12 up, 12 in 2023-03-09T21:43:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:15 smithi038 ceph-mon[92422]: pgmap v13473: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 376 KiB/s wr, 8 op/s 2023-03-09T21:43:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:15 smithi038 ceph-mon[92422]: osdmap e1056: 12 total, 12 up, 12 in 2023-03-09T21:43:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24306.330101 secs 2023-03-09T21:43:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24306.330101 secs 2023-03-09T21:43:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24306.330101 secs 2023-03-09T21:43:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:17 smithi122 ceph-mon[93869]: pgmap v13475: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 470 KiB/s wr, 10 op/s 2023-03-09T21:43:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:17 smithi181 ceph-mon[94189]: pgmap v13475: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 470 KiB/s wr, 10 op/s 2023-03-09T21:43:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:17 smithi038 ceph-mon[92422]: pgmap v13475: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 470 KiB/s wr, 10 op/s 2023-03-09T21:43:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24308.679612 secs 2023-03-09T21:43:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24308.679612 secs 2023-03-09T21:43:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24770.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:43:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24308.679612 secs 2023-03-09T21:43:19.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:19 smithi038 ceph-mon[92422]: pgmap v13476: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 159 KiB/s wr, 7 op/s 2023-03-09T21:43:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:19 smithi122 ceph-mon[93869]: pgmap v13476: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 159 KiB/s wr, 7 op/s 2023-03-09T21:43:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:19 smithi181 ceph-mon[94189]: pgmap v13476: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 159 KiB/s wr, 7 op/s 2023-03-09T21:43:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24311.330164 secs 2023-03-09T21:43:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:21 smithi038 ceph-mon[92422]: pgmap v13477: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 159 KiB/s wr, 7 op/s 2023-03-09T21:43:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24311.330164 secs 2023-03-09T21:43:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:21 smithi122 ceph-mon[93869]: pgmap v13477: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 159 KiB/s wr, 7 op/s 2023-03-09T21:43:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24311.330164 secs 2023-03-09T21:43:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:21 smithi181 ceph-mon[94189]: pgmap v13477: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 159 KiB/s wr, 7 op/s 2023-03-09T21:43:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:23 smithi038 ceph-mon[92422]: pgmap v13478: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 160 KiB/s wr, 8 op/s 2023-03-09T21:43:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24313.679688 secs 2023-03-09T21:43:23.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24775.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:43:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:23 smithi122 ceph-mon[93869]: pgmap v13478: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 160 KiB/s wr, 8 op/s 2023-03-09T21:43:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24313.679688 secs 2023-03-09T21:43:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:23 smithi181 ceph-mon[94189]: pgmap v13478: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 160 KiB/s wr, 8 op/s 2023-03-09T21:43:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24313.679688 secs 2023-03-09T21:43:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:25 smithi122 ceph-mon[93869]: pgmap v13479: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 8 op/s 2023-03-09T21:43:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:25 smithi181 ceph-mon[94189]: pgmap v13479: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 8 op/s 2023-03-09T21:43:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:25 smithi038 ceph-mon[92422]: pgmap v13479: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 8 op/s 2023-03-09T21:43:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24316.330225 secs 2023-03-09T21:43:26.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24316.330225 secs 2023-03-09T21:43:26.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24316.330225 secs 2023-03-09T21:43:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:27 smithi122 ceph-mon[93869]: pgmap v13480: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 9 op/s 2023-03-09T21:43:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:27 smithi181 ceph-mon[94189]: pgmap v13480: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 9 op/s 2023-03-09T21:43:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:27 smithi038 ceph-mon[92422]: pgmap v13480: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 9 op/s 2023-03-09T21:43:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24780.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:43:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24318.679803 secs 2023-03-09T21:43:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24318.679803 secs 2023-03-09T21:43:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24318.679803 secs 2023-03-09T21:43:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:29 smithi122 ceph-mon[93869]: pgmap v13481: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T21:43:29.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:29 smithi181 ceph-mon[94189]: pgmap v13481: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T21:43:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:29 smithi038 ceph-mon[92422]: pgmap v13481: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T21:43:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24321.330287 secs 2023-03-09T21:43:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:31 smithi122 ceph-mon[93869]: pgmap v13482: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 33 KiB/s wr, 7 op/s 2023-03-09T21:43:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24321.330287 secs 2023-03-09T21:43:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:31 smithi181 ceph-mon[94189]: pgmap v13482: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 33 KiB/s wr, 7 op/s 2023-03-09T21:43:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24321.330287 secs 2023-03-09T21:43:32.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:31 smithi038 ceph-mon[92422]: pgmap v13482: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 33 KiB/s wr, 7 op/s 2023-03-09T21:43:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24785.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:43:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:33 smithi122 ceph-mon[93869]: pgmap v13483: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 101 KiB/s wr, 9 op/s 2023-03-09T21:43:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24323.679921 secs 2023-03-09T21:43:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:33 smithi181 ceph-mon[94189]: pgmap v13483: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 101 KiB/s wr, 9 op/s 2023-03-09T21:43:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24323.679921 secs 2023-03-09T21:43:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:33 smithi038 ceph-mon[92422]: pgmap v13483: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 101 KiB/s wr, 9 op/s 2023-03-09T21:43:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24323.679921 secs 2023-03-09T21:43:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:35 smithi122 ceph-mon[93869]: pgmap v13484: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 100 KiB/s wr, 8 op/s 2023-03-09T21:43:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:35 smithi181 ceph-mon[94189]: pgmap v13484: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 100 KiB/s wr, 8 op/s 2023-03-09T21:43:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:35 smithi038 ceph-mon[92422]: pgmap v13484: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 100 KiB/s wr, 8 op/s 2023-03-09T21:43:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24326.330362 secs 2023-03-09T21:43:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24326.330362 secs 2023-03-09T21:43:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24326.330362 secs 2023-03-09T21:43:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:37 smithi122 ceph-mon[93869]: pgmap v13485: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 192 KiB/s wr, 9 op/s 2023-03-09T21:43:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:43:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:37 smithi181 ceph-mon[94189]: pgmap v13485: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 192 KiB/s wr, 9 op/s 2023-03-09T21:43:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:43:38.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:37 smithi038 ceph-mon[92422]: pgmap v13485: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 192 KiB/s wr, 9 op/s 2023-03-09T21:43:38.028 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:43:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24790.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:43:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24328.680029 secs 2023-03-09T21:43:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:38 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:43:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:38 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:43:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:38 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:43:38.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24328.680029 secs 2023-03-09T21:43:38.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:38 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:43:38.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:38 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:43:38.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:38 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:43:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24328.680029 secs 2023-03-09T21:43:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:38 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:43:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:38 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:43:39.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:38 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:43:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:39 smithi122 ceph-mon[93869]: pgmap v13486: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 192 KiB/s wr, 7 op/s 2023-03-09T21:43:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:39 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:43:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:39 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:43:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:39 smithi181 ceph-mon[94189]: pgmap v13486: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 192 KiB/s wr, 7 op/s 2023-03-09T21:43:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:39 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:43:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:39 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:43:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:39 smithi038 ceph-mon[92422]: pgmap v13486: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 192 KiB/s wr, 7 op/s 2023-03-09T21:43:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:39 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:43:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:39 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:43:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:43:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:43:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:43:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:43:41.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:43:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:43:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24331.330426 secs 2023-03-09T21:43:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:41 smithi038 ceph-mon[92422]: pgmap v13487: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 192 KiB/s wr, 8 op/s 2023-03-09T21:43:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24331.330426 secs 2023-03-09T21:43:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:41 smithi122 ceph-mon[93869]: pgmap v13487: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 192 KiB/s wr, 8 op/s 2023-03-09T21:43:42.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24331.330426 secs 2023-03-09T21:43:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:41 smithi181 ceph-mon[94189]: pgmap v13487: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 192 KiB/s wr, 8 op/s 2023-03-09T21:43:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24795.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:43:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:43 smithi038 ceph-mon[92422]: pgmap v13488: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 402 KiB/s rd, 343 KiB/s wr, 10 op/s 2023-03-09T21:43:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24333.680095 secs 2023-03-09T21:43:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:43 smithi122 ceph-mon[93869]: pgmap v13488: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 402 KiB/s rd, 343 KiB/s wr, 10 op/s 2023-03-09T21:43:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24333.680095 secs 2023-03-09T21:43:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:43 smithi181 ceph-mon[94189]: pgmap v13488: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 402 KiB/s rd, 343 KiB/s wr, 10 op/s 2023-03-09T21:43:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24333.680095 secs 2023-03-09T21:43:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:45 smithi038 ceph-mon[92422]: pgmap v13489: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 484 KiB/s rd, 426 KiB/s wr, 8 op/s 2023-03-09T21:43:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:45 smithi122 ceph-mon[93869]: pgmap v13489: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 484 KiB/s rd, 426 KiB/s wr, 8 op/s 2023-03-09T21:43:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:45 smithi181 ceph-mon[94189]: pgmap v13489: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 484 KiB/s rd, 426 KiB/s wr, 8 op/s 2023-03-09T21:43:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24336.330485 secs 2023-03-09T21:43:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24336.330485 secs 2023-03-09T21:43:47.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24336.330485 secs 2023-03-09T21:43:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:47 smithi122 ceph-mon[93869]: pgmap v13490: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 664 KiB/s rd, 435 KiB/s wr, 10 op/s 2023-03-09T21:43:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:43:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:43:48.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:47 smithi181 ceph-mon[94189]: pgmap v13490: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 664 KiB/s rd, 435 KiB/s wr, 10 op/s 2023-03-09T21:43:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:43:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:43:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:47 smithi038 ceph-mon[92422]: pgmap v13490: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 664 KiB/s rd, 435 KiB/s wr, 10 op/s 2023-03-09T21:43:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:43:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:43:48.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24800.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:43:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24338.680182 secs 2023-03-09T21:43:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:43:49.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:43:49.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24338.680182 secs 2023-03-09T21:43:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:43:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:43:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24338.680182 secs 2023-03-09T21:43:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:43:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:43:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:49 smithi122 ceph-mon[93869]: pgmap v13491: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 570 KiB/s rd, 343 KiB/s wr, 9 op/s 2023-03-09T21:43:50.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:49 smithi181 ceph-mon[94189]: pgmap v13491: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 570 KiB/s rd, 343 KiB/s wr, 9 op/s 2023-03-09T21:43:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:49 smithi038 ceph-mon[92422]: pgmap v13491: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 570 KiB/s rd, 343 KiB/s wr, 9 op/s 2023-03-09T21:43:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24341.330573 secs 2023-03-09T21:43:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:51 smithi122 ceph-mon[93869]: pgmap v13492: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 700 KiB/s rd, 461 KiB/s wr, 10 op/s 2023-03-09T21:43:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24341.330573 secs 2023-03-09T21:43:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:51 smithi181 ceph-mon[94189]: pgmap v13492: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 700 KiB/s rd, 461 KiB/s wr, 10 op/s 2023-03-09T21:43:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24341.330573 secs 2023-03-09T21:43:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:51 smithi038 ceph-mon[92422]: pgmap v13492: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 700 KiB/s rd, 461 KiB/s wr, 10 op/s 2023-03-09T21:43:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:53 smithi122 ceph-mon[93869]: pgmap v13493: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 806 KiB/s rd, 586 KiB/s wr, 13 op/s 2023-03-09T21:43:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24343.680262 secs 2023-03-09T21:43:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:53 smithi181 ceph-mon[94189]: pgmap v13493: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 806 KiB/s rd, 586 KiB/s wr, 13 op/s 2023-03-09T21:43:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24343.680262 secs 2023-03-09T21:43:53.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:53 smithi038 ceph-mon[92422]: pgmap v13493: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 806 KiB/s rd, 586 KiB/s wr, 13 op/s 2023-03-09T21:43:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24343.680262 secs 2023-03-09T21:43:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24805.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:43:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:55 smithi122 ceph-mon[93869]: pgmap v13494: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 711 KiB/s rd, 505 KiB/s wr, 12 op/s 2023-03-09T21:43:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:43:55.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:43:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:55 smithi181 ceph-mon[94189]: pgmap v13494: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 711 KiB/s rd, 505 KiB/s wr, 12 op/s 2023-03-09T21:43:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:43:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:43:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:55 smithi038 ceph-mon[92422]: pgmap v13494: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 711 KiB/s rd, 505 KiB/s wr, 12 op/s 2023-03-09T21:43:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:43:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:43:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24346.330638 secs 2023-03-09T21:43:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24346.330638 secs 2023-03-09T21:43:56.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24346.330638 secs 2023-03-09T21:43:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:57 smithi122 ceph-mon[93869]: pgmap v13495: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 846 KiB/s rd, 620 KiB/s wr, 14 op/s 2023-03-09T21:43:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:43:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:43:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:57 smithi181 ceph-mon[94189]: pgmap v13495: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 846 KiB/s rd, 620 KiB/s wr, 14 op/s 2023-03-09T21:43:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:43:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:43:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:57 smithi038 ceph-mon[92422]: pgmap v13495: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 846 KiB/s rd, 620 KiB/s wr, 14 op/s 2023-03-09T21:43:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:43:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:43:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24810.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:43:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:59 smithi122 ceph-mon[93869]: pgmap v13496: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 666 KiB/s rd, 611 KiB/s wr, 12 op/s 2023-03-09T21:43:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:43:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:43:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:43:59.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:59 smithi181 ceph-mon[94189]: pgmap v13496: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 666 KiB/s rd, 611 KiB/s wr, 12 op/s 2023-03-09T21:43:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:43:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:43:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:43:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:59 smithi038 ceph-mon[92422]: pgmap v13496: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 666 KiB/s rd, 611 KiB/s wr, 12 op/s 2023-03-09T21:43:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:43:59.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:43:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:44:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24351.330702 secs 2023-03-09T21:44:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:01 smithi122 ceph-mon[93869]: pgmap v13497: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 666 KiB/s rd, 611 KiB/s wr, 12 op/s 2023-03-09T21:44:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:44:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:44:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:01 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:44:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24351.330702 secs 2023-03-09T21:44:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:01 smithi181 ceph-mon[94189]: pgmap v13497: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 666 KiB/s rd, 611 KiB/s wr, 12 op/s 2023-03-09T21:44:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:44:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:44:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:01 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:44:01.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24351.330702 secs 2023-03-09T21:44:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:01 smithi038 ceph-mon[92422]: pgmap v13497: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 666 KiB/s rd, 611 KiB/s wr, 12 op/s 2023-03-09T21:44:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:44:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:44:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:01 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:44:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:02 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:44:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:02 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:44:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:02 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:44:02.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:02 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:44:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:02 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:44:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:02 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:44:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:02 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:44:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:02 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:44:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:02 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:44:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:03 smithi122 ceph-mon[93869]: pgmap v13498: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 612 KiB/s rd, 590 KiB/s wr, 17 op/s 2023-03-09T21:44:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24353.680505 secs 2023-03-09T21:44:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:03 smithi181 ceph-mon[94189]: pgmap v13498: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 612 KiB/s rd, 590 KiB/s wr, 17 op/s 2023-03-09T21:44:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24353.680505 secs 2023-03-09T21:44:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:03 smithi038 ceph-mon[92422]: pgmap v13498: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 612 KiB/s rd, 590 KiB/s wr, 17 op/s 2023-03-09T21:44:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24353.680505 secs 2023-03-09T21:44:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24815.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:44:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:05 smithi038 ceph-mon[92422]: pgmap v13499: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 487 KiB/s wr, 26 op/s 2023-03-09T21:44:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:05 smithi038 ceph-mon[92422]: osdmap e1057: 12 total, 12 up, 12 in 2023-03-09T21:44:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:05 smithi122 ceph-mon[93869]: pgmap v13499: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 487 KiB/s wr, 26 op/s 2023-03-09T21:44:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:05 smithi122 ceph-mon[93869]: osdmap e1057: 12 total, 12 up, 12 in 2023-03-09T21:44:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:05 smithi181 ceph-mon[94189]: pgmap v13499: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 487 KiB/s wr, 26 op/s 2023-03-09T21:44:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:05 smithi181 ceph-mon[94189]: osdmap e1057: 12 total, 12 up, 12 in 2023-03-09T21:44:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24356.330764 secs 2023-03-09T21:44:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:06 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:44:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:44:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24356.330764 secs 2023-03-09T21:44:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:06 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:44:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:44:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24356.330764 secs 2023-03-09T21:44:06.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:06 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:44:06.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:44:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:07 smithi038 ceph-mon[92422]: pgmap v13501: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 443 KiB/s wr, 28 op/s 2023-03-09T21:44:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:07 smithi122 ceph-mon[93869]: pgmap v13501: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 443 KiB/s wr, 28 op/s 2023-03-09T21:44:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:07 smithi181 ceph-mon[94189]: pgmap v13501: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 443 KiB/s wr, 28 op/s 2023-03-09T21:44:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24820.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:44:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24358.680625 secs 2023-03-09T21:44:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24358.680625 secs 2023-03-09T21:44:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24358.680625 secs 2023-03-09T21:44:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:09 smithi122 ceph-mon[93869]: pgmap v13502: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 443 KiB/s wr, 28 op/s 2023-03-09T21:44:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:09 smithi181 ceph-mon[94189]: pgmap v13502: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 443 KiB/s wr, 28 op/s 2023-03-09T21:44:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:09 smithi038 ceph-mon[92422]: pgmap v13502: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 443 KiB/s wr, 28 op/s 2023-03-09T21:44:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:10 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:44:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:10 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:44:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:10 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:44:11.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24361.330822 secs 2023-03-09T21:44:11.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:11 smithi122 ceph-mon[93869]: pgmap v13503: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 443 KiB/s wr, 27 op/s 2023-03-09T21:44:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24361.330822 secs 2023-03-09T21:44:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:11 smithi181 ceph-mon[94189]: pgmap v13503: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 443 KiB/s wr, 27 op/s 2023-03-09T21:44:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24361.330822 secs 2023-03-09T21:44:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:11 smithi038 ceph-mon[92422]: pgmap v13503: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 443 KiB/s wr, 27 op/s 2023-03-09T21:44:13.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24825.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:44:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:13 smithi122 ceph-mon[93869]: pgmap v13504: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 336 KiB/s wr, 23 op/s 2023-03-09T21:44:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24363.680715 secs 2023-03-09T21:44:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:13 smithi181 ceph-mon[94189]: pgmap v13504: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 336 KiB/s wr, 23 op/s 2023-03-09T21:44:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24363.680715 secs 2023-03-09T21:44:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:13 smithi038 ceph-mon[92422]: pgmap v13504: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 336 KiB/s wr, 23 op/s 2023-03-09T21:44:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24363.680715 secs 2023-03-09T21:44:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:15 smithi122 ceph-mon[93869]: pgmap v13505: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 382 KiB/s wr, 9 op/s 2023-03-09T21:44:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:15 smithi122 ceph-mon[93869]: osdmap e1058: 12 total, 12 up, 12 in 2023-03-09T21:44:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:15 smithi181 ceph-mon[94189]: pgmap v13505: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 382 KiB/s wr, 9 op/s 2023-03-09T21:44:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:15 smithi181 ceph-mon[94189]: osdmap e1058: 12 total, 12 up, 12 in 2023-03-09T21:44:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:15 smithi038 ceph-mon[92422]: pgmap v13505: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 382 KiB/s wr, 9 op/s 2023-03-09T21:44:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:15 smithi038 ceph-mon[92422]: osdmap e1058: 12 total, 12 up, 12 in 2023-03-09T21:44:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24366.330883 secs 2023-03-09T21:44:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24366.330883 secs 2023-03-09T21:44:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24366.330883 secs 2023-03-09T21:44:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:17 smithi122 ceph-mon[93869]: pgmap v13507: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 104 KiB/s wr, 8 op/s 2023-03-09T21:44:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:17 smithi181 ceph-mon[94189]: pgmap v13507: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 104 KiB/s wr, 8 op/s 2023-03-09T21:44:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:17 smithi038 ceph-mon[92422]: pgmap v13507: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 104 KiB/s wr, 8 op/s 2023-03-09T21:44:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24830.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:44:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24368.680835 secs 2023-03-09T21:44:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24368.680835 secs 2023-03-09T21:44:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24368.680835 secs 2023-03-09T21:44:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:19 smithi122 ceph-mon[93869]: pgmap v13508: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 104 KiB/s wr, 8 op/s 2023-03-09T21:44:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:19 smithi181 ceph-mon[94189]: pgmap v13508: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 104 KiB/s wr, 8 op/s 2023-03-09T21:44:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:19 smithi038 ceph-mon[92422]: pgmap v13508: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 104 KiB/s wr, 8 op/s 2023-03-09T21:44:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24371.330945 secs 2023-03-09T21:44:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:21 smithi122 ceph-mon[93869]: pgmap v13509: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 104 KiB/s wr, 8 op/s 2023-03-09T21:44:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24371.330945 secs 2023-03-09T21:44:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:21 smithi181 ceph-mon[94189]: pgmap v13509: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 104 KiB/s wr, 8 op/s 2023-03-09T21:44:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24371.330945 secs 2023-03-09T21:44:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:21 smithi038 ceph-mon[92422]: pgmap v13509: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 104 KiB/s wr, 8 op/s 2023-03-09T21:44:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24835.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:44:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:23 smithi122 ceph-mon[93869]: pgmap v13510: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 118 KiB/s wr, 9 op/s 2023-03-09T21:44:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24373.680956 secs 2023-03-09T21:44:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:23 smithi181 ceph-mon[94189]: pgmap v13510: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 118 KiB/s wr, 9 op/s 2023-03-09T21:44:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24373.680956 secs 2023-03-09T21:44:23.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:23 smithi038 ceph-mon[92422]: pgmap v13510: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 118 KiB/s wr, 9 op/s 2023-03-09T21:44:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24373.680956 secs 2023-03-09T21:44:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:25 smithi122 ceph-mon[93869]: pgmap v13511: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 46 KiB/s wr, 8 op/s 2023-03-09T21:44:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:25 smithi181 ceph-mon[94189]: pgmap v13511: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 46 KiB/s wr, 8 op/s 2023-03-09T21:44:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:25 smithi038 ceph-mon[92422]: pgmap v13511: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 46 KiB/s wr, 8 op/s 2023-03-09T21:44:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24376.331002 secs 2023-03-09T21:44:26.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24376.331002 secs 2023-03-09T21:44:26.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24376.331002 secs 2023-03-09T21:44:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:27 smithi122 ceph-mon[93869]: pgmap v13512: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 10 op/s 2023-03-09T21:44:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:27 smithi181 ceph-mon[94189]: pgmap v13512: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 10 op/s 2023-03-09T21:44:27.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:27 smithi038 ceph-mon[92422]: pgmap v13512: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 10 op/s 2023-03-09T21:44:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24840.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:44:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24378.681079 secs 2023-03-09T21:44:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24378.681079 secs 2023-03-09T21:44:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24378.681079 secs 2023-03-09T21:44:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:29 smithi122 ceph-mon[93869]: pgmap v13513: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T21:44:29.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:29 smithi181 ceph-mon[94189]: pgmap v13513: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T21:44:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:29 smithi038 ceph-mon[92422]: pgmap v13513: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T21:44:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24381.331064 secs 2023-03-09T21:44:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:31 smithi122 ceph-mon[93869]: pgmap v13514: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 118 KiB/s wr, 7 op/s 2023-03-09T21:44:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:31 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:44:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:31 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:44:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24381.331064 secs 2023-03-09T21:44:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:31 smithi181 ceph-mon[94189]: pgmap v13514: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 118 KiB/s wr, 7 op/s 2023-03-09T21:44:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:31 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:44:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:31 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:44:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24381.331064 secs 2023-03-09T21:44:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:31 smithi038 ceph-mon[92422]: pgmap v13514: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 118 KiB/s wr, 7 op/s 2023-03-09T21:44:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:31 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:44:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:31 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:44:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24845.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:44:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:33 smithi122 ceph-mon[93869]: pgmap v13515: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 119 KiB/s wr, 10 op/s 2023-03-09T21:44:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24383.681163 secs 2023-03-09T21:44:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:33 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:44:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:33 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:44:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:33 smithi181 ceph-mon[94189]: pgmap v13515: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 119 KiB/s wr, 10 op/s 2023-03-09T21:44:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24383.681163 secs 2023-03-09T21:44:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:33 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:44:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:33 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:44:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:33 smithi038 ceph-mon[92422]: pgmap v13515: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 119 KiB/s wr, 10 op/s 2023-03-09T21:44:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24383.681163 secs 2023-03-09T21:44:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:33 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:44:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:33 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:44:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:35 smithi122 ceph-mon[93869]: pgmap v13516: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 99 KiB/s wr, 7 op/s 2023-03-09T21:44:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:35 smithi181 ceph-mon[94189]: pgmap v13516: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 99 KiB/s wr, 7 op/s 2023-03-09T21:44:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:35 smithi038 ceph-mon[92422]: pgmap v13516: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 99 KiB/s wr, 7 op/s 2023-03-09T21:44:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24386.331122 secs 2023-03-09T21:44:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24386.331122 secs 2023-03-09T21:44:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24386.331122 secs 2023-03-09T21:44:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:37 smithi122 ceph-mon[93869]: pgmap v13517: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 100 KiB/s wr, 9 op/s 2023-03-09T21:44:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:37 smithi181 ceph-mon[94189]: pgmap v13517: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 100 KiB/s wr, 9 op/s 2023-03-09T21:44:37.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:37 smithi038 ceph-mon[92422]: pgmap v13517: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 100 KiB/s wr, 9 op/s 2023-03-09T21:44:38.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24850.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:44:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24388.681301 secs 2023-03-09T21:44:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:38 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:44:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:38 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:44:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:38 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:44:38.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:38 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:44:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24388.681301 secs 2023-03-09T21:44:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:38 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:44:38.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:38 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:44:38.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:38 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:44:38.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:38 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:44:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24388.681301 secs 2023-03-09T21:44:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:38 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:44:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:38 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:44:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:38 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:44:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:38 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:44:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:39 smithi122 ceph-mon[93869]: pgmap v13518: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 99 KiB/s wr, 7 op/s 2023-03-09T21:44:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:39 smithi181 ceph-mon[94189]: pgmap v13518: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 99 KiB/s wr, 7 op/s 2023-03-09T21:44:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:39 smithi038 ceph-mon[92422]: pgmap v13518: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 99 KiB/s wr, 7 op/s 2023-03-09T21:44:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24391.331183 secs 2023-03-09T21:44:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:41 smithi122 ceph-mon[93869]: pgmap v13519: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 198 KiB/s wr, 7 op/s 2023-03-09T21:44:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24391.331183 secs 2023-03-09T21:44:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:41 smithi181 ceph-mon[94189]: pgmap v13519: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 198 KiB/s wr, 7 op/s 2023-03-09T21:44:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24391.331183 secs 2023-03-09T21:44:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:41 smithi038 ceph-mon[92422]: pgmap v13519: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 198 KiB/s wr, 7 op/s 2023-03-09T21:44:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24855.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:44:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:43 smithi122 ceph-mon[93869]: pgmap v13520: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 99 KiB/s wr, 8 op/s 2023-03-09T21:44:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:43 smithi181 ceph-mon[94189]: pgmap v13520: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 99 KiB/s wr, 8 op/s 2023-03-09T21:44:43.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:43 smithi038 ceph-mon[92422]: pgmap v13520: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 99 KiB/s wr, 8 op/s 2023-03-09T21:44:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:45 smithi122 ceph-mon[93869]: pgmap v13521: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 184 KiB/s wr, 7 op/s 2023-03-09T21:44:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:45 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:44:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:45 smithi181 ceph-mon[94189]: pgmap v13521: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 184 KiB/s wr, 7 op/s 2023-03-09T21:44:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:45 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:44:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:45 smithi038 ceph-mon[92422]: pgmap v13521: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 184 KiB/s wr, 7 op/s 2023-03-09T21:44:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:45 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:44:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24396.331243 secs 2023-03-09T21:44:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:46 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:44:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:46 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:44:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:46 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:44:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24396.331243 secs 2023-03-09T21:44:46.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:46 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:44:46.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:46 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:44:46.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:46 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:44:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24396.331243 secs 2023-03-09T21:44:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:46 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:44:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:46 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:44:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:46 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:44:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:47 smithi122 ceph-mon[93869]: pgmap v13522: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 203 KiB/s wr, 8 op/s 2023-03-09T21:44:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:47 smithi181 ceph-mon[94189]: pgmap v13522: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 203 KiB/s wr, 8 op/s 2023-03-09T21:44:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:47 smithi038 ceph-mon[92422]: pgmap v13522: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 203 KiB/s wr, 8 op/s 2023-03-09T21:44:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24860.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:44:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:49 smithi122 ceph-mon[93869]: pgmap v13523: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 334 KiB/s wr, 7 op/s 2023-03-09T21:44:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:49 smithi181 ceph-mon[94189]: pgmap v13523: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 334 KiB/s wr, 7 op/s 2023-03-09T21:44:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:49 smithi038 ceph-mon[92422]: pgmap v13523: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 334 KiB/s wr, 7 op/s 2023-03-09T21:44:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24401.331304 secs 2023-03-09T21:44:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:51 smithi122 ceph-mon[93869]: pgmap v13524: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 626 KiB/s rd, 334 KiB/s wr, 7 op/s 2023-03-09T21:44:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:44:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:44:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24401.331304 secs 2023-03-09T21:44:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:51 smithi181 ceph-mon[94189]: pgmap v13524: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 626 KiB/s rd, 334 KiB/s wr, 7 op/s 2023-03-09T21:44:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:44:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:44:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24401.331304 secs 2023-03-09T21:44:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:51 smithi038 ceph-mon[92422]: pgmap v13524: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 626 KiB/s rd, 334 KiB/s wr, 7 op/s 2023-03-09T21:44:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:44:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:44:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:44:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:44:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:44:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:44:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:44:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:44:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24865.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:44:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:53 smithi122 ceph-mon[93869]: pgmap v13525: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 550 KiB/s rd, 256 KiB/s wr, 11 op/s 2023-03-09T21:44:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:53 smithi181 ceph-mon[94189]: pgmap v13525: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 550 KiB/s rd, 256 KiB/s wr, 11 op/s 2023-03-09T21:44:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:53 smithi038 ceph-mon[92422]: pgmap v13525: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 550 KiB/s rd, 256 KiB/s wr, 11 op/s 2023-03-09T21:44:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:55 smithi122 ceph-mon[93869]: pgmap v13526: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 466 KiB/s wr, 11 op/s 2023-03-09T21:44:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:44:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:44:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:55 smithi181 ceph-mon[94189]: pgmap v13526: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 466 KiB/s wr, 11 op/s 2023-03-09T21:44:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:44:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:44:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:55 smithi038 ceph-mon[92422]: pgmap v13526: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 466 KiB/s wr, 11 op/s 2023-03-09T21:44:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:44:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:44:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24406.331363 secs 2023-03-09T21:44:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:44:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:44:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24406.331363 secs 2023-03-09T21:44:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:44:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:44:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24406.331363 secs 2023-03-09T21:44:57.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:44:57.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:44:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:57 smithi038 ceph-mon[92422]: pgmap v13527: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 652 KiB/s rd, 638 KiB/s wr, 13 op/s 2023-03-09T21:44:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:57 smithi122 ceph-mon[93869]: pgmap v13527: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 652 KiB/s rd, 638 KiB/s wr, 13 op/s 2023-03-09T21:44:58.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:57 smithi181 ceph-mon[94189]: pgmap v13527: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 652 KiB/s rd, 638 KiB/s wr, 13 op/s 2023-03-09T21:44:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24870.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:44:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:44:59 smithi038 ceph-mon[92422]: pgmap v13528: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 692 KiB/s rd, 620 KiB/s wr, 12 op/s 2023-03-09T21:45:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:44:59 smithi122 ceph-mon[93869]: pgmap v13528: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 692 KiB/s rd, 620 KiB/s wr, 12 op/s 2023-03-09T21:45:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:44:59 smithi181 ceph-mon[94189]: pgmap v13528: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 692 KiB/s rd, 620 KiB/s wr, 12 op/s 2023-03-09T21:45:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24411.331446 secs 2023-03-09T21:45:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:01 smithi038 ceph-mon[92422]: pgmap v13529: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 561 KiB/s rd, 488 KiB/s wr, 11 op/s 2023-03-09T21:45:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24411.331446 secs 2023-03-09T21:45:02.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:01 smithi122 ceph-mon[93869]: pgmap v13529: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 561 KiB/s rd, 488 KiB/s wr, 11 op/s 2023-03-09T21:45:02.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24411.331446 secs 2023-03-09T21:45:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:01 smithi181 ceph-mon[94189]: pgmap v13529: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 561 KiB/s rd, 488 KiB/s wr, 11 op/s 2023-03-09T21:45:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24875.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:45:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:03 smithi038 ceph-mon[92422]: pgmap v13530: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 704 KiB/s rd, 612 KiB/s wr, 15 op/s 2023-03-09T21:45:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:03 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 24413.681925 secs 2023-03-09T21:45:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:03 smithi122 ceph-mon[93869]: pgmap v13530: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 704 KiB/s rd, 612 KiB/s wr, 15 op/s 2023-03-09T21:45:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:03 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 24413.681925 secs 2023-03-09T21:45:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:03 smithi181 ceph-mon[94189]: pgmap v13530: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 704 KiB/s rd, 612 KiB/s wr, 15 op/s 2023-03-09T21:45:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:03 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 24413.681925 secs 2023-03-09T21:45:05.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:04 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:45:05.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:04 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:45:05.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:04 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:45:05.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:04 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:45:05.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:04 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:45:05.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:04 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:45:05.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:04 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:45:05.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:04 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:45:05.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:04 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:45:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:05 smithi122 ceph-mon[93869]: pgmap v13531: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 688 KiB/s rd, 614 KiB/s wr, 23 op/s 2023-03-09T21:45:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24416.331505 secs 2023-03-09T21:45:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:05 smithi181 ceph-mon[94189]: pgmap v13531: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 688 KiB/s rd, 614 KiB/s wr, 23 op/s 2023-03-09T21:45:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24416.331505 secs 2023-03-09T21:45:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:05 smithi038 ceph-mon[92422]: pgmap v13531: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 688 KiB/s rd, 614 KiB/s wr, 23 op/s 2023-03-09T21:45:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24416.331505 secs 2023-03-09T21:45:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:07 smithi122 ceph-mon[93869]: pgmap v13532: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 958 KiB/s rd, 703 KiB/s wr, 28 op/s 2023-03-09T21:45:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:45:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:45:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:07 smithi181 ceph-mon[94189]: pgmap v13532: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 958 KiB/s rd, 703 KiB/s wr, 28 op/s 2023-03-09T21:45:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:45:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:45:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:07 smithi038 ceph-mon[92422]: pgmap v13532: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 958 KiB/s rd, 703 KiB/s wr, 28 op/s 2023-03-09T21:45:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:45:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:45:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24880.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:45:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24418.682043 secs 2023-03-09T21:45:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:45:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:45:09.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24418.682043 secs 2023-03-09T21:45:09.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:45:09.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:45:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24418.682043 secs 2023-03-09T21:45:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:45:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:45:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:09 smithi122 ceph-mon[93869]: pgmap v13533: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 700 KiB/s rd, 446 KiB/s wr, 25 op/s 2023-03-09T21:45:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:45:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:45:10.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:09 smithi181 ceph-mon[94189]: pgmap v13533: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 700 KiB/s rd, 446 KiB/s wr, 25 op/s 2023-03-09T21:45:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:45:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:45:10.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:09 smithi038 ceph-mon[92422]: pgmap v13533: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 700 KiB/s rd, 446 KiB/s wr, 25 op/s 2023-03-09T21:45:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:45:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:45:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:10 smithi122 ceph-mon[93869]: osdmap e1059: 12 total, 12 up, 12 in 2023-03-09T21:45:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24421.331581 secs 2023-03-09T21:45:11.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:10 smithi181 ceph-mon[94189]: osdmap e1059: 12 total, 12 up, 12 in 2023-03-09T21:45:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24421.331581 secs 2023-03-09T21:45:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:10 smithi038 ceph-mon[92422]: osdmap e1059: 12 total, 12 up, 12 in 2023-03-09T21:45:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24421.331581 secs 2023-03-09T21:45:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:11 smithi122 ceph-mon[93869]: pgmap v13535: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 767 KiB/s rd, 535 KiB/s wr, 29 op/s 2023-03-09T21:45:12.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:11 smithi181 ceph-mon[94189]: pgmap v13535: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 767 KiB/s rd, 535 KiB/s wr, 29 op/s 2023-03-09T21:45:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:11 smithi038 ceph-mon[92422]: pgmap v13535: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 767 KiB/s rd, 535 KiB/s wr, 29 op/s 2023-03-09T21:45:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24423.682195 secs 2023-03-09T21:45:13.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24423.682195 secs 2023-03-09T21:45:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24423.682195 secs 2023-03-09T21:45:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24885.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:45:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:13 smithi122 ceph-mon[93869]: pgmap v13536: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 631 KiB/s rd, 491 KiB/s wr, 28 op/s 2023-03-09T21:45:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:13 smithi181 ceph-mon[94189]: pgmap v13536: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 631 KiB/s rd, 491 KiB/s wr, 28 op/s 2023-03-09T21:45:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:13 smithi038 ceph-mon[92422]: pgmap v13536: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 631 KiB/s rd, 491 KiB/s wr, 28 op/s 2023-03-09T21:45:15.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:14 smithi122 ceph-mon[93869]: pgmap v13537: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 622 KiB/s rd, 465 KiB/s wr, 14 op/s 2023-03-09T21:45:15.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:14 smithi181 ceph-mon[94189]: pgmap v13537: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 622 KiB/s rd, 465 KiB/s wr, 14 op/s 2023-03-09T21:45:15.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:14 smithi038 ceph-mon[92422]: pgmap v13537: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 622 KiB/s rd, 465 KiB/s wr, 14 op/s 2023-03-09T21:45:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24426.331640 secs 2023-03-09T21:45:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:16 smithi122 ceph-mon[93869]: osdmap e1060: 12 total, 12 up, 12 in 2023-03-09T21:45:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24426.331640 secs 2023-03-09T21:45:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:16 smithi181 ceph-mon[94189]: osdmap e1060: 12 total, 12 up, 12 in 2023-03-09T21:45:16.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24426.331640 secs 2023-03-09T21:45:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:16 smithi038 ceph-mon[92422]: osdmap e1060: 12 total, 12 up, 12 in 2023-03-09T21:45:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:17 smithi122 ceph-mon[93869]: pgmap v13539: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 131 KiB/s wr, 10 op/s 2023-03-09T21:45:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:17 smithi181 ceph-mon[94189]: pgmap v13539: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 131 KiB/s wr, 10 op/s 2023-03-09T21:45:17.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:17 smithi038 ceph-mon[92422]: pgmap v13539: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 131 KiB/s wr, 10 op/s 2023-03-09T21:45:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24428.682327 secs 2023-03-09T21:45:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24428.682327 secs 2023-03-09T21:45:18.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24428.682327 secs 2023-03-09T21:45:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24890.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:45:19.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:19 smithi122 ceph-mon[93869]: pgmap v13540: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 127 KiB/s wr, 10 op/s 2023-03-09T21:45:19.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:19 smithi181 ceph-mon[94189]: pgmap v13540: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 127 KiB/s wr, 10 op/s 2023-03-09T21:45:19.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:19 smithi038 ceph-mon[92422]: pgmap v13540: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 127 KiB/s wr, 10 op/s 2023-03-09T21:45:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24431.331718 secs 2023-03-09T21:45:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:21 smithi122 ceph-mon[93869]: pgmap v13541: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 105 KiB/s wr, 8 op/s 2023-03-09T21:45:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24431.331718 secs 2023-03-09T21:45:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:21 smithi181 ceph-mon[94189]: pgmap v13541: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 105 KiB/s wr, 8 op/s 2023-03-09T21:45:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24431.331718 secs 2023-03-09T21:45:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:21 smithi038 ceph-mon[92422]: pgmap v13541: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 105 KiB/s wr, 8 op/s 2023-03-09T21:45:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:23 smithi122 ceph-mon[93869]: pgmap v13542: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 7 op/s 2023-03-09T21:45:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24433.682417 secs 2023-03-09T21:45:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:23 smithi181 ceph-mon[94189]: pgmap v13542: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 7 op/s 2023-03-09T21:45:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24433.682417 secs 2023-03-09T21:45:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:23 smithi038 ceph-mon[92422]: pgmap v13542: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 7 op/s 2023-03-09T21:45:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24433.682417 secs 2023-03-09T21:45:23.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24895.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:45:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:25 smithi122 ceph-mon[93869]: pgmap v13543: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2023-03-09T21:45:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:25 smithi181 ceph-mon[94189]: pgmap v13543: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2023-03-09T21:45:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:25 smithi038 ceph-mon[92422]: pgmap v13543: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 307 B/s wr, 6 op/s 2023-03-09T21:45:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24436.331789 secs 2023-03-09T21:45:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24436.331789 secs 2023-03-09T21:45:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24436.331789 secs 2023-03-09T21:45:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:27 smithi122 ceph-mon[93869]: pgmap v13544: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 95 B/s wr, 8 op/s 2023-03-09T21:45:27.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:27 smithi181 ceph-mon[94189]: pgmap v13544: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 95 B/s wr, 8 op/s 2023-03-09T21:45:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:27 smithi038 ceph-mon[92422]: pgmap v13544: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 95 B/s wr, 8 op/s 2023-03-09T21:45:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24438.682530 secs 2023-03-09T21:45:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24438.682530 secs 2023-03-09T21:45:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24900.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:45:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24438.682530 secs 2023-03-09T21:45:29.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:29 smithi181 ceph-mon[94189]: pgmap v13545: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T21:45:29.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:29 smithi038 ceph-mon[92422]: pgmap v13545: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T21:45:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:29 smithi122 ceph-mon[93869]: pgmap v13545: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T21:45:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24441.331851 secs 2023-03-09T21:45:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:31 smithi038 ceph-mon[92422]: pgmap v13546: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T21:45:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24441.331851 secs 2023-03-09T21:45:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:31 smithi122 ceph-mon[93869]: pgmap v13546: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T21:45:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24441.331851 secs 2023-03-09T21:45:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:31 smithi181 ceph-mon[94189]: pgmap v13546: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T21:45:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:33 smithi038 ceph-mon[92422]: pgmap v13547: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 426 B/s wr, 7 op/s 2023-03-09T21:45:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24443.682666 secs 2023-03-09T21:45:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24905.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:45:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:33 smithi122 ceph-mon[93869]: pgmap v13547: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 426 B/s wr, 7 op/s 2023-03-09T21:45:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24443.682666 secs 2023-03-09T21:45:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:33 smithi181 ceph-mon[94189]: pgmap v13547: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 426 B/s wr, 7 op/s 2023-03-09T21:45:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24443.682666 secs 2023-03-09T21:45:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:35 smithi038 ceph-mon[92422]: pgmap v13548: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2023-03-09T21:45:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:35 smithi122 ceph-mon[93869]: pgmap v13548: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2023-03-09T21:45:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:35 smithi181 ceph-mon[94189]: pgmap v13548: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2023-03-09T21:45:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24446.331910 secs 2023-03-09T21:45:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24446.331910 secs 2023-03-09T21:45:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24446.331910 secs 2023-03-09T21:45:37.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:37 smithi038 ceph-mon[92422]: pgmap v13549: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 99 KiB/s wr, 9 op/s 2023-03-09T21:45:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:37 smithi122 ceph-mon[93869]: pgmap v13549: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 99 KiB/s wr, 9 op/s 2023-03-09T21:45:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:37 smithi181 ceph-mon[94189]: pgmap v13549: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 99 KiB/s wr, 9 op/s 2023-03-09T21:45:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24910.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:45:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24448.682804 secs 2023-03-09T21:45:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24448.682804 secs 2023-03-09T21:45:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24448.682804 secs 2023-03-09T21:45:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:39 smithi122 ceph-mon[93869]: pgmap v13550: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 99 KiB/s wr, 6 op/s 2023-03-09T21:45:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:45:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:45:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:45:39.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:45:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:39 smithi181 ceph-mon[94189]: pgmap v13550: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 99 KiB/s wr, 6 op/s 2023-03-09T21:45:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:45:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:45:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:45:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:45:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:39 smithi038 ceph-mon[92422]: pgmap v13550: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 99 KiB/s wr, 6 op/s 2023-03-09T21:45:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:45:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:45:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:45:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:45:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24451.331970 secs 2023-03-09T21:45:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:41 smithi122 ceph-mon[93869]: pgmap v13551: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 100 KiB/s wr, 6 op/s 2023-03-09T21:45:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:45:41.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:45:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24451.331970 secs 2023-03-09T21:45:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:41 smithi181 ceph-mon[94189]: pgmap v13551: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 100 KiB/s wr, 6 op/s 2023-03-09T21:45:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:45:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:45:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24451.331970 secs 2023-03-09T21:45:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:41 smithi038 ceph-mon[92422]: pgmap v13551: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 100 KiB/s wr, 6 op/s 2023-03-09T21:45:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:45:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:45:43.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24915.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:45:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:43 smithi122 ceph-mon[93869]: pgmap v13552: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 380 KiB/s wr, 11 op/s 2023-03-09T21:45:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24453.682918 secs 2023-03-09T21:45:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:43 smithi181 ceph-mon[94189]: pgmap v13552: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 380 KiB/s wr, 11 op/s 2023-03-09T21:45:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24453.682918 secs 2023-03-09T21:45:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:43 smithi038 ceph-mon[92422]: pgmap v13552: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 380 KiB/s wr, 11 op/s 2023-03-09T21:45:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24453.682918 secs 2023-03-09T21:45:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:45 smithi122 ceph-mon[93869]: pgmap v13553: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 395 KiB/s rd, 380 KiB/s wr, 9 op/s 2023-03-09T21:45:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:45 smithi181 ceph-mon[94189]: pgmap v13553: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 395 KiB/s rd, 380 KiB/s wr, 9 op/s 2023-03-09T21:45:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:45 smithi038 ceph-mon[92422]: pgmap v13553: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 395 KiB/s rd, 380 KiB/s wr, 9 op/s 2023-03-09T21:45:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24456.331986 secs 2023-03-09T21:45:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24456.331986 secs 2023-03-09T21:45:47.182 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24456.331986 secs 2023-03-09T21:45:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:47 smithi122 ceph-mon[93869]: pgmap v13554: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 475 KiB/s wr, 13 op/s 2023-03-09T21:45:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:45:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:45:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:47 smithi181 ceph-mon[94189]: pgmap v13554: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 475 KiB/s wr, 13 op/s 2023-03-09T21:45:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:45:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:45:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:47 smithi038 ceph-mon[92422]: pgmap v13554: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 475 KiB/s wr, 13 op/s 2023-03-09T21:45:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:45:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:45:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24920.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:45:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24458.683034 secs 2023-03-09T21:45:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24458.683034 secs 2023-03-09T21:45:48.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24458.683034 secs 2023-03-09T21:45:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:49 smithi122 ceph-mon[93869]: pgmap v13555: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 395 KiB/s rd, 376 KiB/s wr, 10 op/s 2023-03-09T21:45:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:49 smithi181 ceph-mon[94189]: pgmap v13555: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 395 KiB/s rd, 376 KiB/s wr, 10 op/s 2023-03-09T21:45:49.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:49 smithi038 ceph-mon[92422]: pgmap v13555: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 395 KiB/s rd, 376 KiB/s wr, 10 op/s 2023-03-09T21:45:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24461.332002 secs 2023-03-09T21:45:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:51 smithi122 ceph-mon[93869]: pgmap v13556: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 395 KiB/s rd, 376 KiB/s wr, 10 op/s 2023-03-09T21:45:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:45:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:45:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24461.332002 secs 2023-03-09T21:45:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:51 smithi181 ceph-mon[94189]: pgmap v13556: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 395 KiB/s rd, 376 KiB/s wr, 10 op/s 2023-03-09T21:45:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:45:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:45:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24461.332002 secs 2023-03-09T21:45:52.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:51 smithi038 ceph-mon[92422]: pgmap v13556: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 395 KiB/s rd, 376 KiB/s wr, 10 op/s 2023-03-09T21:45:52.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:45:52.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:45:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24925.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:45:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:53 smithi122 ceph-mon[93869]: pgmap v13557: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 742 KiB/s rd, 660 KiB/s wr, 15 op/s 2023-03-09T21:45:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24463.683199 secs 2023-03-09T21:45:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:45:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:45:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:53 smithi181 ceph-mon[94189]: pgmap v13557: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 742 KiB/s rd, 660 KiB/s wr, 15 op/s 2023-03-09T21:45:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24463.683199 secs 2023-03-09T21:45:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:45:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:45:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:53 smithi038 ceph-mon[92422]: pgmap v13557: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 742 KiB/s rd, 660 KiB/s wr, 15 op/s 2023-03-09T21:45:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24463.683199 secs 2023-03-09T21:45:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:45:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:45:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:55 smithi122 ceph-mon[93869]: pgmap v13558: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 379 KiB/s wr, 11 op/s 2023-03-09T21:45:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:55 smithi181 ceph-mon[94189]: pgmap v13558: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 379 KiB/s wr, 11 op/s 2023-03-09T21:45:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:55 smithi038 ceph-mon[92422]: pgmap v13558: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 379 KiB/s wr, 11 op/s 2023-03-09T21:45:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24466.332061 secs 2023-03-09T21:45:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24466.332061 secs 2023-03-09T21:45:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24466.332061 secs 2023-03-09T21:45:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:57 smithi122 ceph-mon[93869]: pgmap v13559: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 953 KiB/s rd, 636 KiB/s wr, 15 op/s 2023-03-09T21:45:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:57 smithi181 ceph-mon[94189]: pgmap v13559: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 953 KiB/s rd, 636 KiB/s wr, 15 op/s 2023-03-09T21:45:57.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:57 smithi038 ceph-mon[92422]: pgmap v13559: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 953 KiB/s rd, 636 KiB/s wr, 15 op/s 2023-03-09T21:45:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24930.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:45:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24468.683334 secs 2023-03-09T21:45:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24468.683334 secs 2023-03-09T21:45:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24468.683334 secs 2023-03-09T21:45:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:59 smithi122 ceph-mon[93869]: pgmap v13560: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 853 KiB/s rd, 542 KiB/s wr, 11 op/s 2023-03-09T21:45:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:45:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:45:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:45:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:59 smithi181 ceph-mon[94189]: pgmap v13560: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 853 KiB/s rd, 542 KiB/s wr, 11 op/s 2023-03-09T21:45:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:45:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:45:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:45:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:59 smithi038 ceph-mon[92422]: pgmap v13560: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 853 KiB/s rd, 542 KiB/s wr, 11 op/s 2023-03-09T21:45:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:45:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:45:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:46:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24471.332122 secs 2023-03-09T21:46:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:01 smithi122 ceph-mon[93869]: pgmap v13561: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 853 KiB/s rd, 542 KiB/s wr, 10 op/s 2023-03-09T21:46:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:46:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:46:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24471.332122 secs 2023-03-09T21:46:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:01 smithi181 ceph-mon[94189]: pgmap v13561: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 853 KiB/s rd, 542 KiB/s wr, 10 op/s 2023-03-09T21:46:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:46:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:46:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24471.332122 secs 2023-03-09T21:46:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:01 smithi038 ceph-mon[92422]: pgmap v13561: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 853 KiB/s rd, 542 KiB/s wr, 10 op/s 2023-03-09T21:46:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:46:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:46:03.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24935.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:46:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:03 smithi122 ceph-mon[93869]: pgmap v13562: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 857 KiB/s rd, 566 KiB/s wr, 16 op/s 2023-03-09T21:46:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24473.683432 secs 2023-03-09T21:46:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:03 smithi181 ceph-mon[94189]: pgmap v13562: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 857 KiB/s rd, 566 KiB/s wr, 16 op/s 2023-03-09T21:46:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24473.683432 secs 2023-03-09T21:46:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:03 smithi038 ceph-mon[92422]: pgmap v13562: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 857 KiB/s rd, 566 KiB/s wr, 16 op/s 2023-03-09T21:46:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24473.683432 secs 2023-03-09T21:46:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:05 smithi122 ceph-mon[93869]: pgmap v13563: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 543 KiB/s rd, 329 KiB/s wr, 24 op/s 2023-03-09T21:46:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:05 smithi122 ceph-mon[93869]: osdmap e1061: 12 total, 12 up, 12 in 2023-03-09T21:46:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:05 smithi181 ceph-mon[94189]: pgmap v13563: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 543 KiB/s rd, 329 KiB/s wr, 24 op/s 2023-03-09T21:46:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:05 smithi181 ceph-mon[94189]: osdmap e1061: 12 total, 12 up, 12 in 2023-03-09T21:46:05.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:05 smithi038 ceph-mon[92422]: pgmap v13563: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 543 KiB/s rd, 329 KiB/s wr, 24 op/s 2023-03-09T21:46:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:05 smithi038 ceph-mon[92422]: osdmap e1061: 12 total, 12 up, 12 in 2023-03-09T21:46:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24476.332186 secs 2023-03-09T21:46:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24476.332186 secs 2023-03-09T21:46:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24476.332186 secs 2023-03-09T21:46:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:07 smithi122 ceph-mon[93869]: pgmap v13565: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 231 KiB/s wr, 26 op/s 2023-03-09T21:46:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:07 smithi181 ceph-mon[94189]: pgmap v13565: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 231 KiB/s wr, 26 op/s 2023-03-09T21:46:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:07 smithi038 ceph-mon[92422]: pgmap v13565: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 231 KiB/s wr, 26 op/s 2023-03-09T21:46:08.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24940.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:46:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24478.683548 secs 2023-03-09T21:46:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24478.683548 secs 2023-03-09T21:46:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24478.683548 secs 2023-03-09T21:46:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:09 smithi122 ceph-mon[93869]: pgmap v13566: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 346 KiB/s wr, 27 op/s 2023-03-09T21:46:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:09 smithi181 ceph-mon[94189]: pgmap v13566: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 346 KiB/s wr, 27 op/s 2023-03-09T21:46:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:09 smithi038 ceph-mon[92422]: pgmap v13566: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 346 KiB/s wr, 27 op/s 2023-03-09T21:46:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24481.332248 secs 2023-03-09T21:46:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:11 smithi122 ceph-mon[93869]: pgmap v13567: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 346 KiB/s wr, 27 op/s 2023-03-09T21:46:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:11 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:46:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:11 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:46:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24481.332248 secs 2023-03-09T21:46:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:11 smithi181 ceph-mon[94189]: pgmap v13567: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 346 KiB/s wr, 27 op/s 2023-03-09T21:46:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:11 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:46:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:11 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:46:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24481.332248 secs 2023-03-09T21:46:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:11 smithi038 ceph-mon[92422]: pgmap v13567: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 346 KiB/s wr, 27 op/s 2023-03-09T21:46:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:11 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:46:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:11 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:46:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:46:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:46:12.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:46:12.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:46:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:46:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:46:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24945.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:46:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:13 smithi122 ceph-mon[93869]: pgmap v13568: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 319 KiB/s wr, 22 op/s 2023-03-09T21:46:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24483.683645 secs 2023-03-09T21:46:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:13 smithi181 ceph-mon[94189]: pgmap v13568: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 319 KiB/s wr, 22 op/s 2023-03-09T21:46:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24483.683645 secs 2023-03-09T21:46:13.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:13 smithi038 ceph-mon[92422]: pgmap v13568: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 319 KiB/s wr, 22 op/s 2023-03-09T21:46:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24483.683645 secs 2023-03-09T21:46:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:15 smithi122 ceph-mon[93869]: pgmap v13569: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 362 KiB/s rd, 360 KiB/s wr, 9 op/s 2023-03-09T21:46:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:15 smithi122 ceph-mon[93869]: osdmap e1062: 12 total, 12 up, 12 in 2023-03-09T21:46:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:15 smithi181 ceph-mon[94189]: pgmap v13569: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 362 KiB/s rd, 360 KiB/s wr, 9 op/s 2023-03-09T21:46:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:15 smithi181 ceph-mon[94189]: osdmap e1062: 12 total, 12 up, 12 in 2023-03-09T21:46:15.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:15 smithi038 ceph-mon[92422]: pgmap v13569: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 362 KiB/s rd, 360 KiB/s wr, 9 op/s 2023-03-09T21:46:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:15 smithi038 ceph-mon[92422]: osdmap e1062: 12 total, 12 up, 12 in 2023-03-09T21:46:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24486.332305 secs 2023-03-09T21:46:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24486.332305 secs 2023-03-09T21:46:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24486.332305 secs 2023-03-09T21:46:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:17 smithi122 ceph-mon[93869]: pgmap v13571: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 237 KiB/s wr, 8 op/s 2023-03-09T21:46:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:17 smithi181 ceph-mon[94189]: pgmap v13571: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 237 KiB/s wr, 8 op/s 2023-03-09T21:46:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:17 smithi038 ceph-mon[92422]: pgmap v13571: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 237 KiB/s wr, 8 op/s 2023-03-09T21:46:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24950.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:46:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24488.683765 secs 2023-03-09T21:46:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24488.683765 secs 2023-03-09T21:46:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24488.683765 secs 2023-03-09T21:46:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:19 smithi122 ceph-mon[93869]: pgmap v13572: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 289 KiB/s wr, 8 op/s 2023-03-09T21:46:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:19 smithi181 ceph-mon[94189]: pgmap v13572: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 289 KiB/s wr, 8 op/s 2023-03-09T21:46:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:19 smithi038 ceph-mon[92422]: pgmap v13572: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 289 KiB/s wr, 8 op/s 2023-03-09T21:46:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24491.332364 secs 2023-03-09T21:46:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:21 smithi122 ceph-mon[93869]: pgmap v13573: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 305 KiB/s wr, 8 op/s 2023-03-09T21:46:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24491.332364 secs 2023-03-09T21:46:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:21 smithi181 ceph-mon[94189]: pgmap v13573: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 305 KiB/s wr, 8 op/s 2023-03-09T21:46:21.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24491.332364 secs 2023-03-09T21:46:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:21 smithi038 ceph-mon[92422]: pgmap v13573: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 305 KiB/s wr, 8 op/s 2023-03-09T21:46:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24955.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:46:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:23 smithi122 ceph-mon[93869]: pgmap v13574: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 305 KiB/s wr, 9 op/s 2023-03-09T21:46:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24493.683885 secs 2023-03-09T21:46:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:23 smithi181 ceph-mon[94189]: pgmap v13574: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 305 KiB/s wr, 9 op/s 2023-03-09T21:46:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24493.683885 secs 2023-03-09T21:46:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:23 smithi038 ceph-mon[92422]: pgmap v13574: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 305 KiB/s wr, 9 op/s 2023-03-09T21:46:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24493.683885 secs 2023-03-09T21:46:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:25 smithi122 ceph-mon[93869]: pgmap v13575: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 208 KiB/s wr, 9 op/s 2023-03-09T21:46:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:25 smithi181 ceph-mon[94189]: pgmap v13575: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 208 KiB/s wr, 9 op/s 2023-03-09T21:46:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:25 smithi038 ceph-mon[92422]: pgmap v13575: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 208 KiB/s wr, 9 op/s 2023-03-09T21:46:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24496.332423 secs 2023-03-09T21:46:26.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24496.332423 secs 2023-03-09T21:46:26.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24496.332423 secs 2023-03-09T21:46:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:27 smithi122 ceph-mon[93869]: pgmap v13576: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 174 KiB/s wr, 10 op/s 2023-03-09T21:46:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:27 smithi181 ceph-mon[94189]: pgmap v13576: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 174 KiB/s wr, 10 op/s 2023-03-09T21:46:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:27 smithi038 ceph-mon[92422]: pgmap v13576: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 174 KiB/s wr, 10 op/s 2023-03-09T21:46:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24960.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:46:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24498.684004 secs 2023-03-09T21:46:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24498.684004 secs 2023-03-09T21:46:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24498.684004 secs 2023-03-09T21:46:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:29 smithi038 ceph-mon[92422]: pgmap v13577: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 154 KiB/s wr, 8 op/s 2023-03-09T21:46:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:29 smithi122 ceph-mon[93869]: pgmap v13577: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 154 KiB/s wr, 8 op/s 2023-03-09T21:46:30.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:29 smithi181 ceph-mon[94189]: pgmap v13577: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 154 KiB/s wr, 8 op/s 2023-03-09T21:46:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24501.332491 secs 2023-03-09T21:46:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:31 smithi038 ceph-mon[92422]: pgmap v13578: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 114 KiB/s wr, 7 op/s 2023-03-09T21:46:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:31 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:46:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:31 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:46:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24501.332491 secs 2023-03-09T21:46:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:31 smithi122 ceph-mon[93869]: pgmap v13578: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 114 KiB/s wr, 7 op/s 2023-03-09T21:46:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:31 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:46:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:31 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:46:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24501.332491 secs 2023-03-09T21:46:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:31 smithi181 ceph-mon[94189]: pgmap v13578: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 114 KiB/s wr, 7 op/s 2023-03-09T21:46:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:31 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:46:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:31 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:46:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:32 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:46:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:32 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:46:33.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:32 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:46:33.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:32 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:46:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:32 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:46:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:32 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:46:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24965.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:46:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:33 smithi122 ceph-mon[93869]: pgmap v13579: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 101 KiB/s wr, 10 op/s 2023-03-09T21:46:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24503.684145 secs 2023-03-09T21:46:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:33 smithi181 ceph-mon[94189]: pgmap v13579: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 101 KiB/s wr, 10 op/s 2023-03-09T21:46:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24503.684145 secs 2023-03-09T21:46:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:33 smithi038 ceph-mon[92422]: pgmap v13579: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 101 KiB/s wr, 10 op/s 2023-03-09T21:46:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24503.684145 secs 2023-03-09T21:46:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:35 smithi122 ceph-mon[93869]: pgmap v13580: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 193 KiB/s wr, 7 op/s 2023-03-09T21:46:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:35 smithi181 ceph-mon[94189]: pgmap v13580: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 193 KiB/s wr, 7 op/s 2023-03-09T21:46:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:35 smithi038 ceph-mon[92422]: pgmap v13580: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 193 KiB/s wr, 7 op/s 2023-03-09T21:46:37.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24506.332551 secs 2023-03-09T21:46:37.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24506.332551 secs 2023-03-09T21:46:37.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24506.332551 secs 2023-03-09T21:46:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:37 smithi122 ceph-mon[93869]: pgmap v13581: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 193 KiB/s wr, 9 op/s 2023-03-09T21:46:38.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:37 smithi181 ceph-mon[94189]: pgmap v13581: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 193 KiB/s wr, 9 op/s 2023-03-09T21:46:38.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:37 smithi038 ceph-mon[92422]: pgmap v13581: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 193 KiB/s wr, 9 op/s 2023-03-09T21:46:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24970.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:46:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24508.684263 secs 2023-03-09T21:46:39.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24508.684263 secs 2023-03-09T21:46:39.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24508.684263 secs 2023-03-09T21:46:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:39 smithi122 ceph-mon[93869]: pgmap v13582: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 193 KiB/s wr, 6 op/s 2023-03-09T21:46:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:46:40.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:39 smithi181 ceph-mon[94189]: pgmap v13582: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 193 KiB/s wr, 6 op/s 2023-03-09T21:46:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:46:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:39 smithi038 ceph-mon[92422]: pgmap v13582: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 193 KiB/s wr, 6 op/s 2023-03-09T21:46:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:46:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24511.332612 secs 2023-03-09T21:46:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:46:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:46:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:46:41.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24511.332612 secs 2023-03-09T21:46:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:46:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:46:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:46:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24511.332612 secs 2023-03-09T21:46:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:46:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:46:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:46:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:41 smithi122 ceph-mon[93869]: pgmap v13583: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 193 KiB/s wr, 6 op/s 2023-03-09T21:46:42.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:41 smithi181 ceph-mon[94189]: pgmap v13583: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 193 KiB/s wr, 6 op/s 2023-03-09T21:46:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:41 smithi038 ceph-mon[92422]: pgmap v13583: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 193 KiB/s wr, 6 op/s 2023-03-09T21:46:43.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24975.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:46:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:43 smithi122 ceph-mon[93869]: pgmap v13584: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 94 KiB/s wr, 8 op/s 2023-03-09T21:46:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24513.684356 secs 2023-03-09T21:46:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:43 smithi181 ceph-mon[94189]: pgmap v13584: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 94 KiB/s wr, 8 op/s 2023-03-09T21:46:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24513.684356 secs 2023-03-09T21:46:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:43 smithi038 ceph-mon[92422]: pgmap v13584: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 94 KiB/s wr, 8 op/s 2023-03-09T21:46:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24513.684356 secs 2023-03-09T21:46:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:45 smithi122 ceph-mon[93869]: pgmap v13585: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 312 KiB/s wr, 7 op/s 2023-03-09T21:46:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24516.332673 secs 2023-03-09T21:46:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:45 smithi181 ceph-mon[94189]: pgmap v13585: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 312 KiB/s wr, 7 op/s 2023-03-09T21:46:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24516.332673 secs 2023-03-09T21:46:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:45 smithi038 ceph-mon[92422]: pgmap v13585: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 312 KiB/s wr, 7 op/s 2023-03-09T21:46:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24516.332673 secs 2023-03-09T21:46:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:47 smithi122 ceph-mon[93869]: pgmap v13586: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 229 KiB/s wr, 15 op/s 2023-03-09T21:46:48.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:47 smithi181 ceph-mon[94189]: pgmap v13586: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 229 KiB/s wr, 15 op/s 2023-03-09T21:46:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:47 smithi038 ceph-mon[92422]: pgmap v13586: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 229 KiB/s wr, 15 op/s 2023-03-09T21:46:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24980.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:46:49.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:48 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:46:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:48 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:46:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:48 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:46:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:49 smithi122 ceph-mon[93869]: pgmap v13587: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 240 KiB/s wr, 14 op/s 2023-03-09T21:46:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:49 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:46:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:46:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:46:50.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:49 smithi181 ceph-mon[94189]: pgmap v13587: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 240 KiB/s wr, 14 op/s 2023-03-09T21:46:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:49 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:46:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:46:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:46:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:49 smithi038 ceph-mon[92422]: pgmap v13587: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 240 KiB/s wr, 14 op/s 2023-03-09T21:46:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:49 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:46:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:46:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:46:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24521.332736 secs 2023-03-09T21:46:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:51 smithi122 ceph-mon[93869]: pgmap v13588: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 514 KiB/s rd, 458 KiB/s wr, 15 op/s 2023-03-09T21:46:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:46:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:46:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24521.332736 secs 2023-03-09T21:46:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:51 smithi181 ceph-mon[94189]: pgmap v13588: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 514 KiB/s rd, 458 KiB/s wr, 15 op/s 2023-03-09T21:46:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:46:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:46:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24521.332736 secs 2023-03-09T21:46:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:51 smithi038 ceph-mon[92422]: pgmap v13588: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 514 KiB/s rd, 458 KiB/s wr, 15 op/s 2023-03-09T21:46:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:46:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:46:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:53 smithi122 ceph-mon[93869]: pgmap v13589: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 543 KiB/s rd, 459 KiB/s wr, 45 op/s 2023-03-09T21:46:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:46:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:46:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:53 smithi181 ceph-mon[94189]: pgmap v13589: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 543 KiB/s rd, 459 KiB/s wr, 45 op/s 2023-03-09T21:46:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:46:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:46:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:53 smithi038 ceph-mon[92422]: pgmap v13589: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 543 KiB/s rd, 459 KiB/s wr, 45 op/s 2023-03-09T21:46:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:46:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:46:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24985.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:46:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:55 smithi038 ceph-mon[92422]: pgmap v13590: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 768 KiB/s rd, 536 KiB/s wr, 45 op/s 2023-03-09T21:46:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:46:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:46:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:55 smithi122 ceph-mon[93869]: pgmap v13590: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 768 KiB/s rd, 536 KiB/s wr, 45 op/s 2023-03-09T21:46:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:46:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:46:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:55 smithi181 ceph-mon[94189]: pgmap v13590: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 768 KiB/s rd, 536 KiB/s wr, 45 op/s 2023-03-09T21:46:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:46:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:46:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24526.332797 secs 2023-03-09T21:46:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24526.332797 secs 2023-03-09T21:46:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24526.332797 secs 2023-03-09T21:46:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:57 smithi038 ceph-mon[92422]: pgmap v13591: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 891 KiB/s rd, 660 KiB/s wr, 47 op/s 2023-03-09T21:46:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:46:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:46:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:57 smithi122 ceph-mon[93869]: pgmap v13591: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 891 KiB/s rd, 660 KiB/s wr, 47 op/s 2023-03-09T21:46:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:46:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:46:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:57 smithi181 ceph-mon[94189]: pgmap v13591: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 891 KiB/s rd, 660 KiB/s wr, 47 op/s 2023-03-09T21:46:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:46:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:46:58.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24990.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:46:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:59 smithi122 ceph-mon[93869]: pgmap v13592: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 966 KiB/s rd, 744 KiB/s wr, 41 op/s 2023-03-09T21:46:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:46:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:46:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:46:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:59 smithi181 ceph-mon[94189]: pgmap v13592: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 966 KiB/s rd, 744 KiB/s wr, 41 op/s 2023-03-09T21:46:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:46:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:46:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:46:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:59 smithi038 ceph-mon[92422]: pgmap v13592: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 966 KiB/s rd, 744 KiB/s wr, 41 op/s 2023-03-09T21:46:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:47:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:46:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:47:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:00 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): Behind on trimming (678/128) 2023-03-09T21:47:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:00 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): Behind on trimming (678/128) 2023-03-09T21:47:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:00 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): Behind on trimming (678/128) 2023-03-09T21:47:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24531.332858 secs 2023-03-09T21:47:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:01 smithi122 ceph-mon[93869]: pgmap v13593: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 955 KiB/s rd, 732 KiB/s wr, 40 op/s 2023-03-09T21:47:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:01 smithi122 ceph-mon[93869]: Health check update: 1 MDSs behind on trimming (MDS_TRIM) 2023-03-09T21:47:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:47:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:47:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24531.332858 secs 2023-03-09T21:47:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:01 smithi181 ceph-mon[94189]: pgmap v13593: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 955 KiB/s rd, 732 KiB/s wr, 40 op/s 2023-03-09T21:47:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:01 smithi181 ceph-mon[94189]: Health check update: 1 MDSs behind on trimming (MDS_TRIM) 2023-03-09T21:47:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:47:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:47:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24531.332858 secs 2023-03-09T21:47:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:01 smithi038 ceph-mon[92422]: pgmap v13593: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 955 KiB/s rd, 732 KiB/s wr, 40 op/s 2023-03-09T21:47:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:01 smithi038 ceph-mon[92422]: Health check update: 1 MDSs behind on trimming (MDS_TRIM) 2023-03-09T21:47:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T21:47:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:47:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 24995.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:47:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:03 smithi122 ceph-mon[93869]: pgmap v13594: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 722 KiB/s rd, 544 KiB/s wr, 74 op/s 2023-03-09T21:47:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24533.684797 secs 2023-03-09T21:47:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:03 smithi181 ceph-mon[94189]: pgmap v13594: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 722 KiB/s rd, 544 KiB/s wr, 74 op/s 2023-03-09T21:47:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24533.684797 secs 2023-03-09T21:47:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:03 smithi038 ceph-mon[92422]: pgmap v13594: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 722 KiB/s rd, 544 KiB/s wr, 74 op/s 2023-03-09T21:47:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24533.684797 secs 2023-03-09T21:47:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:05 smithi122 ceph-mon[93869]: pgmap v13595: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 720 KiB/s rd, 584 KiB/s wr, 72 op/s 2023-03-09T21:47:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:05 smithi122 ceph-mon[93869]: osdmap e1063: 12 total, 12 up, 12 in 2023-03-09T21:47:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:05 smithi181 ceph-mon[94189]: pgmap v13595: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 720 KiB/s rd, 584 KiB/s wr, 72 op/s 2023-03-09T21:47:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:05 smithi181 ceph-mon[94189]: osdmap e1063: 12 total, 12 up, 12 in 2023-03-09T21:47:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:05 smithi038 ceph-mon[92422]: pgmap v13595: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 720 KiB/s rd, 584 KiB/s wr, 72 op/s 2023-03-09T21:47:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:05 smithi038 ceph-mon[92422]: osdmap e1063: 12 total, 12 up, 12 in 2023-03-09T21:47:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24536.332916 secs 2023-03-09T21:47:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:06 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:47:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:06 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:47:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:47:06.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:06 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): Behind on trimming (679/128) 2023-03-09T21:47:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24536.332916 secs 2023-03-09T21:47:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:06 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:47:06.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:06 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:47:06.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:47:06.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:06 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): Behind on trimming (679/128) 2023-03-09T21:47:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24536.332916 secs 2023-03-09T21:47:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:06 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:47:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:06 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:47:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:47:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:06 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): Behind on trimming (679/128) 2023-03-09T21:47:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:07 smithi122 ceph-mon[93869]: pgmap v13597: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 227 KiB/s wr, 84 op/s 2023-03-09T21:47:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:07 smithi122 ceph-mon[93869]: Health check cleared: MDS_TRIM (was: 1 MDSs behind on trimming) 2023-03-09T21:47:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:47:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:47:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:07 smithi181 ceph-mon[94189]: pgmap v13597: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 227 KiB/s wr, 84 op/s 2023-03-09T21:47:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:07 smithi181 ceph-mon[94189]: Health check cleared: MDS_TRIM (was: 1 MDSs behind on trimming) 2023-03-09T21:47:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:47:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:47:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:07 smithi038 ceph-mon[92422]: pgmap v13597: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 227 KiB/s wr, 84 op/s 2023-03-09T21:47:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:07 smithi038 ceph-mon[92422]: Health check cleared: MDS_TRIM (was: 1 MDSs behind on trimming) 2023-03-09T21:47:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T21:47:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:47:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25000.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:47:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24538.684932 secs 2023-03-09T21:47:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24538.684932 secs 2023-03-09T21:47:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24538.684932 secs 2023-03-09T21:47:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:09 smithi122 ceph-mon[93869]: pgmap v13598: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 400 KiB/s wr, 84 op/s 2023-03-09T21:47:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:09 smithi181 ceph-mon[94189]: pgmap v13598: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 400 KiB/s wr, 84 op/s 2023-03-09T21:47:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:09 smithi038 ceph-mon[92422]: pgmap v13598: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 400 KiB/s wr, 84 op/s 2023-03-09T21:47:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24541.332979 secs 2023-03-09T21:47:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:11 smithi122 ceph-mon[93869]: pgmap v13599: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 400 KiB/s wr, 83 op/s 2023-03-09T21:47:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24541.332979 secs 2023-03-09T21:47:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:11 smithi181 ceph-mon[94189]: pgmap v13599: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 400 KiB/s wr, 83 op/s 2023-03-09T21:47:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24541.332979 secs 2023-03-09T21:47:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:11 smithi038 ceph-mon[92422]: pgmap v13599: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 400 KiB/s wr, 83 op/s 2023-03-09T21:47:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25005.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:47:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:13 smithi122 ceph-mon[93869]: pgmap v13600: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 365 KiB/s wr, 43 op/s 2023-03-09T21:47:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24543.685073 secs 2023-03-09T21:47:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:13 smithi181 ceph-mon[94189]: pgmap v13600: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 365 KiB/s wr, 43 op/s 2023-03-09T21:47:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24543.685073 secs 2023-03-09T21:47:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:13 smithi038 ceph-mon[92422]: pgmap v13600: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 365 KiB/s wr, 43 op/s 2023-03-09T21:47:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24543.685073 secs 2023-03-09T21:47:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:15 smithi122 ceph-mon[93869]: pgmap v13601: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 339 KiB/s wr, 11 op/s 2023-03-09T21:47:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:15 smithi122 ceph-mon[93869]: osdmap e1064: 12 total, 12 up, 12 in 2023-03-09T21:47:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:15 smithi181 ceph-mon[94189]: pgmap v13601: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 339 KiB/s wr, 11 op/s 2023-03-09T21:47:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:15 smithi181 ceph-mon[94189]: osdmap e1064: 12 total, 12 up, 12 in 2023-03-09T21:47:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:15 smithi038 ceph-mon[92422]: pgmap v13601: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 339 KiB/s wr, 11 op/s 2023-03-09T21:47:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:15 smithi038 ceph-mon[92422]: osdmap e1064: 12 total, 12 up, 12 in 2023-03-09T21:47:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24546.333040 secs 2023-03-09T21:47:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24546.333040 secs 2023-03-09T21:47:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24546.333040 secs 2023-03-09T21:47:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:17 smithi122 ceph-mon[93869]: pgmap v13603: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 391 KiB/s wr, 9 op/s 2023-03-09T21:47:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:17 smithi181 ceph-mon[94189]: pgmap v13603: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 391 KiB/s wr, 9 op/s 2023-03-09T21:47:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:17 smithi038 ceph-mon[92422]: pgmap v13603: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 391 KiB/s wr, 9 op/s 2023-03-09T21:47:18.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25010.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:47:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24548.685194 secs 2023-03-09T21:47:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24548.685194 secs 2023-03-09T21:47:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24548.685194 secs 2023-03-09T21:47:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:19 smithi122 ceph-mon[93869]: pgmap v13604: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 105 KiB/s wr, 8 op/s 2023-03-09T21:47:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:19 smithi181 ceph-mon[94189]: pgmap v13604: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 105 KiB/s wr, 8 op/s 2023-03-09T21:47:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:19 smithi038 ceph-mon[92422]: pgmap v13604: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 105 KiB/s wr, 8 op/s 2023-03-09T21:47:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24551.333126 secs 2023-03-09T21:47:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:21 smithi122 ceph-mon[93869]: pgmap v13605: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 105 KiB/s wr, 8 op/s 2023-03-09T21:47:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24551.333126 secs 2023-03-09T21:47:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:21 smithi181 ceph-mon[94189]: pgmap v13605: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 105 KiB/s wr, 8 op/s 2023-03-09T21:47:21.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24551.333126 secs 2023-03-09T21:47:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:21 smithi038 ceph-mon[92422]: pgmap v13605: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 105 KiB/s wr, 8 op/s 2023-03-09T21:47:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25015.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:47:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:23 smithi122 ceph-mon[93869]: pgmap v13606: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 105 KiB/s wr, 7 op/s 2023-03-09T21:47:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24553.685294 secs 2023-03-09T21:47:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:23 smithi181 ceph-mon[94189]: pgmap v13606: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 105 KiB/s wr, 7 op/s 2023-03-09T21:47:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24553.685294 secs 2023-03-09T21:47:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:23 smithi038 ceph-mon[92422]: pgmap v13606: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 105 KiB/s wr, 7 op/s 2023-03-09T21:47:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24553.685294 secs 2023-03-09T21:47:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:25 smithi122 ceph-mon[93869]: pgmap v13607: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 83 KiB/s wr, 7 op/s 2023-03-09T21:47:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:25 smithi181 ceph-mon[94189]: pgmap v13607: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 83 KiB/s wr, 7 op/s 2023-03-09T21:47:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:25 smithi038 ceph-mon[92422]: pgmap v13607: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 83 KiB/s wr, 7 op/s 2023-03-09T21:47:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24556.333187 secs 2023-03-09T21:47:26.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24556.333187 secs 2023-03-09T21:47:26.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24556.333187 secs 2023-03-09T21:47:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:27 smithi122 ceph-mon[93869]: pgmap v13608: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.8 KiB/s wr, 8 op/s 2023-03-09T21:47:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:27 smithi181 ceph-mon[94189]: pgmap v13608: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.8 KiB/s wr, 8 op/s 2023-03-09T21:47:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:27 smithi038 ceph-mon[92422]: pgmap v13608: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.8 KiB/s wr, 8 op/s 2023-03-09T21:47:28.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25020.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:47:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24558.685377 secs 2023-03-09T21:47:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24558.685377 secs 2023-03-09T21:47:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24558.685377 secs 2023-03-09T21:47:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:29 smithi122 ceph-mon[93869]: pgmap v13609: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2023-03-09T21:47:29.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:29 smithi181 ceph-mon[94189]: pgmap v13609: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2023-03-09T21:47:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:29 smithi038 ceph-mon[92422]: pgmap v13609: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2023-03-09T21:47:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24561.333271 secs 2023-03-09T21:47:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:31 smithi122 ceph-mon[93869]: pgmap v13610: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2023-03-09T21:47:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24561.333271 secs 2023-03-09T21:47:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:31 smithi181 ceph-mon[94189]: pgmap v13610: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2023-03-09T21:47:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24561.333271 secs 2023-03-09T21:47:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:31 smithi038 ceph-mon[92422]: pgmap v13610: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2023-03-09T21:47:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25025.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:47:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:33 smithi122 ceph-mon[93869]: pgmap v13611: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 101 KiB/s wr, 8 op/s 2023-03-09T21:47:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24563.685488 secs 2023-03-09T21:47:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:33 smithi181 ceph-mon[94189]: pgmap v13611: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 101 KiB/s wr, 8 op/s 2023-03-09T21:47:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24563.685488 secs 2023-03-09T21:47:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:33 smithi038 ceph-mon[92422]: pgmap v13611: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 101 KiB/s wr, 8 op/s 2023-03-09T21:47:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24563.685488 secs 2023-03-09T21:47:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:35 smithi122 ceph-mon[93869]: pgmap v13612: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 101 KiB/s wr, 6 op/s 2023-03-09T21:47:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:35 smithi181 ceph-mon[94189]: pgmap v13612: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 101 KiB/s wr, 6 op/s 2023-03-09T21:47:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:35 smithi038 ceph-mon[92422]: pgmap v13612: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 101 KiB/s wr, 6 op/s 2023-03-09T21:47:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24566.333330 secs 2023-03-09T21:47:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24566.333330 secs 2023-03-09T21:47:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24566.333330 secs 2023-03-09T21:47:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:37 smithi122 ceph-mon[93869]: pgmap v13613: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 100 KiB/s wr, 8 op/s 2023-03-09T21:47:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:37 smithi181 ceph-mon[94189]: pgmap v13613: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 100 KiB/s wr, 8 op/s 2023-03-09T21:47:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:37 smithi038 ceph-mon[92422]: pgmap v13613: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 100 KiB/s wr, 8 op/s 2023-03-09T21:47:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25030.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:47:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24568.685613 secs 2023-03-09T21:47:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24568.685613 secs 2023-03-09T21:47:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24568.685613 secs 2023-03-09T21:47:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:39 smithi122 ceph-mon[93869]: pgmap v13614: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 100 KiB/s wr, 7 op/s 2023-03-09T21:47:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:39 smithi181 ceph-mon[94189]: pgmap v13614: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 100 KiB/s wr, 7 op/s 2023-03-09T21:47:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:39 smithi038 ceph-mon[92422]: pgmap v13614: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 100 KiB/s wr, 7 op/s 2023-03-09T21:47:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:47:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:47:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:47:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24571.333391 secs 2023-03-09T21:47:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:41 smithi122 ceph-mon[93869]: pgmap v13615: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 100 KiB/s wr, 6 op/s 2023-03-09T21:47:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24571.333391 secs 2023-03-09T21:47:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:41 smithi181 ceph-mon[94189]: pgmap v13615: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 100 KiB/s wr, 6 op/s 2023-03-09T21:47:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24571.333391 secs 2023-03-09T21:47:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:41 smithi038 ceph-mon[92422]: pgmap v13615: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 100 KiB/s wr, 6 op/s 2023-03-09T21:47:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25035.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:47:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:43 smithi122 ceph-mon[93869]: pgmap v13616: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 273 KiB/s wr, 8 op/s 2023-03-09T21:47:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24573.685693 secs 2023-03-09T21:47:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:47:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:47:43.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:47:43.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:47:43.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:47:43.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:47:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:43 smithi181 ceph-mon[94189]: pgmap v13616: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 273 KiB/s wr, 8 op/s 2023-03-09T21:47:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24573.685693 secs 2023-03-09T21:47:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:47:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:47:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:47:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:47:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:47:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:47:43.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:43 smithi038 ceph-mon[92422]: pgmap v13616: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 273 KiB/s wr, 8 op/s 2023-03-09T21:47:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24573.685693 secs 2023-03-09T21:47:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:47:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:47:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:47:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:47:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:47:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:47:44.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:44 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:47:44.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:44 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:47:44.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:44 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:47:44.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:44 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:47:44.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:44 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi038 to 37664k 2023-03-09T21:47:44.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:44 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi038 to 38568050: error parsing value: Value '38568050' is below minimum 939524096 2023-03-09T21:47:44.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:44 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:47:44.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:44 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:47:44.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:44 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:47:44.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:44 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:47:44.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:44 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:47:44.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:44 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:47:44.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:44 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:47:44.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:44 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:47:44.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:44 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:47:44.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:44 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:47:44.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:44 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:47:44.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:44 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:47:44.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:44 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:47:44.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:44 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:47:44.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:44 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:47:44.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:44 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi038 to 37664k 2023-03-09T21:47:44.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:44 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi038 to 38568050: error parsing value: Value '38568050' is below minimum 939524096 2023-03-09T21:47:44.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:44 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:47:44.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:44 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:47:44.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:44 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:47:44.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:44 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:47:44.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:44 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:47:44.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:44 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:47:44.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:44 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:47:44.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:44 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:47:44.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:44 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:47:44.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:44 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:47:44.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:44 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:47:44.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:44 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:47:44.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:44 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:47:44.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:44 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:47:44.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:44 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:47:44.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:44 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi038 to 37664k 2023-03-09T21:47:45.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:44 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi038 to 38568050: error parsing value: Value '38568050' is below minimum 939524096 2023-03-09T21:47:45.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:44 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:47:45.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:44 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:47:45.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:44 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:47:45.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:44 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:47:45.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:44 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:47:45.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:44 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:47:45.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:44 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:47:45.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:44 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:47:45.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:44 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:47:45.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:44 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:47:45.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:44 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:47:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:45 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T21:47:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:45 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T21:47:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:45 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T21:47:45.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:45 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T21:47:45.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:45 smithi122 ceph-mon[93869]: pgmap v13617: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 338 KiB/s wr, 7 op/s 2023-03-09T21:47:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:45 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T21:47:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:45 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T21:47:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:45 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T21:47:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:45 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T21:47:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:45 smithi181 ceph-mon[94189]: pgmap v13617: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 338 KiB/s wr, 7 op/s 2023-03-09T21:47:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:45 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T21:47:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:45 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T21:47:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:45 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T21:47:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:45 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T21:47:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:45 smithi038 ceph-mon[92422]: pgmap v13617: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 338 KiB/s wr, 7 op/s 2023-03-09T21:47:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24576.333478 secs 2023-03-09T21:47:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24576.333478 secs 2023-03-09T21:47:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24576.333478 secs 2023-03-09T21:47:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:47 smithi122 ceph-mon[93869]: pgmap v13618: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 338 KiB/s wr, 10 op/s 2023-03-09T21:47:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:47 smithi181 ceph-mon[94189]: pgmap v13618: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 338 KiB/s wr, 10 op/s 2023-03-09T21:47:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:47 smithi038 ceph-mon[92422]: pgmap v13618: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 338 KiB/s wr, 10 op/s 2023-03-09T21:47:48.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25040.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:47:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24578.685820 secs 2023-03-09T21:47:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24578.685820 secs 2023-03-09T21:47:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24578.685820 secs 2023-03-09T21:47:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:49 smithi122 ceph-mon[93869]: pgmap v13619: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 672 KiB/s rd, 467 KiB/s wr, 10 op/s 2023-03-09T21:47:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:49 smithi181 ceph-mon[94189]: pgmap v13619: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 672 KiB/s rd, 467 KiB/s wr, 10 op/s 2023-03-09T21:47:49.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:49 smithi038 ceph-mon[92422]: pgmap v13619: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 672 KiB/s rd, 467 KiB/s wr, 10 op/s 2023-03-09T21:47:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24581.333537 secs 2023-03-09T21:47:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:51 smithi122 ceph-mon[93869]: pgmap v13620: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 672 KiB/s rd, 467 KiB/s wr, 9 op/s 2023-03-09T21:47:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24581.333537 secs 2023-03-09T21:47:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:51 smithi181 ceph-mon[94189]: pgmap v13620: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 672 KiB/s rd, 467 KiB/s wr, 9 op/s 2023-03-09T21:47:51.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24581.333537 secs 2023-03-09T21:47:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:51 smithi038 ceph-mon[92422]: pgmap v13620: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 672 KiB/s rd, 467 KiB/s wr, 9 op/s 2023-03-09T21:47:53.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25045.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:47:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:53 smithi122 ceph-mon[93869]: pgmap v13621: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 673 KiB/s rd, 467 KiB/s wr, 12 op/s 2023-03-09T21:47:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24583.685935 secs 2023-03-09T21:47:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:53 smithi181 ceph-mon[94189]: pgmap v13621: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 673 KiB/s rd, 467 KiB/s wr, 12 op/s 2023-03-09T21:47:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24583.685935 secs 2023-03-09T21:47:53.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:53 smithi038 ceph-mon[92422]: pgmap v13621: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 673 KiB/s rd, 467 KiB/s wr, 12 op/s 2023-03-09T21:47:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24583.685935 secs 2023-03-09T21:47:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:55 smithi122 ceph-mon[93869]: pgmap v13622: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 758 KiB/s rd, 417 KiB/s wr, 11 op/s 2023-03-09T21:47:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:55 smithi181 ceph-mon[94189]: pgmap v13622: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 758 KiB/s rd, 417 KiB/s wr, 11 op/s 2023-03-09T21:47:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:55 smithi038 ceph-mon[92422]: pgmap v13622: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 758 KiB/s rd, 417 KiB/s wr, 11 op/s 2023-03-09T21:47:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24586.333598 secs 2023-03-09T21:47:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24586.333598 secs 2023-03-09T21:47:56.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24586.333598 secs 2023-03-09T21:47:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:57 smithi122 ceph-mon[93869]: pgmap v13623: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 925 KiB/s rd, 594 KiB/s wr, 12 op/s 2023-03-09T21:47:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:57 smithi181 ceph-mon[94189]: pgmap v13623: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 925 KiB/s rd, 594 KiB/s wr, 12 op/s 2023-03-09T21:47:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:57 smithi038 ceph-mon[92422]: pgmap v13623: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 925 KiB/s rd, 594 KiB/s wr, 12 op/s 2023-03-09T21:47:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25050.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:47:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24588.686031 secs 2023-03-09T21:47:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24588.686031 secs 2023-03-09T21:47:58.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24588.686031 secs 2023-03-09T21:47:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:47:59 smithi122 ceph-mon[93869]: pgmap v13624: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 595 KiB/s wr, 11 op/s 2023-03-09T21:47:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:47:59 smithi181 ceph-mon[94189]: pgmap v13624: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 595 KiB/s wr, 11 op/s 2023-03-09T21:47:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:47:59 smithi038 ceph-mon[92422]: pgmap v13624: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 595 KiB/s wr, 11 op/s 2023-03-09T21:48:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:48:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24591.333658 secs 2023-03-09T21:48:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:48:01 smithi181 ceph-mon[94189]: pgmap v13625: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 712 KiB/s rd, 575 KiB/s wr, 9 op/s 2023-03-09T21:48:01.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:48:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24591.333658 secs 2023-03-09T21:48:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:48:01 smithi038 ceph-mon[92422]: pgmap v13625: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 712 KiB/s rd, 575 KiB/s wr, 9 op/s 2023-03-09T21:48:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:48:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24591.333658 secs 2023-03-09T21:48:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:48:01 smithi122 ceph-mon[93869]: pgmap v13625: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 712 KiB/s rd, 575 KiB/s wr, 9 op/s 2023-03-09T21:48:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25055.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:48:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:48:03 smithi038 ceph-mon[92422]: pgmap v13626: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 714 KiB/s rd, 575 KiB/s wr, 11 op/s 2023-03-09T21:48:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:48:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24593.686149 secs 2023-03-09T21:48:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:48:03 smithi122 ceph-mon[93869]: pgmap v13626: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 714 KiB/s rd, 575 KiB/s wr, 11 op/s 2023-03-09T21:48:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:48:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24593.686149 secs 2023-03-09T21:48:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:48:03 smithi181 ceph-mon[94189]: pgmap v13626: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 714 KiB/s rd, 575 KiB/s wr, 11 op/s 2023-03-09T21:48:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:48:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24593.686149 secs 2023-03-09T21:48:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:48:05 smithi038 ceph-mon[92422]: pgmap v13627: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 933 KiB/s rd, 810 KiB/s wr, 23 op/s 2023-03-09T21:48:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:48:05 smithi122 ceph-mon[93869]: pgmap v13627: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 933 KiB/s rd, 810 KiB/s wr, 23 op/s 2023-03-09T21:48:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:48:05 smithi181 ceph-mon[94189]: pgmap v13627: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 933 KiB/s rd, 810 KiB/s wr, 23 op/s 2023-03-09T21:48:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:48:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24596.333718 secs 2023-03-09T21:48:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:48:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24596.333718 secs 2023-03-09T21:48:07.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:48:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24596.333718 secs 2023-03-09T21:48:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:48:07 smithi038 ceph-mon[92422]: pgmap v13628: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 677 KiB/s rd, 709 KiB/s wr, 25 op/s 2023-03-09T21:48:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:48:07 smithi122 ceph-mon[93869]: pgmap v13628: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 677 KiB/s rd, 709 KiB/s wr, 25 op/s 2023-03-09T21:48:08.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:48:07 smithi181 ceph-mon[94189]: pgmap v13628: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 677 KiB/s rd, 709 KiB/s wr, 25 op/s 2023-03-09T21:48:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25060.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:48:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:48:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24598.686266 secs 2023-03-09T21:48:09.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:48:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24598.686266 secs 2023-03-09T21:48:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:48:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24598.686266 secs 2023-03-09T21:48:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:48:09 smithi122 ceph-mon[93869]: pgmap v13629: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 385 KiB/s wr, 23 op/s 2023-03-09T21:48:10.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:48:09 smithi181 ceph-mon[94189]: pgmap v13629: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 385 KiB/s wr, 23 op/s 2023-03-09T21:48:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:48:09 smithi038 ceph-mon[92422]: pgmap v13629: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 385 KiB/s wr, 23 op/s 2023-03-09T21:48:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:48:10 smithi122 ceph-mon[93869]: osdmap e1065: 12 total, 12 up, 12 in 2023-03-09T21:48:11.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:48:10 smithi181 ceph-mon[94189]: osdmap e1065: 12 total, 12 up, 12 in 2023-03-09T21:48:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:48:10 smithi038 ceph-mon[92422]: osdmap e1065: 12 total, 12 up, 12 in 2023-03-09T21:48:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:48:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24601.333780 secs 2023-03-09T21:48:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:48:11 smithi122 ceph-mon[93869]: pgmap v13631: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 569 KiB/s wr, 27 op/s 2023-03-09T21:48:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:48:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24601.333780 secs 2023-03-09T21:48:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:48:11 smithi181 ceph-mon[94189]: pgmap v13631: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 569 KiB/s wr, 27 op/s 2023-03-09T21:48:12.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:48:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24601.333780 secs 2023-03-09T21:48:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:48:11 smithi038 ceph-mon[92422]: pgmap v13631: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 569 KiB/s wr, 27 op/s 2023-03-09T21:48:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25065.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:48:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:48:13 smithi122 ceph-mon[93869]: pgmap v13632: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 569 KiB/s wr, 27 op/s 2023-03-09T21:48:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:48:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24603.686347 secs 2023-03-09T21:48:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:48:13 smithi181 ceph-mon[94189]: pgmap v13632: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 569 KiB/s wr, 27 op/s 2023-03-09T21:48:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:48:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24603.686347 secs 2023-03-09T21:48:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:48:13 smithi038 ceph-mon[92422]: pgmap v13632: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 569 KiB/s wr, 27 op/s 2023-03-09T21:48:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:48:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24603.686347 secs 2023-03-09T21:48:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:48:15 smithi122 ceph-mon[93869]: pgmap v13633: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 408 KiB/s wr, 11 op/s 2023-03-09T21:48:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:48:15 smithi122 ceph-mon[93869]: osdmap e1066: 12 total, 12 up, 12 in 2023-03-09T21:48:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:48:15 smithi181 ceph-mon[94189]: pgmap v13633: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 408 KiB/s wr, 11 op/s 2023-03-09T21:48:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:48:15 smithi181 ceph-mon[94189]: osdmap e1066: 12 total, 12 up, 12 in 2023-03-09T21:48:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:48:15 smithi038 ceph-mon[92422]: pgmap v13633: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 408 KiB/s wr, 11 op/s 2023-03-09T21:48:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:48:15 smithi038 ceph-mon[92422]: osdmap e1066: 12 total, 12 up, 12 in 2023-03-09T21:48:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:48:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24606.333839 secs 2023-03-09T21:48:17.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:48:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24606.333839 secs 2023-03-09T21:48:17.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:48:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24606.333839 secs 2023-03-09T21:48:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:48:17 smithi122 ceph-mon[93869]: pgmap v13635: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 454 KiB/s rd, 451 KiB/s wr, 9 op/s 2023-03-09T21:48:18.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:48:17 smithi181 ceph-mon[94189]: pgmap v13635: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 454 KiB/s rd, 451 KiB/s wr, 9 op/s 2023-03-09T21:48:18.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:48:17 smithi038 ceph-mon[92422]: pgmap v13635: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 454 KiB/s rd, 451 KiB/s wr, 9 op/s 2023-03-09T21:48:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25070.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:48:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:48:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24608.686425 secs 2023-03-09T21:48:19.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:48:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24608.686425 secs 2023-03-09T21:48:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:48:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24608.686425 secs 2023-03-09T21:48:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:48:19 smithi122 ceph-mon[93869]: pgmap v13636: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 173 KiB/s wr, 8 op/s 2023-03-09T21:48:20.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:48:19 smithi181 ceph-mon[94189]: pgmap v13636: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 173 KiB/s wr, 8 op/s 2023-03-09T21:48:20.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:48:19 smithi038 ceph-mon[92422]: pgmap v13636: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 173 KiB/s wr, 8 op/s 2023-03-09T21:48:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:48:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24611.333928 secs 2023-03-09T21:48:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:48:21 smithi038 ceph-mon[92422]: pgmap v13637: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 227 KiB/s wr, 7 op/s 2023-03-09T21:48:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:48:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24611.333928 secs 2023-03-09T21:48:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:48:21 smithi122 ceph-mon[93869]: pgmap v13637: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 227 KiB/s wr, 7 op/s 2023-03-09T21:48:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:48:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24611.333928 secs 2023-03-09T21:48:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:48:21 smithi181 ceph-mon[94189]: pgmap v13637: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 227 KiB/s wr, 7 op/s 2023-03-09T21:48:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:48:23 smithi038 ceph-mon[92422]: pgmap v13638: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 228 KiB/s wr, 8 op/s 2023-03-09T21:48:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:48:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24613.686539 secs 2023-03-09T21:48:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25075.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:48:23.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:48:23 smithi122 ceph-mon[93869]: pgmap v13638: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 228 KiB/s wr, 8 op/s 2023-03-09T21:48:23.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:48:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24613.686539 secs 2023-03-09T21:48:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:48:23 smithi181 ceph-mon[94189]: pgmap v13638: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 228 KiB/s wr, 8 op/s 2023-03-09T21:48:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:48:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24613.686539 secs 2023-03-09T21:48:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:48:25 smithi122 ceph-mon[93869]: pgmap v13639: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 106 KiB/s wr, 8 op/s 2023-03-09T21:48:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:48:25 smithi181 ceph-mon[94189]: pgmap v13639: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 106 KiB/s wr, 8 op/s 2023-03-09T21:48:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:48:25 smithi038 ceph-mon[92422]: pgmap v13639: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 106 KiB/s wr, 8 op/s 2023-03-09T21:48:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:48:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24616.334009 secs 2023-03-09T21:48:26.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:48:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24616.334009 secs 2023-03-09T21:48:26.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:48:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24616.334009 secs 2023-03-09T21:48:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:48:27 smithi122 ceph-mon[93869]: pgmap v13640: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 100 KiB/s wr, 9 op/s 2023-03-09T21:48:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:48:27 smithi181 ceph-mon[94189]: pgmap v13640: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 100 KiB/s wr, 9 op/s 2023-03-09T21:48:27.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:48:27 smithi038 ceph-mon[92422]: pgmap v13640: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 100 KiB/s wr, 9 op/s 2023-03-09T21:48:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25080.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:48:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:48:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24618.686656 secs 2023-03-09T21:48:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:48:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24618.686656 secs 2023-03-09T21:48:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:48:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24618.686656 secs 2023-03-09T21:48:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:48:29 smithi122 ceph-mon[93869]: pgmap v13641: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 89 KiB/s wr, 7 op/s 2023-03-09T21:48:29.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:48:29 smithi181 ceph-mon[94189]: pgmap v13641: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 89 KiB/s wr, 7 op/s 2023-03-09T21:48:29.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:48:29 smithi038 ceph-mon[92422]: pgmap v13641: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 89 KiB/s wr, 7 op/s 2023-03-09T21:48:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:48:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24621.334080 secs 2023-03-09T21:48:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:48:31 smithi122 ceph-mon[93869]: pgmap v13642: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 86 KiB/s wr, 7 op/s 2023-03-09T21:48:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:48:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24621.334080 secs 2023-03-09T21:48:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:48:31 smithi181 ceph-mon[94189]: pgmap v13642: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 86 KiB/s wr, 7 op/s 2023-03-09T21:48:31.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:48:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24621.334080 secs 2023-03-09T21:48:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:48:31 smithi038 ceph-mon[92422]: pgmap v13642: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 86 KiB/s wr, 7 op/s 2023-03-09T21:48:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25085.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:48:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:48:33 smithi122 ceph-mon[93869]: pgmap v13643: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 102 KiB/s wr, 9 op/s 2023-03-09T21:48:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:48:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24623.686778 secs 2023-03-09T21:48:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:48:33 smithi181 ceph-mon[94189]: pgmap v13643: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 102 KiB/s wr, 9 op/s 2023-03-09T21:48:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:48:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24623.686778 secs 2023-03-09T21:48:33.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:48:33 smithi038 ceph-mon[92422]: pgmap v13643: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 102 KiB/s wr, 9 op/s 2023-03-09T21:48:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:48:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24623.686778 secs 2023-03-09T21:48:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:48:35 smithi122 ceph-mon[93869]: pgmap v13644: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 101 KiB/s wr, 8 op/s 2023-03-09T21:48:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:48:35 smithi181 ceph-mon[94189]: pgmap v13644: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 101 KiB/s wr, 8 op/s 2023-03-09T21:48:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:48:35 smithi038 ceph-mon[92422]: pgmap v13644: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 101 KiB/s wr, 8 op/s 2023-03-09T21:48:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:48:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24626.334138 secs 2023-03-09T21:48:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:48:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24626.334138 secs 2023-03-09T21:48:36.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:48:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24626.334138 secs 2023-03-09T21:48:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:48:37 smithi122 ceph-mon[93869]: pgmap v13645: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 195 KiB/s wr, 9 op/s 2023-03-09T21:48:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:48:37 smithi181 ceph-mon[94189]: pgmap v13645: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 195 KiB/s wr, 9 op/s 2023-03-09T21:48:37.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:48:37 smithi038 ceph-mon[92422]: pgmap v13645: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 195 KiB/s wr, 9 op/s 2023-03-09T21:48:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25090.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:48:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:48:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24628.686890 secs 2023-03-09T21:48:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:48:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24628.686890 secs 2023-03-09T21:48:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:48:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24628.686890 secs 2023-03-09T21:48:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:48:39 smithi122 ceph-mon[93869]: pgmap v13646: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 195 KiB/s wr, 8 op/s 2023-03-09T21:48:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:48:39 smithi181 ceph-mon[94189]: pgmap v13646: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 195 KiB/s wr, 8 op/s 2023-03-09T21:48:39.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:48:39 smithi038 ceph-mon[92422]: pgmap v13646: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 195 KiB/s wr, 8 op/s 2023-03-09T21:48:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:48:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24631.334201 secs 2023-03-09T21:48:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:48:41 smithi122 ceph-mon[93869]: pgmap v13647: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 195 KiB/s wr, 7 op/s 2023-03-09T21:48:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:48:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24631.334201 secs 2023-03-09T21:48:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:48:41 smithi181 ceph-mon[94189]: pgmap v13647: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 195 KiB/s wr, 7 op/s 2023-03-09T21:48:41.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:48:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24631.334201 secs 2023-03-09T21:48:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:48:41 smithi038 ceph-mon[92422]: pgmap v13647: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 195 KiB/s wr, 7 op/s 2023-03-09T21:48:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25095.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:48:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:48:43 smithi122 ceph-mon[93869]: pgmap v13648: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 343 KiB/s wr, 10 op/s 2023-03-09T21:48:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:48:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24633.687033 secs 2023-03-09T21:48:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:48:43 smithi181 ceph-mon[94189]: pgmap v13648: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 343 KiB/s wr, 10 op/s 2023-03-09T21:48:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:48:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24633.687033 secs 2023-03-09T21:48:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:48:43 smithi038 ceph-mon[92422]: pgmap v13648: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 343 KiB/s wr, 10 op/s 2023-03-09T21:48:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:48:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24633.687033 secs 2023-03-09T21:48:44.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:48:44 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:48:44.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:48:44 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:48:44.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:48:44 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:48:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:48:45 smithi122 ceph-mon[93869]: pgmap v13649: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 279 KiB/s wr, 8 op/s 2023-03-09T21:48:45.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:48:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:48:45.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:48:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:48:45.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:48:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:48:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:48:45 smithi181 ceph-mon[94189]: pgmap v13649: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 279 KiB/s wr, 8 op/s 2023-03-09T21:48:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:48:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:48:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:48:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:48:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:48:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:48:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:48:45 smithi038 ceph-mon[92422]: pgmap v13649: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 279 KiB/s wr, 8 op/s 2023-03-09T21:48:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:48:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:48:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:48:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:48:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:48:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:48:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:48:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24636.334264 secs 2023-03-09T21:48:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:48:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24636.334264 secs 2023-03-09T21:48:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:48:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24636.334264 secs 2023-03-09T21:48:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:48:47 smithi122 ceph-mon[93869]: pgmap v13650: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 279 KiB/s wr, 10 op/s 2023-03-09T21:48:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:48:47 smithi181 ceph-mon[94189]: pgmap v13650: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 279 KiB/s wr, 10 op/s 2023-03-09T21:48:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:48:47 smithi038 ceph-mon[92422]: pgmap v13650: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 279 KiB/s wr, 10 op/s 2023-03-09T21:48:48.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25100.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:48:49.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:48:49 smithi122 ceph-mon[93869]: pgmap v13651: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 662 KiB/s rd, 483 KiB/s wr, 10 op/s 2023-03-09T21:48:49.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:48:49 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:48:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:48:49 smithi181 ceph-mon[94189]: pgmap v13651: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 662 KiB/s rd, 483 KiB/s wr, 10 op/s 2023-03-09T21:48:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:48:49 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:48:49.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:48:49 smithi038 ceph-mon[92422]: pgmap v13651: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 662 KiB/s rd, 483 KiB/s wr, 10 op/s 2023-03-09T21:48:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:48:49 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:48:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:48:50 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:48:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:48:50 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:48:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:48:50 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:48:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:48:50 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:48:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:48:50 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:48:50.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:48:50 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:48:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:48:50 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:48:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:48:50 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:48:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:48:50 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:48:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:48:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24641.334325 secs 2023-03-09T21:48:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:48:51 smithi122 ceph-mon[93869]: pgmap v13652: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 662 KiB/s rd, 483 KiB/s wr, 9 op/s 2023-03-09T21:48:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:48:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24641.334325 secs 2023-03-09T21:48:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:48:51 smithi181 ceph-mon[94189]: pgmap v13652: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 662 KiB/s rd, 483 KiB/s wr, 9 op/s 2023-03-09T21:48:51.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:48:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24641.334325 secs 2023-03-09T21:48:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:48:51 smithi038 ceph-mon[92422]: pgmap v13652: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 662 KiB/s rd, 483 KiB/s wr, 9 op/s 2023-03-09T21:48:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25105.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:48:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:48:53 smithi122 ceph-mon[93869]: pgmap v13653: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 841 KiB/s rd, 661 KiB/s wr, 13 op/s 2023-03-09T21:48:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:48:53 smithi181 ceph-mon[94189]: pgmap v13653: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 841 KiB/s rd, 661 KiB/s wr, 13 op/s 2023-03-09T21:48:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:48:53 smithi038 ceph-mon[92422]: pgmap v13653: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 841 KiB/s rd, 661 KiB/s wr, 13 op/s 2023-03-09T21:48:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:48:55 smithi122 ceph-mon[93869]: pgmap v13654: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 695 KiB/s rd, 515 KiB/s wr, 11 op/s 2023-03-09T21:48:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:48:55 smithi181 ceph-mon[94189]: pgmap v13654: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 695 KiB/s rd, 515 KiB/s wr, 11 op/s 2023-03-09T21:48:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:48:55 smithi038 ceph-mon[92422]: pgmap v13654: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 695 KiB/s rd, 515 KiB/s wr, 11 op/s 2023-03-09T21:48:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:48:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24646.334386 secs 2023-03-09T21:48:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:48:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24646.334386 secs 2023-03-09T21:48:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:48:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24646.334386 secs 2023-03-09T21:48:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:48:57 smithi122 ceph-mon[93869]: pgmap v13655: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 893 KiB/s rd, 724 KiB/s wr, 13 op/s 2023-03-09T21:48:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:48:57 smithi181 ceph-mon[94189]: pgmap v13655: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 893 KiB/s rd, 724 KiB/s wr, 13 op/s 2023-03-09T21:48:57.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:48:57 smithi038 ceph-mon[92422]: pgmap v13655: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 893 KiB/s rd, 724 KiB/s wr, 13 op/s 2023-03-09T21:48:58.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25110.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:48:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:48:59 smithi122 ceph-mon[93869]: pgmap v13656: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 784 KiB/s wr, 12 op/s 2023-03-09T21:48:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:48:59 smithi181 ceph-mon[94189]: pgmap v13656: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 784 KiB/s wr, 12 op/s 2023-03-09T21:48:59.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:48:59 smithi038 ceph-mon[92422]: pgmap v13656: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 784 KiB/s wr, 12 op/s 2023-03-09T21:49:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:49:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24651.334464 secs 2023-03-09T21:49:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:49:01 smithi122 ceph-mon[93869]: pgmap v13657: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 486 KiB/s wr, 10 op/s 2023-03-09T21:49:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:49:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24651.334464 secs 2023-03-09T21:49:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:49:01 smithi181 ceph-mon[94189]: pgmap v13657: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 486 KiB/s wr, 10 op/s 2023-03-09T21:49:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:49:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24651.334464 secs 2023-03-09T21:49:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:49:01 smithi038 ceph-mon[92422]: pgmap v13657: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 486 KiB/s wr, 10 op/s 2023-03-09T21:49:03.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25115.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:49:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:49:03 smithi122 ceph-mon[93869]: pgmap v13658: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 547 KiB/s rd, 566 KiB/s wr, 16 op/s 2023-03-09T21:49:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:49:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24653.687433 secs 2023-03-09T21:49:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:49:03 smithi181 ceph-mon[94189]: pgmap v13658: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 547 KiB/s rd, 566 KiB/s wr, 16 op/s 2023-03-09T21:49:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:49:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24653.687433 secs 2023-03-09T21:49:03.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:49:03 smithi038 ceph-mon[92422]: pgmap v13658: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 547 KiB/s rd, 566 KiB/s wr, 16 op/s 2023-03-09T21:49:03.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:49:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24653.687433 secs 2023-03-09T21:49:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:49:05 smithi122 ceph-mon[93869]: pgmap v13659: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 395 KiB/s rd, 429 KiB/s wr, 25 op/s 2023-03-09T21:49:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:49:05 smithi122 ceph-mon[93869]: osdmap e1067: 12 total, 12 up, 12 in 2023-03-09T21:49:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:49:05 smithi181 ceph-mon[94189]: pgmap v13659: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 395 KiB/s rd, 429 KiB/s wr, 25 op/s 2023-03-09T21:49:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:49:05 smithi181 ceph-mon[94189]: osdmap e1067: 12 total, 12 up, 12 in 2023-03-09T21:49:05.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:49:05 smithi038 ceph-mon[92422]: pgmap v13659: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 395 KiB/s rd, 429 KiB/s wr, 25 op/s 2023-03-09T21:49:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:49:05 smithi038 ceph-mon[92422]: osdmap e1067: 12 total, 12 up, 12 in 2023-03-09T21:49:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:49:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24656.334527 secs 2023-03-09T21:49:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:49:06 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:49:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:49:06 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:49:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:49:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:49:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:49:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24656.334527 secs 2023-03-09T21:49:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:49:06 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:49:06.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:49:06 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:49:06.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:49:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:49:06.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:49:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24656.334527 secs 2023-03-09T21:49:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:49:06 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:49:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:49:06 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:49:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:49:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:49:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:49:07 smithi122 ceph-mon[93869]: pgmap v13661: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 218 KiB/s wr, 27 op/s 2023-03-09T21:49:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:49:07 smithi181 ceph-mon[94189]: pgmap v13661: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 218 KiB/s wr, 27 op/s 2023-03-09T21:49:07.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:49:07 smithi038 ceph-mon[92422]: pgmap v13661: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 218 KiB/s wr, 27 op/s 2023-03-09T21:49:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25120.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:49:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:49:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24658.687536 secs 2023-03-09T21:49:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:49:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24658.687536 secs 2023-03-09T21:49:08.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:49:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24658.687536 secs 2023-03-09T21:49:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:49:09 smithi122 ceph-mon[93869]: pgmap v13662: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 438 KiB/s wr, 27 op/s 2023-03-09T21:49:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:49:09 smithi181 ceph-mon[94189]: pgmap v13662: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 438 KiB/s wr, 27 op/s 2023-03-09T21:49:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:49:09 smithi038 ceph-mon[92422]: pgmap v13662: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 438 KiB/s wr, 27 op/s 2023-03-09T21:49:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:49:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24661.334602 secs 2023-03-09T21:49:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:49:11 smithi122 ceph-mon[93869]: pgmap v13663: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 532 KiB/s wr, 28 op/s 2023-03-09T21:49:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:49:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24661.334602 secs 2023-03-09T21:49:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:49:11 smithi181 ceph-mon[94189]: pgmap v13663: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 532 KiB/s wr, 28 op/s 2023-03-09T21:49:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:49:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24661.334602 secs 2023-03-09T21:49:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:49:11 smithi038 ceph-mon[92422]: pgmap v13663: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 532 KiB/s wr, 28 op/s 2023-03-09T21:49:13.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25125.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:49:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:49:13 smithi122 ceph-mon[93869]: pgmap v13664: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 436 KiB/s wr, 23 op/s 2023-03-09T21:49:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:49:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24663.687646 secs 2023-03-09T21:49:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:49:13 smithi181 ceph-mon[94189]: pgmap v13664: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 436 KiB/s wr, 23 op/s 2023-03-09T21:49:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:49:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24663.687646 secs 2023-03-09T21:49:13.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:49:13 smithi038 ceph-mon[92422]: pgmap v13664: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 436 KiB/s wr, 23 op/s 2023-03-09T21:49:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:49:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24663.687646 secs 2023-03-09T21:49:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:49:15 smithi122 ceph-mon[93869]: pgmap v13665: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 391 KiB/s wr, 10 op/s 2023-03-09T21:49:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:49:15 smithi122 ceph-mon[93869]: osdmap e1068: 12 total, 12 up, 12 in 2023-03-09T21:49:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:49:15 smithi181 ceph-mon[94189]: pgmap v13665: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 391 KiB/s wr, 10 op/s 2023-03-09T21:49:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:49:15 smithi181 ceph-mon[94189]: osdmap e1068: 12 total, 12 up, 12 in 2023-03-09T21:49:15.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:49:15 smithi038 ceph-mon[92422]: pgmap v13665: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 391 KiB/s wr, 10 op/s 2023-03-09T21:49:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:49:15 smithi038 ceph-mon[92422]: osdmap e1068: 12 total, 12 up, 12 in 2023-03-09T21:49:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:49:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24666.334662 secs 2023-03-09T21:49:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:49:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24666.334662 secs 2023-03-09T21:49:16.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:49:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24666.334662 secs 2023-03-09T21:49:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:49:17 smithi122 ceph-mon[93869]: pgmap v13667: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 391 KiB/s wr, 10 op/s 2023-03-09T21:49:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:49:17 smithi181 ceph-mon[94189]: pgmap v13667: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 391 KiB/s wr, 10 op/s 2023-03-09T21:49:17.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:49:17 smithi038 ceph-mon[92422]: pgmap v13667: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 391 KiB/s wr, 10 op/s 2023-03-09T21:49:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25130.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:49:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:49:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24668.687790 secs 2023-03-09T21:49:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:49:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24668.687790 secs 2023-03-09T21:49:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:49:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24668.687790 secs 2023-03-09T21:49:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:49:19 smithi122 ceph-mon[93869]: pgmap v13668: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 117 KiB/s wr, 9 op/s 2023-03-09T21:49:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:49:19 smithi181 ceph-mon[94189]: pgmap v13668: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 117 KiB/s wr, 9 op/s 2023-03-09T21:49:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:49:19 smithi038 ceph-mon[92422]: pgmap v13668: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 117 KiB/s wr, 9 op/s 2023-03-09T21:49:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:49:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24671.334722 secs 2023-03-09T21:49:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:49:21 smithi122 ceph-mon[93869]: pgmap v13669: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 24 KiB/s wr, 9 op/s 2023-03-09T21:49:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:49:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24671.334722 secs 2023-03-09T21:49:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:49:21 smithi181 ceph-mon[94189]: pgmap v13669: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 24 KiB/s wr, 9 op/s 2023-03-09T21:49:21.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:49:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24671.334722 secs 2023-03-09T21:49:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:49:21 smithi038 ceph-mon[92422]: pgmap v13669: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 24 KiB/s wr, 9 op/s 2023-03-09T21:49:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25135.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:49:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:49:23 smithi122 ceph-mon[93869]: pgmap v13670: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 48 KiB/s wr, 10 op/s 2023-03-09T21:49:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:49:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24673.687923 secs 2023-03-09T21:49:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:49:23 smithi181 ceph-mon[94189]: pgmap v13670: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 48 KiB/s wr, 10 op/s 2023-03-09T21:49:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:49:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24673.687923 secs 2023-03-09T21:49:23.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:49:23 smithi038 ceph-mon[92422]: pgmap v13670: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 48 KiB/s wr, 10 op/s 2023-03-09T21:49:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:49:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24673.687923 secs 2023-03-09T21:49:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:49:25 smithi122 ceph-mon[93869]: pgmap v13671: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 44 KiB/s wr, 9 op/s 2023-03-09T21:49:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:49:25 smithi181 ceph-mon[94189]: pgmap v13671: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 44 KiB/s wr, 9 op/s 2023-03-09T21:49:25.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:49:25 smithi038 ceph-mon[92422]: pgmap v13671: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 44 KiB/s wr, 9 op/s 2023-03-09T21:49:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:49:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24676.334782 secs 2023-03-09T21:49:26.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:49:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24676.334782 secs 2023-03-09T21:49:26.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:49:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24676.334782 secs 2023-03-09T21:49:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:49:27 smithi122 ceph-mon[93869]: pgmap v13672: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 41 KiB/s wr, 10 op/s 2023-03-09T21:49:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:49:27 smithi181 ceph-mon[94189]: pgmap v13672: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 41 KiB/s wr, 10 op/s 2023-03-09T21:49:27.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:49:27 smithi038 ceph-mon[92422]: pgmap v13672: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 41 KiB/s wr, 10 op/s 2023-03-09T21:49:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25140.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:49:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:49:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24678.688040 secs 2023-03-09T21:49:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:49:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24678.688040 secs 2023-03-09T21:49:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:49:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24678.688040 secs 2023-03-09T21:49:29.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:49:29 smithi181 ceph-mon[94189]: pgmap v13673: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 37 KiB/s wr, 9 op/s 2023-03-09T21:49:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:49:29 smithi038 ceph-mon[92422]: pgmap v13673: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 37 KiB/s wr, 9 op/s 2023-03-09T21:49:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:49:29 smithi122 ceph-mon[93869]: pgmap v13673: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 37 KiB/s wr, 9 op/s 2023-03-09T21:49:31.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:49:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24681.334848 secs 2023-03-09T21:49:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:49:31 smithi038 ceph-mon[92422]: pgmap v13674: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 120 KiB/s wr, 8 op/s 2023-03-09T21:49:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:49:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24681.334848 secs 2023-03-09T21:49:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:49:31 smithi122 ceph-mon[93869]: pgmap v13674: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 120 KiB/s wr, 8 op/s 2023-03-09T21:49:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:49:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24681.334848 secs 2023-03-09T21:49:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:49:31 smithi181 ceph-mon[94189]: pgmap v13674: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 120 KiB/s wr, 8 op/s 2023-03-09T21:49:33.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25145.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:49:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:49:33 smithi038 ceph-mon[92422]: pgmap v13675: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 121 KiB/s wr, 10 op/s 2023-03-09T21:49:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:49:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24683.688156 secs 2023-03-09T21:49:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:49:33 smithi122 ceph-mon[93869]: pgmap v13675: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 121 KiB/s wr, 10 op/s 2023-03-09T21:49:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:49:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24683.688156 secs 2023-03-09T21:49:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:49:33 smithi181 ceph-mon[94189]: pgmap v13675: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 121 KiB/s wr, 10 op/s 2023-03-09T21:49:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:49:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24683.688156 secs 2023-03-09T21:49:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:49:35 smithi122 ceph-mon[93869]: pgmap v13676: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 100 KiB/s wr, 7 op/s 2023-03-09T21:49:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:49:35 smithi181 ceph-mon[94189]: pgmap v13676: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 100 KiB/s wr, 7 op/s 2023-03-09T21:49:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:49:35 smithi038 ceph-mon[92422]: pgmap v13676: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 100 KiB/s wr, 7 op/s 2023-03-09T21:49:37.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:49:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24686.334908 secs 2023-03-09T21:49:37.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:49:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24686.334908 secs 2023-03-09T21:49:37.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:49:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24686.334908 secs 2023-03-09T21:49:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:49:37 smithi122 ceph-mon[93869]: pgmap v13677: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 101 KiB/s wr, 8 op/s 2023-03-09T21:49:38.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:49:37 smithi181 ceph-mon[94189]: pgmap v13677: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 101 KiB/s wr, 8 op/s 2023-03-09T21:49:38.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:49:37 smithi038 ceph-mon[92422]: pgmap v13677: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 101 KiB/s wr, 8 op/s 2023-03-09T21:49:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25150.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:49:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:49:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24688.688275 secs 2023-03-09T21:49:39.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:49:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24688.688275 secs 2023-03-09T21:49:39.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:49:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24688.688275 secs 2023-03-09T21:49:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:49:39 smithi122 ceph-mon[93869]: pgmap v13678: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 101 KiB/s wr, 7 op/s 2023-03-09T21:49:40.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:49:39 smithi181 ceph-mon[94189]: pgmap v13678: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 101 KiB/s wr, 7 op/s 2023-03-09T21:49:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:49:39 smithi038 ceph-mon[92422]: pgmap v13678: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 101 KiB/s wr, 7 op/s 2023-03-09T21:49:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:49:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24691.334968 secs 2023-03-09T21:49:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:49:41 smithi122 ceph-mon[93869]: pgmap v13679: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 264 KiB/s wr, 7 op/s 2023-03-09T21:49:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:49:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24691.334968 secs 2023-03-09T21:49:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:49:41 smithi181 ceph-mon[94189]: pgmap v13679: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 264 KiB/s wr, 7 op/s 2023-03-09T21:49:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:49:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24691.334968 secs 2023-03-09T21:49:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:49:41 smithi038 ceph-mon[92422]: pgmap v13679: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 264 KiB/s wr, 7 op/s 2023-03-09T21:49:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25155.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:49:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:49:43 smithi122 ceph-mon[93869]: pgmap v13680: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 165 KiB/s wr, 8 op/s 2023-03-09T21:49:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:49:43 smithi181 ceph-mon[94189]: pgmap v13680: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 165 KiB/s wr, 8 op/s 2023-03-09T21:49:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:49:43 smithi038 ceph-mon[92422]: pgmap v13680: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 165 KiB/s wr, 8 op/s 2023-03-09T21:49:45.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:49:44 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:49:45.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:49:44 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:49:45.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:49:44 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:49:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:49:45 smithi122 ceph-mon[93869]: pgmap v13681: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 164 KiB/s wr, 7 op/s 2023-03-09T21:49:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:49:45 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:49:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:49:45 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:49:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:49:45 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:49:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:49:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24696.335052 secs 2023-03-09T21:49:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:49:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:49:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:49:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:49:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:49:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:49:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:49:45 smithi181 ceph-mon[94189]: pgmap v13681: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 164 KiB/s wr, 7 op/s 2023-03-09T21:49:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:49:45 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:49:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:49:45 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:49:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:49:45 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:49:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:49:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24696.335052 secs 2023-03-09T21:49:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:49:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:49:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:49:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:49:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:49:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:49:46.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:49:45 smithi038 ceph-mon[92422]: pgmap v13681: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 164 KiB/s wr, 7 op/s 2023-03-09T21:49:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:49:45 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:49:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:49:45 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:49:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:49:45 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:49:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:49:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24696.335052 secs 2023-03-09T21:49:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:49:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:49:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:49:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:49:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:49:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:49:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:49:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:49:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:49:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:49:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:49:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:49:47.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:49:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:49:47.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:49:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:49:47.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:49:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:49:47.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:49:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:49:47.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:49:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:49:47.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:49:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:49:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:49:47 smithi122 ceph-mon[93869]: pgmap v13682: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 313 KiB/s rd, 164 KiB/s wr, 8 op/s 2023-03-09T21:49:48.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:49:47 smithi181 ceph-mon[94189]: pgmap v13682: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 313 KiB/s rd, 164 KiB/s wr, 8 op/s 2023-03-09T21:49:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:49:47 smithi038 ceph-mon[92422]: pgmap v13682: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 313 KiB/s rd, 164 KiB/s wr, 8 op/s 2023-03-09T21:49:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25160.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:49:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:49:49 smithi122 ceph-mon[93869]: pgmap v13683: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 340 KiB/s wr, 8 op/s 2023-03-09T21:49:50.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:49:49 smithi181 ceph-mon[94189]: pgmap v13683: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 340 KiB/s wr, 8 op/s 2023-03-09T21:49:50.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:49:49 smithi038 ceph-mon[92422]: pgmap v13683: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 340 KiB/s wr, 8 op/s 2023-03-09T21:49:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:49:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24701.335116 secs 2023-03-09T21:49:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:49:51 smithi122 ceph-mon[93869]: pgmap v13684: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 523 KiB/s rd, 340 KiB/s wr, 8 op/s 2023-03-09T21:49:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:49:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24701.335116 secs 2023-03-09T21:49:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:49:51 smithi181 ceph-mon[94189]: pgmap v13684: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 523 KiB/s rd, 340 KiB/s wr, 8 op/s 2023-03-09T21:49:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:49:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24701.335116 secs 2023-03-09T21:49:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:49:51 smithi038 ceph-mon[92422]: pgmap v13684: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 523 KiB/s rd, 340 KiB/s wr, 8 op/s 2023-03-09T21:49:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25165.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:49:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:49:53 smithi122 ceph-mon[93869]: pgmap v13685: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 734 KiB/s rd, 484 KiB/s wr, 11 op/s 2023-03-09T21:49:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:49:53 smithi181 ceph-mon[94189]: pgmap v13685: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 734 KiB/s rd, 484 KiB/s wr, 11 op/s 2023-03-09T21:49:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:49:53 smithi038 ceph-mon[92422]: pgmap v13685: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 734 KiB/s rd, 484 KiB/s wr, 11 op/s 2023-03-09T21:49:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:49:55 smithi122 ceph-mon[93869]: pgmap v13686: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 484 KiB/s wr, 10 op/s 2023-03-09T21:49:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:49:55 smithi181 ceph-mon[94189]: pgmap v13686: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 484 KiB/s wr, 10 op/s 2023-03-09T21:49:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:49:55 smithi038 ceph-mon[92422]: pgmap v13686: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 484 KiB/s wr, 10 op/s 2023-03-09T21:49:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:49:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24706.335175 secs 2023-03-09T21:49:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:49:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24706.335175 secs 2023-03-09T21:49:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:49:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24706.335175 secs 2023-03-09T21:49:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:49:57 smithi122 ceph-mon[93869]: pgmap v13687: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 867 KiB/s rd, 815 KiB/s wr, 13 op/s 2023-03-09T21:49:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:49:57 smithi181 ceph-mon[94189]: pgmap v13687: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 867 KiB/s rd, 815 KiB/s wr, 13 op/s 2023-03-09T21:49:57.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:49:57 smithi038 ceph-mon[92422]: pgmap v13687: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 867 KiB/s rd, 815 KiB/s wr, 13 op/s 2023-03-09T21:49:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25170.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:49:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:49:58 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 24708.688806 secs 2023-03-09T21:49:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:49:58 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 24708.688806 secs 2023-03-09T21:49:58.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:49:58 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 24708.688806 secs 2023-03-09T21:49:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:49:59 smithi122 ceph-mon[93869]: pgmap v13688: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 886 KiB/s rd, 835 KiB/s wr, 13 op/s 2023-03-09T21:49:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:49:59 smithi181 ceph-mon[94189]: pgmap v13688: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 886 KiB/s rd, 835 KiB/s wr, 13 op/s 2023-03-09T21:49:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:49:59 smithi038 ceph-mon[92422]: pgmap v13688: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 886 KiB/s rd, 835 KiB/s wr, 13 op/s 2023-03-09T21:50:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:00 smithi122 ceph-mon[93869]: Health detail: HEALTH_WARN 1 MDSs report slow requests 2023-03-09T21:50:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:00 smithi122 ceph-mon[93869]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2023-03-09T21:50:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:00 smithi122 ceph-mon[93869]: mds.h(mds.1): 2 slow requests are blocked > 180 secs 2023-03-09T21:50:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:00 smithi181 ceph-mon[94189]: Health detail: HEALTH_WARN 1 MDSs report slow requests 2023-03-09T21:50:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:00 smithi181 ceph-mon[94189]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2023-03-09T21:50:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:00 smithi181 ceph-mon[94189]: mds.h(mds.1): 2 slow requests are blocked > 180 secs 2023-03-09T21:50:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:00 smithi038 ceph-mon[92422]: Health detail: HEALTH_WARN 1 MDSs report slow requests 2023-03-09T21:50:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:00 smithi038 ceph-mon[92422]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2023-03-09T21:50:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:00 smithi038 ceph-mon[92422]: mds.h(mds.1): 2 slow requests are blocked > 180 secs 2023-03-09T21:50:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24711.335236 secs 2023-03-09T21:50:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:01 smithi122 ceph-mon[93869]: pgmap v13689: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 711 KiB/s rd, 660 KiB/s wr, 12 op/s 2023-03-09T21:50:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24711.335236 secs 2023-03-09T21:50:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:01 smithi181 ceph-mon[94189]: pgmap v13689: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 711 KiB/s rd, 660 KiB/s wr, 12 op/s 2023-03-09T21:50:01.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24711.335236 secs 2023-03-09T21:50:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:01 smithi038 ceph-mon[92422]: pgmap v13689: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 711 KiB/s rd, 660 KiB/s wr, 12 op/s 2023-03-09T21:50:02.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:02 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:50:02.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:02 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:50:02.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:02 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:50:02.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:02 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:50:02.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:02 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:50:02.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:02 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:50:02.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:02 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:50:02.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:02 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:50:02.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:02 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:50:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25175.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:50:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:03 smithi122 ceph-mon[93869]: pgmap v13690: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 724 KiB/s rd, 709 KiB/s wr, 15 op/s 2023-03-09T21:50:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:03 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 24713.688952 secs 2023-03-09T21:50:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:03 smithi181 ceph-mon[94189]: pgmap v13690: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 724 KiB/s rd, 709 KiB/s wr, 15 op/s 2023-03-09T21:50:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:03 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 24713.688952 secs 2023-03-09T21:50:03.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:03 smithi038 ceph-mon[92422]: pgmap v13690: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 724 KiB/s rd, 709 KiB/s wr, 15 op/s 2023-03-09T21:50:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:03 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 24713.688952 secs 2023-03-09T21:50:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:05 smithi122 ceph-mon[93869]: pgmap v13691: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 422 KiB/s wr, 23 op/s 2023-03-09T21:50:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:05 smithi181 ceph-mon[94189]: pgmap v13691: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 422 KiB/s wr, 23 op/s 2023-03-09T21:50:05.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:05 smithi038 ceph-mon[92422]: pgmap v13691: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 422 KiB/s wr, 23 op/s 2023-03-09T21:50:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24716.335296 secs 2023-03-09T21:50:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24716.335296 secs 2023-03-09T21:50:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24716.335296 secs 2023-03-09T21:50:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:07 smithi122 ceph-mon[93869]: pgmap v13692: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 750 KiB/s rd, 783 KiB/s wr, 28 op/s 2023-03-09T21:50:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:07 smithi181 ceph-mon[94189]: pgmap v13692: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 750 KiB/s rd, 783 KiB/s wr, 28 op/s 2023-03-09T21:50:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:07 smithi038 ceph-mon[92422]: pgmap v13692: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 750 KiB/s rd, 783 KiB/s wr, 28 op/s 2023-03-09T21:50:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25180.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:50:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24718.689070 secs 2023-03-09T21:50:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24718.689070 secs 2023-03-09T21:50:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24718.689070 secs 2023-03-09T21:50:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:09 smithi122 ceph-mon[93869]: pgmap v13693: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 471 KiB/s wr, 25 op/s 2023-03-09T21:50:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:09 smithi181 ceph-mon[94189]: pgmap v13693: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 471 KiB/s wr, 25 op/s 2023-03-09T21:50:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:09 smithi038 ceph-mon[92422]: pgmap v13693: 129 pgs: 129 active+clean; 9.2 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 471 KiB/s wr, 25 op/s 2023-03-09T21:50:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:10 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:50:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:10 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:50:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:10 smithi122 ceph-mon[93869]: osdmap e1069: 12 total, 12 up, 12 in 2023-03-09T21:50:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:10 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:50:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:10 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:50:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:10 smithi181 ceph-mon[94189]: osdmap e1069: 12 total, 12 up, 12 in 2023-03-09T21:50:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:10 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:50:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:10 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:50:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:10 smithi038 ceph-mon[92422]: osdmap e1069: 12 total, 12 up, 12 in 2023-03-09T21:50:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24721.335362 secs 2023-03-09T21:50:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:11 smithi122 ceph-mon[93869]: pgmap v13695: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 582 KiB/s rd, 624 KiB/s wr, 29 op/s 2023-03-09T21:50:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24721.335362 secs 2023-03-09T21:50:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:11 smithi181 ceph-mon[94189]: pgmap v13695: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 582 KiB/s rd, 624 KiB/s wr, 29 op/s 2023-03-09T21:50:11.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24721.335362 secs 2023-03-09T21:50:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:11 smithi038 ceph-mon[92422]: pgmap v13695: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 582 KiB/s rd, 624 KiB/s wr, 29 op/s 2023-03-09T21:50:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25185.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:50:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:13 smithi122 ceph-mon[93869]: pgmap v13696: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 525 KiB/s rd, 565 KiB/s wr, 27 op/s 2023-03-09T21:50:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24723.689193 secs 2023-03-09T21:50:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:13 smithi181 ceph-mon[94189]: pgmap v13696: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 525 KiB/s rd, 565 KiB/s wr, 27 op/s 2023-03-09T21:50:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24723.689193 secs 2023-03-09T21:50:13.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:13 smithi038 ceph-mon[92422]: pgmap v13696: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 525 KiB/s rd, 565 KiB/s wr, 27 op/s 2023-03-09T21:50:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24723.689193 secs 2023-03-09T21:50:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:15 smithi122 ceph-mon[93869]: pgmap v13697: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 540 KiB/s wr, 12 op/s 2023-03-09T21:50:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:15 smithi122 ceph-mon[93869]: osdmap e1070: 12 total, 12 up, 12 in 2023-03-09T21:50:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:15 smithi181 ceph-mon[94189]: pgmap v13697: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 540 KiB/s wr, 12 op/s 2023-03-09T21:50:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:15 smithi181 ceph-mon[94189]: osdmap e1070: 12 total, 12 up, 12 in 2023-03-09T21:50:15.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:15 smithi038 ceph-mon[92422]: pgmap v13697: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 540 KiB/s wr, 12 op/s 2023-03-09T21:50:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:15 smithi038 ceph-mon[92422]: osdmap e1070: 12 total, 12 up, 12 in 2023-03-09T21:50:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24726.335437 secs 2023-03-09T21:50:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24726.335437 secs 2023-03-09T21:50:16.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24726.335437 secs 2023-03-09T21:50:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:17 smithi122 ceph-mon[93869]: pgmap v13699: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 104 KiB/s wr, 9 op/s 2023-03-09T21:50:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:17 smithi181 ceph-mon[94189]: pgmap v13699: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 104 KiB/s wr, 9 op/s 2023-03-09T21:50:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:17 smithi038 ceph-mon[92422]: pgmap v13699: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 104 KiB/s wr, 9 op/s 2023-03-09T21:50:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25190.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:50:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24728.689309 secs 2023-03-09T21:50:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24728.689309 secs 2023-03-09T21:50:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24728.689309 secs 2023-03-09T21:50:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:19 smithi122 ceph-mon[93869]: pgmap v13700: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 123 KiB/s wr, 10 op/s 2023-03-09T21:50:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:19 smithi181 ceph-mon[94189]: pgmap v13700: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 123 KiB/s wr, 10 op/s 2023-03-09T21:50:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:19 smithi038 ceph-mon[92422]: pgmap v13700: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 123 KiB/s wr, 10 op/s 2023-03-09T21:50:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24731.335498 secs 2023-03-09T21:50:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:21 smithi122 ceph-mon[93869]: pgmap v13701: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 24 KiB/s wr, 8 op/s 2023-03-09T21:50:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24731.335498 secs 2023-03-09T21:50:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:21 smithi181 ceph-mon[94189]: pgmap v13701: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 24 KiB/s wr, 8 op/s 2023-03-09T21:50:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24731.335498 secs 2023-03-09T21:50:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:21 smithi038 ceph-mon[92422]: pgmap v13701: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 24 KiB/s wr, 8 op/s 2023-03-09T21:50:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25195.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:50:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:23 smithi122 ceph-mon[93869]: pgmap v13702: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T21:50:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24733.689439 secs 2023-03-09T21:50:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:23 smithi181 ceph-mon[94189]: pgmap v13702: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T21:50:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24733.689439 secs 2023-03-09T21:50:23.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:23 smithi038 ceph-mon[92422]: pgmap v13702: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T21:50:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24733.689439 secs 2023-03-09T21:50:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:25 smithi122 ceph-mon[93869]: pgmap v13703: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T21:50:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:25 smithi181 ceph-mon[94189]: pgmap v13703: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T21:50:25.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:25 smithi038 ceph-mon[92422]: pgmap v13703: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T21:50:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24736.335584 secs 2023-03-09T21:50:26.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24736.335584 secs 2023-03-09T21:50:26.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24736.335584 secs 2023-03-09T21:50:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:27 smithi122 ceph-mon[93869]: pgmap v13704: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 9 op/s 2023-03-09T21:50:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:27 smithi181 ceph-mon[94189]: pgmap v13704: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 9 op/s 2023-03-09T21:50:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:27 smithi038 ceph-mon[92422]: pgmap v13704: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 23 KiB/s wr, 9 op/s 2023-03-09T21:50:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25200.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:50:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24738.689573 secs 2023-03-09T21:50:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24738.689573 secs 2023-03-09T21:50:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24738.689573 secs 2023-03-09T21:50:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:29 smithi122 ceph-mon[93869]: pgmap v13705: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 40 KiB/s wr, 7 op/s 2023-03-09T21:50:29.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:29 smithi181 ceph-mon[94189]: pgmap v13705: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 40 KiB/s wr, 7 op/s 2023-03-09T21:50:29.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:29 smithi038 ceph-mon[92422]: pgmap v13705: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 40 KiB/s wr, 7 op/s 2023-03-09T21:50:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24741.335646 secs 2023-03-09T21:50:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:31 smithi122 ceph-mon[93869]: pgmap v13706: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 6 op/s 2023-03-09T21:50:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24741.335646 secs 2023-03-09T21:50:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:31 smithi181 ceph-mon[94189]: pgmap v13706: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 6 op/s 2023-03-09T21:50:31.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24741.335646 secs 2023-03-09T21:50:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:31 smithi038 ceph-mon[92422]: pgmap v13706: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 20 KiB/s wr, 6 op/s 2023-03-09T21:50:33.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25205.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:50:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:33 smithi122 ceph-mon[93869]: pgmap v13707: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 21 KiB/s wr, 8 op/s 2023-03-09T21:50:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24743.689680 secs 2023-03-09T21:50:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:33 smithi181 ceph-mon[94189]: pgmap v13707: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 21 KiB/s wr, 8 op/s 2023-03-09T21:50:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24743.689680 secs 2023-03-09T21:50:33.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:33 smithi038 ceph-mon[92422]: pgmap v13707: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 21 KiB/s wr, 8 op/s 2023-03-09T21:50:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24743.689680 secs 2023-03-09T21:50:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:35 smithi122 ceph-mon[93869]: pgmap v13708: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 21 KiB/s wr, 6 op/s 2023-03-09T21:50:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:35 smithi181 ceph-mon[94189]: pgmap v13708: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 21 KiB/s wr, 6 op/s 2023-03-09T21:50:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:35 smithi038 ceph-mon[92422]: pgmap v13708: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 21 KiB/s wr, 6 op/s 2023-03-09T21:50:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24746.335705 secs 2023-03-09T21:50:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24746.335705 secs 2023-03-09T21:50:36.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24746.335705 secs 2023-03-09T21:50:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:37 smithi122 ceph-mon[93869]: pgmap v13709: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 121 KiB/s wr, 9 op/s 2023-03-09T21:50:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:37 smithi181 ceph-mon[94189]: pgmap v13709: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 121 KiB/s wr, 9 op/s 2023-03-09T21:50:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:37 smithi038 ceph-mon[92422]: pgmap v13709: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 121 KiB/s wr, 9 op/s 2023-03-09T21:50:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25210.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:50:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24748.689793 secs 2023-03-09T21:50:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24748.689793 secs 2023-03-09T21:50:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24748.689793 secs 2023-03-09T21:50:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:39 smithi122 ceph-mon[93869]: pgmap v13710: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 120 KiB/s wr, 7 op/s 2023-03-09T21:50:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:39 smithi181 ceph-mon[94189]: pgmap v13710: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 120 KiB/s wr, 7 op/s 2023-03-09T21:50:39.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:39 smithi038 ceph-mon[92422]: pgmap v13710: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 120 KiB/s wr, 7 op/s 2023-03-09T21:50:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24751.335780 secs 2023-03-09T21:50:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:41 smithi122 ceph-mon[93869]: pgmap v13711: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 157 KiB/s wr, 7 op/s 2023-03-09T21:50:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24751.335780 secs 2023-03-09T21:50:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:41 smithi181 ceph-mon[94189]: pgmap v13711: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 157 KiB/s wr, 7 op/s 2023-03-09T21:50:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24751.335780 secs 2023-03-09T21:50:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:41 smithi038 ceph-mon[92422]: pgmap v13711: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 157 KiB/s wr, 7 op/s 2023-03-09T21:50:43.793 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25215.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:50:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:43 smithi122 ceph-mon[93869]: pgmap v13712: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 157 KiB/s wr, 9 op/s 2023-03-09T21:50:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:43 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 24753.689912 secs 2023-03-09T21:50:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:43 smithi181 ceph-mon[94189]: pgmap v13712: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 157 KiB/s wr, 9 op/s 2023-03-09T21:50:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:43 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 24753.689912 secs 2023-03-09T21:50:43.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:43 smithi038 ceph-mon[92422]: pgmap v13712: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 157 KiB/s wr, 9 op/s 2023-03-09T21:50:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:43 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 24753.689912 secs 2023-03-09T21:50:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:45 smithi122 ceph-mon[93869]: pgmap v13713: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 422 KiB/s wr, 8 op/s 2023-03-09T21:50:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:45 smithi181 ceph-mon[94189]: pgmap v13713: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 422 KiB/s wr, 8 op/s 2023-03-09T21:50:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:45 smithi038 ceph-mon[92422]: pgmap v13713: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 422 KiB/s wr, 8 op/s 2023-03-09T21:50:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24756.335844 secs 2023-03-09T21:50:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:46 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:50:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:46 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:50:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:50:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24756.335844 secs 2023-03-09T21:50:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:46 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:50:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:46 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:50:47.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:50:47.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24756.335844 secs 2023-03-09T21:50:47.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:46 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:50:47.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:46 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:50:47.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:50:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:47 smithi122 ceph-mon[93869]: pgmap v13714: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 421 KiB/s wr, 10 op/s 2023-03-09T21:50:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:50:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:50:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:50:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:50:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:50:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:50:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:50:48.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:47 smithi181 ceph-mon[94189]: pgmap v13714: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 421 KiB/s wr, 10 op/s 2023-03-09T21:50:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:50:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:50:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:50:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:50:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:50:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:50:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:50:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:47 smithi038 ceph-mon[92422]: pgmap v13714: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 421 KiB/s wr, 10 op/s 2023-03-09T21:50:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:50:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:50:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:50:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:50:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:50:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:50:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:50:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25220.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:50:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24758.690002 secs 2023-03-09T21:50:49.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24758.690002 secs 2023-03-09T21:50:49.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24758.690002 secs 2023-03-09T21:50:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:49 smithi122 ceph-mon[93869]: pgmap v13715: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 508 KiB/s rd, 437 KiB/s wr, 9 op/s 2023-03-09T21:50:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:50:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:50:50.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:49 smithi181 ceph-mon[94189]: pgmap v13715: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 508 KiB/s rd, 437 KiB/s wr, 9 op/s 2023-03-09T21:50:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:50:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:50:50.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:49 smithi038 ceph-mon[92422]: pgmap v13715: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 508 KiB/s rd, 437 KiB/s wr, 9 op/s 2023-03-09T21:50:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:50:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:50:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24761.335905 secs 2023-03-09T21:50:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:51 smithi122 ceph-mon[93869]: pgmap v13716: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 541 KiB/s rd, 470 KiB/s wr, 10 op/s 2023-03-09T21:50:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24761.335905 secs 2023-03-09T21:50:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:51 smithi181 ceph-mon[94189]: pgmap v13716: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 541 KiB/s rd, 470 KiB/s wr, 10 op/s 2023-03-09T21:50:52.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24761.335905 secs 2023-03-09T21:50:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:51 smithi038 ceph-mon[92422]: pgmap v13716: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 541 KiB/s rd, 470 KiB/s wr, 10 op/s 2023-03-09T21:50:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25225.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:50:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:53 smithi122 ceph-mon[93869]: pgmap v13717: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 415 KiB/s wr, 12 op/s 2023-03-09T21:50:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24763.690138 secs 2023-03-09T21:50:54.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:53 smithi181 ceph-mon[94189]: pgmap v13717: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 415 KiB/s wr, 12 op/s 2023-03-09T21:50:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24763.690138 secs 2023-03-09T21:50:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:53 smithi038 ceph-mon[92422]: pgmap v13717: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 415 KiB/s wr, 12 op/s 2023-03-09T21:50:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24763.690138 secs 2023-03-09T21:50:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:55 smithi122 ceph-mon[93869]: pgmap v13718: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 847 KiB/s rd, 864 KiB/s wr, 11 op/s 2023-03-09T21:50:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:55 smithi181 ceph-mon[94189]: pgmap v13718: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 847 KiB/s rd, 864 KiB/s wr, 11 op/s 2023-03-09T21:50:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:55 smithi038 ceph-mon[92422]: pgmap v13718: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 847 KiB/s rd, 864 KiB/s wr, 11 op/s 2023-03-09T21:50:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24766.335965 secs 2023-03-09T21:50:57.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24766.335965 secs 2023-03-09T21:50:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24766.335965 secs 2023-03-09T21:50:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:57 smithi122 ceph-mon[93869]: pgmap v13719: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 672 KiB/s rd, 599 KiB/s wr, 12 op/s 2023-03-09T21:50:58.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:57 smithi181 ceph-mon[94189]: pgmap v13719: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 672 KiB/s rd, 599 KiB/s wr, 12 op/s 2023-03-09T21:50:58.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:57 smithi038 ceph-mon[92422]: pgmap v13719: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 672 KiB/s rd, 599 KiB/s wr, 12 op/s 2023-03-09T21:50:58.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25230.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:50:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24768.690256 secs 2023-03-09T21:50:59.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24768.690256 secs 2023-03-09T21:50:59.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24768.690256 secs 2023-03-09T21:51:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:50:59 smithi122 ceph-mon[93869]: pgmap v13720: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 831 KiB/s rd, 730 KiB/s wr, 12 op/s 2023-03-09T21:51:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:50:59 smithi181 ceph-mon[94189]: pgmap v13720: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 831 KiB/s rd, 730 KiB/s wr, 12 op/s 2023-03-09T21:51:00.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:50:59 smithi038 ceph-mon[92422]: pgmap v13720: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 831 KiB/s rd, 730 KiB/s wr, 12 op/s 2023-03-09T21:51:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:51:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24771.335982 secs 2023-03-09T21:51:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:51:01 smithi122 ceph-mon[93869]: pgmap v13721: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 615 KiB/s rd, 614 KiB/s wr, 10 op/s 2023-03-09T21:51:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:51:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24771.335982 secs 2023-03-09T21:51:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:51:01 smithi181 ceph-mon[94189]: pgmap v13721: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 615 KiB/s rd, 614 KiB/s wr, 10 op/s 2023-03-09T21:51:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:51:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24771.335982 secs 2023-03-09T21:51:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:51:01 smithi038 ceph-mon[92422]: pgmap v13721: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 615 KiB/s rd, 614 KiB/s wr, 10 op/s 2023-03-09T21:51:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25235.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:51:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:51:03 smithi122 ceph-mon[93869]: pgmap v13722: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 709 KiB/s rd, 706 KiB/s wr, 13 op/s 2023-03-09T21:51:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:51:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24773.690371 secs 2023-03-09T21:51:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:51:03 smithi181 ceph-mon[94189]: pgmap v13722: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 709 KiB/s rd, 706 KiB/s wr, 13 op/s 2023-03-09T21:51:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:51:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24773.690371 secs 2023-03-09T21:51:03.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:51:03 smithi038 ceph-mon[92422]: pgmap v13722: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 709 KiB/s rd, 706 KiB/s wr, 13 op/s 2023-03-09T21:51:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:51:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24773.690371 secs 2023-03-09T21:51:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:51:05 smithi122 ceph-mon[93869]: pgmap v13723: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 805 KiB/s rd, 817 KiB/s wr, 23 op/s 2023-03-09T21:51:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:51:05 smithi181 ceph-mon[94189]: pgmap v13723: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 805 KiB/s rd, 817 KiB/s wr, 23 op/s 2023-03-09T21:51:05.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:51:05 smithi038 ceph-mon[92422]: pgmap v13723: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 805 KiB/s rd, 817 KiB/s wr, 23 op/s 2023-03-09T21:51:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:51:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24776.336042 secs 2023-03-09T21:51:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:51:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24776.336042 secs 2023-03-09T21:51:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:51:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24776.336042 secs 2023-03-09T21:51:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:51:07 smithi122 ceph-mon[93869]: pgmap v13724: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 391 KiB/s wr, 26 op/s 2023-03-09T21:51:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:51:07 smithi181 ceph-mon[94189]: pgmap v13724: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 391 KiB/s wr, 26 op/s 2023-03-09T21:51:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:51:07 smithi038 ceph-mon[92422]: pgmap v13724: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 391 KiB/s wr, 26 op/s 2023-03-09T21:51:08.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25240.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:51:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:51:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24778.690473 secs 2023-03-09T21:51:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:51:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24778.690473 secs 2023-03-09T21:51:08.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:51:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24778.690473 secs 2023-03-09T21:51:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:51:09 smithi122 ceph-mon[93869]: pgmap v13725: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 457 KiB/s rd, 459 KiB/s wr, 25 op/s 2023-03-09T21:51:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:51:09 smithi181 ceph-mon[94189]: pgmap v13725: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 457 KiB/s rd, 459 KiB/s wr, 25 op/s 2023-03-09T21:51:09.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:51:09 smithi038 ceph-mon[92422]: pgmap v13725: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 457 KiB/s rd, 459 KiB/s wr, 25 op/s 2023-03-09T21:51:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:51:10 smithi122 ceph-mon[93869]: osdmap e1071: 12 total, 12 up, 12 in 2023-03-09T21:51:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:51:10 smithi181 ceph-mon[94189]: osdmap e1071: 12 total, 12 up, 12 in 2023-03-09T21:51:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:51:10 smithi038 ceph-mon[92422]: osdmap e1071: 12 total, 12 up, 12 in 2023-03-09T21:51:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:51:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24781.336107 secs 2023-03-09T21:51:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:51:11 smithi122 ceph-mon[93869]: pgmap v13727: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 395 KiB/s wr, 28 op/s 2023-03-09T21:51:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:51:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24781.336107 secs 2023-03-09T21:51:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:51:11 smithi181 ceph-mon[94189]: pgmap v13727: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 395 KiB/s wr, 28 op/s 2023-03-09T21:51:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:51:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24781.336107 secs 2023-03-09T21:51:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:51:11 smithi038 ceph-mon[92422]: pgmap v13727: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 395 KiB/s wr, 28 op/s 2023-03-09T21:51:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25245.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:51:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:51:13 smithi122 ceph-mon[93869]: pgmap v13728: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 522 KiB/s wr, 27 op/s 2023-03-09T21:51:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:51:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24783.690550 secs 2023-03-09T21:51:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:51:13 smithi181 ceph-mon[94189]: pgmap v13728: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 522 KiB/s wr, 27 op/s 2023-03-09T21:51:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:51:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24783.690550 secs 2023-03-09T21:51:13.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:51:13 smithi038 ceph-mon[92422]: pgmap v13728: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 522 KiB/s wr, 27 op/s 2023-03-09T21:51:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:51:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24783.690550 secs 2023-03-09T21:51:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:51:15 smithi122 ceph-mon[93869]: pgmap v13729: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 388 KiB/s wr, 11 op/s 2023-03-09T21:51:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:51:15 smithi122 ceph-mon[93869]: osdmap e1072: 12 total, 12 up, 12 in 2023-03-09T21:51:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:51:15 smithi181 ceph-mon[94189]: pgmap v13729: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 388 KiB/s wr, 11 op/s 2023-03-09T21:51:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:51:15 smithi181 ceph-mon[94189]: osdmap e1072: 12 total, 12 up, 12 in 2023-03-09T21:51:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:51:15 smithi038 ceph-mon[92422]: pgmap v13729: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 388 KiB/s wr, 11 op/s 2023-03-09T21:51:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:51:15 smithi038 ceph-mon[92422]: osdmap e1072: 12 total, 12 up, 12 in 2023-03-09T21:51:16.924 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:51:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24786.336167 secs 2023-03-09T21:51:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:51:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24786.336167 secs 2023-03-09T21:51:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:51:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24786.336167 secs 2023-03-09T21:51:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:51:17 smithi122 ceph-mon[93869]: pgmap v13731: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 349 KiB/s wr, 8 op/s 2023-03-09T21:51:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:51:17 smithi181 ceph-mon[94189]: pgmap v13731: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 349 KiB/s wr, 8 op/s 2023-03-09T21:51:17.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:51:17 smithi038 ceph-mon[92422]: pgmap v13731: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 349 KiB/s wr, 8 op/s 2023-03-09T21:51:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25250.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:51:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:51:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24788.690618 secs 2023-03-09T21:51:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:51:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24788.690618 secs 2023-03-09T21:51:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:51:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24788.690618 secs 2023-03-09T21:51:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:51:19 smithi122 ceph-mon[93869]: pgmap v13732: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 445 KiB/s wr, 10 op/s 2023-03-09T21:51:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:51:19 smithi181 ceph-mon[94189]: pgmap v13732: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 445 KiB/s wr, 10 op/s 2023-03-09T21:51:20.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:51:19 smithi038 ceph-mon[92422]: pgmap v13732: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 445 KiB/s wr, 10 op/s 2023-03-09T21:51:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:51:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24791.336227 secs 2023-03-09T21:51:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:51:21 smithi122 ceph-mon[93869]: pgmap v13733: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 385 KiB/s wr, 9 op/s 2023-03-09T21:51:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:51:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24791.336227 secs 2023-03-09T21:51:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:51:21 smithi181 ceph-mon[94189]: pgmap v13733: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 385 KiB/s wr, 9 op/s 2023-03-09T21:51:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:51:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24791.336227 secs 2023-03-09T21:51:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:51:21 smithi038 ceph-mon[92422]: pgmap v13733: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 385 KiB/s wr, 9 op/s 2023-03-09T21:51:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25255.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:51:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:51:23 smithi122 ceph-mon[93869]: pgmap v13734: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 130 KiB/s wr, 9 op/s 2023-03-09T21:51:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:51:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24793.690676 secs 2023-03-09T21:51:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:51:23 smithi181 ceph-mon[94189]: pgmap v13734: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 130 KiB/s wr, 9 op/s 2023-03-09T21:51:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:51:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24793.690676 secs 2023-03-09T21:51:23.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:51:23 smithi038 ceph-mon[92422]: pgmap v13734: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 130 KiB/s wr, 9 op/s 2023-03-09T21:51:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:51:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24793.690676 secs 2023-03-09T21:51:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:51:25 smithi122 ceph-mon[93869]: pgmap v13735: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 130 KiB/s wr, 9 op/s 2023-03-09T21:51:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:51:25 smithi181 ceph-mon[94189]: pgmap v13735: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 130 KiB/s wr, 9 op/s 2023-03-09T21:51:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:51:25 smithi038 ceph-mon[92422]: pgmap v13735: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 130 KiB/s wr, 9 op/s 2023-03-09T21:51:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:51:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24796.336285 secs 2023-03-09T21:51:26.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:51:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24796.336285 secs 2023-03-09T21:51:26.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:51:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24796.336285 secs 2023-03-09T21:51:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:51:27 smithi122 ceph-mon[93869]: pgmap v13736: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 123 KiB/s wr, 10 op/s 2023-03-09T21:51:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:51:27 smithi181 ceph-mon[94189]: pgmap v13736: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 123 KiB/s wr, 10 op/s 2023-03-09T21:51:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:51:27 smithi038 ceph-mon[92422]: pgmap v13736: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 123 KiB/s wr, 10 op/s 2023-03-09T21:51:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25260.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:51:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:51:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24798.690805 secs 2023-03-09T21:51:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:51:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24798.690805 secs 2023-03-09T21:51:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:51:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24798.690805 secs 2023-03-09T21:51:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:51:29 smithi122 ceph-mon[93869]: pgmap v13737: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 109 KiB/s wr, 9 op/s 2023-03-09T21:51:29.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:51:29 smithi181 ceph-mon[94189]: pgmap v13737: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 109 KiB/s wr, 9 op/s 2023-03-09T21:51:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:51:29 smithi038 ceph-mon[92422]: pgmap v13737: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 109 KiB/s wr, 9 op/s 2023-03-09T21:51:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:51:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24801.336345 secs 2023-03-09T21:51:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:51:31 smithi122 ceph-mon[93869]: pgmap v13738: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 21 KiB/s wr, 7 op/s 2023-03-09T21:51:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:51:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24801.336345 secs 2023-03-09T21:51:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:51:31 smithi181 ceph-mon[94189]: pgmap v13738: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 21 KiB/s wr, 7 op/s 2023-03-09T21:51:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:51:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24801.336345 secs 2023-03-09T21:51:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:51:31 smithi038 ceph-mon[92422]: pgmap v13738: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 21 KiB/s wr, 7 op/s 2023-03-09T21:51:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25265.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:51:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:51:33 smithi122 ceph-mon[93869]: pgmap v13739: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 217 KiB/s wr, 11 op/s 2023-03-09T21:51:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:51:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24803.690942 secs 2023-03-09T21:51:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:51:33 smithi181 ceph-mon[94189]: pgmap v13739: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 217 KiB/s wr, 11 op/s 2023-03-09T21:51:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:51:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24803.690942 secs 2023-03-09T21:51:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:51:33 smithi038 ceph-mon[92422]: pgmap v13739: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 217 KiB/s wr, 11 op/s 2023-03-09T21:51:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:51:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24803.690942 secs 2023-03-09T21:51:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:51:35 smithi122 ceph-mon[93869]: pgmap v13740: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 197 KiB/s wr, 7 op/s 2023-03-09T21:51:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:51:35 smithi181 ceph-mon[94189]: pgmap v13740: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 197 KiB/s wr, 7 op/s 2023-03-09T21:51:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:51:35 smithi038 ceph-mon[92422]: pgmap v13740: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 197 KiB/s wr, 7 op/s 2023-03-09T21:51:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:51:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24806.336428 secs 2023-03-09T21:51:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:51:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24806.336428 secs 2023-03-09T21:51:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:51:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24806.336428 secs 2023-03-09T21:51:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:51:37 smithi122 ceph-mon[93869]: pgmap v13741: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 197 KiB/s wr, 9 op/s 2023-03-09T21:51:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:51:37 smithi181 ceph-mon[94189]: pgmap v13741: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 197 KiB/s wr, 9 op/s 2023-03-09T21:51:37.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:51:37 smithi038 ceph-mon[92422]: pgmap v13741: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 197 KiB/s wr, 9 op/s 2023-03-09T21:51:38.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25270.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:51:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:51:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24808.691060 secs 2023-03-09T21:51:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:51:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24808.691060 secs 2023-03-09T21:51:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:51:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24808.691060 secs 2023-03-09T21:51:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:51:39 smithi122 ceph-mon[93869]: pgmap v13742: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 196 KiB/s wr, 8 op/s 2023-03-09T21:51:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:51:39 smithi181 ceph-mon[94189]: pgmap v13742: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 196 KiB/s wr, 8 op/s 2023-03-09T21:51:39.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:51:39 smithi038 ceph-mon[92422]: pgmap v13742: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 196 KiB/s wr, 8 op/s 2023-03-09T21:51:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:51:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24811.336489 secs 2023-03-09T21:51:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:51:41 smithi122 ceph-mon[93869]: pgmap v13743: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 196 KiB/s wr, 6 op/s 2023-03-09T21:51:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:51:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24811.336489 secs 2023-03-09T21:51:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:51:41 smithi181 ceph-mon[94189]: pgmap v13743: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 196 KiB/s wr, 6 op/s 2023-03-09T21:51:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:51:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24811.336489 secs 2023-03-09T21:51:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:51:41 smithi038 ceph-mon[92422]: pgmap v13743: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 196 KiB/s wr, 6 op/s 2023-03-09T21:51:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25275.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:51:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:51:43 smithi122 ceph-mon[93869]: pgmap v13744: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 307 KiB/s wr, 9 op/s 2023-03-09T21:51:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:51:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24813.691178 secs 2023-03-09T21:51:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:51:43 smithi181 ceph-mon[94189]: pgmap v13744: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 307 KiB/s wr, 9 op/s 2023-03-09T21:51:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:51:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24813.691178 secs 2023-03-09T21:51:43.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:51:43 smithi038 ceph-mon[92422]: pgmap v13744: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 307 KiB/s wr, 9 op/s 2023-03-09T21:51:43.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:51:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24813.691178 secs 2023-03-09T21:51:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:51:45 smithi122 ceph-mon[93869]: pgmap v13745: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 111 KiB/s wr, 6 op/s 2023-03-09T21:51:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:51:45 smithi181 ceph-mon[94189]: pgmap v13745: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 111 KiB/s wr, 6 op/s 2023-03-09T21:51:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:51:45 smithi038 ceph-mon[92422]: pgmap v13745: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 111 KiB/s wr, 6 op/s 2023-03-09T21:51:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:51:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24816.336545 secs 2023-03-09T21:51:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:51:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24816.336545 secs 2023-03-09T21:51:46.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:51:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24816.336545 secs 2023-03-09T21:51:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:51:47 smithi122 ceph-mon[93869]: pgmap v13746: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 618 KiB/s rd, 581 KiB/s wr, 10 op/s 2023-03-09T21:51:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:51:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:51:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:51:47 smithi181 ceph-mon[94189]: pgmap v13746: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 618 KiB/s rd, 581 KiB/s wr, 10 op/s 2023-03-09T21:51:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:51:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:51:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:51:47 smithi038 ceph-mon[92422]: pgmap v13746: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 618 KiB/s rd, 581 KiB/s wr, 10 op/s 2023-03-09T21:51:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:51:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:51:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25280.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:51:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:51:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:51:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:51:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:51:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:51:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:51:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:51:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:51:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:51:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:51:48.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:51:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:51:48.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:51:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:51:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:51:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:51:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:51:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:51:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:51:49 smithi122 ceph-mon[93869]: pgmap v13747: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 638 KiB/s rd, 603 KiB/s wr, 10 op/s 2023-03-09T21:51:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:51:49 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:51:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:51:49 smithi181 ceph-mon[94189]: pgmap v13747: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 638 KiB/s rd, 603 KiB/s wr, 10 op/s 2023-03-09T21:51:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:51:49 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:51:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:51:49 smithi038 ceph-mon[92422]: pgmap v13747: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 638 KiB/s rd, 603 KiB/s wr, 10 op/s 2023-03-09T21:51:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:51:49 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:51:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:51:50 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:51:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:51:50 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:51:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:51:50 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:51:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:51:50 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:51:50.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:51:50 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:51:50.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:51:50 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:51:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:51:50 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:51:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:51:50 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:51:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:51:50 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:51:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:51:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24821.336645 secs 2023-03-09T21:51:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:51:51 smithi122 ceph-mon[93869]: pgmap v13748: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 649 KiB/s rd, 602 KiB/s wr, 9 op/s 2023-03-09T21:51:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:51:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24821.336645 secs 2023-03-09T21:51:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:51:51 smithi181 ceph-mon[94189]: pgmap v13748: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 649 KiB/s rd, 602 KiB/s wr, 9 op/s 2023-03-09T21:51:51.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:51:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24821.336645 secs 2023-03-09T21:51:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:51:51 smithi038 ceph-mon[92422]: pgmap v13748: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 649 KiB/s rd, 602 KiB/s wr, 9 op/s 2023-03-09T21:51:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25285.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:51:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:51:53 smithi038 ceph-mon[92422]: pgmap v13749: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 679 KiB/s rd, 632 KiB/s wr, 13 op/s 2023-03-09T21:51:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:51:53 smithi122 ceph-mon[93869]: pgmap v13749: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 679 KiB/s rd, 632 KiB/s wr, 13 op/s 2023-03-09T21:51:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:51:53 smithi181 ceph-mon[94189]: pgmap v13749: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 679 KiB/s rd, 632 KiB/s wr, 13 op/s 2023-03-09T21:51:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:51:55 smithi038 ceph-mon[92422]: pgmap v13750: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 559 KiB/s rd, 546 KiB/s wr, 10 op/s 2023-03-09T21:51:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:51:55 smithi122 ceph-mon[93869]: pgmap v13750: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 559 KiB/s rd, 546 KiB/s wr, 10 op/s 2023-03-09T21:51:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:51:55 smithi181 ceph-mon[94189]: pgmap v13750: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 559 KiB/s rd, 546 KiB/s wr, 10 op/s 2023-03-09T21:51:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:51:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24826.336702 secs 2023-03-09T21:51:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:51:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24826.336702 secs 2023-03-09T21:51:57.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:51:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24826.336702 secs 2023-03-09T21:51:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:51:57 smithi122 ceph-mon[93869]: pgmap v13751: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 900 KiB/s rd, 885 KiB/s wr, 14 op/s 2023-03-09T21:51:58.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:51:57 smithi181 ceph-mon[94189]: pgmap v13751: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 900 KiB/s rd, 885 KiB/s wr, 14 op/s 2023-03-09T21:51:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:51:57 smithi038 ceph-mon[92422]: pgmap v13751: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 900 KiB/s rd, 885 KiB/s wr, 14 op/s 2023-03-09T21:51:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25290.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:52:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:51:59 smithi122 ceph-mon[93869]: pgmap v13752: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 429 KiB/s wr, 12 op/s 2023-03-09T21:52:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:51:59 smithi181 ceph-mon[94189]: pgmap v13752: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 429 KiB/s wr, 12 op/s 2023-03-09T21:52:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:51:59 smithi038 ceph-mon[92422]: pgmap v13752: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 429 KiB/s wr, 12 op/s 2023-03-09T21:52:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:52:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24831.336762 secs 2023-03-09T21:52:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:52:01 smithi122 ceph-mon[93869]: pgmap v13753: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 407 KiB/s wr, 11 op/s 2023-03-09T21:52:02.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:52:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24831.336762 secs 2023-03-09T21:52:02.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:52:01 smithi181 ceph-mon[94189]: pgmap v13753: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 407 KiB/s wr, 11 op/s 2023-03-09T21:52:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:52:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24831.336762 secs 2023-03-09T21:52:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:52:01 smithi038 ceph-mon[92422]: pgmap v13753: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 407 KiB/s wr, 11 op/s 2023-03-09T21:52:03.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25295.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:52:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:52:03 smithi122 ceph-mon[93869]: pgmap v13754: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 574 KiB/s rd, 518 KiB/s wr, 14 op/s 2023-03-09T21:52:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:52:03 smithi181 ceph-mon[94189]: pgmap v13754: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 574 KiB/s rd, 518 KiB/s wr, 14 op/s 2023-03-09T21:52:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:52:03 smithi038 ceph-mon[92422]: pgmap v13754: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 574 KiB/s rd, 518 KiB/s wr, 14 op/s 2023-03-09T21:52:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:52:05 smithi122 ceph-mon[93869]: pgmap v13755: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 932 KiB/s rd, 890 KiB/s wr, 24 op/s 2023-03-09T21:52:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:52:05 smithi181 ceph-mon[94189]: pgmap v13755: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 932 KiB/s rd, 890 KiB/s wr, 24 op/s 2023-03-09T21:52:06.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:52:05 smithi038 ceph-mon[92422]: pgmap v13755: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 932 KiB/s rd, 890 KiB/s wr, 24 op/s 2023-03-09T21:52:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:52:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24836.336820 secs 2023-03-09T21:52:07.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:52:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24836.336820 secs 2023-03-09T21:52:07.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:52:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24836.336820 secs 2023-03-09T21:52:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:52:07 smithi122 ceph-mon[93869]: pgmap v13756: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 910 KiB/s rd, 887 KiB/s wr, 28 op/s 2023-03-09T21:52:08.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:52:07 smithi181 ceph-mon[94189]: pgmap v13756: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 910 KiB/s rd, 887 KiB/s wr, 28 op/s 2023-03-09T21:52:08.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:52:07 smithi038 ceph-mon[92422]: pgmap v13756: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 910 KiB/s rd, 887 KiB/s wr, 28 op/s 2023-03-09T21:52:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25300.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:52:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:52:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24838.691813 secs 2023-03-09T21:52:09.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:52:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24838.691813 secs 2023-03-09T21:52:09.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:52:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24838.691813 secs 2023-03-09T21:52:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:52:09 smithi122 ceph-mon[93869]: pgmap v13757: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 724 KiB/s rd, 703 KiB/s wr, 26 op/s 2023-03-09T21:52:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:52:09 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:52:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:52:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:52:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:52:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:52:10.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:52:09 smithi181 ceph-mon[94189]: pgmap v13757: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 724 KiB/s rd, 703 KiB/s wr, 26 op/s 2023-03-09T21:52:10.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:52:09 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:52:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:52:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:52:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:52:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:52:10.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:52:09 smithi038 ceph-mon[92422]: pgmap v13757: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 724 KiB/s rd, 703 KiB/s wr, 26 op/s 2023-03-09T21:52:10.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:52:09 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:52:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:52:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:52:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:52:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:52:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:52:10 smithi122 ceph-mon[93869]: osdmap e1073: 12 total, 12 up, 12 in 2023-03-09T21:52:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:52:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24841.339988 secs 2023-03-09T21:52:11.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:52:10 smithi181 ceph-mon[94189]: osdmap e1073: 12 total, 12 up, 12 in 2023-03-09T21:52:11.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:52:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24841.339988 secs 2023-03-09T21:52:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:52:10 smithi038 ceph-mon[92422]: osdmap e1073: 12 total, 12 up, 12 in 2023-03-09T21:52:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:52:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24841.339988 secs 2023-03-09T21:52:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:52:11 smithi122 ceph-mon[93869]: pgmap v13759: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 853 KiB/s rd, 828 KiB/s wr, 30 op/s 2023-03-09T21:52:12.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:52:11 smithi181 ceph-mon[94189]: pgmap v13759: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 853 KiB/s rd, 828 KiB/s wr, 30 op/s 2023-03-09T21:52:12.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:52:11 smithi038 ceph-mon[92422]: pgmap v13759: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 853 KiB/s rd, 828 KiB/s wr, 30 op/s 2023-03-09T21:52:13.801 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25305.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:52:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:52:13 smithi122 ceph-mon[93869]: pgmap v13760: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 875 KiB/s rd, 916 KiB/s wr, 29 op/s 2023-03-09T21:52:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:52:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24843.691891 secs 2023-03-09T21:52:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:52:13 smithi181 ceph-mon[94189]: pgmap v13760: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 875 KiB/s rd, 916 KiB/s wr, 29 op/s 2023-03-09T21:52:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:52:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24843.691891 secs 2023-03-09T21:52:14.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:52:13 smithi038 ceph-mon[92422]: pgmap v13760: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 875 KiB/s rd, 916 KiB/s wr, 29 op/s 2023-03-09T21:52:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:52:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24843.691891 secs 2023-03-09T21:52:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:52:15 smithi122 ceph-mon[93869]: pgmap v13761: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 434 KiB/s wr, 13 op/s 2023-03-09T21:52:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:52:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24846.340048 secs 2023-03-09T21:52:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:52:15 smithi122 ceph-mon[93869]: osdmap e1074: 12 total, 12 up, 12 in 2023-03-09T21:52:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:52:15 smithi181 ceph-mon[94189]: pgmap v13761: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 434 KiB/s wr, 13 op/s 2023-03-09T21:52:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:52:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24846.340048 secs 2023-03-09T21:52:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:52:15 smithi181 ceph-mon[94189]: osdmap e1074: 12 total, 12 up, 12 in 2023-03-09T21:52:16.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:52:15 smithi038 ceph-mon[92422]: pgmap v13761: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 434 KiB/s wr, 13 op/s 2023-03-09T21:52:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:52:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24846.340048 secs 2023-03-09T21:52:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:52:15 smithi038 ceph-mon[92422]: osdmap e1074: 12 total, 12 up, 12 in 2023-03-09T21:52:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:52:17 smithi122 ceph-mon[93869]: pgmap v13763: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 526 KiB/s rd, 522 KiB/s wr, 9 op/s 2023-03-09T21:52:18.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:52:17 smithi181 ceph-mon[94189]: pgmap v13763: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 526 KiB/s rd, 522 KiB/s wr, 9 op/s 2023-03-09T21:52:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:52:17 smithi038 ceph-mon[92422]: pgmap v13763: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 526 KiB/s rd, 522 KiB/s wr, 9 op/s 2023-03-09T21:52:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25310.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:52:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:52:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24848.691970 secs 2023-03-09T21:52:19.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:52:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24848.691970 secs 2023-03-09T21:52:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:52:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24848.691970 secs 2023-03-09T21:52:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:52:19 smithi122 ceph-mon[93869]: pgmap v13764: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 503 KiB/s rd, 500 KiB/s wr, 10 op/s 2023-03-09T21:52:20.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:52:19 smithi181 ceph-mon[94189]: pgmap v13764: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 503 KiB/s rd, 500 KiB/s wr, 10 op/s 2023-03-09T21:52:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:52:19 smithi038 ceph-mon[92422]: pgmap v13764: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 503 KiB/s rd, 500 KiB/s wr, 10 op/s 2023-03-09T21:52:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:52:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24851.340106 secs 2023-03-09T21:52:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:52:21 smithi122 ceph-mon[93869]: pgmap v13765: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 422 KiB/s wr, 9 op/s 2023-03-09T21:52:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:52:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24851.340106 secs 2023-03-09T21:52:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:52:21 smithi181 ceph-mon[94189]: pgmap v13765: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 422 KiB/s wr, 9 op/s 2023-03-09T21:52:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:52:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24851.340106 secs 2023-03-09T21:52:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:52:21 smithi038 ceph-mon[92422]: pgmap v13765: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 422 KiB/s wr, 9 op/s 2023-03-09T21:52:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25315.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:52:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:52:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24853.692090 secs 2023-03-09T21:52:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:52:23 smithi122 ceph-mon[93869]: pgmap v13766: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 202 KiB/s wr, 9 op/s 2023-03-09T21:52:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:52:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24853.692090 secs 2023-03-09T21:52:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:52:23 smithi181 ceph-mon[94189]: pgmap v13766: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 202 KiB/s wr, 9 op/s 2023-03-09T21:52:23.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:52:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24853.692090 secs 2023-03-09T21:52:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:52:23 smithi038 ceph-mon[92422]: pgmap v13766: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 202 KiB/s wr, 9 op/s 2023-03-09T21:52:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:52:25 smithi122 ceph-mon[93869]: pgmap v13767: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 202 KiB/s wr, 9 op/s 2023-03-09T21:52:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:52:25 smithi181 ceph-mon[94189]: pgmap v13767: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 202 KiB/s wr, 9 op/s 2023-03-09T21:52:25.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:52:25 smithi038 ceph-mon[92422]: pgmap v13767: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 202 KiB/s wr, 9 op/s 2023-03-09T21:52:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:52:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24856.340165 secs 2023-03-09T21:52:26.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:52:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24856.340165 secs 2023-03-09T21:52:26.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:52:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24856.340165 secs 2023-03-09T21:52:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:52:27 smithi122 ceph-mon[93869]: pgmap v13768: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 189 KiB/s wr, 11 op/s 2023-03-09T21:52:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:52:27 smithi181 ceph-mon[94189]: pgmap v13768: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 189 KiB/s wr, 11 op/s 2023-03-09T21:52:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:52:27 smithi038 ceph-mon[92422]: pgmap v13768: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 189 KiB/s wr, 11 op/s 2023-03-09T21:52:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25320.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:52:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:52:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24858.692211 secs 2023-03-09T21:52:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:52:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24858.692211 secs 2023-03-09T21:52:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:52:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24858.692211 secs 2023-03-09T21:52:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:52:29 smithi122 ceph-mon[93869]: pgmap v13769: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 4.0 KiB/s wr, 8 op/s 2023-03-09T21:52:29.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:52:29 smithi181 ceph-mon[94189]: pgmap v13769: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 4.0 KiB/s wr, 8 op/s 2023-03-09T21:52:29.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:52:29 smithi038 ceph-mon[92422]: pgmap v13769: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 4.0 KiB/s wr, 8 op/s 2023-03-09T21:52:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:52:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24861.340225 secs 2023-03-09T21:52:31.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:52:31 smithi122 ceph-mon[93869]: pgmap v13770: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 7 op/s 2023-03-09T21:52:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:52:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24861.340225 secs 2023-03-09T21:52:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:52:31 smithi181 ceph-mon[94189]: pgmap v13770: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 7 op/s 2023-03-09T21:52:31.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:52:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24861.340225 secs 2023-03-09T21:52:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:52:31 smithi038 ceph-mon[92422]: pgmap v13770: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 7 op/s 2023-03-09T21:52:33.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25325.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:52:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:52:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24863.692323 secs 2023-03-09T21:52:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:52:33 smithi122 ceph-mon[93869]: pgmap v13771: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 511 B/s wr, 9 op/s 2023-03-09T21:52:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:52:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24863.692323 secs 2023-03-09T21:52:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:52:33 smithi181 ceph-mon[94189]: pgmap v13771: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 511 B/s wr, 9 op/s 2023-03-09T21:52:33.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:52:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24863.692323 secs 2023-03-09T21:52:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:52:33 smithi038 ceph-mon[92422]: pgmap v13771: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 511 B/s wr, 9 op/s 2023-03-09T21:52:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:52:35 smithi122 ceph-mon[93869]: pgmap v13772: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 597 B/s wr, 8 op/s 2023-03-09T21:52:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:52:35 smithi181 ceph-mon[94189]: pgmap v13772: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 597 B/s wr, 8 op/s 2023-03-09T21:52:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:52:35 smithi038 ceph-mon[92422]: pgmap v13772: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 597 B/s wr, 8 op/s 2023-03-09T21:52:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:52:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24866.340294 secs 2023-03-09T21:52:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:52:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24866.340294 secs 2023-03-09T21:52:36.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:52:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24866.340294 secs 2023-03-09T21:52:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:52:37 smithi122 ceph-mon[93869]: pgmap v13773: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 101 KiB/s wr, 9 op/s 2023-03-09T21:52:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:52:37 smithi181 ceph-mon[94189]: pgmap v13773: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 101 KiB/s wr, 9 op/s 2023-03-09T21:52:37.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:52:37 smithi038 ceph-mon[92422]: pgmap v13773: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 101 KiB/s wr, 9 op/s 2023-03-09T21:52:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25330.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:52:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:52:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24868.692427 secs 2023-03-09T21:52:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:52:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24868.692427 secs 2023-03-09T21:52:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:52:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24868.692427 secs 2023-03-09T21:52:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:52:39 smithi122 ceph-mon[93869]: pgmap v13774: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 102 KiB/s wr, 8 op/s 2023-03-09T21:52:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:52:39 smithi181 ceph-mon[94189]: pgmap v13774: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 102 KiB/s wr, 8 op/s 2023-03-09T21:52:39.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:52:39 smithi038 ceph-mon[92422]: pgmap v13774: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 102 KiB/s wr, 8 op/s 2023-03-09T21:52:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:52:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24871.340356 secs 2023-03-09T21:52:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:52:41 smithi122 ceph-mon[93869]: pgmap v13775: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 101 KiB/s wr, 7 op/s 2023-03-09T21:52:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:52:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24871.340356 secs 2023-03-09T21:52:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:52:41 smithi181 ceph-mon[94189]: pgmap v13775: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 101 KiB/s wr, 7 op/s 2023-03-09T21:52:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:52:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24871.340356 secs 2023-03-09T21:52:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:52:41 smithi038 ceph-mon[92422]: pgmap v13775: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 101 KiB/s wr, 7 op/s 2023-03-09T21:52:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25335.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:52:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:52:43 smithi122 ceph-mon[93869]: pgmap v13776: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 212 KiB/s wr, 9 op/s 2023-03-09T21:52:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:52:43 smithi181 ceph-mon[94189]: pgmap v13776: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 212 KiB/s wr, 9 op/s 2023-03-09T21:52:43.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:52:43 smithi038 ceph-mon[92422]: pgmap v13776: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 212 KiB/s wr, 9 op/s 2023-03-09T21:52:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:52:45 smithi122 ceph-mon[93869]: pgmap v13777: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 212 KiB/s wr, 7 op/s 2023-03-09T21:52:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:52:45 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:52:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:52:45 smithi181 ceph-mon[94189]: pgmap v13777: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 212 KiB/s wr, 7 op/s 2023-03-09T21:52:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:52:45 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:52:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:52:45 smithi038 ceph-mon[92422]: pgmap v13777: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 212 KiB/s wr, 7 op/s 2023-03-09T21:52:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:52:45 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:52:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:52:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24876.340418 secs 2023-03-09T21:52:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:52:46 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:52:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:52:46 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:52:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:52:46 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:52:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:52:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24876.340418 secs 2023-03-09T21:52:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:52:46 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:52:46.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:52:46 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:52:46.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:52:46 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:52:46.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:52:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24876.340418 secs 2023-03-09T21:52:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:52:46 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:52:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:52:46 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:52:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:52:46 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:52:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:52:47 smithi122 ceph-mon[93869]: pgmap v13778: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 519 KiB/s wr, 9 op/s 2023-03-09T21:52:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:52:47 smithi181 ceph-mon[94189]: pgmap v13778: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 519 KiB/s wr, 9 op/s 2023-03-09T21:52:47.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:52:47 smithi038 ceph-mon[92422]: pgmap v13778: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 519 KiB/s wr, 9 op/s 2023-03-09T21:52:48.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25340.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:52:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:52:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:52:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:52:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:52:48.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:52:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:52:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:52:49 smithi122 ceph-mon[93869]: pgmap v13779: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 438 KiB/s wr, 8 op/s 2023-03-09T21:52:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:52:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:52:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:52:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:52:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:52:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:52:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:52:49 smithi181 ceph-mon[94189]: pgmap v13779: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 438 KiB/s wr, 8 op/s 2023-03-09T21:52:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:52:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:52:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:52:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:52:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:52:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:52:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:52:49 smithi038 ceph-mon[92422]: pgmap v13779: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 438 KiB/s wr, 8 op/s 2023-03-09T21:52:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:52:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:52:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:52:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:52:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:52:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:52:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:52:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24881.340478 secs 2023-03-09T21:52:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:52:51 smithi122 ceph-mon[93869]: pgmap v13780: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 438 KiB/s wr, 7 op/s 2023-03-09T21:52:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:52:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24881.340478 secs 2023-03-09T21:52:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:52:51 smithi181 ceph-mon[94189]: pgmap v13780: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 438 KiB/s wr, 7 op/s 2023-03-09T21:52:51.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:52:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24881.340478 secs 2023-03-09T21:52:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:52:51 smithi038 ceph-mon[92422]: pgmap v13780: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 438 KiB/s wr, 7 op/s 2023-03-09T21:52:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25345.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:52:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:52:53 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 24883.692826 secs 2023-03-09T21:52:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:52:53 smithi122 ceph-mon[93869]: pgmap v13781: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 778 KiB/s rd, 603 KiB/s wr, 12 op/s 2023-03-09T21:52:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:52:53 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 24883.692826 secs 2023-03-09T21:52:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:52:53 smithi181 ceph-mon[94189]: pgmap v13781: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 778 KiB/s rd, 603 KiB/s wr, 12 op/s 2023-03-09T21:52:53.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:52:53 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 24883.692826 secs 2023-03-09T21:52:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:52:53 smithi038 ceph-mon[92422]: pgmap v13781: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 778 KiB/s rd, 603 KiB/s wr, 12 op/s 2023-03-09T21:52:54.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:52:54 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:52:54.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:52:54 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:52:54.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:52:54 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:52:54.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:52:54 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:52:54.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:52:54 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:52:54.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:52:54 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:52:54.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:52:54 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:52:54.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:52:54 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:52:54.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:52:54 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:52:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:52:55 smithi122 ceph-mon[93869]: pgmap v13782: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 612 KiB/s rd, 516 KiB/s wr, 10 op/s 2023-03-09T21:52:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:52:55 smithi181 ceph-mon[94189]: pgmap v13782: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 612 KiB/s rd, 516 KiB/s wr, 10 op/s 2023-03-09T21:52:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:52:55 smithi038 ceph-mon[92422]: pgmap v13782: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 612 KiB/s rd, 516 KiB/s wr, 10 op/s 2023-03-09T21:52:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:52:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24886.340541 secs 2023-03-09T21:52:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:52:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24886.340541 secs 2023-03-09T21:52:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:52:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24886.340541 secs 2023-03-09T21:52:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:52:57 smithi122 ceph-mon[93869]: pgmap v13783: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 613 KiB/s rd, 516 KiB/s wr, 12 op/s 2023-03-09T21:52:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:52:57 smithi181 ceph-mon[94189]: pgmap v13783: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 613 KiB/s rd, 516 KiB/s wr, 12 op/s 2023-03-09T21:52:57.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:52:57 smithi038 ceph-mon[92422]: pgmap v13783: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 613 KiB/s rd, 516 KiB/s wr, 12 op/s 2023-03-09T21:52:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25350.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:52:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:52:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24888.692943 secs 2023-03-09T21:52:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:52:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24888.692943 secs 2023-03-09T21:52:58.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:52:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24888.692943 secs 2023-03-09T21:52:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:52:59 smithi122 ceph-mon[93869]: pgmap v13784: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 613 KiB/s rd, 518 KiB/s wr, 13 op/s 2023-03-09T21:52:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:52:59 smithi181 ceph-mon[94189]: pgmap v13784: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 613 KiB/s rd, 518 KiB/s wr, 13 op/s 2023-03-09T21:52:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:52:59 smithi038 ceph-mon[92422]: pgmap v13784: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 613 KiB/s rd, 518 KiB/s wr, 13 op/s 2023-03-09T21:53:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:53:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24891.340620 secs 2023-03-09T21:53:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:53:01 smithi122 ceph-mon[93869]: pgmap v13785: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 594 KiB/s rd, 498 KiB/s wr, 11 op/s 2023-03-09T21:53:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:53:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24891.340620 secs 2023-03-09T21:53:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:53:01 smithi181 ceph-mon[94189]: pgmap v13785: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 594 KiB/s rd, 498 KiB/s wr, 11 op/s 2023-03-09T21:53:01.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:53:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24891.340620 secs 2023-03-09T21:53:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:53:01 smithi038 ceph-mon[92422]: pgmap v13785: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 594 KiB/s rd, 498 KiB/s wr, 11 op/s 2023-03-09T21:53:02.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:53:02 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:53:02.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:53:02 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:53:02.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:53:02 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:53:02.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:53:02 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:53:02.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:53:02 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:53:02.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:53:02 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:53:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25355.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:53:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:53:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24893.693056 secs 2023-03-09T21:53:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:53:03 smithi122 ceph-mon[93869]: pgmap v13786: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 584 KiB/s rd, 504 KiB/s wr, 15 op/s 2023-03-09T21:53:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:53:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24893.693056 secs 2023-03-09T21:53:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:53:03 smithi181 ceph-mon[94189]: pgmap v13786: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 584 KiB/s rd, 504 KiB/s wr, 15 op/s 2023-03-09T21:53:03.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:53:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24893.693056 secs 2023-03-09T21:53:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:53:03 smithi038 ceph-mon[92422]: pgmap v13786: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 584 KiB/s rd, 504 KiB/s wr, 15 op/s 2023-03-09T21:53:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:53:05 smithi181 ceph-mon[94189]: pgmap v13787: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 570 KiB/s wr, 25 op/s 2023-03-09T21:53:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:53:05 smithi181 ceph-mon[94189]: osdmap e1075: 12 total, 12 up, 12 in 2023-03-09T21:53:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:53:05 smithi038 ceph-mon[92422]: pgmap v13787: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 570 KiB/s wr, 25 op/s 2023-03-09T21:53:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:53:05 smithi038 ceph-mon[92422]: osdmap e1075: 12 total, 12 up, 12 in 2023-03-09T21:53:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:53:05 smithi122 ceph-mon[93869]: pgmap v13787: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 570 KiB/s wr, 25 op/s 2023-03-09T21:53:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:53:05 smithi122 ceph-mon[93869]: osdmap e1075: 12 total, 12 up, 12 in 2023-03-09T21:53:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:53:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24896.340684 secs 2023-03-09T21:53:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:53:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24896.340684 secs 2023-03-09T21:53:07.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:53:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24896.340684 secs 2023-03-09T21:53:07.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:53:07 smithi038 ceph-mon[92422]: pgmap v13789: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 684 KiB/s rd, 725 KiB/s wr, 30 op/s 2023-03-09T21:53:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:53:07 smithi122 ceph-mon[93869]: pgmap v13789: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 684 KiB/s rd, 725 KiB/s wr, 30 op/s 2023-03-09T21:53:08.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:53:07 smithi181 ceph-mon[94189]: pgmap v13789: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 684 KiB/s rd, 725 KiB/s wr, 30 op/s 2023-03-09T21:53:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25360.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:53:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:53:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24898.693194 secs 2023-03-09T21:53:09.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:53:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24898.693194 secs 2023-03-09T21:53:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:53:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24898.693194 secs 2023-03-09T21:53:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:53:09 smithi122 ceph-mon[93869]: pgmap v13790: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 422 KiB/s wr, 28 op/s 2023-03-09T21:53:10.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:53:09 smithi181 ceph-mon[94189]: pgmap v13790: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 422 KiB/s wr, 28 op/s 2023-03-09T21:53:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:53:09 smithi038 ceph-mon[92422]: pgmap v13790: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 422 KiB/s wr, 28 op/s 2023-03-09T21:53:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:53:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24901.340745 secs 2023-03-09T21:53:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:53:11 smithi122 ceph-mon[93869]: pgmap v13791: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 422 KiB/s wr, 28 op/s 2023-03-09T21:53:12.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:53:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24901.340745 secs 2023-03-09T21:53:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:53:11 smithi181 ceph-mon[94189]: pgmap v13791: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 422 KiB/s wr, 28 op/s 2023-03-09T21:53:12.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:53:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24901.340745 secs 2023-03-09T21:53:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:53:11 smithi038 ceph-mon[92422]: pgmap v13791: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 422 KiB/s wr, 28 op/s 2023-03-09T21:53:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25365.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:53:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:53:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24903.693313 secs 2023-03-09T21:53:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:53:13 smithi122 ceph-mon[93869]: pgmap v13792: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 464 KiB/s rd, 498 KiB/s wr, 27 op/s 2023-03-09T21:53:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:53:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24903.693313 secs 2023-03-09T21:53:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:53:13 smithi181 ceph-mon[94189]: pgmap v13792: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 464 KiB/s rd, 498 KiB/s wr, 27 op/s 2023-03-09T21:53:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:53:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24903.693313 secs 2023-03-09T21:53:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:53:13 smithi038 ceph-mon[92422]: pgmap v13792: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 464 KiB/s rd, 498 KiB/s wr, 27 op/s 2023-03-09T21:53:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:53:15 smithi122 ceph-mon[93869]: pgmap v13793: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 221 KiB/s wr, 11 op/s 2023-03-09T21:53:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:53:15 smithi122 ceph-mon[93869]: osdmap e1076: 12 total, 12 up, 12 in 2023-03-09T21:53:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:53:15 smithi181 ceph-mon[94189]: pgmap v13793: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 221 KiB/s wr, 11 op/s 2023-03-09T21:53:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:53:15 smithi181 ceph-mon[94189]: osdmap e1076: 12 total, 12 up, 12 in 2023-03-09T21:53:16.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:53:15 smithi038 ceph-mon[92422]: pgmap v13793: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 221 KiB/s wr, 11 op/s 2023-03-09T21:53:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:53:15 smithi038 ceph-mon[92422]: osdmap e1076: 12 total, 12 up, 12 in 2023-03-09T21:53:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:53:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24906.340807 secs 2023-03-09T21:53:17.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:53:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24906.340807 secs 2023-03-09T21:53:17.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:53:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24906.340807 secs 2023-03-09T21:53:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:53:17 smithi122 ceph-mon[93869]: pgmap v13795: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 151 KiB/s wr, 9 op/s 2023-03-09T21:53:18.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:53:17 smithi181 ceph-mon[94189]: pgmap v13795: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 151 KiB/s wr, 9 op/s 2023-03-09T21:53:18.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:53:17 smithi038 ceph-mon[92422]: pgmap v13795: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 151 KiB/s wr, 9 op/s 2023-03-09T21:53:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25370.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:53:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:53:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24908.693405 secs 2023-03-09T21:53:19.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:53:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24908.693405 secs 2023-03-09T21:53:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:53:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24908.693405 secs 2023-03-09T21:53:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:53:19 smithi122 ceph-mon[93869]: pgmap v13796: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 83 KiB/s wr, 8 op/s 2023-03-09T21:53:20.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:53:19 smithi181 ceph-mon[94189]: pgmap v13796: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 83 KiB/s wr, 8 op/s 2023-03-09T21:53:20.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:53:19 smithi038 ceph-mon[92422]: pgmap v13796: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 83 KiB/s wr, 8 op/s 2023-03-09T21:53:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:53:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24911.340867 secs 2023-03-09T21:53:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:53:21 smithi122 ceph-mon[93869]: pgmap v13797: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 83 KiB/s wr, 8 op/s 2023-03-09T21:53:22.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:53:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24911.340867 secs 2023-03-09T21:53:22.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:53:21 smithi181 ceph-mon[94189]: pgmap v13797: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 83 KiB/s wr, 8 op/s 2023-03-09T21:53:22.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:53:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24911.340867 secs 2023-03-09T21:53:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:53:21 smithi038 ceph-mon[92422]: pgmap v13797: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 83 KiB/s wr, 8 op/s 2023-03-09T21:53:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25375.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:53:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:53:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24913.693525 secs 2023-03-09T21:53:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:53:23 smithi122 ceph-mon[93869]: pgmap v13798: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 204 B/s wr, 7 op/s 2023-03-09T21:53:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:53:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24913.693525 secs 2023-03-09T21:53:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:53:23 smithi181 ceph-mon[94189]: pgmap v13798: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 204 B/s wr, 7 op/s 2023-03-09T21:53:24.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:53:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24913.693525 secs 2023-03-09T21:53:24.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:53:23 smithi038 ceph-mon[92422]: pgmap v13798: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 204 B/s wr, 7 op/s 2023-03-09T21:53:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:53:25 smithi122 ceph-mon[93869]: pgmap v13799: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 409 B/s wr, 7 op/s 2023-03-09T21:53:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:53:25 smithi181 ceph-mon[94189]: pgmap v13799: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 409 B/s wr, 7 op/s 2023-03-09T21:53:26.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:53:25 smithi038 ceph-mon[92422]: pgmap v13799: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 409 B/s wr, 7 op/s 2023-03-09T21:53:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:53:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24916.340926 secs 2023-03-09T21:53:27.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:53:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24916.340926 secs 2023-03-09T21:53:27.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:53:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24916.340926 secs 2023-03-09T21:53:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:53:27 smithi122 ceph-mon[93869]: pgmap v13800: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 478 B/s wr, 9 op/s 2023-03-09T21:53:28.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:53:27 smithi181 ceph-mon[94189]: pgmap v13800: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 478 B/s wr, 9 op/s 2023-03-09T21:53:28.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:53:27 smithi038 ceph-mon[92422]: pgmap v13800: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 478 B/s wr, 9 op/s 2023-03-09T21:53:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25380.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:53:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:53:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24918.693646 secs 2023-03-09T21:53:29.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:53:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24918.693646 secs 2023-03-09T21:53:29.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:53:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24918.693646 secs 2023-03-09T21:53:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:53:29 smithi122 ceph-mon[93869]: pgmap v13801: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 767 B/s wr, 7 op/s 2023-03-09T21:53:30.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:53:29 smithi181 ceph-mon[94189]: pgmap v13801: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 767 B/s wr, 7 op/s 2023-03-09T21:53:30.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:53:29 smithi038 ceph-mon[92422]: pgmap v13801: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 767 B/s wr, 7 op/s 2023-03-09T21:53:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:53:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24921.340985 secs 2023-03-09T21:53:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:53:31 smithi122 ceph-mon[93869]: pgmap v13802: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 853 B/s wr, 6 op/s 2023-03-09T21:53:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:53:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24921.340985 secs 2023-03-09T21:53:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:53:31 smithi181 ceph-mon[94189]: pgmap v13802: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 853 B/s wr, 6 op/s 2023-03-09T21:53:32.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:53:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24921.340985 secs 2023-03-09T21:53:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:53:31 smithi038 ceph-mon[92422]: pgmap v13802: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 853 B/s wr, 6 op/s 2023-03-09T21:53:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25385.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:53:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:53:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24923.693769 secs 2023-03-09T21:53:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:53:33 smithi122 ceph-mon[93869]: pgmap v13803: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 116 KiB/s wr, 9 op/s 2023-03-09T21:53:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:53:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24923.693769 secs 2023-03-09T21:53:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:53:33 smithi181 ceph-mon[94189]: pgmap v13803: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 116 KiB/s wr, 9 op/s 2023-03-09T21:53:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:53:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24923.693769 secs 2023-03-09T21:53:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:53:33 smithi038 ceph-mon[92422]: pgmap v13803: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 116 KiB/s wr, 9 op/s 2023-03-09T21:53:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:53:35 smithi122 ceph-mon[93869]: pgmap v13804: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 116 KiB/s wr, 8 op/s 2023-03-09T21:53:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:53:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24926.341047 secs 2023-03-09T21:53:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:53:35 smithi181 ceph-mon[94189]: pgmap v13804: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 116 KiB/s wr, 8 op/s 2023-03-09T21:53:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:53:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24926.341047 secs 2023-03-09T21:53:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:53:35 smithi038 ceph-mon[92422]: pgmap v13804: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 116 KiB/s wr, 8 op/s 2023-03-09T21:53:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:53:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24926.341047 secs 2023-03-09T21:53:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:53:37 smithi122 ceph-mon[93869]: pgmap v13805: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 226 KiB/s wr, 9 op/s 2023-03-09T21:53:38.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:53:37 smithi181 ceph-mon[94189]: pgmap v13805: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 226 KiB/s wr, 9 op/s 2023-03-09T21:53:38.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:53:37 smithi038 ceph-mon[92422]: pgmap v13805: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 226 KiB/s wr, 9 op/s 2023-03-09T21:53:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25390.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:53:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:53:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24928.693871 secs 2023-03-09T21:53:39.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:53:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24928.693871 secs 2023-03-09T21:53:39.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:53:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24928.693871 secs 2023-03-09T21:53:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:53:39 smithi122 ceph-mon[93869]: pgmap v13806: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 226 KiB/s wr, 7 op/s 2023-03-09T21:53:40.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:53:39 smithi181 ceph-mon[94189]: pgmap v13806: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 226 KiB/s wr, 7 op/s 2023-03-09T21:53:40.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:53:39 smithi038 ceph-mon[92422]: pgmap v13806: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 226 KiB/s wr, 7 op/s 2023-03-09T21:53:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:53:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24931.341111 secs 2023-03-09T21:53:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:53:41 smithi122 ceph-mon[93869]: pgmap v13807: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 225 KiB/s wr, 6 op/s 2023-03-09T21:53:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:53:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24931.341111 secs 2023-03-09T21:53:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:53:41 smithi181 ceph-mon[94189]: pgmap v13807: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 225 KiB/s wr, 6 op/s 2023-03-09T21:53:41.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:53:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24931.341111 secs 2023-03-09T21:53:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:53:41 smithi038 ceph-mon[92422]: pgmap v13807: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 225 KiB/s wr, 6 op/s 2023-03-09T21:53:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25395.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:53:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:53:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24933.693989 secs 2023-03-09T21:53:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:53:43 smithi122 ceph-mon[93869]: pgmap v13808: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 337 KiB/s wr, 9 op/s 2023-03-09T21:53:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:53:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24933.693989 secs 2023-03-09T21:53:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:53:43 smithi181 ceph-mon[94189]: pgmap v13808: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 337 KiB/s wr, 9 op/s 2023-03-09T21:53:43.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:53:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24933.693989 secs 2023-03-09T21:53:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:53:43 smithi038 ceph-mon[92422]: pgmap v13808: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 337 KiB/s wr, 9 op/s 2023-03-09T21:53:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:53:45 smithi122 ceph-mon[93869]: pgmap v13809: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 221 KiB/s wr, 7 op/s 2023-03-09T21:53:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:53:45 smithi181 ceph-mon[94189]: pgmap v13809: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 221 KiB/s wr, 7 op/s 2023-03-09T21:53:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:53:45 smithi038 ceph-mon[92422]: pgmap v13809: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 221 KiB/s wr, 7 op/s 2023-03-09T21:53:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:53:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24936.341172 secs 2023-03-09T21:53:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:53:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24936.341172 secs 2023-03-09T21:53:46.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:53:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24936.341172 secs 2023-03-09T21:53:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:53:47 smithi122 ceph-mon[93869]: pgmap v13810: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 416 KiB/s wr, 9 op/s 2023-03-09T21:53:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:53:47 smithi181 ceph-mon[94189]: pgmap v13810: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 416 KiB/s wr, 9 op/s 2023-03-09T21:53:47.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:53:47 smithi038 ceph-mon[92422]: pgmap v13810: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 416 KiB/s wr, 9 op/s 2023-03-09T21:53:48.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25400.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:53:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:53:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24938.694116 secs 2023-03-09T21:53:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:53:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24938.694116 secs 2023-03-09T21:53:48.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:53:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24938.694116 secs 2023-03-09T21:53:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:53:49 smithi122 ceph-mon[93869]: pgmap v13811: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 326 KiB/s wr, 8 op/s 2023-03-09T21:53:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:53:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:53:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:53:49 smithi181 ceph-mon[94189]: pgmap v13811: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 326 KiB/s wr, 8 op/s 2023-03-09T21:53:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:53:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:53:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:53:49 smithi038 ceph-mon[92422]: pgmap v13811: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 326 KiB/s wr, 8 op/s 2023-03-09T21:53:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:53:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:53:51.871 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:53:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24941.341235 secs 2023-03-09T21:53:51.871 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:53:51 smithi038 ceph-mon[92422]: pgmap v13812: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 326 KiB/s wr, 8 op/s 2023-03-09T21:53:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:53:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24941.341235 secs 2023-03-09T21:53:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:53:51 smithi122 ceph-mon[93869]: pgmap v13812: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 326 KiB/s wr, 8 op/s 2023-03-09T21:53:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:53:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24941.341235 secs 2023-03-09T21:53:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:53:51 smithi181 ceph-mon[94189]: pgmap v13812: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 326 KiB/s wr, 8 op/s 2023-03-09T21:53:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25405.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:53:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:53:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24943.694232 secs 2023-03-09T21:53:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:53:53 smithi122 ceph-mon[93869]: pgmap v13813: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 440 KiB/s rd, 369 KiB/s wr, 11 op/s 2023-03-09T21:53:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:53:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:53:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:53:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:53:53.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:53:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:53:53.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:53:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:53:53.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:53:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:53:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:53:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24943.694232 secs 2023-03-09T21:53:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:53:53 smithi181 ceph-mon[94189]: pgmap v13813: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 440 KiB/s rd, 369 KiB/s wr, 11 op/s 2023-03-09T21:53:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:53:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:53:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:53:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:53:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:53:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:53:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:53:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:53:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:53:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:53:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:53:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24943.694232 secs 2023-03-09T21:53:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:53:53 smithi038 ceph-mon[92422]: pgmap v13813: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 440 KiB/s rd, 369 KiB/s wr, 11 op/s 2023-03-09T21:53:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:53:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:53:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:53:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:53:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:53:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:53:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:53:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:53:54.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:53:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:53:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:53:55 smithi122 ceph-mon[93869]: pgmap v13814: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 278 KiB/s wr, 9 op/s 2023-03-09T21:53:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:53:55 smithi181 ceph-mon[94189]: pgmap v13814: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 278 KiB/s wr, 9 op/s 2023-03-09T21:53:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:53:55 smithi038 ceph-mon[92422]: pgmap v13814: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 278 KiB/s wr, 9 op/s 2023-03-09T21:53:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:53:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24946.341297 secs 2023-03-09T21:53:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:53:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24946.341297 secs 2023-03-09T21:53:56.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:53:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24946.341297 secs 2023-03-09T21:53:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:53:57 smithi122 ceph-mon[93869]: pgmap v13815: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 636 KiB/s rd, 619 KiB/s wr, 12 op/s 2023-03-09T21:53:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:53:57 smithi181 ceph-mon[94189]: pgmap v13815: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 636 KiB/s rd, 619 KiB/s wr, 12 op/s 2023-03-09T21:53:57.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:53:57 smithi038 ceph-mon[92422]: pgmap v13815: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 636 KiB/s rd, 619 KiB/s wr, 12 op/s 2023-03-09T21:53:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25410.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:53:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:53:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24948.694297 secs 2023-03-09T21:53:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:53:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24948.694297 secs 2023-03-09T21:53:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:53:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24948.694297 secs 2023-03-09T21:53:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:53:59 smithi122 ceph-mon[93869]: pgmap v13816: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 445 KiB/s wr, 10 op/s 2023-03-09T21:53:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:53:59 smithi181 ceph-mon[94189]: pgmap v13816: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 445 KiB/s wr, 10 op/s 2023-03-09T21:53:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:53:59 smithi038 ceph-mon[92422]: pgmap v13816: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 445 KiB/s wr, 10 op/s 2023-03-09T21:54:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:54:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24951.341360 secs 2023-03-09T21:54:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:54:01 smithi122 ceph-mon[93869]: pgmap v13817: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 469 KiB/s rd, 453 KiB/s wr, 10 op/s 2023-03-09T21:54:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:54:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24951.341360 secs 2023-03-09T21:54:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:54:01 smithi181 ceph-mon[94189]: pgmap v13817: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 469 KiB/s rd, 453 KiB/s wr, 10 op/s 2023-03-09T21:54:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:54:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24951.341360 secs 2023-03-09T21:54:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:54:01 smithi038 ceph-mon[92422]: pgmap v13817: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 469 KiB/s rd, 453 KiB/s wr, 10 op/s 2023-03-09T21:54:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25415.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:54:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:54:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24953.694423 secs 2023-03-09T21:54:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:54:03 smithi122 ceph-mon[93869]: pgmap v13818: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 453 KiB/s wr, 13 op/s 2023-03-09T21:54:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:54:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24953.694423 secs 2023-03-09T21:54:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:54:03 smithi181 ceph-mon[94189]: pgmap v13818: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 453 KiB/s wr, 13 op/s 2023-03-09T21:54:03.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:54:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24953.694423 secs 2023-03-09T21:54:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:54:03 smithi038 ceph-mon[92422]: pgmap v13818: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 453 KiB/s wr, 13 op/s 2023-03-09T21:54:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:54:05 smithi122 ceph-mon[93869]: pgmap v13819: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 738 KiB/s rd, 740 KiB/s wr, 23 op/s 2023-03-09T21:54:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:54:05 smithi181 ceph-mon[94189]: pgmap v13819: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 738 KiB/s rd, 740 KiB/s wr, 23 op/s 2023-03-09T21:54:05.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:54:05 smithi038 ceph-mon[92422]: pgmap v13819: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 738 KiB/s rd, 740 KiB/s wr, 23 op/s 2023-03-09T21:54:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:54:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24956.341445 secs 2023-03-09T21:54:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:54:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24956.341445 secs 2023-03-09T21:54:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:54:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24956.341445 secs 2023-03-09T21:54:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:54:07 smithi122 ceph-mon[93869]: pgmap v13820: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 720 KiB/s rd, 731 KiB/s wr, 24 op/s 2023-03-09T21:54:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:54:07 smithi181 ceph-mon[94189]: pgmap v13820: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 720 KiB/s rd, 731 KiB/s wr, 24 op/s 2023-03-09T21:54:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:54:07 smithi038 ceph-mon[92422]: pgmap v13820: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 720 KiB/s rd, 731 KiB/s wr, 24 op/s 2023-03-09T21:54:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25420.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:54:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:54:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24958.694538 secs 2023-03-09T21:54:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:54:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24958.694538 secs 2023-03-09T21:54:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:54:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24958.694538 secs 2023-03-09T21:54:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:54:09 smithi122 ceph-mon[93869]: pgmap v13821: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 390 KiB/s wr, 21 op/s 2023-03-09T21:54:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:54:09 smithi181 ceph-mon[94189]: pgmap v13821: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 390 KiB/s wr, 21 op/s 2023-03-09T21:54:09.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:54:09 smithi038 ceph-mon[92422]: pgmap v13821: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 390 KiB/s wr, 21 op/s 2023-03-09T21:54:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:54:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24961.341506 secs 2023-03-09T21:54:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:54:11 smithi122 ceph-mon[93869]: pgmap v13822: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 494 KiB/s rd, 507 KiB/s wr, 22 op/s 2023-03-09T21:54:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:54:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24961.341506 secs 2023-03-09T21:54:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:54:11 smithi181 ceph-mon[94189]: pgmap v13822: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 494 KiB/s rd, 507 KiB/s wr, 22 op/s 2023-03-09T21:54:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:54:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24961.341506 secs 2023-03-09T21:54:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:54:11 smithi038 ceph-mon[92422]: pgmap v13822: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 494 KiB/s rd, 507 KiB/s wr, 22 op/s 2023-03-09T21:54:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25425.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:54:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:54:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24963.694644 secs 2023-03-09T21:54:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:54:13 smithi122 ceph-mon[93869]: pgmap v13823: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 467 KiB/s rd, 479 KiB/s wr, 23 op/s 2023-03-09T21:54:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:54:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24963.694644 secs 2023-03-09T21:54:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:54:13 smithi181 ceph-mon[94189]: pgmap v13823: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 467 KiB/s rd, 479 KiB/s wr, 23 op/s 2023-03-09T21:54:13.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:54:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24963.694644 secs 2023-03-09T21:54:13.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:54:13 smithi038 ceph-mon[92422]: pgmap v13823: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 467 KiB/s rd, 479 KiB/s wr, 23 op/s 2023-03-09T21:54:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:54:15 smithi122 ceph-mon[93869]: pgmap v13824: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 554 KiB/s rd, 567 KiB/s wr, 22 op/s 2023-03-09T21:54:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:54:15 smithi181 ceph-mon[94189]: pgmap v13824: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 554 KiB/s rd, 567 KiB/s wr, 22 op/s 2023-03-09T21:54:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:54:15 smithi038 ceph-mon[92422]: pgmap v13824: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 554 KiB/s rd, 567 KiB/s wr, 22 op/s 2023-03-09T21:54:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:54:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24966.341565 secs 2023-03-09T21:54:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:54:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24966.341565 secs 2023-03-09T21:54:16.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:54:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24966.341565 secs 2023-03-09T21:54:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:54:17 smithi122 ceph-mon[93869]: pgmap v13825: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 237 KiB/s wr, 11 op/s 2023-03-09T21:54:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:54:17 smithi181 ceph-mon[94189]: pgmap v13825: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 237 KiB/s wr, 11 op/s 2023-03-09T21:54:17.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:54:17 smithi038 ceph-mon[92422]: pgmap v13825: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 237 KiB/s wr, 11 op/s 2023-03-09T21:54:18.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25430.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:54:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:54:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24968.694762 secs 2023-03-09T21:54:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:54:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24968.694762 secs 2023-03-09T21:54:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:54:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24968.694762 secs 2023-03-09T21:54:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:54:19 smithi122 ceph-mon[93869]: pgmap v13826: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 226 KiB/s wr, 8 op/s 2023-03-09T21:54:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:54:19 smithi181 ceph-mon[94189]: pgmap v13826: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 226 KiB/s wr, 8 op/s 2023-03-09T21:54:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:54:19 smithi038 ceph-mon[92422]: pgmap v13826: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 226 KiB/s wr, 8 op/s 2023-03-09T21:54:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:54:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24971.341625 secs 2023-03-09T21:54:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:54:21 smithi122 ceph-mon[93869]: pgmap v13827: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 283 KiB/s rd, 281 KiB/s wr, 8 op/s 2023-03-09T21:54:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:54:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24971.341625 secs 2023-03-09T21:54:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:54:21 smithi181 ceph-mon[94189]: pgmap v13827: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 283 KiB/s rd, 281 KiB/s wr, 8 op/s 2023-03-09T21:54:21.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:54:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24971.341625 secs 2023-03-09T21:54:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:54:21 smithi038 ceph-mon[92422]: pgmap v13827: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 283 KiB/s rd, 281 KiB/s wr, 8 op/s 2023-03-09T21:54:23.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25435.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:54:23.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:54:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24973.694893 secs 2023-03-09T21:54:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:54:23 smithi038 ceph-mon[92422]: pgmap v13828: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 299 KiB/s wr, 10 op/s 2023-03-09T21:54:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:54:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24973.694893 secs 2023-03-09T21:54:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:54:23 smithi122 ceph-mon[93869]: pgmap v13828: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 299 KiB/s wr, 10 op/s 2023-03-09T21:54:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:54:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24973.694893 secs 2023-03-09T21:54:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:54:23 smithi181 ceph-mon[94189]: pgmap v13828: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 299 KiB/s wr, 10 op/s 2023-03-09T21:54:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:54:25 smithi038 ceph-mon[92422]: pgmap v13829: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 424 KiB/s wr, 11 op/s 2023-03-09T21:54:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:54:25 smithi038 ceph-mon[92422]: osdmap e1077: 12 total, 12 up, 12 in 2023-03-09T21:54:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:54:25 smithi122 ceph-mon[93869]: pgmap v13829: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 424 KiB/s wr, 11 op/s 2023-03-09T21:54:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:54:25 smithi122 ceph-mon[93869]: osdmap e1077: 12 total, 12 up, 12 in 2023-03-09T21:54:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:54:25 smithi181 ceph-mon[94189]: pgmap v13829: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 424 KiB/s wr, 11 op/s 2023-03-09T21:54:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:54:25 smithi181 ceph-mon[94189]: osdmap e1077: 12 total, 12 up, 12 in 2023-03-09T21:54:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:54:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24976.341688 secs 2023-03-09T21:54:27.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:54:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24976.341688 secs 2023-03-09T21:54:27.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:54:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24976.341688 secs 2023-03-09T21:54:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:54:27 smithi122 ceph-mon[93869]: pgmap v13831: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 504 KiB/s wr, 12 op/s 2023-03-09T21:54:28.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:54:27 smithi181 ceph-mon[94189]: pgmap v13831: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 504 KiB/s wr, 12 op/s 2023-03-09T21:54:28.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:54:27 smithi038 ceph-mon[92422]: pgmap v13831: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 504 KiB/s wr, 12 op/s 2023-03-09T21:54:28.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25440.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:54:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:54:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24978.695031 secs 2023-03-09T21:54:29.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:54:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24978.695031 secs 2023-03-09T21:54:29.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:54:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24978.695031 secs 2023-03-09T21:54:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:54:29 smithi122 ceph-mon[93869]: pgmap v13832: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 528 KiB/s wr, 12 op/s 2023-03-09T21:54:30.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:54:29 smithi181 ceph-mon[94189]: pgmap v13832: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 528 KiB/s wr, 12 op/s 2023-03-09T21:54:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:54:29 smithi038 ceph-mon[92422]: pgmap v13832: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 528 KiB/s wr, 12 op/s 2023-03-09T21:54:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:54:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24981.341749 secs 2023-03-09T21:54:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:54:31 smithi122 ceph-mon[93869]: pgmap v13833: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 462 KiB/s wr, 12 op/s 2023-03-09T21:54:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:54:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24981.341749 secs 2023-03-09T21:54:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:54:31 smithi181 ceph-mon[94189]: pgmap v13833: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 462 KiB/s wr, 12 op/s 2023-03-09T21:54:32.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:54:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24981.341749 secs 2023-03-09T21:54:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:54:31 smithi038 ceph-mon[92422]: pgmap v13833: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 462 KiB/s wr, 12 op/s 2023-03-09T21:54:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25445.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:54:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:54:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24983.695154 secs 2023-03-09T21:54:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:54:33 smithi122 ceph-mon[93869]: pgmap v13834: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 413 KiB/s wr, 11 op/s 2023-03-09T21:54:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:54:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24983.695154 secs 2023-03-09T21:54:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:54:33 smithi181 ceph-mon[94189]: pgmap v13834: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 413 KiB/s wr, 11 op/s 2023-03-09T21:54:34.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:54:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24983.695154 secs 2023-03-09T21:54:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:54:33 smithi038 ceph-mon[92422]: pgmap v13834: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 413 KiB/s wr, 11 op/s 2023-03-09T21:54:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:54:35 smithi122 ceph-mon[93869]: pgmap v13835: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 263 KiB/s wr, 10 op/s 2023-03-09T21:54:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:54:35 smithi122 ceph-mon[93869]: osdmap e1078: 12 total, 12 up, 12 in 2023-03-09T21:54:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:54:35 smithi181 ceph-mon[94189]: pgmap v13835: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 263 KiB/s wr, 10 op/s 2023-03-09T21:54:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:54:35 smithi181 ceph-mon[94189]: osdmap e1078: 12 total, 12 up, 12 in 2023-03-09T21:54:36.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:54:35 smithi038 ceph-mon[92422]: pgmap v13835: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 263 KiB/s wr, 10 op/s 2023-03-09T21:54:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:54:35 smithi038 ceph-mon[92422]: osdmap e1078: 12 total, 12 up, 12 in 2023-03-09T21:54:37.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:54:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24986.341810 secs 2023-03-09T21:54:37.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:54:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24986.341810 secs 2023-03-09T21:54:37.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:54:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24986.341810 secs 2023-03-09T21:54:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:54:37 smithi122 ceph-mon[93869]: pgmap v13837: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 162 KiB/s wr, 10 op/s 2023-03-09T21:54:38.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:54:37 smithi181 ceph-mon[94189]: pgmap v13837: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 162 KiB/s wr, 10 op/s 2023-03-09T21:54:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:54:37 smithi038 ceph-mon[92422]: pgmap v13837: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 162 KiB/s wr, 10 op/s 2023-03-09T21:54:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25450.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:54:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:54:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24988.695274 secs 2023-03-09T21:54:39.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:54:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24988.695274 secs 2023-03-09T21:54:39.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:54:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24988.695274 secs 2023-03-09T21:54:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:54:39 smithi122 ceph-mon[93869]: pgmap v13838: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 344 KiB/s wr, 10 op/s 2023-03-09T21:54:40.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:54:39 smithi181 ceph-mon[94189]: pgmap v13838: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 344 KiB/s wr, 10 op/s 2023-03-09T21:54:40.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:54:39 smithi038 ceph-mon[92422]: pgmap v13838: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 344 KiB/s wr, 10 op/s 2023-03-09T21:54:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:54:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24991.341870 secs 2023-03-09T21:54:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:54:41 smithi122 ceph-mon[93869]: pgmap v13839: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 343 KiB/s wr, 9 op/s 2023-03-09T21:54:42.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:54:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24991.341870 secs 2023-03-09T21:54:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:54:41 smithi181 ceph-mon[94189]: pgmap v13839: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 343 KiB/s wr, 9 op/s 2023-03-09T21:54:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:54:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24991.341870 secs 2023-03-09T21:54:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:54:41 smithi038 ceph-mon[92422]: pgmap v13839: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 343 KiB/s wr, 9 op/s 2023-03-09T21:54:43.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25455.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:54:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:54:43 smithi122 ceph-mon[93869]: pgmap v13840: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 243 KiB/s wr, 9 op/s 2023-03-09T21:54:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:54:43 smithi181 ceph-mon[94189]: pgmap v13840: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 243 KiB/s wr, 9 op/s 2023-03-09T21:54:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:54:43 smithi038 ceph-mon[92422]: pgmap v13840: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 243 KiB/s wr, 9 op/s 2023-03-09T21:54:45.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:54:44 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:54:45.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:54:44 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:54:45.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:54:44 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:54:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:54:45 smithi122 ceph-mon[93869]: pgmap v13841: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 448 KiB/s wr, 9 op/s 2023-03-09T21:54:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:54:45 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:54:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:54:45 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:54:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:54:45 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:54:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:54:45 smithi181 ceph-mon[94189]: pgmap v13841: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 448 KiB/s wr, 9 op/s 2023-03-09T21:54:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:54:45 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:54:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:54:45 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:54:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:54:45 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:54:46.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:54:45 smithi038 ceph-mon[92422]: pgmap v13841: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 448 KiB/s wr, 9 op/s 2023-03-09T21:54:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:54:45 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:54:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:54:45 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:54:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:54:45 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:54:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:54:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 24996.341954 secs 2023-03-09T21:54:47.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:54:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 24996.341954 secs 2023-03-09T21:54:47.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:54:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 24996.341954 secs 2023-03-09T21:54:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:54:47 smithi122 ceph-mon[93869]: pgmap v13842: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 419 KiB/s wr, 11 op/s 2023-03-09T21:54:48.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:54:47 smithi181 ceph-mon[94189]: pgmap v13842: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 419 KiB/s wr, 11 op/s 2023-03-09T21:54:48.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:54:47 smithi038 ceph-mon[92422]: pgmap v13842: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 419 KiB/s wr, 11 op/s 2023-03-09T21:54:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25460.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:54:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:54:49 smithi122 ceph-mon[93869]: pgmap v13843: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 520 KiB/s rd, 505 KiB/s wr, 10 op/s 2023-03-09T21:54:50.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:54:49 smithi181 ceph-mon[94189]: pgmap v13843: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 520 KiB/s rd, 505 KiB/s wr, 10 op/s 2023-03-09T21:54:50.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:54:49 smithi038 ceph-mon[92422]: pgmap v13843: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 520 KiB/s rd, 505 KiB/s wr, 10 op/s 2023-03-09T21:54:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:54:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25001.341986 secs 2023-03-09T21:54:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:54:51 smithi122 ceph-mon[93869]: pgmap v13844: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 334 KiB/s wr, 9 op/s 2023-03-09T21:54:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:54:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25001.341986 secs 2023-03-09T21:54:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:54:51 smithi181 ceph-mon[94189]: pgmap v13844: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 334 KiB/s wr, 9 op/s 2023-03-09T21:54:51.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:54:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25001.341986 secs 2023-03-09T21:54:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:54:51 smithi038 ceph-mon[92422]: pgmap v13844: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 334 KiB/s wr, 9 op/s 2023-03-09T21:54:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25465.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:54:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:54:53 smithi122 ceph-mon[93869]: pgmap v13845: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 534 KiB/s wr, 12 op/s 2023-03-09T21:54:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:54:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:54:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:54:53 smithi181 ceph-mon[94189]: pgmap v13845: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 534 KiB/s wr, 12 op/s 2023-03-09T21:54:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:54:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:54:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:54:53 smithi038 ceph-mon[92422]: pgmap v13845: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 534 KiB/s wr, 12 op/s 2023-03-09T21:54:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:54:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:54:55.606 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:54:55 smithi122 ceph-mon[93869]: pgmap v13846: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 714 KiB/s rd, 598 KiB/s wr, 12 op/s 2023-03-09T21:54:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:54:55 smithi181 ceph-mon[94189]: pgmap v13846: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 714 KiB/s rd, 598 KiB/s wr, 12 op/s 2023-03-09T21:54:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:54:55 smithi038 ceph-mon[92422]: pgmap v13846: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 714 KiB/s rd, 598 KiB/s wr, 12 op/s 2023-03-09T21:54:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:54:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25006.342047 secs 2023-03-09T21:54:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:54:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25006.342047 secs 2023-03-09T21:54:56.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:54:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25006.342047 secs 2023-03-09T21:54:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:54:57 smithi122 ceph-mon[93869]: pgmap v13847: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 447 KiB/s wr, 12 op/s 2023-03-09T21:54:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:54:57 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:54:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:54:57 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:54:57.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:54:57 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:54:57.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:54:57 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:54:57.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:54:57 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:54:57.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:54:57 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:54:57.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:54:57 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:54:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:54:57 smithi181 ceph-mon[94189]: pgmap v13847: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 447 KiB/s wr, 12 op/s 2023-03-09T21:54:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:54:57 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:54:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:54:57 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:54:57.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:54:57 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:54:57.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:54:57 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:54:57.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:54:57 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:54:57.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:54:57 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:54:57.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:54:57 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:54:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:54:57 smithi038 ceph-mon[92422]: pgmap v13847: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 447 KiB/s wr, 12 op/s 2023-03-09T21:54:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:54:57 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:54:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:54:57 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:54:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:54:57 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:54:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:54:57 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:54:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:54:57 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:54:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:54:57 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:54:58.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:54:57 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:54:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25470.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:54:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:54:59 smithi122 ceph-mon[93869]: pgmap v13848: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 447 KiB/s wr, 11 op/s 2023-03-09T21:54:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:54:59 smithi181 ceph-mon[94189]: pgmap v13848: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 447 KiB/s wr, 11 op/s 2023-03-09T21:54:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:54:59 smithi038 ceph-mon[92422]: pgmap v13848: 129 pgs: 129 active+clean; 9.3 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 447 KiB/s wr, 11 op/s 2023-03-09T21:55:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:55:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25011.342106 secs 2023-03-09T21:55:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:55:01 smithi122 ceph-mon[93869]: pgmap v13849: 129 pgs: 129 active+clean; 9.3 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 688 KiB/s rd, 687 KiB/s wr, 10 op/s 2023-03-09T21:55:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:55:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25011.342106 secs 2023-03-09T21:55:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:55:01 smithi181 ceph-mon[94189]: pgmap v13849: 129 pgs: 129 active+clean; 9.3 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 688 KiB/s rd, 687 KiB/s wr, 10 op/s 2023-03-09T21:55:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:55:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25011.342106 secs 2023-03-09T21:55:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:55:01 smithi038 ceph-mon[92422]: pgmap v13849: 129 pgs: 129 active+clean; 9.3 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 688 KiB/s rd, 687 KiB/s wr, 10 op/s 2023-03-09T21:55:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25475.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:55:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:55:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25013.695897 secs 2023-03-09T21:55:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:55:03 smithi181 ceph-mon[94189]: pgmap v13850: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 709 KiB/s rd, 718 KiB/s wr, 14 op/s 2023-03-09T21:55:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:55:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25013.695897 secs 2023-03-09T21:55:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:55:03 smithi038 ceph-mon[92422]: pgmap v13850: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 709 KiB/s rd, 718 KiB/s wr, 14 op/s 2023-03-09T21:55:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:55:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25013.695897 secs 2023-03-09T21:55:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:55:03 smithi122 ceph-mon[93869]: pgmap v13850: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 709 KiB/s rd, 718 KiB/s wr, 14 op/s 2023-03-09T21:55:05.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:55:05 smithi038 ceph-mon[92422]: pgmap v13851: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 517 KiB/s rd, 541 KiB/s wr, 25 op/s 2023-03-09T21:55:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:55:05 smithi122 ceph-mon[93869]: pgmap v13851: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 517 KiB/s rd, 541 KiB/s wr, 25 op/s 2023-03-09T21:55:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:55:05 smithi181 ceph-mon[94189]: pgmap v13851: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 517 KiB/s rd, 541 KiB/s wr, 25 op/s 2023-03-09T21:55:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:55:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25016.342168 secs 2023-03-09T21:55:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:55:06 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:55:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:55:06 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:55:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:55:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:55:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:55:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25016.342168 secs 2023-03-09T21:55:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:55:06 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:55:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:55:06 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:55:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:55:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:55:07.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:55:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25016.342168 secs 2023-03-09T21:55:07.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:55:06 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:55:07.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:55:06 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:55:07.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:55:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:55:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:55:07 smithi038 ceph-mon[92422]: pgmap v13852: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 457 KiB/s wr, 27 op/s 2023-03-09T21:55:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:55:07 smithi122 ceph-mon[93869]: pgmap v13852: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 457 KiB/s wr, 27 op/s 2023-03-09T21:55:08.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:55:07 smithi181 ceph-mon[94189]: pgmap v13852: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 457 KiB/s wr, 27 op/s 2023-03-09T21:55:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25480.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:55:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:55:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25018.695978 secs 2023-03-09T21:55:09.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:55:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25018.695978 secs 2023-03-09T21:55:09.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:55:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25018.695978 secs 2023-03-09T21:55:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:55:09 smithi122 ceph-mon[93869]: pgmap v13853: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 438 KiB/s wr, 25 op/s 2023-03-09T21:55:10.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:55:09 smithi181 ceph-mon[94189]: pgmap v13853: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 438 KiB/s wr, 25 op/s 2023-03-09T21:55:10.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:55:09 smithi038 ceph-mon[92422]: pgmap v13853: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 438 KiB/s wr, 25 op/s 2023-03-09T21:55:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:55:10 smithi122 ceph-mon[93869]: osdmap e1079: 12 total, 12 up, 12 in 2023-03-09T21:55:11.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:55:10 smithi181 ceph-mon[94189]: osdmap e1079: 12 total, 12 up, 12 in 2023-03-09T21:55:11.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:55:10 smithi038 ceph-mon[92422]: osdmap e1079: 12 total, 12 up, 12 in 2023-03-09T21:55:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:55:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25021.342233 secs 2023-03-09T21:55:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:55:11 smithi122 ceph-mon[93869]: pgmap v13855: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 179 KiB/s wr, 28 op/s 2023-03-09T21:55:12.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:55:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25021.342233 secs 2023-03-09T21:55:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:55:11 smithi181 ceph-mon[94189]: pgmap v13855: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 179 KiB/s wr, 28 op/s 2023-03-09T21:55:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:55:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25021.342233 secs 2023-03-09T21:55:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:55:11 smithi038 ceph-mon[92422]: pgmap v13855: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 179 KiB/s wr, 28 op/s 2023-03-09T21:55:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25485.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:55:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:55:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25023.696131 secs 2023-03-09T21:55:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:55:13 smithi122 ceph-mon[93869]: pgmap v13856: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 142 KiB/s wr, 26 op/s 2023-03-09T21:55:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:55:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25023.696131 secs 2023-03-09T21:55:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:55:13 smithi181 ceph-mon[94189]: pgmap v13856: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 142 KiB/s wr, 26 op/s 2023-03-09T21:55:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:55:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25023.696131 secs 2023-03-09T21:55:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:55:13 smithi038 ceph-mon[92422]: pgmap v13856: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 142 KiB/s wr, 26 op/s 2023-03-09T21:55:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:55:15 smithi122 ceph-mon[93869]: pgmap v13857: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 115 KiB/s wr, 11 op/s 2023-03-09T21:55:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:55:15 smithi122 ceph-mon[93869]: osdmap e1080: 12 total, 12 up, 12 in 2023-03-09T21:55:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:55:15 smithi181 ceph-mon[94189]: pgmap v13857: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 115 KiB/s wr, 11 op/s 2023-03-09T21:55:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:55:15 smithi181 ceph-mon[94189]: osdmap e1080: 12 total, 12 up, 12 in 2023-03-09T21:55:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:55:15 smithi038 ceph-mon[92422]: pgmap v13857: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 115 KiB/s wr, 11 op/s 2023-03-09T21:55:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:55:15 smithi038 ceph-mon[92422]: osdmap e1080: 12 total, 12 up, 12 in 2023-03-09T21:55:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:55:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25026.342292 secs 2023-03-09T21:55:17.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:55:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25026.342292 secs 2023-03-09T21:55:17.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:55:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25026.342292 secs 2023-03-09T21:55:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:55:17 smithi122 ceph-mon[93869]: pgmap v13859: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 157 KiB/s wr, 11 op/s 2023-03-09T21:55:18.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:55:17 smithi181 ceph-mon[94189]: pgmap v13859: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 157 KiB/s wr, 11 op/s 2023-03-09T21:55:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:55:17 smithi038 ceph-mon[92422]: pgmap v13859: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 157 KiB/s wr, 11 op/s 2023-03-09T21:55:18.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25490.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:55:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:55:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25028.696241 secs 2023-03-09T21:55:19.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:55:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25028.696241 secs 2023-03-09T21:55:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:55:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25028.696241 secs 2023-03-09T21:55:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:55:19 smithi122 ceph-mon[93869]: pgmap v13860: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 29 KiB/s wr, 9 op/s 2023-03-09T21:55:20.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:55:19 smithi181 ceph-mon[94189]: pgmap v13860: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 29 KiB/s wr, 9 op/s 2023-03-09T21:55:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:55:19 smithi038 ceph-mon[92422]: pgmap v13860: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 29 KiB/s wr, 9 op/s 2023-03-09T21:55:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:55:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25031.342351 secs 2023-03-09T21:55:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:55:21 smithi122 ceph-mon[93869]: pgmap v13861: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 143 KiB/s wr, 9 op/s 2023-03-09T21:55:22.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:55:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25031.342351 secs 2023-03-09T21:55:22.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:55:21 smithi181 ceph-mon[94189]: pgmap v13861: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 143 KiB/s wr, 9 op/s 2023-03-09T21:55:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:55:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25031.342351 secs 2023-03-09T21:55:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:55:21 smithi038 ceph-mon[92422]: pgmap v13861: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 143 KiB/s wr, 9 op/s 2023-03-09T21:55:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25495.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:55:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:55:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25033.696324 secs 2023-03-09T21:55:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:55:23 smithi122 ceph-mon[93869]: pgmap v13862: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 166 KiB/s wr, 9 op/s 2023-03-09T21:55:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:55:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25033.696324 secs 2023-03-09T21:55:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:55:23 smithi181 ceph-mon[94189]: pgmap v13862: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 166 KiB/s wr, 9 op/s 2023-03-09T21:55:24.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:55:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25033.696324 secs 2023-03-09T21:55:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:55:23 smithi038 ceph-mon[92422]: pgmap v13862: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 166 KiB/s wr, 9 op/s 2023-03-09T21:55:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:55:25 smithi122 ceph-mon[93869]: pgmap v13863: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 289 KiB/s wr, 8 op/s 2023-03-09T21:55:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:55:25 smithi181 ceph-mon[94189]: pgmap v13863: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 289 KiB/s wr, 8 op/s 2023-03-09T21:55:26.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:55:25 smithi038 ceph-mon[92422]: pgmap v13863: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 289 KiB/s wr, 8 op/s 2023-03-09T21:55:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:55:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25036.342409 secs 2023-03-09T21:55:27.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:55:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25036.342409 secs 2023-03-09T21:55:27.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:55:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25036.342409 secs 2023-03-09T21:55:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:55:27 smithi122 ceph-mon[93869]: pgmap v13864: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 270 KiB/s wr, 10 op/s 2023-03-09T21:55:28.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:55:27 smithi181 ceph-mon[94189]: pgmap v13864: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 270 KiB/s wr, 10 op/s 2023-03-09T21:55:28.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:55:27 smithi038 ceph-mon[92422]: pgmap v13864: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 270 KiB/s wr, 10 op/s 2023-03-09T21:55:28.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25500.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:55:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:55:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25038.696390 secs 2023-03-09T21:55:29.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:55:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25038.696390 secs 2023-03-09T21:55:29.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:55:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25038.696390 secs 2023-03-09T21:55:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:55:29 smithi122 ceph-mon[93869]: pgmap v13865: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 290 KiB/s wr, 8 op/s 2023-03-09T21:55:30.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:55:29 smithi181 ceph-mon[94189]: pgmap v13865: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 290 KiB/s wr, 8 op/s 2023-03-09T21:55:30.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:55:29 smithi038 ceph-mon[92422]: pgmap v13865: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 290 KiB/s wr, 8 op/s 2023-03-09T21:55:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:55:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25041.342488 secs 2023-03-09T21:55:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:55:31 smithi122 ceph-mon[93869]: pgmap v13866: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 290 KiB/s wr, 8 op/s 2023-03-09T21:55:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:55:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25041.342488 secs 2023-03-09T21:55:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:55:31 smithi181 ceph-mon[94189]: pgmap v13866: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 290 KiB/s wr, 8 op/s 2023-03-09T21:55:31.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:55:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25041.342488 secs 2023-03-09T21:55:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:55:31 smithi038 ceph-mon[92422]: pgmap v13866: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 290 KiB/s wr, 8 op/s 2023-03-09T21:55:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25505.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:55:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:55:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25043.696511 secs 2023-03-09T21:55:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:55:33 smithi122 ceph-mon[93869]: pgmap v13867: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 193 KiB/s wr, 10 op/s 2023-03-09T21:55:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:55:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25043.696511 secs 2023-03-09T21:55:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:55:33 smithi181 ceph-mon[94189]: pgmap v13867: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 193 KiB/s wr, 10 op/s 2023-03-09T21:55:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:55:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25043.696511 secs 2023-03-09T21:55:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:55:33 smithi038 ceph-mon[92422]: pgmap v13867: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 193 KiB/s wr, 10 op/s 2023-03-09T21:55:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:55:35 smithi122 ceph-mon[93869]: pgmap v13868: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 344 KiB/s wr, 8 op/s 2023-03-09T21:55:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:55:35 smithi181 ceph-mon[94189]: pgmap v13868: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 344 KiB/s wr, 8 op/s 2023-03-09T21:55:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:55:35 smithi038 ceph-mon[92422]: pgmap v13868: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 344 KiB/s wr, 8 op/s 2023-03-09T21:55:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:55:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25046.342549 secs 2023-03-09T21:55:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:55:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25046.342549 secs 2023-03-09T21:55:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:55:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25046.342549 secs 2023-03-09T21:55:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:55:37 smithi122 ceph-mon[93869]: pgmap v13869: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 287 KiB/s wr, 9 op/s 2023-03-09T21:55:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:55:37 smithi181 ceph-mon[94189]: pgmap v13869: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 287 KiB/s wr, 9 op/s 2023-03-09T21:55:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:55:37 smithi038 ceph-mon[92422]: pgmap v13869: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 287 KiB/s wr, 9 op/s 2023-03-09T21:55:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25510.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:55:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:55:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25048.696623 secs 2023-03-09T21:55:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:55:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25048.696623 secs 2023-03-09T21:55:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:55:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25048.696623 secs 2023-03-09T21:55:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:55:39 smithi122 ceph-mon[93869]: pgmap v13870: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 338 KiB/s wr, 8 op/s 2023-03-09T21:55:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:55:39 smithi181 ceph-mon[94189]: pgmap v13870: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 338 KiB/s wr, 8 op/s 2023-03-09T21:55:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:55:39 smithi038 ceph-mon[92422]: pgmap v13870: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 338 KiB/s wr, 8 op/s 2023-03-09T21:55:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:55:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25051.342612 secs 2023-03-09T21:55:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:55:41 smithi122 ceph-mon[93869]: pgmap v13871: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 268 KiB/s wr, 7 op/s 2023-03-09T21:55:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:55:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25051.342612 secs 2023-03-09T21:55:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:55:41 smithi181 ceph-mon[94189]: pgmap v13871: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 268 KiB/s wr, 7 op/s 2023-03-09T21:55:41.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:55:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25051.342612 secs 2023-03-09T21:55:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:55:41 smithi038 ceph-mon[92422]: pgmap v13871: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 268 KiB/s wr, 7 op/s 2023-03-09T21:55:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25515.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:55:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:55:43 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 25053.696758 secs 2023-03-09T21:55:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:55:43 smithi122 ceph-mon[93869]: pgmap v13872: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 384 KiB/s rd, 381 KiB/s wr, 9 op/s 2023-03-09T21:55:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:55:43 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 25053.696758 secs 2023-03-09T21:55:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:55:43 smithi181 ceph-mon[94189]: pgmap v13872: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 384 KiB/s rd, 381 KiB/s wr, 9 op/s 2023-03-09T21:55:43.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:55:43 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 25053.696758 secs 2023-03-09T21:55:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:55:43 smithi038 ceph-mon[92422]: pgmap v13872: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 384 KiB/s rd, 381 KiB/s wr, 9 op/s 2023-03-09T21:55:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:55:45 smithi122 ceph-mon[93869]: pgmap v13873: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 380 KiB/s wr, 7 op/s 2023-03-09T21:55:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:55:45 smithi181 ceph-mon[94189]: pgmap v13873: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 380 KiB/s wr, 7 op/s 2023-03-09T21:55:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:55:45 smithi038 ceph-mon[92422]: pgmap v13873: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 380 KiB/s wr, 7 op/s 2023-03-09T21:55:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:55:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25056.342673 secs 2023-03-09T21:55:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:55:46 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:55:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:55:46 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:55:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:55:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25056.342673 secs 2023-03-09T21:55:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:55:46 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:55:46.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:55:46 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:55:46.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:55:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25056.342673 secs 2023-03-09T21:55:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:55:46 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:55:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:55:46 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:55:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:55:47 smithi122 ceph-mon[93869]: pgmap v13874: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 519 KiB/s rd, 414 KiB/s wr, 10 op/s 2023-03-09T21:55:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:55:47 smithi181 ceph-mon[94189]: pgmap v13874: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 519 KiB/s rd, 414 KiB/s wr, 10 op/s 2023-03-09T21:55:47.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:55:47 smithi038 ceph-mon[92422]: pgmap v13874: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 519 KiB/s rd, 414 KiB/s wr, 10 op/s 2023-03-09T21:55:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25520.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:55:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:55:48 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 25058.696880 secs 2023-03-09T21:55:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:55:48 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 25058.696880 secs 2023-03-09T21:55:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:55:48 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 25058.696880 secs 2023-03-09T21:55:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:55:49 smithi122 ceph-mon[93869]: pgmap v13875: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 484 KiB/s rd, 381 KiB/s wr, 9 op/s 2023-03-09T21:55:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:55:49 smithi181 ceph-mon[94189]: pgmap v13875: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 484 KiB/s rd, 381 KiB/s wr, 9 op/s 2023-03-09T21:55:49.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:55:49 smithi038 ceph-mon[92422]: pgmap v13875: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 484 KiB/s rd, 381 KiB/s wr, 9 op/s 2023-03-09T21:55:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:55:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25061.342732 secs 2023-03-09T21:55:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:55:51 smithi122 ceph-mon[93869]: pgmap v13876: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 329 KiB/s wr, 8 op/s 2023-03-09T21:55:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:55:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25061.342732 secs 2023-03-09T21:55:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:55:51 smithi181 ceph-mon[94189]: pgmap v13876: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 329 KiB/s wr, 8 op/s 2023-03-09T21:55:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:55:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25061.342732 secs 2023-03-09T21:55:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:55:51 smithi038 ceph-mon[92422]: pgmap v13876: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 329 KiB/s wr, 8 op/s 2023-03-09T21:55:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25525.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:55:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:55:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25063.696998 secs 2023-03-09T21:55:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:55:53 smithi122 ceph-mon[93869]: pgmap v13877: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 547 KiB/s rd, 532 KiB/s wr, 13 op/s 2023-03-09T21:55:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:55:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25063.696998 secs 2023-03-09T21:55:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:55:53 smithi181 ceph-mon[94189]: pgmap v13877: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 547 KiB/s rd, 532 KiB/s wr, 13 op/s 2023-03-09T21:55:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:55:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25063.696998 secs 2023-03-09T21:55:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:55:53 smithi038 ceph-mon[92422]: pgmap v13877: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 547 KiB/s rd, 532 KiB/s wr, 13 op/s 2023-03-09T21:55:54.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:55:54 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:55:54.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:55:54 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:55:54.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:55:54 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:55:54.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:55:54 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:55:54.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:55:54 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:55:54.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:55:54 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:55:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:55:55 smithi181 ceph-mon[94189]: pgmap v13878: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 433 KiB/s wr, 10 op/s 2023-03-09T21:55:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:55:55 smithi038 ceph-mon[92422]: pgmap v13878: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 433 KiB/s wr, 10 op/s 2023-03-09T21:55:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:55:55 smithi122 ceph-mon[93869]: pgmap v13878: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 433 KiB/s wr, 10 op/s 2023-03-09T21:55:56.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:55:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25066.342795 secs 2023-03-09T21:55:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:55:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25066.342795 secs 2023-03-09T21:55:57.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:55:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25066.342795 secs 2023-03-09T21:55:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:55:57 smithi038 ceph-mon[92422]: pgmap v13879: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 455 KiB/s rd, 453 KiB/s wr, 13 op/s 2023-03-09T21:55:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:55:57 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:55:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:55:57 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:55:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:55:57 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:55:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:55:57 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:55:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:55:57 smithi122 ceph-mon[93869]: pgmap v13879: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 455 KiB/s rd, 453 KiB/s wr, 13 op/s 2023-03-09T21:55:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:55:57 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:55:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:55:57 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:55:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:55:57 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:55:58.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:55:57 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:55:58.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:55:57 smithi181 ceph-mon[94189]: pgmap v13879: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 455 KiB/s rd, 453 KiB/s wr, 13 op/s 2023-03-09T21:55:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:55:57 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:55:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:55:57 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:55:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:55:57 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:55:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:55:57 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:55:58.793 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25530.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:55:58.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:55:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25068.697135 secs 2023-03-09T21:55:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:55:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25068.697135 secs 2023-03-09T21:55:59.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:55:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25068.697135 secs 2023-03-09T21:55:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:55:59 smithi038 ceph-mon[92422]: pgmap v13880: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 524 KiB/s rd, 509 KiB/s wr, 12 op/s 2023-03-09T21:56:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:55:59 smithi122 ceph-mon[93869]: pgmap v13880: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 524 KiB/s rd, 509 KiB/s wr, 12 op/s 2023-03-09T21:56:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:55:59 smithi181 ceph-mon[94189]: pgmap v13880: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 524 KiB/s rd, 509 KiB/s wr, 12 op/s 2023-03-09T21:56:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:56:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25071.342856 secs 2023-03-09T21:56:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:56:01 smithi122 ceph-mon[93869]: pgmap v13881: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 513 KiB/s rd, 498 KiB/s wr, 11 op/s 2023-03-09T21:56:02.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:56:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25071.342856 secs 2023-03-09T21:56:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:56:01 smithi181 ceph-mon[94189]: pgmap v13881: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 513 KiB/s rd, 498 KiB/s wr, 11 op/s 2023-03-09T21:56:02.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:56:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25071.342856 secs 2023-03-09T21:56:02.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:56:01 smithi038 ceph-mon[92422]: pgmap v13881: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 513 KiB/s rd, 498 KiB/s wr, 11 op/s 2023-03-09T21:56:03.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25535.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:56:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:56:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25073.697194 secs 2023-03-09T21:56:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:56:03 smithi122 ceph-mon[93869]: pgmap v13882: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 523 KiB/s rd, 518 KiB/s wr, 14 op/s 2023-03-09T21:56:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:56:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25073.697194 secs 2023-03-09T21:56:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:56:03 smithi181 ceph-mon[94189]: pgmap v13882: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 523 KiB/s rd, 518 KiB/s wr, 14 op/s 2023-03-09T21:56:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:56:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25073.697194 secs 2023-03-09T21:56:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:56:03 smithi038 ceph-mon[92422]: pgmap v13882: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 523 KiB/s rd, 518 KiB/s wr, 14 op/s 2023-03-09T21:56:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:56:05 smithi122 ceph-mon[93869]: pgmap v13883: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 359 KiB/s wr, 11 op/s 2023-03-09T21:56:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:56:05 smithi181 ceph-mon[94189]: pgmap v13883: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 359 KiB/s wr, 11 op/s 2023-03-09T21:56:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:56:05 smithi038 ceph-mon[92422]: pgmap v13883: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 359 KiB/s wr, 11 op/s 2023-03-09T21:56:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:56:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25076.342916 secs 2023-03-09T21:56:07.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:56:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25076.342916 secs 2023-03-09T21:56:07.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:56:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25076.342916 secs 2023-03-09T21:56:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:56:07 smithi122 ceph-mon[93869]: pgmap v13884: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 439 KiB/s wr, 16 op/s 2023-03-09T21:56:08.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:56:07 smithi181 ceph-mon[94189]: pgmap v13884: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 439 KiB/s wr, 16 op/s 2023-03-09T21:56:08.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:56:07 smithi038 ceph-mon[92422]: pgmap v13884: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 439 KiB/s wr, 16 op/s 2023-03-09T21:56:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25540.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:56:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:56:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25078.697308 secs 2023-03-09T21:56:09.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:56:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25078.697308 secs 2023-03-09T21:56:09.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:56:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25078.697308 secs 2023-03-09T21:56:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:56:09 smithi122 ceph-mon[93869]: pgmap v13885: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 592 KiB/s rd, 613 KiB/s wr, 26 op/s 2023-03-09T21:56:10.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:56:09 smithi181 ceph-mon[94189]: pgmap v13885: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 592 KiB/s rd, 613 KiB/s wr, 26 op/s 2023-03-09T21:56:10.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:56:09 smithi038 ceph-mon[92422]: pgmap v13885: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 592 KiB/s rd, 613 KiB/s wr, 26 op/s 2023-03-09T21:56:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:56:10 smithi122 ceph-mon[93869]: osdmap e1081: 12 total, 12 up, 12 in 2023-03-09T21:56:11.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:56:10 smithi181 ceph-mon[94189]: osdmap e1081: 12 total, 12 up, 12 in 2023-03-09T21:56:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:56:10 smithi038 ceph-mon[92422]: osdmap e1081: 12 total, 12 up, 12 in 2023-03-09T21:56:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:56:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25081.342989 secs 2023-03-09T21:56:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:56:11 smithi122 ceph-mon[93869]: pgmap v13887: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 422 KiB/s wr, 28 op/s 2023-03-09T21:56:12.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:56:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25081.342989 secs 2023-03-09T21:56:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:56:11 smithi181 ceph-mon[94189]: pgmap v13887: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 422 KiB/s wr, 28 op/s 2023-03-09T21:56:12.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:56:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25081.342989 secs 2023-03-09T21:56:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:56:11 smithi038 ceph-mon[92422]: pgmap v13887: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 422 KiB/s wr, 28 op/s 2023-03-09T21:56:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25545.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:56:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:56:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25083.697426 secs 2023-03-09T21:56:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:56:13 smithi122 ceph-mon[93869]: pgmap v13888: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 619 KiB/s rd, 660 KiB/s wr, 28 op/s 2023-03-09T21:56:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:56:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25083.697426 secs 2023-03-09T21:56:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:56:13 smithi181 ceph-mon[94189]: pgmap v13888: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 619 KiB/s rd, 660 KiB/s wr, 28 op/s 2023-03-09T21:56:14.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:56:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25083.697426 secs 2023-03-09T21:56:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:56:13 smithi038 ceph-mon[92422]: pgmap v13888: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 619 KiB/s rd, 660 KiB/s wr, 28 op/s 2023-03-09T21:56:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:56:15 smithi122 ceph-mon[93869]: pgmap v13889: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 567 KiB/s rd, 609 KiB/s wr, 26 op/s 2023-03-09T21:56:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:56:15 smithi122 ceph-mon[93869]: osdmap e1082: 12 total, 12 up, 12 in 2023-03-09T21:56:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:56:15 smithi181 ceph-mon[94189]: pgmap v13889: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 567 KiB/s rd, 609 KiB/s wr, 26 op/s 2023-03-09T21:56:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:56:15 smithi181 ceph-mon[94189]: osdmap e1082: 12 total, 12 up, 12 in 2023-03-09T21:56:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:56:15 smithi038 ceph-mon[92422]: pgmap v13889: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 567 KiB/s rd, 609 KiB/s wr, 26 op/s 2023-03-09T21:56:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:56:15 smithi038 ceph-mon[92422]: osdmap e1082: 12 total, 12 up, 12 in 2023-03-09T21:56:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:56:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25086.343048 secs 2023-03-09T21:56:17.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:56:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25086.343048 secs 2023-03-09T21:56:17.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:56:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25086.343048 secs 2023-03-09T21:56:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:56:17 smithi122 ceph-mon[93869]: pgmap v13891: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 466 KiB/s rd, 464 KiB/s wr, 11 op/s 2023-03-09T21:56:18.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:56:17 smithi181 ceph-mon[94189]: pgmap v13891: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 466 KiB/s rd, 464 KiB/s wr, 11 op/s 2023-03-09T21:56:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:56:17 smithi038 ceph-mon[92422]: pgmap v13891: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 466 KiB/s rd, 464 KiB/s wr, 11 op/s 2023-03-09T21:56:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25550.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:56:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:56:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25088.697547 secs 2023-03-09T21:56:19.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:56:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25088.697547 secs 2023-03-09T21:56:19.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:56:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25088.697547 secs 2023-03-09T21:56:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:56:19 smithi122 ceph-mon[93869]: pgmap v13892: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 434 KiB/s wr, 12 op/s 2023-03-09T21:56:20.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:56:19 smithi181 ceph-mon[94189]: pgmap v13892: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 434 KiB/s wr, 12 op/s 2023-03-09T21:56:20.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:56:19 smithi038 ceph-mon[92422]: pgmap v13892: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 434 KiB/s wr, 12 op/s 2023-03-09T21:56:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:56:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25091.343108 secs 2023-03-09T21:56:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:56:21 smithi122 ceph-mon[93869]: pgmap v13893: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 371 KiB/s wr, 10 op/s 2023-03-09T21:56:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:56:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25091.343108 secs 2023-03-09T21:56:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:56:21 smithi181 ceph-mon[94189]: pgmap v13893: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 371 KiB/s wr, 10 op/s 2023-03-09T21:56:21.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:56:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25091.343108 secs 2023-03-09T21:56:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:56:21 smithi038 ceph-mon[92422]: pgmap v13893: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 371 KiB/s wr, 10 op/s 2023-03-09T21:56:23.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25555.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:56:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:56:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25093.697663 secs 2023-03-09T21:56:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:56:23 smithi122 ceph-mon[93869]: pgmap v13894: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 108 KiB/s wr, 8 op/s 2023-03-09T21:56:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:56:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25093.697663 secs 2023-03-09T21:56:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:56:23 smithi181 ceph-mon[94189]: pgmap v13894: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 108 KiB/s wr, 8 op/s 2023-03-09T21:56:23.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:56:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25093.697663 secs 2023-03-09T21:56:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:56:23 smithi038 ceph-mon[92422]: pgmap v13894: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 108 KiB/s wr, 8 op/s 2023-03-09T21:56:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:56:25 smithi122 ceph-mon[93869]: pgmap v13895: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 190 KiB/s wr, 8 op/s 2023-03-09T21:56:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:56:25 smithi181 ceph-mon[94189]: pgmap v13895: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 190 KiB/s wr, 8 op/s 2023-03-09T21:56:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:56:25 smithi038 ceph-mon[92422]: pgmap v13895: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 190 KiB/s wr, 8 op/s 2023-03-09T21:56:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:56:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25096.343171 secs 2023-03-09T21:56:26.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:56:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25096.343171 secs 2023-03-09T21:56:26.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:56:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25096.343171 secs 2023-03-09T21:56:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:56:27 smithi122 ceph-mon[93869]: pgmap v13896: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 180 KiB/s wr, 11 op/s 2023-03-09T21:56:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:56:27 smithi181 ceph-mon[94189]: pgmap v13896: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 180 KiB/s wr, 11 op/s 2023-03-09T21:56:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:56:27 smithi038 ceph-mon[92422]: pgmap v13896: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 180 KiB/s wr, 11 op/s 2023-03-09T21:56:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25560.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:56:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:56:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25098.697781 secs 2023-03-09T21:56:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:56:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25098.697781 secs 2023-03-09T21:56:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:56:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25098.697781 secs 2023-03-09T21:56:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:56:29 smithi122 ceph-mon[93869]: pgmap v13897: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 71 KiB/s wr, 6 op/s 2023-03-09T21:56:29.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:56:29 smithi181 ceph-mon[94189]: pgmap v13897: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 71 KiB/s wr, 6 op/s 2023-03-09T21:56:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:56:29 smithi038 ceph-mon[92422]: pgmap v13897: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 71 KiB/s wr, 6 op/s 2023-03-09T21:56:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:56:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25101.343255 secs 2023-03-09T21:56:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:56:31 smithi122 ceph-mon[93869]: pgmap v13898: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 71 KiB/s wr, 6 op/s 2023-03-09T21:56:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:56:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25101.343255 secs 2023-03-09T21:56:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:56:31 smithi181 ceph-mon[94189]: pgmap v13898: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 71 KiB/s wr, 6 op/s 2023-03-09T21:56:31.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:56:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25101.343255 secs 2023-03-09T21:56:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:56:31 smithi038 ceph-mon[92422]: pgmap v13898: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 71 KiB/s wr, 6 op/s 2023-03-09T21:56:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25565.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:56:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:56:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25103.697901 secs 2023-03-09T21:56:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:56:33 smithi122 ceph-mon[93869]: pgmap v13899: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 71 KiB/s wr, 9 op/s 2023-03-09T21:56:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:56:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25103.697901 secs 2023-03-09T21:56:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:56:33 smithi181 ceph-mon[94189]: pgmap v13899: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 71 KiB/s wr, 9 op/s 2023-03-09T21:56:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:56:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25103.697901 secs 2023-03-09T21:56:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:56:33 smithi038 ceph-mon[92422]: pgmap v13899: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 71 KiB/s wr, 9 op/s 2023-03-09T21:56:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:56:35 smithi122 ceph-mon[93869]: pgmap v13900: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 173 KiB/s wr, 7 op/s 2023-03-09T21:56:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:56:35 smithi181 ceph-mon[94189]: pgmap v13900: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 173 KiB/s wr, 7 op/s 2023-03-09T21:56:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:56:35 smithi038 ceph-mon[92422]: pgmap v13900: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 173 KiB/s wr, 7 op/s 2023-03-09T21:56:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:56:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25106.343314 secs 2023-03-09T21:56:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:56:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25106.343314 secs 2023-03-09T21:56:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:56:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25106.343314 secs 2023-03-09T21:56:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:56:37 smithi122 ceph-mon[93869]: pgmap v13901: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 105 KiB/s wr, 9 op/s 2023-03-09T21:56:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:56:37 smithi181 ceph-mon[94189]: pgmap v13901: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 105 KiB/s wr, 9 op/s 2023-03-09T21:56:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:56:37 smithi038 ceph-mon[92422]: pgmap v13901: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 105 KiB/s wr, 9 op/s 2023-03-09T21:56:38.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25570.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:56:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:56:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25108.698014 secs 2023-03-09T21:56:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:56:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25108.698014 secs 2023-03-09T21:56:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:56:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25108.698014 secs 2023-03-09T21:56:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:56:39 smithi122 ceph-mon[93869]: pgmap v13902: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 102 KiB/s wr, 7 op/s 2023-03-09T21:56:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:56:39 smithi181 ceph-mon[94189]: pgmap v13902: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 102 KiB/s wr, 7 op/s 2023-03-09T21:56:39.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:56:39 smithi038 ceph-mon[92422]: pgmap v13902: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 102 KiB/s wr, 7 op/s 2023-03-09T21:56:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:56:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25111.343398 secs 2023-03-09T21:56:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:56:41 smithi122 ceph-mon[93869]: pgmap v13903: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 274 KiB/s wr, 7 op/s 2023-03-09T21:56:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:56:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25111.343398 secs 2023-03-09T21:56:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:56:41 smithi181 ceph-mon[94189]: pgmap v13903: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 274 KiB/s wr, 7 op/s 2023-03-09T21:56:41.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:56:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25111.343398 secs 2023-03-09T21:56:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:56:41 smithi038 ceph-mon[92422]: pgmap v13903: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 274 KiB/s wr, 7 op/s 2023-03-09T21:56:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25575.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:56:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:56:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25113.698138 secs 2023-03-09T21:56:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:56:43 smithi122 ceph-mon[93869]: pgmap v13904: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 274 KiB/s wr, 9 op/s 2023-03-09T21:56:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:56:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25113.698138 secs 2023-03-09T21:56:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:56:43 smithi181 ceph-mon[94189]: pgmap v13904: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 274 KiB/s wr, 9 op/s 2023-03-09T21:56:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:56:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25113.698138 secs 2023-03-09T21:56:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:56:43 smithi038 ceph-mon[92422]: pgmap v13904: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 274 KiB/s wr, 9 op/s 2023-03-09T21:56:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:56:45 smithi122 ceph-mon[93869]: pgmap v13905: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 511 KiB/s rd, 274 KiB/s wr, 8 op/s 2023-03-09T21:56:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:56:45 smithi181 ceph-mon[94189]: pgmap v13905: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 511 KiB/s rd, 274 KiB/s wr, 8 op/s 2023-03-09T21:56:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:56:45 smithi038 ceph-mon[92422]: pgmap v13905: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 511 KiB/s rd, 274 KiB/s wr, 8 op/s 2023-03-09T21:56:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:56:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25116.343460 secs 2023-03-09T21:56:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:56:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25116.343460 secs 2023-03-09T21:56:46.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:56:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25116.343460 secs 2023-03-09T21:56:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:56:47 smithi122 ceph-mon[93869]: pgmap v13906: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 329 KiB/s wr, 10 op/s 2023-03-09T21:56:47.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:56:47 smithi181 ceph-mon[94189]: pgmap v13906: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 329 KiB/s wr, 10 op/s 2023-03-09T21:56:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:56:47 smithi038 ceph-mon[92422]: pgmap v13906: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 329 KiB/s wr, 10 op/s 2023-03-09T21:56:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25580.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:56:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:56:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25118.698245 secs 2023-03-09T21:56:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:56:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25118.698245 secs 2023-03-09T21:56:48.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:56:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25118.698245 secs 2023-03-09T21:56:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:56:49 smithi122 ceph-mon[93869]: pgmap v13907: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 567 KiB/s rd, 330 KiB/s wr, 8 op/s 2023-03-09T21:56:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:56:49 smithi181 ceph-mon[94189]: pgmap v13907: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 567 KiB/s rd, 330 KiB/s wr, 8 op/s 2023-03-09T21:56:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:56:49 smithi038 ceph-mon[92422]: pgmap v13907: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 567 KiB/s rd, 330 KiB/s wr, 8 op/s 2023-03-09T21:56:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:56:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25121.343544 secs 2023-03-09T21:56:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:56:51 smithi038 ceph-mon[92422]: pgmap v13908: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 637 KiB/s rd, 329 KiB/s wr, 8 op/s 2023-03-09T21:56:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:56:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25121.343544 secs 2023-03-09T21:56:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:56:51 smithi122 ceph-mon[93869]: pgmap v13908: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 637 KiB/s rd, 329 KiB/s wr, 8 op/s 2023-03-09T21:56:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:56:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25121.343544 secs 2023-03-09T21:56:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:56:51 smithi181 ceph-mon[94189]: pgmap v13908: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 637 KiB/s rd, 329 KiB/s wr, 8 op/s 2023-03-09T21:56:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25585.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:56:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:56:53 smithi038 ceph-mon[92422]: pgmap v13909: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 703 KiB/s rd, 221 KiB/s wr, 12 op/s 2023-03-09T21:56:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:56:53 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:56:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:56:53 smithi122 ceph-mon[93869]: pgmap v13909: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 703 KiB/s rd, 221 KiB/s wr, 12 op/s 2023-03-09T21:56:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:56:53 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:56:54.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:56:53 smithi181 ceph-mon[94189]: pgmap v13909: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 703 KiB/s rd, 221 KiB/s wr, 12 op/s 2023-03-09T21:56:54.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:56:53 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:56:54.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:56:54 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:56:54.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:56:54 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:56:54.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:56:54 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:56:55.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:56:54 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:56:55.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:56:54 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:56:55.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:56:54 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:56:55.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:56:54 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:56:55.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:56:54 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:56:55.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:56:54 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:56:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:56:55 smithi038 ceph-mon[92422]: pgmap v13910: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 787 KiB/s rd, 307 KiB/s wr, 10 op/s 2023-03-09T21:56:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:56:55 smithi122 ceph-mon[93869]: pgmap v13910: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 787 KiB/s rd, 307 KiB/s wr, 10 op/s 2023-03-09T21:56:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:56:55 smithi181 ceph-mon[94189]: pgmap v13910: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 787 KiB/s rd, 307 KiB/s wr, 10 op/s 2023-03-09T21:56:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:56:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25126.343602 secs 2023-03-09T21:56:57.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:56:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25126.343602 secs 2023-03-09T21:56:57.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:56:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25126.343602 secs 2023-03-09T21:56:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:56:57 smithi122 ceph-mon[93869]: pgmap v13911: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 488 KiB/s wr, 12 op/s 2023-03-09T21:56:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:56:57 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:56:58.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:56:57 smithi181 ceph-mon[94189]: pgmap v13911: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 488 KiB/s wr, 12 op/s 2023-03-09T21:56:58.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:56:57 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:56:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:56:57 smithi038 ceph-mon[92422]: pgmap v13911: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 488 KiB/s wr, 12 op/s 2023-03-09T21:56:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:56:57 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:56:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25590.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:57:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:56:59 smithi122 ceph-mon[93869]: pgmap v13912: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 531 KiB/s rd, 586 KiB/s wr, 11 op/s 2023-03-09T21:57:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:56:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:57:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:56:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:57:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:56:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:57:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:56:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:57:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:56:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:57:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:56:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:57:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:56:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:57:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:56:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:57:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:56:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:57:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:56:59 smithi181 ceph-mon[94189]: pgmap v13912: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 531 KiB/s rd, 586 KiB/s wr, 11 op/s 2023-03-09T21:57:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:56:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:57:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:56:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:57:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:56:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:57:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:56:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:57:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:56:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:57:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:56:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:57:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:56:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:57:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:56:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:57:00.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:56:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:57:00.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:56:59 smithi038 ceph-mon[92422]: pgmap v13912: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 531 KiB/s rd, 586 KiB/s wr, 11 op/s 2023-03-09T21:57:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:56:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:57:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:56:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:57:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:56:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:57:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:56:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:57:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:56:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:57:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:56:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:57:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:56:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:57:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:56:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:57:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:56:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:57:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:57:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25131.343679 secs 2023-03-09T21:57:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:57:01 smithi122 ceph-mon[93869]: pgmap v13913: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 531 KiB/s rd, 586 KiB/s wr, 11 op/s 2023-03-09T21:57:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:57:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25131.343679 secs 2023-03-09T21:57:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:57:01 smithi181 ceph-mon[94189]: pgmap v13913: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 531 KiB/s rd, 586 KiB/s wr, 11 op/s 2023-03-09T21:57:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:57:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25131.343679 secs 2023-03-09T21:57:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:57:01 smithi038 ceph-mon[92422]: pgmap v13913: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 531 KiB/s rd, 586 KiB/s wr, 11 op/s 2023-03-09T21:57:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25595.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:57:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:57:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25133.698582 secs 2023-03-09T21:57:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:57:03 smithi122 ceph-mon[93869]: pgmap v13914: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 560 KiB/s rd, 706 KiB/s wr, 17 op/s 2023-03-09T21:57:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:57:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25133.698582 secs 2023-03-09T21:57:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:57:03 smithi181 ceph-mon[94189]: pgmap v13914: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 560 KiB/s rd, 706 KiB/s wr, 17 op/s 2023-03-09T21:57:03.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:57:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25133.698582 secs 2023-03-09T21:57:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:57:03 smithi038 ceph-mon[92422]: pgmap v13914: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 560 KiB/s rd, 706 KiB/s wr, 17 op/s 2023-03-09T21:57:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:57:05 smithi122 ceph-mon[93869]: pgmap v13915: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 630 KiB/s rd, 665 KiB/s wr, 26 op/s 2023-03-09T21:57:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:57:05 smithi122 ceph-mon[93869]: osdmap e1083: 12 total, 12 up, 12 in 2023-03-09T21:57:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:57:05 smithi181 ceph-mon[94189]: pgmap v13915: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 630 KiB/s rd, 665 KiB/s wr, 26 op/s 2023-03-09T21:57:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:57:05 smithi181 ceph-mon[94189]: osdmap e1083: 12 total, 12 up, 12 in 2023-03-09T21:57:05.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:57:05 smithi038 ceph-mon[92422]: pgmap v13915: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 630 KiB/s rd, 665 KiB/s wr, 26 op/s 2023-03-09T21:57:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:57:05 smithi038 ceph-mon[92422]: osdmap e1083: 12 total, 12 up, 12 in 2023-03-09T21:57:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:57:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25136.343768 secs 2023-03-09T21:57:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:57:06 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:57:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:57:06 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:57:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:57:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:57:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:57:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25136.343768 secs 2023-03-09T21:57:06.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:57:06 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:57:06.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:57:06 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:57:06.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:57:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:57:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:57:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25136.343768 secs 2023-03-09T21:57:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:57:06 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:57:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:57:06 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:57:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:57:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:57:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:57:07 smithi122 ceph-mon[93869]: pgmap v13917: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 715 KiB/s rd, 757 KiB/s wr, 30 op/s 2023-03-09T21:57:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:57:07 smithi181 ceph-mon[94189]: pgmap v13917: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 715 KiB/s rd, 757 KiB/s wr, 30 op/s 2023-03-09T21:57:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:57:07 smithi038 ceph-mon[92422]: pgmap v13917: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 715 KiB/s rd, 757 KiB/s wr, 30 op/s 2023-03-09T21:57:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25600.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:57:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:57:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25138.698717 secs 2023-03-09T21:57:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:57:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25138.698717 secs 2023-03-09T21:57:08.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:57:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25138.698717 secs 2023-03-09T21:57:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:57:09 smithi122 ceph-mon[93869]: pgmap v13918: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 451 KiB/s wr, 29 op/s 2023-03-09T21:57:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:57:09 smithi181 ceph-mon[94189]: pgmap v13918: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 451 KiB/s wr, 29 op/s 2023-03-09T21:57:09.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:57:09 smithi038 ceph-mon[92422]: pgmap v13918: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 451 KiB/s wr, 29 op/s 2023-03-09T21:57:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:57:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25141.343830 secs 2023-03-09T21:57:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:57:11 smithi122 ceph-mon[93869]: pgmap v13919: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 451 KiB/s wr, 28 op/s 2023-03-09T21:57:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:57:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25141.343830 secs 2023-03-09T21:57:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:57:11 smithi181 ceph-mon[94189]: pgmap v13919: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 451 KiB/s wr, 28 op/s 2023-03-09T21:57:11.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:57:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25141.343830 secs 2023-03-09T21:57:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:57:11 smithi038 ceph-mon[92422]: pgmap v13919: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 451 KiB/s wr, 28 op/s 2023-03-09T21:57:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25605.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:57:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:57:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25143.698823 secs 2023-03-09T21:57:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:57:13 smithi122 ceph-mon[93869]: pgmap v13920: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 530 KiB/s rd, 391 KiB/s wr, 24 op/s 2023-03-09T21:57:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:57:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25143.698823 secs 2023-03-09T21:57:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:57:13 smithi181 ceph-mon[94189]: pgmap v13920: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 530 KiB/s rd, 391 KiB/s wr, 24 op/s 2023-03-09T21:57:13.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:57:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25143.698823 secs 2023-03-09T21:57:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:57:13 smithi038 ceph-mon[92422]: pgmap v13920: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 530 KiB/s rd, 391 KiB/s wr, 24 op/s 2023-03-09T21:57:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:57:15 smithi122 ceph-mon[93869]: pgmap v13921: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 366 KiB/s wr, 9 op/s 2023-03-09T21:57:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:57:15 smithi122 ceph-mon[93869]: osdmap e1084: 12 total, 12 up, 12 in 2023-03-09T21:57:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:57:15 smithi181 ceph-mon[94189]: pgmap v13921: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 366 KiB/s wr, 9 op/s 2023-03-09T21:57:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:57:15 smithi181 ceph-mon[94189]: osdmap e1084: 12 total, 12 up, 12 in 2023-03-09T21:57:15.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:57:15 smithi038 ceph-mon[92422]: pgmap v13921: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 366 KiB/s wr, 9 op/s 2023-03-09T21:57:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:57:15 smithi038 ceph-mon[92422]: osdmap e1084: 12 total, 12 up, 12 in 2023-03-09T21:57:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:57:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25146.343885 secs 2023-03-09T21:57:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:57:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25146.343885 secs 2023-03-09T21:57:16.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:57:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25146.343885 secs 2023-03-09T21:57:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:57:17 smithi122 ceph-mon[93869]: pgmap v13923: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 85 KiB/s wr, 8 op/s 2023-03-09T21:57:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:57:17 smithi181 ceph-mon[94189]: pgmap v13923: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 85 KiB/s wr, 8 op/s 2023-03-09T21:57:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:57:17 smithi038 ceph-mon[92422]: pgmap v13923: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 85 KiB/s wr, 8 op/s 2023-03-09T21:57:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25610.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:57:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:57:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25148.698933 secs 2023-03-09T21:57:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:57:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25148.698933 secs 2023-03-09T21:57:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:57:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25148.698933 secs 2023-03-09T21:57:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:57:19 smithi122 ceph-mon[93869]: pgmap v13924: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 85 KiB/s wr, 7 op/s 2023-03-09T21:57:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:57:19 smithi181 ceph-mon[94189]: pgmap v13924: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 85 KiB/s wr, 7 op/s 2023-03-09T21:57:19.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:57:19 smithi038 ceph-mon[92422]: pgmap v13924: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 85 KiB/s wr, 7 op/s 2023-03-09T21:57:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:57:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25151.343948 secs 2023-03-09T21:57:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:57:21 smithi122 ceph-mon[93869]: pgmap v13925: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 85 KiB/s wr, 7 op/s 2023-03-09T21:57:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:57:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25151.343948 secs 2023-03-09T21:57:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:57:21 smithi181 ceph-mon[94189]: pgmap v13925: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 85 KiB/s wr, 7 op/s 2023-03-09T21:57:21.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:57:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25151.343948 secs 2023-03-09T21:57:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:57:21 smithi038 ceph-mon[92422]: pgmap v13925: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 85 KiB/s wr, 7 op/s 2023-03-09T21:57:23.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25615.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:57:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:57:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25153.699049 secs 2023-03-09T21:57:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:57:23 smithi122 ceph-mon[93869]: pgmap v13926: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T21:57:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:57:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25153.699049 secs 2023-03-09T21:57:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:57:23 smithi181 ceph-mon[94189]: pgmap v13926: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T21:57:23.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:57:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25153.699049 secs 2023-03-09T21:57:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:57:23 smithi038 ceph-mon[92422]: pgmap v13926: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T21:57:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:57:25 smithi122 ceph-mon[93869]: pgmap v13927: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T21:57:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:57:25 smithi181 ceph-mon[94189]: pgmap v13927: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T21:57:25.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:57:25 smithi038 ceph-mon[92422]: pgmap v13927: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T21:57:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:57:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25156.344008 secs 2023-03-09T21:57:26.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:57:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25156.344008 secs 2023-03-09T21:57:26.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:57:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25156.344008 secs 2023-03-09T21:57:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:57:27 smithi122 ceph-mon[93869]: pgmap v13928: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 9 op/s 2023-03-09T21:57:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:57:27 smithi181 ceph-mon[94189]: pgmap v13928: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 9 op/s 2023-03-09T21:57:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:57:27 smithi038 ceph-mon[92422]: pgmap v13928: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 9 op/s 2023-03-09T21:57:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25620.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:57:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:57:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25158.699190 secs 2023-03-09T21:57:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:57:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25158.699190 secs 2023-03-09T21:57:29.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:57:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25158.699190 secs 2023-03-09T21:57:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:57:29 smithi038 ceph-mon[92422]: pgmap v13929: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T21:57:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:57:29 smithi122 ceph-mon[93869]: pgmap v13929: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T21:57:30.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:57:29 smithi181 ceph-mon[94189]: pgmap v13929: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T21:57:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:57:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25161.344068 secs 2023-03-09T21:57:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:57:31 smithi038 ceph-mon[92422]: pgmap v13930: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T21:57:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:57:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25161.344068 secs 2023-03-09T21:57:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:57:31 smithi122 ceph-mon[93869]: pgmap v13930: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T21:57:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:57:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25161.344068 secs 2023-03-09T21:57:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:57:31 smithi181 ceph-mon[94189]: pgmap v13930: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T21:57:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25625.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:57:33.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:57:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25163.699311 secs 2023-03-09T21:57:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:57:33 smithi038 ceph-mon[92422]: pgmap v13931: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T21:57:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:57:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25163.699311 secs 2023-03-09T21:57:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:57:33 smithi122 ceph-mon[93869]: pgmap v13931: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T21:57:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:57:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25163.699311 secs 2023-03-09T21:57:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:57:33 smithi181 ceph-mon[94189]: pgmap v13931: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T21:57:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:57:35 smithi038 ceph-mon[92422]: pgmap v13932: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-09T21:57:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:57:35 smithi122 ceph-mon[93869]: pgmap v13932: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-09T21:57:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:57:35 smithi181 ceph-mon[94189]: pgmap v13932: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-09T21:57:37.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:57:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25166.344130 secs 2023-03-09T21:57:37.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:57:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25166.344130 secs 2023-03-09T21:57:37.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:57:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25166.344130 secs 2023-03-09T21:57:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:57:37 smithi122 ceph-mon[93869]: pgmap v13933: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 511 B/s wr, 9 op/s 2023-03-09T21:57:38.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:57:37 smithi181 ceph-mon[94189]: pgmap v13933: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 511 B/s wr, 9 op/s 2023-03-09T21:57:38.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:57:37 smithi038 ceph-mon[92422]: pgmap v13933: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 511 B/s wr, 9 op/s 2023-03-09T21:57:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25630.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:57:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:57:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25168.699424 secs 2023-03-09T21:57:39.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:57:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25168.699424 secs 2023-03-09T21:57:39.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:57:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25168.699424 secs 2023-03-09T21:57:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:57:39 smithi122 ceph-mon[93869]: pgmap v13934: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 597 B/s wr, 6 op/s 2023-03-09T21:57:40.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:57:39 smithi181 ceph-mon[94189]: pgmap v13934: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 597 B/s wr, 6 op/s 2023-03-09T21:57:40.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:57:39 smithi038 ceph-mon[92422]: pgmap v13934: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 597 B/s wr, 6 op/s 2023-03-09T21:57:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:57:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25171.344190 secs 2023-03-09T21:57:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:57:41 smithi122 ceph-mon[93869]: pgmap v13935: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 597 B/s wr, 6 op/s 2023-03-09T21:57:42.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:57:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25171.344190 secs 2023-03-09T21:57:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:57:41 smithi181 ceph-mon[94189]: pgmap v13935: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 597 B/s wr, 6 op/s 2023-03-09T21:57:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:57:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25171.344190 secs 2023-03-09T21:57:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:57:41 smithi038 ceph-mon[92422]: pgmap v13935: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 597 B/s wr, 6 op/s 2023-03-09T21:57:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25635.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:57:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:57:43 smithi122 ceph-mon[93869]: pgmap v13936: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 132 KiB/s wr, 9 op/s 2023-03-09T21:57:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:57:43 smithi181 ceph-mon[94189]: pgmap v13936: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 132 KiB/s wr, 9 op/s 2023-03-09T21:57:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:57:43 smithi038 ceph-mon[92422]: pgmap v13936: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 132 KiB/s wr, 9 op/s 2023-03-09T21:57:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:57:45 smithi122 ceph-mon[93869]: pgmap v13937: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 132 KiB/s wr, 7 op/s 2023-03-09T21:57:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:57:45 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:57:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:57:45 smithi181 ceph-mon[94189]: pgmap v13937: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 132 KiB/s wr, 7 op/s 2023-03-09T21:57:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:57:45 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:57:46.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:57:45 smithi038 ceph-mon[92422]: pgmap v13937: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 132 KiB/s wr, 7 op/s 2023-03-09T21:57:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:57:45 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:57:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:57:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25176.344251 secs 2023-03-09T21:57:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:57:46 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:57:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:57:46 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:57:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:57:46 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:57:47.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:57:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25176.344251 secs 2023-03-09T21:57:47.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:57:46 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:57:47.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:57:46 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:57:47.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:57:46 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:57:47.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:57:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25176.344251 secs 2023-03-09T21:57:47.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:57:46 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:57:47.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:57:46 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:57:47.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:57:46 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:57:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:57:47 smithi122 ceph-mon[93869]: pgmap v13938: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 344 KiB/s wr, 10 op/s 2023-03-09T21:57:48.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:57:47 smithi181 ceph-mon[94189]: pgmap v13938: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 344 KiB/s wr, 10 op/s 2023-03-09T21:57:48.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:57:47 smithi038 ceph-mon[92422]: pgmap v13938: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 344 KiB/s wr, 10 op/s 2023-03-09T21:57:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25640.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:57:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:57:49 smithi122 ceph-mon[93869]: pgmap v13939: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 349 KiB/s wr, 7 op/s 2023-03-09T21:57:50.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:57:49 smithi181 ceph-mon[94189]: pgmap v13939: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 349 KiB/s wr, 7 op/s 2023-03-09T21:57:50.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:57:49 smithi038 ceph-mon[92422]: pgmap v13939: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 349 KiB/s wr, 7 op/s 2023-03-09T21:57:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:57:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25181.344311 secs 2023-03-09T21:57:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:57:51 smithi122 ceph-mon[93869]: pgmap v13940: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 362 KiB/s rd, 348 KiB/s wr, 7 op/s 2023-03-09T21:57:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:57:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25181.344311 secs 2023-03-09T21:57:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:57:51 smithi181 ceph-mon[94189]: pgmap v13940: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 362 KiB/s rd, 348 KiB/s wr, 7 op/s 2023-03-09T21:57:52.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:57:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25181.344311 secs 2023-03-09T21:57:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:57:51 smithi038 ceph-mon[92422]: pgmap v13940: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 362 KiB/s rd, 348 KiB/s wr, 7 op/s 2023-03-09T21:57:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25645.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:57:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:57:53 smithi122 ceph-mon[93869]: pgmap v13941: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 412 KiB/s wr, 11 op/s 2023-03-09T21:57:54.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:57:53 smithi181 ceph-mon[94189]: pgmap v13941: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 412 KiB/s wr, 11 op/s 2023-03-09T21:57:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:57:53 smithi038 ceph-mon[92422]: pgmap v13941: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 412 KiB/s wr, 11 op/s 2023-03-09T21:57:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:57:55 smithi122 ceph-mon[93869]: pgmap v13942: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 281 KiB/s wr, 9 op/s 2023-03-09T21:57:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:57:55 smithi181 ceph-mon[94189]: pgmap v13942: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 281 KiB/s wr, 9 op/s 2023-03-09T21:57:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:57:55 smithi038 ceph-mon[92422]: pgmap v13942: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 281 KiB/s wr, 9 op/s 2023-03-09T21:57:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:57:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25186.344370 secs 2023-03-09T21:57:57.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:57:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25186.344370 secs 2023-03-09T21:57:57.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:57:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25186.344370 secs 2023-03-09T21:57:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:57:57 smithi122 ceph-mon[93869]: pgmap v13943: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 301 KiB/s wr, 12 op/s 2023-03-09T21:57:58.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:57:57 smithi181 ceph-mon[94189]: pgmap v13943: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 301 KiB/s wr, 12 op/s 2023-03-09T21:57:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:57:57 smithi038 ceph-mon[92422]: pgmap v13943: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 301 KiB/s wr, 12 op/s 2023-03-09T21:57:58.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25650.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:57:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:57:58 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 25188.699888 secs 2023-03-09T21:57:59.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:57:58 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 25188.699888 secs 2023-03-09T21:57:59.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:57:58 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 25188.699888 secs 2023-03-09T21:58:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:57:59 smithi122 ceph-mon[93869]: pgmap v13944: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 465 KiB/s wr, 11 op/s 2023-03-09T21:58:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:57:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:58:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:57:59 smithi181 ceph-mon[94189]: pgmap v13944: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 465 KiB/s wr, 11 op/s 2023-03-09T21:58:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:57:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:58:00.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:57:59 smithi038 ceph-mon[92422]: pgmap v13944: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 465 KiB/s wr, 11 op/s 2023-03-09T21:58:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:57:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:58:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25191.344431 secs 2023-03-09T21:58:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:01 smithi122 ceph-mon[93869]: pgmap v13945: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 474 KiB/s rd, 459 KiB/s wr, 10 op/s 2023-03-09T21:58:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:01 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:58:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:58:02.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:58:02.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25191.344431 secs 2023-03-09T21:58:02.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:01 smithi181 ceph-mon[94189]: pgmap v13945: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 474 KiB/s rd, 459 KiB/s wr, 10 op/s 2023-03-09T21:58:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:01 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:58:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:58:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:58:02.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25191.344431 secs 2023-03-09T21:58:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:01 smithi038 ceph-mon[92422]: pgmap v13945: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 474 KiB/s rd, 459 KiB/s wr, 10 op/s 2023-03-09T21:58:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:01 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:58:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:58:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:58:03.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:58:03.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:58:03.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:58:03.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:58:03.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:58:03.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:58:03.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:58:03.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:58:03.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:58:03.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:58:03.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:58:03.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:58:03.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:58:03.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:58:03.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:58:03.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:58:03.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:58:03.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:58:03.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:58:03.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:58:03.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:58:03.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:58:03.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:58:03.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:58:03.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:58:03.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:58:03.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:58:03.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:58:03.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:58:03.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:58:03.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:58:03.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:58:03.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:58:03.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:58:03.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:58:03.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:58:03.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:58:03.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:58:03.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:58:03.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:58:03.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:58:03.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:58:03.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25655.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:58:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:03 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 25193.699980 secs 2023-03-09T21:58:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:03 smithi122 ceph-mon[93869]: pgmap v13946: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 507 KiB/s rd, 503 KiB/s wr, 14 op/s 2023-03-09T21:58:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:03 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T21:58:04.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:03 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T21:58:04.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:03 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi038 to 30061k 2023-03-09T21:58:04.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:03 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi038 to 30783422: error parsing value: Value '30783422' is below minimum 939524096 2023-03-09T21:58:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:03 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 25193.699980 secs 2023-03-09T21:58:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:03 smithi181 ceph-mon[94189]: pgmap v13946: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 507 KiB/s rd, 503 KiB/s wr, 14 op/s 2023-03-09T21:58:04.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:03 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T21:58:04.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:03 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T21:58:04.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:03 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi038 to 30061k 2023-03-09T21:58:04.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:03 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi038 to 30783422: error parsing value: Value '30783422' is below minimum 939524096 2023-03-09T21:58:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:03 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 25193.699980 secs 2023-03-09T21:58:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:03 smithi038 ceph-mon[92422]: pgmap v13946: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 507 KiB/s rd, 503 KiB/s wr, 14 op/s 2023-03-09T21:58:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:03 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T21:58:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:03 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T21:58:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:03 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi038 to 30061k 2023-03-09T21:58:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:03 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi038 to 30783422: error parsing value: Value '30783422' is below minimum 939524096 2023-03-09T21:58:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:05 smithi122 ceph-mon[93869]: pgmap v13947: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 451 KiB/s wr, 21 op/s 2023-03-09T21:58:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25196.344492 secs 2023-03-09T21:58:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:05 smithi181 ceph-mon[94189]: pgmap v13947: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 451 KiB/s wr, 21 op/s 2023-03-09T21:58:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25196.344492 secs 2023-03-09T21:58:06.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:05 smithi038 ceph-mon[92422]: pgmap v13947: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 451 KiB/s wr, 21 op/s 2023-03-09T21:58:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25196.344492 secs 2023-03-09T21:58:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:07 smithi122 ceph-mon[93869]: pgmap v13948: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 615 KiB/s rd, 654 KiB/s wr, 25 op/s 2023-03-09T21:58:08.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:07 smithi181 ceph-mon[94189]: pgmap v13948: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 615 KiB/s rd, 654 KiB/s wr, 25 op/s 2023-03-09T21:58:08.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:07 smithi038 ceph-mon[92422]: pgmap v13948: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 615 KiB/s rd, 654 KiB/s wr, 25 op/s 2023-03-09T21:58:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25660.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:58:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:08 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 25198.700078 secs 2023-03-09T21:58:09.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:08 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 25198.700078 secs 2023-03-09T21:58:09.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:08 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 25198.700078 secs 2023-03-09T21:58:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:09 smithi122 ceph-mon[93869]: pgmap v13949: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 810 KiB/s rd, 850 KiB/s wr, 24 op/s 2023-03-09T21:58:10.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:09 smithi181 ceph-mon[94189]: pgmap v13949: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 810 KiB/s rd, 850 KiB/s wr, 24 op/s 2023-03-09T21:58:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:09 smithi038 ceph-mon[92422]: pgmap v13949: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 810 KiB/s rd, 850 KiB/s wr, 24 op/s 2023-03-09T21:58:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25201.344565 secs 2023-03-09T21:58:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:11 smithi122 ceph-mon[93869]: pgmap v13950: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 435 KiB/s rd, 475 KiB/s wr, 22 op/s 2023-03-09T21:58:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25201.344565 secs 2023-03-09T21:58:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:11 smithi181 ceph-mon[94189]: pgmap v13950: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 435 KiB/s rd, 475 KiB/s wr, 22 op/s 2023-03-09T21:58:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25201.344565 secs 2023-03-09T21:58:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:11 smithi038 ceph-mon[92422]: pgmap v13950: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 435 KiB/s rd, 475 KiB/s wr, 22 op/s 2023-03-09T21:58:13.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25665.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:58:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:13 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 25203.700220 secs 2023-03-09T21:58:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:13 smithi122 ceph-mon[93869]: pgmap v13951: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 545 KiB/s wr, 25 op/s 2023-03-09T21:58:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:13 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 25203.700220 secs 2023-03-09T21:58:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:13 smithi181 ceph-mon[94189]: pgmap v13951: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 545 KiB/s wr, 25 op/s 2023-03-09T21:58:13.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:13 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 25203.700220 secs 2023-03-09T21:58:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:13 smithi038 ceph-mon[92422]: pgmap v13951: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 545 KiB/s wr, 25 op/s 2023-03-09T21:58:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:15 smithi122 ceph-mon[93869]: pgmap v13952: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 502 KiB/s wr, 21 op/s 2023-03-09T21:58:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:15 smithi181 ceph-mon[94189]: pgmap v13952: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 502 KiB/s wr, 21 op/s 2023-03-09T21:58:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:15 smithi038 ceph-mon[92422]: pgmap v13952: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 502 KiB/s wr, 21 op/s 2023-03-09T21:58:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25206.344646 secs 2023-03-09T21:58:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25206.344646 secs 2023-03-09T21:58:16.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25206.344646 secs 2023-03-09T21:58:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:17 smithi122 ceph-mon[93869]: pgmap v13953: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 603 KiB/s rd, 611 KiB/s wr, 13 op/s 2023-03-09T21:58:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:17 smithi181 ceph-mon[94189]: pgmap v13953: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 603 KiB/s rd, 611 KiB/s wr, 13 op/s 2023-03-09T21:58:17.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:17 smithi038 ceph-mon[92422]: pgmap v13953: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 603 KiB/s rd, 611 KiB/s wr, 13 op/s 2023-03-09T21:58:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25670.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:58:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:18 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 25208.700316 secs 2023-03-09T21:58:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:18 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 25208.700316 secs 2023-03-09T21:58:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:18 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 25208.700316 secs 2023-03-09T21:58:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:19 smithi122 ceph-mon[93869]: pgmap v13954: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 436 KiB/s rd, 407 KiB/s wr, 8 op/s 2023-03-09T21:58:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:58:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:58:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:58:19.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:58:19.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:58:19.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:58:19.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:58:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:19 smithi181 ceph-mon[94189]: pgmap v13954: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 436 KiB/s rd, 407 KiB/s wr, 8 op/s 2023-03-09T21:58:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:58:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:58:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:58:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:58:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:58:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:58:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:58:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:19 smithi038 ceph-mon[92422]: pgmap v13954: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 436 KiB/s rd, 407 KiB/s wr, 8 op/s 2023-03-09T21:58:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:58:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:58:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:58:20.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T21:58:20.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:58:20.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:58:20.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:58:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:20 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T21:58:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:20 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T21:58:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:20 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T21:58:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:20 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T21:58:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:20 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T21:58:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:20 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T21:58:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25211.344731 secs 2023-03-09T21:58:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:21 smithi122 ceph-mon[93869]: pgmap v13955: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 190 KiB/s wr, 7 op/s 2023-03-09T21:58:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25211.344731 secs 2023-03-09T21:58:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:21 smithi181 ceph-mon[94189]: pgmap v13955: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 190 KiB/s wr, 7 op/s 2023-03-09T21:58:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25211.344731 secs 2023-03-09T21:58:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:21 smithi038 ceph-mon[92422]: pgmap v13955: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 190 KiB/s wr, 7 op/s 2023-03-09T21:58:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25675.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:58:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25213.700458 secs 2023-03-09T21:58:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:23 smithi122 ceph-mon[93869]: pgmap v13956: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 335 KiB/s wr, 11 op/s 2023-03-09T21:58:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25213.700458 secs 2023-03-09T21:58:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:23 smithi181 ceph-mon[94189]: pgmap v13956: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 335 KiB/s wr, 11 op/s 2023-03-09T21:58:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25213.700458 secs 2023-03-09T21:58:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:23 smithi038 ceph-mon[92422]: pgmap v13956: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 335 KiB/s wr, 11 op/s 2023-03-09T21:58:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:25 smithi122 ceph-mon[93869]: pgmap v13957: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 275 KiB/s wr, 9 op/s 2023-03-09T21:58:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:25 smithi181 ceph-mon[94189]: pgmap v13957: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 275 KiB/s wr, 9 op/s 2023-03-09T21:58:25.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:25 smithi038 ceph-mon[92422]: pgmap v13957: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 275 KiB/s wr, 9 op/s 2023-03-09T21:58:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25216.344792 secs 2023-03-09T21:58:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:26 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:58:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:26 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:58:26.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25216.344792 secs 2023-03-09T21:58:26.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:26 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:58:26.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:26 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:58:26.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25216.344792 secs 2023-03-09T21:58:26.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:26 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:58:26.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:26 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:58:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:27 smithi122 ceph-mon[93869]: pgmap v13958: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 361 KiB/s wr, 13 op/s 2023-03-09T21:58:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:27 smithi181 ceph-mon[94189]: pgmap v13958: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 361 KiB/s wr, 13 op/s 2023-03-09T21:58:27.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:27 smithi038 ceph-mon[92422]: pgmap v13958: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 361 KiB/s wr, 13 op/s 2023-03-09T21:58:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25680.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:58:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25218.700572 secs 2023-03-09T21:58:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25218.700572 secs 2023-03-09T21:58:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25218.700572 secs 2023-03-09T21:58:29.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:29 smithi038 ceph-mon[92422]: pgmap v13959: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 242 KiB/s wr, 10 op/s 2023-03-09T21:58:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:29 smithi122 ceph-mon[93869]: pgmap v13959: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 242 KiB/s wr, 10 op/s 2023-03-09T21:58:30.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:29 smithi181 ceph-mon[94189]: pgmap v13959: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 242 KiB/s wr, 10 op/s 2023-03-09T21:58:30.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:30 smithi038 ceph-mon[92422]: osdmap e1085: 12 total, 12 up, 12 in 2023-03-09T21:58:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:30 smithi122 ceph-mon[93869]: osdmap e1085: 12 total, 12 up, 12 in 2023-03-09T21:58:31.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:30 smithi181 ceph-mon[94189]: osdmap e1085: 12 total, 12 up, 12 in 2023-03-09T21:58:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25221.344872 secs 2023-03-09T21:58:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:31 smithi038 ceph-mon[92422]: pgmap v13961: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 290 KiB/s wr, 12 op/s 2023-03-09T21:58:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25221.344872 secs 2023-03-09T21:58:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:31 smithi122 ceph-mon[93869]: pgmap v13961: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 290 KiB/s wr, 12 op/s 2023-03-09T21:58:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25221.344872 secs 2023-03-09T21:58:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:31 smithi181 ceph-mon[94189]: pgmap v13961: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 290 KiB/s wr, 12 op/s 2023-03-09T21:58:33.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25685.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:58:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25223.700686 secs 2023-03-09T21:58:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:33 smithi122 ceph-mon[93869]: pgmap v13962: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 279 KiB/s wr, 10 op/s 2023-03-09T21:58:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25223.700686 secs 2023-03-09T21:58:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:33 smithi181 ceph-mon[94189]: pgmap v13962: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 279 KiB/s wr, 10 op/s 2023-03-09T21:58:34.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25223.700686 secs 2023-03-09T21:58:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:33 smithi038 ceph-mon[92422]: pgmap v13962: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 279 KiB/s wr, 10 op/s 2023-03-09T21:58:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:35 smithi122 ceph-mon[93869]: pgmap v13963: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 267 KiB/s wr, 8 op/s 2023-03-09T21:58:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:35 smithi122 ceph-mon[93869]: osdmap e1086: 12 total, 12 up, 12 in 2023-03-09T21:58:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:35 smithi181 ceph-mon[94189]: pgmap v13963: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 267 KiB/s wr, 8 op/s 2023-03-09T21:58:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:35 smithi181 ceph-mon[94189]: osdmap e1086: 12 total, 12 up, 12 in 2023-03-09T21:58:36.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:35 smithi038 ceph-mon[92422]: pgmap v13963: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 267 KiB/s wr, 8 op/s 2023-03-09T21:58:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:35 smithi038 ceph-mon[92422]: osdmap e1086: 12 total, 12 up, 12 in 2023-03-09T21:58:37.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25226.344931 secs 2023-03-09T21:58:37.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25226.344931 secs 2023-03-09T21:58:37.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25226.344931 secs 2023-03-09T21:58:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:37 smithi122 ceph-mon[93869]: pgmap v13965: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 218 KiB/s wr, 10 op/s 2023-03-09T21:58:38.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:37 smithi181 ceph-mon[94189]: pgmap v13965: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 218 KiB/s wr, 10 op/s 2023-03-09T21:58:38.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:37 smithi038 ceph-mon[92422]: pgmap v13965: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 218 KiB/s wr, 10 op/s 2023-03-09T21:58:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25690.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:58:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25228.700774 secs 2023-03-09T21:58:39.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25228.700774 secs 2023-03-09T21:58:39.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25228.700774 secs 2023-03-09T21:58:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:39 smithi122 ceph-mon[93869]: pgmap v13966: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 202 KiB/s wr, 10 op/s 2023-03-09T21:58:40.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:39 smithi181 ceph-mon[94189]: pgmap v13966: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 202 KiB/s wr, 10 op/s 2023-03-09T21:58:40.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:39 smithi038 ceph-mon[92422]: pgmap v13966: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 202 KiB/s wr, 10 op/s 2023-03-09T21:58:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25231.344983 secs 2023-03-09T21:58:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:41 smithi122 ceph-mon[93869]: pgmap v13967: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 373 KiB/s wr, 9 op/s 2023-03-09T21:58:42.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25231.344983 secs 2023-03-09T21:58:42.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:41 smithi181 ceph-mon[94189]: pgmap v13967: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 373 KiB/s wr, 9 op/s 2023-03-09T21:58:42.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25231.344983 secs 2023-03-09T21:58:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:41 smithi038 ceph-mon[92422]: pgmap v13967: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 373 KiB/s wr, 9 op/s 2023-03-09T21:58:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25695.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:58:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25233.700909 secs 2023-03-09T21:58:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:43 smithi122 ceph-mon[93869]: pgmap v13968: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 234 KiB/s wr, 9 op/s 2023-03-09T21:58:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25233.700909 secs 2023-03-09T21:58:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:43 smithi181 ceph-mon[94189]: pgmap v13968: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 234 KiB/s wr, 9 op/s 2023-03-09T21:58:44.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25233.700909 secs 2023-03-09T21:58:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:43 smithi038 ceph-mon[92422]: pgmap v13968: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 234 KiB/s wr, 9 op/s 2023-03-09T21:58:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:45 smithi122 ceph-mon[93869]: pgmap v13969: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 471 KiB/s rd, 455 KiB/s wr, 11 op/s 2023-03-09T21:58:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:45 smithi181 ceph-mon[94189]: pgmap v13969: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 471 KiB/s rd, 455 KiB/s wr, 11 op/s 2023-03-09T21:58:46.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:45 smithi038 ceph-mon[92422]: pgmap v13969: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 471 KiB/s rd, 455 KiB/s wr, 11 op/s 2023-03-09T21:58:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25236.345063 secs 2023-03-09T21:58:47.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25236.345063 secs 2023-03-09T21:58:47.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25236.345063 secs 2023-03-09T21:58:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:47 smithi122 ceph-mon[93869]: pgmap v13970: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 447 KiB/s wr, 13 op/s 2023-03-09T21:58:48.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:47 smithi181 ceph-mon[94189]: pgmap v13970: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 447 KiB/s wr, 13 op/s 2023-03-09T21:58:48.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:47 smithi038 ceph-mon[92422]: pgmap v13970: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 447 KiB/s wr, 13 op/s 2023-03-09T21:58:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25700.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:58:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25238.701023 secs 2023-03-09T21:58:49.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25238.701023 secs 2023-03-09T21:58:49.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25238.701023 secs 2023-03-09T21:58:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:49 smithi122 ceph-mon[93869]: pgmap v13971: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 683 KiB/s wr, 10 op/s 2023-03-09T21:58:50.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:49 smithi181 ceph-mon[94189]: pgmap v13971: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 683 KiB/s wr, 10 op/s 2023-03-09T21:58:50.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:49 smithi038 ceph-mon[92422]: pgmap v13971: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 683 KiB/s wr, 10 op/s 2023-03-09T21:58:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25241.345123 secs 2023-03-09T21:58:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:51 smithi122 ceph-mon[93869]: pgmap v13972: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 683 KiB/s wr, 10 op/s 2023-03-09T21:58:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25241.345123 secs 2023-03-09T21:58:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:51 smithi181 ceph-mon[94189]: pgmap v13972: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 683 KiB/s wr, 10 op/s 2023-03-09T21:58:52.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25241.345123 secs 2023-03-09T21:58:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:51 smithi038 ceph-mon[92422]: pgmap v13972: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 683 KiB/s wr, 10 op/s 2023-03-09T21:58:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25705.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:58:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25243.701140 secs 2023-03-09T21:58:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:53 smithi122 ceph-mon[93869]: pgmap v13973: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 613 KiB/s rd, 598 KiB/s wr, 13 op/s 2023-03-09T21:58:54.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25243.701140 secs 2023-03-09T21:58:54.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:53 smithi181 ceph-mon[94189]: pgmap v13973: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 613 KiB/s rd, 598 KiB/s wr, 13 op/s 2023-03-09T21:58:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25243.701140 secs 2023-03-09T21:58:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:53 smithi038 ceph-mon[92422]: pgmap v13973: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 613 KiB/s rd, 598 KiB/s wr, 13 op/s 2023-03-09T21:58:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:55 smithi122 ceph-mon[93869]: pgmap v13974: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 633 KiB/s rd, 618 KiB/s wr, 12 op/s 2023-03-09T21:58:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:55 smithi181 ceph-mon[94189]: pgmap v13974: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 633 KiB/s rd, 618 KiB/s wr, 12 op/s 2023-03-09T21:58:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:55 smithi038 ceph-mon[92422]: pgmap v13974: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 633 KiB/s rd, 618 KiB/s wr, 12 op/s 2023-03-09T21:58:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25246.345183 secs 2023-03-09T21:58:57.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25246.345183 secs 2023-03-09T21:58:57.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25246.345183 secs 2023-03-09T21:58:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:57 smithi122 ceph-mon[93869]: pgmap v13975: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 629 KiB/s rd, 653 KiB/s wr, 14 op/s 2023-03-09T21:58:58.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:57 smithi181 ceph-mon[94189]: pgmap v13975: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 629 KiB/s rd, 653 KiB/s wr, 14 op/s 2023-03-09T21:58:58.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:57 smithi038 ceph-mon[92422]: pgmap v13975: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 629 KiB/s rd, 653 KiB/s wr, 14 op/s 2023-03-09T21:58:58.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25710.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:58:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25248.701216 secs 2023-03-09T21:58:59.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25248.701216 secs 2023-03-09T21:58:59.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25248.701216 secs 2023-03-09T21:59:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:58:59 smithi122 ceph-mon[93869]: pgmap v13976: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 605 KiB/s rd, 633 KiB/s wr, 11 op/s 2023-03-09T21:59:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:58:59 smithi181 ceph-mon[94189]: pgmap v13976: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 605 KiB/s rd, 633 KiB/s wr, 11 op/s 2023-03-09T21:59:00.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:58:59 smithi038 ceph-mon[92422]: pgmap v13976: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 605 KiB/s rd, 633 KiB/s wr, 11 op/s 2023-03-09T21:59:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:59:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25251.345267 secs 2023-03-09T21:59:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:59:01 smithi122 ceph-mon[93869]: pgmap v13977: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 475 KiB/s wr, 10 op/s 2023-03-09T21:59:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:59:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25251.345267 secs 2023-03-09T21:59:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:59:01 smithi181 ceph-mon[94189]: pgmap v13977: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 475 KiB/s wr, 10 op/s 2023-03-09T21:59:01.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:59:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25251.345267 secs 2023-03-09T21:59:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:59:01 smithi038 ceph-mon[92422]: pgmap v13977: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 475 KiB/s wr, 10 op/s 2023-03-09T21:59:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25715.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:59:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:59:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25253.701330 secs 2023-03-09T21:59:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:59:03 smithi122 ceph-mon[93869]: pgmap v13978: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 584 KiB/s rd, 592 KiB/s wr, 14 op/s 2023-03-09T21:59:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:59:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25253.701330 secs 2023-03-09T21:59:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:59:03 smithi181 ceph-mon[94189]: pgmap v13978: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 584 KiB/s rd, 592 KiB/s wr, 14 op/s 2023-03-09T21:59:03.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:59:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25253.701330 secs 2023-03-09T21:59:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:59:03 smithi038 ceph-mon[92422]: pgmap v13978: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 584 KiB/s rd, 592 KiB/s wr, 14 op/s 2023-03-09T21:59:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:59:05 smithi122 ceph-mon[93869]: pgmap v13979: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 511 KiB/s rd, 534 KiB/s wr, 23 op/s 2023-03-09T21:59:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:59:05 smithi181 ceph-mon[94189]: pgmap v13979: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 511 KiB/s rd, 534 KiB/s wr, 23 op/s 2023-03-09T21:59:05.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:59:05 smithi038 ceph-mon[92422]: pgmap v13979: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 511 KiB/s rd, 534 KiB/s wr, 23 op/s 2023-03-09T21:59:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:59:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25256.345327 secs 2023-03-09T21:59:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:59:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25256.345327 secs 2023-03-09T21:59:06.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:59:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25256.345327 secs 2023-03-09T21:59:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:59:07 smithi122 ceph-mon[93869]: pgmap v13980: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 573 KiB/s rd, 605 KiB/s wr, 26 op/s 2023-03-09T21:59:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:59:07 smithi181 ceph-mon[94189]: pgmap v13980: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 573 KiB/s rd, 605 KiB/s wr, 26 op/s 2023-03-09T21:59:07.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:59:07 smithi038 ceph-mon[92422]: pgmap v13980: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 573 KiB/s rd, 605 KiB/s wr, 26 op/s 2023-03-09T21:59:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25720.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:59:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:59:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25258.701433 secs 2023-03-09T21:59:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:59:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25258.701433 secs 2023-03-09T21:59:08.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:59:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25258.701433 secs 2023-03-09T21:59:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:59:09 smithi122 ceph-mon[93869]: pgmap v13981: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 386 KiB/s wr, 23 op/s 2023-03-09T21:59:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:59:09 smithi181 ceph-mon[94189]: pgmap v13981: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 386 KiB/s wr, 23 op/s 2023-03-09T21:59:09.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:59:09 smithi038 ceph-mon[92422]: pgmap v13981: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 386 KiB/s wr, 23 op/s 2023-03-09T21:59:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:59:10 smithi181 ceph-mon[94189]: osdmap e1087: 12 total, 12 up, 12 in 2023-03-09T21:59:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:59:10 smithi038 ceph-mon[92422]: osdmap e1087: 12 total, 12 up, 12 in 2023-03-09T21:59:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:59:10 smithi122 ceph-mon[93869]: osdmap e1087: 12 total, 12 up, 12 in 2023-03-09T21:59:11.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:59:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25261.345389 secs 2023-03-09T21:59:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:59:11 smithi038 ceph-mon[92422]: pgmap v13983: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 301 KiB/s wr, 27 op/s 2023-03-09T21:59:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:59:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25261.345389 secs 2023-03-09T21:59:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:59:11 smithi122 ceph-mon[93869]: pgmap v13983: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 301 KiB/s wr, 27 op/s 2023-03-09T21:59:12.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:59:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25261.345389 secs 2023-03-09T21:59:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:59:11 smithi181 ceph-mon[94189]: pgmap v13983: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 301 KiB/s wr, 27 op/s 2023-03-09T21:59:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25725.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:59:13.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:59:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25263.701508 secs 2023-03-09T21:59:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:59:13 smithi038 ceph-mon[92422]: pgmap v13984: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 161 KiB/s wr, 24 op/s 2023-03-09T21:59:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:59:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25263.701508 secs 2023-03-09T21:59:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:59:13 smithi122 ceph-mon[93869]: pgmap v13984: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 161 KiB/s wr, 24 op/s 2023-03-09T21:59:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:59:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25263.701508 secs 2023-03-09T21:59:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:59:13 smithi181 ceph-mon[94189]: pgmap v13984: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 161 KiB/s wr, 24 op/s 2023-03-09T21:59:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:59:15 smithi038 ceph-mon[92422]: pgmap v13985: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 135 KiB/s wr, 10 op/s 2023-03-09T21:59:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:59:15 smithi038 ceph-mon[92422]: osdmap e1088: 12 total, 12 up, 12 in 2023-03-09T21:59:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:59:15 smithi122 ceph-mon[93869]: pgmap v13985: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 135 KiB/s wr, 10 op/s 2023-03-09T21:59:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:59:15 smithi122 ceph-mon[93869]: osdmap e1088: 12 total, 12 up, 12 in 2023-03-09T21:59:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:59:15 smithi181 ceph-mon[94189]: pgmap v13985: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 135 KiB/s wr, 10 op/s 2023-03-09T21:59:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:59:15 smithi181 ceph-mon[94189]: osdmap e1088: 12 total, 12 up, 12 in 2023-03-09T21:59:16.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:59:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25266.345490 secs 2023-03-09T21:59:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:59:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25266.345490 secs 2023-03-09T21:59:17.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:59:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25266.345490 secs 2023-03-09T21:59:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:59:17 smithi038 ceph-mon[92422]: pgmap v13987: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 31 KiB/s wr, 10 op/s 2023-03-09T21:59:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:59:17 smithi122 ceph-mon[93869]: pgmap v13987: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 31 KiB/s wr, 10 op/s 2023-03-09T21:59:18.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:59:17 smithi181 ceph-mon[94189]: pgmap v13987: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 31 KiB/s wr, 10 op/s 2023-03-09T21:59:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25730.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:59:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:59:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25268.701626 secs 2023-03-09T21:59:19.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:59:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25268.701626 secs 2023-03-09T21:59:19.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:59:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25268.701626 secs 2023-03-09T21:59:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:59:19 smithi122 ceph-mon[93869]: pgmap v13988: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 29 KiB/s wr, 9 op/s 2023-03-09T21:59:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:59:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:59:20.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:59:19 smithi181 ceph-mon[94189]: pgmap v13988: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 29 KiB/s wr, 9 op/s 2023-03-09T21:59:20.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:59:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:59:20.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:59:19 smithi038 ceph-mon[92422]: pgmap v13988: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 29 KiB/s wr, 9 op/s 2023-03-09T21:59:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:59:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T21:59:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:59:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:59:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:59:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:59:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:59:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:59:21.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:59:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:59:21.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:59:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:59:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:59:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:59:21.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:59:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T21:59:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:59:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T21:59:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:59:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T21:59:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:59:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25271.345551 secs 2023-03-09T21:59:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:59:21 smithi122 ceph-mon[93869]: pgmap v13989: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 8 op/s 2023-03-09T21:59:22.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:59:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25271.345551 secs 2023-03-09T21:59:22.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:59:21 smithi181 ceph-mon[94189]: pgmap v13989: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 8 op/s 2023-03-09T21:59:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:59:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25271.345551 secs 2023-03-09T21:59:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:59:21 smithi038 ceph-mon[92422]: pgmap v13989: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 8 op/s 2023-03-09T21:59:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25735.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:59:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:59:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25273.701750 secs 2023-03-09T21:59:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:59:23 smithi122 ceph-mon[93869]: pgmap v13990: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 8 op/s 2023-03-09T21:59:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:59:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25273.701750 secs 2023-03-09T21:59:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:59:23 smithi181 ceph-mon[94189]: pgmap v13990: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 8 op/s 2023-03-09T21:59:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:59:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25273.701750 secs 2023-03-09T21:59:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:59:23 smithi038 ceph-mon[92422]: pgmap v13990: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 8 op/s 2023-03-09T21:59:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:59:25 smithi122 ceph-mon[93869]: pgmap v13991: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 8 op/s 2023-03-09T21:59:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:59:25 smithi181 ceph-mon[94189]: pgmap v13991: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 8 op/s 2023-03-09T21:59:26.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:59:25 smithi038 ceph-mon[92422]: pgmap v13991: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 50 KiB/s wr, 8 op/s 2023-03-09T21:59:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:59:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25276.345633 secs 2023-03-09T21:59:27.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:59:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25276.345633 secs 2023-03-09T21:59:27.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:59:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25276.345633 secs 2023-03-09T21:59:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:59:27 smithi122 ceph-mon[93869]: pgmap v13992: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 47 KiB/s wr, 10 op/s 2023-03-09T21:59:28.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:59:27 smithi181 ceph-mon[94189]: pgmap v13992: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 47 KiB/s wr, 10 op/s 2023-03-09T21:59:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:59:27 smithi038 ceph-mon[92422]: pgmap v13992: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 47 KiB/s wr, 10 op/s 2023-03-09T21:59:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25740.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:59:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:59:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25278.701867 secs 2023-03-09T21:59:29.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:59:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25278.701867 secs 2023-03-09T21:59:29.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:59:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25278.701867 secs 2023-03-09T21:59:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:59:29 smithi122 ceph-mon[93869]: pgmap v13993: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 22 KiB/s wr, 6 op/s 2023-03-09T21:59:30.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:59:29 smithi181 ceph-mon[94189]: pgmap v13993: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 22 KiB/s wr, 6 op/s 2023-03-09T21:59:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:59:29 smithi038 ceph-mon[92422]: pgmap v13993: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 22 KiB/s wr, 6 op/s 2023-03-09T21:59:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:59:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25281.345695 secs 2023-03-09T21:59:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:59:31 smithi122 ceph-mon[93869]: pgmap v13994: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 125 KiB/s wr, 7 op/s 2023-03-09T21:59:32.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:59:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25281.345695 secs 2023-03-09T21:59:32.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:59:31 smithi181 ceph-mon[94189]: pgmap v13994: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 125 KiB/s wr, 7 op/s 2023-03-09T21:59:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:59:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25281.345695 secs 2023-03-09T21:59:32.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:59:31 smithi038 ceph-mon[92422]: pgmap v13994: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 125 KiB/s wr, 7 op/s 2023-03-09T21:59:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25745.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:59:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:59:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25283.701975 secs 2023-03-09T21:59:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:59:33 smithi122 ceph-mon[93869]: pgmap v13995: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 126 KiB/s wr, 9 op/s 2023-03-09T21:59:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:59:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25283.701975 secs 2023-03-09T21:59:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:59:33 smithi181 ceph-mon[94189]: pgmap v13995: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 126 KiB/s wr, 9 op/s 2023-03-09T21:59:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:59:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25283.701975 secs 2023-03-09T21:59:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:59:33 smithi038 ceph-mon[92422]: pgmap v13995: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 126 KiB/s wr, 9 op/s 2023-03-09T21:59:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:59:35 smithi122 ceph-mon[93869]: pgmap v13996: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 104 KiB/s wr, 6 op/s 2023-03-09T21:59:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:59:35 smithi181 ceph-mon[94189]: pgmap v13996: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 104 KiB/s wr, 6 op/s 2023-03-09T21:59:36.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:59:35 smithi038 ceph-mon[92422]: pgmap v13996: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 104 KiB/s wr, 6 op/s 2023-03-09T21:59:37.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:59:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25286.345754 secs 2023-03-09T21:59:37.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:59:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25286.345754 secs 2023-03-09T21:59:37.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:59:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25286.345754 secs 2023-03-09T21:59:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:59:37 smithi122 ceph-mon[93869]: pgmap v13997: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 107 KiB/s wr, 9 op/s 2023-03-09T21:59:38.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:59:37 smithi181 ceph-mon[94189]: pgmap v13997: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 107 KiB/s wr, 9 op/s 2023-03-09T21:59:38.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:59:37 smithi038 ceph-mon[92422]: pgmap v13997: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 107 KiB/s wr, 9 op/s 2023-03-09T21:59:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25750.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:59:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:59:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25288.702071 secs 2023-03-09T21:59:39.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:59:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25288.702071 secs 2023-03-09T21:59:39.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:59:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25288.702071 secs 2023-03-09T21:59:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:59:39 smithi122 ceph-mon[93869]: pgmap v13998: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 202 KiB/s wr, 7 op/s 2023-03-09T21:59:40.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:59:39 smithi181 ceph-mon[94189]: pgmap v13998: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 202 KiB/s wr, 7 op/s 2023-03-09T21:59:40.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:59:39 smithi038 ceph-mon[92422]: pgmap v13998: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 202 KiB/s wr, 7 op/s 2023-03-09T21:59:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:59:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25291.345814 secs 2023-03-09T21:59:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:59:41 smithi122 ceph-mon[93869]: pgmap v13999: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 202 KiB/s wr, 7 op/s 2023-03-09T21:59:42.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:59:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25291.345814 secs 2023-03-09T21:59:42.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:59:41 smithi181 ceph-mon[94189]: pgmap v13999: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 202 KiB/s wr, 7 op/s 2023-03-09T21:59:42.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:59:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25291.345814 secs 2023-03-09T21:59:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:59:41 smithi038 ceph-mon[92422]: pgmap v13999: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 202 KiB/s wr, 7 op/s 2023-03-09T21:59:43.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25755.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:59:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:59:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25293.702193 secs 2023-03-09T21:59:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:59:43 smithi122 ceph-mon[93869]: pgmap v14000: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 231 KiB/s wr, 9 op/s 2023-03-09T21:59:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:59:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25293.702193 secs 2023-03-09T21:59:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:59:43 smithi181 ceph-mon[94189]: pgmap v14000: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 231 KiB/s wr, 9 op/s 2023-03-09T21:59:44.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:59:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25293.702193 secs 2023-03-09T21:59:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:59:43 smithi038 ceph-mon[92422]: pgmap v14000: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 231 KiB/s wr, 9 op/s 2023-03-09T21:59:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:59:45 smithi122 ceph-mon[93869]: pgmap v14001: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 230 KiB/s wr, 7 op/s 2023-03-09T21:59:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:59:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25296.345877 secs 2023-03-09T21:59:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:59:45 smithi181 ceph-mon[94189]: pgmap v14001: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 230 KiB/s wr, 7 op/s 2023-03-09T21:59:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:59:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25296.345877 secs 2023-03-09T21:59:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:59:45 smithi038 ceph-mon[92422]: pgmap v14001: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 230 KiB/s wr, 7 op/s 2023-03-09T21:59:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:59:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25296.345877 secs 2023-03-09T21:59:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:59:47 smithi122 ceph-mon[93869]: pgmap v14002: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 336 KiB/s wr, 10 op/s 2023-03-09T21:59:48.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:59:47 smithi181 ceph-mon[94189]: pgmap v14002: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 336 KiB/s wr, 10 op/s 2023-03-09T21:59:48.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:59:47 smithi038 ceph-mon[92422]: pgmap v14002: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 336 KiB/s wr, 10 op/s 2023-03-09T21:59:48.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25760.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:59:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:59:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25298.702286 secs 2023-03-09T21:59:49.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:59:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25298.702286 secs 2023-03-09T21:59:49.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:59:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25298.702286 secs 2023-03-09T21:59:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:59:49 smithi122 ceph-mon[93869]: pgmap v14003: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 447 KiB/s wr, 9 op/s 2023-03-09T21:59:50.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:59:49 smithi181 ceph-mon[94189]: pgmap v14003: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 447 KiB/s wr, 9 op/s 2023-03-09T21:59:50.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:59:49 smithi038 ceph-mon[92422]: pgmap v14003: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 447 KiB/s wr, 9 op/s 2023-03-09T21:59:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:59:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25301.345946 secs 2023-03-09T21:59:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:59:51 smithi122 ceph-mon[93869]: pgmap v14004: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 352 KiB/s wr, 9 op/s 2023-03-09T21:59:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:59:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25301.345946 secs 2023-03-09T21:59:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:59:51 smithi181 ceph-mon[94189]: pgmap v14004: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 352 KiB/s wr, 9 op/s 2023-03-09T21:59:51.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:59:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25301.345946 secs 2023-03-09T21:59:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:59:51 smithi038 ceph-mon[92422]: pgmap v14004: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 352 KiB/s wr, 9 op/s 2023-03-09T21:59:53.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25765.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:59:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:59:53 smithi122 ceph-mon[93869]: pgmap v14005: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 574 KiB/s rd, 558 KiB/s wr, 13 op/s 2023-03-09T21:59:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:59:53 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:59:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:59:53 smithi181 ceph-mon[94189]: pgmap v14005: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 574 KiB/s rd, 558 KiB/s wr, 13 op/s 2023-03-09T21:59:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:59:53 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:59:53.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:59:53 smithi038 ceph-mon[92422]: pgmap v14005: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 574 KiB/s rd, 558 KiB/s wr, 13 op/s 2023-03-09T21:59:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:59:53 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T21:59:54.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:59:54 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:59:54.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:59:54 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:59:54.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:59:54 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:59:54.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:59:54 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:59:54.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:59:54 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:59:54.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:59:54 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:59:54.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:59:54 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T21:59:54.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:59:54 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T21:59:54.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:59:54 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T21:59:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:59:55 smithi122 ceph-mon[93869]: pgmap v14006: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 440 KiB/s rd, 426 KiB/s wr, 10 op/s 2023-03-09T21:59:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:59:55 smithi181 ceph-mon[94189]: pgmap v14006: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 440 KiB/s rd, 426 KiB/s wr, 10 op/s 2023-03-09T21:59:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:59:55 smithi038 ceph-mon[92422]: pgmap v14006: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 440 KiB/s rd, 426 KiB/s wr, 10 op/s 2023-03-09T21:59:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:59:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25306.345985 secs 2023-03-09T21:59:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:59:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25306.345985 secs 2023-03-09T21:59:56.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:59:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25306.345985 secs 2023-03-09T21:59:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:59:57 smithi122 ceph-mon[93869]: pgmap v14007: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 741 KiB/s rd, 726 KiB/s wr, 15 op/s 2023-03-09T21:59:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:59:57 smithi181 ceph-mon[94189]: pgmap v14007: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 741 KiB/s rd, 726 KiB/s wr, 15 op/s 2023-03-09T21:59:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:59:57 smithi038 ceph-mon[92422]: pgmap v14007: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 741 KiB/s rd, 726 KiB/s wr, 15 op/s 2023-03-09T21:59:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25770.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T21:59:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 21:59:59 smithi122 ceph-mon[93869]: pgmap v14008: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 634 KiB/s rd, 620 KiB/s wr, 12 op/s 2023-03-09T21:59:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 21:59:59 smithi181 ceph-mon[94189]: pgmap v14008: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 634 KiB/s rd, 620 KiB/s wr, 12 op/s 2023-03-09T21:59:59.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 21:59:59 smithi038 ceph-mon[92422]: pgmap v14008: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 634 KiB/s rd, 620 KiB/s wr, 12 op/s 2023-03-09T22:00:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:00:00 smithi122 ceph-mon[93869]: overall HEALTH_WARN 1 MDSs report slow requests 2023-03-09T22:00:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:00:00 smithi181 ceph-mon[94189]: overall HEALTH_WARN 1 MDSs report slow requests 2023-03-09T22:00:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:00:00 smithi038 ceph-mon[92422]: overall HEALTH_WARN 1 MDSs report slow requests 2023-03-09T22:00:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:00:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25311.346045 secs 2023-03-09T22:00:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:00:01 smithi181 ceph-mon[94189]: pgmap v14009: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 520 KiB/s rd, 506 KiB/s wr, 11 op/s 2023-03-09T22:00:01.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:00:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25311.346045 secs 2023-03-09T22:00:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:00:01 smithi038 ceph-mon[92422]: pgmap v14009: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 520 KiB/s rd, 506 KiB/s wr, 11 op/s 2023-03-09T22:00:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:00:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25311.346045 secs 2023-03-09T22:00:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:00:01 smithi122 ceph-mon[93869]: pgmap v14009: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 520 KiB/s rd, 506 KiB/s wr, 11 op/s 2023-03-09T22:00:03.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25775.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:00:03.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:00:03 smithi038 ceph-mon[92422]: pgmap v14010: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 759 KiB/s rd, 755 KiB/s wr, 14 op/s 2023-03-09T22:00:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:00:03 smithi122 ceph-mon[93869]: pgmap v14010: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 759 KiB/s rd, 755 KiB/s wr, 14 op/s 2023-03-09T22:00:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:00:03 smithi181 ceph-mon[94189]: pgmap v14010: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 759 KiB/s rd, 755 KiB/s wr, 14 op/s 2023-03-09T22:00:05.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:00:05 smithi038 ceph-mon[92422]: pgmap v14011: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 549 KiB/s wr, 12 op/s 2023-03-09T22:00:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:00:05 smithi122 ceph-mon[93869]: pgmap v14011: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 549 KiB/s wr, 12 op/s 2023-03-09T22:00:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:00:05 smithi181 ceph-mon[94189]: pgmap v14011: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 549 KiB/s wr, 12 op/s 2023-03-09T22:00:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:00:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25316.346104 secs 2023-03-09T22:00:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:00:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25316.346104 secs 2023-03-09T22:00:07.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:00:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25316.346104 secs 2023-03-09T22:00:07.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:00:07 smithi038 ceph-mon[92422]: pgmap v14012: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 814 KiB/s rd, 833 KiB/s wr, 18 op/s 2023-03-09T22:00:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:00:07 smithi122 ceph-mon[93869]: pgmap v14012: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 814 KiB/s rd, 833 KiB/s wr, 18 op/s 2023-03-09T22:00:08.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:00:07 smithi181 ceph-mon[94189]: pgmap v14012: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 814 KiB/s rd, 833 KiB/s wr, 18 op/s 2023-03-09T22:00:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25780.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:00:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:00:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25318.702605 secs 2023-03-09T22:00:09.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:00:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25318.702605 secs 2023-03-09T22:00:09.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:00:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25318.702605 secs 2023-03-09T22:00:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:00:09 smithi122 ceph-mon[93869]: pgmap v14013: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 520 KiB/s rd, 554 KiB/s wr, 24 op/s 2023-03-09T22:00:10.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:00:09 smithi181 ceph-mon[94189]: pgmap v14013: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 520 KiB/s rd, 554 KiB/s wr, 24 op/s 2023-03-09T22:00:10.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:00:09 smithi038 ceph-mon[92422]: pgmap v14013: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 520 KiB/s rd, 554 KiB/s wr, 24 op/s 2023-03-09T22:00:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:00:10 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:00:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:00:10 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:00:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:00:10 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:00:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:00:10 smithi122 ceph-mon[93869]: osdmap e1089: 12 total, 12 up, 12 in 2023-03-09T22:00:11.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:00:10 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:00:11.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:00:10 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:00:11.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:00:10 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:00:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:00:10 smithi181 ceph-mon[94189]: osdmap e1089: 12 total, 12 up, 12 in 2023-03-09T22:00:11.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:00:10 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:00:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:00:10 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:00:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:00:10 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:00:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:00:10 smithi038 ceph-mon[92422]: osdmap e1089: 12 total, 12 up, 12 in 2023-03-09T22:00:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:00:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25321.348009 secs 2023-03-09T22:00:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:00:11 smithi122 ceph-mon[93869]: pgmap v14015: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 665 KiB/s wr, 29 op/s 2023-03-09T22:00:12.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:00:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25321.348009 secs 2023-03-09T22:00:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:00:11 smithi181 ceph-mon[94189]: pgmap v14015: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 665 KiB/s wr, 29 op/s 2023-03-09T22:00:12.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:00:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25321.348009 secs 2023-03-09T22:00:12.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:00:11 smithi038 ceph-mon[92422]: pgmap v14015: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 665 KiB/s wr, 29 op/s 2023-03-09T22:00:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25785.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:00:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:00:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25323.702725 secs 2023-03-09T22:00:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:00:13 smithi122 ceph-mon[93869]: pgmap v14016: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 536 KiB/s rd, 554 KiB/s wr, 29 op/s 2023-03-09T22:00:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:00:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25323.702725 secs 2023-03-09T22:00:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:00:13 smithi181 ceph-mon[94189]: pgmap v14016: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 536 KiB/s rd, 554 KiB/s wr, 29 op/s 2023-03-09T22:00:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:00:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25323.702725 secs 2023-03-09T22:00:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:00:13 smithi038 ceph-mon[92422]: pgmap v14016: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 536 KiB/s rd, 554 KiB/s wr, 29 op/s 2023-03-09T22:00:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:00:15 smithi122 ceph-mon[93869]: pgmap v14017: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 535 KiB/s rd, 554 KiB/s wr, 27 op/s 2023-03-09T22:00:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:00:15 smithi122 ceph-mon[93869]: osdmap e1090: 12 total, 12 up, 12 in 2023-03-09T22:00:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:00:15 smithi181 ceph-mon[94189]: pgmap v14017: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 535 KiB/s rd, 554 KiB/s wr, 27 op/s 2023-03-09T22:00:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:00:15 smithi181 ceph-mon[94189]: osdmap e1090: 12 total, 12 up, 12 in 2023-03-09T22:00:16.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:00:15 smithi038 ceph-mon[92422]: pgmap v14017: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 535 KiB/s rd, 554 KiB/s wr, 27 op/s 2023-03-09T22:00:16.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:00:15 smithi038 ceph-mon[92422]: osdmap e1090: 12 total, 12 up, 12 in 2023-03-09T22:00:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:00:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25326.348066 secs 2023-03-09T22:00:17.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:00:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25326.348066 secs 2023-03-09T22:00:17.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:00:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25326.348066 secs 2023-03-09T22:00:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:00:17 smithi122 ceph-mon[93869]: pgmap v14019: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 476 KiB/s rd, 387 KiB/s wr, 13 op/s 2023-03-09T22:00:18.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:00:17 smithi181 ceph-mon[94189]: pgmap v14019: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 476 KiB/s rd, 387 KiB/s wr, 13 op/s 2023-03-09T22:00:18.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:00:17 smithi038 ceph-mon[92422]: pgmap v14019: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 476 KiB/s rd, 387 KiB/s wr, 13 op/s 2023-03-09T22:00:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25790.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:00:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:00:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25328.702842 secs 2023-03-09T22:00:19.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:00:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25328.702842 secs 2023-03-09T22:00:19.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:00:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25328.702842 secs 2023-03-09T22:00:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:00:19 smithi122 ceph-mon[93869]: pgmap v14020: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 357 KiB/s wr, 13 op/s 2023-03-09T22:00:20.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:00:19 smithi181 ceph-mon[94189]: pgmap v14020: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 357 KiB/s wr, 13 op/s 2023-03-09T22:00:20.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:00:19 smithi038 ceph-mon[92422]: pgmap v14020: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 357 KiB/s wr, 13 op/s 2023-03-09T22:00:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:00:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:00:21.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:00:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:00:21.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:00:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:00:21.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:00:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:00:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:00:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:00:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:00:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:00:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:00:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:00:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:00:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:00:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:00:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:00:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:00:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25331.348129 secs 2023-03-09T22:00:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:00:21 smithi122 ceph-mon[93869]: pgmap v14021: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 310 KiB/s wr, 11 op/s 2023-03-09T22:00:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:00:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:00:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:00:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:00:22.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:00:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:00:22.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:00:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25331.348129 secs 2023-03-09T22:00:22.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:00:21 smithi181 ceph-mon[94189]: pgmap v14021: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 310 KiB/s wr, 11 op/s 2023-03-09T22:00:22.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:00:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:00:22.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:00:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:00:22.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:00:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:00:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:00:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25331.348129 secs 2023-03-09T22:00:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:00:21 smithi038 ceph-mon[92422]: pgmap v14021: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 310 KiB/s wr, 11 op/s 2023-03-09T22:00:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:00:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:00:22.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:00:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:00:22.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:00:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:00:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25795.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:00:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:00:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25333.702944 secs 2023-03-09T22:00:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:00:23 smithi122 ceph-mon[93869]: pgmap v14022: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 123 KiB/s wr, 10 op/s 2023-03-09T22:00:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:00:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25333.702944 secs 2023-03-09T22:00:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:00:23 smithi181 ceph-mon[94189]: pgmap v14022: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 123 KiB/s wr, 10 op/s 2023-03-09T22:00:24.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:00:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25333.702944 secs 2023-03-09T22:00:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:00:23 smithi038 ceph-mon[92422]: pgmap v14022: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 123 KiB/s wr, 10 op/s 2023-03-09T22:00:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:00:25 smithi122 ceph-mon[93869]: pgmap v14023: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 123 KiB/s wr, 10 op/s 2023-03-09T22:00:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:00:25 smithi181 ceph-mon[94189]: pgmap v14023: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 123 KiB/s wr, 10 op/s 2023-03-09T22:00:26.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:00:25 smithi038 ceph-mon[92422]: pgmap v14023: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 123 KiB/s wr, 10 op/s 2023-03-09T22:00:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:00:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25336.348190 secs 2023-03-09T22:00:27.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:00:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25336.348190 secs 2023-03-09T22:00:27.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:00:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25336.348190 secs 2023-03-09T22:00:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:00:27 smithi122 ceph-mon[93869]: pgmap v14024: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 201 KiB/s wr, 13 op/s 2023-03-09T22:00:28.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:00:27 smithi181 ceph-mon[94189]: pgmap v14024: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 201 KiB/s wr, 13 op/s 2023-03-09T22:00:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:00:27 smithi038 ceph-mon[92422]: pgmap v14024: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 201 KiB/s wr, 13 op/s 2023-03-09T22:00:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25800.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:00:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:00:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25338.703066 secs 2023-03-09T22:00:29.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:00:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25338.703066 secs 2023-03-09T22:00:29.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:00:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25338.703066 secs 2023-03-09T22:00:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:00:29 smithi122 ceph-mon[93869]: pgmap v14025: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 78 KiB/s wr, 8 op/s 2023-03-09T22:00:30.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:00:29 smithi181 ceph-mon[94189]: pgmap v14025: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 78 KiB/s wr, 8 op/s 2023-03-09T22:00:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:00:29 smithi038 ceph-mon[92422]: pgmap v14025: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 78 KiB/s wr, 8 op/s 2023-03-09T22:00:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:00:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25341.348252 secs 2023-03-09T22:00:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:00:31 smithi122 ceph-mon[93869]: pgmap v14026: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 78 KiB/s wr, 7 op/s 2023-03-09T22:00:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:00:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25341.348252 secs 2023-03-09T22:00:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:00:31 smithi181 ceph-mon[94189]: pgmap v14026: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 78 KiB/s wr, 7 op/s 2023-03-09T22:00:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:00:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25341.348252 secs 2023-03-09T22:00:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:00:31 smithi038 ceph-mon[92422]: pgmap v14026: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 78 KiB/s wr, 7 op/s 2023-03-09T22:00:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25805.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:00:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:00:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25343.703186 secs 2023-03-09T22:00:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:00:33 smithi122 ceph-mon[93869]: pgmap v14027: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 182 KiB/s wr, 11 op/s 2023-03-09T22:00:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:00:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25343.703186 secs 2023-03-09T22:00:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:00:33 smithi181 ceph-mon[94189]: pgmap v14027: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 182 KiB/s wr, 11 op/s 2023-03-09T22:00:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:00:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25343.703186 secs 2023-03-09T22:00:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:00:33 smithi038 ceph-mon[92422]: pgmap v14027: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 182 KiB/s wr, 11 op/s 2023-03-09T22:00:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:00:35 smithi122 ceph-mon[93869]: pgmap v14028: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 181 KiB/s wr, 8 op/s 2023-03-09T22:00:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:00:35 smithi181 ceph-mon[94189]: pgmap v14028: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 181 KiB/s wr, 8 op/s 2023-03-09T22:00:36.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:00:35 smithi038 ceph-mon[92422]: pgmap v14028: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 181 KiB/s wr, 8 op/s 2023-03-09T22:00:37.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:00:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25346.348314 secs 2023-03-09T22:00:37.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:00:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25346.348314 secs 2023-03-09T22:00:37.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:00:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25346.348314 secs 2023-03-09T22:00:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:00:37 smithi122 ceph-mon[93869]: pgmap v14029: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 181 KiB/s wr, 10 op/s 2023-03-09T22:00:38.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:00:37 smithi181 ceph-mon[94189]: pgmap v14029: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 181 KiB/s wr, 10 op/s 2023-03-09T22:00:38.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:00:37 smithi038 ceph-mon[92422]: pgmap v14029: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 181 KiB/s wr, 10 op/s 2023-03-09T22:00:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25810.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:00:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:00:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25348.703300 secs 2023-03-09T22:00:39.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:00:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25348.703300 secs 2023-03-09T22:00:39.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:00:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25348.703300 secs 2023-03-09T22:00:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:00:39 smithi122 ceph-mon[93869]: pgmap v14030: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 104 KiB/s wr, 7 op/s 2023-03-09T22:00:40.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:00:39 smithi181 ceph-mon[94189]: pgmap v14030: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 104 KiB/s wr, 7 op/s 2023-03-09T22:00:40.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:00:39 smithi038 ceph-mon[92422]: pgmap v14030: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 104 KiB/s wr, 7 op/s 2023-03-09T22:00:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:00:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25351.348375 secs 2023-03-09T22:00:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:00:41 smithi122 ceph-mon[93869]: pgmap v14031: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 104 KiB/s wr, 7 op/s 2023-03-09T22:00:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:00:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25351.348375 secs 2023-03-09T22:00:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:00:41 smithi181 ceph-mon[94189]: pgmap v14031: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 104 KiB/s wr, 7 op/s 2023-03-09T22:00:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:00:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25351.348375 secs 2023-03-09T22:00:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:00:41 smithi038 ceph-mon[92422]: pgmap v14031: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 104 KiB/s wr, 7 op/s 2023-03-09T22:00:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25815.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:00:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:00:43 smithi122 ceph-mon[93869]: pgmap v14032: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 218 KiB/s wr, 10 op/s 2023-03-09T22:00:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:00:43 smithi181 ceph-mon[94189]: pgmap v14032: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 218 KiB/s wr, 10 op/s 2023-03-09T22:00:43.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:00:43 smithi038 ceph-mon[92422]: pgmap v14032: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 218 KiB/s wr, 10 op/s 2023-03-09T22:00:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:00:45 smithi122 ceph-mon[93869]: pgmap v14033: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 115 KiB/s wr, 6 op/s 2023-03-09T22:00:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:00:45 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:00:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:00:45 smithi181 ceph-mon[94189]: pgmap v14033: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 115 KiB/s wr, 6 op/s 2023-03-09T22:00:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:00:45 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:00:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:00:45 smithi038 ceph-mon[92422]: pgmap v14033: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 115 KiB/s wr, 6 op/s 2023-03-09T22:00:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:00:45 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:00:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:00:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25356.348443 secs 2023-03-09T22:00:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:00:46 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:00:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:00:46 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:00:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:00:46 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:00:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:00:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25356.348443 secs 2023-03-09T22:00:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:00:46 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:00:46.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:00:46 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:00:46.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:00:46 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:00:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:00:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25356.348443 secs 2023-03-09T22:00:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:00:46 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:00:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:00:46 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:00:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:00:46 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:00:47.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:00:47 smithi038 ceph-mon[92422]: pgmap v14034: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 135 KiB/s wr, 9 op/s 2023-03-09T22:00:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:00:47 smithi122 ceph-mon[93869]: pgmap v14034: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 135 KiB/s wr, 9 op/s 2023-03-09T22:00:48.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:00:47 smithi181 ceph-mon[94189]: pgmap v14034: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 135 KiB/s wr, 9 op/s 2023-03-09T22:00:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25820.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:00:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:00:49 smithi038 ceph-mon[92422]: pgmap v14035: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 337 KiB/s rd, 335 KiB/s wr, 8 op/s 2023-03-09T22:00:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:00:49 smithi122 ceph-mon[93869]: pgmap v14035: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 337 KiB/s rd, 335 KiB/s wr, 8 op/s 2023-03-09T22:00:50.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:00:49 smithi181 ceph-mon[94189]: pgmap v14035: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 337 KiB/s rd, 335 KiB/s wr, 8 op/s 2023-03-09T22:00:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:00:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25361.348521 secs 2023-03-09T22:00:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:00:51 smithi038 ceph-mon[92422]: pgmap v14036: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 348 KiB/s rd, 335 KiB/s wr, 7 op/s 2023-03-09T22:00:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:00:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25361.348521 secs 2023-03-09T22:00:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:00:51 smithi122 ceph-mon[93869]: pgmap v14036: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 348 KiB/s rd, 335 KiB/s wr, 7 op/s 2023-03-09T22:00:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:00:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25361.348521 secs 2023-03-09T22:00:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:00:51 smithi181 ceph-mon[94189]: pgmap v14036: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 348 KiB/s rd, 335 KiB/s wr, 7 op/s 2023-03-09T22:00:53.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25825.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:00:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:00:53 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 25363.703610 secs 2023-03-09T22:00:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:00:53 smithi122 ceph-mon[93869]: pgmap v14037: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 399 KiB/s wr, 11 op/s 2023-03-09T22:00:54.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:00:53 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 25363.703610 secs 2023-03-09T22:00:54.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:00:53 smithi181 ceph-mon[94189]: pgmap v14037: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 399 KiB/s wr, 11 op/s 2023-03-09T22:00:54.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:00:53 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 25363.703610 secs 2023-03-09T22:00:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:00:53 smithi038 ceph-mon[92422]: pgmap v14037: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 399 KiB/s wr, 11 op/s 2023-03-09T22:00:55.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:00:54 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:00:55.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:00:54 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:00:55.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:00:54 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:00:55.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:00:54 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:00:55.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:00:54 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:00:55.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:00:54 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:00:55.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:00:54 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:00:55.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:00:54 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:00:55.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:00:54 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:00:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:00:55 smithi122 ceph-mon[93869]: pgmap v14038: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 285 KiB/s wr, 9 op/s 2023-03-09T22:00:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:00:55 smithi181 ceph-mon[94189]: pgmap v14038: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 285 KiB/s wr, 9 op/s 2023-03-09T22:00:56.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:00:55 smithi038 ceph-mon[92422]: pgmap v14038: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 285 KiB/s wr, 9 op/s 2023-03-09T22:00:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:00:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25366.348579 secs 2023-03-09T22:00:57.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:00:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25366.348579 secs 2023-03-09T22:00:57.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:00:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25366.348579 secs 2023-03-09T22:00:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:00:57 smithi122 ceph-mon[93869]: pgmap v14039: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 407 KiB/s wr, 12 op/s 2023-03-09T22:00:58.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:00:57 smithi181 ceph-mon[94189]: pgmap v14039: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 407 KiB/s wr, 12 op/s 2023-03-09T22:00:58.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:00:57 smithi038 ceph-mon[92422]: pgmap v14039: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 407 KiB/s wr, 12 op/s 2023-03-09T22:00:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25830.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:00:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:00:58 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 25368.703727 secs 2023-03-09T22:00:59.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:00:58 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 25368.703727 secs 2023-03-09T22:00:59.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:00:58 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 25368.703727 secs 2023-03-09T22:01:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:00:59 smithi122 ceph-mon[93869]: pgmap v14040: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 680 KiB/s rd, 665 KiB/s wr, 11 op/s 2023-03-09T22:01:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:00:59 smithi181 ceph-mon[94189]: pgmap v14040: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 680 KiB/s rd, 665 KiB/s wr, 11 op/s 2023-03-09T22:01:00.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:00:59 smithi038 ceph-mon[92422]: pgmap v14040: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 680 KiB/s rd, 665 KiB/s wr, 11 op/s 2023-03-09T22:01:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:01:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25371.348640 secs 2023-03-09T22:01:01.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:01:01 smithi181 ceph-mon[94189]: pgmap v14041: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 465 KiB/s wr, 10 op/s 2023-03-09T22:01:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:01:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25371.348640 secs 2023-03-09T22:01:02.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:01:01 smithi122 ceph-mon[93869]: pgmap v14041: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 465 KiB/s wr, 10 op/s 2023-03-09T22:01:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:01:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25371.348640 secs 2023-03-09T22:01:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:01:01 smithi038 ceph-mon[92422]: pgmap v14041: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 465 KiB/s wr, 10 op/s 2023-03-09T22:01:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25835.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:01:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:01:03 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 25373.703838 secs 2023-03-09T22:01:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:01:03 smithi122 ceph-mon[93869]: pgmap v14042: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 514 KiB/s rd, 510 KiB/s wr, 14 op/s 2023-03-09T22:01:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:01:03 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 25373.703838 secs 2023-03-09T22:01:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:01:03 smithi181 ceph-mon[94189]: pgmap v14042: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 514 KiB/s rd, 510 KiB/s wr, 14 op/s 2023-03-09T22:01:04.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:01:03 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 25373.703838 secs 2023-03-09T22:01:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:01:03 smithi038 ceph-mon[92422]: pgmap v14042: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 514 KiB/s rd, 510 KiB/s wr, 14 op/s 2023-03-09T22:01:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:01:05 smithi122 ceph-mon[93869]: pgmap v14043: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 454 KiB/s rd, 456 KiB/s wr, 21 op/s 2023-03-09T22:01:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:01:05 smithi181 ceph-mon[94189]: pgmap v14043: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 454 KiB/s rd, 456 KiB/s wr, 21 op/s 2023-03-09T22:01:06.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:01:05 smithi038 ceph-mon[92422]: pgmap v14043: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 454 KiB/s rd, 456 KiB/s wr, 21 op/s 2023-03-09T22:01:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:01:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25376.348701 secs 2023-03-09T22:01:07.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:01:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25376.348701 secs 2023-03-09T22:01:07.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:01:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25376.348701 secs 2023-03-09T22:01:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:01:07 smithi122 ceph-mon[93869]: pgmap v14044: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 884 KiB/s rd, 896 KiB/s wr, 26 op/s 2023-03-09T22:01:08.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:01:07 smithi181 ceph-mon[94189]: pgmap v14044: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 884 KiB/s rd, 896 KiB/s wr, 26 op/s 2023-03-09T22:01:08.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:01:07 smithi038 ceph-mon[92422]: pgmap v14044: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 884 KiB/s rd, 896 KiB/s wr, 26 op/s 2023-03-09T22:01:08.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25840.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:01:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:01:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25378.703912 secs 2023-03-09T22:01:09.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:01:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25378.703912 secs 2023-03-09T22:01:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:01:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25378.703912 secs 2023-03-09T22:01:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:01:09 smithi122 ceph-mon[93869]: pgmap v14045: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 761 KiB/s rd, 775 KiB/s wr, 24 op/s 2023-03-09T22:01:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:01:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:01:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:01:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:01:10.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:01:09 smithi181 ceph-mon[94189]: pgmap v14045: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 761 KiB/s rd, 775 KiB/s wr, 24 op/s 2023-03-09T22:01:10.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:01:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:01:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:01:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:01:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:01:09 smithi038 ceph-mon[92422]: pgmap v14045: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 761 KiB/s rd, 775 KiB/s wr, 24 op/s 2023-03-09T22:01:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:01:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:01:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:01:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:01:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:01:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25381.348768 secs 2023-03-09T22:01:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:01:11 smithi122 ceph-mon[93869]: pgmap v14046: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 483 KiB/s rd, 496 KiB/s wr, 22 op/s 2023-03-09T22:01:12.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:01:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25381.348768 secs 2023-03-09T22:01:12.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:01:11 smithi181 ceph-mon[94189]: pgmap v14046: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 483 KiB/s rd, 496 KiB/s wr, 22 op/s 2023-03-09T22:01:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:01:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25381.348768 secs 2023-03-09T22:01:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:01:11 smithi038 ceph-mon[92422]: pgmap v14046: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 483 KiB/s rd, 496 KiB/s wr, 22 op/s 2023-03-09T22:01:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25845.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:01:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:01:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25383.704045 secs 2023-03-09T22:01:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:01:13 smithi122 ceph-mon[93869]: pgmap v14047: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 590 KiB/s rd, 602 KiB/s wr, 26 op/s 2023-03-09T22:01:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:01:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25383.704045 secs 2023-03-09T22:01:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:01:13 smithi181 ceph-mon[94189]: pgmap v14047: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 590 KiB/s rd, 602 KiB/s wr, 26 op/s 2023-03-09T22:01:14.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:01:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25383.704045 secs 2023-03-09T22:01:14.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:01:13 smithi038 ceph-mon[92422]: pgmap v14047: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 590 KiB/s rd, 602 KiB/s wr, 26 op/s 2023-03-09T22:01:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:01:15 smithi122 ceph-mon[93869]: pgmap v14048: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 545 KiB/s rd, 558 KiB/s wr, 21 op/s 2023-03-09T22:01:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:01:15 smithi181 ceph-mon[94189]: pgmap v14048: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 545 KiB/s rd, 558 KiB/s wr, 21 op/s 2023-03-09T22:01:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:01:15 smithi038 ceph-mon[92422]: pgmap v14048: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 545 KiB/s rd, 558 KiB/s wr, 21 op/s 2023-03-09T22:01:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:01:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25386.348839 secs 2023-03-09T22:01:17.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:01:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25386.348839 secs 2023-03-09T22:01:17.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:01:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25386.348839 secs 2023-03-09T22:01:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:01:17 smithi122 ceph-mon[93869]: pgmap v14049: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 585 KiB/s rd, 593 KiB/s wr, 14 op/s 2023-03-09T22:01:18.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:01:17 smithi181 ceph-mon[94189]: pgmap v14049: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 585 KiB/s rd, 593 KiB/s wr, 14 op/s 2023-03-09T22:01:18.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:01:17 smithi038 ceph-mon[92422]: pgmap v14049: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 585 KiB/s rd, 593 KiB/s wr, 14 op/s 2023-03-09T22:01:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25850.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:01:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:01:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25388.704144 secs 2023-03-09T22:01:19.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:01:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25388.704144 secs 2023-03-09T22:01:19.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:01:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25388.704144 secs 2023-03-09T22:01:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:01:19 smithi122 ceph-mon[93869]: pgmap v14050: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 216 KiB/s wr, 9 op/s 2023-03-09T22:01:20.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:01:19 smithi181 ceph-mon[94189]: pgmap v14050: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 216 KiB/s wr, 9 op/s 2023-03-09T22:01:20.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:01:19 smithi038 ceph-mon[92422]: pgmap v14050: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 216 KiB/s wr, 9 op/s 2023-03-09T22:01:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:01:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25391.348900 secs 2023-03-09T22:01:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:01:21 smithi122 ceph-mon[93869]: pgmap v14051: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 216 KiB/s wr, 8 op/s 2023-03-09T22:01:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:01:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:01:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:01:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25391.348900 secs 2023-03-09T22:01:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:01:21 smithi181 ceph-mon[94189]: pgmap v14051: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 216 KiB/s wr, 8 op/s 2023-03-09T22:01:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:01:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:01:21.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:01:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25391.348900 secs 2023-03-09T22:01:21.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:01:21 smithi038 ceph-mon[92422]: pgmap v14051: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 216 KiB/s wr, 8 op/s 2023-03-09T22:01:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:01:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:01:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:01:22 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:01:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:01:22 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:01:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:01:22 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:01:23.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:01:22 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:01:23.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:01:22 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:01:23.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:01:22 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:01:23.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:01:22 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:01:23.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:01:22 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:01:23.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:01:22 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:01:23.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:01:22 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:01:23.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:01:22 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:01:23.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:01:22 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:01:23.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:01:22 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:01:23.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:01:22 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:01:23.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:01:22 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:01:23.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:01:22 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:01:23.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:01:22 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:01:23.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:01:22 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:01:23.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:01:22 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:01:23.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:01:22 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:01:23.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:01:22 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:01:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25855.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:01:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:01:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25393.704219 secs 2023-03-09T22:01:24.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:01:23 smithi122 ceph-mon[93869]: pgmap v14052: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 526 KiB/s wr, 13 op/s 2023-03-09T22:01:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:01:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25393.704219 secs 2023-03-09T22:01:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:01:23 smithi181 ceph-mon[94189]: pgmap v14052: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 526 KiB/s wr, 13 op/s 2023-03-09T22:01:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:01:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25393.704219 secs 2023-03-09T22:01:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:01:23 smithi038 ceph-mon[92422]: pgmap v14052: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 526 KiB/s wr, 13 op/s 2023-03-09T22:01:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:01:25 smithi038 ceph-mon[92422]: pgmap v14053: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 511 KiB/s wr, 12 op/s 2023-03-09T22:01:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:01:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25396.348957 secs 2023-03-09T22:01:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:01:25 smithi038 ceph-mon[92422]: osdmap e1091: 12 total, 12 up, 12 in 2023-03-09T22:01:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:01:25 smithi122 ceph-mon[93869]: pgmap v14053: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 511 KiB/s wr, 12 op/s 2023-03-09T22:01:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:01:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25396.348957 secs 2023-03-09T22:01:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:01:25 smithi122 ceph-mon[93869]: osdmap e1091: 12 total, 12 up, 12 in 2023-03-09T22:01:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:01:25 smithi181 ceph-mon[94189]: pgmap v14053: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 511 KiB/s wr, 12 op/s 2023-03-09T22:01:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:01:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25396.348957 secs 2023-03-09T22:01:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:01:25 smithi181 ceph-mon[94189]: osdmap e1091: 12 total, 12 up, 12 in 2023-03-09T22:01:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:01:27 smithi038 ceph-mon[92422]: pgmap v14055: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 538 KiB/s rd, 563 KiB/s wr, 13 op/s 2023-03-09T22:01:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:01:27 smithi122 ceph-mon[93869]: pgmap v14055: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 538 KiB/s rd, 563 KiB/s wr, 13 op/s 2023-03-09T22:01:28.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:01:27 smithi181 ceph-mon[94189]: pgmap v14055: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 538 KiB/s rd, 563 KiB/s wr, 13 op/s 2023-03-09T22:01:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25860.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:01:29.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:01:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25398.704293 secs 2023-03-09T22:01:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:01:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25398.704293 secs 2023-03-09T22:01:29.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:01:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25398.704293 secs 2023-03-09T22:01:30.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:01:29 smithi038 ceph-mon[92422]: pgmap v14056: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 487 KiB/s wr, 13 op/s 2023-03-09T22:01:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:01:29 smithi122 ceph-mon[93869]: pgmap v14056: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 487 KiB/s wr, 13 op/s 2023-03-09T22:01:30.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:01:29 smithi181 ceph-mon[94189]: pgmap v14056: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 487 KiB/s wr, 13 op/s 2023-03-09T22:01:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:01:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25401.348983 secs 2023-03-09T22:01:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:01:31 smithi122 ceph-mon[93869]: pgmap v14057: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 775 KiB/s rd, 800 KiB/s wr, 15 op/s 2023-03-09T22:01:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:01:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25401.348983 secs 2023-03-09T22:01:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:01:31 smithi181 ceph-mon[94189]: pgmap v14057: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 775 KiB/s rd, 800 KiB/s wr, 15 op/s 2023-03-09T22:01:31.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:01:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25401.348983 secs 2023-03-09T22:01:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:01:31 smithi038 ceph-mon[92422]: pgmap v14057: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 775 KiB/s rd, 800 KiB/s wr, 15 op/s 2023-03-09T22:01:33.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25865.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:01:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:01:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25403.704412 secs 2023-03-09T22:01:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:01:33 smithi181 ceph-mon[94189]: pgmap v14058: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 451 KiB/s wr, 12 op/s 2023-03-09T22:01:33.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:01:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25403.704412 secs 2023-03-09T22:01:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:01:33 smithi038 ceph-mon[92422]: pgmap v14058: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 451 KiB/s wr, 12 op/s 2023-03-09T22:01:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:01:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25403.704412 secs 2023-03-09T22:01:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:01:33 smithi122 ceph-mon[93869]: pgmap v14058: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 451 KiB/s wr, 12 op/s 2023-03-09T22:01:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:01:35 smithi038 ceph-mon[92422]: pgmap v14059: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 342 KiB/s wr, 11 op/s 2023-03-09T22:01:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:01:35 smithi038 ceph-mon[92422]: osdmap e1092: 12 total, 12 up, 12 in 2023-03-09T22:01:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:01:35 smithi122 ceph-mon[93869]: pgmap v14059: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 342 KiB/s wr, 11 op/s 2023-03-09T22:01:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:01:35 smithi122 ceph-mon[93869]: osdmap e1092: 12 total, 12 up, 12 in 2023-03-09T22:01:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:01:35 smithi181 ceph-mon[94189]: pgmap v14059: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 342 KiB/s wr, 11 op/s 2023-03-09T22:01:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:01:35 smithi181 ceph-mon[94189]: osdmap e1092: 12 total, 12 up, 12 in 2023-03-09T22:01:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:01:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25406.349041 secs 2023-03-09T22:01:37.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:01:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25406.349041 secs 2023-03-09T22:01:37.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:01:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25406.349041 secs 2023-03-09T22:01:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:01:37 smithi038 ceph-mon[92422]: pgmap v14061: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 473 KiB/s wr, 11 op/s 2023-03-09T22:01:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:01:37 smithi122 ceph-mon[93869]: pgmap v14061: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 473 KiB/s wr, 11 op/s 2023-03-09T22:01:38.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:01:37 smithi181 ceph-mon[94189]: pgmap v14061: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 473 KiB/s wr, 11 op/s 2023-03-09T22:01:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25870.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:01:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:01:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25408.704516 secs 2023-03-09T22:01:39.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:01:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25408.704516 secs 2023-03-09T22:01:39.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:01:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25408.704516 secs 2023-03-09T22:01:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:01:39 smithi122 ceph-mon[93869]: pgmap v14062: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 473 KiB/s wr, 10 op/s 2023-03-09T22:01:40.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:01:39 smithi181 ceph-mon[94189]: pgmap v14062: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 473 KiB/s wr, 10 op/s 2023-03-09T22:01:40.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:01:39 smithi038 ceph-mon[92422]: pgmap v14062: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 473 KiB/s wr, 10 op/s 2023-03-09T22:01:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:01:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25411.349101 secs 2023-03-09T22:01:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:01:41 smithi122 ceph-mon[93869]: pgmap v14063: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 161 KiB/s wr, 9 op/s 2023-03-09T22:01:42.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:01:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25411.349101 secs 2023-03-09T22:01:42.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:01:41 smithi181 ceph-mon[94189]: pgmap v14063: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 161 KiB/s wr, 9 op/s 2023-03-09T22:01:42.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:01:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25411.349101 secs 2023-03-09T22:01:42.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:01:41 smithi038 ceph-mon[92422]: pgmap v14063: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 161 KiB/s wr, 9 op/s 2023-03-09T22:01:43.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25875.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:01:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:01:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25413.704660 secs 2023-03-09T22:01:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:01:43 smithi122 ceph-mon[93869]: pgmap v14064: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 275 KiB/s wr, 8 op/s 2023-03-09T22:01:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:01:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25413.704660 secs 2023-03-09T22:01:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:01:43 smithi181 ceph-mon[94189]: pgmap v14064: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 275 KiB/s wr, 8 op/s 2023-03-09T22:01:44.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:01:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25413.704660 secs 2023-03-09T22:01:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:01:43 smithi038 ceph-mon[92422]: pgmap v14064: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 275 KiB/s wr, 8 op/s 2023-03-09T22:01:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:01:45 smithi122 ceph-mon[93869]: pgmap v14065: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 275 KiB/s wr, 8 op/s 2023-03-09T22:01:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:01:45 smithi181 ceph-mon[94189]: pgmap v14065: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 275 KiB/s wr, 8 op/s 2023-03-09T22:01:46.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:01:45 smithi038 ceph-mon[92422]: pgmap v14065: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 275 KiB/s wr, 8 op/s 2023-03-09T22:01:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:01:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25416.349160 secs 2023-03-09T22:01:47.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:01:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25416.349160 secs 2023-03-09T22:01:47.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:01:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25416.349160 secs 2023-03-09T22:01:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:01:47 smithi122 ceph-mon[93869]: pgmap v14066: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 283 KiB/s rd, 280 KiB/s wr, 8 op/s 2023-03-09T22:01:48.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:01:47 smithi181 ceph-mon[94189]: pgmap v14066: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 283 KiB/s rd, 280 KiB/s wr, 8 op/s 2023-03-09T22:01:48.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:01:47 smithi038 ceph-mon[92422]: pgmap v14066: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 283 KiB/s rd, 280 KiB/s wr, 8 op/s 2023-03-09T22:01:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25880.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:01:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:01:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25418.704781 secs 2023-03-09T22:01:49.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:01:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25418.704781 secs 2023-03-09T22:01:49.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:01:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25418.704781 secs 2023-03-09T22:01:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:01:49 smithi122 ceph-mon[93869]: pgmap v14067: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 338 KiB/s wr, 8 op/s 2023-03-09T22:01:50.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:01:49 smithi181 ceph-mon[94189]: pgmap v14067: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 338 KiB/s wr, 8 op/s 2023-03-09T22:01:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:01:49 smithi038 ceph-mon[92422]: pgmap v14067: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 338 KiB/s wr, 8 op/s 2023-03-09T22:01:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:01:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25421.349220 secs 2023-03-09T22:01:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:01:51 smithi122 ceph-mon[93869]: pgmap v14068: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 338 KiB/s wr, 8 op/s 2023-03-09T22:01:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:01:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25421.349220 secs 2023-03-09T22:01:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:01:51 smithi181 ceph-mon[94189]: pgmap v14068: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 338 KiB/s wr, 8 op/s 2023-03-09T22:01:52.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:01:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25421.349220 secs 2023-03-09T22:01:52.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:01:51 smithi038 ceph-mon[92422]: pgmap v14068: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 338 KiB/s wr, 8 op/s 2023-03-09T22:01:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25885.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:01:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:01:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25423.704891 secs 2023-03-09T22:01:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:01:53 smithi122 ceph-mon[93869]: pgmap v14069: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 560 KiB/s rd, 544 KiB/s wr, 12 op/s 2023-03-09T22:01:54.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:01:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25423.704891 secs 2023-03-09T22:01:54.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:01:53 smithi181 ceph-mon[94189]: pgmap v14069: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 560 KiB/s rd, 544 KiB/s wr, 12 op/s 2023-03-09T22:01:54.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:01:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25423.704891 secs 2023-03-09T22:01:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:01:53 smithi038 ceph-mon[92422]: pgmap v14069: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 560 KiB/s rd, 544 KiB/s wr, 12 op/s 2023-03-09T22:01:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:01:55 smithi122 ceph-mon[93869]: pgmap v14070: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 444 KiB/s rd, 429 KiB/s wr, 10 op/s 2023-03-09T22:01:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:01:55 smithi181 ceph-mon[94189]: pgmap v14070: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 444 KiB/s rd, 429 KiB/s wr, 10 op/s 2023-03-09T22:01:56.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:01:55 smithi038 ceph-mon[92422]: pgmap v14070: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 444 KiB/s rd, 429 KiB/s wr, 10 op/s 2023-03-09T22:01:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:01:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25426.349280 secs 2023-03-09T22:01:57.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:01:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25426.349280 secs 2023-03-09T22:01:57.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:01:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25426.349280 secs 2023-03-09T22:01:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:01:57 smithi122 ceph-mon[93869]: pgmap v14071: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 680 KiB/s rd, 664 KiB/s wr, 13 op/s 2023-03-09T22:01:58.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:01:57 smithi181 ceph-mon[94189]: pgmap v14071: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 680 KiB/s rd, 664 KiB/s wr, 13 op/s 2023-03-09T22:01:58.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:01:57 smithi038 ceph-mon[92422]: pgmap v14071: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 680 KiB/s rd, 664 KiB/s wr, 13 op/s 2023-03-09T22:01:58.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25890.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:01:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:01:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25428.705027 secs 2023-03-09T22:01:59.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:01:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25428.705027 secs 2023-03-09T22:01:59.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:01:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25428.705027 secs 2023-03-09T22:02:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:01:59 smithi122 ceph-mon[93869]: pgmap v14072: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 612 KiB/s rd, 598 KiB/s wr, 11 op/s 2023-03-09T22:02:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:01:59 smithi181 ceph-mon[94189]: pgmap v14072: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 612 KiB/s rd, 598 KiB/s wr, 11 op/s 2023-03-09T22:02:00.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:01:59 smithi038 ceph-mon[92422]: pgmap v14072: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 612 KiB/s rd, 598 KiB/s wr, 11 op/s 2023-03-09T22:02:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:02:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25431.349371 secs 2023-03-09T22:02:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:02:01 smithi122 ceph-mon[93869]: pgmap v14073: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 525 KiB/s rd, 510 KiB/s wr, 10 op/s 2023-03-09T22:02:02.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:02:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25431.349371 secs 2023-03-09T22:02:02.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:02:01 smithi181 ceph-mon[94189]: pgmap v14073: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 525 KiB/s rd, 510 KiB/s wr, 10 op/s 2023-03-09T22:02:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:02:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25431.349371 secs 2023-03-09T22:02:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:02:01 smithi038 ceph-mon[92422]: pgmap v14073: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 525 KiB/s rd, 510 KiB/s wr, 10 op/s 2023-03-09T22:02:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25895.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:02:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:02:03 smithi122 ceph-mon[93869]: pgmap v14074: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 720 KiB/s rd, 659 KiB/s wr, 14 op/s 2023-03-09T22:02:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:02:03 smithi181 ceph-mon[94189]: pgmap v14074: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 720 KiB/s rd, 659 KiB/s wr, 14 op/s 2023-03-09T22:02:04.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:02:03 smithi038 ceph-mon[92422]: pgmap v14074: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 720 KiB/s rd, 659 KiB/s wr, 14 op/s 2023-03-09T22:02:05.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:02:04 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:02:05.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:02:04 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:02:05.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:02:04 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:02:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:02:05 smithi122 ceph-mon[93869]: pgmap v14075: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 511 KiB/s rd, 452 KiB/s wr, 11 op/s 2023-03-09T22:02:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:02:05 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:02:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:02:05 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:02:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:02:05 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:02:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:02:05 smithi181 ceph-mon[94189]: pgmap v14075: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 511 KiB/s rd, 452 KiB/s wr, 11 op/s 2023-03-09T22:02:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:02:05 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:02:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:02:05 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:02:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:02:05 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:02:06.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:02:05 smithi038 ceph-mon[92422]: pgmap v14075: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 511 KiB/s rd, 452 KiB/s wr, 11 op/s 2023-03-09T22:02:06.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:02:05 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:02:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:02:05 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:02:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:02:05 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:02:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:02:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25436.349432 secs 2023-03-09T22:02:07.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:02:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25436.349432 secs 2023-03-09T22:02:07.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:02:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25436.349432 secs 2023-03-09T22:02:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:02:07 smithi122 ceph-mon[93869]: pgmap v14076: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 888 KiB/s rd, 624 KiB/s wr, 17 op/s 2023-03-09T22:02:08.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:02:07 smithi181 ceph-mon[94189]: pgmap v14076: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 888 KiB/s rd, 624 KiB/s wr, 17 op/s 2023-03-09T22:02:08.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:02:07 smithi038 ceph-mon[92422]: pgmap v14076: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 888 KiB/s rd, 624 KiB/s wr, 17 op/s 2023-03-09T22:02:08.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25900.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:02:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:02:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25438.705256 secs 2023-03-09T22:02:09.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:02:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25438.705256 secs 2023-03-09T22:02:09.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:02:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25438.705256 secs 2023-03-09T22:02:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:02:09 smithi122 ceph-mon[93869]: pgmap v14077: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 660 KiB/s rd, 411 KiB/s wr, 24 op/s 2023-03-09T22:02:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:02:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:02:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:02:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:02:10.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:02:09 smithi181 ceph-mon[94189]: pgmap v14077: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 660 KiB/s rd, 411 KiB/s wr, 24 op/s 2023-03-09T22:02:10.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:02:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:02:10.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:02:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:02:10.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:02:09 smithi038 ceph-mon[92422]: pgmap v14077: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 660 KiB/s rd, 411 KiB/s wr, 24 op/s 2023-03-09T22:02:10.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:02:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:02:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:02:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:02:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:02:10 smithi122 ceph-mon[93869]: osdmap e1093: 12 total, 12 up, 12 in 2023-03-09T22:02:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:02:10 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:02:11.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:02:10 smithi181 ceph-mon[94189]: osdmap e1093: 12 total, 12 up, 12 in 2023-03-09T22:02:11.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:02:10 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:02:11.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:02:10 smithi038 ceph-mon[92422]: osdmap e1093: 12 total, 12 up, 12 in 2023-03-09T22:02:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:02:10 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:02:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:02:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25441.349497 secs 2023-03-09T22:02:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:02:11 smithi122 ceph-mon[93869]: pgmap v14079: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 965 KiB/s rd, 666 KiB/s wr, 29 op/s 2023-03-09T22:02:12.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:02:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25441.349497 secs 2023-03-09T22:02:12.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:02:11 smithi181 ceph-mon[94189]: pgmap v14079: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 965 KiB/s rd, 666 KiB/s wr, 29 op/s 2023-03-09T22:02:12.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:02:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25441.349497 secs 2023-03-09T22:02:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:02:11 smithi038 ceph-mon[92422]: pgmap v14079: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 965 KiB/s rd, 666 KiB/s wr, 29 op/s 2023-03-09T22:02:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25905.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:02:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:02:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25443.705366 secs 2023-03-09T22:02:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:02:13 smithi122 ceph-mon[93869]: pgmap v14080: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 765 KiB/s rd, 535 KiB/s wr, 27 op/s 2023-03-09T22:02:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:02:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25443.705366 secs 2023-03-09T22:02:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:02:13 smithi181 ceph-mon[94189]: pgmap v14080: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 765 KiB/s rd, 535 KiB/s wr, 27 op/s 2023-03-09T22:02:14.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:02:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25443.705366 secs 2023-03-09T22:02:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:02:13 smithi038 ceph-mon[92422]: pgmap v14080: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 765 KiB/s rd, 535 KiB/s wr, 27 op/s 2023-03-09T22:02:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:02:15 smithi122 ceph-mon[93869]: pgmap v14081: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 764 KiB/s rd, 535 KiB/s wr, 26 op/s 2023-03-09T22:02:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:02:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25446.349587 secs 2023-03-09T22:02:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:02:15 smithi122 ceph-mon[93869]: osdmap e1094: 12 total, 12 up, 12 in 2023-03-09T22:02:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:02:15 smithi181 ceph-mon[94189]: pgmap v14081: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 764 KiB/s rd, 535 KiB/s wr, 26 op/s 2023-03-09T22:02:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:02:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25446.349587 secs 2023-03-09T22:02:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:02:15 smithi181 ceph-mon[94189]: osdmap e1094: 12 total, 12 up, 12 in 2023-03-09T22:02:16.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:02:15 smithi038 ceph-mon[92422]: pgmap v14081: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 764 KiB/s rd, 535 KiB/s wr, 26 op/s 2023-03-09T22:02:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:02:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25446.349587 secs 2023-03-09T22:02:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:02:15 smithi038 ceph-mon[92422]: osdmap e1094: 12 total, 12 up, 12 in 2023-03-09T22:02:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:02:17 smithi122 ceph-mon[93869]: pgmap v14083: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 409 KiB/s wr, 11 op/s 2023-03-09T22:02:18.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:02:17 smithi181 ceph-mon[94189]: pgmap v14083: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 409 KiB/s wr, 11 op/s 2023-03-09T22:02:18.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:02:17 smithi038 ceph-mon[92422]: pgmap v14083: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 409 KiB/s wr, 11 op/s 2023-03-09T22:02:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25910.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:02:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:02:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25448.705468 secs 2023-03-09T22:02:19.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:02:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25448.705468 secs 2023-03-09T22:02:19.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:02:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25448.705468 secs 2023-03-09T22:02:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:02:19 smithi122 ceph-mon[93869]: pgmap v14084: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 478 KiB/s wr, 11 op/s 2023-03-09T22:02:20.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:02:19 smithi181 ceph-mon[94189]: pgmap v14084: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 478 KiB/s wr, 11 op/s 2023-03-09T22:02:20.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:02:19 smithi038 ceph-mon[92422]: pgmap v14084: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 478 KiB/s wr, 11 op/s 2023-03-09T22:02:21.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:02:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25451.349679 secs 2023-03-09T22:02:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:02:21 smithi038 ceph-mon[92422]: pgmap v14085: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 158 KiB/s wr, 9 op/s 2023-03-09T22:02:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:02:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25451.349679 secs 2023-03-09T22:02:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:02:21 smithi122 ceph-mon[93869]: pgmap v14085: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 158 KiB/s wr, 9 op/s 2023-03-09T22:02:22.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:02:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25451.349679 secs 2023-03-09T22:02:22.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:02:21 smithi181 ceph-mon[94189]: pgmap v14085: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 158 KiB/s wr, 9 op/s 2023-03-09T22:02:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:02:22 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:02:23.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:02:22 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:02:23.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:02:22 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:02:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25915.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:02:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:02:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25453.705602 secs 2023-03-09T22:02:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:02:23 smithi122 ceph-mon[93869]: pgmap v14086: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 136 KiB/s wr, 9 op/s 2023-03-09T22:02:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:02:23 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:02:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:02:23 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:02:24.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:02:23 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:02:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:02:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25453.705602 secs 2023-03-09T22:02:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:02:23 smithi181 ceph-mon[94189]: pgmap v14086: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 136 KiB/s wr, 9 op/s 2023-03-09T22:02:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:02:23 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:02:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:02:23 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:02:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:02:23 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:02:24.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:02:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25453.705602 secs 2023-03-09T22:02:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:02:23 smithi038 ceph-mon[92422]: pgmap v14086: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 136 KiB/s wr, 9 op/s 2023-03-09T22:02:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:02:23 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:02:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:02:23 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:02:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:02:23 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:02:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:02:25 smithi122 ceph-mon[93869]: pgmap v14087: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 137 KiB/s wr, 9 op/s 2023-03-09T22:02:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:02:25 smithi181 ceph-mon[94189]: pgmap v14087: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 137 KiB/s wr, 9 op/s 2023-03-09T22:02:26.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:02:25 smithi038 ceph-mon[92422]: pgmap v14087: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 137 KiB/s wr, 9 op/s 2023-03-09T22:02:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:02:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25456.349758 secs 2023-03-09T22:02:27.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:02:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25456.349758 secs 2023-03-09T22:02:27.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:02:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25456.349758 secs 2023-03-09T22:02:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:02:27 smithi122 ceph-mon[93869]: pgmap v14088: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 104 KiB/s wr, 8 op/s 2023-03-09T22:02:28.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:02:27 smithi181 ceph-mon[94189]: pgmap v14088: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 104 KiB/s wr, 8 op/s 2023-03-09T22:02:28.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:02:27 smithi038 ceph-mon[92422]: pgmap v14088: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 104 KiB/s wr, 8 op/s 2023-03-09T22:02:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25920.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:02:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:02:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25458.705740 secs 2023-03-09T22:02:29.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:02:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25458.705740 secs 2023-03-09T22:02:29.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:02:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25458.705740 secs 2023-03-09T22:02:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:02:29 smithi122 ceph-mon[93869]: pgmap v14089: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 179 KiB/s wr, 8 op/s 2023-03-09T22:02:30.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:02:29 smithi181 ceph-mon[94189]: pgmap v14089: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 179 KiB/s wr, 8 op/s 2023-03-09T22:02:30.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:02:29 smithi038 ceph-mon[92422]: pgmap v14089: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 179 KiB/s wr, 8 op/s 2023-03-09T22:02:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:02:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25461.349819 secs 2023-03-09T22:02:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:02:31 smithi122 ceph-mon[93869]: pgmap v14090: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 108 KiB/s wr, 7 op/s 2023-03-09T22:02:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:02:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25461.349819 secs 2023-03-09T22:02:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:02:31 smithi181 ceph-mon[94189]: pgmap v14090: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 108 KiB/s wr, 7 op/s 2023-03-09T22:02:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:02:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25461.349819 secs 2023-03-09T22:02:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:02:31 smithi038 ceph-mon[92422]: pgmap v14090: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 108 KiB/s wr, 7 op/s 2023-03-09T22:02:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25925.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:02:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:02:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25463.705853 secs 2023-03-09T22:02:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:02:33 smithi122 ceph-mon[93869]: pgmap v14091: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 213 KiB/s wr, 9 op/s 2023-03-09T22:02:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:02:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25463.705853 secs 2023-03-09T22:02:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:02:33 smithi181 ceph-mon[94189]: pgmap v14091: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 213 KiB/s wr, 9 op/s 2023-03-09T22:02:34.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:02:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25463.705853 secs 2023-03-09T22:02:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:02:33 smithi038 ceph-mon[92422]: pgmap v14091: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 213 KiB/s wr, 9 op/s 2023-03-09T22:02:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:02:35 smithi122 ceph-mon[93869]: pgmap v14092: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 192 KiB/s wr, 7 op/s 2023-03-09T22:02:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:02:35 smithi181 ceph-mon[94189]: pgmap v14092: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 192 KiB/s wr, 7 op/s 2023-03-09T22:02:36.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:02:35 smithi038 ceph-mon[92422]: pgmap v14092: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 192 KiB/s wr, 7 op/s 2023-03-09T22:02:37.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:02:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25466.349892 secs 2023-03-09T22:02:37.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:02:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25466.349892 secs 2023-03-09T22:02:37.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:02:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25466.349892 secs 2023-03-09T22:02:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:02:37 smithi122 ceph-mon[93869]: pgmap v14093: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 192 KiB/s wr, 9 op/s 2023-03-09T22:02:38.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:02:37 smithi181 ceph-mon[94189]: pgmap v14093: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 192 KiB/s wr, 9 op/s 2023-03-09T22:02:38.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:02:37 smithi038 ceph-mon[92422]: pgmap v14093: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 192 KiB/s wr, 9 op/s 2023-03-09T22:02:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25930.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:02:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:02:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25468.705971 secs 2023-03-09T22:02:39.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:02:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25468.705971 secs 2023-03-09T22:02:39.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:02:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25468.705971 secs 2023-03-09T22:02:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:02:39 smithi122 ceph-mon[93869]: pgmap v14094: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 191 KiB/s wr, 7 op/s 2023-03-09T22:02:40.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:02:39 smithi181 ceph-mon[94189]: pgmap v14094: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 191 KiB/s wr, 7 op/s 2023-03-09T22:02:40.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:02:39 smithi038 ceph-mon[92422]: pgmap v14094: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 191 KiB/s wr, 7 op/s 2023-03-09T22:02:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:02:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25471.349964 secs 2023-03-09T22:02:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:02:41 smithi122 ceph-mon[93869]: pgmap v14095: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 105 KiB/s wr, 6 op/s 2023-03-09T22:02:42.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:02:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25471.349964 secs 2023-03-09T22:02:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:02:41 smithi181 ceph-mon[94189]: pgmap v14095: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 105 KiB/s wr, 6 op/s 2023-03-09T22:02:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:02:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25471.349964 secs 2023-03-09T22:02:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:02:41 smithi038 ceph-mon[92422]: pgmap v14095: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 105 KiB/s wr, 6 op/s 2023-03-09T22:02:43.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25935.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:02:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:02:43 smithi122 ceph-mon[93869]: pgmap v14096: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 239 KiB/s wr, 9 op/s 2023-03-09T22:02:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:02:43 smithi181 ceph-mon[94189]: pgmap v14096: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 239 KiB/s wr, 9 op/s 2023-03-09T22:02:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:02:43 smithi038 ceph-mon[92422]: pgmap v14096: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 239 KiB/s wr, 9 op/s 2023-03-09T22:02:45.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:02:44 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:02:45.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:02:44 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:02:45.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:02:44 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:02:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:02:45 smithi122 ceph-mon[93869]: pgmap v14097: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 134 KiB/s wr, 6 op/s 2023-03-09T22:02:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:02:45 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:02:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:02:45 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:02:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:02:45 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:02:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:02:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25476.350050 secs 2023-03-09T22:02:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:02:45 smithi181 ceph-mon[94189]: pgmap v14097: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 134 KiB/s wr, 6 op/s 2023-03-09T22:02:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:02:45 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:02:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:02:45 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:02:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:02:45 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:02:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:02:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25476.350050 secs 2023-03-09T22:02:46.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:02:45 smithi038 ceph-mon[92422]: pgmap v14097: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 134 KiB/s wr, 6 op/s 2023-03-09T22:02:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:02:45 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:02:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:02:45 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:02:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:02:45 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:02:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:02:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25476.350050 secs 2023-03-09T22:02:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:02:47 smithi122 ceph-mon[93869]: pgmap v14098: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 155 KiB/s wr, 9 op/s 2023-03-09T22:02:48.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:02:47 smithi181 ceph-mon[94189]: pgmap v14098: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 155 KiB/s wr, 9 op/s 2023-03-09T22:02:48.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:02:47 smithi038 ceph-mon[92422]: pgmap v14098: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 155 KiB/s wr, 9 op/s 2023-03-09T22:02:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25940.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:02:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:02:49 smithi122 ceph-mon[93869]: pgmap v14099: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 356 KiB/s wr, 8 op/s 2023-03-09T22:02:50.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:02:49 smithi181 ceph-mon[94189]: pgmap v14099: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 356 KiB/s wr, 8 op/s 2023-03-09T22:02:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:02:49 smithi038 ceph-mon[92422]: pgmap v14099: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 356 KiB/s wr, 8 op/s 2023-03-09T22:02:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:02:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25481.350107 secs 2023-03-09T22:02:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:02:51 smithi038 ceph-mon[92422]: pgmap v14100: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 448 KiB/s wr, 8 op/s 2023-03-09T22:02:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:02:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25481.350107 secs 2023-03-09T22:02:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:02:51 smithi122 ceph-mon[93869]: pgmap v14100: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 448 KiB/s wr, 8 op/s 2023-03-09T22:02:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:02:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25481.350107 secs 2023-03-09T22:02:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:02:51 smithi181 ceph-mon[94189]: pgmap v14100: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 448 KiB/s wr, 8 op/s 2023-03-09T22:02:53.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25945.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:02:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:02:53 smithi122 ceph-mon[93869]: pgmap v14101: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 523 KiB/s rd, 507 KiB/s wr, 12 op/s 2023-03-09T22:02:54.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:02:53 smithi181 ceph-mon[94189]: pgmap v14101: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 523 KiB/s rd, 507 KiB/s wr, 12 op/s 2023-03-09T22:02:54.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:02:53 smithi038 ceph-mon[92422]: pgmap v14101: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 523 KiB/s rd, 507 KiB/s wr, 12 op/s 2023-03-09T22:02:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:02:55 smithi122 ceph-mon[93869]: pgmap v14102: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 413 KiB/s wr, 9 op/s 2023-03-09T22:02:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:02:55 smithi181 ceph-mon[94189]: pgmap v14102: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 413 KiB/s wr, 9 op/s 2023-03-09T22:02:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:02:55 smithi038 ceph-mon[92422]: pgmap v14102: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 413 KiB/s wr, 9 op/s 2023-03-09T22:02:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:02:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25486.350172 secs 2023-03-09T22:02:57.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:02:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25486.350172 secs 2023-03-09T22:02:57.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:02:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25486.350172 secs 2023-03-09T22:02:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:02:57 smithi122 ceph-mon[93869]: pgmap v14103: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 669 KiB/s rd, 692 KiB/s wr, 13 op/s 2023-03-09T22:02:58.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:02:57 smithi181 ceph-mon[94189]: pgmap v14103: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 669 KiB/s rd, 692 KiB/s wr, 13 op/s 2023-03-09T22:02:58.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:02:57 smithi038 ceph-mon[92422]: pgmap v14103: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 669 KiB/s rd, 692 KiB/s wr, 13 op/s 2023-03-09T22:02:58.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25950.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:03:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:02:59 smithi122 ceph-mon[93869]: pgmap v14104: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 647 KiB/s rd, 671 KiB/s wr, 11 op/s 2023-03-09T22:03:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:02:59 smithi181 ceph-mon[94189]: pgmap v14104: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 647 KiB/s rd, 671 KiB/s wr, 11 op/s 2023-03-09T22:03:00.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:02:59 smithi038 ceph-mon[92422]: pgmap v14104: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 647 KiB/s rd, 671 KiB/s wr, 11 op/s 2023-03-09T22:03:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:03:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25491.350232 secs 2023-03-09T22:03:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:03:01 smithi122 ceph-mon[93869]: pgmap v14105: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 484 KiB/s rd, 513 KiB/s wr, 11 op/s 2023-03-09T22:03:02.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:03:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25491.350232 secs 2023-03-09T22:03:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:03:01 smithi181 ceph-mon[94189]: pgmap v14105: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 484 KiB/s rd, 513 KiB/s wr, 11 op/s 2023-03-09T22:03:02.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:03:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25491.350232 secs 2023-03-09T22:03:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:03:01 smithi038 ceph-mon[92422]: pgmap v14105: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 484 KiB/s rd, 513 KiB/s wr, 11 op/s 2023-03-09T22:03:03.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25955.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:03:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:03:03 smithi122 ceph-mon[93869]: pgmap v14106: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 383 KiB/s rd, 422 KiB/s wr, 13 op/s 2023-03-09T22:03:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:03:03 smithi181 ceph-mon[94189]: pgmap v14106: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 383 KiB/s rd, 422 KiB/s wr, 13 op/s 2023-03-09T22:03:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:03:03 smithi038 ceph-mon[92422]: pgmap v14106: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 383 KiB/s rd, 422 KiB/s wr, 13 op/s 2023-03-09T22:03:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:03:05 smithi122 ceph-mon[93869]: pgmap v14107: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 632 KiB/s wr, 22 op/s 2023-03-09T22:03:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:03:05 smithi181 ceph-mon[94189]: pgmap v14107: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 632 KiB/s wr, 22 op/s 2023-03-09T22:03:06.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:03:05 smithi038 ceph-mon[92422]: pgmap v14107: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 632 KiB/s wr, 22 op/s 2023-03-09T22:03:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:03:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25496.350292 secs 2023-03-09T22:03:07.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:03:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25496.350292 secs 2023-03-09T22:03:07.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:03:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25496.350292 secs 2023-03-09T22:03:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:03:07 smithi122 ceph-mon[93869]: pgmap v14108: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 768 KiB/s rd, 763 KiB/s wr, 26 op/s 2023-03-09T22:03:08.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:03:07 smithi181 ceph-mon[94189]: pgmap v14108: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 768 KiB/s rd, 763 KiB/s wr, 26 op/s 2023-03-09T22:03:08.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:03:07 smithi038 ceph-mon[92422]: pgmap v14108: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 768 KiB/s rd, 763 KiB/s wr, 26 op/s 2023-03-09T22:03:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25960.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:03:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:03:08 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 25498.706685 secs 2023-03-09T22:03:09.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:03:08 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 25498.706685 secs 2023-03-09T22:03:09.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:03:08 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 25498.706685 secs 2023-03-09T22:03:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:03:09 smithi122 ceph-mon[93869]: pgmap v14109: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 487 KiB/s wr, 22 op/s 2023-03-09T22:03:10.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:03:09 smithi181 ceph-mon[94189]: pgmap v14109: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 487 KiB/s wr, 22 op/s 2023-03-09T22:03:10.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:03:09 smithi038 ceph-mon[92422]: pgmap v14109: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 487 KiB/s wr, 22 op/s 2023-03-09T22:03:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:03:10 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:03:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:03:10 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:03:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:03:10 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:03:11.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:03:10 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:03:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:03:10 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:03:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:03:10 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:03:11.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:03:10 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:03:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:03:10 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:03:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:03:10 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:03:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:03:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25501.350353 secs 2023-03-09T22:03:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:03:11 smithi122 ceph-mon[93869]: pgmap v14110: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 493 KiB/s rd, 487 KiB/s wr, 22 op/s 2023-03-09T22:03:12.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:03:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25501.350353 secs 2023-03-09T22:03:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:03:11 smithi181 ceph-mon[94189]: pgmap v14110: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 493 KiB/s rd, 487 KiB/s wr, 22 op/s 2023-03-09T22:03:12.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:03:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25501.350353 secs 2023-03-09T22:03:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:03:11 smithi038 ceph-mon[92422]: pgmap v14110: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 493 KiB/s rd, 487 KiB/s wr, 22 op/s 2023-03-09T22:03:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25965.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:03:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:03:13 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 25503.706816 secs 2023-03-09T22:03:14.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:03:13 smithi122 ceph-mon[93869]: pgmap v14111: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 555 KiB/s rd, 515 KiB/s wr, 23 op/s 2023-03-09T22:03:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:03:13 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 25503.706816 secs 2023-03-09T22:03:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:03:13 smithi181 ceph-mon[94189]: pgmap v14111: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 555 KiB/s rd, 515 KiB/s wr, 23 op/s 2023-03-09T22:03:14.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:03:13 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 25503.706816 secs 2023-03-09T22:03:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:03:13 smithi038 ceph-mon[92422]: pgmap v14111: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 555 KiB/s rd, 515 KiB/s wr, 23 op/s 2023-03-09T22:03:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:03:15 smithi122 ceph-mon[93869]: pgmap v14112: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 554 KiB/s rd, 514 KiB/s wr, 21 op/s 2023-03-09T22:03:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:03:15 smithi181 ceph-mon[94189]: pgmap v14112: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 554 KiB/s rd, 514 KiB/s wr, 21 op/s 2023-03-09T22:03:16.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:03:15 smithi038 ceph-mon[92422]: pgmap v14112: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 554 KiB/s rd, 514 KiB/s wr, 21 op/s 2023-03-09T22:03:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:03:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25506.350418 secs 2023-03-09T22:03:17.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:03:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25506.350418 secs 2023-03-09T22:03:17.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:03:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25506.350418 secs 2023-03-09T22:03:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:03:17 smithi122 ceph-mon[93869]: pgmap v14113: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 265 KiB/s wr, 12 op/s 2023-03-09T22:03:18.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:03:17 smithi181 ceph-mon[94189]: pgmap v14113: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 265 KiB/s wr, 12 op/s 2023-03-09T22:03:18.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:03:17 smithi038 ceph-mon[92422]: pgmap v14113: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 265 KiB/s wr, 12 op/s 2023-03-09T22:03:18.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25970.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:03:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:03:18 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 25508.706948 secs 2023-03-09T22:03:19.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:03:18 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 25508.706948 secs 2023-03-09T22:03:19.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:03:18 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 25508.706948 secs 2023-03-09T22:03:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:03:19 smithi122 ceph-mon[93869]: pgmap v14114: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 95 KiB/s wr, 7 op/s 2023-03-09T22:03:20.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:03:19 smithi181 ceph-mon[94189]: pgmap v14114: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 95 KiB/s wr, 7 op/s 2023-03-09T22:03:20.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:03:19 smithi038 ceph-mon[92422]: pgmap v14114: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 95 KiB/s wr, 7 op/s 2023-03-09T22:03:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:03:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25511.350478 secs 2023-03-09T22:03:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:03:21 smithi122 ceph-mon[93869]: pgmap v14115: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 113 KiB/s wr, 8 op/s 2023-03-09T22:03:22.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:03:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25511.350478 secs 2023-03-09T22:03:22.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:03:21 smithi181 ceph-mon[94189]: pgmap v14115: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 113 KiB/s wr, 8 op/s 2023-03-09T22:03:22.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:03:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25511.350478 secs 2023-03-09T22:03:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:03:21 smithi038 ceph-mon[92422]: pgmap v14115: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 113 KiB/s wr, 8 op/s 2023-03-09T22:03:23.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25975.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:03:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:03:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25513.707074 secs 2023-03-09T22:03:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:03:23 smithi122 ceph-mon[93869]: pgmap v14116: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 341 KiB/s wr, 10 op/s 2023-03-09T22:03:24.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:03:23 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:03:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:03:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25513.707074 secs 2023-03-09T22:03:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:03:23 smithi181 ceph-mon[94189]: pgmap v14116: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 341 KiB/s wr, 10 op/s 2023-03-09T22:03:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:03:23 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:03:24.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:03:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25513.707074 secs 2023-03-09T22:03:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:03:23 smithi038 ceph-mon[92422]: pgmap v14116: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 341 KiB/s wr, 10 op/s 2023-03-09T22:03:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:03:23 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:03:25.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:03:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:03:25.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:03:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:03:25.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:03:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:03:25.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:03:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:03:25.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:03:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:03:25.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:03:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:03:25.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:03:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:03:25.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:03:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:03:25.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:03:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:03:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:03:25 smithi122 ceph-mon[93869]: pgmap v14117: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 280 KiB/s wr, 9 op/s 2023-03-09T22:03:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:03:25 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:03:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:03:25 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:03:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:03:25 smithi181 ceph-mon[94189]: pgmap v14117: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 280 KiB/s wr, 9 op/s 2023-03-09T22:03:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:03:25 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:03:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:03:25 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:03:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:03:25 smithi038 ceph-mon[92422]: pgmap v14117: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 280 KiB/s wr, 9 op/s 2023-03-09T22:03:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:03:25 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:03:26.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:03:25 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:03:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:03:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25516.350538 secs 2023-03-09T22:03:27.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:03:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25516.350538 secs 2023-03-09T22:03:27.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:03:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25516.350538 secs 2023-03-09T22:03:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:03:27 smithi122 ceph-mon[93869]: pgmap v14118: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 368 KiB/s wr, 12 op/s 2023-03-09T22:03:28.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:03:27 smithi181 ceph-mon[94189]: pgmap v14118: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 368 KiB/s wr, 12 op/s 2023-03-09T22:03:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:03:27 smithi038 ceph-mon[92422]: pgmap v14118: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 368 KiB/s wr, 12 op/s 2023-03-09T22:03:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25980.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:03:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:03:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25518.707194 secs 2023-03-09T22:03:29.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:03:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25518.707194 secs 2023-03-09T22:03:29.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:03:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25518.707194 secs 2023-03-09T22:03:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:03:29 smithi122 ceph-mon[93869]: pgmap v14119: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 328 KiB/s rd, 350 KiB/s wr, 10 op/s 2023-03-09T22:03:30.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:03:29 smithi181 ceph-mon[94189]: pgmap v14119: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 328 KiB/s rd, 350 KiB/s wr, 10 op/s 2023-03-09T22:03:30.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:03:29 smithi038 ceph-mon[92422]: pgmap v14119: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 328 KiB/s rd, 350 KiB/s wr, 10 op/s 2023-03-09T22:03:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:03:30 smithi122 ceph-mon[93869]: osdmap e1095: 12 total, 12 up, 12 in 2023-03-09T22:03:31.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:03:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25521.350600 secs 2023-03-09T22:03:31.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:03:30 smithi181 ceph-mon[94189]: osdmap e1095: 12 total, 12 up, 12 in 2023-03-09T22:03:31.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:03:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25521.350600 secs 2023-03-09T22:03:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:03:30 smithi038 ceph-mon[92422]: osdmap e1095: 12 total, 12 up, 12 in 2023-03-09T22:03:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:03:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25521.350600 secs 2023-03-09T22:03:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:03:31 smithi122 ceph-mon[93869]: pgmap v14121: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 396 KiB/s wr, 11 op/s 2023-03-09T22:03:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:03:31 smithi181 ceph-mon[94189]: pgmap v14121: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 396 KiB/s wr, 11 op/s 2023-03-09T22:03:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:03:31 smithi038 ceph-mon[92422]: pgmap v14121: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 396 KiB/s wr, 11 op/s 2023-03-09T22:03:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25985.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:03:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:03:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25523.707292 secs 2023-03-09T22:03:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:03:33 smithi122 ceph-mon[93869]: pgmap v14122: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 288 KiB/s wr, 10 op/s 2023-03-09T22:03:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:03:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25523.707292 secs 2023-03-09T22:03:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:03:33 smithi181 ceph-mon[94189]: pgmap v14122: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 288 KiB/s wr, 10 op/s 2023-03-09T22:03:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:03:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25523.707292 secs 2023-03-09T22:03:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:03:33 smithi038 ceph-mon[92422]: pgmap v14122: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 288 KiB/s wr, 10 op/s 2023-03-09T22:03:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:03:35 smithi122 ceph-mon[93869]: pgmap v14123: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 276 KiB/s wr, 9 op/s 2023-03-09T22:03:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:03:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25526.350661 secs 2023-03-09T22:03:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:03:35 smithi122 ceph-mon[93869]: osdmap e1096: 12 total, 12 up, 12 in 2023-03-09T22:03:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:03:35 smithi181 ceph-mon[94189]: pgmap v14123: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 276 KiB/s wr, 9 op/s 2023-03-09T22:03:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:03:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25526.350661 secs 2023-03-09T22:03:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:03:35 smithi181 ceph-mon[94189]: osdmap e1096: 12 total, 12 up, 12 in 2023-03-09T22:03:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:03:35 smithi038 ceph-mon[92422]: pgmap v14123: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 276 KiB/s wr, 9 op/s 2023-03-09T22:03:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:03:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25526.350661 secs 2023-03-09T22:03:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:03:35 smithi038 ceph-mon[92422]: osdmap e1096: 12 total, 12 up, 12 in 2023-03-09T22:03:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:03:37 smithi122 ceph-mon[93869]: pgmap v14125: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 321 KiB/s rd, 318 KiB/s wr, 10 op/s 2023-03-09T22:03:38.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:03:37 smithi181 ceph-mon[94189]: pgmap v14125: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 321 KiB/s rd, 318 KiB/s wr, 10 op/s 2023-03-09T22:03:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:03:37 smithi038 ceph-mon[92422]: pgmap v14125: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 321 KiB/s rd, 318 KiB/s wr, 10 op/s 2023-03-09T22:03:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25990.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:03:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:03:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25528.707409 secs 2023-03-09T22:03:39.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:03:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25528.707409 secs 2023-03-09T22:03:39.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:03:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25528.707409 secs 2023-03-09T22:03:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:03:39 smithi122 ceph-mon[93869]: pgmap v14126: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 295 KiB/s wr, 10 op/s 2023-03-09T22:03:40.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:03:39 smithi181 ceph-mon[94189]: pgmap v14126: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 295 KiB/s wr, 10 op/s 2023-03-09T22:03:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:03:39 smithi038 ceph-mon[92422]: pgmap v14126: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 295 KiB/s wr, 10 op/s 2023-03-09T22:03:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:03:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25531.350718 secs 2023-03-09T22:03:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:03:41 smithi122 ceph-mon[93869]: pgmap v14127: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 640 KiB/s wr, 10 op/s 2023-03-09T22:03:42.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:03:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25531.350718 secs 2023-03-09T22:03:42.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:03:41 smithi181 ceph-mon[94189]: pgmap v14127: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 640 KiB/s wr, 10 op/s 2023-03-09T22:03:42.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:03:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25531.350718 secs 2023-03-09T22:03:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:03:41 smithi038 ceph-mon[92422]: pgmap v14127: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 640 KiB/s wr, 10 op/s 2023-03-09T22:03:43.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 25995.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:03:44.033 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:03:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25533.707563 secs 2023-03-09T22:03:44.034 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:03:43 smithi181 ceph-mon[94189]: pgmap v14128: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 474 KiB/s wr, 9 op/s 2023-03-09T22:03:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:03:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25533.707563 secs 2023-03-09T22:03:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:03:43 smithi122 ceph-mon[93869]: pgmap v14128: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 474 KiB/s wr, 9 op/s 2023-03-09T22:03:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:03:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25533.707563 secs 2023-03-09T22:03:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:03:43 smithi038 ceph-mon[92422]: pgmap v14128: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 474 KiB/s wr, 9 op/s 2023-03-09T22:03:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:03:45 smithi122 ceph-mon[93869]: pgmap v14129: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 474 KiB/s wr, 9 op/s 2023-03-09T22:03:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:03:45 smithi181 ceph-mon[94189]: pgmap v14129: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 474 KiB/s wr, 9 op/s 2023-03-09T22:03:46.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:03:45 smithi038 ceph-mon[92422]: pgmap v14129: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 474 KiB/s wr, 9 op/s 2023-03-09T22:03:47.037 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:03:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25536.350777 secs 2023-03-09T22:03:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:03:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25536.350777 secs 2023-03-09T22:03:47.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:03:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25536.350777 secs 2023-03-09T22:03:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:03:47 smithi122 ceph-mon[93869]: pgmap v14130: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 555 KiB/s rd, 503 KiB/s wr, 10 op/s 2023-03-09T22:03:48.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:03:47 smithi181 ceph-mon[94189]: pgmap v14130: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 555 KiB/s rd, 503 KiB/s wr, 10 op/s 2023-03-09T22:03:48.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:03:47 smithi038 ceph-mon[92422]: pgmap v14130: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 555 KiB/s rd, 503 KiB/s wr, 10 op/s 2023-03-09T22:03:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26000.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:03:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:03:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25538.707646 secs 2023-03-09T22:03:49.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:03:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25538.707646 secs 2023-03-09T22:03:49.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:03:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25538.707646 secs 2023-03-09T22:03:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:03:49 smithi122 ceph-mon[93869]: pgmap v14131: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 451 KiB/s wr, 9 op/s 2023-03-09T22:03:50.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:03:49 smithi181 ceph-mon[94189]: pgmap v14131: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 451 KiB/s wr, 9 op/s 2023-03-09T22:03:50.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:03:49 smithi038 ceph-mon[92422]: pgmap v14131: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 451 KiB/s wr, 9 op/s 2023-03-09T22:03:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:03:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25541.350863 secs 2023-03-09T22:03:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:03:51 smithi122 ceph-mon[93869]: pgmap v14132: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 519 KiB/s rd, 472 KiB/s wr, 10 op/s 2023-03-09T22:03:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:03:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25541.350863 secs 2023-03-09T22:03:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:03:51 smithi181 ceph-mon[94189]: pgmap v14132: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 519 KiB/s rd, 472 KiB/s wr, 10 op/s 2023-03-09T22:03:52.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:03:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25541.350863 secs 2023-03-09T22:03:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:03:51 smithi038 ceph-mon[92422]: pgmap v14132: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 519 KiB/s rd, 472 KiB/s wr, 10 op/s 2023-03-09T22:03:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26005.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:03:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:03:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25543.707768 secs 2023-03-09T22:03:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:03:53 smithi122 ceph-mon[93869]: pgmap v14133: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 619 KiB/s rd, 384 KiB/s wr, 12 op/s 2023-03-09T22:03:54.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:03:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25543.707768 secs 2023-03-09T22:03:54.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:03:53 smithi181 ceph-mon[94189]: pgmap v14133: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 619 KiB/s rd, 384 KiB/s wr, 12 op/s 2023-03-09T22:03:54.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:03:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25543.707768 secs 2023-03-09T22:03:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:03:53 smithi038 ceph-mon[92422]: pgmap v14133: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 619 KiB/s rd, 384 KiB/s wr, 12 op/s 2023-03-09T22:03:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:03:55 smithi122 ceph-mon[93869]: pgmap v14134: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 668 KiB/s rd, 435 KiB/s wr, 11 op/s 2023-03-09T22:03:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:03:55 smithi181 ceph-mon[94189]: pgmap v14134: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 668 KiB/s rd, 435 KiB/s wr, 11 op/s 2023-03-09T22:03:56.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:03:55 smithi038 ceph-mon[92422]: pgmap v14134: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 668 KiB/s rd, 435 KiB/s wr, 11 op/s 2023-03-09T22:03:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:03:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25546.350923 secs 2023-03-09T22:03:57.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:03:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25546.350923 secs 2023-03-09T22:03:57.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:03:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25546.350923 secs 2023-03-09T22:03:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:03:57 smithi122 ceph-mon[93869]: pgmap v14135: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 658 KiB/s rd, 455 KiB/s wr, 14 op/s 2023-03-09T22:03:58.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:03:57 smithi181 ceph-mon[94189]: pgmap v14135: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 658 KiB/s rd, 455 KiB/s wr, 14 op/s 2023-03-09T22:03:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:03:57 smithi038 ceph-mon[92422]: pgmap v14135: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 658 KiB/s rd, 455 KiB/s wr, 14 op/s 2023-03-09T22:03:58.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26010.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:03:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:03:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25548.707886 secs 2023-03-09T22:03:59.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:03:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25548.707886 secs 2023-03-09T22:03:59.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:03:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25548.707886 secs 2023-03-09T22:04:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:03:59 smithi122 ceph-mon[93869]: pgmap v14136: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 618 KiB/s wr, 11 op/s 2023-03-09T22:04:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:03:59 smithi181 ceph-mon[94189]: pgmap v14136: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 618 KiB/s wr, 11 op/s 2023-03-09T22:04:00.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:03:59 smithi038 ceph-mon[92422]: pgmap v14136: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 618 KiB/s wr, 11 op/s 2023-03-09T22:04:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:04:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25551.350982 secs 2023-03-09T22:04:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:04:01 smithi122 ceph-mon[93869]: pgmap v14137: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 639 KiB/s rd, 638 KiB/s wr, 11 op/s 2023-03-09T22:04:02.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:04:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25551.350982 secs 2023-03-09T22:04:02.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:04:01 smithi181 ceph-mon[94189]: pgmap v14137: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 639 KiB/s rd, 638 KiB/s wr, 11 op/s 2023-03-09T22:04:02.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:04:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25551.350982 secs 2023-03-09T22:04:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:04:01 smithi038 ceph-mon[92422]: pgmap v14137: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 639 KiB/s rd, 638 KiB/s wr, 11 op/s 2023-03-09T22:04:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26015.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:04:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:04:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25553.708004 secs 2023-03-09T22:04:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:04:03 smithi122 ceph-mon[93869]: pgmap v14138: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 757 KiB/s rd, 766 KiB/s wr, 14 op/s 2023-03-09T22:04:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:04:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25553.708004 secs 2023-03-09T22:04:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:04:03 smithi181 ceph-mon[94189]: pgmap v14138: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 757 KiB/s rd, 766 KiB/s wr, 14 op/s 2023-03-09T22:04:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:04:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25553.708004 secs 2023-03-09T22:04:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:04:03 smithi038 ceph-mon[92422]: pgmap v14138: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 757 KiB/s rd, 766 KiB/s wr, 14 op/s 2023-03-09T22:04:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:04:05 smithi122 ceph-mon[93869]: pgmap v14139: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 530 KiB/s rd, 554 KiB/s wr, 23 op/s 2023-03-09T22:04:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:04:05 smithi181 ceph-mon[94189]: pgmap v14139: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 530 KiB/s rd, 554 KiB/s wr, 23 op/s 2023-03-09T22:04:06.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:04:05 smithi038 ceph-mon[92422]: pgmap v14139: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 530 KiB/s rd, 554 KiB/s wr, 23 op/s 2023-03-09T22:04:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:04:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25556.351045 secs 2023-03-09T22:04:07.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:04:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25556.351045 secs 2023-03-09T22:04:07.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:04:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25556.351045 secs 2023-03-09T22:04:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:04:07 smithi122 ceph-mon[93869]: pgmap v14140: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 514 KiB/s wr, 26 op/s 2023-03-09T22:04:08.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:04:07 smithi181 ceph-mon[94189]: pgmap v14140: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 514 KiB/s wr, 26 op/s 2023-03-09T22:04:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:04:07 smithi038 ceph-mon[92422]: pgmap v14140: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 514 KiB/s wr, 26 op/s 2023-03-09T22:04:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26020.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:04:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:04:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25558.708141 secs 2023-03-09T22:04:09.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:04:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25558.708141 secs 2023-03-09T22:04:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:04:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25558.708141 secs 2023-03-09T22:04:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:04:09 smithi122 ceph-mon[93869]: pgmap v14141: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 582 KiB/s rd, 596 KiB/s wr, 24 op/s 2023-03-09T22:04:10.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:04:09 smithi181 ceph-mon[94189]: pgmap v14141: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 582 KiB/s rd, 596 KiB/s wr, 24 op/s 2023-03-09T22:04:10.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:04:09 smithi038 ceph-mon[92422]: pgmap v14141: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 582 KiB/s rd, 596 KiB/s wr, 24 op/s 2023-03-09T22:04:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:04:10 smithi122 ceph-mon[93869]: osdmap e1097: 12 total, 12 up, 12 in 2023-03-09T22:04:11.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:04:10 smithi181 ceph-mon[94189]: osdmap e1097: 12 total, 12 up, 12 in 2023-03-09T22:04:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:04:10 smithi038 ceph-mon[92422]: osdmap e1097: 12 total, 12 up, 12 in 2023-03-09T22:04:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:04:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25561.351109 secs 2023-03-09T22:04:12.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:04:11 smithi122 ceph-mon[93869]: pgmap v14143: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 341 KiB/s wr, 26 op/s 2023-03-09T22:04:12.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:04:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25561.351109 secs 2023-03-09T22:04:12.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:04:11 smithi181 ceph-mon[94189]: pgmap v14143: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 341 KiB/s wr, 26 op/s 2023-03-09T22:04:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:04:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25561.351109 secs 2023-03-09T22:04:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:04:11 smithi038 ceph-mon[92422]: pgmap v14143: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 341 KiB/s wr, 26 op/s 2023-03-09T22:04:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26025.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:04:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:04:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25563.708264 secs 2023-03-09T22:04:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:04:13 smithi122 ceph-mon[93869]: pgmap v14144: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 162 KiB/s wr, 24 op/s 2023-03-09T22:04:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:04:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25563.708264 secs 2023-03-09T22:04:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:04:13 smithi181 ceph-mon[94189]: pgmap v14144: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 162 KiB/s wr, 24 op/s 2023-03-09T22:04:14.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:04:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25563.708264 secs 2023-03-09T22:04:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:04:13 smithi038 ceph-mon[92422]: pgmap v14144: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 162 KiB/s wr, 24 op/s 2023-03-09T22:04:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:04:15 smithi122 ceph-mon[93869]: pgmap v14145: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 137 KiB/s wr, 9 op/s 2023-03-09T22:04:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:04:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25566.351170 secs 2023-03-09T22:04:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:04:15 smithi122 ceph-mon[93869]: osdmap e1098: 12 total, 12 up, 12 in 2023-03-09T22:04:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:04:15 smithi181 ceph-mon[94189]: pgmap v14145: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 137 KiB/s wr, 9 op/s 2023-03-09T22:04:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:04:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25566.351170 secs 2023-03-09T22:04:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:04:15 smithi181 ceph-mon[94189]: osdmap e1098: 12 total, 12 up, 12 in 2023-03-09T22:04:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:04:15 smithi038 ceph-mon[92422]: pgmap v14145: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 137 KiB/s wr, 9 op/s 2023-03-09T22:04:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:04:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25566.351170 secs 2023-03-09T22:04:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:04:15 smithi038 ceph-mon[92422]: osdmap e1098: 12 total, 12 up, 12 in 2023-03-09T22:04:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:04:17 smithi122 ceph-mon[93869]: pgmap v14147: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 32 KiB/s wr, 8 op/s 2023-03-09T22:04:18.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:04:17 smithi181 ceph-mon[94189]: pgmap v14147: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 32 KiB/s wr, 8 op/s 2023-03-09T22:04:18.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:04:17 smithi038 ceph-mon[92422]: pgmap v14147: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 32 KiB/s wr, 8 op/s 2023-03-09T22:04:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26030.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:04:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:04:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25568.708368 secs 2023-03-09T22:04:19.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:04:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25568.708368 secs 2023-03-09T22:04:19.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:04:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25568.708368 secs 2023-03-09T22:04:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:04:19 smithi122 ceph-mon[93869]: pgmap v14148: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 9 op/s 2023-03-09T22:04:20.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:04:19 smithi181 ceph-mon[94189]: pgmap v14148: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 9 op/s 2023-03-09T22:04:20.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:04:19 smithi038 ceph-mon[92422]: pgmap v14148: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 9 op/s 2023-03-09T22:04:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:04:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25571.351229 secs 2023-03-09T22:04:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:04:21 smithi122 ceph-mon[93869]: pgmap v14149: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 51 KiB/s wr, 8 op/s 2023-03-09T22:04:22.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:04:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25571.351229 secs 2023-03-09T22:04:22.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:04:21 smithi181 ceph-mon[94189]: pgmap v14149: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 51 KiB/s wr, 8 op/s 2023-03-09T22:04:22.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:04:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25571.351229 secs 2023-03-09T22:04:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:04:21 smithi038 ceph-mon[92422]: pgmap v14149: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 51 KiB/s wr, 8 op/s 2023-03-09T22:04:23.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26035.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:04:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:04:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25573.708491 secs 2023-03-09T22:04:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:04:23 smithi122 ceph-mon[93869]: pgmap v14150: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 51 KiB/s wr, 8 op/s 2023-03-09T22:04:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:04:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25573.708491 secs 2023-03-09T22:04:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:04:23 smithi181 ceph-mon[94189]: pgmap v14150: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 51 KiB/s wr, 8 op/s 2023-03-09T22:04:24.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:04:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25573.708491 secs 2023-03-09T22:04:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:04:23 smithi038 ceph-mon[92422]: pgmap v14150: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 51 KiB/s wr, 8 op/s 2023-03-09T22:04:25.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:04:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:04:25.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:04:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:04:25.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:04:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:04:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:04:25 smithi122 ceph-mon[93869]: pgmap v14151: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 55 KiB/s wr, 8 op/s 2023-03-09T22:04:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:04:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:04:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:04:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:04:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:04:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:04:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:04:25 smithi181 ceph-mon[94189]: pgmap v14151: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 55 KiB/s wr, 8 op/s 2023-03-09T22:04:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:04:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:04:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:04:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:04:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:04:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:04:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:04:25 smithi038 ceph-mon[92422]: pgmap v14151: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 55 KiB/s wr, 8 op/s 2023-03-09T22:04:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:04:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:04:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:04:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:04:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:04:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:04:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:04:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25576.351291 secs 2023-03-09T22:04:27.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:04:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25576.351291 secs 2023-03-09T22:04:27.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:04:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25576.351291 secs 2023-03-09T22:04:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:04:27 smithi122 ceph-mon[93869]: pgmap v14152: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 28 KiB/s wr, 8 op/s 2023-03-09T22:04:28.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:04:27 smithi181 ceph-mon[94189]: pgmap v14152: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 28 KiB/s wr, 8 op/s 2023-03-09T22:04:28.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:04:27 smithi038 ceph-mon[92422]: pgmap v14152: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 28 KiB/s wr, 8 op/s 2023-03-09T22:04:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26040.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:04:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:04:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25578.708610 secs 2023-03-09T22:04:29.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:04:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25578.708610 secs 2023-03-09T22:04:29.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:04:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25578.708610 secs 2023-03-09T22:04:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:04:29 smithi122 ceph-mon[93869]: pgmap v14153: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 127 KiB/s wr, 8 op/s 2023-03-09T22:04:30.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:04:29 smithi181 ceph-mon[94189]: pgmap v14153: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 127 KiB/s wr, 8 op/s 2023-03-09T22:04:30.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:04:29 smithi038 ceph-mon[92422]: pgmap v14153: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 127 KiB/s wr, 8 op/s 2023-03-09T22:04:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:04:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25581.351353 secs 2023-03-09T22:04:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:04:31 smithi122 ceph-mon[93869]: pgmap v14154: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 127 KiB/s wr, 8 op/s 2023-03-09T22:04:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:04:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25581.351353 secs 2023-03-09T22:04:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:04:31 smithi181 ceph-mon[94189]: pgmap v14154: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 127 KiB/s wr, 8 op/s 2023-03-09T22:04:32.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:04:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25581.351353 secs 2023-03-09T22:04:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:04:31 smithi038 ceph-mon[92422]: pgmap v14154: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 127 KiB/s wr, 8 op/s 2023-03-09T22:04:33.793 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26045.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:04:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:04:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25583.708727 secs 2023-03-09T22:04:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:04:33 smithi122 ceph-mon[93869]: pgmap v14155: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 107 KiB/s wr, 9 op/s 2023-03-09T22:04:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:04:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25583.708727 secs 2023-03-09T22:04:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:04:33 smithi181 ceph-mon[94189]: pgmap v14155: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 107 KiB/s wr, 9 op/s 2023-03-09T22:04:34.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:04:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25583.708727 secs 2023-03-09T22:04:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:04:33 smithi038 ceph-mon[92422]: pgmap v14155: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 107 KiB/s wr, 9 op/s 2023-03-09T22:04:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:04:35 smithi122 ceph-mon[93869]: pgmap v14156: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 206 KiB/s wr, 7 op/s 2023-03-09T22:04:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:04:35 smithi181 ceph-mon[94189]: pgmap v14156: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 206 KiB/s wr, 7 op/s 2023-03-09T22:04:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:04:35 smithi038 ceph-mon[92422]: pgmap v14156: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 206 KiB/s wr, 7 op/s 2023-03-09T22:04:37.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:04:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25586.351416 secs 2023-03-09T22:04:37.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:04:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25586.351416 secs 2023-03-09T22:04:37.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:04:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25586.351416 secs 2023-03-09T22:04:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:04:37 smithi122 ceph-mon[93869]: pgmap v14157: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 203 KiB/s wr, 9 op/s 2023-03-09T22:04:38.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:04:37 smithi181 ceph-mon[94189]: pgmap v14157: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 203 KiB/s wr, 9 op/s 2023-03-09T22:04:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:04:37 smithi038 ceph-mon[92422]: pgmap v14157: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 203 KiB/s wr, 9 op/s 2023-03-09T22:04:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26050.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:04:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:04:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25588.708864 secs 2023-03-09T22:04:39.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:04:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25588.708864 secs 2023-03-09T22:04:39.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:04:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25588.708864 secs 2023-03-09T22:04:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:04:39 smithi122 ceph-mon[93869]: pgmap v14158: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 203 KiB/s wr, 7 op/s 2023-03-09T22:04:40.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:04:39 smithi181 ceph-mon[94189]: pgmap v14158: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 203 KiB/s wr, 7 op/s 2023-03-09T22:04:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:04:39 smithi038 ceph-mon[92422]: pgmap v14158: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 203 KiB/s wr, 7 op/s 2023-03-09T22:04:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:04:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25591.351474 secs 2023-03-09T22:04:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:04:41 smithi122 ceph-mon[93869]: pgmap v14159: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 206 KiB/s wr, 7 op/s 2023-03-09T22:04:42.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:04:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25591.351474 secs 2023-03-09T22:04:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:04:41 smithi181 ceph-mon[94189]: pgmap v14159: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 206 KiB/s wr, 7 op/s 2023-03-09T22:04:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:04:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25591.351474 secs 2023-03-09T22:04:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:04:41 smithi038 ceph-mon[92422]: pgmap v14159: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 206 KiB/s wr, 7 op/s 2023-03-09T22:04:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26055.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:04:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:04:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25593.708997 secs 2023-03-09T22:04:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:04:43 smithi122 ceph-mon[93869]: pgmap v14160: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 206 KiB/s wr, 8 op/s 2023-03-09T22:04:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:04:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25593.708997 secs 2023-03-09T22:04:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:04:43 smithi181 ceph-mon[94189]: pgmap v14160: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 206 KiB/s wr, 8 op/s 2023-03-09T22:04:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:04:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25593.708997 secs 2023-03-09T22:04:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:04:43 smithi038 ceph-mon[92422]: pgmap v14160: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 206 KiB/s wr, 8 op/s 2023-03-09T22:04:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:04:45 smithi122 ceph-mon[93869]: pgmap v14161: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 235 KiB/s wr, 7 op/s 2023-03-09T22:04:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:04:45 smithi181 ceph-mon[94189]: pgmap v14161: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 235 KiB/s wr, 7 op/s 2023-03-09T22:04:46.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:04:45 smithi038 ceph-mon[92422]: pgmap v14161: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 235 KiB/s wr, 7 op/s 2023-03-09T22:04:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:04:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25596.351556 secs 2023-03-09T22:04:47.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:04:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25596.351556 secs 2023-03-09T22:04:47.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:04:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25596.351556 secs 2023-03-09T22:04:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:04:47 smithi122 ceph-mon[93869]: pgmap v14162: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 348 KiB/s rd, 345 KiB/s wr, 10 op/s 2023-03-09T22:04:48.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:04:47 smithi181 ceph-mon[94189]: pgmap v14162: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 348 KiB/s rd, 345 KiB/s wr, 10 op/s 2023-03-09T22:04:48.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:04:47 smithi038 ceph-mon[92422]: pgmap v14162: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 348 KiB/s rd, 345 KiB/s wr, 10 op/s 2023-03-09T22:04:48.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26060.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:04:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:04:48 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:04:49.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:04:48 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:04:49.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:04:48 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:04:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:04:49 smithi122 ceph-mon[93869]: pgmap v14163: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 345 KiB/s wr, 8 op/s 2023-03-09T22:04:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:04:49 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:04:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:04:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:04:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:04:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:04:50.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:04:49 smithi181 ceph-mon[94189]: pgmap v14163: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 345 KiB/s wr, 8 op/s 2023-03-09T22:04:50.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:04:49 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:04:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:04:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:04:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:04:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:04:50.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:04:49 smithi038 ceph-mon[92422]: pgmap v14163: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 345 KiB/s wr, 8 op/s 2023-03-09T22:04:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:04:49 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:04:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:04:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:04:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:04:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:04:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:04:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25601.351615 secs 2023-03-09T22:04:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:04:51 smithi122 ceph-mon[93869]: pgmap v14164: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 494 KiB/s rd, 479 KiB/s wr, 10 op/s 2023-03-09T22:04:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:04:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25601.351615 secs 2023-03-09T22:04:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:04:51 smithi181 ceph-mon[94189]: pgmap v14164: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 494 KiB/s rd, 479 KiB/s wr, 10 op/s 2023-03-09T22:04:52.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:04:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25601.351615 secs 2023-03-09T22:04:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:04:51 smithi038 ceph-mon[92422]: pgmap v14164: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 494 KiB/s rd, 479 KiB/s wr, 10 op/s 2023-03-09T22:04:53.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26065.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:04:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:04:53 smithi122 ceph-mon[93869]: pgmap v14165: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 374 KiB/s wr, 11 op/s 2023-03-09T22:04:54.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:04:53 smithi181 ceph-mon[94189]: pgmap v14165: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 374 KiB/s wr, 11 op/s 2023-03-09T22:04:54.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:04:53 smithi038 ceph-mon[92422]: pgmap v14165: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 374 KiB/s wr, 11 op/s 2023-03-09T22:04:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:04:55 smithi122 ceph-mon[93869]: pgmap v14166: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 374 KiB/s wr, 9 op/s 2023-03-09T22:04:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:04:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25606.351674 secs 2023-03-09T22:04:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:04:55 smithi181 ceph-mon[94189]: pgmap v14166: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 374 KiB/s wr, 9 op/s 2023-03-09T22:04:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:04:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25606.351674 secs 2023-03-09T22:04:56.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:04:55 smithi038 ceph-mon[92422]: pgmap v14166: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 374 KiB/s wr, 9 op/s 2023-03-09T22:04:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:04:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25606.351674 secs 2023-03-09T22:04:58.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:04:57 smithi122 ceph-mon[93869]: pgmap v14167: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 770 KiB/s rd, 755 KiB/s wr, 13 op/s 2023-03-09T22:04:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:04:57 smithi181 ceph-mon[94189]: pgmap v14167: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 770 KiB/s rd, 755 KiB/s wr, 13 op/s 2023-03-09T22:04:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:04:57 smithi038 ceph-mon[92422]: pgmap v14167: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 770 KiB/s rd, 755 KiB/s wr, 13 op/s 2023-03-09T22:04:58.793 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26070.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:05:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:04:59 smithi122 ceph-mon[93869]: pgmap v14168: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 560 KiB/s rd, 545 KiB/s wr, 9 op/s 2023-03-09T22:05:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:04:59 smithi181 ceph-mon[94189]: pgmap v14168: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 560 KiB/s rd, 545 KiB/s wr, 9 op/s 2023-03-09T22:05:00.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:04:59 smithi038 ceph-mon[92422]: pgmap v14168: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 560 KiB/s rd, 545 KiB/s wr, 9 op/s 2023-03-09T22:05:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:05:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25611.351736 secs 2023-03-09T22:05:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:05:01 smithi122 ceph-mon[93869]: pgmap v14169: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 810 KiB/s rd, 795 KiB/s wr, 11 op/s 2023-03-09T22:05:02.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:05:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25611.351736 secs 2023-03-09T22:05:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:05:01 smithi181 ceph-mon[94189]: pgmap v14169: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 810 KiB/s rd, 795 KiB/s wr, 11 op/s 2023-03-09T22:05:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:05:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25611.351736 secs 2023-03-09T22:05:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:05:01 smithi038 ceph-mon[92422]: pgmap v14169: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 810 KiB/s rd, 795 KiB/s wr, 11 op/s 2023-03-09T22:05:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26075.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:05:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:05:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25613.709503 secs 2023-03-09T22:05:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:05:03 smithi122 ceph-mon[93869]: pgmap v14170: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 759 KiB/s rd, 757 KiB/s wr, 12 op/s 2023-03-09T22:05:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:05:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25613.709503 secs 2023-03-09T22:05:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:05:03 smithi181 ceph-mon[94189]: pgmap v14170: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 759 KiB/s rd, 757 KiB/s wr, 12 op/s 2023-03-09T22:05:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:05:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25613.709503 secs 2023-03-09T22:05:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:05:03 smithi038 ceph-mon[92422]: pgmap v14170: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 759 KiB/s rd, 757 KiB/s wr, 12 op/s 2023-03-09T22:05:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:05:05 smithi122 ceph-mon[93869]: pgmap v14171: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1022 KiB/s rd, 1.0 MiB/s wr, 23 op/s 2023-03-09T22:05:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:05:05 smithi122 ceph-mon[93869]: osdmap e1099: 12 total, 12 up, 12 in 2023-03-09T22:05:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:05:05 smithi181 ceph-mon[94189]: pgmap v14171: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1022 KiB/s rd, 1.0 MiB/s wr, 23 op/s 2023-03-09T22:05:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:05:05 smithi181 ceph-mon[94189]: osdmap e1099: 12 total, 12 up, 12 in 2023-03-09T22:05:06.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:05:05 smithi038 ceph-mon[92422]: pgmap v14171: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1022 KiB/s rd, 1.0 MiB/s wr, 23 op/s 2023-03-09T22:05:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:05:05 smithi038 ceph-mon[92422]: osdmap e1099: 12 total, 12 up, 12 in 2023-03-09T22:05:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:05:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25616.351796 secs 2023-03-09T22:05:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:05:06 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:05:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:05:06 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:05:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:05:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:05:07.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:05:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25616.351796 secs 2023-03-09T22:05:07.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:05:06 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:05:07.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:05:06 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:05:07.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:05:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:05:07.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:05:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25616.351796 secs 2023-03-09T22:05:07.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:05:06 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:05:07.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:05:06 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:05:07.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:05:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:05:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:05:07 smithi122 ceph-mon[93869]: pgmap v14173: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 736 KiB/s rd, 777 KiB/s wr, 28 op/s 2023-03-09T22:05:08.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:05:07 smithi181 ceph-mon[94189]: pgmap v14173: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 736 KiB/s rd, 777 KiB/s wr, 28 op/s 2023-03-09T22:05:08.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:05:07 smithi038 ceph-mon[92422]: pgmap v14173: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 736 KiB/s rd, 777 KiB/s wr, 28 op/s 2023-03-09T22:05:08.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26080.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:05:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:05:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25618.709647 secs 2023-03-09T22:05:09.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:05:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25618.709647 secs 2023-03-09T22:05:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:05:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25618.709647 secs 2023-03-09T22:05:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:05:09 smithi122 ceph-mon[93869]: pgmap v14174: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 736 KiB/s rd, 777 KiB/s wr, 28 op/s 2023-03-09T22:05:10.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:05:09 smithi181 ceph-mon[94189]: pgmap v14174: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 736 KiB/s rd, 777 KiB/s wr, 28 op/s 2023-03-09T22:05:10.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:05:09 smithi038 ceph-mon[92422]: pgmap v14174: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 736 KiB/s rd, 777 KiB/s wr, 28 op/s 2023-03-09T22:05:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:05:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25621.351873 secs 2023-03-09T22:05:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:05:11 smithi122 ceph-mon[93869]: pgmap v14175: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 443 KiB/s rd, 484 KiB/s wr, 28 op/s 2023-03-09T22:05:12.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:05:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25621.351873 secs 2023-03-09T22:05:12.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:05:11 smithi181 ceph-mon[94189]: pgmap v14175: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 443 KiB/s rd, 484 KiB/s wr, 28 op/s 2023-03-09T22:05:12.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:05:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25621.351873 secs 2023-03-09T22:05:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:05:11 smithi038 ceph-mon[92422]: pgmap v14175: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 443 KiB/s rd, 484 KiB/s wr, 28 op/s 2023-03-09T22:05:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26085.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:05:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:05:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25623.709758 secs 2023-03-09T22:05:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:05:13 smithi122 ceph-mon[93869]: pgmap v14176: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 328 KiB/s rd, 369 KiB/s wr, 27 op/s 2023-03-09T22:05:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:05:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25623.709758 secs 2023-03-09T22:05:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:05:13 smithi181 ceph-mon[94189]: pgmap v14176: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 328 KiB/s rd, 369 KiB/s wr, 27 op/s 2023-03-09T22:05:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:05:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25623.709758 secs 2023-03-09T22:05:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:05:13 smithi038 ceph-mon[92422]: pgmap v14176: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 328 KiB/s rd, 369 KiB/s wr, 27 op/s 2023-03-09T22:05:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:05:15 smithi122 ceph-mon[93869]: pgmap v14177: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 139 KiB/s wr, 11 op/s 2023-03-09T22:05:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:05:15 smithi122 ceph-mon[93869]: osdmap e1100: 12 total, 12 up, 12 in 2023-03-09T22:05:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:05:15 smithi181 ceph-mon[94189]: pgmap v14177: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 139 KiB/s wr, 11 op/s 2023-03-09T22:05:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:05:15 smithi181 ceph-mon[94189]: osdmap e1100: 12 total, 12 up, 12 in 2023-03-09T22:05:16.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:05:15 smithi038 ceph-mon[92422]: pgmap v14177: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 139 KiB/s wr, 11 op/s 2023-03-09T22:05:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:05:15 smithi038 ceph-mon[92422]: osdmap e1100: 12 total, 12 up, 12 in 2023-03-09T22:05:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:05:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25626.351933 secs 2023-03-09T22:05:17.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:05:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25626.351933 secs 2023-03-09T22:05:17.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:05:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25626.351933 secs 2023-03-09T22:05:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:05:17 smithi122 ceph-mon[93869]: pgmap v14179: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 130 KiB/s wr, 9 op/s 2023-03-09T22:05:18.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:05:17 smithi181 ceph-mon[94189]: pgmap v14179: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 130 KiB/s wr, 9 op/s 2023-03-09T22:05:18.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:05:17 smithi038 ceph-mon[92422]: pgmap v14179: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 130 KiB/s wr, 9 op/s 2023-03-09T22:05:18.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26090.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:05:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:05:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25628.709869 secs 2023-03-09T22:05:19.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:05:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25628.709869 secs 2023-03-09T22:05:19.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:05:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25628.709869 secs 2023-03-09T22:05:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:05:19 smithi122 ceph-mon[93869]: pgmap v14180: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 130 KiB/s wr, 9 op/s 2023-03-09T22:05:20.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:05:19 smithi181 ceph-mon[94189]: pgmap v14180: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 130 KiB/s wr, 9 op/s 2023-03-09T22:05:20.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:05:19 smithi038 ceph-mon[92422]: pgmap v14180: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 130 KiB/s wr, 9 op/s 2023-03-09T22:05:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:05:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25631.351992 secs 2023-03-09T22:05:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:05:21 smithi122 ceph-mon[93869]: pgmap v14181: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 122 KiB/s wr, 8 op/s 2023-03-09T22:05:22.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:05:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25631.351992 secs 2023-03-09T22:05:22.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:05:21 smithi181 ceph-mon[94189]: pgmap v14181: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 122 KiB/s wr, 8 op/s 2023-03-09T22:05:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:05:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25631.351992 secs 2023-03-09T22:05:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:05:21 smithi038 ceph-mon[92422]: pgmap v14181: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 122 KiB/s wr, 8 op/s 2023-03-09T22:05:23.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26095.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:05:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:05:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25633.709985 secs 2023-03-09T22:05:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:05:23 smithi122 ceph-mon[93869]: pgmap v14182: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 148 KiB/s wr, 9 op/s 2023-03-09T22:05:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:05:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25633.709985 secs 2023-03-09T22:05:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:05:23 smithi181 ceph-mon[94189]: pgmap v14182: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 148 KiB/s wr, 9 op/s 2023-03-09T22:05:24.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:05:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25633.709985 secs 2023-03-09T22:05:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:05:23 smithi038 ceph-mon[92422]: pgmap v14182: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 148 KiB/s wr, 9 op/s 2023-03-09T22:05:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:05:25 smithi122 ceph-mon[93869]: pgmap v14183: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 44 KiB/s wr, 9 op/s 2023-03-09T22:05:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:05:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:05:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:05:25 smithi181 ceph-mon[94189]: pgmap v14183: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 44 KiB/s wr, 9 op/s 2023-03-09T22:05:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:05:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:05:26.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:05:25 smithi038 ceph-mon[92422]: pgmap v14183: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 44 KiB/s wr, 9 op/s 2023-03-09T22:05:26.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:05:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:05:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:05:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25636.352053 secs 2023-03-09T22:05:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:05:26 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:05:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:05:26 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:05:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:05:26 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:05:27.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:05:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25636.352053 secs 2023-03-09T22:05:27.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:05:26 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:05:27.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:05:26 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:05:27.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:05:26 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:05:27.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:05:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25636.352053 secs 2023-03-09T22:05:27.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:05:26 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:05:27.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:05:26 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:05:27.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:05:26 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:05:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:05:27 smithi122 ceph-mon[93869]: pgmap v14184: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 8 op/s 2023-03-09T22:05:28.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:05:27 smithi181 ceph-mon[94189]: pgmap v14184: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 8 op/s 2023-03-09T22:05:28.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:05:27 smithi038 ceph-mon[92422]: pgmap v14184: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 24 KiB/s wr, 8 op/s 2023-03-09T22:05:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26100.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:05:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:05:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25638.710080 secs 2023-03-09T22:05:29.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:05:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25638.710080 secs 2023-03-09T22:05:29.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:05:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25638.710080 secs 2023-03-09T22:05:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:05:29 smithi122 ceph-mon[93869]: pgmap v14185: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 22 KiB/s wr, 7 op/s 2023-03-09T22:05:30.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:05:29 smithi181 ceph-mon[94189]: pgmap v14185: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 22 KiB/s wr, 7 op/s 2023-03-09T22:05:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:05:29 smithi038 ceph-mon[92422]: pgmap v14185: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 22 KiB/s wr, 7 op/s 2023-03-09T22:05:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:05:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25641.352112 secs 2023-03-09T22:05:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:05:31 smithi122 ceph-mon[93869]: pgmap v14186: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 128 KiB/s wr, 8 op/s 2023-03-09T22:05:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:05:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25641.352112 secs 2023-03-09T22:05:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:05:31 smithi181 ceph-mon[94189]: pgmap v14186: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 128 KiB/s wr, 8 op/s 2023-03-09T22:05:32.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:05:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25641.352112 secs 2023-03-09T22:05:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:05:31 smithi038 ceph-mon[92422]: pgmap v14186: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 128 KiB/s wr, 8 op/s 2023-03-09T22:05:33.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26105.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:05:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:05:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25643.710148 secs 2023-03-09T22:05:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:05:33 smithi122 ceph-mon[93869]: pgmap v14187: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 128 KiB/s wr, 10 op/s 2023-03-09T22:05:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:05:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25643.710148 secs 2023-03-09T22:05:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:05:33 smithi181 ceph-mon[94189]: pgmap v14187: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 128 KiB/s wr, 10 op/s 2023-03-09T22:05:34.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:05:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25643.710148 secs 2023-03-09T22:05:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:05:33 smithi038 ceph-mon[92422]: pgmap v14187: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 128 KiB/s wr, 10 op/s 2023-03-09T22:05:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:05:35 smithi122 ceph-mon[93869]: pgmap v14188: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 107 KiB/s wr, 7 op/s 2023-03-09T22:05:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:05:35 smithi181 ceph-mon[94189]: pgmap v14188: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 107 KiB/s wr, 7 op/s 2023-03-09T22:05:36.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:05:35 smithi038 ceph-mon[92422]: pgmap v14188: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 107 KiB/s wr, 7 op/s 2023-03-09T22:05:37.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:05:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25646.352175 secs 2023-03-09T22:05:37.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:05:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25646.352175 secs 2023-03-09T22:05:37.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:05:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25646.352175 secs 2023-03-09T22:05:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:05:37 smithi122 ceph-mon[93869]: pgmap v14189: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 107 KiB/s wr, 9 op/s 2023-03-09T22:05:38.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:05:37 smithi181 ceph-mon[94189]: pgmap v14189: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 107 KiB/s wr, 9 op/s 2023-03-09T22:05:38.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:05:37 smithi038 ceph-mon[92422]: pgmap v14189: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 107 KiB/s wr, 9 op/s 2023-03-09T22:05:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26110.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:05:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:05:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25648.710289 secs 2023-03-09T22:05:39.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:05:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25648.710289 secs 2023-03-09T22:05:39.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:05:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25648.710289 secs 2023-03-09T22:05:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:05:39 smithi122 ceph-mon[93869]: pgmap v14190: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 107 KiB/s wr, 7 op/s 2023-03-09T22:05:40.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:05:39 smithi181 ceph-mon[94189]: pgmap v14190: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 107 KiB/s wr, 7 op/s 2023-03-09T22:05:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:05:39 smithi038 ceph-mon[92422]: pgmap v14190: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 107 KiB/s wr, 7 op/s 2023-03-09T22:05:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:05:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25651.352234 secs 2023-03-09T22:05:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:05:41 smithi122 ceph-mon[93869]: pgmap v14191: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 312 KiB/s wr, 8 op/s 2023-03-09T22:05:42.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:05:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25651.352234 secs 2023-03-09T22:05:42.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:05:41 smithi181 ceph-mon[94189]: pgmap v14191: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 312 KiB/s wr, 8 op/s 2023-03-09T22:05:42.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:05:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25651.352234 secs 2023-03-09T22:05:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:05:41 smithi038 ceph-mon[92422]: pgmap v14191: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 312 KiB/s wr, 8 op/s 2023-03-09T22:05:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26115.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:05:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:05:43 smithi122 ceph-mon[93869]: pgmap v14192: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 207 KiB/s wr, 9 op/s 2023-03-09T22:05:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:05:43 smithi181 ceph-mon[94189]: pgmap v14192: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 207 KiB/s wr, 9 op/s 2023-03-09T22:05:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:05:43 smithi038 ceph-mon[92422]: pgmap v14192: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 207 KiB/s wr, 9 op/s 2023-03-09T22:05:45.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:05:44 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:05:45.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:05:44 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:05:45.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:05:44 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:05:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:05:45 smithi122 ceph-mon[93869]: pgmap v14193: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 236 KiB/s wr, 7 op/s 2023-03-09T22:05:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:05:45 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:05:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:05:45 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:05:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:05:45 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:05:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:05:45 smithi181 ceph-mon[94189]: pgmap v14193: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 236 KiB/s wr, 7 op/s 2023-03-09T22:05:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:05:45 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:05:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:05:45 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:05:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:05:45 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:05:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:05:45 smithi038 ceph-mon[92422]: pgmap v14193: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 236 KiB/s wr, 7 op/s 2023-03-09T22:05:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:05:45 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:05:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:05:45 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:05:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:05:45 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:05:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:05:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25656.352296 secs 2023-03-09T22:05:47.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:05:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25656.352296 secs 2023-03-09T22:05:47.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:05:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25656.352296 secs 2023-03-09T22:05:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:05:47 smithi122 ceph-mon[93869]: pgmap v14194: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 330 KiB/s wr, 10 op/s 2023-03-09T22:05:48.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:05:47 smithi181 ceph-mon[94189]: pgmap v14194: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 330 KiB/s wr, 10 op/s 2023-03-09T22:05:48.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:05:47 smithi038 ceph-mon[92422]: pgmap v14194: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 330 KiB/s wr, 10 op/s 2023-03-09T22:05:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26120.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:05:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:05:48 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 25658.710572 secs 2023-03-09T22:05:49.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:05:48 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 25658.710572 secs 2023-03-09T22:05:49.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:05:48 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 25658.710572 secs 2023-03-09T22:05:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:05:49 smithi122 ceph-mon[93869]: pgmap v14195: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 330 KiB/s wr, 8 op/s 2023-03-09T22:05:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:05:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:05:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:05:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:05:50.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:05:49 smithi181 ceph-mon[94189]: pgmap v14195: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 330 KiB/s wr, 8 op/s 2023-03-09T22:05:50.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:05:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:05:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:05:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:05:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:05:49 smithi038 ceph-mon[92422]: pgmap v14195: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 330 KiB/s wr, 8 op/s 2023-03-09T22:05:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:05:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:05:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:05:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:05:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:05:50 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:05:51.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:05:50 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:05:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:05:50 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:05:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:05:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25661.352357 secs 2023-03-09T22:05:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:05:51 smithi122 ceph-mon[93869]: pgmap v14196: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 650 KiB/s wr, 10 op/s 2023-03-09T22:05:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:05:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25661.352357 secs 2023-03-09T22:05:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:05:51 smithi181 ceph-mon[94189]: pgmap v14196: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 650 KiB/s wr, 10 op/s 2023-03-09T22:05:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:05:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25661.352357 secs 2023-03-09T22:05:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:05:51 smithi038 ceph-mon[92422]: pgmap v14196: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 650 KiB/s wr, 10 op/s 2023-03-09T22:05:53.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26125.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:05:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:05:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25663.710694 secs 2023-03-09T22:05:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:05:53 smithi122 ceph-mon[93869]: pgmap v14197: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 336 KiB/s rd, 465 KiB/s wr, 12 op/s 2023-03-09T22:05:54.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:05:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:05:54.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:05:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:05:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:05:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25663.710694 secs 2023-03-09T22:05:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:05:53 smithi181 ceph-mon[94189]: pgmap v14197: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 336 KiB/s rd, 465 KiB/s wr, 12 op/s 2023-03-09T22:05:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:05:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:05:54.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:05:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:05:54.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:05:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25663.710694 secs 2023-03-09T22:05:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:05:53 smithi038 ceph-mon[92422]: pgmap v14197: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 336 KiB/s rd, 465 KiB/s wr, 12 op/s 2023-03-09T22:05:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:05:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:05:54.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:05:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:05:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:05:55 smithi122 ceph-mon[93869]: pgmap v14198: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 335 KiB/s rd, 554 KiB/s wr, 10 op/s 2023-03-09T22:05:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:05:55 smithi181 ceph-mon[94189]: pgmap v14198: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 335 KiB/s rd, 554 KiB/s wr, 10 op/s 2023-03-09T22:05:56.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:05:55 smithi038 ceph-mon[92422]: pgmap v14198: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 335 KiB/s rd, 554 KiB/s wr, 10 op/s 2023-03-09T22:05:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:05:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25666.352440 secs 2023-03-09T22:05:57.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:05:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25666.352440 secs 2023-03-09T22:05:57.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:05:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25666.352440 secs 2023-03-09T22:05:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:05:57 smithi122 ceph-mon[93869]: pgmap v14199: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 625 KiB/s wr, 13 op/s 2023-03-09T22:05:58.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:05:57 smithi181 ceph-mon[94189]: pgmap v14199: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 625 KiB/s wr, 13 op/s 2023-03-09T22:05:58.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:05:57 smithi038 ceph-mon[92422]: pgmap v14199: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 625 KiB/s wr, 13 op/s 2023-03-09T22:05:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26130.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:05:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:05:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25668.710806 secs 2023-03-09T22:05:59.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:05:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25668.710806 secs 2023-03-09T22:05:59.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:05:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25668.710806 secs 2023-03-09T22:06:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:05:59 smithi122 ceph-mon[93869]: pgmap v14200: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 662 KiB/s rd, 648 KiB/s wr, 11 op/s 2023-03-09T22:06:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:05:59 smithi181 ceph-mon[94189]: pgmap v14200: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 662 KiB/s rd, 648 KiB/s wr, 11 op/s 2023-03-09T22:06:00.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:05:59 smithi038 ceph-mon[92422]: pgmap v14200: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 662 KiB/s rd, 648 KiB/s wr, 11 op/s 2023-03-09T22:06:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:06:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25671.352501 secs 2023-03-09T22:06:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:06:01 smithi122 ceph-mon[93869]: pgmap v14201: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 760 KiB/s rd, 746 KiB/s wr, 12 op/s 2023-03-09T22:06:02.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:06:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25671.352501 secs 2023-03-09T22:06:02.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:06:01 smithi181 ceph-mon[94189]: pgmap v14201: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 760 KiB/s rd, 746 KiB/s wr, 12 op/s 2023-03-09T22:06:02.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:06:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25671.352501 secs 2023-03-09T22:06:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:06:01 smithi038 ceph-mon[92422]: pgmap v14201: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 760 KiB/s rd, 746 KiB/s wr, 12 op/s 2023-03-09T22:06:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26135.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:06:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:06:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25673.710919 secs 2023-03-09T22:06:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:06:03 smithi122 ceph-mon[93869]: pgmap v14202: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 596 KiB/s rd, 447 KiB/s wr, 13 op/s 2023-03-09T22:06:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:06:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25673.710919 secs 2023-03-09T22:06:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:06:03 smithi181 ceph-mon[94189]: pgmap v14202: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 596 KiB/s rd, 447 KiB/s wr, 13 op/s 2023-03-09T22:06:04.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:06:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25673.710919 secs 2023-03-09T22:06:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:06:03 smithi038 ceph-mon[92422]: pgmap v14202: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 596 KiB/s rd, 447 KiB/s wr, 13 op/s 2023-03-09T22:06:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:06:05 smithi122 ceph-mon[93869]: pgmap v14203: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 708 KiB/s rd, 562 KiB/s wr, 12 op/s 2023-03-09T22:06:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:06:05 smithi181 ceph-mon[94189]: pgmap v14203: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 708 KiB/s rd, 562 KiB/s wr, 12 op/s 2023-03-09T22:06:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:06:05 smithi038 ceph-mon[92422]: pgmap v14203: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 708 KiB/s rd, 562 KiB/s wr, 12 op/s 2023-03-09T22:06:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:06:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25676.352561 secs 2023-03-09T22:06:07.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:06:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25676.352561 secs 2023-03-09T22:06:07.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:06:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25676.352561 secs 2023-03-09T22:06:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:06:07 smithi122 ceph-mon[93869]: pgmap v14204: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 949 KiB/s rd, 759 KiB/s wr, 17 op/s 2023-03-09T22:06:08.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:06:07 smithi181 ceph-mon[94189]: pgmap v14204: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 949 KiB/s rd, 759 KiB/s wr, 17 op/s 2023-03-09T22:06:08.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:06:07 smithi038 ceph-mon[92422]: pgmap v14204: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 949 KiB/s rd, 759 KiB/s wr, 17 op/s 2023-03-09T22:06:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26140.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:06:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:06:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25678.711040 secs 2023-03-09T22:06:09.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:06:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25678.711040 secs 2023-03-09T22:06:09.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:06:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25678.711040 secs 2023-03-09T22:06:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:06:09 smithi122 ceph-mon[93869]: pgmap v14205: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 680 KiB/s wr, 26 op/s 2023-03-09T22:06:10.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:06:09 smithi181 ceph-mon[94189]: pgmap v14205: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 680 KiB/s wr, 26 op/s 2023-03-09T22:06:10.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:06:09 smithi038 ceph-mon[92422]: pgmap v14205: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 680 KiB/s wr, 26 op/s 2023-03-09T22:06:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:06:10 smithi122 ceph-mon[93869]: osdmap e1101: 12 total, 12 up, 12 in 2023-03-09T22:06:11.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:06:10 smithi181 ceph-mon[94189]: osdmap e1101: 12 total, 12 up, 12 in 2023-03-09T22:06:11.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:06:10 smithi038 ceph-mon[92422]: osdmap e1101: 12 total, 12 up, 12 in 2023-03-09T22:06:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:06:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25681.352638 secs 2023-03-09T22:06:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:06:11 smithi122 ceph-mon[93869]: pgmap v14207: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 757 KiB/s wr, 30 op/s 2023-03-09T22:06:12.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:06:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25681.352638 secs 2023-03-09T22:06:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:06:11 smithi181 ceph-mon[94189]: pgmap v14207: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 757 KiB/s wr, 30 op/s 2023-03-09T22:06:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:06:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25681.352638 secs 2023-03-09T22:06:12.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:06:11 smithi038 ceph-mon[92422]: pgmap v14207: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 757 KiB/s wr, 30 op/s 2023-03-09T22:06:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26145.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:06:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:06:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25683.711164 secs 2023-03-09T22:06:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:06:13 smithi122 ceph-mon[93869]: pgmap v14208: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 692 KiB/s rd, 733 KiB/s wr, 29 op/s 2023-03-09T22:06:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:06:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25683.711164 secs 2023-03-09T22:06:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:06:13 smithi181 ceph-mon[94189]: pgmap v14208: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 692 KiB/s rd, 733 KiB/s wr, 29 op/s 2023-03-09T22:06:14.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:06:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25683.711164 secs 2023-03-09T22:06:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:06:13 smithi038 ceph-mon[92422]: pgmap v14208: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 692 KiB/s rd, 733 KiB/s wr, 29 op/s 2023-03-09T22:06:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:06:15 smithi122 ceph-mon[93869]: pgmap v14209: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 529 KiB/s rd, 571 KiB/s wr, 27 op/s 2023-03-09T22:06:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:06:15 smithi122 ceph-mon[93869]: osdmap e1102: 12 total, 12 up, 12 in 2023-03-09T22:06:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:06:15 smithi181 ceph-mon[94189]: pgmap v14209: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 529 KiB/s rd, 571 KiB/s wr, 27 op/s 2023-03-09T22:06:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:06:15 smithi181 ceph-mon[94189]: osdmap e1102: 12 total, 12 up, 12 in 2023-03-09T22:06:16.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:06:15 smithi038 ceph-mon[92422]: pgmap v14209: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 529 KiB/s rd, 571 KiB/s wr, 27 op/s 2023-03-09T22:06:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:06:15 smithi038 ceph-mon[92422]: osdmap e1102: 12 total, 12 up, 12 in 2023-03-09T22:06:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:06:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25686.352699 secs 2023-03-09T22:06:17.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:06:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25686.352699 secs 2023-03-09T22:06:17.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:06:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25686.352699 secs 2023-03-09T22:06:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:06:17 smithi122 ceph-mon[93869]: pgmap v14211: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 283 KiB/s wr, 12 op/s 2023-03-09T22:06:18.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:06:17 smithi181 ceph-mon[94189]: pgmap v14211: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 283 KiB/s wr, 12 op/s 2023-03-09T22:06:18.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:06:17 smithi038 ceph-mon[92422]: pgmap v14211: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 283 KiB/s wr, 12 op/s 2023-03-09T22:06:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26150.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:06:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:06:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25688.711279 secs 2023-03-09T22:06:19.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:06:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25688.711279 secs 2023-03-09T22:06:19.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:06:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25688.711279 secs 2023-03-09T22:06:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:06:19 smithi122 ceph-mon[93869]: pgmap v14212: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 32 KiB/s wr, 10 op/s 2023-03-09T22:06:20.247 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:06:19 smithi181 ceph-mon[94189]: pgmap v14212: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 32 KiB/s wr, 10 op/s 2023-03-09T22:06:20.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:06:19 smithi038 ceph-mon[92422]: pgmap v14212: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 32 KiB/s wr, 10 op/s 2023-03-09T22:06:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:06:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25691.352759 secs 2023-03-09T22:06:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:06:21 smithi122 ceph-mon[93869]: pgmap v14213: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 152 KiB/s wr, 9 op/s 2023-03-09T22:06:22.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:06:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25691.352759 secs 2023-03-09T22:06:22.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:06:21 smithi181 ceph-mon[94189]: pgmap v14213: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 152 KiB/s wr, 9 op/s 2023-03-09T22:06:22.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:06:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25691.352759 secs 2023-03-09T22:06:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:06:21 smithi038 ceph-mon[92422]: pgmap v14213: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 152 KiB/s wr, 9 op/s 2023-03-09T22:06:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26155.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:06:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:06:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25693.711431 secs 2023-03-09T22:06:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:06:23 smithi122 ceph-mon[93869]: pgmap v14214: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 150 KiB/s wr, 8 op/s 2023-03-09T22:06:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:06:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25693.711431 secs 2023-03-09T22:06:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:06:23 smithi181 ceph-mon[94189]: pgmap v14214: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 150 KiB/s wr, 8 op/s 2023-03-09T22:06:24.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:06:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25693.711431 secs 2023-03-09T22:06:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:06:23 smithi038 ceph-mon[92422]: pgmap v14214: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 150 KiB/s wr, 8 op/s 2023-03-09T22:06:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:06:25 smithi122 ceph-mon[93869]: pgmap v14215: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 150 KiB/s wr, 8 op/s 2023-03-09T22:06:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:06:25 smithi181 ceph-mon[94189]: pgmap v14215: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 150 KiB/s wr, 8 op/s 2023-03-09T22:06:26.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:06:25 smithi038 ceph-mon[92422]: pgmap v14215: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 150 KiB/s wr, 8 op/s 2023-03-09T22:06:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:06:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25696.352842 secs 2023-03-09T22:06:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:06:26 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:06:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:06:26 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:06:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:06:26 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:06:27.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:06:26 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:06:27.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:06:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25696.352842 secs 2023-03-09T22:06:27.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:06:26 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:06:27.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:06:26 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:06:27.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:06:26 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:06:27.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:06:26 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:06:27.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:06:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25696.352842 secs 2023-03-09T22:06:27.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:06:26 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:06:27.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:06:26 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:06:27.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:06:26 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:06:27.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:06:26 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:06:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:06:27 smithi122 ceph-mon[93869]: pgmap v14216: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 139 KiB/s wr, 7 op/s 2023-03-09T22:06:28.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:06:27 smithi181 ceph-mon[94189]: pgmap v14216: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 139 KiB/s wr, 7 op/s 2023-03-09T22:06:28.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:06:27 smithi038 ceph-mon[92422]: pgmap v14216: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 139 KiB/s wr, 7 op/s 2023-03-09T22:06:28.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26160.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:06:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:06:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25698.711566 secs 2023-03-09T22:06:29.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:06:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25698.711566 secs 2023-03-09T22:06:29.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:06:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25698.711566 secs 2023-03-09T22:06:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:06:29 smithi122 ceph-mon[93869]: pgmap v14217: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 125 KiB/s wr, 7 op/s 2023-03-09T22:06:30.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:06:29 smithi181 ceph-mon[94189]: pgmap v14217: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 125 KiB/s wr, 7 op/s 2023-03-09T22:06:30.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:06:29 smithi038 ceph-mon[92422]: pgmap v14217: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 125 KiB/s wr, 7 op/s 2023-03-09T22:06:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:06:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25701.352904 secs 2023-03-09T22:06:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:06:31 smithi122 ceph-mon[93869]: pgmap v14218: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 198 KiB/s wr, 7 op/s 2023-03-09T22:06:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:06:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25701.352904 secs 2023-03-09T22:06:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:06:31 smithi181 ceph-mon[94189]: pgmap v14218: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 198 KiB/s wr, 7 op/s 2023-03-09T22:06:32.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:06:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25701.352904 secs 2023-03-09T22:06:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:06:31 smithi038 ceph-mon[92422]: pgmap v14218: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 198 KiB/s wr, 7 op/s 2023-03-09T22:06:33.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26165.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:06:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:06:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25703.711628 secs 2023-03-09T22:06:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:06:33 smithi122 ceph-mon[93869]: pgmap v14219: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 185 KiB/s wr, 9 op/s 2023-03-09T22:06:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:06:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25703.711628 secs 2023-03-09T22:06:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:06:33 smithi181 ceph-mon[94189]: pgmap v14219: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 185 KiB/s wr, 9 op/s 2023-03-09T22:06:34.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:06:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25703.711628 secs 2023-03-09T22:06:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:06:33 smithi038 ceph-mon[92422]: pgmap v14219: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 185 KiB/s wr, 9 op/s 2023-03-09T22:06:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:06:35 smithi122 ceph-mon[93869]: pgmap v14220: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 185 KiB/s wr, 7 op/s 2023-03-09T22:06:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:06:35 smithi181 ceph-mon[94189]: pgmap v14220: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 185 KiB/s wr, 7 op/s 2023-03-09T22:06:36.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:06:35 smithi038 ceph-mon[92422]: pgmap v14220: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 185 KiB/s wr, 7 op/s 2023-03-09T22:06:37.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:06:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25706.352976 secs 2023-03-09T22:06:37.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:06:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25706.352976 secs 2023-03-09T22:06:37.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:06:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25706.352976 secs 2023-03-09T22:06:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:06:37 smithi122 ceph-mon[93869]: pgmap v14221: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 185 KiB/s wr, 10 op/s 2023-03-09T22:06:38.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:06:37 smithi181 ceph-mon[94189]: pgmap v14221: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 185 KiB/s wr, 10 op/s 2023-03-09T22:06:38.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:06:37 smithi038 ceph-mon[92422]: pgmap v14221: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 185 KiB/s wr, 10 op/s 2023-03-09T22:06:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26170.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:06:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:06:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25708.711751 secs 2023-03-09T22:06:39.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:06:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25708.711751 secs 2023-03-09T22:06:39.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:06:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25708.711751 secs 2023-03-09T22:06:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:06:39 smithi122 ceph-mon[93869]: pgmap v14222: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 164 KiB/s wr, 7 op/s 2023-03-09T22:06:40.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:06:39 smithi181 ceph-mon[94189]: pgmap v14222: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 164 KiB/s wr, 7 op/s 2023-03-09T22:06:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:06:39 smithi038 ceph-mon[92422]: pgmap v14222: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 164 KiB/s wr, 7 op/s 2023-03-09T22:06:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:06:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25711.353038 secs 2023-03-09T22:06:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:06:41 smithi122 ceph-mon[93869]: pgmap v14223: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 164 KiB/s wr, 7 op/s 2023-03-09T22:06:42.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:06:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25711.353038 secs 2023-03-09T22:06:42.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:06:41 smithi181 ceph-mon[94189]: pgmap v14223: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 164 KiB/s wr, 7 op/s 2023-03-09T22:06:42.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:06:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25711.353038 secs 2023-03-09T22:06:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:06:41 smithi038 ceph-mon[92422]: pgmap v14223: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 164 KiB/s wr, 7 op/s 2023-03-09T22:06:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26175.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:06:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:06:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25713.711892 secs 2023-03-09T22:06:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:06:43 smithi122 ceph-mon[93869]: pgmap v14224: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 227 KiB/s wr, 9 op/s 2023-03-09T22:06:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:06:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25713.711892 secs 2023-03-09T22:06:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:06:43 smithi181 ceph-mon[94189]: pgmap v14224: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 227 KiB/s wr, 9 op/s 2023-03-09T22:06:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:06:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25713.711892 secs 2023-03-09T22:06:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:06:43 smithi038 ceph-mon[92422]: pgmap v14224: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 227 KiB/s wr, 9 op/s 2023-03-09T22:06:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:06:45 smithi122 ceph-mon[93869]: pgmap v14225: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 136 KiB/s wr, 7 op/s 2023-03-09T22:06:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:06:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25716.353097 secs 2023-03-09T22:06:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:06:45 smithi181 ceph-mon[94189]: pgmap v14225: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 136 KiB/s wr, 7 op/s 2023-03-09T22:06:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:06:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25716.353097 secs 2023-03-09T22:06:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:06:45 smithi038 ceph-mon[92422]: pgmap v14225: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 136 KiB/s wr, 7 op/s 2023-03-09T22:06:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:06:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25716.353097 secs 2023-03-09T22:06:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:06:47 smithi122 ceph-mon[93869]: pgmap v14226: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 272 KiB/s wr, 9 op/s 2023-03-09T22:06:48.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:06:47 smithi181 ceph-mon[94189]: pgmap v14226: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 272 KiB/s wr, 9 op/s 2023-03-09T22:06:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:06:47 smithi038 ceph-mon[92422]: pgmap v14226: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 272 KiB/s wr, 9 op/s 2023-03-09T22:06:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26180.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:06:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:06:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25718.712039 secs 2023-03-09T22:06:49.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:06:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25718.712039 secs 2023-03-09T22:06:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:06:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25718.712039 secs 2023-03-09T22:06:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:06:49 smithi122 ceph-mon[93869]: pgmap v14227: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 272 KiB/s wr, 7 op/s 2023-03-09T22:06:50.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:06:49 smithi181 ceph-mon[94189]: pgmap v14227: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 272 KiB/s wr, 7 op/s 2023-03-09T22:06:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:06:49 smithi038 ceph-mon[92422]: pgmap v14227: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 272 KiB/s wr, 7 op/s 2023-03-09T22:06:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:06:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25721.353158 secs 2023-03-09T22:06:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:06:51 smithi122 ceph-mon[93869]: pgmap v14228: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 361 KiB/s wr, 8 op/s 2023-03-09T22:06:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:06:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25721.353158 secs 2023-03-09T22:06:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:06:51 smithi181 ceph-mon[94189]: pgmap v14228: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 361 KiB/s wr, 8 op/s 2023-03-09T22:06:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:06:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25721.353158 secs 2023-03-09T22:06:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:06:51 smithi038 ceph-mon[92422]: pgmap v14228: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 361 KiB/s wr, 8 op/s 2023-03-09T22:06:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26185.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:06:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:06:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25723.712173 secs 2023-03-09T22:06:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:06:53 smithi122 ceph-mon[93869]: pgmap v14229: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 398 KiB/s rd, 382 KiB/s wr, 11 op/s 2023-03-09T22:06:54.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:06:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25723.712173 secs 2023-03-09T22:06:54.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:06:53 smithi181 ceph-mon[94189]: pgmap v14229: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 398 KiB/s rd, 382 KiB/s wr, 11 op/s 2023-03-09T22:06:54.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:06:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25723.712173 secs 2023-03-09T22:06:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:06:53 smithi038 ceph-mon[92422]: pgmap v14229: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 398 KiB/s rd, 382 KiB/s wr, 11 op/s 2023-03-09T22:06:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:06:55 smithi122 ceph-mon[93869]: pgmap v14230: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 514 KiB/s rd, 499 KiB/s wr, 9 op/s 2023-03-09T22:06:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:06:55 smithi181 ceph-mon[94189]: pgmap v14230: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 514 KiB/s rd, 499 KiB/s wr, 9 op/s 2023-03-09T22:06:56.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:06:55 smithi038 ceph-mon[92422]: pgmap v14230: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 514 KiB/s rd, 499 KiB/s wr, 9 op/s 2023-03-09T22:06:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:06:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25726.353220 secs 2023-03-09T22:06:57.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:06:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25726.353220 secs 2023-03-09T22:06:57.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:06:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25726.353220 secs 2023-03-09T22:06:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:06:57 smithi122 ceph-mon[93869]: pgmap v14231: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 717 KiB/s rd, 701 KiB/s wr, 13 op/s 2023-03-09T22:06:58.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:06:57 smithi181 ceph-mon[94189]: pgmap v14231: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 717 KiB/s rd, 701 KiB/s wr, 13 op/s 2023-03-09T22:06:58.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:06:57 smithi038 ceph-mon[92422]: pgmap v14231: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 717 KiB/s rd, 701 KiB/s wr, 13 op/s 2023-03-09T22:06:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26190.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:06:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:06:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25728.712295 secs 2023-03-09T22:06:59.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:06:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25728.712295 secs 2023-03-09T22:06:59.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:06:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25728.712295 secs 2023-03-09T22:07:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:06:59 smithi122 ceph-mon[93869]: pgmap v14232: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 580 KiB/s rd, 565 KiB/s wr, 10 op/s 2023-03-09T22:07:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:06:59 smithi181 ceph-mon[94189]: pgmap v14232: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 580 KiB/s rd, 565 KiB/s wr, 10 op/s 2023-03-09T22:07:00.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:06:59 smithi038 ceph-mon[92422]: pgmap v14232: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 580 KiB/s rd, 565 KiB/s wr, 10 op/s 2023-03-09T22:07:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:07:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25731.353279 secs 2023-03-09T22:07:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:07:01 smithi122 ceph-mon[93869]: pgmap v14233: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 580 KiB/s rd, 565 KiB/s wr, 11 op/s 2023-03-09T22:07:02.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:07:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25731.353279 secs 2023-03-09T22:07:02.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:07:01 smithi181 ceph-mon[94189]: pgmap v14233: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 580 KiB/s rd, 565 KiB/s wr, 11 op/s 2023-03-09T22:07:02.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:07:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25731.353279 secs 2023-03-09T22:07:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:07:01 smithi038 ceph-mon[92422]: pgmap v14233: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 580 KiB/s rd, 565 KiB/s wr, 11 op/s 2023-03-09T22:07:03.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26195.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:07:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:07:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25733.712424 secs 2023-03-09T22:07:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:07:03 smithi122 ceph-mon[93869]: pgmap v14234: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 526 KiB/s rd, 521 KiB/s wr, 13 op/s 2023-03-09T22:07:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:07:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25733.712424 secs 2023-03-09T22:07:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:07:03 smithi181 ceph-mon[94189]: pgmap v14234: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 526 KiB/s rd, 521 KiB/s wr, 13 op/s 2023-03-09T22:07:04.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:07:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25733.712424 secs 2023-03-09T22:07:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:07:03 smithi038 ceph-mon[92422]: pgmap v14234: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 526 KiB/s rd, 521 KiB/s wr, 13 op/s 2023-03-09T22:07:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:07:05 smithi122 ceph-mon[93869]: pgmap v14235: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 509 KiB/s rd, 511 KiB/s wr, 21 op/s 2023-03-09T22:07:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:07:05 smithi181 ceph-mon[94189]: pgmap v14235: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 509 KiB/s rd, 511 KiB/s wr, 21 op/s 2023-03-09T22:07:06.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:07:05 smithi038 ceph-mon[92422]: pgmap v14235: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 509 KiB/s rd, 511 KiB/s wr, 21 op/s 2023-03-09T22:07:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:07:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25736.353340 secs 2023-03-09T22:07:07.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:07:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25736.353340 secs 2023-03-09T22:07:07.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:07:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25736.353340 secs 2023-03-09T22:07:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:07:07 smithi122 ceph-mon[93869]: pgmap v14236: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 682 KiB/s rd, 694 KiB/s wr, 26 op/s 2023-03-09T22:07:08.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:07:07 smithi181 ceph-mon[94189]: pgmap v14236: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 682 KiB/s rd, 694 KiB/s wr, 26 op/s 2023-03-09T22:07:08.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:07:07 smithi038 ceph-mon[92422]: pgmap v14236: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 682 KiB/s rd, 694 KiB/s wr, 26 op/s 2023-03-09T22:07:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26200.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:07:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:07:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25738.712498 secs 2023-03-09T22:07:09.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:07:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25738.712498 secs 2023-03-09T22:07:09.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:07:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25738.712498 secs 2023-03-09T22:07:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:07:09 smithi122 ceph-mon[93869]: pgmap v14237: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 492 KiB/s wr, 22 op/s 2023-03-09T22:07:10.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:07:09 smithi181 ceph-mon[94189]: pgmap v14237: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 492 KiB/s wr, 22 op/s 2023-03-09T22:07:10.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:07:09 smithi038 ceph-mon[92422]: pgmap v14237: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 492 KiB/s wr, 22 op/s 2023-03-09T22:07:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:07:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25741.353401 secs 2023-03-09T22:07:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:07:11 smithi122 ceph-mon[93869]: pgmap v14238: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 491 KiB/s wr, 22 op/s 2023-03-09T22:07:12.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:07:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25741.353401 secs 2023-03-09T22:07:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:07:11 smithi181 ceph-mon[94189]: pgmap v14238: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 491 KiB/s wr, 22 op/s 2023-03-09T22:07:12.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:07:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25741.353401 secs 2023-03-09T22:07:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:07:11 smithi038 ceph-mon[92422]: pgmap v14238: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 491 KiB/s wr, 22 op/s 2023-03-09T22:07:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26205.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:07:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:07:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25743.712616 secs 2023-03-09T22:07:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:07:13 smithi122 ceph-mon[93869]: pgmap v14239: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 596 KiB/s rd, 608 KiB/s wr, 25 op/s 2023-03-09T22:07:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:07:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25743.712616 secs 2023-03-09T22:07:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:07:13 smithi181 ceph-mon[94189]: pgmap v14239: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 596 KiB/s rd, 608 KiB/s wr, 25 op/s 2023-03-09T22:07:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:07:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25743.712616 secs 2023-03-09T22:07:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:07:13 smithi038 ceph-mon[92422]: pgmap v14239: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 596 KiB/s rd, 608 KiB/s wr, 25 op/s 2023-03-09T22:07:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:07:15 smithi122 ceph-mon[93869]: pgmap v14240: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 549 KiB/s rd, 563 KiB/s wr, 22 op/s 2023-03-09T22:07:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:07:15 smithi181 ceph-mon[94189]: pgmap v14240: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 549 KiB/s rd, 563 KiB/s wr, 22 op/s 2023-03-09T22:07:16.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:07:15 smithi038 ceph-mon[92422]: pgmap v14240: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 549 KiB/s rd, 563 KiB/s wr, 22 op/s 2023-03-09T22:07:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:07:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25746.353461 secs 2023-03-09T22:07:17.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:07:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25746.353461 secs 2023-03-09T22:07:17.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:07:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25746.353461 secs 2023-03-09T22:07:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:07:17 smithi122 ceph-mon[93869]: pgmap v14241: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 901 KiB/s rd, 921 KiB/s wr, 16 op/s 2023-03-09T22:07:18.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:07:17 smithi181 ceph-mon[94189]: pgmap v14241: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 901 KiB/s rd, 921 KiB/s wr, 16 op/s 2023-03-09T22:07:18.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:07:17 smithi038 ceph-mon[92422]: pgmap v14241: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 901 KiB/s rd, 921 KiB/s wr, 16 op/s 2023-03-09T22:07:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26210.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:07:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:07:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25748.712745 secs 2023-03-09T22:07:19.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:07:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25748.712745 secs 2023-03-09T22:07:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:07:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25748.712745 secs 2023-03-09T22:07:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:07:19 smithi122 ceph-mon[93869]: pgmap v14242: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 496 KiB/s wr, 12 op/s 2023-03-09T22:07:20.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:07:19 smithi181 ceph-mon[94189]: pgmap v14242: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 496 KiB/s wr, 12 op/s 2023-03-09T22:07:20.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:07:19 smithi038 ceph-mon[92422]: pgmap v14242: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 496 KiB/s wr, 12 op/s 2023-03-09T22:07:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:07:20 smithi122 ceph-mon[93869]: osdmap e1103: 12 total, 12 up, 12 in 2023-03-09T22:07:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:07:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25751.353523 secs 2023-03-09T22:07:21.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:07:20 smithi181 ceph-mon[94189]: osdmap e1103: 12 total, 12 up, 12 in 2023-03-09T22:07:21.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:07:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25751.353523 secs 2023-03-09T22:07:21.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:07:20 smithi038 ceph-mon[92422]: osdmap e1103: 12 total, 12 up, 12 in 2023-03-09T22:07:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:07:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25751.353523 secs 2023-03-09T22:07:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:07:21 smithi122 ceph-mon[93869]: pgmap v14244: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 596 KiB/s wr, 13 op/s 2023-03-09T22:07:22.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:07:21 smithi181 ceph-mon[94189]: pgmap v14244: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 596 KiB/s wr, 13 op/s 2023-03-09T22:07:22.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:07:21 smithi038 ceph-mon[92422]: pgmap v14244: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 596 KiB/s wr, 13 op/s 2023-03-09T22:07:23.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26215.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:07:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:07:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25753.712884 secs 2023-03-09T22:07:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:07:23 smithi122 ceph-mon[93869]: pgmap v14245: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 594 KiB/s rd, 618 KiB/s wr, 14 op/s 2023-03-09T22:07:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:07:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25753.712884 secs 2023-03-09T22:07:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:07:23 smithi181 ceph-mon[94189]: pgmap v14245: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 594 KiB/s rd, 618 KiB/s wr, 14 op/s 2023-03-09T22:07:24.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:07:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25753.712884 secs 2023-03-09T22:07:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:07:23 smithi038 ceph-mon[92422]: pgmap v14245: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 594 KiB/s rd, 618 KiB/s wr, 14 op/s 2023-03-09T22:07:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:07:25 smithi122 ceph-mon[93869]: pgmap v14246: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 594 KiB/s rd, 618 KiB/s wr, 14 op/s 2023-03-09T22:07:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:07:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25756.353584 secs 2023-03-09T22:07:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:07:25 smithi122 ceph-mon[93869]: osdmap e1104: 12 total, 12 up, 12 in 2023-03-09T22:07:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:07:25 smithi181 ceph-mon[94189]: pgmap v14246: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 594 KiB/s rd, 618 KiB/s wr, 14 op/s 2023-03-09T22:07:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:07:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25756.353584 secs 2023-03-09T22:07:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:07:25 smithi181 ceph-mon[94189]: osdmap e1104: 12 total, 12 up, 12 in 2023-03-09T22:07:26.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:07:25 smithi038 ceph-mon[92422]: pgmap v14246: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 594 KiB/s rd, 618 KiB/s wr, 14 op/s 2023-03-09T22:07:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:07:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25756.353584 secs 2023-03-09T22:07:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:07:25 smithi038 ceph-mon[92422]: osdmap e1104: 12 total, 12 up, 12 in 2023-03-09T22:07:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:07:26 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:07:27.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:07:26 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:07:27.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:07:26 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:07:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:07:27 smithi122 ceph-mon[93869]: pgmap v14248: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 612 KiB/s rd, 611 KiB/s wr, 13 op/s 2023-03-09T22:07:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:07:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:07:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:07:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:07:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:07:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:07:28.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:07:27 smithi181 ceph-mon[94189]: pgmap v14248: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 612 KiB/s rd, 611 KiB/s wr, 13 op/s 2023-03-09T22:07:28.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:07:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:07:28.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:07:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:07:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:07:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:07:28.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:07:27 smithi038 ceph-mon[92422]: pgmap v14248: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 612 KiB/s rd, 611 KiB/s wr, 13 op/s 2023-03-09T22:07:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:07:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:07:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:07:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:07:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:07:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:07:28.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26220.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:07:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:07:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25758.712973 secs 2023-03-09T22:07:29.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:07:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25758.712973 secs 2023-03-09T22:07:29.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:07:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25758.712973 secs 2023-03-09T22:07:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:07:29 smithi122 ceph-mon[93869]: pgmap v14249: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 562 KiB/s wr, 13 op/s 2023-03-09T22:07:30.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:07:29 smithi181 ceph-mon[94189]: pgmap v14249: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 562 KiB/s wr, 13 op/s 2023-03-09T22:07:30.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:07:29 smithi038 ceph-mon[92422]: pgmap v14249: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 562 KiB/s wr, 13 op/s 2023-03-09T22:07:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:07:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25761.353662 secs 2023-03-09T22:07:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:07:31 smithi122 ceph-mon[93869]: pgmap v14250: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 489 KiB/s wr, 11 op/s 2023-03-09T22:07:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:07:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25761.353662 secs 2023-03-09T22:07:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:07:31 smithi181 ceph-mon[94189]: pgmap v14250: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 489 KiB/s wr, 11 op/s 2023-03-09T22:07:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:07:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25761.353662 secs 2023-03-09T22:07:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:07:31 smithi038 ceph-mon[92422]: pgmap v14250: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 489 KiB/s wr, 11 op/s 2023-03-09T22:07:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26225.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:07:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:07:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25763.713091 secs 2023-03-09T22:07:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:07:33 smithi122 ceph-mon[93869]: pgmap v14251: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 417 KiB/s wr, 11 op/s 2023-03-09T22:07:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:07:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25763.713091 secs 2023-03-09T22:07:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:07:33 smithi181 ceph-mon[94189]: pgmap v14251: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 417 KiB/s wr, 11 op/s 2023-03-09T22:07:34.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:07:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25763.713091 secs 2023-03-09T22:07:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:07:33 smithi038 ceph-mon[92422]: pgmap v14251: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 417 KiB/s wr, 11 op/s 2023-03-09T22:07:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:07:35 smithi122 ceph-mon[93869]: pgmap v14252: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 417 KiB/s wr, 10 op/s 2023-03-09T22:07:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:07:35 smithi181 ceph-mon[94189]: pgmap v14252: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 417 KiB/s wr, 10 op/s 2023-03-09T22:07:36.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:07:35 smithi038 ceph-mon[92422]: pgmap v14252: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 417 KiB/s wr, 10 op/s 2023-03-09T22:07:37.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:07:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25766.353721 secs 2023-03-09T22:07:37.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:07:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25766.353721 secs 2023-03-09T22:07:37.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:07:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25766.353721 secs 2023-03-09T22:07:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:07:37 smithi122 ceph-mon[93869]: pgmap v14253: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 316 KiB/s wr, 8 op/s 2023-03-09T22:07:38.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:07:37 smithi181 ceph-mon[94189]: pgmap v14253: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 316 KiB/s wr, 8 op/s 2023-03-09T22:07:38.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:07:37 smithi038 ceph-mon[92422]: pgmap v14253: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 316 KiB/s wr, 8 op/s 2023-03-09T22:07:38.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26230.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:07:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:07:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25768.713166 secs 2023-03-09T22:07:39.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:07:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25768.713166 secs 2023-03-09T22:07:39.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:07:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25768.713166 secs 2023-03-09T22:07:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:07:39 smithi122 ceph-mon[93869]: pgmap v14254: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 283 KiB/s wr, 7 op/s 2023-03-09T22:07:40.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:07:39 smithi181 ceph-mon[94189]: pgmap v14254: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 283 KiB/s wr, 7 op/s 2023-03-09T22:07:40.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:07:39 smithi038 ceph-mon[92422]: pgmap v14254: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 283 KiB/s wr, 7 op/s 2023-03-09T22:07:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:07:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25771.353798 secs 2023-03-09T22:07:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:07:41 smithi122 ceph-mon[93869]: pgmap v14255: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 283 KiB/s wr, 7 op/s 2023-03-09T22:07:42.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:07:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25771.353798 secs 2023-03-09T22:07:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:07:41 smithi181 ceph-mon[94189]: pgmap v14255: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 283 KiB/s wr, 7 op/s 2023-03-09T22:07:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:07:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25771.353798 secs 2023-03-09T22:07:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:07:41 smithi038 ceph-mon[92422]: pgmap v14255: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 283 KiB/s wr, 7 op/s 2023-03-09T22:07:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26235.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:07:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:07:43 smithi122 ceph-mon[93869]: pgmap v14256: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 401 KiB/s wr, 11 op/s 2023-03-09T22:07:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:07:43 smithi181 ceph-mon[94189]: pgmap v14256: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 401 KiB/s wr, 11 op/s 2023-03-09T22:07:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:07:43 smithi038 ceph-mon[92422]: pgmap v14256: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 401 KiB/s wr, 11 op/s 2023-03-09T22:07:45.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:07:44 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:07:45.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:07:44 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:07:45.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:07:44 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:07:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:07:45 smithi122 ceph-mon[93869]: pgmap v14257: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 328 KiB/s rd, 326 KiB/s wr, 7 op/s 2023-03-09T22:07:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:07:45 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:07:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:07:45 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:07:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:07:45 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:07:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:07:45 smithi181 ceph-mon[94189]: pgmap v14257: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 328 KiB/s rd, 326 KiB/s wr, 7 op/s 2023-03-09T22:07:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:07:45 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:07:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:07:45 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:07:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:07:45 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:07:46.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:07:45 smithi038 ceph-mon[92422]: pgmap v14257: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 328 KiB/s rd, 326 KiB/s wr, 7 op/s 2023-03-09T22:07:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:07:45 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:07:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:07:45 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:07:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:07:45 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:07:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:07:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25776.353857 secs 2023-03-09T22:07:47.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:07:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25776.353857 secs 2023-03-09T22:07:47.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:07:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25776.353857 secs 2023-03-09T22:07:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:07:47 smithi122 ceph-mon[93869]: pgmap v14258: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 464 KiB/s rd, 461 KiB/s wr, 11 op/s 2023-03-09T22:07:48.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:07:47 smithi181 ceph-mon[94189]: pgmap v14258: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 464 KiB/s rd, 461 KiB/s wr, 11 op/s 2023-03-09T22:07:48.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:07:47 smithi038 ceph-mon[92422]: pgmap v14258: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 464 KiB/s rd, 461 KiB/s wr, 11 op/s 2023-03-09T22:07:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26240.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:07:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:07:49 smithi122 ceph-mon[93869]: pgmap v14259: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 253 KiB/s wr, 8 op/s 2023-03-09T22:07:50.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:07:49 smithi181 ceph-mon[94189]: pgmap v14259: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 253 KiB/s wr, 8 op/s 2023-03-09T22:07:50.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:07:49 smithi038 ceph-mon[92422]: pgmap v14259: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 253 KiB/s wr, 8 op/s 2023-03-09T22:07:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:07:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25781.353918 secs 2023-03-09T22:07:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:07:51 smithi122 ceph-mon[93869]: pgmap v14260: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 257 KiB/s wr, 9 op/s 2023-03-09T22:07:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:07:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25781.353918 secs 2023-03-09T22:07:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:07:51 smithi181 ceph-mon[94189]: pgmap v14260: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 257 KiB/s wr, 9 op/s 2023-03-09T22:07:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:07:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25781.353918 secs 2023-03-09T22:07:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:07:51 smithi038 ceph-mon[92422]: pgmap v14260: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 257 KiB/s wr, 9 op/s 2023-03-09T22:07:53.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26245.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:07:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:07:53 smithi122 ceph-mon[93869]: pgmap v14261: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 538 KiB/s rd, 448 KiB/s wr, 13 op/s 2023-03-09T22:07:54.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:07:53 smithi181 ceph-mon[94189]: pgmap v14261: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 538 KiB/s rd, 448 KiB/s wr, 13 op/s 2023-03-09T22:07:54.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:07:53 smithi038 ceph-mon[92422]: pgmap v14261: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 538 KiB/s rd, 448 KiB/s wr, 13 op/s 2023-03-09T22:07:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:07:55 smithi122 ceph-mon[93869]: pgmap v14262: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 386 KiB/s rd, 371 KiB/s wr, 10 op/s 2023-03-09T22:07:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:07:55 smithi181 ceph-mon[94189]: pgmap v14262: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 386 KiB/s rd, 371 KiB/s wr, 10 op/s 2023-03-09T22:07:56.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:07:55 smithi038 ceph-mon[92422]: pgmap v14262: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 386 KiB/s rd, 371 KiB/s wr, 10 op/s 2023-03-09T22:07:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:07:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25786.354011 secs 2023-03-09T22:07:57.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:07:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25786.354011 secs 2023-03-09T22:07:57.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:07:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25786.354011 secs 2023-03-09T22:07:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:07:57 smithi122 ceph-mon[93869]: pgmap v14263: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 671 KiB/s rd, 611 KiB/s wr, 15 op/s 2023-03-09T22:07:58.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:07:57 smithi181 ceph-mon[94189]: pgmap v14263: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 671 KiB/s rd, 611 KiB/s wr, 15 op/s 2023-03-09T22:07:58.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:07:57 smithi038 ceph-mon[92422]: pgmap v14263: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 671 KiB/s rd, 611 KiB/s wr, 15 op/s 2023-03-09T22:07:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26250.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:08:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:07:59 smithi122 ceph-mon[93869]: pgmap v14264: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 535 KiB/s rd, 476 KiB/s wr, 11 op/s 2023-03-09T22:08:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:07:59 smithi181 ceph-mon[94189]: pgmap v14264: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 535 KiB/s rd, 476 KiB/s wr, 11 op/s 2023-03-09T22:08:00.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:07:59 smithi038 ceph-mon[92422]: pgmap v14264: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 535 KiB/s rd, 476 KiB/s wr, 11 op/s 2023-03-09T22:08:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25791.354085 secs 2023-03-09T22:08:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:01 smithi122 ceph-mon[93869]: pgmap v14265: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 584 KiB/s rd, 525 KiB/s wr, 12 op/s 2023-03-09T22:08:02.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25791.354085 secs 2023-03-09T22:08:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:01 smithi181 ceph-mon[94189]: pgmap v14265: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 584 KiB/s rd, 525 KiB/s wr, 12 op/s 2023-03-09T22:08:02.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25791.354085 secs 2023-03-09T22:08:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:01 smithi038 ceph-mon[92422]: pgmap v14265: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 584 KiB/s rd, 525 KiB/s wr, 12 op/s 2023-03-09T22:08:03.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26255.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:08:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:03 smithi122 ceph-mon[93869]: pgmap v14266: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 581 KiB/s rd, 521 KiB/s wr, 14 op/s 2023-03-09T22:08:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:03 smithi181 ceph-mon[94189]: pgmap v14266: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 581 KiB/s rd, 521 KiB/s wr, 14 op/s 2023-03-09T22:08:04.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:03 smithi038 ceph-mon[92422]: pgmap v14266: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 581 KiB/s rd, 521 KiB/s wr, 14 op/s 2023-03-09T22:08:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:05 smithi122 ceph-mon[93869]: pgmap v14267: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 453 KiB/s rd, 395 KiB/s wr, 13 op/s 2023-03-09T22:08:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:05 smithi181 ceph-mon[94189]: pgmap v14267: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 453 KiB/s rd, 395 KiB/s wr, 13 op/s 2023-03-09T22:08:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:05 smithi038 ceph-mon[92422]: pgmap v14267: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 453 KiB/s rd, 395 KiB/s wr, 13 op/s 2023-03-09T22:08:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25796.354145 secs 2023-03-09T22:08:07.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25796.354145 secs 2023-03-09T22:08:07.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25796.354145 secs 2023-03-09T22:08:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:07 smithi122 ceph-mon[93869]: pgmap v14268: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 377 KiB/s wr, 15 op/s 2023-03-09T22:08:08.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:07 smithi181 ceph-mon[94189]: pgmap v14268: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 377 KiB/s wr, 15 op/s 2023-03-09T22:08:08.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:07 smithi038 ceph-mon[92422]: pgmap v14268: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 377 KiB/s wr, 15 op/s 2023-03-09T22:08:08.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26260.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:08:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25798.713967 secs 2023-03-09T22:08:09.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25798.713967 secs 2023-03-09T22:08:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25798.713967 secs 2023-03-09T22:08:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:09 smithi122 ceph-mon[93869]: pgmap v14269: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 159 KiB/s wr, 22 op/s 2023-03-09T22:08:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:09 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:08:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:08:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:08:10.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:09 smithi181 ceph-mon[94189]: pgmap v14269: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 159 KiB/s wr, 22 op/s 2023-03-09T22:08:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:09 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:08:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:08:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:08:10.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:09 smithi038 ceph-mon[92422]: pgmap v14269: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 159 KiB/s wr, 22 op/s 2023-03-09T22:08:10.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:09 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:08:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:08:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:08:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:10 smithi122 ceph-mon[93869]: osdmap e1105: 12 total, 12 up, 12 in 2023-03-09T22:08:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25801.354207 secs 2023-03-09T22:08:11.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:10 smithi181 ceph-mon[94189]: osdmap e1105: 12 total, 12 up, 12 in 2023-03-09T22:08:11.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25801.354207 secs 2023-03-09T22:08:11.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:10 smithi038 ceph-mon[92422]: osdmap e1105: 12 total, 12 up, 12 in 2023-03-09T22:08:11.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25801.354207 secs 2023-03-09T22:08:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:11 smithi122 ceph-mon[93869]: pgmap v14271: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 453 KiB/s wr, 27 op/s 2023-03-09T22:08:12.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:11 smithi181 ceph-mon[94189]: pgmap v14271: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 453 KiB/s wr, 27 op/s 2023-03-09T22:08:12.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:11 smithi038 ceph-mon[92422]: pgmap v14271: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 453 KiB/s wr, 27 op/s 2023-03-09T22:08:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26265.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:08:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25803.714140 secs 2023-03-09T22:08:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:13 smithi122 ceph-mon[93869]: pgmap v14272: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 653 KiB/s rd, 695 KiB/s wr, 27 op/s 2023-03-09T22:08:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25803.714140 secs 2023-03-09T22:08:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:13 smithi181 ceph-mon[94189]: pgmap v14272: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 653 KiB/s rd, 695 KiB/s wr, 27 op/s 2023-03-09T22:08:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25803.714140 secs 2023-03-09T22:08:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:13 smithi038 ceph-mon[92422]: pgmap v14272: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 653 KiB/s rd, 695 KiB/s wr, 27 op/s 2023-03-09T22:08:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:15 smithi122 ceph-mon[93869]: pgmap v14273: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 641 KiB/s wr, 26 op/s 2023-03-09T22:08:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25806.354269 secs 2023-03-09T22:08:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:15 smithi122 ceph-mon[93869]: osdmap e1106: 12 total, 12 up, 12 in 2023-03-09T22:08:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:15 smithi181 ceph-mon[94189]: pgmap v14273: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 641 KiB/s wr, 26 op/s 2023-03-09T22:08:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25806.354269 secs 2023-03-09T22:08:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:15 smithi181 ceph-mon[94189]: osdmap e1106: 12 total, 12 up, 12 in 2023-03-09T22:08:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:15 smithi038 ceph-mon[92422]: pgmap v14273: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 641 KiB/s wr, 26 op/s 2023-03-09T22:08:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25806.354269 secs 2023-03-09T22:08:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:15 smithi038 ceph-mon[92422]: osdmap e1106: 12 total, 12 up, 12 in 2023-03-09T22:08:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:17 smithi122 ceph-mon[93869]: pgmap v14275: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 737 KiB/s rd, 736 KiB/s wr, 13 op/s 2023-03-09T22:08:18.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:17 smithi181 ceph-mon[94189]: pgmap v14275: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 737 KiB/s rd, 736 KiB/s wr, 13 op/s 2023-03-09T22:08:18.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:17 smithi038 ceph-mon[92422]: pgmap v14275: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 737 KiB/s rd, 736 KiB/s wr, 13 op/s 2023-03-09T22:08:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26270.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:08:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25808.714239 secs 2023-03-09T22:08:19.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25808.714239 secs 2023-03-09T22:08:19.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25808.714239 secs 2023-03-09T22:08:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:19 smithi122 ceph-mon[93869]: pgmap v14276: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 407 KiB/s wr, 10 op/s 2023-03-09T22:08:20.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:19 smithi181 ceph-mon[94189]: pgmap v14276: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 407 KiB/s wr, 10 op/s 2023-03-09T22:08:20.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:19 smithi038 ceph-mon[92422]: pgmap v14276: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 407 KiB/s wr, 10 op/s 2023-03-09T22:08:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25811.354327 secs 2023-03-09T22:08:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:21 smithi122 ceph-mon[93869]: pgmap v14277: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 378 KiB/s wr, 9 op/s 2023-03-09T22:08:22.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25811.354327 secs 2023-03-09T22:08:22.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:21 smithi181 ceph-mon[94189]: pgmap v14277: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 378 KiB/s wr, 9 op/s 2023-03-09T22:08:22.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25811.354327 secs 2023-03-09T22:08:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:21 smithi038 ceph-mon[92422]: pgmap v14277: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 378 KiB/s wr, 9 op/s 2023-03-09T22:08:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26275.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:08:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25813.714359 secs 2023-03-09T22:08:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:23 smithi122 ceph-mon[93869]: pgmap v14278: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 219 KiB/s wr, 9 op/s 2023-03-09T22:08:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25813.714359 secs 2023-03-09T22:08:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:23 smithi181 ceph-mon[94189]: pgmap v14278: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 219 KiB/s wr, 9 op/s 2023-03-09T22:08:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25813.714359 secs 2023-03-09T22:08:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:23 smithi038 ceph-mon[92422]: pgmap v14278: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 219 KiB/s wr, 9 op/s 2023-03-09T22:08:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:25 smithi122 ceph-mon[93869]: pgmap v14279: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 195 KiB/s wr, 8 op/s 2023-03-09T22:08:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:25 smithi181 ceph-mon[94189]: pgmap v14279: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 195 KiB/s wr, 8 op/s 2023-03-09T22:08:26.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:25 smithi038 ceph-mon[92422]: pgmap v14279: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 195 KiB/s wr, 8 op/s 2023-03-09T22:08:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25816.354386 secs 2023-03-09T22:08:27.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25816.354386 secs 2023-03-09T22:08:27.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25816.354386 secs 2023-03-09T22:08:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:27 smithi122 ceph-mon[93869]: pgmap v14280: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 181 KiB/s wr, 7 op/s 2023-03-09T22:08:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:08:28.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:27 smithi181 ceph-mon[94189]: pgmap v14280: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 181 KiB/s wr, 7 op/s 2023-03-09T22:08:28.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:08:28.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:27 smithi038 ceph-mon[92422]: pgmap v14280: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 181 KiB/s wr, 7 op/s 2023-03-09T22:08:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:08:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26280.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:08:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25818.714460 secs 2023-03-09T22:08:29.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25818.714460 secs 2023-03-09T22:08:29.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25818.714460 secs 2023-03-09T22:08:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:29 smithi122 ceph-mon[93869]: pgmap v14281: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 162 KiB/s wr, 7 op/s 2023-03-09T22:08:30.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:29 smithi181 ceph-mon[94189]: pgmap v14281: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 162 KiB/s wr, 7 op/s 2023-03-09T22:08:30.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:29 smithi038 ceph-mon[92422]: pgmap v14281: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 162 KiB/s wr, 7 op/s 2023-03-09T22:08:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25821.354446 secs 2023-03-09T22:08:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:08:31.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:08:31.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:08:31.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:08:31.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:08:31.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:08:31.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:31 smithi122 ceph-mon[93869]: pgmap v14282: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 217 KiB/s wr, 8 op/s 2023-03-09T22:08:31.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:08:31.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:08:31.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:08:31.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:08:31.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:08:31.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:08:31.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:08:31.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:08:31.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:08:31.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:08:31.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:08:31.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:08:31.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:08:31.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:08:31.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:08:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25821.354446 secs 2023-03-09T22:08:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:08:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:08:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:08:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:08:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:08:31.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:08:31.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:31 smithi181 ceph-mon[94189]: pgmap v14282: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 217 KiB/s wr, 8 op/s 2023-03-09T22:08:31.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:08:31.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:08:31.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:08:31.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:08:31.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:08:31.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:08:31.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:08:31.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:08:31.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:08:31.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:08:31.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:08:31.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:08:31.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:08:31.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:08:31.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:08:31.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25821.354446 secs 2023-03-09T22:08:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:08:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:08:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:08:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:08:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:08:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:08:31.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:31 smithi038 ceph-mon[92422]: pgmap v14282: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 217 KiB/s wr, 8 op/s 2023-03-09T22:08:31.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:08:31.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:08:31.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:08:31.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:08:31.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:08:31.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:08:31.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:08:31.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:08:31.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:08:31.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:08:31.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:08:31.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:08:31.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:08:31.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:08:31.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:08:32.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:32 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi038 to 22984k 2023-03-09T22:08:32.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:32 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T22:08:32.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:32 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi038 to 23535665: error parsing value: Value '23535665' is below minimum 939524096 2023-03-09T22:08:32.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:32 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T22:08:32.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:32 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T22:08:32.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:32 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T22:08:32.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:32 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi038 to 22984k 2023-03-09T22:08:32.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:32 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T22:08:32.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:32 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi038 to 23535665: error parsing value: Value '23535665' is below minimum 939524096 2023-03-09T22:08:32.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:32 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T22:08:32.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:32 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T22:08:32.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:32 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T22:08:32.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:32 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi038 to 22984k 2023-03-09T22:08:32.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:32 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T22:08:32.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:32 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi038 to 23535665: error parsing value: Value '23535665' is below minimum 939524096 2023-03-09T22:08:32.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:32 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T22:08:32.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:32 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T22:08:32.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:32 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T22:08:33.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26285.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:08:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25823.714577 secs 2023-03-09T22:08:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:33 smithi122 ceph-mon[93869]: pgmap v14283: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 402 KiB/s rd, 197 KiB/s wr, 9 op/s 2023-03-09T22:08:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25823.714577 secs 2023-03-09T22:08:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:33 smithi181 ceph-mon[94189]: pgmap v14283: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 402 KiB/s rd, 197 KiB/s wr, 9 op/s 2023-03-09T22:08:33.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25823.714577 secs 2023-03-09T22:08:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:33 smithi038 ceph-mon[92422]: pgmap v14283: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 402 KiB/s rd, 197 KiB/s wr, 9 op/s 2023-03-09T22:08:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:35 smithi122 ceph-mon[93869]: pgmap v14284: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 129 KiB/s wr, 8 op/s 2023-03-09T22:08:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:35 smithi181 ceph-mon[94189]: pgmap v14284: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 129 KiB/s wr, 8 op/s 2023-03-09T22:08:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:35 smithi038 ceph-mon[92422]: pgmap v14284: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 129 KiB/s wr, 8 op/s 2023-03-09T22:08:37.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25826.354504 secs 2023-03-09T22:08:37.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25826.354504 secs 2023-03-09T22:08:37.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25826.354504 secs 2023-03-09T22:08:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:37 smithi122 ceph-mon[93869]: pgmap v14285: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 129 KiB/s wr, 9 op/s 2023-03-09T22:08:38.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:37 smithi181 ceph-mon[94189]: pgmap v14285: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 129 KiB/s wr, 9 op/s 2023-03-09T22:08:38.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:37 smithi038 ceph-mon[92422]: pgmap v14285: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 129 KiB/s wr, 9 op/s 2023-03-09T22:08:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26290.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:08:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25828.714699 secs 2023-03-09T22:08:39.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25828.714699 secs 2023-03-09T22:08:39.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25828.714699 secs 2023-03-09T22:08:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:39 smithi122 ceph-mon[93869]: pgmap v14286: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 128 KiB/s wr, 8 op/s 2023-03-09T22:08:40.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:39 smithi181 ceph-mon[94189]: pgmap v14286: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 128 KiB/s wr, 8 op/s 2023-03-09T22:08:40.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:39 smithi038 ceph-mon[92422]: pgmap v14286: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 128 KiB/s wr, 8 op/s 2023-03-09T22:08:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25831.354565 secs 2023-03-09T22:08:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:41 smithi122 ceph-mon[93869]: pgmap v14287: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 235 KiB/s wr, 9 op/s 2023-03-09T22:08:42.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25831.354565 secs 2023-03-09T22:08:42.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:41 smithi181 ceph-mon[94189]: pgmap v14287: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 235 KiB/s wr, 9 op/s 2023-03-09T22:08:42.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25831.354565 secs 2023-03-09T22:08:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:41 smithi038 ceph-mon[92422]: pgmap v14287: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 235 KiB/s wr, 9 op/s 2023-03-09T22:08:43.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26295.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:08:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:43 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 25833.714812 secs 2023-03-09T22:08:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:43 smithi122 ceph-mon[93869]: pgmap v14288: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 108 KiB/s wr, 8 op/s 2023-03-09T22:08:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:43 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 25833.714812 secs 2023-03-09T22:08:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:43 smithi181 ceph-mon[94189]: pgmap v14288: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 108 KiB/s wr, 8 op/s 2023-03-09T22:08:44.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:43 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 25833.714812 secs 2023-03-09T22:08:44.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:43 smithi038 ceph-mon[92422]: pgmap v14288: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 108 KiB/s wr, 8 op/s 2023-03-09T22:08:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:45 smithi122 ceph-mon[93869]: pgmap v14289: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 526 KiB/s rd, 215 KiB/s wr, 8 op/s 2023-03-09T22:08:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:45 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:08:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:45 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:08:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:45 smithi181 ceph-mon[94189]: pgmap v14289: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 526 KiB/s rd, 215 KiB/s wr, 8 op/s 2023-03-09T22:08:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:45 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:08:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:45 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:08:46.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:45 smithi038 ceph-mon[92422]: pgmap v14289: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 526 KiB/s rd, 215 KiB/s wr, 8 op/s 2023-03-09T22:08:46.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:45 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:08:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:45 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:08:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25836.354627 secs 2023-03-09T22:08:47.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25836.354627 secs 2023-03-09T22:08:47.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25836.354627 secs 2023-03-09T22:08:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:47 smithi122 ceph-mon[93869]: pgmap v14290: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 656 KiB/s rd, 344 KiB/s wr, 9 op/s 2023-03-09T22:08:48.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:47 smithi181 ceph-mon[94189]: pgmap v14290: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 656 KiB/s rd, 344 KiB/s wr, 9 op/s 2023-03-09T22:08:48.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:47 smithi038 ceph-mon[92422]: pgmap v14290: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 656 KiB/s rd, 344 KiB/s wr, 9 op/s 2023-03-09T22:08:48.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26300.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:08:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:48 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 25838.714931 secs 2023-03-09T22:08:49.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:48 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 25838.714931 secs 2023-03-09T22:08:49.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:48 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 25838.714931 secs 2023-03-09T22:08:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:49 smithi122 ceph-mon[93869]: pgmap v14291: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 654 KiB/s rd, 344 KiB/s wr, 7 op/s 2023-03-09T22:08:50.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:49 smithi181 ceph-mon[94189]: pgmap v14291: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 654 KiB/s rd, 344 KiB/s wr, 7 op/s 2023-03-09T22:08:50.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:49 smithi038 ceph-mon[92422]: pgmap v14291: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 654 KiB/s rd, 344 KiB/s wr, 7 op/s 2023-03-09T22:08:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25841.354686 secs 2023-03-09T22:08:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:51 smithi122 ceph-mon[93869]: pgmap v14292: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 753 KiB/s rd, 364 KiB/s wr, 9 op/s 2023-03-09T22:08:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25841.354686 secs 2023-03-09T22:08:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:51 smithi181 ceph-mon[94189]: pgmap v14292: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 753 KiB/s rd, 364 KiB/s wr, 9 op/s 2023-03-09T22:08:52.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25841.354686 secs 2023-03-09T22:08:52.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:51 smithi038 ceph-mon[92422]: pgmap v14292: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 753 KiB/s rd, 364 KiB/s wr, 9 op/s 2023-03-09T22:08:53.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26305.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:08:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25843.715078 secs 2023-03-09T22:08:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:53 smithi122 ceph-mon[93869]: pgmap v14293: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 781 KiB/s rd, 391 KiB/s wr, 9 op/s 2023-03-09T22:08:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:08:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:08:54.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25843.715078 secs 2023-03-09T22:08:54.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:53 smithi181 ceph-mon[94189]: pgmap v14293: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 781 KiB/s rd, 391 KiB/s wr, 9 op/s 2023-03-09T22:08:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:08:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:08:54.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25843.715078 secs 2023-03-09T22:08:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:53 smithi038 ceph-mon[92422]: pgmap v14293: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 781 KiB/s rd, 391 KiB/s wr, 9 op/s 2023-03-09T22:08:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:08:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:08:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:55 smithi122 ceph-mon[93869]: pgmap v14294: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 812 KiB/s rd, 411 KiB/s wr, 10 op/s 2023-03-09T22:08:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:55 smithi181 ceph-mon[94189]: pgmap v14294: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 812 KiB/s rd, 411 KiB/s wr, 10 op/s 2023-03-09T22:08:56.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:55 smithi038 ceph-mon[92422]: pgmap v14294: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 812 KiB/s rd, 411 KiB/s wr, 10 op/s 2023-03-09T22:08:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25846.354746 secs 2023-03-09T22:08:57.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25846.354746 secs 2023-03-09T22:08:57.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25846.354746 secs 2023-03-09T22:08:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:57 smithi122 ceph-mon[93869]: pgmap v14295: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 701 KiB/s rd, 608 KiB/s wr, 12 op/s 2023-03-09T22:08:58.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:57 smithi181 ceph-mon[94189]: pgmap v14295: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 701 KiB/s rd, 608 KiB/s wr, 12 op/s 2023-03-09T22:08:58.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:57 smithi038 ceph-mon[92422]: pgmap v14295: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 701 KiB/s rd, 608 KiB/s wr, 12 op/s 2023-03-09T22:08:58.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26310.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:08:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25848.715192 secs 2023-03-09T22:08:59.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25848.715192 secs 2023-03-09T22:08:59.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25848.715192 secs 2023-03-09T22:09:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:08:59 smithi122 ceph-mon[93869]: pgmap v14296: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 571 KiB/s rd, 479 KiB/s wr, 10 op/s 2023-03-09T22:09:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:08:59 smithi181 ceph-mon[94189]: pgmap v14296: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 571 KiB/s rd, 479 KiB/s wr, 10 op/s 2023-03-09T22:09:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:08:59 smithi038 ceph-mon[92422]: pgmap v14296: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 571 KiB/s rd, 479 KiB/s wr, 10 op/s 2023-03-09T22:09:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:09:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25851.354806 secs 2023-03-09T22:09:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:09:01 smithi122 ceph-mon[93869]: pgmap v14297: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 592 KiB/s rd, 500 KiB/s wr, 12 op/s 2023-03-09T22:09:02.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:09:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25851.354806 secs 2023-03-09T22:09:02.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:09:01 smithi181 ceph-mon[94189]: pgmap v14297: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 592 KiB/s rd, 500 KiB/s wr, 12 op/s 2023-03-09T22:09:02.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:09:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25851.354806 secs 2023-03-09T22:09:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:09:01 smithi038 ceph-mon[92422]: pgmap v14297: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 592 KiB/s rd, 500 KiB/s wr, 12 op/s 2023-03-09T22:09:03.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26315.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:09:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:09:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25853.715257 secs 2023-03-09T22:09:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:09:03 smithi122 ceph-mon[93869]: pgmap v14298: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 611 KiB/s rd, 596 KiB/s wr, 12 op/s 2023-03-09T22:09:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:09:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25853.715257 secs 2023-03-09T22:09:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:09:03 smithi181 ceph-mon[94189]: pgmap v14298: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 611 KiB/s rd, 596 KiB/s wr, 12 op/s 2023-03-09T22:09:04.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:09:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25853.715257 secs 2023-03-09T22:09:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:09:03 smithi038 ceph-mon[92422]: pgmap v14298: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 611 KiB/s rd, 596 KiB/s wr, 12 op/s 2023-03-09T22:09:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:09:05 smithi122 ceph-mon[93869]: pgmap v14299: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 503 KiB/s rd, 682 KiB/s wr, 22 op/s 2023-03-09T22:09:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:09:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25856.354874 secs 2023-03-09T22:09:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:09:05 smithi181 ceph-mon[94189]: pgmap v14299: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 503 KiB/s rd, 682 KiB/s wr, 22 op/s 2023-03-09T22:09:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:09:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25856.354874 secs 2023-03-09T22:09:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:09:05 smithi038 ceph-mon[92422]: pgmap v14299: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 503 KiB/s rd, 682 KiB/s wr, 22 op/s 2023-03-09T22:09:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:09:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25856.354874 secs 2023-03-09T22:09:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:09:07 smithi122 ceph-mon[93869]: pgmap v14300: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 673 KiB/s wr, 25 op/s 2023-03-09T22:09:08.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:09:07 smithi181 ceph-mon[94189]: pgmap v14300: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 673 KiB/s wr, 25 op/s 2023-03-09T22:09:08.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:09:07 smithi038 ceph-mon[92422]: pgmap v14300: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 673 KiB/s wr, 25 op/s 2023-03-09T22:09:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26320.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:09:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:09:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25858.715368 secs 2023-03-09T22:09:09.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:09:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25858.715368 secs 2023-03-09T22:09:09.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:09:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25858.715368 secs 2023-03-09T22:09:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:09:09 smithi122 ceph-mon[93869]: pgmap v14301: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 717 KiB/s rd, 516 KiB/s wr, 23 op/s 2023-03-09T22:09:10.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:09:09 smithi181 ceph-mon[94189]: pgmap v14301: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 717 KiB/s rd, 516 KiB/s wr, 23 op/s 2023-03-09T22:09:10.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:09:09 smithi038 ceph-mon[92422]: pgmap v14301: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 717 KiB/s rd, 516 KiB/s wr, 23 op/s 2023-03-09T22:09:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:09:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25861.354934 secs 2023-03-09T22:09:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:09:11 smithi122 ceph-mon[93869]: pgmap v14302: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 515 KiB/s wr, 23 op/s 2023-03-09T22:09:12.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:09:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25861.354934 secs 2023-03-09T22:09:12.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:09:11 smithi181 ceph-mon[94189]: pgmap v14302: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 515 KiB/s wr, 23 op/s 2023-03-09T22:09:12.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:09:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25861.354934 secs 2023-03-09T22:09:12.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:09:11 smithi038 ceph-mon[92422]: pgmap v14302: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 515 KiB/s wr, 23 op/s 2023-03-09T22:09:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26325.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:09:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:09:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25863.715483 secs 2023-03-09T22:09:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:09:13 smithi122 ceph-mon[93869]: pgmap v14303: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 763 KiB/s rd, 562 KiB/s wr, 24 op/s 2023-03-09T22:09:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:09:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25863.715483 secs 2023-03-09T22:09:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:09:13 smithi181 ceph-mon[94189]: pgmap v14303: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 763 KiB/s rd, 562 KiB/s wr, 24 op/s 2023-03-09T22:09:14.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:09:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25863.715483 secs 2023-03-09T22:09:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:09:13 smithi038 ceph-mon[92422]: pgmap v14303: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 763 KiB/s rd, 562 KiB/s wr, 24 op/s 2023-03-09T22:09:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:09:15 smithi122 ceph-mon[93869]: pgmap v14304: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 673 KiB/s rd, 471 KiB/s wr, 22 op/s 2023-03-09T22:09:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:09:15 smithi181 ceph-mon[94189]: pgmap v14304: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 673 KiB/s rd, 471 KiB/s wr, 22 op/s 2023-03-09T22:09:16.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:09:15 smithi038 ceph-mon[92422]: pgmap v14304: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 673 KiB/s rd, 471 KiB/s wr, 22 op/s 2023-03-09T22:09:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:09:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25866.354982 secs 2023-03-09T22:09:17.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:09:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25866.354982 secs 2023-03-09T22:09:17.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:09:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25866.354982 secs 2023-03-09T22:09:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:09:17 smithi122 ceph-mon[93869]: pgmap v14305: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 647 KiB/s rd, 252 KiB/s wr, 12 op/s 2023-03-09T22:09:18.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:09:17 smithi181 ceph-mon[94189]: pgmap v14305: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 647 KiB/s rd, 252 KiB/s wr, 12 op/s 2023-03-09T22:09:18.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:09:17 smithi038 ceph-mon[92422]: pgmap v14305: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 647 KiB/s rd, 252 KiB/s wr, 12 op/s 2023-03-09T22:09:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26330.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:09:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:09:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25868.715620 secs 2023-03-09T22:09:19.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:09:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25868.715620 secs 2023-03-09T22:09:19.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:09:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25868.715620 secs 2023-03-09T22:09:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:09:19 smithi122 ceph-mon[93869]: pgmap v14306: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 241 KiB/s wr, 9 op/s 2023-03-09T22:09:20.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:09:19 smithi181 ceph-mon[94189]: pgmap v14306: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 241 KiB/s wr, 9 op/s 2023-03-09T22:09:20.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:09:19 smithi038 ceph-mon[92422]: pgmap v14306: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 241 KiB/s wr, 9 op/s 2023-03-09T22:09:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:09:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25871.355051 secs 2023-03-09T22:09:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:09:21 smithi122 ceph-mon[93869]: pgmap v14307: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 94 KiB/s wr, 9 op/s 2023-03-09T22:09:22.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:09:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25871.355051 secs 2023-03-09T22:09:22.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:09:21 smithi181 ceph-mon[94189]: pgmap v14307: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 94 KiB/s wr, 9 op/s 2023-03-09T22:09:22.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:09:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25871.355051 secs 2023-03-09T22:09:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:09:21 smithi038 ceph-mon[92422]: pgmap v14307: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 94 KiB/s wr, 9 op/s 2023-03-09T22:09:23.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26335.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:09:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:09:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25873.715768 secs 2023-03-09T22:09:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:09:23 smithi122 ceph-mon[93869]: pgmap v14308: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 328 KiB/s wr, 10 op/s 2023-03-09T22:09:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:09:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25873.715768 secs 2023-03-09T22:09:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:09:23 smithi181 ceph-mon[94189]: pgmap v14308: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 328 KiB/s wr, 10 op/s 2023-03-09T22:09:24.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:09:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25873.715768 secs 2023-03-09T22:09:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:09:23 smithi038 ceph-mon[92422]: pgmap v14308: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 328 KiB/s wr, 10 op/s 2023-03-09T22:09:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:09:25 smithi122 ceph-mon[93869]: pgmap v14309: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 293 KiB/s wr, 11 op/s 2023-03-09T22:09:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:09:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25876.355112 secs 2023-03-09T22:09:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:09:25 smithi181 ceph-mon[94189]: pgmap v14309: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 293 KiB/s wr, 11 op/s 2023-03-09T22:09:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:09:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25876.355112 secs 2023-03-09T22:09:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:09:25 smithi038 ceph-mon[92422]: pgmap v14309: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 293 KiB/s wr, 11 op/s 2023-03-09T22:09:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:09:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25876.355112 secs 2023-03-09T22:09:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:09:27 smithi122 ceph-mon[93869]: pgmap v14310: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 319 KiB/s wr, 12 op/s 2023-03-09T22:09:28.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:09:27 smithi181 ceph-mon[94189]: pgmap v14310: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 319 KiB/s wr, 12 op/s 2023-03-09T22:09:28.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:09:27 smithi038 ceph-mon[92422]: pgmap v14310: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 319 KiB/s wr, 12 op/s 2023-03-09T22:09:28.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26340.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:09:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:09:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25878.715904 secs 2023-03-09T22:09:29.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:09:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25878.715904 secs 2023-03-09T22:09:29.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:09:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25878.715904 secs 2023-03-09T22:09:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:09:29 smithi122 ceph-mon[93869]: pgmap v14311: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 319 KiB/s wr, 10 op/s 2023-03-09T22:09:30.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:09:29 smithi181 ceph-mon[94189]: pgmap v14311: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 319 KiB/s wr, 10 op/s 2023-03-09T22:09:30.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:09:29 smithi038 ceph-mon[92422]: pgmap v14311: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 319 KiB/s wr, 10 op/s 2023-03-09T22:09:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:09:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25881.355173 secs 2023-03-09T22:09:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:09:30 smithi122 ceph-mon[93869]: osdmap e1107: 12 total, 12 up, 12 in 2023-03-09T22:09:31.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:09:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25881.355173 secs 2023-03-09T22:09:31.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:09:30 smithi181 ceph-mon[94189]: osdmap e1107: 12 total, 12 up, 12 in 2023-03-09T22:09:31.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:09:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25881.355173 secs 2023-03-09T22:09:31.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:09:30 smithi038 ceph-mon[92422]: osdmap e1107: 12 total, 12 up, 12 in 2023-03-09T22:09:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:09:31 smithi122 ceph-mon[93869]: pgmap v14313: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 571 KiB/s rd, 596 KiB/s wr, 13 op/s 2023-03-09T22:09:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:09:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:09:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:09:31 smithi181 ceph-mon[94189]: pgmap v14313: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 571 KiB/s rd, 596 KiB/s wr, 13 op/s 2023-03-09T22:09:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:09:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:09:32.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:09:31 smithi038 ceph-mon[92422]: pgmap v14313: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 571 KiB/s rd, 596 KiB/s wr, 13 op/s 2023-03-09T22:09:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:09:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:09:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:09:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:09:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:09:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:09:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:09:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:09:33.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:09:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:09:33.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:09:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:09:33.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:09:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:09:33.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:09:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:09:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:09:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:09:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:09:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:09:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26345.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:09:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:09:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25883.715994 secs 2023-03-09T22:09:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:09:33 smithi122 ceph-mon[93869]: pgmap v14314: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 316 KiB/s wr, 12 op/s 2023-03-09T22:09:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:09:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25883.715994 secs 2023-03-09T22:09:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:09:33 smithi181 ceph-mon[94189]: pgmap v14314: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 316 KiB/s wr, 12 op/s 2023-03-09T22:09:34.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:09:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25883.715994 secs 2023-03-09T22:09:34.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:09:33 smithi038 ceph-mon[92422]: pgmap v14314: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 316 KiB/s wr, 12 op/s 2023-03-09T22:09:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:09:35 smithi122 ceph-mon[93869]: pgmap v14315: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 278 KiB/s wr, 10 op/s 2023-03-09T22:09:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:09:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25886.355230 secs 2023-03-09T22:09:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:09:35 smithi122 ceph-mon[93869]: osdmap e1108: 12 total, 12 up, 12 in 2023-03-09T22:09:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:09:35 smithi181 ceph-mon[94189]: pgmap v14315: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 278 KiB/s wr, 10 op/s 2023-03-09T22:09:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:09:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25886.355230 secs 2023-03-09T22:09:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:09:35 smithi181 ceph-mon[94189]: osdmap e1108: 12 total, 12 up, 12 in 2023-03-09T22:09:36.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:09:35 smithi038 ceph-mon[92422]: pgmap v14315: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 278 KiB/s wr, 10 op/s 2023-03-09T22:09:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:09:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25886.355230 secs 2023-03-09T22:09:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:09:35 smithi038 ceph-mon[92422]: osdmap e1108: 12 total, 12 up, 12 in 2023-03-09T22:09:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:09:37 smithi122 ceph-mon[93869]: pgmap v14317: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 268 KiB/s wr, 11 op/s 2023-03-09T22:09:38.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:09:37 smithi181 ceph-mon[94189]: pgmap v14317: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 268 KiB/s wr, 11 op/s 2023-03-09T22:09:38.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:09:37 smithi038 ceph-mon[92422]: pgmap v14317: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 268 KiB/s wr, 11 op/s 2023-03-09T22:09:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26350.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:09:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:09:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25888.716125 secs 2023-03-09T22:09:39.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:09:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25888.716125 secs 2023-03-09T22:09:39.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:09:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25888.716125 secs 2023-03-09T22:09:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:09:39 smithi122 ceph-mon[93869]: pgmap v14318: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.5 KiB/s wr, 8 op/s 2023-03-09T22:09:40.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:09:39 smithi181 ceph-mon[94189]: pgmap v14318: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.5 KiB/s wr, 8 op/s 2023-03-09T22:09:40.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:09:39 smithi038 ceph-mon[92422]: pgmap v14318: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.5 KiB/s wr, 8 op/s 2023-03-09T22:09:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:09:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25891.355290 secs 2023-03-09T22:09:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:09:41 smithi122 ceph-mon[93869]: pgmap v14319: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 198 KiB/s wr, 8 op/s 2023-03-09T22:09:42.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:09:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25891.355290 secs 2023-03-09T22:09:42.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:09:41 smithi181 ceph-mon[94189]: pgmap v14319: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 198 KiB/s wr, 8 op/s 2023-03-09T22:09:42.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:09:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25891.355290 secs 2023-03-09T22:09:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:09:41 smithi038 ceph-mon[92422]: pgmap v14319: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 198 KiB/s wr, 8 op/s 2023-03-09T22:09:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26355.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:09:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:09:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25893.716227 secs 2023-03-09T22:09:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:09:43 smithi122 ceph-mon[93869]: pgmap v14320: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 284 KiB/s wr, 8 op/s 2023-03-09T22:09:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:09:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25893.716227 secs 2023-03-09T22:09:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:09:43 smithi181 ceph-mon[94189]: pgmap v14320: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 284 KiB/s wr, 8 op/s 2023-03-09T22:09:44.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:09:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25893.716227 secs 2023-03-09T22:09:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:09:43 smithi038 ceph-mon[92422]: pgmap v14320: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 284 KiB/s wr, 8 op/s 2023-03-09T22:09:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:09:45 smithi122 ceph-mon[93869]: pgmap v14321: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 284 KiB/s wr, 8 op/s 2023-03-09T22:09:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:09:45 smithi181 ceph-mon[94189]: pgmap v14321: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 284 KiB/s wr, 8 op/s 2023-03-09T22:09:46.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:09:45 smithi038 ceph-mon[92422]: pgmap v14321: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 284 KiB/s wr, 8 op/s 2023-03-09T22:09:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:09:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25896.355351 secs 2023-03-09T22:09:47.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:09:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25896.355351 secs 2023-03-09T22:09:47.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:09:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25896.355351 secs 2023-03-09T22:09:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:09:47 smithi122 ceph-mon[93869]: pgmap v14322: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 396 KiB/s wr, 9 op/s 2023-03-09T22:09:48.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:09:47 smithi181 ceph-mon[94189]: pgmap v14322: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 396 KiB/s wr, 9 op/s 2023-03-09T22:09:48.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:09:47 smithi038 ceph-mon[92422]: pgmap v14322: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 396 KiB/s wr, 9 op/s 2023-03-09T22:09:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26360.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:09:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:09:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25898.716320 secs 2023-03-09T22:09:49.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:09:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25898.716320 secs 2023-03-09T22:09:49.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:09:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25898.716320 secs 2023-03-09T22:09:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:09:49 smithi122 ceph-mon[93869]: pgmap v14323: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 356 KiB/s wr, 8 op/s 2023-03-09T22:09:50.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:09:49 smithi181 ceph-mon[94189]: pgmap v14323: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 356 KiB/s wr, 8 op/s 2023-03-09T22:09:50.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:09:49 smithi038 ceph-mon[92422]: pgmap v14323: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 356 KiB/s wr, 8 op/s 2023-03-09T22:09:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:09:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25901.355413 secs 2023-03-09T22:09:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:09:51 smithi122 ceph-mon[93869]: pgmap v14324: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 376 KiB/s wr, 10 op/s 2023-03-09T22:09:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:09:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25901.355413 secs 2023-03-09T22:09:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:09:51 smithi181 ceph-mon[94189]: pgmap v14324: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 376 KiB/s wr, 10 op/s 2023-03-09T22:09:52.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:09:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25901.355413 secs 2023-03-09T22:09:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:09:51 smithi038 ceph-mon[92422]: pgmap v14324: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 376 KiB/s wr, 10 op/s 2023-03-09T22:09:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:09:52 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:09:53.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:09:52 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:09:53.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:09:52 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:09:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26365.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:09:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:09:53 smithi122 ceph-mon[93869]: pgmap v14325: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 363 KiB/s wr, 10 op/s 2023-03-09T22:09:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:09:53 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:09:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:09:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:09:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:09:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:09:54.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:09:53 smithi181 ceph-mon[94189]: pgmap v14325: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 363 KiB/s wr, 10 op/s 2023-03-09T22:09:54.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:09:53 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:09:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:09:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:09:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:09:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:09:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:09:53 smithi038 ceph-mon[92422]: pgmap v14325: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 363 KiB/s wr, 10 op/s 2023-03-09T22:09:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:09:53 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:09:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:09:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:09:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:09:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:09:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:09:55 smithi122 ceph-mon[93869]: pgmap v14326: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 654 KiB/s rd, 545 KiB/s wr, 11 op/s 2023-03-09T22:09:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:09:55 smithi181 ceph-mon[94189]: pgmap v14326: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 654 KiB/s rd, 545 KiB/s wr, 11 op/s 2023-03-09T22:09:56.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:09:55 smithi038 ceph-mon[92422]: pgmap v14326: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 654 KiB/s rd, 545 KiB/s wr, 11 op/s 2023-03-09T22:09:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:09:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25906.355501 secs 2023-03-09T22:09:57.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:09:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25906.355501 secs 2023-03-09T22:09:57.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:09:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25906.355501 secs 2023-03-09T22:09:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:09:57 smithi122 ceph-mon[93869]: pgmap v14327: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 561 KiB/s rd, 545 KiB/s wr, 12 op/s 2023-03-09T22:09:58.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:09:57 smithi181 ceph-mon[94189]: pgmap v14327: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 561 KiB/s rd, 545 KiB/s wr, 12 op/s 2023-03-09T22:09:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:09:57 smithi038 ceph-mon[92422]: pgmap v14327: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 561 KiB/s rd, 545 KiB/s wr, 12 op/s 2023-03-09T22:09:58.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26370.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:10:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:09:59 smithi122 ceph-mon[93869]: pgmap v14328: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 426 KiB/s wr, 10 op/s 2023-03-09T22:10:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:09:59 smithi181 ceph-mon[94189]: pgmap v14328: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 426 KiB/s wr, 10 op/s 2023-03-09T22:10:00.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:09:59 smithi038 ceph-mon[92422]: pgmap v14328: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 426 KiB/s wr, 10 op/s 2023-03-09T22:10:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:10:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25911.355572 secs 2023-03-09T22:10:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:10:00 smithi122 ceph-mon[93869]: overall HEALTH_WARN 1 MDSs report slow requests 2023-03-09T22:10:01.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:10:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25911.355572 secs 2023-03-09T22:10:01.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:10:00 smithi181 ceph-mon[94189]: overall HEALTH_WARN 1 MDSs report slow requests 2023-03-09T22:10:01.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:10:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25911.355572 secs 2023-03-09T22:10:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:10:00 smithi038 ceph-mon[92422]: overall HEALTH_WARN 1 MDSs report slow requests 2023-03-09T22:10:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:10:01 smithi122 ceph-mon[93869]: pgmap v14329: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 607 KiB/s rd, 555 KiB/s wr, 12 op/s 2023-03-09T22:10:02.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:10:01 smithi181 ceph-mon[94189]: pgmap v14329: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 607 KiB/s rd, 555 KiB/s wr, 12 op/s 2023-03-09T22:10:02.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:10:01 smithi038 ceph-mon[92422]: pgmap v14329: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 607 KiB/s rd, 555 KiB/s wr, 12 op/s 2023-03-09T22:10:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26375.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:10:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:10:03 smithi122 ceph-mon[93869]: pgmap v14330: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 600 KiB/s wr, 13 op/s 2023-03-09T22:10:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:10:03 smithi181 ceph-mon[94189]: pgmap v14330: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 600 KiB/s wr, 13 op/s 2023-03-09T22:10:04.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:10:03 smithi038 ceph-mon[92422]: pgmap v14330: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 600 KiB/s wr, 13 op/s 2023-03-09T22:10:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:10:05 smithi122 ceph-mon[93869]: pgmap v14331: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 469 KiB/s wr, 13 op/s 2023-03-09T22:10:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:10:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25916.355632 secs 2023-03-09T22:10:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:10:05 smithi181 ceph-mon[94189]: pgmap v14331: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 469 KiB/s wr, 13 op/s 2023-03-09T22:10:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:10:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25916.355632 secs 2023-03-09T22:10:06.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:10:05 smithi038 ceph-mon[92422]: pgmap v14331: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 469 KiB/s wr, 13 op/s 2023-03-09T22:10:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:10:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25916.355632 secs 2023-03-09T22:10:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:10:07 smithi122 ceph-mon[93869]: pgmap v14332: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 537 KiB/s wr, 16 op/s 2023-03-09T22:10:08.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:10:07 smithi181 ceph-mon[94189]: pgmap v14332: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 537 KiB/s wr, 16 op/s 2023-03-09T22:10:08.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:10:07 smithi038 ceph-mon[92422]: pgmap v14332: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 537 KiB/s wr, 16 op/s 2023-03-09T22:10:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26380.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:10:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:10:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25918.716785 secs 2023-03-09T22:10:09.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:10:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25918.716785 secs 2023-03-09T22:10:09.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:10:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25918.716785 secs 2023-03-09T22:10:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:10:09 smithi122 ceph-mon[93869]: pgmap v14333: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 553 KiB/s rd, 559 KiB/s wr, 25 op/s 2023-03-09T22:10:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:10:09 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:10:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:10:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:10:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:10:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:10:10.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:10:09 smithi181 ceph-mon[94189]: pgmap v14333: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 553 KiB/s rd, 559 KiB/s wr, 25 op/s 2023-03-09T22:10:10.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:10:09 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:10:10.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:10:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:10:10.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:10:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:10:10.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:10:09 smithi038 ceph-mon[92422]: pgmap v14333: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 553 KiB/s rd, 559 KiB/s wr, 25 op/s 2023-03-09T22:10:10.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:10:09 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:10:10.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:10:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:10:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:10:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:10:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:10:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25921.355693 secs 2023-03-09T22:10:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:10:10 smithi122 ceph-mon[93869]: osdmap e1109: 12 total, 12 up, 12 in 2023-03-09T22:10:11.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:10:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25921.355693 secs 2023-03-09T22:10:11.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:10:10 smithi181 ceph-mon[94189]: osdmap e1109: 12 total, 12 up, 12 in 2023-03-09T22:10:11.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:10:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25921.355693 secs 2023-03-09T22:10:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:10:10 smithi038 ceph-mon[92422]: osdmap e1109: 12 total, 12 up, 12 in 2023-03-09T22:10:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:10:11 smithi122 ceph-mon[93869]: pgmap v14335: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 796 KiB/s rd, 827 KiB/s wr, 30 op/s 2023-03-09T22:10:12.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:10:11 smithi181 ceph-mon[94189]: pgmap v14335: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 796 KiB/s rd, 827 KiB/s wr, 30 op/s 2023-03-09T22:10:12.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:10:11 smithi038 ceph-mon[92422]: pgmap v14335: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 796 KiB/s rd, 827 KiB/s wr, 30 op/s 2023-03-09T22:10:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26385.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:10:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:10:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25923.716907 secs 2023-03-09T22:10:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:10:13 smithi122 ceph-mon[93869]: pgmap v14336: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 719 KiB/s rd, 749 KiB/s wr, 29 op/s 2023-03-09T22:10:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:10:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25923.716907 secs 2023-03-09T22:10:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:10:13 smithi181 ceph-mon[94189]: pgmap v14336: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 719 KiB/s rd, 749 KiB/s wr, 29 op/s 2023-03-09T22:10:14.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:10:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25923.716907 secs 2023-03-09T22:10:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:10:13 smithi038 ceph-mon[92422]: pgmap v14336: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 719 KiB/s rd, 749 KiB/s wr, 29 op/s 2023-03-09T22:10:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:10:15 smithi122 ceph-mon[93869]: pgmap v14337: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 682 KiB/s rd, 724 KiB/s wr, 27 op/s 2023-03-09T22:10:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:10:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25926.355770 secs 2023-03-09T22:10:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:10:15 smithi122 ceph-mon[93869]: osdmap e1110: 12 total, 12 up, 12 in 2023-03-09T22:10:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:10:15 smithi181 ceph-mon[94189]: pgmap v14337: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 682 KiB/s rd, 724 KiB/s wr, 27 op/s 2023-03-09T22:10:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:10:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25926.355770 secs 2023-03-09T22:10:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:10:15 smithi181 ceph-mon[94189]: osdmap e1110: 12 total, 12 up, 12 in 2023-03-09T22:10:16.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:10:15 smithi038 ceph-mon[92422]: pgmap v14337: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 682 KiB/s rd, 724 KiB/s wr, 27 op/s 2023-03-09T22:10:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:10:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25926.355770 secs 2023-03-09T22:10:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:10:15 smithi038 ceph-mon[92422]: osdmap e1110: 12 total, 12 up, 12 in 2023-03-09T22:10:18.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:10:17 smithi181 ceph-mon[94189]: pgmap v14339: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 469 KiB/s wr, 14 op/s 2023-03-09T22:10:18.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:10:17 smithi038 ceph-mon[92422]: pgmap v14339: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 469 KiB/s wr, 14 op/s 2023-03-09T22:10:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:10:17 smithi122 ceph-mon[93869]: pgmap v14339: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 469 KiB/s wr, 14 op/s 2023-03-09T22:10:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26390.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:10:19.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:10:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25928.717042 secs 2023-03-09T22:10:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:10:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25928.717042 secs 2023-03-09T22:10:19.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:10:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25928.717042 secs 2023-03-09T22:10:20.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:10:19 smithi038 ceph-mon[92422]: pgmap v14340: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 436 KiB/s rd, 412 KiB/s wr, 12 op/s 2023-03-09T22:10:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:10:19 smithi122 ceph-mon[93869]: pgmap v14340: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 436 KiB/s rd, 412 KiB/s wr, 12 op/s 2023-03-09T22:10:20.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:10:19 smithi181 ceph-mon[94189]: pgmap v14340: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 436 KiB/s rd, 412 KiB/s wr, 12 op/s 2023-03-09T22:10:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:10:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25931.355842 secs 2023-03-09T22:10:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:10:21 smithi122 ceph-mon[93869]: pgmap v14341: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 88 KiB/s wr, 10 op/s 2023-03-09T22:10:22.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:10:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25931.355842 secs 2023-03-09T22:10:22.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:10:21 smithi181 ceph-mon[94189]: pgmap v14341: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 88 KiB/s wr, 10 op/s 2023-03-09T22:10:22.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:10:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25931.355842 secs 2023-03-09T22:10:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:10:21 smithi038 ceph-mon[92422]: pgmap v14341: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 88 KiB/s wr, 10 op/s 2023-03-09T22:10:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26395.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:10:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:10:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25933.717159 secs 2023-03-09T22:10:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:10:23 smithi122 ceph-mon[93869]: pgmap v14342: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 90 KiB/s wr, 9 op/s 2023-03-09T22:10:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:10:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25933.717159 secs 2023-03-09T22:10:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:10:23 smithi181 ceph-mon[94189]: pgmap v14342: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 90 KiB/s wr, 9 op/s 2023-03-09T22:10:24.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:10:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25933.717159 secs 2023-03-09T22:10:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:10:23 smithi038 ceph-mon[92422]: pgmap v14342: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 90 KiB/s wr, 9 op/s 2023-03-09T22:10:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:10:25 smithi122 ceph-mon[93869]: pgmap v14343: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 89 KiB/s wr, 8 op/s 2023-03-09T22:10:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:10:25 smithi181 ceph-mon[94189]: pgmap v14343: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 89 KiB/s wr, 8 op/s 2023-03-09T22:10:26.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:10:25 smithi038 ceph-mon[92422]: pgmap v14343: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 89 KiB/s wr, 8 op/s 2023-03-09T22:10:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:10:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25936.355901 secs 2023-03-09T22:10:27.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:10:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25936.355901 secs 2023-03-09T22:10:27.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:10:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25936.355901 secs 2023-03-09T22:10:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:10:27 smithi122 ceph-mon[93869]: pgmap v14344: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 144 KiB/s wr, 7 op/s 2023-03-09T22:10:28.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:10:27 smithi181 ceph-mon[94189]: pgmap v14344: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 144 KiB/s wr, 7 op/s 2023-03-09T22:10:28.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:10:27 smithi038 ceph-mon[92422]: pgmap v14344: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 144 KiB/s wr, 7 op/s 2023-03-09T22:10:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26400.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:10:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:10:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25938.717283 secs 2023-03-09T22:10:29.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:10:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25938.717283 secs 2023-03-09T22:10:29.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:10:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25938.717283 secs 2023-03-09T22:10:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:10:29 smithi122 ceph-mon[93869]: pgmap v14345: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 130 KiB/s wr, 6 op/s 2023-03-09T22:10:30.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:10:29 smithi181 ceph-mon[94189]: pgmap v14345: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 130 KiB/s wr, 6 op/s 2023-03-09T22:10:30.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:10:29 smithi038 ceph-mon[92422]: pgmap v14345: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 130 KiB/s wr, 6 op/s 2023-03-09T22:10:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:10:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25941.355962 secs 2023-03-09T22:10:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:10:31 smithi122 ceph-mon[93869]: pgmap v14346: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 130 KiB/s wr, 8 op/s 2023-03-09T22:10:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:10:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25941.355962 secs 2023-03-09T22:10:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:10:31 smithi181 ceph-mon[94189]: pgmap v14346: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 130 KiB/s wr, 8 op/s 2023-03-09T22:10:32.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:10:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25941.355962 secs 2023-03-09T22:10:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:10:31 smithi038 ceph-mon[92422]: pgmap v14346: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 130 KiB/s wr, 8 op/s 2023-03-09T22:10:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:10:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:10:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:10:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:10:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:10:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:10:33.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:10:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:10:33.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:10:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:10:33.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:10:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:10:33.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:10:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:10:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:10:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:10:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:10:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:10:33.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26405.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:10:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:10:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25943.717358 secs 2023-03-09T22:10:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:10:33 smithi122 ceph-mon[93869]: pgmap v14347: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 109 KiB/s wr, 7 op/s 2023-03-09T22:10:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:10:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:10:34.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:10:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:10:34.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:10:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:10:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:10:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25943.717358 secs 2023-03-09T22:10:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:10:33 smithi181 ceph-mon[94189]: pgmap v14347: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 109 KiB/s wr, 7 op/s 2023-03-09T22:10:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:10:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:10:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:10:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:10:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:10:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:10:34.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:10:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25943.717358 secs 2023-03-09T22:10:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:10:33 smithi038 ceph-mon[92422]: pgmap v14347: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 109 KiB/s wr, 7 op/s 2023-03-09T22:10:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:10:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:10:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:10:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:10:34.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:10:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:10:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:10:35 smithi122 ceph-mon[93869]: pgmap v14348: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 108 KiB/s wr, 6 op/s 2023-03-09T22:10:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:10:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25946.356030 secs 2023-03-09T22:10:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:10:35 smithi181 ceph-mon[94189]: pgmap v14348: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 108 KiB/s wr, 6 op/s 2023-03-09T22:10:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:10:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25946.356030 secs 2023-03-09T22:10:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:10:35 smithi038 ceph-mon[92422]: pgmap v14348: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 108 KiB/s wr, 6 op/s 2023-03-09T22:10:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:10:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25946.356030 secs 2023-03-09T22:10:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:10:37 smithi122 ceph-mon[93869]: pgmap v14349: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 108 KiB/s wr, 9 op/s 2023-03-09T22:10:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:10:37 smithi181 ceph-mon[94189]: pgmap v14349: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 108 KiB/s wr, 9 op/s 2023-03-09T22:10:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:10:37 smithi038 ceph-mon[92422]: pgmap v14349: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 108 KiB/s wr, 9 op/s 2023-03-09T22:10:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26410.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:10:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:10:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25948.717483 secs 2023-03-09T22:10:39.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:10:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25948.717483 secs 2023-03-09T22:10:39.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:10:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25948.717483 secs 2023-03-09T22:10:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:10:39 smithi122 ceph-mon[93869]: pgmap v14350: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 6 op/s 2023-03-09T22:10:40.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:10:39 smithi181 ceph-mon[94189]: pgmap v14350: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 6 op/s 2023-03-09T22:10:40.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:10:39 smithi038 ceph-mon[92422]: pgmap v14350: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 6 op/s 2023-03-09T22:10:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:10:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25951.356089 secs 2023-03-09T22:10:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:10:41 smithi122 ceph-mon[93869]: pgmap v14351: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 119 KiB/s wr, 8 op/s 2023-03-09T22:10:42.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:10:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25951.356089 secs 2023-03-09T22:10:42.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:10:41 smithi181 ceph-mon[94189]: pgmap v14351: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 119 KiB/s wr, 8 op/s 2023-03-09T22:10:42.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:10:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25951.356089 secs 2023-03-09T22:10:42.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:10:41 smithi038 ceph-mon[92422]: pgmap v14351: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 119 KiB/s wr, 8 op/s 2023-03-09T22:10:43.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26415.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:10:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:10:43 smithi122 ceph-mon[93869]: pgmap v14352: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 119 KiB/s wr, 7 op/s 2023-03-09T22:10:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:10:43 smithi181 ceph-mon[94189]: pgmap v14352: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 119 KiB/s wr, 7 op/s 2023-03-09T22:10:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:10:43 smithi038 ceph-mon[92422]: pgmap v14352: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 119 KiB/s wr, 7 op/s 2023-03-09T22:10:45.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:10:44 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:10:45.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:10:44 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:10:45.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:10:44 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:10:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:10:45 smithi122 ceph-mon[93869]: pgmap v14353: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 119 KiB/s wr, 6 op/s 2023-03-09T22:10:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:10:45 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:10:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:10:45 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:10:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:10:45 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:10:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:10:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25956.356151 secs 2023-03-09T22:10:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:10:45 smithi181 ceph-mon[94189]: pgmap v14353: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 119 KiB/s wr, 6 op/s 2023-03-09T22:10:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:10:45 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:10:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:10:45 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:10:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:10:45 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:10:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:10:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25956.356151 secs 2023-03-09T22:10:46.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:10:45 smithi038 ceph-mon[92422]: pgmap v14353: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 119 KiB/s wr, 6 op/s 2023-03-09T22:10:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:10:45 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:10:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:10:45 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:10:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:10:45 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:10:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:10:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25956.356151 secs 2023-03-09T22:10:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:10:47 smithi122 ceph-mon[93869]: pgmap v14354: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 259 KiB/s wr, 9 op/s 2023-03-09T22:10:48.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:10:47 smithi181 ceph-mon[94189]: pgmap v14354: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 259 KiB/s wr, 9 op/s 2023-03-09T22:10:48.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:10:47 smithi038 ceph-mon[92422]: pgmap v14354: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 259 KiB/s wr, 9 op/s 2023-03-09T22:10:48.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26420.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:10:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:10:49 smithi122 ceph-mon[93869]: pgmap v14355: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 259 KiB/s wr, 7 op/s 2023-03-09T22:10:50.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:10:49 smithi181 ceph-mon[94189]: pgmap v14355: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 259 KiB/s wr, 7 op/s 2023-03-09T22:10:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:10:49 smithi038 ceph-mon[92422]: pgmap v14355: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 259 KiB/s wr, 7 op/s 2023-03-09T22:10:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:10:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25961.356212 secs 2023-03-09T22:10:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:10:51 smithi122 ceph-mon[93869]: pgmap v14356: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 323 KiB/s wr, 9 op/s 2023-03-09T22:10:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:10:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25961.356212 secs 2023-03-09T22:10:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:10:51 smithi181 ceph-mon[94189]: pgmap v14356: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 323 KiB/s wr, 9 op/s 2023-03-09T22:10:52.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:10:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25961.356212 secs 2023-03-09T22:10:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:10:51 smithi038 ceph-mon[92422]: pgmap v14356: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 323 KiB/s wr, 9 op/s 2023-03-09T22:10:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26425.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:10:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:10:53 smithi122 ceph-mon[93869]: pgmap v14357: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 295 KiB/s wr, 9 op/s 2023-03-09T22:10:54.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:10:53 smithi181 ceph-mon[94189]: pgmap v14357: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 295 KiB/s wr, 9 op/s 2023-03-09T22:10:54.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:10:53 smithi038 ceph-mon[92422]: pgmap v14357: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 295 KiB/s wr, 9 op/s 2023-03-09T22:10:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:10:55 smithi122 ceph-mon[93869]: pgmap v14358: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 310 KiB/s rd, 295 KiB/s wr, 9 op/s 2023-03-09T22:10:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:10:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25966.356274 secs 2023-03-09T22:10:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:10:55 smithi181 ceph-mon[94189]: pgmap v14358: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 310 KiB/s rd, 295 KiB/s wr, 9 op/s 2023-03-09T22:10:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:10:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25966.356274 secs 2023-03-09T22:10:56.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:10:55 smithi038 ceph-mon[92422]: pgmap v14358: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 310 KiB/s rd, 295 KiB/s wr, 9 op/s 2023-03-09T22:10:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:10:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25966.356274 secs 2023-03-09T22:10:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:10:57 smithi122 ceph-mon[93869]: pgmap v14359: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 316 KiB/s wr, 11 op/s 2023-03-09T22:10:58.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:10:57 smithi181 ceph-mon[94189]: pgmap v14359: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 316 KiB/s wr, 11 op/s 2023-03-09T22:10:58.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:10:57 smithi038 ceph-mon[92422]: pgmap v14359: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 316 KiB/s wr, 11 op/s 2023-03-09T22:10:58.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26430.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:11:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:10:59 smithi122 ceph-mon[93869]: pgmap v14360: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 442 KiB/s rd, 427 KiB/s wr, 10 op/s 2023-03-09T22:11:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:10:59 smithi181 ceph-mon[94189]: pgmap v14360: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 442 KiB/s rd, 427 KiB/s wr, 10 op/s 2023-03-09T22:11:00.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:10:59 smithi038 ceph-mon[92422]: pgmap v14360: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 442 KiB/s rd, 427 KiB/s wr, 10 op/s 2023-03-09T22:11:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25971.356336 secs 2023-03-09T22:11:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:01 smithi122 ceph-mon[93869]: pgmap v14361: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 499 KiB/s rd, 483 KiB/s wr, 12 op/s 2023-03-09T22:11:02.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25971.356336 secs 2023-03-09T22:11:02.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:01 smithi181 ceph-mon[94189]: pgmap v14361: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 499 KiB/s rd, 483 KiB/s wr, 12 op/s 2023-03-09T22:11:02.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25971.356336 secs 2023-03-09T22:11:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:01 smithi038 ceph-mon[92422]: pgmap v14361: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 499 KiB/s rd, 483 KiB/s wr, 12 op/s 2023-03-09T22:11:03.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26435.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:11:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:03 smithi122 ceph-mon[93869]: pgmap v14362: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 420 KiB/s wr, 11 op/s 2023-03-09T22:11:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:03 smithi181 ceph-mon[94189]: pgmap v14362: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 420 KiB/s wr, 11 op/s 2023-03-09T22:11:04.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:03 smithi038 ceph-mon[92422]: pgmap v14362: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 420 KiB/s wr, 11 op/s 2023-03-09T22:11:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:05 smithi122 ceph-mon[93869]: pgmap v14363: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 633 KiB/s rd, 625 KiB/s wr, 21 op/s 2023-03-09T22:11:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25976.356434 secs 2023-03-09T22:11:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:05 smithi181 ceph-mon[94189]: pgmap v14363: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 633 KiB/s rd, 625 KiB/s wr, 21 op/s 2023-03-09T22:11:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25976.356434 secs 2023-03-09T22:11:06.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:05 smithi038 ceph-mon[92422]: pgmap v14363: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 633 KiB/s rd, 625 KiB/s wr, 21 op/s 2023-03-09T22:11:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25976.356434 secs 2023-03-09T22:11:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:07 smithi122 ceph-mon[93869]: pgmap v14364: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 644 KiB/s rd, 656 KiB/s wr, 24 op/s 2023-03-09T22:11:08.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:07 smithi181 ceph-mon[94189]: pgmap v14364: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 644 KiB/s rd, 656 KiB/s wr, 24 op/s 2023-03-09T22:11:08.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:07 smithi038 ceph-mon[92422]: pgmap v14364: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 644 KiB/s rd, 656 KiB/s wr, 24 op/s 2023-03-09T22:11:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26440.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:11:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:08 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 25978.718231 secs 2023-03-09T22:11:09.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:08 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 25978.718231 secs 2023-03-09T22:11:09.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:08 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 25978.718231 secs 2023-03-09T22:11:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:09 smithi122 ceph-mon[93869]: pgmap v14365: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 792 KiB/s rd, 805 KiB/s wr, 23 op/s 2023-03-09T22:11:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:09 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:11:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:11:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:11:10.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:09 smithi181 ceph-mon[94189]: pgmap v14365: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 792 KiB/s rd, 805 KiB/s wr, 23 op/s 2023-03-09T22:11:10.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:09 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:11:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:11:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:11:10.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:09 smithi038 ceph-mon[92422]: pgmap v14365: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 792 KiB/s rd, 805 KiB/s wr, 23 op/s 2023-03-09T22:11:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:09 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:11:10.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:11:10.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:11:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25981.356517 secs 2023-03-09T22:11:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:11 smithi122 ceph-mon[93869]: pgmap v14366: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 542 KiB/s rd, 555 KiB/s wr, 24 op/s 2023-03-09T22:11:12.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25981.356517 secs 2023-03-09T22:11:12.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:11 smithi181 ceph-mon[94189]: pgmap v14366: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 542 KiB/s rd, 555 KiB/s wr, 24 op/s 2023-03-09T22:11:12.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25981.356517 secs 2023-03-09T22:11:12.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:11 smithi038 ceph-mon[92422]: pgmap v14366: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 542 KiB/s rd, 555 KiB/s wr, 24 op/s 2023-03-09T22:11:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26445.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:11:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:13 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 25983.718344 secs 2023-03-09T22:11:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:13 smithi122 ceph-mon[93869]: pgmap v14367: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 560 KiB/s rd, 572 KiB/s wr, 23 op/s 2023-03-09T22:11:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:13 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 25983.718344 secs 2023-03-09T22:11:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:13 smithi181 ceph-mon[94189]: pgmap v14367: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 560 KiB/s rd, 572 KiB/s wr, 23 op/s 2023-03-09T22:11:14.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:13 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 25983.718344 secs 2023-03-09T22:11:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:13 smithi038 ceph-mon[92422]: pgmap v14367: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 560 KiB/s rd, 572 KiB/s wr, 23 op/s 2023-03-09T22:11:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:15 smithi122 ceph-mon[93869]: pgmap v14368: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 559 KiB/s rd, 572 KiB/s wr, 21 op/s 2023-03-09T22:11:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25986.356579 secs 2023-03-09T22:11:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:15 smithi181 ceph-mon[94189]: pgmap v14368: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 559 KiB/s rd, 572 KiB/s wr, 21 op/s 2023-03-09T22:11:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25986.356579 secs 2023-03-09T22:11:16.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:15 smithi038 ceph-mon[92422]: pgmap v14368: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 559 KiB/s rd, 572 KiB/s wr, 21 op/s 2023-03-09T22:11:16.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25986.356579 secs 2023-03-09T22:11:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:17 smithi122 ceph-mon[93869]: pgmap v14369: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 296 KiB/s wr, 12 op/s 2023-03-09T22:11:18.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:17 smithi181 ceph-mon[94189]: pgmap v14369: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 296 KiB/s wr, 12 op/s 2023-03-09T22:11:18.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:17 smithi038 ceph-mon[92422]: pgmap v14369: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 296 KiB/s wr, 12 op/s 2023-03-09T22:11:18.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26450.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:11:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:18 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 25988.718422 secs 2023-03-09T22:11:19.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:18 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 25988.718422 secs 2023-03-09T22:11:19.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:18 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 25988.718422 secs 2023-03-09T22:11:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:19 smithi122 ceph-mon[93869]: pgmap v14370: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 560 KiB/s rd, 363 KiB/s wr, 9 op/s 2023-03-09T22:11:20.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:19 smithi181 ceph-mon[94189]: pgmap v14370: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 560 KiB/s rd, 363 KiB/s wr, 9 op/s 2023-03-09T22:11:20.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:19 smithi038 ceph-mon[92422]: pgmap v14370: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 560 KiB/s rd, 363 KiB/s wr, 9 op/s 2023-03-09T22:11:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25991.356641 secs 2023-03-09T22:11:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:21 smithi122 ceph-mon[93869]: pgmap v14371: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 214 KiB/s wr, 8 op/s 2023-03-09T22:11:22.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25991.356641 secs 2023-03-09T22:11:22.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:21 smithi181 ceph-mon[94189]: pgmap v14371: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 214 KiB/s wr, 8 op/s 2023-03-09T22:11:22.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25991.356641 secs 2023-03-09T22:11:22.334 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:21 smithi038 ceph-mon[92422]: pgmap v14371: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 214 KiB/s wr, 8 op/s 2023-03-09T22:11:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26455.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:11:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25993.718506 secs 2023-03-09T22:11:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:23 smithi122 ceph-mon[93869]: pgmap v14372: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 706 KiB/s rd, 225 KiB/s wr, 9 op/s 2023-03-09T22:11:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25993.718506 secs 2023-03-09T22:11:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:23 smithi181 ceph-mon[94189]: pgmap v14372: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 706 KiB/s rd, 225 KiB/s wr, 9 op/s 2023-03-09T22:11:24.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25993.718506 secs 2023-03-09T22:11:24.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:23 smithi038 ceph-mon[92422]: pgmap v14372: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 706 KiB/s rd, 225 KiB/s wr, 9 op/s 2023-03-09T22:11:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:25 smithi122 ceph-mon[93869]: pgmap v14373: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 758 KiB/s rd, 288 KiB/s wr, 10 op/s 2023-03-09T22:11:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:25 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:11:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25996.356700 secs 2023-03-09T22:11:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:25 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:11:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:25 smithi181 ceph-mon[94189]: pgmap v14373: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 758 KiB/s rd, 288 KiB/s wr, 10 op/s 2023-03-09T22:11:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:25 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:11:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25996.356700 secs 2023-03-09T22:11:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:25 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:11:26.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:25 smithi038 ceph-mon[92422]: pgmap v14373: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 758 KiB/s rd, 288 KiB/s wr, 10 op/s 2023-03-09T22:11:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:25 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:11:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25996.356700 secs 2023-03-09T22:11:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:25 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:11:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:27 smithi122 ceph-mon[93869]: pgmap v14374: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 762 KiB/s rd, 290 KiB/s wr, 12 op/s 2023-03-09T22:11:28.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:27 smithi181 ceph-mon[94189]: pgmap v14374: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 762 KiB/s rd, 290 KiB/s wr, 12 op/s 2023-03-09T22:11:28.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:27 smithi038 ceph-mon[92422]: pgmap v14374: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 762 KiB/s rd, 290 KiB/s wr, 12 op/s 2023-03-09T22:11:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26460.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:11:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 25998.718645 secs 2023-03-09T22:11:29.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 25998.718645 secs 2023-03-09T22:11:29.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 25998.718645 secs 2023-03-09T22:11:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:29 smithi122 ceph-mon[93869]: pgmap v14375: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 741 KiB/s rd, 270 KiB/s wr, 10 op/s 2023-03-09T22:11:30.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:29 smithi181 ceph-mon[94189]: pgmap v14375: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 741 KiB/s rd, 270 KiB/s wr, 10 op/s 2023-03-09T22:11:30.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:29 smithi038 ceph-mon[92422]: pgmap v14375: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 741 KiB/s rd, 270 KiB/s wr, 10 op/s 2023-03-09T22:11:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26001.356769 secs 2023-03-09T22:11:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:30 smithi122 ceph-mon[93869]: osdmap e1111: 12 total, 12 up, 12 in 2023-03-09T22:11:31.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26001.356769 secs 2023-03-09T22:11:31.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:30 smithi181 ceph-mon[94189]: osdmap e1111: 12 total, 12 up, 12 in 2023-03-09T22:11:31.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26001.356769 secs 2023-03-09T22:11:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:30 smithi038 ceph-mon[92422]: osdmap e1111: 12 total, 12 up, 12 in 2023-03-09T22:11:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:31 smithi122 ceph-mon[93869]: pgmap v14377: 129 pgs: 7 active+clean+snaptrim_wait, 122 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 811 KiB/s rd, 482 KiB/s wr, 13 op/s 2023-03-09T22:11:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:31 smithi181 ceph-mon[94189]: pgmap v14377: 129 pgs: 7 active+clean+snaptrim_wait, 122 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 811 KiB/s rd, 482 KiB/s wr, 13 op/s 2023-03-09T22:11:32.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:31 smithi038 ceph-mon[92422]: pgmap v14377: 129 pgs: 7 active+clean+snaptrim_wait, 122 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 811 KiB/s rd, 482 KiB/s wr, 13 op/s 2023-03-09T22:11:33.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26465.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:11:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26003.718784 secs 2023-03-09T22:11:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:33 smithi122 ceph-mon[93869]: pgmap v14378: 129 pgs: 7 active+clean+snaptrim_wait, 122 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 469 KiB/s wr, 12 op/s 2023-03-09T22:11:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:11:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26003.718784 secs 2023-03-09T22:11:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:33 smithi181 ceph-mon[94189]: pgmap v14378: 129 pgs: 7 active+clean+snaptrim_wait, 122 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 469 KiB/s wr, 12 op/s 2023-03-09T22:11:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:11:34.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26003.718784 secs 2023-03-09T22:11:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:33 smithi038 ceph-mon[92422]: pgmap v14378: 129 pgs: 7 active+clean+snaptrim_wait, 122 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 469 KiB/s wr, 12 op/s 2023-03-09T22:11:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:11:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:11:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:11:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:11:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:11:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:11:35.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:11:35.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:11:35.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:11:35.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:11:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:11:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:11:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:11:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:11:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:11:35.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:11:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:11:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:11:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:11:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:11:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:11:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:11:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:36 smithi122 ceph-mon[93869]: pgmap v14379: 129 pgs: 7 active+clean+snaptrim_wait, 122 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 305 KiB/s wr, 10 op/s 2023-03-09T22:11:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26006.356829 secs 2023-03-09T22:11:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:36 smithi122 ceph-mon[93869]: osdmap e1112: 12 total, 12 up, 12 in 2023-03-09T22:11:36.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:36 smithi181 ceph-mon[94189]: pgmap v14379: 129 pgs: 7 active+clean+snaptrim_wait, 122 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 305 KiB/s wr, 10 op/s 2023-03-09T22:11:36.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26006.356829 secs 2023-03-09T22:11:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:36 smithi181 ceph-mon[94189]: osdmap e1112: 12 total, 12 up, 12 in 2023-03-09T22:11:36.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:36 smithi038 ceph-mon[92422]: pgmap v14379: 129 pgs: 7 active+clean+snaptrim_wait, 122 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 305 KiB/s wr, 10 op/s 2023-03-09T22:11:36.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26006.356829 secs 2023-03-09T22:11:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:36 smithi038 ceph-mon[92422]: osdmap e1112: 12 total, 12 up, 12 in 2023-03-09T22:11:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:38 smithi122 ceph-mon[93869]: pgmap v14381: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 557 KiB/s rd, 554 KiB/s wr, 12 op/s 2023-03-09T22:11:38.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:38 smithi181 ceph-mon[94189]: pgmap v14381: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 557 KiB/s rd, 554 KiB/s wr, 12 op/s 2023-03-09T22:11:38.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:38 smithi038 ceph-mon[92422]: pgmap v14381: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 557 KiB/s rd, 554 KiB/s wr, 12 op/s 2023-03-09T22:11:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26470.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:11:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:39 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26008.718901 secs 2023-03-09T22:11:39.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:39 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26008.718901 secs 2023-03-09T22:11:39.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:39 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26008.718901 secs 2023-03-09T22:11:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:40 smithi122 ceph-mon[93869]: pgmap v14382: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 211 KiB/s wr, 10 op/s 2023-03-09T22:11:40.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:40 smithi181 ceph-mon[94189]: pgmap v14382: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 211 KiB/s wr, 10 op/s 2023-03-09T22:11:40.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:40 smithi038 ceph-mon[92422]: pgmap v14382: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 211 KiB/s wr, 10 op/s 2023-03-09T22:11:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26011.356890 secs 2023-03-09T22:11:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:41 smithi122 ceph-mon[93869]: osdmap e1113: 12 total, 12 up, 12 in 2023-03-09T22:11:41.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26011.356890 secs 2023-03-09T22:11:41.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:41 smithi181 ceph-mon[94189]: osdmap e1113: 12 total, 12 up, 12 in 2023-03-09T22:11:41.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26011.356890 secs 2023-03-09T22:11:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:41 smithi038 ceph-mon[92422]: osdmap e1113: 12 total, 12 up, 12 in 2023-03-09T22:11:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:42 smithi122 ceph-mon[93869]: pgmap v14383: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 286 KiB/s wr, 8 op/s 2023-03-09T22:11:42.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:42 smithi181 ceph-mon[94189]: pgmap v14383: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 286 KiB/s wr, 8 op/s 2023-03-09T22:11:42.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:42 smithi038 ceph-mon[92422]: pgmap v14383: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 286 KiB/s wr, 8 op/s 2023-03-09T22:11:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:43 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 26013.719033 secs 2023-03-09T22:11:43.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:43 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 26013.719033 secs 2023-03-09T22:11:43.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:43 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 26013.719033 secs 2023-03-09T22:11:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26475.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:11:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:44 smithi122 ceph-mon[93869]: pgmap v14385: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 357 KiB/s wr, 9 op/s 2023-03-09T22:11:44.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:44 smithi181 ceph-mon[94189]: pgmap v14385: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 357 KiB/s wr, 9 op/s 2023-03-09T22:11:44.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:44 smithi038 ceph-mon[92422]: pgmap v14385: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 357 KiB/s wr, 9 op/s 2023-03-09T22:11:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:46 smithi122 ceph-mon[93869]: pgmap v14386: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 164 KiB/s wr, 5 op/s 2023-03-09T22:11:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26016.356952 secs 2023-03-09T22:11:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:46 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:11:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:46 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:11:46.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:46 smithi181 ceph-mon[94189]: pgmap v14386: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 164 KiB/s wr, 5 op/s 2023-03-09T22:11:46.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26016.356952 secs 2023-03-09T22:11:46.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:46 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:11:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:46 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:11:46.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:46 smithi038 ceph-mon[92422]: pgmap v14386: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 164 KiB/s wr, 5 op/s 2023-03-09T22:11:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26016.356952 secs 2023-03-09T22:11:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:46 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:11:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:46 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:11:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:48 smithi122 ceph-mon[93869]: pgmap v14387: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 313 KiB/s wr, 10 op/s 2023-03-09T22:11:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26018.719164 secs 2023-03-09T22:11:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:48 smithi181 ceph-mon[94189]: pgmap v14387: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 313 KiB/s wr, 10 op/s 2023-03-09T22:11:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26018.719164 secs 2023-03-09T22:11:48.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:48 smithi038 ceph-mon[92422]: pgmap v14387: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 313 KiB/s wr, 10 op/s 2023-03-09T22:11:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26018.719164 secs 2023-03-09T22:11:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26480.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:11:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:50 smithi122 ceph-mon[93869]: pgmap v14388: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 313 KiB/s wr, 10 op/s 2023-03-09T22:11:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:50 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:11:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:50 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:11:50.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:50 smithi181 ceph-mon[94189]: pgmap v14388: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 313 KiB/s wr, 10 op/s 2023-03-09T22:11:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:50 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:11:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:50 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:11:50.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:50 smithi038 ceph-mon[92422]: pgmap v14388: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 313 KiB/s wr, 10 op/s 2023-03-09T22:11:50.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:50 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:11:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:50 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:11:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26021.356988 secs 2023-03-09T22:11:51.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26021.356988 secs 2023-03-09T22:11:51.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26021.356988 secs 2023-03-09T22:11:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:52 smithi122 ceph-mon[93869]: pgmap v14389: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 432 KiB/s rd, 427 KiB/s wr, 11 op/s 2023-03-09T22:11:52.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:52 smithi181 ceph-mon[94189]: pgmap v14389: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 432 KiB/s rd, 427 KiB/s wr, 11 op/s 2023-03-09T22:11:52.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:52 smithi038 ceph-mon[92422]: pgmap v14389: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 432 KiB/s rd, 427 KiB/s wr, 11 op/s 2023-03-09T22:11:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26023.719252 secs 2023-03-09T22:11:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:53 smithi122 ceph-mon[93869]: pgmap v14390: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 363 KiB/s wr, 10 op/s 2023-03-09T22:11:53.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26023.719252 secs 2023-03-09T22:11:53.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:53 smithi181 ceph-mon[94189]: pgmap v14390: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 363 KiB/s wr, 10 op/s 2023-03-09T22:11:53.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26023.719252 secs 2023-03-09T22:11:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:53 smithi038 ceph-mon[92422]: pgmap v14390: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 363 KiB/s wr, 10 op/s 2023-03-09T22:11:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26485.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:11:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:55 smithi122 ceph-mon[93869]: pgmap v14391: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 356 KiB/s wr, 10 op/s 2023-03-09T22:11:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:55 smithi181 ceph-mon[94189]: pgmap v14391: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 356 KiB/s wr, 10 op/s 2023-03-09T22:11:56.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:55 smithi038 ceph-mon[92422]: pgmap v14391: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 356 KiB/s wr, 10 op/s 2023-03-09T22:11:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26026.357049 secs 2023-03-09T22:11:57.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26026.357049 secs 2023-03-09T22:11:57.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26026.357049 secs 2023-03-09T22:11:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:57 smithi122 ceph-mon[93869]: pgmap v14392: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 685 KiB/s rd, 670 KiB/s wr, 14 op/s 2023-03-09T22:11:58.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:57 smithi181 ceph-mon[94189]: pgmap v14392: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 685 KiB/s rd, 670 KiB/s wr, 14 op/s 2023-03-09T22:11:58.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:57 smithi038 ceph-mon[92422]: pgmap v14392: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 685 KiB/s rd, 670 KiB/s wr, 14 op/s 2023-03-09T22:11:58.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26490.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:11:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26028.719370 secs 2023-03-09T22:11:59.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26028.719370 secs 2023-03-09T22:11:59.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26028.719370 secs 2023-03-09T22:12:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:11:59 smithi122 ceph-mon[93869]: pgmap v14393: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 529 KiB/s wr, 10 op/s 2023-03-09T22:12:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:11:59 smithi181 ceph-mon[94189]: pgmap v14393: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 529 KiB/s wr, 10 op/s 2023-03-09T22:12:00.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:11:59 smithi038 ceph-mon[92422]: pgmap v14393: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 529 KiB/s wr, 10 op/s 2023-03-09T22:12:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:12:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26031.357108 secs 2023-03-09T22:12:01.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:12:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26031.357108 secs 2023-03-09T22:12:01.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:12:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26031.357108 secs 2023-03-09T22:12:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:12:01 smithi122 ceph-mon[93869]: pgmap v14394: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 565 KiB/s rd, 550 KiB/s wr, 13 op/s 2023-03-09T22:12:02.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:12:01 smithi181 ceph-mon[94189]: pgmap v14394: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 565 KiB/s rd, 550 KiB/s wr, 13 op/s 2023-03-09T22:12:02.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:12:01 smithi038 ceph-mon[92422]: pgmap v14394: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 565 KiB/s rd, 550 KiB/s wr, 13 op/s 2023-03-09T22:12:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26495.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:12:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:12:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26033.719479 secs 2023-03-09T22:12:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:12:03 smithi122 ceph-mon[93869]: pgmap v14395: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 498 KiB/s wr, 11 op/s 2023-03-09T22:12:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:12:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26033.719479 secs 2023-03-09T22:12:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:12:03 smithi181 ceph-mon[94189]: pgmap v14395: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 498 KiB/s wr, 11 op/s 2023-03-09T22:12:04.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:12:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26033.719479 secs 2023-03-09T22:12:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:12:03 smithi038 ceph-mon[92422]: pgmap v14395: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 498 KiB/s wr, 11 op/s 2023-03-09T22:12:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:12:05 smithi122 ceph-mon[93869]: pgmap v14396: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 498 KiB/s wr, 12 op/s 2023-03-09T22:12:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:12:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26036.357170 secs 2023-03-09T22:12:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:12:05 smithi181 ceph-mon[94189]: pgmap v14396: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 498 KiB/s wr, 12 op/s 2023-03-09T22:12:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:12:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26036.357170 secs 2023-03-09T22:12:06.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:12:05 smithi038 ceph-mon[92422]: pgmap v14396: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 498 KiB/s wr, 12 op/s 2023-03-09T22:12:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:12:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26036.357170 secs 2023-03-09T22:12:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:12:07 smithi122 ceph-mon[93869]: pgmap v14397: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 966 KiB/s rd, 541 KiB/s wr, 16 op/s 2023-03-09T22:12:08.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:12:07 smithi181 ceph-mon[94189]: pgmap v14397: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 966 KiB/s rd, 541 KiB/s wr, 16 op/s 2023-03-09T22:12:08.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:12:07 smithi038 ceph-mon[92422]: pgmap v14397: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 966 KiB/s rd, 541 KiB/s wr, 16 op/s 2023-03-09T22:12:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26500.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:12:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:12:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26038.719615 secs 2023-03-09T22:12:09.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:12:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26038.719615 secs 2023-03-09T22:12:09.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:12:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26038.719615 secs 2023-03-09T22:12:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:12:09 smithi122 ceph-mon[93869]: pgmap v14398: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 658 KiB/s rd, 249 KiB/s wr, 23 op/s 2023-03-09T22:12:10.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:12:09 smithi181 ceph-mon[94189]: pgmap v14398: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 658 KiB/s rd, 249 KiB/s wr, 23 op/s 2023-03-09T22:12:10.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:12:09 smithi038 ceph-mon[92422]: pgmap v14398: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 658 KiB/s rd, 249 KiB/s wr, 23 op/s 2023-03-09T22:12:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:12:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26041.357230 secs 2023-03-09T22:12:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:12:10 smithi038 ceph-mon[92422]: osdmap e1114: 12 total, 12 up, 12 in 2023-03-09T22:12:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:12:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26041.357230 secs 2023-03-09T22:12:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:12:10 smithi122 ceph-mon[93869]: osdmap e1114: 12 total, 12 up, 12 in 2023-03-09T22:12:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:12:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26041.357230 secs 2023-03-09T22:12:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:12:10 smithi181 ceph-mon[94189]: osdmap e1114: 12 total, 12 up, 12 in 2023-03-09T22:12:12.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:12:11 smithi038 ceph-mon[92422]: pgmap v14400: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 699 KiB/s wr, 28 op/s 2023-03-09T22:12:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:12:11 smithi122 ceph-mon[93869]: pgmap v14400: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 699 KiB/s wr, 28 op/s 2023-03-09T22:12:12.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:12:11 smithi181 ceph-mon[94189]: pgmap v14400: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 699 KiB/s wr, 28 op/s 2023-03-09T22:12:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26505.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:12:14.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:12:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26043.719731 secs 2023-03-09T22:12:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:12:13 smithi038 ceph-mon[92422]: pgmap v14401: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 888 KiB/s rd, 503 KiB/s wr, 27 op/s 2023-03-09T22:12:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:12:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26043.719731 secs 2023-03-09T22:12:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:12:13 smithi122 ceph-mon[93869]: pgmap v14401: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 888 KiB/s rd, 503 KiB/s wr, 27 op/s 2023-03-09T22:12:14.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:12:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26043.719731 secs 2023-03-09T22:12:14.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:12:13 smithi181 ceph-mon[94189]: pgmap v14401: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 888 KiB/s rd, 503 KiB/s wr, 27 op/s 2023-03-09T22:12:16.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:12:15 smithi038 ceph-mon[92422]: pgmap v14402: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 643 KiB/s wr, 27 op/s 2023-03-09T22:12:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:12:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26046.357290 secs 2023-03-09T22:12:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:12:15 smithi038 ceph-mon[92422]: osdmap e1115: 12 total, 12 up, 12 in 2023-03-09T22:12:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:12:15 smithi122 ceph-mon[93869]: pgmap v14402: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 643 KiB/s wr, 27 op/s 2023-03-09T22:12:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:12:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26046.357290 secs 2023-03-09T22:12:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:12:15 smithi122 ceph-mon[93869]: osdmap e1115: 12 total, 12 up, 12 in 2023-03-09T22:12:16.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:12:15 smithi181 ceph-mon[94189]: pgmap v14402: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 643 KiB/s wr, 27 op/s 2023-03-09T22:12:16.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:12:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26046.357290 secs 2023-03-09T22:12:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:12:15 smithi181 ceph-mon[94189]: osdmap e1115: 12 total, 12 up, 12 in 2023-03-09T22:12:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:12:17 smithi122 ceph-mon[93869]: pgmap v14404: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 740 KiB/s rd, 738 KiB/s wr, 14 op/s 2023-03-09T22:12:18.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:12:17 smithi181 ceph-mon[94189]: pgmap v14404: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 740 KiB/s rd, 738 KiB/s wr, 14 op/s 2023-03-09T22:12:18.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:12:17 smithi038 ceph-mon[92422]: pgmap v14404: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 740 KiB/s rd, 738 KiB/s wr, 14 op/s 2023-03-09T22:12:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26510.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:12:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:12:19 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26048.719844 secs 2023-03-09T22:12:19.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:12:19 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26048.719844 secs 2023-03-09T22:12:19.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:12:19 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26048.719844 secs 2023-03-09T22:12:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:12:20 smithi122 ceph-mon[93869]: pgmap v14405: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 190 KiB/s wr, 10 op/s 2023-03-09T22:12:20.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:12:20 smithi181 ceph-mon[94189]: pgmap v14405: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 190 KiB/s wr, 10 op/s 2023-03-09T22:12:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:12:20 smithi038 ceph-mon[92422]: pgmap v14405: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 190 KiB/s wr, 10 op/s 2023-03-09T22:12:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:12:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26051.357353 secs 2023-03-09T22:12:21.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:12:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26051.357353 secs 2023-03-09T22:12:21.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:12:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26051.357353 secs 2023-03-09T22:12:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:12:22 smithi122 ceph-mon[93869]: pgmap v14406: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 318 KiB/s rd, 317 KiB/s wr, 10 op/s 2023-03-09T22:12:22.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:12:22 smithi181 ceph-mon[94189]: pgmap v14406: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 318 KiB/s rd, 317 KiB/s wr, 10 op/s 2023-03-09T22:12:22.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:12:22 smithi038 ceph-mon[92422]: pgmap v14406: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 318 KiB/s rd, 317 KiB/s wr, 10 op/s 2023-03-09T22:12:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26515.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:12:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:12:24 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26053.719966 secs 2023-03-09T22:12:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:12:24 smithi122 ceph-mon[93869]: pgmap v14407: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 318 KiB/s rd, 317 KiB/s wr, 10 op/s 2023-03-09T22:12:24.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:12:24 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26053.719966 secs 2023-03-09T22:12:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:12:24 smithi181 ceph-mon[94189]: pgmap v14407: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 318 KiB/s rd, 317 KiB/s wr, 10 op/s 2023-03-09T22:12:24.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:12:24 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26053.719966 secs 2023-03-09T22:12:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:12:24 smithi038 ceph-mon[92422]: pgmap v14407: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 318 KiB/s rd, 317 KiB/s wr, 10 op/s 2023-03-09T22:12:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:12:26 smithi122 ceph-mon[93869]: pgmap v14408: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 215 KiB/s wr, 9 op/s 2023-03-09T22:12:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:12:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26056.357416 secs 2023-03-09T22:12:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:12:26 smithi181 ceph-mon[94189]: pgmap v14408: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 215 KiB/s wr, 9 op/s 2023-03-09T22:12:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:12:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26056.357416 secs 2023-03-09T22:12:26.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:12:26 smithi038 ceph-mon[92422]: pgmap v14408: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 215 KiB/s wr, 9 op/s 2023-03-09T22:12:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:12:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26056.357416 secs 2023-03-09T22:12:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:12:28 smithi122 ceph-mon[93869]: pgmap v14409: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 177 KiB/s wr, 7 op/s 2023-03-09T22:12:28.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:12:28 smithi181 ceph-mon[94189]: pgmap v14409: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 177 KiB/s wr, 7 op/s 2023-03-09T22:12:28.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:12:28 smithi038 ceph-mon[92422]: pgmap v14409: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 177 KiB/s wr, 7 op/s 2023-03-09T22:12:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26520.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:12:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:12:29 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26058.720125 secs 2023-03-09T22:12:29.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:12:29 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26058.720125 secs 2023-03-09T22:12:29.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:12:29 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26058.720125 secs 2023-03-09T22:12:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:12:30 smithi122 ceph-mon[93869]: pgmap v14410: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 159 KiB/s wr, 6 op/s 2023-03-09T22:12:30.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:12:30 smithi181 ceph-mon[94189]: pgmap v14410: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 159 KiB/s wr, 6 op/s 2023-03-09T22:12:30.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:12:30 smithi038 ceph-mon[92422]: pgmap v14410: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 159 KiB/s wr, 6 op/s 2023-03-09T22:12:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:12:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26061.357475 secs 2023-03-09T22:12:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:12:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26061.357475 secs 2023-03-09T22:12:31.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:12:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26061.357475 secs 2023-03-09T22:12:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:12:32 smithi122 ceph-mon[93869]: pgmap v14411: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 342 KiB/s wr, 9 op/s 2023-03-09T22:12:32.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:12:32 smithi181 ceph-mon[94189]: pgmap v14411: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 342 KiB/s wr, 9 op/s 2023-03-09T22:12:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:12:32 smithi038 ceph-mon[92422]: pgmap v14411: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 342 KiB/s wr, 9 op/s 2023-03-09T22:12:33.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26525.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:12:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:12:34 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26063.720211 secs 2023-03-09T22:12:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:12:34 smithi122 ceph-mon[93869]: pgmap v14412: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 216 KiB/s wr, 7 op/s 2023-03-09T22:12:34.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:12:34 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26063.720211 secs 2023-03-09T22:12:34.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:12:34 smithi181 ceph-mon[94189]: pgmap v14412: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 216 KiB/s wr, 7 op/s 2023-03-09T22:12:34.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:12:34 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26063.720211 secs 2023-03-09T22:12:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:12:34 smithi038 ceph-mon[92422]: pgmap v14412: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 216 KiB/s wr, 7 op/s 2023-03-09T22:12:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:12:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:12:35.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:12:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:12:35.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:12:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:12:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:12:36 smithi122 ceph-mon[93869]: pgmap v14413: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 215 KiB/s wr, 7 op/s 2023-03-09T22:12:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:12:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26066.357535 secs 2023-03-09T22:12:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:12:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:12:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:12:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:12:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:12:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:12:36.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:12:36 smithi181 ceph-mon[94189]: pgmap v14413: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 215 KiB/s wr, 7 op/s 2023-03-09T22:12:36.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:12:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26066.357535 secs 2023-03-09T22:12:36.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:12:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:12:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:12:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:12:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:12:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:12:36.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:12:36 smithi038 ceph-mon[92422]: pgmap v14413: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 215 KiB/s wr, 7 op/s 2023-03-09T22:12:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:12:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26066.357535 secs 2023-03-09T22:12:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:12:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:12:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:12:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:12:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:12:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:12:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:12:38 smithi122 ceph-mon[93869]: pgmap v14414: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 184 KiB/s wr, 9 op/s 2023-03-09T22:12:38.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:12:38 smithi181 ceph-mon[94189]: pgmap v14414: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 184 KiB/s wr, 9 op/s 2023-03-09T22:12:38.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:12:38 smithi038 ceph-mon[92422]: pgmap v14414: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 184 KiB/s wr, 9 op/s 2023-03-09T22:12:38.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26530.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:12:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:12:39 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26068.720322 secs 2023-03-09T22:12:39.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:12:39 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26068.720322 secs 2023-03-09T22:12:39.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:12:39 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26068.720322 secs 2023-03-09T22:12:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:12:40 smithi122 ceph-mon[93869]: pgmap v14415: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 183 KiB/s wr, 7 op/s 2023-03-09T22:12:40.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:12:40 smithi181 ceph-mon[94189]: pgmap v14415: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 183 KiB/s wr, 7 op/s 2023-03-09T22:12:40.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:12:40 smithi038 ceph-mon[92422]: pgmap v14415: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 183 KiB/s wr, 7 op/s 2023-03-09T22:12:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:12:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26071.357599 secs 2023-03-09T22:12:41.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:12:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26071.357599 secs 2023-03-09T22:12:41.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:12:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26071.357599 secs 2023-03-09T22:12:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:12:42 smithi122 ceph-mon[93869]: pgmap v14416: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 292 KiB/s wr, 9 op/s 2023-03-09T22:12:42.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:12:42 smithi181 ceph-mon[94189]: pgmap v14416: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 292 KiB/s wr, 9 op/s 2023-03-09T22:12:42.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:12:42 smithi038 ceph-mon[92422]: pgmap v14416: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 292 KiB/s wr, 9 op/s 2023-03-09T22:12:43.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26535.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:12:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:12:44 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26073.720476 secs 2023-03-09T22:12:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:12:44 smithi122 ceph-mon[93869]: pgmap v14417: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 109 KiB/s wr, 7 op/s 2023-03-09T22:12:44.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:12:44 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26073.720476 secs 2023-03-09T22:12:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:12:44 smithi181 ceph-mon[94189]: pgmap v14417: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 109 KiB/s wr, 7 op/s 2023-03-09T22:12:44.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:12:44 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26073.720476 secs 2023-03-09T22:12:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:12:44 smithi038 ceph-mon[92422]: pgmap v14417: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 109 KiB/s wr, 7 op/s 2023-03-09T22:12:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:12:46 smithi122 ceph-mon[93869]: pgmap v14418: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 617 KiB/s rd, 235 KiB/s wr, 7 op/s 2023-03-09T22:12:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:12:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26076.357659 secs 2023-03-09T22:12:46.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:12:46 smithi181 ceph-mon[94189]: pgmap v14418: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 617 KiB/s rd, 235 KiB/s wr, 7 op/s 2023-03-09T22:12:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:12:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26076.357659 secs 2023-03-09T22:12:46.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:12:46 smithi038 ceph-mon[92422]: pgmap v14418: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 617 KiB/s rd, 235 KiB/s wr, 7 op/s 2023-03-09T22:12:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:12:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26076.357659 secs 2023-03-09T22:12:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:12:48 smithi122 ceph-mon[93869]: pgmap v14419: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 741 KiB/s rd, 358 KiB/s wr, 9 op/s 2023-03-09T22:12:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:12:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26078.720595 secs 2023-03-09T22:12:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:12:48 smithi181 ceph-mon[94189]: pgmap v14419: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 741 KiB/s rd, 358 KiB/s wr, 9 op/s 2023-03-09T22:12:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:12:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26078.720595 secs 2023-03-09T22:12:48.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:12:48 smithi038 ceph-mon[92422]: pgmap v14419: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 741 KiB/s rd, 358 KiB/s wr, 9 op/s 2023-03-09T22:12:48.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:12:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26078.720595 secs 2023-03-09T22:12:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26540.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:12:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:12:50 smithi122 ceph-mon[93869]: pgmap v14420: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 740 KiB/s rd, 358 KiB/s wr, 7 op/s 2023-03-09T22:12:50.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:12:50 smithi181 ceph-mon[94189]: pgmap v14420: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 740 KiB/s rd, 358 KiB/s wr, 7 op/s 2023-03-09T22:12:50.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:12:50 smithi038 ceph-mon[92422]: pgmap v14420: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 740 KiB/s rd, 358 KiB/s wr, 7 op/s 2023-03-09T22:12:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:12:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26081.357719 secs 2023-03-09T22:12:51.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:12:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26081.357719 secs 2023-03-09T22:12:51.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:12:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26081.357719 secs 2023-03-09T22:12:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:12:52 smithi122 ceph-mon[93869]: pgmap v14421: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 986 KiB/s rd, 506 KiB/s wr, 11 op/s 2023-03-09T22:12:52.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:12:52 smithi181 ceph-mon[94189]: pgmap v14421: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 986 KiB/s rd, 506 KiB/s wr, 11 op/s 2023-03-09T22:12:52.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:12:52 smithi038 ceph-mon[92422]: pgmap v14421: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 986 KiB/s rd, 506 KiB/s wr, 11 op/s 2023-03-09T22:12:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:12:53 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:12:53.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:12:53 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:12:53.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:12:53 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:12:53.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26545.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:12:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:12:54 smithi122 ceph-mon[93869]: pgmap v14422: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 877 KiB/s rd, 397 KiB/s wr, 10 op/s 2023-03-09T22:12:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:12:54 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:12:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:12:54 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:12:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:12:54 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:12:54.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:12:54 smithi181 ceph-mon[94189]: pgmap v14422: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 877 KiB/s rd, 397 KiB/s wr, 10 op/s 2023-03-09T22:12:54.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:12:54 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:12:54.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:12:54 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:12:54.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:12:54 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:12:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:12:54 smithi038 ceph-mon[92422]: pgmap v14422: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 877 KiB/s rd, 397 KiB/s wr, 10 op/s 2023-03-09T22:12:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:12:54 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:12:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:12:54 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:12:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:12:54 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:12:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:12:56 smithi122 ceph-mon[93869]: pgmap v14423: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 908 KiB/s rd, 418 KiB/s wr, 11 op/s 2023-03-09T22:12:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:12:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26086.357778 secs 2023-03-09T22:12:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:12:56 smithi181 ceph-mon[94189]: pgmap v14423: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 908 KiB/s rd, 418 KiB/s wr, 11 op/s 2023-03-09T22:12:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:12:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26086.357778 secs 2023-03-09T22:12:56.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:12:56 smithi038 ceph-mon[92422]: pgmap v14423: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 908 KiB/s rd, 418 KiB/s wr, 11 op/s 2023-03-09T22:12:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:12:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26086.357778 secs 2023-03-09T22:12:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:12:58 smithi122 ceph-mon[93869]: pgmap v14424: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 602 KiB/s wr, 14 op/s 2023-03-09T22:12:58.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:12:58 smithi181 ceph-mon[94189]: pgmap v14424: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 602 KiB/s wr, 14 op/s 2023-03-09T22:12:58.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:12:58 smithi038 ceph-mon[92422]: pgmap v14424: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 602 KiB/s wr, 14 op/s 2023-03-09T22:12:58.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26550.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:13:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:00 smithi122 ceph-mon[93869]: pgmap v14425: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 589 KiB/s rd, 479 KiB/s wr, 11 op/s 2023-03-09T22:13:00.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:00 smithi181 ceph-mon[94189]: pgmap v14425: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 589 KiB/s rd, 479 KiB/s wr, 11 op/s 2023-03-09T22:13:00.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:00 smithi038 ceph-mon[92422]: pgmap v14425: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 589 KiB/s rd, 479 KiB/s wr, 11 op/s 2023-03-09T22:13:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26091.357841 secs 2023-03-09T22:13:01.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26091.357841 secs 2023-03-09T22:13:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26091.357841 secs 2023-03-09T22:13:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:02 smithi122 ceph-mon[93869]: pgmap v14426: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 610 KiB/s rd, 500 KiB/s wr, 13 op/s 2023-03-09T22:13:02.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:02 smithi181 ceph-mon[94189]: pgmap v14426: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 610 KiB/s rd, 500 KiB/s wr, 13 op/s 2023-03-09T22:13:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:02 smithi038 ceph-mon[92422]: pgmap v14426: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 610 KiB/s rd, 500 KiB/s wr, 13 op/s 2023-03-09T22:13:03.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26555.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:13:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:04 smithi122 ceph-mon[93869]: pgmap v14427: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 567 KiB/s rd, 476 KiB/s wr, 11 op/s 2023-03-09T22:13:04.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:04 smithi181 ceph-mon[94189]: pgmap v14427: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 567 KiB/s rd, 476 KiB/s wr, 11 op/s 2023-03-09T22:13:04.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:04 smithi038 ceph-mon[92422]: pgmap v14427: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 567 KiB/s rd, 476 KiB/s wr, 11 op/s 2023-03-09T22:13:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:05 smithi122 ceph-mon[93869]: pgmap v14428: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 594 KiB/s rd, 507 KiB/s wr, 22 op/s 2023-03-09T22:13:05.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:05 smithi181 ceph-mon[94189]: pgmap v14428: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 594 KiB/s rd, 507 KiB/s wr, 22 op/s 2023-03-09T22:13:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:05 smithi038 ceph-mon[92422]: pgmap v14428: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 594 KiB/s rd, 507 KiB/s wr, 22 op/s 2023-03-09T22:13:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26096.357902 secs 2023-03-09T22:13:06.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26096.357902 secs 2023-03-09T22:13:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26096.357902 secs 2023-03-09T22:13:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:07 smithi122 ceph-mon[93869]: pgmap v14429: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 833 KiB/s wr, 26 op/s 2023-03-09T22:13:07.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:07 smithi181 ceph-mon[94189]: pgmap v14429: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 833 KiB/s wr, 26 op/s 2023-03-09T22:13:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:07 smithi038 ceph-mon[92422]: pgmap v14429: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 833 KiB/s wr, 26 op/s 2023-03-09T22:13:08.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26560.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:13:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:09 smithi122 ceph-mon[93869]: pgmap v14430: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 523 KiB/s wr, 22 op/s 2023-03-09T22:13:10.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:09 smithi181 ceph-mon[94189]: pgmap v14430: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 523 KiB/s wr, 22 op/s 2023-03-09T22:13:10.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:09 smithi038 ceph-mon[92422]: pgmap v14430: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 523 KiB/s wr, 22 op/s 2023-03-09T22:13:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26101.357960 secs 2023-03-09T22:13:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:11 smithi122 ceph-mon[93869]: pgmap v14431: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 598 KiB/s wr, 24 op/s 2023-03-09T22:13:12.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26101.357960 secs 2023-03-09T22:13:12.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:11 smithi181 ceph-mon[94189]: pgmap v14431: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 598 KiB/s wr, 24 op/s 2023-03-09T22:13:12.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26101.357960 secs 2023-03-09T22:13:12.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:11 smithi038 ceph-mon[92422]: pgmap v14431: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 598 KiB/s wr, 24 op/s 2023-03-09T22:13:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26565.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:13:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:13 smithi122 ceph-mon[93869]: pgmap v14432: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 577 KiB/s wr, 23 op/s 2023-03-09T22:13:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:13 smithi181 ceph-mon[94189]: pgmap v14432: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 577 KiB/s wr, 23 op/s 2023-03-09T22:13:14.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:13 smithi038 ceph-mon[92422]: pgmap v14432: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 577 KiB/s wr, 23 op/s 2023-03-09T22:13:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:15 smithi122 ceph-mon[93869]: pgmap v14433: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 616 KiB/s rd, 473 KiB/s wr, 22 op/s 2023-03-09T22:13:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:15 smithi181 ceph-mon[94189]: pgmap v14433: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 616 KiB/s rd, 473 KiB/s wr, 22 op/s 2023-03-09T22:13:16.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:15 smithi038 ceph-mon[92422]: pgmap v14433: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 616 KiB/s rd, 473 KiB/s wr, 22 op/s 2023-03-09T22:13:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26106.357984 secs 2023-03-09T22:13:17.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26106.357984 secs 2023-03-09T22:13:17.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26106.357984 secs 2023-03-09T22:13:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:17 smithi122 ceph-mon[93869]: pgmap v14434: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 590 KiB/s rd, 442 KiB/s wr, 13 op/s 2023-03-09T22:13:18.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:17 smithi181 ceph-mon[94189]: pgmap v14434: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 590 KiB/s rd, 442 KiB/s wr, 13 op/s 2023-03-09T22:13:18.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:17 smithi038 ceph-mon[92422]: pgmap v14434: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 590 KiB/s rd, 442 KiB/s wr, 13 op/s 2023-03-09T22:13:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26570.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:13:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:19 smithi122 ceph-mon[93869]: pgmap v14435: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 96 KiB/s wr, 8 op/s 2023-03-09T22:13:20.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:19 smithi181 ceph-mon[94189]: pgmap v14435: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 96 KiB/s wr, 8 op/s 2023-03-09T22:13:20.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:19 smithi038 ceph-mon[92422]: pgmap v14435: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 96 KiB/s wr, 8 op/s 2023-03-09T22:13:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26111.358045 secs 2023-03-09T22:13:21.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26111.358045 secs 2023-03-09T22:13:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26111.358045 secs 2023-03-09T22:13:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:21 smithi122 ceph-mon[93869]: pgmap v14436: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 323 KiB/s wr, 10 op/s 2023-03-09T22:13:22.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:21 smithi181 ceph-mon[94189]: pgmap v14436: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 323 KiB/s wr, 10 op/s 2023-03-09T22:13:22.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:21 smithi038 ceph-mon[92422]: pgmap v14436: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 323 KiB/s wr, 10 op/s 2023-03-09T22:13:23.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26575.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:13:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26113.721388 secs 2023-03-09T22:13:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:23 smithi122 ceph-mon[93869]: pgmap v14437: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 259 KiB/s wr, 9 op/s 2023-03-09T22:13:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26113.721388 secs 2023-03-09T22:13:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:23 smithi181 ceph-mon[94189]: pgmap v14437: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 259 KiB/s wr, 9 op/s 2023-03-09T22:13:24.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26113.721388 secs 2023-03-09T22:13:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:23 smithi038 ceph-mon[92422]: pgmap v14437: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 259 KiB/s wr, 9 op/s 2023-03-09T22:13:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:25 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:13:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:25 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:13:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:25 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:13:25.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:25 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:13:25.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:25 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:13:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:25 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:13:25.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:25 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:13:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:25 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:13:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:25 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:13:26.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:26 smithi122 ceph-mon[93869]: pgmap v14438: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 291 KiB/s wr, 10 op/s 2023-03-09T22:13:26.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26116.358104 secs 2023-03-09T22:13:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:26 smithi181 ceph-mon[94189]: pgmap v14438: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 291 KiB/s wr, 10 op/s 2023-03-09T22:13:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26116.358104 secs 2023-03-09T22:13:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:26 smithi038 ceph-mon[92422]: pgmap v14438: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 291 KiB/s wr, 10 op/s 2023-03-09T22:13:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26116.358104 secs 2023-03-09T22:13:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:28 smithi122 ceph-mon[93869]: pgmap v14439: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 361 KiB/s wr, 13 op/s 2023-03-09T22:13:28.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:28 smithi181 ceph-mon[94189]: pgmap v14439: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 361 KiB/s wr, 13 op/s 2023-03-09T22:13:28.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:28 smithi038 ceph-mon[92422]: pgmap v14439: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 361 KiB/s wr, 13 op/s 2023-03-09T22:13:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26580.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:13:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:29 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26118.721502 secs 2023-03-09T22:13:29.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:29 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26118.721502 secs 2023-03-09T22:13:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:29 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26118.721502 secs 2023-03-09T22:13:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:30 smithi122 ceph-mon[93869]: pgmap v14440: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 361 KiB/s wr, 10 op/s 2023-03-09T22:13:30.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:30 smithi181 ceph-mon[94189]: pgmap v14440: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 361 KiB/s wr, 10 op/s 2023-03-09T22:13:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:30 smithi038 ceph-mon[92422]: pgmap v14440: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 361 KiB/s wr, 10 op/s 2023-03-09T22:13:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26121.358168 secs 2023-03-09T22:13:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:31 smithi122 ceph-mon[93869]: osdmap e1116: 12 total, 12 up, 12 in 2023-03-09T22:13:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26121.358168 secs 2023-03-09T22:13:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:31 smithi181 ceph-mon[94189]: osdmap e1116: 12 total, 12 up, 12 in 2023-03-09T22:13:31.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26121.358168 secs 2023-03-09T22:13:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:31 smithi038 ceph-mon[92422]: osdmap e1116: 12 total, 12 up, 12 in 2023-03-09T22:13:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:32 smithi122 ceph-mon[93869]: pgmap v14442: 129 pgs: 2 active+clean+snaptrim, 127 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 161 KiB/s wr, 11 op/s 2023-03-09T22:13:32.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:32 smithi181 ceph-mon[94189]: pgmap v14442: 129 pgs: 2 active+clean+snaptrim, 127 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 161 KiB/s wr, 11 op/s 2023-03-09T22:13:32.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:32 smithi038 ceph-mon[92422]: pgmap v14442: 129 pgs: 2 active+clean+snaptrim, 127 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 161 KiB/s wr, 11 op/s 2023-03-09T22:13:33.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26585.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:13:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:34 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26123.721641 secs 2023-03-09T22:13:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:34 smithi122 ceph-mon[93869]: pgmap v14443: 129 pgs: 2 active+clean+snaptrim, 127 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 310 KiB/s rd, 321 KiB/s wr, 12 op/s 2023-03-09T22:13:34.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:34 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26123.721641 secs 2023-03-09T22:13:34.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:34 smithi181 ceph-mon[94189]: pgmap v14443: 129 pgs: 2 active+clean+snaptrim, 127 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 310 KiB/s rd, 321 KiB/s wr, 12 op/s 2023-03-09T22:13:34.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:34 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26123.721641 secs 2023-03-09T22:13:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:34 smithi038 ceph-mon[92422]: pgmap v14443: 129 pgs: 2 active+clean+snaptrim, 127 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 310 KiB/s rd, 321 KiB/s wr, 12 op/s 2023-03-09T22:13:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:36 smithi122 ceph-mon[93869]: pgmap v14444: 129 pgs: 2 active+clean+snaptrim, 127 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 283 KiB/s wr, 10 op/s 2023-03-09T22:13:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26126.358229 secs 2023-03-09T22:13:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:13:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:36 smithi122 ceph-mon[93869]: osdmap e1117: 12 total, 12 up, 12 in 2023-03-09T22:13:36.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:36 smithi181 ceph-mon[94189]: pgmap v14444: 129 pgs: 2 active+clean+snaptrim, 127 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 283 KiB/s wr, 10 op/s 2023-03-09T22:13:36.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26126.358229 secs 2023-03-09T22:13:36.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:13:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:36 smithi181 ceph-mon[94189]: osdmap e1117: 12 total, 12 up, 12 in 2023-03-09T22:13:36.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:36 smithi038 ceph-mon[92422]: pgmap v14444: 129 pgs: 2 active+clean+snaptrim, 127 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 283 KiB/s wr, 10 op/s 2023-03-09T22:13:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26126.358229 secs 2023-03-09T22:13:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:13:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:36 smithi038 ceph-mon[92422]: osdmap e1117: 12 total, 12 up, 12 in 2023-03-09T22:13:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:13:37.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:13:37.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:13:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:13:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:13:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:13:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:13:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:13:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:13:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:38 smithi122 ceph-mon[93869]: pgmap v14446: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 397 KiB/s wr, 11 op/s 2023-03-09T22:13:38.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:38 smithi181 ceph-mon[94189]: pgmap v14446: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 397 KiB/s wr, 11 op/s 2023-03-09T22:13:38.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:38 smithi038 ceph-mon[92422]: pgmap v14446: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 397 KiB/s wr, 11 op/s 2023-03-09T22:13:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26590.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:13:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:39 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26128.721738 secs 2023-03-09T22:13:39.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:39 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26128.721738 secs 2023-03-09T22:13:39.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:39 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26128.721738 secs 2023-03-09T22:13:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:40 smithi122 ceph-mon[93869]: pgmap v14447: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 364 KiB/s wr, 10 op/s 2023-03-09T22:13:40.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:40 smithi181 ceph-mon[94189]: pgmap v14447: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 364 KiB/s wr, 10 op/s 2023-03-09T22:13:40.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:40 smithi038 ceph-mon[92422]: pgmap v14447: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 364 KiB/s wr, 10 op/s 2023-03-09T22:13:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26131.358288 secs 2023-03-09T22:13:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:41 smithi122 ceph-mon[93869]: Health check failed: 1 MDSs behind on trimming (MDS_TRIM) 2023-03-09T22:13:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:13:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:13:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:41 smithi122 ceph-mon[93869]: osdmap e1118: 12 total, 12 up, 12 in 2023-03-09T22:13:41.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26131.358288 secs 2023-03-09T22:13:41.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:41 smithi181 ceph-mon[94189]: Health check failed: 1 MDSs behind on trimming (MDS_TRIM) 2023-03-09T22:13:41.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:13:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:13:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:41 smithi181 ceph-mon[94189]: osdmap e1118: 12 total, 12 up, 12 in 2023-03-09T22:13:41.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26131.358288 secs 2023-03-09T22:13:41.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:41 smithi038 ceph-mon[92422]: Health check failed: 1 MDSs behind on trimming (MDS_TRIM) 2023-03-09T22:13:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:13:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:13:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:41 smithi038 ceph-mon[92422]: osdmap e1118: 12 total, 12 up, 12 in 2023-03-09T22:13:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:42 smithi122 ceph-mon[93869]: pgmap v14448: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 317 KiB/s wr, 9 op/s 2023-03-09T22:13:42.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:42 smithi181 ceph-mon[94189]: pgmap v14448: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 317 KiB/s wr, 9 op/s 2023-03-09T22:13:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:42 smithi038 ceph-mon[92422]: pgmap v14448: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 317 KiB/s wr, 9 op/s 2023-03-09T22:13:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26595.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:13:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:44 smithi122 ceph-mon[93869]: pgmap v14450: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 360 KiB/s wr, 10 op/s 2023-03-09T22:13:44.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:44 smithi181 ceph-mon[94189]: pgmap v14450: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 360 KiB/s wr, 10 op/s 2023-03-09T22:13:44.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:44 smithi038 ceph-mon[92422]: pgmap v14450: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 360 KiB/s wr, 10 op/s 2023-03-09T22:13:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:45 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:13:45.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:45 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:13:45.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:45 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:13:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:46 smithi122 ceph-mon[93869]: pgmap v14451: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 165 KiB/s wr, 6 op/s 2023-03-09T22:13:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26136.358349 secs 2023-03-09T22:13:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:46 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:13:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:46 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:13:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:46 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:13:46.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:46 smithi181 ceph-mon[94189]: pgmap v14451: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 165 KiB/s wr, 6 op/s 2023-03-09T22:13:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26136.358349 secs 2023-03-09T22:13:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:46 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:13:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:46 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:13:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:46 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:13:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:46 smithi038 ceph-mon[92422]: pgmap v14451: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 165 KiB/s wr, 6 op/s 2023-03-09T22:13:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26136.358349 secs 2023-03-09T22:13:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:46 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:13:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:46 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:13:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:46 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:13:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:47 smithi122 ceph-mon[93869]: pgmap v14452: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 146 KiB/s wr, 9 op/s 2023-03-09T22:13:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:47 smithi122 ceph-mon[93869]: Health check update: 2 MDSs behind on trimming (MDS_TRIM) 2023-03-09T22:13:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:13:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:13:47.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:47 smithi181 ceph-mon[94189]: pgmap v14452: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 146 KiB/s wr, 9 op/s 2023-03-09T22:13:47.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:47 smithi181 ceph-mon[94189]: Health check update: 2 MDSs behind on trimming (MDS_TRIM) 2023-03-09T22:13:47.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:13:47.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:13:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:47 smithi038 ceph-mon[92422]: pgmap v14452: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 146 KiB/s wr, 9 op/s 2023-03-09T22:13:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:47 smithi038 ceph-mon[92422]: Health check update: 2 MDSs behind on trimming (MDS_TRIM) 2023-03-09T22:13:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:13:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:13:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26600.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:13:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:49 smithi122 ceph-mon[93869]: pgmap v14453: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 291 KiB/s wr, 10 op/s 2023-03-09T22:13:50.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:49 smithi181 ceph-mon[94189]: pgmap v14453: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 291 KiB/s wr, 10 op/s 2023-03-09T22:13:50.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:49 smithi038 ceph-mon[92422]: pgmap v14453: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 291 KiB/s wr, 10 op/s 2023-03-09T22:13:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26141.358410 secs 2023-03-09T22:13:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:51 smithi122 ceph-mon[93869]: pgmap v14454: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 388 KiB/s wr, 10 op/s 2023-03-09T22:13:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26141.358410 secs 2023-03-09T22:13:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:51 smithi181 ceph-mon[94189]: pgmap v14454: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 388 KiB/s wr, 10 op/s 2023-03-09T22:13:52.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26141.358410 secs 2023-03-09T22:13:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:51 smithi038 ceph-mon[92422]: pgmap v14454: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 388 KiB/s wr, 10 op/s 2023-03-09T22:13:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26605.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:13:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:53 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 26143.722091 secs 2023-03-09T22:13:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:53 smithi122 ceph-mon[93869]: pgmap v14455: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 431 KiB/s wr, 9 op/s 2023-03-09T22:13:54.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:53 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 26143.722091 secs 2023-03-09T22:13:54.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:53 smithi181 ceph-mon[94189]: pgmap v14455: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 431 KiB/s wr, 9 op/s 2023-03-09T22:13:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:53 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 26143.722091 secs 2023-03-09T22:13:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:53 smithi038 ceph-mon[92422]: pgmap v14455: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 431 KiB/s wr, 9 op/s 2023-03-09T22:13:55.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:54 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:13:55.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:54 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:13:55.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:54 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:13:55.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:54 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:13:55.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:54 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:13:55.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:54 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:13:55.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:54 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:13:55.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:54 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:13:55.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:54 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:13:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:55 smithi122 ceph-mon[93869]: pgmap v14456: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 381 KiB/s wr, 10 op/s 2023-03-09T22:13:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:13:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:13:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:55 smithi181 ceph-mon[94189]: pgmap v14456: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 381 KiB/s wr, 10 op/s 2023-03-09T22:13:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:13:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:13:56.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:55 smithi038 ceph-mon[92422]: pgmap v14456: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 381 KiB/s wr, 10 op/s 2023-03-09T22:13:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:13:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:13:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26146.358480 secs 2023-03-09T22:13:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:13:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:13:57.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26146.358480 secs 2023-03-09T22:13:57.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:13:57.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:13:57.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26146.358480 secs 2023-03-09T22:13:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:13:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:13:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:57 smithi122 ceph-mon[93869]: pgmap v14457: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 781 KiB/s rd, 676 KiB/s wr, 13 op/s 2023-03-09T22:13:58.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:57 smithi181 ceph-mon[94189]: pgmap v14457: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 781 KiB/s rd, 676 KiB/s wr, 13 op/s 2023-03-09T22:13:58.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:57 smithi038 ceph-mon[92422]: pgmap v14457: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 781 KiB/s rd, 676 KiB/s wr, 13 op/s 2023-03-09T22:13:58.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26610.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:13:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:58 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 26148.722206 secs 2023-03-09T22:13:59.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:58 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 26148.722206 secs 2023-03-09T22:13:59.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:58 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 26148.722206 secs 2023-03-09T22:14:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:13:59 smithi122 ceph-mon[93869]: pgmap v14458: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 780 KiB/s rd, 676 KiB/s wr, 11 op/s 2023-03-09T22:14:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:13:59 smithi181 ceph-mon[94189]: pgmap v14458: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 780 KiB/s rd, 676 KiB/s wr, 11 op/s 2023-03-09T22:14:00.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:13:59 smithi038 ceph-mon[92422]: pgmap v14458: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 780 KiB/s rd, 676 KiB/s wr, 11 op/s 2023-03-09T22:14:01.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26151.358540 secs 2023-03-09T22:14:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:14:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:14:01.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26151.358540 secs 2023-03-09T22:14:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:14:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:14:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26151.358540 secs 2023-03-09T22:14:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:14:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:14:02.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:01 smithi038 ceph-mon[92422]: pgmap v14459: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 661 KiB/s rd, 555 KiB/s wr, 12 op/s 2023-03-09T22:14:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:01 smithi122 ceph-mon[93869]: pgmap v14459: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 661 KiB/s rd, 555 KiB/s wr, 12 op/s 2023-03-09T22:14:02.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:01 smithi181 ceph-mon[94189]: pgmap v14459: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 661 KiB/s rd, 555 KiB/s wr, 12 op/s 2023-03-09T22:14:03.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26615.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:14:04.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:03 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 26153.722342 secs 2023-03-09T22:14:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:03 smithi038 ceph-mon[92422]: pgmap v14460: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 550 KiB/s wr, 12 op/s 2023-03-09T22:14:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:03 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 26153.722342 secs 2023-03-09T22:14:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:03 smithi122 ceph-mon[93869]: pgmap v14460: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 550 KiB/s wr, 12 op/s 2023-03-09T22:14:04.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:03 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 26153.722342 secs 2023-03-09T22:14:04.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:03 smithi181 ceph-mon[94189]: pgmap v14460: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 550 KiB/s wr, 12 op/s 2023-03-09T22:14:06.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:05 smithi038 ceph-mon[92422]: pgmap v14461: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 508 KiB/s rd, 435 KiB/s wr, 12 op/s 2023-03-09T22:14:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26156.358602 secs 2023-03-09T22:14:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:05 smithi122 ceph-mon[93869]: pgmap v14461: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 508 KiB/s rd, 435 KiB/s wr, 12 op/s 2023-03-09T22:14:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26156.358602 secs 2023-03-09T22:14:06.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:05 smithi181 ceph-mon[94189]: pgmap v14461: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 508 KiB/s rd, 435 KiB/s wr, 12 op/s 2023-03-09T22:14:06.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26156.358602 secs 2023-03-09T22:14:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:07 smithi038 ceph-mon[92422]: pgmap v14462: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 772 KiB/s rd, 732 KiB/s wr, 16 op/s 2023-03-09T22:14:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:14:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:14:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:07 smithi122 ceph-mon[93869]: pgmap v14462: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 772 KiB/s rd, 732 KiB/s wr, 16 op/s 2023-03-09T22:14:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:14:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:14:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:07 smithi181 ceph-mon[94189]: pgmap v14462: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 772 KiB/s rd, 732 KiB/s wr, 16 op/s 2023-03-09T22:14:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:14:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:14:08.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26620.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:14:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:09 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26158.722484 secs 2023-03-09T22:14:09.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:09 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26158.722484 secs 2023-03-09T22:14:09.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:09 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26158.722484 secs 2023-03-09T22:14:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:10 smithi122 ceph-mon[93869]: pgmap v14463: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 457 KiB/s wr, 24 op/s 2023-03-09T22:14:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:10 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:14:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:10 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:14:10.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:10 smithi181 ceph-mon[94189]: pgmap v14463: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 457 KiB/s wr, 24 op/s 2023-03-09T22:14:10.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:10 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:14:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:10 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:14:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:10 smithi038 ceph-mon[92422]: pgmap v14463: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 457 KiB/s wr, 24 op/s 2023-03-09T22:14:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:10 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:14:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:10 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:14:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26161.358693 secs 2023-03-09T22:14:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:11 smithi122 ceph-mon[93869]: osdmap e1119: 12 total, 12 up, 12 in 2023-03-09T22:14:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26161.358693 secs 2023-03-09T22:14:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:11 smithi181 ceph-mon[94189]: osdmap e1119: 12 total, 12 up, 12 in 2023-03-09T22:14:11.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26161.358693 secs 2023-03-09T22:14:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:11 smithi038 ceph-mon[92422]: osdmap e1119: 12 total, 12 up, 12 in 2023-03-09T22:14:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:12 smithi122 ceph-mon[93869]: pgmap v14465: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 813 KiB/s rd, 889 KiB/s wr, 28 op/s 2023-03-09T22:14:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:14:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:14:12.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:12 smithi181 ceph-mon[94189]: pgmap v14465: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 813 KiB/s rd, 889 KiB/s wr, 28 op/s 2023-03-09T22:14:12.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:14:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:14:12.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:12 smithi038 ceph-mon[92422]: pgmap v14465: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 813 KiB/s rd, 889 KiB/s wr, 28 op/s 2023-03-09T22:14:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:14:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:14:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:14:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:14:13.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:14:13.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:14:13.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:14:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:14:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26625.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:14:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:14 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26163.722582 secs 2023-03-09T22:14:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:14 smithi122 ceph-mon[93869]: pgmap v14466: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 723 KiB/s rd, 799 KiB/s wr, 27 op/s 2023-03-09T22:14:14.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:14 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26163.722582 secs 2023-03-09T22:14:14.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:14 smithi181 ceph-mon[94189]: pgmap v14466: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 723 KiB/s rd, 799 KiB/s wr, 27 op/s 2023-03-09T22:14:14.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:14 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26163.722582 secs 2023-03-09T22:14:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:14 smithi038 ceph-mon[92422]: pgmap v14466: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 723 KiB/s rd, 799 KiB/s wr, 27 op/s 2023-03-09T22:14:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:16 smithi122 ceph-mon[93869]: pgmap v14467: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 722 KiB/s rd, 799 KiB/s wr, 26 op/s 2023-03-09T22:14:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26166.358750 secs 2023-03-09T22:14:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:16 smithi181 ceph-mon[94189]: pgmap v14467: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 722 KiB/s rd, 799 KiB/s wr, 26 op/s 2023-03-09T22:14:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26166.358750 secs 2023-03-09T22:14:16.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:16 smithi038 ceph-mon[92422]: pgmap v14467: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 722 KiB/s rd, 799 KiB/s wr, 26 op/s 2023-03-09T22:14:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26166.358750 secs 2023-03-09T22:14:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:18 smithi122 ceph-mon[93869]: pgmap v14468: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 411 KiB/s wr, 23 op/s 2023-03-09T22:14:18.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:18 smithi181 ceph-mon[94189]: pgmap v14468: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 411 KiB/s wr, 23 op/s 2023-03-09T22:14:18.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:18 smithi038 ceph-mon[92422]: pgmap v14468: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 411 KiB/s wr, 23 op/s 2023-03-09T22:14:18.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26630.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:14:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:19 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26168.722702 secs 2023-03-09T22:14:19.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:19 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26168.722702 secs 2023-03-09T22:14:19.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:19 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26168.722702 secs 2023-03-09T22:14:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:20 smithi122 ceph-mon[93869]: pgmap v14469: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 457 KiB/s wr, 11 op/s 2023-03-09T22:14:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:14:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:14:20.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:20 smithi181 ceph-mon[94189]: pgmap v14469: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 457 KiB/s wr, 11 op/s 2023-03-09T22:14:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:14:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:14:20.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:20 smithi038 ceph-mon[92422]: pgmap v14469: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 457 KiB/s wr, 11 op/s 2023-03-09T22:14:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:14:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:14:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26171.358811 secs 2023-03-09T22:14:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:14:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:14:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:21 smithi122 ceph-mon[93869]: osdmap e1120: 12 total, 12 up, 12 in 2023-03-09T22:14:21.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26171.358811 secs 2023-03-09T22:14:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:14:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:14:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:21 smithi181 ceph-mon[94189]: osdmap e1120: 12 total, 12 up, 12 in 2023-03-09T22:14:21.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26171.358811 secs 2023-03-09T22:14:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:14:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:14:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:21 smithi038 ceph-mon[92422]: osdmap e1120: 12 total, 12 up, 12 in 2023-03-09T22:14:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:22 smithi122 ceph-mon[93869]: pgmap v14470: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 426 KiB/s wr, 11 op/s 2023-03-09T22:14:22.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:22 smithi181 ceph-mon[94189]: pgmap v14470: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 426 KiB/s wr, 11 op/s 2023-03-09T22:14:22.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:22 smithi038 ceph-mon[92422]: pgmap v14470: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 426 KiB/s wr, 11 op/s 2023-03-09T22:14:23.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26635.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:14:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:24 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26173.722820 secs 2023-03-09T22:14:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:24 smithi122 ceph-mon[93869]: pgmap v14472: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 141 KiB/s wr, 9 op/s 2023-03-09T22:14:24.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:24 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26173.722820 secs 2023-03-09T22:14:24.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:24 smithi181 ceph-mon[94189]: pgmap v14472: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 141 KiB/s wr, 9 op/s 2023-03-09T22:14:24.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:24 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26173.722820 secs 2023-03-09T22:14:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:24 smithi038 ceph-mon[92422]: pgmap v14472: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 141 KiB/s wr, 9 op/s 2023-03-09T22:14:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:26 smithi122 ceph-mon[93869]: pgmap v14473: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 117 KiB/s wr, 10 op/s 2023-03-09T22:14:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26176.358870 secs 2023-03-09T22:14:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:26 smithi181 ceph-mon[94189]: pgmap v14473: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 117 KiB/s wr, 10 op/s 2023-03-09T22:14:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26176.358870 secs 2023-03-09T22:14:26.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:26 smithi038 ceph-mon[92422]: pgmap v14473: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 117 KiB/s wr, 10 op/s 2023-03-09T22:14:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26176.358870 secs 2023-03-09T22:14:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:28 smithi122 ceph-mon[93869]: pgmap v14474: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 98 KiB/s wr, 8 op/s 2023-03-09T22:14:28.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:28 smithi181 ceph-mon[94189]: pgmap v14474: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 98 KiB/s wr, 8 op/s 2023-03-09T22:14:28.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:28 smithi038 ceph-mon[92422]: pgmap v14474: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 98 KiB/s wr, 8 op/s 2023-03-09T22:14:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26640.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:14:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:29 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26178.722973 secs 2023-03-09T22:14:29.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:29 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26178.722973 secs 2023-03-09T22:14:29.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:29 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26178.722973 secs 2023-03-09T22:14:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:30 smithi122 ceph-mon[93869]: pgmap v14475: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 247 KiB/s wr, 8 op/s 2023-03-09T22:14:30.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:30 smithi181 ceph-mon[94189]: pgmap v14475: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 247 KiB/s wr, 8 op/s 2023-03-09T22:14:30.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:30 smithi038 ceph-mon[92422]: pgmap v14475: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 247 KiB/s wr, 8 op/s 2023-03-09T22:14:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26181.358929 secs 2023-03-09T22:14:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26181.358929 secs 2023-03-09T22:14:31.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26181.358929 secs 2023-03-09T22:14:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:32 smithi122 ceph-mon[93869]: pgmap v14476: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 223 KiB/s wr, 8 op/s 2023-03-09T22:14:32.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:32 smithi181 ceph-mon[94189]: pgmap v14476: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 223 KiB/s wr, 8 op/s 2023-03-09T22:14:32.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:32 smithi038 ceph-mon[92422]: pgmap v14476: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 223 KiB/s wr, 8 op/s 2023-03-09T22:14:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26183.723092 secs 2023-03-09T22:14:33.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26183.723092 secs 2023-03-09T22:14:33.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26183.723092 secs 2023-03-09T22:14:33.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26645.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:14:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:34 smithi122 ceph-mon[93869]: pgmap v14477: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 189 KiB/s wr, 7 op/s 2023-03-09T22:14:34.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:34 smithi181 ceph-mon[94189]: pgmap v14477: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 189 KiB/s wr, 7 op/s 2023-03-09T22:14:34.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:34 smithi038 ceph-mon[92422]: pgmap v14477: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 189 KiB/s wr, 7 op/s 2023-03-09T22:14:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:36 smithi122 ceph-mon[93869]: pgmap v14478: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 186 KiB/s wr, 8 op/s 2023-03-09T22:14:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26186.358994 secs 2023-03-09T22:14:36.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:36 smithi181 ceph-mon[94189]: pgmap v14478: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 186 KiB/s wr, 8 op/s 2023-03-09T22:14:36.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26186.358994 secs 2023-03-09T22:14:36.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:36 smithi038 ceph-mon[92422]: pgmap v14478: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 186 KiB/s wr, 8 op/s 2023-03-09T22:14:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26186.358994 secs 2023-03-09T22:14:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:14:37.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:14:37.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:14:38.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:38 smithi122 ceph-mon[93869]: pgmap v14479: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 185 KiB/s wr, 9 op/s 2023-03-09T22:14:38.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26188.723211 secs 2023-03-09T22:14:38.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:38 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:14:38.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:38 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:14:38.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:38 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:14:38.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:38 smithi181 ceph-mon[94189]: pgmap v14479: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 185 KiB/s wr, 9 op/s 2023-03-09T22:14:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26188.723211 secs 2023-03-09T22:14:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:38 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:14:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:38 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:14:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:38 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:14:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:38 smithi038 ceph-mon[92422]: pgmap v14479: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 185 KiB/s wr, 9 op/s 2023-03-09T22:14:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26188.723211 secs 2023-03-09T22:14:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:38 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:14:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:38 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:14:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:38 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:14:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26650.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:14:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:40 smithi122 ceph-mon[93869]: pgmap v14480: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 283 KiB/s wr, 8 op/s 2023-03-09T22:14:40.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:40 smithi181 ceph-mon[94189]: pgmap v14480: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 283 KiB/s wr, 8 op/s 2023-03-09T22:14:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:40 smithi038 ceph-mon[92422]: pgmap v14480: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 283 KiB/s wr, 8 op/s 2023-03-09T22:14:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26191.359056 secs 2023-03-09T22:14:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:14:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:14:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:41 smithi122 ceph-mon[93869]: pgmap v14481: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 98 KiB/s wr, 8 op/s 2023-03-09T22:14:41.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26191.359056 secs 2023-03-09T22:14:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:14:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:14:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:41 smithi181 ceph-mon[94189]: pgmap v14481: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 98 KiB/s wr, 8 op/s 2023-03-09T22:14:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26191.359056 secs 2023-03-09T22:14:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:14:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:14:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:41 smithi038 ceph-mon[92422]: pgmap v14481: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 98 KiB/s wr, 8 op/s 2023-03-09T22:14:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26655.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:14:44.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26193.723357 secs 2023-03-09T22:14:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:43 smithi122 ceph-mon[93869]: pgmap v14482: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 141 KiB/s wr, 8 op/s 2023-03-09T22:14:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26193.723357 secs 2023-03-09T22:14:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:43 smithi181 ceph-mon[94189]: pgmap v14482: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 141 KiB/s wr, 8 op/s 2023-03-09T22:14:44.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26193.723357 secs 2023-03-09T22:14:44.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:43 smithi038 ceph-mon[92422]: pgmap v14482: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 141 KiB/s wr, 8 op/s 2023-03-09T22:14:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:45 smithi122 ceph-mon[93869]: pgmap v14483: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 141 KiB/s wr, 8 op/s 2023-03-09T22:14:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:45 smithi181 ceph-mon[94189]: pgmap v14483: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 141 KiB/s wr, 8 op/s 2023-03-09T22:14:46.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:45 smithi038 ceph-mon[92422]: pgmap v14483: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 141 KiB/s wr, 8 op/s 2023-03-09T22:14:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26196.359117 secs 2023-03-09T22:14:47.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26196.359117 secs 2023-03-09T22:14:47.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26196.359117 secs 2023-03-09T22:14:48.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:47 smithi181 ceph-mon[94189]: pgmap v14484: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 141 KiB/s wr, 9 op/s 2023-03-09T22:14:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:14:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:14:48.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:47 smithi038 ceph-mon[92422]: pgmap v14484: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 141 KiB/s wr, 9 op/s 2023-03-09T22:14:48.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:14:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:14:48.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:47 smithi122 ceph-mon[93869]: pgmap v14484: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 141 KiB/s wr, 9 op/s 2023-03-09T22:14:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:14:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:14:48.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26660.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:14:49.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26198.723466 secs 2023-03-09T22:14:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26198.723466 secs 2023-03-09T22:14:49.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26198.723466 secs 2023-03-09T22:14:50.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:49 smithi038 ceph-mon[92422]: pgmap v14485: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 352 KiB/s wr, 8 op/s 2023-03-09T22:14:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:49 smithi122 ceph-mon[93869]: pgmap v14485: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 352 KiB/s wr, 8 op/s 2023-03-09T22:14:50.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:49 smithi181 ceph-mon[94189]: pgmap v14485: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 352 KiB/s wr, 8 op/s 2023-03-09T22:14:51.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26201.359175 secs 2023-03-09T22:14:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26201.359175 secs 2023-03-09T22:14:51.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26201.359175 secs 2023-03-09T22:14:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:51 smithi038 ceph-mon[92422]: pgmap v14486: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 276 KiB/s wr, 9 op/s 2023-03-09T22:14:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:51 smithi122 ceph-mon[93869]: pgmap v14486: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 276 KiB/s wr, 9 op/s 2023-03-09T22:14:52.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:51 smithi181 ceph-mon[94189]: pgmap v14486: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 276 KiB/s wr, 9 op/s 2023-03-09T22:14:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26665.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:14:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26203.723603 secs 2023-03-09T22:14:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:53 smithi122 ceph-mon[93869]: pgmap v14487: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 514 KiB/s rd, 413 KiB/s wr, 10 op/s 2023-03-09T22:14:54.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:54 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26203.723603 secs 2023-03-09T22:14:54.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:54 smithi181 ceph-mon[94189]: pgmap v14487: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 514 KiB/s rd, 413 KiB/s wr, 10 op/s 2023-03-09T22:14:54.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26203.723603 secs 2023-03-09T22:14:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:53 smithi038 ceph-mon[92422]: pgmap v14487: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 514 KiB/s rd, 413 KiB/s wr, 10 op/s 2023-03-09T22:14:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:56 smithi122 ceph-mon[93869]: pgmap v14488: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 391 KiB/s wr, 11 op/s 2023-03-09T22:14:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26206.359233 secs 2023-03-09T22:14:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:14:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:14:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:56 smithi181 ceph-mon[94189]: pgmap v14488: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 391 KiB/s wr, 11 op/s 2023-03-09T22:14:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26206.359233 secs 2023-03-09T22:14:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:14:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:14:56.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:56 smithi038 ceph-mon[92422]: pgmap v14488: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 391 KiB/s wr, 11 op/s 2023-03-09T22:14:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26206.359233 secs 2023-03-09T22:14:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:14:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:14:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:14:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:14:57.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:14:57.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:14:57.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:14:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:14:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:58 smithi122 ceph-mon[93869]: pgmap v14489: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 391 KiB/s wr, 11 op/s 2023-03-09T22:14:58.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:58 smithi181 ceph-mon[94189]: pgmap v14489: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 391 KiB/s wr, 11 op/s 2023-03-09T22:14:58.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:58 smithi038 ceph-mon[92422]: pgmap v14489: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 391 KiB/s wr, 11 op/s 2023-03-09T22:14:58.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26670.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:14:59.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:14:59 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26208.723741 secs 2023-03-09T22:14:59.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:14:59 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26208.723741 secs 2023-03-09T22:14:59.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:14:59 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26208.723741 secs 2023-03-09T22:15:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:00 smithi122 ceph-mon[93869]: pgmap v14490: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 717 KiB/s rd, 703 KiB/s wr, 12 op/s 2023-03-09T22:15:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:15:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:15:00.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:00 smithi181 ceph-mon[94189]: pgmap v14490: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 717 KiB/s rd, 703 KiB/s wr, 12 op/s 2023-03-09T22:15:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:15:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:15:00.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:00 smithi038 ceph-mon[92422]: pgmap v14490: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 717 KiB/s rd, 703 KiB/s wr, 12 op/s 2023-03-09T22:15:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:15:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:15:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26211.359293 secs 2023-03-09T22:15:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:15:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:15:01.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26211.359293 secs 2023-03-09T22:15:01.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:15:01.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:15:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26211.359293 secs 2023-03-09T22:15:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:15:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:15:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:02 smithi122 ceph-mon[93869]: pgmap v14491: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 513 KiB/s wr, 12 op/s 2023-03-09T22:15:02.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:02 smithi181 ceph-mon[94189]: pgmap v14491: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 513 KiB/s wr, 12 op/s 2023-03-09T22:15:02.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:02 smithi038 ceph-mon[92422]: pgmap v14491: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 513 KiB/s wr, 12 op/s 2023-03-09T22:15:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26675.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:15:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:04 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26213.723875 secs 2023-03-09T22:15:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:04 smithi122 ceph-mon[93869]: pgmap v14492: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 538 KiB/s wr, 12 op/s 2023-03-09T22:15:04.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:04 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26213.723875 secs 2023-03-09T22:15:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:04 smithi181 ceph-mon[94189]: pgmap v14492: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 538 KiB/s wr, 12 op/s 2023-03-09T22:15:04.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:04 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26213.723875 secs 2023-03-09T22:15:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:04 smithi038 ceph-mon[92422]: pgmap v14492: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 538 KiB/s wr, 12 op/s 2023-03-09T22:15:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:06 smithi122 ceph-mon[93869]: pgmap v14493: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 429 KiB/s wr, 22 op/s 2023-03-09T22:15:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26216.359374 secs 2023-03-09T22:15:06.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:06 smithi181 ceph-mon[94189]: pgmap v14493: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 429 KiB/s wr, 22 op/s 2023-03-09T22:15:06.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26216.359374 secs 2023-03-09T22:15:06.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:06 smithi038 ceph-mon[92422]: pgmap v14493: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 429 KiB/s wr, 22 op/s 2023-03-09T22:15:06.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26216.359374 secs 2023-03-09T22:15:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:08 smithi122 ceph-mon[93869]: pgmap v14494: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 564 KiB/s wr, 24 op/s 2023-03-09T22:15:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:08 smithi181 ceph-mon[94189]: pgmap v14494: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 564 KiB/s wr, 24 op/s 2023-03-09T22:15:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:08 smithi038 ceph-mon[92422]: pgmap v14494: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 564 KiB/s wr, 24 op/s 2023-03-09T22:15:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26680.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:15:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:09 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26218.724014 secs 2023-03-09T22:15:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:15:09.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:15:09.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:09 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26218.724014 secs 2023-03-09T22:15:09.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:15:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:15:09.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:09 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26218.724014 secs 2023-03-09T22:15:09.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:15:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:15:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:10 smithi122 ceph-mon[93869]: pgmap v14495: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 837 KiB/s rd, 840 KiB/s wr, 24 op/s 2023-03-09T22:15:10.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:10 smithi181 ceph-mon[94189]: pgmap v14495: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 837 KiB/s rd, 840 KiB/s wr, 24 op/s 2023-03-09T22:15:10.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:10 smithi038 ceph-mon[92422]: pgmap v14495: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 837 KiB/s rd, 840 KiB/s wr, 24 op/s 2023-03-09T22:15:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26221.359433 secs 2023-03-09T22:15:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26221.359433 secs 2023-03-09T22:15:11.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26221.359433 secs 2023-03-09T22:15:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:12 smithi122 ceph-mon[93869]: pgmap v14496: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 527 KiB/s rd, 528 KiB/s wr, 24 op/s 2023-03-09T22:15:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:15:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:15:12.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:12 smithi181 ceph-mon[94189]: pgmap v14496: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 527 KiB/s rd, 528 KiB/s wr, 24 op/s 2023-03-09T22:15:12.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:15:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:15:12.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:12 smithi038 ceph-mon[92422]: pgmap v14496: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 527 KiB/s rd, 528 KiB/s wr, 24 op/s 2023-03-09T22:15:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:15:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:15:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26223.724153 secs 2023-03-09T22:15:13.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26223.724153 secs 2023-03-09T22:15:13.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26223.724153 secs 2023-03-09T22:15:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26685.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:15:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:14 smithi122 ceph-mon[93869]: pgmap v14497: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 580 KiB/s rd, 582 KiB/s wr, 24 op/s 2023-03-09T22:15:14.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:14 smithi181 ceph-mon[94189]: pgmap v14497: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 580 KiB/s rd, 582 KiB/s wr, 24 op/s 2023-03-09T22:15:14.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:14 smithi038 ceph-mon[92422]: pgmap v14497: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 580 KiB/s rd, 582 KiB/s wr, 24 op/s 2023-03-09T22:15:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:15 smithi122 ceph-mon[93869]: pgmap v14498: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 533 KiB/s rd, 535 KiB/s wr, 22 op/s 2023-03-09T22:15:15.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:15 smithi181 ceph-mon[94189]: pgmap v14498: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 533 KiB/s rd, 535 KiB/s wr, 22 op/s 2023-03-09T22:15:15.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:15 smithi038 ceph-mon[92422]: pgmap v14498: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 533 KiB/s rd, 535 KiB/s wr, 22 op/s 2023-03-09T22:15:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26226.359518 secs 2023-03-09T22:15:16.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26226.359518 secs 2023-03-09T22:15:16.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26226.359518 secs 2023-03-09T22:15:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:17 smithi122 ceph-mon[93869]: pgmap v14499: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 532 KiB/s rd, 529 KiB/s wr, 13 op/s 2023-03-09T22:15:17.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:17 smithi181 ceph-mon[94189]: pgmap v14499: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 532 KiB/s rd, 529 KiB/s wr, 13 op/s 2023-03-09T22:15:17.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:17 smithi038 ceph-mon[92422]: pgmap v14499: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 532 KiB/s rd, 529 KiB/s wr, 13 op/s 2023-03-09T22:15:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26228.724255 secs 2023-03-09T22:15:18.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26228.724255 secs 2023-03-09T22:15:18.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26228.724255 secs 2023-03-09T22:15:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26690.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:15:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:19 smithi122 ceph-mon[93869]: pgmap v14500: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 373 KiB/s wr, 9 op/s 2023-03-09T22:15:19.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:19 smithi181 ceph-mon[94189]: pgmap v14500: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 373 KiB/s wr, 9 op/s 2023-03-09T22:15:19.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:19 smithi038 ceph-mon[92422]: pgmap v14500: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 373 KiB/s wr, 9 op/s 2023-03-09T22:15:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26231.359582 secs 2023-03-09T22:15:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:21 smithi122 ceph-mon[93869]: pgmap v14501: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 219 KiB/s wr, 9 op/s 2023-03-09T22:15:22.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26231.359582 secs 2023-03-09T22:15:22.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:21 smithi181 ceph-mon[94189]: pgmap v14501: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 219 KiB/s wr, 9 op/s 2023-03-09T22:15:22.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26231.359582 secs 2023-03-09T22:15:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:21 smithi038 ceph-mon[92422]: pgmap v14501: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 219 KiB/s wr, 9 op/s 2023-03-09T22:15:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26695.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:15:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26233.724364 secs 2023-03-09T22:15:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:23 smithi122 ceph-mon[93869]: pgmap v14502: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 348 KiB/s rd, 357 KiB/s wr, 10 op/s 2023-03-09T22:15:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26233.724364 secs 2023-03-09T22:15:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:23 smithi181 ceph-mon[94189]: pgmap v14502: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 348 KiB/s rd, 357 KiB/s wr, 10 op/s 2023-03-09T22:15:24.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26233.724364 secs 2023-03-09T22:15:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:23 smithi038 ceph-mon[92422]: pgmap v14502: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 348 KiB/s rd, 357 KiB/s wr, 10 op/s 2023-03-09T22:15:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:25 smithi122 ceph-mon[93869]: pgmap v14503: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 293 KiB/s wr, 10 op/s 2023-03-09T22:15:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:25 smithi181 ceph-mon[94189]: pgmap v14503: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 293 KiB/s wr, 10 op/s 2023-03-09T22:15:26.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:25 smithi038 ceph-mon[92422]: pgmap v14503: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 293 KiB/s wr, 10 op/s 2023-03-09T22:15:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26236.359646 secs 2023-03-09T22:15:27.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26236.359646 secs 2023-03-09T22:15:27.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26236.359646 secs 2023-03-09T22:15:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:27 smithi038 ceph-mon[92422]: pgmap v14504: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 384 KiB/s wr, 13 op/s 2023-03-09T22:15:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:27 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:15:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:27 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:15:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:27 smithi122 ceph-mon[93869]: pgmap v14504: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 384 KiB/s wr, 13 op/s 2023-03-09T22:15:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:27 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:15:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:27 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:15:28.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:27 smithi181 ceph-mon[94189]: pgmap v14504: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 384 KiB/s wr, 13 op/s 2023-03-09T22:15:28.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:27 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:15:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:27 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:15:28.793 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26700.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:15:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:29 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26238.724488 secs 2023-03-09T22:15:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:29 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:15:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:29 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:15:29.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:29 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26238.724488 secs 2023-03-09T22:15:29.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:29 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:15:29.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:29 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:15:29.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:29 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26238.724488 secs 2023-03-09T22:15:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:29 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:15:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:29 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:15:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:30 smithi122 ceph-mon[93869]: pgmap v14505: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 363 KiB/s wr, 10 op/s 2023-03-09T22:15:30.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:30 smithi181 ceph-mon[94189]: pgmap v14505: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 363 KiB/s wr, 10 op/s 2023-03-09T22:15:30.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:30 smithi038 ceph-mon[92422]: pgmap v14505: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 363 KiB/s wr, 10 op/s 2023-03-09T22:15:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26241.359707 secs 2023-03-09T22:15:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:31 smithi122 ceph-mon[93869]: osdmap e1121: 12 total, 12 up, 12 in 2023-03-09T22:15:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26241.359707 secs 2023-03-09T22:15:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:31 smithi181 ceph-mon[94189]: osdmap e1121: 12 total, 12 up, 12 in 2023-03-09T22:15:31.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26241.359707 secs 2023-03-09T22:15:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:31 smithi038 ceph-mon[92422]: osdmap e1121: 12 total, 12 up, 12 in 2023-03-09T22:15:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:32 smithi122 ceph-mon[93869]: pgmap v14507: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 463 KiB/s wr, 12 op/s 2023-03-09T22:15:32.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:32 smithi181 ceph-mon[94189]: pgmap v14507: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 463 KiB/s wr, 12 op/s 2023-03-09T22:15:32.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:32 smithi038 ceph-mon[92422]: pgmap v14507: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 463 KiB/s wr, 12 op/s 2023-03-09T22:15:33.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26705.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:15:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:34 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26243.724611 secs 2023-03-09T22:15:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:34 smithi122 ceph-mon[93869]: pgmap v14508: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 298 KiB/s wr, 10 op/s 2023-03-09T22:15:34.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:34 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26243.724611 secs 2023-03-09T22:15:34.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:34 smithi181 ceph-mon[94189]: pgmap v14508: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 298 KiB/s wr, 10 op/s 2023-03-09T22:15:34.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:34 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26243.724611 secs 2023-03-09T22:15:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:34 smithi038 ceph-mon[92422]: pgmap v14508: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 298 KiB/s wr, 10 op/s 2023-03-09T22:15:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:36 smithi122 ceph-mon[93869]: pgmap v14509: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 285 KiB/s wr, 9 op/s 2023-03-09T22:15:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26246.359782 secs 2023-03-09T22:15:36.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:36 smithi181 ceph-mon[94189]: pgmap v14509: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 285 KiB/s wr, 9 op/s 2023-03-09T22:15:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26246.359782 secs 2023-03-09T22:15:36.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:36 smithi038 ceph-mon[92422]: pgmap v14509: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 285 KiB/s wr, 9 op/s 2023-03-09T22:15:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26246.359782 secs 2023-03-09T22:15:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:38 smithi122 ceph-mon[93869]: pgmap v14510: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 181 KiB/s wr, 8 op/s 2023-03-09T22:15:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:38 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:15:38.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:38 smithi181 ceph-mon[94189]: pgmap v14510: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 181 KiB/s wr, 8 op/s 2023-03-09T22:15:38.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:38 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:15:38.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:38 smithi038 ceph-mon[92422]: pgmap v14510: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 181 KiB/s wr, 8 op/s 2023-03-09T22:15:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:38 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:15:38.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26710.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:15:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:39 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26248.724730 secs 2023-03-09T22:15:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:15:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:15:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:15:39.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:39 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26248.724730 secs 2023-03-09T22:15:39.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:15:39.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:15:39.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:15:39.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:39 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26248.724730 secs 2023-03-09T22:15:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:15:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:15:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:15:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:40 smithi122 ceph-mon[93869]: pgmap v14511: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 181 KiB/s wr, 8 op/s 2023-03-09T22:15:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:15:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:15:40.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:40 smithi181 ceph-mon[94189]: pgmap v14511: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 181 KiB/s wr, 8 op/s 2023-03-09T22:15:40.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:15:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:15:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:40 smithi038 ceph-mon[92422]: pgmap v14511: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 181 KiB/s wr, 8 op/s 2023-03-09T22:15:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:15:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:15:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26251.359841 secs 2023-03-09T22:15:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:15:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:15:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:41 smithi122 ceph-mon[93869]: osdmap e1122: 12 total, 12 up, 12 in 2023-03-09T22:15:41.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26251.359841 secs 2023-03-09T22:15:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:15:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:15:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:41 smithi181 ceph-mon[94189]: osdmap e1122: 12 total, 12 up, 12 in 2023-03-09T22:15:41.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26251.359841 secs 2023-03-09T22:15:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:15:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:15:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:41 smithi038 ceph-mon[92422]: osdmap e1122: 12 total, 12 up, 12 in 2023-03-09T22:15:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:42 smithi122 ceph-mon[93869]: pgmap v14512: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 297 KiB/s wr, 10 op/s 2023-03-09T22:15:42.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:42 smithi181 ceph-mon[94189]: pgmap v14512: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 297 KiB/s wr, 10 op/s 2023-03-09T22:15:42.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:42 smithi038 ceph-mon[92422]: pgmap v14512: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 297 KiB/s wr, 10 op/s 2023-03-09T22:15:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26715.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:15:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:44 smithi122 ceph-mon[93869]: pgmap v14514: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 145 KiB/s wr, 7 op/s 2023-03-09T22:15:44.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:44 smithi181 ceph-mon[94189]: pgmap v14514: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 145 KiB/s wr, 7 op/s 2023-03-09T22:15:44.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:44 smithi038 ceph-mon[92422]: pgmap v14514: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 145 KiB/s wr, 7 op/s 2023-03-09T22:15:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:45 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:15:45.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:45 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:15:45.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:45 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:15:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:46 smithi122 ceph-mon[93869]: pgmap v14515: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 293 KiB/s rd, 291 KiB/s wr, 8 op/s 2023-03-09T22:15:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26256.359904 secs 2023-03-09T22:15:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:46 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:15:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:46 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:15:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:46 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:15:46.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:46 smithi181 ceph-mon[94189]: pgmap v14515: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 293 KiB/s rd, 291 KiB/s wr, 8 op/s 2023-03-09T22:15:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26256.359904 secs 2023-03-09T22:15:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:46 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:15:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:46 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:15:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:46 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:15:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:46 smithi038 ceph-mon[92422]: pgmap v14515: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 293 KiB/s rd, 291 KiB/s wr, 8 op/s 2023-03-09T22:15:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26256.359904 secs 2023-03-09T22:15:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:46 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:15:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:46 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:15:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:46 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:15:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:47 smithi122 ceph-mon[93869]: pgmap v14516: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 332 KiB/s wr, 9 op/s 2023-03-09T22:15:47.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:47 smithi181 ceph-mon[94189]: pgmap v14516: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 332 KiB/s wr, 9 op/s 2023-03-09T22:15:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:47 smithi038 ceph-mon[92422]: pgmap v14516: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 332 KiB/s wr, 9 op/s 2023-03-09T22:15:48.793 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26720.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:15:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:49 smithi122 ceph-mon[93869]: pgmap v14517: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 332 KiB/s wr, 9 op/s 2023-03-09T22:15:50.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:49 smithi181 ceph-mon[94189]: pgmap v14517: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 332 KiB/s wr, 9 op/s 2023-03-09T22:15:50.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:49 smithi038 ceph-mon[92422]: pgmap v14517: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 332 KiB/s wr, 9 op/s 2023-03-09T22:15:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26261.359964 secs 2023-03-09T22:15:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:51 smithi122 ceph-mon[93869]: pgmap v14518: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 585 KiB/s rd, 581 KiB/s wr, 10 op/s 2023-03-09T22:15:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:15:52.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:15:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26261.359964 secs 2023-03-09T22:15:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:51 smithi181 ceph-mon[94189]: pgmap v14518: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 585 KiB/s rd, 581 KiB/s wr, 10 op/s 2023-03-09T22:15:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:15:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:15:52.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26261.359964 secs 2023-03-09T22:15:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:51 smithi038 ceph-mon[92422]: pgmap v14518: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 585 KiB/s rd, 581 KiB/s wr, 10 op/s 2023-03-09T22:15:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:15:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:15:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:15:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:15:53.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:15:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:15:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:15:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:15:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26725.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:15:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:54 smithi122 ceph-mon[93869]: pgmap v14519: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 492 KiB/s wr, 9 op/s 2023-03-09T22:15:54.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:54 smithi181 ceph-mon[94189]: pgmap v14519: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 492 KiB/s wr, 9 op/s 2023-03-09T22:15:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:54 smithi038 ceph-mon[92422]: pgmap v14519: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 492 KiB/s wr, 9 op/s 2023-03-09T22:15:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:56 smithi122 ceph-mon[93869]: pgmap v14520: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 499 KiB/s rd, 484 KiB/s wr, 10 op/s 2023-03-09T22:15:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26266.359997 secs 2023-03-09T22:15:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:15:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:15:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:56 smithi181 ceph-mon[94189]: pgmap v14520: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 499 KiB/s rd, 484 KiB/s wr, 10 op/s 2023-03-09T22:15:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26266.359997 secs 2023-03-09T22:15:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:15:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:15:56.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:56 smithi038 ceph-mon[92422]: pgmap v14520: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 499 KiB/s rd, 484 KiB/s wr, 10 op/s 2023-03-09T22:15:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26266.359997 secs 2023-03-09T22:15:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:15:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:15:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:15:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:15:57.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:15:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:15:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:15:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:15:58.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:15:58 smithi122 ceph-mon[93869]: pgmap v14521: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 697 KiB/s rd, 681 KiB/s wr, 14 op/s 2023-03-09T22:15:58.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:15:58 smithi181 ceph-mon[94189]: pgmap v14521: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 697 KiB/s rd, 681 KiB/s wr, 14 op/s 2023-03-09T22:15:58.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:15:58 smithi038 ceph-mon[92422]: pgmap v14521: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 697 KiB/s rd, 681 KiB/s wr, 14 op/s 2023-03-09T22:15:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26730.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:16:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:00 smithi122 ceph-mon[93869]: pgmap v14522: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 643 KiB/s wr, 11 op/s 2023-03-09T22:16:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26271.360055 secs 2023-03-09T22:16:00.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:00 smithi181 ceph-mon[94189]: pgmap v14522: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 643 KiB/s wr, 11 op/s 2023-03-09T22:16:00.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26271.360055 secs 2023-03-09T22:16:00.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:00 smithi038 ceph-mon[92422]: pgmap v14522: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 643 KiB/s wr, 11 op/s 2023-03-09T22:16:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26271.360055 secs 2023-03-09T22:16:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:02 smithi122 ceph-mon[93869]: pgmap v14523: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 679 KiB/s rd, 664 KiB/s wr, 13 op/s 2023-03-09T22:16:02.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:02 smithi181 ceph-mon[94189]: pgmap v14523: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 679 KiB/s rd, 664 KiB/s wr, 13 op/s 2023-03-09T22:16:02.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:02 smithi038 ceph-mon[92422]: pgmap v14523: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 679 KiB/s rd, 664 KiB/s wr, 13 op/s 2023-03-09T22:16:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26735.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:16:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:04 smithi122 ceph-mon[93869]: pgmap v14524: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 618 KiB/s rd, 511 KiB/s wr, 12 op/s 2023-03-09T22:16:04.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:04 smithi181 ceph-mon[94189]: pgmap v14524: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 618 KiB/s rd, 511 KiB/s wr, 12 op/s 2023-03-09T22:16:04.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:04 smithi038 ceph-mon[92422]: pgmap v14524: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 618 KiB/s rd, 511 KiB/s wr, 12 op/s 2023-03-09T22:16:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26276.360117 secs 2023-03-09T22:16:05.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26276.360117 secs 2023-03-09T22:16:05.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26276.360117 secs 2023-03-09T22:16:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:06 smithi122 ceph-mon[93869]: pgmap v14525: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 619 KiB/s rd, 511 KiB/s wr, 12 op/s 2023-03-09T22:16:06.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:06 smithi181 ceph-mon[94189]: pgmap v14525: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 619 KiB/s rd, 511 KiB/s wr, 12 op/s 2023-03-09T22:16:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:06 smithi038 ceph-mon[92422]: pgmap v14525: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 619 KiB/s rd, 511 KiB/s wr, 12 op/s 2023-03-09T22:16:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:08 smithi122 ceph-mon[93869]: pgmap v14526: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 760 KiB/s wr, 17 op/s 2023-03-09T22:16:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:08 smithi181 ceph-mon[94189]: pgmap v14526: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 760 KiB/s wr, 17 op/s 2023-03-09T22:16:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:08 smithi038 ceph-mon[92422]: pgmap v14526: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 760 KiB/s wr, 17 op/s 2023-03-09T22:16:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26740.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:16:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:09 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26278.725377 secs 2023-03-09T22:16:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:16:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:16:09.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:09 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26278.725377 secs 2023-03-09T22:16:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:16:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:16:09.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:09 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26278.725377 secs 2023-03-09T22:16:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:16:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:16:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:10 smithi122 ceph-mon[93869]: pgmap v14527: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 899 KiB/s rd, 462 KiB/s wr, 24 op/s 2023-03-09T22:16:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:10 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:16:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:10 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:16:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:10 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:16:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26281.360176 secs 2023-03-09T22:16:10.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:10 smithi181 ceph-mon[94189]: pgmap v14527: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 899 KiB/s rd, 462 KiB/s wr, 24 op/s 2023-03-09T22:16:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:10 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:16:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:10 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:16:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:10 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:16:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26281.360176 secs 2023-03-09T22:16:10.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:10 smithi038 ceph-mon[92422]: pgmap v14527: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 899 KiB/s rd, 462 KiB/s wr, 24 op/s 2023-03-09T22:16:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:10 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:16:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:10 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:16:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:10 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:16:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26281.360176 secs 2023-03-09T22:16:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:11 smithi122 ceph-mon[93869]: osdmap e1123: 12 total, 12 up, 12 in 2023-03-09T22:16:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:11 smithi181 ceph-mon[94189]: osdmap e1123: 12 total, 12 up, 12 in 2023-03-09T22:16:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:11 smithi038 ceph-mon[92422]: osdmap e1123: 12 total, 12 up, 12 in 2023-03-09T22:16:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:12 smithi122 ceph-mon[93869]: pgmap v14529: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 684 KiB/s wr, 28 op/s 2023-03-09T22:16:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:16:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:16:12.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:12 smithi181 ceph-mon[94189]: pgmap v14529: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 684 KiB/s wr, 28 op/s 2023-03-09T22:16:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:16:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:16:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:12 smithi038 ceph-mon[92422]: pgmap v14529: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 684 KiB/s wr, 28 op/s 2023-03-09T22:16:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:16:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:16:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26745.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:16:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:14 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26283.725489 secs 2023-03-09T22:16:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:14 smithi122 ceph-mon[93869]: pgmap v14530: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1020 KiB/s rd, 644 KiB/s wr, 28 op/s 2023-03-09T22:16:14.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:14 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26283.725489 secs 2023-03-09T22:16:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:14 smithi181 ceph-mon[94189]: pgmap v14530: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1020 KiB/s rd, 644 KiB/s wr, 28 op/s 2023-03-09T22:16:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:14 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26283.725489 secs 2023-03-09T22:16:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:14 smithi038 ceph-mon[92422]: pgmap v14530: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1020 KiB/s rd, 644 KiB/s wr, 28 op/s 2023-03-09T22:16:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26286.360238 secs 2023-03-09T22:16:15.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26286.360238 secs 2023-03-09T22:16:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26286.360238 secs 2023-03-09T22:16:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:16 smithi122 ceph-mon[93869]: pgmap v14531: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1019 KiB/s rd, 644 KiB/s wr, 26 op/s 2023-03-09T22:16:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:16:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:16:16.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:16 smithi181 ceph-mon[94189]: pgmap v14531: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1019 KiB/s rd, 644 KiB/s wr, 26 op/s 2023-03-09T22:16:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:16:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:16:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:16 smithi038 ceph-mon[92422]: pgmap v14531: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1019 KiB/s rd, 644 KiB/s wr, 26 op/s 2023-03-09T22:16:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:16:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:16:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:16:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:16:17.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:16:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:16:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:16:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:16:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:18 smithi122 ceph-mon[93869]: pgmap v14532: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 446 KiB/s rd, 462 KiB/s wr, 24 op/s 2023-03-09T22:16:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:18 smithi181 ceph-mon[94189]: pgmap v14532: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 446 KiB/s rd, 462 KiB/s wr, 24 op/s 2023-03-09T22:16:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:18 smithi038 ceph-mon[92422]: pgmap v14532: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 446 KiB/s rd, 462 KiB/s wr, 24 op/s 2023-03-09T22:16:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26750.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:16:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:19 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26288.725603 secs 2023-03-09T22:16:19.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:19 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26288.725603 secs 2023-03-09T22:16:19.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:19 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26288.725603 secs 2023-03-09T22:16:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:20 smithi122 ceph-mon[93869]: pgmap v14533: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 436 KiB/s wr, 11 op/s 2023-03-09T22:16:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26291.360297 secs 2023-03-09T22:16:20.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:20 smithi181 ceph-mon[94189]: pgmap v14533: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 436 KiB/s wr, 11 op/s 2023-03-09T22:16:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26291.360297 secs 2023-03-09T22:16:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:20 smithi038 ceph-mon[92422]: pgmap v14533: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 436 KiB/s wr, 11 op/s 2023-03-09T22:16:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26291.360297 secs 2023-03-09T22:16:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:22 smithi122 ceph-mon[93869]: pgmap v14534: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 408 KiB/s wr, 13 op/s 2023-03-09T22:16:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:22 smithi122 ceph-mon[93869]: osdmap e1124: 12 total, 12 up, 12 in 2023-03-09T22:16:22.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:22 smithi181 ceph-mon[94189]: pgmap v14534: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 408 KiB/s wr, 13 op/s 2023-03-09T22:16:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:22 smithi181 ceph-mon[94189]: osdmap e1124: 12 total, 12 up, 12 in 2023-03-09T22:16:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:22 smithi038 ceph-mon[92422]: pgmap v14534: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 408 KiB/s wr, 13 op/s 2023-03-09T22:16:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:22 smithi038 ceph-mon[92422]: osdmap e1124: 12 total, 12 up, 12 in 2023-03-09T22:16:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26755.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:16:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:24 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26293.725733 secs 2023-03-09T22:16:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:24 smithi122 ceph-mon[93869]: pgmap v14536: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 118 KiB/s wr, 9 op/s 2023-03-09T22:16:24.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:24 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26293.725733 secs 2023-03-09T22:16:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:24 smithi181 ceph-mon[94189]: pgmap v14536: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 118 KiB/s wr, 9 op/s 2023-03-09T22:16:24.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:24 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26293.725733 secs 2023-03-09T22:16:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:24 smithi038 ceph-mon[92422]: pgmap v14536: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 118 KiB/s wr, 9 op/s 2023-03-09T22:16:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26296.360358 secs 2023-03-09T22:16:25.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26296.360358 secs 2023-03-09T22:16:25.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26296.360358 secs 2023-03-09T22:16:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:26 smithi122 ceph-mon[93869]: pgmap v14537: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 118 KiB/s wr, 8 op/s 2023-03-09T22:16:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:26 smithi181 ceph-mon[94189]: pgmap v14537: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 118 KiB/s wr, 8 op/s 2023-03-09T22:16:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:26 smithi038 ceph-mon[92422]: pgmap v14537: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 118 KiB/s wr, 8 op/s 2023-03-09T22:16:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:28 smithi122 ceph-mon[93869]: pgmap v14538: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 133 KiB/s wr, 7 op/s 2023-03-09T22:16:28.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:28 smithi181 ceph-mon[94189]: pgmap v14538: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 133 KiB/s wr, 7 op/s 2023-03-09T22:16:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:28 smithi038 ceph-mon[92422]: pgmap v14538: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 133 KiB/s wr, 7 op/s 2023-03-09T22:16:28.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26760.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:16:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:29 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26298.725858 secs 2023-03-09T22:16:29.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:29 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26298.725858 secs 2023-03-09T22:16:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:29 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26298.725858 secs 2023-03-09T22:16:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:30 smithi122 ceph-mon[93869]: pgmap v14539: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 208 KiB/s wr, 7 op/s 2023-03-09T22:16:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26301.360419 secs 2023-03-09T22:16:30.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:30 smithi181 ceph-mon[94189]: pgmap v14539: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 208 KiB/s wr, 7 op/s 2023-03-09T22:16:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26301.360419 secs 2023-03-09T22:16:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:30 smithi038 ceph-mon[92422]: pgmap v14539: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 208 KiB/s wr, 7 op/s 2023-03-09T22:16:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26301.360419 secs 2023-03-09T22:16:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:32 smithi122 ceph-mon[93869]: pgmap v14540: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 265 KiB/s wr, 7 op/s 2023-03-09T22:16:32.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:32 smithi181 ceph-mon[94189]: pgmap v14540: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 265 KiB/s wr, 7 op/s 2023-03-09T22:16:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:32 smithi038 ceph-mon[92422]: pgmap v14540: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 265 KiB/s wr, 7 op/s 2023-03-09T22:16:33.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26765.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:16:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:34 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26303.725954 secs 2023-03-09T22:16:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:34 smithi122 ceph-mon[93869]: pgmap v14541: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 224 KiB/s wr, 6 op/s 2023-03-09T22:16:34.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:34 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26303.725954 secs 2023-03-09T22:16:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:34 smithi181 ceph-mon[94189]: pgmap v14541: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 224 KiB/s wr, 6 op/s 2023-03-09T22:16:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:34 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26303.725954 secs 2023-03-09T22:16:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:34 smithi038 ceph-mon[92422]: pgmap v14541: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 224 KiB/s wr, 6 op/s 2023-03-09T22:16:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26306.360479 secs 2023-03-09T22:16:35.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26306.360479 secs 2023-03-09T22:16:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26306.360479 secs 2023-03-09T22:16:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:36 smithi122 ceph-mon[93869]: pgmap v14542: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 221 KiB/s wr, 7 op/s 2023-03-09T22:16:36.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:36 smithi181 ceph-mon[94189]: pgmap v14542: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 221 KiB/s wr, 7 op/s 2023-03-09T22:16:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:36 smithi038 ceph-mon[92422]: pgmap v14542: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 221 KiB/s wr, 7 op/s 2023-03-09T22:16:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:38 smithi122 ceph-mon[93869]: pgmap v14543: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 221 KiB/s wr, 10 op/s 2023-03-09T22:16:38.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:38 smithi181 ceph-mon[94189]: pgmap v14543: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 221 KiB/s wr, 10 op/s 2023-03-09T22:16:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:38 smithi038 ceph-mon[92422]: pgmap v14543: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 221 KiB/s wr, 10 op/s 2023-03-09T22:16:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26770.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:16:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:39 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26308.726063 secs 2023-03-09T22:16:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:16:39.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:16:39.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:16:39.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:16:39.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:39 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26308.726063 secs 2023-03-09T22:16:39.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:16:39.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:16:39.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:16:39.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:16:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:39 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26308.726063 secs 2023-03-09T22:16:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:16:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:16:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:16:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:16:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:40 smithi122 ceph-mon[93869]: pgmap v14544: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 240 KiB/s wr, 7 op/s 2023-03-09T22:16:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:16:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:16:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26311.360541 secs 2023-03-09T22:16:40.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:40 smithi181 ceph-mon[94189]: pgmap v14544: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 240 KiB/s wr, 7 op/s 2023-03-09T22:16:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:16:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:16:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26311.360541 secs 2023-03-09T22:16:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:40 smithi038 ceph-mon[92422]: pgmap v14544: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 240 KiB/s wr, 7 op/s 2023-03-09T22:16:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:16:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:16:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26311.360541 secs 2023-03-09T22:16:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:16:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:16:41.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:16:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:16:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:16:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:16:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:42 smithi122 ceph-mon[93869]: pgmap v14545: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 178 KiB/s wr, 9 op/s 2023-03-09T22:16:42.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:42 smithi181 ceph-mon[94189]: pgmap v14545: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 178 KiB/s wr, 9 op/s 2023-03-09T22:16:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:42 smithi038 ceph-mon[92422]: pgmap v14545: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 178 KiB/s wr, 9 op/s 2023-03-09T22:16:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26775.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:16:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:44 smithi122 ceph-mon[93869]: pgmap v14546: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 130 KiB/s wr, 6 op/s 2023-03-09T22:16:44.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:44 smithi181 ceph-mon[94189]: pgmap v14546: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 130 KiB/s wr, 6 op/s 2023-03-09T22:16:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:44 smithi038 ceph-mon[92422]: pgmap v14546: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 130 KiB/s wr, 6 op/s 2023-03-09T22:16:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:45 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:16:45.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26316.360668 secs 2023-03-09T22:16:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:45 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:16:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26316.360668 secs 2023-03-09T22:16:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:45 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:16:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26316.360668 secs 2023-03-09T22:16:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:46 smithi122 ceph-mon[93869]: pgmap v14547: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 161 KiB/s wr, 7 op/s 2023-03-09T22:16:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:46 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:16:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:46 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:16:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:46 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:16:46.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:46 smithi181 ceph-mon[94189]: pgmap v14547: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 161 KiB/s wr, 7 op/s 2023-03-09T22:16:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:46 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:16:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:46 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:16:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:46 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:16:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:46 smithi038 ceph-mon[92422]: pgmap v14547: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 161 KiB/s wr, 7 op/s 2023-03-09T22:16:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:46 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:16:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:46 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:16:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:46 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:16:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:48 smithi122 ceph-mon[93869]: pgmap v14548: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 276 KiB/s wr, 9 op/s 2023-03-09T22:16:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:48 smithi181 ceph-mon[94189]: pgmap v14548: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 276 KiB/s wr, 9 op/s 2023-03-09T22:16:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:48 smithi038 ceph-mon[92422]: pgmap v14548: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 276 KiB/s wr, 9 op/s 2023-03-09T22:16:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26780.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:16:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:49 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 26318.726295 secs 2023-03-09T22:16:49.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:49 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 26318.726295 secs 2023-03-09T22:16:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:49 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 26318.726295 secs 2023-03-09T22:16:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:50 smithi122 ceph-mon[93869]: pgmap v14549: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 395 KiB/s wr, 7 op/s 2023-03-09T22:16:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:50 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:16:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:50 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:16:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26321.360728 secs 2023-03-09T22:16:50.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:50 smithi181 ceph-mon[94189]: pgmap v14549: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 395 KiB/s wr, 7 op/s 2023-03-09T22:16:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:50 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:16:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:50 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:16:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26321.360728 secs 2023-03-09T22:16:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:50 smithi038 ceph-mon[92422]: pgmap v14549: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 395 KiB/s wr, 7 op/s 2023-03-09T22:16:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:50 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:16:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:50 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:16:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26321.360728 secs 2023-03-09T22:16:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:51 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:16:51.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:51 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:16:51.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:51 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:16:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:52 smithi122 ceph-mon[93869]: pgmap v14550: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 540 KiB/s rd, 424 KiB/s wr, 10 op/s 2023-03-09T22:16:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:16:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:16:52.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:52 smithi181 ceph-mon[94189]: pgmap v14550: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 540 KiB/s rd, 424 KiB/s wr, 10 op/s 2023-03-09T22:16:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:16:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:16:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:52 smithi038 ceph-mon[92422]: pgmap v14550: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 540 KiB/s rd, 424 KiB/s wr, 10 op/s 2023-03-09T22:16:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:16:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:16:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:16:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:16:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:53 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 26323.726405 secs 2023-03-09T22:16:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:16:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:16:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:53 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 26323.726405 secs 2023-03-09T22:16:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:16:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:16:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:53 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 26323.726405 secs 2023-03-09T22:16:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26785.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:16:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:54 smithi122 ceph-mon[93869]: pgmap v14551: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 538 KiB/s rd, 425 KiB/s wr, 8 op/s 2023-03-09T22:16:54.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:54 smithi181 ceph-mon[94189]: pgmap v14551: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 538 KiB/s rd, 425 KiB/s wr, 8 op/s 2023-03-09T22:16:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:54 smithi038 ceph-mon[92422]: pgmap v14551: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 538 KiB/s rd, 425 KiB/s wr, 8 op/s 2023-03-09T22:16:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26326.360787 secs 2023-03-09T22:16:55.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26326.360787 secs 2023-03-09T22:16:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26326.360787 secs 2023-03-09T22:16:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:56 smithi122 ceph-mon[93869]: pgmap v14552: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 550 KiB/s rd, 425 KiB/s wr, 9 op/s 2023-03-09T22:16:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:56 smithi181 ceph-mon[94189]: pgmap v14552: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 550 KiB/s rd, 425 KiB/s wr, 9 op/s 2023-03-09T22:16:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:56 smithi038 ceph-mon[92422]: pgmap v14552: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 550 KiB/s rd, 425 KiB/s wr, 9 op/s 2023-03-09T22:16:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:16:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:16:57.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:16:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:16:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:16:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:16:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:58 smithi122 ceph-mon[93869]: pgmap v14553: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 745 KiB/s rd, 729 KiB/s wr, 13 op/s 2023-03-09T22:16:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:16:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26328.726489 secs 2023-03-09T22:16:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:58 smithi181 ceph-mon[94189]: pgmap v14553: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 745 KiB/s rd, 729 KiB/s wr, 13 op/s 2023-03-09T22:16:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:16:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26328.726489 secs 2023-03-09T22:16:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:58 smithi038 ceph-mon[92422]: pgmap v14553: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 745 KiB/s rd, 729 KiB/s wr, 13 op/s 2023-03-09T22:16:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:16:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26328.726489 secs 2023-03-09T22:16:58.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26790.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:17:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:00 smithi122 ceph-mon[93869]: pgmap v14554: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 614 KiB/s wr, 10 op/s 2023-03-09T22:17:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:17:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:17:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26331.360849 secs 2023-03-09T22:17:00.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:00 smithi181 ceph-mon[94189]: pgmap v14554: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 614 KiB/s wr, 10 op/s 2023-03-09T22:17:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:17:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:17:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26331.360849 secs 2023-03-09T22:17:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:00 smithi038 ceph-mon[92422]: pgmap v14554: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 614 KiB/s wr, 10 op/s 2023-03-09T22:17:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:17:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:17:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26331.360849 secs 2023-03-09T22:17:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:02 smithi122 ceph-mon[93869]: pgmap v14555: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 633 KiB/s wr, 13 op/s 2023-03-09T22:17:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:02 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:17:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:02 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:17:02.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:02 smithi181 ceph-mon[94189]: pgmap v14555: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 633 KiB/s wr, 13 op/s 2023-03-09T22:17:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:02 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:17:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:02 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:17:02.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:02 smithi038 ceph-mon[92422]: pgmap v14555: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 633 KiB/s wr, 13 op/s 2023-03-09T22:17:02.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:02 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:17:02.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:02 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:17:03.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26333.726622 secs 2023-03-09T22:17:03.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:03 smithi122 ceph-mon[93869]: pgmap v14556: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 486 KiB/s rd, 474 KiB/s wr, 10 op/s 2023-03-09T22:17:03.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26333.726622 secs 2023-03-09T22:17:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:03 smithi181 ceph-mon[94189]: pgmap v14556: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 486 KiB/s rd, 474 KiB/s wr, 10 op/s 2023-03-09T22:17:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26333.726622 secs 2023-03-09T22:17:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:03 smithi038 ceph-mon[92422]: pgmap v14556: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 486 KiB/s rd, 474 KiB/s wr, 10 op/s 2023-03-09T22:17:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26795.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:17:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:05 smithi122 ceph-mon[93869]: pgmap v14557: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 493 KiB/s rd, 484 KiB/s wr, 21 op/s 2023-03-09T22:17:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26336.360905 secs 2023-03-09T22:17:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:05 smithi181 ceph-mon[94189]: pgmap v14557: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 493 KiB/s rd, 484 KiB/s wr, 21 op/s 2023-03-09T22:17:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26336.360905 secs 2023-03-09T22:17:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:05 smithi038 ceph-mon[92422]: pgmap v14557: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 493 KiB/s rd, 484 KiB/s wr, 21 op/s 2023-03-09T22:17:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26336.360905 secs 2023-03-09T22:17:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:08 smithi122 ceph-mon[93869]: pgmap v14558: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 776 KiB/s rd, 517 KiB/s wr, 25 op/s 2023-03-09T22:17:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:17:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:17:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:08 smithi181 ceph-mon[94189]: pgmap v14558: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 776 KiB/s rd, 517 KiB/s wr, 25 op/s 2023-03-09T22:17:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:17:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:17:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:08 smithi038 ceph-mon[92422]: pgmap v14558: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 776 KiB/s rd, 517 KiB/s wr, 25 op/s 2023-03-09T22:17:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:17:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:17:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26800.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:17:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:09 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26338.726745 secs 2023-03-09T22:17:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:17:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:17:09.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:09 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26338.726745 secs 2023-03-09T22:17:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:17:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:17:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:09 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26338.726745 secs 2023-03-09T22:17:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:17:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:17:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:10 smithi122 ceph-mon[93869]: pgmap v14559: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 769 KiB/s rd, 511 KiB/s wr, 22 op/s 2023-03-09T22:17:10.548 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26341.361011 secs 2023-03-09T22:17:10.550 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:10 smithi038 ceph-mon[92422]: pgmap v14559: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 769 KiB/s rd, 511 KiB/s wr, 22 op/s 2023-03-09T22:17:10.550 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26341.361011 secs 2023-03-09T22:17:10.550 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:10 smithi181 ceph-mon[94189]: pgmap v14559: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 769 KiB/s rd, 511 KiB/s wr, 22 op/s 2023-03-09T22:17:10.550 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26341.361011 secs 2023-03-09T22:17:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:12 smithi122 ceph-mon[93869]: pgmap v14560: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 834 KiB/s rd, 574 KiB/s wr, 24 op/s 2023-03-09T22:17:12.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:12 smithi181 ceph-mon[94189]: pgmap v14560: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 834 KiB/s rd, 574 KiB/s wr, 24 op/s 2023-03-09T22:17:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:12 smithi038 ceph-mon[92422]: pgmap v14560: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 834 KiB/s rd, 574 KiB/s wr, 24 op/s 2023-03-09T22:17:13.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26805.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:17:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:14 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26343.726890 secs 2023-03-09T22:17:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:14 smithi122 ceph-mon[93869]: pgmap v14561: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 707 KiB/s rd, 449 KiB/s wr, 21 op/s 2023-03-09T22:17:14.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:14 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26343.726890 secs 2023-03-09T22:17:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:14 smithi181 ceph-mon[94189]: pgmap v14561: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 707 KiB/s rd, 449 KiB/s wr, 21 op/s 2023-03-09T22:17:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:14 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26343.726890 secs 2023-03-09T22:17:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:14 smithi038 ceph-mon[92422]: pgmap v14561: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 707 KiB/s rd, 449 KiB/s wr, 21 op/s 2023-03-09T22:17:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26346.361081 secs 2023-03-09T22:17:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26346.361081 secs 2023-03-09T22:17:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26346.361081 secs 2023-03-09T22:17:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:16 smithi122 ceph-mon[93869]: pgmap v14562: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 707 KiB/s rd, 449 KiB/s wr, 21 op/s 2023-03-09T22:17:16.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:16 smithi181 ceph-mon[94189]: pgmap v14562: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 707 KiB/s rd, 449 KiB/s wr, 21 op/s 2023-03-09T22:17:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:16 smithi038 ceph-mon[92422]: pgmap v14562: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 707 KiB/s rd, 449 KiB/s wr, 21 op/s 2023-03-09T22:17:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:18 smithi122 ceph-mon[93869]: pgmap v14563: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 723 KiB/s rd, 459 KiB/s wr, 13 op/s 2023-03-09T22:17:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:18 smithi181 ceph-mon[94189]: pgmap v14563: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 723 KiB/s rd, 459 KiB/s wr, 13 op/s 2023-03-09T22:17:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:18 smithi038 ceph-mon[92422]: pgmap v14563: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 723 KiB/s rd, 459 KiB/s wr, 13 op/s 2023-03-09T22:17:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26810.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:17:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:19 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26348.727010 secs 2023-03-09T22:17:19.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:19 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26348.727010 secs 2023-03-09T22:17:19.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:19 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26348.727010 secs 2023-03-09T22:17:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:20 smithi122 ceph-mon[93869]: pgmap v14564: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 427 KiB/s wr, 9 op/s 2023-03-09T22:17:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26351.361140 secs 2023-03-09T22:17:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:20 smithi181 ceph-mon[94189]: pgmap v14564: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 427 KiB/s wr, 9 op/s 2023-03-09T22:17:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26351.361140 secs 2023-03-09T22:17:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:20 smithi038 ceph-mon[92422]: pgmap v14564: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 427 KiB/s wr, 9 op/s 2023-03-09T22:17:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26351.361140 secs 2023-03-09T22:17:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:17:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:17:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:17:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:17:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:17:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:17:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:22 smithi122 ceph-mon[93869]: pgmap v14565: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 360 KiB/s wr, 11 op/s 2023-03-09T22:17:22.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:22 smithi181 ceph-mon[94189]: pgmap v14565: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 360 KiB/s wr, 11 op/s 2023-03-09T22:17:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:22 smithi038 ceph-mon[92422]: pgmap v14565: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 360 KiB/s wr, 11 op/s 2023-03-09T22:17:23.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26815.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:17:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:24 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26353.727121 secs 2023-03-09T22:17:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:24 smithi122 ceph-mon[93869]: pgmap v14566: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 297 KiB/s wr, 8 op/s 2023-03-09T22:17:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:24 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26353.727121 secs 2023-03-09T22:17:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:24 smithi181 ceph-mon[94189]: pgmap v14566: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 297 KiB/s wr, 8 op/s 2023-03-09T22:17:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:24 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26353.727121 secs 2023-03-09T22:17:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:24 smithi038 ceph-mon[92422]: pgmap v14566: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 297 KiB/s wr, 8 op/s 2023-03-09T22:17:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26356.361203 secs 2023-03-09T22:17:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26356.361203 secs 2023-03-09T22:17:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26356.361203 secs 2023-03-09T22:17:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:26 smithi122 ceph-mon[93869]: pgmap v14567: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 295 KiB/s wr, 9 op/s 2023-03-09T22:17:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:26 smithi181 ceph-mon[94189]: pgmap v14567: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 295 KiB/s wr, 9 op/s 2023-03-09T22:17:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:26 smithi038 ceph-mon[92422]: pgmap v14567: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 295 KiB/s wr, 9 op/s 2023-03-09T22:17:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:28 smithi122 ceph-mon[93869]: pgmap v14568: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 344 KiB/s wr, 12 op/s 2023-03-09T22:17:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:28 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:17:28.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:28 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:17:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:28 smithi181 ceph-mon[94189]: pgmap v14568: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 344 KiB/s wr, 12 op/s 2023-03-09T22:17:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:28 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:17:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:28 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:17:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:28 smithi038 ceph-mon[92422]: pgmap v14568: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 344 KiB/s wr, 12 op/s 2023-03-09T22:17:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:28 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:17:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:28 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:17:28.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26820.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:17:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:29 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26358.727239 secs 2023-03-09T22:17:29.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:29 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26358.727239 secs 2023-03-09T22:17:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:29 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26358.727239 secs 2023-03-09T22:17:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:30 smithi122 ceph-mon[93869]: pgmap v14569: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 512 KiB/s wr, 11 op/s 2023-03-09T22:17:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:30 smithi122 ceph-mon[93869]: osdmap e1125: 12 total, 12 up, 12 in 2023-03-09T22:17:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26361.361264 secs 2023-03-09T22:17:30.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:30 smithi181 ceph-mon[94189]: pgmap v14569: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 512 KiB/s wr, 11 op/s 2023-03-09T22:17:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:30 smithi181 ceph-mon[94189]: osdmap e1125: 12 total, 12 up, 12 in 2023-03-09T22:17:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26361.361264 secs 2023-03-09T22:17:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:30 smithi038 ceph-mon[92422]: pgmap v14569: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 512 KiB/s wr, 11 op/s 2023-03-09T22:17:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:30 smithi038 ceph-mon[92422]: osdmap e1125: 12 total, 12 up, 12 in 2023-03-09T22:17:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26361.361264 secs 2023-03-09T22:17:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:32 smithi122 ceph-mon[93869]: pgmap v14571: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 300 KiB/s wr, 11 op/s 2023-03-09T22:17:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:32 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:17:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:32 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:17:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:32 smithi181 ceph-mon[94189]: pgmap v14571: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 300 KiB/s wr, 11 op/s 2023-03-09T22:17:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:32 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:17:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:32 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:17:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:32 smithi038 ceph-mon[92422]: pgmap v14571: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 300 KiB/s wr, 11 op/s 2023-03-09T22:17:32.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:32 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:17:32.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:32 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:17:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:33 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:17:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:33 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:17:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26363.727357 secs 2023-03-09T22:17:33.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:33 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:17:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:33 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:17:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26363.727357 secs 2023-03-09T22:17:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:33 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:17:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:33 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:17:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26363.727357 secs 2023-03-09T22:17:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26825.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:17:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:34 smithi122 ceph-mon[93869]: pgmap v14572: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 300 KiB/s wr, 11 op/s 2023-03-09T22:17:34.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:34 smithi181 ceph-mon[94189]: pgmap v14572: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 300 KiB/s wr, 11 op/s 2023-03-09T22:17:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:34 smithi038 ceph-mon[92422]: pgmap v14572: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 300 KiB/s wr, 11 op/s 2023-03-09T22:17:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26366.361320 secs 2023-03-09T22:17:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26366.361320 secs 2023-03-09T22:17:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26366.361320 secs 2023-03-09T22:17:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:36 smithi122 ceph-mon[93869]: pgmap v14573: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 287 KiB/s wr, 10 op/s 2023-03-09T22:17:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:36 smithi122 ceph-mon[93869]: osdmap e1126: 12 total, 12 up, 12 in 2023-03-09T22:17:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:36 smithi181 ceph-mon[94189]: pgmap v14573: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 287 KiB/s wr, 10 op/s 2023-03-09T22:17:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:36 smithi181 ceph-mon[94189]: osdmap e1126: 12 total, 12 up, 12 in 2023-03-09T22:17:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:36 smithi038 ceph-mon[92422]: pgmap v14573: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 287 KiB/s wr, 10 op/s 2023-03-09T22:17:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:36 smithi038 ceph-mon[92422]: osdmap e1126: 12 total, 12 up, 12 in 2023-03-09T22:17:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:38 smithi122 ceph-mon[93869]: pgmap v14575: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 2.4 KiB/s wr, 10 op/s 2023-03-09T22:17:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26368.727460 secs 2023-03-09T22:17:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:38 smithi181 ceph-mon[94189]: pgmap v14575: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 2.4 KiB/s wr, 10 op/s 2023-03-09T22:17:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26368.727460 secs 2023-03-09T22:17:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:38 smithi038 ceph-mon[92422]: pgmap v14575: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 2.4 KiB/s wr, 10 op/s 2023-03-09T22:17:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26368.727460 secs 2023-03-09T22:17:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26830.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:17:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:39 smithi122 ceph-mon[93869]: pgmap v14576: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 152 KiB/s wr, 9 op/s 2023-03-09T22:17:39.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:17:39.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:39 smithi181 ceph-mon[94189]: pgmap v14576: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 152 KiB/s wr, 9 op/s 2023-03-09T22:17:39.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:17:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:39 smithi038 ceph-mon[92422]: pgmap v14576: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 152 KiB/s wr, 9 op/s 2023-03-09T22:17:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:17:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:17:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:17:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:17:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26371.361381 secs 2023-03-09T22:17:40.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:17:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:17:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:17:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26371.361381 secs 2023-03-09T22:17:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:17:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:17:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:17:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26371.361381 secs 2023-03-09T22:17:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:17:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:17:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:41 smithi122 ceph-mon[93869]: pgmap v14577: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 255 KiB/s wr, 8 op/s 2023-03-09T22:17:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:17:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:17:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:41 smithi181 ceph-mon[94189]: pgmap v14577: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 255 KiB/s wr, 8 op/s 2023-03-09T22:17:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:17:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:17:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:41 smithi038 ceph-mon[92422]: pgmap v14577: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 255 KiB/s wr, 8 op/s 2023-03-09T22:17:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26835.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:17:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26373.727575 secs 2023-03-09T22:17:44.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:43 smithi122 ceph-mon[93869]: pgmap v14578: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 255 KiB/s wr, 8 op/s 2023-03-09T22:17:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26373.727575 secs 2023-03-09T22:17:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:43 smithi181 ceph-mon[94189]: pgmap v14578: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 255 KiB/s wr, 8 op/s 2023-03-09T22:17:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26373.727575 secs 2023-03-09T22:17:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:43 smithi038 ceph-mon[92422]: pgmap v14578: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 255 KiB/s wr, 8 op/s 2023-03-09T22:17:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:45 smithi122 ceph-mon[93869]: pgmap v14579: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 293 KiB/s wr, 8 op/s 2023-03-09T22:17:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26376.361440 secs 2023-03-09T22:17:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:45 smithi181 ceph-mon[94189]: pgmap v14579: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 293 KiB/s wr, 8 op/s 2023-03-09T22:17:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26376.361440 secs 2023-03-09T22:17:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:45 smithi038 ceph-mon[92422]: pgmap v14579: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 293 KiB/s wr, 8 op/s 2023-03-09T22:17:46.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26376.361440 secs 2023-03-09T22:17:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:48 smithi122 ceph-mon[93869]: pgmap v14580: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 431 KiB/s wr, 8 op/s 2023-03-09T22:17:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:17:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:17:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:48 smithi181 ceph-mon[94189]: pgmap v14580: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 431 KiB/s wr, 8 op/s 2023-03-09T22:17:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:17:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:17:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:48 smithi038 ceph-mon[92422]: pgmap v14580: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 431 KiB/s wr, 8 op/s 2023-03-09T22:17:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:17:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:17:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26840.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:17:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:49 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26378.727710 secs 2023-03-09T22:17:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:49 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26378.727710 secs 2023-03-09T22:17:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:49 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26378.727710 secs 2023-03-09T22:17:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:50 smithi122 ceph-mon[93869]: pgmap v14581: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 422 KiB/s rd, 388 KiB/s wr, 8 op/s 2023-03-09T22:17:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26381.361500 secs 2023-03-09T22:17:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:50 smithi181 ceph-mon[94189]: pgmap v14581: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 422 KiB/s rd, 388 KiB/s wr, 8 op/s 2023-03-09T22:17:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26381.361500 secs 2023-03-09T22:17:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:50 smithi038 ceph-mon[92422]: pgmap v14581: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 422 KiB/s rd, 388 KiB/s wr, 8 op/s 2023-03-09T22:17:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26381.361500 secs 2023-03-09T22:17:52.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:52 smithi122 ceph-mon[93869]: pgmap v14582: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 533 KiB/s rd, 416 KiB/s wr, 11 op/s 2023-03-09T22:17:52.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:52 smithi181 ceph-mon[94189]: pgmap v14582: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 533 KiB/s rd, 416 KiB/s wr, 11 op/s 2023-03-09T22:17:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:52 smithi038 ceph-mon[92422]: pgmap v14582: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 533 KiB/s rd, 416 KiB/s wr, 11 op/s 2023-03-09T22:17:53.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26845.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:17:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:54 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26383.727839 secs 2023-03-09T22:17:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:54 smithi122 ceph-mon[93869]: pgmap v14583: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 440 KiB/s rd, 325 KiB/s wr, 9 op/s 2023-03-09T22:17:54.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:54 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26383.727839 secs 2023-03-09T22:17:54.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:54 smithi181 ceph-mon[94189]: pgmap v14583: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 440 KiB/s rd, 325 KiB/s wr, 9 op/s 2023-03-09T22:17:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:54 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26383.727839 secs 2023-03-09T22:17:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:54 smithi038 ceph-mon[92422]: pgmap v14583: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 440 KiB/s rd, 325 KiB/s wr, 9 op/s 2023-03-09T22:17:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26386.361562 secs 2023-03-09T22:17:55.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26386.361562 secs 2023-03-09T22:17:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26386.361562 secs 2023-03-09T22:17:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:56 smithi122 ceph-mon[93869]: pgmap v14584: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 700 KiB/s rd, 574 KiB/s wr, 11 op/s 2023-03-09T22:17:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:17:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:17:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:56 smithi181 ceph-mon[94189]: pgmap v14584: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 700 KiB/s rd, 574 KiB/s wr, 11 op/s 2023-03-09T22:17:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:17:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:17:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:56 smithi038 ceph-mon[92422]: pgmap v14584: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 700 KiB/s rd, 574 KiB/s wr, 11 op/s 2023-03-09T22:17:56.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:17:56.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:17:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:17:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:17:57.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:17:57.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:17:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:17:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:17:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:58 smithi122 ceph-mon[93869]: pgmap v14585: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 660 KiB/s rd, 564 KiB/s wr, 14 op/s 2023-03-09T22:17:58.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:58 smithi181 ceph-mon[94189]: pgmap v14585: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 660 KiB/s rd, 564 KiB/s wr, 14 op/s 2023-03-09T22:17:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:58 smithi038 ceph-mon[92422]: pgmap v14585: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 660 KiB/s rd, 564 KiB/s wr, 14 op/s 2023-03-09T22:17:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26850.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:17:59.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:17:59 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26388.727990 secs 2023-03-09T22:17:59.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:17:59 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26388.727990 secs 2023-03-09T22:17:59.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:17:59 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26388.727990 secs 2023-03-09T22:18:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:00 smithi122 ceph-mon[93869]: pgmap v14586: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 545 KiB/s rd, 450 KiB/s wr, 11 op/s 2023-03-09T22:18:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26391.361622 secs 2023-03-09T22:18:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:00 smithi181 ceph-mon[94189]: pgmap v14586: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 545 KiB/s rd, 450 KiB/s wr, 11 op/s 2023-03-09T22:18:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26391.361622 secs 2023-03-09T22:18:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:00 smithi038 ceph-mon[92422]: pgmap v14586: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 545 KiB/s rd, 450 KiB/s wr, 11 op/s 2023-03-09T22:18:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26391.361622 secs 2023-03-09T22:18:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:18:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:18:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:18:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:18:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:18:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:18:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:02 smithi122 ceph-mon[93869]: pgmap v14587: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 660 KiB/s rd, 564 KiB/s wr, 14 op/s 2023-03-09T22:18:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:02 smithi181 ceph-mon[94189]: pgmap v14587: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 660 KiB/s rd, 564 KiB/s wr, 14 op/s 2023-03-09T22:18:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:02 smithi038 ceph-mon[92422]: pgmap v14587: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 660 KiB/s rd, 564 KiB/s wr, 14 op/s 2023-03-09T22:18:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26855.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:18:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:04 smithi122 ceph-mon[93869]: pgmap v14588: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 416 KiB/s wr, 10 op/s 2023-03-09T22:18:04.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:04 smithi181 ceph-mon[94189]: pgmap v14588: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 416 KiB/s wr, 10 op/s 2023-03-09T22:18:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:04 smithi038 ceph-mon[92422]: pgmap v14588: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 416 KiB/s wr, 10 op/s 2023-03-09T22:18:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:05 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:18:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26396.361683 secs 2023-03-09T22:18:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:05 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:18:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26396.361683 secs 2023-03-09T22:18:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:05 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:18:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26396.361683 secs 2023-03-09T22:18:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:06 smithi122 ceph-mon[93869]: pgmap v14589: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 505 KiB/s rd, 492 KiB/s wr, 12 op/s 2023-03-09T22:18:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:06 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:18:06.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:06 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:18:06.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:18:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:06 smithi181 ceph-mon[94189]: pgmap v14589: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 505 KiB/s rd, 492 KiB/s wr, 12 op/s 2023-03-09T22:18:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:06 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:18:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:06 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:18:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:18:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:06 smithi038 ceph-mon[92422]: pgmap v14589: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 505 KiB/s rd, 492 KiB/s wr, 12 op/s 2023-03-09T22:18:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:06 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:18:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:06 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:18:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:18:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:08 smithi122 ceph-mon[93869]: pgmap v14590: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 286 KiB/s wr, 15 op/s 2023-03-09T22:18:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:18:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:18:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26398.728232 secs 2023-03-09T22:18:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:08 smithi181 ceph-mon[94189]: pgmap v14590: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 286 KiB/s wr, 15 op/s 2023-03-09T22:18:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:18:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:18:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26398.728232 secs 2023-03-09T22:18:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:08 smithi038 ceph-mon[92422]: pgmap v14590: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 286 KiB/s wr, 15 op/s 2023-03-09T22:18:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:18:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:18:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26398.728232 secs 2023-03-09T22:18:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26860.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:18:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:10 smithi122 ceph-mon[93869]: pgmap v14591: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 521 KiB/s wr, 24 op/s 2023-03-09T22:18:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:10 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:18:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:10 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:18:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:10 smithi122 ceph-mon[93869]: osdmap e1127: 12 total, 12 up, 12 in 2023-03-09T22:18:10.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26401.361741 secs 2023-03-09T22:18:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:10 smithi181 ceph-mon[94189]: pgmap v14591: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 521 KiB/s wr, 24 op/s 2023-03-09T22:18:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:10 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:18:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:10 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:18:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:10 smithi181 ceph-mon[94189]: osdmap e1127: 12 total, 12 up, 12 in 2023-03-09T22:18:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26401.361741 secs 2023-03-09T22:18:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:10 smithi038 ceph-mon[92422]: pgmap v14591: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 521 KiB/s wr, 24 op/s 2023-03-09T22:18:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:10 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:18:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:10 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:18:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:10 smithi038 ceph-mon[92422]: osdmap e1127: 12 total, 12 up, 12 in 2023-03-09T22:18:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26401.361741 secs 2023-03-09T22:18:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:11 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:18:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:11 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:18:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:11 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:18:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:12 smithi122 ceph-mon[93869]: pgmap v14593: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 690 KiB/s rd, 731 KiB/s wr, 28 op/s 2023-03-09T22:18:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:18:12.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:18:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:12 smithi181 ceph-mon[94189]: pgmap v14593: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 690 KiB/s rd, 731 KiB/s wr, 28 op/s 2023-03-09T22:18:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:18:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:18:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:12 smithi038 ceph-mon[92422]: pgmap v14593: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 690 KiB/s rd, 731 KiB/s wr, 28 op/s 2023-03-09T22:18:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:18:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:18:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:18:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:18:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26403.728320 secs 2023-03-09T22:18:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:18:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:18:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26403.728320 secs 2023-03-09T22:18:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:18:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:18:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26403.728320 secs 2023-03-09T22:18:13.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26865.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:18:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:14 smithi122 ceph-mon[93869]: pgmap v14594: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 690 KiB/s rd, 731 KiB/s wr, 28 op/s 2023-03-09T22:18:14.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:14 smithi181 ceph-mon[94189]: pgmap v14594: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 690 KiB/s rd, 731 KiB/s wr, 28 op/s 2023-03-09T22:18:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:14 smithi038 ceph-mon[92422]: pgmap v14594: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 690 KiB/s rd, 731 KiB/s wr, 28 op/s 2023-03-09T22:18:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26406.361801 secs 2023-03-09T22:18:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26406.361801 secs 2023-03-09T22:18:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26406.361801 secs 2023-03-09T22:18:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:16 smithi122 ceph-mon[93869]: pgmap v14595: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 786 KiB/s wr, 27 op/s 2023-03-09T22:18:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:16 smithi122 ceph-mon[93869]: osdmap e1128: 12 total, 12 up, 12 in 2023-03-09T22:18:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:16 smithi181 ceph-mon[94189]: pgmap v14595: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 786 KiB/s wr, 27 op/s 2023-03-09T22:18:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:16 smithi181 ceph-mon[94189]: osdmap e1128: 12 total, 12 up, 12 in 2023-03-09T22:18:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:16 smithi038 ceph-mon[92422]: pgmap v14595: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 786 KiB/s wr, 27 op/s 2023-03-09T22:18:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:16 smithi038 ceph-mon[92422]: osdmap e1128: 12 total, 12 up, 12 in 2023-03-09T22:18:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:17 smithi122 ceph-mon[93869]: pgmap v14597: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 566 KiB/s rd, 564 KiB/s wr, 12 op/s 2023-03-09T22:18:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:17 smithi181 ceph-mon[94189]: pgmap v14597: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 566 KiB/s rd, 564 KiB/s wr, 12 op/s 2023-03-09T22:18:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:17 smithi038 ceph-mon[92422]: pgmap v14597: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 566 KiB/s rd, 564 KiB/s wr, 12 op/s 2023-03-09T22:18:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26408.728420 secs 2023-03-09T22:18:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26408.728420 secs 2023-03-09T22:18:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26408.728420 secs 2023-03-09T22:18:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26870.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:18:19.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:19 smithi038 ceph-mon[92422]: pgmap v14598: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 273 KiB/s wr, 9 op/s 2023-03-09T22:18:19.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:19 smithi122 ceph-mon[93869]: pgmap v14598: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 273 KiB/s wr, 9 op/s 2023-03-09T22:18:19.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:19 smithi181 ceph-mon[94189]: pgmap v14598: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 273 KiB/s wr, 9 op/s 2023-03-09T22:18:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:18:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:18:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26411.361862 secs 2023-03-09T22:18:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:18:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:18:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26411.361862 secs 2023-03-09T22:18:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:18:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:18:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26411.361862 secs 2023-03-09T22:18:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:18:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:18:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:21 smithi038 ceph-mon[92422]: pgmap v14599: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 289 KiB/s wr, 9 op/s 2023-03-09T22:18:21.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:18:21.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:18:21.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:21 smithi122 ceph-mon[93869]: pgmap v14599: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 289 KiB/s wr, 9 op/s 2023-03-09T22:18:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:18:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:18:21.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:21 smithi181 ceph-mon[94189]: pgmap v14599: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 289 KiB/s wr, 9 op/s 2023-03-09T22:18:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26875.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:18:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26413.728506 secs 2023-03-09T22:18:24.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:23 smithi122 ceph-mon[93869]: pgmap v14600: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 289 KiB/s wr, 10 op/s 2023-03-09T22:18:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26413.728506 secs 2023-03-09T22:18:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:23 smithi181 ceph-mon[94189]: pgmap v14600: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 289 KiB/s wr, 10 op/s 2023-03-09T22:18:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26413.728506 secs 2023-03-09T22:18:24.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:23 smithi038 ceph-mon[92422]: pgmap v14600: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 289 KiB/s wr, 10 op/s 2023-03-09T22:18:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:25 smithi122 ceph-mon[93869]: pgmap v14601: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 143 KiB/s wr, 9 op/s 2023-03-09T22:18:26.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26416.361924 secs 2023-03-09T22:18:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:25 smithi181 ceph-mon[94189]: pgmap v14601: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 143 KiB/s wr, 9 op/s 2023-03-09T22:18:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26416.361924 secs 2023-03-09T22:18:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:25 smithi038 ceph-mon[92422]: pgmap v14601: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 143 KiB/s wr, 9 op/s 2023-03-09T22:18:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26416.361924 secs 2023-03-09T22:18:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:28 smithi122 ceph-mon[93869]: pgmap v14602: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 109 KiB/s wr, 8 op/s 2023-03-09T22:18:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:28 smithi181 ceph-mon[94189]: pgmap v14602: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 109 KiB/s wr, 8 op/s 2023-03-09T22:18:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:28 smithi038 ceph-mon[92422]: pgmap v14602: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 109 KiB/s wr, 8 op/s 2023-03-09T22:18:28.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26880.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:18:29.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:29 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26418.728627 secs 2023-03-09T22:18:29.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:29 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26418.728627 secs 2023-03-09T22:18:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:29 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26418.728627 secs 2023-03-09T22:18:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:30 smithi122 ceph-mon[93869]: pgmap v14603: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 285 KiB/s wr, 8 op/s 2023-03-09T22:18:30.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26421.361986 secs 2023-03-09T22:18:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:30 smithi181 ceph-mon[94189]: pgmap v14603: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 285 KiB/s wr, 8 op/s 2023-03-09T22:18:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26421.361986 secs 2023-03-09T22:18:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:30 smithi038 ceph-mon[92422]: pgmap v14603: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 285 KiB/s wr, 8 op/s 2023-03-09T22:18:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26421.361986 secs 2023-03-09T22:18:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:32 smithi122 ceph-mon[93869]: pgmap v14604: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 210 KiB/s wr, 9 op/s 2023-03-09T22:18:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:32 smithi181 ceph-mon[94189]: pgmap v14604: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 210 KiB/s wr, 9 op/s 2023-03-09T22:18:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:32 smithi038 ceph-mon[92422]: pgmap v14604: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 210 KiB/s wr, 9 op/s 2023-03-09T22:18:33.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26885.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:18:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:34 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26423.728765 secs 2023-03-09T22:18:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:34 smithi122 ceph-mon[93869]: pgmap v14605: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 189 KiB/s wr, 7 op/s 2023-03-09T22:18:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:34 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26423.728765 secs 2023-03-09T22:18:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:34 smithi181 ceph-mon[94189]: pgmap v14605: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 189 KiB/s wr, 7 op/s 2023-03-09T22:18:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:34 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26423.728765 secs 2023-03-09T22:18:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:34 smithi038 ceph-mon[92422]: pgmap v14605: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 189 KiB/s wr, 7 op/s 2023-03-09T22:18:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26426.362044 secs 2023-03-09T22:18:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26426.362044 secs 2023-03-09T22:18:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26426.362044 secs 2023-03-09T22:18:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:36 smithi122 ceph-mon[93869]: pgmap v14606: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 189 KiB/s wr, 7 op/s 2023-03-09T22:18:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:36 smithi181 ceph-mon[94189]: pgmap v14606: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 189 KiB/s wr, 7 op/s 2023-03-09T22:18:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:36 smithi038 ceph-mon[92422]: pgmap v14606: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 189 KiB/s wr, 7 op/s 2023-03-09T22:18:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:38 smithi122 ceph-mon[93869]: pgmap v14607: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 188 KiB/s wr, 8 op/s 2023-03-09T22:18:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:38 smithi181 ceph-mon[94189]: pgmap v14607: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 188 KiB/s wr, 8 op/s 2023-03-09T22:18:38.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:38 smithi038 ceph-mon[92422]: pgmap v14607: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 188 KiB/s wr, 8 op/s 2023-03-09T22:18:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26890.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:18:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:39 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26428.728884 secs 2023-03-09T22:18:39.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:39 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26428.728884 secs 2023-03-09T22:18:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:39 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26428.728884 secs 2023-03-09T22:18:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:40 smithi122 ceph-mon[93869]: pgmap v14608: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 254 KiB/s wr, 7 op/s 2023-03-09T22:18:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26431.362107 secs 2023-03-09T22:18:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:18:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:40 smithi181 ceph-mon[94189]: pgmap v14608: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 254 KiB/s wr, 7 op/s 2023-03-09T22:18:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26431.362107 secs 2023-03-09T22:18:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:18:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:40 smithi038 ceph-mon[92422]: pgmap v14608: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 254 KiB/s wr, 7 op/s 2023-03-09T22:18:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26431.362107 secs 2023-03-09T22:18:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:18:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:18:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:18:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:18:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:18:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:18:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:18:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:42 smithi122 ceph-mon[93869]: pgmap v14609: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 143 KiB/s wr, 9 op/s 2023-03-09T22:18:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:42 smithi181 ceph-mon[94189]: pgmap v14609: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 143 KiB/s wr, 9 op/s 2023-03-09T22:18:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:42 smithi038 ceph-mon[92422]: pgmap v14609: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 143 KiB/s wr, 9 op/s 2023-03-09T22:18:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26895.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:18:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:18:44.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:18:44.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:18:44.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:18:44.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:43 smithi122 ceph-mon[93869]: pgmap v14610: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 143 KiB/s wr, 6 op/s 2023-03-09T22:18:44.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:18:44.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:18:44.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:18:44.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:18:44.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:18:44.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:18:44.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:18:44.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:18:44.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:18:44.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:18:44.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:18:44.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:18:44.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:18:44.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:18:44.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:18:44.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:18:44.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:18:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:18:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:18:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:18:44.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:18:44.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:43 smithi181 ceph-mon[94189]: pgmap v14610: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 143 KiB/s wr, 6 op/s 2023-03-09T22:18:44.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:18:44.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:18:44.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:18:44.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:18:44.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:18:44.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:18:44.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:18:44.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:18:44.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:18:44.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:18:44.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:18:44.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:18:44.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:18:44.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:18:44.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:18:44.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:18:44.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:18:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:18:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:18:44.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:18:44.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:18:44.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:43 smithi038 ceph-mon[92422]: pgmap v14610: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 143 KiB/s wr, 6 op/s 2023-03-09T22:18:44.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:18:44.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:18:44.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:18:44.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:18:44.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:18:44.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:18:44.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:18:44.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:18:44.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:18:44.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:18:44.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:18:44.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:18:44.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:18:44.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:18:44.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:18:44.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:18:44.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:18:45.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:44 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T22:18:45.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:44 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T22:18:45.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:44 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi038 to 16168k 2023-03-09T22:18:45.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:44 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi038 to 16556343: error parsing value: Value '16556343' is below minimum 939524096 2023-03-09T22:18:45.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:44 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T22:18:45.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:44 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T22:18:45.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:44 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T22:18:45.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:44 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T22:18:45.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:44 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi038 to 16168k 2023-03-09T22:18:45.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:44 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi038 to 16556343: error parsing value: Value '16556343' is below minimum 939524096 2023-03-09T22:18:45.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:44 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T22:18:45.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:44 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T22:18:45.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:44 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T22:18:45.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:44 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T22:18:45.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:44 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi038 to 16168k 2023-03-09T22:18:45.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:44 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi038 to 16556343: error parsing value: Value '16556343' is below minimum 939524096 2023-03-09T22:18:45.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:44 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T22:18:45.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:44 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T22:18:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:45 smithi122 ceph-mon[93869]: pgmap v14611: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 143 KiB/s wr, 7 op/s 2023-03-09T22:18:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:45 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:18:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26436.362167 secs 2023-03-09T22:18:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:45 smithi181 ceph-mon[94189]: pgmap v14611: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 143 KiB/s wr, 7 op/s 2023-03-09T22:18:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:45 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:18:46.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26436.362167 secs 2023-03-09T22:18:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:45 smithi038 ceph-mon[92422]: pgmap v14611: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 143 KiB/s wr, 7 op/s 2023-03-09T22:18:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:45 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:18:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26436.362167 secs 2023-03-09T22:18:47.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:46 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:18:47.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:46 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:18:47.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:46 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:18:47.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:46 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:18:47.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:46 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:18:47.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:46 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:18:47.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:46 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:18:47.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:46 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:18:47.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:46 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:18:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:48 smithi122 ceph-mon[93869]: pgmap v14612: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 379 KiB/s wr, 9 op/s 2023-03-09T22:18:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:18:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:18:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:48 smithi181 ceph-mon[94189]: pgmap v14612: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 379 KiB/s wr, 9 op/s 2023-03-09T22:18:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:18:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:18:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:48 smithi038 ceph-mon[92422]: pgmap v14612: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 379 KiB/s wr, 9 op/s 2023-03-09T22:18:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:18:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:18:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26900.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:18:50.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:50 smithi122 ceph-mon[93869]: pgmap v14613: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 379 KiB/s wr, 8 op/s 2023-03-09T22:18:50.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26441.362246 secs 2023-03-09T22:18:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:50 smithi181 ceph-mon[94189]: pgmap v14613: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 379 KiB/s wr, 8 op/s 2023-03-09T22:18:50.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26441.362246 secs 2023-03-09T22:18:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:50 smithi038 ceph-mon[92422]: pgmap v14613: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 379 KiB/s wr, 8 op/s 2023-03-09T22:18:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26441.362246 secs 2023-03-09T22:18:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:52 smithi122 ceph-mon[93869]: pgmap v14614: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 385 KiB/s wr, 10 op/s 2023-03-09T22:18:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:52 smithi181 ceph-mon[94189]: pgmap v14614: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 385 KiB/s wr, 10 op/s 2023-03-09T22:18:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:52 smithi038 ceph-mon[92422]: pgmap v14614: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 385 KiB/s wr, 10 op/s 2023-03-09T22:18:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26905.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:18:54.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:54 smithi122 ceph-mon[93869]: pgmap v14615: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 313 KiB/s rd, 309 KiB/s wr, 8 op/s 2023-03-09T22:18:54.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:54 smithi181 ceph-mon[94189]: pgmap v14615: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 313 KiB/s rd, 309 KiB/s wr, 8 op/s 2023-03-09T22:18:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:54 smithi038 ceph-mon[92422]: pgmap v14615: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 313 KiB/s rd, 309 KiB/s wr, 8 op/s 2023-03-09T22:18:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26446.362304 secs 2023-03-09T22:18:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26446.362304 secs 2023-03-09T22:18:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26446.362304 secs 2023-03-09T22:18:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:56 smithi122 ceph-mon[93869]: pgmap v14616: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 590 KiB/s rd, 606 KiB/s wr, 10 op/s 2023-03-09T22:18:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:18:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:18:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:56 smithi181 ceph-mon[94189]: pgmap v14616: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 590 KiB/s rd, 606 KiB/s wr, 10 op/s 2023-03-09T22:18:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:18:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:18:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:56 smithi038 ceph-mon[92422]: pgmap v14616: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 590 KiB/s rd, 606 KiB/s wr, 10 op/s 2023-03-09T22:18:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:18:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:18:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:18:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:18:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:18:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:18:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:18:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:18:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:18:58 smithi122 ceph-mon[93869]: pgmap v14617: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 613 KiB/s rd, 629 KiB/s wr, 13 op/s 2023-03-09T22:18:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:18:58 smithi181 ceph-mon[94189]: pgmap v14617: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 613 KiB/s rd, 629 KiB/s wr, 13 op/s 2023-03-09T22:18:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:18:58 smithi038 ceph-mon[92422]: pgmap v14617: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 613 KiB/s rd, 629 KiB/s wr, 13 op/s 2023-03-09T22:18:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26910.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:19:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:00 smithi122 ceph-mon[93869]: pgmap v14618: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 456 KiB/s wr, 10 op/s 2023-03-09T22:19:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:19:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:19:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26451.362364 secs 2023-03-09T22:19:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:00 smithi181 ceph-mon[94189]: pgmap v14618: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 456 KiB/s wr, 10 op/s 2023-03-09T22:19:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:19:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:19:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26451.362364 secs 2023-03-09T22:19:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:00 smithi038 ceph-mon[92422]: pgmap v14618: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 456 KiB/s wr, 10 op/s 2023-03-09T22:19:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:19:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:19:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26451.362364 secs 2023-03-09T22:19:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:19:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:19:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:19:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:19:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:19:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:19:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:02 smithi122 ceph-mon[93869]: pgmap v14619: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 502 KiB/s wr, 14 op/s 2023-03-09T22:19:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:02 smithi181 ceph-mon[94189]: pgmap v14619: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 502 KiB/s wr, 14 op/s 2023-03-09T22:19:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:02 smithi038 ceph-mon[92422]: pgmap v14619: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 502 KiB/s wr, 14 op/s 2023-03-09T22:19:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26915.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:19:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:04 smithi122 ceph-mon[93869]: pgmap v14620: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 430 KiB/s wr, 10 op/s 2023-03-09T22:19:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:04 smithi181 ceph-mon[94189]: pgmap v14620: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 430 KiB/s wr, 10 op/s 2023-03-09T22:19:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:04 smithi038 ceph-mon[92422]: pgmap v14620: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 430 KiB/s wr, 10 op/s 2023-03-09T22:19:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26456.362423 secs 2023-03-09T22:19:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26456.362423 secs 2023-03-09T22:19:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26456.362423 secs 2023-03-09T22:19:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:06 smithi122 ceph-mon[93869]: pgmap v14621: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 494 KiB/s rd, 719 KiB/s wr, 22 op/s 2023-03-09T22:19:06.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:06 smithi181 ceph-mon[94189]: pgmap v14621: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 494 KiB/s rd, 719 KiB/s wr, 22 op/s 2023-03-09T22:19:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:06 smithi038 ceph-mon[92422]: pgmap v14621: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 494 KiB/s rd, 719 KiB/s wr, 22 op/s 2023-03-09T22:19:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:08 smithi122 ceph-mon[93869]: pgmap v14622: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 454 KiB/s wr, 24 op/s 2023-03-09T22:19:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:19:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:19:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:08 smithi181 ceph-mon[94189]: pgmap v14622: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 454 KiB/s wr, 24 op/s 2023-03-09T22:19:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:19:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:19:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:08 smithi038 ceph-mon[92422]: pgmap v14622: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 454 KiB/s wr, 24 op/s 2023-03-09T22:19:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:19:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:19:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26920.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:19:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:09 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 26458.729428 secs 2023-03-09T22:19:09.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:19:09.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:19:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:09 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 26458.729428 secs 2023-03-09T22:19:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:19:09.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:19:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:09 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 26458.729428 secs 2023-03-09T22:19:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:19:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:19:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:10 smithi122 ceph-mon[93869]: pgmap v14623: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 649 KiB/s wr, 24 op/s 2023-03-09T22:19:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:10 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:19:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:10 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:19:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:10 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:19:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26461.362482 secs 2023-03-09T22:19:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:10 smithi181 ceph-mon[94189]: pgmap v14623: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 649 KiB/s wr, 24 op/s 2023-03-09T22:19:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:10 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:19:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:10 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:19:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:10 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:19:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26461.362482 secs 2023-03-09T22:19:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:10 smithi038 ceph-mon[92422]: pgmap v14623: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 649 KiB/s wr, 24 op/s 2023-03-09T22:19:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:10 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:19:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:10 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:19:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:10 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:19:10.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26461.362482 secs 2023-03-09T22:19:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:12 smithi122 ceph-mon[93869]: pgmap v14624: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 774 KiB/s rd, 777 KiB/s wr, 27 op/s 2023-03-09T22:19:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:19:12.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:19:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:12 smithi181 ceph-mon[94189]: pgmap v14624: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 774 KiB/s rd, 777 KiB/s wr, 27 op/s 2023-03-09T22:19:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:19:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:19:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:12 smithi038 ceph-mon[92422]: pgmap v14624: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 774 KiB/s rd, 777 KiB/s wr, 27 op/s 2023-03-09T22:19:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:19:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:19:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:19:13.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:19:13.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26463.729535 secs 2023-03-09T22:19:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:19:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:19:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26463.729535 secs 2023-03-09T22:19:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:19:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:19:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26463.729535 secs 2023-03-09T22:19:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26925.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:19:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:14 smithi122 ceph-mon[93869]: pgmap v14625: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 831 KiB/s rd, 865 KiB/s wr, 24 op/s 2023-03-09T22:19:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:14 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:19:14.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:14 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:19:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:14 smithi181 ceph-mon[94189]: pgmap v14625: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 831 KiB/s rd, 865 KiB/s wr, 24 op/s 2023-03-09T22:19:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:14 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:19:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:14 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:19:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:14 smithi038 ceph-mon[92422]: pgmap v14625: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 831 KiB/s rd, 865 KiB/s wr, 24 op/s 2023-03-09T22:19:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:14 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:19:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:14 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:19:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:15 smithi122 ceph-mon[93869]: osdmap e1129: 12 total, 12 up, 12 in 2023-03-09T22:19:15.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26466.362567 secs 2023-03-09T22:19:15.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:15 smithi181 ceph-mon[94189]: osdmap e1129: 12 total, 12 up, 12 in 2023-03-09T22:19:15.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26466.362567 secs 2023-03-09T22:19:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:15 smithi038 ceph-mon[92422]: osdmap e1129: 12 total, 12 up, 12 in 2023-03-09T22:19:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26466.362567 secs 2023-03-09T22:19:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:16 smithi122 ceph-mon[93869]: pgmap v14627: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 933 KiB/s rd, 692 KiB/s wr, 15 op/s 2023-03-09T22:19:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:16 smithi181 ceph-mon[94189]: pgmap v14627: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 933 KiB/s rd, 692 KiB/s wr, 15 op/s 2023-03-09T22:19:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:16 smithi038 ceph-mon[92422]: pgmap v14627: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 933 KiB/s rd, 692 KiB/s wr, 15 op/s 2023-03-09T22:19:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:17 smithi122 ceph-mon[93869]: pgmap v14628: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 907 KiB/s rd, 653 KiB/s wr, 15 op/s 2023-03-09T22:19:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:17 smithi181 ceph-mon[94189]: pgmap v14628: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 907 KiB/s rd, 653 KiB/s wr, 15 op/s 2023-03-09T22:19:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:17 smithi038 ceph-mon[92422]: pgmap v14628: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 907 KiB/s rd, 653 KiB/s wr, 15 op/s 2023-03-09T22:19:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26468.729658 secs 2023-03-09T22:19:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26468.729658 secs 2023-03-09T22:19:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26468.729658 secs 2023-03-09T22:19:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26930.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:19:19.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:19 smithi122 ceph-mon[93869]: pgmap v14629: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 391 KiB/s wr, 12 op/s 2023-03-09T22:19:19.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:19 smithi181 ceph-mon[94189]: pgmap v14629: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 391 KiB/s wr, 12 op/s 2023-03-09T22:19:19.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:19 smithi038 ceph-mon[92422]: pgmap v14629: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 391 KiB/s wr, 12 op/s 2023-03-09T22:19:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26471.362629 secs 2023-03-09T22:19:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26471.362629 secs 2023-03-09T22:19:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26471.362629 secs 2023-03-09T22:19:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:22 smithi122 ceph-mon[93869]: pgmap v14630: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 466 KiB/s wr, 11 op/s 2023-03-09T22:19:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:22 smithi181 ceph-mon[94189]: pgmap v14630: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 466 KiB/s wr, 11 op/s 2023-03-09T22:19:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:22 smithi038 ceph-mon[92422]: pgmap v14630: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 466 KiB/s wr, 11 op/s 2023-03-09T22:19:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26935.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:19:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:24 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26473.729793 secs 2023-03-09T22:19:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:24 smithi122 ceph-mon[93869]: pgmap v14631: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 305 KiB/s wr, 10 op/s 2023-03-09T22:19:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:24 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26473.729793 secs 2023-03-09T22:19:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:24 smithi181 ceph-mon[94189]: pgmap v14631: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 305 KiB/s wr, 10 op/s 2023-03-09T22:19:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:24 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26473.729793 secs 2023-03-09T22:19:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:24 smithi038 ceph-mon[92422]: pgmap v14631: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 305 KiB/s wr, 10 op/s 2023-03-09T22:19:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26476.362690 secs 2023-03-09T22:19:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26476.362690 secs 2023-03-09T22:19:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26476.362690 secs 2023-03-09T22:19:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:26 smithi122 ceph-mon[93869]: pgmap v14632: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 286 KiB/s wr, 9 op/s 2023-03-09T22:19:26.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:26 smithi122 ceph-mon[93869]: osdmap e1130: 12 total, 12 up, 12 in 2023-03-09T22:19:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:26 smithi181 ceph-mon[94189]: pgmap v14632: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 286 KiB/s wr, 9 op/s 2023-03-09T22:19:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:26 smithi181 ceph-mon[94189]: osdmap e1130: 12 total, 12 up, 12 in 2023-03-09T22:19:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:26 smithi038 ceph-mon[92422]: pgmap v14632: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 286 KiB/s wr, 9 op/s 2023-03-09T22:19:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:26 smithi038 ceph-mon[92422]: osdmap e1130: 12 total, 12 up, 12 in 2023-03-09T22:19:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:28 smithi122 ceph-mon[93869]: pgmap v14634: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 399 KiB/s rd, 396 KiB/s wr, 9 op/s 2023-03-09T22:19:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:28 smithi181 ceph-mon[94189]: pgmap v14634: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 399 KiB/s rd, 396 KiB/s wr, 9 op/s 2023-03-09T22:19:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:28 smithi038 ceph-mon[92422]: pgmap v14634: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 399 KiB/s rd, 396 KiB/s wr, 9 op/s 2023-03-09T22:19:28.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26940.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:19:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:29 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26478.729913 secs 2023-03-09T22:19:29.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:29 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26478.729913 secs 2023-03-09T22:19:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:29 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26478.729913 secs 2023-03-09T22:19:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:30 smithi122 ceph-mon[93869]: pgmap v14635: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 399 KiB/s rd, 396 KiB/s wr, 8 op/s 2023-03-09T22:19:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26481.362749 secs 2023-03-09T22:19:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:30 smithi181 ceph-mon[94189]: pgmap v14635: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 399 KiB/s rd, 396 KiB/s wr, 8 op/s 2023-03-09T22:19:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26481.362749 secs 2023-03-09T22:19:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:30 smithi038 ceph-mon[92422]: pgmap v14635: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 399 KiB/s rd, 396 KiB/s wr, 8 op/s 2023-03-09T22:19:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26481.362749 secs 2023-03-09T22:19:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:32 smithi122 ceph-mon[93869]: pgmap v14636: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 KiB/s wr, 7 op/s 2023-03-09T22:19:32.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:32 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:19:32.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:32 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:19:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:32 smithi181 ceph-mon[94189]: pgmap v14636: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 KiB/s wr, 7 op/s 2023-03-09T22:19:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:32 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:19:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:32 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:19:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:32 smithi038 ceph-mon[92422]: pgmap v14636: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 KiB/s wr, 7 op/s 2023-03-09T22:19:32.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:32 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:19:32.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:32 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:19:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:33 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:19:33.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:33 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:19:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:33 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:19:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:33 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:19:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:33 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:19:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:33 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:19:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26945.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:19:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:34 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26483.730003 secs 2023-03-09T22:19:34.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:34 smithi122 ceph-mon[93869]: pgmap v14637: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 KiB/s wr, 7 op/s 2023-03-09T22:19:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:34 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26483.730003 secs 2023-03-09T22:19:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:34 smithi181 ceph-mon[94189]: pgmap v14637: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 KiB/s wr, 7 op/s 2023-03-09T22:19:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:34 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26483.730003 secs 2023-03-09T22:19:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:34 smithi038 ceph-mon[92422]: pgmap v14637: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 KiB/s wr, 7 op/s 2023-03-09T22:19:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26486.362810 secs 2023-03-09T22:19:35.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26486.362810 secs 2023-03-09T22:19:35.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26486.362810 secs 2023-03-09T22:19:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:36 smithi122 ceph-mon[93869]: pgmap v14638: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 KiB/s wr, 8 op/s 2023-03-09T22:19:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:36 smithi181 ceph-mon[94189]: pgmap v14638: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 KiB/s wr, 8 op/s 2023-03-09T22:19:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:36 smithi038 ceph-mon[92422]: pgmap v14638: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 KiB/s wr, 8 op/s 2023-03-09T22:19:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:38 smithi122 ceph-mon[93869]: pgmap v14639: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 94 B/s wr, 7 op/s 2023-03-09T22:19:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:38 smithi181 ceph-mon[94189]: pgmap v14639: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 94 B/s wr, 7 op/s 2023-03-09T22:19:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:38 smithi038 ceph-mon[92422]: pgmap v14639: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 94 B/s wr, 7 op/s 2023-03-09T22:19:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26950.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:19:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:39 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26488.730140 secs 2023-03-09T22:19:39.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:39 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26488.730140 secs 2023-03-09T22:19:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:39 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26488.730140 secs 2023-03-09T22:19:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:40 smithi122 ceph-mon[93869]: pgmap v14640: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 170 B/s wr, 6 op/s 2023-03-09T22:19:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26491.362868 secs 2023-03-09T22:19:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:40 smithi181 ceph-mon[94189]: pgmap v14640: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 170 B/s wr, 6 op/s 2023-03-09T22:19:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26491.362868 secs 2023-03-09T22:19:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:40 smithi038 ceph-mon[92422]: pgmap v14640: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 170 B/s wr, 6 op/s 2023-03-09T22:19:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26491.362868 secs 2023-03-09T22:19:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:42 smithi122 ceph-mon[93869]: pgmap v14641: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 140 KiB/s wr, 9 op/s 2023-03-09T22:19:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:42 smithi181 ceph-mon[94189]: pgmap v14641: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 140 KiB/s wr, 9 op/s 2023-03-09T22:19:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:42 smithi038 ceph-mon[92422]: pgmap v14641: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 140 KiB/s wr, 9 op/s 2023-03-09T22:19:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26493.730280 secs 2023-03-09T22:19:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26493.730280 secs 2023-03-09T22:19:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26493.730280 secs 2023-03-09T22:19:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26955.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:19:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:44 smithi122 ceph-mon[93869]: pgmap v14642: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 140 KiB/s wr, 6 op/s 2023-03-09T22:19:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:44 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:19:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:44 smithi181 ceph-mon[94189]: pgmap v14642: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 140 KiB/s wr, 6 op/s 2023-03-09T22:19:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:44 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:19:44.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:44 smithi038 ceph-mon[92422]: pgmap v14642: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 140 KiB/s wr, 6 op/s 2023-03-09T22:19:44.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:44 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:19:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:19:45.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:19:45.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:19:45.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26496.362929 secs 2023-03-09T22:19:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:19:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:19:45.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:19:45.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26496.362929 secs 2023-03-09T22:19:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:19:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:19:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:19:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26496.362929 secs 2023-03-09T22:19:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:46 smithi122 ceph-mon[93869]: pgmap v14643: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 140 KiB/s wr, 7 op/s 2023-03-09T22:19:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:46 smithi181 ceph-mon[94189]: pgmap v14643: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 140 KiB/s wr, 7 op/s 2023-03-09T22:19:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:46 smithi038 ceph-mon[92422]: pgmap v14643: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 140 KiB/s wr, 7 op/s 2023-03-09T22:19:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:48 smithi122 ceph-mon[93869]: pgmap v14644: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 332 KiB/s wr, 10 op/s 2023-03-09T22:19:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26498.730332 secs 2023-03-09T22:19:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:48 smithi181 ceph-mon[94189]: pgmap v14644: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 332 KiB/s wr, 10 op/s 2023-03-09T22:19:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26498.730332 secs 2023-03-09T22:19:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:48 smithi038 ceph-mon[92422]: pgmap v14644: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 332 KiB/s wr, 10 op/s 2023-03-09T22:19:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26498.730332 secs 2023-03-09T22:19:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26960.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:19:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:50 smithi122 ceph-mon[93869]: pgmap v14645: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 333 KiB/s wr, 8 op/s 2023-03-09T22:19:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26501.362985 secs 2023-03-09T22:19:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:50 smithi181 ceph-mon[94189]: pgmap v14645: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 333 KiB/s wr, 8 op/s 2023-03-09T22:19:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26501.362985 secs 2023-03-09T22:19:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:50 smithi038 ceph-mon[92422]: pgmap v14645: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 333 KiB/s wr, 8 op/s 2023-03-09T22:19:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26501.362985 secs 2023-03-09T22:19:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:52 smithi122 ceph-mon[93869]: pgmap v14646: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 688 KiB/s rd, 580 KiB/s wr, 12 op/s 2023-03-09T22:19:52.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:19:52.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:19:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:52 smithi181 ceph-mon[94189]: pgmap v14646: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 688 KiB/s rd, 580 KiB/s wr, 12 op/s 2023-03-09T22:19:52.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:19:52.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:19:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:52 smithi038 ceph-mon[92422]: pgmap v14646: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 688 KiB/s rd, 580 KiB/s wr, 12 op/s 2023-03-09T22:19:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:19:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:19:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:19:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:19:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26503.730445 secs 2023-03-09T22:19:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:19:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:19:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26503.730445 secs 2023-03-09T22:19:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:19:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:19:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26503.730445 secs 2023-03-09T22:19:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26965.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:19:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:54 smithi122 ceph-mon[93869]: pgmap v14647: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 441 KiB/s wr, 9 op/s 2023-03-09T22:19:54.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:54 smithi181 ceph-mon[94189]: pgmap v14647: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 441 KiB/s wr, 9 op/s 2023-03-09T22:19:54.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:54 smithi038 ceph-mon[92422]: pgmap v14647: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 441 KiB/s wr, 9 op/s 2023-03-09T22:19:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26506.363044 secs 2023-03-09T22:19:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26506.363044 secs 2023-03-09T22:19:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26506.363044 secs 2023-03-09T22:19:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:56 smithi122 ceph-mon[93869]: pgmap v14648: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 625 KiB/s wr, 10 op/s 2023-03-09T22:19:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:19:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:19:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:56 smithi181 ceph-mon[94189]: pgmap v14648: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 625 KiB/s wr, 10 op/s 2023-03-09T22:19:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:19:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:19:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:56 smithi038 ceph-mon[92422]: pgmap v14648: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 625 KiB/s wr, 10 op/s 2023-03-09T22:19:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:19:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:19:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:19:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:19:57.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:57 smithi181 ceph-mon[94189]: pgmap v14649: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 726 KiB/s rd, 647 KiB/s wr, 13 op/s 2023-03-09T22:19:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:19:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:19:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:57 smithi038 ceph-mon[92422]: pgmap v14649: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 726 KiB/s rd, 647 KiB/s wr, 13 op/s 2023-03-09T22:19:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:19:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:19:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:57 smithi122 ceph-mon[93869]: pgmap v14649: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 726 KiB/s rd, 647 KiB/s wr, 13 op/s 2023-03-09T22:19:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26508.730570 secs 2023-03-09T22:19:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26508.730570 secs 2023-03-09T22:19:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26508.730570 secs 2023-03-09T22:19:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26970.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:19:59.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:19:59 smithi038 ceph-mon[92422]: pgmap v14650: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 784 KiB/s rd, 613 KiB/s wr, 11 op/s 2023-03-09T22:19:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:19:59 smithi122 ceph-mon[93869]: pgmap v14650: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 784 KiB/s rd, 613 KiB/s wr, 11 op/s 2023-03-09T22:19:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:19:59 smithi181 ceph-mon[94189]: pgmap v14650: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 784 KiB/s rd, 613 KiB/s wr, 11 op/s 2023-03-09T22:20:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26511.363106 secs 2023-03-09T22:20:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:00 smithi038 ceph-mon[92422]: Health detail: HEALTH_WARN 2 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-09T22:20:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:00 smithi038 ceph-mon[92422]: [WRN] MDS_SLOW_REQUEST: 2 MDSs report slow requests 2023-03-09T22:20:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:00 smithi038 ceph-mon[92422]: mds.i(mds.2): 2 slow requests are blocked > 180 secs 2023-03-09T22:20:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:00 smithi038 ceph-mon[92422]: mds.h(mds.1): 2 slow requests are blocked > 180 secs 2023-03-09T22:20:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:00 smithi038 ceph-mon[92422]: [WRN] MDS_TRIM: 2 MDSs behind on trimming 2023-03-09T22:20:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:00 smithi038 ceph-mon[92422]: mds.e(mds.0): Behind on trimming (290/128) max_segments: 128, num_segments: 290 2023-03-09T22:20:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:00 smithi038 ceph-mon[92422]: mds.b(mds.0): Behind on trimming (290/128) max_segments: 128, num_segments: 290 2023-03-09T22:20:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26511.363106 secs 2023-03-09T22:20:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:00 smithi122 ceph-mon[93869]: Health detail: HEALTH_WARN 2 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-09T22:20:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:00 smithi122 ceph-mon[93869]: [WRN] MDS_SLOW_REQUEST: 2 MDSs report slow requests 2023-03-09T22:20:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:00 smithi122 ceph-mon[93869]: mds.i(mds.2): 2 slow requests are blocked > 180 secs 2023-03-09T22:20:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:00 smithi122 ceph-mon[93869]: mds.h(mds.1): 2 slow requests are blocked > 180 secs 2023-03-09T22:20:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:00 smithi122 ceph-mon[93869]: [WRN] MDS_TRIM: 2 MDSs behind on trimming 2023-03-09T22:20:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:00 smithi122 ceph-mon[93869]: mds.e(mds.0): Behind on trimming (290/128) max_segments: 128, num_segments: 290 2023-03-09T22:20:00.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:00 smithi122 ceph-mon[93869]: mds.b(mds.0): Behind on trimming (290/128) max_segments: 128, num_segments: 290 2023-03-09T22:20:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26511.363106 secs 2023-03-09T22:20:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:00 smithi181 ceph-mon[94189]: Health detail: HEALTH_WARN 2 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-09T22:20:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:00 smithi181 ceph-mon[94189]: [WRN] MDS_SLOW_REQUEST: 2 MDSs report slow requests 2023-03-09T22:20:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:00 smithi181 ceph-mon[94189]: mds.i(mds.2): 2 slow requests are blocked > 180 secs 2023-03-09T22:20:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:00 smithi181 ceph-mon[94189]: mds.h(mds.1): 2 slow requests are blocked > 180 secs 2023-03-09T22:20:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:00 smithi181 ceph-mon[94189]: [WRN] MDS_TRIM: 2 MDSs behind on trimming 2023-03-09T22:20:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:00 smithi181 ceph-mon[94189]: mds.e(mds.0): Behind on trimming (290/128) max_segments: 128, num_segments: 290 2023-03-09T22:20:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:00 smithi181 ceph-mon[94189]: mds.b(mds.0): Behind on trimming (290/128) max_segments: 128, num_segments: 290 2023-03-09T22:20:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:02 smithi122 ceph-mon[93869]: pgmap v14651: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 807 KiB/s rd, 635 KiB/s wr, 13 op/s 2023-03-09T22:20:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:02 smithi181 ceph-mon[94189]: pgmap v14651: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 807 KiB/s rd, 635 KiB/s wr, 13 op/s 2023-03-09T22:20:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:02 smithi038 ceph-mon[92422]: pgmap v14651: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 807 KiB/s rd, 635 KiB/s wr, 13 op/s 2023-03-09T22:20:03.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26975.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:20:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:04 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26513.730691 secs 2023-03-09T22:20:04.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:04 smithi122 ceph-mon[93869]: pgmap v14652: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 493 KiB/s rd, 387 KiB/s wr, 9 op/s 2023-03-09T22:20:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:04 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26513.730691 secs 2023-03-09T22:20:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:04 smithi181 ceph-mon[94189]: pgmap v14652: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 493 KiB/s rd, 387 KiB/s wr, 9 op/s 2023-03-09T22:20:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:04 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26513.730691 secs 2023-03-09T22:20:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:04 smithi038 ceph-mon[92422]: pgmap v14652: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 493 KiB/s rd, 387 KiB/s wr, 9 op/s 2023-03-09T22:20:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26516.363163 secs 2023-03-09T22:20:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26516.363163 secs 2023-03-09T22:20:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26516.363163 secs 2023-03-09T22:20:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:06 smithi122 ceph-mon[93869]: pgmap v14653: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1003 KiB/s rd, 843 KiB/s wr, 23 op/s 2023-03-09T22:20:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:06 smithi181 ceph-mon[94189]: pgmap v14653: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1003 KiB/s rd, 843 KiB/s wr, 23 op/s 2023-03-09T22:20:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:06 smithi038 ceph-mon[92422]: pgmap v14653: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1003 KiB/s rd, 843 KiB/s wr, 23 op/s 2023-03-09T22:20:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:08 smithi122 ceph-mon[93869]: pgmap v14654: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 702 KiB/s wr, 27 op/s 2023-03-09T22:20:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:20:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:20:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:08 smithi181 ceph-mon[94189]: pgmap v14654: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 702 KiB/s wr, 27 op/s 2023-03-09T22:20:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:20:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:20:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:08 smithi038 ceph-mon[92422]: pgmap v14654: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 702 KiB/s wr, 27 op/s 2023-03-09T22:20:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:20:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:20:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26980.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:20:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:09 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26518.730802 secs 2023-03-09T22:20:09.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:20:09.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:20:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:09 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26518.730802 secs 2023-03-09T22:20:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:20:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:20:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:09 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26518.730802 secs 2023-03-09T22:20:09.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:20:09.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:20:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:10 smithi122 ceph-mon[93869]: pgmap v14655: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 848 KiB/s rd, 681 KiB/s wr, 25 op/s 2023-03-09T22:20:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:10 smithi122 ceph-mon[93869]: osdmap e1131: 12 total, 12 up, 12 in 2023-03-09T22:20:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26521.363225 secs 2023-03-09T22:20:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:10 smithi181 ceph-mon[94189]: pgmap v14655: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 848 KiB/s rd, 681 KiB/s wr, 25 op/s 2023-03-09T22:20:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:10 smithi181 ceph-mon[94189]: osdmap e1131: 12 total, 12 up, 12 in 2023-03-09T22:20:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26521.363225 secs 2023-03-09T22:20:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:10 smithi038 ceph-mon[92422]: pgmap v14655: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 848 KiB/s rd, 681 KiB/s wr, 25 op/s 2023-03-09T22:20:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:10 smithi038 ceph-mon[92422]: osdmap e1131: 12 total, 12 up, 12 in 2023-03-09T22:20:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26521.363225 secs 2023-03-09T22:20:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:12 smithi122 ceph-mon[93869]: pgmap v14657: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1009 KiB/s wr, 30 op/s 2023-03-09T22:20:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:12 smithi181 ceph-mon[94189]: pgmap v14657: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1009 KiB/s wr, 30 op/s 2023-03-09T22:20:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:12 smithi038 ceph-mon[92422]: pgmap v14657: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1009 KiB/s wr, 30 op/s 2023-03-09T22:20:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26985.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:20:14.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:14 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26523.730864 secs 2023-03-09T22:20:14.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:14 smithi122 ceph-mon[93869]: pgmap v14658: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1009 KiB/s wr, 30 op/s 2023-03-09T22:20:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:14 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26523.730864 secs 2023-03-09T22:20:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:14 smithi181 ceph-mon[94189]: pgmap v14658: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1009 KiB/s wr, 30 op/s 2023-03-09T22:20:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:14 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26523.730864 secs 2023-03-09T22:20:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:14 smithi038 ceph-mon[92422]: pgmap v14658: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1009 KiB/s wr, 30 op/s 2023-03-09T22:20:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26526.363284 secs 2023-03-09T22:20:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26526.363284 secs 2023-03-09T22:20:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26526.363284 secs 2023-03-09T22:20:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:16 smithi122 ceph-mon[93869]: pgmap v14659: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 463 KiB/s wr, 13 op/s 2023-03-09T22:20:16.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:20:16.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:20:16.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:16 smithi122 ceph-mon[93869]: osdmap e1132: 12 total, 12 up, 12 in 2023-03-09T22:20:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:16 smithi181 ceph-mon[94189]: pgmap v14659: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 463 KiB/s wr, 13 op/s 2023-03-09T22:20:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:20:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:20:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:16 smithi181 ceph-mon[94189]: osdmap e1132: 12 total, 12 up, 12 in 2023-03-09T22:20:16.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:16 smithi038 ceph-mon[92422]: pgmap v14659: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 463 KiB/s wr, 13 op/s 2023-03-09T22:20:16.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:20:16.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:20:16.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:16 smithi038 ceph-mon[92422]: osdmap e1132: 12 total, 12 up, 12 in 2023-03-09T22:20:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:20:17.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:20:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:20:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:20:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:20:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:20:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:18 smithi122 ceph-mon[93869]: pgmap v14661: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 796 KiB/s rd, 793 KiB/s wr, 13 op/s 2023-03-09T22:20:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:18 smithi181 ceph-mon[94189]: pgmap v14661: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 796 KiB/s rd, 793 KiB/s wr, 13 op/s 2023-03-09T22:20:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:18 smithi038 ceph-mon[92422]: pgmap v14661: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 796 KiB/s rd, 793 KiB/s wr, 13 op/s 2023-03-09T22:20:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26990.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:20:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:19 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26528.730982 secs 2023-03-09T22:20:19.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:19 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26528.730982 secs 2023-03-09T22:20:19.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:19 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26528.730982 secs 2023-03-09T22:20:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:20 smithi122 ceph-mon[93869]: pgmap v14662: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 344 KiB/s wr, 9 op/s 2023-03-09T22:20:20.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26531.363343 secs 2023-03-09T22:20:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:20 smithi181 ceph-mon[94189]: pgmap v14662: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 344 KiB/s wr, 9 op/s 2023-03-09T22:20:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26531.363343 secs 2023-03-09T22:20:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:20 smithi038 ceph-mon[92422]: pgmap v14662: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 344 KiB/s wr, 9 op/s 2023-03-09T22:20:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26531.363343 secs 2023-03-09T22:20:22.398 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:22 smithi181 ceph-mon[94189]: pgmap v14663: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 302 KiB/s wr, 9 op/s 2023-03-09T22:20:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:22 smithi122 ceph-mon[93869]: pgmap v14663: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 302 KiB/s wr, 9 op/s 2023-03-09T22:20:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:22 smithi038 ceph-mon[92422]: pgmap v14663: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 302 KiB/s wr, 9 op/s 2023-03-09T22:20:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 26995.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:20:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:24 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26533.731138 secs 2023-03-09T22:20:24.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:24 smithi122 ceph-mon[93869]: pgmap v14664: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 354 KiB/s wr, 9 op/s 2023-03-09T22:20:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:24 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26533.731138 secs 2023-03-09T22:20:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:24 smithi181 ceph-mon[94189]: pgmap v14664: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 354 KiB/s wr, 9 op/s 2023-03-09T22:20:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:24 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26533.731138 secs 2023-03-09T22:20:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:24 smithi038 ceph-mon[92422]: pgmap v14664: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 354 KiB/s wr, 9 op/s 2023-03-09T22:20:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26536.363403 secs 2023-03-09T22:20:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26536.363403 secs 2023-03-09T22:20:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26536.363403 secs 2023-03-09T22:20:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:26 smithi122 ceph-mon[93869]: pgmap v14665: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 353 KiB/s wr, 9 op/s 2023-03-09T22:20:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:26 smithi181 ceph-mon[94189]: pgmap v14665: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 353 KiB/s wr, 9 op/s 2023-03-09T22:20:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:26 smithi038 ceph-mon[92422]: pgmap v14665: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 353 KiB/s wr, 9 op/s 2023-03-09T22:20:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:28 smithi122 ceph-mon[93869]: pgmap v14666: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 118 KiB/s wr, 7 op/s 2023-03-09T22:20:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:28 smithi181 ceph-mon[94189]: pgmap v14666: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 118 KiB/s wr, 7 op/s 2023-03-09T22:20:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:28 smithi038 ceph-mon[92422]: pgmap v14666: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 118 KiB/s wr, 7 op/s 2023-03-09T22:20:28.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27000.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:20:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:29 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26538.731200 secs 2023-03-09T22:20:29.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:29 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26538.731200 secs 2023-03-09T22:20:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:29 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26538.731200 secs 2023-03-09T22:20:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:30 smithi122 ceph-mon[93869]: pgmap v14667: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 107 KiB/s wr, 7 op/s 2023-03-09T22:20:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26541.363464 secs 2023-03-09T22:20:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:30 smithi181 ceph-mon[94189]: pgmap v14667: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 107 KiB/s wr, 7 op/s 2023-03-09T22:20:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26541.363464 secs 2023-03-09T22:20:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:30 smithi038 ceph-mon[92422]: pgmap v14667: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 107 KiB/s wr, 7 op/s 2023-03-09T22:20:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26541.363464 secs 2023-03-09T22:20:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:32 smithi122 ceph-mon[93869]: pgmap v14668: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 107 KiB/s wr, 10 op/s 2023-03-09T22:20:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:32 smithi181 ceph-mon[94189]: pgmap v14668: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 107 KiB/s wr, 10 op/s 2023-03-09T22:20:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:32 smithi038 ceph-mon[92422]: pgmap v14668: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 107 KiB/s wr, 10 op/s 2023-03-09T22:20:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26543.731331 secs 2023-03-09T22:20:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26543.731331 secs 2023-03-09T22:20:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26543.731331 secs 2023-03-09T22:20:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27005.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:20:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:34 smithi122 ceph-mon[93869]: pgmap v14669: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 44 KiB/s wr, 8 op/s 2023-03-09T22:20:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:34 smithi181 ceph-mon[94189]: pgmap v14669: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 44 KiB/s wr, 8 op/s 2023-03-09T22:20:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:34 smithi038 ceph-mon[92422]: pgmap v14669: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 44 KiB/s wr, 8 op/s 2023-03-09T22:20:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26546.363526 secs 2023-03-09T22:20:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26546.363526 secs 2023-03-09T22:20:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26546.363526 secs 2023-03-09T22:20:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:36 smithi122 ceph-mon[93869]: pgmap v14670: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 511 B/s wr, 8 op/s 2023-03-09T22:20:36.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:20:36.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:20:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:36 smithi181 ceph-mon[94189]: pgmap v14670: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 511 B/s wr, 8 op/s 2023-03-09T22:20:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:20:36.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:20:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:36 smithi038 ceph-mon[92422]: pgmap v14670: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 511 B/s wr, 8 op/s 2023-03-09T22:20:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:20:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:20:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:20:37.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:20:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:20:37.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:20:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:20:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:20:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:38 smithi038 ceph-mon[92422]: pgmap v14671: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 853 B/s wr, 10 op/s 2023-03-09T22:20:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26548.731463 secs 2023-03-09T22:20:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:38 smithi122 ceph-mon[93869]: pgmap v14671: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 853 B/s wr, 10 op/s 2023-03-09T22:20:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26548.731463 secs 2023-03-09T22:20:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:38 smithi181 ceph-mon[94189]: pgmap v14671: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 853 B/s wr, 10 op/s 2023-03-09T22:20:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26548.731463 secs 2023-03-09T22:20:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27010.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:20:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:39 smithi038 ceph-mon[92422]: pgmap v14672: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 113 KiB/s wr, 8 op/s 2023-03-09T22:20:39.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:39 smithi122 ceph-mon[93869]: pgmap v14672: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 113 KiB/s wr, 8 op/s 2023-03-09T22:20:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:39 smithi181 ceph-mon[94189]: pgmap v14672: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 113 KiB/s wr, 8 op/s 2023-03-09T22:20:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26551.363588 secs 2023-03-09T22:20:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26551.363588 secs 2023-03-09T22:20:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26551.363588 secs 2023-03-09T22:20:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:42 smithi122 ceph-mon[93869]: pgmap v14673: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 113 KiB/s wr, 9 op/s 2023-03-09T22:20:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:42 smithi181 ceph-mon[94189]: pgmap v14673: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 113 KiB/s wr, 9 op/s 2023-03-09T22:20:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:42 smithi038 ceph-mon[92422]: pgmap v14673: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 113 KiB/s wr, 9 op/s 2023-03-09T22:20:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27015.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:20:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:44 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26553.731591 secs 2023-03-09T22:20:44.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:44 smithi122 ceph-mon[93869]: pgmap v14674: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 236 KiB/s wr, 7 op/s 2023-03-09T22:20:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:44 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26553.731591 secs 2023-03-09T22:20:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:44 smithi181 ceph-mon[94189]: pgmap v14674: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 236 KiB/s wr, 7 op/s 2023-03-09T22:20:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:44 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26553.731591 secs 2023-03-09T22:20:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:44 smithi038 ceph-mon[92422]: pgmap v14674: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 236 KiB/s wr, 7 op/s 2023-03-09T22:20:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:20:45.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26556.363648 secs 2023-03-09T22:20:45.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:20:45.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:20:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:20:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26556.363648 secs 2023-03-09T22:20:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:20:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:20:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:20:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26556.363648 secs 2023-03-09T22:20:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:20:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:20:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:46 smithi122 ceph-mon[93869]: pgmap v14675: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 236 KiB/s wr, 6 op/s 2023-03-09T22:20:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:20:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:20:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:20:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:46 smithi181 ceph-mon[94189]: pgmap v14675: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 236 KiB/s wr, 6 op/s 2023-03-09T22:20:46.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:20:46.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:20:46.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:20:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:46 smithi038 ceph-mon[92422]: pgmap v14675: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 236 KiB/s wr, 6 op/s 2023-03-09T22:20:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:20:46.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:20:46.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:20:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:48 smithi122 ceph-mon[93869]: pgmap v14676: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 460 KiB/s wr, 9 op/s 2023-03-09T22:20:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:48 smithi181 ceph-mon[94189]: pgmap v14676: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 460 KiB/s wr, 9 op/s 2023-03-09T22:20:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:48 smithi038 ceph-mon[92422]: pgmap v14676: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 460 KiB/s wr, 9 op/s 2023-03-09T22:20:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27020.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:20:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:49 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:20:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:49 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:20:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:49 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:20:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:50 smithi122 ceph-mon[93869]: pgmap v14677: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 524 KiB/s rd, 522 KiB/s wr, 8 op/s 2023-03-09T22:20:50.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:50 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:20:50.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:50 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:20:50.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:50 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:20:50.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26561.363709 secs 2023-03-09T22:20:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:50 smithi181 ceph-mon[94189]: pgmap v14677: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 524 KiB/s rd, 522 KiB/s wr, 8 op/s 2023-03-09T22:20:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:50 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:20:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:50 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:20:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:50 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:20:50.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26561.363709 secs 2023-03-09T22:20:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:50 smithi038 ceph-mon[92422]: pgmap v14677: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 524 KiB/s rd, 522 KiB/s wr, 8 op/s 2023-03-09T22:20:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:50 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:20:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:50 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:20:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:50 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:20:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26561.363709 secs 2023-03-09T22:20:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:52 smithi122 ceph-mon[93869]: pgmap v14678: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 714 KiB/s wr, 12 op/s 2023-03-09T22:20:52.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:20:52.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:20:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:52 smithi181 ceph-mon[94189]: pgmap v14678: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 714 KiB/s wr, 12 op/s 2023-03-09T22:20:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:20:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:20:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:52 smithi038 ceph-mon[92422]: pgmap v14678: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 714 KiB/s wr, 12 op/s 2023-03-09T22:20:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:20:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:20:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:20:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:20:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:20:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:20:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:20:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:20:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27025.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:20:54.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:54 smithi122 ceph-mon[93869]: pgmap v14679: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 717 KiB/s rd, 714 KiB/s wr, 10 op/s 2023-03-09T22:20:54.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:54 smithi181 ceph-mon[94189]: pgmap v14679: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 717 KiB/s rd, 714 KiB/s wr, 10 op/s 2023-03-09T22:20:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:54 smithi038 ceph-mon[92422]: pgmap v14679: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 717 KiB/s rd, 714 KiB/s wr, 10 op/s 2023-03-09T22:20:55.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26566.363770 secs 2023-03-09T22:20:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26566.363770 secs 2023-03-09T22:20:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26566.363770 secs 2023-03-09T22:20:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:56 smithi122 ceph-mon[93869]: pgmap v14680: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 604 KiB/s rd, 591 KiB/s wr, 10 op/s 2023-03-09T22:20:56.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:20:56.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:20:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:56 smithi181 ceph-mon[94189]: pgmap v14680: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 604 KiB/s rd, 591 KiB/s wr, 10 op/s 2023-03-09T22:20:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:20:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:20:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:56 smithi038 ceph-mon[92422]: pgmap v14680: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 604 KiB/s rd, 591 KiB/s wr, 10 op/s 2023-03-09T22:20:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:20:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:20:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:20:57.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:20:57.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:20:57.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:20:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:20:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:20:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:20:58 smithi122 ceph-mon[93869]: pgmap v14681: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 898 KiB/s rd, 868 KiB/s wr, 15 op/s 2023-03-09T22:20:58.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:20:58 smithi181 ceph-mon[94189]: pgmap v14681: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 898 KiB/s rd, 868 KiB/s wr, 15 op/s 2023-03-09T22:20:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:20:58 smithi038 ceph-mon[92422]: pgmap v14681: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 898 KiB/s rd, 868 KiB/s wr, 15 op/s 2023-03-09T22:20:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27030.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:21:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:00 smithi122 ceph-mon[93869]: pgmap v14682: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 674 KiB/s rd, 645 KiB/s wr, 13 op/s 2023-03-09T22:21:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:21:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:21:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26571.363828 secs 2023-03-09T22:21:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:00 smithi181 ceph-mon[94189]: pgmap v14682: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 674 KiB/s rd, 645 KiB/s wr, 13 op/s 2023-03-09T22:21:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:21:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:21:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26571.363828 secs 2023-03-09T22:21:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:00 smithi038 ceph-mon[92422]: pgmap v14682: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 674 KiB/s rd, 645 KiB/s wr, 13 op/s 2023-03-09T22:21:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:21:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:21:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26571.363828 secs 2023-03-09T22:21:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:21:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:21:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:21:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:21:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:21:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:21:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:02 smithi122 ceph-mon[93869]: pgmap v14683: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 838 KiB/s rd, 808 KiB/s wr, 16 op/s 2023-03-09T22:21:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:02 smithi181 ceph-mon[94189]: pgmap v14683: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 838 KiB/s rd, 808 KiB/s wr, 16 op/s 2023-03-09T22:21:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:02 smithi038 ceph-mon[92422]: pgmap v14683: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 838 KiB/s rd, 808 KiB/s wr, 16 op/s 2023-03-09T22:21:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26573.732025 secs 2023-03-09T22:21:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26573.732025 secs 2023-03-09T22:21:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26573.732025 secs 2023-03-09T22:21:03.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27035.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:21:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:04 smithi122 ceph-mon[93869]: pgmap v14684: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 532 KiB/s rd, 505 KiB/s wr, 12 op/s 2023-03-09T22:21:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:04 smithi181 ceph-mon[94189]: pgmap v14684: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 532 KiB/s rd, 505 KiB/s wr, 12 op/s 2023-03-09T22:21:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:04 smithi038 ceph-mon[92422]: pgmap v14684: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 532 KiB/s rd, 505 KiB/s wr, 12 op/s 2023-03-09T22:21:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:05 smithi122 ceph-mon[93869]: osdmap e1133: 12 total, 12 up, 12 in 2023-03-09T22:21:05.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26576.363887 secs 2023-03-09T22:21:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:05 smithi181 ceph-mon[94189]: osdmap e1133: 12 total, 12 up, 12 in 2023-03-09T22:21:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26576.363887 secs 2023-03-09T22:21:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:05 smithi038 ceph-mon[92422]: osdmap e1133: 12 total, 12 up, 12 in 2023-03-09T22:21:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26576.363887 secs 2023-03-09T22:21:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:06 smithi122 ceph-mon[93869]: pgmap v14686: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 634 KiB/s rd, 630 KiB/s wr, 28 op/s 2023-03-09T22:21:06.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:06 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:21:06.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:06 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:21:06.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:21:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:06 smithi181 ceph-mon[94189]: pgmap v14686: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 634 KiB/s rd, 630 KiB/s wr, 28 op/s 2023-03-09T22:21:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:06 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:21:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:06 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:21:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:21:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:06 smithi038 ceph-mon[92422]: pgmap v14686: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 634 KiB/s rd, 630 KiB/s wr, 28 op/s 2023-03-09T22:21:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:06 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:21:06.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:06 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:21:06.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:21:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:08 smithi122 ceph-mon[93869]: pgmap v14687: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 519 KiB/s rd, 559 KiB/s wr, 30 op/s 2023-03-09T22:21:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26578.732089 secs 2023-03-09T22:21:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:08 smithi181 ceph-mon[94189]: pgmap v14687: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 519 KiB/s rd, 559 KiB/s wr, 30 op/s 2023-03-09T22:21:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26578.732089 secs 2023-03-09T22:21:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:08 smithi038 ceph-mon[92422]: pgmap v14687: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 519 KiB/s rd, 559 KiB/s wr, 30 op/s 2023-03-09T22:21:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26578.732089 secs 2023-03-09T22:21:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27040.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:21:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:21:09.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:21:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:21:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:21:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:21:09.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:21:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:10 smithi122 ceph-mon[93869]: pgmap v14688: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 519 KiB/s rd, 559 KiB/s wr, 30 op/s 2023-03-09T22:21:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26581.363957 secs 2023-03-09T22:21:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:10 smithi181 ceph-mon[94189]: pgmap v14688: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 519 KiB/s rd, 559 KiB/s wr, 30 op/s 2023-03-09T22:21:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26581.363957 secs 2023-03-09T22:21:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:10 smithi038 ceph-mon[92422]: pgmap v14688: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 519 KiB/s rd, 559 KiB/s wr, 30 op/s 2023-03-09T22:21:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26581.363957 secs 2023-03-09T22:21:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:12 smithi181 ceph-mon[94189]: pgmap v14689: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 420 KiB/s wr, 28 op/s 2023-03-09T22:21:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:21:12.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:21:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:12 smithi038 ceph-mon[92422]: pgmap v14689: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 420 KiB/s wr, 28 op/s 2023-03-09T22:21:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:21:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:21:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:12 smithi122 ceph-mon[93869]: pgmap v14689: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 420 KiB/s wr, 28 op/s 2023-03-09T22:21:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:21:12.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:21:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26583.732202 secs 2023-03-09T22:21:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26583.732202 secs 2023-03-09T22:21:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26583.732202 secs 2023-03-09T22:21:13.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27045.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:21:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:14 smithi038 ceph-mon[92422]: pgmap v14690: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 419 KiB/s wr, 27 op/s 2023-03-09T22:21:14.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:14 smithi122 ceph-mon[93869]: pgmap v14690: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 419 KiB/s wr, 27 op/s 2023-03-09T22:21:14.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:14 smithi181 ceph-mon[94189]: pgmap v14690: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 419 KiB/s wr, 27 op/s 2023-03-09T22:21:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:15 smithi122 ceph-mon[93869]: pgmap v14691: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 368 KiB/s wr, 12 op/s 2023-03-09T22:21:15.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26586.364006 secs 2023-03-09T22:21:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:15 smithi181 ceph-mon[94189]: pgmap v14691: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 368 KiB/s wr, 12 op/s 2023-03-09T22:21:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26586.364006 secs 2023-03-09T22:21:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:15 smithi038 ceph-mon[92422]: pgmap v14691: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 368 KiB/s wr, 12 op/s 2023-03-09T22:21:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26586.364006 secs 2023-03-09T22:21:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:17 smithi122 ceph-mon[93869]: osdmap e1134: 12 total, 12 up, 12 in 2023-03-09T22:21:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:17 smithi181 ceph-mon[94189]: osdmap e1134: 12 total, 12 up, 12 in 2023-03-09T22:21:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:17 smithi038 ceph-mon[92422]: osdmap e1134: 12 total, 12 up, 12 in 2023-03-09T22:21:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:18 smithi122 ceph-mon[93869]: pgmap v14693: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 131 KiB/s wr, 7 op/s 2023-03-09T22:21:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:18 smithi181 ceph-mon[94189]: pgmap v14693: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 131 KiB/s wr, 7 op/s 2023-03-09T22:21:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:18 smithi038 ceph-mon[92422]: pgmap v14693: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 131 KiB/s wr, 7 op/s 2023-03-09T22:21:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27050.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:21:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:19 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26588.732309 secs 2023-03-09T22:21:19.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:19 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26588.732309 secs 2023-03-09T22:21:19.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:19 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26588.732309 secs 2023-03-09T22:21:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:20 smithi122 ceph-mon[93869]: pgmap v14694: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 131 KiB/s wr, 6 op/s 2023-03-09T22:21:20.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26591.364067 secs 2023-03-09T22:21:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:20 smithi181 ceph-mon[94189]: pgmap v14694: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 131 KiB/s wr, 6 op/s 2023-03-09T22:21:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26591.364067 secs 2023-03-09T22:21:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:20 smithi038 ceph-mon[92422]: pgmap v14694: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 131 KiB/s wr, 6 op/s 2023-03-09T22:21:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26591.364067 secs 2023-03-09T22:21:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:22 smithi122 ceph-mon[93869]: pgmap v14695: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T22:21:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:22 smithi181 ceph-mon[94189]: pgmap v14695: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T22:21:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:22 smithi038 ceph-mon[92422]: pgmap v14695: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T22:21:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27055.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:21:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:24 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26593.732390 secs 2023-03-09T22:21:24.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:24 smithi122 ceph-mon[93869]: pgmap v14696: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T22:21:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:24 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26593.732390 secs 2023-03-09T22:21:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:24 smithi181 ceph-mon[94189]: pgmap v14696: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T22:21:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:24 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26593.732390 secs 2023-03-09T22:21:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:24 smithi038 ceph-mon[92422]: pgmap v14696: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T22:21:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26596.364130 secs 2023-03-09T22:21:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26596.364130 secs 2023-03-09T22:21:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26596.364130 secs 2023-03-09T22:21:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:26 smithi122 ceph-mon[93869]: pgmap v14697: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T22:21:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:26 smithi181 ceph-mon[94189]: pgmap v14697: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T22:21:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:26 smithi038 ceph-mon[92422]: pgmap v14697: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T22:21:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:28 smithi122 ceph-mon[93869]: pgmap v14698: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T22:21:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:28 smithi181 ceph-mon[94189]: pgmap v14698: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T22:21:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:28 smithi038 ceph-mon[92422]: pgmap v14698: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T22:21:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27060.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:21:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:29 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26598.732481 secs 2023-03-09T22:21:29.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:29 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26598.732481 secs 2023-03-09T22:21:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:29 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26598.732481 secs 2023-03-09T22:21:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:30 smithi122 ceph-mon[93869]: pgmap v14699: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T22:21:30.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26601.364190 secs 2023-03-09T22:21:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:30 smithi181 ceph-mon[94189]: pgmap v14699: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T22:21:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26601.364190 secs 2023-03-09T22:21:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:30 smithi038 ceph-mon[92422]: pgmap v14699: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T22:21:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26601.364190 secs 2023-03-09T22:21:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:32 smithi122 ceph-mon[93869]: pgmap v14700: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T22:21:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:32 smithi181 ceph-mon[94189]: pgmap v14700: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T22:21:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:32 smithi038 ceph-mon[92422]: pgmap v14700: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T22:21:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26603.732582 secs 2023-03-09T22:21:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26603.732582 secs 2023-03-09T22:21:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26603.732582 secs 2023-03-09T22:21:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27065.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:21:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:34 smithi122 ceph-mon[93869]: pgmap v14701: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T22:21:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:34 smithi181 ceph-mon[94189]: pgmap v14701: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T22:21:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:34 smithi038 ceph-mon[92422]: pgmap v14701: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T22:21:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26606.364268 secs 2023-03-09T22:21:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26606.364268 secs 2023-03-09T22:21:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26606.364268 secs 2023-03-09T22:21:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:36 smithi122 ceph-mon[93869]: pgmap v14702: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T22:21:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:36 smithi181 ceph-mon[94189]: pgmap v14702: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T22:21:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:36 smithi038 ceph-mon[92422]: pgmap v14702: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T22:21:38.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:38 smithi122 ceph-mon[93869]: pgmap v14703: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 93 KiB/s wr, 8 op/s 2023-03-09T22:21:38.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26608.732675 secs 2023-03-09T22:21:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:38 smithi181 ceph-mon[94189]: pgmap v14703: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 93 KiB/s wr, 8 op/s 2023-03-09T22:21:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26608.732675 secs 2023-03-09T22:21:38.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:38 smithi038 ceph-mon[92422]: pgmap v14703: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 93 KiB/s wr, 8 op/s 2023-03-09T22:21:38.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26608.732675 secs 2023-03-09T22:21:38.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27070.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:21:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:40 smithi122 ceph-mon[93869]: pgmap v14704: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 94 KiB/s wr, 6 op/s 2023-03-09T22:21:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26611.364328 secs 2023-03-09T22:21:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:40 smithi181 ceph-mon[94189]: pgmap v14704: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 94 KiB/s wr, 6 op/s 2023-03-09T22:21:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26611.364328 secs 2023-03-09T22:21:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:40 smithi038 ceph-mon[92422]: pgmap v14704: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 94 KiB/s wr, 6 op/s 2023-03-09T22:21:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26611.364328 secs 2023-03-09T22:21:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:21:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:21:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:21:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:21:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:21:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:21:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:42 smithi038 ceph-mon[92422]: pgmap v14705: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 429 KiB/s wr, 10 op/s 2023-03-09T22:21:42.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:42 smithi122 ceph-mon[93869]: pgmap v14705: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 429 KiB/s wr, 10 op/s 2023-03-09T22:21:42.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:42 smithi181 ceph-mon[94189]: pgmap v14705: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 429 KiB/s wr, 10 op/s 2023-03-09T22:21:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:43 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 26613.732768 secs 2023-03-09T22:21:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:43 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 26613.732768 secs 2023-03-09T22:21:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:43 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 26613.732768 secs 2023-03-09T22:21:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27075.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:21:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:44 smithi038 ceph-mon[92422]: pgmap v14706: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 429 KiB/s wr, 8 op/s 2023-03-09T22:21:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:44 smithi122 ceph-mon[93869]: pgmap v14706: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 429 KiB/s wr, 8 op/s 2023-03-09T22:21:44.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:44 smithi181 ceph-mon[94189]: pgmap v14706: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 429 KiB/s wr, 8 op/s 2023-03-09T22:21:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:45 smithi122 ceph-mon[93869]: pgmap v14707: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 429 KiB/s wr, 8 op/s 2023-03-09T22:21:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26616.364390 secs 2023-03-09T22:21:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:45 smithi181 ceph-mon[94189]: pgmap v14707: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 429 KiB/s wr, 8 op/s 2023-03-09T22:21:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26616.364390 secs 2023-03-09T22:21:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:45 smithi038 ceph-mon[92422]: pgmap v14707: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 429 KiB/s wr, 8 op/s 2023-03-09T22:21:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26616.364390 secs 2023-03-09T22:21:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:46 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:21:46.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:46 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:21:46.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:21:46.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:21:46.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:21:46.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:21:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:46 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:21:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:46 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:21:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:21:46.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:21:46.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:21:46.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:21:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:46 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:21:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:46 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:21:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:21:46.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:21:46.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:21:46.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:21:47.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:21:47.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:47 smithi122 ceph-mon[93869]: pgmap v14708: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 530 KiB/s rd, 525 KiB/s wr, 12 op/s 2023-03-09T22:21:47.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:21:47.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:21:47.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:21:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:21:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:47 smithi181 ceph-mon[94189]: pgmap v14708: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 530 KiB/s rd, 525 KiB/s wr, 12 op/s 2023-03-09T22:21:47.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:21:47.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:21:47.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:21:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:21:47.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:47 smithi038 ceph-mon[92422]: pgmap v14708: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 530 KiB/s rd, 525 KiB/s wr, 12 op/s 2023-03-09T22:21:47.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:21:47.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:21:47.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:21:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:21:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:21:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:48 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 26618.732892 secs 2023-03-09T22:21:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:21:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:21:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:48 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 26618.732892 secs 2023-03-09T22:21:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:21:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:21:48.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:48 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 26618.732892 secs 2023-03-09T22:21:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27080.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:21:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:49 smithi122 ceph-mon[93869]: pgmap v14709: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 681 KiB/s rd, 678 KiB/s wr, 11 op/s 2023-03-09T22:21:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:49 smithi181 ceph-mon[94189]: pgmap v14709: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 681 KiB/s rd, 678 KiB/s wr, 11 op/s 2023-03-09T22:21:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:49 smithi038 ceph-mon[92422]: pgmap v14709: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 681 KiB/s rd, 678 KiB/s wr, 11 op/s 2023-03-09T22:21:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26621.364447 secs 2023-03-09T22:21:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26621.364447 secs 2023-03-09T22:21:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26621.364447 secs 2023-03-09T22:21:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:52 smithi122 ceph-mon[93869]: pgmap v14710: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 703 KiB/s wr, 14 op/s 2023-03-09T22:21:52.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:21:52.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:21:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:52 smithi181 ceph-mon[94189]: pgmap v14710: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 703 KiB/s wr, 14 op/s 2023-03-09T22:21:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:21:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:21:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:52 smithi038 ceph-mon[92422]: pgmap v14710: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 703 KiB/s wr, 14 op/s 2023-03-09T22:21:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:21:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:21:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26623.732966 secs 2023-03-09T22:21:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:21:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:21:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:53 smithi122 ceph-mon[93869]: pgmap v14711: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 590 KiB/s rd, 555 KiB/s wr, 12 op/s 2023-03-09T22:21:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26623.732966 secs 2023-03-09T22:21:53.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:21:53.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:21:53.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:53 smithi181 ceph-mon[94189]: pgmap v14711: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 590 KiB/s rd, 555 KiB/s wr, 12 op/s 2023-03-09T22:21:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26623.732966 secs 2023-03-09T22:21:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:21:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:21:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:53 smithi038 ceph-mon[92422]: pgmap v14711: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 590 KiB/s rd, 555 KiB/s wr, 12 op/s 2023-03-09T22:21:53.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27085.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:21:54.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:54 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:21:54.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:54 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:21:54.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:54 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:21:54.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:54 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:21:54.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:54 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:21:54.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:54 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:21:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:55 smithi122 ceph-mon[93869]: pgmap v14712: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 590 KiB/s rd, 555 KiB/s wr, 12 op/s 2023-03-09T22:21:55.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26626.364531 secs 2023-03-09T22:21:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:55 smithi181 ceph-mon[94189]: pgmap v14712: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 590 KiB/s rd, 555 KiB/s wr, 12 op/s 2023-03-09T22:21:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26626.364531 secs 2023-03-09T22:21:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:55 smithi038 ceph-mon[92422]: pgmap v14712: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 590 KiB/s rd, 555 KiB/s wr, 12 op/s 2023-03-09T22:21:55.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26626.364531 secs 2023-03-09T22:21:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:21:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:21:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:21:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:21:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:21:56.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:21:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:21:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:21:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:57 smithi122 ceph-mon[93869]: pgmap v14713: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 612 KiB/s rd, 577 KiB/s wr, 14 op/s 2023-03-09T22:21:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:21:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:21:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:57 smithi181 ceph-mon[94189]: pgmap v14713: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 612 KiB/s rd, 577 KiB/s wr, 14 op/s 2023-03-09T22:21:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:21:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:21:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:57 smithi038 ceph-mon[92422]: pgmap v14713: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 612 KiB/s rd, 577 KiB/s wr, 14 op/s 2023-03-09T22:21:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26628.733081 secs 2023-03-09T22:21:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26628.733081 secs 2023-03-09T22:21:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26628.733081 secs 2023-03-09T22:21:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27090.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:21:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:21:59 smithi122 ceph-mon[93869]: pgmap v14714: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 617 KiB/s rd, 606 KiB/s wr, 12 op/s 2023-03-09T22:21:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:21:59 smithi181 ceph-mon[94189]: pgmap v14714: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 617 KiB/s rd, 606 KiB/s wr, 12 op/s 2023-03-09T22:21:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:21:59 smithi038 ceph-mon[92422]: pgmap v14714: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 617 KiB/s rd, 606 KiB/s wr, 12 op/s 2023-03-09T22:22:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:22:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26631.364617 secs 2023-03-09T22:22:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:22:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26631.364617 secs 2023-03-09T22:22:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:22:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26631.364617 secs 2023-03-09T22:22:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:22:02 smithi122 ceph-mon[93869]: pgmap v14715: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 401 KiB/s wr, 13 op/s 2023-03-09T22:22:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:22:02 smithi181 ceph-mon[94189]: pgmap v14715: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 401 KiB/s wr, 13 op/s 2023-03-09T22:22:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:22:02 smithi038 ceph-mon[92422]: pgmap v14715: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 401 KiB/s wr, 13 op/s 2023-03-09T22:22:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27095.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:22:04.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:22:04 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26633.733192 secs 2023-03-09T22:22:04.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:22:04 smithi122 ceph-mon[93869]: pgmap v14716: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 501 KiB/s wr, 10 op/s 2023-03-09T22:22:04.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:22:04 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26633.733192 secs 2023-03-09T22:22:04.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:22:04 smithi181 ceph-mon[94189]: pgmap v14716: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 501 KiB/s wr, 10 op/s 2023-03-09T22:22:04.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:22:04 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26633.733192 secs 2023-03-09T22:22:04.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:22:04 smithi038 ceph-mon[92422]: pgmap v14716: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 501 KiB/s wr, 10 op/s 2023-03-09T22:22:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:22:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26636.364678 secs 2023-03-09T22:22:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:22:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26636.364678 secs 2023-03-09T22:22:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:22:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26636.364678 secs 2023-03-09T22:22:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:22:06 smithi122 ceph-mon[93869]: pgmap v14717: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 384 KiB/s rd, 397 KiB/s wr, 22 op/s 2023-03-09T22:22:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:22:06 smithi181 ceph-mon[94189]: pgmap v14717: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 384 KiB/s rd, 397 KiB/s wr, 22 op/s 2023-03-09T22:22:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:22:06 smithi038 ceph-mon[92422]: pgmap v14717: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 384 KiB/s rd, 397 KiB/s wr, 22 op/s 2023-03-09T22:22:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:22:08 smithi122 ceph-mon[93869]: pgmap v14718: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 386 KiB/s rd, 419 KiB/s wr, 26 op/s 2023-03-09T22:22:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:22:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:22:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:22:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:22:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:22:08 smithi181 ceph-mon[94189]: pgmap v14718: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 386 KiB/s rd, 419 KiB/s wr, 26 op/s 2023-03-09T22:22:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:22:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:22:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:22:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:22:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:22:08 smithi038 ceph-mon[92422]: pgmap v14718: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 386 KiB/s rd, 419 KiB/s wr, 26 op/s 2023-03-09T22:22:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:22:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:22:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:22:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:22:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27100.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:22:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:22:09 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26638.733314 secs 2023-03-09T22:22:09.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:22:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:22:09.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:22:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:22:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:22:09 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26638.733314 secs 2023-03-09T22:22:09.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:22:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:22:09.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:22:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:22:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:22:09 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26638.733314 secs 2023-03-09T22:22:09.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:22:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:22:09.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:22:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:22:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:22:10 smithi122 ceph-mon[93869]: pgmap v14719: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 398 KiB/s wr, 24 op/s 2023-03-09T22:22:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:22:10 smithi122 ceph-mon[93869]: osdmap e1135: 12 total, 12 up, 12 in 2023-03-09T22:22:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:22:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26641.364738 secs 2023-03-09T22:22:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:22:10 smithi181 ceph-mon[94189]: pgmap v14719: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 398 KiB/s wr, 24 op/s 2023-03-09T22:22:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:22:10 smithi181 ceph-mon[94189]: osdmap e1135: 12 total, 12 up, 12 in 2023-03-09T22:22:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:22:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26641.364738 secs 2023-03-09T22:22:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:22:10 smithi038 ceph-mon[92422]: pgmap v14719: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 398 KiB/s wr, 24 op/s 2023-03-09T22:22:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:22:10 smithi038 ceph-mon[92422]: osdmap e1135: 12 total, 12 up, 12 in 2023-03-09T22:22:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:22:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26641.364738 secs 2023-03-09T22:22:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:22:12 smithi122 ceph-mon[93869]: pgmap v14721: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 409 KiB/s wr, 26 op/s 2023-03-09T22:22:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:22:12 smithi181 ceph-mon[94189]: pgmap v14721: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 409 KiB/s wr, 26 op/s 2023-03-09T22:22:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:22:12 smithi038 ceph-mon[92422]: pgmap v14721: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 409 KiB/s wr, 26 op/s 2023-03-09T22:22:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27105.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:22:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:22:14 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26643.733471 secs 2023-03-09T22:22:14.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:22:14 smithi122 ceph-mon[93869]: pgmap v14722: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 259 KiB/s wr, 25 op/s 2023-03-09T22:22:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:22:14 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26643.733471 secs 2023-03-09T22:22:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:22:14 smithi181 ceph-mon[94189]: pgmap v14722: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 259 KiB/s wr, 25 op/s 2023-03-09T22:22:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:22:14 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26643.733471 secs 2023-03-09T22:22:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:22:14 smithi038 ceph-mon[92422]: pgmap v14722: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 259 KiB/s wr, 25 op/s 2023-03-09T22:22:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:22:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26646.364801 secs 2023-03-09T22:22:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:22:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26646.364801 secs 2023-03-09T22:22:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:22:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26646.364801 secs 2023-03-09T22:22:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:22:16 smithi122 ceph-mon[93869]: pgmap v14723: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 158 KiB/s wr, 10 op/s 2023-03-09T22:22:16.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:22:16 smithi122 ceph-mon[93869]: osdmap e1136: 12 total, 12 up, 12 in 2023-03-09T22:22:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:22:16 smithi181 ceph-mon[94189]: pgmap v14723: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 158 KiB/s wr, 10 op/s 2023-03-09T22:22:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:22:16 smithi181 ceph-mon[94189]: osdmap e1136: 12 total, 12 up, 12 in 2023-03-09T22:22:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:22:16 smithi038 ceph-mon[92422]: pgmap v14723: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 158 KiB/s wr, 10 op/s 2023-03-09T22:22:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:22:16 smithi038 ceph-mon[92422]: osdmap e1136: 12 total, 12 up, 12 in 2023-03-09T22:22:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:22:18 smithi122 ceph-mon[93869]: pgmap v14725: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 164 KiB/s wr, 8 op/s 2023-03-09T22:22:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:22:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26648.733587 secs 2023-03-09T22:22:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:22:18 smithi181 ceph-mon[94189]: pgmap v14725: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 164 KiB/s wr, 8 op/s 2023-03-09T22:22:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:22:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26648.733587 secs 2023-03-09T22:22:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:22:18 smithi038 ceph-mon[92422]: pgmap v14725: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 164 KiB/s wr, 8 op/s 2023-03-09T22:22:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:22:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26648.733587 secs 2023-03-09T22:22:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27110.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:22:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:22:20 smithi122 ceph-mon[93869]: pgmap v14726: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 104 B/s wr, 6 op/s 2023-03-09T22:22:20.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:22:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26651.364860 secs 2023-03-09T22:22:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:22:20 smithi181 ceph-mon[94189]: pgmap v14726: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 104 B/s wr, 6 op/s 2023-03-09T22:22:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:22:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26651.364860 secs 2023-03-09T22:22:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:22:20 smithi038 ceph-mon[92422]: pgmap v14726: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 104 B/s wr, 6 op/s 2023-03-09T22:22:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:22:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26651.364860 secs 2023-03-09T22:22:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:22:22 smithi122 ceph-mon[93869]: pgmap v14727: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T22:22:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:22:22 smithi181 ceph-mon[94189]: pgmap v14727: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T22:22:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:22:22 smithi038 ceph-mon[92422]: pgmap v14727: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T22:22:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:22:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26653.733707 secs 2023-03-09T22:22:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:22:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26653.733707 secs 2023-03-09T22:22:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:22:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26653.733707 secs 2023-03-09T22:22:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27115.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:22:24.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:22:24 smithi122 ceph-mon[93869]: pgmap v14728: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T22:22:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:22:24 smithi181 ceph-mon[94189]: pgmap v14728: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T22:22:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:22:24 smithi038 ceph-mon[92422]: pgmap v14728: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2023-03-09T22:22:25.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:22:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26656.364920 secs 2023-03-09T22:22:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:22:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26656.364920 secs 2023-03-09T22:22:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:22:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26656.364920 secs 2023-03-09T22:22:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:22:26 smithi181 ceph-mon[94189]: pgmap v14729: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T22:22:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:22:26 smithi038 ceph-mon[92422]: pgmap v14729: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T22:22:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:22:26 smithi122 ceph-mon[93869]: pgmap v14729: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T22:22:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:22:28 smithi038 ceph-mon[92422]: pgmap v14730: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 94 B/s wr, 5 op/s 2023-03-09T22:22:28.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:22:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26658.733819 secs 2023-03-09T22:22:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:22:28 smithi122 ceph-mon[93869]: pgmap v14730: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 94 B/s wr, 5 op/s 2023-03-09T22:22:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:22:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26658.733819 secs 2023-03-09T22:22:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:22:28 smithi181 ceph-mon[94189]: pgmap v14730: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 94 B/s wr, 5 op/s 2023-03-09T22:22:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:22:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26658.733819 secs 2023-03-09T22:22:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27120.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:22:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:22:30 smithi038 ceph-mon[92422]: pgmap v14731: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-09T22:22:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:22:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26661.365007 secs 2023-03-09T22:22:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:22:30 smithi122 ceph-mon[93869]: pgmap v14731: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-09T22:22:30.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:22:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26661.365007 secs 2023-03-09T22:22:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:22:30 smithi181 ceph-mon[94189]: pgmap v14731: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-09T22:22:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:22:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26661.365007 secs 2023-03-09T22:22:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:22:32 smithi038 ceph-mon[92422]: pgmap v14732: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 7 op/s 2023-03-09T22:22:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:22:32 smithi122 ceph-mon[93869]: pgmap v14732: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 7 op/s 2023-03-09T22:22:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:22:32 smithi181 ceph-mon[94189]: pgmap v14732: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 7 op/s 2023-03-09T22:22:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:22:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26663.733938 secs 2023-03-09T22:22:33.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:22:33 smithi122 ceph-mon[93869]: pgmap v14733: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 597 B/s wr, 5 op/s 2023-03-09T22:22:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:22:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26663.733938 secs 2023-03-09T22:22:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:22:33 smithi181 ceph-mon[94189]: pgmap v14733: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 597 B/s wr, 5 op/s 2023-03-09T22:22:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:22:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26663.733938 secs 2023-03-09T22:22:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:22:33 smithi038 ceph-mon[92422]: pgmap v14733: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 597 B/s wr, 5 op/s 2023-03-09T22:22:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27125.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:22:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:22:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26666.365068 secs 2023-03-09T22:22:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:22:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26666.365068 secs 2023-03-09T22:22:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:22:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26666.365068 secs 2023-03-09T22:22:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:22:36 smithi122 ceph-mon[93869]: pgmap v14734: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 113 KiB/s wr, 6 op/s 2023-03-09T22:22:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:22:36 smithi181 ceph-mon[94189]: pgmap v14734: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 113 KiB/s wr, 6 op/s 2023-03-09T22:22:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:22:36 smithi038 ceph-mon[92422]: pgmap v14734: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 113 KiB/s wr, 6 op/s 2023-03-09T22:22:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:22:38 smithi122 ceph-mon[93869]: pgmap v14735: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 113 KiB/s wr, 8 op/s 2023-03-09T22:22:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:22:38 smithi181 ceph-mon[94189]: pgmap v14735: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 113 KiB/s wr, 8 op/s 2023-03-09T22:22:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:22:38 smithi038 ceph-mon[92422]: pgmap v14735: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 113 KiB/s wr, 8 op/s 2023-03-09T22:22:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27130.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:22:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:22:39 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26668.734062 secs 2023-03-09T22:22:39.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:22:39 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26668.734062 secs 2023-03-09T22:22:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:22:39 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26668.734062 secs 2023-03-09T22:22:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:22:40 smithi122 ceph-mon[93869]: pgmap v14736: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 113 KiB/s wr, 6 op/s 2023-03-09T22:22:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:22:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26671.365129 secs 2023-03-09T22:22:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:22:40 smithi181 ceph-mon[94189]: pgmap v14736: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 113 KiB/s wr, 6 op/s 2023-03-09T22:22:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:22:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26671.365129 secs 2023-03-09T22:22:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:22:40 smithi038 ceph-mon[92422]: pgmap v14736: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 113 KiB/s wr, 6 op/s 2023-03-09T22:22:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:22:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26671.365129 secs 2023-03-09T22:22:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:22:42 smithi122 ceph-mon[93869]: pgmap v14737: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 149 KiB/s wr, 8 op/s 2023-03-09T22:22:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:22:42 smithi181 ceph-mon[94189]: pgmap v14737: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 149 KiB/s wr, 8 op/s 2023-03-09T22:22:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:22:42 smithi038 ceph-mon[92422]: pgmap v14737: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 149 KiB/s wr, 8 op/s 2023-03-09T22:22:43.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27135.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:22:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:22:44 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26673.734217 secs 2023-03-09T22:22:44.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:22:44 smithi122 ceph-mon[93869]: pgmap v14738: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 237 KiB/s wr, 6 op/s 2023-03-09T22:22:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:22:44 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26673.734217 secs 2023-03-09T22:22:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:22:44 smithi181 ceph-mon[94189]: pgmap v14738: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 237 KiB/s wr, 6 op/s 2023-03-09T22:22:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:22:44 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26673.734217 secs 2023-03-09T22:22:44.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:22:44 smithi038 ceph-mon[92422]: pgmap v14738: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 237 KiB/s wr, 6 op/s 2023-03-09T22:22:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:22:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26676.365191 secs 2023-03-09T22:22:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:22:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26676.365191 secs 2023-03-09T22:22:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:22:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26676.365191 secs 2023-03-09T22:22:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:22:46 smithi122 ceph-mon[93869]: pgmap v14739: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 237 KiB/s wr, 6 op/s 2023-03-09T22:22:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:22:46 smithi181 ceph-mon[94189]: pgmap v14739: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 237 KiB/s wr, 6 op/s 2023-03-09T22:22:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:22:46 smithi038 ceph-mon[92422]: pgmap v14739: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 237 KiB/s wr, 6 op/s 2023-03-09T22:22:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:22:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:22:47.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:22:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:22:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:22:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:22:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:22:48 smithi122 ceph-mon[93869]: pgmap v14740: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 146 KiB/s wr, 8 op/s 2023-03-09T22:22:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:22:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:22:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:22:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:22:48.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:22:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:22:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:22:48 smithi181 ceph-mon[94189]: pgmap v14740: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 146 KiB/s wr, 8 op/s 2023-03-09T22:22:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:22:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:22:48.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:22:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:22:48.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:22:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:22:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:22:48 smithi038 ceph-mon[92422]: pgmap v14740: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 146 KiB/s wr, 8 op/s 2023-03-09T22:22:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:22:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:22:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:22:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:22:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:22:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:22:48.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27140.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:22:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:22:49 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26678.734283 secs 2023-03-09T22:22:49.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:22:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:22:49.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:22:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:22:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:22:49 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26678.734283 secs 2023-03-09T22:22:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:22:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:22:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:22:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:22:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:22:49 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26678.734283 secs 2023-03-09T22:22:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:22:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:22:49.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:22:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:22:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:22:50 smithi122 ceph-mon[93869]: pgmap v14741: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 412 KiB/s wr, 8 op/s 2023-03-09T22:22:50.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:22:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26681.365253 secs 2023-03-09T22:22:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:22:50 smithi181 ceph-mon[94189]: pgmap v14741: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 412 KiB/s wr, 8 op/s 2023-03-09T22:22:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:22:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26681.365253 secs 2023-03-09T22:22:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:22:50 smithi038 ceph-mon[92422]: pgmap v14741: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 412 KiB/s wr, 8 op/s 2023-03-09T22:22:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:22:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26681.365253 secs 2023-03-09T22:22:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:22:52 smithi122 ceph-mon[93869]: pgmap v14742: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 476 KiB/s rd, 437 KiB/s wr, 11 op/s 2023-03-09T22:22:52.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:22:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:22:52.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:22:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:22:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:22:52 smithi181 ceph-mon[94189]: pgmap v14742: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 476 KiB/s rd, 437 KiB/s wr, 11 op/s 2023-03-09T22:22:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:22:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:22:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:22:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:22:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:22:52 smithi038 ceph-mon[92422]: pgmap v14742: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 476 KiB/s rd, 437 KiB/s wr, 11 op/s 2023-03-09T22:22:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:22:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:22:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:22:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:22:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:22:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:22:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:22:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:22:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:22:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26683.734409 secs 2023-03-09T22:22:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:22:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:22:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:22:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:22:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:22:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26683.734409 secs 2023-03-09T22:22:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:22:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:22:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:22:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:22:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:22:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26683.734409 secs 2023-03-09T22:22:53.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27145.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:22:54.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:22:54 smithi122 ceph-mon[93869]: pgmap v14743: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 837 KiB/s rd, 799 KiB/s wr, 10 op/s 2023-03-09T22:22:54.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:22:54 smithi181 ceph-mon[94189]: pgmap v14743: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 837 KiB/s rd, 799 KiB/s wr, 10 op/s 2023-03-09T22:22:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:22:54 smithi038 ceph-mon[92422]: pgmap v14743: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 837 KiB/s rd, 799 KiB/s wr, 10 op/s 2023-03-09T22:22:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:22:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26686.365311 secs 2023-03-09T22:22:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:22:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26686.365311 secs 2023-03-09T22:22:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:22:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26686.365311 secs 2023-03-09T22:22:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:22:56 smithi122 ceph-mon[93869]: pgmap v14744: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 759 KiB/s rd, 710 KiB/s wr, 10 op/s 2023-03-09T22:22:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:22:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:22:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:22:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:22:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:22:56 smithi181 ceph-mon[94189]: pgmap v14744: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 759 KiB/s rd, 710 KiB/s wr, 10 op/s 2023-03-09T22:22:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:22:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:22:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:22:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:22:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:22:56 smithi038 ceph-mon[92422]: pgmap v14744: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 759 KiB/s rd, 710 KiB/s wr, 10 op/s 2023-03-09T22:22:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:22:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:22:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:22:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:22:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:22:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:22:57.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:22:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:22:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:22:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:22:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:22:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:22:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:22:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:22:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:22:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:22:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:22:58 smithi122 ceph-mon[93869]: pgmap v14745: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 746 KiB/s rd, 732 KiB/s wr, 13 op/s 2023-03-09T22:22:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:22:58 smithi181 ceph-mon[94189]: pgmap v14745: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 746 KiB/s rd, 732 KiB/s wr, 13 op/s 2023-03-09T22:22:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:22:58 smithi038 ceph-mon[92422]: pgmap v14745: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 746 KiB/s rd, 732 KiB/s wr, 13 op/s 2023-03-09T22:22:58.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27150.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:23:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:00 smithi181 ceph-mon[94189]: pgmap v14746: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 864 KiB/s rd, 851 KiB/s wr, 12 op/s 2023-03-09T22:23:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26691.365374 secs 2023-03-09T22:23:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:00 smithi038 ceph-mon[92422]: pgmap v14746: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 864 KiB/s rd, 851 KiB/s wr, 12 op/s 2023-03-09T22:23:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26691.365374 secs 2023-03-09T22:23:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:00 smithi122 ceph-mon[93869]: pgmap v14746: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 864 KiB/s rd, 851 KiB/s wr, 12 op/s 2023-03-09T22:23:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26691.365374 secs 2023-03-09T22:23:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:01 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:23:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:01 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:23:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:01 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:23:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:02 smithi038 ceph-mon[92422]: pgmap v14747: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 695 KiB/s rd, 680 KiB/s wr, 14 op/s 2023-03-09T22:23:02.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:02 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:23:02.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:02 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:23:02.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:02 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:23:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:02 smithi122 ceph-mon[93869]: pgmap v14747: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 695 KiB/s rd, 680 KiB/s wr, 14 op/s 2023-03-09T22:23:02.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:02 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:23:02.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:02 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:23:02.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:02 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:23:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:02 smithi181 ceph-mon[94189]: pgmap v14747: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 695 KiB/s rd, 680 KiB/s wr, 14 op/s 2023-03-09T22:23:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:02 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:23:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:02 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:23:02.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:02 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:23:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26693.734622 secs 2023-03-09T22:23:03.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26693.734622 secs 2023-03-09T22:23:03.811 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27155.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:23:03.812 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26693.734622 secs 2023-03-09T22:23:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:04 smithi038 ceph-mon[92422]: pgmap v14748: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 800 KiB/s rd, 788 KiB/s wr, 12 op/s 2023-03-09T22:23:04.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:04 smithi122 ceph-mon[93869]: pgmap v14748: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 800 KiB/s rd, 788 KiB/s wr, 12 op/s 2023-03-09T22:23:04.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:04 smithi181 ceph-mon[94189]: pgmap v14748: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 800 KiB/s rd, 788 KiB/s wr, 12 op/s 2023-03-09T22:23:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:05 smithi122 ceph-mon[93869]: osdmap e1137: 12 total, 12 up, 12 in 2023-03-09T22:23:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26696.365436 secs 2023-03-09T22:23:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:05 smithi181 ceph-mon[94189]: osdmap e1137: 12 total, 12 up, 12 in 2023-03-09T22:23:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26696.365436 secs 2023-03-09T22:23:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:05 smithi038 ceph-mon[92422]: osdmap e1137: 12 total, 12 up, 12 in 2023-03-09T22:23:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26696.365436 secs 2023-03-09T22:23:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:06 smithi122 ceph-mon[93869]: pgmap v14750: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 494 KiB/s wr, 26 op/s 2023-03-09T22:23:06.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:06 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:23:06.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:23:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:06 smithi181 ceph-mon[94189]: pgmap v14750: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 494 KiB/s wr, 26 op/s 2023-03-09T22:23:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:06 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:23:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:23:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:06 smithi038 ceph-mon[92422]: pgmap v14750: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 494 KiB/s wr, 26 op/s 2023-03-09T22:23:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:06 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:23:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:23:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:07 smithi122 ceph-mon[93869]: pgmap v14751: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 689 KiB/s rd, 731 KiB/s wr, 30 op/s 2023-03-09T22:23:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:07 smithi181 ceph-mon[94189]: pgmap v14751: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 689 KiB/s rd, 731 KiB/s wr, 30 op/s 2023-03-09T22:23:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:07 smithi038 ceph-mon[92422]: pgmap v14751: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 689 KiB/s rd, 731 KiB/s wr, 30 op/s 2023-03-09T22:23:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:23:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:23:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26698.734743 secs 2023-03-09T22:23:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:23:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:23:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26698.734743 secs 2023-03-09T22:23:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:23:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:23:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26698.734743 secs 2023-03-09T22:23:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27160.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:23:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:23:09.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:23:09.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:09 smithi122 ceph-mon[93869]: pgmap v14752: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 521 KiB/s rd, 562 KiB/s wr, 28 op/s 2023-03-09T22:23:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:23:09.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:23:09.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:09 smithi181 ceph-mon[94189]: pgmap v14752: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 521 KiB/s rd, 562 KiB/s wr, 28 op/s 2023-03-09T22:23:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:23:09.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:23:09.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:09 smithi038 ceph-mon[92422]: pgmap v14752: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 521 KiB/s rd, 562 KiB/s wr, 28 op/s 2023-03-09T22:23:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26701.365496 secs 2023-03-09T22:23:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26701.365496 secs 2023-03-09T22:23:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26701.365496 secs 2023-03-09T22:23:11.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:11 smithi122 ceph-mon[93869]: pgmap v14753: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 539 KiB/s rd, 580 KiB/s wr, 28 op/s 2023-03-09T22:23:11.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:11 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:23:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:11 smithi181 ceph-mon[94189]: pgmap v14753: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 539 KiB/s rd, 580 KiB/s wr, 28 op/s 2023-03-09T22:23:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:11 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:23:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:11 smithi038 ceph-mon[92422]: pgmap v14753: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 539 KiB/s rd, 580 KiB/s wr, 28 op/s 2023-03-09T22:23:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:11 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:23:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27165.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:23:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:14 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26703.734867 secs 2023-03-09T22:23:14.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:14 smithi122 ceph-mon[93869]: pgmap v14754: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 421 KiB/s wr, 27 op/s 2023-03-09T22:23:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:14 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26703.734867 secs 2023-03-09T22:23:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:14 smithi181 ceph-mon[94189]: pgmap v14754: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 421 KiB/s wr, 27 op/s 2023-03-09T22:23:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:14 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26703.734867 secs 2023-03-09T22:23:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:14 smithi038 ceph-mon[92422]: pgmap v14754: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 421 KiB/s wr, 27 op/s 2023-03-09T22:23:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26706.365565 secs 2023-03-09T22:23:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26706.365565 secs 2023-03-09T22:23:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26706.365565 secs 2023-03-09T22:23:16.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:16 smithi122 ceph-mon[93869]: pgmap v14755: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 370 KiB/s wr, 11 op/s 2023-03-09T22:23:16.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:16 smithi181 ceph-mon[94189]: pgmap v14755: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 370 KiB/s wr, 11 op/s 2023-03-09T22:23:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:16 smithi038 ceph-mon[92422]: pgmap v14755: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 370 KiB/s wr, 11 op/s 2023-03-09T22:23:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:17 smithi122 ceph-mon[93869]: osdmap e1138: 12 total, 12 up, 12 in 2023-03-09T22:23:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:17 smithi181 ceph-mon[94189]: osdmap e1138: 12 total, 12 up, 12 in 2023-03-09T22:23:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:17 smithi038 ceph-mon[92422]: osdmap e1138: 12 total, 12 up, 12 in 2023-03-09T22:23:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:18 smithi122 ceph-mon[93869]: pgmap v14757: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 132 KiB/s wr, 8 op/s 2023-03-09T22:23:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:18 smithi181 ceph-mon[94189]: pgmap v14757: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 132 KiB/s wr, 8 op/s 2023-03-09T22:23:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:18 smithi038 ceph-mon[92422]: pgmap v14757: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 132 KiB/s wr, 8 op/s 2023-03-09T22:23:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27170.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:23:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:19 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26708.734982 secs 2023-03-09T22:23:19.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:19 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26708.734982 secs 2023-03-09T22:23:19.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:19 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26708.734982 secs 2023-03-09T22:23:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:20 smithi122 ceph-mon[93869]: pgmap v14758: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 132 KiB/s wr, 7 op/s 2023-03-09T22:23:20.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26711.365626 secs 2023-03-09T22:23:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:20 smithi181 ceph-mon[94189]: pgmap v14758: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 132 KiB/s wr, 7 op/s 2023-03-09T22:23:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26711.365626 secs 2023-03-09T22:23:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:20 smithi038 ceph-mon[92422]: pgmap v14758: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 132 KiB/s wr, 7 op/s 2023-03-09T22:23:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26711.365626 secs 2023-03-09T22:23:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:22 smithi122 ceph-mon[93869]: pgmap v14759: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T22:23:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:22 smithi181 ceph-mon[94189]: pgmap v14759: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T22:23:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:22 smithi038 ceph-mon[92422]: pgmap v14759: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T22:23:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27175.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:23:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:24 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26713.735116 secs 2023-03-09T22:23:24.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:24 smithi122 ceph-mon[93869]: pgmap v14760: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T22:23:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:24 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26713.735116 secs 2023-03-09T22:23:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:24 smithi181 ceph-mon[94189]: pgmap v14760: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T22:23:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:24 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26713.735116 secs 2023-03-09T22:23:24.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:24 smithi038 ceph-mon[92422]: pgmap v14760: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T22:23:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26716.365687 secs 2023-03-09T22:23:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26716.365687 secs 2023-03-09T22:23:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26716.365687 secs 2023-03-09T22:23:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:26 smithi122 ceph-mon[93869]: pgmap v14761: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T22:23:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:26 smithi181 ceph-mon[94189]: pgmap v14761: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T22:23:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:26 smithi038 ceph-mon[92422]: pgmap v14761: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T22:23:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:28 smithi122 ceph-mon[93869]: pgmap v14762: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T22:23:28.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26718.735238 secs 2023-03-09T22:23:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:28 smithi181 ceph-mon[94189]: pgmap v14762: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T22:23:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26718.735238 secs 2023-03-09T22:23:28.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:28 smithi038 ceph-mon[92422]: pgmap v14762: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T22:23:28.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26718.735238 secs 2023-03-09T22:23:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27180.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:23:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:30 smithi122 ceph-mon[93869]: pgmap v14763: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T22:23:30.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26721.365744 secs 2023-03-09T22:23:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:30 smithi181 ceph-mon[94189]: pgmap v14763: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T22:23:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26721.365744 secs 2023-03-09T22:23:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:30 smithi038 ceph-mon[92422]: pgmap v14763: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T22:23:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26721.365744 secs 2023-03-09T22:23:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:32 smithi122 ceph-mon[93869]: pgmap v14764: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 341 B/s wr, 8 op/s 2023-03-09T22:23:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:32 smithi181 ceph-mon[94189]: pgmap v14764: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 341 B/s wr, 8 op/s 2023-03-09T22:23:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:32 smithi038 ceph-mon[92422]: pgmap v14764: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 341 B/s wr, 8 op/s 2023-03-09T22:23:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26723.735358 secs 2023-03-09T22:23:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26723.735358 secs 2023-03-09T22:23:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26723.735358 secs 2023-03-09T22:23:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27185.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:23:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:34 smithi181 ceph-mon[94189]: pgmap v14765: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-09T22:23:34.526 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:34 smithi038 ceph-mon[92422]: pgmap v14765: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-09T22:23:34.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:34 smithi122 ceph-mon[93869]: pgmap v14765: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-09T22:23:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26726.365805 secs 2023-03-09T22:23:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26726.365805 secs 2023-03-09T22:23:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26726.365805 secs 2023-03-09T22:23:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:36 smithi122 ceph-mon[93869]: pgmap v14766: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-09T22:23:36.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:23:36.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:23:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:36 smithi181 ceph-mon[94189]: pgmap v14766: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-09T22:23:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:23:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:23:36.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:36 smithi038 ceph-mon[92422]: pgmap v14766: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-09T22:23:36.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:23:36.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:23:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:23:37.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:23:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:23:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:23:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:23:37.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:23:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:38 smithi122 ceph-mon[93869]: pgmap v14767: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 114 KiB/s wr, 9 op/s 2023-03-09T22:23:38.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26728.735430 secs 2023-03-09T22:23:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:38 smithi181 ceph-mon[94189]: pgmap v14767: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 114 KiB/s wr, 9 op/s 2023-03-09T22:23:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26728.735430 secs 2023-03-09T22:23:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:38 smithi038 ceph-mon[92422]: pgmap v14767: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 114 KiB/s wr, 9 op/s 2023-03-09T22:23:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26728.735430 secs 2023-03-09T22:23:38.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27190.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:23:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:39 smithi122 ceph-mon[93869]: pgmap v14768: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 114 KiB/s wr, 7 op/s 2023-03-09T22:23:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:39 smithi181 ceph-mon[94189]: pgmap v14768: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 114 KiB/s wr, 7 op/s 2023-03-09T22:23:39.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:39 smithi038 ceph-mon[92422]: pgmap v14768: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 114 KiB/s wr, 7 op/s 2023-03-09T22:23:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26731.365867 secs 2023-03-09T22:23:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26731.365867 secs 2023-03-09T22:23:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26731.365867 secs 2023-03-09T22:23:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:42 smithi122 ceph-mon[93869]: pgmap v14769: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 254 KiB/s wr, 10 op/s 2023-03-09T22:23:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:42 smithi181 ceph-mon[94189]: pgmap v14769: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 254 KiB/s wr, 10 op/s 2023-03-09T22:23:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:42 smithi038 ceph-mon[92422]: pgmap v14769: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 254 KiB/s wr, 10 op/s 2023-03-09T22:23:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27195.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:23:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:44 smithi122 ceph-mon[93869]: pgmap v14770: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 254 KiB/s wr, 7 op/s 2023-03-09T22:23:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:44 smithi181 ceph-mon[94189]: pgmap v14770: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 254 KiB/s wr, 7 op/s 2023-03-09T22:23:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:44 smithi038 ceph-mon[92422]: pgmap v14770: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 254 KiB/s wr, 7 op/s 2023-03-09T22:23:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26736.365946 secs 2023-03-09T22:23:45.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:45 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:23:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26736.365946 secs 2023-03-09T22:23:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:45 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:23:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26736.365946 secs 2023-03-09T22:23:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:45 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:23:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:46 smithi122 ceph-mon[93869]: pgmap v14771: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 254 KiB/s wr, 6 op/s 2023-03-09T22:23:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:46 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:23:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:46 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:23:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:46 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:23:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:46 smithi181 ceph-mon[94189]: pgmap v14771: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 254 KiB/s wr, 6 op/s 2023-03-09T22:23:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:46 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:23:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:46 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:23:46.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:46 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:23:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:46 smithi038 ceph-mon[92422]: pgmap v14771: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 254 KiB/s wr, 6 op/s 2023-03-09T22:23:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:46 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:23:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:46 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:23:46.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:46 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:23:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:48 smithi122 ceph-mon[93869]: pgmap v14772: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 505 KiB/s rd, 449 KiB/s wr, 10 op/s 2023-03-09T22:23:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:23:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:48 smithi181 ceph-mon[94189]: pgmap v14772: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 505 KiB/s rd, 449 KiB/s wr, 10 op/s 2023-03-09T22:23:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:23:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:48 smithi038 ceph-mon[92422]: pgmap v14772: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 505 KiB/s rd, 449 KiB/s wr, 10 op/s 2023-03-09T22:23:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:23:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27200.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:23:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:23:49.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:23:49.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:23:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:23:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:23:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:23:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:23:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:23:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:23:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:50 smithi122 ceph-mon[93869]: pgmap v14773: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 336 KiB/s wr, 7 op/s 2023-03-09T22:23:50.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26741.365992 secs 2023-03-09T22:23:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:50 smithi181 ceph-mon[94189]: pgmap v14773: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 336 KiB/s wr, 7 op/s 2023-03-09T22:23:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26741.365992 secs 2023-03-09T22:23:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:50 smithi038 ceph-mon[92422]: pgmap v14773: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 336 KiB/s wr, 7 op/s 2023-03-09T22:23:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26741.365992 secs 2023-03-09T22:23:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:52 smithi122 ceph-mon[93869]: pgmap v14774: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 619 KiB/s rd, 550 KiB/s wr, 12 op/s 2023-03-09T22:23:52.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:23:52.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:23:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:52 smithi181 ceph-mon[94189]: pgmap v14774: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 619 KiB/s rd, 550 KiB/s wr, 12 op/s 2023-03-09T22:23:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:23:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:23:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:52 smithi038 ceph-mon[92422]: pgmap v14774: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 619 KiB/s rd, 550 KiB/s wr, 12 op/s 2023-03-09T22:23:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:23:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:23:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:23:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:23:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:23:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:23:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:23:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:23:53.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27205.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:23:54.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:54 smithi122 ceph-mon[93869]: pgmap v14775: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 409 KiB/s wr, 10 op/s 2023-03-09T22:23:54.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:54 smithi181 ceph-mon[94189]: pgmap v14775: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 409 KiB/s wr, 10 op/s 2023-03-09T22:23:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:54 smithi038 ceph-mon[92422]: pgmap v14775: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 409 KiB/s wr, 10 op/s 2023-03-09T22:23:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26746.366054 secs 2023-03-09T22:23:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26746.366054 secs 2023-03-09T22:23:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26746.366054 secs 2023-03-09T22:23:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:56 smithi122 ceph-mon[93869]: pgmap v14776: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 409 KiB/s wr, 10 op/s 2023-03-09T22:23:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:23:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:23:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:56 smithi181 ceph-mon[94189]: pgmap v14776: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 409 KiB/s wr, 10 op/s 2023-03-09T22:23:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:23:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:23:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:56 smithi038 ceph-mon[92422]: pgmap v14776: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 409 KiB/s wr, 10 op/s 2023-03-09T22:23:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:23:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:23:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:23:57.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:23:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:23:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:23:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:23:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:23:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:58 smithi122 ceph-mon[93869]: pgmap v14777: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 789 KiB/s rd, 721 KiB/s wr, 15 op/s 2023-03-09T22:23:58.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:23:58 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 26748.735886 secs 2023-03-09T22:23:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:58 smithi181 ceph-mon[94189]: pgmap v14777: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 789 KiB/s rd, 721 KiB/s wr, 15 op/s 2023-03-09T22:23:58.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:23:58 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 26748.735886 secs 2023-03-09T22:23:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:58 smithi038 ceph-mon[92422]: pgmap v14777: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 789 KiB/s rd, 721 KiB/s wr, 15 op/s 2023-03-09T22:23:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:23:58 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 26748.735886 secs 2023-03-09T22:23:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27210.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:24:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:00 smithi122 ceph-mon[93869]: pgmap v14778: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 541 KiB/s rd, 527 KiB/s wr, 12 op/s 2023-03-09T22:24:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26751.366112 secs 2023-03-09T22:24:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:00 smithi181 ceph-mon[94189]: pgmap v14778: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 541 KiB/s rd, 527 KiB/s wr, 12 op/s 2023-03-09T22:24:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26751.366112 secs 2023-03-09T22:24:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:00 smithi038 ceph-mon[92422]: pgmap v14778: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 541 KiB/s rd, 527 KiB/s wr, 12 op/s 2023-03-09T22:24:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26751.366112 secs 2023-03-09T22:24:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:02 smithi122 ceph-mon[93869]: pgmap v14779: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 584 KiB/s rd, 590 KiB/s wr, 17 op/s 2023-03-09T22:24:02.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:02 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:24:02.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:02 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:24:02.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:02 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:24:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:02 smithi181 ceph-mon[94189]: pgmap v14779: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 584 KiB/s rd, 590 KiB/s wr, 17 op/s 2023-03-09T22:24:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:02 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:24:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:02 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:24:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:02 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:24:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:02 smithi038 ceph-mon[92422]: pgmap v14779: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 584 KiB/s rd, 590 KiB/s wr, 17 op/s 2023-03-09T22:24:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:02 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:24:02.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:02 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:24:02.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:02 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:24:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27215.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:24:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:04 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26753.735994 secs 2023-03-09T22:24:04.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:04 smithi122 ceph-mon[93869]: pgmap v14780: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 524 KiB/s wr, 26 op/s 2023-03-09T22:24:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:04 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26753.735994 secs 2023-03-09T22:24:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:04 smithi181 ceph-mon[94189]: pgmap v14780: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 524 KiB/s wr, 26 op/s 2023-03-09T22:24:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:04 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26753.735994 secs 2023-03-09T22:24:04.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:04 smithi038 ceph-mon[92422]: pgmap v14780: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 524 KiB/s wr, 26 op/s 2023-03-09T22:24:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:05 smithi122 ceph-mon[93869]: osdmap e1139: 12 total, 12 up, 12 in 2023-03-09T22:24:05.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26756.366173 secs 2023-03-09T22:24:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:05 smithi181 ceph-mon[94189]: osdmap e1139: 12 total, 12 up, 12 in 2023-03-09T22:24:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26756.366173 secs 2023-03-09T22:24:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:05 smithi038 ceph-mon[92422]: osdmap e1139: 12 total, 12 up, 12 in 2023-03-09T22:24:05.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26756.366173 secs 2023-03-09T22:24:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:06 smithi122 ceph-mon[93869]: pgmap v14782: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 628 KiB/s wr, 30 op/s 2023-03-09T22:24:06.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:06 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:24:06.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:24:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:06 smithi181 ceph-mon[94189]: pgmap v14782: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 628 KiB/s wr, 30 op/s 2023-03-09T22:24:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:06 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:24:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:24:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:06 smithi038 ceph-mon[92422]: pgmap v14782: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 628 KiB/s wr, 30 op/s 2023-03-09T22:24:06.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:06 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:24:06.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:24:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:08 smithi122 ceph-mon[93869]: pgmap v14783: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 545 KiB/s wr, 29 op/s 2023-03-09T22:24:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:24:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:24:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:08 smithi181 ceph-mon[94189]: pgmap v14783: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 545 KiB/s wr, 29 op/s 2023-03-09T22:24:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:24:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:24:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:08 smithi038 ceph-mon[92422]: pgmap v14783: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 545 KiB/s wr, 29 op/s 2023-03-09T22:24:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:24:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:24:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27220.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:24:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:09 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26758.736135 secs 2023-03-09T22:24:09.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:24:09.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:24:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:09 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26758.736135 secs 2023-03-09T22:24:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:24:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:24:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:09 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26758.736135 secs 2023-03-09T22:24:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:24:09.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:24:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:10 smithi122 ceph-mon[93869]: pgmap v14784: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 545 KiB/s wr, 29 op/s 2023-03-09T22:24:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26761.366233 secs 2023-03-09T22:24:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:10 smithi181 ceph-mon[94189]: pgmap v14784: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 545 KiB/s wr, 29 op/s 2023-03-09T22:24:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26761.366233 secs 2023-03-09T22:24:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:10 smithi038 ceph-mon[92422]: pgmap v14784: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 545 KiB/s wr, 29 op/s 2023-03-09T22:24:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26761.366233 secs 2023-03-09T22:24:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:12 smithi122 ceph-mon[93869]: pgmap v14785: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 473 KiB/s wr, 25 op/s 2023-03-09T22:24:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:12 smithi181 ceph-mon[94189]: pgmap v14785: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 473 KiB/s wr, 25 op/s 2023-03-09T22:24:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:12 smithi038 ceph-mon[92422]: pgmap v14785: 129 pgs: 129 active+clean; 10 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 473 KiB/s wr, 25 op/s 2023-03-09T22:24:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26763.736248 secs 2023-03-09T22:24:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26763.736248 secs 2023-03-09T22:24:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26763.736248 secs 2023-03-09T22:24:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27225.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:24:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:14 smithi122 ceph-mon[93869]: pgmap v14786: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 385 KiB/s wr, 9 op/s 2023-03-09T22:24:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:14 smithi181 ceph-mon[94189]: pgmap v14786: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 385 KiB/s wr, 9 op/s 2023-03-09T22:24:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:14 smithi038 ceph-mon[92422]: pgmap v14786: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 385 KiB/s wr, 9 op/s 2023-03-09T22:24:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26766.366295 secs 2023-03-09T22:24:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26766.366295 secs 2023-03-09T22:24:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26766.366295 secs 2023-03-09T22:24:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:16 smithi038 ceph-mon[92422]: pgmap v14787: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 357 KiB/s wr, 9 op/s 2023-03-09T22:24:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:16 smithi038 ceph-mon[92422]: osdmap e1140: 12 total, 12 up, 12 in 2023-03-09T22:24:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:16 smithi122 ceph-mon[93869]: pgmap v14787: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 357 KiB/s wr, 9 op/s 2023-03-09T22:24:16.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:16 smithi122 ceph-mon[93869]: osdmap e1140: 12 total, 12 up, 12 in 2023-03-09T22:24:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:16 smithi181 ceph-mon[94189]: pgmap v14787: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 357 KiB/s wr, 9 op/s 2023-03-09T22:24:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:16 smithi181 ceph-mon[94189]: osdmap e1140: 12 total, 12 up, 12 in 2023-03-09T22:24:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:18 smithi038 ceph-mon[92422]: pgmap v14789: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 167 KiB/s wr, 8 op/s 2023-03-09T22:24:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26768.736373 secs 2023-03-09T22:24:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:18 smithi122 ceph-mon[93869]: pgmap v14789: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 167 KiB/s wr, 8 op/s 2023-03-09T22:24:18.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26768.736373 secs 2023-03-09T22:24:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:18 smithi181 ceph-mon[94189]: pgmap v14789: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 167 KiB/s wr, 8 op/s 2023-03-09T22:24:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26768.736373 secs 2023-03-09T22:24:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27230.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:24:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:20 smithi122 ceph-mon[93869]: pgmap v14790: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 167 KiB/s wr, 7 op/s 2023-03-09T22:24:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:24:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:24:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26771.366357 secs 2023-03-09T22:24:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:20 smithi181 ceph-mon[94189]: pgmap v14790: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 167 KiB/s wr, 7 op/s 2023-03-09T22:24:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:24:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:24:20.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26771.366357 secs 2023-03-09T22:24:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:20 smithi038 ceph-mon[92422]: pgmap v14790: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 167 KiB/s wr, 7 op/s 2023-03-09T22:24:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:24:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:24:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26771.366357 secs 2023-03-09T22:24:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:24:21.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:24:21.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:21 smithi122 ceph-mon[93869]: pgmap v14791: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 163 KiB/s wr, 7 op/s 2023-03-09T22:24:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:24:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:24:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:21 smithi181 ceph-mon[94189]: pgmap v14791: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 163 KiB/s wr, 7 op/s 2023-03-09T22:24:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:24:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:24:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:21 smithi038 ceph-mon[92422]: pgmap v14791: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 163 KiB/s wr, 7 op/s 2023-03-09T22:24:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27235.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:24:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:24 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26773.736446 secs 2023-03-09T22:24:24.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:24 smithi122 ceph-mon[93869]: pgmap v14792: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 74 KiB/s wr, 7 op/s 2023-03-09T22:24:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:24 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26773.736446 secs 2023-03-09T22:24:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:24 smithi181 ceph-mon[94189]: pgmap v14792: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 74 KiB/s wr, 7 op/s 2023-03-09T22:24:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:24 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26773.736446 secs 2023-03-09T22:24:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:24 smithi038 ceph-mon[92422]: pgmap v14792: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 74 KiB/s wr, 7 op/s 2023-03-09T22:24:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26776.366418 secs 2023-03-09T22:24:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26776.366418 secs 2023-03-09T22:24:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26776.366418 secs 2023-03-09T22:24:26.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:26 smithi122 ceph-mon[93869]: pgmap v14793: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 74 KiB/s wr, 7 op/s 2023-03-09T22:24:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:26 smithi181 ceph-mon[94189]: pgmap v14793: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 74 KiB/s wr, 7 op/s 2023-03-09T22:24:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:26 smithi038 ceph-mon[92422]: pgmap v14793: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 74 KiB/s wr, 7 op/s 2023-03-09T22:24:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:28 smithi122 ceph-mon[93869]: pgmap v14794: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T22:24:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:28 smithi181 ceph-mon[94189]: pgmap v14794: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T22:24:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:28 smithi038 ceph-mon[92422]: pgmap v14794: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T22:24:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27240.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:24:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:29 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26778.736566 secs 2023-03-09T22:24:29.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:29 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26778.736566 secs 2023-03-09T22:24:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:29 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26778.736566 secs 2023-03-09T22:24:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:30 smithi122 ceph-mon[93869]: pgmap v14795: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T22:24:30.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26781.366473 secs 2023-03-09T22:24:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:30 smithi181 ceph-mon[94189]: pgmap v14795: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T22:24:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26781.366473 secs 2023-03-09T22:24:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:30 smithi038 ceph-mon[92422]: pgmap v14795: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T22:24:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26781.366473 secs 2023-03-09T22:24:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:32 smithi122 ceph-mon[93869]: pgmap v14796: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 94 KiB/s wr, 8 op/s 2023-03-09T22:24:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:32 smithi181 ceph-mon[94189]: pgmap v14796: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 94 KiB/s wr, 8 op/s 2023-03-09T22:24:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:32 smithi038 ceph-mon[92422]: pgmap v14796: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 94 KiB/s wr, 8 op/s 2023-03-09T22:24:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27245.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:24:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:34 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26783.736677 secs 2023-03-09T22:24:34.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:34 smithi122 ceph-mon[93869]: pgmap v14797: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 94 KiB/s wr, 7 op/s 2023-03-09T22:24:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:34 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26783.736677 secs 2023-03-09T22:24:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:34 smithi181 ceph-mon[94189]: pgmap v14797: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 94 KiB/s wr, 7 op/s 2023-03-09T22:24:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:34 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26783.736677 secs 2023-03-09T22:24:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:34 smithi038 ceph-mon[92422]: pgmap v14797: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 94 KiB/s wr, 7 op/s 2023-03-09T22:24:35.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26786.366534 secs 2023-03-09T22:24:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26786.366534 secs 2023-03-09T22:24:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26786.366534 secs 2023-03-09T22:24:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:36 smithi122 ceph-mon[93869]: pgmap v14798: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 94 KiB/s wr, 6 op/s 2023-03-09T22:24:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:36 smithi181 ceph-mon[94189]: pgmap v14798: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 94 KiB/s wr, 6 op/s 2023-03-09T22:24:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:36 smithi038 ceph-mon[92422]: pgmap v14798: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 94 KiB/s wr, 6 op/s 2023-03-09T22:24:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:38 smithi122 ceph-mon[93869]: pgmap v14799: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 207 KiB/s wr, 9 op/s 2023-03-09T22:24:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:38 smithi181 ceph-mon[94189]: pgmap v14799: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 207 KiB/s wr, 9 op/s 2023-03-09T22:24:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:38 smithi038 ceph-mon[92422]: pgmap v14799: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 207 KiB/s wr, 9 op/s 2023-03-09T22:24:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27250.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:24:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:39 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26788.736791 secs 2023-03-09T22:24:39.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:39 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26788.736791 secs 2023-03-09T22:24:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:39 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26788.736791 secs 2023-03-09T22:24:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:40 smithi122 ceph-mon[93869]: pgmap v14800: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 207 KiB/s wr, 6 op/s 2023-03-09T22:24:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26791.366614 secs 2023-03-09T22:24:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:40 smithi181 ceph-mon[94189]: pgmap v14800: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 207 KiB/s wr, 6 op/s 2023-03-09T22:24:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26791.366614 secs 2023-03-09T22:24:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:40 smithi038 ceph-mon[92422]: pgmap v14800: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 207 KiB/s wr, 6 op/s 2023-03-09T22:24:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26791.366614 secs 2023-03-09T22:24:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:42 smithi122 ceph-mon[93869]: pgmap v14801: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 278 KiB/s wr, 9 op/s 2023-03-09T22:24:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:42 smithi181 ceph-mon[94189]: pgmap v14801: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 278 KiB/s wr, 9 op/s 2023-03-09T22:24:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:42 smithi038 ceph-mon[92422]: pgmap v14801: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 278 KiB/s wr, 9 op/s 2023-03-09T22:24:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27255.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:24:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:44 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26793.736875 secs 2023-03-09T22:24:44.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:44 smithi122 ceph-mon[93869]: pgmap v14802: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 239 KiB/s wr, 7 op/s 2023-03-09T22:24:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:44 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26793.736875 secs 2023-03-09T22:24:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:44 smithi181 ceph-mon[94189]: pgmap v14802: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 239 KiB/s wr, 7 op/s 2023-03-09T22:24:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:44 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26793.736875 secs 2023-03-09T22:24:44.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:44 smithi038 ceph-mon[92422]: pgmap v14802: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 239 KiB/s wr, 7 op/s 2023-03-09T22:24:45.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26796.366686 secs 2023-03-09T22:24:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26796.366686 secs 2023-03-09T22:24:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26796.366686 secs 2023-03-09T22:24:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:46 smithi122 ceph-mon[93869]: pgmap v14803: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 239 KiB/s wr, 6 op/s 2023-03-09T22:24:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:46 smithi181 ceph-mon[94189]: pgmap v14803: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 239 KiB/s wr, 6 op/s 2023-03-09T22:24:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:46 smithi038 ceph-mon[92422]: pgmap v14803: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 239 KiB/s wr, 6 op/s 2023-03-09T22:24:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:48 smithi122 ceph-mon[93869]: pgmap v14804: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 335 KiB/s rd, 261 KiB/s wr, 9 op/s 2023-03-09T22:24:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26798.736989 secs 2023-03-09T22:24:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:48 smithi181 ceph-mon[94189]: pgmap v14804: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 335 KiB/s rd, 261 KiB/s wr, 9 op/s 2023-03-09T22:24:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26798.736989 secs 2023-03-09T22:24:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:48 smithi038 ceph-mon[92422]: pgmap v14804: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 335 KiB/s rd, 261 KiB/s wr, 9 op/s 2023-03-09T22:24:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26798.736989 secs 2023-03-09T22:24:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27260.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:24:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:24:49.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:24:49.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:24:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:24:49.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:24:49.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:24:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:24:49.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:24:49.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:24:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:50 smithi038 ceph-mon[92422]: pgmap v14805: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 435 KiB/s wr, 8 op/s 2023-03-09T22:24:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26801.366749 secs 2023-03-09T22:24:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:50 smithi122 ceph-mon[93869]: pgmap v14805: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 435 KiB/s wr, 8 op/s 2023-03-09T22:24:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26801.366749 secs 2023-03-09T22:24:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:50 smithi181 ceph-mon[94189]: pgmap v14805: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 435 KiB/s wr, 8 op/s 2023-03-09T22:24:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26801.366749 secs 2023-03-09T22:24:52.353 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:52 smithi038 ceph-mon[92422]: pgmap v14806: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 850 KiB/s rd, 722 KiB/s wr, 11 op/s 2023-03-09T22:24:52.354 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:24:52.354 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:24:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:52 smithi122 ceph-mon[93869]: pgmap v14806: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 850 KiB/s rd, 722 KiB/s wr, 11 op/s 2023-03-09T22:24:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:24:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:24:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:52 smithi181 ceph-mon[94189]: pgmap v14806: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 850 KiB/s rd, 722 KiB/s wr, 11 op/s 2023-03-09T22:24:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:24:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:24:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26803.737058 secs 2023-03-09T22:24:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:24:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:24:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:24:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:24:53.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:24:53.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:53 smithi122 ceph-mon[93869]: pgmap v14807: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 768 KiB/s rd, 653 KiB/s wr, 10 op/s 2023-03-09T22:24:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26803.737058 secs 2023-03-09T22:24:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:24:53.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:24:53.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:24:53.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:24:53.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:24:53.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:53 smithi181 ceph-mon[94189]: pgmap v14807: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 768 KiB/s rd, 653 KiB/s wr, 10 op/s 2023-03-09T22:24:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26803.737058 secs 2023-03-09T22:24:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:24:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:24:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:24:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:24:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:24:53.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:53 smithi038 ceph-mon[92422]: pgmap v14807: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 768 KiB/s rd, 653 KiB/s wr, 10 op/s 2023-03-09T22:24:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27265.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:24:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:24:55.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:24:55.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26806.366810 secs 2023-03-09T22:24:55.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:55 smithi122 ceph-mon[93869]: pgmap v14808: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 598 KiB/s wr, 9 op/s 2023-03-09T22:24:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:24:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:24:55.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26806.366810 secs 2023-03-09T22:24:55.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:55 smithi181 ceph-mon[94189]: pgmap v14808: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 598 KiB/s wr, 9 op/s 2023-03-09T22:24:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:24:55.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:24:55.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26806.366810 secs 2023-03-09T22:24:55.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:55 smithi038 ceph-mon[92422]: pgmap v14808: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 598 KiB/s wr, 9 op/s 2023-03-09T22:24:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:24:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:24:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:57 smithi038 ceph-mon[92422]: pgmap v14809: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 995 KiB/s rd, 879 KiB/s wr, 14 op/s 2023-03-09T22:24:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:24:57.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:24:57.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:57 smithi122 ceph-mon[93869]: pgmap v14809: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 995 KiB/s rd, 879 KiB/s wr, 14 op/s 2023-03-09T22:24:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:24:57.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:24:57.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:57 smithi181 ceph-mon[94189]: pgmap v14809: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 995 KiB/s rd, 879 KiB/s wr, 14 op/s 2023-03-09T22:24:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27270.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:24:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26808.737220 secs 2023-03-09T22:24:58.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26808.737220 secs 2023-03-09T22:24:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26808.737220 secs 2023-03-09T22:24:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:24:59.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:24:59.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:24:59 smithi122 ceph-mon[93869]: pgmap v14810: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 972 KiB/s rd, 857 KiB/s wr, 12 op/s 2023-03-09T22:24:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:24:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:24:59.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:24:59 smithi181 ceph-mon[94189]: pgmap v14810: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 972 KiB/s rd, 857 KiB/s wr, 12 op/s 2023-03-09T22:24:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:25:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:25:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:24:59 smithi038 ceph-mon[92422]: pgmap v14810: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 972 KiB/s rd, 857 KiB/s wr, 12 op/s 2023-03-09T22:25:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26811.366872 secs 2023-03-09T22:25:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26811.366872 secs 2023-03-09T22:25:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26811.366872 secs 2023-03-09T22:25:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:02 smithi122 ceph-mon[93869]: pgmap v14811: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 850 KiB/s rd, 821 KiB/s wr, 17 op/s 2023-03-09T22:25:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:02 smithi181 ceph-mon[94189]: pgmap v14811: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 850 KiB/s rd, 821 KiB/s wr, 17 op/s 2023-03-09T22:25:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:02 smithi038 ceph-mon[92422]: pgmap v14811: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 850 KiB/s rd, 821 KiB/s wr, 17 op/s 2023-03-09T22:25:03.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27275.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:25:04.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:04 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26813.737343 secs 2023-03-09T22:25:04.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:04 smithi122 ceph-mon[93869]: pgmap v14812: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 555 KiB/s rd, 555 KiB/s wr, 25 op/s 2023-03-09T22:25:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:04 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26813.737343 secs 2023-03-09T22:25:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:04 smithi181 ceph-mon[94189]: pgmap v14812: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 555 KiB/s rd, 555 KiB/s wr, 25 op/s 2023-03-09T22:25:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:04 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26813.737343 secs 2023-03-09T22:25:04.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:04 smithi038 ceph-mon[92422]: pgmap v14812: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 555 KiB/s rd, 555 KiB/s wr, 25 op/s 2023-03-09T22:25:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:05 smithi122 ceph-mon[93869]: osdmap e1141: 12 total, 12 up, 12 in 2023-03-09T22:25:05.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26816.366934 secs 2023-03-09T22:25:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:05 smithi181 ceph-mon[94189]: osdmap e1141: 12 total, 12 up, 12 in 2023-03-09T22:25:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26816.366934 secs 2023-03-09T22:25:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:05 smithi038 ceph-mon[92422]: osdmap e1141: 12 total, 12 up, 12 in 2023-03-09T22:25:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26816.366934 secs 2023-03-09T22:25:06.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:06 smithi122 ceph-mon[93869]: pgmap v14814: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 860 KiB/s rd, 902 KiB/s wr, 31 op/s 2023-03-09T22:25:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:06 smithi181 ceph-mon[94189]: pgmap v14814: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 860 KiB/s rd, 902 KiB/s wr, 31 op/s 2023-03-09T22:25:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:06 smithi038 ceph-mon[92422]: pgmap v14814: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 860 KiB/s rd, 902 KiB/s wr, 31 op/s 2023-03-09T22:25:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:08 smithi122 ceph-mon[93869]: pgmap v14815: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 524 KiB/s rd, 565 KiB/s wr, 28 op/s 2023-03-09T22:25:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:25:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:25:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:08 smithi181 ceph-mon[94189]: pgmap v14815: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 524 KiB/s rd, 565 KiB/s wr, 28 op/s 2023-03-09T22:25:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:25:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:25:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:08 smithi038 ceph-mon[92422]: pgmap v14815: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 524 KiB/s rd, 565 KiB/s wr, 28 op/s 2023-03-09T22:25:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:25:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:25:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27280.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:25:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:09 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26818.737455 secs 2023-03-09T22:25:09.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:25:09.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:25:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:09 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26818.737455 secs 2023-03-09T22:25:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:25:09.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:25:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:09 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26818.737455 secs 2023-03-09T22:25:09.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:25:09.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:25:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:10 smithi122 ceph-mon[93869]: pgmap v14816: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 524 KiB/s rd, 565 KiB/s wr, 28 op/s 2023-03-09T22:25:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26821.366981 secs 2023-03-09T22:25:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:10 smithi181 ceph-mon[94189]: pgmap v14816: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 524 KiB/s rd, 565 KiB/s wr, 28 op/s 2023-03-09T22:25:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26821.366981 secs 2023-03-09T22:25:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:10 smithi038 ceph-mon[92422]: pgmap v14816: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 524 KiB/s rd, 565 KiB/s wr, 28 op/s 2023-03-09T22:25:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26821.366981 secs 2023-03-09T22:25:12.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:12 smithi122 ceph-mon[93869]: pgmap v14817: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 396 KiB/s wr, 25 op/s 2023-03-09T22:25:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:12 smithi181 ceph-mon[94189]: pgmap v14817: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 396 KiB/s wr, 25 op/s 2023-03-09T22:25:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:12 smithi038 ceph-mon[92422]: pgmap v14817: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 396 KiB/s wr, 25 op/s 2023-03-09T22:25:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26823.737587 secs 2023-03-09T22:25:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26823.737587 secs 2023-03-09T22:25:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26823.737587 secs 2023-03-09T22:25:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27285.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:25:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:14 smithi122 ceph-mon[93869]: pgmap v14818: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 371 KiB/s wr, 10 op/s 2023-03-09T22:25:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:14 smithi181 ceph-mon[94189]: pgmap v14818: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 371 KiB/s wr, 10 op/s 2023-03-09T22:25:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:14 smithi038 ceph-mon[92422]: pgmap v14818: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 371 KiB/s wr, 10 op/s 2023-03-09T22:25:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26826.367041 secs 2023-03-09T22:25:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26826.367041 secs 2023-03-09T22:25:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26826.367041 secs 2023-03-09T22:25:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:16 smithi122 ceph-mon[93869]: pgmap v14819: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 123 KiB/s wr, 8 op/s 2023-03-09T22:25:16.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:16 smithi122 ceph-mon[93869]: osdmap e1142: 12 total, 12 up, 12 in 2023-03-09T22:25:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:16 smithi181 ceph-mon[94189]: pgmap v14819: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 123 KiB/s wr, 8 op/s 2023-03-09T22:25:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:16 smithi181 ceph-mon[94189]: osdmap e1142: 12 total, 12 up, 12 in 2023-03-09T22:25:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:16 smithi038 ceph-mon[92422]: pgmap v14819: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 123 KiB/s wr, 8 op/s 2023-03-09T22:25:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:16 smithi038 ceph-mon[92422]: osdmap e1142: 12 total, 12 up, 12 in 2023-03-09T22:25:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:18 smithi181 ceph-mon[94189]: pgmap v14821: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 133 KiB/s wr, 8 op/s 2023-03-09T22:25:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26828.737711 secs 2023-03-09T22:25:18.503 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:18 smithi038 ceph-mon[92422]: pgmap v14821: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 133 KiB/s wr, 8 op/s 2023-03-09T22:25:18.504 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26828.737711 secs 2023-03-09T22:25:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:18 smithi122 ceph-mon[93869]: pgmap v14821: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 133 KiB/s wr, 8 op/s 2023-03-09T22:25:18.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26828.737711 secs 2023-03-09T22:25:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27290.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:25:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:20 smithi038 ceph-mon[92422]: pgmap v14822: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 133 KiB/s wr, 8 op/s 2023-03-09T22:25:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26831.367102 secs 2023-03-09T22:25:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:20 smithi122 ceph-mon[93869]: pgmap v14822: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 133 KiB/s wr, 8 op/s 2023-03-09T22:25:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26831.367102 secs 2023-03-09T22:25:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:20 smithi181 ceph-mon[94189]: pgmap v14822: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 133 KiB/s wr, 8 op/s 2023-03-09T22:25:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26831.367102 secs 2023-03-09T22:25:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:22 smithi038 ceph-mon[92422]: pgmap v14823: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 7 op/s 2023-03-09T22:25:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:22 smithi122 ceph-mon[93869]: pgmap v14823: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 7 op/s 2023-03-09T22:25:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:22 smithi181 ceph-mon[94189]: pgmap v14823: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 7 op/s 2023-03-09T22:25:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26833.737840 secs 2023-03-09T22:25:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26833.737840 secs 2023-03-09T22:25:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26833.737840 secs 2023-03-09T22:25:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27295.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:25:24.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:24 smithi122 ceph-mon[93869]: pgmap v14824: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T22:25:24.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:24 smithi181 ceph-mon[94189]: pgmap v14824: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T22:25:24.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:24 smithi038 ceph-mon[92422]: pgmap v14824: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T22:25:25.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26836.367161 secs 2023-03-09T22:25:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26836.367161 secs 2023-03-09T22:25:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26836.367161 secs 2023-03-09T22:25:26.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:26 smithi122 ceph-mon[93869]: pgmap v14825: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T22:25:26.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:26 smithi181 ceph-mon[94189]: pgmap v14825: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T22:25:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:26 smithi038 ceph-mon[92422]: pgmap v14825: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T22:25:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:28 smithi122 ceph-mon[93869]: pgmap v14826: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T22:25:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26838.737940 secs 2023-03-09T22:25:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:28 smithi181 ceph-mon[94189]: pgmap v14826: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T22:25:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26838.737940 secs 2023-03-09T22:25:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:28 smithi038 ceph-mon[92422]: pgmap v14826: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T22:25:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26838.737940 secs 2023-03-09T22:25:28.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27300.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:25:29.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:29 smithi122 ceph-mon[93869]: pgmap v14827: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T22:25:29.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:29 smithi181 ceph-mon[94189]: pgmap v14827: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T22:25:29.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:29 smithi038 ceph-mon[92422]: pgmap v14827: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T22:25:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26841.367224 secs 2023-03-09T22:25:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26841.367224 secs 2023-03-09T22:25:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26841.367224 secs 2023-03-09T22:25:32.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:32 smithi122 ceph-mon[93869]: pgmap v14828: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 511 B/s wr, 8 op/s 2023-03-09T22:25:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:32 smithi181 ceph-mon[94189]: pgmap v14828: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 511 B/s wr, 8 op/s 2023-03-09T22:25:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:32 smithi038 ceph-mon[92422]: pgmap v14828: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 511 B/s wr, 8 op/s 2023-03-09T22:25:33.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27305.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:25:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:34 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26843.738062 secs 2023-03-09T22:25:34.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:34 smithi122 ceph-mon[93869]: pgmap v14829: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 105 KiB/s wr, 6 op/s 2023-03-09T22:25:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:34 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26843.738062 secs 2023-03-09T22:25:34.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:34 smithi181 ceph-mon[94189]: pgmap v14829: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 105 KiB/s wr, 6 op/s 2023-03-09T22:25:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:34 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26843.738062 secs 2023-03-09T22:25:34.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:34 smithi038 ceph-mon[92422]: pgmap v14829: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 105 KiB/s wr, 6 op/s 2023-03-09T22:25:35.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26846.367283 secs 2023-03-09T22:25:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26846.367283 secs 2023-03-09T22:25:35.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26846.367283 secs 2023-03-09T22:25:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:36 smithi122 ceph-mon[93869]: pgmap v14830: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 114 KiB/s wr, 7 op/s 2023-03-09T22:25:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:36 smithi181 ceph-mon[94189]: pgmap v14830: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 114 KiB/s wr, 7 op/s 2023-03-09T22:25:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:36 smithi038 ceph-mon[92422]: pgmap v14830: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 114 KiB/s wr, 7 op/s 2023-03-09T22:25:38.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:38 smithi122 ceph-mon[93869]: pgmap v14831: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 114 KiB/s wr, 9 op/s 2023-03-09T22:25:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:38 smithi181 ceph-mon[94189]: pgmap v14831: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 114 KiB/s wr, 9 op/s 2023-03-09T22:25:38.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:38 smithi038 ceph-mon[92422]: pgmap v14831: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 114 KiB/s wr, 9 op/s 2023-03-09T22:25:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27310.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:25:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:39 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26848.738179 secs 2023-03-09T22:25:39.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:39 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26848.738179 secs 2023-03-09T22:25:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:39 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26848.738179 secs 2023-03-09T22:25:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:40 smithi122 ceph-mon[93869]: pgmap v14832: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 114 KiB/s wr, 7 op/s 2023-03-09T22:25:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26851.367345 secs 2023-03-09T22:25:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:40 smithi181 ceph-mon[94189]: pgmap v14832: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 114 KiB/s wr, 7 op/s 2023-03-09T22:25:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26851.367345 secs 2023-03-09T22:25:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:40 smithi038 ceph-mon[92422]: pgmap v14832: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 114 KiB/s wr, 7 op/s 2023-03-09T22:25:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26851.367345 secs 2023-03-09T22:25:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:25:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:25:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:25:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:25:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:25:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:25:42.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:42 smithi122 ceph-mon[93869]: pgmap v14833: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 114 KiB/s wr, 9 op/s 2023-03-09T22:25:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:42 smithi181 ceph-mon[94189]: pgmap v14833: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 114 KiB/s wr, 9 op/s 2023-03-09T22:25:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:42 smithi038 ceph-mon[92422]: pgmap v14833: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 114 KiB/s wr, 9 op/s 2023-03-09T22:25:43.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26853.738243 secs 2023-03-09T22:25:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26853.738243 secs 2023-03-09T22:25:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26853.738243 secs 2023-03-09T22:25:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27315.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:25:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:44 smithi181 ceph-mon[94189]: pgmap v14834: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 351 KiB/s wr, 8 op/s 2023-03-09T22:25:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:44 smithi038 ceph-mon[92422]: pgmap v14834: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 351 KiB/s wr, 8 op/s 2023-03-09T22:25:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:44 smithi122 ceph-mon[93869]: pgmap v14834: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 351 KiB/s wr, 8 op/s 2023-03-09T22:25:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26856.367404 secs 2023-03-09T22:25:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26856.367404 secs 2023-03-09T22:25:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26856.367404 secs 2023-03-09T22:25:46.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:46 smithi038 ceph-mon[92422]: pgmap v14835: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 247 KiB/s wr, 8 op/s 2023-03-09T22:25:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:46 smithi122 ceph-mon[93869]: pgmap v14835: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 247 KiB/s wr, 8 op/s 2023-03-09T22:25:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:46 smithi181 ceph-mon[94189]: pgmap v14835: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 247 KiB/s wr, 8 op/s 2023-03-09T22:25:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:48 smithi122 ceph-mon[93869]: pgmap v14836: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 442 KiB/s wr, 11 op/s 2023-03-09T22:25:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:25:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:25:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:48 smithi181 ceph-mon[94189]: pgmap v14836: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 442 KiB/s wr, 11 op/s 2023-03-09T22:25:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:25:48.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:25:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:48 smithi038 ceph-mon[92422]: pgmap v14836: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 442 KiB/s wr, 11 op/s 2023-03-09T22:25:48.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:25:48.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:25:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27320.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:25:49.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:25:49.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:25:49.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:49 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:25:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:25:49.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:25:49.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:49 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:25:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:25:49.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:25:49.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:49 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:25:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:50 smithi122 ceph-mon[93869]: pgmap v14837: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 561 KiB/s rd, 442 KiB/s wr, 10 op/s 2023-03-09T22:25:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:50 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:25:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:50 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:25:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:50 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:25:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26861.367465 secs 2023-03-09T22:25:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:50 smithi181 ceph-mon[94189]: pgmap v14837: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 561 KiB/s rd, 442 KiB/s wr, 10 op/s 2023-03-09T22:25:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:50 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:25:50.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:50 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:25:50.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:50 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:25:50.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26861.367465 secs 2023-03-09T22:25:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:50 smithi038 ceph-mon[92422]: pgmap v14837: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 561 KiB/s rd, 442 KiB/s wr, 10 op/s 2023-03-09T22:25:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:50 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:25:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:50 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:25:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:50 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:25:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26861.367465 secs 2023-03-09T22:25:52.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:52 smithi122 ceph-mon[93869]: pgmap v14838: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 490 KiB/s wr, 13 op/s 2023-03-09T22:25:52.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:25:52.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:25:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:52 smithi181 ceph-mon[94189]: pgmap v14838: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 490 KiB/s wr, 13 op/s 2023-03-09T22:25:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:25:52.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:25:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:52 smithi038 ceph-mon[92422]: pgmap v14838: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 490 KiB/s wr, 13 op/s 2023-03-09T22:25:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:25:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:25:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:25:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:25:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:25:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27325.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:25:54.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:54 smithi122 ceph-mon[93869]: pgmap v14839: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 491 KiB/s wr, 11 op/s 2023-03-09T22:25:54.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:54 smithi181 ceph-mon[94189]: pgmap v14839: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 491 KiB/s wr, 11 op/s 2023-03-09T22:25:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:54 smithi038 ceph-mon[92422]: pgmap v14839: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 491 KiB/s wr, 11 op/s 2023-03-09T22:25:55.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26866.367524 secs 2023-03-09T22:25:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26866.367524 secs 2023-03-09T22:25:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26866.367524 secs 2023-03-09T22:25:56.370 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:56 smithi122 ceph-mon[93869]: pgmap v14840: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 536 KiB/s rd, 537 KiB/s wr, 10 op/s 2023-03-09T22:25:56.370 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:25:56.371 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:25:56.371 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:25:56.371 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:25:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:56 smithi181 ceph-mon[94189]: pgmap v14840: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 536 KiB/s rd, 537 KiB/s wr, 10 op/s 2023-03-09T22:25:56.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:25:56.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:25:56.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:25:56.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:25:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:56 smithi038 ceph-mon[92422]: pgmap v14840: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 536 KiB/s rd, 537 KiB/s wr, 10 op/s 2023-03-09T22:25:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:25:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:25:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:25:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:25:57.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:25:57.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:25:57.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:57 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:25:57.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:57 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:25:57.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:57 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:25:57.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:57 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:25:57.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:57 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:25:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:25:57.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:25:57.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:57 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:25:57.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:57 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:25:57.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:57 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:25:57.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:57 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:25:57.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:57 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:25:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:25:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:25:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:57 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:25:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:57 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:25:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:57 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:25:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:57 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:25:57.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:57 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:25:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:25:58 smithi122 ceph-mon[93869]: pgmap v14841: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 704 KiB/s rd, 715 KiB/s wr, 14 op/s 2023-03-09T22:25:58.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:25:58 smithi181 ceph-mon[94189]: pgmap v14841: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 704 KiB/s rd, 715 KiB/s wr, 14 op/s 2023-03-09T22:25:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:25:58 smithi038 ceph-mon[92422]: pgmap v14841: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 704 KiB/s rd, 715 KiB/s wr, 14 op/s 2023-03-09T22:25:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27330.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:26:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:00 smithi122 ceph-mon[93869]: pgmap v14842: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 567 KiB/s rd, 581 KiB/s wr, 11 op/s 2023-03-09T22:26:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:26:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:26:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26871.367584 secs 2023-03-09T22:26:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:00 smithi181 ceph-mon[94189]: pgmap v14842: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 567 KiB/s rd, 581 KiB/s wr, 11 op/s 2023-03-09T22:26:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:26:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:26:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26871.367584 secs 2023-03-09T22:26:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:00 smithi038 ceph-mon[92422]: pgmap v14842: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 567 KiB/s rd, 581 KiB/s wr, 11 op/s 2023-03-09T22:26:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:26:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:26:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26871.367584 secs 2023-03-09T22:26:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:26:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:26:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:26:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:26:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:26:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:26:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:02 smithi038 ceph-mon[92422]: pgmap v14843: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 665 KiB/s rd, 685 KiB/s wr, 16 op/s 2023-03-09T22:26:02.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:02 smithi122 ceph-mon[93869]: pgmap v14843: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 665 KiB/s rd, 685 KiB/s wr, 16 op/s 2023-03-09T22:26:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:02 smithi181 ceph-mon[94189]: pgmap v14843: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 665 KiB/s rd, 685 KiB/s wr, 16 op/s 2023-03-09T22:26:03.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26873.738678 secs 2023-03-09T22:26:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26873.738678 secs 2023-03-09T22:26:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26873.738678 secs 2023-03-09T22:26:03.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27335.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:26:04.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:04 smithi122 ceph-mon[93869]: pgmap v14844: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 763 KiB/s rd, 930 KiB/s wr, 27 op/s 2023-03-09T22:26:04.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:04 smithi181 ceph-mon[94189]: pgmap v14844: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 763 KiB/s rd, 930 KiB/s wr, 27 op/s 2023-03-09T22:26:04.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:04 smithi038 ceph-mon[92422]: pgmap v14844: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 763 KiB/s rd, 930 KiB/s wr, 27 op/s 2023-03-09T22:26:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:05 smithi122 ceph-mon[93869]: osdmap e1143: 12 total, 12 up, 12 in 2023-03-09T22:26:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26876.367643 secs 2023-03-09T22:26:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:05 smithi181 ceph-mon[94189]: osdmap e1143: 12 total, 12 up, 12 in 2023-03-09T22:26:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26876.367643 secs 2023-03-09T22:26:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:05 smithi038 ceph-mon[92422]: osdmap e1143: 12 total, 12 up, 12 in 2023-03-09T22:26:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26876.367643 secs 2023-03-09T22:26:06.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:06 smithi122 ceph-mon[93869]: pgmap v14846: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 728 KiB/s rd, 775 KiB/s wr, 31 op/s 2023-03-09T22:26:06.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:06 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:26:06.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:06 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:26:06.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:26:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:06 smithi181 ceph-mon[94189]: pgmap v14846: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 728 KiB/s rd, 775 KiB/s wr, 31 op/s 2023-03-09T22:26:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:06 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:26:06.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:06 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:26:06.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:26:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:06 smithi038 ceph-mon[92422]: pgmap v14846: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 728 KiB/s rd, 775 KiB/s wr, 31 op/s 2023-03-09T22:26:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:06 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:26:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:06 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:26:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:26:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:08 smithi122 ceph-mon[93869]: pgmap v14847: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 669 KiB/s rd, 717 KiB/s wr, 31 op/s 2023-03-09T22:26:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26878.738811 secs 2023-03-09T22:26:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:08 smithi181 ceph-mon[94189]: pgmap v14847: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 669 KiB/s rd, 717 KiB/s wr, 31 op/s 2023-03-09T22:26:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26878.738811 secs 2023-03-09T22:26:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:08 smithi038 ceph-mon[92422]: pgmap v14847: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 669 KiB/s rd, 717 KiB/s wr, 31 op/s 2023-03-09T22:26:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26878.738811 secs 2023-03-09T22:26:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27340.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:26:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:26:09.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:26:09.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:09 smithi122 ceph-mon[93869]: pgmap v14848: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 742 KiB/s rd, 632 KiB/s wr, 30 op/s 2023-03-09T22:26:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:26:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:26:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:09 smithi181 ceph-mon[94189]: pgmap v14848: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 742 KiB/s rd, 632 KiB/s wr, 30 op/s 2023-03-09T22:26:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:26:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:26:09.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:09 smithi038 ceph-mon[92422]: pgmap v14848: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 742 KiB/s rd, 632 KiB/s wr, 30 op/s 2023-03-09T22:26:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26881.367703 secs 2023-03-09T22:26:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26881.367703 secs 2023-03-09T22:26:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26881.367703 secs 2023-03-09T22:26:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:12 smithi122 ceph-mon[93869]: pgmap v14849: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 493 KiB/s rd, 508 KiB/s wr, 25 op/s 2023-03-09T22:26:12.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:26:12.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:26:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:12 smithi181 ceph-mon[94189]: pgmap v14849: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 493 KiB/s rd, 508 KiB/s wr, 25 op/s 2023-03-09T22:26:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:26:12.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:26:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:12 smithi038 ceph-mon[92422]: pgmap v14849: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 493 KiB/s rd, 508 KiB/s wr, 25 op/s 2023-03-09T22:26:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:26:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:26:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27345.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:26:14.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:14 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26883.738927 secs 2023-03-09T22:26:14.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:14 smithi122 ceph-mon[93869]: pgmap v14850: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 156 KiB/s wr, 9 op/s 2023-03-09T22:26:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:14 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26883.738927 secs 2023-03-09T22:26:14.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:14 smithi181 ceph-mon[94189]: pgmap v14850: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 156 KiB/s wr, 9 op/s 2023-03-09T22:26:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:14 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26883.738927 secs 2023-03-09T22:26:14.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:14 smithi038 ceph-mon[92422]: pgmap v14850: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 156 KiB/s wr, 9 op/s 2023-03-09T22:26:15.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26886.367764 secs 2023-03-09T22:26:15.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26886.367764 secs 2023-03-09T22:26:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26886.367764 secs 2023-03-09T22:26:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:16 smithi122 ceph-mon[93869]: pgmap v14851: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 293 KiB/s rd, 145 KiB/s wr, 8 op/s 2023-03-09T22:26:16.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:16 smithi122 ceph-mon[93869]: osdmap e1144: 12 total, 12 up, 12 in 2023-03-09T22:26:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:16 smithi181 ceph-mon[94189]: pgmap v14851: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 293 KiB/s rd, 145 KiB/s wr, 8 op/s 2023-03-09T22:26:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:16 smithi181 ceph-mon[94189]: osdmap e1144: 12 total, 12 up, 12 in 2023-03-09T22:26:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:16 smithi038 ceph-mon[92422]: pgmap v14851: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 293 KiB/s rd, 145 KiB/s wr, 8 op/s 2023-03-09T22:26:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:16 smithi038 ceph-mon[92422]: osdmap e1144: 12 total, 12 up, 12 in 2023-03-09T22:26:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:18 smithi122 ceph-mon[93869]: pgmap v14853: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 114 KiB/s wr, 7 op/s 2023-03-09T22:26:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:18 smithi181 ceph-mon[94189]: pgmap v14853: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 114 KiB/s wr, 7 op/s 2023-03-09T22:26:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:18 smithi038 ceph-mon[92422]: pgmap v14853: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 114 KiB/s wr, 7 op/s 2023-03-09T22:26:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27350.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:26:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:19 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26888.739047 secs 2023-03-09T22:26:19.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:19 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26888.739047 secs 2023-03-09T22:26:19.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:19 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26888.739047 secs 2023-03-09T22:26:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:20 smithi122 ceph-mon[93869]: pgmap v14854: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 114 KiB/s wr, 7 op/s 2023-03-09T22:26:20.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26891.367824 secs 2023-03-09T22:26:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:20 smithi181 ceph-mon[94189]: pgmap v14854: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 114 KiB/s wr, 7 op/s 2023-03-09T22:26:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26891.367824 secs 2023-03-09T22:26:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:20 smithi038 ceph-mon[92422]: pgmap v14854: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 114 KiB/s wr, 7 op/s 2023-03-09T22:26:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26891.367824 secs 2023-03-09T22:26:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:22 smithi181 ceph-mon[94189]: pgmap v14855: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 243 KiB/s wr, 6 op/s 2023-03-09T22:26:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:22 smithi038 ceph-mon[92422]: pgmap v14855: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 243 KiB/s wr, 6 op/s 2023-03-09T22:26:22.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:22 smithi122 ceph-mon[93869]: pgmap v14855: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 243 KiB/s wr, 6 op/s 2023-03-09T22:26:23.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26893.739163 secs 2023-03-09T22:26:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26893.739163 secs 2023-03-09T22:26:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26893.739163 secs 2023-03-09T22:26:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27355.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:26:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:24 smithi038 ceph-mon[92422]: pgmap v14856: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 243 KiB/s wr, 6 op/s 2023-03-09T22:26:24.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:24 smithi122 ceph-mon[93869]: pgmap v14856: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 243 KiB/s wr, 6 op/s 2023-03-09T22:26:24.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:24 smithi181 ceph-mon[94189]: pgmap v14856: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 243 KiB/s wr, 6 op/s 2023-03-09T22:26:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26896.367904 secs 2023-03-09T22:26:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26896.367904 secs 2023-03-09T22:26:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26896.367904 secs 2023-03-09T22:26:26.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:26 smithi122 ceph-mon[93869]: pgmap v14857: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 243 KiB/s wr, 6 op/s 2023-03-09T22:26:26.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:26 smithi181 ceph-mon[94189]: pgmap v14857: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 243 KiB/s wr, 6 op/s 2023-03-09T22:26:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:26 smithi038 ceph-mon[92422]: pgmap v14857: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 243 KiB/s wr, 6 op/s 2023-03-09T22:26:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:28 smithi122 ceph-mon[93869]: pgmap v14858: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 120 KiB/s wr, 6 op/s 2023-03-09T22:26:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26898.739237 secs 2023-03-09T22:26:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:28 smithi181 ceph-mon[94189]: pgmap v14858: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 120 KiB/s wr, 6 op/s 2023-03-09T22:26:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26898.739237 secs 2023-03-09T22:26:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:28 smithi038 ceph-mon[92422]: pgmap v14858: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 120 KiB/s wr, 6 op/s 2023-03-09T22:26:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26898.739237 secs 2023-03-09T22:26:28.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27360.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:26:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:30 smithi122 ceph-mon[93869]: pgmap v14859: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 108 KiB/s wr, 5 op/s 2023-03-09T22:26:30.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26901.367964 secs 2023-03-09T22:26:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:30 smithi181 ceph-mon[94189]: pgmap v14859: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 108 KiB/s wr, 5 op/s 2023-03-09T22:26:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26901.367964 secs 2023-03-09T22:26:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:30 smithi038 ceph-mon[92422]: pgmap v14859: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 108 KiB/s wr, 5 op/s 2023-03-09T22:26:30.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26901.367964 secs 2023-03-09T22:26:32.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:32 smithi122 ceph-mon[93869]: pgmap v14860: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 108 KiB/s wr, 8 op/s 2023-03-09T22:26:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:32 smithi181 ceph-mon[94189]: pgmap v14860: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 108 KiB/s wr, 8 op/s 2023-03-09T22:26:32.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:32 smithi038 ceph-mon[92422]: pgmap v14860: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 108 KiB/s wr, 8 op/s 2023-03-09T22:26:33.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26903.739376 secs 2023-03-09T22:26:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26903.739376 secs 2023-03-09T22:26:33.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26903.739376 secs 2023-03-09T22:26:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27365.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:26:34.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:34 smithi122 ceph-mon[93869]: pgmap v14861: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2023-03-09T22:26:34.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:34 smithi181 ceph-mon[94189]: pgmap v14861: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2023-03-09T22:26:34.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:34 smithi038 ceph-mon[92422]: pgmap v14861: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2023-03-09T22:26:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26906.367985 secs 2023-03-09T22:26:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26906.367985 secs 2023-03-09T22:26:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26906.367985 secs 2023-03-09T22:26:36.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:36 smithi122 ceph-mon[93869]: pgmap v14862: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2023-03-09T22:26:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:36 smithi181 ceph-mon[94189]: pgmap v14862: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2023-03-09T22:26:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:36 smithi038 ceph-mon[92422]: pgmap v14862: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2023-03-09T22:26:37.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:37 smithi122 ceph-mon[93869]: pgmap v14863: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 170 B/s wr, 8 op/s 2023-03-09T22:26:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:37 smithi181 ceph-mon[94189]: pgmap v14863: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 170 B/s wr, 8 op/s 2023-03-09T22:26:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:37 smithi038 ceph-mon[92422]: pgmap v14863: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 170 B/s wr, 8 op/s 2023-03-09T22:26:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26908.739505 secs 2023-03-09T22:26:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26908.739505 secs 2023-03-09T22:26:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26908.739505 secs 2023-03-09T22:26:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27370.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:26:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:39 smithi122 ceph-mon[93869]: pgmap v14864: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T22:26:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:39 smithi181 ceph-mon[94189]: pgmap v14864: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T22:26:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:39 smithi038 ceph-mon[92422]: pgmap v14864: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T22:26:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26911.368045 secs 2023-03-09T22:26:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26911.368045 secs 2023-03-09T22:26:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26911.368045 secs 2023-03-09T22:26:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:26:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:26:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:41 smithi122 ceph-mon[93869]: pgmap v14865: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 142 KiB/s wr, 8 op/s 2023-03-09T22:26:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:26:41.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:26:41.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:41 smithi181 ceph-mon[94189]: pgmap v14865: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 142 KiB/s wr, 8 op/s 2023-03-09T22:26:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:26:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:26:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:41 smithi038 ceph-mon[92422]: pgmap v14865: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 142 KiB/s wr, 8 op/s 2023-03-09T22:26:43.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27375.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:26:44.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:44 smithi122 ceph-mon[93869]: pgmap v14866: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 142 KiB/s wr, 6 op/s 2023-03-09T22:26:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:44 smithi181 ceph-mon[94189]: pgmap v14866: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 142 KiB/s wr, 6 op/s 2023-03-09T22:26:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:44 smithi038 ceph-mon[92422]: pgmap v14866: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 142 KiB/s wr, 6 op/s 2023-03-09T22:26:45.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26916.368105 secs 2023-03-09T22:26:45.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:45 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:26:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26916.368105 secs 2023-03-09T22:26:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:45 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:26:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26916.368105 secs 2023-03-09T22:26:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:45 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:26:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:46 smithi122 ceph-mon[93869]: pgmap v14867: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 142 KiB/s wr, 6 op/s 2023-03-09T22:26:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:46 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:26:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:46 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:26:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:46 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:26:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:46 smithi181 ceph-mon[94189]: pgmap v14867: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 142 KiB/s wr, 6 op/s 2023-03-09T22:26:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:46 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:26:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:46 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:26:46.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:46 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:26:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:46 smithi038 ceph-mon[92422]: pgmap v14867: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 142 KiB/s wr, 6 op/s 2023-03-09T22:26:46.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:46 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:26:46.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:46 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:26:46.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:46 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:26:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:48 smithi122 ceph-mon[93869]: pgmap v14868: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 459 KiB/s wr, 12 op/s 2023-03-09T22:26:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:26:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:26:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:48 smithi181 ceph-mon[94189]: pgmap v14868: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 459 KiB/s wr, 12 op/s 2023-03-09T22:26:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:26:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:26:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:48 smithi038 ceph-mon[92422]: pgmap v14868: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 459 KiB/s wr, 12 op/s 2023-03-09T22:26:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:26:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:26:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27380.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:26:50.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:50 smithi122 ceph-mon[93869]: pgmap v14869: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 474 KiB/s rd, 459 KiB/s wr, 10 op/s 2023-03-09T22:26:50.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26921.368164 secs 2023-03-09T22:26:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:50 smithi181 ceph-mon[94189]: pgmap v14869: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 474 KiB/s rd, 459 KiB/s wr, 10 op/s 2023-03-09T22:26:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26921.368164 secs 2023-03-09T22:26:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:50 smithi038 ceph-mon[92422]: pgmap v14869: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 474 KiB/s rd, 459 KiB/s wr, 10 op/s 2023-03-09T22:26:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26921.368164 secs 2023-03-09T22:26:52.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:52 smithi122 ceph-mon[93869]: pgmap v14870: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 460 KiB/s wr, 13 op/s 2023-03-09T22:26:52.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:26:52.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:26:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:52 smithi181 ceph-mon[94189]: pgmap v14870: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 460 KiB/s wr, 13 op/s 2023-03-09T22:26:52.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:26:52.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:26:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:52 smithi038 ceph-mon[92422]: pgmap v14870: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 460 KiB/s wr, 13 op/s 2023-03-09T22:26:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:26:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:26:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:26:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:26:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:26:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:26:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:26:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:26:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27385.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:26:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:54 smithi038 ceph-mon[92422]: pgmap v14871: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 366 KiB/s wr, 11 op/s 2023-03-09T22:26:54.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:54 smithi122 ceph-mon[93869]: pgmap v14871: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 366 KiB/s wr, 11 op/s 2023-03-09T22:26:54.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:54 smithi181 ceph-mon[94189]: pgmap v14871: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 366 KiB/s wr, 11 op/s 2023-03-09T22:26:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26926.368224 secs 2023-03-09T22:26:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26926.368224 secs 2023-03-09T22:26:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26926.368224 secs 2023-03-09T22:26:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:56 smithi038 ceph-mon[92422]: pgmap v14872: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 366 KiB/s wr, 11 op/s 2023-03-09T22:26:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:56 smithi122 ceph-mon[93869]: pgmap v14872: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 366 KiB/s wr, 11 op/s 2023-03-09T22:26:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:56 smithi181 ceph-mon[94189]: pgmap v14872: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 366 KiB/s wr, 11 op/s 2023-03-09T22:26:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:26:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:26:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:57 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:26:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:26:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:26:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:57 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:26:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:26:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:26:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:57 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:26:58.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:58 smithi122 ceph-mon[93869]: pgmap v14873: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 810 KiB/s rd, 806 KiB/s wr, 16 op/s 2023-03-09T22:26:58.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:26:58.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:26:58.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:26:58.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:26:58 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 26928.739938 secs 2023-03-09T22:26:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:58 smithi181 ceph-mon[94189]: pgmap v14873: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 810 KiB/s rd, 806 KiB/s wr, 16 op/s 2023-03-09T22:26:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:26:58.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:26:58.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:26:58.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:26:58 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 26928.739938 secs 2023-03-09T22:26:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:58 smithi038 ceph-mon[92422]: pgmap v14873: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 810 KiB/s rd, 806 KiB/s wr, 16 op/s 2023-03-09T22:26:58.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:26:58.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:26:58.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:26:58.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:26:58 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 26928.739938 secs 2023-03-09T22:26:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27390.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:27:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:00 smithi122 ceph-mon[93869]: pgmap v14874: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 490 KiB/s wr, 11 op/s 2023-03-09T22:27:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:27:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:27:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26931.368281 secs 2023-03-09T22:27:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:00 smithi181 ceph-mon[94189]: pgmap v14874: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 490 KiB/s wr, 11 op/s 2023-03-09T22:27:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:27:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:27:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26931.368281 secs 2023-03-09T22:27:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:00 smithi038 ceph-mon[92422]: pgmap v14874: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 490 KiB/s wr, 11 op/s 2023-03-09T22:27:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:27:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:27:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26931.368281 secs 2023-03-09T22:27:02.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:02 smithi122 ceph-mon[93869]: pgmap v14875: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 984 KiB/s rd, 953 KiB/s wr, 15 op/s 2023-03-09T22:27:02.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:02 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:27:02.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:02 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:27:02.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:02 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:27:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:02 smithi181 ceph-mon[94189]: pgmap v14875: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 984 KiB/s rd, 953 KiB/s wr, 15 op/s 2023-03-09T22:27:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:02 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:27:02.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:02 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:27:02.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:02 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:27:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:02 smithi038 ceph-mon[92422]: pgmap v14875: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 984 KiB/s rd, 953 KiB/s wr, 15 op/s 2023-03-09T22:27:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:02 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:27:02.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:02 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:27:02.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:02 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:27:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27395.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:27:04.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:04 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 26933.740060 secs 2023-03-09T22:27:04.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:04 smithi122 ceph-mon[93869]: pgmap v14876: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 990 KiB/s rd, 973 KiB/s wr, 23 op/s 2023-03-09T22:27:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:04 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 26933.740060 secs 2023-03-09T22:27:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:04 smithi181 ceph-mon[94189]: pgmap v14876: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 990 KiB/s rd, 973 KiB/s wr, 23 op/s 2023-03-09T22:27:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:04 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 26933.740060 secs 2023-03-09T22:27:04.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:04 smithi038 ceph-mon[92422]: pgmap v14876: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 990 KiB/s rd, 973 KiB/s wr, 23 op/s 2023-03-09T22:27:05.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26936.368364 secs 2023-03-09T22:27:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26936.368364 secs 2023-03-09T22:27:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26936.368364 secs 2023-03-09T22:27:06.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:06 smithi122 ceph-mon[93869]: pgmap v14877: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 942 KiB/s rd, 925 KiB/s wr, 22 op/s 2023-03-09T22:27:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:06 smithi181 ceph-mon[94189]: pgmap v14877: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 942 KiB/s rd, 925 KiB/s wr, 22 op/s 2023-03-09T22:27:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:06 smithi038 ceph-mon[92422]: pgmap v14877: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 942 KiB/s rd, 925 KiB/s wr, 22 op/s 2023-03-09T22:27:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:08 smithi122 ceph-mon[93869]: pgmap v14878: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 30 op/s 2023-03-09T22:27:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:27:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:27:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26938.740176 secs 2023-03-09T22:27:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:08 smithi181 ceph-mon[94189]: pgmap v14878: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 30 op/s 2023-03-09T22:27:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:27:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:27:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26938.740176 secs 2023-03-09T22:27:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:08 smithi038 ceph-mon[92422]: pgmap v14878: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 30 op/s 2023-03-09T22:27:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:27:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:27:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26938.740176 secs 2023-03-09T22:27:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27400.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:27:09.503 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:27:09.503 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:27:09.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:27:09.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:27:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:27:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:27:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:10 smithi122 ceph-mon[93869]: pgmap v14879: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1016 KiB/s rd, 1021 KiB/s wr, 25 op/s 2023-03-09T22:27:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:10 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:27:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:10 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:27:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:10 smithi122 ceph-mon[93869]: osdmap e1145: 12 total, 12 up, 12 in 2023-03-09T22:27:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26941.368425 secs 2023-03-09T22:27:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:10 smithi181 ceph-mon[94189]: pgmap v14879: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1016 KiB/s rd, 1021 KiB/s wr, 25 op/s 2023-03-09T22:27:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:10 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:27:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:10 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:27:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:10 smithi181 ceph-mon[94189]: osdmap e1145: 12 total, 12 up, 12 in 2023-03-09T22:27:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26941.368425 secs 2023-03-09T22:27:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:10 smithi038 ceph-mon[92422]: pgmap v14879: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1016 KiB/s rd, 1021 KiB/s wr, 25 op/s 2023-03-09T22:27:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:10 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:27:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:10 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:27:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:10 smithi038 ceph-mon[92422]: osdmap e1145: 12 total, 12 up, 12 in 2023-03-09T22:27:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26941.368425 secs 2023-03-09T22:27:12.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:12 smithi122 ceph-mon[93869]: pgmap v14881: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 631 KiB/s rd, 672 KiB/s wr, 29 op/s 2023-03-09T22:27:12.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:27:12.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:27:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:12 smithi181 ceph-mon[94189]: pgmap v14881: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 631 KiB/s rd, 672 KiB/s wr, 29 op/s 2023-03-09T22:27:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:27:12.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:27:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:12 smithi038 ceph-mon[92422]: pgmap v14881: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 631 KiB/s rd, 672 KiB/s wr, 29 op/s 2023-03-09T22:27:12.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:27:12.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:27:13.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:27:13.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:27:13.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26943.740303 secs 2023-03-09T22:27:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:27:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:27:13.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26943.740303 secs 2023-03-09T22:27:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:27:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:27:13.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26943.740303 secs 2023-03-09T22:27:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27405.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:27:14.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:14 smithi122 ceph-mon[93869]: pgmap v14882: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 656 KiB/s rd, 681 KiB/s wr, 14 op/s 2023-03-09T22:27:14.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:14 smithi181 ceph-mon[94189]: pgmap v14882: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 656 KiB/s rd, 681 KiB/s wr, 14 op/s 2023-03-09T22:27:14.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:14 smithi038 ceph-mon[92422]: pgmap v14882: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 656 KiB/s rd, 681 KiB/s wr, 14 op/s 2023-03-09T22:27:15.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26946.368485 secs 2023-03-09T22:27:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26946.368485 secs 2023-03-09T22:27:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26946.368485 secs 2023-03-09T22:27:16.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:16 smithi122 ceph-mon[93869]: pgmap v14883: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 769 KiB/s rd, 794 KiB/s wr, 15 op/s 2023-03-09T22:27:16.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:16 smithi122 ceph-mon[93869]: osdmap e1146: 12 total, 12 up, 12 in 2023-03-09T22:27:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:16 smithi181 ceph-mon[94189]: pgmap v14883: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 769 KiB/s rd, 794 KiB/s wr, 15 op/s 2023-03-09T22:27:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:16 smithi181 ceph-mon[94189]: osdmap e1146: 12 total, 12 up, 12 in 2023-03-09T22:27:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:16 smithi038 ceph-mon[92422]: pgmap v14883: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 769 KiB/s rd, 794 KiB/s wr, 15 op/s 2023-03-09T22:27:16.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:16 smithi038 ceph-mon[92422]: osdmap e1146: 12 total, 12 up, 12 in 2023-03-09T22:27:18.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:18 smithi122 ceph-mon[93869]: pgmap v14885: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 188 KiB/s wr, 10 op/s 2023-03-09T22:27:18.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26948.740467 secs 2023-03-09T22:27:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:18 smithi181 ceph-mon[94189]: pgmap v14885: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 188 KiB/s wr, 10 op/s 2023-03-09T22:27:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26948.740467 secs 2023-03-09T22:27:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:18 smithi038 ceph-mon[92422]: pgmap v14885: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 188 KiB/s wr, 10 op/s 2023-03-09T22:27:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26948.740467 secs 2023-03-09T22:27:18.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27410.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:27:19.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:19 smithi122 ceph-mon[93869]: pgmap v14886: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 250 KiB/s wr, 9 op/s 2023-03-09T22:27:19.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:19 smithi181 ceph-mon[94189]: pgmap v14886: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 250 KiB/s wr, 9 op/s 2023-03-09T22:27:19.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:19 smithi038 ceph-mon[92422]: pgmap v14886: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 250 KiB/s wr, 9 op/s 2023-03-09T22:27:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26951.368545 secs 2023-03-09T22:27:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26951.368545 secs 2023-03-09T22:27:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26951.368545 secs 2023-03-09T22:27:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:22 smithi122 ceph-mon[93869]: pgmap v14887: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 242 KiB/s wr, 8 op/s 2023-03-09T22:27:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:22 smithi181 ceph-mon[94189]: pgmap v14887: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 242 KiB/s wr, 8 op/s 2023-03-09T22:27:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:22 smithi038 ceph-mon[92422]: pgmap v14887: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 242 KiB/s wr, 8 op/s 2023-03-09T22:27:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27415.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:27:24.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:24 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26953.740589 secs 2023-03-09T22:27:24.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:24 smithi122 ceph-mon[93869]: pgmap v14888: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 207 KiB/s wr, 7 op/s 2023-03-09T22:27:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:24 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26953.740589 secs 2023-03-09T22:27:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:24 smithi181 ceph-mon[94189]: pgmap v14888: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 207 KiB/s wr, 7 op/s 2023-03-09T22:27:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:24 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26953.740589 secs 2023-03-09T22:27:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:24 smithi038 ceph-mon[92422]: pgmap v14888: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 207 KiB/s wr, 7 op/s 2023-03-09T22:27:25.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26956.368607 secs 2023-03-09T22:27:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26956.368607 secs 2023-03-09T22:27:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26956.368607 secs 2023-03-09T22:27:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:26 smithi181 ceph-mon[94189]: pgmap v14889: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 299 KiB/s wr, 7 op/s 2023-03-09T22:27:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:26 smithi038 ceph-mon[92422]: pgmap v14889: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 299 KiB/s wr, 7 op/s 2023-03-09T22:27:26.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:26 smithi122 ceph-mon[93869]: pgmap v14889: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 299 KiB/s wr, 7 op/s 2023-03-09T22:27:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:28 smithi038 ceph-mon[92422]: pgmap v14890: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 275 KiB/s wr, 7 op/s 2023-03-09T22:27:28.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26958.740728 secs 2023-03-09T22:27:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:28 smithi122 ceph-mon[93869]: pgmap v14890: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 275 KiB/s wr, 7 op/s 2023-03-09T22:27:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26958.740728 secs 2023-03-09T22:27:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:28 smithi181 ceph-mon[94189]: pgmap v14890: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 275 KiB/s wr, 7 op/s 2023-03-09T22:27:28.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26958.740728 secs 2023-03-09T22:27:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27420.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:27:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:30 smithi038 ceph-mon[92422]: pgmap v14891: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 248 KiB/s wr, 6 op/s 2023-03-09T22:27:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26961.368668 secs 2023-03-09T22:27:30.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:30 smithi122 ceph-mon[93869]: pgmap v14891: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 248 KiB/s wr, 6 op/s 2023-03-09T22:27:30.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26961.368668 secs 2023-03-09T22:27:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:30 smithi181 ceph-mon[94189]: pgmap v14891: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 248 KiB/s wr, 6 op/s 2023-03-09T22:27:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26961.368668 secs 2023-03-09T22:27:32.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:32 smithi122 ceph-mon[93869]: pgmap v14892: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 170 KiB/s wr, 8 op/s 2023-03-09T22:27:32.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:32 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:27:32.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:32 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:27:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:32 smithi181 ceph-mon[94189]: pgmap v14892: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 170 KiB/s wr, 8 op/s 2023-03-09T22:27:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:32 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:27:32.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:32 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:27:32.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:32 smithi038 ceph-mon[92422]: pgmap v14892: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 170 KiB/s wr, 8 op/s 2023-03-09T22:27:32.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:32 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:27:32.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:32 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:27:33.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:33 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:27:33.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:33 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:27:33.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26963.740891 secs 2023-03-09T22:27:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:33 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:27:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:33 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:27:33.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26963.740891 secs 2023-03-09T22:27:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:33 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:27:33.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:33 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:27:33.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26963.740891 secs 2023-03-09T22:27:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27425.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:27:34.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:34 smithi122 ceph-mon[93869]: pgmap v14893: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 170 KiB/s wr, 7 op/s 2023-03-09T22:27:34.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:34 smithi181 ceph-mon[94189]: pgmap v14893: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 170 KiB/s wr, 7 op/s 2023-03-09T22:27:34.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:34 smithi038 ceph-mon[92422]: pgmap v14893: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 170 KiB/s wr, 7 op/s 2023-03-09T22:27:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26966.368738 secs 2023-03-09T22:27:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26966.368738 secs 2023-03-09T22:27:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26966.368738 secs 2023-03-09T22:27:36.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:36 smithi122 ceph-mon[93869]: pgmap v14894: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 170 KiB/s wr, 7 op/s 2023-03-09T22:27:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:36 smithi181 ceph-mon[94189]: pgmap v14894: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 170 KiB/s wr, 7 op/s 2023-03-09T22:27:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:36 smithi038 ceph-mon[92422]: pgmap v14894: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 170 KiB/s wr, 7 op/s 2023-03-09T22:27:38.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:38 smithi122 ceph-mon[93869]: pgmap v14895: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 597 B/s wr, 9 op/s 2023-03-09T22:27:38.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26968.741011 secs 2023-03-09T22:27:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:38 smithi181 ceph-mon[94189]: pgmap v14895: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 597 B/s wr, 9 op/s 2023-03-09T22:27:38.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26968.741011 secs 2023-03-09T22:27:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:38 smithi038 ceph-mon[92422]: pgmap v14895: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 597 B/s wr, 9 op/s 2023-03-09T22:27:38.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26968.741011 secs 2023-03-09T22:27:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27430.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:27:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:40 smithi122 ceph-mon[93869]: pgmap v14896: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 511 B/s wr, 7 op/s 2023-03-09T22:27:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26971.368797 secs 2023-03-09T22:27:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:40 smithi181 ceph-mon[94189]: pgmap v14896: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 511 B/s wr, 7 op/s 2023-03-09T22:27:40.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26971.368797 secs 2023-03-09T22:27:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:40 smithi038 ceph-mon[92422]: pgmap v14896: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 511 B/s wr, 7 op/s 2023-03-09T22:27:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26971.368797 secs 2023-03-09T22:27:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:27:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:27:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:27:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:27:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:27:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:27:42.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:42 smithi122 ceph-mon[93869]: pgmap v14897: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 220 KiB/s wr, 9 op/s 2023-03-09T22:27:42.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:42 smithi181 ceph-mon[94189]: pgmap v14897: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 220 KiB/s wr, 9 op/s 2023-03-09T22:27:42.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:42 smithi038 ceph-mon[92422]: pgmap v14897: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 220 KiB/s wr, 9 op/s 2023-03-09T22:27:43.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:43 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 26973.741080 secs 2023-03-09T22:27:43.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:43 smithi122 ceph-mon[93869]: pgmap v14898: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 241 KiB/s wr, 7 op/s 2023-03-09T22:27:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:43 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 26973.741080 secs 2023-03-09T22:27:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:43 smithi181 ceph-mon[94189]: pgmap v14898: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 241 KiB/s wr, 7 op/s 2023-03-09T22:27:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:43 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 26973.741080 secs 2023-03-09T22:27:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:43 smithi038 ceph-mon[92422]: pgmap v14898: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 241 KiB/s wr, 7 op/s 2023-03-09T22:27:43.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27435.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:27:45.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26976.368859 secs 2023-03-09T22:27:45.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:45 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:27:45.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:45 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:27:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26976.368859 secs 2023-03-09T22:27:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:45 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:27:45.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:45 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:27:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26976.368859 secs 2023-03-09T22:27:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:45 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:27:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:45 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:27:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:46 smithi122 ceph-mon[93869]: pgmap v14899: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 241 KiB/s wr, 7 op/s 2023-03-09T22:27:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:46 smithi181 ceph-mon[94189]: pgmap v14899: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 241 KiB/s wr, 7 op/s 2023-03-09T22:27:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:46 smithi038 ceph-mon[92422]: pgmap v14899: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 241 KiB/s wr, 7 op/s 2023-03-09T22:27:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:48 smithi122 ceph-mon[93869]: pgmap v14900: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 690 KiB/s rd, 583 KiB/s wr, 10 op/s 2023-03-09T22:27:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:27:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:27:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:48 smithi181 ceph-mon[94189]: pgmap v14900: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 690 KiB/s rd, 583 KiB/s wr, 10 op/s 2023-03-09T22:27:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:27:48.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:27:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:48 smithi038 ceph-mon[92422]: pgmap v14900: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 690 KiB/s rd, 583 KiB/s wr, 10 op/s 2023-03-09T22:27:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:27:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:27:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27440.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:27:49.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:49 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26978.741196 secs 2023-03-09T22:27:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:49 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26978.741196 secs 2023-03-09T22:27:49.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:49 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26978.741196 secs 2023-03-09T22:27:50.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:50 smithi181 ceph-mon[94189]: pgmap v14901: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 689 KiB/s rd, 583 KiB/s wr, 8 op/s 2023-03-09T22:27:50.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:50 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:27:50.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:50 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:27:50.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26981.368918 secs 2023-03-09T22:27:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:50 smithi038 ceph-mon[92422]: pgmap v14901: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 689 KiB/s rd, 583 KiB/s wr, 8 op/s 2023-03-09T22:27:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:50 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:27:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:50 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:27:50.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26981.368918 secs 2023-03-09T22:27:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:50 smithi122 ceph-mon[93869]: pgmap v14901: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 689 KiB/s rd, 583 KiB/s wr, 8 op/s 2023-03-09T22:27:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:50 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:27:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:50 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:27:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26981.368918 secs 2023-03-09T22:27:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:52 smithi038 ceph-mon[92422]: pgmap v14902: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 728 KiB/s rd, 609 KiB/s wr, 12 op/s 2023-03-09T22:27:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:52 smithi122 ceph-mon[93869]: pgmap v14902: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 728 KiB/s rd, 609 KiB/s wr, 12 op/s 2023-03-09T22:27:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:52 smithi181 ceph-mon[94189]: pgmap v14902: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 728 KiB/s rd, 609 KiB/s wr, 12 op/s 2023-03-09T22:27:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26983.741317 secs 2023-03-09T22:27:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26983.741317 secs 2023-03-09T22:27:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26983.741317 secs 2023-03-09T22:27:53.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27445.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:27:54.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:54 smithi038 ceph-mon[92422]: pgmap v14903: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1001 KiB/s rd, 624 KiB/s wr, 11 op/s 2023-03-09T22:27:54.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:54 smithi122 ceph-mon[93869]: pgmap v14903: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1001 KiB/s rd, 624 KiB/s wr, 11 op/s 2023-03-09T22:27:54.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:54 smithi181 ceph-mon[94189]: pgmap v14903: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1001 KiB/s rd, 624 KiB/s wr, 11 op/s 2023-03-09T22:27:55.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26986.368983 secs 2023-03-09T22:27:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26986.368983 secs 2023-03-09T22:27:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26986.368983 secs 2023-03-09T22:27:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:56 smithi122 ceph-mon[93869]: pgmap v14904: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 981 KiB/s rd, 603 KiB/s wr, 11 op/s 2023-03-09T22:27:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:27:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:27:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:56 smithi181 ceph-mon[94189]: pgmap v14904: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 981 KiB/s rd, 603 KiB/s wr, 11 op/s 2023-03-09T22:27:56.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:27:56.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:27:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:56 smithi038 ceph-mon[92422]: pgmap v14904: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 981 KiB/s rd, 603 KiB/s wr, 11 op/s 2023-03-09T22:27:56.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:27:56.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:27:57.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:27:57.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:27:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:27:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:27:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:27:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:27:58.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:58 smithi122 ceph-mon[93869]: pgmap v14905: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 885 KiB/s wr, 15 op/s 2023-03-09T22:27:58.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26988.741456 secs 2023-03-09T22:27:58.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:27:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:58 smithi181 ceph-mon[94189]: pgmap v14905: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 885 KiB/s wr, 15 op/s 2023-03-09T22:27:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26988.741456 secs 2023-03-09T22:27:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:27:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:58 smithi038 ceph-mon[92422]: pgmap v14905: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 885 KiB/s wr, 15 op/s 2023-03-09T22:27:58.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26988.741456 secs 2023-03-09T22:27:58.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:27:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27450.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:27:59.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:27:59.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:27:59.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:27:59.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:27:59.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:27:59.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:27:59.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:27:59.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:27:59.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:27:59.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:27:59 smithi122 ceph-mon[93869]: pgmap v14906: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 711 KiB/s rd, 543 KiB/s wr, 11 op/s 2023-03-09T22:27:59.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:27:59.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:27:59.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:27:59.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:27:59.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:27:59.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:27:59.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:27:59.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:27:59.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:27:59.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:27:59 smithi181 ceph-mon[94189]: pgmap v14906: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 711 KiB/s rd, 543 KiB/s wr, 11 op/s 2023-03-09T22:27:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:28:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:28:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:28:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:28:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:28:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:28:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:28:00.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:28:00.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:28:00.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:27:59 smithi038 ceph-mon[92422]: pgmap v14906: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 711 KiB/s rd, 543 KiB/s wr, 11 op/s 2023-03-09T22:28:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26991.369042 secs 2023-03-09T22:28:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26991.369042 secs 2023-03-09T22:28:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26991.369042 secs 2023-03-09T22:28:02.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:02 smithi122 ceph-mon[93869]: pgmap v14907: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 944 KiB/s rd, 796 KiB/s wr, 17 op/s 2023-03-09T22:28:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:02 smithi181 ceph-mon[94189]: pgmap v14907: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 944 KiB/s rd, 796 KiB/s wr, 17 op/s 2023-03-09T22:28:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:02 smithi038 ceph-mon[92422]: pgmap v14907: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 944 KiB/s rd, 796 KiB/s wr, 17 op/s 2023-03-09T22:28:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27455.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:28:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:04 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26993.741584 secs 2023-03-09T22:28:04.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:04 smithi181 ceph-mon[94189]: pgmap v14908: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 912 KiB/s rd, 792 KiB/s wr, 26 op/s 2023-03-09T22:28:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:04 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26993.741584 secs 2023-03-09T22:28:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:04 smithi038 ceph-mon[92422]: pgmap v14908: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 912 KiB/s rd, 792 KiB/s wr, 26 op/s 2023-03-09T22:28:04.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:04 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26993.741584 secs 2023-03-09T22:28:04.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:04 smithi122 ceph-mon[93869]: pgmap v14908: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 912 KiB/s rd, 792 KiB/s wr, 26 op/s 2023-03-09T22:28:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:05 smithi038 ceph-mon[92422]: osdmap e1147: 12 total, 12 up, 12 in 2023-03-09T22:28:05.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26996.369104 secs 2023-03-09T22:28:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:05 smithi122 ceph-mon[93869]: osdmap e1147: 12 total, 12 up, 12 in 2023-03-09T22:28:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26996.369104 secs 2023-03-09T22:28:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:05 smithi181 ceph-mon[94189]: osdmap e1147: 12 total, 12 up, 12 in 2023-03-09T22:28:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26996.369104 secs 2023-03-09T22:28:06.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:06 smithi122 ceph-mon[93869]: pgmap v14910: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 670 KiB/s wr, 28 op/s 2023-03-09T22:28:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:06 smithi181 ceph-mon[94189]: pgmap v14910: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 670 KiB/s wr, 28 op/s 2023-03-09T22:28:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:06 smithi038 ceph-mon[92422]: pgmap v14910: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 670 KiB/s wr, 28 op/s 2023-03-09T22:28:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:08 smithi122 ceph-mon[93869]: pgmap v14911: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 569 KiB/s wr, 29 op/s 2023-03-09T22:28:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:28:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:28:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 26998.741705 secs 2023-03-09T22:28:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:08 smithi181 ceph-mon[94189]: pgmap v14911: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 569 KiB/s wr, 29 op/s 2023-03-09T22:28:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:28:08.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:28:08.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 26998.741705 secs 2023-03-09T22:28:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:08 smithi038 ceph-mon[92422]: pgmap v14911: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 569 KiB/s wr, 29 op/s 2023-03-09T22:28:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:28:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:28:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 26998.741705 secs 2023-03-09T22:28:08.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27460.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:28:09.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:28:09.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:28:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:28:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:28:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:28:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:28:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:10 smithi122 ceph-mon[93869]: pgmap v14912: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 569 KiB/s wr, 29 op/s 2023-03-09T22:28:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27001.369163 secs 2023-03-09T22:28:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:10 smithi181 ceph-mon[94189]: pgmap v14912: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 569 KiB/s wr, 29 op/s 2023-03-09T22:28:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27001.369163 secs 2023-03-09T22:28:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:10 smithi038 ceph-mon[92422]: pgmap v14912: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 569 KiB/s wr, 29 op/s 2023-03-09T22:28:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27001.369163 secs 2023-03-09T22:28:12.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:12 smithi122 ceph-mon[93869]: pgmap v14913: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 265 KiB/s wr, 24 op/s 2023-03-09T22:28:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:12 smithi181 ceph-mon[94189]: pgmap v14913: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 265 KiB/s wr, 24 op/s 2023-03-09T22:28:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:12 smithi038 ceph-mon[92422]: pgmap v14913: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 265 KiB/s wr, 24 op/s 2023-03-09T22:28:13.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27003.741821 secs 2023-03-09T22:28:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27003.741821 secs 2023-03-09T22:28:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27003.741821 secs 2023-03-09T22:28:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27465.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:28:14.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:14 smithi122 ceph-mon[93869]: pgmap v14914: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 372 KiB/s wr, 10 op/s 2023-03-09T22:28:14.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:14 smithi181 ceph-mon[94189]: pgmap v14914: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 372 KiB/s wr, 10 op/s 2023-03-09T22:28:14.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:14 smithi038 ceph-mon[92422]: pgmap v14914: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 372 KiB/s wr, 10 op/s 2023-03-09T22:28:15.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27006.369220 secs 2023-03-09T22:28:15.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:15 smithi122 ceph-mon[93869]: pgmap v14915: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 344 KiB/s wr, 9 op/s 2023-03-09T22:28:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27006.369220 secs 2023-03-09T22:28:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:15 smithi181 ceph-mon[94189]: pgmap v14915: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 344 KiB/s wr, 9 op/s 2023-03-09T22:28:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27006.369220 secs 2023-03-09T22:28:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:15 smithi038 ceph-mon[92422]: pgmap v14915: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 344 KiB/s wr, 9 op/s 2023-03-09T22:28:17.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:17 smithi122 ceph-mon[93869]: osdmap e1148: 12 total, 12 up, 12 in 2023-03-09T22:28:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:17 smithi181 ceph-mon[94189]: osdmap e1148: 12 total, 12 up, 12 in 2023-03-09T22:28:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:17 smithi038 ceph-mon[92422]: osdmap e1148: 12 total, 12 up, 12 in 2023-03-09T22:28:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:18 smithi038 ceph-mon[92422]: pgmap v14917: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 134 KiB/s wr, 7 op/s 2023-03-09T22:28:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27008.741915 secs 2023-03-09T22:28:18.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:18 smithi122 ceph-mon[93869]: pgmap v14917: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 134 KiB/s wr, 7 op/s 2023-03-09T22:28:18.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27008.741915 secs 2023-03-09T22:28:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:18 smithi181 ceph-mon[94189]: pgmap v14917: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 134 KiB/s wr, 7 op/s 2023-03-09T22:28:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27008.741915 secs 2023-03-09T22:28:18.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27470.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:28:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:20 smithi038 ceph-mon[92422]: pgmap v14918: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 133 KiB/s wr, 7 op/s 2023-03-09T22:28:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27011.369235 secs 2023-03-09T22:28:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:20 smithi122 ceph-mon[93869]: pgmap v14918: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 133 KiB/s wr, 7 op/s 2023-03-09T22:28:20.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27011.369235 secs 2023-03-09T22:28:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:20 smithi181 ceph-mon[94189]: pgmap v14918: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 133 KiB/s wr, 7 op/s 2023-03-09T22:28:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27011.369235 secs 2023-03-09T22:28:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:22 smithi038 ceph-mon[92422]: pgmap v14919: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 133 KiB/s wr, 6 op/s 2023-03-09T22:28:22.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:22 smithi122 ceph-mon[93869]: pgmap v14919: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 133 KiB/s wr, 6 op/s 2023-03-09T22:28:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:22 smithi181 ceph-mon[94189]: pgmap v14919: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 133 KiB/s wr, 6 op/s 2023-03-09T22:28:23.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27013.742045 secs 2023-03-09T22:28:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27013.742045 secs 2023-03-09T22:28:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27013.742045 secs 2023-03-09T22:28:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27475.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:28:24.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:24 smithi122 ceph-mon[93869]: pgmap v14920: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-09T22:28:24.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:24 smithi181 ceph-mon[94189]: pgmap v14920: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-09T22:28:24.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:24 smithi038 ceph-mon[92422]: pgmap v14920: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-09T22:28:25.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27016.369296 secs 2023-03-09T22:28:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27016.369296 secs 2023-03-09T22:28:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27016.369296 secs 2023-03-09T22:28:26.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:26 smithi122 ceph-mon[93869]: pgmap v14921: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-09T22:28:26.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:26 smithi181 ceph-mon[94189]: pgmap v14921: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-09T22:28:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:26 smithi038 ceph-mon[92422]: pgmap v14921: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-09T22:28:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:28 smithi122 ceph-mon[93869]: pgmap v14922: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2023-03-09T22:28:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27018.742184 secs 2023-03-09T22:28:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:28 smithi181 ceph-mon[94189]: pgmap v14922: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2023-03-09T22:28:28.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27018.742184 secs 2023-03-09T22:28:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:28 smithi038 ceph-mon[92422]: pgmap v14922: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2023-03-09T22:28:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27018.742184 secs 2023-03-09T22:28:28.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27480.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:28:30.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:30 smithi122 ceph-mon[93869]: pgmap v14923: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T22:28:30.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27021.369358 secs 2023-03-09T22:28:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:30 smithi181 ceph-mon[94189]: pgmap v14923: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T22:28:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27021.369358 secs 2023-03-09T22:28:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:30 smithi038 ceph-mon[92422]: pgmap v14923: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T22:28:30.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27021.369358 secs 2023-03-09T22:28:32.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:32 smithi122 ceph-mon[93869]: pgmap v14924: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 341 B/s wr, 7 op/s 2023-03-09T22:28:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:32 smithi181 ceph-mon[94189]: pgmap v14924: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 341 B/s wr, 7 op/s 2023-03-09T22:28:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:32 smithi038 ceph-mon[92422]: pgmap v14924: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 341 B/s wr, 7 op/s 2023-03-09T22:28:33.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27023.742269 secs 2023-03-09T22:28:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27023.742269 secs 2023-03-09T22:28:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27023.742269 secs 2023-03-09T22:28:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27485.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:28:34.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:34 smithi122 ceph-mon[93869]: pgmap v14925: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 95 KiB/s wr, 6 op/s 2023-03-09T22:28:34.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:34 smithi181 ceph-mon[94189]: pgmap v14925: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 95 KiB/s wr, 6 op/s 2023-03-09T22:28:34.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:34 smithi038 ceph-mon[92422]: pgmap v14925: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 95 KiB/s wr, 6 op/s 2023-03-09T22:28:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27026.369417 secs 2023-03-09T22:28:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:35 smithi122 ceph-mon[93869]: pgmap v14926: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 95 KiB/s wr, 6 op/s 2023-03-09T22:28:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27026.369417 secs 2023-03-09T22:28:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:35 smithi181 ceph-mon[94189]: pgmap v14926: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 95 KiB/s wr, 6 op/s 2023-03-09T22:28:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27026.369417 secs 2023-03-09T22:28:35.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:35 smithi038 ceph-mon[92422]: pgmap v14926: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 95 KiB/s wr, 6 op/s 2023-03-09T22:28:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:38 smithi038 ceph-mon[92422]: pgmap v14927: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 95 KiB/s wr, 9 op/s 2023-03-09T22:28:38.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27028.742370 secs 2023-03-09T22:28:38.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:38 smithi122 ceph-mon[93869]: pgmap v14927: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 95 KiB/s wr, 9 op/s 2023-03-09T22:28:38.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27028.742370 secs 2023-03-09T22:28:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:38 smithi181 ceph-mon[94189]: pgmap v14927: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 95 KiB/s wr, 9 op/s 2023-03-09T22:28:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27028.742370 secs 2023-03-09T22:28:38.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27490.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:28:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:40 smithi122 ceph-mon[93869]: pgmap v14928: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 318 KiB/s wr, 7 op/s 2023-03-09T22:28:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:28:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:28:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27031.369495 secs 2023-03-09T22:28:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:40 smithi181 ceph-mon[94189]: pgmap v14928: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 318 KiB/s wr, 7 op/s 2023-03-09T22:28:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:28:40.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:28:40.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27031.369495 secs 2023-03-09T22:28:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:40 smithi038 ceph-mon[92422]: pgmap v14928: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 318 KiB/s wr, 7 op/s 2023-03-09T22:28:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:28:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:28:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27031.369495 secs 2023-03-09T22:28:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:28:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:28:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:28:41.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:28:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:28:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:28:42.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:42 smithi122 ceph-mon[93869]: pgmap v14929: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 388 KiB/s wr, 10 op/s 2023-03-09T22:28:42.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:42 smithi181 ceph-mon[94189]: pgmap v14929: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 388 KiB/s wr, 10 op/s 2023-03-09T22:28:42.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:42 smithi038 ceph-mon[92422]: pgmap v14929: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 388 KiB/s wr, 10 op/s 2023-03-09T22:28:43.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27033.742471 secs 2023-03-09T22:28:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27033.742471 secs 2023-03-09T22:28:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27033.742471 secs 2023-03-09T22:28:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27495.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:28:44.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:44 smithi122 ceph-mon[93869]: pgmap v14930: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 537 KiB/s rd, 445 KiB/s wr, 9 op/s 2023-03-09T22:28:44.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:44 smithi181 ceph-mon[94189]: pgmap v14930: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 537 KiB/s rd, 445 KiB/s wr, 9 op/s 2023-03-09T22:28:44.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:44 smithi038 ceph-mon[92422]: pgmap v14930: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 537 KiB/s rd, 445 KiB/s wr, 9 op/s 2023-03-09T22:28:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27036.369571 secs 2023-03-09T22:28:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27036.369571 secs 2023-03-09T22:28:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27036.369571 secs 2023-03-09T22:28:46.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:46 smithi122 ceph-mon[93869]: pgmap v14931: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 351 KiB/s wr, 8 op/s 2023-03-09T22:28:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:46 smithi181 ceph-mon[94189]: pgmap v14931: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 351 KiB/s wr, 8 op/s 2023-03-09T22:28:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:46 smithi038 ceph-mon[92422]: pgmap v14931: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 351 KiB/s wr, 8 op/s 2023-03-09T22:28:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:48 smithi122 ceph-mon[93869]: pgmap v14932: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 559 KiB/s rd, 500 KiB/s wr, 12 op/s 2023-03-09T22:28:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:28:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:28:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27038.742564 secs 2023-03-09T22:28:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:48 smithi181 ceph-mon[94189]: pgmap v14932: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 559 KiB/s rd, 500 KiB/s wr, 12 op/s 2023-03-09T22:28:48.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:28:48.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:28:48.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27038.742564 secs 2023-03-09T22:28:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:48 smithi038 ceph-mon[92422]: pgmap v14932: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 559 KiB/s rd, 500 KiB/s wr, 12 op/s 2023-03-09T22:28:48.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:28:48.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:28:48.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27038.742564 secs 2023-03-09T22:28:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27500.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:28:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:50 smithi122 ceph-mon[93869]: pgmap v14933: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 558 KiB/s rd, 499 KiB/s wr, 9 op/s 2023-03-09T22:28:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27041.369648 secs 2023-03-09T22:28:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:50 smithi181 ceph-mon[94189]: pgmap v14933: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 558 KiB/s rd, 499 KiB/s wr, 9 op/s 2023-03-09T22:28:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27041.369648 secs 2023-03-09T22:28:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:50 smithi038 ceph-mon[92422]: pgmap v14933: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 558 KiB/s rd, 499 KiB/s wr, 9 op/s 2023-03-09T22:28:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27041.369648 secs 2023-03-09T22:28:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:52 smithi122 ceph-mon[93869]: pgmap v14934: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 517 KiB/s rd, 444 KiB/s wr, 13 op/s 2023-03-09T22:28:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:52 smithi181 ceph-mon[94189]: pgmap v14934: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 517 KiB/s rd, 444 KiB/s wr, 13 op/s 2023-03-09T22:28:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:52 smithi038 ceph-mon[92422]: pgmap v14934: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 517 KiB/s rd, 444 KiB/s wr, 13 op/s 2023-03-09T22:28:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:53 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:28:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:53 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:28:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:53 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:28:53.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27505.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:28:54.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:54 smithi122 ceph-mon[93869]: pgmap v14935: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 446 KiB/s rd, 374 KiB/s wr, 10 op/s 2023-03-09T22:28:54.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:54 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:28:54.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:54 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:28:54.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:54 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:28:54.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:54 smithi181 ceph-mon[94189]: pgmap v14935: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 446 KiB/s rd, 374 KiB/s wr, 10 op/s 2023-03-09T22:28:54.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:54 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:28:54.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:54 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:28:54.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:54 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:28:54.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:54 smithi038 ceph-mon[92422]: pgmap v14935: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 446 KiB/s rd, 374 KiB/s wr, 10 op/s 2023-03-09T22:28:54.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:54 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:28:54.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:54 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:28:54.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:54 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:28:55.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27046.369708 secs 2023-03-09T22:28:55.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:55 smithi122 ceph-mon[93869]: pgmap v14936: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 317 KiB/s wr, 9 op/s 2023-03-09T22:28:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27046.369708 secs 2023-03-09T22:28:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:55 smithi181 ceph-mon[94189]: pgmap v14936: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 317 KiB/s wr, 9 op/s 2023-03-09T22:28:55.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27046.369708 secs 2023-03-09T22:28:55.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:55 smithi038 ceph-mon[92422]: pgmap v14936: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 317 KiB/s wr, 9 op/s 2023-03-09T22:28:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:28:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:28:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:28:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:28:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:28:56.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:28:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:28:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:28:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:57 smithi122 ceph-mon[93869]: pgmap v14937: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 749 KiB/s wr, 14 op/s 2023-03-09T22:28:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:28:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:28:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:57 smithi181 ceph-mon[94189]: pgmap v14937: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 749 KiB/s wr, 14 op/s 2023-03-09T22:28:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:28:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:28:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:57 smithi038 ceph-mon[92422]: pgmap v14937: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 749 KiB/s wr, 14 op/s 2023-03-09T22:28:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27510.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:28:59.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:28:59.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:28:59.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:28:59.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:28:59 smithi122 ceph-mon[93869]: pgmap v14938: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 902 KiB/s rd, 601 KiB/s wr, 11 op/s 2023-03-09T22:28:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:28:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:28:59.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:28:59.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:28:59 smithi181 ceph-mon[94189]: pgmap v14938: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 902 KiB/s rd, 601 KiB/s wr, 11 op/s 2023-03-09T22:29:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:29:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:29:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:29:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:28:59 smithi038 ceph-mon[92422]: pgmap v14938: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 902 KiB/s rd, 601 KiB/s wr, 11 op/s 2023-03-09T22:29:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27051.369770 secs 2023-03-09T22:29:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27051.369770 secs 2023-03-09T22:29:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27051.369770 secs 2023-03-09T22:29:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:29:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:29:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:01 smithi122 ceph-mon[93869]: pgmap v14939: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 925 KiB/s rd, 635 KiB/s wr, 15 op/s 2023-03-09T22:29:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:29:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:29:01.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:01 smithi181 ceph-mon[94189]: pgmap v14939: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 925 KiB/s rd, 635 KiB/s wr, 15 op/s 2023-03-09T22:29:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:29:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:29:02.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:01 smithi038 ceph-mon[92422]: pgmap v14939: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 925 KiB/s rd, 635 KiB/s wr, 15 op/s 2023-03-09T22:29:02.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:29:02.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:29:02.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:29:02.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:29:02.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:29:02.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:29:02.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:29:02.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:29:02.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:29:02.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:29:02.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:02 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T22:29:02.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:02 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T22:29:02.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:29:02.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:29:02.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:29:02.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:29:02.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:02 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi038 to 8303k 2023-03-09T22:29:02.980 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:02 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi038 to 8503279: error parsing value: Value '8503279' is below minimum 939524096 2023-03-09T22:29:02.980 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:29:02.980 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:29:02.980 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:29:02.980 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:29:02.980 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:29:02.980 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:29:02.981 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:29:02.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:29:02.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:29:02.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:29:02.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:29:02.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:29:02.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:29:02.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:29:02.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:29:02.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:29:02.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:29:02.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:02 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T22:29:02.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:02 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T22:29:02.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:29:02.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:29:02.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:29:02.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:29:02.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:02 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi038 to 8303k 2023-03-09T22:29:02.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:02 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi038 to 8503279: error parsing value: Value '8503279' is below minimum 939524096 2023-03-09T22:29:02.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:29:02.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:29:02.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:29:02.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:29:02.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:29:02.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:29:02.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:29:02.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:29:03.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:29:03.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:29:03.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:29:03.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:29:03.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:29:03.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:29:03.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:29:03.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:29:03.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:29:03.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:02 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T22:29:03.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:02 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T22:29:03.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:29:03.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:29:03.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:29:03.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:29:03.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:02 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi038 to 8303k 2023-03-09T22:29:03.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:02 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi038 to 8503279: error parsing value: Value '8503279' is below minimum 939524096 2023-03-09T22:29:03.004 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:29:03.004 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:29:03.004 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:29:03.004 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:29:03.005 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:29:03.005 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:29:03.005 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:29:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27515.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:29:03.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27053.742941 secs 2023-03-09T22:29:03.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:03 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T22:29:03.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:03 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T22:29:03.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:03 smithi122 ceph-mon[93869]: pgmap v14940: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 879 KiB/s rd, 617 KiB/s wr, 24 op/s 2023-03-09T22:29:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27053.742941 secs 2023-03-09T22:29:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:03 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T22:29:03.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:03 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T22:29:03.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:03 smithi181 ceph-mon[94189]: pgmap v14940: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 879 KiB/s rd, 617 KiB/s wr, 24 op/s 2023-03-09T22:29:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27053.742941 secs 2023-03-09T22:29:04.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:03 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T22:29:04.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:03 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T22:29:04.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:03 smithi038 ceph-mon[92422]: pgmap v14940: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 879 KiB/s rd, 617 KiB/s wr, 24 op/s 2023-03-09T22:29:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:05 smithi122 ceph-mon[93869]: osdmap e1149: 12 total, 12 up, 12 in 2023-03-09T22:29:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27056.369827 secs 2023-03-09T22:29:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:05 smithi122 ceph-mon[93869]: pgmap v14942: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 740 KiB/s wr, 28 op/s 2023-03-09T22:29:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:05 smithi181 ceph-mon[94189]: osdmap e1149: 12 total, 12 up, 12 in 2023-03-09T22:29:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27056.369827 secs 2023-03-09T22:29:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:05 smithi181 ceph-mon[94189]: pgmap v14942: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 740 KiB/s wr, 28 op/s 2023-03-09T22:29:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:05 smithi038 ceph-mon[92422]: osdmap e1149: 12 total, 12 up, 12 in 2023-03-09T22:29:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27056.369827 secs 2023-03-09T22:29:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:05 smithi038 ceph-mon[92422]: pgmap v14942: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 740 KiB/s wr, 28 op/s 2023-03-09T22:29:06.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:06 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:29:06.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:06 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:29:06.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:29:06.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:06 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:29:06.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:06 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:29:06.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:29:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:06 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:29:07.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:06 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:29:07.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:29:07.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:07 smithi122 ceph-mon[93869]: pgmap v14943: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 439 KiB/s wr, 28 op/s 2023-03-09T22:29:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:07 smithi181 ceph-mon[94189]: pgmap v14943: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 439 KiB/s wr, 28 op/s 2023-03-09T22:29:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:07 smithi038 ceph-mon[92422]: pgmap v14943: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 439 KiB/s wr, 28 op/s 2023-03-09T22:29:08.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27520.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:29:08.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27058.743064 secs 2023-03-09T22:29:08.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27058.743064 secs 2023-03-09T22:29:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27058.743064 secs 2023-03-09T22:29:09.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:09 smithi122 ceph-mon[93869]: pgmap v14944: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 530 KiB/s rd, 571 KiB/s wr, 29 op/s 2023-03-09T22:29:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:09 smithi181 ceph-mon[94189]: pgmap v14944: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 530 KiB/s rd, 571 KiB/s wr, 29 op/s 2023-03-09T22:29:10.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:09 smithi038 ceph-mon[92422]: pgmap v14944: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 530 KiB/s rd, 571 KiB/s wr, 29 op/s 2023-03-09T22:29:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27061.369887 secs 2023-03-09T22:29:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27061.369887 secs 2023-03-09T22:29:11.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27061.369887 secs 2023-03-09T22:29:11.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:11 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:29:11.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:11 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:29:11.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:11 smithi122 ceph-mon[93869]: pgmap v14945: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 503 KiB/s rd, 531 KiB/s wr, 26 op/s 2023-03-09T22:29:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:11 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:29:11.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:11 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:29:11.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:11 smithi181 ceph-mon[94189]: pgmap v14945: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 503 KiB/s rd, 531 KiB/s wr, 26 op/s 2023-03-09T22:29:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:11 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:29:12.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:11 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:29:12.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:11 smithi038 ceph-mon[92422]: pgmap v14945: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 503 KiB/s rd, 531 KiB/s wr, 26 op/s 2023-03-09T22:29:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27525.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:29:13.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27063.743162 secs 2023-03-09T22:29:13.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:29:13.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:29:13.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:13 smithi122 ceph-mon[93869]: pgmap v14946: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 351 KiB/s wr, 10 op/s 2023-03-09T22:29:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27063.743162 secs 2023-03-09T22:29:13.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:29:13.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:29:13.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:13 smithi181 ceph-mon[94189]: pgmap v14946: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 351 KiB/s wr, 10 op/s 2023-03-09T22:29:14.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27063.743162 secs 2023-03-09T22:29:14.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:29:14.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:29:14.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:13 smithi038 ceph-mon[92422]: pgmap v14946: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 351 KiB/s wr, 10 op/s 2023-03-09T22:29:15.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27066.369950 secs 2023-03-09T22:29:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27066.369950 secs 2023-03-09T22:29:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27066.369950 secs 2023-03-09T22:29:16.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:16 smithi122 ceph-mon[93869]: pgmap v14947: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 424 KiB/s wr, 11 op/s 2023-03-09T22:29:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:16 smithi181 ceph-mon[94189]: pgmap v14947: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 424 KiB/s wr, 11 op/s 2023-03-09T22:29:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:16 smithi038 ceph-mon[92422]: pgmap v14947: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 424 KiB/s wr, 11 op/s 2023-03-09T22:29:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:17 smithi038 ceph-mon[92422]: osdmap e1150: 12 total, 12 up, 12 in 2023-03-09T22:29:17.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:17 smithi122 ceph-mon[93869]: osdmap e1150: 12 total, 12 up, 12 in 2023-03-09T22:29:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:17 smithi181 ceph-mon[94189]: osdmap e1150: 12 total, 12 up, 12 in 2023-03-09T22:29:18.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:18 smithi122 ceph-mon[93869]: pgmap v14949: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 227 KiB/s wr, 8 op/s 2023-03-09T22:29:18.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27068.743284 secs 2023-03-09T22:29:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:18 smithi181 ceph-mon[94189]: pgmap v14949: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 227 KiB/s wr, 8 op/s 2023-03-09T22:29:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27068.743284 secs 2023-03-09T22:29:18.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:18 smithi038 ceph-mon[92422]: pgmap v14949: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 227 KiB/s wr, 8 op/s 2023-03-09T22:29:18.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27068.743284 secs 2023-03-09T22:29:18.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27530.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:29:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:20 smithi122 ceph-mon[93869]: pgmap v14950: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 95 KiB/s wr, 7 op/s 2023-03-09T22:29:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27071.369987 secs 2023-03-09T22:29:20.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:20 smithi181 ceph-mon[94189]: pgmap v14950: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 95 KiB/s wr, 7 op/s 2023-03-09T22:29:20.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27071.369987 secs 2023-03-09T22:29:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:20 smithi038 ceph-mon[92422]: pgmap v14950: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 95 KiB/s wr, 7 op/s 2023-03-09T22:29:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27071.369987 secs 2023-03-09T22:29:22.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:22 smithi122 ceph-mon[93869]: pgmap v14951: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 94 KiB/s wr, 7 op/s 2023-03-09T22:29:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:22 smithi181 ceph-mon[94189]: pgmap v14951: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 94 KiB/s wr, 7 op/s 2023-03-09T22:29:22.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:22 smithi038 ceph-mon[92422]: pgmap v14951: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 94 KiB/s wr, 7 op/s 2023-03-09T22:29:23.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27073.743408 secs 2023-03-09T22:29:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27073.743408 secs 2023-03-09T22:29:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27073.743408 secs 2023-03-09T22:29:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27535.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:29:24.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:24 smithi122 ceph-mon[93869]: pgmap v14952: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 94 KiB/s wr, 7 op/s 2023-03-09T22:29:24.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:24 smithi181 ceph-mon[94189]: pgmap v14952: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 94 KiB/s wr, 7 op/s 2023-03-09T22:29:24.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:24 smithi038 ceph-mon[92422]: pgmap v14952: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 94 KiB/s wr, 7 op/s 2023-03-09T22:29:25.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27076.370047 secs 2023-03-09T22:29:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27076.370047 secs 2023-03-09T22:29:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27076.370047 secs 2023-03-09T22:29:26.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:26 smithi122 ceph-mon[93869]: pgmap v14953: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-03-09T22:29:26.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:26 smithi181 ceph-mon[94189]: pgmap v14953: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-03-09T22:29:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:26 smithi038 ceph-mon[92422]: pgmap v14953: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-03-09T22:29:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:28 smithi122 ceph-mon[93869]: pgmap v14954: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2023-03-09T22:29:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27078.743519 secs 2023-03-09T22:29:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:28 smithi181 ceph-mon[94189]: pgmap v14954: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2023-03-09T22:29:28.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27078.743519 secs 2023-03-09T22:29:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:28 smithi038 ceph-mon[92422]: pgmap v14954: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2023-03-09T22:29:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27078.743519 secs 2023-03-09T22:29:28.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27540.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:29:30.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:30 smithi122 ceph-mon[93869]: pgmap v14955: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T22:29:30.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27081.370109 secs 2023-03-09T22:29:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:30 smithi181 ceph-mon[94189]: pgmap v14955: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T22:29:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27081.370109 secs 2023-03-09T22:29:30.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:30 smithi038 ceph-mon[92422]: pgmap v14955: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T22:29:30.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27081.370109 secs 2023-03-09T22:29:32.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:32 smithi122 ceph-mon[93869]: pgmap v14956: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T22:29:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:32 smithi181 ceph-mon[94189]: pgmap v14956: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T22:29:32.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:32 smithi038 ceph-mon[92422]: pgmap v14956: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T22:29:33.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27083.743657 secs 2023-03-09T22:29:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27083.743657 secs 2023-03-09T22:29:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27083.743657 secs 2023-03-09T22:29:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27545.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:29:34.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:34 smithi122 ceph-mon[93869]: pgmap v14957: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T22:29:34.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:34 smithi181 ceph-mon[94189]: pgmap v14957: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T22:29:34.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:34 smithi038 ceph-mon[92422]: pgmap v14957: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T22:29:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27086.370171 secs 2023-03-09T22:29:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:35 smithi122 ceph-mon[93869]: pgmap v14958: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T22:29:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27086.370171 secs 2023-03-09T22:29:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:35 smithi181 ceph-mon[94189]: pgmap v14958: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T22:29:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27086.370171 secs 2023-03-09T22:29:35.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:35 smithi038 ceph-mon[92422]: pgmap v14958: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T22:29:38.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:38 smithi122 ceph-mon[93869]: pgmap v14959: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 8 op/s 2023-03-09T22:29:38.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27088.743781 secs 2023-03-09T22:29:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:38 smithi181 ceph-mon[94189]: pgmap v14959: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 8 op/s 2023-03-09T22:29:38.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27088.743781 secs 2023-03-09T22:29:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:38 smithi038 ceph-mon[92422]: pgmap v14959: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 8 op/s 2023-03-09T22:29:38.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27088.743781 secs 2023-03-09T22:29:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27550.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:29:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:40 smithi122 ceph-mon[93869]: pgmap v14960: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 153 KiB/s wr, 7 op/s 2023-03-09T22:29:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27091.370255 secs 2023-03-09T22:29:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:40 smithi181 ceph-mon[94189]: pgmap v14960: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 153 KiB/s wr, 7 op/s 2023-03-09T22:29:40.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27091.370255 secs 2023-03-09T22:29:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:40 smithi038 ceph-mon[92422]: pgmap v14960: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 153 KiB/s wr, 7 op/s 2023-03-09T22:29:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27091.370255 secs 2023-03-09T22:29:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:29:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:29:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:29:41.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:29:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:29:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:29:42.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:42 smithi122 ceph-mon[93869]: pgmap v14961: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 242 KiB/s wr, 9 op/s 2023-03-09T22:29:42.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:42 smithi181 ceph-mon[94189]: pgmap v14961: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 242 KiB/s wr, 9 op/s 2023-03-09T22:29:42.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:42 smithi038 ceph-mon[92422]: pgmap v14961: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 242 KiB/s wr, 9 op/s 2023-03-09T22:29:43.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27555.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:29:44.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:44 smithi122 ceph-mon[93869]: pgmap v14962: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 242 KiB/s wr, 7 op/s 2023-03-09T22:29:44.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:44 smithi181 ceph-mon[94189]: pgmap v14962: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 242 KiB/s wr, 7 op/s 2023-03-09T22:29:44.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:44 smithi038 ceph-mon[92422]: pgmap v14962: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 242 KiB/s wr, 7 op/s 2023-03-09T22:29:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27096.370315 secs 2023-03-09T22:29:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:45 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:29:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27096.370315 secs 2023-03-09T22:29:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:45 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:29:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27096.370315 secs 2023-03-09T22:29:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:45 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:29:46.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:46 smithi122 ceph-mon[93869]: pgmap v14963: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 283 KiB/s rd, 242 KiB/s wr, 7 op/s 2023-03-09T22:29:46.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:46 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:29:46.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:46 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:29:46.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:46 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:29:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:46 smithi181 ceph-mon[94189]: pgmap v14963: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 283 KiB/s rd, 242 KiB/s wr, 7 op/s 2023-03-09T22:29:46.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:46 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:29:46.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:46 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:29:46.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:46 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:29:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:46 smithi038 ceph-mon[92422]: pgmap v14963: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 283 KiB/s rd, 242 KiB/s wr, 7 op/s 2023-03-09T22:29:46.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:46 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:29:46.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:46 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:29:46.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:46 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:29:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:48 smithi122 ceph-mon[93869]: pgmap v14964: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 576 KiB/s rd, 535 KiB/s wr, 11 op/s 2023-03-09T22:29:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:29:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:29:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:48 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 27098.743955 secs 2023-03-09T22:29:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:48 smithi181 ceph-mon[94189]: pgmap v14964: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 576 KiB/s rd, 535 KiB/s wr, 11 op/s 2023-03-09T22:29:48.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:29:48.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:29:48.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:48 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 27098.743955 secs 2023-03-09T22:29:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:48 smithi038 ceph-mon[92422]: pgmap v14964: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 576 KiB/s rd, 535 KiB/s wr, 11 op/s 2023-03-09T22:29:48.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:29:48.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:29:48.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:48 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 27098.743955 secs 2023-03-09T22:29:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27560.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:29:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:50 smithi122 ceph-mon[93869]: pgmap v14965: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 574 KiB/s rd, 534 KiB/s wr, 8 op/s 2023-03-09T22:29:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:50 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:29:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:50 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:29:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27101.370377 secs 2023-03-09T22:29:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:50 smithi181 ceph-mon[94189]: pgmap v14965: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 574 KiB/s rd, 534 KiB/s wr, 8 op/s 2023-03-09T22:29:50.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:50 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:29:50.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:50 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:29:50.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27101.370377 secs 2023-03-09T22:29:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:50 smithi038 ceph-mon[92422]: pgmap v14965: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 574 KiB/s rd, 534 KiB/s wr, 8 op/s 2023-03-09T22:29:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:50 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:29:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:50 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:29:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27101.370377 secs 2023-03-09T22:29:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:51 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:29:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:51 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:29:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:51 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:29:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:52 smithi122 ceph-mon[93869]: pgmap v14966: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 693 KiB/s wr, 12 op/s 2023-03-09T22:29:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:29:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:29:52.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:52 smithi181 ceph-mon[94189]: pgmap v14966: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 693 KiB/s wr, 12 op/s 2023-03-09T22:29:52.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:29:52.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:29:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:52 smithi038 ceph-mon[92422]: pgmap v14966: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 693 KiB/s wr, 12 op/s 2023-03-09T22:29:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:29:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:29:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27103.744083 secs 2023-03-09T22:29:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:29:53.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:29:53.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:53 smithi122 ceph-mon[93869]: pgmap v14967: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 695 KiB/s rd, 605 KiB/s wr, 11 op/s 2023-03-09T22:29:53.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27103.744083 secs 2023-03-09T22:29:53.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:29:53.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:29:53.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:53 smithi181 ceph-mon[94189]: pgmap v14967: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 695 KiB/s rd, 605 KiB/s wr, 11 op/s 2023-03-09T22:29:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27103.744083 secs 2023-03-09T22:29:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:29:53.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:29:53.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:53 smithi038 ceph-mon[92422]: pgmap v14967: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 695 KiB/s rd, 605 KiB/s wr, 11 op/s 2023-03-09T22:29:53.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27565.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:29:54.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:54 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:29:54.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:54 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:29:54.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:54 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:29:54.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:54 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:29:54.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:54 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:29:54.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:54 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:29:55.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27106.370435 secs 2023-03-09T22:29:55.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:55 smithi122 ceph-mon[93869]: pgmap v14968: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 824 KiB/s rd, 734 KiB/s wr, 11 op/s 2023-03-09T22:29:55.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27106.370435 secs 2023-03-09T22:29:55.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:55 smithi181 ceph-mon[94189]: pgmap v14968: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 824 KiB/s rd, 734 KiB/s wr, 11 op/s 2023-03-09T22:29:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27106.370435 secs 2023-03-09T22:29:55.753 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:55 smithi038 ceph-mon[92422]: pgmap v14968: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 824 KiB/s rd, 734 KiB/s wr, 11 op/s 2023-03-09T22:29:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:29:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:29:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:29:56.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:29:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:29:56.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:29:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:29:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:29:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:57 smithi122 ceph-mon[93869]: pgmap v14969: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 809 KiB/s rd, 795 KiB/s wr, 15 op/s 2023-03-09T22:29:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:29:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:29:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:57 smithi181 ceph-mon[94189]: pgmap v14969: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 809 KiB/s rd, 795 KiB/s wr, 15 op/s 2023-03-09T22:29:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:29:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:29:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:57 smithi038 ceph-mon[92422]: pgmap v14969: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 809 KiB/s rd, 795 KiB/s wr, 15 op/s 2023-03-09T22:29:58.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27108.744223 secs 2023-03-09T22:29:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27108.744223 secs 2023-03-09T22:29:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27108.744223 secs 2023-03-09T22:29:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27570.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:29:59.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:29:59 smithi122 ceph-mon[93869]: pgmap v14970: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 610 KiB/s rd, 597 KiB/s wr, 13 op/s 2023-03-09T22:29:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:29:59 smithi181 ceph-mon[94189]: pgmap v14970: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 610 KiB/s rd, 597 KiB/s wr, 13 op/s 2023-03-09T22:29:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:29:59 smithi038 ceph-mon[92422]: pgmap v14970: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 610 KiB/s rd, 597 KiB/s wr, 13 op/s 2023-03-09T22:30:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27111.370497 secs 2023-03-09T22:30:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:00 smithi122 ceph-mon[93869]: overall HEALTH_WARN 2 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-09T22:30:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27111.370497 secs 2023-03-09T22:30:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:00 smithi181 ceph-mon[94189]: overall HEALTH_WARN 2 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-09T22:30:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27111.370497 secs 2023-03-09T22:30:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:00 smithi038 ceph-mon[92422]: overall HEALTH_WARN 2 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-09T22:30:02.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:02 smithi122 ceph-mon[93869]: pgmap v14971: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 724 KiB/s rd, 710 KiB/s wr, 16 op/s 2023-03-09T22:30:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:02 smithi181 ceph-mon[94189]: pgmap v14971: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 724 KiB/s rd, 710 KiB/s wr, 16 op/s 2023-03-09T22:30:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:02 smithi038 ceph-mon[92422]: pgmap v14971: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 724 KiB/s rd, 710 KiB/s wr, 16 op/s 2023-03-09T22:30:03.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27113.744317 secs 2023-03-09T22:30:03.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:03 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:30:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27113.744317 secs 2023-03-09T22:30:03.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:03 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:30:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27113.744317 secs 2023-03-09T22:30:03.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:03 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:30:03.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27575.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:30:04.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:04 smithi122 ceph-mon[93869]: pgmap v14972: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 406 KiB/s rd, 419 KiB/s wr, 23 op/s 2023-03-09T22:30:04.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:30:04.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:30:04.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:30:04.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:04 smithi181 ceph-mon[94189]: pgmap v14972: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 406 KiB/s rd, 419 KiB/s wr, 23 op/s 2023-03-09T22:30:04.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:30:04.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:30:04.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:30:04.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:04 smithi038 ceph-mon[92422]: pgmap v14972: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 406 KiB/s rd, 419 KiB/s wr, 23 op/s 2023-03-09T22:30:04.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:30:04.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:30:04.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:30:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:05 smithi122 ceph-mon[93869]: osdmap e1151: 12 total, 12 up, 12 in 2023-03-09T22:30:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27116.370582 secs 2023-03-09T22:30:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:05 smithi181 ceph-mon[94189]: osdmap e1151: 12 total, 12 up, 12 in 2023-03-09T22:30:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27116.370582 secs 2023-03-09T22:30:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:05 smithi038 ceph-mon[92422]: osdmap e1151: 12 total, 12 up, 12 in 2023-03-09T22:30:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27116.370582 secs 2023-03-09T22:30:06.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:06 smithi122 ceph-mon[93869]: pgmap v14974: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 490 KiB/s wr, 27 op/s 2023-03-09T22:30:06.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:06 smithi181 ceph-mon[94189]: pgmap v14974: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 490 KiB/s wr, 27 op/s 2023-03-09T22:30:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:06 smithi038 ceph-mon[92422]: pgmap v14974: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 490 KiB/s wr, 27 op/s 2023-03-09T22:30:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:08 smithi122 ceph-mon[93869]: pgmap v14975: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 644 KiB/s rd, 685 KiB/s wr, 30 op/s 2023-03-09T22:30:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:30:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:30:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27118.744459 secs 2023-03-09T22:30:08.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:08 smithi181 ceph-mon[94189]: pgmap v14975: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 644 KiB/s rd, 685 KiB/s wr, 30 op/s 2023-03-09T22:30:08.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:30:08.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:30:08.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27118.744459 secs 2023-03-09T22:30:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:08 smithi038 ceph-mon[92422]: pgmap v14975: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 644 KiB/s rd, 685 KiB/s wr, 30 op/s 2023-03-09T22:30:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:30:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:30:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27118.744459 secs 2023-03-09T22:30:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27580.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:30:09.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:30:09.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:30:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:30:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:30:09.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:30:09.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:30:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:10 smithi122 ceph-mon[93869]: pgmap v14976: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 531 KiB/s rd, 571 KiB/s wr, 29 op/s 2023-03-09T22:30:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27121.370645 secs 2023-03-09T22:30:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:10 smithi181 ceph-mon[94189]: pgmap v14976: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 531 KiB/s rd, 571 KiB/s wr, 29 op/s 2023-03-09T22:30:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27121.370645 secs 2023-03-09T22:30:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:10 smithi038 ceph-mon[92422]: pgmap v14976: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 531 KiB/s rd, 571 KiB/s wr, 29 op/s 2023-03-09T22:30:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27121.370645 secs 2023-03-09T22:30:12.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:12 smithi122 ceph-mon[93869]: pgmap v14977: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 520 KiB/s wr, 29 op/s 2023-03-09T22:30:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:12 smithi181 ceph-mon[94189]: pgmap v14977: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 520 KiB/s wr, 29 op/s 2023-03-09T22:30:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:12 smithi038 ceph-mon[92422]: pgmap v14977: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 520 KiB/s wr, 29 op/s 2023-03-09T22:30:13.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27123.744527 secs 2023-03-09T22:30:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27123.744527 secs 2023-03-09T22:30:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27123.744527 secs 2023-03-09T22:30:13.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27585.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:30:14.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:14 smithi122 ceph-mon[93869]: pgmap v14978: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 470 KiB/s rd, 494 KiB/s wr, 14 op/s 2023-03-09T22:30:14.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:14 smithi181 ceph-mon[94189]: pgmap v14978: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 470 KiB/s rd, 494 KiB/s wr, 14 op/s 2023-03-09T22:30:14.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:14 smithi038 ceph-mon[92422]: pgmap v14978: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 470 KiB/s rd, 494 KiB/s wr, 14 op/s 2023-03-09T22:30:15.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27126.370707 secs 2023-03-09T22:30:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27126.370707 secs 2023-03-09T22:30:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27126.370707 secs 2023-03-09T22:30:16.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:16 smithi122 ceph-mon[93869]: pgmap v14979: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 371 KiB/s wr, 12 op/s 2023-03-09T22:30:16.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:16 smithi122 ceph-mon[93869]: osdmap e1152: 12 total, 12 up, 12 in 2023-03-09T22:30:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:16 smithi181 ceph-mon[94189]: pgmap v14979: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 371 KiB/s wr, 12 op/s 2023-03-09T22:30:16.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:16 smithi181 ceph-mon[94189]: osdmap e1152: 12 total, 12 up, 12 in 2023-03-09T22:30:16.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:16 smithi038 ceph-mon[92422]: pgmap v14979: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 371 KiB/s wr, 12 op/s 2023-03-09T22:30:16.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:16 smithi038 ceph-mon[92422]: osdmap e1152: 12 total, 12 up, 12 in 2023-03-09T22:30:17.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:17 smithi122 ceph-mon[93869]: pgmap v14981: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 134 KiB/s wr, 8 op/s 2023-03-09T22:30:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:17 smithi181 ceph-mon[94189]: pgmap v14981: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 134 KiB/s wr, 8 op/s 2023-03-09T22:30:17.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:17 smithi038 ceph-mon[92422]: pgmap v14981: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 134 KiB/s wr, 8 op/s 2023-03-09T22:30:18.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27128.744652 secs 2023-03-09T22:30:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27128.744652 secs 2023-03-09T22:30:18.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27128.744652 secs 2023-03-09T22:30:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27590.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:30:19.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:19 smithi122 ceph-mon[93869]: pgmap v14982: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 134 KiB/s wr, 8 op/s 2023-03-09T22:30:19.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:19 smithi181 ceph-mon[94189]: pgmap v14982: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 134 KiB/s wr, 8 op/s 2023-03-09T22:30:19.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:19 smithi038 ceph-mon[92422]: pgmap v14982: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 134 KiB/s wr, 8 op/s 2023-03-09T22:30:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27131.370771 secs 2023-03-09T22:30:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27131.370771 secs 2023-03-09T22:30:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27131.370771 secs 2023-03-09T22:30:22.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:22 smithi122 ceph-mon[93869]: pgmap v14983: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 50 KiB/s wr, 7 op/s 2023-03-09T22:30:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:22 smithi181 ceph-mon[94189]: pgmap v14983: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 50 KiB/s wr, 7 op/s 2023-03-09T22:30:22.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:22 smithi038 ceph-mon[92422]: pgmap v14983: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 50 KiB/s wr, 7 op/s 2023-03-09T22:30:23.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27133.744776 secs 2023-03-09T22:30:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27133.744776 secs 2023-03-09T22:30:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27133.744776 secs 2023-03-09T22:30:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27595.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:30:24.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:24 smithi122 ceph-mon[93869]: pgmap v14984: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 50 KiB/s wr, 7 op/s 2023-03-09T22:30:24.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:24 smithi181 ceph-mon[94189]: pgmap v14984: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 50 KiB/s wr, 7 op/s 2023-03-09T22:30:24.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:24 smithi038 ceph-mon[92422]: pgmap v14984: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 50 KiB/s wr, 7 op/s 2023-03-09T22:30:25.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27136.370831 secs 2023-03-09T22:30:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27136.370831 secs 2023-03-09T22:30:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27136.370831 secs 2023-03-09T22:30:26.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:26 smithi122 ceph-mon[93869]: pgmap v14985: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-03-09T22:30:26.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:26 smithi181 ceph-mon[94189]: pgmap v14985: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-03-09T22:30:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:26 smithi038 ceph-mon[92422]: pgmap v14985: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-03-09T22:30:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:28 smithi122 ceph-mon[93869]: pgmap v14986: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2023-03-09T22:30:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27138.744914 secs 2023-03-09T22:30:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:28 smithi181 ceph-mon[94189]: pgmap v14986: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2023-03-09T22:30:28.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27138.744914 secs 2023-03-09T22:30:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:28 smithi038 ceph-mon[92422]: pgmap v14986: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2023-03-09T22:30:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27138.744914 secs 2023-03-09T22:30:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27600.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:30:30.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:30 smithi122 ceph-mon[93869]: pgmap v14987: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T22:30:30.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27141.370905 secs 2023-03-09T22:30:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:30 smithi181 ceph-mon[94189]: pgmap v14987: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T22:30:30.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27141.370905 secs 2023-03-09T22:30:30.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:30 smithi038 ceph-mon[92422]: pgmap v14987: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T22:30:30.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27141.370905 secs 2023-03-09T22:30:32.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:32 smithi122 ceph-mon[93869]: pgmap v14988: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T22:30:32.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:32 smithi181 ceph-mon[94189]: pgmap v14988: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T22:30:32.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:32 smithi038 ceph-mon[92422]: pgmap v14988: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T22:30:33.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27143.745034 secs 2023-03-09T22:30:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27143.745034 secs 2023-03-09T22:30:33.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27143.745034 secs 2023-03-09T22:30:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27605.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:30:34.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:34 smithi122 ceph-mon[93869]: pgmap v14989: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T22:30:34.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:34 smithi181 ceph-mon[94189]: pgmap v14989: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T22:30:34.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:34 smithi038 ceph-mon[92422]: pgmap v14989: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T22:30:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27146.370964 secs 2023-03-09T22:30:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27146.370964 secs 2023-03-09T22:30:35.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27146.370964 secs 2023-03-09T22:30:36.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:36 smithi122 ceph-mon[93869]: pgmap v14990: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T22:30:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:36 smithi181 ceph-mon[94189]: pgmap v14990: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T22:30:36.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:36 smithi038 ceph-mon[92422]: pgmap v14990: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T22:30:37.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:30:37.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:30:37.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:37 smithi122 ceph-mon[93869]: pgmap v14991: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 853 B/s wr, 8 op/s 2023-03-09T22:30:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:30:37.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:30:37.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:37 smithi181 ceph-mon[94189]: pgmap v14991: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 853 B/s wr, 8 op/s 2023-03-09T22:30:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:30:37.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:30:37.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:37 smithi038 ceph-mon[92422]: pgmap v14991: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 853 B/s wr, 8 op/s 2023-03-09T22:30:38.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27148.745187 secs 2023-03-09T22:30:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27148.745187 secs 2023-03-09T22:30:38.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27148.745187 secs 2023-03-09T22:30:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27610.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:30:39.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:39 smithi122 ceph-mon[93869]: pgmap v14992: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 300 KiB/s wr, 7 op/s 2023-03-09T22:30:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:39 smithi181 ceph-mon[94189]: pgmap v14992: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 300 KiB/s wr, 7 op/s 2023-03-09T22:30:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:39 smithi038 ceph-mon[92422]: pgmap v14992: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 300 KiB/s wr, 7 op/s 2023-03-09T22:30:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:30:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:30:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27151.370995 secs 2023-03-09T22:30:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:30:40.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:30:40.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27151.370995 secs 2023-03-09T22:30:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:30:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:30:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27151.370995 secs 2023-03-09T22:30:42.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:42 smithi122 ceph-mon[93869]: pgmap v14993: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 338 KiB/s wr, 9 op/s 2023-03-09T22:30:42.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:42 smithi181 ceph-mon[94189]: pgmap v14993: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 338 KiB/s wr, 9 op/s 2023-03-09T22:30:42.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:42 smithi038 ceph-mon[92422]: pgmap v14993: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 338 KiB/s wr, 9 op/s 2023-03-09T22:30:43.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27153.745310 secs 2023-03-09T22:30:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27153.745310 secs 2023-03-09T22:30:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27153.745310 secs 2023-03-09T22:30:43.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27615.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:30:44.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:44 smithi122 ceph-mon[93869]: pgmap v14994: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 338 KiB/s wr, 7 op/s 2023-03-09T22:30:44.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:44 smithi181 ceph-mon[94189]: pgmap v14994: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 338 KiB/s wr, 7 op/s 2023-03-09T22:30:44.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:44 smithi038 ceph-mon[92422]: pgmap v14994: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 338 KiB/s wr, 7 op/s 2023-03-09T22:30:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27156.371054 secs 2023-03-09T22:30:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27156.371054 secs 2023-03-09T22:30:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27156.371054 secs 2023-03-09T22:30:46.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:46 smithi122 ceph-mon[93869]: pgmap v14995: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 338 KiB/s wr, 7 op/s 2023-03-09T22:30:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:46 smithi181 ceph-mon[94189]: pgmap v14995: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 338 KiB/s wr, 7 op/s 2023-03-09T22:30:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:46 smithi038 ceph-mon[92422]: pgmap v14995: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 338 KiB/s wr, 7 op/s 2023-03-09T22:30:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:48 smithi122 ceph-mon[93869]: pgmap v14996: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 700 KiB/s rd, 664 KiB/s wr, 11 op/s 2023-03-09T22:30:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27158.745379 secs 2023-03-09T22:30:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:48 smithi181 ceph-mon[94189]: pgmap v14996: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 700 KiB/s rd, 664 KiB/s wr, 11 op/s 2023-03-09T22:30:48.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27158.745379 secs 2023-03-09T22:30:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:48 smithi038 ceph-mon[92422]: pgmap v14996: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 700 KiB/s rd, 664 KiB/s wr, 11 op/s 2023-03-09T22:30:48.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27158.745379 secs 2023-03-09T22:30:48.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27620.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:30:49.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:30:49.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:30:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:30:49.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:30:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:30:49.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:30:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:50 smithi122 ceph-mon[93869]: pgmap v14997: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 699 KiB/s rd, 663 KiB/s wr, 8 op/s 2023-03-09T22:30:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27161.371112 secs 2023-03-09T22:30:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:50 smithi181 ceph-mon[94189]: pgmap v14997: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 699 KiB/s rd, 663 KiB/s wr, 8 op/s 2023-03-09T22:30:50.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27161.371112 secs 2023-03-09T22:30:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:50 smithi038 ceph-mon[92422]: pgmap v14997: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 699 KiB/s rd, 663 KiB/s wr, 8 op/s 2023-03-09T22:30:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27161.371112 secs 2023-03-09T22:30:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:52 smithi122 ceph-mon[93869]: pgmap v14998: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 662 KiB/s rd, 389 KiB/s wr, 11 op/s 2023-03-09T22:30:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:30:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:30:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:52 smithi181 ceph-mon[94189]: pgmap v14998: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 662 KiB/s rd, 389 KiB/s wr, 11 op/s 2023-03-09T22:30:52.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:30:52.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:30:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:52 smithi038 ceph-mon[92422]: pgmap v14998: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 662 KiB/s rd, 389 KiB/s wr, 11 op/s 2023-03-09T22:30:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:30:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:30:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:53 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:30:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:53 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:30:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:53 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:30:53.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27625.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:30:54.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:54 smithi122 ceph-mon[93869]: pgmap v14999: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 877 KiB/s rd, 686 KiB/s wr, 11 op/s 2023-03-09T22:30:54.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:54 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:30:54.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:54 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:30:54.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:54 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:30:54.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:54 smithi181 ceph-mon[94189]: pgmap v14999: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 877 KiB/s rd, 686 KiB/s wr, 11 op/s 2023-03-09T22:30:54.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:54 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:30:54.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:54 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:30:54.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:54 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:30:54.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:54 smithi038 ceph-mon[92422]: pgmap v14999: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 877 KiB/s rd, 686 KiB/s wr, 11 op/s 2023-03-09T22:30:54.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:54 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:30:54.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:54 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:30:54.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:54 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:30:55.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27166.371177 secs 2023-03-09T22:30:55.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:55 smithi122 ceph-mon[93869]: pgmap v15000: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 877 KiB/s rd, 686 KiB/s wr, 11 op/s 2023-03-09T22:30:55.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27166.371177 secs 2023-03-09T22:30:55.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:55 smithi181 ceph-mon[94189]: pgmap v15000: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 877 KiB/s rd, 686 KiB/s wr, 11 op/s 2023-03-09T22:30:55.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27166.371177 secs 2023-03-09T22:30:55.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:55 smithi038 ceph-mon[92422]: pgmap v15000: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 877 KiB/s rd, 686 KiB/s wr, 11 op/s 2023-03-09T22:30:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:30:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:30:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:30:56.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:30:56.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:30:56.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:30:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:30:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:30:57.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:30:57 smithi122 ceph-mon[93869]: pgmap v15001: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 899 KiB/s rd, 797 KiB/s wr, 15 op/s 2023-03-09T22:30:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:30:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:30:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:30:57 smithi181 ceph-mon[94189]: pgmap v15001: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 899 KiB/s rd, 797 KiB/s wr, 15 op/s 2023-03-09T22:30:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:30:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:30:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:30:57 smithi038 ceph-mon[92422]: pgmap v15001: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 899 KiB/s rd, 797 KiB/s wr, 15 op/s 2023-03-09T22:30:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27630.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:31:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:00 smithi122 ceph-mon[93869]: pgmap v15002: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 712 KiB/s rd, 565 KiB/s wr, 12 op/s 2023-03-09T22:31:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27171.371253 secs 2023-03-09T22:31:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:00 smithi181 ceph-mon[94189]: pgmap v15002: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 712 KiB/s rd, 565 KiB/s wr, 12 op/s 2023-03-09T22:31:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27171.371253 secs 2023-03-09T22:31:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:00 smithi038 ceph-mon[92422]: pgmap v15002: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 712 KiB/s rd, 565 KiB/s wr, 12 op/s 2023-03-09T22:31:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27171.371253 secs 2023-03-09T22:31:02.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:02 smithi122 ceph-mon[93869]: pgmap v15003: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 810 KiB/s rd, 662 KiB/s wr, 15 op/s 2023-03-09T22:31:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:02 smithi181 ceph-mon[94189]: pgmap v15003: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 810 KiB/s rd, 662 KiB/s wr, 15 op/s 2023-03-09T22:31:02.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:02 smithi038 ceph-mon[92422]: pgmap v15003: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 810 KiB/s rd, 662 KiB/s wr, 15 op/s 2023-03-09T22:31:03.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27173.745770 secs 2023-03-09T22:31:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27173.745770 secs 2023-03-09T22:31:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27173.745770 secs 2023-03-09T22:31:03.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27635.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:31:04.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:04 smithi122 ceph-mon[93869]: pgmap v15004: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 813 KiB/s rd, 826 KiB/s wr, 25 op/s 2023-03-09T22:31:04.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:31:04.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:31:04.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:31:04.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:04 smithi181 ceph-mon[94189]: pgmap v15004: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 813 KiB/s rd, 826 KiB/s wr, 25 op/s 2023-03-09T22:31:04.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:31:04.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:31:04.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:31:04.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:04 smithi038 ceph-mon[92422]: pgmap v15004: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 813 KiB/s rd, 826 KiB/s wr, 25 op/s 2023-03-09T22:31:04.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:31:04.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:31:04.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:31:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:05 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:31:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:05 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:31:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:05 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:31:05.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:05 smithi122 ceph-mon[93869]: osdmap e1153: 12 total, 12 up, 12 in 2023-03-09T22:31:05.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27176.375993 secs 2023-03-09T22:31:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:05 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:31:05.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:05 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:31:05.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:05 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:31:05.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:05 smithi181 ceph-mon[94189]: osdmap e1153: 12 total, 12 up, 12 in 2023-03-09T22:31:05.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27176.375993 secs 2023-03-09T22:31:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:05 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:31:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:05 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:31:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:05 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:31:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:05 smithi038 ceph-mon[92422]: osdmap e1153: 12 total, 12 up, 12 in 2023-03-09T22:31:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27176.375993 secs 2023-03-09T22:31:06.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:06 smithi122 ceph-mon[93869]: pgmap v15006: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 671 KiB/s rd, 590 KiB/s wr, 27 op/s 2023-03-09T22:31:06.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:06 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:31:06.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:06 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:31:06.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:31:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:06 smithi181 ceph-mon[94189]: pgmap v15006: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 671 KiB/s rd, 590 KiB/s wr, 27 op/s 2023-03-09T22:31:06.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:06 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:31:06.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:06 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:31:06.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:31:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:06 smithi038 ceph-mon[92422]: pgmap v15006: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 671 KiB/s rd, 590 KiB/s wr, 27 op/s 2023-03-09T22:31:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:06 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:31:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:06 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:31:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:31:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:08 smithi122 ceph-mon[93869]: pgmap v15007: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 780 KiB/s rd, 726 KiB/s wr, 30 op/s 2023-03-09T22:31:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:31:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:31:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27178.745916 secs 2023-03-09T22:31:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:08 smithi181 ceph-mon[94189]: pgmap v15007: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 780 KiB/s rd, 726 KiB/s wr, 30 op/s 2023-03-09T22:31:08.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:31:08.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:31:08.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27178.745916 secs 2023-03-09T22:31:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:08 smithi038 ceph-mon[92422]: pgmap v15007: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 780 KiB/s rd, 726 KiB/s wr, 30 op/s 2023-03-09T22:31:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:31:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:31:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27178.745916 secs 2023-03-09T22:31:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27640.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:31:09.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:31:09.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:31:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:31:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:31:09.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:31:09.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:31:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:10 smithi122 ceph-mon[93869]: pgmap v15008: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 571 KiB/s rd, 612 KiB/s wr, 29 op/s 2023-03-09T22:31:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27181.376056 secs 2023-03-09T22:31:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:10 smithi181 ceph-mon[94189]: pgmap v15008: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 571 KiB/s rd, 612 KiB/s wr, 29 op/s 2023-03-09T22:31:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27181.376056 secs 2023-03-09T22:31:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:10 smithi038 ceph-mon[92422]: pgmap v15008: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 571 KiB/s rd, 612 KiB/s wr, 29 op/s 2023-03-09T22:31:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27181.376056 secs 2023-03-09T22:31:12.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:12 smithi122 ceph-mon[93869]: pgmap v15009: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 589 KiB/s rd, 630 KiB/s wr, 29 op/s 2023-03-09T22:31:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:12 smithi181 ceph-mon[94189]: pgmap v15009: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 589 KiB/s rd, 630 KiB/s wr, 29 op/s 2023-03-09T22:31:12.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:12 smithi038 ceph-mon[92422]: pgmap v15009: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 589 KiB/s rd, 630 KiB/s wr, 29 op/s 2023-03-09T22:31:13.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27183.746033 secs 2023-03-09T22:31:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27183.746033 secs 2023-03-09T22:31:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27183.746033 secs 2023-03-09T22:31:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27645.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:31:14.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:14 smithi122 ceph-mon[93869]: pgmap v15010: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 403 KiB/s wr, 13 op/s 2023-03-09T22:31:14.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:14 smithi181 ceph-mon[94189]: pgmap v15010: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 403 KiB/s wr, 13 op/s 2023-03-09T22:31:14.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:14 smithi038 ceph-mon[92422]: pgmap v15010: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 403 KiB/s wr, 13 op/s 2023-03-09T22:31:15.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27186.376119 secs 2023-03-09T22:31:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27186.376119 secs 2023-03-09T22:31:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27186.376119 secs 2023-03-09T22:31:16.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:16 smithi122 ceph-mon[93869]: pgmap v15011: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 394 KiB/s wr, 12 op/s 2023-03-09T22:31:16.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:16 smithi122 ceph-mon[93869]: osdmap e1154: 12 total, 12 up, 12 in 2023-03-09T22:31:16.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:16 smithi181 ceph-mon[94189]: pgmap v15011: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 394 KiB/s wr, 12 op/s 2023-03-09T22:31:16.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:16 smithi181 ceph-mon[94189]: osdmap e1154: 12 total, 12 up, 12 in 2023-03-09T22:31:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:16 smithi038 ceph-mon[92422]: pgmap v15011: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 394 KiB/s wr, 12 op/s 2023-03-09T22:31:16.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:16 smithi038 ceph-mon[92422]: osdmap e1154: 12 total, 12 up, 12 in 2023-03-09T22:31:17.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:17 smithi122 ceph-mon[93869]: pgmap v15013: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 135 KiB/s wr, 8 op/s 2023-03-09T22:31:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:17 smithi181 ceph-mon[94189]: pgmap v15013: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 135 KiB/s wr, 8 op/s 2023-03-09T22:31:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:17 smithi038 ceph-mon[92422]: pgmap v15013: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 135 KiB/s wr, 8 op/s 2023-03-09T22:31:18.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27188.746154 secs 2023-03-09T22:31:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27188.746154 secs 2023-03-09T22:31:18.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27188.746154 secs 2023-03-09T22:31:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27650.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:31:19.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:19 smithi122 ceph-mon[93869]: pgmap v15014: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 135 KiB/s wr, 7 op/s 2023-03-09T22:31:19.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:19 smithi181 ceph-mon[94189]: pgmap v15014: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 135 KiB/s wr, 7 op/s 2023-03-09T22:31:19.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:19 smithi038 ceph-mon[92422]: pgmap v15014: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 135 KiB/s wr, 7 op/s 2023-03-09T22:31:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27191.376189 secs 2023-03-09T22:31:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27191.376189 secs 2023-03-09T22:31:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27191.376189 secs 2023-03-09T22:31:22.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:22 smithi122 ceph-mon[93869]: pgmap v15015: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 95 KiB/s wr, 7 op/s 2023-03-09T22:31:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:22 smithi181 ceph-mon[94189]: pgmap v15015: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 95 KiB/s wr, 7 op/s 2023-03-09T22:31:22.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:22 smithi038 ceph-mon[92422]: pgmap v15015: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 95 KiB/s wr, 7 op/s 2023-03-09T22:31:23.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27193.746251 secs 2023-03-09T22:31:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27193.746251 secs 2023-03-09T22:31:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27193.746251 secs 2023-03-09T22:31:23.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27655.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:31:24.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:24 smithi122 ceph-mon[93869]: pgmap v15016: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 95 KiB/s wr, 7 op/s 2023-03-09T22:31:24.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:24 smithi181 ceph-mon[94189]: pgmap v15016: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 95 KiB/s wr, 7 op/s 2023-03-09T22:31:24.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:24 smithi038 ceph-mon[92422]: pgmap v15016: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 95 KiB/s wr, 7 op/s 2023-03-09T22:31:25.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27196.376250 secs 2023-03-09T22:31:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27196.376250 secs 2023-03-09T22:31:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27196.376250 secs 2023-03-09T22:31:26.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:26 smithi122 ceph-mon[93869]: pgmap v15017: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 95 KiB/s wr, 7 op/s 2023-03-09T22:31:26.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:26 smithi181 ceph-mon[94189]: pgmap v15017: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 95 KiB/s wr, 7 op/s 2023-03-09T22:31:26.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:26 smithi038 ceph-mon[92422]: pgmap v15017: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 95 KiB/s wr, 7 op/s 2023-03-09T22:31:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:28 smithi122 ceph-mon[93869]: pgmap v15018: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 87 KiB/s wr, 6 op/s 2023-03-09T22:31:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27198.746355 secs 2023-03-09T22:31:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:28 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:31:28.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:28 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:31:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:28 smithi181 ceph-mon[94189]: pgmap v15018: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 87 KiB/s wr, 6 op/s 2023-03-09T22:31:28.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27198.746355 secs 2023-03-09T22:31:28.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:28 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:31:28.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:28 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:31:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:28 smithi038 ceph-mon[92422]: pgmap v15018: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 87 KiB/s wr, 6 op/s 2023-03-09T22:31:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27198.746355 secs 2023-03-09T22:31:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:28 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:31:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:28 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:31:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27660.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:31:29.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:29 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:31:29.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:29 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:31:29.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:29 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:31:29.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:29 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:31:29.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:29 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:31:29.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:29 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:31:30.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:30 smithi122 ceph-mon[93869]: pgmap v15019: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 79 KiB/s wr, 5 op/s 2023-03-09T22:31:30.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27201.376334 secs 2023-03-09T22:31:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:30 smithi181 ceph-mon[94189]: pgmap v15019: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 79 KiB/s wr, 5 op/s 2023-03-09T22:31:30.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27201.376334 secs 2023-03-09T22:31:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:30 smithi038 ceph-mon[92422]: pgmap v15019: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 79 KiB/s wr, 5 op/s 2023-03-09T22:31:30.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27201.376334 secs 2023-03-09T22:31:32.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:32 smithi122 ceph-mon[93869]: pgmap v15020: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 79 KiB/s wr, 8 op/s 2023-03-09T22:31:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:32 smithi181 ceph-mon[94189]: pgmap v15020: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 79 KiB/s wr, 8 op/s 2023-03-09T22:31:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:32 smithi038 ceph-mon[92422]: pgmap v15020: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 79 KiB/s wr, 8 op/s 2023-03-09T22:31:33.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27203.746425 secs 2023-03-09T22:31:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27203.746425 secs 2023-03-09T22:31:33.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27203.746425 secs 2023-03-09T22:31:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27665.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:31:34.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:34 smithi122 ceph-mon[93869]: pgmap v15021: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 511 B/s wr, 5 op/s 2023-03-09T22:31:34.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:34 smithi181 ceph-mon[94189]: pgmap v15021: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 511 B/s wr, 5 op/s 2023-03-09T22:31:34.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:34 smithi038 ceph-mon[92422]: pgmap v15021: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 511 B/s wr, 5 op/s 2023-03-09T22:31:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27206.376396 secs 2023-03-09T22:31:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:35 smithi122 ceph-mon[93869]: pgmap v15022: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 511 B/s wr, 5 op/s 2023-03-09T22:31:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27206.376396 secs 2023-03-09T22:31:35.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:35 smithi181 ceph-mon[94189]: pgmap v15022: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 511 B/s wr, 5 op/s 2023-03-09T22:31:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27206.376396 secs 2023-03-09T22:31:35.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:35 smithi038 ceph-mon[92422]: pgmap v15022: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 511 B/s wr, 5 op/s 2023-03-09T22:31:38.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:38 smithi122 ceph-mon[93869]: pgmap v15023: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 116 KiB/s wr, 8 op/s 2023-03-09T22:31:38.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27208.746552 secs 2023-03-09T22:31:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:38 smithi181 ceph-mon[94189]: pgmap v15023: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 116 KiB/s wr, 8 op/s 2023-03-09T22:31:38.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27208.746552 secs 2023-03-09T22:31:38.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:38 smithi038 ceph-mon[92422]: pgmap v15023: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 116 KiB/s wr, 8 op/s 2023-03-09T22:31:38.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27208.746552 secs 2023-03-09T22:31:38.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27670.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:31:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:40 smithi122 ceph-mon[93869]: pgmap v15024: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 116 KiB/s wr, 6 op/s 2023-03-09T22:31:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27211.376459 secs 2023-03-09T22:31:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:40 smithi181 ceph-mon[94189]: pgmap v15024: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 116 KiB/s wr, 6 op/s 2023-03-09T22:31:40.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27211.376459 secs 2023-03-09T22:31:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:40 smithi038 ceph-mon[92422]: pgmap v15024: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 116 KiB/s wr, 6 op/s 2023-03-09T22:31:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27211.376459 secs 2023-03-09T22:31:42.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:42 smithi122 ceph-mon[93869]: pgmap v15025: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 237 KiB/s wr, 9 op/s 2023-03-09T22:31:42.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:42 smithi181 ceph-mon[94189]: pgmap v15025: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 237 KiB/s wr, 9 op/s 2023-03-09T22:31:42.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:42 smithi038 ceph-mon[92422]: pgmap v15025: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 237 KiB/s wr, 9 op/s 2023-03-09T22:31:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27675.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:31:44.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:44 smithi122 ceph-mon[93869]: pgmap v15026: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 243 KiB/s wr, 7 op/s 2023-03-09T22:31:44.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:44 smithi181 ceph-mon[94189]: pgmap v15026: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 243 KiB/s wr, 7 op/s 2023-03-09T22:31:44.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:44 smithi038 ceph-mon[92422]: pgmap v15026: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 243 KiB/s wr, 7 op/s 2023-03-09T22:31:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27216.376518 secs 2023-03-09T22:31:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:45 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:31:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27216.376518 secs 2023-03-09T22:31:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:45 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:31:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27216.376518 secs 2023-03-09T22:31:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:45 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:31:46.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:46 smithi122 ceph-mon[93869]: pgmap v15027: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 243 KiB/s wr, 6 op/s 2023-03-09T22:31:46.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:46 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:31:46.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:46 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:31:46.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:46 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:31:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:46 smithi181 ceph-mon[94189]: pgmap v15027: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 243 KiB/s wr, 6 op/s 2023-03-09T22:31:46.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:46 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:31:46.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:46 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:31:46.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:46 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:31:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:46 smithi038 ceph-mon[92422]: pgmap v15027: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 243 KiB/s wr, 6 op/s 2023-03-09T22:31:46.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:46 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:31:46.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:46 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:31:46.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:46 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:31:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:48 smithi122 ceph-mon[93869]: pgmap v15028: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 265 KiB/s wr, 9 op/s 2023-03-09T22:31:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:48 smithi181 ceph-mon[94189]: pgmap v15028: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 265 KiB/s wr, 9 op/s 2023-03-09T22:31:48.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:48 smithi038 ceph-mon[92422]: pgmap v15028: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 265 KiB/s wr, 9 op/s 2023-03-09T22:31:48.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27680.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:31:49.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:31:49.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:31:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:31:49.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:31:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:31:49.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:31:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:50 smithi122 ceph-mon[93869]: pgmap v15029: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 677 KiB/s rd, 485 KiB/s wr, 8 op/s 2023-03-09T22:31:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27221.376580 secs 2023-03-09T22:31:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:50 smithi181 ceph-mon[94189]: pgmap v15029: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 677 KiB/s rd, 485 KiB/s wr, 8 op/s 2023-03-09T22:31:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27221.376580 secs 2023-03-09T22:31:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:50 smithi038 ceph-mon[92422]: pgmap v15029: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 677 KiB/s rd, 485 KiB/s wr, 8 op/s 2023-03-09T22:31:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27221.376580 secs 2023-03-09T22:31:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:52 smithi122 ceph-mon[93869]: pgmap v15030: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 511 KiB/s wr, 11 op/s 2023-03-09T22:31:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:31:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:31:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:52 smithi181 ceph-mon[94189]: pgmap v15030: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 511 KiB/s wr, 11 op/s 2023-03-09T22:31:52.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:31:52.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:31:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:52 smithi038 ceph-mon[92422]: pgmap v15030: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 511 KiB/s wr, 11 op/s 2023-03-09T22:31:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:31:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:31:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:53 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 27223.746915 secs 2023-03-09T22:31:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:53 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 27223.746915 secs 2023-03-09T22:31:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:53 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 27223.746915 secs 2023-03-09T22:31:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27685.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:31:54.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:54 smithi122 ceph-mon[93869]: pgmap v15031: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 961 KiB/s rd, 731 KiB/s wr, 11 op/s 2023-03-09T22:31:54.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:54 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:31:54.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:54 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:31:54.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:54 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:31:54.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:54 smithi181 ceph-mon[94189]: pgmap v15031: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 961 KiB/s rd, 731 KiB/s wr, 11 op/s 2023-03-09T22:31:54.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:54 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:31:54.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:54 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:31:54.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:54 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:31:54.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:54 smithi038 ceph-mon[92422]: pgmap v15031: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 961 KiB/s rd, 731 KiB/s wr, 11 op/s 2023-03-09T22:31:54.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:54 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:31:54.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:54 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:31:54.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:54 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:31:55.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27226.376663 secs 2023-03-09T22:31:55.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:55 smithi122 ceph-mon[93869]: pgmap v15032: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 954 KiB/s rd, 725 KiB/s wr, 10 op/s 2023-03-09T22:31:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27226.376663 secs 2023-03-09T22:31:55.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:55 smithi181 ceph-mon[94189]: pgmap v15032: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 954 KiB/s rd, 725 KiB/s wr, 10 op/s 2023-03-09T22:31:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27226.376663 secs 2023-03-09T22:31:55.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:55 smithi038 ceph-mon[92422]: pgmap v15032: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 954 KiB/s rd, 725 KiB/s wr, 10 op/s 2023-03-09T22:31:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:31:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:31:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:31:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:31:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:31:56.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:31:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:31:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:31:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:57 smithi122 ceph-mon[93869]: pgmap v15033: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 893 KiB/s wr, 15 op/s 2023-03-09T22:31:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:31:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:31:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:57 smithi181 ceph-mon[94189]: pgmap v15033: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 893 KiB/s wr, 15 op/s 2023-03-09T22:31:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:31:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:31:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:57 smithi038 ceph-mon[92422]: pgmap v15033: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 893 KiB/s wr, 15 op/s 2023-03-09T22:31:58.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:58 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 27228.746994 secs 2023-03-09T22:31:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:58 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 27228.746994 secs 2023-03-09T22:31:58.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:58 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 27228.746994 secs 2023-03-09T22:31:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27690.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:31:59.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:31:59 smithi122 ceph-mon[93869]: pgmap v15034: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 871 KiB/s wr, 12 op/s 2023-03-09T22:31:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:31:59 smithi181 ceph-mon[94189]: pgmap v15034: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 871 KiB/s wr, 12 op/s 2023-03-09T22:31:59.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:31:59 smithi038 ceph-mon[92422]: pgmap v15034: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 871 KiB/s wr, 12 op/s 2023-03-09T22:32:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:32:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:32:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27231.376740 secs 2023-03-09T22:32:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:32:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:32:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27231.376740 secs 2023-03-09T22:32:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:32:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:32:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27231.376740 secs 2023-03-09T22:32:02.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:02 smithi122 ceph-mon[93869]: pgmap v15035: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 912 KiB/s rd, 792 KiB/s wr, 17 op/s 2023-03-09T22:32:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:02 smithi181 ceph-mon[94189]: pgmap v15035: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 912 KiB/s rd, 792 KiB/s wr, 17 op/s 2023-03-09T22:32:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:02 smithi038 ceph-mon[92422]: pgmap v15035: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 912 KiB/s rd, 792 KiB/s wr, 17 op/s 2023-03-09T22:32:03.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27233.747135 secs 2023-03-09T22:32:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27233.747135 secs 2023-03-09T22:32:03.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27233.747135 secs 2023-03-09T22:32:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27695.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:32:04.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:04 smithi122 ceph-mon[93869]: pgmap v15036: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 880 KiB/s rd, 788 KiB/s wr, 26 op/s 2023-03-09T22:32:04.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:04 smithi181 ceph-mon[94189]: pgmap v15036: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 880 KiB/s rd, 788 KiB/s wr, 26 op/s 2023-03-09T22:32:04.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:04 smithi038 ceph-mon[92422]: pgmap v15036: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 880 KiB/s rd, 788 KiB/s wr, 26 op/s 2023-03-09T22:32:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:05 smithi122 ceph-mon[93869]: osdmap e1155: 12 total, 12 up, 12 in 2023-03-09T22:32:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:05 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:32:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27236.376801 secs 2023-03-09T22:32:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:05 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:32:05.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:05 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:32:05.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:05 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:32:05.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:05 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:32:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:05 smithi181 ceph-mon[94189]: osdmap e1155: 12 total, 12 up, 12 in 2023-03-09T22:32:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:05 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:32:05.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27236.376801 secs 2023-03-09T22:32:05.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:05 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:32:05.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:05 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:32:05.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:05 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:32:05.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:05 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:32:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:05 smithi038 ceph-mon[92422]: osdmap e1155: 12 total, 12 up, 12 in 2023-03-09T22:32:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:05 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:32:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27236.376801 secs 2023-03-09T22:32:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:05 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:32:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:05 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:32:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:05 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:32:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:05 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:32:06.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:06 smithi122 ceph-mon[93869]: pgmap v15038: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 792 KiB/s rd, 713 KiB/s wr, 29 op/s 2023-03-09T22:32:06.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:06 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:32:06.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:32:06.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:06 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:32:06.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:06 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:32:06.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:06 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:32:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:06 smithi181 ceph-mon[94189]: pgmap v15038: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 792 KiB/s rd, 713 KiB/s wr, 29 op/s 2023-03-09T22:32:06.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:06 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:32:06.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:32:06.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:06 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:32:06.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:06 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:32:06.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:06 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:32:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:06 smithi038 ceph-mon[92422]: pgmap v15038: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 792 KiB/s rd, 713 KiB/s wr, 29 op/s 2023-03-09T22:32:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:06 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:32:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:32:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:06 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:32:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:06 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:32:06.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:06 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:32:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:08 smithi122 ceph-mon[93869]: pgmap v15039: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 596 KiB/s rd, 516 KiB/s wr, 27 op/s 2023-03-09T22:32:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27238.747256 secs 2023-03-09T22:32:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:08 smithi181 ceph-mon[94189]: pgmap v15039: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 596 KiB/s rd, 516 KiB/s wr, 27 op/s 2023-03-09T22:32:08.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27238.747256 secs 2023-03-09T22:32:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:08 smithi038 ceph-mon[92422]: pgmap v15039: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 596 KiB/s rd, 516 KiB/s wr, 27 op/s 2023-03-09T22:32:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27238.747256 secs 2023-03-09T22:32:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27700.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:32:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:10 smithi122 ceph-mon[93869]: pgmap v15040: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 533 KiB/s rd, 574 KiB/s wr, 28 op/s 2023-03-09T22:32:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27241.376862 secs 2023-03-09T22:32:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:10 smithi181 ceph-mon[94189]: pgmap v15040: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 533 KiB/s rd, 574 KiB/s wr, 28 op/s 2023-03-09T22:32:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27241.376862 secs 2023-03-09T22:32:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:10 smithi038 ceph-mon[92422]: pgmap v15040: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 533 KiB/s rd, 574 KiB/s wr, 28 op/s 2023-03-09T22:32:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27241.376862 secs 2023-03-09T22:32:12.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:12 smithi122 ceph-mon[93869]: pgmap v15041: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 267 KiB/s wr, 23 op/s 2023-03-09T22:32:12.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:32:12.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:32:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:12 smithi181 ceph-mon[94189]: pgmap v15041: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 267 KiB/s wr, 23 op/s 2023-03-09T22:32:12.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:32:12.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:32:12.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:12 smithi038 ceph-mon[92422]: pgmap v15041: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 267 KiB/s wr, 23 op/s 2023-03-09T22:32:12.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:32:12.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:32:13.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27243.747379 secs 2023-03-09T22:32:13.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:32:13.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:32:13.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:13 smithi122 ceph-mon[93869]: pgmap v15042: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 242 KiB/s wr, 9 op/s 2023-03-09T22:32:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27243.747379 secs 2023-03-09T22:32:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:32:13.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:32:13.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:13 smithi181 ceph-mon[94189]: pgmap v15042: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 242 KiB/s wr, 9 op/s 2023-03-09T22:32:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27243.747379 secs 2023-03-09T22:32:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:32:13.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:32:13.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:13 smithi038 ceph-mon[92422]: pgmap v15042: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 242 KiB/s wr, 9 op/s 2023-03-09T22:32:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27705.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:32:15.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27246.376921 secs 2023-03-09T22:32:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27246.376921 secs 2023-03-09T22:32:15.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27246.376921 secs 2023-03-09T22:32:16.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:16 smithi122 ceph-mon[93869]: pgmap v15043: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 183 KiB/s wr, 8 op/s 2023-03-09T22:32:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:16 smithi181 ceph-mon[94189]: pgmap v15043: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 183 KiB/s wr, 8 op/s 2023-03-09T22:32:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:16 smithi038 ceph-mon[92422]: pgmap v15043: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 183 KiB/s wr, 8 op/s 2023-03-09T22:32:17.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:17 smithi122 ceph-mon[93869]: osdmap e1156: 12 total, 12 up, 12 in 2023-03-09T22:32:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:17 smithi181 ceph-mon[94189]: osdmap e1156: 12 total, 12 up, 12 in 2023-03-09T22:32:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:17 smithi038 ceph-mon[92422]: osdmap e1156: 12 total, 12 up, 12 in 2023-03-09T22:32:18.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:18 smithi122 ceph-mon[93869]: pgmap v15045: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 193 KiB/s wr, 8 op/s 2023-03-09T22:32:18.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27248.747446 secs 2023-03-09T22:32:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:18 smithi181 ceph-mon[94189]: pgmap v15045: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 193 KiB/s wr, 8 op/s 2023-03-09T22:32:18.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27248.747446 secs 2023-03-09T22:32:18.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:18 smithi038 ceph-mon[92422]: pgmap v15045: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 193 KiB/s wr, 8 op/s 2023-03-09T22:32:18.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27248.747446 secs 2023-03-09T22:32:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27710.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:32:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:20 smithi122 ceph-mon[93869]: pgmap v15046: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 135 KiB/s wr, 7 op/s 2023-03-09T22:32:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27251.376983 secs 2023-03-09T22:32:20.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:20 smithi181 ceph-mon[94189]: pgmap v15046: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 135 KiB/s wr, 7 op/s 2023-03-09T22:32:20.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27251.376983 secs 2023-03-09T22:32:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:20 smithi038 ceph-mon[92422]: pgmap v15046: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 135 KiB/s wr, 7 op/s 2023-03-09T22:32:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27251.376983 secs 2023-03-09T22:32:22.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:22 smithi122 ceph-mon[93869]: pgmap v15047: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 135 KiB/s wr, 6 op/s 2023-03-09T22:32:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:22 smithi181 ceph-mon[94189]: pgmap v15047: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 135 KiB/s wr, 6 op/s 2023-03-09T22:32:22.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:22 smithi038 ceph-mon[92422]: pgmap v15047: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 135 KiB/s wr, 6 op/s 2023-03-09T22:32:23.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27253.747566 secs 2023-03-09T22:32:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27253.747566 secs 2023-03-09T22:32:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27253.747566 secs 2023-03-09T22:32:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27715.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:32:24.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:24 smithi122 ceph-mon[93869]: pgmap v15048: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 134 KiB/s wr, 6 op/s 2023-03-09T22:32:24.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:24 smithi181 ceph-mon[94189]: pgmap v15048: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 134 KiB/s wr, 6 op/s 2023-03-09T22:32:24.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:24 smithi038 ceph-mon[92422]: pgmap v15048: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 134 KiB/s wr, 6 op/s 2023-03-09T22:32:25.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27256.377044 secs 2023-03-09T22:32:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27256.377044 secs 2023-03-09T22:32:25.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27256.377044 secs 2023-03-09T22:32:26.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:26 smithi122 ceph-mon[93869]: pgmap v15049: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T22:32:26.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:26 smithi181 ceph-mon[94189]: pgmap v15049: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T22:32:26.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:26 smithi038 ceph-mon[92422]: pgmap v15049: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T22:32:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:28 smithi122 ceph-mon[93869]: pgmap v15050: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T22:32:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27258.747679 secs 2023-03-09T22:32:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:28 smithi181 ceph-mon[94189]: pgmap v15050: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T22:32:28.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27258.747679 secs 2023-03-09T22:32:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:28 smithi038 ceph-mon[92422]: pgmap v15050: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T22:32:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27258.747679 secs 2023-03-09T22:32:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27720.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:32:29.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:29 smithi122 ceph-mon[93869]: pgmap v15051: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 34 KiB/s wr, 5 op/s 2023-03-09T22:32:29.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:29 smithi181 ceph-mon[94189]: pgmap v15051: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 34 KiB/s wr, 5 op/s 2023-03-09T22:32:29.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:29 smithi038 ceph-mon[92422]: pgmap v15051: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 34 KiB/s wr, 5 op/s 2023-03-09T22:32:30.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27261.377104 secs 2023-03-09T22:32:30.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27261.377104 secs 2023-03-09T22:32:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27261.377104 secs 2023-03-09T22:32:32.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:32 smithi122 ceph-mon[93869]: pgmap v15052: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 34 KiB/s wr, 7 op/s 2023-03-09T22:32:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:32 smithi181 ceph-mon[94189]: pgmap v15052: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 34 KiB/s wr, 7 op/s 2023-03-09T22:32:32.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:32 smithi038 ceph-mon[92422]: pgmap v15052: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 34 KiB/s wr, 7 op/s 2023-03-09T22:32:33.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27263.747772 secs 2023-03-09T22:32:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27263.747772 secs 2023-03-09T22:32:33.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27263.747772 secs 2023-03-09T22:32:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27725.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:32:34.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:34 smithi122 ceph-mon[93869]: pgmap v15053: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 96 KiB/s wr, 6 op/s 2023-03-09T22:32:34.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:34 smithi181 ceph-mon[94189]: pgmap v15053: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 96 KiB/s wr, 6 op/s 2023-03-09T22:32:34.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:34 smithi038 ceph-mon[92422]: pgmap v15053: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 96 KiB/s wr, 6 op/s 2023-03-09T22:32:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27266.377167 secs 2023-03-09T22:32:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27266.377167 secs 2023-03-09T22:32:35.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27266.377167 secs 2023-03-09T22:32:36.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:36 smithi122 ceph-mon[93869]: pgmap v15054: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 96 KiB/s wr, 7 op/s 2023-03-09T22:32:36.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:32:36.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:32:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:36 smithi181 ceph-mon[94189]: pgmap v15054: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 96 KiB/s wr, 7 op/s 2023-03-09T22:32:36.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:32:36.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:32:36.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:36 smithi038 ceph-mon[92422]: pgmap v15054: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 96 KiB/s wr, 7 op/s 2023-03-09T22:32:36.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:32:36.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:32:37.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:32:37.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:32:37.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:32:37.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:32:37.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:32:37.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:32:38.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:38 smithi122 ceph-mon[93869]: pgmap v15055: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 212 KiB/s wr, 9 op/s 2023-03-09T22:32:38.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27268.747898 secs 2023-03-09T22:32:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:38 smithi181 ceph-mon[94189]: pgmap v15055: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 212 KiB/s wr, 9 op/s 2023-03-09T22:32:38.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27268.747898 secs 2023-03-09T22:32:38.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:38 smithi038 ceph-mon[92422]: pgmap v15055: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 212 KiB/s wr, 9 op/s 2023-03-09T22:32:38.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27268.747898 secs 2023-03-09T22:32:38.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27730.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:32:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:40 smithi122 ceph-mon[93869]: pgmap v15056: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 212 KiB/s wr, 8 op/s 2023-03-09T22:32:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27271.377244 secs 2023-03-09T22:32:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:40 smithi181 ceph-mon[94189]: pgmap v15056: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 212 KiB/s wr, 8 op/s 2023-03-09T22:32:40.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27271.377244 secs 2023-03-09T22:32:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:40 smithi038 ceph-mon[92422]: pgmap v15056: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 212 KiB/s wr, 8 op/s 2023-03-09T22:32:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27271.377244 secs 2023-03-09T22:32:42.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:42 smithi122 ceph-mon[93869]: pgmap v15057: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 178 KiB/s wr, 9 op/s 2023-03-09T22:32:42.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:42 smithi181 ceph-mon[94189]: pgmap v15057: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 178 KiB/s wr, 9 op/s 2023-03-09T22:32:42.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:42 smithi038 ceph-mon[92422]: pgmap v15057: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 178 KiB/s wr, 9 op/s 2023-03-09T22:32:43.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27273.747997 secs 2023-03-09T22:32:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27273.747997 secs 2023-03-09T22:32:43.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27273.747997 secs 2023-03-09T22:32:43.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27735.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:32:44.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:44 smithi122 ceph-mon[93869]: pgmap v15058: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 306 KiB/s wr, 8 op/s 2023-03-09T22:32:44.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:44 smithi181 ceph-mon[94189]: pgmap v15058: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 306 KiB/s wr, 8 op/s 2023-03-09T22:32:44.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:44 smithi038 ceph-mon[92422]: pgmap v15058: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 306 KiB/s wr, 8 op/s 2023-03-09T22:32:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27276.377305 secs 2023-03-09T22:32:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:45 smithi122 ceph-mon[93869]: pgmap v15059: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 244 KiB/s wr, 7 op/s 2023-03-09T22:32:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27276.377305 secs 2023-03-09T22:32:45.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:45 smithi181 ceph-mon[94189]: pgmap v15059: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 244 KiB/s wr, 7 op/s 2023-03-09T22:32:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27276.377305 secs 2023-03-09T22:32:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:45 smithi038 ceph-mon[92422]: pgmap v15059: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 244 KiB/s wr, 7 op/s 2023-03-09T22:32:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:48 smithi122 ceph-mon[93869]: pgmap v15060: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 394 KiB/s wr, 11 op/s 2023-03-09T22:32:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27278.748089 secs 2023-03-09T22:32:48.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:48 smithi181 ceph-mon[94189]: pgmap v15060: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 394 KiB/s wr, 11 op/s 2023-03-09T22:32:48.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27278.748089 secs 2023-03-09T22:32:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:48 smithi038 ceph-mon[92422]: pgmap v15060: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 394 KiB/s wr, 11 op/s 2023-03-09T22:32:48.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27278.748089 secs 2023-03-09T22:32:48.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27740.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:32:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:50 smithi122 ceph-mon[93869]: pgmap v15061: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 279 KiB/s wr, 8 op/s 2023-03-09T22:32:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27281.377364 secs 2023-03-09T22:32:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:50 smithi181 ceph-mon[94189]: pgmap v15061: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 279 KiB/s wr, 8 op/s 2023-03-09T22:32:50.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27281.377364 secs 2023-03-09T22:32:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:50 smithi038 ceph-mon[92422]: pgmap v15061: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 280 KiB/s rd, 279 KiB/s wr, 8 op/s 2023-03-09T22:32:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27281.377364 secs 2023-03-09T22:32:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:52 smithi122 ceph-mon[93869]: pgmap v15062: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 301 KiB/s wr, 10 op/s 2023-03-09T22:32:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:32:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:32:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:52 smithi181 ceph-mon[94189]: pgmap v15062: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 301 KiB/s wr, 10 op/s 2023-03-09T22:32:52.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:32:52.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:32:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:52 smithi038 ceph-mon[92422]: pgmap v15062: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 301 KiB/s wr, 10 op/s 2023-03-09T22:32:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:32:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:32:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27283.748215 secs 2023-03-09T22:32:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:32:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:32:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27283.748215 secs 2023-03-09T22:32:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:32:53.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:32:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27283.748215 secs 2023-03-09T22:32:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:32:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:32:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27745.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:32:54.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:54 smithi122 ceph-mon[93869]: pgmap v15063: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 443 KiB/s rd, 429 KiB/s wr, 10 op/s 2023-03-09T22:32:54.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:54 smithi181 ceph-mon[94189]: pgmap v15063: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 443 KiB/s rd, 429 KiB/s wr, 10 op/s 2023-03-09T22:32:54.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:54 smithi038 ceph-mon[92422]: pgmap v15063: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 443 KiB/s rd, 429 KiB/s wr, 10 op/s 2023-03-09T22:32:55.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27286.377425 secs 2023-03-09T22:32:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27286.377425 secs 2023-03-09T22:32:55.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27286.377425 secs 2023-03-09T22:32:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:56 smithi122 ceph-mon[93869]: pgmap v15064: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 301 KiB/s wr, 9 op/s 2023-03-09T22:32:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:32:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:32:56.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:56 smithi181 ceph-mon[94189]: pgmap v15064: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 301 KiB/s wr, 9 op/s 2023-03-09T22:32:56.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:32:56.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:32:56.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:56 smithi038 ceph-mon[92422]: pgmap v15064: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 301 KiB/s wr, 9 op/s 2023-03-09T22:32:56.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:32:56.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:32:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:32:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:32:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:57 smithi122 ceph-mon[93869]: pgmap v15065: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 543 KiB/s rd, 528 KiB/s wr, 14 op/s 2023-03-09T22:32:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:32:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:32:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:57 smithi181 ceph-mon[94189]: pgmap v15065: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 543 KiB/s rd, 528 KiB/s wr, 14 op/s 2023-03-09T22:32:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:32:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:32:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:57 smithi038 ceph-mon[92422]: pgmap v15065: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 543 KiB/s rd, 528 KiB/s wr, 14 op/s 2023-03-09T22:32:58.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27288.748321 secs 2023-03-09T22:32:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27288.748321 secs 2023-03-09T22:32:58.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27288.748321 secs 2023-03-09T22:32:58.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27750.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:32:59.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:32:59 smithi122 ceph-mon[93869]: pgmap v15066: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 618 KiB/s rd, 604 KiB/s wr, 11 op/s 2023-03-09T22:32:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:32:59 smithi181 ceph-mon[94189]: pgmap v15066: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 618 KiB/s rd, 604 KiB/s wr, 11 op/s 2023-03-09T22:32:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:32:59 smithi038 ceph-mon[92422]: pgmap v15066: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 618 KiB/s rd, 604 KiB/s wr, 11 op/s 2023-03-09T22:33:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27291.377488 secs 2023-03-09T22:33:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27291.377488 secs 2023-03-09T22:33:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27291.377488 secs 2023-03-09T22:33:02.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:02 smithi122 ceph-mon[93869]: pgmap v15067: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 734 KiB/s rd, 741 KiB/s wr, 17 op/s 2023-03-09T22:33:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:02 smithi181 ceph-mon[94189]: pgmap v15067: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 734 KiB/s rd, 741 KiB/s wr, 17 op/s 2023-03-09T22:33:02.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:02 smithi038 ceph-mon[92422]: pgmap v15067: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 734 KiB/s rd, 741 KiB/s wr, 17 op/s 2023-03-09T22:33:03.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27293.748387 secs 2023-03-09T22:33:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27293.748387 secs 2023-03-09T22:33:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27293.748387 secs 2023-03-09T22:33:03.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27755.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:33:04.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:04 smithi122 ceph-mon[93869]: pgmap v15068: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 742 KiB/s rd, 777 KiB/s wr, 28 op/s 2023-03-09T22:33:04.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:04 smithi181 ceph-mon[94189]: pgmap v15068: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 742 KiB/s rd, 777 KiB/s wr, 28 op/s 2023-03-09T22:33:04.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:04 smithi038 ceph-mon[92422]: pgmap v15068: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 742 KiB/s rd, 777 KiB/s wr, 28 op/s 2023-03-09T22:33:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:05 smithi122 ceph-mon[93869]: osdmap e1157: 12 total, 12 up, 12 in 2023-03-09T22:33:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27296.377547 secs 2023-03-09T22:33:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:05 smithi181 ceph-mon[94189]: osdmap e1157: 12 total, 12 up, 12 in 2023-03-09T22:33:05.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27296.377547 secs 2023-03-09T22:33:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:05 smithi038 ceph-mon[92422]: osdmap e1157: 12 total, 12 up, 12 in 2023-03-09T22:33:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27296.377547 secs 2023-03-09T22:33:06.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:06 smithi122 ceph-mon[93869]: pgmap v15070: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 737 KiB/s rd, 779 KiB/s wr, 31 op/s 2023-03-09T22:33:06.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:06 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:33:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:06 smithi181 ceph-mon[94189]: pgmap v15070: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 737 KiB/s rd, 779 KiB/s wr, 31 op/s 2023-03-09T22:33:06.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:06 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:33:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:06 smithi038 ceph-mon[92422]: pgmap v15070: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 737 KiB/s rd, 779 KiB/s wr, 31 op/s 2023-03-09T22:33:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:06 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:33:07.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:07 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:33:07.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:07 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:33:07.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:07 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:33:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:07 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:33:07.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:07 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:33:07.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:07 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:33:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:07 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:33:07.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:07 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:33:07.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:07 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:33:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:08 smithi122 ceph-mon[93869]: pgmap v15071: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 787 KiB/s rd, 829 KiB/s wr, 30 op/s 2023-03-09T22:33:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27298.748468 secs 2023-03-09T22:33:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:33:08.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:33:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:08 smithi181 ceph-mon[94189]: pgmap v15071: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 787 KiB/s rd, 829 KiB/s wr, 30 op/s 2023-03-09T22:33:08.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27298.748468 secs 2023-03-09T22:33:08.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:33:08.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:33:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:08 smithi038 ceph-mon[92422]: pgmap v15071: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 787 KiB/s rd, 829 KiB/s wr, 30 op/s 2023-03-09T22:33:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27298.748468 secs 2023-03-09T22:33:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:33:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:33:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27760.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:33:09.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:33:09.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:33:09.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:09 smithi122 ceph-mon[93869]: pgmap v15072: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 575 KiB/s wr, 30 op/s 2023-03-09T22:33:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:33:09.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:33:09.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:09 smithi181 ceph-mon[94189]: pgmap v15072: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 575 KiB/s wr, 30 op/s 2023-03-09T22:33:09.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:33:09.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:33:09.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:09 smithi038 ceph-mon[92422]: pgmap v15072: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 575 KiB/s wr, 30 op/s 2023-03-09T22:33:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27301.377610 secs 2023-03-09T22:33:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27301.377610 secs 2023-03-09T22:33:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27301.377610 secs 2023-03-09T22:33:12.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:12 smithi122 ceph-mon[93869]: pgmap v15073: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 411 KiB/s wr, 25 op/s 2023-03-09T22:33:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:12 smithi181 ceph-mon[94189]: pgmap v15073: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 411 KiB/s wr, 25 op/s 2023-03-09T22:33:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:12 smithi038 ceph-mon[92422]: pgmap v15073: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 411 KiB/s wr, 25 op/s 2023-03-09T22:33:13.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27303.748609 secs 2023-03-09T22:33:13.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:33:13.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:33:13.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:13 smithi122 ceph-mon[93869]: pgmap v15074: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 341 KiB/s wr, 9 op/s 2023-03-09T22:33:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27303.748609 secs 2023-03-09T22:33:13.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:33:13.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:33:13.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:13 smithi181 ceph-mon[94189]: pgmap v15074: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 341 KiB/s wr, 9 op/s 2023-03-09T22:33:13.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27303.748609 secs 2023-03-09T22:33:13.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:33:13.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:33:13.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:13 smithi038 ceph-mon[92422]: pgmap v15074: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 341 KiB/s wr, 9 op/s 2023-03-09T22:33:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27765.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:33:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:15 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:33:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:15 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:33:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27306.377685 secs 2023-03-09T22:33:15.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:15 smithi122 ceph-mon[93869]: pgmap v15075: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 403 KiB/s wr, 9 op/s 2023-03-09T22:33:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:15 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:33:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:15 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:33:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27306.377685 secs 2023-03-09T22:33:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:15 smithi181 ceph-mon[94189]: pgmap v15075: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 403 KiB/s wr, 9 op/s 2023-03-09T22:33:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:15 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:33:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:15 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:33:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27306.377685 secs 2023-03-09T22:33:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:15 smithi038 ceph-mon[92422]: pgmap v15075: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 403 KiB/s wr, 9 op/s 2023-03-09T22:33:17.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:17 smithi122 ceph-mon[93869]: osdmap e1158: 12 total, 12 up, 12 in 2023-03-09T22:33:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:17 smithi181 ceph-mon[94189]: osdmap e1158: 12 total, 12 up, 12 in 2023-03-09T22:33:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:17 smithi038 ceph-mon[92422]: osdmap e1158: 12 total, 12 up, 12 in 2023-03-09T22:33:18.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:18 smithi122 ceph-mon[93869]: pgmap v15077: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 114 KiB/s wr, 9 op/s 2023-03-09T22:33:18.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27308.748745 secs 2023-03-09T22:33:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:18 smithi181 ceph-mon[94189]: pgmap v15077: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 114 KiB/s wr, 9 op/s 2023-03-09T22:33:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27308.748745 secs 2023-03-09T22:33:18.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:18 smithi038 ceph-mon[92422]: pgmap v15077: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 114 KiB/s wr, 9 op/s 2023-03-09T22:33:18.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27308.748745 secs 2023-03-09T22:33:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27770.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:33:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:20 smithi122 ceph-mon[93869]: pgmap v15078: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 95 KiB/s wr, 8 op/s 2023-03-09T22:33:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27311.377771 secs 2023-03-09T22:33:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:20 smithi181 ceph-mon[94189]: pgmap v15078: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 95 KiB/s wr, 8 op/s 2023-03-09T22:33:20.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27311.377771 secs 2023-03-09T22:33:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:20 smithi038 ceph-mon[92422]: pgmap v15078: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 95 KiB/s wr, 8 op/s 2023-03-09T22:33:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27311.377771 secs 2023-03-09T22:33:22.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:22 smithi122 ceph-mon[93869]: pgmap v15079: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 95 KiB/s wr, 8 op/s 2023-03-09T22:33:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:22 smithi181 ceph-mon[94189]: pgmap v15079: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 95 KiB/s wr, 8 op/s 2023-03-09T22:33:22.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:22 smithi038 ceph-mon[92422]: pgmap v15079: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 95 KiB/s wr, 8 op/s 2023-03-09T22:33:23.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27313.748889 secs 2023-03-09T22:33:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27313.748889 secs 2023-03-09T22:33:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27313.748889 secs 2023-03-09T22:33:23.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27775.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:33:24.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:24 smithi122 ceph-mon[93869]: pgmap v15080: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 95 KiB/s wr, 8 op/s 2023-03-09T22:33:24.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:24 smithi181 ceph-mon[94189]: pgmap v15080: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 95 KiB/s wr, 8 op/s 2023-03-09T22:33:24.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:24 smithi038 ceph-mon[92422]: pgmap v15080: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 95 KiB/s wr, 8 op/s 2023-03-09T22:33:25.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27316.377831 secs 2023-03-09T22:33:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27316.377831 secs 2023-03-09T22:33:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27316.377831 secs 2023-03-09T22:33:26.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:26 smithi122 ceph-mon[93869]: pgmap v15081: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 7 op/s 2023-03-09T22:33:26.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:26 smithi181 ceph-mon[94189]: pgmap v15081: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 7 op/s 2023-03-09T22:33:26.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:26 smithi038 ceph-mon[92422]: pgmap v15081: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 7 op/s 2023-03-09T22:33:27.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:27 smithi122 ceph-mon[93869]: pgmap v15082: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 7 op/s 2023-03-09T22:33:27.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:27 smithi181 ceph-mon[94189]: pgmap v15082: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 7 op/s 2023-03-09T22:33:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:27 smithi038 ceph-mon[92422]: pgmap v15082: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 7 op/s 2023-03-09T22:33:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27318.749008 secs 2023-03-09T22:33:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27318.749008 secs 2023-03-09T22:33:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27318.749008 secs 2023-03-09T22:33:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27780.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:33:29.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:29 smithi122 ceph-mon[93869]: pgmap v15083: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 7 op/s 2023-03-09T22:33:29.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:29 smithi181 ceph-mon[94189]: pgmap v15083: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 7 op/s 2023-03-09T22:33:29.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:29 smithi038 ceph-mon[92422]: pgmap v15083: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 7 op/s 2023-03-09T22:33:30.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27321.377892 secs 2023-03-09T22:33:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27321.377892 secs 2023-03-09T22:33:30.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27321.377892 secs 2023-03-09T22:33:32.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:32 smithi122 ceph-mon[93869]: pgmap v15084: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 9 op/s 2023-03-09T22:33:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:32 smithi181 ceph-mon[94189]: pgmap v15084: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 9 op/s 2023-03-09T22:33:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:32 smithi038 ceph-mon[92422]: pgmap v15084: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 9 op/s 2023-03-09T22:33:33.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27323.749132 secs 2023-03-09T22:33:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27323.749132 secs 2023-03-09T22:33:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27323.749132 secs 2023-03-09T22:33:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27785.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:33:34.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:34 smithi122 ceph-mon[93869]: pgmap v15085: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 7 op/s 2023-03-09T22:33:34.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:34 smithi181 ceph-mon[94189]: pgmap v15085: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 7 op/s 2023-03-09T22:33:34.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:34 smithi038 ceph-mon[92422]: pgmap v15085: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 7 op/s 2023-03-09T22:33:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27326.377954 secs 2023-03-09T22:33:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27326.377954 secs 2023-03-09T22:33:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27326.377954 secs 2023-03-09T22:33:36.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:36 smithi122 ceph-mon[93869]: pgmap v15086: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 7 op/s 2023-03-09T22:33:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:36 smithi181 ceph-mon[94189]: pgmap v15086: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 7 op/s 2023-03-09T22:33:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:36 smithi038 ceph-mon[92422]: pgmap v15086: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 7 op/s 2023-03-09T22:33:38.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:38 smithi122 ceph-mon[93869]: pgmap v15087: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 10 op/s 2023-03-09T22:33:38.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27328.749267 secs 2023-03-09T22:33:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:38 smithi181 ceph-mon[94189]: pgmap v15087: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 10 op/s 2023-03-09T22:33:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27328.749267 secs 2023-03-09T22:33:38.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:38 smithi038 ceph-mon[92422]: pgmap v15087: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 10 op/s 2023-03-09T22:33:38.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27328.749267 secs 2023-03-09T22:33:38.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27790.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:33:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:40 smithi122 ceph-mon[93869]: pgmap v15088: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 116 KiB/s wr, 8 op/s 2023-03-09T22:33:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27331.377984 secs 2023-03-09T22:33:40.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:40 smithi181 ceph-mon[94189]: pgmap v15088: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 116 KiB/s wr, 8 op/s 2023-03-09T22:33:40.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27331.377984 secs 2023-03-09T22:33:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:40 smithi038 ceph-mon[92422]: pgmap v15088: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 116 KiB/s wr, 8 op/s 2023-03-09T22:33:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27331.377984 secs 2023-03-09T22:33:42.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:42 smithi122 ceph-mon[93869]: pgmap v15089: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 116 KiB/s wr, 10 op/s 2023-03-09T22:33:42.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:42 smithi181 ceph-mon[94189]: pgmap v15089: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 116 KiB/s wr, 10 op/s 2023-03-09T22:33:42.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:42 smithi038 ceph-mon[92422]: pgmap v15089: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 116 KiB/s wr, 10 op/s 2023-03-09T22:33:43.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27795.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:33:44.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:44 smithi122 ceph-mon[93869]: pgmap v15090: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 116 KiB/s wr, 7 op/s 2023-03-09T22:33:44.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:44 smithi181 ceph-mon[94189]: pgmap v15090: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 116 KiB/s wr, 7 op/s 2023-03-09T22:33:44.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:44 smithi038 ceph-mon[92422]: pgmap v15090: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 116 KiB/s wr, 7 op/s 2023-03-09T22:33:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27336.378043 secs 2023-03-09T22:33:45.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:45 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:33:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27336.378043 secs 2023-03-09T22:33:45.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:45 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:33:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27336.378043 secs 2023-03-09T22:33:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:45 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:33:46.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:46 smithi122 ceph-mon[93869]: pgmap v15091: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 248 KiB/s wr, 7 op/s 2023-03-09T22:33:46.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:46 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:33:46.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:46 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:33:46.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:46 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:33:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:46 smithi181 ceph-mon[94189]: pgmap v15091: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 248 KiB/s wr, 7 op/s 2023-03-09T22:33:46.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:46 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:33:46.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:46 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:33:46.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:46 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:33:46.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:46 smithi038 ceph-mon[92422]: pgmap v15091: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 248 KiB/s wr, 7 op/s 2023-03-09T22:33:46.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:46 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:33:46.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:46 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:33:46.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:46 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:33:47.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:47 smithi122 ceph-mon[93869]: pgmap v15092: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 270 KiB/s wr, 10 op/s 2023-03-09T22:33:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:47 smithi181 ceph-mon[94189]: pgmap v15092: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 270 KiB/s wr, 10 op/s 2023-03-09T22:33:47.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:47 smithi038 ceph-mon[92422]: pgmap v15092: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 270 KiB/s wr, 10 op/s 2023-03-09T22:33:48.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27800.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:33:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:50 smithi038 ceph-mon[92422]: pgmap v15093: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 395 KiB/s wr, 8 op/s 2023-03-09T22:33:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27341.378105 secs 2023-03-09T22:33:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:50 smithi122 ceph-mon[93869]: pgmap v15093: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 395 KiB/s wr, 8 op/s 2023-03-09T22:33:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27341.378105 secs 2023-03-09T22:33:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:50 smithi181 ceph-mon[94189]: pgmap v15093: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 395 KiB/s wr, 8 op/s 2023-03-09T22:33:50.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27341.378105 secs 2023-03-09T22:33:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:52 smithi122 ceph-mon[93869]: pgmap v15094: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 301 KiB/s wr, 10 op/s 2023-03-09T22:33:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:33:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:33:52.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:52 smithi181 ceph-mon[94189]: pgmap v15094: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 301 KiB/s wr, 10 op/s 2023-03-09T22:33:52.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:33:52.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:33:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:52 smithi038 ceph-mon[92422]: pgmap v15094: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 301 KiB/s wr, 10 op/s 2023-03-09T22:33:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:33:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:33:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:33:53.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:33:53.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:33:53.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:33:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:33:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:33:53.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27805.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:33:54.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:54 smithi122 ceph-mon[93869]: pgmap v15095: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 302 KiB/s wr, 9 op/s 2023-03-09T22:33:54.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:54 smithi181 ceph-mon[94189]: pgmap v15095: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 302 KiB/s wr, 9 op/s 2023-03-09T22:33:54.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:54 smithi038 ceph-mon[92422]: pgmap v15095: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 302 KiB/s wr, 9 op/s 2023-03-09T22:33:55.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27346.378165 secs 2023-03-09T22:33:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27346.378165 secs 2023-03-09T22:33:55.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27346.378165 secs 2023-03-09T22:33:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:56 smithi122 ceph-mon[93869]: pgmap v15096: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 549 KiB/s rd, 518 KiB/s wr, 10 op/s 2023-03-09T22:33:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:33:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:33:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:56 smithi181 ceph-mon[94189]: pgmap v15096: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 549 KiB/s rd, 518 KiB/s wr, 10 op/s 2023-03-09T22:33:56.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:33:56.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:33:56.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:56 smithi038 ceph-mon[92422]: pgmap v15096: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 549 KiB/s rd, 518 KiB/s wr, 10 op/s 2023-03-09T22:33:56.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:33:56.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:33:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:33:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:33:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:33:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:33:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:33:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:33:58.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:58 smithi122 ceph-mon[93869]: pgmap v15097: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 770 KiB/s rd, 755 KiB/s wr, 14 op/s 2023-03-09T22:33:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:58 smithi181 ceph-mon[94189]: pgmap v15097: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 770 KiB/s rd, 755 KiB/s wr, 14 op/s 2023-03-09T22:33:58.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:58 smithi038 ceph-mon[92422]: pgmap v15097: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 770 KiB/s rd, 755 KiB/s wr, 14 op/s 2023-03-09T22:33:58.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27810.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:33:59.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:33:59 smithi122 ceph-mon[93869]: pgmap v15098: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 746 KiB/s rd, 733 KiB/s wr, 12 op/s 2023-03-09T22:33:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:33:59 smithi181 ceph-mon[94189]: pgmap v15098: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 746 KiB/s rd, 733 KiB/s wr, 12 op/s 2023-03-09T22:33:59.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:33:59 smithi038 ceph-mon[92422]: pgmap v15098: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 746 KiB/s rd, 733 KiB/s wr, 12 op/s 2023-03-09T22:34:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27351.378251 secs 2023-03-09T22:34:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27351.378251 secs 2023-03-09T22:34:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27351.378251 secs 2023-03-09T22:34:02.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:02 smithi122 ceph-mon[93869]: pgmap v15099: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 655 KiB/s rd, 653 KiB/s wr, 16 op/s 2023-03-09T22:34:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:02 smithi181 ceph-mon[94189]: pgmap v15099: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 655 KiB/s rd, 653 KiB/s wr, 16 op/s 2023-03-09T22:34:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:02 smithi038 ceph-mon[92422]: pgmap v15099: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 655 KiB/s rd, 653 KiB/s wr, 16 op/s 2023-03-09T22:34:03.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27353.749811 secs 2023-03-09T22:34:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27353.749811 secs 2023-03-09T22:34:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27353.749811 secs 2023-03-09T22:34:03.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27815.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:34:04.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:04 smithi122 ceph-mon[93869]: pgmap v15100: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 658 KiB/s rd, 684 KiB/s wr, 27 op/s 2023-03-09T22:34:04.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:04 smithi181 ceph-mon[94189]: pgmap v15100: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 658 KiB/s rd, 684 KiB/s wr, 27 op/s 2023-03-09T22:34:04.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:04 smithi038 ceph-mon[92422]: pgmap v15100: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 658 KiB/s rd, 684 KiB/s wr, 27 op/s 2023-03-09T22:34:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:05 smithi122 ceph-mon[93869]: osdmap e1159: 12 total, 12 up, 12 in 2023-03-09T22:34:05.750 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27356.378335 secs 2023-03-09T22:34:05.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:05 smithi038 ceph-mon[92422]: osdmap e1159: 12 total, 12 up, 12 in 2023-03-09T22:34:05.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27356.378335 secs 2023-03-09T22:34:05.751 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:05 smithi181 ceph-mon[94189]: osdmap e1159: 12 total, 12 up, 12 in 2023-03-09T22:34:05.751 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27356.378335 secs 2023-03-09T22:34:06.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:06 smithi122 ceph-mon[93869]: pgmap v15102: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 529 KiB/s rd, 560 KiB/s wr, 30 op/s 2023-03-09T22:34:06.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:06 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:34:06.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:06 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:34:06.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:34:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:06 smithi181 ceph-mon[94189]: pgmap v15102: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 529 KiB/s rd, 560 KiB/s wr, 30 op/s 2023-03-09T22:34:06.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:06 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:34:06.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:06 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:34:06.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:34:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:06 smithi038 ceph-mon[92422]: pgmap v15102: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 529 KiB/s rd, 560 KiB/s wr, 30 op/s 2023-03-09T22:34:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:06 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:34:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:06 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:34:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:34:07.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:07 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:34:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:07 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:34:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:07 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:34:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:08 smithi122 ceph-mon[93869]: pgmap v15103: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 460 KiB/s wr, 29 op/s 2023-03-09T22:34:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27358.749882 secs 2023-03-09T22:34:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:34:08.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:34:08.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:08 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:34:08.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:08 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:34:08.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:08 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:34:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:08 smithi181 ceph-mon[94189]: pgmap v15103: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 460 KiB/s wr, 29 op/s 2023-03-09T22:34:08.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27358.749882 secs 2023-03-09T22:34:08.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:34:08.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:34:08.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:08 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:34:08.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:08 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:34:08.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:08 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:34:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:08 smithi038 ceph-mon[92422]: pgmap v15103: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 460 KiB/s wr, 29 op/s 2023-03-09T22:34:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27358.749882 secs 2023-03-09T22:34:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:34:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:34:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:08 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:34:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:08 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:34:08.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:08 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:34:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27820.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:34:09.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:34:09.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:34:09.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:09 smithi122 ceph-mon[93869]: pgmap v15104: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 459 KiB/s wr, 29 op/s 2023-03-09T22:34:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:34:09.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:34:09.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:09 smithi181 ceph-mon[94189]: pgmap v15104: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 459 KiB/s wr, 29 op/s 2023-03-09T22:34:10.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:34:10.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:34:10.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:09 smithi038 ceph-mon[92422]: pgmap v15104: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 459 KiB/s wr, 29 op/s 2023-03-09T22:34:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27361.378420 secs 2023-03-09T22:34:10.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27361.378420 secs 2023-03-09T22:34:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27361.378420 secs 2023-03-09T22:34:12.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:12 smithi122 ceph-mon[93869]: pgmap v15105: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 448 KiB/s wr, 26 op/s 2023-03-09T22:34:12.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:12 smithi181 ceph-mon[94189]: pgmap v15105: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 448 KiB/s wr, 26 op/s 2023-03-09T22:34:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:12 smithi038 ceph-mon[92422]: pgmap v15105: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 448 KiB/s wr, 26 op/s 2023-03-09T22:34:13.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27363.750007 secs 2023-03-09T22:34:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27363.750007 secs 2023-03-09T22:34:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27363.750007 secs 2023-03-09T22:34:13.802 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27825.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:34:14.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:14 smithi122 ceph-mon[93869]: pgmap v15106: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 440 KiB/s rd, 438 KiB/s wr, 10 op/s 2023-03-09T22:34:14.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:14 smithi181 ceph-mon[94189]: pgmap v15106: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 440 KiB/s rd, 438 KiB/s wr, 10 op/s 2023-03-09T22:34:14.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:14 smithi038 ceph-mon[92422]: pgmap v15106: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 440 KiB/s rd, 438 KiB/s wr, 10 op/s 2023-03-09T22:34:15.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27366.378481 secs 2023-03-09T22:34:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27366.378481 secs 2023-03-09T22:34:15.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27366.378481 secs 2023-03-09T22:34:16.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:16 smithi122 ceph-mon[93869]: pgmap v15107: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 404 KiB/s wr, 9 op/s 2023-03-09T22:34:16.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:16 smithi122 ceph-mon[93869]: osdmap e1160: 12 total, 12 up, 12 in 2023-03-09T22:34:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:16 smithi181 ceph-mon[94189]: pgmap v15107: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 404 KiB/s wr, 9 op/s 2023-03-09T22:34:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:16 smithi181 ceph-mon[94189]: osdmap e1160: 12 total, 12 up, 12 in 2023-03-09T22:34:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:16 smithi038 ceph-mon[92422]: pgmap v15107: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 404 KiB/s wr, 9 op/s 2023-03-09T22:34:16.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:16 smithi038 ceph-mon[92422]: osdmap e1160: 12 total, 12 up, 12 in 2023-03-09T22:34:18.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:18 smithi122 ceph-mon[93869]: pgmap v15109: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 96 KiB/s wr, 8 op/s 2023-03-09T22:34:18.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27368.750127 secs 2023-03-09T22:34:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:18 smithi181 ceph-mon[94189]: pgmap v15109: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 96 KiB/s wr, 8 op/s 2023-03-09T22:34:18.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27368.750127 secs 2023-03-09T22:34:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:18 smithi038 ceph-mon[92422]: pgmap v15109: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 96 KiB/s wr, 8 op/s 2023-03-09T22:34:18.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27368.750127 secs 2023-03-09T22:34:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27830.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:34:19.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:19 smithi122 ceph-mon[93869]: pgmap v15110: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 96 KiB/s wr, 7 op/s 2023-03-09T22:34:19.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:19 smithi181 ceph-mon[94189]: pgmap v15110: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 96 KiB/s wr, 7 op/s 2023-03-09T22:34:19.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:19 smithi038 ceph-mon[92422]: pgmap v15110: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 96 KiB/s wr, 7 op/s 2023-03-09T22:34:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27371.378544 secs 2023-03-09T22:34:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27371.378544 secs 2023-03-09T22:34:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27371.378544 secs 2023-03-09T22:34:22.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:22 smithi122 ceph-mon[93869]: pgmap v15111: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 54 KiB/s wr, 7 op/s 2023-03-09T22:34:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:22 smithi181 ceph-mon[94189]: pgmap v15111: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 54 KiB/s wr, 7 op/s 2023-03-09T22:34:22.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:22 smithi038 ceph-mon[92422]: pgmap v15111: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 54 KiB/s wr, 7 op/s 2023-03-09T22:34:23.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27373.750211 secs 2023-03-09T22:34:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27373.750211 secs 2023-03-09T22:34:23.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27373.750211 secs 2023-03-09T22:34:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27835.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:34:24.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:24 smithi122 ceph-mon[93869]: pgmap v15112: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-03-09T22:34:24.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:24 smithi181 ceph-mon[94189]: pgmap v15112: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-03-09T22:34:24.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:24 smithi038 ceph-mon[92422]: pgmap v15112: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-03-09T22:34:25.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27376.378604 secs 2023-03-09T22:34:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27376.378604 secs 2023-03-09T22:34:25.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27376.378604 secs 2023-03-09T22:34:26.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:26 smithi122 ceph-mon[93869]: pgmap v15113: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-03-09T22:34:26.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:26 smithi181 ceph-mon[94189]: pgmap v15113: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-03-09T22:34:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:26 smithi038 ceph-mon[92422]: pgmap v15113: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-03-09T22:34:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:28 smithi122 ceph-mon[93869]: pgmap v15114: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2023-03-09T22:34:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27378.750284 secs 2023-03-09T22:34:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:28 smithi181 ceph-mon[94189]: pgmap v15114: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2023-03-09T22:34:28.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27378.750284 secs 2023-03-09T22:34:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:28 smithi038 ceph-mon[92422]: pgmap v15114: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2023-03-09T22:34:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27378.750284 secs 2023-03-09T22:34:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27840.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:34:29.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:29 smithi122 ceph-mon[93869]: pgmap v15115: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T22:34:29.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:29 smithi181 ceph-mon[94189]: pgmap v15115: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T22:34:29.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:29 smithi038 ceph-mon[92422]: pgmap v15115: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T22:34:30.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27381.378663 secs 2023-03-09T22:34:30.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27381.378663 secs 2023-03-09T22:34:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27381.378663 secs 2023-03-09T22:34:32.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:32 smithi122 ceph-mon[93869]: pgmap v15116: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T22:34:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:32 smithi181 ceph-mon[94189]: pgmap v15116: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T22:34:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:32 smithi038 ceph-mon[92422]: pgmap v15116: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T22:34:33.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27383.750416 secs 2023-03-09T22:34:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27383.750416 secs 2023-03-09T22:34:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27383.750416 secs 2023-03-09T22:34:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27845.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:34:34.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:34 smithi122 ceph-mon[93869]: pgmap v15117: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T22:34:34.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:34 smithi181 ceph-mon[94189]: pgmap v15117: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T22:34:34.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:34 smithi038 ceph-mon[92422]: pgmap v15117: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T22:34:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27386.378718 secs 2023-03-09T22:34:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27386.378718 secs 2023-03-09T22:34:35.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27386.378718 secs 2023-03-09T22:34:36.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:36 smithi122 ceph-mon[93869]: pgmap v15118: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T22:34:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:36 smithi181 ceph-mon[94189]: pgmap v15118: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T22:34:36.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:36 smithi038 ceph-mon[92422]: pgmap v15118: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T22:34:37.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:34:37.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:34:37.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:37 smithi122 ceph-mon[93869]: pgmap v15119: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 96 KiB/s wr, 9 op/s 2023-03-09T22:34:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:34:37.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:34:37.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:37 smithi181 ceph-mon[94189]: pgmap v15119: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 96 KiB/s wr, 9 op/s 2023-03-09T22:34:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:34:37.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:34:37.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:37 smithi038 ceph-mon[92422]: pgmap v15119: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 96 KiB/s wr, 9 op/s 2023-03-09T22:34:38.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27388.750532 secs 2023-03-09T22:34:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27388.750532 secs 2023-03-09T22:34:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27388.750532 secs 2023-03-09T22:34:38.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27850.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:34:39.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:39 smithi122 ceph-mon[93869]: pgmap v15120: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 96 KiB/s wr, 6 op/s 2023-03-09T22:34:39.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:39 smithi181 ceph-mon[94189]: pgmap v15120: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 96 KiB/s wr, 6 op/s 2023-03-09T22:34:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:39 smithi038 ceph-mon[92422]: pgmap v15120: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 96 KiB/s wr, 6 op/s 2023-03-09T22:34:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:34:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:34:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27391.378778 secs 2023-03-09T22:34:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:34:40.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:34:40.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27391.378778 secs 2023-03-09T22:34:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:34:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:34:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27391.378778 secs 2023-03-09T22:34:42.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:42 smithi122 ceph-mon[93869]: pgmap v15121: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 241 KiB/s wr, 9 op/s 2023-03-09T22:34:42.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:42 smithi181 ceph-mon[94189]: pgmap v15121: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 241 KiB/s wr, 9 op/s 2023-03-09T22:34:42.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:42 smithi038 ceph-mon[92422]: pgmap v15121: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 241 KiB/s wr, 9 op/s 2023-03-09T22:34:43.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27855.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:34:44.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:44 smithi122 ceph-mon[93869]: pgmap v15122: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 241 KiB/s wr, 7 op/s 2023-03-09T22:34:44.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:44 smithi181 ceph-mon[94189]: pgmap v15122: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 241 KiB/s wr, 7 op/s 2023-03-09T22:34:44.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:44 smithi038 ceph-mon[92422]: pgmap v15122: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 241 KiB/s wr, 7 op/s 2023-03-09T22:34:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27396.378837 secs 2023-03-09T22:34:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:45 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:34:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27396.378837 secs 2023-03-09T22:34:45.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:45 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:34:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27396.378837 secs 2023-03-09T22:34:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:45 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:34:46.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:46 smithi122 ceph-mon[93869]: pgmap v15123: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 241 KiB/s wr, 6 op/s 2023-03-09T22:34:46.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:46 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:34:46.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:46 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:34:46.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:46 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:34:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:46 smithi181 ceph-mon[94189]: pgmap v15123: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 241 KiB/s wr, 6 op/s 2023-03-09T22:34:46.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:46 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:34:46.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:46 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:34:46.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:46 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:34:46.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:46 smithi038 ceph-mon[92422]: pgmap v15123: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 241 KiB/s wr, 6 op/s 2023-03-09T22:34:46.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:46 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:34:46.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:46 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:34:46.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:46 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:34:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:48 smithi122 ceph-mon[93869]: pgmap v15124: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 579 KiB/s rd, 563 KiB/s wr, 12 op/s 2023-03-09T22:34:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:48 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 27398.750790 secs 2023-03-09T22:34:48.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:48 smithi181 ceph-mon[94189]: pgmap v15124: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 579 KiB/s rd, 563 KiB/s wr, 12 op/s 2023-03-09T22:34:48.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:48 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 27398.750790 secs 2023-03-09T22:34:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:48 smithi038 ceph-mon[92422]: pgmap v15124: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 579 KiB/s rd, 563 KiB/s wr, 12 op/s 2023-03-09T22:34:48.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:48 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 27398.750790 secs 2023-03-09T22:34:48.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27860.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:34:49.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:34:49.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:34:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:34:49.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:34:49.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:34:49.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:34:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:50 smithi122 ceph-mon[93869]: pgmap v15125: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 467 KiB/s wr, 9 op/s 2023-03-09T22:34:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:50 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:34:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:50 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:34:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27401.378896 secs 2023-03-09T22:34:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:50 smithi181 ceph-mon[94189]: pgmap v15125: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 467 KiB/s wr, 9 op/s 2023-03-09T22:34:50.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:50 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:34:50.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:50 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:34:50.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27401.378896 secs 2023-03-09T22:34:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:50 smithi038 ceph-mon[92422]: pgmap v15125: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 467 KiB/s wr, 9 op/s 2023-03-09T22:34:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:50 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:34:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:50 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:34:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27401.378896 secs 2023-03-09T22:34:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:51 smithi122 ceph-mon[93869]: pgmap v15126: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 614 KiB/s rd, 599 KiB/s wr, 12 op/s 2023-03-09T22:34:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:51 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:34:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:51 smithi181 ceph-mon[94189]: pgmap v15126: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 614 KiB/s rd, 599 KiB/s wr, 12 op/s 2023-03-09T22:34:51.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:51 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:34:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:51 smithi038 ceph-mon[92422]: pgmap v15126: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 614 KiB/s rd, 599 KiB/s wr, 12 op/s 2023-03-09T22:34:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:51 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:34:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:34:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:34:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:34:52.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:34:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:34:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:34:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:53 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 27403.750924 secs 2023-03-09T22:34:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:53 smithi122 ceph-mon[93869]: pgmap v15127: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 454 KiB/s wr, 10 op/s 2023-03-09T22:34:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:53 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 27403.750924 secs 2023-03-09T22:34:53.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:53 smithi181 ceph-mon[94189]: pgmap v15127: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 454 KiB/s wr, 10 op/s 2023-03-09T22:34:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:53 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 27403.750924 secs 2023-03-09T22:34:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:53 smithi038 ceph-mon[92422]: pgmap v15127: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 454 KiB/s wr, 10 op/s 2023-03-09T22:34:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27865.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:34:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:34:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:34:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27406.378957 secs 2023-03-09T22:34:55.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:55 smithi122 ceph-mon[93869]: pgmap v15128: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 454 KiB/s wr, 9 op/s 2023-03-09T22:34:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:34:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:34:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27406.378957 secs 2023-03-09T22:34:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:55 smithi181 ceph-mon[94189]: pgmap v15128: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 454 KiB/s wr, 9 op/s 2023-03-09T22:34:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:34:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:34:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27406.378957 secs 2023-03-09T22:34:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:55 smithi038 ceph-mon[92422]: pgmap v15128: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 454 KiB/s wr, 9 op/s 2023-03-09T22:34:57.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:34:57.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:34:57.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:57 smithi122 ceph-mon[93869]: pgmap v15129: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 822 KiB/s wr, 14 op/s 2023-03-09T22:34:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:34:57.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:34:57.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:57 smithi181 ceph-mon[94189]: pgmap v15129: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 822 KiB/s wr, 14 op/s 2023-03-09T22:34:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:34:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:34:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:57 smithi038 ceph-mon[92422]: pgmap v15129: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 822 KiB/s wr, 14 op/s 2023-03-09T22:34:58.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27870.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:34:58.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:58 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 27408.751046 secs 2023-03-09T22:34:58.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:58 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 27408.751046 secs 2023-03-09T22:34:59.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:58 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 27408.751046 secs 2023-03-09T22:34:59.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:34:59 smithi122 ceph-mon[93869]: pgmap v15130: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 500 KiB/s wr, 10 op/s 2023-03-09T22:34:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:34:59 smithi181 ceph-mon[94189]: pgmap v15130: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 500 KiB/s wr, 10 op/s 2023-03-09T22:35:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:34:59 smithi038 ceph-mon[92422]: pgmap v15130: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 500 KiB/s wr, 10 op/s 2023-03-09T22:35:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27411.378981 secs 2023-03-09T22:35:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27411.378981 secs 2023-03-09T22:35:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27411.378981 secs 2023-03-09T22:35:01.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:35:01.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:35:01.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:01 smithi122 ceph-mon[93869]: pgmap v15131: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 886 KiB/s wr, 15 op/s 2023-03-09T22:35:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:35:01.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:35:01.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:01 smithi181 ceph-mon[94189]: pgmap v15131: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 886 KiB/s wr, 15 op/s 2023-03-09T22:35:02.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:35:02.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:35:02.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:01 smithi038 ceph-mon[92422]: pgmap v15131: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 886 KiB/s wr, 15 op/s 2023-03-09T22:35:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27875.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:35:04.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:04 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27413.751163 secs 2023-03-09T22:35:04.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:04 smithi122 ceph-mon[93869]: pgmap v15132: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 872 KiB/s wr, 26 op/s 2023-03-09T22:35:04.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:04 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27413.751163 secs 2023-03-09T22:35:04.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:04 smithi181 ceph-mon[94189]: pgmap v15132: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 872 KiB/s wr, 26 op/s 2023-03-09T22:35:04.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:04 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27413.751163 secs 2023-03-09T22:35:04.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:04 smithi038 ceph-mon[92422]: pgmap v15132: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 872 KiB/s wr, 26 op/s 2023-03-09T22:35:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27416.379044 secs 2023-03-09T22:35:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27416.379044 secs 2023-03-09T22:35:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27416.379044 secs 2023-03-09T22:35:06.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:06 smithi122 ceph-mon[93869]: pgmap v15133: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 872 KiB/s wr, 25 op/s 2023-03-09T22:35:06.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:06 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:35:06.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:35:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:06 smithi181 ceph-mon[94189]: pgmap v15133: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 872 KiB/s wr, 25 op/s 2023-03-09T22:35:06.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:06 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:35:06.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:35:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:06 smithi038 ceph-mon[92422]: pgmap v15133: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 872 KiB/s wr, 25 op/s 2023-03-09T22:35:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:06 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:35:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:35:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:08 smithi122 ceph-mon[93869]: pgmap v15134: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 873 KiB/s wr, 30 op/s 2023-03-09T22:35:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27418.751288 secs 2023-03-09T22:35:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:35:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:35:08.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:08 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:35:08.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:08 smithi181 ceph-mon[94189]: pgmap v15134: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 873 KiB/s wr, 30 op/s 2023-03-09T22:35:08.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27418.751288 secs 2023-03-09T22:35:08.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:35:08.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:35:08.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:08 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:35:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:08 smithi038 ceph-mon[92422]: pgmap v15134: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 873 KiB/s wr, 30 op/s 2023-03-09T22:35:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27418.751288 secs 2023-03-09T22:35:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:35:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:35:08.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:08 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:35:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27880.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:35:09.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:35:09.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:35:09.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:09 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:35:09.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:09 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:35:09.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:09 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:35:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:35:09.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:35:09.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:09 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:35:09.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:09 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:35:09.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:09 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:35:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:35:09.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:35:09.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:09 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:35:09.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:09 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:35:09.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:09 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:35:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:10 smithi122 ceph-mon[93869]: pgmap v15135: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 550 KiB/s rd, 737 KiB/s wr, 26 op/s 2023-03-09T22:35:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:10 smithi122 ceph-mon[93869]: osdmap e1161: 12 total, 12 up, 12 in 2023-03-09T22:35:10.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27421.379104 secs 2023-03-09T22:35:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:10 smithi181 ceph-mon[94189]: pgmap v15135: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 550 KiB/s rd, 737 KiB/s wr, 26 op/s 2023-03-09T22:35:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:10 smithi181 ceph-mon[94189]: osdmap e1161: 12 total, 12 up, 12 in 2023-03-09T22:35:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27421.379104 secs 2023-03-09T22:35:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:10 smithi038 ceph-mon[92422]: pgmap v15135: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 550 KiB/s rd, 737 KiB/s wr, 26 op/s 2023-03-09T22:35:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:10 smithi038 ceph-mon[92422]: osdmap e1161: 12 total, 12 up, 12 in 2023-03-09T22:35:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27421.379104 secs 2023-03-09T22:35:12.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:12 smithi122 ceph-mon[93869]: pgmap v15137: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 545 KiB/s wr, 26 op/s 2023-03-09T22:35:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:12 smithi181 ceph-mon[94189]: pgmap v15137: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 545 KiB/s wr, 26 op/s 2023-03-09T22:35:12.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:12 smithi038 ceph-mon[92422]: pgmap v15137: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 545 KiB/s wr, 26 op/s 2023-03-09T22:35:13.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27423.751405 secs 2023-03-09T22:35:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27423.751405 secs 2023-03-09T22:35:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27423.751405 secs 2023-03-09T22:35:13.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27885.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:35:14.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:14 smithi122 ceph-mon[93869]: pgmap v15138: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 419 KiB/s wr, 10 op/s 2023-03-09T22:35:14.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:14 smithi181 ceph-mon[94189]: pgmap v15138: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 419 KiB/s wr, 10 op/s 2023-03-09T22:35:14.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:14 smithi038 ceph-mon[92422]: pgmap v15138: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 419 KiB/s wr, 10 op/s 2023-03-09T22:35:15.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27426.379185 secs 2023-03-09T22:35:15.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:15 smithi122 ceph-mon[93869]: pgmap v15139: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 419 KiB/s wr, 10 op/s 2023-03-09T22:35:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27426.379185 secs 2023-03-09T22:35:15.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:15 smithi181 ceph-mon[94189]: pgmap v15139: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 419 KiB/s wr, 10 op/s 2023-03-09T22:35:15.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27426.379185 secs 2023-03-09T22:35:15.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:15 smithi038 ceph-mon[92422]: pgmap v15139: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 419 KiB/s wr, 10 op/s 2023-03-09T22:35:17.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:17 smithi122 ceph-mon[93869]: osdmap e1162: 12 total, 12 up, 12 in 2023-03-09T22:35:17.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:35:17.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:35:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:17 smithi181 ceph-mon[94189]: osdmap e1162: 12 total, 12 up, 12 in 2023-03-09T22:35:17.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:35:17.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:35:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:17 smithi038 ceph-mon[92422]: osdmap e1162: 12 total, 12 up, 12 in 2023-03-09T22:35:17.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:35:17.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:35:18.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:18 smithi122 ceph-mon[93869]: pgmap v15141: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 527 KiB/s rd, 295 KiB/s wr, 10 op/s 2023-03-09T22:35:18.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27428.751519 secs 2023-03-09T22:35:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:18 smithi181 ceph-mon[94189]: pgmap v15141: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 527 KiB/s rd, 295 KiB/s wr, 10 op/s 2023-03-09T22:35:18.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27428.751519 secs 2023-03-09T22:35:18.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:18 smithi038 ceph-mon[92422]: pgmap v15141: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 527 KiB/s rd, 295 KiB/s wr, 10 op/s 2023-03-09T22:35:18.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27428.751519 secs 2023-03-09T22:35:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27890.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:35:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:20 smithi122 ceph-mon[93869]: pgmap v15142: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 240 KiB/s wr, 8 op/s 2023-03-09T22:35:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:35:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:35:20.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27431.379247 secs 2023-03-09T22:35:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:20 smithi181 ceph-mon[94189]: pgmap v15142: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 240 KiB/s wr, 8 op/s 2023-03-09T22:35:20.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:35:20.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:35:20.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27431.379247 secs 2023-03-09T22:35:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:20 smithi038 ceph-mon[92422]: pgmap v15142: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 240 KiB/s wr, 8 op/s 2023-03-09T22:35:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:35:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:35:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27431.379247 secs 2023-03-09T22:35:22.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:22 smithi122 ceph-mon[93869]: pgmap v15143: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 245 KiB/s wr, 8 op/s 2023-03-09T22:35:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:22 smithi181 ceph-mon[94189]: pgmap v15143: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 245 KiB/s wr, 8 op/s 2023-03-09T22:35:22.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:22 smithi038 ceph-mon[92422]: pgmap v15143: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 245 KiB/s wr, 8 op/s 2023-03-09T22:35:23.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27433.751582 secs 2023-03-09T22:35:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27433.751582 secs 2023-03-09T22:35:23.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27433.751582 secs 2023-03-09T22:35:23.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27895.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:35:24.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:24 smithi122 ceph-mon[93869]: pgmap v15144: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 229 KiB/s wr, 7 op/s 2023-03-09T22:35:24.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:24 smithi181 ceph-mon[94189]: pgmap v15144: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 229 KiB/s wr, 7 op/s 2023-03-09T22:35:24.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:24 smithi038 ceph-mon[92422]: pgmap v15144: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 229 KiB/s wr, 7 op/s 2023-03-09T22:35:25.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27436.379307 secs 2023-03-09T22:35:25.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:25 smithi122 ceph-mon[93869]: pgmap v15145: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 229 KiB/s wr, 7 op/s 2023-03-09T22:35:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27436.379307 secs 2023-03-09T22:35:25.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:25 smithi181 ceph-mon[94189]: pgmap v15145: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 229 KiB/s wr, 7 op/s 2023-03-09T22:35:25.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27436.379307 secs 2023-03-09T22:35:25.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:25 smithi038 ceph-mon[92422]: pgmap v15145: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 229 KiB/s wr, 7 op/s 2023-03-09T22:35:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:28 smithi122 ceph-mon[93869]: pgmap v15146: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 122 KiB/s wr, 6 op/s 2023-03-09T22:35:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27438.751670 secs 2023-03-09T22:35:28.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:28 smithi181 ceph-mon[94189]: pgmap v15146: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 122 KiB/s wr, 6 op/s 2023-03-09T22:35:28.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27438.751670 secs 2023-03-09T22:35:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:28 smithi038 ceph-mon[92422]: pgmap v15146: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 122 KiB/s wr, 6 op/s 2023-03-09T22:35:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27438.751670 secs 2023-03-09T22:35:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27900.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:35:30.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:30 smithi122 ceph-mon[93869]: pgmap v15147: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 110 KiB/s wr, 6 op/s 2023-03-09T22:35:30.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27441.379370 secs 2023-03-09T22:35:30.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:30 smithi181 ceph-mon[94189]: pgmap v15147: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 110 KiB/s wr, 6 op/s 2023-03-09T22:35:30.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27441.379370 secs 2023-03-09T22:35:30.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:30 smithi038 ceph-mon[92422]: pgmap v15147: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 110 KiB/s wr, 6 op/s 2023-03-09T22:35:30.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27441.379370 secs 2023-03-09T22:35:32.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:32 smithi122 ceph-mon[93869]: pgmap v15148: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 110 KiB/s wr, 8 op/s 2023-03-09T22:35:32.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:32 smithi181 ceph-mon[94189]: pgmap v15148: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 110 KiB/s wr, 8 op/s 2023-03-09T22:35:32.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:32 smithi038 ceph-mon[92422]: pgmap v15148: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 110 KiB/s wr, 8 op/s 2023-03-09T22:35:33.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27443.751789 secs 2023-03-09T22:35:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27443.751789 secs 2023-03-09T22:35:33.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27443.751789 secs 2023-03-09T22:35:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27905.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:35:34.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:34 smithi122 ceph-mon[93869]: pgmap v15149: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2023-03-09T22:35:34.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:34 smithi181 ceph-mon[94189]: pgmap v15149: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2023-03-09T22:35:34.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:34 smithi038 ceph-mon[92422]: pgmap v15149: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2023-03-09T22:35:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27446.379429 secs 2023-03-09T22:35:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:35 smithi122 ceph-mon[93869]: pgmap v15150: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2023-03-09T22:35:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27446.379429 secs 2023-03-09T22:35:35.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:35 smithi181 ceph-mon[94189]: pgmap v15150: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2023-03-09T22:35:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27446.379429 secs 2023-03-09T22:35:35.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:35 smithi038 ceph-mon[92422]: pgmap v15150: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 5 op/s 2023-03-09T22:35:38.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:38 smithi122 ceph-mon[93869]: pgmap v15151: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 597 B/s wr, 8 op/s 2023-03-09T22:35:38.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27448.751901 secs 2023-03-09T22:35:38.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:38 smithi181 ceph-mon[94189]: pgmap v15151: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 597 B/s wr, 8 op/s 2023-03-09T22:35:38.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27448.751901 secs 2023-03-09T22:35:38.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:38 smithi038 ceph-mon[92422]: pgmap v15151: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 597 B/s wr, 8 op/s 2023-03-09T22:35:38.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27448.751901 secs 2023-03-09T22:35:38.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27910.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:35:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:40 smithi122 ceph-mon[93869]: pgmap v15152: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T22:35:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27451.379492 secs 2023-03-09T22:35:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:40 smithi181 ceph-mon[94189]: pgmap v15152: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T22:35:40.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27451.379492 secs 2023-03-09T22:35:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:40 smithi038 ceph-mon[92422]: pgmap v15152: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T22:35:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27451.379492 secs 2023-03-09T22:35:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:35:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:35:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:41 smithi122 ceph-mon[93869]: pgmap v15153: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 69 KiB/s wr, 8 op/s 2023-03-09T22:35:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:35:41.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:35:41.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:41 smithi181 ceph-mon[94189]: pgmap v15153: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 69 KiB/s wr, 8 op/s 2023-03-09T22:35:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:35:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:35:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:41 smithi038 ceph-mon[92422]: pgmap v15153: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 69 KiB/s wr, 8 op/s 2023-03-09T22:35:43.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27915.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:35:44.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:44 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27453.752014 secs 2023-03-09T22:35:44.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:44 smithi122 ceph-mon[93869]: pgmap v15154: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 246 KiB/s wr, 6 op/s 2023-03-09T22:35:44.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:44 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27453.752014 secs 2023-03-09T22:35:44.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:44 smithi181 ceph-mon[94189]: pgmap v15154: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 246 KiB/s wr, 6 op/s 2023-03-09T22:35:44.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:44 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27453.752014 secs 2023-03-09T22:35:44.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:44 smithi038 ceph-mon[92422]: pgmap v15154: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 246 KiB/s wr, 6 op/s 2023-03-09T22:35:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27456.379553 secs 2023-03-09T22:35:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27456.379553 secs 2023-03-09T22:35:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27456.379553 secs 2023-03-09T22:35:46.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:46 smithi122 ceph-mon[93869]: pgmap v15155: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 246 KiB/s wr, 6 op/s 2023-03-09T22:35:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:46 smithi181 ceph-mon[94189]: pgmap v15155: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 246 KiB/s wr, 6 op/s 2023-03-09T22:35:46.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:46 smithi038 ceph-mon[92422]: pgmap v15155: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 246 KiB/s wr, 6 op/s 2023-03-09T22:35:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:48 smithi122 ceph-mon[93869]: pgmap v15156: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 546 KiB/s rd, 543 KiB/s wr, 11 op/s 2023-03-09T22:35:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27458.752132 secs 2023-03-09T22:35:48.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:35:48.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:35:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:48 smithi181 ceph-mon[94189]: pgmap v15156: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 546 KiB/s rd, 543 KiB/s wr, 11 op/s 2023-03-09T22:35:48.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27458.752132 secs 2023-03-09T22:35:48.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:35:48.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:35:48.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:48 smithi038 ceph-mon[92422]: pgmap v15156: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 546 KiB/s rd, 543 KiB/s wr, 11 op/s 2023-03-09T22:35:48.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27458.752132 secs 2023-03-09T22:35:48.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:35:48.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:35:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27920.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:35:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:50 smithi122 ceph-mon[93869]: pgmap v15157: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 555 KiB/s rd, 542 KiB/s wr, 8 op/s 2023-03-09T22:35:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27461.379612 secs 2023-03-09T22:35:50.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:50 smithi181 ceph-mon[94189]: pgmap v15157: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 555 KiB/s rd, 542 KiB/s wr, 8 op/s 2023-03-09T22:35:50.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27461.379612 secs 2023-03-09T22:35:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:50 smithi038 ceph-mon[92422]: pgmap v15157: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 555 KiB/s rd, 542 KiB/s wr, 8 op/s 2023-03-09T22:35:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27461.379612 secs 2023-03-09T22:35:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:52 smithi122 ceph-mon[93869]: pgmap v15158: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 974 KiB/s rd, 858 KiB/s wr, 13 op/s 2023-03-09T22:35:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:35:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:35:52.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:52 smithi181 ceph-mon[94189]: pgmap v15158: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 974 KiB/s rd, 858 KiB/s wr, 13 op/s 2023-03-09T22:35:52.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:35:52.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:35:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:52 smithi038 ceph-mon[92422]: pgmap v15158: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 974 KiB/s rd, 858 KiB/s wr, 13 op/s 2023-03-09T22:35:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:35:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:35:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:35:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:35:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:53 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:35:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:53 smithi122 ceph-mon[93869]: pgmap v15159: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 905 KiB/s rd, 791 KiB/s wr, 12 op/s 2023-03-09T22:35:53.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:35:53.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:35:53.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:53 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:35:53.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:53 smithi181 ceph-mon[94189]: pgmap v15159: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 905 KiB/s rd, 791 KiB/s wr, 12 op/s 2023-03-09T22:35:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:35:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:35:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:53 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:35:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:53 smithi038 ceph-mon[92422]: pgmap v15159: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 905 KiB/s rd, 791 KiB/s wr, 12 op/s 2023-03-09T22:35:53.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27925.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:35:54.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:54 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:35:54.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:54 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:35:54.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:54 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:35:54.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:54 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:35:54.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:54 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:35:54.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:54 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:35:54.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:54 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:35:54.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:54 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:35:54.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:54 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:35:55.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27466.379672 secs 2023-03-09T22:35:55.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:55 smithi122 ceph-mon[93869]: pgmap v15160: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 728 KiB/s rd, 614 KiB/s wr, 11 op/s 2023-03-09T22:35:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27466.379672 secs 2023-03-09T22:35:55.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:55 smithi181 ceph-mon[94189]: pgmap v15160: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 728 KiB/s rd, 614 KiB/s wr, 11 op/s 2023-03-09T22:35:55.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27466.379672 secs 2023-03-09T22:35:55.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:55 smithi038 ceph-mon[92422]: pgmap v15160: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 728 KiB/s rd, 614 KiB/s wr, 11 op/s 2023-03-09T22:35:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:35:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:35:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:35:56.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:35:56.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:35:56.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:35:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:35:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:35:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:35:57 smithi122 ceph-mon[93869]: pgmap v15161: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 912 KiB/s wr, 16 op/s 2023-03-09T22:35:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:35:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:35:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:35:57 smithi181 ceph-mon[94189]: pgmap v15161: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 912 KiB/s wr, 16 op/s 2023-03-09T22:35:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:35:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:35:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:35:57 smithi038 ceph-mon[92422]: pgmap v15161: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 912 KiB/s wr, 16 op/s 2023-03-09T22:35:58.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27930.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:36:00.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:00 smithi122 ceph-mon[93869]: pgmap v15162: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 776 KiB/s rd, 643 KiB/s wr, 12 op/s 2023-03-09T22:36:00.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27471.379730 secs 2023-03-09T22:36:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:00 smithi181 ceph-mon[94189]: pgmap v15162: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 776 KiB/s rd, 643 KiB/s wr, 12 op/s 2023-03-09T22:36:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27471.379730 secs 2023-03-09T22:36:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:00 smithi038 ceph-mon[92422]: pgmap v15162: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 776 KiB/s rd, 643 KiB/s wr, 12 op/s 2023-03-09T22:36:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27471.379730 secs 2023-03-09T22:36:02.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:02 smithi122 ceph-mon[93869]: pgmap v15163: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 835 KiB/s rd, 654 KiB/s wr, 16 op/s 2023-03-09T22:36:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:02 smithi181 ceph-mon[94189]: pgmap v15163: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 835 KiB/s rd, 654 KiB/s wr, 16 op/s 2023-03-09T22:36:02.759 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:02 smithi038 ceph-mon[92422]: pgmap v15163: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 835 KiB/s rd, 654 KiB/s wr, 16 op/s 2023-03-09T22:36:03.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27473.752444 secs 2023-03-09T22:36:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27473.752444 secs 2023-03-09T22:36:03.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27473.752444 secs 2023-03-09T22:36:03.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27935.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:36:04.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:04 smithi122 ceph-mon[93869]: pgmap v15164: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 623 KiB/s rd, 559 KiB/s wr, 25 op/s 2023-03-09T22:36:04.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:04 smithi181 ceph-mon[94189]: pgmap v15164: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 623 KiB/s rd, 559 KiB/s wr, 25 op/s 2023-03-09T22:36:04.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:04 smithi038 ceph-mon[92422]: pgmap v15164: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 623 KiB/s rd, 559 KiB/s wr, 25 op/s 2023-03-09T22:36:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:05 smithi122 ceph-mon[93869]: osdmap e1163: 12 total, 12 up, 12 in 2023-03-09T22:36:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27476.379790 secs 2023-03-09T22:36:05.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:05 smithi122 ceph-mon[93869]: pgmap v15166: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 748 KiB/s rd, 669 KiB/s wr, 29 op/s 2023-03-09T22:36:05.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:05 smithi181 ceph-mon[94189]: osdmap e1163: 12 total, 12 up, 12 in 2023-03-09T22:36:05.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27476.379790 secs 2023-03-09T22:36:05.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:05 smithi181 ceph-mon[94189]: pgmap v15166: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 748 KiB/s rd, 669 KiB/s wr, 29 op/s 2023-03-09T22:36:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:05 smithi038 ceph-mon[92422]: osdmap e1163: 12 total, 12 up, 12 in 2023-03-09T22:36:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27476.379790 secs 2023-03-09T22:36:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:05 smithi038 ceph-mon[92422]: pgmap v15166: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 748 KiB/s rd, 669 KiB/s wr, 29 op/s 2023-03-09T22:36:06.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:06 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:36:06.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:06 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:36:06.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:36:06.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:06 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:36:06.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:06 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:36:06.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:36:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:06 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:36:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:06 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:36:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:36:07.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:07 smithi122 ceph-mon[93869]: pgmap v15167: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 611 KiB/s rd, 568 KiB/s wr, 29 op/s 2023-03-09T22:36:07.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:07 smithi181 ceph-mon[94189]: pgmap v15167: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 611 KiB/s rd, 568 KiB/s wr, 29 op/s 2023-03-09T22:36:07.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:07 smithi038 ceph-mon[92422]: pgmap v15167: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 611 KiB/s rd, 568 KiB/s wr, 29 op/s 2023-03-09T22:36:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27478.752562 secs 2023-03-09T22:36:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:36:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:36:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27478.752562 secs 2023-03-09T22:36:08.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:36:08.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:36:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27478.752562 secs 2023-03-09T22:36:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:36:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:36:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27940.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:36:09.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:36:09.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:36:09.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:09 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:36:09.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:09 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:36:09.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:09 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:36:09.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:09 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:36:09.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:09 smithi122 ceph-mon[93869]: pgmap v15168: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 578 KiB/s rd, 535 KiB/s wr, 29 op/s 2023-03-09T22:36:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:36:09.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:36:09.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:09 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:36:09.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:09 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:36:09.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:09 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:36:09.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:09 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:36:09.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:09 smithi181 ceph-mon[94189]: pgmap v15168: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 578 KiB/s rd, 535 KiB/s wr, 29 op/s 2023-03-09T22:36:09.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:36:09.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:36:09.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:09 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:36:09.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:09 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:36:09.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:09 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:36:09.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:09 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:36:09.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:09 smithi038 ceph-mon[92422]: pgmap v15168: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 578 KiB/s rd, 535 KiB/s wr, 29 op/s 2023-03-09T22:36:10.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27481.379850 secs 2023-03-09T22:36:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27481.379850 secs 2023-03-09T22:36:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27481.379850 secs 2023-03-09T22:36:12.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:12 smithi122 ceph-mon[93869]: pgmap v15169: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 630 KiB/s rd, 658 KiB/s wr, 28 op/s 2023-03-09T22:36:12.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:12 smithi122 ceph-mon[93869]: osdmap e1164: 12 total, 12 up, 12 in 2023-03-09T22:36:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:12 smithi181 ceph-mon[94189]: pgmap v15169: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 630 KiB/s rd, 658 KiB/s wr, 28 op/s 2023-03-09T22:36:12.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:12 smithi181 ceph-mon[94189]: osdmap e1164: 12 total, 12 up, 12 in 2023-03-09T22:36:12.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:12 smithi038 ceph-mon[92422]: pgmap v15169: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 630 KiB/s rd, 658 KiB/s wr, 28 op/s 2023-03-09T22:36:12.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:12 smithi038 ceph-mon[92422]: osdmap e1164: 12 total, 12 up, 12 in 2023-03-09T22:36:13.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27483.752679 secs 2023-03-09T22:36:13.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:36:13.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:36:13.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:13 smithi122 ceph-mon[93869]: pgmap v15171: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 432 KiB/s rd, 444 KiB/s wr, 13 op/s 2023-03-09T22:36:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27483.752679 secs 2023-03-09T22:36:13.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:36:13.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:36:13.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:13 smithi181 ceph-mon[94189]: pgmap v15171: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 432 KiB/s rd, 444 KiB/s wr, 13 op/s 2023-03-09T22:36:13.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27483.752679 secs 2023-03-09T22:36:13.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:36:13.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:36:13.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:13 smithi038 ceph-mon[92422]: pgmap v15171: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 432 KiB/s rd, 444 KiB/s wr, 13 op/s 2023-03-09T22:36:13.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27945.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:36:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:15 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:36:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:15 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:36:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27486.379909 secs 2023-03-09T22:36:15.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:15 smithi122 ceph-mon[93869]: pgmap v15172: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 393 KiB/s wr, 11 op/s 2023-03-09T22:36:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:15 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:36:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:15 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:36:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27486.379909 secs 2023-03-09T22:36:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:15 smithi181 ceph-mon[94189]: pgmap v15172: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 393 KiB/s wr, 11 op/s 2023-03-09T22:36:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:15 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:36:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:15 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:36:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27486.379909 secs 2023-03-09T22:36:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:15 smithi038 ceph-mon[92422]: pgmap v15172: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 393 KiB/s wr, 11 op/s 2023-03-09T22:36:18.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:18 smithi122 ceph-mon[93869]: pgmap v15173: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 136 KiB/s wr, 7 op/s 2023-03-09T22:36:18.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27488.752795 secs 2023-03-09T22:36:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:18 smithi181 ceph-mon[94189]: pgmap v15173: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 136 KiB/s wr, 7 op/s 2023-03-09T22:36:18.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27488.752795 secs 2023-03-09T22:36:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:18 smithi038 ceph-mon[92422]: pgmap v15173: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 136 KiB/s wr, 7 op/s 2023-03-09T22:36:18.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27488.752795 secs 2023-03-09T22:36:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27950.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:36:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:20 smithi122 ceph-mon[93869]: pgmap v15174: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 136 KiB/s wr, 7 op/s 2023-03-09T22:36:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27491.379969 secs 2023-03-09T22:36:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:20 smithi181 ceph-mon[94189]: pgmap v15174: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 136 KiB/s wr, 7 op/s 2023-03-09T22:36:20.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27491.379969 secs 2023-03-09T22:36:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:20 smithi038 ceph-mon[92422]: pgmap v15174: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 136 KiB/s wr, 7 op/s 2023-03-09T22:36:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27491.379969 secs 2023-03-09T22:36:22.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:22 smithi122 ceph-mon[93869]: pgmap v15175: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T22:36:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:22 smithi181 ceph-mon[94189]: pgmap v15175: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T22:36:22.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:22 smithi038 ceph-mon[92422]: pgmap v15175: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T22:36:23.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27493.752949 secs 2023-03-09T22:36:23.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:23 smithi122 ceph-mon[93869]: pgmap v15176: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T22:36:23.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27493.752949 secs 2023-03-09T22:36:23.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:23 smithi181 ceph-mon[94189]: pgmap v15176: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T22:36:23.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27493.752949 secs 2023-03-09T22:36:23.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:23 smithi038 ceph-mon[92422]: pgmap v15176: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T22:36:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27955.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:36:25.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27496.379981 secs 2023-03-09T22:36:25.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27496.379981 secs 2023-03-09T22:36:25.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27496.379981 secs 2023-03-09T22:36:26.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:26 smithi122 ceph-mon[93869]: pgmap v15177: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T22:36:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:26 smithi181 ceph-mon[94189]: pgmap v15177: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T22:36:26.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:26 smithi038 ceph-mon[92422]: pgmap v15177: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T22:36:27.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:27 smithi122 ceph-mon[93869]: pgmap v15178: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T22:36:27.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:27 smithi181 ceph-mon[94189]: pgmap v15178: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T22:36:27.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:27 smithi038 ceph-mon[92422]: pgmap v15178: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T22:36:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27498.753085 secs 2023-03-09T22:36:28.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27498.753085 secs 2023-03-09T22:36:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27498.753085 secs 2023-03-09T22:36:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27960.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:36:29.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:29 smithi122 ceph-mon[93869]: pgmap v15179: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T22:36:29.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:29 smithi181 ceph-mon[94189]: pgmap v15179: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T22:36:29.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:29 smithi038 ceph-mon[92422]: pgmap v15179: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T22:36:30.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27501.380039 secs 2023-03-09T22:36:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27501.380039 secs 2023-03-09T22:36:30.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27501.380039 secs 2023-03-09T22:36:32.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:32 smithi122 ceph-mon[93869]: pgmap v15180: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 97 KiB/s wr, 7 op/s 2023-03-09T22:36:32.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:32 smithi181 ceph-mon[94189]: pgmap v15180: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 97 KiB/s wr, 7 op/s 2023-03-09T22:36:32.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:32 smithi038 ceph-mon[92422]: pgmap v15180: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 97 KiB/s wr, 7 op/s 2023-03-09T22:36:33.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27503.753184 secs 2023-03-09T22:36:33.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:33 smithi122 ceph-mon[93869]: pgmap v15181: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 97 KiB/s wr, 6 op/s 2023-03-09T22:36:33.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27503.753184 secs 2023-03-09T22:36:33.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:33 smithi181 ceph-mon[94189]: pgmap v15181: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 97 KiB/s wr, 6 op/s 2023-03-09T22:36:33.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27503.753184 secs 2023-03-09T22:36:33.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:33 smithi038 ceph-mon[92422]: pgmap v15181: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 97 KiB/s wr, 6 op/s 2023-03-09T22:36:33.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27965.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:36:35.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27506.380098 secs 2023-03-09T22:36:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27506.380098 secs 2023-03-09T22:36:35.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27506.380098 secs 2023-03-09T22:36:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:36 smithi122 ceph-mon[93869]: pgmap v15182: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 113 KiB/s wr, 6 op/s 2023-03-09T22:36:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:36 smithi181 ceph-mon[94189]: pgmap v15182: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 113 KiB/s wr, 6 op/s 2023-03-09T22:36:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:36 smithi038 ceph-mon[92422]: pgmap v15182: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 113 KiB/s wr, 6 op/s 2023-03-09T22:36:38.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:38 smithi122 ceph-mon[93869]: pgmap v15183: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 208 KiB/s wr, 8 op/s 2023-03-09T22:36:38.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27508.753304 secs 2023-03-09T22:36:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:38 smithi181 ceph-mon[94189]: pgmap v15183: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 208 KiB/s wr, 8 op/s 2023-03-09T22:36:38.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27508.753304 secs 2023-03-09T22:36:38.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:38 smithi038 ceph-mon[92422]: pgmap v15183: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 208 KiB/s wr, 8 op/s 2023-03-09T22:36:38.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27508.753304 secs 2023-03-09T22:36:38.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27970.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:36:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:40 smithi122 ceph-mon[93869]: pgmap v15184: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 208 KiB/s wr, 6 op/s 2023-03-09T22:36:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:36:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:36:40.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27511.380157 secs 2023-03-09T22:36:40.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:40 smithi181 ceph-mon[94189]: pgmap v15184: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 208 KiB/s wr, 6 op/s 2023-03-09T22:36:40.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:36:40.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:36:40.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27511.380157 secs 2023-03-09T22:36:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:40 smithi038 ceph-mon[92422]: pgmap v15184: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 208 KiB/s wr, 6 op/s 2023-03-09T22:36:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:36:40.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:36:40.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27511.380157 secs 2023-03-09T22:36:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:36:41.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:36:41.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:41 smithi122 ceph-mon[93869]: pgmap v15185: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 457 KiB/s rd, 453 KiB/s wr, 9 op/s 2023-03-09T22:36:41.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:36:41.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:36:41.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:41 smithi181 ceph-mon[94189]: pgmap v15185: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 457 KiB/s rd, 453 KiB/s wr, 9 op/s 2023-03-09T22:36:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:36:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:36:41.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:41 smithi038 ceph-mon[92422]: pgmap v15185: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 457 KiB/s rd, 453 KiB/s wr, 9 op/s 2023-03-09T22:36:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27975.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:36:44.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:44 smithi122 ceph-mon[93869]: pgmap v15186: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 357 KiB/s wr, 8 op/s 2023-03-09T22:36:44.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:44 smithi181 ceph-mon[94189]: pgmap v15186: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 357 KiB/s wr, 8 op/s 2023-03-09T22:36:44.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:44 smithi038 ceph-mon[92422]: pgmap v15186: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 357 KiB/s wr, 8 op/s 2023-03-09T22:36:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27516.380217 secs 2023-03-09T22:36:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:45 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:36:45.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27516.380217 secs 2023-03-09T22:36:45.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:45 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:36:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27516.380217 secs 2023-03-09T22:36:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:45 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:36:46.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:46 smithi122 ceph-mon[93869]: pgmap v15187: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 486 KiB/s wr, 8 op/s 2023-03-09T22:36:46.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:46 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:36:46.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:46 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:36:46.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:46 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:36:46.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:46 smithi181 ceph-mon[94189]: pgmap v15187: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 486 KiB/s wr, 8 op/s 2023-03-09T22:36:46.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:46 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:36:46.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:46 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:36:46.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:46 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:36:46.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:46 smithi038 ceph-mon[92422]: pgmap v15187: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 486 KiB/s wr, 8 op/s 2023-03-09T22:36:46.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:46 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:36:46.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:46 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:36:46.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:46 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:36:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:48 smithi122 ceph-mon[93869]: pgmap v15188: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 490 KiB/s rd, 471 KiB/s wr, 10 op/s 2023-03-09T22:36:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:48 smithi181 ceph-mon[94189]: pgmap v15188: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 490 KiB/s rd, 471 KiB/s wr, 10 op/s 2023-03-09T22:36:48.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:48 smithi038 ceph-mon[92422]: pgmap v15188: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 490 KiB/s rd, 471 KiB/s wr, 10 op/s 2023-03-09T22:36:48.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27980.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:36:49.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:49 smithi122 ceph-mon[93869]: pgmap v15189: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 398 KiB/s wr, 9 op/s 2023-03-09T22:36:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:49 smithi181 ceph-mon[94189]: pgmap v15189: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 398 KiB/s wr, 9 op/s 2023-03-09T22:36:49.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:49 smithi038 ceph-mon[92422]: pgmap v15189: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 398 KiB/s wr, 9 op/s 2023-03-09T22:36:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27521.380279 secs 2023-03-09T22:36:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27521.380279 secs 2023-03-09T22:36:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27521.380279 secs 2023-03-09T22:36:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:52 smithi122 ceph-mon[93869]: pgmap v15190: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 447 KiB/s wr, 12 op/s 2023-03-09T22:36:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:36:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:36:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:52 smithi181 ceph-mon[94189]: pgmap v15190: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 447 KiB/s wr, 12 op/s 2023-03-09T22:36:52.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:36:52.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:36:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:52 smithi038 ceph-mon[92422]: pgmap v15190: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 447 KiB/s wr, 12 op/s 2023-03-09T22:36:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:36:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:36:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:36:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:36:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:53 smithi122 ceph-mon[93869]: pgmap v15191: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 224 KiB/s wr, 10 op/s 2023-03-09T22:36:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:36:53.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:36:53.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:53 smithi181 ceph-mon[94189]: pgmap v15191: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 224 KiB/s wr, 10 op/s 2023-03-09T22:36:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:36:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:36:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:53 smithi038 ceph-mon[92422]: pgmap v15191: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 224 KiB/s wr, 10 op/s 2023-03-09T22:36:53.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27985.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:36:55.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27526.380338 secs 2023-03-09T22:36:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27526.380338 secs 2023-03-09T22:36:55.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27526.380338 secs 2023-03-09T22:36:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:56 smithi122 ceph-mon[93869]: pgmap v15192: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 481 KiB/s rd, 467 KiB/s wr, 10 op/s 2023-03-09T22:36:56.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:56 smithi181 ceph-mon[94189]: pgmap v15192: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 481 KiB/s rd, 467 KiB/s wr, 10 op/s 2023-03-09T22:36:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:56 smithi038 ceph-mon[92422]: pgmap v15192: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 481 KiB/s rd, 467 KiB/s wr, 10 op/s 2023-03-09T22:36:57.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:36:57.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:36:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:36:57.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:36:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:36:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:36:58.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:58 smithi122 ceph-mon[93869]: pgmap v15193: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 621 KiB/s rd, 607 KiB/s wr, 13 op/s 2023-03-09T22:36:58.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:58 smithi181 ceph-mon[94189]: pgmap v15193: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 621 KiB/s rd, 607 KiB/s wr, 13 op/s 2023-03-09T22:36:58.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:58 smithi038 ceph-mon[92422]: pgmap v15193: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 621 KiB/s rd, 607 KiB/s wr, 13 op/s 2023-03-09T22:36:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27990.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:36:59.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:36:59 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 27528.753755 secs 2023-03-09T22:36:59.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:36:59 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 27528.753755 secs 2023-03-09T22:36:59.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:36:59 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 27528.753755 secs 2023-03-09T22:37:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:37:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:37:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:00 smithi122 ceph-mon[93869]: pgmap v15194: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 644 KiB/s rd, 631 KiB/s wr, 12 op/s 2023-03-09T22:37:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27531.380397 secs 2023-03-09T22:37:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:37:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:37:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:00 smithi181 ceph-mon[94189]: pgmap v15194: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 644 KiB/s rd, 631 KiB/s wr, 12 op/s 2023-03-09T22:37:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27531.380397 secs 2023-03-09T22:37:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:37:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:37:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:00 smithi038 ceph-mon[92422]: pgmap v15194: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 644 KiB/s rd, 631 KiB/s wr, 12 op/s 2023-03-09T22:37:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27531.380397 secs 2023-03-09T22:37:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:01 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:37:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:37:01.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:37:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:01 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:37:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:37:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:37:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:01 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:37:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:37:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:37:02.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:02 smithi122 ceph-mon[93869]: pgmap v15195: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 742 KiB/s rd, 739 KiB/s wr, 15 op/s 2023-03-09T22:37:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:02 smithi181 ceph-mon[94189]: pgmap v15195: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 742 KiB/s rd, 739 KiB/s wr, 15 op/s 2023-03-09T22:37:02.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:02 smithi038 ceph-mon[92422]: pgmap v15195: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 742 KiB/s rd, 739 KiB/s wr, 15 op/s 2023-03-09T22:37:03.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27533.753871 secs 2023-03-09T22:37:03.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:03 smithi122 ceph-mon[93869]: pgmap v15196: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 737 KiB/s rd, 752 KiB/s wr, 25 op/s 2023-03-09T22:37:03.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27533.753871 secs 2023-03-09T22:37:03.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:03 smithi181 ceph-mon[94189]: pgmap v15196: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 737 KiB/s rd, 752 KiB/s wr, 25 op/s 2023-03-09T22:37:03.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27533.753871 secs 2023-03-09T22:37:03.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:03 smithi038 ceph-mon[92422]: pgmap v15196: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 737 KiB/s rd, 752 KiB/s wr, 25 op/s 2023-03-09T22:37:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 27995.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:37:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:05 smithi122 ceph-mon[93869]: osdmap e1165: 12 total, 12 up, 12 in 2023-03-09T22:37:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27536.380456 secs 2023-03-09T22:37:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:05 smithi122 ceph-mon[93869]: pgmap v15198: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 583 KiB/s wr, 26 op/s 2023-03-09T22:37:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:05 smithi181 ceph-mon[94189]: osdmap e1165: 12 total, 12 up, 12 in 2023-03-09T22:37:05.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27536.380456 secs 2023-03-09T22:37:05.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:05 smithi181 ceph-mon[94189]: pgmap v15198: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 583 KiB/s wr, 26 op/s 2023-03-09T22:37:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:05 smithi038 ceph-mon[92422]: osdmap e1165: 12 total, 12 up, 12 in 2023-03-09T22:37:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27536.380456 secs 2023-03-09T22:37:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:05 smithi038 ceph-mon[92422]: pgmap v15198: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 583 KiB/s wr, 26 op/s 2023-03-09T22:37:06.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:06 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:37:06.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:37:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:06 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:37:06.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:37:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:06 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:37:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:37:07.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:07 smithi122 ceph-mon[93869]: pgmap v15199: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 609 KiB/s wr, 30 op/s 2023-03-09T22:37:07.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:07 smithi181 ceph-mon[94189]: pgmap v15199: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 609 KiB/s wr, 30 op/s 2023-03-09T22:37:07.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:07 smithi038 ceph-mon[92422]: pgmap v15199: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 609 KiB/s wr, 30 op/s 2023-03-09T22:37:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27538.754004 secs 2023-03-09T22:37:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:37:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:37:08.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27538.754004 secs 2023-03-09T22:37:08.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:37:08.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:37:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27538.754004 secs 2023-03-09T22:37:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:37:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:37:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28000.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:37:09.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:37:09.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:37:09.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:09 smithi122 ceph-mon[93869]: pgmap v15200: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 539 KiB/s rd, 580 KiB/s wr, 29 op/s 2023-03-09T22:37:09.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:09 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:37:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:37:09.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:37:09.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:09 smithi181 ceph-mon[94189]: pgmap v15200: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 539 KiB/s rd, 580 KiB/s wr, 29 op/s 2023-03-09T22:37:09.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:09 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:37:09.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:37:09.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:37:09.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:09 smithi038 ceph-mon[92422]: pgmap v15200: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 539 KiB/s rd, 580 KiB/s wr, 29 op/s 2023-03-09T22:37:09.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:09 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:37:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27541.380517 secs 2023-03-09T22:37:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:37:10.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:37:10.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:37:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27541.380517 secs 2023-03-09T22:37:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:37:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:37:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:37:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27541.380517 secs 2023-03-09T22:37:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:37:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:37:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:37:12.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:12 smithi122 ceph-mon[93869]: pgmap v15201: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 446 KiB/s wr, 28 op/s 2023-03-09T22:37:12.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:12 smithi122 ceph-mon[93869]: osdmap e1166: 12 total, 12 up, 12 in 2023-03-09T22:37:12.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:12 smithi181 ceph-mon[94189]: pgmap v15201: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 446 KiB/s wr, 28 op/s 2023-03-09T22:37:12.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:12 smithi181 ceph-mon[94189]: osdmap e1166: 12 total, 12 up, 12 in 2023-03-09T22:37:12.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:12 smithi038 ceph-mon[92422]: pgmap v15201: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 446 KiB/s wr, 28 op/s 2023-03-09T22:37:12.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:12 smithi038 ceph-mon[92422]: osdmap e1166: 12 total, 12 up, 12 in 2023-03-09T22:37:13.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27543.754095 secs 2023-03-09T22:37:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27543.754095 secs 2023-03-09T22:37:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27543.754095 secs 2023-03-09T22:37:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28005.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:37:14.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:14 smithi122 ceph-mon[93869]: pgmap v15203: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 504 KiB/s wr, 14 op/s 2023-03-09T22:37:14.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:14 smithi181 ceph-mon[94189]: pgmap v15203: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 504 KiB/s wr, 14 op/s 2023-03-09T22:37:14.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:14 smithi038 ceph-mon[92422]: pgmap v15203: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 504 KiB/s wr, 14 op/s 2023-03-09T22:37:15.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27546.380576 secs 2023-03-09T22:37:15.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:15 smithi122 ceph-mon[93869]: pgmap v15204: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 422 KiB/s rd, 446 KiB/s wr, 13 op/s 2023-03-09T22:37:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27546.380576 secs 2023-03-09T22:37:15.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:15 smithi181 ceph-mon[94189]: pgmap v15204: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 422 KiB/s rd, 446 KiB/s wr, 13 op/s 2023-03-09T22:37:15.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27546.380576 secs 2023-03-09T22:37:15.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:15 smithi038 ceph-mon[92422]: pgmap v15204: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 422 KiB/s rd, 446 KiB/s wr, 13 op/s 2023-03-09T22:37:17.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:17 smithi122 ceph-mon[93869]: pgmap v15205: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 96 KiB/s wr, 8 op/s 2023-03-09T22:37:17.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:17 smithi181 ceph-mon[94189]: pgmap v15205: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 96 KiB/s wr, 8 op/s 2023-03-09T22:37:17.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:17 smithi038 ceph-mon[92422]: pgmap v15205: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 96 KiB/s wr, 8 op/s 2023-03-09T22:37:18.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27548.754207 secs 2023-03-09T22:37:18.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27548.754207 secs 2023-03-09T22:37:18.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27548.754207 secs 2023-03-09T22:37:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28010.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:37:19.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:19 smithi122 ceph-mon[93869]: pgmap v15206: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 96 KiB/s wr, 8 op/s 2023-03-09T22:37:19.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:19 smithi181 ceph-mon[94189]: pgmap v15206: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 96 KiB/s wr, 8 op/s 2023-03-09T22:37:19.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:19 smithi038 ceph-mon[92422]: pgmap v15206: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 96 KiB/s wr, 8 op/s 2023-03-09T22:37:20.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27551.380662 secs 2023-03-09T22:37:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27551.380662 secs 2023-03-09T22:37:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27551.380662 secs 2023-03-09T22:37:21.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:21 smithi038 ceph-mon[92422]: pgmap v15207: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 75 KiB/s wr, 8 op/s 2023-03-09T22:37:21.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:21 smithi122 ceph-mon[93869]: pgmap v15207: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 75 KiB/s wr, 8 op/s 2023-03-09T22:37:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:21 smithi181 ceph-mon[94189]: pgmap v15207: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 75 KiB/s wr, 8 op/s 2023-03-09T22:37:23.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27553.754312 secs 2023-03-09T22:37:23.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27553.754312 secs 2023-03-09T22:37:23.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27553.754312 secs 2023-03-09T22:37:23.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28015.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:37:24.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:24 smithi122 ceph-mon[93869]: pgmap v15208: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 63 KiB/s wr, 7 op/s 2023-03-09T22:37:24.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:24 smithi181 ceph-mon[94189]: pgmap v15208: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 63 KiB/s wr, 7 op/s 2023-03-09T22:37:24.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:24 smithi038 ceph-mon[92422]: pgmap v15208: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 63 KiB/s wr, 7 op/s 2023-03-09T22:37:25.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27556.380737 secs 2023-03-09T22:37:25.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27556.380737 secs 2023-03-09T22:37:25.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27556.380737 secs 2023-03-09T22:37:26.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:26 smithi122 ceph-mon[93869]: pgmap v15209: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T22:37:26.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:26 smithi181 ceph-mon[94189]: pgmap v15209: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T22:37:26.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:26 smithi038 ceph-mon[92422]: pgmap v15209: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T22:37:27.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:27 smithi122 ceph-mon[93869]: pgmap v15210: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 9 op/s 2023-03-09T22:37:27.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:27 smithi181 ceph-mon[94189]: pgmap v15210: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 9 op/s 2023-03-09T22:37:27.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:27 smithi038 ceph-mon[92422]: pgmap v15210: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 9 op/s 2023-03-09T22:37:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27558.754409 secs 2023-03-09T22:37:28.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27558.754409 secs 2023-03-09T22:37:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27558.754409 secs 2023-03-09T22:37:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28020.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:37:29.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:29 smithi122 ceph-mon[93869]: pgmap v15211: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 7 op/s 2023-03-09T22:37:29.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:29 smithi181 ceph-mon[94189]: pgmap v15211: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 7 op/s 2023-03-09T22:37:29.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:29 smithi038 ceph-mon[92422]: pgmap v15211: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 7 op/s 2023-03-09T22:37:30.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27561.380796 secs 2023-03-09T22:37:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27561.380796 secs 2023-03-09T22:37:30.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27561.380796 secs 2023-03-09T22:37:31.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:31 smithi122 ceph-mon[93869]: pgmap v15212: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 9 op/s 2023-03-09T22:37:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:31 smithi181 ceph-mon[94189]: pgmap v15212: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 9 op/s 2023-03-09T22:37:31.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:31 smithi038 ceph-mon[92422]: pgmap v15212: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 9 op/s 2023-03-09T22:37:33.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27563.754528 secs 2023-03-09T22:37:33.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27563.754528 secs 2023-03-09T22:37:33.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27563.754528 secs 2023-03-09T22:37:33.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28025.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:37:34.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:34 smithi122 ceph-mon[93869]: pgmap v15213: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 7 op/s 2023-03-09T22:37:34.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:34 smithi181 ceph-mon[94189]: pgmap v15213: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 7 op/s 2023-03-09T22:37:34.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:34 smithi038 ceph-mon[92422]: pgmap v15213: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 7 op/s 2023-03-09T22:37:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27566.380856 secs 2023-03-09T22:37:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27566.380856 secs 2023-03-09T22:37:35.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27566.380856 secs 2023-03-09T22:37:36.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:36 smithi122 ceph-mon[93869]: pgmap v15214: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T22:37:36.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:36 smithi181 ceph-mon[94189]: pgmap v15214: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T22:37:36.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:36 smithi038 ceph-mon[92422]: pgmap v15214: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T22:37:37.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:37 smithi122 ceph-mon[93869]: pgmap v15215: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 9 op/s 2023-03-09T22:37:37.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:37 smithi181 ceph-mon[94189]: pgmap v15215: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 9 op/s 2023-03-09T22:37:37.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:37 smithi038 ceph-mon[92422]: pgmap v15215: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 9 op/s 2023-03-09T22:37:38.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27568.754626 secs 2023-03-09T22:37:38.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27568.754626 secs 2023-03-09T22:37:38.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27568.754626 secs 2023-03-09T22:37:38.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28030.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:37:39.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:39 smithi122 ceph-mon[93869]: pgmap v15216: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 511 B/s wr, 7 op/s 2023-03-09T22:37:39.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:39 smithi181 ceph-mon[94189]: pgmap v15216: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 511 B/s wr, 7 op/s 2023-03-09T22:37:39.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:39 smithi038 ceph-mon[92422]: pgmap v15216: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 511 B/s wr, 7 op/s 2023-03-09T22:37:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27571.380937 secs 2023-03-09T22:37:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27571.380937 secs 2023-03-09T22:37:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27571.380937 secs 2023-03-09T22:37:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:37:41.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:37:41.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:41 smithi122 ceph-mon[93869]: pgmap v15217: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 9 op/s 2023-03-09T22:37:41.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:37:41.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:37:41.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:41 smithi181 ceph-mon[94189]: pgmap v15217: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 9 op/s 2023-03-09T22:37:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:37:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:37:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:41 smithi038 ceph-mon[92422]: pgmap v15217: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 9 op/s 2023-03-09T22:37:42.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:42 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 27573.754762 secs 2023-03-09T22:37:42.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:42 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 27573.754762 secs 2023-03-09T22:37:42.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:42 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 27573.754762 secs 2023-03-09T22:37:43.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:43 smithi122 ceph-mon[93869]: pgmap v15218: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 152 KiB/s wr, 8 op/s 2023-03-09T22:37:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:43 smithi181 ceph-mon[94189]: pgmap v15218: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 152 KiB/s wr, 8 op/s 2023-03-09T22:37:43.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:43 smithi038 ceph-mon[92422]: pgmap v15218: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 152 KiB/s wr, 8 op/s 2023-03-09T22:37:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28035.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:37:45.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27576.380993 secs 2023-03-09T22:37:45.481 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:45 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:37:45.481 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:45 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:37:45.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27576.380993 secs 2023-03-09T22:37:45.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:45 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:37:45.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:45 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:37:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27576.380993 secs 2023-03-09T22:37:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:45 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:37:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:45 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:37:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:46 smithi122 ceph-mon[93869]: pgmap v15219: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 153 KiB/s wr, 8 op/s 2023-03-09T22:37:46.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:46 smithi181 ceph-mon[94189]: pgmap v15219: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 153 KiB/s wr, 8 op/s 2023-03-09T22:37:46.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:46 smithi038 ceph-mon[92422]: pgmap v15219: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 153 KiB/s wr, 8 op/s 2023-03-09T22:37:47.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:47 smithi122 ceph-mon[93869]: pgmap v15220: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 303 KiB/s wr, 11 op/s 2023-03-09T22:37:47.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27578.754880 secs 2023-03-09T22:37:47.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:47 smithi181 ceph-mon[94189]: pgmap v15220: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 303 KiB/s wr, 11 op/s 2023-03-09T22:37:47.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27578.754880 secs 2023-03-09T22:37:47.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:47 smithi038 ceph-mon[92422]: pgmap v15220: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 303 KiB/s wr, 11 op/s 2023-03-09T22:37:47.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27578.754880 secs 2023-03-09T22:37:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:37:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:37:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:37:48.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:37:48.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:37:48.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:37:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28040.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:37:49.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:37:49.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:37:49.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:49 smithi122 ceph-mon[93869]: pgmap v15221: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 553 KiB/s rd, 540 KiB/s wr, 10 op/s 2023-03-09T22:37:49.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:37:49.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:37:49.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:49 smithi181 ceph-mon[94189]: pgmap v15221: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 553 KiB/s rd, 540 KiB/s wr, 10 op/s 2023-03-09T22:37:49.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:37:49.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:37:49.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:49 smithi038 ceph-mon[92422]: pgmap v15221: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 553 KiB/s rd, 540 KiB/s wr, 10 op/s 2023-03-09T22:37:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:50 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:37:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:50 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:37:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27581.381054 secs 2023-03-09T22:37:50.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:50 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:37:50.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:50 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:37:50.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27581.381054 secs 2023-03-09T22:37:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:50 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:37:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:50 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:37:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27581.381054 secs 2023-03-09T22:37:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:37:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:37:51.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:51 smithi122 ceph-mon[93869]: pgmap v15222: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 554 KiB/s rd, 540 KiB/s wr, 12 op/s 2023-03-09T22:37:51.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:37:51.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:37:51.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:51 smithi181 ceph-mon[94189]: pgmap v15222: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 554 KiB/s rd, 540 KiB/s wr, 12 op/s 2023-03-09T22:37:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:37:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:37:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:51 smithi038 ceph-mon[92422]: pgmap v15222: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 554 KiB/s rd, 540 KiB/s wr, 12 op/s 2023-03-09T22:37:52.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27583.754979 secs 2023-03-09T22:37:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27583.754979 secs 2023-03-09T22:37:52.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27583.754979 secs 2023-03-09T22:37:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:53 smithi038 ceph-mon[92422]: pgmap v15223: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 740 KiB/s rd, 768 KiB/s wr, 12 op/s 2023-03-09T22:37:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28045.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:37:53.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:53 smithi122 ceph-mon[93869]: pgmap v15223: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 740 KiB/s rd, 768 KiB/s wr, 12 op/s 2023-03-09T22:37:53.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:53 smithi181 ceph-mon[94189]: pgmap v15223: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 740 KiB/s rd, 768 KiB/s wr, 12 op/s 2023-03-09T22:37:55.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:37:55.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:37:55.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27586.381117 secs 2023-03-09T22:37:55.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:55 smithi038 ceph-mon[92422]: pgmap v15224: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 746 KiB/s wr, 11 op/s 2023-03-09T22:37:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:37:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:37:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27586.381117 secs 2023-03-09T22:37:55.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:55 smithi122 ceph-mon[93869]: pgmap v15224: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 746 KiB/s wr, 11 op/s 2023-03-09T22:37:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:37:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:37:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27586.381117 secs 2023-03-09T22:37:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:55 smithi181 ceph-mon[94189]: pgmap v15224: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 746 KiB/s wr, 11 op/s 2023-03-09T22:37:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:37:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:37:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:57 smithi038 ceph-mon[92422]: pgmap v15225: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 745 KiB/s wr, 13 op/s 2023-03-09T22:37:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:37:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27588.755095 secs 2023-03-09T22:37:57.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:37:57.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:37:57.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:57 smithi122 ceph-mon[93869]: pgmap v15225: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 745 KiB/s wr, 13 op/s 2023-03-09T22:37:57.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:37:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27588.755095 secs 2023-03-09T22:37:57.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:37:57.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:37:57.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:57 smithi181 ceph-mon[94189]: pgmap v15225: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 745 KiB/s wr, 13 op/s 2023-03-09T22:37:57.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:37:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27588.755095 secs 2023-03-09T22:37:58.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28050.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:38:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:00 smithi122 ceph-mon[93869]: pgmap v15226: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 763 KiB/s wr, 11 op/s 2023-03-09T22:38:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27591.381175 secs 2023-03-09T22:38:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:00 smithi181 ceph-mon[94189]: pgmap v15226: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 763 KiB/s wr, 11 op/s 2023-03-09T22:38:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27591.381175 secs 2023-03-09T22:38:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:00 smithi038 ceph-mon[92422]: pgmap v15226: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 763 KiB/s wr, 11 op/s 2023-03-09T22:38:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27591.381175 secs 2023-03-09T22:38:02.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:02 smithi122 ceph-mon[93869]: pgmap v15227: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 867 KiB/s rd, 526 KiB/s wr, 11 op/s 2023-03-09T22:38:02.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27593.755185 secs 2023-03-09T22:38:02.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:02 smithi181 ceph-mon[94189]: pgmap v15227: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 867 KiB/s rd, 526 KiB/s wr, 11 op/s 2023-03-09T22:38:02.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27593.755185 secs 2023-03-09T22:38:02.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:02 smithi038 ceph-mon[92422]: pgmap v15227: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 867 KiB/s rd, 526 KiB/s wr, 11 op/s 2023-03-09T22:38:02.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27593.755185 secs 2023-03-09T22:38:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28055.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:38:04.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:04 smithi122 ceph-mon[93869]: pgmap v15228: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 881 KiB/s rd, 556 KiB/s wr, 23 op/s 2023-03-09T22:38:04.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:04 smithi181 ceph-mon[94189]: pgmap v15228: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 881 KiB/s rd, 556 KiB/s wr, 23 op/s 2023-03-09T22:38:04.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:04 smithi038 ceph-mon[92422]: pgmap v15228: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 881 KiB/s rd, 556 KiB/s wr, 23 op/s 2023-03-09T22:38:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:05 smithi122 ceph-mon[93869]: osdmap e1167: 12 total, 12 up, 12 in 2023-03-09T22:38:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27596.381236 secs 2023-03-09T22:38:05.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:05 smithi181 ceph-mon[94189]: osdmap e1167: 12 total, 12 up, 12 in 2023-03-09T22:38:05.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27596.381236 secs 2023-03-09T22:38:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:05 smithi038 ceph-mon[92422]: osdmap e1167: 12 total, 12 up, 12 in 2023-03-09T22:38:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27596.381236 secs 2023-03-09T22:38:06.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:06 smithi122 ceph-mon[93869]: pgmap v15230: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 239 KiB/s wr, 24 op/s 2023-03-09T22:38:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:06 smithi181 ceph-mon[94189]: pgmap v15230: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 239 KiB/s wr, 24 op/s 2023-03-09T22:38:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:06 smithi038 ceph-mon[92422]: pgmap v15230: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 239 KiB/s wr, 24 op/s 2023-03-09T22:38:07.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:07 smithi122 ceph-mon[93869]: pgmap v15231: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 566 KiB/s rd, 558 KiB/s wr, 27 op/s 2023-03-09T22:38:07.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27598.755294 secs 2023-03-09T22:38:07.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:07 smithi181 ceph-mon[94189]: pgmap v15231: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 566 KiB/s rd, 558 KiB/s wr, 27 op/s 2023-03-09T22:38:07.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27598.755294 secs 2023-03-09T22:38:07.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:07 smithi038 ceph-mon[92422]: pgmap v15231: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 566 KiB/s rd, 558 KiB/s wr, 27 op/s 2023-03-09T22:38:07.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27598.755294 secs 2023-03-09T22:38:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:38:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:38:08.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:38:08.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:38:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:38:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:38:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28060.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:38:09.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:38:09.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:38:09.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:09 smithi122 ceph-mon[93869]: pgmap v15232: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 356 KiB/s wr, 26 op/s 2023-03-09T22:38:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:38:09.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:38:09.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:09 smithi181 ceph-mon[94189]: pgmap v15232: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 356 KiB/s wr, 26 op/s 2023-03-09T22:38:09.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:38:09.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:38:09.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:09 smithi038 ceph-mon[92422]: pgmap v15232: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 356 KiB/s wr, 26 op/s 2023-03-09T22:38:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27601.381311 secs 2023-03-09T22:38:10.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:38:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27601.381311 secs 2023-03-09T22:38:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:38:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27601.381311 secs 2023-03-09T22:38:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:38:11.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:11 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:38:11.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:11 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:38:11.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:11 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:38:11.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:11 smithi122 ceph-mon[93869]: pgmap v15233: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 356 KiB/s wr, 26 op/s 2023-03-09T22:38:11.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:11 smithi122 ceph-mon[93869]: osdmap e1168: 12 total, 12 up, 12 in 2023-03-09T22:38:11.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:11 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:38:11.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:11 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:38:11.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:11 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:38:11.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:11 smithi181 ceph-mon[94189]: pgmap v15233: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 356 KiB/s wr, 26 op/s 2023-03-09T22:38:11.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:11 smithi181 ceph-mon[94189]: osdmap e1168: 12 total, 12 up, 12 in 2023-03-09T22:38:11.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:11 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:38:11.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:11 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:38:11.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:11 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:38:11.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:11 smithi038 ceph-mon[92422]: pgmap v15233: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 356 KiB/s wr, 26 op/s 2023-03-09T22:38:11.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:11 smithi038 ceph-mon[92422]: osdmap e1168: 12 total, 12 up, 12 in 2023-03-09T22:38:13.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27603.755411 secs 2023-03-09T22:38:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27603.755411 secs 2023-03-09T22:38:13.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27603.755411 secs 2023-03-09T22:38:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28065.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:38:14.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:13 smithi122 ceph-mon[93869]: pgmap v15235: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 467 KiB/s wr, 12 op/s 2023-03-09T22:38:14.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:13 smithi181 ceph-mon[94189]: pgmap v15235: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 467 KiB/s wr, 12 op/s 2023-03-09T22:38:14.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:13 smithi038 ceph-mon[92422]: pgmap v15235: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 467 KiB/s wr, 12 op/s 2023-03-09T22:38:15.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27606.381373 secs 2023-03-09T22:38:15.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27606.381373 secs 2023-03-09T22:38:15.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27606.381373 secs 2023-03-09T22:38:16.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:16 smithi122 ceph-mon[93869]: pgmap v15236: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 416 KiB/s wr, 10 op/s 2023-03-09T22:38:16.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:16 smithi181 ceph-mon[94189]: pgmap v15236: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 416 KiB/s wr, 10 op/s 2023-03-09T22:38:16.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:16 smithi038 ceph-mon[92422]: pgmap v15236: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 416 KiB/s wr, 10 op/s 2023-03-09T22:38:17.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:17 smithi122 ceph-mon[93869]: pgmap v15237: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 97 KiB/s wr, 7 op/s 2023-03-09T22:38:17.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27608.755478 secs 2023-03-09T22:38:17.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:17 smithi181 ceph-mon[94189]: pgmap v15237: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 97 KiB/s wr, 7 op/s 2023-03-09T22:38:17.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27608.755478 secs 2023-03-09T22:38:17.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:17 smithi038 ceph-mon[92422]: pgmap v15237: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 97 KiB/s wr, 7 op/s 2023-03-09T22:38:17.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27608.755478 secs 2023-03-09T22:38:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28070.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:38:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:20 smithi122 ceph-mon[93869]: pgmap v15238: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 96 KiB/s wr, 6 op/s 2023-03-09T22:38:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27611.381457 secs 2023-03-09T22:38:20.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:20 smithi181 ceph-mon[94189]: pgmap v15238: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 96 KiB/s wr, 6 op/s 2023-03-09T22:38:20.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27611.381457 secs 2023-03-09T22:38:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:20 smithi038 ceph-mon[92422]: pgmap v15238: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 96 KiB/s wr, 6 op/s 2023-03-09T22:38:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27611.381457 secs 2023-03-09T22:38:22.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:22 smithi122 ceph-mon[93869]: pgmap v15239: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 96 KiB/s wr, 6 op/s 2023-03-09T22:38:22.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27613.755544 secs 2023-03-09T22:38:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:22 smithi181 ceph-mon[94189]: pgmap v15239: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 96 KiB/s wr, 6 op/s 2023-03-09T22:38:22.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27613.755544 secs 2023-03-09T22:38:22.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:22 smithi038 ceph-mon[92422]: pgmap v15239: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 96 KiB/s wr, 6 op/s 2023-03-09T22:38:22.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27613.755544 secs 2023-03-09T22:38:23.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28075.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:38:24.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:24 smithi122 ceph-mon[93869]: pgmap v15240: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 81 KiB/s wr, 6 op/s 2023-03-09T22:38:24.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:24 smithi181 ceph-mon[94189]: pgmap v15240: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 81 KiB/s wr, 6 op/s 2023-03-09T22:38:24.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:24 smithi038 ceph-mon[92422]: pgmap v15240: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 81 KiB/s wr, 6 op/s 2023-03-09T22:38:25.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27616.381519 secs 2023-03-09T22:38:25.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27616.381519 secs 2023-03-09T22:38:25.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27616.381519 secs 2023-03-09T22:38:26.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:26 smithi122 ceph-mon[93869]: pgmap v15241: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T22:38:26.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:26 smithi181 ceph-mon[94189]: pgmap v15241: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T22:38:26.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:26 smithi038 ceph-mon[92422]: pgmap v15241: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T22:38:27.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:27 smithi122 ceph-mon[93869]: pgmap v15242: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 18 KiB/s wr, 7 op/s 2023-03-09T22:38:27.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27618.755662 secs 2023-03-09T22:38:27.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:27 smithi181 ceph-mon[94189]: pgmap v15242: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 18 KiB/s wr, 7 op/s 2023-03-09T22:38:27.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27618.755662 secs 2023-03-09T22:38:27.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:27 smithi038 ceph-mon[92422]: pgmap v15242: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 18 KiB/s wr, 7 op/s 2023-03-09T22:38:27.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27618.755662 secs 2023-03-09T22:38:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28080.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:38:30.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:30 smithi122 ceph-mon[93869]: pgmap v15243: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 18 KiB/s wr, 5 op/s 2023-03-09T22:38:30.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27621.381579 secs 2023-03-09T22:38:30.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:30 smithi181 ceph-mon[94189]: pgmap v15243: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 18 KiB/s wr, 5 op/s 2023-03-09T22:38:30.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27621.381579 secs 2023-03-09T22:38:30.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:30 smithi038 ceph-mon[92422]: pgmap v15243: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 18 KiB/s wr, 5 op/s 2023-03-09T22:38:30.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27621.381579 secs 2023-03-09T22:38:32.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:32 smithi122 ceph-mon[93869]: pgmap v15244: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 97 KiB/s wr, 7 op/s 2023-03-09T22:38:32.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27623.755728 secs 2023-03-09T22:38:32.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:32 smithi181 ceph-mon[94189]: pgmap v15244: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 97 KiB/s wr, 7 op/s 2023-03-09T22:38:32.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27623.755728 secs 2023-03-09T22:38:32.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:32 smithi038 ceph-mon[92422]: pgmap v15244: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 97 KiB/s wr, 7 op/s 2023-03-09T22:38:32.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27623.755728 secs 2023-03-09T22:38:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28085.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:38:34.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:34 smithi122 ceph-mon[93869]: pgmap v15245: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 98 KiB/s wr, 7 op/s 2023-03-09T22:38:34.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:34 smithi181 ceph-mon[94189]: pgmap v15245: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 98 KiB/s wr, 7 op/s 2023-03-09T22:38:34.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:34 smithi038 ceph-mon[92422]: pgmap v15245: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 98 KiB/s wr, 7 op/s 2023-03-09T22:38:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27626.381640 secs 2023-03-09T22:38:35.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27626.381640 secs 2023-03-09T22:38:35.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27626.381640 secs 2023-03-09T22:38:36.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:36 smithi122 ceph-mon[93869]: pgmap v15246: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 98 KiB/s wr, 6 op/s 2023-03-09T22:38:36.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:38:36.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:38:36.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:36 smithi181 ceph-mon[94189]: pgmap v15246: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 98 KiB/s wr, 6 op/s 2023-03-09T22:38:36.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:38:36.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:38:36.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:36 smithi038 ceph-mon[92422]: pgmap v15246: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 98 KiB/s wr, 6 op/s 2023-03-09T22:38:36.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:38:36.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:38:37.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:38:37.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:38:37.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27628.755833 secs 2023-03-09T22:38:37.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:38:37.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:38:37.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27628.755833 secs 2023-03-09T22:38:37.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:38:37.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:38:37.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27628.755833 secs 2023-03-09T22:38:38.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:38 smithi181 ceph-mon[94189]: pgmap v15247: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 215 KiB/s wr, 9 op/s 2023-03-09T22:38:38.732 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:38 smithi122 ceph-mon[93869]: pgmap v15247: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 215 KiB/s wr, 9 op/s 2023-03-09T22:38:38.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:38 smithi038 ceph-mon[92422]: pgmap v15247: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 215 KiB/s wr, 9 op/s 2023-03-09T22:38:38.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28090.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:38:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:40 smithi122 ceph-mon[93869]: pgmap v15248: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 197 KiB/s wr, 7 op/s 2023-03-09T22:38:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27631.381699 secs 2023-03-09T22:38:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:40 smithi181 ceph-mon[94189]: pgmap v15248: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 197 KiB/s wr, 7 op/s 2023-03-09T22:38:40.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27631.381699 secs 2023-03-09T22:38:40.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:40 smithi038 ceph-mon[92422]: pgmap v15248: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 197 KiB/s wr, 7 op/s 2023-03-09T22:38:40.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27631.381699 secs 2023-03-09T22:38:42.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:42 smithi122 ceph-mon[93869]: pgmap v15249: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 343 KiB/s wr, 9 op/s 2023-03-09T22:38:42.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27633.755999 secs 2023-03-09T22:38:42.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:42 smithi181 ceph-mon[94189]: pgmap v15249: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 343 KiB/s wr, 9 op/s 2023-03-09T22:38:42.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27633.755999 secs 2023-03-09T22:38:42.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:42 smithi038 ceph-mon[92422]: pgmap v15249: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 343 KiB/s wr, 9 op/s 2023-03-09T22:38:42.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27633.755999 secs 2023-03-09T22:38:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28095.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:38:44.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:44 smithi122 ceph-mon[93869]: pgmap v15250: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 264 KiB/s wr, 8 op/s 2023-03-09T22:38:44.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:44 smithi181 ceph-mon[94189]: pgmap v15250: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 264 KiB/s wr, 8 op/s 2023-03-09T22:38:44.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:44 smithi038 ceph-mon[92422]: pgmap v15250: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 264 KiB/s wr, 8 op/s 2023-03-09T22:38:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27636.381760 secs 2023-03-09T22:38:45.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27636.381760 secs 2023-03-09T22:38:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27636.381760 secs 2023-03-09T22:38:46.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:46 smithi122 ceph-mon[93869]: pgmap v15251: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 394 KiB/s wr, 8 op/s 2023-03-09T22:38:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:46 smithi181 ceph-mon[94189]: pgmap v15251: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 394 KiB/s wr, 8 op/s 2023-03-09T22:38:46.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:46 smithi038 ceph-mon[92422]: pgmap v15251: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 394 KiB/s wr, 8 op/s 2023-03-09T22:38:47.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:47 smithi122 ceph-mon[93869]: pgmap v15252: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 394 KiB/s wr, 10 op/s 2023-03-09T22:38:47.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27638.756063 secs 2023-03-09T22:38:47.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:47 smithi181 ceph-mon[94189]: pgmap v15252: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 394 KiB/s wr, 10 op/s 2023-03-09T22:38:47.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27638.756063 secs 2023-03-09T22:38:47.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:47 smithi038 ceph-mon[92422]: pgmap v15252: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 394 KiB/s wr, 10 op/s 2023-03-09T22:38:47.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27638.756063 secs 2023-03-09T22:38:48.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28100.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:38:49.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:38:49.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:38:49.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:49 smithi122 ceph-mon[93869]: pgmap v15253: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 299 KiB/s wr, 8 op/s 2023-03-09T22:38:49.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:38:49.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:38:49.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:49 smithi181 ceph-mon[94189]: pgmap v15253: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 299 KiB/s wr, 8 op/s 2023-03-09T22:38:49.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:38:49.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:38:49.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:49 smithi038 ceph-mon[92422]: pgmap v15253: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 299 KiB/s wr, 8 op/s 2023-03-09T22:38:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27641.381818 secs 2023-03-09T22:38:50.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27641.381818 secs 2023-03-09T22:38:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27641.381818 secs 2023-03-09T22:38:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:38:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:38:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:51 smithi122 ceph-mon[93869]: pgmap v15254: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 446 KiB/s wr, 10 op/s 2023-03-09T22:38:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:38:51.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:38:51.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:51 smithi181 ceph-mon[94189]: pgmap v15254: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 446 KiB/s wr, 10 op/s 2023-03-09T22:38:52.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:38:52.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:38:52.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:51 smithi038 ceph-mon[92422]: pgmap v15254: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 446 KiB/s wr, 10 op/s 2023-03-09T22:38:52.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27643.756212 secs 2023-03-09T22:38:52.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27643.756212 secs 2023-03-09T22:38:53.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27643.756212 secs 2023-03-09T22:38:53.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28105.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:38:53.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:53 smithi122 ceph-mon[93869]: pgmap v15255: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 322 KiB/s wr, 9 op/s 2023-03-09T22:38:53.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:53 smithi181 ceph-mon[94189]: pgmap v15255: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 322 KiB/s wr, 9 op/s 2023-03-09T22:38:54.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:53 smithi038 ceph-mon[92422]: pgmap v15255: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 322 KiB/s wr, 9 op/s 2023-03-09T22:38:55.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27646.381893 secs 2023-03-09T22:38:55.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27646.381893 secs 2023-03-09T22:38:55.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27646.381893 secs 2023-03-09T22:38:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:56 smithi122 ceph-mon[93869]: pgmap v15256: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 385 KiB/s wr, 9 op/s 2023-03-09T22:38:56.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:56 smithi181 ceph-mon[94189]: pgmap v15256: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 385 KiB/s wr, 9 op/s 2023-03-09T22:38:56.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:56 smithi038 ceph-mon[92422]: pgmap v15256: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 385 KiB/s wr, 9 op/s 2023-03-09T22:38:57.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:38:57.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:38:57.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:38:57.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:38:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:38:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:38:58.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:38:58 smithi122 ceph-mon[93869]: pgmap v15257: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 612 KiB/s rd, 598 KiB/s wr, 12 op/s 2023-03-09T22:38:58.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:38:58 smithi181 ceph-mon[94189]: pgmap v15257: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 612 KiB/s rd, 598 KiB/s wr, 12 op/s 2023-03-09T22:38:58.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:38:58 smithi038 ceph-mon[92422]: pgmap v15257: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 612 KiB/s rd, 598 KiB/s wr, 12 op/s 2023-03-09T22:38:58.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28110.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:39:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:39:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:39:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:00 smithi122 ceph-mon[93869]: pgmap v15258: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 647 KiB/s rd, 634 KiB/s wr, 11 op/s 2023-03-09T22:39:00.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27651.381954 secs 2023-03-09T22:39:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:39:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:39:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:00 smithi181 ceph-mon[94189]: pgmap v15258: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 647 KiB/s rd, 634 KiB/s wr, 11 op/s 2023-03-09T22:39:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27651.381954 secs 2023-03-09T22:39:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:39:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:39:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:00 smithi038 ceph-mon[92422]: pgmap v15258: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 647 KiB/s rd, 634 KiB/s wr, 11 op/s 2023-03-09T22:39:00.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27651.381954 secs 2023-03-09T22:39:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:39:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:39:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:01 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:39:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:39:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:39:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:01 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:39:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:39:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:39:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:01 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:39:02.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:02 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:39:02.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:02 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:39:02.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:02 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:39:02.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:02 smithi122 ceph-mon[93869]: pgmap v15259: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 646 KiB/s rd, 653 KiB/s wr, 15 op/s 2023-03-09T22:39:02.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:02 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:39:02.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:02 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:39:02.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:02 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:39:02.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:02 smithi181 ceph-mon[94189]: pgmap v15259: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 646 KiB/s rd, 653 KiB/s wr, 15 op/s 2023-03-09T22:39:02.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:02 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:39:02.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:02 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:39:02.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:02 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:39:02.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:02 smithi038 ceph-mon[92422]: pgmap v15259: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 646 KiB/s rd, 653 KiB/s wr, 15 op/s 2023-03-09T22:39:03.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27653.756464 secs 2023-03-09T22:39:03.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27653.756464 secs 2023-03-09T22:39:03.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27653.756464 secs 2023-03-09T22:39:03.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28115.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:39:04.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:04 smithi122 ceph-mon[93869]: pgmap v15260: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 507 KiB/s rd, 531 KiB/s wr, 26 op/s 2023-03-09T22:39:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:04 smithi181 ceph-mon[94189]: pgmap v15260: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 507 KiB/s rd, 531 KiB/s wr, 26 op/s 2023-03-09T22:39:04.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:04 smithi038 ceph-mon[92422]: pgmap v15260: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 507 KiB/s rd, 531 KiB/s wr, 26 op/s 2023-03-09T22:39:05.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:05 smithi122 ceph-mon[93869]: osdmap e1169: 12 total, 12 up, 12 in 2023-03-09T22:39:05.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27656.381989 secs 2023-03-09T22:39:05.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:05 smithi181 ceph-mon[94189]: osdmap e1169: 12 total, 12 up, 12 in 2023-03-09T22:39:05.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27656.381989 secs 2023-03-09T22:39:05.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:05 smithi038 ceph-mon[92422]: osdmap e1169: 12 total, 12 up, 12 in 2023-03-09T22:39:05.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27656.381989 secs 2023-03-09T22:39:06.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:06 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:39:06.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:39:06.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:06 smithi122 ceph-mon[93869]: pgmap v15262: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 742 KiB/s rd, 784 KiB/s wr, 30 op/s 2023-03-09T22:39:06.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:06 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:39:06.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:39:06.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:06 smithi181 ceph-mon[94189]: pgmap v15262: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 742 KiB/s rd, 784 KiB/s wr, 30 op/s 2023-03-09T22:39:06.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:06 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:39:06.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:39:06.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:06 smithi038 ceph-mon[92422]: pgmap v15262: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 742 KiB/s rd, 784 KiB/s wr, 30 op/s 2023-03-09T22:39:07.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:07 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:39:07.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:07 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:39:07.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:07 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:39:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:08 smithi122 ceph-mon[93869]: pgmap v15263: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 373 KiB/s wr, 29 op/s 2023-03-09T22:39:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27658.756563 secs 2023-03-09T22:39:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:08 smithi181 ceph-mon[94189]: pgmap v15263: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 373 KiB/s wr, 29 op/s 2023-03-09T22:39:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27658.756563 secs 2023-03-09T22:39:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:08 smithi038 ceph-mon[92422]: pgmap v15263: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 373 KiB/s wr, 29 op/s 2023-03-09T22:39:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27658.756563 secs 2023-03-09T22:39:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28120.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:39:10.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:10 smithi122 ceph-mon[93869]: pgmap v15264: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 466 KiB/s wr, 28 op/s 2023-03-09T22:39:10.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27661.382071 secs 2023-03-09T22:39:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:10 smithi181 ceph-mon[94189]: pgmap v15264: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 466 KiB/s wr, 28 op/s 2023-03-09T22:39:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27661.382071 secs 2023-03-09T22:39:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:10 smithi038 ceph-mon[92422]: pgmap v15264: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 466 KiB/s wr, 28 op/s 2023-03-09T22:39:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27661.382071 secs 2023-03-09T22:39:11.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:11 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:39:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:11 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:39:11.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:11 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:39:12.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:12 smithi122 ceph-mon[93869]: pgmap v15265: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 466 KiB/s rd, 481 KiB/s wr, 25 op/s 2023-03-09T22:39:12.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:12 smithi122 ceph-mon[93869]: osdmap e1170: 12 total, 12 up, 12 in 2023-03-09T22:39:12.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27663.756631 secs 2023-03-09T22:39:12.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:12 smithi181 ceph-mon[94189]: pgmap v15265: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 466 KiB/s rd, 481 KiB/s wr, 25 op/s 2023-03-09T22:39:12.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:12 smithi181 ceph-mon[94189]: osdmap e1170: 12 total, 12 up, 12 in 2023-03-09T22:39:12.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27663.756631 secs 2023-03-09T22:39:12.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:12 smithi038 ceph-mon[92422]: pgmap v15265: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 466 KiB/s rd, 481 KiB/s wr, 25 op/s 2023-03-09T22:39:12.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:12 smithi038 ceph-mon[92422]: osdmap e1170: 12 total, 12 up, 12 in 2023-03-09T22:39:12.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27663.756631 secs 2023-03-09T22:39:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28125.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:39:14.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:14 smithi122 ceph-mon[93869]: pgmap v15267: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 247 KiB/s rd, 245 KiB/s wr, 8 op/s 2023-03-09T22:39:14.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:14 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:39:14.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:14 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:39:14.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:14 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:39:14.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:14 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:39:14.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:14 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:39:14.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:14 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:39:14.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:14 smithi181 ceph-mon[94189]: pgmap v15267: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 247 KiB/s rd, 245 KiB/s wr, 8 op/s 2023-03-09T22:39:14.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:14 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:39:14.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:14 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:39:14.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:14 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:39:14.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:14 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:39:14.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:14 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:39:14.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:14 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:39:14.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:14 smithi038 ceph-mon[92422]: pgmap v15267: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 247 KiB/s rd, 245 KiB/s wr, 8 op/s 2023-03-09T22:39:14.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:14 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:39:14.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:14 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:39:14.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:14 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:39:14.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:14 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:39:14.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:14 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:39:14.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:14 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:39:15.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:39:15.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:39:15.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:39:15.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:39:15.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:39:15.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:39:15.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:39:15.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:39:15.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:39:15.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:39:15.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:39:15.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:39:15.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:39:15.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:39:15.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:39:15.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27666.382132 secs 2023-03-09T22:39:15.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:39:15.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:39:15.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:39:15.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:39:15.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:39:15.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:39:15.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:39:15.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:39:15.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:39:15.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:39:15.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:39:15.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:39:15.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:39:15.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:39:15.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:39:15.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27666.382132 secs 2023-03-09T22:39:15.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:39:15.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:39:15.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:39:15.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:39:15.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:39:15.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:39:15.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:39:15.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:39:15.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:39:15.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:39:15.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:39:15.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:39:15.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:39:15.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:39:15.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:39:15.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27666.382132 secs 2023-03-09T22:39:16.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:16 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T22:39:16.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:16 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi038 to 701.8k 2023-03-09T22:39:16.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:16 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T22:39:16.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:16 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi038 to 718651: error parsing value: Value '718651' is below minimum 939524096 2023-03-09T22:39:16.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:16 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T22:39:16.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:16 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T22:39:16.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:16 smithi122 ceph-mon[93869]: pgmap v15268: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 234 KiB/s wr, 8 op/s 2023-03-09T22:39:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:16 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T22:39:16.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:16 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi038 to 701.8k 2023-03-09T22:39:16.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:16 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T22:39:16.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:16 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi038 to 718651: error parsing value: Value '718651' is below minimum 939524096 2023-03-09T22:39:16.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:16 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T22:39:16.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:16 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T22:39:16.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:16 smithi181 ceph-mon[94189]: pgmap v15268: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 234 KiB/s wr, 8 op/s 2023-03-09T22:39:16.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:16 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T22:39:16.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:16 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi038 to 701.8k 2023-03-09T22:39:16.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:16 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T22:39:16.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:16 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi038 to 718651: error parsing value: Value '718651' is below minimum 939524096 2023-03-09T22:39:16.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:16 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T22:39:16.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:16 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T22:39:16.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:16 smithi038 ceph-mon[92422]: pgmap v15268: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 234 KiB/s wr, 8 op/s 2023-03-09T22:39:17.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:17 smithi122 ceph-mon[93869]: pgmap v15269: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 234 KiB/s wr, 8 op/s 2023-03-09T22:39:17.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27668.756746 secs 2023-03-09T22:39:17.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:17 smithi181 ceph-mon[94189]: pgmap v15269: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 234 KiB/s wr, 8 op/s 2023-03-09T22:39:17.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27668.756746 secs 2023-03-09T22:39:17.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:17 smithi038 ceph-mon[92422]: pgmap v15269: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 234 KiB/s wr, 8 op/s 2023-03-09T22:39:17.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27668.756746 secs 2023-03-09T22:39:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28130.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:39:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:20 smithi122 ceph-mon[93869]: pgmap v15270: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 97 KiB/s wr, 7 op/s 2023-03-09T22:39:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27671.382193 secs 2023-03-09T22:39:20.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:20 smithi181 ceph-mon[94189]: pgmap v15270: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 97 KiB/s wr, 7 op/s 2023-03-09T22:39:20.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27671.382193 secs 2023-03-09T22:39:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:20 smithi038 ceph-mon[92422]: pgmap v15270: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 97 KiB/s wr, 7 op/s 2023-03-09T22:39:20.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27671.382193 secs 2023-03-09T22:39:22.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:22 smithi122 ceph-mon[93869]: pgmap v15271: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 33 KiB/s wr, 7 op/s 2023-03-09T22:39:22.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27673.756894 secs 2023-03-09T22:39:22.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:22 smithi181 ceph-mon[94189]: pgmap v15271: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 33 KiB/s wr, 7 op/s 2023-03-09T22:39:22.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27673.756894 secs 2023-03-09T22:39:22.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:22 smithi038 ceph-mon[92422]: pgmap v15271: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 33 KiB/s wr, 7 op/s 2023-03-09T22:39:22.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27673.756894 secs 2023-03-09T22:39:23.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28135.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:39:24.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:24 smithi122 ceph-mon[93869]: pgmap v15272: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 28 KiB/s wr, 6 op/s 2023-03-09T22:39:24.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:24 smithi181 ceph-mon[94189]: pgmap v15272: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 28 KiB/s wr, 6 op/s 2023-03-09T22:39:24.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:24 smithi038 ceph-mon[92422]: pgmap v15272: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 28 KiB/s wr, 6 op/s 2023-03-09T22:39:25.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27676.382252 secs 2023-03-09T22:39:25.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27676.382252 secs 2023-03-09T22:39:25.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27676.382252 secs 2023-03-09T22:39:26.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:26 smithi122 ceph-mon[93869]: pgmap v15273: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T22:39:26.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:26 smithi181 ceph-mon[94189]: pgmap v15273: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T22:39:26.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:26 smithi038 ceph-mon[92422]: pgmap v15273: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 28 KiB/s wr, 7 op/s 2023-03-09T22:39:27.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:27 smithi122 ceph-mon[93869]: pgmap v15274: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 8 op/s 2023-03-09T22:39:27.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27678.757004 secs 2023-03-09T22:39:27.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:27 smithi181 ceph-mon[94189]: pgmap v15274: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 8 op/s 2023-03-09T22:39:27.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27678.757004 secs 2023-03-09T22:39:27.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:27 smithi038 ceph-mon[92422]: pgmap v15274: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 8 op/s 2023-03-09T22:39:27.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27678.757004 secs 2023-03-09T22:39:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28140.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:39:30.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:30 smithi122 ceph-mon[93869]: pgmap v15275: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T22:39:30.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27681.382311 secs 2023-03-09T22:39:30.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:30 smithi181 ceph-mon[94189]: pgmap v15275: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T22:39:30.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27681.382311 secs 2023-03-09T22:39:30.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:30 smithi038 ceph-mon[92422]: pgmap v15275: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T22:39:30.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27681.382311 secs 2023-03-09T22:39:32.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:32 smithi122 ceph-mon[93869]: pgmap v15276: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T22:39:32.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27683.757144 secs 2023-03-09T22:39:32.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:32 smithi181 ceph-mon[94189]: pgmap v15276: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T22:39:32.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27683.757144 secs 2023-03-09T22:39:32.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:32 smithi038 ceph-mon[92422]: pgmap v15276: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T22:39:32.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27683.757144 secs 2023-03-09T22:39:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28145.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:39:34.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:34 smithi122 ceph-mon[93869]: pgmap v15277: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-09T22:39:34.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:34 smithi181 ceph-mon[94189]: pgmap v15277: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-09T22:39:34.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:34 smithi038 ceph-mon[92422]: pgmap v15277: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-09T22:39:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27686.382370 secs 2023-03-09T22:39:35.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27686.382370 secs 2023-03-09T22:39:35.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27686.382370 secs 2023-03-09T22:39:36.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:36 smithi122 ceph-mon[93869]: pgmap v15278: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 511 B/s wr, 7 op/s 2023-03-09T22:39:36.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:36 smithi181 ceph-mon[94189]: pgmap v15278: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 511 B/s wr, 7 op/s 2023-03-09T22:39:36.753 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:36 smithi038 ceph-mon[92422]: pgmap v15278: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 511 B/s wr, 7 op/s 2023-03-09T22:39:37.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:37 smithi122 ceph-mon[93869]: pgmap v15279: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 8 op/s 2023-03-09T22:39:37.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27688.757277 secs 2023-03-09T22:39:37.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:37 smithi181 ceph-mon[94189]: pgmap v15279: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 8 op/s 2023-03-09T22:39:37.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27688.757277 secs 2023-03-09T22:39:37.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:37 smithi038 ceph-mon[92422]: pgmap v15279: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 8 op/s 2023-03-09T22:39:37.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27688.757277 secs 2023-03-09T22:39:38.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28150.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:39:39.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:39 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:39:39.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:39 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:39:39.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:39 smithi122 ceph-mon[93869]: pgmap v15280: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 682 B/s wr, 7 op/s 2023-03-09T22:39:39.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:39 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:39:39.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:39 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:39:39.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:39 smithi181 ceph-mon[94189]: pgmap v15280: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 682 B/s wr, 7 op/s 2023-03-09T22:39:40.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:39 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:39:40.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:39 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:39:40.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:39 smithi038 ceph-mon[92422]: pgmap v15280: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 682 B/s wr, 7 op/s 2023-03-09T22:39:40.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27691.382454 secs 2023-03-09T22:39:40.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27691.382454 secs 2023-03-09T22:39:41.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27691.382454 secs 2023-03-09T22:39:41.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:39:41.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:39:41.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:41 smithi122 ceph-mon[93869]: pgmap v15281: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 232 KiB/s wr, 9 op/s 2023-03-09T22:39:41.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:39:41.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:39:41.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:41 smithi181 ceph-mon[94189]: pgmap v15281: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 232 KiB/s wr, 9 op/s 2023-03-09T22:39:42.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:39:42.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:39:42.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:41 smithi038 ceph-mon[92422]: pgmap v15281: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 232 KiB/s wr, 9 op/s 2023-03-09T22:39:43.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28155.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:39:44.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:44 smithi122 ceph-mon[93869]: pgmap v15282: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 232 KiB/s wr, 7 op/s 2023-03-09T22:39:44.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:44 smithi181 ceph-mon[94189]: pgmap v15282: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 232 KiB/s wr, 7 op/s 2023-03-09T22:39:44.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:44 smithi038 ceph-mon[92422]: pgmap v15282: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 232 KiB/s wr, 7 op/s 2023-03-09T22:39:45.737 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27696.382515 secs 2023-03-09T22:39:45.738 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:45 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:39:45.739 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27696.382515 secs 2023-03-09T22:39:45.739 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:45 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:39:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27696.382515 secs 2023-03-09T22:39:45.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:45 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:39:46.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:46 smithi122 ceph-mon[93869]: pgmap v15283: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 232 KiB/s wr, 7 op/s 2023-03-09T22:39:46.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:46 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:39:46.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:46 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:39:46.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:46 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:39:46.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:46 smithi181 ceph-mon[94189]: pgmap v15283: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 232 KiB/s wr, 7 op/s 2023-03-09T22:39:46.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:46 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:39:46.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:46 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:39:46.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:46 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:39:46.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:46 smithi038 ceph-mon[92422]: pgmap v15283: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 232 KiB/s wr, 7 op/s 2023-03-09T22:39:46.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:46 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:39:46.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:46 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:39:46.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:46 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:39:47.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:47 smithi122 ceph-mon[93869]: pgmap v15284: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 293 KiB/s rd, 290 KiB/s wr, 9 op/s 2023-03-09T22:39:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:47 smithi181 ceph-mon[94189]: pgmap v15284: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 293 KiB/s rd, 290 KiB/s wr, 9 op/s 2023-03-09T22:39:47.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:47 smithi038 ceph-mon[92422]: pgmap v15284: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 293 KiB/s rd, 290 KiB/s wr, 9 op/s 2023-03-09T22:39:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28160.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:39:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:50 smithi122 ceph-mon[93869]: pgmap v15285: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 290 KiB/s wr, 6 op/s 2023-03-09T22:39:50.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27701.382575 secs 2023-03-09T22:39:50.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:50 smithi181 ceph-mon[94189]: pgmap v15285: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 290 KiB/s wr, 6 op/s 2023-03-09T22:39:50.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27701.382575 secs 2023-03-09T22:39:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:50 smithi038 ceph-mon[92422]: pgmap v15285: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 290 KiB/s wr, 6 op/s 2023-03-09T22:39:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27701.382575 secs 2023-03-09T22:39:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:52 smithi122 ceph-mon[93869]: pgmap v15286: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 724 KiB/s rd, 514 KiB/s wr, 10 op/s 2023-03-09T22:39:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:52 smithi181 ceph-mon[94189]: pgmap v15286: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 724 KiB/s rd, 514 KiB/s wr, 10 op/s 2023-03-09T22:39:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:52 smithi038 ceph-mon[92422]: pgmap v15286: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 724 KiB/s rd, 514 KiB/s wr, 10 op/s 2023-03-09T22:39:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:39:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:39:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:53 smithi122 ceph-mon[93869]: pgmap v15287: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 283 KiB/s wr, 7 op/s 2023-03-09T22:39:53.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:39:53.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:39:53.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:53 smithi181 ceph-mon[94189]: pgmap v15287: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 283 KiB/s wr, 7 op/s 2023-03-09T22:39:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:39:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:39:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:53 smithi038 ceph-mon[92422]: pgmap v15287: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 283 KiB/s wr, 7 op/s 2023-03-09T22:39:53.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28165.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:39:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:39:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:39:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27706.382637 secs 2023-03-09T22:39:55.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:55 smithi122 ceph-mon[93869]: pgmap v15288: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 654 KiB/s rd, 446 KiB/s wr, 8 op/s 2023-03-09T22:39:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:39:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:39:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27706.382637 secs 2023-03-09T22:39:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:55 smithi181 ceph-mon[94189]: pgmap v15288: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 654 KiB/s rd, 446 KiB/s wr, 8 op/s 2023-03-09T22:39:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:39:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:39:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27706.382637 secs 2023-03-09T22:39:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:55 smithi038 ceph-mon[92422]: pgmap v15288: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 654 KiB/s rd, 446 KiB/s wr, 8 op/s 2023-03-09T22:39:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:57 smithi122 ceph-mon[93869]: pgmap v15289: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 926 KiB/s rd, 715 KiB/s wr, 11 op/s 2023-03-09T22:39:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:57 smithi181 ceph-mon[94189]: pgmap v15289: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 926 KiB/s rd, 715 KiB/s wr, 11 op/s 2023-03-09T22:39:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:57 smithi038 ceph-mon[92422]: pgmap v15289: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 926 KiB/s rd, 715 KiB/s wr, 11 op/s 2023-03-09T22:39:58.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28170.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:39:59.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:39:59.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:39:59.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:39:59 smithi122 ceph-mon[93869]: pgmap v15290: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 878 KiB/s rd, 658 KiB/s wr, 10 op/s 2023-03-09T22:39:59.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:39:59.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:39:59.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:39:59 smithi181 ceph-mon[94189]: pgmap v15290: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 878 KiB/s rd, 658 KiB/s wr, 10 op/s 2023-03-09T22:39:59.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:39:59.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:39:59.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:39:59 smithi038 ceph-mon[92422]: pgmap v15290: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 878 KiB/s rd, 658 KiB/s wr, 10 op/s 2023-03-09T22:40:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27711.382698 secs 2023-03-09T22:40:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:00 smithi122 ceph-mon[93869]: Health detail: HEALTH_WARN 1 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-09T22:40:00.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:00 smithi122 ceph-mon[93869]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2023-03-09T22:40:00.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:00 smithi122 ceph-mon[93869]: mds.h(mds.1): 2 slow requests are blocked > 180 secs 2023-03-09T22:40:00.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:00 smithi122 ceph-mon[93869]: [WRN] MDS_TRIM: 2 MDSs behind on trimming 2023-03-09T22:40:00.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:00 smithi122 ceph-mon[93869]: mds.e(mds.0): Behind on trimming (385/128) max_segments: 128, num_segments: 385 2023-03-09T22:40:00.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:00 smithi122 ceph-mon[93869]: mds.b(mds.0): Behind on trimming (383/128) max_segments: 128, num_segments: 383 2023-03-09T22:40:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27711.382698 secs 2023-03-09T22:40:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:00 smithi181 ceph-mon[94189]: Health detail: HEALTH_WARN 1 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-09T22:40:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:00 smithi181 ceph-mon[94189]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2023-03-09T22:40:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:00 smithi181 ceph-mon[94189]: mds.h(mds.1): 2 slow requests are blocked > 180 secs 2023-03-09T22:40:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:00 smithi181 ceph-mon[94189]: [WRN] MDS_TRIM: 2 MDSs behind on trimming 2023-03-09T22:40:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:00 smithi181 ceph-mon[94189]: mds.e(mds.0): Behind on trimming (385/128) max_segments: 128, num_segments: 385 2023-03-09T22:40:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:00 smithi181 ceph-mon[94189]: mds.b(mds.0): Behind on trimming (383/128) max_segments: 128, num_segments: 383 2023-03-09T22:40:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27711.382698 secs 2023-03-09T22:40:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:00 smithi038 ceph-mon[92422]: Health detail: HEALTH_WARN 1 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-09T22:40:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:00 smithi038 ceph-mon[92422]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2023-03-09T22:40:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:00 smithi038 ceph-mon[92422]: mds.h(mds.1): 2 slow requests are blocked > 180 secs 2023-03-09T22:40:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:00 smithi038 ceph-mon[92422]: [WRN] MDS_TRIM: 2 MDSs behind on trimming 2023-03-09T22:40:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:00 smithi038 ceph-mon[92422]: mds.e(mds.0): Behind on trimming (385/128) max_segments: 128, num_segments: 385 2023-03-09T22:40:00.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:00 smithi038 ceph-mon[92422]: mds.b(mds.0): Behind on trimming (383/128) max_segments: 128, num_segments: 383 2023-03-09T22:40:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:40:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:40:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:01 smithi122 ceph-mon[93869]: pgmap v15291: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 17 op/s 2023-03-09T22:40:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:40:01.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:40:01.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:01 smithi181 ceph-mon[94189]: pgmap v15291: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 17 op/s 2023-03-09T22:40:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:40:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:40:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:01 smithi038 ceph-mon[92422]: pgmap v15291: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 17 op/s 2023-03-09T22:40:02.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27713.757903 secs 2023-03-09T22:40:02.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27713.757903 secs 2023-03-09T22:40:02.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27713.757903 secs 2023-03-09T22:40:03.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:03 smithi038 ceph-mon[92422]: pgmap v15292: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 871 KiB/s rd, 895 KiB/s wr, 26 op/s 2023-03-09T22:40:03.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28175.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:40:03.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:03 smithi122 ceph-mon[93869]: pgmap v15292: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 871 KiB/s rd, 895 KiB/s wr, 26 op/s 2023-03-09T22:40:03.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:03 smithi181 ceph-mon[94189]: pgmap v15292: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 871 KiB/s rd, 895 KiB/s wr, 26 op/s 2023-03-09T22:40:05.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27716.382790 secs 2023-03-09T22:40:05.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:05 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:40:05.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:05 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:40:05.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:05 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:40:05.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27716.382790 secs 2023-03-09T22:40:05.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:05 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:40:05.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:05 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:40:05.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:05 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:40:05.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27716.382790 secs 2023-03-09T22:40:05.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:05 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:40:05.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:05 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:40:05.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:05 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:40:06.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:06 smithi122 ceph-mon[93869]: pgmap v15293: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 891 KiB/s rd, 914 KiB/s wr, 27 op/s 2023-03-09T22:40:06.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:06 smithi181 ceph-mon[94189]: pgmap v15293: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 891 KiB/s rd, 914 KiB/s wr, 27 op/s 2023-03-09T22:40:06.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:06 smithi038 ceph-mon[92422]: pgmap v15293: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 891 KiB/s rd, 914 KiB/s wr, 27 op/s 2023-03-09T22:40:07.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27718.758016 secs 2023-03-09T22:40:07.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:07 smithi122 ceph-mon[93869]: pgmap v15294: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 731 KiB/s rd, 753 KiB/s wr, 29 op/s 2023-03-09T22:40:07.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27718.758016 secs 2023-03-09T22:40:07.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:07 smithi181 ceph-mon[94189]: pgmap v15294: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 731 KiB/s rd, 753 KiB/s wr, 29 op/s 2023-03-09T22:40:07.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27718.758016 secs 2023-03-09T22:40:07.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:07 smithi038 ceph-mon[92422]: pgmap v15294: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 731 KiB/s rd, 753 KiB/s wr, 29 op/s 2023-03-09T22:40:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28180.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:40:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:10 smithi122 ceph-mon[93869]: osdmap e1171: 12 total, 12 up, 12 in 2023-03-09T22:40:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:10 smithi122 ceph-mon[93869]: pgmap v15296: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 538 KiB/s rd, 579 KiB/s wr, 30 op/s 2023-03-09T22:40:10.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27721.382851 secs 2023-03-09T22:40:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:10 smithi181 ceph-mon[94189]: osdmap e1171: 12 total, 12 up, 12 in 2023-03-09T22:40:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:10 smithi181 ceph-mon[94189]: pgmap v15296: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 538 KiB/s rd, 579 KiB/s wr, 30 op/s 2023-03-09T22:40:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27721.382851 secs 2023-03-09T22:40:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:10 smithi038 ceph-mon[92422]: osdmap e1171: 12 total, 12 up, 12 in 2023-03-09T22:40:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:10 smithi038 ceph-mon[92422]: pgmap v15296: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 538 KiB/s rd, 579 KiB/s wr, 30 op/s 2023-03-09T22:40:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27721.382851 secs 2023-03-09T22:40:12.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:40:12.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:40:12.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:12 smithi122 ceph-mon[93869]: pgmap v15297: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 462 KiB/s wr, 26 op/s 2023-03-09T22:40:12.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:40:12.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:40:12.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:12 smithi181 ceph-mon[94189]: pgmap v15297: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 462 KiB/s wr, 26 op/s 2023-03-09T22:40:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:40:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:40:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:12 smithi038 ceph-mon[92422]: pgmap v15297: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 462 KiB/s wr, 26 op/s 2023-03-09T22:40:13.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27723.758145 secs 2023-03-09T22:40:13.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:40:13.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:40:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27723.758145 secs 2023-03-09T22:40:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:40:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:40:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27723.758145 secs 2023-03-09T22:40:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:40:13.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:40:13.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28185.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:40:14.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:14 smithi122 ceph-mon[93869]: pgmap v15298: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 349 KiB/s wr, 11 op/s 2023-03-09T22:40:14.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:14 smithi181 ceph-mon[94189]: pgmap v15298: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 349 KiB/s wr, 11 op/s 2023-03-09T22:40:14.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:14 smithi038 ceph-mon[92422]: pgmap v15298: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 349 KiB/s wr, 11 op/s 2023-03-09T22:40:15.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:40:15.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27726.382910 secs 2023-03-09T22:40:15.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:40:15.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27726.382910 secs 2023-03-09T22:40:15.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:40:15.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27726.382910 secs 2023-03-09T22:40:16.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:16 smithi122 ceph-mon[93869]: pgmap v15299: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 329 KiB/s rd, 326 KiB/s wr, 10 op/s 2023-03-09T22:40:16.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:16 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:40:16.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:16 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:40:16.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:16 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:40:16.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:16 smithi181 ceph-mon[94189]: pgmap v15299: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 329 KiB/s rd, 326 KiB/s wr, 10 op/s 2023-03-09T22:40:16.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:16 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:40:16.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:16 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:40:16.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:16 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:40:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:16 smithi038 ceph-mon[92422]: pgmap v15299: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 329 KiB/s rd, 326 KiB/s wr, 10 op/s 2023-03-09T22:40:16.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:16 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:40:16.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:16 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:40:16.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:16 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:40:17.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:17 smithi122 ceph-mon[93869]: osdmap e1172: 12 total, 12 up, 12 in 2023-03-09T22:40:17.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27728.758263 secs 2023-03-09T22:40:17.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:17 smithi122 ceph-mon[93869]: pgmap v15301: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 515 KiB/s wr, 11 op/s 2023-03-09T22:40:17.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:17 smithi181 ceph-mon[94189]: osdmap e1172: 12 total, 12 up, 12 in 2023-03-09T22:40:17.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27728.758263 secs 2023-03-09T22:40:17.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:17 smithi181 ceph-mon[94189]: pgmap v15301: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 515 KiB/s wr, 11 op/s 2023-03-09T22:40:17.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:17 smithi038 ceph-mon[92422]: osdmap e1172: 12 total, 12 up, 12 in 2023-03-09T22:40:17.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27728.758263 secs 2023-03-09T22:40:17.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:17 smithi038 ceph-mon[92422]: pgmap v15301: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 515 KiB/s wr, 11 op/s 2023-03-09T22:40:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28190.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:40:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:20 smithi122 ceph-mon[93869]: pgmap v15302: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 420 KiB/s wr, 9 op/s 2023-03-09T22:40:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27731.382976 secs 2023-03-09T22:40:20.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:20 smithi181 ceph-mon[94189]: pgmap v15302: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 420 KiB/s wr, 9 op/s 2023-03-09T22:40:20.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27731.382976 secs 2023-03-09T22:40:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:20 smithi038 ceph-mon[92422]: pgmap v15302: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 420 KiB/s wr, 9 op/s 2023-03-09T22:40:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27731.382976 secs 2023-03-09T22:40:22.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:22 smithi122 ceph-mon[93869]: pgmap v15303: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 97 KiB/s wr, 7 op/s 2023-03-09T22:40:22.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27733.758380 secs 2023-03-09T22:40:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:22 smithi181 ceph-mon[94189]: pgmap v15303: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 97 KiB/s wr, 7 op/s 2023-03-09T22:40:22.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27733.758380 secs 2023-03-09T22:40:22.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:22 smithi038 ceph-mon[92422]: pgmap v15303: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 97 KiB/s wr, 7 op/s 2023-03-09T22:40:22.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27733.758380 secs 2023-03-09T22:40:23.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28195.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:40:24.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:24 smithi122 ceph-mon[93869]: pgmap v15304: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 97 KiB/s wr, 6 op/s 2023-03-09T22:40:24.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:24 smithi181 ceph-mon[94189]: pgmap v15304: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 97 KiB/s wr, 6 op/s 2023-03-09T22:40:24.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:24 smithi038 ceph-mon[92422]: pgmap v15304: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 97 KiB/s wr, 6 op/s 2023-03-09T22:40:25.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27736.382985 secs 2023-03-09T22:40:25.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27736.382985 secs 2023-03-09T22:40:25.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27736.382985 secs 2023-03-09T22:40:26.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:26 smithi122 ceph-mon[93869]: pgmap v15305: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 97 KiB/s wr, 6 op/s 2023-03-09T22:40:26.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:26 smithi181 ceph-mon[94189]: pgmap v15305: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 97 KiB/s wr, 6 op/s 2023-03-09T22:40:26.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:26 smithi038 ceph-mon[92422]: pgmap v15305: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 97 KiB/s wr, 6 op/s 2023-03-09T22:40:27.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27738.758490 secs 2023-03-09T22:40:27.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:27 smithi122 ceph-mon[93869]: pgmap v15306: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T22:40:27.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27738.758490 secs 2023-03-09T22:40:27.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:27 smithi181 ceph-mon[94189]: pgmap v15306: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T22:40:27.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27738.758490 secs 2023-03-09T22:40:27.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:27 smithi038 ceph-mon[92422]: pgmap v15306: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T22:40:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28200.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:40:30.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:30 smithi122 ceph-mon[93869]: pgmap v15307: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T22:40:30.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27741.383046 secs 2023-03-09T22:40:30.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:30 smithi181 ceph-mon[94189]: pgmap v15307: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T22:40:30.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27741.383046 secs 2023-03-09T22:40:30.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:30 smithi038 ceph-mon[92422]: pgmap v15307: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T22:40:30.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27741.383046 secs 2023-03-09T22:40:32.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:32 smithi122 ceph-mon[93869]: pgmap v15308: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T22:40:32.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27743.758618 secs 2023-03-09T22:40:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:32 smithi181 ceph-mon[94189]: pgmap v15308: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T22:40:32.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27743.758618 secs 2023-03-09T22:40:32.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:32 smithi038 ceph-mon[92422]: pgmap v15308: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T22:40:32.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27743.758618 secs 2023-03-09T22:40:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28205.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:40:34.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:34 smithi122 ceph-mon[93869]: pgmap v15309: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T22:40:34.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:34 smithi181 ceph-mon[94189]: pgmap v15309: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T22:40:34.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:34 smithi038 ceph-mon[92422]: pgmap v15309: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T22:40:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27746.383108 secs 2023-03-09T22:40:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27746.383108 secs 2023-03-09T22:40:35.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27746.383108 secs 2023-03-09T22:40:36.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:36 smithi122 ceph-mon[93869]: pgmap v15310: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2023-03-09T22:40:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:36 smithi181 ceph-mon[94189]: pgmap v15310: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2023-03-09T22:40:36.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:36 smithi038 ceph-mon[92422]: pgmap v15310: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 5 op/s 2023-03-09T22:40:37.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27748.758754 secs 2023-03-09T22:40:37.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:37 smithi122 ceph-mon[93869]: pgmap v15311: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 426 B/s wr, 7 op/s 2023-03-09T22:40:37.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27748.758754 secs 2023-03-09T22:40:37.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:37 smithi181 ceph-mon[94189]: pgmap v15311: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 426 B/s wr, 7 op/s 2023-03-09T22:40:37.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27748.758754 secs 2023-03-09T22:40:37.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:37 smithi038 ceph-mon[92422]: pgmap v15311: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 426 B/s wr, 7 op/s 2023-03-09T22:40:38.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28210.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:40:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:40 smithi122 ceph-mon[93869]: pgmap v15312: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2023-03-09T22:40:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27751.383165 secs 2023-03-09T22:40:40.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:40 smithi181 ceph-mon[94189]: pgmap v15312: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2023-03-09T22:40:40.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27751.383165 secs 2023-03-09T22:40:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:40 smithi038 ceph-mon[92422]: pgmap v15312: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2023-03-09T22:40:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27751.383165 secs 2023-03-09T22:40:42.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:42 smithi122 ceph-mon[93869]: pgmap v15313: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 118 KiB/s wr, 9 op/s 2023-03-09T22:40:42.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:42 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 27753.758906 secs 2023-03-09T22:40:42.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:42 smithi181 ceph-mon[94189]: pgmap v15313: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 118 KiB/s wr, 9 op/s 2023-03-09T22:40:42.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:42 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 27753.758906 secs 2023-03-09T22:40:42.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:42 smithi038 ceph-mon[92422]: pgmap v15313: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 118 KiB/s wr, 9 op/s 2023-03-09T22:40:42.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:42 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 27753.758906 secs 2023-03-09T22:40:43.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28215.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:40:44.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:44 smithi122 ceph-mon[93869]: pgmap v15314: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 118 KiB/s wr, 6 op/s 2023-03-09T22:40:44.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:44 smithi181 ceph-mon[94189]: pgmap v15314: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 118 KiB/s wr, 6 op/s 2023-03-09T22:40:44.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:44 smithi038 ceph-mon[92422]: pgmap v15314: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 118 KiB/s wr, 6 op/s 2023-03-09T22:40:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27756.383223 secs 2023-03-09T22:40:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27756.383223 secs 2023-03-09T22:40:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27756.383223 secs 2023-03-09T22:40:46.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:46 smithi122 ceph-mon[93869]: pgmap v15315: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 119 KiB/s wr, 7 op/s 2023-03-09T22:40:46.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:46 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:40:46.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:46 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:40:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:46 smithi181 ceph-mon[94189]: pgmap v15315: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 119 KiB/s wr, 7 op/s 2023-03-09T22:40:46.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:46 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:40:46.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:46 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:40:46.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:46 smithi038 ceph-mon[92422]: pgmap v15315: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 119 KiB/s wr, 7 op/s 2023-03-09T22:40:46.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:46 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:40:46.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:46 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:40:47.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:47 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 27758.759029 secs 2023-03-09T22:40:47.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:47 smithi122 ceph-mon[93869]: pgmap v15316: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 357 KiB/s wr, 10 op/s 2023-03-09T22:40:47.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:47 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 27758.759029 secs 2023-03-09T22:40:47.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:47 smithi181 ceph-mon[94189]: pgmap v15316: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 357 KiB/s wr, 10 op/s 2023-03-09T22:40:47.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:47 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 27758.759029 secs 2023-03-09T22:40:47.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:47 smithi038 ceph-mon[92422]: pgmap v15316: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 357 KiB/s wr, 10 op/s 2023-03-09T22:40:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28220.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:40:49.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:40:49.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:40:49.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:49 smithi122 ceph-mon[93869]: pgmap v15317: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 357 KiB/s wr, 7 op/s 2023-03-09T22:40:49.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:40:49.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:40:49.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:49 smithi181 ceph-mon[94189]: pgmap v15317: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 357 KiB/s wr, 7 op/s 2023-03-09T22:40:49.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:40:49.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:40:49.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:49 smithi038 ceph-mon[92422]: pgmap v15317: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 357 KiB/s wr, 7 op/s 2023-03-09T22:40:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27761.383284 secs 2023-03-09T22:40:50.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27761.383284 secs 2023-03-09T22:40:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27761.383284 secs 2023-03-09T22:40:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:40:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:40:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:51 smithi038 ceph-mon[92422]: pgmap v15318: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 515 KiB/s wr, 11 op/s 2023-03-09T22:40:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:40:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:40:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:51 smithi122 ceph-mon[93869]: pgmap v15318: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 515 KiB/s wr, 11 op/s 2023-03-09T22:40:51.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:40:51.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:40:51.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:51 smithi181 ceph-mon[94189]: pgmap v15318: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 515 KiB/s wr, 11 op/s 2023-03-09T22:40:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:52 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 27763.759130 secs 2023-03-09T22:40:52.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:52 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 27763.759130 secs 2023-03-09T22:40:52.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:52 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 27763.759130 secs 2023-03-09T22:40:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:40:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:40:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:53 smithi038 ceph-mon[92422]: pgmap v15319: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 938 KiB/s rd, 397 KiB/s wr, 8 op/s 2023-03-09T22:40:53.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28225.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:40:53.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:40:53.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:40:53.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:53 smithi122 ceph-mon[93869]: pgmap v15319: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 938 KiB/s rd, 397 KiB/s wr, 8 op/s 2023-03-09T22:40:53.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:40:53.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:40:53.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:53 smithi181 ceph-mon[94189]: pgmap v15319: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 938 KiB/s rd, 397 KiB/s wr, 8 op/s 2023-03-09T22:40:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:40:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:40:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27766.383344 secs 2023-03-09T22:40:55.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:55 smithi122 ceph-mon[93869]: pgmap v15320: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 960 KiB/s rd, 421 KiB/s wr, 9 op/s 2023-03-09T22:40:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:40:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:40:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27766.383344 secs 2023-03-09T22:40:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:55 smithi181 ceph-mon[94189]: pgmap v15320: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 960 KiB/s rd, 421 KiB/s wr, 9 op/s 2023-03-09T22:40:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:40:56.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:40:56.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27766.383344 secs 2023-03-09T22:40:56.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:55 smithi038 ceph-mon[92422]: pgmap v15320: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 960 KiB/s rd, 421 KiB/s wr, 9 op/s 2023-03-09T22:40:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27768.759199 secs 2023-03-09T22:40:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:57 smithi122 ceph-mon[93869]: pgmap v15321: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 636 KiB/s wr, 12 op/s 2023-03-09T22:40:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27768.759199 secs 2023-03-09T22:40:57.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:57 smithi181 ceph-mon[94189]: pgmap v15321: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 636 KiB/s wr, 12 op/s 2023-03-09T22:40:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27768.759199 secs 2023-03-09T22:40:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:57 smithi038 ceph-mon[92422]: pgmap v15321: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 636 KiB/s wr, 12 op/s 2023-03-09T22:40:58.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28230.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:40:59.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:40:59.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:40:59.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:40:59 smithi181 ceph-mon[94189]: pgmap v15322: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 949 KiB/s rd, 398 KiB/s wr, 10 op/s 2023-03-09T22:40:59.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:40:59.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:40:59.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:40:59 smithi038 ceph-mon[92422]: pgmap v15322: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 949 KiB/s rd, 398 KiB/s wr, 10 op/s 2023-03-09T22:40:59.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:40:59.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:40:59.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:40:59 smithi122 ceph-mon[93869]: pgmap v15322: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 949 KiB/s rd, 398 KiB/s wr, 10 op/s 2023-03-09T22:41:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27771.383404 secs 2023-03-09T22:41:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27771.383404 secs 2023-03-09T22:41:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27771.383404 secs 2023-03-09T22:41:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:41:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:41:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:01 smithi038 ceph-mon[92422]: pgmap v15323: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 975 KiB/s rd, 420 KiB/s wr, 13 op/s 2023-03-09T22:41:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:41:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:41:01.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:01 smithi122 ceph-mon[93869]: pgmap v15323: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 975 KiB/s rd, 420 KiB/s wr, 13 op/s 2023-03-09T22:41:01.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:41:01.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:41:01.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:01 smithi181 ceph-mon[94189]: pgmap v15323: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 975 KiB/s rd, 420 KiB/s wr, 13 op/s 2023-03-09T22:41:02.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:02 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:41:02.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:02 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:41:02.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27773.759275 secs 2023-03-09T22:41:02.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:02 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:41:02.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:02 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:41:02.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27773.759275 secs 2023-03-09T22:41:02.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:02 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:41:02.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:02 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:41:02.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27773.759275 secs 2023-03-09T22:41:03.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28235.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:41:03.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:03 smithi122 ceph-mon[93869]: pgmap v15324: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 637 KiB/s rd, 595 KiB/s wr, 12 op/s 2023-03-09T22:41:03.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:03 smithi181 ceph-mon[94189]: pgmap v15324: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 637 KiB/s rd, 595 KiB/s wr, 12 op/s 2023-03-09T22:41:04.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:03 smithi038 ceph-mon[92422]: pgmap v15324: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 637 KiB/s rd, 595 KiB/s wr, 12 op/s 2023-03-09T22:41:05.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27776.383464 secs 2023-03-09T22:41:05.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27776.383464 secs 2023-03-09T22:41:05.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27776.383464 secs 2023-03-09T22:41:06.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:06 smithi122 ceph-mon[93869]: pgmap v15325: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 678 KiB/s rd, 636 KiB/s wr, 13 op/s 2023-03-09T22:41:06.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:06 smithi181 ceph-mon[94189]: pgmap v15325: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 678 KiB/s rd, 636 KiB/s wr, 13 op/s 2023-03-09T22:41:06.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:06 smithi038 ceph-mon[92422]: pgmap v15325: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 678 KiB/s rd, 636 KiB/s wr, 13 op/s 2023-03-09T22:41:07.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27778.759392 secs 2023-03-09T22:41:07.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:07 smithi122 ceph-mon[93869]: pgmap v15326: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 754 KiB/s rd, 689 KiB/s wr, 19 op/s 2023-03-09T22:41:07.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27778.759392 secs 2023-03-09T22:41:07.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:07 smithi181 ceph-mon[94189]: pgmap v15326: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 754 KiB/s rd, 689 KiB/s wr, 19 op/s 2023-03-09T22:41:07.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27778.759392 secs 2023-03-09T22:41:07.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:07 smithi038 ceph-mon[92422]: pgmap v15326: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 754 KiB/s rd, 689 KiB/s wr, 19 op/s 2023-03-09T22:41:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28240.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:41:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:10 smithi122 ceph-mon[93869]: osdmap e1173: 12 total, 12 up, 12 in 2023-03-09T22:41:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:10 smithi122 ceph-mon[93869]: pgmap v15328: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 639 KiB/s rd, 594 KiB/s wr, 30 op/s 2023-03-09T22:41:10.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27781.383524 secs 2023-03-09T22:41:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:10 smithi181 ceph-mon[94189]: osdmap e1173: 12 total, 12 up, 12 in 2023-03-09T22:41:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:10 smithi181 ceph-mon[94189]: pgmap v15328: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 639 KiB/s rd, 594 KiB/s wr, 30 op/s 2023-03-09T22:41:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27781.383524 secs 2023-03-09T22:41:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:10 smithi038 ceph-mon[92422]: osdmap e1173: 12 total, 12 up, 12 in 2023-03-09T22:41:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:10 smithi038 ceph-mon[92422]: pgmap v15328: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 639 KiB/s rd, 594 KiB/s wr, 30 op/s 2023-03-09T22:41:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27781.383524 secs 2023-03-09T22:41:12.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:41:12.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:41:12.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:12 smithi122 ceph-mon[93869]: pgmap v15329: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 610 KiB/s rd, 568 KiB/s wr, 29 op/s 2023-03-09T22:41:12.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:41:12.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:41:12.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:12 smithi181 ceph-mon[94189]: pgmap v15329: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 610 KiB/s rd, 568 KiB/s wr, 29 op/s 2023-03-09T22:41:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:41:12.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:41:12.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:12 smithi038 ceph-mon[92422]: pgmap v15329: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 610 KiB/s rd, 568 KiB/s wr, 29 op/s 2023-03-09T22:41:13.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27783.759502 secs 2023-03-09T22:41:13.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:41:13.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:41:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27783.759502 secs 2023-03-09T22:41:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:41:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:41:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27783.759502 secs 2023-03-09T22:41:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:41:13.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:41:13.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28245.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:41:14.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:14 smithi122 ceph-mon[93869]: pgmap v15330: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 492 KiB/s wr, 27 op/s 2023-03-09T22:41:14.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:14 smithi181 ceph-mon[94189]: pgmap v15330: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 492 KiB/s wr, 27 op/s 2023-03-09T22:41:14.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:14 smithi038 ceph-mon[92422]: pgmap v15330: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 492 KiB/s wr, 27 op/s 2023-03-09T22:41:15.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27786.383587 secs 2023-03-09T22:41:15.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27786.383587 secs 2023-03-09T22:41:15.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27786.383587 secs 2023-03-09T22:41:16.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:16 smithi122 ceph-mon[93869]: pgmap v15331: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 442 KiB/s wr, 27 op/s 2023-03-09T22:41:16.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:16 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:41:16.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:16 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:41:16.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:16 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:41:16.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:16 smithi181 ceph-mon[94189]: pgmap v15331: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 442 KiB/s wr, 27 op/s 2023-03-09T22:41:16.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:16 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:41:16.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:16 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:41:16.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:16 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:41:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:16 smithi038 ceph-mon[92422]: pgmap v15331: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 442 KiB/s wr, 27 op/s 2023-03-09T22:41:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:16 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:41:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:16 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:41:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:16 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:41:17.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:17 smithi122 ceph-mon[93869]: osdmap e1174: 12 total, 12 up, 12 in 2023-03-09T22:41:17.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:17 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:41:17.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:17 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:41:17.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:17 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:41:17.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27788.759623 secs 2023-03-09T22:41:17.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:17 smithi122 ceph-mon[93869]: pgmap v15333: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 515 KiB/s wr, 11 op/s 2023-03-09T22:41:17.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:17 smithi181 ceph-mon[94189]: osdmap e1174: 12 total, 12 up, 12 in 2023-03-09T22:41:17.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:17 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:41:17.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:17 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:41:17.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:17 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:41:17.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27788.759623 secs 2023-03-09T22:41:17.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:17 smithi181 ceph-mon[94189]: pgmap v15333: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 515 KiB/s wr, 11 op/s 2023-03-09T22:41:17.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:17 smithi038 ceph-mon[92422]: osdmap e1174: 12 total, 12 up, 12 in 2023-03-09T22:41:17.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:17 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:41:17.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:17 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:41:17.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:17 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:41:17.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27788.759623 secs 2023-03-09T22:41:17.753 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:17 smithi038 ceph-mon[92422]: pgmap v15333: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 515 KiB/s wr, 11 op/s 2023-03-09T22:41:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28250.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:41:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:20 smithi122 ceph-mon[93869]: pgmap v15334: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 422 KiB/s wr, 9 op/s 2023-03-09T22:41:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27791.383645 secs 2023-03-09T22:41:20.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:20 smithi181 ceph-mon[94189]: pgmap v15334: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 422 KiB/s wr, 9 op/s 2023-03-09T22:41:20.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27791.383645 secs 2023-03-09T22:41:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:20 smithi038 ceph-mon[92422]: pgmap v15334: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 422 KiB/s wr, 9 op/s 2023-03-09T22:41:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27791.383645 secs 2023-03-09T22:41:22.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:22 smithi122 ceph-mon[93869]: pgmap v15335: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 421 KiB/s wr, 8 op/s 2023-03-09T22:41:22.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27793.759726 secs 2023-03-09T22:41:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:22 smithi181 ceph-mon[94189]: pgmap v15335: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 421 KiB/s wr, 8 op/s 2023-03-09T22:41:22.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27793.759726 secs 2023-03-09T22:41:22.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:22 smithi038 ceph-mon[92422]: pgmap v15335: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 421 KiB/s wr, 8 op/s 2023-03-09T22:41:22.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27793.759726 secs 2023-03-09T22:41:23.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28255.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:41:24.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:24 smithi122 ceph-mon[93869]: pgmap v15336: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 97 KiB/s wr, 7 op/s 2023-03-09T22:41:24.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:24 smithi181 ceph-mon[94189]: pgmap v15336: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 97 KiB/s wr, 7 op/s 2023-03-09T22:41:24.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:24 smithi038 ceph-mon[92422]: pgmap v15336: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 97 KiB/s wr, 7 op/s 2023-03-09T22:41:25.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27796.383704 secs 2023-03-09T22:41:25.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27796.383704 secs 2023-03-09T22:41:25.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27796.383704 secs 2023-03-09T22:41:26.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:26 smithi122 ceph-mon[93869]: pgmap v15337: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 97 KiB/s wr, 7 op/s 2023-03-09T22:41:26.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:26 smithi181 ceph-mon[94189]: pgmap v15337: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 97 KiB/s wr, 7 op/s 2023-03-09T22:41:26.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:26 smithi038 ceph-mon[92422]: pgmap v15337: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 97 KiB/s wr, 7 op/s 2023-03-09T22:41:27.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27798.759831 secs 2023-03-09T22:41:27.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:27 smithi038 ceph-mon[92422]: pgmap v15338: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T22:41:27.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27798.759831 secs 2023-03-09T22:41:27.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:27 smithi122 ceph-mon[93869]: pgmap v15338: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T22:41:27.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27798.759831 secs 2023-03-09T22:41:27.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:27 smithi181 ceph-mon[94189]: pgmap v15338: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T22:41:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28260.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:41:30.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:30 smithi122 ceph-mon[93869]: pgmap v15339: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T22:41:30.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27801.383767 secs 2023-03-09T22:41:30.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:30 smithi181 ceph-mon[94189]: pgmap v15339: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T22:41:30.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27801.383767 secs 2023-03-09T22:41:30.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:30 smithi038 ceph-mon[92422]: pgmap v15339: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T22:41:30.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27801.383767 secs 2023-03-09T22:41:32.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:32 smithi122 ceph-mon[93869]: pgmap v15340: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T22:41:32.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27803.759963 secs 2023-03-09T22:41:32.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:32 smithi181 ceph-mon[94189]: pgmap v15340: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T22:41:32.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27803.759963 secs 2023-03-09T22:41:32.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:32 smithi038 ceph-mon[92422]: pgmap v15340: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T22:41:32.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27803.759963 secs 2023-03-09T22:41:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28265.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:41:34.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:34 smithi122 ceph-mon[93869]: pgmap v15341: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T22:41:34.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:34 smithi181 ceph-mon[94189]: pgmap v15341: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T22:41:34.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:34 smithi038 ceph-mon[92422]: pgmap v15341: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T22:41:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27806.383827 secs 2023-03-09T22:41:35.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27806.383827 secs 2023-03-09T22:41:35.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27806.383827 secs 2023-03-09T22:41:36.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:36 smithi122 ceph-mon[93869]: pgmap v15342: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T22:41:36.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:36 smithi181 ceph-mon[94189]: pgmap v15342: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T22:41:36.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:36 smithi038 ceph-mon[92422]: pgmap v15342: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T22:41:37.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:41:37.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:41:37.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27808.760092 secs 2023-03-09T22:41:37.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:37 smithi122 ceph-mon[93869]: pgmap v15343: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 98 KiB/s wr, 8 op/s 2023-03-09T22:41:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:41:37.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:41:37.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27808.760092 secs 2023-03-09T22:41:37.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:37 smithi181 ceph-mon[94189]: pgmap v15343: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 98 KiB/s wr, 8 op/s 2023-03-09T22:41:37.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:41:37.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:41:37.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27808.760092 secs 2023-03-09T22:41:37.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:37 smithi038 ceph-mon[92422]: pgmap v15343: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 98 KiB/s wr, 8 op/s 2023-03-09T22:41:38.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28270.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:41:39.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:39 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:41:39.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:39 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:41:39.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:39 smithi038 ceph-mon[92422]: pgmap v15344: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 98 KiB/s wr, 6 op/s 2023-03-09T22:41:39.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:39 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:41:39.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:39 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:41:39.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:39 smithi122 ceph-mon[93869]: pgmap v15344: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 98 KiB/s wr, 6 op/s 2023-03-09T22:41:39.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:39 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:41:39.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:39 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:41:39.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:39 smithi181 ceph-mon[94189]: pgmap v15344: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 98 KiB/s wr, 6 op/s 2023-03-09T22:41:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27811.383888 secs 2023-03-09T22:41:40.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27811.383888 secs 2023-03-09T22:41:40.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27811.383888 secs 2023-03-09T22:41:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:41 smithi038 ceph-mon[92422]: pgmap v15345: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 98 KiB/s wr, 9 op/s 2023-03-09T22:41:41.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:41 smithi122 ceph-mon[93869]: pgmap v15345: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 98 KiB/s wr, 9 op/s 2023-03-09T22:41:41.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:41 smithi181 ceph-mon[94189]: pgmap v15345: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 98 KiB/s wr, 9 op/s 2023-03-09T22:41:42.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27813.760315 secs 2023-03-09T22:41:42.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27813.760315 secs 2023-03-09T22:41:42.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27813.760315 secs 2023-03-09T22:41:43.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28275.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:41:43.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:43 smithi122 ceph-mon[93869]: pgmap v15346: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 182 KiB/s wr, 6 op/s 2023-03-09T22:41:43.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:43 smithi181 ceph-mon[94189]: pgmap v15346: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 182 KiB/s wr, 6 op/s 2023-03-09T22:41:44.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:43 smithi038 ceph-mon[92422]: pgmap v15346: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 182 KiB/s wr, 6 op/s 2023-03-09T22:41:45.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27816.383979 secs 2023-03-09T22:41:45.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27816.383979 secs 2023-03-09T22:41:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27816.383979 secs 2023-03-09T22:41:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:46 smithi122 ceph-mon[93869]: pgmap v15347: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 204 KiB/s wr, 7 op/s 2023-03-09T22:41:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:46 smithi181 ceph-mon[94189]: pgmap v15347: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 204 KiB/s wr, 7 op/s 2023-03-09T22:41:46.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:46 smithi038 ceph-mon[92422]: pgmap v15347: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 204 KiB/s wr, 7 op/s 2023-03-09T22:41:47.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27818.760395 secs 2023-03-09T22:41:47.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:47 smithi181 ceph-mon[94189]: pgmap v15348: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 386 KiB/s wr, 9 op/s 2023-03-09T22:41:47.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27818.760395 secs 2023-03-09T22:41:47.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:47 smithi038 ceph-mon[92422]: pgmap v15348: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 386 KiB/s wr, 9 op/s 2023-03-09T22:41:47.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27818.760395 secs 2023-03-09T22:41:47.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:47 smithi122 ceph-mon[93869]: pgmap v15348: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 386 KiB/s wr, 9 op/s 2023-03-09T22:41:48.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28280.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:41:49.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:41:49.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:41:49.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:49 smithi038 ceph-mon[92422]: pgmap v15349: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 322 KiB/s wr, 7 op/s 2023-03-09T22:41:49.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:41:49.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:41:49.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:49 smithi122 ceph-mon[93869]: pgmap v15349: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 322 KiB/s wr, 7 op/s 2023-03-09T22:41:49.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:41:49.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:41:49.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:49 smithi181 ceph-mon[94189]: pgmap v15349: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 322 KiB/s wr, 7 op/s 2023-03-09T22:41:50.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27821.384039 secs 2023-03-09T22:41:50.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27821.384039 secs 2023-03-09T22:41:51.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27821.384039 secs 2023-03-09T22:41:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:41:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:41:51.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:51 smithi122 ceph-mon[93869]: pgmap v15350: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 344 KiB/s wr, 10 op/s 2023-03-09T22:41:51.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:41:51.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:41:51.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:51 smithi181 ceph-mon[94189]: pgmap v15350: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 344 KiB/s wr, 10 op/s 2023-03-09T22:41:52.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:41:52.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:41:52.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:51 smithi038 ceph-mon[92422]: pgmap v15350: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 344 KiB/s wr, 10 op/s 2023-03-09T22:41:52.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27823.760493 secs 2023-03-09T22:41:52.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27823.760493 secs 2023-03-09T22:41:53.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27823.760493 secs 2023-03-09T22:41:53.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28285.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:41:53.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:53 smithi122 ceph-mon[93869]: pgmap v15351: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 576 KiB/s rd, 572 KiB/s wr, 10 op/s 2023-03-09T22:41:53.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:53 smithi181 ceph-mon[94189]: pgmap v15351: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 576 KiB/s rd, 572 KiB/s wr, 10 op/s 2023-03-09T22:41:54.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:53 smithi038 ceph-mon[92422]: pgmap v15351: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 576 KiB/s rd, 572 KiB/s wr, 10 op/s 2023-03-09T22:41:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:41:55.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:41:55.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27826.384102 secs 2023-03-09T22:41:55.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:55 smithi122 ceph-mon[93869]: pgmap v15352: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 611 KiB/s rd, 514 KiB/s wr, 10 op/s 2023-03-09T22:41:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:41:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:41:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27826.384102 secs 2023-03-09T22:41:55.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:55 smithi181 ceph-mon[94189]: pgmap v15352: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 611 KiB/s rd, 514 KiB/s wr, 10 op/s 2023-03-09T22:41:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:41:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:41:56.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27826.384102 secs 2023-03-09T22:41:56.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:55 smithi038 ceph-mon[92422]: pgmap v15352: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 611 KiB/s rd, 514 KiB/s wr, 10 op/s 2023-03-09T22:41:57.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:41:57.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:41:57.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:57 smithi122 ceph-mon[93869]: pgmap v15353: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 977 KiB/s rd, 878 KiB/s wr, 14 op/s 2023-03-09T22:41:57.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:41:57.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:41:57.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:57 smithi181 ceph-mon[94189]: pgmap v15353: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 977 KiB/s rd, 878 KiB/s wr, 14 op/s 2023-03-09T22:41:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:41:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:41:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:57 smithi038 ceph-mon[92422]: pgmap v15353: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 977 KiB/s rd, 878 KiB/s wr, 14 op/s 2023-03-09T22:41:58.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28290.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:41:59.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:41:59.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:41:59.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:41:59 smithi122 ceph-mon[93869]: pgmap v15354: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 794 KiB/s rd, 697 KiB/s wr, 12 op/s 2023-03-09T22:41:59.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:41:59.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:41:59.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:41:59 smithi181 ceph-mon[94189]: pgmap v15354: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 794 KiB/s rd, 697 KiB/s wr, 12 op/s 2023-03-09T22:42:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:42:00.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:42:00.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:41:59 smithi038 ceph-mon[92422]: pgmap v15354: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 794 KiB/s rd, 697 KiB/s wr, 12 op/s 2023-03-09T22:42:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27831.384162 secs 2023-03-09T22:42:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27831.384162 secs 2023-03-09T22:42:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27831.384162 secs 2023-03-09T22:42:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:42:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:42:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:01 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:42:01.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:01 smithi122 ceph-mon[93869]: pgmap v15355: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 883 KiB/s rd, 891 KiB/s wr, 17 op/s 2023-03-09T22:42:01.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:42:01.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:42:01.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:01 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:42:01.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:01 smithi181 ceph-mon[94189]: pgmap v15355: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 883 KiB/s rd, 891 KiB/s wr, 17 op/s 2023-03-09T22:42:02.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:42:02.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:42:02.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:01 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:42:02.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:01 smithi038 ceph-mon[92422]: pgmap v15355: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 883 KiB/s rd, 891 KiB/s wr, 17 op/s 2023-03-09T22:42:02.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:02 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:42:02.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:02 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:42:02.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:02 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:42:02.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27833.760726 secs 2023-03-09T22:42:02.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:02 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:42:02.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:02 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:42:02.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:02 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:42:02.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27833.760726 secs 2023-03-09T22:42:03.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:02 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:42:03.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:02 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:42:03.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:02 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:42:03.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27833.760726 secs 2023-03-09T22:42:03.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28295.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:42:03.980 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:03 smithi122 ceph-mon[93869]: pgmap v15356: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 890 KiB/s wr, 27 op/s 2023-03-09T22:42:03.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:03 smithi181 ceph-mon[94189]: pgmap v15356: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 890 KiB/s wr, 27 op/s 2023-03-09T22:42:04.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:03 smithi038 ceph-mon[92422]: pgmap v15356: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 890 KiB/s wr, 27 op/s 2023-03-09T22:42:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:05 smithi122 ceph-mon[93869]: osdmap e1175: 12 total, 12 up, 12 in 2023-03-09T22:42:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27836.384240 secs 2023-03-09T22:42:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:05 smithi122 ceph-mon[93869]: pgmap v15358: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 737 KiB/s rd, 779 KiB/s wr, 30 op/s 2023-03-09T22:42:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:05 smithi181 ceph-mon[94189]: osdmap e1175: 12 total, 12 up, 12 in 2023-03-09T22:42:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27836.384240 secs 2023-03-09T22:42:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:05 smithi181 ceph-mon[94189]: pgmap v15358: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 737 KiB/s rd, 779 KiB/s wr, 30 op/s 2023-03-09T22:42:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:05 smithi038 ceph-mon[92422]: osdmap e1175: 12 total, 12 up, 12 in 2023-03-09T22:42:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27836.384240 secs 2023-03-09T22:42:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:05 smithi038 ceph-mon[92422]: pgmap v15358: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 737 KiB/s rd, 779 KiB/s wr, 30 op/s 2023-03-09T22:42:06.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:06 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:42:06.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:42:06.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:06 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:42:06.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:42:07.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:06 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:42:07.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:42:07.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27838.760850 secs 2023-03-09T22:42:07.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:07 smithi122 ceph-mon[93869]: pgmap v15359: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 509 KiB/s wr, 30 op/s 2023-03-09T22:42:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27838.760850 secs 2023-03-09T22:42:07.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:07 smithi181 ceph-mon[94189]: pgmap v15359: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 509 KiB/s wr, 30 op/s 2023-03-09T22:42:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27838.760850 secs 2023-03-09T22:42:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:07 smithi038 ceph-mon[92422]: pgmap v15359: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 509 KiB/s wr, 30 op/s 2023-03-09T22:42:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28300.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:42:09.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:42:09.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:42:09.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:09 smithi122 ceph-mon[93869]: pgmap v15360: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 509 KiB/s wr, 30 op/s 2023-03-09T22:42:09.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:42:09.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:42:09.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:09 smithi181 ceph-mon[94189]: pgmap v15360: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 509 KiB/s wr, 30 op/s 2023-03-09T22:42:10.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:42:10.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:42:10.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:09 smithi038 ceph-mon[92422]: pgmap v15360: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 509 KiB/s wr, 30 op/s 2023-03-09T22:42:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27841.384302 secs 2023-03-09T22:42:10.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27841.384302 secs 2023-03-09T22:42:11.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27841.384302 secs 2023-03-09T22:42:11.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:11 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:42:11.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:11 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:42:11.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:11 smithi122 ceph-mon[93869]: pgmap v15361: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 375 KiB/s wr, 27 op/s 2023-03-09T22:42:11.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:11 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:42:11.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:11 smithi122 ceph-mon[93869]: osdmap e1176: 12 total, 12 up, 12 in 2023-03-09T22:42:11.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:11 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:42:11.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:11 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:42:11.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:11 smithi181 ceph-mon[94189]: pgmap v15361: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 375 KiB/s wr, 27 op/s 2023-03-09T22:42:11.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:11 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:42:11.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:11 smithi181 ceph-mon[94189]: osdmap e1176: 12 total, 12 up, 12 in 2023-03-09T22:42:12.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:11 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:42:12.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:11 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:42:12.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:11 smithi038 ceph-mon[92422]: pgmap v15361: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 375 KiB/s wr, 27 op/s 2023-03-09T22:42:12.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:11 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:42:12.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:11 smithi038 ceph-mon[92422]: osdmap e1176: 12 total, 12 up, 12 in 2023-03-09T22:42:12.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27843.760970 secs 2023-03-09T22:42:12.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27843.760970 secs 2023-03-09T22:42:13.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27843.760970 secs 2023-03-09T22:42:13.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28305.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:42:13.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:13 smithi122 ceph-mon[93869]: pgmap v15363: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 397 KiB/s wr, 13 op/s 2023-03-09T22:42:13.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:13 smithi181 ceph-mon[94189]: pgmap v15363: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 397 KiB/s wr, 13 op/s 2023-03-09T22:42:14.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:13 smithi038 ceph-mon[92422]: pgmap v15363: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 397 KiB/s wr, 13 op/s 2023-03-09T22:42:15.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27846.384360 secs 2023-03-09T22:42:15.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27846.384360 secs 2023-03-09T22:42:15.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27846.384360 secs 2023-03-09T22:42:16.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:16 smithi122 ceph-mon[93869]: pgmap v15364: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 336 KiB/s rd, 333 KiB/s wr, 10 op/s 2023-03-09T22:42:16.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:16 smithi181 ceph-mon[94189]: pgmap v15364: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 336 KiB/s rd, 333 KiB/s wr, 10 op/s 2023-03-09T22:42:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:16 smithi038 ceph-mon[92422]: pgmap v15364: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 336 KiB/s rd, 333 KiB/s wr, 10 op/s 2023-03-09T22:42:17.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:17 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:42:17.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27848.761048 secs 2023-03-09T22:42:17.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:17 smithi038 ceph-mon[92422]: pgmap v15365: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 139 KiB/s wr, 8 op/s 2023-03-09T22:42:17.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:17 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:42:17.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27848.761048 secs 2023-03-09T22:42:17.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:17 smithi122 ceph-mon[93869]: pgmap v15365: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 139 KiB/s wr, 8 op/s 2023-03-09T22:42:17.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:17 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:42:17.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27848.761048 secs 2023-03-09T22:42:17.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:17 smithi181 ceph-mon[94189]: pgmap v15365: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 139 KiB/s wr, 8 op/s 2023-03-09T22:42:18.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:18 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:42:18.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:18 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:42:18.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:18 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:42:18.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:18 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:42:18.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:18 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:42:18.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:18 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:42:18.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:18 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:42:18.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28310.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:42:18.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:18 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:42:18.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:18 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:42:18.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:18 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:42:18.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:18 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:42:18.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:18 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:42:18.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:18 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:42:18.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:18 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:42:18.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:18 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:42:18.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:18 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:42:18.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:18 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:42:18.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:18 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:42:18.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:18 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:42:18.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:18 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:42:18.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:18 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:42:19.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:19 smithi122 ceph-mon[93869]: pgmap v15366: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 139 KiB/s wr, 7 op/s 2023-03-09T22:42:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:19 smithi181 ceph-mon[94189]: pgmap v15366: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 139 KiB/s wr, 7 op/s 2023-03-09T22:42:20.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:19 smithi038 ceph-mon[92422]: pgmap v15366: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 139 KiB/s wr, 7 op/s 2023-03-09T22:42:20.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27851.384421 secs 2023-03-09T22:42:20.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27851.384421 secs 2023-03-09T22:42:21.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27851.384421 secs 2023-03-09T22:42:21.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:21 smithi122 ceph-mon[93869]: pgmap v15367: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 97 KiB/s wr, 6 op/s 2023-03-09T22:42:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:21 smithi181 ceph-mon[94189]: pgmap v15367: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 97 KiB/s wr, 6 op/s 2023-03-09T22:42:22.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:21 smithi038 ceph-mon[92422]: pgmap v15367: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 97 KiB/s wr, 6 op/s 2023-03-09T22:42:22.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27853.761193 secs 2023-03-09T22:42:22.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27853.761193 secs 2023-03-09T22:42:23.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27853.761193 secs 2023-03-09T22:42:23.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28315.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:42:23.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:23 smithi122 ceph-mon[93869]: pgmap v15368: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 82 KiB/s wr, 5 op/s 2023-03-09T22:42:23.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:23 smithi181 ceph-mon[94189]: pgmap v15368: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 82 KiB/s wr, 5 op/s 2023-03-09T22:42:24.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:23 smithi038 ceph-mon[92422]: pgmap v15368: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 82 KiB/s wr, 5 op/s 2023-03-09T22:42:25.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27856.384505 secs 2023-03-09T22:42:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27856.384505 secs 2023-03-09T22:42:25.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27856.384505 secs 2023-03-09T22:42:26.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:26 smithi122 ceph-mon[93869]: pgmap v15369: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 81 KiB/s wr, 5 op/s 2023-03-09T22:42:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:26 smithi181 ceph-mon[94189]: pgmap v15369: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 81 KiB/s wr, 5 op/s 2023-03-09T22:42:26.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:26 smithi038 ceph-mon[92422]: pgmap v15369: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 81 KiB/s wr, 5 op/s 2023-03-09T22:42:27.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27858.761309 secs 2023-03-09T22:42:27.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:27 smithi122 ceph-mon[93869]: pgmap v15370: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 81 KiB/s wr, 7 op/s 2023-03-09T22:42:27.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27858.761309 secs 2023-03-09T22:42:27.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:27 smithi181 ceph-mon[94189]: pgmap v15370: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 81 KiB/s wr, 7 op/s 2023-03-09T22:42:28.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27858.761309 secs 2023-03-09T22:42:28.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:27 smithi038 ceph-mon[92422]: pgmap v15370: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 81 KiB/s wr, 7 op/s 2023-03-09T22:42:28.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28320.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:42:30.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:30 smithi122 ceph-mon[93869]: pgmap v15371: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 81 KiB/s wr, 5 op/s 2023-03-09T22:42:30.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27861.384584 secs 2023-03-09T22:42:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:30 smithi181 ceph-mon[94189]: pgmap v15371: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 81 KiB/s wr, 5 op/s 2023-03-09T22:42:30.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27861.384584 secs 2023-03-09T22:42:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:30 smithi038 ceph-mon[92422]: pgmap v15371: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 81 KiB/s wr, 5 op/s 2023-03-09T22:42:30.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27861.384584 secs 2023-03-09T22:42:32.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:32 smithi122 ceph-mon[93869]: pgmap v15372: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 81 KiB/s wr, 7 op/s 2023-03-09T22:42:32.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27863.761376 secs 2023-03-09T22:42:32.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:32 smithi181 ceph-mon[94189]: pgmap v15372: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 81 KiB/s wr, 7 op/s 2023-03-09T22:42:32.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27863.761376 secs 2023-03-09T22:42:32.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:32 smithi038 ceph-mon[92422]: pgmap v15372: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 81 KiB/s wr, 7 op/s 2023-03-09T22:42:32.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27863.761376 secs 2023-03-09T22:42:33.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28325.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:42:34.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:34 smithi122 ceph-mon[93869]: pgmap v15373: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T22:42:34.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:34 smithi181 ceph-mon[94189]: pgmap v15373: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T22:42:34.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:34 smithi038 ceph-mon[92422]: pgmap v15373: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T22:42:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27866.384646 secs 2023-03-09T22:42:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:35 smithi122 ceph-mon[93869]: pgmap v15374: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T22:42:35.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27866.384646 secs 2023-03-09T22:42:35.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:35 smithi181 ceph-mon[94189]: pgmap v15374: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T22:42:35.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27866.384646 secs 2023-03-09T22:42:35.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:35 smithi038 ceph-mon[92422]: pgmap v15374: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T22:42:37.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27868.761497 secs 2023-03-09T22:42:37.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:37 smithi038 ceph-mon[92422]: pgmap v15375: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 8 op/s 2023-03-09T22:42:37.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27868.761497 secs 2023-03-09T22:42:37.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:37 smithi122 ceph-mon[93869]: pgmap v15375: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 8 op/s 2023-03-09T22:42:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27868.761497 secs 2023-03-09T22:42:37.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:37 smithi181 ceph-mon[94189]: pgmap v15375: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 8 op/s 2023-03-09T22:42:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28330.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:42:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:40 smithi122 ceph-mon[93869]: pgmap v15376: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 170 B/s wr, 5 op/s 2023-03-09T22:42:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27871.384708 secs 2023-03-09T22:42:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:40 smithi181 ceph-mon[94189]: pgmap v15376: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 170 B/s wr, 5 op/s 2023-03-09T22:42:40.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27871.384708 secs 2023-03-09T22:42:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:40 smithi038 ceph-mon[92422]: pgmap v15376: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 170 B/s wr, 5 op/s 2023-03-09T22:42:40.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27871.384708 secs 2023-03-09T22:42:42.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:42 smithi122 ceph-mon[93869]: pgmap v15377: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 101 KiB/s wr, 8 op/s 2023-03-09T22:42:42.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:42 smithi181 ceph-mon[94189]: pgmap v15377: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 101 KiB/s wr, 8 op/s 2023-03-09T22:42:42.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:42 smithi038 ceph-mon[92422]: pgmap v15377: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 101 KiB/s wr, 8 op/s 2023-03-09T22:42:43.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28335.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:42:44.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:44 smithi122 ceph-mon[93869]: pgmap v15378: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 101 KiB/s wr, 6 op/s 2023-03-09T22:42:44.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:44 smithi181 ceph-mon[94189]: pgmap v15378: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 101 KiB/s wr, 6 op/s 2023-03-09T22:42:44.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:44 smithi038 ceph-mon[92422]: pgmap v15378: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 101 KiB/s wr, 6 op/s 2023-03-09T22:42:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27876.384767 secs 2023-03-09T22:42:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:45 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:42:45.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27876.384767 secs 2023-03-09T22:42:45.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:45 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:42:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27876.384767 secs 2023-03-09T22:42:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:45 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:42:46.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:46 smithi122 ceph-mon[93869]: pgmap v15379: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 386 KiB/s rd, 284 KiB/s wr, 7 op/s 2023-03-09T22:42:46.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:46 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:42:46.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:46 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:42:46.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:46 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:42:46.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:46 smithi181 ceph-mon[94189]: pgmap v15379: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 386 KiB/s rd, 284 KiB/s wr, 7 op/s 2023-03-09T22:42:46.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:46 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:42:46.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:46 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:42:46.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:46 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:42:46.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:46 smithi038 ceph-mon[92422]: pgmap v15379: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 386 KiB/s rd, 284 KiB/s wr, 7 op/s 2023-03-09T22:42:46.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:46 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:42:46.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:46 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:42:46.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:46 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:42:47.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:47 smithi122 ceph-mon[93869]: pgmap v15380: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 610 KiB/s rd, 306 KiB/s wr, 10 op/s 2023-03-09T22:42:47.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:47 smithi181 ceph-mon[94189]: pgmap v15380: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 610 KiB/s rd, 306 KiB/s wr, 10 op/s 2023-03-09T22:42:48.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:47 smithi038 ceph-mon[92422]: pgmap v15380: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 610 KiB/s rd, 306 KiB/s wr, 10 op/s 2023-03-09T22:42:48.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28340.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:42:49.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:42:49.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:42:49.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:49 smithi122 ceph-mon[93869]: pgmap v15381: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 609 KiB/s rd, 306 KiB/s wr, 7 op/s 2023-03-09T22:42:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:42:49.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:42:49.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:49 smithi181 ceph-mon[94189]: pgmap v15381: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 609 KiB/s rd, 306 KiB/s wr, 7 op/s 2023-03-09T22:42:50.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:42:50.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:42:50.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:49 smithi038 ceph-mon[92422]: pgmap v15381: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 609 KiB/s rd, 306 KiB/s wr, 7 op/s 2023-03-09T22:42:50.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27881.384826 secs 2023-03-09T22:42:50.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27881.384826 secs 2023-03-09T22:42:51.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27881.384826 secs 2023-03-09T22:42:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:42:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:42:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:51 smithi122 ceph-mon[93869]: pgmap v15382: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 863 KiB/s rd, 557 KiB/s wr, 12 op/s 2023-03-09T22:42:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:42:51.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:42:51.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:51 smithi181 ceph-mon[94189]: pgmap v15382: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 863 KiB/s rd, 557 KiB/s wr, 12 op/s 2023-03-09T22:42:52.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:42:52.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:42:52.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:51 smithi038 ceph-mon[92422]: pgmap v15382: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 863 KiB/s rd, 557 KiB/s wr, 12 op/s 2023-03-09T22:42:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28345.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:42:54.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:54 smithi122 ceph-mon[93869]: pgmap v15383: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 661 KiB/s rd, 457 KiB/s wr, 10 op/s 2023-03-09T22:42:54.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:54 smithi181 ceph-mon[94189]: pgmap v15383: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 661 KiB/s rd, 457 KiB/s wr, 10 op/s 2023-03-09T22:42:54.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:54 smithi038 ceph-mon[92422]: pgmap v15383: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 661 KiB/s rd, 457 KiB/s wr, 10 op/s 2023-03-09T22:42:55.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27886.384890 secs 2023-03-09T22:42:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27886.384890 secs 2023-03-09T22:42:55.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27886.384890 secs 2023-03-09T22:42:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:56 smithi122 ceph-mon[93869]: pgmap v15384: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 672 KiB/s rd, 457 KiB/s wr, 10 op/s 2023-03-09T22:42:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:42:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:42:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:56 smithi181 ceph-mon[94189]: pgmap v15384: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 672 KiB/s rd, 457 KiB/s wr, 10 op/s 2023-03-09T22:42:56.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:42:56.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:42:56.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:56 smithi038 ceph-mon[92422]: pgmap v15384: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 672 KiB/s rd, 457 KiB/s wr, 10 op/s 2023-03-09T22:42:56.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:42:56.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:42:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:42:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:42:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:57 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 27888.762029 secs 2023-03-09T22:42:57.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:57 smithi122 ceph-mon[93869]: pgmap v15385: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 936 KiB/s rd, 688 KiB/s wr, 15 op/s 2023-03-09T22:42:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:42:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:42:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:57 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 27888.762029 secs 2023-03-09T22:42:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:57 smithi181 ceph-mon[94189]: pgmap v15385: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 936 KiB/s rd, 688 KiB/s wr, 15 op/s 2023-03-09T22:42:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:42:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:42:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:57 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 27888.762029 secs 2023-03-09T22:42:57.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:57 smithi038 ceph-mon[92422]: pgmap v15385: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 936 KiB/s rd, 688 KiB/s wr, 15 op/s 2023-03-09T22:42:58.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28350.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:42:59.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:42:59.980 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:42:59.981 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:42:59 smithi122 ceph-mon[93869]: pgmap v15386: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 712 KiB/s rd, 666 KiB/s wr, 12 op/s 2023-03-09T22:42:59.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:42:59.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:42:59.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:42:59 smithi181 ceph-mon[94189]: pgmap v15386: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 712 KiB/s rd, 666 KiB/s wr, 12 op/s 2023-03-09T22:43:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:43:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:43:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:42:59 smithi038 ceph-mon[92422]: pgmap v15386: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 712 KiB/s rd, 666 KiB/s wr, 12 op/s 2023-03-09T22:43:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27891.384947 secs 2023-03-09T22:43:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27891.384947 secs 2023-03-09T22:43:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27891.384947 secs 2023-03-09T22:43:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:43:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:43:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:01 smithi122 ceph-mon[93869]: pgmap v15387: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 736 KiB/s rd, 711 KiB/s wr, 18 op/s 2023-03-09T22:43:01.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:43:01.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:43:01.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:01 smithi181 ceph-mon[94189]: pgmap v15387: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 736 KiB/s rd, 711 KiB/s wr, 18 op/s 2023-03-09T22:43:02.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:43:02.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:43:02.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:01 smithi038 ceph-mon[92422]: pgmap v15387: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 736 KiB/s rd, 711 KiB/s wr, 18 op/s 2023-03-09T22:43:02.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:02 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:43:02.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:02 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:43:02.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:02 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:43:02.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27893.762123 secs 2023-03-09T22:43:02.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:02 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:43:02.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:02 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:43:02.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:02 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:43:02.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27893.762123 secs 2023-03-09T22:43:03.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:02 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:43:03.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:02 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:43:03.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:02 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:43:03.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27893.762123 secs 2023-03-09T22:43:03.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28355.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:43:03.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:03 smithi122 ceph-mon[93869]: pgmap v15388: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 638 KiB/s rd, 630 KiB/s wr, 27 op/s 2023-03-09T22:43:03.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:03 smithi181 ceph-mon[94189]: pgmap v15388: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 638 KiB/s rd, 630 KiB/s wr, 27 op/s 2023-03-09T22:43:04.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:03 smithi038 ceph-mon[92422]: pgmap v15388: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 638 KiB/s rd, 630 KiB/s wr, 27 op/s 2023-03-09T22:43:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:05 smithi122 ceph-mon[93869]: osdmap e1177: 12 total, 12 up, 12 in 2023-03-09T22:43:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27896.385005 secs 2023-03-09T22:43:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:05 smithi122 ceph-mon[93869]: pgmap v15390: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 847 KiB/s rd, 851 KiB/s wr, 31 op/s 2023-03-09T22:43:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:05 smithi181 ceph-mon[94189]: osdmap e1177: 12 total, 12 up, 12 in 2023-03-09T22:43:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27896.385005 secs 2023-03-09T22:43:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:05 smithi181 ceph-mon[94189]: pgmap v15390: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 847 KiB/s rd, 851 KiB/s wr, 31 op/s 2023-03-09T22:43:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:05 smithi038 ceph-mon[92422]: osdmap e1177: 12 total, 12 up, 12 in 2023-03-09T22:43:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27896.385005 secs 2023-03-09T22:43:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:05 smithi038 ceph-mon[92422]: pgmap v15390: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 847 KiB/s rd, 851 KiB/s wr, 31 op/s 2023-03-09T22:43:06.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:06 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:43:06.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:43:06.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:06 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:43:06.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:43:07.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:06 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:43:07.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:43:07.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:43:07.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:43:07.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27898.762248 secs 2023-03-09T22:43:07.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:07 smithi122 ceph-mon[93869]: pgmap v15391: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 470 KiB/s wr, 28 op/s 2023-03-09T22:43:07.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:43:07.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:43:07.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27898.762248 secs 2023-03-09T22:43:07.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:07 smithi181 ceph-mon[94189]: pgmap v15391: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 470 KiB/s wr, 28 op/s 2023-03-09T22:43:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:43:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:43:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27898.762248 secs 2023-03-09T22:43:08.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:07 smithi038 ceph-mon[92422]: pgmap v15391: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 470 KiB/s wr, 28 op/s 2023-03-09T22:43:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28360.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:43:09.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:43:09.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:43:09.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:09 smithi122 ceph-mon[93869]: pgmap v15392: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 470 KiB/s wr, 29 op/s 2023-03-09T22:43:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:43:09.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:43:09.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:09 smithi181 ceph-mon[94189]: pgmap v15392: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 470 KiB/s wr, 29 op/s 2023-03-09T22:43:10.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:43:10.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:43:10.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:09 smithi038 ceph-mon[92422]: pgmap v15392: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 470 KiB/s wr, 29 op/s 2023-03-09T22:43:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27901.385066 secs 2023-03-09T22:43:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27901.385066 secs 2023-03-09T22:43:11.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27901.385066 secs 2023-03-09T22:43:11.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:11 smithi122 ceph-mon[93869]: pgmap v15393: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 402 KiB/s rd, 417 KiB/s wr, 24 op/s 2023-03-09T22:43:11.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:11 smithi122 ceph-mon[93869]: osdmap e1178: 12 total, 12 up, 12 in 2023-03-09T22:43:11.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:11 smithi181 ceph-mon[94189]: pgmap v15393: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 402 KiB/s rd, 417 KiB/s wr, 24 op/s 2023-03-09T22:43:11.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:11 smithi181 ceph-mon[94189]: osdmap e1178: 12 total, 12 up, 12 in 2023-03-09T22:43:12.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:11 smithi038 ceph-mon[92422]: pgmap v15393: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 402 KiB/s rd, 417 KiB/s wr, 24 op/s 2023-03-09T22:43:12.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:11 smithi038 ceph-mon[92422]: osdmap e1178: 12 total, 12 up, 12 in 2023-03-09T22:43:12.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27903.762397 secs 2023-03-09T22:43:12.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27903.762397 secs 2023-03-09T22:43:13.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27903.762397 secs 2023-03-09T22:43:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28365.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:43:13.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:13 smithi122 ceph-mon[93869]: pgmap v15395: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 132 KiB/s wr, 9 op/s 2023-03-09T22:43:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:13 smithi181 ceph-mon[94189]: pgmap v15395: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 132 KiB/s wr, 9 op/s 2023-03-09T22:43:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:13 smithi038 ceph-mon[92422]: pgmap v15395: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 132 KiB/s wr, 9 op/s 2023-03-09T22:43:15.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27906.385119 secs 2023-03-09T22:43:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27906.385119 secs 2023-03-09T22:43:15.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27906.385119 secs 2023-03-09T22:43:16.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:16 smithi122 ceph-mon[93869]: pgmap v15396: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 255 KiB/s wr, 9 op/s 2023-03-09T22:43:16.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:16 smithi181 ceph-mon[94189]: pgmap v15396: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 255 KiB/s wr, 9 op/s 2023-03-09T22:43:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:16 smithi038 ceph-mon[92422]: pgmap v15396: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 255 KiB/s wr, 9 op/s 2023-03-09T22:43:17.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27908.762492 secs 2023-03-09T22:43:17.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:17 smithi122 ceph-mon[93869]: pgmap v15397: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 139 KiB/s wr, 8 op/s 2023-03-09T22:43:17.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27908.762492 secs 2023-03-09T22:43:17.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:17 smithi181 ceph-mon[94189]: pgmap v15397: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 139 KiB/s wr, 8 op/s 2023-03-09T22:43:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27908.762492 secs 2023-03-09T22:43:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:17 smithi038 ceph-mon[92422]: pgmap v15397: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 139 KiB/s wr, 8 op/s 2023-03-09T22:43:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28370.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:43:18.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:18 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:43:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:18 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:43:19.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:18 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:43:19.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:43:19.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:43:19.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:43:19.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:19 smithi122 ceph-mon[93869]: pgmap v15398: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 139 KiB/s wr, 7 op/s 2023-03-09T22:43:19.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:43:19.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:43:19.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:43:19.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:19 smithi181 ceph-mon[94189]: pgmap v15398: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 139 KiB/s wr, 7 op/s 2023-03-09T22:43:20.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:43:20.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:43:20.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:43:20.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:19 smithi038 ceph-mon[92422]: pgmap v15398: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 139 KiB/s wr, 7 op/s 2023-03-09T22:43:20.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27911.385202 secs 2023-03-09T22:43:20.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27911.385202 secs 2023-03-09T22:43:21.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27911.385202 secs 2023-03-09T22:43:21.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:21 smithi122 ceph-mon[93869]: pgmap v15399: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 138 KiB/s wr, 7 op/s 2023-03-09T22:43:21.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:21 smithi181 ceph-mon[94189]: pgmap v15399: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 138 KiB/s wr, 7 op/s 2023-03-09T22:43:22.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:21 smithi038 ceph-mon[92422]: pgmap v15399: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 138 KiB/s wr, 7 op/s 2023-03-09T22:43:22.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27913.762611 secs 2023-03-09T22:43:22.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27913.762611 secs 2023-03-09T22:43:23.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27913.762611 secs 2023-03-09T22:43:23.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28375.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:43:23.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:23 smithi122 ceph-mon[93869]: pgmap v15400: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 116 KiB/s wr, 6 op/s 2023-03-09T22:43:23.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:23 smithi181 ceph-mon[94189]: pgmap v15400: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 116 KiB/s wr, 6 op/s 2023-03-09T22:43:24.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:23 smithi038 ceph-mon[92422]: pgmap v15400: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 116 KiB/s wr, 6 op/s 2023-03-09T22:43:25.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27916.385262 secs 2023-03-09T22:43:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27916.385262 secs 2023-03-09T22:43:25.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27916.385262 secs 2023-03-09T22:43:26.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:26 smithi122 ceph-mon[93869]: pgmap v15401: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 115 KiB/s wr, 6 op/s 2023-03-09T22:43:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:26 smithi181 ceph-mon[94189]: pgmap v15401: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 115 KiB/s wr, 6 op/s 2023-03-09T22:43:26.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:26 smithi038 ceph-mon[92422]: pgmap v15401: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 115 KiB/s wr, 6 op/s 2023-03-09T22:43:27.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27918.762749 secs 2023-03-09T22:43:27.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:27 smithi122 ceph-mon[93869]: pgmap v15402: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T22:43:27.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27918.762749 secs 2023-03-09T22:43:27.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:27 smithi181 ceph-mon[94189]: pgmap v15402: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T22:43:28.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27918.762749 secs 2023-03-09T22:43:28.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:27 smithi038 ceph-mon[92422]: pgmap v15402: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T22:43:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28380.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:43:30.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:30 smithi122 ceph-mon[93869]: pgmap v15403: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T22:43:30.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27921.385341 secs 2023-03-09T22:43:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:30 smithi181 ceph-mon[94189]: pgmap v15403: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T22:43:30.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27921.385341 secs 2023-03-09T22:43:30.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:30 smithi038 ceph-mon[92422]: pgmap v15403: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T22:43:30.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27921.385341 secs 2023-03-09T22:43:31.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:31 smithi122 ceph-mon[93869]: pgmap v15404: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T22:43:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:31 smithi181 ceph-mon[94189]: pgmap v15404: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T22:43:31.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:31 smithi038 ceph-mon[92422]: pgmap v15404: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T22:43:32.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27923.762884 secs 2023-03-09T22:43:32.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27923.762884 secs 2023-03-09T22:43:33.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27923.762884 secs 2023-03-09T22:43:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28385.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:43:33.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:33 smithi122 ceph-mon[93869]: pgmap v15405: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T22:43:33.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:33 smithi181 ceph-mon[94189]: pgmap v15405: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T22:43:34.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:33 smithi038 ceph-mon[92422]: pgmap v15405: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T22:43:35.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27926.385419 secs 2023-03-09T22:43:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27926.385419 secs 2023-03-09T22:43:35.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27926.385419 secs 2023-03-09T22:43:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:36 smithi122 ceph-mon[93869]: pgmap v15406: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T22:43:36.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:36 smithi181 ceph-mon[94189]: pgmap v15406: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T22:43:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:36 smithi038 ceph-mon[92422]: pgmap v15406: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T22:43:37.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27928.763029 secs 2023-03-09T22:43:37.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:37 smithi122 ceph-mon[93869]: pgmap v15407: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 8 op/s 2023-03-09T22:43:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27928.763029 secs 2023-03-09T22:43:37.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:37 smithi181 ceph-mon[94189]: pgmap v15407: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 8 op/s 2023-03-09T22:43:38.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27928.763029 secs 2023-03-09T22:43:38.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:37 smithi038 ceph-mon[92422]: pgmap v15407: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 8 op/s 2023-03-09T22:43:38.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28390.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:43:39.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:39 smithi122 ceph-mon[93869]: pgmap v15408: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 98 KiB/s wr, 6 op/s 2023-03-09T22:43:39.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:39 smithi181 ceph-mon[94189]: pgmap v15408: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 98 KiB/s wr, 6 op/s 2023-03-09T22:43:40.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:39 smithi038 ceph-mon[92422]: pgmap v15408: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 98 KiB/s wr, 6 op/s 2023-03-09T22:43:40.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27931.385480 secs 2023-03-09T22:43:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27931.385480 secs 2023-03-09T22:43:41.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27931.385480 secs 2023-03-09T22:43:41.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:41 smithi122 ceph-mon[93869]: pgmap v15409: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 121 KiB/s wr, 9 op/s 2023-03-09T22:43:41.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:41 smithi181 ceph-mon[94189]: pgmap v15409: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 121 KiB/s wr, 9 op/s 2023-03-09T22:43:42.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:41 smithi038 ceph-mon[92422]: pgmap v15409: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 121 KiB/s wr, 9 op/s 2023-03-09T22:43:42.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27933.763149 secs 2023-03-09T22:43:42.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27933.763149 secs 2023-03-09T22:43:43.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27933.763149 secs 2023-03-09T22:43:43.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28395.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:43:43.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:43 smithi122 ceph-mon[93869]: pgmap v15410: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 389 KiB/s wr, 8 op/s 2023-03-09T22:43:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:43 smithi181 ceph-mon[94189]: pgmap v15410: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 389 KiB/s wr, 8 op/s 2023-03-09T22:43:44.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:43 smithi038 ceph-mon[92422]: pgmap v15410: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 389 KiB/s wr, 8 op/s 2023-03-09T22:43:45.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27936.385538 secs 2023-03-09T22:43:45.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27936.385538 secs 2023-03-09T22:43:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27936.385538 secs 2023-03-09T22:43:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:46 smithi122 ceph-mon[93869]: pgmap v15411: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 438 KiB/s wr, 8 op/s 2023-03-09T22:43:46.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:46 smithi181 ceph-mon[94189]: pgmap v15411: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 438 KiB/s wr, 8 op/s 2023-03-09T22:43:46.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:46 smithi038 ceph-mon[92422]: pgmap v15411: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 438 KiB/s wr, 8 op/s 2023-03-09T22:43:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:43:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:43:48.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27938.763263 secs 2023-03-09T22:43:48.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:48 smithi122 ceph-mon[93869]: pgmap v15412: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 461 KiB/s wr, 11 op/s 2023-03-09T22:43:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:43:48.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:43:48.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27938.763263 secs 2023-03-09T22:43:48.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:48 smithi181 ceph-mon[94189]: pgmap v15412: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 461 KiB/s wr, 11 op/s 2023-03-09T22:43:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:43:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:43:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27938.763263 secs 2023-03-09T22:43:48.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:48 smithi038 ceph-mon[92422]: pgmap v15412: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 461 KiB/s wr, 11 op/s 2023-03-09T22:43:48.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28400.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:43:49.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:43:49.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:43:49.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:43:49.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:43:49.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:43:49.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:43:50.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:50 smithi122 ceph-mon[93869]: pgmap v15413: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 670 KiB/s rd, 535 KiB/s wr, 10 op/s 2023-03-09T22:43:50.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27941.385598 secs 2023-03-09T22:43:50.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:50 smithi181 ceph-mon[94189]: pgmap v15413: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 670 KiB/s rd, 535 KiB/s wr, 10 op/s 2023-03-09T22:43:50.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27941.385598 secs 2023-03-09T22:43:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:50 smithi038 ceph-mon[92422]: pgmap v15413: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 670 KiB/s rd, 535 KiB/s wr, 10 op/s 2023-03-09T22:43:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27941.385598 secs 2023-03-09T22:43:52.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:43:52.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:43:52.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:52 smithi122 ceph-mon[93869]: pgmap v15414: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 920 KiB/s rd, 784 KiB/s wr, 13 op/s 2023-03-09T22:43:52.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:43:52.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:43:52.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:52 smithi181 ceph-mon[94189]: pgmap v15414: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 920 KiB/s rd, 784 KiB/s wr, 13 op/s 2023-03-09T22:43:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:43:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:43:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:52 smithi038 ceph-mon[92422]: pgmap v15414: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 920 KiB/s rd, 784 KiB/s wr, 13 op/s 2023-03-09T22:43:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27943.763382 secs 2023-03-09T22:43:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:43:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:43:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27943.763382 secs 2023-03-09T22:43:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:43:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:43:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27943.763382 secs 2023-03-09T22:43:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:43:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:43:53.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28405.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:43:54.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:54 smithi122 ceph-mon[93869]: pgmap v15415: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 896 KiB/s rd, 762 KiB/s wr, 11 op/s 2023-03-09T22:43:54.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:54 smithi181 ceph-mon[94189]: pgmap v15415: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 896 KiB/s rd, 762 KiB/s wr, 11 op/s 2023-03-09T22:43:54.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:54 smithi038 ceph-mon[92422]: pgmap v15415: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 896 KiB/s rd, 762 KiB/s wr, 11 op/s 2023-03-09T22:43:55.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27946.385658 secs 2023-03-09T22:43:55.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27946.385658 secs 2023-03-09T22:43:55.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27946.385658 secs 2023-03-09T22:43:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:56 smithi122 ceph-mon[93869]: pgmap v15416: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 494 KiB/s wr, 9 op/s 2023-03-09T22:43:56.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:56 smithi181 ceph-mon[94189]: pgmap v15416: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 494 KiB/s wr, 9 op/s 2023-03-09T22:43:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:56 smithi038 ceph-mon[92422]: pgmap v15416: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 494 KiB/s wr, 9 op/s 2023-03-09T22:43:57.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27948.763496 secs 2023-03-09T22:43:57.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:57 smithi122 ceph-mon[93869]: pgmap v15417: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 642 KiB/s rd, 518 KiB/s wr, 13 op/s 2023-03-09T22:43:57.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27948.763496 secs 2023-03-09T22:43:57.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:57 smithi181 ceph-mon[94189]: pgmap v15417: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 642 KiB/s rd, 518 KiB/s wr, 13 op/s 2023-03-09T22:43:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27948.763496 secs 2023-03-09T22:43:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:57 smithi038 ceph-mon[92422]: pgmap v15417: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 642 KiB/s rd, 518 KiB/s wr, 13 op/s 2023-03-09T22:43:58.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28410.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:43:59.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:43:59.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:43:59.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:43:59 smithi122 ceph-mon[93869]: pgmap v15418: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 617 KiB/s rd, 496 KiB/s wr, 10 op/s 2023-03-09T22:43:59.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:43:59.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:43:59.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:43:59 smithi181 ceph-mon[94189]: pgmap v15418: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 617 KiB/s rd, 496 KiB/s wr, 10 op/s 2023-03-09T22:44:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:44:00.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:44:00.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:43:59 smithi038 ceph-mon[92422]: pgmap v15418: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 617 KiB/s rd, 496 KiB/s wr, 10 op/s 2023-03-09T22:44:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27951.385738 secs 2023-03-09T22:44:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27951.385738 secs 2023-03-09T22:44:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27951.385738 secs 2023-03-09T22:44:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:44:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:44:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:01 smithi122 ceph-mon[93869]: pgmap v15419: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 754 KiB/s rd, 774 KiB/s wr, 14 op/s 2023-03-09T22:44:01.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:44:01.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:44:01.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:01 smithi181 ceph-mon[94189]: pgmap v15419: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 754 KiB/s rd, 774 KiB/s wr, 14 op/s 2023-03-09T22:44:02.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:44:02.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:44:02.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:01 smithi038 ceph-mon[92422]: pgmap v15419: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 754 KiB/s rd, 774 KiB/s wr, 14 op/s 2023-03-09T22:44:02.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27953.763629 secs 2023-03-09T22:44:02.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27953.763629 secs 2023-03-09T22:44:03.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27953.763629 secs 2023-03-09T22:44:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28415.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:44:03.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:03 smithi122 ceph-mon[93869]: pgmap v15420: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 448 KiB/s wr, 23 op/s 2023-03-09T22:44:03.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:03 smithi181 ceph-mon[94189]: pgmap v15420: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 448 KiB/s wr, 23 op/s 2023-03-09T22:44:04.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:03 smithi038 ceph-mon[92422]: pgmap v15420: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 448 KiB/s wr, 23 op/s 2023-03-09T22:44:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:05 smithi122 ceph-mon[93869]: osdmap e1179: 12 total, 12 up, 12 in 2023-03-09T22:44:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27956.385799 secs 2023-03-09T22:44:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:05 smithi122 ceph-mon[93869]: pgmap v15422: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 726 KiB/s rd, 767 KiB/s wr, 29 op/s 2023-03-09T22:44:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:05 smithi181 ceph-mon[94189]: osdmap e1179: 12 total, 12 up, 12 in 2023-03-09T22:44:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27956.385799 secs 2023-03-09T22:44:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:05 smithi181 ceph-mon[94189]: pgmap v15422: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 726 KiB/s rd, 767 KiB/s wr, 29 op/s 2023-03-09T22:44:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:05 smithi038 ceph-mon[92422]: osdmap e1179: 12 total, 12 up, 12 in 2023-03-09T22:44:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27956.385799 secs 2023-03-09T22:44:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:05 smithi038 ceph-mon[92422]: pgmap v15422: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 726 KiB/s rd, 767 KiB/s wr, 29 op/s 2023-03-09T22:44:07.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:44:07.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:44:07.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27958.763766 secs 2023-03-09T22:44:07.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:07 smithi122 ceph-mon[93869]: pgmap v15423: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 639 KiB/s rd, 680 KiB/s wr, 28 op/s 2023-03-09T22:44:07.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:44:07.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:44:07.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27958.763766 secs 2023-03-09T22:44:07.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:07 smithi181 ceph-mon[94189]: pgmap v15423: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 639 KiB/s rd, 680 KiB/s wr, 28 op/s 2023-03-09T22:44:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:44:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:44:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27958.763766 secs 2023-03-09T22:44:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:07 smithi038 ceph-mon[92422]: pgmap v15423: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 639 KiB/s rd, 680 KiB/s wr, 28 op/s 2023-03-09T22:44:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28420.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:44:09.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:44:09.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:44:09.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:09 smithi122 ceph-mon[93869]: pgmap v15424: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 736 KiB/s rd, 777 KiB/s wr, 29 op/s 2023-03-09T22:44:09.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:44:09.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:44:09.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:09 smithi181 ceph-mon[94189]: pgmap v15424: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 736 KiB/s rd, 777 KiB/s wr, 29 op/s 2023-03-09T22:44:10.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:44:10.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:44:10.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:09 smithi038 ceph-mon[92422]: pgmap v15424: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 736 KiB/s rd, 777 KiB/s wr, 29 op/s 2023-03-09T22:44:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27961.385859 secs 2023-03-09T22:44:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27961.385859 secs 2023-03-09T22:44:11.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27961.385859 secs 2023-03-09T22:44:11.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:11 smithi122 ceph-mon[93869]: pgmap v15425: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 354 KiB/s wr, 24 op/s 2023-03-09T22:44:11.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:11 smithi122 ceph-mon[93869]: osdmap e1180: 12 total, 12 up, 12 in 2023-03-09T22:44:11.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:11 smithi181 ceph-mon[94189]: pgmap v15425: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 354 KiB/s wr, 24 op/s 2023-03-09T22:44:11.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:11 smithi181 ceph-mon[94189]: osdmap e1180: 12 total, 12 up, 12 in 2023-03-09T22:44:12.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:11 smithi038 ceph-mon[92422]: pgmap v15425: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 354 KiB/s wr, 24 op/s 2023-03-09T22:44:12.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:11 smithi038 ceph-mon[92422]: osdmap e1180: 12 total, 12 up, 12 in 2023-03-09T22:44:12.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27963.763903 secs 2023-03-09T22:44:12.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27963.763903 secs 2023-03-09T22:44:13.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27963.763903 secs 2023-03-09T22:44:13.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28425.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:44:13.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:13 smithi122 ceph-mon[93869]: pgmap v15427: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 112 KiB/s wr, 9 op/s 2023-03-09T22:44:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:13 smithi181 ceph-mon[94189]: pgmap v15427: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 112 KiB/s wr, 9 op/s 2023-03-09T22:44:14.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:13 smithi038 ceph-mon[92422]: pgmap v15427: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 112 KiB/s wr, 9 op/s 2023-03-09T22:44:15.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27966.385937 secs 2023-03-09T22:44:15.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27966.385937 secs 2023-03-09T22:44:15.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27966.385937 secs 2023-03-09T22:44:16.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:16 smithi122 ceph-mon[93869]: pgmap v15428: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 99 KiB/s wr, 8 op/s 2023-03-09T22:44:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:16 smithi181 ceph-mon[94189]: pgmap v15428: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 99 KiB/s wr, 8 op/s 2023-03-09T22:44:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:16 smithi038 ceph-mon[92422]: pgmap v15428: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 99 KiB/s wr, 8 op/s 2023-03-09T22:44:17.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27968.763990 secs 2023-03-09T22:44:17.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:17 smithi122 ceph-mon[93869]: pgmap v15429: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 98 KiB/s wr, 7 op/s 2023-03-09T22:44:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27968.763990 secs 2023-03-09T22:44:17.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:17 smithi181 ceph-mon[94189]: pgmap v15429: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 98 KiB/s wr, 7 op/s 2023-03-09T22:44:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27968.763990 secs 2023-03-09T22:44:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:17 smithi038 ceph-mon[92422]: pgmap v15429: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 98 KiB/s wr, 7 op/s 2023-03-09T22:44:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28430.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:44:19.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:44:19.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:44:19.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:44:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:20 smithi122 ceph-mon[93869]: pgmap v15430: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T22:44:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:44:20.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:44:20.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:44:20.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27971.385989 secs 2023-03-09T22:44:20.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:20 smithi181 ceph-mon[94189]: pgmap v15430: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T22:44:20.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:44:20.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:44:20.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:44:20.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27971.385989 secs 2023-03-09T22:44:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:20 smithi038 ceph-mon[92422]: pgmap v15430: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T22:44:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:44:20.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:44:20.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:44:20.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27971.385989 secs 2023-03-09T22:44:21.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:21 smithi122 ceph-mon[93869]: pgmap v15431: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2023-03-09T22:44:21.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:21 smithi181 ceph-mon[94189]: pgmap v15431: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2023-03-09T22:44:22.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:21 smithi038 ceph-mon[92422]: pgmap v15431: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2023-03-09T22:44:22.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27973.764143 secs 2023-03-09T22:44:22.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27973.764143 secs 2023-03-09T22:44:23.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27973.764143 secs 2023-03-09T22:44:23.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28435.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:44:23.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:23 smithi122 ceph-mon[93869]: pgmap v15432: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 1.0 KiB/s wr, 5 op/s 2023-03-09T22:44:23.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:23 smithi181 ceph-mon[94189]: pgmap v15432: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 1.0 KiB/s wr, 5 op/s 2023-03-09T22:44:24.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:23 smithi038 ceph-mon[92422]: pgmap v15432: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 1.0 KiB/s wr, 5 op/s 2023-03-09T22:44:25.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27976.386059 secs 2023-03-09T22:44:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27976.386059 secs 2023-03-09T22:44:25.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27976.386059 secs 2023-03-09T22:44:26.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:26 smithi122 ceph-mon[93869]: pgmap v15433: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2023-03-09T22:44:26.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:26 smithi181 ceph-mon[94189]: pgmap v15433: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2023-03-09T22:44:26.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:26 smithi038 ceph-mon[92422]: pgmap v15433: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 1.1 KiB/s wr, 6 op/s 2023-03-09T22:44:27.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27978.764245 secs 2023-03-09T22:44:27.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:27 smithi122 ceph-mon[93869]: pgmap v15434: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 45 KiB/s wr, 8 op/s 2023-03-09T22:44:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27978.764245 secs 2023-03-09T22:44:27.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:27 smithi181 ceph-mon[94189]: pgmap v15434: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 45 KiB/s wr, 8 op/s 2023-03-09T22:44:28.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27978.764245 secs 2023-03-09T22:44:28.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:27 smithi038 ceph-mon[92422]: pgmap v15434: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 45 KiB/s wr, 8 op/s 2023-03-09T22:44:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28440.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:44:29.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:29 smithi122 ceph-mon[93869]: pgmap v15435: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 114 KiB/s wr, 6 op/s 2023-03-09T22:44:29.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:29 smithi181 ceph-mon[94189]: pgmap v15435: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 114 KiB/s wr, 6 op/s 2023-03-09T22:44:30.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:29 smithi038 ceph-mon[92422]: pgmap v15435: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 114 KiB/s wr, 6 op/s 2023-03-09T22:44:30.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27981.386119 secs 2023-03-09T22:44:30.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27981.386119 secs 2023-03-09T22:44:31.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27981.386119 secs 2023-03-09T22:44:31.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:31 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:44:31.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:31 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:44:31.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:31 smithi122 ceph-mon[93869]: pgmap v15436: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 233 KiB/s wr, 9 op/s 2023-03-09T22:44:31.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:31 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:44:31.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:31 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:44:31.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:31 smithi181 ceph-mon[94189]: pgmap v15436: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 233 KiB/s wr, 9 op/s 2023-03-09T22:44:32.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:31 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:44:32.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:31 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:44:32.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:31 smithi038 ceph-mon[92422]: pgmap v15436: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 233 KiB/s wr, 9 op/s 2023-03-09T22:44:32.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27983.764400 secs 2023-03-09T22:44:32.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27983.764400 secs 2023-03-09T22:44:33.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27983.764400 secs 2023-03-09T22:44:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28445.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:44:33.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:33 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:44:33.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:33 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:44:33.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:33 smithi122 ceph-mon[93869]: pgmap v15437: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 232 KiB/s wr, 7 op/s 2023-03-09T22:44:33.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:33 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:44:33.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:33 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:44:33.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:33 smithi181 ceph-mon[94189]: pgmap v15437: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 232 KiB/s wr, 7 op/s 2023-03-09T22:44:34.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:33 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:44:34.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:33 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:44:34.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:33 smithi038 ceph-mon[92422]: pgmap v15437: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 232 KiB/s wr, 7 op/s 2023-03-09T22:44:35.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27986.386180 secs 2023-03-09T22:44:35.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27986.386180 secs 2023-03-09T22:44:35.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27986.386180 secs 2023-03-09T22:44:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:36 smithi122 ceph-mon[93869]: pgmap v15438: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 232 KiB/s wr, 7 op/s 2023-03-09T22:44:36.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:36 smithi181 ceph-mon[94189]: pgmap v15438: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 232 KiB/s wr, 7 op/s 2023-03-09T22:44:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:36 smithi038 ceph-mon[92422]: pgmap v15438: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 232 KiB/s wr, 7 op/s 2023-03-09T22:44:37.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27988.764516 secs 2023-03-09T22:44:37.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:37 smithi122 ceph-mon[93869]: pgmap v15439: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 232 KiB/s wr, 10 op/s 2023-03-09T22:44:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27988.764516 secs 2023-03-09T22:44:37.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:37 smithi181 ceph-mon[94189]: pgmap v15439: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 232 KiB/s wr, 10 op/s 2023-03-09T22:44:38.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27988.764516 secs 2023-03-09T22:44:38.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:37 smithi038 ceph-mon[92422]: pgmap v15439: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 232 KiB/s wr, 10 op/s 2023-03-09T22:44:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28450.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:44:39.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:39 smithi122 ceph-mon[93869]: pgmap v15440: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 188 KiB/s wr, 7 op/s 2023-03-09T22:44:39.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:39 smithi181 ceph-mon[94189]: pgmap v15440: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 188 KiB/s wr, 7 op/s 2023-03-09T22:44:40.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:39 smithi038 ceph-mon[92422]: pgmap v15440: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 188 KiB/s wr, 7 op/s 2023-03-09T22:44:40.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27991.386243 secs 2023-03-09T22:44:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27991.386243 secs 2023-03-09T22:44:41.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27991.386243 secs 2023-03-09T22:44:41.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:41 smithi122 ceph-mon[93869]: pgmap v15441: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 298 KiB/s wr, 9 op/s 2023-03-09T22:44:41.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:41 smithi181 ceph-mon[94189]: pgmap v15441: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 298 KiB/s wr, 9 op/s 2023-03-09T22:44:42.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:41 smithi038 ceph-mon[92422]: pgmap v15441: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 298 KiB/s wr, 9 op/s 2023-03-09T22:44:43.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28455.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:44:43.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:43 smithi122 ceph-mon[93869]: pgmap v15442: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 180 KiB/s wr, 7 op/s 2023-03-09T22:44:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:43 smithi181 ceph-mon[94189]: pgmap v15442: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 180 KiB/s wr, 7 op/s 2023-03-09T22:44:44.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:43 smithi038 ceph-mon[92422]: pgmap v15442: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 180 KiB/s wr, 7 op/s 2023-03-09T22:44:45.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 27996.386303 secs 2023-03-09T22:44:45.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:45 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:44:45.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 27996.386303 secs 2023-03-09T22:44:45.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:45 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:44:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 27996.386303 secs 2023-03-09T22:44:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:45 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:44:46.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:46 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:44:46.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:46 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:44:46.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:46 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:44:46.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:46 smithi122 ceph-mon[93869]: pgmap v15443: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 180 KiB/s wr, 7 op/s 2023-03-09T22:44:46.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:46 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:44:46.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:46 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:44:46.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:46 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:44:46.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:46 smithi181 ceph-mon[94189]: pgmap v15443: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 180 KiB/s wr, 7 op/s 2023-03-09T22:44:46.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:46 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:44:46.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:46 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:44:46.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:46 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:44:46.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:46 smithi038 ceph-mon[92422]: pgmap v15443: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 180 KiB/s wr, 7 op/s 2023-03-09T22:44:47.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:47 smithi122 ceph-mon[93869]: pgmap v15444: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 220 KiB/s wr, 10 op/s 2023-03-09T22:44:47.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:47 smithi181 ceph-mon[94189]: pgmap v15444: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 220 KiB/s wr, 10 op/s 2023-03-09T22:44:48.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:47 smithi038 ceph-mon[92422]: pgmap v15444: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 220 KiB/s wr, 10 op/s 2023-03-09T22:44:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28460.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:44:49.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:49 smithi122 ceph-mon[93869]: pgmap v15445: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 220 KiB/s wr, 8 op/s 2023-03-09T22:44:49.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:49 smithi181 ceph-mon[94189]: pgmap v15445: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 220 KiB/s wr, 8 op/s 2023-03-09T22:44:50.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:49 smithi038 ceph-mon[92422]: pgmap v15445: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 220 KiB/s wr, 8 op/s 2023-03-09T22:44:50.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28001.386363 secs 2023-03-09T22:44:50.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28001.386363 secs 2023-03-09T22:44:51.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28001.386363 secs 2023-03-09T22:44:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:44:51.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:44:51.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:51 smithi122 ceph-mon[93869]: pgmap v15446: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 592 KiB/s rd, 523 KiB/s wr, 12 op/s 2023-03-09T22:44:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:44:51.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:44:51.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:51 smithi181 ceph-mon[94189]: pgmap v15446: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 592 KiB/s rd, 523 KiB/s wr, 12 op/s 2023-03-09T22:44:52.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:44:52.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:44:52.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:51 smithi038 ceph-mon[92422]: pgmap v15446: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 592 KiB/s rd, 523 KiB/s wr, 12 op/s 2023-03-09T22:44:53.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28465.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:44:53.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:44:53.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:44:53.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:53 smithi122 ceph-mon[93869]: pgmap v15447: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 591 KiB/s rd, 492 KiB/s wr, 10 op/s 2023-03-09T22:44:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:44:53.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:44:53.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:53 smithi181 ceph-mon[94189]: pgmap v15447: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 591 KiB/s rd, 492 KiB/s wr, 10 op/s 2023-03-09T22:44:54.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:44:54.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:44:54.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:53 smithi038 ceph-mon[92422]: pgmap v15447: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 591 KiB/s rd, 492 KiB/s wr, 10 op/s 2023-03-09T22:44:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:44:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:44:55.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28006.386425 secs 2023-03-09T22:44:55.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:55 smithi122 ceph-mon[93869]: pgmap v15448: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 602 KiB/s rd, 492 KiB/s wr, 10 op/s 2023-03-09T22:44:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:44:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:44:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28006.386425 secs 2023-03-09T22:44:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:55 smithi181 ceph-mon[94189]: pgmap v15448: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 602 KiB/s rd, 492 KiB/s wr, 10 op/s 2023-03-09T22:44:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:44:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:44:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28006.386425 secs 2023-03-09T22:44:56.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:55 smithi038 ceph-mon[92422]: pgmap v15448: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 602 KiB/s rd, 492 KiB/s wr, 10 op/s 2023-03-09T22:44:57.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:44:57.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:44:57.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:57 smithi122 ceph-mon[93869]: pgmap v15449: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 669 KiB/s rd, 557 KiB/s wr, 13 op/s 2023-03-09T22:44:57.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:44:57.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:44:57.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:57 smithi181 ceph-mon[94189]: pgmap v15449: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 669 KiB/s rd, 557 KiB/s wr, 13 op/s 2023-03-09T22:44:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:44:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:44:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:57 smithi038 ceph-mon[92422]: pgmap v15449: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 669 KiB/s rd, 557 KiB/s wr, 13 op/s 2023-03-09T22:44:58.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28470.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:44:59.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:44:59 smithi122 ceph-mon[93869]: pgmap v15450: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 565 KiB/s rd, 550 KiB/s wr, 10 op/s 2023-03-09T22:44:59.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:44:59 smithi181 ceph-mon[94189]: pgmap v15450: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 565 KiB/s rd, 550 KiB/s wr, 10 op/s 2023-03-09T22:45:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:44:59 smithi038 ceph-mon[92422]: pgmap v15450: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 565 KiB/s rd, 550 KiB/s wr, 10 op/s 2023-03-09T22:45:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28011.386487 secs 2023-03-09T22:45:00.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28011.386487 secs 2023-03-09T22:45:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28011.386487 secs 2023-03-09T22:45:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:01 smithi122 ceph-mon[93869]: pgmap v15451: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 567 KiB/s rd, 551 KiB/s wr, 14 op/s 2023-03-09T22:45:01.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:01 smithi181 ceph-mon[94189]: pgmap v15451: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 567 KiB/s rd, 551 KiB/s wr, 14 op/s 2023-03-09T22:45:02.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:01 smithi038 ceph-mon[92422]: pgmap v15451: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 567 KiB/s rd, 551 KiB/s wr, 14 op/s 2023-03-09T22:45:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28475.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:45:03.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:03 smithi122 ceph-mon[93869]: pgmap v15452: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 497 KiB/s rd, 696 KiB/s wr, 22 op/s 2023-03-09T22:45:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:03 smithi181 ceph-mon[94189]: pgmap v15452: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 497 KiB/s rd, 696 KiB/s wr, 22 op/s 2023-03-09T22:45:04.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:03 smithi038 ceph-mon[92422]: pgmap v15452: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 497 KiB/s rd, 696 KiB/s wr, 22 op/s 2023-03-09T22:45:05.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28016.386548 secs 2023-03-09T22:45:05.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28016.386548 secs 2023-03-09T22:45:05.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28016.386548 secs 2023-03-09T22:45:06.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:06 smithi122 ceph-mon[93869]: pgmap v15453: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 497 KiB/s rd, 549 KiB/s wr, 21 op/s 2023-03-09T22:45:06.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:06 smithi181 ceph-mon[94189]: pgmap v15453: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 497 KiB/s rd, 549 KiB/s wr, 21 op/s 2023-03-09T22:45:06.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:06 smithi038 ceph-mon[92422]: pgmap v15453: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 497 KiB/s rd, 549 KiB/s wr, 21 op/s 2023-03-09T22:45:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:45:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:45:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:08 smithi122 ceph-mon[93869]: pgmap v15454: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 511 KiB/s rd, 583 KiB/s wr, 25 op/s 2023-03-09T22:45:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:45:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:45:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:08 smithi181 ceph-mon[94189]: pgmap v15454: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 511 KiB/s rd, 583 KiB/s wr, 25 op/s 2023-03-09T22:45:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:45:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:45:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:08 smithi038 ceph-mon[92422]: pgmap v15454: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 511 KiB/s rd, 583 KiB/s wr, 25 op/s 2023-03-09T22:45:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28480.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:45:09.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:45:09.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:45:09.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:45:09.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:45:09.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:45:09.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:45:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:10 smithi122 ceph-mon[93869]: pgmap v15455: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 781 KiB/s rd, 647 KiB/s wr, 23 op/s 2023-03-09T22:45:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28021.386608 secs 2023-03-09T22:45:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:10 smithi181 ceph-mon[94189]: pgmap v15455: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 781 KiB/s rd, 647 KiB/s wr, 23 op/s 2023-03-09T22:45:10.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28021.386608 secs 2023-03-09T22:45:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:10 smithi038 ceph-mon[92422]: pgmap v15455: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 781 KiB/s rd, 647 KiB/s wr, 23 op/s 2023-03-09T22:45:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28021.386608 secs 2023-03-09T22:45:11.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:11 smithi122 ceph-mon[93869]: pgmap v15456: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 636 KiB/s wr, 25 op/s 2023-03-09T22:45:11.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:11 smithi181 ceph-mon[94189]: pgmap v15456: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 636 KiB/s wr, 25 op/s 2023-03-09T22:45:12.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:11 smithi038 ceph-mon[92422]: pgmap v15456: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 636 KiB/s wr, 25 op/s 2023-03-09T22:45:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28485.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:45:13.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:13 smithi122 ceph-mon[93869]: pgmap v15457: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 776 KiB/s rd, 890 KiB/s wr, 23 op/s 2023-03-09T22:45:13.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:13 smithi181 ceph-mon[94189]: pgmap v15457: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 776 KiB/s rd, 890 KiB/s wr, 23 op/s 2023-03-09T22:45:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:13 smithi038 ceph-mon[92422]: pgmap v15457: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 776 KiB/s rd, 890 KiB/s wr, 23 op/s 2023-03-09T22:45:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:15 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:45:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:15 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:45:15.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28026.386668 secs 2023-03-09T22:45:15.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:15 smithi122 ceph-mon[93869]: pgmap v15458: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 539 KiB/s rd, 441 KiB/s wr, 10 op/s 2023-03-09T22:45:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:15 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:45:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:15 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:45:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28026.386668 secs 2023-03-09T22:45:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:15 smithi181 ceph-mon[94189]: pgmap v15458: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 539 KiB/s rd, 441 KiB/s wr, 10 op/s 2023-03-09T22:45:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:15 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:45:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:15 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:45:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28026.386668 secs 2023-03-09T22:45:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:15 smithi038 ceph-mon[92422]: pgmap v15458: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 539 KiB/s rd, 441 KiB/s wr, 10 op/s 2023-03-09T22:45:17.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:45:17.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:45:17.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:17 smithi122 ceph-mon[93869]: pgmap v15459: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 585 KiB/s rd, 443 KiB/s wr, 14 op/s 2023-03-09T22:45:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:45:17.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:45:17.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:17 smithi181 ceph-mon[94189]: pgmap v15459: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 585 KiB/s rd, 443 KiB/s wr, 14 op/s 2023-03-09T22:45:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:45:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:45:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:17 smithi038 ceph-mon[92422]: pgmap v15459: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 585 KiB/s rd, 443 KiB/s wr, 14 op/s 2023-03-09T22:45:18.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28490.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:45:19.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:19 smithi122 ceph-mon[93869]: pgmap v15460: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 726 KiB/s rd, 431 KiB/s wr, 12 op/s 2023-03-09T22:45:19.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:19 smithi181 ceph-mon[94189]: pgmap v15460: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 726 KiB/s rd, 431 KiB/s wr, 12 op/s 2023-03-09T22:45:20.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:19 smithi038 ceph-mon[92422]: pgmap v15460: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 726 KiB/s rd, 431 KiB/s wr, 12 op/s 2023-03-09T22:45:20.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28031.386728 secs 2023-03-09T22:45:20.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:45:20.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28031.386728 secs 2023-03-09T22:45:20.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:45:21.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28031.386728 secs 2023-03-09T22:45:21.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:45:21.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:45:21.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:45:21.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:45:21.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:21 smithi122 ceph-mon[93869]: pgmap v15461: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 313 KiB/s wr, 13 op/s 2023-03-09T22:45:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:45:21.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:45:21.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:45:21.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:21 smithi181 ceph-mon[94189]: pgmap v15461: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 313 KiB/s wr, 13 op/s 2023-03-09T22:45:22.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:45:22.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:45:22.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:45:22.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:21 smithi038 ceph-mon[92422]: pgmap v15461: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 313 KiB/s wr, 13 op/s 2023-03-09T22:45:23.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28033.765604 secs 2023-03-09T22:45:23.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28033.765604 secs 2023-03-09T22:45:23.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28033.765604 secs 2023-03-09T22:45:23.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28495.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:45:24.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:23 smithi122 ceph-mon[93869]: pgmap v15462: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 697 KiB/s rd, 567 KiB/s wr, 15 op/s 2023-03-09T22:45:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:23 smithi181 ceph-mon[94189]: pgmap v15462: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 697 KiB/s rd, 567 KiB/s wr, 15 op/s 2023-03-09T22:45:24.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:23 smithi038 ceph-mon[92422]: pgmap v15462: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 697 KiB/s rd, 567 KiB/s wr, 15 op/s 2023-03-09T22:45:25.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28036.386813 secs 2023-03-09T22:45:25.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:25 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:45:25.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:25 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:45:25.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:25 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:45:25.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28036.386813 secs 2023-03-09T22:45:25.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:25 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:45:25.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:25 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:45:25.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:25 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:45:25.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28036.386813 secs 2023-03-09T22:45:25.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:25 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:45:25.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:25 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:45:25.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:25 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:45:26.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:26 smithi122 ceph-mon[93869]: pgmap v15463: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 543 KiB/s rd, 312 KiB/s wr, 13 op/s 2023-03-09T22:45:26.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:26 smithi181 ceph-mon[94189]: pgmap v15463: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 543 KiB/s rd, 312 KiB/s wr, 13 op/s 2023-03-09T22:45:26.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:26 smithi038 ceph-mon[92422]: pgmap v15463: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 543 KiB/s rd, 312 KiB/s wr, 13 op/s 2023-03-09T22:45:28.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:28 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:45:28.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:28 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:45:28.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28038.765717 secs 2023-03-09T22:45:28.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:28 smithi122 ceph-mon[93869]: pgmap v15464: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 312 KiB/s wr, 16 op/s 2023-03-09T22:45:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:28 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:45:28.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:28 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:45:28.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28038.765717 secs 2023-03-09T22:45:28.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:28 smithi181 ceph-mon[94189]: pgmap v15464: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 312 KiB/s wr, 16 op/s 2023-03-09T22:45:28.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:28 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:45:28.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:28 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:45:28.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28038.765717 secs 2023-03-09T22:45:28.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:28 smithi038 ceph-mon[92422]: pgmap v15464: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 312 KiB/s wr, 16 op/s 2023-03-09T22:45:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28500.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:45:29.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:29 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:45:29.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:29 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:45:29.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:29 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:45:29.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:29 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:45:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:29 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:45:29.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:29 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:45:30.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:30 smithi122 ceph-mon[93869]: osdmap e1181: 12 total, 12 up, 12 in 2023-03-09T22:45:30.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:30 smithi122 ceph-mon[93869]: pgmap v15466: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 758 KiB/s rd, 705 KiB/s wr, 15 op/s 2023-03-09T22:45:30.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28041.386872 secs 2023-03-09T22:45:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:30 smithi181 ceph-mon[94189]: osdmap e1181: 12 total, 12 up, 12 in 2023-03-09T22:45:30.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:30 smithi181 ceph-mon[94189]: pgmap v15466: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 758 KiB/s rd, 705 KiB/s wr, 15 op/s 2023-03-09T22:45:30.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28041.386872 secs 2023-03-09T22:45:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:30 smithi038 ceph-mon[92422]: osdmap e1181: 12 total, 12 up, 12 in 2023-03-09T22:45:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:30 smithi038 ceph-mon[92422]: pgmap v15466: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 758 KiB/s rd, 705 KiB/s wr, 15 op/s 2023-03-09T22:45:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28041.386872 secs 2023-03-09T22:45:32.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:32 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:45:32.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:32 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:45:32.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:32 smithi122 ceph-mon[93869]: pgmap v15467: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 759 KiB/s rd, 693 KiB/s wr, 14 op/s 2023-03-09T22:45:32.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:32 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:45:32.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:32 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:45:32.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:32 smithi181 ceph-mon[94189]: pgmap v15467: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 759 KiB/s rd, 693 KiB/s wr, 14 op/s 2023-03-09T22:45:32.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:32 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:45:32.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:32 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:45:32.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:32 smithi038 ceph-mon[92422]: pgmap v15467: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 759 KiB/s rd, 693 KiB/s wr, 14 op/s 2023-03-09T22:45:33.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28043.765813 secs 2023-03-09T22:45:33.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:33 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:45:33.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:33 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:45:33.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28043.765813 secs 2023-03-09T22:45:33.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:33 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:45:33.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:33 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:45:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28043.765813 secs 2023-03-09T22:45:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:33 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:45:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:33 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:45:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28505.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:45:34.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:34 smithi122 ceph-mon[93869]: pgmap v15468: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 362 KiB/s wr, 10 op/s 2023-03-09T22:45:34.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:34 smithi181 ceph-mon[94189]: pgmap v15468: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 362 KiB/s wr, 10 op/s 2023-03-09T22:45:34.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:34 smithi038 ceph-mon[92422]: pgmap v15468: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 362 KiB/s wr, 10 op/s 2023-03-09T22:45:35.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28046.386933 secs 2023-03-09T22:45:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28046.386933 secs 2023-03-09T22:45:35.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28046.386933 secs 2023-03-09T22:45:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:36 smithi122 ceph-mon[93869]: pgmap v15469: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 461 KiB/s wr, 11 op/s 2023-03-09T22:45:36.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:36 smithi181 ceph-mon[94189]: pgmap v15469: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 461 KiB/s wr, 11 op/s 2023-03-09T22:45:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:36 smithi038 ceph-mon[92422]: pgmap v15469: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 461 KiB/s wr, 11 op/s 2023-03-09T22:45:37.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:37 smithi122 ceph-mon[93869]: osdmap e1182: 12 total, 12 up, 12 in 2023-03-09T22:45:37.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28048.765934 secs 2023-03-09T22:45:37.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:37 smithi122 ceph-mon[93869]: pgmap v15471: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 123 KiB/s wr, 10 op/s 2023-03-09T22:45:37.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:37 smithi181 ceph-mon[94189]: osdmap e1182: 12 total, 12 up, 12 in 2023-03-09T22:45:37.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28048.765934 secs 2023-03-09T22:45:37.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:37 smithi181 ceph-mon[94189]: pgmap v15471: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 123 KiB/s wr, 10 op/s 2023-03-09T22:45:38.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:37 smithi038 ceph-mon[92422]: osdmap e1182: 12 total, 12 up, 12 in 2023-03-09T22:45:38.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28048.765934 secs 2023-03-09T22:45:38.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:37 smithi038 ceph-mon[92422]: pgmap v15471: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 123 KiB/s wr, 10 op/s 2023-03-09T22:45:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28510.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:45:39.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:39 smithi122 ceph-mon[93869]: pgmap v15472: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 102 KiB/s wr, 10 op/s 2023-03-09T22:45:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:39 smithi181 ceph-mon[94189]: pgmap v15472: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 102 KiB/s wr, 10 op/s 2023-03-09T22:45:40.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:39 smithi038 ceph-mon[92422]: pgmap v15472: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 102 KiB/s wr, 10 op/s 2023-03-09T22:45:40.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28051.386986 secs 2023-03-09T22:45:40.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28051.386986 secs 2023-03-09T22:45:41.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28051.386986 secs 2023-03-09T22:45:41.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:41 smithi122 ceph-mon[93869]: pgmap v15473: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 100 KiB/s wr, 10 op/s 2023-03-09T22:45:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:41 smithi181 ceph-mon[94189]: pgmap v15473: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 100 KiB/s wr, 10 op/s 2023-03-09T22:45:42.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:41 smithi038 ceph-mon[92422]: pgmap v15473: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 100 KiB/s wr, 10 op/s 2023-03-09T22:45:42.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:42 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 28053.766053 secs 2023-03-09T22:45:42.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:42 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 28053.766053 secs 2023-03-09T22:45:43.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:42 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 28053.766053 secs 2023-03-09T22:45:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28515.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:45:43.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:43 smithi122 ceph-mon[93869]: pgmap v15474: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 262 KiB/s wr, 9 op/s 2023-03-09T22:45:43.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:43 smithi181 ceph-mon[94189]: pgmap v15474: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 262 KiB/s wr, 9 op/s 2023-03-09T22:45:44.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:43 smithi038 ceph-mon[92422]: pgmap v15474: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 262 KiB/s wr, 9 op/s 2023-03-09T22:45:45.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28056.387070 secs 2023-03-09T22:45:45.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:45 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:45:45.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:45 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:45:45.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28056.387070 secs 2023-03-09T22:45:45.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:45 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:45:45.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:45 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:45:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28056.387070 secs 2023-03-09T22:45:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:45 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:45:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:45 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:45:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:46 smithi122 ceph-mon[93869]: pgmap v15475: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 236 KiB/s wr, 9 op/s 2023-03-09T22:45:46.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:46 smithi181 ceph-mon[94189]: pgmap v15475: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 236 KiB/s wr, 9 op/s 2023-03-09T22:45:46.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:46 smithi038 ceph-mon[92422]: pgmap v15475: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 236 KiB/s wr, 9 op/s 2023-03-09T22:45:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:45:48.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:45:48.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:48 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 28058.766209 secs 2023-03-09T22:45:48.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:48 smithi122 ceph-mon[93869]: pgmap v15476: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 279 KiB/s wr, 10 op/s 2023-03-09T22:45:48.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:45:48.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:45:48.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:48 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 28058.766209 secs 2023-03-09T22:45:48.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:48 smithi181 ceph-mon[94189]: pgmap v15476: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 279 KiB/s wr, 10 op/s 2023-03-09T22:45:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:45:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:45:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:48 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 28058.766209 secs 2023-03-09T22:45:48.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:48 smithi038 ceph-mon[92422]: pgmap v15476: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 279 KiB/s wr, 10 op/s 2023-03-09T22:45:48.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28520.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:45:49.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:45:49.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:45:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:45:49.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:45:49.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:45:49.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:45:50.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:50 smithi122 ceph-mon[93869]: pgmap v15477: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 253 KiB/s wr, 9 op/s 2023-03-09T22:45:50.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28061.387132 secs 2023-03-09T22:45:50.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:50 smithi181 ceph-mon[94189]: pgmap v15477: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 253 KiB/s wr, 9 op/s 2023-03-09T22:45:50.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28061.387132 secs 2023-03-09T22:45:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:50 smithi038 ceph-mon[92422]: pgmap v15477: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 253 KiB/s wr, 9 op/s 2023-03-09T22:45:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28061.387132 secs 2023-03-09T22:45:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:51 smithi122 ceph-mon[93869]: pgmap v15478: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 254 KiB/s wr, 10 op/s 2023-03-09T22:45:51.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:51 smithi181 ceph-mon[94189]: pgmap v15478: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 254 KiB/s wr, 10 op/s 2023-03-09T22:45:52.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:51 smithi038 ceph-mon[92422]: pgmap v15478: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 254 KiB/s wr, 10 op/s 2023-03-09T22:45:52.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28063.766333 secs 2023-03-09T22:45:52.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28063.766333 secs 2023-03-09T22:45:53.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28063.766333 secs 2023-03-09T22:45:53.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28525.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:45:53.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:53 smithi122 ceph-mon[93869]: pgmap v15479: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 793 KiB/s wr, 12 op/s 2023-03-09T22:45:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:53 smithi181 ceph-mon[94189]: pgmap v15479: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 793 KiB/s wr, 12 op/s 2023-03-09T22:45:54.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:53 smithi038 ceph-mon[92422]: pgmap v15479: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 793 KiB/s wr, 12 op/s 2023-03-09T22:45:54.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:54 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:45:54.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:54 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:45:54.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:54 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:45:54.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:54 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:45:55.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:54 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:45:55.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:54 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:45:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:45:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:45:55.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28066.387210 secs 2023-03-09T22:45:55.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:55 smithi122 ceph-mon[93869]: pgmap v15480: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 508 KiB/s rd, 658 KiB/s wr, 11 op/s 2023-03-09T22:45:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:45:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:45:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28066.387210 secs 2023-03-09T22:45:55.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:55 smithi181 ceph-mon[94189]: pgmap v15480: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 508 KiB/s rd, 658 KiB/s wr, 11 op/s 2023-03-09T22:45:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:45:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:45:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28066.387210 secs 2023-03-09T22:45:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:55 smithi038 ceph-mon[92422]: pgmap v15480: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 508 KiB/s rd, 658 KiB/s wr, 11 op/s 2023-03-09T22:45:57.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:45:57.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:45:57.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28068.766422 secs 2023-03-09T22:45:57.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:57 smithi122 ceph-mon[93869]: pgmap v15481: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 699 KiB/s rd, 848 KiB/s wr, 15 op/s 2023-03-09T22:45:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:45:57.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:45:57.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28068.766422 secs 2023-03-09T22:45:57.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:57 smithi181 ceph-mon[94189]: pgmap v15481: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 699 KiB/s rd, 848 KiB/s wr, 15 op/s 2023-03-09T22:45:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:45:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:45:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28068.766422 secs 2023-03-09T22:45:58.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:57 smithi038 ceph-mon[92422]: pgmap v15481: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 699 KiB/s rd, 848 KiB/s wr, 15 op/s 2023-03-09T22:45:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28530.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:45:59.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:45:59.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:45:59.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:45:59 smithi122 ceph-mon[93869]: pgmap v15482: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 805 KiB/s rd, 792 KiB/s wr, 12 op/s 2023-03-09T22:45:59.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:45:59.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:45:59.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:45:59 smithi181 ceph-mon[94189]: pgmap v15482: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 805 KiB/s rd, 792 KiB/s wr, 12 op/s 2023-03-09T22:46:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:46:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:46:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:45:59 smithi038 ceph-mon[92422]: pgmap v15482: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 805 KiB/s rd, 792 KiB/s wr, 12 op/s 2023-03-09T22:46:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28071.387269 secs 2023-03-09T22:46:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28071.387269 secs 2023-03-09T22:46:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28071.387269 secs 2023-03-09T22:46:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:01 smithi122 ceph-mon[93869]: pgmap v15483: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 923 KiB/s rd, 920 KiB/s wr, 16 op/s 2023-03-09T22:46:01.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:01 smithi181 ceph-mon[94189]: pgmap v15483: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 923 KiB/s rd, 920 KiB/s wr, 16 op/s 2023-03-09T22:46:02.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:01 smithi038 ceph-mon[92422]: pgmap v15483: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 923 KiB/s rd, 920 KiB/s wr, 16 op/s 2023-03-09T22:46:02.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28073.766547 secs 2023-03-09T22:46:02.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28073.766547 secs 2023-03-09T22:46:03.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28073.766547 secs 2023-03-09T22:46:03.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28535.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:46:03.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:03 smithi122 ceph-mon[93869]: pgmap v15484: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 946 KiB/s rd, 960 KiB/s wr, 27 op/s 2023-03-09T22:46:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:03 smithi181 ceph-mon[94189]: pgmap v15484: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 946 KiB/s rd, 960 KiB/s wr, 27 op/s 2023-03-09T22:46:04.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:03 smithi038 ceph-mon[92422]: pgmap v15484: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 946 KiB/s rd, 960 KiB/s wr, 27 op/s 2023-03-09T22:46:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:05 smithi122 ceph-mon[93869]: osdmap e1183: 12 total, 12 up, 12 in 2023-03-09T22:46:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28076.387348 secs 2023-03-09T22:46:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:05 smithi122 ceph-mon[93869]: pgmap v15486: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 672 KiB/s rd, 505 KiB/s wr, 27 op/s 2023-03-09T22:46:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:05 smithi181 ceph-mon[94189]: osdmap e1183: 12 total, 12 up, 12 in 2023-03-09T22:46:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28076.387348 secs 2023-03-09T22:46:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:05 smithi181 ceph-mon[94189]: pgmap v15486: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 672 KiB/s rd, 505 KiB/s wr, 27 op/s 2023-03-09T22:46:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:05 smithi038 ceph-mon[92422]: osdmap e1183: 12 total, 12 up, 12 in 2023-03-09T22:46:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28076.387348 secs 2023-03-09T22:46:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:05 smithi038 ceph-mon[92422]: pgmap v15486: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 672 KiB/s rd, 505 KiB/s wr, 27 op/s 2023-03-09T22:46:07.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:46:07.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:46:07.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28078.766676 secs 2023-03-09T22:46:07.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:07 smithi122 ceph-mon[93869]: pgmap v15487: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 407 KiB/s wr, 28 op/s 2023-03-09T22:46:07.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:46:07.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:46:07.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28078.766676 secs 2023-03-09T22:46:07.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:07 smithi181 ceph-mon[94189]: pgmap v15487: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 407 KiB/s wr, 28 op/s 2023-03-09T22:46:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:46:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:46:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28078.766676 secs 2023-03-09T22:46:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:07 smithi038 ceph-mon[92422]: pgmap v15487: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 407 KiB/s wr, 28 op/s 2023-03-09T22:46:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28540.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:46:09.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:46:09.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:46:09.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:09 smithi122 ceph-mon[93869]: pgmap v15488: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 407 KiB/s wr, 27 op/s 2023-03-09T22:46:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:46:09.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:46:09.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:09 smithi181 ceph-mon[94189]: pgmap v15488: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 407 KiB/s wr, 27 op/s 2023-03-09T22:46:10.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:46:10.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:46:10.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:09 smithi038 ceph-mon[92422]: pgmap v15488: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 407 KiB/s wr, 27 op/s 2023-03-09T22:46:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28081.387407 secs 2023-03-09T22:46:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28081.387407 secs 2023-03-09T22:46:11.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28081.387407 secs 2023-03-09T22:46:11.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:11 smithi122 ceph-mon[93869]: pgmap v15489: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 352 KiB/s wr, 25 op/s 2023-03-09T22:46:11.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:11 smithi122 ceph-mon[93869]: osdmap e1184: 12 total, 12 up, 12 in 2023-03-09T22:46:11.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:11 smithi181 ceph-mon[94189]: pgmap v15489: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 352 KiB/s wr, 25 op/s 2023-03-09T22:46:11.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:11 smithi181 ceph-mon[94189]: osdmap e1184: 12 total, 12 up, 12 in 2023-03-09T22:46:12.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:11 smithi038 ceph-mon[92422]: pgmap v15489: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 352 KiB/s wr, 25 op/s 2023-03-09T22:46:12.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:11 smithi038 ceph-mon[92422]: osdmap e1184: 12 total, 12 up, 12 in 2023-03-09T22:46:12.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28083.766752 secs 2023-03-09T22:46:12.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28083.766752 secs 2023-03-09T22:46:13.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28083.766752 secs 2023-03-09T22:46:13.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28545.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:46:13.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:13 smithi122 ceph-mon[93869]: pgmap v15491: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 342 KiB/s wr, 11 op/s 2023-03-09T22:46:13.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:13 smithi181 ceph-mon[94189]: pgmap v15491: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 342 KiB/s wr, 11 op/s 2023-03-09T22:46:14.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:13 smithi038 ceph-mon[92422]: pgmap v15491: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 342 KiB/s wr, 11 op/s 2023-03-09T22:46:15.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28086.387464 secs 2023-03-09T22:46:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28086.387464 secs 2023-03-09T22:46:15.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28086.387464 secs 2023-03-09T22:46:16.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:16 smithi122 ceph-mon[93869]: pgmap v15492: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 293 KiB/s rd, 302 KiB/s wr, 10 op/s 2023-03-09T22:46:16.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:16 smithi181 ceph-mon[94189]: pgmap v15492: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 293 KiB/s rd, 302 KiB/s wr, 10 op/s 2023-03-09T22:46:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:16 smithi038 ceph-mon[92422]: pgmap v15492: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 293 KiB/s rd, 302 KiB/s wr, 10 op/s 2023-03-09T22:46:17.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28088.766886 secs 2023-03-09T22:46:17.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:17 smithi122 ceph-mon[93869]: pgmap v15493: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 99 KiB/s wr, 7 op/s 2023-03-09T22:46:17.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28088.766886 secs 2023-03-09T22:46:17.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:17 smithi181 ceph-mon[94189]: pgmap v15493: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 99 KiB/s wr, 7 op/s 2023-03-09T22:46:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28088.766886 secs 2023-03-09T22:46:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:17 smithi038 ceph-mon[92422]: pgmap v15493: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 99 KiB/s wr, 7 op/s 2023-03-09T22:46:18.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28550.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:46:19.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:19 smithi122 ceph-mon[93869]: pgmap v15494: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 99 KiB/s wr, 7 op/s 2023-03-09T22:46:19.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:19 smithi181 ceph-mon[94189]: pgmap v15494: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 99 KiB/s wr, 7 op/s 2023-03-09T22:46:20.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:19 smithi038 ceph-mon[92422]: pgmap v15494: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 99 KiB/s wr, 7 op/s 2023-03-09T22:46:20.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28091.387523 secs 2023-03-09T22:46:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28091.387523 secs 2023-03-09T22:46:21.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28091.387523 secs 2023-03-09T22:46:21.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:46:21.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:21 smithi122 ceph-mon[93869]: pgmap v15495: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 117 KiB/s wr, 7 op/s 2023-03-09T22:46:21.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:46:21.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:21 smithi181 ceph-mon[94189]: pgmap v15495: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 117 KiB/s wr, 7 op/s 2023-03-09T22:46:22.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:46:22.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:21 smithi038 ceph-mon[92422]: pgmap v15495: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 117 KiB/s wr, 7 op/s 2023-03-09T22:46:22.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:22 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:46:22.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:22 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:46:22.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:22 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:46:22.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28093.767001 secs 2023-03-09T22:46:22.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:22 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:46:22.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:22 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:46:22.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:22 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:46:22.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28093.767001 secs 2023-03-09T22:46:23.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:22 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:46:23.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:22 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:46:23.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:22 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:46:23.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28093.767001 secs 2023-03-09T22:46:23.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28555.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:46:23.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:23 smithi122 ceph-mon[93869]: pgmap v15496: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 98 KiB/s wr, 6 op/s 2023-03-09T22:46:23.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:23 smithi181 ceph-mon[94189]: pgmap v15496: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 98 KiB/s wr, 6 op/s 2023-03-09T22:46:24.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:23 smithi038 ceph-mon[92422]: pgmap v15496: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 98 KiB/s wr, 6 op/s 2023-03-09T22:46:25.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28096.387586 secs 2023-03-09T22:46:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28096.387586 secs 2023-03-09T22:46:25.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28096.387586 secs 2023-03-09T22:46:26.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:26 smithi122 ceph-mon[93869]: pgmap v15497: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 97 KiB/s wr, 5 op/s 2023-03-09T22:46:26.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:26 smithi181 ceph-mon[94189]: pgmap v15497: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 97 KiB/s wr, 5 op/s 2023-03-09T22:46:26.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:26 smithi038 ceph-mon[92422]: pgmap v15497: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 97 KiB/s wr, 5 op/s 2023-03-09T22:46:27.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28098.767143 secs 2023-03-09T22:46:27.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:27 smithi122 ceph-mon[93869]: pgmap v15498: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 97 KiB/s wr, 8 op/s 2023-03-09T22:46:27.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28098.767143 secs 2023-03-09T22:46:27.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:27 smithi181 ceph-mon[94189]: pgmap v15498: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 97 KiB/s wr, 8 op/s 2023-03-09T22:46:28.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28098.767143 secs 2023-03-09T22:46:28.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:27 smithi038 ceph-mon[92422]: pgmap v15498: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 97 KiB/s wr, 8 op/s 2023-03-09T22:46:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28560.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:46:29.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:29 smithi122 ceph-mon[93869]: pgmap v15499: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 97 KiB/s wr, 5 op/s 2023-03-09T22:46:29.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:29 smithi181 ceph-mon[94189]: pgmap v15499: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 97 KiB/s wr, 5 op/s 2023-03-09T22:46:30.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:29 smithi038 ceph-mon[92422]: pgmap v15499: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 97 KiB/s wr, 5 op/s 2023-03-09T22:46:30.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28101.387647 secs 2023-03-09T22:46:30.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28101.387647 secs 2023-03-09T22:46:31.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28101.387647 secs 2023-03-09T22:46:31.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:31 smithi122 ceph-mon[93869]: pgmap v15500: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 162 KiB/s wr, 8 op/s 2023-03-09T22:46:31.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:31 smithi181 ceph-mon[94189]: pgmap v15500: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 162 KiB/s wr, 8 op/s 2023-03-09T22:46:32.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:31 smithi038 ceph-mon[92422]: pgmap v15500: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 162 KiB/s wr, 8 op/s 2023-03-09T22:46:32.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28103.767191 secs 2023-03-09T22:46:32.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28103.767191 secs 2023-03-09T22:46:33.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28103.767191 secs 2023-03-09T22:46:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28565.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:46:33.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:33 smithi122 ceph-mon[93869]: pgmap v15501: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 65 KiB/s wr, 5 op/s 2023-03-09T22:46:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:33 smithi181 ceph-mon[94189]: pgmap v15501: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 65 KiB/s wr, 5 op/s 2023-03-09T22:46:34.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:33 smithi038 ceph-mon[92422]: pgmap v15501: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 65 KiB/s wr, 5 op/s 2023-03-09T22:46:35.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28106.387716 secs 2023-03-09T22:46:35.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28106.387716 secs 2023-03-09T22:46:35.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28106.387716 secs 2023-03-09T22:46:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:36 smithi122 ceph-mon[93869]: pgmap v15502: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 65 KiB/s wr, 5 op/s 2023-03-09T22:46:36.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:36 smithi181 ceph-mon[94189]: pgmap v15502: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 65 KiB/s wr, 5 op/s 2023-03-09T22:46:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:36 smithi038 ceph-mon[92422]: pgmap v15502: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 65 KiB/s wr, 5 op/s 2023-03-09T22:46:37.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:46:37.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:46:37.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:46:37.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:46:37.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:46:37.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:46:38.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28108.767285 secs 2023-03-09T22:46:38.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:38 smithi122 ceph-mon[93869]: pgmap v15503: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 165 KiB/s wr, 9 op/s 2023-03-09T22:46:38.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28108.767285 secs 2023-03-09T22:46:38.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:38 smithi181 ceph-mon[94189]: pgmap v15503: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 165 KiB/s wr, 9 op/s 2023-03-09T22:46:38.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28108.767285 secs 2023-03-09T22:46:38.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:38 smithi038 ceph-mon[92422]: pgmap v15503: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 165 KiB/s wr, 9 op/s 2023-03-09T22:46:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28570.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:46:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:46:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:46:40.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:40 smithi122 ceph-mon[93869]: pgmap v15504: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 165 KiB/s wr, 7 op/s 2023-03-09T22:46:40.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28111.387776 secs 2023-03-09T22:46:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:46:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:46:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:40 smithi181 ceph-mon[94189]: pgmap v15504: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 165 KiB/s wr, 7 op/s 2023-03-09T22:46:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28111.387776 secs 2023-03-09T22:46:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:46:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:46:40.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:40 smithi038 ceph-mon[92422]: pgmap v15504: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 165 KiB/s wr, 7 op/s 2023-03-09T22:46:40.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28111.387776 secs 2023-03-09T22:46:41.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:41 smithi122 ceph-mon[93869]: pgmap v15505: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 347 KiB/s wr, 9 op/s 2023-03-09T22:46:41.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:41 smithi181 ceph-mon[94189]: pgmap v15505: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 347 KiB/s wr, 9 op/s 2023-03-09T22:46:42.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:41 smithi038 ceph-mon[92422]: pgmap v15505: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 347 KiB/s wr, 9 op/s 2023-03-09T22:46:42.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28113.767405 secs 2023-03-09T22:46:42.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28113.767405 secs 2023-03-09T22:46:43.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28113.767405 secs 2023-03-09T22:46:43.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28575.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:46:43.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:43 smithi122 ceph-mon[93869]: pgmap v15506: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 384 KiB/s wr, 8 op/s 2023-03-09T22:46:43.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:43 smithi181 ceph-mon[94189]: pgmap v15506: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 384 KiB/s wr, 8 op/s 2023-03-09T22:46:44.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:43 smithi038 ceph-mon[92422]: pgmap v15506: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 384 KiB/s wr, 8 op/s 2023-03-09T22:46:45.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28116.387837 secs 2023-03-09T22:46:45.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28116.387837 secs 2023-03-09T22:46:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28116.387837 secs 2023-03-09T22:46:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:46 smithi122 ceph-mon[93869]: pgmap v15507: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 384 KiB/s wr, 7 op/s 2023-03-09T22:46:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:46 smithi181 ceph-mon[94189]: pgmap v15507: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 384 KiB/s wr, 7 op/s 2023-03-09T22:46:46.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:46 smithi038 ceph-mon[92422]: pgmap v15507: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 384 KiB/s wr, 7 op/s 2023-03-09T22:46:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:46:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:46:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28118.767521 secs 2023-03-09T22:46:48.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:48 smithi122 ceph-mon[93869]: pgmap v15508: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 472 KiB/s rd, 407 KiB/s wr, 11 op/s 2023-03-09T22:46:48.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:46:48.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:46:48.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28118.767521 secs 2023-03-09T22:46:48.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:48 smithi181 ceph-mon[94189]: pgmap v15508: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 472 KiB/s rd, 407 KiB/s wr, 11 op/s 2023-03-09T22:46:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:46:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:46:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28118.767521 secs 2023-03-09T22:46:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:48 smithi038 ceph-mon[92422]: pgmap v15508: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 472 KiB/s rd, 407 KiB/s wr, 11 op/s 2023-03-09T22:46:48.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28580.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:46:49.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:46:49.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:46:49.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:46:49.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:46:49.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:46:49.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:46:50.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:50 smithi122 ceph-mon[93869]: pgmap v15509: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 341 KiB/s wr, 8 op/s 2023-03-09T22:46:50.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28121.387898 secs 2023-03-09T22:46:50.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:50 smithi181 ceph-mon[94189]: pgmap v15509: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 341 KiB/s wr, 8 op/s 2023-03-09T22:46:50.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28121.387898 secs 2023-03-09T22:46:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:50 smithi038 ceph-mon[92422]: pgmap v15509: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 341 KiB/s wr, 8 op/s 2023-03-09T22:46:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28121.387898 secs 2023-03-09T22:46:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:51 smithi122 ceph-mon[93869]: pgmap v15510: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 341 KiB/s wr, 10 op/s 2023-03-09T22:46:51.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:51 smithi181 ceph-mon[94189]: pgmap v15510: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 341 KiB/s wr, 10 op/s 2023-03-09T22:46:52.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:51 smithi038 ceph-mon[92422]: pgmap v15510: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 341 KiB/s wr, 10 op/s 2023-03-09T22:46:52.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28123.767587 secs 2023-03-09T22:46:52.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28123.767587 secs 2023-03-09T22:46:53.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28123.767587 secs 2023-03-09T22:46:53.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28585.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:46:53.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:53 smithi122 ceph-mon[93869]: pgmap v15511: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 465 KiB/s wr, 10 op/s 2023-03-09T22:46:53.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:53 smithi181 ceph-mon[94189]: pgmap v15511: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 465 KiB/s wr, 10 op/s 2023-03-09T22:46:54.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:53 smithi038 ceph-mon[92422]: pgmap v15511: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 465 KiB/s wr, 10 op/s 2023-03-09T22:46:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:46:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:46:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28126.387960 secs 2023-03-09T22:46:55.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:55 smithi122 ceph-mon[93869]: pgmap v15512: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 363 KiB/s wr, 10 op/s 2023-03-09T22:46:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:46:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:46:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28126.387960 secs 2023-03-09T22:46:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:55 smithi181 ceph-mon[94189]: pgmap v15512: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 363 KiB/s wr, 10 op/s 2023-03-09T22:46:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:46:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:46:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28126.387960 secs 2023-03-09T22:46:56.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:55 smithi038 ceph-mon[92422]: pgmap v15512: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 363 KiB/s wr, 10 op/s 2023-03-09T22:46:57.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:46:57.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:46:57.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28128.767708 secs 2023-03-09T22:46:57.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:57 smithi122 ceph-mon[93869]: pgmap v15513: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 668 KiB/s rd, 654 KiB/s wr, 14 op/s 2023-03-09T22:46:57.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:46:57.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:46:57.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28128.767708 secs 2023-03-09T22:46:57.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:57 smithi181 ceph-mon[94189]: pgmap v15513: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 668 KiB/s rd, 654 KiB/s wr, 14 op/s 2023-03-09T22:46:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:46:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:46:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28128.767708 secs 2023-03-09T22:46:58.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:57 smithi038 ceph-mon[92422]: pgmap v15513: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 668 KiB/s rd, 654 KiB/s wr, 14 op/s 2023-03-09T22:46:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28590.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:46:59.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:46:59.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:46:59.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:46:59 smithi122 ceph-mon[93869]: pgmap v15514: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 755 KiB/s rd, 742 KiB/s wr, 12 op/s 2023-03-09T22:46:59.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:46:59.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:46:59.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:46:59 smithi181 ceph-mon[94189]: pgmap v15514: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 755 KiB/s rd, 742 KiB/s wr, 12 op/s 2023-03-09T22:47:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:47:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:47:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:46:59 smithi038 ceph-mon[92422]: pgmap v15514: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 755 KiB/s rd, 742 KiB/s wr, 12 op/s 2023-03-09T22:47:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28131.387986 secs 2023-03-09T22:47:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28131.387986 secs 2023-03-09T22:47:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28131.387986 secs 2023-03-09T22:47:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:47:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:47:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:01 smithi122 ceph-mon[93869]: pgmap v15515: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 723 KiB/s rd, 708 KiB/s wr, 13 op/s 2023-03-09T22:47:01.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:47:01.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:47:01.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:01 smithi181 ceph-mon[94189]: pgmap v15515: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 723 KiB/s rd, 708 KiB/s wr, 13 op/s 2023-03-09T22:47:02.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:47:02.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:47:02.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:01 smithi038 ceph-mon[92422]: pgmap v15515: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 723 KiB/s rd, 708 KiB/s wr, 13 op/s 2023-03-09T22:47:02.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28133.767845 secs 2023-03-09T22:47:02.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28133.767845 secs 2023-03-09T22:47:03.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28133.767845 secs 2023-03-09T22:47:03.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28595.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:47:03.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:03 smithi122 ceph-mon[93869]: pgmap v15516: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 900 KiB/s rd, 983 KiB/s wr, 26 op/s 2023-03-09T22:47:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:03 smithi181 ceph-mon[94189]: pgmap v15516: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 900 KiB/s rd, 983 KiB/s wr, 26 op/s 2023-03-09T22:47:04.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:03 smithi038 ceph-mon[92422]: pgmap v15516: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 900 KiB/s rd, 983 KiB/s wr, 26 op/s 2023-03-09T22:47:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:05 smithi122 ceph-mon[93869]: osdmap e1185: 12 total, 12 up, 12 in 2023-03-09T22:47:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28136.388068 secs 2023-03-09T22:47:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:05 smithi122 ceph-mon[93869]: pgmap v15518: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 700 KiB/s rd, 812 KiB/s wr, 27 op/s 2023-03-09T22:47:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:05 smithi181 ceph-mon[94189]: osdmap e1185: 12 total, 12 up, 12 in 2023-03-09T22:47:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28136.388068 secs 2023-03-09T22:47:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:05 smithi181 ceph-mon[94189]: pgmap v15518: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 700 KiB/s rd, 812 KiB/s wr, 27 op/s 2023-03-09T22:47:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:05 smithi038 ceph-mon[92422]: osdmap e1185: 12 total, 12 up, 12 in 2023-03-09T22:47:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28136.388068 secs 2023-03-09T22:47:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:05 smithi038 ceph-mon[92422]: pgmap v15518: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 700 KiB/s rd, 812 KiB/s wr, 27 op/s 2023-03-09T22:47:07.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28138.767986 secs 2023-03-09T22:47:07.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:07 smithi122 ceph-mon[93869]: pgmap v15519: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 472 KiB/s rd, 608 KiB/s wr, 29 op/s 2023-03-09T22:47:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28138.767986 secs 2023-03-09T22:47:07.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:07 smithi181 ceph-mon[94189]: pgmap v15519: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 472 KiB/s rd, 608 KiB/s wr, 29 op/s 2023-03-09T22:47:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28138.767986 secs 2023-03-09T22:47:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:07 smithi038 ceph-mon[92422]: pgmap v15519: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 472 KiB/s rd, 608 KiB/s wr, 29 op/s 2023-03-09T22:47:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28600.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:47:09.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:47:09.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:47:09.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:09 smithi122 ceph-mon[93869]: pgmap v15520: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 475 KiB/s wr, 29 op/s 2023-03-09T22:47:09.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:47:09.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:47:09.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:09 smithi181 ceph-mon[94189]: pgmap v15520: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 475 KiB/s wr, 29 op/s 2023-03-09T22:47:10.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:47:10.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:47:10.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:09 smithi038 ceph-mon[92422]: pgmap v15520: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 475 KiB/s wr, 29 op/s 2023-03-09T22:47:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28141.388129 secs 2023-03-09T22:47:10.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28141.388129 secs 2023-03-09T22:47:11.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28141.388129 secs 2023-03-09T22:47:11.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:11 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:47:11.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:11 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:47:11.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:11 smithi122 ceph-mon[93869]: pgmap v15521: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 574 KiB/s rd, 615 KiB/s wr, 29 op/s 2023-03-09T22:47:11.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:11 smithi122 ceph-mon[93869]: osdmap e1186: 12 total, 12 up, 12 in 2023-03-09T22:47:11.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:11 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:47:11.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:11 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:47:11.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:11 smithi181 ceph-mon[94189]: pgmap v15521: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 574 KiB/s rd, 615 KiB/s wr, 29 op/s 2023-03-09T22:47:11.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:11 smithi181 ceph-mon[94189]: osdmap e1186: 12 total, 12 up, 12 in 2023-03-09T22:47:12.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:11 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:47:12.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:11 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:47:12.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:11 smithi038 ceph-mon[92422]: pgmap v15521: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 574 KiB/s rd, 615 KiB/s wr, 29 op/s 2023-03-09T22:47:12.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:11 smithi038 ceph-mon[92422]: osdmap e1186: 12 total, 12 up, 12 in 2023-03-09T22:47:12.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28143.768140 secs 2023-03-09T22:47:12.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28143.768140 secs 2023-03-09T22:47:13.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28143.768140 secs 2023-03-09T22:47:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28605.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:47:13.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:13 smithi122 ceph-mon[93869]: pgmap v15523: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 323 KiB/s wr, 14 op/s 2023-03-09T22:47:13.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:13 smithi181 ceph-mon[94189]: pgmap v15523: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 323 KiB/s wr, 14 op/s 2023-03-09T22:47:14.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:13 smithi038 ceph-mon[92422]: pgmap v15523: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 323 KiB/s wr, 14 op/s 2023-03-09T22:47:15.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28146.388189 secs 2023-03-09T22:47:15.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28146.388189 secs 2023-03-09T22:47:15.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28146.388189 secs 2023-03-09T22:47:16.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:16 smithi122 ceph-mon[93869]: pgmap v15524: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 357 KiB/s rd, 286 KiB/s wr, 12 op/s 2023-03-09T22:47:16.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:16 smithi181 ceph-mon[94189]: pgmap v15524: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 357 KiB/s rd, 286 KiB/s wr, 12 op/s 2023-03-09T22:47:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:16 smithi038 ceph-mon[92422]: pgmap v15524: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 357 KiB/s rd, 286 KiB/s wr, 12 op/s 2023-03-09T22:47:17.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28148.768285 secs 2023-03-09T22:47:17.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:17 smithi122 ceph-mon[93869]: pgmap v15525: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 141 KiB/s wr, 8 op/s 2023-03-09T22:47:17.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28148.768285 secs 2023-03-09T22:47:17.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:17 smithi181 ceph-mon[94189]: pgmap v15525: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 141 KiB/s wr, 8 op/s 2023-03-09T22:47:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28148.768285 secs 2023-03-09T22:47:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:17 smithi038 ceph-mon[92422]: pgmap v15525: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 141 KiB/s wr, 8 op/s 2023-03-09T22:47:18.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28610.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:47:19.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:19 smithi122 ceph-mon[93869]: pgmap v15526: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 141 KiB/s wr, 7 op/s 2023-03-09T22:47:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:19 smithi181 ceph-mon[94189]: pgmap v15526: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 141 KiB/s wr, 7 op/s 2023-03-09T22:47:20.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:19 smithi038 ceph-mon[92422]: pgmap v15526: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 141 KiB/s wr, 7 op/s 2023-03-09T22:47:20.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28151.388251 secs 2023-03-09T22:47:20.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28151.388251 secs 2023-03-09T22:47:21.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28151.388251 secs 2023-03-09T22:47:21.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:21 smithi122 ceph-mon[93869]: pgmap v15527: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 99 KiB/s wr, 6 op/s 2023-03-09T22:47:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:21 smithi181 ceph-mon[94189]: pgmap v15527: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 99 KiB/s wr, 6 op/s 2023-03-09T22:47:22.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:21 smithi038 ceph-mon[92422]: pgmap v15527: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 99 KiB/s wr, 6 op/s 2023-03-09T22:47:22.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:22 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:47:22.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28153.768391 secs 2023-03-09T22:47:22.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:22 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:47:22.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:22 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:47:22.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:22 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:47:22.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:22 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:47:22.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28153.768391 secs 2023-03-09T22:47:22.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:22 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:47:22.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:22 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:47:22.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:22 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:47:23.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:22 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:47:23.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28153.768391 secs 2023-03-09T22:47:23.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:22 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:47:23.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:22 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:47:23.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:22 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:47:23.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28615.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:47:23.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:23 smithi122 ceph-mon[93869]: pgmap v15528: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 83 KiB/s wr, 6 op/s 2023-03-09T22:47:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:23 smithi181 ceph-mon[94189]: pgmap v15528: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 83 KiB/s wr, 6 op/s 2023-03-09T22:47:24.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:23 smithi038 ceph-mon[92422]: pgmap v15528: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 83 KiB/s wr, 6 op/s 2023-03-09T22:47:25.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28156.388310 secs 2023-03-09T22:47:25.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28156.388310 secs 2023-03-09T22:47:25.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28156.388310 secs 2023-03-09T22:47:26.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:26 smithi122 ceph-mon[93869]: pgmap v15529: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 82 KiB/s wr, 5 op/s 2023-03-09T22:47:26.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:26 smithi181 ceph-mon[94189]: pgmap v15529: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 82 KiB/s wr, 5 op/s 2023-03-09T22:47:26.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:26 smithi038 ceph-mon[92422]: pgmap v15529: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 82 KiB/s wr, 5 op/s 2023-03-09T22:47:27.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28158.768512 secs 2023-03-09T22:47:27.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:27 smithi122 ceph-mon[93869]: pgmap v15530: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 82 KiB/s wr, 7 op/s 2023-03-09T22:47:27.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28158.768512 secs 2023-03-09T22:47:27.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:27 smithi181 ceph-mon[94189]: pgmap v15530: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 82 KiB/s wr, 7 op/s 2023-03-09T22:47:28.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28158.768512 secs 2023-03-09T22:47:28.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:27 smithi038 ceph-mon[92422]: pgmap v15530: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 82 KiB/s wr, 7 op/s 2023-03-09T22:47:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28620.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:47:29.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:29 smithi122 ceph-mon[93869]: pgmap v15531: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 82 KiB/s wr, 5 op/s 2023-03-09T22:47:29.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:29 smithi181 ceph-mon[94189]: pgmap v15531: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 82 KiB/s wr, 5 op/s 2023-03-09T22:47:30.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:29 smithi038 ceph-mon[92422]: pgmap v15531: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 82 KiB/s wr, 5 op/s 2023-03-09T22:47:30.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28161.388378 secs 2023-03-09T22:47:30.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28161.388378 secs 2023-03-09T22:47:31.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28161.388378 secs 2023-03-09T22:47:31.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:31 smithi122 ceph-mon[93869]: pgmap v15532: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 82 KiB/s wr, 7 op/s 2023-03-09T22:47:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:31 smithi181 ceph-mon[94189]: pgmap v15532: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 82 KiB/s wr, 7 op/s 2023-03-09T22:47:32.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:31 smithi038 ceph-mon[92422]: pgmap v15532: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 82 KiB/s wr, 7 op/s 2023-03-09T22:47:32.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28163.768631 secs 2023-03-09T22:47:32.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28163.768631 secs 2023-03-09T22:47:33.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28163.768631 secs 2023-03-09T22:47:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28625.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:47:33.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:33 smithi122 ceph-mon[93869]: pgmap v15533: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-09T22:47:33.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:33 smithi181 ceph-mon[94189]: pgmap v15533: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-09T22:47:34.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:33 smithi038 ceph-mon[92422]: pgmap v15533: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-09T22:47:35.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28166.388438 secs 2023-03-09T22:47:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28166.388438 secs 2023-03-09T22:47:35.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28166.388438 secs 2023-03-09T22:47:36.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:36 smithi122 ceph-mon[93869]: pgmap v15534: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-09T22:47:36.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:36 smithi181 ceph-mon[94189]: pgmap v15534: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-09T22:47:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:36 smithi038 ceph-mon[92422]: pgmap v15534: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-09T22:47:37.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28168.768770 secs 2023-03-09T22:47:37.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:37 smithi122 ceph-mon[93869]: pgmap v15535: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 8 op/s 2023-03-09T22:47:37.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28168.768770 secs 2023-03-09T22:47:37.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:37 smithi181 ceph-mon[94189]: pgmap v15535: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 8 op/s 2023-03-09T22:47:38.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28168.768770 secs 2023-03-09T22:47:38.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:37 smithi038 ceph-mon[92422]: pgmap v15535: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 8 op/s 2023-03-09T22:47:38.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28630.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:47:39.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:39 smithi122 ceph-mon[93869]: pgmap v15536: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 682 B/s wr, 6 op/s 2023-03-09T22:47:39.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:39 smithi181 ceph-mon[94189]: pgmap v15536: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 682 B/s wr, 6 op/s 2023-03-09T22:47:40.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:39 smithi038 ceph-mon[92422]: pgmap v15536: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 682 B/s wr, 6 op/s 2023-03-09T22:47:40.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28171.388521 secs 2023-03-09T22:47:40.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28171.388521 secs 2023-03-09T22:47:41.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28171.388521 secs 2023-03-09T22:47:41.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:41 smithi122 ceph-mon[93869]: pgmap v15537: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 121 KiB/s wr, 8 op/s 2023-03-09T22:47:41.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:41 smithi181 ceph-mon[94189]: pgmap v15537: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 121 KiB/s wr, 8 op/s 2023-03-09T22:47:42.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:41 smithi038 ceph-mon[92422]: pgmap v15537: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 121 KiB/s wr, 8 op/s 2023-03-09T22:47:43.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28635.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:47:43.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:43 smithi122 ceph-mon[93869]: pgmap v15538: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 121 KiB/s wr, 7 op/s 2023-03-09T22:47:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:43 smithi181 ceph-mon[94189]: pgmap v15538: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 121 KiB/s wr, 7 op/s 2023-03-09T22:47:44.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:43 smithi038 ceph-mon[92422]: pgmap v15538: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 121 KiB/s wr, 7 op/s 2023-03-09T22:47:45.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28176.388580 secs 2023-03-09T22:47:45.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:45 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:47:45.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28176.388580 secs 2023-03-09T22:47:45.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:45 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:47:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28176.388580 secs 2023-03-09T22:47:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:45 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:47:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:46 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:47:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:46 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:47:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:46 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:47:46.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:46 smithi122 ceph-mon[93869]: pgmap v15539: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 254 KiB/s wr, 6 op/s 2023-03-09T22:47:46.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:46 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:47:46.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:46 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:47:46.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:46 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:47:46.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:46 smithi181 ceph-mon[94189]: pgmap v15539: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 254 KiB/s wr, 6 op/s 2023-03-09T22:47:46.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:46 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:47:46.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:46 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:47:46.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:46 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:47:46.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:46 smithi038 ceph-mon[92422]: pgmap v15539: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 254 KiB/s wr, 6 op/s 2023-03-09T22:47:48.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28640.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:47:48.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:48 smithi122 ceph-mon[93869]: pgmap v15540: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 254 KiB/s wr, 9 op/s 2023-03-09T22:47:48.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:48 smithi181 ceph-mon[94189]: pgmap v15540: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 254 KiB/s wr, 9 op/s 2023-03-09T22:47:49.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:48 smithi038 ceph-mon[92422]: pgmap v15540: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 254 KiB/s wr, 9 op/s 2023-03-09T22:47:49.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:47:49.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:47:49.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:47:49.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:47:50.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:47:50.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:47:50.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:50 smithi122 ceph-mon[93869]: pgmap v15541: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 253 KiB/s wr, 6 op/s 2023-03-09T22:47:50.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28181.388639 secs 2023-03-09T22:47:50.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:50 smithi181 ceph-mon[94189]: pgmap v15541: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 253 KiB/s wr, 6 op/s 2023-03-09T22:47:50.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28181.388639 secs 2023-03-09T22:47:51.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:50 smithi038 ceph-mon[92422]: pgmap v15541: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 253 KiB/s wr, 6 op/s 2023-03-09T22:47:51.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28181.388639 secs 2023-03-09T22:47:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:47:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:47:51.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:47:51.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:47:52.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:47:52.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:47:52.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:52 smithi122 ceph-mon[93869]: pgmap v15542: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 407 KiB/s wr, 9 op/s 2023-03-09T22:47:52.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:52 smithi181 ceph-mon[94189]: pgmap v15542: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 407 KiB/s wr, 9 op/s 2023-03-09T22:47:53.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:52 smithi038 ceph-mon[92422]: pgmap v15542: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 407 KiB/s wr, 9 op/s 2023-03-09T22:47:53.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28645.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:47:54.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:54 smithi122 ceph-mon[93869]: pgmap v15543: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 405 KiB/s wr, 8 op/s 2023-03-09T22:47:54.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:54 smithi181 ceph-mon[94189]: pgmap v15543: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 405 KiB/s wr, 8 op/s 2023-03-09T22:47:55.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:54 smithi038 ceph-mon[92422]: pgmap v15543: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 405 KiB/s wr, 8 op/s 2023-03-09T22:47:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:47:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:47:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28186.388721 secs 2023-03-09T22:47:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:47:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:47:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28186.388721 secs 2023-03-09T22:47:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:47:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:47:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28186.388721 secs 2023-03-09T22:47:56.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:56 smithi122 ceph-mon[93869]: pgmap v15544: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 405 KiB/s wr, 8 op/s 2023-03-09T22:47:56.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:56 smithi181 ceph-mon[94189]: pgmap v15544: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 405 KiB/s wr, 8 op/s 2023-03-09T22:47:57.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:56 smithi038 ceph-mon[92422]: pgmap v15544: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 405 KiB/s wr, 8 op/s 2023-03-09T22:47:57.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:47:57.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:47:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:47:57.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:47:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:47:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:47:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28650.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:47:58.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:58 smithi122 ceph-mon[93869]: pgmap v15545: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 719 KiB/s rd, 716 KiB/s wr, 13 op/s 2023-03-09T22:47:58.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:58 smithi181 ceph-mon[94189]: pgmap v15545: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 719 KiB/s rd, 716 KiB/s wr, 13 op/s 2023-03-09T22:47:59.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:58 smithi038 ceph-mon[92422]: pgmap v15545: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 719 KiB/s rd, 716 KiB/s wr, 13 op/s 2023-03-09T22:47:59.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:47:59.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:47:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:47:59.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:47:59.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:47:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:48:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:48:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:47:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:48:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:00 smithi122 ceph-mon[93869]: pgmap v15546: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 729 KiB/s rd, 716 KiB/s wr, 10 op/s 2023-03-09T22:48:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28191.388781 secs 2023-03-09T22:48:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:00 smithi181 ceph-mon[94189]: pgmap v15546: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 729 KiB/s rd, 716 KiB/s wr, 10 op/s 2023-03-09T22:48:00.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28191.388781 secs 2023-03-09T22:48:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:00 smithi038 ceph-mon[92422]: pgmap v15546: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 729 KiB/s rd, 716 KiB/s wr, 10 op/s 2023-03-09T22:48:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28191.388781 secs 2023-03-09T22:48:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:48:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:48:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:48:01.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:48:02.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:48:02.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:48:02.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:02 smithi122 ceph-mon[93869]: pgmap v15547: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 17 op/s 2023-03-09T22:48:02.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28193.769412 secs 2023-03-09T22:48:02.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:02 smithi181 ceph-mon[94189]: pgmap v15547: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 17 op/s 2023-03-09T22:48:02.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28193.769412 secs 2023-03-09T22:48:03.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:02 smithi038 ceph-mon[92422]: pgmap v15547: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 17 op/s 2023-03-09T22:48:03.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28193.769412 secs 2023-03-09T22:48:03.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28655.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:48:04.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:04 smithi122 ceph-mon[93869]: pgmap v15548: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 935 KiB/s rd, 1.0 MiB/s wr, 28 op/s 2023-03-09T22:48:04.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:04 smithi181 ceph-mon[94189]: pgmap v15548: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 935 KiB/s rd, 1.0 MiB/s wr, 28 op/s 2023-03-09T22:48:05.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:04 smithi038 ceph-mon[92422]: pgmap v15548: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 935 KiB/s rd, 1.0 MiB/s wr, 28 op/s 2023-03-09T22:48:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28196.388840 secs 2023-03-09T22:48:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28196.388840 secs 2023-03-09T22:48:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28196.388840 secs 2023-03-09T22:48:07.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:06 smithi038 ceph-mon[92422]: pgmap v15549: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 892 KiB/s rd, 915 KiB/s wr, 27 op/s 2023-03-09T22:48:07.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:06 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:48:07.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:06 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:48:07.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:48:07.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:06 smithi122 ceph-mon[93869]: pgmap v15549: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 892 KiB/s rd, 915 KiB/s wr, 27 op/s 2023-03-09T22:48:07.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:06 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:48:07.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:06 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:48:07.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:48:07.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:06 smithi181 ceph-mon[94189]: pgmap v15549: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 892 KiB/s rd, 915 KiB/s wr, 27 op/s 2023-03-09T22:48:07.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:06 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:48:07.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:06 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:48:07.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:48:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:48:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:48:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28198.769534 secs 2023-03-09T22:48:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:48:08.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:48:08.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28198.769534 secs 2023-03-09T22:48:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:48:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:48:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28198.769534 secs 2023-03-09T22:48:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28660.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:48:09.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:08 smithi122 ceph-mon[93869]: pgmap v15550: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 915 KiB/s rd, 937 KiB/s wr, 31 op/s 2023-03-09T22:48:09.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:08 smithi181 ceph-mon[94189]: pgmap v15550: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 915 KiB/s rd, 937 KiB/s wr, 31 op/s 2023-03-09T22:48:09.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:08 smithi038 ceph-mon[92422]: pgmap v15550: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 915 KiB/s rd, 937 KiB/s wr, 31 op/s 2023-03-09T22:48:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:10 smithi122 ceph-mon[93869]: pgmap v15551: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 470 KiB/s rd, 493 KiB/s wr, 26 op/s 2023-03-09T22:48:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:10 smithi122 ceph-mon[93869]: osdmap e1187: 12 total, 12 up, 12 in 2023-03-09T22:48:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28201.388902 secs 2023-03-09T22:48:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:10 smithi181 ceph-mon[94189]: pgmap v15551: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 470 KiB/s rd, 493 KiB/s wr, 26 op/s 2023-03-09T22:48:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:10 smithi181 ceph-mon[94189]: osdmap e1187: 12 total, 12 up, 12 in 2023-03-09T22:48:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28201.388902 secs 2023-03-09T22:48:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:10 smithi038 ceph-mon[92422]: pgmap v15551: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 470 KiB/s rd, 493 KiB/s wr, 26 op/s 2023-03-09T22:48:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:10 smithi038 ceph-mon[92422]: osdmap e1187: 12 total, 12 up, 12 in 2023-03-09T22:48:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28201.388902 secs 2023-03-09T22:48:12.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:11 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:48:12.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:11 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:48:12.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:11 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:48:12.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:11 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:48:12.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:11 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:48:12.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:11 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:48:12.788 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:12 smithi122 ceph-mon[93869]: pgmap v15553: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 413 KiB/s wr, 27 op/s 2023-03-09T22:48:12.789 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28203.769638 secs 2023-03-09T22:48:12.789 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:48:12.790 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:48:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:12 smithi181 ceph-mon[94189]: pgmap v15553: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 413 KiB/s wr, 27 op/s 2023-03-09T22:48:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28203.769638 secs 2023-03-09T22:48:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:48:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:48:13.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:12 smithi038 ceph-mon[92422]: pgmap v15553: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 413 KiB/s wr, 27 op/s 2023-03-09T22:48:13.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28203.769638 secs 2023-03-09T22:48:13.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:48:13.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:48:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28665.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:48:15.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:14 smithi122 ceph-mon[93869]: pgmap v15554: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 357 KiB/s wr, 11 op/s 2023-03-09T22:48:15.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:14 smithi181 ceph-mon[94189]: pgmap v15554: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 357 KiB/s wr, 11 op/s 2023-03-09T22:48:15.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:14 smithi038 ceph-mon[92422]: pgmap v15554: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 357 KiB/s wr, 11 op/s 2023-03-09T22:48:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28206.388984 secs 2023-03-09T22:48:16.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28206.388984 secs 2023-03-09T22:48:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28206.388984 secs 2023-03-09T22:48:17.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:16 smithi122 ceph-mon[93869]: pgmap v15555: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 457 KiB/s rd, 455 KiB/s wr, 11 op/s 2023-03-09T22:48:17.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:16 smithi122 ceph-mon[93869]: osdmap e1188: 12 total, 12 up, 12 in 2023-03-09T22:48:17.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:16 smithi181 ceph-mon[94189]: pgmap v15555: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 457 KiB/s rd, 455 KiB/s wr, 11 op/s 2023-03-09T22:48:17.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:16 smithi181 ceph-mon[94189]: osdmap e1188: 12 total, 12 up, 12 in 2023-03-09T22:48:17.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:16 smithi038 ceph-mon[92422]: pgmap v15555: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 457 KiB/s rd, 455 KiB/s wr, 11 op/s 2023-03-09T22:48:17.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:16 smithi038 ceph-mon[92422]: osdmap e1188: 12 total, 12 up, 12 in 2023-03-09T22:48:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28208.769756 secs 2023-03-09T22:48:18.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28208.769756 secs 2023-03-09T22:48:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28208.769756 secs 2023-03-09T22:48:18.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28670.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:48:19.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:18 smithi122 ceph-mon[93869]: pgmap v15557: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 540 KiB/s rd, 537 KiB/s wr, 11 op/s 2023-03-09T22:48:19.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:18 smithi181 ceph-mon[94189]: pgmap v15557: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 540 KiB/s rd, 537 KiB/s wr, 11 op/s 2023-03-09T22:48:19.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:18 smithi038 ceph-mon[92422]: pgmap v15557: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 540 KiB/s rd, 537 KiB/s wr, 11 op/s 2023-03-09T22:48:21.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:20 smithi122 ceph-mon[93869]: pgmap v15558: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 443 KiB/s rd, 441 KiB/s wr, 9 op/s 2023-03-09T22:48:21.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28211.389044 secs 2023-03-09T22:48:21.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:20 smithi181 ceph-mon[94189]: pgmap v15558: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 443 KiB/s rd, 441 KiB/s wr, 9 op/s 2023-03-09T22:48:21.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28211.389044 secs 2023-03-09T22:48:21.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:20 smithi038 ceph-mon[92422]: pgmap v15558: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 443 KiB/s rd, 441 KiB/s wr, 9 op/s 2023-03-09T22:48:21.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28211.389044 secs 2023-03-09T22:48:23.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:22 smithi122 ceph-mon[93869]: pgmap v15559: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 100 KiB/s wr, 7 op/s 2023-03-09T22:48:23.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28213.769905 secs 2023-03-09T22:48:23.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:22 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:48:23.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:22 smithi181 ceph-mon[94189]: pgmap v15559: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 100 KiB/s wr, 7 op/s 2023-03-09T22:48:23.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28213.769905 secs 2023-03-09T22:48:23.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:22 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:48:23.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:22 smithi038 ceph-mon[92422]: pgmap v15559: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 100 KiB/s wr, 7 op/s 2023-03-09T22:48:23.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28213.769905 secs 2023-03-09T22:48:23.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:22 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:48:23.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28675.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:48:24.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:23 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:48:24.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:23 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:48:24.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:23 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:48:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:23 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:48:24.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:23 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:48:24.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:23 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:48:24.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:23 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:48:24.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:23 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:48:24.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:23 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:48:25.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:25 smithi122 ceph-mon[93869]: pgmap v15560: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 99 KiB/s wr, 6 op/s 2023-03-09T22:48:25.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28216.389104 secs 2023-03-09T22:48:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:25 smithi181 ceph-mon[94189]: pgmap v15560: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 99 KiB/s wr, 6 op/s 2023-03-09T22:48:25.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28216.389104 secs 2023-03-09T22:48:25.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:25 smithi038 ceph-mon[92422]: pgmap v15560: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 99 KiB/s wr, 6 op/s 2023-03-09T22:48:25.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28216.389104 secs 2023-03-09T22:48:27.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:27 smithi122 ceph-mon[93869]: pgmap v15561: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 409 B/s wr, 6 op/s 2023-03-09T22:48:27.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:27 smithi181 ceph-mon[94189]: pgmap v15561: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 409 B/s wr, 6 op/s 2023-03-09T22:48:27.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:27 smithi038 ceph-mon[92422]: pgmap v15561: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 409 B/s wr, 6 op/s 2023-03-09T22:48:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28218.770027 secs 2023-03-09T22:48:28.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28218.770027 secs 2023-03-09T22:48:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28218.770027 secs 2023-03-09T22:48:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28680.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:48:29.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:29 smithi122 ceph-mon[93869]: pgmap v15562: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 375 B/s wr, 6 op/s 2023-03-09T22:48:29.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:29 smithi181 ceph-mon[94189]: pgmap v15562: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 375 B/s wr, 6 op/s 2023-03-09T22:48:29.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:29 smithi038 ceph-mon[92422]: pgmap v15562: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 375 B/s wr, 6 op/s 2023-03-09T22:48:30.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28221.389163 secs 2023-03-09T22:48:30.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28221.389163 secs 2023-03-09T22:48:30.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28221.389163 secs 2023-03-09T22:48:31.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:31 smithi122 ceph-mon[93869]: pgmap v15563: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T22:48:31.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:31 smithi181 ceph-mon[94189]: pgmap v15563: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T22:48:31.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:31 smithi038 ceph-mon[92422]: pgmap v15563: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T22:48:33.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:33 smithi122 ceph-mon[93869]: pgmap v15564: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T22:48:33.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28223.770139 secs 2023-03-09T22:48:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:33 smithi181 ceph-mon[94189]: pgmap v15564: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T22:48:33.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28223.770139 secs 2023-03-09T22:48:33.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:33 smithi038 ceph-mon[92422]: pgmap v15564: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T22:48:33.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28223.770139 secs 2023-03-09T22:48:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28685.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:48:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:35 smithi122 ceph-mon[93869]: pgmap v15565: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T22:48:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28226.389222 secs 2023-03-09T22:48:35.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:35 smithi181 ceph-mon[94189]: pgmap v15565: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T22:48:35.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28226.389222 secs 2023-03-09T22:48:35.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:35 smithi038 ceph-mon[92422]: pgmap v15565: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T22:48:35.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28226.389222 secs 2023-03-09T22:48:37.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:37 smithi122 ceph-mon[93869]: pgmap v15566: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T22:48:37.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:37 smithi181 ceph-mon[94189]: pgmap v15566: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T22:48:37.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:37 smithi038 ceph-mon[92422]: pgmap v15566: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T22:48:38.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28228.770249 secs 2023-03-09T22:48:38.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28228.770249 secs 2023-03-09T22:48:38.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28228.770249 secs 2023-03-09T22:48:38.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28690.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:48:39.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:39 smithi122 ceph-mon[93869]: pgmap v15567: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 426 B/s wr, 7 op/s 2023-03-09T22:48:39.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:39 smithi181 ceph-mon[94189]: pgmap v15567: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 426 B/s wr, 7 op/s 2023-03-09T22:48:39.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:39 smithi038 ceph-mon[92422]: pgmap v15567: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 426 B/s wr, 7 op/s 2023-03-09T22:48:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28231.389291 secs 2023-03-09T22:48:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28231.389291 secs 2023-03-09T22:48:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28231.389291 secs 2023-03-09T22:48:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:41 smithi122 ceph-mon[93869]: pgmap v15568: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2023-03-09T22:48:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:41 smithi181 ceph-mon[94189]: pgmap v15568: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2023-03-09T22:48:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:41 smithi038 ceph-mon[92422]: pgmap v15568: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2023-03-09T22:48:43.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:43 smithi122 ceph-mon[93869]: pgmap v15569: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 95 KiB/s wr, 8 op/s 2023-03-09T22:48:43.731 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:43 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 28233.770376 secs 2023-03-09T22:48:43.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:43 smithi181 ceph-mon[94189]: pgmap v15569: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 95 KiB/s wr, 8 op/s 2023-03-09T22:48:43.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:43 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 28233.770376 secs 2023-03-09T22:48:43.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:43 smithi038 ceph-mon[92422]: pgmap v15569: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 95 KiB/s wr, 8 op/s 2023-03-09T22:48:43.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:43 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 28233.770376 secs 2023-03-09T22:48:43.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28695.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:48:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:45 smithi122 ceph-mon[93869]: pgmap v15570: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 121 KiB/s wr, 7 op/s 2023-03-09T22:48:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28236.389350 secs 2023-03-09T22:48:45.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:45 smithi181 ceph-mon[94189]: pgmap v15570: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 121 KiB/s wr, 7 op/s 2023-03-09T22:48:45.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28236.389350 secs 2023-03-09T22:48:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:45 smithi038 ceph-mon[92422]: pgmap v15570: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 121 KiB/s wr, 7 op/s 2023-03-09T22:48:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28236.389350 secs 2023-03-09T22:48:46.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:46 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:48:46.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:46 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:48:46.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:46 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:48:46.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:46 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:48:46.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:46 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:48:46.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:46 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:48:47.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:47 smithi122 ceph-mon[93869]: pgmap v15571: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 121 KiB/s wr, 7 op/s 2023-03-09T22:48:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:47 smithi181 ceph-mon[94189]: pgmap v15571: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 121 KiB/s wr, 7 op/s 2023-03-09T22:48:47.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:47 smithi038 ceph-mon[92422]: pgmap v15571: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 121 KiB/s wr, 7 op/s 2023-03-09T22:48:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:48 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 28238.770498 secs 2023-03-09T22:48:48.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:48 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 28238.770498 secs 2023-03-09T22:48:48.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:48 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 28238.770498 secs 2023-03-09T22:48:48.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28700.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:48:49.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:49 smithi122 ceph-mon[93869]: pgmap v15572: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 254 KiB/s wr, 9 op/s 2023-03-09T22:48:49.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:48:49.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:48:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:49 smithi181 ceph-mon[94189]: pgmap v15572: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 254 KiB/s wr, 9 op/s 2023-03-09T22:48:49.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:48:49.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:48:49.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:49 smithi038 ceph-mon[92422]: pgmap v15572: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 254 KiB/s wr, 9 op/s 2023-03-09T22:48:49.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:48:49.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:48:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28241.389410 secs 2023-03-09T22:48:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28241.389410 secs 2023-03-09T22:48:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28241.389410 secs 2023-03-09T22:48:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:51 smithi122 ceph-mon[93869]: pgmap v15573: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 254 KiB/s wr, 7 op/s 2023-03-09T22:48:51.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:51 smithi181 ceph-mon[94189]: pgmap v15573: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 254 KiB/s wr, 7 op/s 2023-03-09T22:48:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:51 smithi038 ceph-mon[92422]: pgmap v15573: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 254 KiB/s wr, 7 op/s 2023-03-09T22:48:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:48:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:48:52.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:48:52.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:48:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:48:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:48:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:53 smithi122 ceph-mon[93869]: pgmap v15574: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 481 KiB/s rd, 476 KiB/s wr, 10 op/s 2023-03-09T22:48:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28243.770601 secs 2023-03-09T22:48:53.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:48:53.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:48:53.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:53 smithi181 ceph-mon[94189]: pgmap v15574: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 481 KiB/s rd, 476 KiB/s wr, 10 op/s 2023-03-09T22:48:53.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28243.770601 secs 2023-03-09T22:48:53.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:48:53.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:48:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:53 smithi038 ceph-mon[92422]: pgmap v15574: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 481 KiB/s rd, 476 KiB/s wr, 10 op/s 2023-03-09T22:48:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28243.770601 secs 2023-03-09T22:48:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:48:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:48:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28705.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:48:54.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:54 smithi122 ceph-mon[93869]: pgmap v15575: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 612 KiB/s rd, 609 KiB/s wr, 10 op/s 2023-03-09T22:48:54.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:54 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:48:54.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:54 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:48:54.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:54 smithi181 ceph-mon[94189]: pgmap v15575: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 612 KiB/s rd, 609 KiB/s wr, 10 op/s 2023-03-09T22:48:54.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:54 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:48:54.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:54 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:48:55.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:54 smithi038 ceph-mon[92422]: pgmap v15575: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 612 KiB/s rd, 609 KiB/s wr, 10 op/s 2023-03-09T22:48:55.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:54 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:48:55.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:54 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:48:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28246.389473 secs 2023-03-09T22:48:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:48:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:48:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28246.389473 secs 2023-03-09T22:48:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:48:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:48:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28246.389473 secs 2023-03-09T22:48:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:48:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:48:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:57 smithi122 ceph-mon[93869]: pgmap v15576: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 598 KiB/s rd, 583 KiB/s wr, 10 op/s 2023-03-09T22:48:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:48:57.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:48:57.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28248.770717 secs 2023-03-09T22:48:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:57 smithi181 ceph-mon[94189]: pgmap v15576: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 598 KiB/s rd, 583 KiB/s wr, 10 op/s 2023-03-09T22:48:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:48:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:48:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28248.770717 secs 2023-03-09T22:48:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:57 smithi038 ceph-mon[92422]: pgmap v15576: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 598 KiB/s rd, 583 KiB/s wr, 10 op/s 2023-03-09T22:48:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:48:57.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:48:57.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28248.770717 secs 2023-03-09T22:48:58.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28710.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:48:59.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:48:59 smithi122 ceph-mon[93869]: pgmap v15577: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 824 KiB/s rd, 809 KiB/s wr, 15 op/s 2023-03-09T22:48:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:48:59 smithi181 ceph-mon[94189]: pgmap v15577: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 824 KiB/s rd, 809 KiB/s wr, 15 op/s 2023-03-09T22:48:59.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:48:59 smithi038 ceph-mon[92422]: pgmap v15577: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 824 KiB/s rd, 809 KiB/s wr, 15 op/s 2023-03-09T22:49:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28251.389535 secs 2023-03-09T22:49:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28251.389535 secs 2023-03-09T22:49:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28251.389535 secs 2023-03-09T22:49:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:01 smithi122 ceph-mon[93869]: pgmap v15578: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 721 KiB/s rd, 707 KiB/s wr, 12 op/s 2023-03-09T22:49:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:49:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:49:01.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:01 smithi181 ceph-mon[94189]: pgmap v15578: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 721 KiB/s rd, 707 KiB/s wr, 12 op/s 2023-03-09T22:49:01.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:49:01.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:49:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:01 smithi038 ceph-mon[92422]: pgmap v15578: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 721 KiB/s rd, 707 KiB/s wr, 12 op/s 2023-03-09T22:49:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:49:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:49:02.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:02 smithi122 ceph-mon[93869]: pgmap v15579: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 723 KiB/s rd, 730 KiB/s wr, 16 op/s 2023-03-09T22:49:02.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28253.770813 secs 2023-03-09T22:49:02.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:02 smithi181 ceph-mon[94189]: pgmap v15579: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 723 KiB/s rd, 730 KiB/s wr, 16 op/s 2023-03-09T22:49:02.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28253.770813 secs 2023-03-09T22:49:03.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:02 smithi038 ceph-mon[92422]: pgmap v15579: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 723 KiB/s rd, 730 KiB/s wr, 16 op/s 2023-03-09T22:49:03.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28253.770813 secs 2023-03-09T22:49:03.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28715.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:49:04.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:04 smithi122 ceph-mon[93869]: pgmap v15580: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 963 KiB/s rd, 987 KiB/s wr, 29 op/s 2023-03-09T22:49:04.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:04 smithi181 ceph-mon[94189]: pgmap v15580: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 963 KiB/s rd, 987 KiB/s wr, 29 op/s 2023-03-09T22:49:05.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:04 smithi038 ceph-mon[92422]: pgmap v15580: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 963 KiB/s rd, 987 KiB/s wr, 29 op/s 2023-03-09T22:49:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28256.389596 secs 2023-03-09T22:49:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28256.389596 secs 2023-03-09T22:49:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28256.389596 secs 2023-03-09T22:49:06.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:06 smithi122 ceph-mon[93869]: pgmap v15581: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 735 KiB/s rd, 759 KiB/s wr, 26 op/s 2023-03-09T22:49:06.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:06 smithi181 ceph-mon[94189]: pgmap v15581: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 735 KiB/s rd, 759 KiB/s wr, 26 op/s 2023-03-09T22:49:07.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:06 smithi038 ceph-mon[92422]: pgmap v15581: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 735 KiB/s rd, 759 KiB/s wr, 26 op/s 2023-03-09T22:49:07.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:49:07.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:49:07.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28258.770931 secs 2023-03-09T22:49:07.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:49:07.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:49:07.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28258.770931 secs 2023-03-09T22:49:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:49:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:49:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28258.770931 secs 2023-03-09T22:49:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28720.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:49:08.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:08 smithi122 ceph-mon[93869]: pgmap v15582: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 725 KiB/s rd, 759 KiB/s wr, 29 op/s 2023-03-09T22:49:08.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:08 smithi181 ceph-mon[94189]: pgmap v15582: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 725 KiB/s rd, 759 KiB/s wr, 29 op/s 2023-03-09T22:49:09.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:08 smithi038 ceph-mon[92422]: pgmap v15582: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 725 KiB/s rd, 759 KiB/s wr, 29 op/s 2023-03-09T22:49:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:10 smithi122 ceph-mon[93869]: pgmap v15583: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 658 KiB/s rd, 692 KiB/s wr, 25 op/s 2023-03-09T22:49:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:10 smithi122 ceph-mon[93869]: osdmap e1189: 12 total, 12 up, 12 in 2023-03-09T22:49:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28261.389659 secs 2023-03-09T22:49:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:10 smithi181 ceph-mon[94189]: pgmap v15583: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 658 KiB/s rd, 692 KiB/s wr, 25 op/s 2023-03-09T22:49:10.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:10 smithi181 ceph-mon[94189]: osdmap e1189: 12 total, 12 up, 12 in 2023-03-09T22:49:10.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28261.389659 secs 2023-03-09T22:49:11.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:10 smithi038 ceph-mon[92422]: pgmap v15583: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 658 KiB/s rd, 692 KiB/s wr, 25 op/s 2023-03-09T22:49:11.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:10 smithi038 ceph-mon[92422]: osdmap e1189: 12 total, 12 up, 12 in 2023-03-09T22:49:11.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28261.389659 secs 2023-03-09T22:49:12.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:12 smithi122 ceph-mon[93869]: pgmap v15585: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 751 KiB/s rd, 766 KiB/s wr, 27 op/s 2023-03-09T22:49:12.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28263.771045 secs 2023-03-09T22:49:12.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:12 smithi181 ceph-mon[94189]: pgmap v15585: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 751 KiB/s rd, 766 KiB/s wr, 27 op/s 2023-03-09T22:49:12.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28263.771045 secs 2023-03-09T22:49:13.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:12 smithi038 ceph-mon[92422]: pgmap v15585: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 751 KiB/s rd, 766 KiB/s wr, 27 op/s 2023-03-09T22:49:13.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28263.771045 secs 2023-03-09T22:49:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28725.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:49:14.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:14 smithi122 ceph-mon[93869]: pgmap v15586: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 313 KiB/s rd, 310 KiB/s wr, 9 op/s 2023-03-09T22:49:14.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:14 smithi181 ceph-mon[94189]: pgmap v15586: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 313 KiB/s rd, 310 KiB/s wr, 9 op/s 2023-03-09T22:49:15.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:14 smithi038 ceph-mon[92422]: pgmap v15586: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 313 KiB/s rd, 310 KiB/s wr, 9 op/s 2023-03-09T22:49:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:15 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:49:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:15 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:49:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28266.389717 secs 2023-03-09T22:49:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:15 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:49:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:15 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:49:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28266.389717 secs 2023-03-09T22:49:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:15 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:49:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:15 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:49:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28266.389717 secs 2023-03-09T22:49:16.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:16 smithi122 ceph-mon[93869]: pgmap v15587: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 313 KiB/s rd, 310 KiB/s wr, 9 op/s 2023-03-09T22:49:16.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:16 smithi122 ceph-mon[93869]: osdmap e1190: 12 total, 12 up, 12 in 2023-03-09T22:49:16.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:16 smithi181 ceph-mon[94189]: pgmap v15587: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 313 KiB/s rd, 310 KiB/s wr, 9 op/s 2023-03-09T22:49:16.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:16 smithi181 ceph-mon[94189]: osdmap e1190: 12 total, 12 up, 12 in 2023-03-09T22:49:17.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:16 smithi038 ceph-mon[92422]: pgmap v15587: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 313 KiB/s rd, 310 KiB/s wr, 9 op/s 2023-03-09T22:49:17.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:16 smithi038 ceph-mon[92422]: osdmap e1190: 12 total, 12 up, 12 in 2023-03-09T22:49:17.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:49:17.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:49:17.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28268.771172 secs 2023-03-09T22:49:17.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:49:17.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:49:17.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28268.771172 secs 2023-03-09T22:49:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:49:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:49:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28268.771172 secs 2023-03-09T22:49:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28730.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:49:18.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:18 smithi122 ceph-mon[93869]: pgmap v15589: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 150 KiB/s wr, 8 op/s 2023-03-09T22:49:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:18 smithi181 ceph-mon[94189]: pgmap v15589: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 150 KiB/s wr, 8 op/s 2023-03-09T22:49:19.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:18 smithi038 ceph-mon[92422]: pgmap v15589: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 150 KiB/s wr, 8 op/s 2023-03-09T22:49:20.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:20 smithi122 ceph-mon[93869]: pgmap v15590: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 222 KiB/s wr, 7 op/s 2023-03-09T22:49:20.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28271.389779 secs 2023-03-09T22:49:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:20 smithi181 ceph-mon[94189]: pgmap v15590: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 222 KiB/s wr, 7 op/s 2023-03-09T22:49:20.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28271.389779 secs 2023-03-09T22:49:21.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:20 smithi038 ceph-mon[92422]: pgmap v15590: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 222 KiB/s wr, 7 op/s 2023-03-09T22:49:21.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28271.389779 secs 2023-03-09T22:49:22.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:22 smithi122 ceph-mon[93869]: pgmap v15591: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 219 KiB/s wr, 7 op/s 2023-03-09T22:49:22.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28273.771293 secs 2023-03-09T22:49:22.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:22 smithi181 ceph-mon[94189]: pgmap v15591: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 219 KiB/s wr, 7 op/s 2023-03-09T22:49:22.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28273.771293 secs 2023-03-09T22:49:23.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:22 smithi038 ceph-mon[92422]: pgmap v15591: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 219 KiB/s wr, 7 op/s 2023-03-09T22:49:23.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28273.771293 secs 2023-03-09T22:49:23.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28735.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:49:23.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:23 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:49:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:23 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:49:24.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:23 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:49:24.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:24 smithi122 ceph-mon[93869]: pgmap v15592: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 100 KiB/s wr, 6 op/s 2023-03-09T22:49:24.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:24 smithi181 ceph-mon[94189]: pgmap v15592: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 100 KiB/s wr, 6 op/s 2023-03-09T22:49:25.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:24 smithi038 ceph-mon[92422]: pgmap v15592: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 100 KiB/s wr, 6 op/s 2023-03-09T22:49:25.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28276.389839 secs 2023-03-09T22:49:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28276.389839 secs 2023-03-09T22:49:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28276.389839 secs 2023-03-09T22:49:27.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:27 smithi122 ceph-mon[93869]: pgmap v15593: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 100 KiB/s wr, 6 op/s 2023-03-09T22:49:27.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:49:27.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:49:27.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:49:27.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:49:27.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:49:27.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:49:27.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:49:27.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:49:27.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:49:27.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd='[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]': finished 2023-03-09T22:49:27.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:49:27.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:49:27.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:49:27.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:49:27.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:49:27.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:49:27.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:49:27.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:49:27.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:49:27.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:27 smithi181 ceph-mon[94189]: pgmap v15593: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 100 KiB/s wr, 6 op/s 2023-03-09T22:49:27.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:49:27.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:49:27.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:49:27.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:49:27.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:49:27.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:49:27.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:49:27.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:49:27.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:49:27.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd='[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]': finished 2023-03-09T22:49:27.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:49:27.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:49:27.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:49:27.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:49:27.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:49:27.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:49:27.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:49:27.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:49:27.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:49:27.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:27 smithi038 ceph-mon[92422]: pgmap v15593: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 100 KiB/s wr, 6 op/s 2023-03-09T22:49:27.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:49:27.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:49:27.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:49:27.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:49:27.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:49:27.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:49:27.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:49:27.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:49:27.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:49:27.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd='[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]': finished 2023-03-09T22:49:27.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:49:27.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:49:27.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:49:27.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:49:27.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:49:27.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:49:27.503 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:49:27.503 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:49:27.503 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:49:28.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:28 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T22:49:28.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:28 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T22:49:28.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:28 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T22:49:28.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:28 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T22:49:28.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28278.771408 secs 2023-03-09T22:49:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:28 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T22:49:28.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:28 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T22:49:28.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:28 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T22:49:28.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:28 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T22:49:28.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28278.771408 secs 2023-03-09T22:49:28.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:28 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T22:49:28.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:28 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T22:49:28.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:28 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T22:49:28.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:28 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T22:49:28.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28278.771408 secs 2023-03-09T22:49:28.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28740.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:49:29.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:29 smithi122 ceph-mon[93869]: pgmap v15594: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 91 KiB/s wr, 6 op/s 2023-03-09T22:49:29.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:29 smithi181 ceph-mon[94189]: pgmap v15594: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 91 KiB/s wr, 6 op/s 2023-03-09T22:49:29.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:29 smithi038 ceph-mon[92422]: pgmap v15594: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 91 KiB/s wr, 6 op/s 2023-03-09T22:49:30.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28281.389900 secs 2023-03-09T22:49:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28281.389900 secs 2023-03-09T22:49:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28281.389900 secs 2023-03-09T22:49:31.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:31 smithi122 ceph-mon[93869]: pgmap v15595: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 83 KiB/s wr, 5 op/s 2023-03-09T22:49:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:31 smithi181 ceph-mon[94189]: pgmap v15595: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 83 KiB/s wr, 5 op/s 2023-03-09T22:49:31.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:31 smithi038 ceph-mon[92422]: pgmap v15595: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 83 KiB/s wr, 5 op/s 2023-03-09T22:49:33.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:33 smithi122 ceph-mon[93869]: pgmap v15596: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 6 op/s 2023-03-09T22:49:33.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28283.771519 secs 2023-03-09T22:49:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:33 smithi181 ceph-mon[94189]: pgmap v15596: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 6 op/s 2023-03-09T22:49:33.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28283.771519 secs 2023-03-09T22:49:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:33 smithi038 ceph-mon[92422]: pgmap v15596: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 6 op/s 2023-03-09T22:49:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28283.771519 secs 2023-03-09T22:49:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28745.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:49:35.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:35 smithi122 ceph-mon[93869]: pgmap v15597: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 159 KiB/s wr, 7 op/s 2023-03-09T22:49:35.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28286.389960 secs 2023-03-09T22:49:35.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:35 smithi181 ceph-mon[94189]: pgmap v15597: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 159 KiB/s wr, 7 op/s 2023-03-09T22:49:35.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28286.389960 secs 2023-03-09T22:49:35.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:35 smithi038 ceph-mon[92422]: pgmap v15597: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 159 KiB/s wr, 7 op/s 2023-03-09T22:49:35.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28286.389960 secs 2023-03-09T22:49:37.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:37 smithi122 ceph-mon[93869]: pgmap v15598: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 221 KiB/s wr, 6 op/s 2023-03-09T22:49:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:37 smithi181 ceph-mon[94189]: pgmap v15598: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 221 KiB/s wr, 6 op/s 2023-03-09T22:49:37.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:37 smithi038 ceph-mon[92422]: pgmap v15598: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 221 KiB/s wr, 6 op/s 2023-03-09T22:49:38.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28288.771613 secs 2023-03-09T22:49:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28288.771613 secs 2023-03-09T22:49:38.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28288.771613 secs 2023-03-09T22:49:38.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28750.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:49:39.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:39 smithi122 ceph-mon[93869]: pgmap v15599: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 221 KiB/s wr, 8 op/s 2023-03-09T22:49:39.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:39 smithi181 ceph-mon[94189]: pgmap v15599: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 221 KiB/s wr, 8 op/s 2023-03-09T22:49:39.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:39 smithi038 ceph-mon[92422]: pgmap v15599: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 221 KiB/s wr, 8 op/s 2023-03-09T22:49:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28291.390002 secs 2023-03-09T22:49:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28291.390002 secs 2023-03-09T22:49:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28291.390002 secs 2023-03-09T22:49:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:41 smithi122 ceph-mon[93869]: pgmap v15600: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 221 KiB/s wr, 7 op/s 2023-03-09T22:49:41.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:41 smithi181 ceph-mon[94189]: pgmap v15600: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 221 KiB/s wr, 7 op/s 2023-03-09T22:49:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:41 smithi038 ceph-mon[92422]: pgmap v15600: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 221 KiB/s wr, 7 op/s 2023-03-09T22:49:43.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:43 smithi122 ceph-mon[93869]: pgmap v15601: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 221 KiB/s wr, 8 op/s 2023-03-09T22:49:43.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28293.771749 secs 2023-03-09T22:49:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:43 smithi181 ceph-mon[94189]: pgmap v15601: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 221 KiB/s wr, 8 op/s 2023-03-09T22:49:43.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28293.771749 secs 2023-03-09T22:49:43.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:43 smithi038 ceph-mon[92422]: pgmap v15601: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 221 KiB/s wr, 8 op/s 2023-03-09T22:49:43.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28293.771749 secs 2023-03-09T22:49:43.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28755.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:49:45.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:45 smithi122 ceph-mon[93869]: pgmap v15602: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 221 KiB/s wr, 8 op/s 2023-03-09T22:49:45.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28296.390088 secs 2023-03-09T22:49:45.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:45 smithi181 ceph-mon[94189]: pgmap v15602: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 221 KiB/s wr, 8 op/s 2023-03-09T22:49:45.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28296.390088 secs 2023-03-09T22:49:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:45 smithi038 ceph-mon[92422]: pgmap v15602: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 221 KiB/s wr, 8 op/s 2023-03-09T22:49:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28296.390088 secs 2023-03-09T22:49:47.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:47 smithi038 ceph-mon[92422]: pgmap v15603: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 196 KiB/s wr, 7 op/s 2023-03-09T22:49:47.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:47 smithi122 ceph-mon[93869]: pgmap v15603: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 196 KiB/s wr, 7 op/s 2023-03-09T22:49:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:47 smithi181 ceph-mon[94189]: pgmap v15603: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 196 KiB/s wr, 7 op/s 2023-03-09T22:49:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28298.771875 secs 2023-03-09T22:49:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28298.771875 secs 2023-03-09T22:49:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28298.771875 secs 2023-03-09T22:49:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28760.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:49:49.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:49 smithi038 ceph-mon[92422]: pgmap v15604: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 135 KiB/s wr, 9 op/s 2023-03-09T22:49:49.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:49:49.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:49:49.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:49 smithi122 ceph-mon[93869]: pgmap v15604: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 135 KiB/s wr, 9 op/s 2023-03-09T22:49:49.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:49:49.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:49:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:49 smithi181 ceph-mon[94189]: pgmap v15604: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 135 KiB/s wr, 9 op/s 2023-03-09T22:49:49.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:49:49.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:49:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28301.390146 secs 2023-03-09T22:49:50.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28301.390146 secs 2023-03-09T22:49:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28301.390146 secs 2023-03-09T22:49:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:51 smithi122 ceph-mon[93869]: pgmap v15605: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 135 KiB/s wr, 7 op/s 2023-03-09T22:49:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:51 smithi181 ceph-mon[94189]: pgmap v15605: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 135 KiB/s wr, 7 op/s 2023-03-09T22:49:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:51 smithi038 ceph-mon[92422]: pgmap v15605: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 135 KiB/s wr, 7 op/s 2023-03-09T22:49:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:49:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:49:52.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:49:52.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:49:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:49:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:49:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:53 smithi122 ceph-mon[93869]: pgmap v15606: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 545 KiB/s rd, 540 KiB/s wr, 10 op/s 2023-03-09T22:49:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:49:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:49:53.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:53 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:49:53.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:53 smithi181 ceph-mon[94189]: pgmap v15606: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 545 KiB/s rd, 540 KiB/s wr, 10 op/s 2023-03-09T22:49:53.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:49:53.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:49:53.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:53 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:49:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:53 smithi038 ceph-mon[92422]: pgmap v15606: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 545 KiB/s rd, 540 KiB/s wr, 10 op/s 2023-03-09T22:49:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:49:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:49:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:53 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:49:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28765.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:49:54.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:54 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:49:54.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:54 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:49:54.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:54 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:49:54.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:54 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:49:54.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:54 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:49:54.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:54 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:49:54.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:54 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:49:54.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:54 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:49:54.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:54 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:49:55.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:55 smithi122 ceph-mon[93869]: pgmap v15607: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 589 KiB/s rd, 585 KiB/s wr, 9 op/s 2023-03-09T22:49:55.777 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28306.390205 secs 2023-03-09T22:49:55.779 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:55 smithi038 ceph-mon[92422]: pgmap v15607: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 589 KiB/s rd, 585 KiB/s wr, 9 op/s 2023-03-09T22:49:55.779 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28306.390205 secs 2023-03-09T22:49:55.779 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:55 smithi181 ceph-mon[94189]: pgmap v15607: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 589 KiB/s rd, 585 KiB/s wr, 9 op/s 2023-03-09T22:49:55.779 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28306.390205 secs 2023-03-09T22:49:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:49:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:49:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:49:56.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:49:56.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:49:56.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:49:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:57 smithi122 ceph-mon[93869]: pgmap v15608: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 802 KiB/s rd, 788 KiB/s wr, 10 op/s 2023-03-09T22:49:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:49:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:49:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:57 smithi181 ceph-mon[94189]: pgmap v15608: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 802 KiB/s rd, 788 KiB/s wr, 10 op/s 2023-03-09T22:49:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:49:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:49:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:57 smithi038 ceph-mon[92422]: pgmap v15608: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 802 KiB/s rd, 788 KiB/s wr, 10 op/s 2023-03-09T22:49:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:49:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:49:58.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28770.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:49:59.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:49:59 smithi122 ceph-mon[93869]: pgmap v15609: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 692 KiB/s rd, 678 KiB/s wr, 13 op/s 2023-03-09T22:49:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:49:59 smithi181 ceph-mon[94189]: pgmap v15609: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 692 KiB/s rd, 678 KiB/s wr, 13 op/s 2023-03-09T22:49:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:49:59 smithi038 ceph-mon[92422]: pgmap v15609: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 692 KiB/s rd, 678 KiB/s wr, 13 op/s 2023-03-09T22:50:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:50:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:50:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28311.390277 secs 2023-03-09T22:50:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:00 smithi122 ceph-mon[93869]: overall HEALTH_WARN 1 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-09T22:50:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:50:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:50:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28311.390277 secs 2023-03-09T22:50:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:00 smithi181 ceph-mon[94189]: overall HEALTH_WARN 1 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-09T22:50:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:50:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:50:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28311.390277 secs 2023-03-09T22:50:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:00 smithi038 ceph-mon[92422]: overall HEALTH_WARN 1 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-09T22:50:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:01 smithi122 ceph-mon[93869]: pgmap v15610: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 691 KiB/s rd, 765 KiB/s wr, 11 op/s 2023-03-09T22:50:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:50:01.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:50:01.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:01 smithi181 ceph-mon[94189]: pgmap v15610: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 691 KiB/s rd, 765 KiB/s wr, 11 op/s 2023-03-09T22:50:01.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:50:01.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:50:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:01 smithi038 ceph-mon[92422]: pgmap v15610: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 691 KiB/s rd, 765 KiB/s wr, 11 op/s 2023-03-09T22:50:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:50:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:50:03.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:03 smithi122 ceph-mon[93869]: pgmap v15611: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 915 KiB/s rd, 1009 KiB/s wr, 16 op/s 2023-03-09T22:50:03.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28313.772199 secs 2023-03-09T22:50:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:03 smithi181 ceph-mon[94189]: pgmap v15611: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 915 KiB/s rd, 1009 KiB/s wr, 16 op/s 2023-03-09T22:50:03.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28313.772199 secs 2023-03-09T22:50:03.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:03 smithi038 ceph-mon[92422]: pgmap v15611: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 915 KiB/s rd, 1009 KiB/s wr, 16 op/s 2023-03-09T22:50:03.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28313.772199 secs 2023-03-09T22:50:03.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28775.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:50:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:05 smithi122 ceph-mon[93869]: pgmap v15612: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 745 KiB/s rd, 770 KiB/s wr, 27 op/s 2023-03-09T22:50:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28316.390345 secs 2023-03-09T22:50:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:05 smithi181 ceph-mon[94189]: pgmap v15612: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 745 KiB/s rd, 770 KiB/s wr, 27 op/s 2023-03-09T22:50:05.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28316.390345 secs 2023-03-09T22:50:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:05 smithi038 ceph-mon[92422]: pgmap v15612: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 745 KiB/s rd, 770 KiB/s wr, 27 op/s 2023-03-09T22:50:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28316.390345 secs 2023-03-09T22:50:06.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:06 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:50:06.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:06 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:50:06.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:50:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:06 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:50:06.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:06 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:50:06.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:50:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:06 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:50:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:06 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:50:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:50:07.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:07 smithi122 ceph-mon[93869]: pgmap v15613: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 700 KiB/s rd, 724 KiB/s wr, 25 op/s 2023-03-09T22:50:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:07 smithi181 ceph-mon[94189]: pgmap v15613: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 700 KiB/s rd, 724 KiB/s wr, 25 op/s 2023-03-09T22:50:07.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:07 smithi038 ceph-mon[92422]: pgmap v15613: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 700 KiB/s rd, 724 KiB/s wr, 25 op/s 2023-03-09T22:50:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28318.772334 secs 2023-03-09T22:50:08.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28318.772334 secs 2023-03-09T22:50:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28318.772334 secs 2023-03-09T22:50:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28780.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:50:09.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:09 smithi122 ceph-mon[93869]: pgmap v15614: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 522 KiB/s wr, 26 op/s 2023-03-09T22:50:09.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:09 smithi181 ceph-mon[94189]: pgmap v15614: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 522 KiB/s wr, 26 op/s 2023-03-09T22:50:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:09 smithi038 ceph-mon[92422]: pgmap v15614: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 522 KiB/s wr, 26 op/s 2023-03-09T22:50:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:10 smithi122 ceph-mon[93869]: osdmap e1191: 12 total, 12 up, 12 in 2023-03-09T22:50:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28321.390417 secs 2023-03-09T22:50:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:10 smithi181 ceph-mon[94189]: osdmap e1191: 12 total, 12 up, 12 in 2023-03-09T22:50:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28321.390417 secs 2023-03-09T22:50:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:10 smithi038 ceph-mon[92422]: osdmap e1191: 12 total, 12 up, 12 in 2023-03-09T22:50:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28321.390417 secs 2023-03-09T22:50:11.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:11 smithi122 ceph-mon[93869]: pgmap v15616: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 702 KiB/s rd, 656 KiB/s wr, 28 op/s 2023-03-09T22:50:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:11 smithi181 ceph-mon[94189]: pgmap v15616: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 702 KiB/s rd, 656 KiB/s wr, 28 op/s 2023-03-09T22:50:11.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:11 smithi038 ceph-mon[92422]: pgmap v15616: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 702 KiB/s rd, 656 KiB/s wr, 28 op/s 2023-03-09T22:50:13.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:13 smithi122 ceph-mon[93869]: pgmap v15617: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 435 KiB/s rd, 363 KiB/s wr, 24 op/s 2023-03-09T22:50:13.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28323.772442 secs 2023-03-09T22:50:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:13 smithi181 ceph-mon[94189]: pgmap v15617: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 435 KiB/s rd, 363 KiB/s wr, 24 op/s 2023-03-09T22:50:13.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28323.772442 secs 2023-03-09T22:50:13.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:13 smithi038 ceph-mon[92422]: pgmap v15617: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 435 KiB/s rd, 363 KiB/s wr, 24 op/s 2023-03-09T22:50:13.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28323.772442 secs 2023-03-09T22:50:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28785.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:50:15.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:15 smithi122 ceph-mon[93869]: pgmap v15618: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 315 KiB/s wr, 8 op/s 2023-03-09T22:50:15.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28326.390518 secs 2023-03-09T22:50:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:15 smithi181 ceph-mon[94189]: pgmap v15618: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 315 KiB/s wr, 8 op/s 2023-03-09T22:50:15.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28326.390518 secs 2023-03-09T22:50:15.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:15 smithi038 ceph-mon[92422]: pgmap v15618: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 315 KiB/s wr, 8 op/s 2023-03-09T22:50:15.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28326.390518 secs 2023-03-09T22:50:17.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:17 smithi122 ceph-mon[93869]: pgmap v15619: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 315 KiB/s wr, 8 op/s 2023-03-09T22:50:17.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:50:17.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:50:17.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:17 smithi122 ceph-mon[93869]: osdmap e1192: 12 total, 12 up, 12 in 2023-03-09T22:50:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:17 smithi181 ceph-mon[94189]: pgmap v15619: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 315 KiB/s wr, 8 op/s 2023-03-09T22:50:17.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:50:17.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:50:17.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:17 smithi181 ceph-mon[94189]: osdmap e1192: 12 total, 12 up, 12 in 2023-03-09T22:50:17.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:17 smithi038 ceph-mon[92422]: pgmap v15619: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 315 KiB/s wr, 8 op/s 2023-03-09T22:50:17.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:50:17.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:50:17.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:17 smithi038 ceph-mon[92422]: osdmap e1192: 12 total, 12 up, 12 in 2023-03-09T22:50:18.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28328.772580 secs 2023-03-09T22:50:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28328.772580 secs 2023-03-09T22:50:18.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28328.772580 secs 2023-03-09T22:50:18.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28790.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:50:19.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:19 smithi122 ceph-mon[93869]: pgmap v15621: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 185 KiB/s wr, 9 op/s 2023-03-09T22:50:19.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:19 smithi181 ceph-mon[94189]: pgmap v15621: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 185 KiB/s wr, 9 op/s 2023-03-09T22:50:19.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:19 smithi038 ceph-mon[92422]: pgmap v15621: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 185 KiB/s wr, 9 op/s 2023-03-09T22:50:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:50:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:50:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28331.390576 secs 2023-03-09T22:50:20.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:50:20.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:50:20.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28331.390576 secs 2023-03-09T22:50:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:50:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:50:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28331.390576 secs 2023-03-09T22:50:21.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:21 smithi122 ceph-mon[93869]: pgmap v15622: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 151 KiB/s wr, 8 op/s 2023-03-09T22:50:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:21 smithi181 ceph-mon[94189]: pgmap v15622: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 151 KiB/s wr, 8 op/s 2023-03-09T22:50:21.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:21 smithi038 ceph-mon[92422]: pgmap v15622: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 151 KiB/s wr, 8 op/s 2023-03-09T22:50:23.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:23 smithi122 ceph-mon[93869]: pgmap v15623: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 151 KiB/s wr, 8 op/s 2023-03-09T22:50:23.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28333.772718 secs 2023-03-09T22:50:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:23 smithi181 ceph-mon[94189]: pgmap v15623: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 151 KiB/s wr, 8 op/s 2023-03-09T22:50:23.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28333.772718 secs 2023-03-09T22:50:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:23 smithi038 ceph-mon[92422]: pgmap v15623: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 151 KiB/s wr, 8 op/s 2023-03-09T22:50:23.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28333.772718 secs 2023-03-09T22:50:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28795.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:50:25.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:25 smithi122 ceph-mon[93869]: pgmap v15624: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 409 B/s wr, 7 op/s 2023-03-09T22:50:25.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28336.390638 secs 2023-03-09T22:50:25.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:25 smithi181 ceph-mon[94189]: pgmap v15624: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 409 B/s wr, 7 op/s 2023-03-09T22:50:25.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28336.390638 secs 2023-03-09T22:50:25.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:25 smithi038 ceph-mon[92422]: pgmap v15624: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 409 B/s wr, 7 op/s 2023-03-09T22:50:25.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28336.390638 secs 2023-03-09T22:50:27.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:27 smithi122 ceph-mon[93869]: pgmap v15625: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 409 B/s wr, 7 op/s 2023-03-09T22:50:27.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:50:27.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:27 smithi181 ceph-mon[94189]: pgmap v15625: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 409 B/s wr, 7 op/s 2023-03-09T22:50:27.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:50:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:27 smithi038 ceph-mon[92422]: pgmap v15625: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 409 B/s wr, 7 op/s 2023-03-09T22:50:27.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:50:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28338.772808 secs 2023-03-09T22:50:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:50:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:50:28.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:50:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28338.772808 secs 2023-03-09T22:50:28.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:50:28.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:50:28.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:50:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28338.772808 secs 2023-03-09T22:50:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:50:28.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:50:28.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:50:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28800.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:50:29.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:29 smithi122 ceph-mon[93869]: pgmap v15626: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 375 B/s wr, 7 op/s 2023-03-09T22:50:29.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:29 smithi181 ceph-mon[94189]: pgmap v15626: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 375 B/s wr, 7 op/s 2023-03-09T22:50:29.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:29 smithi038 ceph-mon[92422]: pgmap v15626: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 375 B/s wr, 7 op/s 2023-03-09T22:50:30.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28341.390697 secs 2023-03-09T22:50:30.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28341.390697 secs 2023-03-09T22:50:30.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28341.390697 secs 2023-03-09T22:50:31.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:31 smithi122 ceph-mon[93869]: pgmap v15627: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T22:50:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:31 smithi181 ceph-mon[94189]: pgmap v15627: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T22:50:31.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:31 smithi038 ceph-mon[92422]: pgmap v15627: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T22:50:32.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:32 smithi122 ceph-mon[93869]: pgmap v15628: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T22:50:32.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28343.772928 secs 2023-03-09T22:50:32.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:32 smithi181 ceph-mon[94189]: pgmap v15628: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T22:50:32.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28343.772928 secs 2023-03-09T22:50:32.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:32 smithi038 ceph-mon[92422]: pgmap v15628: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T22:50:33.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28343.772928 secs 2023-03-09T22:50:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28805.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:50:34.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:34 smithi122 ceph-mon[93869]: pgmap v15629: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T22:50:34.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:34 smithi181 ceph-mon[94189]: pgmap v15629: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T22:50:35.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:34 smithi038 ceph-mon[92422]: pgmap v15629: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T22:50:35.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28346.390759 secs 2023-03-09T22:50:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28346.390759 secs 2023-03-09T22:50:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28346.390759 secs 2023-03-09T22:50:36.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:36 smithi122 ceph-mon[93869]: pgmap v15630: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T22:50:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:36 smithi181 ceph-mon[94189]: pgmap v15630: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T22:50:37.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:36 smithi038 ceph-mon[92422]: pgmap v15630: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T22:50:37.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28348.773052 secs 2023-03-09T22:50:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28348.773052 secs 2023-03-09T22:50:38.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28348.773052 secs 2023-03-09T22:50:38.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28810.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:50:38.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:38 smithi122 ceph-mon[93869]: pgmap v15631: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 426 B/s wr, 9 op/s 2023-03-09T22:50:38.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:38 smithi181 ceph-mon[94189]: pgmap v15631: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 426 B/s wr, 9 op/s 2023-03-09T22:50:39.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:38 smithi038 ceph-mon[92422]: pgmap v15631: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 426 B/s wr, 9 op/s 2023-03-09T22:50:40.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:40 smithi122 ceph-mon[93869]: pgmap v15632: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 7 op/s 2023-03-09T22:50:40.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28351.390819 secs 2023-03-09T22:50:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:40 smithi181 ceph-mon[94189]: pgmap v15632: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 7 op/s 2023-03-09T22:50:40.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28351.390819 secs 2023-03-09T22:50:41.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:40 smithi038 ceph-mon[92422]: pgmap v15632: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 7 op/s 2023-03-09T22:50:41.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28351.390819 secs 2023-03-09T22:50:42.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:42 smithi122 ceph-mon[93869]: pgmap v15633: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 511 B/s wr, 8 op/s 2023-03-09T22:50:42.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:42 smithi181 ceph-mon[94189]: pgmap v15633: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 511 B/s wr, 8 op/s 2023-03-09T22:50:43.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:42 smithi038 ceph-mon[92422]: pgmap v15633: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 511 B/s wr, 8 op/s 2023-03-09T22:50:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28815.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:50:44.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:44 smithi122 ceph-mon[93869]: pgmap v15634: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 237 KiB/s wr, 8 op/s 2023-03-09T22:50:44.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:44 smithi181 ceph-mon[94189]: pgmap v15634: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 237 KiB/s wr, 8 op/s 2023-03-09T22:50:45.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:44 smithi038 ceph-mon[92422]: pgmap v15634: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 237 KiB/s wr, 8 op/s 2023-03-09T22:50:45.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28356.390879 secs 2023-03-09T22:50:45.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:45 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:50:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28356.390879 secs 2023-03-09T22:50:45.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:45 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:50:46.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28356.390879 secs 2023-03-09T22:50:46.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:45 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:50:46.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:46 smithi122 ceph-mon[93869]: pgmap v15635: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 237 KiB/s wr, 6 op/s 2023-03-09T22:50:46.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:46 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:50:46.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:46 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:50:46.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:46 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:50:46.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:46 smithi181 ceph-mon[94189]: pgmap v15635: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 237 KiB/s wr, 6 op/s 2023-03-09T22:50:46.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:46 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:50:46.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:46 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:50:46.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:46 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:50:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:46 smithi038 ceph-mon[92422]: pgmap v15635: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 237 KiB/s wr, 6 op/s 2023-03-09T22:50:47.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:46 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:50:47.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:46 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:50:47.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:46 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:50:48.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28820.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:50:48.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:48 smithi122 ceph-mon[93869]: pgmap v15636: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 815 KiB/s rd, 237 KiB/s wr, 10 op/s 2023-03-09T22:50:48.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:48 smithi181 ceph-mon[94189]: pgmap v15636: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 815 KiB/s rd, 237 KiB/s wr, 10 op/s 2023-03-09T22:50:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:48 smithi038 ceph-mon[92422]: pgmap v15636: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 815 KiB/s rd, 237 KiB/s wr, 10 op/s 2023-03-09T22:50:50.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:50 smithi122 ceph-mon[93869]: pgmap v15637: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 833 KiB/s rd, 256 KiB/s wr, 7 op/s 2023-03-09T22:50:50.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28361.390937 secs 2023-03-09T22:50:50.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:50 smithi181 ceph-mon[94189]: pgmap v15637: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 833 KiB/s rd, 256 KiB/s wr, 7 op/s 2023-03-09T22:50:50.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28361.390937 secs 2023-03-09T22:50:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:50 smithi038 ceph-mon[92422]: pgmap v15637: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 833 KiB/s rd, 256 KiB/s wr, 7 op/s 2023-03-09T22:50:51.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28361.390937 secs 2023-03-09T22:50:52.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:52 smithi122 ceph-mon[93869]: pgmap v15638: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 834 KiB/s rd, 256 KiB/s wr, 9 op/s 2023-03-09T22:50:52.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:52 smithi181 ceph-mon[94189]: pgmap v15638: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 834 KiB/s rd, 256 KiB/s wr, 9 op/s 2023-03-09T22:50:53.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:52 smithi038 ceph-mon[92422]: pgmap v15638: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 834 KiB/s rd, 256 KiB/s wr, 9 op/s 2023-03-09T22:50:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28825.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:50:53.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:50:53.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:50:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:50:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:50:54.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:50:54.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:50:54.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:54 smithi181 ceph-mon[94189]: pgmap v15639: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 530 KiB/s wr, 10 op/s 2023-03-09T22:50:55.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:54 smithi038 ceph-mon[92422]: pgmap v15639: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 530 KiB/s wr, 10 op/s 2023-03-09T22:50:55.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:54 smithi122 ceph-mon[93869]: pgmap v15639: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 530 KiB/s wr, 10 op/s 2023-03-09T22:50:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:50:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:50:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28366.390996 secs 2023-03-09T22:50:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:50:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:50:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28366.390996 secs 2023-03-09T22:50:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:50:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:50:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28366.390996 secs 2023-03-09T22:50:57.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:56 smithi038 ceph-mon[92422]: pgmap v15640: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 756 KiB/s rd, 295 KiB/s wr, 8 op/s 2023-03-09T22:50:57.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:56 smithi122 ceph-mon[93869]: pgmap v15640: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 756 KiB/s rd, 295 KiB/s wr, 8 op/s 2023-03-09T22:50:57.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:56 smithi181 ceph-mon[94189]: pgmap v15640: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 756 KiB/s rd, 295 KiB/s wr, 8 op/s 2023-03-09T22:50:58.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:57 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 28368.773571 secs 2023-03-09T22:50:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:57 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 28368.773571 secs 2023-03-09T22:50:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:57 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 28368.773571 secs 2023-03-09T22:50:58.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28830.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:50:59.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:58 smithi122 ceph-mon[93869]: pgmap v15641: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 957 KiB/s rd, 493 KiB/s wr, 13 op/s 2023-03-09T22:50:59.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:58 smithi181 ceph-mon[94189]: pgmap v15641: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 957 KiB/s rd, 493 KiB/s wr, 13 op/s 2023-03-09T22:50:59.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:58 smithi038 ceph-mon[92422]: pgmap v15641: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 957 KiB/s rd, 493 KiB/s wr, 13 op/s 2023-03-09T22:51:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:51:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:50:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:51:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:51:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:50:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:51:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:51:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:50:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:51:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:00 smithi122 ceph-mon[93869]: pgmap v15642: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 578 KiB/s rd, 565 KiB/s wr, 11 op/s 2023-03-09T22:51:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:51:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:51:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28371.391057 secs 2023-03-09T22:51:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:00 smithi181 ceph-mon[94189]: pgmap v15642: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 578 KiB/s rd, 565 KiB/s wr, 11 op/s 2023-03-09T22:51:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:51:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:51:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28371.391057 secs 2023-03-09T22:51:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:00 smithi038 ceph-mon[92422]: pgmap v15642: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 578 KiB/s rd, 565 KiB/s wr, 11 op/s 2023-03-09T22:51:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:51:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:51:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28371.391057 secs 2023-03-09T22:51:02.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:02 smithi122 ceph-mon[93869]: pgmap v15643: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 731 KiB/s rd, 727 KiB/s wr, 15 op/s 2023-03-09T22:51:02.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:02 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:51:02.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:02 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:51:02.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:02 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:51:02.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28373.773686 secs 2023-03-09T22:51:02.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:02 smithi181 ceph-mon[94189]: pgmap v15643: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 731 KiB/s rd, 727 KiB/s wr, 15 op/s 2023-03-09T22:51:02.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:02 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:51:02.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:02 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:51:02.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:02 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:51:02.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28373.773686 secs 2023-03-09T22:51:03.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:02 smithi038 ceph-mon[92422]: pgmap v15643: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 731 KiB/s rd, 727 KiB/s wr, 15 op/s 2023-03-09T22:51:03.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:02 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:51:03.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:02 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:51:03.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:02 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:51:03.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28373.773686 secs 2023-03-09T22:51:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28835.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:51:04.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:04 smithi122 ceph-mon[93869]: pgmap v15644: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 959 KiB/s rd, 971 KiB/s wr, 28 op/s 2023-03-09T22:51:04.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:04 smithi181 ceph-mon[94189]: pgmap v15644: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 959 KiB/s rd, 971 KiB/s wr, 28 op/s 2023-03-09T22:51:05.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:04 smithi038 ceph-mon[92422]: pgmap v15644: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 959 KiB/s rd, 971 KiB/s wr, 28 op/s 2023-03-09T22:51:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28376.391118 secs 2023-03-09T22:51:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28376.391118 secs 2023-03-09T22:51:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28376.391118 secs 2023-03-09T22:51:06.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:06 smithi122 ceph-mon[93869]: pgmap v15645: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 686 KiB/s rd, 697 KiB/s wr, 26 op/s 2023-03-09T22:51:06.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:06 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:51:06.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:51:06.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:06 smithi181 ceph-mon[94189]: pgmap v15645: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 686 KiB/s rd, 697 KiB/s wr, 26 op/s 2023-03-09T22:51:06.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:06 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:51:06.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:51:07.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:06 smithi038 ceph-mon[92422]: pgmap v15645: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 686 KiB/s rd, 697 KiB/s wr, 26 op/s 2023-03-09T22:51:07.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:06 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:51:07.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:51:07.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:51:07.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:51:07.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28378.773802 secs 2023-03-09T22:51:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:51:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:51:07.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28378.773802 secs 2023-03-09T22:51:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:51:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:51:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28378.773802 secs 2023-03-09T22:51:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28840.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:51:08.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:08 smithi122 ceph-mon[93869]: pgmap v15646: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 707 KiB/s rd, 727 KiB/s wr, 30 op/s 2023-03-09T22:51:08.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:08 smithi181 ceph-mon[94189]: pgmap v15646: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 707 KiB/s rd, 727 KiB/s wr, 30 op/s 2023-03-09T22:51:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:08 smithi038 ceph-mon[92422]: pgmap v15646: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 707 KiB/s rd, 727 KiB/s wr, 30 op/s 2023-03-09T22:51:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:10 smithi122 ceph-mon[93869]: pgmap v15647: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 529 KiB/s wr, 26 op/s 2023-03-09T22:51:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:10 smithi122 ceph-mon[93869]: osdmap e1193: 12 total, 12 up, 12 in 2023-03-09T22:51:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28381.391182 secs 2023-03-09T22:51:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:10 smithi181 ceph-mon[94189]: pgmap v15647: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 529 KiB/s wr, 26 op/s 2023-03-09T22:51:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:10 smithi181 ceph-mon[94189]: osdmap e1193: 12 total, 12 up, 12 in 2023-03-09T22:51:10.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28381.391182 secs 2023-03-09T22:51:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:10 smithi038 ceph-mon[92422]: pgmap v15647: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 529 KiB/s wr, 26 op/s 2023-03-09T22:51:11.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:10 smithi038 ceph-mon[92422]: osdmap e1193: 12 total, 12 up, 12 in 2023-03-09T22:51:11.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28381.391182 secs 2023-03-09T22:51:12.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:12 smithi122 ceph-mon[93869]: pgmap v15649: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 331 KiB/s wr, 27 op/s 2023-03-09T22:51:12.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28383.773909 secs 2023-03-09T22:51:12.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:12 smithi181 ceph-mon[94189]: pgmap v15649: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 331 KiB/s wr, 27 op/s 2023-03-09T22:51:12.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28383.773909 secs 2023-03-09T22:51:13.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:12 smithi038 ceph-mon[92422]: pgmap v15649: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 331 KiB/s wr, 27 op/s 2023-03-09T22:51:13.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28383.773909 secs 2023-03-09T22:51:13.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28845.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:51:14.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:14 smithi122 ceph-mon[93869]: pgmap v15650: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 236 KiB/s wr, 10 op/s 2023-03-09T22:51:14.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:14 smithi181 ceph-mon[94189]: pgmap v15650: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 236 KiB/s wr, 10 op/s 2023-03-09T22:51:15.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:14 smithi038 ceph-mon[92422]: pgmap v15650: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 236 KiB/s wr, 10 op/s 2023-03-09T22:51:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28386.391243 secs 2023-03-09T22:51:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28386.391243 secs 2023-03-09T22:51:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28386.391243 secs 2023-03-09T22:51:16.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:16 smithi122 ceph-mon[93869]: pgmap v15651: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 349 KiB/s wr, 11 op/s 2023-03-09T22:51:16.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:16 smithi122 ceph-mon[93869]: osdmap e1194: 12 total, 12 up, 12 in 2023-03-09T22:51:16.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:16 smithi181 ceph-mon[94189]: pgmap v15651: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 349 KiB/s wr, 11 op/s 2023-03-09T22:51:16.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:16 smithi181 ceph-mon[94189]: osdmap e1194: 12 total, 12 up, 12 in 2023-03-09T22:51:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:16 smithi038 ceph-mon[92422]: pgmap v15651: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 349 KiB/s wr, 11 op/s 2023-03-09T22:51:17.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:16 smithi038 ceph-mon[92422]: osdmap e1194: 12 total, 12 up, 12 in 2023-03-09T22:51:17.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:51:17.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:51:17.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28388.774032 secs 2023-03-09T22:51:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:51:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:51:17.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28388.774032 secs 2023-03-09T22:51:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:51:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:51:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28388.774032 secs 2023-03-09T22:51:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28850.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:51:18.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:18 smithi122 ceph-mon[93869]: pgmap v15653: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 390 KiB/s wr, 10 op/s 2023-03-09T22:51:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:18 smithi181 ceph-mon[94189]: pgmap v15653: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 390 KiB/s wr, 10 op/s 2023-03-09T22:51:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:18 smithi038 ceph-mon[92422]: pgmap v15653: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 390 KiB/s wr, 10 op/s 2023-03-09T22:51:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:19 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:51:19.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:19 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:51:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:19 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:51:20.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:19 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:51:20.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:19 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:51:20.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:19 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:51:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:20 smithi181 ceph-mon[94189]: pgmap v15654: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 416 KiB/s wr, 9 op/s 2023-03-09T22:51:20.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28391.391303 secs 2023-03-09T22:51:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:20 smithi038 ceph-mon[92422]: pgmap v15654: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 416 KiB/s wr, 9 op/s 2023-03-09T22:51:21.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28391.391303 secs 2023-03-09T22:51:21.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:20 smithi122 ceph-mon[93869]: pgmap v15654: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 416 KiB/s wr, 9 op/s 2023-03-09T22:51:21.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28391.391303 secs 2023-03-09T22:51:22.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:22 smithi038 ceph-mon[92422]: pgmap v15655: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 412 KiB/s wr, 8 op/s 2023-03-09T22:51:23.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28393.774157 secs 2023-03-09T22:51:23.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:22 smithi122 ceph-mon[93869]: pgmap v15655: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 412 KiB/s wr, 8 op/s 2023-03-09T22:51:23.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28393.774157 secs 2023-03-09T22:51:23.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:22 smithi181 ceph-mon[94189]: pgmap v15655: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 412 KiB/s wr, 8 op/s 2023-03-09T22:51:23.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28393.774157 secs 2023-03-09T22:51:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28855.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:51:25.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:24 smithi038 ceph-mon[92422]: pgmap v15656: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 214 KiB/s wr, 7 op/s 2023-03-09T22:51:25.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:24 smithi122 ceph-mon[93869]: pgmap v15656: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 214 KiB/s wr, 7 op/s 2023-03-09T22:51:25.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:24 smithi181 ceph-mon[94189]: pgmap v15656: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 214 KiB/s wr, 7 op/s 2023-03-09T22:51:26.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28396.391364 secs 2023-03-09T22:51:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28396.391364 secs 2023-03-09T22:51:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28396.391364 secs 2023-03-09T22:51:26.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:26 smithi038 ceph-mon[92422]: pgmap v15657: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 100 KiB/s wr, 7 op/s 2023-03-09T22:51:27.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:26 smithi122 ceph-mon[93869]: pgmap v15657: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 100 KiB/s wr, 7 op/s 2023-03-09T22:51:27.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:26 smithi181 ceph-mon[94189]: pgmap v15657: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 100 KiB/s wr, 7 op/s 2023-03-09T22:51:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28398.774272 secs 2023-03-09T22:51:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28398.774272 secs 2023-03-09T22:51:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28398.774272 secs 2023-03-09T22:51:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28860.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:51:29.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:28 smithi122 ceph-mon[93869]: pgmap v15658: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 92 KiB/s wr, 7 op/s 2023-03-09T22:51:29.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:51:29.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:51:29.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:51:29.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:28 smithi181 ceph-mon[94189]: pgmap v15658: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 92 KiB/s wr, 7 op/s 2023-03-09T22:51:29.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:51:29.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:51:29.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:51:29.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:28 smithi038 ceph-mon[92422]: pgmap v15658: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 92 KiB/s wr, 7 op/s 2023-03-09T22:51:29.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:51:29.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:51:29.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:51:30.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:51:30.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:51:30.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:51:30.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:51:30.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:51:30.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:51:30.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:51:30.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:51:30.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:51:31.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:30 smithi122 ceph-mon[93869]: pgmap v15659: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 83 KiB/s wr, 6 op/s 2023-03-09T22:51:31.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28401.391437 secs 2023-03-09T22:51:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:30 smithi181 ceph-mon[94189]: pgmap v15659: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 83 KiB/s wr, 6 op/s 2023-03-09T22:51:31.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28401.391437 secs 2023-03-09T22:51:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:30 smithi038 ceph-mon[92422]: pgmap v15659: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 83 KiB/s wr, 6 op/s 2023-03-09T22:51:31.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28401.391437 secs 2023-03-09T22:51:33.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:32 smithi122 ceph-mon[93869]: pgmap v15660: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T22:51:33.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28403.774384 secs 2023-03-09T22:51:33.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:32 smithi181 ceph-mon[94189]: pgmap v15660: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T22:51:33.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28403.774384 secs 2023-03-09T22:51:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:32 smithi038 ceph-mon[92422]: pgmap v15660: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T22:51:33.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28403.774384 secs 2023-03-09T22:51:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28865.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:51:35.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:34 smithi122 ceph-mon[93869]: pgmap v15661: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T22:51:35.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:34 smithi181 ceph-mon[94189]: pgmap v15661: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T22:51:35.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:34 smithi038 ceph-mon[92422]: pgmap v15661: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T22:51:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28406.391501 secs 2023-03-09T22:51:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28406.391501 secs 2023-03-09T22:51:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28406.391501 secs 2023-03-09T22:51:37.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:36 smithi122 ceph-mon[93869]: pgmap v15662: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T22:51:37.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:36 smithi181 ceph-mon[94189]: pgmap v15662: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T22:51:37.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:36 smithi038 ceph-mon[92422]: pgmap v15662: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T22:51:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28408.774450 secs 2023-03-09T22:51:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28408.774450 secs 2023-03-09T22:51:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28408.774450 secs 2023-03-09T22:51:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28870.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:51:39.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:38 smithi122 ceph-mon[93869]: pgmap v15663: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T22:51:39.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:38 smithi181 ceph-mon[94189]: pgmap v15663: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T22:51:39.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:38 smithi038 ceph-mon[92422]: pgmap v15663: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T22:51:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:40 smithi122 ceph-mon[93869]: pgmap v15664: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T22:51:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28411.391558 secs 2023-03-09T22:51:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:40 smithi181 ceph-mon[94189]: pgmap v15664: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T22:51:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28411.391558 secs 2023-03-09T22:51:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:40 smithi038 ceph-mon[92422]: pgmap v15664: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T22:51:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28411.391558 secs 2023-03-09T22:51:43.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:42 smithi122 ceph-mon[93869]: pgmap v15665: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2023-03-09T22:51:43.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28413.774563 secs 2023-03-09T22:51:43.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:42 smithi181 ceph-mon[94189]: pgmap v15665: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2023-03-09T22:51:43.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28413.774563 secs 2023-03-09T22:51:43.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:42 smithi038 ceph-mon[92422]: pgmap v15665: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2023-03-09T22:51:43.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28413.774563 secs 2023-03-09T22:51:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28875.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:51:45.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:44 smithi122 ceph-mon[93869]: pgmap v15666: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 767 B/s wr, 7 op/s 2023-03-09T22:51:45.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:44 smithi181 ceph-mon[94189]: pgmap v15666: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 767 B/s wr, 7 op/s 2023-03-09T22:51:45.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:44 smithi038 ceph-mon[92422]: pgmap v15666: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 767 B/s wr, 7 op/s 2023-03-09T22:51:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28416.391620 secs 2023-03-09T22:51:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28416.391620 secs 2023-03-09T22:51:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28416.391620 secs 2023-03-09T22:51:47.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:46 smithi122 ceph-mon[93869]: pgmap v15667: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 293 KiB/s rd, 291 KiB/s wr, 8 op/s 2023-03-09T22:51:47.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:46 smithi181 ceph-mon[94189]: pgmap v15667: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 293 KiB/s rd, 291 KiB/s wr, 8 op/s 2023-03-09T22:51:47.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:46 smithi038 ceph-mon[92422]: pgmap v15667: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 293 KiB/s rd, 291 KiB/s wr, 8 op/s 2023-03-09T22:51:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28418.774692 secs 2023-03-09T22:51:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28418.774692 secs 2023-03-09T22:51:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28418.774692 secs 2023-03-09T22:51:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28880.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:51:49.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:48 smithi122 ceph-mon[93869]: pgmap v15668: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 314 KiB/s wr, 10 op/s 2023-03-09T22:51:49.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:51:49.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:51:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:48 smithi181 ceph-mon[94189]: pgmap v15668: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 314 KiB/s wr, 10 op/s 2023-03-09T22:51:49.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:51:49.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:51:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:48 smithi038 ceph-mon[92422]: pgmap v15668: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 314 KiB/s wr, 10 op/s 2023-03-09T22:51:49.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:51:49.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:51:51.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:50 smithi122 ceph-mon[93869]: pgmap v15669: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 314 KiB/s wr, 9 op/s 2023-03-09T22:51:51.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28421.391683 secs 2023-03-09T22:51:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:50 smithi181 ceph-mon[94189]: pgmap v15669: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 314 KiB/s wr, 9 op/s 2023-03-09T22:51:51.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28421.391683 secs 2023-03-09T22:51:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:50 smithi038 ceph-mon[92422]: pgmap v15669: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 314 KiB/s wr, 9 op/s 2023-03-09T22:51:51.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28421.391683 secs 2023-03-09T22:51:52.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:51:52.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:51:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:51:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:51:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:51:52.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:51:53.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:52 smithi122 ceph-mon[93869]: pgmap v15670: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 576 KiB/s rd, 571 KiB/s wr, 12 op/s 2023-03-09T22:51:53.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28423.774798 secs 2023-03-09T22:51:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:52 smithi181 ceph-mon[94189]: pgmap v15670: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 576 KiB/s rd, 571 KiB/s wr, 12 op/s 2023-03-09T22:51:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28423.774798 secs 2023-03-09T22:51:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:52 smithi038 ceph-mon[92422]: pgmap v15670: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 576 KiB/s rd, 571 KiB/s wr, 12 op/s 2023-03-09T22:51:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28423.774798 secs 2023-03-09T22:51:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28885.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:51:55.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:54 smithi122 ceph-mon[93869]: pgmap v15671: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 572 KiB/s wr, 11 op/s 2023-03-09T22:51:55.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:54 smithi181 ceph-mon[94189]: pgmap v15671: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 572 KiB/s wr, 11 op/s 2023-03-09T22:51:55.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:54 smithi038 ceph-mon[92422]: pgmap v15671: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 572 KiB/s wr, 11 op/s 2023-03-09T22:51:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:51:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:51:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28426.391744 secs 2023-03-09T22:51:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:51:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:51:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28426.391744 secs 2023-03-09T22:51:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:51:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:51:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28426.391744 secs 2023-03-09T22:51:57.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:56 smithi122 ceph-mon[93869]: pgmap v15672: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 908 KiB/s rd, 894 KiB/s wr, 12 op/s 2023-03-09T22:51:57.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:51:57.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:51:57.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:56 smithi181 ceph-mon[94189]: pgmap v15672: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 908 KiB/s rd, 894 KiB/s wr, 12 op/s 2023-03-09T22:51:57.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:51:57.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:51:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:56 smithi038 ceph-mon[92422]: pgmap v15672: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 908 KiB/s rd, 894 KiB/s wr, 12 op/s 2023-03-09T22:51:57.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:51:57.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:51:58.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28428.774933 secs 2023-03-09T22:51:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28428.774933 secs 2023-03-09T22:51:58.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28428.774933 secs 2023-03-09T22:51:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28890.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:51:59.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:58 smithi122 ceph-mon[93869]: pgmap v15673: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 719 KiB/s rd, 705 KiB/s wr, 14 op/s 2023-03-09T22:51:59.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:58 smithi181 ceph-mon[94189]: pgmap v15673: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 719 KiB/s rd, 705 KiB/s wr, 14 op/s 2023-03-09T22:51:59.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:58 smithi038 ceph-mon[92422]: pgmap v15673: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 719 KiB/s rd, 705 KiB/s wr, 14 op/s 2023-03-09T22:52:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:52:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:51:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:52:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:52:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:51:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:52:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:52:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:51:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:52:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:00 smithi122 ceph-mon[93869]: pgmap v15674: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 695 KiB/s rd, 682 KiB/s wr, 12 op/s 2023-03-09T22:52:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:52:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:52:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28431.391806 secs 2023-03-09T22:52:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:00 smithi181 ceph-mon[94189]: pgmap v15674: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 695 KiB/s rd, 682 KiB/s wr, 12 op/s 2023-03-09T22:52:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:52:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:52:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28431.391806 secs 2023-03-09T22:52:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:00 smithi038 ceph-mon[92422]: pgmap v15674: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 695 KiB/s rd, 682 KiB/s wr, 12 op/s 2023-03-09T22:52:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:52:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:52:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28431.391806 secs 2023-03-09T22:52:03.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:02 smithi122 ceph-mon[93869]: pgmap v15675: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 871 KiB/s rd, 857 KiB/s wr, 16 op/s 2023-03-09T22:52:03.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28433.775054 secs 2023-03-09T22:52:03.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:02 smithi181 ceph-mon[94189]: pgmap v15675: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 871 KiB/s rd, 857 KiB/s wr, 16 op/s 2023-03-09T22:52:03.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28433.775054 secs 2023-03-09T22:52:03.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:02 smithi038 ceph-mon[92422]: pgmap v15675: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 871 KiB/s rd, 857 KiB/s wr, 16 op/s 2023-03-09T22:52:03.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28433.775054 secs 2023-03-09T22:52:03.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28895.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:52:05.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:04 smithi122 ceph-mon[93869]: pgmap v15676: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 618 KiB/s rd, 621 KiB/s wr, 25 op/s 2023-03-09T22:52:05.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:04 smithi181 ceph-mon[94189]: pgmap v15676: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 618 KiB/s rd, 621 KiB/s wr, 25 op/s 2023-03-09T22:52:05.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:04 smithi038 ceph-mon[92422]: pgmap v15676: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 618 KiB/s rd, 621 KiB/s wr, 25 op/s 2023-03-09T22:52:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:05 smithi122 ceph-mon[93869]: osdmap e1195: 12 total, 12 up, 12 in 2023-03-09T22:52:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28436.391870 secs 2023-03-09T22:52:06.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:05 smithi181 ceph-mon[94189]: osdmap e1195: 12 total, 12 up, 12 in 2023-03-09T22:52:06.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28436.391870 secs 2023-03-09T22:52:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:05 smithi038 ceph-mon[92422]: osdmap e1195: 12 total, 12 up, 12 in 2023-03-09T22:52:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28436.391870 secs 2023-03-09T22:52:07.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:06 smithi122 ceph-mon[93869]: pgmap v15678: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 357 KiB/s wr, 26 op/s 2023-03-09T22:52:07.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:06 smithi181 ceph-mon[94189]: pgmap v15678: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 357 KiB/s wr, 26 op/s 2023-03-09T22:52:07.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:06 smithi038 ceph-mon[92422]: pgmap v15678: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 357 KiB/s wr, 26 op/s 2023-03-09T22:52:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:52:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:52:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28438.775191 secs 2023-03-09T22:52:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:52:08.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:52:08.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28438.775191 secs 2023-03-09T22:52:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:52:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:52:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28438.775191 secs 2023-03-09T22:52:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28900.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:52:09.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:08 smithi122 ceph-mon[93869]: pgmap v15679: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 480 KiB/s wr, 29 op/s 2023-03-09T22:52:09.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:52:09.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:52:09.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:08 smithi181 ceph-mon[94189]: pgmap v15679: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 480 KiB/s wr, 29 op/s 2023-03-09T22:52:09.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:52:09.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:52:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:08 smithi038 ceph-mon[92422]: pgmap v15679: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 480 KiB/s wr, 29 op/s 2023-03-09T22:52:09.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:52:09.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:52:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:10 smithi122 ceph-mon[93869]: pgmap v15680: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 480 KiB/s wr, 28 op/s 2023-03-09T22:52:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28441.391930 secs 2023-03-09T22:52:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:10 smithi181 ceph-mon[94189]: pgmap v15680: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 480 KiB/s wr, 28 op/s 2023-03-09T22:52:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28441.391930 secs 2023-03-09T22:52:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:10 smithi038 ceph-mon[92422]: pgmap v15680: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 480 KiB/s wr, 28 op/s 2023-03-09T22:52:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28441.391930 secs 2023-03-09T22:52:12.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:12 smithi122 ceph-mon[93869]: pgmap v15681: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 412 KiB/s wr, 27 op/s 2023-03-09T22:52:12.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:12 smithi122 ceph-mon[93869]: osdmap e1196: 12 total, 12 up, 12 in 2023-03-09T22:52:12.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28443.775288 secs 2023-03-09T22:52:12.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:12 smithi181 ceph-mon[94189]: pgmap v15681: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 412 KiB/s wr, 27 op/s 2023-03-09T22:52:12.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:12 smithi181 ceph-mon[94189]: osdmap e1196: 12 total, 12 up, 12 in 2023-03-09T22:52:12.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28443.775288 secs 2023-03-09T22:52:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:12 smithi038 ceph-mon[92422]: pgmap v15681: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 412 KiB/s wr, 27 op/s 2023-03-09T22:52:13.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:12 smithi038 ceph-mon[92422]: osdmap e1196: 12 total, 12 up, 12 in 2023-03-09T22:52:13.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28443.775288 secs 2023-03-09T22:52:13.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28905.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:52:14.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:14 smithi122 ceph-mon[93869]: pgmap v15683: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 445 KiB/s wr, 14 op/s 2023-03-09T22:52:14.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:14 smithi181 ceph-mon[94189]: pgmap v15683: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 445 KiB/s wr, 14 op/s 2023-03-09T22:52:14.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:14 smithi038 ceph-mon[92422]: pgmap v15683: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 445 KiB/s wr, 14 op/s 2023-03-09T22:52:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28446.392015 secs 2023-03-09T22:52:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28446.392015 secs 2023-03-09T22:52:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28446.392015 secs 2023-03-09T22:52:16.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:16 smithi122 ceph-mon[93869]: pgmap v15684: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 363 KiB/s rd, 386 KiB/s wr, 12 op/s 2023-03-09T22:52:16.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:16 smithi181 ceph-mon[94189]: pgmap v15684: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 363 KiB/s rd, 386 KiB/s wr, 12 op/s 2023-03-09T22:52:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:16 smithi038 ceph-mon[92422]: pgmap v15684: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 363 KiB/s rd, 386 KiB/s wr, 12 op/s 2023-03-09T22:52:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28448.775401 secs 2023-03-09T22:52:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28448.775401 secs 2023-03-09T22:52:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28448.775401 secs 2023-03-09T22:52:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28910.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:52:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:18 smithi122 ceph-mon[93869]: pgmap v15685: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 142 KiB/s wr, 8 op/s 2023-03-09T22:52:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:18 smithi181 ceph-mon[94189]: pgmap v15685: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 142 KiB/s wr, 8 op/s 2023-03-09T22:52:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:18 smithi038 ceph-mon[92422]: pgmap v15685: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 142 KiB/s wr, 8 op/s 2023-03-09T22:52:20.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:20 smithi122 ceph-mon[93869]: pgmap v15686: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 142 KiB/s wr, 7 op/s 2023-03-09T22:52:20.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28451.392077 secs 2023-03-09T22:52:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:20 smithi181 ceph-mon[94189]: pgmap v15686: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 142 KiB/s wr, 7 op/s 2023-03-09T22:52:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28451.392077 secs 2023-03-09T22:52:21.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:20 smithi038 ceph-mon[92422]: pgmap v15686: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 142 KiB/s wr, 7 op/s 2023-03-09T22:52:21.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28451.392077 secs 2023-03-09T22:52:22.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:22 smithi122 ceph-mon[93869]: pgmap v15687: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T22:52:22.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28453.775522 secs 2023-03-09T22:52:22.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:22 smithi181 ceph-mon[94189]: pgmap v15687: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T22:52:22.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28453.775522 secs 2023-03-09T22:52:22.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:22 smithi038 ceph-mon[92422]: pgmap v15687: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T22:52:23.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28453.775522 secs 2023-03-09T22:52:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28915.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:52:24.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:24 smithi122 ceph-mon[93869]: pgmap v15688: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T22:52:24.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:24 smithi181 ceph-mon[94189]: pgmap v15688: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T22:52:24.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:24 smithi038 ceph-mon[92422]: pgmap v15688: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T22:52:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28456.392138 secs 2023-03-09T22:52:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28456.392138 secs 2023-03-09T22:52:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28456.392138 secs 2023-03-09T22:52:26.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:26 smithi122 ceph-mon[93869]: pgmap v15689: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T22:52:26.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:26 smithi181 ceph-mon[94189]: pgmap v15689: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T22:52:26.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:26 smithi038 ceph-mon[92422]: pgmap v15689: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T22:52:27.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28458.775660 secs 2023-03-09T22:52:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28458.775660 secs 2023-03-09T22:52:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28458.775660 secs 2023-03-09T22:52:28.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28920.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:52:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:28 smithi122 ceph-mon[93869]: pgmap v15690: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T22:52:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:28 smithi181 ceph-mon[94189]: pgmap v15690: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T22:52:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:28 smithi038 ceph-mon[92422]: pgmap v15690: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T22:52:29.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:52:29.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:52:29.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:52:29.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:52:29.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:52:29.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:52:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:52:30.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:52:30.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:52:31.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:30 smithi122 ceph-mon[93869]: pgmap v15691: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T22:52:31.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:30 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:52:31.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:30 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:52:31.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28461.392199 secs 2023-03-09T22:52:31.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:30 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:52:31.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:30 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:52:31.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:30 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:52:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:30 smithi181 ceph-mon[94189]: pgmap v15691: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T22:52:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:30 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:52:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:30 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:52:31.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28461.392199 secs 2023-03-09T22:52:31.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:30 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:52:31.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:30 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:52:31.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:30 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:52:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:30 smithi038 ceph-mon[92422]: pgmap v15691: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T22:52:31.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:30 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:52:31.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:30 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:52:31.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28461.392199 secs 2023-03-09T22:52:31.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:30 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:52:31.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:30 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:52:31.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:30 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:52:33.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:32 smithi122 ceph-mon[93869]: pgmap v15692: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T22:52:33.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28463.775747 secs 2023-03-09T22:52:33.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:32 smithi181 ceph-mon[94189]: pgmap v15692: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T22:52:33.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28463.775747 secs 2023-03-09T22:52:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:32 smithi038 ceph-mon[92422]: pgmap v15692: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T22:52:33.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28463.775747 secs 2023-03-09T22:52:33.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28925.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:52:35.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:34 smithi122 ceph-mon[93869]: pgmap v15693: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T22:52:35.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:34 smithi181 ceph-mon[94189]: pgmap v15693: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T22:52:35.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:34 smithi038 ceph-mon[92422]: pgmap v15693: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T22:52:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28466.392263 secs 2023-03-09T22:52:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28466.392263 secs 2023-03-09T22:52:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28466.392263 secs 2023-03-09T22:52:37.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:36 smithi122 ceph-mon[93869]: pgmap v15694: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T22:52:37.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:36 smithi181 ceph-mon[94189]: pgmap v15694: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T22:52:37.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:36 smithi038 ceph-mon[92422]: pgmap v15694: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T22:52:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28468.775897 secs 2023-03-09T22:52:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28468.775897 secs 2023-03-09T22:52:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28468.775897 secs 2023-03-09T22:52:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28930.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:52:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:38 smithi122 ceph-mon[93869]: pgmap v15695: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 9 op/s 2023-03-09T22:52:39.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:38 smithi181 ceph-mon[94189]: pgmap v15695: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 9 op/s 2023-03-09T22:52:39.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:38 smithi038 ceph-mon[92422]: pgmap v15695: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 9 op/s 2023-03-09T22:52:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:40 smithi122 ceph-mon[93869]: pgmap v15696: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-09T22:52:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28471.392334 secs 2023-03-09T22:52:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:40 smithi181 ceph-mon[94189]: pgmap v15696: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-09T22:52:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28471.392334 secs 2023-03-09T22:52:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:40 smithi038 ceph-mon[92422]: pgmap v15696: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-09T22:52:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28471.392334 secs 2023-03-09T22:52:43.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:42 smithi122 ceph-mon[93869]: pgmap v15697: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 179 KiB/s wr, 9 op/s 2023-03-09T22:52:43.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:42 smithi181 ceph-mon[94189]: pgmap v15697: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 179 KiB/s wr, 9 op/s 2023-03-09T22:52:43.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:42 smithi038 ceph-mon[92422]: pgmap v15697: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 179 KiB/s wr, 9 op/s 2023-03-09T22:52:43.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28935.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:52:45.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:44 smithi122 ceph-mon[93869]: pgmap v15698: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 179 KiB/s wr, 7 op/s 2023-03-09T22:52:45.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:44 smithi181 ceph-mon[94189]: pgmap v15698: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 179 KiB/s wr, 7 op/s 2023-03-09T22:52:45.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:44 smithi038 ceph-mon[92422]: pgmap v15698: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 179 KiB/s wr, 7 op/s 2023-03-09T22:52:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28476.392393 secs 2023-03-09T22:52:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:45 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:52:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28476.392393 secs 2023-03-09T22:52:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:45 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:52:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28476.392393 secs 2023-03-09T22:52:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:45 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:52:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:46 smithi122 ceph-mon[93869]: pgmap v15699: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 374 KiB/s wr, 7 op/s 2023-03-09T22:52:47.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:46 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:52:47.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:46 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:52:47.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:46 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:52:47.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:46 smithi181 ceph-mon[94189]: pgmap v15699: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 374 KiB/s wr, 7 op/s 2023-03-09T22:52:47.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:46 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:52:47.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:46 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:52:47.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:46 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:52:47.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:46 smithi038 ceph-mon[92422]: pgmap v15699: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 374 KiB/s wr, 7 op/s 2023-03-09T22:52:47.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:46 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:52:47.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:46 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:52:47.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:46 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:52:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:52:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:52:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:52:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:52:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:52:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:52:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28940.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:52:49.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:48 smithi122 ceph-mon[93869]: pgmap v15700: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 522 KiB/s rd, 397 KiB/s wr, 10 op/s 2023-03-09T22:52:49.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:52:49.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:52:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:48 smithi181 ceph-mon[94189]: pgmap v15700: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 522 KiB/s rd, 397 KiB/s wr, 10 op/s 2023-03-09T22:52:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:52:49.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:52:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:48 smithi038 ceph-mon[92422]: pgmap v15700: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 522 KiB/s rd, 397 KiB/s wr, 10 op/s 2023-03-09T22:52:49.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:52:49.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:52:51.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:50 smithi122 ceph-mon[93869]: pgmap v15701: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 544 KiB/s wr, 9 op/s 2023-03-09T22:52:51.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28481.392476 secs 2023-03-09T22:52:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:50 smithi181 ceph-mon[94189]: pgmap v15701: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 544 KiB/s wr, 9 op/s 2023-03-09T22:52:51.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28481.392476 secs 2023-03-09T22:52:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:50 smithi038 ceph-mon[92422]: pgmap v15701: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 544 KiB/s wr, 9 op/s 2023-03-09T22:52:51.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28481.392476 secs 2023-03-09T22:52:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:52 smithi122 ceph-mon[93869]: pgmap v15702: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 690 KiB/s wr, 12 op/s 2023-03-09T22:52:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:52 smithi181 ceph-mon[94189]: pgmap v15702: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 690 KiB/s wr, 12 op/s 2023-03-09T22:52:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:52 smithi038 ceph-mon[92422]: pgmap v15702: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 690 KiB/s wr, 12 op/s 2023-03-09T22:52:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28945.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:52:55.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:54 smithi122 ceph-mon[93869]: pgmap v15703: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 511 KiB/s wr, 10 op/s 2023-03-09T22:52:55.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:54 smithi181 ceph-mon[94189]: pgmap v15703: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 511 KiB/s wr, 10 op/s 2023-03-09T22:52:55.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:54 smithi038 ceph-mon[92422]: pgmap v15703: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 511 KiB/s wr, 10 op/s 2023-03-09T22:52:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:52:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:52:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28486.392538 secs 2023-03-09T22:52:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:52:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:52:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28486.392538 secs 2023-03-09T22:52:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:52:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:52:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28486.392538 secs 2023-03-09T22:52:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:56 smithi122 ceph-mon[93869]: pgmap v15704: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 511 KiB/s wr, 10 op/s 2023-03-09T22:52:57.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:52:57.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:52:57.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:56 smithi181 ceph-mon[94189]: pgmap v15704: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 511 KiB/s wr, 10 op/s 2023-03-09T22:52:57.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:52:57.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:52:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:56 smithi038 ceph-mon[92422]: pgmap v15704: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 511 KiB/s wr, 10 op/s 2023-03-09T22:52:57.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:52:57.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:52:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28950.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:52:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:58 smithi122 ceph-mon[93869]: pgmap v15705: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 561 KiB/s wr, 14 op/s 2023-03-09T22:52:59.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:58 smithi181 ceph-mon[94189]: pgmap v15705: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 561 KiB/s wr, 14 op/s 2023-03-09T22:52:59.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:58 smithi038 ceph-mon[92422]: pgmap v15705: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 561 KiB/s wr, 14 op/s 2023-03-09T22:53:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:53:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:52:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:53:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:53:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:52:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:53:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:53:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:52:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:53:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:00 smithi122 ceph-mon[93869]: pgmap v15706: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 718 KiB/s wr, 12 op/s 2023-03-09T22:53:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:53:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:53:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28491.392600 secs 2023-03-09T22:53:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:00 smithi181 ceph-mon[94189]: pgmap v15706: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 718 KiB/s wr, 12 op/s 2023-03-09T22:53:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:53:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:53:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28491.392600 secs 2023-03-09T22:53:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:00 smithi038 ceph-mon[92422]: pgmap v15706: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 718 KiB/s wr, 12 op/s 2023-03-09T22:53:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:53:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:53:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28491.392600 secs 2023-03-09T22:53:03.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:02 smithi122 ceph-mon[93869]: pgmap v15707: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 753 KiB/s rd, 619 KiB/s wr, 14 op/s 2023-03-09T22:53:03.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28493.776525 secs 2023-03-09T22:53:03.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:02 smithi181 ceph-mon[94189]: pgmap v15707: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 753 KiB/s rd, 619 KiB/s wr, 14 op/s 2023-03-09T22:53:03.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28493.776525 secs 2023-03-09T22:53:03.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:02 smithi038 ceph-mon[92422]: pgmap v15707: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 753 KiB/s rd, 619 KiB/s wr, 14 op/s 2023-03-09T22:53:03.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28493.776525 secs 2023-03-09T22:53:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28955.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:53:05.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:04 smithi122 ceph-mon[93869]: pgmap v15708: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 773 KiB/s rd, 657 KiB/s wr, 24 op/s 2023-03-09T22:53:05.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:04 smithi181 ceph-mon[94189]: pgmap v15708: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 773 KiB/s rd, 657 KiB/s wr, 24 op/s 2023-03-09T22:53:05.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:04 smithi038 ceph-mon[92422]: pgmap v15708: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 773 KiB/s rd, 657 KiB/s wr, 24 op/s 2023-03-09T22:53:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:05 smithi122 ceph-mon[93869]: osdmap e1197: 12 total, 12 up, 12 in 2023-03-09T22:53:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28496.392663 secs 2023-03-09T22:53:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:05 smithi181 ceph-mon[94189]: osdmap e1197: 12 total, 12 up, 12 in 2023-03-09T22:53:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28496.392663 secs 2023-03-09T22:53:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:05 smithi038 ceph-mon[92422]: osdmap e1197: 12 total, 12 up, 12 in 2023-03-09T22:53:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28496.392663 secs 2023-03-09T22:53:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:06 smithi122 ceph-mon[93869]: pgmap v15710: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 973 KiB/s rd, 981 KiB/s wr, 29 op/s 2023-03-09T22:53:07.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:06 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:53:07.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:06 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:53:07.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:53:07.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:06 smithi181 ceph-mon[94189]: pgmap v15710: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 973 KiB/s rd, 981 KiB/s wr, 29 op/s 2023-03-09T22:53:07.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:06 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:53:07.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:06 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:53:07.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:53:07.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:06 smithi038 ceph-mon[92422]: pgmap v15710: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 973 KiB/s rd, 981 KiB/s wr, 29 op/s 2023-03-09T22:53:07.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:06 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:53:07.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:06 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:53:07.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:53:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:53:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:53:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28498.776672 secs 2023-03-09T22:53:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:53:08.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:53:08.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28498.776672 secs 2023-03-09T22:53:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:53:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:53:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28498.776672 secs 2023-03-09T22:53:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28960.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:53:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:08 smithi122 ceph-mon[93869]: pgmap v15711: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 697 KiB/s rd, 738 KiB/s wr, 30 op/s 2023-03-09T22:53:09.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:53:09.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:53:09.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:08 smithi181 ceph-mon[94189]: pgmap v15711: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 697 KiB/s rd, 738 KiB/s wr, 30 op/s 2023-03-09T22:53:09.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:53:09.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:53:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:08 smithi038 ceph-mon[92422]: pgmap v15711: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 697 KiB/s rd, 738 KiB/s wr, 30 op/s 2023-03-09T22:53:09.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:53:09.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:53:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:10 smithi122 ceph-mon[93869]: pgmap v15712: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 523 KiB/s wr, 29 op/s 2023-03-09T22:53:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28501.392721 secs 2023-03-09T22:53:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:10 smithi181 ceph-mon[94189]: pgmap v15712: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 523 KiB/s wr, 29 op/s 2023-03-09T22:53:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28501.392721 secs 2023-03-09T22:53:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:10 smithi038 ceph-mon[92422]: pgmap v15712: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 523 KiB/s wr, 29 op/s 2023-03-09T22:53:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28501.392721 secs 2023-03-09T22:53:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:12 smithi122 ceph-mon[93869]: pgmap v15713: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 567 KiB/s rd, 608 KiB/s wr, 29 op/s 2023-03-09T22:53:13.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28503.776792 secs 2023-03-09T22:53:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:12 smithi181 ceph-mon[94189]: pgmap v15713: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 567 KiB/s rd, 608 KiB/s wr, 29 op/s 2023-03-09T22:53:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28503.776792 secs 2023-03-09T22:53:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:12 smithi038 ceph-mon[92422]: pgmap v15713: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 567 KiB/s rd, 608 KiB/s wr, 29 op/s 2023-03-09T22:53:13.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28503.776792 secs 2023-03-09T22:53:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28965.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:53:15.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:14 smithi122 ceph-mon[93869]: pgmap v15714: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 387 KiB/s wr, 13 op/s 2023-03-09T22:53:15.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:14 smithi181 ceph-mon[94189]: pgmap v15714: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 387 KiB/s wr, 13 op/s 2023-03-09T22:53:15.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:14 smithi038 ceph-mon[92422]: pgmap v15714: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 387 KiB/s wr, 13 op/s 2023-03-09T22:53:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28506.392785 secs 2023-03-09T22:53:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28506.392785 secs 2023-03-09T22:53:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28506.392785 secs 2023-03-09T22:53:17.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:16 smithi122 ceph-mon[93869]: pgmap v15715: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 360 KiB/s wr, 12 op/s 2023-03-09T22:53:17.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:16 smithi122 ceph-mon[93869]: osdmap e1198: 12 total, 12 up, 12 in 2023-03-09T22:53:17.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:16 smithi181 ceph-mon[94189]: pgmap v15715: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 360 KiB/s wr, 12 op/s 2023-03-09T22:53:17.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:16 smithi181 ceph-mon[94189]: osdmap e1198: 12 total, 12 up, 12 in 2023-03-09T22:53:17.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:16 smithi038 ceph-mon[92422]: pgmap v15715: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 360 KiB/s wr, 12 op/s 2023-03-09T22:53:17.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:16 smithi038 ceph-mon[92422]: osdmap e1198: 12 total, 12 up, 12 in 2023-03-09T22:53:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28508.776909 secs 2023-03-09T22:53:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28508.776909 secs 2023-03-09T22:53:18.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28508.776909 secs 2023-03-09T22:53:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28970.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:53:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:18 smithi122 ceph-mon[93869]: pgmap v15717: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 143 KiB/s wr, 8 op/s 2023-03-09T22:53:19.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:18 smithi181 ceph-mon[94189]: pgmap v15717: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 143 KiB/s wr, 8 op/s 2023-03-09T22:53:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:18 smithi038 ceph-mon[92422]: pgmap v15717: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 143 KiB/s wr, 8 op/s 2023-03-09T22:53:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:20 smithi181 ceph-mon[94189]: pgmap v15718: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 143 KiB/s wr, 7 op/s 2023-03-09T22:53:21.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28511.392883 secs 2023-03-09T22:53:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:20 smithi038 ceph-mon[92422]: pgmap v15718: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 143 KiB/s wr, 7 op/s 2023-03-09T22:53:21.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28511.392883 secs 2023-03-09T22:53:21.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:20 smithi122 ceph-mon[93869]: pgmap v15718: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 143 KiB/s wr, 7 op/s 2023-03-09T22:53:21.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28511.392883 secs 2023-03-09T22:53:23.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:22 smithi038 ceph-mon[92422]: pgmap v15719: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 100 KiB/s wr, 6 op/s 2023-03-09T22:53:23.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28513.777063 secs 2023-03-09T22:53:23.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:22 smithi122 ceph-mon[93869]: pgmap v15719: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 100 KiB/s wr, 6 op/s 2023-03-09T22:53:23.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28513.777063 secs 2023-03-09T22:53:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:22 smithi181 ceph-mon[94189]: pgmap v15719: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 100 KiB/s wr, 6 op/s 2023-03-09T22:53:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28513.777063 secs 2023-03-09T22:53:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28975.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:53:25.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:24 smithi038 ceph-mon[92422]: pgmap v15720: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 100 KiB/s wr, 6 op/s 2023-03-09T22:53:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:24 smithi122 ceph-mon[93869]: pgmap v15720: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 100 KiB/s wr, 6 op/s 2023-03-09T22:53:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:24 smithi181 ceph-mon[94189]: pgmap v15720: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 100 KiB/s wr, 6 op/s 2023-03-09T22:53:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28516.392946 secs 2023-03-09T22:53:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28516.392946 secs 2023-03-09T22:53:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28516.392946 secs 2023-03-09T22:53:27.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:26 smithi038 ceph-mon[92422]: pgmap v15721: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 100 KiB/s wr, 6 op/s 2023-03-09T22:53:27.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:26 smithi122 ceph-mon[93869]: pgmap v15721: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 100 KiB/s wr, 6 op/s 2023-03-09T22:53:27.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:26 smithi181 ceph-mon[94189]: pgmap v15721: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 100 KiB/s wr, 6 op/s 2023-03-09T22:53:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28518.777195 secs 2023-03-09T22:53:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28518.777195 secs 2023-03-09T22:53:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28518.777195 secs 2023-03-09T22:53:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28980.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:53:29.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:29 smithi122 ceph-mon[93869]: pgmap v15722: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 92 KiB/s wr, 6 op/s 2023-03-09T22:53:29.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:29 smithi181 ceph-mon[94189]: pgmap v15722: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 92 KiB/s wr, 6 op/s 2023-03-09T22:53:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:29 smithi038 ceph-mon[92422]: pgmap v15722: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 92 KiB/s wr, 6 op/s 2023-03-09T22:53:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:31 smithi122 ceph-mon[93869]: pgmap v15723: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 83 KiB/s wr, 5 op/s 2023-03-09T22:53:31.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28521.393010 secs 2023-03-09T22:53:31.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:53:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:31 smithi181 ceph-mon[94189]: pgmap v15723: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 83 KiB/s wr, 5 op/s 2023-03-09T22:53:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28521.393010 secs 2023-03-09T22:53:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:53:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:31 smithi038 ceph-mon[92422]: pgmap v15723: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 83 KiB/s wr, 5 op/s 2023-03-09T22:53:31.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28521.393010 secs 2023-03-09T22:53:31.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:53:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:53:32.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:53:32.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:53:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:53:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:53:32.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:53:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:53:32.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:53:32.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:53:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:33 smithi122 ceph-mon[93869]: pgmap v15724: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 83 KiB/s wr, 8 op/s 2023-03-09T22:53:33.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28523.777312 secs 2023-03-09T22:53:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:33 smithi181 ceph-mon[94189]: pgmap v15724: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 83 KiB/s wr, 8 op/s 2023-03-09T22:53:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28523.777312 secs 2023-03-09T22:53:33.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:33 smithi038 ceph-mon[92422]: pgmap v15724: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 83 KiB/s wr, 8 op/s 2023-03-09T22:53:33.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28523.777312 secs 2023-03-09T22:53:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28985.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:53:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:35 smithi122 ceph-mon[93869]: pgmap v15725: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T22:53:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28526.393101 secs 2023-03-09T22:53:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:35 smithi181 ceph-mon[94189]: pgmap v15725: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T22:53:35.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28526.393101 secs 2023-03-09T22:53:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:35 smithi038 ceph-mon[92422]: pgmap v15725: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T22:53:35.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28526.393101 secs 2023-03-09T22:53:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:37 smithi122 ceph-mon[93869]: pgmap v15726: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T22:53:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:37 smithi181 ceph-mon[94189]: pgmap v15726: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T22:53:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:37 smithi038 ceph-mon[92422]: pgmap v15726: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T22:53:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28528.777406 secs 2023-03-09T22:53:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28528.777406 secs 2023-03-09T22:53:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28528.777406 secs 2023-03-09T22:53:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28990.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:53:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:39 smithi122 ceph-mon[93869]: pgmap v15727: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 8 op/s 2023-03-09T22:53:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:39 smithi181 ceph-mon[94189]: pgmap v15727: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 8 op/s 2023-03-09T22:53:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:39 smithi038 ceph-mon[92422]: pgmap v15727: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 8 op/s 2023-03-09T22:53:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28531.393170 secs 2023-03-09T22:53:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28531.393170 secs 2023-03-09T22:53:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28531.393170 secs 2023-03-09T22:53:41.732 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:41 smithi122 ceph-mon[93869]: pgmap v15728: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-09T22:53:41.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:41 smithi181 ceph-mon[94189]: pgmap v15728: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-09T22:53:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:41 smithi038 ceph-mon[92422]: pgmap v15728: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-09T22:53:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:43 smithi122 ceph-mon[93869]: pgmap v15729: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 9 op/s 2023-03-09T22:53:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:43 smithi181 ceph-mon[94189]: pgmap v15729: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 9 op/s 2023-03-09T22:53:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:43 smithi038 ceph-mon[92422]: pgmap v15729: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 9 op/s 2023-03-09T22:53:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 28995.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:53:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:45 smithi122 ceph-mon[93869]: pgmap v15730: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 6 op/s 2023-03-09T22:53:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28536.393232 secs 2023-03-09T22:53:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:45 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:53:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:45 smithi181 ceph-mon[94189]: pgmap v15730: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 6 op/s 2023-03-09T22:53:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28536.393232 secs 2023-03-09T22:53:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:45 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:53:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:45 smithi038 ceph-mon[92422]: pgmap v15730: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 6 op/s 2023-03-09T22:53:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28536.393232 secs 2023-03-09T22:53:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:45 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:53:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:46 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:53:46.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:46 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:53:46.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:46 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:53:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:46 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:53:46.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:46 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:53:46.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:46 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:53:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:46 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:53:46.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:46 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:53:46.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:46 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:53:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:47 smithi122 ceph-mon[93869]: pgmap v15731: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 296 KiB/s wr, 8 op/s 2023-03-09T22:53:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:47 smithi181 ceph-mon[94189]: pgmap v15731: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 296 KiB/s wr, 8 op/s 2023-03-09T22:53:47.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:47 smithi038 ceph-mon[92422]: pgmap v15731: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 296 KiB/s wr, 8 op/s 2023-03-09T22:53:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:53:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:53:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:53:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:53:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:53:48.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:53:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29000.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:53:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:49 smithi122 ceph-mon[93869]: pgmap v15732: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 319 KiB/s wr, 10 op/s 2023-03-09T22:53:49.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:53:49.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:53:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:49 smithi181 ceph-mon[94189]: pgmap v15732: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 319 KiB/s wr, 10 op/s 2023-03-09T22:53:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:53:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:53:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:49 smithi038 ceph-mon[92422]: pgmap v15732: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 319 KiB/s wr, 10 op/s 2023-03-09T22:53:49.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:53:49.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:53:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28541.393289 secs 2023-03-09T22:53:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28541.393289 secs 2023-03-09T22:53:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28541.393289 secs 2023-03-09T22:53:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:51 smithi122 ceph-mon[93869]: pgmap v15733: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 321 KiB/s rd, 320 KiB/s wr, 8 op/s 2023-03-09T22:53:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:51 smithi181 ceph-mon[94189]: pgmap v15733: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 321 KiB/s rd, 320 KiB/s wr, 8 op/s 2023-03-09T22:53:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:51 smithi038 ceph-mon[92422]: pgmap v15733: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 321 KiB/s rd, 320 KiB/s wr, 8 op/s 2023-03-09T22:53:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:53 smithi122 ceph-mon[93869]: pgmap v15734: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 343 KiB/s wr, 11 op/s 2023-03-09T22:53:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:53 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 28543.777715 secs 2023-03-09T22:53:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:53 smithi181 ceph-mon[94189]: pgmap v15734: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 343 KiB/s wr, 11 op/s 2023-03-09T22:53:53.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:53 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 28543.777715 secs 2023-03-09T22:53:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:53 smithi038 ceph-mon[92422]: pgmap v15734: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 343 KiB/s wr, 11 op/s 2023-03-09T22:53:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:53 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 28543.777715 secs 2023-03-09T22:53:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29005.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:53:54.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:54 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:53:54.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:54 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:53:54.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:54 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:53:54.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:54 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:53:54.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:54 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:53:54.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:54 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:53:54.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:54 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:53:54.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:54 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:53:54.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:54 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:53:55.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:55 smithi122 ceph-mon[93869]: pgmap v15735: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 342 KiB/s wr, 8 op/s 2023-03-09T22:53:55.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28546.393351 secs 2023-03-09T22:53:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:55 smithi181 ceph-mon[94189]: pgmap v15735: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 342 KiB/s wr, 8 op/s 2023-03-09T22:53:55.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28546.393351 secs 2023-03-09T22:53:55.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:55 smithi038 ceph-mon[92422]: pgmap v15735: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 342 KiB/s wr, 8 op/s 2023-03-09T22:53:55.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28546.393351 secs 2023-03-09T22:53:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:53:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:53:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:53:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:53:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:53:56.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:53:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:57 smithi122 ceph-mon[93869]: pgmap v15736: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 720 KiB/s rd, 684 KiB/s wr, 10 op/s 2023-03-09T22:53:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:53:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:53:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:57 smithi181 ceph-mon[94189]: pgmap v15736: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 720 KiB/s rd, 684 KiB/s wr, 10 op/s 2023-03-09T22:53:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:53:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:53:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:57 smithi038 ceph-mon[92422]: pgmap v15736: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 720 KiB/s rd, 684 KiB/s wr, 10 op/s 2023-03-09T22:53:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:53:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:53:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:58 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 28548.777831 secs 2023-03-09T22:53:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:58 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 28548.777831 secs 2023-03-09T22:53:58.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:58 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 28548.777831 secs 2023-03-09T22:53:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29010.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:53:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:53:59 smithi122 ceph-mon[93869]: pgmap v15737: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 741 KiB/s rd, 703 KiB/s wr, 13 op/s 2023-03-09T22:53:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:53:59 smithi181 ceph-mon[94189]: pgmap v15737: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 741 KiB/s rd, 703 KiB/s wr, 13 op/s 2023-03-09T22:53:59.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:53:59 smithi038 ceph-mon[92422]: pgmap v15737: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 741 KiB/s rd, 703 KiB/s wr, 13 op/s 2023-03-09T22:54:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28551.393412 secs 2023-03-09T22:54:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28551.393412 secs 2023-03-09T22:54:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28551.393412 secs 2023-03-09T22:54:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:01 smithi122 ceph-mon[93869]: pgmap v15738: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 717 KiB/s rd, 681 KiB/s wr, 11 op/s 2023-03-09T22:54:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:01 smithi181 ceph-mon[94189]: pgmap v15738: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 717 KiB/s rd, 681 KiB/s wr, 11 op/s 2023-03-09T22:54:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:01 smithi038 ceph-mon[92422]: pgmap v15738: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 717 KiB/s rd, 681 KiB/s wr, 11 op/s 2023-03-09T22:54:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:03 smithi122 ceph-mon[93869]: pgmap v15739: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 794 KiB/s rd, 757 KiB/s wr, 15 op/s 2023-03-09T22:54:03.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28553.777948 secs 2023-03-09T22:54:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:03 smithi181 ceph-mon[94189]: pgmap v15739: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 794 KiB/s rd, 757 KiB/s wr, 15 op/s 2023-03-09T22:54:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28553.777948 secs 2023-03-09T22:54:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:03 smithi038 ceph-mon[92422]: pgmap v15739: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 794 KiB/s rd, 757 KiB/s wr, 15 op/s 2023-03-09T22:54:03.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28553.777948 secs 2023-03-09T22:54:03.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29015.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:54:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:05 smithi038 ceph-mon[92422]: pgmap v15740: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 753 KiB/s rd, 756 KiB/s wr, 24 op/s 2023-03-09T22:54:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:05 smithi038 ceph-mon[92422]: osdmap e1199: 12 total, 12 up, 12 in 2023-03-09T22:54:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28556.393471 secs 2023-03-09T22:54:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:05 smithi122 ceph-mon[93869]: pgmap v15740: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 753 KiB/s rd, 756 KiB/s wr, 24 op/s 2023-03-09T22:54:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:05 smithi122 ceph-mon[93869]: osdmap e1199: 12 total, 12 up, 12 in 2023-03-09T22:54:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28556.393471 secs 2023-03-09T22:54:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:05 smithi181 ceph-mon[94189]: pgmap v15740: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 753 KiB/s rd, 756 KiB/s wr, 24 op/s 2023-03-09T22:54:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:05 smithi181 ceph-mon[94189]: osdmap e1199: 12 total, 12 up, 12 in 2023-03-09T22:54:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28556.393471 secs 2023-03-09T22:54:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:06 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:54:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:54:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:06 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:54:06.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:54:06.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:06 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:54:06.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:54:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:07 smithi038 ceph-mon[92422]: pgmap v15742: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 793 KiB/s rd, 810 KiB/s wr, 27 op/s 2023-03-09T22:54:07.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:54:07.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:54:07.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28558.778089 secs 2023-03-09T22:54:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:07 smithi122 ceph-mon[93869]: pgmap v15742: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 793 KiB/s rd, 810 KiB/s wr, 27 op/s 2023-03-09T22:54:07.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:54:07.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:54:07.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28558.778089 secs 2023-03-09T22:54:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:07 smithi181 ceph-mon[94189]: pgmap v15742: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 793 KiB/s rd, 810 KiB/s wr, 27 op/s 2023-03-09T22:54:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:54:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:54:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28558.778089 secs 2023-03-09T22:54:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29020.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:54:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:09 smithi122 ceph-mon[93869]: pgmap v15743: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 483 KiB/s wr, 29 op/s 2023-03-09T22:54:09.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:54:09.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:54:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:09 smithi181 ceph-mon[94189]: pgmap v15743: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 483 KiB/s wr, 29 op/s 2023-03-09T22:54:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:54:09.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:54:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:09 smithi038 ceph-mon[92422]: pgmap v15743: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 483 KiB/s wr, 29 op/s 2023-03-09T22:54:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:54:10.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:54:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28561.393534 secs 2023-03-09T22:54:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28561.393534 secs 2023-03-09T22:54:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28561.393534 secs 2023-03-09T22:54:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:11 smithi122 ceph-mon[93869]: pgmap v15744: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 482 KiB/s wr, 29 op/s 2023-03-09T22:54:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:11 smithi181 ceph-mon[94189]: pgmap v15744: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 482 KiB/s wr, 29 op/s 2023-03-09T22:54:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:11 smithi038 ceph-mon[92422]: pgmap v15744: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 482 KiB/s wr, 29 op/s 2023-03-09T22:54:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:12 smithi122 ceph-mon[93869]: pgmap v15745: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 392 KiB/s wr, 27 op/s 2023-03-09T22:54:12.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:12 smithi122 ceph-mon[93869]: osdmap e1200: 12 total, 12 up, 12 in 2023-03-09T22:54:12.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28563.778213 secs 2023-03-09T22:54:12.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:12 smithi181 ceph-mon[94189]: pgmap v15745: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 392 KiB/s wr, 27 op/s 2023-03-09T22:54:12.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:12 smithi181 ceph-mon[94189]: osdmap e1200: 12 total, 12 up, 12 in 2023-03-09T22:54:12.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28563.778213 secs 2023-03-09T22:54:13.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:12 smithi038 ceph-mon[92422]: pgmap v15745: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 392 KiB/s wr, 27 op/s 2023-03-09T22:54:13.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:12 smithi038 ceph-mon[92422]: osdmap e1200: 12 total, 12 up, 12 in 2023-03-09T22:54:13.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28563.778213 secs 2023-03-09T22:54:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29025.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:54:14.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:14 smithi122 ceph-mon[93869]: pgmap v15747: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 107 KiB/s wr, 12 op/s 2023-03-09T22:54:14.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:14 smithi181 ceph-mon[94189]: pgmap v15747: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 107 KiB/s wr, 12 op/s 2023-03-09T22:54:14.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:14 smithi038 ceph-mon[92422]: pgmap v15747: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 107 KiB/s wr, 12 op/s 2023-03-09T22:54:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28566.393597 secs 2023-03-09T22:54:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28566.393597 secs 2023-03-09T22:54:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28566.393597 secs 2023-03-09T22:54:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:16 smithi038 ceph-mon[92422]: pgmap v15748: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 194 KiB/s wr, 12 op/s 2023-03-09T22:54:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:16 smithi122 ceph-mon[93869]: pgmap v15748: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 194 KiB/s wr, 12 op/s 2023-03-09T22:54:17.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:16 smithi181 ceph-mon[94189]: pgmap v15748: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 194 KiB/s wr, 12 op/s 2023-03-09T22:54:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28568.778307 secs 2023-03-09T22:54:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28568.778307 secs 2023-03-09T22:54:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28568.778307 secs 2023-03-09T22:54:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29030.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:54:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:18 smithi038 ceph-mon[92422]: pgmap v15749: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 143 KiB/s wr, 9 op/s 2023-03-09T22:54:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:18 smithi122 ceph-mon[93869]: pgmap v15749: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 143 KiB/s wr, 9 op/s 2023-03-09T22:54:19.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:18 smithi181 ceph-mon[94189]: pgmap v15749: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 143 KiB/s wr, 9 op/s 2023-03-09T22:54:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:20 smithi038 ceph-mon[92422]: pgmap v15750: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 143 KiB/s wr, 8 op/s 2023-03-09T22:54:21.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28571.393661 secs 2023-03-09T22:54:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:20 smithi122 ceph-mon[93869]: pgmap v15750: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 143 KiB/s wr, 8 op/s 2023-03-09T22:54:21.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28571.393661 secs 2023-03-09T22:54:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:20 smithi181 ceph-mon[94189]: pgmap v15750: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 143 KiB/s wr, 8 op/s 2023-03-09T22:54:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28571.393661 secs 2023-03-09T22:54:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:22 smithi122 ceph-mon[93869]: pgmap v15751: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 142 KiB/s wr, 8 op/s 2023-03-09T22:54:23.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28573.778426 secs 2023-03-09T22:54:23.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:22 smithi181 ceph-mon[94189]: pgmap v15751: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 142 KiB/s wr, 8 op/s 2023-03-09T22:54:23.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28573.778426 secs 2023-03-09T22:54:23.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:22 smithi038 ceph-mon[92422]: pgmap v15751: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 142 KiB/s wr, 8 op/s 2023-03-09T22:54:23.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28573.778426 secs 2023-03-09T22:54:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29035.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:54:25.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:24 smithi122 ceph-mon[93869]: pgmap v15752: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 119 KiB/s wr, 7 op/s 2023-03-09T22:54:25.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:24 smithi181 ceph-mon[94189]: pgmap v15752: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 119 KiB/s wr, 7 op/s 2023-03-09T22:54:25.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:24 smithi038 ceph-mon[92422]: pgmap v15752: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 119 KiB/s wr, 7 op/s 2023-03-09T22:54:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28576.393724 secs 2023-03-09T22:54:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28576.393724 secs 2023-03-09T22:54:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28576.393724 secs 2023-03-09T22:54:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:26 smithi122 ceph-mon[93869]: pgmap v15753: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 81 KiB/s wr, 7 op/s 2023-03-09T22:54:27.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:26 smithi181 ceph-mon[94189]: pgmap v15753: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 81 KiB/s wr, 7 op/s 2023-03-09T22:54:27.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:26 smithi038 ceph-mon[92422]: pgmap v15753: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 81 KiB/s wr, 7 op/s 2023-03-09T22:54:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28578.778536 secs 2023-03-09T22:54:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28578.778536 secs 2023-03-09T22:54:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28578.778536 secs 2023-03-09T22:54:28.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29040.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:54:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:28 smithi122 ceph-mon[93869]: pgmap v15754: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 9 op/s 2023-03-09T22:54:29.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:28 smithi181 ceph-mon[94189]: pgmap v15754: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 9 op/s 2023-03-09T22:54:29.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:28 smithi038 ceph-mon[92422]: pgmap v15754: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 9 op/s 2023-03-09T22:54:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:30 smithi122 ceph-mon[93869]: pgmap v15755: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T22:54:31.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28581.393810 secs 2023-03-09T22:54:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:30 smithi181 ceph-mon[94189]: pgmap v15755: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T22:54:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28581.393810 secs 2023-03-09T22:54:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:30 smithi038 ceph-mon[92422]: pgmap v15755: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T22:54:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28581.393810 secs 2023-03-09T22:54:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:54:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:54:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:54:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:32 smithi122 ceph-mon[93869]: pgmap v15756: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 9 op/s 2023-03-09T22:54:33.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:54:33.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:54:33.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:54:33.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28583.778643 secs 2023-03-09T22:54:33.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:32 smithi181 ceph-mon[94189]: pgmap v15756: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 9 op/s 2023-03-09T22:54:33.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:54:33.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:54:33.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:54:33.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28583.778643 secs 2023-03-09T22:54:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:32 smithi038 ceph-mon[92422]: pgmap v15756: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 9 op/s 2023-03-09T22:54:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:54:33.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:54:33.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:54:33.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28583.778643 secs 2023-03-09T22:54:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29045.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:54:35.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:34 smithi122 ceph-mon[93869]: pgmap v15757: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 7 op/s 2023-03-09T22:54:35.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:34 smithi181 ceph-mon[94189]: pgmap v15757: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 7 op/s 2023-03-09T22:54:35.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:34 smithi038 ceph-mon[92422]: pgmap v15757: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 7 op/s 2023-03-09T22:54:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28586.393871 secs 2023-03-09T22:54:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28586.393871 secs 2023-03-09T22:54:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28586.393871 secs 2023-03-09T22:54:37.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:36 smithi122 ceph-mon[93869]: pgmap v15758: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 597 B/s wr, 9 op/s 2023-03-09T22:54:37.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:54:37.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:54:37.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:36 smithi181 ceph-mon[94189]: pgmap v15758: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 597 B/s wr, 9 op/s 2023-03-09T22:54:37.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:54:37.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:54:37.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:36 smithi038 ceph-mon[92422]: pgmap v15758: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 597 B/s wr, 9 op/s 2023-03-09T22:54:37.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:54:37.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:54:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28588.778762 secs 2023-03-09T22:54:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28588.778762 secs 2023-03-09T22:54:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28588.778762 secs 2023-03-09T22:54:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29050.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:54:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:38 smithi122 ceph-mon[93869]: pgmap v15759: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 102 KiB/s wr, 10 op/s 2023-03-09T22:54:39.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:38 smithi181 ceph-mon[94189]: pgmap v15759: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 102 KiB/s wr, 10 op/s 2023-03-09T22:54:39.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:38 smithi038 ceph-mon[92422]: pgmap v15759: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 102 KiB/s wr, 10 op/s 2023-03-09T22:54:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:39 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:54:40.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:39 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:54:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:39 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:54:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:39 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:54:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:39 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:54:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:39 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:54:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:40 smithi122 ceph-mon[93869]: pgmap v15760: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 102 KiB/s wr, 8 op/s 2023-03-09T22:54:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28591.393943 secs 2023-03-09T22:54:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:40 smithi181 ceph-mon[94189]: pgmap v15760: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 102 KiB/s wr, 8 op/s 2023-03-09T22:54:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28591.393943 secs 2023-03-09T22:54:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:40 smithi038 ceph-mon[92422]: pgmap v15760: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 102 KiB/s wr, 8 op/s 2023-03-09T22:54:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28591.393943 secs 2023-03-09T22:54:43.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:42 smithi122 ceph-mon[93869]: pgmap v15761: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 102 KiB/s wr, 9 op/s 2023-03-09T22:54:43.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28593.778878 secs 2023-03-09T22:54:43.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:42 smithi181 ceph-mon[94189]: pgmap v15761: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 102 KiB/s wr, 9 op/s 2023-03-09T22:54:43.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28593.778878 secs 2023-03-09T22:54:43.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:42 smithi038 ceph-mon[92422]: pgmap v15761: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 102 KiB/s wr, 9 op/s 2023-03-09T22:54:43.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28593.778878 secs 2023-03-09T22:54:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29055.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:54:45.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:44 smithi122 ceph-mon[93869]: pgmap v15762: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 304 KiB/s wr, 7 op/s 2023-03-09T22:54:45.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:44 smithi181 ceph-mon[94189]: pgmap v15762: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 304 KiB/s wr, 7 op/s 2023-03-09T22:54:45.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:44 smithi038 ceph-mon[92422]: pgmap v15762: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 304 KiB/s wr, 7 op/s 2023-03-09T22:54:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28596.393983 secs 2023-03-09T22:54:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28596.393983 secs 2023-03-09T22:54:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28596.393983 secs 2023-03-09T22:54:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:46 smithi122 ceph-mon[93869]: pgmap v15763: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 304 KiB/s wr, 8 op/s 2023-03-09T22:54:47.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:46 smithi181 ceph-mon[94189]: pgmap v15763: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 304 KiB/s wr, 8 op/s 2023-03-09T22:54:47.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:46 smithi038 ceph-mon[92422]: pgmap v15763: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 304 KiB/s wr, 8 op/s 2023-03-09T22:54:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28598.779020 secs 2023-03-09T22:54:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28598.779020 secs 2023-03-09T22:54:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28598.779020 secs 2023-03-09T22:54:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29060.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:54:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:48 smithi122 ceph-mon[93869]: pgmap v15764: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 764 KiB/s rd, 359 KiB/s wr, 8 op/s 2023-03-09T22:54:49.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:54:49.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:54:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:48 smithi181 ceph-mon[94189]: pgmap v15764: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 764 KiB/s rd, 359 KiB/s wr, 8 op/s 2023-03-09T22:54:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:54:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:54:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:48 smithi038 ceph-mon[92422]: pgmap v15764: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 764 KiB/s rd, 359 KiB/s wr, 8 op/s 2023-03-09T22:54:49.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:54:49.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:54:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:50 smithi122 ceph-mon[93869]: pgmap v15765: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 662 KiB/s rd, 258 KiB/s wr, 6 op/s 2023-03-09T22:54:51.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28601.394042 secs 2023-03-09T22:54:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:50 smithi181 ceph-mon[94189]: pgmap v15765: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 662 KiB/s rd, 258 KiB/s wr, 6 op/s 2023-03-09T22:54:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28601.394042 secs 2023-03-09T22:54:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:50 smithi038 ceph-mon[92422]: pgmap v15765: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 662 KiB/s rd, 258 KiB/s wr, 6 op/s 2023-03-09T22:54:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28601.394042 secs 2023-03-09T22:54:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:54:52.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:54:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:54:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:54:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:54:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:54:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:52 smithi122 ceph-mon[93869]: pgmap v15766: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 967 KiB/s rd, 561 KiB/s wr, 10 op/s 2023-03-09T22:54:53.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28603.779137 secs 2023-03-09T22:54:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:52 smithi181 ceph-mon[94189]: pgmap v15766: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 967 KiB/s rd, 561 KiB/s wr, 10 op/s 2023-03-09T22:54:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28603.779137 secs 2023-03-09T22:54:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:52 smithi038 ceph-mon[92422]: pgmap v15766: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 967 KiB/s rd, 561 KiB/s wr, 10 op/s 2023-03-09T22:54:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28603.779137 secs 2023-03-09T22:54:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29065.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:54:55.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:54 smithi122 ceph-mon[93869]: pgmap v15767: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 971 KiB/s rd, 567 KiB/s wr, 9 op/s 2023-03-09T22:54:55.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:54 smithi181 ceph-mon[94189]: pgmap v15767: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 971 KiB/s rd, 567 KiB/s wr, 9 op/s 2023-03-09T22:54:55.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:54 smithi038 ceph-mon[92422]: pgmap v15767: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 971 KiB/s rd, 567 KiB/s wr, 9 op/s 2023-03-09T22:54:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28606.394103 secs 2023-03-09T22:54:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28606.394103 secs 2023-03-09T22:54:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28606.394103 secs 2023-03-09T22:54:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:56 smithi122 ceph-mon[93869]: pgmap v15768: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 689 KiB/s rd, 366 KiB/s wr, 9 op/s 2023-03-09T22:54:57.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:56 smithi181 ceph-mon[94189]: pgmap v15768: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 689 KiB/s rd, 366 KiB/s wr, 9 op/s 2023-03-09T22:54:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:56 smithi038 ceph-mon[92422]: pgmap v15768: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 689 KiB/s rd, 366 KiB/s wr, 9 op/s 2023-03-09T22:54:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28608.779249 secs 2023-03-09T22:54:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28608.779249 secs 2023-03-09T22:54:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28608.779249 secs 2023-03-09T22:54:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29070.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:54:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:58 smithi122 ceph-mon[93869]: pgmap v15769: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 856 KiB/s rd, 366 KiB/s wr, 10 op/s 2023-03-09T22:54:59.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:58 smithi181 ceph-mon[94189]: pgmap v15769: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 856 KiB/s rd, 366 KiB/s wr, 10 op/s 2023-03-09T22:54:59.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:58 smithi038 ceph-mon[92422]: pgmap v15769: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 856 KiB/s rd, 366 KiB/s wr, 10 op/s 2023-03-09T22:55:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:55:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:54:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:55:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:55:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:54:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:55:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:55:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:54:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:55:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:00 smithi122 ceph-mon[93869]: pgmap v15770: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 705 KiB/s rd, 679 KiB/s wr, 10 op/s 2023-03-09T22:55:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:55:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:55:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28611.394162 secs 2023-03-09T22:55:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:00 smithi181 ceph-mon[94189]: pgmap v15770: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 705 KiB/s rd, 679 KiB/s wr, 10 op/s 2023-03-09T22:55:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:55:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:55:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28611.394162 secs 2023-03-09T22:55:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:00 smithi038 ceph-mon[92422]: pgmap v15770: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 705 KiB/s rd, 679 KiB/s wr, 10 op/s 2023-03-09T22:55:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:55:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:55:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28611.394162 secs 2023-03-09T22:55:03.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:02 smithi122 ceph-mon[93869]: pgmap v15771: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 885 KiB/s rd, 857 KiB/s wr, 14 op/s 2023-03-09T22:55:03.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28613.779370 secs 2023-03-09T22:55:03.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:02 smithi181 ceph-mon[94189]: pgmap v15771: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 885 KiB/s rd, 857 KiB/s wr, 14 op/s 2023-03-09T22:55:03.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28613.779370 secs 2023-03-09T22:55:03.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:02 smithi038 ceph-mon[92422]: pgmap v15771: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 885 KiB/s rd, 857 KiB/s wr, 14 op/s 2023-03-09T22:55:03.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28613.779370 secs 2023-03-09T22:55:03.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29075.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:55:05.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:04 smithi122 ceph-mon[93869]: pgmap v15772: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 757 KiB/s rd, 598 KiB/s wr, 14 op/s 2023-03-09T22:55:05.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:04 smithi181 ceph-mon[94189]: pgmap v15772: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 757 KiB/s rd, 598 KiB/s wr, 14 op/s 2023-03-09T22:55:05.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:04 smithi038 ceph-mon[92422]: pgmap v15772: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 757 KiB/s rd, 598 KiB/s wr, 14 op/s 2023-03-09T22:55:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28616.394221 secs 2023-03-09T22:55:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28616.394221 secs 2023-03-09T22:55:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28616.394221 secs 2023-03-09T22:55:07.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:06 smithi038 ceph-mon[92422]: pgmap v15773: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 752 KiB/s rd, 592 KiB/s wr, 14 op/s 2023-03-09T22:55:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:06 smithi122 ceph-mon[93869]: pgmap v15773: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 752 KiB/s rd, 592 KiB/s wr, 14 op/s 2023-03-09T22:55:07.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:06 smithi181 ceph-mon[94189]: pgmap v15773: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 752 KiB/s rd, 592 KiB/s wr, 14 op/s 2023-03-09T22:55:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:55:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:55:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28618.779454 secs 2023-03-09T22:55:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:55:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:55:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28618.779454 secs 2023-03-09T22:55:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:55:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:55:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28618.779454 secs 2023-03-09T22:55:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29080.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:55:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:08 smithi038 ceph-mon[92422]: pgmap v15774: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 916 KiB/s rd, 777 KiB/s wr, 18 op/s 2023-03-09T22:55:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:08 smithi122 ceph-mon[93869]: pgmap v15774: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 916 KiB/s rd, 777 KiB/s wr, 18 op/s 2023-03-09T22:55:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:08 smithi181 ceph-mon[94189]: pgmap v15774: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 916 KiB/s rd, 777 KiB/s wr, 18 op/s 2023-03-09T22:55:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:11 smithi122 ceph-mon[93869]: pgmap v15775: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 968 KiB/s rd, 991 KiB/s wr, 28 op/s 2023-03-09T22:55:11.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28621.394281 secs 2023-03-09T22:55:11.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:11 smithi122 ceph-mon[93869]: osdmap e1201: 12 total, 12 up, 12 in 2023-03-09T22:55:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:11 smithi181 ceph-mon[94189]: pgmap v15775: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 968 KiB/s rd, 991 KiB/s wr, 28 op/s 2023-03-09T22:55:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28621.394281 secs 2023-03-09T22:55:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:11 smithi181 ceph-mon[94189]: osdmap e1201: 12 total, 12 up, 12 in 2023-03-09T22:55:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:11 smithi038 ceph-mon[92422]: pgmap v15775: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 968 KiB/s rd, 991 KiB/s wr, 28 op/s 2023-03-09T22:55:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28621.394281 secs 2023-03-09T22:55:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:11 smithi038 ceph-mon[92422]: osdmap e1201: 12 total, 12 up, 12 in 2023-03-09T22:55:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:55:12.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:55:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:55:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:55:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:55:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:55:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:13 smithi122 ceph-mon[93869]: pgmap v15777: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 675 KiB/s rd, 534 KiB/s wr, 30 op/s 2023-03-09T22:55:13.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:55:13.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:55:13.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28623.779574 secs 2023-03-09T22:55:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:13 smithi181 ceph-mon[94189]: pgmap v15777: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 675 KiB/s rd, 534 KiB/s wr, 30 op/s 2023-03-09T22:55:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:55:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:55:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28623.779574 secs 2023-03-09T22:55:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:13 smithi038 ceph-mon[92422]: pgmap v15777: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 675 KiB/s rd, 534 KiB/s wr, 30 op/s 2023-03-09T22:55:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:55:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:55:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28623.779574 secs 2023-03-09T22:55:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29085.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:55:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:15 smithi122 ceph-mon[93869]: pgmap v15778: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 606 KiB/s rd, 624 KiB/s wr, 28 op/s 2023-03-09T22:55:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:15 smithi181 ceph-mon[94189]: pgmap v15778: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 606 KiB/s rd, 624 KiB/s wr, 28 op/s 2023-03-09T22:55:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:15 smithi038 ceph-mon[92422]: pgmap v15778: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 606 KiB/s rd, 624 KiB/s wr, 28 op/s 2023-03-09T22:55:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28626.394364 secs 2023-03-09T22:55:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28626.394364 secs 2023-03-09T22:55:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28626.394364 secs 2023-03-09T22:55:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:17 smithi122 ceph-mon[93869]: pgmap v15779: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 606 KiB/s rd, 624 KiB/s wr, 27 op/s 2023-03-09T22:55:17.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:17 smithi122 ceph-mon[93869]: osdmap e1202: 12 total, 12 up, 12 in 2023-03-09T22:55:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:17 smithi181 ceph-mon[94189]: pgmap v15779: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 606 KiB/s rd, 624 KiB/s wr, 27 op/s 2023-03-09T22:55:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:17 smithi181 ceph-mon[94189]: osdmap e1202: 12 total, 12 up, 12 in 2023-03-09T22:55:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:17 smithi038 ceph-mon[92422]: pgmap v15779: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 606 KiB/s rd, 624 KiB/s wr, 27 op/s 2023-03-09T22:55:17.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:17 smithi038 ceph-mon[92422]: osdmap e1202: 12 total, 12 up, 12 in 2023-03-09T22:55:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28628.779701 secs 2023-03-09T22:55:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28628.779701 secs 2023-03-09T22:55:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28628.779701 secs 2023-03-09T22:55:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29090.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:55:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:19 smithi122 ceph-mon[93869]: pgmap v15781: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 226 KiB/s wr, 10 op/s 2023-03-09T22:55:19.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:19 smithi181 ceph-mon[94189]: pgmap v15781: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 226 KiB/s wr, 10 op/s 2023-03-09T22:55:19.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:19 smithi038 ceph-mon[92422]: pgmap v15781: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 226 KiB/s wr, 10 op/s 2023-03-09T22:55:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:21 smithi122 ceph-mon[93869]: pgmap v15782: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 253 KiB/s wr, 10 op/s 2023-03-09T22:55:21.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28631.394445 secs 2023-03-09T22:55:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:21 smithi181 ceph-mon[94189]: pgmap v15782: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 253 KiB/s wr, 10 op/s 2023-03-09T22:55:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28631.394445 secs 2023-03-09T22:55:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:21 smithi038 ceph-mon[92422]: pgmap v15782: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 253 KiB/s wr, 10 op/s 2023-03-09T22:55:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28631.394445 secs 2023-03-09T22:55:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:23 smithi122 ceph-mon[93869]: pgmap v15783: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 247 KiB/s rd, 245 KiB/s wr, 9 op/s 2023-03-09T22:55:23.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28633.779815 secs 2023-03-09T22:55:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:23 smithi181 ceph-mon[94189]: pgmap v15783: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 247 KiB/s rd, 245 KiB/s wr, 9 op/s 2023-03-09T22:55:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28633.779815 secs 2023-03-09T22:55:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:23 smithi038 ceph-mon[92422]: pgmap v15783: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 247 KiB/s rd, 245 KiB/s wr, 9 op/s 2023-03-09T22:55:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28633.779815 secs 2023-03-09T22:55:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29095.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:55:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:25 smithi122 ceph-mon[93869]: pgmap v15784: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 101 KiB/s wr, 8 op/s 2023-03-09T22:55:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:25 smithi181 ceph-mon[94189]: pgmap v15784: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 101 KiB/s wr, 8 op/s 2023-03-09T22:55:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:25 smithi038 ceph-mon[92422]: pgmap v15784: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 101 KiB/s wr, 8 op/s 2023-03-09T22:55:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28636.394504 secs 2023-03-09T22:55:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28636.394504 secs 2023-03-09T22:55:26.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28636.394504 secs 2023-03-09T22:55:27.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:27 smithi122 ceph-mon[93869]: pgmap v15785: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 101 KiB/s wr, 8 op/s 2023-03-09T22:55:27.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:27 smithi181 ceph-mon[94189]: pgmap v15785: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 101 KiB/s wr, 8 op/s 2023-03-09T22:55:27.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:27 smithi038 ceph-mon[92422]: pgmap v15785: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 101 KiB/s wr, 8 op/s 2023-03-09T22:55:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28638.779940 secs 2023-03-09T22:55:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28638.779940 secs 2023-03-09T22:55:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28638.779940 secs 2023-03-09T22:55:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29100.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:55:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:29 smithi122 ceph-mon[93869]: pgmap v15786: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 8 op/s 2023-03-09T22:55:29.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:29 smithi181 ceph-mon[94189]: pgmap v15786: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 8 op/s 2023-03-09T22:55:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:29 smithi038 ceph-mon[92422]: pgmap v15786: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 59 KiB/s wr, 8 op/s 2023-03-09T22:55:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:31 smithi122 ceph-mon[93869]: pgmap v15787: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 53 KiB/s wr, 7 op/s 2023-03-09T22:55:31.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28641.394561 secs 2023-03-09T22:55:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:31 smithi181 ceph-mon[94189]: pgmap v15787: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 53 KiB/s wr, 7 op/s 2023-03-09T22:55:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28641.394561 secs 2023-03-09T22:55:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:31 smithi038 ceph-mon[92422]: pgmap v15787: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 53 KiB/s wr, 7 op/s 2023-03-09T22:55:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28641.394561 secs 2023-03-09T22:55:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:33 smithi122 ceph-mon[93869]: pgmap v15788: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 9 op/s 2023-03-09T22:55:33.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:55:33.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28643.780059 secs 2023-03-09T22:55:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:33 smithi181 ceph-mon[94189]: pgmap v15788: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 9 op/s 2023-03-09T22:55:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:55:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28643.780059 secs 2023-03-09T22:55:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:33 smithi038 ceph-mon[92422]: pgmap v15788: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 9 op/s 2023-03-09T22:55:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:55:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28643.780059 secs 2023-03-09T22:55:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29105.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:55:35.367 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:35 smithi038 ceph-mon[92422]: pgmap v15789: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 7 op/s 2023-03-09T22:55:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:35 smithi122 ceph-mon[93869]: pgmap v15789: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 7 op/s 2023-03-09T22:55:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:35 smithi181 ceph-mon[94189]: pgmap v15789: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 7 op/s 2023-03-09T22:55:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28646.394622 secs 2023-03-09T22:55:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:55:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:55:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:55:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:55:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:55:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28646.394622 secs 2023-03-09T22:55:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:55:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:55:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:55:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:55:36.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:55:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28646.394622 secs 2023-03-09T22:55:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:55:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:55:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:55:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:55:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:55:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:37 smithi122 ceph-mon[93869]: pgmap v15790: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 341 B/s wr, 7 op/s 2023-03-09T22:55:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:37 smithi181 ceph-mon[94189]: pgmap v15790: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 341 B/s wr, 7 op/s 2023-03-09T22:55:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:37 smithi038 ceph-mon[92422]: pgmap v15790: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 341 B/s wr, 7 op/s 2023-03-09T22:55:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28648.780175 secs 2023-03-09T22:55:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28648.780175 secs 2023-03-09T22:55:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28648.780175 secs 2023-03-09T22:55:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29110.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:55:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:39 smithi122 ceph-mon[93869]: pgmap v15791: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 426 B/s wr, 8 op/s 2023-03-09T22:55:39.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:39 smithi181 ceph-mon[94189]: pgmap v15791: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 426 B/s wr, 8 op/s 2023-03-09T22:55:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:39 smithi038 ceph-mon[92422]: pgmap v15791: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 426 B/s wr, 8 op/s 2023-03-09T22:55:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:55:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:55:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28651.394684 secs 2023-03-09T22:55:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:55:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:55:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28651.394684 secs 2023-03-09T22:55:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:55:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:55:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28651.394684 secs 2023-03-09T22:55:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:41 smithi122 ceph-mon[93869]: pgmap v15792: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 682 B/s wr, 8 op/s 2023-03-09T22:55:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:55:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:55:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:41 smithi181 ceph-mon[94189]: pgmap v15792: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 682 B/s wr, 8 op/s 2023-03-09T22:55:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:55:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:55:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:41 smithi038 ceph-mon[92422]: pgmap v15792: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 682 B/s wr, 8 op/s 2023-03-09T22:55:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:55:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:55:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:43 smithi122 ceph-mon[93869]: pgmap v15793: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 123 KiB/s wr, 10 op/s 2023-03-09T22:55:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:43 smithi181 ceph-mon[94189]: pgmap v15793: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 123 KiB/s wr, 10 op/s 2023-03-09T22:55:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:43 smithi038 ceph-mon[92422]: pgmap v15793: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 123 KiB/s wr, 10 op/s 2023-03-09T22:55:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29115.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:55:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:45 smithi122 ceph-mon[93869]: pgmap v15794: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 123 KiB/s wr, 8 op/s 2023-03-09T22:55:45.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28656.394744 secs 2023-03-09T22:55:45.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:45 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:55:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:45 smithi181 ceph-mon[94189]: pgmap v15794: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 123 KiB/s wr, 8 op/s 2023-03-09T22:55:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28656.394744 secs 2023-03-09T22:55:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:45 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:55:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:45 smithi038 ceph-mon[92422]: pgmap v15794: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 123 KiB/s wr, 8 op/s 2023-03-09T22:55:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28656.394744 secs 2023-03-09T22:55:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:45 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:55:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:46 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:55:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:46 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:55:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:46 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:55:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:46 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:55:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:46 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:55:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:46 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:55:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:46 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:55:46.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:46 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:55:46.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:46 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:55:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:47 smithi122 ceph-mon[93869]: pgmap v15795: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 123 KiB/s wr, 8 op/s 2023-03-09T22:55:47.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:47 smithi181 ceph-mon[94189]: pgmap v15795: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 123 KiB/s wr, 8 op/s 2023-03-09T22:55:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:47 smithi038 ceph-mon[92422]: pgmap v15795: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 123 KiB/s wr, 8 op/s 2023-03-09T22:55:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29120.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:55:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:49 smithi122 ceph-mon[93869]: pgmap v15796: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 312 KiB/s wr, 9 op/s 2023-03-09T22:55:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:49 smithi181 ceph-mon[94189]: pgmap v15796: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 312 KiB/s wr, 9 op/s 2023-03-09T22:55:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:49 smithi038 ceph-mon[92422]: pgmap v15796: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 312 KiB/s wr, 9 op/s 2023-03-09T22:55:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28661.394827 secs 2023-03-09T22:55:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28661.394827 secs 2023-03-09T22:55:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28661.394827 secs 2023-03-09T22:55:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:51 smithi122 ceph-mon[93869]: pgmap v15797: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 312 KiB/s wr, 7 op/s 2023-03-09T22:55:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:51 smithi181 ceph-mon[94189]: pgmap v15797: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 312 KiB/s wr, 7 op/s 2023-03-09T22:55:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:51 smithi038 ceph-mon[92422]: pgmap v15797: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 312 KiB/s wr, 7 op/s 2023-03-09T22:55:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:53 smithi122 ceph-mon[93869]: pgmap v15798: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 490 KiB/s rd, 311 KiB/s wr, 9 op/s 2023-03-09T22:55:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:55:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:55:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:53 smithi181 ceph-mon[94189]: pgmap v15798: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 490 KiB/s rd, 311 KiB/s wr, 9 op/s 2023-03-09T22:55:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:55:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:55:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:53 smithi038 ceph-mon[92422]: pgmap v15798: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 490 KiB/s rd, 311 KiB/s wr, 9 op/s 2023-03-09T22:55:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:55:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:55:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29125.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:55:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:55 smithi122 ceph-mon[93869]: pgmap v15799: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 609 KiB/s rd, 432 KiB/s wr, 8 op/s 2023-03-09T22:55:55.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28666.394885 secs 2023-03-09T22:55:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:55 smithi181 ceph-mon[94189]: pgmap v15799: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 609 KiB/s rd, 432 KiB/s wr, 8 op/s 2023-03-09T22:55:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28666.394885 secs 2023-03-09T22:55:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:55 smithi038 ceph-mon[92422]: pgmap v15799: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 609 KiB/s rd, 432 KiB/s wr, 8 op/s 2023-03-09T22:55:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28666.394885 secs 2023-03-09T22:55:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:55:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:55:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:55:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:55:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:55:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:55:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:57 smithi038 ceph-mon[92422]: pgmap v15800: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 611 KiB/s rd, 433 KiB/s wr, 8 op/s 2023-03-09T22:55:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:57 smithi122 ceph-mon[93869]: pgmap v15800: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 611 KiB/s rd, 433 KiB/s wr, 8 op/s 2023-03-09T22:55:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:57 smithi181 ceph-mon[94189]: pgmap v15800: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 611 KiB/s rd, 433 KiB/s wr, 8 op/s 2023-03-09T22:55:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29130.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:55:59.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:55:59 smithi038 ceph-mon[92422]: pgmap v15801: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 835 KiB/s rd, 656 KiB/s wr, 10 op/s 2023-03-09T22:55:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:55:59 smithi122 ceph-mon[93869]: pgmap v15801: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 835 KiB/s rd, 656 KiB/s wr, 10 op/s 2023-03-09T22:55:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:55:59 smithi181 ceph-mon[94189]: pgmap v15801: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 835 KiB/s rd, 656 KiB/s wr, 10 op/s 2023-03-09T22:56:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:56:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:56:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28671.394963 secs 2023-03-09T22:56:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:56:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:56:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28671.394963 secs 2023-03-09T22:56:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:56:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:56:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28671.394963 secs 2023-03-09T22:56:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:01 smithi122 ceph-mon[93869]: pgmap v15802: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 684 KiB/s rd, 494 KiB/s wr, 10 op/s 2023-03-09T22:56:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:56:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:56:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:01 smithi181 ceph-mon[94189]: pgmap v15802: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 684 KiB/s rd, 494 KiB/s wr, 10 op/s 2023-03-09T22:56:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:56:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:56:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:01 smithi038 ceph-mon[92422]: pgmap v15802: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 684 KiB/s rd, 494 KiB/s wr, 10 op/s 2023-03-09T22:56:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:56:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:56:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:03 smithi122 ceph-mon[93869]: pgmap v15803: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 685 KiB/s rd, 494 KiB/s wr, 12 op/s 2023-03-09T22:56:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:03 smithi181 ceph-mon[94189]: pgmap v15803: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 685 KiB/s rd, 494 KiB/s wr, 12 op/s 2023-03-09T22:56:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:03 smithi038 ceph-mon[92422]: pgmap v15803: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 685 KiB/s rd, 494 KiB/s wr, 12 op/s 2023-03-09T22:56:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29135.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:56:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:05 smithi122 ceph-mon[93869]: pgmap v15804: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 871 KiB/s wr, 14 op/s 2023-03-09T22:56:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28676.394985 secs 2023-03-09T22:56:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:05 smithi181 ceph-mon[94189]: pgmap v15804: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 871 KiB/s wr, 14 op/s 2023-03-09T22:56:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28676.394985 secs 2023-03-09T22:56:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:05 smithi038 ceph-mon[92422]: pgmap v15804: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 871 KiB/s wr, 14 op/s 2023-03-09T22:56:05.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28676.394985 secs 2023-03-09T22:56:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:07 smithi122 ceph-mon[93869]: pgmap v15805: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 623 KiB/s rd, 650 KiB/s wr, 16 op/s 2023-03-09T22:56:07.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:07 smithi181 ceph-mon[94189]: pgmap v15805: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 623 KiB/s rd, 650 KiB/s wr, 16 op/s 2023-03-09T22:56:07.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:07 smithi038 ceph-mon[92422]: pgmap v15805: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 623 KiB/s rd, 650 KiB/s wr, 16 op/s 2023-03-09T22:56:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:56:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:56:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28678.780882 secs 2023-03-09T22:56:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:56:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:56:08.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28678.780882 secs 2023-03-09T22:56:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:56:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:56:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28678.780882 secs 2023-03-09T22:56:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29140.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:56:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:09 smithi122 ceph-mon[93869]: pgmap v15806: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 622 KiB/s rd, 650 KiB/s wr, 17 op/s 2023-03-09T22:56:09.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:56:09.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:56:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:09 smithi181 ceph-mon[94189]: pgmap v15806: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 622 KiB/s rd, 650 KiB/s wr, 17 op/s 2023-03-09T22:56:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:56:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:56:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:09 smithi038 ceph-mon[92422]: pgmap v15806: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 622 KiB/s rd, 650 KiB/s wr, 17 op/s 2023-03-09T22:56:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:56:09.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:56:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:10 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:56:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:10 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:56:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:10 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:56:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:10 smithi122 ceph-mon[93869]: osdmap e1203: 12 total, 12 up, 12 in 2023-03-09T22:56:10.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28681.395050 secs 2023-03-09T22:56:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:10 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:56:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:10 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:56:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:10 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:56:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:10 smithi181 ceph-mon[94189]: osdmap e1203: 12 total, 12 up, 12 in 2023-03-09T22:56:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28681.395050 secs 2023-03-09T22:56:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:10 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:56:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:10 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:56:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:10 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:56:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:10 smithi038 ceph-mon[92422]: osdmap e1203: 12 total, 12 up, 12 in 2023-03-09T22:56:10.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28681.395050 secs 2023-03-09T22:56:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:11 smithi122 ceph-mon[93869]: pgmap v15808: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 577 KiB/s rd, 812 KiB/s wr, 31 op/s 2023-03-09T22:56:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:11 smithi181 ceph-mon[94189]: pgmap v15808: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 577 KiB/s rd, 812 KiB/s wr, 31 op/s 2023-03-09T22:56:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:11 smithi038 ceph-mon[92422]: pgmap v15808: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 577 KiB/s rd, 812 KiB/s wr, 31 op/s 2023-03-09T22:56:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:13 smithi122 ceph-mon[93869]: pgmap v15809: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 576 KiB/s rd, 812 KiB/s wr, 30 op/s 2023-03-09T22:56:13.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28683.781001 secs 2023-03-09T22:56:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:13 smithi181 ceph-mon[94189]: pgmap v15809: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 576 KiB/s rd, 812 KiB/s wr, 30 op/s 2023-03-09T22:56:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28683.781001 secs 2023-03-09T22:56:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:13 smithi038 ceph-mon[92422]: pgmap v15809: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 576 KiB/s rd, 812 KiB/s wr, 30 op/s 2023-03-09T22:56:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28683.781001 secs 2023-03-09T22:56:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29145.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:56:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:15 smithi122 ceph-mon[93869]: pgmap v15810: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 486 KiB/s rd, 504 KiB/s wr, 28 op/s 2023-03-09T22:56:15.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28686.395109 secs 2023-03-09T22:56:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:15 smithi181 ceph-mon[94189]: pgmap v15810: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 486 KiB/s rd, 504 KiB/s wr, 28 op/s 2023-03-09T22:56:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28686.395109 secs 2023-03-09T22:56:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:15 smithi038 ceph-mon[92422]: pgmap v15810: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 486 KiB/s rd, 504 KiB/s wr, 28 op/s 2023-03-09T22:56:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28686.395109 secs 2023-03-09T22:56:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:56:16.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:56:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:56:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:56:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:56:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:56:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:17 smithi122 ceph-mon[93869]: pgmap v15811: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 485 KiB/s rd, 477 KiB/s wr, 25 op/s 2023-03-09T22:56:17.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:56:17.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:56:17.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:17 smithi122 ceph-mon[93869]: osdmap e1204: 12 total, 12 up, 12 in 2023-03-09T22:56:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:17 smithi181 ceph-mon[94189]: pgmap v15811: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 485 KiB/s rd, 477 KiB/s wr, 25 op/s 2023-03-09T22:56:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:56:17.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:56:17.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:17 smithi181 ceph-mon[94189]: osdmap e1204: 12 total, 12 up, 12 in 2023-03-09T22:56:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:17 smithi038 ceph-mon[92422]: pgmap v15811: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 485 KiB/s rd, 477 KiB/s wr, 25 op/s 2023-03-09T22:56:17.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:56:17.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:56:17.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:17 smithi038 ceph-mon[92422]: osdmap e1204: 12 total, 12 up, 12 in 2023-03-09T22:56:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28688.781146 secs 2023-03-09T22:56:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28688.781146 secs 2023-03-09T22:56:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28688.781146 secs 2023-03-09T22:56:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29150.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:56:19.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:19 smithi122 ceph-mon[93869]: pgmap v15813: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 525 KiB/s rd, 292 KiB/s wr, 10 op/s 2023-03-09T22:56:19.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:19 smithi181 ceph-mon[94189]: pgmap v15813: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 525 KiB/s rd, 292 KiB/s wr, 10 op/s 2023-03-09T22:56:19.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:19 smithi038 ceph-mon[92422]: pgmap v15813: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 525 KiB/s rd, 292 KiB/s wr, 10 op/s 2023-03-09T22:56:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28691.395169 secs 2023-03-09T22:56:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28691.395169 secs 2023-03-09T22:56:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28691.395169 secs 2023-03-09T22:56:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:21 smithi122 ceph-mon[93869]: pgmap v15814: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 245 KiB/s wr, 9 op/s 2023-03-09T22:56:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:21 smithi181 ceph-mon[94189]: pgmap v15814: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 245 KiB/s wr, 9 op/s 2023-03-09T22:56:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:21 smithi038 ceph-mon[92422]: pgmap v15814: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 245 KiB/s wr, 9 op/s 2023-03-09T22:56:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:23 smithi122 ceph-mon[93869]: pgmap v15815: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 245 KiB/s wr, 9 op/s 2023-03-09T22:56:23.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28693.781254 secs 2023-03-09T22:56:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:23 smithi181 ceph-mon[94189]: pgmap v15815: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 245 KiB/s wr, 9 op/s 2023-03-09T22:56:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28693.781254 secs 2023-03-09T22:56:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:23 smithi038 ceph-mon[92422]: pgmap v15815: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 245 KiB/s wr, 9 op/s 2023-03-09T22:56:23.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28693.781254 secs 2023-03-09T22:56:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29155.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:56:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:25 smithi122 ceph-mon[93869]: pgmap v15816: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 101 KiB/s wr, 7 op/s 2023-03-09T22:56:25.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28696.395229 secs 2023-03-09T22:56:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:25 smithi181 ceph-mon[94189]: pgmap v15816: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 101 KiB/s wr, 7 op/s 2023-03-09T22:56:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28696.395229 secs 2023-03-09T22:56:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:25 smithi038 ceph-mon[92422]: pgmap v15816: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 101 KiB/s wr, 7 op/s 2023-03-09T22:56:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28696.395229 secs 2023-03-09T22:56:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:27 smithi122 ceph-mon[93869]: pgmap v15817: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 101 KiB/s wr, 7 op/s 2023-03-09T22:56:27.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:27 smithi181 ceph-mon[94189]: pgmap v15817: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 101 KiB/s wr, 7 op/s 2023-03-09T22:56:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:27 smithi038 ceph-mon[92422]: pgmap v15817: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 101 KiB/s wr, 7 op/s 2023-03-09T22:56:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28698.781378 secs 2023-03-09T22:56:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28698.781378 secs 2023-03-09T22:56:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28698.781378 secs 2023-03-09T22:56:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29160.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:56:29.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:29 smithi122 ceph-mon[93869]: pgmap v15818: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 KiB/s wr, 7 op/s 2023-03-09T22:56:29.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:29 smithi181 ceph-mon[94189]: pgmap v15818: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 KiB/s wr, 7 op/s 2023-03-09T22:56:29.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:29 smithi038 ceph-mon[92422]: pgmap v15818: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 KiB/s wr, 7 op/s 2023-03-09T22:56:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28701.395291 secs 2023-03-09T22:56:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28701.395291 secs 2023-03-09T22:56:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28701.395291 secs 2023-03-09T22:56:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:31 smithi122 ceph-mon[93869]: pgmap v15819: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T22:56:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:31 smithi181 ceph-mon[94189]: pgmap v15819: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T22:56:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:31 smithi038 ceph-mon[92422]: pgmap v15819: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T22:56:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:33 smithi122 ceph-mon[93869]: pgmap v15820: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T22:56:33.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28703.781495 secs 2023-03-09T22:56:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:33 smithi181 ceph-mon[94189]: pgmap v15820: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T22:56:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28703.781495 secs 2023-03-09T22:56:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:33 smithi038 ceph-mon[92422]: pgmap v15820: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T22:56:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28703.781495 secs 2023-03-09T22:56:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29165.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:56:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:35 smithi122 ceph-mon[93869]: pgmap v15821: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 7 op/s 2023-03-09T22:56:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28706.395353 secs 2023-03-09T22:56:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:35 smithi181 ceph-mon[94189]: pgmap v15821: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 7 op/s 2023-03-09T22:56:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28706.395353 secs 2023-03-09T22:56:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:35 smithi038 ceph-mon[92422]: pgmap v15821: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 7 op/s 2023-03-09T22:56:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28706.395353 secs 2023-03-09T22:56:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:56:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:56:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:56:37.692 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:37 smithi122 ceph-mon[93869]: pgmap v15822: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 341 B/s wr, 6 op/s 2023-03-09T22:56:37.703 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:37 smithi181 ceph-mon[94189]: pgmap v15822: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 341 B/s wr, 6 op/s 2023-03-09T22:56:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:37 smithi038 ceph-mon[92422]: pgmap v15822: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 341 B/s wr, 6 op/s 2023-03-09T22:56:38.607 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28708.781566 secs 2023-03-09T22:56:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28708.781566 secs 2023-03-09T22:56:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28708.781566 secs 2023-03-09T22:56:38.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29170.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:56:39.558 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:39 smithi181 ceph-mon[94189]: pgmap v15823: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 426 B/s wr, 7 op/s 2023-03-09T22:56:39.706 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:39 smithi122 ceph-mon[93869]: pgmap v15823: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 426 B/s wr, 7 op/s 2023-03-09T22:56:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:39 smithi038 ceph-mon[92422]: pgmap v15823: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 426 B/s wr, 7 op/s 2023-03-09T22:56:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:56:40.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:56:40.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:56:40.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:56:40.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:56:40.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:56:40.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:56:40.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:40 smithi122 ceph-mon[93869]: pgmap v15824: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 682 B/s wr, 7 op/s 2023-03-09T22:56:40.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28711.395429 secs 2023-03-09T22:56:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:56:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:56:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:56:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:56:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:56:40.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:56:40.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:56:40.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:40 smithi181 ceph-mon[94189]: pgmap v15824: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 682 B/s wr, 7 op/s 2023-03-09T22:56:40.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28711.395429 secs 2023-03-09T22:56:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:56:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:56:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:56:41.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:56:41.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:56:41.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:56:41.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:56:41.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:40 smithi038 ceph-mon[92422]: pgmap v15824: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 682 B/s wr, 7 op/s 2023-03-09T22:56:41.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28711.395429 secs 2023-03-09T22:56:43.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:42 smithi122 ceph-mon[93869]: pgmap v15825: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 767 B/s wr, 8 op/s 2023-03-09T22:56:43.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:42 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 28713.781683 secs 2023-03-09T22:56:43.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:42 smithi181 ceph-mon[94189]: pgmap v15825: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 767 B/s wr, 8 op/s 2023-03-09T22:56:43.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:42 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 28713.781683 secs 2023-03-09T22:56:43.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:42 smithi038 ceph-mon[92422]: pgmap v15825: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 767 B/s wr, 8 op/s 2023-03-09T22:56:43.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:42 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 28713.781683 secs 2023-03-09T22:56:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29175.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:56:45.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:44 smithi122 ceph-mon[93869]: pgmap v15826: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 157 KiB/s wr, 8 op/s 2023-03-09T22:56:45.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:44 smithi181 ceph-mon[94189]: pgmap v15826: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 157 KiB/s wr, 8 op/s 2023-03-09T22:56:45.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:44 smithi038 ceph-mon[92422]: pgmap v15826: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 157 KiB/s wr, 8 op/s 2023-03-09T22:56:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28716.395513 secs 2023-03-09T22:56:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28716.395513 secs 2023-03-09T22:56:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28716.395513 secs 2023-03-09T22:56:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:46 smithi122 ceph-mon[93869]: pgmap v15827: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 157 KiB/s wr, 8 op/s 2023-03-09T22:56:47.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:46 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:56:47.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:46 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:56:47.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:46 smithi181 ceph-mon[94189]: pgmap v15827: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 157 KiB/s wr, 8 op/s 2023-03-09T22:56:47.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:46 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:56:47.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:46 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:56:47.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:46 smithi038 ceph-mon[92422]: pgmap v15827: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 157 KiB/s wr, 8 op/s 2023-03-09T22:56:47.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:46 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:56:47.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:46 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:56:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:47 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 28718.781809 secs 2023-03-09T22:56:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:47 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 28718.781809 secs 2023-03-09T22:56:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:47 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 28718.781809 secs 2023-03-09T22:56:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29180.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:56:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:48 smithi122 ceph-mon[93869]: pgmap v15828: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 362 KiB/s wr, 9 op/s 2023-03-09T22:56:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:48 smithi181 ceph-mon[94189]: pgmap v15828: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 362 KiB/s wr, 9 op/s 2023-03-09T22:56:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:48 smithi038 ceph-mon[92422]: pgmap v15828: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 362 KiB/s wr, 9 op/s 2023-03-09T22:56:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:50 smithi122 ceph-mon[93869]: pgmap v15829: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 362 KiB/s wr, 8 op/s 2023-03-09T22:56:51.318 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28721.395579 secs 2023-03-09T22:56:51.319 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:50 smithi038 ceph-mon[92422]: pgmap v15829: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 362 KiB/s wr, 8 op/s 2023-03-09T22:56:51.319 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28721.395579 secs 2023-03-09T22:56:51.319 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:50 smithi181 ceph-mon[94189]: pgmap v15829: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 362 KiB/s wr, 8 op/s 2023-03-09T22:56:51.320 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28721.395579 secs 2023-03-09T22:56:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:56:52.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:56:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:56:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:56:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:56:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:56:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:52 smithi122 ceph-mon[93869]: pgmap v15830: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 498 KiB/s wr, 9 op/s 2023-03-09T22:56:53.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:56:53.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:56:53.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:52 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 28723.781898 secs 2023-03-09T22:56:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:52 smithi181 ceph-mon[94189]: pgmap v15830: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 498 KiB/s wr, 9 op/s 2023-03-09T22:56:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:56:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:56:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:52 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 28723.781898 secs 2023-03-09T22:56:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:52 smithi038 ceph-mon[92422]: pgmap v15830: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 498 KiB/s wr, 9 op/s 2023-03-09T22:56:53.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:56:53.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:56:53.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:52 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 28723.781898 secs 2023-03-09T22:56:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29185.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:56:55.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:54 smithi122 ceph-mon[93869]: pgmap v15831: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 656 KiB/s rd, 591 KiB/s wr, 9 op/s 2023-03-09T22:56:55.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:54 smithi181 ceph-mon[94189]: pgmap v15831: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 656 KiB/s rd, 591 KiB/s wr, 9 op/s 2023-03-09T22:56:55.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:54 smithi038 ceph-mon[92422]: pgmap v15831: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 656 KiB/s rd, 591 KiB/s wr, 9 op/s 2023-03-09T22:56:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:56:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:56:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28726.395660 secs 2023-03-09T22:56:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:56:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:56:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28726.395660 secs 2023-03-09T22:56:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:56:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:56:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28726.395660 secs 2023-03-09T22:56:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:56 smithi122 ceph-mon[93869]: pgmap v15832: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 486 KiB/s rd, 435 KiB/s wr, 9 op/s 2023-03-09T22:56:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:56:57.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:56:57.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:56 smithi181 ceph-mon[94189]: pgmap v15832: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 486 KiB/s rd, 435 KiB/s wr, 9 op/s 2023-03-09T22:56:57.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:56:57.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:56:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:56 smithi038 ceph-mon[92422]: pgmap v15832: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 486 KiB/s rd, 435 KiB/s wr, 9 op/s 2023-03-09T22:56:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:56:57.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:56:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28728.782021 secs 2023-03-09T22:56:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28728.782021 secs 2023-03-09T22:56:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28728.782021 secs 2023-03-09T22:56:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29190.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:56:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:58 smithi122 ceph-mon[93869]: pgmap v15833: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 662 KiB/s wr, 11 op/s 2023-03-09T22:56:59.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:58 smithi181 ceph-mon[94189]: pgmap v15833: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 662 KiB/s wr, 11 op/s 2023-03-09T22:56:59.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:58 smithi038 ceph-mon[92422]: pgmap v15833: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 662 KiB/s wr, 11 op/s 2023-03-09T22:57:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:57:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:56:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:57:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:57:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:56:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:57:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:57:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:56:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:57:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:00 smithi122 ceph-mon[93869]: pgmap v15834: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 615 KiB/s wr, 10 op/s 2023-03-09T22:57:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:57:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:57:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28731.395729 secs 2023-03-09T22:57:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:00 smithi181 ceph-mon[94189]: pgmap v15834: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 615 KiB/s wr, 10 op/s 2023-03-09T22:57:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:57:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:57:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28731.395729 secs 2023-03-09T22:57:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:00 smithi038 ceph-mon[92422]: pgmap v15834: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 615 KiB/s wr, 10 op/s 2023-03-09T22:57:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:57:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:57:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28731.395729 secs 2023-03-09T22:57:03.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:02 smithi122 ceph-mon[93869]: pgmap v15835: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 497 KiB/s rd, 615 KiB/s wr, 12 op/s 2023-03-09T22:57:03.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:02 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:57:03.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:02 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:57:03.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28733.782132 secs 2023-03-09T22:57:03.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:02 smithi181 ceph-mon[94189]: pgmap v15835: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 497 KiB/s rd, 615 KiB/s wr, 12 op/s 2023-03-09T22:57:03.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:02 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:57:03.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:02 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:57:03.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28733.782132 secs 2023-03-09T22:57:03.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:02 smithi038 ceph-mon[92422]: pgmap v15835: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 497 KiB/s rd, 615 KiB/s wr, 12 op/s 2023-03-09T22:57:03.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:02 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:57:03.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:02 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:57:03.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28733.782132 secs 2023-03-09T22:57:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29195.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:57:05.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:04 smithi122 ceph-mon[93869]: pgmap v15836: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 707 KiB/s wr, 14 op/s 2023-03-09T22:57:05.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:04 smithi181 ceph-mon[94189]: pgmap v15836: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 707 KiB/s wr, 14 op/s 2023-03-09T22:57:05.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:04 smithi038 ceph-mon[92422]: pgmap v15836: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 707 KiB/s wr, 14 op/s 2023-03-09T22:57:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28736.395789 secs 2023-03-09T22:57:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28736.395789 secs 2023-03-09T22:57:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28736.395789 secs 2023-03-09T22:57:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:06 smithi122 ceph-mon[93869]: pgmap v15837: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 704 KiB/s rd, 735 KiB/s wr, 16 op/s 2023-03-09T22:57:07.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:06 smithi181 ceph-mon[94189]: pgmap v15837: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 704 KiB/s rd, 735 KiB/s wr, 16 op/s 2023-03-09T22:57:07.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:06 smithi038 ceph-mon[92422]: pgmap v15837: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 704 KiB/s rd, 735 KiB/s wr, 16 op/s 2023-03-09T22:57:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28738.782252 secs 2023-03-09T22:57:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28738.782252 secs 2023-03-09T22:57:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28738.782252 secs 2023-03-09T22:57:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29200.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:57:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:08 smithi122 ceph-mon[93869]: pgmap v15838: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 702 KiB/s rd, 735 KiB/s wr, 16 op/s 2023-03-09T22:57:09.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:08 smithi181 ceph-mon[94189]: pgmap v15838: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 702 KiB/s rd, 735 KiB/s wr, 16 op/s 2023-03-09T22:57:09.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:08 smithi038 ceph-mon[92422]: pgmap v15838: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 702 KiB/s rd, 735 KiB/s wr, 16 op/s 2023-03-09T22:57:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:10 smithi122 ceph-mon[93869]: pgmap v15839: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 603 KiB/s rd, 627 KiB/s wr, 26 op/s 2023-03-09T22:57:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:10 smithi122 ceph-mon[93869]: osdmap e1205: 12 total, 12 up, 12 in 2023-03-09T22:57:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28741.395854 secs 2023-03-09T22:57:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:10 smithi181 ceph-mon[94189]: pgmap v15839: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 603 KiB/s rd, 627 KiB/s wr, 26 op/s 2023-03-09T22:57:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:10 smithi181 ceph-mon[94189]: osdmap e1205: 12 total, 12 up, 12 in 2023-03-09T22:57:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28741.395854 secs 2023-03-09T22:57:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:10 smithi038 ceph-mon[92422]: pgmap v15839: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 603 KiB/s rd, 627 KiB/s wr, 26 op/s 2023-03-09T22:57:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:10 smithi038 ceph-mon[92422]: osdmap e1205: 12 total, 12 up, 12 in 2023-03-09T22:57:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28741.395854 secs 2023-03-09T22:57:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:11 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:57:12.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:11 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:57:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:11 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:57:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:11 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:57:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:11 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:57:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:11 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:57:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:12 smithi122 ceph-mon[93869]: pgmap v15841: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 971 KiB/s rd, 857 KiB/s wr, 30 op/s 2023-03-09T22:57:13.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:57:13.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:57:13.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28743.782364 secs 2023-03-09T22:57:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:12 smithi181 ceph-mon[94189]: pgmap v15841: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 971 KiB/s rd, 857 KiB/s wr, 30 op/s 2023-03-09T22:57:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:57:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:57:13.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28743.782364 secs 2023-03-09T22:57:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:12 smithi038 ceph-mon[92422]: pgmap v15841: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 971 KiB/s rd, 857 KiB/s wr, 30 op/s 2023-03-09T22:57:13.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:57:13.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:57:13.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28743.782364 secs 2023-03-09T22:57:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29205.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:57:15.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:14 smithi122 ceph-mon[93869]: pgmap v15842: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 582 KiB/s wr, 27 op/s 2023-03-09T22:57:15.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:14 smithi181 ceph-mon[94189]: pgmap v15842: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 582 KiB/s wr, 27 op/s 2023-03-09T22:57:15.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:14 smithi038 ceph-mon[92422]: pgmap v15842: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 582 KiB/s wr, 27 op/s 2023-03-09T22:57:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28746.395914 secs 2023-03-09T22:57:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28746.395914 secs 2023-03-09T22:57:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28746.395914 secs 2023-03-09T22:57:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:16 smithi122 ceph-mon[93869]: pgmap v15843: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 471 KiB/s rd, 462 KiB/s wr, 24 op/s 2023-03-09T22:57:17.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:57:17.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:57:17.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:16 smithi122 ceph-mon[93869]: osdmap e1206: 12 total, 12 up, 12 in 2023-03-09T22:57:17.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:16 smithi181 ceph-mon[94189]: pgmap v15843: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 471 KiB/s rd, 462 KiB/s wr, 24 op/s 2023-03-09T22:57:17.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:57:17.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:57:17.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:16 smithi181 ceph-mon[94189]: osdmap e1206: 12 total, 12 up, 12 in 2023-03-09T22:57:17.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:16 smithi038 ceph-mon[92422]: pgmap v15843: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 471 KiB/s rd, 462 KiB/s wr, 24 op/s 2023-03-09T22:57:17.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:57:17.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:57:17.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:16 smithi038 ceph-mon[92422]: osdmap e1206: 12 total, 12 up, 12 in 2023-03-09T22:57:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28748.782494 secs 2023-03-09T22:57:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28748.782494 secs 2023-03-09T22:57:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28748.782494 secs 2023-03-09T22:57:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29210.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:57:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:18 smithi122 ceph-mon[93869]: pgmap v15845: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 397 KiB/s wr, 10 op/s 2023-03-09T22:57:19.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:18 smithi181 ceph-mon[94189]: pgmap v15845: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 397 KiB/s wr, 10 op/s 2023-03-09T22:57:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:18 smithi038 ceph-mon[92422]: pgmap v15845: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 397 KiB/s wr, 10 op/s 2023-03-09T22:57:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:19 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:57:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:19 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:57:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:19 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:57:20.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:19 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:57:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:19 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:57:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:19 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:57:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:20 smithi038 ceph-mon[92422]: pgmap v15846: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 427 KiB/s wr, 10 op/s 2023-03-09T22:57:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28751.395978 secs 2023-03-09T22:57:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:20 smithi122 ceph-mon[93869]: pgmap v15846: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 427 KiB/s wr, 10 op/s 2023-03-09T22:57:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28751.395978 secs 2023-03-09T22:57:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:20 smithi181 ceph-mon[94189]: pgmap v15846: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 427 KiB/s wr, 10 op/s 2023-03-09T22:57:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28751.395978 secs 2023-03-09T22:57:23.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:22 smithi038 ceph-mon[92422]: pgmap v15847: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 126 KiB/s wr, 8 op/s 2023-03-09T22:57:23.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28753.782620 secs 2023-03-09T22:57:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:22 smithi122 ceph-mon[93869]: pgmap v15847: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 126 KiB/s wr, 8 op/s 2023-03-09T22:57:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28753.782620 secs 2023-03-09T22:57:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:22 smithi181 ceph-mon[94189]: pgmap v15847: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 126 KiB/s wr, 8 op/s 2023-03-09T22:57:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28753.782620 secs 2023-03-09T22:57:23.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29215.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:57:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:25 smithi122 ceph-mon[93869]: pgmap v15848: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 126 KiB/s wr, 7 op/s 2023-03-09T22:57:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:25 smithi181 ceph-mon[94189]: pgmap v15848: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 126 KiB/s wr, 7 op/s 2023-03-09T22:57:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:25 smithi038 ceph-mon[92422]: pgmap v15848: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 126 KiB/s wr, 7 op/s 2023-03-09T22:57:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28756.396038 secs 2023-03-09T22:57:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28756.396038 secs 2023-03-09T22:57:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28756.396038 secs 2023-03-09T22:57:27.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:27 smithi122 ceph-mon[93869]: pgmap v15849: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 181 KiB/s wr, 8 op/s 2023-03-09T22:57:27.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:27 smithi181 ceph-mon[94189]: pgmap v15849: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 181 KiB/s wr, 8 op/s 2023-03-09T22:57:27.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:27 smithi038 ceph-mon[92422]: pgmap v15849: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 181 KiB/s wr, 8 op/s 2023-03-09T22:57:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28758.782736 secs 2023-03-09T22:57:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28758.782736 secs 2023-03-09T22:57:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28758.782736 secs 2023-03-09T22:57:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29220.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:57:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:29 smithi122 ceph-mon[93869]: pgmap v15850: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 166 KiB/s wr, 7 op/s 2023-03-09T22:57:29.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:29 smithi181 ceph-mon[94189]: pgmap v15850: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 166 KiB/s wr, 7 op/s 2023-03-09T22:57:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:29 smithi038 ceph-mon[92422]: pgmap v15850: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 166 KiB/s wr, 7 op/s 2023-03-09T22:57:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:31 smithi122 ceph-mon[93869]: pgmap v15851: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 151 KiB/s wr, 7 op/s 2023-03-09T22:57:31.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28761.396099 secs 2023-03-09T22:57:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:31 smithi181 ceph-mon[94189]: pgmap v15851: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 151 KiB/s wr, 7 op/s 2023-03-09T22:57:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28761.396099 secs 2023-03-09T22:57:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:31 smithi038 ceph-mon[92422]: pgmap v15851: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 151 KiB/s wr, 7 op/s 2023-03-09T22:57:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28761.396099 secs 2023-03-09T22:57:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:33 smithi122 ceph-mon[93869]: pgmap v15852: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 67 KiB/s wr, 8 op/s 2023-03-09T22:57:33.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28763.782857 secs 2023-03-09T22:57:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:33 smithi181 ceph-mon[94189]: pgmap v15852: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 67 KiB/s wr, 8 op/s 2023-03-09T22:57:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28763.782857 secs 2023-03-09T22:57:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:33 smithi038 ceph-mon[92422]: pgmap v15852: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 67 KiB/s wr, 8 op/s 2023-03-09T22:57:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28763.782857 secs 2023-03-09T22:57:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29225.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:57:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:35 smithi122 ceph-mon[93869]: pgmap v15853: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 67 KiB/s wr, 6 op/s 2023-03-09T22:57:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:35 smithi181 ceph-mon[94189]: pgmap v15853: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 67 KiB/s wr, 6 op/s 2023-03-09T22:57:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:35 smithi038 ceph-mon[92422]: pgmap v15853: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 67 KiB/s wr, 6 op/s 2023-03-09T22:57:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28766.396158 secs 2023-03-09T22:57:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28766.396158 secs 2023-03-09T22:57:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28766.396158 secs 2023-03-09T22:57:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:37 smithi122 ceph-mon[93869]: pgmap v15854: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 67 KiB/s wr, 8 op/s 2023-03-09T22:57:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:37 smithi181 ceph-mon[94189]: pgmap v15854: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 67 KiB/s wr, 8 op/s 2023-03-09T22:57:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:37 smithi038 ceph-mon[92422]: pgmap v15854: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 67 KiB/s wr, 8 op/s 2023-03-09T22:57:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28768.782973 secs 2023-03-09T22:57:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28768.782973 secs 2023-03-09T22:57:38.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28768.782973 secs 2023-03-09T22:57:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29230.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:57:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:39 smithi122 ceph-mon[93869]: pgmap v15855: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T22:57:39.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:39 smithi181 ceph-mon[94189]: pgmap v15855: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T22:57:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:39 smithi038 ceph-mon[92422]: pgmap v15855: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T22:57:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:57:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:57:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:57:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:41 smithi122 ceph-mon[93869]: pgmap v15856: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 7 op/s 2023-03-09T22:57:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28771.396220 secs 2023-03-09T22:57:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:57:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:57:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:57:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:41 smithi181 ceph-mon[94189]: pgmap v15856: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 7 op/s 2023-03-09T22:57:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28771.396220 secs 2023-03-09T22:57:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:57:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:57:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:57:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:41 smithi038 ceph-mon[92422]: pgmap v15856: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 7 op/s 2023-03-09T22:57:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28771.396220 secs 2023-03-09T22:57:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:57:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:57:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:57:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:43 smithi122 ceph-mon[93869]: pgmap v15857: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 341 B/s wr, 8 op/s 2023-03-09T22:57:43.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28773.783066 secs 2023-03-09T22:57:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:43 smithi181 ceph-mon[94189]: pgmap v15857: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 341 B/s wr, 8 op/s 2023-03-09T22:57:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28773.783066 secs 2023-03-09T22:57:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:43 smithi038 ceph-mon[92422]: pgmap v15857: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 341 B/s wr, 8 op/s 2023-03-09T22:57:43.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28773.783066 secs 2023-03-09T22:57:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29235.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:57:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:45 smithi122 ceph-mon[93869]: pgmap v15858: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 682 B/s wr, 7 op/s 2023-03-09T22:57:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:45 smithi181 ceph-mon[94189]: pgmap v15858: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 682 B/s wr, 7 op/s 2023-03-09T22:57:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:45 smithi038 ceph-mon[92422]: pgmap v15858: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 682 B/s wr, 7 op/s 2023-03-09T22:57:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28776.396282 secs 2023-03-09T22:57:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28776.396282 secs 2023-03-09T22:57:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28776.396282 secs 2023-03-09T22:57:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:47 smithi122 ceph-mon[93869]: pgmap v15859: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 171 KiB/s wr, 9 op/s 2023-03-09T22:57:47.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:47 smithi181 ceph-mon[94189]: pgmap v15859: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 171 KiB/s wr, 9 op/s 2023-03-09T22:57:47.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:47 smithi038 ceph-mon[92422]: pgmap v15859: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 171 KiB/s wr, 9 op/s 2023-03-09T22:57:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29240.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:57:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:49 smithi122 ceph-mon[93869]: pgmap v15860: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 295 KiB/s wr, 8 op/s 2023-03-09T22:57:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:57:49.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:57:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:49 smithi181 ceph-mon[94189]: pgmap v15860: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 295 KiB/s wr, 8 op/s 2023-03-09T22:57:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:57:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:57:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:49 smithi038 ceph-mon[92422]: pgmap v15860: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 295 KiB/s wr, 8 op/s 2023-03-09T22:57:49.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:57:49.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:57:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:50 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:57:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:50 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:57:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:50 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:57:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:51 smithi122 ceph-mon[93869]: pgmap v15861: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 321 KiB/s rd, 320 KiB/s wr, 10 op/s 2023-03-09T22:57:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28781.396347 secs 2023-03-09T22:57:51.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:51 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:57:51.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:57:51.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:57:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:51 smithi181 ceph-mon[94189]: pgmap v15861: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 321 KiB/s rd, 320 KiB/s wr, 10 op/s 2023-03-09T22:57:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28781.396347 secs 2023-03-09T22:57:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:51 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:57:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:57:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:57:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:51 smithi038 ceph-mon[92422]: pgmap v15861: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 321 KiB/s rd, 320 KiB/s wr, 10 op/s 2023-03-09T22:57:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28781.396347 secs 2023-03-09T22:57:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:51 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:57:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:57:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:57:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:57:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:57:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:57:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:57:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:57:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:57:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:53 smithi122 ceph-mon[93869]: pgmap v15862: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 319 KiB/s wr, 10 op/s 2023-03-09T22:57:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:53 smithi181 ceph-mon[94189]: pgmap v15862: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 319 KiB/s wr, 10 op/s 2023-03-09T22:57:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:53 smithi038 ceph-mon[92422]: pgmap v15862: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 319 KiB/s wr, 10 op/s 2023-03-09T22:57:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29245.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:57:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:55 smithi122 ceph-mon[93869]: pgmap v15863: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 585 KiB/s rd, 582 KiB/s wr, 11 op/s 2023-03-09T22:57:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28786.396406 secs 2023-03-09T22:57:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:55 smithi181 ceph-mon[94189]: pgmap v15863: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 585 KiB/s rd, 582 KiB/s wr, 11 op/s 2023-03-09T22:57:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28786.396406 secs 2023-03-09T22:57:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:55 smithi038 ceph-mon[92422]: pgmap v15863: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 585 KiB/s rd, 582 KiB/s wr, 11 op/s 2023-03-09T22:57:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28786.396406 secs 2023-03-09T22:57:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:57:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:57:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:57:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:57:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:57:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:57:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:57 smithi122 ceph-mon[93869]: pgmap v15864: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 681 KiB/s rd, 665 KiB/s wr, 14 op/s 2023-03-09T22:57:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:57:57.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:57:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:57 smithi181 ceph-mon[94189]: pgmap v15864: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 681 KiB/s rd, 665 KiB/s wr, 14 op/s 2023-03-09T22:57:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:57:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:57:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:57 smithi038 ceph-mon[92422]: pgmap v15864: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 681 KiB/s rd, 665 KiB/s wr, 14 op/s 2023-03-09T22:57:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:57:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:57:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29250.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:57:59.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:57:59 smithi122 ceph-mon[93869]: pgmap v15865: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 829 KiB/s rd, 815 KiB/s wr, 13 op/s 2023-03-09T22:57:59.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:57:59 smithi181 ceph-mon[94189]: pgmap v15865: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 829 KiB/s rd, 815 KiB/s wr, 13 op/s 2023-03-09T22:57:59.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:57:59 smithi038 ceph-mon[92422]: pgmap v15865: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 829 KiB/s rd, 815 KiB/s wr, 13 op/s 2023-03-09T22:58:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:58:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:58:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28791.396468 secs 2023-03-09T22:58:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:58:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:58:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28791.396468 secs 2023-03-09T22:58:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:58:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:58:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28791.396468 secs 2023-03-09T22:58:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:01 smithi122 ceph-mon[93869]: pgmap v15866: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 731 KiB/s rd, 719 KiB/s wr, 14 op/s 2023-03-09T22:58:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:58:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:58:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:01 smithi181 ceph-mon[94189]: pgmap v15866: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 731 KiB/s rd, 719 KiB/s wr, 14 op/s 2023-03-09T22:58:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:58:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:58:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:01 smithi038 ceph-mon[92422]: pgmap v15866: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 731 KiB/s rd, 719 KiB/s wr, 14 op/s 2023-03-09T22:58:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:58:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:58:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:03 smithi122 ceph-mon[93869]: pgmap v15867: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 873 KiB/s wr, 18 op/s 2023-03-09T22:58:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28793.783547 secs 2023-03-09T22:58:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:03 smithi181 ceph-mon[94189]: pgmap v15867: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 873 KiB/s wr, 18 op/s 2023-03-09T22:58:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28793.783547 secs 2023-03-09T22:58:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:03 smithi038 ceph-mon[92422]: pgmap v15867: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 873 KiB/s wr, 18 op/s 2023-03-09T22:58:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28793.783547 secs 2023-03-09T22:58:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29255.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:58:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:05 smithi122 ceph-mon[93869]: pgmap v15868: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 892 KiB/s rd, 938 KiB/s wr, 29 op/s 2023-03-09T22:58:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:05 smithi122 ceph-mon[93869]: osdmap e1207: 12 total, 12 up, 12 in 2023-03-09T22:58:05.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28796.396529 secs 2023-03-09T22:58:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:05 smithi181 ceph-mon[94189]: pgmap v15868: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 892 KiB/s rd, 938 KiB/s wr, 29 op/s 2023-03-09T22:58:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:05 smithi181 ceph-mon[94189]: osdmap e1207: 12 total, 12 up, 12 in 2023-03-09T22:58:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28796.396529 secs 2023-03-09T22:58:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:05 smithi038 ceph-mon[92422]: pgmap v15868: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 892 KiB/s rd, 938 KiB/s wr, 29 op/s 2023-03-09T22:58:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:05 smithi038 ceph-mon[92422]: osdmap e1207: 12 total, 12 up, 12 in 2023-03-09T22:58:05.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28796.396529 secs 2023-03-09T22:58:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:06 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:58:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:06 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:58:06.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:58:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:06 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:58:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:06 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:58:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:58:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:06 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:58:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:06 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:58:06.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:58:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:07 smithi122 ceph-mon[93869]: pgmap v15870: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 838 KiB/s rd, 904 KiB/s wr, 31 op/s 2023-03-09T22:58:07.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:07 smithi181 ceph-mon[94189]: pgmap v15870: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 838 KiB/s rd, 904 KiB/s wr, 31 op/s 2023-03-09T22:58:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:07 smithi038 ceph-mon[92422]: pgmap v15870: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 838 KiB/s rd, 904 KiB/s wr, 31 op/s 2023-03-09T22:58:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28798.783697 secs 2023-03-09T22:58:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28798.783697 secs 2023-03-09T22:58:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28798.783697 secs 2023-03-09T22:58:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29260.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:58:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:09 smithi122 ceph-mon[93869]: pgmap v15871: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 455 KiB/s rd, 520 KiB/s wr, 29 op/s 2023-03-09T22:58:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:58:09.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:58:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:09 smithi181 ceph-mon[94189]: pgmap v15871: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 455 KiB/s rd, 520 KiB/s wr, 29 op/s 2023-03-09T22:58:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:58:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:58:09.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:09 smithi038 ceph-mon[92422]: pgmap v15871: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 455 KiB/s rd, 520 KiB/s wr, 29 op/s 2023-03-09T22:58:09.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:58:09.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:58:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28801.396592 secs 2023-03-09T22:58:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28801.396592 secs 2023-03-09T22:58:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28801.396592 secs 2023-03-09T22:58:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:11 smithi038 ceph-mon[92422]: pgmap v15872: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 487 KiB/s wr, 28 op/s 2023-03-09T22:58:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:11 smithi122 ceph-mon[93869]: pgmap v15872: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 487 KiB/s wr, 28 op/s 2023-03-09T22:58:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:11 smithi181 ceph-mon[94189]: pgmap v15872: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 487 KiB/s wr, 28 op/s 2023-03-09T22:58:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:58:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:58:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:12 smithi038 ceph-mon[92422]: osdmap e1208: 12 total, 12 up, 12 in 2023-03-09T22:58:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:58:12.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:58:12.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:12 smithi122 ceph-mon[93869]: osdmap e1208: 12 total, 12 up, 12 in 2023-03-09T22:58:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:58:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:58:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:12 smithi181 ceph-mon[94189]: osdmap e1208: 12 total, 12 up, 12 in 2023-03-09T22:58:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:13 smithi038 ceph-mon[92422]: pgmap v15873: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 418 KiB/s wr, 25 op/s 2023-03-09T22:58:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28803.783815 secs 2023-03-09T22:58:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:13 smithi122 ceph-mon[93869]: pgmap v15873: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 418 KiB/s wr, 25 op/s 2023-03-09T22:58:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28803.783815 secs 2023-03-09T22:58:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:13 smithi181 ceph-mon[94189]: pgmap v15873: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 418 KiB/s wr, 25 op/s 2023-03-09T22:58:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28803.783815 secs 2023-03-09T22:58:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29265.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:58:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:15 smithi122 ceph-mon[93869]: pgmap v15875: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 356 KiB/s wr, 9 op/s 2023-03-09T22:58:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28806.396650 secs 2023-03-09T22:58:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:15 smithi181 ceph-mon[94189]: pgmap v15875: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 356 KiB/s wr, 9 op/s 2023-03-09T22:58:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28806.396650 secs 2023-03-09T22:58:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:15 smithi038 ceph-mon[92422]: pgmap v15875: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 356 KiB/s wr, 9 op/s 2023-03-09T22:58:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28806.396650 secs 2023-03-09T22:58:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:17 smithi122 ceph-mon[93869]: pgmap v15876: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 145 KiB/s wr, 8 op/s 2023-03-09T22:58:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:17 smithi181 ceph-mon[94189]: pgmap v15876: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 145 KiB/s wr, 8 op/s 2023-03-09T22:58:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:17 smithi038 ceph-mon[92422]: pgmap v15876: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 145 KiB/s wr, 8 op/s 2023-03-09T22:58:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28808.783938 secs 2023-03-09T22:58:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28808.783938 secs 2023-03-09T22:58:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28808.783938 secs 2023-03-09T22:58:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29270.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:58:19.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:19 smithi122 ceph-mon[93869]: pgmap v15877: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 145 KiB/s wr, 8 op/s 2023-03-09T22:58:19.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:19 smithi181 ceph-mon[94189]: pgmap v15877: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 145 KiB/s wr, 8 op/s 2023-03-09T22:58:19.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:19 smithi038 ceph-mon[92422]: pgmap v15877: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 145 KiB/s wr, 8 op/s 2023-03-09T22:58:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28811.396745 secs 2023-03-09T22:58:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28811.396745 secs 2023-03-09T22:58:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28811.396745 secs 2023-03-09T22:58:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:21 smithi122 ceph-mon[93869]: pgmap v15878: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 144 KiB/s wr, 7 op/s 2023-03-09T22:58:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:21 smithi181 ceph-mon[94189]: pgmap v15878: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 144 KiB/s wr, 7 op/s 2023-03-09T22:58:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:21 smithi038 ceph-mon[92422]: pgmap v15878: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 144 KiB/s wr, 7 op/s 2023-03-09T22:58:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:23 smithi122 ceph-mon[93869]: pgmap v15879: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2023-03-09T22:58:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28813.784074 secs 2023-03-09T22:58:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:23 smithi181 ceph-mon[94189]: pgmap v15879: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2023-03-09T22:58:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28813.784074 secs 2023-03-09T22:58:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:23 smithi038 ceph-mon[92422]: pgmap v15879: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2023-03-09T22:58:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28813.784074 secs 2023-03-09T22:58:23.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29275.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:58:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:25 smithi122 ceph-mon[93869]: pgmap v15880: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2023-03-09T22:58:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28816.396805 secs 2023-03-09T22:58:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:25 smithi181 ceph-mon[94189]: pgmap v15880: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2023-03-09T22:58:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28816.396805 secs 2023-03-09T22:58:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:25 smithi038 ceph-mon[92422]: pgmap v15880: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2023-03-09T22:58:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28816.396805 secs 2023-03-09T22:58:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:27 smithi122 ceph-mon[93869]: pgmap v15881: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T22:58:27.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:27 smithi181 ceph-mon[94189]: pgmap v15881: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T22:58:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:27 smithi038 ceph-mon[92422]: pgmap v15881: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T22:58:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28818.784197 secs 2023-03-09T22:58:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28818.784197 secs 2023-03-09T22:58:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28818.784197 secs 2023-03-09T22:58:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29280.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:58:29.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:29 smithi122 ceph-mon[93869]: pgmap v15882: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T22:58:29.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:29 smithi181 ceph-mon[94189]: pgmap v15882: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T22:58:29.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:29 smithi038 ceph-mon[92422]: pgmap v15882: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T22:58:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28821.396869 secs 2023-03-09T22:58:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28821.396869 secs 2023-03-09T22:58:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28821.396869 secs 2023-03-09T22:58:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:31 smithi122 ceph-mon[93869]: pgmap v15883: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T22:58:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:31 smithi181 ceph-mon[94189]: pgmap v15883: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T22:58:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:31 smithi038 ceph-mon[92422]: pgmap v15883: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T22:58:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:33 smithi122 ceph-mon[93869]: pgmap v15884: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T22:58:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28823.784331 secs 2023-03-09T22:58:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:33 smithi181 ceph-mon[94189]: pgmap v15884: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T22:58:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28823.784331 secs 2023-03-09T22:58:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:33 smithi038 ceph-mon[92422]: pgmap v15884: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T22:58:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28823.784331 secs 2023-03-09T22:58:33.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29285.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:58:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:35 smithi122 ceph-mon[93869]: pgmap v15885: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T22:58:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28826.396929 secs 2023-03-09T22:58:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:35 smithi181 ceph-mon[94189]: pgmap v15885: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T22:58:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28826.396929 secs 2023-03-09T22:58:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:35 smithi038 ceph-mon[92422]: pgmap v15885: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T22:58:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28826.396929 secs 2023-03-09T22:58:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:37 smithi122 ceph-mon[93869]: pgmap v15886: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 102 KiB/s wr, 8 op/s 2023-03-09T22:58:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:37 smithi181 ceph-mon[94189]: pgmap v15886: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 102 KiB/s wr, 8 op/s 2023-03-09T22:58:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:37 smithi038 ceph-mon[92422]: pgmap v15886: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 102 KiB/s wr, 8 op/s 2023-03-09T22:58:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28828.784438 secs 2023-03-09T22:58:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28828.784438 secs 2023-03-09T22:58:38.814 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28828.784438 secs 2023-03-09T22:58:38.814 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29290.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:58:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:39 smithi122 ceph-mon[93869]: pgmap v15887: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 102 KiB/s wr, 6 op/s 2023-03-09T22:58:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:39 smithi181 ceph-mon[94189]: pgmap v15887: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 102 KiB/s wr, 6 op/s 2023-03-09T22:58:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:39 smithi038 ceph-mon[92422]: pgmap v15887: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 102 KiB/s wr, 6 op/s 2023-03-09T22:58:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28831.396982 secs 2023-03-09T22:58:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28831.396982 secs 2023-03-09T22:58:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28831.396982 secs 2023-03-09T22:58:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:41 smithi122 ceph-mon[93869]: pgmap v15888: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 103 KiB/s wr, 7 op/s 2023-03-09T22:58:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:58:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:41 smithi181 ceph-mon[94189]: pgmap v15888: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 103 KiB/s wr, 7 op/s 2023-03-09T22:58:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:58:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:41 smithi038 ceph-mon[92422]: pgmap v15888: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 103 KiB/s wr, 7 op/s 2023-03-09T22:58:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:58:43.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:42 smithi122 ceph-mon[93869]: pgmap v15889: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 162 KiB/s wr, 9 op/s 2023-03-09T22:58:43.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:58:43.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:58:43.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:58:43.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:58:43.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:58:43.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:58:43.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:58:43.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:58:43.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:58:43.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:42 smithi181 ceph-mon[94189]: pgmap v15889: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 162 KiB/s wr, 9 op/s 2023-03-09T22:58:43.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:58:43.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:58:43.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:58:43.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:58:43.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:58:43.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:58:43.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:58:43.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:58:43.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:58:43.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:42 smithi038 ceph-mon[92422]: pgmap v15889: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 162 KiB/s wr, 9 op/s 2023-03-09T22:58:43.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:58:43.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:58:43.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:58:43.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:58:43.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:58:43.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:58:43.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:58:43.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:58:43.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:58:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29295.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:58:45.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:44 smithi122 ceph-mon[93869]: pgmap v15890: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 162 KiB/s wr, 7 op/s 2023-03-09T22:58:45.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:44 smithi181 ceph-mon[94189]: pgmap v15890: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 162 KiB/s wr, 7 op/s 2023-03-09T22:58:45.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:44 smithi038 ceph-mon[92422]: pgmap v15890: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 162 KiB/s wr, 7 op/s 2023-03-09T22:58:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28836.397081 secs 2023-03-09T22:58:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:45 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:58:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28836.397081 secs 2023-03-09T22:58:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:45 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:58:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28836.397081 secs 2023-03-09T22:58:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:45 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:58:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:46 smithi122 ceph-mon[93869]: pgmap v15891: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 504 KiB/s wr, 10 op/s 2023-03-09T22:58:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:46 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:58:47.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:46 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:58:47.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:46 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:58:47.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:46 smithi181 ceph-mon[94189]: pgmap v15891: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 504 KiB/s wr, 10 op/s 2023-03-09T22:58:47.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:46 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:58:47.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:46 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:58:47.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:46 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:58:47.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:46 smithi038 ceph-mon[92422]: pgmap v15891: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 504 KiB/s wr, 10 op/s 2023-03-09T22:58:47.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:46 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:58:47.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:46 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:58:47.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:46 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:58:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:58:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:58:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:47 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 28838.784669 secs 2023-03-09T22:58:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:58:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:58:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:47 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 28838.784669 secs 2023-03-09T22:58:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:58:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:58:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:47 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 28838.784669 secs 2023-03-09T22:58:48.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29300.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:58:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:48 smithi122 ceph-mon[93869]: pgmap v15892: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 402 KiB/s wr, 7 op/s 2023-03-09T22:58:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:58:49.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:58:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:48 smithi181 ceph-mon[94189]: pgmap v15892: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 402 KiB/s wr, 7 op/s 2023-03-09T22:58:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:58:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:58:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:48 smithi038 ceph-mon[92422]: pgmap v15892: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 402 KiB/s wr, 7 op/s 2023-03-09T22:58:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:58:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:58:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:50 smithi122 ceph-mon[93869]: pgmap v15893: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 425 KiB/s wr, 8 op/s 2023-03-09T22:58:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:50 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:58:51.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:50 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:58:51.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28841.397140 secs 2023-03-09T22:58:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:50 smithi181 ceph-mon[94189]: pgmap v15893: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 425 KiB/s wr, 8 op/s 2023-03-09T22:58:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:50 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:58:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:50 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:58:51.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28841.397140 secs 2023-03-09T22:58:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:50 smithi038 ceph-mon[92422]: pgmap v15893: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 425 KiB/s wr, 8 op/s 2023-03-09T22:58:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:50 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:58:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:50 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:58:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28841.397140 secs 2023-03-09T22:58:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:51 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:58:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:51 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:58:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:51 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:58:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:52 smithi122 ceph-mon[93869]: pgmap v15894: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 577 KiB/s wr, 11 op/s 2023-03-09T22:58:53.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28843.784734 secs 2023-03-09T22:58:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:52 smithi181 ceph-mon[94189]: pgmap v15894: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 577 KiB/s wr, 11 op/s 2023-03-09T22:58:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28843.784734 secs 2023-03-09T22:58:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:52 smithi038 ceph-mon[92422]: pgmap v15894: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 577 KiB/s wr, 11 op/s 2023-03-09T22:58:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28843.784734 secs 2023-03-09T22:58:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29305.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:58:55.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:54 smithi122 ceph-mon[93869]: pgmap v15895: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 541 KiB/s wr, 10 op/s 2023-03-09T22:58:55.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:54 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:58:55.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:54 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:58:55.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:54 smithi181 ceph-mon[94189]: pgmap v15895: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 541 KiB/s wr, 10 op/s 2023-03-09T22:58:55.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:54 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:58:55.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:54 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:58:55.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:54 smithi038 ceph-mon[92422]: pgmap v15895: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 541 KiB/s wr, 10 op/s 2023-03-09T22:58:55.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:54 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:58:55.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:54 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:58:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:58:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:58:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28846.397200 secs 2023-03-09T22:58:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:58:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:58:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28846.397200 secs 2023-03-09T22:58:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:58:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:58:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28846.397200 secs 2023-03-09T22:58:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:56 smithi122 ceph-mon[93869]: pgmap v15896: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 731 KiB/s wr, 13 op/s 2023-03-09T22:58:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:58:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:58:57.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:56 smithi181 ceph-mon[94189]: pgmap v15896: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 731 KiB/s wr, 13 op/s 2023-03-09T22:58:57.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:58:57.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:58:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:56 smithi038 ceph-mon[92422]: pgmap v15896: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 731 KiB/s wr, 13 op/s 2023-03-09T22:58:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:58:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:58:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28848.784869 secs 2023-03-09T22:58:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28848.784869 secs 2023-03-09T22:58:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28848.784869 secs 2023-03-09T22:58:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29310.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:58:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:58 smithi122 ceph-mon[93869]: pgmap v15897: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 KiB/s rd, 586 KiB/s wr, 11 op/s 2023-03-09T22:58:59.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:58 smithi181 ceph-mon[94189]: pgmap v15897: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 KiB/s rd, 586 KiB/s wr, 11 op/s 2023-03-09T22:58:59.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:58 smithi038 ceph-mon[92422]: pgmap v15897: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 KiB/s rd, 586 KiB/s wr, 11 op/s 2023-03-09T22:59:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:59:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:58:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:59:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:59:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:58:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:59:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:59:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:58:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:59:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:00 smithi038 ceph-mon[92422]: pgmap v15898: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 614 KiB/s wr, 12 op/s 2023-03-09T22:59:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28851.397263 secs 2023-03-09T22:59:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:00 smithi122 ceph-mon[93869]: pgmap v15898: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 614 KiB/s wr, 12 op/s 2023-03-09T22:59:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28851.397263 secs 2023-03-09T22:59:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:00 smithi181 ceph-mon[94189]: pgmap v15898: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 614 KiB/s wr, 12 op/s 2023-03-09T22:59:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28851.397263 secs 2023-03-09T22:59:03.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:02 smithi038 ceph-mon[92422]: pgmap v15899: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 624 KiB/s wr, 17 op/s 2023-03-09T22:59:03.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28853.785006 secs 2023-03-09T22:59:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:02 smithi122 ceph-mon[93869]: pgmap v15899: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 624 KiB/s wr, 17 op/s 2023-03-09T22:59:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28853.785006 secs 2023-03-09T22:59:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:02 smithi181 ceph-mon[94189]: pgmap v15899: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 624 KiB/s wr, 17 op/s 2023-03-09T22:59:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28853.785006 secs 2023-03-09T22:59:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29315.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:59:05.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:04 smithi038 ceph-mon[92422]: pgmap v15900: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 638 KiB/s rd, 682 KiB/s wr, 28 op/s 2023-03-09T22:59:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:04 smithi122 ceph-mon[93869]: pgmap v15900: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 638 KiB/s rd, 682 KiB/s wr, 28 op/s 2023-03-09T22:59:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:04 smithi181 ceph-mon[94189]: pgmap v15900: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 638 KiB/s rd, 682 KiB/s wr, 28 op/s 2023-03-09T22:59:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28856.397327 secs 2023-03-09T22:59:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:06 smithi122 ceph-mon[93869]: osdmap e1209: 12 total, 12 up, 12 in 2023-03-09T22:59:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28856.397327 secs 2023-03-09T22:59:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:06 smithi181 ceph-mon[94189]: osdmap e1209: 12 total, 12 up, 12 in 2023-03-09T22:59:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28856.397327 secs 2023-03-09T22:59:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:06 smithi038 ceph-mon[92422]: osdmap e1209: 12 total, 12 up, 12 in 2023-03-09T22:59:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:07 smithi122 ceph-mon[93869]: pgmap v15902: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 691 KiB/s rd, 758 KiB/s wr, 31 op/s 2023-03-09T22:59:07.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:07 smithi181 ceph-mon[94189]: pgmap v15902: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 691 KiB/s rd, 758 KiB/s wr, 31 op/s 2023-03-09T22:59:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:07 smithi038 ceph-mon[92422]: pgmap v15902: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 691 KiB/s rd, 758 KiB/s wr, 31 op/s 2023-03-09T22:59:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:59:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:59:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28858.785139 secs 2023-03-09T22:59:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:59:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:59:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28858.785139 secs 2023-03-09T22:59:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:59:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:59:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28858.785139 secs 2023-03-09T22:59:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29320.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:59:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:09 smithi122 ceph-mon[93869]: pgmap v15903: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 455 KiB/s rd, 522 KiB/s wr, 30 op/s 2023-03-09T22:59:09.501 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:59:09.502 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:59:09.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:09 smithi038 ceph-mon[92422]: pgmap v15903: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 455 KiB/s rd, 522 KiB/s wr, 30 op/s 2023-03-09T22:59:09.503 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:59:09.503 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:59:09.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:09 smithi181 ceph-mon[94189]: pgmap v15903: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 455 KiB/s rd, 522 KiB/s wr, 30 op/s 2023-03-09T22:59:09.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:59:09.503 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:59:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:11 smithi122 ceph-mon[93869]: pgmap v15904: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 489 KiB/s wr, 30 op/s 2023-03-09T22:59:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28861.397380 secs 2023-03-09T22:59:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:11 smithi181 ceph-mon[94189]: pgmap v15904: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 489 KiB/s wr, 30 op/s 2023-03-09T22:59:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28861.397380 secs 2023-03-09T22:59:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:11 smithi038 ceph-mon[92422]: pgmap v15904: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 489 KiB/s wr, 30 op/s 2023-03-09T22:59:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28861.397380 secs 2023-03-09T22:59:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:13 smithi122 ceph-mon[93869]: pgmap v15905: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 578 KiB/s rd, 593 KiB/s wr, 25 op/s 2023-03-09T22:59:13.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28863.785252 secs 2023-03-09T22:59:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:13 smithi181 ceph-mon[94189]: pgmap v15905: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 578 KiB/s rd, 593 KiB/s wr, 25 op/s 2023-03-09T22:59:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28863.785252 secs 2023-03-09T22:59:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:13 smithi038 ceph-mon[92422]: pgmap v15905: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 578 KiB/s rd, 593 KiB/s wr, 25 op/s 2023-03-09T22:59:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28863.785252 secs 2023-03-09T22:59:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29325.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:59:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:15 smithi122 ceph-mon[93869]: pgmap v15906: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 341 KiB/s wr, 9 op/s 2023-03-09T22:59:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:15 smithi181 ceph-mon[94189]: pgmap v15906: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 341 KiB/s wr, 9 op/s 2023-03-09T22:59:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:15 smithi038 ceph-mon[92422]: pgmap v15906: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 341 KiB/s wr, 9 op/s 2023-03-09T22:59:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28866.397441 secs 2023-03-09T22:59:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28866.397441 secs 2023-03-09T22:59:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28866.397441 secs 2023-03-09T22:59:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:17 smithi122 ceph-mon[93869]: pgmap v15907: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 317 KiB/s wr, 11 op/s 2023-03-09T22:59:17.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:17 smithi122 ceph-mon[93869]: osdmap e1210: 12 total, 12 up, 12 in 2023-03-09T22:59:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:17 smithi181 ceph-mon[94189]: pgmap v15907: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 317 KiB/s wr, 11 op/s 2023-03-09T22:59:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:17 smithi181 ceph-mon[94189]: osdmap e1210: 12 total, 12 up, 12 in 2023-03-09T22:59:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:17 smithi038 ceph-mon[92422]: pgmap v15907: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 317 KiB/s wr, 11 op/s 2023-03-09T22:59:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:17 smithi038 ceph-mon[92422]: osdmap e1210: 12 total, 12 up, 12 in 2023-03-09T22:59:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28868.785368 secs 2023-03-09T22:59:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28868.785368 secs 2023-03-09T22:59:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28868.785368 secs 2023-03-09T22:59:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29330.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:59:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:19 smithi122 ceph-mon[93869]: pgmap v15909: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 145 KiB/s wr, 7 op/s 2023-03-09T22:59:19.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:19 smithi181 ceph-mon[94189]: pgmap v15909: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 145 KiB/s wr, 7 op/s 2023-03-09T22:59:19.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:19 smithi038 ceph-mon[92422]: pgmap v15909: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 145 KiB/s wr, 7 op/s 2023-03-09T22:59:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:21 smithi122 ceph-mon[93869]: pgmap v15910: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 145 KiB/s wr, 6 op/s 2023-03-09T22:59:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28871.397500 secs 2023-03-09T22:59:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:21 smithi181 ceph-mon[94189]: pgmap v15910: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 145 KiB/s wr, 6 op/s 2023-03-09T22:59:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28871.397500 secs 2023-03-09T22:59:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:21 smithi038 ceph-mon[92422]: pgmap v15910: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 145 KiB/s wr, 6 op/s 2023-03-09T22:59:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28871.397500 secs 2023-03-09T22:59:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:23 smithi122 ceph-mon[93869]: pgmap v15911: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T22:59:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28873.785450 secs 2023-03-09T22:59:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:23 smithi181 ceph-mon[94189]: pgmap v15911: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T22:59:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28873.785450 secs 2023-03-09T22:59:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:23 smithi038 ceph-mon[92422]: pgmap v15911: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T22:59:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28873.785450 secs 2023-03-09T22:59:23.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29335.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:59:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:25 smithi122 ceph-mon[93869]: pgmap v15912: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-09T22:59:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:25 smithi181 ceph-mon[94189]: pgmap v15912: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-09T22:59:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:25 smithi038 ceph-mon[92422]: pgmap v15912: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-09T22:59:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28876.397575 secs 2023-03-09T22:59:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28876.397575 secs 2023-03-09T22:59:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28876.397575 secs 2023-03-09T22:59:27.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:27 smithi122 ceph-mon[93869]: pgmap v15913: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-09T22:59:27.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:27 smithi181 ceph-mon[94189]: pgmap v15913: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-09T22:59:27.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:27 smithi038 ceph-mon[92422]: pgmap v15913: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-09T22:59:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28878.785545 secs 2023-03-09T22:59:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28878.785545 secs 2023-03-09T22:59:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28878.785545 secs 2023-03-09T22:59:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29340.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:59:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:29 smithi122 ceph-mon[93869]: pgmap v15914: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2023-03-09T22:59:29.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:29 smithi181 ceph-mon[94189]: pgmap v15914: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2023-03-09T22:59:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:29 smithi038 ceph-mon[92422]: pgmap v15914: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2023-03-09T22:59:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:31 smithi122 ceph-mon[93869]: pgmap v15915: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T22:59:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28881.397654 secs 2023-03-09T22:59:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:31 smithi181 ceph-mon[94189]: pgmap v15915: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T22:59:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28881.397654 secs 2023-03-09T22:59:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:31 smithi038 ceph-mon[92422]: pgmap v15915: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T22:59:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28881.397654 secs 2023-03-09T22:59:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:33 smithi122 ceph-mon[93869]: pgmap v15916: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T22:59:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28883.785626 secs 2023-03-09T22:59:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:33 smithi181 ceph-mon[94189]: pgmap v15916: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T22:59:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28883.785626 secs 2023-03-09T22:59:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:33 smithi038 ceph-mon[92422]: pgmap v15916: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T22:59:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28883.785626 secs 2023-03-09T22:59:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29345.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:59:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:35 smithi122 ceph-mon[93869]: pgmap v15917: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T22:59:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:35 smithi181 ceph-mon[94189]: pgmap v15917: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T22:59:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:35 smithi038 ceph-mon[92422]: pgmap v15917: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T22:59:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28886.397712 secs 2023-03-09T22:59:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28886.397712 secs 2023-03-09T22:59:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28886.397712 secs 2023-03-09T22:59:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:37 smithi122 ceph-mon[93869]: pgmap v15918: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T22:59:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:37 smithi181 ceph-mon[94189]: pgmap v15918: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T22:59:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:37 smithi038 ceph-mon[92422]: pgmap v15918: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T22:59:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28888.785747 secs 2023-03-09T22:59:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28888.785747 secs 2023-03-09T22:59:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28888.785747 secs 2023-03-09T22:59:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29350.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:59:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:39 smithi122 ceph-mon[93869]: pgmap v15919: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-09T22:59:39.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:39 smithi181 ceph-mon[94189]: pgmap v15919: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-09T22:59:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:39 smithi038 ceph-mon[92422]: pgmap v15919: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-09T22:59:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:41 smithi122 ceph-mon[93869]: pgmap v15920: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T22:59:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28891.397775 secs 2023-03-09T22:59:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:41 smithi181 ceph-mon[94189]: pgmap v15920: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T22:59:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28891.397775 secs 2023-03-09T22:59:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:41 smithi038 ceph-mon[92422]: pgmap v15920: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T22:59:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28891.397775 secs 2023-03-09T22:59:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:43 smithi122 ceph-mon[93869]: pgmap v15921: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 48 KiB/s wr, 8 op/s 2023-03-09T22:59:43.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:59:43.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28893.785837 secs 2023-03-09T22:59:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:43 smithi181 ceph-mon[94189]: pgmap v15921: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 48 KiB/s wr, 8 op/s 2023-03-09T22:59:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:59:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28893.785837 secs 2023-03-09T22:59:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:43 smithi038 ceph-mon[92422]: pgmap v15921: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 48 KiB/s wr, 8 op/s 2023-03-09T22:59:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T22:59:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28893.785837 secs 2023-03-09T22:59:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29355.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:59:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:45 smithi122 ceph-mon[93869]: pgmap v15922: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 48 KiB/s wr, 6 op/s 2023-03-09T22:59:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28896.397836 secs 2023-03-09T22:59:45.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:59:45.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:59:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:45 smithi181 ceph-mon[94189]: pgmap v15922: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 48 KiB/s wr, 6 op/s 2023-03-09T22:59:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28896.397836 secs 2023-03-09T22:59:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:59:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:59:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:45 smithi038 ceph-mon[92422]: pgmap v15922: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 48 KiB/s wr, 6 op/s 2023-03-09T22:59:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28896.397836 secs 2023-03-09T22:59:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:59:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:59:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:59:46.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:59:46.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:59:46.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:59:46.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:59:46.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:59:46.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:59:46.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:59:46.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:59:46.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:59:46.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:59:46.503 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:59:46.503 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:59:46.503 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:59:46.503 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:59:46.504 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:59:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:59:46.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:59:46.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:59:46.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:59:46.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:59:46.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:59:46.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:59:46.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:59:46.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:59:46.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:59:46.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:59:46.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:59:46.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:59:46.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:59:46.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:59:46.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:59:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:59:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:59:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:59:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:59:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:59:46.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:59:46.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:59:46.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:59:46.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:59:46.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:59:46.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:59:46.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:59:46.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T22:59:46.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T22:59:46.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T22:59:46.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T22:59:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:47 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T22:59:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:47 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T22:59:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:47 smithi122 ceph-mon[93869]: pgmap v15923: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 48 KiB/s wr, 8 op/s 2023-03-09T22:59:47.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:47 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T22:59:47.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:47 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T22:59:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:47 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T22:59:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:47 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T22:59:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:47 smithi181 ceph-mon[94189]: pgmap v15923: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 48 KiB/s wr, 8 op/s 2023-03-09T22:59:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:47 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T22:59:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:47 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T22:59:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:47 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T22:59:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:47 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T22:59:47.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:47 smithi038 ceph-mon[92422]: pgmap v15923: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 48 KiB/s wr, 8 op/s 2023-03-09T22:59:47.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:47 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T22:59:47.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:47 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T22:59:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:59:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:59:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28898.785958 secs 2023-03-09T22:59:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:59:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:59:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28898.785958 secs 2023-03-09T22:59:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:59:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:59:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28898.785958 secs 2023-03-09T22:59:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29360.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:59:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:49 smithi122 ceph-mon[93869]: pgmap v15924: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 720 KiB/s rd, 390 KiB/s wr, 7 op/s 2023-03-09T22:59:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:59:49.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:59:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:49 smithi181 ceph-mon[94189]: pgmap v15924: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 720 KiB/s rd, 390 KiB/s wr, 7 op/s 2023-03-09T22:59:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:59:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:59:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:49 smithi038 ceph-mon[92422]: pgmap v15924: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 720 KiB/s rd, 390 KiB/s wr, 7 op/s 2023-03-09T22:59:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:59:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:59:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28901.397898 secs 2023-03-09T22:59:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28901.397898 secs 2023-03-09T22:59:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28901.397898 secs 2023-03-09T22:59:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:51 smithi122 ceph-mon[93869]: pgmap v15925: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 414 KiB/s wr, 8 op/s 2023-03-09T22:59:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:51 smithi181 ceph-mon[94189]: pgmap v15925: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 414 KiB/s wr, 8 op/s 2023-03-09T22:59:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:51 smithi038 ceph-mon[92422]: pgmap v15925: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 414 KiB/s wr, 8 op/s 2023-03-09T22:59:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:53 smithi122 ceph-mon[93869]: pgmap v15926: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 746 KiB/s rd, 414 KiB/s wr, 10 op/s 2023-03-09T22:59:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:53 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:59:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:53 smithi181 ceph-mon[94189]: pgmap v15926: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 746 KiB/s rd, 414 KiB/s wr, 10 op/s 2023-03-09T22:59:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:53 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:59:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:53 smithi038 ceph-mon[92422]: pgmap v15926: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 746 KiB/s rd, 414 KiB/s wr, 10 op/s 2023-03-09T22:59:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:53 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T22:59:53.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29365.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:59:54.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:54 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:59:54.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:54 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:59:54.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:54 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:59:54.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:54 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:59:54.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:54 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:59:54.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:54 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:59:54.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:54 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T22:59:54.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:54 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T22:59:54.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:54 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:59:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:55 smithi122 ceph-mon[93869]: pgmap v15927: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 735 KiB/s wr, 11 op/s 2023-03-09T22:59:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28906.397961 secs 2023-03-09T22:59:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:55 smithi181 ceph-mon[94189]: pgmap v15927: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 735 KiB/s wr, 11 op/s 2023-03-09T22:59:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28906.397961 secs 2023-03-09T22:59:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:55 smithi038 ceph-mon[92422]: pgmap v15927: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 735 KiB/s wr, 11 op/s 2023-03-09T22:59:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28906.397961 secs 2023-03-09T22:59:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:59:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:59:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:59:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:59:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T22:59:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:59:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:57 smithi122 ceph-mon[93869]: pgmap v15928: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 736 KiB/s wr, 14 op/s 2023-03-09T22:59:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:59:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:59:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:57 smithi181 ceph-mon[94189]: pgmap v15928: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 736 KiB/s wr, 14 op/s 2023-03-09T22:59:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:59:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:59:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:57 smithi038 ceph-mon[92422]: pgmap v15928: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 736 KiB/s wr, 14 op/s 2023-03-09T22:59:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T22:59:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T22:59:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29370.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T22:59:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 22:59:59 smithi122 ceph-mon[93869]: pgmap v15929: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 844 KiB/s wr, 13 op/s 2023-03-09T22:59:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 22:59:59 smithi181 ceph-mon[94189]: pgmap v15929: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 844 KiB/s wr, 13 op/s 2023-03-09T22:59:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 22:59:59 smithi038 ceph-mon[92422]: pgmap v15929: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 844 KiB/s wr, 13 op/s 2023-03-09T23:00:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28911.397982 secs 2023-03-09T23:00:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:00 smithi122 ceph-mon[93869]: overall HEALTH_WARN 1 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-09T23:00:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28911.397982 secs 2023-03-09T23:00:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:00 smithi181 ceph-mon[94189]: overall HEALTH_WARN 1 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-09T23:00:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28911.397982 secs 2023-03-09T23:00:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:00 smithi038 ceph-mon[92422]: overall HEALTH_WARN 1 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-09T23:00:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:01 smithi122 ceph-mon[93869]: pgmap v15930: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 838 KiB/s rd, 546 KiB/s wr, 13 op/s 2023-03-09T23:00:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:01 smithi181 ceph-mon[94189]: pgmap v15930: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 838 KiB/s rd, 546 KiB/s wr, 13 op/s 2023-03-09T23:00:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:01 smithi038 ceph-mon[92422]: pgmap v15930: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 838 KiB/s rd, 546 KiB/s wr, 13 op/s 2023-03-09T23:00:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:03 smithi122 ceph-mon[93869]: pgmap v15931: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 817 KiB/s rd, 534 KiB/s wr, 15 op/s 2023-03-09T23:00:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28913.786267 secs 2023-03-09T23:00:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:03 smithi181 ceph-mon[94189]: pgmap v15931: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 817 KiB/s rd, 534 KiB/s wr, 15 op/s 2023-03-09T23:00:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28913.786267 secs 2023-03-09T23:00:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:03 smithi038 ceph-mon[92422]: pgmap v15931: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 817 KiB/s rd, 534 KiB/s wr, 15 op/s 2023-03-09T23:00:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28913.786267 secs 2023-03-09T23:00:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29375.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:00:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:05 smithi122 ceph-mon[93869]: pgmap v15932: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 837 KiB/s wr, 28 op/s 2023-03-09T23:00:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:05 smithi122 ceph-mon[93869]: osdmap e1211: 12 total, 12 up, 12 in 2023-03-09T23:00:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28916.398041 secs 2023-03-09T23:00:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:05 smithi181 ceph-mon[94189]: pgmap v15932: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 837 KiB/s wr, 28 op/s 2023-03-09T23:00:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:05 smithi181 ceph-mon[94189]: osdmap e1211: 12 total, 12 up, 12 in 2023-03-09T23:00:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28916.398041 secs 2023-03-09T23:00:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:05 smithi038 ceph-mon[92422]: pgmap v15932: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 837 KiB/s wr, 28 op/s 2023-03-09T23:00:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:05 smithi038 ceph-mon[92422]: osdmap e1211: 12 total, 12 up, 12 in 2023-03-09T23:00:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28916.398041 secs 2023-03-09T23:00:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:06 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:00:06.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:06 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:00:06.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:00:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:06 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:00:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:06 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:00:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:00:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:06 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:00:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:06 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:00:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:00:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:07 smithi122 ceph-mon[93869]: pgmap v15934: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 575 KiB/s wr, 29 op/s 2023-03-09T23:00:07.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:07 smithi181 ceph-mon[94189]: pgmap v15934: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 575 KiB/s wr, 29 op/s 2023-03-09T23:00:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:07 smithi038 ceph-mon[92422]: pgmap v15934: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 575 KiB/s wr, 29 op/s 2023-03-09T23:00:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28918.786391 secs 2023-03-09T23:00:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:00:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:00:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28918.786391 secs 2023-03-09T23:00:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:00:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:00:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28918.786391 secs 2023-03-09T23:00:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:00:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:00:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29380.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:00:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:09 smithi122 ceph-mon[93869]: pgmap v15935: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 641 KiB/s wr, 30 op/s 2023-03-09T23:00:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:00:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:00:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:09 smithi181 ceph-mon[94189]: pgmap v15935: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 641 KiB/s wr, 30 op/s 2023-03-09T23:00:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:00:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:00:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:09 smithi038 ceph-mon[92422]: pgmap v15935: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 641 KiB/s wr, 30 op/s 2023-03-09T23:00:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:00:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:00:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28921.398103 secs 2023-03-09T23:00:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28921.398103 secs 2023-03-09T23:00:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28921.398103 secs 2023-03-09T23:00:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:11 smithi122 ceph-mon[93869]: pgmap v15936: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 547 KiB/s rd, 588 KiB/s wr, 29 op/s 2023-03-09T23:00:11.962 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:11 smithi038 ceph-mon[92422]: pgmap v15936: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 547 KiB/s rd, 588 KiB/s wr, 29 op/s 2023-03-09T23:00:11.962 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:11 smithi181 ceph-mon[94189]: pgmap v15936: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 547 KiB/s rd, 588 KiB/s wr, 29 op/s 2023-03-09T23:00:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:12 smithi122 ceph-mon[93869]: pgmap v15937: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 546 KiB/s rd, 574 KiB/s wr, 27 op/s 2023-03-09T23:00:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:12 smithi122 ceph-mon[93869]: osdmap e1212: 12 total, 12 up, 12 in 2023-03-09T23:00:13.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28923.786508 secs 2023-03-09T23:00:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:12 smithi181 ceph-mon[94189]: pgmap v15937: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 546 KiB/s rd, 574 KiB/s wr, 27 op/s 2023-03-09T23:00:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:12 smithi181 ceph-mon[94189]: osdmap e1212: 12 total, 12 up, 12 in 2023-03-09T23:00:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28923.786508 secs 2023-03-09T23:00:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:12 smithi038 ceph-mon[92422]: pgmap v15937: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 546 KiB/s rd, 574 KiB/s wr, 27 op/s 2023-03-09T23:00:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:12 smithi038 ceph-mon[92422]: osdmap e1212: 12 total, 12 up, 12 in 2023-03-09T23:00:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28923.786508 secs 2023-03-09T23:00:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29385.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:00:15.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:14 smithi122 ceph-mon[93869]: pgmap v15939: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 378 KiB/s wr, 12 op/s 2023-03-09T23:00:15.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:14 smithi181 ceph-mon[94189]: pgmap v15939: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 378 KiB/s wr, 12 op/s 2023-03-09T23:00:15.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:14 smithi038 ceph-mon[92422]: pgmap v15939: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 378 KiB/s wr, 12 op/s 2023-03-09T23:00:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28926.398163 secs 2023-03-09T23:00:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28926.398163 secs 2023-03-09T23:00:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28926.398163 secs 2023-03-09T23:00:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:16 smithi122 ceph-mon[93869]: pgmap v15940: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 342 KiB/s wr, 9 op/s 2023-03-09T23:00:17.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:16 smithi181 ceph-mon[94189]: pgmap v15940: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 342 KiB/s wr, 9 op/s 2023-03-09T23:00:17.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:16 smithi038 ceph-mon[92422]: pgmap v15940: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 342 KiB/s wr, 9 op/s 2023-03-09T23:00:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28928.786623 secs 2023-03-09T23:00:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28928.786623 secs 2023-03-09T23:00:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28928.786623 secs 2023-03-09T23:00:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29390.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:00:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:18 smithi122 ceph-mon[93869]: pgmap v15941: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 145 KiB/s wr, 8 op/s 2023-03-09T23:00:19.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:18 smithi181 ceph-mon[94189]: pgmap v15941: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 145 KiB/s wr, 8 op/s 2023-03-09T23:00:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:18 smithi038 ceph-mon[92422]: pgmap v15941: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 145 KiB/s wr, 8 op/s 2023-03-09T23:00:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:20 smithi122 ceph-mon[93869]: pgmap v15942: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 145 KiB/s wr, 7 op/s 2023-03-09T23:00:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28931.398222 secs 2023-03-09T23:00:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:20 smithi181 ceph-mon[94189]: pgmap v15942: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 145 KiB/s wr, 7 op/s 2023-03-09T23:00:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28931.398222 secs 2023-03-09T23:00:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:20 smithi038 ceph-mon[92422]: pgmap v15942: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 145 KiB/s wr, 7 op/s 2023-03-09T23:00:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28931.398222 secs 2023-03-09T23:00:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:22 smithi122 ceph-mon[93869]: pgmap v15943: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 145 KiB/s wr, 7 op/s 2023-03-09T23:00:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28933.786757 secs 2023-03-09T23:00:23.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:22 smithi181 ceph-mon[94189]: pgmap v15943: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 145 KiB/s wr, 7 op/s 2023-03-09T23:00:23.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28933.786757 secs 2023-03-09T23:00:23.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:22 smithi038 ceph-mon[92422]: pgmap v15943: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 145 KiB/s wr, 7 op/s 2023-03-09T23:00:23.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28933.786757 secs 2023-03-09T23:00:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29395.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:00:25.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:24 smithi122 ceph-mon[93869]: pgmap v15944: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T23:00:25.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:24 smithi181 ceph-mon[94189]: pgmap v15944: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T23:00:25.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:24 smithi038 ceph-mon[92422]: pgmap v15944: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T23:00:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28936.398284 secs 2023-03-09T23:00:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28936.398284 secs 2023-03-09T23:00:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28936.398284 secs 2023-03-09T23:00:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:26 smithi122 ceph-mon[93869]: pgmap v15945: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T23:00:27.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:26 smithi181 ceph-mon[94189]: pgmap v15945: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T23:00:27.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:26 smithi038 ceph-mon[92422]: pgmap v15945: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T23:00:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28938.786859 secs 2023-03-09T23:00:28.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28938.786859 secs 2023-03-09T23:00:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28938.786859 secs 2023-03-09T23:00:28.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29400.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:00:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:28 smithi122 ceph-mon[93869]: pgmap v15946: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 7 op/s 2023-03-09T23:00:29.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:28 smithi181 ceph-mon[94189]: pgmap v15946: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 7 op/s 2023-03-09T23:00:29.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:28 smithi038 ceph-mon[92422]: pgmap v15946: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 7 op/s 2023-03-09T23:00:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:30 smithi122 ceph-mon[93869]: pgmap v15947: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T23:00:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28941.398344 secs 2023-03-09T23:00:31.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:30 smithi181 ceph-mon[94189]: pgmap v15947: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T23:00:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28941.398344 secs 2023-03-09T23:00:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:30 smithi038 ceph-mon[92422]: pgmap v15947: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T23:00:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28941.398344 secs 2023-03-09T23:00:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:32 smithi122 ceph-mon[93869]: pgmap v15948: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2023-03-09T23:00:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28943.786976 secs 2023-03-09T23:00:33.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:32 smithi181 ceph-mon[94189]: pgmap v15948: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2023-03-09T23:00:33.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28943.786976 secs 2023-03-09T23:00:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:32 smithi038 ceph-mon[92422]: pgmap v15948: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2023-03-09T23:00:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28943.786976 secs 2023-03-09T23:00:33.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29405.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:00:35.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:34 smithi122 ceph-mon[93869]: pgmap v15949: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 103 KiB/s wr, 8 op/s 2023-03-09T23:00:35.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:34 smithi181 ceph-mon[94189]: pgmap v15949: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 103 KiB/s wr, 8 op/s 2023-03-09T23:00:35.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:34 smithi038 ceph-mon[92422]: pgmap v15949: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 103 KiB/s wr, 8 op/s 2023-03-09T23:00:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28946.398405 secs 2023-03-09T23:00:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28946.398405 secs 2023-03-09T23:00:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28946.398405 secs 2023-03-09T23:00:37.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:36 smithi122 ceph-mon[93869]: pgmap v15950: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 103 KiB/s wr, 8 op/s 2023-03-09T23:00:37.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:36 smithi181 ceph-mon[94189]: pgmap v15950: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 103 KiB/s wr, 8 op/s 2023-03-09T23:00:37.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:36 smithi038 ceph-mon[92422]: pgmap v15950: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 103 KiB/s wr, 8 op/s 2023-03-09T23:00:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28948.787123 secs 2023-03-09T23:00:38.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28948.787123 secs 2023-03-09T23:00:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28948.787123 secs 2023-03-09T23:00:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29410.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:00:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:38 smithi122 ceph-mon[93869]: pgmap v15951: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 103 KiB/s wr, 8 op/s 2023-03-09T23:00:39.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:38 smithi181 ceph-mon[94189]: pgmap v15951: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 103 KiB/s wr, 8 op/s 2023-03-09T23:00:39.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:38 smithi038 ceph-mon[92422]: pgmap v15951: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 103 KiB/s wr, 8 op/s 2023-03-09T23:00:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:39 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:00:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:39 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:00:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:39 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:00:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:39 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:00:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:39 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:00:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:39 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:00:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:40 smithi122 ceph-mon[93869]: pgmap v15952: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 227 KiB/s wr, 7 op/s 2023-03-09T23:00:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:00:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:00:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28951.398467 secs 2023-03-09T23:00:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:40 smithi181 ceph-mon[94189]: pgmap v15952: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 227 KiB/s wr, 7 op/s 2023-03-09T23:00:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:00:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:00:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28951.398467 secs 2023-03-09T23:00:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:40 smithi038 ceph-mon[92422]: pgmap v15952: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 227 KiB/s wr, 7 op/s 2023-03-09T23:00:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:00:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:00:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28951.398467 secs 2023-03-09T23:00:43.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:42 smithi122 ceph-mon[93869]: pgmap v15953: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 227 KiB/s wr, 9 op/s 2023-03-09T23:00:43.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:42 smithi181 ceph-mon[94189]: pgmap v15953: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 227 KiB/s wr, 9 op/s 2023-03-09T23:00:43.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:42 smithi038 ceph-mon[92422]: pgmap v15953: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 227 KiB/s wr, 9 op/s 2023-03-09T23:00:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29415.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:00:45.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:44 smithi122 ceph-mon[93869]: pgmap v15954: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 227 KiB/s wr, 9 op/s 2023-03-09T23:00:45.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:44 smithi181 ceph-mon[94189]: pgmap v15954: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 227 KiB/s wr, 9 op/s 2023-03-09T23:00:45.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:44 smithi038 ceph-mon[92422]: pgmap v15954: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 227 KiB/s wr, 9 op/s 2023-03-09T23:00:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28956.398528 secs 2023-03-09T23:00:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:45 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:00:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28956.398528 secs 2023-03-09T23:00:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:45 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:00:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28956.398528 secs 2023-03-09T23:00:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:45 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:00:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:46 smithi122 ceph-mon[93869]: pgmap v15955: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 125 KiB/s wr, 8 op/s 2023-03-09T23:00:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:46 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:00:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:46 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:00:47.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:46 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:00:47.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:00:47.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:00:47.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:00:47.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:00:47.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:46 smithi181 ceph-mon[94189]: pgmap v15955: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 125 KiB/s wr, 8 op/s 2023-03-09T23:00:47.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:46 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:00:47.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:46 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:00:47.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:46 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:00:47.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:00:47.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:00:47.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:00:47.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:00:47.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:46 smithi038 ceph-mon[92422]: pgmap v15955: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 125 KiB/s wr, 8 op/s 2023-03-09T23:00:47.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:46 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:00:47.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:46 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:00:47.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:46 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:00:47.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:00:47.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:00:47.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:00:47.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:00:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29420.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:00:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:48 smithi122 ceph-mon[93869]: pgmap v15956: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 125 KiB/s wr, 8 op/s 2023-03-09T23:00:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:48 smithi181 ceph-mon[94189]: pgmap v15956: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 125 KiB/s wr, 8 op/s 2023-03-09T23:00:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:48 smithi038 ceph-mon[92422]: pgmap v15956: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 125 KiB/s wr, 8 op/s 2023-03-09T23:00:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:50 smithi122 ceph-mon[93869]: pgmap v15957: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 272 KiB/s wr, 7 op/s 2023-03-09T23:00:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28961.398613 secs 2023-03-09T23:00:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:50 smithi181 ceph-mon[94189]: pgmap v15957: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 272 KiB/s wr, 7 op/s 2023-03-09T23:00:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28961.398613 secs 2023-03-09T23:00:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:50 smithi038 ceph-mon[92422]: pgmap v15957: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 272 KiB/s wr, 7 op/s 2023-03-09T23:00:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28961.398613 secs 2023-03-09T23:00:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:52 smithi122 ceph-mon[93869]: pgmap v15958: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 397 KiB/s wr, 9 op/s 2023-03-09T23:00:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:00:53.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:00:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:52 smithi181 ceph-mon[94189]: pgmap v15958: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 397 KiB/s wr, 9 op/s 2023-03-09T23:00:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:00:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:00:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:52 smithi038 ceph-mon[92422]: pgmap v15958: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 397 KiB/s wr, 9 op/s 2023-03-09T23:00:53.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:00:53.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:00:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29425.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:00:55.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:54 smithi122 ceph-mon[93869]: pgmap v15959: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 470 KiB/s rd, 420 KiB/s wr, 9 op/s 2023-03-09T23:00:55.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:54 smithi181 ceph-mon[94189]: pgmap v15959: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 470 KiB/s rd, 420 KiB/s wr, 9 op/s 2023-03-09T23:00:55.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:54 smithi038 ceph-mon[92422]: pgmap v15959: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 470 KiB/s rd, 420 KiB/s wr, 9 op/s 2023-03-09T23:00:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28966.398687 secs 2023-03-09T23:00:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:00:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:00:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28966.398687 secs 2023-03-09T23:00:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:00:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:00:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28966.398687 secs 2023-03-09T23:00:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:00:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:00:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:56 smithi038 ceph-mon[92422]: pgmap v15960: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 566 KiB/s rd, 514 KiB/s wr, 10 op/s 2023-03-09T23:00:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:56 smithi122 ceph-mon[93869]: pgmap v15960: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 566 KiB/s rd, 514 KiB/s wr, 10 op/s 2023-03-09T23:00:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:56 smithi181 ceph-mon[94189]: pgmap v15960: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 566 KiB/s rd, 514 KiB/s wr, 10 op/s 2023-03-09T23:00:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:57 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 28968.787540 secs 2023-03-09T23:00:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:57 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 28968.787540 secs 2023-03-09T23:00:58.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:57 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 28968.787540 secs 2023-03-09T23:00:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29430.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:00:59.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:00:58 smithi038 ceph-mon[92422]: pgmap v15961: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 591 KiB/s rd, 540 KiB/s wr, 9 op/s 2023-03-09T23:00:59.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:00:58 smithi122 ceph-mon[93869]: pgmap v15961: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 591 KiB/s rd, 540 KiB/s wr, 9 op/s 2023-03-09T23:00:59.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:00:58 smithi181 ceph-mon[94189]: pgmap v15961: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 591 KiB/s rd, 540 KiB/s wr, 9 op/s 2023-03-09T23:01:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:01:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:01:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:01:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:01:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:01:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:01:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:01 smithi122 ceph-mon[93869]: pgmap v15962: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 735 KiB/s rd, 853 KiB/s wr, 10 op/s 2023-03-09T23:01:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28971.398750 secs 2023-03-09T23:01:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:01:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:01:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:01 smithi181 ceph-mon[94189]: pgmap v15962: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 735 KiB/s rd, 853 KiB/s wr, 10 op/s 2023-03-09T23:01:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28971.398750 secs 2023-03-09T23:01:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:01:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:01:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:01 smithi038 ceph-mon[92422]: pgmap v15962: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 735 KiB/s rd, 853 KiB/s wr, 10 op/s 2023-03-09T23:01:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28971.398750 secs 2023-03-09T23:01:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:01:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:01:03.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:02 smithi122 ceph-mon[93869]: pgmap v15963: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 560 KiB/s rd, 706 KiB/s wr, 12 op/s 2023-03-09T23:01:03.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:02 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:01:03.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:02 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:01:03.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:02 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:01:03.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:02 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 28973.787664 secs 2023-03-09T23:01:03.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:02 smithi181 ceph-mon[94189]: pgmap v15963: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 560 KiB/s rd, 706 KiB/s wr, 12 op/s 2023-03-09T23:01:03.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:02 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:01:03.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:02 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:01:03.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:02 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:01:03.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:02 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 28973.787664 secs 2023-03-09T23:01:03.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:02 smithi038 ceph-mon[92422]: pgmap v15963: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 560 KiB/s rd, 706 KiB/s wr, 12 op/s 2023-03-09T23:01:03.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:02 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:01:03.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:02 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:01:03.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:02 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:01:03.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:02 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 28973.787664 secs 2023-03-09T23:01:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29435.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:01:05.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:04 smithi122 ceph-mon[93869]: pgmap v15964: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 671 KiB/s rd, 638 KiB/s wr, 14 op/s 2023-03-09T23:01:05.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:04 smithi181 ceph-mon[94189]: pgmap v15964: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 671 KiB/s rd, 638 KiB/s wr, 14 op/s 2023-03-09T23:01:05.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:04 smithi038 ceph-mon[92422]: pgmap v15964: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 671 KiB/s rd, 638 KiB/s wr, 14 op/s 2023-03-09T23:01:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28976.398814 secs 2023-03-09T23:01:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28976.398814 secs 2023-03-09T23:01:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28976.398814 secs 2023-03-09T23:01:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:06 smithi122 ceph-mon[93869]: pgmap v15965: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 631 KiB/s rd, 637 KiB/s wr, 16 op/s 2023-03-09T23:01:07.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:06 smithi181 ceph-mon[94189]: pgmap v15965: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 631 KiB/s rd, 637 KiB/s wr, 16 op/s 2023-03-09T23:01:07.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:06 smithi038 ceph-mon[92422]: pgmap v15965: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 631 KiB/s rd, 637 KiB/s wr, 16 op/s 2023-03-09T23:01:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:01:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:01:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28978.787808 secs 2023-03-09T23:01:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:01:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:01:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28978.787808 secs 2023-03-09T23:01:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:01:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:01:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28978.787808 secs 2023-03-09T23:01:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29440.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:01:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:08 smithi122 ceph-mon[93869]: pgmap v15966: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 649 KiB/s rd, 658 KiB/s wr, 15 op/s 2023-03-09T23:01:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:01:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:01:09.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:08 smithi181 ceph-mon[94189]: pgmap v15966: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 649 KiB/s rd, 658 KiB/s wr, 15 op/s 2023-03-09T23:01:09.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:01:09.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:01:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:08 smithi038 ceph-mon[92422]: pgmap v15966: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 649 KiB/s rd, 658 KiB/s wr, 15 op/s 2023-03-09T23:01:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:01:09.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:01:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:10 smithi122 ceph-mon[93869]: pgmap v15967: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 630 KiB/s rd, 653 KiB/s wr, 26 op/s 2023-03-09T23:01:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:10 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:01:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:10 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:01:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:10 smithi122 ceph-mon[93869]: osdmap e1213: 12 total, 12 up, 12 in 2023-03-09T23:01:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28981.398878 secs 2023-03-09T23:01:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:10 smithi181 ceph-mon[94189]: pgmap v15967: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 630 KiB/s rd, 653 KiB/s wr, 26 op/s 2023-03-09T23:01:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:10 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:01:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:10 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:01:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:10 smithi181 ceph-mon[94189]: osdmap e1213: 12 total, 12 up, 12 in 2023-03-09T23:01:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28981.398878 secs 2023-03-09T23:01:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:10 smithi038 ceph-mon[92422]: pgmap v15967: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 630 KiB/s rd, 653 KiB/s wr, 26 op/s 2023-03-09T23:01:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:10 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:01:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:10 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:01:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:10 smithi038 ceph-mon[92422]: osdmap e1213: 12 total, 12 up, 12 in 2023-03-09T23:01:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28981.398878 secs 2023-03-09T23:01:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:12 smithi122 ceph-mon[93869]: pgmap v15969: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 752 KiB/s rd, 577 KiB/s wr, 29 op/s 2023-03-09T23:01:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28983.787921 secs 2023-03-09T23:01:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:12 smithi181 ceph-mon[94189]: pgmap v15969: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 752 KiB/s rd, 577 KiB/s wr, 29 op/s 2023-03-09T23:01:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28983.787921 secs 2023-03-09T23:01:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:12 smithi038 ceph-mon[92422]: pgmap v15969: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 752 KiB/s rd, 577 KiB/s wr, 29 op/s 2023-03-09T23:01:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28983.787921 secs 2023-03-09T23:01:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29445.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:01:15.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:14 smithi122 ceph-mon[93869]: pgmap v15970: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 533 KiB/s wr, 27 op/s 2023-03-09T23:01:15.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:14 smithi181 ceph-mon[94189]: pgmap v15970: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 533 KiB/s wr, 27 op/s 2023-03-09T23:01:15.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:14 smithi038 ceph-mon[92422]: pgmap v15970: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 533 KiB/s wr, 27 op/s 2023-03-09T23:01:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:15 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:01:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:15 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:01:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28986.398939 secs 2023-03-09T23:01:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:15 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:01:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:15 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:01:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28986.398939 secs 2023-03-09T23:01:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:15 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:01:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:15 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:01:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28986.398939 secs 2023-03-09T23:01:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:16 smithi122 ceph-mon[93869]: pgmap v15971: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 592 KiB/s rd, 608 KiB/s wr, 25 op/s 2023-03-09T23:01:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:01:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:01:17.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:16 smithi122 ceph-mon[93869]: osdmap e1214: 12 total, 12 up, 12 in 2023-03-09T23:01:17.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:16 smithi181 ceph-mon[94189]: pgmap v15971: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 592 KiB/s rd, 608 KiB/s wr, 25 op/s 2023-03-09T23:01:17.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:01:17.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:01:17.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:16 smithi181 ceph-mon[94189]: osdmap e1214: 12 total, 12 up, 12 in 2023-03-09T23:01:17.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:16 smithi038 ceph-mon[92422]: pgmap v15971: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 592 KiB/s rd, 608 KiB/s wr, 25 op/s 2023-03-09T23:01:17.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:01:17.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:01:17.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:16 smithi038 ceph-mon[92422]: osdmap e1214: 12 total, 12 up, 12 in 2023-03-09T23:01:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28988.788020 secs 2023-03-09T23:01:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28988.788020 secs 2023-03-09T23:01:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28988.788020 secs 2023-03-09T23:01:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29450.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:01:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:18 smithi122 ceph-mon[93869]: pgmap v15973: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 558 KiB/s rd, 556 KiB/s wr, 12 op/s 2023-03-09T23:01:19.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:18 smithi181 ceph-mon[94189]: pgmap v15973: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 558 KiB/s rd, 556 KiB/s wr, 12 op/s 2023-03-09T23:01:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:18 smithi038 ceph-mon[92422]: pgmap v15973: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 558 KiB/s rd, 556 KiB/s wr, 12 op/s 2023-03-09T23:01:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:20 smithi122 ceph-mon[93869]: pgmap v15974: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 449 KiB/s wr, 10 op/s 2023-03-09T23:01:21.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28991.398987 secs 2023-03-09T23:01:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:20 smithi181 ceph-mon[94189]: pgmap v15974: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 449 KiB/s wr, 10 op/s 2023-03-09T23:01:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28991.398987 secs 2023-03-09T23:01:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:20 smithi038 ceph-mon[92422]: pgmap v15974: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 449 KiB/s wr, 10 op/s 2023-03-09T23:01:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28991.398987 secs 2023-03-09T23:01:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:22 smithi122 ceph-mon[93869]: pgmap v15975: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 274 KiB/s wr, 9 op/s 2023-03-09T23:01:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28993.788119 secs 2023-03-09T23:01:23.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:22 smithi181 ceph-mon[94189]: pgmap v15975: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 274 KiB/s wr, 9 op/s 2023-03-09T23:01:23.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28993.788119 secs 2023-03-09T23:01:23.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:22 smithi038 ceph-mon[92422]: pgmap v15975: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 274 KiB/s wr, 9 op/s 2023-03-09T23:01:23.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28993.788119 secs 2023-03-09T23:01:23.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29455.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:01:25.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:24 smithi122 ceph-mon[93869]: pgmap v15976: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 103 KiB/s wr, 7 op/s 2023-03-09T23:01:25.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:24 smithi181 ceph-mon[94189]: pgmap v15976: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 103 KiB/s wr, 7 op/s 2023-03-09T23:01:25.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:24 smithi038 ceph-mon[92422]: pgmap v15976: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 103 KiB/s wr, 7 op/s 2023-03-09T23:01:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28996.399050 secs 2023-03-09T23:01:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28996.399050 secs 2023-03-09T23:01:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28996.399050 secs 2023-03-09T23:01:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:26 smithi122 ceph-mon[93869]: pgmap v15977: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 7 op/s 2023-03-09T23:01:27.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:26 smithi181 ceph-mon[94189]: pgmap v15977: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 7 op/s 2023-03-09T23:01:27.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:26 smithi038 ceph-mon[92422]: pgmap v15977: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 7 op/s 2023-03-09T23:01:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 28998.788185 secs 2023-03-09T23:01:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 28998.788185 secs 2023-03-09T23:01:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 28998.788185 secs 2023-03-09T23:01:28.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29460.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:01:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:28 smithi122 ceph-mon[93869]: pgmap v15978: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T23:01:29.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:28 smithi181 ceph-mon[94189]: pgmap v15978: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T23:01:29.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:28 smithi038 ceph-mon[92422]: pgmap v15978: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T23:01:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:30 smithi122 ceph-mon[93869]: pgmap v15979: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T23:01:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29001.399109 secs 2023-03-09T23:01:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:30 smithi181 ceph-mon[94189]: pgmap v15979: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T23:01:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29001.399109 secs 2023-03-09T23:01:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:30 smithi038 ceph-mon[92422]: pgmap v15979: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T23:01:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29001.399109 secs 2023-03-09T23:01:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:32 smithi122 ceph-mon[93869]: pgmap v15980: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T23:01:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29003.788305 secs 2023-03-09T23:01:33.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:32 smithi181 ceph-mon[94189]: pgmap v15980: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T23:01:33.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29003.788305 secs 2023-03-09T23:01:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:32 smithi038 ceph-mon[92422]: pgmap v15980: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T23:01:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29003.788305 secs 2023-03-09T23:01:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29465.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:01:35.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:34 smithi038 ceph-mon[92422]: pgmap v15981: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T23:01:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:34 smithi122 ceph-mon[93869]: pgmap v15981: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T23:01:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:34 smithi181 ceph-mon[94189]: pgmap v15981: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T23:01:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29006.399168 secs 2023-03-09T23:01:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29006.399168 secs 2023-03-09T23:01:36.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29006.399168 secs 2023-03-09T23:01:37.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:36 smithi038 ceph-mon[92422]: pgmap v15982: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2023-03-09T23:01:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:36 smithi122 ceph-mon[93869]: pgmap v15982: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2023-03-09T23:01:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:36 smithi181 ceph-mon[94189]: pgmap v15982: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2023-03-09T23:01:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29008.788442 secs 2023-03-09T23:01:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29008.788442 secs 2023-03-09T23:01:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29008.788442 secs 2023-03-09T23:01:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29470.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:01:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:39 smithi122 ceph-mon[93869]: pgmap v15983: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-09T23:01:39.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:39 smithi181 ceph-mon[94189]: pgmap v15983: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-09T23:01:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:39 smithi038 ceph-mon[92422]: pgmap v15983: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-09T23:01:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:41 smithi122 ceph-mon[93869]: pgmap v15984: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2023-03-09T23:01:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29011.399230 secs 2023-03-09T23:01:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:41 smithi181 ceph-mon[94189]: pgmap v15984: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2023-03-09T23:01:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29011.399230 secs 2023-03-09T23:01:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:41 smithi038 ceph-mon[92422]: pgmap v15984: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2023-03-09T23:01:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29011.399230 secs 2023-03-09T23:01:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:43 smithi122 ceph-mon[93869]: pgmap v15985: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 115 KiB/s wr, 8 op/s 2023-03-09T23:01:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29013.788582 secs 2023-03-09T23:01:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:43 smithi181 ceph-mon[94189]: pgmap v15985: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 115 KiB/s wr, 8 op/s 2023-03-09T23:01:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29013.788582 secs 2023-03-09T23:01:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:43 smithi038 ceph-mon[92422]: pgmap v15985: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 115 KiB/s wr, 8 op/s 2023-03-09T23:01:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29013.788582 secs 2023-03-09T23:01:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29475.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:01:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:45 smithi122 ceph-mon[93869]: pgmap v15986: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 125 KiB/s wr, 7 op/s 2023-03-09T23:01:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:45 smithi181 ceph-mon[94189]: pgmap v15986: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 125 KiB/s wr, 7 op/s 2023-03-09T23:01:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:45 smithi038 ceph-mon[92422]: pgmap v15986: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 125 KiB/s wr, 7 op/s 2023-03-09T23:01:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29016.399288 secs 2023-03-09T23:01:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29016.399288 secs 2023-03-09T23:01:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29016.399288 secs 2023-03-09T23:01:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:47 smithi122 ceph-mon[93869]: pgmap v15987: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 125 KiB/s wr, 8 op/s 2023-03-09T23:01:47.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:01:47.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:47 smithi181 ceph-mon[94189]: pgmap v15987: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 125 KiB/s wr, 8 op/s 2023-03-09T23:01:47.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:01:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:47 smithi038 ceph-mon[92422]: pgmap v15987: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 125 KiB/s wr, 8 op/s 2023-03-09T23:01:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:01:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29018.788677 secs 2023-03-09T23:01:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:01:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:01:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:01:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:01:48.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:01:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29018.788677 secs 2023-03-09T23:01:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:01:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:01:48.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:01:48.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:01:48.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:01:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29018.788677 secs 2023-03-09T23:01:48.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:01:48.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:01:48.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:01:48.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:01:48.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:01:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29480.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:01:49.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:49 smithi122 ceph-mon[93869]: pgmap v15988: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 263 KiB/s wr, 8 op/s 2023-03-09T23:01:49.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:01:49.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:01:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:49 smithi181 ceph-mon[94189]: pgmap v15988: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 263 KiB/s wr, 8 op/s 2023-03-09T23:01:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:01:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:01:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:49 smithi038 ceph-mon[92422]: pgmap v15988: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 263 KiB/s wr, 8 op/s 2023-03-09T23:01:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:01:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:01:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29021.399349 secs 2023-03-09T23:01:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29021.399349 secs 2023-03-09T23:01:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29021.399349 secs 2023-03-09T23:01:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:51 smithi122 ceph-mon[93869]: pgmap v15989: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 263 KiB/s wr, 7 op/s 2023-03-09T23:01:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:51 smithi181 ceph-mon[94189]: pgmap v15989: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 263 KiB/s wr, 7 op/s 2023-03-09T23:01:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:51 smithi038 ceph-mon[92422]: pgmap v15989: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 263 KiB/s wr, 7 op/s 2023-03-09T23:01:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:01:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:01:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:01:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:01:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:01:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:01:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:53 smithi122 ceph-mon[93869]: pgmap v15990: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 328 KiB/s wr, 9 op/s 2023-03-09T23:01:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29023.788802 secs 2023-03-09T23:01:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:53 smithi181 ceph-mon[94189]: pgmap v15990: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 328 KiB/s wr, 9 op/s 2023-03-09T23:01:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29023.788802 secs 2023-03-09T23:01:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:53 smithi038 ceph-mon[92422]: pgmap v15990: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 328 KiB/s wr, 9 op/s 2023-03-09T23:01:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29023.788802 secs 2023-03-09T23:01:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29485.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:01:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:55 smithi122 ceph-mon[93869]: pgmap v15991: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 432 KiB/s rd, 430 KiB/s wr, 9 op/s 2023-03-09T23:01:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29026.399408 secs 2023-03-09T23:01:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:55 smithi181 ceph-mon[94189]: pgmap v15991: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 432 KiB/s rd, 430 KiB/s wr, 9 op/s 2023-03-09T23:01:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29026.399408 secs 2023-03-09T23:01:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:55 smithi038 ceph-mon[92422]: pgmap v15991: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 432 KiB/s rd, 430 KiB/s wr, 9 op/s 2023-03-09T23:01:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29026.399408 secs 2023-03-09T23:01:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:01:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:01:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:01:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:01:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:01:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:01:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:57 smithi122 ceph-mon[93869]: pgmap v15992: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 652 KiB/s rd, 648 KiB/s wr, 11 op/s 2023-03-09T23:01:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:01:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:01:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:57 smithi181 ceph-mon[94189]: pgmap v15992: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 652 KiB/s rd, 648 KiB/s wr, 11 op/s 2023-03-09T23:01:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:01:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:01:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:57 smithi038 ceph-mon[92422]: pgmap v15992: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 652 KiB/s rd, 648 KiB/s wr, 11 op/s 2023-03-09T23:01:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:01:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:01:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29028.788922 secs 2023-03-09T23:01:58.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29490.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:01:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29028.788922 secs 2023-03-09T23:01:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29028.788922 secs 2023-03-09T23:01:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:01:59 smithi038 ceph-mon[92422]: pgmap v15993: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 677 KiB/s rd, 674 KiB/s wr, 10 op/s 2023-03-09T23:01:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:01:59 smithi122 ceph-mon[93869]: pgmap v15993: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 677 KiB/s rd, 674 KiB/s wr, 10 op/s 2023-03-09T23:01:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:01:59 smithi181 ceph-mon[94189]: pgmap v15993: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 677 KiB/s rd, 674 KiB/s wr, 10 op/s 2023-03-09T23:02:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:02:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:02:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29031.399467 secs 2023-03-09T23:02:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:02:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:02:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29031.399467 secs 2023-03-09T23:02:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:02:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:02:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29031.399467 secs 2023-03-09T23:02:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:01 smithi122 ceph-mon[93869]: pgmap v15994: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 680 KiB/s rd, 666 KiB/s wr, 10 op/s 2023-03-09T23:02:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:02:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:02:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:01 smithi181 ceph-mon[94189]: pgmap v15994: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 680 KiB/s rd, 666 KiB/s wr, 10 op/s 2023-03-09T23:02:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:02:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:02:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:01 smithi038 ceph-mon[92422]: pgmap v15994: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 680 KiB/s rd, 666 KiB/s wr, 10 op/s 2023-03-09T23:02:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:02:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:02:03.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29495.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:02:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:03 smithi122 ceph-mon[93869]: pgmap v15995: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 717 KiB/s wr, 13 op/s 2023-03-09T23:02:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29033.789021 secs 2023-03-09T23:02:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:03 smithi181 ceph-mon[94189]: pgmap v15995: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 717 KiB/s wr, 13 op/s 2023-03-09T23:02:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29033.789021 secs 2023-03-09T23:02:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:03 smithi038 ceph-mon[92422]: pgmap v15995: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 717 KiB/s wr, 13 op/s 2023-03-09T23:02:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29033.789021 secs 2023-03-09T23:02:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:05 smithi122 ceph-mon[93869]: pgmap v15996: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 691 KiB/s rd, 677 KiB/s wr, 13 op/s 2023-03-09T23:02:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29036.399528 secs 2023-03-09T23:02:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:05 smithi181 ceph-mon[94189]: pgmap v15996: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 691 KiB/s rd, 677 KiB/s wr, 13 op/s 2023-03-09T23:02:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29036.399528 secs 2023-03-09T23:02:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:05 smithi038 ceph-mon[92422]: pgmap v15996: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 691 KiB/s rd, 677 KiB/s wr, 13 op/s 2023-03-09T23:02:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29036.399528 secs 2023-03-09T23:02:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:07 smithi122 ceph-mon[93869]: pgmap v15997: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 638 KiB/s wr, 17 op/s 2023-03-09T23:02:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:07 smithi181 ceph-mon[94189]: pgmap v15997: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 638 KiB/s wr, 17 op/s 2023-03-09T23:02:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:07 smithi038 ceph-mon[92422]: pgmap v15997: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 638 KiB/s wr, 17 op/s 2023-03-09T23:02:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29500.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:02:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29038.789153 secs 2023-03-09T23:02:08.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29038.789153 secs 2023-03-09T23:02:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29038.789153 secs 2023-03-09T23:02:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:09 smithi122 ceph-mon[93869]: pgmap v15998: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 446 KiB/s rd, 454 KiB/s wr, 14 op/s 2023-03-09T23:02:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:09 smithi181 ceph-mon[94189]: pgmap v15998: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 446 KiB/s rd, 454 KiB/s wr, 14 op/s 2023-03-09T23:02:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:09 smithi038 ceph-mon[92422]: pgmap v15998: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 446 KiB/s rd, 454 KiB/s wr, 14 op/s 2023-03-09T23:02:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:10 smithi122 ceph-mon[93869]: osdmap e1215: 12 total, 12 up, 12 in 2023-03-09T23:02:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29041.399588 secs 2023-03-09T23:02:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:10 smithi181 ceph-mon[94189]: osdmap e1215: 12 total, 12 up, 12 in 2023-03-09T23:02:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29041.399588 secs 2023-03-09T23:02:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:10 smithi038 ceph-mon[92422]: osdmap e1215: 12 total, 12 up, 12 in 2023-03-09T23:02:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29041.399588 secs 2023-03-09T23:02:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:11 smithi122 ceph-mon[93869]: pgmap v16000: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 666 KiB/s wr, 29 op/s 2023-03-09T23:02:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:11 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:02:11.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:11 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:02:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:11 smithi181 ceph-mon[94189]: pgmap v16000: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 666 KiB/s wr, 29 op/s 2023-03-09T23:02:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:11 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:02:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:11 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:02:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:11 smithi038 ceph-mon[92422]: pgmap v16000: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 666 KiB/s wr, 29 op/s 2023-03-09T23:02:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:11 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:02:12.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:11 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:02:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:12 smithi122 ceph-mon[93869]: pgmap v16001: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 604 KiB/s wr, 29 op/s 2023-03-09T23:02:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29043.789261 secs 2023-03-09T23:02:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:02:13.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:02:13.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:12 smithi181 ceph-mon[94189]: pgmap v16001: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 604 KiB/s wr, 29 op/s 2023-03-09T23:02:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29043.789261 secs 2023-03-09T23:02:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:02:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:02:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:12 smithi038 ceph-mon[92422]: pgmap v16001: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 604 KiB/s wr, 29 op/s 2023-03-09T23:02:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29043.789261 secs 2023-03-09T23:02:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:02:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:02:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29505.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:02:15.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:14 smithi122 ceph-mon[93869]: pgmap v16002: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 679 KiB/s rd, 721 KiB/s wr, 28 op/s 2023-03-09T23:02:15.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:14 smithi181 ceph-mon[94189]: pgmap v16002: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 679 KiB/s rd, 721 KiB/s wr, 28 op/s 2023-03-09T23:02:15.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:14 smithi038 ceph-mon[92422]: pgmap v16002: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 679 KiB/s rd, 721 KiB/s wr, 28 op/s 2023-03-09T23:02:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29046.399649 secs 2023-03-09T23:02:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29046.399649 secs 2023-03-09T23:02:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29046.399649 secs 2023-03-09T23:02:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:16 smithi122 ceph-mon[93869]: pgmap v16003: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 507 KiB/s wr, 24 op/s 2023-03-09T23:02:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:16 smithi122 ceph-mon[93869]: osdmap e1216: 12 total, 12 up, 12 in 2023-03-09T23:02:17.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:16 smithi181 ceph-mon[94189]: pgmap v16003: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 507 KiB/s wr, 24 op/s 2023-03-09T23:02:17.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:16 smithi181 ceph-mon[94189]: osdmap e1216: 12 total, 12 up, 12 in 2023-03-09T23:02:17.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:16 smithi038 ceph-mon[92422]: pgmap v16003: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 507 KiB/s wr, 24 op/s 2023-03-09T23:02:17.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:16 smithi038 ceph-mon[92422]: osdmap e1216: 12 total, 12 up, 12 in 2023-03-09T23:02:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29048.789357 secs 2023-03-09T23:02:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29048.789357 secs 2023-03-09T23:02:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29048.789357 secs 2023-03-09T23:02:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29510.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:02:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:18 smithi122 ceph-mon[93869]: pgmap v16005: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 177 KiB/s wr, 9 op/s 2023-03-09T23:02:19.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:18 smithi181 ceph-mon[94189]: pgmap v16005: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 177 KiB/s wr, 9 op/s 2023-03-09T23:02:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:18 smithi038 ceph-mon[92422]: pgmap v16005: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 177 KiB/s wr, 9 op/s 2023-03-09T23:02:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:20 smithi122 ceph-mon[93869]: pgmap v16006: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 249 KiB/s wr, 9 op/s 2023-03-09T23:02:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29051.399710 secs 2023-03-09T23:02:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:20 smithi181 ceph-mon[94189]: pgmap v16006: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 249 KiB/s wr, 9 op/s 2023-03-09T23:02:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29051.399710 secs 2023-03-09T23:02:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:20 smithi038 ceph-mon[92422]: pgmap v16006: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 249 KiB/s wr, 9 op/s 2023-03-09T23:02:21.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29051.399710 secs 2023-03-09T23:02:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:22 smithi122 ceph-mon[93869]: pgmap v16007: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 249 KiB/s wr, 8 op/s 2023-03-09T23:02:23.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29053.789456 secs 2023-03-09T23:02:23.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:22 smithi181 ceph-mon[94189]: pgmap v16007: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 249 KiB/s wr, 8 op/s 2023-03-09T23:02:23.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29053.789456 secs 2023-03-09T23:02:23.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:22 smithi038 ceph-mon[92422]: pgmap v16007: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 249 KiB/s wr, 8 op/s 2023-03-09T23:02:23.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29053.789456 secs 2023-03-09T23:02:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29515.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:02:25.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:24 smithi122 ceph-mon[93869]: pgmap v16008: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 103 KiB/s wr, 7 op/s 2023-03-09T23:02:25.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:24 smithi181 ceph-mon[94189]: pgmap v16008: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 103 KiB/s wr, 7 op/s 2023-03-09T23:02:25.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:24 smithi038 ceph-mon[92422]: pgmap v16008: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 103 KiB/s wr, 7 op/s 2023-03-09T23:02:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29056.399770 secs 2023-03-09T23:02:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29056.399770 secs 2023-03-09T23:02:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29056.399770 secs 2023-03-09T23:02:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:26 smithi122 ceph-mon[93869]: pgmap v16009: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 103 KiB/s wr, 7 op/s 2023-03-09T23:02:27.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:26 smithi181 ceph-mon[94189]: pgmap v16009: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 103 KiB/s wr, 7 op/s 2023-03-09T23:02:27.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:26 smithi038 ceph-mon[92422]: pgmap v16009: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 103 KiB/s wr, 7 op/s 2023-03-09T23:02:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29058.789549 secs 2023-03-09T23:02:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29058.789549 secs 2023-03-09T23:02:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29058.789549 secs 2023-03-09T23:02:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29520.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:02:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:28 smithi122 ceph-mon[93869]: pgmap v16010: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 94 KiB/s wr, 6 op/s 2023-03-09T23:02:29.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:28 smithi181 ceph-mon[94189]: pgmap v16010: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 94 KiB/s wr, 6 op/s 2023-03-09T23:02:29.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:28 smithi038 ceph-mon[92422]: pgmap v16010: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 94 KiB/s wr, 6 op/s 2023-03-09T23:02:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:30 smithi122 ceph-mon[93869]: pgmap v16011: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 85 KiB/s wr, 7 op/s 2023-03-09T23:02:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29061.399827 secs 2023-03-09T23:02:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:30 smithi181 ceph-mon[94189]: pgmap v16011: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 85 KiB/s wr, 7 op/s 2023-03-09T23:02:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29061.399827 secs 2023-03-09T23:02:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:30 smithi038 ceph-mon[92422]: pgmap v16011: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 85 KiB/s wr, 7 op/s 2023-03-09T23:02:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29061.399827 secs 2023-03-09T23:02:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:32 smithi122 ceph-mon[93869]: pgmap v16012: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T23:02:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29063.789692 secs 2023-03-09T23:02:33.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:32 smithi181 ceph-mon[94189]: pgmap v16012: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T23:02:33.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29063.789692 secs 2023-03-09T23:02:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:32 smithi038 ceph-mon[92422]: pgmap v16012: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T23:02:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29063.789692 secs 2023-03-09T23:02:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29525.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:02:35.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:34 smithi122 ceph-mon[93869]: pgmap v16013: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T23:02:35.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:34 smithi181 ceph-mon[94189]: pgmap v16013: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T23:02:35.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:34 smithi038 ceph-mon[92422]: pgmap v16013: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T23:02:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29066.399891 secs 2023-03-09T23:02:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29066.399891 secs 2023-03-09T23:02:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29066.399891 secs 2023-03-09T23:02:37.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:36 smithi122 ceph-mon[93869]: pgmap v16014: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 8 op/s 2023-03-09T23:02:37.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:02:37.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:02:37.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:36 smithi181 ceph-mon[94189]: pgmap v16014: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 8 op/s 2023-03-09T23:02:37.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:02:37.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:02:37.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:36 smithi038 ceph-mon[92422]: pgmap v16014: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 8 op/s 2023-03-09T23:02:37.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:02:37.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:02:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29068.789805 secs 2023-03-09T23:02:38.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29068.789805 secs 2023-03-09T23:02:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29068.789805 secs 2023-03-09T23:02:38.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29530.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:02:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:38 smithi122 ceph-mon[93869]: pgmap v16015: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T23:02:39.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:38 smithi181 ceph-mon[94189]: pgmap v16015: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T23:02:39.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:38 smithi038 ceph-mon[92422]: pgmap v16015: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T23:02:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:39 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:02:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:39 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:02:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:39 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:02:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:39 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:02:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:39 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:02:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:39 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:02:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:40 smithi122 ceph-mon[93869]: pgmap v16016: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 228 KiB/s wr, 7 op/s 2023-03-09T23:02:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29071.399967 secs 2023-03-09T23:02:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:40 smithi181 ceph-mon[94189]: pgmap v16016: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 228 KiB/s wr, 7 op/s 2023-03-09T23:02:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29071.399967 secs 2023-03-09T23:02:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:40 smithi038 ceph-mon[92422]: pgmap v16016: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 228 KiB/s wr, 7 op/s 2023-03-09T23:02:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29071.399967 secs 2023-03-09T23:02:43.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:42 smithi122 ceph-mon[93869]: pgmap v16017: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 228 KiB/s wr, 8 op/s 2023-03-09T23:02:43.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:42 smithi181 ceph-mon[94189]: pgmap v16017: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 228 KiB/s wr, 8 op/s 2023-03-09T23:02:43.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:42 smithi038 ceph-mon[92422]: pgmap v16017: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 228 KiB/s wr, 8 op/s 2023-03-09T23:02:43.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29535.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:02:45.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:44 smithi122 ceph-mon[93869]: pgmap v16018: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 229 KiB/s wr, 7 op/s 2023-03-09T23:02:45.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:44 smithi181 ceph-mon[94189]: pgmap v16018: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 229 KiB/s wr, 7 op/s 2023-03-09T23:02:45.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:44 smithi038 ceph-mon[92422]: pgmap v16018: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 229 KiB/s wr, 7 op/s 2023-03-09T23:02:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29076.399986 secs 2023-03-09T23:02:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:45 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:02:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29076.399986 secs 2023-03-09T23:02:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:45 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:02:46.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29076.399986 secs 2023-03-09T23:02:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:45 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:02:47.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:46 smithi038 ceph-mon[92422]: pgmap v16019: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 507 KiB/s rd, 297 KiB/s wr, 9 op/s 2023-03-09T23:02:47.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:46 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:02:47.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:46 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:02:47.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:46 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:02:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:46 smithi122 ceph-mon[93869]: pgmap v16019: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 507 KiB/s rd, 297 KiB/s wr, 9 op/s 2023-03-09T23:02:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:46 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:02:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:46 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:02:47.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:46 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:02:47.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:46 smithi181 ceph-mon[94189]: pgmap v16019: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 507 KiB/s rd, 297 KiB/s wr, 9 op/s 2023-03-09T23:02:47.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:46 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:02:47.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:46 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:02:47.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:46 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:02:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29540.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:02:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:48 smithi038 ceph-mon[92422]: pgmap v16020: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 367 KiB/s wr, 8 op/s 2023-03-09T23:02:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:02:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:02:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:02:49.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:02:49.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:02:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:48 smithi122 ceph-mon[93869]: pgmap v16020: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 367 KiB/s wr, 8 op/s 2023-03-09T23:02:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:02:49.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:02:49.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:02:49.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:02:49.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:02:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:48 smithi181 ceph-mon[94189]: pgmap v16020: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 367 KiB/s wr, 8 op/s 2023-03-09T23:02:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:02:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:02:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:02:49.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:02:49.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:02:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:02:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:02:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:02:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:02:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:02:50.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:02:50.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:02:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:02:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:02:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:50 smithi038 ceph-mon[92422]: pgmap v16021: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 644 KiB/s rd, 367 KiB/s wr, 8 op/s 2023-03-09T23:02:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29081.400045 secs 2023-03-09T23:02:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:50 smithi122 ceph-mon[93869]: pgmap v16021: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 644 KiB/s rd, 367 KiB/s wr, 8 op/s 2023-03-09T23:02:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29081.400045 secs 2023-03-09T23:02:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:50 smithi181 ceph-mon[94189]: pgmap v16021: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 644 KiB/s rd, 367 KiB/s wr, 8 op/s 2023-03-09T23:02:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29081.400045 secs 2023-03-09T23:02:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:53 smithi122 ceph-mon[93869]: pgmap v16022: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 738 KiB/s rd, 435 KiB/s wr, 10 op/s 2023-03-09T23:02:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:02:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:02:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:53 smithi181 ceph-mon[94189]: pgmap v16022: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 738 KiB/s rd, 435 KiB/s wr, 10 op/s 2023-03-09T23:02:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:02:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:02:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:53 smithi038 ceph-mon[92422]: pgmap v16022: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 738 KiB/s rd, 435 KiB/s wr, 10 op/s 2023-03-09T23:02:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:02:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:02:53.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29545.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:02:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:55 smithi122 ceph-mon[93869]: pgmap v16023: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 737 KiB/s rd, 435 KiB/s wr, 9 op/s 2023-03-09T23:02:55.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:55 smithi181 ceph-mon[94189]: pgmap v16023: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 737 KiB/s rd, 435 KiB/s wr, 9 op/s 2023-03-09T23:02:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:55 smithi038 ceph-mon[92422]: pgmap v16023: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 737 KiB/s rd, 435 KiB/s wr, 9 op/s 2023-03-09T23:02:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29086.400114 secs 2023-03-09T23:02:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:02:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:02:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29086.400114 secs 2023-03-09T23:02:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:02:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:02:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29086.400114 secs 2023-03-09T23:02:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:02:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:02:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:57 smithi122 ceph-mon[93869]: pgmap v16024: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 954 KiB/s rd, 649 KiB/s wr, 11 op/s 2023-03-09T23:02:57.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:57 smithi181 ceph-mon[94189]: pgmap v16024: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 954 KiB/s rd, 649 KiB/s wr, 11 op/s 2023-03-09T23:02:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:57 smithi038 ceph-mon[92422]: pgmap v16024: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 954 KiB/s rd, 649 KiB/s wr, 11 op/s 2023-03-09T23:02:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29550.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:02:59.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:02:59 smithi122 ceph-mon[93869]: pgmap v16025: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 742 KiB/s rd, 606 KiB/s wr, 10 op/s 2023-03-09T23:02:59.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:02:59 smithi181 ceph-mon[94189]: pgmap v16025: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 742 KiB/s rd, 606 KiB/s wr, 10 op/s 2023-03-09T23:02:59.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:02:59 smithi038 ceph-mon[92422]: pgmap v16025: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 742 KiB/s rd, 606 KiB/s wr, 10 op/s 2023-03-09T23:03:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:03:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:03:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:03:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:03:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:03:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:03:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:01 smithi122 ceph-mon[93869]: pgmap v16026: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 684 KiB/s rd, 538 KiB/s wr, 9 op/s 2023-03-09T23:03:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29091.400170 secs 2023-03-09T23:03:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:03:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:03:01.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:01 smithi181 ceph-mon[94189]: pgmap v16026: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 684 KiB/s rd, 538 KiB/s wr, 9 op/s 2023-03-09T23:03:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29091.400170 secs 2023-03-09T23:03:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:03:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:03:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:01 smithi038 ceph-mon[92422]: pgmap v16026: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 684 KiB/s rd, 538 KiB/s wr, 9 op/s 2023-03-09T23:03:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29091.400170 secs 2023-03-09T23:03:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:03:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:03:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:03 smithi122 ceph-mon[93869]: pgmap v16027: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 978 KiB/s rd, 900 KiB/s wr, 14 op/s 2023-03-09T23:03:03.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:03 smithi181 ceph-mon[94189]: pgmap v16027: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 978 KiB/s rd, 900 KiB/s wr, 14 op/s 2023-03-09T23:03:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:03 smithi038 ceph-mon[92422]: pgmap v16027: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 978 KiB/s rd, 900 KiB/s wr, 14 op/s 2023-03-09T23:03:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29555.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:03:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:05 smithi122 ceph-mon[93869]: pgmap v16028: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 604 KiB/s wr, 12 op/s 2023-03-09T23:03:05.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:05 smithi181 ceph-mon[94189]: pgmap v16028: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 604 KiB/s wr, 12 op/s 2023-03-09T23:03:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:05 smithi038 ceph-mon[92422]: pgmap v16028: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 604 KiB/s wr, 12 op/s 2023-03-09T23:03:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29096.400230 secs 2023-03-09T23:03:06.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29096.400230 secs 2023-03-09T23:03:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29096.400230 secs 2023-03-09T23:03:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:07 smithi122 ceph-mon[93869]: pgmap v16029: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 673 KiB/s rd, 640 KiB/s wr, 17 op/s 2023-03-09T23:03:07.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:07 smithi181 ceph-mon[94189]: pgmap v16029: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 673 KiB/s rd, 640 KiB/s wr, 17 op/s 2023-03-09T23:03:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:07 smithi038 ceph-mon[92422]: pgmap v16029: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 673 KiB/s rd, 640 KiB/s wr, 17 op/s 2023-03-09T23:03:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:03:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:03:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29098.790560 secs 2023-03-09T23:03:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:03:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:03:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29098.790560 secs 2023-03-09T23:03:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:03:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:03:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29098.790560 secs 2023-03-09T23:03:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29560.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:03:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:09 smithi122 ceph-mon[93869]: pgmap v16030: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 557 KiB/s rd, 528 KiB/s wr, 16 op/s 2023-03-09T23:03:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:09 smithi181 ceph-mon[94189]: pgmap v16030: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 557 KiB/s rd, 528 KiB/s wr, 16 op/s 2023-03-09T23:03:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:09 smithi038 ceph-mon[92422]: pgmap v16030: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 557 KiB/s rd, 528 KiB/s wr, 16 op/s 2023-03-09T23:03:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:10 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:03:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:10 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:03:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:10 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:03:10.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:10 smithi122 ceph-mon[93869]: osdmap e1217: 12 total, 12 up, 12 in 2023-03-09T23:03:10.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:10 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:03:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:10 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:03:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:10 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:03:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:10 smithi181 ceph-mon[94189]: osdmap e1217: 12 total, 12 up, 12 in 2023-03-09T23:03:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:10 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:03:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:10 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:03:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:10 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:03:10.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:10 smithi038 ceph-mon[92422]: osdmap e1217: 12 total, 12 up, 12 in 2023-03-09T23:03:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:11 smithi122 ceph-mon[93869]: pgmap v16031: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 499 KiB/s rd, 523 KiB/s wr, 25 op/s 2023-03-09T23:03:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29101.400314 secs 2023-03-09T23:03:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:11 smithi181 ceph-mon[94189]: pgmap v16031: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 499 KiB/s rd, 523 KiB/s wr, 25 op/s 2023-03-09T23:03:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29101.400314 secs 2023-03-09T23:03:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:11 smithi038 ceph-mon[92422]: pgmap v16031: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 499 KiB/s rd, 523 KiB/s wr, 25 op/s 2023-03-09T23:03:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29101.400314 secs 2023-03-09T23:03:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:13 smithi122 ceph-mon[93869]: pgmap v16033: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 494 KiB/s rd, 535 KiB/s wr, 28 op/s 2023-03-09T23:03:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:03:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:03:13.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29103.790635 secs 2023-03-09T23:03:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:13 smithi181 ceph-mon[94189]: pgmap v16033: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 494 KiB/s rd, 535 KiB/s wr, 28 op/s 2023-03-09T23:03:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:03:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:03:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29103.790635 secs 2023-03-09T23:03:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:13 smithi038 ceph-mon[92422]: pgmap v16033: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 494 KiB/s rd, 535 KiB/s wr, 28 op/s 2023-03-09T23:03:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:03:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:03:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29103.790635 secs 2023-03-09T23:03:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29565.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:03:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:15 smithi122 ceph-mon[93869]: pgmap v16034: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 493 KiB/s rd, 535 KiB/s wr, 27 op/s 2023-03-09T23:03:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29106.400375 secs 2023-03-09T23:03:15.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:15 smithi181 ceph-mon[94189]: pgmap v16034: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 493 KiB/s rd, 535 KiB/s wr, 27 op/s 2023-03-09T23:03:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29106.400375 secs 2023-03-09T23:03:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:15 smithi038 ceph-mon[92422]: pgmap v16034: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 493 KiB/s rd, 535 KiB/s wr, 27 op/s 2023-03-09T23:03:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29106.400375 secs 2023-03-09T23:03:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:03:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:03:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:03:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:03:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:03:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:03:17.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:17 smithi122 ceph-mon[93869]: pgmap v16035: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 579 KiB/s rd, 595 KiB/s wr, 25 op/s 2023-03-09T23:03:17.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:17 smithi122 ceph-mon[93869]: osdmap e1218: 12 total, 12 up, 12 in 2023-03-09T23:03:17.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:17 smithi181 ceph-mon[94189]: pgmap v16035: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 579 KiB/s rd, 595 KiB/s wr, 25 op/s 2023-03-09T23:03:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:17 smithi181 ceph-mon[94189]: osdmap e1218: 12 total, 12 up, 12 in 2023-03-09T23:03:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:17 smithi038 ceph-mon[92422]: pgmap v16035: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 579 KiB/s rd, 595 KiB/s wr, 25 op/s 2023-03-09T23:03:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:17 smithi038 ceph-mon[92422]: osdmap e1218: 12 total, 12 up, 12 in 2023-03-09T23:03:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29108.790759 secs 2023-03-09T23:03:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29108.790759 secs 2023-03-09T23:03:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29108.790759 secs 2023-03-09T23:03:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29570.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:03:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:19 smithi122 ceph-mon[93869]: pgmap v16037: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 561 KiB/s rd, 558 KiB/s wr, 12 op/s 2023-03-09T23:03:19.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:19 smithi181 ceph-mon[94189]: pgmap v16037: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 561 KiB/s rd, 558 KiB/s wr, 12 op/s 2023-03-09T23:03:19.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:19 smithi038 ceph-mon[92422]: pgmap v16037: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 561 KiB/s rd, 558 KiB/s wr, 12 op/s 2023-03-09T23:03:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29111.400437 secs 2023-03-09T23:03:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29111.400437 secs 2023-03-09T23:03:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29111.400437 secs 2023-03-09T23:03:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:21 smithi122 ceph-mon[93869]: pgmap v16038: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 458 KiB/s wr, 10 op/s 2023-03-09T23:03:21.485 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:21 smithi181 ceph-mon[94189]: pgmap v16038: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 458 KiB/s wr, 10 op/s 2023-03-09T23:03:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:21 smithi038 ceph-mon[92422]: pgmap v16038: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 458 KiB/s wr, 10 op/s 2023-03-09T23:03:23.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:23 smithi122 ceph-mon[93869]: pgmap v16039: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 103 KiB/s wr, 7 op/s 2023-03-09T23:03:23.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29113.790820 secs 2023-03-09T23:03:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:23 smithi181 ceph-mon[94189]: pgmap v16039: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 103 KiB/s wr, 7 op/s 2023-03-09T23:03:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29113.790820 secs 2023-03-09T23:03:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:23 smithi038 ceph-mon[92422]: pgmap v16039: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 103 KiB/s wr, 7 op/s 2023-03-09T23:03:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29113.790820 secs 2023-03-09T23:03:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29575.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:03:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:25 smithi122 ceph-mon[93869]: pgmap v16040: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 103 KiB/s wr, 7 op/s 2023-03-09T23:03:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29116.400512 secs 2023-03-09T23:03:25.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:25 smithi181 ceph-mon[94189]: pgmap v16040: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 103 KiB/s wr, 7 op/s 2023-03-09T23:03:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29116.400512 secs 2023-03-09T23:03:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:25 smithi038 ceph-mon[92422]: pgmap v16040: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 103 KiB/s wr, 7 op/s 2023-03-09T23:03:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29116.400512 secs 2023-03-09T23:03:27.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:27 smithi122 ceph-mon[93869]: pgmap v16041: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T23:03:27.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:27 smithi181 ceph-mon[94189]: pgmap v16041: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T23:03:27.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:27 smithi038 ceph-mon[92422]: pgmap v16041: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T23:03:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29118.790947 secs 2023-03-09T23:03:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29118.790947 secs 2023-03-09T23:03:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29118.790947 secs 2023-03-09T23:03:28.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29580.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:03:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:29 smithi122 ceph-mon[93869]: pgmap v16042: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2023-03-09T23:03:29.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:29 smithi181 ceph-mon[94189]: pgmap v16042: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2023-03-09T23:03:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:29 smithi038 ceph-mon[92422]: pgmap v16042: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2023-03-09T23:03:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29121.400598 secs 2023-03-09T23:03:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29121.400598 secs 2023-03-09T23:03:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29121.400598 secs 2023-03-09T23:03:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:31 smithi038 ceph-mon[92422]: pgmap v16043: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T23:03:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:31 smithi122 ceph-mon[93869]: pgmap v16043: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T23:03:31.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:31 smithi181 ceph-mon[94189]: pgmap v16043: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T23:03:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:33 smithi038 ceph-mon[92422]: pgmap v16044: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T23:03:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29123.791067 secs 2023-03-09T23:03:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:33 smithi122 ceph-mon[93869]: pgmap v16044: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T23:03:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29123.791067 secs 2023-03-09T23:03:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:33 smithi181 ceph-mon[94189]: pgmap v16044: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T23:03:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29123.791067 secs 2023-03-09T23:03:33.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29585.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:03:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:35 smithi038 ceph-mon[92422]: pgmap v16045: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T23:03:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29126.400658 secs 2023-03-09T23:03:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:35 smithi122 ceph-mon[93869]: pgmap v16045: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T23:03:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29126.400658 secs 2023-03-09T23:03:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:35 smithi181 ceph-mon[94189]: pgmap v16045: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T23:03:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29126.400658 secs 2023-03-09T23:03:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:37 smithi038 ceph-mon[92422]: pgmap v16046: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 341 B/s wr, 8 op/s 2023-03-09T23:03:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:37 smithi122 ceph-mon[93869]: pgmap v16046: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 341 B/s wr, 8 op/s 2023-03-09T23:03:37.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:37 smithi181 ceph-mon[94189]: pgmap v16046: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 341 B/s wr, 8 op/s 2023-03-09T23:03:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29128.791182 secs 2023-03-09T23:03:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29128.791182 secs 2023-03-09T23:03:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29128.791182 secs 2023-03-09T23:03:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29590.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:03:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:39 smithi122 ceph-mon[93869]: pgmap v16047: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-09T23:03:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:39 smithi181 ceph-mon[94189]: pgmap v16047: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-09T23:03:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:39 smithi038 ceph-mon[92422]: pgmap v16047: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-09T23:03:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29131.400717 secs 2023-03-09T23:03:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29131.400717 secs 2023-03-09T23:03:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29131.400717 secs 2023-03-09T23:03:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:41 smithi122 ceph-mon[93869]: pgmap v16048: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-09T23:03:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:41 smithi181 ceph-mon[94189]: pgmap v16048: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-09T23:03:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:41 smithi038 ceph-mon[92422]: pgmap v16048: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-09T23:03:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:43 smithi122 ceph-mon[93869]: pgmap v16049: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 126 KiB/s wr, 10 op/s 2023-03-09T23:03:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:43 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 29133.791303 secs 2023-03-09T23:03:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:43 smithi181 ceph-mon[94189]: pgmap v16049: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 126 KiB/s wr, 10 op/s 2023-03-09T23:03:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:43 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 29133.791303 secs 2023-03-09T23:03:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:43 smithi038 ceph-mon[92422]: pgmap v16049: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 126 KiB/s wr, 10 op/s 2023-03-09T23:03:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:43 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 29133.791303 secs 2023-03-09T23:03:43.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29595.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:03:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:45 smithi122 ceph-mon[93869]: pgmap v16050: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 126 KiB/s wr, 7 op/s 2023-03-09T23:03:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29136.400778 secs 2023-03-09T23:03:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:45 smithi181 ceph-mon[94189]: pgmap v16050: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 126 KiB/s wr, 7 op/s 2023-03-09T23:03:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29136.400778 secs 2023-03-09T23:03:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:45 smithi038 ceph-mon[92422]: pgmap v16050: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 126 KiB/s wr, 7 op/s 2023-03-09T23:03:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29136.400778 secs 2023-03-09T23:03:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:46 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:03:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:46 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:03:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:46 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:03:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:46 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:03:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:46 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:03:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:46 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:03:47.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:47 smithi122 ceph-mon[93869]: pgmap v16051: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 483 KiB/s rd, 372 KiB/s wr, 10 op/s 2023-03-09T23:03:47.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:47 smithi181 ceph-mon[94189]: pgmap v16051: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 483 KiB/s rd, 372 KiB/s wr, 10 op/s 2023-03-09T23:03:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:47 smithi038 ceph-mon[92422]: pgmap v16051: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 483 KiB/s rd, 372 KiB/s wr, 10 op/s 2023-03-09T23:03:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:48 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 29138.791417 secs 2023-03-09T23:03:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:48 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 29138.791417 secs 2023-03-09T23:03:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:48 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 29138.791417 secs 2023-03-09T23:03:48.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29600.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:03:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:49 smithi122 ceph-mon[93869]: pgmap v16052: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 372 KiB/s wr, 8 op/s 2023-03-09T23:03:49.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:03:49.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:03:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:49 smithi181 ceph-mon[94189]: pgmap v16052: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 372 KiB/s wr, 8 op/s 2023-03-09T23:03:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:03:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:03:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:49 smithi038 ceph-mon[92422]: pgmap v16052: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 372 KiB/s wr, 8 op/s 2023-03-09T23:03:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:03:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:03:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:50 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:03:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29141.400862 secs 2023-03-09T23:03:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:50 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:03:50.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:50 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:03:50.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:50 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:03:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:50 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:03:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29141.400862 secs 2023-03-09T23:03:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:50 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:03:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:50 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:03:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:50 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:03:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:50 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:03:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29141.400862 secs 2023-03-09T23:03:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:50 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:03:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:50 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:03:50.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:50 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:03:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:51 smithi122 ceph-mon[93869]: pgmap v16053: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 372 KiB/s wr, 7 op/s 2023-03-09T23:03:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:51 smithi181 ceph-mon[94189]: pgmap v16053: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 372 KiB/s wr, 7 op/s 2023-03-09T23:03:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:51 smithi038 ceph-mon[92422]: pgmap v16053: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 372 KiB/s wr, 7 op/s 2023-03-09T23:03:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:03:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:03:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:03:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:03:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:03:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:03:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:53 smithi122 ceph-mon[93869]: pgmap v16054: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 937 KiB/s rd, 395 KiB/s wr, 11 op/s 2023-03-09T23:03:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:03:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:03:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:53 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 29143.791487 secs 2023-03-09T23:03:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:53 smithi181 ceph-mon[94189]: pgmap v16054: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 937 KiB/s rd, 395 KiB/s wr, 11 op/s 2023-03-09T23:03:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:03:53.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:03:53.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:53 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 29143.791487 secs 2023-03-09T23:03:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:53 smithi038 ceph-mon[92422]: pgmap v16054: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 937 KiB/s rd, 395 KiB/s wr, 11 op/s 2023-03-09T23:03:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:03:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:03:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:53 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 29143.791487 secs 2023-03-09T23:03:53.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29605.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:03:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:55 smithi122 ceph-mon[93869]: pgmap v16055: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 963 KiB/s rd, 422 KiB/s wr, 8 op/s 2023-03-09T23:03:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29146.400920 secs 2023-03-09T23:03:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:55 smithi181 ceph-mon[94189]: pgmap v16055: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 963 KiB/s rd, 422 KiB/s wr, 8 op/s 2023-03-09T23:03:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29146.400920 secs 2023-03-09T23:03:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:55 smithi038 ceph-mon[92422]: pgmap v16055: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 963 KiB/s rd, 422 KiB/s wr, 8 op/s 2023-03-09T23:03:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29146.400920 secs 2023-03-09T23:03:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:03:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:03:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:03:56.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:03:56.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:03:56.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:03:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:57 smithi122 ceph-mon[93869]: pgmap v16056: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 992 KiB/s rd, 450 KiB/s wr, 11 op/s 2023-03-09T23:03:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:57 smithi181 ceph-mon[94189]: pgmap v16056: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 992 KiB/s rd, 450 KiB/s wr, 11 op/s 2023-03-09T23:03:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:57 smithi038 ceph-mon[92422]: pgmap v16056: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 992 KiB/s rd, 450 KiB/s wr, 11 op/s 2023-03-09T23:03:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29148.791600 secs 2023-03-09T23:03:58.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29148.791600 secs 2023-03-09T23:03:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29148.791600 secs 2023-03-09T23:03:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29610.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:03:59.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:03:59 smithi122 ceph-mon[93869]: pgmap v16057: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 637 KiB/s rd, 203 KiB/s wr, 8 op/s 2023-03-09T23:03:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:03:59 smithi181 ceph-mon[94189]: pgmap v16057: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 637 KiB/s rd, 203 KiB/s wr, 8 op/s 2023-03-09T23:03:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:03:59 smithi038 ceph-mon[92422]: pgmap v16057: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 637 KiB/s rd, 203 KiB/s wr, 8 op/s 2023-03-09T23:04:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:04:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:04:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29151.400983 secs 2023-03-09T23:04:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:04:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:04:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29151.400983 secs 2023-03-09T23:04:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:04:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:04:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29151.400983 secs 2023-03-09T23:04:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:01 smithi122 ceph-mon[93869]: pgmap v16058: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 612 KiB/s wr, 11 op/s 2023-03-09T23:04:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:04:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:04:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:01 smithi181 ceph-mon[94189]: pgmap v16058: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 612 KiB/s wr, 11 op/s 2023-03-09T23:04:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:04:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:04:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:01 smithi038 ceph-mon[92422]: pgmap v16058: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 612 KiB/s wr, 11 op/s 2023-03-09T23:04:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:04:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:04:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:02 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:04:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:02 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:04:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:02 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:04:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:02 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:04:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:02 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:04:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:02 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:04:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:03 smithi122 ceph-mon[93869]: pgmap v16059: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 672 KiB/s wr, 14 op/s 2023-03-09T23:04:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29153.791722 secs 2023-03-09T23:04:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:03 smithi181 ceph-mon[94189]: pgmap v16059: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 672 KiB/s wr, 14 op/s 2023-03-09T23:04:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29153.791722 secs 2023-03-09T23:04:03.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:03 smithi038 ceph-mon[92422]: pgmap v16059: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 672 KiB/s wr, 14 op/s 2023-03-09T23:04:03.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29153.791722 secs 2023-03-09T23:04:03.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29615.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:04:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:05 smithi122 ceph-mon[93869]: pgmap v16060: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 678 KiB/s rd, 648 KiB/s wr, 13 op/s 2023-03-09T23:04:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29156.401041 secs 2023-03-09T23:04:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:05 smithi181 ceph-mon[94189]: pgmap v16060: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 678 KiB/s rd, 648 KiB/s wr, 13 op/s 2023-03-09T23:04:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29156.401041 secs 2023-03-09T23:04:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:05 smithi038 ceph-mon[92422]: pgmap v16060: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 678 KiB/s rd, 648 KiB/s wr, 13 op/s 2023-03-09T23:04:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29156.401041 secs 2023-03-09T23:04:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:07 smithi122 ceph-mon[93869]: pgmap v16061: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 765 KiB/s rd, 757 KiB/s wr, 19 op/s 2023-03-09T23:04:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:07 smithi181 ceph-mon[94189]: pgmap v16061: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 765 KiB/s rd, 757 KiB/s wr, 19 op/s 2023-03-09T23:04:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:07 smithi038 ceph-mon[92422]: pgmap v16061: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 765 KiB/s rd, 757 KiB/s wr, 19 op/s 2023-03-09T23:04:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29158.791841 secs 2023-03-09T23:04:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:04:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:04:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29158.791841 secs 2023-03-09T23:04:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:04:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:04:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29158.791841 secs 2023-03-09T23:04:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:04:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:04:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29620.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:04:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:09 smithi122 ceph-mon[93869]: pgmap v16062: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 736 KiB/s rd, 729 KiB/s wr, 15 op/s 2023-03-09T23:04:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:04:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:04:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:09 smithi181 ceph-mon[94189]: pgmap v16062: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 736 KiB/s rd, 729 KiB/s wr, 15 op/s 2023-03-09T23:04:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:04:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:04:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:09 smithi038 ceph-mon[92422]: pgmap v16062: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 736 KiB/s rd, 729 KiB/s wr, 15 op/s 2023-03-09T23:04:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:04:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:04:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:10 smithi181 ceph-mon[94189]: osdmap e1219: 12 total, 12 up, 12 in 2023-03-09T23:04:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29161.401104 secs 2023-03-09T23:04:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:10 smithi038 ceph-mon[92422]: osdmap e1219: 12 total, 12 up, 12 in 2023-03-09T23:04:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29161.401104 secs 2023-03-09T23:04:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:10 smithi122 ceph-mon[93869]: osdmap e1219: 12 total, 12 up, 12 in 2023-03-09T23:04:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29161.401104 secs 2023-03-09T23:04:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:11 smithi181 ceph-mon[94189]: pgmap v16064: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 410 KiB/s wr, 29 op/s 2023-03-09T23:04:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:11 smithi038 ceph-mon[92422]: pgmap v16064: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 410 KiB/s wr, 29 op/s 2023-03-09T23:04:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:11 smithi122 ceph-mon[93869]: pgmap v16064: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 410 KiB/s wr, 29 op/s 2023-03-09T23:04:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:13 smithi038 ceph-mon[92422]: pgmap v16065: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 579 KiB/s rd, 621 KiB/s wr, 29 op/s 2023-03-09T23:04:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29163.791958 secs 2023-03-09T23:04:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29625.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:04:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:13 smithi122 ceph-mon[93869]: pgmap v16065: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 579 KiB/s rd, 621 KiB/s wr, 29 op/s 2023-03-09T23:04:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29163.791958 secs 2023-03-09T23:04:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:13 smithi181 ceph-mon[94189]: pgmap v16065: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 579 KiB/s rd, 621 KiB/s wr, 29 op/s 2023-03-09T23:04:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29163.791958 secs 2023-03-09T23:04:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:15 smithi038 ceph-mon[92422]: pgmap v16066: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 578 KiB/s rd, 621 KiB/s wr, 27 op/s 2023-03-09T23:04:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29166.401163 secs 2023-03-09T23:04:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:15 smithi122 ceph-mon[93869]: pgmap v16066: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 578 KiB/s rd, 621 KiB/s wr, 27 op/s 2023-03-09T23:04:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29166.401163 secs 2023-03-09T23:04:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:15 smithi181 ceph-mon[94189]: pgmap v16066: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 578 KiB/s rd, 621 KiB/s wr, 27 op/s 2023-03-09T23:04:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29166.401163 secs 2023-03-09T23:04:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:17 smithi122 ceph-mon[93869]: pgmap v16067: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 355 KiB/s rd, 370 KiB/s wr, 23 op/s 2023-03-09T23:04:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:04:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:04:17.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:17 smithi122 ceph-mon[93869]: osdmap e1220: 12 total, 12 up, 12 in 2023-03-09T23:04:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:17 smithi181 ceph-mon[94189]: pgmap v16067: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 355 KiB/s rd, 370 KiB/s wr, 23 op/s 2023-03-09T23:04:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:04:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:04:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:17 smithi181 ceph-mon[94189]: osdmap e1220: 12 total, 12 up, 12 in 2023-03-09T23:04:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:17 smithi038 ceph-mon[92422]: pgmap v16067: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 355 KiB/s rd, 370 KiB/s wr, 23 op/s 2023-03-09T23:04:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:04:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:04:18.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:17 smithi038 ceph-mon[92422]: osdmap e1220: 12 total, 12 up, 12 in 2023-03-09T23:04:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29630.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:04:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29168.792089 secs 2023-03-09T23:04:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29168.792089 secs 2023-03-09T23:04:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29168.792089 secs 2023-03-09T23:04:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:19 smithi122 ceph-mon[93869]: pgmap v16069: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 411 KiB/s wr, 11 op/s 2023-03-09T23:04:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:19 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:04:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:19 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:04:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:19 smithi181 ceph-mon[94189]: pgmap v16069: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 411 KiB/s wr, 11 op/s 2023-03-09T23:04:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:19 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:04:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:19 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:04:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:19 smithi038 ceph-mon[92422]: pgmap v16069: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 411 KiB/s wr, 11 op/s 2023-03-09T23:04:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:19 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:04:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:19 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:04:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29171.401223 secs 2023-03-09T23:04:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29171.401223 secs 2023-03-09T23:04:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29171.401223 secs 2023-03-09T23:04:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:21 smithi122 ceph-mon[93869]: pgmap v16070: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 448 KiB/s wr, 10 op/s 2023-03-09T23:04:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:21 smithi181 ceph-mon[94189]: pgmap v16070: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 448 KiB/s wr, 10 op/s 2023-03-09T23:04:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:21 smithi038 ceph-mon[92422]: pgmap v16070: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 448 KiB/s wr, 10 op/s 2023-03-09T23:04:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29635.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:04:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:23 smithi122 ceph-mon[93869]: pgmap v16071: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 165 KiB/s wr, 8 op/s 2023-03-09T23:04:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29173.792169 secs 2023-03-09T23:04:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:23 smithi181 ceph-mon[94189]: pgmap v16071: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 165 KiB/s wr, 8 op/s 2023-03-09T23:04:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29173.792169 secs 2023-03-09T23:04:24.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:23 smithi038 ceph-mon[92422]: pgmap v16071: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 165 KiB/s wr, 8 op/s 2023-03-09T23:04:24.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29173.792169 secs 2023-03-09T23:04:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:25 smithi122 ceph-mon[93869]: pgmap v16072: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 165 KiB/s wr, 8 op/s 2023-03-09T23:04:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29176.401284 secs 2023-03-09T23:04:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:25 smithi181 ceph-mon[94189]: pgmap v16072: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 165 KiB/s wr, 8 op/s 2023-03-09T23:04:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29176.401284 secs 2023-03-09T23:04:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:25 smithi038 ceph-mon[92422]: pgmap v16072: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 165 KiB/s wr, 8 op/s 2023-03-09T23:04:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29176.401284 secs 2023-03-09T23:04:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:27 smithi122 ceph-mon[93869]: pgmap v16073: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 103 KiB/s wr, 7 op/s 2023-03-09T23:04:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:27 smithi181 ceph-mon[94189]: pgmap v16073: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 103 KiB/s wr, 7 op/s 2023-03-09T23:04:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:27 smithi038 ceph-mon[92422]: pgmap v16073: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 103 KiB/s wr, 7 op/s 2023-03-09T23:04:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29640.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:04:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29178.792236 secs 2023-03-09T23:04:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29178.792236 secs 2023-03-09T23:04:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29178.792236 secs 2023-03-09T23:04:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:29 smithi122 ceph-mon[93869]: pgmap v16074: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 94 KiB/s wr, 6 op/s 2023-03-09T23:04:29.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:29 smithi181 ceph-mon[94189]: pgmap v16074: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 94 KiB/s wr, 6 op/s 2023-03-09T23:04:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:29 smithi038 ceph-mon[92422]: pgmap v16074: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 94 KiB/s wr, 6 op/s 2023-03-09T23:04:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29181.401363 secs 2023-03-09T23:04:30.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29181.401363 secs 2023-03-09T23:04:30.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29181.401363 secs 2023-03-09T23:04:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:31 smithi122 ceph-mon[93869]: pgmap v16075: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 86 KiB/s wr, 6 op/s 2023-03-09T23:04:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:31 smithi181 ceph-mon[94189]: pgmap v16075: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 86 KiB/s wr, 6 op/s 2023-03-09T23:04:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:31 smithi038 ceph-mon[92422]: pgmap v16075: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 86 KiB/s wr, 6 op/s 2023-03-09T23:04:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:32 smithi122 ceph-mon[93869]: pgmap v16076: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 8 op/s 2023-03-09T23:04:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29183.792348 secs 2023-03-09T23:04:33.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:32 smithi181 ceph-mon[94189]: pgmap v16076: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 8 op/s 2023-03-09T23:04:33.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29183.792348 secs 2023-03-09T23:04:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:32 smithi038 ceph-mon[92422]: pgmap v16076: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 8 op/s 2023-03-09T23:04:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29183.792348 secs 2023-03-09T23:04:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29645.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:04:35.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:34 smithi122 ceph-mon[93869]: pgmap v16077: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T23:04:35.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:34 smithi181 ceph-mon[94189]: pgmap v16077: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T23:04:35.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:34 smithi038 ceph-mon[92422]: pgmap v16077: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T23:04:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29186.401421 secs 2023-03-09T23:04:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29186.401421 secs 2023-03-09T23:04:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29186.401421 secs 2023-03-09T23:04:37.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:36 smithi122 ceph-mon[93869]: pgmap v16078: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 426 B/s wr, 8 op/s 2023-03-09T23:04:37.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:36 smithi181 ceph-mon[94189]: pgmap v16078: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 426 B/s wr, 8 op/s 2023-03-09T23:04:37.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:36 smithi038 ceph-mon[92422]: pgmap v16078: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 426 B/s wr, 8 op/s 2023-03-09T23:04:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29188.792469 secs 2023-03-09T23:04:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29188.792469 secs 2023-03-09T23:04:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29188.792469 secs 2023-03-09T23:04:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29650.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:04:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:38 smithi122 ceph-mon[93869]: pgmap v16079: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T23:04:39.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:38 smithi181 ceph-mon[94189]: pgmap v16079: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T23:04:39.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:38 smithi038 ceph-mon[92422]: pgmap v16079: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T23:04:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:40 smithi122 ceph-mon[93869]: pgmap v16080: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 205 KiB/s wr, 7 op/s 2023-03-09T23:04:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29191.401483 secs 2023-03-09T23:04:41.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:40 smithi181 ceph-mon[94189]: pgmap v16080: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 205 KiB/s wr, 7 op/s 2023-03-09T23:04:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29191.401483 secs 2023-03-09T23:04:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:40 smithi038 ceph-mon[92422]: pgmap v16080: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 205 KiB/s wr, 7 op/s 2023-03-09T23:04:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29191.401483 secs 2023-03-09T23:04:43.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:42 smithi122 ceph-mon[93869]: pgmap v16081: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 230 KiB/s wr, 9 op/s 2023-03-09T23:04:43.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29193.792551 secs 2023-03-09T23:04:43.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:42 smithi181 ceph-mon[94189]: pgmap v16081: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 230 KiB/s wr, 9 op/s 2023-03-09T23:04:43.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29193.792551 secs 2023-03-09T23:04:43.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:42 smithi038 ceph-mon[92422]: pgmap v16081: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 230 KiB/s wr, 9 op/s 2023-03-09T23:04:43.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29193.792551 secs 2023-03-09T23:04:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29655.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:04:45.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:44 smithi122 ceph-mon[93869]: pgmap v16082: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 230 KiB/s wr, 7 op/s 2023-03-09T23:04:45.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:44 smithi181 ceph-mon[94189]: pgmap v16082: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 230 KiB/s wr, 7 op/s 2023-03-09T23:04:45.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:44 smithi038 ceph-mon[92422]: pgmap v16082: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 230 KiB/s wr, 7 op/s 2023-03-09T23:04:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29196.401543 secs 2023-03-09T23:04:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29196.401543 secs 2023-03-09T23:04:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29196.401543 secs 2023-03-09T23:04:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:46 smithi122 ceph-mon[93869]: pgmap v16083: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 369 KiB/s wr, 10 op/s 2023-03-09T23:04:47.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:46 smithi181 ceph-mon[94189]: pgmap v16083: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 369 KiB/s wr, 10 op/s 2023-03-09T23:04:47.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:46 smithi038 ceph-mon[92422]: pgmap v16083: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 369 KiB/s wr, 10 op/s 2023-03-09T23:04:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29198.792670 secs 2023-03-09T23:04:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29198.792670 secs 2023-03-09T23:04:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29198.792670 secs 2023-03-09T23:04:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29660.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:04:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:48 smithi038 ceph-mon[92422]: pgmap v16084: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 368 KiB/s wr, 7 op/s 2023-03-09T23:04:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:04:49.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:04:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:48 smithi122 ceph-mon[93869]: pgmap v16084: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 368 KiB/s wr, 7 op/s 2023-03-09T23:04:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:04:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:04:49.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:48 smithi181 ceph-mon[94189]: pgmap v16084: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 368 KiB/s wr, 7 op/s 2023-03-09T23:04:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:04:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:04:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:50 smithi038 ceph-mon[92422]: pgmap v16085: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 368 KiB/s wr, 7 op/s 2023-03-09T23:04:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29201.401605 secs 2023-03-09T23:04:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:50 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:04:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:50 smithi122 ceph-mon[93869]: pgmap v16085: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 368 KiB/s wr, 7 op/s 2023-03-09T23:04:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29201.401605 secs 2023-03-09T23:04:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:50 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:04:51.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:50 smithi181 ceph-mon[94189]: pgmap v16085: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 368 KiB/s wr, 7 op/s 2023-03-09T23:04:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29201.401605 secs 2023-03-09T23:04:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:50 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:04:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:04:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:04:52.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:04:52.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:04:52.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:04:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:04:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:04:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:04:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:04:52.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:04:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:04:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:04:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:04:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:04:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:04:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:53 smithi122 ceph-mon[93869]: pgmap v16086: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 637 KiB/s rd, 632 KiB/s wr, 11 op/s 2023-03-09T23:04:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:04:53.480 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:04:53.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:53 smithi181 ceph-mon[94189]: pgmap v16086: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 637 KiB/s rd, 632 KiB/s wr, 11 op/s 2023-03-09T23:04:53.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:04:53.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:04:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:53 smithi038 ceph-mon[92422]: pgmap v16086: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 637 KiB/s rd, 632 KiB/s wr, 11 op/s 2023-03-09T23:04:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:04:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:04:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29665.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:04:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:54 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:04:54.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:54 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:04:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:54 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:04:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:55 smithi122 ceph-mon[93869]: pgmap v16087: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 611 KiB/s rd, 607 KiB/s wr, 9 op/s 2023-03-09T23:04:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:55 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:04:55.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:04:55.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:04:55.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29206.401662 secs 2023-03-09T23:04:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:55 smithi181 ceph-mon[94189]: pgmap v16087: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 611 KiB/s rd, 607 KiB/s wr, 9 op/s 2023-03-09T23:04:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:55 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:04:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:04:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:04:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29206.401662 secs 2023-03-09T23:04:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:55 smithi038 ceph-mon[92422]: pgmap v16087: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 611 KiB/s rd, 607 KiB/s wr, 9 op/s 2023-03-09T23:04:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:55 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:04:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:04:55.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:04:55.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29206.401662 secs 2023-03-09T23:04:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:04:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:04:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:04:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:04:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:04:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:04:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:57 smithi122 ceph-mon[93869]: pgmap v16088: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 759 KiB/s rd, 743 KiB/s wr, 13 op/s 2023-03-09T23:04:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:04:57.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:04:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:57 smithi181 ceph-mon[94189]: pgmap v16088: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 759 KiB/s rd, 743 KiB/s wr, 13 op/s 2023-03-09T23:04:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:04:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:04:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:57 smithi038 ceph-mon[92422]: pgmap v16088: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 759 KiB/s rd, 743 KiB/s wr, 13 op/s 2023-03-09T23:04:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:04:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:04:58.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29670.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:04:59.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:04:59 smithi122 ceph-mon[93869]: pgmap v16089: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 717 KiB/s rd, 703 KiB/s wr, 10 op/s 2023-03-09T23:04:59.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:04:59 smithi181 ceph-mon[94189]: pgmap v16089: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 717 KiB/s rd, 703 KiB/s wr, 10 op/s 2023-03-09T23:04:59.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:04:59 smithi038 ceph-mon[92422]: pgmap v16089: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 717 KiB/s rd, 703 KiB/s wr, 10 op/s 2023-03-09T23:05:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:05:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:05:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29211.401756 secs 2023-03-09T23:05:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:05:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:05:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29211.401756 secs 2023-03-09T23:05:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:05:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:05:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29211.401756 secs 2023-03-09T23:05:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:01 smithi122 ceph-mon[93869]: pgmap v16090: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 717 KiB/s rd, 703 KiB/s wr, 11 op/s 2023-03-09T23:05:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:05:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:05:01.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:01 smithi181 ceph-mon[94189]: pgmap v16090: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 717 KiB/s rd, 703 KiB/s wr, 11 op/s 2023-03-09T23:05:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:05:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:05:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:01 smithi038 ceph-mon[92422]: pgmap v16090: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 717 KiB/s rd, 703 KiB/s wr, 11 op/s 2023-03-09T23:05:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:05:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:05:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:03 smithi122 ceph-mon[93869]: pgmap v16091: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 18 op/s 2023-03-09T23:05:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29213.793090 secs 2023-03-09T23:05:03.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:03 smithi181 ceph-mon[94189]: pgmap v16091: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 18 op/s 2023-03-09T23:05:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29213.793090 secs 2023-03-09T23:05:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:03 smithi038 ceph-mon[92422]: pgmap v16091: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 18 op/s 2023-03-09T23:05:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29213.793090 secs 2023-03-09T23:05:03.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29675.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:05:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:05 smithi038 ceph-mon[92422]: pgmap v16092: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 769 KiB/s wr, 26 op/s 2023-03-09T23:05:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29216.401816 secs 2023-03-09T23:05:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:05 smithi122 ceph-mon[93869]: pgmap v16092: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 769 KiB/s wr, 26 op/s 2023-03-09T23:05:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29216.401816 secs 2023-03-09T23:05:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:05 smithi181 ceph-mon[94189]: pgmap v16092: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 769 KiB/s wr, 26 op/s 2023-03-09T23:05:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29216.401816 secs 2023-03-09T23:05:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:06 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:05:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:06 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:05:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:05:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:06 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:05:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:06 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:05:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:05:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:06 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:05:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:06 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:05:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:05:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:07 smithi038 ceph-mon[92422]: pgmap v16093: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 855 KiB/s rd, 878 KiB/s wr, 28 op/s 2023-03-09T23:05:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:07 smithi122 ceph-mon[93869]: pgmap v16093: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 855 KiB/s rd, 878 KiB/s wr, 28 op/s 2023-03-09T23:05:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:07 smithi181 ceph-mon[94189]: pgmap v16093: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 855 KiB/s rd, 878 KiB/s wr, 28 op/s 2023-03-09T23:05:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29218.793194 secs 2023-03-09T23:05:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29218.793194 secs 2023-03-09T23:05:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29218.793194 secs 2023-03-09T23:05:08.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29680.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:05:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:09 smithi122 ceph-mon[93869]: pgmap v16094: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 708 KiB/s rd, 742 KiB/s wr, 25 op/s 2023-03-09T23:05:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:09 smithi181 ceph-mon[94189]: pgmap v16094: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 708 KiB/s rd, 742 KiB/s wr, 25 op/s 2023-03-09T23:05:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:09 smithi038 ceph-mon[92422]: pgmap v16094: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 708 KiB/s rd, 742 KiB/s wr, 25 op/s 2023-03-09T23:05:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:10 smithi122 ceph-mon[93869]: osdmap e1221: 12 total, 12 up, 12 in 2023-03-09T23:05:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29221.401878 secs 2023-03-09T23:05:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:10 smithi181 ceph-mon[94189]: osdmap e1221: 12 total, 12 up, 12 in 2023-03-09T23:05:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29221.401878 secs 2023-03-09T23:05:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:10 smithi038 ceph-mon[92422]: osdmap e1221: 12 total, 12 up, 12 in 2023-03-09T23:05:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29221.401878 secs 2023-03-09T23:05:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:11 smithi122 ceph-mon[93869]: pgmap v16096: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 901 KiB/s rd, 942 KiB/s wr, 30 op/s 2023-03-09T23:05:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:11 smithi181 ceph-mon[94189]: pgmap v16096: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 901 KiB/s rd, 942 KiB/s wr, 30 op/s 2023-03-09T23:05:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:11 smithi038 ceph-mon[92422]: pgmap v16096: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 901 KiB/s rd, 942 KiB/s wr, 30 op/s 2023-03-09T23:05:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:05:12.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:05:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:05:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:05:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:05:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:05:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:13 smithi122 ceph-mon[93869]: pgmap v16097: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 352 KiB/s wr, 23 op/s 2023-03-09T23:05:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:05:13.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:05:13.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29223.793310 secs 2023-03-09T23:05:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:13 smithi181 ceph-mon[94189]: pgmap v16097: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 352 KiB/s wr, 23 op/s 2023-03-09T23:05:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:05:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:05:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29223.793310 secs 2023-03-09T23:05:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:13 smithi038 ceph-mon[92422]: pgmap v16097: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 352 KiB/s wr, 23 op/s 2023-03-09T23:05:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:05:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:05:13.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29223.793310 secs 2023-03-09T23:05:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29685.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:05:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:15 smithi122 ceph-mon[93869]: pgmap v16098: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 326 KiB/s wr, 9 op/s 2023-03-09T23:05:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29226.401962 secs 2023-03-09T23:05:15.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:15 smithi181 ceph-mon[94189]: pgmap v16098: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 326 KiB/s wr, 9 op/s 2023-03-09T23:05:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29226.401962 secs 2023-03-09T23:05:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:15 smithi038 ceph-mon[92422]: pgmap v16098: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 326 KiB/s wr, 9 op/s 2023-03-09T23:05:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29226.401962 secs 2023-03-09T23:05:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:17 smithi122 ceph-mon[93869]: pgmap v16099: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 195 KiB/s wr, 8 op/s 2023-03-09T23:05:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:17 smithi122 ceph-mon[93869]: osdmap e1222: 12 total, 12 up, 12 in 2023-03-09T23:05:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:17 smithi181 ceph-mon[94189]: pgmap v16099: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 195 KiB/s wr, 8 op/s 2023-03-09T23:05:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:17 smithi181 ceph-mon[94189]: osdmap e1222: 12 total, 12 up, 12 in 2023-03-09T23:05:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:17 smithi038 ceph-mon[92422]: pgmap v16099: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 195 KiB/s wr, 8 op/s 2023-03-09T23:05:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:17 smithi038 ceph-mon[92422]: osdmap e1222: 12 total, 12 up, 12 in 2023-03-09T23:05:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29228.793449 secs 2023-03-09T23:05:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29228.793449 secs 2023-03-09T23:05:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29228.793449 secs 2023-03-09T23:05:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29690.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:05:19.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:19 smithi122 ceph-mon[93869]: pgmap v16101: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 29 KiB/s wr, 8 op/s 2023-03-09T23:05:19.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:19 smithi181 ceph-mon[94189]: pgmap v16101: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 29 KiB/s wr, 8 op/s 2023-03-09T23:05:19.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:19 smithi038 ceph-mon[92422]: pgmap v16101: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 29 KiB/s wr, 8 op/s 2023-03-09T23:05:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29231.402003 secs 2023-03-09T23:05:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29231.402003 secs 2023-03-09T23:05:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29231.402003 secs 2023-03-09T23:05:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:21 smithi122 ceph-mon[93869]: pgmap v16102: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 25 KiB/s wr, 7 op/s 2023-03-09T23:05:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:21 smithi181 ceph-mon[94189]: pgmap v16102: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 25 KiB/s wr, 7 op/s 2023-03-09T23:05:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:21 smithi038 ceph-mon[92422]: pgmap v16102: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 25 KiB/s wr, 7 op/s 2023-03-09T23:05:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:23 smithi122 ceph-mon[93869]: pgmap v16103: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 307 B/s wr, 7 op/s 2023-03-09T23:05:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29233.793522 secs 2023-03-09T23:05:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:23 smithi181 ceph-mon[94189]: pgmap v16103: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 307 B/s wr, 7 op/s 2023-03-09T23:05:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29233.793522 secs 2023-03-09T23:05:23.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:23 smithi038 ceph-mon[92422]: pgmap v16103: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 307 B/s wr, 7 op/s 2023-03-09T23:05:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29233.793522 secs 2023-03-09T23:05:23.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29695.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:05:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:25 smithi122 ceph-mon[93869]: pgmap v16104: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 307 B/s wr, 7 op/s 2023-03-09T23:05:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29236.402061 secs 2023-03-09T23:05:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:25 smithi181 ceph-mon[94189]: pgmap v16104: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 307 B/s wr, 7 op/s 2023-03-09T23:05:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29236.402061 secs 2023-03-09T23:05:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:25 smithi038 ceph-mon[92422]: pgmap v16104: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 307 B/s wr, 7 op/s 2023-03-09T23:05:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29236.402061 secs 2023-03-09T23:05:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:27 smithi122 ceph-mon[93869]: pgmap v16105: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 7 op/s 2023-03-09T23:05:27.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:27 smithi181 ceph-mon[94189]: pgmap v16105: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 7 op/s 2023-03-09T23:05:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:27 smithi038 ceph-mon[92422]: pgmap v16105: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 102 B/s wr, 7 op/s 2023-03-09T23:05:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29238.793665 secs 2023-03-09T23:05:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29238.793665 secs 2023-03-09T23:05:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29238.793665 secs 2023-03-09T23:05:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29700.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:05:29.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:29 smithi122 ceph-mon[93869]: pgmap v16106: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 93 B/s wr, 6 op/s 2023-03-09T23:05:29.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:29 smithi181 ceph-mon[94189]: pgmap v16106: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 93 B/s wr, 6 op/s 2023-03-09T23:05:29.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:29 smithi038 ceph-mon[92422]: pgmap v16106: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 93 B/s wr, 6 op/s 2023-03-09T23:05:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29241.402122 secs 2023-03-09T23:05:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29241.402122 secs 2023-03-09T23:05:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29241.402122 secs 2023-03-09T23:05:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:31 smithi122 ceph-mon[93869]: pgmap v16107: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-09T23:05:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:31 smithi181 ceph-mon[94189]: pgmap v16107: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-09T23:05:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:31 smithi038 ceph-mon[92422]: pgmap v16107: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-09T23:05:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:33 smithi122 ceph-mon[93869]: pgmap v16108: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 426 B/s wr, 8 op/s 2023-03-09T23:05:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29243.793748 secs 2023-03-09T23:05:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:33 smithi181 ceph-mon[94189]: pgmap v16108: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 426 B/s wr, 8 op/s 2023-03-09T23:05:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29243.793748 secs 2023-03-09T23:05:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:33 smithi038 ceph-mon[92422]: pgmap v16108: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 426 B/s wr, 8 op/s 2023-03-09T23:05:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29243.793748 secs 2023-03-09T23:05:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29705.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:05:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:35 smithi122 ceph-mon[93869]: pgmap v16109: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-09T23:05:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29246.402186 secs 2023-03-09T23:05:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:35 smithi181 ceph-mon[94189]: pgmap v16109: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-09T23:05:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29246.402186 secs 2023-03-09T23:05:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:35 smithi038 ceph-mon[92422]: pgmap v16109: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-09T23:05:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29246.402186 secs 2023-03-09T23:05:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:37 smithi122 ceph-mon[93869]: pgmap v16110: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 126 KiB/s wr, 9 op/s 2023-03-09T23:05:37.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:37 smithi181 ceph-mon[94189]: pgmap v16110: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 126 KiB/s wr, 9 op/s 2023-03-09T23:05:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:37 smithi038 ceph-mon[92422]: pgmap v16110: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 126 KiB/s wr, 9 op/s 2023-03-09T23:05:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29248.793865 secs 2023-03-09T23:05:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29248.793865 secs 2023-03-09T23:05:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29248.793865 secs 2023-03-09T23:05:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29710.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:05:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:39 smithi122 ceph-mon[93869]: pgmap v16111: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 126 KiB/s wr, 6 op/s 2023-03-09T23:05:39.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:39 smithi181 ceph-mon[94189]: pgmap v16111: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 126 KiB/s wr, 6 op/s 2023-03-09T23:05:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:39 smithi038 ceph-mon[92422]: pgmap v16111: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 126 KiB/s wr, 6 op/s 2023-03-09T23:05:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29251.402244 secs 2023-03-09T23:05:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29251.402244 secs 2023-03-09T23:05:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29251.402244 secs 2023-03-09T23:05:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:41 smithi122 ceph-mon[93869]: pgmap v16112: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 126 KiB/s wr, 7 op/s 2023-03-09T23:05:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:41 smithi181 ceph-mon[94189]: pgmap v16112: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 126 KiB/s wr, 7 op/s 2023-03-09T23:05:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:41 smithi038 ceph-mon[92422]: pgmap v16112: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 126 KiB/s wr, 7 op/s 2023-03-09T23:05:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:43 smithi122 ceph-mon[93869]: pgmap v16113: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 126 KiB/s wr, 9 op/s 2023-03-09T23:05:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:43 smithi181 ceph-mon[94189]: pgmap v16113: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 126 KiB/s wr, 9 op/s 2023-03-09T23:05:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:43 smithi038 ceph-mon[92422]: pgmap v16113: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 126 KiB/s wr, 9 op/s 2023-03-09T23:05:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29715.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:05:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:45 smithi122 ceph-mon[93869]: pgmap v16114: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 126 KiB/s wr, 6 op/s 2023-03-09T23:05:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29256.402305 secs 2023-03-09T23:05:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:45 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:05:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:45 smithi181 ceph-mon[94189]: pgmap v16114: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 126 KiB/s wr, 6 op/s 2023-03-09T23:05:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29256.402305 secs 2023-03-09T23:05:45.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:45 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:05:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:45 smithi038 ceph-mon[92422]: pgmap v16114: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 126 KiB/s wr, 6 op/s 2023-03-09T23:05:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29256.402305 secs 2023-03-09T23:05:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:45 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:05:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:46 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:05:46.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:46 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:05:46.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:46 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:05:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:46 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:05:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:46 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:05:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:46 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:05:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:46 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:05:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:46 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:05:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:46 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:05:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:47 smithi122 ceph-mon[93869]: pgmap v16115: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 567 KiB/s rd, 265 KiB/s wr, 9 op/s 2023-03-09T23:05:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:47 smithi181 ceph-mon[94189]: pgmap v16115: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 567 KiB/s rd, 265 KiB/s wr, 9 op/s 2023-03-09T23:05:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:47 smithi038 ceph-mon[92422]: pgmap v16115: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 567 KiB/s rd, 265 KiB/s wr, 9 op/s 2023-03-09T23:05:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29720.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:05:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:49 smithi122 ceph-mon[93869]: pgmap v16116: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 440 KiB/s rd, 140 KiB/s wr, 6 op/s 2023-03-09T23:05:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:05:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:05:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:49 smithi181 ceph-mon[94189]: pgmap v16116: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 440 KiB/s rd, 140 KiB/s wr, 6 op/s 2023-03-09T23:05:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:05:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:05:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:49 smithi038 ceph-mon[92422]: pgmap v16116: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 440 KiB/s rd, 140 KiB/s wr, 6 op/s 2023-03-09T23:05:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:05:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:05:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29261.402363 secs 2023-03-09T23:05:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29261.402363 secs 2023-03-09T23:05:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29261.402363 secs 2023-03-09T23:05:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:51 smithi122 ceph-mon[93869]: pgmap v16117: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 728 KiB/s rd, 354 KiB/s wr, 7 op/s 2023-03-09T23:05:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:05:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:51 smithi181 ceph-mon[94189]: pgmap v16117: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 728 KiB/s rd, 354 KiB/s wr, 7 op/s 2023-03-09T23:05:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:05:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:51 smithi038 ceph-mon[92422]: pgmap v16117: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 728 KiB/s rd, 354 KiB/s wr, 7 op/s 2023-03-09T23:05:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:05:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:05:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:05:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:05:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:05:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:05:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:05:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:05:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:05:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:05:52.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:05:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:05:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:05:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:05:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:05:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:05:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:53 smithi122 ceph-mon[93869]: pgmap v16118: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 677 KiB/s wr, 10 op/s 2023-03-09T23:05:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:53 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 29263.794246 secs 2023-03-09T23:05:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:05:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:05:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:53 smithi181 ceph-mon[94189]: pgmap v16118: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 677 KiB/s wr, 10 op/s 2023-03-09T23:05:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:53 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 29263.794246 secs 2023-03-09T23:05:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:05:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:05:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:53 smithi038 ceph-mon[92422]: pgmap v16118: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 677 KiB/s wr, 10 op/s 2023-03-09T23:05:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:53 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 29263.794246 secs 2023-03-09T23:05:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:05:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:05:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29725.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:05:54.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:54 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:05:54.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:54 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:05:54.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:54 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:05:54.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:54 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:05:54.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:54 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:05:54.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:54 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:05:54.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:54 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:05:54.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:54 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:05:54.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:54 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:05:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:55 smithi122 ceph-mon[93869]: pgmap v16119: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 805 KiB/s wr, 9 op/s 2023-03-09T23:05:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29266.402432 secs 2023-03-09T23:05:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:55 smithi181 ceph-mon[94189]: pgmap v16119: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 805 KiB/s wr, 9 op/s 2023-03-09T23:05:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29266.402432 secs 2023-03-09T23:05:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:55 smithi038 ceph-mon[92422]: pgmap v16119: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 805 KiB/s wr, 9 op/s 2023-03-09T23:05:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29266.402432 secs 2023-03-09T23:05:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:05:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:05:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:05:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:05:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:05:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:05:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:57 smithi122 ceph-mon[93869]: pgmap v16120: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.0 MiB/s wr, 12 op/s 2023-03-09T23:05:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:05:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:05:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:57 smithi181 ceph-mon[94189]: pgmap v16120: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.0 MiB/s wr, 12 op/s 2023-03-09T23:05:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:05:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:05:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:57 smithi038 ceph-mon[92422]: pgmap v16120: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.0 MiB/s wr, 12 op/s 2023-03-09T23:05:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:05:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:05:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:58 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 29268.794341 secs 2023-03-09T23:05:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29730.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:05:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:58 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 29268.794341 secs 2023-03-09T23:05:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:58 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 29268.794341 secs 2023-03-09T23:05:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:05:59 smithi038 ceph-mon[92422]: pgmap v16121: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 983 KiB/s rd, 895 KiB/s wr, 10 op/s 2023-03-09T23:05:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:05:59 smithi122 ceph-mon[93869]: pgmap v16121: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 983 KiB/s rd, 895 KiB/s wr, 10 op/s 2023-03-09T23:05:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:05:59 smithi181 ceph-mon[94189]: pgmap v16121: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 983 KiB/s rd, 895 KiB/s wr, 10 op/s 2023-03-09T23:06:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:06:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:06:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29271.402494 secs 2023-03-09T23:06:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:06:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:06:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29271.402494 secs 2023-03-09T23:06:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:06:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:06:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29271.402494 secs 2023-03-09T23:06:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:01 smithi038 ceph-mon[92422]: pgmap v16122: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.0 MiB/s wr, 11 op/s 2023-03-09T23:06:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:06:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:06:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:01 smithi122 ceph-mon[93869]: pgmap v16122: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.0 MiB/s wr, 11 op/s 2023-03-09T23:06:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:06:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:06:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:01 smithi181 ceph-mon[94189]: pgmap v16122: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.0 MiB/s wr, 11 op/s 2023-03-09T23:06:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:06:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:06:03.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29735.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:06:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:03 smithi122 ceph-mon[93869]: pgmap v16123: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.2 MiB/s wr, 16 op/s 2023-03-09T23:06:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29273.794463 secs 2023-03-09T23:06:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:03 smithi181 ceph-mon[94189]: pgmap v16123: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.2 MiB/s wr, 16 op/s 2023-03-09T23:06:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29273.794463 secs 2023-03-09T23:06:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:03 smithi038 ceph-mon[92422]: pgmap v16123: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.2 MiB/s wr, 16 op/s 2023-03-09T23:06:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29273.794463 secs 2023-03-09T23:06:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:05 smithi122 ceph-mon[93869]: pgmap v16124: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 929 KiB/s wr, 27 op/s 2023-03-09T23:06:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29276.402577 secs 2023-03-09T23:06:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:05 smithi181 ceph-mon[94189]: pgmap v16124: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 929 KiB/s wr, 27 op/s 2023-03-09T23:06:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29276.402577 secs 2023-03-09T23:06:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:05 smithi038 ceph-mon[92422]: pgmap v16124: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 929 KiB/s wr, 27 op/s 2023-03-09T23:06:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29276.402577 secs 2023-03-09T23:06:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:06 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:06:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:06:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:06 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:06:06.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:06:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:06 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:06:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:06:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:07 smithi122 ceph-mon[93869]: pgmap v16125: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 801 KiB/s wr, 29 op/s 2023-03-09T23:06:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:06:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:06:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:07 smithi181 ceph-mon[94189]: pgmap v16125: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 801 KiB/s wr, 29 op/s 2023-03-09T23:06:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:06:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:06:07.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:07 smithi038 ceph-mon[92422]: pgmap v16125: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 801 KiB/s wr, 29 op/s 2023-03-09T23:06:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:06:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:06:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29740.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:06:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29278.794582 secs 2023-03-09T23:06:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29278.794582 secs 2023-03-09T23:06:09.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29278.794582 secs 2023-03-09T23:06:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:09 smithi122 ceph-mon[93869]: pgmap v16126: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 896 KiB/s rd, 572 KiB/s wr, 26 op/s 2023-03-09T23:06:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:06:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:06:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:09 smithi181 ceph-mon[94189]: pgmap v16126: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 896 KiB/s rd, 572 KiB/s wr, 26 op/s 2023-03-09T23:06:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:06:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:06:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:09 smithi038 ceph-mon[92422]: pgmap v16126: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 896 KiB/s rd, 572 KiB/s wr, 26 op/s 2023-03-09T23:06:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:06:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:06:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:10 smithi122 ceph-mon[93869]: osdmap e1223: 12 total, 12 up, 12 in 2023-03-09T23:06:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29281.402657 secs 2023-03-09T23:06:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:10 smithi181 ceph-mon[94189]: osdmap e1223: 12 total, 12 up, 12 in 2023-03-09T23:06:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29281.402657 secs 2023-03-09T23:06:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:10 smithi038 ceph-mon[92422]: osdmap e1223: 12 total, 12 up, 12 in 2023-03-09T23:06:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29281.402657 secs 2023-03-09T23:06:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:11 smithi122 ceph-mon[93869]: pgmap v16128: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 680 KiB/s rd, 718 KiB/s wr, 31 op/s 2023-03-09T23:06:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:11 smithi181 ceph-mon[94189]: pgmap v16128: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 680 KiB/s rd, 718 KiB/s wr, 31 op/s 2023-03-09T23:06:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:11 smithi038 ceph-mon[92422]: pgmap v16128: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 680 KiB/s rd, 718 KiB/s wr, 31 op/s 2023-03-09T23:06:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29745.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:06:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:13 smithi122 ceph-mon[93869]: pgmap v16129: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 255 KiB/s wr, 26 op/s 2023-03-09T23:06:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29283.794699 secs 2023-03-09T23:06:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:13 smithi181 ceph-mon[94189]: pgmap v16129: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 255 KiB/s wr, 26 op/s 2023-03-09T23:06:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29283.794699 secs 2023-03-09T23:06:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:13 smithi038 ceph-mon[92422]: pgmap v16129: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 255 KiB/s wr, 26 op/s 2023-03-09T23:06:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29283.794699 secs 2023-03-09T23:06:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:15 smithi122 ceph-mon[93869]: pgmap v16130: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 324 KiB/s wr, 10 op/s 2023-03-09T23:06:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29286.402718 secs 2023-03-09T23:06:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:15 smithi181 ceph-mon[94189]: pgmap v16130: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 324 KiB/s wr, 10 op/s 2023-03-09T23:06:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29286.402718 secs 2023-03-09T23:06:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:15 smithi038 ceph-mon[92422]: pgmap v16130: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 324 KiB/s wr, 10 op/s 2023-03-09T23:06:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29286.402718 secs 2023-03-09T23:06:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:17 smithi122 ceph-mon[93869]: pgmap v16131: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 326 KiB/s rd, 324 KiB/s wr, 8 op/s 2023-03-09T23:06:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:17 smithi122 ceph-mon[93869]: osdmap e1224: 12 total, 12 up, 12 in 2023-03-09T23:06:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:17 smithi181 ceph-mon[94189]: pgmap v16131: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 326 KiB/s rd, 324 KiB/s wr, 8 op/s 2023-03-09T23:06:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:17 smithi181 ceph-mon[94189]: osdmap e1224: 12 total, 12 up, 12 in 2023-03-09T23:06:18.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:17 smithi038 ceph-mon[92422]: pgmap v16131: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 326 KiB/s rd, 324 KiB/s wr, 8 op/s 2023-03-09T23:06:18.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:17 smithi038 ceph-mon[92422]: osdmap e1224: 12 total, 12 up, 12 in 2023-03-09T23:06:18.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29750.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:06:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29288.794844 secs 2023-03-09T23:06:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29288.794844 secs 2023-03-09T23:06:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29288.794844 secs 2023-03-09T23:06:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:19 smithi122 ceph-mon[93869]: pgmap v16133: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 150 KiB/s wr, 7 op/s 2023-03-09T23:06:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:19 smithi181 ceph-mon[94189]: pgmap v16133: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 150 KiB/s wr, 7 op/s 2023-03-09T23:06:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:19 smithi038 ceph-mon[92422]: pgmap v16133: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 150 KiB/s wr, 7 op/s 2023-03-09T23:06:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29291.402779 secs 2023-03-09T23:06:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29291.402779 secs 2023-03-09T23:06:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29291.402779 secs 2023-03-09T23:06:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:21 smithi122 ceph-mon[93869]: pgmap v16134: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 229 KiB/s wr, 8 op/s 2023-03-09T23:06:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:21 smithi181 ceph-mon[94189]: pgmap v16134: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 229 KiB/s wr, 8 op/s 2023-03-09T23:06:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:21 smithi038 ceph-mon[92422]: pgmap v16134: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 229 KiB/s wr, 8 op/s 2023-03-09T23:06:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:22 smithi122 ceph-mon[93869]: pgmap v16135: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 229 KiB/s wr, 7 op/s 2023-03-09T23:06:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29293.794967 secs 2023-03-09T23:06:23.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:22 smithi181 ceph-mon[94189]: pgmap v16135: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 229 KiB/s wr, 7 op/s 2023-03-09T23:06:23.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29293.794967 secs 2023-03-09T23:06:23.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:22 smithi038 ceph-mon[92422]: pgmap v16135: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 229 KiB/s wr, 7 op/s 2023-03-09T23:06:23.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29293.794967 secs 2023-03-09T23:06:23.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29755.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:06:25.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:24 smithi122 ceph-mon[93869]: pgmap v16136: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 104 KiB/s wr, 7 op/s 2023-03-09T23:06:25.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:24 smithi181 ceph-mon[94189]: pgmap v16136: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 104 KiB/s wr, 7 op/s 2023-03-09T23:06:25.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:24 smithi038 ceph-mon[92422]: pgmap v16136: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 104 KiB/s wr, 7 op/s 2023-03-09T23:06:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29296.402841 secs 2023-03-09T23:06:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29296.402841 secs 2023-03-09T23:06:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29296.402841 secs 2023-03-09T23:06:27.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:26 smithi181 ceph-mon[94189]: pgmap v16137: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 104 KiB/s wr, 7 op/s 2023-03-09T23:06:27.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:26 smithi038 ceph-mon[92422]: pgmap v16137: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 104 KiB/s wr, 7 op/s 2023-03-09T23:06:27.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:26 smithi122 ceph-mon[93869]: pgmap v16137: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 104 KiB/s wr, 7 op/s 2023-03-09T23:06:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29298.795098 secs 2023-03-09T23:06:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29298.795098 secs 2023-03-09T23:06:28.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29298.795098 secs 2023-03-09T23:06:28.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29760.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:06:29.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:28 smithi038 ceph-mon[92422]: pgmap v16138: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 95 KiB/s wr, 7 op/s 2023-03-09T23:06:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:28 smithi122 ceph-mon[93869]: pgmap v16138: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 95 KiB/s wr, 7 op/s 2023-03-09T23:06:29.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:28 smithi181 ceph-mon[94189]: pgmap v16138: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 95 KiB/s wr, 7 op/s 2023-03-09T23:06:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:30 smithi038 ceph-mon[92422]: pgmap v16139: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 212 KiB/s wr, 7 op/s 2023-03-09T23:06:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29301.402900 secs 2023-03-09T23:06:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:30 smithi122 ceph-mon[93869]: pgmap v16139: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 212 KiB/s wr, 7 op/s 2023-03-09T23:06:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29301.402900 secs 2023-03-09T23:06:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:30 smithi181 ceph-mon[94189]: pgmap v16139: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 212 KiB/s wr, 7 op/s 2023-03-09T23:06:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29301.402900 secs 2023-03-09T23:06:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:32 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:06:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:32 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:06:32.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:32 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:06:32.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:32 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:06:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:32 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:06:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:32 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:06:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:33 smithi122 ceph-mon[93869]: pgmap v16140: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 126 KiB/s wr, 7 op/s 2023-03-09T23:06:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:33 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:06:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:33 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:06:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29303.795214 secs 2023-03-09T23:06:33.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:33 smithi181 ceph-mon[94189]: pgmap v16140: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 126 KiB/s wr, 7 op/s 2023-03-09T23:06:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:33 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:06:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:33 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:06:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29303.795214 secs 2023-03-09T23:06:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:33 smithi038 ceph-mon[92422]: pgmap v16140: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 126 KiB/s wr, 7 op/s 2023-03-09T23:06:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:33 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:06:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:33 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:06:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29303.795214 secs 2023-03-09T23:06:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29765.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:06:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:35 smithi122 ceph-mon[93869]: pgmap v16141: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 145 KiB/s wr, 7 op/s 2023-03-09T23:06:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:35 smithi181 ceph-mon[94189]: pgmap v16141: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 145 KiB/s wr, 7 op/s 2023-03-09T23:06:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:35 smithi038 ceph-mon[92422]: pgmap v16141: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 145 KiB/s wr, 7 op/s 2023-03-09T23:06:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29306.402961 secs 2023-03-09T23:06:36.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29306.402961 secs 2023-03-09T23:06:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29306.402961 secs 2023-03-09T23:06:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:37 smithi122 ceph-mon[93869]: pgmap v16142: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 145 KiB/s wr, 8 op/s 2023-03-09T23:06:37.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:37 smithi181 ceph-mon[94189]: pgmap v16142: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 145 KiB/s wr, 8 op/s 2023-03-09T23:06:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:37 smithi038 ceph-mon[92422]: pgmap v16142: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 145 KiB/s wr, 8 op/s 2023-03-09T23:06:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29308.795333 secs 2023-03-09T23:06:38.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29308.795333 secs 2023-03-09T23:06:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29308.795333 secs 2023-03-09T23:06:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29770.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:06:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:39 smithi122 ceph-mon[93869]: pgmap v16143: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 175 KiB/s wr, 6 op/s 2023-03-09T23:06:39.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:39 smithi181 ceph-mon[94189]: pgmap v16143: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 175 KiB/s wr, 6 op/s 2023-03-09T23:06:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:39 smithi038 ceph-mon[92422]: pgmap v16143: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 175 KiB/s wr, 6 op/s 2023-03-09T23:06:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:41 smithi122 ceph-mon[93869]: pgmap v16144: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 356 KiB/s wr, 7 op/s 2023-03-09T23:06:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29311.403007 secs 2023-03-09T23:06:41.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:41 smithi181 ceph-mon[94189]: pgmap v16144: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 356 KiB/s wr, 7 op/s 2023-03-09T23:06:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29311.403007 secs 2023-03-09T23:06:41.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:41 smithi038 ceph-mon[92422]: pgmap v16144: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 356 KiB/s wr, 7 op/s 2023-03-09T23:06:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29311.403007 secs 2023-03-09T23:06:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:43 smithi122 ceph-mon[93869]: pgmap v16145: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 231 KiB/s wr, 8 op/s 2023-03-09T23:06:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29313.795457 secs 2023-03-09T23:06:43.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:43 smithi181 ceph-mon[94189]: pgmap v16145: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 231 KiB/s wr, 8 op/s 2023-03-09T23:06:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29313.795457 secs 2023-03-09T23:06:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:43 smithi038 ceph-mon[92422]: pgmap v16145: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 231 KiB/s wr, 8 op/s 2023-03-09T23:06:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29313.795457 secs 2023-03-09T23:06:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29775.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:06:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:45 smithi122 ceph-mon[93869]: pgmap v16146: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 231 KiB/s wr, 8 op/s 2023-03-09T23:06:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:45 smithi181 ceph-mon[94189]: pgmap v16146: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 231 KiB/s wr, 8 op/s 2023-03-09T23:06:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:45 smithi038 ceph-mon[92422]: pgmap v16146: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 231 KiB/s wr, 8 op/s 2023-03-09T23:06:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29316.403066 secs 2023-03-09T23:06:46.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29316.403066 secs 2023-03-09T23:06:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29316.403066 secs 2023-03-09T23:06:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:47 smithi122 ceph-mon[93869]: pgmap v16147: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 212 KiB/s wr, 8 op/s 2023-03-09T23:06:47.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:47 smithi181 ceph-mon[94189]: pgmap v16147: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 212 KiB/s wr, 8 op/s 2023-03-09T23:06:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:47 smithi038 ceph-mon[92422]: pgmap v16147: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 212 KiB/s wr, 8 op/s 2023-03-09T23:06:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29318.795575 secs 2023-03-09T23:06:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29318.795575 secs 2023-03-09T23:06:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29318.795575 secs 2023-03-09T23:06:48.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29780.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:06:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:49 smithi122 ceph-mon[93869]: pgmap v16148: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 352 KiB/s wr, 7 op/s 2023-03-09T23:06:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:06:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:06:49.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:49 smithi181 ceph-mon[94189]: pgmap v16148: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 352 KiB/s wr, 7 op/s 2023-03-09T23:06:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:06:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:06:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:49 smithi038 ceph-mon[92422]: pgmap v16148: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 352 KiB/s wr, 7 op/s 2023-03-09T23:06:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:06:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:06:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:51 smithi122 ceph-mon[93869]: pgmap v16149: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 355 KiB/s rd, 322 KiB/s wr, 7 op/s 2023-03-09T23:06:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29321.403144 secs 2023-03-09T23:06:51.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:51 smithi181 ceph-mon[94189]: pgmap v16149: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 355 KiB/s rd, 322 KiB/s wr, 7 op/s 2023-03-09T23:06:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29321.403144 secs 2023-03-09T23:06:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:51 smithi038 ceph-mon[92422]: pgmap v16149: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 355 KiB/s rd, 322 KiB/s wr, 7 op/s 2023-03-09T23:06:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29321.403144 secs 2023-03-09T23:06:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:06:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:06:52.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:06:52.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:06:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:06:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:06:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:53 smithi122 ceph-mon[93869]: pgmap v16150: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 141 KiB/s wr, 9 op/s 2023-03-09T23:06:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:06:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29323.795664 secs 2023-03-09T23:06:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:06:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:06:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:06:53.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:53 smithi181 ceph-mon[94189]: pgmap v16150: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 141 KiB/s wr, 9 op/s 2023-03-09T23:06:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:06:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29323.795664 secs 2023-03-09T23:06:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:06:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:06:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:06:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:53 smithi038 ceph-mon[92422]: pgmap v16150: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 141 KiB/s wr, 9 op/s 2023-03-09T23:06:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:06:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29323.795664 secs 2023-03-09T23:06:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:06:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:06:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:06:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29785.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:06:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:55 smithi122 ceph-mon[93869]: pgmap v16151: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 535 KiB/s rd, 408 KiB/s wr, 9 op/s 2023-03-09T23:06:55.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:55 smithi181 ceph-mon[94189]: pgmap v16151: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 535 KiB/s rd, 408 KiB/s wr, 9 op/s 2023-03-09T23:06:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:55 smithi038 ceph-mon[92422]: pgmap v16151: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 535 KiB/s rd, 408 KiB/s wr, 9 op/s 2023-03-09T23:06:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29326.403205 secs 2023-03-09T23:06:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:06:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:06:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29326.403205 secs 2023-03-09T23:06:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:06:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:06:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29326.403205 secs 2023-03-09T23:06:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:06:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:06:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:57 smithi122 ceph-mon[93869]: pgmap v16152: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 408 KiB/s wr, 10 op/s 2023-03-09T23:06:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:06:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:06:57.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:57 smithi181 ceph-mon[94189]: pgmap v16152: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 408 KiB/s wr, 10 op/s 2023-03-09T23:06:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:06:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:06:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:57 smithi038 ceph-mon[92422]: pgmap v16152: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 408 KiB/s wr, 10 op/s 2023-03-09T23:06:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:06:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:06:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29328.795783 secs 2023-03-09T23:06:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29328.795783 secs 2023-03-09T23:06:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29328.795783 secs 2023-03-09T23:06:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29790.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:06:59.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:06:59 smithi122 ceph-mon[93869]: pgmap v16153: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 408 KiB/s wr, 9 op/s 2023-03-09T23:06:59.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:06:59 smithi181 ceph-mon[94189]: pgmap v16153: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 408 KiB/s wr, 9 op/s 2023-03-09T23:06:59.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:06:59 smithi038 ceph-mon[92422]: pgmap v16153: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 408 KiB/s wr, 9 op/s 2023-03-09T23:07:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:07:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:07:00.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:07:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:07:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:07:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:07:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:01 smithi122 ceph-mon[93869]: pgmap v16154: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 576 KiB/s rd, 716 KiB/s wr, 11 op/s 2023-03-09T23:07:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29331.403268 secs 2023-03-09T23:07:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:07:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:07:01.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:01 smithi181 ceph-mon[94189]: pgmap v16154: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 576 KiB/s rd, 716 KiB/s wr, 11 op/s 2023-03-09T23:07:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29331.403268 secs 2023-03-09T23:07:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:07:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:07:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:01 smithi038 ceph-mon[92422]: pgmap v16154: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 576 KiB/s rd, 716 KiB/s wr, 11 op/s 2023-03-09T23:07:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29331.403268 secs 2023-03-09T23:07:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:07:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:07:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:03 smithi122 ceph-mon[93869]: pgmap v16155: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 578 KiB/s rd, 717 KiB/s wr, 13 op/s 2023-03-09T23:07:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29333.795905 secs 2023-03-09T23:07:03.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:03 smithi181 ceph-mon[94189]: pgmap v16155: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 578 KiB/s rd, 717 KiB/s wr, 13 op/s 2023-03-09T23:07:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29333.795905 secs 2023-03-09T23:07:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:03 smithi038 ceph-mon[92422]: pgmap v16155: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 578 KiB/s rd, 717 KiB/s wr, 13 op/s 2023-03-09T23:07:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29333.795905 secs 2023-03-09T23:07:03.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29795.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:07:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:05 smithi122 ceph-mon[93869]: pgmap v16156: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 872 KiB/s rd, 857 KiB/s wr, 14 op/s 2023-03-09T23:07:05.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:05 smithi181 ceph-mon[94189]: pgmap v16156: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 872 KiB/s rd, 857 KiB/s wr, 14 op/s 2023-03-09T23:07:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:05 smithi038 ceph-mon[92422]: pgmap v16156: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 872 KiB/s rd, 857 KiB/s wr, 14 op/s 2023-03-09T23:07:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29336.403330 secs 2023-03-09T23:07:06.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29336.403330 secs 2023-03-09T23:07:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29336.403330 secs 2023-03-09T23:07:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:07 smithi122 ceph-mon[93869]: pgmap v16157: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 608 KiB/s rd, 602 KiB/s wr, 15 op/s 2023-03-09T23:07:07.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:07 smithi181 ceph-mon[94189]: pgmap v16157: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 608 KiB/s rd, 602 KiB/s wr, 15 op/s 2023-03-09T23:07:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:07 smithi038 ceph-mon[92422]: pgmap v16157: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 608 KiB/s rd, 602 KiB/s wr, 15 op/s 2023-03-09T23:07:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29338.796007 secs 2023-03-09T23:07:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29338.796007 secs 2023-03-09T23:07:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29338.796007 secs 2023-03-09T23:07:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29800.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:07:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:09 smithi122 ceph-mon[93869]: pgmap v16158: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 604 KiB/s rd, 602 KiB/s wr, 13 op/s 2023-03-09T23:07:09.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:09 smithi181 ceph-mon[94189]: pgmap v16158: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 604 KiB/s rd, 602 KiB/s wr, 13 op/s 2023-03-09T23:07:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:09 smithi038 ceph-mon[92422]: pgmap v16158: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 604 KiB/s rd, 602 KiB/s wr, 13 op/s 2023-03-09T23:07:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29341.403392 secs 2023-03-09T23:07:10.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29341.403392 secs 2023-03-09T23:07:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29341.403392 secs 2023-03-09T23:07:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:11 smithi122 ceph-mon[93869]: pgmap v16159: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 750 KiB/s rd, 705 KiB/s wr, 24 op/s 2023-03-09T23:07:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:11 smithi181 ceph-mon[94189]: pgmap v16159: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 750 KiB/s rd, 705 KiB/s wr, 24 op/s 2023-03-09T23:07:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:11 smithi038 ceph-mon[92422]: pgmap v16159: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 750 KiB/s rd, 705 KiB/s wr, 24 op/s 2023-03-09T23:07:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:07:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:07:12.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:07:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:07:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:07:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:07:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:13 smithi122 ceph-mon[93869]: pgmap v16160: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 791 KiB/s rd, 600 KiB/s wr, 24 op/s 2023-03-09T23:07:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:07:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:07:13.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29343.796143 secs 2023-03-09T23:07:13.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:13 smithi181 ceph-mon[94189]: pgmap v16160: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 791 KiB/s rd, 600 KiB/s wr, 24 op/s 2023-03-09T23:07:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:07:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:07:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29343.796143 secs 2023-03-09T23:07:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:13 smithi038 ceph-mon[92422]: pgmap v16160: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 791 KiB/s rd, 600 KiB/s wr, 24 op/s 2023-03-09T23:07:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:07:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:07:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29343.796143 secs 2023-03-09T23:07:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29805.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:07:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:15 smithi122 ceph-mon[93869]: pgmap v16161: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 804 KiB/s rd, 622 KiB/s wr, 23 op/s 2023-03-09T23:07:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29346.403452 secs 2023-03-09T23:07:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:15 smithi181 ceph-mon[94189]: pgmap v16161: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 804 KiB/s rd, 622 KiB/s wr, 23 op/s 2023-03-09T23:07:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29346.403452 secs 2023-03-09T23:07:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:15 smithi038 ceph-mon[92422]: pgmap v16161: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 804 KiB/s rd, 622 KiB/s wr, 23 op/s 2023-03-09T23:07:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29346.403452 secs 2023-03-09T23:07:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:17 smithi122 ceph-mon[93869]: pgmap v16162: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 511 KiB/s rd, 483 KiB/s wr, 22 op/s 2023-03-09T23:07:17.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:17 smithi181 ceph-mon[94189]: pgmap v16162: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 511 KiB/s rd, 483 KiB/s wr, 22 op/s 2023-03-09T23:07:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:17 smithi038 ceph-mon[92422]: pgmap v16162: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 511 KiB/s rd, 483 KiB/s wr, 22 op/s 2023-03-09T23:07:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29348.796201 secs 2023-03-09T23:07:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29348.796201 secs 2023-03-09T23:07:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29348.796201 secs 2023-03-09T23:07:18.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29810.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:07:19.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:19 smithi038 ceph-mon[92422]: pgmap v16163: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 613 KiB/s rd, 576 KiB/s wr, 19 op/s 2023-03-09T23:07:19.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:19 smithi122 ceph-mon[93869]: pgmap v16163: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 613 KiB/s rd, 576 KiB/s wr, 19 op/s 2023-03-09T23:07:19.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:19 smithi181 ceph-mon[94189]: pgmap v16163: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 613 KiB/s rd, 576 KiB/s wr, 19 op/s 2023-03-09T23:07:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:07:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:07:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29351.403512 secs 2023-03-09T23:07:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:07:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:07:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29351.403512 secs 2023-03-09T23:07:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:07:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:07:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29351.403512 secs 2023-03-09T23:07:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:21 smithi122 ceph-mon[93869]: pgmap v16164: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 636 KiB/s rd, 576 KiB/s wr, 20 op/s 2023-03-09T23:07:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:07:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:07:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:21 smithi181 ceph-mon[94189]: pgmap v16164: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 636 KiB/s rd, 576 KiB/s wr, 20 op/s 2023-03-09T23:07:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:07:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:07:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:21 smithi038 ceph-mon[92422]: pgmap v16164: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 636 KiB/s rd, 576 KiB/s wr, 20 op/s 2023-03-09T23:07:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:07:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:07:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:23 smithi122 ceph-mon[93869]: pgmap v16165: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 560 KiB/s rd, 541 KiB/s wr, 10 op/s 2023-03-09T23:07:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29353.796282 secs 2023-03-09T23:07:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:23 smithi181 ceph-mon[94189]: pgmap v16165: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 560 KiB/s rd, 541 KiB/s wr, 10 op/s 2023-03-09T23:07:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29353.796282 secs 2023-03-09T23:07:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:23 smithi038 ceph-mon[92422]: pgmap v16165: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 560 KiB/s rd, 541 KiB/s wr, 10 op/s 2023-03-09T23:07:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29353.796282 secs 2023-03-09T23:07:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29815.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:07:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:25 smithi122 ceph-mon[93869]: pgmap v16166: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 232 KiB/s wr, 9 op/s 2023-03-09T23:07:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:25 smithi122 ceph-mon[93869]: osdmap e1225: 12 total, 12 up, 12 in 2023-03-09T23:07:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29356.403579 secs 2023-03-09T23:07:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:25 smithi181 ceph-mon[94189]: pgmap v16166: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 232 KiB/s wr, 9 op/s 2023-03-09T23:07:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:25 smithi181 ceph-mon[94189]: osdmap e1225: 12 total, 12 up, 12 in 2023-03-09T23:07:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29356.403579 secs 2023-03-09T23:07:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:25 smithi038 ceph-mon[92422]: pgmap v16166: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 232 KiB/s wr, 9 op/s 2023-03-09T23:07:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:25 smithi038 ceph-mon[92422]: osdmap e1225: 12 total, 12 up, 12 in 2023-03-09T23:07:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29356.403579 secs 2023-03-09T23:07:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:27 smithi122 ceph-mon[93869]: pgmap v16168: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 430 KiB/s wr, 12 op/s 2023-03-09T23:07:27.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:27 smithi181 ceph-mon[94189]: pgmap v16168: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 430 KiB/s wr, 12 op/s 2023-03-09T23:07:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:27 smithi038 ceph-mon[92422]: pgmap v16168: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 430 KiB/s wr, 12 op/s 2023-03-09T23:07:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:28 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:07:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:28 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:07:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29358.796361 secs 2023-03-09T23:07:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:28 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:07:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:28 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:07:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29358.796361 secs 2023-03-09T23:07:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:28 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:07:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:28 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:07:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29358.796361 secs 2023-03-09T23:07:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29820.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:07:29.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:29 smithi122 ceph-mon[93869]: pgmap v16169: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 483 KiB/s rd, 506 KiB/s wr, 12 op/s 2023-03-09T23:07:29.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:29 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:07:29.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:29 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:07:29.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:29 smithi181 ceph-mon[94189]: pgmap v16169: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 483 KiB/s rd, 506 KiB/s wr, 12 op/s 2023-03-09T23:07:29.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:29 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:07:29.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:29 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:07:29.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:29 smithi038 ceph-mon[92422]: pgmap v16169: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 483 KiB/s rd, 506 KiB/s wr, 12 op/s 2023-03-09T23:07:29.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:29 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:07:29.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:29 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:07:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29361.403639 secs 2023-03-09T23:07:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29361.403639 secs 2023-03-09T23:07:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29361.403639 secs 2023-03-09T23:07:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:31 smithi122 ceph-mon[93869]: pgmap v16170: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 483 KiB/s rd, 506 KiB/s wr, 12 op/s 2023-03-09T23:07:31.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:31 smithi181 ceph-mon[94189]: pgmap v16170: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 483 KiB/s rd, 506 KiB/s wr, 12 op/s 2023-03-09T23:07:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:31 smithi038 ceph-mon[92422]: pgmap v16170: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 483 KiB/s rd, 506 KiB/s wr, 12 op/s 2023-03-09T23:07:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:32 smithi122 ceph-mon[93869]: pgmap v16171: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 556 KiB/s rd, 581 KiB/s wr, 13 op/s 2023-03-09T23:07:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:32 smithi122 ceph-mon[93869]: osdmap e1226: 12 total, 12 up, 12 in 2023-03-09T23:07:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29363.796476 secs 2023-03-09T23:07:33.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:32 smithi181 ceph-mon[94189]: pgmap v16171: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 556 KiB/s rd, 581 KiB/s wr, 13 op/s 2023-03-09T23:07:33.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:32 smithi181 ceph-mon[94189]: osdmap e1226: 12 total, 12 up, 12 in 2023-03-09T23:07:33.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29363.796476 secs 2023-03-09T23:07:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:32 smithi038 ceph-mon[92422]: pgmap v16171: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 556 KiB/s rd, 581 KiB/s wr, 13 op/s 2023-03-09T23:07:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:32 smithi038 ceph-mon[92422]: osdmap e1226: 12 total, 12 up, 12 in 2023-03-09T23:07:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29363.796476 secs 2023-03-09T23:07:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29825.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:07:35.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:34 smithi181 ceph-mon[94189]: pgmap v16173: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 603 KiB/s wr, 12 op/s 2023-03-09T23:07:35.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:34 smithi038 ceph-mon[92422]: pgmap v16173: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 603 KiB/s wr, 12 op/s 2023-03-09T23:07:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:34 smithi122 ceph-mon[93869]: pgmap v16173: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 603 KiB/s wr, 12 op/s 2023-03-09T23:07:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29366.403699 secs 2023-03-09T23:07:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29366.403699 secs 2023-03-09T23:07:36.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29366.403699 secs 2023-03-09T23:07:37.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:36 smithi038 ceph-mon[92422]: pgmap v16174: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 476 KiB/s rd, 489 KiB/s wr, 9 op/s 2023-03-09T23:07:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:36 smithi122 ceph-mon[93869]: pgmap v16174: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 476 KiB/s rd, 489 KiB/s wr, 9 op/s 2023-03-09T23:07:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:36 smithi181 ceph-mon[94189]: pgmap v16174: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 476 KiB/s rd, 489 KiB/s wr, 9 op/s 2023-03-09T23:07:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29368.796611 secs 2023-03-09T23:07:38.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29368.796611 secs 2023-03-09T23:07:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29368.796611 secs 2023-03-09T23:07:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29830.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:07:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:39 smithi122 ceph-mon[93869]: pgmap v16175: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 288 KiB/s wr, 8 op/s 2023-03-09T23:07:39.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:39 smithi181 ceph-mon[94189]: pgmap v16175: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 288 KiB/s wr, 8 op/s 2023-03-09T23:07:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:39 smithi038 ceph-mon[92422]: pgmap v16175: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 288 KiB/s wr, 8 op/s 2023-03-09T23:07:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:07:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:07:40.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:07:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:07:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:07:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:07:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:41 smithi122 ceph-mon[93869]: pgmap v16176: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 439 KiB/s wr, 9 op/s 2023-03-09T23:07:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29371.403785 secs 2023-03-09T23:07:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:07:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:07:41.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:41 smithi181 ceph-mon[94189]: pgmap v16176: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 439 KiB/s wr, 9 op/s 2023-03-09T23:07:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29371.403785 secs 2023-03-09T23:07:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:07:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:07:41.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:41 smithi038 ceph-mon[92422]: pgmap v16176: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 439 KiB/s wr, 9 op/s 2023-03-09T23:07:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29371.403785 secs 2023-03-09T23:07:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:07:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:07:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:43 smithi122 ceph-mon[93869]: pgmap v16177: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 282 KiB/s wr, 8 op/s 2023-03-09T23:07:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29373.796679 secs 2023-03-09T23:07:43.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:43 smithi181 ceph-mon[94189]: pgmap v16177: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 282 KiB/s wr, 8 op/s 2023-03-09T23:07:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29373.796679 secs 2023-03-09T23:07:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:43 smithi038 ceph-mon[92422]: pgmap v16177: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 282 KiB/s wr, 8 op/s 2023-03-09T23:07:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29373.796679 secs 2023-03-09T23:07:43.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29835.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:07:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:45 smithi122 ceph-mon[93869]: pgmap v16178: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 305 KiB/s wr, 8 op/s 2023-03-09T23:07:45.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:45 smithi181 ceph-mon[94189]: pgmap v16178: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 305 KiB/s wr, 8 op/s 2023-03-09T23:07:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:45 smithi038 ceph-mon[92422]: pgmap v16178: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 305 KiB/s wr, 8 op/s 2023-03-09T23:07:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29376.403846 secs 2023-03-09T23:07:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29376.403846 secs 2023-03-09T23:07:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29376.403846 secs 2023-03-09T23:07:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:47 smithi122 ceph-mon[93869]: pgmap v16179: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 269 KiB/s wr, 9 op/s 2023-03-09T23:07:47.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:47 smithi181 ceph-mon[94189]: pgmap v16179: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 269 KiB/s wr, 9 op/s 2023-03-09T23:07:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:47 smithi038 ceph-mon[92422]: pgmap v16179: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 269 KiB/s wr, 9 op/s 2023-03-09T23:07:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29378.796801 secs 2023-03-09T23:07:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29378.796801 secs 2023-03-09T23:07:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29378.796801 secs 2023-03-09T23:07:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29840.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:07:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:49 smithi122 ceph-mon[93869]: pgmap v16180: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 307 KiB/s wr, 9 op/s 2023-03-09T23:07:49.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:49 smithi181 ceph-mon[94189]: pgmap v16180: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 307 KiB/s wr, 9 op/s 2023-03-09T23:07:49.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:49 smithi038 ceph-mon[92422]: pgmap v16180: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 307 KiB/s wr, 9 op/s 2023-03-09T23:07:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:51 smithi122 ceph-mon[93869]: pgmap v16181: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 307 KiB/s wr, 8 op/s 2023-03-09T23:07:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29381.403903 secs 2023-03-09T23:07:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:51 smithi181 ceph-mon[94189]: pgmap v16181: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 307 KiB/s wr, 8 op/s 2023-03-09T23:07:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29381.403903 secs 2023-03-09T23:07:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:51 smithi038 ceph-mon[92422]: pgmap v16181: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 307 KiB/s wr, 8 op/s 2023-03-09T23:07:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29381.403903 secs 2023-03-09T23:07:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:53 smithi122 ceph-mon[93869]: pgmap v16182: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 604 KiB/s rd, 402 KiB/s wr, 10 op/s 2023-03-09T23:07:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29383.796919 secs 2023-03-09T23:07:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:07:53.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:53 smithi181 ceph-mon[94189]: pgmap v16182: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 604 KiB/s rd, 402 KiB/s wr, 10 op/s 2023-03-09T23:07:53.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29383.796919 secs 2023-03-09T23:07:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:07:53.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:53 smithi038 ceph-mon[92422]: pgmap v16182: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 604 KiB/s rd, 402 KiB/s wr, 10 op/s 2023-03-09T23:07:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29383.796919 secs 2023-03-09T23:07:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:07:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29845.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:07:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:54 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:07:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:54 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:07:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:54 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:07:54.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:54 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:07:54.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:54 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:07:54.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:54 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:07:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:54 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:07:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:54 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:07:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:54 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:07:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:55 smithi122 ceph-mon[93869]: pgmap v16183: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 674 KiB/s rd, 425 KiB/s wr, 9 op/s 2023-03-09T23:07:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29386.403963 secs 2023-03-09T23:07:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:07:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:07:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:55 smithi181 ceph-mon[94189]: pgmap v16183: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 674 KiB/s rd, 425 KiB/s wr, 9 op/s 2023-03-09T23:07:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29386.403963 secs 2023-03-09T23:07:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:07:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:07:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:55 smithi038 ceph-mon[92422]: pgmap v16183: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 674 KiB/s rd, 425 KiB/s wr, 9 op/s 2023-03-09T23:07:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29386.403963 secs 2023-03-09T23:07:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:07:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:07:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:57 smithi122 ceph-mon[93869]: pgmap v16184: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 668 KiB/s rd, 477 KiB/s wr, 11 op/s 2023-03-09T23:07:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:07:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:07:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:57 smithi181 ceph-mon[94189]: pgmap v16184: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 668 KiB/s rd, 477 KiB/s wr, 11 op/s 2023-03-09T23:07:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:07:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:07:57.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:57 smithi038 ceph-mon[92422]: pgmap v16184: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 668 KiB/s rd, 477 KiB/s wr, 11 op/s 2023-03-09T23:07:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:07:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:07:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29850.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:07:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:59 smithi122 ceph-mon[93869]: pgmap v16185: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 690 KiB/s rd, 500 KiB/s wr, 10 op/s 2023-03-09T23:07:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:07:59.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:07:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:07:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:59 smithi181 ceph-mon[94189]: pgmap v16185: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 690 KiB/s rd, 500 KiB/s wr, 10 op/s 2023-03-09T23:07:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:07:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:07:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:07:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:59 smithi038 ceph-mon[92422]: pgmap v16185: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 690 KiB/s rd, 500 KiB/s wr, 10 op/s 2023-03-09T23:08:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:08:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:07:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:08:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:08:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:08:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:00 smithi038 ceph-mon[92422]: pgmap v16186: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 758 KiB/s rd, 629 KiB/s wr, 9 op/s 2023-03-09T23:08:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29391.403982 secs 2023-03-09T23:08:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:08:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:08:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:00 smithi122 ceph-mon[93869]: pgmap v16186: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 758 KiB/s rd, 629 KiB/s wr, 9 op/s 2023-03-09T23:08:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29391.403982 secs 2023-03-09T23:08:01.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:08:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:08:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:00 smithi181 ceph-mon[94189]: pgmap v16186: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 758 KiB/s rd, 629 KiB/s wr, 9 op/s 2023-03-09T23:08:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29391.403982 secs 2023-03-09T23:08:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:01 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:08:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:01 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:08:02.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:01 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:08:02.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:02 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:08:02.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:02 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:08:03.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:02 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:08:03.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:02 smithi038 ceph-mon[92422]: pgmap v16187: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 949 KiB/s rd, 697 KiB/s wr, 14 op/s 2023-03-09T23:08:03.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:02 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:08:03.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:02 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:08:03.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:02 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:08:03.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:02 smithi122 ceph-mon[93869]: pgmap v16187: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 949 KiB/s rd, 697 KiB/s wr, 14 op/s 2023-03-09T23:08:03.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:02 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:08:03.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:02 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:08:03.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:02 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:08:03.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:02 smithi181 ceph-mon[94189]: pgmap v16187: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 949 KiB/s rd, 697 KiB/s wr, 14 op/s 2023-03-09T23:08:03.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29855.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:08:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:05 smithi122 ceph-mon[93869]: pgmap v16188: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 757 KiB/s rd, 510 KiB/s wr, 24 op/s 2023-03-09T23:08:05.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:05 smithi181 ceph-mon[94189]: pgmap v16188: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 757 KiB/s rd, 510 KiB/s wr, 24 op/s 2023-03-09T23:08:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:05 smithi038 ceph-mon[92422]: pgmap v16188: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 757 KiB/s rd, 510 KiB/s wr, 24 op/s 2023-03-09T23:08:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29396.404043 secs 2023-03-09T23:08:06.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29396.404043 secs 2023-03-09T23:08:06.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29396.404043 secs 2023-03-09T23:08:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:07 smithi122 ceph-mon[93869]: pgmap v16189: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 967 KiB/s rd, 767 KiB/s wr, 26 op/s 2023-03-09T23:08:07.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:07 smithi181 ceph-mon[94189]: pgmap v16189: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 967 KiB/s rd, 767 KiB/s wr, 26 op/s 2023-03-09T23:08:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:07 smithi038 ceph-mon[92422]: pgmap v16189: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 967 KiB/s rd, 767 KiB/s wr, 26 op/s 2023-03-09T23:08:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:08:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:08:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:08:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:08:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:08:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:08:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29860.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:08:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:09 smithi122 ceph-mon[93869]: pgmap v16190: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 777 KiB/s rd, 646 KiB/s wr, 25 op/s 2023-03-09T23:08:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:08:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:08:09.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:09 smithi181 ceph-mon[94189]: pgmap v16190: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 777 KiB/s rd, 646 KiB/s wr, 25 op/s 2023-03-09T23:08:09.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:08:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:08:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:09 smithi038 ceph-mon[92422]: pgmap v16190: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 777 KiB/s rd, 646 KiB/s wr, 25 op/s 2023-03-09T23:08:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:08:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:08:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:11 smithi122 ceph-mon[93869]: pgmap v16191: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 754 KiB/s rd, 623 KiB/s wr, 23 op/s 2023-03-09T23:08:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29401.404103 secs 2023-03-09T23:08:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:11 smithi181 ceph-mon[94189]: pgmap v16191: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 754 KiB/s rd, 623 KiB/s wr, 23 op/s 2023-03-09T23:08:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29401.404103 secs 2023-03-09T23:08:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:11 smithi038 ceph-mon[92422]: pgmap v16191: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 754 KiB/s rd, 623 KiB/s wr, 23 op/s 2023-03-09T23:08:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29401.404103 secs 2023-03-09T23:08:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:13 smithi122 ceph-mon[93869]: pgmap v16192: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 660 KiB/s rd, 539 KiB/s wr, 25 op/s 2023-03-09T23:08:13.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:13 smithi181 ceph-mon[94189]: pgmap v16192: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 660 KiB/s rd, 539 KiB/s wr, 25 op/s 2023-03-09T23:08:13.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:13 smithi038 ceph-mon[92422]: pgmap v16192: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 660 KiB/s rd, 539 KiB/s wr, 25 op/s 2023-03-09T23:08:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29865.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:08:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:15 smithi122 ceph-mon[93869]: pgmap v16193: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 470 KiB/s wr, 21 op/s 2023-03-09T23:08:15.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:15 smithi181 ceph-mon[94189]: pgmap v16193: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 470 KiB/s wr, 21 op/s 2023-03-09T23:08:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:15 smithi038 ceph-mon[92422]: pgmap v16193: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 470 KiB/s wr, 21 op/s 2023-03-09T23:08:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29406.404162 secs 2023-03-09T23:08:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:08:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:08:16.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29406.404162 secs 2023-03-09T23:08:16.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:08:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:08:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29406.404162 secs 2023-03-09T23:08:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:08:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:08:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:17 smithi122 ceph-mon[93869]: pgmap v16194: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 436 KiB/s wr, 11 op/s 2023-03-09T23:08:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:08:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:08:17.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:17 smithi181 ceph-mon[94189]: pgmap v16194: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 436 KiB/s wr, 11 op/s 2023-03-09T23:08:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:08:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:08:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:17 smithi038 ceph-mon[92422]: pgmap v16194: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 436 KiB/s wr, 11 op/s 2023-03-09T23:08:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:08:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:08:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29870.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:08:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:19 smithi122 ceph-mon[93869]: pgmap v16195: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 191 KiB/s wr, 10 op/s 2023-03-09T23:08:19.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:19 smithi181 ceph-mon[94189]: pgmap v16195: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 191 KiB/s wr, 10 op/s 2023-03-09T23:08:19.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:19 smithi038 ceph-mon[92422]: pgmap v16195: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 191 KiB/s wr, 10 op/s 2023-03-09T23:08:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:21 smithi122 ceph-mon[93869]: pgmap v16196: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 191 KiB/s wr, 8 op/s 2023-03-09T23:08:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29411.404224 secs 2023-03-09T23:08:21.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:21 smithi181 ceph-mon[94189]: pgmap v16196: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 191 KiB/s wr, 8 op/s 2023-03-09T23:08:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29411.404224 secs 2023-03-09T23:08:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:21 smithi038 ceph-mon[92422]: pgmap v16196: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 191 KiB/s wr, 8 op/s 2023-03-09T23:08:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29411.404224 secs 2023-03-09T23:08:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:23 smithi122 ceph-mon[93869]: pgmap v16197: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 529 KiB/s wr, 13 op/s 2023-03-09T23:08:23.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:23 smithi181 ceph-mon[94189]: pgmap v16197: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 529 KiB/s wr, 13 op/s 2023-03-09T23:08:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:23 smithi038 ceph-mon[92422]: pgmap v16197: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 529 KiB/s wr, 13 op/s 2023-03-09T23:08:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29875.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:08:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:25 smithi122 ceph-mon[93869]: pgmap v16198: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 373 KiB/s wr, 11 op/s 2023-03-09T23:08:25.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:25 smithi181 ceph-mon[94189]: pgmap v16198: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 373 KiB/s wr, 11 op/s 2023-03-09T23:08:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:25 smithi038 ceph-mon[92422]: pgmap v16198: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 373 KiB/s wr, 11 op/s 2023-03-09T23:08:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29416.404284 secs 2023-03-09T23:08:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:26 smithi122 ceph-mon[93869]: osdmap e1227: 12 total, 12 up, 12 in 2023-03-09T23:08:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29416.404284 secs 2023-03-09T23:08:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:26 smithi181 ceph-mon[94189]: osdmap e1227: 12 total, 12 up, 12 in 2023-03-09T23:08:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29416.404284 secs 2023-03-09T23:08:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:26 smithi038 ceph-mon[92422]: osdmap e1227: 12 total, 12 up, 12 in 2023-03-09T23:08:27.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:27 smithi122 ceph-mon[93869]: pgmap v16200: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 677 KiB/s rd, 700 KiB/s wr, 13 op/s 2023-03-09T23:08:27.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:27 smithi181 ceph-mon[94189]: pgmap v16200: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 677 KiB/s rd, 700 KiB/s wr, 13 op/s 2023-03-09T23:08:27.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:27 smithi038 ceph-mon[92422]: pgmap v16200: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 677 KiB/s rd, 700 KiB/s wr, 13 op/s 2023-03-09T23:08:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29880.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:08:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:29 smithi122 ceph-mon[93869]: pgmap v16201: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 676 KiB/s rd, 687 KiB/s wr, 12 op/s 2023-03-09T23:08:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:29 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:08:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:29 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:08:29.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:29 smithi181 ceph-mon[94189]: pgmap v16201: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 676 KiB/s rd, 687 KiB/s wr, 12 op/s 2023-03-09T23:08:29.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:29 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:08:29.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:29 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:08:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:29 smithi038 ceph-mon[92422]: pgmap v16201: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 676 KiB/s rd, 687 KiB/s wr, 12 op/s 2023-03-09T23:08:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:29 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:08:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:29 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:08:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:31 smithi122 ceph-mon[93869]: pgmap v16202: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 676 KiB/s rd, 687 KiB/s wr, 12 op/s 2023-03-09T23:08:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29421.404344 secs 2023-03-09T23:08:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:31 smithi181 ceph-mon[94189]: pgmap v16202: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 676 KiB/s rd, 687 KiB/s wr, 12 op/s 2023-03-09T23:08:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29421.404344 secs 2023-03-09T23:08:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:31 smithi038 ceph-mon[92422]: pgmap v16202: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 676 KiB/s rd, 687 KiB/s wr, 12 op/s 2023-03-09T23:08:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29421.404344 secs 2023-03-09T23:08:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:32 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:08:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:32 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:08:32.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:32 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:08:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:32 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:08:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:32 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:08:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:32 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:08:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:33 smithi122 ceph-mon[93869]: pgmap v16203: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 455 KiB/s rd, 401 KiB/s wr, 10 op/s 2023-03-09T23:08:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29423.797951 secs 2023-03-09T23:08:33.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:33 smithi181 ceph-mon[94189]: pgmap v16203: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 455 KiB/s rd, 401 KiB/s wr, 10 op/s 2023-03-09T23:08:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29423.797951 secs 2023-03-09T23:08:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:33 smithi038 ceph-mon[92422]: pgmap v16203: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 455 KiB/s rd, 401 KiB/s wr, 10 op/s 2023-03-09T23:08:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29423.797951 secs 2023-03-09T23:08:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29885.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:08:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:34 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:08:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:34 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:08:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:34 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:08:34.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:34 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:08:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:34 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:08:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:34 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:08:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:34 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:08:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:34 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:08:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:34 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:08:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:35 smithi122 ceph-mon[93869]: pgmap v16204: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 947 KiB/s rd, 893 KiB/s wr, 11 op/s 2023-03-09T23:08:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29426.404401 secs 2023-03-09T23:08:35.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:35 smithi181 ceph-mon[94189]: pgmap v16204: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 947 KiB/s rd, 893 KiB/s wr, 11 op/s 2023-03-09T23:08:35.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29426.404401 secs 2023-03-09T23:08:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:35 smithi038 ceph-mon[92422]: pgmap v16204: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 947 KiB/s rd, 893 KiB/s wr, 11 op/s 2023-03-09T23:08:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29426.404401 secs 2023-03-09T23:08:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:08:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:08:36.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:08:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:08:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:08:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:08:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:37 smithi122 ceph-mon[93869]: pgmap v16205: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 873 KiB/s rd, 822 KiB/s wr, 13 op/s 2023-03-09T23:08:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:08:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:08:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:37 smithi122 ceph-mon[93869]: osdmap e1228: 12 total, 12 up, 12 in 2023-03-09T23:08:37.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:37 smithi181 ceph-mon[94189]: pgmap v16205: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 873 KiB/s rd, 822 KiB/s wr, 13 op/s 2023-03-09T23:08:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:08:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:08:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:37 smithi181 ceph-mon[94189]: osdmap e1228: 12 total, 12 up, 12 in 2023-03-09T23:08:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:37 smithi038 ceph-mon[92422]: pgmap v16205: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 873 KiB/s rd, 822 KiB/s wr, 13 op/s 2023-03-09T23:08:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:08:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:08:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:37 smithi038 ceph-mon[92422]: osdmap e1228: 12 total, 12 up, 12 in 2023-03-09T23:08:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29428.798074 secs 2023-03-09T23:08:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29428.798074 secs 2023-03-09T23:08:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29428.798074 secs 2023-03-09T23:08:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29890.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:08:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:39 smithi038 ceph-mon[92422]: pgmap v16207: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 667 KiB/s rd, 614 KiB/s wr, 11 op/s 2023-03-09T23:08:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:39 smithi122 ceph-mon[93869]: pgmap v16207: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 667 KiB/s rd, 614 KiB/s wr, 11 op/s 2023-03-09T23:08:39.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:39 smithi181 ceph-mon[94189]: pgmap v16207: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 667 KiB/s rd, 614 KiB/s wr, 11 op/s 2023-03-09T23:08:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:08:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:08:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29431.404461 secs 2023-03-09T23:08:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:08:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:08:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29431.404461 secs 2023-03-09T23:08:40.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:08:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:08:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29431.404461 secs 2023-03-09T23:08:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:41 smithi122 ceph-mon[93869]: pgmap v16208: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 950 KiB/s rd, 896 KiB/s wr, 12 op/s 2023-03-09T23:08:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:08:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:08:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:41 smithi181 ceph-mon[94189]: pgmap v16208: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 950 KiB/s rd, 896 KiB/s wr, 12 op/s 2023-03-09T23:08:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:08:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:08:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:41 smithi038 ceph-mon[92422]: pgmap v16208: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 950 KiB/s rd, 896 KiB/s wr, 12 op/s 2023-03-09T23:08:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:08:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:08:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:43 smithi122 ceph-mon[93869]: pgmap v16209: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 907 KiB/s rd, 905 KiB/s wr, 11 op/s 2023-03-09T23:08:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:43 smithi181 ceph-mon[94189]: pgmap v16209: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 907 KiB/s rd, 905 KiB/s wr, 11 op/s 2023-03-09T23:08:43.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:43 smithi038 ceph-mon[92422]: pgmap v16209: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 907 KiB/s rd, 905 KiB/s wr, 11 op/s 2023-03-09T23:08:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29895.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:08:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:45 smithi122 ceph-mon[93869]: pgmap v16210: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 485 KiB/s wr, 10 op/s 2023-03-09T23:08:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29436.404520 secs 2023-03-09T23:08:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:45 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:08:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:45 smithi181 ceph-mon[94189]: pgmap v16210: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 485 KiB/s wr, 10 op/s 2023-03-09T23:08:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29436.404520 secs 2023-03-09T23:08:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:45 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:08:45.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:45 smithi038 ceph-mon[92422]: pgmap v16210: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 485 KiB/s wr, 10 op/s 2023-03-09T23:08:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29436.404520 secs 2023-03-09T23:08:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:45 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:08:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:46 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:08:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:46 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:08:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:46 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:08:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:46 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:08:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:46 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:08:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:46 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:08:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:46 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:08:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:46 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:08:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:46 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:08:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:47 smithi122 ceph-mon[93869]: pgmap v16211: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 773 KiB/s rd, 769 KiB/s wr, 12 op/s 2023-03-09T23:08:47.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:47 smithi181 ceph-mon[94189]: pgmap v16211: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 773 KiB/s rd, 769 KiB/s wr, 12 op/s 2023-03-09T23:08:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:47 smithi038 ceph-mon[92422]: pgmap v16211: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 773 KiB/s rd, 769 KiB/s wr, 12 op/s 2023-03-09T23:08:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:08:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:08:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:48 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 29438.798287 secs 2023-03-09T23:08:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:08:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:08:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:48 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 29438.798287 secs 2023-03-09T23:08:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:08:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:08:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:48 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 29438.798287 secs 2023-03-09T23:08:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29900.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:08:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:49 smithi122 ceph-mon[93869]: pgmap v16212: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 745 KiB/s rd, 741 KiB/s wr, 12 op/s 2023-03-09T23:08:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:08:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:08:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:49 smithi181 ceph-mon[94189]: pgmap v16212: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 745 KiB/s rd, 741 KiB/s wr, 12 op/s 2023-03-09T23:08:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:08:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:08:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:49 smithi038 ceph-mon[92422]: pgmap v16212: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 745 KiB/s rd, 741 KiB/s wr, 12 op/s 2023-03-09T23:08:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:08:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:08:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:50 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:08:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:50 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:08:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29441.404579 secs 2023-03-09T23:08:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:50 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:08:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:50 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:08:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29441.404579 secs 2023-03-09T23:08:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:50 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:08:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:50 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:08:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29441.404579 secs 2023-03-09T23:08:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:51 smithi122 ceph-mon[93869]: pgmap v16213: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 692 KiB/s rd, 677 KiB/s wr, 11 op/s 2023-03-09T23:08:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:51 smithi181 ceph-mon[94189]: pgmap v16213: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 692 KiB/s rd, 677 KiB/s wr, 11 op/s 2023-03-09T23:08:51.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:51 smithi038 ceph-mon[92422]: pgmap v16213: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 692 KiB/s rd, 677 KiB/s wr, 11 op/s 2023-03-09T23:08:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:08:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:08:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:52 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:08:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:08:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:08:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:52 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:08:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:08:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:08:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:52 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:08:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:53 smithi122 ceph-mon[93869]: pgmap v16214: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 442 KiB/s wr, 13 op/s 2023-03-09T23:08:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:53 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 29443.798352 secs 2023-03-09T23:08:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:08:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:08:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:53 smithi181 ceph-mon[94189]: pgmap v16214: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 442 KiB/s wr, 13 op/s 2023-03-09T23:08:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:53 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 29443.798352 secs 2023-03-09T23:08:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:08:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:08:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:53 smithi038 ceph-mon[92422]: pgmap v16214: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 442 KiB/s wr, 13 op/s 2023-03-09T23:08:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:53 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 29443.798352 secs 2023-03-09T23:08:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:08:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:08:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29905.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:08:54.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:54 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:08:54.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:54 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:08:54.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:54 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:08:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:55 smithi122 ceph-mon[93869]: pgmap v16215: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 724 KiB/s rd, 710 KiB/s wr, 12 op/s 2023-03-09T23:08:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:55 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:08:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:55 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:08:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:55 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:08:55.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29446.404637 secs 2023-03-09T23:08:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:55 smithi181 ceph-mon[94189]: pgmap v16215: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 724 KiB/s rd, 710 KiB/s wr, 12 op/s 2023-03-09T23:08:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:55 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:08:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:55 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:08:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:55 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:08:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29446.404637 secs 2023-03-09T23:08:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:55 smithi038 ceph-mon[92422]: pgmap v16215: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 724 KiB/s rd, 710 KiB/s wr, 12 op/s 2023-03-09T23:08:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:55 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:08:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:55 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:08:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:55 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:08:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29446.404637 secs 2023-03-09T23:08:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:57 smithi122 ceph-mon[93869]: pgmap v16216: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 802 KiB/s rd, 787 KiB/s wr, 14 op/s 2023-03-09T23:08:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:08:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:08:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:57 smithi181 ceph-mon[94189]: pgmap v16216: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 802 KiB/s rd, 787 KiB/s wr, 14 op/s 2023-03-09T23:08:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:08:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:08:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:57 smithi038 ceph-mon[92422]: pgmap v16216: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 802 KiB/s rd, 787 KiB/s wr, 14 op/s 2023-03-09T23:08:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:08:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:08:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:58 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 29448.798470 secs 2023-03-09T23:08:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:58 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 29448.798470 secs 2023-03-09T23:08:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:58 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 29448.798470 secs 2023-03-09T23:08:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29910.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:08:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:08:59 smithi122 ceph-mon[93869]: pgmap v16217: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 550 KiB/s wr, 12 op/s 2023-03-09T23:08:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:08:59 smithi181 ceph-mon[94189]: pgmap v16217: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 550 KiB/s wr, 12 op/s 2023-03-09T23:08:59.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:08:59 smithi038 ceph-mon[92422]: pgmap v16217: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 550 KiB/s wr, 12 op/s 2023-03-09T23:09:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:09:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:09:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29451.404700 secs 2023-03-09T23:09:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:09:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:09:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29451.404700 secs 2023-03-09T23:09:00.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:09:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:09:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29451.404700 secs 2023-03-09T23:09:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:01 smithi122 ceph-mon[93869]: pgmap v16218: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 526 KiB/s rd, 513 KiB/s wr, 10 op/s 2023-03-09T23:09:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:01 smithi181 ceph-mon[94189]: pgmap v16218: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 526 KiB/s rd, 513 KiB/s wr, 10 op/s 2023-03-09T23:09:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:01 smithi038 ceph-mon[92422]: pgmap v16218: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 526 KiB/s rd, 513 KiB/s wr, 10 op/s 2023-03-09T23:09:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:03 smithi122 ceph-mon[93869]: pgmap v16219: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 828 KiB/s rd, 836 KiB/s wr, 15 op/s 2023-03-09T23:09:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29453.798588 secs 2023-03-09T23:09:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:03 smithi181 ceph-mon[94189]: pgmap v16219: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 828 KiB/s rd, 836 KiB/s wr, 15 op/s 2023-03-09T23:09:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29453.798588 secs 2023-03-09T23:09:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:03 smithi038 ceph-mon[92422]: pgmap v16219: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 828 KiB/s rd, 836 KiB/s wr, 15 op/s 2023-03-09T23:09:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29453.798588 secs 2023-03-09T23:09:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29915.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:09:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:05 smithi122 ceph-mon[93869]: pgmap v16220: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 859 KiB/s rd, 882 KiB/s wr, 26 op/s 2023-03-09T23:09:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29456.404783 secs 2023-03-09T23:09:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:05 smithi181 ceph-mon[94189]: pgmap v16220: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 859 KiB/s rd, 882 KiB/s wr, 26 op/s 2023-03-09T23:09:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29456.404783 secs 2023-03-09T23:09:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:05 smithi038 ceph-mon[92422]: pgmap v16220: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 859 KiB/s rd, 882 KiB/s wr, 26 op/s 2023-03-09T23:09:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29456.404783 secs 2023-03-09T23:09:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:06 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:09:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:09:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:06 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:09:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:09:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:06 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:09:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:09:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:07 smithi122 ceph-mon[93869]: pgmap v16221: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 486 KiB/s rd, 508 KiB/s wr, 26 op/s 2023-03-09T23:09:07.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:07 smithi181 ceph-mon[94189]: pgmap v16221: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 486 KiB/s rd, 508 KiB/s wr, 26 op/s 2023-03-09T23:09:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:07 smithi038 ceph-mon[92422]: pgmap v16221: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 486 KiB/s rd, 508 KiB/s wr, 26 op/s 2023-03-09T23:09:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29458.798721 secs 2023-03-09T23:09:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:09:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:09:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29458.798721 secs 2023-03-09T23:09:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:09:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:09:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29458.798721 secs 2023-03-09T23:09:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:09:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:09:08.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29920.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:09:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:09 smithi122 ceph-mon[93869]: pgmap v16222: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 460 KiB/s wr, 26 op/s 2023-03-09T23:09:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:09:09.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:09:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:09 smithi181 ceph-mon[94189]: pgmap v16222: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 460 KiB/s wr, 26 op/s 2023-03-09T23:09:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:09:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:09:09.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:09 smithi038 ceph-mon[92422]: pgmap v16222: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 460 KiB/s wr, 26 op/s 2023-03-09T23:09:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:09:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:09:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:10 smithi122 ceph-mon[93869]: osdmap e1229: 12 total, 12 up, 12 in 2023-03-09T23:09:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29461.404869 secs 2023-03-09T23:09:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:10 smithi181 ceph-mon[94189]: osdmap e1229: 12 total, 12 up, 12 in 2023-03-09T23:09:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29461.404869 secs 2023-03-09T23:09:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:10 smithi038 ceph-mon[92422]: osdmap e1229: 12 total, 12 up, 12 in 2023-03-09T23:09:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29461.404869 secs 2023-03-09T23:09:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:11 smithi122 ceph-mon[93869]: pgmap v16224: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 525 KiB/s rd, 551 KiB/s wr, 29 op/s 2023-03-09T23:09:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:11 smithi181 ceph-mon[94189]: pgmap v16224: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 525 KiB/s rd, 551 KiB/s wr, 29 op/s 2023-03-09T23:09:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:11 smithi038 ceph-mon[92422]: pgmap v16224: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 525 KiB/s rd, 551 KiB/s wr, 29 op/s 2023-03-09T23:09:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:13 smithi122 ceph-mon[93869]: pgmap v16225: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 499 KiB/s rd, 513 KiB/s wr, 25 op/s 2023-03-09T23:09:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29463.798837 secs 2023-03-09T23:09:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:13 smithi181 ceph-mon[94189]: pgmap v16225: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 499 KiB/s rd, 513 KiB/s wr, 25 op/s 2023-03-09T23:09:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29463.798837 secs 2023-03-09T23:09:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:13 smithi038 ceph-mon[92422]: pgmap v16225: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 499 KiB/s rd, 513 KiB/s wr, 25 op/s 2023-03-09T23:09:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29463.798837 secs 2023-03-09T23:09:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29925.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:09:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:15 smithi122 ceph-mon[93869]: pgmap v16226: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 458 KiB/s wr, 9 op/s 2023-03-09T23:09:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29466.404928 secs 2023-03-09T23:09:15.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:15 smithi181 ceph-mon[94189]: pgmap v16226: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 458 KiB/s wr, 9 op/s 2023-03-09T23:09:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29466.404928 secs 2023-03-09T23:09:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:15 smithi038 ceph-mon[92422]: pgmap v16226: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 458 KiB/s wr, 9 op/s 2023-03-09T23:09:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29466.404928 secs 2023-03-09T23:09:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:09:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:09:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:09:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:09:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:09:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:09:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:17 smithi038 ceph-mon[92422]: pgmap v16227: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 458 KiB/s wr, 9 op/s 2023-03-09T23:09:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:09:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:09:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:17 smithi038 ceph-mon[92422]: osdmap e1230: 12 total, 12 up, 12 in 2023-03-09T23:09:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:17 smithi122 ceph-mon[93869]: pgmap v16227: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 458 KiB/s wr, 9 op/s 2023-03-09T23:09:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:09:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:09:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:17 smithi122 ceph-mon[93869]: osdmap e1230: 12 total, 12 up, 12 in 2023-03-09T23:09:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:17 smithi181 ceph-mon[94189]: pgmap v16227: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 458 KiB/s wr, 9 op/s 2023-03-09T23:09:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:09:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:09:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:17 smithi181 ceph-mon[94189]: osdmap e1230: 12 total, 12 up, 12 in 2023-03-09T23:09:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29468.798961 secs 2023-03-09T23:09:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29930.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:09:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29468.798961 secs 2023-03-09T23:09:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29468.798961 secs 2023-03-09T23:09:19.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:19 smithi038 ceph-mon[92422]: pgmap v16229: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 445 KiB/s rd, 443 KiB/s wr, 10 op/s 2023-03-09T23:09:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:19 smithi122 ceph-mon[93869]: pgmap v16229: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 445 KiB/s rd, 443 KiB/s wr, 10 op/s 2023-03-09T23:09:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:19 smithi181 ceph-mon[94189]: pgmap v16229: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 445 KiB/s rd, 443 KiB/s wr, 10 op/s 2023-03-09T23:09:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29471.404984 secs 2023-03-09T23:09:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29471.404984 secs 2023-03-09T23:09:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29471.404984 secs 2023-03-09T23:09:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:21 smithi122 ceph-mon[93869]: pgmap v16230: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 377 KiB/s wr, 8 op/s 2023-03-09T23:09:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:21 smithi181 ceph-mon[94189]: pgmap v16230: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 377 KiB/s wr, 8 op/s 2023-03-09T23:09:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:21 smithi038 ceph-mon[92422]: pgmap v16230: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 377 KiB/s wr, 8 op/s 2023-03-09T23:09:23.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29935.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:09:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:23 smithi122 ceph-mon[93869]: pgmap v16231: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 111 KiB/s wr, 8 op/s 2023-03-09T23:09:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29473.799078 secs 2023-03-09T23:09:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:23 smithi181 ceph-mon[94189]: pgmap v16231: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 111 KiB/s wr, 8 op/s 2023-03-09T23:09:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29473.799078 secs 2023-03-09T23:09:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:23 smithi038 ceph-mon[92422]: pgmap v16231: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 111 KiB/s wr, 8 op/s 2023-03-09T23:09:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29473.799078 secs 2023-03-09T23:09:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:25 smithi122 ceph-mon[93869]: pgmap v16232: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 139 KiB/s wr, 8 op/s 2023-03-09T23:09:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29476.405040 secs 2023-03-09T23:09:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:25 smithi181 ceph-mon[94189]: pgmap v16232: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 139 KiB/s wr, 8 op/s 2023-03-09T23:09:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29476.405040 secs 2023-03-09T23:09:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:25 smithi038 ceph-mon[92422]: pgmap v16232: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 139 KiB/s wr, 8 op/s 2023-03-09T23:09:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29476.405040 secs 2023-03-09T23:09:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:27 smithi122 ceph-mon[93869]: pgmap v16233: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 139 KiB/s wr, 8 op/s 2023-03-09T23:09:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:27 smithi181 ceph-mon[94189]: pgmap v16233: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 139 KiB/s wr, 8 op/s 2023-03-09T23:09:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:27 smithi038 ceph-mon[92422]: pgmap v16233: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 139 KiB/s wr, 8 op/s 2023-03-09T23:09:28.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29940.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:09:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29478.799170 secs 2023-03-09T23:09:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29478.799170 secs 2023-03-09T23:09:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29478.799170 secs 2023-03-09T23:09:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:29 smithi122 ceph-mon[93869]: pgmap v16234: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 127 KiB/s wr, 9 op/s 2023-03-09T23:09:29.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:29 smithi181 ceph-mon[94189]: pgmap v16234: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 127 KiB/s wr, 9 op/s 2023-03-09T23:09:29.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:29 smithi038 ceph-mon[92422]: pgmap v16234: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 127 KiB/s wr, 9 op/s 2023-03-09T23:09:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29481.405100 secs 2023-03-09T23:09:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29481.405100 secs 2023-03-09T23:09:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29481.405100 secs 2023-03-09T23:09:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:31 smithi122 ceph-mon[93869]: pgmap v16235: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 93 KiB/s wr, 7 op/s 2023-03-09T23:09:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:31 smithi181 ceph-mon[94189]: pgmap v16235: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 93 KiB/s wr, 7 op/s 2023-03-09T23:09:31.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:31 smithi038 ceph-mon[92422]: pgmap v16235: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 93 KiB/s wr, 7 op/s 2023-03-09T23:09:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29945.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:09:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:33 smithi122 ceph-mon[93869]: pgmap v16236: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 221 KiB/s wr, 10 op/s 2023-03-09T23:09:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29483.799236 secs 2023-03-09T23:09:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:33 smithi181 ceph-mon[94189]: pgmap v16236: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 221 KiB/s wr, 10 op/s 2023-03-09T23:09:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29483.799236 secs 2023-03-09T23:09:33.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:33 smithi038 ceph-mon[92422]: pgmap v16236: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 223 KiB/s rd, 221 KiB/s wr, 10 op/s 2023-03-09T23:09:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29483.799236 secs 2023-03-09T23:09:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:35 smithi122 ceph-mon[93869]: pgmap v16237: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 152 KiB/s wr, 7 op/s 2023-03-09T23:09:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29486.405160 secs 2023-03-09T23:09:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:35 smithi181 ceph-mon[94189]: pgmap v16237: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 152 KiB/s wr, 7 op/s 2023-03-09T23:09:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29486.405160 secs 2023-03-09T23:09:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:35 smithi038 ceph-mon[92422]: pgmap v16237: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 152 KiB/s wr, 7 op/s 2023-03-09T23:09:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29486.405160 secs 2023-03-09T23:09:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:37 smithi122 ceph-mon[93869]: pgmap v16238: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 161 KiB/s wr, 9 op/s 2023-03-09T23:09:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:37 smithi181 ceph-mon[94189]: pgmap v16238: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 161 KiB/s wr, 9 op/s 2023-03-09T23:09:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:37 smithi038 ceph-mon[92422]: pgmap v16238: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 161 KiB/s wr, 9 op/s 2023-03-09T23:09:38.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29950.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:09:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29488.799351 secs 2023-03-09T23:09:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29488.799351 secs 2023-03-09T23:09:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29488.799351 secs 2023-03-09T23:09:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:39 smithi122 ceph-mon[93869]: pgmap v16239: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 249 KiB/s wr, 7 op/s 2023-03-09T23:09:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:39 smithi181 ceph-mon[94189]: pgmap v16239: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 249 KiB/s wr, 7 op/s 2023-03-09T23:09:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:39 smithi038 ceph-mon[92422]: pgmap v16239: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 249 KiB/s wr, 7 op/s 2023-03-09T23:09:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29491.405221 secs 2023-03-09T23:09:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29491.405221 secs 2023-03-09T23:09:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29491.405221 secs 2023-03-09T23:09:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:41 smithi122 ceph-mon[93869]: pgmap v16240: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 248 KiB/s wr, 6 op/s 2023-03-09T23:09:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:41 smithi181 ceph-mon[94189]: pgmap v16240: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 248 KiB/s wr, 6 op/s 2023-03-09T23:09:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:41 smithi038 ceph-mon[92422]: pgmap v16240: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 248 KiB/s wr, 6 op/s 2023-03-09T23:09:43.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29955.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:09:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:43 smithi122 ceph-mon[93869]: pgmap v16241: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 248 KiB/s wr, 9 op/s 2023-03-09T23:09:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29493.799469 secs 2023-03-09T23:09:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:43 smithi181 ceph-mon[94189]: pgmap v16241: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 248 KiB/s wr, 9 op/s 2023-03-09T23:09:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29493.799469 secs 2023-03-09T23:09:43.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:43 smithi038 ceph-mon[92422]: pgmap v16241: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 248 KiB/s wr, 9 op/s 2023-03-09T23:09:43.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29493.799469 secs 2023-03-09T23:09:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:45 smithi122 ceph-mon[93869]: pgmap v16242: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 121 KiB/s wr, 7 op/s 2023-03-09T23:09:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29496.405289 secs 2023-03-09T23:09:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:45 smithi181 ceph-mon[94189]: pgmap v16242: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 121 KiB/s wr, 7 op/s 2023-03-09T23:09:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29496.405289 secs 2023-03-09T23:09:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:45 smithi038 ceph-mon[92422]: pgmap v16242: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 121 KiB/s wr, 7 op/s 2023-03-09T23:09:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29496.405289 secs 2023-03-09T23:09:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:47 smithi122 ceph-mon[93869]: pgmap v16243: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 171 KiB/s wr, 10 op/s 2023-03-09T23:09:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:09:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:09:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:47 smithi181 ceph-mon[94189]: pgmap v16243: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 171 KiB/s wr, 10 op/s 2023-03-09T23:09:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:09:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:09:47.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:47 smithi038 ceph-mon[92422]: pgmap v16243: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 171 KiB/s wr, 10 op/s 2023-03-09T23:09:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:09:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:09:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29960.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:09:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29498.799584 secs 2023-03-09T23:09:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29498.799584 secs 2023-03-09T23:09:48.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29498.799584 secs 2023-03-09T23:09:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:49 smithi122 ceph-mon[93869]: pgmap v16244: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 416 KiB/s wr, 9 op/s 2023-03-09T23:09:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:09:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:09:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:49 smithi181 ceph-mon[94189]: pgmap v16244: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 416 KiB/s wr, 9 op/s 2023-03-09T23:09:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:09:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:09:49.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:49 smithi038 ceph-mon[92422]: pgmap v16244: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 416 KiB/s wr, 9 op/s 2023-03-09T23:09:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:09:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:09:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29501.405350 secs 2023-03-09T23:09:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29501.405350 secs 2023-03-09T23:09:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29501.405350 secs 2023-03-09T23:09:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:51 smithi122 ceph-mon[93869]: pgmap v16245: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 363 KiB/s rd, 328 KiB/s wr, 8 op/s 2023-03-09T23:09:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:09:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:09:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:51 smithi181 ceph-mon[94189]: pgmap v16245: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 363 KiB/s rd, 328 KiB/s wr, 8 op/s 2023-03-09T23:09:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:09:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:09:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:51 smithi038 ceph-mon[92422]: pgmap v16245: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 363 KiB/s rd, 328 KiB/s wr, 8 op/s 2023-03-09T23:09:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:09:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:09:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:53 smithi122 ceph-mon[93869]: pgmap v16246: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 620 KiB/s wr, 11 op/s 2023-03-09T23:09:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29503.799726 secs 2023-03-09T23:09:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:53 smithi181 ceph-mon[94189]: pgmap v16246: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 620 KiB/s wr, 11 op/s 2023-03-09T23:09:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29503.799726 secs 2023-03-09T23:09:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:53 smithi038 ceph-mon[92422]: pgmap v16246: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 620 KiB/s wr, 11 op/s 2023-03-09T23:09:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29503.799726 secs 2023-03-09T23:09:53.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29965.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:09:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:55 smithi122 ceph-mon[93869]: pgmap v16247: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 915 KiB/s rd, 912 KiB/s wr, 11 op/s 2023-03-09T23:09:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:55 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:09:55.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:55 smithi181 ceph-mon[94189]: pgmap v16247: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 915 KiB/s rd, 912 KiB/s wr, 11 op/s 2023-03-09T23:09:55.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:55 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:09:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:55 smithi038 ceph-mon[92422]: pgmap v16247: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 915 KiB/s rd, 912 KiB/s wr, 11 op/s 2023-03-09T23:09:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:55 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:09:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29506.405411 secs 2023-03-09T23:09:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:09:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:09:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29506.405411 secs 2023-03-09T23:09:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:09:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:09:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29506.405411 secs 2023-03-09T23:09:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:09:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:09:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:57 smithi122 ceph-mon[93869]: pgmap v16248: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 936 KiB/s rd, 912 KiB/s wr, 13 op/s 2023-03-09T23:09:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:09:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:09:57.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:57 smithi181 ceph-mon[94189]: pgmap v16248: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 936 KiB/s rd, 912 KiB/s wr, 13 op/s 2023-03-09T23:09:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:09:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:09:57.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:57 smithi038 ceph-mon[92422]: pgmap v16248: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 936 KiB/s rd, 912 KiB/s wr, 13 op/s 2023-03-09T23:09:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:09:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:09:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29508.799853 secs 2023-03-09T23:09:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:09:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:09:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:09:58.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:09:58.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:09:58.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:09:58.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:09:58.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:09:58.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:09:58.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:09:58.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:09:58.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:09:58.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:09:58.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:09:58.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:09:58.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:09:58.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:09:58.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:09:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29508.799853 secs 2023-03-09T23:09:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:09:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:09:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:09:58.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:09:58.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:09:58.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:09:58.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:09:58.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:09:58.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:09:58.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:09:58.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:09:58.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:09:58.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:09:58.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:09:58.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:09:58.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:09:58.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:09:58.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:09:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29970.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:09:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29508.799853 secs 2023-03-09T23:09:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:09:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:09:58.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:09:58.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:09:58.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:09:58.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:09:58.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:09:58.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:09:58.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:09:58.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:09:58.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:09:58.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:09:58.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:09:58.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:09:58.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:09:58.980 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:09:58.980 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:09:58.980 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:09:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:59 smithi122 ceph-mon[93869]: pgmap v16249: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 885 KiB/s rd, 862 KiB/s wr, 11 op/s 2023-03-09T23:09:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:59 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T23:09:59.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:59 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T23:09:59.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:59 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T23:09:59.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:09:59 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T23:09:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:59 smithi181 ceph-mon[94189]: pgmap v16249: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 885 KiB/s rd, 862 KiB/s wr, 11 op/s 2023-03-09T23:09:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:59 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T23:09:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:59 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T23:09:59.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:59 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T23:09:59.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:09:59 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T23:09:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:59 smithi038 ceph-mon[92422]: pgmap v16249: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 885 KiB/s rd, 862 KiB/s wr, 11 op/s 2023-03-09T23:09:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:59 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-09T23:10:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:59 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-09T23:10:00.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:59 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T23:10:00.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:09:59 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T23:10:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29511.405469 secs 2023-03-09T23:10:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:00 smithi122 ceph-mon[93869]: Health detail: HEALTH_WARN 2 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-09T23:10:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:00 smithi122 ceph-mon[93869]: [WRN] MDS_SLOW_REQUEST: 2 MDSs report slow requests 2023-03-09T23:10:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:00 smithi122 ceph-mon[93869]: mds.i(mds.2): 2 slow requests are blocked > 180 secs 2023-03-09T23:10:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:00 smithi122 ceph-mon[93869]: mds.h(mds.1): 2 slow requests are blocked > 180 secs 2023-03-09T23:10:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:00 smithi122 ceph-mon[93869]: [WRN] MDS_TRIM: 2 MDSs behind on trimming 2023-03-09T23:10:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:00 smithi122 ceph-mon[93869]: mds.e(mds.0): Behind on trimming (529/128) max_segments: 128, num_segments: 529 2023-03-09T23:10:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:00 smithi122 ceph-mon[93869]: mds.b(mds.0): Behind on trimming (529/128) max_segments: 128, num_segments: 529 2023-03-09T23:10:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29511.405469 secs 2023-03-09T23:10:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:00 smithi181 ceph-mon[94189]: Health detail: HEALTH_WARN 2 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-09T23:10:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:00 smithi181 ceph-mon[94189]: [WRN] MDS_SLOW_REQUEST: 2 MDSs report slow requests 2023-03-09T23:10:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:00 smithi181 ceph-mon[94189]: mds.i(mds.2): 2 slow requests are blocked > 180 secs 2023-03-09T23:10:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:00 smithi181 ceph-mon[94189]: mds.h(mds.1): 2 slow requests are blocked > 180 secs 2023-03-09T23:10:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:00 smithi181 ceph-mon[94189]: [WRN] MDS_TRIM: 2 MDSs behind on trimming 2023-03-09T23:10:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:00 smithi181 ceph-mon[94189]: mds.e(mds.0): Behind on trimming (529/128) max_segments: 128, num_segments: 529 2023-03-09T23:10:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:00 smithi181 ceph-mon[94189]: mds.b(mds.0): Behind on trimming (529/128) max_segments: 128, num_segments: 529 2023-03-09T23:10:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29511.405469 secs 2023-03-09T23:10:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:00 smithi038 ceph-mon[92422]: Health detail: HEALTH_WARN 2 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-09T23:10:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:00 smithi038 ceph-mon[92422]: [WRN] MDS_SLOW_REQUEST: 2 MDSs report slow requests 2023-03-09T23:10:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:00 smithi038 ceph-mon[92422]: mds.i(mds.2): 2 slow requests are blocked > 180 secs 2023-03-09T23:10:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:00 smithi038 ceph-mon[92422]: mds.h(mds.1): 2 slow requests are blocked > 180 secs 2023-03-09T23:10:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:00 smithi038 ceph-mon[92422]: [WRN] MDS_TRIM: 2 MDSs behind on trimming 2023-03-09T23:10:01.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:00 smithi038 ceph-mon[92422]: mds.e(mds.0): Behind on trimming (529/128) max_segments: 128, num_segments: 529 2023-03-09T23:10:01.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:00 smithi038 ceph-mon[92422]: mds.b(mds.0): Behind on trimming (529/128) max_segments: 128, num_segments: 529 2023-03-09T23:10:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:01 smithi122 ceph-mon[93869]: pgmap v16250: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1013 KiB/s rd, 680 KiB/s wr, 10 op/s 2023-03-09T23:10:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:10:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:10:01.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:01 smithi181 ceph-mon[94189]: pgmap v16250: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1013 KiB/s rd, 680 KiB/s wr, 10 op/s 2023-03-09T23:10:01.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:10:01.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:10:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:01 smithi038 ceph-mon[92422]: pgmap v16250: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1013 KiB/s rd, 680 KiB/s wr, 10 op/s 2023-03-09T23:10:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:10:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:10:03.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29975.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:10:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:03 smithi122 ceph-mon[93869]: pgmap v16251: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1014 KiB/s rd, 681 KiB/s wr, 13 op/s 2023-03-09T23:10:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29513.799973 secs 2023-03-09T23:10:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:03 smithi181 ceph-mon[94189]: pgmap v16251: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1014 KiB/s rd, 681 KiB/s wr, 13 op/s 2023-03-09T23:10:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29513.799973 secs 2023-03-09T23:10:03.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:03 smithi038 ceph-mon[92422]: pgmap v16251: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1014 KiB/s rd, 681 KiB/s wr, 13 op/s 2023-03-09T23:10:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29513.799973 secs 2023-03-09T23:10:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:05 smithi122 ceph-mon[93869]: pgmap v16252: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 821 KiB/s rd, 413 KiB/s wr, 13 op/s 2023-03-09T23:10:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29516.405548 secs 2023-03-09T23:10:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:05 smithi181 ceph-mon[94189]: pgmap v16252: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 821 KiB/s rd, 413 KiB/s wr, 13 op/s 2023-03-09T23:10:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29516.405548 secs 2023-03-09T23:10:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:05 smithi038 ceph-mon[92422]: pgmap v16252: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 821 KiB/s rd, 413 KiB/s wr, 13 op/s 2023-03-09T23:10:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29516.405548 secs 2023-03-09T23:10:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:07 smithi122 ceph-mon[93869]: pgmap v16253: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 484 KiB/s wr, 15 op/s 2023-03-09T23:10:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:10:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:10:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:07 smithi181 ceph-mon[94189]: pgmap v16253: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 484 KiB/s wr, 15 op/s 2023-03-09T23:10:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:10:07.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:10:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:07 smithi038 ceph-mon[92422]: pgmap v16253: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 484 KiB/s wr, 15 op/s 2023-03-09T23:10:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:10:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:10:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29980.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:10:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29518.800091 secs 2023-03-09T23:10:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29518.800091 secs 2023-03-09T23:10:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29518.800091 secs 2023-03-09T23:10:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:09 smithi122 ceph-mon[93869]: pgmap v16254: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 528 KiB/s wr, 25 op/s 2023-03-09T23:10:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:09 smithi181 ceph-mon[94189]: pgmap v16254: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 528 KiB/s wr, 25 op/s 2023-03-09T23:10:09.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:09 smithi038 ceph-mon[92422]: pgmap v16254: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 528 KiB/s wr, 25 op/s 2023-03-09T23:10:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:10 smithi122 ceph-mon[93869]: osdmap e1231: 12 total, 12 up, 12 in 2023-03-09T23:10:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29521.405610 secs 2023-03-09T23:10:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:10 smithi181 ceph-mon[94189]: osdmap e1231: 12 total, 12 up, 12 in 2023-03-09T23:10:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29521.405610 secs 2023-03-09T23:10:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:10 smithi038 ceph-mon[92422]: osdmap e1231: 12 total, 12 up, 12 in 2023-03-09T23:10:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29521.405610 secs 2023-03-09T23:10:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:11 smithi122 ceph-mon[93869]: pgmap v16256: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 894 KiB/s wr, 30 op/s 2023-03-09T23:10:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:11 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:10:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:11 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:10:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:11 smithi181 ceph-mon[94189]: pgmap v16256: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 894 KiB/s wr, 30 op/s 2023-03-09T23:10:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:11 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:10:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:11 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:10:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:11 smithi038 ceph-mon[92422]: pgmap v16256: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 894 KiB/s wr, 30 op/s 2023-03-09T23:10:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:11 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:10:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:11 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:10:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:13 smithi122 ceph-mon[93869]: pgmap v16257: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1008 KiB/s wr, 30 op/s 2023-03-09T23:10:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29523.800217 secs 2023-03-09T23:10:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:10:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:10:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:13 smithi181 ceph-mon[94189]: pgmap v16257: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1008 KiB/s wr, 30 op/s 2023-03-09T23:10:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29523.800217 secs 2023-03-09T23:10:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:10:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:10:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:13 smithi038 ceph-mon[92422]: pgmap v16257: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1008 KiB/s wr, 30 op/s 2023-03-09T23:10:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29523.800217 secs 2023-03-09T23:10:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:10:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:10:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29985.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:10:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:15 smithi122 ceph-mon[93869]: pgmap v16258: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1008 KiB/s wr, 28 op/s 2023-03-09T23:10:15.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:15 smithi181 ceph-mon[94189]: pgmap v16258: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1008 KiB/s wr, 28 op/s 2023-03-09T23:10:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:15 smithi038 ceph-mon[92422]: pgmap v16258: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1008 KiB/s wr, 28 op/s 2023-03-09T23:10:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29526.405671 secs 2023-03-09T23:10:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:10:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:10:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29526.405671 secs 2023-03-09T23:10:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:10:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:10:16.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29526.405671 secs 2023-03-09T23:10:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:10:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:10:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:17 smithi122 ceph-mon[93869]: pgmap v16259: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 754 KiB/s rd, 712 KiB/s wr, 26 op/s 2023-03-09T23:10:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:10:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:10:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:17 smithi122 ceph-mon[93869]: osdmap e1232: 12 total, 12 up, 12 in 2023-03-09T23:10:17.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:17 smithi181 ceph-mon[94189]: pgmap v16259: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 754 KiB/s rd, 712 KiB/s wr, 26 op/s 2023-03-09T23:10:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:10:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:10:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:17 smithi181 ceph-mon[94189]: osdmap e1232: 12 total, 12 up, 12 in 2023-03-09T23:10:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:17 smithi038 ceph-mon[92422]: pgmap v16259: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 754 KiB/s rd, 712 KiB/s wr, 26 op/s 2023-03-09T23:10:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:10:17.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:10:17.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:17 smithi038 ceph-mon[92422]: osdmap e1232: 12 total, 12 up, 12 in 2023-03-09T23:10:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29528.800303 secs 2023-03-09T23:10:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29528.800303 secs 2023-03-09T23:10:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29528.800303 secs 2023-03-09T23:10:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29990.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:10:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:19 smithi122 ceph-mon[93869]: pgmap v16261: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 337 KiB/s wr, 12 op/s 2023-03-09T23:10:19.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:19 smithi181 ceph-mon[94189]: pgmap v16261: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 337 KiB/s wr, 12 op/s 2023-03-09T23:10:19.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:19 smithi038 ceph-mon[92422]: pgmap v16261: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 337 KiB/s wr, 12 op/s 2023-03-09T23:10:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29531.405755 secs 2023-03-09T23:10:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:21 smithi122 ceph-mon[93869]: pgmap v16262: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 282 KiB/s wr, 10 op/s 2023-03-09T23:10:21.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29531.405755 secs 2023-03-09T23:10:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:21 smithi181 ceph-mon[94189]: pgmap v16262: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 282 KiB/s wr, 10 op/s 2023-03-09T23:10:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29531.405755 secs 2023-03-09T23:10:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:21 smithi038 ceph-mon[92422]: pgmap v16262: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 282 KiB/s wr, 10 op/s 2023-03-09T23:10:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:23 smithi122 ceph-mon[93869]: pgmap v16263: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 168 KiB/s wr, 9 op/s 2023-03-09T23:10:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29533.800415 secs 2023-03-09T23:10:23.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:23 smithi181 ceph-mon[94189]: pgmap v16263: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 168 KiB/s wr, 9 op/s 2023-03-09T23:10:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29533.800415 secs 2023-03-09T23:10:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:23 smithi038 ceph-mon[92422]: pgmap v16263: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 168 KiB/s wr, 9 op/s 2023-03-09T23:10:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29533.800415 secs 2023-03-09T23:10:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 29995.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:10:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:25 smithi122 ceph-mon[93869]: pgmap v16264: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 140 KiB/s wr, 9 op/s 2023-03-09T23:10:25.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:25 smithi181 ceph-mon[94189]: pgmap v16264: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 140 KiB/s wr, 9 op/s 2023-03-09T23:10:25.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:25 smithi038 ceph-mon[92422]: pgmap v16264: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 140 KiB/s wr, 9 op/s 2023-03-09T23:10:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29536.405839 secs 2023-03-09T23:10:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29536.405839 secs 2023-03-09T23:10:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29536.405839 secs 2023-03-09T23:10:27.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:27 smithi122 ceph-mon[93869]: pgmap v16265: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2023-03-09T23:10:27.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:27 smithi181 ceph-mon[94189]: pgmap v16265: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2023-03-09T23:10:27.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:27 smithi038 ceph-mon[92422]: pgmap v16265: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2023-03-09T23:10:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29538.800535 secs 2023-03-09T23:10:28.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29538.800535 secs 2023-03-09T23:10:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29538.800535 secs 2023-03-09T23:10:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30000.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:10:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:29 smithi122 ceph-mon[93869]: pgmap v16266: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2023-03-09T23:10:29.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:29 smithi181 ceph-mon[94189]: pgmap v16266: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2023-03-09T23:10:29.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:29 smithi038 ceph-mon[92422]: pgmap v16266: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2023-03-09T23:10:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29541.405901 secs 2023-03-09T23:10:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:31 smithi122 ceph-mon[93869]: pgmap v16267: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2023-03-09T23:10:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29541.405901 secs 2023-03-09T23:10:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:31 smithi181 ceph-mon[94189]: pgmap v16267: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2023-03-09T23:10:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29541.405901 secs 2023-03-09T23:10:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:31 smithi038 ceph-mon[92422]: pgmap v16267: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2023-03-09T23:10:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:33 smithi122 ceph-mon[93869]: pgmap v16268: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 129 KiB/s wr, 8 op/s 2023-03-09T23:10:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29543.800666 secs 2023-03-09T23:10:33.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:33 smithi181 ceph-mon[94189]: pgmap v16268: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 129 KiB/s wr, 8 op/s 2023-03-09T23:10:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29543.800666 secs 2023-03-09T23:10:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:33 smithi038 ceph-mon[92422]: pgmap v16268: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 129 KiB/s wr, 8 op/s 2023-03-09T23:10:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29543.800666 secs 2023-03-09T23:10:33.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30005.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:10:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:35 smithi122 ceph-mon[93869]: pgmap v16269: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 129 KiB/s wr, 6 op/s 2023-03-09T23:10:35.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:35 smithi181 ceph-mon[94189]: pgmap v16269: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 129 KiB/s wr, 6 op/s 2023-03-09T23:10:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:35 smithi038 ceph-mon[92422]: pgmap v16269: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 129 KiB/s wr, 6 op/s 2023-03-09T23:10:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29546.405961 secs 2023-03-09T23:10:36.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29546.405961 secs 2023-03-09T23:10:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29546.405961 secs 2023-03-09T23:10:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:37 smithi122 ceph-mon[93869]: pgmap v16270: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 129 KiB/s wr, 8 op/s 2023-03-09T23:10:37.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:37 smithi181 ceph-mon[94189]: pgmap v16270: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 129 KiB/s wr, 8 op/s 2023-03-09T23:10:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:37 smithi038 ceph-mon[92422]: pgmap v16270: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 129 KiB/s wr, 8 op/s 2023-03-09T23:10:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29548.800759 secs 2023-03-09T23:10:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29548.800759 secs 2023-03-09T23:10:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29548.800759 secs 2023-03-09T23:10:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30010.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:10:39.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:39 smithi122 ceph-mon[93869]: pgmap v16271: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 128 KiB/s wr, 6 op/s 2023-03-09T23:10:39.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:39 smithi181 ceph-mon[94189]: pgmap v16271: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 128 KiB/s wr, 6 op/s 2023-03-09T23:10:39.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:39 smithi038 ceph-mon[92422]: pgmap v16271: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 128 KiB/s wr, 6 op/s 2023-03-09T23:10:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29551.405989 secs 2023-03-09T23:10:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:41 smithi122 ceph-mon[93869]: pgmap v16272: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 128 KiB/s wr, 6 op/s 2023-03-09T23:10:41.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29551.405989 secs 2023-03-09T23:10:41.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:41 smithi181 ceph-mon[94189]: pgmap v16272: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 128 KiB/s wr, 6 op/s 2023-03-09T23:10:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29551.405989 secs 2023-03-09T23:10:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:41 smithi038 ceph-mon[92422]: pgmap v16272: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 128 KiB/s wr, 6 op/s 2023-03-09T23:10:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:43 smithi122 ceph-mon[93869]: pgmap v16273: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 160 KiB/s wr, 8 op/s 2023-03-09T23:10:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29553.800887 secs 2023-03-09T23:10:43.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:43 smithi181 ceph-mon[94189]: pgmap v16273: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 160 KiB/s wr, 8 op/s 2023-03-09T23:10:43.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29553.800887 secs 2023-03-09T23:10:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:43 smithi038 ceph-mon[92422]: pgmap v16273: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 160 KiB/s wr, 8 op/s 2023-03-09T23:10:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29553.800887 secs 2023-03-09T23:10:43.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30015.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:10:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:45 smithi122 ceph-mon[93869]: pgmap v16274: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 32 KiB/s wr, 6 op/s 2023-03-09T23:10:45.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:45 smithi181 ceph-mon[94189]: pgmap v16274: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 32 KiB/s wr, 6 op/s 2023-03-09T23:10:45.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:45 smithi038 ceph-mon[92422]: pgmap v16274: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 32 KiB/s wr, 6 op/s 2023-03-09T23:10:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29556.406048 secs 2023-03-09T23:10:46.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29556.406048 secs 2023-03-09T23:10:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29556.406048 secs 2023-03-09T23:10:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:47 smithi122 ceph-mon[93869]: pgmap v16275: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 142 KiB/s wr, 8 op/s 2023-03-09T23:10:47.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:47 smithi181 ceph-mon[94189]: pgmap v16275: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 142 KiB/s wr, 8 op/s 2023-03-09T23:10:47.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:47 smithi038 ceph-mon[92422]: pgmap v16275: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 142 KiB/s wr, 8 op/s 2023-03-09T23:10:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29558.801008 secs 2023-03-09T23:10:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29558.801008 secs 2023-03-09T23:10:48.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29558.801008 secs 2023-03-09T23:10:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30020.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:10:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:49 smithi122 ceph-mon[93869]: pgmap v16276: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 142 KiB/s wr, 7 op/s 2023-03-09T23:10:49.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:49 smithi181 ceph-mon[94189]: pgmap v16276: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 142 KiB/s wr, 7 op/s 2023-03-09T23:10:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:49 smithi038 ceph-mon[92422]: pgmap v16276: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 142 KiB/s wr, 7 op/s 2023-03-09T23:10:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29561.406107 secs 2023-03-09T23:10:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:51 smithi122 ceph-mon[93869]: pgmap v16277: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 142 KiB/s wr, 6 op/s 2023-03-09T23:10:51.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29561.406107 secs 2023-03-09T23:10:51.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:51 smithi181 ceph-mon[94189]: pgmap v16277: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 142 KiB/s wr, 6 op/s 2023-03-09T23:10:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29561.406107 secs 2023-03-09T23:10:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:51 smithi038 ceph-mon[92422]: pgmap v16277: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 142 KiB/s wr, 6 op/s 2023-03-09T23:10:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:53 smithi122 ceph-mon[93869]: pgmap v16278: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 536 KiB/s rd, 374 KiB/s wr, 9 op/s 2023-03-09T23:10:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:10:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:10:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29563.801148 secs 2023-03-09T23:10:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:53 smithi181 ceph-mon[94189]: pgmap v16278: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 536 KiB/s rd, 374 KiB/s wr, 9 op/s 2023-03-09T23:10:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:10:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:10:53.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29563.801148 secs 2023-03-09T23:10:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:53 smithi038 ceph-mon[92422]: pgmap v16278: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 536 KiB/s rd, 374 KiB/s wr, 9 op/s 2023-03-09T23:10:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:10:53.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:10:53.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29563.801148 secs 2023-03-09T23:10:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30025.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:10:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:55 smithi122 ceph-mon[93869]: pgmap v16279: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 542 KiB/s rd, 381 KiB/s wr, 8 op/s 2023-03-09T23:10:55.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:55 smithi181 ceph-mon[94189]: pgmap v16279: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 542 KiB/s rd, 381 KiB/s wr, 8 op/s 2023-03-09T23:10:55.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:55 smithi038 ceph-mon[92422]: pgmap v16279: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 542 KiB/s rd, 381 KiB/s wr, 8 op/s 2023-03-09T23:10:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29566.406168 secs 2023-03-09T23:10:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:10:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:10:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29566.406168 secs 2023-03-09T23:10:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:10:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:10:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29566.406168 secs 2023-03-09T23:10:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:10:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:10:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:57 smithi122 ceph-mon[93869]: pgmap v16280: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 545 KiB/s rd, 381 KiB/s wr, 11 op/s 2023-03-09T23:10:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:10:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:10:57.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:57 smithi181 ceph-mon[94189]: pgmap v16280: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 545 KiB/s rd, 381 KiB/s wr, 11 op/s 2023-03-09T23:10:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:10:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:10:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:57 smithi038 ceph-mon[92422]: pgmap v16280: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 545 KiB/s rd, 381 KiB/s wr, 11 op/s 2023-03-09T23:10:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:10:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:10:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30030.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:10:59.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:59 smithi122 ceph-mon[93869]: pgmap v16281: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 621 KiB/s wr, 10 op/s 2023-03-09T23:10:59.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:10:59.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:10:59.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:10:59.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:10:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:10:59.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:59 smithi181 ceph-mon[94189]: pgmap v16281: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 621 KiB/s wr, 10 op/s 2023-03-09T23:10:59.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:10:59.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:10:59.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:10:59.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:10:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:10:59.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:59 smithi038 ceph-mon[92422]: pgmap v16281: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 621 KiB/s wr, 10 op/s 2023-03-09T23:10:59.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:10:59.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:10:59.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:10:59.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:10:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:11:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29571.406229 secs 2023-03-09T23:11:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:01 smithi122 ceph-mon[93869]: pgmap v16282: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 508 KiB/s rd, 695 KiB/s wr, 10 op/s 2023-03-09T23:11:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:11:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:11:01.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29571.406229 secs 2023-03-09T23:11:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:01 smithi181 ceph-mon[94189]: pgmap v16282: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 508 KiB/s rd, 695 KiB/s wr, 10 op/s 2023-03-09T23:11:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:11:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:11:01.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29571.406229 secs 2023-03-09T23:11:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:01 smithi038 ceph-mon[92422]: pgmap v16282: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 508 KiB/s rd, 695 KiB/s wr, 10 op/s 2023-03-09T23:11:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:11:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:11:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:02 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:11:02.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:02 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:11:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:02 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:11:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:03 smithi122 ceph-mon[93869]: pgmap v16283: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 509 KiB/s rd, 695 KiB/s wr, 13 op/s 2023-03-09T23:11:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:03 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:11:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:03 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:11:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:03 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:11:03.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:03 smithi181 ceph-mon[94189]: pgmap v16283: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 509 KiB/s rd, 695 KiB/s wr, 13 op/s 2023-03-09T23:11:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:03 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:11:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:03 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:11:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:03 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:11:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:03 smithi038 ceph-mon[92422]: pgmap v16283: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 509 KiB/s rd, 695 KiB/s wr, 13 op/s 2023-03-09T23:11:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:03 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:11:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:03 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:11:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:03 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:11:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30035.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:11:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:05 smithi122 ceph-mon[93869]: pgmap v16284: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 466 KiB/s rd, 487 KiB/s wr, 13 op/s 2023-03-09T23:11:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29576.406287 secs 2023-03-09T23:11:05.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:05 smithi181 ceph-mon[94189]: pgmap v16284: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 466 KiB/s rd, 487 KiB/s wr, 13 op/s 2023-03-09T23:11:05.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29576.406287 secs 2023-03-09T23:11:05.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:05 smithi038 ceph-mon[92422]: pgmap v16284: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 466 KiB/s rd, 487 KiB/s wr, 13 op/s 2023-03-09T23:11:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29576.406287 secs 2023-03-09T23:11:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:07 smithi122 ceph-mon[93869]: pgmap v16285: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 670 KiB/s rd, 700 KiB/s wr, 16 op/s 2023-03-09T23:11:07.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:07 smithi181 ceph-mon[94189]: pgmap v16285: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 670 KiB/s rd, 700 KiB/s wr, 16 op/s 2023-03-09T23:11:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:07 smithi038 ceph-mon[92422]: pgmap v16285: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 670 KiB/s rd, 700 KiB/s wr, 16 op/s 2023-03-09T23:11:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:11:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:11:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:11:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:11:08.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:11:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:11:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30040.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:11:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:09 smithi122 ceph-mon[93869]: pgmap v16286: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 678 KiB/s rd, 716 KiB/s wr, 23 op/s 2023-03-09T23:11:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:09 smithi181 ceph-mon[94189]: pgmap v16286: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 678 KiB/s rd, 716 KiB/s wr, 23 op/s 2023-03-09T23:11:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:09 smithi038 ceph-mon[92422]: pgmap v16286: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 678 KiB/s rd, 716 KiB/s wr, 23 op/s 2023-03-09T23:11:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29581.406348 secs 2023-03-09T23:11:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29581.406348 secs 2023-03-09T23:11:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29581.406348 secs 2023-03-09T23:11:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:11 smithi038 ceph-mon[92422]: pgmap v16287: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 712 KiB/s rd, 413 KiB/s wr, 22 op/s 2023-03-09T23:11:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:11 smithi122 ceph-mon[93869]: pgmap v16287: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 712 KiB/s rd, 413 KiB/s wr, 22 op/s 2023-03-09T23:11:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:11 smithi181 ceph-mon[94189]: pgmap v16287: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 712 KiB/s rd, 413 KiB/s wr, 22 op/s 2023-03-09T23:11:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:11:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:11:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:11:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:11:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:11:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:11:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:13 smithi038 ceph-mon[92422]: pgmap v16288: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 867 KiB/s rd, 542 KiB/s wr, 25 op/s 2023-03-09T23:11:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:11:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:11:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:13 smithi122 ceph-mon[93869]: pgmap v16288: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 867 KiB/s rd, 542 KiB/s wr, 25 op/s 2023-03-09T23:11:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:11:13.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:11:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:13 smithi181 ceph-mon[94189]: pgmap v16288: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 867 KiB/s rd, 542 KiB/s wr, 25 op/s 2023-03-09T23:11:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:11:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:11:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30045.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:11:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:15 smithi122 ceph-mon[93869]: pgmap v16289: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 541 KiB/s wr, 22 op/s 2023-03-09T23:11:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29586.406408 secs 2023-03-09T23:11:15.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:15 smithi181 ceph-mon[94189]: pgmap v16289: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 541 KiB/s wr, 22 op/s 2023-03-09T23:11:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29586.406408 secs 2023-03-09T23:11:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:15 smithi038 ceph-mon[92422]: pgmap v16289: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 541 KiB/s wr, 22 op/s 2023-03-09T23:11:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29586.406408 secs 2023-03-09T23:11:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:17 smithi122 ceph-mon[93869]: pgmap v16290: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 813 KiB/s rd, 654 KiB/s wr, 23 op/s 2023-03-09T23:11:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:17 smithi181 ceph-mon[94189]: pgmap v16290: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 813 KiB/s rd, 654 KiB/s wr, 23 op/s 2023-03-09T23:11:17.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:17 smithi038 ceph-mon[92422]: pgmap v16290: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 813 KiB/s rd, 654 KiB/s wr, 23 op/s 2023-03-09T23:11:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30050.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:11:19.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:19 smithi122 ceph-mon[93869]: pgmap v16291: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 571 KiB/s rd, 403 KiB/s wr, 19 op/s 2023-03-09T23:11:19.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:19 smithi181 ceph-mon[94189]: pgmap v16291: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 571 KiB/s rd, 403 KiB/s wr, 19 op/s 2023-03-09T23:11:19.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:19 smithi038 ceph-mon[92422]: pgmap v16291: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 571 KiB/s rd, 403 KiB/s wr, 19 op/s 2023-03-09T23:11:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29591.406473 secs 2023-03-09T23:11:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29591.406473 secs 2023-03-09T23:11:20.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29591.406473 secs 2023-03-09T23:11:21.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:21 smithi122 ceph-mon[93869]: pgmap v16292: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 559 KiB/s rd, 386 KiB/s wr, 8 op/s 2023-03-09T23:11:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:21 smithi181 ceph-mon[94189]: pgmap v16292: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 559 KiB/s rd, 386 KiB/s wr, 8 op/s 2023-03-09T23:11:21.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:21 smithi038 ceph-mon[92422]: pgmap v16292: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 559 KiB/s rd, 386 KiB/s wr, 8 op/s 2023-03-09T23:11:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:23 smithi122 ceph-mon[93869]: pgmap v16293: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 453 KiB/s rd, 450 KiB/s wr, 11 op/s 2023-03-09T23:11:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:23 smithi181 ceph-mon[94189]: pgmap v16293: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 453 KiB/s rd, 450 KiB/s wr, 11 op/s 2023-03-09T23:11:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:23 smithi038 ceph-mon[92422]: pgmap v16293: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 453 KiB/s rd, 450 KiB/s wr, 11 op/s 2023-03-09T23:11:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30055.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:11:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:25 smithi122 ceph-mon[93869]: pgmap v16294: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 257 KiB/s wr, 9 op/s 2023-03-09T23:11:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29596.406532 secs 2023-03-09T23:11:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:25 smithi181 ceph-mon[94189]: pgmap v16294: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 257 KiB/s wr, 9 op/s 2023-03-09T23:11:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29596.406532 secs 2023-03-09T23:11:25.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:25 smithi038 ceph-mon[92422]: pgmap v16294: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 257 KiB/s wr, 9 op/s 2023-03-09T23:11:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29596.406532 secs 2023-03-09T23:11:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:27 smithi122 ceph-mon[93869]: pgmap v16295: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 330 KiB/s wr, 13 op/s 2023-03-09T23:11:27.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:27 smithi181 ceph-mon[94189]: pgmap v16295: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 330 KiB/s wr, 13 op/s 2023-03-09T23:11:27.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:27 smithi038 ceph-mon[92422]: pgmap v16295: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 330 KiB/s wr, 13 op/s 2023-03-09T23:11:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:28 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:11:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:28 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:11:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29598.801894 secs 2023-03-09T23:11:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:28 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:11:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:28 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:11:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29598.801894 secs 2023-03-09T23:11:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:28 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:11:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:28 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:11:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29598.801894 secs 2023-03-09T23:11:28.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30060.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:11:29.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:29 smithi122 ceph-mon[93869]: pgmap v16296: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 452 KiB/s wr, 11 op/s 2023-03-09T23:11:29.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:29 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:11:29.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:29 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:11:29.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:29 smithi181 ceph-mon[94189]: pgmap v16296: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 452 KiB/s wr, 11 op/s 2023-03-09T23:11:29.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:29 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:11:29.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:29 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:11:29.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:29 smithi038 ceph-mon[92422]: pgmap v16296: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 452 KiB/s wr, 11 op/s 2023-03-09T23:11:29.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:29 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:11:29.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:29 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:11:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:30 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:11:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:30 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:11:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:30 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:11:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:30 smithi122 ceph-mon[93869]: osdmap e1233: 12 total, 12 up, 12 in 2023-03-09T23:11:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29601.406590 secs 2023-03-09T23:11:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:30 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:11:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:30 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:11:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:30 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:11:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:30 smithi181 ceph-mon[94189]: osdmap e1233: 12 total, 12 up, 12 in 2023-03-09T23:11:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29601.406590 secs 2023-03-09T23:11:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:30 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:11:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:30 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:11:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:30 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:11:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:30 smithi038 ceph-mon[92422]: osdmap e1233: 12 total, 12 up, 12 in 2023-03-09T23:11:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29601.406590 secs 2023-03-09T23:11:31.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:31 smithi122 ceph-mon[93869]: pgmap v16298: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 519 KiB/s rd, 543 KiB/s wr, 13 op/s 2023-03-09T23:11:31.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:31 smithi181 ceph-mon[94189]: pgmap v16298: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 519 KiB/s rd, 543 KiB/s wr, 13 op/s 2023-03-09T23:11:31.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:31 smithi038 ceph-mon[92422]: pgmap v16298: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 519 KiB/s rd, 543 KiB/s wr, 13 op/s 2023-03-09T23:11:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:33 smithi122 ceph-mon[93869]: pgmap v16299: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 546 KiB/s rd, 571 KiB/s wr, 14 op/s 2023-03-09T23:11:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29603.801973 secs 2023-03-09T23:11:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:33 smithi181 ceph-mon[94189]: pgmap v16299: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 546 KiB/s rd, 571 KiB/s wr, 14 op/s 2023-03-09T23:11:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29603.801973 secs 2023-03-09T23:11:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:33 smithi038 ceph-mon[92422]: pgmap v16299: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 546 KiB/s rd, 571 KiB/s wr, 14 op/s 2023-03-09T23:11:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29603.801973 secs 2023-03-09T23:11:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30065.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:11:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:35 smithi122 ceph-mon[93869]: pgmap v16300: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 545 KiB/s rd, 558 KiB/s wr, 12 op/s 2023-03-09T23:11:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29606.406668 secs 2023-03-09T23:11:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:35 smithi181 ceph-mon[94189]: pgmap v16300: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 545 KiB/s rd, 558 KiB/s wr, 12 op/s 2023-03-09T23:11:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29606.406668 secs 2023-03-09T23:11:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:35 smithi038 ceph-mon[92422]: pgmap v16300: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 545 KiB/s rd, 558 KiB/s wr, 12 op/s 2023-03-09T23:11:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29606.406668 secs 2023-03-09T23:11:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:37 smithi122 ceph-mon[93869]: pgmap v16301: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 676 KiB/s rd, 676 KiB/s wr, 11 op/s 2023-03-09T23:11:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:37 smithi122 ceph-mon[93869]: osdmap e1234: 12 total, 12 up, 12 in 2023-03-09T23:11:37.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:37 smithi181 ceph-mon[94189]: pgmap v16301: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 676 KiB/s rd, 676 KiB/s wr, 11 op/s 2023-03-09T23:11:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:37 smithi181 ceph-mon[94189]: osdmap e1234: 12 total, 12 up, 12 in 2023-03-09T23:11:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:37 smithi038 ceph-mon[92422]: pgmap v16301: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 676 KiB/s rd, 676 KiB/s wr, 11 op/s 2023-03-09T23:11:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:37 smithi038 ceph-mon[92422]: osdmap e1234: 12 total, 12 up, 12 in 2023-03-09T23:11:38.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29608.802092 secs 2023-03-09T23:11:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29608.802092 secs 2023-03-09T23:11:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29608.802092 secs 2023-03-09T23:11:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30070.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:11:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:39 smithi122 ceph-mon[93869]: pgmap v16303: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 419 KiB/s wr, 11 op/s 2023-03-09T23:11:39.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:39 smithi181 ceph-mon[94189]: pgmap v16303: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 419 KiB/s wr, 11 op/s 2023-03-09T23:11:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:39 smithi038 ceph-mon[92422]: pgmap v16303: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 419 KiB/s wr, 11 op/s 2023-03-09T23:11:40.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29611.406743 secs 2023-03-09T23:11:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29611.406743 secs 2023-03-09T23:11:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29611.406743 secs 2023-03-09T23:11:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:41 smithi122 ceph-mon[93869]: pgmap v16304: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 365 KiB/s wr, 10 op/s 2023-03-09T23:11:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:41 smithi181 ceph-mon[94189]: pgmap v16304: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 365 KiB/s wr, 10 op/s 2023-03-09T23:11:41.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:41 smithi038 ceph-mon[92422]: pgmap v16304: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 365 KiB/s wr, 10 op/s 2023-03-09T23:11:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:43 smithi122 ceph-mon[93869]: pgmap v16305: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 312 KiB/s wr, 9 op/s 2023-03-09T23:11:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:43 smithi181 ceph-mon[94189]: pgmap v16305: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 312 KiB/s wr, 9 op/s 2023-03-09T23:11:43.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:43 smithi038 ceph-mon[92422]: pgmap v16305: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 312 KiB/s wr, 9 op/s 2023-03-09T23:11:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30075.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:11:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:45 smithi122 ceph-mon[93869]: pgmap v16306: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 312 KiB/s wr, 8 op/s 2023-03-09T23:11:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29616.406805 secs 2023-03-09T23:11:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:45 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:11:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:45 smithi181 ceph-mon[94189]: pgmap v16306: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 312 KiB/s wr, 8 op/s 2023-03-09T23:11:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29616.406805 secs 2023-03-09T23:11:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:45 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:11:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:45 smithi038 ceph-mon[92422]: pgmap v16306: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 312 KiB/s wr, 8 op/s 2023-03-09T23:11:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29616.406805 secs 2023-03-09T23:11:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:45 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:11:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:46 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:11:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:46 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:11:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:46 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:11:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:46 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:11:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:46 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:11:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:46 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:11:46.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:46 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:11:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:46 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:11:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:46 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:11:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:47 smithi122 ceph-mon[93869]: pgmap v16307: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 213 KiB/s wr, 8 op/s 2023-03-09T23:11:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:47 smithi181 ceph-mon[94189]: pgmap v16307: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 213 KiB/s wr, 8 op/s 2023-03-09T23:11:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:47 smithi038 ceph-mon[92422]: pgmap v16307: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 213 KiB/s wr, 8 op/s 2023-03-09T23:11:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:11:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:11:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:11:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:11:48.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:11:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:11:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30080.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:11:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:49 smithi122 ceph-mon[93869]: pgmap v16308: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 194 KiB/s wr, 7 op/s 2023-03-09T23:11:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:11:49.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:11:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:49 smithi181 ceph-mon[94189]: pgmap v16308: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 194 KiB/s wr, 7 op/s 2023-03-09T23:11:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:11:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:11:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:49 smithi038 ceph-mon[92422]: pgmap v16308: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 194 KiB/s wr, 7 op/s 2023-03-09T23:11:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:11:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:11:50.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29621.406870 secs 2023-03-09T23:11:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29621.406870 secs 2023-03-09T23:11:50.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29621.406870 secs 2023-03-09T23:11:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:51 smithi122 ceph-mon[93869]: pgmap v16309: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 178 KiB/s wr, 6 op/s 2023-03-09T23:11:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:51 smithi181 ceph-mon[94189]: pgmap v16309: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 178 KiB/s wr, 6 op/s 2023-03-09T23:11:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:51 smithi038 ceph-mon[92422]: pgmap v16309: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 178 KiB/s wr, 6 op/s 2023-03-09T23:11:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:53 smithi122 ceph-mon[93869]: pgmap v16310: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 393 KiB/s wr, 10 op/s 2023-03-09T23:11:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:53 smithi181 ceph-mon[94189]: pgmap v16310: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 393 KiB/s wr, 10 op/s 2023-03-09T23:11:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:53 smithi038 ceph-mon[92422]: pgmap v16310: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 393 KiB/s wr, 10 op/s 2023-03-09T23:11:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30085.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:11:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:55 smithi122 ceph-mon[93869]: pgmap v16311: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 304 KiB/s wr, 6 op/s 2023-03-09T23:11:55.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29626.406930 secs 2023-03-09T23:11:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:55 smithi181 ceph-mon[94189]: pgmap v16311: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 304 KiB/s wr, 6 op/s 2023-03-09T23:11:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29626.406930 secs 2023-03-09T23:11:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:55 smithi038 ceph-mon[92422]: pgmap v16311: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 304 KiB/s wr, 6 op/s 2023-03-09T23:11:55.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29626.406930 secs 2023-03-09T23:11:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:11:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:11:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:11:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:11:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:11:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:11:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:57 smithi122 ceph-mon[93869]: pgmap v16312: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 456 KiB/s wr, 11 op/s 2023-03-09T23:11:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:11:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:11:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:57 smithi181 ceph-mon[94189]: pgmap v16312: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 456 KiB/s wr, 11 op/s 2023-03-09T23:11:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:11:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:11:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:57 smithi038 ceph-mon[92422]: pgmap v16312: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 456 KiB/s wr, 11 op/s 2023-03-09T23:11:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:11:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:11:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:58 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 29628.802564 secs 2023-03-09T23:11:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:58 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 29628.802564 secs 2023-03-09T23:11:58.835 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:58 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 29628.802564 secs 2023-03-09T23:11:58.835 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30090.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:11:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:59 smithi122 ceph-mon[93869]: pgmap v16313: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 368 KiB/s wr, 8 op/s 2023-03-09T23:11:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:11:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:11:59.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:59 smithi181 ceph-mon[94189]: pgmap v16313: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 368 KiB/s wr, 8 op/s 2023-03-09T23:11:59.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:11:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:11:59.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:59 smithi038 ceph-mon[92422]: pgmap v16313: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 368 KiB/s wr, 8 op/s 2023-03-09T23:11:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:11:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:12:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:12:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:12:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:12:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:12:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29631.406992 secs 2023-03-09T23:12:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:12:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:12:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:12:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:12:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:12:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:12:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:12:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29631.406992 secs 2023-03-09T23:12:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:12:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:12:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:12:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:12:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:12:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:12:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:12:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29631.406992 secs 2023-03-09T23:12:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:12:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:12:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:12:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:01 smithi122 ceph-mon[93869]: pgmap v16314: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 596 KiB/s rd, 581 KiB/s wr, 9 op/s 2023-03-09T23:12:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:12:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:12:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:01 smithi181 ceph-mon[94189]: pgmap v16314: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 596 KiB/s rd, 581 KiB/s wr, 9 op/s 2023-03-09T23:12:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:12:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:12:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:01 smithi038 ceph-mon[92422]: pgmap v16314: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 596 KiB/s rd, 581 KiB/s wr, 9 op/s 2023-03-09T23:12:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:12:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:12:02.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:02 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:12:02.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:02 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:12:02.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:02 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:12:02.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:02 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 29633.802682 secs 2023-03-09T23:12:02.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:02 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:12:02.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:02 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:12:02.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:02 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:12:02.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:02 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 29633.802682 secs 2023-03-09T23:12:02.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:02 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:12:02.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:02 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:12:02.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:02 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:12:02.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:02 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 29633.802682 secs 2023-03-09T23:12:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30095.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:12:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:03 smithi122 ceph-mon[93869]: pgmap v16315: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 880 KiB/s rd, 958 KiB/s wr, 15 op/s 2023-03-09T23:12:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:03 smithi181 ceph-mon[94189]: pgmap v16315: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 880 KiB/s rd, 958 KiB/s wr, 15 op/s 2023-03-09T23:12:03.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:03 smithi038 ceph-mon[92422]: pgmap v16315: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 880 KiB/s rd, 958 KiB/s wr, 15 op/s 2023-03-09T23:12:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:05 smithi122 ceph-mon[93869]: pgmap v16316: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 671 KiB/s rd, 754 KiB/s wr, 23 op/s 2023-03-09T23:12:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29636.407051 secs 2023-03-09T23:12:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:05 smithi181 ceph-mon[94189]: pgmap v16316: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 671 KiB/s rd, 754 KiB/s wr, 23 op/s 2023-03-09T23:12:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29636.407051 secs 2023-03-09T23:12:05.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:05 smithi038 ceph-mon[92422]: pgmap v16316: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 671 KiB/s rd, 754 KiB/s wr, 23 op/s 2023-03-09T23:12:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29636.407051 secs 2023-03-09T23:12:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:07 smithi122 ceph-mon[93869]: pgmap v16317: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 791 KiB/s rd, 754 KiB/s wr, 27 op/s 2023-03-09T23:12:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:07 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 29638.802818 secs 2023-03-09T23:12:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:07 smithi181 ceph-mon[94189]: pgmap v16317: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 791 KiB/s rd, 754 KiB/s wr, 27 op/s 2023-03-09T23:12:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:07 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 29638.802818 secs 2023-03-09T23:12:07.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:07 smithi038 ceph-mon[92422]: pgmap v16317: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 791 KiB/s rd, 754 KiB/s wr, 27 op/s 2023-03-09T23:12:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:07 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 29638.802818 secs 2023-03-09T23:12:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30100.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:12:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:09 smithi122 ceph-mon[93869]: pgmap v16318: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 803 KiB/s rd, 715 KiB/s wr, 24 op/s 2023-03-09T23:12:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:12:09.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:12:09.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:09 smithi181 ceph-mon[94189]: pgmap v16318: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 803 KiB/s rd, 715 KiB/s wr, 24 op/s 2023-03-09T23:12:09.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:12:09.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:12:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:09 smithi038 ceph-mon[92422]: pgmap v16318: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 803 KiB/s rd, 715 KiB/s wr, 24 op/s 2023-03-09T23:12:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:12:10.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:12:10.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29641.407114 secs 2023-03-09T23:12:10.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29641.407114 secs 2023-03-09T23:12:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29641.407114 secs 2023-03-09T23:12:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:11 smithi122 ceph-mon[93869]: pgmap v16319: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 803 KiB/s rd, 715 KiB/s wr, 23 op/s 2023-03-09T23:12:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:11 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:12:11.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:11 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:12:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:11 smithi181 ceph-mon[94189]: pgmap v16319: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 803 KiB/s rd, 715 KiB/s wr, 23 op/s 2023-03-09T23:12:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:11 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:12:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:11 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:12:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:11 smithi038 ceph-mon[92422]: pgmap v16319: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 803 KiB/s rd, 715 KiB/s wr, 23 op/s 2023-03-09T23:12:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:11 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:12:12.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:11 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:12:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:13 smithi122 ceph-mon[93869]: pgmap v16320: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 525 KiB/s rd, 502 KiB/s wr, 25 op/s 2023-03-09T23:12:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:13 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 29643.802954 secs 2023-03-09T23:12:13.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:13 smithi181 ceph-mon[94189]: pgmap v16320: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 525 KiB/s rd, 502 KiB/s wr, 25 op/s 2023-03-09T23:12:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:13 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 29643.802954 secs 2023-03-09T23:12:13.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:13 smithi038 ceph-mon[92422]: pgmap v16320: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 525 KiB/s rd, 502 KiB/s wr, 25 op/s 2023-03-09T23:12:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:13 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 29643.802954 secs 2023-03-09T23:12:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30105.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:12:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:15 smithi122 ceph-mon[93869]: pgmap v16321: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 299 KiB/s wr, 20 op/s 2023-03-09T23:12:15.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:15 smithi181 ceph-mon[94189]: pgmap v16321: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 299 KiB/s wr, 20 op/s 2023-03-09T23:12:15.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:15 smithi038 ceph-mon[92422]: pgmap v16321: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 299 KiB/s wr, 20 op/s 2023-03-09T23:12:16.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29646.407171 secs 2023-03-09T23:12:16.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29646.407171 secs 2023-03-09T23:12:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29646.407171 secs 2023-03-09T23:12:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:17 smithi122 ceph-mon[93869]: pgmap v16322: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 288 KiB/s wr, 12 op/s 2023-03-09T23:12:17.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:17 smithi181 ceph-mon[94189]: pgmap v16322: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 288 KiB/s wr, 12 op/s 2023-03-09T23:12:17.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:17 smithi038 ceph-mon[92422]: pgmap v16322: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 288 KiB/s wr, 12 op/s 2023-03-09T23:12:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:18 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 29648.803096 secs 2023-03-09T23:12:18.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:18 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 29648.803096 secs 2023-03-09T23:12:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:18 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 29648.803096 secs 2023-03-09T23:12:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30110.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:12:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:19 smithi122 ceph-mon[93869]: pgmap v16323: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 353 KiB/s wr, 10 op/s 2023-03-09T23:12:19.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:19 smithi181 ceph-mon[94189]: pgmap v16323: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 353 KiB/s wr, 10 op/s 2023-03-09T23:12:19.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:19 smithi038 ceph-mon[92422]: pgmap v16323: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 353 KiB/s wr, 10 op/s 2023-03-09T23:12:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:12:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:12:20.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:12:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:12:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:12:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:12:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29651.407248 secs 2023-03-09T23:12:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:21 smithi122 ceph-mon[93869]: pgmap v16324: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 240 KiB/s wr, 8 op/s 2023-03-09T23:12:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:12:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:12:21.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29651.407248 secs 2023-03-09T23:12:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:21 smithi181 ceph-mon[94189]: pgmap v16324: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 240 KiB/s wr, 8 op/s 2023-03-09T23:12:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:12:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:12:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29651.407248 secs 2023-03-09T23:12:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:21 smithi038 ceph-mon[92422]: pgmap v16324: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 240 KiB/s wr, 8 op/s 2023-03-09T23:12:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:12:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:12:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:23 smithi122 ceph-mon[93869]: pgmap v16325: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 551 KiB/s wr, 13 op/s 2023-03-09T23:12:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:23 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 29653.803210 secs 2023-03-09T23:12:23.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:23 smithi181 ceph-mon[94189]: pgmap v16325: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 551 KiB/s wr, 13 op/s 2023-03-09T23:12:23.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:23 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 29653.803210 secs 2023-03-09T23:12:23.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:23 smithi038 ceph-mon[92422]: pgmap v16325: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 551 KiB/s wr, 13 op/s 2023-03-09T23:12:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:23 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 29653.803210 secs 2023-03-09T23:12:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30115.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:12:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:25 smithi122 ceph-mon[93869]: pgmap v16326: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 532 KiB/s rd, 551 KiB/s wr, 10 op/s 2023-03-09T23:12:25.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:25 smithi181 ceph-mon[94189]: pgmap v16326: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 532 KiB/s rd, 551 KiB/s wr, 10 op/s 2023-03-09T23:12:25.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:25 smithi038 ceph-mon[92422]: pgmap v16326: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 532 KiB/s rd, 551 KiB/s wr, 10 op/s 2023-03-09T23:12:26.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29656.407309 secs 2023-03-09T23:12:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:26 smithi122 ceph-mon[93869]: osdmap e1235: 12 total, 12 up, 12 in 2023-03-09T23:12:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29656.407309 secs 2023-03-09T23:12:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:26 smithi181 ceph-mon[94189]: osdmap e1235: 12 total, 12 up, 12 in 2023-03-09T23:12:26.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29656.407309 secs 2023-03-09T23:12:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:26 smithi038 ceph-mon[92422]: osdmap e1235: 12 total, 12 up, 12 in 2023-03-09T23:12:27.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:27 smithi122 ceph-mon[93869]: pgmap v16328: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 558 KiB/s rd, 582 KiB/s wr, 13 op/s 2023-03-09T23:12:27.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:27 smithi181 ceph-mon[94189]: pgmap v16328: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 558 KiB/s rd, 582 KiB/s wr, 13 op/s 2023-03-09T23:12:27.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:27 smithi038 ceph-mon[92422]: pgmap v16328: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 558 KiB/s rd, 582 KiB/s wr, 13 op/s 2023-03-09T23:12:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:28 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 29658.803275 secs 2023-03-09T23:12:28.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:28 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 29658.803275 secs 2023-03-09T23:12:28.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:28 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 29658.803275 secs 2023-03-09T23:12:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30120.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:12:29.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:29 smithi122 ceph-mon[93869]: pgmap v16329: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 557 KiB/s rd, 504 KiB/s wr, 10 op/s 2023-03-09T23:12:29.489 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:29 smithi181 ceph-mon[94189]: pgmap v16329: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 557 KiB/s rd, 504 KiB/s wr, 10 op/s 2023-03-09T23:12:29.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:29 smithi038 ceph-mon[92422]: pgmap v16329: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 557 KiB/s rd, 504 KiB/s wr, 10 op/s 2023-03-09T23:12:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29661.407370 secs 2023-03-09T23:12:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:31 smithi122 ceph-mon[93869]: pgmap v16330: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 557 KiB/s rd, 504 KiB/s wr, 10 op/s 2023-03-09T23:12:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29661.407370 secs 2023-03-09T23:12:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:31 smithi181 ceph-mon[94189]: pgmap v16330: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 557 KiB/s rd, 504 KiB/s wr, 10 op/s 2023-03-09T23:12:31.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29661.407370 secs 2023-03-09T23:12:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:31 smithi038 ceph-mon[92422]: pgmap v16330: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 557 KiB/s rd, 504 KiB/s wr, 10 op/s 2023-03-09T23:12:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:32 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:12:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:32 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:12:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:32 smithi122 ceph-mon[93869]: osdmap e1236: 12 total, 12 up, 12 in 2023-03-09T23:12:32.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:32 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:12:32.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:32 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:12:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:32 smithi181 ceph-mon[94189]: osdmap e1236: 12 total, 12 up, 12 in 2023-03-09T23:12:32.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:32 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:12:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:32 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:12:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:32 smithi038 ceph-mon[92422]: osdmap e1236: 12 total, 12 up, 12 in 2023-03-09T23:12:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:33 smithi122 ceph-mon[93869]: pgmap v16331: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 131 KiB/s wr, 9 op/s 2023-03-09T23:12:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:33 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:12:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:33 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:12:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:33 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 29663.803376 secs 2023-03-09T23:12:33.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:33 smithi181 ceph-mon[94189]: pgmap v16331: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 131 KiB/s wr, 9 op/s 2023-03-09T23:12:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:33 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:12:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:33 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:12:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:33 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 29663.803376 secs 2023-03-09T23:12:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:33 smithi038 ceph-mon[92422]: pgmap v16331: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 131 KiB/s wr, 9 op/s 2023-03-09T23:12:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:33 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:12:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:33 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:12:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:33 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 29663.803376 secs 2023-03-09T23:12:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30125.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:12:35.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:35 smithi122 ceph-mon[93869]: pgmap v16333: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 146 KiB/s wr, 10 op/s 2023-03-09T23:12:35.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:35 smithi181 ceph-mon[94189]: pgmap v16333: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 146 KiB/s wr, 10 op/s 2023-03-09T23:12:35.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:35 smithi038 ceph-mon[92422]: pgmap v16333: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 146 KiB/s wr, 10 op/s 2023-03-09T23:12:36.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29666.407454 secs 2023-03-09T23:12:36.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29666.407454 secs 2023-03-09T23:12:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29666.407454 secs 2023-03-09T23:12:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:37 smithi122 ceph-mon[93869]: pgmap v16334: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 392 KiB/s wr, 9 op/s 2023-03-09T23:12:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:12:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:12:37.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:37 smithi181 ceph-mon[94189]: pgmap v16334: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 392 KiB/s wr, 9 op/s 2023-03-09T23:12:37.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:12:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:12:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:37 smithi038 ceph-mon[92422]: pgmap v16334: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 392 KiB/s wr, 9 op/s 2023-03-09T23:12:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:12:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:12:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29668.803508 secs 2023-03-09T23:12:38.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29668.803508 secs 2023-03-09T23:12:38.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29668.803508 secs 2023-03-09T23:12:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30130.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:12:39.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:39 smithi122 ceph-mon[93869]: pgmap v16335: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 797 KiB/s rd, 795 KiB/s wr, 11 op/s 2023-03-09T23:12:39.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:39 smithi181 ceph-mon[94189]: pgmap v16335: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 797 KiB/s rd, 795 KiB/s wr, 11 op/s 2023-03-09T23:12:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:39 smithi038 ceph-mon[92422]: pgmap v16335: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 797 KiB/s rd, 795 KiB/s wr, 11 op/s 2023-03-09T23:12:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:12:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:12:40.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:12:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:12:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:12:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:12:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29671.407515 secs 2023-03-09T23:12:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:41 smithi122 ceph-mon[93869]: pgmap v16336: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 797 KiB/s rd, 795 KiB/s wr, 11 op/s 2023-03-09T23:12:41.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29671.407515 secs 2023-03-09T23:12:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:41 smithi181 ceph-mon[94189]: pgmap v16336: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 797 KiB/s rd, 795 KiB/s wr, 11 op/s 2023-03-09T23:12:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29671.407515 secs 2023-03-09T23:12:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:41 smithi038 ceph-mon[92422]: pgmap v16336: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 797 KiB/s rd, 795 KiB/s wr, 11 op/s 2023-03-09T23:12:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:43 smithi122 ceph-mon[93869]: pgmap v16337: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 804 KiB/s rd, 802 KiB/s wr, 11 op/s 2023-03-09T23:12:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:43 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:12:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:43 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:12:43.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29673.803659 secs 2023-03-09T23:12:43.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:43 smithi181 ceph-mon[94189]: pgmap v16337: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 804 KiB/s rd, 802 KiB/s wr, 11 op/s 2023-03-09T23:12:43.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:43 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:12:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:43 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:12:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29673.803659 secs 2023-03-09T23:12:43.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:43 smithi038 ceph-mon[92422]: pgmap v16337: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 804 KiB/s rd, 802 KiB/s wr, 11 op/s 2023-03-09T23:12:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:43 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:12:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:43 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:12:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29673.803659 secs 2023-03-09T23:12:43.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30135.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:12:45.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:45 smithi181 ceph-mon[94189]: pgmap v16338: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 976 KiB/s rd, 1010 KiB/s wr, 10 op/s 2023-03-09T23:12:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29676.407576 secs 2023-03-09T23:12:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:45 smithi038 ceph-mon[92422]: pgmap v16338: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 976 KiB/s rd, 1010 KiB/s wr, 10 op/s 2023-03-09T23:12:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29676.407576 secs 2023-03-09T23:12:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:45 smithi122 ceph-mon[93869]: pgmap v16338: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 976 KiB/s rd, 1010 KiB/s wr, 10 op/s 2023-03-09T23:12:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29676.407576 secs 2023-03-09T23:12:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:47 smithi038 ceph-mon[92422]: pgmap v16339: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 14 op/s 2023-03-09T23:12:47.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:47 smithi122 ceph-mon[93869]: pgmap v16339: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 14 op/s 2023-03-09T23:12:47.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:47 smithi181 ceph-mon[94189]: pgmap v16339: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 14 op/s 2023-03-09T23:12:48.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:12:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:12:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29678.803783 secs 2023-03-09T23:12:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:12:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:12:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29678.803783 secs 2023-03-09T23:12:48.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:12:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:12:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29678.803783 secs 2023-03-09T23:12:48.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30140.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:12:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:49 smithi122 ceph-mon[93869]: pgmap v16340: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 964 KiB/s rd, 964 KiB/s wr, 11 op/s 2023-03-09T23:12:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:12:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:12:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:49 smithi181 ceph-mon[94189]: pgmap v16340: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 964 KiB/s rd, 964 KiB/s wr, 11 op/s 2023-03-09T23:12:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:12:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:12:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:49 smithi038 ceph-mon[92422]: pgmap v16340: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 964 KiB/s rd, 964 KiB/s wr, 11 op/s 2023-03-09T23:12:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:12:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:12:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29681.407636 secs 2023-03-09T23:12:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29681.407636 secs 2023-03-09T23:12:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29681.407636 secs 2023-03-09T23:12:51.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:51 smithi122 ceph-mon[93869]: pgmap v16341: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 639 KiB/s rd, 628 KiB/s wr, 10 op/s 2023-03-09T23:12:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:51 smithi181 ceph-mon[94189]: pgmap v16341: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 639 KiB/s rd, 628 KiB/s wr, 10 op/s 2023-03-09T23:12:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:51 smithi038 ceph-mon[92422]: pgmap v16341: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 639 KiB/s rd, 628 KiB/s wr, 10 op/s 2023-03-09T23:12:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:12:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:12:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:12:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:12:52.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:12:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:12:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:53 smithi122 ceph-mon[93869]: pgmap v16342: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 787 KiB/s rd, 772 KiB/s wr, 14 op/s 2023-03-09T23:12:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29683.803936 secs 2023-03-09T23:12:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:53 smithi181 ceph-mon[94189]: pgmap v16342: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 787 KiB/s rd, 772 KiB/s wr, 14 op/s 2023-03-09T23:12:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29683.803936 secs 2023-03-09T23:12:53.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:53 smithi038 ceph-mon[92422]: pgmap v16342: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 787 KiB/s rd, 772 KiB/s wr, 14 op/s 2023-03-09T23:12:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29683.803936 secs 2023-03-09T23:12:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30145.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:12:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:55 smithi122 ceph-mon[93869]: pgmap v16343: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 780 KiB/s rd, 765 KiB/s wr, 12 op/s 2023-03-09T23:12:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29686.407697 secs 2023-03-09T23:12:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:55 smithi181 ceph-mon[94189]: pgmap v16343: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 780 KiB/s rd, 765 KiB/s wr, 12 op/s 2023-03-09T23:12:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29686.407697 secs 2023-03-09T23:12:55.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:55 smithi038 ceph-mon[92422]: pgmap v16343: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 780 KiB/s rd, 765 KiB/s wr, 12 op/s 2023-03-09T23:12:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29686.407697 secs 2023-03-09T23:12:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:12:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:12:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:12:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:12:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:12:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:12:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:57 smithi122 ceph-mon[93869]: pgmap v16344: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 531 KiB/s rd, 479 KiB/s wr, 14 op/s 2023-03-09T23:12:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:12:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:12:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:57 smithi181 ceph-mon[94189]: pgmap v16344: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 531 KiB/s rd, 479 KiB/s wr, 14 op/s 2023-03-09T23:12:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:12:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:12:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:57 smithi038 ceph-mon[92422]: pgmap v16344: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 531 KiB/s rd, 479 KiB/s wr, 14 op/s 2023-03-09T23:12:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:12:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:12:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29688.804070 secs 2023-03-09T23:12:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29688.804070 secs 2023-03-09T23:12:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29688.804070 secs 2023-03-09T23:12:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30150.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:12:59.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:12:59 smithi122 ceph-mon[93869]: pgmap v16345: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 539 KiB/s rd, 489 KiB/s wr, 12 op/s 2023-03-09T23:12:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:12:59 smithi181 ceph-mon[94189]: pgmap v16345: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 539 KiB/s rd, 489 KiB/s wr, 12 op/s 2023-03-09T23:12:59.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:12:59 smithi038 ceph-mon[92422]: pgmap v16345: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 539 KiB/s rd, 489 KiB/s wr, 12 op/s 2023-03-09T23:13:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29691.407757 secs 2023-03-09T23:13:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29691.407757 secs 2023-03-09T23:13:00.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29691.407757 secs 2023-03-09T23:13:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:01 smithi122 ceph-mon[93869]: pgmap v16346: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 503 KiB/s rd, 488 KiB/s wr, 11 op/s 2023-03-09T23:13:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:13:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:13:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:01 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:13:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:01 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:13:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:01 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:13:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:01 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:13:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:01 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:13:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:01 smithi181 ceph-mon[94189]: pgmap v16346: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 503 KiB/s rd, 488 KiB/s wr, 11 op/s 2023-03-09T23:13:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:13:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:13:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:01 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:13:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:01 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:13:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:01 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:13:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:01 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:13:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:01 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:13:01.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:01 smithi038 ceph-mon[92422]: pgmap v16346: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 503 KiB/s rd, 488 KiB/s wr, 11 op/s 2023-03-09T23:13:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:13:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:13:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:01 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:13:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:01 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:13:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:01 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:13:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:01 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:13:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:01 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:13:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:13:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:13:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:13:02.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:13:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:13:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:13:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:13:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:13:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:13:03.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:03 smithi122 ceph-mon[93869]: pgmap v16347: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 850 KiB/s rd, 914 KiB/s wr, 16 op/s 2023-03-09T23:13:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29693.804180 secs 2023-03-09T23:13:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:03 smithi181 ceph-mon[94189]: pgmap v16347: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 850 KiB/s rd, 914 KiB/s wr, 16 op/s 2023-03-09T23:13:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29693.804180 secs 2023-03-09T23:13:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:03 smithi038 ceph-mon[92422]: pgmap v16347: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 850 KiB/s rd, 914 KiB/s wr, 16 op/s 2023-03-09T23:13:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29693.804180 secs 2023-03-09T23:13:03.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30155.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:13:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:05 smithi122 ceph-mon[93869]: pgmap v16348: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 709 KiB/s rd, 792 KiB/s wr, 24 op/s 2023-03-09T23:13:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29696.407817 secs 2023-03-09T23:13:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:05 smithi181 ceph-mon[94189]: pgmap v16348: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 709 KiB/s rd, 792 KiB/s wr, 24 op/s 2023-03-09T23:13:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29696.407817 secs 2023-03-09T23:13:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:05 smithi038 ceph-mon[92422]: pgmap v16348: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 709 KiB/s rd, 792 KiB/s wr, 24 op/s 2023-03-09T23:13:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29696.407817 secs 2023-03-09T23:13:07.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:07 smithi122 ceph-mon[93869]: pgmap v16349: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 793 KiB/s rd, 826 KiB/s wr, 28 op/s 2023-03-09T23:13:07.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:07 smithi181 ceph-mon[94189]: pgmap v16349: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 793 KiB/s rd, 826 KiB/s wr, 28 op/s 2023-03-09T23:13:07.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:07 smithi038 ceph-mon[92422]: pgmap v16349: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 793 KiB/s rd, 826 KiB/s wr, 28 op/s 2023-03-09T23:13:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:13:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:13:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29698.804301 secs 2023-03-09T23:13:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:13:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:13:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29698.804301 secs 2023-03-09T23:13:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:13:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:13:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29698.804301 secs 2023-03-09T23:13:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30160.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:13:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:09 smithi122 ceph-mon[93869]: pgmap v16350: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 737 KiB/s rd, 784 KiB/s wr, 26 op/s 2023-03-09T23:13:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:13:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:13:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:09 smithi181 ceph-mon[94189]: pgmap v16350: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 737 KiB/s rd, 784 KiB/s wr, 26 op/s 2023-03-09T23:13:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:13:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:13:09.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:09 smithi038 ceph-mon[92422]: pgmap v16350: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 737 KiB/s rd, 784 KiB/s wr, 26 op/s 2023-03-09T23:13:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:13:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:13:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:10 smithi122 ceph-mon[93869]: osdmap e1237: 12 total, 12 up, 12 in 2023-03-09T23:13:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29701.407876 secs 2023-03-09T23:13:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:10 smithi181 ceph-mon[94189]: osdmap e1237: 12 total, 12 up, 12 in 2023-03-09T23:13:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29701.407876 secs 2023-03-09T23:13:10.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:10 smithi038 ceph-mon[92422]: osdmap e1237: 12 total, 12 up, 12 in 2023-03-09T23:13:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29701.407876 secs 2023-03-09T23:13:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:11 smithi122 ceph-mon[93869]: pgmap v16352: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 538 KiB/s rd, 594 KiB/s wr, 29 op/s 2023-03-09T23:13:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:11 smithi181 ceph-mon[94189]: pgmap v16352: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 538 KiB/s rd, 594 KiB/s wr, 29 op/s 2023-03-09T23:13:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:11 smithi038 ceph-mon[92422]: pgmap v16352: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 538 KiB/s rd, 594 KiB/s wr, 29 op/s 2023-03-09T23:13:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:13 smithi122 ceph-mon[93869]: pgmap v16353: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 424 KiB/s wr, 26 op/s 2023-03-09T23:13:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29703.804368 secs 2023-03-09T23:13:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:13 smithi181 ceph-mon[94189]: pgmap v16353: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 424 KiB/s wr, 26 op/s 2023-03-09T23:13:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29703.804368 secs 2023-03-09T23:13:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:13 smithi038 ceph-mon[92422]: pgmap v16353: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 424 KiB/s wr, 26 op/s 2023-03-09T23:13:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29703.804368 secs 2023-03-09T23:13:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30165.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:13:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:15 smithi122 ceph-mon[93869]: pgmap v16354: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 605 KiB/s rd, 546 KiB/s wr, 13 op/s 2023-03-09T23:13:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29706.407935 secs 2023-03-09T23:13:15.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:15 smithi181 ceph-mon[94189]: pgmap v16354: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 605 KiB/s rd, 546 KiB/s wr, 13 op/s 2023-03-09T23:13:15.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29706.407935 secs 2023-03-09T23:13:15.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:15 smithi038 ceph-mon[92422]: pgmap v16354: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 605 KiB/s rd, 546 KiB/s wr, 13 op/s 2023-03-09T23:13:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29706.407935 secs 2023-03-09T23:13:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:13:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:13:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:13:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:13:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:13:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:13:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:17 smithi122 ceph-mon[93869]: pgmap v16355: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 507 KiB/s rd, 505 KiB/s wr, 10 op/s 2023-03-09T23:13:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:13:17.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:13:17.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:17 smithi122 ceph-mon[93869]: osdmap e1238: 12 total, 12 up, 12 in 2023-03-09T23:13:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:17 smithi181 ceph-mon[94189]: pgmap v16355: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 507 KiB/s rd, 505 KiB/s wr, 10 op/s 2023-03-09T23:13:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:13:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:13:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:17 smithi181 ceph-mon[94189]: osdmap e1238: 12 total, 12 up, 12 in 2023-03-09T23:13:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:17 smithi038 ceph-mon[92422]: pgmap v16355: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 507 KiB/s rd, 505 KiB/s wr, 10 op/s 2023-03-09T23:13:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:13:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:13:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:17 smithi038 ceph-mon[92422]: osdmap e1238: 12 total, 12 up, 12 in 2023-03-09T23:13:18.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29708.804439 secs 2023-03-09T23:13:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29708.804439 secs 2023-03-09T23:13:18.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29708.804439 secs 2023-03-09T23:13:18.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30170.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:13:19.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:19 smithi122 ceph-mon[93869]: pgmap v16357: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 583 KiB/s rd, 562 KiB/s wr, 11 op/s 2023-03-09T23:13:19.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:19 smithi181 ceph-mon[94189]: pgmap v16357: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 583 KiB/s rd, 562 KiB/s wr, 11 op/s 2023-03-09T23:13:19.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:19 smithi038 ceph-mon[92422]: pgmap v16357: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 583 KiB/s rd, 562 KiB/s wr, 11 op/s 2023-03-09T23:13:20.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29711.407997 secs 2023-03-09T23:13:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29711.407997 secs 2023-03-09T23:13:20.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29711.407997 secs 2023-03-09T23:13:21.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:21 smithi122 ceph-mon[93869]: pgmap v16358: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 508 KiB/s rd, 491 KiB/s wr, 9 op/s 2023-03-09T23:13:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:21 smithi181 ceph-mon[94189]: pgmap v16358: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 508 KiB/s rd, 491 KiB/s wr, 9 op/s 2023-03-09T23:13:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:21 smithi038 ceph-mon[92422]: pgmap v16358: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 508 KiB/s rd, 491 KiB/s wr, 9 op/s 2023-03-09T23:13:23.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:23 smithi122 ceph-mon[93869]: pgmap v16359: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 150 KiB/s wr, 9 op/s 2023-03-09T23:13:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29713.804565 secs 2023-03-09T23:13:23.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:23 smithi181 ceph-mon[94189]: pgmap v16359: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 150 KiB/s wr, 9 op/s 2023-03-09T23:13:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29713.804565 secs 2023-03-09T23:13:23.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:23 smithi038 ceph-mon[92422]: pgmap v16359: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 150 KiB/s wr, 9 op/s 2023-03-09T23:13:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29713.804565 secs 2023-03-09T23:13:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30175.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:13:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:25 smithi122 ceph-mon[93869]: pgmap v16360: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 248 KiB/s wr, 8 op/s 2023-03-09T23:13:25.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:25 smithi181 ceph-mon[94189]: pgmap v16360: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 248 KiB/s wr, 8 op/s 2023-03-09T23:13:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:25 smithi038 ceph-mon[92422]: pgmap v16360: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 248 KiB/s wr, 8 op/s 2023-03-09T23:13:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29716.408056 secs 2023-03-09T23:13:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29716.408056 secs 2023-03-09T23:13:26.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29716.408056 secs 2023-03-09T23:13:27.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:27 smithi122 ceph-mon[93869]: pgmap v16361: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 248 KiB/s wr, 8 op/s 2023-03-09T23:13:27.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:27 smithi181 ceph-mon[94189]: pgmap v16361: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 248 KiB/s wr, 8 op/s 2023-03-09T23:13:27.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:27 smithi038 ceph-mon[92422]: pgmap v16361: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 248 KiB/s wr, 8 op/s 2023-03-09T23:13:28.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29718.804692 secs 2023-03-09T23:13:28.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29718.804692 secs 2023-03-09T23:13:28.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29718.804692 secs 2023-03-09T23:13:28.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30180.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:13:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:29 smithi122 ceph-mon[93869]: pgmap v16362: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 259 KiB/s wr, 8 op/s 2023-03-09T23:13:29.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:29 smithi181 ceph-mon[94189]: pgmap v16362: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 259 KiB/s wr, 8 op/s 2023-03-09T23:13:29.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:29 smithi038 ceph-mon[92422]: pgmap v16362: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 259 KiB/s wr, 8 op/s 2023-03-09T23:13:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29721.408115 secs 2023-03-09T23:13:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:31 smithi122 ceph-mon[93869]: pgmap v16363: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 237 KiB/s wr, 6 op/s 2023-03-09T23:13:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29721.408115 secs 2023-03-09T23:13:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:31 smithi181 ceph-mon[94189]: pgmap v16363: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 237 KiB/s wr, 6 op/s 2023-03-09T23:13:31.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29721.408115 secs 2023-03-09T23:13:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:31 smithi038 ceph-mon[92422]: pgmap v16363: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 237 KiB/s wr, 6 op/s 2023-03-09T23:13:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:33 smithi122 ceph-mon[93869]: pgmap v16364: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 237 KiB/s wr, 10 op/s 2023-03-09T23:13:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29723.804813 secs 2023-03-09T23:13:33.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:33 smithi181 ceph-mon[94189]: pgmap v16364: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 237 KiB/s wr, 10 op/s 2023-03-09T23:13:33.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29723.804813 secs 2023-03-09T23:13:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:33 smithi038 ceph-mon[92422]: pgmap v16364: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 237 KiB/s wr, 10 op/s 2023-03-09T23:13:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29723.804813 secs 2023-03-09T23:13:33.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30185.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:13:35.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:35 smithi122 ceph-mon[93869]: pgmap v16365: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 237 KiB/s wr, 7 op/s 2023-03-09T23:13:35.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:35 smithi181 ceph-mon[94189]: pgmap v16365: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 237 KiB/s wr, 7 op/s 2023-03-09T23:13:35.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:35 smithi038 ceph-mon[92422]: pgmap v16365: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 237 KiB/s wr, 7 op/s 2023-03-09T23:13:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29726.408177 secs 2023-03-09T23:13:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:13:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:13:36.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29726.408177 secs 2023-03-09T23:13:36.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:13:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:13:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29726.408177 secs 2023-03-09T23:13:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:13:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:13:37.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:37 smithi122 ceph-mon[93869]: pgmap v16366: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 32 KiB/s wr, 9 op/s 2023-03-09T23:13:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:13:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:13:37.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:37 smithi181 ceph-mon[94189]: pgmap v16366: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 32 KiB/s wr, 9 op/s 2023-03-09T23:13:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:13:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:13:37.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:37 smithi038 ceph-mon[92422]: pgmap v16366: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 32 KiB/s wr, 9 op/s 2023-03-09T23:13:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:13:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:13:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29728.804886 secs 2023-03-09T23:13:38.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29728.804886 secs 2023-03-09T23:13:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29728.804886 secs 2023-03-09T23:13:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30190.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:13:39.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:39 smithi122 ceph-mon[93869]: pgmap v16367: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 282 KiB/s wr, 8 op/s 2023-03-09T23:13:39.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:39 smithi181 ceph-mon[94189]: pgmap v16367: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 282 KiB/s wr, 8 op/s 2023-03-09T23:13:39.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:39 smithi038 ceph-mon[92422]: pgmap v16367: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 282 KiB/s wr, 8 op/s 2023-03-09T23:13:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29731.408239 secs 2023-03-09T23:13:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:41 smithi122 ceph-mon[93869]: pgmap v16368: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 252 KiB/s wr, 7 op/s 2023-03-09T23:13:41.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29731.408239 secs 2023-03-09T23:13:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:41 smithi181 ceph-mon[94189]: pgmap v16368: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 252 KiB/s wr, 7 op/s 2023-03-09T23:13:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29731.408239 secs 2023-03-09T23:13:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:41 smithi038 ceph-mon[92422]: pgmap v16368: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 252 KiB/s wr, 7 op/s 2023-03-09T23:13:43.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:43 smithi122 ceph-mon[93869]: pgmap v16369: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 357 KiB/s rd, 312 KiB/s wr, 10 op/s 2023-03-09T23:13:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29733.804951 secs 2023-03-09T23:13:43.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:43 smithi181 ceph-mon[94189]: pgmap v16369: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 357 KiB/s rd, 312 KiB/s wr, 10 op/s 2023-03-09T23:13:43.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29733.804951 secs 2023-03-09T23:13:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:43 smithi038 ceph-mon[92422]: pgmap v16369: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 357 KiB/s rd, 312 KiB/s wr, 10 op/s 2023-03-09T23:13:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29733.804951 secs 2023-03-09T23:13:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30195.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:13:45.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:45 smithi122 ceph-mon[93869]: pgmap v16370: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 311 KiB/s wr, 8 op/s 2023-03-09T23:13:45.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:45 smithi181 ceph-mon[94189]: pgmap v16370: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 311 KiB/s wr, 8 op/s 2023-03-09T23:13:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:45 smithi038 ceph-mon[92422]: pgmap v16370: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 311 KiB/s wr, 8 op/s 2023-03-09T23:13:46.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29736.408298 secs 2023-03-09T23:13:46.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29736.408298 secs 2023-03-09T23:13:46.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29736.408298 secs 2023-03-09T23:13:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:47 smithi122 ceph-mon[93869]: pgmap v16371: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 357 KiB/s rd, 311 KiB/s wr, 10 op/s 2023-03-09T23:13:47.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:47 smithi181 ceph-mon[94189]: pgmap v16371: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 357 KiB/s rd, 311 KiB/s wr, 10 op/s 2023-03-09T23:13:47.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:47 smithi038 ceph-mon[92422]: pgmap v16371: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 357 KiB/s rd, 311 KiB/s wr, 10 op/s 2023-03-09T23:13:48.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29738.805069 secs 2023-03-09T23:13:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29738.805069 secs 2023-03-09T23:13:48.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29738.805069 secs 2023-03-09T23:13:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30200.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:13:49.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:49 smithi122 ceph-mon[93869]: pgmap v16372: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 583 KiB/s rd, 412 KiB/s wr, 9 op/s 2023-03-09T23:13:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:13:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:13:49.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:49 smithi181 ceph-mon[94189]: pgmap v16372: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 583 KiB/s rd, 412 KiB/s wr, 9 op/s 2023-03-09T23:13:49.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:13:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:13:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:49 smithi038 ceph-mon[92422]: pgmap v16372: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 583 KiB/s rd, 412 KiB/s wr, 9 op/s 2023-03-09T23:13:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:13:49.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:13:51.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29741.408361 secs 2023-03-09T23:13:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:51 smithi122 ceph-mon[93869]: pgmap v16373: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 162 KiB/s wr, 7 op/s 2023-03-09T23:13:51.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29741.408361 secs 2023-03-09T23:13:51.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:51 smithi181 ceph-mon[94189]: pgmap v16373: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 162 KiB/s wr, 7 op/s 2023-03-09T23:13:51.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29741.408361 secs 2023-03-09T23:13:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:51 smithi038 ceph-mon[92422]: pgmap v16373: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 162 KiB/s wr, 7 op/s 2023-03-09T23:13:52.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:13:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:13:52.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:13:52.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:13:52.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:13:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:13:53.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:53 smithi122 ceph-mon[93869]: pgmap v16374: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 306 KiB/s wr, 17 op/s 2023-03-09T23:13:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29743.805117 secs 2023-03-09T23:13:53.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:53 smithi181 ceph-mon[94189]: pgmap v16374: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 306 KiB/s wr, 17 op/s 2023-03-09T23:13:53.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29743.805117 secs 2023-03-09T23:13:53.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:53 smithi038 ceph-mon[92422]: pgmap v16374: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 306 KiB/s wr, 17 op/s 2023-03-09T23:13:53.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29743.805117 secs 2023-03-09T23:13:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30205.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:13:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:55 smithi122 ceph-mon[93869]: pgmap v16375: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 514 KiB/s rd, 486 KiB/s wr, 15 op/s 2023-03-09T23:13:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:55 smithi181 ceph-mon[94189]: pgmap v16375: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 514 KiB/s rd, 486 KiB/s wr, 15 op/s 2023-03-09T23:13:55.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:55 smithi038 ceph-mon[92422]: pgmap v16375: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 514 KiB/s rd, 486 KiB/s wr, 15 op/s 2023-03-09T23:13:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29746.408445 secs 2023-03-09T23:13:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:13:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:13:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29746.408445 secs 2023-03-09T23:13:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:13:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:13:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29746.408445 secs 2023-03-09T23:13:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:13:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:13:57.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:57 smithi122 ceph-mon[93869]: pgmap v16376: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 913 KiB/s rd, 736 KiB/s wr, 50 op/s 2023-03-09T23:13:57.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:13:57.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:13:57.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:57 smithi181 ceph-mon[94189]: pgmap v16376: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 913 KiB/s rd, 736 KiB/s wr, 50 op/s 2023-03-09T23:13:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:13:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:13:57.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:57 smithi038 ceph-mon[92422]: pgmap v16376: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 913 KiB/s rd, 736 KiB/s wr, 50 op/s 2023-03-09T23:13:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:13:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:13:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29748.805250 secs 2023-03-09T23:13:58.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29748.805250 secs 2023-03-09T23:13:58.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29748.805250 secs 2023-03-09T23:13:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30210.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:13:59.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:13:59 smithi122 ceph-mon[93869]: pgmap v16377: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1022 KiB/s wr, 50 op/s 2023-03-09T23:13:59.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:13:59 smithi181 ceph-mon[94189]: pgmap v16377: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1022 KiB/s wr, 50 op/s 2023-03-09T23:13:59.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:13:59 smithi038 ceph-mon[92422]: pgmap v16377: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1022 KiB/s wr, 50 op/s 2023-03-09T23:14:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:14:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:14:00.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:14:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:14:00.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:14:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:14:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29751.408502 secs 2023-03-09T23:14:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:01 smithi122 ceph-mon[93869]: pgmap v16378: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 970 KiB/s rd, 921 KiB/s wr, 48 op/s 2023-03-09T23:14:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:14:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:14:01.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29751.408502 secs 2023-03-09T23:14:01.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:01 smithi181 ceph-mon[94189]: pgmap v16378: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 970 KiB/s rd, 921 KiB/s wr, 48 op/s 2023-03-09T23:14:01.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:14:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:14:01.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29751.408502 secs 2023-03-09T23:14:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:01 smithi038 ceph-mon[92422]: pgmap v16378: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 970 KiB/s rd, 921 KiB/s wr, 48 op/s 2023-03-09T23:14:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:14:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:14:02.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:14:02.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:14:02.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:14:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:03 smithi122 ceph-mon[93869]: pgmap v16379: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 55 op/s 2023-03-09T23:14:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29753.805301 secs 2023-03-09T23:14:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:03 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:14:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:03 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:14:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:03 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:14:03.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:03 smithi181 ceph-mon[94189]: pgmap v16379: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 55 op/s 2023-03-09T23:14:03.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29753.805301 secs 2023-03-09T23:14:03.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:03 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:14:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:03 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:14:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:03 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:14:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:03 smithi038 ceph-mon[92422]: pgmap v16379: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 55 op/s 2023-03-09T23:14:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29753.805301 secs 2023-03-09T23:14:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:03 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:14:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:03 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:14:03.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:03 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:14:03.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30215.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:14:05.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:05 smithi122 ceph-mon[93869]: pgmap v16380: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 58 op/s 2023-03-09T23:14:05.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:05 smithi181 ceph-mon[94189]: pgmap v16380: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 58 op/s 2023-03-09T23:14:05.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:05 smithi038 ceph-mon[92422]: pgmap v16380: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 58 op/s 2023-03-09T23:14:06.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29756.408562 secs 2023-03-09T23:14:06.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29756.408562 secs 2023-03-09T23:14:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29756.408562 secs 2023-03-09T23:14:07.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:07 smithi122 ceph-mon[93869]: pgmap v16381: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 850 KiB/s wr, 61 op/s 2023-03-09T23:14:07.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:07 smithi181 ceph-mon[94189]: pgmap v16381: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 850 KiB/s wr, 61 op/s 2023-03-09T23:14:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:07 smithi038 ceph-mon[92422]: pgmap v16381: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 850 KiB/s wr, 61 op/s 2023-03-09T23:14:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:14:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:14:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29758.805379 secs 2023-03-09T23:14:08.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:14:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:14:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29758.805379 secs 2023-03-09T23:14:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:14:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:14:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29758.805379 secs 2023-03-09T23:14:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30220.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:14:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:09 smithi038 ceph-mon[92422]: pgmap v16382: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 645 KiB/s rd, 600 KiB/s wr, 27 op/s 2023-03-09T23:14:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:14:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:14:09.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:09 smithi122 ceph-mon[93869]: pgmap v16382: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 645 KiB/s rd, 600 KiB/s wr, 27 op/s 2023-03-09T23:14:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:14:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:14:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:09 smithi181 ceph-mon[94189]: pgmap v16382: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 645 KiB/s rd, 600 KiB/s wr, 27 op/s 2023-03-09T23:14:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:14:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:14:10.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:10 smithi038 ceph-mon[92422]: osdmap e1239: 12 total, 12 up, 12 in 2023-03-09T23:14:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29761.408622 secs 2023-03-09T23:14:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:10 smithi122 ceph-mon[93869]: osdmap e1239: 12 total, 12 up, 12 in 2023-03-09T23:14:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29761.408622 secs 2023-03-09T23:14:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:10 smithi181 ceph-mon[94189]: osdmap e1239: 12 total, 12 up, 12 in 2023-03-09T23:14:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29761.408622 secs 2023-03-09T23:14:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:11 smithi038 ceph-mon[92422]: pgmap v16384: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 376 KiB/s wr, 30 op/s 2023-03-09T23:14:11.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:11 smithi122 ceph-mon[93869]: pgmap v16384: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 376 KiB/s wr, 30 op/s 2023-03-09T23:14:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:11 smithi181 ceph-mon[94189]: pgmap v16384: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 376 KiB/s wr, 30 op/s 2023-03-09T23:14:13.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:13 smithi122 ceph-mon[93869]: pgmap v16385: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 461 KiB/s wr, 27 op/s 2023-03-09T23:14:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29763.805489 secs 2023-03-09T23:14:13.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:13 smithi181 ceph-mon[94189]: pgmap v16385: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 461 KiB/s wr, 27 op/s 2023-03-09T23:14:13.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29763.805489 secs 2023-03-09T23:14:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:13 smithi038 ceph-mon[92422]: pgmap v16385: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 461 KiB/s wr, 27 op/s 2023-03-09T23:14:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29763.805489 secs 2023-03-09T23:14:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30225.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:14:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:15 smithi122 ceph-mon[93869]: pgmap v16386: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 369 KiB/s wr, 11 op/s 2023-03-09T23:14:15.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:15 smithi181 ceph-mon[94189]: pgmap v16386: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 369 KiB/s wr, 11 op/s 2023-03-09T23:14:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:15 smithi038 ceph-mon[92422]: pgmap v16386: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 369 KiB/s wr, 11 op/s 2023-03-09T23:14:16.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29766.408687 secs 2023-03-09T23:14:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:14:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:14:16.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29766.408687 secs 2023-03-09T23:14:16.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:14:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:14:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29766.408687 secs 2023-03-09T23:14:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:14:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:14:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:17 smithi122 ceph-mon[93869]: pgmap v16387: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 355 KiB/s wr, 10 op/s 2023-03-09T23:14:17.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:14:17.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:14:17.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:17 smithi122 ceph-mon[93869]: osdmap e1240: 12 total, 12 up, 12 in 2023-03-09T23:14:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:17 smithi181 ceph-mon[94189]: pgmap v16387: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 355 KiB/s wr, 10 op/s 2023-03-09T23:14:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:14:17.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:14:17.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:17 smithi181 ceph-mon[94189]: osdmap e1240: 12 total, 12 up, 12 in 2023-03-09T23:14:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:17 smithi038 ceph-mon[92422]: pgmap v16387: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 355 KiB/s wr, 10 op/s 2023-03-09T23:14:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:14:17.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:14:17.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:17 smithi038 ceph-mon[92422]: osdmap e1240: 12 total, 12 up, 12 in 2023-03-09T23:14:18.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29768.805606 secs 2023-03-09T23:14:18.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29768.805606 secs 2023-03-09T23:14:18.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29768.805606 secs 2023-03-09T23:14:18.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30230.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:14:19.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:19 smithi122 ceph-mon[93869]: pgmap v16389: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 524 KiB/s rd, 521 KiB/s wr, 10 op/s 2023-03-09T23:14:19.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:19 smithi181 ceph-mon[94189]: pgmap v16389: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 524 KiB/s rd, 521 KiB/s wr, 10 op/s 2023-03-09T23:14:19.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:19 smithi038 ceph-mon[92422]: pgmap v16389: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 524 KiB/s rd, 521 KiB/s wr, 10 op/s 2023-03-09T23:14:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29771.408747 secs 2023-03-09T23:14:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:21 smithi122 ceph-mon[93869]: pgmap v16390: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 464 KiB/s rd, 461 KiB/s wr, 9 op/s 2023-03-09T23:14:21.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29771.408747 secs 2023-03-09T23:14:21.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:21 smithi181 ceph-mon[94189]: pgmap v16390: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 464 KiB/s rd, 461 KiB/s wr, 9 op/s 2023-03-09T23:14:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29771.408747 secs 2023-03-09T23:14:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:21 smithi038 ceph-mon[92422]: pgmap v16390: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 464 KiB/s rd, 461 KiB/s wr, 9 op/s 2023-03-09T23:14:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:23 smithi122 ceph-mon[93869]: pgmap v16391: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 107 KiB/s wr, 7 op/s 2023-03-09T23:14:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29773.805708 secs 2023-03-09T23:14:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:23 smithi181 ceph-mon[94189]: pgmap v16391: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 107 KiB/s wr, 7 op/s 2023-03-09T23:14:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29773.805708 secs 2023-03-09T23:14:23.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:23 smithi038 ceph-mon[92422]: pgmap v16391: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 107 KiB/s wr, 7 op/s 2023-03-09T23:14:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29773.805708 secs 2023-03-09T23:14:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30235.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:14:25.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:25 smithi122 ceph-mon[93869]: pgmap v16392: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 107 KiB/s wr, 7 op/s 2023-03-09T23:14:25.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:25 smithi181 ceph-mon[94189]: pgmap v16392: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 107 KiB/s wr, 7 op/s 2023-03-09T23:14:25.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:25 smithi038 ceph-mon[92422]: pgmap v16392: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 107 KiB/s wr, 7 op/s 2023-03-09T23:14:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29776.408807 secs 2023-03-09T23:14:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29776.408807 secs 2023-03-09T23:14:26.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29776.408807 secs 2023-03-09T23:14:27.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:27 smithi122 ceph-mon[93869]: pgmap v16393: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 107 KiB/s wr, 7 op/s 2023-03-09T23:14:27.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:27 smithi181 ceph-mon[94189]: pgmap v16393: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 107 KiB/s wr, 7 op/s 2023-03-09T23:14:27.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:27 smithi038 ceph-mon[92422]: pgmap v16393: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 107 KiB/s wr, 7 op/s 2023-03-09T23:14:28.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29778.805831 secs 2023-03-09T23:14:28.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29778.805831 secs 2023-03-09T23:14:28.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29778.805831 secs 2023-03-09T23:14:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30240.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:14:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:29 smithi122 ceph-mon[93869]: pgmap v16394: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 372 B/s wr, 6 op/s 2023-03-09T23:14:29.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:29 smithi181 ceph-mon[94189]: pgmap v16394: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 372 B/s wr, 6 op/s 2023-03-09T23:14:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:29 smithi038 ceph-mon[92422]: pgmap v16394: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 372 B/s wr, 6 op/s 2023-03-09T23:14:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29781.408892 secs 2023-03-09T23:14:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:31 smithi122 ceph-mon[93869]: pgmap v16395: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2023-03-09T23:14:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29781.408892 secs 2023-03-09T23:14:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:31 smithi181 ceph-mon[94189]: pgmap v16395: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2023-03-09T23:14:31.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29781.408892 secs 2023-03-09T23:14:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:31 smithi038 ceph-mon[92422]: pgmap v16395: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 5 op/s 2023-03-09T23:14:33.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:33 smithi122 ceph-mon[93869]: pgmap v16396: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 130 KiB/s wr, 8 op/s 2023-03-09T23:14:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29783.805975 secs 2023-03-09T23:14:33.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:33 smithi181 ceph-mon[94189]: pgmap v16396: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 130 KiB/s wr, 8 op/s 2023-03-09T23:14:33.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29783.805975 secs 2023-03-09T23:14:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:33 smithi038 ceph-mon[92422]: pgmap v16396: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 130 KiB/s wr, 8 op/s 2023-03-09T23:14:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29783.805975 secs 2023-03-09T23:14:33.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30245.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:14:35.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:35 smithi122 ceph-mon[93869]: pgmap v16397: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 130 KiB/s wr, 6 op/s 2023-03-09T23:14:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:35 smithi181 ceph-mon[94189]: pgmap v16397: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 130 KiB/s wr, 6 op/s 2023-03-09T23:14:35.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:35 smithi038 ceph-mon[92422]: pgmap v16397: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 130 KiB/s wr, 6 op/s 2023-03-09T23:14:36.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29786.408954 secs 2023-03-09T23:14:36.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29786.408954 secs 2023-03-09T23:14:36.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29786.408954 secs 2023-03-09T23:14:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:37 smithi122 ceph-mon[93869]: pgmap v16398: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 226 KiB/s wr, 8 op/s 2023-03-09T23:14:37.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:37 smithi181 ceph-mon[94189]: pgmap v16398: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 226 KiB/s wr, 8 op/s 2023-03-09T23:14:37.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:37 smithi038 ceph-mon[92422]: pgmap v16398: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 226 KiB/s wr, 8 op/s 2023-03-09T23:14:38.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29788.806133 secs 2023-03-09T23:14:38.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29788.806133 secs 2023-03-09T23:14:38.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29788.806133 secs 2023-03-09T23:14:38.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30250.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:14:39.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:39 smithi122 ceph-mon[93869]: pgmap v16399: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 252 KiB/s wr, 7 op/s 2023-03-09T23:14:39.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:39 smithi181 ceph-mon[94189]: pgmap v16399: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 252 KiB/s wr, 7 op/s 2023-03-09T23:14:39.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:39 smithi038 ceph-mon[92422]: pgmap v16399: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 252 KiB/s wr, 7 op/s 2023-03-09T23:14:40.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:14:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:14:40.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:14:40.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:14:40.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:14:40.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:14:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29791.408983 secs 2023-03-09T23:14:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:41 smithi122 ceph-mon[93869]: pgmap v16400: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 252 KiB/s wr, 6 op/s 2023-03-09T23:14:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:14:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:14:41.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29791.408983 secs 2023-03-09T23:14:41.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:41 smithi181 ceph-mon[94189]: pgmap v16400: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 252 KiB/s wr, 6 op/s 2023-03-09T23:14:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:14:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:14:41.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29791.408983 secs 2023-03-09T23:14:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:41 smithi038 ceph-mon[92422]: pgmap v16400: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 252 KiB/s wr, 6 op/s 2023-03-09T23:14:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:14:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:14:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:43 smithi122 ceph-mon[93869]: pgmap v16401: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 384 KiB/s rd, 497 KiB/s wr, 10 op/s 2023-03-09T23:14:43.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:43 smithi181 ceph-mon[94189]: pgmap v16401: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 384 KiB/s rd, 497 KiB/s wr, 10 op/s 2023-03-09T23:14:43.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:43 smithi038 ceph-mon[92422]: pgmap v16401: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 384 KiB/s rd, 497 KiB/s wr, 10 op/s 2023-03-09T23:14:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30255.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:14:45.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:45 smithi122 ceph-mon[93869]: pgmap v16402: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 368 KiB/s wr, 7 op/s 2023-03-09T23:14:45.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:45 smithi181 ceph-mon[94189]: pgmap v16402: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 368 KiB/s wr, 7 op/s 2023-03-09T23:14:45.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:45 smithi038 ceph-mon[92422]: pgmap v16402: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 368 KiB/s wr, 7 op/s 2023-03-09T23:14:46.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29796.409043 secs 2023-03-09T23:14:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:46 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:14:46.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29796.409043 secs 2023-03-09T23:14:46.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:46 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:14:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29796.409043 secs 2023-03-09T23:14:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:46 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:14:47.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:47 smithi122 ceph-mon[93869]: pgmap v16403: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 763 KiB/s rd, 368 KiB/s wr, 10 op/s 2023-03-09T23:14:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:47 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:14:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:14:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:14:47.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:47 smithi181 ceph-mon[94189]: pgmap v16403: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 763 KiB/s rd, 368 KiB/s wr, 10 op/s 2023-03-09T23:14:47.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:47 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:14:47.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:14:47.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:14:47.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:47 smithi038 ceph-mon[92422]: pgmap v16403: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 763 KiB/s rd, 368 KiB/s wr, 10 op/s 2023-03-09T23:14:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:47 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:14:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:14:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:14:48.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30260.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:14:49.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:49 smithi122 ceph-mon[93869]: pgmap v16404: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 807 KiB/s rd, 317 KiB/s wr, 8 op/s 2023-03-09T23:14:49.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:49 smithi181 ceph-mon[94189]: pgmap v16404: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 807 KiB/s rd, 317 KiB/s wr, 8 op/s 2023-03-09T23:14:49.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:49 smithi038 ceph-mon[92422]: pgmap v16404: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 807 KiB/s rd, 317 KiB/s wr, 8 op/s 2023-03-09T23:14:51.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29801.409104 secs 2023-03-09T23:14:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:51 smithi122 ceph-mon[93869]: pgmap v16405: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 781 KiB/s rd, 291 KiB/s wr, 7 op/s 2023-03-09T23:14:51.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29801.409104 secs 2023-03-09T23:14:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:51 smithi181 ceph-mon[94189]: pgmap v16405: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 781 KiB/s rd, 291 KiB/s wr, 7 op/s 2023-03-09T23:14:51.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29801.409104 secs 2023-03-09T23:14:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:51 smithi038 ceph-mon[92422]: pgmap v16405: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 781 KiB/s rd, 291 KiB/s wr, 7 op/s 2023-03-09T23:14:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:53 smithi122 ceph-mon[93869]: pgmap v16406: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 806 KiB/s rd, 315 KiB/s wr, 11 op/s 2023-03-09T23:14:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:14:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:14:53.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:53 smithi181 ceph-mon[94189]: pgmap v16406: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 806 KiB/s rd, 315 KiB/s wr, 11 op/s 2023-03-09T23:14:53.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:14:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:14:53.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:53 smithi038 ceph-mon[92422]: pgmap v16406: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 806 KiB/s rd, 315 KiB/s wr, 11 op/s 2023-03-09T23:14:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:14:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:14:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30265.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:14:55.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:55 smithi122 ceph-mon[93869]: pgmap v16407: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 830 KiB/s rd, 321 KiB/s wr, 8 op/s 2023-03-09T23:14:55.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:55 smithi181 ceph-mon[94189]: pgmap v16407: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 830 KiB/s rd, 321 KiB/s wr, 8 op/s 2023-03-09T23:14:55.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:55 smithi038 ceph-mon[92422]: pgmap v16407: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 830 KiB/s rd, 321 KiB/s wr, 8 op/s 2023-03-09T23:14:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29806.409168 secs 2023-03-09T23:14:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:14:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:14:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29806.409168 secs 2023-03-09T23:14:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:14:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:14:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29806.409168 secs 2023-03-09T23:14:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:14:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:14:57.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:57 smithi122 ceph-mon[93869]: pgmap v16408: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 869 KiB/s rd, 345 KiB/s wr, 12 op/s 2023-03-09T23:14:57.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:57 smithi181 ceph-mon[94189]: pgmap v16408: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 869 KiB/s rd, 345 KiB/s wr, 12 op/s 2023-03-09T23:14:57.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:57 smithi038 ceph-mon[92422]: pgmap v16408: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 869 KiB/s rd, 345 KiB/s wr, 12 op/s 2023-03-09T23:14:58.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:58 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 29808.806617 secs 2023-03-09T23:14:58.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:58 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 29808.806617 secs 2023-03-09T23:14:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:58 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 29808.806617 secs 2023-03-09T23:14:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30270.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:14:59.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:14:59 smithi122 ceph-mon[93869]: pgmap v16409: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 400 KiB/s wr, 11 op/s 2023-03-09T23:14:59.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:14:59 smithi181 ceph-mon[94189]: pgmap v16409: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 400 KiB/s wr, 11 op/s 2023-03-09T23:14:59.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:14:59 smithi038 ceph-mon[92422]: pgmap v16409: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 400 KiB/s wr, 11 op/s 2023-03-09T23:15:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:15:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:15:00.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:15:00.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:15:00.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:15:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:15:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29811.409228 secs 2023-03-09T23:15:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:01 smithi122 ceph-mon[93869]: pgmap v16410: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 499 KiB/s wr, 10 op/s 2023-03-09T23:15:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:15:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:15:01.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29811.409228 secs 2023-03-09T23:15:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:01 smithi181 ceph-mon[94189]: pgmap v16410: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 499 KiB/s wr, 10 op/s 2023-03-09T23:15:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:15:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:15:01.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29811.409228 secs 2023-03-09T23:15:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:01 smithi038 ceph-mon[92422]: pgmap v16410: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 499 KiB/s wr, 10 op/s 2023-03-09T23:15:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:15:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:15:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:03 smithi122 ceph-mon[93869]: pgmap v16411: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 762 KiB/s rd, 855 KiB/s wr, 14 op/s 2023-03-09T23:15:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:03 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:15:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:03 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:15:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:03 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:15:03.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:03 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 29813.806768 secs 2023-03-09T23:15:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:03 smithi181 ceph-mon[94189]: pgmap v16411: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 762 KiB/s rd, 855 KiB/s wr, 14 op/s 2023-03-09T23:15:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:03 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:15:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:03 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:15:03.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:03 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:15:03.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:03 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 29813.806768 secs 2023-03-09T23:15:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:03 smithi038 ceph-mon[92422]: pgmap v16411: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 762 KiB/s rd, 855 KiB/s wr, 14 op/s 2023-03-09T23:15:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:03 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:15:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:03 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:15:03.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:03 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:15:03.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:03 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 29813.806768 secs 2023-03-09T23:15:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30275.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:15:05.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:05 smithi122 ceph-mon[93869]: pgmap v16412: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 743 KiB/s rd, 841 KiB/s wr, 22 op/s 2023-03-09T23:15:05.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:05 smithi181 ceph-mon[94189]: pgmap v16412: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 743 KiB/s rd, 841 KiB/s wr, 22 op/s 2023-03-09T23:15:05.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:05 smithi038 ceph-mon[92422]: pgmap v16412: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 743 KiB/s rd, 841 KiB/s wr, 22 op/s 2023-03-09T23:15:06.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29816.409305 secs 2023-03-09T23:15:06.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29816.409305 secs 2023-03-09T23:15:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29816.409305 secs 2023-03-09T23:15:07.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:07 smithi122 ceph-mon[93869]: pgmap v16413: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 803 KiB/s rd, 784 KiB/s wr, 26 op/s 2023-03-09T23:15:07.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:07 smithi181 ceph-mon[94189]: pgmap v16413: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 803 KiB/s rd, 784 KiB/s wr, 26 op/s 2023-03-09T23:15:07.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:07 smithi038 ceph-mon[92422]: pgmap v16413: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 803 KiB/s rd, 784 KiB/s wr, 26 op/s 2023-03-09T23:15:08.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:15:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:15:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:08 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 29818.806895 secs 2023-03-09T23:15:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:15:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:15:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:08 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 29818.806895 secs 2023-03-09T23:15:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:15:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:15:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:08 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 29818.806895 secs 2023-03-09T23:15:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30280.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:15:09.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:09 smithi122 ceph-mon[93869]: pgmap v16414: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 764 KiB/s rd, 760 KiB/s wr, 24 op/s 2023-03-09T23:15:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:15:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:15:09.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:09 smithi181 ceph-mon[94189]: pgmap v16414: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 764 KiB/s rd, 760 KiB/s wr, 24 op/s 2023-03-09T23:15:09.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:15:09.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:15:09.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:09 smithi038 ceph-mon[92422]: pgmap v16414: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 764 KiB/s rd, 760 KiB/s wr, 24 op/s 2023-03-09T23:15:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:15:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:15:11.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29821.409367 secs 2023-03-09T23:15:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:11 smithi122 ceph-mon[93869]: pgmap v16415: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 837 KiB/s rd, 814 KiB/s wr, 23 op/s 2023-03-09T23:15:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29821.409367 secs 2023-03-09T23:15:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:11 smithi181 ceph-mon[94189]: pgmap v16415: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 837 KiB/s rd, 814 KiB/s wr, 23 op/s 2023-03-09T23:15:11.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29821.409367 secs 2023-03-09T23:15:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:11 smithi038 ceph-mon[92422]: pgmap v16415: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 837 KiB/s rd, 814 KiB/s wr, 23 op/s 2023-03-09T23:15:13.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:13 smithi122 ceph-mon[93869]: pgmap v16416: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 838 KiB/s rd, 670 KiB/s wr, 25 op/s 2023-03-09T23:15:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:13 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 29823.807013 secs 2023-03-09T23:15:13.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:13 smithi181 ceph-mon[94189]: pgmap v16416: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 838 KiB/s rd, 670 KiB/s wr, 25 op/s 2023-03-09T23:15:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:13 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 29823.807013 secs 2023-03-09T23:15:13.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:13 smithi038 ceph-mon[92422]: pgmap v16416: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 838 KiB/s rd, 670 KiB/s wr, 25 op/s 2023-03-09T23:15:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:13 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 29823.807013 secs 2023-03-09T23:15:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30285.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:15:15.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:15 smithi122 ceph-mon[93869]: pgmap v16417: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 500 KiB/s wr, 22 op/s 2023-03-09T23:15:15.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:15 smithi181 ceph-mon[94189]: pgmap v16417: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 500 KiB/s wr, 22 op/s 2023-03-09T23:15:15.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:15 smithi038 ceph-mon[92422]: pgmap v16417: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 500 KiB/s wr, 22 op/s 2023-03-09T23:15:16.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29826.409431 secs 2023-03-09T23:15:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:15:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:15:16.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29826.409431 secs 2023-03-09T23:15:16.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:15:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:15:16.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29826.409431 secs 2023-03-09T23:15:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:15:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:15:17.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:17 smithi122 ceph-mon[93869]: pgmap v16418: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 627 KiB/s rd, 491 KiB/s wr, 13 op/s 2023-03-09T23:15:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:15:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:15:17.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:17 smithi181 ceph-mon[94189]: pgmap v16418: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 627 KiB/s rd, 491 KiB/s wr, 13 op/s 2023-03-09T23:15:17.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:15:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:15:17.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:17 smithi038 ceph-mon[92422]: pgmap v16418: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 627 KiB/s rd, 491 KiB/s wr, 13 op/s 2023-03-09T23:15:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:15:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:15:18.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:18 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 29828.807119 secs 2023-03-09T23:15:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:18 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:15:18.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:18 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 29828.807119 secs 2023-03-09T23:15:18.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:18 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:15:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:18 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 29828.807119 secs 2023-03-09T23:15:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:18 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:15:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30290.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:15:19.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:19 smithi122 ceph-mon[93869]: pgmap v16419: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 318 KiB/s rd, 544 KiB/s wr, 10 op/s 2023-03-09T23:15:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:15:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:15:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:15:19.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:19 smithi181 ceph-mon[94189]: pgmap v16419: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 318 KiB/s rd, 544 KiB/s wr, 10 op/s 2023-03-09T23:15:19.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:15:19.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:15:19.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:15:19.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:19 smithi038 ceph-mon[92422]: pgmap v16419: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 318 KiB/s rd, 544 KiB/s wr, 10 op/s 2023-03-09T23:15:19.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:15:19.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:15:19.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:15:20.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:15:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:15:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29831.409491 secs 2023-03-09T23:15:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:15:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:15:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29831.409491 secs 2023-03-09T23:15:20.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:15:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:15:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29831.409491 secs 2023-03-09T23:15:21.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:21 smithi122 ceph-mon[93869]: pgmap v16420: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 605 KiB/s wr, 9 op/s 2023-03-09T23:15:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:15:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:15:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:21 smithi181 ceph-mon[94189]: pgmap v16420: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 605 KiB/s wr, 9 op/s 2023-03-09T23:15:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:15:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:15:21.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:21 smithi038 ceph-mon[92422]: pgmap v16420: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 605 KiB/s wr, 9 op/s 2023-03-09T23:15:21.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:15:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:15:23.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:23 smithi122 ceph-mon[93869]: pgmap v16421: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 530 KiB/s wr, 11 op/s 2023-03-09T23:15:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29833.807263 secs 2023-03-09T23:15:23.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:23 smithi181 ceph-mon[94189]: pgmap v16421: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 530 KiB/s wr, 11 op/s 2023-03-09T23:15:23.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29833.807263 secs 2023-03-09T23:15:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:23 smithi038 ceph-mon[92422]: pgmap v16421: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 530 KiB/s wr, 11 op/s 2023-03-09T23:15:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29833.807263 secs 2023-03-09T23:15:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30295.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:15:25.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:25 smithi122 ceph-mon[93869]: pgmap v16422: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 618 KiB/s rd, 638 KiB/s wr, 11 op/s 2023-03-09T23:15:25.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:25 smithi181 ceph-mon[94189]: pgmap v16422: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 618 KiB/s rd, 638 KiB/s wr, 11 op/s 2023-03-09T23:15:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:25 smithi038 ceph-mon[92422]: pgmap v16422: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 618 KiB/s rd, 638 KiB/s wr, 11 op/s 2023-03-09T23:15:26.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29836.409551 secs 2023-03-09T23:15:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:26 smithi122 ceph-mon[93869]: osdmap e1241: 12 total, 12 up, 12 in 2023-03-09T23:15:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29836.409551 secs 2023-03-09T23:15:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:26 smithi181 ceph-mon[94189]: osdmap e1241: 12 total, 12 up, 12 in 2023-03-09T23:15:26.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29836.409551 secs 2023-03-09T23:15:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:26 smithi038 ceph-mon[92422]: osdmap e1241: 12 total, 12 up, 12 in 2023-03-09T23:15:27.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:27 smithi122 ceph-mon[93869]: pgmap v16424: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 519 KiB/s rd, 540 KiB/s wr, 12 op/s 2023-03-09T23:15:27.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:27 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:15:27.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:27 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:15:27.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:27 smithi181 ceph-mon[94189]: pgmap v16424: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 519 KiB/s rd, 540 KiB/s wr, 12 op/s 2023-03-09T23:15:27.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:27 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:15:27.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:27 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:15:27.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:27 smithi038 ceph-mon[92422]: pgmap v16424: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 519 KiB/s rd, 540 KiB/s wr, 12 op/s 2023-03-09T23:15:27.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:27 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:15:27.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:27 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:15:28.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29838.807360 secs 2023-03-09T23:15:28.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29838.807360 secs 2023-03-09T23:15:28.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29838.807360 secs 2023-03-09T23:15:28.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30300.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:15:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:29 smithi122 ceph-mon[93869]: pgmap v16425: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 647 KiB/s rd, 375 KiB/s wr, 11 op/s 2023-03-09T23:15:29.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:29 smithi181 ceph-mon[94189]: pgmap v16425: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 647 KiB/s rd, 375 KiB/s wr, 11 op/s 2023-03-09T23:15:29.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:29 smithi038 ceph-mon[92422]: pgmap v16425: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 647 KiB/s rd, 375 KiB/s wr, 11 op/s 2023-03-09T23:15:31.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29841.409610 secs 2023-03-09T23:15:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:31 smithi122 ceph-mon[93869]: pgmap v16426: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 573 KiB/s rd, 301 KiB/s wr, 10 op/s 2023-03-09T23:15:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29841.409610 secs 2023-03-09T23:15:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:31 smithi181 ceph-mon[94189]: pgmap v16426: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 573 KiB/s rd, 301 KiB/s wr, 10 op/s 2023-03-09T23:15:31.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29841.409610 secs 2023-03-09T23:15:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:31 smithi038 ceph-mon[92422]: pgmap v16426: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 573 KiB/s rd, 301 KiB/s wr, 10 op/s 2023-03-09T23:15:32.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:32 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:15:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:32 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:15:32.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:32 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:15:32.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:32 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:15:32.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:32 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:15:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:32 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:15:33.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:33 smithi122 ceph-mon[93869]: pgmap v16427: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 571 KiB/s rd, 289 KiB/s wr, 9 op/s 2023-03-09T23:15:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:33 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:15:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:33 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:15:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29843.807440 secs 2023-03-09T23:15:33.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:33 smithi181 ceph-mon[94189]: pgmap v16427: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 571 KiB/s rd, 289 KiB/s wr, 9 op/s 2023-03-09T23:15:33.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:33 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:15:33.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:33 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:15:33.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29843.807440 secs 2023-03-09T23:15:33.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:33 smithi038 ceph-mon[92422]: pgmap v16427: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 571 KiB/s rd, 289 KiB/s wr, 9 op/s 2023-03-09T23:15:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:33 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:15:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:33 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:15:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29843.807440 secs 2023-03-09T23:15:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30305.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:15:35.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:35 smithi122 ceph-mon[93869]: pgmap v16428: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 440 KiB/s wr, 8 op/s 2023-03-09T23:15:35.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:35 smithi181 ceph-mon[94189]: pgmap v16428: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 440 KiB/s wr, 8 op/s 2023-03-09T23:15:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:35 smithi038 ceph-mon[92422]: pgmap v16428: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 440 KiB/s wr, 8 op/s 2023-03-09T23:15:36.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29846.409671 secs 2023-03-09T23:15:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:15:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:15:36.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29846.409671 secs 2023-03-09T23:15:36.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:15:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:15:36.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29846.409671 secs 2023-03-09T23:15:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:15:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:15:37.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:37 smithi122 ceph-mon[93869]: pgmap v16429: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 630 KiB/s wr, 11 op/s 2023-03-09T23:15:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:15:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:15:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:37 smithi122 ceph-mon[93869]: osdmap e1242: 12 total, 12 up, 12 in 2023-03-09T23:15:37.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:37 smithi181 ceph-mon[94189]: pgmap v16429: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 630 KiB/s wr, 11 op/s 2023-03-09T23:15:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:15:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:15:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:37 smithi181 ceph-mon[94189]: osdmap e1242: 12 total, 12 up, 12 in 2023-03-09T23:15:37.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:37 smithi038 ceph-mon[92422]: pgmap v16429: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 630 KiB/s wr, 11 op/s 2023-03-09T23:15:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:15:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:15:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:37 smithi038 ceph-mon[92422]: osdmap e1242: 12 total, 12 up, 12 in 2023-03-09T23:15:38.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29848.807569 secs 2023-03-09T23:15:38.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29848.807569 secs 2023-03-09T23:15:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29848.807569 secs 2023-03-09T23:15:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30310.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:15:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:39 smithi122 ceph-mon[93869]: pgmap v16431: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 562 KiB/s wr, 9 op/s 2023-03-09T23:15:39.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:39 smithi181 ceph-mon[94189]: pgmap v16431: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 562 KiB/s wr, 9 op/s 2023-03-09T23:15:39.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:39 smithi038 ceph-mon[92422]: pgmap v16431: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 562 KiB/s wr, 9 op/s 2023-03-09T23:15:40.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29851.409732 secs 2023-03-09T23:15:40.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29851.409732 secs 2023-03-09T23:15:40.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29851.409732 secs 2023-03-09T23:15:41.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:41 smithi038 ceph-mon[92422]: pgmap v16432: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 562 KiB/s wr, 9 op/s 2023-03-09T23:15:41.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:41 smithi122 ceph-mon[93869]: pgmap v16432: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 562 KiB/s wr, 9 op/s 2023-03-09T23:15:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:41 smithi181 ceph-mon[94189]: pgmap v16432: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 562 KiB/s wr, 9 op/s 2023-03-09T23:15:43.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:43 smithi122 ceph-mon[93869]: pgmap v16433: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 784 KiB/s rd, 783 KiB/s wr, 11 op/s 2023-03-09T23:15:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29853.807693 secs 2023-03-09T23:15:43.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:43 smithi181 ceph-mon[94189]: pgmap v16433: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 784 KiB/s rd, 783 KiB/s wr, 11 op/s 2023-03-09T23:15:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29853.807693 secs 2023-03-09T23:15:43.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:43 smithi038 ceph-mon[92422]: pgmap v16433: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 784 KiB/s rd, 783 KiB/s wr, 11 op/s 2023-03-09T23:15:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29853.807693 secs 2023-03-09T23:15:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30315.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:15:45.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:45 smithi122 ceph-mon[93869]: pgmap v16434: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 897 KiB/s rd, 616 KiB/s wr, 10 op/s 2023-03-09T23:15:45.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:45 smithi181 ceph-mon[94189]: pgmap v16434: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 897 KiB/s rd, 616 KiB/s wr, 10 op/s 2023-03-09T23:15:45.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:45 smithi038 ceph-mon[92422]: pgmap v16434: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 897 KiB/s rd, 616 KiB/s wr, 10 op/s 2023-03-09T23:15:46.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29856.409794 secs 2023-03-09T23:15:46.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29856.409794 secs 2023-03-09T23:15:46.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29856.409794 secs 2023-03-09T23:15:47.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:47 smithi122 ceph-mon[93869]: pgmap v16435: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 720 KiB/s rd, 424 KiB/s wr, 11 op/s 2023-03-09T23:15:47.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:47 smithi181 ceph-mon[94189]: pgmap v16435: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 720 KiB/s rd, 424 KiB/s wr, 11 op/s 2023-03-09T23:15:47.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:47 smithi038 ceph-mon[92422]: pgmap v16435: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 720 KiB/s rd, 424 KiB/s wr, 11 op/s 2023-03-09T23:15:48.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:15:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:15:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29858.807832 secs 2023-03-09T23:15:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:15:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:15:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29858.807832 secs 2023-03-09T23:15:48.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:15:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:15:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29858.807832 secs 2023-03-09T23:15:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30320.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:15:49.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:49 smithi122 ceph-mon[93869]: pgmap v16436: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 970 KiB/s rd, 701 KiB/s wr, 11 op/s 2023-03-09T23:15:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:15:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:15:49.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:49 smithi181 ceph-mon[94189]: pgmap v16436: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 970 KiB/s rd, 701 KiB/s wr, 11 op/s 2023-03-09T23:15:49.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:15:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:15:49.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:49 smithi038 ceph-mon[92422]: pgmap v16436: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 970 KiB/s rd, 701 KiB/s wr, 11 op/s 2023-03-09T23:15:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:15:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:15:51.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29861.409850 secs 2023-03-09T23:15:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:51 smithi122 ceph-mon[93869]: pgmap v16437: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 871 KiB/s rd, 858 KiB/s wr, 10 op/s 2023-03-09T23:15:51.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29861.409850 secs 2023-03-09T23:15:51.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:51 smithi181 ceph-mon[94189]: pgmap v16437: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 871 KiB/s rd, 858 KiB/s wr, 10 op/s 2023-03-09T23:15:51.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29861.409850 secs 2023-03-09T23:15:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:51 smithi038 ceph-mon[92422]: pgmap v16437: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 871 KiB/s rd, 858 KiB/s wr, 10 op/s 2023-03-09T23:15:52.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:15:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:15:52.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:15:52.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:15:52.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:15:52.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:15:53.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:53 smithi122 ceph-mon[93869]: pgmap v16438: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1010 KiB/s rd, 995 KiB/s wr, 15 op/s 2023-03-09T23:15:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29863.807955 secs 2023-03-09T23:15:53.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:53 smithi181 ceph-mon[94189]: pgmap v16438: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1010 KiB/s rd, 995 KiB/s wr, 15 op/s 2023-03-09T23:15:53.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29863.807955 secs 2023-03-09T23:15:53.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:53 smithi038 ceph-mon[92422]: pgmap v16438: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1010 KiB/s rd, 995 KiB/s wr, 15 op/s 2023-03-09T23:15:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29863.807955 secs 2023-03-09T23:15:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30325.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:15:55.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:55 smithi122 ceph-mon[93869]: pgmap v16439: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 832 KiB/s rd, 818 KiB/s wr, 12 op/s 2023-03-09T23:15:55.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:55 smithi181 ceph-mon[94189]: pgmap v16439: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 832 KiB/s rd, 818 KiB/s wr, 12 op/s 2023-03-09T23:15:55.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:55 smithi038 ceph-mon[92422]: pgmap v16439: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 832 KiB/s rd, 818 KiB/s wr, 12 op/s 2023-03-09T23:15:56.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29866.409911 secs 2023-03-09T23:15:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:15:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:15:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29866.409911 secs 2023-03-09T23:15:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:15:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:15:56.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29866.409911 secs 2023-03-09T23:15:56.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:15:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:15:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:57 smithi122 ceph-mon[93869]: pgmap v16440: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 992 KiB/s rd, 977 KiB/s wr, 16 op/s 2023-03-09T23:15:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:15:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:15:57.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:57 smithi181 ceph-mon[94189]: pgmap v16440: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 992 KiB/s rd, 977 KiB/s wr, 16 op/s 2023-03-09T23:15:57.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:15:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:15:57.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:57 smithi038 ceph-mon[92422]: pgmap v16440: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 992 KiB/s rd, 977 KiB/s wr, 16 op/s 2023-03-09T23:15:57.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:15:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:15:58.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29868.808071 secs 2023-03-09T23:15:58.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29868.808071 secs 2023-03-09T23:15:58.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29868.808071 secs 2023-03-09T23:15:58.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30330.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:15:59.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:15:59 smithi122 ceph-mon[93869]: pgmap v16441: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 13 op/s 2023-03-09T23:15:59.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:15:59 smithi181 ceph-mon[94189]: pgmap v16441: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 13 op/s 2023-03-09T23:15:59.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:15:59 smithi038 ceph-mon[92422]: pgmap v16441: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 13 op/s 2023-03-09T23:16:00.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:16:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:16:00.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:16:00.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:16:00.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:16:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:16:01.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29871.409974 secs 2023-03-09T23:16:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:01 smithi122 ceph-mon[93869]: pgmap v16442: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 742 KiB/s wr, 12 op/s 2023-03-09T23:16:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:16:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:16:01.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29871.409974 secs 2023-03-09T23:16:01.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:01 smithi181 ceph-mon[94189]: pgmap v16442: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 742 KiB/s wr, 12 op/s 2023-03-09T23:16:01.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:16:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:16:01.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29871.409974 secs 2023-03-09T23:16:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:01 smithi038 ceph-mon[92422]: pgmap v16442: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 742 KiB/s wr, 12 op/s 2023-03-09T23:16:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:16:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:16:03.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:03 smithi122 ceph-mon[93869]: pgmap v16443: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 555 KiB/s rd, 551 KiB/s wr, 14 op/s 2023-03-09T23:16:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29873.808193 secs 2023-03-09T23:16:03.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:03 smithi181 ceph-mon[94189]: pgmap v16443: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 555 KiB/s rd, 551 KiB/s wr, 14 op/s 2023-03-09T23:16:03.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29873.808193 secs 2023-03-09T23:16:03.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:03 smithi038 ceph-mon[92422]: pgmap v16443: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 555 KiB/s rd, 551 KiB/s wr, 14 op/s 2023-03-09T23:16:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29873.808193 secs 2023-03-09T23:16:03.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30335.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:16:05.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:05 smithi122 ceph-mon[93869]: pgmap v16444: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 640 KiB/s rd, 674 KiB/s wr, 23 op/s 2023-03-09T23:16:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:05 smithi122 ceph-mon[93869]: osdmap e1243: 12 total, 12 up, 12 in 2023-03-09T23:16:05.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:05 smithi181 ceph-mon[94189]: pgmap v16444: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 640 KiB/s rd, 674 KiB/s wr, 23 op/s 2023-03-09T23:16:05.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:05 smithi181 ceph-mon[94189]: osdmap e1243: 12 total, 12 up, 12 in 2023-03-09T23:16:05.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:05 smithi038 ceph-mon[92422]: pgmap v16444: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 640 KiB/s rd, 674 KiB/s wr, 23 op/s 2023-03-09T23:16:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:05 smithi038 ceph-mon[92422]: osdmap e1243: 12 total, 12 up, 12 in 2023-03-09T23:16:06.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29876.410032 secs 2023-03-09T23:16:06.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29876.410032 secs 2023-03-09T23:16:06.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29876.410032 secs 2023-03-09T23:16:07.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:07 smithi122 ceph-mon[93869]: pgmap v16446: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 606 KiB/s rd, 672 KiB/s wr, 28 op/s 2023-03-09T23:16:07.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:07 smithi181 ceph-mon[94189]: pgmap v16446: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 606 KiB/s rd, 672 KiB/s wr, 28 op/s 2023-03-09T23:16:07.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:07 smithi038 ceph-mon[92422]: pgmap v16446: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 606 KiB/s rd, 672 KiB/s wr, 28 op/s 2023-03-09T23:16:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:16:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:16:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29878.808311 secs 2023-03-09T23:16:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:16:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:16:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29878.808311 secs 2023-03-09T23:16:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:16:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:16:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29878.808311 secs 2023-03-09T23:16:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30340.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:16:09.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:09 smithi181 ceph-mon[94189]: pgmap v16447: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 547 KiB/s wr, 28 op/s 2023-03-09T23:16:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:16:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:16:09.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:09 smithi038 ceph-mon[92422]: pgmap v16447: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 547 KiB/s wr, 28 op/s 2023-03-09T23:16:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:16:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:16:09.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:09 smithi122 ceph-mon[93869]: pgmap v16447: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 547 KiB/s wr, 28 op/s 2023-03-09T23:16:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:16:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:16:10.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29881.410089 secs 2023-03-09T23:16:10.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29881.410089 secs 2023-03-09T23:16:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29881.410089 secs 2023-03-09T23:16:11.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:11 smithi038 ceph-mon[92422]: pgmap v16448: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 547 KiB/s wr, 28 op/s 2023-03-09T23:16:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:11 smithi122 ceph-mon[93869]: pgmap v16448: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 547 KiB/s wr, 28 op/s 2023-03-09T23:16:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:11 smithi181 ceph-mon[94189]: pgmap v16448: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 547 KiB/s wr, 28 op/s 2023-03-09T23:16:13.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:13 smithi122 ceph-mon[93869]: osdmap e1244: 12 total, 12 up, 12 in 2023-03-09T23:16:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:13 smithi122 ceph-mon[93869]: pgmap v16450: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 452 KiB/s rd, 450 KiB/s wr, 15 op/s 2023-03-09T23:16:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29883.808413 secs 2023-03-09T23:16:13.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:13 smithi181 ceph-mon[94189]: osdmap e1244: 12 total, 12 up, 12 in 2023-03-09T23:16:13.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:13 smithi181 ceph-mon[94189]: pgmap v16450: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 452 KiB/s rd, 450 KiB/s wr, 15 op/s 2023-03-09T23:16:13.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29883.808413 secs 2023-03-09T23:16:13.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:13 smithi038 ceph-mon[92422]: osdmap e1244: 12 total, 12 up, 12 in 2023-03-09T23:16:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:13 smithi038 ceph-mon[92422]: pgmap v16450: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 452 KiB/s rd, 450 KiB/s wr, 15 op/s 2023-03-09T23:16:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29883.808413 secs 2023-03-09T23:16:13.801 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30345.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:16:15.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:15 smithi122 ceph-mon[93869]: pgmap v16451: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 362 KiB/s wr, 12 op/s 2023-03-09T23:16:15.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:15 smithi181 ceph-mon[94189]: pgmap v16451: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 362 KiB/s wr, 12 op/s 2023-03-09T23:16:15.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:15 smithi038 ceph-mon[92422]: pgmap v16451: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 362 KiB/s wr, 12 op/s 2023-03-09T23:16:16.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29886.410151 secs 2023-03-09T23:16:16.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29886.410151 secs 2023-03-09T23:16:16.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29886.410151 secs 2023-03-09T23:16:17.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:17 smithi122 ceph-mon[93869]: pgmap v16452: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 231 KiB/s wr, 8 op/s 2023-03-09T23:16:17.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:17 smithi181 ceph-mon[94189]: pgmap v16452: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 231 KiB/s wr, 8 op/s 2023-03-09T23:16:17.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:17 smithi038 ceph-mon[92422]: pgmap v16452: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 231 KiB/s wr, 8 op/s 2023-03-09T23:16:18.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29888.808532 secs 2023-03-09T23:16:18.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29888.808532 secs 2023-03-09T23:16:18.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29888.808532 secs 2023-03-09T23:16:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30350.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:16:19.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:19 smithi122 ceph-mon[93869]: pgmap v16453: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 300 KiB/s wr, 8 op/s 2023-03-09T23:16:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:16:19.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:19 smithi181 ceph-mon[94189]: pgmap v16453: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 300 KiB/s wr, 8 op/s 2023-03-09T23:16:19.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:16:19.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:19 smithi038 ceph-mon[92422]: pgmap v16453: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 300 KiB/s wr, 8 op/s 2023-03-09T23:16:19.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:16:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:16:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:16:20.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:16:20.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:16:20.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:16:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:16:20.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:16:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:16:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:16:21.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29891.410231 secs 2023-03-09T23:16:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:21 smithi038 ceph-mon[92422]: pgmap v16454: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 300 KiB/s wr, 8 op/s 2023-03-09T23:16:21.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29891.410231 secs 2023-03-09T23:16:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:21 smithi122 ceph-mon[93869]: pgmap v16454: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 300 KiB/s wr, 8 op/s 2023-03-09T23:16:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29891.410231 secs 2023-03-09T23:16:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:21 smithi181 ceph-mon[94189]: pgmap v16454: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 300 KiB/s wr, 8 op/s 2023-03-09T23:16:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:23 smithi122 ceph-mon[93869]: pgmap v16455: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 265 KiB/s wr, 9 op/s 2023-03-09T23:16:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29893.808648 secs 2023-03-09T23:16:23.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:23 smithi181 ceph-mon[94189]: pgmap v16455: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 265 KiB/s wr, 9 op/s 2023-03-09T23:16:23.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29893.808648 secs 2023-03-09T23:16:23.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:23 smithi038 ceph-mon[92422]: pgmap v16455: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 265 KiB/s wr, 9 op/s 2023-03-09T23:16:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29893.808648 secs 2023-03-09T23:16:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30355.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:16:25.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:25 smithi122 ceph-mon[93869]: pgmap v16456: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 221 KiB/s wr, 7 op/s 2023-03-09T23:16:25.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:25 smithi181 ceph-mon[94189]: pgmap v16456: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 221 KiB/s wr, 7 op/s 2023-03-09T23:16:25.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:25 smithi038 ceph-mon[92422]: pgmap v16456: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 221 KiB/s wr, 7 op/s 2023-03-09T23:16:26.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29896.410293 secs 2023-03-09T23:16:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29896.410293 secs 2023-03-09T23:16:26.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29896.410293 secs 2023-03-09T23:16:27.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:27 smithi122 ceph-mon[93869]: pgmap v16457: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 243 KiB/s wr, 10 op/s 2023-03-09T23:16:27.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:27 smithi181 ceph-mon[94189]: pgmap v16457: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 243 KiB/s wr, 10 op/s 2023-03-09T23:16:27.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:27 smithi038 ceph-mon[92422]: pgmap v16457: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 243 KiB/s wr, 10 op/s 2023-03-09T23:16:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:28 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:16:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:28 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:16:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29898.808762 secs 2023-03-09T23:16:28.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:28 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:16:28.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:28 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:16:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29898.808762 secs 2023-03-09T23:16:28.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:28 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:16:28.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:28 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:16:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29898.808762 secs 2023-03-09T23:16:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30360.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:16:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:29 smithi122 ceph-mon[93869]: pgmap v16458: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 177 KiB/s wr, 8 op/s 2023-03-09T23:16:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:29 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:16:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:29 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:16:29.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:29 smithi181 ceph-mon[94189]: pgmap v16458: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 177 KiB/s wr, 8 op/s 2023-03-09T23:16:29.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:29 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:16:29.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:29 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:16:29.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:29 smithi038 ceph-mon[92422]: pgmap v16458: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 177 KiB/s wr, 8 op/s 2023-03-09T23:16:29.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:29 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:16:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:29 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:16:31.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29901.410352 secs 2023-03-09T23:16:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:31 smithi122 ceph-mon[93869]: pgmap v16459: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 120 KiB/s wr, 8 op/s 2023-03-09T23:16:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29901.410352 secs 2023-03-09T23:16:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:31 smithi181 ceph-mon[94189]: pgmap v16459: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 120 KiB/s wr, 8 op/s 2023-03-09T23:16:31.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29901.410352 secs 2023-03-09T23:16:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:31 smithi038 ceph-mon[92422]: pgmap v16459: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 120 KiB/s wr, 8 op/s 2023-03-09T23:16:33.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:33 smithi122 ceph-mon[93869]: pgmap v16460: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 250 KiB/s wr, 11 op/s 2023-03-09T23:16:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29903.808860 secs 2023-03-09T23:16:33.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:33 smithi181 ceph-mon[94189]: pgmap v16460: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 250 KiB/s wr, 11 op/s 2023-03-09T23:16:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29903.808860 secs 2023-03-09T23:16:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:33 smithi038 ceph-mon[92422]: pgmap v16460: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 250 KiB/s wr, 11 op/s 2023-03-09T23:16:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29903.808860 secs 2023-03-09T23:16:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30365.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:16:35.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:35 smithi122 ceph-mon[93869]: pgmap v16461: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 152 KiB/s wr, 6 op/s 2023-03-09T23:16:35.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:35 smithi181 ceph-mon[94189]: pgmap v16461: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 152 KiB/s wr, 6 op/s 2023-03-09T23:16:35.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:35 smithi038 ceph-mon[92422]: pgmap v16461: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 152 KiB/s wr, 6 op/s 2023-03-09T23:16:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29906.410416 secs 2023-03-09T23:16:36.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29906.410416 secs 2023-03-09T23:16:36.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29906.410416 secs 2023-03-09T23:16:37.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:37 smithi122 ceph-mon[93869]: pgmap v16462: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 152 KiB/s wr, 9 op/s 2023-03-09T23:16:37.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:37 smithi181 ceph-mon[94189]: pgmap v16462: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 152 KiB/s wr, 9 op/s 2023-03-09T23:16:37.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:37 smithi038 ceph-mon[92422]: pgmap v16462: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 152 KiB/s wr, 9 op/s 2023-03-09T23:16:38.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29908.808976 secs 2023-03-09T23:16:38.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29908.808976 secs 2023-03-09T23:16:38.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29908.808976 secs 2023-03-09T23:16:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30370.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:16:39.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:39 smithi122 ceph-mon[93869]: pgmap v16463: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 214 KiB/s wr, 7 op/s 2023-03-09T23:16:39.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:39 smithi181 ceph-mon[94189]: pgmap v16463: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 214 KiB/s wr, 7 op/s 2023-03-09T23:16:39.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:39 smithi038 ceph-mon[92422]: pgmap v16463: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 214 KiB/s wr, 7 op/s 2023-03-09T23:16:41.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29911.410712 secs 2023-03-09T23:16:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:41 smithi122 ceph-mon[93869]: pgmap v16464: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 214 KiB/s wr, 6 op/s 2023-03-09T23:16:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:16:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:16:41.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29911.410712 secs 2023-03-09T23:16:41.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:41 smithi181 ceph-mon[94189]: pgmap v16464: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 214 KiB/s wr, 6 op/s 2023-03-09T23:16:41.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:16:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:16:41.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29911.410712 secs 2023-03-09T23:16:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:41 smithi038 ceph-mon[92422]: pgmap v16464: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 214 KiB/s wr, 6 op/s 2023-03-09T23:16:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:16:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:16:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:43 smithi122 ceph-mon[93869]: pgmap v16465: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 539 KiB/s rd, 368 KiB/s wr, 9 op/s 2023-03-09T23:16:43.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:43 smithi181 ceph-mon[94189]: pgmap v16465: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 539 KiB/s rd, 368 KiB/s wr, 9 op/s 2023-03-09T23:16:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:43 smithi038 ceph-mon[92422]: pgmap v16465: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 539 KiB/s rd, 368 KiB/s wr, 9 op/s 2023-03-09T23:16:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30375.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:16:45.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:45 smithi122 ceph-mon[93869]: pgmap v16466: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 238 KiB/s wr, 6 op/s 2023-03-09T23:16:45.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:45 smithi181 ceph-mon[94189]: pgmap v16466: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 238 KiB/s wr, 6 op/s 2023-03-09T23:16:45.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:45 smithi038 ceph-mon[92422]: pgmap v16466: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 238 KiB/s wr, 6 op/s 2023-03-09T23:16:46.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29916.410774 secs 2023-03-09T23:16:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:46 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:16:46.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29916.410774 secs 2023-03-09T23:16:46.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:46 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:16:46.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29916.410774 secs 2023-03-09T23:16:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:46 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:16:47.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:47 smithi122 ceph-mon[93869]: pgmap v16467: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 596 KiB/s rd, 425 KiB/s wr, 9 op/s 2023-03-09T23:16:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:47 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:16:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:16:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:16:47.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:47 smithi181 ceph-mon[94189]: pgmap v16467: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 596 KiB/s rd, 425 KiB/s wr, 9 op/s 2023-03-09T23:16:47.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:47 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:16:47.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:16:47.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:16:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:47 smithi038 ceph-mon[92422]: pgmap v16467: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 596 KiB/s rd, 425 KiB/s wr, 9 op/s 2023-03-09T23:16:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:47 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:16:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:16:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:16:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:16:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:16:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:16:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:16:48.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:16:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:16:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30380.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:16:49.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:49 smithi181 ceph-mon[94189]: pgmap v16468: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 594 KiB/s rd, 425 KiB/s wr, 8 op/s 2023-03-09T23:16:49.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:49 smithi038 ceph-mon[92422]: pgmap v16468: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 594 KiB/s rd, 425 KiB/s wr, 8 op/s 2023-03-09T23:16:49.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:49 smithi122 ceph-mon[93869]: pgmap v16468: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 594 KiB/s rd, 425 KiB/s wr, 8 op/s 2023-03-09T23:16:50.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29921.410832 secs 2023-03-09T23:16:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29921.410832 secs 2023-03-09T23:16:50.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29921.410832 secs 2023-03-09T23:16:51.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:51 smithi122 ceph-mon[93869]: pgmap v16469: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 342 KiB/s wr, 7 op/s 2023-03-09T23:16:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:51 smithi181 ceph-mon[94189]: pgmap v16469: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 342 KiB/s wr, 7 op/s 2023-03-09T23:16:51.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:51 smithi038 ceph-mon[92422]: pgmap v16469: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 342 KiB/s wr, 7 op/s 2023-03-09T23:16:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:53 smithi122 ceph-mon[93869]: pgmap v16470: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 666 KiB/s rd, 366 KiB/s wr, 10 op/s 2023-03-09T23:16:53.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:53 smithi181 ceph-mon[94189]: pgmap v16470: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 666 KiB/s rd, 366 KiB/s wr, 10 op/s 2023-03-09T23:16:53.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:53 smithi038 ceph-mon[92422]: pgmap v16470: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 666 KiB/s rd, 366 KiB/s wr, 10 op/s 2023-03-09T23:16:53.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30385.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:16:55.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:55 smithi122 ceph-mon[93869]: pgmap v16471: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 637 KiB/s rd, 422 KiB/s wr, 8 op/s 2023-03-09T23:16:55.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:55 smithi181 ceph-mon[94189]: pgmap v16471: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 637 KiB/s rd, 422 KiB/s wr, 8 op/s 2023-03-09T23:16:55.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:55 smithi038 ceph-mon[92422]: pgmap v16471: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 637 KiB/s rd, 422 KiB/s wr, 8 op/s 2023-03-09T23:16:56.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29926.410894 secs 2023-03-09T23:16:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:16:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:16:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29926.410894 secs 2023-03-09T23:16:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:16:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:16:56.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29926.410894 secs 2023-03-09T23:16:56.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:16:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:16:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:57 smithi122 ceph-mon[93869]: pgmap v16472: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 675 KiB/s rd, 446 KiB/s wr, 11 op/s 2023-03-09T23:16:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:16:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:16:57.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:57 smithi181 ceph-mon[94189]: pgmap v16472: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 675 KiB/s rd, 446 KiB/s wr, 11 op/s 2023-03-09T23:16:57.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:16:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:16:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:57 smithi038 ceph-mon[92422]: pgmap v16472: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 675 KiB/s rd, 446 KiB/s wr, 11 op/s 2023-03-09T23:16:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:16:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:16:58.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30390.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:16:59.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:16:59 smithi181 ceph-mon[94189]: pgmap v16473: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 259 KiB/s wr, 9 op/s 2023-03-09T23:16:59.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:16:59 smithi038 ceph-mon[92422]: pgmap v16473: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 259 KiB/s wr, 9 op/s 2023-03-09T23:16:59.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:16:59 smithi122 ceph-mon[93869]: pgmap v16473: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 259 KiB/s wr, 9 op/s 2023-03-09T23:17:00.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:17:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:17:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29931.410954 secs 2023-03-09T23:17:00.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:17:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:17:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29931.410954 secs 2023-03-09T23:17:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:17:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:17:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29931.410954 secs 2023-03-09T23:17:01.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:01 smithi122 ceph-mon[93869]: pgmap v16474: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 858 KiB/s rd, 630 KiB/s wr, 10 op/s 2023-03-09T23:17:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:17:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:17:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:01 smithi181 ceph-mon[94189]: pgmap v16474: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 858 KiB/s rd, 630 KiB/s wr, 10 op/s 2023-03-09T23:17:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:17:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:17:01.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:01 smithi038 ceph-mon[92422]: pgmap v16474: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 858 KiB/s rd, 630 KiB/s wr, 10 op/s 2023-03-09T23:17:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:17:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:17:03.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:03 smithi122 ceph-mon[93869]: pgmap v16475: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1002 KiB/s rd, 784 KiB/s wr, 16 op/s 2023-03-09T23:17:03.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:03 smithi181 ceph-mon[94189]: pgmap v16475: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1002 KiB/s rd, 784 KiB/s wr, 16 op/s 2023-03-09T23:17:03.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:03 smithi038 ceph-mon[92422]: pgmap v16475: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1002 KiB/s rd, 784 KiB/s wr, 16 op/s 2023-03-09T23:17:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30395.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:17:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:05 smithi122 ceph-mon[93869]: pgmap v16476: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 872 KiB/s rd, 872 KiB/s wr, 24 op/s 2023-03-09T23:17:05.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:05 smithi181 ceph-mon[94189]: pgmap v16476: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 872 KiB/s rd, 872 KiB/s wr, 24 op/s 2023-03-09T23:17:05.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:05 smithi038 ceph-mon[92422]: pgmap v16476: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 872 KiB/s rd, 872 KiB/s wr, 24 op/s 2023-03-09T23:17:06.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29936.410985 secs 2023-03-09T23:17:06.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29936.410985 secs 2023-03-09T23:17:06.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29936.410985 secs 2023-03-09T23:17:07.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:07 smithi122 ceph-mon[93869]: pgmap v16477: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 682 KiB/s wr, 26 op/s 2023-03-09T23:17:07.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:07 smithi181 ceph-mon[94189]: pgmap v16477: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 682 KiB/s wr, 26 op/s 2023-03-09T23:17:07.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:07 smithi038 ceph-mon[92422]: pgmap v16477: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 682 KiB/s wr, 26 op/s 2023-03-09T23:17:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:17:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:17:08.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:17:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:17:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:17:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:17:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30400.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:17:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:09 smithi038 ceph-mon[92422]: pgmap v16478: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 644 KiB/s rd, 658 KiB/s wr, 24 op/s 2023-03-09T23:17:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:17:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:17:09.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:09 smithi122 ceph-mon[93869]: pgmap v16478: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 644 KiB/s rd, 658 KiB/s wr, 24 op/s 2023-03-09T23:17:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:17:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:17:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:09 smithi181 ceph-mon[94189]: pgmap v16478: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 644 KiB/s rd, 658 KiB/s wr, 24 op/s 2023-03-09T23:17:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:17:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:17:10.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29941.411046 secs 2023-03-09T23:17:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29941.411046 secs 2023-03-09T23:17:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29941.411046 secs 2023-03-09T23:17:11.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:11 smithi038 ceph-mon[92422]: pgmap v16479: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 644 KiB/s rd, 658 KiB/s wr, 23 op/s 2023-03-09T23:17:11.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:11 smithi122 ceph-mon[93869]: pgmap v16479: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 644 KiB/s rd, 658 KiB/s wr, 23 op/s 2023-03-09T23:17:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:11 smithi181 ceph-mon[94189]: pgmap v16479: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 644 KiB/s rd, 658 KiB/s wr, 23 op/s 2023-03-09T23:17:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:13 smithi122 ceph-mon[93869]: pgmap v16480: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 435 KiB/s rd, 447 KiB/s wr, 25 op/s 2023-03-09T23:17:13.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:13 smithi181 ceph-mon[94189]: pgmap v16480: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 435 KiB/s rd, 447 KiB/s wr, 25 op/s 2023-03-09T23:17:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:13 smithi038 ceph-mon[92422]: pgmap v16480: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 435 KiB/s rd, 447 KiB/s wr, 25 op/s 2023-03-09T23:17:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30405.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:17:15.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:15 smithi122 ceph-mon[93869]: pgmap v16481: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 293 KiB/s wr, 19 op/s 2023-03-09T23:17:15.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:15 smithi181 ceph-mon[94189]: pgmap v16481: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 293 KiB/s wr, 19 op/s 2023-03-09T23:17:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:15 smithi038 ceph-mon[92422]: pgmap v16481: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 293 KiB/s wr, 19 op/s 2023-03-09T23:17:16.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29946.411105 secs 2023-03-09T23:17:16.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29946.411105 secs 2023-03-09T23:17:16.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29946.411105 secs 2023-03-09T23:17:17.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:17 smithi122 ceph-mon[93869]: pgmap v16482: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 180 KiB/s wr, 11 op/s 2023-03-09T23:17:17.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:17 smithi181 ceph-mon[94189]: pgmap v16482: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 180 KiB/s wr, 11 op/s 2023-03-09T23:17:17.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:17 smithi038 ceph-mon[92422]: pgmap v16482: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 180 KiB/s wr, 11 op/s 2023-03-09T23:17:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30410.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:17:19.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:19 smithi038 ceph-mon[92422]: pgmap v16483: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 172 KiB/s wr, 9 op/s 2023-03-09T23:17:19.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:19 smithi122 ceph-mon[93869]: pgmap v16483: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 172 KiB/s wr, 9 op/s 2023-03-09T23:17:19.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:19 smithi181 ceph-mon[94189]: pgmap v16483: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 172 KiB/s wr, 9 op/s 2023-03-09T23:17:20.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:17:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:17:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:17:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29951.411168 secs 2023-03-09T23:17:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:17:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:17:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:17:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:17:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:17:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:17:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29951.411168 secs 2023-03-09T23:17:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:17:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:17:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:17:20.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:17:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:17:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:17:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29951.411168 secs 2023-03-09T23:17:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:17:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:17:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:17:21.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:21 smithi122 ceph-mon[93869]: pgmap v16484: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 172 KiB/s wr, 8 op/s 2023-03-09T23:17:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:17:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:17:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:21 smithi181 ceph-mon[94189]: pgmap v16484: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 172 KiB/s wr, 8 op/s 2023-03-09T23:17:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:17:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:17:21.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:21 smithi038 ceph-mon[92422]: pgmap v16484: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 172 KiB/s wr, 8 op/s 2023-03-09T23:17:21.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:17:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:17:23.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:23 smithi122 ceph-mon[93869]: pgmap v16485: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 596 KiB/s rd, 614 KiB/s wr, 14 op/s 2023-03-09T23:17:23.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:23 smithi181 ceph-mon[94189]: pgmap v16485: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 596 KiB/s rd, 614 KiB/s wr, 14 op/s 2023-03-09T23:17:23.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:23 smithi038 ceph-mon[92422]: pgmap v16485: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 596 KiB/s rd, 614 KiB/s wr, 14 op/s 2023-03-09T23:17:23.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30415.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:17:25.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:25 smithi122 ceph-mon[93869]: pgmap v16486: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 453 KiB/s wr, 11 op/s 2023-03-09T23:17:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:25 smithi122 ceph-mon[93869]: osdmap e1245: 12 total, 12 up, 12 in 2023-03-09T23:17:25.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:25 smithi181 ceph-mon[94189]: pgmap v16486: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 453 KiB/s wr, 11 op/s 2023-03-09T23:17:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:25 smithi181 ceph-mon[94189]: osdmap e1245: 12 total, 12 up, 12 in 2023-03-09T23:17:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:25 smithi038 ceph-mon[92422]: pgmap v16486: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 453 KiB/s wr, 11 op/s 2023-03-09T23:17:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:25 smithi038 ceph-mon[92422]: osdmap e1245: 12 total, 12 up, 12 in 2023-03-09T23:17:26.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29956.411227 secs 2023-03-09T23:17:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29956.411227 secs 2023-03-09T23:17:26.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29956.411227 secs 2023-03-09T23:17:27.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:27 smithi122 ceph-mon[93869]: pgmap v16488: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 674 KiB/s wr, 13 op/s 2023-03-09T23:17:27.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:27 smithi181 ceph-mon[94189]: pgmap v16488: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 674 KiB/s wr, 13 op/s 2023-03-09T23:17:27.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:27 smithi038 ceph-mon[92422]: pgmap v16488: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 674 KiB/s wr, 13 op/s 2023-03-09T23:17:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30420.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:17:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:29 smithi122 ceph-mon[93869]: pgmap v16489: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 650 KiB/s rd, 663 KiB/s wr, 12 op/s 2023-03-09T23:17:29.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:29 smithi181 ceph-mon[94189]: pgmap v16489: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 650 KiB/s rd, 663 KiB/s wr, 12 op/s 2023-03-09T23:17:29.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:29 smithi038 ceph-mon[92422]: pgmap v16489: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 650 KiB/s rd, 663 KiB/s wr, 12 op/s 2023-03-09T23:17:31.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29961.411290 secs 2023-03-09T23:17:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:31 smithi122 ceph-mon[93869]: pgmap v16490: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 650 KiB/s rd, 663 KiB/s wr, 11 op/s 2023-03-09T23:17:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29961.411290 secs 2023-03-09T23:17:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:31 smithi181 ceph-mon[94189]: pgmap v16490: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 650 KiB/s rd, 663 KiB/s wr, 11 op/s 2023-03-09T23:17:31.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29961.411290 secs 2023-03-09T23:17:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:31 smithi038 ceph-mon[92422]: pgmap v16490: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 650 KiB/s rd, 663 KiB/s wr, 11 op/s 2023-03-09T23:17:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:32 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:17:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:32 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:17:32.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:32 smithi122 ceph-mon[93869]: osdmap e1246: 12 total, 12 up, 12 in 2023-03-09T23:17:32.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:32 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:17:32.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:32 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:17:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:32 smithi181 ceph-mon[94189]: osdmap e1246: 12 total, 12 up, 12 in 2023-03-09T23:17:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:32 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:17:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:32 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:17:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:32 smithi038 ceph-mon[92422]: osdmap e1246: 12 total, 12 up, 12 in 2023-03-09T23:17:33.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:33 smithi122 ceph-mon[93869]: pgmap v16492: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 201 KiB/s wr, 11 op/s 2023-03-09T23:17:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:33 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:17:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:33 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:17:33.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:33 smithi181 ceph-mon[94189]: pgmap v16492: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 201 KiB/s wr, 11 op/s 2023-03-09T23:17:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:33 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:17:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:33 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:17:33.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:33 smithi038 ceph-mon[92422]: pgmap v16492: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 201 KiB/s wr, 11 op/s 2023-03-09T23:17:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:33 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:17:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:33 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:17:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30425.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:17:35.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:35 smithi122 ceph-mon[93869]: pgmap v16493: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 389 KiB/s wr, 10 op/s 2023-03-09T23:17:35.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:35 smithi181 ceph-mon[94189]: pgmap v16493: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 389 KiB/s wr, 10 op/s 2023-03-09T23:17:35.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:35 smithi038 ceph-mon[92422]: pgmap v16493: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 389 KiB/s wr, 10 op/s 2023-03-09T23:17:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29966.411350 secs 2023-03-09T23:17:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:17:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:17:36.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29966.411350 secs 2023-03-09T23:17:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:17:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:17:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29966.411350 secs 2023-03-09T23:17:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:17:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:17:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:37 smithi038 ceph-mon[92422]: pgmap v16494: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 542 KiB/s wr, 10 op/s 2023-03-09T23:17:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:17:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:17:37.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:37 smithi122 ceph-mon[93869]: pgmap v16494: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 542 KiB/s wr, 10 op/s 2023-03-09T23:17:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:17:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:17:37.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:37 smithi181 ceph-mon[94189]: pgmap v16494: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 542 KiB/s wr, 10 op/s 2023-03-09T23:17:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:17:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:17:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29968.810475 secs 2023-03-09T23:17:38.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29968.810475 secs 2023-03-09T23:17:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29968.810475 secs 2023-03-09T23:17:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30430.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:17:39.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:39 smithi038 ceph-mon[92422]: pgmap v16495: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 542 KiB/s wr, 11 op/s 2023-03-09T23:17:39.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:39 smithi122 ceph-mon[93869]: pgmap v16495: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 542 KiB/s wr, 11 op/s 2023-03-09T23:17:39.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:39 smithi181 ceph-mon[94189]: pgmap v16495: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 542 KiB/s wr, 11 op/s 2023-03-09T23:17:40.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29971.411437 secs 2023-03-09T23:17:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29971.411437 secs 2023-03-09T23:17:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29971.411437 secs 2023-03-09T23:17:41.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:41 smithi122 ceph-mon[93869]: pgmap v16496: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 542 KiB/s wr, 11 op/s 2023-03-09T23:17:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:41 smithi181 ceph-mon[94189]: pgmap v16496: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 542 KiB/s wr, 11 op/s 2023-03-09T23:17:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:41 smithi038 ceph-mon[92422]: pgmap v16496: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 542 KiB/s wr, 11 op/s 2023-03-09T23:17:42.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:42 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:17:42.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:42 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:17:42.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:42 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:17:42.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:42 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:17:42.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:42 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:17:42.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:42 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:17:42.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:42 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:17:42.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:42 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:17:42.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:42 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:17:43.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:43 smithi122 ceph-mon[93869]: pgmap v16497: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 515 KiB/s rd, 513 KiB/s wr, 10 op/s 2023-03-09T23:17:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:43 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 29973.810618 secs 2023-03-09T23:17:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:43 smithi181 ceph-mon[94189]: pgmap v16497: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 515 KiB/s rd, 513 KiB/s wr, 10 op/s 2023-03-09T23:17:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:43 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 29973.810618 secs 2023-03-09T23:17:43.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:43 smithi038 ceph-mon[92422]: pgmap v16497: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 515 KiB/s rd, 513 KiB/s wr, 10 op/s 2023-03-09T23:17:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:43 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 29973.810618 secs 2023-03-09T23:17:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30435.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:17:45.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:45 smithi122 ceph-mon[93869]: pgmap v16498: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 777 KiB/s rd, 764 KiB/s wr, 9 op/s 2023-03-09T23:17:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29976.411495 secs 2023-03-09T23:17:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:45 smithi181 ceph-mon[94189]: pgmap v16498: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 777 KiB/s rd, 764 KiB/s wr, 9 op/s 2023-03-09T23:17:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29976.411495 secs 2023-03-09T23:17:45.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:45 smithi038 ceph-mon[92422]: pgmap v16498: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 777 KiB/s rd, 764 KiB/s wr, 9 op/s 2023-03-09T23:17:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29976.411495 secs 2023-03-09T23:17:46.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:46 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:17:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:46 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:17:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:46 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:17:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:46 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:17:46.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:46 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:17:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:46 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:17:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:47 smithi122 ceph-mon[93869]: pgmap v16499: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 749 KiB/s rd, 679 KiB/s wr, 12 op/s 2023-03-09T23:17:47.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:47 smithi181 ceph-mon[94189]: pgmap v16499: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 749 KiB/s rd, 679 KiB/s wr, 12 op/s 2023-03-09T23:17:47.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:47 smithi038 ceph-mon[92422]: pgmap v16499: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 749 KiB/s rd, 679 KiB/s wr, 12 op/s 2023-03-09T23:17:48.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:17:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:17:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29978.810761 secs 2023-03-09T23:17:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:17:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:17:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29978.810761 secs 2023-03-09T23:17:48.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:17:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:17:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29978.810761 secs 2023-03-09T23:17:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30440.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:17:49.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:49 smithi122 ceph-mon[93869]: pgmap v16500: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 788 KiB/s rd, 600 KiB/s wr, 11 op/s 2023-03-09T23:17:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:17:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:17:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:49 smithi181 ceph-mon[94189]: pgmap v16500: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 788 KiB/s rd, 600 KiB/s wr, 11 op/s 2023-03-09T23:17:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:17:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:17:49.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:49 smithi038 ceph-mon[92422]: pgmap v16500: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 788 KiB/s rd, 600 KiB/s wr, 11 op/s 2023-03-09T23:17:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:17:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:17:50.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:50 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:17:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:50 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:17:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29981.411556 secs 2023-03-09T23:17:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:50 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:17:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:50 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:17:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29981.411556 secs 2023-03-09T23:17:50.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:50 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:17:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:50 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:17:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29981.411556 secs 2023-03-09T23:17:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:51 smithi122 ceph-mon[93869]: pgmap v16501: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 669 KiB/s rd, 599 KiB/s wr, 10 op/s 2023-03-09T23:17:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:51 smithi181 ceph-mon[94189]: pgmap v16501: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 669 KiB/s rd, 599 KiB/s wr, 10 op/s 2023-03-09T23:17:51.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:51 smithi038 ceph-mon[92422]: pgmap v16501: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 669 KiB/s rd, 599 KiB/s wr, 10 op/s 2023-03-09T23:17:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:17:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:17:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:17:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:17:52.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:17:52.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:17:53.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:53 smithi122 ceph-mon[93869]: pgmap v16502: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 894 KiB/s rd, 745 KiB/s wr, 14 op/s 2023-03-09T23:17:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:17:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:17:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29983.810896 secs 2023-03-09T23:17:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:53 smithi181 ceph-mon[94189]: pgmap v16502: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 894 KiB/s rd, 745 KiB/s wr, 14 op/s 2023-03-09T23:17:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:17:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:17:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29983.810896 secs 2023-03-09T23:17:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:53 smithi038 ceph-mon[92422]: pgmap v16502: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 894 KiB/s rd, 745 KiB/s wr, 14 op/s 2023-03-09T23:17:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:17:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:17:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29983.810896 secs 2023-03-09T23:17:53.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30445.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:17:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:55 smithi122 ceph-mon[93869]: pgmap v16503: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 892 KiB/s rd, 745 KiB/s wr, 11 op/s 2023-03-09T23:17:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29986.411616 secs 2023-03-09T23:17:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:55 smithi181 ceph-mon[94189]: pgmap v16503: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 892 KiB/s rd, 745 KiB/s wr, 11 op/s 2023-03-09T23:17:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29986.411616 secs 2023-03-09T23:17:55.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:55 smithi038 ceph-mon[92422]: pgmap v16503: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 892 KiB/s rd, 745 KiB/s wr, 11 op/s 2023-03-09T23:17:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29986.411616 secs 2023-03-09T23:17:56.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:17:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:17:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:17:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:17:56.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:17:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:17:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:57 smithi122 ceph-mon[93869]: pgmap v16504: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 574 KiB/s wr, 14 op/s 2023-03-09T23:17:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:17:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:17:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:57 smithi181 ceph-mon[94189]: pgmap v16504: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 574 KiB/s wr, 14 op/s 2023-03-09T23:17:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:17:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:17:57.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:57 smithi038 ceph-mon[92422]: pgmap v16504: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 574 KiB/s wr, 14 op/s 2023-03-09T23:17:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:17:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:17:58.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29988.811039 secs 2023-03-09T23:17:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29988.811039 secs 2023-03-09T23:17:58.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29988.811039 secs 2023-03-09T23:17:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30450.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:17:59.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:17:59 smithi122 ceph-mon[93869]: pgmap v16505: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 890 KiB/s wr, 12 op/s 2023-03-09T23:17:59.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:17:59 smithi181 ceph-mon[94189]: pgmap v16505: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 890 KiB/s wr, 12 op/s 2023-03-09T23:17:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:17:59 smithi038 ceph-mon[92422]: pgmap v16505: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 890 KiB/s wr, 12 op/s 2023-03-09T23:18:00.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:18:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:18:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29991.411677 secs 2023-03-09T23:18:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:18:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:18:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29991.411677 secs 2023-03-09T23:18:00.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:18:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:18:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29991.411677 secs 2023-03-09T23:18:01.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:01 smithi122 ceph-mon[93869]: pgmap v16506: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 822 KiB/s rd, 732 KiB/s wr, 10 op/s 2023-03-09T23:18:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:18:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:18:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:01 smithi181 ceph-mon[94189]: pgmap v16506: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 822 KiB/s rd, 732 KiB/s wr, 10 op/s 2023-03-09T23:18:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:18:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:18:01.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:01 smithi038 ceph-mon[92422]: pgmap v16506: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 822 KiB/s rd, 732 KiB/s wr, 10 op/s 2023-03-09T23:18:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:18:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:18:03.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:03 smithi122 ceph-mon[93869]: pgmap v16507: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 16 op/s 2023-03-09T23:18:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29993.811175 secs 2023-03-09T23:18:03.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:03 smithi181 ceph-mon[94189]: pgmap v16507: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 16 op/s 2023-03-09T23:18:03.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29993.811175 secs 2023-03-09T23:18:03.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:03 smithi038 ceph-mon[92422]: pgmap v16507: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 16 op/s 2023-03-09T23:18:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29993.811175 secs 2023-03-09T23:18:03.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30455.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:18:05.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:05 smithi122 ceph-mon[93869]: pgmap v16508: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1003 KiB/s rd, 1.1 MiB/s wr, 26 op/s 2023-03-09T23:18:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:05 smithi122 ceph-mon[93869]: osdmap e1247: 12 total, 12 up, 12 in 2023-03-09T23:18:05.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:05 smithi181 ceph-mon[94189]: pgmap v16508: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1003 KiB/s rd, 1.1 MiB/s wr, 26 op/s 2023-03-09T23:18:05.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:05 smithi181 ceph-mon[94189]: osdmap e1247: 12 total, 12 up, 12 in 2023-03-09T23:18:05.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:05 smithi038 ceph-mon[92422]: pgmap v16508: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1003 KiB/s rd, 1.1 MiB/s wr, 26 op/s 2023-03-09T23:18:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:05 smithi038 ceph-mon[92422]: osdmap e1247: 12 total, 12 up, 12 in 2023-03-09T23:18:06.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29996.411736 secs 2023-03-09T23:18:06.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29996.411736 secs 2023-03-09T23:18:06.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29996.411736 secs 2023-03-09T23:18:07.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:07 smithi122 ceph-mon[93869]: pgmap v16510: 129 pgs: 1 active+clean+snaptrim, 9 active+clean+snaptrim_wait, 119 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 31 op/s 2023-03-09T23:18:07.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:07 smithi181 ceph-mon[94189]: pgmap v16510: 129 pgs: 1 active+clean+snaptrim, 9 active+clean+snaptrim_wait, 119 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 31 op/s 2023-03-09T23:18:07.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:07 smithi038 ceph-mon[92422]: pgmap v16510: 129 pgs: 1 active+clean+snaptrim, 9 active+clean+snaptrim_wait, 119 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 31 op/s 2023-03-09T23:18:08.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:18:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:18:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 29998.811257 secs 2023-03-09T23:18:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:18:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:18:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 29998.811257 secs 2023-03-09T23:18:08.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:18:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:18:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 29998.811257 secs 2023-03-09T23:18:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30460.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:18:09.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:09 smithi122 ceph-mon[93869]: pgmap v16511: 129 pgs: 1 active+clean+snaptrim, 9 active+clean+snaptrim_wait, 119 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 793 KiB/s rd, 834 KiB/s wr, 31 op/s 2023-03-09T23:18:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:18:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:18:09.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:09 smithi181 ceph-mon[94189]: pgmap v16511: 129 pgs: 1 active+clean+snaptrim, 9 active+clean+snaptrim_wait, 119 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 793 KiB/s rd, 834 KiB/s wr, 31 op/s 2023-03-09T23:18:09.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:18:09.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:18:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:09 smithi038 ceph-mon[92422]: pgmap v16511: 129 pgs: 1 active+clean+snaptrim, 9 active+clean+snaptrim_wait, 119 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 793 KiB/s rd, 834 KiB/s wr, 31 op/s 2023-03-09T23:18:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:18:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:18:11.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30001.411796 secs 2023-03-09T23:18:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:11 smithi122 ceph-mon[93869]: pgmap v16512: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 793 KiB/s rd, 834 KiB/s wr, 31 op/s 2023-03-09T23:18:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30001.411796 secs 2023-03-09T23:18:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:11 smithi181 ceph-mon[94189]: pgmap v16512: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 793 KiB/s rd, 834 KiB/s wr, 31 op/s 2023-03-09T23:18:11.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30001.411796 secs 2023-03-09T23:18:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:11 smithi038 ceph-mon[92422]: pgmap v16512: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 793 KiB/s rd, 834 KiB/s wr, 31 op/s 2023-03-09T23:18:13.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:13 smithi122 ceph-mon[93869]: osdmap e1248: 12 total, 12 up, 12 in 2023-03-09T23:18:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:13 smithi122 ceph-mon[93869]: pgmap v16514: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 555 KiB/s rd, 449 KiB/s wr, 13 op/s 2023-03-09T23:18:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30003.811339 secs 2023-03-09T23:18:13.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:13 smithi181 ceph-mon[94189]: osdmap e1248: 12 total, 12 up, 12 in 2023-03-09T23:18:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:13 smithi181 ceph-mon[94189]: pgmap v16514: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 555 KiB/s rd, 449 KiB/s wr, 13 op/s 2023-03-09T23:18:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30003.811339 secs 2023-03-09T23:18:13.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:13 smithi038 ceph-mon[92422]: osdmap e1248: 12 total, 12 up, 12 in 2023-03-09T23:18:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:13 smithi038 ceph-mon[92422]: pgmap v16514: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 555 KiB/s rd, 449 KiB/s wr, 13 op/s 2023-03-09T23:18:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30003.811339 secs 2023-03-09T23:18:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30465.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:18:15.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:15 smithi122 ceph-mon[93869]: pgmap v16515: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 388 KiB/s wr, 12 op/s 2023-03-09T23:18:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:15 smithi181 ceph-mon[94189]: pgmap v16515: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 388 KiB/s wr, 12 op/s 2023-03-09T23:18:15.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:15 smithi038 ceph-mon[92422]: pgmap v16515: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 388 KiB/s wr, 12 op/s 2023-03-09T23:18:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30006.411857 secs 2023-03-09T23:18:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:18:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:18:16.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30006.411857 secs 2023-03-09T23:18:16.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:18:16.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:18:16.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30006.411857 secs 2023-03-09T23:18:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:18:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:18:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:17 smithi122 ceph-mon[93869]: pgmap v16516: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 180 KiB/s wr, 10 op/s 2023-03-09T23:18:17.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:18:17.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:18:17.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:17 smithi181 ceph-mon[94189]: pgmap v16516: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 180 KiB/s wr, 10 op/s 2023-03-09T23:18:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:18:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:18:17.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:17 smithi038 ceph-mon[92422]: pgmap v16516: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 180 KiB/s wr, 10 op/s 2023-03-09T23:18:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:18:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:18:18.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30008.811459 secs 2023-03-09T23:18:18.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30008.811459 secs 2023-03-09T23:18:18.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30008.811459 secs 2023-03-09T23:18:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30470.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:18:19.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:19 smithi038 ceph-mon[92422]: pgmap v16517: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 209 KiB/s wr, 9 op/s 2023-03-09T23:18:19.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:19 smithi122 ceph-mon[93869]: pgmap v16517: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 209 KiB/s wr, 9 op/s 2023-03-09T23:18:19.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:19 smithi181 ceph-mon[94189]: pgmap v16517: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 209 KiB/s wr, 9 op/s 2023-03-09T23:18:20.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:18:20.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:18:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:18:21.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30011.411918 secs 2023-03-09T23:18:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:21 smithi122 ceph-mon[93869]: pgmap v16518: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 209 KiB/s wr, 9 op/s 2023-03-09T23:18:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:18:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:18:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:18:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30011.411918 secs 2023-03-09T23:18:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:21 smithi181 ceph-mon[94189]: pgmap v16518: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 209 KiB/s wr, 9 op/s 2023-03-09T23:18:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:18:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:18:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:18:21.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30011.411918 secs 2023-03-09T23:18:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:21 smithi038 ceph-mon[92422]: pgmap v16518: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 209 KiB/s wr, 9 op/s 2023-03-09T23:18:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:18:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:18:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:18:23.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:23 smithi122 ceph-mon[93869]: pgmap v16519: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 189 KiB/s wr, 9 op/s 2023-03-09T23:18:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30013.811522 secs 2023-03-09T23:18:23.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:23 smithi181 ceph-mon[94189]: pgmap v16519: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 189 KiB/s wr, 9 op/s 2023-03-09T23:18:23.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30013.811522 secs 2023-03-09T23:18:23.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:23 smithi038 ceph-mon[92422]: pgmap v16519: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 189 KiB/s wr, 9 op/s 2023-03-09T23:18:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30013.811522 secs 2023-03-09T23:18:23.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30475.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:18:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:25 smithi038 ceph-mon[92422]: pgmap v16520: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 158 KiB/s wr, 8 op/s 2023-03-09T23:18:25.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:25 smithi122 ceph-mon[93869]: pgmap v16520: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 158 KiB/s wr, 8 op/s 2023-03-09T23:18:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:25 smithi181 ceph-mon[94189]: pgmap v16520: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 158 KiB/s wr, 8 op/s 2023-03-09T23:18:26.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30016.411979 secs 2023-03-09T23:18:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30016.411979 secs 2023-03-09T23:18:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30016.411979 secs 2023-03-09T23:18:27.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:27 smithi122 ceph-mon[93869]: pgmap v16521: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 265 KiB/s wr, 10 op/s 2023-03-09T23:18:27.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:27 smithi181 ceph-mon[94189]: pgmap v16521: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 265 KiB/s wr, 10 op/s 2023-03-09T23:18:27.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:27 smithi038 ceph-mon[92422]: pgmap v16521: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 265 KiB/s wr, 10 op/s 2023-03-09T23:18:28.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30018.811616 secs 2023-03-09T23:18:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30018.811616 secs 2023-03-09T23:18:28.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30018.811616 secs 2023-03-09T23:18:28.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30480.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:18:29.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:29 smithi122 ceph-mon[93869]: pgmap v16522: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 265 KiB/s wr, 8 op/s 2023-03-09T23:18:29.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:29 smithi181 ceph-mon[94189]: pgmap v16522: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 265 KiB/s wr, 8 op/s 2023-03-09T23:18:29.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:29 smithi038 ceph-mon[92422]: pgmap v16522: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 265 KiB/s wr, 8 op/s 2023-03-09T23:18:30.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30021.412039 secs 2023-03-09T23:18:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30021.412039 secs 2023-03-09T23:18:30.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30021.412039 secs 2023-03-09T23:18:31.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:31 smithi122 ceph-mon[93869]: pgmap v16523: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 240 KiB/s wr, 7 op/s 2023-03-09T23:18:31.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:31 smithi181 ceph-mon[94189]: pgmap v16523: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 240 KiB/s wr, 7 op/s 2023-03-09T23:18:31.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:31 smithi038 ceph-mon[92422]: pgmap v16523: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 240 KiB/s wr, 7 op/s 2023-03-09T23:18:33.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:33 smithi122 ceph-mon[93869]: pgmap v16524: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 241 KiB/s wr, 10 op/s 2023-03-09T23:18:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30023.811754 secs 2023-03-09T23:18:33.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:33 smithi181 ceph-mon[94189]: pgmap v16524: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 241 KiB/s wr, 10 op/s 2023-03-09T23:18:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30023.811754 secs 2023-03-09T23:18:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:33 smithi038 ceph-mon[92422]: pgmap v16524: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 241 KiB/s wr, 10 op/s 2023-03-09T23:18:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30023.811754 secs 2023-03-09T23:18:33.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30485.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:18:35.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:35 smithi038 ceph-mon[92422]: pgmap v16525: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 132 KiB/s wr, 8 op/s 2023-03-09T23:18:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30026.412099 secs 2023-03-09T23:18:35.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:35 smithi122 ceph-mon[93869]: pgmap v16525: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 132 KiB/s wr, 8 op/s 2023-03-09T23:18:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30026.412099 secs 2023-03-09T23:18:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:35 smithi181 ceph-mon[94189]: pgmap v16525: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 132 KiB/s wr, 8 op/s 2023-03-09T23:18:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30026.412099 secs 2023-03-09T23:18:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:37 smithi038 ceph-mon[92422]: pgmap v16526: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 132 KiB/s wr, 10 op/s 2023-03-09T23:18:37.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:37 smithi122 ceph-mon[93869]: pgmap v16526: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 132 KiB/s wr, 10 op/s 2023-03-09T23:18:37.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:37 smithi181 ceph-mon[94189]: pgmap v16526: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 132 KiB/s wr, 10 op/s 2023-03-09T23:18:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30028.811870 secs 2023-03-09T23:18:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30028.811870 secs 2023-03-09T23:18:38.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30028.811870 secs 2023-03-09T23:18:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30490.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:18:39.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:39 smithi122 ceph-mon[93869]: pgmap v16527: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.5 KiB/s wr, 8 op/s 2023-03-09T23:18:39.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:39 smithi181 ceph-mon[94189]: pgmap v16527: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.5 KiB/s wr, 8 op/s 2023-03-09T23:18:39.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:39 smithi038 ceph-mon[92422]: pgmap v16527: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 1.5 KiB/s wr, 8 op/s 2023-03-09T23:18:40.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:18:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:18:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30031.412159 secs 2023-03-09T23:18:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:18:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:18:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30031.412159 secs 2023-03-09T23:18:40.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:18:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:18:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30031.412159 secs 2023-03-09T23:18:41.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:41 smithi122 ceph-mon[93869]: pgmap v16528: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 256 KiB/s wr, 8 op/s 2023-03-09T23:18:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:18:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:18:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:41 smithi181 ceph-mon[94189]: pgmap v16528: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 256 KiB/s wr, 8 op/s 2023-03-09T23:18:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:18:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:18:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:41 smithi038 ceph-mon[92422]: pgmap v16528: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 256 KiB/s wr, 8 op/s 2023-03-09T23:18:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:18:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:18:43.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:43 smithi122 ceph-mon[93869]: pgmap v16529: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 256 KiB/s wr, 10 op/s 2023-03-09T23:18:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30033.811994 secs 2023-03-09T23:18:43.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:43 smithi181 ceph-mon[94189]: pgmap v16529: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 256 KiB/s wr, 10 op/s 2023-03-09T23:18:43.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30033.811994 secs 2023-03-09T23:18:43.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:43 smithi038 ceph-mon[92422]: pgmap v16529: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 256 KiB/s wr, 10 op/s 2023-03-09T23:18:43.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30033.811994 secs 2023-03-09T23:18:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30495.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:18:45.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:45 smithi038 ceph-mon[92422]: pgmap v16530: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 255 KiB/s wr, 7 op/s 2023-03-09T23:18:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30036.412220 secs 2023-03-09T23:18:45.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:45 smithi122 ceph-mon[93869]: pgmap v16530: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 255 KiB/s wr, 7 op/s 2023-03-09T23:18:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30036.412220 secs 2023-03-09T23:18:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:45 smithi181 ceph-mon[94189]: pgmap v16530: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 255 KiB/s wr, 7 op/s 2023-03-09T23:18:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30036.412220 secs 2023-03-09T23:18:47.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:47 smithi122 ceph-mon[93869]: pgmap v16531: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 602 KiB/s rd, 401 KiB/s wr, 10 op/s 2023-03-09T23:18:47.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:47 smithi181 ceph-mon[94189]: pgmap v16531: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 602 KiB/s rd, 401 KiB/s wr, 10 op/s 2023-03-09T23:18:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:47 smithi038 ceph-mon[92422]: pgmap v16531: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 602 KiB/s rd, 401 KiB/s wr, 10 op/s 2023-03-09T23:18:48.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30038.812134 secs 2023-03-09T23:18:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30038.812134 secs 2023-03-09T23:18:48.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30038.812134 secs 2023-03-09T23:18:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30500.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:18:49.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:49 smithi122 ceph-mon[93869]: pgmap v16532: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 601 KiB/s rd, 401 KiB/s wr, 8 op/s 2023-03-09T23:18:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:18:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:18:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:49 smithi181 ceph-mon[94189]: pgmap v16532: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 601 KiB/s rd, 401 KiB/s wr, 8 op/s 2023-03-09T23:18:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:18:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:18:49.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:49 smithi038 ceph-mon[92422]: pgmap v16532: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 601 KiB/s rd, 401 KiB/s wr, 8 op/s 2023-03-09T23:18:49.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:18:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:18:50.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30041.412280 secs 2023-03-09T23:18:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30041.412280 secs 2023-03-09T23:18:50.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30041.412280 secs 2023-03-09T23:18:51.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:51 smithi122 ceph-mon[93869]: pgmap v16533: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 667 KiB/s rd, 401 KiB/s wr, 8 op/s 2023-03-09T23:18:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:51 smithi181 ceph-mon[94189]: pgmap v16533: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 667 KiB/s rd, 401 KiB/s wr, 8 op/s 2023-03-09T23:18:51.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:51 smithi038 ceph-mon[92422]: pgmap v16533: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 667 KiB/s rd, 401 KiB/s wr, 8 op/s 2023-03-09T23:18:52.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:18:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:18:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:18:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:18:52.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:18:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:18:53.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:53 smithi122 ceph-mon[93869]: pgmap v16534: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 678 KiB/s rd, 408 KiB/s wr, 10 op/s 2023-03-09T23:18:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30043.812194 secs 2023-03-09T23:18:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:53 smithi181 ceph-mon[94189]: pgmap v16534: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 678 KiB/s rd, 408 KiB/s wr, 10 op/s 2023-03-09T23:18:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30043.812194 secs 2023-03-09T23:18:53.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:53 smithi038 ceph-mon[92422]: pgmap v16534: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 678 KiB/s rd, 408 KiB/s wr, 10 op/s 2023-03-09T23:18:53.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30043.812194 secs 2023-03-09T23:18:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30505.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:18:55.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:55 smithi122 ceph-mon[93869]: pgmap v16535: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 735 KiB/s rd, 435 KiB/s wr, 41 op/s 2023-03-09T23:18:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30046.412342 secs 2023-03-09T23:18:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:55 smithi181 ceph-mon[94189]: pgmap v16535: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 735 KiB/s rd, 435 KiB/s wr, 41 op/s 2023-03-09T23:18:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30046.412342 secs 2023-03-09T23:18:55.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:55 smithi038 ceph-mon[92422]: pgmap v16535: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 735 KiB/s rd, 435 KiB/s wr, 41 op/s 2023-03-09T23:18:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30046.412342 secs 2023-03-09T23:18:56.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:18:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:18:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:18:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:18:56.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:18:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:18:57.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:57 smithi122 ceph-mon[93869]: pgmap v16536: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 857 KiB/s rd, 574 KiB/s wr, 43 op/s 2023-03-09T23:18:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:18:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:18:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:57 smithi181 ceph-mon[94189]: pgmap v16536: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 857 KiB/s rd, 574 KiB/s wr, 43 op/s 2023-03-09T23:18:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:18:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:18:57.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:57 smithi038 ceph-mon[92422]: pgmap v16536: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 857 KiB/s rd, 574 KiB/s wr, 43 op/s 2023-03-09T23:18:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:18:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:18:58.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30048.812317 secs 2023-03-09T23:18:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30048.812317 secs 2023-03-09T23:18:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30048.812317 secs 2023-03-09T23:18:58.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30510.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:18:59.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:18:59 smithi122 ceph-mon[93869]: pgmap v16537: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 557 KiB/s rd, 474 KiB/s wr, 42 op/s 2023-03-09T23:18:59.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:18:59 smithi181 ceph-mon[94189]: pgmap v16537: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 557 KiB/s rd, 474 KiB/s wr, 42 op/s 2023-03-09T23:18:59.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:18:59 smithi038 ceph-mon[92422]: pgmap v16537: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 557 KiB/s rd, 474 KiB/s wr, 42 op/s 2023-03-09T23:19:00.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:19:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:19:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30051.412401 secs 2023-03-09T23:19:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:19:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:19:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30051.412401 secs 2023-03-09T23:19:00.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:19:00.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:19:00.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30051.412401 secs 2023-03-09T23:19:01.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:01 smithi122 ceph-mon[93869]: pgmap v16538: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 915 KiB/s rd, 956 KiB/s wr, 46 op/s 2023-03-09T23:19:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:19:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:19:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:01 smithi181 ceph-mon[94189]: pgmap v16538: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 915 KiB/s rd, 956 KiB/s wr, 46 op/s 2023-03-09T23:19:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:19:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:19:01.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:01 smithi038 ceph-mon[92422]: pgmap v16538: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 915 KiB/s rd, 956 KiB/s wr, 46 op/s 2023-03-09T23:19:01.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:19:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:19:03.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:03 smithi122 ceph-mon[93869]: pgmap v16539: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 851 KiB/s rd, 956 KiB/s wr, 48 op/s 2023-03-09T23:19:03.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:03 smithi181 ceph-mon[94189]: pgmap v16539: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 851 KiB/s rd, 956 KiB/s wr, 48 op/s 2023-03-09T23:19:03.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:03 smithi038 ceph-mon[92422]: pgmap v16539: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 851 KiB/s rd, 956 KiB/s wr, 48 op/s 2023-03-09T23:19:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30515.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:19:05.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:05 smithi122 ceph-mon[93869]: pgmap v16540: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 695 KiB/s wr, 48 op/s 2023-03-09T23:19:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30056.412459 secs 2023-03-09T23:19:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:05 smithi181 ceph-mon[94189]: pgmap v16540: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 695 KiB/s wr, 48 op/s 2023-03-09T23:19:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30056.412459 secs 2023-03-09T23:19:05.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:05 smithi038 ceph-mon[92422]: pgmap v16540: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 695 KiB/s wr, 48 op/s 2023-03-09T23:19:05.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30056.412459 secs 2023-03-09T23:19:06.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:06 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:19:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:06 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:19:06.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:06 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:19:07.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:07 smithi122 ceph-mon[93869]: pgmap v16541: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 695 KiB/s rd, 742 KiB/s wr, 19 op/s 2023-03-09T23:19:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:07 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:19:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:19:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:19:07.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:07 smithi181 ceph-mon[94189]: pgmap v16541: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 695 KiB/s rd, 742 KiB/s wr, 19 op/s 2023-03-09T23:19:07.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:07 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:19:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:19:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:19:07.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:07 smithi038 ceph-mon[92422]: pgmap v16541: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 695 KiB/s rd, 742 KiB/s wr, 19 op/s 2023-03-09T23:19:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:07 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:19:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:19:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:19:08.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30520.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:19:09.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:09 smithi122 ceph-mon[93869]: pgmap v16542: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 766 KiB/s rd, 801 KiB/s wr, 27 op/s 2023-03-09T23:19:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:09 smithi181 ceph-mon[94189]: pgmap v16542: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 766 KiB/s rd, 801 KiB/s wr, 27 op/s 2023-03-09T23:19:09.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:09 smithi038 ceph-mon[92422]: pgmap v16542: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 766 KiB/s rd, 801 KiB/s wr, 27 op/s 2023-03-09T23:19:10.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30061.412521 secs 2023-03-09T23:19:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30061.412521 secs 2023-03-09T23:19:10.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30061.412521 secs 2023-03-09T23:19:11.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:11 smithi122 ceph-mon[93869]: pgmap v16543: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 783 KiB/s rd, 756 KiB/s wr, 26 op/s 2023-03-09T23:19:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:11 smithi181 ceph-mon[94189]: pgmap v16543: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 783 KiB/s rd, 756 KiB/s wr, 26 op/s 2023-03-09T23:19:11.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:11 smithi038 ceph-mon[92422]: pgmap v16543: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 783 KiB/s rd, 756 KiB/s wr, 26 op/s 2023-03-09T23:19:12.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:19:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:19:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:19:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:19:12.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:19:12.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:19:13.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:13 smithi122 ceph-mon[93869]: pgmap v16544: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 629 KiB/s rd, 476 KiB/s wr, 24 op/s 2023-03-09T23:19:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:19:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:19:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:13 smithi181 ceph-mon[94189]: pgmap v16544: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 629 KiB/s rd, 476 KiB/s wr, 24 op/s 2023-03-09T23:19:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:19:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:19:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:13 smithi038 ceph-mon[92422]: pgmap v16544: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 629 KiB/s rd, 476 KiB/s wr, 24 op/s 2023-03-09T23:19:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:19:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:19:13.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30525.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:19:15.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:15 smithi122 ceph-mon[93869]: pgmap v16545: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 652 KiB/s rd, 500 KiB/s wr, 22 op/s 2023-03-09T23:19:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30066.412580 secs 2023-03-09T23:19:15.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:15 smithi181 ceph-mon[94189]: pgmap v16545: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 652 KiB/s rd, 500 KiB/s wr, 22 op/s 2023-03-09T23:19:15.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30066.412580 secs 2023-03-09T23:19:15.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:15 smithi038 ceph-mon[92422]: pgmap v16545: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 652 KiB/s rd, 500 KiB/s wr, 22 op/s 2023-03-09T23:19:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30066.412580 secs 2023-03-09T23:19:17.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:17 smithi122 ceph-mon[93869]: pgmap v16546: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 760 KiB/s rd, 608 KiB/s wr, 22 op/s 2023-03-09T23:19:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:17 smithi181 ceph-mon[94189]: pgmap v16546: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 760 KiB/s rd, 608 KiB/s wr, 22 op/s 2023-03-09T23:19:17.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:17 smithi038 ceph-mon[92422]: pgmap v16546: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 760 KiB/s rd, 608 KiB/s wr, 22 op/s 2023-03-09T23:19:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30530.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:19:19.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:19 smithi122 ceph-mon[93869]: pgmap v16547: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 594 KiB/s rd, 534 KiB/s wr, 19 op/s 2023-03-09T23:19:19.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:19 smithi181 ceph-mon[94189]: pgmap v16547: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 594 KiB/s rd, 534 KiB/s wr, 19 op/s 2023-03-09T23:19:19.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:19 smithi038 ceph-mon[92422]: pgmap v16547: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 594 KiB/s rd, 534 KiB/s wr, 19 op/s 2023-03-09T23:19:20.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:19:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:19:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30071.412639 secs 2023-03-09T23:19:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:19:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:19:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30071.412639 secs 2023-03-09T23:19:20.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:19:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:19:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30071.412639 secs 2023-03-09T23:19:21.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:21 smithi122 ceph-mon[93869]: pgmap v16548: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 336 KiB/s wr, 8 op/s 2023-03-09T23:19:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:19:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:19:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:21 smithi181 ceph-mon[94189]: pgmap v16548: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 336 KiB/s wr, 8 op/s 2023-03-09T23:19:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:19:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:19:21.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:21 smithi038 ceph-mon[92422]: pgmap v16548: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 336 KiB/s wr, 8 op/s 2023-03-09T23:19:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:19:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:19:23.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:23 smithi122 ceph-mon[93869]: pgmap v16549: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 340 KiB/s wr, 10 op/s 2023-03-09T23:19:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:23 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:19:23.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:23 smithi181 ceph-mon[94189]: pgmap v16549: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 340 KiB/s wr, 10 op/s 2023-03-09T23:19:23.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:23 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:19:23.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:23 smithi038 ceph-mon[92422]: pgmap v16549: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 340 KiB/s wr, 10 op/s 2023-03-09T23:19:23.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:23 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:19:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30535.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:19:24.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:19:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:19:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:19:24.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:19:24.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:19:24.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:19:24.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:19:24.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:19:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:19:25.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:25 smithi122 ceph-mon[93869]: pgmap v16550: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 172 KiB/s wr, 9 op/s 2023-03-09T23:19:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:25 smithi122 ceph-mon[93869]: osdmap e1249: 12 total, 12 up, 12 in 2023-03-09T23:19:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30076.412701 secs 2023-03-09T23:19:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:25 smithi181 ceph-mon[94189]: pgmap v16550: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 172 KiB/s wr, 9 op/s 2023-03-09T23:19:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:25 smithi181 ceph-mon[94189]: osdmap e1249: 12 total, 12 up, 12 in 2023-03-09T23:19:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30076.412701 secs 2023-03-09T23:19:25.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:25 smithi038 ceph-mon[92422]: pgmap v16550: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 172 KiB/s wr, 9 op/s 2023-03-09T23:19:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:25 smithi038 ceph-mon[92422]: osdmap e1249: 12 total, 12 up, 12 in 2023-03-09T23:19:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30076.412701 secs 2023-03-09T23:19:27.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:27 smithi122 ceph-mon[93869]: pgmap v16552: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 445 KiB/s wr, 12 op/s 2023-03-09T23:19:27.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:27 smithi181 ceph-mon[94189]: pgmap v16552: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 445 KiB/s wr, 12 op/s 2023-03-09T23:19:27.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:27 smithi038 ceph-mon[92422]: pgmap v16552: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 445 KiB/s wr, 12 op/s 2023-03-09T23:19:28.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30078.813077 secs 2023-03-09T23:19:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:28 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:19:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:28 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:19:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30078.813077 secs 2023-03-09T23:19:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:28 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:19:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:28 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:19:28.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30078.813077 secs 2023-03-09T23:19:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:28 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:19:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:28 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:19:28.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30540.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:19:29.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:29 smithi122 ceph-mon[93869]: pgmap v16553: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 449 KiB/s wr, 13 op/s 2023-03-09T23:19:29.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:29 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:19:29.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:29 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:19:29.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:29 smithi181 ceph-mon[94189]: pgmap v16553: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 449 KiB/s wr, 13 op/s 2023-03-09T23:19:29.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:29 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:19:29.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:29 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:19:29.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:29 smithi038 ceph-mon[92422]: pgmap v16553: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 449 KiB/s wr, 13 op/s 2023-03-09T23:19:29.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:29 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:19:29.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:29 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:19:30.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:30 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:19:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:30 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:19:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:30 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:19:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30081.412761 secs 2023-03-09T23:19:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:30 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:19:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:30 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:19:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:30 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:19:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30081.412761 secs 2023-03-09T23:19:30.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:30 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:19:30.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:30 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:19:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:30 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:19:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30081.412761 secs 2023-03-09T23:19:31.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:31 smithi122 ceph-mon[93869]: pgmap v16554: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 449 KiB/s wr, 13 op/s 2023-03-09T23:19:31.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:31 smithi181 ceph-mon[94189]: pgmap v16554: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 449 KiB/s wr, 13 op/s 2023-03-09T23:19:31.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:31 smithi038 ceph-mon[92422]: pgmap v16554: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 449 KiB/s wr, 13 op/s 2023-03-09T23:19:33.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:33 smithi122 ceph-mon[93869]: osdmap e1250: 12 total, 12 up, 12 in 2023-03-09T23:19:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:33 smithi122 ceph-mon[93869]: pgmap v16556: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 707 KiB/s wr, 13 op/s 2023-03-09T23:19:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30083.813127 secs 2023-03-09T23:19:33.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:33 smithi181 ceph-mon[94189]: osdmap e1250: 12 total, 12 up, 12 in 2023-03-09T23:19:33.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:33 smithi181 ceph-mon[94189]: pgmap v16556: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 707 KiB/s wr, 13 op/s 2023-03-09T23:19:33.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30083.813127 secs 2023-03-09T23:19:33.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:33 smithi038 ceph-mon[92422]: osdmap e1250: 12 total, 12 up, 12 in 2023-03-09T23:19:33.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:33 smithi038 ceph-mon[92422]: pgmap v16556: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 707 KiB/s wr, 13 op/s 2023-03-09T23:19:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30083.813127 secs 2023-03-09T23:19:33.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30545.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:19:35.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:35 smithi122 ceph-mon[93869]: pgmap v16557: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 582 KiB/s rd, 599 KiB/s wr, 12 op/s 2023-03-09T23:19:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30086.412823 secs 2023-03-09T23:19:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:35 smithi181 ceph-mon[94189]: pgmap v16557: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 582 KiB/s rd, 599 KiB/s wr, 12 op/s 2023-03-09T23:19:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30086.412823 secs 2023-03-09T23:19:35.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:35 smithi038 ceph-mon[92422]: pgmap v16557: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 582 KiB/s rd, 599 KiB/s wr, 12 op/s 2023-03-09T23:19:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30086.412823 secs 2023-03-09T23:19:37.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:37 smithi122 ceph-mon[93869]: pgmap v16558: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 304 KiB/s wr, 10 op/s 2023-03-09T23:19:37.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:37 smithi181 ceph-mon[94189]: pgmap v16558: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 304 KiB/s wr, 10 op/s 2023-03-09T23:19:37.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:37 smithi038 ceph-mon[92422]: pgmap v16558: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 304 KiB/s wr, 10 op/s 2023-03-09T23:19:38.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30088.813282 secs 2023-03-09T23:19:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30088.813282 secs 2023-03-09T23:19:38.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30088.813282 secs 2023-03-09T23:19:38.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30550.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:19:39.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:39 smithi122 ceph-mon[93869]: pgmap v16559: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 300 KiB/s wr, 10 op/s 2023-03-09T23:19:39.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:39 smithi181 ceph-mon[94189]: pgmap v16559: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 300 KiB/s wr, 10 op/s 2023-03-09T23:19:39.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:39 smithi038 ceph-mon[92422]: pgmap v16559: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 300 KiB/s wr, 10 op/s 2023-03-09T23:19:40.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:19:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:19:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30091.412900 secs 2023-03-09T23:19:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:19:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:19:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30091.412900 secs 2023-03-09T23:19:40.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:19:40.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:19:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30091.412900 secs 2023-03-09T23:19:41.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:41 smithi122 ceph-mon[93869]: pgmap v16560: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 458 KiB/s wr, 10 op/s 2023-03-09T23:19:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:19:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:19:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:41 smithi181 ceph-mon[94189]: pgmap v16560: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 458 KiB/s wr, 10 op/s 2023-03-09T23:19:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:19:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:19:41.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:41 smithi038 ceph-mon[92422]: pgmap v16560: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 458 KiB/s wr, 10 op/s 2023-03-09T23:19:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:19:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:19:43.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:43 smithi122 ceph-mon[93869]: pgmap v16561: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 294 KiB/s wr, 9 op/s 2023-03-09T23:19:43.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:43 smithi181 ceph-mon[94189]: pgmap v16561: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 294 KiB/s wr, 9 op/s 2023-03-09T23:19:43.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:43 smithi038 ceph-mon[92422]: pgmap v16561: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 294 KiB/s wr, 9 op/s 2023-03-09T23:19:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30555.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:19:45.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:45 smithi122 ceph-mon[93869]: pgmap v16562: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 387 KiB/s wr, 9 op/s 2023-03-09T23:19:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30096.412961 secs 2023-03-09T23:19:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:45 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:19:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:45 smithi181 ceph-mon[94189]: pgmap v16562: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 387 KiB/s wr, 9 op/s 2023-03-09T23:19:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30096.412961 secs 2023-03-09T23:19:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:45 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:19:45.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:45 smithi038 ceph-mon[92422]: pgmap v16562: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 387 KiB/s wr, 9 op/s 2023-03-09T23:19:45.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30096.412961 secs 2023-03-09T23:19:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:45 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:19:46.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:46 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:19:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:46 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:19:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:46 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:19:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:46 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:19:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:46 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:19:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:46 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:19:46.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:46 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:19:46.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:46 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:19:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:46 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:19:47.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:47 smithi122 ceph-mon[93869]: pgmap v16563: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 371 KiB/s wr, 10 op/s 2023-03-09T23:19:47.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:47 smithi181 ceph-mon[94189]: pgmap v16563: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 371 KiB/s wr, 10 op/s 2023-03-09T23:19:47.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:47 smithi038 ceph-mon[92422]: pgmap v16563: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 371 KiB/s wr, 10 op/s 2023-03-09T23:19:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30560.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:19:49.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:49 smithi122 ceph-mon[93869]: pgmap v16564: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 273 KiB/s wr, 7 op/s 2023-03-09T23:19:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:49 smithi181 ceph-mon[94189]: pgmap v16564: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 273 KiB/s wr, 7 op/s 2023-03-09T23:19:49.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:49 smithi038 ceph-mon[92422]: pgmap v16564: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 273 KiB/s wr, 7 op/s 2023-03-09T23:19:50.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30101.412983 secs 2023-03-09T23:19:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30101.412983 secs 2023-03-09T23:19:50.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30101.412983 secs 2023-03-09T23:19:51.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:51 smithi122 ceph-mon[93869]: pgmap v16565: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 326 KiB/s wr, 7 op/s 2023-03-09T23:19:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:51 smithi181 ceph-mon[94189]: pgmap v16565: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 326 KiB/s wr, 7 op/s 2023-03-09T23:19:51.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:51 smithi038 ceph-mon[92422]: pgmap v16565: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 326 KiB/s wr, 7 op/s 2023-03-09T23:19:53.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:53 smithi122 ceph-mon[93869]: pgmap v16566: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 247 KiB/s rd, 195 KiB/s wr, 8 op/s 2023-03-09T23:19:53.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:53 smithi181 ceph-mon[94189]: pgmap v16566: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 247 KiB/s rd, 195 KiB/s wr, 8 op/s 2023-03-09T23:19:53.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:53 smithi038 ceph-mon[92422]: pgmap v16566: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 247 KiB/s rd, 195 KiB/s wr, 8 op/s 2023-03-09T23:19:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30565.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:19:55.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:55 smithi122 ceph-mon[93869]: pgmap v16567: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 525 KiB/s rd, 459 KiB/s wr, 8 op/s 2023-03-09T23:19:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30106.413050 secs 2023-03-09T23:19:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:55 smithi181 ceph-mon[94189]: pgmap v16567: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 525 KiB/s rd, 459 KiB/s wr, 8 op/s 2023-03-09T23:19:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30106.413050 secs 2023-03-09T23:19:55.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:55 smithi038 ceph-mon[92422]: pgmap v16567: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 525 KiB/s rd, 459 KiB/s wr, 8 op/s 2023-03-09T23:19:55.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30106.413050 secs 2023-03-09T23:19:56.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:19:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:19:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:19:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:19:56.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:19:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:19:57.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:57 smithi122 ceph-mon[93869]: pgmap v16568: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 533 KiB/s rd, 452 KiB/s wr, 10 op/s 2023-03-09T23:19:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:19:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:19:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:57 smithi181 ceph-mon[94189]: pgmap v16568: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 533 KiB/s rd, 452 KiB/s wr, 10 op/s 2023-03-09T23:19:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:19:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:19:57.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:57 smithi038 ceph-mon[92422]: pgmap v16568: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 533 KiB/s rd, 452 KiB/s wr, 10 op/s 2023-03-09T23:19:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:19:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:19:58.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30570.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:19:59.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:19:59 smithi122 ceph-mon[93869]: pgmap v16569: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 682 KiB/s rd, 603 KiB/s wr, 10 op/s 2023-03-09T23:19:59.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:19:59 smithi181 ceph-mon[94189]: pgmap v16569: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 682 KiB/s rd, 603 KiB/s wr, 10 op/s 2023-03-09T23:19:59.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:19:59 smithi038 ceph-mon[92422]: pgmap v16569: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 682 KiB/s rd, 603 KiB/s wr, 10 op/s 2023-03-09T23:20:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:20:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:20:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30111.413110 secs 2023-03-09T23:20:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:00 smithi122 ceph-mon[93869]: Health detail: HEALTH_WARN 1 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-09T23:20:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:00 smithi122 ceph-mon[93869]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2023-03-09T23:20:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:00 smithi122 ceph-mon[93869]: mds.h(mds.1): 2 slow requests are blocked > 180 secs 2023-03-09T23:20:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:00 smithi122 ceph-mon[93869]: [WRN] MDS_TRIM: 2 MDSs behind on trimming 2023-03-09T23:20:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:00 smithi122 ceph-mon[93869]: mds.e(mds.0): Behind on trimming (594/128) max_segments: 128, num_segments: 594 2023-03-09T23:20:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:00 smithi122 ceph-mon[93869]: mds.b(mds.0): Behind on trimming (593/128) max_segments: 128, num_segments: 593 2023-03-09T23:20:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:20:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:20:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30111.413110 secs 2023-03-09T23:20:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:00 smithi181 ceph-mon[94189]: Health detail: HEALTH_WARN 1 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-09T23:20:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:00 smithi181 ceph-mon[94189]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2023-03-09T23:20:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:00 smithi181 ceph-mon[94189]: mds.h(mds.1): 2 slow requests are blocked > 180 secs 2023-03-09T23:20:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:00 smithi181 ceph-mon[94189]: [WRN] MDS_TRIM: 2 MDSs behind on trimming 2023-03-09T23:20:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:00 smithi181 ceph-mon[94189]: mds.e(mds.0): Behind on trimming (594/128) max_segments: 128, num_segments: 594 2023-03-09T23:20:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:00 smithi181 ceph-mon[94189]: mds.b(mds.0): Behind on trimming (593/128) max_segments: 128, num_segments: 593 2023-03-09T23:20:00.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:20:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:20:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30111.413110 secs 2023-03-09T23:20:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:00 smithi038 ceph-mon[92422]: Health detail: HEALTH_WARN 1 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-09T23:20:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:00 smithi038 ceph-mon[92422]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2023-03-09T23:20:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:00 smithi038 ceph-mon[92422]: mds.h(mds.1): 2 slow requests are blocked > 180 secs 2023-03-09T23:20:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:00 smithi038 ceph-mon[92422]: [WRN] MDS_TRIM: 2 MDSs behind on trimming 2023-03-09T23:20:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:00 smithi038 ceph-mon[92422]: mds.e(mds.0): Behind on trimming (594/128) max_segments: 128, num_segments: 594 2023-03-09T23:20:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:00 smithi038 ceph-mon[92422]: mds.b(mds.0): Behind on trimming (593/128) max_segments: 128, num_segments: 593 2023-03-09T23:20:01.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:01 smithi122 ceph-mon[93869]: pgmap v16570: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 685 KiB/s rd, 606 KiB/s wr, 10 op/s 2023-03-09T23:20:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:20:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:20:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:01 smithi181 ceph-mon[94189]: pgmap v16570: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 685 KiB/s rd, 606 KiB/s wr, 10 op/s 2023-03-09T23:20:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:20:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:20:01.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:01 smithi038 ceph-mon[92422]: pgmap v16570: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 685 KiB/s rd, 606 KiB/s wr, 10 op/s 2023-03-09T23:20:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:20:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:20:03.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:03 smithi122 ceph-mon[93869]: pgmap v16571: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 792 KiB/s rd, 760 KiB/s wr, 13 op/s 2023-03-09T23:20:03.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:03 smithi181 ceph-mon[94189]: pgmap v16571: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 792 KiB/s rd, 760 KiB/s wr, 13 op/s 2023-03-09T23:20:03.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:03 smithi038 ceph-mon[92422]: pgmap v16571: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 792 KiB/s rd, 760 KiB/s wr, 13 op/s 2023-03-09T23:20:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30575.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:20:05.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:05 smithi122 ceph-mon[93869]: pgmap v16572: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 938 KiB/s rd, 907 KiB/s wr, 14 op/s 2023-03-09T23:20:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30116.413168 secs 2023-03-09T23:20:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:05 smithi181 ceph-mon[94189]: pgmap v16572: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 938 KiB/s rd, 907 KiB/s wr, 14 op/s 2023-03-09T23:20:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30116.413168 secs 2023-03-09T23:20:05.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:05 smithi038 ceph-mon[92422]: pgmap v16572: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 938 KiB/s rd, 907 KiB/s wr, 14 op/s 2023-03-09T23:20:05.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30116.413168 secs 2023-03-09T23:20:07.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:07 smithi122 ceph-mon[93869]: pgmap v16573: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 661 KiB/s rd, 655 KiB/s wr, 16 op/s 2023-03-09T23:20:07.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:07 smithi181 ceph-mon[94189]: pgmap v16573: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 661 KiB/s rd, 655 KiB/s wr, 16 op/s 2023-03-09T23:20:07.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:07 smithi038 ceph-mon[92422]: pgmap v16573: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 661 KiB/s rd, 655 KiB/s wr, 16 op/s 2023-03-09T23:20:08.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:20:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:20:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:20:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:20:08.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:20:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:20:08.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30580.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:20:09.560 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:09 smithi038 ceph-mon[92422]: pgmap v16574: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 542 KiB/s rd, 555 KiB/s wr, 24 op/s 2023-03-09T23:20:09.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:09 smithi122 ceph-mon[93869]: pgmap v16574: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 542 KiB/s rd, 555 KiB/s wr, 24 op/s 2023-03-09T23:20:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:09 smithi181 ceph-mon[94189]: pgmap v16574: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 542 KiB/s rd, 555 KiB/s wr, 24 op/s 2023-03-09T23:20:10.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30121.413228 secs 2023-03-09T23:20:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30121.413228 secs 2023-03-09T23:20:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30121.413228 secs 2023-03-09T23:20:11.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:11 smithi122 ceph-mon[93869]: pgmap v16575: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 596 KiB/s wr, 23 op/s 2023-03-09T23:20:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:11 smithi181 ceph-mon[94189]: pgmap v16575: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 596 KiB/s wr, 23 op/s 2023-03-09T23:20:11.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:11 smithi038 ceph-mon[92422]: pgmap v16575: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 596 KiB/s wr, 23 op/s 2023-03-09T23:20:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:13 smithi122 ceph-mon[93869]: pgmap v16576: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 763 KiB/s wr, 26 op/s 2023-03-09T23:20:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:20:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:20:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:13 smithi181 ceph-mon[94189]: pgmap v16576: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 763 KiB/s wr, 26 op/s 2023-03-09T23:20:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:20:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:20:13.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:13 smithi038 ceph-mon[92422]: pgmap v16576: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 763 KiB/s wr, 26 op/s 2023-03-09T23:20:13.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:20:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:20:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30585.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:20:15.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:15 smithi122 ceph-mon[93869]: pgmap v16577: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 556 KiB/s wr, 23 op/s 2023-03-09T23:20:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30126.413291 secs 2023-03-09T23:20:15.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:15 smithi181 ceph-mon[94189]: pgmap v16577: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 556 KiB/s wr, 23 op/s 2023-03-09T23:20:15.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30126.413291 secs 2023-03-09T23:20:15.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:15 smithi038 ceph-mon[92422]: pgmap v16577: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 556 KiB/s wr, 23 op/s 2023-03-09T23:20:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30126.413291 secs 2023-03-09T23:20:16.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:20:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:20:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:20:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:20:16.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:20:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:20:17.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:17 smithi122 ceph-mon[93869]: pgmap v16578: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 518 KiB/s wr, 22 op/s 2023-03-09T23:20:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:17 smithi181 ceph-mon[94189]: pgmap v16578: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 518 KiB/s wr, 22 op/s 2023-03-09T23:20:17.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:17 smithi038 ceph-mon[92422]: pgmap v16578: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 518 KiB/s wr, 22 op/s 2023-03-09T23:20:18.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30590.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:20:19.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:19 smithi122 ceph-mon[93869]: pgmap v16579: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 530 KiB/s wr, 19 op/s 2023-03-09T23:20:19.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:19 smithi181 ceph-mon[94189]: pgmap v16579: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 530 KiB/s wr, 19 op/s 2023-03-09T23:20:19.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:19 smithi038 ceph-mon[92422]: pgmap v16579: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 530 KiB/s wr, 19 op/s 2023-03-09T23:20:20.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30131.413352 secs 2023-03-09T23:20:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30131.413352 secs 2023-03-09T23:20:20.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30131.413352 secs 2023-03-09T23:20:21.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:21 smithi122 ceph-mon[93869]: pgmap v16580: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 496 KiB/s wr, 9 op/s 2023-03-09T23:20:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:21 smithi181 ceph-mon[94189]: pgmap v16580: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 496 KiB/s wr, 9 op/s 2023-03-09T23:20:21.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:21 smithi038 ceph-mon[92422]: pgmap v16580: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 496 KiB/s wr, 9 op/s 2023-03-09T23:20:23.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:23 smithi122 ceph-mon[93869]: pgmap v16581: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 494 KiB/s rd, 375 KiB/s wr, 10 op/s 2023-03-09T23:20:23.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:23 smithi181 ceph-mon[94189]: pgmap v16581: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 494 KiB/s rd, 375 KiB/s wr, 10 op/s 2023-03-09T23:20:23.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:23 smithi038 ceph-mon[92422]: pgmap v16581: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 494 KiB/s rd, 375 KiB/s wr, 10 op/s 2023-03-09T23:20:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30595.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:20:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:20:24.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:20:24.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:20:25.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:25 smithi122 ceph-mon[93869]: pgmap v16582: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 241 KiB/s wr, 10 op/s 2023-03-09T23:20:25.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:25 smithi181 ceph-mon[94189]: pgmap v16582: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 241 KiB/s wr, 10 op/s 2023-03-09T23:20:25.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:25 smithi038 ceph-mon[92422]: pgmap v16582: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 241 KiB/s wr, 10 op/s 2023-03-09T23:20:26.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30136.413415 secs 2023-03-09T23:20:26.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30136.413415 secs 2023-03-09T23:20:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30136.413415 secs 2023-03-09T23:20:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:27 smithi122 ceph-mon[93869]: pgmap v16583: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 536 KiB/s rd, 555 KiB/s wr, 13 op/s 2023-03-09T23:20:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:20:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:20:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:20:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:20:27.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:20:27.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:20:27.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:20:27.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:20:27.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:20:27.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:20:27.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:20:27.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:20:27.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:20:27.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:20:27.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:20:27.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:20:27.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:20:27.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:20:27.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:27 smithi181 ceph-mon[94189]: pgmap v16583: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 536 KiB/s rd, 555 KiB/s wr, 13 op/s 2023-03-09T23:20:27.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:20:27.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:20:27.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:20:27.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:20:27.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:20:27.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:20:27.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:20:27.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:20:27.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:20:27.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:20:27.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:20:27.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:20:27.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:20:27.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:20:27.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:20:27.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:20:27.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:20:27.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:20:27.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:27 smithi038 ceph-mon[92422]: pgmap v16583: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 536 KiB/s rd, 555 KiB/s wr, 13 op/s 2023-03-09T23:20:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:20:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:20:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:20:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:20:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:20:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:20:27.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:20:27.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:20:27.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:20:27.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:20:27.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:20:27.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:20:27.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:20:27.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:20:27.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:20:27.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:20:27.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:20:27.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:20:28.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:28 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 192.8M 2023-03-09T23:20:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:28 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi181 to 202247708: error parsing value: Value '202247708' is below minimum 939524096 2023-03-09T23:20:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:28 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T23:20:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:28 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T23:20:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30138.814434 secs 2023-03-09T23:20:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:28 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:20:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:28 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:20:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:28 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 192.8M 2023-03-09T23:20:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:28 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi181 to 202247708: error parsing value: Value '202247708' is below minimum 939524096 2023-03-09T23:20:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:28 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T23:20:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:28 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T23:20:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30138.814434 secs 2023-03-09T23:20:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:28 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:20:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:28 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:20:28.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:28 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 192.8M 2023-03-09T23:20:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:28 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi181 to 202247708: error parsing value: Value '202247708' is below minimum 939524096 2023-03-09T23:20:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:28 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T23:20:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:28 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T23:20:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30138.814434 secs 2023-03-09T23:20:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:28 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:20:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:28 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:20:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30600.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:20:29.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:29 smithi122 ceph-mon[93869]: pgmap v16584: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 445 KiB/s wr, 12 op/s 2023-03-09T23:20:29.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:29 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:20:29.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:29 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:20:29.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:29 smithi181 ceph-mon[94189]: pgmap v16584: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 445 KiB/s wr, 12 op/s 2023-03-09T23:20:29.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:29 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:20:29.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:29 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:20:29.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:29 smithi038 ceph-mon[92422]: pgmap v16584: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 445 KiB/s wr, 12 op/s 2023-03-09T23:20:29.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:29 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:20:29.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:29 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:20:30.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:30 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:20:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:30 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:20:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:30 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:20:30.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:30 smithi122 ceph-mon[93869]: osdmap e1251: 12 total, 12 up, 12 in 2023-03-09T23:20:30.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30141.413473 secs 2023-03-09T23:20:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:30 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:20:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:30 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:20:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:30 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:20:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:30 smithi181 ceph-mon[94189]: osdmap e1251: 12 total, 12 up, 12 in 2023-03-09T23:20:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30141.413473 secs 2023-03-09T23:20:30.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:30 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:20:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:30 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:20:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:30 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:20:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:30 smithi038 ceph-mon[92422]: osdmap e1251: 12 total, 12 up, 12 in 2023-03-09T23:20:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30141.413473 secs 2023-03-09T23:20:31.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:31 smithi122 ceph-mon[93869]: pgmap v16586: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 486 KiB/s rd, 506 KiB/s wr, 13 op/s 2023-03-09T23:20:31.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:31 smithi181 ceph-mon[94189]: pgmap v16586: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 486 KiB/s rd, 506 KiB/s wr, 13 op/s 2023-03-09T23:20:31.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:31 smithi038 ceph-mon[92422]: pgmap v16586: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 486 KiB/s rd, 506 KiB/s wr, 13 op/s 2023-03-09T23:20:33.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:33 smithi038 ceph-mon[92422]: pgmap v16587: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 440 KiB/s rd, 461 KiB/s wr, 13 op/s 2023-03-09T23:20:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30143.814530 secs 2023-03-09T23:20:33.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:33 smithi122 ceph-mon[93869]: pgmap v16587: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 440 KiB/s rd, 461 KiB/s wr, 13 op/s 2023-03-09T23:20:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30143.814530 secs 2023-03-09T23:20:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:33 smithi181 ceph-mon[94189]: pgmap v16587: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 440 KiB/s rd, 461 KiB/s wr, 13 op/s 2023-03-09T23:20:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30143.814530 secs 2023-03-09T23:20:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30605.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:20:35.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:35 smithi122 ceph-mon[93869]: pgmap v16588: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 481 KiB/s rd, 647 KiB/s wr, 11 op/s 2023-03-09T23:20:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30146.413532 secs 2023-03-09T23:20:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:35 smithi181 ceph-mon[94189]: pgmap v16588: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 481 KiB/s rd, 647 KiB/s wr, 11 op/s 2023-03-09T23:20:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30146.413532 secs 2023-03-09T23:20:35.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:35 smithi038 ceph-mon[92422]: pgmap v16588: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 481 KiB/s rd, 647 KiB/s wr, 11 op/s 2023-03-09T23:20:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30146.413532 secs 2023-03-09T23:20:36.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:20:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:20:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:20:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:20:36.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:20:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:20:37.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:37 smithi122 ceph-mon[93869]: pgmap v16589: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 297 KiB/s wr, 10 op/s 2023-03-09T23:20:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:20:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:20:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:37 smithi122 ceph-mon[93869]: osdmap e1252: 12 total, 12 up, 12 in 2023-03-09T23:20:37.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:37 smithi181 ceph-mon[94189]: pgmap v16589: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 297 KiB/s wr, 10 op/s 2023-03-09T23:20:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:20:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:20:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:37 smithi181 ceph-mon[94189]: osdmap e1252: 12 total, 12 up, 12 in 2023-03-09T23:20:37.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:37 smithi038 ceph-mon[92422]: pgmap v16589: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 297 KiB/s wr, 10 op/s 2023-03-09T23:20:37.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:20:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:20:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:37 smithi038 ceph-mon[92422]: osdmap e1252: 12 total, 12 up, 12 in 2023-03-09T23:20:38.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30148.814643 secs 2023-03-09T23:20:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30148.814643 secs 2023-03-09T23:20:38.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30148.814643 secs 2023-03-09T23:20:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30610.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:20:39.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:39 smithi122 ceph-mon[93869]: pgmap v16591: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 337 KiB/s wr, 11 op/s 2023-03-09T23:20:39.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:39 smithi181 ceph-mon[94189]: pgmap v16591: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 337 KiB/s wr, 11 op/s 2023-03-09T23:20:39.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:39 smithi038 ceph-mon[92422]: pgmap v16591: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 337 KiB/s wr, 11 op/s 2023-03-09T23:20:40.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30151.413593 secs 2023-03-09T23:20:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30151.413593 secs 2023-03-09T23:20:40.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30151.413593 secs 2023-03-09T23:20:41.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:41 smithi122 ceph-mon[93869]: pgmap v16592: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 297 KiB/s wr, 10 op/s 2023-03-09T23:20:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:41 smithi181 ceph-mon[94189]: pgmap v16592: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 297 KiB/s wr, 10 op/s 2023-03-09T23:20:41.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:41 smithi038 ceph-mon[92422]: pgmap v16592: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 297 KiB/s wr, 10 op/s 2023-03-09T23:20:43.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:43 smithi038 ceph-mon[92422]: pgmap v16593: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 256 KiB/s wr, 9 op/s 2023-03-09T23:20:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:43 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 30153.814771 secs 2023-03-09T23:20:43.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:43 smithi122 ceph-mon[93869]: pgmap v16593: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 256 KiB/s wr, 9 op/s 2023-03-09T23:20:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:43 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 30153.814771 secs 2023-03-09T23:20:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:43 smithi181 ceph-mon[94189]: pgmap v16593: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 256 KiB/s wr, 9 op/s 2023-03-09T23:20:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:43 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 30153.814771 secs 2023-03-09T23:20:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30615.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:20:45.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:45 smithi122 ceph-mon[93869]: pgmap v16594: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 26 KiB/s wr, 8 op/s 2023-03-09T23:20:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30156.413678 secs 2023-03-09T23:20:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:45 smithi181 ceph-mon[94189]: pgmap v16594: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 26 KiB/s wr, 8 op/s 2023-03-09T23:20:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30156.413678 secs 2023-03-09T23:20:45.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:45 smithi038 ceph-mon[92422]: pgmap v16594: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 26 KiB/s wr, 8 op/s 2023-03-09T23:20:45.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30156.413678 secs 2023-03-09T23:20:46.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:46 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:20:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:46 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:20:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:46 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:20:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:46 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:20:46.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:46 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:20:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:46 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:20:47.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:47 smithi122 ceph-mon[93869]: pgmap v16595: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 716 B/s wr, 8 op/s 2023-03-09T23:20:47.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:47 smithi181 ceph-mon[94189]: pgmap v16595: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 716 B/s wr, 8 op/s 2023-03-09T23:20:47.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:47 smithi038 ceph-mon[92422]: pgmap v16595: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 716 B/s wr, 8 op/s 2023-03-09T23:20:48.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:48 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 30158.814930 secs 2023-03-09T23:20:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:48 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 30158.814930 secs 2023-03-09T23:20:48.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:48 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 30158.814930 secs 2023-03-09T23:20:48.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30620.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:20:49.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:49 smithi122 ceph-mon[93869]: pgmap v16596: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 164 KiB/s wr, 8 op/s 2023-03-09T23:20:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:49 smithi181 ceph-mon[94189]: pgmap v16596: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 164 KiB/s wr, 8 op/s 2023-03-09T23:20:49.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:49 smithi038 ceph-mon[92422]: pgmap v16596: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 164 KiB/s wr, 8 op/s 2023-03-09T23:20:50.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30161.413738 secs 2023-03-09T23:20:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30161.413738 secs 2023-03-09T23:20:50.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30161.413738 secs 2023-03-09T23:20:51.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:51 smithi122 ceph-mon[93869]: pgmap v16597: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 150 KiB/s wr, 6 op/s 2023-03-09T23:20:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:51 smithi181 ceph-mon[94189]: pgmap v16597: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 150 KiB/s wr, 6 op/s 2023-03-09T23:20:51.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:51 smithi038 ceph-mon[92422]: pgmap v16597: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 150 KiB/s wr, 6 op/s 2023-03-09T23:20:53.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:53 smithi038 ceph-mon[92422]: pgmap v16598: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 175 KiB/s wr, 9 op/s 2023-03-09T23:20:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:53 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 30163.815087 secs 2023-03-09T23:20:53.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:53 smithi122 ceph-mon[93869]: pgmap v16598: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 175 KiB/s wr, 9 op/s 2023-03-09T23:20:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:53 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 30163.815087 secs 2023-03-09T23:20:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:53 smithi181 ceph-mon[94189]: pgmap v16598: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 175 KiB/s wr, 9 op/s 2023-03-09T23:20:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:53 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 30163.815087 secs 2023-03-09T23:20:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30625.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:20:55.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:55 smithi122 ceph-mon[93869]: pgmap v16599: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 445 KiB/s rd, 428 KiB/s wr, 7 op/s 2023-03-09T23:20:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30166.413798 secs 2023-03-09T23:20:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:55 smithi181 ceph-mon[94189]: pgmap v16599: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 445 KiB/s rd, 428 KiB/s wr, 7 op/s 2023-03-09T23:20:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30166.413798 secs 2023-03-09T23:20:55.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:55 smithi038 ceph-mon[92422]: pgmap v16599: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 445 KiB/s rd, 428 KiB/s wr, 7 op/s 2023-03-09T23:20:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30166.413798 secs 2023-03-09T23:20:56.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:20:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:20:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:20:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:20:56.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:20:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:20:57.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:57 smithi122 ceph-mon[93869]: pgmap v16600: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 485 KiB/s rd, 453 KiB/s wr, 10 op/s 2023-03-09T23:20:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:20:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:20:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:57 smithi181 ceph-mon[94189]: pgmap v16600: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 485 KiB/s rd, 453 KiB/s wr, 10 op/s 2023-03-09T23:20:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:20:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:20:57.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:57 smithi038 ceph-mon[92422]: pgmap v16600: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 485 KiB/s rd, 453 KiB/s wr, 10 op/s 2023-03-09T23:20:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:20:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:20:58.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30168.815224 secs 2023-03-09T23:20:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30168.815224 secs 2023-03-09T23:20:58.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30168.815224 secs 2023-03-09T23:20:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30630.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:20:59.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:20:59 smithi122 ceph-mon[93869]: pgmap v16601: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 665 KiB/s rd, 541 KiB/s wr, 10 op/s 2023-03-09T23:20:59.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:20:59 smithi181 ceph-mon[94189]: pgmap v16601: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 665 KiB/s rd, 541 KiB/s wr, 10 op/s 2023-03-09T23:20:59.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:20:59 smithi038 ceph-mon[92422]: pgmap v16601: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 665 KiB/s rd, 541 KiB/s wr, 10 op/s 2023-03-09T23:21:00.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30171.413859 secs 2023-03-09T23:21:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30171.413859 secs 2023-03-09T23:21:00.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30171.413859 secs 2023-03-09T23:21:01.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:01 smithi122 ceph-mon[93869]: pgmap v16602: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 391 KiB/s wr, 9 op/s 2023-03-09T23:21:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:21:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:21:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:01 smithi181 ceph-mon[94189]: pgmap v16602: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 391 KiB/s wr, 9 op/s 2023-03-09T23:21:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:21:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:21:01.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:01 smithi038 ceph-mon[92422]: pgmap v16602: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 391 KiB/s wr, 9 op/s 2023-03-09T23:21:01.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:21:01.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:21:02.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:02 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:21:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:02 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:21:02.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:02 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:21:02.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:02 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:21:02.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:02 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:21:02.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:02 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:21:03.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:03 smithi122 ceph-mon[93869]: pgmap v16603: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 875 KiB/s rd, 769 KiB/s wr, 14 op/s 2023-03-09T23:21:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30173.815341 secs 2023-03-09T23:21:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:03 smithi181 ceph-mon[94189]: pgmap v16603: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 875 KiB/s rd, 769 KiB/s wr, 14 op/s 2023-03-09T23:21:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30173.815341 secs 2023-03-09T23:21:03.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:03 smithi038 ceph-mon[92422]: pgmap v16603: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 875 KiB/s rd, 769 KiB/s wr, 14 op/s 2023-03-09T23:21:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30173.815341 secs 2023-03-09T23:21:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30635.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:21:05.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:05 smithi122 ceph-mon[93869]: pgmap v16604: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 927 KiB/s wr, 24 op/s 2023-03-09T23:21:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30176.413919 secs 2023-03-09T23:21:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:05 smithi181 ceph-mon[94189]: pgmap v16604: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 927 KiB/s wr, 24 op/s 2023-03-09T23:21:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30176.413919 secs 2023-03-09T23:21:05.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:05 smithi038 ceph-mon[92422]: pgmap v16604: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 927 KiB/s wr, 24 op/s 2023-03-09T23:21:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30176.413919 secs 2023-03-09T23:21:07.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:07 smithi122 ceph-mon[93869]: pgmap v16605: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 769 KiB/s rd, 674 KiB/s wr, 26 op/s 2023-03-09T23:21:07.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:07 smithi181 ceph-mon[94189]: pgmap v16605: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 769 KiB/s rd, 674 KiB/s wr, 26 op/s 2023-03-09T23:21:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:07 smithi038 ceph-mon[92422]: pgmap v16605: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 769 KiB/s rd, 674 KiB/s wr, 26 op/s 2023-03-09T23:21:08.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30178.815415 secs 2023-03-09T23:21:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:21:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:21:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30178.815415 secs 2023-03-09T23:21:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:21:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:21:08.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30178.815415 secs 2023-03-09T23:21:08.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:21:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:21:08.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30640.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:21:09.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:09 smithi122 ceph-mon[93869]: pgmap v16606: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 730 KiB/s rd, 649 KiB/s wr, 24 op/s 2023-03-09T23:21:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:21:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:21:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:09 smithi181 ceph-mon[94189]: pgmap v16606: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 730 KiB/s rd, 649 KiB/s wr, 24 op/s 2023-03-09T23:21:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:21:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:21:09.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:09 smithi038 ceph-mon[92422]: pgmap v16606: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 730 KiB/s rd, 649 KiB/s wr, 24 op/s 2023-03-09T23:21:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:21:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:21:10.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30181.413992 secs 2023-03-09T23:21:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30181.413992 secs 2023-03-09T23:21:10.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30181.413992 secs 2023-03-09T23:21:11.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:11 smithi122 ceph-mon[93869]: pgmap v16607: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 676 KiB/s rd, 689 KiB/s wr, 22 op/s 2023-03-09T23:21:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:11 smithi181 ceph-mon[94189]: pgmap v16607: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 676 KiB/s rd, 689 KiB/s wr, 22 op/s 2023-03-09T23:21:11.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:11 smithi038 ceph-mon[92422]: pgmap v16607: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 676 KiB/s rd, 689 KiB/s wr, 22 op/s 2023-03-09T23:21:13.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:13 smithi038 ceph-mon[92422]: pgmap v16608: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 799 KiB/s rd, 810 KiB/s wr, 25 op/s 2023-03-09T23:21:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30183.815521 secs 2023-03-09T23:21:13.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:13 smithi122 ceph-mon[93869]: pgmap v16608: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 799 KiB/s rd, 810 KiB/s wr, 25 op/s 2023-03-09T23:21:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30183.815521 secs 2023-03-09T23:21:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:13 smithi181 ceph-mon[94189]: pgmap v16608: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 799 KiB/s rd, 810 KiB/s wr, 25 op/s 2023-03-09T23:21:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30183.815521 secs 2023-03-09T23:21:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30645.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:21:15.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:15 smithi122 ceph-mon[93869]: pgmap v16609: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 472 KiB/s rd, 475 KiB/s wr, 20 op/s 2023-03-09T23:21:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30186.414050 secs 2023-03-09T23:21:15.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:15 smithi181 ceph-mon[94189]: pgmap v16609: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 472 KiB/s rd, 475 KiB/s wr, 20 op/s 2023-03-09T23:21:15.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30186.414050 secs 2023-03-09T23:21:15.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:15 smithi038 ceph-mon[92422]: pgmap v16609: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 472 KiB/s rd, 475 KiB/s wr, 20 op/s 2023-03-09T23:21:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30186.414050 secs 2023-03-09T23:21:16.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:21:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:21:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:21:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:21:16.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:21:16.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:21:17.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:17 smithi122 ceph-mon[93869]: pgmap v16610: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 293 KiB/s wr, 11 op/s 2023-03-09T23:21:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:21:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:21:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:17 smithi181 ceph-mon[94189]: pgmap v16610: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 293 KiB/s wr, 11 op/s 2023-03-09T23:21:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:21:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:21:17.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:17 smithi038 ceph-mon[92422]: pgmap v16610: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 293 KiB/s wr, 11 op/s 2023-03-09T23:21:17.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:21:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:21:18.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30188.815663 secs 2023-03-09T23:21:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30188.815663 secs 2023-03-09T23:21:18.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30188.815663 secs 2023-03-09T23:21:18.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30650.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:21:19.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:19 smithi122 ceph-mon[93869]: pgmap v16611: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 303 KiB/s wr, 10 op/s 2023-03-09T23:21:19.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:19 smithi181 ceph-mon[94189]: pgmap v16611: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 303 KiB/s wr, 10 op/s 2023-03-09T23:21:19.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:19 smithi038 ceph-mon[92422]: pgmap v16611: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 303 KiB/s wr, 10 op/s 2023-03-09T23:21:20.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30191.414114 secs 2023-03-09T23:21:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30191.414114 secs 2023-03-09T23:21:20.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30191.414114 secs 2023-03-09T23:21:21.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:21 smithi122 ceph-mon[93869]: pgmap v16612: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 303 KiB/s wr, 9 op/s 2023-03-09T23:21:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:21:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:21:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:21 smithi181 ceph-mon[94189]: pgmap v16612: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 303 KiB/s wr, 9 op/s 2023-03-09T23:21:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:21:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:21:21.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:21 smithi038 ceph-mon[92422]: pgmap v16612: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 303 KiB/s wr, 9 op/s 2023-03-09T23:21:21.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:21:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:21:23.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:23 smithi038 ceph-mon[92422]: pgmap v16613: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 552 KiB/s wr, 13 op/s 2023-03-09T23:21:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30193.815764 secs 2023-03-09T23:21:23.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:23 smithi122 ceph-mon[93869]: pgmap v16613: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 552 KiB/s wr, 13 op/s 2023-03-09T23:21:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30193.815764 secs 2023-03-09T23:21:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:23 smithi181 ceph-mon[94189]: pgmap v16613: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 552 KiB/s wr, 13 op/s 2023-03-09T23:21:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30193.815764 secs 2023-03-09T23:21:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30655.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:21:25.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:25 smithi122 ceph-mon[93869]: osdmap e1253: 12 total, 12 up, 12 in 2023-03-09T23:21:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:25 smithi122 ceph-mon[93869]: pgmap v16615: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 443 KiB/s rd, 467 KiB/s wr, 12 op/s 2023-03-09T23:21:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:25 smithi181 ceph-mon[94189]: osdmap e1253: 12 total, 12 up, 12 in 2023-03-09T23:21:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:25 smithi181 ceph-mon[94189]: pgmap v16615: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 443 KiB/s rd, 467 KiB/s wr, 12 op/s 2023-03-09T23:21:25.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:25 smithi038 ceph-mon[92422]: osdmap e1253: 12 total, 12 up, 12 in 2023-03-09T23:21:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:25 smithi038 ceph-mon[92422]: pgmap v16615: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 443 KiB/s rd, 467 KiB/s wr, 12 op/s 2023-03-09T23:21:26.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30196.414173 secs 2023-03-09T23:21:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30196.414173 secs 2023-03-09T23:21:26.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30196.414173 secs 2023-03-09T23:21:27.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:27 smithi122 ceph-mon[93869]: pgmap v16616: 129 pgs: 1 active+clean+snaptrim, 9 active+clean+snaptrim_wait, 119 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 599 KiB/s wr, 13 op/s 2023-03-09T23:21:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:21:27.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:27 smithi181 ceph-mon[94189]: pgmap v16616: 129 pgs: 1 active+clean+snaptrim, 9 active+clean+snaptrim_wait, 119 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 599 KiB/s wr, 13 op/s 2023-03-09T23:21:27.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:21:27.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:27 smithi038 ceph-mon[92422]: pgmap v16616: 129 pgs: 1 active+clean+snaptrim, 9 active+clean+snaptrim_wait, 119 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 599 KiB/s wr, 13 op/s 2023-03-09T23:21:27.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:21:28.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:28 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:21:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:28 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:21:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30198.815860 secs 2023-03-09T23:21:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:21:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:21:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:21:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:28 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:21:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:28 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:21:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30198.815860 secs 2023-03-09T23:21:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:21:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:21:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:21:28.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:28 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:21:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:28 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:21:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30198.815860 secs 2023-03-09T23:21:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:21:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:21:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:21:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30660.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:21:29.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:29 smithi122 ceph-mon[93869]: pgmap v16617: 129 pgs: 1 active+clean+snaptrim, 9 active+clean+snaptrim_wait, 119 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 586 KiB/s wr, 11 op/s 2023-03-09T23:21:29.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:29 smithi181 ceph-mon[94189]: pgmap v16617: 129 pgs: 1 active+clean+snaptrim, 9 active+clean+snaptrim_wait, 119 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 586 KiB/s wr, 11 op/s 2023-03-09T23:21:29.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:29 smithi038 ceph-mon[92422]: pgmap v16617: 129 pgs: 1 active+clean+snaptrim, 9 active+clean+snaptrim_wait, 119 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 586 KiB/s wr, 11 op/s 2023-03-09T23:21:30.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30201.414233 secs 2023-03-09T23:21:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30201.414233 secs 2023-03-09T23:21:30.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30201.414233 secs 2023-03-09T23:21:31.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:31 smithi122 ceph-mon[93869]: pgmap v16618: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 604 KiB/s rd, 617 KiB/s wr, 12 op/s 2023-03-09T23:21:31.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:31 smithi181 ceph-mon[94189]: pgmap v16618: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 604 KiB/s rd, 617 KiB/s wr, 12 op/s 2023-03-09T23:21:31.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:31 smithi038 ceph-mon[92422]: pgmap v16618: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 604 KiB/s rd, 617 KiB/s wr, 12 op/s 2023-03-09T23:21:33.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:33 smithi038 ceph-mon[92422]: osdmap e1254: 12 total, 12 up, 12 in 2023-03-09T23:21:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:33 smithi038 ceph-mon[92422]: pgmap v16620: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 371 KiB/s wr, 11 op/s 2023-03-09T23:21:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30203.815949 secs 2023-03-09T23:21:33.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:33 smithi122 ceph-mon[93869]: osdmap e1254: 12 total, 12 up, 12 in 2023-03-09T23:21:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:33 smithi122 ceph-mon[93869]: pgmap v16620: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 371 KiB/s wr, 11 op/s 2023-03-09T23:21:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30203.815949 secs 2023-03-09T23:21:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:33 smithi181 ceph-mon[94189]: osdmap e1254: 12 total, 12 up, 12 in 2023-03-09T23:21:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:33 smithi181 ceph-mon[94189]: pgmap v16620: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 371 KiB/s wr, 11 op/s 2023-03-09T23:21:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30203.815949 secs 2023-03-09T23:21:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30665.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:21:35.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:35 smithi122 ceph-mon[93869]: pgmap v16621: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 297 KiB/s wr, 10 op/s 2023-03-09T23:21:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30206.414292 secs 2023-03-09T23:21:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:35 smithi181 ceph-mon[94189]: pgmap v16621: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 297 KiB/s wr, 10 op/s 2023-03-09T23:21:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30206.414292 secs 2023-03-09T23:21:35.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:35 smithi038 ceph-mon[92422]: pgmap v16621: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 297 KiB/s wr, 10 op/s 2023-03-09T23:21:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30206.414292 secs 2023-03-09T23:21:36.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:21:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:21:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:21:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:21:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:21:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:21:37.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:37 smithi122 ceph-mon[93869]: pgmap v16622: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1008 KiB/s rd, 1004 KiB/s wr, 11 op/s 2023-03-09T23:21:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:21:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:21:37.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:37 smithi181 ceph-mon[94189]: pgmap v16622: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1008 KiB/s rd, 1004 KiB/s wr, 11 op/s 2023-03-09T23:21:37.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:21:37.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:21:37.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:37 smithi038 ceph-mon[92422]: pgmap v16622: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1008 KiB/s rd, 1004 KiB/s wr, 11 op/s 2023-03-09T23:21:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:21:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:21:38.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30208.816078 secs 2023-03-09T23:21:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30208.816078 secs 2023-03-09T23:21:38.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30208.816078 secs 2023-03-09T23:21:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30670.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:21:39.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:39 smithi122 ceph-mon[93869]: pgmap v16623: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1008 KiB/s rd, 1005 KiB/s wr, 12 op/s 2023-03-09T23:21:39.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:39 smithi181 ceph-mon[94189]: pgmap v16623: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1008 KiB/s rd, 1005 KiB/s wr, 12 op/s 2023-03-09T23:21:39.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:39 smithi038 ceph-mon[92422]: pgmap v16623: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1008 KiB/s rd, 1005 KiB/s wr, 12 op/s 2023-03-09T23:21:40.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30211.414377 secs 2023-03-09T23:21:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30211.414377 secs 2023-03-09T23:21:40.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30211.414377 secs 2023-03-09T23:21:41.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:41 smithi122 ceph-mon[93869]: pgmap v16624: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 979 KiB/s rd, 974 KiB/s wr, 12 op/s 2023-03-09T23:21:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:41 smithi181 ceph-mon[94189]: pgmap v16624: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 979 KiB/s rd, 974 KiB/s wr, 12 op/s 2023-03-09T23:21:41.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:41 smithi038 ceph-mon[92422]: pgmap v16624: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 979 KiB/s rd, 974 KiB/s wr, 12 op/s 2023-03-09T23:21:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:43 smithi038 ceph-mon[92422]: pgmap v16625: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 855 KiB/s rd, 851 KiB/s wr, 12 op/s 2023-03-09T23:21:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30213.816204 secs 2023-03-09T23:21:43.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:43 smithi122 ceph-mon[93869]: pgmap v16625: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 855 KiB/s rd, 851 KiB/s wr, 12 op/s 2023-03-09T23:21:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30213.816204 secs 2023-03-09T23:21:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:43 smithi181 ceph-mon[94189]: pgmap v16625: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 855 KiB/s rd, 851 KiB/s wr, 12 op/s 2023-03-09T23:21:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30213.816204 secs 2023-03-09T23:21:43.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30675.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:21:45.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:45 smithi122 ceph-mon[93869]: pgmap v16626: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 11 op/s 2023-03-09T23:21:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30216.414436 secs 2023-03-09T23:21:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:45 smithi181 ceph-mon[94189]: pgmap v16626: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 11 op/s 2023-03-09T23:21:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30216.414436 secs 2023-03-09T23:21:45.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:45 smithi038 ceph-mon[92422]: pgmap v16626: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 11 op/s 2023-03-09T23:21:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30216.414436 secs 2023-03-09T23:21:47.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:47 smithi122 ceph-mon[93869]: pgmap v16627: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 15 op/s 2023-03-09T23:21:47.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:47 smithi181 ceph-mon[94189]: pgmap v16627: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 15 op/s 2023-03-09T23:21:47.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:47 smithi038 ceph-mon[92422]: pgmap v16627: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 15 op/s 2023-03-09T23:21:48.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:21:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:21:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:21:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:21:48.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:21:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:21:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30680.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:21:49.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:49 smithi122 ceph-mon[93869]: pgmap v16628: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 646 KiB/s rd, 646 KiB/s wr, 12 op/s 2023-03-09T23:21:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:21:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:21:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:49 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:21:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:49 smithi181 ceph-mon[94189]: pgmap v16628: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 646 KiB/s rd, 646 KiB/s wr, 12 op/s 2023-03-09T23:21:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:21:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:21:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:49 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:21:49.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:49 smithi038 ceph-mon[92422]: pgmap v16628: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 646 KiB/s rd, 646 KiB/s wr, 12 op/s 2023-03-09T23:21:49.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:21:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:21:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:49 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:21:50.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:50 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:21:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:50 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:21:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:50 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:21:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30221.414505 secs 2023-03-09T23:21:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:50 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:21:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:50 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:21:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:50 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:21:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30221.414505 secs 2023-03-09T23:21:50.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:50 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:21:50.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:50 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:21:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:50 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:21:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30221.414505 secs 2023-03-09T23:21:51.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:51 smithi122 ceph-mon[93869]: pgmap v16629: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 946 KiB/s rd, 932 KiB/s wr, 13 op/s 2023-03-09T23:21:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:51 smithi181 ceph-mon[94189]: pgmap v16629: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 946 KiB/s rd, 932 KiB/s wr, 13 op/s 2023-03-09T23:21:51.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:51 smithi038 ceph-mon[92422]: pgmap v16629: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 946 KiB/s rd, 932 KiB/s wr, 13 op/s 2023-03-09T23:21:52.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:21:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:21:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:21:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:21:52.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:21:52.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:21:53.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:53 smithi122 ceph-mon[93869]: pgmap v16630: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1009 KiB/s wr, 16 op/s 2023-03-09T23:21:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:21:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:21:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:53 smithi181 ceph-mon[94189]: pgmap v16630: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1009 KiB/s wr, 16 op/s 2023-03-09T23:21:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:21:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:21:53.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:53 smithi038 ceph-mon[92422]: pgmap v16630: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1009 KiB/s wr, 16 op/s 2023-03-09T23:21:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:21:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:21:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30685.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:21:55.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:55 smithi122 ceph-mon[93869]: pgmap v16631: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1013 KiB/s rd, 999 KiB/s wr, 13 op/s 2023-03-09T23:21:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30226.414562 secs 2023-03-09T23:21:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:55 smithi181 ceph-mon[94189]: pgmap v16631: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1013 KiB/s rd, 999 KiB/s wr, 13 op/s 2023-03-09T23:21:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30226.414562 secs 2023-03-09T23:21:55.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:55 smithi038 ceph-mon[92422]: pgmap v16631: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1013 KiB/s rd, 999 KiB/s wr, 13 op/s 2023-03-09T23:21:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30226.414562 secs 2023-03-09T23:21:56.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:21:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:21:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:21:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:21:56.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:21:56.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:21:57.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:57 smithi122 ceph-mon[93869]: pgmap v16632: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 800 KiB/s rd, 785 KiB/s wr, 15 op/s 2023-03-09T23:21:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:21:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:21:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:57 smithi181 ceph-mon[94189]: pgmap v16632: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 800 KiB/s rd, 785 KiB/s wr, 15 op/s 2023-03-09T23:21:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:21:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:21:57.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:57 smithi038 ceph-mon[92422]: pgmap v16632: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 800 KiB/s rd, 785 KiB/s wr, 15 op/s 2023-03-09T23:21:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:21:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:21:58.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30690.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:21:59.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:21:59 smithi122 ceph-mon[93869]: pgmap v16633: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 973 KiB/s rd, 958 KiB/s wr, 13 op/s 2023-03-09T23:21:59.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:21:59 smithi181 ceph-mon[94189]: pgmap v16633: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 973 KiB/s rd, 958 KiB/s wr, 13 op/s 2023-03-09T23:21:59.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:21:59 smithi038 ceph-mon[92422]: pgmap v16633: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 973 KiB/s rd, 958 KiB/s wr, 13 op/s 2023-03-09T23:22:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:22:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:22:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30231.414621 secs 2023-03-09T23:22:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:22:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:22:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30231.414621 secs 2023-03-09T23:22:00.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:22:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:22:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30231.414621 secs 2023-03-09T23:22:01.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:01 smithi122 ceph-mon[93869]: pgmap v16634: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 978 KiB/s rd, 997 KiB/s wr, 13 op/s 2023-03-09T23:22:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:01 smithi181 ceph-mon[94189]: pgmap v16634: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 978 KiB/s rd, 997 KiB/s wr, 13 op/s 2023-03-09T23:22:01.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:01 smithi038 ceph-mon[92422]: pgmap v16634: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 978 KiB/s rd, 997 KiB/s wr, 13 op/s 2023-03-09T23:22:03.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:03 smithi122 ceph-mon[93869]: pgmap v16635: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 874 KiB/s wr, 16 op/s 2023-03-09T23:22:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30233.816553 secs 2023-03-09T23:22:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:03 smithi181 ceph-mon[94189]: pgmap v16635: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 874 KiB/s wr, 16 op/s 2023-03-09T23:22:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30233.816553 secs 2023-03-09T23:22:03.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:03 smithi038 ceph-mon[92422]: pgmap v16635: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 874 KiB/s wr, 16 op/s 2023-03-09T23:22:03.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30233.816553 secs 2023-03-09T23:22:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30695.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:22:05.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:05 smithi122 ceph-mon[93869]: osdmap e1255: 12 total, 12 up, 12 in 2023-03-09T23:22:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:05 smithi122 ceph-mon[93869]: pgmap v16637: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 899 KiB/s rd, 981 KiB/s wr, 30 op/s 2023-03-09T23:22:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:05 smithi181 ceph-mon[94189]: osdmap e1255: 12 total, 12 up, 12 in 2023-03-09T23:22:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:05 smithi181 ceph-mon[94189]: pgmap v16637: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 899 KiB/s rd, 981 KiB/s wr, 30 op/s 2023-03-09T23:22:05.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:05 smithi038 ceph-mon[92422]: osdmap e1255: 12 total, 12 up, 12 in 2023-03-09T23:22:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:05 smithi038 ceph-mon[92422]: pgmap v16637: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 899 KiB/s rd, 981 KiB/s wr, 30 op/s 2023-03-09T23:22:06.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30236.414682 secs 2023-03-09T23:22:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30236.414682 secs 2023-03-09T23:22:06.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30236.414682 secs 2023-03-09T23:22:07.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:07 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:22:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:07 smithi122 ceph-mon[93869]: pgmap v16638: 129 pgs: 9 active+clean+snaptrim_wait, 120 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 862 KiB/s rd, 903 KiB/s wr, 30 op/s 2023-03-09T23:22:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:22:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:22:07.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:07 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:22:07.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:07 smithi181 ceph-mon[94189]: pgmap v16638: 129 pgs: 9 active+clean+snaptrim_wait, 120 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 862 KiB/s rd, 903 KiB/s wr, 30 op/s 2023-03-09T23:22:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:22:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:22:07.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:07 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:22:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:07 smithi038 ceph-mon[92422]: pgmap v16638: 129 pgs: 9 active+clean+snaptrim_wait, 120 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 862 KiB/s rd, 903 KiB/s wr, 30 op/s 2023-03-09T23:22:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:22:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:22:08.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:22:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:22:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30238.816688 secs 2023-03-09T23:22:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:22:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:22:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30238.816688 secs 2023-03-09T23:22:08.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:22:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:22:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30238.816688 secs 2023-03-09T23:22:08.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30700.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:22:09.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:09 smithi122 ceph-mon[93869]: pgmap v16639: 129 pgs: 9 active+clean+snaptrim_wait, 120 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 733 KiB/s rd, 774 KiB/s wr, 29 op/s 2023-03-09T23:22:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:22:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:22:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:09 smithi181 ceph-mon[94189]: pgmap v16639: 129 pgs: 9 active+clean+snaptrim_wait, 120 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 733 KiB/s rd, 774 KiB/s wr, 29 op/s 2023-03-09T23:22:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:22:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:22:09.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:09 smithi038 ceph-mon[92422]: pgmap v16639: 129 pgs: 9 active+clean+snaptrim_wait, 120 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 733 KiB/s rd, 774 KiB/s wr, 29 op/s 2023-03-09T23:22:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:22:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:22:10.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30241.414741 secs 2023-03-09T23:22:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30241.414741 secs 2023-03-09T23:22:10.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30241.414741 secs 2023-03-09T23:22:11.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:11 smithi122 ceph-mon[93869]: pgmap v16640: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 409 KiB/s wr, 28 op/s 2023-03-09T23:22:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:11 smithi181 ceph-mon[94189]: pgmap v16640: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 409 KiB/s wr, 28 op/s 2023-03-09T23:22:11.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:11 smithi038 ceph-mon[92422]: pgmap v16640: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 409 KiB/s wr, 28 op/s 2023-03-09T23:22:12.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:22:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:22:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:12 smithi122 ceph-mon[93869]: osdmap e1256: 12 total, 12 up, 12 in 2023-03-09T23:22:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:22:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:22:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:12 smithi181 ceph-mon[94189]: osdmap e1256: 12 total, 12 up, 12 in 2023-03-09T23:22:12.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:22:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:22:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:12 smithi038 ceph-mon[92422]: osdmap e1256: 12 total, 12 up, 12 in 2023-03-09T23:22:13.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:13 smithi122 ceph-mon[93869]: pgmap v16642: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 372 KiB/s wr, 12 op/s 2023-03-09T23:22:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:22:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:22:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30243.816808 secs 2023-03-09T23:22:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:13 smithi181 ceph-mon[94189]: pgmap v16642: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 372 KiB/s wr, 12 op/s 2023-03-09T23:22:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:22:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:22:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30243.816808 secs 2023-03-09T23:22:13.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:13 smithi038 ceph-mon[92422]: pgmap v16642: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 372 KiB/s wr, 12 op/s 2023-03-09T23:22:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:22:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:22:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30243.816808 secs 2023-03-09T23:22:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30705.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:22:15.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:15 smithi122 ceph-mon[93869]: pgmap v16643: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 298 KiB/s wr, 10 op/s 2023-03-09T23:22:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30246.414801 secs 2023-03-09T23:22:15.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:15 smithi181 ceph-mon[94189]: pgmap v16643: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 298 KiB/s wr, 10 op/s 2023-03-09T23:22:15.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30246.414801 secs 2023-03-09T23:22:15.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:15 smithi038 ceph-mon[92422]: pgmap v16643: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 298 KiB/s wr, 10 op/s 2023-03-09T23:22:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30246.414801 secs 2023-03-09T23:22:17.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:17 smithi122 ceph-mon[93869]: pgmap v16644: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 253 KiB/s wr, 9 op/s 2023-03-09T23:22:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:17 smithi181 ceph-mon[94189]: pgmap v16644: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 253 KiB/s wr, 9 op/s 2023-03-09T23:22:17.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:17 smithi038 ceph-mon[92422]: pgmap v16644: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 253 KiB/s wr, 9 op/s 2023-03-09T23:22:18.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30248.816928 secs 2023-03-09T23:22:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30248.816928 secs 2023-03-09T23:22:18.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30248.816928 secs 2023-03-09T23:22:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30710.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:22:19.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:19 smithi122 ceph-mon[93869]: pgmap v16645: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 293 KiB/s rd, 290 KiB/s wr, 9 op/s 2023-03-09T23:22:19.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:19 smithi181 ceph-mon[94189]: pgmap v16645: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 293 KiB/s rd, 290 KiB/s wr, 9 op/s 2023-03-09T23:22:19.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:19 smithi038 ceph-mon[92422]: pgmap v16645: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 293 KiB/s rd, 290 KiB/s wr, 9 op/s 2023-03-09T23:22:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30251.414858 secs 2023-03-09T23:22:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30251.414858 secs 2023-03-09T23:22:20.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30251.414858 secs 2023-03-09T23:22:21.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:21 smithi122 ceph-mon[93869]: pgmap v16646: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 314 KiB/s wr, 9 op/s 2023-03-09T23:22:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:21 smithi181 ceph-mon[94189]: pgmap v16646: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 314 KiB/s wr, 9 op/s 2023-03-09T23:22:21.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:21 smithi038 ceph-mon[92422]: pgmap v16646: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 314 KiB/s wr, 9 op/s 2023-03-09T23:22:23.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:23 smithi122 ceph-mon[93869]: pgmap v16647: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 290 KiB/s wr, 9 op/s 2023-03-09T23:22:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30253.817065 secs 2023-03-09T23:22:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:23 smithi181 ceph-mon[94189]: pgmap v16647: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 290 KiB/s wr, 9 op/s 2023-03-09T23:22:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30253.817065 secs 2023-03-09T23:22:23.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:23 smithi038 ceph-mon[92422]: pgmap v16647: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 290 KiB/s wr, 9 op/s 2023-03-09T23:22:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30253.817065 secs 2023-03-09T23:22:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30715.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:22:25.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:25 smithi122 ceph-mon[93869]: pgmap v16648: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 243 KiB/s wr, 8 op/s 2023-03-09T23:22:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30256.414918 secs 2023-03-09T23:22:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:25 smithi181 ceph-mon[94189]: pgmap v16648: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 243 KiB/s wr, 8 op/s 2023-03-09T23:22:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30256.414918 secs 2023-03-09T23:22:25.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:25 smithi038 ceph-mon[92422]: pgmap v16648: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 243 KiB/s wr, 8 op/s 2023-03-09T23:22:25.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30256.414918 secs 2023-03-09T23:22:27.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:27 smithi122 ceph-mon[93869]: pgmap v16649: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 249 KiB/s wr, 11 op/s 2023-03-09T23:22:27.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:27 smithi181 ceph-mon[94189]: pgmap v16649: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 249 KiB/s wr, 11 op/s 2023-03-09T23:22:27.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:27 smithi038 ceph-mon[92422]: pgmap v16649: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 249 KiB/s wr, 11 op/s 2023-03-09T23:22:28.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30258.817214 secs 2023-03-09T23:22:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:22:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30258.817214 secs 2023-03-09T23:22:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:22:28.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30258.817214 secs 2023-03-09T23:22:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:22:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30720.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:22:29.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:29 smithi122 ceph-mon[93869]: pgmap v16650: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 318 KiB/s rd, 224 KiB/s wr, 9 op/s 2023-03-09T23:22:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:22:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:22:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:22:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:22:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:22:29.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:29 smithi181 ceph-mon[94189]: pgmap v16650: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 318 KiB/s rd, 224 KiB/s wr, 9 op/s 2023-03-09T23:22:29.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:22:29.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:22:29.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:22:29.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:22:29.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:22:29.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:29 smithi038 ceph-mon[92422]: pgmap v16650: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 318 KiB/s rd, 224 KiB/s wr, 9 op/s 2023-03-09T23:22:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:22:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:22:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:22:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:22:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:22:30.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30261.414981 secs 2023-03-09T23:22:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30261.414981 secs 2023-03-09T23:22:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30261.414981 secs 2023-03-09T23:22:31.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:31 smithi122 ceph-mon[93869]: pgmap v16651: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 98 KiB/s wr, 8 op/s 2023-03-09T23:22:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:31 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:22:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:31 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:22:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:31 smithi181 ceph-mon[94189]: pgmap v16651: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 98 KiB/s wr, 8 op/s 2023-03-09T23:22:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:31 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:22:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:31 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:22:31.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:31 smithi038 ceph-mon[92422]: pgmap v16651: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 98 KiB/s wr, 8 op/s 2023-03-09T23:22:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:31 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:22:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:31 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:22:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30725.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:22:33.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:33 smithi122 ceph-mon[93869]: pgmap v16652: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 212 KiB/s wr, 10 op/s 2023-03-09T23:22:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30263.817341 secs 2023-03-09T23:22:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:33 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:22:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:33 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:22:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:33 smithi181 ceph-mon[94189]: pgmap v16652: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 212 KiB/s wr, 10 op/s 2023-03-09T23:22:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30263.817341 secs 2023-03-09T23:22:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:33 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:22:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:33 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:22:33.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:33 smithi038 ceph-mon[92422]: pgmap v16652: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 212 KiB/s wr, 10 op/s 2023-03-09T23:22:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30263.817341 secs 2023-03-09T23:22:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:33 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:22:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:33 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:22:35.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:35 smithi122 ceph-mon[93869]: pgmap v16653: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 140 KiB/s wr, 8 op/s 2023-03-09T23:22:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30266.415043 secs 2023-03-09T23:22:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:35 smithi181 ceph-mon[94189]: pgmap v16653: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 140 KiB/s wr, 8 op/s 2023-03-09T23:22:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30266.415043 secs 2023-03-09T23:22:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:35 smithi038 ceph-mon[92422]: pgmap v16653: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 140 KiB/s wr, 8 op/s 2023-03-09T23:22:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30266.415043 secs 2023-03-09T23:22:37.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:37 smithi122 ceph-mon[93869]: pgmap v16654: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 328 KiB/s rd, 250 KiB/s wr, 10 op/s 2023-03-09T23:22:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:37 smithi181 ceph-mon[94189]: pgmap v16654: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 328 KiB/s rd, 250 KiB/s wr, 10 op/s 2023-03-09T23:22:37.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:37 smithi038 ceph-mon[92422]: pgmap v16654: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 328 KiB/s rd, 250 KiB/s wr, 10 op/s 2023-03-09T23:22:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30730.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:22:38.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30268.817488 secs 2023-03-09T23:22:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30268.817488 secs 2023-03-09T23:22:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30268.817488 secs 2023-03-09T23:22:39.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:39 smithi122 ceph-mon[93869]: pgmap v16655: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 244 KiB/s wr, 8 op/s 2023-03-09T23:22:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:39 smithi181 ceph-mon[94189]: pgmap v16655: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 244 KiB/s wr, 8 op/s 2023-03-09T23:22:39.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:39 smithi038 ceph-mon[92422]: pgmap v16655: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 244 KiB/s wr, 8 op/s 2023-03-09T23:22:40.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30271.415103 secs 2023-03-09T23:22:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30271.415103 secs 2023-03-09T23:22:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30271.415103 secs 2023-03-09T23:22:41.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:41 smithi122 ceph-mon[93869]: pgmap v16656: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 243 KiB/s wr, 7 op/s 2023-03-09T23:22:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:41 smithi181 ceph-mon[94189]: pgmap v16656: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 243 KiB/s wr, 7 op/s 2023-03-09T23:22:41.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:41 smithi038 ceph-mon[92422]: pgmap v16656: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 243 KiB/s wr, 7 op/s 2023-03-09T23:22:43.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:43 smithi122 ceph-mon[93869]: pgmap v16657: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 376 KiB/s wr, 10 op/s 2023-03-09T23:22:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:43 smithi181 ceph-mon[94189]: pgmap v16657: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 376 KiB/s wr, 10 op/s 2023-03-09T23:22:43.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:43 smithi038 ceph-mon[92422]: pgmap v16657: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 376 KiB/s wr, 10 op/s 2023-03-09T23:22:43.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30735.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:22:45.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:45 smithi122 ceph-mon[93869]: pgmap v16658: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 243 KiB/s wr, 7 op/s 2023-03-09T23:22:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30276.415186 secs 2023-03-09T23:22:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:45 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:22:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:45 smithi181 ceph-mon[94189]: pgmap v16658: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 243 KiB/s wr, 7 op/s 2023-03-09T23:22:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30276.415186 secs 2023-03-09T23:22:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:45 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:22:45.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:45 smithi038 ceph-mon[92422]: pgmap v16658: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 243 KiB/s wr, 7 op/s 2023-03-09T23:22:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30276.415186 secs 2023-03-09T23:22:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:45 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:22:46.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:46 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:22:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:46 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:22:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:46 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:22:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:46 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:22:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:46 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:22:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:46 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:22:46.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:46 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:22:46.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:46 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:22:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:46 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:22:47.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:47 smithi122 ceph-mon[93869]: pgmap v16659: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 513 KiB/s rd, 510 KiB/s wr, 10 op/s 2023-03-09T23:22:47.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:47 smithi181 ceph-mon[94189]: pgmap v16659: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 513 KiB/s rd, 510 KiB/s wr, 10 op/s 2023-03-09T23:22:47.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:47 smithi038 ceph-mon[92422]: pgmap v16659: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 513 KiB/s rd, 510 KiB/s wr, 10 op/s 2023-03-09T23:22:48.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30740.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:22:49.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:49 smithi122 ceph-mon[93869]: pgmap v16660: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 402 KiB/s rd, 400 KiB/s wr, 8 op/s 2023-03-09T23:22:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:22:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:22:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:49 smithi181 ceph-mon[94189]: pgmap v16660: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 402 KiB/s rd, 400 KiB/s wr, 8 op/s 2023-03-09T23:22:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:22:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:22:49.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:49 smithi038 ceph-mon[92422]: pgmap v16660: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 402 KiB/s rd, 400 KiB/s wr, 8 op/s 2023-03-09T23:22:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:22:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:22:50.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30281.415245 secs 2023-03-09T23:22:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30281.415245 secs 2023-03-09T23:22:50.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30281.415245 secs 2023-03-09T23:22:51.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:51 smithi122 ceph-mon[93869]: pgmap v16661: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 402 KiB/s rd, 400 KiB/s wr, 7 op/s 2023-03-09T23:22:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:51 smithi181 ceph-mon[94189]: pgmap v16661: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 402 KiB/s rd, 400 KiB/s wr, 7 op/s 2023-03-09T23:22:51.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:51 smithi038 ceph-mon[92422]: pgmap v16661: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 402 KiB/s rd, 400 KiB/s wr, 7 op/s 2023-03-09T23:22:52.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:22:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:22:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:22:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:22:52.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:22:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:22:53.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:53 smithi122 ceph-mon[93869]: pgmap v16662: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1019 KiB/s rd, 424 KiB/s wr, 11 op/s 2023-03-09T23:22:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:53 smithi181 ceph-mon[94189]: pgmap v16662: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1019 KiB/s rd, 424 KiB/s wr, 11 op/s 2023-03-09T23:22:53.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:53 smithi038 ceph-mon[92422]: pgmap v16662: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1019 KiB/s rd, 424 KiB/s wr, 11 op/s 2023-03-09T23:22:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30745.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:22:55.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:55 smithi122 ceph-mon[93869]: pgmap v16663: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 431 KiB/s wr, 8 op/s 2023-03-09T23:22:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30286.415320 secs 2023-03-09T23:22:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:55 smithi181 ceph-mon[94189]: pgmap v16663: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 431 KiB/s wr, 8 op/s 2023-03-09T23:22:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30286.415320 secs 2023-03-09T23:22:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:55 smithi038 ceph-mon[92422]: pgmap v16663: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 431 KiB/s wr, 8 op/s 2023-03-09T23:22:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30286.415320 secs 2023-03-09T23:22:56.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:22:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:22:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:22:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:22:56.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:22:56.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:22:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:57 smithi122 ceph-mon[93869]: pgmap v16664: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 456 KiB/s wr, 12 op/s 2023-03-09T23:22:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:22:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:22:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:57 smithi181 ceph-mon[94189]: pgmap v16664: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 456 KiB/s wr, 12 op/s 2023-03-09T23:22:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:22:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:22:57.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:57 smithi038 ceph-mon[92422]: pgmap v16664: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 456 KiB/s wr, 12 op/s 2023-03-09T23:22:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:22:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:22:58.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:58 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 30288.818020 secs 2023-03-09T23:22:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:58 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 30288.818020 secs 2023-03-09T23:22:58.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:58 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 30288.818020 secs 2023-03-09T23:22:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30750.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:22:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:22:59 smithi122 ceph-mon[93869]: pgmap v16665: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 392 KiB/s wr, 10 op/s 2023-03-09T23:22:59.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:22:59 smithi181 ceph-mon[94189]: pgmap v16665: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 392 KiB/s wr, 10 op/s 2023-03-09T23:22:59.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:22:59 smithi038 ceph-mon[92422]: pgmap v16665: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 392 KiB/s wr, 10 op/s 2023-03-09T23:23:00.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:23:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:23:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30291.415379 secs 2023-03-09T23:23:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:23:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:23:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30291.415379 secs 2023-03-09T23:23:00.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:23:00.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:23:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30291.415379 secs 2023-03-09T23:23:01.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:01 smithi122 ceph-mon[93869]: pgmap v16666: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 569 KiB/s wr, 10 op/s 2023-03-09T23:23:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:23:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:23:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:01 smithi181 ceph-mon[94189]: pgmap v16666: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 569 KiB/s wr, 10 op/s 2023-03-09T23:23:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:23:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:23:01.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:01 smithi038 ceph-mon[92422]: pgmap v16666: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 569 KiB/s wr, 10 op/s 2023-03-09T23:23:01.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:23:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:23:02.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:02 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:23:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:02 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:23:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:02 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:23:02.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:02 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:23:02.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:02 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:23:02.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:02 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:23:02.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:02 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:23:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:02 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:23:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:02 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:23:03.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:03 smithi122 ceph-mon[93869]: pgmap v16667: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 667 KiB/s wr, 15 op/s 2023-03-09T23:23:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:03 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 30293.818144 secs 2023-03-09T23:23:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:03 smithi181 ceph-mon[94189]: pgmap v16667: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 667 KiB/s wr, 15 op/s 2023-03-09T23:23:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:03 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 30293.818144 secs 2023-03-09T23:23:03.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:03 smithi038 ceph-mon[92422]: pgmap v16667: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 667 KiB/s wr, 15 op/s 2023-03-09T23:23:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:03 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 30293.818144 secs 2023-03-09T23:23:03.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30755.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:23:05.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:05 smithi122 ceph-mon[93869]: pgmap v16668: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 806 KiB/s rd, 817 KiB/s wr, 24 op/s 2023-03-09T23:23:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30296.415439 secs 2023-03-09T23:23:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:05 smithi181 ceph-mon[94189]: pgmap v16668: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 806 KiB/s rd, 817 KiB/s wr, 24 op/s 2023-03-09T23:23:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30296.415439 secs 2023-03-09T23:23:05.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:05 smithi038 ceph-mon[92422]: pgmap v16668: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 806 KiB/s rd, 817 KiB/s wr, 24 op/s 2023-03-09T23:23:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30296.415439 secs 2023-03-09T23:23:07.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:07 smithi122 ceph-mon[93869]: pgmap v16669: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 854 KiB/s rd, 807 KiB/s wr, 26 op/s 2023-03-09T23:23:07.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:07 smithi181 ceph-mon[94189]: pgmap v16669: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 854 KiB/s rd, 807 KiB/s wr, 26 op/s 2023-03-09T23:23:07.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:07 smithi038 ceph-mon[92422]: pgmap v16669: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 854 KiB/s rd, 807 KiB/s wr, 26 op/s 2023-03-09T23:23:08.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:08 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 30298.818266 secs 2023-03-09T23:23:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:23:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:23:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:08 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 30298.818266 secs 2023-03-09T23:23:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:23:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:23:08.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:08 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 30298.818266 secs 2023-03-09T23:23:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:23:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:23:08.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30760.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:23:09.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:09 smithi122 ceph-mon[93869]: pgmap v16670: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 827 KiB/s rd, 782 KiB/s wr, 23 op/s 2023-03-09T23:23:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:23:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:23:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:09 smithi181 ceph-mon[94189]: pgmap v16670: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 827 KiB/s rd, 782 KiB/s wr, 23 op/s 2023-03-09T23:23:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:23:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:23:09.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:09 smithi038 ceph-mon[92422]: pgmap v16670: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 827 KiB/s rd, 782 KiB/s wr, 23 op/s 2023-03-09T23:23:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:23:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:23:10.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30301.415503 secs 2023-03-09T23:23:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30301.415503 secs 2023-03-09T23:23:10.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30301.415503 secs 2023-03-09T23:23:11.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:11 smithi122 ceph-mon[93869]: pgmap v16671: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 578 KiB/s rd, 580 KiB/s wr, 22 op/s 2023-03-09T23:23:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:11 smithi181 ceph-mon[94189]: pgmap v16671: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 578 KiB/s rd, 580 KiB/s wr, 22 op/s 2023-03-09T23:23:11.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:11 smithi038 ceph-mon[92422]: pgmap v16671: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 578 KiB/s rd, 580 KiB/s wr, 22 op/s 2023-03-09T23:23:13.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:13 smithi122 ceph-mon[93869]: pgmap v16672: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 520 KiB/s rd, 567 KiB/s wr, 25 op/s 2023-03-09T23:23:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:13 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 30303.818351 secs 2023-03-09T23:23:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:13 smithi181 ceph-mon[94189]: pgmap v16672: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 520 KiB/s rd, 567 KiB/s wr, 25 op/s 2023-03-09T23:23:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:13 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 30303.818351 secs 2023-03-09T23:23:13.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:13 smithi038 ceph-mon[92422]: pgmap v16672: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 520 KiB/s rd, 567 KiB/s wr, 25 op/s 2023-03-09T23:23:13.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:13 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 30303.818351 secs 2023-03-09T23:23:13.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30765.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:23:15.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:15 smithi122 ceph-mon[93869]: pgmap v16673: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 490 KiB/s rd, 469 KiB/s wr, 20 op/s 2023-03-09T23:23:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30306.415561 secs 2023-03-09T23:23:15.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:15 smithi181 ceph-mon[94189]: pgmap v16673: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 490 KiB/s rd, 469 KiB/s wr, 20 op/s 2023-03-09T23:23:15.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30306.415561 secs 2023-03-09T23:23:15.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:15 smithi038 ceph-mon[92422]: pgmap v16673: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 490 KiB/s rd, 469 KiB/s wr, 20 op/s 2023-03-09T23:23:15.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30306.415561 secs 2023-03-09T23:23:17.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:17 smithi122 ceph-mon[93869]: pgmap v16674: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 355 KiB/s rd, 295 KiB/s wr, 11 op/s 2023-03-09T23:23:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:17 smithi181 ceph-mon[94189]: pgmap v16674: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 355 KiB/s rd, 295 KiB/s wr, 11 op/s 2023-03-09T23:23:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:17 smithi038 ceph-mon[92422]: pgmap v16674: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 355 KiB/s rd, 295 KiB/s wr, 11 op/s 2023-03-09T23:23:18.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:18 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 30308.818452 secs 2023-03-09T23:23:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:18 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 30308.818452 secs 2023-03-09T23:23:18.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:18 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 30308.818452 secs 2023-03-09T23:23:18.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30770.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:23:19.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:19 smithi122 ceph-mon[93869]: pgmap v16675: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 176 KiB/s wr, 9 op/s 2023-03-09T23:23:19.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:19 smithi181 ceph-mon[94189]: pgmap v16675: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 176 KiB/s wr, 9 op/s 2023-03-09T23:23:19.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:19 smithi038 ceph-mon[92422]: pgmap v16675: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 176 KiB/s wr, 9 op/s 2023-03-09T23:23:20.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:23:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:23:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30311.415621 secs 2023-03-09T23:23:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:23:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:23:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30311.415621 secs 2023-03-09T23:23:20.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:23:20.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:23:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30311.415621 secs 2023-03-09T23:23:21.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:21 smithi122 ceph-mon[93869]: pgmap v16676: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 176 KiB/s wr, 8 op/s 2023-03-09T23:23:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:23:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:23:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:21 smithi181 ceph-mon[94189]: pgmap v16676: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 176 KiB/s wr, 8 op/s 2023-03-09T23:23:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:23:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:23:21.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:21 smithi038 ceph-mon[92422]: pgmap v16676: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 176 KiB/s wr, 8 op/s 2023-03-09T23:23:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:23:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:23:23.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:23 smithi122 ceph-mon[93869]: pgmap v16677: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 537 KiB/s rd, 555 KiB/s wr, 13 op/s 2023-03-09T23:23:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:23 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 30313.818585 secs 2023-03-09T23:23:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:23 smithi181 ceph-mon[94189]: pgmap v16677: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 537 KiB/s rd, 555 KiB/s wr, 13 op/s 2023-03-09T23:23:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:23 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 30313.818585 secs 2023-03-09T23:23:23.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:23 smithi038 ceph-mon[92422]: pgmap v16677: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 537 KiB/s rd, 555 KiB/s wr, 13 op/s 2023-03-09T23:23:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:23 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 30313.818585 secs 2023-03-09T23:23:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30775.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:23:25.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:25 smithi122 ceph-mon[93869]: osdmap e1257: 12 total, 12 up, 12 in 2023-03-09T23:23:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:25 smithi122 ceph-mon[93869]: pgmap v16679: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 469 KiB/s wr, 12 op/s 2023-03-09T23:23:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30316.415687 secs 2023-03-09T23:23:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:25 smithi181 ceph-mon[94189]: osdmap e1257: 12 total, 12 up, 12 in 2023-03-09T23:23:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:25 smithi181 ceph-mon[94189]: pgmap v16679: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 469 KiB/s wr, 12 op/s 2023-03-09T23:23:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30316.415687 secs 2023-03-09T23:23:25.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:25 smithi038 ceph-mon[92422]: osdmap e1257: 12 total, 12 up, 12 in 2023-03-09T23:23:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:25 smithi038 ceph-mon[92422]: pgmap v16679: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 469 KiB/s wr, 12 op/s 2023-03-09T23:23:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30316.415687 secs 2023-03-09T23:23:27.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:27 smithi122 ceph-mon[93869]: pgmap v16680: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 446 KiB/s rd, 469 KiB/s wr, 12 op/s 2023-03-09T23:23:27.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:27 smithi181 ceph-mon[94189]: pgmap v16680: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 446 KiB/s rd, 469 KiB/s wr, 12 op/s 2023-03-09T23:23:27.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:27 smithi038 ceph-mon[92422]: pgmap v16680: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 446 KiB/s rd, 469 KiB/s wr, 12 op/s 2023-03-09T23:23:28.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:28 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 30318.818701 secs 2023-03-09T23:23:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:28 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 30318.818701 secs 2023-03-09T23:23:28.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:28 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 30318.818701 secs 2023-03-09T23:23:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30780.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:23:29.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:29 smithi122 ceph-mon[93869]: pgmap v16681: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 578 KiB/s rd, 590 KiB/s wr, 11 op/s 2023-03-09T23:23:29.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:29 smithi181 ceph-mon[94189]: pgmap v16681: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 578 KiB/s rd, 590 KiB/s wr, 11 op/s 2023-03-09T23:23:29.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:29 smithi038 ceph-mon[92422]: pgmap v16681: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 578 KiB/s rd, 590 KiB/s wr, 11 op/s 2023-03-09T23:23:30.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:30 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:23:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:30 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:23:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:30 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:23:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:30 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:23:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:30 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:23:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30321.415751 secs 2023-03-09T23:23:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:30 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:23:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:30 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:23:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:30 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:23:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:30 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:23:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:30 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:23:30.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30321.415751 secs 2023-03-09T23:23:30.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:30 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:23:30.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:30 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:23:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:30 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:23:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:30 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:23:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:30 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:23:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30321.415751 secs 2023-03-09T23:23:31.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:31 smithi122 ceph-mon[93869]: pgmap v16682: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 578 KiB/s rd, 590 KiB/s wr, 11 op/s 2023-03-09T23:23:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:23:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:23:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:23:31.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:31 smithi181 ceph-mon[94189]: pgmap v16682: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 578 KiB/s rd, 590 KiB/s wr, 11 op/s 2023-03-09T23:23:31.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:23:31.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:23:31.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:23:31.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:31 smithi038 ceph-mon[92422]: pgmap v16682: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 578 KiB/s rd, 590 KiB/s wr, 11 op/s 2023-03-09T23:23:31.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:23:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:23:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:23:32.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:32 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:23:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:32 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:23:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:32 smithi122 ceph-mon[93869]: osdmap e1258: 12 total, 12 up, 12 in 2023-03-09T23:23:32.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:32 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:23:32.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:32 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:23:32.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:32 smithi181 ceph-mon[94189]: osdmap e1258: 12 total, 12 up, 12 in 2023-03-09T23:23:32.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:32 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:23:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:32 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:23:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:32 smithi038 ceph-mon[92422]: osdmap e1258: 12 total, 12 up, 12 in 2023-03-09T23:23:33.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:33 smithi122 ceph-mon[93869]: pgmap v16684: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 206 KiB/s wr, 11 op/s 2023-03-09T23:23:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:33 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 30323.818772 secs 2023-03-09T23:23:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:33 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:23:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:33 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:23:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:33 smithi181 ceph-mon[94189]: pgmap v16684: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 206 KiB/s wr, 11 op/s 2023-03-09T23:23:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:33 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 30323.818772 secs 2023-03-09T23:23:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:33 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:23:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:33 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:23:33.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:33 smithi038 ceph-mon[92422]: pgmap v16684: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 206 KiB/s wr, 11 op/s 2023-03-09T23:23:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:33 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 30323.818772 secs 2023-03-09T23:23:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:33 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:23:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:33 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:23:33.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30785.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:23:35.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:35 smithi122 ceph-mon[93869]: pgmap v16685: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 165 KiB/s wr, 9 op/s 2023-03-09T23:23:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30326.415812 secs 2023-03-09T23:23:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:35 smithi181 ceph-mon[94189]: pgmap v16685: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 165 KiB/s wr, 9 op/s 2023-03-09T23:23:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30326.415812 secs 2023-03-09T23:23:35.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:35 smithi038 ceph-mon[92422]: pgmap v16685: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 165 KiB/s wr, 9 op/s 2023-03-09T23:23:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30326.415812 secs 2023-03-09T23:23:36.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:23:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:23:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:23:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:23:36.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:23:36.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:23:37.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:37 smithi122 ceph-mon[93869]: pgmap v16686: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 517 KiB/s rd, 516 KiB/s wr, 9 op/s 2023-03-09T23:23:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:23:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:23:37.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:37 smithi181 ceph-mon[94189]: pgmap v16686: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 517 KiB/s rd, 516 KiB/s wr, 9 op/s 2023-03-09T23:23:37.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:23:37.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:23:37.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:37 smithi038 ceph-mon[92422]: pgmap v16686: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 517 KiB/s rd, 516 KiB/s wr, 9 op/s 2023-03-09T23:23:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:23:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:23:38.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30328.818889 secs 2023-03-09T23:23:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30328.818889 secs 2023-03-09T23:23:38.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30328.818889 secs 2023-03-09T23:23:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30790.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:23:39.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:39 smithi122 ceph-mon[93869]: pgmap v16687: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 643 KiB/s rd, 640 KiB/s wr, 9 op/s 2023-03-09T23:23:39.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:39 smithi181 ceph-mon[94189]: pgmap v16687: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 643 KiB/s rd, 640 KiB/s wr, 9 op/s 2023-03-09T23:23:39.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:39 smithi038 ceph-mon[92422]: pgmap v16687: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 643 KiB/s rd, 640 KiB/s wr, 9 op/s 2023-03-09T23:23:40.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30331.415874 secs 2023-03-09T23:23:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30331.415874 secs 2023-03-09T23:23:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30331.415874 secs 2023-03-09T23:23:41.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:41 smithi122 ceph-mon[93869]: pgmap v16688: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 643 KiB/s rd, 640 KiB/s wr, 9 op/s 2023-03-09T23:23:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:41 smithi181 ceph-mon[94189]: pgmap v16688: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 643 KiB/s rd, 640 KiB/s wr, 9 op/s 2023-03-09T23:23:41.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:41 smithi038 ceph-mon[92422]: pgmap v16688: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 643 KiB/s rd, 640 KiB/s wr, 9 op/s 2023-03-09T23:23:42.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:42 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:23:42.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:42 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:23:42.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:42 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:23:42.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:42 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:23:42.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:42 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:23:42.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:42 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:23:43.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:43 smithi122 ceph-mon[93869]: pgmap v16689: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 745 KiB/s rd, 743 KiB/s wr, 10 op/s 2023-03-09T23:23:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30333.818992 secs 2023-03-09T23:23:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:43 smithi181 ceph-mon[94189]: pgmap v16689: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 745 KiB/s rd, 743 KiB/s wr, 10 op/s 2023-03-09T23:23:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30333.818992 secs 2023-03-09T23:23:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:43 smithi038 ceph-mon[92422]: pgmap v16689: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 745 KiB/s rd, 743 KiB/s wr, 10 op/s 2023-03-09T23:23:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30333.818992 secs 2023-03-09T23:23:43.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30795.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:23:45.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:45 smithi122 ceph-mon[93869]: pgmap v16690: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 622 KiB/s rd, 620 KiB/s wr, 8 op/s 2023-03-09T23:23:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30336.415932 secs 2023-03-09T23:23:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:45 smithi181 ceph-mon[94189]: pgmap v16690: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 622 KiB/s rd, 620 KiB/s wr, 8 op/s 2023-03-09T23:23:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30336.415932 secs 2023-03-09T23:23:45.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:45 smithi038 ceph-mon[92422]: pgmap v16690: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 622 KiB/s rd, 620 KiB/s wr, 8 op/s 2023-03-09T23:23:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30336.415932 secs 2023-03-09T23:23:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:47 smithi122 ceph-mon[93869]: pgmap v16691: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 823 KiB/s rd, 818 KiB/s wr, 12 op/s 2023-03-09T23:23:47.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:47 smithi181 ceph-mon[94189]: pgmap v16691: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 823 KiB/s rd, 818 KiB/s wr, 12 op/s 2023-03-09T23:23:47.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:47 smithi038 ceph-mon[92422]: pgmap v16691: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 823 KiB/s rd, 818 KiB/s wr, 12 op/s 2023-03-09T23:23:48.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30338.819126 secs 2023-03-09T23:23:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:23:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:23:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30338.819126 secs 2023-03-09T23:23:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:23:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:23:48.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30338.819126 secs 2023-03-09T23:23:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:23:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:23:48.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30800.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:23:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:49 smithi181 ceph-mon[94189]: pgmap v16692: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 832 KiB/s rd, 829 KiB/s wr, 10 op/s 2023-03-09T23:23:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:23:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:23:49.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:49 smithi038 ceph-mon[92422]: pgmap v16692: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 832 KiB/s rd, 829 KiB/s wr, 10 op/s 2023-03-09T23:23:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:23:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:23:49.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:49 smithi122 ceph-mon[93869]: pgmap v16692: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 832 KiB/s rd, 829 KiB/s wr, 10 op/s 2023-03-09T23:23:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:23:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:23:50.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30341.415983 secs 2023-03-09T23:23:50.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30341.415983 secs 2023-03-09T23:23:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30341.415983 secs 2023-03-09T23:23:51.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:51 smithi122 ceph-mon[93869]: pgmap v16693: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 613 KiB/s wr, 10 op/s 2023-03-09T23:23:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:51 smithi181 ceph-mon[94189]: pgmap v16693: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 613 KiB/s wr, 10 op/s 2023-03-09T23:23:51.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:51 smithi038 ceph-mon[92422]: pgmap v16693: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 613 KiB/s wr, 10 op/s 2023-03-09T23:23:52.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:23:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:23:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:23:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:23:52.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:23:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:23:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30805.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:23:53.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:53 smithi122 ceph-mon[93869]: pgmap v16694: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 802 KiB/s rd, 786 KiB/s wr, 14 op/s 2023-03-09T23:23:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30343.819197 secs 2023-03-09T23:23:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:23:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:23:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:53 smithi181 ceph-mon[94189]: pgmap v16694: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 802 KiB/s rd, 786 KiB/s wr, 14 op/s 2023-03-09T23:23:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30343.819197 secs 2023-03-09T23:23:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:23:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:23:53.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:53 smithi038 ceph-mon[92422]: pgmap v16694: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 802 KiB/s rd, 786 KiB/s wr, 14 op/s 2023-03-09T23:23:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30343.819197 secs 2023-03-09T23:23:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:23:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:23:55.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:55 smithi122 ceph-mon[93869]: pgmap v16695: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 689 KiB/s rd, 675 KiB/s wr, 10 op/s 2023-03-09T23:23:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:23:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:23:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30346.416041 secs 2023-03-09T23:23:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:55 smithi181 ceph-mon[94189]: pgmap v16695: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 689 KiB/s rd, 675 KiB/s wr, 10 op/s 2023-03-09T23:23:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:23:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:23:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30346.416041 secs 2023-03-09T23:23:56.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:55 smithi038 ceph-mon[92422]: pgmap v16695: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 689 KiB/s rd, 675 KiB/s wr, 10 op/s 2023-03-09T23:23:56.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:23:56.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:23:56.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30346.416041 secs 2023-03-09T23:23:57.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:57 smithi122 ceph-mon[93869]: pgmap v16696: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 809 KiB/s rd, 722 KiB/s wr, 14 op/s 2023-03-09T23:23:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:23:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:23:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:57 smithi181 ceph-mon[94189]: pgmap v16696: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 809 KiB/s rd, 722 KiB/s wr, 14 op/s 2023-03-09T23:23:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:23:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:23:57.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:57 smithi038 ceph-mon[92422]: pgmap v16696: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 809 KiB/s rd, 722 KiB/s wr, 14 op/s 2023-03-09T23:23:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:23:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:23:58.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30810.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:23:58.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30348.819283 secs 2023-03-09T23:23:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30348.819283 secs 2023-03-09T23:23:58.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30348.819283 secs 2023-03-09T23:23:59.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:23:59 smithi122 ceph-mon[93869]: pgmap v16697: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 866 KiB/s wr, 12 op/s 2023-03-09T23:23:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:23:59 smithi181 ceph-mon[94189]: pgmap v16697: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 866 KiB/s wr, 12 op/s 2023-03-09T23:23:59.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:23:59 smithi038 ceph-mon[92422]: pgmap v16697: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 866 KiB/s wr, 12 op/s 2023-03-09T23:24:00.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30351.416117 secs 2023-03-09T23:24:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30351.416117 secs 2023-03-09T23:24:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30351.416117 secs 2023-03-09T23:24:01.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:01 smithi122 ceph-mon[93869]: pgmap v16698: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 741 KiB/s rd, 562 KiB/s wr, 11 op/s 2023-03-09T23:24:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:24:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:24:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:01 smithi181 ceph-mon[94189]: pgmap v16698: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 741 KiB/s rd, 562 KiB/s wr, 11 op/s 2023-03-09T23:24:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:24:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:24:01.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:01 smithi038 ceph-mon[92422]: pgmap v16698: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 741 KiB/s rd, 562 KiB/s wr, 11 op/s 2023-03-09T23:24:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:24:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:24:03.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:03 smithi122 ceph-mon[93869]: pgmap v16699: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 931 KiB/s wr, 16 op/s 2023-03-09T23:24:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30353.819412 secs 2023-03-09T23:24:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:03 smithi181 ceph-mon[94189]: pgmap v16699: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 931 KiB/s wr, 16 op/s 2023-03-09T23:24:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30353.819412 secs 2023-03-09T23:24:03.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:03 smithi038 ceph-mon[92422]: pgmap v16699: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 931 KiB/s wr, 16 op/s 2023-03-09T23:24:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30353.819412 secs 2023-03-09T23:24:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30815.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:24:05.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:05 smithi122 ceph-mon[93869]: pgmap v16700: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 882 KiB/s wr, 25 op/s 2023-03-09T23:24:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30356.416190 secs 2023-03-09T23:24:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:05 smithi181 ceph-mon[94189]: pgmap v16700: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 882 KiB/s wr, 25 op/s 2023-03-09T23:24:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30356.416190 secs 2023-03-09T23:24:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:05 smithi038 ceph-mon[92422]: pgmap v16700: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 882 KiB/s wr, 25 op/s 2023-03-09T23:24:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30356.416190 secs 2023-03-09T23:24:07.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:07 smithi122 ceph-mon[93869]: pgmap v16701: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 904 KiB/s wr, 30 op/s 2023-03-09T23:24:07.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:07 smithi181 ceph-mon[94189]: pgmap v16701: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 904 KiB/s wr, 30 op/s 2023-03-09T23:24:07.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:07 smithi038 ceph-mon[92422]: pgmap v16701: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 904 KiB/s wr, 30 op/s 2023-03-09T23:24:08.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:24:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:24:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30358.819497 secs 2023-03-09T23:24:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:24:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:24:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30358.819497 secs 2023-03-09T23:24:08.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:24:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:24:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30358.819497 secs 2023-03-09T23:24:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30820.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:24:09.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:09 smithi122 ceph-mon[93869]: pgmap v16702: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 857 KiB/s wr, 26 op/s 2023-03-09T23:24:09.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:24:09.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:24:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:09 smithi181 ceph-mon[94189]: pgmap v16702: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 857 KiB/s wr, 26 op/s 2023-03-09T23:24:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:24:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:24:09.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:09 smithi038 ceph-mon[92422]: pgmap v16702: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 857 KiB/s wr, 26 op/s 2023-03-09T23:24:09.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:24:09.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:24:10.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:10 smithi122 ceph-mon[93869]: osdmap e1259: 12 total, 12 up, 12 in 2023-03-09T23:24:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30361.416262 secs 2023-03-09T23:24:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:10 smithi181 ceph-mon[94189]: osdmap e1259: 12 total, 12 up, 12 in 2023-03-09T23:24:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30361.416262 secs 2023-03-09T23:24:10.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:10 smithi038 ceph-mon[92422]: osdmap e1259: 12 total, 12 up, 12 in 2023-03-09T23:24:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30361.416262 secs 2023-03-09T23:24:11.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:11 smithi122 ceph-mon[93869]: pgmap v16704: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 829 KiB/s wr, 30 op/s 2023-03-09T23:24:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:11 smithi181 ceph-mon[94189]: pgmap v16704: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 829 KiB/s wr, 30 op/s 2023-03-09T23:24:11.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:11 smithi038 ceph-mon[92422]: pgmap v16704: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 829 KiB/s wr, 30 op/s 2023-03-09T23:24:13.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:13 smithi122 ceph-mon[93869]: pgmap v16705: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 488 KiB/s wr, 28 op/s 2023-03-09T23:24:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30363.819625 secs 2023-03-09T23:24:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:13 smithi181 ceph-mon[94189]: pgmap v16705: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 488 KiB/s wr, 28 op/s 2023-03-09T23:24:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30363.819625 secs 2023-03-09T23:24:13.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:13 smithi038 ceph-mon[92422]: pgmap v16705: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 488 KiB/s wr, 28 op/s 2023-03-09T23:24:13.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30363.819625 secs 2023-03-09T23:24:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30825.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:24:15.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:15 smithi122 ceph-mon[93869]: pgmap v16706: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 418 KiB/s wr, 13 op/s 2023-03-09T23:24:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30366.416347 secs 2023-03-09T23:24:15.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:15 smithi181 ceph-mon[94189]: pgmap v16706: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 418 KiB/s wr, 13 op/s 2023-03-09T23:24:15.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30366.416347 secs 2023-03-09T23:24:15.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:15 smithi038 ceph-mon[92422]: pgmap v16706: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 418 KiB/s wr, 13 op/s 2023-03-09T23:24:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30366.416347 secs 2023-03-09T23:24:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:17 smithi122 ceph-mon[93869]: pgmap v16707: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 421 KiB/s wr, 10 op/s 2023-03-09T23:24:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:24:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:24:17.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:17 smithi122 ceph-mon[93869]: osdmap e1260: 12 total, 12 up, 12 in 2023-03-09T23:24:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:17 smithi181 ceph-mon[94189]: pgmap v16707: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 421 KiB/s wr, 10 op/s 2023-03-09T23:24:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:24:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:24:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:17 smithi181 ceph-mon[94189]: osdmap e1260: 12 total, 12 up, 12 in 2023-03-09T23:24:17.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:17 smithi038 ceph-mon[92422]: pgmap v16707: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 421 KiB/s wr, 10 op/s 2023-03-09T23:24:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:24:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:24:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:17 smithi038 ceph-mon[92422]: osdmap e1260: 12 total, 12 up, 12 in 2023-03-09T23:24:18.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30368.819700 secs 2023-03-09T23:24:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30368.819700 secs 2023-03-09T23:24:18.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30368.819700 secs 2023-03-09T23:24:18.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30830.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:24:19.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:19 smithi122 ceph-mon[93869]: pgmap v16709: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 235 KiB/s wr, 11 op/s 2023-03-09T23:24:19.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:19 smithi181 ceph-mon[94189]: pgmap v16709: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 235 KiB/s wr, 11 op/s 2023-03-09T23:24:19.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:19 smithi038 ceph-mon[92422]: pgmap v16709: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 235 KiB/s wr, 11 op/s 2023-03-09T23:24:20.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:24:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:24:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30371.416408 secs 2023-03-09T23:24:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:24:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:24:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30371.416408 secs 2023-03-09T23:24:20.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:24:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:24:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30371.416408 secs 2023-03-09T23:24:21.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:21 smithi122 ceph-mon[93869]: pgmap v16710: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 210 KiB/s wr, 10 op/s 2023-03-09T23:24:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:21 smithi181 ceph-mon[94189]: pgmap v16710: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 210 KiB/s wr, 10 op/s 2023-03-09T23:24:21.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:21 smithi038 ceph-mon[92422]: pgmap v16710: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 212 KiB/s rd, 210 KiB/s wr, 10 op/s 2023-03-09T23:24:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:23 smithi122 ceph-mon[93869]: pgmap v16711: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 272 KiB/s wr, 11 op/s 2023-03-09T23:24:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30373.819829 secs 2023-03-09T23:24:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:23 smithi181 ceph-mon[94189]: pgmap v16711: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 272 KiB/s wr, 11 op/s 2023-03-09T23:24:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30373.819829 secs 2023-03-09T23:24:23.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:23 smithi038 ceph-mon[92422]: pgmap v16711: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 272 KiB/s wr, 11 op/s 2023-03-09T23:24:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30373.819829 secs 2023-03-09T23:24:23.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30835.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:24:25.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:25 smithi122 ceph-mon[93869]: pgmap v16712: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 194 KiB/s wr, 10 op/s 2023-03-09T23:24:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30376.416468 secs 2023-03-09T23:24:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:25 smithi181 ceph-mon[94189]: pgmap v16712: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 194 KiB/s wr, 10 op/s 2023-03-09T23:24:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30376.416468 secs 2023-03-09T23:24:25.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:25 smithi038 ceph-mon[92422]: pgmap v16712: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 194 KiB/s wr, 10 op/s 2023-03-09T23:24:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30376.416468 secs 2023-03-09T23:24:27.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:27 smithi122 ceph-mon[93869]: pgmap v16713: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 164 KiB/s wr, 9 op/s 2023-03-09T23:24:27.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:27 smithi181 ceph-mon[94189]: pgmap v16713: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 164 KiB/s wr, 9 op/s 2023-03-09T23:24:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:27 smithi038 ceph-mon[92422]: pgmap v16713: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 164 KiB/s wr, 9 op/s 2023-03-09T23:24:28.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30378.819949 secs 2023-03-09T23:24:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30378.819949 secs 2023-03-09T23:24:28.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30378.819949 secs 2023-03-09T23:24:28.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30840.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:24:29.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:29 smithi122 ceph-mon[93869]: pgmap v16714: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 294 KiB/s wr, 8 op/s 2023-03-09T23:24:29.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:29 smithi181 ceph-mon[94189]: pgmap v16714: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 294 KiB/s wr, 8 op/s 2023-03-09T23:24:29.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:29 smithi038 ceph-mon[92422]: pgmap v16714: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 294 KiB/s wr, 8 op/s 2023-03-09T23:24:30.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30381.416530 secs 2023-03-09T23:24:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30381.416530 secs 2023-03-09T23:24:30.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30381.416530 secs 2023-03-09T23:24:31.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:31 smithi122 ceph-mon[93869]: pgmap v16715: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 271 KiB/s wr, 7 op/s 2023-03-09T23:24:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:24:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:31 smithi181 ceph-mon[94189]: pgmap v16715: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 271 KiB/s wr, 7 op/s 2023-03-09T23:24:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:24:31.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:31 smithi038 ceph-mon[92422]: pgmap v16715: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 271 KiB/s wr, 7 op/s 2023-03-09T23:24:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:24:32.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:24:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:24:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:24:32.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:24:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:24:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:24:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:24:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:24:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:24:33.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:33 smithi122 ceph-mon[93869]: pgmap v16716: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 271 KiB/s wr, 10 op/s 2023-03-09T23:24:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30383.820013 secs 2023-03-09T23:24:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:33 smithi181 ceph-mon[94189]: pgmap v16716: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 271 KiB/s wr, 10 op/s 2023-03-09T23:24:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30383.820013 secs 2023-03-09T23:24:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:33 smithi038 ceph-mon[92422]: pgmap v16716: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 271 KiB/s wr, 10 op/s 2023-03-09T23:24:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30383.820013 secs 2023-03-09T23:24:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30845.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:24:35.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:35 smithi122 ceph-mon[93869]: pgmap v16717: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 135 KiB/s wr, 6 op/s 2023-03-09T23:24:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30386.416591 secs 2023-03-09T23:24:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:35 smithi181 ceph-mon[94189]: pgmap v16717: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 135 KiB/s wr, 6 op/s 2023-03-09T23:24:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30386.416591 secs 2023-03-09T23:24:35.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:35 smithi038 ceph-mon[92422]: pgmap v16717: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 135 KiB/s wr, 6 op/s 2023-03-09T23:24:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30386.416591 secs 2023-03-09T23:24:37.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:37 smithi122 ceph-mon[93869]: pgmap v16718: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 135 KiB/s wr, 8 op/s 2023-03-09T23:24:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:37 smithi181 ceph-mon[94189]: pgmap v16718: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 135 KiB/s wr, 8 op/s 2023-03-09T23:24:37.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:37 smithi038 ceph-mon[92422]: pgmap v16718: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 135 KiB/s wr, 8 op/s 2023-03-09T23:24:38.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30388.820133 secs 2023-03-09T23:24:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30388.820133 secs 2023-03-09T23:24:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30388.820133 secs 2023-03-09T23:24:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30850.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:24:39.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:39 smithi122 ceph-mon[93869]: pgmap v16719: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 168 KiB/s wr, 7 op/s 2023-03-09T23:24:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:39 smithi181 ceph-mon[94189]: pgmap v16719: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 168 KiB/s wr, 7 op/s 2023-03-09T23:24:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:39 smithi038 ceph-mon[92422]: pgmap v16719: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 168 KiB/s wr, 7 op/s 2023-03-09T23:24:40.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:24:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:24:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30391.416652 secs 2023-03-09T23:24:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:24:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:24:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30391.416652 secs 2023-03-09T23:24:40.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:24:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:24:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30391.416652 secs 2023-03-09T23:24:41.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:41 smithi122 ceph-mon[93869]: pgmap v16720: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 261 KiB/s wr, 6 op/s 2023-03-09T23:24:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:24:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:24:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:41 smithi181 ceph-mon[94189]: pgmap v16720: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 261 KiB/s wr, 6 op/s 2023-03-09T23:24:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:24:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:24:41.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:41 smithi038 ceph-mon[92422]: pgmap v16720: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 261 KiB/s wr, 6 op/s 2023-03-09T23:24:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:24:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:24:43.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:43 smithi122 ceph-mon[93869]: pgmap v16721: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 261 KiB/s wr, 9 op/s 2023-03-09T23:24:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30393.820260 secs 2023-03-09T23:24:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:43 smithi181 ceph-mon[94189]: pgmap v16721: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 261 KiB/s wr, 9 op/s 2023-03-09T23:24:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30393.820260 secs 2023-03-09T23:24:43.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:43 smithi038 ceph-mon[92422]: pgmap v16721: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 261 KiB/s wr, 9 op/s 2023-03-09T23:24:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30393.820260 secs 2023-03-09T23:24:43.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30855.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:24:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:45 smithi122 ceph-mon[93869]: pgmap v16722: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 723 KiB/s rd, 376 KiB/s wr, 7 op/s 2023-03-09T23:24:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30396.416710 secs 2023-03-09T23:24:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:45 smithi181 ceph-mon[94189]: pgmap v16722: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 723 KiB/s rd, 376 KiB/s wr, 7 op/s 2023-03-09T23:24:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30396.416710 secs 2023-03-09T23:24:45.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:45 smithi038 ceph-mon[92422]: pgmap v16722: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 723 KiB/s rd, 376 KiB/s wr, 7 op/s 2023-03-09T23:24:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30396.416710 secs 2023-03-09T23:24:47.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:47 smithi122 ceph-mon[93869]: pgmap v16723: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 724 KiB/s rd, 376 KiB/s wr, 10 op/s 2023-03-09T23:24:47.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:47 smithi181 ceph-mon[94189]: pgmap v16723: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 724 KiB/s rd, 376 KiB/s wr, 10 op/s 2023-03-09T23:24:47.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:47 smithi038 ceph-mon[92422]: pgmap v16723: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 724 KiB/s rd, 376 KiB/s wr, 10 op/s 2023-03-09T23:24:48.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30398.820330 secs 2023-03-09T23:24:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30398.820330 secs 2023-03-09T23:24:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30398.820330 secs 2023-03-09T23:24:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30860.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:24:49.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:49 smithi122 ceph-mon[93869]: pgmap v16724: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 757 KiB/s rd, 410 KiB/s wr, 9 op/s 2023-03-09T23:24:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:49 smithi181 ceph-mon[94189]: pgmap v16724: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 757 KiB/s rd, 410 KiB/s wr, 9 op/s 2023-03-09T23:24:49.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:49 smithi038 ceph-mon[92422]: pgmap v16724: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 757 KiB/s rd, 410 KiB/s wr, 9 op/s 2023-03-09T23:24:50.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30401.416796 secs 2023-03-09T23:24:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30401.416796 secs 2023-03-09T23:24:50.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30401.416796 secs 2023-03-09T23:24:51.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:51 smithi122 ceph-mon[93869]: pgmap v16725: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 838 KiB/s rd, 376 KiB/s wr, 9 op/s 2023-03-09T23:24:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:51 smithi181 ceph-mon[94189]: pgmap v16725: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 838 KiB/s rd, 376 KiB/s wr, 9 op/s 2023-03-09T23:24:51.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:51 smithi038 ceph-mon[92422]: pgmap v16725: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 838 KiB/s rd, 376 KiB/s wr, 9 op/s 2023-03-09T23:24:53.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:53 smithi122 ceph-mon[93869]: pgmap v16726: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 951 KiB/s rd, 510 KiB/s wr, 13 op/s 2023-03-09T23:24:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:24:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:24:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:53 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:24:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:53 smithi181 ceph-mon[94189]: pgmap v16726: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 951 KiB/s rd, 510 KiB/s wr, 13 op/s 2023-03-09T23:24:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:24:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:24:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:53 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:24:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:53 smithi038 ceph-mon[92422]: pgmap v16726: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 951 KiB/s rd, 510 KiB/s wr, 13 op/s 2023-03-09T23:24:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:24:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:24:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:53 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:24:53.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30865.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:24:54.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:54 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:24:54.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:54 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:24:54.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:54 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:24:54.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:54 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:24:54.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:54 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:24:54.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:54 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:24:54.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:54 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:24:54.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:54 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:24:54.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:54 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:24:55.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:55 smithi122 ceph-mon[93869]: pgmap v16727: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 761 KiB/s wr, 12 op/s 2023-03-09T23:24:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30406.416853 secs 2023-03-09T23:24:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:55 smithi181 ceph-mon[94189]: pgmap v16727: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 761 KiB/s wr, 12 op/s 2023-03-09T23:24:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30406.416853 secs 2023-03-09T23:24:55.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:55 smithi038 ceph-mon[92422]: pgmap v16727: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 761 KiB/s wr, 12 op/s 2023-03-09T23:24:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30406.416853 secs 2023-03-09T23:24:56.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:24:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:24:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:24:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:24:56.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:24:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:24:57.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:57 smithi122 ceph-mon[93869]: pgmap v16728: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 777 KiB/s rd, 647 KiB/s wr, 14 op/s 2023-03-09T23:24:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:24:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:24:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:57 smithi181 ceph-mon[94189]: pgmap v16728: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 777 KiB/s rd, 647 KiB/s wr, 14 op/s 2023-03-09T23:24:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:24:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:24:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:57 smithi038 ceph-mon[92422]: pgmap v16728: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 777 KiB/s rd, 647 KiB/s wr, 14 op/s 2023-03-09T23:24:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:24:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:24:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30870.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:24:59.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:24:59 smithi122 ceph-mon[93869]: pgmap v16729: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 775 KiB/s rd, 647 KiB/s wr, 12 op/s 2023-03-09T23:24:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:24:59 smithi181 ceph-mon[94189]: pgmap v16729: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 775 KiB/s rd, 647 KiB/s wr, 12 op/s 2023-03-09T23:24:59.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:24:59 smithi038 ceph-mon[92422]: pgmap v16729: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 775 KiB/s rd, 647 KiB/s wr, 12 op/s 2023-03-09T23:25:00.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:25:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:25:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30411.416921 secs 2023-03-09T23:25:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:25:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:25:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30411.416921 secs 2023-03-09T23:25:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:25:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:25:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30411.416921 secs 2023-03-09T23:25:01.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:01 smithi122 ceph-mon[93869]: pgmap v16730: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 704 KiB/s wr, 12 op/s 2023-03-09T23:25:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:25:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:25:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:01 smithi181 ceph-mon[94189]: pgmap v16730: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 704 KiB/s wr, 12 op/s 2023-03-09T23:25:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:25:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:25:01.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:01 smithi038 ceph-mon[92422]: pgmap v16730: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 704 KiB/s wr, 12 op/s 2023-03-09T23:25:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:25:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:25:03.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:03 smithi122 ceph-mon[93869]: pgmap v16731: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 16 op/s 2023-03-09T23:25:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30413.820698 secs 2023-03-09T23:25:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:03 smithi181 ceph-mon[94189]: pgmap v16731: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 16 op/s 2023-03-09T23:25:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30413.820698 secs 2023-03-09T23:25:03.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:03 smithi038 ceph-mon[92422]: pgmap v16731: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 16 op/s 2023-03-09T23:25:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30413.820698 secs 2023-03-09T23:25:03.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30875.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:25:05.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:05 smithi122 ceph-mon[93869]: pgmap v16732: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 24 op/s 2023-03-09T23:25:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30416.416981 secs 2023-03-09T23:25:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:05 smithi181 ceph-mon[94189]: pgmap v16732: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 24 op/s 2023-03-09T23:25:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30416.416981 secs 2023-03-09T23:25:05.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:05 smithi038 ceph-mon[92422]: pgmap v16732: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 24 op/s 2023-03-09T23:25:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30416.416981 secs 2023-03-09T23:25:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:06 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:25:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:06 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:25:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:25:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:06 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:25:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:06 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:25:06.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:25:06.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:06 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:25:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:06 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:25:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:25:07.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:07 smithi122 ceph-mon[93869]: pgmap v16733: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 870 KiB/s wr, 28 op/s 2023-03-09T23:25:07.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:07 smithi181 ceph-mon[94189]: pgmap v16733: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 870 KiB/s wr, 28 op/s 2023-03-09T23:25:07.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:07 smithi038 ceph-mon[92422]: pgmap v16733: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 870 KiB/s wr, 28 op/s 2023-03-09T23:25:08.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30418.820834 secs 2023-03-09T23:25:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:25:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:25:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30418.820834 secs 2023-03-09T23:25:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:25:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:25:08.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30418.820834 secs 2023-03-09T23:25:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:25:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:25:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30880.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:25:09.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:09 smithi122 ceph-mon[93869]: pgmap v16734: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 869 KiB/s wr, 25 op/s 2023-03-09T23:25:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:09 smithi181 ceph-mon[94189]: pgmap v16734: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 869 KiB/s wr, 25 op/s 2023-03-09T23:25:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:09 smithi038 ceph-mon[92422]: pgmap v16734: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 869 KiB/s wr, 25 op/s 2023-03-09T23:25:10.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:10 smithi122 ceph-mon[93869]: osdmap e1261: 12 total, 12 up, 12 in 2023-03-09T23:25:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30421.417042 secs 2023-03-09T23:25:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:10 smithi181 ceph-mon[94189]: osdmap e1261: 12 total, 12 up, 12 in 2023-03-09T23:25:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30421.417042 secs 2023-03-09T23:25:10.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:10 smithi038 ceph-mon[92422]: osdmap e1261: 12 total, 12 up, 12 in 2023-03-09T23:25:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30421.417042 secs 2023-03-09T23:25:11.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:11 smithi122 ceph-mon[93869]: pgmap v16736: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 893 KiB/s rd, 933 KiB/s wr, 29 op/s 2023-03-09T23:25:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:11 smithi181 ceph-mon[94189]: pgmap v16736: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 893 KiB/s rd, 933 KiB/s wr, 29 op/s 2023-03-09T23:25:11.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:11 smithi038 ceph-mon[92422]: pgmap v16736: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 893 KiB/s rd, 933 KiB/s wr, 29 op/s 2023-03-09T23:25:13.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:13 smithi122 ceph-mon[93869]: pgmap v16737: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 400 KiB/s wr, 28 op/s 2023-03-09T23:25:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30423.820906 secs 2023-03-09T23:25:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:25:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:25:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:13 smithi181 ceph-mon[94189]: pgmap v16737: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 400 KiB/s wr, 28 op/s 2023-03-09T23:25:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30423.820906 secs 2023-03-09T23:25:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:25:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:25:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:13 smithi038 ceph-mon[92422]: pgmap v16737: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 400 KiB/s wr, 28 op/s 2023-03-09T23:25:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30423.820906 secs 2023-03-09T23:25:13.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:25:13.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:25:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30885.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:25:15.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:15 smithi122 ceph-mon[93869]: pgmap v16738: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 374 KiB/s wr, 13 op/s 2023-03-09T23:25:15.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30426.417127 secs 2023-03-09T23:25:15.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:15 smithi181 ceph-mon[94189]: pgmap v16738: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 374 KiB/s wr, 13 op/s 2023-03-09T23:25:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30426.417127 secs 2023-03-09T23:25:15.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:15 smithi038 ceph-mon[92422]: pgmap v16738: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 374 KiB/s wr, 13 op/s 2023-03-09T23:25:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30426.417127 secs 2023-03-09T23:25:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:25:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:25:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:25:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:25:16.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:25:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:25:17.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:17 smithi122 ceph-mon[93869]: pgmap v16739: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 436 KiB/s wr, 10 op/s 2023-03-09T23:25:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:17 smithi122 ceph-mon[93869]: osdmap e1262: 12 total, 12 up, 12 in 2023-03-09T23:25:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:17 smithi181 ceph-mon[94189]: pgmap v16739: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 436 KiB/s wr, 10 op/s 2023-03-09T23:25:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:17 smithi181 ceph-mon[94189]: osdmap e1262: 12 total, 12 up, 12 in 2023-03-09T23:25:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:17 smithi038 ceph-mon[92422]: pgmap v16739: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 436 KiB/s wr, 10 op/s 2023-03-09T23:25:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:17 smithi038 ceph-mon[92422]: osdmap e1262: 12 total, 12 up, 12 in 2023-03-09T23:25:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30428.821026 secs 2023-03-09T23:25:18.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30428.821026 secs 2023-03-09T23:25:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30890.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:25:18.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30428.821026 secs 2023-03-09T23:25:19.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:19 smithi181 ceph-mon[94189]: pgmap v16741: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 515 KiB/s rd, 512 KiB/s wr, 10 op/s 2023-03-09T23:25:19.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:19 smithi038 ceph-mon[92422]: pgmap v16741: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 515 KiB/s rd, 512 KiB/s wr, 10 op/s 2023-03-09T23:25:19.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:19 smithi122 ceph-mon[93869]: pgmap v16741: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 515 KiB/s rd, 512 KiB/s wr, 10 op/s 2023-03-09T23:25:20.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30431.417187 secs 2023-03-09T23:25:20.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30431.417187 secs 2023-03-09T23:25:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30431.417187 secs 2023-03-09T23:25:21.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:21 smithi038 ceph-mon[92422]: pgmap v16742: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 456 KiB/s wr, 9 op/s 2023-03-09T23:25:21.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:21 smithi122 ceph-mon[93869]: pgmap v16742: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 456 KiB/s wr, 9 op/s 2023-03-09T23:25:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:21 smithi181 ceph-mon[94189]: pgmap v16742: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 456 KiB/s wr, 9 op/s 2023-03-09T23:25:23.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:23 smithi122 ceph-mon[93869]: pgmap v16743: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 111 KiB/s wr, 8 op/s 2023-03-09T23:25:23.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30433.821171 secs 2023-03-09T23:25:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:23 smithi181 ceph-mon[94189]: pgmap v16743: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 111 KiB/s wr, 8 op/s 2023-03-09T23:25:23.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30433.821171 secs 2023-03-09T23:25:23.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:23 smithi038 ceph-mon[92422]: pgmap v16743: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 111 KiB/s wr, 8 op/s 2023-03-09T23:25:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30433.821171 secs 2023-03-09T23:25:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30895.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:25:25.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:25 smithi122 ceph-mon[93869]: pgmap v16744: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 111 KiB/s wr, 8 op/s 2023-03-09T23:25:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30436.417246 secs 2023-03-09T23:25:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:25 smithi181 ceph-mon[94189]: pgmap v16744: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 111 KiB/s wr, 8 op/s 2023-03-09T23:25:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30436.417246 secs 2023-03-09T23:25:25.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:25 smithi038 ceph-mon[92422]: pgmap v16744: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 111 KiB/s wr, 8 op/s 2023-03-09T23:25:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30436.417246 secs 2023-03-09T23:25:27.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:27 smithi122 ceph-mon[93869]: pgmap v16745: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T23:25:27.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:27 smithi181 ceph-mon[94189]: pgmap v16745: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T23:25:27.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:27 smithi038 ceph-mon[92422]: pgmap v16745: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 20 KiB/s wr, 7 op/s 2023-03-09T23:25:28.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30438.821310 secs 2023-03-09T23:25:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30438.821310 secs 2023-03-09T23:25:28.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30438.821310 secs 2023-03-09T23:25:28.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30900.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:25:29.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:29 smithi122 ceph-mon[93869]: pgmap v16746: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 7 op/s 2023-03-09T23:25:29.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:29 smithi181 ceph-mon[94189]: pgmap v16746: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 7 op/s 2023-03-09T23:25:29.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:29 smithi038 ceph-mon[92422]: pgmap v16746: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 7 op/s 2023-03-09T23:25:30.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30441.417309 secs 2023-03-09T23:25:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30441.417309 secs 2023-03-09T23:25:30.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30441.417309 secs 2023-03-09T23:25:31.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:31 smithi122 ceph-mon[93869]: pgmap v16747: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T23:25:31.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:31 smithi181 ceph-mon[94189]: pgmap v16747: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T23:25:31.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:31 smithi038 ceph-mon[92422]: pgmap v16747: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T23:25:32.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:25:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:25:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:25:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:25:32.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:25:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:25:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:25:32.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:25:32.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:25:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:25:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:25:32.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:25:33.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:33 smithi122 ceph-mon[93869]: pgmap v16748: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 9 op/s 2023-03-09T23:25:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30443.821415 secs 2023-03-09T23:25:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:33 smithi181 ceph-mon[94189]: pgmap v16748: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 9 op/s 2023-03-09T23:25:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30443.821415 secs 2023-03-09T23:25:33.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:33 smithi038 ceph-mon[92422]: pgmap v16748: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 9 op/s 2023-03-09T23:25:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30443.821415 secs 2023-03-09T23:25:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30905.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:25:35.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:35 smithi122 ceph-mon[93869]: pgmap v16749: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T23:25:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30446.417591 secs 2023-03-09T23:25:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:35 smithi181 ceph-mon[94189]: pgmap v16749: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T23:25:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30446.417591 secs 2023-03-09T23:25:35.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:35 smithi038 ceph-mon[92422]: pgmap v16749: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T23:25:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30446.417591 secs 2023-03-09T23:25:37.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:37 smithi122 ceph-mon[93869]: pgmap v16750: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 9 op/s 2023-03-09T23:25:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:25:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:25:37.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:37 smithi181 ceph-mon[94189]: pgmap v16750: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 9 op/s 2023-03-09T23:25:37.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:25:37.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:25:37.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:37 smithi038 ceph-mon[92422]: pgmap v16750: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 9 op/s 2023-03-09T23:25:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:25:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:25:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30448.821553 secs 2023-03-09T23:25:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30448.821553 secs 2023-03-09T23:25:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30448.821553 secs 2023-03-09T23:25:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30910.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:25:39.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:39 smithi122 ceph-mon[93869]: pgmap v16751: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 112 KiB/s wr, 8 op/s 2023-03-09T23:25:39.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:39 smithi181 ceph-mon[94189]: pgmap v16751: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 112 KiB/s wr, 8 op/s 2023-03-09T23:25:39.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:39 smithi038 ceph-mon[92422]: pgmap v16751: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 112 KiB/s wr, 8 op/s 2023-03-09T23:25:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:25:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:25:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30451.417674 secs 2023-03-09T23:25:40.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:25:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:25:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30451.417674 secs 2023-03-09T23:25:40.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:25:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:25:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30451.417674 secs 2023-03-09T23:25:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:41 smithi038 ceph-mon[92422]: pgmap v16752: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 112 KiB/s wr, 7 op/s 2023-03-09T23:25:41.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:41 smithi122 ceph-mon[93869]: pgmap v16752: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 112 KiB/s wr, 7 op/s 2023-03-09T23:25:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:41 smithi181 ceph-mon[94189]: pgmap v16752: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 112 KiB/s wr, 7 op/s 2023-03-09T23:25:43.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:43 smithi122 ceph-mon[93869]: pgmap v16753: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 373 KiB/s wr, 11 op/s 2023-03-09T23:25:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:43 smithi181 ceph-mon[94189]: pgmap v16753: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 373 KiB/s wr, 11 op/s 2023-03-09T23:25:43.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:43 smithi038 ceph-mon[92422]: pgmap v16753: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 373 KiB/s wr, 11 op/s 2023-03-09T23:25:43.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30915.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:25:45.733 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:45 smithi122 ceph-mon[93869]: pgmap v16754: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 373 KiB/s wr, 7 op/s 2023-03-09T23:25:45.733 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30456.417733 secs 2023-03-09T23:25:45.733 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:45 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:25:45.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:45 smithi181 ceph-mon[94189]: pgmap v16754: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 373 KiB/s wr, 7 op/s 2023-03-09T23:25:45.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30456.417733 secs 2023-03-09T23:25:45.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:45 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:25:45.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:45 smithi038 ceph-mon[92422]: pgmap v16754: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 373 KiB/s wr, 7 op/s 2023-03-09T23:25:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30456.417733 secs 2023-03-09T23:25:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:45 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:25:46.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:46 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:25:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:46 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:25:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:46 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:25:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:46 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:25:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:46 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:25:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:46 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:25:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:46 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:25:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:46 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:25:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:46 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:25:47.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:47 smithi122 ceph-mon[93869]: pgmap v16755: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 373 KiB/s wr, 10 op/s 2023-03-09T23:25:47.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:47 smithi181 ceph-mon[94189]: pgmap v16755: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 373 KiB/s wr, 10 op/s 2023-03-09T23:25:47.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:47 smithi038 ceph-mon[92422]: pgmap v16755: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 373 KiB/s wr, 10 op/s 2023-03-09T23:25:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30920.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:25:49.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:49 smithi122 ceph-mon[93869]: pgmap v16756: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 795 KiB/s rd, 667 KiB/s wr, 8 op/s 2023-03-09T23:25:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:25:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:25:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:49 smithi181 ceph-mon[94189]: pgmap v16756: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 795 KiB/s rd, 667 KiB/s wr, 8 op/s 2023-03-09T23:25:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:25:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:25:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:49 smithi038 ceph-mon[92422]: pgmap v16756: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 795 KiB/s rd, 667 KiB/s wr, 8 op/s 2023-03-09T23:25:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:25:49.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:25:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30461.417793 secs 2023-03-09T23:25:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30461.417793 secs 2023-03-09T23:25:50.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30461.417793 secs 2023-03-09T23:25:51.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:51 smithi122 ceph-mon[93869]: pgmap v16757: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 684 KiB/s rd, 556 KiB/s wr, 7 op/s 2023-03-09T23:25:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:51 smithi181 ceph-mon[94189]: pgmap v16757: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 684 KiB/s rd, 556 KiB/s wr, 7 op/s 2023-03-09T23:25:51.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:51 smithi038 ceph-mon[92422]: pgmap v16757: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 684 KiB/s rd, 556 KiB/s wr, 7 op/s 2023-03-09T23:25:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:25:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:25:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:25:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:25:52.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:25:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:25:53.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:53 smithi122 ceph-mon[93869]: pgmap v16758: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 580 KiB/s wr, 10 op/s 2023-03-09T23:25:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:53 smithi181 ceph-mon[94189]: pgmap v16758: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 580 KiB/s wr, 10 op/s 2023-03-09T23:25:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:53 smithi038 ceph-mon[92422]: pgmap v16758: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 580 KiB/s wr, 10 op/s 2023-03-09T23:25:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30925.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:25:55.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:55 smithi122 ceph-mon[93869]: pgmap v16759: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 959 KiB/s rd, 454 KiB/s wr, 8 op/s 2023-03-09T23:25:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30466.417854 secs 2023-03-09T23:25:55.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:55 smithi181 ceph-mon[94189]: pgmap v16759: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 959 KiB/s rd, 454 KiB/s wr, 8 op/s 2023-03-09T23:25:55.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30466.417854 secs 2023-03-09T23:25:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:55 smithi038 ceph-mon[92422]: pgmap v16759: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 959 KiB/s rd, 454 KiB/s wr, 8 op/s 2023-03-09T23:25:55.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30466.417854 secs 2023-03-09T23:25:56.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:25:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:25:56.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:25:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:25:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:25:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:25:57.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:57 smithi038 ceph-mon[92422]: pgmap v16760: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 991 KiB/s rd, 482 KiB/s wr, 11 op/s 2023-03-09T23:25:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:25:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:25:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:57 smithi122 ceph-mon[93869]: pgmap v16760: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 991 KiB/s rd, 482 KiB/s wr, 11 op/s 2023-03-09T23:25:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:25:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:25:57.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:57 smithi181 ceph-mon[94189]: pgmap v16760: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 991 KiB/s rd, 482 KiB/s wr, 11 op/s 2023-03-09T23:25:57.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:25:57.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:25:58.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30930.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:25:58.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:58 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 30468.821982 secs 2023-03-09T23:25:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:58 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 30468.821982 secs 2023-03-09T23:25:58.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:58 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 30468.821982 secs 2023-03-09T23:25:59.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:25:59 smithi122 ceph-mon[93869]: pgmap v16761: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 716 KiB/s wr, 11 op/s 2023-03-09T23:25:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:25:59 smithi181 ceph-mon[94189]: pgmap v16761: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 716 KiB/s wr, 11 op/s 2023-03-09T23:25:59.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:25:59 smithi038 ceph-mon[92422]: pgmap v16761: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 716 KiB/s wr, 11 op/s 2023-03-09T23:26:00.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:26:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:26:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30471.417915 secs 2023-03-09T23:26:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:26:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:26:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30471.417915 secs 2023-03-09T23:26:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:26:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:26:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30471.417915 secs 2023-03-09T23:26:01.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:01 smithi122 ceph-mon[93869]: pgmap v16762: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1020 KiB/s rd, 629 KiB/s wr, 10 op/s 2023-03-09T23:26:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:26:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:26:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:01 smithi181 ceph-mon[94189]: pgmap v16762: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1020 KiB/s rd, 629 KiB/s wr, 10 op/s 2023-03-09T23:26:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:26:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:26:01.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:01 smithi038 ceph-mon[92422]: pgmap v16762: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1020 KiB/s rd, 629 KiB/s wr, 10 op/s 2023-03-09T23:26:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:26:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:26:02.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:02 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:26:02.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:02 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:26:02.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:02 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:26:02.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:02 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:26:02.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:02 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:26:02.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:02 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:26:02.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:02 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:26:02.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:02 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:26:02.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:02 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:26:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30935.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:26:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:03 smithi122 ceph-mon[93869]: pgmap v16763: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 845 KiB/s wr, 17 op/s 2023-03-09T23:26:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30473.822082 secs 2023-03-09T23:26:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:03 smithi181 ceph-mon[94189]: pgmap v16763: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 845 KiB/s wr, 17 op/s 2023-03-09T23:26:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30473.822082 secs 2023-03-09T23:26:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:03 smithi038 ceph-mon[92422]: pgmap v16763: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 845 KiB/s wr, 17 op/s 2023-03-09T23:26:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30473.822082 secs 2023-03-09T23:26:05.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:05 smithi122 ceph-mon[93869]: pgmap v16764: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 928 KiB/s rd, 951 KiB/s wr, 27 op/s 2023-03-09T23:26:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30476.418000 secs 2023-03-09T23:26:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:05 smithi181 ceph-mon[94189]: pgmap v16764: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 928 KiB/s rd, 951 KiB/s wr, 27 op/s 2023-03-09T23:26:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30476.418000 secs 2023-03-09T23:26:05.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:05 smithi038 ceph-mon[92422]: pgmap v16764: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 928 KiB/s rd, 951 KiB/s wr, 27 op/s 2023-03-09T23:26:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30476.418000 secs 2023-03-09T23:26:06.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:06 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:26:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:26:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:06 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:26:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:26:06.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:06 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:26:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:26:07.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:07 smithi122 ceph-mon[93869]: pgmap v16765: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 818 KiB/s rd, 840 KiB/s wr, 30 op/s 2023-03-09T23:26:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:26:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:26:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:07 smithi181 ceph-mon[94189]: pgmap v16765: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 818 KiB/s rd, 840 KiB/s wr, 30 op/s 2023-03-09T23:26:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:26:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:26:07.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:07 smithi038 ceph-mon[92422]: pgmap v16765: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 818 KiB/s rd, 840 KiB/s wr, 30 op/s 2023-03-09T23:26:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:26:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:26:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30940.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:26:08.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30478.822184 secs 2023-03-09T23:26:08.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30478.822184 secs 2023-03-09T23:26:08.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30478.822184 secs 2023-03-09T23:26:09.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:09 smithi122 ceph-mon[93869]: pgmap v16766: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 787 KiB/s rd, 812 KiB/s wr, 27 op/s 2023-03-09T23:26:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:26:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:26:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:09 smithi181 ceph-mon[94189]: pgmap v16766: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 787 KiB/s rd, 812 KiB/s wr, 27 op/s 2023-03-09T23:26:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:26:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:26:09.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:09 smithi038 ceph-mon[92422]: pgmap v16766: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 787 KiB/s rd, 812 KiB/s wr, 27 op/s 2023-03-09T23:26:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:26:10.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:26:10.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:10 smithi122 ceph-mon[93869]: osdmap e1263: 12 total, 12 up, 12 in 2023-03-09T23:26:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30481.418059 secs 2023-03-09T23:26:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:10 smithi181 ceph-mon[94189]: osdmap e1263: 12 total, 12 up, 12 in 2023-03-09T23:26:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30481.418059 secs 2023-03-09T23:26:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:10 smithi038 ceph-mon[92422]: osdmap e1263: 12 total, 12 up, 12 in 2023-03-09T23:26:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30481.418059 secs 2023-03-09T23:26:11.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:11 smithi122 ceph-mon[93869]: pgmap v16768: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 444 KiB/s wr, 30 op/s 2023-03-09T23:26:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:11 smithi181 ceph-mon[94189]: pgmap v16768: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 444 KiB/s wr, 30 op/s 2023-03-09T23:26:11.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:11 smithi038 ceph-mon[92422]: pgmap v16768: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 444 KiB/s wr, 30 op/s 2023-03-09T23:26:13.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:13 smithi122 ceph-mon[93869]: pgmap v16769: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 539 KiB/s rd, 554 KiB/s wr, 26 op/s 2023-03-09T23:26:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30483.822324 secs 2023-03-09T23:26:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:13 smithi181 ceph-mon[94189]: pgmap v16769: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 539 KiB/s rd, 554 KiB/s wr, 26 op/s 2023-03-09T23:26:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30483.822324 secs 2023-03-09T23:26:13.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:13 smithi038 ceph-mon[92422]: pgmap v16769: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 539 KiB/s rd, 554 KiB/s wr, 26 op/s 2023-03-09T23:26:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30483.822324 secs 2023-03-09T23:26:13.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30945.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:26:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:15 smithi122 ceph-mon[93869]: pgmap v16770: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 398 KiB/s wr, 10 op/s 2023-03-09T23:26:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30486.418119 secs 2023-03-09T23:26:15.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:15 smithi181 ceph-mon[94189]: pgmap v16770: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 398 KiB/s wr, 10 op/s 2023-03-09T23:26:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30486.418119 secs 2023-03-09T23:26:15.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:15 smithi038 ceph-mon[92422]: pgmap v16770: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 398 KiB/s wr, 10 op/s 2023-03-09T23:26:15.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30486.418119 secs 2023-03-09T23:26:16.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:26:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:26:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:26:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:26:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:26:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:26:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:17 smithi122 ceph-mon[93869]: pgmap v16771: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 370 KiB/s wr, 9 op/s 2023-03-09T23:26:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:26:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:26:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:17 smithi122 ceph-mon[93869]: osdmap e1264: 12 total, 12 up, 12 in 2023-03-09T23:26:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:17 smithi181 ceph-mon[94189]: pgmap v16771: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 370 KiB/s wr, 9 op/s 2023-03-09T23:26:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:26:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:26:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:17 smithi181 ceph-mon[94189]: osdmap e1264: 12 total, 12 up, 12 in 2023-03-09T23:26:17.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:17 smithi038 ceph-mon[92422]: pgmap v16771: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 370 KiB/s wr, 9 op/s 2023-03-09T23:26:17.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:26:17.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:26:17.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:17 smithi038 ceph-mon[92422]: osdmap e1264: 12 total, 12 up, 12 in 2023-03-09T23:26:18.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30488.822397 secs 2023-03-09T23:26:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30488.822397 secs 2023-03-09T23:26:18.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30488.822397 secs 2023-03-09T23:26:18.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30950.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:26:19.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:19 smithi122 ceph-mon[93869]: pgmap v16773: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 420 KiB/s wr, 10 op/s 2023-03-09T23:26:19.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:19 smithi181 ceph-mon[94189]: pgmap v16773: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 420 KiB/s wr, 10 op/s 2023-03-09T23:26:19.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:19 smithi038 ceph-mon[92422]: pgmap v16773: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 420 KiB/s wr, 10 op/s 2023-03-09T23:26:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30491.418194 secs 2023-03-09T23:26:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30491.418194 secs 2023-03-09T23:26:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30491.418194 secs 2023-03-09T23:26:21.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:21 smithi122 ceph-mon[93869]: pgmap v16774: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 370 KiB/s wr, 8 op/s 2023-03-09T23:26:21.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:21 smithi181 ceph-mon[94189]: pgmap v16774: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 370 KiB/s wr, 8 op/s 2023-03-09T23:26:21.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:21 smithi038 ceph-mon[92422]: pgmap v16774: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 370 KiB/s wr, 8 op/s 2023-03-09T23:26:23.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:23 smithi122 ceph-mon[93869]: pgmap v16775: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 111 KiB/s wr, 7 op/s 2023-03-09T23:26:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30493.822518 secs 2023-03-09T23:26:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:23 smithi181 ceph-mon[94189]: pgmap v16775: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 111 KiB/s wr, 7 op/s 2023-03-09T23:26:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30493.822518 secs 2023-03-09T23:26:23.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:23 smithi038 ceph-mon[92422]: pgmap v16775: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 111 KiB/s wr, 7 op/s 2023-03-09T23:26:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30493.822518 secs 2023-03-09T23:26:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30955.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:26:25.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:25 smithi122 ceph-mon[93869]: pgmap v16776: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 111 KiB/s wr, 7 op/s 2023-03-09T23:26:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30496.418250 secs 2023-03-09T23:26:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:25 smithi181 ceph-mon[94189]: pgmap v16776: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 111 KiB/s wr, 7 op/s 2023-03-09T23:26:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30496.418250 secs 2023-03-09T23:26:25.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:25 smithi038 ceph-mon[92422]: pgmap v16776: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 111 KiB/s wr, 7 op/s 2023-03-09T23:26:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30496.418250 secs 2023-03-09T23:26:27.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:27 smithi122 ceph-mon[93869]: pgmap v16777: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 110 KiB/s wr, 7 op/s 2023-03-09T23:26:27.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:27 smithi181 ceph-mon[94189]: pgmap v16777: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 110 KiB/s wr, 7 op/s 2023-03-09T23:26:27.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:27 smithi038 ceph-mon[92422]: pgmap v16777: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 110 KiB/s wr, 7 op/s 2023-03-09T23:26:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30498.822611 secs 2023-03-09T23:26:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30498.822611 secs 2023-03-09T23:26:28.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30498.822611 secs 2023-03-09T23:26:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30960.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:26:29.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:29 smithi122 ceph-mon[93869]: pgmap v16778: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 100 KiB/s wr, 6 op/s 2023-03-09T23:26:29.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:29 smithi181 ceph-mon[94189]: pgmap v16778: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 100 KiB/s wr, 6 op/s 2023-03-09T23:26:29.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:29 smithi038 ceph-mon[92422]: pgmap v16778: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 100 KiB/s wr, 6 op/s 2023-03-09T23:26:30.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30501.418312 secs 2023-03-09T23:26:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30501.418312 secs 2023-03-09T23:26:30.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30501.418312 secs 2023-03-09T23:26:31.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:31 smithi122 ceph-mon[93869]: pgmap v16779: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 92 KiB/s wr, 5 op/s 2023-03-09T23:26:31.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:31 smithi181 ceph-mon[94189]: pgmap v16779: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 92 KiB/s wr, 5 op/s 2023-03-09T23:26:31.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:31 smithi038 ceph-mon[92422]: pgmap v16779: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 92 KiB/s wr, 5 op/s 2023-03-09T23:26:33.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:33 smithi122 ceph-mon[93869]: pgmap v16780: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 92 KiB/s wr, 8 op/s 2023-03-09T23:26:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30503.822735 secs 2023-03-09T23:26:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:26:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:33 smithi181 ceph-mon[94189]: pgmap v16780: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 92 KiB/s wr, 8 op/s 2023-03-09T23:26:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30503.822735 secs 2023-03-09T23:26:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:26:33.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:33 smithi038 ceph-mon[92422]: pgmap v16780: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 92 KiB/s wr, 8 op/s 2023-03-09T23:26:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30503.822735 secs 2023-03-09T23:26:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:26:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30965.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:26:35.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:35 smithi122 ceph-mon[93869]: pgmap v16781: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T23:26:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30506.418398 secs 2023-03-09T23:26:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:35 smithi181 ceph-mon[94189]: pgmap v16781: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T23:26:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30506.418398 secs 2023-03-09T23:26:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:35 smithi038 ceph-mon[92422]: pgmap v16781: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T23:26:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30506.418398 secs 2023-03-09T23:26:37.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:37 smithi122 ceph-mon[93869]: pgmap v16782: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 341 B/s wr, 8 op/s 2023-03-09T23:26:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:26:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:26:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:26:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:26:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:26:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:37 smithi181 ceph-mon[94189]: pgmap v16782: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 341 B/s wr, 8 op/s 2023-03-09T23:26:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:26:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:26:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:26:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:26:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:26:38.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:37 smithi038 ceph-mon[92422]: pgmap v16782: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 341 B/s wr, 8 op/s 2023-03-09T23:26:38.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:26:38.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:26:38.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:26:38.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:26:38.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:26:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30970.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:26:38.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30508.822850 secs 2023-03-09T23:26:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30508.822850 secs 2023-03-09T23:26:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30508.822850 secs 2023-03-09T23:26:39.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:39 smithi122 ceph-mon[93869]: pgmap v16783: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-09T23:26:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:39 smithi181 ceph-mon[94189]: pgmap v16783: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-09T23:26:39.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:39 smithi038 ceph-mon[92422]: pgmap v16783: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-09T23:26:40.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30511.418480 secs 2023-03-09T23:26:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30511.418480 secs 2023-03-09T23:26:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30511.418480 secs 2023-03-09T23:26:41.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:41 smithi122 ceph-mon[93869]: pgmap v16784: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-09T23:26:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:41 smithi181 ceph-mon[94189]: pgmap v16784: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-09T23:26:42.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:41 smithi038 ceph-mon[92422]: pgmap v16784: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-09T23:26:43.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:43 smithi122 ceph-mon[93869]: pgmap v16785: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 135 KiB/s wr, 10 op/s 2023-03-09T23:26:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30513.822989 secs 2023-03-09T23:26:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:43 smithi181 ceph-mon[94189]: pgmap v16785: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 135 KiB/s wr, 10 op/s 2023-03-09T23:26:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30513.822989 secs 2023-03-09T23:26:43.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:43 smithi038 ceph-mon[92422]: pgmap v16785: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 135 KiB/s wr, 10 op/s 2023-03-09T23:26:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30513.822989 secs 2023-03-09T23:26:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30975.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:26:45.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:45 smithi122 ceph-mon[93869]: pgmap v16786: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 135 KiB/s wr, 7 op/s 2023-03-09T23:26:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30516.418539 secs 2023-03-09T23:26:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:45 smithi181 ceph-mon[94189]: pgmap v16786: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 135 KiB/s wr, 7 op/s 2023-03-09T23:26:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30516.418539 secs 2023-03-09T23:26:45.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:45 smithi038 ceph-mon[92422]: pgmap v16786: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 135 KiB/s wr, 7 op/s 2023-03-09T23:26:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30516.418539 secs 2023-03-09T23:26:47.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:47 smithi122 ceph-mon[93869]: pgmap v16787: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 461 KiB/s wr, 10 op/s 2023-03-09T23:26:47.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:47 smithi181 ceph-mon[94189]: pgmap v16787: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 461 KiB/s wr, 10 op/s 2023-03-09T23:26:47.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:47 smithi038 ceph-mon[92422]: pgmap v16787: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 461 KiB/s wr, 10 op/s 2023-03-09T23:26:48.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30518.823145 secs 2023-03-09T23:26:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30518.823145 secs 2023-03-09T23:26:48.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30518.823145 secs 2023-03-09T23:26:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30980.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:26:49.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:49 smithi122 ceph-mon[93869]: pgmap v16788: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 461 KiB/s wr, 8 op/s 2023-03-09T23:26:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:26:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:26:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:49 smithi181 ceph-mon[94189]: pgmap v16788: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 461 KiB/s wr, 8 op/s 2023-03-09T23:26:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:26:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:26:49.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:49 smithi038 ceph-mon[92422]: pgmap v16788: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 461 KiB/s wr, 8 op/s 2023-03-09T23:26:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:26:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:26:50.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30521.418599 secs 2023-03-09T23:26:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30521.418599 secs 2023-03-09T23:26:50.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30521.418599 secs 2023-03-09T23:26:51.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:51 smithi122 ceph-mon[93869]: pgmap v16789: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 461 KiB/s wr, 7 op/s 2023-03-09T23:26:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:51 smithi181 ceph-mon[94189]: pgmap v16789: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 461 KiB/s wr, 7 op/s 2023-03-09T23:26:51.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:51 smithi038 ceph-mon[92422]: pgmap v16789: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 461 KiB/s wr, 7 op/s 2023-03-09T23:26:52.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:26:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:26:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:26:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:26:52.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:26:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:26:53.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:53 smithi122 ceph-mon[93869]: pgmap v16790: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 889 KiB/s rd, 701 KiB/s wr, 11 op/s 2023-03-09T23:26:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30523.823279 secs 2023-03-09T23:26:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:53 smithi181 ceph-mon[94189]: pgmap v16790: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 889 KiB/s rd, 701 KiB/s wr, 11 op/s 2023-03-09T23:26:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30523.823279 secs 2023-03-09T23:26:53.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:53 smithi038 ceph-mon[92422]: pgmap v16790: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 889 KiB/s rd, 701 KiB/s wr, 11 op/s 2023-03-09T23:26:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30523.823279 secs 2023-03-09T23:26:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30985.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:26:55.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:55 smithi122 ceph-mon[93869]: pgmap v16791: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 753 KiB/s rd, 566 KiB/s wr, 8 op/s 2023-03-09T23:26:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30526.418657 secs 2023-03-09T23:26:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:55 smithi181 ceph-mon[94189]: pgmap v16791: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 753 KiB/s rd, 566 KiB/s wr, 8 op/s 2023-03-09T23:26:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30526.418657 secs 2023-03-09T23:26:55.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:55 smithi038 ceph-mon[92422]: pgmap v16791: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 753 KiB/s rd, 566 KiB/s wr, 8 op/s 2023-03-09T23:26:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30526.418657 secs 2023-03-09T23:26:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:26:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:26:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:26:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:26:56.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:26:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:26:57.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:57 smithi122 ceph-mon[93869]: pgmap v16792: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 902 KiB/s rd, 714 KiB/s wr, 12 op/s 2023-03-09T23:26:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:26:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:26:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:57 smithi181 ceph-mon[94189]: pgmap v16792: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 902 KiB/s rd, 714 KiB/s wr, 12 op/s 2023-03-09T23:26:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:26:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:26:57.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:57 smithi038 ceph-mon[92422]: pgmap v16792: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 902 KiB/s rd, 714 KiB/s wr, 12 op/s 2023-03-09T23:26:57.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:26:57.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:26:58.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30528.823371 secs 2023-03-09T23:26:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30990.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:26:58.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30528.823371 secs 2023-03-09T23:26:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30528.823371 secs 2023-03-09T23:26:59.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:26:59 smithi038 ceph-mon[92422]: pgmap v16793: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 878 KiB/s rd, 515 KiB/s wr, 10 op/s 2023-03-09T23:26:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:26:59 smithi122 ceph-mon[93869]: pgmap v16793: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 878 KiB/s rd, 515 KiB/s wr, 10 op/s 2023-03-09T23:26:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:26:59 smithi181 ceph-mon[94189]: pgmap v16793: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 878 KiB/s rd, 515 KiB/s wr, 10 op/s 2023-03-09T23:27:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:27:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:27:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30531.418715 secs 2023-03-09T23:27:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:27:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:27:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30531.418715 secs 2023-03-09T23:27:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:27:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:27:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30531.418715 secs 2023-03-09T23:27:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:01 smithi122 ceph-mon[93869]: pgmap v16794: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 737 KiB/s wr, 11 op/s 2023-03-09T23:27:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:27:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:27:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:01 smithi181 ceph-mon[94189]: pgmap v16794: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 737 KiB/s wr, 11 op/s 2023-03-09T23:27:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:27:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:27:01.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:01 smithi038 ceph-mon[92422]: pgmap v16794: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 737 KiB/s wr, 11 op/s 2023-03-09T23:27:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:27:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:27:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:03 smithi122 ceph-mon[93869]: pgmap v16795: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 905 KiB/s wr, 15 op/s 2023-03-09T23:27:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30533.823488 secs 2023-03-09T23:27:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:03 smithi181 ceph-mon[94189]: pgmap v16795: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 905 KiB/s wr, 15 op/s 2023-03-09T23:27:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30533.823488 secs 2023-03-09T23:27:03.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:03 smithi038 ceph-mon[92422]: pgmap v16795: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 905 KiB/s wr, 15 op/s 2023-03-09T23:27:03.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30533.823488 secs 2023-03-09T23:27:03.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 30995.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:27:05.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:05 smithi122 ceph-mon[93869]: pgmap v16796: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 679 KiB/s rd, 665 KiB/s wr, 12 op/s 2023-03-09T23:27:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30536.418788 secs 2023-03-09T23:27:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:05 smithi181 ceph-mon[94189]: pgmap v16796: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 679 KiB/s rd, 665 KiB/s wr, 12 op/s 2023-03-09T23:27:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30536.418788 secs 2023-03-09T23:27:05.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:05 smithi038 ceph-mon[92422]: pgmap v16796: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 679 KiB/s rd, 665 KiB/s wr, 12 op/s 2023-03-09T23:27:05.867 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30536.418788 secs 2023-03-09T23:27:07.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:07 smithi122 ceph-mon[93869]: pgmap v16797: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 728 KiB/s rd, 734 KiB/s wr, 17 op/s 2023-03-09T23:27:07.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:07 smithi181 ceph-mon[94189]: pgmap v16797: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 728 KiB/s rd, 734 KiB/s wr, 17 op/s 2023-03-09T23:27:07.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:07 smithi038 ceph-mon[92422]: pgmap v16797: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 728 KiB/s rd, 734 KiB/s wr, 17 op/s 2023-03-09T23:27:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30538.823624 secs 2023-03-09T23:27:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:27:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:27:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30538.823624 secs 2023-03-09T23:27:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:27:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:27:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30538.823624 secs 2023-03-09T23:27:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:27:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:27:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31000.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:27:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:09 smithi122 ceph-mon[93869]: pgmap v16798: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 663 KiB/s rd, 686 KiB/s wr, 26 op/s 2023-03-09T23:27:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:27:09.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:27:09.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:09 smithi181 ceph-mon[94189]: pgmap v16798: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 663 KiB/s rd, 686 KiB/s wr, 26 op/s 2023-03-09T23:27:09.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:27:09.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:27:09.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:09 smithi038 ceph-mon[92422]: pgmap v16798: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 663 KiB/s rd, 686 KiB/s wr, 26 op/s 2023-03-09T23:27:09.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:27:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:27:10.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:10 smithi122 ceph-mon[93869]: osdmap e1265: 12 total, 12 up, 12 in 2023-03-09T23:27:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30541.418847 secs 2023-03-09T23:27:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:10 smithi181 ceph-mon[94189]: osdmap e1265: 12 total, 12 up, 12 in 2023-03-09T23:27:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30541.418847 secs 2023-03-09T23:27:10.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:10 smithi038 ceph-mon[92422]: osdmap e1265: 12 total, 12 up, 12 in 2023-03-09T23:27:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30541.418847 secs 2023-03-09T23:27:11.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:11 smithi122 ceph-mon[93869]: pgmap v16800: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 469 KiB/s wr, 29 op/s 2023-03-09T23:27:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:11 smithi181 ceph-mon[94189]: pgmap v16800: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 469 KiB/s wr, 29 op/s 2023-03-09T23:27:11.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:11 smithi038 ceph-mon[92422]: pgmap v16800: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 469 KiB/s wr, 29 op/s 2023-03-09T23:27:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:27:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:27:12.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:27:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:27:12.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:27:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:27:13.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30543.823743 secs 2023-03-09T23:27:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:13 smithi038 ceph-mon[92422]: pgmap v16801: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 415 KiB/s wr, 29 op/s 2023-03-09T23:27:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:27:13.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:27:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31005.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:27:13.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30543.823743 secs 2023-03-09T23:27:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:13 smithi122 ceph-mon[93869]: pgmap v16801: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 415 KiB/s wr, 29 op/s 2023-03-09T23:27:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:27:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:27:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30543.823743 secs 2023-03-09T23:27:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:13 smithi181 ceph-mon[94189]: pgmap v16801: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 415 KiB/s wr, 29 op/s 2023-03-09T23:27:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:27:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:27:15.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:15 smithi038 ceph-mon[92422]: pgmap v16802: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 531 KiB/s rd, 572 KiB/s wr, 28 op/s 2023-03-09T23:27:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30546.418923 secs 2023-03-09T23:27:15.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:15 smithi122 ceph-mon[93869]: pgmap v16802: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 531 KiB/s rd, 572 KiB/s wr, 28 op/s 2023-03-09T23:27:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30546.418923 secs 2023-03-09T23:27:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:15 smithi181 ceph-mon[94189]: pgmap v16802: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 531 KiB/s rd, 572 KiB/s wr, 28 op/s 2023-03-09T23:27:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30546.418923 secs 2023-03-09T23:27:17.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:17 smithi038 ceph-mon[92422]: pgmap v16803: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 474 KiB/s rd, 490 KiB/s wr, 24 op/s 2023-03-09T23:27:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:17 smithi038 ceph-mon[92422]: osdmap e1266: 12 total, 12 up, 12 in 2023-03-09T23:27:17.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:17 smithi122 ceph-mon[93869]: pgmap v16803: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 474 KiB/s rd, 490 KiB/s wr, 24 op/s 2023-03-09T23:27:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:17 smithi122 ceph-mon[93869]: osdmap e1266: 12 total, 12 up, 12 in 2023-03-09T23:27:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:17 smithi181 ceph-mon[94189]: pgmap v16803: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 474 KiB/s rd, 490 KiB/s wr, 24 op/s 2023-03-09T23:27:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:17 smithi181 ceph-mon[94189]: osdmap e1266: 12 total, 12 up, 12 in 2023-03-09T23:27:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31010.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:27:18.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30548.823857 secs 2023-03-09T23:27:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30548.823857 secs 2023-03-09T23:27:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30548.823857 secs 2023-03-09T23:27:19.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:19 smithi122 ceph-mon[93869]: pgmap v16805: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 342 KiB/s wr, 9 op/s 2023-03-09T23:27:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:19 smithi181 ceph-mon[94189]: pgmap v16805: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 342 KiB/s wr, 9 op/s 2023-03-09T23:27:19.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:19 smithi038 ceph-mon[92422]: pgmap v16805: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 342 KiB/s wr, 9 op/s 2023-03-09T23:27:20.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30551.418988 secs 2023-03-09T23:27:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30551.418988 secs 2023-03-09T23:27:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30551.418988 secs 2023-03-09T23:27:21.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:21 smithi122 ceph-mon[93869]: pgmap v16806: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 416 KiB/s wr, 9 op/s 2023-03-09T23:27:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:21 smithi181 ceph-mon[94189]: pgmap v16806: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 416 KiB/s wr, 9 op/s 2023-03-09T23:27:21.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:21 smithi038 ceph-mon[92422]: pgmap v16806: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 416 KiB/s wr, 9 op/s 2023-03-09T23:27:23.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30553.823989 secs 2023-03-09T23:27:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:23 smithi122 ceph-mon[93869]: pgmap v16807: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 269 KiB/s wr, 7 op/s 2023-03-09T23:27:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30553.823989 secs 2023-03-09T23:27:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:23 smithi181 ceph-mon[94189]: pgmap v16807: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 269 KiB/s wr, 7 op/s 2023-03-09T23:27:23.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30553.823989 secs 2023-03-09T23:27:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:23 smithi038 ceph-mon[92422]: pgmap v16807: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 269 KiB/s wr, 7 op/s 2023-03-09T23:27:23.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31015.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:27:25.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:25 smithi122 ceph-mon[93869]: pgmap v16808: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 111 KiB/s wr, 6 op/s 2023-03-09T23:27:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30556.419047 secs 2023-03-09T23:27:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:25 smithi181 ceph-mon[94189]: pgmap v16808: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 111 KiB/s wr, 6 op/s 2023-03-09T23:27:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30556.419047 secs 2023-03-09T23:27:25.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:25 smithi038 ceph-mon[92422]: pgmap v16808: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 111 KiB/s wr, 6 op/s 2023-03-09T23:27:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30556.419047 secs 2023-03-09T23:27:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:27 smithi122 ceph-mon[93869]: pgmap v16809: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 111 KiB/s wr, 6 op/s 2023-03-09T23:27:27.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:27 smithi181 ceph-mon[94189]: pgmap v16809: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 111 KiB/s wr, 6 op/s 2023-03-09T23:27:27.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:27 smithi038 ceph-mon[92422]: pgmap v16809: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 111 KiB/s wr, 6 op/s 2023-03-09T23:27:28.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30558.824132 secs 2023-03-09T23:27:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30558.824132 secs 2023-03-09T23:27:28.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30558.824132 secs 2023-03-09T23:27:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31020.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:27:29.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:29 smithi122 ceph-mon[93869]: pgmap v16810: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 100 KiB/s wr, 5 op/s 2023-03-09T23:27:29.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:29 smithi181 ceph-mon[94189]: pgmap v16810: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 100 KiB/s wr, 5 op/s 2023-03-09T23:27:29.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:29 smithi038 ceph-mon[92422]: pgmap v16810: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 100 KiB/s wr, 5 op/s 2023-03-09T23:27:30.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30561.419107 secs 2023-03-09T23:27:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30561.419107 secs 2023-03-09T23:27:30.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30561.419107 secs 2023-03-09T23:27:31.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:31 smithi122 ceph-mon[93869]: pgmap v16811: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 KiB/s wr, 5 op/s 2023-03-09T23:27:31.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:31 smithi181 ceph-mon[94189]: pgmap v16811: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 KiB/s wr, 5 op/s 2023-03-09T23:27:31.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:31 smithi038 ceph-mon[92422]: pgmap v16811: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 KiB/s wr, 5 op/s 2023-03-09T23:27:33.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30563.824235 secs 2023-03-09T23:27:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:33 smithi122 ceph-mon[93869]: pgmap v16812: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T23:27:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30563.824235 secs 2023-03-09T23:27:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:33 smithi181 ceph-mon[94189]: pgmap v16812: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T23:27:33.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30563.824235 secs 2023-03-09T23:27:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:33 smithi038 ceph-mon[92422]: pgmap v16812: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T23:27:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31025.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:27:35.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:35 smithi122 ceph-mon[93869]: pgmap v16813: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 64 KiB/s wr, 5 op/s 2023-03-09T23:27:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30566.419180 secs 2023-03-09T23:27:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:35 smithi181 ceph-mon[94189]: pgmap v16813: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 64 KiB/s wr, 5 op/s 2023-03-09T23:27:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30566.419180 secs 2023-03-09T23:27:35.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:35 smithi038 ceph-mon[92422]: pgmap v16813: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 64 KiB/s wr, 5 op/s 2023-03-09T23:27:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30566.419180 secs 2023-03-09T23:27:37.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:37 smithi122 ceph-mon[93869]: pgmap v16814: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 112 KiB/s wr, 8 op/s 2023-03-09T23:27:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:27:37.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:37 smithi181 ceph-mon[94189]: pgmap v16814: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 112 KiB/s wr, 8 op/s 2023-03-09T23:27:37.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:27:37.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:37 smithi038 ceph-mon[92422]: pgmap v16814: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 112 KiB/s wr, 8 op/s 2023-03-09T23:27:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:27:38.712 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30568.824302 secs 2023-03-09T23:27:38.723 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30568.824302 secs 2023-03-09T23:27:38.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30568.824302 secs 2023-03-09T23:27:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31030.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:27:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:39 smithi122 ceph-mon[93869]: pgmap v16815: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 112 KiB/s wr, 6 op/s 2023-03-09T23:27:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:39 smithi181 ceph-mon[94189]: pgmap v16815: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 112 KiB/s wr, 6 op/s 2023-03-09T23:27:39.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:39 smithi038 ceph-mon[92422]: pgmap v16815: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 112 KiB/s wr, 6 op/s 2023-03-09T23:27:40.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:27:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:27:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30571.419243 secs 2023-03-09T23:27:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:27:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:27:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:27:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:27:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:27:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:27:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:27:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:27:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:27:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30571.419243 secs 2023-03-09T23:27:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:27:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:27:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:27:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:27:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:27:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:27:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:27:40.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:27:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:27:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30571.419243 secs 2023-03-09T23:27:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:27:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:27:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:27:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:27:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:27:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:27:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:27:41.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:41 smithi038 ceph-mon[92422]: pgmap v16816: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 112 KiB/s wr, 6 op/s 2023-03-09T23:27:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:27:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:27:41.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:41 smithi122 ceph-mon[93869]: pgmap v16816: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 112 KiB/s wr, 6 op/s 2023-03-09T23:27:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:27:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:27:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:41 smithi181 ceph-mon[94189]: pgmap v16816: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 112 KiB/s wr, 6 op/s 2023-03-09T23:27:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:27:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:27:43.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:43 smithi122 ceph-mon[93869]: pgmap v16817: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 505 KiB/s rd, 247 KiB/s wr, 9 op/s 2023-03-09T23:27:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:43 smithi181 ceph-mon[94189]: pgmap v16817: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 505 KiB/s rd, 247 KiB/s wr, 9 op/s 2023-03-09T23:27:43.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:43 smithi038 ceph-mon[92422]: pgmap v16817: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 505 KiB/s rd, 247 KiB/s wr, 9 op/s 2023-03-09T23:27:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31035.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:27:45.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:45 smithi122 ceph-mon[93869]: pgmap v16818: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 247 KiB/s wr, 8 op/s 2023-03-09T23:27:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30576.419304 secs 2023-03-09T23:27:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:45 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:27:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:45 smithi181 ceph-mon[94189]: pgmap v16818: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 247 KiB/s wr, 8 op/s 2023-03-09T23:27:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30576.419304 secs 2023-03-09T23:27:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:45 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:27:45.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:45 smithi038 ceph-mon[92422]: pgmap v16818: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 247 KiB/s wr, 8 op/s 2023-03-09T23:27:45.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30576.419304 secs 2023-03-09T23:27:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:45 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:27:46.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:46 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:27:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:46 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:27:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:46 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:27:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:46 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:27:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:46 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:27:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:46 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:27:46.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:46 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:27:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:46 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:27:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:46 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:27:47.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:47 smithi122 ceph-mon[93869]: pgmap v16819: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 342 KiB/s wr, 10 op/s 2023-03-09T23:27:47.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:47 smithi181 ceph-mon[94189]: pgmap v16819: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 342 KiB/s wr, 10 op/s 2023-03-09T23:27:47.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:47 smithi038 ceph-mon[92422]: pgmap v16819: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 342 KiB/s wr, 10 op/s 2023-03-09T23:27:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31040.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:27:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:49 smithi122 ceph-mon[93869]: pgmap v16820: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 294 KiB/s wr, 7 op/s 2023-03-09T23:27:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:49 smithi181 ceph-mon[94189]: pgmap v16820: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 294 KiB/s wr, 7 op/s 2023-03-09T23:27:49.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:49 smithi038 ceph-mon[92422]: pgmap v16820: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 294 KiB/s wr, 7 op/s 2023-03-09T23:27:50.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30581.419389 secs 2023-03-09T23:27:50.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30581.419389 secs 2023-03-09T23:27:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30581.419389 secs 2023-03-09T23:27:51.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:51 smithi038 ceph-mon[92422]: pgmap v16821: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 402 KiB/s rd, 336 KiB/s wr, 8 op/s 2023-03-09T23:27:51.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:51 smithi122 ceph-mon[93869]: pgmap v16821: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 402 KiB/s rd, 336 KiB/s wr, 8 op/s 2023-03-09T23:27:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:51 smithi181 ceph-mon[94189]: pgmap v16821: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 402 KiB/s rd, 336 KiB/s wr, 8 op/s 2023-03-09T23:27:53.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:53 smithi038 ceph-mon[92422]: pgmap v16822: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 729 KiB/s rd, 680 KiB/s wr, 11 op/s 2023-03-09T23:27:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:27:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:27:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31045.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:27:53.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:53 smithi122 ceph-mon[93869]: pgmap v16822: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 729 KiB/s rd, 680 KiB/s wr, 11 op/s 2023-03-09T23:27:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:27:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:27:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:53 smithi181 ceph-mon[94189]: pgmap v16822: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 729 KiB/s rd, 680 KiB/s wr, 11 op/s 2023-03-09T23:27:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:27:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:27:55.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:55 smithi122 ceph-mon[93869]: pgmap v16823: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 616 KiB/s rd, 568 KiB/s wr, 9 op/s 2023-03-09T23:27:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30586.419465 secs 2023-03-09T23:27:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:55 smithi181 ceph-mon[94189]: pgmap v16823: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 616 KiB/s rd, 568 KiB/s wr, 9 op/s 2023-03-09T23:27:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30586.419465 secs 2023-03-09T23:27:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:55 smithi038 ceph-mon[92422]: pgmap v16823: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 616 KiB/s rd, 568 KiB/s wr, 9 op/s 2023-03-09T23:27:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30586.419465 secs 2023-03-09T23:27:56.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:27:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:27:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:27:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:27:56.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:27:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:27:57.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:57 smithi122 ceph-mon[93869]: pgmap v16824: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 814 KiB/s wr, 13 op/s 2023-03-09T23:27:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:27:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:27:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:57 smithi181 ceph-mon[94189]: pgmap v16824: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 814 KiB/s wr, 13 op/s 2023-03-09T23:27:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:27:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:27:57.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:57 smithi038 ceph-mon[92422]: pgmap v16824: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 814 KiB/s wr, 13 op/s 2023-03-09T23:27:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:27:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:27:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31050.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:27:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:59 smithi122 ceph-mon[93869]: pgmap v16825: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 704 KiB/s rd, 797 KiB/s wr, 11 op/s 2023-03-09T23:27:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:27:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:27:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:27:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:59 smithi181 ceph-mon[94189]: pgmap v16825: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 704 KiB/s rd, 797 KiB/s wr, 11 op/s 2023-03-09T23:27:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:27:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:27:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:27:59.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:59 smithi038 ceph-mon[92422]: pgmap v16825: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 704 KiB/s rd, 797 KiB/s wr, 11 op/s 2023-03-09T23:27:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:27:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:27:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:28:00.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30591.419543 secs 2023-03-09T23:28:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30591.419543 secs 2023-03-09T23:28:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30591.419543 secs 2023-03-09T23:28:01.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:01 smithi122 ceph-mon[93869]: pgmap v16826: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 797 KiB/s rd, 879 KiB/s wr, 12 op/s 2023-03-09T23:28:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:28:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:28:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:01 smithi181 ceph-mon[94189]: pgmap v16826: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 797 KiB/s rd, 879 KiB/s wr, 12 op/s 2023-03-09T23:28:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:28:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:28:01.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:01 smithi038 ceph-mon[92422]: pgmap v16826: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 797 KiB/s rd, 879 KiB/s wr, 12 op/s 2023-03-09T23:28:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:28:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:28:03.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:03 smithi122 ceph-mon[93869]: pgmap v16827: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 757 KiB/s rd, 838 KiB/s wr, 15 op/s 2023-03-09T23:28:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:03 smithi181 ceph-mon[94189]: pgmap v16827: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 757 KiB/s rd, 838 KiB/s wr, 15 op/s 2023-03-09T23:28:03.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:03 smithi038 ceph-mon[92422]: pgmap v16827: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 757 KiB/s rd, 838 KiB/s wr, 15 op/s 2023-03-09T23:28:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31055.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:28:05.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:05 smithi122 ceph-mon[93869]: pgmap v16828: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 816 KiB/s rd, 741 KiB/s wr, 15 op/s 2023-03-09T23:28:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30596.419604 secs 2023-03-09T23:28:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:05 smithi181 ceph-mon[94189]: pgmap v16828: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 816 KiB/s rd, 741 KiB/s wr, 15 op/s 2023-03-09T23:28:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30596.419604 secs 2023-03-09T23:28:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:05 smithi038 ceph-mon[92422]: pgmap v16828: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 816 KiB/s rd, 741 KiB/s wr, 15 op/s 2023-03-09T23:28:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30596.419604 secs 2023-03-09T23:28:07.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:07 smithi122 ceph-mon[93869]: pgmap v16829: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 796 KiB/s rd, 741 KiB/s wr, 20 op/s 2023-03-09T23:28:07.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:07 smithi181 ceph-mon[94189]: pgmap v16829: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 796 KiB/s rd, 741 KiB/s wr, 20 op/s 2023-03-09T23:28:07.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:07 smithi038 ceph-mon[92422]: pgmap v16829: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 796 KiB/s rd, 741 KiB/s wr, 20 op/s 2023-03-09T23:28:08.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30598.825043 secs 2023-03-09T23:28:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30598.825043 secs 2023-03-09T23:28:08.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30598.825043 secs 2023-03-09T23:28:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31060.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:28:09.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:09 smithi122 ceph-mon[93869]: pgmap v16830: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 774 KiB/s rd, 677 KiB/s wr, 28 op/s 2023-03-09T23:28:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:09 smithi181 ceph-mon[94189]: pgmap v16830: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 774 KiB/s rd, 677 KiB/s wr, 28 op/s 2023-03-09T23:28:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:09 smithi038 ceph-mon[92422]: pgmap v16830: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 774 KiB/s rd, 677 KiB/s wr, 28 op/s 2023-03-09T23:28:10.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:10 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:28:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:10 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:28:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:10 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:28:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:10 smithi122 ceph-mon[93869]: osdmap e1267: 12 total, 12 up, 12 in 2023-03-09T23:28:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30601.419663 secs 2023-03-09T23:28:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:10 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:28:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:10 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:28:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:10 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:28:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:10 smithi181 ceph-mon[94189]: osdmap e1267: 12 total, 12 up, 12 in 2023-03-09T23:28:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30601.419663 secs 2023-03-09T23:28:10.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:10 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:28:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:10 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:28:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:10 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:28:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:10 smithi038 ceph-mon[92422]: osdmap e1267: 12 total, 12 up, 12 in 2023-03-09T23:28:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30601.419663 secs 2023-03-09T23:28:11.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:11 smithi038 ceph-mon[92422]: pgmap v16832: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 846 KiB/s rd, 572 KiB/s wr, 31 op/s 2023-03-09T23:28:11.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:11 smithi122 ceph-mon[93869]: pgmap v16832: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 846 KiB/s rd, 572 KiB/s wr, 31 op/s 2023-03-09T23:28:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:11 smithi181 ceph-mon[94189]: pgmap v16832: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 846 KiB/s rd, 572 KiB/s wr, 31 op/s 2023-03-09T23:28:12.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:28:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:28:12.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:28:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:28:12.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:28:12.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:28:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31065.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:28:13.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30603.825166 secs 2023-03-09T23:28:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:13 smithi122 ceph-mon[93869]: pgmap v16833: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 942 KiB/s wr, 31 op/s 2023-03-09T23:28:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:28:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:28:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30603.825166 secs 2023-03-09T23:28:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:13 smithi181 ceph-mon[94189]: pgmap v16833: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 942 KiB/s wr, 31 op/s 2023-03-09T23:28:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:28:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:28:13.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30603.825166 secs 2023-03-09T23:28:13.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:13 smithi038 ceph-mon[92422]: pgmap v16833: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 942 KiB/s wr, 31 op/s 2023-03-09T23:28:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:28:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:28:15.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:15 smithi122 ceph-mon[93869]: pgmap v16834: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 678 KiB/s rd, 646 KiB/s wr, 28 op/s 2023-03-09T23:28:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30606.419724 secs 2023-03-09T23:28:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:15 smithi181 ceph-mon[94189]: pgmap v16834: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 678 KiB/s rd, 646 KiB/s wr, 28 op/s 2023-03-09T23:28:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30606.419724 secs 2023-03-09T23:28:15.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:15 smithi038 ceph-mon[92422]: pgmap v16834: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 678 KiB/s rd, 646 KiB/s wr, 28 op/s 2023-03-09T23:28:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30606.419724 secs 2023-03-09T23:28:16.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:28:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:28:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:28:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:28:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:28:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:28:17.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:17 smithi122 ceph-mon[93869]: pgmap v16835: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 676 KiB/s rd, 619 KiB/s wr, 23 op/s 2023-03-09T23:28:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:28:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:28:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:17 smithi122 ceph-mon[93869]: osdmap e1268: 12 total, 12 up, 12 in 2023-03-09T23:28:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:17 smithi181 ceph-mon[94189]: pgmap v16835: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 676 KiB/s rd, 619 KiB/s wr, 23 op/s 2023-03-09T23:28:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:28:17.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:28:17.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:17 smithi181 ceph-mon[94189]: osdmap e1268: 12 total, 12 up, 12 in 2023-03-09T23:28:17.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:17 smithi038 ceph-mon[92422]: pgmap v16835: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 676 KiB/s rd, 619 KiB/s wr, 23 op/s 2023-03-09T23:28:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:28:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:28:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:17 smithi038 ceph-mon[92422]: osdmap e1268: 12 total, 12 up, 12 in 2023-03-09T23:28:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31070.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:28:18.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30608.825289 secs 2023-03-09T23:28:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30608.825289 secs 2023-03-09T23:28:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30608.825289 secs 2023-03-09T23:28:19.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:19 smithi122 ceph-mon[93869]: pgmap v16837: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 541 KiB/s wr, 9 op/s 2023-03-09T23:28:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:19 smithi181 ceph-mon[94189]: pgmap v16837: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 541 KiB/s wr, 9 op/s 2023-03-09T23:28:19.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:19 smithi038 ceph-mon[92422]: pgmap v16837: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 541 KiB/s wr, 9 op/s 2023-03-09T23:28:20.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30611.419787 secs 2023-03-09T23:28:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30611.419787 secs 2023-03-09T23:28:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30611.419787 secs 2023-03-09T23:28:21.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:21 smithi122 ceph-mon[93869]: pgmap v16838: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 485 KiB/s rd, 483 KiB/s wr, 9 op/s 2023-03-09T23:28:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:21 smithi181 ceph-mon[94189]: pgmap v16838: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 485 KiB/s rd, 483 KiB/s wr, 9 op/s 2023-03-09T23:28:21.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:21 smithi038 ceph-mon[92422]: pgmap v16838: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 485 KiB/s rd, 483 KiB/s wr, 9 op/s 2023-03-09T23:28:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30613.825414 secs 2023-03-09T23:28:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:23 smithi122 ceph-mon[93869]: pgmap v16839: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 112 KiB/s wr, 8 op/s 2023-03-09T23:28:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30613.825414 secs 2023-03-09T23:28:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:23 smithi181 ceph-mon[94189]: pgmap v16839: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 112 KiB/s wr, 8 op/s 2023-03-09T23:28:23.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30613.825414 secs 2023-03-09T23:28:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:23 smithi038 ceph-mon[92422]: pgmap v16839: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 112 KiB/s wr, 8 op/s 2023-03-09T23:28:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31075.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:28:25.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:25 smithi122 ceph-mon[93869]: pgmap v16840: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 111 KiB/s wr, 7 op/s 2023-03-09T23:28:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30616.419845 secs 2023-03-09T23:28:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:25 smithi181 ceph-mon[94189]: pgmap v16840: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 111 KiB/s wr, 7 op/s 2023-03-09T23:28:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30616.419845 secs 2023-03-09T23:28:25.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:25 smithi038 ceph-mon[92422]: pgmap v16840: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 111 KiB/s wr, 7 op/s 2023-03-09T23:28:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30616.419845 secs 2023-03-09T23:28:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:27 smithi122 ceph-mon[93869]: pgmap v16841: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 111 KiB/s wr, 7 op/s 2023-03-09T23:28:27.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:27 smithi181 ceph-mon[94189]: pgmap v16841: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 111 KiB/s wr, 7 op/s 2023-03-09T23:28:27.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:27 smithi038 ceph-mon[92422]: pgmap v16841: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 111 KiB/s wr, 7 op/s 2023-03-09T23:28:28.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30618.825540 secs 2023-03-09T23:28:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31080.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:28:28.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30618.825540 secs 2023-03-09T23:28:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30618.825540 secs 2023-03-09T23:28:29.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:29 smithi038 ceph-mon[92422]: pgmap v16842: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T23:28:29.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:29 smithi122 ceph-mon[93869]: pgmap v16842: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T23:28:29.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:29 smithi181 ceph-mon[94189]: pgmap v16842: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T23:28:30.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30621.419912 secs 2023-03-09T23:28:30.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30621.419912 secs 2023-03-09T23:28:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30621.419912 secs 2023-03-09T23:28:31.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:31 smithi038 ceph-mon[92422]: pgmap v16843: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T23:28:31.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:31 smithi122 ceph-mon[93869]: pgmap v16843: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T23:28:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:31 smithi181 ceph-mon[94189]: pgmap v16843: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T23:28:33.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30623.825665 secs 2023-03-09T23:28:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:33 smithi122 ceph-mon[93869]: pgmap v16844: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T23:28:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30623.825665 secs 2023-03-09T23:28:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:33 smithi181 ceph-mon[94189]: pgmap v16844: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T23:28:33.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30623.825665 secs 2023-03-09T23:28:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:33 smithi038 ceph-mon[92422]: pgmap v16844: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T23:28:33.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31085.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:28:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:35 smithi181 ceph-mon[94189]: pgmap v16845: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 6 op/s 2023-03-09T23:28:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30626.419974 secs 2023-03-09T23:28:35.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:35 smithi038 ceph-mon[92422]: pgmap v16845: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 6 op/s 2023-03-09T23:28:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30626.419974 secs 2023-03-09T23:28:35.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:35 smithi122 ceph-mon[93869]: pgmap v16845: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 6 op/s 2023-03-09T23:28:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30626.419974 secs 2023-03-09T23:28:37.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:37 smithi038 ceph-mon[92422]: pgmap v16846: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 341 B/s wr, 7 op/s 2023-03-09T23:28:37.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:37 smithi122 ceph-mon[93869]: pgmap v16846: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 341 B/s wr, 7 op/s 2023-03-09T23:28:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:37 smithi181 ceph-mon[94189]: pgmap v16846: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 341 B/s wr, 7 op/s 2023-03-09T23:28:38.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30628.825780 secs 2023-03-09T23:28:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31090.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:28:38.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30628.825780 secs 2023-03-09T23:28:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30628.825780 secs 2023-03-09T23:28:39.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:39 smithi038 ceph-mon[92422]: pgmap v16847: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 7 op/s 2023-03-09T23:28:39.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:39 smithi122 ceph-mon[93869]: pgmap v16847: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 7 op/s 2023-03-09T23:28:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:39 smithi181 ceph-mon[94189]: pgmap v16847: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 7 op/s 2023-03-09T23:28:40.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30631.419985 secs 2023-03-09T23:28:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30631.419985 secs 2023-03-09T23:28:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30631.419985 secs 2023-03-09T23:28:41.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:41 smithi122 ceph-mon[93869]: pgmap v16848: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 7 op/s 2023-03-09T23:28:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:28:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:28:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:28:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:28:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:41 smithi181 ceph-mon[94189]: pgmap v16848: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 7 op/s 2023-03-09T23:28:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:28:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:28:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:28:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:28:41.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:41 smithi038 ceph-mon[92422]: pgmap v16848: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 7 op/s 2023-03-09T23:28:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:28:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:28:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:28:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:28:43.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30633.825904 secs 2023-03-09T23:28:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:43 smithi122 ceph-mon[93869]: pgmap v16849: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 136 KiB/s wr, 10 op/s 2023-03-09T23:28:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30633.825904 secs 2023-03-09T23:28:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:43 smithi181 ceph-mon[94189]: pgmap v16849: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 136 KiB/s wr, 10 op/s 2023-03-09T23:28:43.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30633.825904 secs 2023-03-09T23:28:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:43 smithi038 ceph-mon[92422]: pgmap v16849: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 136 KiB/s wr, 10 op/s 2023-03-09T23:28:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31095.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:28:45.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:45 smithi122 ceph-mon[93869]: pgmap v16850: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 136 KiB/s wr, 8 op/s 2023-03-09T23:28:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30636.420044 secs 2023-03-09T23:28:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:45 smithi181 ceph-mon[94189]: pgmap v16850: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 136 KiB/s wr, 8 op/s 2023-03-09T23:28:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30636.420044 secs 2023-03-09T23:28:45.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:45 smithi038 ceph-mon[92422]: pgmap v16850: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 136 KiB/s wr, 8 op/s 2023-03-09T23:28:45.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30636.420044 secs 2023-03-09T23:28:47.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:47 smithi181 ceph-mon[94189]: pgmap v16851: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 136 KiB/s wr, 9 op/s 2023-03-09T23:28:47.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:47 smithi038 ceph-mon[92422]: pgmap v16851: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 136 KiB/s wr, 9 op/s 2023-03-09T23:28:47.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:47 smithi122 ceph-mon[93869]: pgmap v16851: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 136 KiB/s wr, 9 op/s 2023-03-09T23:28:48.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30638.826034 secs 2023-03-09T23:28:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31100.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:28:48.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30638.826034 secs 2023-03-09T23:28:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30638.826034 secs 2023-03-09T23:28:49.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:49 smithi038 ceph-mon[92422]: pgmap v16852: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 466 KiB/s rd, 464 KiB/s wr, 8 op/s 2023-03-09T23:28:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:28:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:28:49.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:49 smithi122 ceph-mon[93869]: pgmap v16852: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 466 KiB/s rd, 464 KiB/s wr, 8 op/s 2023-03-09T23:28:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:28:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:28:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:49 smithi181 ceph-mon[94189]: pgmap v16852: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 466 KiB/s rd, 464 KiB/s wr, 8 op/s 2023-03-09T23:28:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:28:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:28:50.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30641.420104 secs 2023-03-09T23:28:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30641.420104 secs 2023-03-09T23:28:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30641.420104 secs 2023-03-09T23:28:51.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:51 smithi038 ceph-mon[92422]: pgmap v16853: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 466 KiB/s rd, 464 KiB/s wr, 8 op/s 2023-03-09T23:28:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:51 smithi122 ceph-mon[93869]: pgmap v16853: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 466 KiB/s rd, 464 KiB/s wr, 8 op/s 2023-03-09T23:28:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:51 smithi181 ceph-mon[94189]: pgmap v16853: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 466 KiB/s rd, 464 KiB/s wr, 8 op/s 2023-03-09T23:28:52.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:28:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:28:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:28:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:28:52.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:28:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:28:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31105.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:28:53.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30643.826152 secs 2023-03-09T23:28:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:53 smithi122 ceph-mon[93869]: pgmap v16854: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 606 KiB/s rd, 464 KiB/s wr, 10 op/s 2023-03-09T23:28:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30643.826152 secs 2023-03-09T23:28:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:53 smithi181 ceph-mon[94189]: pgmap v16854: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 606 KiB/s rd, 464 KiB/s wr, 10 op/s 2023-03-09T23:28:53.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30643.826152 secs 2023-03-09T23:28:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:53 smithi038 ceph-mon[92422]: pgmap v16854: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 606 KiB/s rd, 464 KiB/s wr, 10 op/s 2023-03-09T23:28:55.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:55 smithi122 ceph-mon[93869]: pgmap v16855: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 570 KiB/s wr, 8 op/s 2023-03-09T23:28:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:28:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:28:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30646.420166 secs 2023-03-09T23:28:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:55 smithi181 ceph-mon[94189]: pgmap v16855: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 570 KiB/s wr, 8 op/s 2023-03-09T23:28:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:28:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:28:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30646.420166 secs 2023-03-09T23:28:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:55 smithi038 ceph-mon[92422]: pgmap v16855: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 570 KiB/s wr, 8 op/s 2023-03-09T23:28:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:28:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:28:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30646.420166 secs 2023-03-09T23:28:57.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:57 smithi122 ceph-mon[93869]: pgmap v16856: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 570 KiB/s wr, 10 op/s 2023-03-09T23:28:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:28:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:28:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:57 smithi181 ceph-mon[94189]: pgmap v16856: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 570 KiB/s wr, 10 op/s 2023-03-09T23:28:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:28:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:28:57.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:57 smithi038 ceph-mon[92422]: pgmap v16856: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 570 KiB/s wr, 10 op/s 2023-03-09T23:28:57.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:28:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:28:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31110.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:28:58.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30648.826272 secs 2023-03-09T23:28:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30648.826272 secs 2023-03-09T23:28:58.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30648.826272 secs 2023-03-09T23:28:59.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:59 smithi122 ceph-mon[93869]: pgmap v16857: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 959 KiB/s rd, 818 KiB/s wr, 11 op/s 2023-03-09T23:28:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:28:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:28:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:28:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:59 smithi181 ceph-mon[94189]: pgmap v16857: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 959 KiB/s rd, 818 KiB/s wr, 11 op/s 2023-03-09T23:28:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:28:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:28:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:28:59.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:59 smithi038 ceph-mon[92422]: pgmap v16857: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 959 KiB/s rd, 818 KiB/s wr, 11 op/s 2023-03-09T23:28:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:28:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:28:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:29:00.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30651.420233 secs 2023-03-09T23:29:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30651.420233 secs 2023-03-09T23:29:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30651.420233 secs 2023-03-09T23:29:01.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:01 smithi122 ceph-mon[93869]: pgmap v16858: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 670 KiB/s rd, 517 KiB/s wr, 10 op/s 2023-03-09T23:29:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:29:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:29:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:01 smithi181 ceph-mon[94189]: pgmap v16858: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 670 KiB/s rd, 517 KiB/s wr, 10 op/s 2023-03-09T23:29:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:29:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:29:01.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:01 smithi038 ceph-mon[92422]: pgmap v16858: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 670 KiB/s rd, 517 KiB/s wr, 10 op/s 2023-03-09T23:29:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:29:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:29:03.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30653.826413 secs 2023-03-09T23:29:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:03 smithi122 ceph-mon[93869]: pgmap v16859: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 884 KiB/s wr, 14 op/s 2023-03-09T23:29:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30653.826413 secs 2023-03-09T23:29:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:03 smithi181 ceph-mon[94189]: pgmap v16859: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 884 KiB/s wr, 14 op/s 2023-03-09T23:29:03.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30653.826413 secs 2023-03-09T23:29:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:03 smithi038 ceph-mon[92422]: pgmap v16859: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 884 KiB/s wr, 14 op/s 2023-03-09T23:29:03.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31115.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:29:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:05 smithi038 ceph-mon[92422]: pgmap v16860: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 945 KiB/s rd, 931 KiB/s wr, 14 op/s 2023-03-09T23:29:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30656.420323 secs 2023-03-09T23:29:05.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:05 smithi122 ceph-mon[93869]: pgmap v16860: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 945 KiB/s rd, 931 KiB/s wr, 14 op/s 2023-03-09T23:29:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30656.420323 secs 2023-03-09T23:29:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:05 smithi181 ceph-mon[94189]: pgmap v16860: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 945 KiB/s rd, 931 KiB/s wr, 14 op/s 2023-03-09T23:29:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30656.420323 secs 2023-03-09T23:29:07.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:07 smithi038 ceph-mon[92422]: pgmap v16861: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 791 KiB/s wr, 17 op/s 2023-03-09T23:29:07.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:07 smithi122 ceph-mon[93869]: pgmap v16861: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 791 KiB/s wr, 17 op/s 2023-03-09T23:29:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:07 smithi181 ceph-mon[94189]: pgmap v16861: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 791 KiB/s wr, 17 op/s 2023-03-09T23:29:08.793 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31120.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:29:08.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30658.826510 secs 2023-03-09T23:29:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:29:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:29:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30658.826510 secs 2023-03-09T23:29:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:29:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:29:08.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30658.826510 secs 2023-03-09T23:29:08.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:29:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:29:09.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:09 smithi122 ceph-mon[93869]: pgmap v16862: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 788 KiB/s rd, 812 KiB/s wr, 27 op/s 2023-03-09T23:29:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:29:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:29:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:09 smithi181 ceph-mon[94189]: pgmap v16862: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 788 KiB/s rd, 812 KiB/s wr, 27 op/s 2023-03-09T23:29:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:29:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:29:09.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:09 smithi038 ceph-mon[92422]: pgmap v16862: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 788 KiB/s rd, 812 KiB/s wr, 27 op/s 2023-03-09T23:29:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:29:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:29:10.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:10 smithi122 ceph-mon[93869]: osdmap e1269: 12 total, 12 up, 12 in 2023-03-09T23:29:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30661.420387 secs 2023-03-09T23:29:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:10 smithi181 ceph-mon[94189]: osdmap e1269: 12 total, 12 up, 12 in 2023-03-09T23:29:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30661.420387 secs 2023-03-09T23:29:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:10 smithi038 ceph-mon[92422]: osdmap e1269: 12 total, 12 up, 12 in 2023-03-09T23:29:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30661.420387 secs 2023-03-09T23:29:11.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:11 smithi122 ceph-mon[93869]: pgmap v16864: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 633 KiB/s rd, 674 KiB/s wr, 30 op/s 2023-03-09T23:29:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:11 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:29:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:11 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:29:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:11 smithi181 ceph-mon[94189]: pgmap v16864: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 633 KiB/s rd, 674 KiB/s wr, 30 op/s 2023-03-09T23:29:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:11 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:29:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:11 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:29:11.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:11 smithi038 ceph-mon[92422]: pgmap v16864: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 633 KiB/s rd, 674 KiB/s wr, 30 op/s 2023-03-09T23:29:11.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:11 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:29:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:11 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:29:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31125.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:29:13.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30663.826581 secs 2023-03-09T23:29:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:13 smithi122 ceph-mon[93869]: pgmap v16865: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 565 KiB/s rd, 606 KiB/s wr, 29 op/s 2023-03-09T23:29:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:29:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:29:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30663.826581 secs 2023-03-09T23:29:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:13 smithi181 ceph-mon[94189]: pgmap v16865: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 565 KiB/s rd, 606 KiB/s wr, 29 op/s 2023-03-09T23:29:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:29:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:29:13.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30663.826581 secs 2023-03-09T23:29:13.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:13 smithi038 ceph-mon[92422]: pgmap v16865: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 565 KiB/s rd, 606 KiB/s wr, 29 op/s 2023-03-09T23:29:13.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:29:13.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:29:15.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:15 smithi122 ceph-mon[93869]: pgmap v16866: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 508 KiB/s rd, 550 KiB/s wr, 27 op/s 2023-03-09T23:29:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30666.420446 secs 2023-03-09T23:29:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:15 smithi181 ceph-mon[94189]: pgmap v16866: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 508 KiB/s rd, 550 KiB/s wr, 27 op/s 2023-03-09T23:29:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30666.420446 secs 2023-03-09T23:29:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:15 smithi038 ceph-mon[92422]: pgmap v16866: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 508 KiB/s rd, 550 KiB/s wr, 27 op/s 2023-03-09T23:29:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30666.420446 secs 2023-03-09T23:29:17.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:17 smithi122 ceph-mon[93869]: pgmap v16867: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 429 KiB/s wr, 23 op/s 2023-03-09T23:29:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:17 smithi122 ceph-mon[93869]: osdmap e1270: 12 total, 12 up, 12 in 2023-03-09T23:29:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:17 smithi181 ceph-mon[94189]: pgmap v16867: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 429 KiB/s wr, 23 op/s 2023-03-09T23:29:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:17 smithi181 ceph-mon[94189]: osdmap e1270: 12 total, 12 up, 12 in 2023-03-09T23:29:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:17 smithi038 ceph-mon[92422]: pgmap v16867: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 429 KiB/s wr, 23 op/s 2023-03-09T23:29:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:17 smithi038 ceph-mon[92422]: osdmap e1270: 12 total, 12 up, 12 in 2023-03-09T23:29:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31130.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:29:18.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30668.826646 secs 2023-03-09T23:29:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30668.826646 secs 2023-03-09T23:29:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30668.826646 secs 2023-03-09T23:29:19.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:19 smithi122 ceph-mon[93869]: pgmap v16869: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 546 KiB/s rd, 543 KiB/s wr, 10 op/s 2023-03-09T23:29:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:19 smithi181 ceph-mon[94189]: pgmap v16869: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 546 KiB/s rd, 543 KiB/s wr, 10 op/s 2023-03-09T23:29:19.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:19 smithi038 ceph-mon[92422]: pgmap v16869: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 546 KiB/s rd, 543 KiB/s wr, 10 op/s 2023-03-09T23:29:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30671.420507 secs 2023-03-09T23:29:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30671.420507 secs 2023-03-09T23:29:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30671.420507 secs 2023-03-09T23:29:21.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:21 smithi122 ceph-mon[93869]: pgmap v16870: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 484 KiB/s wr, 9 op/s 2023-03-09T23:29:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:21 smithi181 ceph-mon[94189]: pgmap v16870: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 484 KiB/s wr, 9 op/s 2023-03-09T23:29:21.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:21 smithi038 ceph-mon[92422]: pgmap v16870: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 484 KiB/s wr, 9 op/s 2023-03-09T23:29:23.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30673.826788 secs 2023-03-09T23:29:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:23 smithi122 ceph-mon[93869]: pgmap v16871: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 112 KiB/s wr, 7 op/s 2023-03-09T23:29:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30673.826788 secs 2023-03-09T23:29:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:23 smithi181 ceph-mon[94189]: pgmap v16871: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 112 KiB/s wr, 7 op/s 2023-03-09T23:29:23.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30673.826788 secs 2023-03-09T23:29:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:23 smithi038 ceph-mon[92422]: pgmap v16871: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 112 KiB/s wr, 7 op/s 2023-03-09T23:29:23.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31135.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:29:25.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:25 smithi038 ceph-mon[92422]: pgmap v16872: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 111 KiB/s wr, 6 op/s 2023-03-09T23:29:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30676.420566 secs 2023-03-09T23:29:25.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:25 smithi122 ceph-mon[93869]: pgmap v16872: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 111 KiB/s wr, 6 op/s 2023-03-09T23:29:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30676.420566 secs 2023-03-09T23:29:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:25 smithi181 ceph-mon[94189]: pgmap v16872: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 111 KiB/s wr, 6 op/s 2023-03-09T23:29:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30676.420566 secs 2023-03-09T23:29:27.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:27 smithi038 ceph-mon[92422]: pgmap v16873: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 111 KiB/s wr, 6 op/s 2023-03-09T23:29:27.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:27 smithi122 ceph-mon[93869]: pgmap v16873: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 111 KiB/s wr, 6 op/s 2023-03-09T23:29:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:27 smithi181 ceph-mon[94189]: pgmap v16873: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 111 KiB/s wr, 6 op/s 2023-03-09T23:29:28.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31140.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:29:28.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30678.826876 secs 2023-03-09T23:29:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30678.826876 secs 2023-03-09T23:29:29.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30678.826876 secs 2023-03-09T23:29:29.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:29 smithi122 ceph-mon[93869]: pgmap v16874: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2023-03-09T23:29:29.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:29 smithi181 ceph-mon[94189]: pgmap v16874: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2023-03-09T23:29:29.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:29 smithi038 ceph-mon[92422]: pgmap v16874: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2023-03-09T23:29:30.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30681.420628 secs 2023-03-09T23:29:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30681.420628 secs 2023-03-09T23:29:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30681.420628 secs 2023-03-09T23:29:31.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:31 smithi122 ceph-mon[93869]: pgmap v16875: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T23:29:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:31 smithi181 ceph-mon[94189]: pgmap v16875: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T23:29:31.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:31 smithi038 ceph-mon[92422]: pgmap v16875: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T23:29:33.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30683.826945 secs 2023-03-09T23:29:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:33 smithi122 ceph-mon[93869]: pgmap v16876: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T23:29:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30683.826945 secs 2023-03-09T23:29:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:33 smithi181 ceph-mon[94189]: pgmap v16876: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T23:29:33.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30683.826945 secs 2023-03-09T23:29:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:33 smithi038 ceph-mon[92422]: pgmap v16876: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T23:29:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31145.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:29:35.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:35 smithi038 ceph-mon[92422]: pgmap v16877: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T23:29:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30686.420705 secs 2023-03-09T23:29:35.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:35 smithi122 ceph-mon[93869]: pgmap v16877: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T23:29:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30686.420705 secs 2023-03-09T23:29:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:35 smithi181 ceph-mon[94189]: pgmap v16877: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T23:29:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30686.420705 secs 2023-03-09T23:29:37.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:37 smithi122 ceph-mon[93869]: pgmap v16878: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-09T23:29:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:37 smithi181 ceph-mon[94189]: pgmap v16878: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-09T23:29:37.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:37 smithi038 ceph-mon[92422]: pgmap v16878: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-09T23:29:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31150.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:29:38.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30688.827076 secs 2023-03-09T23:29:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30688.827076 secs 2023-03-09T23:29:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30688.827076 secs 2023-03-09T23:29:39.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:39 smithi122 ceph-mon[93869]: pgmap v16879: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 112 KiB/s wr, 6 op/s 2023-03-09T23:29:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:39 smithi181 ceph-mon[94189]: pgmap v16879: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 112 KiB/s wr, 6 op/s 2023-03-09T23:29:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:39 smithi038 ceph-mon[92422]: pgmap v16879: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 112 KiB/s wr, 6 op/s 2023-03-09T23:29:40.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30691.420768 secs 2023-03-09T23:29:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30691.420768 secs 2023-03-09T23:29:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30691.420768 secs 2023-03-09T23:29:41.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:41 smithi122 ceph-mon[93869]: pgmap v16880: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 113 KiB/s wr, 6 op/s 2023-03-09T23:29:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:29:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:41 smithi181 ceph-mon[94189]: pgmap v16880: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 113 KiB/s wr, 6 op/s 2023-03-09T23:29:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:29:41.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:41 smithi038 ceph-mon[92422]: pgmap v16880: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 113 KiB/s wr, 6 op/s 2023-03-09T23:29:41.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:29:43.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31155.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:29:43.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30693.827132 secs 2023-03-09T23:29:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:43 smithi122 ceph-mon[93869]: pgmap v16881: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 113 KiB/s wr, 9 op/s 2023-03-09T23:29:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:29:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:29:43.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:29:43.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:29:43.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:29:43.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:29:43.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:29:43.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:29:43.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:29:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30693.827132 secs 2023-03-09T23:29:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:43 smithi181 ceph-mon[94189]: pgmap v16881: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 113 KiB/s wr, 9 op/s 2023-03-09T23:29:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:29:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:29:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:29:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:29:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:29:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:29:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:29:43.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:29:43.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:29:43.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30693.827132 secs 2023-03-09T23:29:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:43 smithi038 ceph-mon[92422]: pgmap v16881: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 113 KiB/s wr, 9 op/s 2023-03-09T23:29:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:29:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:29:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:29:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:29:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:29:44.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:29:44.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:29:44.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:29:44.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:29:45.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:45 smithi122 ceph-mon[93869]: pgmap v16882: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 113 KiB/s wr, 7 op/s 2023-03-09T23:29:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30696.420864 secs 2023-03-09T23:29:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:45 smithi181 ceph-mon[94189]: pgmap v16882: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 113 KiB/s wr, 7 op/s 2023-03-09T23:29:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30696.420864 secs 2023-03-09T23:29:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:45 smithi038 ceph-mon[92422]: pgmap v16882: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 113 KiB/s wr, 7 op/s 2023-03-09T23:29:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30696.420864 secs 2023-03-09T23:29:47.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:47 smithi122 ceph-mon[93869]: pgmap v16883: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 113 KiB/s wr, 8 op/s 2023-03-09T23:29:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:29:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:29:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:47 smithi181 ceph-mon[94189]: pgmap v16883: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 113 KiB/s wr, 8 op/s 2023-03-09T23:29:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:29:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:29:47.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:47 smithi038 ceph-mon[92422]: pgmap v16883: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 113 KiB/s wr, 8 op/s 2023-03-09T23:29:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:29:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:29:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31160.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:29:48.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30698.827222 secs 2023-03-09T23:29:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30698.827222 secs 2023-03-09T23:29:48.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30698.827222 secs 2023-03-09T23:29:49.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:49 smithi122 ceph-mon[93869]: pgmap v16884: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 514 KiB/s rd, 457 KiB/s wr, 9 op/s 2023-03-09T23:29:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:29:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:29:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:49 smithi181 ceph-mon[94189]: pgmap v16884: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 514 KiB/s rd, 457 KiB/s wr, 9 op/s 2023-03-09T23:29:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:29:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:29:49.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:49 smithi038 ceph-mon[92422]: pgmap v16884: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 514 KiB/s rd, 457 KiB/s wr, 9 op/s 2023-03-09T23:29:49.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:29:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:29:50.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30701.420925 secs 2023-03-09T23:29:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30701.420925 secs 2023-03-09T23:29:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30701.420925 secs 2023-03-09T23:29:51.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:51 smithi122 ceph-mon[93869]: pgmap v16885: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 370 KiB/s wr, 8 op/s 2023-03-09T23:29:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:29:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:29:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:51 smithi181 ceph-mon[94189]: pgmap v16885: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 370 KiB/s wr, 8 op/s 2023-03-09T23:29:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:29:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:29:51.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:51 smithi038 ceph-mon[92422]: pgmap v16885: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 370 KiB/s wr, 8 op/s 2023-03-09T23:29:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:29:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:29:53.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:53 smithi122 ceph-mon[93869]: pgmap v16886: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 695 KiB/s rd, 652 KiB/s wr, 12 op/s 2023-03-09T23:29:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:53 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:29:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:53 smithi181 ceph-mon[94189]: pgmap v16886: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 695 KiB/s rd, 652 KiB/s wr, 12 op/s 2023-03-09T23:29:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:53 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:29:53.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:53 smithi038 ceph-mon[92422]: pgmap v16886: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 695 KiB/s rd, 652 KiB/s wr, 12 op/s 2023-03-09T23:29:53.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:53 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:29:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31165.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:29:54.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:54 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:29:54.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:54 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:29:54.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:54 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:29:54.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:54 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:29:54.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:54 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:29:54.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:54 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:29:54.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:54 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:29:54.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:54 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:29:54.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:54 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:29:55.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:55 smithi122 ceph-mon[93869]: pgmap v16887: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 730 KiB/s rd, 677 KiB/s wr, 10 op/s 2023-03-09T23:29:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30706.420987 secs 2023-03-09T23:29:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:55 smithi181 ceph-mon[94189]: pgmap v16887: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 730 KiB/s rd, 677 KiB/s wr, 10 op/s 2023-03-09T23:29:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30706.420987 secs 2023-03-09T23:29:55.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:55 smithi038 ceph-mon[92422]: pgmap v16887: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 730 KiB/s rd, 677 KiB/s wr, 10 op/s 2023-03-09T23:29:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30706.420987 secs 2023-03-09T23:29:56.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:29:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:29:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:29:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:29:56.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:29:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:29:57.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:29:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:29:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:57 smithi122 ceph-mon[93869]: pgmap v16888: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 14 op/s 2023-03-09T23:29:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:29:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:29:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:57 smithi181 ceph-mon[94189]: pgmap v16888: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 14 op/s 2023-03-09T23:29:57.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:29:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:29:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:57 smithi038 ceph-mon[92422]: pgmap v16888: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 14 op/s 2023-03-09T23:29:58.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31170.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:29:59.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:29:59 smithi122 ceph-mon[93869]: pgmap v16889: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 13 op/s 2023-03-09T23:29:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:29:59 smithi181 ceph-mon[94189]: pgmap v16889: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 13 op/s 2023-03-09T23:29:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:29:59 smithi038 ceph-mon[92422]: pgmap v16889: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 13 op/s 2023-03-09T23:30:00.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:30:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:30:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30711.421049 secs 2023-03-09T23:30:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:00 smithi122 ceph-mon[93869]: overall HEALTH_WARN 1 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-09T23:30:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:30:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:30:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30711.421049 secs 2023-03-09T23:30:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:00 smithi181 ceph-mon[94189]: overall HEALTH_WARN 1 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-09T23:30:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:30:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:30:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30711.421049 secs 2023-03-09T23:30:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:00 smithi038 ceph-mon[92422]: overall HEALTH_WARN 1 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-09T23:30:01.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:01 smithi122 ceph-mon[93869]: pgmap v16890: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 946 KiB/s rd, 788 KiB/s wr, 12 op/s 2023-03-09T23:30:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:30:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:30:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:01 smithi181 ceph-mon[94189]: pgmap v16890: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 946 KiB/s rd, 788 KiB/s wr, 12 op/s 2023-03-09T23:30:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:30:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:30:01.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:01 smithi038 ceph-mon[92422]: pgmap v16890: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 946 KiB/s rd, 788 KiB/s wr, 12 op/s 2023-03-09T23:30:01.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:30:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:30:03.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30713.827595 secs 2023-03-09T23:30:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:03 smithi122 ceph-mon[93869]: pgmap v16891: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 957 KiB/s wr, 17 op/s 2023-03-09T23:30:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30713.827595 secs 2023-03-09T23:30:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:03 smithi181 ceph-mon[94189]: pgmap v16891: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 957 KiB/s wr, 17 op/s 2023-03-09T23:30:03.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30713.827595 secs 2023-03-09T23:30:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:03 smithi038 ceph-mon[92422]: pgmap v16891: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 957 KiB/s wr, 17 op/s 2023-03-09T23:30:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31175.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:30:05.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:05 smithi122 ceph-mon[93869]: osdmap e1271: 12 total, 12 up, 12 in 2023-03-09T23:30:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:05 smithi122 ceph-mon[93869]: pgmap v16893: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 945 KiB/s rd, 861 KiB/s wr, 31 op/s 2023-03-09T23:30:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30716.421110 secs 2023-03-09T23:30:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:05 smithi181 ceph-mon[94189]: osdmap e1271: 12 total, 12 up, 12 in 2023-03-09T23:30:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:05 smithi181 ceph-mon[94189]: pgmap v16893: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 945 KiB/s rd, 861 KiB/s wr, 31 op/s 2023-03-09T23:30:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30716.421110 secs 2023-03-09T23:30:05.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:05 smithi038 ceph-mon[92422]: osdmap e1271: 12 total, 12 up, 12 in 2023-03-09T23:30:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:05 smithi038 ceph-mon[92422]: pgmap v16893: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 945 KiB/s rd, 861 KiB/s wr, 31 op/s 2023-03-09T23:30:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30716.421110 secs 2023-03-09T23:30:06.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:06 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:30:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:06 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:30:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:30:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:06 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:30:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:06 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:30:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:30:06.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:06 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:30:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:06 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:30:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:30:07.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:07 smithi122 ceph-mon[93869]: pgmap v16894: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 436 KiB/s rd, 455 KiB/s wr, 30 op/s 2023-03-09T23:30:07.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:07 smithi181 ceph-mon[94189]: pgmap v16894: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 436 KiB/s rd, 455 KiB/s wr, 30 op/s 2023-03-09T23:30:07.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:07 smithi038 ceph-mon[92422]: pgmap v16894: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 436 KiB/s rd, 455 KiB/s wr, 30 op/s 2023-03-09T23:30:08.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30718.827707 secs 2023-03-09T23:30:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30718.827707 secs 2023-03-09T23:30:08.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30718.827707 secs 2023-03-09T23:30:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31180.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:30:09.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:30:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:30:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:09 smithi122 ceph-mon[93869]: pgmap v16895: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 523 KiB/s rd, 564 KiB/s wr, 31 op/s 2023-03-09T23:30:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:30:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:30:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:09 smithi181 ceph-mon[94189]: pgmap v16895: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 523 KiB/s rd, 564 KiB/s wr, 31 op/s 2023-03-09T23:30:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:30:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:30:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:09 smithi038 ceph-mon[92422]: pgmap v16895: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 523 KiB/s rd, 564 KiB/s wr, 31 op/s 2023-03-09T23:30:10.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30721.421169 secs 2023-03-09T23:30:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30721.421169 secs 2023-03-09T23:30:10.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30721.421169 secs 2023-03-09T23:30:11.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:11 smithi122 ceph-mon[93869]: pgmap v16896: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 530 KiB/s wr, 31 op/s 2023-03-09T23:30:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:11 smithi181 ceph-mon[94189]: pgmap v16896: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 530 KiB/s wr, 31 op/s 2023-03-09T23:30:11.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:11 smithi038 ceph-mon[92422]: pgmap v16896: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 530 KiB/s wr, 31 op/s 2023-03-09T23:30:12.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:30:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:30:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:12 smithi122 ceph-mon[93869]: osdmap e1272: 12 total, 12 up, 12 in 2023-03-09T23:30:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:30:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:30:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:12 smithi181 ceph-mon[94189]: osdmap e1272: 12 total, 12 up, 12 in 2023-03-09T23:30:12.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:30:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:30:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:12 smithi038 ceph-mon[92422]: osdmap e1272: 12 total, 12 up, 12 in 2023-03-09T23:30:13.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30723.827828 secs 2023-03-09T23:30:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:13 smithi122 ceph-mon[93869]: pgmap v16898: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 465 KiB/s wr, 13 op/s 2023-03-09T23:30:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30723.827828 secs 2023-03-09T23:30:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:13 smithi181 ceph-mon[94189]: pgmap v16898: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 465 KiB/s wr, 13 op/s 2023-03-09T23:30:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30723.827828 secs 2023-03-09T23:30:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:13 smithi038 ceph-mon[92422]: pgmap v16898: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 465 KiB/s wr, 13 op/s 2023-03-09T23:30:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31185.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:30:15.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:15 smithi122 ceph-mon[93869]: pgmap v16899: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 374 KiB/s wr, 11 op/s 2023-03-09T23:30:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30726.421245 secs 2023-03-09T23:30:15.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:15 smithi181 ceph-mon[94189]: pgmap v16899: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 374 KiB/s wr, 11 op/s 2023-03-09T23:30:15.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30726.421245 secs 2023-03-09T23:30:15.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:15 smithi038 ceph-mon[92422]: pgmap v16899: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 374 KiB/s wr, 11 op/s 2023-03-09T23:30:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30726.421245 secs 2023-03-09T23:30:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:17 smithi181 ceph-mon[94189]: pgmap v16900: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 269 KiB/s wr, 9 op/s 2023-03-09T23:30:17.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:17 smithi038 ceph-mon[92422]: pgmap v16900: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 269 KiB/s wr, 9 op/s 2023-03-09T23:30:17.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:17 smithi122 ceph-mon[93869]: pgmap v16900: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 269 KiB/s wr, 9 op/s 2023-03-09T23:30:18.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30728.827959 secs 2023-03-09T23:30:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31190.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:30:18.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30728.827959 secs 2023-03-09T23:30:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30728.827959 secs 2023-03-09T23:30:19.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:19 smithi038 ceph-mon[92422]: pgmap v16901: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 159 KiB/s wr, 8 op/s 2023-03-09T23:30:19.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:19 smithi122 ceph-mon[93869]: pgmap v16901: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 159 KiB/s wr, 8 op/s 2023-03-09T23:30:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:19 smithi181 ceph-mon[94189]: pgmap v16901: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 159 KiB/s wr, 8 op/s 2023-03-09T23:30:20.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30731.421306 secs 2023-03-09T23:30:20.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30731.421306 secs 2023-03-09T23:30:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30731.421306 secs 2023-03-09T23:30:21.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:21 smithi038 ceph-mon[92422]: pgmap v16902: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 159 KiB/s wr, 7 op/s 2023-03-09T23:30:21.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:21 smithi122 ceph-mon[93869]: pgmap v16902: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 159 KiB/s wr, 7 op/s 2023-03-09T23:30:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:21 smithi181 ceph-mon[94189]: pgmap v16902: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 159 KiB/s wr, 7 op/s 2023-03-09T23:30:23.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30733.828101 secs 2023-03-09T23:30:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:23 smithi122 ceph-mon[93869]: pgmap v16903: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-09T23:30:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30733.828101 secs 2023-03-09T23:30:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:23 smithi181 ceph-mon[94189]: pgmap v16903: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-09T23:30:23.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30733.828101 secs 2023-03-09T23:30:23.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:23 smithi038 ceph-mon[92422]: pgmap v16903: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-09T23:30:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31195.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:30:25.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:25 smithi122 ceph-mon[93869]: pgmap v16904: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T23:30:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30736.421390 secs 2023-03-09T23:30:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:25 smithi181 ceph-mon[94189]: pgmap v16904: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T23:30:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30736.421390 secs 2023-03-09T23:30:25.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:25 smithi038 ceph-mon[92422]: pgmap v16904: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T23:30:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30736.421390 secs 2023-03-09T23:30:27.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:27 smithi122 ceph-mon[93869]: pgmap v16905: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-03-09T23:30:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:27 smithi181 ceph-mon[94189]: pgmap v16905: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-03-09T23:30:27.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:27 smithi038 ceph-mon[92422]: pgmap v16905: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-03-09T23:30:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31200.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:30:28.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30738.828178 secs 2023-03-09T23:30:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30738.828178 secs 2023-03-09T23:30:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30738.828178 secs 2023-03-09T23:30:29.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:29 smithi122 ceph-mon[93869]: pgmap v16906: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T23:30:29.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:29 smithi181 ceph-mon[94189]: pgmap v16906: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T23:30:29.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:29 smithi038 ceph-mon[92422]: pgmap v16906: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T23:30:30.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30741.421451 secs 2023-03-09T23:30:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30741.421451 secs 2023-03-09T23:30:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30741.421451 secs 2023-03-09T23:30:31.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:31 smithi122 ceph-mon[93869]: pgmap v16907: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T23:30:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:31 smithi181 ceph-mon[94189]: pgmap v16907: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T23:30:31.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:31 smithi038 ceph-mon[92422]: pgmap v16907: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T23:30:33.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30743.828277 secs 2023-03-09T23:30:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:33 smithi122 ceph-mon[93869]: pgmap v16908: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T23:30:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30743.828277 secs 2023-03-09T23:30:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:33 smithi181 ceph-mon[94189]: pgmap v16908: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T23:30:33.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30743.828277 secs 2023-03-09T23:30:33.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:33 smithi038 ceph-mon[92422]: pgmap v16908: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T23:30:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31205.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:30:35.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:35 smithi122 ceph-mon[93869]: pgmap v16909: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T23:30:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30746.421532 secs 2023-03-09T23:30:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:35 smithi181 ceph-mon[94189]: pgmap v16909: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T23:30:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30746.421532 secs 2023-03-09T23:30:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:35 smithi038 ceph-mon[92422]: pgmap v16909: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T23:30:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30746.421532 secs 2023-03-09T23:30:37.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:37 smithi122 ceph-mon[93869]: pgmap v16910: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T23:30:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:37 smithi181 ceph-mon[94189]: pgmap v16910: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T23:30:37.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:37 smithi038 ceph-mon[92422]: pgmap v16910: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T23:30:38.793 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31210.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:30:38.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30748.828388 secs 2023-03-09T23:30:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30748.828388 secs 2023-03-09T23:30:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30748.828388 secs 2023-03-09T23:30:39.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:39 smithi122 ceph-mon[93869]: pgmap v16911: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-09T23:30:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:39 smithi181 ceph-mon[94189]: pgmap v16911: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-09T23:30:39.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:39 smithi038 ceph-mon[92422]: pgmap v16911: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-09T23:30:40.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30751.421616 secs 2023-03-09T23:30:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30751.421616 secs 2023-03-09T23:30:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30751.421616 secs 2023-03-09T23:30:41.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:41 smithi122 ceph-mon[93869]: pgmap v16912: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T23:30:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:41 smithi181 ceph-mon[94189]: pgmap v16912: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T23:30:41.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:41 smithi038 ceph-mon[92422]: pgmap v16912: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T23:30:43.420 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:43 smithi122 ceph-mon[93869]: pgmap v16913: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 119 KiB/s wr, 8 op/s 2023-03-09T23:30:43.421 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:30:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:43 smithi181 ceph-mon[94189]: pgmap v16913: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 119 KiB/s wr, 8 op/s 2023-03-09T23:30:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:30:43.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:43 smithi038 ceph-mon[92422]: pgmap v16913: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 119 KiB/s wr, 8 op/s 2023-03-09T23:30:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:30:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31215.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:30:45.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:45 smithi122 ceph-mon[93869]: pgmap v16914: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 119 KiB/s wr, 6 op/s 2023-03-09T23:30:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30756.421676 secs 2023-03-09T23:30:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:45 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:30:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:45 smithi181 ceph-mon[94189]: pgmap v16914: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 119 KiB/s wr, 6 op/s 2023-03-09T23:30:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30756.421676 secs 2023-03-09T23:30:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:45 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:30:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:45 smithi038 ceph-mon[92422]: pgmap v16914: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 119 KiB/s wr, 6 op/s 2023-03-09T23:30:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30756.421676 secs 2023-03-09T23:30:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:45 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:30:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:46 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:30:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:46 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:30:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:46 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:30:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:30:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:30:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:30:46.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:30:46.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:30:46.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:30:46.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:30:46.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:30:46.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:30:46.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:30:46.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:30:46.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:30:46.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:30:46.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:30:46.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:30:46.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:30:46.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:30:46.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:30:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:46 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:30:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:46 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:30:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:46 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:30:46.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:30:46.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:30:46.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:30:46.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:30:46.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:30:46.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:30:46.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:30:46.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:30:46.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:30:46.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:30:46.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:30:46.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:30:46.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:30:46.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:30:46.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:30:46.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:30:46.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:30:46.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:30:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:46 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:30:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:46 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:30:47.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:46 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:30:47.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:30:47.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:30:47.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:30:47.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:30:47.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:30:47.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:30:47.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:30:47.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:30:47.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:30:47.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:30:47.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:30:47.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:30:47.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:30:47.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:30:47.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:30:47.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:30:47.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:30:47.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:30:47.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:47 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 190.8M 2023-03-09T23:30:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:47 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi181 to 200100225: error parsing value: Value '200100225' is below minimum 939524096 2023-03-09T23:30:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:47 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T23:30:47.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:47 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T23:30:47.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:47 smithi122 ceph-mon[93869]: pgmap v16915: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 326 KiB/s wr, 8 op/s 2023-03-09T23:30:47.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:30:47.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:30:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:47 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 190.8M 2023-03-09T23:30:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:47 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi181 to 200100225: error parsing value: Value '200100225' is below minimum 939524096 2023-03-09T23:30:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:47 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T23:30:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:47 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T23:30:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:47 smithi181 ceph-mon[94189]: pgmap v16915: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 326 KiB/s wr, 8 op/s 2023-03-09T23:30:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:30:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:30:47.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:47 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 190.8M 2023-03-09T23:30:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:47 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi181 to 200100225: error parsing value: Value '200100225' is below minimum 939524096 2023-03-09T23:30:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:47 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T23:30:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:47 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T23:30:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:47 smithi038 ceph-mon[92422]: pgmap v16915: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 326 KiB/s wr, 8 op/s 2023-03-09T23:30:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:30:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:30:48.793 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31220.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:30:49.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:49 smithi122 ceph-mon[93869]: pgmap v16916: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 326 KiB/s wr, 7 op/s 2023-03-09T23:30:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:30:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:30:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:49 smithi181 ceph-mon[94189]: pgmap v16916: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 326 KiB/s wr, 7 op/s 2023-03-09T23:30:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:30:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:30:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:49 smithi038 ceph-mon[92422]: pgmap v16916: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 326 KiB/s wr, 7 op/s 2023-03-09T23:30:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:30:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:30:50.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30761.421738 secs 2023-03-09T23:30:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30761.421738 secs 2023-03-09T23:30:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30761.421738 secs 2023-03-09T23:30:51.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:51 smithi122 ceph-mon[93869]: pgmap v16917: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 588 KiB/s rd, 351 KiB/s wr, 8 op/s 2023-03-09T23:30:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:51 smithi181 ceph-mon[94189]: pgmap v16917: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 588 KiB/s rd, 351 KiB/s wr, 8 op/s 2023-03-09T23:30:51.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:51 smithi038 ceph-mon[92422]: pgmap v16917: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 588 KiB/s rd, 351 KiB/s wr, 8 op/s 2023-03-09T23:30:53.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:53 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 30763.828772 secs 2023-03-09T23:30:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:53 smithi122 ceph-mon[93869]: pgmap v16918: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 875 KiB/s rd, 634 KiB/s wr, 11 op/s 2023-03-09T23:30:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:53 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 30763.828772 secs 2023-03-09T23:30:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:53 smithi181 ceph-mon[94189]: pgmap v16918: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 875 KiB/s rd, 634 KiB/s wr, 11 op/s 2023-03-09T23:30:53.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:53 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 30763.828772 secs 2023-03-09T23:30:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:53 smithi038 ceph-mon[92422]: pgmap v16918: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 875 KiB/s rd, 634 KiB/s wr, 11 op/s 2023-03-09T23:30:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31225.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:30:54.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:54 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:30:54.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:54 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:30:54.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:54 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:30:54.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:54 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:30:54.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:54 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:30:54.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:54 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:30:54.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:54 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:30:54.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:54 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:30:54.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:54 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:30:55.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:55 smithi122 ceph-mon[93869]: pgmap v16919: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 674 KiB/s rd, 541 KiB/s wr, 10 op/s 2023-03-09T23:30:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30766.421811 secs 2023-03-09T23:30:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:55 smithi181 ceph-mon[94189]: pgmap v16919: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 674 KiB/s rd, 541 KiB/s wr, 10 op/s 2023-03-09T23:30:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30766.421811 secs 2023-03-09T23:30:55.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:55 smithi038 ceph-mon[92422]: pgmap v16919: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 674 KiB/s rd, 541 KiB/s wr, 10 op/s 2023-03-09T23:30:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30766.421811 secs 2023-03-09T23:30:56.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:30:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:30:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:30:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:30:56.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:30:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:30:57.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:30:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:30:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:57 smithi122 ceph-mon[93869]: pgmap v16920: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 967 KiB/s wr, 14 op/s 2023-03-09T23:30:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:30:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:30:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:57 smithi181 ceph-mon[94189]: pgmap v16920: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 967 KiB/s wr, 14 op/s 2023-03-09T23:30:57.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:30:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:30:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:57 smithi038 ceph-mon[92422]: pgmap v16920: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 967 KiB/s wr, 14 op/s 2023-03-09T23:30:58.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:58 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 30768.828886 secs 2023-03-09T23:30:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:58 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 30768.828886 secs 2023-03-09T23:30:58.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:58 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 30768.828886 secs 2023-03-09T23:30:58.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31230.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:30:59.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:30:59 smithi122 ceph-mon[93869]: pgmap v16921: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 892 KiB/s rd, 761 KiB/s wr, 12 op/s 2023-03-09T23:30:59.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:30:59 smithi181 ceph-mon[94189]: pgmap v16921: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 892 KiB/s rd, 761 KiB/s wr, 12 op/s 2023-03-09T23:30:59.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:30:59 smithi038 ceph-mon[92422]: pgmap v16921: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 892 KiB/s rd, 761 KiB/s wr, 12 op/s 2023-03-09T23:31:00.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:31:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:31:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30771.421869 secs 2023-03-09T23:31:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:31:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:31:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30771.421869 secs 2023-03-09T23:31:00.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:31:00.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:31:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30771.421869 secs 2023-03-09T23:31:01.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:01 smithi122 ceph-mon[93869]: pgmap v16922: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 864 KiB/s rd, 851 KiB/s wr, 12 op/s 2023-03-09T23:31:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:01 smithi181 ceph-mon[94189]: pgmap v16922: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 864 KiB/s rd, 851 KiB/s wr, 12 op/s 2023-03-09T23:31:01.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:01 smithi038 ceph-mon[92422]: pgmap v16922: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 864 KiB/s rd, 851 KiB/s wr, 12 op/s 2023-03-09T23:31:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30773.829006 secs 2023-03-09T23:31:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:03 smithi122 ceph-mon[93869]: pgmap v16923: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 950 KiB/s rd, 957 KiB/s wr, 18 op/s 2023-03-09T23:31:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30773.829006 secs 2023-03-09T23:31:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:03 smithi181 ceph-mon[94189]: pgmap v16923: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 950 KiB/s rd, 957 KiB/s wr, 18 op/s 2023-03-09T23:31:03.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30773.829006 secs 2023-03-09T23:31:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:03 smithi038 ceph-mon[92422]: pgmap v16923: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 950 KiB/s rd, 957 KiB/s wr, 18 op/s 2023-03-09T23:31:03.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31235.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:31:05.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:05 smithi122 ceph-mon[93869]: osdmap e1273: 12 total, 12 up, 12 in 2023-03-09T23:31:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:05 smithi122 ceph-mon[93869]: pgmap v16925: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 817 KiB/s rd, 860 KiB/s wr, 31 op/s 2023-03-09T23:31:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30776.421930 secs 2023-03-09T23:31:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:05 smithi181 ceph-mon[94189]: osdmap e1273: 12 total, 12 up, 12 in 2023-03-09T23:31:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:05 smithi181 ceph-mon[94189]: pgmap v16925: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 817 KiB/s rd, 860 KiB/s wr, 31 op/s 2023-03-09T23:31:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30776.421930 secs 2023-03-09T23:31:05.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:05 smithi038 ceph-mon[92422]: osdmap e1273: 12 total, 12 up, 12 in 2023-03-09T23:31:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:05 smithi038 ceph-mon[92422]: pgmap v16925: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 817 KiB/s rd, 860 KiB/s wr, 31 op/s 2023-03-09T23:31:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30776.421930 secs 2023-03-09T23:31:06.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:06 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:31:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:31:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:06 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:31:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:31:06.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:06 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:31:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:31:07.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:07 smithi122 ceph-mon[93869]: pgmap v16926: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 522 KiB/s rd, 564 KiB/s wr, 30 op/s 2023-03-09T23:31:07.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:07 smithi181 ceph-mon[94189]: pgmap v16926: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 522 KiB/s rd, 564 KiB/s wr, 30 op/s 2023-03-09T23:31:07.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:07 smithi038 ceph-mon[92422]: pgmap v16926: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 522 KiB/s rd, 564 KiB/s wr, 30 op/s 2023-03-09T23:31:08.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30778.829138 secs 2023-03-09T23:31:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:31:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:31:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30778.829138 secs 2023-03-09T23:31:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:31:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:31:08.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30778.829138 secs 2023-03-09T23:31:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:31:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:31:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31240.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:31:09.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:31:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:31:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:09 smithi122 ceph-mon[93869]: pgmap v16927: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 522 KiB/s rd, 563 KiB/s wr, 29 op/s 2023-03-09T23:31:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:31:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:31:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:09 smithi181 ceph-mon[94189]: pgmap v16927: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 522 KiB/s rd, 563 KiB/s wr, 29 op/s 2023-03-09T23:31:09.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:31:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:31:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:09 smithi038 ceph-mon[92422]: pgmap v16927: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 522 KiB/s rd, 563 KiB/s wr, 29 op/s 2023-03-09T23:31:10.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30781.421981 secs 2023-03-09T23:31:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30781.421981 secs 2023-03-09T23:31:10.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30781.421981 secs 2023-03-09T23:31:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:11 smithi181 ceph-mon[94189]: pgmap v16928: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 574 KiB/s wr, 29 op/s 2023-03-09T23:31:11.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:11 smithi038 ceph-mon[92422]: pgmap v16928: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 574 KiB/s wr, 29 op/s 2023-03-09T23:31:11.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:11 smithi122 ceph-mon[93869]: pgmap v16928: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 574 KiB/s wr, 29 op/s 2023-03-09T23:31:13.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30783.829212 secs 2023-03-09T23:31:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:13 smithi122 ceph-mon[93869]: osdmap e1274: 12 total, 12 up, 12 in 2023-03-09T23:31:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:13 smithi122 ceph-mon[93869]: pgmap v16930: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 470 KiB/s rd, 467 KiB/s wr, 11 op/s 2023-03-09T23:31:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30783.829212 secs 2023-03-09T23:31:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:13 smithi181 ceph-mon[94189]: osdmap e1274: 12 total, 12 up, 12 in 2023-03-09T23:31:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:13 smithi181 ceph-mon[94189]: pgmap v16930: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 470 KiB/s rd, 467 KiB/s wr, 11 op/s 2023-03-09T23:31:13.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30783.829212 secs 2023-03-09T23:31:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:13 smithi038 ceph-mon[92422]: osdmap e1274: 12 total, 12 up, 12 in 2023-03-09T23:31:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:13 smithi038 ceph-mon[92422]: pgmap v16930: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 470 KiB/s rd, 467 KiB/s wr, 11 op/s 2023-03-09T23:31:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31245.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:31:15.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:15 smithi122 ceph-mon[93869]: pgmap v16931: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 376 KiB/s wr, 10 op/s 2023-03-09T23:31:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30786.422053 secs 2023-03-09T23:31:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:15 smithi181 ceph-mon[94189]: pgmap v16931: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 376 KiB/s wr, 10 op/s 2023-03-09T23:31:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30786.422053 secs 2023-03-09T23:31:15.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:15 smithi038 ceph-mon[92422]: pgmap v16931: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 376 KiB/s wr, 10 op/s 2023-03-09T23:31:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30786.422053 secs 2023-03-09T23:31:17.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:17 smithi122 ceph-mon[93869]: pgmap v16932: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 160 KiB/s wr, 8 op/s 2023-03-09T23:31:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:17 smithi181 ceph-mon[94189]: pgmap v16932: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 160 KiB/s wr, 8 op/s 2023-03-09T23:31:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:17 smithi038 ceph-mon[92422]: pgmap v16932: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 160 KiB/s wr, 8 op/s 2023-03-09T23:31:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31250.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:31:18.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30788.829345 secs 2023-03-09T23:31:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30788.829345 secs 2023-03-09T23:31:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30788.829345 secs 2023-03-09T23:31:19.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:19 smithi122 ceph-mon[93869]: pgmap v16933: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 160 KiB/s wr, 8 op/s 2023-03-09T23:31:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:19 smithi181 ceph-mon[94189]: pgmap v16933: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 160 KiB/s wr, 8 op/s 2023-03-09T23:31:19.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:19 smithi038 ceph-mon[92422]: pgmap v16933: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 160 KiB/s wr, 8 op/s 2023-03-09T23:31:20.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30791.422112 secs 2023-03-09T23:31:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30791.422112 secs 2023-03-09T23:31:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30791.422112 secs 2023-03-09T23:31:21.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:21 smithi122 ceph-mon[93869]: pgmap v16934: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 40 KiB/s wr, 8 op/s 2023-03-09T23:31:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:21 smithi181 ceph-mon[94189]: pgmap v16934: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 40 KiB/s wr, 8 op/s 2023-03-09T23:31:21.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:21 smithi038 ceph-mon[92422]: pgmap v16934: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 40 KiB/s wr, 8 op/s 2023-03-09T23:31:23.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30793.829485 secs 2023-03-09T23:31:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:23 smithi122 ceph-mon[93869]: pgmap v16935: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2023-03-09T23:31:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30793.829485 secs 2023-03-09T23:31:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:23 smithi181 ceph-mon[94189]: pgmap v16935: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2023-03-09T23:31:23.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30793.829485 secs 2023-03-09T23:31:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:23 smithi038 ceph-mon[92422]: pgmap v16935: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2023-03-09T23:31:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31255.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:31:25.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:25 smithi122 ceph-mon[93869]: pgmap v16936: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 6 op/s 2023-03-09T23:31:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30796.422172 secs 2023-03-09T23:31:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:25 smithi181 ceph-mon[94189]: pgmap v16936: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 6 op/s 2023-03-09T23:31:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30796.422172 secs 2023-03-09T23:31:25.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:25 smithi038 ceph-mon[92422]: pgmap v16936: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 6 op/s 2023-03-09T23:31:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30796.422172 secs 2023-03-09T23:31:27.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:27 smithi122 ceph-mon[93869]: pgmap v16937: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T23:31:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:27 smithi181 ceph-mon[94189]: pgmap v16937: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T23:31:27.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:27 smithi038 ceph-mon[92422]: pgmap v16937: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T23:31:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31260.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:31:28.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30798.829605 secs 2023-03-09T23:31:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30798.829605 secs 2023-03-09T23:31:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30798.829605 secs 2023-03-09T23:31:29.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:29 smithi122 ceph-mon[93869]: pgmap v16938: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 6 op/s 2023-03-09T23:31:29.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:29 smithi181 ceph-mon[94189]: pgmap v16938: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 6 op/s 2023-03-09T23:31:29.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:29 smithi038 ceph-mon[92422]: pgmap v16938: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 6 op/s 2023-03-09T23:31:30.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30801.422238 secs 2023-03-09T23:31:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30801.422238 secs 2023-03-09T23:31:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30801.422238 secs 2023-03-09T23:31:31.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:31 smithi122 ceph-mon[93869]: pgmap v16939: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 7 op/s 2023-03-09T23:31:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:31 smithi181 ceph-mon[94189]: pgmap v16939: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 7 op/s 2023-03-09T23:31:31.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:31 smithi038 ceph-mon[92422]: pgmap v16939: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 7 op/s 2023-03-09T23:31:33.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30803.829725 secs 2023-03-09T23:31:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:33 smithi122 ceph-mon[93869]: pgmap v16940: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T23:31:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30803.829725 secs 2023-03-09T23:31:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:33 smithi181 ceph-mon[94189]: pgmap v16940: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T23:31:33.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30803.829725 secs 2023-03-09T23:31:33.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:33 smithi038 ceph-mon[92422]: pgmap v16940: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T23:31:33.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31265.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:31:35.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:35 smithi122 ceph-mon[93869]: pgmap v16941: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T23:31:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30806.422298 secs 2023-03-09T23:31:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:35 smithi181 ceph-mon[94189]: pgmap v16941: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T23:31:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30806.422298 secs 2023-03-09T23:31:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:35 smithi038 ceph-mon[92422]: pgmap v16941: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T23:31:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30806.422298 secs 2023-03-09T23:31:37.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:37 smithi122 ceph-mon[93869]: pgmap v16942: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 9 op/s 2023-03-09T23:31:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:37 smithi181 ceph-mon[94189]: pgmap v16942: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 9 op/s 2023-03-09T23:31:37.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:37 smithi038 ceph-mon[92422]: pgmap v16942: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 9 op/s 2023-03-09T23:31:38.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31270.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:31:38.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30808.829843 secs 2023-03-09T23:31:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30808.829843 secs 2023-03-09T23:31:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30808.829843 secs 2023-03-09T23:31:39.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:39 smithi122 ceph-mon[93869]: pgmap v16943: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 7 op/s 2023-03-09T23:31:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:39 smithi181 ceph-mon[94189]: pgmap v16943: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 7 op/s 2023-03-09T23:31:39.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:39 smithi038 ceph-mon[92422]: pgmap v16943: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 7 op/s 2023-03-09T23:31:40.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30811.422356 secs 2023-03-09T23:31:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30811.422356 secs 2023-03-09T23:31:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30811.422356 secs 2023-03-09T23:31:41.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:41 smithi122 ceph-mon[93869]: pgmap v16944: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 511 B/s wr, 8 op/s 2023-03-09T23:31:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:41 smithi181 ceph-mon[94189]: pgmap v16944: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 511 B/s wr, 8 op/s 2023-03-09T23:31:41.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:41 smithi038 ceph-mon[92422]: pgmap v16944: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 511 B/s wr, 8 op/s 2023-03-09T23:31:43.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30813.829937 secs 2023-03-09T23:31:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:43 smithi122 ceph-mon[93869]: pgmap v16945: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 10 op/s 2023-03-09T23:31:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30813.829937 secs 2023-03-09T23:31:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:43 smithi181 ceph-mon[94189]: pgmap v16945: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 10 op/s 2023-03-09T23:31:43.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30813.829937 secs 2023-03-09T23:31:43.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:43 smithi038 ceph-mon[92422]: pgmap v16945: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 10 op/s 2023-03-09T23:31:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31275.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:31:45.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:45 smithi122 ceph-mon[93869]: pgmap v16946: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 8 op/s 2023-03-09T23:31:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30816.422415 secs 2023-03-09T23:31:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:45 smithi181 ceph-mon[94189]: pgmap v16946: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 8 op/s 2023-03-09T23:31:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30816.422415 secs 2023-03-09T23:31:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:45 smithi038 ceph-mon[92422]: pgmap v16946: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 8 op/s 2023-03-09T23:31:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30816.422415 secs 2023-03-09T23:31:46.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:31:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:31:46.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:31:47.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:47 smithi122 ceph-mon[93869]: pgmap v16947: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 853 B/s wr, 9 op/s 2023-03-09T23:31:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:31:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:31:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:31:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:31:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:31:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:47 smithi181 ceph-mon[94189]: pgmap v16947: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 853 B/s wr, 9 op/s 2023-03-09T23:31:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:31:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:31:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:31:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:31:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:31:47.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:47 smithi038 ceph-mon[92422]: pgmap v16947: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 853 B/s wr, 9 op/s 2023-03-09T23:31:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:31:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:31:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:31:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:31:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:31:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31280.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:31:48.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30818.830055 secs 2023-03-09T23:31:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30818.830055 secs 2023-03-09T23:31:48.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30818.830055 secs 2023-03-09T23:31:49.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:49 smithi122 ceph-mon[93869]: pgmap v16948: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 309 KiB/s wr, 8 op/s 2023-03-09T23:31:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:31:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:31:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:49 smithi181 ceph-mon[94189]: pgmap v16948: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 309 KiB/s wr, 8 op/s 2023-03-09T23:31:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:31:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:31:49.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:49 smithi038 ceph-mon[92422]: pgmap v16948: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 309 KiB/s wr, 8 op/s 2023-03-09T23:31:49.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:31:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:31:50.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30821.422499 secs 2023-03-09T23:31:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30821.422499 secs 2023-03-09T23:31:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30821.422499 secs 2023-03-09T23:31:51.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:51 smithi122 ceph-mon[93869]: pgmap v16949: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 352 KiB/s wr, 9 op/s 2023-03-09T23:31:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:31:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:31:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:51 smithi181 ceph-mon[94189]: pgmap v16949: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 352 KiB/s wr, 9 op/s 2023-03-09T23:31:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:31:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:31:51.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:51 smithi038 ceph-mon[92422]: pgmap v16949: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 352 KiB/s wr, 9 op/s 2023-03-09T23:31:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:31:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:31:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31285.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:31:53.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30823.830173 secs 2023-03-09T23:31:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:53 smithi122 ceph-mon[93869]: pgmap v16950: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 352 KiB/s wr, 11 op/s 2023-03-09T23:31:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:31:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:31:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30823.830173 secs 2023-03-09T23:31:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:53 smithi181 ceph-mon[94189]: pgmap v16950: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 352 KiB/s wr, 11 op/s 2023-03-09T23:31:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:31:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:31:53.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30823.830173 secs 2023-03-09T23:31:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:53 smithi038 ceph-mon[92422]: pgmap v16950: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 352 KiB/s wr, 11 op/s 2023-03-09T23:31:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:31:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:31:55.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:55 smithi122 ceph-mon[93869]: pgmap v16951: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 715 KiB/s rd, 877 KiB/s wr, 10 op/s 2023-03-09T23:31:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:31:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:31:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30826.422560 secs 2023-03-09T23:31:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:55 smithi181 ceph-mon[94189]: pgmap v16951: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 715 KiB/s rd, 877 KiB/s wr, 10 op/s 2023-03-09T23:31:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:31:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:31:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30826.422560 secs 2023-03-09T23:31:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:55 smithi038 ceph-mon[92422]: pgmap v16951: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 715 KiB/s rd, 877 KiB/s wr, 10 op/s 2023-03-09T23:31:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:31:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:31:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30826.422560 secs 2023-03-09T23:31:57.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:57 smithi122 ceph-mon[93869]: pgmap v16952: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 761 KiB/s rd, 924 KiB/s wr, 13 op/s 2023-03-09T23:31:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:31:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:31:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:57 smithi181 ceph-mon[94189]: pgmap v16952: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 761 KiB/s rd, 924 KiB/s wr, 13 op/s 2023-03-09T23:31:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:31:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:31:57.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:57 smithi038 ceph-mon[92422]: pgmap v16952: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 761 KiB/s rd, 924 KiB/s wr, 13 op/s 2023-03-09T23:31:57.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:31:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:31:58.793 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31290.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:31:58.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30828.830264 secs 2023-03-09T23:31:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30828.830264 secs 2023-03-09T23:31:58.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30828.830264 secs 2023-03-09T23:31:59.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:59 smithi122 ceph-mon[93869]: pgmap v16953: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 13 op/s 2023-03-09T23:31:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:31:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:31:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:31:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:59 smithi181 ceph-mon[94189]: pgmap v16953: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 13 op/s 2023-03-09T23:31:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:31:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:31:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:31:59.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:59 smithi038 ceph-mon[92422]: pgmap v16953: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 13 op/s 2023-03-09T23:31:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:31:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:31:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:32:00.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30831.422620 secs 2023-03-09T23:32:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30831.422620 secs 2023-03-09T23:32:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30831.422620 secs 2023-03-09T23:32:01.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:01 smithi122 ceph-mon[93869]: pgmap v16954: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 992 KiB/s rd, 979 KiB/s wr, 14 op/s 2023-03-09T23:32:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:32:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:32:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:01 smithi181 ceph-mon[94189]: pgmap v16954: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 992 KiB/s rd, 979 KiB/s wr, 14 op/s 2023-03-09T23:32:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:32:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:32:01.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:01 smithi038 ceph-mon[92422]: pgmap v16954: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 992 KiB/s rd, 979 KiB/s wr, 14 op/s 2023-03-09T23:32:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:32:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:32:03.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30833.830399 secs 2023-03-09T23:32:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:03 smithi122 ceph-mon[93869]: pgmap v16955: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 951 KiB/s rd, 937 KiB/s wr, 15 op/s 2023-03-09T23:32:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30833.830399 secs 2023-03-09T23:32:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:03 smithi181 ceph-mon[94189]: pgmap v16955: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 951 KiB/s rd, 937 KiB/s wr, 15 op/s 2023-03-09T23:32:03.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30833.830399 secs 2023-03-09T23:32:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:03 smithi038 ceph-mon[92422]: pgmap v16955: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 951 KiB/s rd, 937 KiB/s wr, 15 op/s 2023-03-09T23:32:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31295.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:32:05.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:05 smithi122 ceph-mon[93869]: osdmap e1275: 12 total, 12 up, 12 in 2023-03-09T23:32:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:05 smithi122 ceph-mon[93869]: pgmap v16957: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 770 KiB/s rd, 575 KiB/s wr, 27 op/s 2023-03-09T23:32:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30836.422686 secs 2023-03-09T23:32:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:05 smithi181 ceph-mon[94189]: osdmap e1275: 12 total, 12 up, 12 in 2023-03-09T23:32:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:05 smithi181 ceph-mon[94189]: pgmap v16957: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 770 KiB/s rd, 575 KiB/s wr, 27 op/s 2023-03-09T23:32:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30836.422686 secs 2023-03-09T23:32:05.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:05 smithi038 ceph-mon[92422]: osdmap e1275: 12 total, 12 up, 12 in 2023-03-09T23:32:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:05 smithi038 ceph-mon[92422]: pgmap v16957: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 770 KiB/s rd, 575 KiB/s wr, 27 op/s 2023-03-09T23:32:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30836.422686 secs 2023-03-09T23:32:07.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:07 smithi122 ceph-mon[93869]: pgmap v16958: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 933 KiB/s rd, 762 KiB/s wr, 31 op/s 2023-03-09T23:32:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:32:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:32:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:07 smithi181 ceph-mon[94189]: pgmap v16958: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 933 KiB/s rd, 762 KiB/s wr, 31 op/s 2023-03-09T23:32:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:32:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:32:07.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:07 smithi038 ceph-mon[92422]: pgmap v16958: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 933 KiB/s rd, 762 KiB/s wr, 31 op/s 2023-03-09T23:32:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:32:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:32:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31300.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:32:08.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30838.830512 secs 2023-03-09T23:32:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30838.830512 secs 2023-03-09T23:32:08.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30838.830512 secs 2023-03-09T23:32:09.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:09 smithi122 ceph-mon[93869]: pgmap v16959: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 524 KiB/s rd, 565 KiB/s wr, 29 op/s 2023-03-09T23:32:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:32:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:32:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:09 smithi181 ceph-mon[94189]: pgmap v16959: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 524 KiB/s rd, 565 KiB/s wr, 29 op/s 2023-03-09T23:32:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:32:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:32:09.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:09 smithi038 ceph-mon[92422]: pgmap v16959: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 524 KiB/s rd, 565 KiB/s wr, 29 op/s 2023-03-09T23:32:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:32:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:32:10.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30841.422744 secs 2023-03-09T23:32:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30841.422744 secs 2023-03-09T23:32:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30841.422744 secs 2023-03-09T23:32:11.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:11 smithi122 ceph-mon[93869]: pgmap v16960: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 326 KiB/s wr, 27 op/s 2023-03-09T23:32:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:11 smithi181 ceph-mon[94189]: pgmap v16960: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 326 KiB/s wr, 27 op/s 2023-03-09T23:32:11.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:11 smithi038 ceph-mon[92422]: pgmap v16960: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 326 KiB/s wr, 27 op/s 2023-03-09T23:32:13.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30843.830634 secs 2023-03-09T23:32:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:13 smithi122 ceph-mon[93869]: osdmap e1276: 12 total, 12 up, 12 in 2023-03-09T23:32:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:13 smithi122 ceph-mon[93869]: pgmap v16962: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 472 KiB/s rd, 501 KiB/s wr, 14 op/s 2023-03-09T23:32:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30843.830634 secs 2023-03-09T23:32:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:13 smithi181 ceph-mon[94189]: osdmap e1276: 12 total, 12 up, 12 in 2023-03-09T23:32:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:13 smithi181 ceph-mon[94189]: pgmap v16962: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 472 KiB/s rd, 501 KiB/s wr, 14 op/s 2023-03-09T23:32:13.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30843.830634 secs 2023-03-09T23:32:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:13 smithi038 ceph-mon[92422]: osdmap e1276: 12 total, 12 up, 12 in 2023-03-09T23:32:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:13 smithi038 ceph-mon[92422]: pgmap v16962: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 472 KiB/s rd, 501 KiB/s wr, 14 op/s 2023-03-09T23:32:13.793 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31305.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:32:15.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:15 smithi122 ceph-mon[93869]: pgmap v16963: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 403 KiB/s wr, 12 op/s 2023-03-09T23:32:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30846.422803 secs 2023-03-09T23:32:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:15 smithi181 ceph-mon[94189]: pgmap v16963: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 403 KiB/s wr, 12 op/s 2023-03-09T23:32:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30846.422803 secs 2023-03-09T23:32:15.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:15 smithi038 ceph-mon[92422]: pgmap v16963: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 403 KiB/s wr, 12 op/s 2023-03-09T23:32:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30846.422803 secs 2023-03-09T23:32:17.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:17 smithi122 ceph-mon[93869]: pgmap v16964: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 160 KiB/s wr, 8 op/s 2023-03-09T23:32:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:17 smithi181 ceph-mon[94189]: pgmap v16964: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 160 KiB/s wr, 8 op/s 2023-03-09T23:32:17.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:17 smithi038 ceph-mon[92422]: pgmap v16964: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 160 KiB/s wr, 8 op/s 2023-03-09T23:32:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31310.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:32:18.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30848.830746 secs 2023-03-09T23:32:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30848.830746 secs 2023-03-09T23:32:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30848.830746 secs 2023-03-09T23:32:19.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:19 smithi122 ceph-mon[93869]: pgmap v16965: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 160 KiB/s wr, 7 op/s 2023-03-09T23:32:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:19 smithi181 ceph-mon[94189]: pgmap v16965: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 160 KiB/s wr, 7 op/s 2023-03-09T23:32:19.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:19 smithi038 ceph-mon[92422]: pgmap v16965: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 160 KiB/s wr, 7 op/s 2023-03-09T23:32:20.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30851.422865 secs 2023-03-09T23:32:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30851.422865 secs 2023-03-09T23:32:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30851.422865 secs 2023-03-09T23:32:21.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:21 smithi122 ceph-mon[93869]: pgmap v16966: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 160 KiB/s wr, 7 op/s 2023-03-09T23:32:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:21 smithi181 ceph-mon[94189]: pgmap v16966: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 160 KiB/s wr, 7 op/s 2023-03-09T23:32:21.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:21 smithi038 ceph-mon[92422]: pgmap v16966: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 160 KiB/s wr, 7 op/s 2023-03-09T23:32:23.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30853.830858 secs 2023-03-09T23:32:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:23 smithi122 ceph-mon[93869]: pgmap v16967: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-03-09T23:32:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30853.830858 secs 2023-03-09T23:32:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:23 smithi181 ceph-mon[94189]: pgmap v16967: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-03-09T23:32:23.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30853.830858 secs 2023-03-09T23:32:23.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:23 smithi038 ceph-mon[92422]: pgmap v16967: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-03-09T23:32:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31315.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:32:25.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:25 smithi122 ceph-mon[93869]: pgmap v16968: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T23:32:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30856.422927 secs 2023-03-09T23:32:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:25 smithi181 ceph-mon[94189]: pgmap v16968: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T23:32:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30856.422927 secs 2023-03-09T23:32:25.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:25 smithi038 ceph-mon[92422]: pgmap v16968: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T23:32:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30856.422927 secs 2023-03-09T23:32:27.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:27 smithi122 ceph-mon[93869]: pgmap v16969: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T23:32:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:27 smithi181 ceph-mon[94189]: pgmap v16969: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T23:32:27.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:27 smithi038 ceph-mon[92422]: pgmap v16969: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T23:32:28.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31320.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:32:28.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30858.830974 secs 2023-03-09T23:32:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30858.830974 secs 2023-03-09T23:32:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30858.830974 secs 2023-03-09T23:32:29.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:29 smithi122 ceph-mon[93869]: pgmap v16970: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T23:32:29.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:29 smithi181 ceph-mon[94189]: pgmap v16970: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T23:32:29.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:29 smithi038 ceph-mon[92422]: pgmap v16970: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T23:32:30.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30861.422987 secs 2023-03-09T23:32:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30861.422987 secs 2023-03-09T23:32:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30861.422987 secs 2023-03-09T23:32:31.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:31 smithi122 ceph-mon[93869]: pgmap v16971: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T23:32:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:31 smithi181 ceph-mon[94189]: pgmap v16971: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T23:32:31.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:31 smithi038 ceph-mon[92422]: pgmap v16971: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T23:32:33.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30863.831067 secs 2023-03-09T23:32:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:33 smithi122 ceph-mon[93869]: pgmap v16972: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T23:32:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30863.831067 secs 2023-03-09T23:32:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:33 smithi181 ceph-mon[94189]: pgmap v16972: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T23:32:33.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30863.831067 secs 2023-03-09T23:32:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:33 smithi038 ceph-mon[92422]: pgmap v16972: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T23:32:33.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31325.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:32:35.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:35 smithi122 ceph-mon[93869]: pgmap v16973: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T23:32:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30866.423046 secs 2023-03-09T23:32:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:35 smithi181 ceph-mon[94189]: pgmap v16973: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T23:32:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30866.423046 secs 2023-03-09T23:32:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:35 smithi038 ceph-mon[92422]: pgmap v16973: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T23:32:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30866.423046 secs 2023-03-09T23:32:37.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:37 smithi122 ceph-mon[93869]: pgmap v16974: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 8 op/s 2023-03-09T23:32:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:37 smithi181 ceph-mon[94189]: pgmap v16974: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 8 op/s 2023-03-09T23:32:37.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:37 smithi038 ceph-mon[92422]: pgmap v16974: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 8 op/s 2023-03-09T23:32:38.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31330.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:32:38.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30868.831147 secs 2023-03-09T23:32:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30868.831147 secs 2023-03-09T23:32:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30868.831147 secs 2023-03-09T23:32:39.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:39 smithi122 ceph-mon[93869]: pgmap v16975: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-09T23:32:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:39 smithi181 ceph-mon[94189]: pgmap v16975: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-09T23:32:39.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:39 smithi038 ceph-mon[92422]: pgmap v16975: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-09T23:32:40.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30871.423107 secs 2023-03-09T23:32:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30871.423107 secs 2023-03-09T23:32:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30871.423107 secs 2023-03-09T23:32:41.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:41 smithi122 ceph-mon[93869]: pgmap v16976: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 511 B/s wr, 7 op/s 2023-03-09T23:32:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:41 smithi181 ceph-mon[94189]: pgmap v16976: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 511 B/s wr, 7 op/s 2023-03-09T23:32:41.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:41 smithi038 ceph-mon[92422]: pgmap v16976: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 511 B/s wr, 7 op/s 2023-03-09T23:32:43.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30873.831267 secs 2023-03-09T23:32:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:43 smithi122 ceph-mon[93869]: pgmap v16977: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 8 op/s 2023-03-09T23:32:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30873.831267 secs 2023-03-09T23:32:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:43 smithi181 ceph-mon[94189]: pgmap v16977: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 8 op/s 2023-03-09T23:32:43.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30873.831267 secs 2023-03-09T23:32:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:43 smithi038 ceph-mon[92422]: pgmap v16977: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 8 op/s 2023-03-09T23:32:43.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31335.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:32:45.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:45 smithi122 ceph-mon[93869]: pgmap v16978: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 134 KiB/s wr, 7 op/s 2023-03-09T23:32:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30876.423167 secs 2023-03-09T23:32:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:45 smithi181 ceph-mon[94189]: pgmap v16978: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 134 KiB/s wr, 7 op/s 2023-03-09T23:32:45.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30876.423167 secs 2023-03-09T23:32:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:45 smithi038 ceph-mon[92422]: pgmap v16978: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 134 KiB/s wr, 7 op/s 2023-03-09T23:32:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30876.423167 secs 2023-03-09T23:32:47.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:47 smithi122 ceph-mon[93869]: pgmap v16979: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 288 KiB/s wr, 9 op/s 2023-03-09T23:32:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:32:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:47 smithi181 ceph-mon[94189]: pgmap v16979: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 288 KiB/s wr, 9 op/s 2023-03-09T23:32:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:32:47.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:47 smithi038 ceph-mon[92422]: pgmap v16979: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 288 KiB/s wr, 9 op/s 2023-03-09T23:32:47.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:32:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31340.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:32:49.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:32:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:32:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:32:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:32:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:32:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:32:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:32:49.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:32:49.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:32:49.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:32:49.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:32:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:32:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:32:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:32:49.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:32:49.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:32:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:32:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:32:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:32:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:32:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:32:50.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:49 smithi122 ceph-mon[93869]: pgmap v16980: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 329 KiB/s rd, 327 KiB/s wr, 7 op/s 2023-03-09T23:32:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:49 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:32:50.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:49 smithi181 ceph-mon[94189]: pgmap v16980: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 329 KiB/s rd, 327 KiB/s wr, 7 op/s 2023-03-09T23:32:50.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:49 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:32:50.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:49 smithi038 ceph-mon[92422]: pgmap v16980: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 329 KiB/s rd, 327 KiB/s wr, 7 op/s 2023-03-09T23:32:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:49 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:32:51.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:50 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:32:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:50 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:32:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:50 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:32:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30881.423228 secs 2023-03-09T23:32:51.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:50 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:32:51.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:50 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:32:51.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:50 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:32:51.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30881.423228 secs 2023-03-09T23:32:51.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:50 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:32:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:50 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:32:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:50 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:32:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30881.423228 secs 2023-03-09T23:32:52.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:51 smithi122 ceph-mon[93869]: pgmap v16981: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 448 KiB/s wr, 9 op/s 2023-03-09T23:32:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:32:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:32:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:51 smithi181 ceph-mon[94189]: pgmap v16981: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 448 KiB/s wr, 9 op/s 2023-03-09T23:32:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:32:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:32:52.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:51 smithi038 ceph-mon[92422]: pgmap v16981: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 448 KiB/s wr, 9 op/s 2023-03-09T23:32:52.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:32:52.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:32:53.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:53 smithi122 ceph-mon[93869]: pgmap v16982: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 453 KiB/s rd, 448 KiB/s wr, 10 op/s 2023-03-09T23:32:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:53 smithi181 ceph-mon[94189]: pgmap v16982: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 453 KiB/s rd, 448 KiB/s wr, 10 op/s 2023-03-09T23:32:53.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:53 smithi038 ceph-mon[92422]: pgmap v16982: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 453 KiB/s rd, 448 KiB/s wr, 10 op/s 2023-03-09T23:32:53.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31345.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:32:55.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:55 smithi122 ceph-mon[93869]: pgmap v16983: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 656 KiB/s rd, 642 KiB/s wr, 10 op/s 2023-03-09T23:32:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:32:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:32:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30886.423290 secs 2023-03-09T23:32:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:55 smithi181 ceph-mon[94189]: pgmap v16983: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 656 KiB/s rd, 642 KiB/s wr, 10 op/s 2023-03-09T23:32:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:32:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:32:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30886.423290 secs 2023-03-09T23:32:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:55 smithi038 ceph-mon[92422]: pgmap v16983: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 656 KiB/s rd, 642 KiB/s wr, 10 op/s 2023-03-09T23:32:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:32:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:32:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30886.423290 secs 2023-03-09T23:32:57.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:57 smithi122 ceph-mon[93869]: pgmap v16984: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 921 KiB/s rd, 754 KiB/s wr, 13 op/s 2023-03-09T23:32:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:32:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:32:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:57 smithi181 ceph-mon[94189]: pgmap v16984: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 921 KiB/s rd, 754 KiB/s wr, 13 op/s 2023-03-09T23:32:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:32:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:32:57.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:57 smithi038 ceph-mon[92422]: pgmap v16984: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 921 KiB/s rd, 754 KiB/s wr, 13 op/s 2023-03-09T23:32:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:32:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:32:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31350.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:32:59.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:59 smithi122 ceph-mon[93869]: pgmap v16985: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 779 KiB/s rd, 614 KiB/s wr, 12 op/s 2023-03-09T23:32:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:32:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:32:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:32:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:59 smithi181 ceph-mon[94189]: pgmap v16985: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 779 KiB/s rd, 614 KiB/s wr, 12 op/s 2023-03-09T23:32:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:32:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:32:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:32:59.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:59 smithi038 ceph-mon[92422]: pgmap v16985: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 779 KiB/s rd, 614 KiB/s wr, 12 op/s 2023-03-09T23:32:59.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:32:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:32:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:33:00.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30891.423347 secs 2023-03-09T23:33:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30891.423347 secs 2023-03-09T23:33:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30891.423347 secs 2023-03-09T23:33:01.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:01 smithi122 ceph-mon[93869]: pgmap v16986: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 822 KiB/s rd, 791 KiB/s wr, 13 op/s 2023-03-09T23:33:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:33:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:33:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:01 smithi181 ceph-mon[94189]: pgmap v16986: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 822 KiB/s rd, 791 KiB/s wr, 13 op/s 2023-03-09T23:33:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:33:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:33:01.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:01 smithi038 ceph-mon[92422]: pgmap v16986: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 822 KiB/s rd, 791 KiB/s wr, 13 op/s 2023-03-09T23:33:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:33:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:33:03.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30893.831710 secs 2023-03-09T23:33:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:03 smithi122 ceph-mon[93869]: pgmap v16987: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 727 KiB/s rd, 695 KiB/s wr, 15 op/s 2023-03-09T23:33:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30893.831710 secs 2023-03-09T23:33:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:03 smithi181 ceph-mon[94189]: pgmap v16987: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 727 KiB/s rd, 695 KiB/s wr, 15 op/s 2023-03-09T23:33:03.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30893.831710 secs 2023-03-09T23:33:03.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:03 smithi038 ceph-mon[92422]: pgmap v16987: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 727 KiB/s rd, 695 KiB/s wr, 15 op/s 2023-03-09T23:33:03.793 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31355.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:33:05.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:05 smithi122 ceph-mon[93869]: osdmap e1277: 12 total, 12 up, 12 in 2023-03-09T23:33:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:05 smithi122 ceph-mon[93869]: pgmap v16989: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 30 op/s 2023-03-09T23:33:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30896.423409 secs 2023-03-09T23:33:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:05 smithi181 ceph-mon[94189]: osdmap e1277: 12 total, 12 up, 12 in 2023-03-09T23:33:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:05 smithi181 ceph-mon[94189]: pgmap v16989: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 30 op/s 2023-03-09T23:33:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30896.423409 secs 2023-03-09T23:33:05.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:05 smithi038 ceph-mon[92422]: osdmap e1277: 12 total, 12 up, 12 in 2023-03-09T23:33:05.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:05 smithi038 ceph-mon[92422]: pgmap v16989: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 30 op/s 2023-03-09T23:33:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30896.423409 secs 2023-03-09T23:33:06.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:06 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:33:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:06 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:33:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:33:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:06 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:33:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:06 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:33:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:33:06.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:06 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:33:06.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:06 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:33:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:33:07.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:07 smithi038 ceph-mon[92422]: pgmap v16990: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 739 KiB/s rd, 781 KiB/s wr, 31 op/s 2023-03-09T23:33:07.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:07 smithi122 ceph-mon[93869]: pgmap v16990: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 739 KiB/s rd, 781 KiB/s wr, 31 op/s 2023-03-09T23:33:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:07 smithi181 ceph-mon[94189]: pgmap v16990: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 739 KiB/s rd, 781 KiB/s wr, 31 op/s 2023-03-09T23:33:08.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30898.831825 secs 2023-03-09T23:33:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:33:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:33:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31360.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:33:08.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30898.831825 secs 2023-03-09T23:33:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:33:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:33:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30898.831825 secs 2023-03-09T23:33:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:33:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:33:09.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:33:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:33:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:09 smithi038 ceph-mon[92422]: pgmap v16991: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 749 KiB/s rd, 791 KiB/s wr, 31 op/s 2023-03-09T23:33:09.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:33:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:33:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:09 smithi122 ceph-mon[93869]: pgmap v16991: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 749 KiB/s rd, 791 KiB/s wr, 31 op/s 2023-03-09T23:33:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:33:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:33:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:09 smithi181 ceph-mon[94189]: pgmap v16991: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 749 KiB/s rd, 791 KiB/s wr, 31 op/s 2023-03-09T23:33:10.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30901.423490 secs 2023-03-09T23:33:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30901.423490 secs 2023-03-09T23:33:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30901.423490 secs 2023-03-09T23:33:11.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:11 smithi122 ceph-mon[93869]: pgmap v16992: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 532 KiB/s wr, 30 op/s 2023-03-09T23:33:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:11 smithi181 ceph-mon[94189]: pgmap v16992: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 532 KiB/s wr, 30 op/s 2023-03-09T23:33:11.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:11 smithi038 ceph-mon[92422]: pgmap v16992: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 532 KiB/s wr, 30 op/s 2023-03-09T23:33:13.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30903.831963 secs 2023-03-09T23:33:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:13 smithi122 ceph-mon[93869]: osdmap e1278: 12 total, 12 up, 12 in 2023-03-09T23:33:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:13 smithi122 ceph-mon[93869]: pgmap v16994: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 67 KiB/s wr, 13 op/s 2023-03-09T23:33:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30903.831963 secs 2023-03-09T23:33:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:13 smithi181 ceph-mon[94189]: osdmap e1278: 12 total, 12 up, 12 in 2023-03-09T23:33:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:13 smithi181 ceph-mon[94189]: pgmap v16994: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 67 KiB/s wr, 13 op/s 2023-03-09T23:33:13.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30903.831963 secs 2023-03-09T23:33:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:13 smithi038 ceph-mon[92422]: osdmap e1278: 12 total, 12 up, 12 in 2023-03-09T23:33:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:13 smithi038 ceph-mon[92422]: pgmap v16994: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 67 KiB/s wr, 13 op/s 2023-03-09T23:33:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31365.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:33:15.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:15 smithi122 ceph-mon[93869]: pgmap v16995: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 214 KiB/s wr, 12 op/s 2023-03-09T23:33:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30906.423548 secs 2023-03-09T23:33:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:15 smithi181 ceph-mon[94189]: pgmap v16995: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 214 KiB/s wr, 12 op/s 2023-03-09T23:33:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30906.423548 secs 2023-03-09T23:33:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:15 smithi038 ceph-mon[92422]: pgmap v16995: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 214 KiB/s wr, 12 op/s 2023-03-09T23:33:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30906.423548 secs 2023-03-09T23:33:17.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:17 smithi122 ceph-mon[93869]: pgmap v16996: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 187 KiB/s wr, 10 op/s 2023-03-09T23:33:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:17 smithi181 ceph-mon[94189]: pgmap v16996: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 187 KiB/s wr, 10 op/s 2023-03-09T23:33:17.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:17 smithi038 ceph-mon[92422]: pgmap v16996: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 187 KiB/s wr, 10 op/s 2023-03-09T23:33:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31370.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:33:18.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30908.832066 secs 2023-03-09T23:33:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30908.832066 secs 2023-03-09T23:33:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30908.832066 secs 2023-03-09T23:33:19.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:19 smithi122 ceph-mon[93869]: pgmap v16997: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 160 KiB/s wr, 9 op/s 2023-03-09T23:33:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:19 smithi181 ceph-mon[94189]: pgmap v16997: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 160 KiB/s wr, 9 op/s 2023-03-09T23:33:19.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:19 smithi038 ceph-mon[92422]: pgmap v16997: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 160 KiB/s wr, 9 op/s 2023-03-09T23:33:20.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30911.423610 secs 2023-03-09T23:33:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30911.423610 secs 2023-03-09T23:33:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30911.423610 secs 2023-03-09T23:33:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:21 smithi122 ceph-mon[93869]: pgmap v16998: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 160 KiB/s wr, 9 op/s 2023-03-09T23:33:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:21 smithi181 ceph-mon[94189]: pgmap v16998: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 160 KiB/s wr, 9 op/s 2023-03-09T23:33:21.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:21 smithi038 ceph-mon[92422]: pgmap v16998: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 160 KiB/s wr, 9 op/s 2023-03-09T23:33:23.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30913.832184 secs 2023-03-09T23:33:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:23 smithi122 ceph-mon[93869]: pgmap v16999: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 159 KiB/s wr, 8 op/s 2023-03-09T23:33:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30913.832184 secs 2023-03-09T23:33:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:23 smithi181 ceph-mon[94189]: pgmap v16999: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 159 KiB/s wr, 8 op/s 2023-03-09T23:33:23.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30913.832184 secs 2023-03-09T23:33:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:23 smithi038 ceph-mon[92422]: pgmap v16999: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 159 KiB/s wr, 8 op/s 2023-03-09T23:33:23.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31375.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:33:25.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:25 smithi122 ceph-mon[93869]: pgmap v17000: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 133 KiB/s wr, 8 op/s 2023-03-09T23:33:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30916.423669 secs 2023-03-09T23:33:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:25 smithi181 ceph-mon[94189]: pgmap v17000: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 133 KiB/s wr, 8 op/s 2023-03-09T23:33:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30916.423669 secs 2023-03-09T23:33:25.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:25 smithi038 ceph-mon[92422]: pgmap v17000: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 133 KiB/s wr, 8 op/s 2023-03-09T23:33:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30916.423669 secs 2023-03-09T23:33:27.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:27 smithi122 ceph-mon[93869]: pgmap v17001: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T23:33:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:27 smithi181 ceph-mon[94189]: pgmap v17001: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T23:33:27.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:27 smithi038 ceph-mon[92422]: pgmap v17001: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T23:33:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31380.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:33:28.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30918.832302 secs 2023-03-09T23:33:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30918.832302 secs 2023-03-09T23:33:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30918.832302 secs 2023-03-09T23:33:29.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:29 smithi122 ceph-mon[93869]: pgmap v17002: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 7 op/s 2023-03-09T23:33:29.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:29 smithi181 ceph-mon[94189]: pgmap v17002: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 7 op/s 2023-03-09T23:33:29.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:29 smithi038 ceph-mon[92422]: pgmap v17002: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 7 op/s 2023-03-09T23:33:30.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30921.423728 secs 2023-03-09T23:33:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30921.423728 secs 2023-03-09T23:33:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30921.423728 secs 2023-03-09T23:33:31.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:31 smithi122 ceph-mon[93869]: pgmap v17003: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 7 op/s 2023-03-09T23:33:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:31 smithi181 ceph-mon[94189]: pgmap v17003: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 7 op/s 2023-03-09T23:33:31.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:31 smithi038 ceph-mon[92422]: pgmap v17003: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 7 op/s 2023-03-09T23:33:33.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30923.832419 secs 2023-03-09T23:33:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:33 smithi038 ceph-mon[92422]: pgmap v17004: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T23:33:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31385.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:33:33.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30923.832419 secs 2023-03-09T23:33:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:33 smithi122 ceph-mon[93869]: pgmap v17004: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T23:33:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30923.832419 secs 2023-03-09T23:33:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:33 smithi181 ceph-mon[94189]: pgmap v17004: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T23:33:35.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:35 smithi122 ceph-mon[93869]: pgmap v17005: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 8 op/s 2023-03-09T23:33:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30926.423787 secs 2023-03-09T23:33:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:35 smithi181 ceph-mon[94189]: pgmap v17005: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 8 op/s 2023-03-09T23:33:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30926.423787 secs 2023-03-09T23:33:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:35 smithi038 ceph-mon[92422]: pgmap v17005: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 8 op/s 2023-03-09T23:33:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30926.423787 secs 2023-03-09T23:33:37.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:37 smithi122 ceph-mon[93869]: pgmap v17006: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 8 op/s 2023-03-09T23:33:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:37 smithi181 ceph-mon[94189]: pgmap v17006: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 8 op/s 2023-03-09T23:33:37.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:37 smithi038 ceph-mon[92422]: pgmap v17006: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 8 op/s 2023-03-09T23:33:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31390.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:33:38.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30928.832550 secs 2023-03-09T23:33:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30928.832550 secs 2023-03-09T23:33:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30928.832550 secs 2023-03-09T23:33:39.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:39 smithi122 ceph-mon[93869]: pgmap v17007: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 597 B/s wr, 7 op/s 2023-03-09T23:33:40.032 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:39 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:33:40.032 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:39 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:33:40.033 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:39 smithi038 ceph-mon[92422]: pgmap v17007: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 597 B/s wr, 7 op/s 2023-03-09T23:33:40.034 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:39 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:33:40.034 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:39 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:33:40.034 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:39 smithi181 ceph-mon[94189]: pgmap v17007: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 597 B/s wr, 7 op/s 2023-03-09T23:33:40.035 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:39 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:33:40.035 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:39 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:33:40.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30931.423847 secs 2023-03-09T23:33:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30931.423847 secs 2023-03-09T23:33:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30931.423847 secs 2023-03-09T23:33:41.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:41 smithi122 ceph-mon[93869]: pgmap v17008: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 137 KiB/s wr, 8 op/s 2023-03-09T23:33:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:33:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:33:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:41 smithi181 ceph-mon[94189]: pgmap v17008: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 137 KiB/s wr, 8 op/s 2023-03-09T23:33:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:33:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:33:41.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:41 smithi038 ceph-mon[92422]: pgmap v17008: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 137 KiB/s wr, 8 op/s 2023-03-09T23:33:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:33:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:33:43.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:43 smithi122 ceph-mon[93869]: pgmap v17009: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 137 KiB/s wr, 9 op/s 2023-03-09T23:33:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:43 smithi181 ceph-mon[94189]: pgmap v17009: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 137 KiB/s wr, 9 op/s 2023-03-09T23:33:43.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:43 smithi038 ceph-mon[92422]: pgmap v17009: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 137 KiB/s wr, 9 op/s 2023-03-09T23:33:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31395.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:33:45.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:45 smithi122 ceph-mon[93869]: pgmap v17010: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 137 KiB/s wr, 8 op/s 2023-03-09T23:33:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30936.423907 secs 2023-03-09T23:33:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:45 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:33:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:45 smithi181 ceph-mon[94189]: pgmap v17010: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 137 KiB/s wr, 8 op/s 2023-03-09T23:33:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30936.423907 secs 2023-03-09T23:33:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:45 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:33:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:45 smithi038 ceph-mon[92422]: pgmap v17010: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 137 KiB/s wr, 8 op/s 2023-03-09T23:33:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30936.423907 secs 2023-03-09T23:33:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:45 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:33:46.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:46 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:33:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:46 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:33:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:46 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:33:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:46 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:33:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:46 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:33:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:46 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:33:46.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:46 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:33:46.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:46 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:33:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:46 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:33:47.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:47 smithi122 ceph-mon[93869]: pgmap v17011: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 137 KiB/s wr, 8 op/s 2023-03-09T23:33:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:47 smithi181 ceph-mon[94189]: pgmap v17011: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 137 KiB/s wr, 8 op/s 2023-03-09T23:33:47.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:47 smithi038 ceph-mon[92422]: pgmap v17011: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 137 KiB/s wr, 8 op/s 2023-03-09T23:33:48.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31400.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:33:48.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:33:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:33:48.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:33:49.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:49 smithi122 ceph-mon[93869]: pgmap v17012: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 137 KiB/s wr, 7 op/s 2023-03-09T23:33:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:33:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:33:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:33:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:33:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:49 smithi181 ceph-mon[94189]: pgmap v17012: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 137 KiB/s wr, 7 op/s 2023-03-09T23:33:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:33:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:33:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:33:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:33:49.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:49 smithi038 ceph-mon[92422]: pgmap v17012: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 137 KiB/s wr, 7 op/s 2023-03-09T23:33:49.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:33:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:33:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:33:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:33:50.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:50 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:33:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:50 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:33:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:50 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:33:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30941.423993 secs 2023-03-09T23:33:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:50 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:33:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:50 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:33:50.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:50 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:33:50.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30941.423993 secs 2023-03-09T23:33:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:50 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:33:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:50 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:33:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:50 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:33:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30941.423993 secs 2023-03-09T23:33:52.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:51 smithi122 ceph-mon[93869]: pgmap v17013: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 490 KiB/s rd, 365 KiB/s wr, 8 op/s 2023-03-09T23:33:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:51 smithi181 ceph-mon[94189]: pgmap v17013: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 490 KiB/s rd, 365 KiB/s wr, 8 op/s 2023-03-09T23:33:52.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:51 smithi038 ceph-mon[92422]: pgmap v17013: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 490 KiB/s rd, 365 KiB/s wr, 8 op/s 2023-03-09T23:33:53.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:33:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:33:53.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:33:53.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:33:53.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:33:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:33:53.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31405.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:33:54.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:53 smithi122 ceph-mon[93869]: pgmap v17014: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 253 KiB/s wr, 9 op/s 2023-03-09T23:33:54.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:53 smithi181 ceph-mon[94189]: pgmap v17014: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 253 KiB/s wr, 9 op/s 2023-03-09T23:33:54.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:53 smithi038 ceph-mon[92422]: pgmap v17014: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 253 KiB/s wr, 9 op/s 2023-03-09T23:33:56.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:55 smithi122 ceph-mon[93869]: pgmap v17015: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 750 KiB/s rd, 594 KiB/s wr, 8 op/s 2023-03-09T23:33:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:33:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:33:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30946.424052 secs 2023-03-09T23:33:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:55 smithi181 ceph-mon[94189]: pgmap v17015: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 750 KiB/s rd, 594 KiB/s wr, 8 op/s 2023-03-09T23:33:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:33:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:33:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30946.424052 secs 2023-03-09T23:33:56.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:55 smithi038 ceph-mon[92422]: pgmap v17015: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 750 KiB/s rd, 594 KiB/s wr, 8 op/s 2023-03-09T23:33:56.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:33:56.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:33:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30946.424052 secs 2023-03-09T23:33:57.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:33:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:33:57.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:33:57.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:33:57.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:33:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:33:58.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:57 smithi122 ceph-mon[93869]: pgmap v17016: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 910 KiB/s wr, 11 op/s 2023-03-09T23:33:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:57 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 30948.833035 secs 2023-03-09T23:33:58.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:57 smithi181 ceph-mon[94189]: pgmap v17016: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 910 KiB/s wr, 11 op/s 2023-03-09T23:33:58.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:57 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 30948.833035 secs 2023-03-09T23:33:58.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:57 smithi038 ceph-mon[92422]: pgmap v17016: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 910 KiB/s wr, 11 op/s 2023-03-09T23:33:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:57 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 30948.833035 secs 2023-03-09T23:33:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31410.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:34:00.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:59 smithi122 ceph-mon[93869]: pgmap v17017: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 12 op/s 2023-03-09T23:34:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:34:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:33:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:34:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:59 smithi181 ceph-mon[94189]: pgmap v17017: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 12 op/s 2023-03-09T23:34:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:34:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:33:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:34:00.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:59 smithi038 ceph-mon[92422]: pgmap v17017: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 12 op/s 2023-03-09T23:34:00.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:34:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:33:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:34:01.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:34:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:34:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30951.424109 secs 2023-03-09T23:34:01.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:34:01.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:34:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30951.424109 secs 2023-03-09T23:34:01.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:34:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:34:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30951.424109 secs 2023-03-09T23:34:02.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:01 smithi122 ceph-mon[93869]: pgmap v17018: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 12 op/s 2023-03-09T23:34:02.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:01 smithi181 ceph-mon[94189]: pgmap v17018: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 12 op/s 2023-03-09T23:34:02.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:01 smithi038 ceph-mon[92422]: pgmap v17018: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 12 op/s 2023-03-09T23:34:03.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:02 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:34:03.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:02 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:34:03.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:02 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:34:03.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30953.833160 secs 2023-03-09T23:34:03.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:02 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:34:03.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:02 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:34:03.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:02 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:34:03.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30953.833160 secs 2023-03-09T23:34:03.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:02 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:34:03.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:02 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:34:03.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:02 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:34:03.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30953.833160 secs 2023-03-09T23:34:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31415.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:34:04.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:03 smithi122 ceph-mon[93869]: pgmap v17019: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 944 KiB/s rd, 1.1 MiB/s wr, 14 op/s 2023-03-09T23:34:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:03 smithi181 ceph-mon[94189]: pgmap v17019: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 944 KiB/s rd, 1.1 MiB/s wr, 14 op/s 2023-03-09T23:34:04.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:03 smithi038 ceph-mon[92422]: pgmap v17019: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 944 KiB/s rd, 1.1 MiB/s wr, 14 op/s 2023-03-09T23:34:06.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:05 smithi122 ceph-mon[93869]: pgmap v17020: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 26 op/s 2023-03-09T23:34:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30956.424185 secs 2023-03-09T23:34:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:05 smithi181 ceph-mon[94189]: pgmap v17020: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 26 op/s 2023-03-09T23:34:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30956.424185 secs 2023-03-09T23:34:06.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:05 smithi038 ceph-mon[92422]: pgmap v17020: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 26 op/s 2023-03-09T23:34:06.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30956.424185 secs 2023-03-09T23:34:07.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:06 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:34:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:34:07.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:06 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:34:07.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:34:07.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:06 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:34:07.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:34:08.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:07 smithi122 ceph-mon[93869]: pgmap v17021: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1014 KiB/s rd, 1.0 MiB/s wr, 29 op/s 2023-03-09T23:34:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:34:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:34:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30958.833286 secs 2023-03-09T23:34:08.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:07 smithi181 ceph-mon[94189]: pgmap v17021: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1014 KiB/s rd, 1.0 MiB/s wr, 29 op/s 2023-03-09T23:34:08.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:34:08.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:34:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30958.833286 secs 2023-03-09T23:34:08.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:07 smithi038 ceph-mon[92422]: pgmap v17021: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1014 KiB/s rd, 1.0 MiB/s wr, 29 op/s 2023-03-09T23:34:08.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:34:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:34:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30958.833286 secs 2023-03-09T23:34:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31420.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:34:10.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:09 smithi122 ceph-mon[93869]: pgmap v17022: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 717 KiB/s rd, 808 KiB/s wr, 28 op/s 2023-03-09T23:34:10.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:09 smithi181 ceph-mon[94189]: pgmap v17022: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 717 KiB/s rd, 808 KiB/s wr, 28 op/s 2023-03-09T23:34:10.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:09 smithi038 ceph-mon[92422]: pgmap v17022: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 717 KiB/s rd, 808 KiB/s wr, 28 op/s 2023-03-09T23:34:11.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:10 smithi181 ceph-mon[94189]: osdmap e1279: 12 total, 12 up, 12 in 2023-03-09T23:34:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30961.424250 secs 2023-03-09T23:34:11.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:10 smithi038 ceph-mon[92422]: osdmap e1279: 12 total, 12 up, 12 in 2023-03-09T23:34:11.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30961.424250 secs 2023-03-09T23:34:11.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:10 smithi122 ceph-mon[93869]: osdmap e1279: 12 total, 12 up, 12 in 2023-03-09T23:34:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30961.424250 secs 2023-03-09T23:34:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:11 smithi038 ceph-mon[92422]: pgmap v17024: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 919 KiB/s rd, 802 KiB/s wr, 31 op/s 2023-03-09T23:34:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:11 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:34:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:11 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:34:12.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:11 smithi122 ceph-mon[93869]: pgmap v17024: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 919 KiB/s rd, 802 KiB/s wr, 31 op/s 2023-03-09T23:34:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:11 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:34:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:11 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:34:12.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:11 smithi181 ceph-mon[94189]: pgmap v17024: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 919 KiB/s rd, 802 KiB/s wr, 31 op/s 2023-03-09T23:34:12.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:11 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:34:12.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:11 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:34:13.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:34:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:34:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30963.833398 secs 2023-03-09T23:34:13.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:34:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:34:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30963.833398 secs 2023-03-09T23:34:13.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:34:13.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:34:13.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30963.833398 secs 2023-03-09T23:34:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31425.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:34:14.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:13 smithi038 ceph-mon[92422]: pgmap v17025: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 888 KiB/s rd, 769 KiB/s wr, 30 op/s 2023-03-09T23:34:14.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:13 smithi122 ceph-mon[93869]: pgmap v17025: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 888 KiB/s rd, 769 KiB/s wr, 30 op/s 2023-03-09T23:34:14.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:13 smithi181 ceph-mon[94189]: pgmap v17025: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 888 KiB/s rd, 769 KiB/s wr, 30 op/s 2023-03-09T23:34:16.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:15 smithi038 ceph-mon[92422]: pgmap v17026: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 432 KiB/s wr, 13 op/s 2023-03-09T23:34:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30966.424330 secs 2023-03-09T23:34:16.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:15 smithi122 ceph-mon[93869]: pgmap v17026: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 432 KiB/s wr, 13 op/s 2023-03-09T23:34:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30966.424330 secs 2023-03-09T23:34:16.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:15 smithi181 ceph-mon[94189]: pgmap v17026: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 432 KiB/s wr, 13 op/s 2023-03-09T23:34:16.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30966.424330 secs 2023-03-09T23:34:18.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:17 smithi038 ceph-mon[92422]: pgmap v17027: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 519 KiB/s rd, 515 KiB/s wr, 10 op/s 2023-03-09T23:34:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30968.833492 secs 2023-03-09T23:34:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:17 smithi038 ceph-mon[92422]: osdmap e1280: 12 total, 12 up, 12 in 2023-03-09T23:34:18.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:17 smithi122 ceph-mon[93869]: pgmap v17027: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 519 KiB/s rd, 515 KiB/s wr, 10 op/s 2023-03-09T23:34:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30968.833492 secs 2023-03-09T23:34:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:17 smithi122 ceph-mon[93869]: osdmap e1280: 12 total, 12 up, 12 in 2023-03-09T23:34:18.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:17 smithi181 ceph-mon[94189]: pgmap v17027: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 519 KiB/s rd, 515 KiB/s wr, 10 op/s 2023-03-09T23:34:18.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30968.833492 secs 2023-03-09T23:34:18.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:17 smithi181 ceph-mon[94189]: osdmap e1280: 12 total, 12 up, 12 in 2023-03-09T23:34:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31430.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:34:20.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:20 smithi122 ceph-mon[93869]: pgmap v17029: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 576 KiB/s rd, 480 KiB/s wr, 10 op/s 2023-03-09T23:34:20.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:20 smithi181 ceph-mon[94189]: pgmap v17029: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 576 KiB/s rd, 480 KiB/s wr, 10 op/s 2023-03-09T23:34:20.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:20 smithi038 ceph-mon[92422]: pgmap v17029: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 576 KiB/s rd, 480 KiB/s wr, 10 op/s 2023-03-09T23:34:21.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30971.424392 secs 2023-03-09T23:34:21.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30971.424392 secs 2023-03-09T23:34:21.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30971.424392 secs 2023-03-09T23:34:22.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:22 smithi122 ceph-mon[93869]: pgmap v17030: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 114 KiB/s wr, 8 op/s 2023-03-09T23:34:22.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:22 smithi181 ceph-mon[94189]: pgmap v17030: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 114 KiB/s wr, 8 op/s 2023-03-09T23:34:22.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:22 smithi038 ceph-mon[92422]: pgmap v17030: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 114 KiB/s wr, 8 op/s 2023-03-09T23:34:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31435.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:34:23.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30973.833611 secs 2023-03-09T23:34:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:23 smithi122 ceph-mon[93869]: pgmap v17031: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 113 KiB/s wr, 7 op/s 2023-03-09T23:34:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30973.833611 secs 2023-03-09T23:34:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:23 smithi181 ceph-mon[94189]: pgmap v17031: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 113 KiB/s wr, 7 op/s 2023-03-09T23:34:23.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30973.833611 secs 2023-03-09T23:34:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:23 smithi038 ceph-mon[92422]: pgmap v17031: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 113 KiB/s wr, 7 op/s 2023-03-09T23:34:25.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:25 smithi122 ceph-mon[93869]: pgmap v17032: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 113 KiB/s wr, 7 op/s 2023-03-09T23:34:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30976.424453 secs 2023-03-09T23:34:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:25 smithi181 ceph-mon[94189]: pgmap v17032: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 113 KiB/s wr, 7 op/s 2023-03-09T23:34:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30976.424453 secs 2023-03-09T23:34:25.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:25 smithi038 ceph-mon[92422]: pgmap v17032: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 113 KiB/s wr, 7 op/s 2023-03-09T23:34:25.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30976.424453 secs 2023-03-09T23:34:27.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:27 smithi122 ceph-mon[93869]: pgmap v17033: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.1 KiB/s wr, 7 op/s 2023-03-09T23:34:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:27 smithi181 ceph-mon[94189]: pgmap v17033: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.1 KiB/s wr, 7 op/s 2023-03-09T23:34:27.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:27 smithi038 ceph-mon[92422]: pgmap v17033: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.1 KiB/s wr, 7 op/s 2023-03-09T23:34:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31440.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:34:28.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30978.833729 secs 2023-03-09T23:34:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30978.833729 secs 2023-03-09T23:34:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30978.833729 secs 2023-03-09T23:34:29.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:29 smithi122 ceph-mon[93869]: pgmap v17034: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2023-03-09T23:34:29.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:29 smithi181 ceph-mon[94189]: pgmap v17034: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2023-03-09T23:34:29.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:29 smithi038 ceph-mon[92422]: pgmap v17034: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2023-03-09T23:34:30.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30981.424515 secs 2023-03-09T23:34:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30981.424515 secs 2023-03-09T23:34:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30981.424515 secs 2023-03-09T23:34:31.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:31 smithi122 ceph-mon[93869]: pgmap v17035: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T23:34:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:31 smithi181 ceph-mon[94189]: pgmap v17035: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T23:34:31.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:31 smithi038 ceph-mon[92422]: pgmap v17035: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T23:34:33.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30983.833848 secs 2023-03-09T23:34:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:33 smithi038 ceph-mon[92422]: pgmap v17036: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T23:34:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31445.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:34:33.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30983.833848 secs 2023-03-09T23:34:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:33 smithi122 ceph-mon[93869]: pgmap v17036: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T23:34:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30983.833848 secs 2023-03-09T23:34:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:33 smithi181 ceph-mon[94189]: pgmap v17036: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T23:34:35.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:35 smithi122 ceph-mon[93869]: pgmap v17037: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T23:34:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30986.424576 secs 2023-03-09T23:34:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:35 smithi181 ceph-mon[94189]: pgmap v17037: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T23:34:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30986.424576 secs 2023-03-09T23:34:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:35 smithi038 ceph-mon[92422]: pgmap v17037: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T23:34:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30986.424576 secs 2023-03-09T23:34:37.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:37 smithi122 ceph-mon[93869]: pgmap v17038: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2023-03-09T23:34:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:37 smithi181 ceph-mon[94189]: pgmap v17038: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2023-03-09T23:34:37.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:37 smithi038 ceph-mon[92422]: pgmap v17038: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2023-03-09T23:34:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31450.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:34:38.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30988.833964 secs 2023-03-09T23:34:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30988.833964 secs 2023-03-09T23:34:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30988.833964 secs 2023-03-09T23:34:39.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:39 smithi122 ceph-mon[93869]: pgmap v17039: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 7 op/s 2023-03-09T23:34:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:39 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:34:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:39 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:34:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:39 smithi181 ceph-mon[94189]: pgmap v17039: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 7 op/s 2023-03-09T23:34:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:39 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:34:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:39 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:34:39.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:39 smithi038 ceph-mon[92422]: pgmap v17039: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 7 op/s 2023-03-09T23:34:39.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:39 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:34:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:39 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:34:40.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30991.424660 secs 2023-03-09T23:34:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30991.424660 secs 2023-03-09T23:34:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30991.424660 secs 2023-03-09T23:34:41.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:41 smithi122 ceph-mon[93869]: pgmap v17040: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-09T23:34:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:34:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:34:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:41 smithi181 ceph-mon[94189]: pgmap v17040: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-09T23:34:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:34:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:34:41.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:41 smithi038 ceph-mon[92422]: pgmap v17040: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-09T23:34:41.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:34:41.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:34:43.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30993.834083 secs 2023-03-09T23:34:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:43 smithi038 ceph-mon[92422]: pgmap v17041: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 138 KiB/s wr, 9 op/s 2023-03-09T23:34:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31455.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:34:43.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30993.834083 secs 2023-03-09T23:34:43.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:43 smithi122 ceph-mon[93869]: pgmap v17041: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 138 KiB/s wr, 9 op/s 2023-03-09T23:34:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30993.834083 secs 2023-03-09T23:34:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:43 smithi181 ceph-mon[94189]: pgmap v17041: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 138 KiB/s wr, 9 op/s 2023-03-09T23:34:45.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:45 smithi122 ceph-mon[93869]: pgmap v17042: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 138 KiB/s wr, 7 op/s 2023-03-09T23:34:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30996.424722 secs 2023-03-09T23:34:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:45 smithi181 ceph-mon[94189]: pgmap v17042: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 138 KiB/s wr, 7 op/s 2023-03-09T23:34:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30996.424722 secs 2023-03-09T23:34:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:45 smithi038 ceph-mon[92422]: pgmap v17042: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 138 KiB/s wr, 7 op/s 2023-03-09T23:34:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30996.424722 secs 2023-03-09T23:34:47.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:47 smithi122 ceph-mon[93869]: pgmap v17043: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 138 KiB/s wr, 9 op/s 2023-03-09T23:34:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:47 smithi181 ceph-mon[94189]: pgmap v17043: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 138 KiB/s wr, 9 op/s 2023-03-09T23:34:47.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:47 smithi038 ceph-mon[92422]: pgmap v17043: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 138 KiB/s wr, 9 op/s 2023-03-09T23:34:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31460.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:34:48.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 30998.834200 secs 2023-03-09T23:34:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 30998.834200 secs 2023-03-09T23:34:48.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 30998.834200 secs 2023-03-09T23:34:49.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:49 smithi122 ceph-mon[93869]: pgmap v17044: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 342 KiB/s wr, 8 op/s 2023-03-09T23:34:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:49 smithi181 ceph-mon[94189]: pgmap v17044: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 342 KiB/s wr, 8 op/s 2023-03-09T23:34:49.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:49 smithi038 ceph-mon[92422]: pgmap v17044: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 342 KiB/s wr, 8 op/s 2023-03-09T23:34:50.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:50 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:34:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31001.424784 secs 2023-03-09T23:34:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:50 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:34:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31001.424784 secs 2023-03-09T23:34:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:50 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:34:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31001.424784 secs 2023-03-09T23:34:51.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:51 smithi122 ceph-mon[93869]: pgmap v17045: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 342 KiB/s wr, 7 op/s 2023-03-09T23:34:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:34:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:34:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:34:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:51 smithi181 ceph-mon[94189]: pgmap v17045: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 342 KiB/s wr, 7 op/s 2023-03-09T23:34:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:34:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:34:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:34:51.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:51 smithi038 ceph-mon[92422]: pgmap v17045: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 342 KiB/s wr, 7 op/s 2023-03-09T23:34:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:34:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:34:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:34:53.793 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31465.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:34:53.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31003.834309 secs 2023-03-09T23:34:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:53 smithi122 ceph-mon[93869]: pgmap v17046: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 976 KiB/s rd, 713 KiB/s wr, 11 op/s 2023-03-09T23:34:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:34:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:34:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31003.834309 secs 2023-03-09T23:34:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:53 smithi181 ceph-mon[94189]: pgmap v17046: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 976 KiB/s rd, 713 KiB/s wr, 11 op/s 2023-03-09T23:34:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:34:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:34:53.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31003.834309 secs 2023-03-09T23:34:53.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:53 smithi038 ceph-mon[92422]: pgmap v17046: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 976 KiB/s rd, 713 KiB/s wr, 11 op/s 2023-03-09T23:34:53.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:34:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:34:55.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:55 smithi122 ceph-mon[93869]: pgmap v17047: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 838 KiB/s rd, 576 KiB/s wr, 8 op/s 2023-03-09T23:34:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:34:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:34:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31006.424843 secs 2023-03-09T23:34:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:55 smithi181 ceph-mon[94189]: pgmap v17047: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 838 KiB/s rd, 576 KiB/s wr, 8 op/s 2023-03-09T23:34:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:34:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:34:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31006.424843 secs 2023-03-09T23:34:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:55 smithi038 ceph-mon[92422]: pgmap v17047: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 838 KiB/s rd, 576 KiB/s wr, 8 op/s 2023-03-09T23:34:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:34:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:34:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31006.424843 secs 2023-03-09T23:34:57.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:57 smithi122 ceph-mon[93869]: pgmap v17048: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 841 KiB/s rd, 577 KiB/s wr, 10 op/s 2023-03-09T23:34:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:34:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:34:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:57 smithi181 ceph-mon[94189]: pgmap v17048: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 841 KiB/s rd, 577 KiB/s wr, 10 op/s 2023-03-09T23:34:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:34:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:34:57.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:57 smithi038 ceph-mon[92422]: pgmap v17048: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 841 KiB/s rd, 577 KiB/s wr, 10 op/s 2023-03-09T23:34:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:34:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:34:58.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31470.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:34:58.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31008.834426 secs 2023-03-09T23:34:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31008.834426 secs 2023-03-09T23:34:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31008.834426 secs 2023-03-09T23:34:59.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:34:59 smithi122 ceph-mon[93869]: pgmap v17049: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 923 KiB/s rd, 715 KiB/s wr, 11 op/s 2023-03-09T23:34:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:34:59 smithi181 ceph-mon[94189]: pgmap v17049: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 923 KiB/s rd, 715 KiB/s wr, 11 op/s 2023-03-09T23:34:59.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:34:59 smithi038 ceph-mon[92422]: pgmap v17049: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 923 KiB/s rd, 715 KiB/s wr, 11 op/s 2023-03-09T23:35:00.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31011.424902 secs 2023-03-09T23:35:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31011.424902 secs 2023-03-09T23:35:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31011.424902 secs 2023-03-09T23:35:01.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:01 smithi122 ceph-mon[93869]: pgmap v17050: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 626 KiB/s rd, 511 KiB/s wr, 9 op/s 2023-03-09T23:35:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:35:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:35:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:01 smithi181 ceph-mon[94189]: pgmap v17050: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 626 KiB/s rd, 511 KiB/s wr, 9 op/s 2023-03-09T23:35:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:35:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:35:01.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:01 smithi038 ceph-mon[92422]: pgmap v17050: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 626 KiB/s rd, 511 KiB/s wr, 9 op/s 2023-03-09T23:35:01.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:35:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:35:03.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31475.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:35:03.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31013.834553 secs 2023-03-09T23:35:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:03 smithi122 ceph-mon[93869]: pgmap v17051: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 932 KiB/s rd, 816 KiB/s wr, 13 op/s 2023-03-09T23:35:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31013.834553 secs 2023-03-09T23:35:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:03 smithi181 ceph-mon[94189]: pgmap v17051: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 932 KiB/s rd, 816 KiB/s wr, 13 op/s 2023-03-09T23:35:03.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31013.834553 secs 2023-03-09T23:35:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:03 smithi038 ceph-mon[92422]: pgmap v17051: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 932 KiB/s rd, 816 KiB/s wr, 13 op/s 2023-03-09T23:35:05.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:05 smithi122 ceph-mon[93869]: pgmap v17052: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 797 KiB/s rd, 732 KiB/s wr, 12 op/s 2023-03-09T23:35:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31016.424959 secs 2023-03-09T23:35:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:05 smithi181 ceph-mon[94189]: pgmap v17052: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 797 KiB/s rd, 732 KiB/s wr, 12 op/s 2023-03-09T23:35:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31016.424959 secs 2023-03-09T23:35:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:05 smithi038 ceph-mon[92422]: pgmap v17052: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 797 KiB/s rd, 732 KiB/s wr, 12 op/s 2023-03-09T23:35:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31016.424959 secs 2023-03-09T23:35:07.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:07 smithi122 ceph-mon[93869]: pgmap v17053: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 822 KiB/s rd, 778 KiB/s wr, 17 op/s 2023-03-09T23:35:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:35:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:35:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:07 smithi181 ceph-mon[94189]: pgmap v17053: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 822 KiB/s rd, 778 KiB/s wr, 17 op/s 2023-03-09T23:35:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:35:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:35:07.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:07 smithi038 ceph-mon[92422]: pgmap v17053: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 822 KiB/s rd, 778 KiB/s wr, 17 op/s 2023-03-09T23:35:07.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:35:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:35:08.793 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31480.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:35:08.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31018.834670 secs 2023-03-09T23:35:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31018.834670 secs 2023-03-09T23:35:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31018.834670 secs 2023-03-09T23:35:09.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:09 smithi122 ceph-mon[93869]: pgmap v17054: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 883 KiB/s rd, 824 KiB/s wr, 27 op/s 2023-03-09T23:35:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:09 smithi181 ceph-mon[94189]: pgmap v17054: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 883 KiB/s rd, 824 KiB/s wr, 27 op/s 2023-03-09T23:35:09.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:09 smithi038 ceph-mon[92422]: pgmap v17054: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 883 KiB/s rd, 824 KiB/s wr, 27 op/s 2023-03-09T23:35:10.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:10 smithi122 ceph-mon[93869]: osdmap e1281: 12 total, 12 up, 12 in 2023-03-09T23:35:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31021.424992 secs 2023-03-09T23:35:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:10 smithi181 ceph-mon[94189]: osdmap e1281: 12 total, 12 up, 12 in 2023-03-09T23:35:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31021.424992 secs 2023-03-09T23:35:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:10 smithi038 ceph-mon[92422]: osdmap e1281: 12 total, 12 up, 12 in 2023-03-09T23:35:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31021.424992 secs 2023-03-09T23:35:11.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:11 smithi122 ceph-mon[93869]: pgmap v17056: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 947 KiB/s rd, 822 KiB/s wr, 30 op/s 2023-03-09T23:35:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:11 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:35:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:11 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:35:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:11 smithi181 ceph-mon[94189]: pgmap v17056: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 947 KiB/s rd, 822 KiB/s wr, 30 op/s 2023-03-09T23:35:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:11 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:35:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:11 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:35:11.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:11 smithi038 ceph-mon[92422]: pgmap v17056: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 947 KiB/s rd, 822 KiB/s wr, 30 op/s 2023-03-09T23:35:11.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:11 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:35:11.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:11 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:35:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31485.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:35:13.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31023.834819 secs 2023-03-09T23:35:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:13 smithi122 ceph-mon[93869]: pgmap v17057: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 983 KiB/s rd, 859 KiB/s wr, 30 op/s 2023-03-09T23:35:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:35:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:35:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31023.834819 secs 2023-03-09T23:35:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:13 smithi181 ceph-mon[94189]: pgmap v17057: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 983 KiB/s rd, 859 KiB/s wr, 30 op/s 2023-03-09T23:35:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:35:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:35:13.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31023.834819 secs 2023-03-09T23:35:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:13 smithi038 ceph-mon[92422]: pgmap v17057: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 983 KiB/s rd, 859 KiB/s wr, 30 op/s 2023-03-09T23:35:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:35:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:35:15.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:15 smithi122 ceph-mon[93869]: pgmap v17058: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 514 KiB/s wr, 27 op/s 2023-03-09T23:35:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31026.425052 secs 2023-03-09T23:35:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:15 smithi181 ceph-mon[94189]: pgmap v17058: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 514 KiB/s wr, 27 op/s 2023-03-09T23:35:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31026.425052 secs 2023-03-09T23:35:15.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:15 smithi038 ceph-mon[92422]: pgmap v17058: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 514 KiB/s wr, 27 op/s 2023-03-09T23:35:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31026.425052 secs 2023-03-09T23:35:17.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:17 smithi122 ceph-mon[93869]: pgmap v17059: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 596 KiB/s rd, 572 KiB/s wr, 24 op/s 2023-03-09T23:35:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:17 smithi122 ceph-mon[93869]: osdmap e1282: 12 total, 12 up, 12 in 2023-03-09T23:35:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:17 smithi181 ceph-mon[94189]: pgmap v17059: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 596 KiB/s rd, 572 KiB/s wr, 24 op/s 2023-03-09T23:35:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:17 smithi181 ceph-mon[94189]: osdmap e1282: 12 total, 12 up, 12 in 2023-03-09T23:35:17.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:17 smithi038 ceph-mon[92422]: pgmap v17059: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 596 KiB/s rd, 572 KiB/s wr, 24 op/s 2023-03-09T23:35:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:17 smithi038 ceph-mon[92422]: osdmap e1282: 12 total, 12 up, 12 in 2023-03-09T23:35:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31490.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:35:18.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31028.834947 secs 2023-03-09T23:35:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31028.834947 secs 2023-03-09T23:35:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31028.834947 secs 2023-03-09T23:35:19.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:19 smithi122 ceph-mon[93869]: pgmap v17061: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 583 KiB/s rd, 581 KiB/s wr, 10 op/s 2023-03-09T23:35:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:19 smithi181 ceph-mon[94189]: pgmap v17061: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 583 KiB/s rd, 581 KiB/s wr, 10 op/s 2023-03-09T23:35:19.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:19 smithi038 ceph-mon[92422]: pgmap v17061: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 583 KiB/s rd, 581 KiB/s wr, 10 op/s 2023-03-09T23:35:20.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31031.425112 secs 2023-03-09T23:35:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31031.425112 secs 2023-03-09T23:35:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31031.425112 secs 2023-03-09T23:35:21.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:21 smithi122 ceph-mon[93869]: pgmap v17062: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 519 KiB/s rd, 517 KiB/s wr, 9 op/s 2023-03-09T23:35:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:21 smithi181 ceph-mon[94189]: pgmap v17062: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 519 KiB/s rd, 517 KiB/s wr, 9 op/s 2023-03-09T23:35:21.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:21 smithi038 ceph-mon[92422]: pgmap v17062: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 519 KiB/s rd, 517 KiB/s wr, 9 op/s 2023-03-09T23:35:23.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31495.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:35:23.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31033.835084 secs 2023-03-09T23:35:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:23 smithi122 ceph-mon[93869]: pgmap v17063: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 114 KiB/s wr, 8 op/s 2023-03-09T23:35:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31033.835084 secs 2023-03-09T23:35:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:23 smithi181 ceph-mon[94189]: pgmap v17063: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 114 KiB/s wr, 8 op/s 2023-03-09T23:35:23.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31033.835084 secs 2023-03-09T23:35:23.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:23 smithi038 ceph-mon[92422]: pgmap v17063: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 114 KiB/s wr, 8 op/s 2023-03-09T23:35:25.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:25 smithi122 ceph-mon[93869]: pgmap v17064: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 113 KiB/s wr, 7 op/s 2023-03-09T23:35:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31036.425174 secs 2023-03-09T23:35:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:25 smithi181 ceph-mon[94189]: pgmap v17064: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 113 KiB/s wr, 7 op/s 2023-03-09T23:35:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31036.425174 secs 2023-03-09T23:35:25.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:25 smithi038 ceph-mon[92422]: pgmap v17064: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 113 KiB/s wr, 7 op/s 2023-03-09T23:35:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31036.425174 secs 2023-03-09T23:35:27.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:27 smithi122 ceph-mon[93869]: pgmap v17065: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 511 B/s wr, 7 op/s 2023-03-09T23:35:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:27 smithi181 ceph-mon[94189]: pgmap v17065: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 511 B/s wr, 7 op/s 2023-03-09T23:35:27.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:27 smithi038 ceph-mon[92422]: pgmap v17065: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 511 B/s wr, 7 op/s 2023-03-09T23:35:28.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31500.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:35:28.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31038.835176 secs 2023-03-09T23:35:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31038.835176 secs 2023-03-09T23:35:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31038.835176 secs 2023-03-09T23:35:29.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:29 smithi122 ceph-mon[93869]: pgmap v17066: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T23:35:29.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:29 smithi181 ceph-mon[94189]: pgmap v17066: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T23:35:29.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:29 smithi038 ceph-mon[92422]: pgmap v17066: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T23:35:30.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31041.425234 secs 2023-03-09T23:35:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31041.425234 secs 2023-03-09T23:35:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31041.425234 secs 2023-03-09T23:35:31.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:31 smithi122 ceph-mon[93869]: pgmap v17067: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T23:35:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:31 smithi181 ceph-mon[94189]: pgmap v17067: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T23:35:31.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:31 smithi038 ceph-mon[92422]: pgmap v17067: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T23:35:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31505.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:35:33.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31043.835274 secs 2023-03-09T23:35:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:33 smithi122 ceph-mon[93869]: pgmap v17068: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 8 op/s 2023-03-09T23:35:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31043.835274 secs 2023-03-09T23:35:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:33 smithi181 ceph-mon[94189]: pgmap v17068: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 8 op/s 2023-03-09T23:35:33.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31043.835274 secs 2023-03-09T23:35:33.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:33 smithi038 ceph-mon[92422]: pgmap v17068: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 8 op/s 2023-03-09T23:35:35.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:35 smithi122 ceph-mon[93869]: pgmap v17069: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T23:35:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31046.425292 secs 2023-03-09T23:35:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:35 smithi181 ceph-mon[94189]: pgmap v17069: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T23:35:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31046.425292 secs 2023-03-09T23:35:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:35 smithi038 ceph-mon[92422]: pgmap v17069: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T23:35:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31046.425292 secs 2023-03-09T23:35:37.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:37 smithi122 ceph-mon[93869]: pgmap v17070: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T23:35:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:37 smithi181 ceph-mon[94189]: pgmap v17070: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T23:35:37.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:37 smithi038 ceph-mon[92422]: pgmap v17070: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T23:35:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31510.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:35:38.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31048.835426 secs 2023-03-09T23:35:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31048.835426 secs 2023-03-09T23:35:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31048.835426 secs 2023-03-09T23:35:39.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:39 smithi122 ceph-mon[93869]: pgmap v17071: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-09T23:35:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:39 smithi181 ceph-mon[94189]: pgmap v17071: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-09T23:35:39.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:39 smithi038 ceph-mon[92422]: pgmap v17071: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-09T23:35:40.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31051.425354 secs 2023-03-09T23:35:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31051.425354 secs 2023-03-09T23:35:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31051.425354 secs 2023-03-09T23:35:41.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:41 smithi122 ceph-mon[93869]: pgmap v17072: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-09T23:35:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:41 smithi181 ceph-mon[94189]: pgmap v17072: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-09T23:35:41.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:41 smithi038 ceph-mon[92422]: pgmap v17072: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-09T23:35:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31515.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:35:43.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:43 smithi122 ceph-mon[93869]: pgmap v17073: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 9 op/s 2023-03-09T23:35:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:43 smithi181 ceph-mon[94189]: pgmap v17073: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 9 op/s 2023-03-09T23:35:43.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:43 smithi038 ceph-mon[92422]: pgmap v17073: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 9 op/s 2023-03-09T23:35:45.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:45 smithi122 ceph-mon[93869]: pgmap v17074: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 6 op/s 2023-03-09T23:35:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31056.425424 secs 2023-03-09T23:35:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:45 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:35:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:45 smithi181 ceph-mon[94189]: pgmap v17074: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 6 op/s 2023-03-09T23:35:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31056.425424 secs 2023-03-09T23:35:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:45 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:35:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:45 smithi038 ceph-mon[92422]: pgmap v17074: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 6 op/s 2023-03-09T23:35:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31056.425424 secs 2023-03-09T23:35:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:45 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:35:46.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:46 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:35:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:46 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:35:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:46 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:35:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:46 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:35:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:46 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:35:46.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:46 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:35:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:46 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:35:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:46 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:35:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:46 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:35:47.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:47 smithi122 ceph-mon[93869]: pgmap v17075: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 329 KiB/s wr, 10 op/s 2023-03-09T23:35:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:35:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:35:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:47 smithi181 ceph-mon[94189]: pgmap v17075: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 329 KiB/s wr, 10 op/s 2023-03-09T23:35:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:35:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:35:47.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:47 smithi038 ceph-mon[92422]: pgmap v17075: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 329 KiB/s wr, 10 op/s 2023-03-09T23:35:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:35:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:35:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31520.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:35:49.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:49 smithi122 ceph-mon[93869]: pgmap v17076: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 354 KiB/s wr, 8 op/s 2023-03-09T23:35:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:35:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:35:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:49 smithi181 ceph-mon[94189]: pgmap v17076: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 354 KiB/s wr, 8 op/s 2023-03-09T23:35:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:35:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:35:49.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:49 smithi038 ceph-mon[92422]: pgmap v17076: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 354 KiB/s wr, 8 op/s 2023-03-09T23:35:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:35:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:35:50.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31061.425484 secs 2023-03-09T23:35:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31061.425484 secs 2023-03-09T23:35:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31061.425484 secs 2023-03-09T23:35:51.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:51 smithi122 ceph-mon[93869]: pgmap v17077: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 354 KiB/s wr, 7 op/s 2023-03-09T23:35:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:35:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:35:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:35:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:35:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:35:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:35:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:51 smithi181 ceph-mon[94189]: pgmap v17077: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 354 KiB/s wr, 7 op/s 2023-03-09T23:35:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:35:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:35:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:35:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:35:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:35:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:35:51.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:51 smithi038 ceph-mon[92422]: pgmap v17077: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 354 KiB/s wr, 7 op/s 2023-03-09T23:35:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:35:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:35:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:35:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:35:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:35:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:35:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31525.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:35:53.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:53 smithi122 ceph-mon[93869]: pgmap v17078: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 646 KiB/s wr, 12 op/s 2023-03-09T23:35:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:53 smithi181 ceph-mon[94189]: pgmap v17078: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 646 KiB/s wr, 12 op/s 2023-03-09T23:35:53.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:53 smithi038 ceph-mon[92422]: pgmap v17078: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 646 KiB/s wr, 12 op/s 2023-03-09T23:35:55.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:35:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:35:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:55 smithi122 ceph-mon[93869]: pgmap v17079: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 660 KiB/s rd, 646 KiB/s wr, 9 op/s 2023-03-09T23:35:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31066.425543 secs 2023-03-09T23:35:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:35:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:35:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:55 smithi181 ceph-mon[94189]: pgmap v17079: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 660 KiB/s rd, 646 KiB/s wr, 9 op/s 2023-03-09T23:35:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31066.425543 secs 2023-03-09T23:35:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:35:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:35:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:55 smithi038 ceph-mon[92422]: pgmap v17079: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 660 KiB/s rd, 646 KiB/s wr, 9 op/s 2023-03-09T23:35:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31066.425543 secs 2023-03-09T23:35:57.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:35:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:35:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:57 smithi122 ceph-mon[93869]: pgmap v17080: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 14 op/s 2023-03-09T23:35:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:35:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:35:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:57 smithi181 ceph-mon[94189]: pgmap v17080: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 14 op/s 2023-03-09T23:35:57.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:35:57.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:35:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:57 smithi038 ceph-mon[92422]: pgmap v17080: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 14 op/s 2023-03-09T23:35:58.793 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31530.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:35:59.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:35:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:35:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:35:59 smithi122 ceph-mon[93869]: pgmap v17081: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 810 KiB/s rd, 797 KiB/s wr, 12 op/s 2023-03-09T23:35:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:35:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:35:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:35:59 smithi181 ceph-mon[94189]: pgmap v17081: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 810 KiB/s rd, 797 KiB/s wr, 12 op/s 2023-03-09T23:35:59.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:35:59.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:35:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:35:59 smithi038 ceph-mon[92422]: pgmap v17081: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 810 KiB/s rd, 797 KiB/s wr, 12 op/s 2023-03-09T23:36:00.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31071.425604 secs 2023-03-09T23:36:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31071.425604 secs 2023-03-09T23:36:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31071.425604 secs 2023-03-09T23:36:01.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:36:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:36:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:01 smithi122 ceph-mon[93869]: pgmap v17082: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 772 KiB/s wr, 11 op/s 2023-03-09T23:36:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:36:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:36:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:01 smithi181 ceph-mon[94189]: pgmap v17082: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 772 KiB/s wr, 11 op/s 2023-03-09T23:36:01.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:36:01.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:36:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:01 smithi038 ceph-mon[92422]: pgmap v17082: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 772 KiB/s wr, 11 op/s 2023-03-09T23:36:03.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31535.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:36:03.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31073.835916 secs 2023-03-09T23:36:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:03 smithi122 ceph-mon[93869]: pgmap v17083: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 985 KiB/s rd, 993 KiB/s wr, 19 op/s 2023-03-09T23:36:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31073.835916 secs 2023-03-09T23:36:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:03 smithi181 ceph-mon[94189]: pgmap v17083: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 985 KiB/s rd, 993 KiB/s wr, 19 op/s 2023-03-09T23:36:03.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31073.835916 secs 2023-03-09T23:36:03.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:03 smithi038 ceph-mon[92422]: pgmap v17083: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 985 KiB/s rd, 993 KiB/s wr, 19 op/s 2023-03-09T23:36:05.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:05 smithi122 ceph-mon[93869]: osdmap e1283: 12 total, 12 up, 12 in 2023-03-09T23:36:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:05 smithi122 ceph-mon[93869]: pgmap v17085: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 824 KiB/s rd, 867 KiB/s wr, 31 op/s 2023-03-09T23:36:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31076.425682 secs 2023-03-09T23:36:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:05 smithi181 ceph-mon[94189]: osdmap e1283: 12 total, 12 up, 12 in 2023-03-09T23:36:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:05 smithi181 ceph-mon[94189]: pgmap v17085: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 824 KiB/s rd, 867 KiB/s wr, 31 op/s 2023-03-09T23:36:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31076.425682 secs 2023-03-09T23:36:05.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:05 smithi038 ceph-mon[92422]: osdmap e1283: 12 total, 12 up, 12 in 2023-03-09T23:36:05.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:05 smithi038 ceph-mon[92422]: pgmap v17085: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 824 KiB/s rd, 867 KiB/s wr, 31 op/s 2023-03-09T23:36:05.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31076.425682 secs 2023-03-09T23:36:06.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:06 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:36:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:06 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:36:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:36:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:06 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:36:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:06 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:36:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:36:06.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:06 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:36:06.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:06 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:36:06.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:36:07.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:36:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:36:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:07 smithi122 ceph-mon[93869]: pgmap v17086: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 544 KiB/s wr, 30 op/s 2023-03-09T23:36:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:36:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:36:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:07 smithi181 ceph-mon[94189]: pgmap v17086: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 544 KiB/s wr, 30 op/s 2023-03-09T23:36:07.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:36:07.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:36:07.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:07 smithi038 ceph-mon[92422]: pgmap v17086: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 544 KiB/s wr, 30 op/s 2023-03-09T23:36:08.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31540.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:36:08.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31078.836031 secs 2023-03-09T23:36:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31078.836031 secs 2023-03-09T23:36:08.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31078.836031 secs 2023-03-09T23:36:09.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:36:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:36:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:09 smithi122 ceph-mon[93869]: pgmap v17087: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 494 KiB/s rd, 535 KiB/s wr, 30 op/s 2023-03-09T23:36:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:36:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:36:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:09 smithi181 ceph-mon[94189]: pgmap v17087: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 494 KiB/s rd, 535 KiB/s wr, 30 op/s 2023-03-09T23:36:09.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:36:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:36:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:09 smithi038 ceph-mon[92422]: pgmap v17087: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 494 KiB/s rd, 535 KiB/s wr, 30 op/s 2023-03-09T23:36:10.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31081.425742 secs 2023-03-09T23:36:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31081.425742 secs 2023-03-09T23:36:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31081.425742 secs 2023-03-09T23:36:11.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:11 smithi122 ceph-mon[93869]: pgmap v17088: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 655 KiB/s rd, 696 KiB/s wr, 31 op/s 2023-03-09T23:36:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:11 smithi181 ceph-mon[94189]: pgmap v17088: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 655 KiB/s rd, 696 KiB/s wr, 31 op/s 2023-03-09T23:36:11.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:11 smithi038 ceph-mon[92422]: pgmap v17088: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 655 KiB/s rd, 696 KiB/s wr, 31 op/s 2023-03-09T23:36:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31545.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:36:13.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31083.836151 secs 2023-03-09T23:36:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:13 smithi122 ceph-mon[93869]: osdmap e1284: 12 total, 12 up, 12 in 2023-03-09T23:36:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:13 smithi122 ceph-mon[93869]: pgmap v17090: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 502 KiB/s wr, 12 op/s 2023-03-09T23:36:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31083.836151 secs 2023-03-09T23:36:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:13 smithi181 ceph-mon[94189]: osdmap e1284: 12 total, 12 up, 12 in 2023-03-09T23:36:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:13 smithi181 ceph-mon[94189]: pgmap v17090: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 502 KiB/s wr, 12 op/s 2023-03-09T23:36:13.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31083.836151 secs 2023-03-09T23:36:13.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:13 smithi038 ceph-mon[92422]: osdmap e1284: 12 total, 12 up, 12 in 2023-03-09T23:36:13.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:13 smithi038 ceph-mon[92422]: pgmap v17090: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 502 KiB/s wr, 12 op/s 2023-03-09T23:36:15.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:15 smithi122 ceph-mon[93869]: pgmap v17091: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 405 KiB/s wr, 10 op/s 2023-03-09T23:36:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31086.425803 secs 2023-03-09T23:36:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:15 smithi181 ceph-mon[94189]: pgmap v17091: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 405 KiB/s wr, 10 op/s 2023-03-09T23:36:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31086.425803 secs 2023-03-09T23:36:15.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:15 smithi038 ceph-mon[92422]: pgmap v17091: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 405 KiB/s wr, 10 op/s 2023-03-09T23:36:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31086.425803 secs 2023-03-09T23:36:17.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:17 smithi122 ceph-mon[93869]: pgmap v17092: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 162 KiB/s wr, 9 op/s 2023-03-09T23:36:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:17 smithi181 ceph-mon[94189]: pgmap v17092: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 162 KiB/s wr, 9 op/s 2023-03-09T23:36:17.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:17 smithi038 ceph-mon[92422]: pgmap v17092: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 162 KiB/s wr, 9 op/s 2023-03-09T23:36:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31550.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:36:18.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31088.836229 secs 2023-03-09T23:36:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31088.836229 secs 2023-03-09T23:36:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31088.836229 secs 2023-03-09T23:36:19.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:19 smithi122 ceph-mon[93869]: pgmap v17093: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 161 KiB/s wr, 8 op/s 2023-03-09T23:36:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:19 smithi181 ceph-mon[94189]: pgmap v17093: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 161 KiB/s wr, 8 op/s 2023-03-09T23:36:19.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:19 smithi038 ceph-mon[92422]: pgmap v17093: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 161 KiB/s wr, 8 op/s 2023-03-09T23:36:20.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31091.425862 secs 2023-03-09T23:36:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31091.425862 secs 2023-03-09T23:36:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31091.425862 secs 2023-03-09T23:36:21.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:21 smithi122 ceph-mon[93869]: pgmap v17094: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 614 B/s wr, 7 op/s 2023-03-09T23:36:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:21 smithi181 ceph-mon[94189]: pgmap v17094: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 614 B/s wr, 7 op/s 2023-03-09T23:36:21.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:21 smithi038 ceph-mon[92422]: pgmap v17094: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 614 B/s wr, 7 op/s 2023-03-09T23:36:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31555.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:36:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31093.836310 secs 2023-03-09T23:36:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:23 smithi122 ceph-mon[93869]: pgmap v17095: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T23:36:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31093.836310 secs 2023-03-09T23:36:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:23 smithi181 ceph-mon[94189]: pgmap v17095: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T23:36:23.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31093.836310 secs 2023-03-09T23:36:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:23 smithi038 ceph-mon[92422]: pgmap v17095: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T23:36:25.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:25 smithi122 ceph-mon[93869]: pgmap v17096: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T23:36:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31096.425947 secs 2023-03-09T23:36:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:25 smithi181 ceph-mon[94189]: pgmap v17096: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T23:36:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31096.425947 secs 2023-03-09T23:36:25.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:25 smithi038 ceph-mon[92422]: pgmap v17096: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T23:36:25.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31096.425947 secs 2023-03-09T23:36:27.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:27 smithi122 ceph-mon[93869]: pgmap v17097: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 8 op/s 2023-03-09T23:36:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:27 smithi181 ceph-mon[94189]: pgmap v17097: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 8 op/s 2023-03-09T23:36:27.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:27 smithi038 ceph-mon[92422]: pgmap v17097: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 8 op/s 2023-03-09T23:36:28.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31560.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:36:28.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31098.836407 secs 2023-03-09T23:36:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31098.836407 secs 2023-03-09T23:36:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31098.836407 secs 2023-03-09T23:36:29.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:29 smithi122 ceph-mon[93869]: pgmap v17098: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T23:36:29.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:29 smithi181 ceph-mon[94189]: pgmap v17098: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T23:36:29.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:29 smithi038 ceph-mon[92422]: pgmap v17098: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T23:36:30.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31101.426006 secs 2023-03-09T23:36:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31101.426006 secs 2023-03-09T23:36:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31101.426006 secs 2023-03-09T23:36:31.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:31 smithi122 ceph-mon[93869]: pgmap v17099: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T23:36:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:31 smithi181 ceph-mon[94189]: pgmap v17099: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T23:36:31.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:31 smithi038 ceph-mon[92422]: pgmap v17099: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T23:36:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31565.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:36:33.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31103.836523 secs 2023-03-09T23:36:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:33 smithi122 ceph-mon[93869]: pgmap v17100: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 8 op/s 2023-03-09T23:36:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31103.836523 secs 2023-03-09T23:36:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:33 smithi181 ceph-mon[94189]: pgmap v17100: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 8 op/s 2023-03-09T23:36:33.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31103.836523 secs 2023-03-09T23:36:33.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:33 smithi038 ceph-mon[92422]: pgmap v17100: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 8 op/s 2023-03-09T23:36:35.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:35 smithi122 ceph-mon[93869]: pgmap v17101: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-09T23:36:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31106.426068 secs 2023-03-09T23:36:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:35 smithi181 ceph-mon[94189]: pgmap v17101: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-09T23:36:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31106.426068 secs 2023-03-09T23:36:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:35 smithi038 ceph-mon[92422]: pgmap v17101: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-09T23:36:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31106.426068 secs 2023-03-09T23:36:37.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:37 smithi122 ceph-mon[93869]: pgmap v17102: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 115 KiB/s wr, 10 op/s 2023-03-09T23:36:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:37 smithi181 ceph-mon[94189]: pgmap v17102: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 115 KiB/s wr, 10 op/s 2023-03-09T23:36:37.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:37 smithi038 ceph-mon[92422]: pgmap v17102: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 115 KiB/s wr, 10 op/s 2023-03-09T23:36:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31570.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:36:38.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31108.836640 secs 2023-03-09T23:36:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31108.836640 secs 2023-03-09T23:36:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31108.836640 secs 2023-03-09T23:36:39.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:39 smithi122 ceph-mon[93869]: pgmap v17103: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 115 KiB/s wr, 7 op/s 2023-03-09T23:36:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:39 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:36:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:39 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:36:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:39 smithi181 ceph-mon[94189]: pgmap v17103: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 115 KiB/s wr, 7 op/s 2023-03-09T23:36:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:39 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:36:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:39 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:36:39.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:39 smithi038 ceph-mon[92422]: pgmap v17103: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 115 KiB/s wr, 7 op/s 2023-03-09T23:36:39.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:39 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:36:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:39 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:36:40.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31111.426131 secs 2023-03-09T23:36:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31111.426131 secs 2023-03-09T23:36:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31111.426131 secs 2023-03-09T23:36:41.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:41 smithi122 ceph-mon[93869]: pgmap v17104: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 115 KiB/s wr, 8 op/s 2023-03-09T23:36:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:36:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:36:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:41 smithi181 ceph-mon[94189]: pgmap v17104: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 115 KiB/s wr, 8 op/s 2023-03-09T23:36:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:36:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:36:41.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:41 smithi038 ceph-mon[92422]: pgmap v17104: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 115 KiB/s wr, 8 op/s 2023-03-09T23:36:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:36:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:36:43.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31575.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:36:43.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:43 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 31113.836774 secs 2023-03-09T23:36:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:43 smithi122 ceph-mon[93869]: pgmap v17105: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 252 KiB/s wr, 10 op/s 2023-03-09T23:36:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:43 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 31113.836774 secs 2023-03-09T23:36:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:43 smithi181 ceph-mon[94189]: pgmap v17105: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 252 KiB/s wr, 10 op/s 2023-03-09T23:36:43.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:43 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 31113.836774 secs 2023-03-09T23:36:43.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:43 smithi038 ceph-mon[92422]: pgmap v17105: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 252 KiB/s wr, 10 op/s 2023-03-09T23:36:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:45 smithi122 ceph-mon[93869]: pgmap v17106: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 252 KiB/s wr, 7 op/s 2023-03-09T23:36:45.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31116.426191 secs 2023-03-09T23:36:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:45 smithi181 ceph-mon[94189]: pgmap v17106: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 252 KiB/s wr, 7 op/s 2023-03-09T23:36:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31116.426191 secs 2023-03-09T23:36:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:45 smithi038 ceph-mon[92422]: pgmap v17106: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 252 KiB/s wr, 7 op/s 2023-03-09T23:36:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31116.426191 secs 2023-03-09T23:36:46.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:46 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:36:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:46 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:36:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:46 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:36:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:46 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:36:46.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:46 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:36:46.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:46 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:36:47.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:47 smithi122 ceph-mon[93869]: pgmap v17107: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 601 KiB/s rd, 427 KiB/s wr, 10 op/s 2023-03-09T23:36:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:47 smithi181 ceph-mon[94189]: pgmap v17107: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 601 KiB/s rd, 427 KiB/s wr, 10 op/s 2023-03-09T23:36:47.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:47 smithi038 ceph-mon[92422]: pgmap v17107: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 601 KiB/s rd, 427 KiB/s wr, 10 op/s 2023-03-09T23:36:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31580.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:36:48.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:48 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 31118.836918 secs 2023-03-09T23:36:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:48 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 31118.836918 secs 2023-03-09T23:36:48.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:48 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 31118.836918 secs 2023-03-09T23:36:49.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:49 smithi122 ceph-mon[93869]: pgmap v17108: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 313 KiB/s wr, 7 op/s 2023-03-09T23:36:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:49 smithi181 ceph-mon[94189]: pgmap v17108: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 313 KiB/s wr, 7 op/s 2023-03-09T23:36:49.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:49 smithi038 ceph-mon[92422]: pgmap v17108: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 313 KiB/s wr, 7 op/s 2023-03-09T23:36:50.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31121.426252 secs 2023-03-09T23:36:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31121.426252 secs 2023-03-09T23:36:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31121.426252 secs 2023-03-09T23:36:51.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:51 smithi122 ceph-mon[93869]: pgmap v17109: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 313 KiB/s wr, 7 op/s 2023-03-09T23:36:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:36:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:51 smithi181 ceph-mon[94189]: pgmap v17109: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 313 KiB/s wr, 7 op/s 2023-03-09T23:36:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:36:51.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:51 smithi038 ceph-mon[92422]: pgmap v17109: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 313 KiB/s wr, 7 op/s 2023-03-09T23:36:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:36:52.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:36:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:36:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:36:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:36:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:36:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:36:52.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:36:52.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:36:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:36:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31585.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:36:53.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:53 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 31123.837047 secs 2023-03-09T23:36:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:53 smithi122 ceph-mon[93869]: pgmap v17110: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 831 KiB/s rd, 716 KiB/s wr, 11 op/s 2023-03-09T23:36:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:36:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:36:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:53 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 31123.837047 secs 2023-03-09T23:36:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:53 smithi181 ceph-mon[94189]: pgmap v17110: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 831 KiB/s rd, 716 KiB/s wr, 11 op/s 2023-03-09T23:36:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:36:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:36:53.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:53 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 31123.837047 secs 2023-03-09T23:36:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:53 smithi038 ceph-mon[92422]: pgmap v17110: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 831 KiB/s rd, 716 KiB/s wr, 11 op/s 2023-03-09T23:36:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:36:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:36:55.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:55 smithi122 ceph-mon[93869]: pgmap v17111: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 693 KiB/s rd, 579 KiB/s wr, 8 op/s 2023-03-09T23:36:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:36:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:36:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31126.426311 secs 2023-03-09T23:36:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:55 smithi181 ceph-mon[94189]: pgmap v17111: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 693 KiB/s rd, 579 KiB/s wr, 8 op/s 2023-03-09T23:36:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:36:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:36:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31126.426311 secs 2023-03-09T23:36:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:55 smithi038 ceph-mon[92422]: pgmap v17111: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 693 KiB/s rd, 579 KiB/s wr, 8 op/s 2023-03-09T23:36:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:36:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:36:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31126.426311 secs 2023-03-09T23:36:57.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:57 smithi038 ceph-mon[92422]: pgmap v17112: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 721 KiB/s rd, 604 KiB/s wr, 11 op/s 2023-03-09T23:36:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:36:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:36:58.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:57 smithi122 ceph-mon[93869]: pgmap v17112: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 721 KiB/s rd, 604 KiB/s wr, 11 op/s 2023-03-09T23:36:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:36:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:36:58.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:57 smithi181 ceph-mon[94189]: pgmap v17112: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 721 KiB/s rd, 604 KiB/s wr, 11 op/s 2023-03-09T23:36:58.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:36:58.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:36:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31590.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:36:58.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31128.837165 secs 2023-03-09T23:36:59.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31128.837165 secs 2023-03-09T23:36:59.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31128.837165 secs 2023-03-09T23:36:59.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:59 smithi038 ceph-mon[92422]: pgmap v17113: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 691 KiB/s rd, 576 KiB/s wr, 10 op/s 2023-03-09T23:36:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:36:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:36:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:37:00.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:59 smithi122 ceph-mon[93869]: pgmap v17113: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 691 KiB/s rd, 576 KiB/s wr, 10 op/s 2023-03-09T23:37:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:37:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:36:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:37:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:59 smithi181 ceph-mon[94189]: pgmap v17113: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 691 KiB/s rd, 576 KiB/s wr, 10 op/s 2023-03-09T23:37:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:37:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:36:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:37:01.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:37:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:37:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31131.426371 secs 2023-03-09T23:37:01.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:37:01.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:37:01.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31131.426371 secs 2023-03-09T23:37:01.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:37:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:37:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31131.426371 secs 2023-03-09T23:37:02.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:01 smithi122 ceph-mon[93869]: pgmap v17114: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 702 KiB/s rd, 577 KiB/s wr, 10 op/s 2023-03-09T23:37:02.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:01 smithi181 ceph-mon[94189]: pgmap v17114: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 702 KiB/s rd, 577 KiB/s wr, 10 op/s 2023-03-09T23:37:02.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:01 smithi038 ceph-mon[92422]: pgmap v17114: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 702 KiB/s rd, 577 KiB/s wr, 10 op/s 2023-03-09T23:37:03.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:02 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:37:03.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:02 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:37:03.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:02 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:37:03.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:02 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:37:03.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:02 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:37:03.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:02 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:37:03.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31595.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:37:04.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31133.837281 secs 2023-03-09T23:37:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:03 smithi122 ceph-mon[93869]: pgmap v17115: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 995 KiB/s rd, 869 KiB/s wr, 15 op/s 2023-03-09T23:37:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31133.837281 secs 2023-03-09T23:37:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:03 smithi181 ceph-mon[94189]: pgmap v17115: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 995 KiB/s rd, 869 KiB/s wr, 15 op/s 2023-03-09T23:37:04.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31133.837281 secs 2023-03-09T23:37:04.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:03 smithi038 ceph-mon[92422]: pgmap v17115: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 995 KiB/s rd, 869 KiB/s wr, 15 op/s 2023-03-09T23:37:06.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:05 smithi122 ceph-mon[93869]: pgmap v17116: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 734 KiB/s rd, 719 KiB/s wr, 13 op/s 2023-03-09T23:37:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31136.426431 secs 2023-03-09T23:37:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:05 smithi181 ceph-mon[94189]: pgmap v17116: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 734 KiB/s rd, 719 KiB/s wr, 13 op/s 2023-03-09T23:37:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31136.426431 secs 2023-03-09T23:37:06.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:05 smithi038 ceph-mon[92422]: pgmap v17116: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 734 KiB/s rd, 719 KiB/s wr, 13 op/s 2023-03-09T23:37:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31136.426431 secs 2023-03-09T23:37:08.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:07 smithi122 ceph-mon[93869]: pgmap v17117: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 761 KiB/s rd, 767 KiB/s wr, 19 op/s 2023-03-09T23:37:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:37:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:37:08.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:07 smithi181 ceph-mon[94189]: pgmap v17117: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 761 KiB/s rd, 767 KiB/s wr, 19 op/s 2023-03-09T23:37:08.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:37:08.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:37:08.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:07 smithi038 ceph-mon[92422]: pgmap v17117: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 761 KiB/s rd, 767 KiB/s wr, 19 op/s 2023-03-09T23:37:08.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:37:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:37:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31600.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:37:09.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31138.837395 secs 2023-03-09T23:37:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:37:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:37:09.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31138.837395 secs 2023-03-09T23:37:09.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:37:09.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:37:09.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31138.837395 secs 2023-03-09T23:37:09.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:37:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:37:10.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:09 smithi122 ceph-mon[93869]: pgmap v17118: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 905 KiB/s rd, 930 KiB/s wr, 29 op/s 2023-03-09T23:37:10.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:09 smithi181 ceph-mon[94189]: pgmap v17118: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 905 KiB/s rd, 930 KiB/s wr, 29 op/s 2023-03-09T23:37:10.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:09 smithi038 ceph-mon[92422]: pgmap v17118: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 905 KiB/s rd, 930 KiB/s wr, 29 op/s 2023-03-09T23:37:11.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:10 smithi122 ceph-mon[93869]: osdmap e1285: 12 total, 12 up, 12 in 2023-03-09T23:37:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31141.427481 secs 2023-03-09T23:37:11.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:10 smithi181 ceph-mon[94189]: osdmap e1285: 12 total, 12 up, 12 in 2023-03-09T23:37:11.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31141.427481 secs 2023-03-09T23:37:11.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:10 smithi038 ceph-mon[92422]: osdmap e1285: 12 total, 12 up, 12 in 2023-03-09T23:37:11.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31141.427481 secs 2023-03-09T23:37:12.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:11 smithi122 ceph-mon[93869]: pgmap v17120: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 897 KiB/s rd, 938 KiB/s wr, 32 op/s 2023-03-09T23:37:12.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:11 smithi181 ceph-mon[94189]: pgmap v17120: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 897 KiB/s rd, 938 KiB/s wr, 32 op/s 2023-03-09T23:37:12.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:11 smithi038 ceph-mon[92422]: pgmap v17120: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 897 KiB/s rd, 938 KiB/s wr, 32 op/s 2023-03-09T23:37:13.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31605.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:37:13.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31143.837505 secs 2023-03-09T23:37:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:13 smithi122 ceph-mon[93869]: pgmap v17121: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 777 KiB/s rd, 818 KiB/s wr, 29 op/s 2023-03-09T23:37:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31143.837505 secs 2023-03-09T23:37:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:13 smithi181 ceph-mon[94189]: pgmap v17121: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 777 KiB/s rd, 818 KiB/s wr, 29 op/s 2023-03-09T23:37:13.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31143.837505 secs 2023-03-09T23:37:13.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:13 smithi038 ceph-mon[92422]: pgmap v17121: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 777 KiB/s rd, 818 KiB/s wr, 29 op/s 2023-03-09T23:37:15.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:15 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:37:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:15 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:37:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:15 smithi122 ceph-mon[93869]: pgmap v17122: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 633 KiB/s rd, 675 KiB/s wr, 27 op/s 2023-03-09T23:37:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31146.427557 secs 2023-03-09T23:37:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:15 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:37:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:15 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:37:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:15 smithi181 ceph-mon[94189]: pgmap v17122: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 633 KiB/s rd, 675 KiB/s wr, 27 op/s 2023-03-09T23:37:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31146.427557 secs 2023-03-09T23:37:15.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:15 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:37:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:15 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:37:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:15 smithi038 ceph-mon[92422]: pgmap v17122: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 633 KiB/s rd, 675 KiB/s wr, 27 op/s 2023-03-09T23:37:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31146.427557 secs 2023-03-09T23:37:17.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:37:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:37:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:17 smithi122 ceph-mon[93869]: pgmap v17123: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 602 KiB/s rd, 618 KiB/s wr, 24 op/s 2023-03-09T23:37:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:17 smithi122 ceph-mon[93869]: osdmap e1286: 12 total, 12 up, 12 in 2023-03-09T23:37:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:37:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:37:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:17 smithi181 ceph-mon[94189]: pgmap v17123: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 602 KiB/s rd, 618 KiB/s wr, 24 op/s 2023-03-09T23:37:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:17 smithi181 ceph-mon[94189]: osdmap e1286: 12 total, 12 up, 12 in 2023-03-09T23:37:17.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:37:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:37:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:17 smithi038 ceph-mon[92422]: pgmap v17123: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 602 KiB/s rd, 618 KiB/s wr, 24 op/s 2023-03-09T23:37:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:17 smithi038 ceph-mon[92422]: osdmap e1286: 12 total, 12 up, 12 in 2023-03-09T23:37:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31610.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:37:18.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31148.837613 secs 2023-03-09T23:37:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31148.837613 secs 2023-03-09T23:37:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31148.837613 secs 2023-03-09T23:37:19.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:19 smithi122 ceph-mon[93869]: pgmap v17125: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 580 KiB/s rd, 577 KiB/s wr, 10 op/s 2023-03-09T23:37:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:19 smithi181 ceph-mon[94189]: pgmap v17125: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 580 KiB/s rd, 577 KiB/s wr, 10 op/s 2023-03-09T23:37:19.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:19 smithi038 ceph-mon[92422]: pgmap v17125: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 580 KiB/s rd, 577 KiB/s wr, 10 op/s 2023-03-09T23:37:20.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31151.427617 secs 2023-03-09T23:37:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31151.427617 secs 2023-03-09T23:37:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31151.427617 secs 2023-03-09T23:37:21.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:21 smithi122 ceph-mon[93869]: pgmap v17126: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 508 KiB/s rd, 506 KiB/s wr, 9 op/s 2023-03-09T23:37:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:21 smithi181 ceph-mon[94189]: pgmap v17126: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 508 KiB/s rd, 506 KiB/s wr, 9 op/s 2023-03-09T23:37:21.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:21 smithi038 ceph-mon[92422]: pgmap v17126: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 508 KiB/s rd, 506 KiB/s wr, 9 op/s 2023-03-09T23:37:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31615.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:37:23.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31153.837729 secs 2023-03-09T23:37:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:23 smithi122 ceph-mon[93869]: pgmap v17127: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 275 KiB/s wr, 8 op/s 2023-03-09T23:37:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31153.837729 secs 2023-03-09T23:37:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:23 smithi181 ceph-mon[94189]: pgmap v17127: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 275 KiB/s wr, 8 op/s 2023-03-09T23:37:23.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31153.837729 secs 2023-03-09T23:37:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:23 smithi038 ceph-mon[92422]: pgmap v17127: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 275 KiB/s wr, 8 op/s 2023-03-09T23:37:25.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:25 smithi122 ceph-mon[93869]: pgmap v17128: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 114 KiB/s wr, 7 op/s 2023-03-09T23:37:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31156.427677 secs 2023-03-09T23:37:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:25 smithi181 ceph-mon[94189]: pgmap v17128: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 114 KiB/s wr, 7 op/s 2023-03-09T23:37:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31156.427677 secs 2023-03-09T23:37:25.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:25 smithi038 ceph-mon[92422]: pgmap v17128: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 114 KiB/s wr, 7 op/s 2023-03-09T23:37:25.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31156.427677 secs 2023-03-09T23:37:27.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:27 smithi122 ceph-mon[93869]: pgmap v17129: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 114 KiB/s wr, 7 op/s 2023-03-09T23:37:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:27 smithi181 ceph-mon[94189]: pgmap v17129: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 114 KiB/s wr, 7 op/s 2023-03-09T23:37:27.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:27 smithi038 ceph-mon[92422]: pgmap v17129: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 114 KiB/s wr, 7 op/s 2023-03-09T23:37:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31620.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:37:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31158.837850 secs 2023-03-09T23:37:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31158.837850 secs 2023-03-09T23:37:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31158.837850 secs 2023-03-09T23:37:29.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:29 smithi122 ceph-mon[93869]: pgmap v17130: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 103 KiB/s wr, 7 op/s 2023-03-09T23:37:29.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:29 smithi181 ceph-mon[94189]: pgmap v17130: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 103 KiB/s wr, 7 op/s 2023-03-09T23:37:29.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:29 smithi038 ceph-mon[92422]: pgmap v17130: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 103 KiB/s wr, 7 op/s 2023-03-09T23:37:30.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31161.427736 secs 2023-03-09T23:37:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31161.427736 secs 2023-03-09T23:37:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31161.427736 secs 2023-03-09T23:37:31.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:31 smithi122 ceph-mon[93869]: pgmap v17131: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T23:37:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:31 smithi181 ceph-mon[94189]: pgmap v17131: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T23:37:31.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:31 smithi038 ceph-mon[92422]: pgmap v17131: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T23:37:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31625.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:37:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31163.837992 secs 2023-03-09T23:37:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:33 smithi122 ceph-mon[93869]: pgmap v17132: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T23:37:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31163.837992 secs 2023-03-09T23:37:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:33 smithi181 ceph-mon[94189]: pgmap v17132: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T23:37:33.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31163.837992 secs 2023-03-09T23:37:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:33 smithi038 ceph-mon[92422]: pgmap v17132: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T23:37:35.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:35 smithi122 ceph-mon[93869]: pgmap v17133: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T23:37:35.980 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31166.427794 secs 2023-03-09T23:37:35.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:35 smithi181 ceph-mon[94189]: pgmap v17133: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T23:37:35.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31166.427794 secs 2023-03-09T23:37:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:35 smithi038 ceph-mon[92422]: pgmap v17133: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T23:37:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31166.427794 secs 2023-03-09T23:37:37.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:37 smithi122 ceph-mon[93869]: pgmap v17134: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T23:37:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:37 smithi181 ceph-mon[94189]: pgmap v17134: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T23:37:37.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:37 smithi038 ceph-mon[92422]: pgmap v17134: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T23:37:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31630.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:37:38.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31168.838124 secs 2023-03-09T23:37:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31168.838124 secs 2023-03-09T23:37:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31168.838124 secs 2023-03-09T23:37:39.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:39 smithi122 ceph-mon[93869]: pgmap v17135: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 7 op/s 2023-03-09T23:37:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:39 smithi181 ceph-mon[94189]: pgmap v17135: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 7 op/s 2023-03-09T23:37:39.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:39 smithi038 ceph-mon[92422]: pgmap v17135: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 7 op/s 2023-03-09T23:37:40.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31171.427854 secs 2023-03-09T23:37:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31171.427854 secs 2023-03-09T23:37:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31171.427854 secs 2023-03-09T23:37:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:41 smithi122 ceph-mon[93869]: pgmap v17136: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 7 op/s 2023-03-09T23:37:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:41 smithi181 ceph-mon[94189]: pgmap v17136: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 7 op/s 2023-03-09T23:37:41.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:41 smithi038 ceph-mon[92422]: pgmap v17136: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 7 op/s 2023-03-09T23:37:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31635.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:37:43.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31173.838242 secs 2023-03-09T23:37:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:43 smithi122 ceph-mon[93869]: pgmap v17137: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 597 B/s wr, 9 op/s 2023-03-09T23:37:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31173.838242 secs 2023-03-09T23:37:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:43 smithi181 ceph-mon[94189]: pgmap v17137: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 597 B/s wr, 9 op/s 2023-03-09T23:37:43.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31173.838242 secs 2023-03-09T23:37:43.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:43 smithi038 ceph-mon[92422]: pgmap v17137: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 597 B/s wr, 9 op/s 2023-03-09T23:37:45.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:45 smithi122 ceph-mon[93869]: pgmap v17138: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 101 KiB/s wr, 7 op/s 2023-03-09T23:37:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31176.427929 secs 2023-03-09T23:37:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:45 smithi181 ceph-mon[94189]: pgmap v17138: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 101 KiB/s wr, 7 op/s 2023-03-09T23:37:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31176.427929 secs 2023-03-09T23:37:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:45 smithi038 ceph-mon[92422]: pgmap v17138: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 101 KiB/s wr, 7 op/s 2023-03-09T23:37:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31176.427929 secs 2023-03-09T23:37:47.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:47 smithi122 ceph-mon[93869]: pgmap v17139: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 102 KiB/s wr, 9 op/s 2023-03-09T23:37:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:47 smithi181 ceph-mon[94189]: pgmap v17139: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 102 KiB/s wr, 9 op/s 2023-03-09T23:37:47.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:47 smithi038 ceph-mon[92422]: pgmap v17139: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 102 KiB/s wr, 9 op/s 2023-03-09T23:37:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31640.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:37:48.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31178.838359 secs 2023-03-09T23:37:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31178.838359 secs 2023-03-09T23:37:48.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31178.838359 secs 2023-03-09T23:37:49.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:49 smithi122 ceph-mon[93869]: pgmap v17140: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 507 KiB/s rd, 102 KiB/s wr, 8 op/s 2023-03-09T23:37:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:49 smithi181 ceph-mon[94189]: pgmap v17140: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 507 KiB/s rd, 102 KiB/s wr, 8 op/s 2023-03-09T23:37:49.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:49 smithi038 ceph-mon[92422]: pgmap v17140: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 507 KiB/s rd, 102 KiB/s wr, 8 op/s 2023-03-09T23:37:50.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31181.427984 secs 2023-03-09T23:37:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31181.427984 secs 2023-03-09T23:37:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31181.427984 secs 2023-03-09T23:37:51.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:51 smithi122 ceph-mon[93869]: pgmap v17141: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 861 KiB/s rd, 443 KiB/s wr, 9 op/s 2023-03-09T23:37:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:37:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:37:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:51 smithi181 ceph-mon[94189]: pgmap v17141: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 861 KiB/s rd, 443 KiB/s wr, 9 op/s 2023-03-09T23:37:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:37:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:37:51.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:51 smithi038 ceph-mon[92422]: pgmap v17141: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 861 KiB/s rd, 443 KiB/s wr, 9 op/s 2023-03-09T23:37:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:37:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:37:52.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:37:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:37:52.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:37:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31645.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:37:53.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:53 smithi122 ceph-mon[93869]: pgmap v17142: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 813 KiB/s wr, 11 op/s 2023-03-09T23:37:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:37:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:37:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:37:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:37:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:37:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:53 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:37:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:53 smithi181 ceph-mon[94189]: pgmap v17142: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 813 KiB/s wr, 11 op/s 2023-03-09T23:37:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:37:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:37:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:37:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:37:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:37:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:53 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:37:53.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:53 smithi038 ceph-mon[92422]: pgmap v17142: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 813 KiB/s wr, 11 op/s 2023-03-09T23:37:53.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:37:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:37:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:37:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:37:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:37:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:53 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:37:54.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:54 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:37:54.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:54 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:37:54.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:54 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:37:54.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:54 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:37:54.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:54 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:37:54.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:54 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:37:54.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:54 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:37:54.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:54 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:37:54.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:54 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:37:55.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:55 smithi122 ceph-mon[93869]: pgmap v17143: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 896 KiB/s wr, 11 op/s 2023-03-09T23:37:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31186.428044 secs 2023-03-09T23:37:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:55 smithi181 ceph-mon[94189]: pgmap v17143: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 896 KiB/s wr, 11 op/s 2023-03-09T23:37:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31186.428044 secs 2023-03-09T23:37:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:55 smithi038 ceph-mon[92422]: pgmap v17143: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 896 KiB/s wr, 11 op/s 2023-03-09T23:37:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31186.428044 secs 2023-03-09T23:37:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:57 smithi181 ceph-mon[94189]: pgmap v17144: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 821 KiB/s wr, 13 op/s 2023-03-09T23:37:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:37:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:37:57.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:57 smithi038 ceph-mon[92422]: pgmap v17144: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 821 KiB/s wr, 13 op/s 2023-03-09T23:37:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:37:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:37:58.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:57 smithi122 ceph-mon[93869]: pgmap v17144: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 821 KiB/s wr, 13 op/s 2023-03-09T23:37:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:37:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:37:58.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31650.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:37:59.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:59 smithi038 ceph-mon[92422]: pgmap v17145: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 822 KiB/s wr, 12 op/s 2023-03-09T23:37:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:37:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:37:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:38:00.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:59 smithi122 ceph-mon[93869]: pgmap v17145: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 822 KiB/s wr, 12 op/s 2023-03-09T23:38:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:38:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:37:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:38:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:59 smithi181 ceph-mon[94189]: pgmap v17145: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 822 KiB/s wr, 12 op/s 2023-03-09T23:38:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:38:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:37:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:38:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31191.428107 secs 2023-03-09T23:38:01.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31191.428107 secs 2023-03-09T23:38:01.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31191.428107 secs 2023-03-09T23:38:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:01 smithi038 ceph-mon[92422]: pgmap v17146: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 14 op/s 2023-03-09T23:38:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:38:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:38:02.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:01 smithi122 ceph-mon[93869]: pgmap v17146: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 14 op/s 2023-03-09T23:38:02.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:38:02.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:38:02.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:01 smithi181 ceph-mon[94189]: pgmap v17146: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 14 op/s 2023-03-09T23:38:02.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:38:02.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:38:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31655.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:38:03.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:03 smithi122 ceph-mon[93869]: pgmap v17147: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 15 op/s 2023-03-09T23:38:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:03 smithi181 ceph-mon[94189]: pgmap v17147: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 15 op/s 2023-03-09T23:38:03.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:03 smithi038 ceph-mon[92422]: pgmap v17147: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 15 op/s 2023-03-09T23:38:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:05 smithi122 ceph-mon[93869]: pgmap v17148: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 917 KiB/s rd, 900 KiB/s wr, 15 op/s 2023-03-09T23:38:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31196.428191 secs 2023-03-09T23:38:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:05 smithi181 ceph-mon[94189]: pgmap v17148: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 917 KiB/s rd, 900 KiB/s wr, 15 op/s 2023-03-09T23:38:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31196.428191 secs 2023-03-09T23:38:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:05 smithi038 ceph-mon[92422]: pgmap v17148: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 917 KiB/s rd, 900 KiB/s wr, 15 op/s 2023-03-09T23:38:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31196.428191 secs 2023-03-09T23:38:07.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:07 smithi122 ceph-mon[93869]: pgmap v17149: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 861 KiB/s rd, 864 KiB/s wr, 18 op/s 2023-03-09T23:38:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:38:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:38:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:07 smithi181 ceph-mon[94189]: pgmap v17149: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 861 KiB/s rd, 864 KiB/s wr, 18 op/s 2023-03-09T23:38:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:38:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:38:07.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:07 smithi038 ceph-mon[92422]: pgmap v17149: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 861 KiB/s rd, 864 KiB/s wr, 18 op/s 2023-03-09T23:38:07.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:38:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:38:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31660.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:38:08.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31198.838819 secs 2023-03-09T23:38:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31198.838819 secs 2023-03-09T23:38:08.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31198.838819 secs 2023-03-09T23:38:09.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:09 smithi122 ceph-mon[93869]: pgmap v17150: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 841 KiB/s rd, 860 KiB/s wr, 26 op/s 2023-03-09T23:38:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:38:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:38:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:09 smithi181 ceph-mon[94189]: pgmap v17150: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 841 KiB/s rd, 860 KiB/s wr, 26 op/s 2023-03-09T23:38:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:38:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:38:09.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:09 smithi038 ceph-mon[92422]: pgmap v17150: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 841 KiB/s rd, 860 KiB/s wr, 26 op/s 2023-03-09T23:38:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:38:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:38:10.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:10 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:38:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:10 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:38:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:10 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:38:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:10 smithi122 ceph-mon[93869]: osdmap e1287: 12 total, 12 up, 12 in 2023-03-09T23:38:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31201.428252 secs 2023-03-09T23:38:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:10 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:38:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:10 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:38:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:10 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:38:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:10 smithi181 ceph-mon[94189]: osdmap e1287: 12 total, 12 up, 12 in 2023-03-09T23:38:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31201.428252 secs 2023-03-09T23:38:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:10 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:38:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:10 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:38:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:10 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:38:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:10 smithi038 ceph-mon[92422]: osdmap e1287: 12 total, 12 up, 12 in 2023-03-09T23:38:11.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31201.428252 secs 2023-03-09T23:38:11.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:11 smithi122 ceph-mon[93869]: pgmap v17152: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 898 KiB/s rd, 920 KiB/s wr, 30 op/s 2023-03-09T23:38:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:11 smithi181 ceph-mon[94189]: pgmap v17152: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 898 KiB/s rd, 920 KiB/s wr, 30 op/s 2023-03-09T23:38:11.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:11 smithi038 ceph-mon[92422]: pgmap v17152: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 898 KiB/s rd, 920 KiB/s wr, 30 op/s 2023-03-09T23:38:13.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31665.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:38:13.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31203.838935 secs 2023-03-09T23:38:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:13 smithi122 ceph-mon[93869]: pgmap v17153: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 835 KiB/s rd, 856 KiB/s wr, 29 op/s 2023-03-09T23:38:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31203.838935 secs 2023-03-09T23:38:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:13 smithi181 ceph-mon[94189]: pgmap v17153: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 835 KiB/s rd, 856 KiB/s wr, 29 op/s 2023-03-09T23:38:13.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31203.838935 secs 2023-03-09T23:38:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:13 smithi038 ceph-mon[92422]: pgmap v17153: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 835 KiB/s rd, 856 KiB/s wr, 29 op/s 2023-03-09T23:38:15.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:15 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:38:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:15 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:38:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:15 smithi122 ceph-mon[93869]: pgmap v17154: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 650 KiB/s rd, 692 KiB/s wr, 26 op/s 2023-03-09T23:38:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31206.428310 secs 2023-03-09T23:38:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:15 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:38:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:15 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:38:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:15 smithi181 ceph-mon[94189]: pgmap v17154: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 650 KiB/s rd, 692 KiB/s wr, 26 op/s 2023-03-09T23:38:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31206.428310 secs 2023-03-09T23:38:15.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:15 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:38:15.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:15 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:38:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:15 smithi038 ceph-mon[92422]: pgmap v17154: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 650 KiB/s rd, 692 KiB/s wr, 26 op/s 2023-03-09T23:38:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31206.428310 secs 2023-03-09T23:38:17.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:38:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:38:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:17 smithi122 ceph-mon[93869]: pgmap v17155: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 620 KiB/s rd, 636 KiB/s wr, 24 op/s 2023-03-09T23:38:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:38:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:38:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:17 smithi181 ceph-mon[94189]: pgmap v17155: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 620 KiB/s rd, 636 KiB/s wr, 24 op/s 2023-03-09T23:38:17.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:38:17.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:38:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:17 smithi038 ceph-mon[92422]: pgmap v17155: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 620 KiB/s rd, 636 KiB/s wr, 24 op/s 2023-03-09T23:38:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31670.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:38:18.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31208.839050 secs 2023-03-09T23:38:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31208.839050 secs 2023-03-09T23:38:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31208.839050 secs 2023-03-09T23:38:19.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:19 smithi122 ceph-mon[93869]: pgmap v17156: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 612 KiB/s rd, 610 KiB/s wr, 11 op/s 2023-03-09T23:38:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:19 smithi181 ceph-mon[94189]: pgmap v17156: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 612 KiB/s rd, 610 KiB/s wr, 11 op/s 2023-03-09T23:38:19.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:19 smithi038 ceph-mon[92422]: pgmap v17156: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 612 KiB/s rd, 610 KiB/s wr, 11 op/s 2023-03-09T23:38:20.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31211.428370 secs 2023-03-09T23:38:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31211.428370 secs 2023-03-09T23:38:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31211.428370 secs 2023-03-09T23:38:21.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:21 smithi122 ceph-mon[93869]: pgmap v17157: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 377 KiB/s wr, 10 op/s 2023-03-09T23:38:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:21 smithi181 ceph-mon[94189]: pgmap v17157: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 377 KiB/s wr, 10 op/s 2023-03-09T23:38:21.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:21 smithi038 ceph-mon[92422]: pgmap v17157: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 377 KiB/s wr, 10 op/s 2023-03-09T23:38:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31675.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:38:23.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31213.839175 secs 2023-03-09T23:38:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:23 smithi122 ceph-mon[93869]: osdmap e1288: 12 total, 12 up, 12 in 2023-03-09T23:38:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:23 smithi122 ceph-mon[93869]: pgmap v17159: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 158 KiB/s wr, 8 op/s 2023-03-09T23:38:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31213.839175 secs 2023-03-09T23:38:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:23 smithi181 ceph-mon[94189]: osdmap e1288: 12 total, 12 up, 12 in 2023-03-09T23:38:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:23 smithi181 ceph-mon[94189]: pgmap v17159: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 158 KiB/s wr, 8 op/s 2023-03-09T23:38:23.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31213.839175 secs 2023-03-09T23:38:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:23 smithi038 ceph-mon[92422]: osdmap e1288: 12 total, 12 up, 12 in 2023-03-09T23:38:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:23 smithi038 ceph-mon[92422]: pgmap v17159: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 158 KiB/s wr, 8 op/s 2023-03-09T23:38:25.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:25 smithi122 ceph-mon[93869]: pgmap v17160: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 158 KiB/s wr, 8 op/s 2023-03-09T23:38:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31216.428431 secs 2023-03-09T23:38:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:25 smithi181 ceph-mon[94189]: pgmap v17160: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 158 KiB/s wr, 8 op/s 2023-03-09T23:38:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31216.428431 secs 2023-03-09T23:38:25.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:25 smithi038 ceph-mon[92422]: pgmap v17160: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 158 KiB/s wr, 8 op/s 2023-03-09T23:38:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31216.428431 secs 2023-03-09T23:38:27.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:27 smithi122 ceph-mon[93869]: pgmap v17161: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 157 KiB/s wr, 8 op/s 2023-03-09T23:38:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:27 smithi181 ceph-mon[94189]: pgmap v17161: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 157 KiB/s wr, 8 op/s 2023-03-09T23:38:27.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:27 smithi038 ceph-mon[92422]: pgmap v17161: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 157 KiB/s wr, 8 op/s 2023-03-09T23:38:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31680.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:38:28.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31218.839264 secs 2023-03-09T23:38:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31218.839264 secs 2023-03-09T23:38:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31218.839264 secs 2023-03-09T23:38:29.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:29 smithi122 ceph-mon[93869]: pgmap v17162: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 157 KiB/s wr, 7 op/s 2023-03-09T23:38:29.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:29 smithi181 ceph-mon[94189]: pgmap v17162: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 157 KiB/s wr, 7 op/s 2023-03-09T23:38:29.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:29 smithi038 ceph-mon[92422]: pgmap v17162: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 157 KiB/s wr, 7 op/s 2023-03-09T23:38:30.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31221.428493 secs 2023-03-09T23:38:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31221.428493 secs 2023-03-09T23:38:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31221.428493 secs 2023-03-09T23:38:31.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:31 smithi122 ceph-mon[93869]: pgmap v17163: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 109 KiB/s wr, 7 op/s 2023-03-09T23:38:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:31 smithi181 ceph-mon[94189]: pgmap v17163: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 109 KiB/s wr, 7 op/s 2023-03-09T23:38:31.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:31 smithi038 ceph-mon[92422]: pgmap v17163: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 109 KiB/s wr, 7 op/s 2023-03-09T23:38:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31685.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:38:33.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31223.839347 secs 2023-03-09T23:38:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:33 smithi122 ceph-mon[93869]: pgmap v17164: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 101 B/s wr, 7 op/s 2023-03-09T23:38:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31223.839347 secs 2023-03-09T23:38:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:33 smithi181 ceph-mon[94189]: pgmap v17164: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 101 B/s wr, 7 op/s 2023-03-09T23:38:33.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31223.839347 secs 2023-03-09T23:38:33.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:33 smithi038 ceph-mon[92422]: pgmap v17164: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 101 B/s wr, 7 op/s 2023-03-09T23:38:35.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:35 smithi122 ceph-mon[93869]: pgmap v17165: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-09T23:38:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31226.428567 secs 2023-03-09T23:38:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:35 smithi181 ceph-mon[94189]: pgmap v17165: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-09T23:38:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31226.428567 secs 2023-03-09T23:38:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:35 smithi038 ceph-mon[92422]: pgmap v17165: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-09T23:38:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31226.428567 secs 2023-03-09T23:38:37.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:37 smithi122 ceph-mon[93869]: pgmap v17166: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 9 op/s 2023-03-09T23:38:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:37 smithi181 ceph-mon[94189]: pgmap v17166: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 9 op/s 2023-03-09T23:38:37.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:37 smithi038 ceph-mon[92422]: pgmap v17166: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 85 B/s wr, 9 op/s 2023-03-09T23:38:38.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31690.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:38:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31228.839448 secs 2023-03-09T23:38:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31228.839448 secs 2023-03-09T23:38:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31228.839448 secs 2023-03-09T23:38:39.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:39 smithi122 ceph-mon[93869]: pgmap v17167: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-09T23:38:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:39 smithi181 ceph-mon[94189]: pgmap v17167: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-09T23:38:39.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:39 smithi038 ceph-mon[92422]: pgmap v17167: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-09T23:38:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31231.428628 secs 2023-03-09T23:38:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31231.428628 secs 2023-03-09T23:38:41.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31231.428628 secs 2023-03-09T23:38:41.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:41 smithi038 ceph-mon[92422]: pgmap v17168: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 7 op/s 2023-03-09T23:38:42.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:41 smithi122 ceph-mon[93869]: pgmap v17168: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 7 op/s 2023-03-09T23:38:42.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:41 smithi181 ceph-mon[94189]: pgmap v17168: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 7 op/s 2023-03-09T23:38:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31695.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:38:43.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:43 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 31233.839575 secs 2023-03-09T23:38:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:43 smithi122 ceph-mon[93869]: pgmap v17169: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 767 B/s wr, 8 op/s 2023-03-09T23:38:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:43 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 31233.839575 secs 2023-03-09T23:38:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:43 smithi181 ceph-mon[94189]: pgmap v17169: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 767 B/s wr, 8 op/s 2023-03-09T23:38:43.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:43 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 31233.839575 secs 2023-03-09T23:38:43.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:43 smithi038 ceph-mon[92422]: pgmap v17169: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 767 B/s wr, 8 op/s 2023-03-09T23:38:45.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:45 smithi122 ceph-mon[93869]: pgmap v17170: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 767 B/s wr, 7 op/s 2023-03-09T23:38:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31236.428691 secs 2023-03-09T23:38:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:45 smithi181 ceph-mon[94189]: pgmap v17170: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 767 B/s wr, 7 op/s 2023-03-09T23:38:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31236.428691 secs 2023-03-09T23:38:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:45 smithi038 ceph-mon[92422]: pgmap v17170: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 767 B/s wr, 7 op/s 2023-03-09T23:38:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31236.428691 secs 2023-03-09T23:38:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:46 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:38:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:46 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:38:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:46 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:38:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:46 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:38:46.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:46 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:38:46.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:46 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:38:47.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:47 smithi122 ceph-mon[93869]: pgmap v17171: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 357 KiB/s wr, 11 op/s 2023-03-09T23:38:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:38:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:38:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:47 smithi181 ceph-mon[94189]: pgmap v17171: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 357 KiB/s wr, 11 op/s 2023-03-09T23:38:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:38:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:38:47.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:47 smithi038 ceph-mon[92422]: pgmap v17171: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 357 KiB/s wr, 11 op/s 2023-03-09T23:38:47.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:38:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:38:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31700.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:38:48.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:48 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 31238.839713 secs 2023-03-09T23:38:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:48 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 31238.839713 secs 2023-03-09T23:38:48.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:48 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 31238.839713 secs 2023-03-09T23:38:49.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:49 smithi122 ceph-mon[93869]: pgmap v17172: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 358 KiB/s wr, 8 op/s 2023-03-09T23:38:49.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:38:49.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:38:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:49 smithi181 ceph-mon[94189]: pgmap v17172: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 358 KiB/s wr, 8 op/s 2023-03-09T23:38:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:38:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:38:49.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:49 smithi038 ceph-mon[92422]: pgmap v17172: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 358 KiB/s wr, 8 op/s 2023-03-09T23:38:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:38:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:38:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31241.428754 secs 2023-03-09T23:38:51.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31241.428754 secs 2023-03-09T23:38:51.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31241.428754 secs 2023-03-09T23:38:51.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:51 smithi038 ceph-mon[92422]: pgmap v17173: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 366 KiB/s wr, 10 op/s 2023-03-09T23:38:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:38:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:38:52.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:51 smithi122 ceph-mon[93869]: pgmap v17173: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 366 KiB/s wr, 10 op/s 2023-03-09T23:38:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:38:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:38:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:51 smithi181 ceph-mon[94189]: pgmap v17173: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 366 KiB/s wr, 10 op/s 2023-03-09T23:38:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:38:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:38:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31705.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:38:53.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31243.839842 secs 2023-03-09T23:38:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:53 smithi122 ceph-mon[93869]: pgmap v17174: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 658 KiB/s rd, 655 KiB/s wr, 12 op/s 2023-03-09T23:38:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:38:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31243.839842 secs 2023-03-09T23:38:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:53 smithi181 ceph-mon[94189]: pgmap v17174: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 658 KiB/s rd, 655 KiB/s wr, 12 op/s 2023-03-09T23:38:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:38:53.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31243.839842 secs 2023-03-09T23:38:53.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:53 smithi038 ceph-mon[92422]: pgmap v17174: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 658 KiB/s rd, 655 KiB/s wr, 12 op/s 2023-03-09T23:38:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:38:54.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:54 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:38:54.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:54 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:38:54.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:54 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:38:54.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:54 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:38:54.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:54 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:38:54.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:54 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:38:54.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:54 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:38:54.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:54 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:38:54.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:54 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:38:54.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:54 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:38:54.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:54 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:38:54.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:54 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:38:54.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:54 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:38:54.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:54 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:38:54.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:54 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:38:56.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:56 smithi122 ceph-mon[93869]: pgmap v17175: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 669 KiB/s rd, 654 KiB/s wr, 11 op/s 2023-03-09T23:38:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31246.428814 secs 2023-03-09T23:38:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:38:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:38:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:56 smithi181 ceph-mon[94189]: pgmap v17175: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 669 KiB/s rd, 654 KiB/s wr, 11 op/s 2023-03-09T23:38:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31246.428814 secs 2023-03-09T23:38:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:38:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:38:56.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:56 smithi038 ceph-mon[92422]: pgmap v17175: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 669 KiB/s rd, 654 KiB/s wr, 11 op/s 2023-03-09T23:38:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31246.428814 secs 2023-03-09T23:38:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:38:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:38:57.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:38:57.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:38:57.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:57 smithi122 ceph-mon[93869]: pgmap v17176: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 16 op/s 2023-03-09T23:38:57.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:38:57.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:38:57.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:57 smithi181 ceph-mon[94189]: pgmap v17176: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 16 op/s 2023-03-09T23:38:57.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:38:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:38:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:57 smithi038 ceph-mon[92422]: pgmap v17176: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 16 op/s 2023-03-09T23:38:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31710.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:38:58.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31248.839965 secs 2023-03-09T23:38:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31248.839965 secs 2023-03-09T23:38:58.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31248.839965 secs 2023-03-09T23:38:59.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:38:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:38:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:38:59 smithi122 ceph-mon[93869]: pgmap v17177: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 775 KiB/s rd, 781 KiB/s wr, 12 op/s 2023-03-09T23:38:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:38:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:38:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:38:59 smithi181 ceph-mon[94189]: pgmap v17177: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 775 KiB/s rd, 781 KiB/s wr, 12 op/s 2023-03-09T23:38:59.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:38:59.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:38:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:38:59 smithi038 ceph-mon[92422]: pgmap v17177: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 775 KiB/s rd, 781 KiB/s wr, 12 op/s 2023-03-09T23:39:00.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31251.428874 secs 2023-03-09T23:39:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31251.428874 secs 2023-03-09T23:39:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31251.428874 secs 2023-03-09T23:39:01.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:39:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:39:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:01 smithi122 ceph-mon[93869]: pgmap v17178: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 969 KiB/s rd, 955 KiB/s wr, 14 op/s 2023-03-09T23:39:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:39:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:39:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:01 smithi181 ceph-mon[94189]: pgmap v17178: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 969 KiB/s rd, 955 KiB/s wr, 14 op/s 2023-03-09T23:39:01.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:39:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:39:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:01 smithi038 ceph-mon[92422]: pgmap v17178: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 969 KiB/s rd, 955 KiB/s wr, 14 op/s 2023-03-09T23:39:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31715.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:39:03.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31253.840083 secs 2023-03-09T23:39:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:03 smithi122 ceph-mon[93869]: pgmap v17179: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 987 KiB/s rd, 995 KiB/s wr, 17 op/s 2023-03-09T23:39:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31253.840083 secs 2023-03-09T23:39:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:03 smithi181 ceph-mon[94189]: pgmap v17179: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 987 KiB/s rd, 995 KiB/s wr, 17 op/s 2023-03-09T23:39:03.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31253.840083 secs 2023-03-09T23:39:03.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:03 smithi038 ceph-mon[92422]: pgmap v17179: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 987 KiB/s rd, 995 KiB/s wr, 17 op/s 2023-03-09T23:39:05.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:05 smithi122 ceph-mon[93869]: osdmap e1289: 12 total, 12 up, 12 in 2023-03-09T23:39:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:05 smithi122 ceph-mon[93869]: pgmap v17181: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 831 KiB/s rd, 873 KiB/s wr, 30 op/s 2023-03-09T23:39:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31256.428935 secs 2023-03-09T23:39:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:05 smithi181 ceph-mon[94189]: osdmap e1289: 12 total, 12 up, 12 in 2023-03-09T23:39:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:05 smithi181 ceph-mon[94189]: pgmap v17181: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 831 KiB/s rd, 873 KiB/s wr, 30 op/s 2023-03-09T23:39:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31256.428935 secs 2023-03-09T23:39:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:05 smithi038 ceph-mon[92422]: osdmap e1289: 12 total, 12 up, 12 in 2023-03-09T23:39:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:05 smithi038 ceph-mon[92422]: pgmap v17181: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 831 KiB/s rd, 873 KiB/s wr, 30 op/s 2023-03-09T23:39:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31256.428935 secs 2023-03-09T23:39:07.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:39:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:39:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:07 smithi122 ceph-mon[93869]: pgmap v17182: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 377 KiB/s wr, 28 op/s 2023-03-09T23:39:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:39:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:39:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:07 smithi181 ceph-mon[94189]: pgmap v17182: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 377 KiB/s wr, 28 op/s 2023-03-09T23:39:07.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:39:07.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:39:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:07 smithi038 ceph-mon[92422]: pgmap v17182: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 377 KiB/s wr, 28 op/s 2023-03-09T23:39:08.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31720.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:39:08.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31258.840197 secs 2023-03-09T23:39:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31258.840197 secs 2023-03-09T23:39:08.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31258.840197 secs 2023-03-09T23:39:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:39:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:39:09.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:09 smithi122 ceph-mon[93869]: pgmap v17183: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 522 KiB/s rd, 539 KiB/s wr, 28 op/s 2023-03-09T23:39:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:39:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:39:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:09 smithi181 ceph-mon[94189]: pgmap v17183: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 522 KiB/s rd, 539 KiB/s wr, 28 op/s 2023-03-09T23:39:09.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:39:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:39:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:09 smithi038 ceph-mon[92422]: pgmap v17183: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 522 KiB/s rd, 539 KiB/s wr, 28 op/s 2023-03-09T23:39:10.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31261.429006 secs 2023-03-09T23:39:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31261.429006 secs 2023-03-09T23:39:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31261.429006 secs 2023-03-09T23:39:11.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:11 smithi122 ceph-mon[93869]: pgmap v17184: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 330 KiB/s wr, 27 op/s 2023-03-09T23:39:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:11 smithi181 ceph-mon[94189]: pgmap v17184: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 330 KiB/s wr, 27 op/s 2023-03-09T23:39:11.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:11 smithi038 ceph-mon[92422]: pgmap v17184: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 330 KiB/s wr, 27 op/s 2023-03-09T23:39:13.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31725.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:39:13.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31263.840319 secs 2023-03-09T23:39:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:13 smithi122 ceph-mon[93869]: osdmap e1290: 12 total, 12 up, 12 in 2023-03-09T23:39:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:13 smithi122 ceph-mon[93869]: pgmap v17186: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 508 KiB/s wr, 12 op/s 2023-03-09T23:39:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31263.840319 secs 2023-03-09T23:39:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:13 smithi181 ceph-mon[94189]: osdmap e1290: 12 total, 12 up, 12 in 2023-03-09T23:39:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:13 smithi181 ceph-mon[94189]: pgmap v17186: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 508 KiB/s wr, 12 op/s 2023-03-09T23:39:13.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31263.840319 secs 2023-03-09T23:39:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:13 smithi038 ceph-mon[92422]: osdmap e1290: 12 total, 12 up, 12 in 2023-03-09T23:39:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:13 smithi038 ceph-mon[92422]: pgmap v17186: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 508 KiB/s wr, 12 op/s 2023-03-09T23:39:15.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:15 smithi122 ceph-mon[93869]: pgmap v17187: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 409 KiB/s wr, 10 op/s 2023-03-09T23:39:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31266.429064 secs 2023-03-09T23:39:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:15 smithi181 ceph-mon[94189]: pgmap v17187: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 409 KiB/s wr, 10 op/s 2023-03-09T23:39:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31266.429064 secs 2023-03-09T23:39:15.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:15 smithi038 ceph-mon[92422]: pgmap v17187: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 409 KiB/s wr, 10 op/s 2023-03-09T23:39:15.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31266.429064 secs 2023-03-09T23:39:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:17 smithi181 ceph-mon[94189]: pgmap v17188: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 328 KiB/s rd, 325 KiB/s wr, 8 op/s 2023-03-09T23:39:17.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:17 smithi038 ceph-mon[92422]: pgmap v17188: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 328 KiB/s rd, 325 KiB/s wr, 8 op/s 2023-03-09T23:39:18.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:17 smithi122 ceph-mon[93869]: pgmap v17188: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 328 KiB/s rd, 325 KiB/s wr, 8 op/s 2023-03-09T23:39:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31730.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:39:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31268.840435 secs 2023-03-09T23:39:19.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31268.840435 secs 2023-03-09T23:39:19.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31268.840435 secs 2023-03-09T23:39:19.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:19 smithi038 ceph-mon[92422]: pgmap v17189: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 163 KiB/s wr, 7 op/s 2023-03-09T23:39:20.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:19 smithi122 ceph-mon[93869]: pgmap v17189: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 163 KiB/s wr, 7 op/s 2023-03-09T23:39:20.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:19 smithi181 ceph-mon[94189]: pgmap v17189: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 163 KiB/s wr, 7 op/s 2023-03-09T23:39:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31271.429126 secs 2023-03-09T23:39:21.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31271.429126 secs 2023-03-09T23:39:21.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31271.429126 secs 2023-03-09T23:39:22.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:21 smithi122 ceph-mon[93869]: pgmap v17190: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 163 KiB/s wr, 7 op/s 2023-03-09T23:39:22.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:21 smithi181 ceph-mon[94189]: pgmap v17190: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 163 KiB/s wr, 7 op/s 2023-03-09T23:39:22.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:21 smithi038 ceph-mon[92422]: pgmap v17190: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 163 KiB/s wr, 7 op/s 2023-03-09T23:39:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31735.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:39:23.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31273.840554 secs 2023-03-09T23:39:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:23 smithi122 ceph-mon[93869]: pgmap v17191: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-03-09T23:39:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31273.840554 secs 2023-03-09T23:39:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:23 smithi181 ceph-mon[94189]: pgmap v17191: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-03-09T23:39:23.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31273.840554 secs 2023-03-09T23:39:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:23 smithi038 ceph-mon[92422]: pgmap v17191: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-03-09T23:39:25.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:25 smithi122 ceph-mon[93869]: pgmap v17192: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T23:39:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31276.429186 secs 2023-03-09T23:39:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:25 smithi181 ceph-mon[94189]: pgmap v17192: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T23:39:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31276.429186 secs 2023-03-09T23:39:25.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:25 smithi038 ceph-mon[92422]: pgmap v17192: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T23:39:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31276.429186 secs 2023-03-09T23:39:27.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:27 smithi122 ceph-mon[93869]: pgmap v17193: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T23:39:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:27 smithi181 ceph-mon[94189]: pgmap v17193: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T23:39:27.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:27 smithi038 ceph-mon[92422]: pgmap v17193: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T23:39:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31740.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:39:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31278.840676 secs 2023-03-09T23:39:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31278.840676 secs 2023-03-09T23:39:29.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31278.840676 secs 2023-03-09T23:39:29.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:29 smithi181 ceph-mon[94189]: pgmap v17194: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T23:39:29.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:29 smithi038 ceph-mon[92422]: pgmap v17194: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T23:39:30.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:29 smithi122 ceph-mon[93869]: pgmap v17194: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T23:39:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31281.429245 secs 2023-03-09T23:39:31.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31281.429245 secs 2023-03-09T23:39:31.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31281.429245 secs 2023-03-09T23:39:31.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:31 smithi038 ceph-mon[92422]: pgmap v17195: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2023-03-09T23:39:32.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:31 smithi122 ceph-mon[93869]: pgmap v17195: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2023-03-09T23:39:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:31 smithi181 ceph-mon[94189]: pgmap v17195: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2023-03-09T23:39:33.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31745.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:39:33.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31283.840788 secs 2023-03-09T23:39:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:33 smithi122 ceph-mon[93869]: pgmap v17196: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2023-03-09T23:39:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31283.840788 secs 2023-03-09T23:39:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:33 smithi181 ceph-mon[94189]: pgmap v17196: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2023-03-09T23:39:33.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31283.840788 secs 2023-03-09T23:39:33.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:33 smithi038 ceph-mon[92422]: pgmap v17196: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2023-03-09T23:39:35.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:35 smithi122 ceph-mon[93869]: pgmap v17197: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T23:39:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31286.429323 secs 2023-03-09T23:39:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:35 smithi181 ceph-mon[94189]: pgmap v17197: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T23:39:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31286.429323 secs 2023-03-09T23:39:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:35 smithi038 ceph-mon[92422]: pgmap v17197: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T23:39:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31286.429323 secs 2023-03-09T23:39:37.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:37 smithi038 ceph-mon[92422]: pgmap v17198: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 341 B/s wr, 8 op/s 2023-03-09T23:39:38.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:37 smithi122 ceph-mon[93869]: pgmap v17198: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 341 B/s wr, 8 op/s 2023-03-09T23:39:38.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:37 smithi181 ceph-mon[94189]: pgmap v17198: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 341 B/s wr, 8 op/s 2023-03-09T23:39:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31750.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:39:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31288.840910 secs 2023-03-09T23:39:39.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31288.840910 secs 2023-03-09T23:39:39.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31288.840910 secs 2023-03-09T23:39:39.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:39 smithi038 ceph-mon[92422]: pgmap v17199: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 115 KiB/s wr, 7 op/s 2023-03-09T23:39:40.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:39 smithi122 ceph-mon[93869]: pgmap v17199: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 115 KiB/s wr, 7 op/s 2023-03-09T23:39:40.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:39 smithi181 ceph-mon[94189]: pgmap v17199: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 115 KiB/s wr, 7 op/s 2023-03-09T23:39:41.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31291.429402 secs 2023-03-09T23:39:41.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31291.429402 secs 2023-03-09T23:39:41.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31291.429402 secs 2023-03-09T23:39:42.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:41 smithi122 ceph-mon[93869]: pgmap v17200: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 115 KiB/s wr, 8 op/s 2023-03-09T23:39:42.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:41 smithi181 ceph-mon[94189]: pgmap v17200: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 115 KiB/s wr, 8 op/s 2023-03-09T23:39:42.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:41 smithi038 ceph-mon[92422]: pgmap v17200: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 115 KiB/s wr, 8 op/s 2023-03-09T23:39:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31755.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:39:43.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31293.841040 secs 2023-03-09T23:39:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:43 smithi122 ceph-mon[93869]: pgmap v17201: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 247 KiB/s wr, 9 op/s 2023-03-09T23:39:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31293.841040 secs 2023-03-09T23:39:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:43 smithi181 ceph-mon[94189]: pgmap v17201: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 247 KiB/s wr, 9 op/s 2023-03-09T23:39:43.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31293.841040 secs 2023-03-09T23:39:43.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:43 smithi038 ceph-mon[92422]: pgmap v17201: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 247 KiB/s wr, 9 op/s 2023-03-09T23:39:45.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:45 smithi122 ceph-mon[93869]: pgmap v17202: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 277 KiB/s wr, 8 op/s 2023-03-09T23:39:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31296.429461 secs 2023-03-09T23:39:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:45 smithi181 ceph-mon[94189]: pgmap v17202: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 277 KiB/s wr, 8 op/s 2023-03-09T23:39:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31296.429461 secs 2023-03-09T23:39:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:45 smithi038 ceph-mon[92422]: pgmap v17202: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 277 KiB/s wr, 8 op/s 2023-03-09T23:39:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31296.429461 secs 2023-03-09T23:39:47.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:47 smithi038 ceph-mon[92422]: pgmap v17203: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 302 KiB/s wr, 10 op/s 2023-03-09T23:39:47.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:39:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:39:48.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:47 smithi122 ceph-mon[93869]: pgmap v17203: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 302 KiB/s wr, 10 op/s 2023-03-09T23:39:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:39:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:39:48.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:47 smithi181 ceph-mon[94189]: pgmap v17203: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 302 KiB/s wr, 10 op/s 2023-03-09T23:39:48.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:39:48.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:39:48.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31760.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:39:48.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31298.841160 secs 2023-03-09T23:39:49.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31298.841160 secs 2023-03-09T23:39:49.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31298.841160 secs 2023-03-09T23:39:50.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:49 smithi122 ceph-mon[93869]: pgmap v17204: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 471 KiB/s rd, 469 KiB/s wr, 9 op/s 2023-03-09T23:39:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:39:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:39:50.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:49 smithi181 ceph-mon[94189]: pgmap v17204: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 471 KiB/s rd, 469 KiB/s wr, 9 op/s 2023-03-09T23:39:50.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:39:50.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:39:50.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:49 smithi038 ceph-mon[92422]: pgmap v17204: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 471 KiB/s rd, 469 KiB/s wr, 9 op/s 2023-03-09T23:39:50.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:39:50.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:39:51.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31301.429521 secs 2023-03-09T23:39:51.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31301.429521 secs 2023-03-09T23:39:51.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31301.429521 secs 2023-03-09T23:39:52.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:51 smithi122 ceph-mon[93869]: pgmap v17205: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 411 KiB/s wr, 10 op/s 2023-03-09T23:39:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:39:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:39:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:51 smithi181 ceph-mon[94189]: pgmap v17205: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 411 KiB/s wr, 10 op/s 2023-03-09T23:39:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:39:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:39:52.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:51 smithi038 ceph-mon[92422]: pgmap v17205: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 411 KiB/s wr, 10 op/s 2023-03-09T23:39:52.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:39:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:39:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31765.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:39:53.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31303.841302 secs 2023-03-09T23:39:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:53 smithi122 ceph-mon[93869]: pgmap v17206: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 437 KiB/s wr, 11 op/s 2023-03-09T23:39:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31303.841302 secs 2023-03-09T23:39:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:53 smithi181 ceph-mon[94189]: pgmap v17206: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 437 KiB/s wr, 11 op/s 2023-03-09T23:39:53.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31303.841302 secs 2023-03-09T23:39:53.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:53 smithi038 ceph-mon[92422]: pgmap v17206: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 437 KiB/s wr, 11 op/s 2023-03-09T23:39:54.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:54 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:39:54.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:54 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:39:54.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:54 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:39:55.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:39:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:39:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:55 smithi122 ceph-mon[93869]: pgmap v17207: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 593 KiB/s rd, 578 KiB/s wr, 10 op/s 2023-03-09T23:39:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:55 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:39:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:55 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:39:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:55 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:39:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31306.429582 secs 2023-03-09T23:39:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:39:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:39:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:55 smithi181 ceph-mon[94189]: pgmap v17207: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 593 KiB/s rd, 578 KiB/s wr, 10 op/s 2023-03-09T23:39:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:55 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:39:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:55 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:39:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:55 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:39:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31306.429582 secs 2023-03-09T23:39:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:39:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:39:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:55 smithi038 ceph-mon[92422]: pgmap v17207: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 593 KiB/s rd, 578 KiB/s wr, 10 op/s 2023-03-09T23:39:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:55 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:39:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:55 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:39:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:55 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:39:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31306.429582 secs 2023-03-09T23:39:57.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:39:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:39:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:57 smithi122 ceph-mon[93869]: pgmap v17208: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 916 KiB/s rd, 901 KiB/s wr, 14 op/s 2023-03-09T23:39:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:39:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:39:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:57 smithi181 ceph-mon[94189]: pgmap v17208: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 916 KiB/s rd, 901 KiB/s wr, 14 op/s 2023-03-09T23:39:57.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:39:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:39:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:57 smithi038 ceph-mon[92422]: pgmap v17208: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 916 KiB/s rd, 901 KiB/s wr, 14 op/s 2023-03-09T23:39:58.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31770.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:39:59.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:39:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:39:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:39:59 smithi122 ceph-mon[93869]: pgmap v17209: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1000 KiB/s rd, 987 KiB/s wr, 13 op/s 2023-03-09T23:39:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:39:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:39:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:39:59 smithi181 ceph-mon[94189]: pgmap v17209: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1000 KiB/s rd, 987 KiB/s wr, 13 op/s 2023-03-09T23:39:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:39:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:39:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:39:59 smithi038 ceph-mon[92422]: pgmap v17209: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1000 KiB/s rd, 987 KiB/s wr, 13 op/s 2023-03-09T23:40:00.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31311.429642 secs 2023-03-09T23:40:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:00 smithi122 ceph-mon[93869]: Health detail: HEALTH_WARN 2 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-09T23:40:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:00 smithi122 ceph-mon[93869]: [WRN] MDS_SLOW_REQUEST: 2 MDSs report slow requests 2023-03-09T23:40:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:00 smithi122 ceph-mon[93869]: mds.i(mds.2): 2 slow requests are blocked > 180 secs 2023-03-09T23:40:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:00 smithi122 ceph-mon[93869]: mds.h(mds.1): 2 slow requests are blocked > 180 secs 2023-03-09T23:40:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:00 smithi122 ceph-mon[93869]: [WRN] MDS_TRIM: 2 MDSs behind on trimming 2023-03-09T23:40:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:00 smithi122 ceph-mon[93869]: mds.e(mds.0): Behind on trimming (708/128) max_segments: 128, num_segments: 708 2023-03-09T23:40:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:00 smithi122 ceph-mon[93869]: mds.b(mds.0): Behind on trimming (707/128) max_segments: 128, num_segments: 707 2023-03-09T23:40:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31311.429642 secs 2023-03-09T23:40:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:00 smithi181 ceph-mon[94189]: Health detail: HEALTH_WARN 2 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-09T23:40:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:00 smithi181 ceph-mon[94189]: [WRN] MDS_SLOW_REQUEST: 2 MDSs report slow requests 2023-03-09T23:40:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:00 smithi181 ceph-mon[94189]: mds.i(mds.2): 2 slow requests are blocked > 180 secs 2023-03-09T23:40:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:00 smithi181 ceph-mon[94189]: mds.h(mds.1): 2 slow requests are blocked > 180 secs 2023-03-09T23:40:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:00 smithi181 ceph-mon[94189]: [WRN] MDS_TRIM: 2 MDSs behind on trimming 2023-03-09T23:40:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:00 smithi181 ceph-mon[94189]: mds.e(mds.0): Behind on trimming (708/128) max_segments: 128, num_segments: 708 2023-03-09T23:40:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:00 smithi181 ceph-mon[94189]: mds.b(mds.0): Behind on trimming (707/128) max_segments: 128, num_segments: 707 2023-03-09T23:40:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31311.429642 secs 2023-03-09T23:40:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:00 smithi038 ceph-mon[92422]: Health detail: HEALTH_WARN 2 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-09T23:40:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:00 smithi038 ceph-mon[92422]: [WRN] MDS_SLOW_REQUEST: 2 MDSs report slow requests 2023-03-09T23:40:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:00 smithi038 ceph-mon[92422]: mds.i(mds.2): 2 slow requests are blocked > 180 secs 2023-03-09T23:40:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:00 smithi038 ceph-mon[92422]: mds.h(mds.1): 2 slow requests are blocked > 180 secs 2023-03-09T23:40:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:00 smithi038 ceph-mon[92422]: [WRN] MDS_TRIM: 2 MDSs behind on trimming 2023-03-09T23:40:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:00 smithi038 ceph-mon[92422]: mds.e(mds.0): Behind on trimming (708/128) max_segments: 128, num_segments: 708 2023-03-09T23:40:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:00 smithi038 ceph-mon[92422]: mds.b(mds.0): Behind on trimming (707/128) max_segments: 128, num_segments: 707 2023-03-09T23:40:01.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:40:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:40:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:01 smithi122 ceph-mon[93869]: pgmap v17210: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 835 KiB/s rd, 842 KiB/s wr, 15 op/s 2023-03-09T23:40:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:01 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:40:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:40:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:40:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:01 smithi181 ceph-mon[94189]: pgmap v17210: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 835 KiB/s rd, 842 KiB/s wr, 15 op/s 2023-03-09T23:40:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:01 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:40:01.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:40:01.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:40:01.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:01 smithi038 ceph-mon[92422]: pgmap v17210: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 835 KiB/s rd, 842 KiB/s wr, 15 op/s 2023-03-09T23:40:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:01 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:40:02.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:02 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:40:02.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:02 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:40:02.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:02 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:40:02.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:02 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:40:02.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:02 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:40:02.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:02 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:40:02.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:02 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:40:02.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:02 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:40:02.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:02 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:40:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31775.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:40:03.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31313.841533 secs 2023-03-09T23:40:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:03 smithi122 ceph-mon[93869]: pgmap v17211: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 803 KiB/s rd, 811 KiB/s wr, 15 op/s 2023-03-09T23:40:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31313.841533 secs 2023-03-09T23:40:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:03 smithi181 ceph-mon[94189]: pgmap v17211: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 803 KiB/s rd, 811 KiB/s wr, 15 op/s 2023-03-09T23:40:03.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31313.841533 secs 2023-03-09T23:40:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:03 smithi038 ceph-mon[92422]: pgmap v17211: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 803 KiB/s rd, 811 KiB/s wr, 15 op/s 2023-03-09T23:40:05.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:05 smithi122 ceph-mon[93869]: osdmap e1291: 12 total, 12 up, 12 in 2023-03-09T23:40:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:05 smithi122 ceph-mon[93869]: pgmap v17213: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 810 KiB/s rd, 919 KiB/s wr, 31 op/s 2023-03-09T23:40:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31316.429704 secs 2023-03-09T23:40:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:05 smithi181 ceph-mon[94189]: osdmap e1291: 12 total, 12 up, 12 in 2023-03-09T23:40:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:05 smithi181 ceph-mon[94189]: pgmap v17213: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 810 KiB/s rd, 919 KiB/s wr, 31 op/s 2023-03-09T23:40:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31316.429704 secs 2023-03-09T23:40:05.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:05 smithi038 ceph-mon[92422]: osdmap e1291: 12 total, 12 up, 12 in 2023-03-09T23:40:05.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:05 smithi038 ceph-mon[92422]: pgmap v17213: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 810 KiB/s rd, 919 KiB/s wr, 31 op/s 2023-03-09T23:40:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31316.429704 secs 2023-03-09T23:40:06.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:06 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:40:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:40:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:06 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:40:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:40:06.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:06 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:40:06.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:40:07.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:07 smithi122 ceph-mon[93869]: pgmap v17214: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 565 KiB/s rd, 673 KiB/s wr, 30 op/s 2023-03-09T23:40:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:07 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:40:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:07 smithi181 ceph-mon[94189]: pgmap v17214: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 565 KiB/s rd, 673 KiB/s wr, 30 op/s 2023-03-09T23:40:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:07 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:40:07.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:07 smithi038 ceph-mon[92422]: pgmap v17214: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 565 KiB/s rd, 673 KiB/s wr, 30 op/s 2023-03-09T23:40:07.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:07 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:40:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31780.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:40:08.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31318.841652 secs 2023-03-09T23:40:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31318.841652 secs 2023-03-09T23:40:08.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31318.841652 secs 2023-03-09T23:40:09.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:09 smithi122 ceph-mon[93869]: pgmap v17215: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 499 KiB/s rd, 540 KiB/s wr, 30 op/s 2023-03-09T23:40:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:40:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:40:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:09 smithi181 ceph-mon[94189]: pgmap v17215: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 499 KiB/s rd, 540 KiB/s wr, 30 op/s 2023-03-09T23:40:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:40:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:40:09.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:09 smithi038 ceph-mon[92422]: pgmap v17215: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 499 KiB/s rd, 540 KiB/s wr, 30 op/s 2023-03-09T23:40:09.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:40:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:40:10.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31321.429790 secs 2023-03-09T23:40:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31321.429790 secs 2023-03-09T23:40:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31321.429790 secs 2023-03-09T23:40:11.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:11 smithi122 ceph-mon[93869]: pgmap v17216: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 660 KiB/s rd, 675 KiB/s wr, 28 op/s 2023-03-09T23:40:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:11 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:40:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:11 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:40:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:11 smithi181 ceph-mon[94189]: pgmap v17216: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 660 KiB/s rd, 675 KiB/s wr, 28 op/s 2023-03-09T23:40:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:11 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:40:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:11 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:40:11.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:11 smithi038 ceph-mon[92422]: pgmap v17216: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 660 KiB/s rd, 675 KiB/s wr, 28 op/s 2023-03-09T23:40:11.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:11 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:40:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:11 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:40:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31785.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:40:13.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31323.841772 secs 2023-03-09T23:40:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:13 smithi122 ceph-mon[93869]: osdmap e1292: 12 total, 12 up, 12 in 2023-03-09T23:40:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:13 smithi122 ceph-mon[93869]: pgmap v17218: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 513 KiB/s rd, 427 KiB/s wr, 12 op/s 2023-03-09T23:40:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31323.841772 secs 2023-03-09T23:40:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:13 smithi181 ceph-mon[94189]: osdmap e1292: 12 total, 12 up, 12 in 2023-03-09T23:40:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:13 smithi181 ceph-mon[94189]: pgmap v17218: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 513 KiB/s rd, 427 KiB/s wr, 12 op/s 2023-03-09T23:40:13.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31323.841772 secs 2023-03-09T23:40:13.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:13 smithi038 ceph-mon[92422]: osdmap e1292: 12 total, 12 up, 12 in 2023-03-09T23:40:13.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:13 smithi038 ceph-mon[92422]: pgmap v17218: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 513 KiB/s rd, 427 KiB/s wr, 12 op/s 2023-03-09T23:40:15.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:15 smithi038 ceph-mon[92422]: pgmap v17219: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 342 KiB/s wr, 10 op/s 2023-03-09T23:40:15.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31326.429867 secs 2023-03-09T23:40:16.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:15 smithi122 ceph-mon[93869]: pgmap v17219: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 342 KiB/s wr, 10 op/s 2023-03-09T23:40:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31326.429867 secs 2023-03-09T23:40:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:15 smithi181 ceph-mon[94189]: pgmap v17219: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 342 KiB/s wr, 10 op/s 2023-03-09T23:40:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31326.429867 secs 2023-03-09T23:40:17.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:17 smithi038 ceph-mon[92422]: pgmap v17220: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 163 KiB/s wr, 8 op/s 2023-03-09T23:40:18.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:17 smithi122 ceph-mon[93869]: pgmap v17220: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 163 KiB/s wr, 8 op/s 2023-03-09T23:40:18.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:17 smithi181 ceph-mon[94189]: pgmap v17220: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 163 KiB/s wr, 8 op/s 2023-03-09T23:40:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31790.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:40:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31328.841888 secs 2023-03-09T23:40:19.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31328.841888 secs 2023-03-09T23:40:19.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31328.841888 secs 2023-03-09T23:40:19.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:19 smithi038 ceph-mon[92422]: pgmap v17221: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 163 KiB/s wr, 7 op/s 2023-03-09T23:40:20.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:19 smithi122 ceph-mon[93869]: pgmap v17221: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 163 KiB/s wr, 7 op/s 2023-03-09T23:40:20.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:19 smithi181 ceph-mon[94189]: pgmap v17221: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 163 KiB/s wr, 7 op/s 2023-03-09T23:40:21.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31331.429927 secs 2023-03-09T23:40:21.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31331.429927 secs 2023-03-09T23:40:21.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31331.429927 secs 2023-03-09T23:40:22.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:21 smithi122 ceph-mon[93869]: pgmap v17222: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T23:40:22.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:21 smithi181 ceph-mon[94189]: pgmap v17222: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T23:40:22.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:21 smithi038 ceph-mon[92422]: pgmap v17222: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T23:40:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31795.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:40:23.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31333.842004 secs 2023-03-09T23:40:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:23 smithi122 ceph-mon[93869]: pgmap v17223: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-03-09T23:40:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31333.842004 secs 2023-03-09T23:40:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:23 smithi181 ceph-mon[94189]: pgmap v17223: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-03-09T23:40:23.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31333.842004 secs 2023-03-09T23:40:23.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:23 smithi038 ceph-mon[92422]: pgmap v17223: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-03-09T23:40:25.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:25 smithi038 ceph-mon[92422]: pgmap v17224: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2023-03-09T23:40:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31336.429985 secs 2023-03-09T23:40:26.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:25 smithi122 ceph-mon[93869]: pgmap v17224: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2023-03-09T23:40:26.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31336.429985 secs 2023-03-09T23:40:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:25 smithi181 ceph-mon[94189]: pgmap v17224: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2023-03-09T23:40:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31336.429985 secs 2023-03-09T23:40:27.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:27 smithi038 ceph-mon[92422]: pgmap v17225: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T23:40:28.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:27 smithi122 ceph-mon[93869]: pgmap v17225: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T23:40:28.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:27 smithi181 ceph-mon[94189]: pgmap v17225: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T23:40:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31800.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:40:29.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31338.842136 secs 2023-03-09T23:40:29.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31338.842136 secs 2023-03-09T23:40:29.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31338.842136 secs 2023-03-09T23:40:30.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:29 smithi122 ceph-mon[93869]: pgmap v17226: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2023-03-09T23:40:30.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:29 smithi181 ceph-mon[94189]: pgmap v17226: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2023-03-09T23:40:30.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:29 smithi038 ceph-mon[92422]: pgmap v17226: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2023-03-09T23:40:31.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31341.430045 secs 2023-03-09T23:40:31.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31341.430045 secs 2023-03-09T23:40:31.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31341.430045 secs 2023-03-09T23:40:32.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:31 smithi122 ceph-mon[93869]: pgmap v17227: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2023-03-09T23:40:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:31 smithi181 ceph-mon[94189]: pgmap v17227: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2023-03-09T23:40:32.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:31 smithi038 ceph-mon[92422]: pgmap v17227: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2023-03-09T23:40:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31805.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:40:33.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31343.842259 secs 2023-03-09T23:40:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:33 smithi122 ceph-mon[93869]: pgmap v17228: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2023-03-09T23:40:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31343.842259 secs 2023-03-09T23:40:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:33 smithi181 ceph-mon[94189]: pgmap v17228: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2023-03-09T23:40:33.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31343.842259 secs 2023-03-09T23:40:33.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:33 smithi038 ceph-mon[92422]: pgmap v17228: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2023-03-09T23:40:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:35 smithi038 ceph-mon[92422]: pgmap v17229: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T23:40:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31346.430105 secs 2023-03-09T23:40:36.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:35 smithi122 ceph-mon[93869]: pgmap v17229: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T23:40:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31346.430105 secs 2023-03-09T23:40:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:35 smithi181 ceph-mon[94189]: pgmap v17229: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T23:40:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31346.430105 secs 2023-03-09T23:40:37.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:37 smithi038 ceph-mon[92422]: pgmap v17230: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T23:40:38.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:37 smithi122 ceph-mon[93869]: pgmap v17230: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T23:40:38.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:37 smithi181 ceph-mon[94189]: pgmap v17230: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T23:40:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31810.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:40:39.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31348.842375 secs 2023-03-09T23:40:39.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31348.842375 secs 2023-03-09T23:40:39.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31348.842375 secs 2023-03-09T23:40:40.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:39 smithi122 ceph-mon[93869]: pgmap v17231: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-09T23:40:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:39 smithi181 ceph-mon[94189]: pgmap v17231: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-09T23:40:40.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:39 smithi038 ceph-mon[92422]: pgmap v17231: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-09T23:40:41.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31351.430173 secs 2023-03-09T23:40:41.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31351.430173 secs 2023-03-09T23:40:41.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31351.430173 secs 2023-03-09T23:40:42.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:41 smithi122 ceph-mon[93869]: pgmap v17232: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-09T23:40:42.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:41 smithi181 ceph-mon[94189]: pgmap v17232: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-09T23:40:42.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:41 smithi038 ceph-mon[92422]: pgmap v17232: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-09T23:40:43.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31815.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:40:43.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:43 smithi122 ceph-mon[93869]: pgmap v17233: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 682 B/s wr, 7 op/s 2023-03-09T23:40:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:43 smithi181 ceph-mon[94189]: pgmap v17233: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 682 B/s wr, 7 op/s 2023-03-09T23:40:43.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:43 smithi038 ceph-mon[92422]: pgmap v17233: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 682 B/s wr, 7 op/s 2023-03-09T23:40:46.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:45 smithi122 ceph-mon[93869]: pgmap v17234: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 767 B/s wr, 6 op/s 2023-03-09T23:40:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31356.430232 secs 2023-03-09T23:40:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:45 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:40:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:45 smithi181 ceph-mon[94189]: pgmap v17234: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 767 B/s wr, 6 op/s 2023-03-09T23:40:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31356.430232 secs 2023-03-09T23:40:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:45 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:40:46.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:45 smithi038 ceph-mon[92422]: pgmap v17234: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 767 B/s wr, 6 op/s 2023-03-09T23:40:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31356.430232 secs 2023-03-09T23:40:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:45 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:40:47.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:46 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:40:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:46 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:40:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:46 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:40:47.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:46 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:40:47.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:46 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:40:47.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:46 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:40:47.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:46 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:40:47.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:46 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:40:47.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:46 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:40:48.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:47 smithi122 ceph-mon[93869]: pgmap v17235: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 106 KiB/s wr, 9 op/s 2023-03-09T23:40:48.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:47 smithi181 ceph-mon[94189]: pgmap v17235: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 106 KiB/s wr, 9 op/s 2023-03-09T23:40:48.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:47 smithi038 ceph-mon[92422]: pgmap v17235: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 106 KiB/s wr, 9 op/s 2023-03-09T23:40:48.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31820.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:40:49.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:40:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:40:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:49 smithi122 ceph-mon[93869]: pgmap v17236: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 706 KiB/s rd, 448 KiB/s wr, 8 op/s 2023-03-09T23:40:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:40:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:40:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:49 smithi181 ceph-mon[94189]: pgmap v17236: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 706 KiB/s rd, 448 KiB/s wr, 8 op/s 2023-03-09T23:40:49.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:40:49.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:40:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:49 smithi038 ceph-mon[92422]: pgmap v17236: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 706 KiB/s rd, 448 KiB/s wr, 8 op/s 2023-03-09T23:40:50.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31361.430293 secs 2023-03-09T23:40:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31361.430293 secs 2023-03-09T23:40:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31361.430293 secs 2023-03-09T23:40:51.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:40:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:40:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:51 smithi122 ceph-mon[93869]: pgmap v17237: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 890 KiB/s rd, 629 KiB/s wr, 10 op/s 2023-03-09T23:40:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:40:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:40:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:51 smithi181 ceph-mon[94189]: pgmap v17237: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 890 KiB/s rd, 629 KiB/s wr, 10 op/s 2023-03-09T23:40:51.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:40:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:40:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:51 smithi038 ceph-mon[92422]: pgmap v17237: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 890 KiB/s rd, 629 KiB/s wr, 10 op/s 2023-03-09T23:40:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31825.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:40:53.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:53 smithi122 ceph-mon[93869]: pgmap v17238: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 655 KiB/s wr, 11 op/s 2023-03-09T23:40:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:53 smithi181 ceph-mon[94189]: pgmap v17238: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 655 KiB/s wr, 11 op/s 2023-03-09T23:40:53.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:53 smithi038 ceph-mon[92422]: pgmap v17238: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 655 KiB/s wr, 11 op/s 2023-03-09T23:40:55.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:40:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:40:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:55 smithi122 ceph-mon[93869]: pgmap v17239: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 654 KiB/s wr, 10 op/s 2023-03-09T23:40:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:55 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:40:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31366.430357 secs 2023-03-09T23:40:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:40:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:40:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:55 smithi181 ceph-mon[94189]: pgmap v17239: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 654 KiB/s wr, 10 op/s 2023-03-09T23:40:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:55 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:40:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31366.430357 secs 2023-03-09T23:40:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:40:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:40:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:55 smithi038 ceph-mon[92422]: pgmap v17239: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 654 KiB/s wr, 10 op/s 2023-03-09T23:40:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:55 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:40:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31366.430357 secs 2023-03-09T23:40:57.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:40:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:40:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:57 smithi122 ceph-mon[93869]: pgmap v17240: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.1 MiB/s wr, 15 op/s 2023-03-09T23:40:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:40:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:40:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:57 smithi181 ceph-mon[94189]: pgmap v17240: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.1 MiB/s wr, 15 op/s 2023-03-09T23:40:57.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:40:57.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:40:57.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:57 smithi038 ceph-mon[92422]: pgmap v17240: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.1 MiB/s wr, 15 op/s 2023-03-09T23:40:58.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31830.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:40:59.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:40:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:40:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:40:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:40:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:40:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:40:59.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:40:59.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:40:59.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:40:59.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:40:59.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:40:59.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:40:59.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:40:59.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:40:59.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:40:59.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:40:59.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:40:59.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:40:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:40:59.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:40:59.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:40:59.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:40:59.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:40:59.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:40:59.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:40:59.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:40:59.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:40:59.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:40:59.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:40:59.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:40:59.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:40:59.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:40:59.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:40:59.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:40:59.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:40:59.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:40:59.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:40:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:40:59.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:40:59.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:40:59.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:40:59.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:40:59.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:40:59.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:40:59.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:40:59.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:40:59.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:40:59.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:40:59.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:40:59.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:40:59.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:40:59.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:40:59.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:40:59.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:40:59.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:40:59.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:40:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:41:00.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:00 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 187.7M 2023-03-09T23:41:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:00 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi181 to 196878999: error parsing value: Value '196878999' is below minimum 939524096 2023-03-09T23:41:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:00 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T23:41:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:00 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T23:41:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:00 smithi122 ceph-mon[93869]: pgmap v17241: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.0 MiB/s wr, 13 op/s 2023-03-09T23:41:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:41:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:41:00.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:00 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 187.7M 2023-03-09T23:41:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:00 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi181 to 196878999: error parsing value: Value '196878999' is below minimum 939524096 2023-03-09T23:41:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:00 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T23:41:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:00 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T23:41:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:00 smithi181 ceph-mon[94189]: pgmap v17241: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.0 MiB/s wr, 13 op/s 2023-03-09T23:41:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:41:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:41:00.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:00 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 187.7M 2023-03-09T23:41:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:00 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi181 to 196878999: error parsing value: Value '196878999' is below minimum 939524096 2023-03-09T23:41:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:00 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T23:41:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:00 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T23:41:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:00 smithi038 ceph-mon[92422]: pgmap v17241: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.0 MiB/s wr, 13 op/s 2023-03-09T23:41:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:41:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:41:01.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31371.430441 secs 2023-03-09T23:41:01.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31371.430441 secs 2023-03-09T23:41:01.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31371.430441 secs 2023-03-09T23:41:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:02 smithi122 ceph-mon[93869]: pgmap v17242: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1020 KiB/s rd, 751 KiB/s wr, 15 op/s 2023-03-09T23:41:02.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:02 smithi181 ceph-mon[94189]: pgmap v17242: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1020 KiB/s rd, 751 KiB/s wr, 15 op/s 2023-03-09T23:41:02.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:02 smithi038 ceph-mon[92422]: pgmap v17242: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1020 KiB/s rd, 751 KiB/s wr, 15 op/s 2023-03-09T23:41:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31835.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:41:03.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31373.842955 secs 2023-03-09T23:41:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:03 smithi122 ceph-mon[93869]: pgmap v17243: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 856 KiB/s rd, 590 KiB/s wr, 15 op/s 2023-03-09T23:41:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31373.842955 secs 2023-03-09T23:41:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:03 smithi181 ceph-mon[94189]: pgmap v17243: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 856 KiB/s rd, 590 KiB/s wr, 15 op/s 2023-03-09T23:41:03.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31373.842955 secs 2023-03-09T23:41:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:03 smithi038 ceph-mon[92422]: pgmap v17243: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 856 KiB/s rd, 590 KiB/s wr, 15 op/s 2023-03-09T23:41:05.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:05 smithi122 ceph-mon[93869]: osdmap e1293: 12 total, 12 up, 12 in 2023-03-09T23:41:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:05 smithi122 ceph-mon[93869]: pgmap v17245: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 31 op/s 2023-03-09T23:41:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31376.430502 secs 2023-03-09T23:41:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:05 smithi181 ceph-mon[94189]: osdmap e1293: 12 total, 12 up, 12 in 2023-03-09T23:41:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:05 smithi181 ceph-mon[94189]: pgmap v17245: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 31 op/s 2023-03-09T23:41:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31376.430502 secs 2023-03-09T23:41:05.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:05 smithi038 ceph-mon[92422]: osdmap e1293: 12 total, 12 up, 12 in 2023-03-09T23:41:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:05 smithi038 ceph-mon[92422]: pgmap v17245: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 31 op/s 2023-03-09T23:41:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31376.430502 secs 2023-03-09T23:41:06.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:06 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:41:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:06 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:41:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:41:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:06 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:41:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:06 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:41:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:41:06.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:06 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:41:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:06 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:41:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:41:07.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:41:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:41:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:07 smithi122 ceph-mon[93869]: pgmap v17246: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 541 KiB/s wr, 30 op/s 2023-03-09T23:41:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:41:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:41:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:07 smithi181 ceph-mon[94189]: pgmap v17246: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 541 KiB/s wr, 30 op/s 2023-03-09T23:41:07.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:41:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:41:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:07 smithi038 ceph-mon[92422]: pgmap v17246: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 541 KiB/s wr, 30 op/s 2023-03-09T23:41:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31840.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:41:08.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31378.843071 secs 2023-03-09T23:41:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31378.843071 secs 2023-03-09T23:41:08.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31378.843071 secs 2023-03-09T23:41:09.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:41:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:41:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:09 smithi122 ceph-mon[93869]: pgmap v17247: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 541 KiB/s wr, 30 op/s 2023-03-09T23:41:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:41:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:41:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:09 smithi181 ceph-mon[94189]: pgmap v17247: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 541 KiB/s wr, 30 op/s 2023-03-09T23:41:09.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:41:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:41:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:09 smithi038 ceph-mon[92422]: pgmap v17247: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 541 KiB/s wr, 30 op/s 2023-03-09T23:41:10.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31381.430561 secs 2023-03-09T23:41:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31381.430561 secs 2023-03-09T23:41:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31381.430561 secs 2023-03-09T23:41:11.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:11 smithi122 ceph-mon[93869]: pgmap v17248: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 445 KiB/s rd, 472 KiB/s wr, 28 op/s 2023-03-09T23:41:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:11 smithi181 ceph-mon[94189]: pgmap v17248: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 445 KiB/s rd, 472 KiB/s wr, 28 op/s 2023-03-09T23:41:11.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:11 smithi038 ceph-mon[92422]: pgmap v17248: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 445 KiB/s rd, 472 KiB/s wr, 28 op/s 2023-03-09T23:41:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31845.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:41:13.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31383.843194 secs 2023-03-09T23:41:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:13 smithi122 ceph-mon[93869]: osdmap e1294: 12 total, 12 up, 12 in 2023-03-09T23:41:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:13 smithi122 ceph-mon[93869]: pgmap v17250: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 175 KiB/s wr, 12 op/s 2023-03-09T23:41:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31383.843194 secs 2023-03-09T23:41:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:13 smithi181 ceph-mon[94189]: osdmap e1294: 12 total, 12 up, 12 in 2023-03-09T23:41:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:13 smithi181 ceph-mon[94189]: pgmap v17250: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 175 KiB/s wr, 12 op/s 2023-03-09T23:41:13.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31383.843194 secs 2023-03-09T23:41:13.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:13 smithi038 ceph-mon[92422]: osdmap e1294: 12 total, 12 up, 12 in 2023-03-09T23:41:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:13 smithi038 ceph-mon[92422]: pgmap v17250: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 175 KiB/s wr, 12 op/s 2023-03-09T23:41:16.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:15 smithi122 ceph-mon[93869]: pgmap v17251: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 177 KiB/s wr, 11 op/s 2023-03-09T23:41:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31386.430620 secs 2023-03-09T23:41:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:15 smithi181 ceph-mon[94189]: pgmap v17251: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 177 KiB/s wr, 11 op/s 2023-03-09T23:41:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31386.430620 secs 2023-03-09T23:41:16.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:15 smithi038 ceph-mon[92422]: pgmap v17251: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 177 KiB/s wr, 11 op/s 2023-03-09T23:41:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31386.430620 secs 2023-03-09T23:41:18.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:17 smithi122 ceph-mon[93869]: pgmap v17252: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 164 KiB/s wr, 9 op/s 2023-03-09T23:41:18.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:17 smithi181 ceph-mon[94189]: pgmap v17252: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 164 KiB/s wr, 9 op/s 2023-03-09T23:41:18.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:17 smithi038 ceph-mon[92422]: pgmap v17252: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 164 KiB/s wr, 9 op/s 2023-03-09T23:41:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31850.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:41:19.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31388.843307 secs 2023-03-09T23:41:19.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31388.843307 secs 2023-03-09T23:41:19.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31388.843307 secs 2023-03-09T23:41:20.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:19 smithi122 ceph-mon[93869]: pgmap v17253: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 163 KiB/s wr, 8 op/s 2023-03-09T23:41:20.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:19 smithi181 ceph-mon[94189]: pgmap v17253: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 163 KiB/s wr, 8 op/s 2023-03-09T23:41:20.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:19 smithi038 ceph-mon[92422]: pgmap v17253: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 163 KiB/s wr, 8 op/s 2023-03-09T23:41:21.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31391.430705 secs 2023-03-09T23:41:21.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31391.430705 secs 2023-03-09T23:41:21.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31391.430705 secs 2023-03-09T23:41:22.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:21 smithi122 ceph-mon[93869]: pgmap v17254: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 163 KiB/s wr, 8 op/s 2023-03-09T23:41:22.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:21 smithi181 ceph-mon[94189]: pgmap v17254: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 163 KiB/s wr, 8 op/s 2023-03-09T23:41:22.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:21 smithi038 ceph-mon[92422]: pgmap v17254: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 163 KiB/s wr, 8 op/s 2023-03-09T23:41:23.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31855.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:41:23.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31393.843428 secs 2023-03-09T23:41:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:23 smithi122 ceph-mon[93869]: pgmap v17255: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 35 KiB/s wr, 8 op/s 2023-03-09T23:41:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31393.843428 secs 2023-03-09T23:41:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:23 smithi181 ceph-mon[94189]: pgmap v17255: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 35 KiB/s wr, 8 op/s 2023-03-09T23:41:23.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31393.843428 secs 2023-03-09T23:41:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:23 smithi038 ceph-mon[92422]: pgmap v17255: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 35 KiB/s wr, 8 op/s 2023-03-09T23:41:26.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:25 smithi122 ceph-mon[93869]: pgmap v17256: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 7 op/s 2023-03-09T23:41:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31396.430765 secs 2023-03-09T23:41:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:25 smithi181 ceph-mon[94189]: pgmap v17256: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 7 op/s 2023-03-09T23:41:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31396.430765 secs 2023-03-09T23:41:26.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:25 smithi038 ceph-mon[92422]: pgmap v17256: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 7 op/s 2023-03-09T23:41:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31396.430765 secs 2023-03-09T23:41:28.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:27 smithi122 ceph-mon[93869]: pgmap v17257: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T23:41:28.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:27 smithi181 ceph-mon[94189]: pgmap v17257: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T23:41:28.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:27 smithi038 ceph-mon[92422]: pgmap v17257: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T23:41:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31860.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:41:29.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31398.843550 secs 2023-03-09T23:41:29.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31398.843550 secs 2023-03-09T23:41:29.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31398.843550 secs 2023-03-09T23:41:30.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:29 smithi122 ceph-mon[93869]: pgmap v17258: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 7 op/s 2023-03-09T23:41:30.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:29 smithi181 ceph-mon[94189]: pgmap v17258: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 7 op/s 2023-03-09T23:41:30.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:29 smithi038 ceph-mon[92422]: pgmap v17258: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 7 op/s 2023-03-09T23:41:31.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31401.430826 secs 2023-03-09T23:41:31.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31401.430826 secs 2023-03-09T23:41:31.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31401.430826 secs 2023-03-09T23:41:32.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:31 smithi122 ceph-mon[93869]: pgmap v17259: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 8 op/s 2023-03-09T23:41:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:31 smithi181 ceph-mon[94189]: pgmap v17259: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 8 op/s 2023-03-09T23:41:32.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:31 smithi038 ceph-mon[92422]: pgmap v17259: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 8 op/s 2023-03-09T23:41:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31865.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:41:33.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31403.843666 secs 2023-03-09T23:41:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:33 smithi122 ceph-mon[93869]: pgmap v17260: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2023-03-09T23:41:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31403.843666 secs 2023-03-09T23:41:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:33 smithi181 ceph-mon[94189]: pgmap v17260: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2023-03-09T23:41:33.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31403.843666 secs 2023-03-09T23:41:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:33 smithi038 ceph-mon[92422]: pgmap v17260: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2023-03-09T23:41:36.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:35 smithi122 ceph-mon[93869]: pgmap v17261: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T23:41:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31406.430889 secs 2023-03-09T23:41:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:35 smithi181 ceph-mon[94189]: pgmap v17261: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T23:41:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31406.430889 secs 2023-03-09T23:41:36.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:35 smithi038 ceph-mon[92422]: pgmap v17261: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T23:41:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31406.430889 secs 2023-03-09T23:41:38.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:37 smithi122 ceph-mon[93869]: pgmap v17262: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 8 op/s 2023-03-09T23:41:38.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:37 smithi181 ceph-mon[94189]: pgmap v17262: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 8 op/s 2023-03-09T23:41:38.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:37 smithi038 ceph-mon[92422]: pgmap v17262: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 8 op/s 2023-03-09T23:41:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31870.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:41:39.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31408.843783 secs 2023-03-09T23:41:39.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31408.843783 secs 2023-03-09T23:41:39.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31408.843783 secs 2023-03-09T23:41:40.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:39 smithi122 ceph-mon[93869]: pgmap v17263: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 7 op/s 2023-03-09T23:41:40.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:39 smithi181 ceph-mon[94189]: pgmap v17263: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 7 op/s 2023-03-09T23:41:40.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:39 smithi038 ceph-mon[92422]: pgmap v17263: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 7 op/s 2023-03-09T23:41:41.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31411.430968 secs 2023-03-09T23:41:41.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31411.430968 secs 2023-03-09T23:41:41.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31411.430968 secs 2023-03-09T23:41:42.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:41 smithi122 ceph-mon[93869]: pgmap v17264: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 511 B/s wr, 9 op/s 2023-03-09T23:41:42.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:41 smithi181 ceph-mon[94189]: pgmap v17264: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 511 B/s wr, 9 op/s 2023-03-09T23:41:42.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:41 smithi038 ceph-mon[92422]: pgmap v17264: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 511 B/s wr, 9 op/s 2023-03-09T23:41:43.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31875.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:41:43.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:43 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 31413.843909 secs 2023-03-09T23:41:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:43 smithi122 ceph-mon[93869]: pgmap v17265: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 682 B/s wr, 8 op/s 2023-03-09T23:41:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:43 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 31413.843909 secs 2023-03-09T23:41:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:43 smithi181 ceph-mon[94189]: pgmap v17265: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 682 B/s wr, 8 op/s 2023-03-09T23:41:43.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:43 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 31413.843909 secs 2023-03-09T23:41:43.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:43 smithi038 ceph-mon[92422]: pgmap v17265: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 682 B/s wr, 8 op/s 2023-03-09T23:41:46.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:45 smithi122 ceph-mon[93869]: pgmap v17266: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 8 op/s 2023-03-09T23:41:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31416.430980 secs 2023-03-09T23:41:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:45 smithi181 ceph-mon[94189]: pgmap v17266: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 8 op/s 2023-03-09T23:41:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31416.430980 secs 2023-03-09T23:41:46.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:45 smithi038 ceph-mon[92422]: pgmap v17266: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 8 op/s 2023-03-09T23:41:46.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31416.430980 secs 2023-03-09T23:41:47.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:46 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:41:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:46 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:41:47.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:46 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:41:47.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:46 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:41:47.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:46 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:41:47.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:46 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:41:48.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:47 smithi122 ceph-mon[93869]: pgmap v17267: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 99 KiB/s wr, 10 op/s 2023-03-09T23:41:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:41:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:41:48.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:47 smithi181 ceph-mon[94189]: pgmap v17267: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 99 KiB/s wr, 10 op/s 2023-03-09T23:41:48.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:41:48.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:41:48.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:47 smithi038 ceph-mon[92422]: pgmap v17267: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 99 KiB/s wr, 10 op/s 2023-03-09T23:41:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:41:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:41:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31880.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:41:49.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:48 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 31418.843991 secs 2023-03-09T23:41:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:41:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:41:49.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:48 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 31418.843991 secs 2023-03-09T23:41:49.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:41:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:41:49.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:48 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 31418.843991 secs 2023-03-09T23:41:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:41:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:41:50.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:49 smithi122 ceph-mon[93869]: pgmap v17268: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 336 KiB/s rd, 334 KiB/s wr, 9 op/s 2023-03-09T23:41:50.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:49 smithi181 ceph-mon[94189]: pgmap v17268: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 336 KiB/s rd, 334 KiB/s wr, 9 op/s 2023-03-09T23:41:50.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:49 smithi038 ceph-mon[92422]: pgmap v17268: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 336 KiB/s rd, 334 KiB/s wr, 9 op/s 2023-03-09T23:41:51.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31421.431060 secs 2023-03-09T23:41:51.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31421.431060 secs 2023-03-09T23:41:51.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31421.431060 secs 2023-03-09T23:41:52.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:51 smithi122 ceph-mon[93869]: pgmap v17269: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 611 KiB/s rd, 606 KiB/s wr, 10 op/s 2023-03-09T23:41:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:41:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:41:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:51 smithi181 ceph-mon[94189]: pgmap v17269: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 611 KiB/s rd, 606 KiB/s wr, 10 op/s 2023-03-09T23:41:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:41:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:41:52.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:51 smithi038 ceph-mon[92422]: pgmap v17269: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 611 KiB/s rd, 606 KiB/s wr, 10 op/s 2023-03-09T23:41:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:41:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:41:53.806 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31885.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:41:53.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31423.844129 secs 2023-03-09T23:41:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:53 smithi122 ceph-mon[93869]: pgmap v17270: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 637 KiB/s rd, 633 KiB/s wr, 10 op/s 2023-03-09T23:41:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31423.844129 secs 2023-03-09T23:41:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:53 smithi181 ceph-mon[94189]: pgmap v17270: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 637 KiB/s rd, 633 KiB/s wr, 10 op/s 2023-03-09T23:41:53.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31423.844129 secs 2023-03-09T23:41:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:53 smithi038 ceph-mon[92422]: pgmap v17270: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 637 KiB/s rd, 633 KiB/s wr, 10 op/s 2023-03-09T23:41:54.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:54 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:41:54.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:54 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:41:54.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:54 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:41:54.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:54 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:41:54.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:54 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:41:54.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:54 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:41:55.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:41:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:41:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:55 smithi122 ceph-mon[93869]: pgmap v17271: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 673 KiB/s rd, 658 KiB/s wr, 10 op/s 2023-03-09T23:41:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31426.431121 secs 2023-03-09T23:41:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:41:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:41:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:55 smithi181 ceph-mon[94189]: pgmap v17271: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 673 KiB/s rd, 658 KiB/s wr, 10 op/s 2023-03-09T23:41:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31426.431121 secs 2023-03-09T23:41:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:41:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:41:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:55 smithi038 ceph-mon[92422]: pgmap v17271: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 673 KiB/s rd, 658 KiB/s wr, 10 op/s 2023-03-09T23:41:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31426.431121 secs 2023-03-09T23:41:57.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:41:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:41:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:57 smithi122 ceph-mon[93869]: pgmap v17272: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 13 op/s 2023-03-09T23:41:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:41:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:41:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:57 smithi181 ceph-mon[94189]: pgmap v17272: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 13 op/s 2023-03-09T23:41:57.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:41:57.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:41:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:57 smithi038 ceph-mon[92422]: pgmap v17272: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 13 op/s 2023-03-09T23:41:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31890.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:41:58.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31428.844248 secs 2023-03-09T23:41:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31428.844248 secs 2023-03-09T23:41:58.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31428.844248 secs 2023-03-09T23:41:59.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:41:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:41:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:59 smithi122 ceph-mon[93869]: pgmap v17273: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1017 KiB/s wr, 12 op/s 2023-03-09T23:41:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:41:59.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:41:59.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:41:59.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:41:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:41:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:41:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:41:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:59 smithi181 ceph-mon[94189]: pgmap v17273: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1017 KiB/s wr, 12 op/s 2023-03-09T23:41:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:41:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:41:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:41:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:41:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:41:59.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:41:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:41:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:59 smithi038 ceph-mon[92422]: pgmap v17273: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1017 KiB/s wr, 12 op/s 2023-03-09T23:41:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:41:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:41:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:41:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:41:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:42:00.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31431.431206 secs 2023-03-09T23:42:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31431.431206 secs 2023-03-09T23:42:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31431.431206 secs 2023-03-09T23:42:01.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:42:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:42:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:01 smithi122 ceph-mon[93869]: pgmap v17274: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1001 KiB/s rd, 986 KiB/s wr, 14 op/s 2023-03-09T23:42:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:42:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:42:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:01 smithi181 ceph-mon[94189]: pgmap v17274: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1001 KiB/s rd, 986 KiB/s wr, 14 op/s 2023-03-09T23:42:01.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:42:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:42:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:01 smithi038 ceph-mon[92422]: pgmap v17274: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1001 KiB/s rd, 986 KiB/s wr, 14 op/s 2023-03-09T23:42:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31895.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:42:03.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31433.844376 secs 2023-03-09T23:42:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:03 smithi122 ceph-mon[93869]: pgmap v17275: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 727 KiB/s rd, 715 KiB/s wr, 13 op/s 2023-03-09T23:42:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31433.844376 secs 2023-03-09T23:42:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:03 smithi181 ceph-mon[94189]: pgmap v17275: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 727 KiB/s rd, 715 KiB/s wr, 13 op/s 2023-03-09T23:42:03.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31433.844376 secs 2023-03-09T23:42:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:03 smithi038 ceph-mon[92422]: pgmap v17275: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 727 KiB/s rd, 715 KiB/s wr, 13 op/s 2023-03-09T23:42:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:05 smithi122 ceph-mon[93869]: osdmap e1295: 12 total, 12 up, 12 in 2023-03-09T23:42:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:05 smithi122 ceph-mon[93869]: pgmap v17277: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 861 KiB/s rd, 877 KiB/s wr, 28 op/s 2023-03-09T23:42:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31436.431267 secs 2023-03-09T23:42:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:05 smithi181 ceph-mon[94189]: osdmap e1295: 12 total, 12 up, 12 in 2023-03-09T23:42:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:05 smithi181 ceph-mon[94189]: pgmap v17277: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 861 KiB/s rd, 877 KiB/s wr, 28 op/s 2023-03-09T23:42:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31436.431267 secs 2023-03-09T23:42:05.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:05 smithi038 ceph-mon[92422]: osdmap e1295: 12 total, 12 up, 12 in 2023-03-09T23:42:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:05 smithi038 ceph-mon[92422]: pgmap v17277: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 861 KiB/s rd, 877 KiB/s wr, 28 op/s 2023-03-09T23:42:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31436.431267 secs 2023-03-09T23:42:07.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:42:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:42:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:07 smithi122 ceph-mon[93869]: pgmap v17278: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 535 KiB/s rd, 577 KiB/s wr, 30 op/s 2023-03-09T23:42:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:42:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:42:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:07 smithi181 ceph-mon[94189]: pgmap v17278: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 535 KiB/s rd, 577 KiB/s wr, 30 op/s 2023-03-09T23:42:07.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:42:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:42:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:07 smithi038 ceph-mon[92422]: pgmap v17278: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 535 KiB/s rd, 577 KiB/s wr, 30 op/s 2023-03-09T23:42:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31900.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:42:08.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31438.844494 secs 2023-03-09T23:42:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31438.844494 secs 2023-03-09T23:42:08.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31438.844494 secs 2023-03-09T23:42:09.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:42:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:42:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:09 smithi122 ceph-mon[93869]: pgmap v17279: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 535 KiB/s rd, 577 KiB/s wr, 29 op/s 2023-03-09T23:42:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:42:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:42:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:09 smithi181 ceph-mon[94189]: pgmap v17279: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 535 KiB/s rd, 577 KiB/s wr, 29 op/s 2023-03-09T23:42:09.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:42:09.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:42:09.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:09 smithi038 ceph-mon[92422]: pgmap v17279: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 535 KiB/s rd, 577 KiB/s wr, 29 op/s 2023-03-09T23:42:10.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31441.431324 secs 2023-03-09T23:42:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31441.431324 secs 2023-03-09T23:42:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31441.431324 secs 2023-03-09T23:42:11.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:11 smithi122 ceph-mon[93869]: pgmap v17280: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 331 KiB/s wr, 27 op/s 2023-03-09T23:42:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:11 smithi181 ceph-mon[94189]: pgmap v17280: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 331 KiB/s wr, 27 op/s 2023-03-09T23:42:11.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:11 smithi038 ceph-mon[92422]: pgmap v17280: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 331 KiB/s wr, 27 op/s 2023-03-09T23:42:13.793 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31905.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:42:13.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31443.844595 secs 2023-03-09T23:42:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:13 smithi122 ceph-mon[93869]: osdmap e1296: 12 total, 12 up, 12 in 2023-03-09T23:42:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:13 smithi122 ceph-mon[93869]: pgmap v17282: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 509 KiB/s wr, 14 op/s 2023-03-09T23:42:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31443.844595 secs 2023-03-09T23:42:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:13 smithi181 ceph-mon[94189]: osdmap e1296: 12 total, 12 up, 12 in 2023-03-09T23:42:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:13 smithi181 ceph-mon[94189]: pgmap v17282: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 509 KiB/s wr, 14 op/s 2023-03-09T23:42:13.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31443.844595 secs 2023-03-09T23:42:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:13 smithi038 ceph-mon[92422]: osdmap e1296: 12 total, 12 up, 12 in 2023-03-09T23:42:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:13 smithi038 ceph-mon[92422]: pgmap v17282: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 509 KiB/s wr, 14 op/s 2023-03-09T23:42:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:15 smithi122 ceph-mon[93869]: pgmap v17283: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 412 KiB/s wr, 12 op/s 2023-03-09T23:42:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31446.431386 secs 2023-03-09T23:42:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:15 smithi181 ceph-mon[94189]: pgmap v17283: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 412 KiB/s wr, 12 op/s 2023-03-09T23:42:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31446.431386 secs 2023-03-09T23:42:16.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:15 smithi038 ceph-mon[92422]: pgmap v17283: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 412 KiB/s wr, 12 op/s 2023-03-09T23:42:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31446.431386 secs 2023-03-09T23:42:18.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:17 smithi122 ceph-mon[93869]: pgmap v17284: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 164 KiB/s wr, 8 op/s 2023-03-09T23:42:18.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:17 smithi181 ceph-mon[94189]: pgmap v17284: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 164 KiB/s wr, 8 op/s 2023-03-09T23:42:18.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:17 smithi038 ceph-mon[92422]: pgmap v17284: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 164 KiB/s wr, 8 op/s 2023-03-09T23:42:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31910.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:42:19.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31448.844717 secs 2023-03-09T23:42:19.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31448.844717 secs 2023-03-09T23:42:19.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31448.844717 secs 2023-03-09T23:42:20.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:19 smithi122 ceph-mon[93869]: pgmap v17285: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 164 KiB/s wr, 7 op/s 2023-03-09T23:42:20.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:19 smithi181 ceph-mon[94189]: pgmap v17285: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 164 KiB/s wr, 7 op/s 2023-03-09T23:42:20.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:19 smithi038 ceph-mon[92422]: pgmap v17285: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 164 KiB/s wr, 7 op/s 2023-03-09T23:42:21.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31451.431440 secs 2023-03-09T23:42:21.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31451.431440 secs 2023-03-09T23:42:21.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31451.431440 secs 2023-03-09T23:42:22.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:21 smithi122 ceph-mon[93869]: pgmap v17286: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 163 KiB/s wr, 7 op/s 2023-03-09T23:42:22.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:21 smithi181 ceph-mon[94189]: pgmap v17286: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 163 KiB/s wr, 7 op/s 2023-03-09T23:42:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:21 smithi038 ceph-mon[92422]: pgmap v17286: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 163 KiB/s wr, 7 op/s 2023-03-09T23:42:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31915.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:42:23.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31453.844845 secs 2023-03-09T23:42:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:23 smithi122 ceph-mon[93869]: pgmap v17287: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-03-09T23:42:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31453.844845 secs 2023-03-09T23:42:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:23 smithi181 ceph-mon[94189]: pgmap v17287: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-03-09T23:42:23.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31453.844845 secs 2023-03-09T23:42:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:23 smithi038 ceph-mon[92422]: pgmap v17287: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-03-09T23:42:26.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:25 smithi122 ceph-mon[93869]: pgmap v17288: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T23:42:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31456.431501 secs 2023-03-09T23:42:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:25 smithi181 ceph-mon[94189]: pgmap v17288: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T23:42:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31456.431501 secs 2023-03-09T23:42:26.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:25 smithi038 ceph-mon[92422]: pgmap v17288: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-09T23:42:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31456.431501 secs 2023-03-09T23:42:28.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:27 smithi122 ceph-mon[93869]: pgmap v17289: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T23:42:28.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:27 smithi181 ceph-mon[94189]: pgmap v17289: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T23:42:28.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:27 smithi038 ceph-mon[92422]: pgmap v17289: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T23:42:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31920.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:42:29.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31458.844966 secs 2023-03-09T23:42:29.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31458.844966 secs 2023-03-09T23:42:29.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31458.844966 secs 2023-03-09T23:42:30.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:29 smithi122 ceph-mon[93869]: pgmap v17290: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T23:42:30.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:29 smithi181 ceph-mon[94189]: pgmap v17290: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T23:42:30.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:29 smithi038 ceph-mon[92422]: pgmap v17290: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T23:42:31.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31461.431562 secs 2023-03-09T23:42:31.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31461.431562 secs 2023-03-09T23:42:31.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31461.431562 secs 2023-03-09T23:42:32.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:31 smithi122 ceph-mon[93869]: pgmap v17291: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T23:42:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:31 smithi181 ceph-mon[94189]: pgmap v17291: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T23:42:32.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:31 smithi038 ceph-mon[92422]: pgmap v17291: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T23:42:33.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31925.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:42:33.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31463.845116 secs 2023-03-09T23:42:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:33 smithi122 ceph-mon[93869]: pgmap v17292: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 46 KiB/s wr, 7 op/s 2023-03-09T23:42:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31463.845116 secs 2023-03-09T23:42:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:33 smithi181 ceph-mon[94189]: pgmap v17292: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 46 KiB/s wr, 7 op/s 2023-03-09T23:42:33.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31463.845116 secs 2023-03-09T23:42:33.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:33 smithi038 ceph-mon[92422]: pgmap v17292: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 46 KiB/s wr, 7 op/s 2023-03-09T23:42:36.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:35 smithi122 ceph-mon[93869]: pgmap v17293: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 47 KiB/s wr, 7 op/s 2023-03-09T23:42:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31466.431631 secs 2023-03-09T23:42:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:35 smithi181 ceph-mon[94189]: pgmap v17293: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 47 KiB/s wr, 7 op/s 2023-03-09T23:42:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31466.431631 secs 2023-03-09T23:42:36.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:35 smithi038 ceph-mon[92422]: pgmap v17293: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 47 KiB/s wr, 7 op/s 2023-03-09T23:42:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31466.431631 secs 2023-03-09T23:42:38.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:37 smithi122 ceph-mon[93869]: pgmap v17294: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 116 KiB/s wr, 9 op/s 2023-03-09T23:42:38.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:37 smithi181 ceph-mon[94189]: pgmap v17294: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 116 KiB/s wr, 9 op/s 2023-03-09T23:42:38.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:37 smithi038 ceph-mon[92422]: pgmap v17294: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 116 KiB/s wr, 9 op/s 2023-03-09T23:42:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31930.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:42:39.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31468.845233 secs 2023-03-09T23:42:39.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31468.845233 secs 2023-03-09T23:42:39.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31468.845233 secs 2023-03-09T23:42:40.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:39 smithi122 ceph-mon[93869]: pgmap v17295: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 117 KiB/s wr, 7 op/s 2023-03-09T23:42:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:39 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:42:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:39 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:42:40.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:39 smithi181 ceph-mon[94189]: pgmap v17295: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 117 KiB/s wr, 7 op/s 2023-03-09T23:42:40.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:39 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:42:40.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:39 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:42:40.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:39 smithi038 ceph-mon[92422]: pgmap v17295: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 117 KiB/s wr, 7 op/s 2023-03-09T23:42:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:39 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:42:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:39 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:42:41.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:42:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:42:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31471.431691 secs 2023-03-09T23:42:41.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:42:41.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:42:41.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31471.431691 secs 2023-03-09T23:42:41.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:42:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:42:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31471.431691 secs 2023-03-09T23:42:42.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:41 smithi122 ceph-mon[93869]: pgmap v17296: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 117 KiB/s wr, 8 op/s 2023-03-09T23:42:42.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:41 smithi181 ceph-mon[94189]: pgmap v17296: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 117 KiB/s wr, 8 op/s 2023-03-09T23:42:42.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:41 smithi038 ceph-mon[92422]: pgmap v17296: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 117 KiB/s wr, 8 op/s 2023-03-09T23:42:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31935.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:42:43.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31473.845314 secs 2023-03-09T23:42:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:43 smithi122 ceph-mon[93869]: pgmap v17297: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 490 KiB/s rd, 389 KiB/s wr, 9 op/s 2023-03-09T23:42:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31473.845314 secs 2023-03-09T23:42:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:43 smithi181 ceph-mon[94189]: pgmap v17297: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 490 KiB/s rd, 389 KiB/s wr, 9 op/s 2023-03-09T23:42:43.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31473.845314 secs 2023-03-09T23:42:43.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:43 smithi038 ceph-mon[92422]: pgmap v17297: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 490 KiB/s rd, 389 KiB/s wr, 9 op/s 2023-03-09T23:42:46.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:45 smithi122 ceph-mon[93869]: pgmap v17298: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 444 KiB/s rd, 342 KiB/s wr, 8 op/s 2023-03-09T23:42:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31476.431751 secs 2023-03-09T23:42:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:45 smithi181 ceph-mon[94189]: pgmap v17298: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 444 KiB/s rd, 342 KiB/s wr, 8 op/s 2023-03-09T23:42:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31476.431751 secs 2023-03-09T23:42:46.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:45 smithi038 ceph-mon[92422]: pgmap v17298: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 444 KiB/s rd, 342 KiB/s wr, 8 op/s 2023-03-09T23:42:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31476.431751 secs 2023-03-09T23:42:48.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:47 smithi122 ceph-mon[93869]: pgmap v17299: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 364 KiB/s wr, 10 op/s 2023-03-09T23:42:48.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:47 smithi181 ceph-mon[94189]: pgmap v17299: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 364 KiB/s wr, 10 op/s 2023-03-09T23:42:48.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:47 smithi038 ceph-mon[92422]: pgmap v17299: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 364 KiB/s wr, 10 op/s 2023-03-09T23:42:48.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31940.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:42:49.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31478.845420 secs 2023-03-09T23:42:49.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31478.845420 secs 2023-03-09T23:42:49.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31478.845420 secs 2023-03-09T23:42:50.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:49 smithi122 ceph-mon[93869]: pgmap v17300: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 828 KiB/s rd, 295 KiB/s wr, 7 op/s 2023-03-09T23:42:50.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:49 smithi181 ceph-mon[94189]: pgmap v17300: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 828 KiB/s rd, 295 KiB/s wr, 7 op/s 2023-03-09T23:42:50.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:49 smithi038 ceph-mon[92422]: pgmap v17300: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 828 KiB/s rd, 295 KiB/s wr, 7 op/s 2023-03-09T23:42:51.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31481.431811 secs 2023-03-09T23:42:51.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31481.431811 secs 2023-03-09T23:42:51.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31481.431811 secs 2023-03-09T23:42:52.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:51 smithi122 ceph-mon[93869]: pgmap v17301: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 614 KiB/s wr, 9 op/s 2023-03-09T23:42:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:51 smithi181 ceph-mon[94189]: pgmap v17301: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 614 KiB/s wr, 9 op/s 2023-03-09T23:42:52.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:51 smithi038 ceph-mon[92422]: pgmap v17301: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 614 KiB/s wr, 9 op/s 2023-03-09T23:42:53.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:42:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:42:53.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:42:53.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:42:53.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:42:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:42:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31945.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:42:54.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31483.845533 secs 2023-03-09T23:42:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:53 smithi122 ceph-mon[93869]: pgmap v17302: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 617 KiB/s wr, 10 op/s 2023-03-09T23:42:54.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31483.845533 secs 2023-03-09T23:42:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:53 smithi181 ceph-mon[94189]: pgmap v17302: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 617 KiB/s wr, 10 op/s 2023-03-09T23:42:54.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31483.845533 secs 2023-03-09T23:42:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:53 smithi038 ceph-mon[92422]: pgmap v17302: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 617 KiB/s wr, 10 op/s 2023-03-09T23:42:56.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:55 smithi122 ceph-mon[93869]: pgmap v17303: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 458 KiB/s wr, 9 op/s 2023-03-09T23:42:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:42:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:42:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31486.431896 secs 2023-03-09T23:42:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:55 smithi181 ceph-mon[94189]: pgmap v17303: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 458 KiB/s wr, 9 op/s 2023-03-09T23:42:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:42:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:42:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31486.431896 secs 2023-03-09T23:42:56.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:55 smithi038 ceph-mon[92422]: pgmap v17303: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 458 KiB/s wr, 9 op/s 2023-03-09T23:42:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:42:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:42:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31486.431896 secs 2023-03-09T23:42:57.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:42:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:42:57.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:42:57.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:42:57.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:42:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:42:58.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:57 smithi122 ceph-mon[93869]: pgmap v17304: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 459 KiB/s wr, 11 op/s 2023-03-09T23:42:58.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:57 smithi181 ceph-mon[94189]: pgmap v17304: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 459 KiB/s wr, 11 op/s 2023-03-09T23:42:58.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:57 smithi038 ceph-mon[92422]: pgmap v17304: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 459 KiB/s wr, 11 op/s 2023-03-09T23:42:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31950.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:43:00.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:59 smithi122 ceph-mon[93869]: pgmap v17305: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 691 KiB/s rd, 695 KiB/s wr, 10 op/s 2023-03-09T23:43:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:43:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:43:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:42:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:43:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:59 smithi181 ceph-mon[94189]: pgmap v17305: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 691 KiB/s rd, 695 KiB/s wr, 10 op/s 2023-03-09T23:43:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:43:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:43:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:42:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:43:00.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:59 smithi038 ceph-mon[92422]: pgmap v17305: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 691 KiB/s rd, 695 KiB/s wr, 10 op/s 2023-03-09T23:43:00.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:43:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:43:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:42:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:43:01.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:43:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:43:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31491.431957 secs 2023-03-09T23:43:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:43:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:43:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:43:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:43:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:43:01.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:43:01.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:43:01.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31491.431957 secs 2023-03-09T23:43:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:43:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:43:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:43:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:43:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:43:01.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:43:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:43:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31491.431957 secs 2023-03-09T23:43:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:43:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:43:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:43:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:43:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:43:02.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:02 smithi122 ceph-mon[93869]: pgmap v17306: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 14 op/s 2023-03-09T23:43:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:02 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:43:02.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:02 smithi181 ceph-mon[94189]: pgmap v17306: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 14 op/s 2023-03-09T23:43:02.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:02 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:43:02.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:02 smithi038 ceph-mon[92422]: pgmap v17306: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 14 op/s 2023-03-09T23:43:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:02 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:43:03.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:03 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:43:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:03 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:43:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:03 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:43:03.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:03 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:43:03.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:03 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:43:03.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:03 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:43:03.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:03 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:43:03.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:03 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:43:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:03 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:43:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31955.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:43:04.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:04 smithi122 ceph-mon[93869]: pgmap v17307: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 790 KiB/s rd, 783 KiB/s wr, 13 op/s 2023-03-09T23:43:04.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:04 smithi181 ceph-mon[94189]: pgmap v17307: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 790 KiB/s rd, 783 KiB/s wr, 13 op/s 2023-03-09T23:43:04.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:04 smithi038 ceph-mon[92422]: pgmap v17307: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 790 KiB/s rd, 783 KiB/s wr, 13 op/s 2023-03-09T23:43:06.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:06 smithi122 ceph-mon[93869]: pgmap v17308: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 838 KiB/s rd, 828 KiB/s wr, 14 op/s 2023-03-09T23:43:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31496.431982 secs 2023-03-09T23:43:06.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:06 smithi181 ceph-mon[94189]: pgmap v17308: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 838 KiB/s rd, 828 KiB/s wr, 14 op/s 2023-03-09T23:43:06.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31496.431982 secs 2023-03-09T23:43:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:06 smithi038 ceph-mon[92422]: pgmap v17308: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 838 KiB/s rd, 828 KiB/s wr, 14 op/s 2023-03-09T23:43:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31496.431982 secs 2023-03-09T23:43:08.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:08 smithi122 ceph-mon[93869]: pgmap v17309: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 728 KiB/s rd, 737 KiB/s wr, 17 op/s 2023-03-09T23:43:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:43:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:43:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31498.845925 secs 2023-03-09T23:43:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:08 smithi181 ceph-mon[94189]: pgmap v17309: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 728 KiB/s rd, 737 KiB/s wr, 17 op/s 2023-03-09T23:43:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:43:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:43:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31498.845925 secs 2023-03-09T23:43:08.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:08 smithi038 ceph-mon[92422]: pgmap v17309: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 728 KiB/s rd, 737 KiB/s wr, 17 op/s 2023-03-09T23:43:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:43:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:43:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31498.845925 secs 2023-03-09T23:43:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31960.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:43:09.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:43:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:43:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:09 smithi122 ceph-mon[93869]: pgmap v17310: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 811 KiB/s rd, 839 KiB/s wr, 27 op/s 2023-03-09T23:43:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:43:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:43:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:09 smithi181 ceph-mon[94189]: pgmap v17310: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 811 KiB/s rd, 839 KiB/s wr, 27 op/s 2023-03-09T23:43:09.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:43:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:43:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:09 smithi038 ceph-mon[92422]: pgmap v17310: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 811 KiB/s rd, 839 KiB/s wr, 27 op/s 2023-03-09T23:43:10.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:10 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:43:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:10 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:43:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:10 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:43:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:10 smithi122 ceph-mon[93869]: osdmap e1297: 12 total, 12 up, 12 in 2023-03-09T23:43:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31501.432044 secs 2023-03-09T23:43:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:10 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:43:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:10 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:43:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:10 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:43:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:10 smithi181 ceph-mon[94189]: osdmap e1297: 12 total, 12 up, 12 in 2023-03-09T23:43:10.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31501.432044 secs 2023-03-09T23:43:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:10 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:43:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:10 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:43:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:10 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:43:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:10 smithi038 ceph-mon[92422]: osdmap e1297: 12 total, 12 up, 12 in 2023-03-09T23:43:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31501.432044 secs 2023-03-09T23:43:11.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:11 smithi122 ceph-mon[93869]: pgmap v17312: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 284 KiB/s wr, 28 op/s 2023-03-09T23:43:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:11 smithi181 ceph-mon[94189]: pgmap v17312: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 284 KiB/s wr, 28 op/s 2023-03-09T23:43:11.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:11 smithi038 ceph-mon[92422]: pgmap v17312: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 284 KiB/s wr, 28 op/s 2023-03-09T23:43:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31965.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:43:13.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31503.846065 secs 2023-03-09T23:43:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:13 smithi122 ceph-mon[93869]: pgmap v17313: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 466 KiB/s rd, 507 KiB/s wr, 29 op/s 2023-03-09T23:43:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31503.846065 secs 2023-03-09T23:43:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:13 smithi181 ceph-mon[94189]: pgmap v17313: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 466 KiB/s rd, 507 KiB/s wr, 29 op/s 2023-03-09T23:43:13.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31503.846065 secs 2023-03-09T23:43:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:13 smithi038 ceph-mon[92422]: pgmap v17313: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 466 KiB/s rd, 507 KiB/s wr, 29 op/s 2023-03-09T23:43:15.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:15 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:43:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:15 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:43:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:15 smithi122 ceph-mon[93869]: pgmap v17314: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 523 KiB/s rd, 565 KiB/s wr, 27 op/s 2023-03-09T23:43:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31506.432103 secs 2023-03-09T23:43:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:15 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:43:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:15 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:43:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:15 smithi181 ceph-mon[94189]: pgmap v17314: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 523 KiB/s rd, 565 KiB/s wr, 27 op/s 2023-03-09T23:43:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31506.432103 secs 2023-03-09T23:43:15.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:15 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:43:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:15 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:43:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:15 smithi038 ceph-mon[92422]: pgmap v17314: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 523 KiB/s rd, 565 KiB/s wr, 27 op/s 2023-03-09T23:43:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31506.432103 secs 2023-03-09T23:43:17.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:43:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:43:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:17 smithi122 ceph-mon[93869]: pgmap v17315: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 522 KiB/s rd, 538 KiB/s wr, 23 op/s 2023-03-09T23:43:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:17 smithi122 ceph-mon[93869]: osdmap e1298: 12 total, 12 up, 12 in 2023-03-09T23:43:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:43:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:43:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:17 smithi181 ceph-mon[94189]: pgmap v17315: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 522 KiB/s rd, 538 KiB/s wr, 23 op/s 2023-03-09T23:43:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:17 smithi181 ceph-mon[94189]: osdmap e1298: 12 total, 12 up, 12 in 2023-03-09T23:43:17.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:43:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:43:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:17 smithi038 ceph-mon[92422]: pgmap v17315: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 522 KiB/s rd, 538 KiB/s wr, 23 op/s 2023-03-09T23:43:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:17 smithi038 ceph-mon[92422]: osdmap e1298: 12 total, 12 up, 12 in 2023-03-09T23:43:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31970.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:43:18.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31508.846183 secs 2023-03-09T23:43:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31508.846183 secs 2023-03-09T23:43:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31508.846183 secs 2023-03-09T23:43:19.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:19 smithi122 ceph-mon[93869]: pgmap v17317: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 549 KiB/s wr, 10 op/s 2023-03-09T23:43:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:19 smithi181 ceph-mon[94189]: pgmap v17317: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 549 KiB/s wr, 10 op/s 2023-03-09T23:43:19.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:19 smithi038 ceph-mon[92422]: pgmap v17317: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 549 KiB/s wr, 10 op/s 2023-03-09T23:43:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31511.432163 secs 2023-03-09T23:43:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31511.432163 secs 2023-03-09T23:43:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31511.432163 secs 2023-03-09T23:43:21.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:21 smithi122 ceph-mon[93869]: pgmap v17318: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 499 KiB/s wr, 9 op/s 2023-03-09T23:43:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:21 smithi181 ceph-mon[94189]: pgmap v17318: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 499 KiB/s wr, 9 op/s 2023-03-09T23:43:21.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:21 smithi038 ceph-mon[92422]: pgmap v17318: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 499 KiB/s wr, 9 op/s 2023-03-09T23:43:23.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31975.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:43:23.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31513.846328 secs 2023-03-09T23:43:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:23 smithi122 ceph-mon[93869]: pgmap v17319: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 231 KiB/s wr, 8 op/s 2023-03-09T23:43:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31513.846328 secs 2023-03-09T23:43:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:23 smithi181 ceph-mon[94189]: pgmap v17319: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 231 KiB/s wr, 8 op/s 2023-03-09T23:43:23.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31513.846328 secs 2023-03-09T23:43:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:23 smithi038 ceph-mon[92422]: pgmap v17319: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 231 KiB/s wr, 8 op/s 2023-03-09T23:43:26.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:25 smithi122 ceph-mon[93869]: pgmap v17320: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 115 KiB/s wr, 7 op/s 2023-03-09T23:43:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31516.432222 secs 2023-03-09T23:43:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:25 smithi181 ceph-mon[94189]: pgmap v17320: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 115 KiB/s wr, 7 op/s 2023-03-09T23:43:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31516.432222 secs 2023-03-09T23:43:26.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:25 smithi038 ceph-mon[92422]: pgmap v17320: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 115 KiB/s wr, 7 op/s 2023-03-09T23:43:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31516.432222 secs 2023-03-09T23:43:28.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:27 smithi122 ceph-mon[93869]: pgmap v17321: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 115 KiB/s wr, 7 op/s 2023-03-09T23:43:28.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:27 smithi181 ceph-mon[94189]: pgmap v17321: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 115 KiB/s wr, 7 op/s 2023-03-09T23:43:28.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:27 smithi038 ceph-mon[92422]: pgmap v17321: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 115 KiB/s wr, 7 op/s 2023-03-09T23:43:28.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31980.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:43:29.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31518.846448 secs 2023-03-09T23:43:29.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31518.846448 secs 2023-03-09T23:43:29.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31518.846448 secs 2023-03-09T23:43:30.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:29 smithi122 ceph-mon[93869]: pgmap v17322: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 104 KiB/s wr, 7 op/s 2023-03-09T23:43:30.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:29 smithi181 ceph-mon[94189]: pgmap v17322: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 104 KiB/s wr, 7 op/s 2023-03-09T23:43:30.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:29 smithi038 ceph-mon[92422]: pgmap v17322: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 104 KiB/s wr, 7 op/s 2023-03-09T23:43:31.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31521.432283 secs 2023-03-09T23:43:31.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31521.432283 secs 2023-03-09T23:43:31.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31521.432283 secs 2023-03-09T23:43:32.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:31 smithi122 ceph-mon[93869]: pgmap v17323: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T23:43:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:31 smithi181 ceph-mon[94189]: pgmap v17323: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T23:43:32.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:31 smithi038 ceph-mon[92422]: pgmap v17323: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T23:43:33.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31985.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:43:33.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31523.846577 secs 2023-03-09T23:43:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:33 smithi122 ceph-mon[93869]: pgmap v17324: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T23:43:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31523.846577 secs 2023-03-09T23:43:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:33 smithi181 ceph-mon[94189]: pgmap v17324: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T23:43:33.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31523.846577 secs 2023-03-09T23:43:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:33 smithi038 ceph-mon[92422]: pgmap v17324: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T23:43:36.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:35 smithi122 ceph-mon[93869]: pgmap v17325: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 6 op/s 2023-03-09T23:43:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31526.432369 secs 2023-03-09T23:43:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:35 smithi181 ceph-mon[94189]: pgmap v17325: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 6 op/s 2023-03-09T23:43:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31526.432369 secs 2023-03-09T23:43:36.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:35 smithi038 ceph-mon[92422]: pgmap v17325: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 6 op/s 2023-03-09T23:43:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31526.432369 secs 2023-03-09T23:43:38.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:37 smithi122 ceph-mon[93869]: pgmap v17326: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 341 B/s wr, 7 op/s 2023-03-09T23:43:38.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:37 smithi181 ceph-mon[94189]: pgmap v17326: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 341 B/s wr, 7 op/s 2023-03-09T23:43:38.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:37 smithi038 ceph-mon[92422]: pgmap v17326: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 341 B/s wr, 7 op/s 2023-03-09T23:43:38.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31990.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:43:39.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31528.846651 secs 2023-03-09T23:43:39.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31528.846651 secs 2023-03-09T23:43:39.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31528.846651 secs 2023-03-09T23:43:40.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:39 smithi122 ceph-mon[93869]: pgmap v17327: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 140 KiB/s wr, 7 op/s 2023-03-09T23:43:40.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:39 smithi181 ceph-mon[94189]: pgmap v17327: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 140 KiB/s wr, 7 op/s 2023-03-09T23:43:40.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:39 smithi038 ceph-mon[92422]: pgmap v17327: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 140 KiB/s wr, 7 op/s 2023-03-09T23:43:41.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31531.432428 secs 2023-03-09T23:43:41.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31531.432428 secs 2023-03-09T23:43:41.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31531.432428 secs 2023-03-09T23:43:42.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:41 smithi122 ceph-mon[93869]: pgmap v17328: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 140 KiB/s wr, 8 op/s 2023-03-09T23:43:42.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:41 smithi181 ceph-mon[94189]: pgmap v17328: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 140 KiB/s wr, 8 op/s 2023-03-09T23:43:42.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:41 smithi038 ceph-mon[92422]: pgmap v17328: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 140 KiB/s wr, 8 op/s 2023-03-09T23:43:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 31995.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:43:43.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:43 smithi122 ceph-mon[93869]: pgmap v17329: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 140 KiB/s wr, 8 op/s 2023-03-09T23:43:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:43 smithi181 ceph-mon[94189]: pgmap v17329: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 140 KiB/s wr, 8 op/s 2023-03-09T23:43:43.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:43 smithi038 ceph-mon[92422]: pgmap v17329: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 140 KiB/s wr, 8 op/s 2023-03-09T23:43:46.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:45 smithi122 ceph-mon[93869]: pgmap v17330: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 151 KiB/s wr, 8 op/s 2023-03-09T23:43:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31536.432490 secs 2023-03-09T23:43:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:45 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:43:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:45 smithi181 ceph-mon[94189]: pgmap v17330: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 151 KiB/s wr, 8 op/s 2023-03-09T23:43:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31536.432490 secs 2023-03-09T23:43:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:45 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:43:46.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:45 smithi038 ceph-mon[92422]: pgmap v17330: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 151 KiB/s wr, 8 op/s 2023-03-09T23:43:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31536.432490 secs 2023-03-09T23:43:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:45 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:43:47.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:46 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:43:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:46 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:43:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:46 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:43:47.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:46 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:43:47.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:46 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:43:47.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:46 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:43:47.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:46 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:43:47.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:46 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:43:47.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:46 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:43:48.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:47 smithi122 ceph-mon[93869]: pgmap v17331: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 150 KiB/s wr, 8 op/s 2023-03-09T23:43:48.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:47 smithi181 ceph-mon[94189]: pgmap v17331: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 150 KiB/s wr, 8 op/s 2023-03-09T23:43:48.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:47 smithi038 ceph-mon[92422]: pgmap v17331: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 150 KiB/s wr, 8 op/s 2023-03-09T23:43:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32000.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:43:49.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:43:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:43:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:49 smithi122 ceph-mon[93869]: pgmap v17332: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 151 KiB/s wr, 8 op/s 2023-03-09T23:43:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:43:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:43:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:49 smithi181 ceph-mon[94189]: pgmap v17332: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 151 KiB/s wr, 8 op/s 2023-03-09T23:43:49.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:43:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:43:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:49 smithi038 ceph-mon[92422]: pgmap v17332: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 151 KiB/s wr, 8 op/s 2023-03-09T23:43:50.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31541.432548 secs 2023-03-09T23:43:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31541.432548 secs 2023-03-09T23:43:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31541.432548 secs 2023-03-09T23:43:51.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:43:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:43:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:51 smithi122 ceph-mon[93869]: pgmap v17333: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 646 KiB/s rd, 352 KiB/s wr, 8 op/s 2023-03-09T23:43:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:43:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:43:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:51 smithi181 ceph-mon[94189]: pgmap v17333: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 646 KiB/s rd, 352 KiB/s wr, 8 op/s 2023-03-09T23:43:51.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:43:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:43:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:51 smithi038 ceph-mon[92422]: pgmap v17333: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 646 KiB/s rd, 352 KiB/s wr, 8 op/s 2023-03-09T23:43:53.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32005.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:43:53.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:53 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 31543.847002 secs 2023-03-09T23:43:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:53 smithi122 ceph-mon[93869]: pgmap v17334: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 646 KiB/s rd, 352 KiB/s wr, 8 op/s 2023-03-09T23:43:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:53 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 31543.847002 secs 2023-03-09T23:43:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:53 smithi181 ceph-mon[94189]: pgmap v17334: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 646 KiB/s rd, 352 KiB/s wr, 8 op/s 2023-03-09T23:43:53.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:53 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 31543.847002 secs 2023-03-09T23:43:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:53 smithi038 ceph-mon[92422]: pgmap v17334: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 646 KiB/s rd, 352 KiB/s wr, 8 op/s 2023-03-09T23:43:54.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:54 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:43:54.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:54 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:43:54.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:54 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:43:54.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:54 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:43:54.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:54 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:43:54.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:54 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:43:54.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:54 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:43:54.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:54 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:43:54.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:54 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:43:55.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:43:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:43:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:55 smithi122 ceph-mon[93869]: pgmap v17335: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 942 KiB/s rd, 629 KiB/s wr, 9 op/s 2023-03-09T23:43:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31546.432645 secs 2023-03-09T23:43:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:43:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:43:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:55 smithi181 ceph-mon[94189]: pgmap v17335: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 942 KiB/s rd, 629 KiB/s wr, 9 op/s 2023-03-09T23:43:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31546.432645 secs 2023-03-09T23:43:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:43:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:43:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:55 smithi038 ceph-mon[92422]: pgmap v17335: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 942 KiB/s rd, 629 KiB/s wr, 9 op/s 2023-03-09T23:43:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31546.432645 secs 2023-03-09T23:43:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:43:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:43:57.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:57 smithi122 ceph-mon[93869]: pgmap v17336: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 910 KiB/s rd, 744 KiB/s wr, 11 op/s 2023-03-09T23:43:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:43:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:43:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:57 smithi181 ceph-mon[94189]: pgmap v17336: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 910 KiB/s rd, 744 KiB/s wr, 11 op/s 2023-03-09T23:43:57.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:43:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:43:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:57 smithi038 ceph-mon[92422]: pgmap v17336: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 910 KiB/s rd, 744 KiB/s wr, 11 op/s 2023-03-09T23:43:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32010.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:43:58.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31548.847116 secs 2023-03-09T23:43:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31548.847116 secs 2023-03-09T23:43:58.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31548.847116 secs 2023-03-09T23:43:59.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:43:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:43:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:43:59 smithi122 ceph-mon[93869]: pgmap v17337: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 909 KiB/s rd, 745 KiB/s wr, 10 op/s 2023-03-09T23:43:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:43:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:43:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:43:59 smithi181 ceph-mon[94189]: pgmap v17337: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 909 KiB/s rd, 745 KiB/s wr, 10 op/s 2023-03-09T23:43:59.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:43:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:43:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:43:59 smithi038 ceph-mon[92422]: pgmap v17337: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 909 KiB/s rd, 745 KiB/s wr, 10 op/s 2023-03-09T23:44:00.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31551.432723 secs 2023-03-09T23:44:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31551.432723 secs 2023-03-09T23:44:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31551.432723 secs 2023-03-09T23:44:01.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:44:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:44:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:01 smithi122 ceph-mon[93869]: pgmap v17338: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.2 MiB/s wr, 14 op/s 2023-03-09T23:44:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:01 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:44:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:01 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:44:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:01 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:44:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:01 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:44:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:01 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:44:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:44:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:44:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:01 smithi181 ceph-mon[94189]: pgmap v17338: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.2 MiB/s wr, 14 op/s 2023-03-09T23:44:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:01 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:44:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:01 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:44:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:01 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:44:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:01 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:44:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:01 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:44:01.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:44:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:44:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:01 smithi038 ceph-mon[92422]: pgmap v17338: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.2 MiB/s wr, 14 op/s 2023-03-09T23:44:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:01 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:44:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:01 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:44:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:01 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:44:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:01 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:44:02.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:01 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:44:02.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:02 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:44:02.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:02 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:44:02.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:44:02.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:44:02.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:44:02.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:02 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:44:02.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:02 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:44:02.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:44:02.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:44:02.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:44:02.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:02 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:44:02.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:02 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:44:02.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:44:02.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:44:02.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:44:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32015.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:44:03.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31553.847205 secs 2023-03-09T23:44:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:03 smithi038 ceph-mon[92422]: pgmap v17339: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 924 KiB/s wr, 13 op/s 2023-03-09T23:44:04.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31553.847205 secs 2023-03-09T23:44:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:03 smithi122 ceph-mon[93869]: pgmap v17339: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 924 KiB/s wr, 13 op/s 2023-03-09T23:44:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31553.847205 secs 2023-03-09T23:44:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:03 smithi181 ceph-mon[94189]: pgmap v17339: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 924 KiB/s wr, 13 op/s 2023-03-09T23:44:06.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:05 smithi122 ceph-mon[93869]: pgmap v17340: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 927 KiB/s wr, 15 op/s 2023-03-09T23:44:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31556.432801 secs 2023-03-09T23:44:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:05 smithi181 ceph-mon[94189]: pgmap v17340: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 927 KiB/s wr, 15 op/s 2023-03-09T23:44:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31556.432801 secs 2023-03-09T23:44:06.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:05 smithi038 ceph-mon[92422]: pgmap v17340: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 927 KiB/s wr, 15 op/s 2023-03-09T23:44:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31556.432801 secs 2023-03-09T23:44:08.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:07 smithi122 ceph-mon[93869]: pgmap v17341: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 919 KiB/s rd, 802 KiB/s wr, 19 op/s 2023-03-09T23:44:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:44:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:44:08.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:07 smithi181 ceph-mon[94189]: pgmap v17341: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 919 KiB/s rd, 802 KiB/s wr, 19 op/s 2023-03-09T23:44:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:44:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:44:08.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:07 smithi038 ceph-mon[92422]: pgmap v17341: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 919 KiB/s rd, 802 KiB/s wr, 19 op/s 2023-03-09T23:44:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:44:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:44:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32020.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:44:09.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31558.847342 secs 2023-03-09T23:44:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:44:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:44:09.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31558.847342 secs 2023-03-09T23:44:09.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:44:09.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:44:09.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31558.847342 secs 2023-03-09T23:44:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:44:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:44:10.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:09 smithi122 ceph-mon[93869]: pgmap v17342: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 698 KiB/s wr, 27 op/s 2023-03-09T23:44:10.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:09 smithi181 ceph-mon[94189]: pgmap v17342: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 698 KiB/s wr, 27 op/s 2023-03-09T23:44:10.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:09 smithi038 ceph-mon[92422]: pgmap v17342: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 698 KiB/s wr, 27 op/s 2023-03-09T23:44:11.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:10 smithi122 ceph-mon[93869]: osdmap e1299: 12 total, 12 up, 12 in 2023-03-09T23:44:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31561.432885 secs 2023-03-09T23:44:11.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:10 smithi181 ceph-mon[94189]: osdmap e1299: 12 total, 12 up, 12 in 2023-03-09T23:44:11.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31561.432885 secs 2023-03-09T23:44:11.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:10 smithi038 ceph-mon[92422]: osdmap e1299: 12 total, 12 up, 12 in 2023-03-09T23:44:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31561.432885 secs 2023-03-09T23:44:12.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:11 smithi122 ceph-mon[93869]: pgmap v17344: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 432 KiB/s wr, 30 op/s 2023-03-09T23:44:12.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:11 smithi181 ceph-mon[94189]: pgmap v17344: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 432 KiB/s wr, 30 op/s 2023-03-09T23:44:12.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:11 smithi038 ceph-mon[92422]: pgmap v17344: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 432 KiB/s wr, 30 op/s 2023-03-09T23:44:13.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:44:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:44:13.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:44:13.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:44:13.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:44:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:44:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32025.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:44:14.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31563.847494 secs 2023-03-09T23:44:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:13 smithi122 ceph-mon[93869]: pgmap v17345: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 432 KiB/s wr, 30 op/s 2023-03-09T23:44:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31563.847494 secs 2023-03-09T23:44:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:13 smithi181 ceph-mon[94189]: pgmap v17345: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 432 KiB/s wr, 30 op/s 2023-03-09T23:44:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31563.847494 secs 2023-03-09T23:44:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:13 smithi038 ceph-mon[92422]: pgmap v17345: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 432 KiB/s wr, 30 op/s 2023-03-09T23:44:16.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:15 smithi122 ceph-mon[93869]: pgmap v17346: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 592 KiB/s wr, 28 op/s 2023-03-09T23:44:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:15 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:44:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:15 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:44:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31566.432944 secs 2023-03-09T23:44:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:15 smithi181 ceph-mon[94189]: pgmap v17346: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 592 KiB/s wr, 28 op/s 2023-03-09T23:44:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:15 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:44:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:15 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:44:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31566.432944 secs 2023-03-09T23:44:16.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:15 smithi038 ceph-mon[92422]: pgmap v17346: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 592 KiB/s wr, 28 op/s 2023-03-09T23:44:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:15 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:44:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:15 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:44:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31566.432944 secs 2023-03-09T23:44:18.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:17 smithi122 ceph-mon[93869]: pgmap v17347: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 411 KiB/s wr, 23 op/s 2023-03-09T23:44:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:17 smithi122 ceph-mon[93869]: osdmap e1300: 12 total, 12 up, 12 in 2023-03-09T23:44:18.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:17 smithi181 ceph-mon[94189]: pgmap v17347: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 411 KiB/s wr, 23 op/s 2023-03-09T23:44:18.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:17 smithi181 ceph-mon[94189]: osdmap e1300: 12 total, 12 up, 12 in 2023-03-09T23:44:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:17 smithi038 ceph-mon[92422]: pgmap v17347: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 411 KiB/s wr, 23 op/s 2023-03-09T23:44:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:17 smithi038 ceph-mon[92422]: osdmap e1300: 12 total, 12 up, 12 in 2023-03-09T23:44:18.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32030.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:44:19.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31568.847632 secs 2023-03-09T23:44:19.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31568.847632 secs 2023-03-09T23:44:19.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31568.847632 secs 2023-03-09T23:44:20.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:19 smithi122 ceph-mon[93869]: pgmap v17349: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 317 KiB/s wr, 9 op/s 2023-03-09T23:44:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:19 smithi181 ceph-mon[94189]: pgmap v17349: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 317 KiB/s wr, 9 op/s 2023-03-09T23:44:20.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:19 smithi038 ceph-mon[92422]: pgmap v17349: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 317 KiB/s wr, 9 op/s 2023-03-09T23:44:21.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31571.433000 secs 2023-03-09T23:44:21.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31571.433000 secs 2023-03-09T23:44:21.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31571.433000 secs 2023-03-09T23:44:22.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:21 smithi122 ceph-mon[93869]: pgmap v17350: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 280 KiB/s wr, 7 op/s 2023-03-09T23:44:22.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:21 smithi181 ceph-mon[94189]: pgmap v17350: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 280 KiB/s wr, 7 op/s 2023-03-09T23:44:22.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:21 smithi038 ceph-mon[92422]: pgmap v17350: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 280 KiB/s wr, 7 op/s 2023-03-09T23:44:23.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32035.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:44:23.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31573.847711 secs 2023-03-09T23:44:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:23 smithi122 ceph-mon[93869]: pgmap v17351: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 279 KiB/s wr, 7 op/s 2023-03-09T23:44:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31573.847711 secs 2023-03-09T23:44:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:23 smithi181 ceph-mon[94189]: pgmap v17351: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 279 KiB/s wr, 7 op/s 2023-03-09T23:44:23.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31573.847711 secs 2023-03-09T23:44:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:23 smithi038 ceph-mon[92422]: pgmap v17351: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 279 KiB/s wr, 7 op/s 2023-03-09T23:44:26.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:25 smithi122 ceph-mon[93869]: pgmap v17352: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 116 KiB/s wr, 6 op/s 2023-03-09T23:44:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31576.433059 secs 2023-03-09T23:44:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:25 smithi181 ceph-mon[94189]: pgmap v17352: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 116 KiB/s wr, 6 op/s 2023-03-09T23:44:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31576.433059 secs 2023-03-09T23:44:26.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:25 smithi038 ceph-mon[92422]: pgmap v17352: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 116 KiB/s wr, 6 op/s 2023-03-09T23:44:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31576.433059 secs 2023-03-09T23:44:28.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:27 smithi122 ceph-mon[93869]: pgmap v17353: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 115 KiB/s wr, 6 op/s 2023-03-09T23:44:28.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:27 smithi181 ceph-mon[94189]: pgmap v17353: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 115 KiB/s wr, 6 op/s 2023-03-09T23:44:28.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:27 smithi038 ceph-mon[92422]: pgmap v17353: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 115 KiB/s wr, 6 op/s 2023-03-09T23:44:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32040.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:44:29.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31578.847801 secs 2023-03-09T23:44:29.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31578.847801 secs 2023-03-09T23:44:29.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31578.847801 secs 2023-03-09T23:44:30.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:29 smithi122 ceph-mon[93869]: pgmap v17354: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 104 KiB/s wr, 5 op/s 2023-03-09T23:44:30.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:29 smithi181 ceph-mon[94189]: pgmap v17354: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 104 KiB/s wr, 5 op/s 2023-03-09T23:44:30.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:29 smithi038 ceph-mon[92422]: pgmap v17354: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 104 KiB/s wr, 5 op/s 2023-03-09T23:44:31.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31581.433121 secs 2023-03-09T23:44:31.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31581.433121 secs 2023-03-09T23:44:31.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31581.433121 secs 2023-03-09T23:44:32.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:31 smithi122 ceph-mon[93869]: pgmap v17355: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 96 KiB/s wr, 6 op/s 2023-03-09T23:44:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:31 smithi181 ceph-mon[94189]: pgmap v17355: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 96 KiB/s wr, 6 op/s 2023-03-09T23:44:32.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:31 smithi038 ceph-mon[92422]: pgmap v17355: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 96 KiB/s wr, 6 op/s 2023-03-09T23:44:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32045.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:44:33.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31583.847938 secs 2023-03-09T23:44:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:33 smithi122 ceph-mon[93869]: pgmap v17356: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2023-03-09T23:44:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31583.847938 secs 2023-03-09T23:44:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:33 smithi181 ceph-mon[94189]: pgmap v17356: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2023-03-09T23:44:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31583.847938 secs 2023-03-09T23:44:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:33 smithi038 ceph-mon[92422]: pgmap v17356: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2023-03-09T23:44:36.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:35 smithi122 ceph-mon[93869]: pgmap v17357: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 24 KiB/s wr, 5 op/s 2023-03-09T23:44:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31586.433184 secs 2023-03-09T23:44:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:35 smithi181 ceph-mon[94189]: pgmap v17357: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 24 KiB/s wr, 5 op/s 2023-03-09T23:44:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31586.433184 secs 2023-03-09T23:44:36.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:35 smithi038 ceph-mon[92422]: pgmap v17357: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 24 KiB/s wr, 5 op/s 2023-03-09T23:44:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31586.433184 secs 2023-03-09T23:44:38.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:37 smithi122 ceph-mon[93869]: pgmap v17358: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T23:44:38.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:37 smithi181 ceph-mon[94189]: pgmap v17358: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T23:44:38.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:37 smithi038 ceph-mon[92422]: pgmap v17358: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 24 KiB/s wr, 7 op/s 2023-03-09T23:44:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32050.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:44:39.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31588.848071 secs 2023-03-09T23:44:39.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31588.848071 secs 2023-03-09T23:44:39.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31588.848071 secs 2023-03-09T23:44:40.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:39 smithi122 ceph-mon[93869]: pgmap v17359: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 117 KiB/s wr, 6 op/s 2023-03-09T23:44:40.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:39 smithi181 ceph-mon[94189]: pgmap v17359: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 117 KiB/s wr, 6 op/s 2023-03-09T23:44:40.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:39 smithi038 ceph-mon[92422]: pgmap v17359: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 117 KiB/s wr, 6 op/s 2023-03-09T23:44:41.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:44:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:44:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31591.433244 secs 2023-03-09T23:44:41.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:44:41.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:44:41.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31591.433244 secs 2023-03-09T23:44:41.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:44:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:44:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31591.433244 secs 2023-03-09T23:44:42.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:41 smithi122 ceph-mon[93869]: pgmap v17360: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 117 KiB/s wr, 8 op/s 2023-03-09T23:44:42.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:41 smithi181 ceph-mon[94189]: pgmap v17360: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 117 KiB/s wr, 8 op/s 2023-03-09T23:44:42.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:41 smithi038 ceph-mon[92422]: pgmap v17360: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 117 KiB/s wr, 8 op/s 2023-03-09T23:44:43.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32055.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:44:43.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31593.848195 secs 2023-03-09T23:44:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:43 smithi122 ceph-mon[93869]: pgmap v17361: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 416 KiB/s rd, 366 KiB/s wr, 8 op/s 2023-03-09T23:44:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31593.848195 secs 2023-03-09T23:44:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:43 smithi181 ceph-mon[94189]: pgmap v17361: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 416 KiB/s rd, 366 KiB/s wr, 8 op/s 2023-03-09T23:44:43.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31593.848195 secs 2023-03-09T23:44:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:43 smithi038 ceph-mon[92422]: pgmap v17361: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 416 KiB/s rd, 366 KiB/s wr, 8 op/s 2023-03-09T23:44:46.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:45 smithi122 ceph-mon[93869]: pgmap v17362: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 416 KiB/s rd, 432 KiB/s wr, 8 op/s 2023-03-09T23:44:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31596.433303 secs 2023-03-09T23:44:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:45 smithi181 ceph-mon[94189]: pgmap v17362: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 416 KiB/s rd, 432 KiB/s wr, 8 op/s 2023-03-09T23:44:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31596.433303 secs 2023-03-09T23:44:46.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:45 smithi038 ceph-mon[92422]: pgmap v17362: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 416 KiB/s rd, 432 KiB/s wr, 8 op/s 2023-03-09T23:44:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31596.433303 secs 2023-03-09T23:44:48.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:47 smithi122 ceph-mon[93869]: pgmap v17363: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 409 KiB/s wr, 10 op/s 2023-03-09T23:44:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:44:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:44:48.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:47 smithi181 ceph-mon[94189]: pgmap v17363: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 409 KiB/s wr, 10 op/s 2023-03-09T23:44:48.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:44:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:44:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:47 smithi038 ceph-mon[92422]: pgmap v17363: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 409 KiB/s wr, 10 op/s 2023-03-09T23:44:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:44:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:44:48.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32060.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:44:49.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31598.848273 secs 2023-03-09T23:44:49.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31598.848273 secs 2023-03-09T23:44:49.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31598.848273 secs 2023-03-09T23:44:50.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:49 smithi122 ceph-mon[93869]: pgmap v17364: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 636 KiB/s rd, 408 KiB/s wr, 8 op/s 2023-03-09T23:44:50.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:49 smithi181 ceph-mon[94189]: pgmap v17364: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 636 KiB/s rd, 408 KiB/s wr, 8 op/s 2023-03-09T23:44:50.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:49 smithi038 ceph-mon[92422]: pgmap v17364: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 636 KiB/s rd, 408 KiB/s wr, 8 op/s 2023-03-09T23:44:51.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31601.433364 secs 2023-03-09T23:44:51.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31601.433364 secs 2023-03-09T23:44:51.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31601.433364 secs 2023-03-09T23:44:52.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:51 smithi122 ceph-mon[93869]: pgmap v17365: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 640 KiB/s rd, 345 KiB/s wr, 10 op/s 2023-03-09T23:44:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:51 smithi181 ceph-mon[94189]: pgmap v17365: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 640 KiB/s rd, 345 KiB/s wr, 10 op/s 2023-03-09T23:44:52.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:51 smithi038 ceph-mon[92422]: pgmap v17365: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 640 KiB/s rd, 345 KiB/s wr, 10 op/s 2023-03-09T23:44:53.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:44:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:44:53.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:44:53.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:44:53.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:44:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:44:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32065.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:44:54.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31603.848385 secs 2023-03-09T23:44:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:53 smithi122 ceph-mon[93869]: pgmap v17366: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 615 KiB/s rd, 345 KiB/s wr, 9 op/s 2023-03-09T23:44:54.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31603.848385 secs 2023-03-09T23:44:54.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:53 smithi181 ceph-mon[94189]: pgmap v17366: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 615 KiB/s rd, 345 KiB/s wr, 9 op/s 2023-03-09T23:44:54.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31603.848385 secs 2023-03-09T23:44:54.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:53 smithi038 ceph-mon[92422]: pgmap v17366: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 615 KiB/s rd, 345 KiB/s wr, 9 op/s 2023-03-09T23:44:56.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:55 smithi122 ceph-mon[93869]: pgmap v17367: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 774 KiB/s rd, 504 KiB/s wr, 9 op/s 2023-03-09T23:44:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:44:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:44:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31606.433426 secs 2023-03-09T23:44:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:55 smithi181 ceph-mon[94189]: pgmap v17367: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 774 KiB/s rd, 504 KiB/s wr, 9 op/s 2023-03-09T23:44:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:44:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:44:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31606.433426 secs 2023-03-09T23:44:56.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:55 smithi038 ceph-mon[92422]: pgmap v17367: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 774 KiB/s rd, 504 KiB/s wr, 9 op/s 2023-03-09T23:44:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:44:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:44:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31606.433426 secs 2023-03-09T23:44:57.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:44:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:44:57.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:44:57.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:44:57.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:44:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:44:58.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:57 smithi122 ceph-mon[93869]: pgmap v17368: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 855 KiB/s rd, 516 KiB/s wr, 12 op/s 2023-03-09T23:44:58.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:57 smithi181 ceph-mon[94189]: pgmap v17368: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 855 KiB/s rd, 516 KiB/s wr, 12 op/s 2023-03-09T23:44:58.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:57 smithi038 ceph-mon[92422]: pgmap v17368: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 855 KiB/s rd, 516 KiB/s wr, 12 op/s 2023-03-09T23:44:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32070.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:45:00.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:59 smithi122 ceph-mon[93869]: pgmap v17369: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 725 KiB/s wr, 11 op/s 2023-03-09T23:45:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:45:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:44:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:45:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:59 smithi181 ceph-mon[94189]: pgmap v17369: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 725 KiB/s wr, 11 op/s 2023-03-09T23:45:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:45:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:44:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:45:00.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:59 smithi038 ceph-mon[92422]: pgmap v17369: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 725 KiB/s wr, 11 op/s 2023-03-09T23:45:00.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:45:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:44:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:45:01.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:45:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:45:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31611.433486 secs 2023-03-09T23:45:01.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:45:01.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:45:01.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31611.433486 secs 2023-03-09T23:45:01.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:45:01.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:45:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31611.433486 secs 2023-03-09T23:45:02.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:01 smithi122 ceph-mon[93869]: pgmap v17370: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 14 op/s 2023-03-09T23:45:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:01 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:45:02.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:01 smithi181 ceph-mon[94189]: pgmap v17370: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 14 op/s 2023-03-09T23:45:02.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:01 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:45:02.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:01 smithi038 ceph-mon[92422]: pgmap v17370: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 14 op/s 2023-03-09T23:45:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:01 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:45:03.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:02 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:45:03.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:02 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:45:03.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:02 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:45:03.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:45:03.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:02 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:45:03.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:02 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:45:03.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:02 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:45:03.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:45:03.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:02 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:45:03.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:02 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:45:03.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:02 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:45:03.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:45:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32075.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:45:04.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:03 smithi122 ceph-mon[93869]: pgmap v17371: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 13 op/s 2023-03-09T23:45:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:03 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:45:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:03 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:45:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:03 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:45:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:03 smithi181 ceph-mon[94189]: pgmap v17371: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 13 op/s 2023-03-09T23:45:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:03 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:45:04.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:03 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:45:04.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:03 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:45:04.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:03 smithi038 ceph-mon[92422]: pgmap v17371: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 13 op/s 2023-03-09T23:45:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:03 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:45:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:03 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:45:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:03 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:45:06.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:05 smithi122 ceph-mon[93869]: pgmap v17372: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 15 op/s 2023-03-09T23:45:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31616.433548 secs 2023-03-09T23:45:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:05 smithi181 ceph-mon[94189]: pgmap v17372: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 15 op/s 2023-03-09T23:45:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31616.433548 secs 2023-03-09T23:45:06.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:05 smithi038 ceph-mon[92422]: pgmap v17372: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 15 op/s 2023-03-09T23:45:06.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31616.433548 secs 2023-03-09T23:45:08.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:07 smithi122 ceph-mon[93869]: pgmap v17373: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 758 KiB/s rd, 764 KiB/s wr, 18 op/s 2023-03-09T23:45:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:45:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:45:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31618.848714 secs 2023-03-09T23:45:08.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:07 smithi181 ceph-mon[94189]: pgmap v17373: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 758 KiB/s rd, 764 KiB/s wr, 18 op/s 2023-03-09T23:45:08.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:45:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:45:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31618.848714 secs 2023-03-09T23:45:08.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:07 smithi038 ceph-mon[92422]: pgmap v17373: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 758 KiB/s rd, 764 KiB/s wr, 18 op/s 2023-03-09T23:45:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:45:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:45:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31618.848714 secs 2023-03-09T23:45:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32080.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:45:09.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:45:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:45:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:09 smithi122 ceph-mon[93869]: pgmap v17374: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 746 KiB/s rd, 771 KiB/s wr, 28 op/s 2023-03-09T23:45:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:45:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:45:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:09 smithi181 ceph-mon[94189]: pgmap v17374: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 746 KiB/s rd, 771 KiB/s wr, 28 op/s 2023-03-09T23:45:09.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:45:09.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:45:09.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:09 smithi038 ceph-mon[92422]: pgmap v17374: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 746 KiB/s rd, 771 KiB/s wr, 28 op/s 2023-03-09T23:45:10.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:10 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:45:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:10 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:45:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:10 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:45:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:10 smithi122 ceph-mon[93869]: osdmap e1301: 12 total, 12 up, 12 in 2023-03-09T23:45:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31621.433613 secs 2023-03-09T23:45:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:10 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:45:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:10 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:45:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:10 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:45:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:10 smithi181 ceph-mon[94189]: osdmap e1301: 12 total, 12 up, 12 in 2023-03-09T23:45:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31621.433613 secs 2023-03-09T23:45:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:10 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:45:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:10 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:45:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:10 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:45:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:10 smithi038 ceph-mon[92422]: osdmap e1301: 12 total, 12 up, 12 in 2023-03-09T23:45:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31621.433613 secs 2023-03-09T23:45:11.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:11 smithi122 ceph-mon[93869]: pgmap v17376: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 290 KiB/s wr, 28 op/s 2023-03-09T23:45:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:11 smithi181 ceph-mon[94189]: pgmap v17376: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 290 KiB/s wr, 28 op/s 2023-03-09T23:45:11.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:11 smithi038 ceph-mon[92422]: pgmap v17376: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 290 KiB/s wr, 28 op/s 2023-03-09T23:45:13.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32085.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:45:13.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31623.848803 secs 2023-03-09T23:45:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:13 smithi122 ceph-mon[93869]: pgmap v17377: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 484 KiB/s rd, 526 KiB/s wr, 29 op/s 2023-03-09T23:45:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31623.848803 secs 2023-03-09T23:45:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:13 smithi181 ceph-mon[94189]: pgmap v17377: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 484 KiB/s rd, 526 KiB/s wr, 29 op/s 2023-03-09T23:45:13.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31623.848803 secs 2023-03-09T23:45:13.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:13 smithi038 ceph-mon[92422]: pgmap v17377: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 484 KiB/s rd, 526 KiB/s wr, 29 op/s 2023-03-09T23:45:16.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:15 smithi122 ceph-mon[93869]: pgmap v17378: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 483 KiB/s rd, 526 KiB/s wr, 27 op/s 2023-03-09T23:45:16.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31626.433671 secs 2023-03-09T23:45:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:15 smithi181 ceph-mon[94189]: pgmap v17378: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 483 KiB/s rd, 526 KiB/s wr, 27 op/s 2023-03-09T23:45:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31626.433671 secs 2023-03-09T23:45:16.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:15 smithi038 ceph-mon[92422]: pgmap v17378: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 483 KiB/s rd, 526 KiB/s wr, 27 op/s 2023-03-09T23:45:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31626.433671 secs 2023-03-09T23:45:17.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:45:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:45:17.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:45:17.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:45:17.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:45:17.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:45:18.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:17 smithi122 ceph-mon[93869]: pgmap v17379: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 442 KiB/s wr, 23 op/s 2023-03-09T23:45:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:17 smithi122 ceph-mon[93869]: osdmap e1302: 12 total, 12 up, 12 in 2023-03-09T23:45:18.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:17 smithi181 ceph-mon[94189]: pgmap v17379: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 442 KiB/s wr, 23 op/s 2023-03-09T23:45:18.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:17 smithi181 ceph-mon[94189]: osdmap e1302: 12 total, 12 up, 12 in 2023-03-09T23:45:18.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:17 smithi038 ceph-mon[92422]: pgmap v17379: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 442 KiB/s wr, 23 op/s 2023-03-09T23:45:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:17 smithi038 ceph-mon[92422]: osdmap e1302: 12 total, 12 up, 12 in 2023-03-09T23:45:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32090.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:45:19.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31628.848917 secs 2023-03-09T23:45:19.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31628.848917 secs 2023-03-09T23:45:19.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31628.848917 secs 2023-03-09T23:45:20.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:19 smithi122 ceph-mon[93869]: pgmap v17381: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 503 KiB/s rd, 501 KiB/s wr, 10 op/s 2023-03-09T23:45:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:19 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:45:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:19 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:45:20.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:19 smithi181 ceph-mon[94189]: pgmap v17381: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 503 KiB/s rd, 501 KiB/s wr, 10 op/s 2023-03-09T23:45:20.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:19 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:45:20.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:19 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:45:20.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:19 smithi038 ceph-mon[92422]: pgmap v17381: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 503 KiB/s rd, 501 KiB/s wr, 10 op/s 2023-03-09T23:45:20.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:19 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:45:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:19 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:45:21.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31631.433731 secs 2023-03-09T23:45:21.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31631.433731 secs 2023-03-09T23:45:21.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31631.433731 secs 2023-03-09T23:45:22.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:21 smithi122 ceph-mon[93869]: pgmap v17382: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 457 KiB/s rd, 455 KiB/s wr, 9 op/s 2023-03-09T23:45:22.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:21 smithi181 ceph-mon[94189]: pgmap v17382: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 457 KiB/s rd, 455 KiB/s wr, 9 op/s 2023-03-09T23:45:22.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:21 smithi038 ceph-mon[92422]: pgmap v17382: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 457 KiB/s rd, 455 KiB/s wr, 9 op/s 2023-03-09T23:45:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32095.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:45:23.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31633.849031 secs 2023-03-09T23:45:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:23 smithi122 ceph-mon[93869]: pgmap v17383: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 116 KiB/s wr, 7 op/s 2023-03-09T23:45:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31633.849031 secs 2023-03-09T23:45:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:23 smithi181 ceph-mon[94189]: pgmap v17383: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 116 KiB/s wr, 7 op/s 2023-03-09T23:45:23.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31633.849031 secs 2023-03-09T23:45:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:23 smithi038 ceph-mon[92422]: pgmap v17383: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 116 KiB/s wr, 7 op/s 2023-03-09T23:45:26.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:25 smithi122 ceph-mon[93869]: pgmap v17384: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 116 KiB/s wr, 7 op/s 2023-03-09T23:45:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31636.433805 secs 2023-03-09T23:45:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:25 smithi181 ceph-mon[94189]: pgmap v17384: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 116 KiB/s wr, 7 op/s 2023-03-09T23:45:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31636.433805 secs 2023-03-09T23:45:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:25 smithi038 ceph-mon[92422]: pgmap v17384: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 116 KiB/s wr, 7 op/s 2023-03-09T23:45:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31636.433805 secs 2023-03-09T23:45:28.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:27 smithi122 ceph-mon[93869]: pgmap v17385: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 115 KiB/s wr, 7 op/s 2023-03-09T23:45:28.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:27 smithi181 ceph-mon[94189]: pgmap v17385: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 115 KiB/s wr, 7 op/s 2023-03-09T23:45:28.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:27 smithi038 ceph-mon[92422]: pgmap v17385: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 115 KiB/s wr, 7 op/s 2023-03-09T23:45:28.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32100.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:45:29.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31638.849146 secs 2023-03-09T23:45:29.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31638.849146 secs 2023-03-09T23:45:29.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31638.849146 secs 2023-03-09T23:45:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:29 smithi122 ceph-mon[93869]: pgmap v17386: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T23:45:30.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:29 smithi181 ceph-mon[94189]: pgmap v17386: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T23:45:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:29 smithi038 ceph-mon[92422]: pgmap v17386: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T23:45:31.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31641.433869 secs 2023-03-09T23:45:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31641.433869 secs 2023-03-09T23:45:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31641.433869 secs 2023-03-09T23:45:32.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:31 smithi122 ceph-mon[93869]: pgmap v17387: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2023-03-09T23:45:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:31 smithi181 ceph-mon[94189]: pgmap v17387: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2023-03-09T23:45:32.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:31 smithi038 ceph-mon[92422]: pgmap v17387: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2023-03-09T23:45:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32105.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:45:33.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31643.849227 secs 2023-03-09T23:45:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:33 smithi122 ceph-mon[93869]: pgmap v17388: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2023-03-09T23:45:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31643.849227 secs 2023-03-09T23:45:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:33 smithi181 ceph-mon[94189]: pgmap v17388: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2023-03-09T23:45:33.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31643.849227 secs 2023-03-09T23:45:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:33 smithi038 ceph-mon[92422]: pgmap v17388: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 7 op/s 2023-03-09T23:45:36.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:35 smithi122 ceph-mon[93869]: pgmap v17389: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T23:45:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31646.433946 secs 2023-03-09T23:45:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:35 smithi181 ceph-mon[94189]: pgmap v17389: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T23:45:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31646.433946 secs 2023-03-09T23:45:36.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:35 smithi038 ceph-mon[92422]: pgmap v17389: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T23:45:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31646.433946 secs 2023-03-09T23:45:38.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:37 smithi122 ceph-mon[93869]: pgmap v17390: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T23:45:38.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:37 smithi181 ceph-mon[94189]: pgmap v17390: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T23:45:38.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:37 smithi038 ceph-mon[92422]: pgmap v17390: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T23:45:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32110.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:45:39.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31648.849286 secs 2023-03-09T23:45:39.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31648.849286 secs 2023-03-09T23:45:39.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31648.849286 secs 2023-03-09T23:45:40.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:39 smithi122 ceph-mon[93869]: pgmap v17391: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T23:45:40.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:39 smithi181 ceph-mon[94189]: pgmap v17391: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T23:45:40.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:39 smithi038 ceph-mon[92422]: pgmap v17391: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T23:45:41.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31651.433980 secs 2023-03-09T23:45:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31651.433980 secs 2023-03-09T23:45:41.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31651.433980 secs 2023-03-09T23:45:42.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:41 smithi122 ceph-mon[93869]: pgmap v17392: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 255 B/s wr, 7 op/s 2023-03-09T23:45:42.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:41 smithi181 ceph-mon[94189]: pgmap v17392: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 255 B/s wr, 7 op/s 2023-03-09T23:45:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:41 smithi038 ceph-mon[92422]: pgmap v17392: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 255 B/s wr, 7 op/s 2023-03-09T23:45:43.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32115.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:45:43.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:43 smithi122 ceph-mon[93869]: pgmap v17393: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 426 B/s wr, 8 op/s 2023-03-09T23:45:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:43 smithi181 ceph-mon[94189]: pgmap v17393: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 426 B/s wr, 8 op/s 2023-03-09T23:45:43.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:43 smithi038 ceph-mon[92422]: pgmap v17393: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 426 B/s wr, 8 op/s 2023-03-09T23:45:46.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:45 smithi122 ceph-mon[93869]: pgmap v17394: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T23:45:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31656.434049 secs 2023-03-09T23:45:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:45 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:45:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:45 smithi181 ceph-mon[94189]: pgmap v17394: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T23:45:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31656.434049 secs 2023-03-09T23:45:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:45 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:45:46.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:45 smithi038 ceph-mon[92422]: pgmap v17394: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T23:45:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31656.434049 secs 2023-03-09T23:45:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:45 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:45:47.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:46 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:45:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:46 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:45:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:46 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:45:47.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:46 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:45:47.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:46 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:45:47.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:46 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:45:47.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:46 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:45:47.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:46 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:45:47.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:46 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:45:48.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:47 smithi122 ceph-mon[93869]: pgmap v17395: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 9 op/s 2023-03-09T23:45:48.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:47 smithi181 ceph-mon[94189]: pgmap v17395: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 9 op/s 2023-03-09T23:45:48.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:47 smithi038 ceph-mon[92422]: pgmap v17395: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 27 KiB/s wr, 9 op/s 2023-03-09T23:45:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32120.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:45:49.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:45:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:45:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:49 smithi122 ceph-mon[93869]: pgmap v17396: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 267 KiB/s wr, 9 op/s 2023-03-09T23:45:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:45:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:45:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:49 smithi181 ceph-mon[94189]: pgmap v17396: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 267 KiB/s wr, 9 op/s 2023-03-09T23:45:49.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:45:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:45:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:49 smithi038 ceph-mon[92422]: pgmap v17396: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 267 KiB/s wr, 9 op/s 2023-03-09T23:45:50.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31661.434111 secs 2023-03-09T23:45:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31661.434111 secs 2023-03-09T23:45:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31661.434111 secs 2023-03-09T23:45:51.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:45:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:45:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:51 smithi038 ceph-mon[92422]: pgmap v17397: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 293 KiB/s wr, 10 op/s 2023-03-09T23:45:52.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:45:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:45:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:51 smithi122 ceph-mon[93869]: pgmap v17397: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 293 KiB/s wr, 10 op/s 2023-03-09T23:45:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:45:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:45:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:51 smithi181 ceph-mon[94189]: pgmap v17397: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 293 KiB/s wr, 10 op/s 2023-03-09T23:45:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32125.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:45:53.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:53 smithi122 ceph-mon[93869]: pgmap v17398: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 531 KiB/s rd, 527 KiB/s wr, 11 op/s 2023-03-09T23:45:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:53 smithi181 ceph-mon[94189]: pgmap v17398: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 531 KiB/s rd, 527 KiB/s wr, 11 op/s 2023-03-09T23:45:53.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:53 smithi038 ceph-mon[92422]: pgmap v17398: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 531 KiB/s rd, 527 KiB/s wr, 11 op/s 2023-03-09T23:45:55.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:45:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:45:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:55 smithi122 ceph-mon[93869]: pgmap v17399: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 684 KiB/s rd, 671 KiB/s wr, 10 op/s 2023-03-09T23:45:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31666.434196 secs 2023-03-09T23:45:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:45:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:45:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:55 smithi181 ceph-mon[94189]: pgmap v17399: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 684 KiB/s rd, 671 KiB/s wr, 10 op/s 2023-03-09T23:45:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31666.434196 secs 2023-03-09T23:45:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:45:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:45:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:55 smithi038 ceph-mon[92422]: pgmap v17399: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 684 KiB/s rd, 671 KiB/s wr, 10 op/s 2023-03-09T23:45:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31666.434196 secs 2023-03-09T23:45:57.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:45:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:45:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:57 smithi122 ceph-mon[93869]: pgmap v17400: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 701 KiB/s wr, 13 op/s 2023-03-09T23:45:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:45:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:45:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:57 smithi181 ceph-mon[94189]: pgmap v17400: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 701 KiB/s wr, 13 op/s 2023-03-09T23:45:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:45:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:45:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:57 smithi038 ceph-mon[92422]: pgmap v17400: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 701 KiB/s wr, 13 op/s 2023-03-09T23:45:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32130.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:45:58.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:58 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 31668.849763 secs 2023-03-09T23:45:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:58 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:45:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:58 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:45:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:58 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:45:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:58 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 31668.849763 secs 2023-03-09T23:45:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:58 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:45:58.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:58 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:45:58.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:58 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:45:58.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:58 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 31668.849763 secs 2023-03-09T23:45:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:58 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:45:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:58 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:45:59.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:58 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:45:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:45:59 smithi181 ceph-mon[94189]: pgmap v17401: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 701 KiB/s wr, 12 op/s 2023-03-09T23:45:59.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:45:59 smithi038 ceph-mon[92422]: pgmap v17401: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 701 KiB/s wr, 12 op/s 2023-03-09T23:46:00.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:45:59 smithi122 ceph-mon[93869]: pgmap v17401: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 701 KiB/s wr, 12 op/s 2023-03-09T23:46:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31671.434256 secs 2023-03-09T23:46:01.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31671.434256 secs 2023-03-09T23:46:01.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31671.434256 secs 2023-03-09T23:46:01.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:01 smithi038 ceph-mon[92422]: pgmap v17402: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 760 KiB/s rd, 790 KiB/s wr, 16 op/s 2023-03-09T23:46:02.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:01 smithi122 ceph-mon[93869]: pgmap v17402: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 760 KiB/s rd, 790 KiB/s wr, 16 op/s 2023-03-09T23:46:02.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:01 smithi181 ceph-mon[94189]: pgmap v17402: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 760 KiB/s rd, 790 KiB/s wr, 16 op/s 2023-03-09T23:46:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32135.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:46:03.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31673.849884 secs 2023-03-09T23:46:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:03 smithi122 ceph-mon[93869]: pgmap v17403: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 764 KiB/s wr, 16 op/s 2023-03-09T23:46:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:03 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:46:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31673.849884 secs 2023-03-09T23:46:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:03 smithi181 ceph-mon[94189]: pgmap v17403: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 764 KiB/s wr, 16 op/s 2023-03-09T23:46:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:03 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:46:03.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31673.849884 secs 2023-03-09T23:46:03.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:03 smithi038 ceph-mon[92422]: pgmap v17403: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 764 KiB/s wr, 16 op/s 2023-03-09T23:46:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:03 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:46:04.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:46:04.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:46:04.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:46:04.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:46:04.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:46:04.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:46:04.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:46:04.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:46:04.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:46:06.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:05 smithi122 ceph-mon[93869]: pgmap v17404: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 589 KiB/s wr, 26 op/s 2023-03-09T23:46:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:05 smithi122 ceph-mon[93869]: osdmap e1303: 12 total, 12 up, 12 in 2023-03-09T23:46:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31676.434319 secs 2023-03-09T23:46:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:05 smithi181 ceph-mon[94189]: pgmap v17404: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 589 KiB/s wr, 26 op/s 2023-03-09T23:46:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:05 smithi181 ceph-mon[94189]: osdmap e1303: 12 total, 12 up, 12 in 2023-03-09T23:46:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31676.434319 secs 2023-03-09T23:46:06.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:05 smithi038 ceph-mon[92422]: pgmap v17404: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 589 KiB/s wr, 26 op/s 2023-03-09T23:46:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:05 smithi038 ceph-mon[92422]: osdmap e1303: 12 total, 12 up, 12 in 2023-03-09T23:46:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31676.434319 secs 2023-03-09T23:46:07.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:06 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:46:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:46:07.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:06 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:46:07.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:46:07.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:06 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:46:07.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:46:08.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:07 smithi122 ceph-mon[93869]: pgmap v17406: 129 pgs: 3 active+clean+snaptrim_wait, 126 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 659 KiB/s rd, 700 KiB/s wr, 30 op/s 2023-03-09T23:46:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:46:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:46:08.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:07 smithi181 ceph-mon[94189]: pgmap v17406: 129 pgs: 3 active+clean+snaptrim_wait, 126 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 659 KiB/s rd, 700 KiB/s wr, 30 op/s 2023-03-09T23:46:08.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:46:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:46:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:07 smithi038 ceph-mon[92422]: pgmap v17406: 129 pgs: 3 active+clean+snaptrim_wait, 126 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 659 KiB/s rd, 700 KiB/s wr, 30 op/s 2023-03-09T23:46:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:46:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:46:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32140.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:46:09.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31678.849996 secs 2023-03-09T23:46:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:46:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:46:09.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31678.849996 secs 2023-03-09T23:46:09.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:46:09.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:46:09.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31678.849996 secs 2023-03-09T23:46:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:46:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:46:10.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:09 smithi122 ceph-mon[93869]: pgmap v17407: 129 pgs: 3 active+clean+snaptrim_wait, 126 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 630 KiB/s rd, 669 KiB/s wr, 29 op/s 2023-03-09T23:46:10.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:09 smithi181 ceph-mon[94189]: pgmap v17407: 129 pgs: 3 active+clean+snaptrim_wait, 126 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 630 KiB/s rd, 669 KiB/s wr, 29 op/s 2023-03-09T23:46:10.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:09 smithi038 ceph-mon[92422]: pgmap v17407: 129 pgs: 3 active+clean+snaptrim_wait, 126 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 630 KiB/s rd, 669 KiB/s wr, 29 op/s 2023-03-09T23:46:11.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31681.434381 secs 2023-03-09T23:46:11.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31681.434381 secs 2023-03-09T23:46:11.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31681.434381 secs 2023-03-09T23:46:12.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:11 smithi122 ceph-mon[93869]: pgmap v17408: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 439 KiB/s wr, 25 op/s 2023-03-09T23:46:12.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:11 smithi181 ceph-mon[94189]: pgmap v17408: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 439 KiB/s wr, 25 op/s 2023-03-09T23:46:12.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:11 smithi038 ceph-mon[92422]: pgmap v17408: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 376 KiB/s rd, 439 KiB/s wr, 25 op/s 2023-03-09T23:46:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32145.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:46:13.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31683.850137 secs 2023-03-09T23:46:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:13 smithi122 ceph-mon[93869]: osdmap e1304: 12 total, 12 up, 12 in 2023-03-09T23:46:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:13 smithi122 ceph-mon[93869]: pgmap v17410: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 460 KiB/s wr, 11 op/s 2023-03-09T23:46:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31683.850137 secs 2023-03-09T23:46:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:13 smithi181 ceph-mon[94189]: osdmap e1304: 12 total, 12 up, 12 in 2023-03-09T23:46:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:13 smithi181 ceph-mon[94189]: pgmap v17410: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 460 KiB/s wr, 11 op/s 2023-03-09T23:46:13.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31683.850137 secs 2023-03-09T23:46:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:13 smithi038 ceph-mon[92422]: osdmap e1304: 12 total, 12 up, 12 in 2023-03-09T23:46:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:13 smithi038 ceph-mon[92422]: pgmap v17410: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 460 KiB/s wr, 11 op/s 2023-03-09T23:46:15.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:15 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:46:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:15 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:46:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:15 smithi038 ceph-mon[92422]: pgmap v17411: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 371 KiB/s wr, 10 op/s 2023-03-09T23:46:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31686.434443 secs 2023-03-09T23:46:16.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:15 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:46:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:15 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:46:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:15 smithi122 ceph-mon[93869]: pgmap v17411: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 371 KiB/s wr, 10 op/s 2023-03-09T23:46:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31686.434443 secs 2023-03-09T23:46:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:15 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:46:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:15 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:46:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:15 smithi181 ceph-mon[94189]: pgmap v17411: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 371 KiB/s wr, 10 op/s 2023-03-09T23:46:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31686.434443 secs 2023-03-09T23:46:17.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:46:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:46:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:17 smithi038 ceph-mon[92422]: pgmap v17412: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 166 KiB/s wr, 9 op/s 2023-03-09T23:46:18.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:46:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:46:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:17 smithi122 ceph-mon[93869]: pgmap v17412: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 166 KiB/s wr, 9 op/s 2023-03-09T23:46:18.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:46:18.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:46:18.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:17 smithi181 ceph-mon[94189]: pgmap v17412: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 166 KiB/s wr, 9 op/s 2023-03-09T23:46:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32150.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:46:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31688.850201 secs 2023-03-09T23:46:19.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31688.850201 secs 2023-03-09T23:46:19.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31688.850201 secs 2023-03-09T23:46:20.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:19 smithi122 ceph-mon[93869]: pgmap v17413: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 165 KiB/s wr, 8 op/s 2023-03-09T23:46:20.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:19 smithi181 ceph-mon[94189]: pgmap v17413: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 165 KiB/s wr, 8 op/s 2023-03-09T23:46:20.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:19 smithi038 ceph-mon[92422]: pgmap v17413: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 165 KiB/s wr, 8 op/s 2023-03-09T23:46:21.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31691.434523 secs 2023-03-09T23:46:21.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31691.434523 secs 2023-03-09T23:46:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31691.434523 secs 2023-03-09T23:46:22.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:21 smithi122 ceph-mon[93869]: pgmap v17414: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 116 KiB/s wr, 9 op/s 2023-03-09T23:46:22.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:21 smithi181 ceph-mon[94189]: pgmap v17414: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 116 KiB/s wr, 9 op/s 2023-03-09T23:46:22.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:21 smithi038 ceph-mon[92422]: pgmap v17414: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 116 KiB/s wr, 9 op/s 2023-03-09T23:46:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32155.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:46:23.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31693.850259 secs 2023-03-09T23:46:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:23 smithi122 ceph-mon[93869]: pgmap v17415: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 114 KiB/s wr, 8 op/s 2023-03-09T23:46:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31693.850259 secs 2023-03-09T23:46:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:23 smithi181 ceph-mon[94189]: pgmap v17415: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 114 KiB/s wr, 8 op/s 2023-03-09T23:46:23.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31693.850259 secs 2023-03-09T23:46:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:23 smithi038 ceph-mon[92422]: pgmap v17415: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 114 KiB/s wr, 8 op/s 2023-03-09T23:46:26.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:25 smithi122 ceph-mon[93869]: pgmap v17416: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 96 KiB/s wr, 7 op/s 2023-03-09T23:46:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31696.434585 secs 2023-03-09T23:46:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:25 smithi181 ceph-mon[94189]: pgmap v17416: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 96 KiB/s wr, 7 op/s 2023-03-09T23:46:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31696.434585 secs 2023-03-09T23:46:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:25 smithi038 ceph-mon[92422]: pgmap v17416: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 96 KiB/s wr, 7 op/s 2023-03-09T23:46:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31696.434585 secs 2023-03-09T23:46:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:27 smithi122 ceph-mon[93869]: pgmap v17417: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 96 KiB/s wr, 8 op/s 2023-03-09T23:46:28.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:27 smithi181 ceph-mon[94189]: pgmap v17417: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 96 KiB/s wr, 8 op/s 2023-03-09T23:46:28.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:27 smithi038 ceph-mon[92422]: pgmap v17417: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 96 KiB/s wr, 8 op/s 2023-03-09T23:46:28.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32160.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:46:29.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31698.850354 secs 2023-03-09T23:46:29.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31698.850354 secs 2023-03-09T23:46:29.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31698.850354 secs 2023-03-09T23:46:30.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:29 smithi122 ceph-mon[93869]: pgmap v17418: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 96 KiB/s wr, 6 op/s 2023-03-09T23:46:30.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:29 smithi181 ceph-mon[94189]: pgmap v17418: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 96 KiB/s wr, 6 op/s 2023-03-09T23:46:30.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:29 smithi038 ceph-mon[92422]: pgmap v17418: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 96 KiB/s wr, 6 op/s 2023-03-09T23:46:31.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31701.434638 secs 2023-03-09T23:46:31.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31701.434638 secs 2023-03-09T23:46:31.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31701.434638 secs 2023-03-09T23:46:32.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:31 smithi122 ceph-mon[93869]: pgmap v17419: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 96 KiB/s wr, 8 op/s 2023-03-09T23:46:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:31 smithi181 ceph-mon[94189]: pgmap v17419: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 96 KiB/s wr, 8 op/s 2023-03-09T23:46:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:31 smithi038 ceph-mon[92422]: pgmap v17419: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 96 KiB/s wr, 8 op/s 2023-03-09T23:46:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32165.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:46:33.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31703.850500 secs 2023-03-09T23:46:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:33 smithi122 ceph-mon[93869]: pgmap v17420: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 5 op/s 2023-03-09T23:46:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31703.850500 secs 2023-03-09T23:46:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:33 smithi181 ceph-mon[94189]: pgmap v17420: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 5 op/s 2023-03-09T23:46:33.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31703.850500 secs 2023-03-09T23:46:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:33 smithi038 ceph-mon[92422]: pgmap v17420: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 5 op/s 2023-03-09T23:46:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:35 smithi122 ceph-mon[93869]: pgmap v17421: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T23:46:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31706.434698 secs 2023-03-09T23:46:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:35 smithi181 ceph-mon[94189]: pgmap v17421: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T23:46:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31706.434698 secs 2023-03-09T23:46:36.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:35 smithi038 ceph-mon[92422]: pgmap v17421: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T23:46:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31706.434698 secs 2023-03-09T23:46:38.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:37 smithi122 ceph-mon[93869]: pgmap v17422: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T23:46:38.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:37 smithi181 ceph-mon[94189]: pgmap v17422: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T23:46:38.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:37 smithi038 ceph-mon[92422]: pgmap v17422: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T23:46:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32170.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:46:39.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31708.850604 secs 2023-03-09T23:46:39.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31708.850604 secs 2023-03-09T23:46:39.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31708.850604 secs 2023-03-09T23:46:40.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:39 smithi122 ceph-mon[93869]: pgmap v17423: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-09T23:46:40.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:39 smithi181 ceph-mon[94189]: pgmap v17423: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-09T23:46:40.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:39 smithi038 ceph-mon[92422]: pgmap v17423: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-09T23:46:41.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31711.434758 secs 2023-03-09T23:46:41.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31711.434758 secs 2023-03-09T23:46:41.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31711.434758 secs 2023-03-09T23:46:42.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:41 smithi122 ceph-mon[93869]: pgmap v17424: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 8 op/s 2023-03-09T23:46:42.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:41 smithi181 ceph-mon[94189]: pgmap v17424: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 8 op/s 2023-03-09T23:46:42.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:41 smithi038 ceph-mon[92422]: pgmap v17424: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 8 op/s 2023-03-09T23:46:43.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32175.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:46:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:43 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 31713.850735 secs 2023-03-09T23:46:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:43 smithi122 ceph-mon[93869]: pgmap v17425: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 682 B/s wr, 6 op/s 2023-03-09T23:46:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:43 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 31713.850735 secs 2023-03-09T23:46:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:43 smithi181 ceph-mon[94189]: pgmap v17425: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 682 B/s wr, 6 op/s 2023-03-09T23:46:43.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:43 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 31713.850735 secs 2023-03-09T23:46:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:43 smithi038 ceph-mon[92422]: pgmap v17425: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 682 B/s wr, 6 op/s 2023-03-09T23:46:46.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:45 smithi122 ceph-mon[93869]: pgmap v17426: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 183 KiB/s wr, 6 op/s 2023-03-09T23:46:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31716.434817 secs 2023-03-09T23:46:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:45 smithi181 ceph-mon[94189]: pgmap v17426: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 183 KiB/s wr, 6 op/s 2023-03-09T23:46:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31716.434817 secs 2023-03-09T23:46:46.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:45 smithi038 ceph-mon[92422]: pgmap v17426: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 183 KiB/s wr, 6 op/s 2023-03-09T23:46:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31716.434817 secs 2023-03-09T23:46:47.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:46 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:46:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:46 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:46:47.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:46 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:46:47.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:46 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:46:47.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:46 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:46:47.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:46 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:46:48.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:47 smithi122 ceph-mon[93869]: pgmap v17427: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 268 KiB/s wr, 9 op/s 2023-03-09T23:46:48.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:47 smithi181 ceph-mon[94189]: pgmap v17427: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 268 KiB/s wr, 9 op/s 2023-03-09T23:46:48.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:47 smithi038 ceph-mon[92422]: pgmap v17427: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 268 KiB/s wr, 9 op/s 2023-03-09T23:46:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32180.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:46:49.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31718.850827 secs 2023-03-09T23:46:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:46:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:46:49.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31718.850827 secs 2023-03-09T23:46:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:46:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:46:49.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31718.850827 secs 2023-03-09T23:46:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:46:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:46:50.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:49 smithi122 ceph-mon[93869]: pgmap v17428: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 268 KiB/s wr, 7 op/s 2023-03-09T23:46:50.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:49 smithi181 ceph-mon[94189]: pgmap v17428: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 268 KiB/s wr, 7 op/s 2023-03-09T23:46:50.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:49 smithi038 ceph-mon[92422]: pgmap v17428: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 268 KiB/s wr, 7 op/s 2023-03-09T23:46:51.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:50 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:46:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:50 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:46:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31721.434878 secs 2023-03-09T23:46:51.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:50 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:46:51.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:50 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:46:51.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31721.434878 secs 2023-03-09T23:46:51.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:50 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:46:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:50 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:46:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31721.434878 secs 2023-03-09T23:46:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:51 smithi122 ceph-mon[93869]: pgmap v17429: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 537 KiB/s rd, 532 KiB/s wr, 12 op/s 2023-03-09T23:46:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:46:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:46:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:51 smithi181 ceph-mon[94189]: pgmap v17429: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 537 KiB/s rd, 532 KiB/s wr, 12 op/s 2023-03-09T23:46:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:46:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:46:52.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:51 smithi038 ceph-mon[92422]: pgmap v17429: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 537 KiB/s rd, 532 KiB/s wr, 12 op/s 2023-03-09T23:46:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:46:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:46:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32185.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:46:53.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31723.850939 secs 2023-03-09T23:46:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:53 smithi122 ceph-mon[93869]: pgmap v17430: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 535 KiB/s rd, 532 KiB/s wr, 9 op/s 2023-03-09T23:46:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31723.850939 secs 2023-03-09T23:46:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:53 smithi181 ceph-mon[94189]: pgmap v17430: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 535 KiB/s rd, 532 KiB/s wr, 9 op/s 2023-03-09T23:46:53.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31723.850939 secs 2023-03-09T23:46:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:53 smithi038 ceph-mon[92422]: pgmap v17430: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 535 KiB/s rd, 532 KiB/s wr, 9 op/s 2023-03-09T23:46:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:46:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:46:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:55 smithi038 ceph-mon[92422]: pgmap v17431: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 666 KiB/s rd, 652 KiB/s wr, 11 op/s 2023-03-09T23:46:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31726.434938 secs 2023-03-09T23:46:56.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:46:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:46:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:55 smithi122 ceph-mon[93869]: pgmap v17431: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 666 KiB/s rd, 652 KiB/s wr, 11 op/s 2023-03-09T23:46:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31726.434938 secs 2023-03-09T23:46:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:46:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:46:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:55 smithi181 ceph-mon[94189]: pgmap v17431: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 666 KiB/s rd, 652 KiB/s wr, 11 op/s 2023-03-09T23:46:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31726.434938 secs 2023-03-09T23:46:57.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:46:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:46:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:57 smithi038 ceph-mon[92422]: pgmap v17432: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 562 KiB/s rd, 549 KiB/s wr, 14 op/s 2023-03-09T23:46:58.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:46:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:46:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:57 smithi122 ceph-mon[93869]: pgmap v17432: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 562 KiB/s rd, 549 KiB/s wr, 14 op/s 2023-03-09T23:46:58.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:46:58.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:46:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:57 smithi181 ceph-mon[94189]: pgmap v17432: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 562 KiB/s rd, 549 KiB/s wr, 14 op/s 2023-03-09T23:46:58.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32190.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:46:58.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31728.851054 secs 2023-03-09T23:46:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31728.851054 secs 2023-03-09T23:46:59.249 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31728.851054 secs 2023-03-09T23:47:00.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:46:59 smithi122 ceph-mon[93869]: pgmap v17433: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 465 KiB/s wr, 11 op/s 2023-03-09T23:47:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:46:59 smithi181 ceph-mon[94189]: pgmap v17433: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 465 KiB/s wr, 11 op/s 2023-03-09T23:47:00.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:46:59 smithi038 ceph-mon[92422]: pgmap v17433: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 465 KiB/s wr, 11 op/s 2023-03-09T23:47:01.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31731.435001 secs 2023-03-09T23:47:01.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31731.435001 secs 2023-03-09T23:47:01.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31731.435001 secs 2023-03-09T23:47:02.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:01 smithi122 ceph-mon[93869]: pgmap v17434: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 723 KiB/s wr, 17 op/s 2023-03-09T23:47:02.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:01 smithi181 ceph-mon[94189]: pgmap v17434: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 723 KiB/s wr, 17 op/s 2023-03-09T23:47:02.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:01 smithi038 ceph-mon[92422]: pgmap v17434: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 723 KiB/s wr, 17 op/s 2023-03-09T23:47:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32195.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:47:03.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31733.851175 secs 2023-03-09T23:47:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:03 smithi038 ceph-mon[92422]: pgmap v17435: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 459 KiB/s wr, 12 op/s 2023-03-09T23:47:04.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31733.851175 secs 2023-03-09T23:47:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:03 smithi122 ceph-mon[93869]: pgmap v17435: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 459 KiB/s wr, 12 op/s 2023-03-09T23:47:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31733.851175 secs 2023-03-09T23:47:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:03 smithi181 ceph-mon[94189]: pgmap v17435: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 459 KiB/s wr, 12 op/s 2023-03-09T23:47:04.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:47:04.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:47:04.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:47:04.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:04 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:47:05.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:47:05.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:47:05.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:47:05.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:04 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:47:05.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:47:05.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:47:05.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:47:05.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:04 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:47:05.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:05 smithi038 ceph-mon[92422]: osdmap e1305: 12 total, 12 up, 12 in 2023-03-09T23:47:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:05 smithi038 ceph-mon[92422]: pgmap v17437: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 652 KiB/s wr, 29 op/s 2023-03-09T23:47:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31736.435063 secs 2023-03-09T23:47:06.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:05 smithi122 ceph-mon[93869]: osdmap e1305: 12 total, 12 up, 12 in 2023-03-09T23:47:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:05 smithi122 ceph-mon[93869]: pgmap v17437: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 652 KiB/s wr, 29 op/s 2023-03-09T23:47:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31736.435063 secs 2023-03-09T23:47:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:05 smithi181 ceph-mon[94189]: osdmap e1305: 12 total, 12 up, 12 in 2023-03-09T23:47:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:05 smithi181 ceph-mon[94189]: pgmap v17437: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 652 KiB/s wr, 29 op/s 2023-03-09T23:47:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31736.435063 secs 2023-03-09T23:47:08.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:47:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:47:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:07 smithi122 ceph-mon[93869]: pgmap v17438: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 558 KiB/s rd, 597 KiB/s wr, 29 op/s 2023-03-09T23:47:08.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:47:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:47:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:07 smithi181 ceph-mon[94189]: pgmap v17438: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 558 KiB/s rd, 597 KiB/s wr, 29 op/s 2023-03-09T23:47:08.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:47:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:47:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:07 smithi038 ceph-mon[92422]: pgmap v17438: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 558 KiB/s rd, 597 KiB/s wr, 29 op/s 2023-03-09T23:47:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32200.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:47:09.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31738.851276 secs 2023-03-09T23:47:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:47:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:47:09.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31738.851276 secs 2023-03-09T23:47:09.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:47:09.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:47:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31738.851276 secs 2023-03-09T23:47:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:47:09.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:47:10.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:09 smithi122 ceph-mon[93869]: pgmap v17439: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 558 KiB/s rd, 597 KiB/s wr, 29 op/s 2023-03-09T23:47:10.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:09 smithi181 ceph-mon[94189]: pgmap v17439: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 558 KiB/s rd, 597 KiB/s wr, 29 op/s 2023-03-09T23:47:10.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:09 smithi038 ceph-mon[92422]: pgmap v17439: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 558 KiB/s rd, 597 KiB/s wr, 29 op/s 2023-03-09T23:47:11.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31741.435126 secs 2023-03-09T23:47:11.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31741.435126 secs 2023-03-09T23:47:11.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31741.435126 secs 2023-03-09T23:47:12.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:11 smithi122 ceph-mon[93869]: pgmap v17440: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 383 KiB/s rd, 288 KiB/s wr, 25 op/s 2023-03-09T23:47:12.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:11 smithi181 ceph-mon[94189]: pgmap v17440: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 383 KiB/s rd, 288 KiB/s wr, 25 op/s 2023-03-09T23:47:12.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:11 smithi038 ceph-mon[92422]: pgmap v17440: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 383 KiB/s rd, 288 KiB/s wr, 25 op/s 2023-03-09T23:47:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32205.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:47:13.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31743.851394 secs 2023-03-09T23:47:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:13 smithi122 ceph-mon[93869]: osdmap e1306: 12 total, 12 up, 12 in 2023-03-09T23:47:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:13 smithi122 ceph-mon[93869]: pgmap v17442: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 51 KiB/s wr, 10 op/s 2023-03-09T23:47:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31743.851394 secs 2023-03-09T23:47:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:13 smithi181 ceph-mon[94189]: osdmap e1306: 12 total, 12 up, 12 in 2023-03-09T23:47:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:13 smithi181 ceph-mon[94189]: pgmap v17442: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 51 KiB/s wr, 10 op/s 2023-03-09T23:47:13.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31743.851394 secs 2023-03-09T23:47:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:13 smithi038 ceph-mon[92422]: osdmap e1306: 12 total, 12 up, 12 in 2023-03-09T23:47:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:13 smithi038 ceph-mon[92422]: pgmap v17442: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 51 KiB/s wr, 10 op/s 2023-03-09T23:47:16.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:15 smithi122 ceph-mon[93869]: pgmap v17443: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 206 KiB/s wr, 10 op/s 2023-03-09T23:47:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31746.435185 secs 2023-03-09T23:47:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:15 smithi181 ceph-mon[94189]: pgmap v17443: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 206 KiB/s wr, 10 op/s 2023-03-09T23:47:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31746.435185 secs 2023-03-09T23:47:16.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:15 smithi038 ceph-mon[92422]: pgmap v17443: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 206 KiB/s wr, 10 op/s 2023-03-09T23:47:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31746.435185 secs 2023-03-09T23:47:18.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:17 smithi122 ceph-mon[93869]: pgmap v17444: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 166 KiB/s wr, 8 op/s 2023-03-09T23:47:18.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:17 smithi181 ceph-mon[94189]: pgmap v17444: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 166 KiB/s wr, 8 op/s 2023-03-09T23:47:18.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:17 smithi038 ceph-mon[92422]: pgmap v17444: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 166 KiB/s wr, 8 op/s 2023-03-09T23:47:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32210.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:47:19.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31748.851509 secs 2023-03-09T23:47:19.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31748.851509 secs 2023-03-09T23:47:19.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31748.851509 secs 2023-03-09T23:47:20.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:19 smithi122 ceph-mon[93869]: pgmap v17445: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 165 KiB/s wr, 8 op/s 2023-03-09T23:47:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:19 smithi181 ceph-mon[94189]: pgmap v17445: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 165 KiB/s wr, 8 op/s 2023-03-09T23:47:20.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:19 smithi038 ceph-mon[92422]: pgmap v17445: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 165 KiB/s wr, 8 op/s 2023-03-09T23:47:21.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31751.435246 secs 2023-03-09T23:47:21.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31751.435246 secs 2023-03-09T23:47:21.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31751.435246 secs 2023-03-09T23:47:22.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:21 smithi122 ceph-mon[93869]: pgmap v17446: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 165 KiB/s wr, 7 op/s 2023-03-09T23:47:22.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:21 smithi181 ceph-mon[94189]: pgmap v17446: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 165 KiB/s wr, 7 op/s 2023-03-09T23:47:22.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:21 smithi038 ceph-mon[92422]: pgmap v17446: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 165 KiB/s wr, 7 op/s 2023-03-09T23:47:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32215.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:47:23.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31753.851626 secs 2023-03-09T23:47:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:23 smithi038 ceph-mon[92422]: pgmap v17447: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 163 KiB/s wr, 7 op/s 2023-03-09T23:47:24.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31753.851626 secs 2023-03-09T23:47:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:23 smithi122 ceph-mon[93869]: pgmap v17447: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 163 KiB/s wr, 7 op/s 2023-03-09T23:47:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31753.851626 secs 2023-03-09T23:47:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:23 smithi181 ceph-mon[94189]: pgmap v17447: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 163 KiB/s wr, 7 op/s 2023-03-09T23:47:26.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:25 smithi122 ceph-mon[93869]: pgmap v17448: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 137 KiB/s wr, 6 op/s 2023-03-09T23:47:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31756.435313 secs 2023-03-09T23:47:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:25 smithi181 ceph-mon[94189]: pgmap v17448: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 137 KiB/s wr, 6 op/s 2023-03-09T23:47:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31756.435313 secs 2023-03-09T23:47:26.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:25 smithi038 ceph-mon[92422]: pgmap v17448: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 137 KiB/s wr, 6 op/s 2023-03-09T23:47:26.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31756.435313 secs 2023-03-09T23:47:28.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:27 smithi122 ceph-mon[93869]: pgmap v17449: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T23:47:28.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:27 smithi181 ceph-mon[94189]: pgmap v17449: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T23:47:28.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:27 smithi038 ceph-mon[92422]: pgmap v17449: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T23:47:28.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32220.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:47:29.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31758.851742 secs 2023-03-09T23:47:29.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31758.851742 secs 2023-03-09T23:47:29.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31758.851742 secs 2023-03-09T23:47:30.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:29 smithi122 ceph-mon[93869]: pgmap v17450: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T23:47:30.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:29 smithi181 ceph-mon[94189]: pgmap v17450: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T23:47:30.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:29 smithi038 ceph-mon[92422]: pgmap v17450: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T23:47:31.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31761.435370 secs 2023-03-09T23:47:31.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31761.435370 secs 2023-03-09T23:47:31.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31761.435370 secs 2023-03-09T23:47:32.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:31 smithi122 ceph-mon[93869]: pgmap v17451: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T23:47:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:31 smithi181 ceph-mon[94189]: pgmap v17451: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T23:47:32.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:31 smithi038 ceph-mon[92422]: pgmap v17451: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T23:47:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32225.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:47:33.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31763.851839 secs 2023-03-09T23:47:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:33 smithi038 ceph-mon[92422]: pgmap v17452: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T23:47:34.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31763.851839 secs 2023-03-09T23:47:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:33 smithi122 ceph-mon[93869]: pgmap v17452: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T23:47:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31763.851839 secs 2023-03-09T23:47:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:33 smithi181 ceph-mon[94189]: pgmap v17452: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T23:47:36.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:35 smithi122 ceph-mon[93869]: pgmap v17453: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T23:47:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31766.435433 secs 2023-03-09T23:47:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:35 smithi181 ceph-mon[94189]: pgmap v17453: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T23:47:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31766.435433 secs 2023-03-09T23:47:36.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:35 smithi038 ceph-mon[92422]: pgmap v17453: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T23:47:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31766.435433 secs 2023-03-09T23:47:38.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:37 smithi122 ceph-mon[93869]: pgmap v17454: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 96 KiB/s wr, 8 op/s 2023-03-09T23:47:38.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:37 smithi181 ceph-mon[94189]: pgmap v17454: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 96 KiB/s wr, 8 op/s 2023-03-09T23:47:38.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:37 smithi038 ceph-mon[92422]: pgmap v17454: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 96 KiB/s wr, 8 op/s 2023-03-09T23:47:38.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32230.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:47:39.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31768.851957 secs 2023-03-09T23:47:39.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31768.851957 secs 2023-03-09T23:47:39.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31768.851957 secs 2023-03-09T23:47:40.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:39 smithi122 ceph-mon[93869]: pgmap v17455: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 96 KiB/s wr, 6 op/s 2023-03-09T23:47:40.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:39 smithi181 ceph-mon[94189]: pgmap v17455: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 96 KiB/s wr, 6 op/s 2023-03-09T23:47:40.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:39 smithi038 ceph-mon[92422]: pgmap v17455: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 96 KiB/s wr, 6 op/s 2023-03-09T23:47:41.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31771.435528 secs 2023-03-09T23:47:41.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31771.435528 secs 2023-03-09T23:47:41.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31771.435528 secs 2023-03-09T23:47:42.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:41 smithi122 ceph-mon[93869]: pgmap v17456: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 118 KiB/s wr, 10 op/s 2023-03-09T23:47:42.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:41 smithi181 ceph-mon[94189]: pgmap v17456: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 118 KiB/s wr, 10 op/s 2023-03-09T23:47:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:41 smithi038 ceph-mon[92422]: pgmap v17456: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 118 KiB/s wr, 10 op/s 2023-03-09T23:47:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32235.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:47:43.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31773.852078 secs 2023-03-09T23:47:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:43 smithi038 ceph-mon[92422]: pgmap v17457: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 118 KiB/s wr, 7 op/s 2023-03-09T23:47:44.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31773.852078 secs 2023-03-09T23:47:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:43 smithi122 ceph-mon[93869]: pgmap v17457: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 118 KiB/s wr, 7 op/s 2023-03-09T23:47:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31773.852078 secs 2023-03-09T23:47:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:43 smithi181 ceph-mon[94189]: pgmap v17457: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 118 KiB/s wr, 7 op/s 2023-03-09T23:47:46.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:45 smithi122 ceph-mon[93869]: pgmap v17458: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 118 KiB/s wr, 8 op/s 2023-03-09T23:47:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31776.435619 secs 2023-03-09T23:47:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:45 smithi181 ceph-mon[94189]: pgmap v17458: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 118 KiB/s wr, 8 op/s 2023-03-09T23:47:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31776.435619 secs 2023-03-09T23:47:46.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:45 smithi038 ceph-mon[92422]: pgmap v17458: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 118 KiB/s wr, 8 op/s 2023-03-09T23:47:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31776.435619 secs 2023-03-09T23:47:48.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:47 smithi122 ceph-mon[93869]: pgmap v17459: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 389 KiB/s wr, 10 op/s 2023-03-09T23:47:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:47:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:47:48.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:47 smithi181 ceph-mon[94189]: pgmap v17459: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 389 KiB/s wr, 10 op/s 2023-03-09T23:47:48.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:47:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:47:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:47 smithi038 ceph-mon[92422]: pgmap v17459: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 389 KiB/s wr, 10 op/s 2023-03-09T23:47:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:47:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:47:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32240.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:47:49.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31778.852182 secs 2023-03-09T23:47:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:47:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:47:49.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31778.852182 secs 2023-03-09T23:47:49.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:47:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:47:49.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31778.852182 secs 2023-03-09T23:47:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:47:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:47:50.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:49 smithi122 ceph-mon[93869]: pgmap v17460: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 293 KiB/s wr, 8 op/s 2023-03-09T23:47:50.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:49 smithi181 ceph-mon[94189]: pgmap v17460: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 293 KiB/s wr, 8 op/s 2023-03-09T23:47:50.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:49 smithi038 ceph-mon[92422]: pgmap v17460: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 293 KiB/s wr, 8 op/s 2023-03-09T23:47:51.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31781.435690 secs 2023-03-09T23:47:51.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31781.435690 secs 2023-03-09T23:47:51.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31781.435690 secs 2023-03-09T23:47:52.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:51 smithi122 ceph-mon[93869]: pgmap v17461: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 319 KiB/s wr, 12 op/s 2023-03-09T23:47:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:51 smithi181 ceph-mon[94189]: pgmap v17461: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 319 KiB/s wr, 12 op/s 2023-03-09T23:47:52.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:51 smithi038 ceph-mon[92422]: pgmap v17461: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 319 KiB/s wr, 12 op/s 2023-03-09T23:47:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32245.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:47:53.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:53 smithi038 ceph-mon[92422]: pgmap v17462: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 298 KiB/s wr, 9 op/s 2023-03-09T23:47:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:53 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:47:54.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:53 smithi122 ceph-mon[93869]: pgmap v17462: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 298 KiB/s wr, 9 op/s 2023-03-09T23:47:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:53 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:47:54.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:53 smithi181 ceph-mon[94189]: pgmap v17462: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 298 KiB/s wr, 9 op/s 2023-03-09T23:47:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:53 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:47:54.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:54 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:47:54.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:54 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:47:54.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:54 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:47:55.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:54 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:47:55.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:54 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:47:55.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:54 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:47:55.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:54 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:47:55.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:54 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:47:55.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:54 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:47:56.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:47:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:47:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:55 smithi122 ceph-mon[93869]: pgmap v17463: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 674 KiB/s rd, 640 KiB/s wr, 10 op/s 2023-03-09T23:47:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31786.435770 secs 2023-03-09T23:47:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:47:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:47:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:55 smithi181 ceph-mon[94189]: pgmap v17463: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 674 KiB/s rd, 640 KiB/s wr, 10 op/s 2023-03-09T23:47:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31786.435770 secs 2023-03-09T23:47:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:47:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:47:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:55 smithi038 ceph-mon[92422]: pgmap v17463: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 674 KiB/s rd, 640 KiB/s wr, 10 op/s 2023-03-09T23:47:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31786.435770 secs 2023-03-09T23:47:58.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:47:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:47:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:57 smithi122 ceph-mon[93869]: pgmap v17464: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 705 KiB/s rd, 670 KiB/s wr, 13 op/s 2023-03-09T23:47:58.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:47:58.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:47:58.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:57 smithi181 ceph-mon[94189]: pgmap v17464: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 705 KiB/s rd, 670 KiB/s wr, 13 op/s 2023-03-09T23:47:58.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:47:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:47:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:57 smithi038 ceph-mon[92422]: pgmap v17464: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 705 KiB/s rd, 670 KiB/s wr, 13 op/s 2023-03-09T23:47:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32250.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:48:00.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:47:59 smithi122 ceph-mon[93869]: pgmap v17465: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 611 KiB/s rd, 578 KiB/s wr, 11 op/s 2023-03-09T23:48:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:47:59 smithi181 ceph-mon[94189]: pgmap v17465: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 611 KiB/s rd, 578 KiB/s wr, 11 op/s 2023-03-09T23:48:00.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:47:59 smithi038 ceph-mon[92422]: pgmap v17465: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 611 KiB/s rd, 578 KiB/s wr, 11 op/s 2023-03-09T23:48:01.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31791.435830 secs 2023-03-09T23:48:01.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31791.435830 secs 2023-03-09T23:48:01.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31791.435830 secs 2023-03-09T23:48:02.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:01 smithi122 ceph-mon[93869]: pgmap v17466: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 655 KiB/s rd, 621 KiB/s wr, 15 op/s 2023-03-09T23:48:02.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:01 smithi181 ceph-mon[94189]: pgmap v17466: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 655 KiB/s rd, 621 KiB/s wr, 15 op/s 2023-03-09T23:48:02.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:01 smithi038 ceph-mon[92422]: pgmap v17466: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 655 KiB/s rd, 621 KiB/s wr, 15 op/s 2023-03-09T23:48:03.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32255.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:48:03.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31793.852564 secs 2023-03-09T23:48:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:03 smithi038 ceph-mon[92422]: pgmap v17467: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 769 KiB/s rd, 757 KiB/s wr, 12 op/s 2023-03-09T23:48:04.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31793.852564 secs 2023-03-09T23:48:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:03 smithi122 ceph-mon[93869]: pgmap v17467: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 769 KiB/s rd, 757 KiB/s wr, 12 op/s 2023-03-09T23:48:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31793.852564 secs 2023-03-09T23:48:04.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:03 smithi181 ceph-mon[94189]: pgmap v17467: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 769 KiB/s rd, 757 KiB/s wr, 12 op/s 2023-03-09T23:48:06.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:05 smithi122 ceph-mon[93869]: osdmap e1307: 12 total, 12 up, 12 in 2023-03-09T23:48:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:05 smithi122 ceph-mon[93869]: pgmap v17469: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 508 KiB/s rd, 525 KiB/s wr, 27 op/s 2023-03-09T23:48:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:05 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:48:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31796.435894 secs 2023-03-09T23:48:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:05 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:48:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:05 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:48:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:05 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:48:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:05 smithi181 ceph-mon[94189]: osdmap e1307: 12 total, 12 up, 12 in 2023-03-09T23:48:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:05 smithi181 ceph-mon[94189]: pgmap v17469: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 508 KiB/s rd, 525 KiB/s wr, 27 op/s 2023-03-09T23:48:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:05 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:48:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31796.435894 secs 2023-03-09T23:48:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:05 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:48:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:05 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:48:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:05 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:48:06.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:05 smithi038 ceph-mon[92422]: osdmap e1307: 12 total, 12 up, 12 in 2023-03-09T23:48:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:05 smithi038 ceph-mon[92422]: pgmap v17469: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 508 KiB/s rd, 525 KiB/s wr, 27 op/s 2023-03-09T23:48:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:05 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:48:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31796.435894 secs 2023-03-09T23:48:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:05 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:48:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:05 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:48:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:05 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:48:07.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:06 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:48:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:06 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:48:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:48:07.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:06 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:48:07.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:06 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:48:07.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:48:07.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:06 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:48:07.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:06 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:48:07.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:48:08.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:48:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:48:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:07 smithi122 ceph-mon[93869]: pgmap v17470: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 678 KiB/s rd, 719 KiB/s wr, 31 op/s 2023-03-09T23:48:08.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:48:08.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:48:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:07 smithi181 ceph-mon[94189]: pgmap v17470: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 678 KiB/s rd, 719 KiB/s wr, 31 op/s 2023-03-09T23:48:08.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:48:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:48:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:07 smithi038 ceph-mon[92422]: pgmap v17470: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 678 KiB/s rd, 719 KiB/s wr, 31 op/s 2023-03-09T23:48:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32260.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:48:09.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31798.852678 secs 2023-03-09T23:48:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:48:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:48:09.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31798.852678 secs 2023-03-09T23:48:09.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:48:09.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:48:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31798.852678 secs 2023-03-09T23:48:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:48:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:48:10.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:09 smithi122 ceph-mon[93869]: pgmap v17471: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 464 KiB/s rd, 504 KiB/s wr, 30 op/s 2023-03-09T23:48:10.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:09 smithi181 ceph-mon[94189]: pgmap v17471: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 464 KiB/s rd, 504 KiB/s wr, 30 op/s 2023-03-09T23:48:10.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:09 smithi038 ceph-mon[92422]: pgmap v17471: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 464 KiB/s rd, 504 KiB/s wr, 30 op/s 2023-03-09T23:48:11.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31801.435976 secs 2023-03-09T23:48:11.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31801.435976 secs 2023-03-09T23:48:11.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31801.435976 secs 2023-03-09T23:48:12.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:11 smithi122 ceph-mon[93869]: pgmap v17472: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 578 KiB/s rd, 619 KiB/s wr, 30 op/s 2023-03-09T23:48:12.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:11 smithi181 ceph-mon[94189]: pgmap v17472: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 578 KiB/s rd, 619 KiB/s wr, 30 op/s 2023-03-09T23:48:12.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:11 smithi038 ceph-mon[92422]: pgmap v17472: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 578 KiB/s rd, 619 KiB/s wr, 30 op/s 2023-03-09T23:48:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32265.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:48:13.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31803.852810 secs 2023-03-09T23:48:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:13 smithi038 ceph-mon[92422]: osdmap e1308: 12 total, 12 up, 12 in 2023-03-09T23:48:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:13 smithi038 ceph-mon[92422]: pgmap v17474: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 490 KiB/s wr, 16 op/s 2023-03-09T23:48:14.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31803.852810 secs 2023-03-09T23:48:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:13 smithi122 ceph-mon[93869]: osdmap e1308: 12 total, 12 up, 12 in 2023-03-09T23:48:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:13 smithi122 ceph-mon[93869]: pgmap v17474: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 490 KiB/s wr, 16 op/s 2023-03-09T23:48:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31803.852810 secs 2023-03-09T23:48:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:13 smithi181 ceph-mon[94189]: osdmap e1308: 12 total, 12 up, 12 in 2023-03-09T23:48:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:13 smithi181 ceph-mon[94189]: pgmap v17474: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 490 KiB/s wr, 16 op/s 2023-03-09T23:48:16.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:15 smithi122 ceph-mon[93869]: pgmap v17475: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 398 KiB/s wr, 13 op/s 2023-03-09T23:48:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31806.436036 secs 2023-03-09T23:48:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:15 smithi181 ceph-mon[94189]: pgmap v17475: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 398 KiB/s wr, 13 op/s 2023-03-09T23:48:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31806.436036 secs 2023-03-09T23:48:16.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:15 smithi038 ceph-mon[92422]: pgmap v17475: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 398 KiB/s wr, 13 op/s 2023-03-09T23:48:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31806.436036 secs 2023-03-09T23:48:18.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:17 smithi122 ceph-mon[93869]: pgmap v17476: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 166 KiB/s wr, 9 op/s 2023-03-09T23:48:18.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:17 smithi181 ceph-mon[94189]: pgmap v17476: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 166 KiB/s wr, 9 op/s 2023-03-09T23:48:18.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:17 smithi038 ceph-mon[92422]: pgmap v17476: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 166 KiB/s wr, 9 op/s 2023-03-09T23:48:18.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32270.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:48:19.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31808.852928 secs 2023-03-09T23:48:19.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31808.852928 secs 2023-03-09T23:48:19.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31808.852928 secs 2023-03-09T23:48:20.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:19 smithi122 ceph-mon[93869]: pgmap v17477: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 166 KiB/s wr, 7 op/s 2023-03-09T23:48:20.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:19 smithi181 ceph-mon[94189]: pgmap v17477: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 166 KiB/s wr, 7 op/s 2023-03-09T23:48:20.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:19 smithi038 ceph-mon[92422]: pgmap v17477: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 166 KiB/s wr, 7 op/s 2023-03-09T23:48:21.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31811.436099 secs 2023-03-09T23:48:21.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31811.436099 secs 2023-03-09T23:48:21.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31811.436099 secs 2023-03-09T23:48:22.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:21 smithi122 ceph-mon[93869]: pgmap v17478: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 47 KiB/s wr, 6 op/s 2023-03-09T23:48:22.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:21 smithi181 ceph-mon[94189]: pgmap v17478: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 47 KiB/s wr, 6 op/s 2023-03-09T23:48:22.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:21 smithi038 ceph-mon[92422]: pgmap v17478: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 47 KiB/s wr, 6 op/s 2023-03-09T23:48:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32275.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:48:24.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31813.853076 secs 2023-03-09T23:48:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:23 smithi122 ceph-mon[93869]: pgmap v17479: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 47 KiB/s wr, 6 op/s 2023-03-09T23:48:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31813.853076 secs 2023-03-09T23:48:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:23 smithi181 ceph-mon[94189]: pgmap v17479: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 47 KiB/s wr, 6 op/s 2023-03-09T23:48:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31813.853076 secs 2023-03-09T23:48:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:23 smithi038 ceph-mon[92422]: pgmap v17479: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 47 KiB/s wr, 6 op/s 2023-03-09T23:48:26.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:25 smithi122 ceph-mon[93869]: pgmap v17480: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 97 KiB/s wr, 5 op/s 2023-03-09T23:48:26.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31816.436158 secs 2023-03-09T23:48:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:25 smithi181 ceph-mon[94189]: pgmap v17480: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 97 KiB/s wr, 5 op/s 2023-03-09T23:48:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31816.436158 secs 2023-03-09T23:48:26.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:25 smithi038 ceph-mon[92422]: pgmap v17480: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 97 KiB/s wr, 5 op/s 2023-03-09T23:48:26.299 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31816.436158 secs 2023-03-09T23:48:28.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:27 smithi122 ceph-mon[93869]: pgmap v17481: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 97 KiB/s wr, 8 op/s 2023-03-09T23:48:28.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:27 smithi181 ceph-mon[94189]: pgmap v17481: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 97 KiB/s wr, 8 op/s 2023-03-09T23:48:28.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:27 smithi038 ceph-mon[92422]: pgmap v17481: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 97 KiB/s wr, 8 op/s 2023-03-09T23:48:28.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32280.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:48:29.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31818.853199 secs 2023-03-09T23:48:29.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31818.853199 secs 2023-03-09T23:48:29.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31818.853199 secs 2023-03-09T23:48:30.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:29 smithi122 ceph-mon[93869]: pgmap v17482: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 97 KiB/s wr, 6 op/s 2023-03-09T23:48:30.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:29 smithi181 ceph-mon[94189]: pgmap v17482: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 97 KiB/s wr, 6 op/s 2023-03-09T23:48:30.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:29 smithi038 ceph-mon[92422]: pgmap v17482: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 97 KiB/s wr, 6 op/s 2023-03-09T23:48:31.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31821.436221 secs 2023-03-09T23:48:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31821.436221 secs 2023-03-09T23:48:31.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31821.436221 secs 2023-03-09T23:48:32.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:31 smithi122 ceph-mon[93869]: pgmap v17483: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 97 KiB/s wr, 8 op/s 2023-03-09T23:48:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:31 smithi181 ceph-mon[94189]: pgmap v17483: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 97 KiB/s wr, 8 op/s 2023-03-09T23:48:32.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:31 smithi038 ceph-mon[92422]: pgmap v17483: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 97 KiB/s wr, 8 op/s 2023-03-09T23:48:33.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32285.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:48:34.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31823.853325 secs 2023-03-09T23:48:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:33 smithi122 ceph-mon[93869]: pgmap v17484: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 5 op/s 2023-03-09T23:48:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31823.853325 secs 2023-03-09T23:48:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:33 smithi181 ceph-mon[94189]: pgmap v17484: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 5 op/s 2023-03-09T23:48:34.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31823.853325 secs 2023-03-09T23:48:34.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:33 smithi038 ceph-mon[92422]: pgmap v17484: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 5 op/s 2023-03-09T23:48:36.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:35 smithi122 ceph-mon[93869]: pgmap v17485: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 6 op/s 2023-03-09T23:48:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31826.436282 secs 2023-03-09T23:48:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:35 smithi181 ceph-mon[94189]: pgmap v17485: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 6 op/s 2023-03-09T23:48:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31826.436282 secs 2023-03-09T23:48:36.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:35 smithi038 ceph-mon[92422]: pgmap v17485: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 57 KiB/s wr, 6 op/s 2023-03-09T23:48:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31826.436282 secs 2023-03-09T23:48:38.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:37 smithi122 ceph-mon[93869]: pgmap v17486: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 8 op/s 2023-03-09T23:48:38.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:37 smithi181 ceph-mon[94189]: pgmap v17486: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 8 op/s 2023-03-09T23:48:38.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:37 smithi038 ceph-mon[92422]: pgmap v17486: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 8 op/s 2023-03-09T23:48:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32290.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:48:39.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31828.853405 secs 2023-03-09T23:48:39.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31828.853405 secs 2023-03-09T23:48:39.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31828.853405 secs 2023-03-09T23:48:40.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:39 smithi122 ceph-mon[93869]: pgmap v17487: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-09T23:48:40.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:39 smithi181 ceph-mon[94189]: pgmap v17487: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-09T23:48:40.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:39 smithi038 ceph-mon[92422]: pgmap v17487: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-09T23:48:41.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31831.436344 secs 2023-03-09T23:48:41.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31831.436344 secs 2023-03-09T23:48:41.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31831.436344 secs 2023-03-09T23:48:42.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:41 smithi122 ceph-mon[93869]: pgmap v17488: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 9 op/s 2023-03-09T23:48:42.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:41 smithi181 ceph-mon[94189]: pgmap v17488: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 9 op/s 2023-03-09T23:48:42.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:41 smithi038 ceph-mon[92422]: pgmap v17488: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 9 op/s 2023-03-09T23:48:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32295.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:48:44.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:43 smithi122 ceph-mon[93869]: pgmap v17489: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 682 B/s wr, 6 op/s 2023-03-09T23:48:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:43 smithi181 ceph-mon[94189]: pgmap v17489: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 682 B/s wr, 6 op/s 2023-03-09T23:48:44.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:43 smithi038 ceph-mon[92422]: pgmap v17489: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 682 B/s wr, 6 op/s 2023-03-09T23:48:46.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:45 smithi122 ceph-mon[93869]: pgmap v17490: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 241 KiB/s wr, 7 op/s 2023-03-09T23:48:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31836.436406 secs 2023-03-09T23:48:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:45 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:48:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:45 smithi181 ceph-mon[94189]: pgmap v17490: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 241 KiB/s wr, 7 op/s 2023-03-09T23:48:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31836.436406 secs 2023-03-09T23:48:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:45 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:48:46.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:45 smithi038 ceph-mon[92422]: pgmap v17490: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 241 KiB/s wr, 7 op/s 2023-03-09T23:48:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31836.436406 secs 2023-03-09T23:48:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:45 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:48:47.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:46 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:48:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:46 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:48:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:46 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:48:47.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:46 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:48:47.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:46 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:48:47.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:46 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:48:47.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:46 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:48:47.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:46 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:48:47.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:46 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:48:48.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:47 smithi122 ceph-mon[93869]: pgmap v17491: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 267 KiB/s wr, 9 op/s 2023-03-09T23:48:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:48:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:48:48.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:47 smithi181 ceph-mon[94189]: pgmap v17491: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 267 KiB/s wr, 9 op/s 2023-03-09T23:48:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:48:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:48:48.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:47 smithi038 ceph-mon[92422]: pgmap v17491: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 267 KiB/s wr, 9 op/s 2023-03-09T23:48:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:48:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:48:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32300.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:48:50.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:48:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:48:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:49 smithi122 ceph-mon[93869]: pgmap v17492: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 440 KiB/s rd, 311 KiB/s wr, 8 op/s 2023-03-09T23:48:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:48:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:48:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:49 smithi181 ceph-mon[94189]: pgmap v17492: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 440 KiB/s rd, 311 KiB/s wr, 8 op/s 2023-03-09T23:48:50.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:48:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:48:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:49 smithi038 ceph-mon[92422]: pgmap v17492: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 440 KiB/s rd, 311 KiB/s wr, 8 op/s 2023-03-09T23:48:51.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31841.436467 secs 2023-03-09T23:48:51.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31841.436467 secs 2023-03-09T23:48:51.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31841.436467 secs 2023-03-09T23:48:52.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:51 smithi122 ceph-mon[93869]: pgmap v17493: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 663 KiB/s rd, 531 KiB/s wr, 11 op/s 2023-03-09T23:48:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:51 smithi181 ceph-mon[94189]: pgmap v17493: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 663 KiB/s rd, 531 KiB/s wr, 11 op/s 2023-03-09T23:48:52.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:51 smithi038 ceph-mon[92422]: pgmap v17493: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 663 KiB/s rd, 531 KiB/s wr, 11 op/s 2023-03-09T23:48:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32305.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:48:54.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:53 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 31843.853752 secs 2023-03-09T23:48:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:53 smithi122 ceph-mon[93869]: pgmap v17494: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 661 KiB/s rd, 531 KiB/s wr, 9 op/s 2023-03-09T23:48:54.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:53 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 31843.853752 secs 2023-03-09T23:48:54.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:53 smithi181 ceph-mon[94189]: pgmap v17494: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 661 KiB/s rd, 531 KiB/s wr, 9 op/s 2023-03-09T23:48:54.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:53 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 31843.853752 secs 2023-03-09T23:48:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:53 smithi038 ceph-mon[92422]: pgmap v17494: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 661 KiB/s rd, 531 KiB/s wr, 9 op/s 2023-03-09T23:48:55.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:54 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:48:55.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:54 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:48:55.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:54 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:48:55.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:54 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:48:55.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:54 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:48:55.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:54 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:48:55.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:54 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:48:55.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:54 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:48:55.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:54 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:48:56.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:48:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:48:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:55 smithi122 ceph-mon[93869]: pgmap v17495: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 757 KiB/s rd, 531 KiB/s wr, 10 op/s 2023-03-09T23:48:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31846.436570 secs 2023-03-09T23:48:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:48:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:48:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:55 smithi181 ceph-mon[94189]: pgmap v17495: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 757 KiB/s rd, 531 KiB/s wr, 10 op/s 2023-03-09T23:48:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31846.436570 secs 2023-03-09T23:48:56.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:48:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:48:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:55 smithi038 ceph-mon[92422]: pgmap v17495: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 757 KiB/s rd, 531 KiB/s wr, 10 op/s 2023-03-09T23:48:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31846.436570 secs 2023-03-09T23:48:57.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:48:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:48:57.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:48:57.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:48:57.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:48:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:48:58.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:57 smithi122 ceph-mon[93869]: pgmap v17496: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 610 KiB/s wr, 14 op/s 2023-03-09T23:48:58.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:57 smithi181 ceph-mon[94189]: pgmap v17496: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 610 KiB/s wr, 14 op/s 2023-03-09T23:48:58.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:57 smithi038 ceph-mon[92422]: pgmap v17496: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 610 KiB/s wr, 14 op/s 2023-03-09T23:48:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32310.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:48:59.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:58 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 31848.853861 secs 2023-03-09T23:48:59.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:58 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 31848.853861 secs 2023-03-09T23:48:59.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:58 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 31848.853861 secs 2023-03-09T23:49:00.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:49:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:49:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:48:59 smithi122 ceph-mon[93869]: pgmap v17497: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 585 KiB/s wr, 12 op/s 2023-03-09T23:49:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:49:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:49:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:48:59 smithi181 ceph-mon[94189]: pgmap v17497: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 585 KiB/s wr, 12 op/s 2023-03-09T23:49:00.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:49:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:49:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:48:59 smithi038 ceph-mon[92422]: pgmap v17497: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 585 KiB/s wr, 12 op/s 2023-03-09T23:49:01.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31851.436655 secs 2023-03-09T23:49:01.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31851.436655 secs 2023-03-09T23:49:01.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31851.436655 secs 2023-03-09T23:49:02.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:01 smithi122 ceph-mon[93869]: pgmap v17498: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 806 KiB/s rd, 707 KiB/s wr, 15 op/s 2023-03-09T23:49:02.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:01 smithi181 ceph-mon[94189]: pgmap v17498: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 806 KiB/s rd, 707 KiB/s wr, 15 op/s 2023-03-09T23:49:02.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:01 smithi038 ceph-mon[92422]: pgmap v17498: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 806 KiB/s rd, 707 KiB/s wr, 15 op/s 2023-03-09T23:49:03.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32315.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:49:04.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31853.853978 secs 2023-03-09T23:49:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:03 smithi122 ceph-mon[93869]: pgmap v17499: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 583 KiB/s rd, 487 KiB/s wr, 12 op/s 2023-03-09T23:49:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31853.853978 secs 2023-03-09T23:49:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:03 smithi181 ceph-mon[94189]: pgmap v17499: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 583 KiB/s rd, 487 KiB/s wr, 12 op/s 2023-03-09T23:49:04.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31853.853978 secs 2023-03-09T23:49:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:03 smithi038 ceph-mon[92422]: pgmap v17499: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 583 KiB/s rd, 487 KiB/s wr, 12 op/s 2023-03-09T23:49:06.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:05 smithi122 ceph-mon[93869]: osdmap e1309: 12 total, 12 up, 12 in 2023-03-09T23:49:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:05 smithi122 ceph-mon[93869]: pgmap v17501: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 715 KiB/s wr, 28 op/s 2023-03-09T23:49:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31856.436736 secs 2023-03-09T23:49:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:05 smithi181 ceph-mon[94189]: osdmap e1309: 12 total, 12 up, 12 in 2023-03-09T23:49:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:05 smithi181 ceph-mon[94189]: pgmap v17501: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 715 KiB/s wr, 28 op/s 2023-03-09T23:49:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31856.436736 secs 2023-03-09T23:49:06.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:05 smithi038 ceph-mon[92422]: osdmap e1309: 12 total, 12 up, 12 in 2023-03-09T23:49:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:05 smithi038 ceph-mon[92422]: pgmap v17501: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 715 KiB/s wr, 28 op/s 2023-03-09T23:49:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31856.436736 secs 2023-03-09T23:49:07.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:06 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:49:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:06 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:49:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:49:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:06 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:49:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:06 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:49:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:06 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:49:07.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:06 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:49:07.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:06 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:49:07.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:49:07.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:06 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:49:07.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:06 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:49:07.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:06 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:49:07.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:06 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:49:07.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:06 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:49:07.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:49:07.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:06 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:49:07.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:06 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:49:07.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:06 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:49:08.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:07 smithi122 ceph-mon[93869]: pgmap v17502: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 416 KiB/s rd, 458 KiB/s wr, 29 op/s 2023-03-09T23:49:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:49:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:49:08.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:07 smithi181 ceph-mon[94189]: pgmap v17502: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 416 KiB/s rd, 458 KiB/s wr, 29 op/s 2023-03-09T23:49:08.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:49:08.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:49:08.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:07 smithi038 ceph-mon[92422]: pgmap v17502: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 416 KiB/s rd, 458 KiB/s wr, 29 op/s 2023-03-09T23:49:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:49:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:49:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32320.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:49:09.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31858.854090 secs 2023-03-09T23:49:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:49:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:49:09.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31858.854090 secs 2023-03-09T23:49:09.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:49:09.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:49:09.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31858.854090 secs 2023-03-09T23:49:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:49:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:49:10.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:09 smithi122 ceph-mon[93869]: pgmap v17503: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 416 KiB/s rd, 458 KiB/s wr, 29 op/s 2023-03-09T23:49:10.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:09 smithi181 ceph-mon[94189]: pgmap v17503: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 416 KiB/s rd, 458 KiB/s wr, 29 op/s 2023-03-09T23:49:10.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:09 smithi038 ceph-mon[92422]: pgmap v17503: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 416 KiB/s rd, 458 KiB/s wr, 29 op/s 2023-03-09T23:49:11.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31861.436795 secs 2023-03-09T23:49:11.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31861.436795 secs 2023-03-09T23:49:11.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31861.436795 secs 2023-03-09T23:49:12.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:11 smithi122 ceph-mon[93869]: pgmap v17504: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 259 KiB/s wr, 27 op/s 2023-03-09T23:49:12.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:11 smithi181 ceph-mon[94189]: pgmap v17504: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 259 KiB/s wr, 27 op/s 2023-03-09T23:49:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:11 smithi038 ceph-mon[92422]: pgmap v17504: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 259 KiB/s wr, 27 op/s 2023-03-09T23:49:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32325.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:49:14.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31863.854209 secs 2023-03-09T23:49:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:13 smithi122 ceph-mon[93869]: osdmap e1310: 12 total, 12 up, 12 in 2023-03-09T23:49:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:13 smithi122 ceph-mon[93869]: pgmap v17506: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 158 KiB/s wr, 15 op/s 2023-03-09T23:49:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31863.854209 secs 2023-03-09T23:49:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:13 smithi181 ceph-mon[94189]: osdmap e1310: 12 total, 12 up, 12 in 2023-03-09T23:49:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:13 smithi181 ceph-mon[94189]: pgmap v17506: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 158 KiB/s wr, 15 op/s 2023-03-09T23:49:14.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31863.854209 secs 2023-03-09T23:49:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:13 smithi038 ceph-mon[92422]: osdmap e1310: 12 total, 12 up, 12 in 2023-03-09T23:49:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:13 smithi038 ceph-mon[92422]: pgmap v17506: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 158 KiB/s wr, 15 op/s 2023-03-09T23:49:16.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:15 smithi122 ceph-mon[93869]: pgmap v17507: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 294 KiB/s wr, 13 op/s 2023-03-09T23:49:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31866.436856 secs 2023-03-09T23:49:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:15 smithi181 ceph-mon[94189]: pgmap v17507: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 294 KiB/s wr, 13 op/s 2023-03-09T23:49:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31866.436856 secs 2023-03-09T23:49:16.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:15 smithi038 ceph-mon[92422]: pgmap v17507: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 294 KiB/s wr, 13 op/s 2023-03-09T23:49:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31866.436856 secs 2023-03-09T23:49:18.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:17 smithi122 ceph-mon[93869]: pgmap v17508: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 167 KiB/s wr, 8 op/s 2023-03-09T23:49:18.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:17 smithi181 ceph-mon[94189]: pgmap v17508: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 167 KiB/s wr, 8 op/s 2023-03-09T23:49:18.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:17 smithi038 ceph-mon[92422]: pgmap v17508: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 167 KiB/s wr, 8 op/s 2023-03-09T23:49:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32330.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:49:19.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31868.854290 secs 2023-03-09T23:49:19.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31868.854290 secs 2023-03-09T23:49:19.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31868.854290 secs 2023-03-09T23:49:20.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:19 smithi122 ceph-mon[93869]: pgmap v17509: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 166 KiB/s wr, 8 op/s 2023-03-09T23:49:20.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:19 smithi181 ceph-mon[94189]: pgmap v17509: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 166 KiB/s wr, 8 op/s 2023-03-09T23:49:20.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:19 smithi038 ceph-mon[92422]: pgmap v17509: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 166 KiB/s wr, 8 op/s 2023-03-09T23:49:21.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31871.436931 secs 2023-03-09T23:49:21.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31871.436931 secs 2023-03-09T23:49:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31871.436931 secs 2023-03-09T23:49:22.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:21 smithi122 ceph-mon[93869]: pgmap v17510: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 166 KiB/s wr, 7 op/s 2023-03-09T23:49:22.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:21 smithi181 ceph-mon[94189]: pgmap v17510: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 166 KiB/s wr, 7 op/s 2023-03-09T23:49:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:21 smithi038 ceph-mon[92422]: pgmap v17510: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 166 KiB/s wr, 7 op/s 2023-03-09T23:49:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32335.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:49:24.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31873.854418 secs 2023-03-09T23:49:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:23 smithi122 ceph-mon[93869]: pgmap v17511: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 164 KiB/s wr, 7 op/s 2023-03-09T23:49:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31873.854418 secs 2023-03-09T23:49:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:23 smithi181 ceph-mon[94189]: pgmap v17511: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 164 KiB/s wr, 7 op/s 2023-03-09T23:49:24.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31873.854418 secs 2023-03-09T23:49:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:23 smithi038 ceph-mon[92422]: pgmap v17511: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 167 KiB/s rd, 164 KiB/s wr, 7 op/s 2023-03-09T23:49:26.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:25 smithi122 ceph-mon[93869]: pgmap v17512: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 138 KiB/s wr, 6 op/s 2023-03-09T23:49:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31876.436984 secs 2023-03-09T23:49:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:25 smithi181 ceph-mon[94189]: pgmap v17512: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 138 KiB/s wr, 6 op/s 2023-03-09T23:49:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31876.436984 secs 2023-03-09T23:49:26.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:25 smithi038 ceph-mon[92422]: pgmap v17512: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 138 KiB/s wr, 6 op/s 2023-03-09T23:49:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31876.436984 secs 2023-03-09T23:49:28.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:27 smithi122 ceph-mon[93869]: pgmap v17513: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 8 op/s 2023-03-09T23:49:28.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:27 smithi181 ceph-mon[94189]: pgmap v17513: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 8 op/s 2023-03-09T23:49:28.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:27 smithi038 ceph-mon[92422]: pgmap v17513: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 8 op/s 2023-03-09T23:49:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32340.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:49:29.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31878.854500 secs 2023-03-09T23:49:29.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31878.854500 secs 2023-03-09T23:49:29.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31878.854500 secs 2023-03-09T23:49:30.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:29 smithi122 ceph-mon[93869]: pgmap v17514: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T23:49:30.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:29 smithi181 ceph-mon[94189]: pgmap v17514: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T23:49:30.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:29 smithi038 ceph-mon[92422]: pgmap v17514: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T23:49:31.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31881.437045 secs 2023-03-09T23:49:31.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31881.437045 secs 2023-03-09T23:49:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31881.437045 secs 2023-03-09T23:49:32.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:31 smithi122 ceph-mon[93869]: pgmap v17515: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T23:49:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:31 smithi181 ceph-mon[94189]: pgmap v17515: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T23:49:32.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:31 smithi038 ceph-mon[92422]: pgmap v17515: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T23:49:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32345.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:49:34.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31883.854593 secs 2023-03-09T23:49:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:33 smithi122 ceph-mon[93869]: pgmap v17516: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T23:49:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31883.854593 secs 2023-03-09T23:49:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:33 smithi181 ceph-mon[94189]: pgmap v17516: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T23:49:34.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31883.854593 secs 2023-03-09T23:49:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:33 smithi038 ceph-mon[92422]: pgmap v17516: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T23:49:36.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:35 smithi122 ceph-mon[93869]: pgmap v17517: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 467 KiB/s rd, 116 KiB/s wr, 6 op/s 2023-03-09T23:49:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31886.437106 secs 2023-03-09T23:49:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:35 smithi181 ceph-mon[94189]: pgmap v17517: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 467 KiB/s rd, 116 KiB/s wr, 6 op/s 2023-03-09T23:49:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31886.437106 secs 2023-03-09T23:49:36.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:35 smithi038 ceph-mon[92422]: pgmap v17517: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 467 KiB/s rd, 116 KiB/s wr, 6 op/s 2023-03-09T23:49:36.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31886.437106 secs 2023-03-09T23:49:38.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:37 smithi122 ceph-mon[93869]: pgmap v17518: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 470 KiB/s rd, 118 KiB/s wr, 9 op/s 2023-03-09T23:49:38.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:37 smithi181 ceph-mon[94189]: pgmap v17518: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 470 KiB/s rd, 118 KiB/s wr, 9 op/s 2023-03-09T23:49:38.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:37 smithi038 ceph-mon[92422]: pgmap v17518: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 470 KiB/s rd, 118 KiB/s wr, 9 op/s 2023-03-09T23:49:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32350.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:49:39.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31888.854728 secs 2023-03-09T23:49:39.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31888.854728 secs 2023-03-09T23:49:39.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31888.854728 secs 2023-03-09T23:49:40.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:39 smithi122 ceph-mon[93869]: pgmap v17519: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 585 KiB/s rd, 118 KiB/s wr, 7 op/s 2023-03-09T23:49:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:39 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:49:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:39 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:49:40.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:39 smithi181 ceph-mon[94189]: pgmap v17519: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 585 KiB/s rd, 118 KiB/s wr, 7 op/s 2023-03-09T23:49:40.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:39 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:49:40.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:39 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:49:40.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:39 smithi038 ceph-mon[92422]: pgmap v17519: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 585 KiB/s rd, 118 KiB/s wr, 7 op/s 2023-03-09T23:49:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:39 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:49:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:39 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:49:41.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:49:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:49:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31891.437185 secs 2023-03-09T23:49:41.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:49:41.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:49:41.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31891.437185 secs 2023-03-09T23:49:41.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:49:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:49:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31891.437185 secs 2023-03-09T23:49:42.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:41 smithi122 ceph-mon[93869]: pgmap v17520: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 708 KiB/s rd, 240 KiB/s wr, 10 op/s 2023-03-09T23:49:42.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:41 smithi181 ceph-mon[94189]: pgmap v17520: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 708 KiB/s rd, 240 KiB/s wr, 10 op/s 2023-03-09T23:49:42.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:41 smithi038 ceph-mon[92422]: pgmap v17520: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 708 KiB/s rd, 240 KiB/s wr, 10 op/s 2023-03-09T23:49:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32355.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:49:44.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31893.854870 secs 2023-03-09T23:49:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:43 smithi122 ceph-mon[93869]: pgmap v17521: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 706 KiB/s rd, 240 KiB/s wr, 7 op/s 2023-03-09T23:49:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31893.854870 secs 2023-03-09T23:49:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:43 smithi181 ceph-mon[94189]: pgmap v17521: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 706 KiB/s rd, 240 KiB/s wr, 7 op/s 2023-03-09T23:49:44.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31893.854870 secs 2023-03-09T23:49:44.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:43 smithi038 ceph-mon[92422]: pgmap v17521: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 706 KiB/s rd, 240 KiB/s wr, 7 op/s 2023-03-09T23:49:46.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:45 smithi122 ceph-mon[93869]: pgmap v17522: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 707 KiB/s rd, 240 KiB/s wr, 8 op/s 2023-03-09T23:49:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31896.437245 secs 2023-03-09T23:49:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:45 smithi181 ceph-mon[94189]: pgmap v17522: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 707 KiB/s rd, 240 KiB/s wr, 8 op/s 2023-03-09T23:49:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31896.437245 secs 2023-03-09T23:49:46.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:45 smithi038 ceph-mon[92422]: pgmap v17522: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 707 KiB/s rd, 240 KiB/s wr, 8 op/s 2023-03-09T23:49:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31896.437245 secs 2023-03-09T23:49:48.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:47 smithi122 ceph-mon[93869]: pgmap v17523: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 560 KiB/s rd, 342 KiB/s wr, 10 op/s 2023-03-09T23:49:48.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:47 smithi181 ceph-mon[94189]: pgmap v17523: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 560 KiB/s rd, 342 KiB/s wr, 10 op/s 2023-03-09T23:49:48.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:47 smithi038 ceph-mon[92422]: pgmap v17523: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 560 KiB/s rd, 342 KiB/s wr, 10 op/s 2023-03-09T23:49:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32360.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:49:49.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31898.855001 secs 2023-03-09T23:49:49.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31898.855001 secs 2023-03-09T23:49:49.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31898.855001 secs 2023-03-09T23:49:50.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:49 smithi122 ceph-mon[93869]: pgmap v17524: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 557 KiB/s rd, 340 KiB/s wr, 7 op/s 2023-03-09T23:49:50.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:49 smithi181 ceph-mon[94189]: pgmap v17524: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 557 KiB/s rd, 340 KiB/s wr, 7 op/s 2023-03-09T23:49:50.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:49 smithi038 ceph-mon[92422]: pgmap v17524: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 557 KiB/s rd, 340 KiB/s wr, 7 op/s 2023-03-09T23:49:51.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31901.437307 secs 2023-03-09T23:49:51.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31901.437307 secs 2023-03-09T23:49:51.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31901.437307 secs 2023-03-09T23:49:52.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:51 smithi122 ceph-mon[93869]: pgmap v17525: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 625 KiB/s wr, 10 op/s 2023-03-09T23:49:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:51 smithi181 ceph-mon[94189]: pgmap v17525: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 625 KiB/s wr, 10 op/s 2023-03-09T23:49:52.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:51 smithi038 ceph-mon[92422]: pgmap v17525: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 625 KiB/s wr, 10 op/s 2023-03-09T23:49:53.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:49:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:49:53.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:49:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:49:53.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:49:53.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:49:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32365.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:49:54.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31903.855140 secs 2023-03-09T23:49:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:53 smithi038 ceph-mon[92422]: pgmap v17526: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 997 KiB/s rd, 504 KiB/s wr, 8 op/s 2023-03-09T23:49:54.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31903.855140 secs 2023-03-09T23:49:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:53 smithi122 ceph-mon[93869]: pgmap v17526: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 997 KiB/s rd, 504 KiB/s wr, 8 op/s 2023-03-09T23:49:54.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31903.855140 secs 2023-03-09T23:49:54.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:53 smithi181 ceph-mon[94189]: pgmap v17526: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 997 KiB/s rd, 504 KiB/s wr, 8 op/s 2023-03-09T23:49:56.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:55 smithi038 ceph-mon[92422]: pgmap v17527: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1008 KiB/s rd, 504 KiB/s wr, 8 op/s 2023-03-09T23:49:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:49:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:49:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31906.437368 secs 2023-03-09T23:49:56.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:55 smithi122 ceph-mon[93869]: pgmap v17527: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1008 KiB/s rd, 504 KiB/s wr, 8 op/s 2023-03-09T23:49:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:49:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:49:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31906.437368 secs 2023-03-09T23:49:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:55 smithi181 ceph-mon[94189]: pgmap v17527: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1008 KiB/s rd, 504 KiB/s wr, 8 op/s 2023-03-09T23:49:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:49:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:49:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31906.437368 secs 2023-03-09T23:49:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:57 smithi038 ceph-mon[92422]: pgmap v17528: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 552 KiB/s wr, 12 op/s 2023-03-09T23:49:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:49:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31908.855201 secs 2023-03-09T23:49:58.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:57 smithi122 ceph-mon[93869]: pgmap v17528: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 552 KiB/s wr, 12 op/s 2023-03-09T23:49:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:49:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31908.855201 secs 2023-03-09T23:49:58.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:57 smithi181 ceph-mon[94189]: pgmap v17528: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 552 KiB/s wr, 12 op/s 2023-03-09T23:49:58.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:49:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31908.855201 secs 2023-03-09T23:49:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32370.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:50:00.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:00 smithi122 ceph-mon[93869]: pgmap v17529: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 742 KiB/s rd, 334 KiB/s wr, 10 op/s 2023-03-09T23:50:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:50:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:50:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:00 smithi122 ceph-mon[93869]: overall HEALTH_WARN 2 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-09T23:50:00.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:00 smithi181 ceph-mon[94189]: pgmap v17529: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 742 KiB/s rd, 334 KiB/s wr, 10 op/s 2023-03-09T23:50:00.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:50:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:50:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:00 smithi181 ceph-mon[94189]: overall HEALTH_WARN 2 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-09T23:50:00.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:00 smithi038 ceph-mon[92422]: pgmap v17529: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 742 KiB/s rd, 334 KiB/s wr, 10 op/s 2023-03-09T23:50:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:50:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:50:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:00 smithi038 ceph-mon[92422]: overall HEALTH_WARN 2 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-09T23:50:01.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31911.437430 secs 2023-03-09T23:50:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:50:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:50:01.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31911.437430 secs 2023-03-09T23:50:01.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:50:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:50:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31911.437430 secs 2023-03-09T23:50:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:50:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:50:02.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:02 smithi122 ceph-mon[93869]: pgmap v17530: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 833 KiB/s rd, 423 KiB/s wr, 13 op/s 2023-03-09T23:50:02.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:02 smithi181 ceph-mon[94189]: pgmap v17530: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 833 KiB/s rd, 423 KiB/s wr, 13 op/s 2023-03-09T23:50:02.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:02 smithi038 ceph-mon[92422]: pgmap v17530: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 833 KiB/s rd, 423 KiB/s wr, 13 op/s 2023-03-09T23:50:03.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32375.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:50:04.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31913.855296 secs 2023-03-09T23:50:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:03 smithi122 ceph-mon[93869]: pgmap v17531: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 480 KiB/s wr, 11 op/s 2023-03-09T23:50:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31913.855296 secs 2023-03-09T23:50:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:03 smithi181 ceph-mon[94189]: pgmap v17531: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 480 KiB/s wr, 11 op/s 2023-03-09T23:50:04.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31913.855296 secs 2023-03-09T23:50:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:03 smithi038 ceph-mon[92422]: pgmap v17531: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 480 KiB/s wr, 11 op/s 2023-03-09T23:50:06.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:05 smithi122 ceph-mon[93869]: pgmap v17532: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 492 KiB/s wr, 22 op/s 2023-03-09T23:50:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31916.437491 secs 2023-03-09T23:50:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:05 smithi181 ceph-mon[94189]: pgmap v17532: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 492 KiB/s wr, 22 op/s 2023-03-09T23:50:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31916.437491 secs 2023-03-09T23:50:06.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:05 smithi038 ceph-mon[92422]: pgmap v17532: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 492 KiB/s wr, 22 op/s 2023-03-09T23:50:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31916.437491 secs 2023-03-09T23:50:07.201 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:06 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:50:07.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:06 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:50:07.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:06 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:50:08.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:07 smithi122 ceph-mon[93869]: pgmap v17533: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 520 KiB/s rd, 542 KiB/s wr, 27 op/s 2023-03-09T23:50:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:07 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:50:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:07 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:50:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:07 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:50:08.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:07 smithi181 ceph-mon[94189]: pgmap v17533: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 520 KiB/s rd, 542 KiB/s wr, 27 op/s 2023-03-09T23:50:08.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:07 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:50:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:07 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:50:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:07 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:50:08.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:07 smithi038 ceph-mon[92422]: pgmap v17533: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 520 KiB/s rd, 542 KiB/s wr, 27 op/s 2023-03-09T23:50:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:07 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:50:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:07 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:50:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:07 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:50:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32380.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:50:09.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31918.855364 secs 2023-03-09T23:50:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:50:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:50:09.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31918.855364 secs 2023-03-09T23:50:09.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:50:09.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:50:09.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31918.855364 secs 2023-03-09T23:50:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:50:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:50:10.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:09 smithi122 ceph-mon[93869]: pgmap v17534: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 846 KiB/s wr, 27 op/s 2023-03-09T23:50:10.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:09 smithi181 ceph-mon[94189]: pgmap v17534: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 846 KiB/s wr, 27 op/s 2023-03-09T23:50:10.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:09 smithi038 ceph-mon[92422]: pgmap v17534: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 846 KiB/s wr, 27 op/s 2023-03-09T23:50:11.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:10 smithi181 ceph-mon[94189]: osdmap e1311: 12 total, 12 up, 12 in 2023-03-09T23:50:11.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31921.437600 secs 2023-03-09T23:50:11.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:10 smithi038 ceph-mon[92422]: osdmap e1311: 12 total, 12 up, 12 in 2023-03-09T23:50:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31921.437600 secs 2023-03-09T23:50:11.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:10 smithi122 ceph-mon[93869]: osdmap e1311: 12 total, 12 up, 12 in 2023-03-09T23:50:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31921.437600 secs 2023-03-09T23:50:12.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:11 smithi181 ceph-mon[94189]: pgmap v17536: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 908 KiB/s wr, 30 op/s 2023-03-09T23:50:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:11 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:50:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:11 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:50:12.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:11 smithi038 ceph-mon[92422]: pgmap v17536: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 908 KiB/s wr, 30 op/s 2023-03-09T23:50:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:11 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:50:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:11 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:50:12.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:11 smithi122 ceph-mon[93869]: pgmap v17536: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 908 KiB/s wr, 30 op/s 2023-03-09T23:50:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:11 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:50:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:11 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:50:13.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:50:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:50:13.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:50:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:50:13.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:50:13.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:50:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32385.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:50:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31923.855483 secs 2023-03-09T23:50:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:13 smithi038 ceph-mon[92422]: pgmap v17537: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 716 KiB/s wr, 29 op/s 2023-03-09T23:50:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31923.855483 secs 2023-03-09T23:50:14.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:13 smithi122 ceph-mon[93869]: pgmap v17537: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 716 KiB/s wr, 29 op/s 2023-03-09T23:50:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31923.855483 secs 2023-03-09T23:50:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:13 smithi181 ceph-mon[94189]: pgmap v17537: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 716 KiB/s wr, 29 op/s 2023-03-09T23:50:16.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:16 smithi122 ceph-mon[93869]: pgmap v17538: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 998 KiB/s wr, 17 op/s 2023-03-09T23:50:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31926.437687 secs 2023-03-09T23:50:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:50:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:50:16.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:16 smithi181 ceph-mon[94189]: pgmap v17538: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 998 KiB/s wr, 17 op/s 2023-03-09T23:50:16.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31926.437687 secs 2023-03-09T23:50:16.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:50:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:50:16.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:16 smithi038 ceph-mon[92422]: pgmap v17538: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 998 KiB/s wr, 17 op/s 2023-03-09T23:50:16.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31926.437687 secs 2023-03-09T23:50:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:50:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:50:17.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:50:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:50:17.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:50:17.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:50:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:50:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:50:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:18 smithi122 ceph-mon[93869]: pgmap v17539: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 969 KiB/s wr, 14 op/s 2023-03-09T23:50:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31928.855620 secs 2023-03-09T23:50:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:18 smithi122 ceph-mon[93869]: osdmap e1312: 12 total, 12 up, 12 in 2023-03-09T23:50:18.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:18 smithi181 ceph-mon[94189]: pgmap v17539: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 969 KiB/s wr, 14 op/s 2023-03-09T23:50:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31928.855620 secs 2023-03-09T23:50:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:18 smithi181 ceph-mon[94189]: osdmap e1312: 12 total, 12 up, 12 in 2023-03-09T23:50:18.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:18 smithi038 ceph-mon[92422]: pgmap v17539: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 969 KiB/s wr, 14 op/s 2023-03-09T23:50:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31928.855620 secs 2023-03-09T23:50:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:18 smithi038 ceph-mon[92422]: osdmap e1312: 12 total, 12 up, 12 in 2023-03-09T23:50:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32390.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:50:20.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:20 smithi122 ceph-mon[93869]: pgmap v17541: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 828 KiB/s rd, 826 KiB/s wr, 14 op/s 2023-03-09T23:50:20.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:20 smithi181 ceph-mon[94189]: pgmap v17541: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 828 KiB/s rd, 826 KiB/s wr, 14 op/s 2023-03-09T23:50:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:20 smithi038 ceph-mon[92422]: pgmap v17541: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 828 KiB/s rd, 826 KiB/s wr, 14 op/s 2023-03-09T23:50:21.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31931.437767 secs 2023-03-09T23:50:21.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31931.437767 secs 2023-03-09T23:50:21.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31931.437767 secs 2023-03-09T23:50:22.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:22 smithi122 ceph-mon[93869]: pgmap v17542: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 740 KiB/s rd, 738 KiB/s wr, 12 op/s 2023-03-09T23:50:22.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:22 smithi181 ceph-mon[94189]: pgmap v17542: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 740 KiB/s rd, 738 KiB/s wr, 12 op/s 2023-03-09T23:50:22.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:22 smithi038 ceph-mon[92422]: pgmap v17542: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 740 KiB/s rd, 738 KiB/s wr, 12 op/s 2023-03-09T23:50:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32395.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:50:24.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31933.855739 secs 2023-03-09T23:50:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:23 smithi122 ceph-mon[93869]: pgmap v17543: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 522 KiB/s rd, 520 KiB/s wr, 11 op/s 2023-03-09T23:50:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31933.855739 secs 2023-03-09T23:50:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:23 smithi181 ceph-mon[94189]: pgmap v17543: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 522 KiB/s rd, 520 KiB/s wr, 11 op/s 2023-03-09T23:50:24.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31933.855739 secs 2023-03-09T23:50:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:23 smithi038 ceph-mon[92422]: pgmap v17543: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 522 KiB/s rd, 520 KiB/s wr, 11 op/s 2023-03-09T23:50:26.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:25 smithi122 ceph-mon[93869]: pgmap v17544: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 485 KiB/s rd, 483 KiB/s wr, 11 op/s 2023-03-09T23:50:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31936.437828 secs 2023-03-09T23:50:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:25 smithi181 ceph-mon[94189]: pgmap v17544: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 485 KiB/s rd, 483 KiB/s wr, 11 op/s 2023-03-09T23:50:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31936.437828 secs 2023-03-09T23:50:26.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:25 smithi038 ceph-mon[92422]: pgmap v17544: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 485 KiB/s rd, 483 KiB/s wr, 11 op/s 2023-03-09T23:50:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31936.437828 secs 2023-03-09T23:50:28.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:27 smithi122 ceph-mon[93869]: pgmap v17545: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 453 KiB/s rd, 451 KiB/s wr, 10 op/s 2023-03-09T23:50:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:27 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:50:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:27 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:50:28.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:27 smithi181 ceph-mon[94189]: pgmap v17545: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 453 KiB/s rd, 451 KiB/s wr, 10 op/s 2023-03-09T23:50:28.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:27 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:50:28.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:27 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:50:28.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:27 smithi038 ceph-mon[92422]: pgmap v17545: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 453 KiB/s rd, 451 KiB/s wr, 10 op/s 2023-03-09T23:50:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:27 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:50:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:27 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:50:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32400.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:50:29.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31938.855869 secs 2023-03-09T23:50:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:28 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:50:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:28 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:50:29.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31938.855869 secs 2023-03-09T23:50:29.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:28 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:50:29.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:28 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:50:29.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31938.855869 secs 2023-03-09T23:50:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:28 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:50:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:28 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:50:30.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:30 smithi122 ceph-mon[93869]: pgmap v17546: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 233 KiB/s wr, 9 op/s 2023-03-09T23:50:30.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:30 smithi181 ceph-mon[94189]: pgmap v17546: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 233 KiB/s wr, 9 op/s 2023-03-09T23:50:30.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:30 smithi038 ceph-mon[92422]: pgmap v17546: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 233 KiB/s wr, 9 op/s 2023-03-09T23:50:31.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31941.437902 secs 2023-03-09T23:50:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31941.437902 secs 2023-03-09T23:50:31.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31941.437902 secs 2023-03-09T23:50:32.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:32 smithi122 ceph-mon[93869]: pgmap v17547: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 217 KiB/s wr, 9 op/s 2023-03-09T23:50:32.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:32 smithi181 ceph-mon[94189]: pgmap v17547: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 217 KiB/s wr, 9 op/s 2023-03-09T23:50:32.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:32 smithi038 ceph-mon[92422]: pgmap v17547: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 217 KiB/s wr, 9 op/s 2023-03-09T23:50:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32405.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:50:34.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31943.856000 secs 2023-03-09T23:50:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:33 smithi122 ceph-mon[93869]: pgmap v17548: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 216 KiB/s wr, 7 op/s 2023-03-09T23:50:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31943.856000 secs 2023-03-09T23:50:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:33 smithi181 ceph-mon[94189]: pgmap v17548: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 216 KiB/s wr, 7 op/s 2023-03-09T23:50:34.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31943.856000 secs 2023-03-09T23:50:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:33 smithi038 ceph-mon[92422]: pgmap v17548: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 216 KiB/s wr, 7 op/s 2023-03-09T23:50:36.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:35 smithi122 ceph-mon[93869]: pgmap v17549: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 293 KiB/s wr, 7 op/s 2023-03-09T23:50:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31946.437962 secs 2023-03-09T23:50:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:35 smithi181 ceph-mon[94189]: pgmap v17549: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 293 KiB/s wr, 7 op/s 2023-03-09T23:50:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31946.437962 secs 2023-03-09T23:50:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:35 smithi038 ceph-mon[92422]: pgmap v17549: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 293 KiB/s wr, 7 op/s 2023-03-09T23:50:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31946.437962 secs 2023-03-09T23:50:38.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:37 smithi122 ceph-mon[93869]: pgmap v17550: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 78 KiB/s wr, 8 op/s 2023-03-09T23:50:38.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:37 smithi181 ceph-mon[94189]: pgmap v17550: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 78 KiB/s wr, 8 op/s 2023-03-09T23:50:38.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:37 smithi038 ceph-mon[92422]: pgmap v17550: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 78 KiB/s wr, 8 op/s 2023-03-09T23:50:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32410.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:50:39.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31948.856140 secs 2023-03-09T23:50:39.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31948.856140 secs 2023-03-09T23:50:39.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31948.856140 secs 2023-03-09T23:50:40.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:39 smithi122 ceph-mon[93869]: pgmap v17551: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 78 KiB/s wr, 7 op/s 2023-03-09T23:50:40.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:39 smithi181 ceph-mon[94189]: pgmap v17551: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 78 KiB/s wr, 7 op/s 2023-03-09T23:50:40.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:39 smithi038 ceph-mon[92422]: pgmap v17551: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 78 KiB/s wr, 7 op/s 2023-03-09T23:50:41.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31951.438023 secs 2023-03-09T23:50:41.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31951.438023 secs 2023-03-09T23:50:41.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31951.438023 secs 2023-03-09T23:50:42.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:41 smithi038 ceph-mon[92422]: pgmap v17552: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 78 KiB/s wr, 9 op/s 2023-03-09T23:50:42.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:41 smithi122 ceph-mon[93869]: pgmap v17552: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 78 KiB/s wr, 9 op/s 2023-03-09T23:50:42.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:41 smithi181 ceph-mon[94189]: pgmap v17552: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 78 KiB/s wr, 9 op/s 2023-03-09T23:50:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32415.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:50:44.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:43 smithi122 ceph-mon[93869]: pgmap v17553: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 77 KiB/s wr, 6 op/s 2023-03-09T23:50:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:43 smithi181 ceph-mon[94189]: pgmap v17553: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 77 KiB/s wr, 6 op/s 2023-03-09T23:50:44.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:43 smithi038 ceph-mon[92422]: pgmap v17553: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 77 KiB/s wr, 6 op/s 2023-03-09T23:50:46.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:45 smithi122 ceph-mon[93869]: pgmap v17554: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 357 KiB/s rd, 274 KiB/s wr, 7 op/s 2023-03-09T23:50:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31956.438099 secs 2023-03-09T23:50:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:45 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:50:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:45 smithi181 ceph-mon[94189]: pgmap v17554: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 357 KiB/s rd, 274 KiB/s wr, 7 op/s 2023-03-09T23:50:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31956.438099 secs 2023-03-09T23:50:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:45 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:50:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:45 smithi038 ceph-mon[92422]: pgmap v17554: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 357 KiB/s rd, 274 KiB/s wr, 7 op/s 2023-03-09T23:50:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31956.438099 secs 2023-03-09T23:50:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:45 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:50:47.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:46 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:50:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:46 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:50:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:46 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:50:47.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:46 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:50:47.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:46 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:50:47.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:46 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:50:47.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:46 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:50:47.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:46 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:50:47.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:46 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:50:48.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:47 smithi038 ceph-mon[92422]: pgmap v17555: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 197 KiB/s wr, 8 op/s 2023-03-09T23:50:48.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:47 smithi122 ceph-mon[93869]: pgmap v17555: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 197 KiB/s wr, 8 op/s 2023-03-09T23:50:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:47 smithi181 ceph-mon[94189]: pgmap v17555: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 197 KiB/s wr, 8 op/s 2023-03-09T23:50:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32420.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:50:50.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:49 smithi038 ceph-mon[92422]: pgmap v17556: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 197 KiB/s wr, 7 op/s 2023-03-09T23:50:50.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:49 smithi122 ceph-mon[93869]: pgmap v17556: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 197 KiB/s wr, 7 op/s 2023-03-09T23:50:50.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:49 smithi181 ceph-mon[94189]: pgmap v17556: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 197 KiB/s wr, 7 op/s 2023-03-09T23:50:51.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31961.438198 secs 2023-03-09T23:50:51.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31961.438198 secs 2023-03-09T23:50:51.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31961.438198 secs 2023-03-09T23:50:52.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:51 smithi122 ceph-mon[93869]: pgmap v17557: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 511 KiB/s wr, 8 op/s 2023-03-09T23:50:52.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:52 smithi181 ceph-mon[94189]: pgmap v17557: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 511 KiB/s wr, 8 op/s 2023-03-09T23:50:52.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:51 smithi038 ceph-mon[92422]: pgmap v17557: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 511 KiB/s wr, 8 op/s 2023-03-09T23:50:53.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:50:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:50:53.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:50:53.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:50:53.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:50:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:50:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32425.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:50:54.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:54 smithi122 ceph-mon[93869]: pgmap v17558: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 511 KiB/s wr, 6 op/s 2023-03-09T23:50:54.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:54 smithi181 ceph-mon[94189]: pgmap v17558: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 511 KiB/s wr, 6 op/s 2023-03-09T23:50:54.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:54 smithi038 ceph-mon[92422]: pgmap v17558: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 511 KiB/s wr, 6 op/s 2023-03-09T23:50:56.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:56 smithi122 ceph-mon[93869]: pgmap v17559: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 793 KiB/s rd, 579 KiB/s wr, 8 op/s 2023-03-09T23:50:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31966.438259 secs 2023-03-09T23:50:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:50:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:50:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:56 smithi181 ceph-mon[94189]: pgmap v17559: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 793 KiB/s rd, 579 KiB/s wr, 8 op/s 2023-03-09T23:50:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31966.438259 secs 2023-03-09T23:50:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:50:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:50:56.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:56 smithi038 ceph-mon[92422]: pgmap v17559: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 793 KiB/s rd, 579 KiB/s wr, 8 op/s 2023-03-09T23:50:56.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31966.438259 secs 2023-03-09T23:50:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:50:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:50:57.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:50:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:50:57.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:50:57.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:50:57.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:50:57.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:50:58.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:50:58 smithi122 ceph-mon[93869]: pgmap v17560: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 545 KiB/s rd, 409 KiB/s wr, 10 op/s 2023-03-09T23:50:58.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:50:58 smithi181 ceph-mon[94189]: pgmap v17560: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 545 KiB/s rd, 409 KiB/s wr, 10 op/s 2023-03-09T23:50:58.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:50:58 smithi038 ceph-mon[92422]: pgmap v17560: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 545 KiB/s rd, 409 KiB/s wr, 10 op/s 2023-03-09T23:50:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32430.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:51:00.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:00 smithi122 ceph-mon[93869]: pgmap v17561: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 817 KiB/s rd, 683 KiB/s wr, 10 op/s 2023-03-09T23:51:00.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:00 smithi181 ceph-mon[94189]: pgmap v17561: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 817 KiB/s rd, 683 KiB/s wr, 10 op/s 2023-03-09T23:51:00.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:00 smithi038 ceph-mon[92422]: pgmap v17561: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 817 KiB/s rd, 683 KiB/s wr, 10 op/s 2023-03-09T23:51:01.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31971.438338 secs 2023-03-09T23:51:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:51:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:51:01.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31971.438338 secs 2023-03-09T23:51:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:51:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:51:01.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31971.438338 secs 2023-03-09T23:51:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:51:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:51:02.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:02 smithi122 ceph-mon[93869]: pgmap v17562: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 797 KiB/s rd, 870 KiB/s wr, 13 op/s 2023-03-09T23:51:02.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:02 smithi181 ceph-mon[94189]: pgmap v17562: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 797 KiB/s rd, 870 KiB/s wr, 13 op/s 2023-03-09T23:51:02.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:02 smithi038 ceph-mon[92422]: pgmap v17562: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 797 KiB/s rd, 870 KiB/s wr, 13 op/s 2023-03-09T23:51:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32435.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:51:04.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:03 smithi122 ceph-mon[93869]: pgmap v17563: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 556 KiB/s wr, 11 op/s 2023-03-09T23:51:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:03 smithi181 ceph-mon[94189]: pgmap v17563: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 556 KiB/s wr, 11 op/s 2023-03-09T23:51:04.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:03 smithi038 ceph-mon[92422]: pgmap v17563: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 556 KiB/s wr, 11 op/s 2023-03-09T23:51:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:05 smithi181 ceph-mon[94189]: pgmap v17564: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 902 KiB/s rd, 602 KiB/s wr, 14 op/s 2023-03-09T23:51:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31976.438400 secs 2023-03-09T23:51:06.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:05 smithi038 ceph-mon[92422]: pgmap v17564: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 902 KiB/s rd, 602 KiB/s wr, 14 op/s 2023-03-09T23:51:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31976.438400 secs 2023-03-09T23:51:06.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:05 smithi122 ceph-mon[93869]: pgmap v17564: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 902 KiB/s rd, 602 KiB/s wr, 14 op/s 2023-03-09T23:51:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31976.438400 secs 2023-03-09T23:51:08.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:07 smithi038 ceph-mon[92422]: pgmap v17565: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 583 KiB/s wr, 16 op/s 2023-03-09T23:51:08.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:51:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:51:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:07 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:51:08.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:07 smithi122 ceph-mon[93869]: pgmap v17565: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 583 KiB/s wr, 16 op/s 2023-03-09T23:51:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:51:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:51:08.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:07 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:51:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:07 smithi181 ceph-mon[94189]: pgmap v17565: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 583 KiB/s wr, 16 op/s 2023-03-09T23:51:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:51:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:51:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:07 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:51:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32440.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:51:09.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31978.856883 secs 2023-03-09T23:51:09.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31978.856883 secs 2023-03-09T23:51:09.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31978.856883 secs 2023-03-09T23:51:10.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:09 smithi038 ceph-mon[92422]: pgmap v17566: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 540 KiB/s rd, 578 KiB/s wr, 25 op/s 2023-03-09T23:51:10.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:09 smithi122 ceph-mon[93869]: pgmap v17566: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 540 KiB/s rd, 578 KiB/s wr, 25 op/s 2023-03-09T23:51:10.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:09 smithi181 ceph-mon[94189]: pgmap v17566: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 540 KiB/s rd, 578 KiB/s wr, 25 op/s 2023-03-09T23:51:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:10 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:51:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31981.438464 secs 2023-03-09T23:51:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:10 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:51:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:10 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:51:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:10 smithi038 ceph-mon[92422]: osdmap e1313: 12 total, 12 up, 12 in 2023-03-09T23:51:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:51:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:51:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:51:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:51:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:51:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:51:11.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:10 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:51:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31981.438464 secs 2023-03-09T23:51:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:10 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:51:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:10 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:51:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:10 smithi122 ceph-mon[93869]: osdmap e1313: 12 total, 12 up, 12 in 2023-03-09T23:51:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:51:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:51:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:51:11.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:51:11.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:51:11.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:51:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:10 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:51:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31981.438464 secs 2023-03-09T23:51:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:10 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:51:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:10 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:51:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:10 smithi181 ceph-mon[94189]: osdmap e1313: 12 total, 12 up, 12 in 2023-03-09T23:51:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:51:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:51:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:51:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:51:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:51:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:51:12.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:12 smithi122 ceph-mon[93869]: pgmap v17568: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 640 KiB/s rd, 513 KiB/s wr, 29 op/s 2023-03-09T23:51:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:12 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:51:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:12 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:51:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:12 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:51:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:12 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:51:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:12 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 185.4M 2023-03-09T23:51:12.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:12 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi181 to 194463080: error parsing value: Value '194463080' is below minimum 939524096 2023-03-09T23:51:12.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:12 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:51:12.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:12 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:51:12.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:12 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:51:12.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:12 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:51:12.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:12 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:51:12.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:12 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:51:12.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:12 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:51:12.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:12 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:51:12.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:12 smithi181 ceph-mon[94189]: pgmap v17568: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 640 KiB/s rd, 513 KiB/s wr, 29 op/s 2023-03-09T23:51:12.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:12 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:51:12.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:12 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:51:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:12 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:51:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:12 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:51:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:12 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 185.4M 2023-03-09T23:51:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:12 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi181 to 194463080: error parsing value: Value '194463080' is below minimum 939524096 2023-03-09T23:51:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:12 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:51:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:12 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:51:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:12 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:51:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:12 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:51:12.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:12 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:51:12.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:12 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:51:12.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:12 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:51:12.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:12 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:51:12.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:12 smithi038 ceph-mon[92422]: pgmap v17568: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 640 KiB/s rd, 513 KiB/s wr, 29 op/s 2023-03-09T23:51:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:12 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:51:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:12 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:51:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:12 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:51:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:12 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:51:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:12 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 185.4M 2023-03-09T23:51:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:12 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi181 to 194463080: error parsing value: Value '194463080' is below minimum 939524096 2023-03-09T23:51:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:12 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:51:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:12 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:51:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:12 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:51:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:12 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:51:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:12 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-09T23:51:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:12 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:51:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:12 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:51:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:12 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:51:13.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:13 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T23:51:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:13 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T23:51:13.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:13 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T23:51:13.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:13 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T23:51:13.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:13 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-09T23:51:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:13 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-09T23:51:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32445.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:51:14.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:14 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31983.857004 secs 2023-03-09T23:51:14.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:14 smithi122 ceph-mon[93869]: pgmap v17569: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 640 KiB/s rd, 513 KiB/s wr, 29 op/s 2023-03-09T23:51:14.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:14 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31983.857004 secs 2023-03-09T23:51:14.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:14 smithi181 ceph-mon[94189]: pgmap v17569: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 640 KiB/s rd, 513 KiB/s wr, 29 op/s 2023-03-09T23:51:14.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:14 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31983.857004 secs 2023-03-09T23:51:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:14 smithi038 ceph-mon[92422]: pgmap v17569: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 640 KiB/s rd, 513 KiB/s wr, 29 op/s 2023-03-09T23:51:16.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:16 smithi122 ceph-mon[93869]: pgmap v17570: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 457 KiB/s wr, 26 op/s 2023-03-09T23:51:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31986.438521 secs 2023-03-09T23:51:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:51:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:51:16.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:16 smithi181 ceph-mon[94189]: pgmap v17570: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 457 KiB/s wr, 26 op/s 2023-03-09T23:51:16.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31986.438521 secs 2023-03-09T23:51:16.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:51:16.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:51:16.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:16 smithi038 ceph-mon[92422]: pgmap v17570: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 457 KiB/s wr, 26 op/s 2023-03-09T23:51:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31986.438521 secs 2023-03-09T23:51:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:51:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:51:17.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:51:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:51:17.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:51:17.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:51:17.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:51:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:51:18.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:18 smithi122 ceph-mon[93869]: pgmap v17571: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 503 KiB/s rd, 517 KiB/s wr, 24 op/s 2023-03-09T23:51:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31988.857123 secs 2023-03-09T23:51:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:18 smithi122 ceph-mon[93869]: osdmap e1314: 12 total, 12 up, 12 in 2023-03-09T23:51:18.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:18 smithi181 ceph-mon[94189]: pgmap v17571: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 503 KiB/s rd, 517 KiB/s wr, 24 op/s 2023-03-09T23:51:18.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31988.857123 secs 2023-03-09T23:51:18.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:18 smithi181 ceph-mon[94189]: osdmap e1314: 12 total, 12 up, 12 in 2023-03-09T23:51:18.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:18 smithi038 ceph-mon[92422]: pgmap v17571: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 503 KiB/s rd, 517 KiB/s wr, 24 op/s 2023-03-09T23:51:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31988.857123 secs 2023-03-09T23:51:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:18 smithi038 ceph-mon[92422]: osdmap e1314: 12 total, 12 up, 12 in 2023-03-09T23:51:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32450.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:51:20.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:20 smithi122 ceph-mon[93869]: pgmap v17573: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 363 KiB/s wr, 10 op/s 2023-03-09T23:51:20.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:20 smithi181 ceph-mon[94189]: pgmap v17573: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 363 KiB/s wr, 10 op/s 2023-03-09T23:51:20.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:20 smithi038 ceph-mon[92422]: pgmap v17573: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 363 KiB/s wr, 10 op/s 2023-03-09T23:51:21.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31991.438618 secs 2023-03-09T23:51:21.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31991.438618 secs 2023-03-09T23:51:21.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31991.438618 secs 2023-03-09T23:51:22.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:22 smithi122 ceph-mon[93869]: pgmap v17574: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 118 KiB/s wr, 8 op/s 2023-03-09T23:51:22.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:22 smithi181 ceph-mon[94189]: pgmap v17574: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 118 KiB/s wr, 8 op/s 2023-03-09T23:51:22.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:22 smithi038 ceph-mon[92422]: pgmap v17574: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 118 KiB/s wr, 8 op/s 2023-03-09T23:51:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32455.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:51:24.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31993.857247 secs 2023-03-09T23:51:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:23 smithi122 ceph-mon[93869]: pgmap v17575: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 118 KiB/s wr, 8 op/s 2023-03-09T23:51:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31993.857247 secs 2023-03-09T23:51:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:23 smithi181 ceph-mon[94189]: pgmap v17575: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 118 KiB/s wr, 8 op/s 2023-03-09T23:51:24.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31993.857247 secs 2023-03-09T23:51:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:23 smithi038 ceph-mon[92422]: pgmap v17575: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 118 KiB/s wr, 8 op/s 2023-03-09T23:51:26.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:25 smithi038 ceph-mon[92422]: pgmap v17576: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 118 KiB/s wr, 7 op/s 2023-03-09T23:51:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31996.438676 secs 2023-03-09T23:51:26.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:25 smithi122 ceph-mon[93869]: pgmap v17576: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 118 KiB/s wr, 7 op/s 2023-03-09T23:51:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31996.438676 secs 2023-03-09T23:51:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:25 smithi181 ceph-mon[94189]: pgmap v17576: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 118 KiB/s wr, 7 op/s 2023-03-09T23:51:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31996.438676 secs 2023-03-09T23:51:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:27 smithi038 ceph-mon[92422]: pgmap v17577: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T23:51:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:27 smithi122 ceph-mon[93869]: pgmap v17577: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T23:51:28.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:27 smithi181 ceph-mon[94189]: pgmap v17577: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2023-03-09T23:51:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32460.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:51:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 31998.857371 secs 2023-03-09T23:51:29.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 31998.857371 secs 2023-03-09T23:51:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 31998.857371 secs 2023-03-09T23:51:30.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:30 smithi122 ceph-mon[93869]: pgmap v17578: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T23:51:30.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:30 smithi181 ceph-mon[94189]: pgmap v17578: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T23:51:30.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:30 smithi038 ceph-mon[92422]: pgmap v17578: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T23:51:31.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32001.438738 secs 2023-03-09T23:51:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32001.438738 secs 2023-03-09T23:51:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32001.438738 secs 2023-03-09T23:51:32.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:32 smithi122 ceph-mon[93869]: pgmap v17579: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T23:51:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:32 smithi181 ceph-mon[94189]: pgmap v17579: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T23:51:32.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:32 smithi038 ceph-mon[92422]: pgmap v17579: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T23:51:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32465.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:51:34.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32003.857492 secs 2023-03-09T23:51:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:33 smithi122 ceph-mon[93869]: pgmap v17580: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T23:51:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32003.857492 secs 2023-03-09T23:51:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:33 smithi181 ceph-mon[94189]: pgmap v17580: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T23:51:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32003.857492 secs 2023-03-09T23:51:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:33 smithi038 ceph-mon[92422]: pgmap v17580: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T23:51:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:35 smithi038 ceph-mon[92422]: pgmap v17581: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T23:51:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32006.438818 secs 2023-03-09T23:51:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:35 smithi122 ceph-mon[93869]: pgmap v17581: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T23:51:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32006.438818 secs 2023-03-09T23:51:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:35 smithi181 ceph-mon[94189]: pgmap v17581: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T23:51:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32006.438818 secs 2023-03-09T23:51:38.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:37 smithi038 ceph-mon[92422]: pgmap v17582: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T23:51:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:37 smithi122 ceph-mon[93869]: pgmap v17582: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T23:51:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:37 smithi181 ceph-mon[94189]: pgmap v17582: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T23:51:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32470.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:51:39.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:39 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32008.857608 secs 2023-03-09T23:51:39.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:39 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32008.857608 secs 2023-03-09T23:51:39.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:39 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32008.857608 secs 2023-03-09T23:51:40.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:40 smithi122 ceph-mon[93869]: pgmap v17583: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-09T23:51:40.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:40 smithi181 ceph-mon[94189]: pgmap v17583: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-09T23:51:40.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:40 smithi038 ceph-mon[92422]: pgmap v17583: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-09T23:51:41.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32011.438878 secs 2023-03-09T23:51:41.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32011.438878 secs 2023-03-09T23:51:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32011.438878 secs 2023-03-09T23:51:42.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:42 smithi122 ceph-mon[93869]: pgmap v17584: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 386 KiB/s rd, 96 KiB/s wr, 8 op/s 2023-03-09T23:51:42.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:42 smithi181 ceph-mon[94189]: pgmap v17584: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 386 KiB/s rd, 96 KiB/s wr, 8 op/s 2023-03-09T23:51:42.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:42 smithi038 ceph-mon[92422]: pgmap v17584: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 386 KiB/s rd, 96 KiB/s wr, 8 op/s 2023-03-09T23:51:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32475.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:51:44.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:43 smithi122 ceph-mon[93869]: pgmap v17585: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 96 KiB/s wr, 6 op/s 2023-03-09T23:51:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:43 smithi181 ceph-mon[94189]: pgmap v17585: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 96 KiB/s wr, 6 op/s 2023-03-09T23:51:44.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:43 smithi038 ceph-mon[92422]: pgmap v17585: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 96 KiB/s wr, 6 op/s 2023-03-09T23:51:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:45 smithi038 ceph-mon[92422]: pgmap v17586: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 362 KiB/s wr, 7 op/s 2023-03-09T23:51:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32016.438937 secs 2023-03-09T23:51:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:45 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:51:46.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:45 smithi122 ceph-mon[93869]: pgmap v17586: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 362 KiB/s wr, 7 op/s 2023-03-09T23:51:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32016.438937 secs 2023-03-09T23:51:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:45 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:51:46.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:45 smithi181 ceph-mon[94189]: pgmap v17586: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 362 KiB/s wr, 7 op/s 2023-03-09T23:51:46.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32016.438937 secs 2023-03-09T23:51:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:45 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:51:47.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:47 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:51:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:51:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:51:47.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:47 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:51:47.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:51:47.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:51:47.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:47 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:51:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:51:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:51:48.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:48 smithi122 ceph-mon[93869]: pgmap v17587: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 679 KiB/s rd, 388 KiB/s wr, 10 op/s 2023-03-09T23:51:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:51:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:51:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:48 smithi181 ceph-mon[94189]: pgmap v17587: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 679 KiB/s rd, 388 KiB/s wr, 10 op/s 2023-03-09T23:51:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:51:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:51:48.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:48 smithi038 ceph-mon[92422]: pgmap v17587: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 679 KiB/s rd, 388 KiB/s wr, 10 op/s 2023-03-09T23:51:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:51:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:51:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32480.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:51:49.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:51:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:51:49.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:51:49.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:51:49.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:51:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:51:50.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:50 smithi122 ceph-mon[93869]: pgmap v17588: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 972 KiB/s rd, 464 KiB/s wr, 9 op/s 2023-03-09T23:51:50.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:50 smithi181 ceph-mon[94189]: pgmap v17588: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 972 KiB/s rd, 464 KiB/s wr, 9 op/s 2023-03-09T23:51:50.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:50 smithi038 ceph-mon[92422]: pgmap v17588: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 972 KiB/s rd, 464 KiB/s wr, 9 op/s 2023-03-09T23:51:51.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32021.438986 secs 2023-03-09T23:51:51.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32021.438986 secs 2023-03-09T23:51:51.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32021.438986 secs 2023-03-09T23:51:52.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:52 smithi122 ceph-mon[93869]: pgmap v17589: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1013 KiB/s rd, 490 KiB/s wr, 12 op/s 2023-03-09T23:51:52.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:52 smithi181 ceph-mon[94189]: pgmap v17589: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1013 KiB/s rd, 490 KiB/s wr, 12 op/s 2023-03-09T23:51:52.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:52 smithi038 ceph-mon[92422]: pgmap v17589: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1013 KiB/s rd, 490 KiB/s wr, 12 op/s 2023-03-09T23:51:53.793 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32485.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:51:54.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:53 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 32023.857936 secs 2023-03-09T23:51:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:53 smithi122 ceph-mon[93869]: pgmap v17590: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 815 KiB/s rd, 580 KiB/s wr, 10 op/s 2023-03-09T23:51:54.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:53 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 32023.857936 secs 2023-03-09T23:51:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:53 smithi181 ceph-mon[94189]: pgmap v17590: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 815 KiB/s rd, 580 KiB/s wr, 10 op/s 2023-03-09T23:51:54.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:53 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 32023.857936 secs 2023-03-09T23:51:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:53 smithi038 ceph-mon[92422]: pgmap v17590: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 815 KiB/s rd, 580 KiB/s wr, 10 op/s 2023-03-09T23:51:55.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:54 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:51:55.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:54 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:51:55.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:54 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:51:55.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:54 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:51:55.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:54 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:51:55.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:54 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:51:55.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:54 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:51:55.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:54 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:51:55.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:54 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:51:56.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:51:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:51:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:55 smithi122 ceph-mon[93869]: pgmap v17591: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 962 KiB/s rd, 580 KiB/s wr, 11 op/s 2023-03-09T23:51:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32026.439045 secs 2023-03-09T23:51:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:51:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:51:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:55 smithi181 ceph-mon[94189]: pgmap v17591: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 962 KiB/s rd, 580 KiB/s wr, 11 op/s 2023-03-09T23:51:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32026.439045 secs 2023-03-09T23:51:56.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:51:56.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:51:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:55 smithi038 ceph-mon[92422]: pgmap v17591: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 962 KiB/s rd, 580 KiB/s wr, 11 op/s 2023-03-09T23:51:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32026.439045 secs 2023-03-09T23:51:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:51:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:51:57.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:51:57.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:51:57.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:51:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:51:58.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:57 smithi122 ceph-mon[93869]: pgmap v17592: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 843 KiB/s rd, 464 KiB/s wr, 13 op/s 2023-03-09T23:51:58.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:57 smithi181 ceph-mon[94189]: pgmap v17592: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 843 KiB/s rd, 464 KiB/s wr, 13 op/s 2023-03-09T23:51:58.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:57 smithi038 ceph-mon[92422]: pgmap v17592: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 843 KiB/s rd, 464 KiB/s wr, 13 op/s 2023-03-09T23:51:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32490.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:51:59.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32028.858046 secs 2023-03-09T23:51:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:58 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:51:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:58 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:51:59.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32028.858046 secs 2023-03-09T23:51:59.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:58 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:51:59.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:58 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:51:59.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32028.858046 secs 2023-03-09T23:51:59.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:58 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:51:59.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:58 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:52:00.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:52:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:52:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:51:59 smithi122 ceph-mon[93869]: pgmap v17593: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 971 KiB/s rd, 595 KiB/s wr, 13 op/s 2023-03-09T23:52:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:52:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:52:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:51:59 smithi181 ceph-mon[94189]: pgmap v17593: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 971 KiB/s rd, 595 KiB/s wr, 13 op/s 2023-03-09T23:52:00.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:52:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:52:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:51:59 smithi038 ceph-mon[92422]: pgmap v17593: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 971 KiB/s rd, 595 KiB/s wr, 13 op/s 2023-03-09T23:52:01.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:52:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:52:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32031.439124 secs 2023-03-09T23:52:01.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:52:01.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:52:01.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32031.439124 secs 2023-03-09T23:52:01.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:52:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:52:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32031.439124 secs 2023-03-09T23:52:02.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:01 smithi122 ceph-mon[93869]: pgmap v17594: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 848 KiB/s rd, 710 KiB/s wr, 18 op/s 2023-03-09T23:52:02.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:01 smithi181 ceph-mon[94189]: pgmap v17594: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 848 KiB/s rd, 710 KiB/s wr, 18 op/s 2023-03-09T23:52:02.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:01 smithi038 ceph-mon[92422]: pgmap v17594: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 848 KiB/s rd, 710 KiB/s wr, 18 op/s 2023-03-09T23:52:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32495.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:52:04.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32033.858185 secs 2023-03-09T23:52:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:03 smithi122 ceph-mon[93869]: pgmap v17595: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 807 KiB/s rd, 684 KiB/s wr, 15 op/s 2023-03-09T23:52:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32033.858185 secs 2023-03-09T23:52:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:03 smithi181 ceph-mon[94189]: pgmap v17595: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 807 KiB/s rd, 684 KiB/s wr, 15 op/s 2023-03-09T23:52:04.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32033.858185 secs 2023-03-09T23:52:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:03 smithi038 ceph-mon[92422]: pgmap v17595: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 807 KiB/s rd, 684 KiB/s wr, 15 op/s 2023-03-09T23:52:06.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:05 smithi122 ceph-mon[93869]: osdmap e1315: 12 total, 12 up, 12 in 2023-03-09T23:52:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:05 smithi122 ceph-mon[93869]: pgmap v17597: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 579 KiB/s rd, 623 KiB/s wr, 30 op/s 2023-03-09T23:52:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32036.439185 secs 2023-03-09T23:52:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:05 smithi181 ceph-mon[94189]: osdmap e1315: 12 total, 12 up, 12 in 2023-03-09T23:52:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:05 smithi181 ceph-mon[94189]: pgmap v17597: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 579 KiB/s rd, 623 KiB/s wr, 30 op/s 2023-03-09T23:52:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32036.439185 secs 2023-03-09T23:52:06.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:05 smithi038 ceph-mon[92422]: osdmap e1315: 12 total, 12 up, 12 in 2023-03-09T23:52:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:05 smithi038 ceph-mon[92422]: pgmap v17597: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 579 KiB/s rd, 623 KiB/s wr, 30 op/s 2023-03-09T23:52:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32036.439185 secs 2023-03-09T23:52:08.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:52:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:52:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:07 smithi122 ceph-mon[93869]: pgmap v17598: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 611 KiB/s rd, 649 KiB/s wr, 30 op/s 2023-03-09T23:52:08.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:52:08.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:52:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:07 smithi181 ceph-mon[94189]: pgmap v17598: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 611 KiB/s rd, 649 KiB/s wr, 30 op/s 2023-03-09T23:52:08.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:52:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:52:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:07 smithi038 ceph-mon[92422]: pgmap v17598: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 611 KiB/s rd, 649 KiB/s wr, 30 op/s 2023-03-09T23:52:08.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32500.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:52:09.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32038.858297 secs 2023-03-09T23:52:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:52:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:52:09.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32038.858297 secs 2023-03-09T23:52:09.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:52:09.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:52:09.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32038.858297 secs 2023-03-09T23:52:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:52:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:52:10.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:09 smithi122 ceph-mon[93869]: pgmap v17599: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 461 KiB/s wr, 29 op/s 2023-03-09T23:52:10.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:09 smithi181 ceph-mon[94189]: pgmap v17599: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 461 KiB/s wr, 29 op/s 2023-03-09T23:52:10.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:09 smithi038 ceph-mon[92422]: pgmap v17599: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 461 KiB/s wr, 29 op/s 2023-03-09T23:52:11.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32041.439245 secs 2023-03-09T23:52:11.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32041.439245 secs 2023-03-09T23:52:11.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32041.439245 secs 2023-03-09T23:52:12.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:11 smithi122 ceph-mon[93869]: pgmap v17600: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 386 KiB/s rd, 400 KiB/s wr, 25 op/s 2023-03-09T23:52:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:11 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:52:12.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:11 smithi181 ceph-mon[94189]: pgmap v17600: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 386 KiB/s rd, 400 KiB/s wr, 25 op/s 2023-03-09T23:52:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:11 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:52:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:11 smithi038 ceph-mon[92422]: pgmap v17600: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 386 KiB/s rd, 400 KiB/s wr, 25 op/s 2023-03-09T23:52:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:11 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:52:13.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:12 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:52:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:12 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:52:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:12 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:52:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:12 smithi122 ceph-mon[93869]: osdmap e1316: 12 total, 12 up, 12 in 2023-03-09T23:52:13.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:12 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:52:13.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:12 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:52:13.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:12 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:52:13.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:12 smithi181 ceph-mon[94189]: osdmap e1316: 12 total, 12 up, 12 in 2023-03-09T23:52:13.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:12 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:52:13.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:12 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:52:13.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:12 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:52:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:12 smithi038 ceph-mon[92422]: osdmap e1316: 12 total, 12 up, 12 in 2023-03-09T23:52:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32505.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:52:14.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32043.858398 secs 2023-03-09T23:52:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:13 smithi122 ceph-mon[93869]: pgmap v17602: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 461 KiB/s wr, 12 op/s 2023-03-09T23:52:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32043.858398 secs 2023-03-09T23:52:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:13 smithi181 ceph-mon[94189]: pgmap v17602: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 461 KiB/s wr, 12 op/s 2023-03-09T23:52:14.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32043.858398 secs 2023-03-09T23:52:14.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:13 smithi038 ceph-mon[92422]: pgmap v17602: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 461 KiB/s wr, 12 op/s 2023-03-09T23:52:16.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:15 smithi122 ceph-mon[93869]: pgmap v17603: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 374 KiB/s wr, 10 op/s 2023-03-09T23:52:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32046.439317 secs 2023-03-09T23:52:16.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:16 smithi181 ceph-mon[94189]: pgmap v17603: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 374 KiB/s wr, 10 op/s 2023-03-09T23:52:16.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32046.439317 secs 2023-03-09T23:52:16.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:15 smithi038 ceph-mon[92422]: pgmap v17603: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 374 KiB/s wr, 10 op/s 2023-03-09T23:52:16.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32046.439317 secs 2023-03-09T23:52:18.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:18 smithi122 ceph-mon[93869]: pgmap v17604: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 168 KiB/s wr, 8 op/s 2023-03-09T23:52:18.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:18 smithi181 ceph-mon[94189]: pgmap v17604: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 168 KiB/s wr, 8 op/s 2023-03-09T23:52:18.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:18 smithi038 ceph-mon[92422]: pgmap v17604: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 168 KiB/s wr, 8 op/s 2023-03-09T23:52:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32510.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:52:19.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:19 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32048.858515 secs 2023-03-09T23:52:19.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:19 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32048.858515 secs 2023-03-09T23:52:19.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:19 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32048.858515 secs 2023-03-09T23:52:20.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:20 smithi122 ceph-mon[93869]: pgmap v17605: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 168 KiB/s wr, 7 op/s 2023-03-09T23:52:20.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:20 smithi181 ceph-mon[94189]: pgmap v17605: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 168 KiB/s wr, 7 op/s 2023-03-09T23:52:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:20 smithi038 ceph-mon[92422]: pgmap v17605: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 168 KiB/s wr, 7 op/s 2023-03-09T23:52:21.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32051.439396 secs 2023-03-09T23:52:21.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32051.439396 secs 2023-03-09T23:52:21.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32051.439396 secs 2023-03-09T23:52:22.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:22 smithi122 ceph-mon[93869]: pgmap v17606: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T23:52:22.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:22 smithi181 ceph-mon[94189]: pgmap v17606: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T23:52:22.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:22 smithi038 ceph-mon[92422]: pgmap v17606: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T23:52:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32515.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:52:24.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32053.858636 secs 2023-03-09T23:52:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:23 smithi122 ceph-mon[93869]: pgmap v17607: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-09T23:52:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32053.858636 secs 2023-03-09T23:52:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:23 smithi181 ceph-mon[94189]: pgmap v17607: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-09T23:52:24.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32053.858636 secs 2023-03-09T23:52:24.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:23 smithi038 ceph-mon[92422]: pgmap v17607: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-09T23:52:26.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:26 smithi122 ceph-mon[93869]: pgmap v17608: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T23:52:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32056.439458 secs 2023-03-09T23:52:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:26 smithi181 ceph-mon[94189]: pgmap v17608: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T23:52:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32056.439458 secs 2023-03-09T23:52:26.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:26 smithi038 ceph-mon[92422]: pgmap v17608: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T23:52:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32056.439458 secs 2023-03-09T23:52:28.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:28 smithi122 ceph-mon[93869]: pgmap v17609: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T23:52:28.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:28 smithi181 ceph-mon[94189]: pgmap v17609: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T23:52:28.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:28 smithi038 ceph-mon[92422]: pgmap v17609: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T23:52:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32520.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:52:29.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:29 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32058.858751 secs 2023-03-09T23:52:29.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:29 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32058.858751 secs 2023-03-09T23:52:29.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:29 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32058.858751 secs 2023-03-09T23:52:30.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:30 smithi122 ceph-mon[93869]: pgmap v17610: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T23:52:30.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:30 smithi181 ceph-mon[94189]: pgmap v17610: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T23:52:30.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:30 smithi038 ceph-mon[92422]: pgmap v17610: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T23:52:31.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32061.439518 secs 2023-03-09T23:52:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32061.439518 secs 2023-03-09T23:52:31.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32061.439518 secs 2023-03-09T23:52:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:32 smithi122 ceph-mon[93869]: pgmap v17611: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T23:52:32.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:32 smithi181 ceph-mon[94189]: pgmap v17611: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T23:52:32.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:32 smithi038 ceph-mon[92422]: pgmap v17611: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-09T23:52:33.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32525.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:52:34.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32063.858865 secs 2023-03-09T23:52:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:33 smithi122 ceph-mon[93869]: pgmap v17612: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-09T23:52:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32063.858865 secs 2023-03-09T23:52:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:33 smithi181 ceph-mon[94189]: pgmap v17612: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-09T23:52:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32063.858865 secs 2023-03-09T23:52:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:33 smithi038 ceph-mon[92422]: pgmap v17612: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-09T23:52:36.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:36 smithi122 ceph-mon[93869]: pgmap v17613: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-09T23:52:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32066.439579 secs 2023-03-09T23:52:36.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:36 smithi181 ceph-mon[94189]: pgmap v17613: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-09T23:52:36.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32066.439579 secs 2023-03-09T23:52:36.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:36 smithi038 ceph-mon[92422]: pgmap v17613: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-09T23:52:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32066.439579 secs 2023-03-09T23:52:38.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:38 smithi122 ceph-mon[93869]: pgmap v17614: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 8 op/s 2023-03-09T23:52:38.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:38 smithi181 ceph-mon[94189]: pgmap v17614: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 8 op/s 2023-03-09T23:52:38.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:38 smithi038 ceph-mon[92422]: pgmap v17614: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 8 op/s 2023-03-09T23:52:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32530.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:52:39.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:39 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32068.858950 secs 2023-03-09T23:52:39.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:39 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32068.858950 secs 2023-03-09T23:52:39.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:39 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32068.858950 secs 2023-03-09T23:52:40.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:40 smithi122 ceph-mon[93869]: pgmap v17615: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 767 B/s wr, 6 op/s 2023-03-09T23:52:40.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:40 smithi181 ceph-mon[94189]: pgmap v17615: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 767 B/s wr, 6 op/s 2023-03-09T23:52:40.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:40 smithi038 ceph-mon[92422]: pgmap v17615: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 767 B/s wr, 6 op/s 2023-03-09T23:52:41.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32071.439640 secs 2023-03-09T23:52:41.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32071.439640 secs 2023-03-09T23:52:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32071.439640 secs 2023-03-09T23:52:42.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:42 smithi122 ceph-mon[93869]: pgmap v17616: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 124 KiB/s wr, 9 op/s 2023-03-09T23:52:42.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:42 smithi181 ceph-mon[94189]: pgmap v17616: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 124 KiB/s wr, 9 op/s 2023-03-09T23:52:42.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:42 smithi038 ceph-mon[92422]: pgmap v17616: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 124 KiB/s wr, 9 op/s 2023-03-09T23:52:43.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32535.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:52:44.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32073.859085 secs 2023-03-09T23:52:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:43 smithi122 ceph-mon[93869]: pgmap v17617: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 124 KiB/s wr, 6 op/s 2023-03-09T23:52:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32073.859085 secs 2023-03-09T23:52:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:43 smithi181 ceph-mon[94189]: pgmap v17617: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 124 KiB/s wr, 6 op/s 2023-03-09T23:52:44.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32073.859085 secs 2023-03-09T23:52:44.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:43 smithi038 ceph-mon[92422]: pgmap v17617: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 124 KiB/s wr, 6 op/s 2023-03-09T23:52:46.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:46 smithi122 ceph-mon[93869]: pgmap v17618: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 124 KiB/s wr, 6 op/s 2023-03-09T23:52:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32076.439699 secs 2023-03-09T23:52:46.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:46 smithi181 ceph-mon[94189]: pgmap v17618: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 124 KiB/s wr, 6 op/s 2023-03-09T23:52:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32076.439699 secs 2023-03-09T23:52:46.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:46 smithi038 ceph-mon[92422]: pgmap v17618: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 124 KiB/s wr, 6 op/s 2023-03-09T23:52:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32076.439699 secs 2023-03-09T23:52:48.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:48 smithi122 ceph-mon[93869]: pgmap v17619: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 244 KiB/s wr, 9 op/s 2023-03-09T23:52:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:48 smithi181 ceph-mon[94189]: pgmap v17619: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 244 KiB/s wr, 9 op/s 2023-03-09T23:52:48.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:48 smithi038 ceph-mon[92422]: pgmap v17619: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 244 KiB/s wr, 9 op/s 2023-03-09T23:52:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32540.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:52:49.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:49 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32078.859146 secs 2023-03-09T23:52:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:49 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32078.859146 secs 2023-03-09T23:52:49.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:49 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32078.859146 secs 2023-03-09T23:52:50.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:50 smithi122 ceph-mon[93869]: pgmap v17620: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 244 KiB/s wr, 7 op/s 2023-03-09T23:52:50.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:50 smithi181 ceph-mon[94189]: pgmap v17620: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 244 KiB/s wr, 7 op/s 2023-03-09T23:52:50.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:50 smithi038 ceph-mon[92422]: pgmap v17620: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 244 KiB/s wr, 7 op/s 2023-03-09T23:52:51.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32081.439760 secs 2023-03-09T23:52:51.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32081.439760 secs 2023-03-09T23:52:51.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32081.439760 secs 2023-03-09T23:52:52.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:52 smithi122 ceph-mon[93869]: pgmap v17621: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 628 KiB/s wr, 10 op/s 2023-03-09T23:52:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:52:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:52:52.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:52 smithi181 ceph-mon[94189]: pgmap v17621: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 628 KiB/s wr, 10 op/s 2023-03-09T23:52:52.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:52:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:52:52.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:52 smithi038 ceph-mon[92422]: pgmap v17621: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 628 KiB/s wr, 10 op/s 2023-03-09T23:52:52.541 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:52:52.541 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:52:53.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:52:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:52:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32083.859268 secs 2023-03-09T23:52:53.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:52:53.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:52:53.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32083.859268 secs 2023-03-09T23:52:53.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:52:53.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:52:53.509 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32083.859268 secs 2023-03-09T23:52:53.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32545.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:52:54.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:54 smithi122 ceph-mon[93869]: pgmap v17622: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 907 KiB/s rd, 505 KiB/s wr, 7 op/s 2023-03-09T23:52:54.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:54 smithi181 ceph-mon[94189]: pgmap v17622: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 907 KiB/s rd, 505 KiB/s wr, 7 op/s 2023-03-09T23:52:54.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:54 smithi038 ceph-mon[92422]: pgmap v17622: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 907 KiB/s rd, 505 KiB/s wr, 7 op/s 2023-03-09T23:52:56.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:56 smithi122 ceph-mon[93869]: pgmap v17623: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 907 KiB/s rd, 505 KiB/s wr, 8 op/s 2023-03-09T23:52:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32086.439818 secs 2023-03-09T23:52:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:52:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:52:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:56 smithi181 ceph-mon[94189]: pgmap v17623: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 907 KiB/s rd, 505 KiB/s wr, 8 op/s 2023-03-09T23:52:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32086.439818 secs 2023-03-09T23:52:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:52:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:52:56.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:56 smithi038 ceph-mon[92422]: pgmap v17623: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 907 KiB/s rd, 505 KiB/s wr, 8 op/s 2023-03-09T23:52:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32086.439818 secs 2023-03-09T23:52:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:52:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:52:57.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:52:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:52:57.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:52:57.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:52:57.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:52:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:52:58.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:58 smithi122 ceph-mon[93869]: pgmap v17624: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 753 KiB/s wr, 12 op/s 2023-03-09T23:52:58.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:52:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32088.859383 secs 2023-03-09T23:52:58.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:58 smithi181 ceph-mon[94189]: pgmap v17624: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 753 KiB/s wr, 12 op/s 2023-03-09T23:52:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:52:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32088.859383 secs 2023-03-09T23:52:58.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:58 smithi038 ceph-mon[92422]: pgmap v17624: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 753 KiB/s wr, 12 op/s 2023-03-09T23:52:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:52:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32088.859383 secs 2023-03-09T23:52:58.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32550.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:53:00.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:00 smithi122 ceph-mon[93869]: pgmap v17625: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 797 KiB/s wr, 10 op/s 2023-03-09T23:53:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:53:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:53:00.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:00 smithi181 ceph-mon[94189]: pgmap v17625: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 797 KiB/s wr, 10 op/s 2023-03-09T23:53:00.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:53:00.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:53:00.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:00 smithi038 ceph-mon[92422]: pgmap v17625: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 797 KiB/s wr, 10 op/s 2023-03-09T23:53:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:53:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:53:01.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32091.439875 secs 2023-03-09T23:53:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:53:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:53:01.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32091.439875 secs 2023-03-09T23:53:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:53:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:53:01.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32091.439875 secs 2023-03-09T23:53:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:53:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:53:02.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:02 smithi122 ceph-mon[93869]: pgmap v17626: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 823 KiB/s wr, 14 op/s 2023-03-09T23:53:02.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:02 smithi181 ceph-mon[94189]: pgmap v17626: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 823 KiB/s wr, 14 op/s 2023-03-09T23:53:02.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:02 smithi038 ceph-mon[92422]: pgmap v17626: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 823 KiB/s wr, 14 op/s 2023-03-09T23:53:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32555.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:53:04.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:03 smithi122 ceph-mon[93869]: pgmap v17627: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 618 KiB/s wr, 13 op/s 2023-03-09T23:53:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:03 smithi181 ceph-mon[94189]: pgmap v17627: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 618 KiB/s wr, 13 op/s 2023-03-09T23:53:04.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:03 smithi038 ceph-mon[92422]: pgmap v17627: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 618 KiB/s wr, 13 op/s 2023-03-09T23:53:06.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:06 smithi122 ceph-mon[93869]: pgmap v17628: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 730 KiB/s wr, 13 op/s 2023-03-09T23:53:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32096.439957 secs 2023-03-09T23:53:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:06 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:53:06.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:06 smithi181 ceph-mon[94189]: pgmap v17628: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 730 KiB/s wr, 13 op/s 2023-03-09T23:53:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32096.439957 secs 2023-03-09T23:53:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:06 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:53:06.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:06 smithi038 ceph-mon[92422]: pgmap v17628: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 730 KiB/s wr, 13 op/s 2023-03-09T23:53:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32096.439957 secs 2023-03-09T23:53:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:06 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:53:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:07 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:53:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:53:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:53:07.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:07 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:53:07.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:53:07.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:53:07.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:07 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:53:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:53:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:53:08.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:08 smithi122 ceph-mon[93869]: pgmap v17629: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 795 KiB/s rd, 802 KiB/s wr, 18 op/s 2023-03-09T23:53:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:53:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:53:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32098.859611 secs 2023-03-09T23:53:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:08 smithi181 ceph-mon[94189]: pgmap v17629: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 795 KiB/s rd, 802 KiB/s wr, 18 op/s 2023-03-09T23:53:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:53:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:53:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32098.859611 secs 2023-03-09T23:53:08.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:08 smithi038 ceph-mon[92422]: pgmap v17629: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 795 KiB/s rd, 802 KiB/s wr, 18 op/s 2023-03-09T23:53:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:53:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:53:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32098.859611 secs 2023-03-09T23:53:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32560.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:53:09.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:53:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:53:09.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:53:09.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:53:09.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:53:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:53:10.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:10 smithi122 ceph-mon[93869]: pgmap v17630: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 575 KiB/s wr, 26 op/s 2023-03-09T23:53:10.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:10 smithi181 ceph-mon[94189]: pgmap v17630: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 575 KiB/s wr, 26 op/s 2023-03-09T23:53:10.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:10 smithi038 ceph-mon[92422]: pgmap v17630: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 575 KiB/s wr, 26 op/s 2023-03-09T23:53:11.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32101.439983 secs 2023-03-09T23:53:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:11 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:53:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:11 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:53:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:11 smithi122 ceph-mon[93869]: osdmap e1317: 12 total, 12 up, 12 in 2023-03-09T23:53:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32101.439983 secs 2023-03-09T23:53:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:11 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:53:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:11 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:53:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:11 smithi181 ceph-mon[94189]: osdmap e1317: 12 total, 12 up, 12 in 2023-03-09T23:53:11.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32101.439983 secs 2023-03-09T23:53:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:11 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:53:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:11 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:53:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:11 smithi038 ceph-mon[92422]: osdmap e1317: 12 total, 12 up, 12 in 2023-03-09T23:53:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:12 smithi122 ceph-mon[93869]: pgmap v17632: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 693 KiB/s wr, 30 op/s 2023-03-09T23:53:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:12 smithi181 ceph-mon[94189]: pgmap v17632: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 693 KiB/s wr, 30 op/s 2023-03-09T23:53:12.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:12 smithi038 ceph-mon[92422]: pgmap v17632: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 693 KiB/s wr, 30 op/s 2023-03-09T23:53:13.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:13 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:53:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32103.859704 secs 2023-03-09T23:53:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:13 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:53:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:13 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:53:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:13 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:53:13.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:13 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:53:13.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32103.859704 secs 2023-03-09T23:53:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:13 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:53:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:13 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:53:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:13 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:53:13.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:13 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:53:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32103.859704 secs 2023-03-09T23:53:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:13 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:53:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:13 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:53:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:13 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:53:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32565.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:53:14.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:14 smithi122 ceph-mon[93869]: pgmap v17633: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 561 KiB/s rd, 603 KiB/s wr, 27 op/s 2023-03-09T23:53:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:14 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:53:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:14 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:53:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:14 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:53:14.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:14 smithi181 ceph-mon[94189]: pgmap v17633: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 561 KiB/s rd, 603 KiB/s wr, 27 op/s 2023-03-09T23:53:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:14 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:53:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:14 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:53:14.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:14 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:53:14.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:14 smithi038 ceph-mon[92422]: pgmap v17633: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 561 KiB/s rd, 603 KiB/s wr, 27 op/s 2023-03-09T23:53:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:14 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:53:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:14 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:53:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:14 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:53:16.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:16 smithi122 ceph-mon[93869]: pgmap v17634: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 469 KiB/s rd, 511 KiB/s wr, 28 op/s 2023-03-09T23:53:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32106.440061 secs 2023-03-09T23:53:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:53:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:53:16.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:16 smithi181 ceph-mon[94189]: pgmap v17634: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 469 KiB/s rd, 511 KiB/s wr, 28 op/s 2023-03-09T23:53:16.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32106.440061 secs 2023-03-09T23:53:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:53:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:53:16.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:16 smithi038 ceph-mon[92422]: pgmap v17634: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 469 KiB/s rd, 511 KiB/s wr, 28 op/s 2023-03-09T23:53:16.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32106.440061 secs 2023-03-09T23:53:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:53:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:53:17.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:53:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:53:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:53:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:53:17.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:53:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:53:18.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:18 smithi122 ceph-mon[93869]: pgmap v17635: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 426 KiB/s wr, 24 op/s 2023-03-09T23:53:18.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32108.859772 secs 2023-03-09T23:53:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:18 smithi122 ceph-mon[93869]: osdmap e1318: 12 total, 12 up, 12 in 2023-03-09T23:53:18.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:18 smithi181 ceph-mon[94189]: pgmap v17635: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 426 KiB/s wr, 24 op/s 2023-03-09T23:53:18.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32108.859772 secs 2023-03-09T23:53:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:18 smithi181 ceph-mon[94189]: osdmap e1318: 12 total, 12 up, 12 in 2023-03-09T23:53:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:18 smithi038 ceph-mon[92422]: pgmap v17635: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 426 KiB/s wr, 24 op/s 2023-03-09T23:53:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32108.859772 secs 2023-03-09T23:53:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:18 smithi038 ceph-mon[92422]: osdmap e1318: 12 total, 12 up, 12 in 2023-03-09T23:53:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32570.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:53:20.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:20 smithi122 ceph-mon[93869]: pgmap v17637: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 321 KiB/s wr, 10 op/s 2023-03-09T23:53:20.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:20 smithi181 ceph-mon[94189]: pgmap v17637: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 321 KiB/s wr, 10 op/s 2023-03-09T23:53:20.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:20 smithi038 ceph-mon[92422]: pgmap v17637: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 321 KiB/s wr, 10 op/s 2023-03-09T23:53:21.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32111.440126 secs 2023-03-09T23:53:21.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32111.440126 secs 2023-03-09T23:53:21.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32111.440126 secs 2023-03-09T23:53:22.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:22 smithi122 ceph-mon[93869]: pgmap v17638: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 285 KiB/s wr, 9 op/s 2023-03-09T23:53:22.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:22 smithi181 ceph-mon[94189]: pgmap v17638: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 285 KiB/s wr, 9 op/s 2023-03-09T23:53:22.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:22 smithi038 ceph-mon[92422]: pgmap v17638: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 285 KiB/s wr, 9 op/s 2023-03-09T23:53:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32575.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:53:24.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32113.859891 secs 2023-03-09T23:53:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:23 smithi122 ceph-mon[93869]: pgmap v17639: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 161 KiB/s wr, 8 op/s 2023-03-09T23:53:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32113.859891 secs 2023-03-09T23:53:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:23 smithi181 ceph-mon[94189]: pgmap v17639: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 161 KiB/s wr, 8 op/s 2023-03-09T23:53:24.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32113.859891 secs 2023-03-09T23:53:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:23 smithi038 ceph-mon[92422]: pgmap v17639: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 161 KiB/s wr, 8 op/s 2023-03-09T23:53:26.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:26 smithi122 ceph-mon[93869]: pgmap v17640: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 118 KiB/s wr, 7 op/s 2023-03-09T23:53:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32116.440185 secs 2023-03-09T23:53:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:26 smithi181 ceph-mon[94189]: pgmap v17640: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 118 KiB/s wr, 7 op/s 2023-03-09T23:53:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32116.440185 secs 2023-03-09T23:53:26.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:26 smithi038 ceph-mon[92422]: pgmap v17640: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 118 KiB/s wr, 7 op/s 2023-03-09T23:53:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32116.440185 secs 2023-03-09T23:53:28.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:28 smithi122 ceph-mon[93869]: pgmap v17641: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 118 KiB/s wr, 7 op/s 2023-03-09T23:53:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32118.860007 secs 2023-03-09T23:53:28.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:28 smithi181 ceph-mon[94189]: pgmap v17641: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 118 KiB/s wr, 7 op/s 2023-03-09T23:53:28.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32118.860007 secs 2023-03-09T23:53:28.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:28 smithi038 ceph-mon[92422]: pgmap v17641: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 118 KiB/s wr, 7 op/s 2023-03-09T23:53:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32118.860007 secs 2023-03-09T23:53:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32580.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:53:30.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:30 smithi122 ceph-mon[93869]: pgmap v17642: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 106 KiB/s wr, 7 op/s 2023-03-09T23:53:30.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:30 smithi181 ceph-mon[94189]: pgmap v17642: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 106 KiB/s wr, 7 op/s 2023-03-09T23:53:30.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:30 smithi038 ceph-mon[92422]: pgmap v17642: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 106 KiB/s wr, 7 op/s 2023-03-09T23:53:31.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32121.440247 secs 2023-03-09T23:53:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32121.440247 secs 2023-03-09T23:53:31.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32121.440247 secs 2023-03-09T23:53:32.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:32 smithi122 ceph-mon[93869]: pgmap v17643: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T23:53:32.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:32 smithi181 ceph-mon[94189]: pgmap v17643: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T23:53:32.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:32 smithi038 ceph-mon[92422]: pgmap v17643: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T23:53:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32585.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:53:34.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32123.860125 secs 2023-03-09T23:53:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:33 smithi122 ceph-mon[93869]: pgmap v17644: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 6 op/s 2023-03-09T23:53:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32123.860125 secs 2023-03-09T23:53:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:33 smithi181 ceph-mon[94189]: pgmap v17644: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 6 op/s 2023-03-09T23:53:34.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32123.860125 secs 2023-03-09T23:53:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:33 smithi038 ceph-mon[92422]: pgmap v17644: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 6 op/s 2023-03-09T23:53:36.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:36 smithi122 ceph-mon[93869]: pgmap v17645: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 6 op/s 2023-03-09T23:53:36.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32126.440309 secs 2023-03-09T23:53:36.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:36 smithi181 ceph-mon[94189]: pgmap v17645: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 6 op/s 2023-03-09T23:53:36.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32126.440309 secs 2023-03-09T23:53:36.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:36 smithi038 ceph-mon[92422]: pgmap v17645: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 6 op/s 2023-03-09T23:53:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32126.440309 secs 2023-03-09T23:53:38.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:38 smithi122 ceph-mon[93869]: pgmap v17646: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 8 op/s 2023-03-09T23:53:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32128.860280 secs 2023-03-09T23:53:38.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:38 smithi181 ceph-mon[94189]: pgmap v17646: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 8 op/s 2023-03-09T23:53:38.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32128.860280 secs 2023-03-09T23:53:38.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:38 smithi038 ceph-mon[92422]: pgmap v17646: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 426 B/s wr, 8 op/s 2023-03-09T23:53:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32128.860280 secs 2023-03-09T23:53:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32590.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:53:40.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:40 smithi122 ceph-mon[93869]: pgmap v17647: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 124 KiB/s wr, 7 op/s 2023-03-09T23:53:40.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:40 smithi181 ceph-mon[94189]: pgmap v17647: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 124 KiB/s wr, 7 op/s 2023-03-09T23:53:40.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:40 smithi038 ceph-mon[92422]: pgmap v17647: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 124 KiB/s wr, 7 op/s 2023-03-09T23:53:41.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32131.440368 secs 2023-03-09T23:53:41.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32131.440368 secs 2023-03-09T23:53:41.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32131.440368 secs 2023-03-09T23:53:42.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:42 smithi122 ceph-mon[93869]: pgmap v17648: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 124 KiB/s wr, 9 op/s 2023-03-09T23:53:42.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:42 smithi181 ceph-mon[94189]: pgmap v17648: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 124 KiB/s wr, 9 op/s 2023-03-09T23:53:42.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:42 smithi038 ceph-mon[92422]: pgmap v17648: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 124 KiB/s wr, 9 op/s 2023-03-09T23:53:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32595.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:53:44.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:43 smithi122 ceph-mon[93869]: pgmap v17649: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 140 KiB/s wr, 7 op/s 2023-03-09T23:53:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:43 smithi181 ceph-mon[94189]: pgmap v17649: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 140 KiB/s wr, 7 op/s 2023-03-09T23:53:44.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:43 smithi038 ceph-mon[92422]: pgmap v17649: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 140 KiB/s wr, 7 op/s 2023-03-09T23:53:46.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:46 smithi122 ceph-mon[93869]: pgmap v17650: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 140 KiB/s wr, 7 op/s 2023-03-09T23:53:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32136.440430 secs 2023-03-09T23:53:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:46 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:53:46.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:46 smithi181 ceph-mon[94189]: pgmap v17650: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 140 KiB/s wr, 7 op/s 2023-03-09T23:53:46.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32136.440430 secs 2023-03-09T23:53:46.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:46 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:53:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:46 smithi038 ceph-mon[92422]: pgmap v17650: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 140 KiB/s wr, 7 op/s 2023-03-09T23:53:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32136.440430 secs 2023-03-09T23:53:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:46 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:53:47.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:47 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:53:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:53:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:53:47.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:47 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:53:47.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:53:47.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:53:47.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:47 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:53:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:53:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:53:48.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:48 smithi122 ceph-mon[93869]: pgmap v17651: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 245 KiB/s wr, 9 op/s 2023-03-09T23:53:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:48 smithi181 ceph-mon[94189]: pgmap v17651: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 245 KiB/s wr, 9 op/s 2023-03-09T23:53:48.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:48 smithi038 ceph-mon[92422]: pgmap v17651: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 245 KiB/s wr, 9 op/s 2023-03-09T23:53:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32600.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:53:50.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:50 smithi122 ceph-mon[93869]: pgmap v17652: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 245 KiB/s wr, 7 op/s 2023-03-09T23:53:50.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:50 smithi181 ceph-mon[94189]: pgmap v17652: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 245 KiB/s wr, 7 op/s 2023-03-09T23:53:50.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:50 smithi038 ceph-mon[92422]: pgmap v17652: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 245 KiB/s wr, 7 op/s 2023-03-09T23:53:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32141.440489 secs 2023-03-09T23:53:51.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32141.440489 secs 2023-03-09T23:53:51.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32141.440489 secs 2023-03-09T23:53:52.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:52 smithi122 ceph-mon[93869]: pgmap v17653: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 752 KiB/s wr, 11 op/s 2023-03-09T23:53:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:53:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:53:52.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:52 smithi181 ceph-mon[94189]: pgmap v17653: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 752 KiB/s wr, 11 op/s 2023-03-09T23:53:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:53:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:53:52.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:52 smithi038 ceph-mon[92422]: pgmap v17653: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 752 KiB/s wr, 11 op/s 2023-03-09T23:53:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:53:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:53:53.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:53:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:53:53.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:53:53.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:53:53.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:53:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:53:53.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32605.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:53:54.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:54 smithi122 ceph-mon[93869]: pgmap v17654: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 752 KiB/s wr, 10 op/s 2023-03-09T23:53:54.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:54 smithi181 ceph-mon[94189]: pgmap v17654: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 752 KiB/s wr, 10 op/s 2023-03-09T23:53:54.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:54 smithi038 ceph-mon[92422]: pgmap v17654: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 752 KiB/s wr, 10 op/s 2023-03-09T23:53:56.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:56 smithi122 ceph-mon[93869]: pgmap v17655: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 903 KiB/s wr, 10 op/s 2023-03-09T23:53:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32146.440549 secs 2023-03-09T23:53:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:53:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:53:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:56 smithi181 ceph-mon[94189]: pgmap v17655: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 903 KiB/s wr, 10 op/s 2023-03-09T23:53:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32146.440549 secs 2023-03-09T23:53:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:53:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:53:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:56 smithi038 ceph-mon[92422]: pgmap v17655: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 903 KiB/s wr, 10 op/s 2023-03-09T23:53:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32146.440549 secs 2023-03-09T23:53:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:53:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:53:57.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:53:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:53:57.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:53:57.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:53:57.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:53:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:53:58.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:53:58 smithi122 ceph-mon[93869]: pgmap v17656: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 931 KiB/s wr, 14 op/s 2023-03-09T23:53:58.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:53:58 smithi181 ceph-mon[94189]: pgmap v17656: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 931 KiB/s wr, 14 op/s 2023-03-09T23:53:58.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:53:58 smithi038 ceph-mon[92422]: pgmap v17656: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 931 KiB/s wr, 14 op/s 2023-03-09T23:53:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32610.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:54:00.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:00 smithi122 ceph-mon[93869]: pgmap v17657: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1000 KiB/s wr, 12 op/s 2023-03-09T23:54:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:54:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:54:00.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:00 smithi181 ceph-mon[94189]: pgmap v17657: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1000 KiB/s wr, 12 op/s 2023-03-09T23:54:00.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:54:00.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:54:00.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:00 smithi038 ceph-mon[92422]: pgmap v17657: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1000 KiB/s wr, 12 op/s 2023-03-09T23:54:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:54:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:54:01.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32151.440611 secs 2023-03-09T23:54:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:54:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:54:01.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32151.440611 secs 2023-03-09T23:54:01.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:54:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:54:01.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32151.440611 secs 2023-03-09T23:54:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:54:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:54:02.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:02 smithi122 ceph-mon[93869]: pgmap v17658: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 18 op/s 2023-03-09T23:54:02.490 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:02 smithi181 ceph-mon[94189]: pgmap v17658: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 18 op/s 2023-03-09T23:54:02.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:02 smithi038 ceph-mon[92422]: pgmap v17658: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 18 op/s 2023-03-09T23:54:03.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32615.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:54:04.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32153.860891 secs 2023-03-09T23:54:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:03 smithi122 ceph-mon[93869]: pgmap v17659: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 661 KiB/s rd, 686 KiB/s wr, 29 op/s 2023-03-09T23:54:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32153.860891 secs 2023-03-09T23:54:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:03 smithi181 ceph-mon[94189]: pgmap v17659: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 661 KiB/s rd, 686 KiB/s wr, 29 op/s 2023-03-09T23:54:04.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32153.860891 secs 2023-03-09T23:54:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:03 smithi038 ceph-mon[92422]: pgmap v17659: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 661 KiB/s rd, 686 KiB/s wr, 29 op/s 2023-03-09T23:54:06.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:06 smithi122 ceph-mon[93869]: pgmap v17660: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 681 KiB/s rd, 707 KiB/s wr, 33 op/s 2023-03-09T23:54:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32156.440676 secs 2023-03-09T23:54:06.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:06 smithi181 ceph-mon[94189]: pgmap v17660: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 681 KiB/s rd, 707 KiB/s wr, 33 op/s 2023-03-09T23:54:06.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32156.440676 secs 2023-03-09T23:54:06.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:06 smithi038 ceph-mon[92422]: pgmap v17660: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 681 KiB/s rd, 707 KiB/s wr, 33 op/s 2023-03-09T23:54:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32156.440676 secs 2023-03-09T23:54:07.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:07 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:54:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:54:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:54:07.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:07 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:54:07.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:54:07.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:54:07.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:07 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:54:07.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:54:07.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:54:08.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:08 smithi122 ceph-mon[93869]: pgmap v17661: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 547 KiB/s wr, 50 op/s 2023-03-09T23:54:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32158.860991 secs 2023-03-09T23:54:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:08 smithi181 ceph-mon[94189]: pgmap v17661: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 547 KiB/s wr, 50 op/s 2023-03-09T23:54:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32158.860991 secs 2023-03-09T23:54:08.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:08 smithi038 ceph-mon[92422]: pgmap v17661: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 547 KiB/s wr, 50 op/s 2023-03-09T23:54:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32158.860991 secs 2023-03-09T23:54:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32620.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:54:10.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:10 smithi122 ceph-mon[93869]: pgmap v17662: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 520 KiB/s wr, 47 op/s 2023-03-09T23:54:10.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:10 smithi181 ceph-mon[94189]: pgmap v17662: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 520 KiB/s wr, 47 op/s 2023-03-09T23:54:10.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:10 smithi038 ceph-mon[92422]: pgmap v17662: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 520 KiB/s wr, 47 op/s 2023-03-09T23:54:11.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32161.440736 secs 2023-03-09T23:54:11.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:11 smithi122 ceph-mon[93869]: osdmap e1319: 12 total, 12 up, 12 in 2023-03-09T23:54:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32161.440736 secs 2023-03-09T23:54:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:11 smithi181 ceph-mon[94189]: osdmap e1319: 12 total, 12 up, 12 in 2023-03-09T23:54:11.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32161.440736 secs 2023-03-09T23:54:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:11 smithi038 ceph-mon[92422]: osdmap e1319: 12 total, 12 up, 12 in 2023-03-09T23:54:12.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:12 smithi122 ceph-mon[93869]: pgmap v17664: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 188 KiB/s wr, 51 op/s 2023-03-09T23:54:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:54:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:54:12.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:12 smithi181 ceph-mon[94189]: pgmap v17664: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 188 KiB/s wr, 51 op/s 2023-03-09T23:54:12.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:54:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:54:12.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:12 smithi038 ceph-mon[92422]: pgmap v17664: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 188 KiB/s wr, 51 op/s 2023-03-09T23:54:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:54:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:54:13.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:54:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:54:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32163.861063 secs 2023-03-09T23:54:13.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:54:13.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:54:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32163.861063 secs 2023-03-09T23:54:13.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:54:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:54:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32163.861063 secs 2023-03-09T23:54:13.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32625.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:54:14.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:14 smithi122 ceph-mon[93869]: pgmap v17665: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 360 KiB/s wr, 32 op/s 2023-03-09T23:54:14.516 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:14 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:54:14.517 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:14 smithi038 ceph-mon[92422]: pgmap v17665: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 360 KiB/s wr, 32 op/s 2023-03-09T23:54:14.517 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:14 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:54:14.517 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:14 smithi181 ceph-mon[94189]: pgmap v17665: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 360 KiB/s wr, 32 op/s 2023-03-09T23:54:14.517 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:14 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:54:15.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:54:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:54:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:54:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:54:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:54:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:54:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:54:15.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:54:15.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:54:15.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:54:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:54:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:54:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:54:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:54:15.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:54:15.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:54:15.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:54:15.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:54:15.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:54:15.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:54:15.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:54:16.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:16 smithi122 ceph-mon[93869]: pgmap v17666: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 335 KiB/s wr, 27 op/s 2023-03-09T23:54:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32166.440797 secs 2023-03-09T23:54:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:16 smithi181 ceph-mon[94189]: pgmap v17666: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 335 KiB/s wr, 27 op/s 2023-03-09T23:54:16.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32166.440797 secs 2023-03-09T23:54:16.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:16 smithi038 ceph-mon[92422]: pgmap v17666: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 335 KiB/s wr, 27 op/s 2023-03-09T23:54:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32166.440797 secs 2023-03-09T23:54:18.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:18 smithi122 ceph-mon[93869]: pgmap v17667: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 446 KiB/s rd, 444 KiB/s wr, 9 op/s 2023-03-09T23:54:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32168.861174 secs 2023-03-09T23:54:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:18 smithi122 ceph-mon[93869]: osdmap e1320: 12 total, 12 up, 12 in 2023-03-09T23:54:18.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:18 smithi181 ceph-mon[94189]: pgmap v17667: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 446 KiB/s rd, 444 KiB/s wr, 9 op/s 2023-03-09T23:54:18.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32168.861174 secs 2023-03-09T23:54:18.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:18 smithi181 ceph-mon[94189]: osdmap e1320: 12 total, 12 up, 12 in 2023-03-09T23:54:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:18 smithi038 ceph-mon[92422]: pgmap v17667: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 446 KiB/s rd, 444 KiB/s wr, 9 op/s 2023-03-09T23:54:18.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32168.861174 secs 2023-03-09T23:54:18.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:18 smithi038 ceph-mon[92422]: osdmap e1320: 12 total, 12 up, 12 in 2023-03-09T23:54:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32630.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:54:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:20 smithi122 ceph-mon[93869]: pgmap v17669: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 497 KiB/s rd, 494 KiB/s wr, 8 op/s 2023-03-09T23:54:20.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:20 smithi181 ceph-mon[94189]: pgmap v17669: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 497 KiB/s rd, 494 KiB/s wr, 8 op/s 2023-03-09T23:54:20.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:20 smithi038 ceph-mon[92422]: pgmap v17669: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 497 KiB/s rd, 494 KiB/s wr, 8 op/s 2023-03-09T23:54:21.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32171.440859 secs 2023-03-09T23:54:21.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32171.440859 secs 2023-03-09T23:54:21.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32171.440859 secs 2023-03-09T23:54:22.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:22 smithi122 ceph-mon[93869]: pgmap v17670: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 317 KiB/s wr, 7 op/s 2023-03-09T23:54:22.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:22 smithi181 ceph-mon[94189]: pgmap v17670: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 317 KiB/s wr, 7 op/s 2023-03-09T23:54:22.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:22 smithi038 ceph-mon[92422]: pgmap v17670: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 317 KiB/s wr, 7 op/s 2023-03-09T23:54:23.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32635.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:54:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32173.861301 secs 2023-03-09T23:54:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:23 smithi122 ceph-mon[93869]: pgmap v17671: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 118 KiB/s wr, 6 op/s 2023-03-09T23:54:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32173.861301 secs 2023-03-09T23:54:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:23 smithi181 ceph-mon[94189]: pgmap v17671: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 118 KiB/s wr, 6 op/s 2023-03-09T23:54:24.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32173.861301 secs 2023-03-09T23:54:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:23 smithi038 ceph-mon[92422]: pgmap v17671: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 118 KiB/s wr, 6 op/s 2023-03-09T23:54:26.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:26 smithi122 ceph-mon[93869]: pgmap v17672: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 118 KiB/s wr, 6 op/s 2023-03-09T23:54:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32176.440921 secs 2023-03-09T23:54:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:26 smithi181 ceph-mon[94189]: pgmap v17672: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 118 KiB/s wr, 6 op/s 2023-03-09T23:54:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32176.440921 secs 2023-03-09T23:54:26.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:26 smithi038 ceph-mon[92422]: pgmap v17672: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 118 KiB/s wr, 6 op/s 2023-03-09T23:54:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32176.440921 secs 2023-03-09T23:54:28.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:28 smithi122 ceph-mon[93869]: pgmap v17673: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-09T23:54:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32178.861423 secs 2023-03-09T23:54:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:28 smithi181 ceph-mon[94189]: pgmap v17673: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-09T23:54:28.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32178.861423 secs 2023-03-09T23:54:28.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:28 smithi038 ceph-mon[92422]: pgmap v17673: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-09T23:54:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32178.861423 secs 2023-03-09T23:54:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32640.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:54:30.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:30 smithi122 ceph-mon[93869]: pgmap v17674: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2023-03-09T23:54:30.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:30 smithi181 ceph-mon[94189]: pgmap v17674: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2023-03-09T23:54:30.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:30 smithi038 ceph-mon[92422]: pgmap v17674: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2023-03-09T23:54:31.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32181.440979 secs 2023-03-09T23:54:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32181.440979 secs 2023-03-09T23:54:31.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32181.440979 secs 2023-03-09T23:54:32.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:32 smithi122 ceph-mon[93869]: pgmap v17675: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2023-03-09T23:54:32.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:32 smithi181 ceph-mon[94189]: pgmap v17675: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2023-03-09T23:54:32.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:32 smithi038 ceph-mon[92422]: pgmap v17675: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2023-03-09T23:54:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32645.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:54:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32183.861549 secs 2023-03-09T23:54:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:33 smithi122 ceph-mon[93869]: pgmap v17676: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 119 KiB/s wr, 6 op/s 2023-03-09T23:54:34.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32183.861549 secs 2023-03-09T23:54:34.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:33 smithi181 ceph-mon[94189]: pgmap v17676: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 119 KiB/s wr, 6 op/s 2023-03-09T23:54:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32183.861549 secs 2023-03-09T23:54:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:33 smithi038 ceph-mon[92422]: pgmap v17676: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 119 KiB/s wr, 6 op/s 2023-03-09T23:54:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:36 smithi122 ceph-mon[93869]: pgmap v17677: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 119 KiB/s wr, 6 op/s 2023-03-09T23:54:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32186.441041 secs 2023-03-09T23:54:36.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:36 smithi181 ceph-mon[94189]: pgmap v17677: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 119 KiB/s wr, 6 op/s 2023-03-09T23:54:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32186.441041 secs 2023-03-09T23:54:36.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:36 smithi038 ceph-mon[92422]: pgmap v17677: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 119 KiB/s wr, 6 op/s 2023-03-09T23:54:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32186.441041 secs 2023-03-09T23:54:38.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:38 smithi122 ceph-mon[93869]: pgmap v17678: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 119 KiB/s wr, 9 op/s 2023-03-09T23:54:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32188.861669 secs 2023-03-09T23:54:38.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:38 smithi181 ceph-mon[94189]: pgmap v17678: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 119 KiB/s wr, 9 op/s 2023-03-09T23:54:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32188.861669 secs 2023-03-09T23:54:38.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:38 smithi038 ceph-mon[92422]: pgmap v17678: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 119 KiB/s wr, 9 op/s 2023-03-09T23:54:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32188.861669 secs 2023-03-09T23:54:38.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32650.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:54:40.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:40 smithi122 ceph-mon[93869]: pgmap v17679: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 119 KiB/s wr, 6 op/s 2023-03-09T23:54:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:54:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:54:40.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:40 smithi181 ceph-mon[94189]: pgmap v17679: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 119 KiB/s wr, 6 op/s 2023-03-09T23:54:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:54:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:54:40.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:40 smithi038 ceph-mon[92422]: pgmap v17679: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 119 KiB/s wr, 6 op/s 2023-03-09T23:54:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:54:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:54:41.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32191.441102 secs 2023-03-09T23:54:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:54:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:54:41.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32191.441102 secs 2023-03-09T23:54:41.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:54:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:54:41.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32191.441102 secs 2023-03-09T23:54:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:54:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:54:42.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:42 smithi122 ceph-mon[93869]: pgmap v17680: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 120 KiB/s wr, 9 op/s 2023-03-09T23:54:42.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:42 smithi181 ceph-mon[94189]: pgmap v17680: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 120 KiB/s wr, 9 op/s 2023-03-09T23:54:42.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:42 smithi038 ceph-mon[92422]: pgmap v17680: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 120 KiB/s wr, 9 op/s 2023-03-09T23:54:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32655.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:54:44.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:43 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 32193.861748 secs 2023-03-09T23:54:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:43 smithi122 ceph-mon[93869]: pgmap v17681: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 243 KiB/s wr, 8 op/s 2023-03-09T23:54:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:43 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 32193.861748 secs 2023-03-09T23:54:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:43 smithi181 ceph-mon[94189]: pgmap v17681: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 243 KiB/s wr, 8 op/s 2023-03-09T23:54:44.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:43 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 32193.861748 secs 2023-03-09T23:54:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:43 smithi038 ceph-mon[92422]: pgmap v17681: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 243 KiB/s wr, 8 op/s 2023-03-09T23:54:46.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:46 smithi122 ceph-mon[93869]: pgmap v17682: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 125 KiB/s wr, 7 op/s 2023-03-09T23:54:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32196.441181 secs 2023-03-09T23:54:46.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:46 smithi181 ceph-mon[94189]: pgmap v17682: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 125 KiB/s wr, 7 op/s 2023-03-09T23:54:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32196.441181 secs 2023-03-09T23:54:46.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:46 smithi038 ceph-mon[92422]: pgmap v17682: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 125 KiB/s wr, 7 op/s 2023-03-09T23:54:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32196.441181 secs 2023-03-09T23:54:47.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:54:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:54:47.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:54:47.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:54:47.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:54:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:54:48.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:48 smithi122 ceph-mon[93869]: pgmap v17683: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 301 KiB/s wr, 9 op/s 2023-03-09T23:54:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:48 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 32198.861872 secs 2023-03-09T23:54:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:48 smithi181 ceph-mon[94189]: pgmap v17683: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 301 KiB/s wr, 9 op/s 2023-03-09T23:54:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:48 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 32198.861872 secs 2023-03-09T23:54:48.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:48 smithi038 ceph-mon[92422]: pgmap v17683: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 301 KiB/s wr, 9 op/s 2023-03-09T23:54:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:48 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 32198.861872 secs 2023-03-09T23:54:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32660.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:54:50.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:50 smithi122 ceph-mon[93869]: pgmap v17684: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 301 KiB/s wr, 7 op/s 2023-03-09T23:54:50.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:50 smithi181 ceph-mon[94189]: pgmap v17684: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 301 KiB/s wr, 7 op/s 2023-03-09T23:54:50.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:50 smithi038 ceph-mon[92422]: pgmap v17684: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 301 KiB/s wr, 7 op/s 2023-03-09T23:54:51.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32201.441298 secs 2023-03-09T23:54:51.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32201.441298 secs 2023-03-09T23:54:51.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32201.441298 secs 2023-03-09T23:54:52.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:52 smithi122 ceph-mon[93869]: pgmap v17685: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 719 KiB/s rd, 606 KiB/s wr, 10 op/s 2023-03-09T23:54:52.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:52 smithi181 ceph-mon[94189]: pgmap v17685: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 719 KiB/s rd, 606 KiB/s wr, 10 op/s 2023-03-09T23:54:52.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:52 smithi038 ceph-mon[92422]: pgmap v17685: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 719 KiB/s rd, 606 KiB/s wr, 10 op/s 2023-03-09T23:54:53.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:54:53.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:54:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:53 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 32203.862003 secs 2023-03-09T23:54:53.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:54:53.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:54:53.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:53 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 32203.862003 secs 2023-03-09T23:54:53.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:54:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:54:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:53 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 32203.862003 secs 2023-03-09T23:54:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32665.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:54:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:54 smithi122 ceph-mon[93869]: pgmap v17686: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 606 KiB/s wr, 9 op/s 2023-03-09T23:54:54.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:54 smithi181 ceph-mon[94189]: pgmap v17686: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 606 KiB/s wr, 9 op/s 2023-03-09T23:54:54.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:54 smithi038 ceph-mon[92422]: pgmap v17686: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 606 KiB/s wr, 9 op/s 2023-03-09T23:54:56.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:56 smithi122 ceph-mon[93869]: pgmap v17687: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 786 KiB/s rd, 511 KiB/s wr, 8 op/s 2023-03-09T23:54:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32206.441357 secs 2023-03-09T23:54:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:54:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:54:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:56 smithi181 ceph-mon[94189]: pgmap v17687: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 786 KiB/s rd, 511 KiB/s wr, 8 op/s 2023-03-09T23:54:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32206.441357 secs 2023-03-09T23:54:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:54:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:54:56.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:56 smithi038 ceph-mon[92422]: pgmap v17687: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 786 KiB/s rd, 511 KiB/s wr, 8 op/s 2023-03-09T23:54:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32206.441357 secs 2023-03-09T23:54:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:54:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:54:58.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:58 smithi122 ceph-mon[93869]: pgmap v17688: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 936 KiB/s rd, 657 KiB/s wr, 12 op/s 2023-03-09T23:54:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32208.862125 secs 2023-03-09T23:54:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:58 smithi181 ceph-mon[94189]: pgmap v17688: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 936 KiB/s rd, 657 KiB/s wr, 12 op/s 2023-03-09T23:54:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32208.862125 secs 2023-03-09T23:54:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:58 smithi038 ceph-mon[92422]: pgmap v17688: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 936 KiB/s rd, 657 KiB/s wr, 12 op/s 2023-03-09T23:54:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32208.862125 secs 2023-03-09T23:54:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32670.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:54:59.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:54:59.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:54:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:54:59.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:54:59.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:54:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:54:59.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:54:59.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:54:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:55:00.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:00 smithi122 ceph-mon[93869]: pgmap v17689: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 688 KiB/s rd, 510 KiB/s wr, 11 op/s 2023-03-09T23:55:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:55:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:55:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:00 smithi181 ceph-mon[94189]: pgmap v17689: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 688 KiB/s rd, 510 KiB/s wr, 11 op/s 2023-03-09T23:55:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:55:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:55:00.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:00 smithi038 ceph-mon[92422]: pgmap v17689: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 688 KiB/s rd, 510 KiB/s wr, 11 op/s 2023-03-09T23:55:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:55:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:55:01.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32211.441418 secs 2023-03-09T23:55:01.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:55:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:55:01.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32211.441418 secs 2023-03-09T23:55:01.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:55:01.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:55:01.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32211.441418 secs 2023-03-09T23:55:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:55:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:55:02.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:02 smithi122 ceph-mon[93869]: pgmap v17690: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 921 KiB/s rd, 742 KiB/s wr, 14 op/s 2023-03-09T23:55:02.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:02 smithi181 ceph-mon[94189]: pgmap v17690: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 921 KiB/s rd, 742 KiB/s wr, 14 op/s 2023-03-09T23:55:02.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:02 smithi038 ceph-mon[92422]: pgmap v17690: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 921 KiB/s rd, 742 KiB/s wr, 14 op/s 2023-03-09T23:55:03.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32675.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:55:04.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32213.862253 secs 2023-03-09T23:55:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:03 smithi122 ceph-mon[93869]: pgmap v17691: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 676 KiB/s rd, 498 KiB/s wr, 13 op/s 2023-03-09T23:55:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32213.862253 secs 2023-03-09T23:55:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:03 smithi181 ceph-mon[94189]: pgmap v17691: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 676 KiB/s rd, 498 KiB/s wr, 13 op/s 2023-03-09T23:55:04.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32213.862253 secs 2023-03-09T23:55:04.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:03 smithi038 ceph-mon[92422]: pgmap v17691: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 676 KiB/s rd, 498 KiB/s wr, 13 op/s 2023-03-09T23:55:06.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32216.441476 secs 2023-03-09T23:55:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:06 smithi122 ceph-mon[93869]: pgmap v17692: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 561 KiB/s rd, 546 KiB/s wr, 13 op/s 2023-03-09T23:55:06.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32216.441476 secs 2023-03-09T23:55:06.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:06 smithi181 ceph-mon[94189]: pgmap v17692: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 561 KiB/s rd, 546 KiB/s wr, 13 op/s 2023-03-09T23:55:06.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32216.441476 secs 2023-03-09T23:55:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:06 smithi038 ceph-mon[92422]: pgmap v17692: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 561 KiB/s rd, 546 KiB/s wr, 13 op/s 2023-03-09T23:55:08.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:08 smithi122 ceph-mon[93869]: pgmap v17693: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 705 KiB/s wr, 18 op/s 2023-03-09T23:55:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:55:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:55:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32218.862371 secs 2023-03-09T23:55:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:08 smithi181 ceph-mon[94189]: pgmap v17693: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 705 KiB/s wr, 18 op/s 2023-03-09T23:55:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:55:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:55:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32218.862371 secs 2023-03-09T23:55:08.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:08 smithi038 ceph-mon[92422]: pgmap v17693: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 705 KiB/s wr, 18 op/s 2023-03-09T23:55:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:55:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:55:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32218.862371 secs 2023-03-09T23:55:08.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32680.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:55:09.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:55:09.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:55:09.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:55:09.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:55:09.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:55:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:55:10.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:10 smithi122 ceph-mon[93869]: pgmap v17694: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 613 KiB/s rd, 638 KiB/s wr, 26 op/s 2023-03-09T23:55:10.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:10 smithi181 ceph-mon[94189]: pgmap v17694: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 613 KiB/s rd, 638 KiB/s wr, 26 op/s 2023-03-09T23:55:10.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:10 smithi038 ceph-mon[92422]: pgmap v17694: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 613 KiB/s rd, 638 KiB/s wr, 26 op/s 2023-03-09T23:55:11.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32221.441560 secs 2023-03-09T23:55:11.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:11 smithi122 ceph-mon[93869]: osdmap e1321: 12 total, 12 up, 12 in 2023-03-09T23:55:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32221.441560 secs 2023-03-09T23:55:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:11 smithi181 ceph-mon[94189]: osdmap e1321: 12 total, 12 up, 12 in 2023-03-09T23:55:11.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32221.441560 secs 2023-03-09T23:55:11.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:11 smithi038 ceph-mon[92422]: osdmap e1321: 12 total, 12 up, 12 in 2023-03-09T23:55:12.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:12 smithi122 ceph-mon[93869]: pgmap v17696: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 451 KiB/s wr, 28 op/s 2023-03-09T23:55:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:55:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:55:12.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:12 smithi181 ceph-mon[94189]: pgmap v17696: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 451 KiB/s wr, 28 op/s 2023-03-09T23:55:12.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:55:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:55:12.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:12 smithi038 ceph-mon[92422]: pgmap v17696: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 451 KiB/s wr, 28 op/s 2023-03-09T23:55:12.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:55:12.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:55:13.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:55:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:55:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32223.862489 secs 2023-03-09T23:55:13.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:55:13.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:55:13.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32223.862489 secs 2023-03-09T23:55:13.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:55:13.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:55:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32223.862489 secs 2023-03-09T23:55:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32685.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:55:14.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:14 smithi122 ceph-mon[93869]: pgmap v17697: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 478 KiB/s rd, 519 KiB/s wr, 27 op/s 2023-03-09T23:55:14.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:14 smithi181 ceph-mon[94189]: pgmap v17697: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 478 KiB/s rd, 519 KiB/s wr, 27 op/s 2023-03-09T23:55:14.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:14 smithi038 ceph-mon[92422]: pgmap v17697: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 478 KiB/s rd, 519 KiB/s wr, 27 op/s 2023-03-09T23:55:15.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:55:15.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:55:15.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:55:16.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32226.441622 secs 2023-03-09T23:55:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:16 smithi122 ceph-mon[93869]: pgmap v17698: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 629 KiB/s wr, 27 op/s 2023-03-09T23:55:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:16 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:55:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:16 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:55:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:16 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:55:16.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32226.441622 secs 2023-03-09T23:55:16.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:16 smithi181 ceph-mon[94189]: pgmap v17698: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 629 KiB/s wr, 27 op/s 2023-03-09T23:55:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:16 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:55:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:16 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:55:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:16 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:55:16.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32226.441622 secs 2023-03-09T23:55:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:16 smithi038 ceph-mon[92422]: pgmap v17698: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 629 KiB/s wr, 27 op/s 2023-03-09T23:55:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:16 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:55:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:16 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:55:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:16 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:55:18.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:18 smithi122 ceph-mon[93869]: pgmap v17699: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 406 KiB/s wr, 23 op/s 2023-03-09T23:55:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32228.862631 secs 2023-03-09T23:55:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:18 smithi122 ceph-mon[93869]: osdmap e1322: 12 total, 12 up, 12 in 2023-03-09T23:55:18.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:18 smithi181 ceph-mon[94189]: pgmap v17699: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 406 KiB/s wr, 23 op/s 2023-03-09T23:55:18.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32228.862631 secs 2023-03-09T23:55:18.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:18 smithi181 ceph-mon[94189]: osdmap e1322: 12 total, 12 up, 12 in 2023-03-09T23:55:18.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:18 smithi038 ceph-mon[92422]: pgmap v17699: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 406 KiB/s wr, 23 op/s 2023-03-09T23:55:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32228.862631 secs 2023-03-09T23:55:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:18 smithi038 ceph-mon[92422]: osdmap e1322: 12 total, 12 up, 12 in 2023-03-09T23:55:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32690.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:55:20.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:20 smithi122 ceph-mon[93869]: pgmap v17701: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 483 KiB/s rd, 481 KiB/s wr, 11 op/s 2023-03-09T23:55:20.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:20 smithi181 ceph-mon[94189]: pgmap v17701: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 483 KiB/s rd, 481 KiB/s wr, 11 op/s 2023-03-09T23:55:20.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:20 smithi038 ceph-mon[92422]: pgmap v17701: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 483 KiB/s rd, 481 KiB/s wr, 11 op/s 2023-03-09T23:55:21.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32231.441683 secs 2023-03-09T23:55:21.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32231.441683 secs 2023-03-09T23:55:21.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32231.441683 secs 2023-03-09T23:55:22.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:22 smithi122 ceph-mon[93869]: pgmap v17702: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 430 KiB/s rd, 428 KiB/s wr, 9 op/s 2023-03-09T23:55:22.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:22 smithi181 ceph-mon[94189]: pgmap v17702: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 430 KiB/s rd, 428 KiB/s wr, 9 op/s 2023-03-09T23:55:22.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:22 smithi038 ceph-mon[92422]: pgmap v17702: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 430 KiB/s rd, 428 KiB/s wr, 9 op/s 2023-03-09T23:55:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32695.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:55:24.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32233.862766 secs 2023-03-09T23:55:24.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:23 smithi122 ceph-mon[93869]: pgmap v17703: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 287 KiB/s wr, 7 op/s 2023-03-09T23:55:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32233.862766 secs 2023-03-09T23:55:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:23 smithi181 ceph-mon[94189]: pgmap v17703: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 287 KiB/s wr, 7 op/s 2023-03-09T23:55:24.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32233.862766 secs 2023-03-09T23:55:24.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:23 smithi038 ceph-mon[92422]: pgmap v17703: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 287 KiB/s wr, 7 op/s 2023-03-09T23:55:26.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32236.441743 secs 2023-03-09T23:55:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:26 smithi122 ceph-mon[93869]: pgmap v17704: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 118 KiB/s wr, 6 op/s 2023-03-09T23:55:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32236.441743 secs 2023-03-09T23:55:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:26 smithi181 ceph-mon[94189]: pgmap v17704: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 118 KiB/s wr, 6 op/s 2023-03-09T23:55:26.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32236.441743 secs 2023-03-09T23:55:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:26 smithi038 ceph-mon[92422]: pgmap v17704: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 118 KiB/s wr, 6 op/s 2023-03-09T23:55:28.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:28 smithi122 ceph-mon[93869]: pgmap v17705: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 118 KiB/s wr, 6 op/s 2023-03-09T23:55:28.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32238.862915 secs 2023-03-09T23:55:28.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:28 smithi181 ceph-mon[94189]: pgmap v17705: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 118 KiB/s wr, 6 op/s 2023-03-09T23:55:28.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32238.862915 secs 2023-03-09T23:55:28.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:28 smithi038 ceph-mon[92422]: pgmap v17705: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 118 KiB/s wr, 6 op/s 2023-03-09T23:55:28.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32238.862915 secs 2023-03-09T23:55:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32700.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:55:30.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:30 smithi122 ceph-mon[93869]: pgmap v17706: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 106 KiB/s wr, 6 op/s 2023-03-09T23:55:30.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:30 smithi181 ceph-mon[94189]: pgmap v17706: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 106 KiB/s wr, 6 op/s 2023-03-09T23:55:30.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:30 smithi038 ceph-mon[92422]: pgmap v17706: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 106 KiB/s wr, 6 op/s 2023-03-09T23:55:31.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32241.441812 secs 2023-03-09T23:55:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32241.441812 secs 2023-03-09T23:55:31.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32241.441812 secs 2023-03-09T23:55:32.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:32 smithi122 ceph-mon[93869]: pgmap v17707: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-03-09T23:55:32.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:32 smithi181 ceph-mon[94189]: pgmap v17707: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-03-09T23:55:32.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:32 smithi038 ceph-mon[92422]: pgmap v17707: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-03-09T23:55:33.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32705.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:55:34.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32243.863012 secs 2023-03-09T23:55:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:33 smithi122 ceph-mon[93869]: pgmap v17708: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T23:55:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32243.863012 secs 2023-03-09T23:55:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:33 smithi181 ceph-mon[94189]: pgmap v17708: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T23:55:34.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32243.863012 secs 2023-03-09T23:55:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:33 smithi038 ceph-mon[92422]: pgmap v17708: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-09T23:55:36.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32246.441872 secs 2023-03-09T23:55:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:36 smithi122 ceph-mon[93869]: pgmap v17709: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T23:55:36.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32246.441872 secs 2023-03-09T23:55:36.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:36 smithi181 ceph-mon[94189]: pgmap v17709: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T23:55:36.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32246.441872 secs 2023-03-09T23:55:36.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:36 smithi038 ceph-mon[92422]: pgmap v17709: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T23:55:38.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:38 smithi122 ceph-mon[93869]: pgmap v17710: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-03-09T23:55:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32248.863144 secs 2023-03-09T23:55:38.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:38 smithi181 ceph-mon[94189]: pgmap v17710: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-03-09T23:55:38.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32248.863144 secs 2023-03-09T23:55:38.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:38 smithi038 ceph-mon[92422]: pgmap v17710: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-03-09T23:55:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32248.863144 secs 2023-03-09T23:55:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32710.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:55:40.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:40 smithi122 ceph-mon[93869]: pgmap v17711: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 6 op/s 2023-03-09T23:55:40.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:40 smithi181 ceph-mon[94189]: pgmap v17711: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 6 op/s 2023-03-09T23:55:40.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:40 smithi038 ceph-mon[92422]: pgmap v17711: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 6 op/s 2023-03-09T23:55:41.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32251.441932 secs 2023-03-09T23:55:41.489 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32251.441932 secs 2023-03-09T23:55:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32251.441932 secs 2023-03-09T23:55:42.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:42 smithi122 ceph-mon[93869]: pgmap v17712: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 341 B/s wr, 7 op/s 2023-03-09T23:55:42.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:42 smithi181 ceph-mon[94189]: pgmap v17712: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 341 B/s wr, 7 op/s 2023-03-09T23:55:42.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:42 smithi038 ceph-mon[92422]: pgmap v17712: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 341 B/s wr, 7 op/s 2023-03-09T23:55:43.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32715.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:55:44.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32253.863283 secs 2023-03-09T23:55:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:43 smithi122 ceph-mon[93869]: pgmap v17713: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T23:55:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32253.863283 secs 2023-03-09T23:55:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:43 smithi181 ceph-mon[94189]: pgmap v17713: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T23:55:44.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32253.863283 secs 2023-03-09T23:55:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:43 smithi038 ceph-mon[92422]: pgmap v17713: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T23:55:46.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32256.441985 secs 2023-03-09T23:55:46.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:46 smithi122 ceph-mon[93869]: pgmap v17714: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 246 KiB/s wr, 7 op/s 2023-03-09T23:55:46.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32256.441985 secs 2023-03-09T23:55:46.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:46 smithi181 ceph-mon[94189]: pgmap v17714: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 246 KiB/s wr, 7 op/s 2023-03-09T23:55:46.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32256.441985 secs 2023-03-09T23:55:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:46 smithi038 ceph-mon[92422]: pgmap v17714: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 246 KiB/s wr, 7 op/s 2023-03-09T23:55:48.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:48 smithi122 ceph-mon[93869]: pgmap v17715: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 247 KiB/s wr, 9 op/s 2023-03-09T23:55:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:55:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:55:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:48 smithi181 ceph-mon[94189]: pgmap v17715: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 247 KiB/s wr, 9 op/s 2023-03-09T23:55:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:55:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:55:48.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:48 smithi038 ceph-mon[92422]: pgmap v17715: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 247 KiB/s wr, 9 op/s 2023-03-09T23:55:48.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:55:48.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:55:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32720.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:55:49.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:55:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:55:49.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:55:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:55:49.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:55:49.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:55:50.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:50 smithi122 ceph-mon[93869]: pgmap v17716: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 275 KiB/s wr, 9 op/s 2023-03-09T23:55:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:50 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:55:50.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:50 smithi181 ceph-mon[94189]: pgmap v17716: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 275 KiB/s wr, 9 op/s 2023-03-09T23:55:50.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:50 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:55:50.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:50 smithi038 ceph-mon[92422]: pgmap v17716: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 275 KiB/s wr, 9 op/s 2023-03-09T23:55:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:50 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:55:51.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32261.442042 secs 2023-03-09T23:55:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:51 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:55:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:55:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:55:51.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32261.442042 secs 2023-03-09T23:55:51.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:51 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:55:51.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:55:51.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:55:51.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32261.442042 secs 2023-03-09T23:55:51.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:51 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:55:51.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:55:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:55:52.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:52 smithi122 ceph-mon[93869]: pgmap v17717: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 353 KiB/s wr, 10 op/s 2023-03-09T23:55:52.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:52 smithi181 ceph-mon[94189]: pgmap v17717: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 353 KiB/s wr, 10 op/s 2023-03-09T23:55:52.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:52 smithi038 ceph-mon[92422]: pgmap v17717: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 353 KiB/s wr, 10 op/s 2023-03-09T23:55:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32725.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:55:54.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:53 smithi122 ceph-mon[93869]: pgmap v17718: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 353 KiB/s wr, 9 op/s 2023-03-09T23:55:54.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:53 smithi181 ceph-mon[94189]: pgmap v17718: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 353 KiB/s wr, 9 op/s 2023-03-09T23:55:54.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:53 smithi038 ceph-mon[92422]: pgmap v17718: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 353 KiB/s wr, 9 op/s 2023-03-09T23:55:56.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:55:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:55:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32266.442102 secs 2023-03-09T23:55:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:55 smithi122 ceph-mon[93869]: pgmap v17719: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 627 KiB/s rd, 542 KiB/s wr, 11 op/s 2023-03-09T23:55:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:55:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:55:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32266.442102 secs 2023-03-09T23:55:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:55 smithi181 ceph-mon[94189]: pgmap v17719: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 627 KiB/s rd, 542 KiB/s wr, 11 op/s 2023-03-09T23:55:56.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:55:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:55:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32266.442102 secs 2023-03-09T23:55:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:55 smithi038 ceph-mon[92422]: pgmap v17719: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 627 KiB/s rd, 542 KiB/s wr, 11 op/s 2023-03-09T23:55:57.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:55:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:55:57.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:55:57.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:55:57.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:55:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:55:58.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:57 smithi122 ceph-mon[93869]: pgmap v17720: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 610 KiB/s rd, 595 KiB/s wr, 13 op/s 2023-03-09T23:55:58.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:57 smithi181 ceph-mon[94189]: pgmap v17720: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 610 KiB/s rd, 595 KiB/s wr, 13 op/s 2023-03-09T23:55:58.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:57 smithi038 ceph-mon[92422]: pgmap v17720: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 610 KiB/s rd, 595 KiB/s wr, 13 op/s 2023-03-09T23:55:58.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32730.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:56:00.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:56:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:56:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:55:59 smithi122 ceph-mon[93869]: pgmap v17721: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 637 KiB/s rd, 625 KiB/s wr, 13 op/s 2023-03-09T23:56:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:56:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:56:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:55:59 smithi181 ceph-mon[94189]: pgmap v17721: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 637 KiB/s rd, 625 KiB/s wr, 13 op/s 2023-03-09T23:56:00.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:56:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:56:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:55:59 smithi038 ceph-mon[92422]: pgmap v17721: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 637 KiB/s rd, 625 KiB/s wr, 13 op/s 2023-03-09T23:56:01.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:56:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:56:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32271.442179 secs 2023-03-09T23:56:01.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:56:01.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:56:01.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32271.442179 secs 2023-03-09T23:56:01.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:56:01.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:56:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32271.442179 secs 2023-03-09T23:56:02.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:01 smithi122 ceph-mon[93869]: pgmap v17722: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 754 KiB/s rd, 761 KiB/s wr, 16 op/s 2023-03-09T23:56:02.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:01 smithi181 ceph-mon[94189]: pgmap v17722: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 754 KiB/s rd, 761 KiB/s wr, 16 op/s 2023-03-09T23:56:02.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:01 smithi038 ceph-mon[92422]: pgmap v17722: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 754 KiB/s rd, 761 KiB/s wr, 16 op/s 2023-03-09T23:56:03.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32735.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:56:04.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32273.863655 secs 2023-03-09T23:56:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:03 smithi122 ceph-mon[93869]: pgmap v17723: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 681 KiB/s rd, 705 KiB/s wr, 27 op/s 2023-03-09T23:56:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32273.863655 secs 2023-03-09T23:56:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:03 smithi181 ceph-mon[94189]: pgmap v17723: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 681 KiB/s rd, 705 KiB/s wr, 27 op/s 2023-03-09T23:56:04.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32273.863655 secs 2023-03-09T23:56:04.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:03 smithi038 ceph-mon[92422]: pgmap v17723: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 681 KiB/s rd, 705 KiB/s wr, 27 op/s 2023-03-09T23:56:06.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:05 smithi122 ceph-mon[93869]: osdmap e1323: 12 total, 12 up, 12 in 2023-03-09T23:56:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32276.442243 secs 2023-03-09T23:56:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:05 smithi122 ceph-mon[93869]: pgmap v17725: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 634 KiB/s rd, 677 KiB/s wr, 30 op/s 2023-03-09T23:56:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:05 smithi181 ceph-mon[94189]: osdmap e1323: 12 total, 12 up, 12 in 2023-03-09T23:56:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32276.442243 secs 2023-03-09T23:56:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:05 smithi181 ceph-mon[94189]: pgmap v17725: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 634 KiB/s rd, 677 KiB/s wr, 30 op/s 2023-03-09T23:56:06.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:05 smithi038 ceph-mon[92422]: osdmap e1323: 12 total, 12 up, 12 in 2023-03-09T23:56:06.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32276.442243 secs 2023-03-09T23:56:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:05 smithi038 ceph-mon[92422]: pgmap v17725: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 634 KiB/s rd, 677 KiB/s wr, 30 op/s 2023-03-09T23:56:07.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:06 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:56:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:06 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:56:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:56:07.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:06 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:56:07.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:06 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:56:07.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:56:07.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:06 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:56:07.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:06 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:56:07.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:56:08.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:07 smithi122 ceph-mon[93869]: pgmap v17726: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 500 KiB/s wr, 29 op/s 2023-03-09T23:56:08.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:07 smithi181 ceph-mon[94189]: pgmap v17726: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 500 KiB/s wr, 29 op/s 2023-03-09T23:56:08.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:07 smithi038 ceph-mon[92422]: pgmap v17726: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 500 KiB/s wr, 29 op/s 2023-03-09T23:56:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32740.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:56:09.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32278.863809 secs 2023-03-09T23:56:09.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32278.863809 secs 2023-03-09T23:56:09.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32278.863809 secs 2023-03-09T23:56:10.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:09 smithi122 ceph-mon[93869]: pgmap v17727: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 465 KiB/s wr, 27 op/s 2023-03-09T23:56:10.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:09 smithi181 ceph-mon[94189]: pgmap v17727: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 465 KiB/s wr, 27 op/s 2023-03-09T23:56:10.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:09 smithi038 ceph-mon[92422]: pgmap v17727: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 465 KiB/s wr, 27 op/s 2023-03-09T23:56:11.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32281.442302 secs 2023-03-09T23:56:11.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32281.442302 secs 2023-03-09T23:56:11.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32281.442302 secs 2023-03-09T23:56:12.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:11 smithi122 ceph-mon[93869]: pgmap v17728: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 435 KiB/s wr, 24 op/s 2023-03-09T23:56:12.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:11 smithi181 ceph-mon[94189]: pgmap v17728: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 435 KiB/s wr, 24 op/s 2023-03-09T23:56:12.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:11 smithi038 ceph-mon[92422]: pgmap v17728: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 435 KiB/s wr, 24 op/s 2023-03-09T23:56:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32745.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:56:14.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32283.863883 secs 2023-03-09T23:56:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:13 smithi122 ceph-mon[93869]: osdmap e1324: 12 total, 12 up, 12 in 2023-03-09T23:56:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:13 smithi122 ceph-mon[93869]: pgmap v17730: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 435 KiB/s rd, 432 KiB/s wr, 10 op/s 2023-03-09T23:56:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32283.863883 secs 2023-03-09T23:56:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:13 smithi181 ceph-mon[94189]: osdmap e1324: 12 total, 12 up, 12 in 2023-03-09T23:56:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:13 smithi181 ceph-mon[94189]: pgmap v17730: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 435 KiB/s rd, 432 KiB/s wr, 10 op/s 2023-03-09T23:56:14.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32283.863883 secs 2023-03-09T23:56:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:13 smithi038 ceph-mon[92422]: osdmap e1324: 12 total, 12 up, 12 in 2023-03-09T23:56:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:13 smithi038 ceph-mon[92422]: pgmap v17730: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 435 KiB/s rd, 432 KiB/s wr, 10 op/s 2023-03-09T23:56:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:15 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:56:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:15 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:56:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32286.442361 secs 2023-03-09T23:56:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:15 smithi122 ceph-mon[93869]: pgmap v17731: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 351 KiB/s wr, 9 op/s 2023-03-09T23:56:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:15 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:56:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:15 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:56:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:15 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:56:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32286.442361 secs 2023-03-09T23:56:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:15 smithi181 ceph-mon[94189]: pgmap v17731: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 351 KiB/s wr, 9 op/s 2023-03-09T23:56:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:15 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:56:16.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:15 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:56:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:15 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:56:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32286.442361 secs 2023-03-09T23:56:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:15 smithi038 ceph-mon[92422]: pgmap v17731: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 351 KiB/s wr, 9 op/s 2023-03-09T23:56:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:15 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:56:17.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:56:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:56:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:16 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:56:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:16 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:56:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:16 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:56:17.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:56:17.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:56:17.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:16 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:56:17.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:16 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:56:17.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:16 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:56:17.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:56:17.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:56:17.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:16 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:56:17.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:16 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:56:17.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:16 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:56:18.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:17 smithi122 ceph-mon[93869]: pgmap v17732: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 169 KiB/s wr, 7 op/s 2023-03-09T23:56:18.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:17 smithi181 ceph-mon[94189]: pgmap v17732: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 169 KiB/s wr, 7 op/s 2023-03-09T23:56:18.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:17 smithi038 ceph-mon[92422]: pgmap v17732: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 169 KiB/s wr, 7 op/s 2023-03-09T23:56:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32750.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:56:19.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32288.863959 secs 2023-03-09T23:56:19.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32288.863959 secs 2023-03-09T23:56:19.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32288.863959 secs 2023-03-09T23:56:20.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:19 smithi122 ceph-mon[93869]: pgmap v17733: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 169 KiB/s wr, 7 op/s 2023-03-09T23:56:20.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:19 smithi181 ceph-mon[94189]: pgmap v17733: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 169 KiB/s wr, 7 op/s 2023-03-09T23:56:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:19 smithi038 ceph-mon[92422]: pgmap v17733: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 169 KiB/s wr, 7 op/s 2023-03-09T23:56:21.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32291.442451 secs 2023-03-09T23:56:21.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32291.442451 secs 2023-03-09T23:56:21.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32291.442451 secs 2023-03-09T23:56:22.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:21 smithi122 ceph-mon[93869]: pgmap v17734: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T23:56:22.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:21 smithi181 ceph-mon[94189]: pgmap v17734: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T23:56:22.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:21 smithi038 ceph-mon[92422]: pgmap v17734: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T23:56:23.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32755.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:56:24.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32293.864075 secs 2023-03-09T23:56:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:23 smithi122 ceph-mon[93869]: pgmap v17735: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T23:56:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32293.864075 secs 2023-03-09T23:56:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:23 smithi181 ceph-mon[94189]: pgmap v17735: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T23:56:24.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32293.864075 secs 2023-03-09T23:56:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:23 smithi038 ceph-mon[92422]: pgmap v17735: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T23:56:26.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32296.442530 secs 2023-03-09T23:56:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:26 smithi122 ceph-mon[93869]: pgmap v17736: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T23:56:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32296.442530 secs 2023-03-09T23:56:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:26 smithi181 ceph-mon[94189]: pgmap v17736: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T23:56:26.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32296.442530 secs 2023-03-09T23:56:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:26 smithi038 ceph-mon[92422]: pgmap v17736: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T23:56:28.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:28 smithi122 ceph-mon[93869]: pgmap v17737: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 8 op/s 2023-03-09T23:56:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32298.864194 secs 2023-03-09T23:56:28.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:28 smithi181 ceph-mon[94189]: pgmap v17737: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 8 op/s 2023-03-09T23:56:28.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32298.864194 secs 2023-03-09T23:56:28.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:28 smithi038 ceph-mon[92422]: pgmap v17737: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 8 op/s 2023-03-09T23:56:28.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32298.864194 secs 2023-03-09T23:56:28.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32760.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:56:30.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:30 smithi122 ceph-mon[93869]: pgmap v17738: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T23:56:30.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:30 smithi181 ceph-mon[94189]: pgmap v17738: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T23:56:30.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:30 smithi038 ceph-mon[92422]: pgmap v17738: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T23:56:31.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32301.442591 secs 2023-03-09T23:56:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32301.442591 secs 2023-03-09T23:56:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32301.442591 secs 2023-03-09T23:56:32.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:32 smithi122 ceph-mon[93869]: pgmap v17739: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T23:56:32.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:32 smithi181 ceph-mon[94189]: pgmap v17739: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T23:56:32.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:32 smithi038 ceph-mon[92422]: pgmap v17739: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T23:56:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32765.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:56:34.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32303.864279 secs 2023-03-09T23:56:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:33 smithi122 ceph-mon[93869]: pgmap v17740: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T23:56:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32303.864279 secs 2023-03-09T23:56:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:33 smithi181 ceph-mon[94189]: pgmap v17740: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T23:56:34.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32303.864279 secs 2023-03-09T23:56:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:33 smithi038 ceph-mon[92422]: pgmap v17740: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T23:56:36.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32306.442652 secs 2023-03-09T23:56:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:36 smithi122 ceph-mon[93869]: pgmap v17741: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T23:56:36.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32306.442652 secs 2023-03-09T23:56:36.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:36 smithi181 ceph-mon[94189]: pgmap v17741: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T23:56:36.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32306.442652 secs 2023-03-09T23:56:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:36 smithi038 ceph-mon[92422]: pgmap v17741: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T23:56:38.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:38 smithi122 ceph-mon[93869]: pgmap v17742: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 8 op/s 2023-03-09T23:56:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32308.864431 secs 2023-03-09T23:56:38.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:38 smithi181 ceph-mon[94189]: pgmap v17742: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 8 op/s 2023-03-09T23:56:38.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32308.864431 secs 2023-03-09T23:56:38.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:38 smithi038 ceph-mon[92422]: pgmap v17742: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 8 op/s 2023-03-09T23:56:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32308.864431 secs 2023-03-09T23:56:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32770.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:56:40.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:40 smithi122 ceph-mon[93869]: pgmap v17743: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T23:56:40.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:40 smithi181 ceph-mon[94189]: pgmap v17743: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T23:56:40.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:40 smithi038 ceph-mon[92422]: pgmap v17743: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-09T23:56:41.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32311.442736 secs 2023-03-09T23:56:41.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32311.442736 secs 2023-03-09T23:56:41.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32311.442736 secs 2023-03-09T23:56:42.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:42 smithi122 ceph-mon[93869]: pgmap v17744: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 120 KiB/s wr, 9 op/s 2023-03-09T23:56:42.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:42 smithi181 ceph-mon[94189]: pgmap v17744: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 120 KiB/s wr, 9 op/s 2023-03-09T23:56:42.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:42 smithi038 ceph-mon[92422]: pgmap v17744: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 120 KiB/s wr, 9 op/s 2023-03-09T23:56:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32775.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:56:44.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:43 smithi122 ceph-mon[93869]: pgmap v17745: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 120 KiB/s wr, 7 op/s 2023-03-09T23:56:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:43 smithi181 ceph-mon[94189]: pgmap v17745: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 120 KiB/s wr, 7 op/s 2023-03-09T23:56:44.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:43 smithi038 ceph-mon[92422]: pgmap v17745: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 120 KiB/s wr, 7 op/s 2023-03-09T23:56:46.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32316.442795 secs 2023-03-09T23:56:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:46 smithi122 ceph-mon[93869]: pgmap v17746: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 120 KiB/s wr, 7 op/s 2023-03-09T23:56:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:46 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:56:46.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32316.442795 secs 2023-03-09T23:56:46.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:46 smithi181 ceph-mon[94189]: pgmap v17746: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 120 KiB/s wr, 7 op/s 2023-03-09T23:56:46.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:46 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:56:46.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32316.442795 secs 2023-03-09T23:56:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:46 smithi038 ceph-mon[92422]: pgmap v17746: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 120 KiB/s wr, 7 op/s 2023-03-09T23:56:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:46 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:56:47.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:47 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:56:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:56:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:56:47.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:47 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:56:47.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:56:47.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:56:47.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:47 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:56:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:56:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:56:48.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:48 smithi122 ceph-mon[93869]: pgmap v17747: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 369 KiB/s wr, 10 op/s 2023-03-09T23:56:48.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:56:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:56:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:48 smithi181 ceph-mon[94189]: pgmap v17747: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 369 KiB/s wr, 10 op/s 2023-03-09T23:56:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:56:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:56:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:48 smithi038 ceph-mon[92422]: pgmap v17747: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 369 KiB/s wr, 10 op/s 2023-03-09T23:56:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:56:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:56:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32780.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:56:49.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:56:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:56:49.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:56:49.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:56:49.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:56:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:56:50.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:50 smithi122 ceph-mon[93869]: pgmap v17748: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 395 KiB/s wr, 8 op/s 2023-03-09T23:56:50.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:50 smithi181 ceph-mon[94189]: pgmap v17748: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 395 KiB/s wr, 8 op/s 2023-03-09T23:56:50.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:50 smithi038 ceph-mon[92422]: pgmap v17748: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 395 KiB/s wr, 8 op/s 2023-03-09T23:56:51.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32321.442856 secs 2023-03-09T23:56:51.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32321.442856 secs 2023-03-09T23:56:51.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32321.442856 secs 2023-03-09T23:56:52.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:52 smithi122 ceph-mon[93869]: pgmap v17749: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 758 KiB/s rd, 658 KiB/s wr, 11 op/s 2023-03-09T23:56:52.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:52 smithi181 ceph-mon[94189]: pgmap v17749: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 758 KiB/s rd, 658 KiB/s wr, 11 op/s 2023-03-09T23:56:52.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:52 smithi038 ceph-mon[92422]: pgmap v17749: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 758 KiB/s rd, 658 KiB/s wr, 11 op/s 2023-03-09T23:56:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32785.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:56:54.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:53 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 32323.864850 secs 2023-03-09T23:56:54.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:53 smithi122 ceph-mon[93869]: pgmap v17750: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 637 KiB/s rd, 538 KiB/s wr, 9 op/s 2023-03-09T23:56:54.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:53 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 32323.864850 secs 2023-03-09T23:56:54.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:53 smithi181 ceph-mon[94189]: pgmap v17750: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 637 KiB/s rd, 538 KiB/s wr, 9 op/s 2023-03-09T23:56:54.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:53 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 32323.864850 secs 2023-03-09T23:56:54.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:53 smithi038 ceph-mon[92422]: pgmap v17750: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 637 KiB/s rd, 538 KiB/s wr, 9 op/s 2023-03-09T23:56:55.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:54 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:56:55.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:54 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:56:55.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:54 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:56:55.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:54 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:56:55.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:54 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:56:55.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:54 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:56:55.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:54 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:56:55.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:54 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:56:55.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:54 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:56:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:56:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:56:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32326.442913 secs 2023-03-09T23:56:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:55 smithi122 ceph-mon[93869]: pgmap v17751: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 674 KiB/s rd, 564 KiB/s wr, 10 op/s 2023-03-09T23:56:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:56:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:56:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32326.442913 secs 2023-03-09T23:56:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:55 smithi181 ceph-mon[94189]: pgmap v17751: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 674 KiB/s rd, 564 KiB/s wr, 10 op/s 2023-03-09T23:56:56.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:56:56.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:56:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32326.442913 secs 2023-03-09T23:56:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:55 smithi038 ceph-mon[92422]: pgmap v17751: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 674 KiB/s rd, 564 KiB/s wr, 10 op/s 2023-03-09T23:56:57.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:56:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:56:57.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:56:57.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:56:57.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:56:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:56:58.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:57 smithi122 ceph-mon[93869]: pgmap v17752: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 978 KiB/s rd, 868 KiB/s wr, 15 op/s 2023-03-09T23:56:58.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:57 smithi181 ceph-mon[94189]: pgmap v17752: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 978 KiB/s rd, 868 KiB/s wr, 15 op/s 2023-03-09T23:56:58.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:57 smithi038 ceph-mon[92422]: pgmap v17752: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 978 KiB/s rd, 868 KiB/s wr, 15 op/s 2023-03-09T23:56:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32790.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:56:59.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32328.864930 secs 2023-03-09T23:56:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:58 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:56:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:58 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:56:59.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32328.864930 secs 2023-03-09T23:56:59.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:58 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:56:59.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:58 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:56:59.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32328.864930 secs 2023-03-09T23:56:59.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:58 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:56:59.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:58 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:57:00.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:57:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:57:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:56:59 smithi122 ceph-mon[93869]: pgmap v17753: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 639 KiB/s rd, 626 KiB/s wr, 13 op/s 2023-03-09T23:57:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:57:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:57:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:56:59 smithi181 ceph-mon[94189]: pgmap v17753: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 639 KiB/s rd, 626 KiB/s wr, 13 op/s 2023-03-09T23:57:00.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:57:00.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:57:00.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:56:59 smithi038 ceph-mon[92422]: pgmap v17753: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 639 KiB/s rd, 626 KiB/s wr, 13 op/s 2023-03-09T23:57:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:57:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:57:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32331.442976 secs 2023-03-09T23:57:01.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:57:01.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:57:01.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32331.442976 secs 2023-03-09T23:57:01.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:57:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:57:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32331.442976 secs 2023-03-09T23:57:02.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:01 smithi122 ceph-mon[93869]: pgmap v17754: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 758 KiB/s rd, 766 KiB/s wr, 21 op/s 2023-03-09T23:57:02.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:01 smithi181 ceph-mon[94189]: pgmap v17754: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 758 KiB/s rd, 766 KiB/s wr, 21 op/s 2023-03-09T23:57:02.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:01 smithi038 ceph-mon[92422]: pgmap v17754: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 758 KiB/s rd, 766 KiB/s wr, 21 op/s 2023-03-09T23:57:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32795.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:57:04.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32333.865049 secs 2023-03-09T23:57:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:03 smithi122 ceph-mon[93869]: pgmap v17755: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 526 KiB/s rd, 551 KiB/s wr, 31 op/s 2023-03-09T23:57:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32333.865049 secs 2023-03-09T23:57:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:03 smithi181 ceph-mon[94189]: pgmap v17755: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 526 KiB/s rd, 551 KiB/s wr, 31 op/s 2023-03-09T23:57:04.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32333.865049 secs 2023-03-09T23:57:04.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:03 smithi038 ceph-mon[92422]: pgmap v17755: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 526 KiB/s rd, 551 KiB/s wr, 31 op/s 2023-03-09T23:57:06.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:05 smithi122 ceph-mon[93869]: osdmap e1325: 12 total, 12 up, 12 in 2023-03-09T23:57:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32336.443999 secs 2023-03-09T23:57:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:05 smithi122 ceph-mon[93869]: pgmap v17757: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 630 KiB/s wr, 35 op/s 2023-03-09T23:57:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:05 smithi181 ceph-mon[94189]: osdmap e1325: 12 total, 12 up, 12 in 2023-03-09T23:57:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32336.443999 secs 2023-03-09T23:57:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:05 smithi181 ceph-mon[94189]: pgmap v17757: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 630 KiB/s wr, 35 op/s 2023-03-09T23:57:06.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:05 smithi038 ceph-mon[92422]: osdmap e1325: 12 total, 12 up, 12 in 2023-03-09T23:57:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32336.443999 secs 2023-03-09T23:57:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:05 smithi038 ceph-mon[92422]: pgmap v17757: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 630 KiB/s wr, 35 op/s 2023-03-09T23:57:08.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:08 smithi122 ceph-mon[93869]: pgmap v17758: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 475 KiB/s wr, 35 op/s 2023-03-09T23:57:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32338.865193 secs 2023-03-09T23:57:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:08 smithi181 ceph-mon[94189]: pgmap v17758: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 475 KiB/s wr, 35 op/s 2023-03-09T23:57:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32338.865193 secs 2023-03-09T23:57:08.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:08 smithi038 ceph-mon[92422]: pgmap v17758: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 475 KiB/s wr, 35 op/s 2023-03-09T23:57:08.520 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32338.865193 secs 2023-03-09T23:57:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32800.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:57:10.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:10 smithi122 ceph-mon[93869]: pgmap v17759: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 466 KiB/s wr, 33 op/s 2023-03-09T23:57:10.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:10 smithi181 ceph-mon[94189]: pgmap v17759: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 466 KiB/s wr, 33 op/s 2023-03-09T23:57:10.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:10 smithi038 ceph-mon[92422]: pgmap v17759: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 466 KiB/s wr, 33 op/s 2023-03-09T23:57:11.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32341.444058 secs 2023-03-09T23:57:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32341.444058 secs 2023-03-09T23:57:11.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32341.444058 secs 2023-03-09T23:57:12.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:12 smithi122 ceph-mon[93869]: pgmap v17760: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 437 KiB/s wr, 27 op/s 2023-03-09T23:57:12.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:12 smithi181 ceph-mon[94189]: pgmap v17760: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 437 KiB/s wr, 27 op/s 2023-03-09T23:57:12.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:12 smithi038 ceph-mon[92422]: pgmap v17760: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 437 KiB/s wr, 27 op/s 2023-03-09T23:57:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32805.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:57:14.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32343.865309 secs 2023-03-09T23:57:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:13 smithi122 ceph-mon[93869]: osdmap e1326: 12 total, 12 up, 12 in 2023-03-09T23:57:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:13 smithi122 ceph-mon[93869]: pgmap v17762: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 469 KiB/s rd, 467 KiB/s wr, 13 op/s 2023-03-09T23:57:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32343.865309 secs 2023-03-09T23:57:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:13 smithi181 ceph-mon[94189]: osdmap e1326: 12 total, 12 up, 12 in 2023-03-09T23:57:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:13 smithi181 ceph-mon[94189]: pgmap v17762: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 469 KiB/s rd, 467 KiB/s wr, 13 op/s 2023-03-09T23:57:14.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32343.865309 secs 2023-03-09T23:57:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:13 smithi038 ceph-mon[92422]: osdmap e1326: 12 total, 12 up, 12 in 2023-03-09T23:57:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:13 smithi038 ceph-mon[92422]: pgmap v17762: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 469 KiB/s rd, 467 KiB/s wr, 13 op/s 2023-03-09T23:57:16.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32346.444120 secs 2023-03-09T23:57:16.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:16 smithi122 ceph-mon[93869]: pgmap v17763: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 380 KiB/s wr, 10 op/s 2023-03-09T23:57:16.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32346.444120 secs 2023-03-09T23:57:16.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:16 smithi181 ceph-mon[94189]: pgmap v17763: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 380 KiB/s wr, 10 op/s 2023-03-09T23:57:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32346.444120 secs 2023-03-09T23:57:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:16 smithi038 ceph-mon[92422]: pgmap v17763: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 380 KiB/s wr, 10 op/s 2023-03-09T23:57:17.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:17 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:57:17.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:17 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:57:17.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:17 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:57:18.473 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:18 smithi038 ceph-mon[92422]: pgmap v17764: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 170 KiB/s wr, 8 op/s 2023-03-09T23:57:18.474 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32348.865426 secs 2023-03-09T23:57:18.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:18 smithi122 ceph-mon[93869]: pgmap v17764: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 170 KiB/s wr, 8 op/s 2023-03-09T23:57:18.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32348.865426 secs 2023-03-09T23:57:18.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:18 smithi181 ceph-mon[94189]: pgmap v17764: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 170 KiB/s wr, 8 op/s 2023-03-09T23:57:18.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32348.865426 secs 2023-03-09T23:57:18.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32810.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:57:20.239 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:20 smithi038 ceph-mon[92422]: pgmap v17765: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 170 KiB/s wr, 8 op/s 2023-03-09T23:57:20.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:20 smithi122 ceph-mon[93869]: pgmap v17765: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 170 KiB/s wr, 8 op/s 2023-03-09T23:57:20.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:20 smithi181 ceph-mon[94189]: pgmap v17765: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 170 KiB/s wr, 8 op/s 2023-03-09T23:57:21.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32351.444180 secs 2023-03-09T23:57:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:57:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:57:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:57:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:57:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:57:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:21 smithi122 ceph-mon[93869]: pgmap v17766: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 409 B/s wr, 7 op/s 2023-03-09T23:57:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32351.444180 secs 2023-03-09T23:57:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:57:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:57:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:57:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:57:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:57:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:21 smithi181 ceph-mon[94189]: pgmap v17766: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 409 B/s wr, 7 op/s 2023-03-09T23:57:21.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32351.444180 secs 2023-03-09T23:57:21.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:57:21.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:57:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:57:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:57:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:57:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:21 smithi038 ceph-mon[92422]: pgmap v17766: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 409 B/s wr, 7 op/s 2023-03-09T23:57:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32815.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:57:24.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32353.865539 secs 2023-03-09T23:57:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:23 smithi122 ceph-mon[93869]: pgmap v17767: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T23:57:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32353.865539 secs 2023-03-09T23:57:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:23 smithi181 ceph-mon[94189]: pgmap v17767: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T23:57:24.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32353.865539 secs 2023-03-09T23:57:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:23 smithi038 ceph-mon[92422]: pgmap v17767: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T23:57:26.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32356.444240 secs 2023-03-09T23:57:26.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:26 smithi122 ceph-mon[93869]: pgmap v17768: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T23:57:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32356.444240 secs 2023-03-09T23:57:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:26 smithi181 ceph-mon[94189]: pgmap v17768: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T23:57:26.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32356.444240 secs 2023-03-09T23:57:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:26 smithi038 ceph-mon[92422]: pgmap v17768: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T23:57:28.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:28 smithi122 ceph-mon[93869]: pgmap v17769: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 8 op/s 2023-03-09T23:57:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32358.865673 secs 2023-03-09T23:57:28.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:28 smithi181 ceph-mon[94189]: pgmap v17769: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 8 op/s 2023-03-09T23:57:28.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32358.865673 secs 2023-03-09T23:57:28.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:28 smithi038 ceph-mon[92422]: pgmap v17769: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 8 op/s 2023-03-09T23:57:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32358.865673 secs 2023-03-09T23:57:28.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32820.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:57:30.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:30 smithi122 ceph-mon[93869]: pgmap v17770: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T23:57:30.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:30 smithi181 ceph-mon[94189]: pgmap v17770: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T23:57:30.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:30 smithi038 ceph-mon[92422]: pgmap v17770: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T23:57:31.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32361.444303 secs 2023-03-09T23:57:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32361.444303 secs 2023-03-09T23:57:31.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32361.444303 secs 2023-03-09T23:57:32.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:32 smithi122 ceph-mon[93869]: pgmap v17771: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T23:57:32.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:32 smithi181 ceph-mon[94189]: pgmap v17771: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T23:57:32.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:32 smithi038 ceph-mon[92422]: pgmap v17771: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T23:57:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32825.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:57:34.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32363.865812 secs 2023-03-09T23:57:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:33 smithi122 ceph-mon[93869]: pgmap v17772: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T23:57:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32363.865812 secs 2023-03-09T23:57:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:33 smithi181 ceph-mon[94189]: pgmap v17772: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T23:57:34.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32363.865812 secs 2023-03-09T23:57:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:33 smithi038 ceph-mon[92422]: pgmap v17772: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-09T23:57:36.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32366.444401 secs 2023-03-09T23:57:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:36 smithi122 ceph-mon[93869]: pgmap v17773: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T23:57:36.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32366.444401 secs 2023-03-09T23:57:36.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:36 smithi181 ceph-mon[94189]: pgmap v17773: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T23:57:36.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32366.444401 secs 2023-03-09T23:57:36.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:36 smithi038 ceph-mon[92422]: pgmap v17773: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T23:57:38.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:38 smithi122 ceph-mon[93869]: pgmap v17774: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 9 op/s 2023-03-09T23:57:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32368.865924 secs 2023-03-09T23:57:38.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:38 smithi181 ceph-mon[94189]: pgmap v17774: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 9 op/s 2023-03-09T23:57:38.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32368.865924 secs 2023-03-09T23:57:38.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:38 smithi038 ceph-mon[92422]: pgmap v17774: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 9 op/s 2023-03-09T23:57:38.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32368.865924 secs 2023-03-09T23:57:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32830.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:57:40.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:40 smithi122 ceph-mon[93869]: pgmap v17775: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-09T23:57:40.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:40 smithi181 ceph-mon[94189]: pgmap v17775: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-09T23:57:40.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:40 smithi038 ceph-mon[92422]: pgmap v17775: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-09T23:57:41.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32371.444460 secs 2023-03-09T23:57:41.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32371.444460 secs 2023-03-09T23:57:41.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32371.444460 secs 2023-03-09T23:57:42.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:42 smithi122 ceph-mon[93869]: pgmap v17776: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 8 op/s 2023-03-09T23:57:42.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:42 smithi181 ceph-mon[94189]: pgmap v17776: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 8 op/s 2023-03-09T23:57:42.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:42 smithi038 ceph-mon[92422]: pgmap v17776: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 170 B/s wr, 8 op/s 2023-03-09T23:57:43.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32835.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:57:44.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32373.866050 secs 2023-03-09T23:57:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:43 smithi122 ceph-mon[93869]: pgmap v17777: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 682 B/s wr, 7 op/s 2023-03-09T23:57:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32373.866050 secs 2023-03-09T23:57:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:43 smithi181 ceph-mon[94189]: pgmap v17777: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 682 B/s wr, 7 op/s 2023-03-09T23:57:44.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32373.866050 secs 2023-03-09T23:57:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:43 smithi038 ceph-mon[92422]: pgmap v17777: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 682 B/s wr, 7 op/s 2023-03-09T23:57:46.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32376.444520 secs 2023-03-09T23:57:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:46 smithi122 ceph-mon[93869]: pgmap v17778: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 682 B/s wr, 6 op/s 2023-03-09T23:57:46.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32376.444520 secs 2023-03-09T23:57:46.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:46 smithi181 ceph-mon[94189]: pgmap v17778: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 682 B/s wr, 6 op/s 2023-03-09T23:57:46.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32376.444520 secs 2023-03-09T23:57:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:46 smithi038 ceph-mon[92422]: pgmap v17778: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 682 B/s wr, 6 op/s 2023-03-09T23:57:48.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:48 smithi122 ceph-mon[93869]: pgmap v17779: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 276 KiB/s wr, 11 op/s 2023-03-09T23:57:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:57:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:57:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32378.866165 secs 2023-03-09T23:57:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:48 smithi181 ceph-mon[94189]: pgmap v17779: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 276 KiB/s wr, 11 op/s 2023-03-09T23:57:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:57:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:57:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32378.866165 secs 2023-03-09T23:57:48.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:48 smithi038 ceph-mon[92422]: pgmap v17779: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 279 KiB/s rd, 276 KiB/s wr, 11 op/s 2023-03-09T23:57:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:57:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:57:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32378.866165 secs 2023-03-09T23:57:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32840.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:57:49.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:57:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:57:49.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:57:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:57:49.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:57:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:57:50.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:50 smithi122 ceph-mon[93869]: pgmap v17780: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 276 KiB/s wr, 8 op/s 2023-03-09T23:57:50.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:50 smithi181 ceph-mon[94189]: pgmap v17780: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 276 KiB/s wr, 8 op/s 2023-03-09T23:57:50.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:50 smithi038 ceph-mon[92422]: pgmap v17780: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 276 KiB/s wr, 8 op/s 2023-03-09T23:57:51.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32381.444597 secs 2023-03-09T23:57:51.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32381.444597 secs 2023-03-09T23:57:51.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32381.444597 secs 2023-03-09T23:57:52.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:52 smithi122 ceph-mon[93869]: pgmap v17781: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 276 KiB/s wr, 10 op/s 2023-03-09T23:57:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:57:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:57:52.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:52 smithi181 ceph-mon[94189]: pgmap v17781: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 276 KiB/s wr, 10 op/s 2023-03-09T23:57:52.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:57:52.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:57:52.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:52 smithi038 ceph-mon[92422]: pgmap v17781: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 281 KiB/s rd, 276 KiB/s wr, 10 op/s 2023-03-09T23:57:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:57:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:57:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32845.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:57:54.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32383.866233 secs 2023-03-09T23:57:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:53 smithi122 ceph-mon[93869]: pgmap v17782: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 544 KiB/s wr, 10 op/s 2023-03-09T23:57:54.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32383.866233 secs 2023-03-09T23:57:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:53 smithi181 ceph-mon[94189]: pgmap v17782: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 544 KiB/s wr, 10 op/s 2023-03-09T23:57:54.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32383.866233 secs 2023-03-09T23:57:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:53 smithi038 ceph-mon[92422]: pgmap v17782: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 544 KiB/s wr, 10 op/s 2023-03-09T23:57:56.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:57:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:57:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32386.444675 secs 2023-03-09T23:57:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:55 smithi122 ceph-mon[93869]: pgmap v17783: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 559 KiB/s rd, 544 KiB/s wr, 9 op/s 2023-03-09T23:57:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:57:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:57:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32386.444675 secs 2023-03-09T23:57:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:55 smithi181 ceph-mon[94189]: pgmap v17783: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 559 KiB/s rd, 544 KiB/s wr, 9 op/s 2023-03-09T23:57:56.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:57:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:57:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32386.444675 secs 2023-03-09T23:57:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:55 smithi038 ceph-mon[92422]: pgmap v17783: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 559 KiB/s rd, 544 KiB/s wr, 9 op/s 2023-03-09T23:57:57.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:57:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:57:57.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:57:57.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:57:57.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:57:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:57:58.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:57 smithi122 ceph-mon[93869]: pgmap v17784: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 889 KiB/s rd, 873 KiB/s wr, 15 op/s 2023-03-09T23:57:58.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:57 smithi181 ceph-mon[94189]: pgmap v17784: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 889 KiB/s rd, 873 KiB/s wr, 15 op/s 2023-03-09T23:57:58.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:57 smithi038 ceph-mon[92422]: pgmap v17784: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 889 KiB/s rd, 873 KiB/s wr, 15 op/s 2023-03-09T23:57:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32850.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:57:59.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32388.866325 secs 2023-03-09T23:57:59.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32388.866325 secs 2023-03-09T23:57:59.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32388.866325 secs 2023-03-09T23:58:00.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:57:59 smithi122 ceph-mon[93869]: pgmap v17785: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 612 KiB/s rd, 599 KiB/s wr, 11 op/s 2023-03-09T23:58:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:57:59 smithi181 ceph-mon[94189]: pgmap v17785: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 612 KiB/s rd, 599 KiB/s wr, 11 op/s 2023-03-09T23:58:00.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:57:59 smithi038 ceph-mon[92422]: pgmap v17785: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 612 KiB/s rd, 599 KiB/s wr, 11 op/s 2023-03-09T23:58:01.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32391.444743 secs 2023-03-09T23:58:01.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32391.444743 secs 2023-03-09T23:58:01.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32391.444743 secs 2023-03-09T23:58:02.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:01 smithi122 ceph-mon[93869]: pgmap v17786: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 742 KiB/s rd, 733 KiB/s wr, 14 op/s 2023-03-09T23:58:02.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:01 smithi181 ceph-mon[94189]: pgmap v17786: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 742 KiB/s rd, 733 KiB/s wr, 14 op/s 2023-03-09T23:58:02.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:01 smithi038 ceph-mon[92422]: pgmap v17786: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 742 KiB/s rd, 733 KiB/s wr, 14 op/s 2023-03-09T23:58:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32855.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:58:04.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32393.866392 secs 2023-03-09T23:58:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:03 smithi122 ceph-mon[93869]: pgmap v17787: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 788 KiB/s rd, 796 KiB/s wr, 26 op/s 2023-03-09T23:58:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32393.866392 secs 2023-03-09T23:58:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:03 smithi181 ceph-mon[94189]: pgmap v17787: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 788 KiB/s rd, 796 KiB/s wr, 26 op/s 2023-03-09T23:58:04.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32393.866392 secs 2023-03-09T23:58:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:03 smithi038 ceph-mon[92422]: pgmap v17787: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 788 KiB/s rd, 796 KiB/s wr, 26 op/s 2023-03-09T23:58:06.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:05 smithi122 ceph-mon[93869]: osdmap e1327: 12 total, 12 up, 12 in 2023-03-09T23:58:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32396.444824 secs 2023-03-09T23:58:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:05 smithi122 ceph-mon[93869]: pgmap v17789: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 611 KiB/s rd, 634 KiB/s wr, 29 op/s 2023-03-09T23:58:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:05 smithi181 ceph-mon[94189]: osdmap e1327: 12 total, 12 up, 12 in 2023-03-09T23:58:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32396.444824 secs 2023-03-09T23:58:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:05 smithi181 ceph-mon[94189]: pgmap v17789: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 611 KiB/s rd, 634 KiB/s wr, 29 op/s 2023-03-09T23:58:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:05 smithi038 ceph-mon[92422]: osdmap e1327: 12 total, 12 up, 12 in 2023-03-09T23:58:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32396.444824 secs 2023-03-09T23:58:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:05 smithi038 ceph-mon[92422]: pgmap v17789: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 611 KiB/s rd, 634 KiB/s wr, 29 op/s 2023-03-09T23:58:08.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:58:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:58:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:07 smithi122 ceph-mon[93869]: pgmap v17790: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 468 KiB/s wr, 29 op/s 2023-03-09T23:58:08.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:58:08.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:58:08.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:07 smithi181 ceph-mon[94189]: pgmap v17790: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 468 KiB/s wr, 29 op/s 2023-03-09T23:58:08.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:58:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:58:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:07 smithi038 ceph-mon[92422]: pgmap v17790: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 468 KiB/s wr, 29 op/s 2023-03-09T23:58:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32860.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:58:09.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32398.866541 secs 2023-03-09T23:58:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:58:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:58:09.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32398.866541 secs 2023-03-09T23:58:09.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:58:09.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:58:09.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32398.866541 secs 2023-03-09T23:58:09.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:58:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:58:10.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:09 smithi122 ceph-mon[93869]: pgmap v17791: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 467 KiB/s wr, 29 op/s 2023-03-09T23:58:10.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:09 smithi181 ceph-mon[94189]: pgmap v17791: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 467 KiB/s wr, 29 op/s 2023-03-09T23:58:10.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:09 smithi038 ceph-mon[92422]: pgmap v17791: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 467 KiB/s wr, 29 op/s 2023-03-09T23:58:11.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32401.444884 secs 2023-03-09T23:58:11.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32401.444884 secs 2023-03-09T23:58:11.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32401.444884 secs 2023-03-09T23:58:12.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:11 smithi122 ceph-mon[93869]: pgmap v17792: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 306 KiB/s wr, 27 op/s 2023-03-09T23:58:12.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:11 smithi181 ceph-mon[94189]: pgmap v17792: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 306 KiB/s wr, 27 op/s 2023-03-09T23:58:12.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:11 smithi038 ceph-mon[92422]: pgmap v17792: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 306 KiB/s wr, 27 op/s 2023-03-09T23:58:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32865.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:58:14.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32403.866638 secs 2023-03-09T23:58:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:13 smithi122 ceph-mon[93869]: osdmap e1328: 12 total, 12 up, 12 in 2023-03-09T23:58:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:13 smithi122 ceph-mon[93869]: pgmap v17794: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 469 KiB/s rd, 490 KiB/s wr, 14 op/s 2023-03-09T23:58:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32403.866638 secs 2023-03-09T23:58:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:13 smithi181 ceph-mon[94189]: osdmap e1328: 12 total, 12 up, 12 in 2023-03-09T23:58:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:13 smithi181 ceph-mon[94189]: pgmap v17794: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 469 KiB/s rd, 490 KiB/s wr, 14 op/s 2023-03-09T23:58:14.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32403.866638 secs 2023-03-09T23:58:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:13 smithi038 ceph-mon[92422]: osdmap e1328: 12 total, 12 up, 12 in 2023-03-09T23:58:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:13 smithi038 ceph-mon[92422]: pgmap v17794: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 469 KiB/s rd, 490 KiB/s wr, 14 op/s 2023-03-09T23:58:16.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32406.444947 secs 2023-03-09T23:58:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:16 smithi122 ceph-mon[93869]: pgmap v17795: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 399 KiB/s wr, 11 op/s 2023-03-09T23:58:16.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32406.444947 secs 2023-03-09T23:58:16.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:16 smithi181 ceph-mon[94189]: pgmap v17795: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 399 KiB/s wr, 11 op/s 2023-03-09T23:58:16.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32406.444947 secs 2023-03-09T23:58:16.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:16 smithi038 ceph-mon[92422]: pgmap v17795: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 399 KiB/s wr, 11 op/s 2023-03-09T23:58:18.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:18 smithi122 ceph-mon[93869]: pgmap v17796: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 170 KiB/s wr, 8 op/s 2023-03-09T23:58:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32408.866762 secs 2023-03-09T23:58:18.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:18 smithi181 ceph-mon[94189]: pgmap v17796: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 170 KiB/s wr, 8 op/s 2023-03-09T23:58:18.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32408.866762 secs 2023-03-09T23:58:18.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:18 smithi038 ceph-mon[92422]: pgmap v17796: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 170 KiB/s wr, 8 op/s 2023-03-09T23:58:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32408.866762 secs 2023-03-09T23:58:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32870.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:58:20.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:20 smithi122 ceph-mon[93869]: pgmap v17797: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 170 KiB/s wr, 7 op/s 2023-03-09T23:58:20.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:20 smithi181 ceph-mon[94189]: pgmap v17797: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 170 KiB/s wr, 7 op/s 2023-03-09T23:58:20.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:20 smithi038 ceph-mon[92422]: pgmap v17797: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 170 KiB/s wr, 7 op/s 2023-03-09T23:58:21.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32411.444981 secs 2023-03-09T23:58:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:58:21.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32411.444981 secs 2023-03-09T23:58:21.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:58:21.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32411.444981 secs 2023-03-09T23:58:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:58:22.362 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:22 smithi181 ceph-mon[94189]: pgmap v17798: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 170 KiB/s wr, 7 op/s 2023-03-09T23:58:22.452 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:22 smithi122 ceph-mon[93869]: pgmap v17798: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 170 KiB/s wr, 7 op/s 2023-03-09T23:58:22.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:22 smithi038 ceph-mon[92422]: pgmap v17798: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 170 KiB/s wr, 7 op/s 2023-03-09T23:58:23.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32875.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:58:24.149 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32413.866897 secs 2023-03-09T23:58:24.149 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:23 smithi181 ceph-mon[94189]: pgmap v17799: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-03-09T23:58:24.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32413.866897 secs 2023-03-09T23:58:24.189 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:23 smithi122 ceph-mon[93869]: pgmap v17799: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-03-09T23:58:24.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32413.866897 secs 2023-03-09T23:58:24.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:23 smithi038 ceph-mon[92422]: pgmap v17799: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-03-09T23:58:25.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:58:25.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:58:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:58:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:58:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:58:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:58:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:58:25.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:58:25.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:58:25.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:58:25.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:58:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:58:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:58:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:58:25.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:58:25.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:58:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:58:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:58:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:58:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:58:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:58:26.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32416.445040 secs 2023-03-09T23:58:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:26 smithi122 ceph-mon[93869]: pgmap v17800: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T23:58:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32416.445040 secs 2023-03-09T23:58:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:26 smithi181 ceph-mon[94189]: pgmap v17800: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T23:58:26.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32416.445040 secs 2023-03-09T23:58:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:26 smithi038 ceph-mon[92422]: pgmap v17800: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T23:58:28.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:28 smithi122 ceph-mon[93869]: pgmap v17801: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T23:58:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32418.867016 secs 2023-03-09T23:58:28.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:28 smithi181 ceph-mon[94189]: pgmap v17801: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T23:58:28.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32418.867016 secs 2023-03-09T23:58:28.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:28 smithi038 ceph-mon[92422]: pgmap v17801: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-09T23:58:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32418.867016 secs 2023-03-09T23:58:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32880.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:58:30.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:30 smithi122 ceph-mon[93869]: pgmap v17802: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T23:58:30.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:30 smithi181 ceph-mon[94189]: pgmap v17802: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T23:58:30.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:30 smithi038 ceph-mon[92422]: pgmap v17802: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-09T23:58:31.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32421.445099 secs 2023-03-09T23:58:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32421.445099 secs 2023-03-09T23:58:31.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32421.445099 secs 2023-03-09T23:58:32.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:32 smithi122 ceph-mon[93869]: pgmap v17803: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 8 op/s 2023-03-09T23:58:32.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:32 smithi181 ceph-mon[94189]: pgmap v17803: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 8 op/s 2023-03-09T23:58:32.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:32 smithi038 ceph-mon[92422]: pgmap v17803: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 8 op/s 2023-03-09T23:58:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32885.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:58:34.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32423.867139 secs 2023-03-09T23:58:34.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:33 smithi122 ceph-mon[93869]: pgmap v17804: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 34 KiB/s wr, 7 op/s 2023-03-09T23:58:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32423.867139 secs 2023-03-09T23:58:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:33 smithi181 ceph-mon[94189]: pgmap v17804: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 34 KiB/s wr, 7 op/s 2023-03-09T23:58:34.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32423.867139 secs 2023-03-09T23:58:34.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:33 smithi038 ceph-mon[92422]: pgmap v17804: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 34 KiB/s wr, 7 op/s 2023-03-09T23:58:36.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32426.445168 secs 2023-03-09T23:58:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:36 smithi122 ceph-mon[93869]: pgmap v17805: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 34 KiB/s wr, 6 op/s 2023-03-09T23:58:36.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32426.445168 secs 2023-03-09T23:58:36.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:36 smithi181 ceph-mon[94189]: pgmap v17805: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 34 KiB/s wr, 6 op/s 2023-03-09T23:58:36.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32426.445168 secs 2023-03-09T23:58:36.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:36 smithi038 ceph-mon[92422]: pgmap v17805: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 34 KiB/s wr, 6 op/s 2023-03-09T23:58:38.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:38 smithi122 ceph-mon[93869]: pgmap v17806: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 34 KiB/s wr, 10 op/s 2023-03-09T23:58:38.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32428.867263 secs 2023-03-09T23:58:38.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:38 smithi181 ceph-mon[94189]: pgmap v17806: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 34 KiB/s wr, 10 op/s 2023-03-09T23:58:38.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32428.867263 secs 2023-03-09T23:58:38.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:38 smithi038 ceph-mon[92422]: pgmap v17806: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 34 KiB/s wr, 10 op/s 2023-03-09T23:58:38.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32428.867263 secs 2023-03-09T23:58:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32890.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:58:40.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:40 smithi122 ceph-mon[93869]: pgmap v17807: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 246 KiB/s wr, 8 op/s 2023-03-09T23:58:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:58:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:58:40.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:40 smithi181 ceph-mon[94189]: pgmap v17807: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 246 KiB/s wr, 8 op/s 2023-03-09T23:58:40.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:58:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:58:40.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:40 smithi038 ceph-mon[92422]: pgmap v17807: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 246 KiB/s wr, 8 op/s 2023-03-09T23:58:40.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:58:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:58:41.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32431.445228 secs 2023-03-09T23:58:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:58:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:58:41.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32431.445228 secs 2023-03-09T23:58:41.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:58:41.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:58:41.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32431.445228 secs 2023-03-09T23:58:41.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:58:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:58:42.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:42 smithi122 ceph-mon[93869]: pgmap v17808: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 293 KiB/s rd, 246 KiB/s wr, 9 op/s 2023-03-09T23:58:42.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:42 smithi181 ceph-mon[94189]: pgmap v17808: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 293 KiB/s rd, 246 KiB/s wr, 9 op/s 2023-03-09T23:58:42.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:42 smithi038 ceph-mon[92422]: pgmap v17808: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 293 KiB/s rd, 246 KiB/s wr, 9 op/s 2023-03-09T23:58:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32895.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:58:44.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:43 smithi122 ceph-mon[93869]: pgmap v17809: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 416 KiB/s rd, 246 KiB/s wr, 9 op/s 2023-03-09T23:58:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:43 smithi181 ceph-mon[94189]: pgmap v17809: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 416 KiB/s rd, 246 KiB/s wr, 9 op/s 2023-03-09T23:58:44.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:43 smithi038 ceph-mon[92422]: pgmap v17809: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 416 KiB/s rd, 246 KiB/s wr, 9 op/s 2023-03-09T23:58:46.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32436.445290 secs 2023-03-09T23:58:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:46 smithi122 ceph-mon[93869]: pgmap v17810: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 212 KiB/s wr, 7 op/s 2023-03-09T23:58:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:46 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:58:46.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32436.445290 secs 2023-03-09T23:58:46.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:46 smithi181 ceph-mon[94189]: pgmap v17810: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 212 KiB/s wr, 7 op/s 2023-03-09T23:58:46.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:46 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:58:46.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32436.445290 secs 2023-03-09T23:58:46.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:46 smithi038 ceph-mon[92422]: pgmap v17810: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 212 KiB/s wr, 7 op/s 2023-03-09T23:58:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:46 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-09T23:58:47.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:47 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:58:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:58:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:58:47.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:47 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:58:47.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:58:47.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:58:47.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:47 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:58:47.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:58:47.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:58:48.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:48 smithi122 ceph-mon[93869]: pgmap v17811: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 212 KiB/s wr, 10 op/s 2023-03-09T23:58:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:48 smithi181 ceph-mon[94189]: pgmap v17811: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 212 KiB/s wr, 10 op/s 2023-03-09T23:58:48.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:48 smithi038 ceph-mon[92422]: pgmap v17811: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 212 KiB/s wr, 10 op/s 2023-03-09T23:58:48.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32900.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:58:50.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:50 smithi122 ceph-mon[93869]: pgmap v17812: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 342 KiB/s wr, 7 op/s 2023-03-09T23:58:50.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:50 smithi181 ceph-mon[94189]: pgmap v17812: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 342 KiB/s wr, 7 op/s 2023-03-09T23:58:50.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:50 smithi038 ceph-mon[92422]: pgmap v17812: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 342 KiB/s wr, 7 op/s 2023-03-09T23:58:51.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32441.445351 secs 2023-03-09T23:58:51.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32441.445351 secs 2023-03-09T23:58:51.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32441.445351 secs 2023-03-09T23:58:52.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:52 smithi122 ceph-mon[93869]: pgmap v17813: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 146 KiB/s wr, 9 op/s 2023-03-09T23:58:52.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:52 smithi181 ceph-mon[94189]: pgmap v17813: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 146 KiB/s wr, 9 op/s 2023-03-09T23:58:52.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:52 smithi038 ceph-mon[92422]: pgmap v17813: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 146 KiB/s wr, 9 op/s 2023-03-09T23:58:53.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:58:53.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:58:53.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:58:53.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:58:53.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:58:53.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:58:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32905.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:58:54.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:54 smithi122 ceph-mon[93869]: pgmap v17814: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 328 KiB/s rd, 171 KiB/s wr, 9 op/s 2023-03-09T23:58:54.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:54 smithi181 ceph-mon[94189]: pgmap v17814: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 328 KiB/s rd, 171 KiB/s wr, 9 op/s 2023-03-09T23:58:54.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:54 smithi038 ceph-mon[92422]: pgmap v17814: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 328 KiB/s rd, 171 KiB/s wr, 9 op/s 2023-03-09T23:58:56.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32446.445413 secs 2023-03-09T23:58:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:56 smithi122 ceph-mon[93869]: pgmap v17815: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 513 KiB/s wr, 8 op/s 2023-03-09T23:58:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:58:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:58:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32446.445413 secs 2023-03-09T23:58:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:56 smithi181 ceph-mon[94189]: pgmap v17815: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 513 KiB/s wr, 8 op/s 2023-03-09T23:58:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:58:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:58:56.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32446.445413 secs 2023-03-09T23:58:56.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:56 smithi038 ceph-mon[92422]: pgmap v17815: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 513 KiB/s wr, 8 op/s 2023-03-09T23:58:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:58:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:58:57.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:58:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:58:57.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:58:57.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:58:57.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:58:57.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:58:58.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:58:58 smithi122 ceph-mon[93869]: pgmap v17816: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 719 KiB/s rd, 690 KiB/s wr, 12 op/s 2023-03-09T23:58:58.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:58:58 smithi181 ceph-mon[94189]: pgmap v17816: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 719 KiB/s rd, 690 KiB/s wr, 12 op/s 2023-03-09T23:58:58.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:58:58 smithi038 ceph-mon[92422]: pgmap v17816: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 719 KiB/s rd, 690 KiB/s wr, 12 op/s 2023-03-09T23:58:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32910.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:59:00.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:00 smithi122 ceph-mon[93869]: pgmap v17817: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 729 KiB/s rd, 691 KiB/s wr, 11 op/s 2023-03-09T23:59:00.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:00 smithi181 ceph-mon[94189]: pgmap v17817: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 729 KiB/s rd, 691 KiB/s wr, 11 op/s 2023-03-09T23:59:00.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:00 smithi038 ceph-mon[92422]: pgmap v17817: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 729 KiB/s rd, 691 KiB/s wr, 11 op/s 2023-03-09T23:59:01.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32451.445498 secs 2023-03-09T23:59:01.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32451.445498 secs 2023-03-09T23:59:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32451.445498 secs 2023-03-09T23:59:02.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:02 smithi038 ceph-mon[92422]: pgmap v17818: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 769 KiB/s rd, 739 KiB/s wr, 13 op/s 2023-03-09T23:59:02.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:02 smithi122 ceph-mon[93869]: pgmap v17818: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 769 KiB/s rd, 739 KiB/s wr, 13 op/s 2023-03-09T23:59:02.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:02 smithi181 ceph-mon[94189]: pgmap v17818: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 769 KiB/s rd, 739 KiB/s wr, 13 op/s 2023-03-09T23:59:03.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32915.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:59:04.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:03 smithi122 ceph-mon[93869]: pgmap v17819: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 897 KiB/s rd, 868 KiB/s wr, 15 op/s 2023-03-09T23:59:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:03 smithi181 ceph-mon[94189]: pgmap v17819: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 897 KiB/s rd, 868 KiB/s wr, 15 op/s 2023-03-09T23:59:04.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:03 smithi038 ceph-mon[92422]: pgmap v17819: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 897 KiB/s rd, 868 KiB/s wr, 15 op/s 2023-03-09T23:59:06.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32456.445562 secs 2023-03-09T23:59:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:06 smithi122 ceph-mon[93869]: pgmap v17820: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 857 KiB/s rd, 843 KiB/s wr, 14 op/s 2023-03-09T23:59:06.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32456.445562 secs 2023-03-09T23:59:06.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:06 smithi181 ceph-mon[94189]: pgmap v17820: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 857 KiB/s rd, 843 KiB/s wr, 14 op/s 2023-03-09T23:59:06.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32456.445562 secs 2023-03-09T23:59:06.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:06 smithi038 ceph-mon[92422]: pgmap v17820: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 857 KiB/s rd, 843 KiB/s wr, 14 op/s 2023-03-09T23:59:08.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:08 smithi122 ceph-mon[93869]: pgmap v17821: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 612 KiB/s rd, 618 KiB/s wr, 20 op/s 2023-03-09T23:59:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:59:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:59:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32458.867980 secs 2023-03-09T23:59:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:08 smithi181 ceph-mon[94189]: pgmap v17821: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 612 KiB/s rd, 618 KiB/s wr, 20 op/s 2023-03-09T23:59:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:59:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:59:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32458.867980 secs 2023-03-09T23:59:08.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:08 smithi038 ceph-mon[92422]: pgmap v17821: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 612 KiB/s rd, 618 KiB/s wr, 20 op/s 2023-03-09T23:59:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:59:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:59:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32458.867980 secs 2023-03-09T23:59:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32920.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:59:09.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:59:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:59:09.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:59:09.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:59:09.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:59:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:59:10.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:10 smithi122 ceph-mon[93869]: pgmap v17822: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 443 KiB/s rd, 462 KiB/s wr, 26 op/s 2023-03-09T23:59:10.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:10 smithi181 ceph-mon[94189]: pgmap v17822: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 443 KiB/s rd, 462 KiB/s wr, 26 op/s 2023-03-09T23:59:10.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:10 smithi038 ceph-mon[92422]: pgmap v17822: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 443 KiB/s rd, 462 KiB/s wr, 26 op/s 2023-03-09T23:59:11.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32461.445619 secs 2023-03-09T23:59:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:11 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:59:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:11 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:59:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:11 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:59:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:11 smithi122 ceph-mon[93869]: osdmap e1329: 12 total, 12 up, 12 in 2023-03-09T23:59:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32461.445619 secs 2023-03-09T23:59:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:11 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:59:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:11 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:59:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:11 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:59:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:11 smithi181 ceph-mon[94189]: osdmap e1329: 12 total, 12 up, 12 in 2023-03-09T23:59:11.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32461.445619 secs 2023-03-09T23:59:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:11 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-09T23:59:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:11 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:59:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:11 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:59:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:11 smithi038 ceph-mon[92422]: osdmap e1329: 12 total, 12 up, 12 in 2023-03-09T23:59:12.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:12 smithi122 ceph-mon[93869]: pgmap v17824: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 436 KiB/s rd, 477 KiB/s wr, 29 op/s 2023-03-09T23:59:12.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:12 smithi181 ceph-mon[94189]: pgmap v17824: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 436 KiB/s rd, 477 KiB/s wr, 29 op/s 2023-03-09T23:59:12.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:12 smithi038 ceph-mon[92422]: pgmap v17824: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 436 KiB/s rd, 477 KiB/s wr, 29 op/s 2023-03-09T23:59:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32925.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:59:14.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32463.868127 secs 2023-03-09T23:59:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:13 smithi122 ceph-mon[93869]: pgmap v17825: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 476 KiB/s rd, 518 KiB/s wr, 27 op/s 2023-03-09T23:59:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32463.868127 secs 2023-03-09T23:59:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:13 smithi181 ceph-mon[94189]: pgmap v17825: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 476 KiB/s rd, 518 KiB/s wr, 27 op/s 2023-03-09T23:59:14.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32463.868127 secs 2023-03-09T23:59:14.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:13 smithi038 ceph-mon[92422]: pgmap v17825: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 476 KiB/s rd, 518 KiB/s wr, 27 op/s 2023-03-09T23:59:16.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:15 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:59:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:15 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:59:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32466.445720 secs 2023-03-09T23:59:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:15 smithi122 ceph-mon[93869]: pgmap v17826: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 476 KiB/s rd, 518 KiB/s wr, 26 op/s 2023-03-09T23:59:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:15 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:59:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:15 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:59:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32466.445720 secs 2023-03-09T23:59:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:15 smithi181 ceph-mon[94189]: pgmap v17826: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 476 KiB/s rd, 518 KiB/s wr, 26 op/s 2023-03-09T23:59:16.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:15 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:59:16.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:15 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:59:16.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32466.445720 secs 2023-03-09T23:59:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:15 smithi038 ceph-mon[92422]: pgmap v17826: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 476 KiB/s rd, 518 KiB/s wr, 26 op/s 2023-03-09T23:59:17.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:59:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:59:17.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:59:17.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:59:17.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:59:17.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:59:18.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:17 smithi122 ceph-mon[93869]: pgmap v17827: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 497 KiB/s wr, 22 op/s 2023-03-09T23:59:18.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:17 smithi181 ceph-mon[94189]: pgmap v17827: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 497 KiB/s wr, 22 op/s 2023-03-09T23:59:18.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:17 smithi038 ceph-mon[92422]: pgmap v17827: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 497 KiB/s wr, 22 op/s 2023-03-09T23:59:18.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32930.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:59:19.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32468.868178 secs 2023-03-09T23:59:19.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32468.868178 secs 2023-03-09T23:59:19.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32468.868178 secs 2023-03-09T23:59:20.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:19 smithi122 ceph-mon[93869]: pgmap v17828: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 474 KiB/s rd, 472 KiB/s wr, 9 op/s 2023-03-09T23:59:20.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:19 smithi181 ceph-mon[94189]: pgmap v17828: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 474 KiB/s rd, 472 KiB/s wr, 9 op/s 2023-03-09T23:59:20.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:19 smithi038 ceph-mon[92422]: pgmap v17828: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 474 KiB/s rd, 472 KiB/s wr, 9 op/s 2023-03-09T23:59:21.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32471.445780 secs 2023-03-09T23:59:21.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32471.445780 secs 2023-03-09T23:59:21.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32471.445780 secs 2023-03-09T23:59:22.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:21 smithi181 ceph-mon[94189]: pgmap v17829: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 436 KiB/s rd, 432 KiB/s wr, 11 op/s 2023-03-09T23:59:22.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:21 smithi038 ceph-mon[92422]: pgmap v17829: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 436 KiB/s rd, 432 KiB/s wr, 11 op/s 2023-03-09T23:59:22.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:21 smithi122 ceph-mon[93869]: pgmap v17829: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 436 KiB/s rd, 432 KiB/s wr, 11 op/s 2023-03-09T23:59:23.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32935.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:59:24.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32473.868291 secs 2023-03-09T23:59:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:23 smithi122 ceph-mon[93869]: osdmap e1330: 12 total, 12 up, 12 in 2023-03-09T23:59:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:23 smithi122 ceph-mon[93869]: pgmap v17831: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 120 KiB/s wr, 7 op/s 2023-03-09T23:59:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32473.868291 secs 2023-03-09T23:59:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:23 smithi181 ceph-mon[94189]: osdmap e1330: 12 total, 12 up, 12 in 2023-03-09T23:59:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:23 smithi181 ceph-mon[94189]: pgmap v17831: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 120 KiB/s wr, 7 op/s 2023-03-09T23:59:24.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32473.868291 secs 2023-03-09T23:59:24.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:23 smithi038 ceph-mon[92422]: osdmap e1330: 12 total, 12 up, 12 in 2023-03-09T23:59:24.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:23 smithi038 ceph-mon[92422]: pgmap v17831: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 120 KiB/s wr, 7 op/s 2023-03-09T23:59:25.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:59:25.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:59:25.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-09T23:59:26.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32476.445839 secs 2023-03-09T23:59:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:25 smithi122 ceph-mon[93869]: pgmap v17832: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 120 KiB/s wr, 7 op/s 2023-03-09T23:59:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:59:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:59:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:59:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32476.445839 secs 2023-03-09T23:59:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:25 smithi181 ceph-mon[94189]: pgmap v17832: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 120 KiB/s wr, 7 op/s 2023-03-09T23:59:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:59:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:59:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:59:26.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32476.445839 secs 2023-03-09T23:59:26.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:25 smithi038 ceph-mon[92422]: pgmap v17832: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 120 KiB/s wr, 7 op/s 2023-03-09T23:59:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-09T23:59:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-09T23:59:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-09T23:59:28.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:28 smithi122 ceph-mon[93869]: pgmap v17833: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T23:59:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32478.868411 secs 2023-03-09T23:59:28.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:28 smithi181 ceph-mon[94189]: pgmap v17833: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T23:59:28.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32478.868411 secs 2023-03-09T23:59:28.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:28 smithi038 ceph-mon[92422]: pgmap v17833: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-09T23:59:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32478.868411 secs 2023-03-09T23:59:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32940.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:59:30.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:30 smithi122 ceph-mon[93869]: pgmap v17834: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-03-09T23:59:30.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:30 smithi181 ceph-mon[94189]: pgmap v17834: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-03-09T23:59:30.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:30 smithi038 ceph-mon[92422]: pgmap v17834: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-03-09T23:59:31.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32481.445899 secs 2023-03-09T23:59:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32481.445899 secs 2023-03-09T23:59:31.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32481.445899 secs 2023-03-09T23:59:32.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:32 smithi122 ceph-mon[93869]: pgmap v17835: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-03-09T23:59:32.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:32 smithi181 ceph-mon[94189]: pgmap v17835: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-03-09T23:59:32.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:32 smithi038 ceph-mon[92422]: pgmap v17835: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-03-09T23:59:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32945.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:59:34.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32483.868535 secs 2023-03-09T23:59:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:33 smithi122 ceph-mon[93869]: pgmap v17836: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-03-09T23:59:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32483.868535 secs 2023-03-09T23:59:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:33 smithi181 ceph-mon[94189]: pgmap v17836: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-03-09T23:59:34.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32483.868535 secs 2023-03-09T23:59:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:33 smithi038 ceph-mon[92422]: pgmap v17836: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2023-03-09T23:59:36.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32486.445957 secs 2023-03-09T23:59:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:36 smithi122 ceph-mon[93869]: pgmap v17837: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T23:59:36.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32486.445957 secs 2023-03-09T23:59:36.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:36 smithi181 ceph-mon[94189]: pgmap v17837: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T23:59:36.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32486.445957 secs 2023-03-09T23:59:36.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:36 smithi038 ceph-mon[92422]: pgmap v17837: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-09T23:59:38.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:38 smithi122 ceph-mon[93869]: pgmap v17838: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 426 B/s wr, 9 op/s 2023-03-09T23:59:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32488.868658 secs 2023-03-09T23:59:38.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:38 smithi181 ceph-mon[94189]: pgmap v17838: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 426 B/s wr, 9 op/s 2023-03-09T23:59:38.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32488.868658 secs 2023-03-09T23:59:38.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:38 smithi038 ceph-mon[92422]: pgmap v17838: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 426 B/s wr, 9 op/s 2023-03-09T23:59:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32488.868658 secs 2023-03-09T23:59:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32950.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:59:40.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:40 smithi122 ceph-mon[93869]: pgmap v17839: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-09T23:59:40.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:40 smithi181 ceph-mon[94189]: pgmap v17839: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-09T23:59:40.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:40 smithi038 ceph-mon[92422]: pgmap v17839: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-09T23:59:41.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32491.445985 secs 2023-03-09T23:59:41.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32491.445985 secs 2023-03-09T23:59:41.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32491.445985 secs 2023-03-09T23:59:42.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:42 smithi122 ceph-mon[93869]: pgmap v17840: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 126 KiB/s wr, 8 op/s 2023-03-09T23:59:42.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:42 smithi181 ceph-mon[94189]: pgmap v17840: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 126 KiB/s wr, 8 op/s 2023-03-09T23:59:42.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:42 smithi038 ceph-mon[92422]: pgmap v17840: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 126 KiB/s wr, 8 op/s 2023-03-09T23:59:43.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32955.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:59:44.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:43 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 32493.868782 secs 2023-03-09T23:59:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:43 smithi122 ceph-mon[93869]: pgmap v17841: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 126 KiB/s wr, 7 op/s 2023-03-09T23:59:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:43 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 32493.868782 secs 2023-03-09T23:59:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:43 smithi181 ceph-mon[94189]: pgmap v17841: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 126 KiB/s wr, 7 op/s 2023-03-09T23:59:44.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:43 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 32493.868782 secs 2023-03-09T23:59:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:43 smithi038 ceph-mon[92422]: pgmap v17841: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 126 KiB/s wr, 7 op/s 2023-03-09T23:59:46.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32496.446043 secs 2023-03-09T23:59:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:46 smithi122 ceph-mon[93869]: pgmap v17842: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 126 KiB/s wr, 6 op/s 2023-03-09T23:59:46.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32496.446043 secs 2023-03-09T23:59:46.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:46 smithi181 ceph-mon[94189]: pgmap v17842: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 126 KiB/s wr, 6 op/s 2023-03-09T23:59:46.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32496.446043 secs 2023-03-09T23:59:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:46 smithi038 ceph-mon[92422]: pgmap v17842: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 126 KiB/s wr, 6 op/s 2023-03-09T23:59:47.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:59:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:59:47.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:59:47.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:59:47.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:59:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:59:48.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:48 smithi122 ceph-mon[93869]: pgmap v17843: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 249 KiB/s wr, 10 op/s 2023-03-09T23:59:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:48 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 32498.868901 secs 2023-03-09T23:59:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:48 smithi181 ceph-mon[94189]: pgmap v17843: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 249 KiB/s wr, 10 op/s 2023-03-09T23:59:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:48 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 32498.868901 secs 2023-03-09T23:59:48.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:48 smithi038 ceph-mon[92422]: pgmap v17843: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 249 KiB/s wr, 10 op/s 2023-03-09T23:59:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:48 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 32498.868901 secs 2023-03-09T23:59:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32960.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:59:50.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:50 smithi122 ceph-mon[93869]: pgmap v17844: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 249 KiB/s wr, 7 op/s 2023-03-09T23:59:50.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:50 smithi181 ceph-mon[94189]: pgmap v17844: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 249 KiB/s wr, 7 op/s 2023-03-09T23:59:50.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:50 smithi038 ceph-mon[92422]: pgmap v17844: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 249 KiB/s wr, 7 op/s 2023-03-09T23:59:51.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32501.446103 secs 2023-03-09T23:59:51.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32501.446103 secs 2023-03-09T23:59:51.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32501.446103 secs 2023-03-09T23:59:52.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:52 smithi122 ceph-mon[93869]: pgmap v17845: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 696 KiB/s rd, 487 KiB/s wr, 10 op/s 2023-03-09T23:59:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:59:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:59:52.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:52 smithi181 ceph-mon[94189]: pgmap v17845: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 696 KiB/s rd, 487 KiB/s wr, 10 op/s 2023-03-09T23:59:52.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:59:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:59:52.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:52 smithi038 ceph-mon[92422]: pgmap v17845: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 696 KiB/s rd, 487 KiB/s wr, 10 op/s 2023-03-09T23:59:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:59:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:59:53.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:59:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:59:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:53 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 32503.869021 secs 2023-03-09T23:59:53.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:59:53.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:59:53.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:53 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 32503.869021 secs 2023-03-09T23:59:53.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:59:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:59:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:53 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 32503.869021 secs 2023-03-09T23:59:53.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32965.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:59:54.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:54 smithi122 ceph-mon[93869]: pgmap v17846: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 722 KiB/s rd, 515 KiB/s wr, 9 op/s 2023-03-09T23:59:54.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:54 smithi181 ceph-mon[94189]: pgmap v17846: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 722 KiB/s rd, 515 KiB/s wr, 9 op/s 2023-03-09T23:59:54.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:54 smithi038 ceph-mon[92422]: pgmap v17846: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 722 KiB/s rd, 515 KiB/s wr, 9 op/s 2023-03-09T23:59:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32506.446164 secs 2023-03-09T23:59:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:56 smithi181 ceph-mon[94189]: pgmap v17847: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 722 KiB/s rd, 514 KiB/s wr, 9 op/s 2023-03-09T23:59:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:59:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:59:56.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32506.446164 secs 2023-03-09T23:59:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:56 smithi038 ceph-mon[92422]: pgmap v17847: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 722 KiB/s rd, 514 KiB/s wr, 9 op/s 2023-03-09T23:59:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:59:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:59:56.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32506.446164 secs 2023-03-09T23:59:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:56 smithi122 ceph-mon[93869]: pgmap v17847: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 722 KiB/s rd, 514 KiB/s wr, 9 op/s 2023-03-09T23:59:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-09T23:59:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:59:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:59:57.506 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:59:57.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:59:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:59:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-09T23:59:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:59:58.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:58 smithi038 ceph-mon[92422]: pgmap v17848: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 903 KiB/s rd, 693 KiB/s wr, 14 op/s 2023-03-09T23:59:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32508.869169 secs 2023-03-09T23:59:58.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:58 smithi122 ceph-mon[93869]: pgmap v17848: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 903 KiB/s rd, 693 KiB/s wr, 14 op/s 2023-03-09T23:59:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32508.869169 secs 2023-03-09T23:59:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:58 smithi181 ceph-mon[94189]: pgmap v17848: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 903 KiB/s rd, 693 KiB/s wr, 14 op/s 2023-03-09T23:59:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32508.869169 secs 2023-03-09T23:59:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32970.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-09T23:59:59.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:59:59.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 09 23:59:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:59:59.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:59:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 09 23:59:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-09T23:59:59.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-09T23:59:59.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 09 23:59:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:00:00.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:00 smithi122 ceph-mon[93869]: pgmap v17849: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 783 KiB/s rd, 571 KiB/s wr, 11 op/s 2023-03-10T00:00:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:00 smithi122 ceph-mon[93869]: overall HEALTH_WARN 2 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-10T00:00:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:00 smithi181 ceph-mon[94189]: pgmap v17849: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 783 KiB/s rd, 571 KiB/s wr, 11 op/s 2023-03-10T00:00:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:00 smithi181 ceph-mon[94189]: overall HEALTH_WARN 2 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-10T00:00:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:00 smithi038 ceph-mon[92422]: pgmap v17849: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 783 KiB/s rd, 571 KiB/s wr, 11 op/s 2023-03-10T00:00:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:00 smithi038 ceph-mon[92422]: overall HEALTH_WARN 2 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-10T00:00:01.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32511.446224 secs 2023-03-10T00:00:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:00:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:00:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32511.446224 secs 2023-03-10T00:00:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:00:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:00:01.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32511.446224 secs 2023-03-10T00:00:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:00:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:00:02.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:02 smithi122 ceph-mon[93869]: pgmap v17850: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 811 KiB/s rd, 598 KiB/s wr, 14 op/s 2023-03-10T00:00:02.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:02 smithi181 ceph-mon[94189]: pgmap v17850: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 811 KiB/s rd, 598 KiB/s wr, 14 op/s 2023-03-10T00:00:02.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:02 smithi038 ceph-mon[92422]: pgmap v17850: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 811 KiB/s rd, 598 KiB/s wr, 14 op/s 2023-03-10T00:00:03.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32975.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:00:04.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32513.869292 secs 2023-03-10T00:00:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:03 smithi122 ceph-mon[93869]: pgmap v17851: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 670 KiB/s rd, 657 KiB/s wr, 15 op/s 2023-03-10T00:00:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32513.869292 secs 2023-03-10T00:00:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:03 smithi181 ceph-mon[94189]: pgmap v17851: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 670 KiB/s rd, 657 KiB/s wr, 15 op/s 2023-03-10T00:00:04.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32513.869292 secs 2023-03-10T00:00:04.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:03 smithi038 ceph-mon[92422]: pgmap v17851: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 670 KiB/s rd, 657 KiB/s wr, 15 op/s 2023-03-10T00:00:06.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32516.446300 secs 2023-03-10T00:00:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:06 smithi122 ceph-mon[93869]: pgmap v17852: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 504 KiB/s wr, 13 op/s 2023-03-10T00:00:06.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32516.446300 secs 2023-03-10T00:00:06.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:06 smithi181 ceph-mon[94189]: pgmap v17852: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 504 KiB/s wr, 13 op/s 2023-03-10T00:00:06.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32516.446300 secs 2023-03-10T00:00:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:06 smithi038 ceph-mon[92422]: pgmap v17852: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 504 KiB/s wr, 13 op/s 2023-03-10T00:00:08.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:08 smithi122 ceph-mon[93869]: pgmap v17853: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 827 KiB/s wr, 21 op/s 2023-03-10T00:00:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:00:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:00:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32518.869412 secs 2023-03-10T00:00:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:08 smithi181 ceph-mon[94189]: pgmap v17853: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 827 KiB/s wr, 21 op/s 2023-03-10T00:00:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:00:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:00:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32518.869412 secs 2023-03-10T00:00:08.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:08 smithi038 ceph-mon[92422]: pgmap v17853: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 827 KiB/s wr, 21 op/s 2023-03-10T00:00:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:00:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:00:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32518.869412 secs 2023-03-10T00:00:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32980.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:00:10.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:10 smithi122 ceph-mon[93869]: pgmap v17854: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 645 KiB/s rd, 670 KiB/s wr, 27 op/s 2023-03-10T00:00:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:10 smithi122 ceph-mon[93869]: osdmap e1331: 12 total, 12 up, 12 in 2023-03-10T00:00:10.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:10 smithi181 ceph-mon[94189]: pgmap v17854: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 645 KiB/s rd, 670 KiB/s wr, 27 op/s 2023-03-10T00:00:10.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:10 smithi181 ceph-mon[94189]: osdmap e1331: 12 total, 12 up, 12 in 2023-03-10T00:00:10.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:10 smithi038 ceph-mon[92422]: pgmap v17854: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 645 KiB/s rd, 670 KiB/s wr, 27 op/s 2023-03-10T00:00:10.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:10 smithi038 ceph-mon[92422]: osdmap e1331: 12 total, 12 up, 12 in 2023-03-10T00:00:11.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32521.446384 secs 2023-03-10T00:00:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32521.446384 secs 2023-03-10T00:00:11.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32521.446384 secs 2023-03-10T00:00:12.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:12 smithi122 ceph-mon[93869]: pgmap v17856: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 32 op/s 2023-03-10T00:00:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:00:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:00:12.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:12 smithi181 ceph-mon[94189]: pgmap v17856: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 32 op/s 2023-03-10T00:00:12.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:00:12.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:00:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:12 smithi038 ceph-mon[92422]: pgmap v17856: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 32 op/s 2023-03-10T00:00:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:00:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:00:13.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:00:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:00:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32523.869536 secs 2023-03-10T00:00:13.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:00:13.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:00:13.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32523.869536 secs 2023-03-10T00:00:13.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:00:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:00:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32523.869536 secs 2023-03-10T00:00:13.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32985.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:00:14.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:14 smithi122 ceph-mon[93869]: pgmap v17857: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 750 KiB/s rd, 792 KiB/s wr, 28 op/s 2023-03-10T00:00:14.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:14 smithi181 ceph-mon[94189]: pgmap v17857: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 750 KiB/s rd, 792 KiB/s wr, 28 op/s 2023-03-10T00:00:14.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:14 smithi038 ceph-mon[92422]: pgmap v17857: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 750 KiB/s rd, 792 KiB/s wr, 28 op/s 2023-03-10T00:00:16.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32526.446445 secs 2023-03-10T00:00:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:16 smithi038 ceph-mon[92422]: pgmap v17858: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 750 KiB/s rd, 792 KiB/s wr, 28 op/s 2023-03-10T00:00:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:00:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:00:16.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32526.446445 secs 2023-03-10T00:00:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:16 smithi122 ceph-mon[93869]: pgmap v17858: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 750 KiB/s rd, 792 KiB/s wr, 28 op/s 2023-03-10T00:00:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:00:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:00:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32526.446445 secs 2023-03-10T00:00:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:16 smithi181 ceph-mon[94189]: pgmap v17858: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 750 KiB/s rd, 792 KiB/s wr, 28 op/s 2023-03-10T00:00:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:00:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:00:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:00:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:00:17.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:00:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:00:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:00:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:00:18.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:18 smithi038 ceph-mon[92422]: pgmap v17859: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 509 KiB/s rd, 525 KiB/s wr, 23 op/s 2023-03-10T00:00:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32528.869659 secs 2023-03-10T00:00:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:18 smithi038 ceph-mon[92422]: osdmap e1332: 12 total, 12 up, 12 in 2023-03-10T00:00:18.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:18 smithi122 ceph-mon[93869]: pgmap v17859: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 509 KiB/s rd, 525 KiB/s wr, 23 op/s 2023-03-10T00:00:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32528.869659 secs 2023-03-10T00:00:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:18 smithi122 ceph-mon[93869]: osdmap e1332: 12 total, 12 up, 12 in 2023-03-10T00:00:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:18 smithi181 ceph-mon[94189]: pgmap v17859: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 509 KiB/s rd, 525 KiB/s wr, 23 op/s 2023-03-10T00:00:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32528.869659 secs 2023-03-10T00:00:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:18 smithi181 ceph-mon[94189]: osdmap e1332: 12 total, 12 up, 12 in 2023-03-10T00:00:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32990.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:00:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:20 smithi181 ceph-mon[94189]: pgmap v17861: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 505 KiB/s rd, 503 KiB/s wr, 9 op/s 2023-03-10T00:00:20.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:20 smithi122 ceph-mon[93869]: pgmap v17861: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 505 KiB/s rd, 503 KiB/s wr, 9 op/s 2023-03-10T00:00:20.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:20 smithi038 ceph-mon[92422]: pgmap v17861: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 505 KiB/s rd, 503 KiB/s wr, 9 op/s 2023-03-10T00:00:21.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32531.446505 secs 2023-03-10T00:00:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32531.446505 secs 2023-03-10T00:00:21.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32531.446505 secs 2023-03-10T00:00:22.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:22 smithi122 ceph-mon[93869]: pgmap v17862: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 120 KiB/s wr, 8 op/s 2023-03-10T00:00:22.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:22 smithi181 ceph-mon[94189]: pgmap v17862: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 120 KiB/s wr, 8 op/s 2023-03-10T00:00:22.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:22 smithi038 ceph-mon[92422]: pgmap v17862: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 120 KiB/s wr, 8 op/s 2023-03-10T00:00:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 32995.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:00:24.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32533.869731 secs 2023-03-10T00:00:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:23 smithi122 ceph-mon[93869]: pgmap v17863: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 120 KiB/s wr, 7 op/s 2023-03-10T00:00:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32533.869731 secs 2023-03-10T00:00:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:23 smithi181 ceph-mon[94189]: pgmap v17863: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 120 KiB/s wr, 7 op/s 2023-03-10T00:00:24.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32533.869731 secs 2023-03-10T00:00:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:23 smithi038 ceph-mon[92422]: pgmap v17863: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 120 KiB/s wr, 7 op/s 2023-03-10T00:00:26.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32536.446564 secs 2023-03-10T00:00:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:26 smithi122 ceph-mon[93869]: pgmap v17864: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 120 KiB/s wr, 7 op/s 2023-03-10T00:00:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:26 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:00:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32536.446564 secs 2023-03-10T00:00:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:26 smithi181 ceph-mon[94189]: pgmap v17864: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 120 KiB/s wr, 7 op/s 2023-03-10T00:00:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:26 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:00:26.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32536.446564 secs 2023-03-10T00:00:26.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:26 smithi038 ceph-mon[92422]: pgmap v17864: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 120 KiB/s wr, 7 op/s 2023-03-10T00:00:26.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:26 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:00:27.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:00:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:00:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:00:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:00:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:00:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:00:27.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:00:27.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:00:27.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:00:27.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:27 smithi038 ceph-mon[92422]: pgmap v17865: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-10T00:00:27.930 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:00:27.931 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:00:27.931 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:00:27.931 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:00:27.932 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:00:27.932 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:00:27.932 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:00:27.932 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:00:27.932 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:00:27.933 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:27 smithi122 ceph-mon[93869]: pgmap v17865: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-10T00:00:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:00:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:00:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:00:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:00:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:00:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:00:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:00:27.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:00:27.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:00:27.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:27 smithi181 ceph-mon[94189]: pgmap v17865: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-10T00:00:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33000.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:00:29.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32538.869863 secs 2023-03-10T00:00:29.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32538.869863 secs 2023-03-10T00:00:29.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32538.869863 secs 2023-03-10T00:00:30.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:29 smithi122 ceph-mon[93869]: pgmap v17866: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-10T00:00:30.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:29 smithi181 ceph-mon[94189]: pgmap v17866: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-10T00:00:30.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:29 smithi038 ceph-mon[92422]: pgmap v17866: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-10T00:00:31.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32541.446624 secs 2023-03-10T00:00:31.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32541.446624 secs 2023-03-10T00:00:31.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32541.446624 secs 2023-03-10T00:00:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:31 smithi181 ceph-mon[94189]: pgmap v17867: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-10T00:00:32.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:31 smithi038 ceph-mon[92422]: pgmap v17867: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-10T00:00:32.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:31 smithi122 ceph-mon[93869]: pgmap v17867: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-10T00:00:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33005.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:00:34.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32543.869989 secs 2023-03-10T00:00:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:33 smithi122 ceph-mon[93869]: pgmap v17868: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-10T00:00:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32543.869989 secs 2023-03-10T00:00:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:33 smithi181 ceph-mon[94189]: pgmap v17868: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-10T00:00:34.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32543.869989 secs 2023-03-10T00:00:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:33 smithi038 ceph-mon[92422]: pgmap v17868: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-10T00:00:36.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32546.446710 secs 2023-03-10T00:00:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:36 smithi122 ceph-mon[93869]: pgmap v17869: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-10T00:00:36.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32546.446710 secs 2023-03-10T00:00:36.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:36 smithi181 ceph-mon[94189]: pgmap v17869: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-10T00:00:36.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32546.446710 secs 2023-03-10T00:00:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:36 smithi038 ceph-mon[92422]: pgmap v17869: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-10T00:00:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:38 smithi122 ceph-mon[93869]: pgmap v17870: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 426 B/s wr, 9 op/s 2023-03-10T00:00:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32548.870123 secs 2023-03-10T00:00:38.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:38 smithi181 ceph-mon[94189]: pgmap v17870: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 426 B/s wr, 9 op/s 2023-03-10T00:00:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32548.870123 secs 2023-03-10T00:00:38.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:38 smithi038 ceph-mon[92422]: pgmap v17870: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 426 B/s wr, 9 op/s 2023-03-10T00:00:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32548.870123 secs 2023-03-10T00:00:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33010.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:00:40.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:40 smithi122 ceph-mon[93869]: pgmap v17871: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 6 op/s 2023-03-10T00:00:40.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:40 smithi181 ceph-mon[94189]: pgmap v17871: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 6 op/s 2023-03-10T00:00:40.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:40 smithi038 ceph-mon[92422]: pgmap v17871: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 16 KiB/s wr, 6 op/s 2023-03-10T00:00:41.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32551.446771 secs 2023-03-10T00:00:41.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32551.446771 secs 2023-03-10T00:00:41.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32551.446771 secs 2023-03-10T00:00:42.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:42 smithi122 ceph-mon[93869]: pgmap v17872: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 127 KiB/s wr, 10 op/s 2023-03-10T00:00:42.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:42 smithi181 ceph-mon[94189]: pgmap v17872: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 127 KiB/s wr, 10 op/s 2023-03-10T00:00:42.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:42 smithi038 ceph-mon[92422]: pgmap v17872: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 127 KiB/s wr, 10 op/s 2023-03-10T00:00:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33015.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:00:44.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32553.870191 secs 2023-03-10T00:00:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:43 smithi122 ceph-mon[93869]: pgmap v17873: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 127 KiB/s wr, 7 op/s 2023-03-10T00:00:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32553.870191 secs 2023-03-10T00:00:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:43 smithi181 ceph-mon[94189]: pgmap v17873: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 127 KiB/s wr, 7 op/s 2023-03-10T00:00:44.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32553.870191 secs 2023-03-10T00:00:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:43 smithi038 ceph-mon[92422]: pgmap v17873: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 127 KiB/s wr, 7 op/s 2023-03-10T00:00:46.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32556.446851 secs 2023-03-10T00:00:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:46 smithi122 ceph-mon[93869]: pgmap v17874: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 127 KiB/s wr, 7 op/s 2023-03-10T00:00:46.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32556.446851 secs 2023-03-10T00:00:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:46 smithi181 ceph-mon[94189]: pgmap v17874: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 127 KiB/s wr, 7 op/s 2023-03-10T00:00:46.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32556.446851 secs 2023-03-10T00:00:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:46 smithi038 ceph-mon[92422]: pgmap v17874: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 127 KiB/s wr, 7 op/s 2023-03-10T00:00:48.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:48 smithi122 ceph-mon[93869]: pgmap v17875: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 127 KiB/s wr, 10 op/s 2023-03-10T00:00:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32558.870309 secs 2023-03-10T00:00:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:48 smithi181 ceph-mon[94189]: pgmap v17875: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 127 KiB/s wr, 10 op/s 2023-03-10T00:00:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32558.870309 secs 2023-03-10T00:00:48.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:48 smithi038 ceph-mon[92422]: pgmap v17875: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 127 KiB/s wr, 10 op/s 2023-03-10T00:00:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32558.870309 secs 2023-03-10T00:00:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33020.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:00:49.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:00:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:00:49.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:00:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:00:49.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:00:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:00:50.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:50 smithi122 ceph-mon[93869]: pgmap v17876: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 535 KiB/s rd, 453 KiB/s wr, 8 op/s 2023-03-10T00:00:50.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:50 smithi181 ceph-mon[94189]: pgmap v17876: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 535 KiB/s rd, 453 KiB/s wr, 8 op/s 2023-03-10T00:00:50.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:50 smithi038 ceph-mon[92422]: pgmap v17876: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 535 KiB/s rd, 453 KiB/s wr, 8 op/s 2023-03-10T00:00:51.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32561.446912 secs 2023-03-10T00:00:51.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32561.446912 secs 2023-03-10T00:00:51.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32561.446912 secs 2023-03-10T00:00:52.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:52 smithi122 ceph-mon[93869]: pgmap v17877: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 547 KiB/s rd, 463 KiB/s wr, 10 op/s 2023-03-10T00:00:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:00:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:00:52.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:52 smithi181 ceph-mon[94189]: pgmap v17877: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 547 KiB/s rd, 463 KiB/s wr, 10 op/s 2023-03-10T00:00:52.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:00:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:00:52.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:52 smithi038 ceph-mon[92422]: pgmap v17877: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 547 KiB/s rd, 463 KiB/s wr, 10 op/s 2023-03-10T00:00:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:00:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:00:53.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33025.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:00:54.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:53 smithi038 ceph-mon[92422]: pgmap v17878: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 517 KiB/s wr, 8 op/s 2023-03-10T00:00:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:53 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:00:54.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:53 smithi122 ceph-mon[93869]: pgmap v17878: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 517 KiB/s wr, 8 op/s 2023-03-10T00:00:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:53 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:00:54.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:53 smithi181 ceph-mon[94189]: pgmap v17878: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 517 KiB/s wr, 8 op/s 2023-03-10T00:00:54.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:53 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:00:55.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:54 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:00:55.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:54 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:00:55.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:54 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:00:55.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:54 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:00:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:54 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:00:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:54 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:00:55.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:54 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:00:55.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:54 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:00:55.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:54 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:00:56.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:00:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:00:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32566.446978 secs 2023-03-10T00:00:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:56 smithi122 ceph-mon[93869]: pgmap v17879: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 679 KiB/s rd, 516 KiB/s wr, 8 op/s 2023-03-10T00:00:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:00:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:00:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32566.446978 secs 2023-03-10T00:00:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:56 smithi181 ceph-mon[94189]: pgmap v17879: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 679 KiB/s rd, 516 KiB/s wr, 8 op/s 2023-03-10T00:00:56.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:00:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:00:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32566.446978 secs 2023-03-10T00:00:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:56 smithi038 ceph-mon[92422]: pgmap v17879: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 679 KiB/s rd, 516 KiB/s wr, 8 op/s 2023-03-10T00:00:57.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:00:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:00:57.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:00:57.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:00:57.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:00:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:00:58.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:00:58 smithi122 ceph-mon[93869]: pgmap v17880: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 862 KiB/s rd, 697 KiB/s wr, 12 op/s 2023-03-10T00:00:58.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:00:58 smithi181 ceph-mon[94189]: pgmap v17880: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 862 KiB/s rd, 697 KiB/s wr, 12 op/s 2023-03-10T00:00:58.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:00:58 smithi038 ceph-mon[92422]: pgmap v17880: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 862 KiB/s rd, 697 KiB/s wr, 12 op/s 2023-03-10T00:00:58.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33030.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:01:00.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:00 smithi122 ceph-mon[93869]: pgmap v17881: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 872 KiB/s rd, 697 KiB/s wr, 10 op/s 2023-03-10T00:01:00.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:00 smithi181 ceph-mon[94189]: pgmap v17881: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 872 KiB/s rd, 697 KiB/s wr, 10 op/s 2023-03-10T00:01:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:00 smithi038 ceph-mon[92422]: pgmap v17881: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 872 KiB/s rd, 697 KiB/s wr, 10 op/s 2023-03-10T00:01:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32571.447040 secs 2023-03-10T00:01:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:01:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:01:01.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32571.447040 secs 2023-03-10T00:01:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:01:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:01:01.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32571.447040 secs 2023-03-10T00:01:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:01:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:01:02.204 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:02 smithi122 ceph-mon[93869]: pgmap v17882: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 763 KiB/s rd, 668 KiB/s wr, 13 op/s 2023-03-10T00:01:02.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:02 smithi181 ceph-mon[94189]: pgmap v17882: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 763 KiB/s rd, 668 KiB/s wr, 13 op/s 2023-03-10T00:01:02.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:02 smithi038 ceph-mon[92422]: pgmap v17882: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 763 KiB/s rd, 668 KiB/s wr, 13 op/s 2023-03-10T00:01:03.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33035.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:01:04.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:03 smithi038 ceph-mon[92422]: pgmap v17883: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 765 KiB/s rd, 670 KiB/s wr, 13 op/s 2023-03-10T00:01:04.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:03 smithi122 ceph-mon[93869]: pgmap v17883: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 765 KiB/s rd, 670 KiB/s wr, 13 op/s 2023-03-10T00:01:04.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:03 smithi181 ceph-mon[94189]: pgmap v17883: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 765 KiB/s rd, 670 KiB/s wr, 13 op/s 2023-03-10T00:01:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32576.447097 secs 2023-03-10T00:01:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:06 smithi122 ceph-mon[93869]: pgmap v17884: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 601 KiB/s rd, 506 KiB/s wr, 12 op/s 2023-03-10T00:01:06.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32576.447097 secs 2023-03-10T00:01:06.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:06 smithi181 ceph-mon[94189]: pgmap v17884: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 601 KiB/s rd, 506 KiB/s wr, 12 op/s 2023-03-10T00:01:06.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32576.447097 secs 2023-03-10T00:01:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:06 smithi038 ceph-mon[92422]: pgmap v17884: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 601 KiB/s rd, 506 KiB/s wr, 12 op/s 2023-03-10T00:01:08.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:08 smithi122 ceph-mon[93869]: pgmap v17885: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 617 KiB/s rd, 624 KiB/s wr, 18 op/s 2023-03-10T00:01:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:01:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:01:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32578.870747 secs 2023-03-10T00:01:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:08 smithi181 ceph-mon[94189]: pgmap v17885: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 617 KiB/s rd, 624 KiB/s wr, 18 op/s 2023-03-10T00:01:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:01:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:01:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32578.870747 secs 2023-03-10T00:01:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:08 smithi038 ceph-mon[92422]: pgmap v17885: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 617 KiB/s rd, 624 KiB/s wr, 18 op/s 2023-03-10T00:01:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:01:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:01:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32578.870747 secs 2023-03-10T00:01:08.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33040.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:01:10.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:10 smithi122 ceph-mon[93869]: pgmap v17886: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 465 KiB/s wr, 25 op/s 2023-03-10T00:01:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:10 smithi122 ceph-mon[93869]: osdmap e1333: 12 total, 12 up, 12 in 2023-03-10T00:01:10.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:10 smithi181 ceph-mon[94189]: pgmap v17886: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 465 KiB/s wr, 25 op/s 2023-03-10T00:01:10.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:10 smithi181 ceph-mon[94189]: osdmap e1333: 12 total, 12 up, 12 in 2023-03-10T00:01:10.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:10 smithi038 ceph-mon[92422]: pgmap v17886: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 465 KiB/s wr, 25 op/s 2023-03-10T00:01:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:10 smithi038 ceph-mon[92422]: osdmap e1333: 12 total, 12 up, 12 in 2023-03-10T00:01:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32581.447172 secs 2023-03-10T00:01:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:11 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:01:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:11 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:01:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:11 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:01:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32581.447172 secs 2023-03-10T00:01:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:11 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:01:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:11 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:01:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:11 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:01:11.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32581.447172 secs 2023-03-10T00:01:11.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:11 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:01:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:11 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:01:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:11 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:01:12.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:12 smithi122 ceph-mon[93869]: pgmap v17888: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 513 KiB/s rd, 555 KiB/s wr, 29 op/s 2023-03-10T00:01:12.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:12 smithi181 ceph-mon[94189]: pgmap v17888: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 513 KiB/s rd, 555 KiB/s wr, 29 op/s 2023-03-10T00:01:12.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:12 smithi038 ceph-mon[92422]: pgmap v17888: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 513 KiB/s rd, 555 KiB/s wr, 29 op/s 2023-03-10T00:01:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33045.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:01:14.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32583.870840 secs 2023-03-10T00:01:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:13 smithi038 ceph-mon[92422]: pgmap v17889: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 521 KiB/s wr, 26 op/s 2023-03-10T00:01:14.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32583.870840 secs 2023-03-10T00:01:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:13 smithi122 ceph-mon[93869]: pgmap v17889: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 521 KiB/s wr, 26 op/s 2023-03-10T00:01:14.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32583.870840 secs 2023-03-10T00:01:14.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:13 smithi181 ceph-mon[94189]: pgmap v17889: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 521 KiB/s wr, 26 op/s 2023-03-10T00:01:16.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:15 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:01:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:15 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:01:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32586.447231 secs 2023-03-10T00:01:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:15 smithi038 ceph-mon[92422]: pgmap v17890: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 521 KiB/s wr, 26 op/s 2023-03-10T00:01:16.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:15 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:01:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:15 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:01:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32586.447231 secs 2023-03-10T00:01:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:15 smithi122 ceph-mon[93869]: pgmap v17890: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 521 KiB/s wr, 26 op/s 2023-03-10T00:01:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:15 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:01:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:15 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:01:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32586.447231 secs 2023-03-10T00:01:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:15 smithi181 ceph-mon[94189]: pgmap v17890: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 521 KiB/s wr, 26 op/s 2023-03-10T00:01:17.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:01:17.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:01:17.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:01:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:01:17.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:01:17.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:01:18.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:18 smithi122 ceph-mon[93869]: pgmap v17891: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 452 KiB/s rd, 467 KiB/s wr, 22 op/s 2023-03-10T00:01:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:18 smithi122 ceph-mon[93869]: osdmap e1334: 12 total, 12 up, 12 in 2023-03-10T00:01:18.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:18 smithi181 ceph-mon[94189]: pgmap v17891: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 452 KiB/s rd, 467 KiB/s wr, 22 op/s 2023-03-10T00:01:18.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:18 smithi181 ceph-mon[94189]: osdmap e1334: 12 total, 12 up, 12 in 2023-03-10T00:01:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:18 smithi038 ceph-mon[92422]: pgmap v17891: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 452 KiB/s rd, 467 KiB/s wr, 22 op/s 2023-03-10T00:01:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:18 smithi038 ceph-mon[92422]: osdmap e1334: 12 total, 12 up, 12 in 2023-03-10T00:01:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33050.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:01:19.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:19 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32588.870962 secs 2023-03-10T00:01:19.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:19 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32588.870962 secs 2023-03-10T00:01:19.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:19 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32588.870962 secs 2023-03-10T00:01:20.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:20 smithi122 ceph-mon[93869]: pgmap v17893: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 478 KiB/s wr, 9 op/s 2023-03-10T00:01:20.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:20 smithi181 ceph-mon[94189]: pgmap v17893: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 478 KiB/s wr, 9 op/s 2023-03-10T00:01:20.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:20 smithi038 ceph-mon[92422]: pgmap v17893: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 478 KiB/s wr, 9 op/s 2023-03-10T00:01:21.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32591.447291 secs 2023-03-10T00:01:21.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32591.447291 secs 2023-03-10T00:01:21.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32591.447291 secs 2023-03-10T00:01:22.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:22 smithi122 ceph-mon[93869]: pgmap v17894: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 121 KiB/s wr, 7 op/s 2023-03-10T00:01:22.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:22 smithi181 ceph-mon[94189]: pgmap v17894: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 121 KiB/s wr, 7 op/s 2023-03-10T00:01:22.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:22 smithi038 ceph-mon[92422]: pgmap v17894: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 121 KiB/s wr, 7 op/s 2023-03-10T00:01:23.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33055.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:01:24.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32593.871098 secs 2023-03-10T00:01:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:23 smithi038 ceph-mon[92422]: pgmap v17895: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 120 KiB/s wr, 7 op/s 2023-03-10T00:01:24.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32593.871098 secs 2023-03-10T00:01:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:23 smithi122 ceph-mon[93869]: pgmap v17895: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 120 KiB/s wr, 7 op/s 2023-03-10T00:01:24.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32593.871098 secs 2023-03-10T00:01:24.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:23 smithi181 ceph-mon[94189]: pgmap v17895: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 120 KiB/s wr, 7 op/s 2023-03-10T00:01:26.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32596.447349 secs 2023-03-10T00:01:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:26 smithi122 ceph-mon[93869]: pgmap v17896: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 120 KiB/s wr, 7 op/s 2023-03-10T00:01:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32596.447349 secs 2023-03-10T00:01:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:26 smithi181 ceph-mon[94189]: pgmap v17896: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 120 KiB/s wr, 7 op/s 2023-03-10T00:01:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32596.447349 secs 2023-03-10T00:01:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:26 smithi038 ceph-mon[92422]: pgmap v17896: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 120 KiB/s wr, 7 op/s 2023-03-10T00:01:27.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:01:27.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:01:27.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:01:28.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:28 smithi122 ceph-mon[93869]: pgmap v17897: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 33 KiB/s wr, 7 op/s 2023-03-10T00:01:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32598.871199 secs 2023-03-10T00:01:28.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:28 smithi181 ceph-mon[94189]: pgmap v17897: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 33 KiB/s wr, 7 op/s 2023-03-10T00:01:28.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32598.871199 secs 2023-03-10T00:01:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:28 smithi038 ceph-mon[92422]: pgmap v17897: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 33 KiB/s wr, 7 op/s 2023-03-10T00:01:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32598.871199 secs 2023-03-10T00:01:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33060.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:01:30.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:30 smithi122 ceph-mon[93869]: pgmap v17898: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 6 op/s 2023-03-10T00:01:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:30 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:01:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:30 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:01:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:30 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:01:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:30 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:01:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:30 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:01:30.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:30 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:01:30.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:30 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:01:30.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:30 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:01:30.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:30 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:01:30.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:30 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:01:30.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:30 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:01:30.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:30 smithi181 ceph-mon[94189]: pgmap v17898: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 6 op/s 2023-03-10T00:01:30.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:30 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:01:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:30 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:01:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:30 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:01:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:30 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:01:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:30 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:01:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:30 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:01:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:30 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:01:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:30 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:01:30.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:30 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:01:30.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:30 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:01:30.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:30 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:01:30.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:30 smithi038 ceph-mon[92422]: pgmap v17898: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 30 KiB/s wr, 6 op/s 2023-03-10T00:01:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:30 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:01:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:30 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:01:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:30 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:01:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:30 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:01:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:30 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:01:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:30 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:01:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:30 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:01:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:30 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:01:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:30 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:01:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:30 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:01:30.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:30 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:01:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32601.447411 secs 2023-03-10T00:01:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:31 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 183.1M 2023-03-10T00:01:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:31 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi181 to 192047161: error parsing value: Value '192047161' is below minimum 939524096 2023-03-10T00:01:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:01:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:01:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:01:31.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:01:31.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:31 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-10T00:01:31.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:31 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-10T00:01:31.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:01:31.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:01:31.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:01:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32601.447411 secs 2023-03-10T00:01:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:31 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 183.1M 2023-03-10T00:01:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:31 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi181 to 192047161: error parsing value: Value '192047161' is below minimum 939524096 2023-03-10T00:01:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:01:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:01:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:01:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:01:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:31 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-10T00:01:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:31 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-10T00:01:31.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:01:31.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:01:31.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:01:31.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32601.447411 secs 2023-03-10T00:01:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:31 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 183.1M 2023-03-10T00:01:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:31 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi181 to 192047161: error parsing value: Value '192047161' is below minimum 939524096 2023-03-10T00:01:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:01:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:01:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:01:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:01:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:31 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-10T00:01:31.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:31 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-10T00:01:31.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:01:31.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:01:31.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:01:32.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:32 smithi038 ceph-mon[92422]: pgmap v17899: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-10T00:01:32.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:32 smithi122 ceph-mon[93869]: pgmap v17899: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-10T00:01:32.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:32 smithi181 ceph-mon[94189]: pgmap v17899: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-10T00:01:33.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33065.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:01:34.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32603.871311 secs 2023-03-10T00:01:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:33 smithi038 ceph-mon[92422]: pgmap v17900: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-10T00:01:34.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32603.871311 secs 2023-03-10T00:01:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:33 smithi122 ceph-mon[93869]: pgmap v17900: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-10T00:01:34.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32603.871311 secs 2023-03-10T00:01:34.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:33 smithi181 ceph-mon[94189]: pgmap v17900: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-10T00:01:36.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32606.447497 secs 2023-03-10T00:01:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:36 smithi122 ceph-mon[93869]: pgmap v17901: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-10T00:01:36.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32606.447497 secs 2023-03-10T00:01:36.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:36 smithi181 ceph-mon[94189]: pgmap v17901: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-10T00:01:36.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32606.447497 secs 2023-03-10T00:01:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:36 smithi038 ceph-mon[92422]: pgmap v17901: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-10T00:01:37.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:01:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:01:37.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:01:37.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:01:37.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:01:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:01:38.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:38 smithi122 ceph-mon[93869]: pgmap v17902: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 8 op/s 2023-03-10T00:01:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32608.871425 secs 2023-03-10T00:01:38.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:38 smithi181 ceph-mon[94189]: pgmap v17902: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 8 op/s 2023-03-10T00:01:38.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32608.871425 secs 2023-03-10T00:01:38.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:38 smithi038 ceph-mon[92422]: pgmap v17902: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 8 op/s 2023-03-10T00:01:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32608.871425 secs 2023-03-10T00:01:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33070.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:01:40.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:40 smithi181 ceph-mon[94189]: pgmap v17903: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 143 KiB/s wr, 6 op/s 2023-03-10T00:01:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:01:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:01:40.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:40 smithi038 ceph-mon[92422]: pgmap v17903: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 143 KiB/s wr, 6 op/s 2023-03-10T00:01:40.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:01:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:01:40.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:40 smithi122 ceph-mon[93869]: pgmap v17903: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 145 KiB/s rd, 143 KiB/s wr, 6 op/s 2023-03-10T00:01:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:01:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:01:41.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32611.447557 secs 2023-03-10T00:01:41.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32611.447557 secs 2023-03-10T00:01:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32611.447557 secs 2023-03-10T00:01:42.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:42 smithi038 ceph-mon[92422]: pgmap v17904: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 143 KiB/s wr, 8 op/s 2023-03-10T00:01:42.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:42 smithi122 ceph-mon[93869]: pgmap v17904: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 143 KiB/s wr, 8 op/s 2023-03-10T00:01:42.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:42 smithi181 ceph-mon[94189]: pgmap v17904: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 143 KiB/s wr, 8 op/s 2023-03-10T00:01:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33075.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:01:44.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:44 smithi122 ceph-mon[93869]: pgmap v17905: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 314 KiB/s wr, 7 op/s 2023-03-10T00:01:44.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:44 smithi181 ceph-mon[94189]: pgmap v17905: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 314 KiB/s wr, 7 op/s 2023-03-10T00:01:44.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:44 smithi038 ceph-mon[92422]: pgmap v17905: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 314 KiB/s wr, 7 op/s 2023-03-10T00:01:46.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32616.447617 secs 2023-03-10T00:01:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:46 smithi122 ceph-mon[93869]: pgmap v17906: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 567 KiB/s rd, 514 KiB/s wr, 7 op/s 2023-03-10T00:01:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:46 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:01:46.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32616.447617 secs 2023-03-10T00:01:46.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:46 smithi181 ceph-mon[94189]: pgmap v17906: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 567 KiB/s rd, 514 KiB/s wr, 7 op/s 2023-03-10T00:01:46.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:46 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:01:46.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32616.447617 secs 2023-03-10T00:01:46.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:46 smithi038 ceph-mon[92422]: pgmap v17906: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 567 KiB/s rd, 514 KiB/s wr, 7 op/s 2023-03-10T00:01:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:46 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:01:47.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:47 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:01:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:01:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:01:47.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:47 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:01:47.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:01:47.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:01:47.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:47 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:01:47.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:01:47.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:01:48.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:48 smithi122 ceph-mon[93869]: pgmap v17907: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 594 KiB/s rd, 541 KiB/s wr, 10 op/s 2023-03-10T00:01:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:01:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:01:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:48 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 32618.871683 secs 2023-03-10T00:01:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:48 smithi181 ceph-mon[94189]: pgmap v17907: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 594 KiB/s rd, 541 KiB/s wr, 10 op/s 2023-03-10T00:01:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:01:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:01:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:48 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 32618.871683 secs 2023-03-10T00:01:48.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:48 smithi038 ceph-mon[92422]: pgmap v17907: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 594 KiB/s rd, 541 KiB/s wr, 10 op/s 2023-03-10T00:01:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:01:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:01:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:48 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 32618.871683 secs 2023-03-10T00:01:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33080.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:01:49.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:01:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:01:49.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:01:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:01:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:01:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:01:50.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:50 smithi038 ceph-mon[92422]: pgmap v17908: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 596 KiB/s rd, 544 KiB/s wr, 8 op/s 2023-03-10T00:01:50.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:50 smithi122 ceph-mon[93869]: pgmap v17908: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 596 KiB/s rd, 544 KiB/s wr, 8 op/s 2023-03-10T00:01:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:50 smithi181 ceph-mon[94189]: pgmap v17908: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 596 KiB/s rd, 544 KiB/s wr, 8 op/s 2023-03-10T00:01:51.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32621.447677 secs 2023-03-10T00:01:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:01:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:01:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32621.447677 secs 2023-03-10T00:01:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:01:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:01:51.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32621.447677 secs 2023-03-10T00:01:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:01:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:01:52.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:52 smithi122 ceph-mon[93869]: pgmap v17909: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 589 KiB/s rd, 534 KiB/s wr, 11 op/s 2023-03-10T00:01:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:52 smithi181 ceph-mon[94189]: pgmap v17909: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 589 KiB/s rd, 534 KiB/s wr, 11 op/s 2023-03-10T00:01:52.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:52 smithi038 ceph-mon[92422]: pgmap v17909: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 589 KiB/s rd, 534 KiB/s wr, 11 op/s 2023-03-10T00:01:53.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32623.871808 secs 2023-03-10T00:01:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:53 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:01:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32623.871808 secs 2023-03-10T00:01:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:53 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:01:53.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32623.871808 secs 2023-03-10T00:01:53.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:53 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:01:53.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33085.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:01:54.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:54 smithi122 ceph-mon[93869]: pgmap v17910: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 534 KiB/s wr, 10 op/s 2023-03-10T00:01:54.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:54 smithi181 ceph-mon[94189]: pgmap v17910: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 534 KiB/s wr, 10 op/s 2023-03-10T00:01:54.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:54 smithi038 ceph-mon[92422]: pgmap v17910: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 534 KiB/s wr, 10 op/s 2023-03-10T00:01:55.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:01:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:01:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:01:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:01:55.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:01:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:01:56.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32626.447736 secs 2023-03-10T00:01:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:56 smithi122 ceph-mon[93869]: pgmap v17911: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 501 KiB/s wr, 9 op/s 2023-03-10T00:01:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:01:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:01:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32626.447736 secs 2023-03-10T00:01:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:56 smithi181 ceph-mon[94189]: pgmap v17911: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 501 KiB/s wr, 9 op/s 2023-03-10T00:01:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:01:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:01:56.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32626.447736 secs 2023-03-10T00:01:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:56 smithi038 ceph-mon[92422]: pgmap v17911: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 501 KiB/s wr, 9 op/s 2023-03-10T00:01:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:01:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:01:57.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:01:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:01:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:01:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:01:57.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:01:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:01:58.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:58 smithi122 ceph-mon[93869]: pgmap v17912: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 583 KiB/s rd, 568 KiB/s wr, 13 op/s 2023-03-10T00:01:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:01:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32628.871933 secs 2023-03-10T00:01:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:58 smithi181 ceph-mon[94189]: pgmap v17912: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 583 KiB/s rd, 568 KiB/s wr, 13 op/s 2023-03-10T00:01:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:01:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32628.871933 secs 2023-03-10T00:01:58.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:58 smithi038 ceph-mon[92422]: pgmap v17912: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 583 KiB/s rd, 568 KiB/s wr, 13 op/s 2023-03-10T00:01:58.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:01:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32628.871933 secs 2023-03-10T00:01:58.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33090.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:02:00.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:00 smithi122 ceph-mon[93869]: pgmap v17913: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 645 KiB/s rd, 632 KiB/s wr, 12 op/s 2023-03-10T00:02:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:02:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:02:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:00 smithi181 ceph-mon[94189]: pgmap v17913: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 645 KiB/s rd, 632 KiB/s wr, 12 op/s 2023-03-10T00:02:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:02:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:02:00.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:00 smithi038 ceph-mon[92422]: pgmap v17913: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 645 KiB/s rd, 632 KiB/s wr, 12 op/s 2023-03-10T00:02:00.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:02:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:02:01.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32631.447796 secs 2023-03-10T00:02:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:02:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:02:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32631.447796 secs 2023-03-10T00:02:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:02:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:02:01.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32631.447796 secs 2023-03-10T00:02:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:02:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:02:02.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:02 smithi122 ceph-mon[93869]: pgmap v17914: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 670 KiB/s rd, 677 KiB/s wr, 16 op/s 2023-03-10T00:02:02.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:02 smithi181 ceph-mon[94189]: pgmap v17914: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 670 KiB/s rd, 677 KiB/s wr, 16 op/s 2023-03-10T00:02:02.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:02 smithi038 ceph-mon[92422]: pgmap v17914: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 670 KiB/s rd, 677 KiB/s wr, 16 op/s 2023-03-10T00:02:03.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33095.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:02:04.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:04 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32633.872052 secs 2023-03-10T00:02:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:04 smithi122 ceph-mon[93869]: pgmap v17915: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 542 KiB/s rd, 566 KiB/s wr, 27 op/s 2023-03-10T00:02:04.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:04 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32633.872052 secs 2023-03-10T00:02:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:04 smithi181 ceph-mon[94189]: pgmap v17915: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 542 KiB/s rd, 566 KiB/s wr, 27 op/s 2023-03-10T00:02:04.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:04 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32633.872052 secs 2023-03-10T00:02:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:04 smithi038 ceph-mon[92422]: pgmap v17915: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 542 KiB/s rd, 566 KiB/s wr, 27 op/s 2023-03-10T00:02:06.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:06 smithi122 ceph-mon[93869]: osdmap e1335: 12 total, 12 up, 12 in 2023-03-10T00:02:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32636.447858 secs 2023-03-10T00:02:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:06 smithi122 ceph-mon[93869]: pgmap v17917: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 773 KiB/s rd, 815 KiB/s wr, 34 op/s 2023-03-10T00:02:06.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:06 smithi181 ceph-mon[94189]: osdmap e1335: 12 total, 12 up, 12 in 2023-03-10T00:02:06.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32636.447858 secs 2023-03-10T00:02:06.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:06 smithi181 ceph-mon[94189]: pgmap v17917: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 773 KiB/s rd, 815 KiB/s wr, 34 op/s 2023-03-10T00:02:06.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:06 smithi038 ceph-mon[92422]: osdmap e1335: 12 total, 12 up, 12 in 2023-03-10T00:02:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32636.447858 secs 2023-03-10T00:02:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:06 smithi038 ceph-mon[92422]: pgmap v17917: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 773 KiB/s rd, 815 KiB/s wr, 34 op/s 2023-03-10T00:02:08.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:02:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:02:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:08 smithi122 ceph-mon[93869]: pgmap v17918: 129 pgs: 1 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 120 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 611 KiB/s rd, 654 KiB/s wr, 40 op/s 2023-03-10T00:02:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:02:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:02:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:08 smithi181 ceph-mon[94189]: pgmap v17918: 129 pgs: 1 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 120 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 611 KiB/s rd, 654 KiB/s wr, 40 op/s 2023-03-10T00:02:08.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:02:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:02:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:08 smithi038 ceph-mon[92422]: pgmap v17918: 129 pgs: 1 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 120 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 611 KiB/s rd, 654 KiB/s wr, 40 op/s 2023-03-10T00:02:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33100.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:02:09.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:09 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32638.872189 secs 2023-03-10T00:02:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:02:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:02:09.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:09 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32638.872189 secs 2023-03-10T00:02:09.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:02:09.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:02:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:09 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32638.872189 secs 2023-03-10T00:02:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:02:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:02:10.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:10 smithi122 ceph-mon[93869]: pgmap v17919: 129 pgs: 1 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 120 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 546 KiB/s wr, 40 op/s 2023-03-10T00:02:10.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:10 smithi181 ceph-mon[94189]: pgmap v17919: 129 pgs: 1 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 120 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 546 KiB/s wr, 40 op/s 2023-03-10T00:02:10.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:10 smithi038 ceph-mon[92422]: pgmap v17919: 129 pgs: 1 active+clean+snaptrim, 8 active+clean+snaptrim_wait, 120 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 546 KiB/s wr, 40 op/s 2023-03-10T00:02:11.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32641.447917 secs 2023-03-10T00:02:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32641.447917 secs 2023-03-10T00:02:11.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32641.447917 secs 2023-03-10T00:02:12.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:12 smithi122 ceph-mon[93869]: pgmap v17920: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 639 KiB/s rd, 657 KiB/s wr, 38 op/s 2023-03-10T00:02:12.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:12 smithi181 ceph-mon[94189]: pgmap v17920: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 639 KiB/s rd, 657 KiB/s wr, 38 op/s 2023-03-10T00:02:12.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:12 smithi038 ceph-mon[92422]: pgmap v17920: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 639 KiB/s rd, 657 KiB/s wr, 38 op/s 2023-03-10T00:02:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33105.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:02:14.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32643.872284 secs 2023-03-10T00:02:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:13 smithi038 ceph-mon[92422]: osdmap e1336: 12 total, 12 up, 12 in 2023-03-10T00:02:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:13 smithi038 ceph-mon[92422]: pgmap v17922: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 402 KiB/s wr, 22 op/s 2023-03-10T00:02:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32643.872284 secs 2023-03-10T00:02:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:13 smithi122 ceph-mon[93869]: osdmap e1336: 12 total, 12 up, 12 in 2023-03-10T00:02:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:13 smithi122 ceph-mon[93869]: pgmap v17922: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 402 KiB/s wr, 22 op/s 2023-03-10T00:02:14.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32643.872284 secs 2023-03-10T00:02:14.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:13 smithi181 ceph-mon[94189]: osdmap e1336: 12 total, 12 up, 12 in 2023-03-10T00:02:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:13 smithi181 ceph-mon[94189]: pgmap v17922: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 402 KiB/s wr, 22 op/s 2023-03-10T00:02:16.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32646.447981 secs 2023-03-10T00:02:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:16 smithi122 ceph-mon[93869]: pgmap v17923: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 328 KiB/s rd, 328 KiB/s wr, 18 op/s 2023-03-10T00:02:16.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32646.447981 secs 2023-03-10T00:02:16.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:16 smithi181 ceph-mon[94189]: pgmap v17923: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 328 KiB/s rd, 328 KiB/s wr, 18 op/s 2023-03-10T00:02:16.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32646.447981 secs 2023-03-10T00:02:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:16 smithi038 ceph-mon[92422]: pgmap v17923: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 328 KiB/s rd, 328 KiB/s wr, 18 op/s 2023-03-10T00:02:18.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:18 smithi038 ceph-mon[92422]: pgmap v17924: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 169 KiB/s wr, 8 op/s 2023-03-10T00:02:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32648.872401 secs 2023-03-10T00:02:18.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:18 smithi122 ceph-mon[93869]: pgmap v17924: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 169 KiB/s wr, 8 op/s 2023-03-10T00:02:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32648.872401 secs 2023-03-10T00:02:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:18 smithi181 ceph-mon[94189]: pgmap v17924: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 169 KiB/s wr, 8 op/s 2023-03-10T00:02:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32648.872401 secs 2023-03-10T00:02:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33110.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:02:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:20 smithi038 ceph-mon[92422]: pgmap v17925: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 169 KiB/s wr, 7 op/s 2023-03-10T00:02:20.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:20 smithi122 ceph-mon[93869]: pgmap v17925: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 169 KiB/s wr, 7 op/s 2023-03-10T00:02:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:20 smithi181 ceph-mon[94189]: pgmap v17925: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 169 KiB/s wr, 7 op/s 2023-03-10T00:02:21.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32651.448044 secs 2023-03-10T00:02:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32651.448044 secs 2023-03-10T00:02:21.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32651.448044 secs 2023-03-10T00:02:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:22 smithi122 ceph-mon[93869]: pgmap v17926: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 145 KiB/s wr, 6 op/s 2023-03-10T00:02:22.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:22 smithi181 ceph-mon[94189]: pgmap v17926: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 145 KiB/s wr, 6 op/s 2023-03-10T00:02:22.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:22 smithi038 ceph-mon[92422]: pgmap v17926: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 145 KiB/s wr, 6 op/s 2023-03-10T00:02:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33115.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:02:24.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:24 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32653.872499 secs 2023-03-10T00:02:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:24 smithi122 ceph-mon[93869]: pgmap v17927: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 143 KiB/s wr, 6 op/s 2023-03-10T00:02:24.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:24 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32653.872499 secs 2023-03-10T00:02:24.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:24 smithi181 ceph-mon[94189]: pgmap v17927: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 143 KiB/s wr, 6 op/s 2023-03-10T00:02:24.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:24 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32653.872499 secs 2023-03-10T00:02:24.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:24 smithi038 ceph-mon[92422]: pgmap v17927: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 143 KiB/s wr, 6 op/s 2023-03-10T00:02:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32656.448107 secs 2023-03-10T00:02:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:26 smithi181 ceph-mon[94189]: pgmap v17928: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 121 KiB/s wr, 5 op/s 2023-03-10T00:02:26.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32656.448107 secs 2023-03-10T00:02:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:26 smithi038 ceph-mon[92422]: pgmap v17928: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 121 KiB/s wr, 5 op/s 2023-03-10T00:02:26.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32656.448107 secs 2023-03-10T00:02:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:26 smithi122 ceph-mon[93869]: pgmap v17928: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 121 KiB/s wr, 5 op/s 2023-03-10T00:02:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:28 smithi038 ceph-mon[92422]: pgmap v17929: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 121 KiB/s wr, 8 op/s 2023-03-10T00:02:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32658.872625 secs 2023-03-10T00:02:28.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:28 smithi122 ceph-mon[93869]: pgmap v17929: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 121 KiB/s wr, 8 op/s 2023-03-10T00:02:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32658.872625 secs 2023-03-10T00:02:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:28 smithi181 ceph-mon[94189]: pgmap v17929: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 121 KiB/s wr, 8 op/s 2023-03-10T00:02:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32658.872625 secs 2023-03-10T00:02:28.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33120.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:02:30.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:30 smithi038 ceph-mon[92422]: pgmap v17930: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 121 KiB/s wr, 5 op/s 2023-03-10T00:02:30.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:30 smithi122 ceph-mon[93869]: pgmap v17930: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 121 KiB/s wr, 5 op/s 2023-03-10T00:02:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:30 smithi181 ceph-mon[94189]: pgmap v17930: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 121 KiB/s wr, 5 op/s 2023-03-10T00:02:31.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32661.448168 secs 2023-03-10T00:02:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:02:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:02:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:02:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:02:31.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32661.448168 secs 2023-03-10T00:02:31.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:02:31.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:02:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:02:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:02:31.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32661.448168 secs 2023-03-10T00:02:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:02:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:02:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:02:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:02:32.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:32 smithi122 ceph-mon[93869]: pgmap v17931: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 121 KiB/s wr, 8 op/s 2023-03-10T00:02:32.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:32 smithi181 ceph-mon[94189]: pgmap v17931: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 121 KiB/s wr, 8 op/s 2023-03-10T00:02:32.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:32 smithi038 ceph-mon[92422]: pgmap v17931: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 121 KiB/s wr, 8 op/s 2023-03-10T00:02:33.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33125.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:02:34.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:34 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32663.872732 secs 2023-03-10T00:02:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:34 smithi122 ceph-mon[93869]: pgmap v17932: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-10T00:02:34.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:34 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32663.872732 secs 2023-03-10T00:02:34.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:34 smithi181 ceph-mon[94189]: pgmap v17932: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-10T00:02:34.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:34 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32663.872732 secs 2023-03-10T00:02:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:34 smithi038 ceph-mon[92422]: pgmap v17932: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-10T00:02:36.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32666.448230 secs 2023-03-10T00:02:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:36 smithi038 ceph-mon[92422]: pgmap v17933: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-10T00:02:36.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32666.448230 secs 2023-03-10T00:02:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:36 smithi122 ceph-mon[93869]: pgmap v17933: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-10T00:02:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32666.448230 secs 2023-03-10T00:02:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:36 smithi181 ceph-mon[94189]: pgmap v17933: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-10T00:02:38.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:38 smithi038 ceph-mon[92422]: pgmap v17934: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 9 op/s 2023-03-10T00:02:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32668.872851 secs 2023-03-10T00:02:38.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:38 smithi122 ceph-mon[93869]: pgmap v17934: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 9 op/s 2023-03-10T00:02:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32668.872851 secs 2023-03-10T00:02:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:38 smithi181 ceph-mon[94189]: pgmap v17934: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 9 op/s 2023-03-10T00:02:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32668.872851 secs 2023-03-10T00:02:38.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33130.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:02:40.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:40 smithi038 ceph-mon[92422]: pgmap v17935: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-10T00:02:40.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:40 smithi122 ceph-mon[93869]: pgmap v17935: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-10T00:02:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:40 smithi181 ceph-mon[94189]: pgmap v17935: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-10T00:02:41.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32671.448291 secs 2023-03-10T00:02:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32671.448291 secs 2023-03-10T00:02:41.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32671.448291 secs 2023-03-10T00:02:42.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:42 smithi122 ceph-mon[93869]: pgmap v17936: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 9 op/s 2023-03-10T00:02:42.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:42 smithi181 ceph-mon[94189]: pgmap v17936: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 9 op/s 2023-03-10T00:02:42.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:42 smithi038 ceph-mon[92422]: pgmap v17936: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 9 op/s 2023-03-10T00:02:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33135.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:02:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:44 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32673.872976 secs 2023-03-10T00:02:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:44 smithi122 ceph-mon[93869]: pgmap v17937: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 682 B/s wr, 6 op/s 2023-03-10T00:02:44.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:44 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32673.872976 secs 2023-03-10T00:02:44.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:44 smithi181 ceph-mon[94189]: pgmap v17937: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 682 B/s wr, 6 op/s 2023-03-10T00:02:44.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:44 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32673.872976 secs 2023-03-10T00:02:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:44 smithi038 ceph-mon[92422]: pgmap v17937: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 682 B/s wr, 6 op/s 2023-03-10T00:02:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32676.448351 secs 2023-03-10T00:02:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:46 smithi038 ceph-mon[92422]: pgmap v17938: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 6 op/s 2023-03-10T00:02:46.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32676.448351 secs 2023-03-10T00:02:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:46 smithi122 ceph-mon[93869]: pgmap v17938: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 6 op/s 2023-03-10T00:02:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32676.448351 secs 2023-03-10T00:02:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:46 smithi181 ceph-mon[94189]: pgmap v17938: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 6 op/s 2023-03-10T00:02:48.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:48 smithi038 ceph-mon[92422]: pgmap v17939: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 341 KiB/s wr, 10 op/s 2023-03-10T00:02:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:02:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:02:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32678.873041 secs 2023-03-10T00:02:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:48 smithi122 ceph-mon[93869]: pgmap v17939: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 341 KiB/s wr, 10 op/s 2023-03-10T00:02:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:02:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:02:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32678.873041 secs 2023-03-10T00:02:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:48 smithi181 ceph-mon[94189]: pgmap v17939: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 341 KiB/s wr, 10 op/s 2023-03-10T00:02:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:02:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:02:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32678.873041 secs 2023-03-10T00:02:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33140.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:02:49.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:02:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:02:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:02:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:02:49.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:02:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:02:50.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:50 smithi122 ceph-mon[93869]: pgmap v17940: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 341 KiB/s wr, 7 op/s 2023-03-10T00:02:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:50 smithi181 ceph-mon[94189]: pgmap v17940: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 341 KiB/s wr, 7 op/s 2023-03-10T00:02:50.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:50 smithi038 ceph-mon[92422]: pgmap v17940: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 341 KiB/s wr, 7 op/s 2023-03-10T00:02:51.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32681.448418 secs 2023-03-10T00:02:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32681.448418 secs 2023-03-10T00:02:51.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32681.448418 secs 2023-03-10T00:02:52.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:52 smithi122 ceph-mon[93869]: pgmap v17941: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 861 KiB/s rd, 552 KiB/s wr, 11 op/s 2023-03-10T00:02:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:52 smithi181 ceph-mon[94189]: pgmap v17941: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 861 KiB/s rd, 552 KiB/s wr, 11 op/s 2023-03-10T00:02:52.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:52 smithi038 ceph-mon[92422]: pgmap v17941: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 861 KiB/s rd, 552 KiB/s wr, 11 op/s 2023-03-10T00:02:53.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33145.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:02:54.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:54 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32683.873165 secs 2023-03-10T00:02:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:54 smithi122 ceph-mon[93869]: pgmap v17942: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 859 KiB/s rd, 552 KiB/s wr, 9 op/s 2023-03-10T00:02:54.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:54 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32683.873165 secs 2023-03-10T00:02:54.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:54 smithi181 ceph-mon[94189]: pgmap v17942: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 859 KiB/s rd, 552 KiB/s wr, 9 op/s 2023-03-10T00:02:54.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:54 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32683.873165 secs 2023-03-10T00:02:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:54 smithi038 ceph-mon[92422]: pgmap v17942: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 859 KiB/s rd, 552 KiB/s wr, 9 op/s 2023-03-10T00:02:56.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32686.448476 secs 2023-03-10T00:02:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:02:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:02:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:56 smithi122 ceph-mon[93869]: pgmap v17943: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 871 KiB/s rd, 552 KiB/s wr, 9 op/s 2023-03-10T00:02:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32686.448476 secs 2023-03-10T00:02:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:02:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:02:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:56 smithi181 ceph-mon[94189]: pgmap v17943: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 871 KiB/s rd, 552 KiB/s wr, 9 op/s 2023-03-10T00:02:56.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32686.448476 secs 2023-03-10T00:02:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:02:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:02:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:56 smithi038 ceph-mon[92422]: pgmap v17943: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 871 KiB/s rd, 552 KiB/s wr, 9 op/s 2023-03-10T00:02:57.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:02:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:02:57.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:02:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:02:57.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:02:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:02:58.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:58 smithi122 ceph-mon[93869]: pgmap v17944: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 888 KiB/s wr, 14 op/s 2023-03-10T00:02:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:58 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:02:58.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:58 smithi181 ceph-mon[94189]: pgmap v17944: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 888 KiB/s wr, 14 op/s 2023-03-10T00:02:58.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:58 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:02:58.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:58 smithi038 ceph-mon[92422]: pgmap v17944: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 888 KiB/s wr, 14 op/s 2023-03-10T00:02:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:58 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:02:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33150.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:02:59.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:59 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:02:59.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:02:59.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:02:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:02:59.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:59 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:02:59.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:02:59.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:02:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:02:59.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:59 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:02:59.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:02:59.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:02:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:03:00.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:03:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:03:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:00 smithi122 ceph-mon[93869]: pgmap v17945: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 548 KiB/s wr, 11 op/s 2023-03-10T00:03:00.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:03:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:03:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:00 smithi181 ceph-mon[94189]: pgmap v17945: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 548 KiB/s wr, 11 op/s 2023-03-10T00:03:00.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:03:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:03:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:00 smithi038 ceph-mon[92422]: pgmap v17945: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 548 KiB/s wr, 11 op/s 2023-03-10T00:03:01.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32691.448536 secs 2023-03-10T00:03:01.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32691.448536 secs 2023-03-10T00:03:01.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32691.448536 secs 2023-03-10T00:03:02.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:02 smithi122 ceph-mon[93869]: pgmap v17946: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 742 KiB/s wr, 18 op/s 2023-03-10T00:03:02.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:02 smithi181 ceph-mon[94189]: pgmap v17946: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 742 KiB/s wr, 18 op/s 2023-03-10T00:03:02.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:02 smithi038 ceph-mon[92422]: pgmap v17946: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 742 KiB/s wr, 18 op/s 2023-03-10T00:03:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33155.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:03:04.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:04 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32693.873427 secs 2023-03-10T00:03:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:04 smithi122 ceph-mon[93869]: pgmap v17947: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 529 KiB/s rd, 553 KiB/s wr, 26 op/s 2023-03-10T00:03:04.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:04 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32693.873427 secs 2023-03-10T00:03:04.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:04 smithi181 ceph-mon[94189]: pgmap v17947: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 529 KiB/s rd, 553 KiB/s wr, 26 op/s 2023-03-10T00:03:04.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:04 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32693.873427 secs 2023-03-10T00:03:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:04 smithi038 ceph-mon[92422]: pgmap v17947: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 529 KiB/s rd, 553 KiB/s wr, 26 op/s 2023-03-10T00:03:06.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32696.448597 secs 2023-03-10T00:03:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:06 smithi122 ceph-mon[93869]: osdmap e1337: 12 total, 12 up, 12 in 2023-03-10T00:03:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:06 smithi122 ceph-mon[93869]: pgmap v17949: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 621 KiB/s rd, 663 KiB/s wr, 30 op/s 2023-03-10T00:03:06.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32696.448597 secs 2023-03-10T00:03:06.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:06 smithi181 ceph-mon[94189]: osdmap e1337: 12 total, 12 up, 12 in 2023-03-10T00:03:06.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:06 smithi181 ceph-mon[94189]: pgmap v17949: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 621 KiB/s rd, 663 KiB/s wr, 30 op/s 2023-03-10T00:03:06.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32696.448597 secs 2023-03-10T00:03:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:06 smithi038 ceph-mon[92422]: osdmap e1337: 12 total, 12 up, 12 in 2023-03-10T00:03:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:06 smithi038 ceph-mon[92422]: pgmap v17949: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 621 KiB/s rd, 663 KiB/s wr, 30 op/s 2023-03-10T00:03:07.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:07 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:03:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:03:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:03:07.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:07 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:03:07.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:03:07.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:03:07.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:07 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:03:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:03:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:03:08.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:03:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:03:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:08 smithi122 ceph-mon[93869]: pgmap v17950: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 432 KiB/s rd, 474 KiB/s wr, 29 op/s 2023-03-10T00:03:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:03:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:03:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:08 smithi181 ceph-mon[94189]: pgmap v17950: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 432 KiB/s rd, 474 KiB/s wr, 29 op/s 2023-03-10T00:03:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:03:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:03:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:08 smithi038 ceph-mon[92422]: pgmap v17950: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 432 KiB/s rd, 474 KiB/s wr, 29 op/s 2023-03-10T00:03:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33160.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:03:09.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:09 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32698.873536 secs 2023-03-10T00:03:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:03:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:03:09.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:09 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32698.873536 secs 2023-03-10T00:03:09.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:03:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:03:09.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:09 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32698.873536 secs 2023-03-10T00:03:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:03:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:03:10.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:10 smithi122 ceph-mon[93869]: pgmap v17951: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 432 KiB/s rd, 472 KiB/s wr, 29 op/s 2023-03-10T00:03:10.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:10 smithi181 ceph-mon[94189]: pgmap v17951: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 432 KiB/s rd, 472 KiB/s wr, 29 op/s 2023-03-10T00:03:10.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:10 smithi038 ceph-mon[92422]: pgmap v17951: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 432 KiB/s rd, 472 KiB/s wr, 29 op/s 2023-03-10T00:03:11.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32701.448676 secs 2023-03-10T00:03:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32701.448676 secs 2023-03-10T00:03:11.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32701.448676 secs 2023-03-10T00:03:12.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:12 smithi122 ceph-mon[93869]: pgmap v17952: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 240 KiB/s wr, 24 op/s 2023-03-10T00:03:12.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:12 smithi181 ceph-mon[94189]: pgmap v17952: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 240 KiB/s wr, 24 op/s 2023-03-10T00:03:12.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:12 smithi038 ceph-mon[92422]: pgmap v17952: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 240 KiB/s wr, 24 op/s 2023-03-10T00:03:13.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33165.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:03:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:14 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32703.873657 secs 2023-03-10T00:03:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:14 smithi122 ceph-mon[93869]: osdmap e1338: 12 total, 12 up, 12 in 2023-03-10T00:03:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:14 smithi122 ceph-mon[93869]: pgmap v17954: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 472 KiB/s wr, 12 op/s 2023-03-10T00:03:14.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:14 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32703.873657 secs 2023-03-10T00:03:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:14 smithi181 ceph-mon[94189]: osdmap e1338: 12 total, 12 up, 12 in 2023-03-10T00:03:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:14 smithi181 ceph-mon[94189]: pgmap v17954: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 472 KiB/s wr, 12 op/s 2023-03-10T00:03:14.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:14 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32703.873657 secs 2023-03-10T00:03:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:14 smithi038 ceph-mon[92422]: osdmap e1338: 12 total, 12 up, 12 in 2023-03-10T00:03:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:14 smithi038 ceph-mon[92422]: pgmap v17954: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 472 KiB/s wr, 12 op/s 2023-03-10T00:03:16.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32706.448734 secs 2023-03-10T00:03:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:16 smithi038 ceph-mon[92422]: pgmap v17955: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 385 KiB/s wr, 10 op/s 2023-03-10T00:03:16.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32706.448734 secs 2023-03-10T00:03:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:16 smithi122 ceph-mon[93869]: pgmap v17955: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 385 KiB/s wr, 10 op/s 2023-03-10T00:03:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32706.448734 secs 2023-03-10T00:03:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:16 smithi181 ceph-mon[94189]: pgmap v17955: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 385 KiB/s wr, 10 op/s 2023-03-10T00:03:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:18 smithi038 ceph-mon[92422]: pgmap v17956: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 172 KiB/s wr, 8 op/s 2023-03-10T00:03:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32708.873773 secs 2023-03-10T00:03:18.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:18 smithi122 ceph-mon[93869]: pgmap v17956: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 172 KiB/s wr, 8 op/s 2023-03-10T00:03:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32708.873773 secs 2023-03-10T00:03:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:18 smithi181 ceph-mon[94189]: pgmap v17956: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 172 KiB/s wr, 8 op/s 2023-03-10T00:03:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32708.873773 secs 2023-03-10T00:03:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33170.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:03:20.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:20 smithi122 ceph-mon[93869]: pgmap v17957: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 172 KiB/s wr, 7 op/s 2023-03-10T00:03:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:20 smithi181 ceph-mon[94189]: pgmap v17957: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 172 KiB/s wr, 7 op/s 2023-03-10T00:03:20.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:20 smithi038 ceph-mon[92422]: pgmap v17957: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 172 KiB/s wr, 7 op/s 2023-03-10T00:03:21.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32711.448794 secs 2023-03-10T00:03:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32711.448794 secs 2023-03-10T00:03:21.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32711.448794 secs 2023-03-10T00:03:22.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:22 smithi122 ceph-mon[93869]: pgmap v17958: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 171 KiB/s wr, 7 op/s 2023-03-10T00:03:22.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:22 smithi181 ceph-mon[94189]: pgmap v17958: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 171 KiB/s wr, 7 op/s 2023-03-10T00:03:22.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:22 smithi038 ceph-mon[92422]: pgmap v17958: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 171 KiB/s wr, 7 op/s 2023-03-10T00:03:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33175.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:03:24.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:24 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32713.873905 secs 2023-03-10T00:03:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:24 smithi122 ceph-mon[93869]: pgmap v17959: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2023-03-10T00:03:24.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:24 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32713.873905 secs 2023-03-10T00:03:24.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:24 smithi181 ceph-mon[94189]: pgmap v17959: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2023-03-10T00:03:24.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:24 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32713.873905 secs 2023-03-10T00:03:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:24 smithi038 ceph-mon[92422]: pgmap v17959: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2023-03-10T00:03:26.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32716.448877 secs 2023-03-10T00:03:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:26 smithi122 ceph-mon[93869]: pgmap v17960: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-10T00:03:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32716.448877 secs 2023-03-10T00:03:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:26 smithi181 ceph-mon[94189]: pgmap v17960: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-10T00:03:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32716.448877 secs 2023-03-10T00:03:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:26 smithi038 ceph-mon[92422]: pgmap v17960: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-10T00:03:28.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:28 smithi122 ceph-mon[93869]: pgmap v17961: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-10T00:03:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32718.874022 secs 2023-03-10T00:03:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:28 smithi181 ceph-mon[94189]: pgmap v17961: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-10T00:03:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32718.874022 secs 2023-03-10T00:03:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:28 smithi038 ceph-mon[92422]: pgmap v17961: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-10T00:03:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32718.874022 secs 2023-03-10T00:03:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33180.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:03:30.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:30 smithi122 ceph-mon[93869]: pgmap v17962: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-10T00:03:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:30 smithi181 ceph-mon[94189]: pgmap v17962: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-10T00:03:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:30 smithi038 ceph-mon[92422]: pgmap v17962: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-10T00:03:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32721.448937 secs 2023-03-10T00:03:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:03:31.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32721.448937 secs 2023-03-10T00:03:31.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:03:31.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32721.448937 secs 2023-03-10T00:03:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:03:32.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:32 smithi122 ceph-mon[93869]: pgmap v17963: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-10T00:03:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:03:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:03:32.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:32 smithi181 ceph-mon[94189]: pgmap v17963: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-10T00:03:32.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:03:32.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:03:32.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:32 smithi038 ceph-mon[92422]: pgmap v17963: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-10T00:03:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:03:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:03:33.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:03:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:03:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:03:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32723.874121 secs 2023-03-10T00:03:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:03:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:03:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:03:33.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32723.874121 secs 2023-03-10T00:03:33.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:03:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:03:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:03:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32723.874121 secs 2023-03-10T00:03:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33185.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:03:34.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:34 smithi122 ceph-mon[93869]: pgmap v17964: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-10T00:03:34.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:34 smithi181 ceph-mon[94189]: pgmap v17964: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-10T00:03:34.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:34 smithi038 ceph-mon[92422]: pgmap v17964: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-10T00:03:36.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32726.448985 secs 2023-03-10T00:03:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:36 smithi122 ceph-mon[93869]: pgmap v17965: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-10T00:03:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32726.448985 secs 2023-03-10T00:03:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:36 smithi181 ceph-mon[94189]: pgmap v17965: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-10T00:03:36.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32726.448985 secs 2023-03-10T00:03:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:36 smithi038 ceph-mon[92422]: pgmap v17965: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-10T00:03:38.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:38 smithi122 ceph-mon[93869]: pgmap v17966: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 122 KiB/s wr, 9 op/s 2023-03-10T00:03:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32728.874231 secs 2023-03-10T00:03:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:38 smithi181 ceph-mon[94189]: pgmap v17966: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 122 KiB/s wr, 9 op/s 2023-03-10T00:03:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32728.874231 secs 2023-03-10T00:03:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:38 smithi038 ceph-mon[92422]: pgmap v17966: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 122 KiB/s wr, 9 op/s 2023-03-10T00:03:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32728.874231 secs 2023-03-10T00:03:38.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33190.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:03:40.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:40 smithi122 ceph-mon[93869]: pgmap v17967: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 122 KiB/s wr, 7 op/s 2023-03-10T00:03:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:40 smithi181 ceph-mon[94189]: pgmap v17967: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 122 KiB/s wr, 7 op/s 2023-03-10T00:03:40.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:40 smithi038 ceph-mon[92422]: pgmap v17967: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 122 KiB/s wr, 7 op/s 2023-03-10T00:03:41.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32731.449045 secs 2023-03-10T00:03:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32731.449045 secs 2023-03-10T00:03:41.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32731.449045 secs 2023-03-10T00:03:42.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:42 smithi122 ceph-mon[93869]: pgmap v17968: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 122 KiB/s wr, 9 op/s 2023-03-10T00:03:42.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:42 smithi181 ceph-mon[94189]: pgmap v17968: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 122 KiB/s wr, 9 op/s 2023-03-10T00:03:42.764 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:42 smithi038 ceph-mon[92422]: pgmap v17968: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 122 KiB/s wr, 9 op/s 2023-03-10T00:03:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33195.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:03:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:44 smithi122 ceph-mon[93869]: pgmap v17969: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 122 KiB/s wr, 8 op/s 2023-03-10T00:03:44.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:44 smithi181 ceph-mon[94189]: pgmap v17969: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 122 KiB/s wr, 8 op/s 2023-03-10T00:03:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:44 smithi038 ceph-mon[92422]: pgmap v17969: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 122 KiB/s wr, 8 op/s 2023-03-10T00:03:46.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32736.449107 secs 2023-03-10T00:03:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:46 smithi122 ceph-mon[93869]: pgmap v17970: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 122 KiB/s wr, 7 op/s 2023-03-10T00:03:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:46 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:03:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32736.449107 secs 2023-03-10T00:03:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:46 smithi181 ceph-mon[94189]: pgmap v17970: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 122 KiB/s wr, 7 op/s 2023-03-10T00:03:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:46 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:03:46.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32736.449107 secs 2023-03-10T00:03:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:46 smithi038 ceph-mon[92422]: pgmap v17970: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 122 KiB/s wr, 7 op/s 2023-03-10T00:03:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:46 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:03:47.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:47 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:03:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:03:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:03:47.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:47 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:03:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:03:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:03:47.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:47 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:03:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:03:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:03:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:48 smithi122 ceph-mon[93869]: pgmap v17971: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 615 KiB/s rd, 461 KiB/s wr, 11 op/s 2023-03-10T00:03:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:03:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:03:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:48 smithi181 ceph-mon[94189]: pgmap v17971: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 615 KiB/s rd, 461 KiB/s wr, 11 op/s 2023-03-10T00:03:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:03:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:03:48.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:48 smithi038 ceph-mon[92422]: pgmap v17971: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 615 KiB/s rd, 461 KiB/s wr, 11 op/s 2023-03-10T00:03:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:03:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:03:48.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33200.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:03:49.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:03:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:03:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:03:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:03:49.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:03:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:03:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:50 smithi122 ceph-mon[93869]: pgmap v17972: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 340 KiB/s wr, 8 op/s 2023-03-10T00:03:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:50 smithi181 ceph-mon[94189]: pgmap v17972: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 340 KiB/s wr, 8 op/s 2023-03-10T00:03:50.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:50 smithi038 ceph-mon[92422]: pgmap v17972: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 340 KiB/s wr, 8 op/s 2023-03-10T00:03:51.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32741.449166 secs 2023-03-10T00:03:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32741.449166 secs 2023-03-10T00:03:51.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32741.449166 secs 2023-03-10T00:03:52.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:52 smithi122 ceph-mon[93869]: pgmap v17973: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 617 KiB/s rd, 554 KiB/s wr, 11 op/s 2023-03-10T00:03:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:52 smithi181 ceph-mon[94189]: pgmap v17973: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 617 KiB/s rd, 554 KiB/s wr, 11 op/s 2023-03-10T00:03:52.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:52 smithi038 ceph-mon[92422]: pgmap v17973: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 617 KiB/s rd, 554 KiB/s wr, 11 op/s 2023-03-10T00:03:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33205.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:03:54.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:54 smithi122 ceph-mon[93869]: pgmap v17974: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 848 KiB/s rd, 577 KiB/s wr, 10 op/s 2023-03-10T00:03:54.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:54 smithi181 ceph-mon[94189]: pgmap v17974: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 848 KiB/s rd, 577 KiB/s wr, 10 op/s 2023-03-10T00:03:54.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:54 smithi038 ceph-mon[92422]: pgmap v17974: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 848 KiB/s rd, 577 KiB/s wr, 10 op/s 2023-03-10T00:03:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32746.449226 secs 2023-03-10T00:03:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:03:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:03:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:56 smithi122 ceph-mon[93869]: pgmap v17975: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 859 KiB/s rd, 577 KiB/s wr, 9 op/s 2023-03-10T00:03:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32746.449226 secs 2023-03-10T00:03:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:03:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:03:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:56 smithi181 ceph-mon[94189]: pgmap v17975: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 859 KiB/s rd, 577 KiB/s wr, 9 op/s 2023-03-10T00:03:56.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32746.449226 secs 2023-03-10T00:03:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:03:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:03:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:56 smithi038 ceph-mon[92422]: pgmap v17975: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 859 KiB/s rd, 577 KiB/s wr, 9 op/s 2023-03-10T00:03:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:03:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:03:57.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:03:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:03:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:03:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:03:58.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:58 smithi122 ceph-mon[93869]: pgmap v17976: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 761 KiB/s wr, 14 op/s 2023-03-10T00:03:58.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:58 smithi181 ceph-mon[94189]: pgmap v17976: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 761 KiB/s wr, 14 op/s 2023-03-10T00:03:58.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:58 smithi038 ceph-mon[92422]: pgmap v17976: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 761 KiB/s wr, 14 op/s 2023-03-10T00:03:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33210.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:03:59.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:59 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 32748.874731 secs 2023-03-10T00:03:59.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:59 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:03:59.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:03:59.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:03:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:03:59.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:59 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 32748.874731 secs 2023-03-10T00:03:59.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:59 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:03:59.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:03:59.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:03:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:03:59.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:59 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 32748.874731 secs 2023-03-10T00:03:59.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:59 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:03:59.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:03:59.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:03:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:04:00.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:04:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:04:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:00 smithi122 ceph-mon[93869]: pgmap v17977: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 799 KiB/s rd, 552 KiB/s wr, 11 op/s 2023-03-10T00:04:00.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:04:00.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:04:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:00 smithi181 ceph-mon[94189]: pgmap v17977: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 799 KiB/s rd, 552 KiB/s wr, 11 op/s 2023-03-10T00:04:00.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:04:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:04:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:00 smithi038 ceph-mon[92422]: pgmap v17977: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 799 KiB/s rd, 552 KiB/s wr, 11 op/s 2023-03-10T00:04:01.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32751.449289 secs 2023-03-10T00:04:01.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32751.449289 secs 2023-03-10T00:04:01.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32751.449289 secs 2023-03-10T00:04:02.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:02 smithi122 ceph-mon[93869]: pgmap v17978: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 826 KiB/s rd, 589 KiB/s wr, 15 op/s 2023-03-10T00:04:02.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:02 smithi181 ceph-mon[94189]: pgmap v17978: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 826 KiB/s rd, 589 KiB/s wr, 15 op/s 2023-03-10T00:04:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:02 smithi038 ceph-mon[92422]: pgmap v17978: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 826 KiB/s rd, 589 KiB/s wr, 15 op/s 2023-03-10T00:04:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33215.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:04:04.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:04 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32753.874848 secs 2023-03-10T00:04:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:04 smithi122 ceph-mon[93869]: pgmap v17979: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 806 KiB/s rd, 586 KiB/s wr, 27 op/s 2023-03-10T00:04:04.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:04 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32753.874848 secs 2023-03-10T00:04:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:04 smithi181 ceph-mon[94189]: pgmap v17979: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 806 KiB/s rd, 586 KiB/s wr, 27 op/s 2023-03-10T00:04:04.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:04 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32753.874848 secs 2023-03-10T00:04:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:04 smithi038 ceph-mon[92422]: pgmap v17979: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 806 KiB/s rd, 586 KiB/s wr, 27 op/s 2023-03-10T00:04:06.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32756.449352 secs 2023-03-10T00:04:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:06 smithi122 ceph-mon[93869]: osdmap e1339: 12 total, 12 up, 12 in 2023-03-10T00:04:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:06 smithi122 ceph-mon[93869]: pgmap v17981: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 674 KiB/s rd, 675 KiB/s wr, 29 op/s 2023-03-10T00:04:06.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32756.449352 secs 2023-03-10T00:04:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:06 smithi181 ceph-mon[94189]: osdmap e1339: 12 total, 12 up, 12 in 2023-03-10T00:04:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:06 smithi181 ceph-mon[94189]: pgmap v17981: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 674 KiB/s rd, 675 KiB/s wr, 29 op/s 2023-03-10T00:04:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32756.449352 secs 2023-03-10T00:04:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:06 smithi038 ceph-mon[92422]: osdmap e1339: 12 total, 12 up, 12 in 2023-03-10T00:04:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:06 smithi038 ceph-mon[92422]: pgmap v17981: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 674 KiB/s rd, 675 KiB/s wr, 29 op/s 2023-03-10T00:04:07.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:04:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:04:07.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:04:07.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:04:07.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:04:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:04:08.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:04:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:04:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:08 smithi122 ceph-mon[93869]: pgmap v17982: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 667 KiB/s rd, 680 KiB/s wr, 31 op/s 2023-03-10T00:04:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:04:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:04:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:08 smithi181 ceph-mon[94189]: pgmap v17982: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 667 KiB/s rd, 680 KiB/s wr, 31 op/s 2023-03-10T00:04:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:04:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:04:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:08 smithi038 ceph-mon[92422]: pgmap v17982: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 667 KiB/s rd, 680 KiB/s wr, 31 op/s 2023-03-10T00:04:08.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33220.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:04:09.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:09 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32758.874981 secs 2023-03-10T00:04:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:04:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:04:09.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:09 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32758.874981 secs 2023-03-10T00:04:09.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:04:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:04:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:09 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32758.874981 secs 2023-03-10T00:04:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:04:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:04:10.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:10 smithi122 ceph-mon[93869]: pgmap v17983: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 484 KiB/s rd, 524 KiB/s wr, 30 op/s 2023-03-10T00:04:10.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:10 smithi181 ceph-mon[94189]: pgmap v17983: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 484 KiB/s rd, 524 KiB/s wr, 30 op/s 2023-03-10T00:04:10.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:10 smithi038 ceph-mon[92422]: pgmap v17983: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 484 KiB/s rd, 524 KiB/s wr, 30 op/s 2023-03-10T00:04:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32761.449413 secs 2023-03-10T00:04:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32761.449413 secs 2023-03-10T00:04:11.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32761.449413 secs 2023-03-10T00:04:12.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:12 smithi122 ceph-mon[93869]: pgmap v17984: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 452 KiB/s rd, 480 KiB/s wr, 28 op/s 2023-03-10T00:04:12.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:12 smithi181 ceph-mon[94189]: pgmap v17984: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 452 KiB/s rd, 480 KiB/s wr, 28 op/s 2023-03-10T00:04:12.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:12 smithi038 ceph-mon[92422]: pgmap v17984: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 452 KiB/s rd, 480 KiB/s wr, 28 op/s 2023-03-10T00:04:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33225.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:04:14.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:14 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32763.875094 secs 2023-03-10T00:04:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:14 smithi122 ceph-mon[93869]: osdmap e1340: 12 total, 12 up, 12 in 2023-03-10T00:04:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:14 smithi122 ceph-mon[93869]: pgmap v17986: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 488 KiB/s wr, 14 op/s 2023-03-10T00:04:14.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:14 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32763.875094 secs 2023-03-10T00:04:14.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:14 smithi181 ceph-mon[94189]: osdmap e1340: 12 total, 12 up, 12 in 2023-03-10T00:04:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:14 smithi181 ceph-mon[94189]: pgmap v17986: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 488 KiB/s wr, 14 op/s 2023-03-10T00:04:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:14 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32763.875094 secs 2023-03-10T00:04:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:14 smithi038 ceph-mon[92422]: osdmap e1340: 12 total, 12 up, 12 in 2023-03-10T00:04:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:14 smithi038 ceph-mon[92422]: pgmap v17986: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 488 KiB/s wr, 14 op/s 2023-03-10T00:04:16.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32766.449472 secs 2023-03-10T00:04:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:16 smithi122 ceph-mon[93869]: pgmap v17987: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 398 KiB/s wr, 12 op/s 2023-03-10T00:04:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32766.449472 secs 2023-03-10T00:04:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:16 smithi181 ceph-mon[94189]: pgmap v17987: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 398 KiB/s wr, 12 op/s 2023-03-10T00:04:16.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32766.449472 secs 2023-03-10T00:04:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:16 smithi038 ceph-mon[92422]: pgmap v17987: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 398 KiB/s wr, 12 op/s 2023-03-10T00:04:18.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:18 smithi122 ceph-mon[93869]: pgmap v17988: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 173 KiB/s wr, 8 op/s 2023-03-10T00:04:18.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32768.875206 secs 2023-03-10T00:04:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:18 smithi181 ceph-mon[94189]: pgmap v17988: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 173 KiB/s wr, 8 op/s 2023-03-10T00:04:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32768.875206 secs 2023-03-10T00:04:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:18 smithi038 ceph-mon[92422]: pgmap v17988: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 173 KiB/s wr, 8 op/s 2023-03-10T00:04:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32768.875206 secs 2023-03-10T00:04:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33230.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:04:20.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:20 smithi122 ceph-mon[93869]: pgmap v17989: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 172 KiB/s wr, 7 op/s 2023-03-10T00:04:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:20 smithi181 ceph-mon[94189]: pgmap v17989: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 172 KiB/s wr, 7 op/s 2023-03-10T00:04:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:20 smithi038 ceph-mon[92422]: pgmap v17989: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 172 KiB/s wr, 7 op/s 2023-03-10T00:04:21.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32771.449556 secs 2023-03-10T00:04:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32771.449556 secs 2023-03-10T00:04:21.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32771.449556 secs 2023-03-10T00:04:22.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:22 smithi122 ceph-mon[93869]: pgmap v17990: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 239 KiB/s wr, 7 op/s 2023-03-10T00:04:22.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:22 smithi181 ceph-mon[94189]: pgmap v17990: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 239 KiB/s wr, 7 op/s 2023-03-10T00:04:22.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:22 smithi038 ceph-mon[92422]: pgmap v17990: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 239 KiB/s wr, 7 op/s 2023-03-10T00:04:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33235.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:04:24.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:24 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32773.875328 secs 2023-03-10T00:04:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:24 smithi122 ceph-mon[93869]: pgmap v17991: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 118 KiB/s wr, 6 op/s 2023-03-10T00:04:24.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:24 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32773.875328 secs 2023-03-10T00:04:24.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:24 smithi181 ceph-mon[94189]: pgmap v17991: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 118 KiB/s wr, 6 op/s 2023-03-10T00:04:24.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:24 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32773.875328 secs 2023-03-10T00:04:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:24 smithi038 ceph-mon[92422]: pgmap v17991: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 118 KiB/s wr, 6 op/s 2023-03-10T00:04:26.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32776.449615 secs 2023-03-10T00:04:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:26 smithi122 ceph-mon[93869]: pgmap v17992: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 100 KiB/s wr, 5 op/s 2023-03-10T00:04:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32776.449615 secs 2023-03-10T00:04:26.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:26 smithi181 ceph-mon[94189]: pgmap v17992: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 100 KiB/s wr, 5 op/s 2023-03-10T00:04:26.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32776.449615 secs 2023-03-10T00:04:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:26 smithi038 ceph-mon[92422]: pgmap v17992: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 100 KiB/s wr, 5 op/s 2023-03-10T00:04:28.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:28 smithi122 ceph-mon[93869]: pgmap v17993: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 100 KiB/s wr, 8 op/s 2023-03-10T00:04:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32778.875441 secs 2023-03-10T00:04:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:28 smithi181 ceph-mon[94189]: pgmap v17993: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 100 KiB/s wr, 8 op/s 2023-03-10T00:04:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32778.875441 secs 2023-03-10T00:04:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:28 smithi038 ceph-mon[92422]: pgmap v17993: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 100 KiB/s wr, 8 op/s 2023-03-10T00:04:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32778.875441 secs 2023-03-10T00:04:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33240.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:04:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:30 smithi122 ceph-mon[93869]: pgmap v17994: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 100 KiB/s wr, 5 op/s 2023-03-10T00:04:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:30 smithi181 ceph-mon[94189]: pgmap v17994: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 100 KiB/s wr, 5 op/s 2023-03-10T00:04:30.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:30 smithi038 ceph-mon[92422]: pgmap v17994: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 103 KiB/s rd, 100 KiB/s wr, 5 op/s 2023-03-10T00:04:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32781.449674 secs 2023-03-10T00:04:31.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32781.449674 secs 2023-03-10T00:04:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32781.449674 secs 2023-03-10T00:04:32.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:32 smithi122 ceph-mon[93869]: pgmap v17995: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 100 KiB/s wr, 8 op/s 2023-03-10T00:04:32.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:32 smithi181 ceph-mon[94189]: pgmap v17995: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 100 KiB/s wr, 8 op/s 2023-03-10T00:04:32.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:32 smithi038 ceph-mon[92422]: pgmap v17995: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 100 KiB/s wr, 8 op/s 2023-03-10T00:04:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32783.875556 secs 2023-03-10T00:04:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:04:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:04:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:04:33.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:04:33.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:04:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32783.875556 secs 2023-03-10T00:04:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:04:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:04:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:04:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:04:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:04:33.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32783.875556 secs 2023-03-10T00:04:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:04:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:04:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:04:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:04:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:04:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33245.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:04:34.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:34 smithi122 ceph-mon[93869]: pgmap v17996: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 45 KiB/s wr, 6 op/s 2023-03-10T00:04:34.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:34 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:04:34.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:34 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:04:34.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:34 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:04:34.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:34 smithi181 ceph-mon[94189]: pgmap v17996: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 45 KiB/s wr, 6 op/s 2023-03-10T00:04:34.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:34 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:04:34.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:34 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:04:34.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:34 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:04:34.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:34 smithi038 ceph-mon[92422]: pgmap v17996: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 45 KiB/s wr, 6 op/s 2023-03-10T00:04:34.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:34 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:04:34.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:34 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:04:34.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:34 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:04:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32786.449734 secs 2023-03-10T00:04:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:36 smithi122 ceph-mon[93869]: pgmap v17997: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-10T00:04:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32786.449734 secs 2023-03-10T00:04:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:36 smithi181 ceph-mon[94189]: pgmap v17997: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-10T00:04:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32786.449734 secs 2023-03-10T00:04:36.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:36 smithi038 ceph-mon[92422]: pgmap v17997: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-10T00:04:38.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:38 smithi122 ceph-mon[93869]: pgmap v17998: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 9 op/s 2023-03-10T00:04:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32788.875672 secs 2023-03-10T00:04:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:38 smithi181 ceph-mon[94189]: pgmap v17998: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 9 op/s 2023-03-10T00:04:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32788.875672 secs 2023-03-10T00:04:38.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:38 smithi038 ceph-mon[92422]: pgmap v17998: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 9 op/s 2023-03-10T00:04:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32788.875672 secs 2023-03-10T00:04:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33250.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:04:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:40 smithi122 ceph-mon[93869]: pgmap v17999: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-10T00:04:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:40 smithi181 ceph-mon[94189]: pgmap v17999: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-10T00:04:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:40 smithi038 ceph-mon[92422]: pgmap v17999: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-10T00:04:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32791.449796 secs 2023-03-10T00:04:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32791.449796 secs 2023-03-10T00:04:41.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32791.449796 secs 2023-03-10T00:04:42.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:42 smithi122 ceph-mon[93869]: pgmap v18000: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 9 op/s 2023-03-10T00:04:42.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:42 smithi181 ceph-mon[94189]: pgmap v18000: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 9 op/s 2023-03-10T00:04:42.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:42 smithi038 ceph-mon[92422]: pgmap v18000: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 9 op/s 2023-03-10T00:04:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33255.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:04:44.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:44 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32793.875789 secs 2023-03-10T00:04:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:44 smithi122 ceph-mon[93869]: pgmap v18001: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 7 op/s 2023-03-10T00:04:44.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:44 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32793.875789 secs 2023-03-10T00:04:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:44 smithi181 ceph-mon[94189]: pgmap v18001: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 7 op/s 2023-03-10T00:04:44.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:44 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32793.875789 secs 2023-03-10T00:04:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:44 smithi038 ceph-mon[92422]: pgmap v18001: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 7 op/s 2023-03-10T00:04:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32796.449857 secs 2023-03-10T00:04:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:46 smithi122 ceph-mon[93869]: pgmap v18002: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 6 op/s 2023-03-10T00:04:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32796.449857 secs 2023-03-10T00:04:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:46 smithi181 ceph-mon[94189]: pgmap v18002: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 6 op/s 2023-03-10T00:04:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32796.449857 secs 2023-03-10T00:04:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:46 smithi038 ceph-mon[92422]: pgmap v18002: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 6 op/s 2023-03-10T00:04:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:48 smithi122 ceph-mon[93869]: pgmap v18003: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 313 KiB/s wr, 10 op/s 2023-03-10T00:04:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:04:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:04:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32798.875886 secs 2023-03-10T00:04:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:48 smithi181 ceph-mon[94189]: pgmap v18003: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 313 KiB/s wr, 10 op/s 2023-03-10T00:04:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:04:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:04:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32798.875886 secs 2023-03-10T00:04:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:48 smithi038 ceph-mon[92422]: pgmap v18003: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 313 KiB/s wr, 10 op/s 2023-03-10T00:04:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:04:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:04:48.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32798.875886 secs 2023-03-10T00:04:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33260.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:04:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:04:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:04:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:04:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:04:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:04:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:04:50.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:50 smithi122 ceph-mon[93869]: pgmap v18004: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 313 KiB/s wr, 7 op/s 2023-03-10T00:04:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:50 smithi181 ceph-mon[94189]: pgmap v18004: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 313 KiB/s wr, 7 op/s 2023-03-10T00:04:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:50 smithi038 ceph-mon[92422]: pgmap v18004: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 313 KiB/s wr, 7 op/s 2023-03-10T00:04:51.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32801.449915 secs 2023-03-10T00:04:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32801.449915 secs 2023-03-10T00:04:51.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32801.449915 secs 2023-03-10T00:04:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:52 smithi122 ceph-mon[93869]: pgmap v18005: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 697 KiB/s rd, 537 KiB/s wr, 11 op/s 2023-03-10T00:04:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:52 smithi181 ceph-mon[94189]: pgmap v18005: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 697 KiB/s rd, 537 KiB/s wr, 11 op/s 2023-03-10T00:04:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:52 smithi038 ceph-mon[92422]: pgmap v18005: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 697 KiB/s rd, 537 KiB/s wr, 11 op/s 2023-03-10T00:04:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33265.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:04:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:54 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32803.876029 secs 2023-03-10T00:04:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:54 smithi122 ceph-mon[93869]: pgmap v18006: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 714 KiB/s rd, 555 KiB/s wr, 10 op/s 2023-03-10T00:04:54.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:54 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32803.876029 secs 2023-03-10T00:04:54.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:54 smithi181 ceph-mon[94189]: pgmap v18006: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 714 KiB/s rd, 555 KiB/s wr, 10 op/s 2023-03-10T00:04:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:54 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32803.876029 secs 2023-03-10T00:04:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:54 smithi038 ceph-mon[92422]: pgmap v18006: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 714 KiB/s rd, 555 KiB/s wr, 10 op/s 2023-03-10T00:04:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32806.449979 secs 2023-03-10T00:04:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:04:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:04:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:56 smithi122 ceph-mon[93869]: pgmap v18007: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 851 KiB/s rd, 681 KiB/s wr, 10 op/s 2023-03-10T00:04:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32806.449979 secs 2023-03-10T00:04:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:04:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:04:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:56 smithi181 ceph-mon[94189]: pgmap v18007: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 851 KiB/s rd, 681 KiB/s wr, 10 op/s 2023-03-10T00:04:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32806.449979 secs 2023-03-10T00:04:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:04:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:04:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:56 smithi038 ceph-mon[92422]: pgmap v18007: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 851 KiB/s rd, 681 KiB/s wr, 10 op/s 2023-03-10T00:04:57.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:04:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:04:57.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:04:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:04:57.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:04:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:04:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:58 smithi122 ceph-mon[93869]: pgmap v18008: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 864 KiB/s wr, 14 op/s 2023-03-10T00:04:58.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:58 smithi181 ceph-mon[94189]: pgmap v18008: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 864 KiB/s wr, 14 op/s 2023-03-10T00:04:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:58 smithi038 ceph-mon[92422]: pgmap v18008: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 864 KiB/s wr, 14 op/s 2023-03-10T00:04:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33270.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:04:59.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:04:59 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32808.876187 secs 2023-03-10T00:04:59.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:04:59 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32808.876187 secs 2023-03-10T00:04:59.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:04:59 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32808.876187 secs 2023-03-10T00:05:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:00 smithi122 ceph-mon[93869]: pgmap v18009: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 719 KiB/s rd, 582 KiB/s wr, 12 op/s 2023-03-10T00:05:00.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:00 smithi181 ceph-mon[94189]: pgmap v18009: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 719 KiB/s rd, 582 KiB/s wr, 12 op/s 2023-03-10T00:05:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:00 smithi038 ceph-mon[92422]: pgmap v18009: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 719 KiB/s rd, 582 KiB/s wr, 12 op/s 2023-03-10T00:05:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32811.450038 secs 2023-03-10T00:05:01.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32811.450038 secs 2023-03-10T00:05:01.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32811.450038 secs 2023-03-10T00:05:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:02 smithi122 ceph-mon[93869]: pgmap v18010: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 867 KiB/s rd, 728 KiB/s wr, 15 op/s 2023-03-10T00:05:02.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:02 smithi181 ceph-mon[94189]: pgmap v18010: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 867 KiB/s rd, 728 KiB/s wr, 15 op/s 2023-03-10T00:05:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:02 smithi038 ceph-mon[92422]: pgmap v18010: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 867 KiB/s rd, 728 KiB/s wr, 15 op/s 2023-03-10T00:05:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33275.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:05:04.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:04 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32813.876254 secs 2023-03-10T00:05:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:04 smithi122 ceph-mon[93869]: pgmap v18011: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 549 KiB/s rd, 575 KiB/s wr, 24 op/s 2023-03-10T00:05:04.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:04 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32813.876254 secs 2023-03-10T00:05:04.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:04 smithi181 ceph-mon[94189]: pgmap v18011: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 549 KiB/s rd, 575 KiB/s wr, 24 op/s 2023-03-10T00:05:04.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:04 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32813.876254 secs 2023-03-10T00:05:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:04 smithi038 ceph-mon[92422]: pgmap v18011: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 549 KiB/s rd, 575 KiB/s wr, 24 op/s 2023-03-10T00:05:06.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32816.450098 secs 2023-03-10T00:05:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:06 smithi122 ceph-mon[93869]: osdmap e1341: 12 total, 12 up, 12 in 2023-03-10T00:05:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:06 smithi122 ceph-mon[93869]: pgmap v18013: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 641 KiB/s rd, 685 KiB/s wr, 27 op/s 2023-03-10T00:05:06.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32816.450098 secs 2023-03-10T00:05:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:06 smithi181 ceph-mon[94189]: osdmap e1341: 12 total, 12 up, 12 in 2023-03-10T00:05:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:06 smithi181 ceph-mon[94189]: pgmap v18013: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 641 KiB/s rd, 685 KiB/s wr, 27 op/s 2023-03-10T00:05:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32816.450098 secs 2023-03-10T00:05:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:06 smithi038 ceph-mon[92422]: osdmap e1341: 12 total, 12 up, 12 in 2023-03-10T00:05:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:06 smithi038 ceph-mon[92422]: pgmap v18013: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 641 KiB/s rd, 685 KiB/s wr, 27 op/s 2023-03-10T00:05:08.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:05:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:05:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:08 smithi122 ceph-mon[93869]: pgmap v18014: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 442 KiB/s rd, 510 KiB/s wr, 28 op/s 2023-03-10T00:05:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:05:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:05:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:08 smithi181 ceph-mon[94189]: pgmap v18014: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 442 KiB/s rd, 510 KiB/s wr, 28 op/s 2023-03-10T00:05:08.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:05:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:05:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:08 smithi038 ceph-mon[92422]: pgmap v18014: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 442 KiB/s rd, 510 KiB/s wr, 28 op/s 2023-03-10T00:05:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33280.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:05:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:09 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32818.876324 secs 2023-03-10T00:05:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:05:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:05:09.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:09 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32818.876324 secs 2023-03-10T00:05:09.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:05:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:05:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:09 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32818.876324 secs 2023-03-10T00:05:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:05:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:05:10.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:10 smithi122 ceph-mon[93869]: pgmap v18015: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 475 KiB/s wr, 29 op/s 2023-03-10T00:05:10.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:10 smithi181 ceph-mon[94189]: pgmap v18015: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 475 KiB/s wr, 29 op/s 2023-03-10T00:05:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:10 smithi038 ceph-mon[92422]: pgmap v18015: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 475 KiB/s wr, 29 op/s 2023-03-10T00:05:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32821.450161 secs 2023-03-10T00:05:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32821.450161 secs 2023-03-10T00:05:11.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32821.450161 secs 2023-03-10T00:05:12.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:12 smithi122 ceph-mon[93869]: pgmap v18016: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 299 KiB/s wr, 27 op/s 2023-03-10T00:05:12.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:12 smithi181 ceph-mon[94189]: pgmap v18016: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 299 KiB/s wr, 27 op/s 2023-03-10T00:05:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:12 smithi038 ceph-mon[92422]: pgmap v18016: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 299 KiB/s wr, 27 op/s 2023-03-10T00:05:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33285.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:05:14.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:14 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32823.876445 secs 2023-03-10T00:05:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:14 smithi122 ceph-mon[93869]: osdmap e1342: 12 total, 12 up, 12 in 2023-03-10T00:05:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:14 smithi122 ceph-mon[93869]: pgmap v18018: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 55 KiB/s wr, 14 op/s 2023-03-10T00:05:14.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:14 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32823.876445 secs 2023-03-10T00:05:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:14 smithi181 ceph-mon[94189]: osdmap e1342: 12 total, 12 up, 12 in 2023-03-10T00:05:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:14 smithi181 ceph-mon[94189]: pgmap v18018: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 55 KiB/s wr, 14 op/s 2023-03-10T00:05:14.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:14 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32823.876445 secs 2023-03-10T00:05:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:14 smithi038 ceph-mon[92422]: osdmap e1342: 12 total, 12 up, 12 in 2023-03-10T00:05:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:14 smithi038 ceph-mon[92422]: pgmap v18018: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 55 KiB/s wr, 14 op/s 2023-03-10T00:05:16.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32826.450219 secs 2023-03-10T00:05:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:16 smithi122 ceph-mon[93869]: pgmap v18019: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 217 KiB/s wr, 12 op/s 2023-03-10T00:05:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32826.450219 secs 2023-03-10T00:05:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:16 smithi181 ceph-mon[94189]: pgmap v18019: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 217 KiB/s wr, 12 op/s 2023-03-10T00:05:16.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32826.450219 secs 2023-03-10T00:05:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:16 smithi038 ceph-mon[92422]: pgmap v18019: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 217 KiB/s wr, 12 op/s 2023-03-10T00:05:18.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:18 smithi122 ceph-mon[93869]: pgmap v18020: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 173 KiB/s wr, 8 op/s 2023-03-10T00:05:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32828.876585 secs 2023-03-10T00:05:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:18 smithi181 ceph-mon[94189]: pgmap v18020: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 173 KiB/s wr, 8 op/s 2023-03-10T00:05:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32828.876585 secs 2023-03-10T00:05:18.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:18 smithi038 ceph-mon[92422]: pgmap v18020: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 173 KiB/s wr, 8 op/s 2023-03-10T00:05:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32828.876585 secs 2023-03-10T00:05:18.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33290.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:05:20.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:20 smithi122 ceph-mon[93869]: pgmap v18021: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 172 KiB/s wr, 7 op/s 2023-03-10T00:05:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:20 smithi181 ceph-mon[94189]: pgmap v18021: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 172 KiB/s wr, 7 op/s 2023-03-10T00:05:20.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:20 smithi038 ceph-mon[92422]: pgmap v18021: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 172 KiB/s wr, 7 op/s 2023-03-10T00:05:21.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32831.450278 secs 2023-03-10T00:05:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32831.450278 secs 2023-03-10T00:05:21.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32831.450278 secs 2023-03-10T00:05:22.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:22 smithi122 ceph-mon[93869]: pgmap v18022: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 172 KiB/s wr, 7 op/s 2023-03-10T00:05:22.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:22 smithi181 ceph-mon[94189]: pgmap v18022: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 172 KiB/s wr, 7 op/s 2023-03-10T00:05:22.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:22 smithi038 ceph-mon[92422]: pgmap v18022: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 172 KiB/s wr, 7 op/s 2023-03-10T00:05:23.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33295.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:05:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:24 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32833.876731 secs 2023-03-10T00:05:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:24 smithi122 ceph-mon[93869]: pgmap v18023: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 169 KiB/s wr, 7 op/s 2023-03-10T00:05:24.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:24 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32833.876731 secs 2023-03-10T00:05:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:24 smithi181 ceph-mon[94189]: pgmap v18023: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 169 KiB/s wr, 7 op/s 2023-03-10T00:05:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:24 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32833.876731 secs 2023-03-10T00:05:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:24 smithi038 ceph-mon[92422]: pgmap v18023: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 169 KiB/s wr, 7 op/s 2023-03-10T00:05:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32836.450339 secs 2023-03-10T00:05:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:26 smithi122 ceph-mon[93869]: pgmap v18024: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 143 KiB/s wr, 6 op/s 2023-03-10T00:05:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32836.450339 secs 2023-03-10T00:05:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:26 smithi181 ceph-mon[94189]: pgmap v18024: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 143 KiB/s wr, 6 op/s 2023-03-10T00:05:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32836.450339 secs 2023-03-10T00:05:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:26 smithi038 ceph-mon[92422]: pgmap v18024: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 143 KiB/s wr, 6 op/s 2023-03-10T00:05:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:28 smithi122 ceph-mon[93869]: pgmap v18025: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-10T00:05:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32838.876849 secs 2023-03-10T00:05:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:28 smithi181 ceph-mon[94189]: pgmap v18025: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-10T00:05:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32838.876849 secs 2023-03-10T00:05:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:28 smithi038 ceph-mon[92422]: pgmap v18025: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-10T00:05:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32838.876849 secs 2023-03-10T00:05:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33300.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:05:30.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:30 smithi122 ceph-mon[93869]: pgmap v18026: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-10T00:05:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:30 smithi181 ceph-mon[94189]: pgmap v18026: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-10T00:05:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:30 smithi038 ceph-mon[92422]: pgmap v18026: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-10T00:05:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32841.450398 secs 2023-03-10T00:05:31.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32841.450398 secs 2023-03-10T00:05:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32841.450398 secs 2023-03-10T00:05:32.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:32 smithi122 ceph-mon[93869]: pgmap v18027: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-10T00:05:32.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:32 smithi181 ceph-mon[94189]: pgmap v18027: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-10T00:05:32.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:32 smithi038 ceph-mon[92422]: pgmap v18027: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-10T00:05:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33305.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:05:34.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:34 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32843.876993 secs 2023-03-10T00:05:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:34 smithi122 ceph-mon[93869]: pgmap v18028: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-10T00:05:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:34 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:05:34.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:34 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32843.876993 secs 2023-03-10T00:05:34.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:34 smithi181 ceph-mon[94189]: pgmap v18028: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-10T00:05:34.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:34 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:05:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:34 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32843.876993 secs 2023-03-10T00:05:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:34 smithi038 ceph-mon[92422]: pgmap v18028: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-10T00:05:34.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:34 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:05:35.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:05:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:05:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:05:35.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:05:35.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:05:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:05:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:05:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:05:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:05:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32846.450458 secs 2023-03-10T00:05:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:36 smithi122 ceph-mon[93869]: pgmap v18029: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 58 KiB/s wr, 6 op/s 2023-03-10T00:05:36.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32846.450458 secs 2023-03-10T00:05:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:36 smithi181 ceph-mon[94189]: pgmap v18029: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 58 KiB/s wr, 6 op/s 2023-03-10T00:05:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32846.450458 secs 2023-03-10T00:05:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:36 smithi038 ceph-mon[92422]: pgmap v18029: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 58 KiB/s wr, 6 op/s 2023-03-10T00:05:38.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:38 smithi122 ceph-mon[93869]: pgmap v18030: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 123 KiB/s wr, 9 op/s 2023-03-10T00:05:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32848.877081 secs 2023-03-10T00:05:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:38 smithi181 ceph-mon[94189]: pgmap v18030: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 123 KiB/s wr, 9 op/s 2023-03-10T00:05:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32848.877081 secs 2023-03-10T00:05:38.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:38 smithi038 ceph-mon[92422]: pgmap v18030: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 123 KiB/s wr, 9 op/s 2023-03-10T00:05:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32848.877081 secs 2023-03-10T00:05:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33310.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:05:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:40 smithi122 ceph-mon[93869]: pgmap v18031: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 355 KiB/s rd, 123 KiB/s wr, 8 op/s 2023-03-10T00:05:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:05:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:05:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:40 smithi181 ceph-mon[94189]: pgmap v18031: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 355 KiB/s rd, 123 KiB/s wr, 8 op/s 2023-03-10T00:05:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:05:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:05:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:40 smithi038 ceph-mon[92422]: pgmap v18031: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 355 KiB/s rd, 123 KiB/s wr, 8 op/s 2023-03-10T00:05:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:05:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:05:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32851.450517 secs 2023-03-10T00:05:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:05:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:05:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32851.450517 secs 2023-03-10T00:05:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:05:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:05:41.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32851.450517 secs 2023-03-10T00:05:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:05:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:05:42.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:42 smithi122 ceph-mon[93869]: pgmap v18032: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 484 KiB/s rd, 250 KiB/s wr, 9 op/s 2023-03-10T00:05:42.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:42 smithi181 ceph-mon[94189]: pgmap v18032: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 484 KiB/s rd, 250 KiB/s wr, 9 op/s 2023-03-10T00:05:42.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:42 smithi038 ceph-mon[92422]: pgmap v18032: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 484 KiB/s rd, 250 KiB/s wr, 9 op/s 2023-03-10T00:05:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33315.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:05:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:44 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32853.877206 secs 2023-03-10T00:05:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:44 smithi122 ceph-mon[93869]: pgmap v18033: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 250 KiB/s wr, 8 op/s 2023-03-10T00:05:44.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:44 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32853.877206 secs 2023-03-10T00:05:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:44 smithi181 ceph-mon[94189]: pgmap v18033: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 250 KiB/s wr, 8 op/s 2023-03-10T00:05:44.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:44 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32853.877206 secs 2023-03-10T00:05:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:44 smithi038 ceph-mon[92422]: pgmap v18033: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 250 KiB/s wr, 8 op/s 2023-03-10T00:05:46.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32856.450589 secs 2023-03-10T00:05:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:46 smithi122 ceph-mon[93869]: pgmap v18034: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 250 KiB/s wr, 7 op/s 2023-03-10T00:05:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32856.450589 secs 2023-03-10T00:05:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:46 smithi181 ceph-mon[94189]: pgmap v18034: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 250 KiB/s wr, 7 op/s 2023-03-10T00:05:46.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32856.450589 secs 2023-03-10T00:05:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:46 smithi038 ceph-mon[92422]: pgmap v18034: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 250 KiB/s wr, 7 op/s 2023-03-10T00:05:48.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:48 smithi122 ceph-mon[93869]: pgmap v18035: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 342 KiB/s wr, 9 op/s 2023-03-10T00:05:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32858.877300 secs 2023-03-10T00:05:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:48 smithi181 ceph-mon[94189]: pgmap v18035: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 342 KiB/s wr, 9 op/s 2023-03-10T00:05:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32858.877300 secs 2023-03-10T00:05:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:48 smithi038 ceph-mon[92422]: pgmap v18035: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 342 KiB/s wr, 9 op/s 2023-03-10T00:05:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32858.877300 secs 2023-03-10T00:05:48.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33320.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:05:50.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:50 smithi122 ceph-mon[93869]: pgmap v18036: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 362 KiB/s rd, 277 KiB/s wr, 8 op/s 2023-03-10T00:05:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:50 smithi181 ceph-mon[94189]: pgmap v18036: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 362 KiB/s rd, 277 KiB/s wr, 8 op/s 2023-03-10T00:05:50.788 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:50 smithi038 ceph-mon[92422]: pgmap v18036: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 362 KiB/s rd, 277 KiB/s wr, 8 op/s 2023-03-10T00:05:51.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32861.450668 secs 2023-03-10T00:05:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32861.450668 secs 2023-03-10T00:05:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32861.450668 secs 2023-03-10T00:05:52.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:52 smithi122 ceph-mon[93869]: pgmap v18037: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 746 KiB/s rd, 619 KiB/s wr, 10 op/s 2023-03-10T00:05:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:52 smithi181 ceph-mon[94189]: pgmap v18037: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 746 KiB/s rd, 619 KiB/s wr, 10 op/s 2023-03-10T00:05:52.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:52 smithi038 ceph-mon[92422]: pgmap v18037: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 746 KiB/s rd, 619 KiB/s wr, 10 op/s 2023-03-10T00:05:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:05:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:05:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32863.877409 secs 2023-03-10T00:05:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:05:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:05:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32863.877409 secs 2023-03-10T00:05:53.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:05:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:05:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32863.877409 secs 2023-03-10T00:05:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33325.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:05:54.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:54 smithi122 ceph-mon[93869]: pgmap v18038: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 644 KiB/s rd, 518 KiB/s wr, 8 op/s 2023-03-10T00:05:54.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:54 smithi181 ceph-mon[94189]: pgmap v18038: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 644 KiB/s rd, 518 KiB/s wr, 8 op/s 2023-03-10T00:05:54.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:54 smithi038 ceph-mon[92422]: pgmap v18038: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 644 KiB/s rd, 518 KiB/s wr, 8 op/s 2023-03-10T00:05:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32866.450732 secs 2023-03-10T00:05:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:56 smithi122 ceph-mon[93869]: pgmap v18039: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 522 KiB/s wr, 8 op/s 2023-03-10T00:05:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:05:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:05:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32866.450732 secs 2023-03-10T00:05:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:56 smithi181 ceph-mon[94189]: pgmap v18039: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 522 KiB/s wr, 8 op/s 2023-03-10T00:05:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:05:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:05:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32866.450732 secs 2023-03-10T00:05:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:56 smithi038 ceph-mon[92422]: pgmap v18039: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 522 KiB/s wr, 8 op/s 2023-03-10T00:05:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:05:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:05:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:05:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:05:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:05:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:05:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:05:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:05:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:58 smithi122 ceph-mon[93869]: pgmap v18040: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 523 KiB/s wr, 11 op/s 2023-03-10T00:05:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:58 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:05:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:58 smithi181 ceph-mon[94189]: pgmap v18040: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 523 KiB/s wr, 11 op/s 2023-03-10T00:05:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:58 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:05:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:58 smithi038 ceph-mon[92422]: pgmap v18040: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 523 KiB/s wr, 11 op/s 2023-03-10T00:05:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:58 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:05:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33330.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:05:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:59 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:05:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:05:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:05:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:05:59 smithi122 ceph-mon[93869]: pgmap v18041: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 825 KiB/s rd, 712 KiB/s wr, 12 op/s 2023-03-10T00:05:59.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:59 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:05:59.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:05:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:05:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:05:59 smithi181 ceph-mon[94189]: pgmap v18041: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 825 KiB/s rd, 712 KiB/s wr, 12 op/s 2023-03-10T00:05:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:59 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:05:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:05:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:05:59.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:05:59 smithi038 ceph-mon[92422]: pgmap v18041: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 825 KiB/s rd, 712 KiB/s wr, 12 op/s 2023-03-10T00:06:00.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32871.450792 secs 2023-03-10T00:06:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32871.450792 secs 2023-03-10T00:06:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32871.450792 secs 2023-03-10T00:06:01.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:01 smithi122 ceph-mon[93869]: pgmap v18042: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 943 KiB/s rd, 830 KiB/s wr, 14 op/s 2023-03-10T00:06:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:01 smithi181 ceph-mon[94189]: pgmap v18042: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 943 KiB/s rd, 830 KiB/s wr, 14 op/s 2023-03-10T00:06:01.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:01 smithi038 ceph-mon[92422]: pgmap v18042: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 943 KiB/s rd, 830 KiB/s wr, 14 op/s 2023-03-10T00:06:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33335.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:06:04.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:04 smithi122 ceph-mon[93869]: pgmap v18043: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 530 KiB/s rd, 539 KiB/s wr, 14 op/s 2023-03-10T00:06:04.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:04 smithi181 ceph-mon[94189]: pgmap v18043: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 530 KiB/s rd, 539 KiB/s wr, 14 op/s 2023-03-10T00:06:04.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:04 smithi038 ceph-mon[92422]: pgmap v18043: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 530 KiB/s rd, 539 KiB/s wr, 14 op/s 2023-03-10T00:06:06.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32876.450852 secs 2023-03-10T00:06:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:06 smithi122 ceph-mon[93869]: pgmap v18044: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 532 KiB/s rd, 541 KiB/s wr, 14 op/s 2023-03-10T00:06:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32876.450852 secs 2023-03-10T00:06:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:06 smithi181 ceph-mon[94189]: pgmap v18044: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 532 KiB/s rd, 541 KiB/s wr, 14 op/s 2023-03-10T00:06:06.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32876.450852 secs 2023-03-10T00:06:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:06 smithi038 ceph-mon[92422]: pgmap v18044: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 532 KiB/s rd, 541 KiB/s wr, 14 op/s 2023-03-10T00:06:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:08 smithi122 ceph-mon[93869]: pgmap v18045: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 699 KiB/s rd, 729 KiB/s wr, 19 op/s 2023-03-10T00:06:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:06:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:06:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32878.877760 secs 2023-03-10T00:06:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:08 smithi181 ceph-mon[94189]: pgmap v18045: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 699 KiB/s rd, 729 KiB/s wr, 19 op/s 2023-03-10T00:06:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:06:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:06:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32878.877760 secs 2023-03-10T00:06:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:08 smithi038 ceph-mon[92422]: pgmap v18045: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 699 KiB/s rd, 729 KiB/s wr, 19 op/s 2023-03-10T00:06:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:06:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:06:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32878.877760 secs 2023-03-10T00:06:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33340.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:06:09.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:06:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:06:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:06:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:06:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:06:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:06:10.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:10 smithi122 ceph-mon[93869]: pgmap v18046: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 851 KiB/s rd, 876 KiB/s wr, 28 op/s 2023-03-10T00:06:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:10 smithi122 ceph-mon[93869]: osdmap e1343: 12 total, 12 up, 12 in 2023-03-10T00:06:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:10 smithi181 ceph-mon[94189]: pgmap v18046: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 851 KiB/s rd, 876 KiB/s wr, 28 op/s 2023-03-10T00:06:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:10 smithi181 ceph-mon[94189]: osdmap e1343: 12 total, 12 up, 12 in 2023-03-10T00:06:10.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:10 smithi038 ceph-mon[92422]: pgmap v18046: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 851 KiB/s rd, 876 KiB/s wr, 28 op/s 2023-03-10T00:06:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:10 smithi038 ceph-mon[92422]: osdmap e1343: 12 total, 12 up, 12 in 2023-03-10T00:06:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32881.450913 secs 2023-03-10T00:06:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:11 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:06:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:11 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:06:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:11 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:06:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32881.450913 secs 2023-03-10T00:06:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:11 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:06:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:11 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:06:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:11 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:06:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32881.450913 secs 2023-03-10T00:06:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:11 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:06:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:11 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:06:11.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:11 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:06:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:12 smithi122 ceph-mon[93869]: pgmap v18048: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 569 KiB/s wr, 29 op/s 2023-03-10T00:06:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:06:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:06:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:12 smithi181 ceph-mon[94189]: pgmap v18048: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 569 KiB/s wr, 29 op/s 2023-03-10T00:06:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:06:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:06:12.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:12 smithi038 ceph-mon[92422]: pgmap v18048: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 569 KiB/s wr, 29 op/s 2023-03-10T00:06:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:06:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:06:13.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:06:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:06:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32883.877891 secs 2023-03-10T00:06:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:13 smithi122 ceph-mon[93869]: pgmap v18049: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 667 KiB/s rd, 682 KiB/s wr, 28 op/s 2023-03-10T00:06:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:06:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:06:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32883.877891 secs 2023-03-10T00:06:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:13 smithi181 ceph-mon[94189]: pgmap v18049: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 667 KiB/s rd, 682 KiB/s wr, 28 op/s 2023-03-10T00:06:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:06:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:06:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32883.877891 secs 2023-03-10T00:06:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:13 smithi038 ceph-mon[92422]: pgmap v18049: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 667 KiB/s rd, 682 KiB/s wr, 28 op/s 2023-03-10T00:06:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33345.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:06:16.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32886.450977 secs 2023-03-10T00:06:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:16 smithi122 ceph-mon[93869]: pgmap v18050: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 647 KiB/s wr, 27 op/s 2023-03-10T00:06:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32886.450977 secs 2023-03-10T00:06:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:16 smithi181 ceph-mon[94189]: pgmap v18050: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 647 KiB/s wr, 27 op/s 2023-03-10T00:06:16.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32886.450977 secs 2023-03-10T00:06:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:16 smithi038 ceph-mon[92422]: pgmap v18050: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 647 KiB/s wr, 27 op/s 2023-03-10T00:06:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:18 smithi122 ceph-mon[93869]: pgmap v18051: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 417 KiB/s wr, 22 op/s 2023-03-10T00:06:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32888.878016 secs 2023-03-10T00:06:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:18 smithi122 ceph-mon[93869]: osdmap e1344: 12 total, 12 up, 12 in 2023-03-10T00:06:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:18 smithi181 ceph-mon[94189]: pgmap v18051: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 417 KiB/s wr, 22 op/s 2023-03-10T00:06:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32888.878016 secs 2023-03-10T00:06:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:18 smithi181 ceph-mon[94189]: osdmap e1344: 12 total, 12 up, 12 in 2023-03-10T00:06:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:18 smithi038 ceph-mon[92422]: pgmap v18051: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 417 KiB/s wr, 22 op/s 2023-03-10T00:06:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32888.878016 secs 2023-03-10T00:06:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:18 smithi038 ceph-mon[92422]: osdmap e1344: 12 total, 12 up, 12 in 2023-03-10T00:06:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33350.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:06:19.999 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 10 00:06:19 smithi038 sudo[271674]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-03-10T00:06:19.999 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 10 00:06:19 smithi038 sudo[271674]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-03-10T00:06:19.999 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 10 00:06:19 smithi038 sudo[271674]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-03-10T00:06:19.999 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 10 00:06:19 smithi038 sudo[271674]: pam_unix(sudo:session): session closed for user root 2023-03-10T00:06:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:20 smithi122 ceph-mon[93869]: pgmap v18053: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 364 KiB/s wr, 10 op/s 2023-03-10T00:06:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:20 smithi181 ceph-mon[94189]: pgmap v18053: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 364 KiB/s wr, 10 op/s 2023-03-10T00:06:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:20 smithi038 ceph-mon[92422]: pgmap v18053: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 364 KiB/s wr, 10 op/s 2023-03-10T00:06:20.749 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 10 00:06:20 smithi038 sudo[271677]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-03-10T00:06:20.749 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 10 00:06:20 smithi038 sudo[271677]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-03-10T00:06:20.749 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 10 00:06:20 smithi038 sudo[271677]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-03-10T00:06:20.750 INFO:journalctl@ceph.osd.0.smithi038.stdout:Mar 10 00:06:20 smithi038 sudo[271677]: pam_unix(sudo:session): session closed for user root 2023-03-10T00:06:21.191 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 10 00:06:20 smithi122 sudo[253759]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-03-10T00:06:21.191 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 10 00:06:20 smithi122 sudo[253759]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-03-10T00:06:21.191 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 10 00:06:20 smithi122 sudo[253759]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-03-10T00:06:21.192 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 10 00:06:20 smithi122 sudo[253759]: pam_unix(sudo:session): session closed for user root 2023-03-10T00:06:21.192 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 10 00:06:21 smithi122 sudo[253762]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-03-10T00:06:21.192 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 10 00:06:21 smithi122 sudo[253762]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-03-10T00:06:21.192 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 10 00:06:21 smithi122 sudo[253762]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-03-10T00:06:21.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32891.451038 secs 2023-03-10T00:06:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:06:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:06:21.476 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 10 00:06:21 smithi122 sudo[253762]: pam_unix(sudo:session): session closed for user root 2023-03-10T00:06:21.649 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32891.451038 secs 2023-03-10T00:06:21.649 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:06:21.650 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:06:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32891.451038 secs 2023-03-10T00:06:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:06:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:06:21.979 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 10 00:06:21 smithi181 sudo[250993]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-03-10T00:06:21.980 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 10 00:06:21 smithi181 sudo[250993]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-03-10T00:06:21.980 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 10 00:06:21 smithi181 sudo[250993]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-03-10T00:06:21.980 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 10 00:06:21 smithi181 sudo[250993]: pam_unix(sudo:session): session closed for user root 2023-03-10T00:06:22.352 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 10 00:06:21 smithi181 sudo[250996]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-03-10T00:06:22.353 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 10 00:06:21 smithi181 sudo[250996]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-03-10T00:06:22.353 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 10 00:06:21 smithi181 sudo[250996]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-03-10T00:06:22.353 INFO:journalctl@ceph.osd.2.smithi181.stdout:Mar 10 00:06:21 smithi181 sudo[250996]: pam_unix(sudo:session): session closed for user root 2023-03-10T00:06:22.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:22 smithi122 ceph-mon[93869]: pgmap v18054: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 295 KiB/s wr, 9 op/s 2023-03-10T00:06:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:22 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:06:22.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:22 smithi181 ceph-mon[94189]: pgmap v18054: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 295 KiB/s wr, 9 op/s 2023-03-10T00:06:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:22 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:06:22.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:22 smithi038 ceph-mon[92422]: pgmap v18054: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 295 KiB/s wr, 9 op/s 2023-03-10T00:06:22.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:22 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:06:22.749 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 10 00:06:22 smithi038 sudo[271680]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-03-10T00:06:22.749 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 10 00:06:22 smithi038 sudo[271680]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-03-10T00:06:22.749 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 10 00:06:22 smithi038 sudo[271680]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-03-10T00:06:22.750 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 10 00:06:22 smithi038 sudo[271680]: pam_unix(sudo:session): session closed for user root 2023-03-10T00:06:23.498 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 10 00:06:23 smithi038 sudo[271683]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-03-10T00:06:23.499 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 10 00:06:23 smithi038 sudo[271683]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-03-10T00:06:23.499 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 10 00:06:23 smithi038 sudo[271683]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-03-10T00:06:23.499 INFO:journalctl@ceph.osd.3.smithi038.stdout:Mar 10 00:06:23 smithi038 sudo[271683]: pam_unix(sudo:session): session closed for user root 2023-03-10T00:06:23.726 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 10 00:06:23 smithi122 sudo[253765]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-03-10T00:06:23.726 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 10 00:06:23 smithi122 sudo[253765]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-03-10T00:06:23.726 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 10 00:06:23 smithi122 sudo[253765]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-03-10T00:06:23.726 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 10 00:06:23 smithi122 sudo[253765]: pam_unix(sudo:session): session closed for user root 2023-03-10T00:06:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33355.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:06:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:24 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32893.878131 secs 2023-03-10T00:06:24.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:24 smithi122 ceph-mon[93869]: pgmap v18055: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 122 KiB/s wr, 8 op/s 2023-03-10T00:06:24.227 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 10 00:06:23 smithi122 sudo[253768]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-03-10T00:06:24.228 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 10 00:06:23 smithi122 sudo[253768]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-03-10T00:06:24.228 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 10 00:06:23 smithi122 sudo[253768]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-03-10T00:06:24.228 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 10 00:06:23 smithi122 sudo[253768]: pam_unix(sudo:session): session closed for user root 2023-03-10T00:06:24.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:24 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32893.878131 secs 2023-03-10T00:06:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:24 smithi181 ceph-mon[94189]: pgmap v18055: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 122 KiB/s wr, 8 op/s 2023-03-10T00:06:24.480 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 10 00:06:24 smithi181 sudo[250999]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-03-10T00:06:24.481 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 10 00:06:24 smithi181 sudo[250999]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-03-10T00:06:24.481 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 10 00:06:24 smithi181 sudo[250999]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-03-10T00:06:24.481 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 10 00:06:24 smithi181 sudo[250999]: pam_unix(sudo:session): session closed for user root 2023-03-10T00:06:24.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:24 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32893.878131 secs 2023-03-10T00:06:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:24 smithi038 ceph-mon[92422]: pgmap v18055: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 122 KiB/s wr, 8 op/s 2023-03-10T00:06:24.980 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 10 00:06:24 smithi181 sudo[251002]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-03-10T00:06:24.980 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 10 00:06:24 smithi181 sudo[251002]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-03-10T00:06:24.980 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 10 00:06:24 smithi181 sudo[251002]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-03-10T00:06:24.981 INFO:journalctl@ceph.osd.5.smithi181.stdout:Mar 10 00:06:24 smithi181 sudo[251002]: pam_unix(sudo:session): session closed for user root 2023-03-10T00:06:25.498 INFO:journalctl@ceph.osd.6.smithi038.stdout:Mar 10 00:06:25 smithi038 sudo[271686]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-03-10T00:06:25.499 INFO:journalctl@ceph.osd.6.smithi038.stdout:Mar 10 00:06:25 smithi038 sudo[271686]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-03-10T00:06:25.499 INFO:journalctl@ceph.osd.6.smithi038.stdout:Mar 10 00:06:25 smithi038 sudo[271686]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-03-10T00:06:25.499 INFO:journalctl@ceph.osd.6.smithi038.stdout:Mar 10 00:06:25 smithi038 sudo[271686]: pam_unix(sudo:session): session closed for user root 2023-03-10T00:06:25.999 INFO:journalctl@ceph.osd.6.smithi038.stdout:Mar 10 00:06:25 smithi038 sudo[271689]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-03-10T00:06:25.999 INFO:journalctl@ceph.osd.6.smithi038.stdout:Mar 10 00:06:25 smithi038 sudo[271689]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-03-10T00:06:25.999 INFO:journalctl@ceph.osd.6.smithi038.stdout:Mar 10 00:06:25 smithi038 sudo[271689]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-03-10T00:06:25.999 INFO:journalctl@ceph.osd.6.smithi038.stdout:Mar 10 00:06:25 smithi038 sudo[271689]: pam_unix(sudo:session): session closed for user root 2023-03-10T00:06:26.226 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 10 00:06:25 smithi122 sudo[253779]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-03-10T00:06:26.226 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 10 00:06:25 smithi122 sudo[253779]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-03-10T00:06:26.226 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 10 00:06:25 smithi122 sudo[253779]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-03-10T00:06:26.226 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 10 00:06:25 smithi122 sudo[253779]: pam_unix(sudo:session): session closed for user root 2023-03-10T00:06:26.628 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32896.451100 secs 2023-03-10T00:06:26.628 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:26 smithi181 ceph-mon[94189]: pgmap v18056: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 239 KiB/s wr, 19 op/s 2023-03-10T00:06:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32896.451100 secs 2023-03-10T00:06:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:26 smithi122 ceph-mon[93869]: pgmap v18056: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 239 KiB/s wr, 19 op/s 2023-03-10T00:06:26.726 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 10 00:06:26 smithi122 sudo[253782]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-03-10T00:06:26.726 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 10 00:06:26 smithi122 sudo[253782]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-03-10T00:06:26.727 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 10 00:06:26 smithi122 sudo[253782]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-03-10T00:06:26.727 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 10 00:06:26 smithi122 sudo[253782]: pam_unix(sudo:session): session closed for user root 2023-03-10T00:06:26.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32896.451100 secs 2023-03-10T00:06:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:26 smithi038 ceph-mon[92422]: pgmap v18056: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 239 KiB/s wr, 19 op/s 2023-03-10T00:06:26.942 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 10 00:06:26 smithi181 sudo[251005]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-03-10T00:06:26.943 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 10 00:06:26 smithi181 sudo[251005]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-03-10T00:06:26.943 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 10 00:06:26 smithi181 sudo[251005]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-03-10T00:06:26.943 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 10 00:06:26 smithi181 sudo[251005]: pam_unix(sudo:session): session closed for user root 2023-03-10T00:06:27.229 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 10 00:06:26 smithi181 sudo[251008]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-03-10T00:06:27.230 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 10 00:06:26 smithi181 sudo[251008]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-03-10T00:06:27.230 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 10 00:06:26 smithi181 sudo[251008]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-03-10T00:06:27.230 INFO:journalctl@ceph.osd.8.smithi181.stdout:Mar 10 00:06:26 smithi181 sudo[251008]: pam_unix(sudo:session): session closed for user root 2023-03-10T00:06:27.748 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 10 00:06:27 smithi038 sudo[271692]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-03-10T00:06:27.749 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 10 00:06:27 smithi038 sudo[271692]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-03-10T00:06:27.749 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 10 00:06:27 smithi038 sudo[271692]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-03-10T00:06:27.749 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 10 00:06:27 smithi038 sudo[271692]: pam_unix(sudo:session): session closed for user root 2023-03-10T00:06:28.320 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 10 00:06:28 smithi038 sudo[271695]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-03-10T00:06:28.320 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 10 00:06:28 smithi038 sudo[271695]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-03-10T00:06:28.320 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 10 00:06:28 smithi038 sudo[271695]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-03-10T00:06:28.320 INFO:journalctl@ceph.osd.9.smithi038.stdout:Mar 10 00:06:28 smithi038 sudo[271695]: pam_unix(sudo:session): session closed for user root 2023-03-10T00:06:28.725 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 10 00:06:28 smithi122 sudo[253785]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-03-10T00:06:28.726 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 10 00:06:28 smithi122 sudo[253785]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-03-10T00:06:28.726 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 10 00:06:28 smithi122 sudo[253785]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-03-10T00:06:28.726 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 10 00:06:28 smithi122 sudo[253785]: pam_unix(sudo:session): session closed for user root 2023-03-10T00:06:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:28 smithi122 ceph-mon[93869]: pgmap v18057: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 239 KiB/s wr, 18 op/s 2023-03-10T00:06:28.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32898.878157 secs 2023-03-10T00:06:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:28 smithi181 ceph-mon[94189]: pgmap v18057: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 239 KiB/s wr, 18 op/s 2023-03-10T00:06:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32898.878157 secs 2023-03-10T00:06:28.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:28 smithi038 ceph-mon[92422]: pgmap v18057: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 239 KiB/s wr, 18 op/s 2023-03-10T00:06:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32898.878157 secs 2023-03-10T00:06:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33360.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:06:28.975 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 10 00:06:28 smithi122 sudo[253788]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-03-10T00:06:28.976 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 10 00:06:28 smithi122 sudo[253788]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-03-10T00:06:28.976 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 10 00:06:28 smithi122 sudo[253788]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-03-10T00:06:28.976 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 10 00:06:28 smithi122 sudo[253788]: pam_unix(sudo:session): session closed for user root 2023-03-10T00:06:29.352 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 10 00:06:29 smithi181 sudo[251011]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-03-10T00:06:29.352 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 10 00:06:29 smithi181 sudo[251011]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-03-10T00:06:29.352 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 10 00:06:29 smithi181 sudo[251011]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-03-10T00:06:29.353 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 10 00:06:29 smithi181 sudo[251011]: pam_unix(sudo:session): session closed for user root 2023-03-10T00:06:29.730 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 10 00:06:29 smithi181 sudo[251014]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-03-10T00:06:29.730 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 10 00:06:29 smithi181 sudo[251014]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-03-10T00:06:29.730 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 10 00:06:29 smithi181 sudo[251014]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-03-10T00:06:29.730 INFO:journalctl@ceph.osd.11.smithi181.stdout:Mar 10 00:06:29 smithi181 sudo[251014]: pam_unix(sudo:session): session closed for user root 2023-03-10T00:06:30.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:29 smithi038 sudo[271698]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-03-10T00:06:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:29 smithi038 sudo[271698]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-03-10T00:06:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:29 smithi038 sudo[271698]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-03-10T00:06:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:29 smithi038 sudo[271698]: pam_unix(sudo:session): session closed for user root 2023-03-10T00:06:30.611 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:30 smithi181 ceph-mon[94189]: pgmap v18058: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 105 KiB/s wr, 16 op/s 2023-03-10T00:06:30.611 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:30 smithi181 ceph-mon[94189]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-03-10T00:06:30.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:30 smithi122 ceph-mon[93869]: pgmap v18058: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 105 KiB/s wr, 16 op/s 2023-03-10T00:06:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:30 smithi122 ceph-mon[93869]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-03-10T00:06:30.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:30 smithi038 sudo[271701]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-03-10T00:06:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:30 smithi038 sudo[271701]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-03-10T00:06:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:30 smithi038 sudo[271701]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-03-10T00:06:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:30 smithi038 sudo[271701]: pam_unix(sudo:session): session closed for user root 2023-03-10T00:06:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:30 smithi038 ceph-mon[92422]: pgmap v18058: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 105 KiB/s wr, 16 op/s 2023-03-10T00:06:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:30 smithi038 ceph-mon[92422]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-03-10T00:06:30.886 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:30 smithi181 sudo[251017]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-03-10T00:06:30.887 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:30 smithi181 sudo[251017]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-03-10T00:06:30.887 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:30 smithi181 sudo[251017]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-03-10T00:06:30.888 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:30 smithi181 sudo[251017]: pam_unix(sudo:session): session closed for user root 2023-03-10T00:06:31.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:30 smithi181 sudo[251020]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-03-10T00:06:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:30 smithi181 sudo[251020]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-03-10T00:06:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:30 smithi181 sudo[251020]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-03-10T00:06:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:30 smithi181 sudo[251020]: pam_unix(sudo:session): session closed for user root 2023-03-10T00:06:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:31 smithi122 sudo[253791]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-03-10T00:06:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:31 smithi122 sudo[253791]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-03-10T00:06:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:31 smithi122 sudo[253791]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-03-10T00:06:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:31 smithi122 sudo[253791]: pam_unix(sudo:session): session closed for user root 2023-03-10T00:06:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:31 smithi122 sudo[253794]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-03-10T00:06:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:31 smithi122 sudo[253794]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-03-10T00:06:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:31 smithi122 sudo[253794]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-03-10T00:06:31.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:31 smithi122 sudo[253794]: pam_unix(sudo:session): session closed for user root 2023-03-10T00:06:31.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32901.451161 secs 2023-03-10T00:06:31.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:31 smithi122 ceph-mon[93869]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-03-10T00:06:31.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-03-10T00:06:31.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-03-10T00:06:31.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-03-10T00:06:31.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:31 smithi122 ceph-mon[93869]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-03-10T00:06:31.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-03-10T00:06:31.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-03-10T00:06:31.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-03-10T00:06:31.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:31 smithi122 ceph-mon[93869]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-03-10T00:06:31.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-03-10T00:06:31.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-03-10T00:06:31.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-03-10T00:06:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32901.451161 secs 2023-03-10T00:06:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:31 smithi181 ceph-mon[94189]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-03-10T00:06:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-03-10T00:06:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-03-10T00:06:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-03-10T00:06:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:31 smithi181 ceph-mon[94189]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-03-10T00:06:31.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-03-10T00:06:31.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-03-10T00:06:31.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-03-10T00:06:31.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:31 smithi181 ceph-mon[94189]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-03-10T00:06:31.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-03-10T00:06:31.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-03-10T00:06:31.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-03-10T00:06:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32901.451161 secs 2023-03-10T00:06:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:31 smithi038 ceph-mon[92422]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-03-10T00:06:32.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-03-10T00:06:32.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-03-10T00:06:32.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-03-10T00:06:32.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:31 smithi038 ceph-mon[92422]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-03-10T00:06:32.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-03-10T00:06:32.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-03-10T00:06:32.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-03-10T00:06:32.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:31 smithi038 ceph-mon[92422]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-03-10T00:06:32.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-03-10T00:06:32.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-03-10T00:06:32.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-03-10T00:06:32.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:32 smithi122 ceph-mon[93869]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-03-10T00:06:32.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:32 smithi122 ceph-mon[93869]: pgmap v18059: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 98 KiB/s wr, 17 op/s 2023-03-10T00:06:32.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:32 smithi122 ceph-mon[93869]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-03-10T00:06:32.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:32 smithi181 ceph-mon[94189]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-03-10T00:06:32.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:32 smithi181 ceph-mon[94189]: pgmap v18059: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 98 KiB/s wr, 17 op/s 2023-03-10T00:06:32.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:32 smithi181 ceph-mon[94189]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-03-10T00:06:32.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:32 smithi038 ceph-mon[92422]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-03-10T00:06:32.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:32 smithi038 ceph-mon[92422]: pgmap v18059: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 98 KiB/s wr, 17 op/s 2023-03-10T00:06:32.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:32 smithi038 ceph-mon[92422]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-03-10T00:06:33.801 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33365.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:06:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:34 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32903.878278 secs 2023-03-10T00:06:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:34 smithi122 ceph-mon[93869]: pgmap v18060: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 98 KiB/s wr, 16 op/s 2023-03-10T00:06:34.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:34 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32903.878278 secs 2023-03-10T00:06:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:34 smithi181 ceph-mon[94189]: pgmap v18060: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 98 KiB/s wr, 16 op/s 2023-03-10T00:06:34.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:34 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32903.878278 secs 2023-03-10T00:06:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:34 smithi038 ceph-mon[92422]: pgmap v18060: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 98 KiB/s wr, 16 op/s 2023-03-10T00:06:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:06:35.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:06:35.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:06:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32906.451223 secs 2023-03-10T00:06:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:36 smithi122 ceph-mon[93869]: pgmap v18061: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 120 KiB/s wr, 17 op/s 2023-03-10T00:06:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:06:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:06:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:06:36.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32906.451223 secs 2023-03-10T00:06:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:36 smithi181 ceph-mon[94189]: pgmap v18061: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 120 KiB/s wr, 17 op/s 2023-03-10T00:06:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:06:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:06:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:06:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32906.451223 secs 2023-03-10T00:06:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:36 smithi038 ceph-mon[92422]: pgmap v18061: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 120 KiB/s wr, 17 op/s 2023-03-10T00:06:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:06:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:06:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:06:38.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:38 smithi122 ceph-mon[93869]: pgmap v18062: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 22 KiB/s wr, 11 op/s 2023-03-10T00:06:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32908.878397 secs 2023-03-10T00:06:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:38 smithi181 ceph-mon[94189]: pgmap v18062: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 22 KiB/s wr, 11 op/s 2023-03-10T00:06:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32908.878397 secs 2023-03-10T00:06:38.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:38 smithi038 ceph-mon[92422]: pgmap v18062: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 22 KiB/s wr, 11 op/s 2023-03-10T00:06:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32908.878397 secs 2023-03-10T00:06:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33370.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:06:40.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:40 smithi122 ceph-mon[93869]: pgmap v18063: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 150 KiB/s wr, 10 op/s 2023-03-10T00:06:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:06:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:06:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:40 smithi181 ceph-mon[94189]: pgmap v18063: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 150 KiB/s wr, 10 op/s 2023-03-10T00:06:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:06:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:06:40.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:40 smithi038 ceph-mon[92422]: pgmap v18063: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 150 KiB/s wr, 10 op/s 2023-03-10T00:06:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:06:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:06:41.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32911.451295 secs 2023-03-10T00:06:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:06:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:06:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32911.451295 secs 2023-03-10T00:06:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:06:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:06:41.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32911.451295 secs 2023-03-10T00:06:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:06:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:06:42.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:42 smithi122 ceph-mon[93869]: pgmap v18064: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 150 KiB/s wr, 11 op/s 2023-03-10T00:06:42.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:42 smithi181 ceph-mon[94189]: pgmap v18064: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 150 KiB/s wr, 11 op/s 2023-03-10T00:06:42.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:42 smithi038 ceph-mon[92422]: pgmap v18064: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 150 KiB/s wr, 11 op/s 2023-03-10T00:06:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33375.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:06:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:44 smithi122 ceph-mon[93869]: pgmap v18065: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 151 KiB/s wr, 10 op/s 2023-03-10T00:06:44.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:44 smithi181 ceph-mon[94189]: pgmap v18065: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 151 KiB/s wr, 10 op/s 2023-03-10T00:06:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:44 smithi038 ceph-mon[92422]: pgmap v18065: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 151 KiB/s wr, 10 op/s 2023-03-10T00:06:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32916.451355 secs 2023-03-10T00:06:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:46 smithi122 ceph-mon[93869]: pgmap v18066: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 150 KiB/s wr, 9 op/s 2023-03-10T00:06:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:46 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:06:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32916.451355 secs 2023-03-10T00:06:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:46 smithi181 ceph-mon[94189]: pgmap v18066: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 150 KiB/s wr, 9 op/s 2023-03-10T00:06:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:46 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:06:46.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32916.451355 secs 2023-03-10T00:06:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:46 smithi038 ceph-mon[92422]: pgmap v18066: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 150 KiB/s wr, 9 op/s 2023-03-10T00:06:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:46 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:06:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:47 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:06:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:06:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:06:47.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:47 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:06:47.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:06:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:06:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:47 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:06:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:06:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:06:48.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:48 smithi122 ceph-mon[93869]: pgmap v18067: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 129 KiB/s wr, 9 op/s 2023-03-10T00:06:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:48 smithi181 ceph-mon[94189]: pgmap v18067: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 129 KiB/s wr, 9 op/s 2023-03-10T00:06:48.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:48 smithi038 ceph-mon[92422]: pgmap v18067: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 129 KiB/s wr, 9 op/s 2023-03-10T00:06:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33380.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:06:50.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:50 smithi122 ceph-mon[93869]: pgmap v18068: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 591 KiB/s rd, 339 KiB/s wr, 8 op/s 2023-03-10T00:06:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:50 smithi181 ceph-mon[94189]: pgmap v18068: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 591 KiB/s rd, 339 KiB/s wr, 8 op/s 2023-03-10T00:06:50.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:50 smithi038 ceph-mon[92422]: pgmap v18068: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 591 KiB/s rd, 339 KiB/s wr, 8 op/s 2023-03-10T00:06:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32921.451424 secs 2023-03-10T00:06:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32921.451424 secs 2023-03-10T00:06:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32921.451424 secs 2023-03-10T00:06:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:52 smithi122 ceph-mon[93869]: pgmap v18069: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 211 KiB/s wr, 8 op/s 2023-03-10T00:06:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:52 smithi181 ceph-mon[94189]: pgmap v18069: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 211 KiB/s wr, 8 op/s 2023-03-10T00:06:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:52 smithi038 ceph-mon[92422]: pgmap v18069: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 211 KiB/s wr, 8 op/s 2023-03-10T00:06:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:06:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:06:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:06:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:06:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:06:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:06:53.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33385.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:06:54.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:54 smithi122 ceph-mon[93869]: pgmap v18070: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 945 KiB/s rd, 527 KiB/s wr, 8 op/s 2023-03-10T00:06:54.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:54 smithi181 ceph-mon[94189]: pgmap v18070: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 945 KiB/s rd, 527 KiB/s wr, 8 op/s 2023-03-10T00:06:54.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:54 smithi038 ceph-mon[92422]: pgmap v18070: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 945 KiB/s rd, 527 KiB/s wr, 8 op/s 2023-03-10T00:06:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:55 smithi122 ceph-mon[93869]: pgmap v18071: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 945 KiB/s rd, 526 KiB/s wr, 7 op/s 2023-03-10T00:06:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:55 smithi181 ceph-mon[94189]: pgmap v18071: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 945 KiB/s rd, 526 KiB/s wr, 7 op/s 2023-03-10T00:06:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:55 smithi038 ceph-mon[92422]: pgmap v18071: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 945 KiB/s rd, 526 KiB/s wr, 7 op/s 2023-03-10T00:06:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32926.451482 secs 2023-03-10T00:06:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:06:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:06:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32926.451482 secs 2023-03-10T00:06:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:06:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:06:56.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32926.451482 secs 2023-03-10T00:06:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:06:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:06:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:57 smithi122 ceph-mon[93869]: pgmap v18072: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 681 KiB/s wr, 11 op/s 2023-03-10T00:06:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:57 smithi181 ceph-mon[94189]: pgmap v18072: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 681 KiB/s wr, 11 op/s 2023-03-10T00:06:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:57 smithi038 ceph-mon[92422]: pgmap v18072: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 681 KiB/s wr, 11 op/s 2023-03-10T00:06:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33390.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:06:59.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:59 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 32928.878892 secs 2023-03-10T00:06:59.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:59 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:06:59.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:06:59.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:06:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:06:59.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:59 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 32928.878892 secs 2023-03-10T00:06:59.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:59 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:06:59.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:06:59.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:06:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:06:59.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:59 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 32928.878892 secs 2023-03-10T00:06:59.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:59 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:06:59.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:06:59.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:06:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:07:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:00 smithi122 ceph-mon[93869]: pgmap v18073: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 711 KiB/s wr, 11 op/s 2023-03-10T00:07:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:07:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:07:00.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:00 smithi181 ceph-mon[94189]: pgmap v18073: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 711 KiB/s wr, 11 op/s 2023-03-10T00:07:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:07:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:07:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:00 smithi038 ceph-mon[92422]: pgmap v18073: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 711 KiB/s wr, 11 op/s 2023-03-10T00:07:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:07:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:07:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32931.451540 secs 2023-03-10T00:07:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:07:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:07:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:01 smithi122 ceph-mon[93869]: pgmap v18074: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 923 KiB/s rd, 742 KiB/s wr, 13 op/s 2023-03-10T00:07:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32931.451540 secs 2023-03-10T00:07:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:07:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:07:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:01 smithi181 ceph-mon[94189]: pgmap v18074: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 923 KiB/s rd, 742 KiB/s wr, 13 op/s 2023-03-10T00:07:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32931.451540 secs 2023-03-10T00:07:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:07:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:07:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:01 smithi038 ceph-mon[92422]: pgmap v18074: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 923 KiB/s rd, 742 KiB/s wr, 13 op/s 2023-03-10T00:07:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33395.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:07:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:04 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32933.879037 secs 2023-03-10T00:07:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:04 smithi122 ceph-mon[93869]: pgmap v18075: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1012 KiB/s rd, 855 KiB/s wr, 15 op/s 2023-03-10T00:07:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:04 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32933.879037 secs 2023-03-10T00:07:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:04 smithi181 ceph-mon[94189]: pgmap v18075: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1012 KiB/s rd, 855 KiB/s wr, 15 op/s 2023-03-10T00:07:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:04 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32933.879037 secs 2023-03-10T00:07:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:04 smithi038 ceph-mon[92422]: pgmap v18075: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1012 KiB/s rd, 855 KiB/s wr, 15 op/s 2023-03-10T00:07:06.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32936.451602 secs 2023-03-10T00:07:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:06 smithi122 ceph-mon[93869]: pgmap v18076: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 531 KiB/s rd, 539 KiB/s wr, 12 op/s 2023-03-10T00:07:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32936.451602 secs 2023-03-10T00:07:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:06 smithi181 ceph-mon[94189]: pgmap v18076: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 531 KiB/s rd, 539 KiB/s wr, 12 op/s 2023-03-10T00:07:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32936.451602 secs 2023-03-10T00:07:06.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:06 smithi038 ceph-mon[92422]: pgmap v18076: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 531 KiB/s rd, 539 KiB/s wr, 12 op/s 2023-03-10T00:07:07.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:07:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:07:07.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:07:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:07:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:07:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:07:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:08 smithi122 ceph-mon[93869]: pgmap v18077: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 555 KiB/s rd, 585 KiB/s wr, 18 op/s 2023-03-10T00:07:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:07:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:07:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32938.879155 secs 2023-03-10T00:07:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:08 smithi181 ceph-mon[94189]: pgmap v18077: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 555 KiB/s rd, 585 KiB/s wr, 18 op/s 2023-03-10T00:07:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:07:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:07:08.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32938.879155 secs 2023-03-10T00:07:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:08 smithi038 ceph-mon[92422]: pgmap v18077: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 555 KiB/s rd, 585 KiB/s wr, 18 op/s 2023-03-10T00:07:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:07:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:07:08.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32938.879155 secs 2023-03-10T00:07:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33400.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:07:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:07:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:07:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:07:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:07:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:07:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:07:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:10 smithi122 ceph-mon[93869]: pgmap v18078: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 634 KiB/s rd, 657 KiB/s wr, 27 op/s 2023-03-10T00:07:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:10 smithi122 ceph-mon[93869]: osdmap e1345: 12 total, 12 up, 12 in 2023-03-10T00:07:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:10 smithi181 ceph-mon[94189]: pgmap v18078: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 634 KiB/s rd, 657 KiB/s wr, 27 op/s 2023-03-10T00:07:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:10 smithi181 ceph-mon[94189]: osdmap e1345: 12 total, 12 up, 12 in 2023-03-10T00:07:10.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:10 smithi038 ceph-mon[92422]: pgmap v18078: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 634 KiB/s rd, 657 KiB/s wr, 27 op/s 2023-03-10T00:07:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:10 smithi038 ceph-mon[92422]: osdmap e1345: 12 total, 12 up, 12 in 2023-03-10T00:07:11.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32941.451662 secs 2023-03-10T00:07:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32941.451662 secs 2023-03-10T00:07:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32941.451662 secs 2023-03-10T00:07:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:12 smithi122 ceph-mon[93869]: pgmap v18080: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 464 KiB/s wr, 29 op/s 2023-03-10T00:07:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:12 smithi181 ceph-mon[94189]: pgmap v18080: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 464 KiB/s wr, 29 op/s 2023-03-10T00:07:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:12 smithi038 ceph-mon[92422]: pgmap v18080: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 464 KiB/s wr, 29 op/s 2023-03-10T00:07:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33405.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:07:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:14 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32943.879277 secs 2023-03-10T00:07:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:14 smithi122 ceph-mon[93869]: pgmap v18081: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 501 KiB/s wr, 28 op/s 2023-03-10T00:07:14.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:14 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32943.879277 secs 2023-03-10T00:07:14.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:14 smithi181 ceph-mon[94189]: pgmap v18081: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 501 KiB/s wr, 28 op/s 2023-03-10T00:07:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:14 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32943.879277 secs 2023-03-10T00:07:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:14 smithi038 ceph-mon[92422]: pgmap v18081: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 501 KiB/s wr, 28 op/s 2023-03-10T00:07:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:15 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mon-b[93845]: problem writing to /var/log/ceph/ceph-mon.b.log: (28) No space left on device 2023-03-10T00:07:15.726 INFO:journalctl@ceph.osd.1.smithi122.stdout:Mar 10 00:07:15 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-1[130534]: problem writing to /var/log/ceph/ceph-osd.1.log: (28) No space left on device 2023-03-10T00:07:15.727 INFO:journalctl@ceph.osd.4.smithi122.stdout:Mar 10 00:07:15 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-4[124955]: problem writing to /var/log/ceph/ceph-osd.4.log: (28) No space left on device 2023-03-10T00:07:15.727 INFO:journalctl@ceph.osd.7.smithi122.stdout:Mar 10 00:07:15 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-7[132036]: problem writing to /var/log/ceph/ceph-osd.7.log: (28) No space left on device 2023-03-10T00:07:15.727 INFO:journalctl@ceph.osd.10.smithi122.stdout:Mar 10 00:07:15 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-osd-10[115379]: problem writing to /var/log/ceph/ceph-osd.10.log: (28) No space left on device 2023-03-10T00:07:15.728 INFO:journalctl@ceph.mds.h.smithi122.stdout:Mar 10 00:07:15 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mds-h[118806]: problem writing to /var/log/ceph/ceph-mds.h.log: (28) No space left on device 2023-03-10T00:07:15.728 INFO:journalctl@ceph.mds.b.smithi122.stdout:Mar 10 00:07:15 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mds-b[119145]: problem writing to /var/log/ceph/ceph-mds.b.log: (28) No space left on device 2023-03-10T00:07:16.226 INFO:journalctl@ceph.mds.e.smithi122.stdout:Mar 10 00:07:15 smithi122 ceph-c6733fe2-be87-11ed-9af2-001a4aab830c-mds-e[118464]: problem writing to /var/log/ceph/ceph-mds.e.log: (28) No space left on device 2023-03-10T00:07:16.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32946.451720 secs 2023-03-10T00:07:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:16 smithi122 ceph-mon[93869]: pgmap v18082: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 501 KiB/s wr, 29 op/s 2023-03-10T00:07:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32946.451720 secs 2023-03-10T00:07:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:16 smithi181 ceph-mon[94189]: pgmap v18082: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 501 KiB/s wr, 29 op/s 2023-03-10T00:07:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32946.451720 secs 2023-03-10T00:07:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:16 smithi038 ceph-mon[92422]: pgmap v18082: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 501 KiB/s wr, 29 op/s 2023-03-10T00:07:17.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:07:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:07:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:07:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:07:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:07:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:07:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:18 smithi122 ceph-mon[93869]: pgmap v18083: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 447 KiB/s wr, 31 op/s 2023-03-10T00:07:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32948.879406 secs 2023-03-10T00:07:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:18 smithi122 ceph-mon[93869]: osdmap e1346: 12 total, 12 up, 12 in 2023-03-10T00:07:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:18 smithi181 ceph-mon[94189]: pgmap v18083: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 447 KiB/s wr, 31 op/s 2023-03-10T00:07:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32948.879406 secs 2023-03-10T00:07:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:18 smithi181 ceph-mon[94189]: osdmap e1346: 12 total, 12 up, 12 in 2023-03-10T00:07:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:18 smithi038 ceph-mon[92422]: pgmap v18083: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 447 KiB/s wr, 31 op/s 2023-03-10T00:07:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32948.879406 secs 2023-03-10T00:07:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:18 smithi038 ceph-mon[92422]: osdmap e1346: 12 total, 12 up, 12 in 2023-03-10T00:07:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33410.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:07:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:20 smithi122 ceph-mon[93869]: pgmap v18085: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 298 KiB/s wr, 19 op/s 2023-03-10T00:07:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:07:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:07:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:20 smithi181 ceph-mon[94189]: pgmap v18085: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 298 KiB/s wr, 19 op/s 2023-03-10T00:07:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:07:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:07:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:20 smithi038 ceph-mon[92422]: pgmap v18085: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 298 KiB/s wr, 19 op/s 2023-03-10T00:07:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:07:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:07:21.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32951.451785 secs 2023-03-10T00:07:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:07:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32951.451785 secs 2023-03-10T00:07:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:07:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32951.451785 secs 2023-03-10T00:07:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:07:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:22 smithi122 ceph-mon[93869]: pgmap v18086: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 300 KiB/s wr, 21 op/s 2023-03-10T00:07:22.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:22 smithi181 ceph-mon[94189]: pgmap v18086: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 300 KiB/s wr, 21 op/s 2023-03-10T00:07:22.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:22 smithi038 ceph-mon[92422]: pgmap v18086: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 300 KiB/s wr, 21 op/s 2023-03-10T00:07:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33415.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:07:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:24 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32953.879535 secs 2023-03-10T00:07:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:24 smithi122 ceph-mon[93869]: pgmap v18087: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1006 KiB/s rd, 1006 KiB/s wr, 25 op/s 2023-03-10T00:07:24.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:24 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32953.879535 secs 2023-03-10T00:07:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:24 smithi181 ceph-mon[94189]: pgmap v18087: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1006 KiB/s rd, 1006 KiB/s wr, 25 op/s 2023-03-10T00:07:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:24 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32953.879535 secs 2023-03-10T00:07:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:24 smithi038 ceph-mon[92422]: pgmap v18087: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1006 KiB/s rd, 1006 KiB/s wr, 25 op/s 2023-03-10T00:07:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32956.451844 secs 2023-03-10T00:07:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:26 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:07:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:26 smithi122 ceph-mon[93869]: pgmap v18088: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1006 KiB/s rd, 1006 KiB/s wr, 25 op/s 2023-03-10T00:07:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32956.451844 secs 2023-03-10T00:07:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:26 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:07:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:26 smithi181 ceph-mon[94189]: pgmap v18088: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1006 KiB/s rd, 1006 KiB/s wr, 25 op/s 2023-03-10T00:07:26.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32956.451844 secs 2023-03-10T00:07:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:26 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:07:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:26 smithi038 ceph-mon[92422]: pgmap v18088: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1006 KiB/s rd, 1006 KiB/s wr, 25 op/s 2023-03-10T00:07:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:28 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:07:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:28 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:07:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:28 smithi122 ceph-mon[93869]: pgmap v18089: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 20 op/s 2023-03-10T00:07:28.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:28 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:07:28.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:28 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:07:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:28 smithi181 ceph-mon[94189]: pgmap v18089: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 20 op/s 2023-03-10T00:07:28.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:28 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:07:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:28 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:07:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:28 smithi038 ceph-mon[92422]: pgmap v18089: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 20 op/s 2023-03-10T00:07:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33420.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:07:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:29 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32958.879669 secs 2023-03-10T00:07:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:29 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:07:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:29 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:07:29.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:29 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32958.879669 secs 2023-03-10T00:07:29.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:29 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:07:29.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:29 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:07:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:29 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32958.879669 secs 2023-03-10T00:07:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:29 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:07:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:29 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:07:30.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:30 smithi122 ceph-mon[93869]: pgmap v18090: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 925 KiB/s rd, 923 KiB/s wr, 15 op/s 2023-03-10T00:07:30.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:30 smithi181 ceph-mon[94189]: pgmap v18090: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 925 KiB/s rd, 923 KiB/s wr, 15 op/s 2023-03-10T00:07:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:30 smithi038 ceph-mon[92422]: pgmap v18090: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 925 KiB/s rd, 923 KiB/s wr, 15 op/s 2023-03-10T00:07:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:31 smithi122 ceph-mon[93869]: scrub summary: active paths [/] 2023-03-10T00:07:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:31 smithi122 ceph-mon[93869]: scrub summary: idle+waiting paths [/] 2023-03-10T00:07:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32961.451904 secs 2023-03-10T00:07:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:31 smithi181 ceph-mon[94189]: scrub summary: active paths [/] 2023-03-10T00:07:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:31 smithi181 ceph-mon[94189]: scrub summary: idle+waiting paths [/] 2023-03-10T00:07:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32961.451904 secs 2023-03-10T00:07:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:31 smithi038 ceph-mon[92422]: scrub summary: active paths [/] 2023-03-10T00:07:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:31 smithi038 ceph-mon[92422]: scrub summary: idle+waiting paths [/] 2023-03-10T00:07:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32961.451904 secs 2023-03-10T00:07:32.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:32 smithi122 ceph-mon[93869]: pgmap v18091: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.4 MiB/s wr, 20 op/s 2023-03-10T00:07:32.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:32 smithi181 ceph-mon[94189]: pgmap v18091: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.4 MiB/s wr, 20 op/s 2023-03-10T00:07:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:32 smithi038 ceph-mon[92422]: pgmap v18091: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.4 MiB/s wr, 20 op/s 2023-03-10T00:07:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33425.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:07:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:34 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32963.879726 secs 2023-03-10T00:07:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:34 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:07:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:34 smithi122 ceph-mon[93869]: pgmap v18092: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.2 MiB/s wr, 26 op/s 2023-03-10T00:07:34.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:34 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32963.879726 secs 2023-03-10T00:07:34.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:34 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:07:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:34 smithi181 ceph-mon[94189]: pgmap v18092: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.2 MiB/s wr, 26 op/s 2023-03-10T00:07:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:34 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32963.879726 secs 2023-03-10T00:07:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:34 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:07:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:34 smithi038 ceph-mon[92422]: pgmap v18092: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 4.2 MiB/s wr, 26 op/s 2023-03-10T00:07:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32966.451960 secs 2023-03-10T00:07:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:07:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:07:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:36 smithi122 ceph-mon[93869]: pgmap v18093: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.6 MiB/s wr, 22 op/s 2023-03-10T00:07:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:07:36.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32966.451960 secs 2023-03-10T00:07:36.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:07:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:07:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:36 smithi181 ceph-mon[94189]: pgmap v18093: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.6 MiB/s wr, 22 op/s 2023-03-10T00:07:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:07:36.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32966.451960 secs 2023-03-10T00:07:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:07:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:07:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:36 smithi038 ceph-mon[92422]: pgmap v18093: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.6 MiB/s wr, 22 op/s 2023-03-10T00:07:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:07:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:07:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:07:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:07:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:07:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:07:37.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:07:37.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:07:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:07:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:07:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:07:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:07:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:07:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:07:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:07:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:07:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:38 smithi122 ceph-mon[93869]: pgmap v18094: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.6 MiB/s wr, 26 op/s 2023-03-10T00:07:38.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:38 smithi181 ceph-mon[94189]: pgmap v18094: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.6 MiB/s wr, 26 op/s 2023-03-10T00:07:38.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:38 smithi038 ceph-mon[92422]: pgmap v18094: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.6 MiB/s wr, 26 op/s 2023-03-10T00:07:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33430.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:07:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:39 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32968.879787 secs 2023-03-10T00:07:39.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:39 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32968.879787 secs 2023-03-10T00:07:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:39 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32968.879787 secs 2023-03-10T00:07:40.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:07:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:07:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:40 smithi122 ceph-mon[93869]: pgmap v18095: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.6 MiB/s wr, 25 op/s 2023-03-10T00:07:40.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:07:40.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:07:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:40 smithi181 ceph-mon[94189]: pgmap v18095: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.6 MiB/s wr, 25 op/s 2023-03-10T00:07:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:07:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:07:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:40 smithi038 ceph-mon[92422]: pgmap v18095: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.6 MiB/s wr, 25 op/s 2023-03-10T00:07:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32971.452019 secs 2023-03-10T00:07:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:07:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:07:41.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32971.452019 secs 2023-03-10T00:07:41.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:07:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:07:41.505 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32971.452019 secs 2023-03-10T00:07:41.505 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:07:41.505 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:07:42.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:42 smithi122 ceph-mon[93869]: pgmap v18096: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.6 MiB/s wr, 28 op/s 2023-03-10T00:07:42.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:42 smithi181 ceph-mon[94189]: pgmap v18096: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.6 MiB/s wr, 28 op/s 2023-03-10T00:07:42.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:42 smithi038 ceph-mon[92422]: pgmap v18096: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 MiB/s rd, 3.6 MiB/s wr, 28 op/s 2023-03-10T00:07:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33435.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:07:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:44 smithi122 ceph-mon[93869]: pgmap v18097: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.0 MiB/s wr, 25 op/s 2023-03-10T00:07:44.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:44 smithi181 ceph-mon[94189]: pgmap v18097: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.0 MiB/s wr, 25 op/s 2023-03-10T00:07:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:44 smithi038 ceph-mon[92422]: pgmap v18097: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.0 MiB/s wr, 25 op/s 2023-03-10T00:07:46.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32976.452079 secs 2023-03-10T00:07:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:46 smithi122 ceph-mon[93869]: pgmap v18098: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 595 KiB/s rd, 251 KiB/s wr, 16 op/s 2023-03-10T00:07:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:46 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:07:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32976.452079 secs 2023-03-10T00:07:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:46 smithi181 ceph-mon[94189]: pgmap v18098: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 595 KiB/s rd, 251 KiB/s wr, 16 op/s 2023-03-10T00:07:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:46 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:07:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32976.452079 secs 2023-03-10T00:07:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:46 smithi038 ceph-mon[92422]: pgmap v18098: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 595 KiB/s rd, 251 KiB/s wr, 16 op/s 2023-03-10T00:07:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:46 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:07:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:47 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:07:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:07:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:07:47.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:47 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:07:47.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:07:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:07:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:47 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:07:47.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:07:47.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:07:48.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:48 smithi122 ceph-mon[93869]: pgmap v18099: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 138 KiB/s wr, 18 op/s 2023-03-10T00:07:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:48 smithi181 ceph-mon[94189]: pgmap v18099: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 138 KiB/s wr, 18 op/s 2023-03-10T00:07:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:48 smithi038 ceph-mon[92422]: pgmap v18099: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 138 KiB/s wr, 18 op/s 2023-03-10T00:07:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33440.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:07:50.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:50 smithi122 ceph-mon[93869]: pgmap v18100: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 580 KiB/s rd, 342 KiB/s wr, 15 op/s 2023-03-10T00:07:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:50 smithi181 ceph-mon[94189]: pgmap v18100: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 580 KiB/s rd, 342 KiB/s wr, 15 op/s 2023-03-10T00:07:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:50 smithi038 ceph-mon[92422]: pgmap v18100: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 580 KiB/s rd, 342 KiB/s wr, 15 op/s 2023-03-10T00:07:51.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32981.452137 secs 2023-03-10T00:07:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32981.452137 secs 2023-03-10T00:07:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32981.452137 secs 2023-03-10T00:07:52.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:52 smithi122 ceph-mon[93869]: pgmap v18101: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 444 KiB/s rd, 205 KiB/s wr, 15 op/s 2023-03-10T00:07:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:52 smithi181 ceph-mon[94189]: pgmap v18101: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 444 KiB/s rd, 205 KiB/s wr, 15 op/s 2023-03-10T00:07:52.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:52 smithi038 ceph-mon[92422]: pgmap v18101: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 444 KiB/s rd, 205 KiB/s wr, 15 op/s 2023-03-10T00:07:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:07:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:07:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:53 smithi122 ceph-mon[93869]: pgmap v18102: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 763 KiB/s rd, 526 KiB/s wr, 13 op/s 2023-03-10T00:07:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:07:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:07:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:53 smithi181 ceph-mon[94189]: pgmap v18102: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 763 KiB/s rd, 526 KiB/s wr, 13 op/s 2023-03-10T00:07:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:07:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:07:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:53 smithi038 ceph-mon[92422]: pgmap v18102: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 763 KiB/s rd, 526 KiB/s wr, 13 op/s 2023-03-10T00:07:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33445.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:07:56.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32986.452198 secs 2023-03-10T00:07:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:07:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:07:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:56 smithi122 ceph-mon[93869]: pgmap v18103: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 842 KiB/s rd, 526 KiB/s wr, 10 op/s 2023-03-10T00:07:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32986.452198 secs 2023-03-10T00:07:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:07:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:07:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:56 smithi181 ceph-mon[94189]: pgmap v18103: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 842 KiB/s rd, 526 KiB/s wr, 10 op/s 2023-03-10T00:07:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32986.452198 secs 2023-03-10T00:07:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:07:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:07:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:56 smithi038 ceph-mon[92422]: pgmap v18103: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 842 KiB/s rd, 526 KiB/s wr, 10 op/s 2023-03-10T00:07:58.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:58 smithi122 ceph-mon[93869]: pgmap v18104: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 710 KiB/s wr, 13 op/s 2023-03-10T00:07:58.764 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:58 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 32988.880159 secs 2023-03-10T00:07:58.765 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:58 smithi038 ceph-mon[92422]: pgmap v18104: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 710 KiB/s wr, 13 op/s 2023-03-10T00:07:58.765 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:58 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 32988.880159 secs 2023-03-10T00:07:58.765 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:58 smithi181 ceph-mon[94189]: pgmap v18104: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 710 KiB/s wr, 13 op/s 2023-03-10T00:07:58.765 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:58 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 32988.880159 secs 2023-03-10T00:07:58.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33450.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:07:59.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:59 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:07:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:07:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:07:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:07:59.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:59 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:07:59.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:07:59.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:07:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:07:59.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:59 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:07:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:07:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:07:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:08:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:00 smithi122 ceph-mon[93869]: pgmap v18105: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 710 KiB/s wr, 11 op/s 2023-03-10T00:08:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:08:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:08:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:00 smithi181 ceph-mon[94189]: pgmap v18105: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 710 KiB/s wr, 11 op/s 2023-03-10T00:08:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:08:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:08:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:00 smithi038 ceph-mon[92422]: pgmap v18105: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 710 KiB/s wr, 11 op/s 2023-03-10T00:08:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:08:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:08:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32991.452261 secs 2023-03-10T00:08:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:08:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:08:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32991.452261 secs 2023-03-10T00:08:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:08:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:08:01.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32991.452261 secs 2023-03-10T00:08:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:08:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:08:02.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:02 smithi122 ceph-mon[93869]: pgmap v18106: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 834 KiB/s rd, 741 KiB/s wr, 13 op/s 2023-03-10T00:08:02.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:02 smithi181 ceph-mon[94189]: pgmap v18106: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 834 KiB/s rd, 741 KiB/s wr, 13 op/s 2023-03-10T00:08:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:02 smithi038 ceph-mon[92422]: pgmap v18106: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 834 KiB/s rd, 741 KiB/s wr, 13 op/s 2023-03-10T00:08:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33455.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:08:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:04 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 32993.880234 secs 2023-03-10T00:08:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:04 smithi122 ceph-mon[93869]: pgmap v18107: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 930 KiB/s rd, 838 KiB/s wr, 14 op/s 2023-03-10T00:08:04.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:04 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 32993.880234 secs 2023-03-10T00:08:04.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:04 smithi181 ceph-mon[94189]: pgmap v18107: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 930 KiB/s rd, 838 KiB/s wr, 14 op/s 2023-03-10T00:08:04.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:04 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 32993.880234 secs 2023-03-10T00:08:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:04 smithi038 ceph-mon[92422]: pgmap v18107: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 930 KiB/s rd, 838 KiB/s wr, 14 op/s 2023-03-10T00:08:06.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32996.452322 secs 2023-03-10T00:08:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:06 smithi122 ceph-mon[93869]: pgmap v18108: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 610 KiB/s rd, 517 KiB/s wr, 13 op/s 2023-03-10T00:08:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32996.452322 secs 2023-03-10T00:08:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:06 smithi181 ceph-mon[94189]: pgmap v18108: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 610 KiB/s rd, 517 KiB/s wr, 13 op/s 2023-03-10T00:08:06.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32996.452322 secs 2023-03-10T00:08:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:06 smithi038 ceph-mon[92422]: pgmap v18108: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 610 KiB/s rd, 517 KiB/s wr, 13 op/s 2023-03-10T00:08:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:08 smithi122 ceph-mon[93869]: pgmap v18109: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 753 KiB/s rd, 759 KiB/s wr, 19 op/s 2023-03-10T00:08:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:08:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:08:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 32998.880312 secs 2023-03-10T00:08:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:08 smithi181 ceph-mon[94189]: pgmap v18109: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 753 KiB/s rd, 759 KiB/s wr, 19 op/s 2023-03-10T00:08:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:08:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:08:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 32998.880312 secs 2023-03-10T00:08:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:08 smithi038 ceph-mon[92422]: pgmap v18109: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 753 KiB/s rd, 759 KiB/s wr, 19 op/s 2023-03-10T00:08:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:08:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:08:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 32998.880312 secs 2023-03-10T00:08:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33460.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:08:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:08:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:08:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:08:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:08:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:08:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:08:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:10 smithi122 ceph-mon[93869]: pgmap v18110: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 573 KiB/s rd, 597 KiB/s wr, 26 op/s 2023-03-10T00:08:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:10 smithi122 ceph-mon[93869]: osdmap e1347: 12 total, 12 up, 12 in 2023-03-10T00:08:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:10 smithi181 ceph-mon[94189]: pgmap v18110: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 573 KiB/s rd, 597 KiB/s wr, 26 op/s 2023-03-10T00:08:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:10 smithi181 ceph-mon[94189]: osdmap e1347: 12 total, 12 up, 12 in 2023-03-10T00:08:10.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:10 smithi038 ceph-mon[92422]: pgmap v18110: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 573 KiB/s rd, 597 KiB/s wr, 26 op/s 2023-03-10T00:08:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:10 smithi038 ceph-mon[92422]: osdmap e1347: 12 total, 12 up, 12 in 2023-03-10T00:08:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33001.452382 secs 2023-03-10T00:08:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:11 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:08:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:11 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:08:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:11 smithi122 ceph-mon[93869]: pgmap v18112: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 630 KiB/s rd, 671 KiB/s wr, 29 op/s 2023-03-10T00:08:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33001.452382 secs 2023-03-10T00:08:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:11 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:08:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:11 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:08:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:11 smithi181 ceph-mon[94189]: pgmap v18112: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 630 KiB/s rd, 671 KiB/s wr, 29 op/s 2023-03-10T00:08:11.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33001.452382 secs 2023-03-10T00:08:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:11 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:08:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:11 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:08:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:11 smithi038 ceph-mon[92422]: pgmap v18112: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 630 KiB/s rd, 671 KiB/s wr, 29 op/s 2023-03-10T00:08:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:08:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:08:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:08:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:08:12.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:08:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:08:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:08:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:08:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33003.880417 secs 2023-03-10T00:08:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:13 smithi122 ceph-mon[93869]: pgmap v18113: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 636 KiB/s rd, 678 KiB/s wr, 27 op/s 2023-03-10T00:08:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:08:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:08:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33003.880417 secs 2023-03-10T00:08:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:13 smithi181 ceph-mon[94189]: pgmap v18113: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 636 KiB/s rd, 678 KiB/s wr, 27 op/s 2023-03-10T00:08:13.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:08:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:08:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33003.880417 secs 2023-03-10T00:08:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:13 smithi038 ceph-mon[92422]: pgmap v18113: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 636 KiB/s rd, 678 KiB/s wr, 27 op/s 2023-03-10T00:08:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33465.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:08:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33006.452463 secs 2023-03-10T00:08:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:16 smithi122 ceph-mon[93869]: pgmap v18114: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 636 KiB/s rd, 677 KiB/s wr, 27 op/s 2023-03-10T00:08:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33006.452463 secs 2023-03-10T00:08:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:16 smithi181 ceph-mon[94189]: pgmap v18114: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 636 KiB/s rd, 677 KiB/s wr, 27 op/s 2023-03-10T00:08:16.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33006.452463 secs 2023-03-10T00:08:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:16 smithi038 ceph-mon[92422]: pgmap v18114: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 636 KiB/s rd, 677 KiB/s wr, 27 op/s 2023-03-10T00:08:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:18 smithi122 ceph-mon[93869]: pgmap v18115: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 494 KiB/s rd, 509 KiB/s wr, 23 op/s 2023-03-10T00:08:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33008.880476 secs 2023-03-10T00:08:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:18 smithi122 ceph-mon[93869]: osdmap e1348: 12 total, 12 up, 12 in 2023-03-10T00:08:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:18 smithi181 ceph-mon[94189]: pgmap v18115: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 494 KiB/s rd, 509 KiB/s wr, 23 op/s 2023-03-10T00:08:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33008.880476 secs 2023-03-10T00:08:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:18 smithi181 ceph-mon[94189]: osdmap e1348: 12 total, 12 up, 12 in 2023-03-10T00:08:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:18 smithi038 ceph-mon[92422]: pgmap v18115: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 494 KiB/s rd, 509 KiB/s wr, 23 op/s 2023-03-10T00:08:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33008.880476 secs 2023-03-10T00:08:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:18 smithi038 ceph-mon[92422]: osdmap e1348: 12 total, 12 up, 12 in 2023-03-10T00:08:18.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33470.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:08:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:20 smithi122 ceph-mon[93869]: pgmap v18117: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 487 KiB/s wr, 10 op/s 2023-03-10T00:08:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:20 smithi181 ceph-mon[94189]: pgmap v18117: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 487 KiB/s wr, 10 op/s 2023-03-10T00:08:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:20 smithi038 ceph-mon[92422]: pgmap v18117: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 487 KiB/s wr, 10 op/s 2023-03-10T00:08:21.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33011.452520 secs 2023-03-10T00:08:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33011.452520 secs 2023-03-10T00:08:21.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33011.452520 secs 2023-03-10T00:08:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:22 smithi122 ceph-mon[93869]: pgmap v18118: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 245 KiB/s wr, 9 op/s 2023-03-10T00:08:22.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:22 smithi181 ceph-mon[94189]: pgmap v18118: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 245 KiB/s wr, 9 op/s 2023-03-10T00:08:22.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:22 smithi038 ceph-mon[92422]: pgmap v18118: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 245 KiB/s wr, 9 op/s 2023-03-10T00:08:23.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33475.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:08:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:24 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33013.880566 secs 2023-03-10T00:08:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:24 smithi122 ceph-mon[93869]: pgmap v18119: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 123 KiB/s wr, 7 op/s 2023-03-10T00:08:24.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:24 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33013.880566 secs 2023-03-10T00:08:24.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:24 smithi181 ceph-mon[94189]: pgmap v18119: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 123 KiB/s wr, 7 op/s 2023-03-10T00:08:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:24 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33013.880566 secs 2023-03-10T00:08:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:24 smithi038 ceph-mon[92422]: pgmap v18119: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 123 KiB/s wr, 7 op/s 2023-03-10T00:08:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33016.452579 secs 2023-03-10T00:08:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:26 smithi122 ceph-mon[93869]: pgmap v18120: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 123 KiB/s wr, 7 op/s 2023-03-10T00:08:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33016.452579 secs 2023-03-10T00:08:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:26 smithi181 ceph-mon[94189]: pgmap v18120: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 123 KiB/s wr, 7 op/s 2023-03-10T00:08:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33016.452579 secs 2023-03-10T00:08:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:26 smithi038 ceph-mon[92422]: pgmap v18120: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 123 KiB/s wr, 7 op/s 2023-03-10T00:08:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:28 smithi122 ceph-mon[93869]: pgmap v18121: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-10T00:08:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33018.880627 secs 2023-03-10T00:08:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:28 smithi181 ceph-mon[94189]: pgmap v18121: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-10T00:08:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33018.880627 secs 2023-03-10T00:08:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:28 smithi038 ceph-mon[92422]: pgmap v18121: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-10T00:08:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33018.880627 secs 2023-03-10T00:08:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33480.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:08:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:30 smithi122 ceph-mon[93869]: pgmap v18122: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-10T00:08:30.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:30 smithi181 ceph-mon[94189]: pgmap v18122: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-10T00:08:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:30 smithi038 ceph-mon[92422]: pgmap v18122: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-10T00:08:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33021.452634 secs 2023-03-10T00:08:31.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33021.452634 secs 2023-03-10T00:08:31.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33021.452634 secs 2023-03-10T00:08:32.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:32 smithi122 ceph-mon[93869]: pgmap v18123: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-10T00:08:32.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:32 smithi181 ceph-mon[94189]: pgmap v18123: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-10T00:08:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:32 smithi038 ceph-mon[92422]: pgmap v18123: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-10T00:08:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33485.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:08:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:34 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33023.880740 secs 2023-03-10T00:08:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:34 smithi122 ceph-mon[93869]: pgmap v18124: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-10T00:08:34.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:34 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33023.880740 secs 2023-03-10T00:08:34.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:34 smithi181 ceph-mon[94189]: pgmap v18124: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-10T00:08:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:34 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33023.880740 secs 2023-03-10T00:08:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:34 smithi038 ceph-mon[92422]: pgmap v18124: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-10T00:08:36.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33026.452692 secs 2023-03-10T00:08:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:36 smithi122 ceph-mon[93869]: pgmap v18125: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-10T00:08:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33026.452692 secs 2023-03-10T00:08:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:36 smithi181 ceph-mon[94189]: pgmap v18125: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-10T00:08:36.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33026.452692 secs 2023-03-10T00:08:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:36 smithi038 ceph-mon[92422]: pgmap v18125: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-10T00:08:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:08:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:08:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:08:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:08:37.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:08:37.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:08:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:08:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:08:37.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:08:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:08:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:08:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:08:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:38 smithi122 ceph-mon[93869]: pgmap v18126: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 426 B/s wr, 7 op/s 2023-03-10T00:08:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33028.880799 secs 2023-03-10T00:08:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:38 smithi181 ceph-mon[94189]: pgmap v18126: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 426 B/s wr, 7 op/s 2023-03-10T00:08:38.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33028.880799 secs 2023-03-10T00:08:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:38 smithi038 ceph-mon[92422]: pgmap v18126: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 426 B/s wr, 7 op/s 2023-03-10T00:08:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33028.880799 secs 2023-03-10T00:08:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33490.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:08:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:40 smithi122 ceph-mon[93869]: pgmap v18127: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2023-03-10T00:08:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:40 smithi181 ceph-mon[94189]: pgmap v18127: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2023-03-10T00:08:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:40 smithi038 ceph-mon[92422]: pgmap v18127: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2023-03-10T00:08:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33031.452780 secs 2023-03-10T00:08:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33031.452780 secs 2023-03-10T00:08:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33031.452780 secs 2023-03-10T00:08:42.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:42 smithi122 ceph-mon[93869]: pgmap v18128: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 42 KiB/s wr, 7 op/s 2023-03-10T00:08:42.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:42 smithi181 ceph-mon[94189]: pgmap v18128: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 42 KiB/s wr, 7 op/s 2023-03-10T00:08:42.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:42 smithi038 ceph-mon[92422]: pgmap v18128: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 42 KiB/s wr, 7 op/s 2023-03-10T00:08:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33495.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:08:44.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:44 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33033.880880 secs 2023-03-10T00:08:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:44 smithi122 ceph-mon[93869]: pgmap v18129: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 129 KiB/s wr, 7 op/s 2023-03-10T00:08:44.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:44 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33033.880880 secs 2023-03-10T00:08:44.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:44 smithi181 ceph-mon[94189]: pgmap v18129: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 129 KiB/s wr, 7 op/s 2023-03-10T00:08:44.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:44 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33033.880880 secs 2023-03-10T00:08:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:44 smithi038 ceph-mon[92422]: pgmap v18129: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 129 KiB/s wr, 7 op/s 2023-03-10T00:08:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33036.452835 secs 2023-03-10T00:08:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:46 smithi122 ceph-mon[93869]: pgmap v18130: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 129 KiB/s wr, 6 op/s 2023-03-10T00:08:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33036.452835 secs 2023-03-10T00:08:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:46 smithi181 ceph-mon[94189]: pgmap v18130: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 129 KiB/s wr, 6 op/s 2023-03-10T00:08:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33036.452835 secs 2023-03-10T00:08:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:46 smithi038 ceph-mon[92422]: pgmap v18130: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 129 KiB/s wr, 6 op/s 2023-03-10T00:08:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:48 smithi122 ceph-mon[93869]: pgmap v18131: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 256 KiB/s wr, 9 op/s 2023-03-10T00:08:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33038.880938 secs 2023-03-10T00:08:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:48 smithi181 ceph-mon[94189]: pgmap v18131: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 256 KiB/s wr, 9 op/s 2023-03-10T00:08:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33038.880938 secs 2023-03-10T00:08:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:48 smithi038 ceph-mon[92422]: pgmap v18131: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 256 KiB/s wr, 9 op/s 2023-03-10T00:08:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33038.880938 secs 2023-03-10T00:08:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33500.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:08:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:08:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:08:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:49 smithi122 ceph-mon[93869]: pgmap v18132: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 256 KiB/s wr, 7 op/s 2023-03-10T00:08:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:08:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:08:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:49 smithi181 ceph-mon[94189]: pgmap v18132: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 256 KiB/s wr, 7 op/s 2023-03-10T00:08:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:08:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:08:49.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:49 smithi038 ceph-mon[92422]: pgmap v18132: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 256 KiB/s wr, 7 op/s 2023-03-10T00:08:50.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33041.452918 secs 2023-03-10T00:08:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33041.452918 secs 2023-03-10T00:08:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33041.452918 secs 2023-03-10T00:08:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:08:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:08:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:51 smithi122 ceph-mon[93869]: pgmap v18133: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 764 KiB/s rd, 759 KiB/s wr, 11 op/s 2023-03-10T00:08:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:08:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:08:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:51 smithi181 ceph-mon[94189]: pgmap v18133: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 764 KiB/s rd, 759 KiB/s wr, 11 op/s 2023-03-10T00:08:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:08:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:08:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:51 smithi038 ceph-mon[92422]: pgmap v18133: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 764 KiB/s rd, 759 KiB/s wr, 11 op/s 2023-03-10T00:08:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33505.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:08:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33043.881000 secs 2023-03-10T00:08:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:08:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:08:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:53 smithi122 ceph-mon[93869]: pgmap v18134: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 904 KiB/s rd, 898 KiB/s wr, 10 op/s 2023-03-10T00:08:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33043.881000 secs 2023-03-10T00:08:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:08:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:08:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:53 smithi181 ceph-mon[94189]: pgmap v18134: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 904 KiB/s rd, 898 KiB/s wr, 10 op/s 2023-03-10T00:08:53.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33043.881000 secs 2023-03-10T00:08:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:08:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:08:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:53 smithi038 ceph-mon[92422]: pgmap v18134: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 904 KiB/s rd, 898 KiB/s wr, 10 op/s 2023-03-10T00:08:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33046.452980 secs 2023-03-10T00:08:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:08:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:08:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:56 smithi122 ceph-mon[93869]: pgmap v18135: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 828 KiB/s rd, 810 KiB/s wr, 10 op/s 2023-03-10T00:08:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33046.452980 secs 2023-03-10T00:08:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:08:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:08:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:56 smithi181 ceph-mon[94189]: pgmap v18135: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 828 KiB/s rd, 810 KiB/s wr, 10 op/s 2023-03-10T00:08:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33046.452980 secs 2023-03-10T00:08:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:08:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:08:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:56 smithi038 ceph-mon[92422]: pgmap v18135: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 828 KiB/s rd, 810 KiB/s wr, 10 op/s 2023-03-10T00:08:58.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:58 smithi122 ceph-mon[93869]: pgmap v18136: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1018 KiB/s wr, 14 op/s 2023-03-10T00:08:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:58 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:08:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:58 smithi181 ceph-mon[94189]: pgmap v18136: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1018 KiB/s wr, 14 op/s 2023-03-10T00:08:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:58 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:08:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:58 smithi038 ceph-mon[92422]: pgmap v18136: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1018 KiB/s wr, 14 op/s 2023-03-10T00:08:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:58 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:08:58.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33510.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:08:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:59 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:08:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:08:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:08:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:08:59.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:59 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:08:59.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:08:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:08:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:08:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:59 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:08:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:08:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:08:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:09:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:00 smithi122 ceph-mon[93869]: pgmap v18137: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 936 KiB/s rd, 892 KiB/s wr, 12 op/s 2023-03-10T00:09:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:00 smithi181 ceph-mon[94189]: pgmap v18137: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 936 KiB/s rd, 892 KiB/s wr, 12 op/s 2023-03-10T00:09:00.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:00 smithi038 ceph-mon[92422]: pgmap v18137: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 936 KiB/s rd, 892 KiB/s wr, 12 op/s 2023-03-10T00:09:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33051.453038 secs 2023-03-10T00:09:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:09:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:09:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:01 smithi122 ceph-mon[93869]: pgmap v18138: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 20 op/s 2023-03-10T00:09:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33051.453038 secs 2023-03-10T00:09:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:09:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:09:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:01 smithi181 ceph-mon[94189]: pgmap v18138: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 20 op/s 2023-03-10T00:09:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33051.453038 secs 2023-03-10T00:09:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:09:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:09:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:01 smithi038 ceph-mon[92422]: pgmap v18138: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 20 op/s 2023-03-10T00:09:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33515.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:09:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:04 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33053.881183 secs 2023-03-10T00:09:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:04 smithi122 ceph-mon[93869]: pgmap v18139: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 952 KiB/s rd, 892 KiB/s wr, 29 op/s 2023-03-10T00:09:04.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:04 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33053.881183 secs 2023-03-10T00:09:04.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:04 smithi181 ceph-mon[94189]: pgmap v18139: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 952 KiB/s rd, 892 KiB/s wr, 29 op/s 2023-03-10T00:09:04.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:04 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33053.881183 secs 2023-03-10T00:09:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:04 smithi038 ceph-mon[92422]: pgmap v18139: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 952 KiB/s rd, 892 KiB/s wr, 29 op/s 2023-03-10T00:09:06.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33056.453096 secs 2023-03-10T00:09:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:06 smithi122 ceph-mon[93869]: pgmap v18140: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 769 KiB/s rd, 711 KiB/s wr, 27 op/s 2023-03-10T00:09:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33056.453096 secs 2023-03-10T00:09:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:06 smithi181 ceph-mon[94189]: pgmap v18140: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 769 KiB/s rd, 711 KiB/s wr, 27 op/s 2023-03-10T00:09:06.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33056.453096 secs 2023-03-10T00:09:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:06 smithi038 ceph-mon[92422]: pgmap v18140: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 769 KiB/s rd, 711 KiB/s wr, 27 op/s 2023-03-10T00:09:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:07 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:09:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:09:07.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:09:07.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:07 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:09:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:09:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:09:07.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:07 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:09:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:09:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:09:08.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:08 smithi122 ceph-mon[93869]: pgmap v18141: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 760 KiB/s rd, 712 KiB/s wr, 31 op/s 2023-03-10T00:09:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:09:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:09:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33058.881241 secs 2023-03-10T00:09:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:08 smithi181 ceph-mon[94189]: pgmap v18141: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 760 KiB/s rd, 712 KiB/s wr, 31 op/s 2023-03-10T00:09:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:09:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:09:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33058.881241 secs 2023-03-10T00:09:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:08 smithi038 ceph-mon[92422]: pgmap v18141: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 760 KiB/s rd, 712 KiB/s wr, 31 op/s 2023-03-10T00:09:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:09:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:09:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33058.881241 secs 2023-03-10T00:09:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33520.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:09:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:09:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:09:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:09 smithi122 ceph-mon[93869]: pgmap v18142: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 524 KiB/s rd, 503 KiB/s wr, 26 op/s 2023-03-10T00:09:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:09:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:09:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:09 smithi181 ceph-mon[94189]: pgmap v18142: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 524 KiB/s rd, 503 KiB/s wr, 26 op/s 2023-03-10T00:09:09.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:09:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:09:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:09 smithi038 ceph-mon[92422]: pgmap v18142: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 524 KiB/s rd, 503 KiB/s wr, 26 op/s 2023-03-10T00:09:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:10 smithi122 ceph-mon[93869]: osdmap e1349: 12 total, 12 up, 12 in 2023-03-10T00:09:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:10 smithi181 ceph-mon[94189]: osdmap e1349: 12 total, 12 up, 12 in 2023-03-10T00:09:10.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:10 smithi038 ceph-mon[92422]: osdmap e1349: 12 total, 12 up, 12 in 2023-03-10T00:09:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33061.453153 secs 2023-03-10T00:09:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:11 smithi122 ceph-mon[93869]: pgmap v18144: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 348 KiB/s rd, 362 KiB/s wr, 26 op/s 2023-03-10T00:09:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33061.453153 secs 2023-03-10T00:09:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:11 smithi181 ceph-mon[94189]: pgmap v18144: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 348 KiB/s rd, 362 KiB/s wr, 26 op/s 2023-03-10T00:09:11.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33061.453153 secs 2023-03-10T00:09:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:11 smithi038 ceph-mon[92422]: pgmap v18144: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 348 KiB/s rd, 362 KiB/s wr, 26 op/s 2023-03-10T00:09:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33525.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:09:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:14 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33063.881325 secs 2023-03-10T00:09:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:14 smithi122 ceph-mon[93869]: pgmap v18145: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 336 KiB/s wr, 10 op/s 2023-03-10T00:09:14.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:14 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33063.881325 secs 2023-03-10T00:09:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:14 smithi181 ceph-mon[94189]: pgmap v18145: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 336 KiB/s wr, 10 op/s 2023-03-10T00:09:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:14 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33063.881325 secs 2023-03-10T00:09:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:14 smithi038 ceph-mon[92422]: pgmap v18145: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 336 KiB/s wr, 10 op/s 2023-03-10T00:09:16.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33066.453212 secs 2023-03-10T00:09:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:16 smithi122 ceph-mon[93869]: pgmap v18146: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 336 KiB/s wr, 10 op/s 2023-03-10T00:09:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33066.453212 secs 2023-03-10T00:09:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:16 smithi181 ceph-mon[94189]: pgmap v18146: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 336 KiB/s wr, 10 op/s 2023-03-10T00:09:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33066.453212 secs 2023-03-10T00:09:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:16 smithi038 ceph-mon[92422]: pgmap v18146: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 336 KiB/s wr, 10 op/s 2023-03-10T00:09:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:09:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:09:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:09:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:09:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:09:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:09:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:18 smithi122 ceph-mon[93869]: pgmap v18147: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 462 KiB/s wr, 9 op/s 2023-03-10T00:09:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33068.881383 secs 2023-03-10T00:09:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:18 smithi122 ceph-mon[93869]: osdmap e1350: 12 total, 12 up, 12 in 2023-03-10T00:09:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:18 smithi181 ceph-mon[94189]: pgmap v18147: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 462 KiB/s wr, 9 op/s 2023-03-10T00:09:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33068.881383 secs 2023-03-10T00:09:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:18 smithi181 ceph-mon[94189]: osdmap e1350: 12 total, 12 up, 12 in 2023-03-10T00:09:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:18 smithi038 ceph-mon[92422]: pgmap v18147: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 462 KiB/s wr, 9 op/s 2023-03-10T00:09:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33068.881383 secs 2023-03-10T00:09:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:18 smithi038 ceph-mon[92422]: osdmap e1350: 12 total, 12 up, 12 in 2023-03-10T00:09:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33530.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:09:19.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:19 smithi122 ceph-mon[93869]: pgmap v18149: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 466 KiB/s wr, 9 op/s 2023-03-10T00:09:19.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:19 smithi181 ceph-mon[94189]: pgmap v18149: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 466 KiB/s wr, 9 op/s 2023-03-10T00:09:19.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:19 smithi038 ceph-mon[92422]: pgmap v18149: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 466 KiB/s wr, 9 op/s 2023-03-10T00:09:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:09:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:09:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:09:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:09:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:09:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:09:21.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33071.453297 secs 2023-03-10T00:09:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:21 smithi122 ceph-mon[93869]: pgmap v18150: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 127 KiB/s wr, 8 op/s 2023-03-10T00:09:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33071.453297 secs 2023-03-10T00:09:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:21 smithi181 ceph-mon[94189]: pgmap v18150: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 127 KiB/s wr, 8 op/s 2023-03-10T00:09:21.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33071.453297 secs 2023-03-10T00:09:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:21 smithi038 ceph-mon[92422]: pgmap v18150: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 127 KiB/s wr, 8 op/s 2023-03-10T00:09:23.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33535.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:09:24.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:24 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33073.881458 secs 2023-03-10T00:09:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:24 smithi122 ceph-mon[93869]: pgmap v18151: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 127 KiB/s wr, 7 op/s 2023-03-10T00:09:24.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:24 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33073.881458 secs 2023-03-10T00:09:24.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:24 smithi181 ceph-mon[94189]: pgmap v18151: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 127 KiB/s wr, 7 op/s 2023-03-10T00:09:24.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:24 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33073.881458 secs 2023-03-10T00:09:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:24 smithi038 ceph-mon[92422]: pgmap v18151: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 127 KiB/s wr, 7 op/s 2023-03-10T00:09:26.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33076.453381 secs 2023-03-10T00:09:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:26 smithi122 ceph-mon[93869]: pgmap v18152: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 127 KiB/s wr, 7 op/s 2023-03-10T00:09:26.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33076.453381 secs 2023-03-10T00:09:26.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:26 smithi181 ceph-mon[94189]: pgmap v18152: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 127 KiB/s wr, 7 op/s 2023-03-10T00:09:26.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33076.453381 secs 2023-03-10T00:09:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:26 smithi038 ceph-mon[92422]: pgmap v18152: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 127 KiB/s wr, 7 op/s 2023-03-10T00:09:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:27 smithi122 ceph-mon[93869]: pgmap v18153: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-10T00:09:27.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:27 smithi181 ceph-mon[94189]: pgmap v18153: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-10T00:09:27.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:27 smithi038 ceph-mon[92422]: pgmap v18153: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-10T00:09:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33540.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:09:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:29 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33078.881515 secs 2023-03-10T00:09:29.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:29 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33078.881515 secs 2023-03-10T00:09:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:29 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33078.881515 secs 2023-03-10T00:09:30.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:30 smithi122 ceph-mon[93869]: pgmap v18154: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2023-03-10T00:09:30.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:30 smithi181 ceph-mon[94189]: pgmap v18154: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2023-03-10T00:09:30.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:30 smithi038 ceph-mon[92422]: pgmap v18154: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2023-03-10T00:09:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33081.453437 secs 2023-03-10T00:09:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33081.453437 secs 2023-03-10T00:09:31.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33081.453437 secs 2023-03-10T00:09:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:32 smithi122 ceph-mon[93869]: pgmap v18155: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 124 KiB/s wr, 9 op/s 2023-03-10T00:09:32.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:32 smithi181 ceph-mon[94189]: pgmap v18155: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 124 KiB/s wr, 9 op/s 2023-03-10T00:09:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:32 smithi038 ceph-mon[92422]: pgmap v18155: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 124 KiB/s wr, 9 op/s 2023-03-10T00:09:33.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33545.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:09:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:34 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33083.881600 secs 2023-03-10T00:09:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:34 smithi122 ceph-mon[93869]: pgmap v18156: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 124 KiB/s wr, 6 op/s 2023-03-10T00:09:34.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:34 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33083.881600 secs 2023-03-10T00:09:34.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:34 smithi181 ceph-mon[94189]: pgmap v18156: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 124 KiB/s wr, 6 op/s 2023-03-10T00:09:34.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:34 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33083.881600 secs 2023-03-10T00:09:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:34 smithi038 ceph-mon[92422]: pgmap v18156: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 124 KiB/s wr, 6 op/s 2023-03-10T00:09:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33086.453521 secs 2023-03-10T00:09:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:36 smithi122 ceph-mon[93869]: pgmap v18157: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 124 KiB/s wr, 6 op/s 2023-03-10T00:09:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33086.453521 secs 2023-03-10T00:09:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:36 smithi181 ceph-mon[94189]: pgmap v18157: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 124 KiB/s wr, 6 op/s 2023-03-10T00:09:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33086.453521 secs 2023-03-10T00:09:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:36 smithi038 ceph-mon[92422]: pgmap v18157: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 124 KiB/s wr, 6 op/s 2023-03-10T00:09:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:09:37.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:09:37.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:09:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:38 smithi122 ceph-mon[93869]: pgmap v18158: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 253 KiB/s wr, 10 op/s 2023-03-10T00:09:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33088.881728 secs 2023-03-10T00:09:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:38 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:09:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:38 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:09:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:38 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:09:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:38 smithi181 ceph-mon[94189]: pgmap v18158: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 253 KiB/s wr, 10 op/s 2023-03-10T00:09:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33088.881728 secs 2023-03-10T00:09:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:38 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:09:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:38 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:09:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:38 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:09:38.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:38 smithi038 ceph-mon[92422]: pgmap v18158: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 253 KiB/s wr, 10 op/s 2023-03-10T00:09:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33088.881728 secs 2023-03-10T00:09:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:38 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:09:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:38 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:09:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:38 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:09:38.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33550.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:09:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:39 smithi122 ceph-mon[93869]: pgmap v18159: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 253 KiB/s wr, 7 op/s 2023-03-10T00:09:39.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:39 smithi181 ceph-mon[94189]: pgmap v18159: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 253 KiB/s wr, 7 op/s 2023-03-10T00:09:39.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:39 smithi038 ceph-mon[92422]: pgmap v18159: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 253 KiB/s wr, 7 op/s 2023-03-10T00:09:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33091.453608 secs 2023-03-10T00:09:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33091.453608 secs 2023-03-10T00:09:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33091.453608 secs 2023-03-10T00:09:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:41 smithi122 ceph-mon[93869]: pgmap v18160: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 253 KiB/s wr, 10 op/s 2023-03-10T00:09:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:41 smithi181 ceph-mon[94189]: pgmap v18160: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 253 KiB/s wr, 10 op/s 2023-03-10T00:09:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:41 smithi038 ceph-mon[92422]: pgmap v18160: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 253 KiB/s wr, 10 op/s 2023-03-10T00:09:43.793 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33555.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:09:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:44 smithi122 ceph-mon[93869]: pgmap v18161: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 129 KiB/s wr, 7 op/s 2023-03-10T00:09:44.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:44 smithi181 ceph-mon[94189]: pgmap v18161: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 129 KiB/s wr, 7 op/s 2023-03-10T00:09:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:44 smithi038 ceph-mon[92422]: pgmap v18161: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 129 KiB/s wr, 7 op/s 2023-03-10T00:09:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33096.453666 secs 2023-03-10T00:09:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:46 smithi122 ceph-mon[93869]: pgmap v18162: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 129 KiB/s wr, 6 op/s 2023-03-10T00:09:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:46 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:09:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33096.453666 secs 2023-03-10T00:09:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:46 smithi181 ceph-mon[94189]: pgmap v18162: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 129 KiB/s wr, 6 op/s 2023-03-10T00:09:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:46 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:09:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33096.453666 secs 2023-03-10T00:09:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:46 smithi038 ceph-mon[92422]: pgmap v18162: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 129 KiB/s wr, 6 op/s 2023-03-10T00:09:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:46 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:09:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:47 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:09:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:09:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:09:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:47 smithi122 ceph-mon[93869]: pgmap v18163: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 256 KiB/s wr, 9 op/s 2023-03-10T00:09:47.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:47 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:09:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:09:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:09:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:47 smithi181 ceph-mon[94189]: pgmap v18163: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 256 KiB/s wr, 9 op/s 2023-03-10T00:09:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:47 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:09:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:09:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:09:47.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:47 smithi038 ceph-mon[92422]: pgmap v18163: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 256 KiB/s wr, 9 op/s 2023-03-10T00:09:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33560.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:09:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:09:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:09:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:49 smithi122 ceph-mon[93869]: pgmap v18164: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 127 KiB/s wr, 6 op/s 2023-03-10T00:09:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:09:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:09:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:49 smithi181 ceph-mon[94189]: pgmap v18164: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 127 KiB/s wr, 6 op/s 2023-03-10T00:09:49.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:09:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:09:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:49 smithi038 ceph-mon[92422]: pgmap v18164: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 127 KiB/s wr, 6 op/s 2023-03-10T00:09:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33101.453723 secs 2023-03-10T00:09:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33101.453723 secs 2023-03-10T00:09:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33101.453723 secs 2023-03-10T00:09:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:09:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:09:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:51 smithi122 ceph-mon[93869]: pgmap v18165: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 840 KiB/s rd, 835 KiB/s wr, 11 op/s 2023-03-10T00:09:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:09:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:09:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:51 smithi181 ceph-mon[94189]: pgmap v18165: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 840 KiB/s rd, 835 KiB/s wr, 11 op/s 2023-03-10T00:09:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:09:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:09:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:51 smithi038 ceph-mon[92422]: pgmap v18165: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 840 KiB/s rd, 835 KiB/s wr, 11 op/s 2023-03-10T00:09:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33565.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:09:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:09:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:09:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:53 smithi122 ceph-mon[93869]: pgmap v18166: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 838 KiB/s rd, 835 KiB/s wr, 9 op/s 2023-03-10T00:09:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:09:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:09:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:53 smithi181 ceph-mon[94189]: pgmap v18166: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 838 KiB/s rd, 835 KiB/s wr, 9 op/s 2023-03-10T00:09:53.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:09:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:09:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:53 smithi038 ceph-mon[92422]: pgmap v18166: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 838 KiB/s rd, 835 KiB/s wr, 9 op/s 2023-03-10T00:09:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33106.453779 secs 2023-03-10T00:09:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:09:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:09:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:56 smithi122 ceph-mon[93869]: pgmap v18167: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 850 KiB/s rd, 836 KiB/s wr, 9 op/s 2023-03-10T00:09:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33106.453779 secs 2023-03-10T00:09:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:09:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:09:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:56 smithi181 ceph-mon[94189]: pgmap v18167: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 850 KiB/s rd, 836 KiB/s wr, 9 op/s 2023-03-10T00:09:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33106.453779 secs 2023-03-10T00:09:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:09:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:09:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:56 smithi038 ceph-mon[92422]: pgmap v18167: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 850 KiB/s rd, 836 KiB/s wr, 9 op/s 2023-03-10T00:09:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:09:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:09:57.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:09:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:09:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:09:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:09:58.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:58 smithi122 ceph-mon[93869]: pgmap v18168: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 13 op/s 2023-03-10T00:09:58.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:58 smithi181 ceph-mon[94189]: pgmap v18168: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 13 op/s 2023-03-10T00:09:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:58 smithi038 ceph-mon[92422]: pgmap v18168: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 13 op/s 2023-03-10T00:09:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33570.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:09:59.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:59 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 33108.882066 secs 2023-03-10T00:09:59.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:59 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:09:59.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:09:59.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:09:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:09:59.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:59 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 33108.882066 secs 2023-03-10T00:09:59.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:59 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:09:59.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:09:59.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:09:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:09:59.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:59 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 33108.882066 secs 2023-03-10T00:09:59.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:59 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:09:59.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:09:59.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:09:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:10:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:10:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:10:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:00 smithi122 ceph-mon[93869]: pgmap v18169: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 11 op/s 2023-03-10T00:10:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:00 smithi122 ceph-mon[93869]: overall HEALTH_WARN 2 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-10T00:10:00.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:10:00.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:10:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:00 smithi181 ceph-mon[94189]: pgmap v18169: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 11 op/s 2023-03-10T00:10:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:00 smithi181 ceph-mon[94189]: overall HEALTH_WARN 2 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-10T00:10:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:10:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:10:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:00 smithi038 ceph-mon[92422]: pgmap v18169: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 11 op/s 2023-03-10T00:10:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:00 smithi038 ceph-mon[92422]: overall HEALTH_WARN 2 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-10T00:10:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33111.453839 secs 2023-03-10T00:10:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:10:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:10:01.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33111.453839 secs 2023-03-10T00:10:01.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:10:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:10:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33111.453839 secs 2023-03-10T00:10:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:10:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:10:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:02 smithi122 ceph-mon[93869]: pgmap v18170: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.2 MiB/s wr, 17 op/s 2023-03-10T00:10:02.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:02 smithi181 ceph-mon[94189]: pgmap v18170: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.2 MiB/s wr, 17 op/s 2023-03-10T00:10:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:02 smithi038 ceph-mon[92422]: pgmap v18170: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.2 MiB/s wr, 17 op/s 2023-03-10T00:10:03.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33575.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:10:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:04 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33113.882150 secs 2023-03-10T00:10:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:04 smithi122 ceph-mon[93869]: pgmap v18171: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 966 KiB/s rd, 692 KiB/s wr, 26 op/s 2023-03-10T00:10:04.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:04 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33113.882150 secs 2023-03-10T00:10:04.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:04 smithi181 ceph-mon[94189]: pgmap v18171: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 966 KiB/s rd, 692 KiB/s wr, 26 op/s 2023-03-10T00:10:04.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:04 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33113.882150 secs 2023-03-10T00:10:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:04 smithi038 ceph-mon[92422]: pgmap v18171: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 966 KiB/s rd, 692 KiB/s wr, 26 op/s 2023-03-10T00:10:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33116.453925 secs 2023-03-10T00:10:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:10:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:06 smithi122 ceph-mon[93869]: pgmap v18172: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 966 KiB/s rd, 692 KiB/s wr, 25 op/s 2023-03-10T00:10:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33116.453925 secs 2023-03-10T00:10:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:10:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:06 smithi181 ceph-mon[94189]: pgmap v18172: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 966 KiB/s rd, 692 KiB/s wr, 25 op/s 2023-03-10T00:10:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33116.453925 secs 2023-03-10T00:10:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:10:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:06 smithi038 ceph-mon[92422]: pgmap v18172: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 966 KiB/s rd, 692 KiB/s wr, 25 op/s 2023-03-10T00:10:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:10:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:10:07.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:10:07.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:10:07.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:10:07.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:10:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:08 smithi122 ceph-mon[93869]: pgmap v18173: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 956 KiB/s rd, 691 KiB/s wr, 28 op/s 2023-03-10T00:10:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:08 smithi181 ceph-mon[94189]: pgmap v18173: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 956 KiB/s rd, 691 KiB/s wr, 28 op/s 2023-03-10T00:10:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:08 smithi038 ceph-mon[92422]: pgmap v18173: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 956 KiB/s rd, 691 KiB/s wr, 28 op/s 2023-03-10T00:10:08.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33580.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:10:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:09 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33118.882261 secs 2023-03-10T00:10:09.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:09 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33118.882261 secs 2023-03-10T00:10:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:09 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33118.882261 secs 2023-03-10T00:10:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:10 smithi122 ceph-mon[93869]: pgmap v18174: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 853 KiB/s rd, 589 KiB/s wr, 25 op/s 2023-03-10T00:10:10.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:10 smithi181 ceph-mon[94189]: pgmap v18174: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 853 KiB/s rd, 589 KiB/s wr, 25 op/s 2023-03-10T00:10:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:10 smithi038 ceph-mon[92422]: pgmap v18174: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 853 KiB/s rd, 589 KiB/s wr, 25 op/s 2023-03-10T00:10:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33121.453984 secs 2023-03-10T00:10:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:11 smithi122 ceph-mon[93869]: osdmap e1351: 12 total, 12 up, 12 in 2023-03-10T00:10:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33121.453984 secs 2023-03-10T00:10:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:11 smithi181 ceph-mon[94189]: osdmap e1351: 12 total, 12 up, 12 in 2023-03-10T00:10:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33121.453984 secs 2023-03-10T00:10:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:11 smithi038 ceph-mon[92422]: osdmap e1351: 12 total, 12 up, 12 in 2023-03-10T00:10:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:10:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:10:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:12 smithi122 ceph-mon[93869]: pgmap v18176: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 394 KiB/s wr, 24 op/s 2023-03-10T00:10:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:10:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:10:12.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:12 smithi181 ceph-mon[94189]: pgmap v18176: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 394 KiB/s wr, 24 op/s 2023-03-10T00:10:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:10:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:10:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:12 smithi038 ceph-mon[92422]: pgmap v18176: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 379 KiB/s rd, 394 KiB/s wr, 24 op/s 2023-03-10T00:10:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:10:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:10:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33123.882319 secs 2023-03-10T00:10:13.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:10:13.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:10:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33123.882319 secs 2023-03-10T00:10:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:10:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:10:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33123.882319 secs 2023-03-10T00:10:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33585.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:10:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:14 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:10:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:14 smithi122 ceph-mon[93869]: pgmap v18177: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 188 KiB/s wr, 8 op/s 2023-03-10T00:10:14.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:14 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:10:14.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:14 smithi181 ceph-mon[94189]: pgmap v18177: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 188 KiB/s wr, 8 op/s 2023-03-10T00:10:14.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:14 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:10:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:14 smithi038 ceph-mon[92422]: pgmap v18177: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 188 KiB/s wr, 8 op/s 2023-03-10T00:10:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:15 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:10:15.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:15 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:10:15.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:15 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:10:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33126.454068 secs 2023-03-10T00:10:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:16 smithi122 ceph-mon[93869]: pgmap v18178: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 337 KiB/s wr, 9 op/s 2023-03-10T00:10:16.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33126.454068 secs 2023-03-10T00:10:16.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:16 smithi181 ceph-mon[94189]: pgmap v18178: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 337 KiB/s wr, 9 op/s 2023-03-10T00:10:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33126.454068 secs 2023-03-10T00:10:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:16 smithi038 ceph-mon[92422]: pgmap v18178: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 337 KiB/s wr, 9 op/s 2023-03-10T00:10:17.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:17 smithi122 ceph-mon[93869]: pgmap v18179: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 375 KiB/s wr, 8 op/s 2023-03-10T00:10:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:17 smithi181 ceph-mon[94189]: pgmap v18179: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 375 KiB/s wr, 8 op/s 2023-03-10T00:10:17.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:17 smithi038 ceph-mon[92422]: pgmap v18179: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 375 KiB/s wr, 8 op/s 2023-03-10T00:10:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33590.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:10:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:19 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33128.882384 secs 2023-03-10T00:10:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:19 smithi122 ceph-mon[93869]: osdmap e1352: 12 total, 12 up, 12 in 2023-03-10T00:10:19.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:19 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33128.882384 secs 2023-03-10T00:10:19.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:19 smithi181 ceph-mon[94189]: osdmap e1352: 12 total, 12 up, 12 in 2023-03-10T00:10:19.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:19 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33128.882384 secs 2023-03-10T00:10:19.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:19 smithi038 ceph-mon[92422]: osdmap e1352: 12 total, 12 up, 12 in 2023-03-10T00:10:20.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:20 smithi122 ceph-mon[93869]: pgmap v18181: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 296 KiB/s wr, 8 op/s 2023-03-10T00:10:20.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:20 smithi181 ceph-mon[94189]: pgmap v18181: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 296 KiB/s wr, 8 op/s 2023-03-10T00:10:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:20 smithi038 ceph-mon[92422]: pgmap v18181: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 296 KiB/s wr, 8 op/s 2023-03-10T00:10:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33131.454128 secs 2023-03-10T00:10:21.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33131.454128 secs 2023-03-10T00:10:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33131.454128 secs 2023-03-10T00:10:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:22 smithi122 ceph-mon[93869]: pgmap v18182: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 272 KiB/s wr, 8 op/s 2023-03-10T00:10:22.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:22 smithi181 ceph-mon[94189]: pgmap v18182: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 272 KiB/s wr, 8 op/s 2023-03-10T00:10:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:22 smithi038 ceph-mon[92422]: pgmap v18182: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 272 KiB/s wr, 8 op/s 2023-03-10T00:10:23.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33595.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:10:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:24 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33133.882443 secs 2023-03-10T00:10:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:24 smithi122 ceph-mon[93869]: pgmap v18183: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 272 KiB/s wr, 7 op/s 2023-03-10T00:10:24.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:24 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33133.882443 secs 2023-03-10T00:10:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:24 smithi181 ceph-mon[94189]: pgmap v18183: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 272 KiB/s wr, 7 op/s 2023-03-10T00:10:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:24 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33133.882443 secs 2023-03-10T00:10:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:24 smithi038 ceph-mon[92422]: pgmap v18183: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 272 KiB/s wr, 7 op/s 2023-03-10T00:10:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:25 smithi122 ceph-mon[93869]: pgmap v18184: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 123 KiB/s wr, 7 op/s 2023-03-10T00:10:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:25 smithi181 ceph-mon[94189]: pgmap v18184: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 123 KiB/s wr, 7 op/s 2023-03-10T00:10:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:25 smithi038 ceph-mon[92422]: pgmap v18184: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 123 KiB/s wr, 7 op/s 2023-03-10T00:10:26.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33136.454208 secs 2023-03-10T00:10:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33136.454208 secs 2023-03-10T00:10:26.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33136.454208 secs 2023-03-10T00:10:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:27 smithi122 ceph-mon[93869]: pgmap v18185: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 84 KiB/s wr, 7 op/s 2023-03-10T00:10:27.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:27 smithi181 ceph-mon[94189]: pgmap v18185: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 84 KiB/s wr, 7 op/s 2023-03-10T00:10:27.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:27 smithi038 ceph-mon[92422]: pgmap v18185: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 84 KiB/s wr, 7 op/s 2023-03-10T00:10:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33600.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:10:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:29 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33138.882505 secs 2023-03-10T00:10:29.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:29 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33138.882505 secs 2023-03-10T00:10:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:29 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33138.882505 secs 2023-03-10T00:10:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:30 smithi122 ceph-mon[93869]: pgmap v18186: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 75 KiB/s wr, 6 op/s 2023-03-10T00:10:30.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:30 smithi181 ceph-mon[94189]: pgmap v18186: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 75 KiB/s wr, 6 op/s 2023-03-10T00:10:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:30 smithi038 ceph-mon[92422]: pgmap v18186: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 75 KiB/s wr, 6 op/s 2023-03-10T00:10:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33141.454312 secs 2023-03-10T00:10:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:31 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:10:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33141.454312 secs 2023-03-10T00:10:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:31 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:10:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33141.454312 secs 2023-03-10T00:10:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:31 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:10:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:32 smithi122 ceph-mon[93869]: pgmap v18187: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-10T00:10:32.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:32 smithi181 ceph-mon[94189]: pgmap v18187: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-10T00:10:32.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:32 smithi038 ceph-mon[92422]: pgmap v18187: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-10T00:10:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33605.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:10:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:34 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33143.882630 secs 2023-03-10T00:10:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:34 smithi122 ceph-mon[93869]: pgmap v18188: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-10T00:10:34.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:34 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33143.882630 secs 2023-03-10T00:10:34.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:34 smithi181 ceph-mon[94189]: pgmap v18188: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-10T00:10:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:34 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33143.882630 secs 2023-03-10T00:10:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:34 smithi038 ceph-mon[92422]: pgmap v18188: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-10T00:10:35.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:35 smithi122 ceph-mon[93869]: pgmap v18189: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-10T00:10:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:35 smithi181 ceph-mon[94189]: pgmap v18189: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-10T00:10:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:35 smithi038 ceph-mon[92422]: pgmap v18189: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-10T00:10:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33146.454427 secs 2023-03-10T00:10:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33146.454427 secs 2023-03-10T00:10:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33146.454427 secs 2023-03-10T00:10:37.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:37 smithi122 ceph-mon[93869]: pgmap v18190: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 426 B/s wr, 8 op/s 2023-03-10T00:10:37.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:37 smithi181 ceph-mon[94189]: pgmap v18190: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 426 B/s wr, 8 op/s 2023-03-10T00:10:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:37 smithi038 ceph-mon[92422]: pgmap v18190: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 426 B/s wr, 8 op/s 2023-03-10T00:10:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33148.882692 secs 2023-03-10T00:10:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:38 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:10:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33148.882692 secs 2023-03-10T00:10:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:38 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:10:38.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33148.882692 secs 2023-03-10T00:10:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:38 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:10:38.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33610.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:10:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:10:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:10:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:10:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:39 smithi122 ceph-mon[93869]: pgmap v18191: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-10T00:10:39.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:10:39.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:10:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:10:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:39 smithi181 ceph-mon[94189]: pgmap v18191: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-10T00:10:39.754 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:10:39.754 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:10:39.754 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:10:39.754 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:39 smithi038 ceph-mon[92422]: pgmap v18191: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-10T00:10:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:10:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:10:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:10:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:10:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:10:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:10:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33151.454514 secs 2023-03-10T00:10:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:10:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:10:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:41 smithi122 ceph-mon[93869]: pgmap v18192: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 130 KiB/s wr, 9 op/s 2023-03-10T00:10:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33151.454514 secs 2023-03-10T00:10:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:10:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:10:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:41 smithi181 ceph-mon[94189]: pgmap v18192: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 130 KiB/s wr, 9 op/s 2023-03-10T00:10:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33151.454514 secs 2023-03-10T00:10:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:10:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:10:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:41 smithi038 ceph-mon[92422]: pgmap v18192: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 130 KiB/s wr, 9 op/s 2023-03-10T00:10:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33615.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:10:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:44 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33153.882790 secs 2023-03-10T00:10:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:44 smithi122 ceph-mon[93869]: pgmap v18193: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 130 KiB/s wr, 7 op/s 2023-03-10T00:10:44.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:44 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33153.882790 secs 2023-03-10T00:10:44.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:44 smithi181 ceph-mon[94189]: pgmap v18193: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 130 KiB/s wr, 7 op/s 2023-03-10T00:10:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:44 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33153.882790 secs 2023-03-10T00:10:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:44 smithi038 ceph-mon[92422]: pgmap v18193: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 130 KiB/s wr, 7 op/s 2023-03-10T00:10:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:45 smithi122 ceph-mon[93869]: pgmap v18194: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 130 KiB/s wr, 6 op/s 2023-03-10T00:10:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:45 smithi181 ceph-mon[94189]: pgmap v18194: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 130 KiB/s wr, 6 op/s 2023-03-10T00:10:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:45 smithi038 ceph-mon[92422]: pgmap v18194: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 130 KiB/s wr, 6 op/s 2023-03-10T00:10:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33156.454601 secs 2023-03-10T00:10:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33156.454601 secs 2023-03-10T00:10:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33156.454601 secs 2023-03-10T00:10:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:47 smithi122 ceph-mon[93869]: pgmap v18195: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 310 KiB/s wr, 9 op/s 2023-03-10T00:10:47.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:47 smithi181 ceph-mon[94189]: pgmap v18195: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 310 KiB/s wr, 9 op/s 2023-03-10T00:10:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:47 smithi038 ceph-mon[92422]: pgmap v18195: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 310 KiB/s wr, 9 op/s 2023-03-10T00:10:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33620.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:10:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:49 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33158.882849 secs 2023-03-10T00:10:49.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:49 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33158.882849 secs 2023-03-10T00:10:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:49 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33158.882849 secs 2023-03-10T00:10:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:50 smithi122 ceph-mon[93869]: pgmap v18196: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 310 KiB/s wr, 6 op/s 2023-03-10T00:10:50.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:50 smithi181 ceph-mon[94189]: pgmap v18196: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 310 KiB/s wr, 6 op/s 2023-03-10T00:10:50.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:50 smithi038 ceph-mon[92422]: pgmap v18196: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 310 KiB/s wr, 6 op/s 2023-03-10T00:10:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33161.454661 secs 2023-03-10T00:10:51.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33161.454661 secs 2023-03-10T00:10:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33161.454661 secs 2023-03-10T00:10:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:52 smithi122 ceph-mon[93869]: pgmap v18197: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 948 KiB/s rd, 679 KiB/s wr, 10 op/s 2023-03-10T00:10:52.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:52 smithi181 ceph-mon[94189]: pgmap v18197: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 948 KiB/s rd, 679 KiB/s wr, 10 op/s 2023-03-10T00:10:52.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:52 smithi038 ceph-mon[92422]: pgmap v18197: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 948 KiB/s rd, 679 KiB/s wr, 10 op/s 2023-03-10T00:10:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:10:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:10:53.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:10:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:10:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:10:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:10:53.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33625.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:10:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:54 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33163.882939 secs 2023-03-10T00:10:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:54 smithi122 ceph-mon[93869]: pgmap v18198: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 817 KiB/s rd, 550 KiB/s wr, 8 op/s 2023-03-10T00:10:54.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:54 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33163.882939 secs 2023-03-10T00:10:54.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:54 smithi181 ceph-mon[94189]: pgmap v18198: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 817 KiB/s rd, 550 KiB/s wr, 8 op/s 2023-03-10T00:10:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:54 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33163.882939 secs 2023-03-10T00:10:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:54 smithi038 ceph-mon[92422]: pgmap v18198: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 817 KiB/s rd, 550 KiB/s wr, 8 op/s 2023-03-10T00:10:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33166.454720 secs 2023-03-10T00:10:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:10:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:10:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:56 smithi122 ceph-mon[93869]: pgmap v18199: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 817 KiB/s rd, 550 KiB/s wr, 8 op/s 2023-03-10T00:10:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33166.454720 secs 2023-03-10T00:10:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:10:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:10:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:56 smithi181 ceph-mon[94189]: pgmap v18199: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 817 KiB/s rd, 550 KiB/s wr, 8 op/s 2023-03-10T00:10:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33166.454720 secs 2023-03-10T00:10:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:10:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:10:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:56 smithi038 ceph-mon[92422]: pgmap v18199: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 817 KiB/s rd, 550 KiB/s wr, 8 op/s 2023-03-10T00:10:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:10:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:10:57.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:10:57.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:10:57.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:10:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:10:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:58 smithi122 ceph-mon[93869]: pgmap v18200: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 767 KiB/s wr, 14 op/s 2023-03-10T00:10:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:10:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33168.883021 secs 2023-03-10T00:10:58.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:58 smithi181 ceph-mon[94189]: pgmap v18200: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 767 KiB/s wr, 14 op/s 2023-03-10T00:10:58.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:10:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33168.883021 secs 2023-03-10T00:10:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:58 smithi038 ceph-mon[92422]: pgmap v18200: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 767 KiB/s wr, 14 op/s 2023-03-10T00:10:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:10:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33168.883021 secs 2023-03-10T00:10:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33630.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:11:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:11:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:11:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:00 smithi122 ceph-mon[93869]: pgmap v18201: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 953 KiB/s rd, 588 KiB/s wr, 11 op/s 2023-03-10T00:11:00.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:11:00.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:11:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:00 smithi181 ceph-mon[94189]: pgmap v18201: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 953 KiB/s rd, 588 KiB/s wr, 11 op/s 2023-03-10T00:11:00.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:11:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:11:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:00 smithi038 ceph-mon[92422]: pgmap v18201: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 953 KiB/s rd, 588 KiB/s wr, 11 op/s 2023-03-10T00:11:01.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33171.454778 secs 2023-03-10T00:11:01.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33171.454778 secs 2023-03-10T00:11:01.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33171.454778 secs 2023-03-10T00:11:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:02 smithi122 ceph-mon[93869]: pgmap v18202: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 777 KiB/s wr, 16 op/s 2023-03-10T00:11:02.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:02 smithi181 ceph-mon[94189]: pgmap v18202: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 777 KiB/s wr, 16 op/s 2023-03-10T00:11:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:02 smithi038 ceph-mon[92422]: pgmap v18202: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 777 KiB/s wr, 16 op/s 2023-03-10T00:11:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33635.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:11:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:04 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33173.883079 secs 2023-03-10T00:11:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:04 smithi122 ceph-mon[93869]: pgmap v18203: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 521 KiB/s rd, 430 KiB/s wr, 24 op/s 2023-03-10T00:11:04.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:04 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33173.883079 secs 2023-03-10T00:11:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:04 smithi181 ceph-mon[94189]: pgmap v18203: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 521 KiB/s rd, 430 KiB/s wr, 24 op/s 2023-03-10T00:11:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:04 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33173.883079 secs 2023-03-10T00:11:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:04 smithi038 ceph-mon[92422]: pgmap v18203: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 521 KiB/s rd, 430 KiB/s wr, 24 op/s 2023-03-10T00:11:05.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:05 smithi122 ceph-mon[93869]: pgmap v18204: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 862 KiB/s rd, 771 KiB/s wr, 25 op/s 2023-03-10T00:11:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:05 smithi181 ceph-mon[94189]: pgmap v18204: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 862 KiB/s rd, 771 KiB/s wr, 25 op/s 2023-03-10T00:11:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:05 smithi038 ceph-mon[92422]: pgmap v18204: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 862 KiB/s rd, 771 KiB/s wr, 25 op/s 2023-03-10T00:11:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33176.454871 secs 2023-03-10T00:11:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33176.454871 secs 2023-03-10T00:11:06.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33176.454871 secs 2023-03-10T00:11:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:07 smithi122 ceph-mon[93869]: pgmap v18205: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1015 KiB/s rd, 945 KiB/s wr, 31 op/s 2023-03-10T00:11:07.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:07 smithi181 ceph-mon[94189]: pgmap v18205: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1015 KiB/s rd, 945 KiB/s wr, 31 op/s 2023-03-10T00:11:07.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:07 smithi038 ceph-mon[92422]: pgmap v18205: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1015 KiB/s rd, 945 KiB/s wr, 31 op/s 2023-03-10T00:11:08.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33178.883117 secs 2023-03-10T00:11:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:11:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:11:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33640.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:11:08.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33178.883117 secs 2023-03-10T00:11:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:11:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:11:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33178.883117 secs 2023-03-10T00:11:08.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:11:08.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:11:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:11:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:11:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:09 smithi038 ceph-mon[92422]: pgmap v18206: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 708 KiB/s rd, 728 KiB/s wr, 26 op/s 2023-03-10T00:11:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:11:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:11:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:09 smithi122 ceph-mon[93869]: pgmap v18206: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 708 KiB/s rd, 728 KiB/s wr, 26 op/s 2023-03-10T00:11:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:11:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:11:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:09 smithi181 ceph-mon[94189]: pgmap v18206: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 708 KiB/s rd, 728 KiB/s wr, 26 op/s 2023-03-10T00:11:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:10 smithi038 ceph-mon[92422]: osdmap e1353: 12 total, 12 up, 12 in 2023-03-10T00:11:10.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:10 smithi122 ceph-mon[93869]: osdmap e1353: 12 total, 12 up, 12 in 2023-03-10T00:11:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:10 smithi181 ceph-mon[94189]: osdmap e1353: 12 total, 12 up, 12 in 2023-03-10T00:11:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33181.454933 secs 2023-03-10T00:11:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:11 smithi038 ceph-mon[92422]: pgmap v18208: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 966 KiB/s rd, 1007 KiB/s wr, 30 op/s 2023-03-10T00:11:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33181.454933 secs 2023-03-10T00:11:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:11 smithi122 ceph-mon[93869]: pgmap v18208: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 966 KiB/s rd, 1007 KiB/s wr, 30 op/s 2023-03-10T00:11:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33181.454933 secs 2023-03-10T00:11:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:11 smithi181 ceph-mon[94189]: pgmap v18208: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 966 KiB/s rd, 1007 KiB/s wr, 30 op/s 2023-03-10T00:11:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33645.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:11:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:14 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33183.883150 secs 2023-03-10T00:11:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:14 smithi122 ceph-mon[93869]: pgmap v18209: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 957 KiB/s rd, 981 KiB/s wr, 15 op/s 2023-03-10T00:11:14.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:14 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33183.883150 secs 2023-03-10T00:11:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:14 smithi181 ceph-mon[94189]: pgmap v18209: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 957 KiB/s rd, 981 KiB/s wr, 15 op/s 2023-03-10T00:11:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:14 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33183.883150 secs 2023-03-10T00:11:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:14 smithi038 ceph-mon[92422]: pgmap v18209: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 957 KiB/s rd, 981 KiB/s wr, 15 op/s 2023-03-10T00:11:15.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:15 smithi122 ceph-mon[93869]: pgmap v18210: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 670 KiB/s rd, 694 KiB/s wr, 15 op/s 2023-03-10T00:11:15.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:15 smithi181 ceph-mon[94189]: pgmap v18210: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 670 KiB/s rd, 694 KiB/s wr, 15 op/s 2023-03-10T00:11:15.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:15 smithi038 ceph-mon[92422]: pgmap v18210: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 670 KiB/s rd, 694 KiB/s wr, 15 op/s 2023-03-10T00:11:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33186.455017 secs 2023-03-10T00:11:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33186.455017 secs 2023-03-10T00:11:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33186.455017 secs 2023-03-10T00:11:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:11:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:11:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:17 smithi038 ceph-mon[92422]: pgmap v18211: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 487 KiB/s wr, 10 op/s 2023-03-10T00:11:17.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:11:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:11:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:17 smithi122 ceph-mon[93869]: pgmap v18211: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 487 KiB/s wr, 10 op/s 2023-03-10T00:11:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:11:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:11:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:17 smithi181 ceph-mon[94189]: pgmap v18211: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 487 KiB/s wr, 10 op/s 2023-03-10T00:11:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33650.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:11:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:19 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33188.883208 secs 2023-03-10T00:11:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:19 smithi122 ceph-mon[93869]: osdmap e1354: 12 total, 12 up, 12 in 2023-03-10T00:11:19.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:19 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33188.883208 secs 2023-03-10T00:11:19.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:19 smithi181 ceph-mon[94189]: osdmap e1354: 12 total, 12 up, 12 in 2023-03-10T00:11:19.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:19 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33188.883208 secs 2023-03-10T00:11:19.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:19 smithi038 ceph-mon[92422]: osdmap e1354: 12 total, 12 up, 12 in 2023-03-10T00:11:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:11:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:11:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:20 smithi122 ceph-mon[93869]: pgmap v18213: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 492 KiB/s wr, 10 op/s 2023-03-10T00:11:20.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:11:20.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:11:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:20 smithi181 ceph-mon[94189]: pgmap v18213: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 492 KiB/s wr, 10 op/s 2023-03-10T00:11:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:11:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:11:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:20 smithi038 ceph-mon[92422]: pgmap v18213: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 492 KiB/s wr, 10 op/s 2023-03-10T00:11:21.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33191.455074 secs 2023-03-10T00:11:21.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33191.455074 secs 2023-03-10T00:11:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33191.455074 secs 2023-03-10T00:11:22.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:22 smithi122 ceph-mon[93869]: pgmap v18214: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 123 KiB/s wr, 8 op/s 2023-03-10T00:11:22.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:22 smithi181 ceph-mon[94189]: pgmap v18214: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 123 KiB/s wr, 8 op/s 2023-03-10T00:11:22.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:22 smithi038 ceph-mon[92422]: pgmap v18214: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 123 KiB/s wr, 8 op/s 2023-03-10T00:11:23.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33655.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:11:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:24 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33193.883270 secs 2023-03-10T00:11:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:24 smithi122 ceph-mon[93869]: pgmap v18215: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 123 KiB/s wr, 7 op/s 2023-03-10T00:11:24.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:24 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33193.883270 secs 2023-03-10T00:11:24.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:24 smithi181 ceph-mon[94189]: pgmap v18215: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 123 KiB/s wr, 7 op/s 2023-03-10T00:11:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:24 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33193.883270 secs 2023-03-10T00:11:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:24 smithi038 ceph-mon[92422]: pgmap v18215: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 123 KiB/s wr, 7 op/s 2023-03-10T00:11:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:25 smithi122 ceph-mon[93869]: pgmap v18216: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 409 B/s wr, 7 op/s 2023-03-10T00:11:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:25 smithi181 ceph-mon[94189]: pgmap v18216: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 409 B/s wr, 7 op/s 2023-03-10T00:11:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:25 smithi038 ceph-mon[92422]: pgmap v18216: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 409 B/s wr, 7 op/s 2023-03-10T00:11:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33196.455137 secs 2023-03-10T00:11:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33196.455137 secs 2023-03-10T00:11:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33196.455137 secs 2023-03-10T00:11:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:27 smithi122 ceph-mon[93869]: pgmap v18217: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-10T00:11:27.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:27 smithi181 ceph-mon[94189]: pgmap v18217: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-10T00:11:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:27 smithi038 ceph-mon[92422]: pgmap v18217: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-10T00:11:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33660.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:11:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:29 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33198.883358 secs 2023-03-10T00:11:29.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:29 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33198.883358 secs 2023-03-10T00:11:29.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:29 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33198.883358 secs 2023-03-10T00:11:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:30 smithi122 ceph-mon[93869]: pgmap v18218: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2023-03-10T00:11:30.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:30 smithi181 ceph-mon[94189]: pgmap v18218: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2023-03-10T00:11:30.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:30 smithi038 ceph-mon[92422]: pgmap v18218: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2023-03-10T00:11:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33201.455195 secs 2023-03-10T00:11:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33201.455195 secs 2023-03-10T00:11:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33201.455195 secs 2023-03-10T00:11:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:32 smithi122 ceph-mon[93869]: pgmap v18219: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-10T00:11:32.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:32 smithi181 ceph-mon[94189]: pgmap v18219: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-10T00:11:32.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:32 smithi038 ceph-mon[92422]: pgmap v18219: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-10T00:11:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33665.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:11:34.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:34 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33203.883418 secs 2023-03-10T00:11:34.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:34 smithi122 ceph-mon[93869]: pgmap v18220: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-10T00:11:34.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:34 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33203.883418 secs 2023-03-10T00:11:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:34 smithi181 ceph-mon[94189]: pgmap v18220: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-10T00:11:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:34 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33203.883418 secs 2023-03-10T00:11:34.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:34 smithi038 ceph-mon[92422]: pgmap v18220: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-10T00:11:35.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:35 smithi122 ceph-mon[93869]: pgmap v18221: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-10T00:11:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:35 smithi181 ceph-mon[94189]: pgmap v18221: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-10T00:11:35.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:35 smithi038 ceph-mon[92422]: pgmap v18221: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-10T00:11:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33206.455255 secs 2023-03-10T00:11:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33206.455255 secs 2023-03-10T00:11:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33206.455255 secs 2023-03-10T00:11:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:37 smithi122 ceph-mon[93869]: pgmap v18222: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-10T00:11:37.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:37 smithi181 ceph-mon[94189]: pgmap v18222: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-10T00:11:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:37 smithi038 ceph-mon[92422]: pgmap v18222: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-10T00:11:38.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33670.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:11:39.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:39 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33208.883500 secs 2023-03-10T00:11:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:11:39.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:39 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33208.883500 secs 2023-03-10T00:11:39.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:11:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:39 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33208.883500 secs 2023-03-10T00:11:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:11:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:40 smithi122 ceph-mon[93869]: pgmap v18223: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-10T00:11:40.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:40 smithi181 ceph-mon[94189]: pgmap v18223: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-10T00:11:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:40 smithi038 ceph-mon[92422]: pgmap v18223: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-10T00:11:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33211.455313 secs 2023-03-10T00:11:41.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33211.455313 secs 2023-03-10T00:11:41.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33211.455313 secs 2023-03-10T00:11:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:42 smithi122 ceph-mon[93869]: pgmap v18224: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 341 B/s wr, 9 op/s 2023-03-10T00:11:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:11:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:11:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:11:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:11:42.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:11:42.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:11:42.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:42 smithi181 ceph-mon[94189]: pgmap v18224: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 341 B/s wr, 9 op/s 2023-03-10T00:11:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:11:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:11:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:11:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:11:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:11:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:11:42.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:42 smithi038 ceph-mon[92422]: pgmap v18224: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 341 B/s wr, 9 op/s 2023-03-10T00:11:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:11:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:11:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:11:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:11:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:11:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:11:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:11:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:11:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:11:43.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:11:43.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:11:43.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:11:43.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:11:43.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:11:43.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:11:43.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:11:43.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:11:43.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:11:43.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:11:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:11:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:11:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:11:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:11:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:11:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:11:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:11:43.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:11:43.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:11:43.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:11:43.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:11:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:11:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:11:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:11:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:11:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:11:43.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:11:43.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:11:43.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:11:43.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:11:43.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:11:43.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:11:43.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:11:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33675.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:11:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:44 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-10T00:11:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:44 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-10T00:11:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:44 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-10T00:11:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:44 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-10T00:11:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:44 smithi122 ceph-mon[93869]: pgmap v18225: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-10T00:11:44.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:44 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-10T00:11:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:44 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-10T00:11:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:44 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-10T00:11:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:44 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-10T00:11:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:44 smithi181 ceph-mon[94189]: pgmap v18225: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-10T00:11:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:44 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-10T00:11:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:44 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-10T00:11:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:44 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-10T00:11:44.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:44 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-10T00:11:44.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:44 smithi038 ceph-mon[92422]: pgmap v18225: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-10T00:11:45.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:45 smithi122 ceph-mon[93869]: pgmap v18226: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 179 KiB/s wr, 7 op/s 2023-03-10T00:11:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:45 smithi181 ceph-mon[94189]: pgmap v18226: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 179 KiB/s wr, 7 op/s 2023-03-10T00:11:45.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:45 smithi038 ceph-mon[92422]: pgmap v18226: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 179 KiB/s wr, 7 op/s 2023-03-10T00:11:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33216.455403 secs 2023-03-10T00:11:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:46 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:11:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33216.455403 secs 2023-03-10T00:11:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:46 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:11:46.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33216.455403 secs 2023-03-10T00:11:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:46 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:11:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:47 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:11:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:11:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:11:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:47 smithi122 ceph-mon[93869]: pgmap v18227: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 207 KiB/s wr, 10 op/s 2023-03-10T00:11:47.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:47 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:11:47.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:11:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:11:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:47 smithi181 ceph-mon[94189]: pgmap v18227: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 207 KiB/s wr, 10 op/s 2023-03-10T00:11:47.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:47 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:11:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:11:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:11:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:47 smithi038 ceph-mon[92422]: pgmap v18227: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 207 KiB/s wr, 10 op/s 2023-03-10T00:11:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33680.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:11:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:49 smithi122 ceph-mon[93869]: pgmap v18228: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 207 KiB/s wr, 8 op/s 2023-03-10T00:11:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:49 smithi181 ceph-mon[94189]: pgmap v18228: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 207 KiB/s wr, 8 op/s 2023-03-10T00:11:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:49 smithi038 ceph-mon[92422]: pgmap v18228: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 207 KiB/s wr, 8 op/s 2023-03-10T00:11:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33221.455465 secs 2023-03-10T00:11:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33221.455465 secs 2023-03-10T00:11:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33221.455465 secs 2023-03-10T00:11:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:51 smithi122 ceph-mon[93869]: pgmap v18229: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 234 KiB/s wr, 12 op/s 2023-03-10T00:11:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:51 smithi181 ceph-mon[94189]: pgmap v18229: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 234 KiB/s wr, 12 op/s 2023-03-10T00:11:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:51 smithi038 ceph-mon[92422]: pgmap v18229: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 234 KiB/s wr, 12 op/s 2023-03-10T00:11:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33685.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:11:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:54 smithi122 ceph-mon[93869]: pgmap v18230: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 234 KiB/s wr, 10 op/s 2023-03-10T00:11:54.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:54 smithi181 ceph-mon[94189]: pgmap v18230: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 234 KiB/s wr, 10 op/s 2023-03-10T00:11:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:54 smithi038 ceph-mon[92422]: pgmap v18230: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 234 KiB/s wr, 10 op/s 2023-03-10T00:11:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33226.455525 secs 2023-03-10T00:11:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:11:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:11:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:56 smithi122 ceph-mon[93869]: pgmap v18231: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 690 KiB/s rd, 576 KiB/s wr, 11 op/s 2023-03-10T00:11:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33226.455525 secs 2023-03-10T00:11:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:11:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:11:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:56 smithi181 ceph-mon[94189]: pgmap v18231: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 690 KiB/s rd, 576 KiB/s wr, 11 op/s 2023-03-10T00:11:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33226.455525 secs 2023-03-10T00:11:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:11:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:11:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:56 smithi038 ceph-mon[92422]: pgmap v18231: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 690 KiB/s rd, 576 KiB/s wr, 11 op/s 2023-03-10T00:11:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:11:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:11:57.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:11:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:11:57.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:11:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:11:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:58 smithi122 ceph-mon[93869]: pgmap v18232: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 792 KiB/s rd, 677 KiB/s wr, 15 op/s 2023-03-10T00:11:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:58 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 33228.883735 secs 2023-03-10T00:11:58.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:58 smithi181 ceph-mon[94189]: pgmap v18232: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 792 KiB/s rd, 677 KiB/s wr, 15 op/s 2023-03-10T00:11:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:58 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 33228.883735 secs 2023-03-10T00:11:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:58 smithi038 ceph-mon[92422]: pgmap v18232: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 792 KiB/s rd, 677 KiB/s wr, 15 op/s 2023-03-10T00:11:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:58 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 33228.883735 secs 2023-03-10T00:11:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33690.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:11:59.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:59 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:11:59.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:11:59.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:11:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:11:59.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:59 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:11:59.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:11:59.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:11:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:11:59.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:59 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:11:59.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:11:59.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:11:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:12:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:00 smithi122 ceph-mon[93869]: pgmap v18233: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 763 KiB/s rd, 650 KiB/s wr, 12 op/s 2023-03-10T00:12:00.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:00 smithi181 ceph-mon[94189]: pgmap v18233: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 763 KiB/s rd, 650 KiB/s wr, 12 op/s 2023-03-10T00:12:00.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:00 smithi038 ceph-mon[92422]: pgmap v18233: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 763 KiB/s rd, 650 KiB/s wr, 12 op/s 2023-03-10T00:12:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33231.455581 secs 2023-03-10T00:12:01.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33231.455581 secs 2023-03-10T00:12:01.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33231.455581 secs 2023-03-10T00:12:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:02 smithi038 ceph-mon[92422]: pgmap v18234: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 792 KiB/s rd, 688 KiB/s wr, 16 op/s 2023-03-10T00:12:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:02 smithi122 ceph-mon[93869]: pgmap v18234: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 792 KiB/s rd, 688 KiB/s wr, 16 op/s 2023-03-10T00:12:02.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:02 smithi181 ceph-mon[94189]: pgmap v18234: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 792 KiB/s rd, 688 KiB/s wr, 16 op/s 2023-03-10T00:12:03.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33695.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:12:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:04 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33233.883796 secs 2023-03-10T00:12:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:04 smithi122 ceph-mon[93869]: pgmap v18235: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 833 KiB/s wr, 26 op/s 2023-03-10T00:12:04.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:04 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33233.883796 secs 2023-03-10T00:12:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:04 smithi181 ceph-mon[94189]: pgmap v18235: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 833 KiB/s wr, 26 op/s 2023-03-10T00:12:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:04 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33233.883796 secs 2023-03-10T00:12:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:04 smithi038 ceph-mon[92422]: pgmap v18235: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 833 KiB/s wr, 26 op/s 2023-03-10T00:12:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33236.455640 secs 2023-03-10T00:12:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:06 smithi122 ceph-mon[93869]: osdmap e1355: 12 total, 12 up, 12 in 2023-03-10T00:12:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:06 smithi122 ceph-mon[93869]: pgmap v18237: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 560 KiB/s rd, 588 KiB/s wr, 29 op/s 2023-03-10T00:12:06.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33236.455640 secs 2023-03-10T00:12:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:06 smithi181 ceph-mon[94189]: osdmap e1355: 12 total, 12 up, 12 in 2023-03-10T00:12:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:06 smithi181 ceph-mon[94189]: pgmap v18237: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 560 KiB/s rd, 588 KiB/s wr, 29 op/s 2023-03-10T00:12:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33236.455640 secs 2023-03-10T00:12:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:06 smithi038 ceph-mon[92422]: osdmap e1355: 12 total, 12 up, 12 in 2023-03-10T00:12:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:06 smithi038 ceph-mon[92422]: pgmap v18237: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 560 KiB/s rd, 588 KiB/s wr, 29 op/s 2023-03-10T00:12:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:12:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:12:07.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:12:07.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:12:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:12:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:12:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:12:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:12:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:08 smithi122 ceph-mon[93869]: pgmap v18238: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 443 KiB/s rd, 483 KiB/s wr, 29 op/s 2023-03-10T00:12:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33238.883854 secs 2023-03-10T00:12:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:12:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:12:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:08 smithi181 ceph-mon[94189]: pgmap v18238: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 443 KiB/s rd, 483 KiB/s wr, 29 op/s 2023-03-10T00:12:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33238.883854 secs 2023-03-10T00:12:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:12:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:12:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:08 smithi038 ceph-mon[92422]: pgmap v18238: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 443 KiB/s rd, 483 KiB/s wr, 29 op/s 2023-03-10T00:12:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33238.883854 secs 2023-03-10T00:12:08.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33700.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:12:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:12:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:12:09.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:12:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:12:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:12:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:12:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:10 smithi122 ceph-mon[93869]: pgmap v18239: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 443 KiB/s rd, 483 KiB/s wr, 29 op/s 2023-03-10T00:12:10.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:10 smithi181 ceph-mon[94189]: pgmap v18239: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 443 KiB/s rd, 483 KiB/s wr, 29 op/s 2023-03-10T00:12:10.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:10 smithi038 ceph-mon[92422]: pgmap v18239: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 443 KiB/s rd, 483 KiB/s wr, 29 op/s 2023-03-10T00:12:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33241.455698 secs 2023-03-10T00:12:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33241.455698 secs 2023-03-10T00:12:11.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33241.455698 secs 2023-03-10T00:12:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:12 smithi038 ceph-mon[92422]: pgmap v18240: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 438 KiB/s wr, 26 op/s 2023-03-10T00:12:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:12 smithi122 ceph-mon[93869]: pgmap v18240: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 438 KiB/s wr, 26 op/s 2023-03-10T00:12:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:12 smithi181 ceph-mon[94189]: pgmap v18240: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 438 KiB/s wr, 26 op/s 2023-03-10T00:12:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33705.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:12:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:14 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33243.883961 secs 2023-03-10T00:12:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:14 smithi122 ceph-mon[93869]: osdmap e1356: 12 total, 12 up, 12 in 2023-03-10T00:12:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:14 smithi122 ceph-mon[93869]: pgmap v18242: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 496 KiB/s wr, 13 op/s 2023-03-10T00:12:14.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:14 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33243.883961 secs 2023-03-10T00:12:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:14 smithi181 ceph-mon[94189]: osdmap e1356: 12 total, 12 up, 12 in 2023-03-10T00:12:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:14 smithi181 ceph-mon[94189]: pgmap v18242: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 496 KiB/s wr, 13 op/s 2023-03-10T00:12:14.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:14 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33243.883961 secs 2023-03-10T00:12:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:14 smithi038 ceph-mon[92422]: osdmap e1356: 12 total, 12 up, 12 in 2023-03-10T00:12:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:14 smithi038 ceph-mon[92422]: pgmap v18242: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 496 KiB/s wr, 13 op/s 2023-03-10T00:12:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:15 smithi122 ceph-mon[93869]: pgmap v18243: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 395 KiB/s rd, 406 KiB/s wr, 10 op/s 2023-03-10T00:12:15.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:15 smithi181 ceph-mon[94189]: pgmap v18243: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 395 KiB/s rd, 406 KiB/s wr, 10 op/s 2023-03-10T00:12:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:15 smithi038 ceph-mon[92422]: pgmap v18243: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 395 KiB/s rd, 406 KiB/s wr, 10 op/s 2023-03-10T00:12:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33246.455759 secs 2023-03-10T00:12:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33246.455759 secs 2023-03-10T00:12:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33246.455759 secs 2023-03-10T00:12:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:17 smithi038 ceph-mon[92422]: pgmap v18244: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 176 KiB/s wr, 8 op/s 2023-03-10T00:12:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:17 smithi122 ceph-mon[93869]: pgmap v18244: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 176 KiB/s wr, 8 op/s 2023-03-10T00:12:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:17 smithi181 ceph-mon[94189]: pgmap v18244: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 176 KiB/s wr, 8 op/s 2023-03-10T00:12:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33710.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:12:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:19 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33248.884020 secs 2023-03-10T00:12:19.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:19 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33248.884020 secs 2023-03-10T00:12:19.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:19 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33248.884020 secs 2023-03-10T00:12:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:20 smithi122 ceph-mon[93869]: pgmap v18245: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 176 KiB/s wr, 8 op/s 2023-03-10T00:12:20.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:20 smithi181 ceph-mon[94189]: pgmap v18245: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 176 KiB/s wr, 8 op/s 2023-03-10T00:12:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:20 smithi038 ceph-mon[92422]: pgmap v18245: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 176 KiB/s wr, 8 op/s 2023-03-10T00:12:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33251.455816 secs 2023-03-10T00:12:21.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33251.455816 secs 2023-03-10T00:12:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33251.455816 secs 2023-03-10T00:12:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:22 smithi122 ceph-mon[93869]: pgmap v18246: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 175 KiB/s wr, 8 op/s 2023-03-10T00:12:22.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:22 smithi181 ceph-mon[94189]: pgmap v18246: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 175 KiB/s wr, 8 op/s 2023-03-10T00:12:22.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:22 smithi038 ceph-mon[92422]: pgmap v18246: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 175 KiB/s wr, 8 op/s 2023-03-10T00:12:23.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33715.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:12:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:24 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33253.884094 secs 2023-03-10T00:12:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:24 smithi122 ceph-mon[93869]: pgmap v18247: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 121 KiB/s wr, 7 op/s 2023-03-10T00:12:24.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:24 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33253.884094 secs 2023-03-10T00:12:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:24 smithi181 ceph-mon[94189]: pgmap v18247: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 121 KiB/s wr, 7 op/s 2023-03-10T00:12:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:24 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33253.884094 secs 2023-03-10T00:12:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:24 smithi038 ceph-mon[92422]: pgmap v18247: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 121 KiB/s wr, 7 op/s 2023-03-10T00:12:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:25 smithi122 ceph-mon[93869]: pgmap v18248: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 103 KiB/s wr, 6 op/s 2023-03-10T00:12:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:25 smithi181 ceph-mon[94189]: pgmap v18248: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 103 KiB/s wr, 6 op/s 2023-03-10T00:12:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:25 smithi038 ceph-mon[92422]: pgmap v18248: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 103 KiB/s wr, 6 op/s 2023-03-10T00:12:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33256.455873 secs 2023-03-10T00:12:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33256.455873 secs 2023-03-10T00:12:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33256.455873 secs 2023-03-10T00:12:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:27 smithi038 ceph-mon[92422]: pgmap v18249: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 103 KiB/s wr, 9 op/s 2023-03-10T00:12:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:27 smithi122 ceph-mon[93869]: pgmap v18249: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 103 KiB/s wr, 9 op/s 2023-03-10T00:12:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:27 smithi181 ceph-mon[94189]: pgmap v18249: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 103 KiB/s wr, 9 op/s 2023-03-10T00:12:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33258.884142 secs 2023-03-10T00:12:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:28 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:12:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:28 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:12:28.793 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33720.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:12:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33258.884142 secs 2023-03-10T00:12:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:28 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:12:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:28 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:12:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33258.884142 secs 2023-03-10T00:12:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:28 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:12:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:28 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:12:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:29 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:12:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:29 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:12:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:29 smithi122 ceph-mon[93869]: pgmap v18250: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 103 KiB/s wr, 5 op/s 2023-03-10T00:12:29.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:29 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:12:29.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:29 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:12:29.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:29 smithi181 ceph-mon[94189]: pgmap v18250: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 103 KiB/s wr, 5 op/s 2023-03-10T00:12:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:29 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:12:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:29 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:12:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:29 smithi038 ceph-mon[92422]: pgmap v18250: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 103 KiB/s wr, 5 op/s 2023-03-10T00:12:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33261.455959 secs 2023-03-10T00:12:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33261.455959 secs 2023-03-10T00:12:30.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33261.455959 secs 2023-03-10T00:12:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:31 smithi122 ceph-mon[93869]: pgmap v18251: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 103 KiB/s wr, 7 op/s 2023-03-10T00:12:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:31 smithi181 ceph-mon[94189]: pgmap v18251: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 103 KiB/s wr, 7 op/s 2023-03-10T00:12:31.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:31 smithi038 ceph-mon[92422]: pgmap v18251: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 103 KiB/s wr, 7 op/s 2023-03-10T00:12:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33725.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:12:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:34 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33263.884288 secs 2023-03-10T00:12:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:34 smithi122 ceph-mon[93869]: pgmap v18252: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 102 KiB/s wr, 6 op/s 2023-03-10T00:12:34.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:34 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33263.884288 secs 2023-03-10T00:12:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:34 smithi181 ceph-mon[94189]: pgmap v18252: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 102 KiB/s wr, 6 op/s 2023-03-10T00:12:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:34 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33263.884288 secs 2023-03-10T00:12:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:34 smithi038 ceph-mon[92422]: pgmap v18252: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 102 KiB/s wr, 6 op/s 2023-03-10T00:12:35.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:35 smithi122 ceph-mon[93869]: pgmap v18253: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-10T00:12:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:35 smithi181 ceph-mon[94189]: pgmap v18253: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-10T00:12:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:35 smithi038 ceph-mon[92422]: pgmap v18253: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-10T00:12:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33266.456018 secs 2023-03-10T00:12:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33266.456018 secs 2023-03-10T00:12:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33266.456018 secs 2023-03-10T00:12:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:37 smithi038 ceph-mon[92422]: pgmap v18254: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2023-03-10T00:12:37.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:37 smithi122 ceph-mon[93869]: pgmap v18254: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2023-03-10T00:12:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:37 smithi181 ceph-mon[94189]: pgmap v18254: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2023-03-10T00:12:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33730.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:12:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:39 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33268.884346 secs 2023-03-10T00:12:39.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:39 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33268.884346 secs 2023-03-10T00:12:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:39 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33268.884346 secs 2023-03-10T00:12:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:40 smithi122 ceph-mon[93869]: pgmap v18255: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-10T00:12:40.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:40 smithi181 ceph-mon[94189]: pgmap v18255: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-10T00:12:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:40 smithi038 ceph-mon[92422]: pgmap v18255: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-10T00:12:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33271.456076 secs 2023-03-10T00:12:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33271.456076 secs 2023-03-10T00:12:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33271.456076 secs 2023-03-10T00:12:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:42 smithi122 ceph-mon[93869]: pgmap v18256: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 7 op/s 2023-03-10T00:12:42.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:42 smithi181 ceph-mon[94189]: pgmap v18256: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 7 op/s 2023-03-10T00:12:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:42 smithi038 ceph-mon[92422]: pgmap v18256: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 7 op/s 2023-03-10T00:12:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33273.884404 secs 2023-03-10T00:12:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:12:43.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33273.884404 secs 2023-03-10T00:12:43.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:12:43.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33273.884404 secs 2023-03-10T00:12:43.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:12:43.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33735.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:12:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:44 smithi122 ceph-mon[93869]: pgmap v18257: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 221 KiB/s wr, 6 op/s 2023-03-10T00:12:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:44 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:12:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:44 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:12:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:44 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:12:44.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:44 smithi181 ceph-mon[94189]: pgmap v18257: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 221 KiB/s wr, 6 op/s 2023-03-10T00:12:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:44 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:12:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:44 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:12:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:44 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:12:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:44 smithi038 ceph-mon[92422]: pgmap v18257: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 221 KiB/s wr, 6 op/s 2023-03-10T00:12:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:44 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:12:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:44 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:12:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:44 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:12:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:45 smithi181 ceph-mon[94189]: pgmap v18258: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 221 KiB/s wr, 6 op/s 2023-03-10T00:12:45.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:45 smithi038 ceph-mon[92422]: pgmap v18258: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 221 KiB/s wr, 6 op/s 2023-03-10T00:12:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:45 smithi122 ceph-mon[93869]: pgmap v18258: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 221 KiB/s wr, 6 op/s 2023-03-10T00:12:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33276.456139 secs 2023-03-10T00:12:46.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33276.456139 secs 2023-03-10T00:12:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33276.456139 secs 2023-03-10T00:12:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:47 smithi038 ceph-mon[92422]: pgmap v18259: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 311 KiB/s wr, 10 op/s 2023-03-10T00:12:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:47 smithi122 ceph-mon[93869]: pgmap v18259: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 311 KiB/s wr, 10 op/s 2023-03-10T00:12:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:47 smithi181 ceph-mon[94189]: pgmap v18259: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 311 KiB/s wr, 10 op/s 2023-03-10T00:12:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33278.884495 secs 2023-03-10T00:12:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:12:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:12:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33740.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:12:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33278.884495 secs 2023-03-10T00:12:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:12:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:12:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33278.884495 secs 2023-03-10T00:12:48.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:12:48.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:12:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:12:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:12:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:49 smithi038 ceph-mon[92422]: pgmap v18260: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 311 KiB/s wr, 7 op/s 2023-03-10T00:12:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:12:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:12:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:49 smithi122 ceph-mon[93869]: pgmap v18260: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 311 KiB/s wr, 7 op/s 2023-03-10T00:12:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:12:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:12:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:49 smithi181 ceph-mon[94189]: pgmap v18260: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 311 KiB/s wr, 7 op/s 2023-03-10T00:12:50.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33281.456197 secs 2023-03-10T00:12:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33281.456197 secs 2023-03-10T00:12:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33281.456197 secs 2023-03-10T00:12:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:51 smithi122 ceph-mon[93869]: pgmap v18261: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 719 KiB/s rd, 618 KiB/s wr, 11 op/s 2023-03-10T00:12:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:51 smithi181 ceph-mon[94189]: pgmap v18261: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 719 KiB/s rd, 618 KiB/s wr, 11 op/s 2023-03-10T00:12:51.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:51 smithi038 ceph-mon[92422]: pgmap v18261: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 719 KiB/s rd, 618 KiB/s wr, 11 op/s 2023-03-10T00:12:53.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33745.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:12:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:54 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33283.884580 secs 2023-03-10T00:12:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:54 smithi122 ceph-mon[93869]: pgmap v18262: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 717 KiB/s rd, 617 KiB/s wr, 10 op/s 2023-03-10T00:12:54.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:54 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33283.884580 secs 2023-03-10T00:12:54.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:54 smithi181 ceph-mon[94189]: pgmap v18262: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 717 KiB/s rd, 617 KiB/s wr, 10 op/s 2023-03-10T00:12:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:54 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33283.884580 secs 2023-03-10T00:12:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:54 smithi038 ceph-mon[92422]: pgmap v18262: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 717 KiB/s rd, 617 KiB/s wr, 10 op/s 2023-03-10T00:12:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33286.456258 secs 2023-03-10T00:12:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:12:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:12:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:56 smithi122 ceph-mon[93869]: pgmap v18263: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 604 KiB/s rd, 396 KiB/s wr, 9 op/s 2023-03-10T00:12:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33286.456258 secs 2023-03-10T00:12:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:12:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:12:56.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:56 smithi181 ceph-mon[94189]: pgmap v18263: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 604 KiB/s rd, 396 KiB/s wr, 9 op/s 2023-03-10T00:12:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33286.456258 secs 2023-03-10T00:12:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:12:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:12:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:56 smithi038 ceph-mon[92422]: pgmap v18263: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 604 KiB/s rd, 396 KiB/s wr, 9 op/s 2023-03-10T00:12:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:12:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:12:57.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:12:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:12:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:12:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:12:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:58 smithi181 ceph-mon[94189]: pgmap v18264: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 984 KiB/s rd, 761 KiB/s wr, 14 op/s 2023-03-10T00:12:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33288.884638 secs 2023-03-10T00:12:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:58 smithi038 ceph-mon[92422]: pgmap v18264: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 984 KiB/s rd, 761 KiB/s wr, 14 op/s 2023-03-10T00:12:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33288.884638 secs 2023-03-10T00:12:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:58 smithi122 ceph-mon[93869]: pgmap v18264: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 984 KiB/s rd, 761 KiB/s wr, 14 op/s 2023-03-10T00:12:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33288.884638 secs 2023-03-10T00:12:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33750.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:12:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:12:59 smithi181 ceph-mon[94189]: pgmap v18265: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 701 KiB/s rd, 671 KiB/s wr, 11 op/s 2023-03-10T00:12:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:12:59 smithi038 ceph-mon[92422]: pgmap v18265: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 701 KiB/s rd, 671 KiB/s wr, 11 op/s 2023-03-10T00:12:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:12:59 smithi122 ceph-mon[93869]: pgmap v18265: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 701 KiB/s rd, 671 KiB/s wr, 11 op/s 2023-03-10T00:13:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33291.456341 secs 2023-03-10T00:13:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33291.456341 secs 2023-03-10T00:13:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33291.456341 secs 2023-03-10T00:13:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:01 smithi122 ceph-mon[93869]: pgmap v18266: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 729 KiB/s rd, 698 KiB/s wr, 14 op/s 2023-03-10T00:13:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:01 smithi181 ceph-mon[94189]: pgmap v18266: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 729 KiB/s rd, 698 KiB/s wr, 14 op/s 2023-03-10T00:13:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:01 smithi038 ceph-mon[92422]: pgmap v18266: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 729 KiB/s rd, 698 KiB/s wr, 14 op/s 2023-03-10T00:13:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33755.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:13:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:04 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33293.884854 secs 2023-03-10T00:13:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:04 smithi122 ceph-mon[93869]: pgmap v18267: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 640 KiB/s rd, 627 KiB/s wr, 24 op/s 2023-03-10T00:13:04.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:04 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33293.884854 secs 2023-03-10T00:13:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:04 smithi181 ceph-mon[94189]: pgmap v18267: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 640 KiB/s rd, 627 KiB/s wr, 24 op/s 2023-03-10T00:13:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:04 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33293.884854 secs 2023-03-10T00:13:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:04 smithi038 ceph-mon[92422]: pgmap v18267: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 640 KiB/s rd, 627 KiB/s wr, 24 op/s 2023-03-10T00:13:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:05 smithi122 ceph-mon[93869]: osdmap e1357: 12 total, 12 up, 12 in 2023-03-10T00:13:05.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:05 smithi181 ceph-mon[94189]: osdmap e1357: 12 total, 12 up, 12 in 2023-03-10T00:13:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:05 smithi038 ceph-mon[92422]: osdmap e1357: 12 total, 12 up, 12 in 2023-03-10T00:13:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33296.456403 secs 2023-03-10T00:13:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:06 smithi122 ceph-mon[93869]: pgmap v18269: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 778 KiB/s rd, 776 KiB/s wr, 28 op/s 2023-03-10T00:13:06.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33296.456403 secs 2023-03-10T00:13:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:06 smithi181 ceph-mon[94189]: pgmap v18269: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 778 KiB/s rd, 776 KiB/s wr, 28 op/s 2023-03-10T00:13:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33296.456403 secs 2023-03-10T00:13:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:06 smithi038 ceph-mon[92422]: pgmap v18269: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 778 KiB/s rd, 776 KiB/s wr, 28 op/s 2023-03-10T00:13:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:13:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:13:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:08 smithi181 ceph-mon[94189]: pgmap v18270: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 559 KiB/s wr, 28 op/s 2023-03-10T00:13:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33298.884938 secs 2023-03-10T00:13:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:13:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:13:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:08 smithi038 ceph-mon[92422]: pgmap v18270: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 559 KiB/s wr, 28 op/s 2023-03-10T00:13:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33298.884938 secs 2023-03-10T00:13:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:13:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:13:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:08 smithi122 ceph-mon[93869]: pgmap v18270: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 559 KiB/s wr, 28 op/s 2023-03-10T00:13:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33298.884938 secs 2023-03-10T00:13:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33760.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:13:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:13:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:13:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:13:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:13:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:13:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:13:10.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:10 smithi038 ceph-mon[92422]: pgmap v18271: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 559 KiB/s wr, 29 op/s 2023-03-10T00:13:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:10 smithi122 ceph-mon[93869]: pgmap v18271: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 559 KiB/s wr, 29 op/s 2023-03-10T00:13:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:10 smithi181 ceph-mon[94189]: pgmap v18271: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 559 KiB/s wr, 29 op/s 2023-03-10T00:13:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33301.456465 secs 2023-03-10T00:13:11.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33301.456465 secs 2023-03-10T00:13:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33301.456465 secs 2023-03-10T00:13:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:12 smithi122 ceph-mon[93869]: pgmap v18272: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 486 KiB/s rd, 527 KiB/s wr, 28 op/s 2023-03-10T00:13:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:12 smithi181 ceph-mon[94189]: pgmap v18272: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 486 KiB/s rd, 527 KiB/s wr, 28 op/s 2023-03-10T00:13:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:12 smithi038 ceph-mon[92422]: pgmap v18272: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 486 KiB/s rd, 527 KiB/s wr, 28 op/s 2023-03-10T00:13:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:13:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:13:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33303.885006 secs 2023-03-10T00:13:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:13 smithi122 ceph-mon[93869]: osdmap e1358: 12 total, 12 up, 12 in 2023-03-10T00:13:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:13:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:13:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33303.885006 secs 2023-03-10T00:13:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:13 smithi181 ceph-mon[94189]: osdmap e1358: 12 total, 12 up, 12 in 2023-03-10T00:13:13.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:13:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:13:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33303.885006 secs 2023-03-10T00:13:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:13 smithi038 ceph-mon[92422]: osdmap e1358: 12 total, 12 up, 12 in 2023-03-10T00:13:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33765.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:13:14.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:14 smithi122 ceph-mon[93869]: pgmap v18274: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 241 KiB/s wr, 13 op/s 2023-03-10T00:13:14.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:14 smithi181 ceph-mon[94189]: pgmap v18274: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 241 KiB/s wr, 13 op/s 2023-03-10T00:13:14.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:14 smithi038 ceph-mon[92422]: pgmap v18274: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 241 KiB/s wr, 13 op/s 2023-03-10T00:13:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33306.456548 secs 2023-03-10T00:13:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:13:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:13:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:16 smithi122 ceph-mon[93869]: pgmap v18275: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 443 KiB/s wr, 12 op/s 2023-03-10T00:13:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33306.456548 secs 2023-03-10T00:13:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:13:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:13:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:16 smithi181 ceph-mon[94189]: pgmap v18275: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 443 KiB/s wr, 12 op/s 2023-03-10T00:13:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33306.456548 secs 2023-03-10T00:13:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:13:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:13:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:16 smithi038 ceph-mon[92422]: pgmap v18275: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 443 KiB/s wr, 12 op/s 2023-03-10T00:13:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:17 smithi038 ceph-mon[92422]: pgmap v18276: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 222 KiB/s wr, 8 op/s 2023-03-10T00:13:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:17 smithi122 ceph-mon[93869]: pgmap v18276: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 222 KiB/s wr, 8 op/s 2023-03-10T00:13:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:17 smithi181 ceph-mon[94189]: pgmap v18276: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 222 KiB/s wr, 8 op/s 2023-03-10T00:13:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33770.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:13:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:19 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33308.885065 secs 2023-03-10T00:13:19.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:19 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33308.885065 secs 2023-03-10T00:13:19.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:19 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33308.885065 secs 2023-03-10T00:13:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:20 smithi122 ceph-mon[93869]: pgmap v18277: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 222 KiB/s wr, 7 op/s 2023-03-10T00:13:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:20 smithi181 ceph-mon[94189]: pgmap v18277: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 222 KiB/s wr, 7 op/s 2023-03-10T00:13:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:20 smithi038 ceph-mon[92422]: pgmap v18277: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 225 KiB/s rd, 222 KiB/s wr, 7 op/s 2023-03-10T00:13:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33311.456605 secs 2023-03-10T00:13:21.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33311.456605 secs 2023-03-10T00:13:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33311.456605 secs 2023-03-10T00:13:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:22 smithi122 ceph-mon[93869]: pgmap v18278: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 221 KiB/s wr, 7 op/s 2023-03-10T00:13:22.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:22 smithi181 ceph-mon[94189]: pgmap v18278: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 221 KiB/s wr, 7 op/s 2023-03-10T00:13:22.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:22 smithi038 ceph-mon[92422]: pgmap v18278: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 221 KiB/s wr, 7 op/s 2023-03-10T00:13:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33775.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:13:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:24 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33313.885146 secs 2023-03-10T00:13:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:24 smithi122 ceph-mon[93869]: pgmap v18279: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 217 KiB/s wr, 6 op/s 2023-03-10T00:13:24.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:24 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33313.885146 secs 2023-03-10T00:13:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:24 smithi181 ceph-mon[94189]: pgmap v18279: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 217 KiB/s wr, 6 op/s 2023-03-10T00:13:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:24 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33313.885146 secs 2023-03-10T00:13:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:24 smithi038 ceph-mon[92422]: pgmap v18279: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 217 KiB/s wr, 6 op/s 2023-03-10T00:13:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:25 smithi038 ceph-mon[92422]: pgmap v18280: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 266 KiB/s wr, 6 op/s 2023-03-10T00:13:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:25 smithi122 ceph-mon[93869]: pgmap v18280: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 266 KiB/s wr, 6 op/s 2023-03-10T00:13:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:25 smithi181 ceph-mon[94189]: pgmap v18280: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 266 KiB/s wr, 6 op/s 2023-03-10T00:13:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33316.456662 secs 2023-03-10T00:13:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33316.456662 secs 2023-03-10T00:13:26.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33316.456662 secs 2023-03-10T00:13:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:27 smithi038 ceph-mon[92422]: pgmap v18281: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 81 KiB/s wr, 7 op/s 2023-03-10T00:13:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:27 smithi122 ceph-mon[93869]: pgmap v18281: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 81 KiB/s wr, 7 op/s 2023-03-10T00:13:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:27 smithi181 ceph-mon[94189]: pgmap v18281: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 81 KiB/s wr, 7 op/s 2023-03-10T00:13:28.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33780.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:13:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:29 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33318.885208 secs 2023-03-10T00:13:29.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:29 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33318.885208 secs 2023-03-10T00:13:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:29 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33318.885208 secs 2023-03-10T00:13:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:30 smithi122 ceph-mon[93869]: pgmap v18282: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 81 KiB/s wr, 5 op/s 2023-03-10T00:13:30.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:30 smithi181 ceph-mon[94189]: pgmap v18282: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 81 KiB/s wr, 5 op/s 2023-03-10T00:13:30.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:30 smithi038 ceph-mon[92422]: pgmap v18282: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 81 KiB/s wr, 5 op/s 2023-03-10T00:13:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33321.456721 secs 2023-03-10T00:13:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33321.456721 secs 2023-03-10T00:13:31.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33321.456721 secs 2023-03-10T00:13:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:32 smithi181 ceph-mon[94189]: pgmap v18283: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 81 KiB/s wr, 7 op/s 2023-03-10T00:13:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:32 smithi038 ceph-mon[92422]: pgmap v18283: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 81 KiB/s wr, 7 op/s 2023-03-10T00:13:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:32 smithi122 ceph-mon[93869]: pgmap v18283: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 81 KiB/s wr, 7 op/s 2023-03-10T00:13:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33323.885266 secs 2023-03-10T00:13:33.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33323.885266 secs 2023-03-10T00:13:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33323.885266 secs 2023-03-10T00:13:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33785.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:13:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:34 smithi038 ceph-mon[92422]: pgmap v18284: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 81 KiB/s wr, 5 op/s 2023-03-10T00:13:34.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:34 smithi122 ceph-mon[93869]: pgmap v18284: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 81 KiB/s wr, 5 op/s 2023-03-10T00:13:34.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:34 smithi181 ceph-mon[94189]: pgmap v18284: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 81 KiB/s wr, 5 op/s 2023-03-10T00:13:35.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:35 smithi038 ceph-mon[92422]: pgmap v18285: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 81 KiB/s wr, 5 op/s 2023-03-10T00:13:35.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:35 smithi122 ceph-mon[93869]: pgmap v18285: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 81 KiB/s wr, 5 op/s 2023-03-10T00:13:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:35 smithi181 ceph-mon[94189]: pgmap v18285: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 81 KiB/s wr, 5 op/s 2023-03-10T00:13:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33326.456808 secs 2023-03-10T00:13:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33326.456808 secs 2023-03-10T00:13:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33326.456808 secs 2023-03-10T00:13:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:37 smithi122 ceph-mon[93869]: pgmap v18286: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2023-03-10T00:13:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:37 smithi181 ceph-mon[94189]: pgmap v18286: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2023-03-10T00:13:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:37 smithi038 ceph-mon[92422]: pgmap v18286: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2023-03-10T00:13:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33790.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:13:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33328.885327 secs 2023-03-10T00:13:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33328.885327 secs 2023-03-10T00:13:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33328.885327 secs 2023-03-10T00:13:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:39 smithi122 ceph-mon[93869]: pgmap v18287: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-10T00:13:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:39 smithi181 ceph-mon[94189]: pgmap v18287: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-10T00:13:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:39 smithi038 ceph-mon[92422]: pgmap v18287: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-10T00:13:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33331.456885 secs 2023-03-10T00:13:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33331.456885 secs 2023-03-10T00:13:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33331.456885 secs 2023-03-10T00:13:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:41 smithi122 ceph-mon[93869]: pgmap v18288: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 7 op/s 2023-03-10T00:13:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:41 smithi181 ceph-mon[94189]: pgmap v18288: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 7 op/s 2023-03-10T00:13:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:41 smithi038 ceph-mon[92422]: pgmap v18288: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 7 op/s 2023-03-10T00:13:43.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33795.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:13:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:44 smithi122 ceph-mon[93869]: pgmap v18289: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 682 B/s wr, 5 op/s 2023-03-10T00:13:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:44 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:13:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:44 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:13:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:44 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:13:44.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:44 smithi181 ceph-mon[94189]: pgmap v18289: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 682 B/s wr, 5 op/s 2023-03-10T00:13:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:44 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:13:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:44 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:13:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:44 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:13:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:44 smithi038 ceph-mon[92422]: pgmap v18289: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 682 B/s wr, 5 op/s 2023-03-10T00:13:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:44 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:13:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:44 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:13:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:44 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:13:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:13:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:13:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:13:45.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:13:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:13:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:13:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:13:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:13:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:13:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33336.456948 secs 2023-03-10T00:13:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:46 smithi038 ceph-mon[92422]: pgmap v18290: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 57 KiB/s wr, 6 op/s 2023-03-10T00:13:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:46 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:13:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33336.456948 secs 2023-03-10T00:13:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:46 smithi122 ceph-mon[93869]: pgmap v18290: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 57 KiB/s wr, 6 op/s 2023-03-10T00:13:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:46 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:13:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33336.456948 secs 2023-03-10T00:13:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:46 smithi181 ceph-mon[94189]: pgmap v18290: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 57 KiB/s wr, 6 op/s 2023-03-10T00:13:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:46 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:13:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:47 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:13:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:13:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:13:47.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:47 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:13:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:13:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:13:47.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:47 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:13:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:13:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:13:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:13:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:13:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:48 smithi122 ceph-mon[93869]: pgmap v18291: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 247 KiB/s rd, 187 KiB/s wr, 9 op/s 2023-03-10T00:13:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:13:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:13:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:48 smithi181 ceph-mon[94189]: pgmap v18291: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 247 KiB/s rd, 187 KiB/s wr, 9 op/s 2023-03-10T00:13:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:13:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:13:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:48 smithi038 ceph-mon[92422]: pgmap v18291: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 247 KiB/s rd, 187 KiB/s wr, 9 op/s 2023-03-10T00:13:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33800.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:13:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:13:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:13:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:13:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:13:49.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:13:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:13:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:50 smithi122 ceph-mon[93869]: pgmap v18292: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 187 KiB/s wr, 7 op/s 2023-03-10T00:13:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:50 smithi181 ceph-mon[94189]: pgmap v18292: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 187 KiB/s wr, 7 op/s 2023-03-10T00:13:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:50 smithi038 ceph-mon[92422]: pgmap v18292: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 187 KiB/s wr, 7 op/s 2023-03-10T00:13:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33341.457006 secs 2023-03-10T00:13:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33341.457006 secs 2023-03-10T00:13:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33341.457006 secs 2023-03-10T00:13:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:52 smithi122 ceph-mon[93869]: pgmap v18293: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 515 KiB/s rd, 453 KiB/s wr, 11 op/s 2023-03-10T00:13:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:52 smithi181 ceph-mon[94189]: pgmap v18293: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 515 KiB/s rd, 453 KiB/s wr, 11 op/s 2023-03-10T00:13:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:52 smithi038 ceph-mon[92422]: pgmap v18293: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 515 KiB/s rd, 453 KiB/s wr, 11 op/s 2023-03-10T00:13:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33805.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:13:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:54 smithi122 ceph-mon[93869]: pgmap v18294: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 549 KiB/s rd, 693 KiB/s wr, 10 op/s 2023-03-10T00:13:54.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:54 smithi181 ceph-mon[94189]: pgmap v18294: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 549 KiB/s rd, 693 KiB/s wr, 10 op/s 2023-03-10T00:13:54.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:54 smithi038 ceph-mon[92422]: pgmap v18294: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 549 KiB/s rd, 693 KiB/s wr, 10 op/s 2023-03-10T00:13:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33346.457064 secs 2023-03-10T00:13:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:13:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:13:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:56 smithi181 ceph-mon[94189]: pgmap v18295: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 560 KiB/s rd, 693 KiB/s wr, 10 op/s 2023-03-10T00:13:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33346.457064 secs 2023-03-10T00:13:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:13:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:13:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:56 smithi038 ceph-mon[92422]: pgmap v18295: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 560 KiB/s rd, 693 KiB/s wr, 10 op/s 2023-03-10T00:13:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33346.457064 secs 2023-03-10T00:13:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:13:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:13:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:56 smithi122 ceph-mon[93869]: pgmap v18295: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 560 KiB/s rd, 693 KiB/s wr, 10 op/s 2023-03-10T00:13:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:13:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:13:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:13:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:13:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:13:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:13:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:58 smithi038 ceph-mon[92422]: pgmap v18296: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 668 KiB/s wr, 14 op/s 2023-03-10T00:13:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:58 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 33348.885589 secs 2023-03-10T00:13:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:58 smithi122 ceph-mon[93869]: pgmap v18296: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 668 KiB/s wr, 14 op/s 2023-03-10T00:13:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:58 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 33348.885589 secs 2023-03-10T00:13:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:58 smithi181 ceph-mon[94189]: pgmap v18296: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 668 KiB/s wr, 14 op/s 2023-03-10T00:13:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:58 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 33348.885589 secs 2023-03-10T00:13:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33810.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:13:59.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:59 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:13:59.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:13:59.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:13:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:13:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:59 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:13:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:13:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:13:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:13:59.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:59 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:13:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:13:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:13:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:14:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:14:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:14:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:00 smithi122 ceph-mon[93869]: pgmap v18297: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 784 KiB/s rd, 673 KiB/s wr, 13 op/s 2023-03-10T00:14:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:14:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:14:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:00 smithi181 ceph-mon[94189]: pgmap v18297: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 784 KiB/s rd, 673 KiB/s wr, 13 op/s 2023-03-10T00:14:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:14:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:14:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:00 smithi038 ceph-mon[92422]: pgmap v18297: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 784 KiB/s rd, 673 KiB/s wr, 13 op/s 2023-03-10T00:14:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33351.457145 secs 2023-03-10T00:14:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:14:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:14:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33351.457145 secs 2023-03-10T00:14:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:14:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:14:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33351.457145 secs 2023-03-10T00:14:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:14:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:14:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:02 smithi122 ceph-mon[93869]: pgmap v18298: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 673 KiB/s wr, 15 op/s 2023-03-10T00:14:02.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:02 smithi181 ceph-mon[94189]: pgmap v18298: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 673 KiB/s wr, 15 op/s 2023-03-10T00:14:02.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:02 smithi038 ceph-mon[92422]: pgmap v18298: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 673 KiB/s wr, 15 op/s 2023-03-10T00:14:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33353.885647 secs 2023-03-10T00:14:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33353.885647 secs 2023-03-10T00:14:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33353.885647 secs 2023-03-10T00:14:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33815.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:14:04.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:04 smithi122 ceph-mon[93869]: pgmap v18299: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 701 KiB/s rd, 764 KiB/s wr, 26 op/s 2023-03-10T00:14:04.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:04 smithi181 ceph-mon[94189]: pgmap v18299: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 701 KiB/s rd, 764 KiB/s wr, 26 op/s 2023-03-10T00:14:04.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:04 smithi038 ceph-mon[92422]: pgmap v18299: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 701 KiB/s rd, 764 KiB/s wr, 26 op/s 2023-03-10T00:14:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:05 smithi122 ceph-mon[93869]: osdmap e1359: 12 total, 12 up, 12 in 2023-03-10T00:14:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:05 smithi181 ceph-mon[94189]: osdmap e1359: 12 total, 12 up, 12 in 2023-03-10T00:14:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:05 smithi038 ceph-mon[92422]: osdmap e1359: 12 total, 12 up, 12 in 2023-03-10T00:14:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33356.457226 secs 2023-03-10T00:14:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:06 smithi122 ceph-mon[93869]: pgmap v18301: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 629 KiB/s wr, 29 op/s 2023-03-10T00:14:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33356.457226 secs 2023-03-10T00:14:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:06 smithi181 ceph-mon[94189]: pgmap v18301: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 629 KiB/s wr, 29 op/s 2023-03-10T00:14:06.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33356.457226 secs 2023-03-10T00:14:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:06 smithi038 ceph-mon[92422]: pgmap v18301: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 629 KiB/s wr, 29 op/s 2023-03-10T00:14:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:14:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:14:07.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:14:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:14:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:14:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:14:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:08 smithi122 ceph-mon[93869]: pgmap v18302: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 778 KiB/s rd, 647 KiB/s wr, 30 op/s 2023-03-10T00:14:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33358.885710 secs 2023-03-10T00:14:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:08 smithi181 ceph-mon[94189]: pgmap v18302: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 778 KiB/s rd, 647 KiB/s wr, 30 op/s 2023-03-10T00:14:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33358.885710 secs 2023-03-10T00:14:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:08 smithi038 ceph-mon[92422]: pgmap v18302: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 778 KiB/s rd, 647 KiB/s wr, 30 op/s 2023-03-10T00:14:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33358.885710 secs 2023-03-10T00:14:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33820.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:14:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:09 smithi038 ceph-mon[92422]: pgmap v18303: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 444 KiB/s rd, 485 KiB/s wr, 29 op/s 2023-03-10T00:14:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:09 smithi122 ceph-mon[93869]: pgmap v18303: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 444 KiB/s rd, 485 KiB/s wr, 29 op/s 2023-03-10T00:14:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:09 smithi181 ceph-mon[94189]: pgmap v18303: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 444 KiB/s rd, 485 KiB/s wr, 29 op/s 2023-03-10T00:14:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33361.457285 secs 2023-03-10T00:14:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33361.457285 secs 2023-03-10T00:14:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33361.457285 secs 2023-03-10T00:14:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:11 smithi122 ceph-mon[93869]: pgmap v18304: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 619 KiB/s rd, 660 KiB/s wr, 28 op/s 2023-03-10T00:14:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:11 smithi181 ceph-mon[94189]: pgmap v18304: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 619 KiB/s rd, 660 KiB/s wr, 28 op/s 2023-03-10T00:14:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:11 smithi038 ceph-mon[92422]: pgmap v18304: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 619 KiB/s rd, 660 KiB/s wr, 28 op/s 2023-03-10T00:14:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33363.885833 secs 2023-03-10T00:14:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:13 smithi122 ceph-mon[93869]: osdmap e1360: 12 total, 12 up, 12 in 2023-03-10T00:14:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33363.885833 secs 2023-03-10T00:14:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:13 smithi181 ceph-mon[94189]: osdmap e1360: 12 total, 12 up, 12 in 2023-03-10T00:14:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33363.885833 secs 2023-03-10T00:14:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:13 smithi038 ceph-mon[92422]: osdmap e1360: 12 total, 12 up, 12 in 2023-03-10T00:14:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33825.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:14:14.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:14 smithi122 ceph-mon[93869]: pgmap v18306: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 253 KiB/s wr, 13 op/s 2023-03-10T00:14:14.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:14 smithi181 ceph-mon[94189]: pgmap v18306: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 253 KiB/s wr, 13 op/s 2023-03-10T00:14:14.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:14 smithi038 ceph-mon[92422]: pgmap v18306: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 253 KiB/s wr, 13 op/s 2023-03-10T00:14:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33366.457374 secs 2023-03-10T00:14:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:14:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:14:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:16 smithi122 ceph-mon[93869]: pgmap v18307: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 398 KiB/s rd, 232 KiB/s wr, 12 op/s 2023-03-10T00:14:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33366.457374 secs 2023-03-10T00:14:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:14:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:14:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:16 smithi181 ceph-mon[94189]: pgmap v18307: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 398 KiB/s rd, 232 KiB/s wr, 12 op/s 2023-03-10T00:14:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33366.457374 secs 2023-03-10T00:14:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:14:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:14:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:16 smithi038 ceph-mon[92422]: pgmap v18307: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 398 KiB/s rd, 232 KiB/s wr, 12 op/s 2023-03-10T00:14:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:14:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:14:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:14:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:14:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:14:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:14:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:18 smithi122 ceph-mon[93869]: pgmap v18308: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 177 KiB/s wr, 8 op/s 2023-03-10T00:14:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33368.885922 secs 2023-03-10T00:14:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:18 smithi181 ceph-mon[94189]: pgmap v18308: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 177 KiB/s wr, 8 op/s 2023-03-10T00:14:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33368.885922 secs 2023-03-10T00:14:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:18 smithi038 ceph-mon[92422]: pgmap v18308: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 177 KiB/s wr, 8 op/s 2023-03-10T00:14:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33368.885922 secs 2023-03-10T00:14:18.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33830.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:14:19.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:19 smithi038 ceph-mon[92422]: pgmap v18309: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 177 KiB/s wr, 8 op/s 2023-03-10T00:14:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:19 smithi122 ceph-mon[93869]: pgmap v18309: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 177 KiB/s wr, 8 op/s 2023-03-10T00:14:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:19 smithi181 ceph-mon[94189]: pgmap v18309: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 177 KiB/s wr, 8 op/s 2023-03-10T00:14:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33371.457433 secs 2023-03-10T00:14:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33371.457433 secs 2023-03-10T00:14:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33371.457433 secs 2023-03-10T00:14:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:21 smithi122 ceph-mon[93869]: pgmap v18310: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 124 KiB/s wr, 8 op/s 2023-03-10T00:14:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:21 smithi181 ceph-mon[94189]: pgmap v18310: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 124 KiB/s wr, 8 op/s 2023-03-10T00:14:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:21 smithi038 ceph-mon[92422]: pgmap v18310: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 124 KiB/s wr, 8 op/s 2023-03-10T00:14:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33373.886039 secs 2023-03-10T00:14:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33373.886039 secs 2023-03-10T00:14:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33373.886039 secs 2023-03-10T00:14:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33835.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:14:24.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:24 smithi122 ceph-mon[93869]: pgmap v18311: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 121 KiB/s wr, 7 op/s 2023-03-10T00:14:24.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:24 smithi181 ceph-mon[94189]: pgmap v18311: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 121 KiB/s wr, 7 op/s 2023-03-10T00:14:24.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:24 smithi038 ceph-mon[92422]: pgmap v18311: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 121 KiB/s wr, 7 op/s 2023-03-10T00:14:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33376.457489 secs 2023-03-10T00:14:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:26 smithi038 ceph-mon[92422]: pgmap v18312: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 103 KiB/s wr, 5 op/s 2023-03-10T00:14:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33376.457489 secs 2023-03-10T00:14:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:26 smithi122 ceph-mon[93869]: pgmap v18312: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 103 KiB/s wr, 5 op/s 2023-03-10T00:14:26.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33376.457489 secs 2023-03-10T00:14:26.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:26 smithi181 ceph-mon[94189]: pgmap v18312: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 103 KiB/s wr, 5 op/s 2023-03-10T00:14:28.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33840.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:14:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:28 smithi122 ceph-mon[93869]: pgmap v18313: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 103 KiB/s wr, 8 op/s 2023-03-10T00:14:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33378.886107 secs 2023-03-10T00:14:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:28 smithi181 ceph-mon[94189]: pgmap v18313: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 103 KiB/s wr, 8 op/s 2023-03-10T00:14:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33378.886107 secs 2023-03-10T00:14:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:28 smithi038 ceph-mon[92422]: pgmap v18313: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 103 KiB/s wr, 8 op/s 2023-03-10T00:14:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33378.886107 secs 2023-03-10T00:14:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:30 smithi122 ceph-mon[93869]: pgmap v18314: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 103 KiB/s wr, 5 op/s 2023-03-10T00:14:30.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:30 smithi181 ceph-mon[94189]: pgmap v18314: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 103 KiB/s wr, 5 op/s 2023-03-10T00:14:30.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:30 smithi038 ceph-mon[92422]: pgmap v18314: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 103 KiB/s wr, 5 op/s 2023-03-10T00:14:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33381.457547 secs 2023-03-10T00:14:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33381.457547 secs 2023-03-10T00:14:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33381.457547 secs 2023-03-10T00:14:32.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:32 smithi122 ceph-mon[93869]: pgmap v18315: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 195 KiB/s wr, 7 op/s 2023-03-10T00:14:32.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:32 smithi181 ceph-mon[94189]: pgmap v18315: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 195 KiB/s wr, 7 op/s 2023-03-10T00:14:32.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:32 smithi038 ceph-mon[92422]: pgmap v18315: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 195 KiB/s wr, 7 op/s 2023-03-10T00:14:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33845.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:14:33.975 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33383.886192 secs 2023-03-10T00:14:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33383.886192 secs 2023-03-10T00:14:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33383.886192 secs 2023-03-10T00:14:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:35 smithi038 ceph-mon[92422]: pgmap v18316: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 KiB/s wr, 6 op/s 2023-03-10T00:14:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:35 smithi122 ceph-mon[93869]: pgmap v18316: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 KiB/s wr, 6 op/s 2023-03-10T00:14:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:35 smithi181 ceph-mon[94189]: pgmap v18316: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 KiB/s wr, 6 op/s 2023-03-10T00:14:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33386.457632 secs 2023-03-10T00:14:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33386.457632 secs 2023-03-10T00:14:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33386.457632 secs 2023-03-10T00:14:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:37 smithi038 ceph-mon[92422]: pgmap v18317: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 KiB/s wr, 5 op/s 2023-03-10T00:14:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:37 smithi122 ceph-mon[93869]: pgmap v18317: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 KiB/s wr, 5 op/s 2023-03-10T00:14:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:37 smithi181 ceph-mon[94189]: pgmap v18317: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 KiB/s wr, 5 op/s 2023-03-10T00:14:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33388.886253 secs 2023-03-10T00:14:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33388.886253 secs 2023-03-10T00:14:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33388.886253 secs 2023-03-10T00:14:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33850.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:14:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:39 smithi122 ceph-mon[93869]: pgmap v18318: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 125 KiB/s wr, 9 op/s 2023-03-10T00:14:39.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:39 smithi181 ceph-mon[94189]: pgmap v18318: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 125 KiB/s wr, 9 op/s 2023-03-10T00:14:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:39 smithi038 ceph-mon[92422]: pgmap v18318: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 125 KiB/s wr, 9 op/s 2023-03-10T00:14:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:14:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:14:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:14:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:14:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:14:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:14:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:41 smithi122 ceph-mon[93869]: pgmap v18319: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 125 KiB/s wr, 6 op/s 2023-03-10T00:14:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33391.457715 secs 2023-03-10T00:14:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:14:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:14:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:41 smithi181 ceph-mon[94189]: pgmap v18319: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 125 KiB/s wr, 6 op/s 2023-03-10T00:14:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33391.457715 secs 2023-03-10T00:14:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:14:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:14:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:41 smithi038 ceph-mon[92422]: pgmap v18319: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 125 KiB/s wr, 6 op/s 2023-03-10T00:14:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33391.457715 secs 2023-03-10T00:14:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:14:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:14:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:43 smithi122 ceph-mon[93869]: pgmap v18320: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 125 KiB/s wr, 8 op/s 2023-03-10T00:14:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:43 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 33393.886366 secs 2023-03-10T00:14:43.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:43 smithi181 ceph-mon[94189]: pgmap v18320: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 125 KiB/s wr, 8 op/s 2023-03-10T00:14:43.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:43 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 33393.886366 secs 2023-03-10T00:14:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:43 smithi038 ceph-mon[92422]: pgmap v18320: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 125 KiB/s wr, 8 op/s 2023-03-10T00:14:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:43 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 33393.886366 secs 2023-03-10T00:14:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33855.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:14:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:45 smithi122 ceph-mon[93869]: pgmap v18321: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 164 KiB/s wr, 7 op/s 2023-03-10T00:14:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:14:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:14:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:14:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:45 smithi181 ceph-mon[94189]: pgmap v18321: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 164 KiB/s wr, 7 op/s 2023-03-10T00:14:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:14:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:14:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:14:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:45 smithi038 ceph-mon[92422]: pgmap v18321: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 164 KiB/s wr, 7 op/s 2023-03-10T00:14:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:14:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:14:45.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:14:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33396.457800 secs 2023-03-10T00:14:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:14:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:14:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33396.457800 secs 2023-03-10T00:14:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:14:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:14:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33396.457800 secs 2023-03-10T00:14:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:14:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:14:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:47 smithi122 ceph-mon[93869]: pgmap v18322: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 163 KiB/s wr, 6 op/s 2023-03-10T00:14:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:14:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:14:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:14:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:14:47.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:14:47.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:47 smithi181 ceph-mon[94189]: pgmap v18322: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 163 KiB/s wr, 6 op/s 2023-03-10T00:14:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:14:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:14:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:14:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:14:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:14:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:47 smithi038 ceph-mon[92422]: pgmap v18322: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 163 KiB/s wr, 6 op/s 2023-03-10T00:14:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:14:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:14:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:14:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:14:47.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:14:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:48 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 33398.886433 secs 2023-03-10T00:14:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:48 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 33398.886433 secs 2023-03-10T00:14:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:48 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 33398.886433 secs 2023-03-10T00:14:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33860.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:14:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:49 smithi122 ceph-mon[93869]: pgmap v18323: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 811 KiB/s rd, 342 KiB/s wr, 10 op/s 2023-03-10T00:14:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:49 smithi181 ceph-mon[94189]: pgmap v18323: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 811 KiB/s rd, 342 KiB/s wr, 10 op/s 2023-03-10T00:14:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:49 smithi038 ceph-mon[92422]: pgmap v18323: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 811 KiB/s rd, 342 KiB/s wr, 10 op/s 2023-03-10T00:14:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:51 smithi122 ceph-mon[93869]: pgmap v18324: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 309 KiB/s wr, 6 op/s 2023-03-10T00:14:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33401.457870 secs 2023-03-10T00:14:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:51 smithi181 ceph-mon[94189]: pgmap v18324: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 309 KiB/s wr, 6 op/s 2023-03-10T00:14:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33401.457870 secs 2023-03-10T00:14:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:51 smithi038 ceph-mon[92422]: pgmap v18324: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 309 KiB/s wr, 6 op/s 2023-03-10T00:14:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33401.457870 secs 2023-03-10T00:14:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:53 smithi122 ceph-mon[93869]: pgmap v18325: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 751 KiB/s rd, 651 KiB/s wr, 8 op/s 2023-03-10T00:14:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:14:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:14:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:53 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 33403.886503 secs 2023-03-10T00:14:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:53 smithi181 ceph-mon[94189]: pgmap v18325: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 751 KiB/s rd, 651 KiB/s wr, 8 op/s 2023-03-10T00:14:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:14:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:14:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:53 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 33403.886503 secs 2023-03-10T00:14:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:53 smithi038 ceph-mon[92422]: pgmap v18325: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 751 KiB/s rd, 651 KiB/s wr, 8 op/s 2023-03-10T00:14:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:14:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:14:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:53 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 33403.886503 secs 2023-03-10T00:14:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33865.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:14:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:55 smithi122 ceph-mon[93869]: pgmap v18326: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 922 KiB/s rd, 678 KiB/s wr, 8 op/s 2023-03-10T00:14:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:55 smithi181 ceph-mon[94189]: pgmap v18326: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 922 KiB/s rd, 678 KiB/s wr, 8 op/s 2023-03-10T00:14:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:55 smithi038 ceph-mon[92422]: pgmap v18326: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 922 KiB/s rd, 678 KiB/s wr, 8 op/s 2023-03-10T00:14:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33406.457930 secs 2023-03-10T00:14:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:14:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:14:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33406.457930 secs 2023-03-10T00:14:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:14:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:14:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33406.457930 secs 2023-03-10T00:14:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:14:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:14:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:57 smithi122 ceph-mon[93869]: pgmap v18327: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 842 KiB/s rd, 548 KiB/s wr, 8 op/s 2023-03-10T00:14:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:57 smithi181 ceph-mon[94189]: pgmap v18327: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 842 KiB/s rd, 548 KiB/s wr, 8 op/s 2023-03-10T00:14:57.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:57 smithi038 ceph-mon[92422]: pgmap v18327: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 842 KiB/s rd, 548 KiB/s wr, 8 op/s 2023-03-10T00:14:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33408.886564 secs 2023-03-10T00:14:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33408.886564 secs 2023-03-10T00:14:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33408.886564 secs 2023-03-10T00:14:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33870.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:14:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:59 smithi122 ceph-mon[93869]: pgmap v18328: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1024 KiB/s rd, 727 KiB/s wr, 12 op/s 2023-03-10T00:14:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:14:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:14:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:14:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:59 smithi181 ceph-mon[94189]: pgmap v18328: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1024 KiB/s rd, 727 KiB/s wr, 12 op/s 2023-03-10T00:14:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:14:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:14:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:14:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:59 smithi038 ceph-mon[92422]: pgmap v18328: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1024 KiB/s rd, 727 KiB/s wr, 12 op/s 2023-03-10T00:14:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:14:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:14:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:15:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:15:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:15:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:15:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:15:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:15:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:15:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:01 smithi122 ceph-mon[93869]: pgmap v18329: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 759 KiB/s rd, 549 KiB/s wr, 10 op/s 2023-03-10T00:15:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33411.457980 secs 2023-03-10T00:15:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:15:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:15:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:01 smithi181 ceph-mon[94189]: pgmap v18329: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 759 KiB/s rd, 549 KiB/s wr, 10 op/s 2023-03-10T00:15:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33411.457980 secs 2023-03-10T00:15:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:15:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:15:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:01 smithi038 ceph-mon[92422]: pgmap v18329: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 759 KiB/s rd, 549 KiB/s wr, 10 op/s 2023-03-10T00:15:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33411.457980 secs 2023-03-10T00:15:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:15:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:15:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:03 smithi122 ceph-mon[93869]: pgmap v18330: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 876 KiB/s rd, 666 KiB/s wr, 13 op/s 2023-03-10T00:15:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33413.886678 secs 2023-03-10T00:15:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:03 smithi181 ceph-mon[94189]: pgmap v18330: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 876 KiB/s rd, 666 KiB/s wr, 13 op/s 2023-03-10T00:15:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33413.886678 secs 2023-03-10T00:15:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:03 smithi038 ceph-mon[92422]: pgmap v18330: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 876 KiB/s rd, 666 KiB/s wr, 13 op/s 2023-03-10T00:15:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33413.886678 secs 2023-03-10T00:15:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33875.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:15:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:05 smithi122 ceph-mon[93869]: pgmap v18331: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 756 KiB/s rd, 546 KiB/s wr, 14 op/s 2023-03-10T00:15:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:05 smithi181 ceph-mon[94189]: pgmap v18331: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 756 KiB/s rd, 546 KiB/s wr, 14 op/s 2023-03-10T00:15:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:05 smithi038 ceph-mon[92422]: pgmap v18331: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 756 KiB/s rd, 546 KiB/s wr, 14 op/s 2023-03-10T00:15:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33416.458046 secs 2023-03-10T00:15:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33416.458046 secs 2023-03-10T00:15:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33416.458046 secs 2023-03-10T00:15:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:07 smithi122 ceph-mon[93869]: pgmap v18332: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 584 KiB/s rd, 519 KiB/s wr, 13 op/s 2023-03-10T00:15:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:07 smithi181 ceph-mon[94189]: pgmap v18332: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 584 KiB/s rd, 519 KiB/s wr, 13 op/s 2023-03-10T00:15:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:07 smithi038 ceph-mon[92422]: pgmap v18332: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 584 KiB/s rd, 519 KiB/s wr, 13 op/s 2023-03-10T00:15:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:15:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:15:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33418.886794 secs 2023-03-10T00:15:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:15:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:15:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33418.886794 secs 2023-03-10T00:15:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:15:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:15:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33418.886794 secs 2023-03-10T00:15:08.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33880.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:15:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:09 smithi122 ceph-mon[93869]: pgmap v18333: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 759 KiB/s rd, 766 KiB/s wr, 19 op/s 2023-03-10T00:15:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:15:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:15:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:09 smithi181 ceph-mon[94189]: pgmap v18333: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 759 KiB/s rd, 766 KiB/s wr, 19 op/s 2023-03-10T00:15:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:15:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:15:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:09 smithi038 ceph-mon[92422]: pgmap v18333: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 759 KiB/s rd, 766 KiB/s wr, 19 op/s 2023-03-10T00:15:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:15:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:15:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:10 smithi122 ceph-mon[93869]: osdmap e1361: 12 total, 12 up, 12 in 2023-03-10T00:15:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:10 smithi181 ceph-mon[94189]: osdmap e1361: 12 total, 12 up, 12 in 2023-03-10T00:15:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:10 smithi038 ceph-mon[92422]: osdmap e1361: 12 total, 12 up, 12 in 2023-03-10T00:15:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:11 smithi122 ceph-mon[93869]: pgmap v18335: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 815 KiB/s rd, 856 KiB/s wr, 30 op/s 2023-03-10T00:15:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33421.458104 secs 2023-03-10T00:15:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:11 smithi181 ceph-mon[94189]: pgmap v18335: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 815 KiB/s rd, 856 KiB/s wr, 30 op/s 2023-03-10T00:15:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33421.458104 secs 2023-03-10T00:15:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:11 smithi038 ceph-mon[92422]: pgmap v18335: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 815 KiB/s rd, 856 KiB/s wr, 30 op/s 2023-03-10T00:15:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33421.458104 secs 2023-03-10T00:15:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:15:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:15:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:15:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:15:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:15:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:15:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:13 smithi122 ceph-mon[93869]: pgmap v18336: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 737 KiB/s rd, 778 KiB/s wr, 30 op/s 2023-03-10T00:15:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:15:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:15:13.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33423.886880 secs 2023-03-10T00:15:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:13 smithi181 ceph-mon[94189]: pgmap v18336: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 737 KiB/s rd, 778 KiB/s wr, 30 op/s 2023-03-10T00:15:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:15:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:15:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33423.886880 secs 2023-03-10T00:15:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:13 smithi038 ceph-mon[92422]: pgmap v18336: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 737 KiB/s rd, 778 KiB/s wr, 30 op/s 2023-03-10T00:15:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:15:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:15:13.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33423.886880 secs 2023-03-10T00:15:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33885.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:15:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:15 smithi122 ceph-mon[93869]: pgmap v18337: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 647 KiB/s rd, 689 KiB/s wr, 28 op/s 2023-03-10T00:15:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:15 smithi181 ceph-mon[94189]: pgmap v18337: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 647 KiB/s rd, 689 KiB/s wr, 28 op/s 2023-03-10T00:15:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:15 smithi038 ceph-mon[92422]: pgmap v18337: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 647 KiB/s rd, 689 KiB/s wr, 28 op/s 2023-03-10T00:15:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33426.458197 secs 2023-03-10T00:15:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33426.458197 secs 2023-03-10T00:15:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33426.458197 secs 2023-03-10T00:15:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:17 smithi122 ceph-mon[93869]: pgmap v18338: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 666 KiB/s rd, 708 KiB/s wr, 28 op/s 2023-03-10T00:15:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:17 smithi181 ceph-mon[94189]: pgmap v18338: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 666 KiB/s rd, 708 KiB/s wr, 28 op/s 2023-03-10T00:15:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:17 smithi038 ceph-mon[92422]: pgmap v18338: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 666 KiB/s rd, 708 KiB/s wr, 28 op/s 2023-03-10T00:15:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33428.886939 secs 2023-03-10T00:15:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:18 smithi122 ceph-mon[93869]: osdmap e1362: 12 total, 12 up, 12 in 2023-03-10T00:15:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33428.886939 secs 2023-03-10T00:15:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:18 smithi181 ceph-mon[94189]: osdmap e1362: 12 total, 12 up, 12 in 2023-03-10T00:15:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33428.886939 secs 2023-03-10T00:15:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:18 smithi038 ceph-mon[92422]: osdmap e1362: 12 total, 12 up, 12 in 2023-03-10T00:15:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33890.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:15:19.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:19 smithi122 ceph-mon[93869]: pgmap v18339: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 412 KiB/s wr, 22 op/s 2023-03-10T00:15:19.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:19 smithi181 ceph-mon[94189]: pgmap v18339: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 412 KiB/s wr, 22 op/s 2023-03-10T00:15:19.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:19 smithi038 ceph-mon[92422]: pgmap v18339: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 412 KiB/s wr, 22 op/s 2023-03-10T00:15:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:21 smithi122 ceph-mon[93869]: pgmap v18341: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 364 KiB/s wr, 9 op/s 2023-03-10T00:15:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33431.458285 secs 2023-03-10T00:15:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:21 smithi181 ceph-mon[94189]: pgmap v18341: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 364 KiB/s wr, 9 op/s 2023-03-10T00:15:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33431.458285 secs 2023-03-10T00:15:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:21 smithi038 ceph-mon[92422]: pgmap v18341: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 364 KiB/s wr, 9 op/s 2023-03-10T00:15:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33431.458285 secs 2023-03-10T00:15:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:23 smithi122 ceph-mon[93869]: pgmap v18342: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 301 KiB/s wr, 9 op/s 2023-03-10T00:15:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33433.886997 secs 2023-03-10T00:15:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:23 smithi181 ceph-mon[94189]: pgmap v18342: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 301 KiB/s wr, 9 op/s 2023-03-10T00:15:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33433.886997 secs 2023-03-10T00:15:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:23 smithi038 ceph-mon[92422]: pgmap v18342: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 301 KiB/s wr, 9 op/s 2023-03-10T00:15:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33433.886997 secs 2023-03-10T00:15:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33895.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:15:24.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:24 smithi122 ceph-mon[93869]: pgmap v18343: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 124 KiB/s wr, 7 op/s 2023-03-10T00:15:24.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:24 smithi181 ceph-mon[94189]: pgmap v18343: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 124 KiB/s wr, 7 op/s 2023-03-10T00:15:24.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:24 smithi038 ceph-mon[92422]: pgmap v18343: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 124 KiB/s wr, 7 op/s 2023-03-10T00:15:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33436.458342 secs 2023-03-10T00:15:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:26 smithi122 ceph-mon[93869]: pgmap v18344: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 106 KiB/s wr, 7 op/s 2023-03-10T00:15:26.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33436.458342 secs 2023-03-10T00:15:26.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:26 smithi181 ceph-mon[94189]: pgmap v18344: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 106 KiB/s wr, 7 op/s 2023-03-10T00:15:26.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33436.458342 secs 2023-03-10T00:15:26.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:26 smithi038 ceph-mon[92422]: pgmap v18344: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 106 KiB/s wr, 7 op/s 2023-03-10T00:15:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33900.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:15:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33438.887078 secs 2023-03-10T00:15:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:28 smithi122 ceph-mon[93869]: pgmap v18345: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 105 KiB/s wr, 7 op/s 2023-03-10T00:15:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33438.887078 secs 2023-03-10T00:15:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:28 smithi181 ceph-mon[94189]: pgmap v18345: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 105 KiB/s wr, 7 op/s 2023-03-10T00:15:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33438.887078 secs 2023-03-10T00:15:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:28 smithi038 ceph-mon[92422]: pgmap v18345: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 105 KiB/s wr, 7 op/s 2023-03-10T00:15:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:30 smithi122 ceph-mon[93869]: pgmap v18346: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2023-03-10T00:15:30.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:30 smithi181 ceph-mon[94189]: pgmap v18346: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2023-03-10T00:15:30.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:30 smithi038 ceph-mon[92422]: pgmap v18346: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2023-03-10T00:15:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33441.458424 secs 2023-03-10T00:15:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33441.458424 secs 2023-03-10T00:15:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33441.458424 secs 2023-03-10T00:15:32.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:32 smithi122 ceph-mon[93869]: pgmap v18347: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-10T00:15:32.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:32 smithi181 ceph-mon[94189]: pgmap v18347: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-10T00:15:32.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:32 smithi038 ceph-mon[92422]: pgmap v18347: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-10T00:15:33.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33905.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:15:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33443.887113 secs 2023-03-10T00:15:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33443.887113 secs 2023-03-10T00:15:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33443.887113 secs 2023-03-10T00:15:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:35 smithi122 ceph-mon[93869]: pgmap v18348: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-10T00:15:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:35 smithi181 ceph-mon[94189]: pgmap v18348: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-10T00:15:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:35 smithi038 ceph-mon[92422]: pgmap v18348: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-10T00:15:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33446.458483 secs 2023-03-10T00:15:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33446.458483 secs 2023-03-10T00:15:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33446.458483 secs 2023-03-10T00:15:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:37 smithi122 ceph-mon[93869]: pgmap v18349: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-10T00:15:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:37 smithi181 ceph-mon[94189]: pgmap v18349: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-10T00:15:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:37 smithi038 ceph-mon[92422]: pgmap v18349: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-10T00:15:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33448.887170 secs 2023-03-10T00:15:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33448.887170 secs 2023-03-10T00:15:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33448.887170 secs 2023-03-10T00:15:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33910.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:15:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:39 smithi122 ceph-mon[93869]: pgmap v18350: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 426 B/s wr, 8 op/s 2023-03-10T00:15:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:39 smithi181 ceph-mon[94189]: pgmap v18350: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 426 B/s wr, 8 op/s 2023-03-10T00:15:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:39 smithi038 ceph-mon[92422]: pgmap v18350: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 426 B/s wr, 8 op/s 2023-03-10T00:15:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:15:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:15:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:15:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:15:40.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:15:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:15:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:41 smithi122 ceph-mon[93869]: pgmap v18351: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-10T00:15:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33451.458539 secs 2023-03-10T00:15:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:15:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:15:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:41 smithi181 ceph-mon[94189]: pgmap v18351: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-10T00:15:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33451.458539 secs 2023-03-10T00:15:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:15:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:15:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:41 smithi038 ceph-mon[92422]: pgmap v18351: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-10T00:15:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33451.458539 secs 2023-03-10T00:15:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:15:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:15:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:43 smithi122 ceph-mon[93869]: pgmap v18352: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 131 KiB/s wr, 8 op/s 2023-03-10T00:15:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:43 smithi181 ceph-mon[94189]: pgmap v18352: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 131 KiB/s wr, 8 op/s 2023-03-10T00:15:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:43 smithi038 ceph-mon[92422]: pgmap v18352: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 131 KiB/s wr, 8 op/s 2023-03-10T00:15:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33915.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:15:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:45 smithi122 ceph-mon[93869]: pgmap v18353: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 132 KiB/s wr, 7 op/s 2023-03-10T00:15:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:45 smithi181 ceph-mon[94189]: pgmap v18353: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 132 KiB/s wr, 7 op/s 2023-03-10T00:15:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:45 smithi038 ceph-mon[92422]: pgmap v18353: 129 pgs: 129 active+clean; 12 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 132 KiB/s wr, 7 op/s 2023-03-10T00:15:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33456.458598 secs 2023-03-10T00:15:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:46 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:15:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33456.458598 secs 2023-03-10T00:15:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:46 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:15:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33456.458598 secs 2023-03-10T00:15:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:46 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:15:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:47 smithi122 ceph-mon[93869]: pgmap v18354: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 329 KiB/s rd, 327 KiB/s wr, 7 op/s 2023-03-10T00:15:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:47 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:15:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:15:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:15:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:15:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:47 smithi181 ceph-mon[94189]: pgmap v18354: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 329 KiB/s rd, 327 KiB/s wr, 7 op/s 2023-03-10T00:15:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:47 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:15:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:15:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:15:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:15:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:47 smithi038 ceph-mon[92422]: pgmap v18354: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 329 KiB/s rd, 327 KiB/s wr, 7 op/s 2023-03-10T00:15:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:47 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:15:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:15:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:15:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:15:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:15:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:15:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:15:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:15:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:15:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:15:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:15:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:15:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:15:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33920.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:15:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:49 smithi122 ceph-mon[93869]: pgmap v18355: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 327 KiB/s wr, 9 op/s 2023-03-10T00:15:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:49 smithi181 ceph-mon[94189]: pgmap v18355: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 327 KiB/s wr, 9 op/s 2023-03-10T00:15:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:49 smithi038 ceph-mon[92422]: pgmap v18355: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 327 KiB/s wr, 9 op/s 2023-03-10T00:15:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:51 smithi122 ceph-mon[93869]: pgmap v18356: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 329 KiB/s rd, 327 KiB/s wr, 7 op/s 2023-03-10T00:15:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33461.458683 secs 2023-03-10T00:15:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:51 smithi181 ceph-mon[94189]: pgmap v18356: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 329 KiB/s rd, 327 KiB/s wr, 7 op/s 2023-03-10T00:15:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33461.458683 secs 2023-03-10T00:15:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:51 smithi038 ceph-mon[92422]: pgmap v18356: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 329 KiB/s rd, 327 KiB/s wr, 7 op/s 2023-03-10T00:15:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33461.458683 secs 2023-03-10T00:15:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:53 smithi122 ceph-mon[93869]: pgmap v18357: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 975 KiB/s rd, 642 KiB/s wr, 9 op/s 2023-03-10T00:15:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:15:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:15:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:53 smithi181 ceph-mon[94189]: pgmap v18357: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 975 KiB/s rd, 642 KiB/s wr, 9 op/s 2023-03-10T00:15:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:15:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:15:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:53 smithi038 ceph-mon[92422]: pgmap v18357: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 975 KiB/s rd, 642 KiB/s wr, 9 op/s 2023-03-10T00:15:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:15:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:15:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33925.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:15:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:55 smithi122 ceph-mon[93869]: pgmap v18358: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 869 KiB/s rd, 538 KiB/s wr, 8 op/s 2023-03-10T00:15:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:55 smithi181 ceph-mon[94189]: pgmap v18358: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 869 KiB/s rd, 538 KiB/s wr, 8 op/s 2023-03-10T00:15:55.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:55 smithi038 ceph-mon[92422]: pgmap v18358: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 869 KiB/s rd, 538 KiB/s wr, 8 op/s 2023-03-10T00:15:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33466.458741 secs 2023-03-10T00:15:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:15:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:15:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33466.458741 secs 2023-03-10T00:15:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:15:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:15:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33466.458741 secs 2023-03-10T00:15:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:15:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:15:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:57 smithi122 ceph-mon[93869]: pgmap v18359: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 895 KiB/s rd, 564 KiB/s wr, 8 op/s 2023-03-10T00:15:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:57 smithi181 ceph-mon[94189]: pgmap v18359: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 895 KiB/s rd, 564 KiB/s wr, 8 op/s 2023-03-10T00:15:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:57 smithi038 ceph-mon[92422]: pgmap v18359: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 895 KiB/s rd, 564 KiB/s wr, 8 op/s 2023-03-10T00:15:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:58 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 33468.887503 secs 2023-03-10T00:15:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:58 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 33468.887503 secs 2023-03-10T00:15:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:58 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 33468.887503 secs 2023-03-10T00:15:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33930.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:15:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:59 smithi122 ceph-mon[93869]: pgmap v18360: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 867 KiB/s rd, 534 KiB/s wr, 11 op/s 2023-03-10T00:15:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:59 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:15:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:15:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:15:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:15:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:59 smithi181 ceph-mon[94189]: pgmap v18360: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 867 KiB/s rd, 534 KiB/s wr, 11 op/s 2023-03-10T00:15:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:59 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:15:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:15:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:15:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:15:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:59 smithi038 ceph-mon[92422]: pgmap v18360: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 867 KiB/s rd, 534 KiB/s wr, 11 op/s 2023-03-10T00:15:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:59 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:15:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:15:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:15:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:16:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:16:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:16:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:16:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:16:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:16:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:16:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:01 smithi122 ceph-mon[93869]: pgmap v18361: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 877 KiB/s rd, 534 KiB/s wr, 9 op/s 2023-03-10T00:16:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33471.458799 secs 2023-03-10T00:16:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:16:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:16:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:01 smithi181 ceph-mon[94189]: pgmap v18361: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 877 KiB/s rd, 534 KiB/s wr, 9 op/s 2023-03-10T00:16:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33471.458799 secs 2023-03-10T00:16:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:16:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:16:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:01 smithi038 ceph-mon[92422]: pgmap v18361: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 877 KiB/s rd, 534 KiB/s wr, 9 op/s 2023-03-10T00:16:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33471.458799 secs 2023-03-10T00:16:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:16:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:16:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:03 smithi122 ceph-mon[93869]: pgmap v18362: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 742 KiB/s wr, 13 op/s 2023-03-10T00:16:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:03 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 33473.887566 secs 2023-03-10T00:16:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:03 smithi181 ceph-mon[94189]: pgmap v18362: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 742 KiB/s wr, 13 op/s 2023-03-10T00:16:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:03 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 33473.887566 secs 2023-03-10T00:16:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:03 smithi038 ceph-mon[92422]: pgmap v18362: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 742 KiB/s wr, 13 op/s 2023-03-10T00:16:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:03 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 33473.887566 secs 2023-03-10T00:16:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33935.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:16:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:05 smithi122 ceph-mon[93869]: pgmap v18363: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 558 KiB/s wr, 13 op/s 2023-03-10T00:16:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:05 smithi181 ceph-mon[94189]: pgmap v18363: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 558 KiB/s wr, 13 op/s 2023-03-10T00:16:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:05 smithi038 ceph-mon[92422]: pgmap v18363: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 558 KiB/s wr, 13 op/s 2023-03-10T00:16:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33476.458858 secs 2023-03-10T00:16:06.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33476.458858 secs 2023-03-10T00:16:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33476.458858 secs 2023-03-10T00:16:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:07 smithi122 ceph-mon[93869]: pgmap v18364: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 545 KiB/s rd, 531 KiB/s wr, 13 op/s 2023-03-10T00:16:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:07 smithi181 ceph-mon[94189]: pgmap v18364: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 545 KiB/s rd, 531 KiB/s wr, 13 op/s 2023-03-10T00:16:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:07 smithi038 ceph-mon[92422]: pgmap v18364: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 545 KiB/s rd, 531 KiB/s wr, 13 op/s 2023-03-10T00:16:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:16:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:16:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33478.887626 secs 2023-03-10T00:16:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:16:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:16:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33478.887626 secs 2023-03-10T00:16:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:16:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:16:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33478.887626 secs 2023-03-10T00:16:08.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33940.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:16:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:09 smithi122 ceph-mon[93869]: pgmap v18365: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 620 KiB/s rd, 627 KiB/s wr, 18 op/s 2023-03-10T00:16:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:16:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:16:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:09 smithi181 ceph-mon[94189]: pgmap v18365: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 620 KiB/s rd, 627 KiB/s wr, 18 op/s 2023-03-10T00:16:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:16:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:16:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:09 smithi038 ceph-mon[92422]: pgmap v18365: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 620 KiB/s rd, 627 KiB/s wr, 18 op/s 2023-03-10T00:16:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:16:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:16:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:10 smithi122 ceph-mon[93869]: osdmap e1363: 12 total, 12 up, 12 in 2023-03-10T00:16:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:10 smithi181 ceph-mon[94189]: osdmap e1363: 12 total, 12 up, 12 in 2023-03-10T00:16:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:10 smithi038 ceph-mon[92422]: osdmap e1363: 12 total, 12 up, 12 in 2023-03-10T00:16:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:11 smithi122 ceph-mon[93869]: pgmap v18367: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 539 KiB/s rd, 580 KiB/s wr, 30 op/s 2023-03-10T00:16:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33481.458915 secs 2023-03-10T00:16:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:11 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:16:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:11 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:16:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:11 smithi181 ceph-mon[94189]: pgmap v18367: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 539 KiB/s rd, 580 KiB/s wr, 30 op/s 2023-03-10T00:16:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33481.458915 secs 2023-03-10T00:16:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:11 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:16:11.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:11 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:16:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:11 smithi038 ceph-mon[92422]: pgmap v18367: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 539 KiB/s rd, 580 KiB/s wr, 30 op/s 2023-03-10T00:16:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33481.458915 secs 2023-03-10T00:16:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:11 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:16:11.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:11 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:16:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:13 smithi122 ceph-mon[93869]: pgmap v18368: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 469 KiB/s wr, 29 op/s 2023-03-10T00:16:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33483.887686 secs 2023-03-10T00:16:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:13 smithi181 ceph-mon[94189]: pgmap v18368: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 469 KiB/s wr, 29 op/s 2023-03-10T00:16:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33483.887686 secs 2023-03-10T00:16:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:13 smithi038 ceph-mon[92422]: pgmap v18368: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 469 KiB/s wr, 29 op/s 2023-03-10T00:16:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33483.887686 secs 2023-03-10T00:16:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33945.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:16:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:15 smithi122 ceph-mon[93869]: pgmap v18369: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 540 KiB/s wr, 27 op/s 2023-03-10T00:16:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:15 smithi181 ceph-mon[94189]: pgmap v18369: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 540 KiB/s wr, 27 op/s 2023-03-10T00:16:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:15 smithi038 ceph-mon[92422]: pgmap v18369: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 540 KiB/s wr, 27 op/s 2023-03-10T00:16:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33486.458999 secs 2023-03-10T00:16:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:16:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:16:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:16 smithi122 ceph-mon[93869]: pgmap v18370: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 540 KiB/s wr, 27 op/s 2023-03-10T00:16:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33486.458999 secs 2023-03-10T00:16:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:16:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:16:16.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:16 smithi181 ceph-mon[94189]: pgmap v18370: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 540 KiB/s wr, 27 op/s 2023-03-10T00:16:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33486.458999 secs 2023-03-10T00:16:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:16:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:16:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:16 smithi038 ceph-mon[92422]: pgmap v18370: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 540 KiB/s wr, 27 op/s 2023-03-10T00:16:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:16:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:16:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:16:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:16:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:16:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:16:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33488.887774 secs 2023-03-10T00:16:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:18 smithi038 ceph-mon[92422]: pgmap v18371: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 503 KiB/s rd, 518 KiB/s wr, 23 op/s 2023-03-10T00:16:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:18 smithi038 ceph-mon[92422]: osdmap e1364: 12 total, 12 up, 12 in 2023-03-10T00:16:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33950.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:16:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33488.887774 secs 2023-03-10T00:16:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:18 smithi122 ceph-mon[93869]: pgmap v18371: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 503 KiB/s rd, 518 KiB/s wr, 23 op/s 2023-03-10T00:16:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:18 smithi122 ceph-mon[93869]: osdmap e1364: 12 total, 12 up, 12 in 2023-03-10T00:16:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33488.887774 secs 2023-03-10T00:16:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:18 smithi181 ceph-mon[94189]: pgmap v18371: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 503 KiB/s rd, 518 KiB/s wr, 23 op/s 2023-03-10T00:16:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:18 smithi181 ceph-mon[94189]: osdmap e1364: 12 total, 12 up, 12 in 2023-03-10T00:16:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:20 smithi122 ceph-mon[93869]: pgmap v18373: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 493 KiB/s wr, 10 op/s 2023-03-10T00:16:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33491.459057 secs 2023-03-10T00:16:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:20 smithi181 ceph-mon[94189]: pgmap v18373: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 493 KiB/s wr, 10 op/s 2023-03-10T00:16:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33491.459057 secs 2023-03-10T00:16:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:20 smithi038 ceph-mon[92422]: pgmap v18373: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 493 KiB/s wr, 10 op/s 2023-03-10T00:16:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33491.459057 secs 2023-03-10T00:16:22.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:22 smithi122 ceph-mon[93869]: pgmap v18374: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 353 KiB/s wr, 9 op/s 2023-03-10T00:16:22.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:22 smithi181 ceph-mon[94189]: pgmap v18374: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 353 KiB/s wr, 9 op/s 2023-03-10T00:16:22.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:22 smithi038 ceph-mon[92422]: pgmap v18374: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 353 KiB/s wr, 9 op/s 2023-03-10T00:16:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33955.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:16:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33493.887850 secs 2023-03-10T00:16:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33493.887850 secs 2023-03-10T00:16:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33493.887850 secs 2023-03-10T00:16:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:25 smithi122 ceph-mon[93869]: pgmap v18375: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 125 KiB/s wr, 7 op/s 2023-03-10T00:16:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:25 smithi181 ceph-mon[94189]: pgmap v18375: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 125 KiB/s wr, 7 op/s 2023-03-10T00:16:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:25 smithi038 ceph-mon[92422]: pgmap v18375: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 125 KiB/s wr, 7 op/s 2023-03-10T00:16:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33496.459125 secs 2023-03-10T00:16:26.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33496.459125 secs 2023-03-10T00:16:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33496.459125 secs 2023-03-10T00:16:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:27 smithi122 ceph-mon[93869]: pgmap v18376: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 125 KiB/s wr, 7 op/s 2023-03-10T00:16:27.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:27 smithi181 ceph-mon[94189]: pgmap v18376: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 125 KiB/s wr, 7 op/s 2023-03-10T00:16:27.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:27 smithi038 ceph-mon[92422]: pgmap v18376: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 125 KiB/s wr, 7 op/s 2023-03-10T00:16:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33498.887968 secs 2023-03-10T00:16:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33498.887968 secs 2023-03-10T00:16:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33498.887968 secs 2023-03-10T00:16:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33960.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:16:29.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:29 smithi122 ceph-mon[93869]: pgmap v18377: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-10T00:16:29.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:29 smithi181 ceph-mon[94189]: pgmap v18377: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-10T00:16:29.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:29 smithi038 ceph-mon[92422]: pgmap v18377: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-10T00:16:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:31 smithi122 ceph-mon[93869]: pgmap v18378: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2023-03-10T00:16:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33501.459196 secs 2023-03-10T00:16:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:31 smithi181 ceph-mon[94189]: pgmap v18378: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2023-03-10T00:16:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33501.459196 secs 2023-03-10T00:16:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:31 smithi038 ceph-mon[92422]: pgmap v18378: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2023-03-10T00:16:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33501.459196 secs 2023-03-10T00:16:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:33 smithi122 ceph-mon[93869]: pgmap v18379: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-10T00:16:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33503.888038 secs 2023-03-10T00:16:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:33 smithi181 ceph-mon[94189]: pgmap v18379: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-10T00:16:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33503.888038 secs 2023-03-10T00:16:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:33 smithi038 ceph-mon[92422]: pgmap v18379: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-10T00:16:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33503.888038 secs 2023-03-10T00:16:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33965.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:16:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:35 smithi122 ceph-mon[93869]: pgmap v18380: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-10T00:16:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:35 smithi181 ceph-mon[94189]: pgmap v18380: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-10T00:16:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:35 smithi038 ceph-mon[92422]: pgmap v18380: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-10T00:16:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33506.459258 secs 2023-03-10T00:16:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33506.459258 secs 2023-03-10T00:16:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33506.459258 secs 2023-03-10T00:16:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:37 smithi122 ceph-mon[93869]: pgmap v18381: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-10T00:16:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:37 smithi181 ceph-mon[94189]: pgmap v18381: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-10T00:16:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:37 smithi038 ceph-mon[92422]: pgmap v18381: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-10T00:16:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33508.888099 secs 2023-03-10T00:16:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33508.888099 secs 2023-03-10T00:16:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33508.888099 secs 2023-03-10T00:16:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33970.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:16:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:39 smithi122 ceph-mon[93869]: pgmap v18382: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 122 KiB/s wr, 9 op/s 2023-03-10T00:16:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:39 smithi181 ceph-mon[94189]: pgmap v18382: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 122 KiB/s wr, 9 op/s 2023-03-10T00:16:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:39 smithi038 ceph-mon[92422]: pgmap v18382: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 122 KiB/s wr, 9 op/s 2023-03-10T00:16:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:16:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:16:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:16:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:16:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:16:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:16:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:41 smithi122 ceph-mon[93869]: pgmap v18383: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 257 KiB/s wr, 7 op/s 2023-03-10T00:16:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33511.459345 secs 2023-03-10T00:16:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:16:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:16:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:41 smithi181 ceph-mon[94189]: pgmap v18383: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 257 KiB/s wr, 7 op/s 2023-03-10T00:16:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33511.459345 secs 2023-03-10T00:16:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:16:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:16:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:41 smithi038 ceph-mon[92422]: pgmap v18383: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 257 KiB/s wr, 7 op/s 2023-03-10T00:16:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33511.459345 secs 2023-03-10T00:16:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:16:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:16:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:43 smithi122 ceph-mon[93869]: pgmap v18384: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 626 KiB/s rd, 257 KiB/s wr, 9 op/s 2023-03-10T00:16:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:43 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 33513.888187 secs 2023-03-10T00:16:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:43 smithi181 ceph-mon[94189]: pgmap v18384: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 626 KiB/s rd, 257 KiB/s wr, 9 op/s 2023-03-10T00:16:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:43 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 33513.888187 secs 2023-03-10T00:16:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:43 smithi038 ceph-mon[92422]: pgmap v18384: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 626 KiB/s rd, 257 KiB/s wr, 9 op/s 2023-03-10T00:16:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:43 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 33513.888187 secs 2023-03-10T00:16:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33975.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:16:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:45 smithi122 ceph-mon[93869]: pgmap v18385: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 257 KiB/s wr, 8 op/s 2023-03-10T00:16:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:45 smithi181 ceph-mon[94189]: pgmap v18385: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 257 KiB/s wr, 8 op/s 2023-03-10T00:16:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:45 smithi038 ceph-mon[92422]: pgmap v18385: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 257 KiB/s wr, 8 op/s 2023-03-10T00:16:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33516.459443 secs 2023-03-10T00:16:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33516.459443 secs 2023-03-10T00:16:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33516.459443 secs 2023-03-10T00:16:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:47 smithi122 ceph-mon[93869]: pgmap v18386: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 257 KiB/s wr, 8 op/s 2023-03-10T00:16:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:16:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:16:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:47 smithi181 ceph-mon[94189]: pgmap v18386: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 257 KiB/s wr, 8 op/s 2023-03-10T00:16:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:16:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:16:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:47 smithi038 ceph-mon[92422]: pgmap v18386: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 257 KiB/s wr, 8 op/s 2023-03-10T00:16:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:16:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:16:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33518.888247 secs 2023-03-10T00:16:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:16:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33518.888247 secs 2023-03-10T00:16:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:16:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33518.888247 secs 2023-03-10T00:16:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:16:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33980.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:16:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:49 smithi122 ceph-mon[93869]: pgmap v18387: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 659 KiB/s rd, 289 KiB/s wr, 12 op/s 2023-03-10T00:16:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:16:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:16:49.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:16:49.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:16:49.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:16:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:49 smithi181 ceph-mon[94189]: pgmap v18387: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 659 KiB/s rd, 289 KiB/s wr, 12 op/s 2023-03-10T00:16:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:16:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:16:49.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:16:49.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:16:49.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:16:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:49 smithi038 ceph-mon[92422]: pgmap v18387: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 659 KiB/s rd, 289 KiB/s wr, 12 op/s 2023-03-10T00:16:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:16:49.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:16:49.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:16:49.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:16:49.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:16:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:51 smithi122 ceph-mon[93869]: pgmap v18388: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 445 KiB/s rd, 500 KiB/s wr, 11 op/s 2023-03-10T00:16:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33521.459525 secs 2023-03-10T00:16:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:16:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:16:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:51 smithi181 ceph-mon[94189]: pgmap v18388: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 445 KiB/s rd, 500 KiB/s wr, 11 op/s 2023-03-10T00:16:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33521.459525 secs 2023-03-10T00:16:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:16:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:16:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:51 smithi038 ceph-mon[92422]: pgmap v18388: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 445 KiB/s rd, 500 KiB/s wr, 11 op/s 2023-03-10T00:16:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33521.459525 secs 2023-03-10T00:16:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:16:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:16:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:16:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:16:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:16:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:16:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:16:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:16:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:53 smithi122 ceph-mon[93869]: pgmap v18389: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 366 KiB/s wr, 13 op/s 2023-03-10T00:16:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33523.888309 secs 2023-03-10T00:16:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:53 smithi181 ceph-mon[94189]: pgmap v18389: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 366 KiB/s wr, 13 op/s 2023-03-10T00:16:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33523.888309 secs 2023-03-10T00:16:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:53 smithi038 ceph-mon[92422]: pgmap v18389: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 366 KiB/s wr, 13 op/s 2023-03-10T00:16:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33523.888309 secs 2023-03-10T00:16:53.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33985.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:16:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:55 smithi122 ceph-mon[93869]: pgmap v18390: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 622 KiB/s rd, 679 KiB/s wr, 13 op/s 2023-03-10T00:16:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:55 smithi181 ceph-mon[94189]: pgmap v18390: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 622 KiB/s rd, 679 KiB/s wr, 13 op/s 2023-03-10T00:16:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:55 smithi038 ceph-mon[92422]: pgmap v18390: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 622 KiB/s rd, 679 KiB/s wr, 13 op/s 2023-03-10T00:16:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33526.459586 secs 2023-03-10T00:16:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:16:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:16:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33526.459586 secs 2023-03-10T00:16:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:16:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:16:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33526.459586 secs 2023-03-10T00:16:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:16:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:16:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:57 smithi122 ceph-mon[93869]: pgmap v18391: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 622 KiB/s rd, 678 KiB/s wr, 12 op/s 2023-03-10T00:16:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:16:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:16:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:57 smithi181 ceph-mon[94189]: pgmap v18391: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 622 KiB/s rd, 678 KiB/s wr, 12 op/s 2023-03-10T00:16:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:16:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:16:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:57 smithi038 ceph-mon[92422]: pgmap v18391: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 622 KiB/s rd, 678 KiB/s wr, 12 op/s 2023-03-10T00:16:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:16:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:16:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33528.888413 secs 2023-03-10T00:16:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33528.888413 secs 2023-03-10T00:16:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33528.888413 secs 2023-03-10T00:16:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33990.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:16:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:16:59 smithi122 ceph-mon[93869]: pgmap v18392: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 773 KiB/s wr, 16 op/s 2023-03-10T00:16:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:16:59 smithi181 ceph-mon[94189]: pgmap v18392: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 773 KiB/s wr, 16 op/s 2023-03-10T00:16:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:16:59 smithi038 ceph-mon[92422]: pgmap v18392: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 773 KiB/s wr, 16 op/s 2023-03-10T00:17:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:01 smithi122 ceph-mon[93869]: pgmap v18393: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 684 KiB/s rd, 742 KiB/s wr, 13 op/s 2023-03-10T00:17:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33531.459643 secs 2023-03-10T00:17:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:01 smithi181 ceph-mon[94189]: pgmap v18393: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 684 KiB/s rd, 742 KiB/s wr, 13 op/s 2023-03-10T00:17:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33531.459643 secs 2023-03-10T00:17:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:01 smithi038 ceph-mon[92422]: pgmap v18393: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 684 KiB/s rd, 742 KiB/s wr, 13 op/s 2023-03-10T00:17:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33531.459643 secs 2023-03-10T00:17:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:03 smithi122 ceph-mon[93869]: pgmap v18394: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 431 KiB/s wr, 14 op/s 2023-03-10T00:17:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33533.888476 secs 2023-03-10T00:17:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:03 smithi181 ceph-mon[94189]: pgmap v18394: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 431 KiB/s wr, 14 op/s 2023-03-10T00:17:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33533.888476 secs 2023-03-10T00:17:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:03 smithi038 ceph-mon[92422]: pgmap v18394: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 411 KiB/s rd, 431 KiB/s wr, 14 op/s 2023-03-10T00:17:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33533.888476 secs 2023-03-10T00:17:03.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 33995.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:17:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:05 smithi122 ceph-mon[93869]: pgmap v18395: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 719 KiB/s rd, 754 KiB/s wr, 26 op/s 2023-03-10T00:17:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:05 smithi122 ceph-mon[93869]: osdmap e1365: 12 total, 12 up, 12 in 2023-03-10T00:17:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:05 smithi181 ceph-mon[94189]: pgmap v18395: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 719 KiB/s rd, 754 KiB/s wr, 26 op/s 2023-03-10T00:17:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:05 smithi181 ceph-mon[94189]: osdmap e1365: 12 total, 12 up, 12 in 2023-03-10T00:17:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:05 smithi038 ceph-mon[92422]: pgmap v18395: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 719 KiB/s rd, 754 KiB/s wr, 26 op/s 2023-03-10T00:17:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:05 smithi038 ceph-mon[92422]: osdmap e1365: 12 total, 12 up, 12 in 2023-03-10T00:17:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33536.459714 secs 2023-03-10T00:17:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33536.459714 secs 2023-03-10T00:17:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33536.459714 secs 2023-03-10T00:17:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:07 smithi122 ceph-mon[93869]: pgmap v18397: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 650 KiB/s rd, 691 KiB/s wr, 29 op/s 2023-03-10T00:17:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:07 smithi181 ceph-mon[94189]: pgmap v18397: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 650 KiB/s rd, 691 KiB/s wr, 29 op/s 2023-03-10T00:17:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:07 smithi038 ceph-mon[92422]: pgmap v18397: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 650 KiB/s rd, 691 KiB/s wr, 29 op/s 2023-03-10T00:17:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33538.888591 secs 2023-03-10T00:17:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33538.888591 secs 2023-03-10T00:17:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33538.888591 secs 2023-03-10T00:17:08.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34000.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:17:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:09 smithi038 ceph-mon[92422]: pgmap v18398: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 604 KiB/s wr, 28 op/s 2023-03-10T00:17:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:09 smithi122 ceph-mon[93869]: pgmap v18398: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 604 KiB/s wr, 28 op/s 2023-03-10T00:17:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:09 smithi181 ceph-mon[94189]: pgmap v18398: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 604 KiB/s wr, 28 op/s 2023-03-10T00:17:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:10 smithi038 ceph-mon[92422]: pgmap v18399: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 604 KiB/s wr, 28 op/s 2023-03-10T00:17:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:10 smithi122 ceph-mon[93869]: pgmap v18399: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 604 KiB/s wr, 28 op/s 2023-03-10T00:17:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:10 smithi181 ceph-mon[94189]: pgmap v18399: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 604 KiB/s wr, 28 op/s 2023-03-10T00:17:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33541.459789 secs 2023-03-10T00:17:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33541.459789 secs 2023-03-10T00:17:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33541.459789 secs 2023-03-10T00:17:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:12 smithi122 ceph-mon[93869]: pgmap v18400: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 739 KiB/s rd, 754 KiB/s wr, 27 op/s 2023-03-10T00:17:12.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:12 smithi181 ceph-mon[94189]: pgmap v18400: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 739 KiB/s rd, 754 KiB/s wr, 27 op/s 2023-03-10T00:17:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:12 smithi038 ceph-mon[92422]: pgmap v18400: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 739 KiB/s rd, 754 KiB/s wr, 27 op/s 2023-03-10T00:17:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34005.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:17:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33543.888672 secs 2023-03-10T00:17:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:17:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:17:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:13 smithi122 ceph-mon[93869]: osdmap e1366: 12 total, 12 up, 12 in 2023-03-10T00:17:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33543.888672 secs 2023-03-10T00:17:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:17:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:17:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:13 smithi181 ceph-mon[94189]: osdmap e1366: 12 total, 12 up, 12 in 2023-03-10T00:17:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33543.888672 secs 2023-03-10T00:17:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:17:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:17:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:13 smithi038 ceph-mon[92422]: osdmap e1366: 12 total, 12 up, 12 in 2023-03-10T00:17:14.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:14 smithi122 ceph-mon[93869]: pgmap v18402: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 404 KiB/s wr, 11 op/s 2023-03-10T00:17:14.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:14 smithi181 ceph-mon[94189]: pgmap v18402: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 404 KiB/s wr, 11 op/s 2023-03-10T00:17:14.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:14 smithi038 ceph-mon[92422]: pgmap v18402: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 404 KiB/s wr, 11 op/s 2023-03-10T00:17:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33546.459874 secs 2023-03-10T00:17:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:16 smithi122 ceph-mon[93869]: pgmap v18403: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 205 KiB/s wr, 9 op/s 2023-03-10T00:17:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:17:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:17:16.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33546.459874 secs 2023-03-10T00:17:16.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:16 smithi181 ceph-mon[94189]: pgmap v18403: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 205 KiB/s wr, 9 op/s 2023-03-10T00:17:16.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:17:16.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:17:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33546.459874 secs 2023-03-10T00:17:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:16 smithi038 ceph-mon[92422]: pgmap v18403: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 205 KiB/s wr, 9 op/s 2023-03-10T00:17:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:17:17.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:17:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34010.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:17:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33548.888731 secs 2023-03-10T00:17:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:18 smithi122 ceph-mon[93869]: pgmap v18404: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 178 KiB/s wr, 8 op/s 2023-03-10T00:17:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33548.888731 secs 2023-03-10T00:17:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:18 smithi181 ceph-mon[94189]: pgmap v18404: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 178 KiB/s wr, 8 op/s 2023-03-10T00:17:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33548.888731 secs 2023-03-10T00:17:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:18 smithi038 ceph-mon[92422]: pgmap v18404: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 178 KiB/s wr, 8 op/s 2023-03-10T00:17:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:20 smithi122 ceph-mon[93869]: pgmap v18405: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 178 KiB/s wr, 7 op/s 2023-03-10T00:17:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33551.459959 secs 2023-03-10T00:17:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:20 smithi181 ceph-mon[94189]: pgmap v18405: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 178 KiB/s wr, 7 op/s 2023-03-10T00:17:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33551.459959 secs 2023-03-10T00:17:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:20 smithi038 ceph-mon[92422]: pgmap v18405: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 178 KiB/s wr, 7 op/s 2023-03-10T00:17:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33551.459959 secs 2023-03-10T00:17:22.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:22 smithi122 ceph-mon[93869]: pgmap v18406: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 125 KiB/s wr, 7 op/s 2023-03-10T00:17:22.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:22 smithi181 ceph-mon[94189]: pgmap v18406: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 125 KiB/s wr, 7 op/s 2023-03-10T00:17:22.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:22 smithi038 ceph-mon[92422]: pgmap v18406: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 125 KiB/s wr, 7 op/s 2023-03-10T00:17:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34015.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:17:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33553.888834 secs 2023-03-10T00:17:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33553.888834 secs 2023-03-10T00:17:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33553.888834 secs 2023-03-10T00:17:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:25 smithi122 ceph-mon[93869]: pgmap v18407: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 122 KiB/s wr, 6 op/s 2023-03-10T00:17:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:25 smithi181 ceph-mon[94189]: pgmap v18407: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 122 KiB/s wr, 6 op/s 2023-03-10T00:17:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:25 smithi038 ceph-mon[92422]: pgmap v18407: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 125 KiB/s rd, 122 KiB/s wr, 6 op/s 2023-03-10T00:17:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33556.459984 secs 2023-03-10T00:17:26.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33556.459984 secs 2023-03-10T00:17:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33556.459984 secs 2023-03-10T00:17:27.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:27 smithi122 ceph-mon[93869]: pgmap v18408: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 104 KiB/s wr, 5 op/s 2023-03-10T00:17:27.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:27 smithi181 ceph-mon[94189]: pgmap v18408: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 104 KiB/s wr, 5 op/s 2023-03-10T00:17:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:27 smithi038 ceph-mon[92422]: pgmap v18408: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 104 KiB/s wr, 5 op/s 2023-03-10T00:17:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33558.888945 secs 2023-03-10T00:17:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33558.888945 secs 2023-03-10T00:17:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33558.888945 secs 2023-03-10T00:17:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34020.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:17:29.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:29 smithi122 ceph-mon[93869]: pgmap v18409: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 104 KiB/s wr, 8 op/s 2023-03-10T00:17:29.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:29 smithi181 ceph-mon[94189]: pgmap v18409: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 104 KiB/s wr, 8 op/s 2023-03-10T00:17:29.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:29 smithi038 ceph-mon[92422]: pgmap v18409: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 104 KiB/s wr, 8 op/s 2023-03-10T00:17:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:31 smithi122 ceph-mon[93869]: pgmap v18410: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 104 KiB/s wr, 6 op/s 2023-03-10T00:17:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33561.460043 secs 2023-03-10T00:17:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:31 smithi181 ceph-mon[94189]: pgmap v18410: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 104 KiB/s wr, 6 op/s 2023-03-10T00:17:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33561.460043 secs 2023-03-10T00:17:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:31 smithi038 ceph-mon[92422]: pgmap v18410: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 106 KiB/s rd, 104 KiB/s wr, 6 op/s 2023-03-10T00:17:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33561.460043 secs 2023-03-10T00:17:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:33 smithi122 ceph-mon[93869]: pgmap v18411: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 104 KiB/s wr, 8 op/s 2023-03-10T00:17:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33563.889005 secs 2023-03-10T00:17:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:33 smithi181 ceph-mon[94189]: pgmap v18411: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 104 KiB/s wr, 8 op/s 2023-03-10T00:17:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33563.889005 secs 2023-03-10T00:17:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:33 smithi038 ceph-mon[92422]: pgmap v18411: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 104 KiB/s wr, 8 op/s 2023-03-10T00:17:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33563.889005 secs 2023-03-10T00:17:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34025.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:17:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:35 smithi122 ceph-mon[93869]: pgmap v18412: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-10T00:17:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:35 smithi181 ceph-mon[94189]: pgmap v18412: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-10T00:17:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:35 smithi038 ceph-mon[92422]: pgmap v18412: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-10T00:17:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33566.460115 secs 2023-03-10T00:17:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33566.460115 secs 2023-03-10T00:17:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33566.460115 secs 2023-03-10T00:17:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:37 smithi122 ceph-mon[93869]: pgmap v18413: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-10T00:17:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:37 smithi181 ceph-mon[94189]: pgmap v18413: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-10T00:17:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:37 smithi038 ceph-mon[92422]: pgmap v18413: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-10T00:17:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33568.889063 secs 2023-03-10T00:17:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33568.889063 secs 2023-03-10T00:17:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33568.889063 secs 2023-03-10T00:17:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34030.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:17:39.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:39 smithi122 ceph-mon[93869]: pgmap v18414: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 8 op/s 2023-03-10T00:17:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:39 smithi181 ceph-mon[94189]: pgmap v18414: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 8 op/s 2023-03-10T00:17:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:39 smithi038 ceph-mon[92422]: pgmap v18414: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 8 op/s 2023-03-10T00:17:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:41 smithi122 ceph-mon[93869]: pgmap v18415: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-10T00:17:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33571.460177 secs 2023-03-10T00:17:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:41 smithi181 ceph-mon[94189]: pgmap v18415: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-10T00:17:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33571.460177 secs 2023-03-10T00:17:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:41 smithi038 ceph-mon[92422]: pgmap v18415: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-10T00:17:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33571.460177 secs 2023-03-10T00:17:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:43 smithi122 ceph-mon[93869]: pgmap v18416: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-10T00:17:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33573.889117 secs 2023-03-10T00:17:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:43 smithi181 ceph-mon[94189]: pgmap v18416: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-10T00:17:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33573.889117 secs 2023-03-10T00:17:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:43 smithi038 ceph-mon[92422]: pgmap v18416: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-10T00:17:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33573.889117 secs 2023-03-10T00:17:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34035.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:17:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:45 smithi122 ceph-mon[93869]: pgmap v18417: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-10T00:17:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:45 smithi181 ceph-mon[94189]: pgmap v18417: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-10T00:17:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:45 smithi038 ceph-mon[92422]: pgmap v18417: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-10T00:17:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33576.460261 secs 2023-03-10T00:17:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33576.460261 secs 2023-03-10T00:17:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33576.460261 secs 2023-03-10T00:17:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:47 smithi122 ceph-mon[93869]: pgmap v18418: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-10T00:17:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:47 smithi181 ceph-mon[94189]: pgmap v18418: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-10T00:17:47.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:47 smithi038 ceph-mon[92422]: pgmap v18418: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-10T00:17:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33578.889199 secs 2023-03-10T00:17:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33578.889199 secs 2023-03-10T00:17:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33578.889199 secs 2023-03-10T00:17:48.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34040.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:17:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:49 smithi122 ceph-mon[93869]: pgmap v18419: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 128 KiB/s wr, 9 op/s 2023-03-10T00:17:49.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:17:49.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:17:49.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:17:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:49 smithi181 ceph-mon[94189]: pgmap v18419: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 128 KiB/s wr, 9 op/s 2023-03-10T00:17:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:17:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:17:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:17:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:49 smithi038 ceph-mon[92422]: pgmap v18419: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 128 KiB/s wr, 9 op/s 2023-03-10T00:17:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:17:49.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:17:49.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:17:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:50 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:17:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:50 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:17:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:50 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:17:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:50 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:17:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:50 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:17:50.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:50 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:17:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:50 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:17:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:50 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:17:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:50 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:17:51.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:51 smithi122 ceph-mon[93869]: pgmap v18420: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 128 KiB/s wr, 6 op/s 2023-03-10T00:17:51.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33581.460350 secs 2023-03-10T00:17:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:51 smithi181 ceph-mon[94189]: pgmap v18420: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 128 KiB/s wr, 6 op/s 2023-03-10T00:17:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33581.460350 secs 2023-03-10T00:17:51.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:51 smithi038 ceph-mon[92422]: pgmap v18420: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 128 KiB/s wr, 6 op/s 2023-03-10T00:17:51.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33581.460350 secs 2023-03-10T00:17:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:17:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:17:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:52 smithi122 ceph-mon[93869]: pgmap v18421: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 791 KiB/s rd, 786 KiB/s wr, 11 op/s 2023-03-10T00:17:52.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:17:52.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:17:52.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:52 smithi181 ceph-mon[94189]: pgmap v18421: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 791 KiB/s rd, 786 KiB/s wr, 11 op/s 2023-03-10T00:17:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:17:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:17:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:52 smithi038 ceph-mon[92422]: pgmap v18421: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 791 KiB/s rd, 786 KiB/s wr, 11 op/s 2023-03-10T00:17:53.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34045.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:17:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:17:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:17:53.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:17:53.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:17:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:17:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:17:54.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:54 smithi122 ceph-mon[93869]: pgmap v18422: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 844 KiB/s rd, 840 KiB/s wr, 10 op/s 2023-03-10T00:17:54.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:54 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:17:54.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:54 smithi181 ceph-mon[94189]: pgmap v18422: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 844 KiB/s rd, 840 KiB/s wr, 10 op/s 2023-03-10T00:17:54.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:54 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:17:54.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:54 smithi038 ceph-mon[92422]: pgmap v18422: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 844 KiB/s rd, 840 KiB/s wr, 10 op/s 2023-03-10T00:17:54.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:54 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:17:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:55 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:17:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:17:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:17:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:55 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:17:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:17:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:17:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:55 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:17:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:17:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:17:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33586.460409 secs 2023-03-10T00:17:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:17:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:17:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:56 smithi122 ceph-mon[93869]: pgmap v18423: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 855 KiB/s rd, 840 KiB/s wr, 10 op/s 2023-03-10T00:17:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33586.460409 secs 2023-03-10T00:17:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:17:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:17:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:56 smithi181 ceph-mon[94189]: pgmap v18423: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 855 KiB/s rd, 840 KiB/s wr, 10 op/s 2023-03-10T00:17:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33586.460409 secs 2023-03-10T00:17:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:17:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:17:57.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:56 smithi038 ceph-mon[92422]: pgmap v18423: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 855 KiB/s rd, 840 KiB/s wr, 10 op/s 2023-03-10T00:17:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:17:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:17:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:17:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:17:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:17:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:17:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34050.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:17:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:17:59 smithi122 ceph-mon[93869]: pgmap v18424: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 14 op/s 2023-03-10T00:17:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:17:59 smithi181 ceph-mon[94189]: pgmap v18424: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 14 op/s 2023-03-10T00:17:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:17:59 smithi038 ceph-mon[92422]: pgmap v18424: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 14 op/s 2023-03-10T00:18:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:18:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:18:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:18:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:18:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:18:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:18:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:01 smithi122 ceph-mon[93869]: pgmap v18425: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 950 KiB/s rd, 1.1 MiB/s wr, 11 op/s 2023-03-10T00:18:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33591.460493 secs 2023-03-10T00:18:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:18:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:18:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:01 smithi181 ceph-mon[94189]: pgmap v18425: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 950 KiB/s rd, 1.1 MiB/s wr, 11 op/s 2023-03-10T00:18:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33591.460493 secs 2023-03-10T00:18:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:18:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:18:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:01 smithi038 ceph-mon[92422]: pgmap v18425: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 950 KiB/s rd, 1.1 MiB/s wr, 11 op/s 2023-03-10T00:18:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33591.460493 secs 2023-03-10T00:18:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:18:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:18:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:03 smithi122 ceph-mon[93869]: pgmap v18426: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 16 op/s 2023-03-10T00:18:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33593.889432 secs 2023-03-10T00:18:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:03 smithi181 ceph-mon[94189]: pgmap v18426: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 16 op/s 2023-03-10T00:18:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33593.889432 secs 2023-03-10T00:18:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:03 smithi038 ceph-mon[92422]: pgmap v18426: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 16 op/s 2023-03-10T00:18:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33593.889432 secs 2023-03-10T00:18:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34055.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:18:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:05 smithi122 ceph-mon[93869]: pgmap v18427: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 918 KiB/s rd, 732 KiB/s wr, 27 op/s 2023-03-10T00:18:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:05 smithi181 ceph-mon[94189]: pgmap v18427: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 918 KiB/s rd, 732 KiB/s wr, 27 op/s 2023-03-10T00:18:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:05 smithi038 ceph-mon[92422]: pgmap v18427: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 918 KiB/s rd, 732 KiB/s wr, 27 op/s 2023-03-10T00:18:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33596.460549 secs 2023-03-10T00:18:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33596.460549 secs 2023-03-10T00:18:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33596.460549 secs 2023-03-10T00:18:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:07 smithi122 ceph-mon[93869]: pgmap v18428: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 678 KiB/s wr, 25 op/s 2023-03-10T00:18:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:07 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:18:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:18:07.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:18:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:07 smithi181 ceph-mon[94189]: pgmap v18428: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 678 KiB/s wr, 25 op/s 2023-03-10T00:18:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:07 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:18:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:18:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:18:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:07 smithi038 ceph-mon[92422]: pgmap v18428: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 678 KiB/s wr, 25 op/s 2023-03-10T00:18:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:07 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:18:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:18:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:18:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:18:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:18:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33598.889527 secs 2023-03-10T00:18:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:18:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:18:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33598.889527 secs 2023-03-10T00:18:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:18:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:18:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33598.889527 secs 2023-03-10T00:18:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34060.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:18:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:09 smithi122 ceph-mon[93869]: pgmap v18429: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 856 KiB/s rd, 690 KiB/s wr, 29 op/s 2023-03-10T00:18:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:18:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:18:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:09 smithi181 ceph-mon[94189]: pgmap v18429: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 856 KiB/s rd, 690 KiB/s wr, 29 op/s 2023-03-10T00:18:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:18:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:18:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:09 smithi038 ceph-mon[92422]: pgmap v18429: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 856 KiB/s rd, 690 KiB/s wr, 29 op/s 2023-03-10T00:18:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:18:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:18:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:10 smithi122 ceph-mon[93869]: osdmap e1367: 12 total, 12 up, 12 in 2023-03-10T00:18:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:10 smithi181 ceph-mon[94189]: osdmap e1367: 12 total, 12 up, 12 in 2023-03-10T00:18:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:10 smithi038 ceph-mon[92422]: osdmap e1367: 12 total, 12 up, 12 in 2023-03-10T00:18:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:11 smithi122 ceph-mon[93869]: pgmap v18431: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 759 KiB/s rd, 299 KiB/s wr, 29 op/s 2023-03-10T00:18:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33601.460635 secs 2023-03-10T00:18:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:11 smithi181 ceph-mon[94189]: pgmap v18431: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 759 KiB/s rd, 299 KiB/s wr, 29 op/s 2023-03-10T00:18:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33601.460635 secs 2023-03-10T00:18:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:11 smithi038 ceph-mon[92422]: pgmap v18431: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 759 KiB/s rd, 299 KiB/s wr, 29 op/s 2023-03-10T00:18:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33601.460635 secs 2023-03-10T00:18:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:13 smithi122 ceph-mon[93869]: pgmap v18432: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 828 KiB/s rd, 595 KiB/s wr, 26 op/s 2023-03-10T00:18:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33603.889607 secs 2023-03-10T00:18:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:13 smithi181 ceph-mon[94189]: pgmap v18432: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 828 KiB/s rd, 595 KiB/s wr, 26 op/s 2023-03-10T00:18:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33603.889607 secs 2023-03-10T00:18:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:13 smithi038 ceph-mon[92422]: pgmap v18432: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 828 KiB/s rd, 595 KiB/s wr, 26 op/s 2023-03-10T00:18:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33603.889607 secs 2023-03-10T00:18:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34065.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:18:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:15 smithi122 ceph-mon[93869]: pgmap v18433: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 354 KiB/s wr, 10 op/s 2023-03-10T00:18:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:15 smithi181 ceph-mon[94189]: pgmap v18433: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 354 KiB/s wr, 10 op/s 2023-03-10T00:18:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:15 smithi038 ceph-mon[92422]: pgmap v18433: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 354 KiB/s wr, 10 op/s 2023-03-10T00:18:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33606.460733 secs 2023-03-10T00:18:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33606.460733 secs 2023-03-10T00:18:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33606.460733 secs 2023-03-10T00:18:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:17 smithi122 ceph-mon[93869]: pgmap v18434: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 437 KiB/s wr, 10 op/s 2023-03-10T00:18:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:17 smithi181 ceph-mon[94189]: pgmap v18434: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 437 KiB/s wr, 10 op/s 2023-03-10T00:18:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:17 smithi038 ceph-mon[92422]: pgmap v18434: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 437 KiB/s wr, 10 op/s 2023-03-10T00:18:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33608.889674 secs 2023-03-10T00:18:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:18 smithi122 ceph-mon[93869]: osdmap e1368: 12 total, 12 up, 12 in 2023-03-10T00:18:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33608.889674 secs 2023-03-10T00:18:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:18 smithi181 ceph-mon[94189]: osdmap e1368: 12 total, 12 up, 12 in 2023-03-10T00:18:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33608.889674 secs 2023-03-10T00:18:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:18 smithi038 ceph-mon[92422]: osdmap e1368: 12 total, 12 up, 12 in 2023-03-10T00:18:18.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34070.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:18:19.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:19 smithi122 ceph-mon[93869]: pgmap v18435: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 467 KiB/s rd, 464 KiB/s wr, 8 op/s 2023-03-10T00:18:19.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:19 smithi181 ceph-mon[94189]: pgmap v18435: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 467 KiB/s rd, 464 KiB/s wr, 8 op/s 2023-03-10T00:18:19.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:19 smithi038 ceph-mon[92422]: pgmap v18435: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 467 KiB/s rd, 464 KiB/s wr, 8 op/s 2023-03-10T00:18:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:21 smithi122 ceph-mon[93869]: pgmap v18437: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 467 KiB/s rd, 464 KiB/s wr, 8 op/s 2023-03-10T00:18:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33611.460796 secs 2023-03-10T00:18:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:21 smithi181 ceph-mon[94189]: pgmap v18437: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 467 KiB/s rd, 464 KiB/s wr, 8 op/s 2023-03-10T00:18:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33611.460796 secs 2023-03-10T00:18:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:21 smithi038 ceph-mon[92422]: pgmap v18437: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 467 KiB/s rd, 464 KiB/s wr, 8 op/s 2023-03-10T00:18:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33611.460796 secs 2023-03-10T00:18:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:23 smithi122 ceph-mon[93869]: pgmap v18438: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 125 KiB/s wr, 8 op/s 2023-03-10T00:18:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33613.889733 secs 2023-03-10T00:18:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:23 smithi181 ceph-mon[94189]: pgmap v18438: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 125 KiB/s wr, 8 op/s 2023-03-10T00:18:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33613.889733 secs 2023-03-10T00:18:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:23 smithi038 ceph-mon[92422]: pgmap v18438: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 125 KiB/s wr, 8 op/s 2023-03-10T00:18:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33613.889733 secs 2023-03-10T00:18:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34075.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:18:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:25 smithi122 ceph-mon[93869]: pgmap v18439: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 125 KiB/s wr, 7 op/s 2023-03-10T00:18:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:25 smithi181 ceph-mon[94189]: pgmap v18439: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 125 KiB/s wr, 7 op/s 2023-03-10T00:18:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:25 smithi038 ceph-mon[92422]: pgmap v18439: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 125 KiB/s wr, 7 op/s 2023-03-10T00:18:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33616.460856 secs 2023-03-10T00:18:26.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33616.460856 secs 2023-03-10T00:18:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33616.460856 secs 2023-03-10T00:18:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:27 smithi122 ceph-mon[93869]: pgmap v18440: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 42 KiB/s wr, 7 op/s 2023-03-10T00:18:27.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:27 smithi181 ceph-mon[94189]: pgmap v18440: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 42 KiB/s wr, 7 op/s 2023-03-10T00:18:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:27 smithi038 ceph-mon[92422]: pgmap v18440: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 42 KiB/s wr, 7 op/s 2023-03-10T00:18:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33618.889798 secs 2023-03-10T00:18:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33618.889798 secs 2023-03-10T00:18:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33618.889798 secs 2023-03-10T00:18:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34080.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:18:29.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:29 smithi122 ceph-mon[93869]: pgmap v18441: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 7 op/s 2023-03-10T00:18:29.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:29 smithi181 ceph-mon[94189]: pgmap v18441: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 7 op/s 2023-03-10T00:18:29.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:29 smithi038 ceph-mon[92422]: pgmap v18441: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 7 op/s 2023-03-10T00:18:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:31 smithi181 ceph-mon[94189]: pgmap v18442: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2023-03-10T00:18:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33621.460913 secs 2023-03-10T00:18:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:31 smithi038 ceph-mon[92422]: pgmap v18442: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2023-03-10T00:18:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33621.460913 secs 2023-03-10T00:18:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:31 smithi122 ceph-mon[93869]: pgmap v18442: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2023-03-10T00:18:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33621.460913 secs 2023-03-10T00:18:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:33 smithi038 ceph-mon[92422]: pgmap v18443: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-10T00:18:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33623.889861 secs 2023-03-10T00:18:33.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34085.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:18:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:33 smithi122 ceph-mon[93869]: pgmap v18443: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-10T00:18:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33623.889861 secs 2023-03-10T00:18:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:33 smithi181 ceph-mon[94189]: pgmap v18443: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-10T00:18:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33623.889861 secs 2023-03-10T00:18:34.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:34 smithi122 ceph-mon[93869]: pgmap v18444: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-10T00:18:34.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:34 smithi181 ceph-mon[94189]: pgmap v18444: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-10T00:18:34.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:34 smithi038 ceph-mon[92422]: pgmap v18444: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-10T00:18:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33626.460999 secs 2023-03-10T00:18:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:36 smithi122 ceph-mon[93869]: pgmap v18445: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-10T00:18:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33626.460999 secs 2023-03-10T00:18:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:36 smithi181 ceph-mon[94189]: pgmap v18445: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-10T00:18:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33626.460999 secs 2023-03-10T00:18:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:36 smithi038 ceph-mon[92422]: pgmap v18445: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-10T00:18:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34090.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:18:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33628.889923 secs 2023-03-10T00:18:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:38 smithi122 ceph-mon[93869]: pgmap v18446: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 426 B/s wr, 8 op/s 2023-03-10T00:18:38.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33628.889923 secs 2023-03-10T00:18:38.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:38 smithi181 ceph-mon[94189]: pgmap v18446: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 426 B/s wr, 8 op/s 2023-03-10T00:18:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33628.889923 secs 2023-03-10T00:18:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:38 smithi038 ceph-mon[92422]: pgmap v18446: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 426 B/s wr, 8 op/s 2023-03-10T00:18:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:40 smithi122 ceph-mon[93869]: pgmap v18447: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-10T00:18:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:18:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:18:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33631.461063 secs 2023-03-10T00:18:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:40 smithi181 ceph-mon[94189]: pgmap v18447: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-10T00:18:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:18:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:18:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33631.461063 secs 2023-03-10T00:18:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:40 smithi038 ceph-mon[92422]: pgmap v18447: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-10T00:18:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:18:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:18:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33631.461063 secs 2023-03-10T00:18:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:18:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:18:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:18:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:18:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:18:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:18:42.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:42 smithi122 ceph-mon[93869]: pgmap v18448: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 131 KiB/s wr, 8 op/s 2023-03-10T00:18:42.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:42 smithi181 ceph-mon[94189]: pgmap v18448: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 131 KiB/s wr, 8 op/s 2023-03-10T00:18:42.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:42 smithi038 ceph-mon[92422]: pgmap v18448: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 131 KiB/s wr, 8 op/s 2023-03-10T00:18:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34095.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:18:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:43 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 33633.889983 secs 2023-03-10T00:18:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:43 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 33633.889983 secs 2023-03-10T00:18:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:43 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 33633.889983 secs 2023-03-10T00:18:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:45 smithi122 ceph-mon[93869]: pgmap v18449: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 131 KiB/s wr, 7 op/s 2023-03-10T00:18:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:45 smithi181 ceph-mon[94189]: pgmap v18449: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 131 KiB/s wr, 7 op/s 2023-03-10T00:18:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:45 smithi038 ceph-mon[92422]: pgmap v18449: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 131 KiB/s wr, 7 op/s 2023-03-10T00:18:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33636.461122 secs 2023-03-10T00:18:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33636.461122 secs 2023-03-10T00:18:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33636.461122 secs 2023-03-10T00:18:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:47 smithi122 ceph-mon[93869]: pgmap v18450: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 273 KiB/s wr, 7 op/s 2023-03-10T00:18:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:18:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:18:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:47 smithi181 ceph-mon[94189]: pgmap v18450: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 273 KiB/s wr, 7 op/s 2023-03-10T00:18:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:18:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:18:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:47 smithi038 ceph-mon[92422]: pgmap v18450: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 273 KiB/s wr, 7 op/s 2023-03-10T00:18:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:18:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:18:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:48 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 33638.890041 secs 2023-03-10T00:18:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:48 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 33638.890041 secs 2023-03-10T00:18:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:48 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 33638.890041 secs 2023-03-10T00:18:48.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34100.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:18:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:49 smithi122 ceph-mon[93869]: pgmap v18451: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 273 KiB/s wr, 9 op/s 2023-03-10T00:18:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:49 smithi181 ceph-mon[94189]: pgmap v18451: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 273 KiB/s wr, 9 op/s 2023-03-10T00:18:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:49 smithi038 ceph-mon[92422]: pgmap v18451: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 273 KiB/s wr, 9 op/s 2023-03-10T00:18:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:50 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:18:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:50 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:18:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:50 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:18:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:51 smithi122 ceph-mon[93869]: pgmap v18452: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 297 KiB/s wr, 7 op/s 2023-03-10T00:18:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33641.461190 secs 2023-03-10T00:18:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:18:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:18:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:18:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:51 smithi181 ceph-mon[94189]: pgmap v18452: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 297 KiB/s wr, 7 op/s 2023-03-10T00:18:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33641.461190 secs 2023-03-10T00:18:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:18:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:18:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:18:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:51 smithi038 ceph-mon[92422]: pgmap v18452: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 297 KiB/s wr, 7 op/s 2023-03-10T00:18:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33641.461190 secs 2023-03-10T00:18:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:18:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:18:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:18:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:53 smithi122 ceph-mon[93869]: pgmap v18453: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 690 KiB/s rd, 638 KiB/s wr, 10 op/s 2023-03-10T00:18:54.435 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:18:54.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:18:54.436 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:53 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 33643.890111 secs 2023-03-10T00:18:54.437 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:53 smithi038 ceph-mon[92422]: pgmap v18453: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 690 KiB/s rd, 638 KiB/s wr, 10 op/s 2023-03-10T00:18:54.437 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:18:54.437 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:18:54.437 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:53 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 33643.890111 secs 2023-03-10T00:18:54.437 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:53 smithi181 ceph-mon[94189]: pgmap v18453: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 690 KiB/s rd, 638 KiB/s wr, 10 op/s 2023-03-10T00:18:54.438 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:18:54.438 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:18:54.438 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:53 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 33643.890111 secs 2023-03-10T00:18:54.438 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34105.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:18:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:55 smithi122 ceph-mon[93869]: pgmap v18454: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 585 KiB/s rd, 535 KiB/s wr, 8 op/s 2023-03-10T00:18:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:55 smithi181 ceph-mon[94189]: pgmap v18454: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 585 KiB/s rd, 535 KiB/s wr, 8 op/s 2023-03-10T00:18:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:55 smithi038 ceph-mon[92422]: pgmap v18454: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 585 KiB/s rd, 535 KiB/s wr, 8 op/s 2023-03-10T00:18:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33646.461260 secs 2023-03-10T00:18:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:18:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:18:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33646.461260 secs 2023-03-10T00:18:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:18:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:18:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33646.461260 secs 2023-03-10T00:18:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:18:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:18:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:57 smithi122 ceph-mon[93869]: pgmap v18455: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 609 KiB/s rd, 602 KiB/s wr, 9 op/s 2023-03-10T00:18:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:18:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:18:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:57 smithi181 ceph-mon[94189]: pgmap v18455: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 609 KiB/s rd, 602 KiB/s wr, 9 op/s 2023-03-10T00:18:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:18:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:18:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:57 smithi038 ceph-mon[92422]: pgmap v18455: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 609 KiB/s rd, 602 KiB/s wr, 9 op/s 2023-03-10T00:18:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:18:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:18:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33648.890170 secs 2023-03-10T00:18:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33648.890170 secs 2023-03-10T00:18:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33648.890170 secs 2023-03-10T00:18:58.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34110.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:18:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:59 smithi122 ceph-mon[93869]: pgmap v18456: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 505 KiB/s rd, 497 KiB/s wr, 12 op/s 2023-03-10T00:18:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:18:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:18:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:18:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:59 smithi181 ceph-mon[94189]: pgmap v18456: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 505 KiB/s rd, 497 KiB/s wr, 12 op/s 2023-03-10T00:18:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:18:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:18:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:18:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:59 smithi038 ceph-mon[92422]: pgmap v18456: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 505 KiB/s rd, 497 KiB/s wr, 12 op/s 2023-03-10T00:18:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:18:59.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:18:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:19:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:19:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:19:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:19:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:19:00.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:19:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:19:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:01 smithi122 ceph-mon[93869]: pgmap v18457: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 833 KiB/s rd, 754 KiB/s wr, 12 op/s 2023-03-10T00:19:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33651.461342 secs 2023-03-10T00:19:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:19:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:19:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:01 smithi181 ceph-mon[94189]: pgmap v18457: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 833 KiB/s rd, 754 KiB/s wr, 12 op/s 2023-03-10T00:19:01.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33651.461342 secs 2023-03-10T00:19:01.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:19:01.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:19:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:01 smithi038 ceph-mon[92422]: pgmap v18457: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 833 KiB/s rd, 754 KiB/s wr, 12 op/s 2023-03-10T00:19:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33651.461342 secs 2023-03-10T00:19:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:19:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:19:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:03 smithi122 ceph-mon[93869]: pgmap v18458: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 834 KiB/s rd, 731 KiB/s wr, 13 op/s 2023-03-10T00:19:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33653.890230 secs 2023-03-10T00:19:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:03 smithi181 ceph-mon[94189]: pgmap v18458: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 834 KiB/s rd, 731 KiB/s wr, 13 op/s 2023-03-10T00:19:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33653.890230 secs 2023-03-10T00:19:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:03 smithi038 ceph-mon[92422]: pgmap v18458: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 834 KiB/s rd, 731 KiB/s wr, 13 op/s 2023-03-10T00:19:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33653.890230 secs 2023-03-10T00:19:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34115.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:19:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:05 smithi122 ceph-mon[93869]: pgmap v18459: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 759 KiB/s wr, 26 op/s 2023-03-10T00:19:05.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:05 smithi181 ceph-mon[94189]: pgmap v18459: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 759 KiB/s wr, 26 op/s 2023-03-10T00:19:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:05 smithi038 ceph-mon[92422]: pgmap v18459: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 759 KiB/s wr, 26 op/s 2023-03-10T00:19:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33656.461401 secs 2023-03-10T00:19:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33656.461401 secs 2023-03-10T00:19:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33656.461401 secs 2023-03-10T00:19:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:07 smithi122 ceph-mon[93869]: pgmap v18460: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 662 KiB/s rd, 782 KiB/s wr, 26 op/s 2023-03-10T00:19:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:07 smithi181 ceph-mon[94189]: pgmap v18460: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 662 KiB/s rd, 782 KiB/s wr, 26 op/s 2023-03-10T00:19:07.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:07 smithi038 ceph-mon[92422]: pgmap v18460: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 662 KiB/s rd, 782 KiB/s wr, 26 op/s 2023-03-10T00:19:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:19:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:19:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33658.890314 secs 2023-03-10T00:19:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:19:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:19:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33658.890314 secs 2023-03-10T00:19:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:19:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:19:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33658.890314 secs 2023-03-10T00:19:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34120.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:19:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:09 smithi122 ceph-mon[93869]: pgmap v18461: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 799 KiB/s rd, 894 KiB/s wr, 30 op/s 2023-03-10T00:19:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:19:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:19:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:09 smithi181 ceph-mon[94189]: pgmap v18461: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 799 KiB/s rd, 894 KiB/s wr, 30 op/s 2023-03-10T00:19:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:19:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:19:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:09 smithi038 ceph-mon[92422]: pgmap v18461: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 799 KiB/s rd, 894 KiB/s wr, 30 op/s 2023-03-10T00:19:09.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:19:09.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:19:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:10 smithi122 ceph-mon[93869]: osdmap e1369: 12 total, 12 up, 12 in 2023-03-10T00:19:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:10 smithi181 ceph-mon[94189]: osdmap e1369: 12 total, 12 up, 12 in 2023-03-10T00:19:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:10 smithi038 ceph-mon[92422]: osdmap e1369: 12 total, 12 up, 12 in 2023-03-10T00:19:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:11 smithi122 ceph-mon[93869]: pgmap v18463: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 688 KiB/s rd, 909 KiB/s wr, 31 op/s 2023-03-10T00:19:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33661.461463 secs 2023-03-10T00:19:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:11 smithi181 ceph-mon[94189]: pgmap v18463: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 688 KiB/s rd, 909 KiB/s wr, 31 op/s 2023-03-10T00:19:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33661.461463 secs 2023-03-10T00:19:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:11 smithi038 ceph-mon[92422]: pgmap v18463: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 688 KiB/s rd, 909 KiB/s wr, 31 op/s 2023-03-10T00:19:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33661.461463 secs 2023-03-10T00:19:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:13 smithi122 ceph-mon[93869]: pgmap v18464: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 814 KiB/s rd, 1.0 MiB/s wr, 31 op/s 2023-03-10T00:19:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33663.890422 secs 2023-03-10T00:19:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:13 smithi181 ceph-mon[94189]: pgmap v18464: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 814 KiB/s rd, 1.0 MiB/s wr, 31 op/s 2023-03-10T00:19:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33663.890422 secs 2023-03-10T00:19:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:13 smithi038 ceph-mon[92422]: pgmap v18464: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 814 KiB/s rd, 1.0 MiB/s wr, 31 op/s 2023-03-10T00:19:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33663.890422 secs 2023-03-10T00:19:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34125.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:19:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:15 smithi122 ceph-mon[93869]: pgmap v18465: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 642 KiB/s wr, 14 op/s 2023-03-10T00:19:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:15 smithi181 ceph-mon[94189]: pgmap v18465: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 642 KiB/s wr, 14 op/s 2023-03-10T00:19:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:15 smithi038 ceph-mon[92422]: pgmap v18465: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 642 KiB/s wr, 14 op/s 2023-03-10T00:19:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33666.461521 secs 2023-03-10T00:19:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:19:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:19:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33666.461521 secs 2023-03-10T00:19:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:19:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:19:16.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33666.461521 secs 2023-03-10T00:19:16.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:19:16.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:19:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:17 smithi038 ceph-mon[92422]: pgmap v18466: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 730 KiB/s rd, 582 KiB/s wr, 13 op/s 2023-03-10T00:19:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:19:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:19:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:17 smithi122 ceph-mon[93869]: pgmap v18466: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 730 KiB/s rd, 582 KiB/s wr, 13 op/s 2023-03-10T00:19:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:19:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:19:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:17 smithi181 ceph-mon[94189]: pgmap v18466: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 730 KiB/s rd, 582 KiB/s wr, 13 op/s 2023-03-10T00:19:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:19:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:19:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34130.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:19:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33668.890480 secs 2023-03-10T00:19:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:18 smithi122 ceph-mon[93869]: osdmap e1370: 12 total, 12 up, 12 in 2023-03-10T00:19:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33668.890480 secs 2023-03-10T00:19:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:18 smithi181 ceph-mon[94189]: osdmap e1370: 12 total, 12 up, 12 in 2023-03-10T00:19:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33668.890480 secs 2023-03-10T00:19:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:18 smithi038 ceph-mon[92422]: osdmap e1370: 12 total, 12 up, 12 in 2023-03-10T00:19:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:19 smithi122 ceph-mon[93869]: pgmap v18467: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 664 KiB/s rd, 492 KiB/s wr, 10 op/s 2023-03-10T00:19:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:19 smithi181 ceph-mon[94189]: pgmap v18467: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 664 KiB/s rd, 492 KiB/s wr, 10 op/s 2023-03-10T00:19:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:19 smithi038 ceph-mon[92422]: pgmap v18467: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 664 KiB/s rd, 492 KiB/s wr, 10 op/s 2023-03-10T00:19:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:20 smithi122 ceph-mon[93869]: pgmap v18469: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 494 KiB/s rd, 303 KiB/s wr, 9 op/s 2023-03-10T00:19:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:20 smithi181 ceph-mon[94189]: pgmap v18469: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 494 KiB/s rd, 303 KiB/s wr, 9 op/s 2023-03-10T00:19:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:20 smithi038 ceph-mon[92422]: pgmap v18469: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 494 KiB/s rd, 303 KiB/s wr, 9 op/s 2023-03-10T00:19:21.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33671.461583 secs 2023-03-10T00:19:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33671.461583 secs 2023-03-10T00:19:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33671.461583 secs 2023-03-10T00:19:22.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:22 smithi122 ceph-mon[93869]: pgmap v18470: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 177 KiB/s wr, 8 op/s 2023-03-10T00:19:22.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:22 smithi181 ceph-mon[94189]: pgmap v18470: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 177 KiB/s wr, 8 op/s 2023-03-10T00:19:22.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:22 smithi038 ceph-mon[92422]: pgmap v18470: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 177 KiB/s wr, 8 op/s 2023-03-10T00:19:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34135.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:19:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33673.890557 secs 2023-03-10T00:19:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33673.890557 secs 2023-03-10T00:19:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33673.890557 secs 2023-03-10T00:19:24.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:24 smithi122 ceph-mon[93869]: pgmap v18471: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 125 KiB/s wr, 7 op/s 2023-03-10T00:19:24.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:24 smithi181 ceph-mon[94189]: pgmap v18471: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 125 KiB/s wr, 7 op/s 2023-03-10T00:19:24.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:24 smithi038 ceph-mon[92422]: pgmap v18471: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 125 KiB/s wr, 7 op/s 2023-03-10T00:19:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33676.461637 secs 2023-03-10T00:19:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33676.461637 secs 2023-03-10T00:19:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33676.461637 secs 2023-03-10T00:19:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:26 smithi122 ceph-mon[93869]: pgmap v18472: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 125 KiB/s wr, 7 op/s 2023-03-10T00:19:26.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:26 smithi181 ceph-mon[94189]: pgmap v18472: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 125 KiB/s wr, 7 op/s 2023-03-10T00:19:26.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:26 smithi038 ceph-mon[92422]: pgmap v18472: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 125 KiB/s wr, 7 op/s 2023-03-10T00:19:28.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34140.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:19:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33678.890616 secs 2023-03-10T00:19:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:28 smithi122 ceph-mon[93869]: pgmap v18473: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-10T00:19:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33678.890616 secs 2023-03-10T00:19:28.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:28 smithi181 ceph-mon[94189]: pgmap v18473: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-10T00:19:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33678.890616 secs 2023-03-10T00:19:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:28 smithi038 ceph-mon[92422]: pgmap v18473: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-10T00:19:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:30 smithi122 ceph-mon[93869]: pgmap v18474: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2023-03-10T00:19:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33681.461695 secs 2023-03-10T00:19:30.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:30 smithi181 ceph-mon[94189]: pgmap v18474: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2023-03-10T00:19:30.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33681.461695 secs 2023-03-10T00:19:30.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:30 smithi038 ceph-mon[92422]: pgmap v18474: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2023-03-10T00:19:30.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33681.461695 secs 2023-03-10T00:19:32.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:32 smithi122 ceph-mon[93869]: pgmap v18475: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-10T00:19:32.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:32 smithi181 ceph-mon[94189]: pgmap v18475: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-10T00:19:32.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:32 smithi038 ceph-mon[92422]: pgmap v18475: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-10T00:19:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34145.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:19:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33683.890700 secs 2023-03-10T00:19:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33683.890700 secs 2023-03-10T00:19:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33683.890700 secs 2023-03-10T00:19:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:35 smithi122 ceph-mon[93869]: pgmap v18476: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-10T00:19:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:35 smithi181 ceph-mon[94189]: pgmap v18476: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-10T00:19:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:35 smithi038 ceph-mon[92422]: pgmap v18476: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-10T00:19:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33686.461751 secs 2023-03-10T00:19:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33686.461751 secs 2023-03-10T00:19:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33686.461751 secs 2023-03-10T00:19:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:37 smithi122 ceph-mon[93869]: pgmap v18477: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-10T00:19:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:37 smithi181 ceph-mon[94189]: pgmap v18477: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-10T00:19:37.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:37 smithi038 ceph-mon[92422]: pgmap v18477: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-10T00:19:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33688.890762 secs 2023-03-10T00:19:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33688.890762 secs 2023-03-10T00:19:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33688.890762 secs 2023-03-10T00:19:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34150.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:19:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:39 smithi122 ceph-mon[93869]: pgmap v18478: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-10T00:19:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:39 smithi181 ceph-mon[94189]: pgmap v18478: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-10T00:19:39.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:39 smithi038 ceph-mon[92422]: pgmap v18478: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-10T00:19:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:41 smithi122 ceph-mon[93869]: pgmap v18479: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 6 op/s 2023-03-10T00:19:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33691.461809 secs 2023-03-10T00:19:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:41 smithi181 ceph-mon[94189]: pgmap v18479: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 6 op/s 2023-03-10T00:19:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33691.461809 secs 2023-03-10T00:19:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:41 smithi038 ceph-mon[92422]: pgmap v18479: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 6 op/s 2023-03-10T00:19:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33691.461809 secs 2023-03-10T00:19:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:43 smithi122 ceph-mon[93869]: pgmap v18480: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 341 B/s wr, 8 op/s 2023-03-10T00:19:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33693.890850 secs 2023-03-10T00:19:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:43 smithi181 ceph-mon[94189]: pgmap v18480: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 341 B/s wr, 8 op/s 2023-03-10T00:19:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33693.890850 secs 2023-03-10T00:19:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:43 smithi038 ceph-mon[92422]: pgmap v18480: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 341 B/s wr, 8 op/s 2023-03-10T00:19:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33693.890850 secs 2023-03-10T00:19:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34155.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:19:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:45 smithi122 ceph-mon[93869]: pgmap v18481: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-10T00:19:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:45 smithi181 ceph-mon[94189]: pgmap v18481: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-10T00:19:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:45 smithi038 ceph-mon[92422]: pgmap v18481: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-10T00:19:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33696.461886 secs 2023-03-10T00:19:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33696.461886 secs 2023-03-10T00:19:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33696.461886 secs 2023-03-10T00:19:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:47 smithi122 ceph-mon[93869]: pgmap v18482: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-10T00:19:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:47 smithi181 ceph-mon[94189]: pgmap v18482: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-10T00:19:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:47 smithi038 ceph-mon[92422]: pgmap v18482: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 27 KiB/s wr, 7 op/s 2023-03-10T00:19:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33698.890901 secs 2023-03-10T00:19:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33698.890901 secs 2023-03-10T00:19:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33698.890901 secs 2023-03-10T00:19:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34160.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:19:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:49 smithi122 ceph-mon[93869]: pgmap v18483: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 184 KiB/s wr, 10 op/s 2023-03-10T00:19:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:19:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:19:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:49 smithi181 ceph-mon[94189]: pgmap v18483: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 184 KiB/s wr, 10 op/s 2023-03-10T00:19:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:19:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:19:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:49 smithi038 ceph-mon[92422]: pgmap v18483: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 184 KiB/s wr, 10 op/s 2023-03-10T00:19:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:19:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:19:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:51 smithi122 ceph-mon[93869]: pgmap v18484: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 211 KiB/s wr, 9 op/s 2023-03-10T00:19:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33701.461945 secs 2023-03-10T00:19:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:19:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:19:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:19:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:19:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:51 smithi181 ceph-mon[94189]: pgmap v18484: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 211 KiB/s wr, 9 op/s 2023-03-10T00:19:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33701.461945 secs 2023-03-10T00:19:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:19:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:19:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:19:51.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:19:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:51 smithi038 ceph-mon[92422]: pgmap v18484: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 186 KiB/s rd, 211 KiB/s wr, 9 op/s 2023-03-10T00:19:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33701.461945 secs 2023-03-10T00:19:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:19:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:19:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:19:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:19:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:19:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:19:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:19:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:19:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:19:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:19:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:53 smithi122 ceph-mon[93869]: pgmap v18485: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 549 KiB/s rd, 529 KiB/s wr, 11 op/s 2023-03-10T00:19:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:53 smithi181 ceph-mon[94189]: pgmap v18485: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 549 KiB/s rd, 529 KiB/s wr, 11 op/s 2023-03-10T00:19:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:53 smithi038 ceph-mon[92422]: pgmap v18485: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 549 KiB/s rd, 529 KiB/s wr, 11 op/s 2023-03-10T00:19:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34165.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:19:54.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:54 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:19:54.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:54 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:19:54.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:54 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:19:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:55 smithi122 ceph-mon[93869]: pgmap v18486: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 529 KiB/s wr, 10 op/s 2023-03-10T00:19:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:55 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:19:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:19:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:19:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:55 smithi181 ceph-mon[94189]: pgmap v18486: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 529 KiB/s wr, 10 op/s 2023-03-10T00:19:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:55 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:19:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:19:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:19:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:55 smithi038 ceph-mon[92422]: pgmap v18486: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 529 KiB/s wr, 10 op/s 2023-03-10T00:19:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:55 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:19:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:19:55.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:19:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33706.461982 secs 2023-03-10T00:19:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:19:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:19:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33706.461982 secs 2023-03-10T00:19:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:19:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:19:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33706.461982 secs 2023-03-10T00:19:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:19:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:19:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:57 smithi122 ceph-mon[93869]: pgmap v18487: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 557 KiB/s wr, 10 op/s 2023-03-10T00:19:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:19:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:19:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:57 smithi181 ceph-mon[94189]: pgmap v18487: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 557 KiB/s wr, 10 op/s 2023-03-10T00:19:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:19:57.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:19:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:57 smithi038 ceph-mon[92422]: pgmap v18487: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 557 KiB/s wr, 10 op/s 2023-03-10T00:19:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:19:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:19:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34170.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:19:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:19:59 smithi122 ceph-mon[93869]: pgmap v18488: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1003 KiB/s rd, 820 KiB/s wr, 14 op/s 2023-03-10T00:19:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:19:59 smithi181 ceph-mon[94189]: pgmap v18488: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1003 KiB/s rd, 820 KiB/s wr, 14 op/s 2023-03-10T00:19:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:19:59 smithi038 ceph-mon[92422]: pgmap v18488: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1003 KiB/s rd, 820 KiB/s wr, 14 op/s 2023-03-10T00:20:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:00 smithi122 ceph-mon[93869]: Health detail: HEALTH_WARN 1 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-10T00:20:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:00 smithi122 ceph-mon[93869]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2023-03-10T00:20:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:00 smithi122 ceph-mon[93869]: mds.h(mds.1): 2 slow requests are blocked > 180 secs 2023-03-10T00:20:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:00 smithi122 ceph-mon[93869]: [WRN] MDS_TRIM: 2 MDSs behind on trimming 2023-03-10T00:20:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:00 smithi122 ceph-mon[93869]: mds.e(mds.0): Behind on trimming (899/128) max_segments: 128, num_segments: 899 2023-03-10T00:20:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:00 smithi122 ceph-mon[93869]: mds.b(mds.0): Behind on trimming (899/128) max_segments: 128, num_segments: 899 2023-03-10T00:20:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:00 smithi181 ceph-mon[94189]: Health detail: HEALTH_WARN 1 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-10T00:20:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:00 smithi181 ceph-mon[94189]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2023-03-10T00:20:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:00 smithi181 ceph-mon[94189]: mds.h(mds.1): 2 slow requests are blocked > 180 secs 2023-03-10T00:20:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:00 smithi181 ceph-mon[94189]: [WRN] MDS_TRIM: 2 MDSs behind on trimming 2023-03-10T00:20:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:00 smithi181 ceph-mon[94189]: mds.e(mds.0): Behind on trimming (899/128) max_segments: 128, num_segments: 899 2023-03-10T00:20:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:00 smithi181 ceph-mon[94189]: mds.b(mds.0): Behind on trimming (899/128) max_segments: 128, num_segments: 899 2023-03-10T00:20:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:00 smithi038 ceph-mon[92422]: Health detail: HEALTH_WARN 1 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-10T00:20:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:00 smithi038 ceph-mon[92422]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2023-03-10T00:20:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:00 smithi038 ceph-mon[92422]: mds.h(mds.1): 2 slow requests are blocked > 180 secs 2023-03-10T00:20:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:00 smithi038 ceph-mon[92422]: [WRN] MDS_TRIM: 2 MDSs behind on trimming 2023-03-10T00:20:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:00 smithi038 ceph-mon[92422]: mds.e(mds.0): Behind on trimming (899/128) max_segments: 128, num_segments: 899 2023-03-10T00:20:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:00 smithi038 ceph-mon[92422]: mds.b(mds.0): Behind on trimming (899/128) max_segments: 128, num_segments: 899 2023-03-10T00:20:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:01 smithi122 ceph-mon[93869]: pgmap v18489: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 876 KiB/s rd, 694 KiB/s wr, 12 op/s 2023-03-10T00:20:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33711.461999 secs 2023-03-10T00:20:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:01 smithi181 ceph-mon[94189]: pgmap v18489: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 876 KiB/s rd, 694 KiB/s wr, 12 op/s 2023-03-10T00:20:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33711.461999 secs 2023-03-10T00:20:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:01 smithi038 ceph-mon[92422]: pgmap v18489: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 876 KiB/s rd, 694 KiB/s wr, 12 op/s 2023-03-10T00:20:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33711.461999 secs 2023-03-10T00:20:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:03 smithi122 ceph-mon[93869]: pgmap v18490: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 948 KiB/s rd, 766 KiB/s wr, 14 op/s 2023-03-10T00:20:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33713.891118 secs 2023-03-10T00:20:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:03 smithi181 ceph-mon[94189]: pgmap v18490: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 948 KiB/s rd, 766 KiB/s wr, 14 op/s 2023-03-10T00:20:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33713.891118 secs 2023-03-10T00:20:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:03 smithi038 ceph-mon[92422]: pgmap v18490: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 948 KiB/s rd, 766 KiB/s wr, 14 op/s 2023-03-10T00:20:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33713.891118 secs 2023-03-10T00:20:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34175.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:20:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:05 smithi122 ceph-mon[93869]: pgmap v18491: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 644 KiB/s rd, 521 KiB/s wr, 24 op/s 2023-03-10T00:20:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:05 smithi122 ceph-mon[93869]: osdmap e1371: 12 total, 12 up, 12 in 2023-03-10T00:20:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:05 smithi181 ceph-mon[94189]: pgmap v18491: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 644 KiB/s rd, 521 KiB/s wr, 24 op/s 2023-03-10T00:20:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:05 smithi181 ceph-mon[94189]: osdmap e1371: 12 total, 12 up, 12 in 2023-03-10T00:20:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:05 smithi038 ceph-mon[92422]: pgmap v18491: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 644 KiB/s rd, 521 KiB/s wr, 24 op/s 2023-03-10T00:20:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:05 smithi038 ceph-mon[92422]: osdmap e1371: 12 total, 12 up, 12 in 2023-03-10T00:20:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33716.462047 secs 2023-03-10T00:20:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33716.462047 secs 2023-03-10T00:20:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33716.462047 secs 2023-03-10T00:20:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:07 smithi122 ceph-mon[93869]: pgmap v18493: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 788 KiB/s rd, 620 KiB/s wr, 28 op/s 2023-03-10T00:20:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:07 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:20:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:20:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:20:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:07 smithi181 ceph-mon[94189]: pgmap v18493: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 788 KiB/s rd, 620 KiB/s wr, 28 op/s 2023-03-10T00:20:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:07 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:20:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:20:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:20:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:07 smithi038 ceph-mon[92422]: pgmap v18493: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 788 KiB/s rd, 620 KiB/s wr, 28 op/s 2023-03-10T00:20:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:07 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:20:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:20:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:20:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33718.891181 secs 2023-03-10T00:20:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:20:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:20:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34180.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:20:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33718.891181 secs 2023-03-10T00:20:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:20:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:20:08.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33718.891181 secs 2023-03-10T00:20:08.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:20:08.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:20:09.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:09 smithi038 ceph-mon[92422]: pgmap v18494: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 488 KiB/s wr, 29 op/s 2023-03-10T00:20:09.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:20:09.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:20:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:09 smithi122 ceph-mon[93869]: pgmap v18494: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 488 KiB/s wr, 29 op/s 2023-03-10T00:20:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:20:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:20:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:09 smithi181 ceph-mon[94189]: pgmap v18494: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 488 KiB/s wr, 29 op/s 2023-03-10T00:20:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:20:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:20:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:11 smithi038 ceph-mon[92422]: pgmap v18495: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 444 KiB/s rd, 529 KiB/s wr, 29 op/s 2023-03-10T00:20:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33721.462106 secs 2023-03-10T00:20:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:11 smithi122 ceph-mon[93869]: pgmap v18495: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 444 KiB/s rd, 529 KiB/s wr, 29 op/s 2023-03-10T00:20:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33721.462106 secs 2023-03-10T00:20:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:11 smithi181 ceph-mon[94189]: pgmap v18495: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 444 KiB/s rd, 529 KiB/s wr, 29 op/s 2023-03-10T00:20:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33721.462106 secs 2023-03-10T00:20:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:13 smithi038 ceph-mon[92422]: pgmap v18496: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 512 KiB/s wr, 27 op/s 2023-03-10T00:20:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33723.891244 secs 2023-03-10T00:20:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:13 smithi038 ceph-mon[92422]: osdmap e1372: 12 total, 12 up, 12 in 2023-03-10T00:20:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34185.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:20:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:13 smithi122 ceph-mon[93869]: pgmap v18496: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 512 KiB/s wr, 27 op/s 2023-03-10T00:20:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33723.891244 secs 2023-03-10T00:20:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:13 smithi122 ceph-mon[93869]: osdmap e1372: 12 total, 12 up, 12 in 2023-03-10T00:20:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:13 smithi181 ceph-mon[94189]: pgmap v18496: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 512 KiB/s wr, 27 op/s 2023-03-10T00:20:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33723.891244 secs 2023-03-10T00:20:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:13 smithi181 ceph-mon[94189]: osdmap e1372: 12 total, 12 up, 12 in 2023-03-10T00:20:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:15 smithi038 ceph-mon[92422]: pgmap v18498: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 434 KiB/s wr, 13 op/s 2023-03-10T00:20:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:15 smithi122 ceph-mon[93869]: pgmap v18498: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 434 KiB/s wr, 13 op/s 2023-03-10T00:20:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:15 smithi181 ceph-mon[94189]: pgmap v18498: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 434 KiB/s wr, 13 op/s 2023-03-10T00:20:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33726.462162 secs 2023-03-10T00:20:16.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:20:16.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:20:16.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:16 smithi122 ceph-mon[93869]: pgmap v18499: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 396 KiB/s wr, 12 op/s 2023-03-10T00:20:16.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33726.462162 secs 2023-03-10T00:20:16.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:20:16.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:20:16.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:16 smithi181 ceph-mon[94189]: pgmap v18499: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 396 KiB/s wr, 12 op/s 2023-03-10T00:20:17.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33726.462162 secs 2023-03-10T00:20:17.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:20:17.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:20:17.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:16 smithi038 ceph-mon[92422]: pgmap v18499: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 396 KiB/s wr, 12 op/s 2023-03-10T00:20:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:20:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:20:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:20:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:20:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:20:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:20:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34190.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:20:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33728.891322 secs 2023-03-10T00:20:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:18 smithi122 ceph-mon[93869]: pgmap v18500: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 179 KiB/s wr, 8 op/s 2023-03-10T00:20:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33728.891322 secs 2023-03-10T00:20:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:18 smithi181 ceph-mon[94189]: pgmap v18500: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 179 KiB/s wr, 8 op/s 2023-03-10T00:20:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33728.891322 secs 2023-03-10T00:20:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:18 smithi038 ceph-mon[92422]: pgmap v18500: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 179 KiB/s wr, 8 op/s 2023-03-10T00:20:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:20 smithi122 ceph-mon[93869]: pgmap v18501: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 227 KiB/s wr, 8 op/s 2023-03-10T00:20:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33731.462243 secs 2023-03-10T00:20:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:20 smithi181 ceph-mon[94189]: pgmap v18501: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 227 KiB/s wr, 8 op/s 2023-03-10T00:20:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33731.462243 secs 2023-03-10T00:20:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:20 smithi038 ceph-mon[92422]: pgmap v18501: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 227 KiB/s wr, 8 op/s 2023-03-10T00:20:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33731.462243 secs 2023-03-10T00:20:22.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:22 smithi122 ceph-mon[93869]: pgmap v18502: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 125 KiB/s wr, 8 op/s 2023-03-10T00:20:22.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:22 smithi181 ceph-mon[94189]: pgmap v18502: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 125 KiB/s wr, 8 op/s 2023-03-10T00:20:22.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:22 smithi038 ceph-mon[92422]: pgmap v18502: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 125 KiB/s wr, 8 op/s 2023-03-10T00:20:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34195.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:20:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33733.891421 secs 2023-03-10T00:20:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33733.891421 secs 2023-03-10T00:20:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33733.891421 secs 2023-03-10T00:20:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:25 smithi122 ceph-mon[93869]: pgmap v18503: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 122 KiB/s wr, 7 op/s 2023-03-10T00:20:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:25 smithi181 ceph-mon[94189]: pgmap v18503: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 122 KiB/s wr, 7 op/s 2023-03-10T00:20:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:25 smithi038 ceph-mon[92422]: pgmap v18503: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 122 KiB/s wr, 7 op/s 2023-03-10T00:20:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33736.462302 secs 2023-03-10T00:20:26.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33736.462302 secs 2023-03-10T00:20:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33736.462302 secs 2023-03-10T00:20:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:27 smithi122 ceph-mon[93869]: pgmap v18504: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 104 KiB/s wr, 7 op/s 2023-03-10T00:20:27.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:27 smithi181 ceph-mon[94189]: pgmap v18504: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 104 KiB/s wr, 7 op/s 2023-03-10T00:20:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:27 smithi038 ceph-mon[92422]: pgmap v18504: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 104 KiB/s wr, 7 op/s 2023-03-10T00:20:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33738.891483 secs 2023-03-10T00:20:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33738.891483 secs 2023-03-10T00:20:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33738.891483 secs 2023-03-10T00:20:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34200.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:20:29.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:29 smithi122 ceph-mon[93869]: pgmap v18505: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 104 KiB/s wr, 8 op/s 2023-03-10T00:20:29.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:29 smithi181 ceph-mon[94189]: pgmap v18505: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 104 KiB/s wr, 8 op/s 2023-03-10T00:20:29.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:29 smithi038 ceph-mon[92422]: pgmap v18505: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 104 KiB/s wr, 8 op/s 2023-03-10T00:20:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:31 smithi122 ceph-mon[93869]: pgmap v18506: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 104 KiB/s wr, 6 op/s 2023-03-10T00:20:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33741.462390 secs 2023-03-10T00:20:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:31 smithi181 ceph-mon[94189]: pgmap v18506: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 104 KiB/s wr, 6 op/s 2023-03-10T00:20:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33741.462390 secs 2023-03-10T00:20:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:31 smithi038 ceph-mon[92422]: pgmap v18506: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 104 KiB/s wr, 6 op/s 2023-03-10T00:20:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33741.462390 secs 2023-03-10T00:20:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:33 smithi122 ceph-mon[93869]: pgmap v18507: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 85 B/s wr, 7 op/s 2023-03-10T00:20:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33743.891542 secs 2023-03-10T00:20:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:33 smithi181 ceph-mon[94189]: pgmap v18507: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 85 B/s wr, 7 op/s 2023-03-10T00:20:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33743.891542 secs 2023-03-10T00:20:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:33 smithi038 ceph-mon[92422]: pgmap v18507: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 85 B/s wr, 7 op/s 2023-03-10T00:20:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33743.891542 secs 2023-03-10T00:20:33.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34205.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:20:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:35 smithi122 ceph-mon[93869]: pgmap v18508: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-10T00:20:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:35 smithi181 ceph-mon[94189]: pgmap v18508: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-10T00:20:35.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:35 smithi038 ceph-mon[92422]: pgmap v18508: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-10T00:20:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33746.462475 secs 2023-03-10T00:20:36.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33746.462475 secs 2023-03-10T00:20:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33746.462475 secs 2023-03-10T00:20:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:37 smithi122 ceph-mon[93869]: pgmap v18509: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-10T00:20:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:37 smithi181 ceph-mon[94189]: pgmap v18509: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-10T00:20:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:37 smithi038 ceph-mon[92422]: pgmap v18509: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-10T00:20:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33748.891653 secs 2023-03-10T00:20:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33748.891653 secs 2023-03-10T00:20:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33748.891653 secs 2023-03-10T00:20:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34210.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:20:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:39 smithi122 ceph-mon[93869]: pgmap v18510: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2023-03-10T00:20:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:39 smithi181 ceph-mon[94189]: pgmap v18510: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2023-03-10T00:20:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:39 smithi038 ceph-mon[92422]: pgmap v18510: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2023-03-10T00:20:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:41 smithi122 ceph-mon[93869]: pgmap v18511: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 127 KiB/s wr, 6 op/s 2023-03-10T00:20:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33751.462533 secs 2023-03-10T00:20:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:41 smithi181 ceph-mon[94189]: pgmap v18511: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 127 KiB/s wr, 6 op/s 2023-03-10T00:20:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33751.462533 secs 2023-03-10T00:20:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:41 smithi038 ceph-mon[92422]: pgmap v18511: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 127 KiB/s wr, 6 op/s 2023-03-10T00:20:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33751.462533 secs 2023-03-10T00:20:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:43 smithi122 ceph-mon[93869]: pgmap v18512: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 127 KiB/s wr, 7 op/s 2023-03-10T00:20:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:43 smithi181 ceph-mon[94189]: pgmap v18512: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 127 KiB/s wr, 7 op/s 2023-03-10T00:20:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:43 smithi038 ceph-mon[92422]: pgmap v18512: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 127 KiB/s wr, 7 op/s 2023-03-10T00:20:43.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34215.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:20:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:45 smithi122 ceph-mon[93869]: pgmap v18513: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 127 KiB/s wr, 6 op/s 2023-03-10T00:20:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:45 smithi181 ceph-mon[94189]: pgmap v18513: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 127 KiB/s wr, 6 op/s 2023-03-10T00:20:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:45 smithi038 ceph-mon[92422]: pgmap v18513: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 127 KiB/s wr, 6 op/s 2023-03-10T00:20:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33756.462645 secs 2023-03-10T00:20:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:46 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:20:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33756.462645 secs 2023-03-10T00:20:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:46 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:20:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33756.462645 secs 2023-03-10T00:20:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:46 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:20:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:47 smithi122 ceph-mon[93869]: pgmap v18514: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 309 KiB/s wr, 7 op/s 2023-03-10T00:20:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:47 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:20:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:20:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:20:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:47 smithi181 ceph-mon[94189]: pgmap v18514: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 309 KiB/s wr, 7 op/s 2023-03-10T00:20:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:47 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:20:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:20:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:20:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:47 smithi038 ceph-mon[92422]: pgmap v18514: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 309 KiB/s wr, 7 op/s 2023-03-10T00:20:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:47 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:20:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:20:47.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:20:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:20:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:20:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:20:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:20:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:20:48.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:20:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34220.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:20:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:49 smithi122 ceph-mon[93869]: pgmap v18515: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 309 KiB/s wr, 8 op/s 2023-03-10T00:20:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:20:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:20:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:49 smithi181 ceph-mon[94189]: pgmap v18515: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 309 KiB/s wr, 8 op/s 2023-03-10T00:20:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:20:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:20:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:49 smithi038 ceph-mon[92422]: pgmap v18515: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 309 KiB/s wr, 8 op/s 2023-03-10T00:20:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:20:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:20:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:51 smithi122 ceph-mon[93869]: pgmap v18516: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 313 KiB/s rd, 338 KiB/s wr, 9 op/s 2023-03-10T00:20:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33761.462705 secs 2023-03-10T00:20:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:20:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:51 smithi181 ceph-mon[94189]: pgmap v18516: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 313 KiB/s rd, 338 KiB/s wr, 9 op/s 2023-03-10T00:20:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33761.462705 secs 2023-03-10T00:20:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:20:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:51 smithi038 ceph-mon[92422]: pgmap v18516: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 313 KiB/s rd, 338 KiB/s wr, 9 op/s 2023-03-10T00:20:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33761.462705 secs 2023-03-10T00:20:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:20:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:20:52.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:20:52.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:20:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:20:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:20:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:20:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:20:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:20:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:20:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:53 smithi122 ceph-mon[93869]: pgmap v18517: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 256 KiB/s wr, 10 op/s 2023-03-10T00:20:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:53 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 33763.891834 secs 2023-03-10T00:20:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:53 smithi181 ceph-mon[94189]: pgmap v18517: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 256 KiB/s wr, 10 op/s 2023-03-10T00:20:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:53 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 33763.891834 secs 2023-03-10T00:20:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:53 smithi038 ceph-mon[92422]: pgmap v18517: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 256 KiB/s wr, 10 op/s 2023-03-10T00:20:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:53 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 33763.891834 secs 2023-03-10T00:20:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34225.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:20:54.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:54 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:20:54.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:54 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:20:54.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:54 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:20:54.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:54 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:20:54.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:54 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:20:54.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:54 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:20:54.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:54 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:20:54.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:54 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:20:54.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:54 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:20:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:55 smithi122 ceph-mon[93869]: pgmap v18518: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 256 KiB/s wr, 9 op/s 2023-03-10T00:20:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:55 smithi181 ceph-mon[94189]: pgmap v18518: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 256 KiB/s wr, 9 op/s 2023-03-10T00:20:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:55 smithi038 ceph-mon[92422]: pgmap v18518: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 256 KiB/s wr, 9 op/s 2023-03-10T00:20:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33766.462791 secs 2023-03-10T00:20:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:20:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:20:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:56 smithi122 ceph-mon[93869]: pgmap v18519: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 926 KiB/s rd, 812 KiB/s wr, 12 op/s 2023-03-10T00:20:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33766.462791 secs 2023-03-10T00:20:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:20:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:20:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:56 smithi181 ceph-mon[94189]: pgmap v18519: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 926 KiB/s rd, 812 KiB/s wr, 12 op/s 2023-03-10T00:20:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33766.462791 secs 2023-03-10T00:20:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:20:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:20:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:56 smithi038 ceph-mon[92422]: pgmap v18519: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 926 KiB/s rd, 812 KiB/s wr, 12 op/s 2023-03-10T00:20:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:20:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:20:57.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:20:57.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:20:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:20:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:20:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34230.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:20:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:58 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 33768.891893 secs 2023-03-10T00:20:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:20:58 smithi122 ceph-mon[93869]: pgmap v18520: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 646 KiB/s rd, 631 KiB/s wr, 13 op/s 2023-03-10T00:20:58.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:58 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 33768.891893 secs 2023-03-10T00:20:58.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:20:58 smithi181 ceph-mon[94189]: pgmap v18520: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 646 KiB/s rd, 631 KiB/s wr, 13 op/s 2023-03-10T00:20:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:58 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 33768.891893 secs 2023-03-10T00:20:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:20:58 smithi038 ceph-mon[92422]: pgmap v18520: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 646 KiB/s rd, 631 KiB/s wr, 13 op/s 2023-03-10T00:21:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:21:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:21:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:00 smithi122 ceph-mon[93869]: pgmap v18521: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 711 KiB/s rd, 698 KiB/s wr, 13 op/s 2023-03-10T00:21:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:21:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:21:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:00 smithi181 ceph-mon[94189]: pgmap v18521: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 711 KiB/s rd, 698 KiB/s wr, 13 op/s 2023-03-10T00:21:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:21:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:21:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:00 smithi038 ceph-mon[92422]: pgmap v18521: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 711 KiB/s rd, 698 KiB/s wr, 13 op/s 2023-03-10T00:21:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33771.462847 secs 2023-03-10T00:21:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:21:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:21:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33771.462847 secs 2023-03-10T00:21:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:21:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:21:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33771.462847 secs 2023-03-10T00:21:02.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:21:02.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:21:02.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:02 smithi122 ceph-mon[93869]: pgmap v18522: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 684 KiB/s rd, 669 KiB/s wr, 13 op/s 2023-03-10T00:21:02.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:02 smithi181 ceph-mon[94189]: pgmap v18522: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 684 KiB/s rd, 669 KiB/s wr, 13 op/s 2023-03-10T00:21:02.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:02 smithi038 ceph-mon[92422]: pgmap v18522: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 684 KiB/s rd, 669 KiB/s wr, 13 op/s 2023-03-10T00:21:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34235.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:21:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33773.891956 secs 2023-03-10T00:21:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33773.891956 secs 2023-03-10T00:21:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33773.891956 secs 2023-03-10T00:21:04.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:04 smithi122 ceph-mon[93869]: pgmap v18523: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 644 KiB/s rd, 646 KiB/s wr, 23 op/s 2023-03-10T00:21:04.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:04 smithi181 ceph-mon[94189]: pgmap v18523: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 644 KiB/s rd, 646 KiB/s wr, 23 op/s 2023-03-10T00:21:04.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:04 smithi038 ceph-mon[92422]: pgmap v18523: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 644 KiB/s rd, 646 KiB/s wr, 23 op/s 2023-03-10T00:21:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:05 smithi122 ceph-mon[93869]: osdmap e1373: 12 total, 12 up, 12 in 2023-03-10T00:21:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33776.462905 secs 2023-03-10T00:21:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:05 smithi181 ceph-mon[94189]: osdmap e1373: 12 total, 12 up, 12 in 2023-03-10T00:21:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33776.462905 secs 2023-03-10T00:21:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:05 smithi038 ceph-mon[92422]: osdmap e1373: 12 total, 12 up, 12 in 2023-03-10T00:21:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33776.462905 secs 2023-03-10T00:21:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:06 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:21:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:21:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:06 smithi122 ceph-mon[93869]: pgmap v18525: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 583 KiB/s wr, 27 op/s 2023-03-10T00:21:06.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:06 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:21:06.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:21:06.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:06 smithi181 ceph-mon[94189]: pgmap v18525: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 583 KiB/s wr, 27 op/s 2023-03-10T00:21:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:06 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:21:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:21:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:06 smithi038 ceph-mon[92422]: pgmap v18525: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 583 KiB/s wr, 27 op/s 2023-03-10T00:21:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:21:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:21:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:21:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:21:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:21:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:21:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34240.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:21:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33778.892032 secs 2023-03-10T00:21:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:08 smithi122 ceph-mon[93869]: pgmap v18526: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 610 KiB/s wr, 29 op/s 2023-03-10T00:21:08.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33778.892032 secs 2023-03-10T00:21:08.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:08 smithi181 ceph-mon[94189]: pgmap v18526: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 610 KiB/s wr, 29 op/s 2023-03-10T00:21:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33778.892032 secs 2023-03-10T00:21:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:08 smithi038 ceph-mon[92422]: pgmap v18526: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 610 KiB/s wr, 29 op/s 2023-03-10T00:21:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:21:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:21:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:21:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:21:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:21:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:21:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:10 smithi122 ceph-mon[93869]: pgmap v18527: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 521 KiB/s rd, 755 KiB/s wr, 30 op/s 2023-03-10T00:21:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33781.462987 secs 2023-03-10T00:21:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:10 smithi181 ceph-mon[94189]: pgmap v18527: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 521 KiB/s rd, 755 KiB/s wr, 30 op/s 2023-03-10T00:21:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33781.462987 secs 2023-03-10T00:21:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:10 smithi038 ceph-mon[92422]: pgmap v18527: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 521 KiB/s rd, 755 KiB/s wr, 30 op/s 2023-03-10T00:21:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33781.462987 secs 2023-03-10T00:21:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:12 smithi122 ceph-mon[93869]: pgmap v18528: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 521 KiB/s rd, 755 KiB/s wr, 29 op/s 2023-03-10T00:21:12.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:12 smithi181 ceph-mon[94189]: pgmap v18528: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 521 KiB/s rd, 755 KiB/s wr, 29 op/s 2023-03-10T00:21:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:12 smithi038 ceph-mon[92422]: pgmap v18528: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 521 KiB/s rd, 755 KiB/s wr, 29 op/s 2023-03-10T00:21:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34245.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:21:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33783.892095 secs 2023-03-10T00:21:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:13 smithi122 ceph-mon[93869]: osdmap e1374: 12 total, 12 up, 12 in 2023-03-10T00:21:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33783.892095 secs 2023-03-10T00:21:13.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:13 smithi181 ceph-mon[94189]: osdmap e1374: 12 total, 12 up, 12 in 2023-03-10T00:21:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33783.892095 secs 2023-03-10T00:21:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:13 smithi038 ceph-mon[92422]: osdmap e1374: 12 total, 12 up, 12 in 2023-03-10T00:21:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:15 smithi122 ceph-mon[93869]: pgmap v18530: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 280 KiB/s wr, 12 op/s 2023-03-10T00:21:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:15 smithi181 ceph-mon[94189]: pgmap v18530: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 280 KiB/s wr, 12 op/s 2023-03-10T00:21:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:15 smithi038 ceph-mon[92422]: pgmap v18530: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 280 KiB/s wr, 12 op/s 2023-03-10T00:21:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33786.463052 secs 2023-03-10T00:21:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33786.463052 secs 2023-03-10T00:21:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33786.463052 secs 2023-03-10T00:21:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:17 smithi122 ceph-mon[93869]: pgmap v18531: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 253 KiB/s wr, 11 op/s 2023-03-10T00:21:17.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:17 smithi181 ceph-mon[94189]: pgmap v18531: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 253 KiB/s wr, 11 op/s 2023-03-10T00:21:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:17 smithi038 ceph-mon[92422]: pgmap v18531: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 253 KiB/s wr, 11 op/s 2023-03-10T00:21:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33788.892184 secs 2023-03-10T00:21:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33788.892184 secs 2023-03-10T00:21:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33788.892184 secs 2023-03-10T00:21:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34250.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:21:19.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:19 smithi122 ceph-mon[93869]: pgmap v18532: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 226 KiB/s wr, 8 op/s 2023-03-10T00:21:19.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:19 smithi181 ceph-mon[94189]: pgmap v18532: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 226 KiB/s wr, 8 op/s 2023-03-10T00:21:19.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:19 smithi038 ceph-mon[92422]: pgmap v18532: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 229 KiB/s rd, 226 KiB/s wr, 8 op/s 2023-03-10T00:21:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:21 smithi122 ceph-mon[93869]: pgmap v18533: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 99 KiB/s wr, 6 op/s 2023-03-10T00:21:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33791.463108 secs 2023-03-10T00:21:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:21 smithi181 ceph-mon[94189]: pgmap v18533: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 99 KiB/s wr, 6 op/s 2023-03-10T00:21:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33791.463108 secs 2023-03-10T00:21:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:21 smithi038 ceph-mon[92422]: pgmap v18533: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 99 KiB/s wr, 6 op/s 2023-03-10T00:21:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33791.463108 secs 2023-03-10T00:21:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:23 smithi122 ceph-mon[93869]: pgmap v18534: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 99 KiB/s wr, 6 op/s 2023-03-10T00:21:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33793.892252 secs 2023-03-10T00:21:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:23 smithi181 ceph-mon[94189]: pgmap v18534: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 99 KiB/s wr, 6 op/s 2023-03-10T00:21:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33793.892252 secs 2023-03-10T00:21:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:23 smithi038 ceph-mon[92422]: pgmap v18534: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 99 KiB/s wr, 6 op/s 2023-03-10T00:21:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33793.892252 secs 2023-03-10T00:21:23.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34255.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:21:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:25 smithi122 ceph-mon[93869]: pgmap v18535: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 97 KiB/s wr, 5 op/s 2023-03-10T00:21:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:25 smithi181 ceph-mon[94189]: pgmap v18535: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 97 KiB/s wr, 5 op/s 2023-03-10T00:21:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:25 smithi038 ceph-mon[92422]: pgmap v18535: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 97 KiB/s wr, 5 op/s 2023-03-10T00:21:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33796.463168 secs 2023-03-10T00:21:26.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33796.463168 secs 2023-03-10T00:21:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33796.463168 secs 2023-03-10T00:21:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:27 smithi122 ceph-mon[93869]: pgmap v18536: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 82 KiB/s wr, 5 op/s 2023-03-10T00:21:27.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:27 smithi181 ceph-mon[94189]: pgmap v18536: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 82 KiB/s wr, 5 op/s 2023-03-10T00:21:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:27 smithi038 ceph-mon[92422]: pgmap v18536: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 82 KiB/s wr, 5 op/s 2023-03-10T00:21:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33798.892311 secs 2023-03-10T00:21:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33798.892311 secs 2023-03-10T00:21:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33798.892311 secs 2023-03-10T00:21:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34260.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:21:29.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:29 smithi122 ceph-mon[93869]: pgmap v18537: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 82 KiB/s wr, 6 op/s 2023-03-10T00:21:29.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:29 smithi181 ceph-mon[94189]: pgmap v18537: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 82 KiB/s wr, 6 op/s 2023-03-10T00:21:29.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:29 smithi038 ceph-mon[92422]: pgmap v18537: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 82 KiB/s wr, 6 op/s 2023-03-10T00:21:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:31 smithi122 ceph-mon[93869]: pgmap v18538: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 82 KiB/s wr, 5 op/s 2023-03-10T00:21:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33801.463228 secs 2023-03-10T00:21:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:31 smithi181 ceph-mon[94189]: pgmap v18538: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 82 KiB/s wr, 5 op/s 2023-03-10T00:21:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33801.463228 secs 2023-03-10T00:21:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:31 smithi038 ceph-mon[92422]: pgmap v18538: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 82 KiB/s wr, 5 op/s 2023-03-10T00:21:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33801.463228 secs 2023-03-10T00:21:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:33 smithi122 ceph-mon[93869]: pgmap v18539: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 6 op/s 2023-03-10T00:21:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33803.892428 secs 2023-03-10T00:21:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:33 smithi181 ceph-mon[94189]: pgmap v18539: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 6 op/s 2023-03-10T00:21:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33803.892428 secs 2023-03-10T00:21:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:33 smithi038 ceph-mon[92422]: pgmap v18539: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 6 op/s 2023-03-10T00:21:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33803.892428 secs 2023-03-10T00:21:33.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34265.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:21:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:35 smithi122 ceph-mon[93869]: pgmap v18540: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-10T00:21:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:35 smithi181 ceph-mon[94189]: pgmap v18540: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-10T00:21:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:35 smithi038 ceph-mon[92422]: pgmap v18540: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-10T00:21:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33806.463285 secs 2023-03-10T00:21:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33806.463285 secs 2023-03-10T00:21:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33806.463285 secs 2023-03-10T00:21:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:37 smithi122 ceph-mon[93869]: pgmap v18541: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-10T00:21:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:37 smithi181 ceph-mon[94189]: pgmap v18541: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-10T00:21:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:37 smithi038 ceph-mon[92422]: pgmap v18541: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-10T00:21:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33808.892489 secs 2023-03-10T00:21:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33808.892489 secs 2023-03-10T00:21:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34270.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:21:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33808.892489 secs 2023-03-10T00:21:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:39 smithi181 ceph-mon[94189]: pgmap v18542: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 7 op/s 2023-03-10T00:21:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:39 smithi038 ceph-mon[92422]: pgmap v18542: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 7 op/s 2023-03-10T00:21:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:39 smithi122 ceph-mon[93869]: pgmap v18542: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 7 op/s 2023-03-10T00:21:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:41 smithi038 ceph-mon[92422]: pgmap v18543: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 5.2 KiB/s wr, 6 op/s 2023-03-10T00:21:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33811.463343 secs 2023-03-10T00:21:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:41 smithi122 ceph-mon[93869]: pgmap v18543: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 5.2 KiB/s wr, 6 op/s 2023-03-10T00:21:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33811.463343 secs 2023-03-10T00:21:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:41 smithi181 ceph-mon[94189]: pgmap v18543: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 5.2 KiB/s wr, 6 op/s 2023-03-10T00:21:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33811.463343 secs 2023-03-10T00:21:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34275.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:21:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:43 smithi122 ceph-mon[93869]: pgmap v18544: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 5.3 KiB/s wr, 7 op/s 2023-03-10T00:21:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33813.892547 secs 2023-03-10T00:21:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:43 smithi181 ceph-mon[94189]: pgmap v18544: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 5.3 KiB/s wr, 7 op/s 2023-03-10T00:21:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33813.892547 secs 2023-03-10T00:21:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:43 smithi038 ceph-mon[92422]: pgmap v18544: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 KiB/s rd, 5.3 KiB/s wr, 7 op/s 2023-03-10T00:21:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33813.892547 secs 2023-03-10T00:21:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:45 smithi122 ceph-mon[93869]: pgmap v18545: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 132 KiB/s wr, 6 op/s 2023-03-10T00:21:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:45 smithi181 ceph-mon[94189]: pgmap v18545: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 132 KiB/s wr, 6 op/s 2023-03-10T00:21:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:45 smithi038 ceph-mon[92422]: pgmap v18545: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 132 KiB/s wr, 6 op/s 2023-03-10T00:21:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33816.463401 secs 2023-03-10T00:21:46.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33816.463401 secs 2023-03-10T00:21:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33816.463401 secs 2023-03-10T00:21:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:47 smithi122 ceph-mon[93869]: pgmap v18546: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 132 KiB/s wr, 7 op/s 2023-03-10T00:21:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:47 smithi181 ceph-mon[94189]: pgmap v18546: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 132 KiB/s wr, 7 op/s 2023-03-10T00:21:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:47 smithi038 ceph-mon[92422]: pgmap v18546: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 132 KiB/s wr, 7 op/s 2023-03-10T00:21:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34280.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:21:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33818.892625 secs 2023-03-10T00:21:48.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33818.892625 secs 2023-03-10T00:21:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33818.892625 secs 2023-03-10T00:21:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:49 smithi122 ceph-mon[93869]: pgmap v18547: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 524 KiB/s rd, 347 KiB/s wr, 9 op/s 2023-03-10T00:21:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:21:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:21:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:49 smithi181 ceph-mon[94189]: pgmap v18547: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 524 KiB/s rd, 347 KiB/s wr, 9 op/s 2023-03-10T00:21:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:21:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:21:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:49 smithi038 ceph-mon[92422]: pgmap v18547: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 524 KiB/s rd, 347 KiB/s wr, 9 op/s 2023-03-10T00:21:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:21:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:21:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:50 smithi122 ceph-mon[93869]: pgmap v18548: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 522 KiB/s rd, 361 KiB/s wr, 8 op/s 2023-03-10T00:21:50.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:50 smithi181 ceph-mon[94189]: pgmap v18548: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 522 KiB/s rd, 361 KiB/s wr, 8 op/s 2023-03-10T00:21:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:50 smithi038 ceph-mon[92422]: pgmap v18548: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 522 KiB/s rd, 361 KiB/s wr, 8 op/s 2023-03-10T00:21:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33821.463460 secs 2023-03-10T00:21:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33821.463460 secs 2023-03-10T00:21:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33821.463460 secs 2023-03-10T00:21:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:21:52.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:21:52.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:52 smithi122 ceph-mon[93869]: pgmap v18549: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 519 KiB/s rd, 356 KiB/s wr, 8 op/s 2023-03-10T00:21:52.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:21:52.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:21:52.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:21:52.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:52 smithi181 ceph-mon[94189]: pgmap v18549: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 519 KiB/s rd, 356 KiB/s wr, 8 op/s 2023-03-10T00:21:52.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:21:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:21:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:21:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:52 smithi038 ceph-mon[92422]: pgmap v18549: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 519 KiB/s rd, 356 KiB/s wr, 8 op/s 2023-03-10T00:21:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:21:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33823.892714 secs 2023-03-10T00:21:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:21:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:21:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34285.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:21:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33823.892714 secs 2023-03-10T00:21:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:21:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:21:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33823.892714 secs 2023-03-10T00:21:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:21:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:21:54.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:54 smithi122 ceph-mon[93869]: pgmap v18550: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 664 KiB/s wr, 9 op/s 2023-03-10T00:21:54.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:54 smithi181 ceph-mon[94189]: pgmap v18550: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 664 KiB/s wr, 9 op/s 2023-03-10T00:21:54.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:54 smithi038 ceph-mon[92422]: pgmap v18550: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 664 KiB/s wr, 9 op/s 2023-03-10T00:21:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:21:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:21:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33826.463521 secs 2023-03-10T00:21:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:55 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:21:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:55 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:21:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:55 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:21:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:55 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:21:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:21:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:21:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33826.463521 secs 2023-03-10T00:21:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:55 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:21:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:55 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:21:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:55 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:21:55.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:55 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:21:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:21:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:21:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33826.463521 secs 2023-03-10T00:21:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:55 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:21:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:55 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:21:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:55 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:21:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:55 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:21:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:56 smithi122 ceph-mon[93869]: pgmap v18551: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 986 KiB/s rd, 538 KiB/s wr, 10 op/s 2023-03-10T00:21:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:21:56.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:21:56.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:21:56.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:21:56.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:21:56.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:56 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-10T00:21:56.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:56 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-10T00:21:56.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:21:56.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:21:56.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:21:56.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:21:56.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:21:56.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:21:56.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:56 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-10T00:21:56.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:56 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-10T00:21:56.980 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:21:56.980 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:21:56.980 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:21:56.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:56 smithi181 ceph-mon[94189]: pgmap v18551: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 986 KiB/s rd, 538 KiB/s wr, 10 op/s 2023-03-10T00:21:56.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:21:56.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:21:56.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:21:56.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:21:56.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:21:56.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:56 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-10T00:21:56.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:56 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-10T00:21:56.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:21:56.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:21:56.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:21:56.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:21:56.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:21:56.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:21:56.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:56 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-10T00:21:56.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:56 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-10T00:21:56.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:21:56.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:21:56.984 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:21:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:56 smithi038 ceph-mon[92422]: pgmap v18551: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 986 KiB/s rd, 538 KiB/s wr, 10 op/s 2023-03-10T00:21:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:21:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:21:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:21:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:21:57.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:21:57.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:56 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-10T00:21:57.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:56 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-10T00:21:57.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:21:57.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:21:57.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:21:57.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:21:57.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:21:57.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:21:57.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:56 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-10T00:21:57.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:56 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-10T00:21:57.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:21:57.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:21:57.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:21:58.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34290.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:21:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33828.892779 secs 2023-03-10T00:21:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:21:58 smithi122 ceph-mon[93869]: pgmap v18552: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1019 KiB/s rd, 569 KiB/s wr, 13 op/s 2023-03-10T00:21:58.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33828.892779 secs 2023-03-10T00:21:58.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:21:58 smithi181 ceph-mon[94189]: pgmap v18552: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1019 KiB/s rd, 569 KiB/s wr, 13 op/s 2023-03-10T00:21:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33828.892779 secs 2023-03-10T00:21:59.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:21:58 smithi038 ceph-mon[92422]: pgmap v18552: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1019 KiB/s rd, 569 KiB/s wr, 13 op/s 2023-03-10T00:22:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:22:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:22:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:22:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:22:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:22:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:22:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:01 smithi122 ceph-mon[93869]: pgmap v18553: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 941 KiB/s rd, 667 KiB/s wr, 13 op/s 2023-03-10T00:22:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33831.463580 secs 2023-03-10T00:22:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:22:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:22:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:01 smithi181 ceph-mon[94189]: pgmap v18553: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 941 KiB/s rd, 667 KiB/s wr, 13 op/s 2023-03-10T00:22:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33831.463580 secs 2023-03-10T00:22:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:22:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:22:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:01 smithi038 ceph-mon[92422]: pgmap v18553: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 941 KiB/s rd, 667 KiB/s wr, 13 op/s 2023-03-10T00:22:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33831.463580 secs 2023-03-10T00:22:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:22:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:22:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:03 smithi122 ceph-mon[93869]: pgmap v18554: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 942 KiB/s rd, 653 KiB/s wr, 14 op/s 2023-03-10T00:22:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:03 smithi181 ceph-mon[94189]: pgmap v18554: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 942 KiB/s rd, 653 KiB/s wr, 14 op/s 2023-03-10T00:22:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:03 smithi038 ceph-mon[92422]: pgmap v18554: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 942 KiB/s rd, 653 KiB/s wr, 14 op/s 2023-03-10T00:22:03.802 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34295.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:22:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:05 smithi122 ceph-mon[93869]: pgmap v18555: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 759 KiB/s wr, 16 op/s 2023-03-10T00:22:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:05 smithi181 ceph-mon[94189]: pgmap v18555: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 759 KiB/s wr, 16 op/s 2023-03-10T00:22:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:05 smithi038 ceph-mon[92422]: pgmap v18555: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 759 KiB/s wr, 16 op/s 2023-03-10T00:22:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33836.463639 secs 2023-03-10T00:22:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:06 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:22:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33836.463639 secs 2023-03-10T00:22:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:06 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:22:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33836.463639 secs 2023-03-10T00:22:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:06 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:22:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:07 smithi122 ceph-mon[93869]: pgmap v18556: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 555 KiB/s rd, 527 KiB/s wr, 15 op/s 2023-03-10T00:22:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:07 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:22:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:22:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:22:07.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:07 smithi181 ceph-mon[94189]: pgmap v18556: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 555 KiB/s rd, 527 KiB/s wr, 15 op/s 2023-03-10T00:22:07.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:07 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:22:07.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:22:07.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:22:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:07 smithi038 ceph-mon[92422]: pgmap v18556: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 555 KiB/s rd, 527 KiB/s wr, 15 op/s 2023-03-10T00:22:07.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:07 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:22:07.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:22:07.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:22:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:22:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:22:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33838.892899 secs 2023-03-10T00:22:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:22:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:22:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33838.892899 secs 2023-03-10T00:22:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:22:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:22:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33838.892899 secs 2023-03-10T00:22:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34300.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:22:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:09 smithi122 ceph-mon[93869]: pgmap v18557: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 583 KiB/s rd, 601 KiB/s wr, 19 op/s 2023-03-10T00:22:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:22:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:22:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:09 smithi181 ceph-mon[94189]: pgmap v18557: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 583 KiB/s rd, 601 KiB/s wr, 19 op/s 2023-03-10T00:22:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:22:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:22:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:09 smithi038 ceph-mon[92422]: pgmap v18557: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 583 KiB/s rd, 601 KiB/s wr, 19 op/s 2023-03-10T00:22:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:22:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:22:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:10 smithi122 ceph-mon[93869]: osdmap e1375: 12 total, 12 up, 12 in 2023-03-10T00:22:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:10 smithi181 ceph-mon[94189]: osdmap e1375: 12 total, 12 up, 12 in 2023-03-10T00:22:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:10 smithi038 ceph-mon[92422]: osdmap e1375: 12 total, 12 up, 12 in 2023-03-10T00:22:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:11 smithi122 ceph-mon[93869]: pgmap v18559: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 566 KiB/s rd, 607 KiB/s wr, 31 op/s 2023-03-10T00:22:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33841.463697 secs 2023-03-10T00:22:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:11 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:22:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:11 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:22:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:11 smithi181 ceph-mon[94189]: pgmap v18559: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 566 KiB/s rd, 607 KiB/s wr, 31 op/s 2023-03-10T00:22:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33841.463697 secs 2023-03-10T00:22:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:11 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:22:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:11 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:22:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:11 smithi038 ceph-mon[92422]: pgmap v18559: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 566 KiB/s rd, 607 KiB/s wr, 31 op/s 2023-03-10T00:22:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33841.463697 secs 2023-03-10T00:22:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:11 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:22:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:11 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:22:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:13 smithi122 ceph-mon[93869]: pgmap v18560: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 739 KiB/s wr, 31 op/s 2023-03-10T00:22:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33843.893032 secs 2023-03-10T00:22:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:13 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:22:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:13 smithi181 ceph-mon[94189]: pgmap v18560: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 739 KiB/s wr, 31 op/s 2023-03-10T00:22:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33843.893032 secs 2023-03-10T00:22:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:13 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:22:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:13 smithi038 ceph-mon[92422]: pgmap v18560: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 739 KiB/s wr, 31 op/s 2023-03-10T00:22:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33843.893032 secs 2023-03-10T00:22:13.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:13 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:22:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34305.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:22:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:15 smithi122 ceph-mon[93869]: pgmap v18561: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 616 KiB/s rd, 658 KiB/s wr, 29 op/s 2023-03-10T00:22:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:15 smithi181 ceph-mon[94189]: pgmap v18561: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 616 KiB/s rd, 658 KiB/s wr, 29 op/s 2023-03-10T00:22:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:15 smithi038 ceph-mon[92422]: pgmap v18561: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 616 KiB/s rd, 658 KiB/s wr, 29 op/s 2023-03-10T00:22:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33846.463774 secs 2023-03-10T00:22:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:22:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:22:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33846.463774 secs 2023-03-10T00:22:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:22:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:22:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33846.463774 secs 2023-03-10T00:22:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:22:16.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:22:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:17 smithi122 ceph-mon[93869]: pgmap v18562: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 652 KiB/s rd, 693 KiB/s wr, 29 op/s 2023-03-10T00:22:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:22:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:22:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:17 smithi181 ceph-mon[94189]: pgmap v18562: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 652 KiB/s rd, 693 KiB/s wr, 29 op/s 2023-03-10T00:22:17.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:22:17.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:22:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:17 smithi038 ceph-mon[92422]: pgmap v18562: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 652 KiB/s rd, 693 KiB/s wr, 29 op/s 2023-03-10T00:22:17.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:22:17.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:22:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33848.893148 secs 2023-03-10T00:22:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:18 smithi122 ceph-mon[93869]: osdmap e1376: 12 total, 12 up, 12 in 2023-03-10T00:22:18.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33848.893148 secs 2023-03-10T00:22:18.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:18 smithi181 ceph-mon[94189]: osdmap e1376: 12 total, 12 up, 12 in 2023-03-10T00:22:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33848.893148 secs 2023-03-10T00:22:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:18 smithi038 ceph-mon[92422]: osdmap e1376: 12 total, 12 up, 12 in 2023-03-10T00:22:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34310.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:22:19.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:19 smithi122 ceph-mon[93869]: pgmap v18563: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 589 KiB/s rd, 605 KiB/s wr, 25 op/s 2023-03-10T00:22:19.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:19 smithi181 ceph-mon[94189]: pgmap v18563: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 589 KiB/s rd, 605 KiB/s wr, 25 op/s 2023-03-10T00:22:19.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:19 smithi038 ceph-mon[92422]: pgmap v18563: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 589 KiB/s rd, 605 KiB/s wr, 25 op/s 2023-03-10T00:22:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:21 smithi038 ceph-mon[92422]: pgmap v18565: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 306 KiB/s wr, 9 op/s 2023-03-10T00:22:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33851.463858 secs 2023-03-10T00:22:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:21 smithi122 ceph-mon[93869]: pgmap v18565: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 306 KiB/s wr, 9 op/s 2023-03-10T00:22:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33851.463858 secs 2023-03-10T00:22:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:21 smithi181 ceph-mon[94189]: pgmap v18565: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 306 KiB/s wr, 9 op/s 2023-03-10T00:22:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33851.463858 secs 2023-03-10T00:22:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:23 smithi038 ceph-mon[92422]: pgmap v18566: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 174 KiB/s wr, 8 op/s 2023-03-10T00:22:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33853.893217 secs 2023-03-10T00:22:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34315.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:22:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:23 smithi122 ceph-mon[93869]: pgmap v18566: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 174 KiB/s wr, 8 op/s 2023-03-10T00:22:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33853.893217 secs 2023-03-10T00:22:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:23 smithi181 ceph-mon[94189]: pgmap v18566: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 174 KiB/s wr, 8 op/s 2023-03-10T00:22:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33853.893217 secs 2023-03-10T00:22:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:25 smithi122 ceph-mon[93869]: pgmap v18567: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 126 KiB/s wr, 7 op/s 2023-03-10T00:22:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:25 smithi181 ceph-mon[94189]: pgmap v18567: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 126 KiB/s wr, 7 op/s 2023-03-10T00:22:26.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:25 smithi038 ceph-mon[92422]: pgmap v18567: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 126 KiB/s wr, 7 op/s 2023-03-10T00:22:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33856.463918 secs 2023-03-10T00:22:26.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33856.463918 secs 2023-03-10T00:22:26.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33856.463918 secs 2023-03-10T00:22:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:27 smithi122 ceph-mon[93869]: pgmap v18568: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 511 B/s wr, 7 op/s 2023-03-10T00:22:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:27 smithi181 ceph-mon[94189]: pgmap v18568: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 511 B/s wr, 7 op/s 2023-03-10T00:22:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:27 smithi038 ceph-mon[92422]: pgmap v18568: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 511 B/s wr, 7 op/s 2023-03-10T00:22:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34320.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:22:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33858.925578 secs 2023-03-10T00:22:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33858.925578 secs 2023-03-10T00:22:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33858.925578 secs 2023-03-10T00:22:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:29 smithi122 ceph-mon[93869]: pgmap v18569: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-10T00:22:29.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:29 smithi181 ceph-mon[94189]: pgmap v18569: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-10T00:22:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:29 smithi038 ceph-mon[92422]: pgmap v18569: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-10T00:22:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:30 smithi122 ceph-mon[93869]: pgmap v18570: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2023-03-10T00:22:30.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:30 smithi181 ceph-mon[94189]: pgmap v18570: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2023-03-10T00:22:30.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:30 smithi038 ceph-mon[92422]: pgmap v18570: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2023-03-10T00:22:31.981 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33861.463977 secs 2023-03-10T00:22:31.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33861.463977 secs 2023-03-10T00:22:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33861.463977 secs 2023-03-10T00:22:32.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:32 smithi122 ceph-mon[93869]: pgmap v18571: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-10T00:22:32.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:32 smithi181 ceph-mon[94189]: pgmap v18571: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-10T00:22:32.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:32 smithi038 ceph-mon[92422]: pgmap v18571: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-10T00:22:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34325.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:22:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33863.925682 secs 2023-03-10T00:22:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33863.925682 secs 2023-03-10T00:22:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33863.925682 secs 2023-03-10T00:22:34.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:34 smithi122 ceph-mon[93869]: pgmap v18572: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-10T00:22:34.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:34 smithi181 ceph-mon[94189]: pgmap v18572: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-10T00:22:34.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:34 smithi038 ceph-mon[92422]: pgmap v18572: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-10T00:22:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33866.464034 secs 2023-03-10T00:22:35.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33866.464034 secs 2023-03-10T00:22:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33866.464034 secs 2023-03-10T00:22:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:36 smithi122 ceph-mon[93869]: pgmap v18573: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 127 KiB/s wr, 7 op/s 2023-03-10T00:22:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:36 smithi181 ceph-mon[94189]: pgmap v18573: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 127 KiB/s wr, 7 op/s 2023-03-10T00:22:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:36 smithi038 ceph-mon[92422]: pgmap v18573: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 127 KiB/s wr, 7 op/s 2023-03-10T00:22:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34330.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:22:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33868.925787 secs 2023-03-10T00:22:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:38 smithi122 ceph-mon[93869]: pgmap v18574: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 127 KiB/s wr, 7 op/s 2023-03-10T00:22:38.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33868.925787 secs 2023-03-10T00:22:38.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:38 smithi181 ceph-mon[94189]: pgmap v18574: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 127 KiB/s wr, 7 op/s 2023-03-10T00:22:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33868.925787 secs 2023-03-10T00:22:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:38 smithi038 ceph-mon[92422]: pgmap v18574: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 127 KiB/s wr, 7 op/s 2023-03-10T00:22:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:40 smithi122 ceph-mon[93869]: pgmap v18575: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 127 KiB/s wr, 6 op/s 2023-03-10T00:22:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33871.464094 secs 2023-03-10T00:22:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:40 smithi181 ceph-mon[94189]: pgmap v18575: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 127 KiB/s wr, 6 op/s 2023-03-10T00:22:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33871.464094 secs 2023-03-10T00:22:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:40 smithi038 ceph-mon[92422]: pgmap v18575: 129 pgs: 129 active+clean; 12 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 127 KiB/s wr, 6 op/s 2023-03-10T00:22:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33871.464094 secs 2023-03-10T00:22:42.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:42 smithi122 ceph-mon[93869]: pgmap v18576: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 259 KiB/s wr, 9 op/s 2023-03-10T00:22:42.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:42 smithi181 ceph-mon[94189]: pgmap v18576: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 259 KiB/s wr, 9 op/s 2023-03-10T00:22:42.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:42 smithi038 ceph-mon[92422]: pgmap v18576: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 259 KiB/s wr, 9 op/s 2023-03-10T00:22:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34335.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:22:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:45 smithi122 ceph-mon[93869]: pgmap v18577: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 259 KiB/s wr, 8 op/s 2023-03-10T00:22:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:45 smithi181 ceph-mon[94189]: pgmap v18577: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 259 KiB/s wr, 8 op/s 2023-03-10T00:22:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:45 smithi038 ceph-mon[92422]: pgmap v18577: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 259 KiB/s wr, 8 op/s 2023-03-10T00:22:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33876.464155 secs 2023-03-10T00:22:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:46 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:22:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33876.464155 secs 2023-03-10T00:22:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:46 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:22:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33876.464155 secs 2023-03-10T00:22:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:46 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:22:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:47 smithi122 ceph-mon[93869]: pgmap v18578: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 472 KiB/s rd, 470 KiB/s wr, 9 op/s 2023-03-10T00:22:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:47 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:22:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:22:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:22:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:47 smithi181 ceph-mon[94189]: pgmap v18578: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 472 KiB/s rd, 470 KiB/s wr, 9 op/s 2023-03-10T00:22:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:47 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:22:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:22:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:22:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:47 smithi038 ceph-mon[92422]: pgmap v18578: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 472 KiB/s rd, 470 KiB/s wr, 9 op/s 2023-03-10T00:22:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:47 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:22:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:22:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:22:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34340.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:22:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:49 smithi122 ceph-mon[93869]: pgmap v18579: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 343 KiB/s wr, 9 op/s 2023-03-10T00:22:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:22:49.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:22:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:49 smithi181 ceph-mon[94189]: pgmap v18579: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 343 KiB/s wr, 9 op/s 2023-03-10T00:22:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:22:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:22:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:49 smithi038 ceph-mon[92422]: pgmap v18579: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 343 KiB/s wr, 9 op/s 2023-03-10T00:22:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:22:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:22:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:51 smithi122 ceph-mon[93869]: pgmap v18580: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 343 KiB/s wr, 7 op/s 2023-03-10T00:22:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33881.464214 secs 2023-03-10T00:22:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:51 smithi181 ceph-mon[94189]: pgmap v18580: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 343 KiB/s wr, 7 op/s 2023-03-10T00:22:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33881.464214 secs 2023-03-10T00:22:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:51 smithi038 ceph-mon[92422]: pgmap v18580: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 343 KiB/s wr, 7 op/s 2023-03-10T00:22:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33881.464214 secs 2023-03-10T00:22:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:22:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:22:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:22:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:22:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:22:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:22:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:53 smithi122 ceph-mon[93869]: pgmap v18581: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 644 KiB/s wr, 11 op/s 2023-03-10T00:22:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:22:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:22:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:53 smithi181 ceph-mon[94189]: pgmap v18581: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 644 KiB/s wr, 11 op/s 2023-03-10T00:22:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:22:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:22:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:53 smithi038 ceph-mon[92422]: pgmap v18581: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 644 KiB/s wr, 11 op/s 2023-03-10T00:22:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:22:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:22:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34345.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:22:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:55 smithi122 ceph-mon[93869]: pgmap v18582: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 951 KiB/s rd, 539 KiB/s wr, 9 op/s 2023-03-10T00:22:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:55 smithi181 ceph-mon[94189]: pgmap v18582: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 951 KiB/s rd, 539 KiB/s wr, 9 op/s 2023-03-10T00:22:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:55 smithi038 ceph-mon[92422]: pgmap v18582: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 951 KiB/s rd, 539 KiB/s wr, 9 op/s 2023-03-10T00:22:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33886.464274 secs 2023-03-10T00:22:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:22:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:22:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33886.464274 secs 2023-03-10T00:22:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:22:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:22:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33886.464274 secs 2023-03-10T00:22:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:22:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:22:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:57 smithi122 ceph-mon[93869]: pgmap v18583: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 992 KiB/s rd, 580 KiB/s wr, 10 op/s 2023-03-10T00:22:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:57 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:22:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:57 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:22:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:57 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:22:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:57 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:22:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:57 smithi181 ceph-mon[94189]: pgmap v18583: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 992 KiB/s rd, 580 KiB/s wr, 10 op/s 2023-03-10T00:22:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:57 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:22:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:57 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:22:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:57 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:22:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:57 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:22:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:57 smithi038 ceph-mon[92422]: pgmap v18583: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 992 KiB/s rd, 580 KiB/s wr, 10 op/s 2023-03-10T00:22:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:57 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:22:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:57 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:22:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:57 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:22:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:57 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:22:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:58 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 33888.926093 secs 2023-03-10T00:22:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:58 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 33888.926093 secs 2023-03-10T00:22:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:58 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 33888.926093 secs 2023-03-10T00:22:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34350.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:22:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:59 smithi122 ceph-mon[93869]: pgmap v18584: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 926 KiB/s rd, 513 KiB/s wr, 11 op/s 2023-03-10T00:22:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:59 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:22:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:22:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:22:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:22:59.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:59 smithi181 ceph-mon[94189]: pgmap v18584: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 926 KiB/s rd, 513 KiB/s wr, 11 op/s 2023-03-10T00:22:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:59 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:22:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:22:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:22:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:22:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:59 smithi038 ceph-mon[92422]: pgmap v18584: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 926 KiB/s rd, 513 KiB/s wr, 11 op/s 2023-03-10T00:22:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:59 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:22:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:22:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:22:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:23:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:23:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:23:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:23:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:23:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:23:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:23:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:01 smithi122 ceph-mon[93869]: pgmap v18585: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 937 KiB/s rd, 513 KiB/s wr, 10 op/s 2023-03-10T00:23:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33891.464359 secs 2023-03-10T00:23:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:23:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:23:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:01 smithi181 ceph-mon[94189]: pgmap v18585: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 937 KiB/s rd, 513 KiB/s wr, 10 op/s 2023-03-10T00:23:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33891.464359 secs 2023-03-10T00:23:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:23:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:23:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:01 smithi038 ceph-mon[92422]: pgmap v18585: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 937 KiB/s rd, 513 KiB/s wr, 10 op/s 2023-03-10T00:23:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33891.464359 secs 2023-03-10T00:23:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:23:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:23:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:03 smithi122 ceph-mon[93869]: pgmap v18586: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1020 KiB/s rd, 595 KiB/s wr, 13 op/s 2023-03-10T00:23:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:03 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 33893.926135 secs 2023-03-10T00:23:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:03 smithi181 ceph-mon[94189]: pgmap v18586: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1020 KiB/s rd, 595 KiB/s wr, 13 op/s 2023-03-10T00:23:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:03 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 33893.926135 secs 2023-03-10T00:23:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:03 smithi038 ceph-mon[92422]: pgmap v18586: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1020 KiB/s rd, 595 KiB/s wr, 13 op/s 2023-03-10T00:23:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:03 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 33893.926135 secs 2023-03-10T00:23:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34355.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:23:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:05 smithi181 ceph-mon[94189]: pgmap v18587: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 550 KiB/s wr, 13 op/s 2023-03-10T00:23:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:05 smithi038 ceph-mon[92422]: pgmap v18587: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 550 KiB/s wr, 13 op/s 2023-03-10T00:23:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:05 smithi122 ceph-mon[93869]: pgmap v18587: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 550 KiB/s wr, 13 op/s 2023-03-10T00:23:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33896.464419 secs 2023-03-10T00:23:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33896.464419 secs 2023-03-10T00:23:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33896.464419 secs 2023-03-10T00:23:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:07 smithi038 ceph-mon[92422]: pgmap v18588: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 720 KiB/s wr, 18 op/s 2023-03-10T00:23:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:07 smithi122 ceph-mon[93869]: pgmap v18588: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 720 KiB/s wr, 18 op/s 2023-03-10T00:23:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:07 smithi181 ceph-mon[94189]: pgmap v18588: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 720 KiB/s wr, 18 op/s 2023-03-10T00:23:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34360.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:23:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33898.926194 secs 2023-03-10T00:23:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:23:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:23:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33898.926194 secs 2023-03-10T00:23:08.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:23:08.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:23:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33898.926194 secs 2023-03-10T00:23:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:23:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:23:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:09 smithi122 ceph-mon[93869]: pgmap v18589: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 722 KiB/s rd, 730 KiB/s wr, 17 op/s 2023-03-10T00:23:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:23:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:23:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:09 smithi181 ceph-mon[94189]: pgmap v18589: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 722 KiB/s rd, 730 KiB/s wr, 17 op/s 2023-03-10T00:23:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:23:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:23:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:09 smithi038 ceph-mon[92422]: pgmap v18589: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 722 KiB/s rd, 730 KiB/s wr, 17 op/s 2023-03-10T00:23:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:23:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:23:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:10 smithi122 ceph-mon[93869]: osdmap e1377: 12 total, 12 up, 12 in 2023-03-10T00:23:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:10 smithi181 ceph-mon[94189]: osdmap e1377: 12 total, 12 up, 12 in 2023-03-10T00:23:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:10 smithi038 ceph-mon[92422]: osdmap e1377: 12 total, 12 up, 12 in 2023-03-10T00:23:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:11 smithi122 ceph-mon[93869]: pgmap v18591: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 775 KiB/s rd, 816 KiB/s wr, 31 op/s 2023-03-10T00:23:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33901.464478 secs 2023-03-10T00:23:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:11 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:23:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:11 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:23:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:11 smithi181 ceph-mon[94189]: pgmap v18591: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 775 KiB/s rd, 816 KiB/s wr, 31 op/s 2023-03-10T00:23:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33901.464478 secs 2023-03-10T00:23:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:11 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:23:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:11 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:23:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:11 smithi038 ceph-mon[92422]: pgmap v18591: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 775 KiB/s rd, 816 KiB/s wr, 31 op/s 2023-03-10T00:23:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33901.464478 secs 2023-03-10T00:23:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:11 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:23:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:11 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:23:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:23:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:23:12.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:23:12.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:23:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:23:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:23:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34365.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:23:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:13 smithi122 ceph-mon[93869]: pgmap v18592: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 779 KiB/s rd, 821 KiB/s wr, 30 op/s 2023-03-10T00:23:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33903.926287 secs 2023-03-10T00:23:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:23:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:23:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:13 smithi181 ceph-mon[94189]: pgmap v18592: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 779 KiB/s rd, 821 KiB/s wr, 30 op/s 2023-03-10T00:23:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33903.926287 secs 2023-03-10T00:23:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:23:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:23:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:13 smithi038 ceph-mon[92422]: pgmap v18592: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 779 KiB/s rd, 821 KiB/s wr, 30 op/s 2023-03-10T00:23:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33903.926287 secs 2023-03-10T00:23:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:23:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:23:14.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:14 smithi122 ceph-mon[93869]: pgmap v18593: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 472 KiB/s rd, 514 KiB/s wr, 26 op/s 2023-03-10T00:23:14.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:14 smithi181 ceph-mon[94189]: pgmap v18593: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 472 KiB/s rd, 514 KiB/s wr, 26 op/s 2023-03-10T00:23:14.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:14 smithi038 ceph-mon[92422]: pgmap v18593: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 472 KiB/s rd, 514 KiB/s wr, 26 op/s 2023-03-10T00:23:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33906.464562 secs 2023-03-10T00:23:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33906.464562 secs 2023-03-10T00:23:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33906.464562 secs 2023-03-10T00:23:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:16 smithi122 ceph-mon[93869]: pgmap v18594: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 442 KiB/s rd, 457 KiB/s wr, 23 op/s 2023-03-10T00:23:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:16 smithi181 ceph-mon[94189]: pgmap v18594: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 442 KiB/s rd, 457 KiB/s wr, 23 op/s 2023-03-10T00:23:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:16 smithi038 ceph-mon[92422]: pgmap v18594: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 442 KiB/s rd, 457 KiB/s wr, 23 op/s 2023-03-10T00:23:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34370.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:23:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33908.926347 secs 2023-03-10T00:23:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:18 smithi122 ceph-mon[93869]: pgmap v18595: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 397 KiB/s wr, 22 op/s 2023-03-10T00:23:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:18 smithi122 ceph-mon[93869]: osdmap e1378: 12 total, 12 up, 12 in 2023-03-10T00:23:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33908.926347 secs 2023-03-10T00:23:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:18 smithi181 ceph-mon[94189]: pgmap v18595: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 397 KiB/s wr, 22 op/s 2023-03-10T00:23:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:18 smithi181 ceph-mon[94189]: osdmap e1378: 12 total, 12 up, 12 in 2023-03-10T00:23:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33908.926347 secs 2023-03-10T00:23:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:18 smithi038 ceph-mon[92422]: pgmap v18595: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 397 KiB/s wr, 22 op/s 2023-03-10T00:23:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:18 smithi038 ceph-mon[92422]: osdmap e1378: 12 total, 12 up, 12 in 2023-03-10T00:23:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:20 smithi122 ceph-mon[93869]: pgmap v18597: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 409 KiB/s wr, 9 op/s 2023-03-10T00:23:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33911.464620 secs 2023-03-10T00:23:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:20 smithi181 ceph-mon[94189]: pgmap v18597: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 409 KiB/s wr, 9 op/s 2023-03-10T00:23:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33911.464620 secs 2023-03-10T00:23:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:20 smithi038 ceph-mon[92422]: pgmap v18597: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 409 KiB/s wr, 9 op/s 2023-03-10T00:23:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33911.464620 secs 2023-03-10T00:23:22.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:22 smithi122 ceph-mon[93869]: pgmap v18598: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 306 KiB/s wr, 8 op/s 2023-03-10T00:23:22.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:22 smithi181 ceph-mon[94189]: pgmap v18598: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 306 KiB/s wr, 8 op/s 2023-03-10T00:23:22.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:22 smithi038 ceph-mon[92422]: pgmap v18598: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 306 KiB/s wr, 8 op/s 2023-03-10T00:23:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34375.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:23:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33913.926430 secs 2023-03-10T00:23:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33913.926430 secs 2023-03-10T00:23:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33913.926430 secs 2023-03-10T00:23:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:25 smithi122 ceph-mon[93869]: pgmap v18599: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 306 KiB/s wr, 8 op/s 2023-03-10T00:23:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:25 smithi181 ceph-mon[94189]: pgmap v18599: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 306 KiB/s wr, 8 op/s 2023-03-10T00:23:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:25 smithi038 ceph-mon[92422]: pgmap v18599: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 306 KiB/s wr, 8 op/s 2023-03-10T00:23:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33916.464678 secs 2023-03-10T00:23:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33916.464678 secs 2023-03-10T00:23:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33916.464678 secs 2023-03-10T00:23:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:27 smithi122 ceph-mon[93869]: pgmap v18600: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 127 KiB/s wr, 7 op/s 2023-03-10T00:23:27.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:27 smithi181 ceph-mon[94189]: pgmap v18600: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 127 KiB/s wr, 7 op/s 2023-03-10T00:23:27.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:27 smithi038 ceph-mon[92422]: pgmap v18600: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 127 KiB/s wr, 7 op/s 2023-03-10T00:23:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33918.926523 secs 2023-03-10T00:23:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33918.926523 secs 2023-03-10T00:23:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33918.926523 secs 2023-03-10T00:23:28.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34380.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:23:29.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:29 smithi122 ceph-mon[93869]: pgmap v18601: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 126 KiB/s wr, 7 op/s 2023-03-10T00:23:29.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:29 smithi181 ceph-mon[94189]: pgmap v18601: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 126 KiB/s wr, 7 op/s 2023-03-10T00:23:29.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:29 smithi038 ceph-mon[92422]: pgmap v18601: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 126 KiB/s wr, 7 op/s 2023-03-10T00:23:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:31 smithi122 ceph-mon[93869]: pgmap v18602: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 112 KiB/s wr, 6 op/s 2023-03-10T00:23:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33921.464740 secs 2023-03-10T00:23:31.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:31 smithi181 ceph-mon[94189]: pgmap v18602: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 112 KiB/s wr, 6 op/s 2023-03-10T00:23:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33921.464740 secs 2023-03-10T00:23:31.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:31 smithi038 ceph-mon[92422]: pgmap v18602: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 115 KiB/s rd, 112 KiB/s wr, 6 op/s 2023-03-10T00:23:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33921.464740 secs 2023-03-10T00:23:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:33 smithi122 ceph-mon[93869]: pgmap v18603: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-10T00:23:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33923.926658 secs 2023-03-10T00:23:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:33 smithi181 ceph-mon[94189]: pgmap v18603: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-10T00:23:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33923.926658 secs 2023-03-10T00:23:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:33 smithi038 ceph-mon[92422]: pgmap v18603: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-10T00:23:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33923.926658 secs 2023-03-10T00:23:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34385.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:23:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:35 smithi122 ceph-mon[93869]: pgmap v18604: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-10T00:23:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:35 smithi181 ceph-mon[94189]: pgmap v18604: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-10T00:23:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:35 smithi038 ceph-mon[92422]: pgmap v18604: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-10T00:23:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33926.464798 secs 2023-03-10T00:23:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33926.464798 secs 2023-03-10T00:23:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33926.464798 secs 2023-03-10T00:23:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:37 smithi122 ceph-mon[93869]: pgmap v18605: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2023-03-10T00:23:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:37 smithi181 ceph-mon[94189]: pgmap v18605: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2023-03-10T00:23:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:37 smithi038 ceph-mon[92422]: pgmap v18605: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2023-03-10T00:23:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33928.926719 secs 2023-03-10T00:23:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33928.926719 secs 2023-03-10T00:23:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33928.926719 secs 2023-03-10T00:23:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34390.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:23:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:39 smithi122 ceph-mon[93869]: pgmap v18606: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 426 B/s wr, 7 op/s 2023-03-10T00:23:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:39 smithi181 ceph-mon[94189]: pgmap v18606: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 426 B/s wr, 7 op/s 2023-03-10T00:23:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:39 smithi038 ceph-mon[92422]: pgmap v18606: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 426 B/s wr, 7 op/s 2023-03-10T00:23:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:41 smithi181 ceph-mon[94189]: pgmap v18607: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-10T00:23:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33931.464857 secs 2023-03-10T00:23:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:41 smithi038 ceph-mon[92422]: pgmap v18607: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-10T00:23:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33931.464857 secs 2023-03-10T00:23:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:41 smithi122 ceph-mon[93869]: pgmap v18607: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 14 KiB/s wr, 7 op/s 2023-03-10T00:23:41.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33931.464857 secs 2023-03-10T00:23:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:43 smithi038 ceph-mon[92422]: pgmap v18608: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 132 KiB/s wr, 8 op/s 2023-03-10T00:23:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33933.926859 secs 2023-03-10T00:23:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34395.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:23:43.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:43 smithi122 ceph-mon[93869]: pgmap v18608: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 132 KiB/s wr, 8 op/s 2023-03-10T00:23:43.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33933.926859 secs 2023-03-10T00:23:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:43 smithi181 ceph-mon[94189]: pgmap v18608: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 132 KiB/s wr, 8 op/s 2023-03-10T00:23:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33933.926859 secs 2023-03-10T00:23:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:45 smithi038 ceph-mon[92422]: pgmap v18609: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 133 KiB/s wr, 7 op/s 2023-03-10T00:23:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:45 smithi122 ceph-mon[93869]: pgmap v18609: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 133 KiB/s wr, 7 op/s 2023-03-10T00:23:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:45 smithi181 ceph-mon[94189]: pgmap v18609: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 133 KiB/s wr, 7 op/s 2023-03-10T00:23:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33936.464923 secs 2023-03-10T00:23:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33936.464923 secs 2023-03-10T00:23:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33936.464923 secs 2023-03-10T00:23:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:47 smithi122 ceph-mon[93869]: pgmap v18610: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 133 KiB/s wr, 9 op/s 2023-03-10T00:23:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:47 smithi181 ceph-mon[94189]: pgmap v18610: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 133 KiB/s wr, 9 op/s 2023-03-10T00:23:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:47 smithi038 ceph-mon[92422]: pgmap v18610: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 133 KiB/s wr, 9 op/s 2023-03-10T00:23:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34400.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:23:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33938.926937 secs 2023-03-10T00:23:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33938.926937 secs 2023-03-10T00:23:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33938.926937 secs 2023-03-10T00:23:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:49 smithi122 ceph-mon[93869]: pgmap v18611: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 356 KiB/s wr, 9 op/s 2023-03-10T00:23:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:23:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:23:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:49 smithi181 ceph-mon[94189]: pgmap v18611: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 356 KiB/s wr, 9 op/s 2023-03-10T00:23:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:23:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:23:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:49 smithi038 ceph-mon[92422]: pgmap v18611: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 356 KiB/s wr, 9 op/s 2023-03-10T00:23:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:23:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:23:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:51 smithi122 ceph-mon[93869]: pgmap v18612: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 356 KiB/s wr, 8 op/s 2023-03-10T00:23:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33941.464984 secs 2023-03-10T00:23:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:51 smithi181 ceph-mon[94189]: pgmap v18612: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 356 KiB/s wr, 8 op/s 2023-03-10T00:23:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33941.464984 secs 2023-03-10T00:23:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:51 smithi038 ceph-mon[92422]: pgmap v18612: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 356 KiB/s wr, 8 op/s 2023-03-10T00:23:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33941.464984 secs 2023-03-10T00:23:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:23:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:23:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:23:52.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:23:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:23:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:23:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34405.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:23:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:53 smithi122 ceph-mon[93869]: pgmap v18613: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 815 KiB/s rd, 342 KiB/s wr, 9 op/s 2023-03-10T00:23:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33943.926980 secs 2023-03-10T00:23:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:23:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:23:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:53 smithi181 ceph-mon[94189]: pgmap v18613: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 815 KiB/s rd, 342 KiB/s wr, 9 op/s 2023-03-10T00:23:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33943.926980 secs 2023-03-10T00:23:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:23:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:23:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:53 smithi038 ceph-mon[92422]: pgmap v18613: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 815 KiB/s rd, 342 KiB/s wr, 9 op/s 2023-03-10T00:23:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33943.926980 secs 2023-03-10T00:23:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:23:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:23:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:55 smithi122 ceph-mon[93869]: pgmap v18614: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1012 KiB/s rd, 541 KiB/s wr, 9 op/s 2023-03-10T00:23:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:55 smithi181 ceph-mon[94189]: pgmap v18614: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1012 KiB/s rd, 541 KiB/s wr, 9 op/s 2023-03-10T00:23:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:55 smithi038 ceph-mon[92422]: pgmap v18614: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1012 KiB/s rd, 541 KiB/s wr, 9 op/s 2023-03-10T00:23:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33946.465042 secs 2023-03-10T00:23:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:23:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:23:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33946.465042 secs 2023-03-10T00:23:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:23:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:23:56.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33946.465042 secs 2023-03-10T00:23:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:23:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:23:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:57 smithi122 ceph-mon[93869]: pgmap v18615: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 646 KiB/s wr, 11 op/s 2023-03-10T00:23:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:57 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:23:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:57 smithi181 ceph-mon[94189]: pgmap v18615: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 646 KiB/s wr, 11 op/s 2023-03-10T00:23:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:57 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:23:57.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:57 smithi038 ceph-mon[92422]: pgmap v18615: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 646 KiB/s wr, 11 op/s 2023-03-10T00:23:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:57 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:23:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34410.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:23:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33948.927093 secs 2023-03-10T00:23:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:58 smithi122 ceph-mon[93869]: pgmap v18616: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 727 KiB/s wr, 11 op/s 2023-03-10T00:23:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:23:59.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:23:59.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:23:59.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:23:59.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:23:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:23:59.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33948.927093 secs 2023-03-10T00:23:59.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:58 smithi181 ceph-mon[94189]: pgmap v18616: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 727 KiB/s wr, 11 op/s 2023-03-10T00:23:59.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:23:59.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:23:59.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:23:59.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:23:59.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:23:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:23:59.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33948.927093 secs 2023-03-10T00:23:59.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:58 smithi038 ceph-mon[92422]: pgmap v18616: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 727 KiB/s wr, 11 op/s 2023-03-10T00:23:59.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:23:59.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:23:59.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:23:59.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:23:59.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:23:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:24:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:24:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:24:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:00 smithi122 ceph-mon[93869]: pgmap v18617: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 987 KiB/s rd, 504 KiB/s wr, 10 op/s 2023-03-10T00:24:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:24:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:24:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:00 smithi181 ceph-mon[94189]: pgmap v18617: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 987 KiB/s rd, 504 KiB/s wr, 10 op/s 2023-03-10T00:24:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:24:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:24:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:00 smithi038 ceph-mon[92422]: pgmap v18617: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 987 KiB/s rd, 504 KiB/s wr, 10 op/s 2023-03-10T00:24:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33951.465103 secs 2023-03-10T00:24:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:24:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:24:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33951.465103 secs 2023-03-10T00:24:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:24:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:24:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33951.465103 secs 2023-03-10T00:24:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:24:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:24:02.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:02 smithi122 ceph-mon[93869]: pgmap v18618: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 817 KiB/s wr, 14 op/s 2023-03-10T00:24:02.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:02 smithi181 ceph-mon[94189]: pgmap v18618: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 817 KiB/s wr, 14 op/s 2023-03-10T00:24:02.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:02 smithi038 ceph-mon[92422]: pgmap v18618: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 817 KiB/s wr, 14 op/s 2023-03-10T00:24:03.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34415.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:24:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33953.927114 secs 2023-03-10T00:24:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33953.927114 secs 2023-03-10T00:24:04.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33953.927114 secs 2023-03-10T00:24:04.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:04 smithi122 ceph-mon[93869]: pgmap v18619: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 857 KiB/s rd, 842 KiB/s wr, 14 op/s 2023-03-10T00:24:04.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:04 smithi181 ceph-mon[94189]: pgmap v18619: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 857 KiB/s rd, 842 KiB/s wr, 14 op/s 2023-03-10T00:24:04.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:04 smithi038 ceph-mon[92422]: pgmap v18619: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 857 KiB/s rd, 842 KiB/s wr, 14 op/s 2023-03-10T00:24:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33956.465165 secs 2023-03-10T00:24:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33956.465165 secs 2023-03-10T00:24:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33956.465165 secs 2023-03-10T00:24:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:06 smithi122 ceph-mon[93869]: pgmap v18620: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 640 KiB/s rd, 646 KiB/s wr, 19 op/s 2023-03-10T00:24:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:06 smithi181 ceph-mon[94189]: pgmap v18620: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 640 KiB/s rd, 646 KiB/s wr, 19 op/s 2023-03-10T00:24:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:06 smithi038 ceph-mon[92422]: pgmap v18620: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 640 KiB/s rd, 646 KiB/s wr, 19 op/s 2023-03-10T00:24:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:24:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:24:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:24:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:24:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:24:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:24:08.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34420.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:24:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33958.927176 secs 2023-03-10T00:24:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:08 smithi122 ceph-mon[93869]: pgmap v18621: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 542 KiB/s wr, 17 op/s 2023-03-10T00:24:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33958.927176 secs 2023-03-10T00:24:08.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:08 smithi181 ceph-mon[94189]: pgmap v18621: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 542 KiB/s wr, 17 op/s 2023-03-10T00:24:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33958.927176 secs 2023-03-10T00:24:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:08 smithi038 ceph-mon[92422]: pgmap v18621: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 542 KiB/s wr, 17 op/s 2023-03-10T00:24:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:24:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:24:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:24:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:24:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:24:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:24:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:10 smithi122 ceph-mon[93869]: osdmap e1379: 12 total, 12 up, 12 in 2023-03-10T00:24:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:10 smithi122 ceph-mon[93869]: pgmap v18623: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 537 KiB/s rd, 578 KiB/s wr, 30 op/s 2023-03-10T00:24:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33961.465230 secs 2023-03-10T00:24:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:10 smithi181 ceph-mon[94189]: osdmap e1379: 12 total, 12 up, 12 in 2023-03-10T00:24:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:10 smithi181 ceph-mon[94189]: pgmap v18623: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 537 KiB/s rd, 578 KiB/s wr, 30 op/s 2023-03-10T00:24:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33961.465230 secs 2023-03-10T00:24:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:10 smithi038 ceph-mon[92422]: osdmap e1379: 12 total, 12 up, 12 in 2023-03-10T00:24:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:10 smithi038 ceph-mon[92422]: pgmap v18623: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 537 KiB/s rd, 578 KiB/s wr, 30 op/s 2023-03-10T00:24:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33961.465230 secs 2023-03-10T00:24:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:12 smithi122 ceph-mon[93869]: pgmap v18624: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 468 KiB/s wr, 30 op/s 2023-03-10T00:24:12.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:12 smithi181 ceph-mon[94189]: pgmap v18624: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 468 KiB/s wr, 30 op/s 2023-03-10T00:24:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:12 smithi038 ceph-mon[92422]: pgmap v18624: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 468 KiB/s wr, 30 op/s 2023-03-10T00:24:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34425.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:24:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33963.927236 secs 2023-03-10T00:24:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33963.927236 secs 2023-03-10T00:24:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33963.927236 secs 2023-03-10T00:24:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:15 smithi122 ceph-mon[93869]: pgmap v18625: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 543 KiB/s wr, 28 op/s 2023-03-10T00:24:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:15 smithi181 ceph-mon[94189]: pgmap v18625: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 543 KiB/s wr, 28 op/s 2023-03-10T00:24:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:15 smithi038 ceph-mon[92422]: pgmap v18625: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 543 KiB/s wr, 28 op/s 2023-03-10T00:24:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33966.465288 secs 2023-03-10T00:24:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33966.465288 secs 2023-03-10T00:24:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33966.465288 secs 2023-03-10T00:24:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:17 smithi122 ceph-mon[93869]: pgmap v18626: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 398 KiB/s wr, 24 op/s 2023-03-10T00:24:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:24:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:24:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:17 smithi181 ceph-mon[94189]: pgmap v18626: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 398 KiB/s wr, 24 op/s 2023-03-10T00:24:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:24:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:24:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:17 smithi038 ceph-mon[92422]: pgmap v18626: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 398 KiB/s wr, 24 op/s 2023-03-10T00:24:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:24:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:24:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33968.927300 secs 2023-03-10T00:24:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:18 smithi122 ceph-mon[93869]: osdmap e1380: 12 total, 12 up, 12 in 2023-03-10T00:24:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33968.927300 secs 2023-03-10T00:24:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:18 smithi181 ceph-mon[94189]: osdmap e1380: 12 total, 12 up, 12 in 2023-03-10T00:24:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33968.927300 secs 2023-03-10T00:24:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:18 smithi038 ceph-mon[92422]: osdmap e1380: 12 total, 12 up, 12 in 2023-03-10T00:24:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34430.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:24:19.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:19 smithi122 ceph-mon[93869]: pgmap v18627: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 508 KiB/s rd, 524 KiB/s wr, 24 op/s 2023-03-10T00:24:19.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:19 smithi181 ceph-mon[94189]: pgmap v18627: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 508 KiB/s rd, 524 KiB/s wr, 24 op/s 2023-03-10T00:24:19.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:19 smithi038 ceph-mon[92422]: pgmap v18627: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 508 KiB/s rd, 524 KiB/s wr, 24 op/s 2023-03-10T00:24:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:24:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:24:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:24:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:24:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:24:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:24:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:21 smithi122 ceph-mon[93869]: pgmap v18629: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 498 KiB/s wr, 10 op/s 2023-03-10T00:24:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33971.465356 secs 2023-03-10T00:24:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:21 smithi181 ceph-mon[94189]: pgmap v18629: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 498 KiB/s wr, 10 op/s 2023-03-10T00:24:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33971.465356 secs 2023-03-10T00:24:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:21 smithi038 ceph-mon[92422]: pgmap v18629: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 498 KiB/s wr, 10 op/s 2023-03-10T00:24:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33971.465356 secs 2023-03-10T00:24:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:23 smithi122 ceph-mon[93869]: pgmap v18630: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 233 KiB/s wr, 8 op/s 2023-03-10T00:24:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33973.927379 secs 2023-03-10T00:24:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:23 smithi181 ceph-mon[94189]: pgmap v18630: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 233 KiB/s wr, 8 op/s 2023-03-10T00:24:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33973.927379 secs 2023-03-10T00:24:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:23 smithi038 ceph-mon[92422]: pgmap v18630: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 233 KiB/s wr, 8 op/s 2023-03-10T00:24:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33973.927379 secs 2023-03-10T00:24:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34435.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:24:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:25 smithi122 ceph-mon[93869]: pgmap v18631: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 127 KiB/s wr, 7 op/s 2023-03-10T00:24:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:25 smithi181 ceph-mon[94189]: pgmap v18631: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 127 KiB/s wr, 7 op/s 2023-03-10T00:24:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:25 smithi038 ceph-mon[92422]: pgmap v18631: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 127 KiB/s wr, 7 op/s 2023-03-10T00:24:26.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33976.465461 secs 2023-03-10T00:24:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33976.465461 secs 2023-03-10T00:24:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33976.465461 secs 2023-03-10T00:24:27.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:27 smithi181 ceph-mon[94189]: pgmap v18632: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 126 KiB/s wr, 7 op/s 2023-03-10T00:24:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:27 smithi038 ceph-mon[92422]: pgmap v18632: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 126 KiB/s wr, 7 op/s 2023-03-10T00:24:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:27 smithi122 ceph-mon[93869]: pgmap v18632: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 126 KiB/s wr, 7 op/s 2023-03-10T00:24:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33978.927463 secs 2023-03-10T00:24:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34440.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:24:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33978.927463 secs 2023-03-10T00:24:28.994 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33978.927463 secs 2023-03-10T00:24:29.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:29 smithi038 ceph-mon[92422]: pgmap v18633: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-10T00:24:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:29 smithi122 ceph-mon[93869]: pgmap v18633: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-10T00:24:29.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:29 smithi181 ceph-mon[94189]: pgmap v18633: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-10T00:24:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:31 smithi038 ceph-mon[92422]: pgmap v18634: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2023-03-10T00:24:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33981.465518 secs 2023-03-10T00:24:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:31 smithi122 ceph-mon[93869]: pgmap v18634: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2023-03-10T00:24:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33981.465518 secs 2023-03-10T00:24:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:31 smithi181 ceph-mon[94189]: pgmap v18634: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2023-03-10T00:24:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33981.465518 secs 2023-03-10T00:24:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:33 smithi038 ceph-mon[92422]: pgmap v18635: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-10T00:24:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33983.927525 secs 2023-03-10T00:24:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34445.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:24:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:33 smithi122 ceph-mon[93869]: pgmap v18635: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-10T00:24:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33983.927525 secs 2023-03-10T00:24:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:33 smithi181 ceph-mon[94189]: pgmap v18635: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-10T00:24:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33983.927525 secs 2023-03-10T00:24:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:35 smithi122 ceph-mon[93869]: pgmap v18636: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-10T00:24:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:35 smithi181 ceph-mon[94189]: pgmap v18636: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-10T00:24:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:35 smithi038 ceph-mon[92422]: pgmap v18636: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-10T00:24:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33986.465576 secs 2023-03-10T00:24:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33986.465576 secs 2023-03-10T00:24:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33986.465576 secs 2023-03-10T00:24:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:37 smithi122 ceph-mon[93869]: pgmap v18637: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 8 op/s 2023-03-10T00:24:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:37 smithi181 ceph-mon[94189]: pgmap v18637: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 8 op/s 2023-03-10T00:24:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:37 smithi038 ceph-mon[92422]: pgmap v18637: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 8 op/s 2023-03-10T00:24:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34450.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:24:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33988.927585 secs 2023-03-10T00:24:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33988.927585 secs 2023-03-10T00:24:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33988.927585 secs 2023-03-10T00:24:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:39 smithi122 ceph-mon[93869]: pgmap v18638: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 128 KiB/s wr, 6 op/s 2023-03-10T00:24:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:39 smithi181 ceph-mon[94189]: pgmap v18638: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 128 KiB/s wr, 6 op/s 2023-03-10T00:24:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:39 smithi038 ceph-mon[92422]: pgmap v18638: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 128 KiB/s wr, 6 op/s 2023-03-10T00:24:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:41 smithi122 ceph-mon[93869]: pgmap v18639: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 240 KiB/s wr, 7 op/s 2023-03-10T00:24:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33991.465634 secs 2023-03-10T00:24:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:41 smithi181 ceph-mon[94189]: pgmap v18639: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 240 KiB/s wr, 7 op/s 2023-03-10T00:24:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33991.465634 secs 2023-03-10T00:24:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:41 smithi038 ceph-mon[92422]: pgmap v18639: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 240 KiB/s wr, 7 op/s 2023-03-10T00:24:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33991.465634 secs 2023-03-10T00:24:43.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34455.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:24:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:43 smithi122 ceph-mon[93869]: pgmap v18640: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 240 KiB/s wr, 9 op/s 2023-03-10T00:24:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:43 smithi181 ceph-mon[94189]: pgmap v18640: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 240 KiB/s wr, 9 op/s 2023-03-10T00:24:43.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:43 smithi038 ceph-mon[92422]: pgmap v18640: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 240 KiB/s wr, 9 op/s 2023-03-10T00:24:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:45 smithi122 ceph-mon[93869]: pgmap v18641: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 260 KiB/s wr, 7 op/s 2023-03-10T00:24:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:45 smithi181 ceph-mon[94189]: pgmap v18641: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 260 KiB/s wr, 7 op/s 2023-03-10T00:24:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:45 smithi038 ceph-mon[92422]: pgmap v18641: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 260 KiB/s wr, 7 op/s 2023-03-10T00:24:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 33996.465693 secs 2023-03-10T00:24:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:46 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:24:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 33996.465693 secs 2023-03-10T00:24:46.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:46 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:24:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 33996.465693 secs 2023-03-10T00:24:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:46 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:24:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:47 smithi122 ceph-mon[93869]: pgmap v18642: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 261 KiB/s wr, 9 op/s 2023-03-10T00:24:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:47 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:24:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:24:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:24:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:47 smithi181 ceph-mon[94189]: pgmap v18642: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 261 KiB/s wr, 9 op/s 2023-03-10T00:24:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:47 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:24:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:24:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:24:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:47 smithi038 ceph-mon[92422]: pgmap v18642: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 261 KiB/s wr, 9 op/s 2023-03-10T00:24:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:47 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:24:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:24:48.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:24:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34460.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:24:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:49 smithi122 ceph-mon[93869]: pgmap v18643: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 260 KiB/s wr, 7 op/s 2023-03-10T00:24:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:24:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:24:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:49 smithi181 ceph-mon[94189]: pgmap v18643: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 260 KiB/s wr, 7 op/s 2023-03-10T00:24:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:24:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:24:49.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:49 smithi038 ceph-mon[92422]: pgmap v18643: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 260 KiB/s wr, 7 op/s 2023-03-10T00:24:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:24:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:24:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:51 smithi122 ceph-mon[93869]: pgmap v18644: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 664 KiB/s rd, 779 KiB/s wr, 8 op/s 2023-03-10T00:24:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34001.465770 secs 2023-03-10T00:24:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:51 smithi181 ceph-mon[94189]: pgmap v18644: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 664 KiB/s rd, 779 KiB/s wr, 8 op/s 2023-03-10T00:24:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34001.465770 secs 2023-03-10T00:24:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:51 smithi038 ceph-mon[92422]: pgmap v18644: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 664 KiB/s rd, 779 KiB/s wr, 8 op/s 2023-03-10T00:24:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34001.465770 secs 2023-03-10T00:24:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:24:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:24:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:24:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:24:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:24:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:24:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:53 smithi122 ceph-mon[93869]: pgmap v18645: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 582 KiB/s rd, 722 KiB/s wr, 10 op/s 2023-03-10T00:24:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:24:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:24:53.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:53 smithi181 ceph-mon[94189]: pgmap v18645: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 582 KiB/s rd, 722 KiB/s wr, 10 op/s 2023-03-10T00:24:53.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:24:53.735 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:24:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:53 smithi038 ceph-mon[92422]: pgmap v18645: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 582 KiB/s rd, 722 KiB/s wr, 10 op/s 2023-03-10T00:24:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:24:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:24:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34465.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:24:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:55 smithi122 ceph-mon[93869]: pgmap v18646: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 852 KiB/s wr, 10 op/s 2023-03-10T00:24:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:55 smithi181 ceph-mon[94189]: pgmap v18646: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 852 KiB/s wr, 10 op/s 2023-03-10T00:24:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:55 smithi038 ceph-mon[92422]: pgmap v18646: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 852 KiB/s wr, 10 op/s 2023-03-10T00:24:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34006.465826 secs 2023-03-10T00:24:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:24:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:24:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34006.465826 secs 2023-03-10T00:24:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:24:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:24:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34006.465826 secs 2023-03-10T00:24:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:24:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:24:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:57 smithi181 ceph-mon[94189]: pgmap v18647: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 859 KiB/s wr, 12 op/s 2023-03-10T00:24:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:24:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:24:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:57 smithi038 ceph-mon[92422]: pgmap v18647: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 859 KiB/s wr, 12 op/s 2023-03-10T00:24:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:24:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:24:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:57 smithi122 ceph-mon[93869]: pgmap v18647: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 859 KiB/s wr, 12 op/s 2023-03-10T00:24:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:24:57.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:24:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34470.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:24:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:59 smithi038 ceph-mon[92422]: pgmap v18648: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 879 KiB/s wr, 11 op/s 2023-03-10T00:24:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:24:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:24:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:24:59.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:24:59.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:24:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:24:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:59 smithi122 ceph-mon[93869]: pgmap v18648: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 879 KiB/s wr, 11 op/s 2023-03-10T00:24:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:24:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:24:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:24:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:24:59.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:24:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:24:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:59 smithi181 ceph-mon[94189]: pgmap v18648: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 879 KiB/s wr, 11 op/s 2023-03-10T00:24:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:24:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:24:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:24:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:24:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:24:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:25:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:25:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:25:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:25:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:25:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:25:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:25:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:25:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:25:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:25:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:01 smithi038 ceph-mon[92422]: pgmap v18649: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1015 KiB/s wr, 11 op/s 2023-03-10T00:25:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34011.465902 secs 2023-03-10T00:25:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:01 smithi122 ceph-mon[93869]: pgmap v18649: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1015 KiB/s wr, 11 op/s 2023-03-10T00:25:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34011.465902 secs 2023-03-10T00:25:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:01 smithi181 ceph-mon[94189]: pgmap v18649: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1015 KiB/s wr, 11 op/s 2023-03-10T00:25:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34011.465902 secs 2023-03-10T00:25:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:03 smithi038 ceph-mon[92422]: pgmap v18650: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 887 KiB/s wr, 17 op/s 2023-03-10T00:25:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34013.928010 secs 2023-03-10T00:25:03.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34475.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:25:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:03 smithi122 ceph-mon[93869]: pgmap v18650: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 887 KiB/s wr, 17 op/s 2023-03-10T00:25:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34013.928010 secs 2023-03-10T00:25:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:03 smithi181 ceph-mon[94189]: pgmap v18650: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 887 KiB/s wr, 17 op/s 2023-03-10T00:25:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34013.928010 secs 2023-03-10T00:25:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:05 smithi038 ceph-mon[92422]: pgmap v18651: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 854 KiB/s wr, 27 op/s 2023-03-10T00:25:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:05 smithi122 ceph-mon[93869]: pgmap v18651: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 854 KiB/s wr, 27 op/s 2023-03-10T00:25:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:05 smithi181 ceph-mon[94189]: pgmap v18651: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 854 KiB/s wr, 27 op/s 2023-03-10T00:25:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34016.465961 secs 2023-03-10T00:25:06.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34016.465961 secs 2023-03-10T00:25:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34016.465961 secs 2023-03-10T00:25:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:07 smithi122 ceph-mon[93869]: pgmap v18652: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 728 KiB/s rd, 724 KiB/s wr, 27 op/s 2023-03-10T00:25:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:07 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:25:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:25:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:25:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:07 smithi181 ceph-mon[94189]: pgmap v18652: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 728 KiB/s rd, 724 KiB/s wr, 27 op/s 2023-03-10T00:25:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:07 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:25:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:25:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:25:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:07 smithi038 ceph-mon[92422]: pgmap v18652: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 728 KiB/s rd, 724 KiB/s wr, 27 op/s 2023-03-10T00:25:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:07 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:25:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:25:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:25:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34480.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:25:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34018.928069 secs 2023-03-10T00:25:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:25:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:25:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34018.928069 secs 2023-03-10T00:25:08.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:25:08.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:25:08.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34018.928069 secs 2023-03-10T00:25:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:25:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:25:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:09 smithi122 ceph-mon[93869]: pgmap v18653: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 662 KiB/s rd, 697 KiB/s wr, 26 op/s 2023-03-10T00:25:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:25:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:25:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:09 smithi181 ceph-mon[94189]: pgmap v18653: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 662 KiB/s rd, 697 KiB/s wr, 26 op/s 2023-03-10T00:25:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:25:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:25:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:09 smithi038 ceph-mon[92422]: pgmap v18653: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 662 KiB/s rd, 697 KiB/s wr, 26 op/s 2023-03-10T00:25:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:25:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:25:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:10 smithi122 ceph-mon[93869]: osdmap e1381: 12 total, 12 up, 12 in 2023-03-10T00:25:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:10 smithi181 ceph-mon[94189]: osdmap e1381: 12 total, 12 up, 12 in 2023-03-10T00:25:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:10 smithi038 ceph-mon[92422]: osdmap e1381: 12 total, 12 up, 12 in 2023-03-10T00:25:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:11 smithi122 ceph-mon[93869]: pgmap v18655: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 609 KiB/s rd, 649 KiB/s wr, 29 op/s 2023-03-10T00:25:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34021.466018 secs 2023-03-10T00:25:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:11 smithi181 ceph-mon[94189]: pgmap v18655: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 609 KiB/s rd, 649 KiB/s wr, 29 op/s 2023-03-10T00:25:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34021.466018 secs 2023-03-10T00:25:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:11 smithi038 ceph-mon[92422]: pgmap v18655: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 609 KiB/s rd, 649 KiB/s wr, 29 op/s 2023-03-10T00:25:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34021.466018 secs 2023-03-10T00:25:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34485.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:25:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:13 smithi122 ceph-mon[93869]: pgmap v18656: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 294 KiB/s wr, 24 op/s 2023-03-10T00:25:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34023.928128 secs 2023-03-10T00:25:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:13 smithi181 ceph-mon[94189]: pgmap v18656: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 294 KiB/s wr, 24 op/s 2023-03-10T00:25:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34023.928128 secs 2023-03-10T00:25:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:13 smithi038 ceph-mon[92422]: pgmap v18656: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 294 KiB/s wr, 24 op/s 2023-03-10T00:25:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34023.928128 secs 2023-03-10T00:25:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:15 smithi122 ceph-mon[93869]: pgmap v18657: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 345 KiB/s wr, 9 op/s 2023-03-10T00:25:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:15 smithi181 ceph-mon[94189]: pgmap v18657: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 345 KiB/s wr, 9 op/s 2023-03-10T00:25:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:15 smithi038 ceph-mon[92422]: pgmap v18657: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 345 KiB/s wr, 9 op/s 2023-03-10T00:25:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34026.466076 secs 2023-03-10T00:25:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34026.466076 secs 2023-03-10T00:25:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34026.466076 secs 2023-03-10T00:25:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:17 smithi122 ceph-mon[93869]: pgmap v18658: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 346 KiB/s wr, 9 op/s 2023-03-10T00:25:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:25:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:25:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:17 smithi181 ceph-mon[94189]: pgmap v18658: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 346 KiB/s wr, 9 op/s 2023-03-10T00:25:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:25:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:25:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:17 smithi038 ceph-mon[92422]: pgmap v18658: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 346 KiB/s wr, 9 op/s 2023-03-10T00:25:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:25:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:25:18.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34490.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:25:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34028.928185 secs 2023-03-10T00:25:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:18 smithi122 ceph-mon[93869]: pgmap v18659: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 472 KiB/s wr, 9 op/s 2023-03-10T00:25:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:18 smithi122 ceph-mon[93869]: osdmap e1382: 12 total, 12 up, 12 in 2023-03-10T00:25:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34028.928185 secs 2023-03-10T00:25:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:18 smithi181 ceph-mon[94189]: pgmap v18659: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 472 KiB/s wr, 9 op/s 2023-03-10T00:25:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:18 smithi181 ceph-mon[94189]: osdmap e1382: 12 total, 12 up, 12 in 2023-03-10T00:25:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34028.928185 secs 2023-03-10T00:25:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:18 smithi038 ceph-mon[92422]: pgmap v18659: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 472 KiB/s wr, 9 op/s 2023-03-10T00:25:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:18 smithi038 ceph-mon[92422]: osdmap e1382: 12 total, 12 up, 12 in 2023-03-10T00:25:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:25:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:25:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:25:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:25:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:25:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:25:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:21 smithi122 ceph-mon[93869]: pgmap v18661: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 472 KiB/s wr, 8 op/s 2023-03-10T00:25:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34031.466159 secs 2023-03-10T00:25:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:21 smithi181 ceph-mon[94189]: pgmap v18661: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 472 KiB/s wr, 8 op/s 2023-03-10T00:25:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34031.466159 secs 2023-03-10T00:25:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:21 smithi038 ceph-mon[92422]: pgmap v18661: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 472 KiB/s wr, 8 op/s 2023-03-10T00:25:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34031.466159 secs 2023-03-10T00:25:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:23 smithi122 ceph-mon[93869]: pgmap v18662: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 204 KiB/s wr, 7 op/s 2023-03-10T00:25:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34033.928261 secs 2023-03-10T00:25:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:23 smithi181 ceph-mon[94189]: pgmap v18662: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 204 KiB/s wr, 7 op/s 2023-03-10T00:25:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34033.928261 secs 2023-03-10T00:25:23.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:23 smithi038 ceph-mon[92422]: pgmap v18662: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 204 KiB/s wr, 7 op/s 2023-03-10T00:25:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34033.928261 secs 2023-03-10T00:25:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34495.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:25:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:25 smithi122 ceph-mon[93869]: pgmap v18663: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 127 KiB/s wr, 7 op/s 2023-03-10T00:25:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:25 smithi181 ceph-mon[94189]: pgmap v18663: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 127 KiB/s wr, 7 op/s 2023-03-10T00:25:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:25 smithi038 ceph-mon[92422]: pgmap v18663: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 127 KiB/s wr, 7 op/s 2023-03-10T00:25:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34036.466217 secs 2023-03-10T00:25:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34036.466217 secs 2023-03-10T00:25:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34036.466217 secs 2023-03-10T00:25:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:27 smithi122 ceph-mon[93869]: pgmap v18664: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 127 KiB/s wr, 7 op/s 2023-03-10T00:25:27.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:27 smithi181 ceph-mon[94189]: pgmap v18664: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 127 KiB/s wr, 7 op/s 2023-03-10T00:25:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:27 smithi038 ceph-mon[92422]: pgmap v18664: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 127 KiB/s wr, 7 op/s 2023-03-10T00:25:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34038.928319 secs 2023-03-10T00:25:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34038.928319 secs 2023-03-10T00:25:28.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34500.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:25:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34038.928319 secs 2023-03-10T00:25:29.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:29 smithi038 ceph-mon[92422]: pgmap v18665: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-10T00:25:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:29 smithi122 ceph-mon[93869]: pgmap v18665: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-10T00:25:29.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:29 smithi181 ceph-mon[94189]: pgmap v18665: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-10T00:25:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:31 smithi038 ceph-mon[92422]: pgmap v18666: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2023-03-10T00:25:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34041.466279 secs 2023-03-10T00:25:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:31 smithi122 ceph-mon[93869]: pgmap v18666: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2023-03-10T00:25:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34041.466279 secs 2023-03-10T00:25:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:31 smithi181 ceph-mon[94189]: pgmap v18666: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2023-03-10T00:25:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34041.466279 secs 2023-03-10T00:25:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34505.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:25:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:33 smithi122 ceph-mon[93869]: pgmap v18667: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 341 B/s wr, 8 op/s 2023-03-10T00:25:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34043.928377 secs 2023-03-10T00:25:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:33 smithi181 ceph-mon[94189]: pgmap v18667: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 341 B/s wr, 8 op/s 2023-03-10T00:25:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34043.928377 secs 2023-03-10T00:25:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:33 smithi038 ceph-mon[92422]: pgmap v18667: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 341 B/s wr, 8 op/s 2023-03-10T00:25:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34043.928377 secs 2023-03-10T00:25:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:35 smithi122 ceph-mon[93869]: pgmap v18668: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-10T00:25:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:35 smithi181 ceph-mon[94189]: pgmap v18668: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-10T00:25:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:35 smithi038 ceph-mon[92422]: pgmap v18668: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-10T00:25:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34046.466336 secs 2023-03-10T00:25:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34046.466336 secs 2023-03-10T00:25:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34046.466336 secs 2023-03-10T00:25:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:37 smithi122 ceph-mon[93869]: pgmap v18669: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 8 op/s 2023-03-10T00:25:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:37 smithi181 ceph-mon[94189]: pgmap v18669: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 8 op/s 2023-03-10T00:25:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:37 smithi038 ceph-mon[92422]: pgmap v18669: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 8 op/s 2023-03-10T00:25:38.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34510.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:25:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34048.928447 secs 2023-03-10T00:25:38.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34048.928447 secs 2023-03-10T00:25:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34048.928447 secs 2023-03-10T00:25:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:39 smithi122 ceph-mon[93869]: pgmap v18670: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 133 KiB/s wr, 7 op/s 2023-03-10T00:25:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:39 smithi181 ceph-mon[94189]: pgmap v18670: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 133 KiB/s wr, 7 op/s 2023-03-10T00:25:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:39 smithi038 ceph-mon[92422]: pgmap v18670: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 133 KiB/s wr, 7 op/s 2023-03-10T00:25:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:41 smithi122 ceph-mon[93869]: pgmap v18671: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 133 KiB/s wr, 6 op/s 2023-03-10T00:25:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34051.466394 secs 2023-03-10T00:25:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:41 smithi181 ceph-mon[94189]: pgmap v18671: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 133 KiB/s wr, 6 op/s 2023-03-10T00:25:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34051.466394 secs 2023-03-10T00:25:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:41 smithi038 ceph-mon[92422]: pgmap v18671: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 133 KiB/s wr, 6 op/s 2023-03-10T00:25:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34051.466394 secs 2023-03-10T00:25:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34515.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:25:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:43 smithi122 ceph-mon[93869]: pgmap v18672: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 133 KiB/s wr, 9 op/s 2023-03-10T00:25:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:43 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 34053.928528 secs 2023-03-10T00:25:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:43 smithi181 ceph-mon[94189]: pgmap v18672: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 133 KiB/s wr, 9 op/s 2023-03-10T00:25:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:43 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 34053.928528 secs 2023-03-10T00:25:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:43 smithi038 ceph-mon[92422]: pgmap v18672: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 133 KiB/s wr, 9 op/s 2023-03-10T00:25:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:43 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 34053.928528 secs 2023-03-10T00:25:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:45 smithi122 ceph-mon[93869]: pgmap v18673: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 133 KiB/s wr, 6 op/s 2023-03-10T00:25:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:45 smithi181 ceph-mon[94189]: pgmap v18673: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 133 KiB/s wr, 6 op/s 2023-03-10T00:25:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:45 smithi038 ceph-mon[92422]: pgmap v18673: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 133 KiB/s wr, 6 op/s 2023-03-10T00:25:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34056.466478 secs 2023-03-10T00:25:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34056.466478 secs 2023-03-10T00:25:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34056.466478 secs 2023-03-10T00:25:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:47 smithi122 ceph-mon[93869]: pgmap v18674: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 133 KiB/s wr, 8 op/s 2023-03-10T00:25:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:25:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:25:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:47 smithi181 ceph-mon[94189]: pgmap v18674: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 133 KiB/s wr, 8 op/s 2023-03-10T00:25:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:25:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:25:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:47 smithi038 ceph-mon[92422]: pgmap v18674: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 133 KiB/s wr, 8 op/s 2023-03-10T00:25:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:25:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:25:48.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34520.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:25:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:48 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 34058.928641 secs 2023-03-10T00:25:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:48 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 34058.928641 secs 2023-03-10T00:25:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:48 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 34058.928641 secs 2023-03-10T00:25:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:49 smithi122 ceph-mon[93869]: pgmap v18675: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 172 KiB/s wr, 6 op/s 2023-03-10T00:25:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:49 smithi181 ceph-mon[94189]: pgmap v18675: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 172 KiB/s wr, 6 op/s 2023-03-10T00:25:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:49 smithi038 ceph-mon[92422]: pgmap v18675: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 172 KiB/s wr, 6 op/s 2023-03-10T00:25:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:50 smithi122 ceph-mon[93869]: pgmap v18676: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 381 KiB/s wr, 6 op/s 2023-03-10T00:25:50.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:50 smithi181 ceph-mon[94189]: pgmap v18676: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 381 KiB/s wr, 6 op/s 2023-03-10T00:25:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:50 smithi038 ceph-mon[92422]: pgmap v18676: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 381 KiB/s wr, 6 op/s 2023-03-10T00:25:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34061.466537 secs 2023-03-10T00:25:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:25:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:25:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34061.466537 secs 2023-03-10T00:25:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:25:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:25:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34061.466537 secs 2023-03-10T00:25:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:25:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:25:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:52 smithi122 ceph-mon[93869]: pgmap v18677: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 490 KiB/s rd, 408 KiB/s wr, 9 op/s 2023-03-10T00:25:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:52 smithi181 ceph-mon[94189]: pgmap v18677: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 490 KiB/s rd, 408 KiB/s wr, 9 op/s 2023-03-10T00:25:53.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:52 smithi038 ceph-mon[92422]: pgmap v18677: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 490 KiB/s rd, 408 KiB/s wr, 9 op/s 2023-03-10T00:25:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34525.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:25:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:53 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 34063.928740 secs 2023-03-10T00:25:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:25:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:25:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:53 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 34063.928740 secs 2023-03-10T00:25:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:25:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:25:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:53 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 34063.928740 secs 2023-03-10T00:25:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:25:54.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:25:54.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:54 smithi122 ceph-mon[93869]: pgmap v18678: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 868 KiB/s rd, 544 KiB/s wr, 8 op/s 2023-03-10T00:25:54.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:54 smithi181 ceph-mon[94189]: pgmap v18678: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 868 KiB/s rd, 544 KiB/s wr, 8 op/s 2023-03-10T00:25:54.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:54 smithi038 ceph-mon[92422]: pgmap v18678: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 868 KiB/s rd, 544 KiB/s wr, 8 op/s 2023-03-10T00:25:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:25:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:25:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34066.466597 secs 2023-03-10T00:25:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:25:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:25:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34066.466597 secs 2023-03-10T00:25:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:25:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:25:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34066.466597 secs 2023-03-10T00:25:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:56 smithi122 ceph-mon[93869]: pgmap v18679: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 897 KiB/s rd, 575 KiB/s wr, 11 op/s 2023-03-10T00:25:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:56 smithi181 ceph-mon[94189]: pgmap v18679: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 897 KiB/s rd, 575 KiB/s wr, 11 op/s 2023-03-10T00:25:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:56 smithi038 ceph-mon[92422]: pgmap v18679: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 897 KiB/s rd, 575 KiB/s wr, 11 op/s 2023-03-10T00:25:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:25:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:25:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:25:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:25:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:25:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:25:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34530.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:25:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34068.928804 secs 2023-03-10T00:25:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:58 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:25:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:58 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:25:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:25:58 smithi122 ceph-mon[93869]: pgmap v18680: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 897 KiB/s rd, 576 KiB/s wr, 10 op/s 2023-03-10T00:25:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34068.928804 secs 2023-03-10T00:25:58.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:58 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:25:58.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:58 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:25:58.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:25:58 smithi181 ceph-mon[94189]: pgmap v18680: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 897 KiB/s rd, 576 KiB/s wr, 10 op/s 2023-03-10T00:25:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34068.928804 secs 2023-03-10T00:25:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:58 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:25:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:58 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:25:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:25:58 smithi038 ceph-mon[92422]: pgmap v18680: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 897 KiB/s rd, 576 KiB/s wr, 10 op/s 2023-03-10T00:26:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:26:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:26:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:26:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:26:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:26:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:26:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:26:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:26:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:26:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:01 smithi122 ceph-mon[93869]: pgmap v18681: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 948 KiB/s rd, 854 KiB/s wr, 12 op/s 2023-03-10T00:26:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34071.466652 secs 2023-03-10T00:26:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:26:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:26:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:01 smithi181 ceph-mon[94189]: pgmap v18681: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 948 KiB/s rd, 854 KiB/s wr, 12 op/s 2023-03-10T00:26:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34071.466652 secs 2023-03-10T00:26:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:26:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:26:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:01 smithi038 ceph-mon[92422]: pgmap v18681: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 948 KiB/s rd, 854 KiB/s wr, 12 op/s 2023-03-10T00:26:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34071.466652 secs 2023-03-10T00:26:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:26:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:26:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:26:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:26:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:26:02.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:26:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:26:02.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:26:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:26:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:26:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:26:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34535.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:26:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:03 smithi122 ceph-mon[93869]: pgmap v18682: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 639 KiB/s rd, 540 KiB/s wr, 14 op/s 2023-03-10T00:26:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34073.928912 secs 2023-03-10T00:26:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:03 smithi181 ceph-mon[94189]: pgmap v18682: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 639 KiB/s rd, 540 KiB/s wr, 14 op/s 2023-03-10T00:26:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34073.928912 secs 2023-03-10T00:26:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:03 smithi038 ceph-mon[92422]: pgmap v18682: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 639 KiB/s rd, 540 KiB/s wr, 14 op/s 2023-03-10T00:26:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34073.928912 secs 2023-03-10T00:26:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:05 smithi122 ceph-mon[93869]: pgmap v18683: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 924 KiB/s rd, 665 KiB/s wr, 15 op/s 2023-03-10T00:26:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:05 smithi181 ceph-mon[94189]: pgmap v18683: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 924 KiB/s rd, 665 KiB/s wr, 15 op/s 2023-03-10T00:26:05.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:05 smithi038 ceph-mon[92422]: pgmap v18683: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 924 KiB/s rd, 665 KiB/s wr, 15 op/s 2023-03-10T00:26:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34076.466710 secs 2023-03-10T00:26:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34076.466710 secs 2023-03-10T00:26:06.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34076.466710 secs 2023-03-10T00:26:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:07 smithi122 ceph-mon[93869]: pgmap v18684: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 598 KiB/s rd, 604 KiB/s wr, 19 op/s 2023-03-10T00:26:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:07 smithi181 ceph-mon[94189]: pgmap v18684: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 598 KiB/s rd, 604 KiB/s wr, 19 op/s 2023-03-10T00:26:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:07 smithi038 ceph-mon[92422]: pgmap v18684: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 598 KiB/s rd, 604 KiB/s wr, 19 op/s 2023-03-10T00:26:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34540.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:26:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34078.928971 secs 2023-03-10T00:26:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34078.928971 secs 2023-03-10T00:26:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34078.928971 secs 2023-03-10T00:26:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:09 smithi122 ceph-mon[93869]: pgmap v18685: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 569 KiB/s rd, 572 KiB/s wr, 16 op/s 2023-03-10T00:26:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:09 smithi181 ceph-mon[94189]: pgmap v18685: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 569 KiB/s rd, 572 KiB/s wr, 16 op/s 2023-03-10T00:26:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:09 smithi038 ceph-mon[92422]: pgmap v18685: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 569 KiB/s rd, 572 KiB/s wr, 16 op/s 2023-03-10T00:26:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:10 smithi122 ceph-mon[93869]: osdmap e1383: 12 total, 12 up, 12 in 2023-03-10T00:26:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:10 smithi122 ceph-mon[93869]: pgmap v18687: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 581 KiB/s wr, 29 op/s 2023-03-10T00:26:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:10 smithi181 ceph-mon[94189]: osdmap e1383: 12 total, 12 up, 12 in 2023-03-10T00:26:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:10 smithi181 ceph-mon[94189]: pgmap v18687: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 581 KiB/s wr, 29 op/s 2023-03-10T00:26:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:10 smithi038 ceph-mon[92422]: osdmap e1383: 12 total, 12 up, 12 in 2023-03-10T00:26:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:10 smithi038 ceph-mon[92422]: pgmap v18687: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 581 KiB/s wr, 29 op/s 2023-03-10T00:26:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34081.466769 secs 2023-03-10T00:26:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34081.466769 secs 2023-03-10T00:26:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34081.466769 secs 2023-03-10T00:26:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:12 smithi122 ceph-mon[93869]: pgmap v18688: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 507 KiB/s rd, 548 KiB/s wr, 29 op/s 2023-03-10T00:26:12.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:12 smithi181 ceph-mon[94189]: pgmap v18688: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 507 KiB/s rd, 548 KiB/s wr, 29 op/s 2023-03-10T00:26:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:12 smithi038 ceph-mon[92422]: pgmap v18688: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 507 KiB/s rd, 548 KiB/s wr, 29 op/s 2023-03-10T00:26:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34545.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:26:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34083.929032 secs 2023-03-10T00:26:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:26:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:26:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34083.929032 secs 2023-03-10T00:26:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:26:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:26:13.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34083.929032 secs 2023-03-10T00:26:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:26:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:26:14.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:14 smithi122 ceph-mon[93869]: pgmap v18689: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 503 KiB/s rd, 545 KiB/s wr, 26 op/s 2023-03-10T00:26:14.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:14 smithi181 ceph-mon[94189]: pgmap v18689: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 503 KiB/s rd, 545 KiB/s wr, 26 op/s 2023-03-10T00:26:14.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:14 smithi038 ceph-mon[92422]: pgmap v18689: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 503 KiB/s rd, 545 KiB/s wr, 26 op/s 2023-03-10T00:26:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:15 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:26:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:15 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:26:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34086.466826 secs 2023-03-10T00:26:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:15 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:26:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:15 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:26:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34086.466826 secs 2023-03-10T00:26:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:15 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:26:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:15 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:26:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34086.466826 secs 2023-03-10T00:26:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:16 smithi122 ceph-mon[93869]: pgmap v18690: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 493 KiB/s rd, 508 KiB/s wr, 23 op/s 2023-03-10T00:26:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:16 smithi181 ceph-mon[94189]: pgmap v18690: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 493 KiB/s rd, 508 KiB/s wr, 23 op/s 2023-03-10T00:26:16.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:16 smithi038 ceph-mon[92422]: pgmap v18690: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 493 KiB/s rd, 508 KiB/s wr, 23 op/s 2023-03-10T00:26:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34550.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:26:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34088.929090 secs 2023-03-10T00:26:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:18 smithi122 ceph-mon[93869]: pgmap v18691: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 493 KiB/s rd, 508 KiB/s wr, 23 op/s 2023-03-10T00:26:19.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:18 smithi122 ceph-mon[93869]: osdmap e1384: 12 total, 12 up, 12 in 2023-03-10T00:26:19.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34088.929090 secs 2023-03-10T00:26:19.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:18 smithi181 ceph-mon[94189]: pgmap v18691: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 493 KiB/s rd, 508 KiB/s wr, 23 op/s 2023-03-10T00:26:19.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:18 smithi181 ceph-mon[94189]: osdmap e1384: 12 total, 12 up, 12 in 2023-03-10T00:26:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34088.929090 secs 2023-03-10T00:26:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:18 smithi038 ceph-mon[92422]: pgmap v18691: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 493 KiB/s rd, 508 KiB/s wr, 23 op/s 2023-03-10T00:26:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:18 smithi038 ceph-mon[92422]: osdmap e1384: 12 total, 12 up, 12 in 2023-03-10T00:26:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:20 smithi122 ceph-mon[93869]: pgmap v18693: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 503 KiB/s rd, 308 KiB/s wr, 9 op/s 2023-03-10T00:26:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34091.466886 secs 2023-03-10T00:26:21.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:20 smithi181 ceph-mon[94189]: pgmap v18693: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 503 KiB/s rd, 308 KiB/s wr, 9 op/s 2023-03-10T00:26:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34091.466886 secs 2023-03-10T00:26:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:20 smithi038 ceph-mon[92422]: pgmap v18693: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 503 KiB/s rd, 308 KiB/s wr, 9 op/s 2023-03-10T00:26:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34091.466886 secs 2023-03-10T00:26:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:22 smithi122 ceph-mon[93869]: pgmap v18694: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 308 KiB/s wr, 9 op/s 2023-03-10T00:26:23.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:22 smithi181 ceph-mon[94189]: pgmap v18694: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 308 KiB/s wr, 9 op/s 2023-03-10T00:26:23.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:22 smithi038 ceph-mon[92422]: pgmap v18694: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 502 KiB/s rd, 308 KiB/s wr, 9 op/s 2023-03-10T00:26:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34555.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:26:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34093.929113 secs 2023-03-10T00:26:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34093.929113 secs 2023-03-10T00:26:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34093.929113 secs 2023-03-10T00:26:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:25 smithi122 ceph-mon[93869]: pgmap v18695: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 127 KiB/s wr, 8 op/s 2023-03-10T00:26:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:25 smithi181 ceph-mon[94189]: pgmap v18695: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 127 KiB/s wr, 8 op/s 2023-03-10T00:26:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:25 smithi038 ceph-mon[92422]: pgmap v18695: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 127 KiB/s wr, 8 op/s 2023-03-10T00:26:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34096.466942 secs 2023-03-10T00:26:26.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34096.466942 secs 2023-03-10T00:26:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34096.466942 secs 2023-03-10T00:26:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:27 smithi122 ceph-mon[93869]: pgmap v18696: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 75 KiB/s wr, 8 op/s 2023-03-10T00:26:27.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:27 smithi181 ceph-mon[94189]: pgmap v18696: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 75 KiB/s wr, 8 op/s 2023-03-10T00:26:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:27 smithi038 ceph-mon[92422]: pgmap v18696: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 75 KiB/s wr, 8 op/s 2023-03-10T00:26:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34098.929172 secs 2023-03-10T00:26:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34560.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:26:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34098.929172 secs 2023-03-10T00:26:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34098.929172 secs 2023-03-10T00:26:29.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:29 smithi038 ceph-mon[92422]: pgmap v18697: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 75 KiB/s wr, 8 op/s 2023-03-10T00:26:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:29 smithi122 ceph-mon[93869]: pgmap v18697: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 75 KiB/s wr, 8 op/s 2023-03-10T00:26:29.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:29 smithi181 ceph-mon[94189]: pgmap v18697: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 75 KiB/s wr, 8 op/s 2023-03-10T00:26:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:31 smithi038 ceph-mon[92422]: pgmap v18698: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 7 op/s 2023-03-10T00:26:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34101.467001 secs 2023-03-10T00:26:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:31 smithi122 ceph-mon[93869]: pgmap v18698: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 7 op/s 2023-03-10T00:26:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34101.467001 secs 2023-03-10T00:26:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:31 smithi181 ceph-mon[94189]: pgmap v18698: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 7 op/s 2023-03-10T00:26:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34101.467001 secs 2023-03-10T00:26:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:33 smithi038 ceph-mon[92422]: pgmap v18699: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 9 op/s 2023-03-10T00:26:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34103.929256 secs 2023-03-10T00:26:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34565.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:26:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:33 smithi122 ceph-mon[93869]: pgmap v18699: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 9 op/s 2023-03-10T00:26:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34103.929256 secs 2023-03-10T00:26:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:33 smithi181 ceph-mon[94189]: pgmap v18699: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 9 op/s 2023-03-10T00:26:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34103.929256 secs 2023-03-10T00:26:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:35 smithi122 ceph-mon[93869]: pgmap v18700: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 7 op/s 2023-03-10T00:26:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:35 smithi181 ceph-mon[94189]: pgmap v18700: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 7 op/s 2023-03-10T00:26:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:35 smithi038 ceph-mon[92422]: pgmap v18700: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 7 op/s 2023-03-10T00:26:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34106.467084 secs 2023-03-10T00:26:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34106.467084 secs 2023-03-10T00:26:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34106.467084 secs 2023-03-10T00:26:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:37 smithi122 ceph-mon[93869]: pgmap v18701: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 9 op/s 2023-03-10T00:26:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:37 smithi181 ceph-mon[94189]: pgmap v18701: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 9 op/s 2023-03-10T00:26:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:37 smithi038 ceph-mon[92422]: pgmap v18701: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 9 op/s 2023-03-10T00:26:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34570.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:26:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34108.929323 secs 2023-03-10T00:26:38.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34108.929323 secs 2023-03-10T00:26:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34108.929323 secs 2023-03-10T00:26:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:39 smithi122 ceph-mon[93869]: pgmap v18702: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-10T00:26:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:39 smithi181 ceph-mon[94189]: pgmap v18702: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-10T00:26:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:39 smithi038 ceph-mon[92422]: pgmap v18702: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-10T00:26:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:41 smithi122 ceph-mon[93869]: pgmap v18703: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-10T00:26:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34111.467146 secs 2023-03-10T00:26:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:41 smithi181 ceph-mon[94189]: pgmap v18703: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-10T00:26:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34111.467146 secs 2023-03-10T00:26:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:41 smithi038 ceph-mon[92422]: pgmap v18703: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-10T00:26:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34111.467146 secs 2023-03-10T00:26:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34575.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:26:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:43 smithi122 ceph-mon[93869]: pgmap v18704: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 341 B/s wr, 9 op/s 2023-03-10T00:26:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:43 smithi181 ceph-mon[94189]: pgmap v18704: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 341 B/s wr, 9 op/s 2023-03-10T00:26:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:43 smithi038 ceph-mon[92422]: pgmap v18704: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 341 B/s wr, 9 op/s 2023-03-10T00:26:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:45 smithi122 ceph-mon[93869]: pgmap v18705: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 511 B/s wr, 8 op/s 2023-03-10T00:26:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:45 smithi181 ceph-mon[94189]: pgmap v18705: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 511 B/s wr, 8 op/s 2023-03-10T00:26:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:45 smithi038 ceph-mon[92422]: pgmap v18705: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 511 B/s wr, 8 op/s 2023-03-10T00:26:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34116.467229 secs 2023-03-10T00:26:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:46 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:26:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34116.467229 secs 2023-03-10T00:26:46.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:46 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:26:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34116.467229 secs 2023-03-10T00:26:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:46 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:26:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:47 smithi122 ceph-mon[93869]: pgmap v18706: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 188 KiB/s wr, 11 op/s 2023-03-10T00:26:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:47 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:26:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:26:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:26:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:47 smithi181 ceph-mon[94189]: pgmap v18706: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 188 KiB/s wr, 11 op/s 2023-03-10T00:26:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:47 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:26:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:26:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:26:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:47 smithi038 ceph-mon[92422]: pgmap v18706: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 191 KiB/s rd, 188 KiB/s wr, 11 op/s 2023-03-10T00:26:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:47 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:26:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:26:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:26:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34580.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:26:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:26:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:26:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:26:48.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:26:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:26:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:26:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:49 smithi122 ceph-mon[93869]: pgmap v18707: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 188 KiB/s wr, 9 op/s 2023-03-10T00:26:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:26:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:26:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:49 smithi181 ceph-mon[94189]: pgmap v18707: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 188 KiB/s wr, 9 op/s 2023-03-10T00:26:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:26:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:26:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:49 smithi038 ceph-mon[92422]: pgmap v18707: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 188 KiB/s wr, 9 op/s 2023-03-10T00:26:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:26:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:26:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:51 smithi122 ceph-mon[93869]: pgmap v18708: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 188 KiB/s wr, 9 op/s 2023-03-10T00:26:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34121.467321 secs 2023-03-10T00:26:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:51 smithi181 ceph-mon[94189]: pgmap v18708: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 188 KiB/s wr, 9 op/s 2023-03-10T00:26:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34121.467321 secs 2023-03-10T00:26:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:51 smithi038 ceph-mon[92422]: pgmap v18708: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 188 KiB/s wr, 9 op/s 2023-03-10T00:26:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34121.467321 secs 2023-03-10T00:26:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34585.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:26:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:53 smithi122 ceph-mon[93869]: pgmap v18709: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 354 KiB/s wr, 12 op/s 2023-03-10T00:26:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:53 smithi181 ceph-mon[94189]: pgmap v18709: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 354 KiB/s wr, 12 op/s 2023-03-10T00:26:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:53 smithi038 ceph-mon[92422]: pgmap v18709: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 354 KiB/s wr, 12 op/s 2023-03-10T00:26:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:55 smithi122 ceph-mon[93869]: pgmap v18710: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 499 KiB/s rd, 496 KiB/s wr, 11 op/s 2023-03-10T00:26:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:26:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:26:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:55 smithi181 ceph-mon[94189]: pgmap v18710: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 499 KiB/s rd, 496 KiB/s wr, 11 op/s 2023-03-10T00:26:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:26:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:26:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:55 smithi038 ceph-mon[92422]: pgmap v18710: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 499 KiB/s rd, 496 KiB/s wr, 11 op/s 2023-03-10T00:26:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:26:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:26:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34126.467380 secs 2023-03-10T00:26:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34126.467380 secs 2023-03-10T00:26:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34126.467380 secs 2023-03-10T00:26:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:57 smithi122 ceph-mon[93869]: pgmap v18711: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 682 KiB/s rd, 667 KiB/s wr, 15 op/s 2023-03-10T00:26:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:26:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:26:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:57 smithi181 ceph-mon[94189]: pgmap v18711: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 682 KiB/s rd, 667 KiB/s wr, 15 op/s 2023-03-10T00:26:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:26:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:26:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:57 smithi038 ceph-mon[92422]: pgmap v18711: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 682 KiB/s rd, 667 KiB/s wr, 15 op/s 2023-03-10T00:26:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:26:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:26:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34590.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:26:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:26:59 smithi122 ceph-mon[93869]: pgmap v18712: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 694 KiB/s rd, 681 KiB/s wr, 12 op/s 2023-03-10T00:26:59.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:26:59 smithi181 ceph-mon[94189]: pgmap v18712: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 694 KiB/s rd, 681 KiB/s wr, 12 op/s 2023-03-10T00:26:59.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:26:59 smithi038 ceph-mon[92422]: pgmap v18712: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 694 KiB/s rd, 681 KiB/s wr, 12 op/s 2023-03-10T00:27:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:27:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:27:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:27:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:27:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:27:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:27:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:01 smithi181 ceph-mon[94189]: pgmap v18713: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 694 KiB/s rd, 681 KiB/s wr, 12 op/s 2023-03-10T00:27:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34131.467463 secs 2023-03-10T00:27:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:01 smithi038 ceph-mon[92422]: pgmap v18713: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 694 KiB/s rd, 681 KiB/s wr, 12 op/s 2023-03-10T00:27:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34131.467463 secs 2023-03-10T00:27:01.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:01 smithi122 ceph-mon[93869]: pgmap v18713: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 694 KiB/s rd, 681 KiB/s wr, 12 op/s 2023-03-10T00:27:01.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34131.467463 secs 2023-03-10T00:27:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:27:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:27:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:27:02.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:27:02.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:27:02.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:27:02.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:27:02.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:27:02.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:27:02.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:27:02.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:27:02.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:27:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:03 smithi038 ceph-mon[92422]: pgmap v18714: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 877 KiB/s rd, 884 KiB/s wr, 18 op/s 2023-03-10T00:27:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34133.929744 secs 2023-03-10T00:27:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34595.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:27:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:03 smithi122 ceph-mon[93869]: pgmap v18714: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 877 KiB/s rd, 884 KiB/s wr, 18 op/s 2023-03-10T00:27:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34133.929744 secs 2023-03-10T00:27:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:03 smithi181 ceph-mon[94189]: pgmap v18714: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 877 KiB/s rd, 884 KiB/s wr, 18 op/s 2023-03-10T00:27:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34133.929744 secs 2023-03-10T00:27:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:05 smithi122 ceph-mon[93869]: pgmap v18715: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 715 KiB/s rd, 739 KiB/s wr, 27 op/s 2023-03-10T00:27:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:05 smithi122 ceph-mon[93869]: osdmap e1385: 12 total, 12 up, 12 in 2023-03-10T00:27:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:05 smithi181 ceph-mon[94189]: pgmap v18715: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 715 KiB/s rd, 739 KiB/s wr, 27 op/s 2023-03-10T00:27:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:05 smithi181 ceph-mon[94189]: osdmap e1385: 12 total, 12 up, 12 in 2023-03-10T00:27:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:05 smithi038 ceph-mon[92422]: pgmap v18715: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 715 KiB/s rd, 739 KiB/s wr, 27 op/s 2023-03-10T00:27:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:05 smithi038 ceph-mon[92422]: osdmap e1385: 12 total, 12 up, 12 in 2023-03-10T00:27:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34136.467518 secs 2023-03-10T00:27:06.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34136.467518 secs 2023-03-10T00:27:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34136.467518 secs 2023-03-10T00:27:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:07 smithi122 ceph-mon[93869]: pgmap v18717: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 556 KiB/s rd, 598 KiB/s wr, 28 op/s 2023-03-10T00:27:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:07 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:27:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:27:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:27:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:07 smithi181 ceph-mon[94189]: pgmap v18717: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 556 KiB/s rd, 598 KiB/s wr, 28 op/s 2023-03-10T00:27:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:07 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:27:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:27:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:27:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:07 smithi038 ceph-mon[92422]: pgmap v18717: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 556 KiB/s rd, 598 KiB/s wr, 28 op/s 2023-03-10T00:27:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:07 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:27:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:27:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:27:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34600.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:27:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34138.929807 secs 2023-03-10T00:27:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:27:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:27:08.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34138.929807 secs 2023-03-10T00:27:08.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:27:08.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:27:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34138.929807 secs 2023-03-10T00:27:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:27:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:27:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:09 smithi122 ceph-mon[93869]: pgmap v18718: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 492 KiB/s wr, 28 op/s 2023-03-10T00:27:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:27:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:27:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:09 smithi181 ceph-mon[94189]: pgmap v18718: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 492 KiB/s wr, 28 op/s 2023-03-10T00:27:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:27:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:27:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:09 smithi038 ceph-mon[92422]: pgmap v18718: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 492 KiB/s wr, 28 op/s 2023-03-10T00:27:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:27:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:27:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:11 smithi122 ceph-mon[93869]: pgmap v18719: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 492 KiB/s wr, 28 op/s 2023-03-10T00:27:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34141.467575 secs 2023-03-10T00:27:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:11 smithi181 ceph-mon[94189]: pgmap v18719: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 492 KiB/s wr, 28 op/s 2023-03-10T00:27:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34141.467575 secs 2023-03-10T00:27:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:11 smithi038 ceph-mon[92422]: pgmap v18719: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 492 KiB/s wr, 28 op/s 2023-03-10T00:27:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34141.467575 secs 2023-03-10T00:27:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34605.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:27:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:13 smithi122 ceph-mon[93869]: pgmap v18720: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 248 KiB/s wr, 24 op/s 2023-03-10T00:27:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34143.929912 secs 2023-03-10T00:27:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:13 smithi122 ceph-mon[93869]: osdmap e1386: 12 total, 12 up, 12 in 2023-03-10T00:27:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:13 smithi181 ceph-mon[94189]: pgmap v18720: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 248 KiB/s wr, 24 op/s 2023-03-10T00:27:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34143.929912 secs 2023-03-10T00:27:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:13 smithi181 ceph-mon[94189]: osdmap e1386: 12 total, 12 up, 12 in 2023-03-10T00:27:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:13 smithi038 ceph-mon[92422]: pgmap v18720: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 248 KiB/s wr, 24 op/s 2023-03-10T00:27:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34143.929912 secs 2023-03-10T00:27:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:13 smithi038 ceph-mon[92422]: osdmap e1386: 12 total, 12 up, 12 in 2023-03-10T00:27:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:15 smithi122 ceph-mon[93869]: pgmap v18722: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 440 KiB/s rd, 437 KiB/s wr, 11 op/s 2023-03-10T00:27:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:15 smithi181 ceph-mon[94189]: pgmap v18722: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 440 KiB/s rd, 437 KiB/s wr, 11 op/s 2023-03-10T00:27:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:15 smithi038 ceph-mon[92422]: pgmap v18722: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 440 KiB/s rd, 437 KiB/s wr, 11 op/s 2023-03-10T00:27:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34146.467657 secs 2023-03-10T00:27:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34146.467657 secs 2023-03-10T00:27:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34146.467657 secs 2023-03-10T00:27:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:17 smithi122 ceph-mon[93869]: pgmap v18723: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 316 KiB/s wr, 9 op/s 2023-03-10T00:27:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:17 smithi181 ceph-mon[94189]: pgmap v18723: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 316 KiB/s wr, 9 op/s 2023-03-10T00:27:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:17 smithi038 ceph-mon[92422]: pgmap v18723: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 316 KiB/s wr, 9 op/s 2023-03-10T00:27:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34610.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:27:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34148.930010 secs 2023-03-10T00:27:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34148.930010 secs 2023-03-10T00:27:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34148.930010 secs 2023-03-10T00:27:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:19 smithi122 ceph-mon[93869]: pgmap v18724: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 182 KiB/s wr, 8 op/s 2023-03-10T00:27:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:19 smithi181 ceph-mon[94189]: pgmap v18724: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 182 KiB/s wr, 8 op/s 2023-03-10T00:27:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:19 smithi038 ceph-mon[92422]: pgmap v18724: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 182 KiB/s wr, 8 op/s 2023-03-10T00:27:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:21 smithi122 ceph-mon[93869]: pgmap v18725: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 182 KiB/s wr, 7 op/s 2023-03-10T00:27:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34151.467715 secs 2023-03-10T00:27:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:21 smithi181 ceph-mon[94189]: pgmap v18725: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 182 KiB/s wr, 7 op/s 2023-03-10T00:27:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34151.467715 secs 2023-03-10T00:27:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:21 smithi038 ceph-mon[92422]: pgmap v18725: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 182 KiB/s wr, 7 op/s 2023-03-10T00:27:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34151.467715 secs 2023-03-10T00:27:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34615.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:27:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:23 smithi122 ceph-mon[93869]: pgmap v18726: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 214 KiB/s wr, 7 op/s 2023-03-10T00:27:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34153.930091 secs 2023-03-10T00:27:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:23 smithi181 ceph-mon[94189]: pgmap v18726: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 214 KiB/s wr, 7 op/s 2023-03-10T00:27:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34153.930091 secs 2023-03-10T00:27:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:23 smithi038 ceph-mon[92422]: pgmap v18726: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 214 KiB/s wr, 7 op/s 2023-03-10T00:27:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34153.930091 secs 2023-03-10T00:27:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:25 smithi122 ceph-mon[93869]: pgmap v18727: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 124 KiB/s wr, 7 op/s 2023-03-10T00:27:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:25 smithi181 ceph-mon[94189]: pgmap v18727: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 124 KiB/s wr, 7 op/s 2023-03-10T00:27:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:25 smithi038 ceph-mon[92422]: pgmap v18727: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 124 KiB/s wr, 7 op/s 2023-03-10T00:27:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34156.467769 secs 2023-03-10T00:27:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:26 smithi122 ceph-mon[93869]: pgmap v18728: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 106 KiB/s wr, 8 op/s 2023-03-10T00:27:26.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34156.467769 secs 2023-03-10T00:27:26.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:26 smithi181 ceph-mon[94189]: pgmap v18728: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 106 KiB/s wr, 8 op/s 2023-03-10T00:27:26.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34156.467769 secs 2023-03-10T00:27:26.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:26 smithi038 ceph-mon[92422]: pgmap v18728: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 106 KiB/s wr, 8 op/s 2023-03-10T00:27:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34620.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:27:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34158.930116 secs 2023-03-10T00:27:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:28 smithi122 ceph-mon[93869]: pgmap v18729: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 106 KiB/s wr, 6 op/s 2023-03-10T00:27:29.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34158.930116 secs 2023-03-10T00:27:29.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:28 smithi181 ceph-mon[94189]: pgmap v18729: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 106 KiB/s wr, 6 op/s 2023-03-10T00:27:29.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34158.930116 secs 2023-03-10T00:27:29.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:28 smithi038 ceph-mon[92422]: pgmap v18729: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 106 KiB/s wr, 6 op/s 2023-03-10T00:27:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:30 smithi122 ceph-mon[93869]: pgmap v18730: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 106 KiB/s wr, 6 op/s 2023-03-10T00:27:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34161.467827 secs 2023-03-10T00:27:31.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:30 smithi181 ceph-mon[94189]: pgmap v18730: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 106 KiB/s wr, 6 op/s 2023-03-10T00:27:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34161.467827 secs 2023-03-10T00:27:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:30 smithi038 ceph-mon[92422]: pgmap v18730: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 106 KiB/s wr, 6 op/s 2023-03-10T00:27:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34161.467827 secs 2023-03-10T00:27:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:32 smithi122 ceph-mon[93869]: pgmap v18731: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 106 KiB/s wr, 8 op/s 2023-03-10T00:27:33.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:32 smithi181 ceph-mon[94189]: pgmap v18731: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 106 KiB/s wr, 8 op/s 2023-03-10T00:27:33.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:32 smithi038 ceph-mon[92422]: pgmap v18731: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 106 KiB/s wr, 8 op/s 2023-03-10T00:27:33.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34625.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:27:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34163.930177 secs 2023-03-10T00:27:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34163.930177 secs 2023-03-10T00:27:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34163.930177 secs 2023-03-10T00:27:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:35 smithi181 ceph-mon[94189]: pgmap v18732: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 79 KiB/s wr, 6 op/s 2023-03-10T00:27:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:35 smithi038 ceph-mon[92422]: pgmap v18732: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 79 KiB/s wr, 6 op/s 2023-03-10T00:27:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:35 smithi122 ceph-mon[93869]: pgmap v18732: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 79 KiB/s wr, 6 op/s 2023-03-10T00:27:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34166.467894 secs 2023-03-10T00:27:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34166.467894 secs 2023-03-10T00:27:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34166.467894 secs 2023-03-10T00:27:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:37 smithi038 ceph-mon[92422]: pgmap v18733: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 128 KiB/s wr, 8 op/s 2023-03-10T00:27:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:27:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:27:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:37 smithi122 ceph-mon[93869]: pgmap v18733: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 128 KiB/s wr, 8 op/s 2023-03-10T00:27:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:27:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:27:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:37 smithi181 ceph-mon[94189]: pgmap v18733: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 128 KiB/s wr, 8 op/s 2023-03-10T00:27:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:27:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:27:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34168.930258 secs 2023-03-10T00:27:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34630.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:27:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34168.930258 secs 2023-03-10T00:27:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34168.930258 secs 2023-03-10T00:27:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:39 smithi122 ceph-mon[93869]: pgmap v18734: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 128 KiB/s wr, 6 op/s 2023-03-10T00:27:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:39 smithi181 ceph-mon[94189]: pgmap v18734: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 128 KiB/s wr, 6 op/s 2023-03-10T00:27:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:39 smithi038 ceph-mon[92422]: pgmap v18734: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 128 KiB/s wr, 6 op/s 2023-03-10T00:27:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:27:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:27:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:27:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:27:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:27:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:27:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:41 smithi122 ceph-mon[93869]: pgmap v18735: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 128 KiB/s wr, 6 op/s 2023-03-10T00:27:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34171.467954 secs 2023-03-10T00:27:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:41 smithi181 ceph-mon[94189]: pgmap v18735: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 128 KiB/s wr, 6 op/s 2023-03-10T00:27:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34171.467954 secs 2023-03-10T00:27:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:41 smithi038 ceph-mon[92422]: pgmap v18735: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 128 KiB/s wr, 6 op/s 2023-03-10T00:27:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34171.467954 secs 2023-03-10T00:27:43.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34635.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:27:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:43 smithi122 ceph-mon[93869]: pgmap v18736: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 129 KiB/s wr, 9 op/s 2023-03-10T00:27:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:43 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 34173.930316 secs 2023-03-10T00:27:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:43 smithi181 ceph-mon[94189]: pgmap v18736: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 129 KiB/s wr, 9 op/s 2023-03-10T00:27:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:43 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 34173.930316 secs 2023-03-10T00:27:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:43 smithi038 ceph-mon[92422]: pgmap v18736: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 129 KiB/s wr, 9 op/s 2023-03-10T00:27:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:43 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 34173.930316 secs 2023-03-10T00:27:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:45 smithi122 ceph-mon[93869]: pgmap v18737: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 129 KiB/s wr, 7 op/s 2023-03-10T00:27:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:45 smithi181 ceph-mon[94189]: pgmap v18737: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 129 KiB/s wr, 7 op/s 2023-03-10T00:27:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:45 smithi038 ceph-mon[92422]: pgmap v18737: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 129 KiB/s wr, 7 op/s 2023-03-10T00:27:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34176.467984 secs 2023-03-10T00:27:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34176.467984 secs 2023-03-10T00:27:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34176.467984 secs 2023-03-10T00:27:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:47 smithi122 ceph-mon[93869]: pgmap v18738: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 342 KiB/s wr, 10 op/s 2023-03-10T00:27:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:27:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:27:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:47 smithi181 ceph-mon[94189]: pgmap v18738: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 342 KiB/s wr, 10 op/s 2023-03-10T00:27:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:27:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:27:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:47 smithi038 ceph-mon[92422]: pgmap v18738: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 342 KiB/s wr, 10 op/s 2023-03-10T00:27:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:27:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:27:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34640.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:27:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34178.930377 secs 2023-03-10T00:27:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34178.930377 secs 2023-03-10T00:27:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34178.930377 secs 2023-03-10T00:27:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:49 smithi122 ceph-mon[93869]: pgmap v18739: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 214 KiB/s wr, 8 op/s 2023-03-10T00:27:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:49 smithi181 ceph-mon[94189]: pgmap v18739: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 214 KiB/s wr, 8 op/s 2023-03-10T00:27:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:49 smithi038 ceph-mon[92422]: pgmap v18739: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 214 KiB/s wr, 8 op/s 2023-03-10T00:27:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:51 smithi122 ceph-mon[93869]: pgmap v18740: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 214 KiB/s wr, 7 op/s 2023-03-10T00:27:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34181.468090 secs 2023-03-10T00:27:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:27:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:27:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:51 smithi181 ceph-mon[94189]: pgmap v18740: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 214 KiB/s wr, 7 op/s 2023-03-10T00:27:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34181.468090 secs 2023-03-10T00:27:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:27:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:27:52.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:51 smithi038 ceph-mon[92422]: pgmap v18740: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 214 KiB/s wr, 7 op/s 2023-03-10T00:27:52.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34181.468090 secs 2023-03-10T00:27:52.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:27:52.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:27:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34645.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:27:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:53 smithi122 ceph-mon[93869]: pgmap v18741: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 242 KiB/s wr, 11 op/s 2023-03-10T00:27:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34183.930435 secs 2023-03-10T00:27:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:53 smithi181 ceph-mon[94189]: pgmap v18741: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 242 KiB/s wr, 11 op/s 2023-03-10T00:27:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34183.930435 secs 2023-03-10T00:27:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:53 smithi038 ceph-mon[92422]: pgmap v18741: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 242 KiB/s wr, 11 op/s 2023-03-10T00:27:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34183.930435 secs 2023-03-10T00:27:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:55 smithi122 ceph-mon[93869]: pgmap v18742: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 631 KiB/s rd, 496 KiB/s wr, 9 op/s 2023-03-10T00:27:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:55 smithi181 ceph-mon[94189]: pgmap v18742: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 631 KiB/s rd, 496 KiB/s wr, 9 op/s 2023-03-10T00:27:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:55 smithi038 ceph-mon[92422]: pgmap v18742: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 631 KiB/s rd, 496 KiB/s wr, 9 op/s 2023-03-10T00:27:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34186.468144 secs 2023-03-10T00:27:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:27:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:27:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34186.468144 secs 2023-03-10T00:27:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:27:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:27:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34186.468144 secs 2023-03-10T00:27:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:27:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:27:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:57 smithi122 ceph-mon[93869]: pgmap v18743: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 928 KiB/s rd, 782 KiB/s wr, 13 op/s 2023-03-10T00:27:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:27:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:27:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:57 smithi181 ceph-mon[94189]: pgmap v18743: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 928 KiB/s rd, 782 KiB/s wr, 13 op/s 2023-03-10T00:27:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:27:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:27:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:57 smithi038 ceph-mon[92422]: pgmap v18743: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 928 KiB/s rd, 782 KiB/s wr, 13 op/s 2023-03-10T00:27:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:27:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:27:58.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34650.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:27:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34188.930504 secs 2023-03-10T00:27:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34188.930504 secs 2023-03-10T00:27:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34188.930504 secs 2023-03-10T00:27:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:27:59 smithi122 ceph-mon[93869]: pgmap v18744: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 800 KiB/s rd, 656 KiB/s wr, 11 op/s 2023-03-10T00:27:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:27:59 smithi181 ceph-mon[94189]: pgmap v18744: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 800 KiB/s rd, 656 KiB/s wr, 11 op/s 2023-03-10T00:27:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:27:59 smithi038 ceph-mon[92422]: pgmap v18744: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 800 KiB/s rd, 656 KiB/s wr, 11 op/s 2023-03-10T00:28:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:01 smithi122 ceph-mon[93869]: pgmap v18745: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 800 KiB/s rd, 656 KiB/s wr, 11 op/s 2023-03-10T00:28:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34191.468202 secs 2023-03-10T00:28:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:01 smithi181 ceph-mon[94189]: pgmap v18745: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 800 KiB/s rd, 656 KiB/s wr, 11 op/s 2023-03-10T00:28:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34191.468202 secs 2023-03-10T00:28:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:01 smithi038 ceph-mon[92422]: pgmap v18745: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 800 KiB/s rd, 656 KiB/s wr, 11 op/s 2023-03-10T00:28:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34191.468202 secs 2023-03-10T00:28:02.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:02 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:28:02.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:02 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:28:02.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:02 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:28:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34655.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:28:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:03 smithi122 ceph-mon[93869]: pgmap v18746: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 912 KiB/s wr, 17 op/s 2023-03-10T00:28:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34193.930564 secs 2023-03-10T00:28:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:03 smithi181 ceph-mon[94189]: pgmap v18746: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 912 KiB/s wr, 17 op/s 2023-03-10T00:28:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34193.930564 secs 2023-03-10T00:28:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:03 smithi038 ceph-mon[92422]: pgmap v18746: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 912 KiB/s wr, 17 op/s 2023-03-10T00:28:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34193.930564 secs 2023-03-10T00:28:04.658 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:04 smithi038 ceph-mon[92422]: pgmap v18747: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 893 KiB/s rd, 906 KiB/s wr, 26 op/s 2023-03-10T00:28:04.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:04 smithi122 ceph-mon[93869]: pgmap v18747: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 893 KiB/s rd, 906 KiB/s wr, 26 op/s 2023-03-10T00:28:04.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:04 smithi181 ceph-mon[94189]: pgmap v18747: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 893 KiB/s rd, 906 KiB/s wr, 26 op/s 2023-03-10T00:28:05.924 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:05 smithi038 ceph-mon[92422]: osdmap e1387: 12 total, 12 up, 12 in 2023-03-10T00:28:05.924 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34196.468262 secs 2023-03-10T00:28:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:05 smithi122 ceph-mon[93869]: osdmap e1387: 12 total, 12 up, 12 in 2023-03-10T00:28:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34196.468262 secs 2023-03-10T00:28:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:05 smithi181 ceph-mon[94189]: osdmap e1387: 12 total, 12 up, 12 in 2023-03-10T00:28:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34196.468262 secs 2023-03-10T00:28:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:07 smithi122 ceph-mon[93869]: pgmap v18749: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 634 KiB/s rd, 675 KiB/s wr, 30 op/s 2023-03-10T00:28:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:07 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:28:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:07 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:28:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:07 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:28:07.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:07 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:28:07.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:07 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:28:07.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:07 smithi181 ceph-mon[94189]: pgmap v18749: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 634 KiB/s rd, 675 KiB/s wr, 30 op/s 2023-03-10T00:28:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:07 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:28:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:07 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:28:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:07 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:28:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:07 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:28:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:07 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:28:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:07 smithi038 ceph-mon[92422]: pgmap v18749: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 634 KiB/s rd, 675 KiB/s wr, 30 op/s 2023-03-10T00:28:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:07 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:28:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:07 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:28:07.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:07 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:28:07.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:07 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:28:07.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:07 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:28:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:28:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:28:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:28:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:28:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:28:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:28:08.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34660.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:28:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:09 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34198.930670 secs 2023-03-10T00:28:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:09 smithi122 ceph-mon[93869]: pgmap v18750: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 530 KiB/s rd, 570 KiB/s wr, 29 op/s 2023-03-10T00:28:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:28:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:28:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:09 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34198.930670 secs 2023-03-10T00:28:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:09 smithi181 ceph-mon[94189]: pgmap v18750: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 530 KiB/s rd, 570 KiB/s wr, 29 op/s 2023-03-10T00:28:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:28:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:28:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:09 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34198.930670 secs 2023-03-10T00:28:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:09 smithi038 ceph-mon[92422]: pgmap v18750: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 530 KiB/s rd, 570 KiB/s wr, 29 op/s 2023-03-10T00:28:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:28:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:28:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:11 smithi122 ceph-mon[93869]: pgmap v18751: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 530 KiB/s rd, 570 KiB/s wr, 29 op/s 2023-03-10T00:28:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34201.468321 secs 2023-03-10T00:28:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:11 smithi181 ceph-mon[94189]: pgmap v18751: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 530 KiB/s rd, 570 KiB/s wr, 29 op/s 2023-03-10T00:28:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34201.468321 secs 2023-03-10T00:28:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:11 smithi038 ceph-mon[92422]: pgmap v18751: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 530 KiB/s rd, 570 KiB/s wr, 29 op/s 2023-03-10T00:28:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34201.468321 secs 2023-03-10T00:28:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:13 smithi122 ceph-mon[93869]: pgmap v18752: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 323 KiB/s wr, 26 op/s 2023-03-10T00:28:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34203.930772 secs 2023-03-10T00:28:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:13 smithi181 ceph-mon[94189]: pgmap v18752: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 323 KiB/s wr, 26 op/s 2023-03-10T00:28:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34203.930772 secs 2023-03-10T00:28:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:13 smithi038 ceph-mon[92422]: pgmap v18752: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 295 KiB/s rd, 323 KiB/s wr, 26 op/s 2023-03-10T00:28:13.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34203.930772 secs 2023-03-10T00:28:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34665.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:28:14.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:14 smithi038 ceph-mon[92422]: osdmap e1388: 12 total, 12 up, 12 in 2023-03-10T00:28:14.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:14 smithi122 ceph-mon[93869]: osdmap e1388: 12 total, 12 up, 12 in 2023-03-10T00:28:14.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:14 smithi181 ceph-mon[94189]: osdmap e1388: 12 total, 12 up, 12 in 2023-03-10T00:28:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:15 smithi122 ceph-mon[93869]: pgmap v18754: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 510 KiB/s wr, 12 op/s 2023-03-10T00:28:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:15 smithi181 ceph-mon[94189]: pgmap v18754: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 510 KiB/s wr, 12 op/s 2023-03-10T00:28:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:15 smithi038 ceph-mon[92422]: pgmap v18754: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 510 KiB/s wr, 12 op/s 2023-03-10T00:28:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34206.468380 secs 2023-03-10T00:28:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:28:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:28:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34206.468380 secs 2023-03-10T00:28:16.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:28:16.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:28:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34206.468380 secs 2023-03-10T00:28:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:28:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:28:17.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:17 smithi122 ceph-mon[93869]: pgmap v18755: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 229 KiB/s wr, 9 op/s 2023-03-10T00:28:17.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:28:17.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:28:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:17 smithi181 ceph-mon[94189]: pgmap v18755: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 229 KiB/s wr, 9 op/s 2023-03-10T00:28:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:28:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:28:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:17 smithi038 ceph-mon[92422]: pgmap v18755: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 229 KiB/s wr, 9 op/s 2023-03-10T00:28:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:28:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:28:18.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34670.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:28:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34208.930861 secs 2023-03-10T00:28:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34208.930861 secs 2023-03-10T00:28:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34208.930861 secs 2023-03-10T00:28:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:19 smithi122 ceph-mon[93869]: pgmap v18756: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 229 KiB/s wr, 9 op/s 2023-03-10T00:28:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:19 smithi181 ceph-mon[94189]: pgmap v18756: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 229 KiB/s wr, 9 op/s 2023-03-10T00:28:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:19 smithi038 ceph-mon[92422]: pgmap v18756: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 229 KiB/s wr, 9 op/s 2023-03-10T00:28:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:21 smithi122 ceph-mon[93869]: pgmap v18757: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 229 KiB/s wr, 9 op/s 2023-03-10T00:28:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34211.468485 secs 2023-03-10T00:28:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:21 smithi181 ceph-mon[94189]: pgmap v18757: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 229 KiB/s wr, 9 op/s 2023-03-10T00:28:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34211.468485 secs 2023-03-10T00:28:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:21 smithi038 ceph-mon[92422]: pgmap v18757: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 229 KiB/s wr, 9 op/s 2023-03-10T00:28:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34211.468485 secs 2023-03-10T00:28:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34675.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:28:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:23 smithi122 ceph-mon[93869]: pgmap v18758: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 169 KiB/s wr, 8 op/s 2023-03-10T00:28:23.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34213.930921 secs 2023-03-10T00:28:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:23 smithi181 ceph-mon[94189]: pgmap v18758: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 169 KiB/s wr, 8 op/s 2023-03-10T00:28:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34213.930921 secs 2023-03-10T00:28:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:23 smithi038 ceph-mon[92422]: pgmap v18758: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 169 KiB/s wr, 8 op/s 2023-03-10T00:28:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34213.930921 secs 2023-03-10T00:28:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:25 smithi122 ceph-mon[93869]: pgmap v18759: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 98 KiB/s wr, 7 op/s 2023-03-10T00:28:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:25 smithi181 ceph-mon[94189]: pgmap v18759: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 98 KiB/s wr, 7 op/s 2023-03-10T00:28:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:25 smithi038 ceph-mon[92422]: pgmap v18759: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 98 KiB/s wr, 7 op/s 2023-03-10T00:28:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34216.468541 secs 2023-03-10T00:28:26.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34216.468541 secs 2023-03-10T00:28:26.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34216.468541 secs 2023-03-10T00:28:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:27 smithi122 ceph-mon[93869]: pgmap v18760: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 84 KiB/s wr, 8 op/s 2023-03-10T00:28:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:27 smithi181 ceph-mon[94189]: pgmap v18760: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 84 KiB/s wr, 8 op/s 2023-03-10T00:28:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:27 smithi038 ceph-mon[92422]: pgmap v18760: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 84 KiB/s wr, 8 op/s 2023-03-10T00:28:28.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34680.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:28:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34218.930981 secs 2023-03-10T00:28:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34218.930981 secs 2023-03-10T00:28:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34218.930981 secs 2023-03-10T00:28:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:29 smithi122 ceph-mon[93869]: pgmap v18761: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 84 KiB/s wr, 6 op/s 2023-03-10T00:28:29.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:29 smithi181 ceph-mon[94189]: pgmap v18761: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 84 KiB/s wr, 6 op/s 2023-03-10T00:28:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:29 smithi038 ceph-mon[92422]: pgmap v18761: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 84 KiB/s wr, 6 op/s 2023-03-10T00:28:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:31 smithi122 ceph-mon[93869]: pgmap v18762: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 84 KiB/s wr, 5 op/s 2023-03-10T00:28:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34221.468600 secs 2023-03-10T00:28:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:31 smithi181 ceph-mon[94189]: pgmap v18762: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 84 KiB/s wr, 5 op/s 2023-03-10T00:28:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34221.468600 secs 2023-03-10T00:28:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:31 smithi038 ceph-mon[92422]: pgmap v18762: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 84 KiB/s wr, 5 op/s 2023-03-10T00:28:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34221.468600 secs 2023-03-10T00:28:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34685.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:28:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:33 smithi122 ceph-mon[93869]: pgmap v18763: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 84 KiB/s wr, 7 op/s 2023-03-10T00:28:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34223.931043 secs 2023-03-10T00:28:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:33 smithi181 ceph-mon[94189]: pgmap v18763: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 84 KiB/s wr, 7 op/s 2023-03-10T00:28:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34223.931043 secs 2023-03-10T00:28:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:33 smithi038 ceph-mon[92422]: pgmap v18763: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 84 KiB/s wr, 7 op/s 2023-03-10T00:28:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34223.931043 secs 2023-03-10T00:28:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:35 smithi122 ceph-mon[93869]: pgmap v18764: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 84 KiB/s wr, 5 op/s 2023-03-10T00:28:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:35 smithi181 ceph-mon[94189]: pgmap v18764: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 84 KiB/s wr, 5 op/s 2023-03-10T00:28:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:35 smithi038 ceph-mon[92422]: pgmap v18764: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 84 KiB/s wr, 5 op/s 2023-03-10T00:28:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34226.468682 secs 2023-03-10T00:28:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34226.468682 secs 2023-03-10T00:28:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34226.468682 secs 2023-03-10T00:28:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:37 smithi122 ceph-mon[93869]: pgmap v18765: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 341 B/s wr, 7 op/s 2023-03-10T00:28:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:37 smithi181 ceph-mon[94189]: pgmap v18765: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 341 B/s wr, 7 op/s 2023-03-10T00:28:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:37 smithi038 ceph-mon[92422]: pgmap v18765: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 341 B/s wr, 7 op/s 2023-03-10T00:28:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34690.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:28:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34228.931159 secs 2023-03-10T00:28:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34228.931159 secs 2023-03-10T00:28:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34228.931159 secs 2023-03-10T00:28:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:39 smithi122 ceph-mon[93869]: pgmap v18766: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-10T00:28:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:39 smithi181 ceph-mon[94189]: pgmap v18766: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-10T00:28:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:39 smithi038 ceph-mon[92422]: pgmap v18766: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-10T00:28:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:41 smithi122 ceph-mon[93869]: pgmap v18767: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 5 op/s 2023-03-10T00:28:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34231.468741 secs 2023-03-10T00:28:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:41 smithi181 ceph-mon[94189]: pgmap v18767: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 5 op/s 2023-03-10T00:28:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34231.468741 secs 2023-03-10T00:28:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:41 smithi038 ceph-mon[92422]: pgmap v18767: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 17 KiB/s wr, 5 op/s 2023-03-10T00:28:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34231.468741 secs 2023-03-10T00:28:42.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:42 smithi122 ceph-mon[93869]: pgmap v18768: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 17 KiB/s wr, 8 op/s 2023-03-10T00:28:42.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:42 smithi181 ceph-mon[94189]: pgmap v18768: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 17 KiB/s wr, 8 op/s 2023-03-10T00:28:42.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:42 smithi038 ceph-mon[92422]: pgmap v18768: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 17 KiB/s wr, 8 op/s 2023-03-10T00:28:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34695.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:28:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:45 smithi038 ceph-mon[92422]: pgmap v18769: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 134 KiB/s wr, 7 op/s 2023-03-10T00:28:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:45 smithi122 ceph-mon[93869]: pgmap v18769: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 134 KiB/s wr, 7 op/s 2023-03-10T00:28:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:45 smithi181 ceph-mon[94189]: pgmap v18769: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 134 KiB/s wr, 7 op/s 2023-03-10T00:28:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34236.468798 secs 2023-03-10T00:28:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:46 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:28:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34236.468798 secs 2023-03-10T00:28:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:46 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:28:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34236.468798 secs 2023-03-10T00:28:46.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:46 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:28:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:47 smithi122 ceph-mon[93869]: pgmap v18770: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 135 KiB/s wr, 9 op/s 2023-03-10T00:28:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:47 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:28:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:28:47.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:28:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:47 smithi181 ceph-mon[94189]: pgmap v18770: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 135 KiB/s wr, 9 op/s 2023-03-10T00:28:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:47 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:28:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:28:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:28:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:47 smithi038 ceph-mon[92422]: pgmap v18770: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 135 KiB/s wr, 9 op/s 2023-03-10T00:28:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:47 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:28:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:28:48.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:28:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34700.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:28:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:49 smithi122 ceph-mon[93869]: pgmap v18771: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 134 KiB/s wr, 7 op/s 2023-03-10T00:28:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:28:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:28:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:49 smithi181 ceph-mon[94189]: pgmap v18771: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 134 KiB/s wr, 7 op/s 2023-03-10T00:28:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:28:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:28:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:49 smithi038 ceph-mon[92422]: pgmap v18771: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 134 KiB/s wr, 7 op/s 2023-03-10T00:28:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:28:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:28:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:51 smithi122 ceph-mon[93869]: pgmap v18772: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 526 KiB/s rd, 459 KiB/s wr, 7 op/s 2023-03-10T00:28:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34241.468852 secs 2023-03-10T00:28:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:51 smithi181 ceph-mon[94189]: pgmap v18772: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 526 KiB/s rd, 459 KiB/s wr, 7 op/s 2023-03-10T00:28:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34241.468852 secs 2023-03-10T00:28:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:51 smithi038 ceph-mon[92422]: pgmap v18772: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 526 KiB/s rd, 459 KiB/s wr, 7 op/s 2023-03-10T00:28:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34241.468852 secs 2023-03-10T00:28:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:28:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:28:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:28:52.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:28:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:28:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:28:53.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34705.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:28:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:53 smithi122 ceph-mon[93869]: pgmap v18773: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 665 KiB/s wr, 10 op/s 2023-03-10T00:28:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:53 smithi181 ceph-mon[94189]: pgmap v18773: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 665 KiB/s wr, 10 op/s 2023-03-10T00:28:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:53 smithi038 ceph-mon[92422]: pgmap v18773: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 665 KiB/s wr, 10 op/s 2023-03-10T00:28:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:55 smithi122 ceph-mon[93869]: pgmap v18774: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 731 KiB/s rd, 664 KiB/s wr, 8 op/s 2023-03-10T00:28:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:55 smithi181 ceph-mon[94189]: pgmap v18774: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 731 KiB/s rd, 664 KiB/s wr, 8 op/s 2023-03-10T00:28:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:55 smithi038 ceph-mon[92422]: pgmap v18774: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 731 KiB/s rd, 664 KiB/s wr, 8 op/s 2023-03-10T00:28:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34246.468929 secs 2023-03-10T00:28:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:28:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:28:56.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34246.468929 secs 2023-03-10T00:28:56.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:28:56.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:28:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34246.468929 secs 2023-03-10T00:28:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:28:57.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:28:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:57 smithi122 ceph-mon[93869]: pgmap v18775: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 857 KiB/s rd, 725 KiB/s wr, 12 op/s 2023-03-10T00:28:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:28:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:28:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:57 smithi181 ceph-mon[94189]: pgmap v18775: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 857 KiB/s rd, 725 KiB/s wr, 12 op/s 2023-03-10T00:28:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:28:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:28:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:57 smithi038 ceph-mon[92422]: pgmap v18775: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 857 KiB/s rd, 725 KiB/s wr, 12 op/s 2023-03-10T00:28:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:28:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:28:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34710.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:28:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:28:59 smithi122 ceph-mon[93869]: pgmap v18776: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 868 KiB/s rd, 736 KiB/s wr, 10 op/s 2023-03-10T00:28:59.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:28:59 smithi181 ceph-mon[94189]: pgmap v18776: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 868 KiB/s rd, 736 KiB/s wr, 10 op/s 2023-03-10T00:28:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:28:59 smithi038 ceph-mon[92422]: pgmap v18776: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 868 KiB/s rd, 736 KiB/s wr, 10 op/s 2023-03-10T00:29:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:29:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:29:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:29:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:29:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:29:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:29:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:01 smithi122 ceph-mon[93869]: pgmap v18777: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 880 KiB/s rd, 736 KiB/s wr, 10 op/s 2023-03-10T00:29:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34251.468997 secs 2023-03-10T00:29:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:29:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:29:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:01 smithi181 ceph-mon[94189]: pgmap v18777: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 880 KiB/s rd, 736 KiB/s wr, 10 op/s 2023-03-10T00:29:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34251.468997 secs 2023-03-10T00:29:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:29:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:29:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:01 smithi038 ceph-mon[92422]: pgmap v18777: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 880 KiB/s rd, 736 KiB/s wr, 10 op/s 2023-03-10T00:29:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34251.468997 secs 2023-03-10T00:29:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:29:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:29:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34715.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:29:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:03 smithi122 ceph-mon[93869]: pgmap v18778: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 850 KiB/s rd, 771 KiB/s wr, 14 op/s 2023-03-10T00:29:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:03 smithi181 ceph-mon[94189]: pgmap v18778: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 850 KiB/s rd, 771 KiB/s wr, 14 op/s 2023-03-10T00:29:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:03 smithi038 ceph-mon[92422]: pgmap v18778: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 850 KiB/s rd, 771 KiB/s wr, 14 op/s 2023-03-10T00:29:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:05 smithi122 ceph-mon[93869]: pgmap v18779: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 549 KiB/s wr, 13 op/s 2023-03-10T00:29:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:05 smithi181 ceph-mon[94189]: pgmap v18779: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 549 KiB/s wr, 13 op/s 2023-03-10T00:29:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:05 smithi038 ceph-mon[92422]: pgmap v18779: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 549 KiB/s wr, 13 op/s 2023-03-10T00:29:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34256.469057 secs 2023-03-10T00:29:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:06 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:29:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34256.469057 secs 2023-03-10T00:29:06.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:06 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:29:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34256.469057 secs 2023-03-10T00:29:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:06 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:29:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:07 smithi122 ceph-mon[93869]: pgmap v18780: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 682 KiB/s rd, 623 KiB/s wr, 19 op/s 2023-03-10T00:29:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:07 smithi181 ceph-mon[94189]: pgmap v18780: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 682 KiB/s rd, 623 KiB/s wr, 19 op/s 2023-03-10T00:29:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:07 smithi038 ceph-mon[92422]: pgmap v18780: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 682 KiB/s rd, 623 KiB/s wr, 19 op/s 2023-03-10T00:29:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34258.931647 secs 2023-03-10T00:29:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34720.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:29:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34258.931647 secs 2023-03-10T00:29:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34258.931647 secs 2023-03-10T00:29:09.892 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:09 smithi122 ceph-mon[93869]: pgmap v18781: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 446 KiB/s wr, 15 op/s 2023-03-10T00:29:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:09 smithi181 ceph-mon[94189]: pgmap v18781: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 446 KiB/s wr, 15 op/s 2023-03-10T00:29:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:09 smithi038 ceph-mon[92422]: pgmap v18781: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 446 KiB/s wr, 15 op/s 2023-03-10T00:29:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:10 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:29:10.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:10 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:29:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:10 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:29:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:10 smithi181 ceph-mon[94189]: osdmap e1389: 12 total, 12 up, 12 in 2023-03-10T00:29:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:10 smithi181 ceph-mon[94189]: pgmap v18783: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 833 KiB/s rd, 778 KiB/s wr, 31 op/s 2023-03-10T00:29:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:29:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:29:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:29:10.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:29:10.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:29:10.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:29:10.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:29:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:10 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:29:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:10 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:29:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:10 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:29:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:10 smithi122 ceph-mon[93869]: osdmap e1389: 12 total, 12 up, 12 in 2023-03-10T00:29:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:10 smithi122 ceph-mon[93869]: pgmap v18783: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 833 KiB/s rd, 778 KiB/s wr, 31 op/s 2023-03-10T00:29:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:29:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:29:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:29:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:29:10.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:29:10.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:29:10.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:29:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:10 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:29:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:10 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:29:11.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:10 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:29:11.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:10 smithi038 ceph-mon[92422]: osdmap e1389: 12 total, 12 up, 12 in 2023-03-10T00:29:11.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:10 smithi038 ceph-mon[92422]: pgmap v18783: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 833 KiB/s rd, 778 KiB/s wr, 31 op/s 2023-03-10T00:29:11.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:29:11.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:29:11.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:29:11.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:29:11.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:29:11.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:29:11.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:29:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34261.469117 secs 2023-03-10T00:29:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:11 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:29:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:11 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:29:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34261.469117 secs 2023-03-10T00:29:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:11 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:29:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:11 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:29:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34261.469117 secs 2023-03-10T00:29:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:11 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:29:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:11 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:29:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:12 smithi122 ceph-mon[93869]: pgmap v18784: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 348 KiB/s wr, 29 op/s 2023-03-10T00:29:12.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:12 smithi181 ceph-mon[94189]: pgmap v18784: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 348 KiB/s wr, 29 op/s 2023-03-10T00:29:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:12 smithi038 ceph-mon[92422]: pgmap v18784: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 403 KiB/s rd, 348 KiB/s wr, 29 op/s 2023-03-10T00:29:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34725.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:29:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34263.931708 secs 2023-03-10T00:29:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:29:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:29:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34263.931708 secs 2023-03-10T00:29:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:29:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:29:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34263.931708 secs 2023-03-10T00:29:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:29:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:29:14.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:14 smithi122 ceph-mon[93869]: pgmap v18785: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 351 KiB/s wr, 28 op/s 2023-03-10T00:29:14.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:14 smithi181 ceph-mon[94189]: pgmap v18785: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 351 KiB/s wr, 28 op/s 2023-03-10T00:29:14.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:14 smithi038 ceph-mon[92422]: pgmap v18785: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 351 KiB/s wr, 28 op/s 2023-03-10T00:29:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34266.469176 secs 2023-03-10T00:29:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34266.469176 secs 2023-03-10T00:29:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34266.469176 secs 2023-03-10T00:29:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:16 smithi122 ceph-mon[93869]: pgmap v18786: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 521 KiB/s rd, 441 KiB/s wr, 25 op/s 2023-03-10T00:29:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:16 smithi181 ceph-mon[94189]: pgmap v18786: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 521 KiB/s rd, 441 KiB/s wr, 25 op/s 2023-03-10T00:29:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:16 smithi038 ceph-mon[92422]: pgmap v18786: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 521 KiB/s rd, 441 KiB/s wr, 25 op/s 2023-03-10T00:29:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:29:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:29:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:29:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:29:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:29:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:29:18.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34730.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:29:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34268.931769 secs 2023-03-10T00:29:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:18 smithi122 ceph-mon[93869]: pgmap v18787: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 440 KiB/s wr, 23 op/s 2023-03-10T00:29:18.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:18 smithi122 ceph-mon[93869]: osdmap e1390: 12 total, 12 up, 12 in 2023-03-10T00:29:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34268.931769 secs 2023-03-10T00:29:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:18 smithi181 ceph-mon[94189]: pgmap v18787: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 440 KiB/s wr, 23 op/s 2023-03-10T00:29:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:18 smithi181 ceph-mon[94189]: osdmap e1390: 12 total, 12 up, 12 in 2023-03-10T00:29:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34268.931769 secs 2023-03-10T00:29:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:18 smithi038 ceph-mon[92422]: pgmap v18787: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 440 KiB/s wr, 23 op/s 2023-03-10T00:29:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:18 smithi038 ceph-mon[92422]: osdmap e1390: 12 total, 12 up, 12 in 2023-03-10T00:29:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:29:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:29:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:29:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:29:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:29:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:29:21.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:21 smithi122 ceph-mon[93869]: pgmap v18789: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 310 KiB/s wr, 10 op/s 2023-03-10T00:29:21.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34271.469261 secs 2023-03-10T00:29:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:21 smithi181 ceph-mon[94189]: pgmap v18789: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 310 KiB/s wr, 10 op/s 2023-03-10T00:29:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34271.469261 secs 2023-03-10T00:29:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:21 smithi038 ceph-mon[92422]: pgmap v18789: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 310 KiB/s wr, 10 op/s 2023-03-10T00:29:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34271.469261 secs 2023-03-10T00:29:23.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34735.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:29:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:23 smithi122 ceph-mon[93869]: pgmap v18790: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 310 KiB/s wr, 8 op/s 2023-03-10T00:29:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34273.931831 secs 2023-03-10T00:29:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:23 smithi181 ceph-mon[94189]: pgmap v18790: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 310 KiB/s wr, 8 op/s 2023-03-10T00:29:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34273.931831 secs 2023-03-10T00:29:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:23 smithi038 ceph-mon[92422]: pgmap v18790: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 310 KiB/s wr, 8 op/s 2023-03-10T00:29:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34273.931831 secs 2023-03-10T00:29:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:25 smithi122 ceph-mon[93869]: pgmap v18791: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 306 KiB/s wr, 8 op/s 2023-03-10T00:29:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:25 smithi181 ceph-mon[94189]: pgmap v18791: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 306 KiB/s wr, 8 op/s 2023-03-10T00:29:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:25 smithi038 ceph-mon[92422]: pgmap v18791: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 306 KiB/s wr, 8 op/s 2023-03-10T00:29:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34276.469319 secs 2023-03-10T00:29:26.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34276.469319 secs 2023-03-10T00:29:26.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34276.469319 secs 2023-03-10T00:29:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:27 smithi122 ceph-mon[93869]: pgmap v18792: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 128 KiB/s wr, 7 op/s 2023-03-10T00:29:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:27 smithi181 ceph-mon[94189]: pgmap v18792: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 128 KiB/s wr, 7 op/s 2023-03-10T00:29:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:27 smithi038 ceph-mon[92422]: pgmap v18792: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 128 KiB/s wr, 7 op/s 2023-03-10T00:29:28.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34740.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:29:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34278.931891 secs 2023-03-10T00:29:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34278.931891 secs 2023-03-10T00:29:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34278.931891 secs 2023-03-10T00:29:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:29 smithi122 ceph-mon[93869]: pgmap v18793: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 128 KiB/s wr, 7 op/s 2023-03-10T00:29:29.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:29 smithi181 ceph-mon[94189]: pgmap v18793: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 128 KiB/s wr, 7 op/s 2023-03-10T00:29:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:29 smithi038 ceph-mon[92422]: pgmap v18793: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 128 KiB/s wr, 7 op/s 2023-03-10T00:29:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:31 smithi122 ceph-mon[93869]: pgmap v18794: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2023-03-10T00:29:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34281.469396 secs 2023-03-10T00:29:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:31 smithi181 ceph-mon[94189]: pgmap v18794: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2023-03-10T00:29:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34281.469396 secs 2023-03-10T00:29:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:31 smithi038 ceph-mon[92422]: pgmap v18794: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2023-03-10T00:29:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34281.469396 secs 2023-03-10T00:29:33.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34745.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:29:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:33 smithi122 ceph-mon[93869]: pgmap v18795: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-10T00:29:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34283.932021 secs 2023-03-10T00:29:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:33 smithi181 ceph-mon[94189]: pgmap v18795: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-10T00:29:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34283.932021 secs 2023-03-10T00:29:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:33 smithi038 ceph-mon[92422]: pgmap v18795: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-10T00:29:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34283.932021 secs 2023-03-10T00:29:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:35 smithi122 ceph-mon[93869]: pgmap v18796: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-10T00:29:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:35 smithi181 ceph-mon[94189]: pgmap v18796: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-10T00:29:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:35 smithi038 ceph-mon[92422]: pgmap v18796: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-10T00:29:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34286.469480 secs 2023-03-10T00:29:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34286.469480 secs 2023-03-10T00:29:36.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34286.469480 secs 2023-03-10T00:29:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:37 smithi122 ceph-mon[93869]: pgmap v18797: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-10T00:29:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:37 smithi181 ceph-mon[94189]: pgmap v18797: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-10T00:29:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:37 smithi038 ceph-mon[92422]: pgmap v18797: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-10T00:29:38.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34750.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:29:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34288.932075 secs 2023-03-10T00:29:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34288.932075 secs 2023-03-10T00:29:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34288.932075 secs 2023-03-10T00:29:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:39 smithi122 ceph-mon[93869]: pgmap v18798: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-10T00:29:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:39 smithi181 ceph-mon[94189]: pgmap v18798: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-10T00:29:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:39 smithi038 ceph-mon[92422]: pgmap v18798: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-10T00:29:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:41 smithi122 ceph-mon[93869]: pgmap v18799: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-10T00:29:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34291.469539 secs 2023-03-10T00:29:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:41 smithi181 ceph-mon[94189]: pgmap v18799: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-10T00:29:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34291.469539 secs 2023-03-10T00:29:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:41 smithi038 ceph-mon[92422]: pgmap v18799: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-10T00:29:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34291.469539 secs 2023-03-10T00:29:43.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34755.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:29:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:43 smithi122 ceph-mon[93869]: pgmap v18800: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 8 op/s 2023-03-10T00:29:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:43 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 34293.932159 secs 2023-03-10T00:29:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:43 smithi181 ceph-mon[94189]: pgmap v18800: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 8 op/s 2023-03-10T00:29:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:43 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 34293.932159 secs 2023-03-10T00:29:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:43 smithi038 ceph-mon[92422]: pgmap v18800: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 597 B/s wr, 8 op/s 2023-03-10T00:29:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:43 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 34293.932159 secs 2023-03-10T00:29:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:45 smithi122 ceph-mon[93869]: pgmap v18801: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 288 KiB/s wr, 7 op/s 2023-03-10T00:29:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:45 smithi181 ceph-mon[94189]: pgmap v18801: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 288 KiB/s wr, 7 op/s 2023-03-10T00:29:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:45 smithi038 ceph-mon[92422]: pgmap v18801: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 288 KiB/s wr, 7 op/s 2023-03-10T00:29:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34296.469623 secs 2023-03-10T00:29:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:46 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:29:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:46 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:29:46.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:46 smithi122 ceph-mon[93869]: pgmap v18802: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 315 KiB/s wr, 9 op/s 2023-03-10T00:29:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34296.469623 secs 2023-03-10T00:29:46.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:46 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:29:46.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:46 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:29:46.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:46 smithi181 ceph-mon[94189]: pgmap v18802: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 315 KiB/s wr, 9 op/s 2023-03-10T00:29:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34296.469623 secs 2023-03-10T00:29:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:46 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:29:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:46 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:29:47.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:46 smithi038 ceph-mon[92422]: pgmap v18802: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 315 KiB/s wr, 9 op/s 2023-03-10T00:29:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:29:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:29:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:29:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:29:47.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:29:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:29:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34760.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:29:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:48 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 34298.932222 secs 2023-03-10T00:29:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:48 smithi122 ceph-mon[93869]: pgmap v18803: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 316 KiB/s wr, 7 op/s 2023-03-10T00:29:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:48 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 34298.932222 secs 2023-03-10T00:29:48.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:48 smithi181 ceph-mon[94189]: pgmap v18803: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 316 KiB/s wr, 7 op/s 2023-03-10T00:29:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:48 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 34298.932222 secs 2023-03-10T00:29:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:48 smithi038 ceph-mon[92422]: pgmap v18803: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 316 KiB/s wr, 7 op/s 2023-03-10T00:29:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:29:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:29:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:29:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:29:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:29:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:29:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:50 smithi122 ceph-mon[93869]: pgmap v18804: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 318 KiB/s wr, 9 op/s 2023-03-10T00:29:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34301.469688 secs 2023-03-10T00:29:50.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:50 smithi181 ceph-mon[94189]: pgmap v18804: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 318 KiB/s wr, 9 op/s 2023-03-10T00:29:50.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34301.469688 secs 2023-03-10T00:29:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:50 smithi038 ceph-mon[92422]: pgmap v18804: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 318 KiB/s wr, 9 op/s 2023-03-10T00:29:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34301.469688 secs 2023-03-10T00:29:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:52 smithi122 ceph-mon[93869]: pgmap v18805: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 346 KiB/s wr, 11 op/s 2023-03-10T00:29:53.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:52 smithi181 ceph-mon[94189]: pgmap v18805: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 346 KiB/s wr, 11 op/s 2023-03-10T00:29:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:52 smithi038 ceph-mon[92422]: pgmap v18805: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 346 KiB/s wr, 11 op/s 2023-03-10T00:29:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34765.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:29:54.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34303.932318 secs 2023-03-10T00:29:54.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34303.932318 secs 2023-03-10T00:29:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34303.932318 secs 2023-03-10T00:29:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:55 smithi122 ceph-mon[93869]: pgmap v18806: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 637 KiB/s rd, 630 KiB/s wr, 11 op/s 2023-03-10T00:29:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:29:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:29:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:55 smithi181 ceph-mon[94189]: pgmap v18806: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 637 KiB/s rd, 630 KiB/s wr, 11 op/s 2023-03-10T00:29:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:29:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:29:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:55 smithi038 ceph-mon[92422]: pgmap v18806: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 637 KiB/s rd, 630 KiB/s wr, 11 op/s 2023-03-10T00:29:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:29:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:29:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34306.469747 secs 2023-03-10T00:29:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:29:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:29:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34306.469747 secs 2023-03-10T00:29:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:29:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:29:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34306.469747 secs 2023-03-10T00:29:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:29:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:29:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:57 smithi122 ceph-mon[93869]: pgmap v18807: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 894 KiB/s rd, 717 KiB/s wr, 14 op/s 2023-03-10T00:29:57.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:29:57.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:29:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:57 smithi181 ceph-mon[94189]: pgmap v18807: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 894 KiB/s rd, 717 KiB/s wr, 14 op/s 2023-03-10T00:29:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:29:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:29:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:57 smithi038 ceph-mon[92422]: pgmap v18807: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 894 KiB/s rd, 717 KiB/s wr, 14 op/s 2023-03-10T00:29:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:29:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:29:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34770.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:29:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34308.932376 secs 2023-03-10T00:29:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34308.932376 secs 2023-03-10T00:29:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34308.932376 secs 2023-03-10T00:29:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:29:59 smithi122 ceph-mon[93869]: pgmap v18808: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 690 KiB/s wr, 12 op/s 2023-03-10T00:29:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:29:59 smithi181 ceph-mon[94189]: pgmap v18808: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 690 KiB/s wr, 12 op/s 2023-03-10T00:29:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:29:59 smithi038 ceph-mon[92422]: pgmap v18808: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 690 KiB/s wr, 12 op/s 2023-03-10T00:30:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:00 smithi122 ceph-mon[93869]: Health detail: HEALTH_WARN 2 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-10T00:30:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:00 smithi122 ceph-mon[93869]: [WRN] MDS_SLOW_REQUEST: 2 MDSs report slow requests 2023-03-10T00:30:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:00 smithi122 ceph-mon[93869]: mds.i(mds.2): 2 slow requests are blocked > 180 secs 2023-03-10T00:30:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:00 smithi122 ceph-mon[93869]: mds.h(mds.1): 2 slow requests are blocked > 180 secs 2023-03-10T00:30:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:00 smithi122 ceph-mon[93869]: [WRN] MDS_TRIM: 2 MDSs behind on trimming 2023-03-10T00:30:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:00 smithi122 ceph-mon[93869]: mds.e(mds.0): Behind on trimming (946/128) max_segments: 128, num_segments: 946 2023-03-10T00:30:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:00 smithi122 ceph-mon[93869]: mds.b(mds.0): Behind on trimming (946/128) max_segments: 128, num_segments: 946 2023-03-10T00:30:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:00 smithi181 ceph-mon[94189]: Health detail: HEALTH_WARN 2 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-10T00:30:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:00 smithi181 ceph-mon[94189]: [WRN] MDS_SLOW_REQUEST: 2 MDSs report slow requests 2023-03-10T00:30:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:00 smithi181 ceph-mon[94189]: mds.i(mds.2): 2 slow requests are blocked > 180 secs 2023-03-10T00:30:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:00 smithi181 ceph-mon[94189]: mds.h(mds.1): 2 slow requests are blocked > 180 secs 2023-03-10T00:30:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:00 smithi181 ceph-mon[94189]: [WRN] MDS_TRIM: 2 MDSs behind on trimming 2023-03-10T00:30:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:00 smithi181 ceph-mon[94189]: mds.e(mds.0): Behind on trimming (946/128) max_segments: 128, num_segments: 946 2023-03-10T00:30:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:00 smithi181 ceph-mon[94189]: mds.b(mds.0): Behind on trimming (946/128) max_segments: 128, num_segments: 946 2023-03-10T00:30:01.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:00 smithi038 ceph-mon[92422]: Health detail: HEALTH_WARN 2 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-10T00:30:01.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:00 smithi038 ceph-mon[92422]: [WRN] MDS_SLOW_REQUEST: 2 MDSs report slow requests 2023-03-10T00:30:01.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:00 smithi038 ceph-mon[92422]: mds.i(mds.2): 2 slow requests are blocked > 180 secs 2023-03-10T00:30:01.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:00 smithi038 ceph-mon[92422]: mds.h(mds.1): 2 slow requests are blocked > 180 secs 2023-03-10T00:30:01.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:00 smithi038 ceph-mon[92422]: [WRN] MDS_TRIM: 2 MDSs behind on trimming 2023-03-10T00:30:01.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:00 smithi038 ceph-mon[92422]: mds.e(mds.0): Behind on trimming (946/128) max_segments: 128, num_segments: 946 2023-03-10T00:30:01.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:00 smithi038 ceph-mon[92422]: mds.b(mds.0): Behind on trimming (946/128) max_segments: 128, num_segments: 946 2023-03-10T00:30:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:01 smithi122 ceph-mon[93869]: pgmap v18809: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 867 KiB/s rd, 690 KiB/s wr, 12 op/s 2023-03-10T00:30:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34311.469832 secs 2023-03-10T00:30:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:01 smithi181 ceph-mon[94189]: pgmap v18809: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 867 KiB/s rd, 690 KiB/s wr, 12 op/s 2023-03-10T00:30:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34311.469832 secs 2023-03-10T00:30:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:01 smithi038 ceph-mon[92422]: pgmap v18809: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 867 KiB/s rd, 690 KiB/s wr, 12 op/s 2023-03-10T00:30:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34311.469832 secs 2023-03-10T00:30:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34775.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:30:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:03 smithi122 ceph-mon[93869]: pgmap v18810: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 893 KiB/s rd, 891 KiB/s wr, 17 op/s 2023-03-10T00:30:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34313.932434 secs 2023-03-10T00:30:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:03 smithi181 ceph-mon[94189]: pgmap v18810: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 893 KiB/s rd, 891 KiB/s wr, 17 op/s 2023-03-10T00:30:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34313.932434 secs 2023-03-10T00:30:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:03 smithi038 ceph-mon[92422]: pgmap v18810: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 893 KiB/s rd, 891 KiB/s wr, 17 op/s 2023-03-10T00:30:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34313.932434 secs 2023-03-10T00:30:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:05 smithi122 ceph-mon[93869]: pgmap v18811: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 869 KiB/s rd, 885 KiB/s wr, 27 op/s 2023-03-10T00:30:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:05 smithi122 ceph-mon[93869]: osdmap e1391: 12 total, 12 up, 12 in 2023-03-10T00:30:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:05 smithi181 ceph-mon[94189]: pgmap v18811: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 869 KiB/s rd, 885 KiB/s wr, 27 op/s 2023-03-10T00:30:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:05 smithi181 ceph-mon[94189]: osdmap e1391: 12 total, 12 up, 12 in 2023-03-10T00:30:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:05 smithi038 ceph-mon[92422]: pgmap v18811: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 869 KiB/s rd, 885 KiB/s wr, 27 op/s 2023-03-10T00:30:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:05 smithi038 ceph-mon[92422]: osdmap e1391: 12 total, 12 up, 12 in 2023-03-10T00:30:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34316.469915 secs 2023-03-10T00:30:06.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34316.469915 secs 2023-03-10T00:30:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34316.469915 secs 2023-03-10T00:30:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:07 smithi122 ceph-mon[93869]: pgmap v18813: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 494 KiB/s wr, 29 op/s 2023-03-10T00:30:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:30:07.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:30:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:07 smithi181 ceph-mon[94189]: pgmap v18813: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 494 KiB/s wr, 29 op/s 2023-03-10T00:30:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:30:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:30:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:07 smithi038 ceph-mon[92422]: pgmap v18813: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 494 KiB/s wr, 29 op/s 2023-03-10T00:30:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:30:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:30:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34780.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:30:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34318.932492 secs 2023-03-10T00:30:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:08 smithi122 ceph-mon[93869]: pgmap v18814: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 494 KiB/s wr, 29 op/s 2023-03-10T00:30:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34318.932492 secs 2023-03-10T00:30:08.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:08 smithi181 ceph-mon[94189]: pgmap v18814: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 494 KiB/s wr, 29 op/s 2023-03-10T00:30:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34318.932492 secs 2023-03-10T00:30:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:08 smithi038 ceph-mon[92422]: pgmap v18814: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 494 KiB/s wr, 29 op/s 2023-03-10T00:30:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:30:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:30:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:30:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:30:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:30:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:30:10.929 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:10 smithi181 ceph-mon[94189]: pgmap v18815: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 494 KiB/s wr, 28 op/s 2023-03-10T00:30:10.930 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34321.470005 secs 2023-03-10T00:30:10.930 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:10 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:30:10.942 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:10 smithi122 ceph-mon[93869]: pgmap v18815: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 494 KiB/s wr, 28 op/s 2023-03-10T00:30:10.942 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34321.470005 secs 2023-03-10T00:30:10.942 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:10 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:30:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:10 smithi038 ceph-mon[92422]: pgmap v18815: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 494 KiB/s wr, 28 op/s 2023-03-10T00:30:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34321.470005 secs 2023-03-10T00:30:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:10 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:30:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:11 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:30:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:11 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:30:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:11 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:30:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:11 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:30:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:11 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:30:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:11 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:30:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:11 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:30:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:11 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:30:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:11 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:30:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:12 smithi122 ceph-mon[93869]: pgmap v18816: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 281 KiB/s wr, 24 op/s 2023-03-10T00:30:12.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:12 smithi181 ceph-mon[94189]: pgmap v18816: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 281 KiB/s wr, 24 op/s 2023-03-10T00:30:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:12 smithi038 ceph-mon[92422]: pgmap v18816: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 281 KiB/s wr, 24 op/s 2023-03-10T00:30:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34785.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:30:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34323.932596 secs 2023-03-10T00:30:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:13 smithi122 ceph-mon[93869]: osdmap e1392: 12 total, 12 up, 12 in 2023-03-10T00:30:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34323.932596 secs 2023-03-10T00:30:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:13 smithi181 ceph-mon[94189]: osdmap e1392: 12 total, 12 up, 12 in 2023-03-10T00:30:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34323.932596 secs 2023-03-10T00:30:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:13 smithi038 ceph-mon[92422]: osdmap e1392: 12 total, 12 up, 12 in 2023-03-10T00:30:14.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:14 smithi122 ceph-mon[93869]: pgmap v18818: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 445 KiB/s rd, 442 KiB/s wr, 11 op/s 2023-03-10T00:30:14.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:14 smithi181 ceph-mon[94189]: pgmap v18818: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 445 KiB/s rd, 442 KiB/s wr, 11 op/s 2023-03-10T00:30:14.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:14 smithi038 ceph-mon[92422]: pgmap v18818: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 445 KiB/s rd, 442 KiB/s wr, 11 op/s 2023-03-10T00:30:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:15 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:30:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:15 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:30:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34326.470103 secs 2023-03-10T00:30:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:15 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:30:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:15 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:30:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34326.470103 secs 2023-03-10T00:30:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:15 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:30:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:15 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:30:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34326.470103 secs 2023-03-10T00:30:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:16 smithi122 ceph-mon[93869]: pgmap v18819: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 183 KiB/s wr, 8 op/s 2023-03-10T00:30:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:16 smithi181 ceph-mon[94189]: pgmap v18819: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 183 KiB/s wr, 8 op/s 2023-03-10T00:30:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:16 smithi038 ceph-mon[92422]: pgmap v18819: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 183 KiB/s wr, 8 op/s 2023-03-10T00:30:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:30:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:30:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:30:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:30:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:30:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:30:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34790.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:30:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34328.932680 secs 2023-03-10T00:30:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:18 smithi122 ceph-mon[93869]: pgmap v18820: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 183 KiB/s wr, 8 op/s 2023-03-10T00:30:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34328.932680 secs 2023-03-10T00:30:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:18 smithi181 ceph-mon[94189]: pgmap v18820: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 183 KiB/s wr, 8 op/s 2023-03-10T00:30:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34328.932680 secs 2023-03-10T00:30:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:18 smithi038 ceph-mon[92422]: pgmap v18820: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 183 KiB/s wr, 8 op/s 2023-03-10T00:30:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:20 smithi122 ceph-mon[93869]: pgmap v18821: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 183 KiB/s wr, 8 op/s 2023-03-10T00:30:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34331.470165 secs 2023-03-10T00:30:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:20 smithi181 ceph-mon[94189]: pgmap v18821: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 183 KiB/s wr, 8 op/s 2023-03-10T00:30:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34331.470165 secs 2023-03-10T00:30:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:20 smithi038 ceph-mon[92422]: pgmap v18821: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 183 KiB/s wr, 8 op/s 2023-03-10T00:30:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34331.470165 secs 2023-03-10T00:30:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:22 smithi122 ceph-mon[93869]: pgmap v18822: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 151 KiB/s wr, 7 op/s 2023-03-10T00:30:23.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:22 smithi181 ceph-mon[94189]: pgmap v18822: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 151 KiB/s wr, 7 op/s 2023-03-10T00:30:23.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:22 smithi038 ceph-mon[92422]: pgmap v18822: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 151 KiB/s wr, 7 op/s 2023-03-10T00:30:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34795.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:30:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34333.932769 secs 2023-03-10T00:30:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34333.932769 secs 2023-03-10T00:30:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34333.932769 secs 2023-03-10T00:30:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:25 smithi122 ceph-mon[93869]: pgmap v18823: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 124 KiB/s wr, 7 op/s 2023-03-10T00:30:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:25 smithi181 ceph-mon[94189]: pgmap v18823: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 124 KiB/s wr, 7 op/s 2023-03-10T00:30:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:25 smithi038 ceph-mon[92422]: pgmap v18823: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 124 KiB/s wr, 7 op/s 2023-03-10T00:30:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34336.470250 secs 2023-03-10T00:30:26.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34336.470250 secs 2023-03-10T00:30:26.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34336.470250 secs 2023-03-10T00:30:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:27 smithi122 ceph-mon[93869]: pgmap v18824: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 106 KiB/s wr, 8 op/s 2023-03-10T00:30:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:27 smithi181 ceph-mon[94189]: pgmap v18824: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 106 KiB/s wr, 8 op/s 2023-03-10T00:30:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:27 smithi038 ceph-mon[92422]: pgmap v18824: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 106 KiB/s wr, 8 op/s 2023-03-10T00:30:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34800.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:30:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34338.932843 secs 2023-03-10T00:30:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34338.932843 secs 2023-03-10T00:30:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34338.932843 secs 2023-03-10T00:30:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:29 smithi122 ceph-mon[93869]: pgmap v18825: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 106 KiB/s wr, 6 op/s 2023-03-10T00:30:29.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:29 smithi181 ceph-mon[94189]: pgmap v18825: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 106 KiB/s wr, 6 op/s 2023-03-10T00:30:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:29 smithi038 ceph-mon[92422]: pgmap v18825: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 106 KiB/s wr, 6 op/s 2023-03-10T00:30:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:31 smithi122 ceph-mon[93869]: pgmap v18826: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 106 KiB/s wr, 6 op/s 2023-03-10T00:30:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34341.470313 secs 2023-03-10T00:30:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:31 smithi181 ceph-mon[94189]: pgmap v18826: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 106 KiB/s wr, 6 op/s 2023-03-10T00:30:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34341.470313 secs 2023-03-10T00:30:31.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:31 smithi038 ceph-mon[92422]: pgmap v18826: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 106 KiB/s wr, 6 op/s 2023-03-10T00:30:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34341.470313 secs 2023-03-10T00:30:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34805.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:30:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:33 smithi122 ceph-mon[93869]: pgmap v18827: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 106 KiB/s wr, 8 op/s 2023-03-10T00:30:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34343.932901 secs 2023-03-10T00:30:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:33 smithi181 ceph-mon[94189]: pgmap v18827: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 106 KiB/s wr, 8 op/s 2023-03-10T00:30:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34343.932901 secs 2023-03-10T00:30:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:33 smithi038 ceph-mon[92422]: pgmap v18827: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 106 KiB/s wr, 8 op/s 2023-03-10T00:30:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34343.932901 secs 2023-03-10T00:30:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:35 smithi122 ceph-mon[93869]: pgmap v18828: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 106 KiB/s wr, 6 op/s 2023-03-10T00:30:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:35 smithi181 ceph-mon[94189]: pgmap v18828: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 106 KiB/s wr, 6 op/s 2023-03-10T00:30:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:35 smithi038 ceph-mon[92422]: pgmap v18828: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 109 KiB/s rd, 106 KiB/s wr, 6 op/s 2023-03-10T00:30:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34346.470393 secs 2023-03-10T00:30:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34346.470393 secs 2023-03-10T00:30:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34346.470393 secs 2023-03-10T00:30:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:37 smithi122 ceph-mon[93869]: pgmap v18829: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 597 B/s wr, 8 op/s 2023-03-10T00:30:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:37 smithi181 ceph-mon[94189]: pgmap v18829: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 597 B/s wr, 8 op/s 2023-03-10T00:30:37.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:37 smithi038 ceph-mon[92422]: pgmap v18829: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 597 B/s wr, 8 op/s 2023-03-10T00:30:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34810.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:30:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34348.932962 secs 2023-03-10T00:30:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34348.932962 secs 2023-03-10T00:30:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34348.932962 secs 2023-03-10T00:30:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:39 smithi122 ceph-mon[93869]: pgmap v18830: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 682 B/s wr, 6 op/s 2023-03-10T00:30:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:39 smithi181 ceph-mon[94189]: pgmap v18830: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 682 B/s wr, 6 op/s 2023-03-10T00:30:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:39 smithi038 ceph-mon[92422]: pgmap v18830: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 682 B/s wr, 6 op/s 2023-03-10T00:30:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:41 smithi122 ceph-mon[93869]: pgmap v18831: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 6 op/s 2023-03-10T00:30:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34351.470451 secs 2023-03-10T00:30:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:41 smithi181 ceph-mon[94189]: pgmap v18831: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 6 op/s 2023-03-10T00:30:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34351.470451 secs 2023-03-10T00:30:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:41 smithi038 ceph-mon[92422]: pgmap v18831: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 25 KiB/s wr, 6 op/s 2023-03-10T00:30:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34351.470451 secs 2023-03-10T00:30:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34815.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:30:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:43 smithi122 ceph-mon[93869]: pgmap v18832: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 135 KiB/s wr, 9 op/s 2023-03-10T00:30:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34353.933041 secs 2023-03-10T00:30:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:43 smithi181 ceph-mon[94189]: pgmap v18832: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 135 KiB/s wr, 9 op/s 2023-03-10T00:30:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34353.933041 secs 2023-03-10T00:30:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:43 smithi038 ceph-mon[92422]: pgmap v18832: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 135 KiB/s wr, 9 op/s 2023-03-10T00:30:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34353.933041 secs 2023-03-10T00:30:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:45 smithi122 ceph-mon[93869]: pgmap v18833: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 135 KiB/s wr, 7 op/s 2023-03-10T00:30:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:45 smithi181 ceph-mon[94189]: pgmap v18833: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 135 KiB/s wr, 7 op/s 2023-03-10T00:30:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:45 smithi038 ceph-mon[92422]: pgmap v18833: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 135 KiB/s wr, 7 op/s 2023-03-10T00:30:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34356.470509 secs 2023-03-10T00:30:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34356.470509 secs 2023-03-10T00:30:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34356.470509 secs 2023-03-10T00:30:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:47 smithi122 ceph-mon[93869]: pgmap v18834: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 135 KiB/s wr, 9 op/s 2023-03-10T00:30:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:47 smithi181 ceph-mon[94189]: pgmap v18834: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 135 KiB/s wr, 9 op/s 2023-03-10T00:30:47.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:47 smithi038 ceph-mon[92422]: pgmap v18834: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 135 KiB/s wr, 9 op/s 2023-03-10T00:30:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34820.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:30:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34358.933153 secs 2023-03-10T00:30:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:48 smithi122 ceph-mon[93869]: pgmap v18835: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 134 KiB/s wr, 7 op/s 2023-03-10T00:30:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34358.933153 secs 2023-03-10T00:30:48.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:48 smithi181 ceph-mon[94189]: pgmap v18835: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 134 KiB/s wr, 7 op/s 2023-03-10T00:30:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34358.933153 secs 2023-03-10T00:30:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:48 smithi038 ceph-mon[92422]: pgmap v18835: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 134 KiB/s wr, 7 op/s 2023-03-10T00:30:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:30:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:30:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:30:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:30:49.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:30:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:30:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:50 smithi122 ceph-mon[93869]: pgmap v18836: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 266 KiB/s wr, 7 op/s 2023-03-10T00:30:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34361.470569 secs 2023-03-10T00:30:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:50 smithi181 ceph-mon[94189]: pgmap v18836: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 266 KiB/s wr, 7 op/s 2023-03-10T00:30:50.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34361.470569 secs 2023-03-10T00:30:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:50 smithi038 ceph-mon[92422]: pgmap v18836: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 266 KiB/s wr, 7 op/s 2023-03-10T00:30:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34361.470569 secs 2023-03-10T00:30:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:30:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:30:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:30:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:30:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:30:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:30:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:52 smithi122 ceph-mon[93869]: pgmap v18837: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 269 KiB/s wr, 9 op/s 2023-03-10T00:30:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:52 smithi181 ceph-mon[94189]: pgmap v18837: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 269 KiB/s wr, 9 op/s 2023-03-10T00:30:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:52 smithi038 ceph-mon[92422]: pgmap v18837: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 269 KiB/s wr, 9 op/s 2023-03-10T00:30:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34825.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:30:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34363.933214 secs 2023-03-10T00:30:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34363.933214 secs 2023-03-10T00:30:53.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34363.933214 secs 2023-03-10T00:30:54.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:54 smithi122 ceph-mon[93869]: pgmap v18838: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 159 KiB/s wr, 7 op/s 2023-03-10T00:30:54.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:54 smithi181 ceph-mon[94189]: pgmap v18838: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 159 KiB/s wr, 7 op/s 2023-03-10T00:30:54.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:54 smithi038 ceph-mon[92422]: pgmap v18838: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 159 KiB/s wr, 7 op/s 2023-03-10T00:30:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:30:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:30:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34366.470670 secs 2023-03-10T00:30:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:30:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:30:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34366.470670 secs 2023-03-10T00:30:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:30:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:30:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34366.470670 secs 2023-03-10T00:30:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:56 smithi122 ceph-mon[93869]: pgmap v18839: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 456 KiB/s rd, 451 KiB/s wr, 11 op/s 2023-03-10T00:30:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:56 smithi181 ceph-mon[94189]: pgmap v18839: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 456 KiB/s rd, 451 KiB/s wr, 11 op/s 2023-03-10T00:30:56.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:56 smithi038 ceph-mon[92422]: pgmap v18839: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 456 KiB/s rd, 451 KiB/s wr, 11 op/s 2023-03-10T00:30:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:30:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:30:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:57 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:30:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:30:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:30:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:57 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:30:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:30:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:30:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:57 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:30:58.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34830.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:30:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:58 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:30:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:58 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:30:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:58 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:30:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:30:58 smithi122 ceph-mon[93869]: pgmap v18840: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 455 KiB/s rd, 451 KiB/s wr, 9 op/s 2023-03-10T00:30:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:58 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:30:58.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:58 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:30:58.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:58 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:30:58.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:30:58 smithi181 ceph-mon[94189]: pgmap v18840: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 455 KiB/s rd, 451 KiB/s wr, 9 op/s 2023-03-10T00:30:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:58 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:30:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:58 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:30:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:58 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:30:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:30:58 smithi038 ceph-mon[92422]: pgmap v18840: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 455 KiB/s rd, 451 KiB/s wr, 9 op/s 2023-03-10T00:31:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:31:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:31:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:31:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:31:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:31:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:31:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:01 smithi122 ceph-mon[93869]: pgmap v18841: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 733 KiB/s rd, 719 KiB/s wr, 11 op/s 2023-03-10T00:31:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34371.470753 secs 2023-03-10T00:31:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:31:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:31:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:01 smithi181 ceph-mon[94189]: pgmap v18841: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 733 KiB/s rd, 719 KiB/s wr, 11 op/s 2023-03-10T00:31:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34371.470753 secs 2023-03-10T00:31:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:31:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:31:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:01 smithi038 ceph-mon[92422]: pgmap v18841: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 733 KiB/s rd, 719 KiB/s wr, 11 op/s 2023-03-10T00:31:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34371.470753 secs 2023-03-10T00:31:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:31:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:31:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34835.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:31:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:03 smithi122 ceph-mon[93869]: pgmap v18842: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 840 KiB/s rd, 825 KiB/s wr, 15 op/s 2023-03-10T00:31:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:03 smithi181 ceph-mon[94189]: pgmap v18842: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 840 KiB/s rd, 825 KiB/s wr, 15 op/s 2023-03-10T00:31:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:03 smithi038 ceph-mon[92422]: pgmap v18842: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 840 KiB/s rd, 825 KiB/s wr, 15 op/s 2023-03-10T00:31:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:05 smithi122 ceph-mon[93869]: pgmap v18843: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 812 KiB/s rd, 798 KiB/s wr, 13 op/s 2023-03-10T00:31:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:05 smithi181 ceph-mon[94189]: pgmap v18843: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 812 KiB/s rd, 798 KiB/s wr, 13 op/s 2023-03-10T00:31:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:05 smithi038 ceph-mon[92422]: pgmap v18843: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 812 KiB/s rd, 798 KiB/s wr, 13 op/s 2023-03-10T00:31:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34376.470809 secs 2023-03-10T00:31:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34376.470809 secs 2023-03-10T00:31:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34376.470809 secs 2023-03-10T00:31:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:07 smithi122 ceph-mon[93869]: pgmap v18844: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 914 KiB/s rd, 920 KiB/s wr, 19 op/s 2023-03-10T00:31:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:07 smithi181 ceph-mon[94189]: pgmap v18844: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 914 KiB/s rd, 920 KiB/s wr, 19 op/s 2023-03-10T00:31:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:07 smithi038 ceph-mon[92422]: pgmap v18844: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 914 KiB/s rd, 920 KiB/s wr, 19 op/s 2023-03-10T00:31:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34840.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:31:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34378.933469 secs 2023-03-10T00:31:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34378.933469 secs 2023-03-10T00:31:08.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34378.933469 secs 2023-03-10T00:31:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:09 smithi122 ceph-mon[93869]: pgmap v18845: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 619 KiB/s rd, 629 KiB/s wr, 15 op/s 2023-03-10T00:31:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:09 smithi181 ceph-mon[94189]: pgmap v18845: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 619 KiB/s rd, 629 KiB/s wr, 15 op/s 2023-03-10T00:31:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:09 smithi038 ceph-mon[92422]: pgmap v18845: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 619 KiB/s rd, 629 KiB/s wr, 15 op/s 2023-03-10T00:31:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:10 smithi122 ceph-mon[93869]: osdmap e1393: 12 total, 12 up, 12 in 2023-03-10T00:31:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:10 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:31:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:10 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:31:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:10 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:31:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:10 smithi181 ceph-mon[94189]: osdmap e1393: 12 total, 12 up, 12 in 2023-03-10T00:31:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:10 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:31:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:10 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:31:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:10 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:31:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:10 smithi038 ceph-mon[92422]: osdmap e1393: 12 total, 12 up, 12 in 2023-03-10T00:31:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:10 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:31:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:10 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:31:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:10 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:31:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:11 smithi122 ceph-mon[93869]: pgmap v18847: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 458 KiB/s wr, 29 op/s 2023-03-10T00:31:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34381.470869 secs 2023-03-10T00:31:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:11 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:31:11.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:11 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:31:11.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:11 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:31:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:11 smithi181 ceph-mon[94189]: pgmap v18847: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 458 KiB/s wr, 29 op/s 2023-03-10T00:31:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34381.470869 secs 2023-03-10T00:31:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:11 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:31:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:11 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:31:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:11 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:31:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:11 smithi038 ceph-mon[92422]: pgmap v18847: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 458 KiB/s wr, 29 op/s 2023-03-10T00:31:12.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34381.470869 secs 2023-03-10T00:31:12.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:11 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:31:12.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:11 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:31:12.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:11 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:31:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34845.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:31:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:13 smithi122 ceph-mon[93869]: pgmap v18848: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 509 KiB/s rd, 550 KiB/s wr, 28 op/s 2023-03-10T00:31:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34383.933579 secs 2023-03-10T00:31:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:31:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:31:13.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:13 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:31:13.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:13 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:31:13.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:13 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:31:13.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:13 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:31:13.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:13 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:31:13.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:13 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:31:13.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:13 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:31:13.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:13 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:31:13.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:13 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:31:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:13 smithi181 ceph-mon[94189]: pgmap v18848: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 509 KiB/s rd, 550 KiB/s wr, 28 op/s 2023-03-10T00:31:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34383.933579 secs 2023-03-10T00:31:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:31:13.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:31:13.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:13 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:31:13.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:13 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:31:13.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:13 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:31:13.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:13 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:31:13.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:13 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:31:13.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:13 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:31:13.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:13 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:31:13.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:13 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:31:13.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:13 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:31:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:13 smithi038 ceph-mon[92422]: pgmap v18848: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 509 KiB/s rd, 550 KiB/s wr, 28 op/s 2023-03-10T00:31:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34383.933579 secs 2023-03-10T00:31:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:31:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:31:14.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:13 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:31:14.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:13 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:31:14.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:13 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:31:14.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:13 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:31:14.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:13 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:31:14.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:13 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:31:14.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:13 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:31:14.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:13 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:31:14.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:13 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:31:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:15 smithi122 ceph-mon[93869]: pgmap v18849: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 509 KiB/s rd, 550 KiB/s wr, 27 op/s 2023-03-10T00:31:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:15 smithi181 ceph-mon[94189]: pgmap v18849: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 509 KiB/s rd, 550 KiB/s wr, 27 op/s 2023-03-10T00:31:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:15 smithi038 ceph-mon[92422]: pgmap v18849: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 509 KiB/s rd, 550 KiB/s wr, 27 op/s 2023-03-10T00:31:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34386.470932 secs 2023-03-10T00:31:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34386.470932 secs 2023-03-10T00:31:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34386.470932 secs 2023-03-10T00:31:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:17 smithi122 ceph-mon[93869]: pgmap v18850: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 532 KiB/s wr, 23 op/s 2023-03-10T00:31:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:17 smithi181 ceph-mon[94189]: pgmap v18850: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 532 KiB/s wr, 23 op/s 2023-03-10T00:31:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:17 smithi038 ceph-mon[92422]: pgmap v18850: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 532 KiB/s wr, 23 op/s 2023-03-10T00:31:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34850.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:31:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34388.933678 secs 2023-03-10T00:31:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:18 smithi122 ceph-mon[93869]: pgmap v18851: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 531 KiB/s wr, 22 op/s 2023-03-10T00:31:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:18 smithi122 ceph-mon[93869]: osdmap e1394: 12 total, 12 up, 12 in 2023-03-10T00:31:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34388.933678 secs 2023-03-10T00:31:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:18 smithi181 ceph-mon[94189]: pgmap v18851: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 531 KiB/s wr, 22 op/s 2023-03-10T00:31:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:18 smithi181 ceph-mon[94189]: osdmap e1394: 12 total, 12 up, 12 in 2023-03-10T00:31:19.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34388.933678 secs 2023-03-10T00:31:19.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:18 smithi038 ceph-mon[92422]: pgmap v18851: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 531 KiB/s wr, 22 op/s 2023-03-10T00:31:19.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:18 smithi038 ceph-mon[92422]: osdmap e1394: 12 total, 12 up, 12 in 2023-03-10T00:31:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:20 smithi122 ceph-mon[93869]: pgmap v18853: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 508 KiB/s rd, 506 KiB/s wr, 9 op/s 2023-03-10T00:31:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34391.470985 secs 2023-03-10T00:31:21.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:20 smithi181 ceph-mon[94189]: pgmap v18853: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 508 KiB/s rd, 506 KiB/s wr, 9 op/s 2023-03-10T00:31:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34391.470985 secs 2023-03-10T00:31:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:20 smithi038 ceph-mon[92422]: pgmap v18853: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 508 KiB/s rd, 506 KiB/s wr, 9 op/s 2023-03-10T00:31:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34391.470985 secs 2023-03-10T00:31:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:22 smithi122 ceph-mon[93869]: pgmap v18854: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 129 KiB/s wr, 6 op/s 2023-03-10T00:31:23.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:22 smithi181 ceph-mon[94189]: pgmap v18854: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 129 KiB/s wr, 6 op/s 2023-03-10T00:31:23.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:22 smithi038 ceph-mon[92422]: pgmap v18854: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 129 KiB/s wr, 6 op/s 2023-03-10T00:31:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34855.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:31:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34393.933773 secs 2023-03-10T00:31:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34393.933773 secs 2023-03-10T00:31:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34393.933773 secs 2023-03-10T00:31:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:25 smithi122 ceph-mon[93869]: pgmap v18855: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 128 KiB/s wr, 6 op/s 2023-03-10T00:31:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:25 smithi181 ceph-mon[94189]: pgmap v18855: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 128 KiB/s wr, 6 op/s 2023-03-10T00:31:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:25 smithi038 ceph-mon[92422]: pgmap v18855: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 128 KiB/s wr, 6 op/s 2023-03-10T00:31:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34396.471043 secs 2023-03-10T00:31:26.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34396.471043 secs 2023-03-10T00:31:26.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34396.471043 secs 2023-03-10T00:31:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:27 smithi122 ceph-mon[93869]: pgmap v18856: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-10T00:31:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:27 smithi181 ceph-mon[94189]: pgmap v18856: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-10T00:31:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:27 smithi038 ceph-mon[92422]: pgmap v18856: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-10T00:31:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34860.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:31:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34398.933871 secs 2023-03-10T00:31:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34398.933871 secs 2023-03-10T00:31:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34398.933871 secs 2023-03-10T00:31:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:29 smithi122 ceph-mon[93869]: pgmap v18857: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-10T00:31:29.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:29 smithi181 ceph-mon[94189]: pgmap v18857: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-10T00:31:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:29 smithi038 ceph-mon[92422]: pgmap v18857: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-10T00:31:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:31 smithi122 ceph-mon[93869]: pgmap v18858: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2023-03-10T00:31:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34401.471104 secs 2023-03-10T00:31:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:31 smithi181 ceph-mon[94189]: pgmap v18858: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2023-03-10T00:31:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34401.471104 secs 2023-03-10T00:31:31.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:31 smithi038 ceph-mon[92422]: pgmap v18858: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2023-03-10T00:31:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34401.471104 secs 2023-03-10T00:31:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34865.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:31:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:33 smithi122 ceph-mon[93869]: pgmap v18859: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-10T00:31:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34403.933932 secs 2023-03-10T00:31:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:33 smithi181 ceph-mon[94189]: pgmap v18859: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-10T00:31:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34403.933932 secs 2023-03-10T00:31:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:33 smithi038 ceph-mon[92422]: pgmap v18859: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-10T00:31:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34403.933932 secs 2023-03-10T00:31:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:35 smithi122 ceph-mon[93869]: pgmap v18860: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-10T00:31:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:35 smithi181 ceph-mon[94189]: pgmap v18860: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-10T00:31:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:35 smithi038 ceph-mon[92422]: pgmap v18860: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-10T00:31:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34406.471162 secs 2023-03-10T00:31:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34406.471162 secs 2023-03-10T00:31:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34406.471162 secs 2023-03-10T00:31:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:37 smithi122 ceph-mon[93869]: pgmap v18861: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 341 B/s wr, 7 op/s 2023-03-10T00:31:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:37 smithi181 ceph-mon[94189]: pgmap v18861: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 341 B/s wr, 7 op/s 2023-03-10T00:31:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:37 smithi038 ceph-mon[92422]: pgmap v18861: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 341 B/s wr, 7 op/s 2023-03-10T00:31:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34870.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:31:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34408.933998 secs 2023-03-10T00:31:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34408.933998 secs 2023-03-10T00:31:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34408.933998 secs 2023-03-10T00:31:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:39 smithi122 ceph-mon[93869]: pgmap v18862: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2023-03-10T00:31:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:39 smithi181 ceph-mon[94189]: pgmap v18862: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2023-03-10T00:31:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:39 smithi038 ceph-mon[92422]: pgmap v18862: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2023-03-10T00:31:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:41 smithi122 ceph-mon[93869]: pgmap v18863: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2023-03-10T00:31:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34411.471220 secs 2023-03-10T00:31:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:41 smithi181 ceph-mon[94189]: pgmap v18863: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2023-03-10T00:31:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34411.471220 secs 2023-03-10T00:31:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:41 smithi038 ceph-mon[92422]: pgmap v18863: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2023-03-10T00:31:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34411.471220 secs 2023-03-10T00:31:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34875.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:31:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:43 smithi122 ceph-mon[93869]: pgmap v18864: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 135 KiB/s wr, 9 op/s 2023-03-10T00:31:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:43 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 34413.934070 secs 2023-03-10T00:31:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:43 smithi181 ceph-mon[94189]: pgmap v18864: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 135 KiB/s wr, 9 op/s 2023-03-10T00:31:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:43 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 34413.934070 secs 2023-03-10T00:31:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:43 smithi038 ceph-mon[92422]: pgmap v18864: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 135 KiB/s wr, 9 op/s 2023-03-10T00:31:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:43 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 34413.934070 secs 2023-03-10T00:31:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:45 smithi122 ceph-mon[93869]: pgmap v18865: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 135 KiB/s wr, 6 op/s 2023-03-10T00:31:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:45 smithi181 ceph-mon[94189]: pgmap v18865: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 135 KiB/s wr, 6 op/s 2023-03-10T00:31:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:45 smithi038 ceph-mon[92422]: pgmap v18865: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 135 KiB/s wr, 6 op/s 2023-03-10T00:31:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34416.471278 secs 2023-03-10T00:31:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:46 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:31:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:46 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:31:46.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34416.471278 secs 2023-03-10T00:31:46.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:46 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:31:46.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:46 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:31:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34416.471278 secs 2023-03-10T00:31:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:46 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:31:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:46 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:31:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:47 smithi122 ceph-mon[93869]: pgmap v18866: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 348 KiB/s wr, 10 op/s 2023-03-10T00:31:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:47 smithi181 ceph-mon[94189]: pgmap v18866: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 348 KiB/s wr, 10 op/s 2023-03-10T00:31:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:47 smithi038 ceph-mon[92422]: pgmap v18866: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 348 KiB/s wr, 10 op/s 2023-03-10T00:31:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34880.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:31:48.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:48 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 34418.934116 secs 2023-03-10T00:31:48.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:48 smithi122 ceph-mon[93869]: pgmap v18867: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 432 KiB/s rd, 348 KiB/s wr, 7 op/s 2023-03-10T00:31:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:48 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 34418.934116 secs 2023-03-10T00:31:48.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:48 smithi181 ceph-mon[94189]: pgmap v18867: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 432 KiB/s rd, 348 KiB/s wr, 7 op/s 2023-03-10T00:31:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:48 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 34418.934116 secs 2023-03-10T00:31:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:48 smithi038 ceph-mon[92422]: pgmap v18867: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 432 KiB/s rd, 348 KiB/s wr, 7 op/s 2023-03-10T00:31:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:31:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:31:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:31:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:31:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:31:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:31:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:50 smithi122 ceph-mon[93869]: pgmap v18868: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 432 KiB/s rd, 348 KiB/s wr, 7 op/s 2023-03-10T00:31:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34421.471340 secs 2023-03-10T00:31:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:50 smithi181 ceph-mon[94189]: pgmap v18868: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 432 KiB/s rd, 348 KiB/s wr, 7 op/s 2023-03-10T00:31:50.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34421.471340 secs 2023-03-10T00:31:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:50 smithi038 ceph-mon[92422]: pgmap v18868: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 432 KiB/s rd, 348 KiB/s wr, 7 op/s 2023-03-10T00:31:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34421.471340 secs 2023-03-10T00:31:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:31:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:31:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:31:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:31:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:31:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:31:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:52 smithi122 ceph-mon[93869]: pgmap v18869: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 686 KiB/s wr, 12 op/s 2023-03-10T00:31:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:52 smithi181 ceph-mon[94189]: pgmap v18869: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 686 KiB/s wr, 12 op/s 2023-03-10T00:31:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:52 smithi038 ceph-mon[92422]: pgmap v18869: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 686 KiB/s wr, 12 op/s 2023-03-10T00:31:53.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34885.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:31:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:53 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 34423.934248 secs 2023-03-10T00:31:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:31:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:31:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:53 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 34423.934248 secs 2023-03-10T00:31:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:31:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:31:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:53 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 34423.934248 secs 2023-03-10T00:31:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:31:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:31:54.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:54 smithi122 ceph-mon[93869]: pgmap v18870: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 960 KiB/s rd, 551 KiB/s wr, 8 op/s 2023-03-10T00:31:54.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:54 smithi181 ceph-mon[94189]: pgmap v18870: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 960 KiB/s rd, 551 KiB/s wr, 8 op/s 2023-03-10T00:31:54.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:54 smithi038 ceph-mon[92422]: pgmap v18870: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 960 KiB/s rd, 551 KiB/s wr, 8 op/s 2023-03-10T00:31:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:31:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:31:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34426.471432 secs 2023-03-10T00:31:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:31:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:31:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34426.471432 secs 2023-03-10T00:31:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:31:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:31:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34426.471432 secs 2023-03-10T00:31:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:56 smithi122 ceph-mon[93869]: pgmap v18871: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 995 KiB/s rd, 584 KiB/s wr, 12 op/s 2023-03-10T00:31:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:56 smithi181 ceph-mon[94189]: pgmap v18871: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 995 KiB/s rd, 584 KiB/s wr, 12 op/s 2023-03-10T00:31:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:56 smithi038 ceph-mon[92422]: pgmap v18871: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 995 KiB/s rd, 584 KiB/s wr, 12 op/s 2023-03-10T00:31:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34890.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:31:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34428.934308 secs 2023-03-10T00:31:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:58 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:31:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:58 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:31:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:58 smithi122 ceph-mon[93869]: pgmap v18872: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 730 KiB/s rd, 402 KiB/s wr, 10 op/s 2023-03-10T00:31:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34428.934308 secs 2023-03-10T00:31:58.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:58 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:31:58.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:58 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:31:58.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:58 smithi181 ceph-mon[94189]: pgmap v18872: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 730 KiB/s rd, 402 KiB/s wr, 10 op/s 2023-03-10T00:31:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34428.934308 secs 2023-03-10T00:31:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:58 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:31:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:58 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:31:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:58 smithi038 ceph-mon[92422]: pgmap v18872: 129 pgs: 129 active+clean; 13 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 730 KiB/s rd, 402 KiB/s wr, 10 op/s 2023-03-10T00:31:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:31:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:31:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:31:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:31:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:31:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:31:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:32:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:31:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:32:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:00 smithi122 ceph-mon[93869]: pgmap v18873: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 869 KiB/s rd, 693 KiB/s wr, 11 op/s 2023-03-10T00:32:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34431.471492 secs 2023-03-10T00:32:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:00 smithi181 ceph-mon[94189]: pgmap v18873: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 869 KiB/s rd, 693 KiB/s wr, 11 op/s 2023-03-10T00:32:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34431.471492 secs 2023-03-10T00:32:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:00 smithi038 ceph-mon[92422]: pgmap v18873: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 869 KiB/s rd, 693 KiB/s wr, 11 op/s 2023-03-10T00:32:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34431.471492 secs 2023-03-10T00:32:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:32:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:32:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:32:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:32:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:32:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:32:02.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:02 smithi122 ceph-mon[93869]: pgmap v18874: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 1002 KiB/s rd, 825 KiB/s wr, 16 op/s 2023-03-10T00:32:02.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:02 smithi181 ceph-mon[94189]: pgmap v18874: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 1002 KiB/s rd, 825 KiB/s wr, 16 op/s 2023-03-10T00:32:02.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:02 smithi038 ceph-mon[92422]: pgmap v18874: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 1002 KiB/s rd, 825 KiB/s wr, 16 op/s 2023-03-10T00:32:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34895.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:32:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34433.934390 secs 2023-03-10T00:32:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34433.934390 secs 2023-03-10T00:32:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34433.934390 secs 2023-03-10T00:32:04.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:04 smithi181 ceph-mon[94189]: pgmap v18875: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 538 KiB/s wr, 14 op/s 2023-03-10T00:32:04.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:04 smithi038 ceph-mon[92422]: pgmap v18875: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 538 KiB/s wr, 14 op/s 2023-03-10T00:32:05.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:04 smithi122 ceph-mon[93869]: pgmap v18875: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 538 KiB/s wr, 14 op/s 2023-03-10T00:32:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34436.471549 secs 2023-03-10T00:32:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34436.471549 secs 2023-03-10T00:32:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34436.471549 secs 2023-03-10T00:32:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:06 smithi038 ceph-mon[92422]: pgmap v18876: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 554 KiB/s rd, 560 KiB/s wr, 19 op/s 2023-03-10T00:32:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:06 smithi122 ceph-mon[93869]: pgmap v18876: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 554 KiB/s rd, 560 KiB/s wr, 19 op/s 2023-03-10T00:32:07.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:06 smithi181 ceph-mon[94189]: pgmap v18876: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 554 KiB/s rd, 560 KiB/s wr, 19 op/s 2023-03-10T00:32:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:32:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:32:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:32:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:32:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:32:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:32:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34900.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:32:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34438.934478 secs 2023-03-10T00:32:09.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:08 smithi038 ceph-mon[92422]: pgmap v18877: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 619 KiB/s rd, 628 KiB/s wr, 17 op/s 2023-03-10T00:32:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34438.934478 secs 2023-03-10T00:32:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:08 smithi122 ceph-mon[93869]: pgmap v18877: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 619 KiB/s rd, 628 KiB/s wr, 17 op/s 2023-03-10T00:32:09.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34438.934478 secs 2023-03-10T00:32:09.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:08 smithi181 ceph-mon[94189]: pgmap v18877: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 619 KiB/s rd, 628 KiB/s wr, 17 op/s 2023-03-10T00:32:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:32:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:32:10.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:32:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:32:10.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:32:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:32:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:10 smithi122 ceph-mon[93869]: osdmap e1395: 12 total, 12 up, 12 in 2023-03-10T00:32:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:10 smithi122 ceph-mon[93869]: pgmap v18879: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 393 KiB/s wr, 31 op/s 2023-03-10T00:32:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34441.471613 secs 2023-03-10T00:32:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:10 smithi181 ceph-mon[94189]: osdmap e1395: 12 total, 12 up, 12 in 2023-03-10T00:32:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:10 smithi181 ceph-mon[94189]: pgmap v18879: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 393 KiB/s wr, 31 op/s 2023-03-10T00:32:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34441.471613 secs 2023-03-10T00:32:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:10 smithi038 ceph-mon[92422]: osdmap e1395: 12 total, 12 up, 12 in 2023-03-10T00:32:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:10 smithi038 ceph-mon[92422]: pgmap v18879: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 393 KiB/s wr, 31 op/s 2023-03-10T00:32:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34441.471613 secs 2023-03-10T00:32:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:12 smithi122 ceph-mon[93869]: pgmap v18880: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 766 KiB/s rd, 612 KiB/s wr, 30 op/s 2023-03-10T00:32:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:12 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:32:13.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:12 smithi181 ceph-mon[94189]: pgmap v18880: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 766 KiB/s rd, 612 KiB/s wr, 30 op/s 2023-03-10T00:32:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:12 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:32:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:12 smithi038 ceph-mon[92422]: pgmap v18880: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 766 KiB/s rd, 612 KiB/s wr, 30 op/s 2023-03-10T00:32:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:12 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:32:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34905.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:32:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34443.934536 secs 2023-03-10T00:32:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34443.934536 secs 2023-03-10T00:32:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34443.934536 secs 2023-03-10T00:32:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:15 smithi122 ceph-mon[93869]: pgmap v18881: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 509 KiB/s rd, 551 KiB/s wr, 28 op/s 2023-03-10T00:32:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:15 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:32:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:15 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:32:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:15 smithi181 ceph-mon[94189]: pgmap v18881: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 509 KiB/s rd, 551 KiB/s wr, 28 op/s 2023-03-10T00:32:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:15 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:32:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:15 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:32:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:15 smithi038 ceph-mon[92422]: pgmap v18881: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 509 KiB/s rd, 551 KiB/s wr, 28 op/s 2023-03-10T00:32:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:15 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:32:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:15 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:32:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34446.471670 secs 2023-03-10T00:32:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:16 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:32:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:16 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:32:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:16 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:32:16.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:16 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:32:16.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:16 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:32:16.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:16 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:32:16.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:16 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:32:16.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:16 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:32:16.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:16 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:32:16.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34446.471670 secs 2023-03-10T00:32:16.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:16 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:32:16.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:16 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:32:16.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:16 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:32:16.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:16 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:32:16.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:16 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:32:16.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:16 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:32:16.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:16 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:32:16.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:16 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:32:16.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:16 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:32:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34446.471670 secs 2023-03-10T00:32:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:16 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:32:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:16 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:32:17.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:16 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:32:17.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:16 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:32:17.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:16 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:32:17.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:16 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:32:17.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:16 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:32:17.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:16 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:32:17.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:16 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:32:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:17 smithi122 ceph-mon[93869]: pgmap v18882: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 636 KiB/s rd, 652 KiB/s wr, 24 op/s 2023-03-10T00:32:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:17 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-10T00:32:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:17 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-10T00:32:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:32:17.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:32:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:17 smithi181 ceph-mon[94189]: pgmap v18882: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 636 KiB/s rd, 652 KiB/s wr, 24 op/s 2023-03-10T00:32:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:17 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-10T00:32:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:17 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-10T00:32:17.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:32:17.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:32:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:17 smithi038 ceph-mon[92422]: pgmap v18882: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 636 KiB/s rd, 652 KiB/s wr, 24 op/s 2023-03-10T00:32:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:17 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-10T00:32:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:17 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-10T00:32:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:32:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:32:18.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34910.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:32:19.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34448.934616 secs 2023-03-10T00:32:19.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:18 smithi181 ceph-mon[94189]: pgmap v18883: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 517 KiB/s rd, 532 KiB/s wr, 23 op/s 2023-03-10T00:32:19.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:18 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:32:19.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:18 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:32:19.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:18 smithi181 ceph-mon[94189]: osdmap e1396: 12 total, 12 up, 12 in 2023-03-10T00:32:19.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:18 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:32:19.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:18 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:32:19.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:18 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:32:19.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:18 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:32:19.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:18 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-10T00:32:19.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:18 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-10T00:32:19.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:18 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:32:19.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:18 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:32:19.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:18 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:32:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34448.934616 secs 2023-03-10T00:32:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:18 smithi038 ceph-mon[92422]: pgmap v18883: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 517 KiB/s rd, 532 KiB/s wr, 23 op/s 2023-03-10T00:32:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:18 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:32:19.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:18 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:32:19.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:18 smithi038 ceph-mon[92422]: osdmap e1396: 12 total, 12 up, 12 in 2023-03-10T00:32:19.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:18 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:32:19.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:18 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:32:19.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:18 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:32:19.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:18 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:32:19.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:18 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-10T00:32:19.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:18 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-10T00:32:19.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:18 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:32:19.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:18 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:32:19.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:18 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:32:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34448.934616 secs 2023-03-10T00:32:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:18 smithi122 ceph-mon[93869]: pgmap v18883: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 517 KiB/s rd, 532 KiB/s wr, 23 op/s 2023-03-10T00:32:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:18 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:32:19.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:18 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:32:19.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:18 smithi122 ceph-mon[93869]: osdmap e1396: 12 total, 12 up, 12 in 2023-03-10T00:32:19.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:18 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:32:19.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:18 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:32:19.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:18 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:32:19.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:18 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:32:19.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:18 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-10T00:32:19.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:18 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-10T00:32:19.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:18 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:32:19.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:18 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:32:19.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:18 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:32:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:20 smithi038 ceph-mon[92422]: pgmap v18885: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 509 KiB/s rd, 506 KiB/s wr, 9 op/s 2023-03-10T00:32:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34451.471729 secs 2023-03-10T00:32:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:20 smithi122 ceph-mon[93869]: pgmap v18885: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 509 KiB/s rd, 506 KiB/s wr, 9 op/s 2023-03-10T00:32:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34451.471729 secs 2023-03-10T00:32:21.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:20 smithi181 ceph-mon[94189]: pgmap v18885: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 509 KiB/s rd, 506 KiB/s wr, 9 op/s 2023-03-10T00:32:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34451.471729 secs 2023-03-10T00:32:23.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:22 smithi038 ceph-mon[92422]: pgmap v18886: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 129 KiB/s wr, 8 op/s 2023-03-10T00:32:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:22 smithi122 ceph-mon[93869]: pgmap v18886: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 129 KiB/s wr, 8 op/s 2023-03-10T00:32:23.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:22 smithi181 ceph-mon[94189]: pgmap v18886: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 129 KiB/s wr, 8 op/s 2023-03-10T00:32:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34915.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:32:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34453.934673 secs 2023-03-10T00:32:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34453.934673 secs 2023-03-10T00:32:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34453.934673 secs 2023-03-10T00:32:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:25 smithi122 ceph-mon[93869]: pgmap v18887: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 129 KiB/s wr, 7 op/s 2023-03-10T00:32:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:25 smithi181 ceph-mon[94189]: pgmap v18887: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 129 KiB/s wr, 7 op/s 2023-03-10T00:32:26.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:25 smithi038 ceph-mon[92422]: pgmap v18887: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 129 KiB/s wr, 7 op/s 2023-03-10T00:32:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34456.471814 secs 2023-03-10T00:32:26.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34456.471814 secs 2023-03-10T00:32:26.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34456.471814 secs 2023-03-10T00:32:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:27 smithi122 ceph-mon[93869]: pgmap v18888: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 409 B/s wr, 7 op/s 2023-03-10T00:32:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:27 smithi181 ceph-mon[94189]: pgmap v18888: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 409 B/s wr, 7 op/s 2023-03-10T00:32:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:27 smithi038 ceph-mon[92422]: pgmap v18888: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 409 B/s wr, 7 op/s 2023-03-10T00:32:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34920.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:32:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34458.934733 secs 2023-03-10T00:32:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34458.934733 secs 2023-03-10T00:32:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34458.934733 secs 2023-03-10T00:32:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:29 smithi122 ceph-mon[93869]: pgmap v18889: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-10T00:32:29.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:29 smithi181 ceph-mon[94189]: pgmap v18889: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-10T00:32:30.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:29 smithi038 ceph-mon[92422]: pgmap v18889: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-10T00:32:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:31 smithi122 ceph-mon[93869]: pgmap v18890: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2023-03-10T00:32:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34461.471903 secs 2023-03-10T00:32:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:31 smithi181 ceph-mon[94189]: pgmap v18890: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2023-03-10T00:32:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34461.471903 secs 2023-03-10T00:32:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:31 smithi038 ceph-mon[92422]: pgmap v18890: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2023-03-10T00:32:32.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34461.471903 secs 2023-03-10T00:32:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34925.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:32:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:33 smithi122 ceph-mon[93869]: pgmap v18891: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-10T00:32:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34463.934795 secs 2023-03-10T00:32:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:33 smithi181 ceph-mon[94189]: pgmap v18891: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-10T00:32:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34463.934795 secs 2023-03-10T00:32:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:33 smithi038 ceph-mon[92422]: pgmap v18891: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-10T00:32:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34463.934795 secs 2023-03-10T00:32:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:35 smithi122 ceph-mon[93869]: pgmap v18892: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-10T00:32:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:35 smithi181 ceph-mon[94189]: pgmap v18892: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-10T00:32:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:35 smithi038 ceph-mon[92422]: pgmap v18892: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-10T00:32:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34466.471960 secs 2023-03-10T00:32:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34466.471960 secs 2023-03-10T00:32:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34466.471960 secs 2023-03-10T00:32:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:37 smithi122 ceph-mon[93869]: pgmap v18893: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 130 KiB/s wr, 8 op/s 2023-03-10T00:32:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:37 smithi181 ceph-mon[94189]: pgmap v18893: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 130 KiB/s wr, 8 op/s 2023-03-10T00:32:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:37 smithi038 ceph-mon[92422]: pgmap v18893: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 353 KiB/s rd, 130 KiB/s wr, 8 op/s 2023-03-10T00:32:38.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34930.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:32:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34468.934858 secs 2023-03-10T00:32:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34468.934858 secs 2023-03-10T00:32:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34468.934858 secs 2023-03-10T00:32:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:39 smithi122 ceph-mon[93869]: pgmap v18894: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 130 KiB/s wr, 7 op/s 2023-03-10T00:32:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:39 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:32:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:39 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:32:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:39 smithi181 ceph-mon[94189]: pgmap v18894: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 130 KiB/s wr, 7 op/s 2023-03-10T00:32:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:39 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:32:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:39 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:32:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:39 smithi038 ceph-mon[92422]: pgmap v18894: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 130 KiB/s wr, 7 op/s 2023-03-10T00:32:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:39 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:32:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:39 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:32:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:40 smithi122 ceph-mon[93869]: pgmap v18895: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 130 KiB/s wr, 7 op/s 2023-03-10T00:32:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34471.472047 secs 2023-03-10T00:32:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:40 smithi181 ceph-mon[94189]: pgmap v18895: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 130 KiB/s wr, 7 op/s 2023-03-10T00:32:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34471.472047 secs 2023-03-10T00:32:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:40 smithi038 ceph-mon[92422]: pgmap v18895: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 130 KiB/s wr, 7 op/s 2023-03-10T00:32:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34471.472047 secs 2023-03-10T00:32:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:32:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:32:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:32:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:32:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:32:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:32:42.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:42 smithi122 ceph-mon[93869]: pgmap v18896: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 265 KiB/s wr, 10 op/s 2023-03-10T00:32:42.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:42 smithi181 ceph-mon[94189]: pgmap v18896: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 265 KiB/s wr, 10 op/s 2023-03-10T00:32:42.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:42 smithi038 ceph-mon[92422]: pgmap v18896: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 265 KiB/s wr, 10 op/s 2023-03-10T00:32:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34935.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:32:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34473.934919 secs 2023-03-10T00:32:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34473.934919 secs 2023-03-10T00:32:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34473.934919 secs 2023-03-10T00:32:44.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:44 smithi122 ceph-mon[93869]: pgmap v18897: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 486 KiB/s rd, 265 KiB/s wr, 7 op/s 2023-03-10T00:32:44.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:44 smithi181 ceph-mon[94189]: pgmap v18897: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 486 KiB/s rd, 265 KiB/s wr, 7 op/s 2023-03-10T00:32:44.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:44 smithi038 ceph-mon[92422]: pgmap v18897: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 486 KiB/s rd, 265 KiB/s wr, 7 op/s 2023-03-10T00:32:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34476.472106 secs 2023-03-10T00:32:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34476.472106 secs 2023-03-10T00:32:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34476.472106 secs 2023-03-10T00:32:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:46 smithi122 ceph-mon[93869]: pgmap v18898: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 730 KiB/s rd, 452 KiB/s wr, 10 op/s 2023-03-10T00:32:46.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:46 smithi181 ceph-mon[94189]: pgmap v18898: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 730 KiB/s rd, 452 KiB/s wr, 10 op/s 2023-03-10T00:32:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:46 smithi038 ceph-mon[92422]: pgmap v18898: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 730 KiB/s rd, 452 KiB/s wr, 10 op/s 2023-03-10T00:32:48.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34940.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:32:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34478.934982 secs 2023-03-10T00:32:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:48 smithi122 ceph-mon[93869]: pgmap v18899: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 323 KiB/s wr, 8 op/s 2023-03-10T00:32:49.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34478.934982 secs 2023-03-10T00:32:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:48 smithi181 ceph-mon[94189]: pgmap v18899: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 323 KiB/s wr, 8 op/s 2023-03-10T00:32:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34478.934982 secs 2023-03-10T00:32:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:48 smithi038 ceph-mon[92422]: pgmap v18899: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 323 KiB/s wr, 8 op/s 2023-03-10T00:32:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:50 smithi122 ceph-mon[93869]: pgmap v18900: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 322 KiB/s wr, 7 op/s 2023-03-10T00:32:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34481.472164 secs 2023-03-10T00:32:51.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:50 smithi181 ceph-mon[94189]: pgmap v18900: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 322 KiB/s wr, 7 op/s 2023-03-10T00:32:51.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34481.472164 secs 2023-03-10T00:32:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:50 smithi038 ceph-mon[92422]: pgmap v18900: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 380 KiB/s rd, 322 KiB/s wr, 7 op/s 2023-03-10T00:32:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34481.472164 secs 2023-03-10T00:32:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:52 smithi122 ceph-mon[93869]: pgmap v18901: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 979 KiB/s rd, 692 KiB/s wr, 12 op/s 2023-03-10T00:32:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:32:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:32:53.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:52 smithi181 ceph-mon[94189]: pgmap v18901: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 979 KiB/s rd, 692 KiB/s wr, 12 op/s 2023-03-10T00:32:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:32:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:32:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:52 smithi038 ceph-mon[92422]: pgmap v18901: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 979 KiB/s rd, 692 KiB/s wr, 12 op/s 2023-03-10T00:32:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:32:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:32:53.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34945.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:32:54.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34483.935070 secs 2023-03-10T00:32:54.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34483.935070 secs 2023-03-10T00:32:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34483.935070 secs 2023-03-10T00:32:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:55 smithi122 ceph-mon[93869]: pgmap v18902: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 844 KiB/s rd, 557 KiB/s wr, 8 op/s 2023-03-10T00:32:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:32:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:32:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:55 smithi181 ceph-mon[94189]: pgmap v18902: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 844 KiB/s rd, 557 KiB/s wr, 8 op/s 2023-03-10T00:32:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:32:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:32:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:55 smithi038 ceph-mon[92422]: pgmap v18902: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 844 KiB/s rd, 557 KiB/s wr, 8 op/s 2023-03-10T00:32:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:32:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:32:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34486.472222 secs 2023-03-10T00:32:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34486.472222 secs 2023-03-10T00:32:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34486.472222 secs 2023-03-10T00:32:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:57 smithi122 ceph-mon[93869]: pgmap v18903: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 847 KiB/s rd, 558 KiB/s wr, 11 op/s 2023-03-10T00:32:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:57 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:32:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:57 smithi181 ceph-mon[94189]: pgmap v18903: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 847 KiB/s rd, 558 KiB/s wr, 11 op/s 2023-03-10T00:32:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:57 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:32:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:57 smithi038 ceph-mon[92422]: pgmap v18903: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 847 KiB/s rd, 558 KiB/s wr, 11 op/s 2023-03-10T00:32:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:57 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:32:58.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34950.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:32:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:58 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:32:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:58 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:32:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:58 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:32:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:58 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:32:58.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:58 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:32:58.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:58 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:32:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:58 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:32:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:58 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:32:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:58 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:32:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:59 smithi122 ceph-mon[93869]: pgmap v18904: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 803 KiB/s rd, 557 KiB/s wr, 10 op/s 2023-03-10T00:32:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:32:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:32:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:32:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:59 smithi181 ceph-mon[94189]: pgmap v18904: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 803 KiB/s rd, 557 KiB/s wr, 10 op/s 2023-03-10T00:32:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:32:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:32:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:32:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:59 smithi038 ceph-mon[92422]: pgmap v18904: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 803 KiB/s rd, 557 KiB/s wr, 10 op/s 2023-03-10T00:32:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:32:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:32:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:33:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:01 smithi122 ceph-mon[93869]: pgmap v18905: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 814 KiB/s rd, 557 KiB/s wr, 10 op/s 2023-03-10T00:33:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34491.472351 secs 2023-03-10T00:33:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:33:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:33:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:01 smithi181 ceph-mon[94189]: pgmap v18905: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 814 KiB/s rd, 557 KiB/s wr, 10 op/s 2023-03-10T00:33:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34491.472351 secs 2023-03-10T00:33:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:33:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:33:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:01 smithi038 ceph-mon[92422]: pgmap v18905: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 814 KiB/s rd, 557 KiB/s wr, 10 op/s 2023-03-10T00:33:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34491.472351 secs 2023-03-10T00:33:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:33:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:33:03.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34955.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:33:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:03 smithi122 ceph-mon[93869]: pgmap v18906: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 771 KiB/s wr, 16 op/s 2023-03-10T00:33:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:03 smithi181 ceph-mon[94189]: pgmap v18906: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 771 KiB/s wr, 16 op/s 2023-03-10T00:33:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:03 smithi038 ceph-mon[92422]: pgmap v18906: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 771 KiB/s wr, 16 op/s 2023-03-10T00:33:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:05 smithi122 ceph-mon[93869]: pgmap v18907: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 562 KiB/s rd, 535 KiB/s wr, 13 op/s 2023-03-10T00:33:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:05 smithi181 ceph-mon[94189]: pgmap v18907: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 562 KiB/s rd, 535 KiB/s wr, 13 op/s 2023-03-10T00:33:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:05 smithi038 ceph-mon[92422]: pgmap v18907: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 562 KiB/s rd, 535 KiB/s wr, 13 op/s 2023-03-10T00:33:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34496.472448 secs 2023-03-10T00:33:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34496.472448 secs 2023-03-10T00:33:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34496.472448 secs 2023-03-10T00:33:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:07 smithi122 ceph-mon[93869]: pgmap v18908: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 557 KiB/s wr, 17 op/s 2023-03-10T00:33:07.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:33:07.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:33:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:07 smithi181 ceph-mon[94189]: pgmap v18908: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 557 KiB/s wr, 17 op/s 2023-03-10T00:33:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:33:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:33:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:07 smithi038 ceph-mon[92422]: pgmap v18908: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 557 KiB/s wr, 17 op/s 2023-03-10T00:33:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:33:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:33:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34960.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:33:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34498.935294 secs 2023-03-10T00:33:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34498.935294 secs 2023-03-10T00:33:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34498.935294 secs 2023-03-10T00:33:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:09 smithi122 ceph-mon[93869]: pgmap v18909: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 613 KiB/s rd, 608 KiB/s wr, 16 op/s 2023-03-10T00:33:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:33:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:33:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:09 smithi181 ceph-mon[94189]: pgmap v18909: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 613 KiB/s rd, 608 KiB/s wr, 16 op/s 2023-03-10T00:33:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:33:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:33:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:09 smithi038 ceph-mon[92422]: pgmap v18909: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 613 KiB/s rd, 608 KiB/s wr, 16 op/s 2023-03-10T00:33:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:33:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:33:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:10 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:33:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:10 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:33:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:10 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:33:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:10 smithi122 ceph-mon[93869]: osdmap e1397: 12 total, 12 up, 12 in 2023-03-10T00:33:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:10 smithi122 ceph-mon[93869]: pgmap v18911: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 785 KiB/s wr, 31 op/s 2023-03-10T00:33:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34501.472509 secs 2023-03-10T00:33:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:10 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:33:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:10 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:33:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:10 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:33:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:10 smithi181 ceph-mon[94189]: osdmap e1397: 12 total, 12 up, 12 in 2023-03-10T00:33:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:10 smithi181 ceph-mon[94189]: pgmap v18911: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 785 KiB/s wr, 31 op/s 2023-03-10T00:33:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34501.472509 secs 2023-03-10T00:33:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:10 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:33:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:10 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:33:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:10 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:33:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:10 smithi038 ceph-mon[92422]: osdmap e1397: 12 total, 12 up, 12 in 2023-03-10T00:33:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:10 smithi038 ceph-mon[92422]: pgmap v18911: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 785 KiB/s wr, 31 op/s 2023-03-10T00:33:11.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34501.472509 secs 2023-03-10T00:33:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:12 smithi122 ceph-mon[93869]: pgmap v18912: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 528 KiB/s wr, 28 op/s 2023-03-10T00:33:13.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:12 smithi181 ceph-mon[94189]: pgmap v18912: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 528 KiB/s wr, 28 op/s 2023-03-10T00:33:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:12 smithi038 ceph-mon[92422]: pgmap v18912: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 528 KiB/s wr, 28 op/s 2023-03-10T00:33:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34965.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:33:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34503.935350 secs 2023-03-10T00:33:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34503.935350 secs 2023-03-10T00:33:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34503.935350 secs 2023-03-10T00:33:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:15 smithi122 ceph-mon[93869]: pgmap v18913: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 552 KiB/s wr, 27 op/s 2023-03-10T00:33:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:15 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:33:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:15 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:33:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:15 smithi181 ceph-mon[94189]: pgmap v18913: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 552 KiB/s wr, 27 op/s 2023-03-10T00:33:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:15 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:33:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:15 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:33:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:15 smithi038 ceph-mon[92422]: pgmap v18913: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 552 KiB/s wr, 27 op/s 2023-03-10T00:33:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:15 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:33:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:15 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:33:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34506.472566 secs 2023-03-10T00:33:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34506.472566 secs 2023-03-10T00:33:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34506.472566 secs 2023-03-10T00:33:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:17 smithi122 ceph-mon[93869]: pgmap v18914: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 509 KiB/s rd, 525 KiB/s wr, 23 op/s 2023-03-10T00:33:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:33:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:33:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:17 smithi181 ceph-mon[94189]: pgmap v18914: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 509 KiB/s rd, 525 KiB/s wr, 23 op/s 2023-03-10T00:33:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:33:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:33:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:17 smithi038 ceph-mon[92422]: pgmap v18914: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 509 KiB/s rd, 525 KiB/s wr, 23 op/s 2023-03-10T00:33:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:33:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:33:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34970.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:33:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34508.935440 secs 2023-03-10T00:33:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:18 smithi122 ceph-mon[93869]: osdmap e1398: 12 total, 12 up, 12 in 2023-03-10T00:33:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34508.935440 secs 2023-03-10T00:33:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:18 smithi181 ceph-mon[94189]: osdmap e1398: 12 total, 12 up, 12 in 2023-03-10T00:33:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34508.935440 secs 2023-03-10T00:33:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:18 smithi038 ceph-mon[92422]: osdmap e1398: 12 total, 12 up, 12 in 2023-03-10T00:33:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:19 smithi122 ceph-mon[93869]: pgmap v18915: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 463 KiB/s wr, 22 op/s 2023-03-10T00:33:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:33:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:33:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:33:19.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:33:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:19 smithi181 ceph-mon[94189]: pgmap v18915: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 463 KiB/s wr, 22 op/s 2023-03-10T00:33:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:33:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:33:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:33:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:33:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:19 smithi038 ceph-mon[92422]: pgmap v18915: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 463 KiB/s wr, 22 op/s 2023-03-10T00:33:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:33:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:33:20.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:33:20.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:33:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:21 smithi122 ceph-mon[93869]: pgmap v18917: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 312 KiB/s wr, 8 op/s 2023-03-10T00:33:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34511.472645 secs 2023-03-10T00:33:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:21 smithi181 ceph-mon[94189]: pgmap v18917: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 312 KiB/s wr, 8 op/s 2023-03-10T00:33:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34511.472645 secs 2023-03-10T00:33:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:21 smithi038 ceph-mon[92422]: pgmap v18917: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 312 KiB/s wr, 8 op/s 2023-03-10T00:33:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34511.472645 secs 2023-03-10T00:33:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34975.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:33:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:23 smithi122 ceph-mon[93869]: pgmap v18918: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 312 KiB/s wr, 8 op/s 2023-03-10T00:33:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34513.935530 secs 2023-03-10T00:33:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:23 smithi181 ceph-mon[94189]: pgmap v18918: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 312 KiB/s wr, 8 op/s 2023-03-10T00:33:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34513.935530 secs 2023-03-10T00:33:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:23 smithi038 ceph-mon[92422]: pgmap v18918: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 312 KiB/s wr, 8 op/s 2023-03-10T00:33:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34513.935530 secs 2023-03-10T00:33:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:25 smithi122 ceph-mon[93869]: pgmap v18919: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 129 KiB/s wr, 7 op/s 2023-03-10T00:33:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34516.472720 secs 2023-03-10T00:33:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:25 smithi181 ceph-mon[94189]: pgmap v18919: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 129 KiB/s wr, 7 op/s 2023-03-10T00:33:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34516.472720 secs 2023-03-10T00:33:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:25 smithi038 ceph-mon[92422]: pgmap v18919: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 129 KiB/s wr, 7 op/s 2023-03-10T00:33:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34516.472720 secs 2023-03-10T00:33:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:27 smithi122 ceph-mon[93869]: pgmap v18920: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 129 KiB/s wr, 7 op/s 2023-03-10T00:33:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:27 smithi181 ceph-mon[94189]: pgmap v18920: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 129 KiB/s wr, 7 op/s 2023-03-10T00:33:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:27 smithi038 ceph-mon[92422]: pgmap v18920: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 129 KiB/s wr, 7 op/s 2023-03-10T00:33:28.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34980.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:33:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34518.935590 secs 2023-03-10T00:33:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:28 smithi122 ceph-mon[93869]: pgmap v18921: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 129 KiB/s wr, 7 op/s 2023-03-10T00:33:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34518.935590 secs 2023-03-10T00:33:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:28 smithi181 ceph-mon[94189]: pgmap v18921: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 129 KiB/s wr, 7 op/s 2023-03-10T00:33:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34518.935590 secs 2023-03-10T00:33:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:28 smithi038 ceph-mon[92422]: pgmap v18921: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 131 KiB/s rd, 129 KiB/s wr, 7 op/s 2023-03-10T00:33:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:30 smithi122 ceph-mon[93869]: pgmap v18922: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2023-03-10T00:33:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34521.472803 secs 2023-03-10T00:33:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:30 smithi181 ceph-mon[94189]: pgmap v18922: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2023-03-10T00:33:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34521.472803 secs 2023-03-10T00:33:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:30 smithi038 ceph-mon[92422]: pgmap v18922: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s 2023-03-10T00:33:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34521.472803 secs 2023-03-10T00:33:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:32 smithi122 ceph-mon[93869]: pgmap v18923: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-10T00:33:33.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:32 smithi181 ceph-mon[94189]: pgmap v18923: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-10T00:33:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:32 smithi038 ceph-mon[92422]: pgmap v18923: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-10T00:33:33.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34985.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:33:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34523.935649 secs 2023-03-10T00:33:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34523.935649 secs 2023-03-10T00:33:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34523.935649 secs 2023-03-10T00:33:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:35 smithi122 ceph-mon[93869]: pgmap v18924: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-10T00:33:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:35 smithi181 ceph-mon[94189]: pgmap v18924: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-10T00:33:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:35 smithi038 ceph-mon[92422]: pgmap v18924: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-10T00:33:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34526.472861 secs 2023-03-10T00:33:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34526.472861 secs 2023-03-10T00:33:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34526.472861 secs 2023-03-10T00:33:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:37 smithi122 ceph-mon[93869]: pgmap v18925: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-10T00:33:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:37 smithi181 ceph-mon[94189]: pgmap v18925: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-10T00:33:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:37 smithi038 ceph-mon[92422]: pgmap v18925: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-10T00:33:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34990.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:33:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34528.935727 secs 2023-03-10T00:33:38.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34528.935727 secs 2023-03-10T00:33:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34528.935727 secs 2023-03-10T00:33:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:39 smithi122 ceph-mon[93869]: pgmap v18926: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-10T00:33:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:39 smithi181 ceph-mon[94189]: pgmap v18926: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-10T00:33:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:39 smithi038 ceph-mon[92422]: pgmap v18926: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-10T00:33:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:40 smithi122 ceph-mon[93869]: pgmap v18927: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-10T00:33:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34531.472918 secs 2023-03-10T00:33:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:40 smithi181 ceph-mon[94189]: pgmap v18927: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-10T00:33:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34531.472918 secs 2023-03-10T00:33:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:40 smithi038 ceph-mon[92422]: pgmap v18927: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-10T00:33:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34531.472918 secs 2023-03-10T00:33:43.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:42 smithi122 ceph-mon[93869]: pgmap v18928: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 426 B/s wr, 8 op/s 2023-03-10T00:33:43.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:42 smithi181 ceph-mon[94189]: pgmap v18928: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 426 B/s wr, 8 op/s 2023-03-10T00:33:43.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:42 smithi038 ceph-mon[92422]: pgmap v18928: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 426 B/s wr, 8 op/s 2023-03-10T00:33:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 34995.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:33:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:45 smithi122 ceph-mon[93869]: pgmap v18929: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 81 KiB/s wr, 7 op/s 2023-03-10T00:33:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:45 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:33:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:45 smithi181 ceph-mon[94189]: pgmap v18929: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 81 KiB/s wr, 7 op/s 2023-03-10T00:33:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:45 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:33:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:45 smithi038 ceph-mon[92422]: pgmap v18929: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 83 KiB/s rd, 81 KiB/s wr, 7 op/s 2023-03-10T00:33:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:45 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:33:46.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34536.472969 secs 2023-03-10T00:33:46.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:46 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:33:46.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:46 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:33:46.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:46 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:33:46.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34536.472969 secs 2023-03-10T00:33:46.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:46 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:33:46.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:46 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:33:46.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:46 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:33:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34536.472969 secs 2023-03-10T00:33:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:46 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:33:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:46 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:33:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:46 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:33:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:47 smithi122 ceph-mon[93869]: pgmap v18930: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 165 KiB/s wr, 9 op/s 2023-03-10T00:33:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:33:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:33:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:47 smithi181 ceph-mon[94189]: pgmap v18930: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 165 KiB/s wr, 9 op/s 2023-03-10T00:33:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:33:47.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:33:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:47 smithi038 ceph-mon[92422]: pgmap v18930: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 165 KiB/s wr, 9 op/s 2023-03-10T00:33:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:33:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:33:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35000.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:33:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:49 smithi122 ceph-mon[93869]: pgmap v18931: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 193 KiB/s wr, 8 op/s 2023-03-10T00:33:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:33:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:33:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:49 smithi181 ceph-mon[94189]: pgmap v18931: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 193 KiB/s wr, 8 op/s 2023-03-10T00:33:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:33:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:33:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:49 smithi038 ceph-mon[92422]: pgmap v18931: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 193 KiB/s wr, 8 op/s 2023-03-10T00:33:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:33:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:33:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:51 smithi122 ceph-mon[93869]: pgmap v18932: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 193 KiB/s wr, 8 op/s 2023-03-10T00:33:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34541.473028 secs 2023-03-10T00:33:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:51 smithi181 ceph-mon[94189]: pgmap v18932: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 193 KiB/s wr, 8 op/s 2023-03-10T00:33:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34541.473028 secs 2023-03-10T00:33:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:51 smithi038 ceph-mon[92422]: pgmap v18932: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 193 KiB/s wr, 8 op/s 2023-03-10T00:33:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34541.473028 secs 2023-03-10T00:33:53.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35005.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:33:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:53 smithi122 ceph-mon[93869]: pgmap v18933: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 506 KiB/s wr, 12 op/s 2023-03-10T00:33:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34543.935929 secs 2023-03-10T00:33:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:53 smithi181 ceph-mon[94189]: pgmap v18933: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 506 KiB/s wr, 12 op/s 2023-03-10T00:33:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34543.935929 secs 2023-03-10T00:33:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:53 smithi038 ceph-mon[92422]: pgmap v18933: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 506 KiB/s wr, 12 op/s 2023-03-10T00:33:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34543.935929 secs 2023-03-10T00:33:54.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:54 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:33:54.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:54 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:33:54.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:54 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:33:54.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:54 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:33:54.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:54 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:33:54.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:54 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:33:54.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:54 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:33:54.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:54 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:33:54.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:54 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:33:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:55 smithi122 ceph-mon[93869]: pgmap v18934: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 509 KiB/s rd, 505 KiB/s wr, 10 op/s 2023-03-10T00:33:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:33:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:33:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34546.473091 secs 2023-03-10T00:33:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:55 smithi181 ceph-mon[94189]: pgmap v18934: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 509 KiB/s rd, 505 KiB/s wr, 10 op/s 2023-03-10T00:33:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:33:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:33:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34546.473091 secs 2023-03-10T00:33:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:55 smithi038 ceph-mon[92422]: pgmap v18934: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 509 KiB/s rd, 505 KiB/s wr, 10 op/s 2023-03-10T00:33:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:33:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:33:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34546.473091 secs 2023-03-10T00:33:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:57 smithi122 ceph-mon[93869]: pgmap v18935: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 426 KiB/s wr, 12 op/s 2023-03-10T00:33:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:33:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:33:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:57 smithi181 ceph-mon[94189]: pgmap v18935: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 426 KiB/s wr, 12 op/s 2023-03-10T00:33:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:33:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:33:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:57 smithi038 ceph-mon[92422]: pgmap v18935: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 426 KiB/s wr, 12 op/s 2023-03-10T00:33:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:33:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:33:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35010.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:33:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34548.936024 secs 2023-03-10T00:33:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:58 smithi122 ceph-mon[93869]: pgmap v18936: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 689 KiB/s rd, 676 KiB/s wr, 12 op/s 2023-03-10T00:33:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34548.936024 secs 2023-03-10T00:33:58.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:58 smithi181 ceph-mon[94189]: pgmap v18936: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 689 KiB/s rd, 676 KiB/s wr, 12 op/s 2023-03-10T00:33:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34548.936024 secs 2023-03-10T00:33:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:58 smithi038 ceph-mon[92422]: pgmap v18936: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 689 KiB/s rd, 676 KiB/s wr, 12 op/s 2023-03-10T00:33:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:33:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:33:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:33:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:33:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:33:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:33:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:33:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:33:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:34:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:00 smithi122 ceph-mon[93869]: pgmap v18937: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 729 KiB/s rd, 716 KiB/s wr, 13 op/s 2023-03-10T00:34:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34551.473150 secs 2023-03-10T00:34:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:00 smithi181 ceph-mon[94189]: pgmap v18937: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 729 KiB/s rd, 716 KiB/s wr, 13 op/s 2023-03-10T00:34:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34551.473150 secs 2023-03-10T00:34:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:00 smithi038 ceph-mon[92422]: pgmap v18937: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 729 KiB/s rd, 716 KiB/s wr, 13 op/s 2023-03-10T00:34:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34551.473150 secs 2023-03-10T00:34:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:34:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:34:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:34:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:34:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:34:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:34:02.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:02 smithi122 ceph-mon[93869]: pgmap v18938: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 739 KiB/s wr, 18 op/s 2023-03-10T00:34:02.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:02 smithi181 ceph-mon[94189]: pgmap v18938: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 739 KiB/s wr, 18 op/s 2023-03-10T00:34:02.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:02 smithi038 ceph-mon[92422]: pgmap v18938: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 739 KiB/s wr, 18 op/s 2023-03-10T00:34:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35015.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:34:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34553.936083 secs 2023-03-10T00:34:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34553.936083 secs 2023-03-10T00:34:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34553.936083 secs 2023-03-10T00:34:04.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:04 smithi122 ceph-mon[93869]: pgmap v18939: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 836 KiB/s rd, 885 KiB/s wr, 29 op/s 2023-03-10T00:34:04.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:04 smithi181 ceph-mon[94189]: pgmap v18939: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 836 KiB/s rd, 885 KiB/s wr, 29 op/s 2023-03-10T00:34:04.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:04 smithi038 ceph-mon[92422]: pgmap v18939: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 836 KiB/s rd, 885 KiB/s wr, 29 op/s 2023-03-10T00:34:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:05 smithi122 ceph-mon[93869]: osdmap e1399: 12 total, 12 up, 12 in 2023-03-10T00:34:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34556.473213 secs 2023-03-10T00:34:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:05 smithi181 ceph-mon[94189]: osdmap e1399: 12 total, 12 up, 12 in 2023-03-10T00:34:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34556.473213 secs 2023-03-10T00:34:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:05 smithi038 ceph-mon[92422]: osdmap e1399: 12 total, 12 up, 12 in 2023-03-10T00:34:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34556.473213 secs 2023-03-10T00:34:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:06 smithi122 ceph-mon[93869]: pgmap v18941: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 989 KiB/s rd, 1.0 MiB/s wr, 32 op/s 2023-03-10T00:34:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:06 smithi181 ceph-mon[94189]: pgmap v18941: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 989 KiB/s rd, 1.0 MiB/s wr, 32 op/s 2023-03-10T00:34:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:06 smithi038 ceph-mon[92422]: pgmap v18941: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 989 KiB/s rd, 1.0 MiB/s wr, 32 op/s 2023-03-10T00:34:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:34:07.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:34:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:34:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:34:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:34:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:34:08.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35020.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:34:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34558.936198 secs 2023-03-10T00:34:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:08 smithi122 ceph-mon[93869]: pgmap v18942: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 589 KiB/s rd, 660 KiB/s wr, 30 op/s 2023-03-10T00:34:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34558.936198 secs 2023-03-10T00:34:08.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:08 smithi181 ceph-mon[94189]: pgmap v18942: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 589 KiB/s rd, 660 KiB/s wr, 30 op/s 2023-03-10T00:34:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34558.936198 secs 2023-03-10T00:34:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:08 smithi038 ceph-mon[92422]: pgmap v18942: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 589 KiB/s rd, 660 KiB/s wr, 30 op/s 2023-03-10T00:34:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:34:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:34:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:34:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:34:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:34:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:34:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:10 smithi122 ceph-mon[93869]: pgmap v18943: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 537 KiB/s rd, 578 KiB/s wr, 30 op/s 2023-03-10T00:34:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34561.473278 secs 2023-03-10T00:34:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:10 smithi181 ceph-mon[94189]: pgmap v18943: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 537 KiB/s rd, 578 KiB/s wr, 30 op/s 2023-03-10T00:34:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34561.473278 secs 2023-03-10T00:34:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:10 smithi038 ceph-mon[92422]: pgmap v18943: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 537 KiB/s rd, 578 KiB/s wr, 30 op/s 2023-03-10T00:34:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34561.473278 secs 2023-03-10T00:34:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:12 smithi122 ceph-mon[93869]: pgmap v18944: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 536 KiB/s rd, 551 KiB/s wr, 25 op/s 2023-03-10T00:34:13.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:12 smithi181 ceph-mon[94189]: pgmap v18944: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 536 KiB/s rd, 551 KiB/s wr, 25 op/s 2023-03-10T00:34:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:12 smithi038 ceph-mon[92422]: pgmap v18944: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 536 KiB/s rd, 551 KiB/s wr, 25 op/s 2023-03-10T00:34:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35025.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:34:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34563.936255 secs 2023-03-10T00:34:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:13 smithi122 ceph-mon[93869]: osdmap e1400: 12 total, 12 up, 12 in 2023-03-10T00:34:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34563.936255 secs 2023-03-10T00:34:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:13 smithi181 ceph-mon[94189]: osdmap e1400: 12 total, 12 up, 12 in 2023-03-10T00:34:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34563.936255 secs 2023-03-10T00:34:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:13 smithi038 ceph-mon[92422]: osdmap e1400: 12 total, 12 up, 12 in 2023-03-10T00:34:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:15 smithi122 ceph-mon[93869]: pgmap v18946: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 252 KiB/s wr, 10 op/s 2023-03-10T00:34:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:15 smithi181 ceph-mon[94189]: pgmap v18946: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 252 KiB/s wr, 10 op/s 2023-03-10T00:34:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:15 smithi038 ceph-mon[92422]: pgmap v18946: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 252 KiB/s wr, 10 op/s 2023-03-10T00:34:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34566.473352 secs 2023-03-10T00:34:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34566.473352 secs 2023-03-10T00:34:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34566.473352 secs 2023-03-10T00:34:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:17 smithi122 ceph-mon[93869]: pgmap v18947: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 232 KiB/s wr, 9 op/s 2023-03-10T00:34:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:17 smithi181 ceph-mon[94189]: pgmap v18947: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 232 KiB/s wr, 9 op/s 2023-03-10T00:34:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:17 smithi038 ceph-mon[92422]: pgmap v18947: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 232 KiB/s wr, 9 op/s 2023-03-10T00:34:18.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35030.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:34:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34568.936336 secs 2023-03-10T00:34:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34568.936336 secs 2023-03-10T00:34:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34568.936336 secs 2023-03-10T00:34:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:19 smithi122 ceph-mon[93869]: pgmap v18948: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 232 KiB/s wr, 8 op/s 2023-03-10T00:34:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:34:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:19 smithi181 ceph-mon[94189]: pgmap v18948: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 232 KiB/s wr, 8 op/s 2023-03-10T00:34:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:34:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:19 smithi038 ceph-mon[92422]: pgmap v18948: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 232 KiB/s wr, 8 op/s 2023-03-10T00:34:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:34:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:21 smithi122 ceph-mon[93869]: pgmap v18949: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 232 KiB/s wr, 8 op/s 2023-03-10T00:34:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34571.473414 secs 2023-03-10T00:34:21.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:34:21.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:34:21.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:34:21.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:34:21.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:21 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:34:21.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:21 smithi181 ceph-mon[94189]: pgmap v18949: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 232 KiB/s wr, 8 op/s 2023-03-10T00:34:21.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34571.473414 secs 2023-03-10T00:34:21.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:34:21.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:34:21.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:34:21.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:34:21.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:21 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:34:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:21 smithi038 ceph-mon[92422]: pgmap v18949: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 232 KiB/s wr, 8 op/s 2023-03-10T00:34:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34571.473414 secs 2023-03-10T00:34:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:34:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:34:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:34:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:34:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:21 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:34:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:23 smithi122 ceph-mon[93869]: pgmap v18950: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 272 KiB/s wr, 8 op/s 2023-03-10T00:34:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:23 smithi181 ceph-mon[94189]: pgmap v18950: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 272 KiB/s wr, 8 op/s 2023-03-10T00:34:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:23 smithi038 ceph-mon[92422]: pgmap v18950: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 272 KiB/s wr, 8 op/s 2023-03-10T00:34:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35035.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:34:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:24 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34573.936394 secs 2023-03-10T00:34:24.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:24 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34573.936394 secs 2023-03-10T00:34:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:24 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34573.936394 secs 2023-03-10T00:34:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:25 smithi122 ceph-mon[93869]: pgmap v18951: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 326 KiB/s rd, 324 KiB/s wr, 8 op/s 2023-03-10T00:34:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:25 smithi181 ceph-mon[94189]: pgmap v18951: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 326 KiB/s rd, 324 KiB/s wr, 8 op/s 2023-03-10T00:34:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:25 smithi038 ceph-mon[92422]: pgmap v18951: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 326 KiB/s rd, 324 KiB/s wr, 8 op/s 2023-03-10T00:34:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34576.473495 secs 2023-03-10T00:34:26.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34576.473495 secs 2023-03-10T00:34:26.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34576.473495 secs 2023-03-10T00:34:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:27 smithi122 ceph-mon[93869]: pgmap v18952: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 85 KiB/s wr, 7 op/s 2023-03-10T00:34:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:27 smithi181 ceph-mon[94189]: pgmap v18952: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 85 KiB/s wr, 7 op/s 2023-03-10T00:34:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:27 smithi038 ceph-mon[92422]: pgmap v18952: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 85 KiB/s wr, 7 op/s 2023-03-10T00:34:28.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35040.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:34:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34578.936453 secs 2023-03-10T00:34:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34578.936453 secs 2023-03-10T00:34:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34578.936453 secs 2023-03-10T00:34:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:29 smithi122 ceph-mon[93869]: pgmap v18953: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 85 KiB/s wr, 6 op/s 2023-03-10T00:34:29.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:29 smithi181 ceph-mon[94189]: pgmap v18953: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 85 KiB/s wr, 6 op/s 2023-03-10T00:34:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:29 smithi038 ceph-mon[92422]: pgmap v18953: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 85 KiB/s wr, 6 op/s 2023-03-10T00:34:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34581.473557 secs 2023-03-10T00:34:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34581.473557 secs 2023-03-10T00:34:30.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34581.473557 secs 2023-03-10T00:34:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:31 smithi122 ceph-mon[93869]: pgmap v18954: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 85 KiB/s wr, 7 op/s 2023-03-10T00:34:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:31 smithi181 ceph-mon[94189]: pgmap v18954: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 85 KiB/s wr, 7 op/s 2023-03-10T00:34:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:31 smithi038 ceph-mon[92422]: pgmap v18954: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 85 KiB/s wr, 7 op/s 2023-03-10T00:34:32.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:32 smithi122 ceph-mon[93869]: pgmap v18955: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 85 KiB/s wr, 8 op/s 2023-03-10T00:34:32.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:32 smithi181 ceph-mon[94189]: pgmap v18955: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 85 KiB/s wr, 8 op/s 2023-03-10T00:34:32.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:32 smithi038 ceph-mon[92422]: pgmap v18955: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 88 KiB/s rd, 85 KiB/s wr, 8 op/s 2023-03-10T00:34:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35045.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:34:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34583.936514 secs 2023-03-10T00:34:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34583.936514 secs 2023-03-10T00:34:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34583.936514 secs 2023-03-10T00:34:34.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:34 smithi122 ceph-mon[93869]: pgmap v18956: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 51 KiB/s wr, 7 op/s 2023-03-10T00:34:34.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:34 smithi181 ceph-mon[94189]: pgmap v18956: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 51 KiB/s wr, 7 op/s 2023-03-10T00:34:34.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:34 smithi038 ceph-mon[92422]: pgmap v18956: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 51 KiB/s wr, 7 op/s 2023-03-10T00:34:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34586.473651 secs 2023-03-10T00:34:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34586.473651 secs 2023-03-10T00:34:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34586.473651 secs 2023-03-10T00:34:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:36 smithi122 ceph-mon[93869]: pgmap v18957: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-10T00:34:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:36 smithi181 ceph-mon[94189]: pgmap v18957: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-10T00:34:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:36 smithi038 ceph-mon[92422]: pgmap v18957: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-10T00:34:38.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35050.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:34:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34588.936574 secs 2023-03-10T00:34:39.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:38 smithi122 ceph-mon[93869]: pgmap v18958: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-10T00:34:39.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34588.936574 secs 2023-03-10T00:34:39.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:38 smithi181 ceph-mon[94189]: pgmap v18958: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-10T00:34:39.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34588.936574 secs 2023-03-10T00:34:39.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:38 smithi038 ceph-mon[92422]: pgmap v18958: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-10T00:34:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:40 smithi122 ceph-mon[93869]: pgmap v18959: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 511 B/s wr, 7 op/s 2023-03-10T00:34:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34591.473774 secs 2023-03-10T00:34:41.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:40 smithi181 ceph-mon[94189]: pgmap v18959: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 511 B/s wr, 7 op/s 2023-03-10T00:34:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34591.473774 secs 2023-03-10T00:34:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:40 smithi038 ceph-mon[92422]: pgmap v18959: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 511 B/s wr, 7 op/s 2023-03-10T00:34:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34591.473774 secs 2023-03-10T00:34:43.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:42 smithi122 ceph-mon[93869]: pgmap v18960: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 682 B/s wr, 8 op/s 2023-03-10T00:34:43.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:42 smithi181 ceph-mon[94189]: pgmap v18960: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 682 B/s wr, 8 op/s 2023-03-10T00:34:43.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:42 smithi038 ceph-mon[92422]: pgmap v18960: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 682 B/s wr, 8 op/s 2023-03-10T00:34:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35055.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:34:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34593.936635 secs 2023-03-10T00:34:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34593.936635 secs 2023-03-10T00:34:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34593.936635 secs 2023-03-10T00:34:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:45 smithi122 ceph-mon[93869]: pgmap v18961: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 682 B/s wr, 7 op/s 2023-03-10T00:34:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:45 smithi181 ceph-mon[94189]: pgmap v18961: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 682 B/s wr, 7 op/s 2023-03-10T00:34:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:45 smithi038 ceph-mon[92422]: pgmap v18961: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 682 B/s wr, 7 op/s 2023-03-10T00:34:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34596.473833 secs 2023-03-10T00:34:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34596.473833 secs 2023-03-10T00:34:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34596.473833 secs 2023-03-10T00:34:47.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:47 smithi122 ceph-mon[93869]: pgmap v18962: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 767 B/s wr, 8 op/s 2023-03-10T00:34:47.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:34:47.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:34:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:47 smithi181 ceph-mon[94189]: pgmap v18962: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 767 B/s wr, 8 op/s 2023-03-10T00:34:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:34:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:34:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:47 smithi038 ceph-mon[92422]: pgmap v18962: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 767 B/s wr, 8 op/s 2023-03-10T00:34:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:34:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:34:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35060.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:34:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34598.936720 secs 2023-03-10T00:34:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34598.936720 secs 2023-03-10T00:34:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34598.936720 secs 2023-03-10T00:34:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:49 smithi122 ceph-mon[93869]: pgmap v18963: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 164 KiB/s wr, 7 op/s 2023-03-10T00:34:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:34:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:34:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:49 smithi181 ceph-mon[94189]: pgmap v18963: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 164 KiB/s wr, 7 op/s 2023-03-10T00:34:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:34:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:34:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:49 smithi038 ceph-mon[92422]: pgmap v18963: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 164 KiB/s wr, 7 op/s 2023-03-10T00:34:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:34:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:34:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34601.473888 secs 2023-03-10T00:34:50.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34601.473888 secs 2023-03-10T00:34:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34601.473888 secs 2023-03-10T00:34:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:51 smithi122 ceph-mon[93869]: pgmap v18964: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 192 KiB/s wr, 7 op/s 2023-03-10T00:34:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:51 smithi181 ceph-mon[94189]: pgmap v18964: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 192 KiB/s wr, 7 op/s 2023-03-10T00:34:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:51 smithi038 ceph-mon[92422]: pgmap v18964: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 192 KiB/s wr, 7 op/s 2023-03-10T00:34:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35065.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:34:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:53 smithi122 ceph-mon[93869]: pgmap v18965: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 483 KiB/s rd, 478 KiB/s wr, 10 op/s 2023-03-10T00:34:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:53 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:34:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:53 smithi181 ceph-mon[94189]: pgmap v18965: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 483 KiB/s rd, 478 KiB/s wr, 10 op/s 2023-03-10T00:34:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:53 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:34:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:53 smithi038 ceph-mon[92422]: pgmap v18965: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 483 KiB/s rd, 478 KiB/s wr, 10 op/s 2023-03-10T00:34:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:53 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:34:54.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:54 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:34:54.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:54 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:34:54.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:54 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:34:54.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:54 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:34:54.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:54 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:34:54.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:54 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:34:54.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:54 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:34:54.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:54 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:34:54.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:54 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:34:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:55 smithi122 ceph-mon[93869]: pgmap v18966: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 506 KiB/s wr, 9 op/s 2023-03-10T00:34:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:34:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:34:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34606.473957 secs 2023-03-10T00:34:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:55 smithi181 ceph-mon[94189]: pgmap v18966: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 506 KiB/s wr, 9 op/s 2023-03-10T00:34:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:34:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:34:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34606.473957 secs 2023-03-10T00:34:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:55 smithi038 ceph-mon[92422]: pgmap v18966: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 506 KiB/s wr, 9 op/s 2023-03-10T00:34:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:34:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:34:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34606.473957 secs 2023-03-10T00:34:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:56 smithi122 ceph-mon[93869]: pgmap v18967: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 870 KiB/s rd, 856 KiB/s wr, 13 op/s 2023-03-10T00:34:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:56 smithi181 ceph-mon[94189]: pgmap v18967: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 870 KiB/s rd, 856 KiB/s wr, 13 op/s 2023-03-10T00:34:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:56 smithi038 ceph-mon[92422]: pgmap v18967: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 870 KiB/s rd, 856 KiB/s wr, 13 op/s 2023-03-10T00:34:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:34:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:34:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:34:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:34:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:34:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:34:58.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35070.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:34:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:59 smithi122 ceph-mon[93869]: pgmap v18968: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 869 KiB/s rd, 856 KiB/s wr, 12 op/s 2023-03-10T00:34:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:34:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:34:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:34:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:59 smithi181 ceph-mon[94189]: pgmap v18968: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 869 KiB/s rd, 856 KiB/s wr, 12 op/s 2023-03-10T00:34:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:34:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:34:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:34:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:59 smithi038 ceph-mon[92422]: pgmap v18968: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 869 KiB/s rd, 856 KiB/s wr, 12 op/s 2023-03-10T00:34:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:34:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:34:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:35:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:01 smithi122 ceph-mon[93869]: pgmap v18969: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 738 KiB/s rd, 725 KiB/s wr, 12 op/s 2023-03-10T00:35:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34611.473999 secs 2023-03-10T00:35:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:01 smithi181 ceph-mon[94189]: pgmap v18969: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 738 KiB/s rd, 725 KiB/s wr, 12 op/s 2023-03-10T00:35:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34611.473999 secs 2023-03-10T00:35:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:01 smithi038 ceph-mon[92422]: pgmap v18969: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 738 KiB/s rd, 725 KiB/s wr, 12 op/s 2023-03-10T00:35:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34611.473999 secs 2023-03-10T00:35:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35075.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:35:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:03 smithi122 ceph-mon[93869]: pgmap v18970: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 867 KiB/s rd, 864 KiB/s wr, 16 op/s 2023-03-10T00:35:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34613.937002 secs 2023-03-10T00:35:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:03 smithi181 ceph-mon[94189]: pgmap v18970: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 867 KiB/s rd, 864 KiB/s wr, 16 op/s 2023-03-10T00:35:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34613.937002 secs 2023-03-10T00:35:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:03 smithi038 ceph-mon[92422]: pgmap v18970: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 867 KiB/s rd, 864 KiB/s wr, 16 op/s 2023-03-10T00:35:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34613.937002 secs 2023-03-10T00:35:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:05 smithi122 ceph-mon[93869]: pgmap v18971: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 614 KiB/s rd, 777 KiB/s wr, 25 op/s 2023-03-10T00:35:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:05 smithi122 ceph-mon[93869]: osdmap e1401: 12 total, 12 up, 12 in 2023-03-10T00:35:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34616.474064 secs 2023-03-10T00:35:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:05 smithi181 ceph-mon[94189]: pgmap v18971: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 614 KiB/s rd, 777 KiB/s wr, 25 op/s 2023-03-10T00:35:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:05 smithi181 ceph-mon[94189]: osdmap e1401: 12 total, 12 up, 12 in 2023-03-10T00:35:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34616.474064 secs 2023-03-10T00:35:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:05 smithi038 ceph-mon[92422]: pgmap v18971: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 614 KiB/s rd, 777 KiB/s wr, 25 op/s 2023-03-10T00:35:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:05 smithi038 ceph-mon[92422]: osdmap e1401: 12 total, 12 up, 12 in 2023-03-10T00:35:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34616.474064 secs 2023-03-10T00:35:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:06 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:35:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:06 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:35:06.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:35:06.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:06 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:35:06.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:06 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:35:06.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:35:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:06 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:35:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:06 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:35:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:35:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:07 smithi122 ceph-mon[93869]: pgmap v18973: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 539 KiB/s wr, 29 op/s 2023-03-10T00:35:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:07 smithi181 ceph-mon[94189]: pgmap v18973: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 539 KiB/s wr, 29 op/s 2023-03-10T00:35:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:07 smithi038 ceph-mon[92422]: pgmap v18973: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 539 KiB/s wr, 29 op/s 2023-03-10T00:35:08.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35080.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:35:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34618.937059 secs 2023-03-10T00:35:09.019 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34618.937059 secs 2023-03-10T00:35:09.020 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34618.937059 secs 2023-03-10T00:35:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:09 smithi122 ceph-mon[93869]: pgmap v18974: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 539 KiB/s wr, 28 op/s 2023-03-10T00:35:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:09 smithi181 ceph-mon[94189]: pgmap v18974: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 539 KiB/s wr, 28 op/s 2023-03-10T00:35:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:09 smithi038 ceph-mon[92422]: pgmap v18974: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 539 KiB/s wr, 28 op/s 2023-03-10T00:35:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:10 smithi122 ceph-mon[93869]: pgmap v18975: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 501 KiB/s wr, 28 op/s 2023-03-10T00:35:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34621.474120 secs 2023-03-10T00:35:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:10 smithi181 ceph-mon[94189]: pgmap v18975: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 501 KiB/s wr, 28 op/s 2023-03-10T00:35:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34621.474120 secs 2023-03-10T00:35:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:10 smithi038 ceph-mon[92422]: pgmap v18975: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 501 KiB/s wr, 28 op/s 2023-03-10T00:35:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34621.474120 secs 2023-03-10T00:35:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:35:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:35:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:12 smithi122 ceph-mon[93869]: pgmap v18976: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 300 KiB/s wr, 25 op/s 2023-03-10T00:35:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:35:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:35:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:12 smithi181 ceph-mon[94189]: pgmap v18976: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 300 KiB/s wr, 25 op/s 2023-03-10T00:35:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:35:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:35:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:12 smithi038 ceph-mon[92422]: pgmap v18976: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 300 KiB/s wr, 25 op/s 2023-03-10T00:35:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35085.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:35:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34623.937154 secs 2023-03-10T00:35:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:13 smithi122 ceph-mon[93869]: osdmap e1402: 12 total, 12 up, 12 in 2023-03-10T00:35:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34623.937154 secs 2023-03-10T00:35:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:13 smithi181 ceph-mon[94189]: osdmap e1402: 12 total, 12 up, 12 in 2023-03-10T00:35:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34623.937154 secs 2023-03-10T00:35:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:13 smithi038 ceph-mon[92422]: osdmap e1402: 12 total, 12 up, 12 in 2023-03-10T00:35:15.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:14 smithi122 ceph-mon[93869]: pgmap v18978: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 264 KiB/s wr, 12 op/s 2023-03-10T00:35:15.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:14 smithi181 ceph-mon[94189]: pgmap v18978: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 264 KiB/s wr, 12 op/s 2023-03-10T00:35:15.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:14 smithi038 ceph-mon[92422]: pgmap v18978: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 264 KiB/s wr, 12 op/s 2023-03-10T00:35:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:15 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:35:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:15 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:35:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34626.474180 secs 2023-03-10T00:35:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:15 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:35:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:15 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:35:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34626.474180 secs 2023-03-10T00:35:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:15 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:35:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:15 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:35:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34626.474180 secs 2023-03-10T00:35:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:16 smithi122 ceph-mon[93869]: pgmap v18979: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 185 KiB/s wr, 9 op/s 2023-03-10T00:35:17.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:16 smithi181 ceph-mon[94189]: pgmap v18979: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 185 KiB/s wr, 9 op/s 2023-03-10T00:35:17.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:16 smithi038 ceph-mon[92422]: pgmap v18979: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 185 KiB/s wr, 9 op/s 2023-03-10T00:35:18.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35090.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:35:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34628.937241 secs 2023-03-10T00:35:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:18 smithi122 ceph-mon[93869]: pgmap v18980: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 185 KiB/s wr, 8 op/s 2023-03-10T00:35:19.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34628.937241 secs 2023-03-10T00:35:19.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:18 smithi181 ceph-mon[94189]: pgmap v18980: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 185 KiB/s wr, 8 op/s 2023-03-10T00:35:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34628.937241 secs 2023-03-10T00:35:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:18 smithi038 ceph-mon[92422]: pgmap v18980: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 185 KiB/s wr, 8 op/s 2023-03-10T00:35:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:20 smithi181 ceph-mon[94189]: pgmap v18981: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 184 KiB/s wr, 8 op/s 2023-03-10T00:35:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34631.474242 secs 2023-03-10T00:35:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:35:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:20 smithi038 ceph-mon[92422]: pgmap v18981: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 184 KiB/s wr, 8 op/s 2023-03-10T00:35:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34631.474242 secs 2023-03-10T00:35:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:35:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:20 smithi122 ceph-mon[93869]: pgmap v18981: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 184 KiB/s wr, 8 op/s 2023-03-10T00:35:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34631.474242 secs 2023-03-10T00:35:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:35:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:22 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:35:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:22 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:35:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:22 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:35:22.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:22 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:35:22.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:22 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:35:22.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:22 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:35:22.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:22 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:35:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:22 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:35:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:22 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:35:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:22 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:35:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:22 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:35:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:22 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:35:22.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:22 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:35:22.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:22 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:35:22.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:22 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:35:22.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:22 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:35:22.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:22 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:35:22.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:22 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:35:22.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:22 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:35:22.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:22 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:35:22.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:22 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:35:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:23 smithi122 ceph-mon[93869]: pgmap v18982: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 184 KiB/s wr, 7 op/s 2023-03-10T00:35:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:23 smithi181 ceph-mon[94189]: pgmap v18982: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 184 KiB/s wr, 7 op/s 2023-03-10T00:35:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:23 smithi038 ceph-mon[92422]: pgmap v18982: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 184 KiB/s wr, 7 op/s 2023-03-10T00:35:23.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35095.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:35:24.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:24 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34633.937293 secs 2023-03-10T00:35:24.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:24 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34633.937293 secs 2023-03-10T00:35:24.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:24 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34633.937293 secs 2023-03-10T00:35:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:25 smithi122 ceph-mon[93869]: pgmap v18983: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 126 KiB/s wr, 7 op/s 2023-03-10T00:35:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:25 smithi181 ceph-mon[94189]: pgmap v18983: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 126 KiB/s wr, 7 op/s 2023-03-10T00:35:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:25 smithi038 ceph-mon[92422]: pgmap v18983: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 126 KiB/s wr, 7 op/s 2023-03-10T00:35:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34636.474351 secs 2023-03-10T00:35:26.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34636.474351 secs 2023-03-10T00:35:26.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34636.474351 secs 2023-03-10T00:35:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:27 smithi122 ceph-mon[93869]: pgmap v18984: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 108 KiB/s wr, 8 op/s 2023-03-10T00:35:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:27 smithi181 ceph-mon[94189]: pgmap v18984: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 108 KiB/s wr, 8 op/s 2023-03-10T00:35:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:27 smithi038 ceph-mon[92422]: pgmap v18984: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 108 KiB/s wr, 8 op/s 2023-03-10T00:35:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35100.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:35:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34638.937388 secs 2023-03-10T00:35:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34638.937388 secs 2023-03-10T00:35:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34638.937388 secs 2023-03-10T00:35:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:29 smithi122 ceph-mon[93869]: pgmap v18985: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 108 KiB/s wr, 5 op/s 2023-03-10T00:35:29.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:29 smithi181 ceph-mon[94189]: pgmap v18985: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 108 KiB/s wr, 5 op/s 2023-03-10T00:35:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:29 smithi038 ceph-mon[92422]: pgmap v18985: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 108 KiB/s wr, 5 op/s 2023-03-10T00:35:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34641.474450 secs 2023-03-10T00:35:30.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34641.474450 secs 2023-03-10T00:35:30.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34641.474450 secs 2023-03-10T00:35:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:31 smithi122 ceph-mon[93869]: pgmap v18986: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 108 KiB/s wr, 7 op/s 2023-03-10T00:35:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:31 smithi181 ceph-mon[94189]: pgmap v18986: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 108 KiB/s wr, 7 op/s 2023-03-10T00:35:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:31 smithi038 ceph-mon[92422]: pgmap v18986: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 108 KiB/s wr, 7 op/s 2023-03-10T00:35:32.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:32 smithi122 ceph-mon[93869]: pgmap v18987: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 108 KiB/s wr, 8 op/s 2023-03-10T00:35:32.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:32 smithi181 ceph-mon[94189]: pgmap v18987: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 108 KiB/s wr, 8 op/s 2023-03-10T00:35:32.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:32 smithi038 ceph-mon[92422]: pgmap v18987: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 108 KiB/s wr, 8 op/s 2023-03-10T00:35:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35105.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:35:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34643.937503 secs 2023-03-10T00:35:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34643.937503 secs 2023-03-10T00:35:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34643.937503 secs 2023-03-10T00:35:34.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:34 smithi122 ceph-mon[93869]: pgmap v18988: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 107 KiB/s wr, 7 op/s 2023-03-10T00:35:34.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:34 smithi181 ceph-mon[94189]: pgmap v18988: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 107 KiB/s wr, 7 op/s 2023-03-10T00:35:34.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:34 smithi038 ceph-mon[92422]: pgmap v18988: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 107 KiB/s wr, 7 op/s 2023-03-10T00:35:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34646.474510 secs 2023-03-10T00:35:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34646.474510 secs 2023-03-10T00:35:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34646.474510 secs 2023-03-10T00:35:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:36 smithi122 ceph-mon[93869]: pgmap v18989: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-10T00:35:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:36 smithi181 ceph-mon[94189]: pgmap v18989: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-10T00:35:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:36 smithi038 ceph-mon[92422]: pgmap v18989: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-10T00:35:37.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:35:37.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:35:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:35:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:35:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:35:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:35:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35110.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:35:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34648.937562 secs 2023-03-10T00:35:38.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:38 smithi122 ceph-mon[93869]: pgmap v18990: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-10T00:35:38.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34648.937562 secs 2023-03-10T00:35:38.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:38 smithi181 ceph-mon[94189]: pgmap v18990: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-10T00:35:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34648.937562 secs 2023-03-10T00:35:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:38 smithi038 ceph-mon[92422]: pgmap v18990: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-10T00:35:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:39 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:35:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:39 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:35:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:39 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:35:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:39 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:35:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:39 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:35:40.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:39 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:35:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:40 smithi038 ceph-mon[92422]: pgmap v18991: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 131 KiB/s wr, 7 op/s 2023-03-10T00:35:41.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34651.474574 secs 2023-03-10T00:35:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:40 smithi122 ceph-mon[93869]: pgmap v18991: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 131 KiB/s wr, 7 op/s 2023-03-10T00:35:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34651.474574 secs 2023-03-10T00:35:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:40 smithi181 ceph-mon[94189]: pgmap v18991: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 131 KiB/s wr, 7 op/s 2023-03-10T00:35:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34651.474574 secs 2023-03-10T00:35:43.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:42 smithi122 ceph-mon[93869]: pgmap v18992: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 131 KiB/s wr, 8 op/s 2023-03-10T00:35:43.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:42 smithi181 ceph-mon[94189]: pgmap v18992: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 131 KiB/s wr, 8 op/s 2023-03-10T00:35:43.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:42 smithi038 ceph-mon[92422]: pgmap v18992: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 131 KiB/s wr, 8 op/s 2023-03-10T00:35:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35115.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:35:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:45 smithi122 ceph-mon[93869]: pgmap v18993: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 292 KiB/s wr, 8 op/s 2023-03-10T00:35:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:45 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:35:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:45 smithi181 ceph-mon[94189]: pgmap v18993: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 292 KiB/s wr, 8 op/s 2023-03-10T00:35:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:45 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:35:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:45 smithi038 ceph-mon[92422]: pgmap v18993: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 292 KiB/s wr, 8 op/s 2023-03-10T00:35:46.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:45 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:35:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34656.474658 secs 2023-03-10T00:35:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:46 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:35:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:46 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:35:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:46 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:35:46.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34656.474658 secs 2023-03-10T00:35:46.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:46 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:35:46.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:46 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:35:46.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:46 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:35:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34656.474658 secs 2023-03-10T00:35:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:46 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:35:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:46 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:35:47.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:46 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:35:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:47 smithi122 ceph-mon[93869]: pgmap v18994: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 292 KiB/s wr, 9 op/s 2023-03-10T00:35:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:47 smithi181 ceph-mon[94189]: pgmap v18994: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 292 KiB/s wr, 9 op/s 2023-03-10T00:35:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:47 smithi038 ceph-mon[92422]: pgmap v18994: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 292 KiB/s wr, 9 op/s 2023-03-10T00:35:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35120.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:35:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:49 smithi122 ceph-mon[93869]: pgmap v18995: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 292 KiB/s wr, 7 op/s 2023-03-10T00:35:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:49 smithi181 ceph-mon[94189]: pgmap v18995: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 292 KiB/s wr, 7 op/s 2023-03-10T00:35:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:49 smithi038 ceph-mon[92422]: pgmap v18995: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 292 KiB/s wr, 7 op/s 2023-03-10T00:35:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34661.474730 secs 2023-03-10T00:35:50.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34661.474730 secs 2023-03-10T00:35:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34661.474730 secs 2023-03-10T00:35:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:51 smithi122 ceph-mon[93869]: pgmap v18996: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 326 KiB/s wr, 9 op/s 2023-03-10T00:35:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:51 smithi181 ceph-mon[94189]: pgmap v18996: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 326 KiB/s wr, 9 op/s 2023-03-10T00:35:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:51 smithi038 ceph-mon[92422]: pgmap v18996: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 326 KiB/s wr, 9 op/s 2023-03-10T00:35:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:52 smithi122 ceph-mon[93869]: pgmap v18997: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 196 KiB/s wr, 10 op/s 2023-03-10T00:35:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:52 smithi181 ceph-mon[94189]: pgmap v18997: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 196 KiB/s wr, 10 op/s 2023-03-10T00:35:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:52 smithi038 ceph-mon[92422]: pgmap v18997: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 196 KiB/s wr, 10 op/s 2023-03-10T00:35:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35125.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:35:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34663.937786 secs 2023-03-10T00:35:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34663.937786 secs 2023-03-10T00:35:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34663.937786 secs 2023-03-10T00:35:54.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:54 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:35:54.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:54 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:35:54.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:54 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:35:54.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:54 smithi122 ceph-mon[93869]: pgmap v18998: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 584 KiB/s rd, 566 KiB/s wr, 11 op/s 2023-03-10T00:35:54.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:54 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:35:54.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:54 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:35:54.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:54 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:35:54.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:54 smithi181 ceph-mon[94189]: pgmap v18998: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 584 KiB/s rd, 566 KiB/s wr, 11 op/s 2023-03-10T00:35:54.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:54 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:35:54.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:54 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:35:54.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:54 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:35:55.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:54 smithi038 ceph-mon[92422]: pgmap v18998: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 584 KiB/s rd, 566 KiB/s wr, 11 op/s 2023-03-10T00:35:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:35:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:35:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34666.474786 secs 2023-03-10T00:35:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:35:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:35:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34666.474786 secs 2023-03-10T00:35:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:35:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:35:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34666.474786 secs 2023-03-10T00:35:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:56 smithi122 ceph-mon[93869]: pgmap v18999: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 749 KiB/s rd, 558 KiB/s wr, 12 op/s 2023-03-10T00:35:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:56 smithi181 ceph-mon[94189]: pgmap v18999: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 749 KiB/s rd, 558 KiB/s wr, 12 op/s 2023-03-10T00:35:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:56 smithi038 ceph-mon[92422]: pgmap v18999: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 749 KiB/s rd, 558 KiB/s wr, 12 op/s 2023-03-10T00:35:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:35:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:35:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:35:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:35:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:35:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:35:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35130.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:35:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34668.937843 secs 2023-03-10T00:35:58.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:35:58 smithi122 ceph-mon[93869]: pgmap v19000: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 748 KiB/s rd, 559 KiB/s wr, 10 op/s 2023-03-10T00:35:58.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34668.937843 secs 2023-03-10T00:35:58.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:35:58 smithi181 ceph-mon[94189]: pgmap v19000: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 748 KiB/s rd, 559 KiB/s wr, 10 op/s 2023-03-10T00:35:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34668.937843 secs 2023-03-10T00:35:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:35:58 smithi038 ceph-mon[92422]: pgmap v19000: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 748 KiB/s rd, 559 KiB/s wr, 10 op/s 2023-03-10T00:36:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:00 smithi038 ceph-mon[92422]: pgmap v19001: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 918 KiB/s rd, 729 KiB/s wr, 13 op/s 2023-03-10T00:36:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34671.474848 secs 2023-03-10T00:36:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:00 smithi122 ceph-mon[93869]: pgmap v19001: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 918 KiB/s rd, 729 KiB/s wr, 13 op/s 2023-03-10T00:36:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34671.474848 secs 2023-03-10T00:36:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:00 smithi181 ceph-mon[94189]: pgmap v19001: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 918 KiB/s rd, 729 KiB/s wr, 13 op/s 2023-03-10T00:36:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34671.474848 secs 2023-03-10T00:36:03.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:02 smithi038 ceph-mon[92422]: pgmap v19002: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 695 KiB/s wr, 13 op/s 2023-03-10T00:36:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:02 smithi122 ceph-mon[93869]: pgmap v19002: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 695 KiB/s wr, 13 op/s 2023-03-10T00:36:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:02 smithi181 ceph-mon[94189]: pgmap v19002: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 695 KiB/s wr, 13 op/s 2023-03-10T00:36:03.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35135.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:36:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34673.937979 secs 2023-03-10T00:36:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34673.937979 secs 2023-03-10T00:36:04.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34673.937979 secs 2023-03-10T00:36:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:05 smithi122 ceph-mon[93869]: pgmap v19003: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 751 KiB/s rd, 762 KiB/s wr, 24 op/s 2023-03-10T00:36:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:05 smithi122 ceph-mon[93869]: osdmap e1403: 12 total, 12 up, 12 in 2023-03-10T00:36:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34676.474911 secs 2023-03-10T00:36:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:05 smithi181 ceph-mon[94189]: pgmap v19003: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 751 KiB/s rd, 762 KiB/s wr, 24 op/s 2023-03-10T00:36:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:05 smithi181 ceph-mon[94189]: osdmap e1403: 12 total, 12 up, 12 in 2023-03-10T00:36:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34676.474911 secs 2023-03-10T00:36:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:05 smithi038 ceph-mon[92422]: pgmap v19003: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 751 KiB/s rd, 762 KiB/s wr, 24 op/s 2023-03-10T00:36:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:05 smithi038 ceph-mon[92422]: osdmap e1403: 12 total, 12 up, 12 in 2023-03-10T00:36:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34676.474911 secs 2023-03-10T00:36:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:07 smithi122 ceph-mon[93869]: pgmap v19005: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 714 KiB/s rd, 777 KiB/s wr, 30 op/s 2023-03-10T00:36:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:36:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:36:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:07 smithi181 ceph-mon[94189]: pgmap v19005: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 714 KiB/s rd, 777 KiB/s wr, 30 op/s 2023-03-10T00:36:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:36:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:36:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:07 smithi038 ceph-mon[92422]: pgmap v19005: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 714 KiB/s rd, 777 KiB/s wr, 30 op/s 2023-03-10T00:36:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:36:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:36:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35140.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:36:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34678.938062 secs 2023-03-10T00:36:08.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34678.938062 secs 2023-03-10T00:36:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34678.938062 secs 2023-03-10T00:36:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:09 smithi122 ceph-mon[93869]: pgmap v19006: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 722 KiB/s rd, 784 KiB/s wr, 31 op/s 2023-03-10T00:36:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:36:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:36:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:09 smithi181 ceph-mon[94189]: pgmap v19006: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 722 KiB/s rd, 784 KiB/s wr, 31 op/s 2023-03-10T00:36:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:36:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:36:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:09 smithi038 ceph-mon[92422]: pgmap v19006: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 722 KiB/s rd, 784 KiB/s wr, 31 op/s 2023-03-10T00:36:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:36:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:36:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:10 smithi122 ceph-mon[93869]: pgmap v19007: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 539 KiB/s rd, 580 KiB/s wr, 29 op/s 2023-03-10T00:36:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34681.474973 secs 2023-03-10T00:36:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:10 smithi181 ceph-mon[94189]: pgmap v19007: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 539 KiB/s rd, 580 KiB/s wr, 29 op/s 2023-03-10T00:36:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34681.474973 secs 2023-03-10T00:36:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:10 smithi038 ceph-mon[92422]: pgmap v19007: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 539 KiB/s rd, 580 KiB/s wr, 29 op/s 2023-03-10T00:36:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34681.474973 secs 2023-03-10T00:36:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:12 smithi038 ceph-mon[92422]: pgmap v19008: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 540 KiB/s rd, 580 KiB/s wr, 29 op/s 2023-03-10T00:36:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:12 smithi122 ceph-mon[93869]: pgmap v19008: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 540 KiB/s rd, 580 KiB/s wr, 29 op/s 2023-03-10T00:36:13.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:12 smithi181 ceph-mon[94189]: pgmap v19008: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 540 KiB/s rd, 580 KiB/s wr, 29 op/s 2023-03-10T00:36:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35145.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:36:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34683.938118 secs 2023-03-10T00:36:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:13 smithi038 ceph-mon[92422]: osdmap e1404: 12 total, 12 up, 12 in 2023-03-10T00:36:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34683.938118 secs 2023-03-10T00:36:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:13 smithi122 ceph-mon[93869]: osdmap e1404: 12 total, 12 up, 12 in 2023-03-10T00:36:14.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34683.938118 secs 2023-03-10T00:36:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:13 smithi181 ceph-mon[94189]: osdmap e1404: 12 total, 12 up, 12 in 2023-03-10T00:36:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:15 smithi122 ceph-mon[93869]: pgmap v19010: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 292 KiB/s wr, 13 op/s 2023-03-10T00:36:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:15 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:36:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:15 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:36:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34686.474984 secs 2023-03-10T00:36:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:15 smithi181 ceph-mon[94189]: pgmap v19010: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 292 KiB/s wr, 13 op/s 2023-03-10T00:36:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:15 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:36:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:15 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:36:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34686.474984 secs 2023-03-10T00:36:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:15 smithi038 ceph-mon[92422]: pgmap v19010: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 292 KiB/s wr, 13 op/s 2023-03-10T00:36:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:15 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:36:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:15 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:36:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34686.474984 secs 2023-03-10T00:36:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:17 smithi122 ceph-mon[93869]: pgmap v19011: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 242 KiB/s wr, 10 op/s 2023-03-10T00:36:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:36:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:36:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:17 smithi181 ceph-mon[94189]: pgmap v19011: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 242 KiB/s wr, 10 op/s 2023-03-10T00:36:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:36:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:36:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:17 smithi038 ceph-mon[92422]: pgmap v19011: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 242 KiB/s wr, 10 op/s 2023-03-10T00:36:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:36:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:36:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35150.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:36:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34688.938181 secs 2023-03-10T00:36:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34688.938181 secs 2023-03-10T00:36:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34688.938181 secs 2023-03-10T00:36:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:19 smithi122 ceph-mon[93869]: pgmap v19012: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 233 KiB/s wr, 9 op/s 2023-03-10T00:36:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:19 smithi181 ceph-mon[94189]: pgmap v19012: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 233 KiB/s wr, 9 op/s 2023-03-10T00:36:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:19 smithi038 ceph-mon[92422]: pgmap v19012: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 233 KiB/s wr, 9 op/s 2023-03-10T00:36:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34691.475042 secs 2023-03-10T00:36:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34691.475042 secs 2023-03-10T00:36:21.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34691.475042 secs 2023-03-10T00:36:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:21 smithi122 ceph-mon[93869]: pgmap v19013: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 233 KiB/s wr, 9 op/s 2023-03-10T00:36:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:21 smithi181 ceph-mon[94189]: pgmap v19013: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 233 KiB/s wr, 9 op/s 2023-03-10T00:36:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:21 smithi038 ceph-mon[92422]: pgmap v19013: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 233 KiB/s wr, 9 op/s 2023-03-10T00:36:22.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:22 smithi122 ceph-mon[93869]: pgmap v19014: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 232 KiB/s wr, 9 op/s 2023-03-10T00:36:22.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:22 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:36:22.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:22 smithi181 ceph-mon[94189]: pgmap v19014: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 232 KiB/s wr, 9 op/s 2023-03-10T00:36:22.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:22 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:36:22.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:22 smithi038 ceph-mon[92422]: pgmap v19014: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 232 KiB/s wr, 9 op/s 2023-03-10T00:36:22.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:22 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:36:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35155.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:36:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34693.938244 secs 2023-03-10T00:36:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:23 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:36:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:23 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:36:23.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:23 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:36:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34693.938244 secs 2023-03-10T00:36:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:23 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:36:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:23 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:36:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:23 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:36:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34693.938244 secs 2023-03-10T00:36:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:23 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:36:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:23 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:36:24.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:23 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:36:24.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:24 smithi122 ceph-mon[93869]: pgmap v19015: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 107 KiB/s wr, 8 op/s 2023-03-10T00:36:24.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:24 smithi181 ceph-mon[94189]: pgmap v19015: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 107 KiB/s wr, 8 op/s 2023-03-10T00:36:24.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:24 smithi038 ceph-mon[92422]: pgmap v19015: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 107 KiB/s wr, 8 op/s 2023-03-10T00:36:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34696.475102 secs 2023-03-10T00:36:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34696.475102 secs 2023-03-10T00:36:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34696.475102 secs 2023-03-10T00:36:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:26 smithi122 ceph-mon[93869]: pgmap v19016: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 KiB/s wr, 9 op/s 2023-03-10T00:36:26.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:26 smithi181 ceph-mon[94189]: pgmap v19016: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 KiB/s wr, 9 op/s 2023-03-10T00:36:26.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:26 smithi038 ceph-mon[92422]: pgmap v19016: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 KiB/s wr, 9 op/s 2023-03-10T00:36:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35160.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:36:29.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34698.938338 secs 2023-03-10T00:36:29.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:28 smithi038 ceph-mon[92422]: pgmap v19017: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 85 KiB/s wr, 6 op/s 2023-03-10T00:36:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34698.938338 secs 2023-03-10T00:36:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:28 smithi122 ceph-mon[93869]: pgmap v19017: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 85 KiB/s wr, 6 op/s 2023-03-10T00:36:29.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34698.938338 secs 2023-03-10T00:36:29.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:28 smithi181 ceph-mon[94189]: pgmap v19017: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 85 KiB/s wr, 6 op/s 2023-03-10T00:36:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:30 smithi038 ceph-mon[92422]: pgmap v19018: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 85 KiB/s wr, 6 op/s 2023-03-10T00:36:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34701.475163 secs 2023-03-10T00:36:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:30 smithi122 ceph-mon[93869]: pgmap v19018: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 85 KiB/s wr, 6 op/s 2023-03-10T00:36:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34701.475163 secs 2023-03-10T00:36:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:30 smithi181 ceph-mon[94189]: pgmap v19018: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 85 KiB/s wr, 6 op/s 2023-03-10T00:36:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34701.475163 secs 2023-03-10T00:36:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:33 smithi122 ceph-mon[93869]: pgmap v19019: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 85 KiB/s wr, 7 op/s 2023-03-10T00:36:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:33 smithi181 ceph-mon[94189]: pgmap v19019: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 85 KiB/s wr, 7 op/s 2023-03-10T00:36:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:33 smithi038 ceph-mon[92422]: pgmap v19019: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 85 KiB/s wr, 7 op/s 2023-03-10T00:36:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35165.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:36:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:34 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34703.938429 secs 2023-03-10T00:36:34.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:34 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34703.938429 secs 2023-03-10T00:36:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:34 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34703.938429 secs 2023-03-10T00:36:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:35 smithi122 ceph-mon[93869]: pgmap v19020: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 85 KiB/s wr, 5 op/s 2023-03-10T00:36:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34706.475223 secs 2023-03-10T00:36:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:35 smithi181 ceph-mon[94189]: pgmap v19020: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 85 KiB/s wr, 5 op/s 2023-03-10T00:36:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34706.475223 secs 2023-03-10T00:36:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:35 smithi038 ceph-mon[92422]: pgmap v19020: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 85 KiB/s wr, 5 op/s 2023-03-10T00:36:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34706.475223 secs 2023-03-10T00:36:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:37 smithi122 ceph-mon[93869]: pgmap v19021: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-10T00:36:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:37 smithi181 ceph-mon[94189]: pgmap v19021: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-10T00:36:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:37 smithi038 ceph-mon[92422]: pgmap v19021: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-10T00:36:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35170.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:36:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34708.938504 secs 2023-03-10T00:36:38.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34708.938504 secs 2023-03-10T00:36:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34708.938504 secs 2023-03-10T00:36:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:39 smithi122 ceph-mon[93869]: pgmap v19022: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-10T00:36:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:39 smithi181 ceph-mon[94189]: pgmap v19022: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-10T00:36:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:39 smithi038 ceph-mon[92422]: pgmap v19022: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-10T00:36:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34711.475285 secs 2023-03-10T00:36:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34711.475285 secs 2023-03-10T00:36:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34711.475285 secs 2023-03-10T00:36:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:41 smithi122 ceph-mon[93869]: pgmap v19023: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-10T00:36:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:41 smithi181 ceph-mon[94189]: pgmap v19023: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-10T00:36:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:41 smithi038 ceph-mon[92422]: pgmap v19023: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-10T00:36:42.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:42 smithi122 ceph-mon[93869]: pgmap v19024: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 7 op/s 2023-03-10T00:36:42.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:42 smithi181 ceph-mon[94189]: pgmap v19024: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 7 op/s 2023-03-10T00:36:42.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:42 smithi038 ceph-mon[92422]: pgmap v19024: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 682 B/s wr, 7 op/s 2023-03-10T00:36:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35175.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:36:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34713.938565 secs 2023-03-10T00:36:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34713.938565 secs 2023-03-10T00:36:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34713.938565 secs 2023-03-10T00:36:44.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:44 smithi122 ceph-mon[93869]: pgmap v19025: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 70 KiB/s wr, 6 op/s 2023-03-10T00:36:44.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:44 smithi181 ceph-mon[94189]: pgmap v19025: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 70 KiB/s wr, 6 op/s 2023-03-10T00:36:44.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:44 smithi038 ceph-mon[92422]: pgmap v19025: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 70 KiB/s wr, 6 op/s 2023-03-10T00:36:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34716.475391 secs 2023-03-10T00:36:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34716.475391 secs 2023-03-10T00:36:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34716.475391 secs 2023-03-10T00:36:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:46 smithi122 ceph-mon[93869]: pgmap v19026: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 164 KiB/s wr, 7 op/s 2023-03-10T00:36:46.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:46 smithi181 ceph-mon[94189]: pgmap v19026: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 164 KiB/s wr, 7 op/s 2023-03-10T00:36:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:46 smithi038 ceph-mon[92422]: pgmap v19026: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 164 KiB/s wr, 7 op/s 2023-03-10T00:36:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:36:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:36:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:36:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:36:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:36:48.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:36:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35180.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:36:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34718.938625 secs 2023-03-10T00:36:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:48 smithi122 ceph-mon[93869]: pgmap v19027: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 192 KiB/s wr, 7 op/s 2023-03-10T00:36:48.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34718.938625 secs 2023-03-10T00:36:48.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:48 smithi181 ceph-mon[94189]: pgmap v19027: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 192 KiB/s wr, 7 op/s 2023-03-10T00:36:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34718.938625 secs 2023-03-10T00:36:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:48 smithi038 ceph-mon[92422]: pgmap v19027: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 192 KiB/s wr, 7 op/s 2023-03-10T00:36:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:36:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:36:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:36:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:36:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:36:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:36:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:50 smithi038 ceph-mon[92422]: pgmap v19028: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 195 KiB/s wr, 7 op/s 2023-03-10T00:36:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34721.475476 secs 2023-03-10T00:36:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:50 smithi122 ceph-mon[93869]: pgmap v19028: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 195 KiB/s wr, 7 op/s 2023-03-10T00:36:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34721.475476 secs 2023-03-10T00:36:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:50 smithi181 ceph-mon[94189]: pgmap v19028: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 195 KiB/s wr, 7 op/s 2023-03-10T00:36:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34721.475476 secs 2023-03-10T00:36:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:53 smithi122 ceph-mon[93869]: pgmap v19029: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 265 KiB/s wr, 11 op/s 2023-03-10T00:36:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:53 smithi181 ceph-mon[94189]: pgmap v19029: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 265 KiB/s wr, 11 op/s 2023-03-10T00:36:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:53 smithi038 ceph-mon[92422]: pgmap v19029: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 265 KiB/s wr, 11 op/s 2023-03-10T00:36:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35185.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:36:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:54 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:36:54.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:54 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:36:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:54 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:36:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:55 smithi122 ceph-mon[93869]: pgmap v19030: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 264 KiB/s wr, 9 op/s 2023-03-10T00:36:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:55 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:36:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:36:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:36:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:55 smithi181 ceph-mon[94189]: pgmap v19030: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 264 KiB/s wr, 9 op/s 2023-03-10T00:36:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:55 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:36:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:36:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:36:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:55 smithi038 ceph-mon[92422]: pgmap v19030: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 264 KiB/s wr, 9 op/s 2023-03-10T00:36:55.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:55 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:36:55.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:36:55.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:36:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:36:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:36:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34726.475535 secs 2023-03-10T00:36:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:36:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:36:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34726.475535 secs 2023-03-10T00:36:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:36:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:36:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34726.475535 secs 2023-03-10T00:36:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:57 smithi122 ceph-mon[93869]: pgmap v19031: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 491 KiB/s wr, 12 op/s 2023-03-10T00:36:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:36:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:36:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:57 smithi181 ceph-mon[94189]: pgmap v19031: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 491 KiB/s wr, 12 op/s 2023-03-10T00:36:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:36:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:36:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:57 smithi038 ceph-mon[92422]: pgmap v19031: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 491 KiB/s wr, 12 op/s 2023-03-10T00:36:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:36:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:36:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35190.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:36:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:59 smithi122 ceph-mon[93869]: pgmap v19032: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 898 KiB/s rd, 734 KiB/s wr, 13 op/s 2023-03-10T00:36:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:36:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:36:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:36:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:59 smithi181 ceph-mon[94189]: pgmap v19032: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 898 KiB/s rd, 734 KiB/s wr, 13 op/s 2023-03-10T00:36:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:36:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:36:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:36:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:59 smithi038 ceph-mon[92422]: pgmap v19032: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 898 KiB/s rd, 734 KiB/s wr, 13 op/s 2023-03-10T00:36:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:36:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:36:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:37:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34731.475621 secs 2023-03-10T00:37:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34731.475621 secs 2023-03-10T00:37:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34731.475621 secs 2023-03-10T00:37:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:01 smithi122 ceph-mon[93869]: pgmap v19033: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 870 KiB/s rd, 706 KiB/s wr, 13 op/s 2023-03-10T00:37:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:37:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:37:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:01 smithi181 ceph-mon[94189]: pgmap v19033: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 870 KiB/s rd, 706 KiB/s wr, 13 op/s 2023-03-10T00:37:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:37:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:37:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:01 smithi038 ceph-mon[92422]: pgmap v19033: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 870 KiB/s rd, 706 KiB/s wr, 13 op/s 2023-03-10T00:37:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:37:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:37:02.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34733.938832 secs 2023-03-10T00:37:02.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34733.938832 secs 2023-03-10T00:37:02.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34733.938832 secs 2023-03-10T00:37:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35195.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:37:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:03 smithi122 ceph-mon[93869]: pgmap v19034: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 983 KiB/s rd, 887 KiB/s wr, 16 op/s 2023-03-10T00:37:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:03 smithi181 ceph-mon[94189]: pgmap v19034: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 983 KiB/s rd, 887 KiB/s wr, 16 op/s 2023-03-10T00:37:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:03 smithi038 ceph-mon[92422]: pgmap v19034: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 983 KiB/s rd, 887 KiB/s wr, 16 op/s 2023-03-10T00:37:04.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:04 smithi122 ceph-mon[93869]: pgmap v19035: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 918 KiB/s rd, 839 KiB/s wr, 24 op/s 2023-03-10T00:37:04.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:04 smithi181 ceph-mon[94189]: pgmap v19035: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 918 KiB/s rd, 839 KiB/s wr, 24 op/s 2023-03-10T00:37:04.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:04 smithi038 ceph-mon[92422]: pgmap v19035: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 918 KiB/s rd, 839 KiB/s wr, 24 op/s 2023-03-10T00:37:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:05 smithi122 ceph-mon[93869]: osdmap e1405: 12 total, 12 up, 12 in 2023-03-10T00:37:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34736.475702 secs 2023-03-10T00:37:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:05 smithi181 ceph-mon[94189]: osdmap e1405: 12 total, 12 up, 12 in 2023-03-10T00:37:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34736.475702 secs 2023-03-10T00:37:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:05 smithi038 ceph-mon[92422]: osdmap e1405: 12 total, 12 up, 12 in 2023-03-10T00:37:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34736.475702 secs 2023-03-10T00:37:06.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:06 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:37:06.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:06 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:37:06.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:37:06.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:06 smithi122 ceph-mon[93869]: pgmap v19037: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 934 KiB/s rd, 877 KiB/s wr, 31 op/s 2023-03-10T00:37:06.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:06 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:37:06.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:06 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:37:06.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:37:06.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:06 smithi181 ceph-mon[94189]: pgmap v19037: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 934 KiB/s rd, 877 KiB/s wr, 31 op/s 2023-03-10T00:37:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:06 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:37:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:06 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:37:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:37:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:06 smithi038 ceph-mon[92422]: pgmap v19037: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 934 KiB/s rd, 877 KiB/s wr, 31 op/s 2023-03-10T00:37:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:37:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:37:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34738.938921 secs 2023-03-10T00:37:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:37:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:37:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34738.938921 secs 2023-03-10T00:37:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:37:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:37:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34738.938921 secs 2023-03-10T00:37:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35200.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:37:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:09 smithi122 ceph-mon[93869]: pgmap v19038: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 503 KiB/s wr, 29 op/s 2023-03-10T00:37:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:37:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:37:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:09 smithi181 ceph-mon[94189]: pgmap v19038: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 503 KiB/s wr, 29 op/s 2023-03-10T00:37:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:37:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:37:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:09 smithi038 ceph-mon[92422]: pgmap v19038: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 503 KiB/s wr, 29 op/s 2023-03-10T00:37:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:37:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:37:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34741.475758 secs 2023-03-10T00:37:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34741.475758 secs 2023-03-10T00:37:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34741.475758 secs 2023-03-10T00:37:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:11 smithi122 ceph-mon[93869]: pgmap v19039: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 503 KiB/s wr, 28 op/s 2023-03-10T00:37:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:11 smithi181 ceph-mon[94189]: pgmap v19039: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 503 KiB/s wr, 28 op/s 2023-03-10T00:37:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:11 smithi038 ceph-mon[92422]: pgmap v19039: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 503 KiB/s wr, 28 op/s 2023-03-10T00:37:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34743.939020 secs 2023-03-10T00:37:12.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34743.939020 secs 2023-03-10T00:37:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34743.939020 secs 2023-03-10T00:37:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35205.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:37:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:13 smithi122 ceph-mon[93869]: pgmap v19040: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 466 KiB/s wr, 27 op/s 2023-03-10T00:37:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:13 smithi122 ceph-mon[93869]: osdmap e1406: 12 total, 12 up, 12 in 2023-03-10T00:37:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:13 smithi181 ceph-mon[94189]: pgmap v19040: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 466 KiB/s wr, 27 op/s 2023-03-10T00:37:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:13 smithi181 ceph-mon[94189]: osdmap e1406: 12 total, 12 up, 12 in 2023-03-10T00:37:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:13 smithi038 ceph-mon[92422]: pgmap v19040: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 466 KiB/s wr, 27 op/s 2023-03-10T00:37:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:13 smithi038 ceph-mon[92422]: osdmap e1406: 12 total, 12 up, 12 in 2023-03-10T00:37:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:15 smithi122 ceph-mon[93869]: pgmap v19042: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 475 KiB/s wr, 13 op/s 2023-03-10T00:37:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34746.475818 secs 2023-03-10T00:37:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:15 smithi181 ceph-mon[94189]: pgmap v19042: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 475 KiB/s wr, 13 op/s 2023-03-10T00:37:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34746.475818 secs 2023-03-10T00:37:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:15 smithi038 ceph-mon[92422]: pgmap v19042: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 475 KiB/s wr, 13 op/s 2023-03-10T00:37:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34746.475818 secs 2023-03-10T00:37:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:16 smithi038 ceph-mon[92422]: pgmap v19043: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 215 KiB/s wr, 8 op/s 2023-03-10T00:37:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:16 smithi122 ceph-mon[93869]: pgmap v19043: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 215 KiB/s wr, 8 op/s 2023-03-10T00:37:17.284 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:16 smithi181 ceph-mon[94189]: pgmap v19043: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 215 KiB/s wr, 8 op/s 2023-03-10T00:37:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34748.939130 secs 2023-03-10T00:37:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34748.939130 secs 2023-03-10T00:37:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34748.939130 secs 2023-03-10T00:37:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35210.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:37:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:19 smithi122 ceph-mon[93869]: pgmap v19044: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 185 KiB/s wr, 8 op/s 2023-03-10T00:37:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:19 smithi181 ceph-mon[94189]: pgmap v19044: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 185 KiB/s wr, 8 op/s 2023-03-10T00:37:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:19 smithi038 ceph-mon[92422]: pgmap v19044: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 185 KiB/s wr, 8 op/s 2023-03-10T00:37:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34751.475878 secs 2023-03-10T00:37:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34751.475878 secs 2023-03-10T00:37:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34751.475878 secs 2023-03-10T00:37:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:21 smithi122 ceph-mon[93869]: pgmap v19045: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 185 KiB/s wr, 7 op/s 2023-03-10T00:37:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:21 smithi181 ceph-mon[94189]: pgmap v19045: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 185 KiB/s wr, 7 op/s 2023-03-10T00:37:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:21 smithi038 ceph-mon[92422]: pgmap v19045: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 185 KiB/s wr, 7 op/s 2023-03-10T00:37:22.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34753.939210 secs 2023-03-10T00:37:22.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34753.939210 secs 2023-03-10T00:37:22.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34753.939210 secs 2023-03-10T00:37:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35215.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:37:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:23 smithi122 ceph-mon[93869]: pgmap v19046: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 129 KiB/s wr, 7 op/s 2023-03-10T00:37:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:23 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:37:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:23 smithi181 ceph-mon[94189]: pgmap v19046: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 129 KiB/s wr, 7 op/s 2023-03-10T00:37:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:23 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:37:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:23 smithi038 ceph-mon[92422]: pgmap v19046: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 129 KiB/s wr, 7 op/s 2023-03-10T00:37:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:23 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:37:24.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:37:24.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:37:24.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:37:24.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:24 smithi122 ceph-mon[93869]: pgmap v19047: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 126 KiB/s wr, 6 op/s 2023-03-10T00:37:24.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:37:24.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:37:24.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:37:24.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:24 smithi181 ceph-mon[94189]: pgmap v19047: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 126 KiB/s wr, 6 op/s 2023-03-10T00:37:24.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:37:24.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:37:25.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:37:25.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:24 smithi038 ceph-mon[92422]: pgmap v19047: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 126 KiB/s wr, 6 op/s 2023-03-10T00:37:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34756.475967 secs 2023-03-10T00:37:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34756.475967 secs 2023-03-10T00:37:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34756.475967 secs 2023-03-10T00:37:27.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:26 smithi122 ceph-mon[93869]: pgmap v19048: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 108 KiB/s wr, 7 op/s 2023-03-10T00:37:27.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:26 smithi181 ceph-mon[94189]: pgmap v19048: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 108 KiB/s wr, 7 op/s 2023-03-10T00:37:27.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:26 smithi038 ceph-mon[92422]: pgmap v19048: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 108 KiB/s wr, 7 op/s 2023-03-10T00:37:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34758.939312 secs 2023-03-10T00:37:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34758.939312 secs 2023-03-10T00:37:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34758.939312 secs 2023-03-10T00:37:28.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35220.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:37:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:29 smithi122 ceph-mon[93869]: pgmap v19049: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 108 KiB/s wr, 5 op/s 2023-03-10T00:37:29.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:29 smithi181 ceph-mon[94189]: pgmap v19049: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 108 KiB/s wr, 5 op/s 2023-03-10T00:37:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:29 smithi038 ceph-mon[92422]: pgmap v19049: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 108 KiB/s wr, 5 op/s 2023-03-10T00:37:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34761.476026 secs 2023-03-10T00:37:30.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34761.476026 secs 2023-03-10T00:37:30.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34761.476026 secs 2023-03-10T00:37:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:31 smithi122 ceph-mon[93869]: pgmap v19050: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 108 KiB/s wr, 5 op/s 2023-03-10T00:37:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:31 smithi181 ceph-mon[94189]: pgmap v19050: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 108 KiB/s wr, 5 op/s 2023-03-10T00:37:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:31 smithi038 ceph-mon[92422]: pgmap v19050: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 108 KiB/s wr, 5 op/s 2023-03-10T00:37:32.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:32 smithi122 ceph-mon[93869]: pgmap v19051: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 159 KiB/s wr, 8 op/s 2023-03-10T00:37:32.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34763.939423 secs 2023-03-10T00:37:32.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:32 smithi181 ceph-mon[94189]: pgmap v19051: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 159 KiB/s wr, 8 op/s 2023-03-10T00:37:32.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34763.939423 secs 2023-03-10T00:37:32.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:32 smithi038 ceph-mon[92422]: pgmap v19051: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 159 KiB/s wr, 8 op/s 2023-03-10T00:37:33.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34763.939423 secs 2023-03-10T00:37:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35225.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:37:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:35 smithi122 ceph-mon[93869]: pgmap v19052: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 51 KiB/s wr, 5 op/s 2023-03-10T00:37:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34766.476112 secs 2023-03-10T00:37:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:35 smithi181 ceph-mon[94189]: pgmap v19052: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 51 KiB/s wr, 5 op/s 2023-03-10T00:37:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34766.476112 secs 2023-03-10T00:37:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:35 smithi038 ceph-mon[92422]: pgmap v19052: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 51 KiB/s wr, 5 op/s 2023-03-10T00:37:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34766.476112 secs 2023-03-10T00:37:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:37 smithi122 ceph-mon[93869]: pgmap v19053: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 131 KiB/s wr, 8 op/s 2023-03-10T00:37:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34768.939545 secs 2023-03-10T00:37:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:37 smithi181 ceph-mon[94189]: pgmap v19053: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 131 KiB/s wr, 8 op/s 2023-03-10T00:37:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34768.939545 secs 2023-03-10T00:37:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:37 smithi038 ceph-mon[92422]: pgmap v19053: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 131 KiB/s wr, 8 op/s 2023-03-10T00:37:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34768.939545 secs 2023-03-10T00:37:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35230.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:37:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:39 smithi122 ceph-mon[93869]: pgmap v19054: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 337 KiB/s rd, 131 KiB/s wr, 7 op/s 2023-03-10T00:37:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:39 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:37:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:39 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:37:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:39 smithi181 ceph-mon[94189]: pgmap v19054: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 337 KiB/s rd, 131 KiB/s wr, 7 op/s 2023-03-10T00:37:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:39 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:37:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:39 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:37:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:39 smithi038 ceph-mon[92422]: pgmap v19054: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 337 KiB/s rd, 131 KiB/s wr, 7 op/s 2023-03-10T00:37:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:39 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:37:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:39 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:37:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34771.476162 secs 2023-03-10T00:37:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34771.476162 secs 2023-03-10T00:37:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34771.476162 secs 2023-03-10T00:37:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:41 smithi122 ceph-mon[93869]: pgmap v19055: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 337 KiB/s rd, 131 KiB/s wr, 6 op/s 2023-03-10T00:37:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:37:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:37:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:41 smithi181 ceph-mon[94189]: pgmap v19055: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 337 KiB/s rd, 131 KiB/s wr, 6 op/s 2023-03-10T00:37:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:37:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:37:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:41 smithi038 ceph-mon[92422]: pgmap v19055: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 337 KiB/s rd, 131 KiB/s wr, 6 op/s 2023-03-10T00:37:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:37:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:37:42.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:42 smithi122 ceph-mon[93869]: pgmap v19056: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 474 KiB/s rd, 267 KiB/s wr, 9 op/s 2023-03-10T00:37:42.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34773.939602 secs 2023-03-10T00:37:42.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:42 smithi181 ceph-mon[94189]: pgmap v19056: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 474 KiB/s rd, 267 KiB/s wr, 9 op/s 2023-03-10T00:37:42.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34773.939602 secs 2023-03-10T00:37:42.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:42 smithi038 ceph-mon[92422]: pgmap v19056: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 474 KiB/s rd, 267 KiB/s wr, 9 op/s 2023-03-10T00:37:42.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34773.939602 secs 2023-03-10T00:37:43.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35235.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:37:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:45 smithi122 ceph-mon[93869]: pgmap v19057: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 422 KiB/s rd, 216 KiB/s wr, 6 op/s 2023-03-10T00:37:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34776.476221 secs 2023-03-10T00:37:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:45 smithi181 ceph-mon[94189]: pgmap v19057: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 422 KiB/s rd, 216 KiB/s wr, 6 op/s 2023-03-10T00:37:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34776.476221 secs 2023-03-10T00:37:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:45 smithi038 ceph-mon[92422]: pgmap v19057: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 422 KiB/s rd, 216 KiB/s wr, 6 op/s 2023-03-10T00:37:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34776.476221 secs 2023-03-10T00:37:47.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:47 smithi122 ceph-mon[93869]: pgmap v19058: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 674 KiB/s rd, 342 KiB/s wr, 9 op/s 2023-03-10T00:37:48.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:47 smithi038 ceph-mon[92422]: pgmap v19058: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 674 KiB/s rd, 342 KiB/s wr, 9 op/s 2023-03-10T00:37:48.003 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:47 smithi181 ceph-mon[94189]: pgmap v19058: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 674 KiB/s rd, 342 KiB/s wr, 9 op/s 2023-03-10T00:37:48.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35240.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:37:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:49 smithi122 ceph-mon[93869]: pgmap v19059: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 398 KiB/s rd, 271 KiB/s wr, 7 op/s 2023-03-10T00:37:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:49 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:37:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:49 smithi181 ceph-mon[94189]: pgmap v19059: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 398 KiB/s rd, 271 KiB/s wr, 7 op/s 2023-03-10T00:37:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:49 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:37:50.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:49 smithi038 ceph-mon[92422]: pgmap v19059: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 398 KiB/s rd, 271 KiB/s wr, 7 op/s 2023-03-10T00:37:50.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:49 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:37:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:50 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:37:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:50 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:37:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:50 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:37:50.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34781.476280 secs 2023-03-10T00:37:50.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:50 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:37:50.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:50 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:37:50.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:50 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:37:50.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34781.476280 secs 2023-03-10T00:37:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:50 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:37:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:50 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:37:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:50 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:37:51.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34781.476280 secs 2023-03-10T00:37:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:51 smithi122 ceph-mon[93869]: pgmap v19060: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 398 KiB/s rd, 270 KiB/s wr, 6 op/s 2023-03-10T00:37:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:51 smithi181 ceph-mon[94189]: pgmap v19060: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 398 KiB/s rd, 270 KiB/s wr, 6 op/s 2023-03-10T00:37:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:51 smithi038 ceph-mon[92422]: pgmap v19060: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 398 KiB/s rd, 270 KiB/s wr, 6 op/s 2023-03-10T00:37:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:52 smithi122 ceph-mon[93869]: pgmap v19061: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 760 KiB/s rd, 632 KiB/s wr, 10 op/s 2023-03-10T00:37:52.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:52 smithi181 ceph-mon[94189]: pgmap v19061: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 760 KiB/s rd, 632 KiB/s wr, 10 op/s 2023-03-10T00:37:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:52 smithi038 ceph-mon[92422]: pgmap v19061: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 760 KiB/s rd, 632 KiB/s wr, 10 op/s 2023-03-10T00:37:53.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35245.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:37:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:37:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:37:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:37:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:37:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:37:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:37:54.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:54 smithi122 ceph-mon[93869]: pgmap v19062: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 686 KiB/s rd, 559 KiB/s wr, 8 op/s 2023-03-10T00:37:54.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:54 smithi181 ceph-mon[94189]: pgmap v19062: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 686 KiB/s rd, 559 KiB/s wr, 8 op/s 2023-03-10T00:37:54.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:54 smithi038 ceph-mon[92422]: pgmap v19062: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 686 KiB/s rd, 559 KiB/s wr, 8 op/s 2023-03-10T00:37:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:37:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:37:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34786.476339 secs 2023-03-10T00:37:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:37:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:37:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34786.476339 secs 2023-03-10T00:37:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:37:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:37:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34786.476339 secs 2023-03-10T00:37:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:56 smithi122 ceph-mon[93869]: pgmap v19063: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 689 KiB/s rd, 559 KiB/s wr, 11 op/s 2023-03-10T00:37:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:56 smithi181 ceph-mon[94189]: pgmap v19063: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 689 KiB/s rd, 559 KiB/s wr, 11 op/s 2023-03-10T00:37:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:56 smithi038 ceph-mon[92422]: pgmap v19063: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 689 KiB/s rd, 559 KiB/s wr, 11 op/s 2023-03-10T00:37:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:37:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:37:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34788.939850 secs 2023-03-10T00:37:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:37:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:37:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34788.939850 secs 2023-03-10T00:37:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:37:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:37:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34788.939850 secs 2023-03-10T00:37:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35250.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:37:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:58 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:37:59.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:58 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:37:59.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:58 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:37:59.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:37:58 smithi038 ceph-mon[92422]: pgmap v19064: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 794 KiB/s wr, 11 op/s 2023-03-10T00:37:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:58 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:37:59.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:58 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:37:59.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:58 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:37:59.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:37:58 smithi122 ceph-mon[93869]: pgmap v19064: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 794 KiB/s wr, 11 op/s 2023-03-10T00:37:59.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:58 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:37:59.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:58 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:37:59.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:58 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:37:59.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:37:58 smithi181 ceph-mon[94189]: pgmap v19064: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 794 KiB/s wr, 11 op/s 2023-03-10T00:38:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:38:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:38:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:00 smithi122 ceph-mon[93869]: pgmap v19065: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 786 KiB/s wr, 10 op/s 2023-03-10T00:38:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34791.476397 secs 2023-03-10T00:38:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:38:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:38:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:00 smithi181 ceph-mon[94189]: pgmap v19065: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 786 KiB/s wr, 10 op/s 2023-03-10T00:38:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34791.476397 secs 2023-03-10T00:38:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:38:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:38:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:00 smithi038 ceph-mon[92422]: pgmap v19065: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 786 KiB/s wr, 10 op/s 2023-03-10T00:38:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34791.476397 secs 2023-03-10T00:38:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:03 smithi122 ceph-mon[93869]: pgmap v19066: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 665 KiB/s rd, 815 KiB/s wr, 14 op/s 2023-03-10T00:38:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34793.939937 secs 2023-03-10T00:38:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:03 smithi181 ceph-mon[94189]: pgmap v19066: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 665 KiB/s rd, 815 KiB/s wr, 14 op/s 2023-03-10T00:38:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34793.939937 secs 2023-03-10T00:38:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:03 smithi038 ceph-mon[92422]: pgmap v19066: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 665 KiB/s rd, 815 KiB/s wr, 14 op/s 2023-03-10T00:38:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34793.939937 secs 2023-03-10T00:38:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35255.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:38:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:05 smithi122 ceph-mon[93869]: pgmap v19067: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 610 KiB/s wr, 13 op/s 2023-03-10T00:38:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34796.476482 secs 2023-03-10T00:38:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:05 smithi181 ceph-mon[94189]: pgmap v19067: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 610 KiB/s wr, 13 op/s 2023-03-10T00:38:05.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34796.476482 secs 2023-03-10T00:38:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:05 smithi038 ceph-mon[92422]: pgmap v19067: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 610 KiB/s wr, 13 op/s 2023-03-10T00:38:06.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34796.476482 secs 2023-03-10T00:38:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:07 smithi122 ceph-mon[93869]: pgmap v19068: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 570 KiB/s wr, 16 op/s 2023-03-10T00:38:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:38:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:38:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34798.939994 secs 2023-03-10T00:38:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:07 smithi181 ceph-mon[94189]: pgmap v19068: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 570 KiB/s wr, 16 op/s 2023-03-10T00:38:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:38:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:38:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34798.939994 secs 2023-03-10T00:38:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:07 smithi038 ceph-mon[92422]: pgmap v19068: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 570 KiB/s wr, 16 op/s 2023-03-10T00:38:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:38:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:38:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34798.939994 secs 2023-03-10T00:38:08.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35260.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:38:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:09 smithi122 ceph-mon[93869]: pgmap v19069: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 754 KiB/s rd, 779 KiB/s wr, 27 op/s 2023-03-10T00:38:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:09 smithi181 ceph-mon[94189]: pgmap v19069: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 754 KiB/s rd, 779 KiB/s wr, 27 op/s 2023-03-10T00:38:10.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:09 smithi038 ceph-mon[92422]: pgmap v19069: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 754 KiB/s rd, 779 KiB/s wr, 27 op/s 2023-03-10T00:38:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:10 smithi122 ceph-mon[93869]: osdmap e1407: 12 total, 12 up, 12 in 2023-03-10T00:38:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:10 smithi122 ceph-mon[93869]: pgmap v19071: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 909 KiB/s rd, 754 KiB/s wr, 30 op/s 2023-03-10T00:38:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34801.476542 secs 2023-03-10T00:38:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:10 smithi181 ceph-mon[94189]: osdmap e1407: 12 total, 12 up, 12 in 2023-03-10T00:38:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:10 smithi181 ceph-mon[94189]: pgmap v19071: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 909 KiB/s rd, 754 KiB/s wr, 30 op/s 2023-03-10T00:38:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34801.476542 secs 2023-03-10T00:38:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:10 smithi038 ceph-mon[92422]: osdmap e1407: 12 total, 12 up, 12 in 2023-03-10T00:38:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:10 smithi038 ceph-mon[92422]: pgmap v19071: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 909 KiB/s rd, 754 KiB/s wr, 30 op/s 2023-03-10T00:38:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34801.476542 secs 2023-03-10T00:38:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:11 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:38:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:11 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:38:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:11 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:38:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:11 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:38:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:11 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:38:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:11 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:38:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:12 smithi122 ceph-mon[93869]: pgmap v19072: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 902 KiB/s rd, 746 KiB/s wr, 29 op/s 2023-03-10T00:38:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34803.940077 secs 2023-03-10T00:38:12.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:12 smithi181 ceph-mon[94189]: pgmap v19072: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 902 KiB/s rd, 746 KiB/s wr, 29 op/s 2023-03-10T00:38:12.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34803.940077 secs 2023-03-10T00:38:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:12 smithi038 ceph-mon[92422]: pgmap v19072: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 902 KiB/s rd, 746 KiB/s wr, 29 op/s 2023-03-10T00:38:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34803.940077 secs 2023-03-10T00:38:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35265.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:38:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:38:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:38:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:38:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:38:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:38:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:38:14.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:14 smithi181 ceph-mon[94189]: pgmap v19073: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 517 KiB/s rd, 558 KiB/s wr, 26 op/s 2023-03-10T00:38:14.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:14 smithi038 ceph-mon[92422]: pgmap v19073: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 517 KiB/s rd, 558 KiB/s wr, 26 op/s 2023-03-10T00:38:15.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:14 smithi122 ceph-mon[93869]: pgmap v19073: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 517 KiB/s rd, 558 KiB/s wr, 26 op/s 2023-03-10T00:38:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34806.476601 secs 2023-03-10T00:38:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34806.476601 secs 2023-03-10T00:38:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34806.476601 secs 2023-03-10T00:38:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:16 smithi038 ceph-mon[92422]: pgmap v19074: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 803 KiB/s rd, 819 KiB/s wr, 25 op/s 2023-03-10T00:38:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:16 smithi122 ceph-mon[93869]: pgmap v19074: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 803 KiB/s rd, 819 KiB/s wr, 25 op/s 2023-03-10T00:38:17.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:16 smithi181 ceph-mon[94189]: pgmap v19074: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 803 KiB/s rd, 819 KiB/s wr, 25 op/s 2023-03-10T00:38:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:38:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:38:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34808.940175 secs 2023-03-10T00:38:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:38:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:38:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34808.940175 secs 2023-03-10T00:38:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:38:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:38:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34808.940175 secs 2023-03-10T00:38:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35270.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:38:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:19 smithi122 ceph-mon[93869]: pgmap v19075: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 570 KiB/s rd, 568 KiB/s wr, 10 op/s 2023-03-10T00:38:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:19 smithi122 ceph-mon[93869]: osdmap e1408: 12 total, 12 up, 12 in 2023-03-10T00:38:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:19 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:38:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:19 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:38:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:19 smithi181 ceph-mon[94189]: pgmap v19075: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 570 KiB/s rd, 568 KiB/s wr, 10 op/s 2023-03-10T00:38:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:19 smithi181 ceph-mon[94189]: osdmap e1408: 12 total, 12 up, 12 in 2023-03-10T00:38:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:19 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:38:19.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:19 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:38:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:19 smithi038 ceph-mon[92422]: pgmap v19075: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 570 KiB/s rd, 568 KiB/s wr, 10 op/s 2023-03-10T00:38:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:19 smithi038 ceph-mon[92422]: osdmap e1408: 12 total, 12 up, 12 in 2023-03-10T00:38:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:19 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:38:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:19 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:38:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34811.476656 secs 2023-03-10T00:38:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34811.476656 secs 2023-03-10T00:38:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34811.476656 secs 2023-03-10T00:38:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:21 smithi122 ceph-mon[93869]: pgmap v19077: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 318 KiB/s rd, 316 KiB/s wr, 9 op/s 2023-03-10T00:38:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:21 smithi181 ceph-mon[94189]: pgmap v19077: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 318 KiB/s rd, 316 KiB/s wr, 9 op/s 2023-03-10T00:38:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:21 smithi038 ceph-mon[92422]: pgmap v19077: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 318 KiB/s rd, 316 KiB/s wr, 9 op/s 2023-03-10T00:38:22.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34813.940257 secs 2023-03-10T00:38:22.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34813.940257 secs 2023-03-10T00:38:22.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34813.940257 secs 2023-03-10T00:38:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35275.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:38:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:23 smithi122 ceph-mon[93869]: pgmap v19078: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 288 KiB/s wr, 8 op/s 2023-03-10T00:38:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:23 smithi181 ceph-mon[94189]: pgmap v19078: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 288 KiB/s wr, 8 op/s 2023-03-10T00:38:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:23 smithi038 ceph-mon[92422]: pgmap v19078: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 288 KiB/s wr, 8 op/s 2023-03-10T00:38:24.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:38:24.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:24 smithi122 ceph-mon[93869]: pgmap v19079: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 288 KiB/s wr, 8 op/s 2023-03-10T00:38:24.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:38:24.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:38:24.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:38:24.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:38:24.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:24 smithi181 ceph-mon[94189]: pgmap v19079: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 288 KiB/s wr, 8 op/s 2023-03-10T00:38:24.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:38:24.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:38:24.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:38:24.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:38:24.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:24 smithi038 ceph-mon[92422]: pgmap v19079: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 288 KiB/s wr, 8 op/s 2023-03-10T00:38:24.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:38:24.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:38:25.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:38:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34816.476717 secs 2023-03-10T00:38:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34816.476717 secs 2023-03-10T00:38:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34816.476717 secs 2023-03-10T00:38:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:26 smithi122 ceph-mon[93869]: pgmap v19080: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 7 op/s 2023-03-10T00:38:26.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:26 smithi181 ceph-mon[94189]: pgmap v19080: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 7 op/s 2023-03-10T00:38:26.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:26 smithi038 ceph-mon[92422]: pgmap v19080: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 511 B/s wr, 7 op/s 2023-03-10T00:38:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34818.940316 secs 2023-03-10T00:38:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34818.940316 secs 2023-03-10T00:38:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34818.940316 secs 2023-03-10T00:38:28.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35280.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:38:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:29 smithi122 ceph-mon[93869]: pgmap v19081: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-10T00:38:29.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:29 smithi181 ceph-mon[94189]: pgmap v19081: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-10T00:38:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:29 smithi038 ceph-mon[92422]: pgmap v19081: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-10T00:38:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34821.476770 secs 2023-03-10T00:38:30.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34821.476770 secs 2023-03-10T00:38:30.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34821.476770 secs 2023-03-10T00:38:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:31 smithi122 ceph-mon[93869]: pgmap v19082: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-10T00:38:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:31 smithi181 ceph-mon[94189]: pgmap v19082: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-10T00:38:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:31 smithi038 ceph-mon[92422]: pgmap v19082: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-10T00:38:32.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:32 smithi122 ceph-mon[93869]: pgmap v19083: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 8 op/s 2023-03-10T00:38:32.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34823.940419 secs 2023-03-10T00:38:32.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:32 smithi181 ceph-mon[94189]: pgmap v19083: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 8 op/s 2023-03-10T00:38:32.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34823.940419 secs 2023-03-10T00:38:32.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:32 smithi038 ceph-mon[92422]: pgmap v19083: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 8 op/s 2023-03-10T00:38:32.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34823.940419 secs 2023-03-10T00:38:33.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35285.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:38:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:35 smithi122 ceph-mon[93869]: pgmap v19084: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-10T00:38:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34826.476826 secs 2023-03-10T00:38:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:35 smithi181 ceph-mon[94189]: pgmap v19084: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-10T00:38:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34826.476826 secs 2023-03-10T00:38:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:35 smithi038 ceph-mon[92422]: pgmap v19084: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-10T00:38:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34826.476826 secs 2023-03-10T00:38:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:37 smithi122 ceph-mon[93869]: pgmap v19085: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 8 op/s 2023-03-10T00:38:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34828.940499 secs 2023-03-10T00:38:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:37 smithi181 ceph-mon[94189]: pgmap v19085: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 8 op/s 2023-03-10T00:38:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34828.940499 secs 2023-03-10T00:38:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:37 smithi038 ceph-mon[92422]: pgmap v19085: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 8 op/s 2023-03-10T00:38:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34828.940499 secs 2023-03-10T00:38:38.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35290.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:38:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:39 smithi122 ceph-mon[93869]: pgmap v19086: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-10T00:38:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:39 smithi181 ceph-mon[94189]: pgmap v19086: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-10T00:38:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:39 smithi038 ceph-mon[92422]: pgmap v19086: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-10T00:38:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34831.476913 secs 2023-03-10T00:38:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34831.476913 secs 2023-03-10T00:38:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34831.476913 secs 2023-03-10T00:38:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:41 smithi122 ceph-mon[93869]: pgmap v19087: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-10T00:38:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:41 smithi181 ceph-mon[94189]: pgmap v19087: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-10T00:38:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:41 smithi038 ceph-mon[92422]: pgmap v19087: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 6 op/s 2023-03-10T00:38:42.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:42 smithi122 ceph-mon[93869]: pgmap v19088: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 137 KiB/s wr, 9 op/s 2023-03-10T00:38:42.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34833.940616 secs 2023-03-10T00:38:42.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:42 smithi181 ceph-mon[94189]: pgmap v19088: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 137 KiB/s wr, 9 op/s 2023-03-10T00:38:42.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34833.940616 secs 2023-03-10T00:38:42.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:42 smithi038 ceph-mon[92422]: pgmap v19088: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 137 KiB/s wr, 9 op/s 2023-03-10T00:38:42.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34833.940616 secs 2023-03-10T00:38:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35295.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:38:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:45 smithi122 ceph-mon[93869]: pgmap v19089: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 137 KiB/s wr, 6 op/s 2023-03-10T00:38:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34836.477003 secs 2023-03-10T00:38:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:45 smithi181 ceph-mon[94189]: pgmap v19089: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 137 KiB/s wr, 6 op/s 2023-03-10T00:38:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34836.477003 secs 2023-03-10T00:38:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:45 smithi038 ceph-mon[92422]: pgmap v19089: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 137 KiB/s wr, 6 op/s 2023-03-10T00:38:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34836.477003 secs 2023-03-10T00:38:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:47 smithi122 ceph-mon[93869]: pgmap v19090: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 271 KiB/s wr, 9 op/s 2023-03-10T00:38:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34838.940707 secs 2023-03-10T00:38:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:47 smithi181 ceph-mon[94189]: pgmap v19090: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 271 KiB/s wr, 9 op/s 2023-03-10T00:38:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34838.940707 secs 2023-03-10T00:38:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:47 smithi038 ceph-mon[92422]: pgmap v19090: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 271 KiB/s wr, 9 op/s 2023-03-10T00:38:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34838.940707 secs 2023-03-10T00:38:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35300.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:38:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:49 smithi122 ceph-mon[93869]: pgmap v19091: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 422 KiB/s rd, 272 KiB/s wr, 7 op/s 2023-03-10T00:38:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:49 smithi181 ceph-mon[94189]: pgmap v19091: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 422 KiB/s rd, 272 KiB/s wr, 7 op/s 2023-03-10T00:38:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:49 smithi038 ceph-mon[92422]: pgmap v19091: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 422 KiB/s rd, 272 KiB/s wr, 7 op/s 2023-03-10T00:38:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34841.477063 secs 2023-03-10T00:38:50.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34841.477063 secs 2023-03-10T00:38:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34841.477063 secs 2023-03-10T00:38:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:51 smithi122 ceph-mon[93869]: pgmap v19092: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 422 KiB/s rd, 271 KiB/s wr, 7 op/s 2023-03-10T00:38:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:51 smithi181 ceph-mon[94189]: pgmap v19092: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 422 KiB/s rd, 271 KiB/s wr, 7 op/s 2023-03-10T00:38:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:51 smithi038 ceph-mon[92422]: pgmap v19092: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 422 KiB/s rd, 271 KiB/s wr, 7 op/s 2023-03-10T00:38:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34843.940768 secs 2023-03-10T00:38:52.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34843.940768 secs 2023-03-10T00:38:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34843.940768 secs 2023-03-10T00:38:53.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35305.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:38:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:53 smithi122 ceph-mon[93869]: pgmap v19093: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 849 KiB/s rd, 698 KiB/s wr, 10 op/s 2023-03-10T00:38:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:38:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:38:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:53 smithi181 ceph-mon[94189]: pgmap v19093: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 849 KiB/s rd, 698 KiB/s wr, 10 op/s 2023-03-10T00:38:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:38:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:38:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:53 smithi038 ceph-mon[92422]: pgmap v19093: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 849 KiB/s rd, 698 KiB/s wr, 10 op/s 2023-03-10T00:38:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:38:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:38:54.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:54 smithi122 ceph-mon[93869]: pgmap v19094: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 711 KiB/s rd, 561 KiB/s wr, 7 op/s 2023-03-10T00:38:54.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:54 smithi181 ceph-mon[94189]: pgmap v19094: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 711 KiB/s rd, 561 KiB/s wr, 7 op/s 2023-03-10T00:38:54.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:54 smithi038 ceph-mon[92422]: pgmap v19094: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 711 KiB/s rd, 561 KiB/s wr, 7 op/s 2023-03-10T00:38:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:38:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:38:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34846.477147 secs 2023-03-10T00:38:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:38:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:38:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34846.477147 secs 2023-03-10T00:38:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:38:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:38:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34846.477147 secs 2023-03-10T00:38:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:56 smithi038 ceph-mon[92422]: pgmap v19095: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 745 KiB/s rd, 732 KiB/s wr, 12 op/s 2023-03-10T00:38:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:56 smithi122 ceph-mon[93869]: pgmap v19095: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 745 KiB/s rd, 732 KiB/s wr, 12 op/s 2023-03-10T00:38:57.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:56 smithi181 ceph-mon[94189]: pgmap v19095: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 745 KiB/s rd, 732 KiB/s wr, 12 op/s 2023-03-10T00:38:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:38:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:38:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:57 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:38:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:38:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:38:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:57 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:38:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:38:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:38:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:57 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:38:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35310.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:38:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:58 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:38:59.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:58 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:38:59.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:58 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:38:59.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:58 smithi038 ceph-mon[92422]: pgmap v19096: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 485 KiB/s rd, 623 KiB/s wr, 10 op/s 2023-03-10T00:38:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:58 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:38:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:58 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:38:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:58 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:38:59.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:58 smithi122 ceph-mon[93869]: pgmap v19096: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 485 KiB/s rd, 623 KiB/s wr, 10 op/s 2023-03-10T00:38:59.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:58 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:38:59.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:58 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:38:59.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:58 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:38:59.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:58 smithi181 ceph-mon[94189]: pgmap v19096: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 485 KiB/s rd, 623 KiB/s wr, 10 op/s 2023-03-10T00:38:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:38:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:38:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:39:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:39:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:38:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:39:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:39:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:38:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:39:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:00 smithi122 ceph-mon[93869]: pgmap v19097: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 623 KiB/s wr, 10 op/s 2023-03-10T00:39:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34851.477228 secs 2023-03-10T00:39:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:00 smithi181 ceph-mon[94189]: pgmap v19097: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 623 KiB/s wr, 10 op/s 2023-03-10T00:39:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34851.477228 secs 2023-03-10T00:39:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:00 smithi038 ceph-mon[92422]: pgmap v19097: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 623 KiB/s wr, 10 op/s 2023-03-10T00:39:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34851.477228 secs 2023-03-10T00:39:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:39:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:39:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:39:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:39:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:39:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:39:03.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:02 smithi122 ceph-mon[93869]: pgmap v19098: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 990 KiB/s rd, 976 KiB/s wr, 15 op/s 2023-03-10T00:39:03.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:02 smithi181 ceph-mon[94189]: pgmap v19098: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 990 KiB/s rd, 976 KiB/s wr, 15 op/s 2023-03-10T00:39:03.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:02 smithi038 ceph-mon[92422]: pgmap v19098: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 990 KiB/s rd, 976 KiB/s wr, 15 op/s 2023-03-10T00:39:03.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35315.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:39:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:05 smithi122 ceph-mon[93869]: pgmap v19099: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 549 KiB/s wr, 13 op/s 2023-03-10T00:39:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34856.477287 secs 2023-03-10T00:39:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:05 smithi181 ceph-mon[94189]: pgmap v19099: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 549 KiB/s wr, 13 op/s 2023-03-10T00:39:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34856.477287 secs 2023-03-10T00:39:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:05 smithi038 ceph-mon[92422]: pgmap v19099: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 549 KiB/s wr, 13 op/s 2023-03-10T00:39:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34856.477287 secs 2023-03-10T00:39:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:07 smithi122 ceph-mon[93869]: pgmap v19100: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 668 KiB/s rd, 675 KiB/s wr, 19 op/s 2023-03-10T00:39:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34858.941066 secs 2023-03-10T00:39:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:07 smithi181 ceph-mon[94189]: pgmap v19100: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 668 KiB/s rd, 675 KiB/s wr, 19 op/s 2023-03-10T00:39:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34858.941066 secs 2023-03-10T00:39:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:07 smithi038 ceph-mon[92422]: pgmap v19100: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 668 KiB/s rd, 675 KiB/s wr, 19 op/s 2023-03-10T00:39:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34858.941066 secs 2023-03-10T00:39:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35320.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:39:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:09 smithi122 ceph-mon[93869]: pgmap v19101: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 641 KiB/s rd, 525 KiB/s wr, 27 op/s 2023-03-10T00:39:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:09 smithi181 ceph-mon[94189]: pgmap v19101: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 641 KiB/s rd, 525 KiB/s wr, 27 op/s 2023-03-10T00:39:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:09 smithi038 ceph-mon[92422]: pgmap v19101: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 641 KiB/s rd, 525 KiB/s wr, 27 op/s 2023-03-10T00:39:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:10 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:39:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:10 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:39:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:10 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:39:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:10 smithi122 ceph-mon[93869]: osdmap e1409: 12 total, 12 up, 12 in 2023-03-10T00:39:10.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34861.477374 secs 2023-03-10T00:39:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:10 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:39:10.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:10 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:39:10.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:10 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:39:10.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:10 smithi181 ceph-mon[94189]: osdmap e1409: 12 total, 12 up, 12 in 2023-03-10T00:39:10.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34861.477374 secs 2023-03-10T00:39:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:10 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:39:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:10 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:39:11.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:10 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:39:11.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:10 smithi038 ceph-mon[92422]: osdmap e1409: 12 total, 12 up, 12 in 2023-03-10T00:39:11.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34861.477374 secs 2023-03-10T00:39:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:11 smithi122 ceph-mon[93869]: pgmap v19103: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 727 KiB/s rd, 599 KiB/s wr, 30 op/s 2023-03-10T00:39:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:11 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:39:11.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:11 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:39:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:11 smithi181 ceph-mon[94189]: pgmap v19103: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 727 KiB/s rd, 599 KiB/s wr, 30 op/s 2023-03-10T00:39:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:11 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:39:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:11 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:39:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:11 smithi038 ceph-mon[92422]: pgmap v19103: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 727 KiB/s rd, 599 KiB/s wr, 30 op/s 2023-03-10T00:39:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:11 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:39:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:11 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:39:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34863.941114 secs 2023-03-10T00:39:12.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34863.941114 secs 2023-03-10T00:39:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34863.941114 secs 2023-03-10T00:39:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35325.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:39:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:13 smithi122 ceph-mon[93869]: pgmap v19104: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 519 KiB/s rd, 560 KiB/s wr, 29 op/s 2023-03-10T00:39:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:39:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:39:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:13 smithi181 ceph-mon[94189]: pgmap v19104: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 519 KiB/s rd, 560 KiB/s wr, 29 op/s 2023-03-10T00:39:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:39:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:39:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:13 smithi038 ceph-mon[92422]: pgmap v19104: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 519 KiB/s rd, 560 KiB/s wr, 29 op/s 2023-03-10T00:39:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:39:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:39:14.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:14 smithi122 ceph-mon[93869]: pgmap v19105: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 560 KiB/s wr, 27 op/s 2023-03-10T00:39:14.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:14 smithi181 ceph-mon[94189]: pgmap v19105: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 560 KiB/s wr, 27 op/s 2023-03-10T00:39:14.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:14 smithi038 ceph-mon[92422]: pgmap v19105: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 560 KiB/s wr, 27 op/s 2023-03-10T00:39:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34866.477434 secs 2023-03-10T00:39:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34866.477434 secs 2023-03-10T00:39:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34866.477434 secs 2023-03-10T00:39:16.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:16 smithi181 ceph-mon[94189]: pgmap v19106: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 479 KiB/s wr, 23 op/s 2023-03-10T00:39:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:16 smithi038 ceph-mon[92422]: pgmap v19106: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 479 KiB/s wr, 23 op/s 2023-03-10T00:39:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:16 smithi122 ceph-mon[93869]: pgmap v19106: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 479 KiB/s wr, 23 op/s 2023-03-10T00:39:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34868.941199 secs 2023-03-10T00:39:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34868.941199 secs 2023-03-10T00:39:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34868.941199 secs 2023-03-10T00:39:18.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35330.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:39:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:19 smithi122 ceph-mon[93869]: pgmap v19107: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 517 KiB/s rd, 515 KiB/s wr, 9 op/s 2023-03-10T00:39:19.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:19 smithi122 ceph-mon[93869]: osdmap e1410: 12 total, 12 up, 12 in 2023-03-10T00:39:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:19 smithi181 ceph-mon[94189]: pgmap v19107: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 517 KiB/s rd, 515 KiB/s wr, 9 op/s 2023-03-10T00:39:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:19 smithi181 ceph-mon[94189]: osdmap e1410: 12 total, 12 up, 12 in 2023-03-10T00:39:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:19 smithi038 ceph-mon[92422]: pgmap v19107: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 517 KiB/s rd, 515 KiB/s wr, 9 op/s 2023-03-10T00:39:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:19 smithi038 ceph-mon[92422]: osdmap e1410: 12 total, 12 up, 12 in 2023-03-10T00:39:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34871.477491 secs 2023-03-10T00:39:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34871.477491 secs 2023-03-10T00:39:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34871.477491 secs 2023-03-10T00:39:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:21 smithi122 ceph-mon[93869]: pgmap v19109: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 517 KiB/s rd, 515 KiB/s wr, 9 op/s 2023-03-10T00:39:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:21 smithi181 ceph-mon[94189]: pgmap v19109: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 517 KiB/s rd, 515 KiB/s wr, 9 op/s 2023-03-10T00:39:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:21 smithi038 ceph-mon[92422]: pgmap v19109: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 517 KiB/s rd, 515 KiB/s wr, 9 op/s 2023-03-10T00:39:22.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:22 smithi122 ceph-mon[93869]: pgmap v19110: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 131 KiB/s wr, 7 op/s 2023-03-10T00:39:22.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34873.941302 secs 2023-03-10T00:39:22.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:22 smithi181 ceph-mon[94189]: pgmap v19110: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 131 KiB/s wr, 7 op/s 2023-03-10T00:39:22.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34873.941302 secs 2023-03-10T00:39:22.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:22 smithi038 ceph-mon[92422]: pgmap v19110: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 131 KiB/s wr, 7 op/s 2023-03-10T00:39:22.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34873.941302 secs 2023-03-10T00:39:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35335.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:39:24.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:39:24.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:39:24.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:39:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:25 smithi122 ceph-mon[93869]: pgmap v19111: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 131 KiB/s wr, 7 op/s 2023-03-10T00:39:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34876.477552 secs 2023-03-10T00:39:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:39:25.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:39:25.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:39:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:25 smithi181 ceph-mon[94189]: pgmap v19111: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 131 KiB/s wr, 7 op/s 2023-03-10T00:39:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34876.477552 secs 2023-03-10T00:39:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:39:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:39:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:39:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:25 smithi038 ceph-mon[92422]: pgmap v19111: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 133 KiB/s rd, 131 KiB/s wr, 7 op/s 2023-03-10T00:39:26.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34876.477552 secs 2023-03-10T00:39:26.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:39:26.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:39:26.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:39:26.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:26 smithi181 ceph-mon[94189]: pgmap v19112: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 62 KiB/s wr, 7 op/s 2023-03-10T00:39:26.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:26 smithi038 ceph-mon[92422]: pgmap v19112: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 62 KiB/s wr, 7 op/s 2023-03-10T00:39:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:26 smithi122 ceph-mon[93869]: pgmap v19112: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 62 KiB/s wr, 7 op/s 2023-03-10T00:39:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34878.941362 secs 2023-03-10T00:39:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34878.941362 secs 2023-03-10T00:39:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34878.941362 secs 2023-03-10T00:39:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35340.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:39:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:29 smithi122 ceph-mon[93869]: pgmap v19113: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-10T00:39:29.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:29 smithi181 ceph-mon[94189]: pgmap v19113: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-10T00:39:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:29 smithi038 ceph-mon[92422]: pgmap v19113: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-10T00:39:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34881.477633 secs 2023-03-10T00:39:30.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34881.477633 secs 2023-03-10T00:39:30.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34881.477633 secs 2023-03-10T00:39:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:31 smithi122 ceph-mon[93869]: pgmap v19114: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-10T00:39:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:31 smithi181 ceph-mon[94189]: pgmap v19114: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-10T00:39:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:31 smithi038 ceph-mon[92422]: pgmap v19114: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-10T00:39:32.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:32 smithi122 ceph-mon[93869]: pgmap v19115: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 8 op/s 2023-03-10T00:39:32.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34883.941443 secs 2023-03-10T00:39:32.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:32 smithi181 ceph-mon[94189]: pgmap v19115: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 8 op/s 2023-03-10T00:39:32.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34883.941443 secs 2023-03-10T00:39:32.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:32 smithi038 ceph-mon[92422]: pgmap v19115: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 8 op/s 2023-03-10T00:39:32.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34883.941443 secs 2023-03-10T00:39:33.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35345.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:39:34.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:34 smithi122 ceph-mon[93869]: pgmap v19116: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-10T00:39:34.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:34 smithi181 ceph-mon[94189]: pgmap v19116: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-10T00:39:34.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:34 smithi038 ceph-mon[92422]: pgmap v19116: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-10T00:39:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34886.477717 secs 2023-03-10T00:39:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34886.477717 secs 2023-03-10T00:39:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34886.477717 secs 2023-03-10T00:39:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:36 smithi038 ceph-mon[92422]: pgmap v19117: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 132 KiB/s wr, 8 op/s 2023-03-10T00:39:37.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:36 smithi122 ceph-mon[93869]: pgmap v19117: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 132 KiB/s wr, 8 op/s 2023-03-10T00:39:37.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:36 smithi181 ceph-mon[94189]: pgmap v19117: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 132 KiB/s wr, 8 op/s 2023-03-10T00:39:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:39:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:39:38.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34888.941501 secs 2023-03-10T00:39:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:39:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:39:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34888.941501 secs 2023-03-10T00:39:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:39:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:39:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34888.941501 secs 2023-03-10T00:39:38.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35350.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:39:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:38 smithi038 ceph-mon[92422]: pgmap v19118: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 132 KiB/s wr, 7 op/s 2023-03-10T00:39:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:38 smithi122 ceph-mon[93869]: pgmap v19118: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 132 KiB/s wr, 7 op/s 2023-03-10T00:39:39.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:38 smithi181 ceph-mon[94189]: pgmap v19118: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 132 KiB/s wr, 7 op/s 2023-03-10T00:39:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:39 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:39:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:39 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:39:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:39 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:39:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:39 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:39:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:39 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:39:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:39 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:39:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:40 smithi122 ceph-mon[93869]: pgmap v19119: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 132 KiB/s wr, 6 op/s 2023-03-10T00:39:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34891.477774 secs 2023-03-10T00:39:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:40 smithi181 ceph-mon[94189]: pgmap v19119: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 132 KiB/s wr, 6 op/s 2023-03-10T00:39:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34891.477774 secs 2023-03-10T00:39:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:40 smithi038 ceph-mon[92422]: pgmap v19119: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 132 KiB/s wr, 6 op/s 2023-03-10T00:39:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34891.477774 secs 2023-03-10T00:39:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:43 smithi122 ceph-mon[93869]: pgmap v19120: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 269 KiB/s wr, 9 op/s 2023-03-10T00:39:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34893.941559 secs 2023-03-10T00:39:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:43 smithi181 ceph-mon[94189]: pgmap v19120: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 269 KiB/s wr, 9 op/s 2023-03-10T00:39:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34893.941559 secs 2023-03-10T00:39:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:43 smithi038 ceph-mon[92422]: pgmap v19120: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 269 KiB/s wr, 9 op/s 2023-03-10T00:39:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34893.941559 secs 2023-03-10T00:39:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35355.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:39:44.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:44 smithi122 ceph-mon[93869]: pgmap v19121: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 269 KiB/s wr, 6 op/s 2023-03-10T00:39:44.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:44 smithi181 ceph-mon[94189]: pgmap v19121: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 269 KiB/s wr, 6 op/s 2023-03-10T00:39:44.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:44 smithi038 ceph-mon[92422]: pgmap v19121: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 269 KiB/s wr, 6 op/s 2023-03-10T00:39:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34896.477833 secs 2023-03-10T00:39:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34896.477833 secs 2023-03-10T00:39:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34896.477833 secs 2023-03-10T00:39:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:46 smithi122 ceph-mon[93869]: pgmap v19122: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 440 KiB/s rd, 404 KiB/s wr, 10 op/s 2023-03-10T00:39:46.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:46 smithi181 ceph-mon[94189]: pgmap v19122: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 440 KiB/s rd, 404 KiB/s wr, 10 op/s 2023-03-10T00:39:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:46 smithi038 ceph-mon[92422]: pgmap v19122: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 440 KiB/s rd, 404 KiB/s wr, 10 op/s 2023-03-10T00:39:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35360.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:39:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:49 smithi122 ceph-mon[93869]: pgmap v19123: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 272 KiB/s wr, 7 op/s 2023-03-10T00:39:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:49 smithi181 ceph-mon[94189]: pgmap v19123: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 272 KiB/s wr, 7 op/s 2023-03-10T00:39:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:49 smithi038 ceph-mon[92422]: pgmap v19123: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 272 KiB/s wr, 7 op/s 2023-03-10T00:39:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:50 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:39:50.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:50 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:39:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:50 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:39:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:51 smithi122 ceph-mon[93869]: pgmap v19124: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 272 KiB/s wr, 6 op/s 2023-03-10T00:39:51.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34901.477891 secs 2023-03-10T00:39:51.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:51 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:39:51.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:39:51.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:39:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:51 smithi181 ceph-mon[94189]: pgmap v19124: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 272 KiB/s wr, 6 op/s 2023-03-10T00:39:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34901.477891 secs 2023-03-10T00:39:51.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:51 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:39:51.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:39:51.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:39:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:51 smithi038 ceph-mon[92422]: pgmap v19124: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 272 KiB/s wr, 6 op/s 2023-03-10T00:39:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34901.477891 secs 2023-03-10T00:39:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:51 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:39:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:39:51.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:39:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:53 smithi122 ceph-mon[93869]: pgmap v19125: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 825 KiB/s rd, 699 KiB/s wr, 10 op/s 2023-03-10T00:39:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:39:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:39:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:53 smithi181 ceph-mon[94189]: pgmap v19125: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 825 KiB/s rd, 699 KiB/s wr, 10 op/s 2023-03-10T00:39:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:39:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:39:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:53 smithi038 ceph-mon[92422]: pgmap v19125: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 825 KiB/s rd, 699 KiB/s wr, 10 op/s 2023-03-10T00:39:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:39:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:39:53.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35365.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:39:54.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:54 smithi181 ceph-mon[94189]: pgmap v19126: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 686 KiB/s rd, 562 KiB/s wr, 8 op/s 2023-03-10T00:39:54.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:54 smithi038 ceph-mon[92422]: pgmap v19126: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 686 KiB/s rd, 562 KiB/s wr, 8 op/s 2023-03-10T00:39:55.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:54 smithi122 ceph-mon[93869]: pgmap v19126: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 686 KiB/s rd, 562 KiB/s wr, 8 op/s 2023-03-10T00:39:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:39:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:39:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34906.477950 secs 2023-03-10T00:39:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:39:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:39:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34906.477950 secs 2023-03-10T00:39:56.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:39:56.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:39:56.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34906.477950 secs 2023-03-10T00:39:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:56 smithi038 ceph-mon[92422]: pgmap v19127: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 881 KiB/s rd, 758 KiB/s wr, 12 op/s 2023-03-10T00:39:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:56 smithi122 ceph-mon[93869]: pgmap v19127: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 881 KiB/s rd, 758 KiB/s wr, 12 op/s 2023-03-10T00:39:57.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:56 smithi181 ceph-mon[94189]: pgmap v19127: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 881 KiB/s rd, 758 KiB/s wr, 12 op/s 2023-03-10T00:39:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:39:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:39:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:57 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 34908.941830 secs 2023-03-10T00:39:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:39:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:39:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:57 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 34908.941830 secs 2023-03-10T00:39:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:39:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:39:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:57 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 34908.941830 secs 2023-03-10T00:39:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35370.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:39:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:58 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:39:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:58 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:39:59.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:58 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:39:59.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:58 smithi122 ceph-mon[93869]: pgmap v19128: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 624 KiB/s wr, 9 op/s 2023-03-10T00:39:59.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:58 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:39:59.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:58 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:39:59.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:58 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:39:59.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:58 smithi181 ceph-mon[94189]: pgmap v19128: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 624 KiB/s wr, 9 op/s 2023-03-10T00:39:59.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:58 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:39:59.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:58 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:39:59.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:58 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:39:59.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:58 smithi038 ceph-mon[92422]: pgmap v19128: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 624 KiB/s wr, 9 op/s 2023-03-10T00:40:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:40:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:39:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:40:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:40:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:39:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:40:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:40:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:39:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:40:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:00 smithi122 ceph-mon[93869]: overall HEALTH_WARN 2 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-10T00:40:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:00 smithi122 ceph-mon[93869]: pgmap v19129: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 724 KiB/s rd, 625 KiB/s wr, 9 op/s 2023-03-10T00:40:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34911.478038 secs 2023-03-10T00:40:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:00 smithi181 ceph-mon[94189]: overall HEALTH_WARN 2 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-10T00:40:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:00 smithi181 ceph-mon[94189]: pgmap v19129: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 724 KiB/s rd, 625 KiB/s wr, 9 op/s 2023-03-10T00:40:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34911.478038 secs 2023-03-10T00:40:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:00 smithi038 ceph-mon[92422]: overall HEALTH_WARN 2 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-10T00:40:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:00 smithi038 ceph-mon[92422]: pgmap v19129: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 724 KiB/s rd, 625 KiB/s wr, 9 op/s 2023-03-10T00:40:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34911.478038 secs 2023-03-10T00:40:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:40:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:40:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:40:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:40:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:40:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:40:03.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:02 smithi122 ceph-mon[93869]: pgmap v19130: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 997 KiB/s wr, 15 op/s 2023-03-10T00:40:03.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34913.941888 secs 2023-03-10T00:40:03.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:02 smithi181 ceph-mon[94189]: pgmap v19130: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 997 KiB/s wr, 15 op/s 2023-03-10T00:40:03.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34913.941888 secs 2023-03-10T00:40:03.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:02 smithi038 ceph-mon[92422]: pgmap v19130: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 997 KiB/s wr, 15 op/s 2023-03-10T00:40:03.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34913.941888 secs 2023-03-10T00:40:03.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35375.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:40:04.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:04 smithi122 ceph-mon[93869]: pgmap v19131: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 584 KiB/s rd, 570 KiB/s wr, 12 op/s 2023-03-10T00:40:04.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:04 smithi181 ceph-mon[94189]: pgmap v19131: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 584 KiB/s rd, 570 KiB/s wr, 12 op/s 2023-03-10T00:40:04.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:04 smithi038 ceph-mon[92422]: pgmap v19131: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 584 KiB/s rd, 570 KiB/s wr, 12 op/s 2023-03-10T00:40:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34916.478095 secs 2023-03-10T00:40:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34916.478095 secs 2023-03-10T00:40:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34916.478095 secs 2023-03-10T00:40:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:06 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:40:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:40:07.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:06 smithi038 ceph-mon[92422]: pgmap v19132: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 734 KiB/s rd, 645 KiB/s wr, 18 op/s 2023-03-10T00:40:07.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:06 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:40:07.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:40:07.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:06 smithi122 ceph-mon[93869]: pgmap v19132: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 734 KiB/s rd, 645 KiB/s wr, 18 op/s 2023-03-10T00:40:07.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:06 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:40:07.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:40:07.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:06 smithi181 ceph-mon[94189]: pgmap v19132: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 734 KiB/s rd, 645 KiB/s wr, 18 op/s 2023-03-10T00:40:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34918.941946 secs 2023-03-10T00:40:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34918.941946 secs 2023-03-10T00:40:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34918.941946 secs 2023-03-10T00:40:08.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35380.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:40:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:08 smithi122 ceph-mon[93869]: pgmap v19133: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 874 KiB/s rd, 799 KiB/s wr, 27 op/s 2023-03-10T00:40:09.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:08 smithi181 ceph-mon[94189]: pgmap v19133: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 874 KiB/s rd, 799 KiB/s wr, 27 op/s 2023-03-10T00:40:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:08 smithi038 ceph-mon[92422]: pgmap v19133: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 874 KiB/s rd, 799 KiB/s wr, 27 op/s 2023-03-10T00:40:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:10 smithi122 ceph-mon[93869]: osdmap e1411: 12 total, 12 up, 12 in 2023-03-10T00:40:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:10 smithi122 ceph-mon[93869]: pgmap v19135: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 958 KiB/s wr, 31 op/s 2023-03-10T00:40:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34921.478153 secs 2023-03-10T00:40:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:10 smithi181 ceph-mon[94189]: osdmap e1411: 12 total, 12 up, 12 in 2023-03-10T00:40:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:10 smithi181 ceph-mon[94189]: pgmap v19135: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 958 KiB/s wr, 31 op/s 2023-03-10T00:40:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34921.478153 secs 2023-03-10T00:40:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:10 smithi038 ceph-mon[92422]: osdmap e1411: 12 total, 12 up, 12 in 2023-03-10T00:40:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:10 smithi038 ceph-mon[92422]: pgmap v19135: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 958 KiB/s wr, 31 op/s 2023-03-10T00:40:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34921.478153 secs 2023-03-10T00:40:12.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:11 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:40:12.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:11 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:40:12.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:11 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:40:12.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:11 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:40:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:11 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:40:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:11 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:40:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:12 smithi122 ceph-mon[93869]: pgmap v19136: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 755 KiB/s rd, 682 KiB/s wr, 29 op/s 2023-03-10T00:40:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34923.942092 secs 2023-03-10T00:40:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:12 smithi181 ceph-mon[94189]: pgmap v19136: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 755 KiB/s rd, 682 KiB/s wr, 29 op/s 2023-03-10T00:40:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34923.942092 secs 2023-03-10T00:40:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:12 smithi038 ceph-mon[92422]: pgmap v19136: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 755 KiB/s rd, 682 KiB/s wr, 29 op/s 2023-03-10T00:40:13.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34923.942092 secs 2023-03-10T00:40:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35385.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:40:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:40:14.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:40:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:40:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:40:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:40:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:40:15.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:14 smithi122 ceph-mon[93869]: pgmap v19137: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 770 KiB/s rd, 697 KiB/s wr, 28 op/s 2023-03-10T00:40:15.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:14 smithi181 ceph-mon[94189]: pgmap v19137: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 770 KiB/s rd, 697 KiB/s wr, 28 op/s 2023-03-10T00:40:15.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:14 smithi038 ceph-mon[92422]: pgmap v19137: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 770 KiB/s rd, 697 KiB/s wr, 28 op/s 2023-03-10T00:40:16.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:15 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:40:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:15 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:40:16.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34926.478258 secs 2023-03-10T00:40:16.233 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:15 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:40:16.233 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:15 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:40:16.233 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34926.478258 secs 2023-03-10T00:40:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:15 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:40:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:15 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:40:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34926.478258 secs 2023-03-10T00:40:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:16 smithi122 ceph-mon[93869]: pgmap v19138: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 722 KiB/s rd, 738 KiB/s wr, 24 op/s 2023-03-10T00:40:17.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:16 smithi181 ceph-mon[94189]: pgmap v19138: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 722 KiB/s rd, 738 KiB/s wr, 24 op/s 2023-03-10T00:40:17.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:16 smithi038 ceph-mon[92422]: pgmap v19138: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 722 KiB/s rd, 738 KiB/s wr, 24 op/s 2023-03-10T00:40:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:40:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:40:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34928.942114 secs 2023-03-10T00:40:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:40:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:40:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34928.942114 secs 2023-03-10T00:40:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:40:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:40:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34928.942114 secs 2023-03-10T00:40:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35390.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:40:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:18 smithi122 ceph-mon[93869]: pgmap v19139: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 318 KiB/s wr, 10 op/s 2023-03-10T00:40:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:18 smithi122 ceph-mon[93869]: osdmap e1412: 12 total, 12 up, 12 in 2023-03-10T00:40:19.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:18 smithi181 ceph-mon[94189]: pgmap v19139: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 318 KiB/s wr, 10 op/s 2023-03-10T00:40:19.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:18 smithi181 ceph-mon[94189]: osdmap e1412: 12 total, 12 up, 12 in 2023-03-10T00:40:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:18 smithi038 ceph-mon[92422]: pgmap v19139: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 318 KiB/s wr, 10 op/s 2023-03-10T00:40:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:18 smithi038 ceph-mon[92422]: osdmap e1412: 12 total, 12 up, 12 in 2023-03-10T00:40:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:21 smithi122 ceph-mon[93869]: pgmap v19141: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 318 KiB/s wr, 10 op/s 2023-03-10T00:40:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34931.478316 secs 2023-03-10T00:40:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:21 smithi181 ceph-mon[94189]: pgmap v19141: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 318 KiB/s wr, 10 op/s 2023-03-10T00:40:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34931.478316 secs 2023-03-10T00:40:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:21 smithi038 ceph-mon[92422]: pgmap v19141: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 318 KiB/s wr, 10 op/s 2023-03-10T00:40:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34931.478316 secs 2023-03-10T00:40:23.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:23 smithi122 ceph-mon[93869]: pgmap v19142: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 147 KiB/s wr, 9 op/s 2023-03-10T00:40:23.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34933.942177 secs 2023-03-10T00:40:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:23 smithi181 ceph-mon[94189]: pgmap v19142: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 147 KiB/s wr, 9 op/s 2023-03-10T00:40:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34933.942177 secs 2023-03-10T00:40:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:23 smithi038 ceph-mon[92422]: pgmap v19142: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 147 KiB/s wr, 9 op/s 2023-03-10T00:40:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34933.942177 secs 2023-03-10T00:40:23.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35395.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:40:24.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:24 smithi038 ceph-mon[92422]: pgmap v19143: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 131 KiB/s wr, 9 op/s 2023-03-10T00:40:25.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:24 smithi122 ceph-mon[93869]: pgmap v19143: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 131 KiB/s wr, 9 op/s 2023-03-10T00:40:25.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:24 smithi181 ceph-mon[94189]: pgmap v19143: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 131 KiB/s wr, 9 op/s 2023-03-10T00:40:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34936.478372 secs 2023-03-10T00:40:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:40:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34936.478372 secs 2023-03-10T00:40:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:40:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34936.478372 secs 2023-03-10T00:40:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:40:26.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:26 smithi038 ceph-mon[92422]: pgmap v19144: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 204 B/s wr, 8 op/s 2023-03-10T00:40:26.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:26 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:40:26.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:26 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:40:27.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:26 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:40:27.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:26 smithi122 ceph-mon[93869]: pgmap v19144: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 204 B/s wr, 8 op/s 2023-03-10T00:40:27.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:26 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:40:27.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:26 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:40:27.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:26 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:40:27.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:26 smithi181 ceph-mon[94189]: pgmap v19144: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 204 B/s wr, 8 op/s 2023-03-10T00:40:27.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:26 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:40:27.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:26 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:40:27.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:26 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:40:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34938.942322 secs 2023-03-10T00:40:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34938.942322 secs 2023-03-10T00:40:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34938.942322 secs 2023-03-10T00:40:28.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35400.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:40:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:28 smithi122 ceph-mon[93869]: pgmap v19145: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 8 op/s 2023-03-10T00:40:29.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:28 smithi181 ceph-mon[94189]: pgmap v19145: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 8 op/s 2023-03-10T00:40:29.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:28 smithi038 ceph-mon[92422]: pgmap v19145: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 8 op/s 2023-03-10T00:40:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:31 smithi122 ceph-mon[93869]: pgmap v19146: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-10T00:40:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34941.478455 secs 2023-03-10T00:40:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:31 smithi181 ceph-mon[94189]: pgmap v19146: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-10T00:40:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34941.478455 secs 2023-03-10T00:40:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:31 smithi038 ceph-mon[92422]: pgmap v19146: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-10T00:40:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34941.478455 secs 2023-03-10T00:40:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:33 smithi122 ceph-mon[93869]: pgmap v19147: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 9 op/s 2023-03-10T00:40:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34943.942391 secs 2023-03-10T00:40:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:33 smithi181 ceph-mon[94189]: pgmap v19147: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 9 op/s 2023-03-10T00:40:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34943.942391 secs 2023-03-10T00:40:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:33 smithi038 ceph-mon[92422]: pgmap v19147: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 0 B/s wr, 9 op/s 2023-03-10T00:40:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34943.942391 secs 2023-03-10T00:40:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35405.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:40:34.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:34 smithi038 ceph-mon[92422]: pgmap v19148: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 7 op/s 2023-03-10T00:40:35.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:34 smithi122 ceph-mon[93869]: pgmap v19148: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 7 op/s 2023-03-10T00:40:35.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:34 smithi181 ceph-mon[94189]: pgmap v19148: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 7 op/s 2023-03-10T00:40:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34946.478512 secs 2023-03-10T00:40:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34946.478512 secs 2023-03-10T00:40:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34946.478512 secs 2023-03-10T00:40:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:36 smithi038 ceph-mon[92422]: pgmap v19149: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 341 B/s wr, 9 op/s 2023-03-10T00:40:37.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:36 smithi122 ceph-mon[93869]: pgmap v19149: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 341 B/s wr, 9 op/s 2023-03-10T00:40:37.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:36 smithi181 ceph-mon[94189]: pgmap v19149: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 341 B/s wr, 9 op/s 2023-03-10T00:40:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34948.942491 secs 2023-03-10T00:40:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34948.942491 secs 2023-03-10T00:40:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34948.942491 secs 2023-03-10T00:40:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35410.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:40:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:38 smithi122 ceph-mon[93869]: pgmap v19150: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 7 op/s 2023-03-10T00:40:39.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:38 smithi181 ceph-mon[94189]: pgmap v19150: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 7 op/s 2023-03-10T00:40:39.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:38 smithi038 ceph-mon[92422]: pgmap v19150: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 7 op/s 2023-03-10T00:40:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:41 smithi122 ceph-mon[93869]: pgmap v19151: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 7 op/s 2023-03-10T00:40:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34951.478570 secs 2023-03-10T00:40:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:41 smithi181 ceph-mon[94189]: pgmap v19151: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 7 op/s 2023-03-10T00:40:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34951.478570 secs 2023-03-10T00:40:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:41 smithi038 ceph-mon[92422]: pgmap v19151: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 7 op/s 2023-03-10T00:40:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34951.478570 secs 2023-03-10T00:40:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:43 smithi122 ceph-mon[93869]: pgmap v19152: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 138 KiB/s wr, 10 op/s 2023-03-10T00:40:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34953.942548 secs 2023-03-10T00:40:43.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:43 smithi181 ceph-mon[94189]: pgmap v19152: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 138 KiB/s wr, 10 op/s 2023-03-10T00:40:43.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34953.942548 secs 2023-03-10T00:40:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:43 smithi038 ceph-mon[92422]: pgmap v19152: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 138 KiB/s wr, 10 op/s 2023-03-10T00:40:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34953.942548 secs 2023-03-10T00:40:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35415.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:40:44.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:44 smithi038 ceph-mon[92422]: pgmap v19153: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 138 KiB/s wr, 7 op/s 2023-03-10T00:40:45.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:44 smithi122 ceph-mon[93869]: pgmap v19153: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 138 KiB/s wr, 7 op/s 2023-03-10T00:40:45.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:44 smithi181 ceph-mon[94189]: pgmap v19153: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 138 KiB/s wr, 7 op/s 2023-03-10T00:40:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34956.478630 secs 2023-03-10T00:40:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34956.478630 secs 2023-03-10T00:40:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34956.478630 secs 2023-03-10T00:40:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:46 smithi122 ceph-mon[93869]: pgmap v19154: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 273 KiB/s wr, 10 op/s 2023-03-10T00:40:47.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:46 smithi181 ceph-mon[94189]: pgmap v19154: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 273 KiB/s wr, 10 op/s 2023-03-10T00:40:47.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:46 smithi038 ceph-mon[92422]: pgmap v19154: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 273 KiB/s wr, 10 op/s 2023-03-10T00:40:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34958.942647 secs 2023-03-10T00:40:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34958.942647 secs 2023-03-10T00:40:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34958.942647 secs 2023-03-10T00:40:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35420.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:40:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:48 smithi122 ceph-mon[93869]: pgmap v19155: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 273 KiB/s wr, 7 op/s 2023-03-10T00:40:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:48 smithi181 ceph-mon[94189]: pgmap v19155: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 273 KiB/s wr, 7 op/s 2023-03-10T00:40:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:48 smithi038 ceph-mon[92422]: pgmap v19155: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 273 KiB/s wr, 7 op/s 2023-03-10T00:40:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:40:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:40:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:40:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:40:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:40:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:40:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:50 smithi122 ceph-mon[93869]: pgmap v19156: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 273 KiB/s wr, 7 op/s 2023-03-10T00:40:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34961.478710 secs 2023-03-10T00:40:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:50 smithi181 ceph-mon[94189]: pgmap v19156: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 273 KiB/s wr, 7 op/s 2023-03-10T00:40:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34961.478710 secs 2023-03-10T00:40:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:50 smithi038 ceph-mon[92422]: pgmap v19156: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 273 KiB/s wr, 7 op/s 2023-03-10T00:40:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34961.478710 secs 2023-03-10T00:40:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:40:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:40:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:40:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:40:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:40:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:40:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:52 smithi122 ceph-mon[93869]: pgmap v19157: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 742 KiB/s rd, 738 KiB/s wr, 12 op/s 2023-03-10T00:40:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:52 smithi181 ceph-mon[94189]: pgmap v19157: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 742 KiB/s rd, 738 KiB/s wr, 12 op/s 2023-03-10T00:40:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:52 smithi038 ceph-mon[92422]: pgmap v19157: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 742 KiB/s rd, 738 KiB/s wr, 12 op/s 2023-03-10T00:40:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35425.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:40:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:40:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:40:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:53 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:40:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:40:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:40:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:53 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:40:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:40:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:40:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:53 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:40:55.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:54 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:40:55.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:54 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:40:55.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:54 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:40:55.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:54 smithi122 ceph-mon[93869]: pgmap v19158: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 604 KiB/s rd, 600 KiB/s wr, 9 op/s 2023-03-10T00:40:55.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:54 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:40:55.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:54 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:40:55.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:54 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:40:55.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:54 smithi181 ceph-mon[94189]: pgmap v19158: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 604 KiB/s rd, 600 KiB/s wr, 9 op/s 2023-03-10T00:40:55.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:54 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:40:55.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:54 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:40:55.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:54 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:40:55.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:54 smithi038 ceph-mon[92422]: pgmap v19158: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 604 KiB/s rd, 600 KiB/s wr, 9 op/s 2023-03-10T00:40:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:40:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:40:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34966.478742 secs 2023-03-10T00:40:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:40:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:40:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34966.478742 secs 2023-03-10T00:40:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:40:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:40:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34966.478742 secs 2023-03-10T00:40:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:56 smithi122 ceph-mon[93869]: pgmap v19159: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 695 KiB/s wr, 13 op/s 2023-03-10T00:40:57.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:56 smithi181 ceph-mon[94189]: pgmap v19159: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 695 KiB/s wr, 13 op/s 2023-03-10T00:40:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:56 smithi038 ceph-mon[92422]: pgmap v19159: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 695 KiB/s wr, 13 op/s 2023-03-10T00:40:58.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35430.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:40:59.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:40:59 smithi122 ceph-mon[93869]: pgmap v19160: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 573 KiB/s rd, 795 KiB/s wr, 11 op/s 2023-03-10T00:40:59.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:40:59 smithi181 ceph-mon[94189]: pgmap v19160: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 573 KiB/s rd, 795 KiB/s wr, 11 op/s 2023-03-10T00:40:59.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:40:59 smithi038 ceph-mon[92422]: pgmap v19160: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 573 KiB/s rd, 795 KiB/s wr, 11 op/s 2023-03-10T00:41:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:41:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:41:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:41:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:41:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:41:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:41:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:01 smithi122 ceph-mon[93869]: pgmap v19161: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 573 KiB/s rd, 795 KiB/s wr, 10 op/s 2023-03-10T00:41:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34971.478821 secs 2023-03-10T00:41:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:41:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:41:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:01 smithi181 ceph-mon[94189]: pgmap v19161: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 573 KiB/s rd, 795 KiB/s wr, 10 op/s 2023-03-10T00:41:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34971.478821 secs 2023-03-10T00:41:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:41:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:41:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:01 smithi038 ceph-mon[92422]: pgmap v19161: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 573 KiB/s rd, 795 KiB/s wr, 10 op/s 2023-03-10T00:41:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34971.478821 secs 2023-03-10T00:41:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:41:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:41:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:03 smithi122 ceph-mon[93869]: pgmap v19162: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 910 KiB/s rd, 929 KiB/s wr, 15 op/s 2023-03-10T00:41:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:03 smithi181 ceph-mon[94189]: pgmap v19162: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 910 KiB/s rd, 929 KiB/s wr, 15 op/s 2023-03-10T00:41:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:03 smithi038 ceph-mon[92422]: pgmap v19162: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 910 KiB/s rd, 929 KiB/s wr, 15 op/s 2023-03-10T00:41:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35435.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:41:04.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:04 smithi038 ceph-mon[92422]: pgmap v19163: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 791 KiB/s rd, 582 KiB/s wr, 23 op/s 2023-03-10T00:41:05.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:04 smithi122 ceph-mon[93869]: pgmap v19163: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 791 KiB/s rd, 582 KiB/s wr, 23 op/s 2023-03-10T00:41:05.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:04 smithi181 ceph-mon[94189]: pgmap v19163: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 791 KiB/s rd, 582 KiB/s wr, 23 op/s 2023-03-10T00:41:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34976.478911 secs 2023-03-10T00:41:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34976.478911 secs 2023-03-10T00:41:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34976.478911 secs 2023-03-10T00:41:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:06 smithi122 ceph-mon[93869]: pgmap v19164: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 873 KiB/s wr, 29 op/s 2023-03-10T00:41:07.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:06 smithi181 ceph-mon[94189]: pgmap v19164: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 873 KiB/s wr, 29 op/s 2023-03-10T00:41:07.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:06 smithi038 ceph-mon[92422]: pgmap v19164: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 873 KiB/s wr, 29 op/s 2023-03-10T00:41:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:41:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:41:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34978.942973 secs 2023-03-10T00:41:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:41:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:41:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34978.942973 secs 2023-03-10T00:41:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:41:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:41:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34978.942973 secs 2023-03-10T00:41:08.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35440.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:41:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:08 smithi122 ceph-mon[93869]: pgmap v19165: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 984 KiB/s rd, 817 KiB/s wr, 27 op/s 2023-03-10T00:41:09.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:08 smithi181 ceph-mon[94189]: pgmap v19165: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 984 KiB/s rd, 817 KiB/s wr, 27 op/s 2023-03-10T00:41:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:08 smithi038 ceph-mon[92422]: pgmap v19165: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 984 KiB/s rd, 817 KiB/s wr, 27 op/s 2023-03-10T00:41:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:41:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:41:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:41:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:41:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:41:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:41:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:10 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:41:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:10 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:41:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:10 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:41:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:10 smithi122 ceph-mon[93869]: osdmap e1413: 12 total, 12 up, 12 in 2023-03-10T00:41:11.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:10 smithi122 ceph-mon[93869]: pgmap v19167: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 698 KiB/s wr, 31 op/s 2023-03-10T00:41:11.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34981.480003 secs 2023-03-10T00:41:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:10 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:41:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:10 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:41:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:10 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:41:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:10 smithi181 ceph-mon[94189]: osdmap e1413: 12 total, 12 up, 12 in 2023-03-10T00:41:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:10 smithi181 ceph-mon[94189]: pgmap v19167: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 698 KiB/s wr, 31 op/s 2023-03-10T00:41:11.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34981.480003 secs 2023-03-10T00:41:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:10 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:41:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:10 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:41:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:10 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:41:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:10 smithi038 ceph-mon[92422]: osdmap e1413: 12 total, 12 up, 12 in 2023-03-10T00:41:11.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:10 smithi038 ceph-mon[92422]: pgmap v19167: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 698 KiB/s wr, 31 op/s 2023-03-10T00:41:11.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34981.480003 secs 2023-03-10T00:41:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:13 smithi122 ceph-mon[93869]: pgmap v19168: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 895 KiB/s wr, 30 op/s 2023-03-10T00:41:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34983.943065 secs 2023-03-10T00:41:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:13 smithi181 ceph-mon[94189]: pgmap v19168: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 895 KiB/s wr, 30 op/s 2023-03-10T00:41:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34983.943065 secs 2023-03-10T00:41:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:13 smithi038 ceph-mon[92422]: pgmap v19168: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 895 KiB/s wr, 30 op/s 2023-03-10T00:41:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34983.943065 secs 2023-03-10T00:41:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35445.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:41:15.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:14 smithi122 ceph-mon[93869]: pgmap v19169: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 755 KiB/s wr, 15 op/s 2023-03-10T00:41:15.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:14 smithi181 ceph-mon[94189]: pgmap v19169: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 755 KiB/s wr, 15 op/s 2023-03-10T00:41:15.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:14 smithi038 ceph-mon[92422]: pgmap v19169: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 755 KiB/s wr, 15 op/s 2023-03-10T00:41:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34986.480064 secs 2023-03-10T00:41:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34986.480064 secs 2023-03-10T00:41:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34986.480064 secs 2023-03-10T00:41:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:16 smithi122 ceph-mon[93869]: pgmap v19170: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 452 KiB/s wr, 10 op/s 2023-03-10T00:41:17.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:16 smithi181 ceph-mon[94189]: pgmap v19170: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 452 KiB/s wr, 10 op/s 2023-03-10T00:41:17.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:16 smithi038 ceph-mon[92422]: pgmap v19170: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 452 KiB/s wr, 10 op/s 2023-03-10T00:41:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34988.943119 secs 2023-03-10T00:41:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34988.943119 secs 2023-03-10T00:41:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34988.943119 secs 2023-03-10T00:41:18.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35450.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:41:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:18 smithi122 ceph-mon[93869]: pgmap v19171: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 405 KiB/s wr, 8 op/s 2023-03-10T00:41:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:18 smithi122 ceph-mon[93869]: osdmap e1414: 12 total, 12 up, 12 in 2023-03-10T00:41:19.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:18 smithi181 ceph-mon[94189]: pgmap v19171: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 405 KiB/s wr, 8 op/s 2023-03-10T00:41:19.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:18 smithi181 ceph-mon[94189]: osdmap e1414: 12 total, 12 up, 12 in 2023-03-10T00:41:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:18 smithi038 ceph-mon[92422]: pgmap v19171: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 405 KiB/s wr, 8 op/s 2023-03-10T00:41:19.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:18 smithi038 ceph-mon[92422]: osdmap e1414: 12 total, 12 up, 12 in 2023-03-10T00:41:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34991.480123 secs 2023-03-10T00:41:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:21 smithi122 ceph-mon[93869]: pgmap v19173: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 405 KiB/s wr, 8 op/s 2023-03-10T00:41:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34991.480123 secs 2023-03-10T00:41:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:21 smithi181 ceph-mon[94189]: pgmap v19173: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 405 KiB/s wr, 8 op/s 2023-03-10T00:41:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34991.480123 secs 2023-03-10T00:41:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:21 smithi038 ceph-mon[92422]: pgmap v19173: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 405 KiB/s wr, 8 op/s 2023-03-10T00:41:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:23 smithi122 ceph-mon[93869]: pgmap v19174: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 529 KiB/s rd, 293 KiB/s wr, 7 op/s 2023-03-10T00:41:23.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34993.943178 secs 2023-03-10T00:41:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:23 smithi181 ceph-mon[94189]: pgmap v19174: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 529 KiB/s rd, 293 KiB/s wr, 7 op/s 2023-03-10T00:41:23.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34993.943178 secs 2023-03-10T00:41:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:23 smithi038 ceph-mon[92422]: pgmap v19174: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 529 KiB/s rd, 293 KiB/s wr, 7 op/s 2023-03-10T00:41:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34993.943178 secs 2023-03-10T00:41:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35455.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:41:25.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:24 smithi122 ceph-mon[93869]: pgmap v19175: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 293 KiB/s wr, 7 op/s 2023-03-10T00:41:25.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:24 smithi181 ceph-mon[94189]: pgmap v19175: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 293 KiB/s wr, 7 op/s 2023-03-10T00:41:25.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:24 smithi038 ceph-mon[92422]: pgmap v19175: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 293 KiB/s wr, 7 op/s 2023-03-10T00:41:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34996.480185 secs 2023-03-10T00:41:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34996.480185 secs 2023-03-10T00:41:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34996.480185 secs 2023-03-10T00:41:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:26 smithi122 ceph-mon[93869]: pgmap v19176: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 246 KiB/s wr, 7 op/s 2023-03-10T00:41:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:26 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:41:27.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:26 smithi181 ceph-mon[94189]: pgmap v19176: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 246 KiB/s wr, 7 op/s 2023-03-10T00:41:27.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:26 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:41:27.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:26 smithi038 ceph-mon[92422]: pgmap v19176: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 246 KiB/s wr, 7 op/s 2023-03-10T00:41:27.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:26 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:41:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:27 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:41:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:27 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:41:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:41:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:41:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:41:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 34998.943235 secs 2023-03-10T00:41:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:27 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:41:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:27 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:41:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:41:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:41:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:41:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 34998.943235 secs 2023-03-10T00:41:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:27 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:41:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:27 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:41:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:41:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:41:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:41:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 34998.943235 secs 2023-03-10T00:41:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35460.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:41:29.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:28 smithi122 ceph-mon[93869]: pgmap v19177: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 640 KiB/s rd, 404 KiB/s wr, 7 op/s 2023-03-10T00:41:29.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:28 smithi181 ceph-mon[94189]: pgmap v19177: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 640 KiB/s rd, 404 KiB/s wr, 7 op/s 2023-03-10T00:41:29.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:28 smithi038 ceph-mon[92422]: pgmap v19177: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 640 KiB/s rd, 404 KiB/s wr, 7 op/s 2023-03-10T00:41:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:29 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:41:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:29 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:41:30.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:29 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:41:30.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:29 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:41:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:29 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:41:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:29 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:41:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35001.480270 secs 2023-03-10T00:41:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:30 smithi122 ceph-mon[93869]: pgmap v19178: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 566 KiB/s rd, 358 KiB/s wr, 6 op/s 2023-03-10T00:41:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35001.480270 secs 2023-03-10T00:41:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:30 smithi181 ceph-mon[94189]: pgmap v19178: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 566 KiB/s rd, 358 KiB/s wr, 6 op/s 2023-03-10T00:41:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35001.480270 secs 2023-03-10T00:41:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:30 smithi038 ceph-mon[92422]: pgmap v19178: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 566 KiB/s rd, 358 KiB/s wr, 6 op/s 2023-03-10T00:41:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:33 smithi122 ceph-mon[93869]: pgmap v19179: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 535 KiB/s rd, 337 KiB/s wr, 9 op/s 2023-03-10T00:41:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35003.943294 secs 2023-03-10T00:41:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:33 smithi181 ceph-mon[94189]: pgmap v19179: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 535 KiB/s rd, 337 KiB/s wr, 9 op/s 2023-03-10T00:41:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35003.943294 secs 2023-03-10T00:41:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:33 smithi038 ceph-mon[92422]: pgmap v19179: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 535 KiB/s rd, 337 KiB/s wr, 9 op/s 2023-03-10T00:41:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35003.943294 secs 2023-03-10T00:41:33.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35465.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:41:35.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:34 smithi122 ceph-mon[93869]: pgmap v19180: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 132 KiB/s wr, 5 op/s 2023-03-10T00:41:35.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:34 smithi181 ceph-mon[94189]: pgmap v19180: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 132 KiB/s wr, 5 op/s 2023-03-10T00:41:35.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:34 smithi038 ceph-mon[92422]: pgmap v19180: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 132 KiB/s wr, 5 op/s 2023-03-10T00:41:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35006.480356 secs 2023-03-10T00:41:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35006.480356 secs 2023-03-10T00:41:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35006.480356 secs 2023-03-10T00:41:37.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:36 smithi122 ceph-mon[93869]: pgmap v19181: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 132 KiB/s wr, 7 op/s 2023-03-10T00:41:37.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:36 smithi181 ceph-mon[94189]: pgmap v19181: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 132 KiB/s wr, 7 op/s 2023-03-10T00:41:37.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:36 smithi038 ceph-mon[92422]: pgmap v19181: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 132 KiB/s wr, 7 op/s 2023-03-10T00:41:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35008.943358 secs 2023-03-10T00:41:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35008.943358 secs 2023-03-10T00:41:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35008.943358 secs 2023-03-10T00:41:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35470.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:41:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:38 smithi122 ceph-mon[93869]: pgmap v19182: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 132 KiB/s wr, 6 op/s 2023-03-10T00:41:39.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:38 smithi181 ceph-mon[94189]: pgmap v19182: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 132 KiB/s wr, 6 op/s 2023-03-10T00:41:39.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:38 smithi038 ceph-mon[92422]: pgmap v19182: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 132 KiB/s wr, 6 op/s 2023-03-10T00:41:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35011.480417 secs 2023-03-10T00:41:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:41 smithi122 ceph-mon[93869]: pgmap v19183: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-10T00:41:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35011.480417 secs 2023-03-10T00:41:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:41 smithi181 ceph-mon[94189]: pgmap v19183: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-10T00:41:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35011.480417 secs 2023-03-10T00:41:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:41 smithi038 ceph-mon[92422]: pgmap v19183: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-10T00:41:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:43 smithi122 ceph-mon[93869]: pgmap v19184: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2023-03-10T00:41:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35013.943419 secs 2023-03-10T00:41:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:43 smithi181 ceph-mon[94189]: pgmap v19184: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2023-03-10T00:41:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35013.943419 secs 2023-03-10T00:41:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:43 smithi038 ceph-mon[92422]: pgmap v19184: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 7 op/s 2023-03-10T00:41:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35013.943419 secs 2023-03-10T00:41:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35475.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:41:45.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:44 smithi122 ceph-mon[93869]: pgmap v19185: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-10T00:41:45.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:44 smithi181 ceph-mon[94189]: pgmap v19185: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-10T00:41:45.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:44 smithi038 ceph-mon[92422]: pgmap v19185: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-10T00:41:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35016.480477 secs 2023-03-10T00:41:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35016.480477 secs 2023-03-10T00:41:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35016.480477 secs 2023-03-10T00:41:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:46 smithi122 ceph-mon[93869]: pgmap v19186: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 133 KiB/s wr, 7 op/s 2023-03-10T00:41:47.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:46 smithi181 ceph-mon[94189]: pgmap v19186: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 133 KiB/s wr, 7 op/s 2023-03-10T00:41:47.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:46 smithi038 ceph-mon[92422]: pgmap v19186: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 133 KiB/s wr, 7 op/s 2023-03-10T00:41:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35480.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:41:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:49 smithi122 ceph-mon[93869]: pgmap v19187: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 138 KiB/s wr, 7 op/s 2023-03-10T00:41:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:49 smithi181 ceph-mon[94189]: pgmap v19187: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 138 KiB/s wr, 7 op/s 2023-03-10T00:41:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:49 smithi038 ceph-mon[92422]: pgmap v19187: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 138 KiB/s wr, 7 op/s 2023-03-10T00:41:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:50 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:41:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:50 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:41:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:50 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:41:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35021.480569 secs 2023-03-10T00:41:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:51 smithi122 ceph-mon[93869]: pgmap v19188: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 138 KiB/s wr, 6 op/s 2023-03-10T00:41:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:51 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:41:51.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:41:51.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:41:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35021.480569 secs 2023-03-10T00:41:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:51 smithi181 ceph-mon[94189]: pgmap v19188: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 138 KiB/s wr, 6 op/s 2023-03-10T00:41:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:51 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:41:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:41:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:41:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35021.480569 secs 2023-03-10T00:41:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:51 smithi038 ceph-mon[92422]: pgmap v19188: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 138 KiB/s wr, 6 op/s 2023-03-10T00:41:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:51 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:41:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:41:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:41:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:41:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:41:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:41:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:41:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:41:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:41:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:53 smithi122 ceph-mon[93869]: pgmap v19189: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 610 KiB/s rd, 602 KiB/s wr, 11 op/s 2023-03-10T00:41:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:41:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:41:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:53 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 35023.943565 secs 2023-03-10T00:41:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:53 smithi181 ceph-mon[94189]: pgmap v19189: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 610 KiB/s rd, 602 KiB/s wr, 11 op/s 2023-03-10T00:41:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:41:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:41:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:53 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 35023.943565 secs 2023-03-10T00:41:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:53 smithi038 ceph-mon[92422]: pgmap v19189: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 610 KiB/s rd, 602 KiB/s wr, 11 op/s 2023-03-10T00:41:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:41:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:41:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:53 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 35023.943565 secs 2023-03-10T00:41:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35485.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:41:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:55 smithi122 ceph-mon[93869]: pgmap v19190: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 609 KiB/s rd, 603 KiB/s wr, 9 op/s 2023-03-10T00:41:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:41:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:41:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:55 smithi181 ceph-mon[94189]: pgmap v19190: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 609 KiB/s rd, 603 KiB/s wr, 9 op/s 2023-03-10T00:41:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:41:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:41:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:55 smithi038 ceph-mon[92422]: pgmap v19190: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 609 KiB/s rd, 603 KiB/s wr, 9 op/s 2023-03-10T00:41:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:41:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:41:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35026.480626 secs 2023-03-10T00:41:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:41:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:41:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35026.480626 secs 2023-03-10T00:41:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:41:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:41:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35026.480626 secs 2023-03-10T00:41:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:41:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:41:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:57 smithi122 ceph-mon[93869]: pgmap v19191: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 622 KiB/s rd, 603 KiB/s wr, 11 op/s 2023-03-10T00:41:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:57 smithi181 ceph-mon[94189]: pgmap v19191: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 622 KiB/s rd, 603 KiB/s wr, 11 op/s 2023-03-10T00:41:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:57 smithi038 ceph-mon[92422]: pgmap v19191: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 622 KiB/s rd, 603 KiB/s wr, 11 op/s 2023-03-10T00:41:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:58 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 35028.943676 secs 2023-03-10T00:41:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:58 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 35028.943676 secs 2023-03-10T00:41:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:58 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 35028.943676 secs 2023-03-10T00:41:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35490.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:41:59.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:59 smithi122 ceph-mon[93869]: pgmap v19192: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 862 KiB/s rd, 613 KiB/s wr, 12 op/s 2023-03-10T00:41:59.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:41:59 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:41:59.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:59 smithi181 ceph-mon[94189]: pgmap v19192: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 862 KiB/s rd, 613 KiB/s wr, 12 op/s 2023-03-10T00:41:59.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:41:59 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:41:59.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:59 smithi038 ceph-mon[92422]: pgmap v19192: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 862 KiB/s rd, 613 KiB/s wr, 12 op/s 2023-03-10T00:41:59.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:41:59 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:42:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:42:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:42:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:42:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:42:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:42:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:42:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35031.480702 secs 2023-03-10T00:42:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:01 smithi122 ceph-mon[93869]: pgmap v19193: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 854 KiB/s rd, 608 KiB/s wr, 10 op/s 2023-03-10T00:42:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:42:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:42:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35031.480702 secs 2023-03-10T00:42:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:01 smithi181 ceph-mon[94189]: pgmap v19193: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 854 KiB/s rd, 608 KiB/s wr, 10 op/s 2023-03-10T00:42:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:42:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:42:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35031.480702 secs 2023-03-10T00:42:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:01 smithi038 ceph-mon[92422]: pgmap v19193: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 854 KiB/s rd, 608 KiB/s wr, 10 op/s 2023-03-10T00:42:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:42:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:42:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:03 smithi122 ceph-mon[93869]: pgmap v19194: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 980 KiB/s wr, 15 op/s 2023-03-10T00:42:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:03 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 35033.943779 secs 2023-03-10T00:42:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:03 smithi181 ceph-mon[94189]: pgmap v19194: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 980 KiB/s wr, 15 op/s 2023-03-10T00:42:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:03 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 35033.943779 secs 2023-03-10T00:42:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:03 smithi038 ceph-mon[92422]: pgmap v19194: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 980 KiB/s wr, 15 op/s 2023-03-10T00:42:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:03 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 35033.943779 secs 2023-03-10T00:42:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35495.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:42:05.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:04 smithi122 ceph-mon[93869]: pgmap v19195: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 990 KiB/s rd, 515 KiB/s wr, 11 op/s 2023-03-10T00:42:05.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:04 smithi181 ceph-mon[94189]: pgmap v19195: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 990 KiB/s rd, 515 KiB/s wr, 11 op/s 2023-03-10T00:42:05.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:04 smithi038 ceph-mon[92422]: pgmap v19195: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 990 KiB/s rd, 515 KiB/s wr, 11 op/s 2023-03-10T00:42:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35036.480762 secs 2023-03-10T00:42:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35036.480762 secs 2023-03-10T00:42:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35036.480762 secs 2023-03-10T00:42:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:06 smithi122 ceph-mon[93869]: pgmap v19196: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 866 KiB/s wr, 16 op/s 2023-03-10T00:42:07.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:06 smithi181 ceph-mon[94189]: pgmap v19196: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 866 KiB/s wr, 16 op/s 2023-03-10T00:42:07.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:06 smithi038 ceph-mon[92422]: pgmap v19196: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 866 KiB/s wr, 16 op/s 2023-03-10T00:42:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:07 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 35038.943838 secs 2023-03-10T00:42:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:07 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 35038.943838 secs 2023-03-10T00:42:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:07 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 35038.943838 secs 2023-03-10T00:42:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35500.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:42:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:08 smithi122 ceph-mon[93869]: pgmap v19197: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 25 op/s 2023-03-10T00:42:09.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:08 smithi181 ceph-mon[94189]: pgmap v19197: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 25 op/s 2023-03-10T00:42:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:08 smithi038 ceph-mon[92422]: pgmap v19197: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 25 op/s 2023-03-10T00:42:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:42:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:42:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:42:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:42:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:42:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:42:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35041.480819 secs 2023-03-10T00:42:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:10 smithi122 ceph-mon[93869]: pgmap v19198: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 980 KiB/s wr, 22 op/s 2023-03-10T00:42:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35041.480819 secs 2023-03-10T00:42:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:10 smithi181 ceph-mon[94189]: pgmap v19198: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 980 KiB/s wr, 22 op/s 2023-03-10T00:42:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35041.480819 secs 2023-03-10T00:42:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:10 smithi038 ceph-mon[92422]: pgmap v19198: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 980 KiB/s wr, 22 op/s 2023-03-10T00:42:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:11 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:42:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:11 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:42:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:11 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:42:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:11 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:42:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:11 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:42:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:11 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:42:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:12 smithi122 ceph-mon[93869]: pgmap v19199: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 985 KiB/s wr, 26 op/s 2023-03-10T00:42:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:12 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 35043.943898 secs 2023-03-10T00:42:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:12 smithi181 ceph-mon[94189]: pgmap v19199: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 985 KiB/s wr, 26 op/s 2023-03-10T00:42:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:12 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 35043.943898 secs 2023-03-10T00:42:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:12 smithi038 ceph-mon[92422]: pgmap v19199: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 985 KiB/s wr, 26 op/s 2023-03-10T00:42:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:12 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 35043.943898 secs 2023-03-10T00:42:13.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35505.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:42:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:42:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:42:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:42:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:42:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:42:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:42:15.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:14 smithi122 ceph-mon[93869]: pgmap v19200: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 905 KiB/s rd, 722 KiB/s wr, 21 op/s 2023-03-10T00:42:15.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:14 smithi181 ceph-mon[94189]: pgmap v19200: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 905 KiB/s rd, 722 KiB/s wr, 21 op/s 2023-03-10T00:42:15.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:14 smithi038 ceph-mon[92422]: pgmap v19200: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 905 KiB/s rd, 722 KiB/s wr, 21 op/s 2023-03-10T00:42:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35046.480877 secs 2023-03-10T00:42:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35046.480877 secs 2023-03-10T00:42:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35046.480877 secs 2023-03-10T00:42:17.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:16 smithi122 ceph-mon[93869]: pgmap v19201: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 907 KiB/s rd, 724 KiB/s wr, 22 op/s 2023-03-10T00:42:17.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:16 smithi181 ceph-mon[94189]: pgmap v19201: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 907 KiB/s rd, 724 KiB/s wr, 22 op/s 2023-03-10T00:42:17.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:16 smithi038 ceph-mon[92422]: pgmap v19201: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 907 KiB/s rd, 724 KiB/s wr, 22 op/s 2023-03-10T00:42:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:17 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 35048.943956 secs 2023-03-10T00:42:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:17 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 35048.943956 secs 2023-03-10T00:42:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:17 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 35048.943956 secs 2023-03-10T00:42:18.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35510.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:42:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:18 smithi122 ceph-mon[93869]: pgmap v19202: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 505 KiB/s wr, 19 op/s 2023-03-10T00:42:19.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:18 smithi181 ceph-mon[94189]: pgmap v19202: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 505 KiB/s wr, 19 op/s 2023-03-10T00:42:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:18 smithi038 ceph-mon[92422]: pgmap v19202: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 505 KiB/s wr, 19 op/s 2023-03-10T00:42:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35051.480935 secs 2023-03-10T00:42:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:21 smithi122 ceph-mon[93869]: pgmap v19203: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 249 KiB/s wr, 8 op/s 2023-03-10T00:42:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35051.480935 secs 2023-03-10T00:42:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:21 smithi181 ceph-mon[94189]: pgmap v19203: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 249 KiB/s wr, 8 op/s 2023-03-10T00:42:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35051.480935 secs 2023-03-10T00:42:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:21 smithi038 ceph-mon[92422]: pgmap v19203: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 249 KiB/s wr, 8 op/s 2023-03-10T00:42:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:23 smithi122 ceph-mon[93869]: pgmap v19204: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 511 KiB/s rd, 277 KiB/s wr, 11 op/s 2023-03-10T00:42:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:23 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 35053.944043 secs 2023-03-10T00:42:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:23 smithi181 ceph-mon[94189]: pgmap v19204: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 511 KiB/s rd, 277 KiB/s wr, 11 op/s 2023-03-10T00:42:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:23 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 35053.944043 secs 2023-03-10T00:42:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:23 smithi038 ceph-mon[92422]: pgmap v19204: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 511 KiB/s rd, 277 KiB/s wr, 11 op/s 2023-03-10T00:42:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:23 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 35053.944043 secs 2023-03-10T00:42:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35515.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:42:25.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:24 smithi122 ceph-mon[93869]: pgmap v19205: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 505 KiB/s wr, 10 op/s 2023-03-10T00:42:25.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:24 smithi181 ceph-mon[94189]: pgmap v19205: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 505 KiB/s wr, 10 op/s 2023-03-10T00:42:25.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:24 smithi038 ceph-mon[92422]: pgmap v19205: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 505 KiB/s wr, 10 op/s 2023-03-10T00:42:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35056.481023 secs 2023-03-10T00:42:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35056.481023 secs 2023-03-10T00:42:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35056.481023 secs 2023-03-10T00:42:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:26 smithi122 ceph-mon[93869]: pgmap v19206: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 487 KiB/s wr, 13 op/s 2023-03-10T00:42:27.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:26 smithi181 ceph-mon[94189]: pgmap v19206: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 487 KiB/s wr, 13 op/s 2023-03-10T00:42:27.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:26 smithi038 ceph-mon[92422]: pgmap v19206: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 487 KiB/s wr, 13 op/s 2023-03-10T00:42:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:27 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:42:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:27 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:42:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:42:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35058.944128 secs 2023-03-10T00:42:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:27 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:42:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:27 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:42:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:42:28.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35058.944128 secs 2023-03-10T00:42:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:27 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:42:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:27 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:42:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:42:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35058.944128 secs 2023-03-10T00:42:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35520.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:42:29.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:28 smithi122 ceph-mon[93869]: pgmap v19207: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 494 KiB/s rd, 515 KiB/s wr, 13 op/s 2023-03-10T00:42:29.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:28 smithi181 ceph-mon[94189]: pgmap v19207: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 494 KiB/s rd, 515 KiB/s wr, 13 op/s 2023-03-10T00:42:29.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:28 smithi038 ceph-mon[92422]: pgmap v19207: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 494 KiB/s rd, 515 KiB/s wr, 13 op/s 2023-03-10T00:42:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:29 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:42:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:29 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:42:30.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:29 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:42:30.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:29 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:42:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:29 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:42:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:29 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:42:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:30 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:42:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:30 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:42:31.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:30 smithi122 ceph-mon[93869]: osdmap e1415: 12 total, 12 up, 12 in 2023-03-10T00:42:31.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35061.481083 secs 2023-03-10T00:42:31.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:30 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:42:31.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:30 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:42:31.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:30 smithi122 ceph-mon[93869]: pgmap v19209: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 593 KiB/s rd, 458 KiB/s wr, 14 op/s 2023-03-10T00:42:31.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:30 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:42:31.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:30 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:42:31.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:30 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:42:31.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:30 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:42:31.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:30 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:42:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:30 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:42:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:30 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:42:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:30 smithi181 ceph-mon[94189]: osdmap e1415: 12 total, 12 up, 12 in 2023-03-10T00:42:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35061.481083 secs 2023-03-10T00:42:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:30 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:42:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:30 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:42:31.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:30 smithi181 ceph-mon[94189]: pgmap v19209: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 593 KiB/s rd, 458 KiB/s wr, 14 op/s 2023-03-10T00:42:31.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:30 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:42:31.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:30 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:42:31.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:30 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:42:31.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:30 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:42:31.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:30 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:42:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:30 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:42:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:30 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:42:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:30 smithi038 ceph-mon[92422]: osdmap e1415: 12 total, 12 up, 12 in 2023-03-10T00:42:31.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35061.481083 secs 2023-03-10T00:42:31.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:30 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:42:31.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:30 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:42:31.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:30 smithi038 ceph-mon[92422]: pgmap v19209: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 593 KiB/s rd, 458 KiB/s wr, 14 op/s 2023-03-10T00:42:31.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:30 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:42:31.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:30 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:42:31.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:30 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:42:31.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:30 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:42:31.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:30 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:42:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:42:32.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:42:32.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:42:32.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:42:32.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:42:32.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:42:32.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:42:32.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:42:32.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:31 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-10T00:42:32.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:31 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-10T00:42:32.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:31 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-10T00:42:32.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:31 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-10T00:42:32.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:42:32.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:42:32.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:42:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:42:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:42:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:42:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:42:32.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:42:32.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:42:32.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:42:32.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:42:32.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:31 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-10T00:42:32.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:31 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-10T00:42:32.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:31 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-10T00:42:32.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:31 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-10T00:42:32.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:42:32.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:42:32.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:42:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:42:32.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:42:32.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:42:32.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:42:32.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:42:32.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:42:32.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:42:32.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:42:32.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:31 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-10T00:42:32.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:31 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-10T00:42:32.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:31 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-10T00:42:32.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:31 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-10T00:42:32.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:42:32.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:42:32.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:42:33.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:32 smithi122 ceph-mon[93869]: pgmap v19210: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 697 KiB/s rd, 562 KiB/s wr, 14 op/s 2023-03-10T00:42:33.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35063.944188 secs 2023-03-10T00:42:33.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:32 smithi181 ceph-mon[94189]: pgmap v19210: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 697 KiB/s rd, 562 KiB/s wr, 14 op/s 2023-03-10T00:42:33.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35063.944188 secs 2023-03-10T00:42:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:32 smithi038 ceph-mon[92422]: pgmap v19210: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 697 KiB/s rd, 562 KiB/s wr, 14 op/s 2023-03-10T00:42:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35063.944188 secs 2023-03-10T00:42:33.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35525.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:42:35.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:34 smithi122 ceph-mon[93869]: pgmap v19211: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 536 KiB/s rd, 281 KiB/s wr, 12 op/s 2023-03-10T00:42:35.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:34 smithi181 ceph-mon[94189]: pgmap v19211: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 536 KiB/s rd, 281 KiB/s wr, 12 op/s 2023-03-10T00:42:35.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:34 smithi038 ceph-mon[92422]: pgmap v19211: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 536 KiB/s rd, 281 KiB/s wr, 12 op/s 2023-03-10T00:42:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35066.481142 secs 2023-03-10T00:42:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35066.481142 secs 2023-03-10T00:42:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35066.481142 secs 2023-03-10T00:42:37.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:36 smithi122 ceph-mon[93869]: pgmap v19212: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 358 KiB/s wr, 11 op/s 2023-03-10T00:42:37.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:36 smithi181 ceph-mon[94189]: pgmap v19212: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 358 KiB/s wr, 11 op/s 2023-03-10T00:42:37.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:36 smithi038 ceph-mon[92422]: pgmap v19212: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 358 KiB/s wr, 11 op/s 2023-03-10T00:42:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:42:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:42:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35068.944248 secs 2023-03-10T00:42:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:42:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:42:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35068.944248 secs 2023-03-10T00:42:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:42:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:42:38.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35068.944248 secs 2023-03-10T00:42:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35530.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:42:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:38 smithi122 ceph-mon[93869]: pgmap v19213: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 494 KiB/s wr, 10 op/s 2023-03-10T00:42:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:38 smithi122 ceph-mon[93869]: osdmap e1416: 12 total, 12 up, 12 in 2023-03-10T00:42:39.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:38 smithi181 ceph-mon[94189]: pgmap v19213: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 494 KiB/s wr, 10 op/s 2023-03-10T00:42:39.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:38 smithi181 ceph-mon[94189]: osdmap e1416: 12 total, 12 up, 12 in 2023-03-10T00:42:39.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:38 smithi038 ceph-mon[92422]: pgmap v19213: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 494 KiB/s wr, 10 op/s 2023-03-10T00:42:39.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:38 smithi038 ceph-mon[92422]: osdmap e1416: 12 total, 12 up, 12 in 2023-03-10T00:42:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:39 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:42:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:39 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:42:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:39 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:42:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:39 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:42:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:39 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:42:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:39 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:42:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35071.481225 secs 2023-03-10T00:42:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:40 smithi122 ceph-mon[93869]: pgmap v19215: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 494 KiB/s wr, 10 op/s 2023-03-10T00:42:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35071.481225 secs 2023-03-10T00:42:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:40 smithi181 ceph-mon[94189]: pgmap v19215: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 494 KiB/s wr, 10 op/s 2023-03-10T00:42:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35071.481225 secs 2023-03-10T00:42:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:40 smithi038 ceph-mon[92422]: pgmap v19215: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 494 KiB/s wr, 10 op/s 2023-03-10T00:42:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:43 smithi122 ceph-mon[93869]: pgmap v19216: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 597 KiB/s wr, 10 op/s 2023-03-10T00:42:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35073.944335 secs 2023-03-10T00:42:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:43 smithi181 ceph-mon[94189]: pgmap v19216: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 597 KiB/s wr, 10 op/s 2023-03-10T00:42:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35073.944335 secs 2023-03-10T00:42:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:43 smithi038 ceph-mon[92422]: pgmap v19216: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 597 KiB/s wr, 10 op/s 2023-03-10T00:42:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35073.944335 secs 2023-03-10T00:42:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35535.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:42:45.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:44 smithi122 ceph-mon[93869]: pgmap v19217: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 597 KiB/s wr, 9 op/s 2023-03-10T00:42:45.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:44 smithi181 ceph-mon[94189]: pgmap v19217: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 597 KiB/s wr, 9 op/s 2023-03-10T00:42:45.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:44 smithi038 ceph-mon[92422]: pgmap v19217: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 597 KiB/s wr, 9 op/s 2023-03-10T00:42:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35076.481285 secs 2023-03-10T00:42:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35076.481285 secs 2023-03-10T00:42:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35076.481285 secs 2023-03-10T00:42:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:46 smithi122 ceph-mon[93869]: pgmap v19218: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 490 KiB/s wr, 9 op/s 2023-03-10T00:42:47.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:46 smithi181 ceph-mon[94189]: pgmap v19218: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 490 KiB/s wr, 9 op/s 2023-03-10T00:42:47.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:46 smithi038 ceph-mon[92422]: pgmap v19218: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 490 KiB/s wr, 9 op/s 2023-03-10T00:42:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35078.944395 secs 2023-03-10T00:42:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35078.944395 secs 2023-03-10T00:42:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35078.944395 secs 2023-03-10T00:42:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35540.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:42:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:48 smithi122 ceph-mon[93869]: pgmap v19219: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 320 KiB/s wr, 8 op/s 2023-03-10T00:42:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:48 smithi181 ceph-mon[94189]: pgmap v19219: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 320 KiB/s wr, 8 op/s 2023-03-10T00:42:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:48 smithi038 ceph-mon[92422]: pgmap v19219: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 320 KiB/s wr, 8 op/s 2023-03-10T00:42:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:42:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:42:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:42:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:42:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:42:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:42:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35081.481359 secs 2023-03-10T00:42:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:50 smithi122 ceph-mon[93869]: pgmap v19220: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 430 KiB/s rd, 427 KiB/s wr, 8 op/s 2023-03-10T00:42:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35081.481359 secs 2023-03-10T00:42:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:50 smithi181 ceph-mon[94189]: pgmap v19220: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 430 KiB/s rd, 427 KiB/s wr, 8 op/s 2023-03-10T00:42:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35081.481359 secs 2023-03-10T00:42:51.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:50 smithi038 ceph-mon[92422]: pgmap v19220: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 430 KiB/s rd, 427 KiB/s wr, 8 op/s 2023-03-10T00:42:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:42:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:42:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:42:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:42:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:42:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:42:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:52 smithi122 ceph-mon[93869]: pgmap v19221: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 561 KiB/s wr, 11 op/s 2023-03-10T00:42:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35083.944457 secs 2023-03-10T00:42:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:52 smithi181 ceph-mon[94189]: pgmap v19221: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 561 KiB/s wr, 11 op/s 2023-03-10T00:42:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35083.944457 secs 2023-03-10T00:42:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:52 smithi038 ceph-mon[92422]: pgmap v19221: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 561 KiB/s wr, 11 op/s 2023-03-10T00:42:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35083.944457 secs 2023-03-10T00:42:53.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35545.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:42:55.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:54 smithi122 ceph-mon[93869]: pgmap v19222: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 363 KiB/s rd, 361 KiB/s wr, 8 op/s 2023-03-10T00:42:55.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:54 smithi181 ceph-mon[94189]: pgmap v19222: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 363 KiB/s rd, 361 KiB/s wr, 8 op/s 2023-03-10T00:42:55.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:54 smithi038 ceph-mon[92422]: pgmap v19222: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 363 KiB/s rd, 361 KiB/s wr, 8 op/s 2023-03-10T00:42:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:42:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:42:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35086.481416 secs 2023-03-10T00:42:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:42:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:42:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35086.481416 secs 2023-03-10T00:42:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:42:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:42:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35086.481416 secs 2023-03-10T00:42:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:56 smithi122 ceph-mon[93869]: pgmap v19223: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 505 KiB/s rd, 500 KiB/s wr, 11 op/s 2023-03-10T00:42:57.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:56 smithi181 ceph-mon[94189]: pgmap v19223: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 505 KiB/s rd, 500 KiB/s wr, 11 op/s 2023-03-10T00:42:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:56 smithi038 ceph-mon[92422]: pgmap v19223: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 505 KiB/s rd, 500 KiB/s wr, 11 op/s 2023-03-10T00:42:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:42:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:42:58.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35088.944518 secs 2023-03-10T00:42:58.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:42:58.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:42:58.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35088.944518 secs 2023-03-10T00:42:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:42:58.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:42:58.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35088.944518 secs 2023-03-10T00:42:58.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35550.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:42:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:42:58 smithi122 ceph-mon[93869]: pgmap v19224: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 649 KiB/s rd, 645 KiB/s wr, 10 op/s 2023-03-10T00:42:59.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:42:58 smithi181 ceph-mon[94189]: pgmap v19224: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 649 KiB/s rd, 645 KiB/s wr, 10 op/s 2023-03-10T00:42:59.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:42:58 smithi038 ceph-mon[92422]: pgmap v19224: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 649 KiB/s rd, 645 KiB/s wr, 10 op/s 2023-03-10T00:43:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:43:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:43:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35091.481501 secs 2023-03-10T00:43:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:00 smithi122 ceph-mon[93869]: pgmap v19225: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 660 KiB/s rd, 645 KiB/s wr, 10 op/s 2023-03-10T00:43:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:43:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:43:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35091.481501 secs 2023-03-10T00:43:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:00 smithi181 ceph-mon[94189]: pgmap v19225: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 660 KiB/s rd, 645 KiB/s wr, 10 op/s 2023-03-10T00:43:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:43:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:43:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35091.481501 secs 2023-03-10T00:43:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:00 smithi038 ceph-mon[92422]: pgmap v19225: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 660 KiB/s rd, 645 KiB/s wr, 10 op/s 2023-03-10T00:43:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:03 smithi122 ceph-mon[93869]: pgmap v19226: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 925 KiB/s rd, 920 KiB/s wr, 16 op/s 2023-03-10T00:43:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35093.944577 secs 2023-03-10T00:43:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:03 smithi181 ceph-mon[94189]: pgmap v19226: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 925 KiB/s rd, 920 KiB/s wr, 16 op/s 2023-03-10T00:43:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35093.944577 secs 2023-03-10T00:43:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:03 smithi038 ceph-mon[92422]: pgmap v19226: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 925 KiB/s rd, 920 KiB/s wr, 16 op/s 2023-03-10T00:43:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35093.944577 secs 2023-03-10T00:43:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35555.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:43:05.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:04 smithi122 ceph-mon[93869]: pgmap v19227: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 800 KiB/s rd, 800 KiB/s wr, 23 op/s 2023-03-10T00:43:05.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:04 smithi181 ceph-mon[94189]: pgmap v19227: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 800 KiB/s rd, 800 KiB/s wr, 23 op/s 2023-03-10T00:43:05.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:04 smithi038 ceph-mon[92422]: pgmap v19227: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 800 KiB/s rd, 800 KiB/s wr, 23 op/s 2023-03-10T00:43:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35096.481562 secs 2023-03-10T00:43:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35096.481562 secs 2023-03-10T00:43:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35096.481562 secs 2023-03-10T00:43:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:06 smithi122 ceph-mon[93869]: pgmap v19228: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 843 KiB/s rd, 800 KiB/s wr, 26 op/s 2023-03-10T00:43:07.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:06 smithi181 ceph-mon[94189]: pgmap v19228: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 843 KiB/s rd, 800 KiB/s wr, 26 op/s 2023-03-10T00:43:07.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:06 smithi038 ceph-mon[92422]: pgmap v19228: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 843 KiB/s rd, 800 KiB/s wr, 26 op/s 2023-03-10T00:43:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:43:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:43:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35098.944635 secs 2023-03-10T00:43:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:43:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:43:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35098.944635 secs 2023-03-10T00:43:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:43:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:43:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35098.944635 secs 2023-03-10T00:43:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35560.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:43:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:08 smithi122 ceph-mon[93869]: pgmap v19229: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 968 KiB/s rd, 930 KiB/s wr, 25 op/s 2023-03-10T00:43:09.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:08 smithi181 ceph-mon[94189]: pgmap v19229: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 968 KiB/s rd, 930 KiB/s wr, 25 op/s 2023-03-10T00:43:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:08 smithi038 ceph-mon[92422]: pgmap v19229: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 968 KiB/s rd, 930 KiB/s wr, 25 op/s 2023-03-10T00:43:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:43:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:43:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:43:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:43:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:43:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:43:10.811 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35101.481623 secs 2023-03-10T00:43:10.812 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:10 smithi038 ceph-mon[92422]: pgmap v19230: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 758 KiB/s rd, 719 KiB/s wr, 23 op/s 2023-03-10T00:43:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35101.481623 secs 2023-03-10T00:43:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:10 smithi122 ceph-mon[93869]: pgmap v19230: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 758 KiB/s rd, 719 KiB/s wr, 23 op/s 2023-03-10T00:43:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35101.481623 secs 2023-03-10T00:43:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:10 smithi181 ceph-mon[94189]: pgmap v19230: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 758 KiB/s rd, 719 KiB/s wr, 23 op/s 2023-03-10T00:43:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:13 smithi122 ceph-mon[93869]: pgmap v19231: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 856 KiB/s rd, 827 KiB/s wr, 25 op/s 2023-03-10T00:43:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35103.944695 secs 2023-03-10T00:43:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:13 smithi181 ceph-mon[94189]: pgmap v19231: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 856 KiB/s rd, 827 KiB/s wr, 25 op/s 2023-03-10T00:43:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35103.944695 secs 2023-03-10T00:43:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:13 smithi038 ceph-mon[92422]: pgmap v19231: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 856 KiB/s rd, 827 KiB/s wr, 25 op/s 2023-03-10T00:43:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35103.944695 secs 2023-03-10T00:43:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35565.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:43:15.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:14 smithi122 ceph-mon[93869]: pgmap v19232: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 462 KiB/s wr, 19 op/s 2023-03-10T00:43:15.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:14 smithi181 ceph-mon[94189]: pgmap v19232: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 462 KiB/s wr, 19 op/s 2023-03-10T00:43:15.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:14 smithi038 ceph-mon[92422]: pgmap v19232: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 462 KiB/s wr, 19 op/s 2023-03-10T00:43:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:15 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:43:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:15 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:43:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35106.481706 secs 2023-03-10T00:43:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:15 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:43:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:15 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:43:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35106.481706 secs 2023-03-10T00:43:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:15 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:43:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:15 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:43:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35106.481706 secs 2023-03-10T00:43:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:16 smithi122 ceph-mon[93869]: pgmap v19233: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 467 KiB/s rd, 423 KiB/s wr, 10 op/s 2023-03-10T00:43:17.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:16 smithi181 ceph-mon[94189]: pgmap v19233: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 467 KiB/s rd, 423 KiB/s wr, 10 op/s 2023-03-10T00:43:17.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:16 smithi038 ceph-mon[92422]: pgmap v19233: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 467 KiB/s rd, 423 KiB/s wr, 10 op/s 2023-03-10T00:43:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:43:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:43:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35108.944753 secs 2023-03-10T00:43:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:43:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:43:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35108.944753 secs 2023-03-10T00:43:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:43:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:43:18.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35108.944753 secs 2023-03-10T00:43:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35570.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:43:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:18 smithi122 ceph-mon[93869]: pgmap v19234: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 434 KiB/s wr, 9 op/s 2023-03-10T00:43:19.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:18 smithi181 ceph-mon[94189]: pgmap v19234: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 434 KiB/s wr, 9 op/s 2023-03-10T00:43:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:18 smithi038 ceph-mon[92422]: pgmap v19234: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 434 KiB/s wr, 9 op/s 2023-03-10T00:43:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:19 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:43:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:19 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:43:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:19 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:43:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:19 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:43:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:19 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:43:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:19 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:43:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35111.481765 secs 2023-03-10T00:43:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:20 smithi122 ceph-mon[93869]: pgmap v19235: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 235 KiB/s wr, 8 op/s 2023-03-10T00:43:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35111.481765 secs 2023-03-10T00:43:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:20 smithi181 ceph-mon[94189]: pgmap v19235: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 235 KiB/s wr, 8 op/s 2023-03-10T00:43:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35111.481765 secs 2023-03-10T00:43:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:20 smithi038 ceph-mon[92422]: pgmap v19235: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 235 KiB/s wr, 8 op/s 2023-03-10T00:43:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:43:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:43:22.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:43:22.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:43:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:43:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:43:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:22 smithi122 ceph-mon[93869]: pgmap v19236: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 507 KiB/s wr, 12 op/s 2023-03-10T00:43:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35113.944837 secs 2023-03-10T00:43:23.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:22 smithi181 ceph-mon[94189]: pgmap v19236: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 507 KiB/s wr, 12 op/s 2023-03-10T00:43:23.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35113.944837 secs 2023-03-10T00:43:23.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:22 smithi038 ceph-mon[92422]: pgmap v19236: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 507 KiB/s wr, 12 op/s 2023-03-10T00:43:23.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35113.944837 secs 2023-03-10T00:43:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35575.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:43:25.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:24 smithi122 ceph-mon[93869]: pgmap v19237: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 510 KiB/s wr, 11 op/s 2023-03-10T00:43:25.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:24 smithi181 ceph-mon[94189]: pgmap v19237: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 510 KiB/s wr, 11 op/s 2023-03-10T00:43:25.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:24 smithi038 ceph-mon[92422]: pgmap v19237: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 510 KiB/s wr, 11 op/s 2023-03-10T00:43:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:25 smithi122 ceph-mon[93869]: osdmap e1417: 12 total, 12 up, 12 in 2023-03-10T00:43:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35116.481825 secs 2023-03-10T00:43:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:25 smithi181 ceph-mon[94189]: osdmap e1417: 12 total, 12 up, 12 in 2023-03-10T00:43:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35116.481825 secs 2023-03-10T00:43:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:25 smithi038 ceph-mon[92422]: osdmap e1417: 12 total, 12 up, 12 in 2023-03-10T00:43:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35116.481825 secs 2023-03-10T00:43:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:26 smithi122 ceph-mon[93869]: pgmap v19239: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 535 KiB/s rd, 559 KiB/s wr, 12 op/s 2023-03-10T00:43:27.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:26 smithi181 ceph-mon[94189]: pgmap v19239: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 535 KiB/s rd, 559 KiB/s wr, 12 op/s 2023-03-10T00:43:27.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:26 smithi038 ceph-mon[92422]: pgmap v19239: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 535 KiB/s rd, 559 KiB/s wr, 12 op/s 2023-03-10T00:43:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35118.944920 secs 2023-03-10T00:43:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35118.944920 secs 2023-03-10T00:43:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35118.944920 secs 2023-03-10T00:43:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35580.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:43:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:28 smithi122 ceph-mon[93869]: pgmap v19240: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 535 KiB/s rd, 546 KiB/s wr, 10 op/s 2023-03-10T00:43:29.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:28 smithi181 ceph-mon[94189]: pgmap v19240: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 535 KiB/s rd, 546 KiB/s wr, 10 op/s 2023-03-10T00:43:29.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:28 smithi038 ceph-mon[92422]: pgmap v19240: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 535 KiB/s rd, 546 KiB/s wr, 10 op/s 2023-03-10T00:43:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35121.481883 secs 2023-03-10T00:43:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:31 smithi122 ceph-mon[93869]: pgmap v19241: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 620 KiB/s rd, 632 KiB/s wr, 11 op/s 2023-03-10T00:43:31.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:43:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35121.481883 secs 2023-03-10T00:43:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:31 smithi181 ceph-mon[94189]: pgmap v19241: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 620 KiB/s rd, 632 KiB/s wr, 11 op/s 2023-03-10T00:43:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:43:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35121.481883 secs 2023-03-10T00:43:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:31 smithi038 ceph-mon[92422]: pgmap v19241: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 620 KiB/s rd, 632 KiB/s wr, 11 op/s 2023-03-10T00:43:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:43:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:43:32.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:43:32.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:43:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:43:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:43:32.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:43:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:43:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:43:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:43:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:33 smithi122 ceph-mon[93869]: pgmap v19242: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 330 KiB/s wr, 9 op/s 2023-03-10T00:43:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35123.945024 secs 2023-03-10T00:43:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:33 smithi181 ceph-mon[94189]: pgmap v19242: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 330 KiB/s wr, 9 op/s 2023-03-10T00:43:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35123.945024 secs 2023-03-10T00:43:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:33 smithi038 ceph-mon[92422]: pgmap v19242: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 330 KiB/s wr, 9 op/s 2023-03-10T00:43:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35123.945024 secs 2023-03-10T00:43:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35585.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:43:35.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:34 smithi122 ceph-mon[93869]: osdmap e1418: 12 total, 12 up, 12 in 2023-03-10T00:43:35.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:34 smithi122 ceph-mon[93869]: pgmap v19244: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 245 KiB/s wr, 10 op/s 2023-03-10T00:43:35.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:34 smithi181 ceph-mon[94189]: osdmap e1418: 12 total, 12 up, 12 in 2023-03-10T00:43:35.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:34 smithi181 ceph-mon[94189]: pgmap v19244: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 245 KiB/s wr, 10 op/s 2023-03-10T00:43:35.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:34 smithi038 ceph-mon[92422]: osdmap e1418: 12 total, 12 up, 12 in 2023-03-10T00:43:35.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:34 smithi038 ceph-mon[92422]: pgmap v19244: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 245 KiB/s wr, 10 op/s 2023-03-10T00:43:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:35 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:43:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:35 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:43:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35126.481943 secs 2023-03-10T00:43:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:35 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:43:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:35 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:43:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35126.481943 secs 2023-03-10T00:43:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:35 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:43:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:35 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:43:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35126.481943 secs 2023-03-10T00:43:37.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:36 smithi122 ceph-mon[93869]: pgmap v19245: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 821 KiB/s rd, 724 KiB/s wr, 11 op/s 2023-03-10T00:43:37.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:36 smithi181 ceph-mon[94189]: pgmap v19245: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 821 KiB/s rd, 724 KiB/s wr, 11 op/s 2023-03-10T00:43:37.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:36 smithi038 ceph-mon[92422]: pgmap v19245: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 821 KiB/s rd, 724 KiB/s wr, 11 op/s 2023-03-10T00:43:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:43:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:43:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35128.945082 secs 2023-03-10T00:43:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:43:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:43:38.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35128.945082 secs 2023-03-10T00:43:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:43:38.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:43:38.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35128.945082 secs 2023-03-10T00:43:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35590.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:43:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:38 smithi122 ceph-mon[93869]: pgmap v19246: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 723 KiB/s wr, 11 op/s 2023-03-10T00:43:39.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:38 smithi181 ceph-mon[94189]: pgmap v19246: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 723 KiB/s wr, 11 op/s 2023-03-10T00:43:39.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:38 smithi038 ceph-mon[92422]: pgmap v19246: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 723 KiB/s wr, 11 op/s 2023-03-10T00:43:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35131.481980 secs 2023-03-10T00:43:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:41 smithi122 ceph-mon[93869]: pgmap v19247: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 554 KiB/s wr, 11 op/s 2023-03-10T00:43:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35131.481980 secs 2023-03-10T00:43:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:41 smithi181 ceph-mon[94189]: pgmap v19247: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 554 KiB/s wr, 11 op/s 2023-03-10T00:43:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35131.481980 secs 2023-03-10T00:43:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:41 smithi038 ceph-mon[92422]: pgmap v19247: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 554 KiB/s wr, 11 op/s 2023-03-10T00:43:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:43 smithi122 ceph-mon[93869]: pgmap v19248: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 940 KiB/s wr, 12 op/s 2023-03-10T00:43:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35133.945159 secs 2023-03-10T00:43:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:43 smithi181 ceph-mon[94189]: pgmap v19248: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 940 KiB/s wr, 12 op/s 2023-03-10T00:43:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35133.945159 secs 2023-03-10T00:43:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:43 smithi038 ceph-mon[92422]: pgmap v19248: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 940 KiB/s wr, 12 op/s 2023-03-10T00:43:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35133.945159 secs 2023-03-10T00:43:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35595.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:43:45.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:44 smithi122 ceph-mon[93869]: pgmap v19249: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 883 KiB/s rd, 881 KiB/s wr, 10 op/s 2023-03-10T00:43:45.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:44 smithi181 ceph-mon[94189]: pgmap v19249: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 883 KiB/s rd, 881 KiB/s wr, 10 op/s 2023-03-10T00:43:45.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:44 smithi038 ceph-mon[92422]: pgmap v19249: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 883 KiB/s rd, 881 KiB/s wr, 10 op/s 2023-03-10T00:43:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35136.482042 secs 2023-03-10T00:43:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35136.482042 secs 2023-03-10T00:43:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35136.482042 secs 2023-03-10T00:43:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:46 smithi122 ceph-mon[93869]: pgmap v19250: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 13 op/s 2023-03-10T00:43:47.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:46 smithi181 ceph-mon[94189]: pgmap v19250: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 13 op/s 2023-03-10T00:43:47.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:46 smithi038 ceph-mon[92422]: pgmap v19250: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 13 op/s 2023-03-10T00:43:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:43:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:43:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:43:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:43:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:43:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:43:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35600.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:43:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:48 smithi122 ceph-mon[93869]: pgmap v19251: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 625 KiB/s wr, 9 op/s 2023-03-10T00:43:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:48 smithi181 ceph-mon[94189]: pgmap v19251: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 625 KiB/s wr, 9 op/s 2023-03-10T00:43:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:48 smithi038 ceph-mon[92422]: pgmap v19251: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 625 KiB/s wr, 9 op/s 2023-03-10T00:43:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:43:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:43:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:49 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:43:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:43:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:43:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:49 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:43:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:43:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:43:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:49 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:43:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:50 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:43:51.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:50 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:43:51.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:50 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:43:51.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35141.482096 secs 2023-03-10T00:43:51.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:50 smithi122 ceph-mon[93869]: pgmap v19252: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 995 KiB/s wr, 9 op/s 2023-03-10T00:43:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:50 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:43:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:50 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:43:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:50 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:43:51.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35141.482096 secs 2023-03-10T00:43:51.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:50 smithi181 ceph-mon[94189]: pgmap v19252: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 995 KiB/s wr, 9 op/s 2023-03-10T00:43:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:50 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:43:51.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:50 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:43:51.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:50 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:43:51.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35141.482096 secs 2023-03-10T00:43:51.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:50 smithi038 ceph-mon[92422]: pgmap v19252: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 995 KiB/s wr, 9 op/s 2023-03-10T00:43:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:43:52.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:43:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:43:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:43:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:43:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:43:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:43:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:43:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:52 smithi122 ceph-mon[93869]: pgmap v19253: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 12 op/s 2023-03-10T00:43:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:43:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:43:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:52 smithi181 ceph-mon[94189]: pgmap v19253: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 12 op/s 2023-03-10T00:43:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:43:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:43:53.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:52 smithi038 ceph-mon[92422]: pgmap v19253: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 12 op/s 2023-03-10T00:43:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35605.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:43:55.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:54 smithi122 ceph-mon[93869]: pgmap v19254: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 954 KiB/s rd, 876 KiB/s wr, 10 op/s 2023-03-10T00:43:55.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:54 smithi181 ceph-mon[94189]: pgmap v19254: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 954 KiB/s rd, 876 KiB/s wr, 10 op/s 2023-03-10T00:43:55.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:54 smithi038 ceph-mon[92422]: pgmap v19254: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 954 KiB/s rd, 876 KiB/s wr, 10 op/s 2023-03-10T00:43:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:43:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:43:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35146.482153 secs 2023-03-10T00:43:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:43:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:43:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35146.482153 secs 2023-03-10T00:43:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:43:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:43:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35146.482153 secs 2023-03-10T00:43:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:56 smithi122 ceph-mon[93869]: pgmap v19255: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 15 op/s 2023-03-10T00:43:57.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:56 smithi181 ceph-mon[94189]: pgmap v19255: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 15 op/s 2023-03-10T00:43:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:56 smithi038 ceph-mon[92422]: pgmap v19255: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 15 op/s 2023-03-10T00:43:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:43:58.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:43:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:43:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:43:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:43:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:43:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35610.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:43:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:58 smithi122 ceph-mon[93869]: pgmap v19256: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 934 KiB/s rd, 768 KiB/s wr, 11 op/s 2023-03-10T00:43:59.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:58 smithi181 ceph-mon[94189]: pgmap v19256: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 934 KiB/s rd, 768 KiB/s wr, 11 op/s 2023-03-10T00:43:59.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:58 smithi038 ceph-mon[92422]: pgmap v19256: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 934 KiB/s rd, 768 KiB/s wr, 11 op/s 2023-03-10T00:44:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:44:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:43:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:44:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:44:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:43:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:44:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:44:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:43:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:44:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35151.482212 secs 2023-03-10T00:44:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:00 smithi122 ceph-mon[93869]: pgmap v19257: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 962 KiB/s rd, 796 KiB/s wr, 11 op/s 2023-03-10T00:44:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35151.482212 secs 2023-03-10T00:44:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:00 smithi181 ceph-mon[94189]: pgmap v19257: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 962 KiB/s rd, 796 KiB/s wr, 11 op/s 2023-03-10T00:44:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35151.482212 secs 2023-03-10T00:44:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:00 smithi038 ceph-mon[92422]: pgmap v19257: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 962 KiB/s rd, 796 KiB/s wr, 11 op/s 2023-03-10T00:44:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:44:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:44:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:44:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:44:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:44:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:44:03.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:02 smithi122 ceph-mon[93869]: pgmap v19258: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1013 KiB/s wr, 14 op/s 2023-03-10T00:44:03.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:02 smithi181 ceph-mon[94189]: pgmap v19258: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1013 KiB/s wr, 14 op/s 2023-03-10T00:44:03.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:02 smithi038 ceph-mon[92422]: pgmap v19258: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1013 KiB/s wr, 14 op/s 2023-03-10T00:44:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35615.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:44:05.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:04 smithi122 ceph-mon[93869]: pgmap v19259: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 905 KiB/s rd, 789 KiB/s wr, 11 op/s 2023-03-10T00:44:05.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:04 smithi181 ceph-mon[94189]: pgmap v19259: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 905 KiB/s rd, 789 KiB/s wr, 11 op/s 2023-03-10T00:44:05.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:04 smithi038 ceph-mon[92422]: pgmap v19259: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 905 KiB/s rd, 789 KiB/s wr, 11 op/s 2023-03-10T00:44:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35156.482273 secs 2023-03-10T00:44:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35156.482273 secs 2023-03-10T00:44:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35156.482273 secs 2023-03-10T00:44:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:06 smithi122 ceph-mon[93869]: pgmap v19260: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 938 KiB/s wr, 17 op/s 2023-03-10T00:44:07.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:06 smithi181 ceph-mon[94189]: pgmap v19260: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 938 KiB/s wr, 17 op/s 2023-03-10T00:44:07.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:06 smithi038 ceph-mon[92422]: pgmap v19260: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 938 KiB/s wr, 17 op/s 2023-03-10T00:44:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:07 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 35158.945626 secs 2023-03-10T00:44:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:07 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 35158.945626 secs 2023-03-10T00:44:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:07 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 35158.945626 secs 2023-03-10T00:44:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35620.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:44:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:08 smithi122 ceph-mon[93869]: pgmap v19261: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 789 KiB/s rd, 775 KiB/s wr, 22 op/s 2023-03-10T00:44:09.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:08 smithi181 ceph-mon[94189]: pgmap v19261: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 789 KiB/s rd, 775 KiB/s wr, 22 op/s 2023-03-10T00:44:09.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:08 smithi038 ceph-mon[92422]: pgmap v19261: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 789 KiB/s rd, 775 KiB/s wr, 22 op/s 2023-03-10T00:44:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:09 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:44:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:44:10.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:44:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:09 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:44:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:44:10.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:44:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:09 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:44:10.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:44:10.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:44:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35161.482355 secs 2023-03-10T00:44:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:10 smithi122 ceph-mon[93869]: pgmap v19262: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 817 KiB/s rd, 803 KiB/s wr, 22 op/s 2023-03-10T00:44:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35161.482355 secs 2023-03-10T00:44:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:10 smithi181 ceph-mon[94189]: pgmap v19262: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 817 KiB/s rd, 803 KiB/s wr, 22 op/s 2023-03-10T00:44:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35161.482355 secs 2023-03-10T00:44:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:10 smithi038 ceph-mon[92422]: pgmap v19262: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 817 KiB/s rd, 803 KiB/s wr, 22 op/s 2023-03-10T00:44:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:11 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:44:12.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:11 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:44:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:11 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:44:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:11 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:44:12.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:11 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:44:12.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:11 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:44:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:44:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:44:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:12 smithi122 ceph-mon[93869]: pgmap v19263: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 926 KiB/s rd, 911 KiB/s wr, 25 op/s 2023-03-10T00:44:13.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:12 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 35163.945687 secs 2023-03-10T00:44:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:44:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:44:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:12 smithi181 ceph-mon[94189]: pgmap v19263: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 926 KiB/s rd, 911 KiB/s wr, 25 op/s 2023-03-10T00:44:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:12 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 35163.945687 secs 2023-03-10T00:44:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:44:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:44:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:12 smithi038 ceph-mon[92422]: pgmap v19263: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 926 KiB/s rd, 911 KiB/s wr, 25 op/s 2023-03-10T00:44:13.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:12 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 35163.945687 secs 2023-03-10T00:44:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35625.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:44:15.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:14 smithi122 ceph-mon[93869]: pgmap v19264: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 324 KiB/s wr, 22 op/s 2023-03-10T00:44:15.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:14 smithi181 ceph-mon[94189]: pgmap v19264: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 324 KiB/s wr, 22 op/s 2023-03-10T00:44:15.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:14 smithi038 ceph-mon[92422]: pgmap v19264: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 324 KiB/s wr, 22 op/s 2023-03-10T00:44:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35166.482414 secs 2023-03-10T00:44:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35166.482414 secs 2023-03-10T00:44:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35166.482414 secs 2023-03-10T00:44:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:16 smithi122 ceph-mon[93869]: pgmap v19265: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 445 KiB/s rd, 457 KiB/s wr, 23 op/s 2023-03-10T00:44:17.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:16 smithi181 ceph-mon[94189]: pgmap v19265: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 445 KiB/s rd, 457 KiB/s wr, 23 op/s 2023-03-10T00:44:17.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:16 smithi038 ceph-mon[92422]: pgmap v19265: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 445 KiB/s rd, 457 KiB/s wr, 23 op/s 2023-03-10T00:44:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:17 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 35168.945771 secs 2023-03-10T00:44:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:17 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 35168.945771 secs 2023-03-10T00:44:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:17 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 35168.945771 secs 2023-03-10T00:44:18.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35630.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:44:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:18 smithi122 ceph-mon[93869]: pgmap v19266: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 308 KiB/s wr, 17 op/s 2023-03-10T00:44:19.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:18 smithi181 ceph-mon[94189]: pgmap v19266: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 308 KiB/s wr, 17 op/s 2023-03-10T00:44:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:18 smithi038 ceph-mon[92422]: pgmap v19266: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 308 KiB/s wr, 17 op/s 2023-03-10T00:44:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35171.482499 secs 2023-03-10T00:44:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:21 smithi122 ceph-mon[93869]: pgmap v19267: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 325 KiB/s wr, 8 op/s 2023-03-10T00:44:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35171.482499 secs 2023-03-10T00:44:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:21 smithi181 ceph-mon[94189]: pgmap v19267: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 325 KiB/s wr, 8 op/s 2023-03-10T00:44:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35171.482499 secs 2023-03-10T00:44:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:21 smithi038 ceph-mon[92422]: pgmap v19267: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 325 KiB/s wr, 8 op/s 2023-03-10T00:44:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:23 smithi122 ceph-mon[93869]: pgmap v19268: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 298 KiB/s wr, 10 op/s 2023-03-10T00:44:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:23 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 35173.945858 secs 2023-03-10T00:44:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:23 smithi181 ceph-mon[94189]: pgmap v19268: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 298 KiB/s wr, 10 op/s 2023-03-10T00:44:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:23 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 35173.945858 secs 2023-03-10T00:44:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:23 smithi038 ceph-mon[92422]: pgmap v19268: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 298 KiB/s wr, 10 op/s 2023-03-10T00:44:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:23 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 35173.945858 secs 2023-03-10T00:44:23.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35635.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:44:25.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:24 smithi122 ceph-mon[93869]: pgmap v19269: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 336 KiB/s wr, 8 op/s 2023-03-10T00:44:25.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:24 smithi181 ceph-mon[94189]: pgmap v19269: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 336 KiB/s wr, 8 op/s 2023-03-10T00:44:25.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:24 smithi038 ceph-mon[92422]: pgmap v19269: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 336 KiB/s wr, 8 op/s 2023-03-10T00:44:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:25 smithi122 ceph-mon[93869]: osdmap e1419: 12 total, 12 up, 12 in 2023-03-10T00:44:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35176.482557 secs 2023-03-10T00:44:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:25 smithi181 ceph-mon[94189]: osdmap e1419: 12 total, 12 up, 12 in 2023-03-10T00:44:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35176.482557 secs 2023-03-10T00:44:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:25 smithi038 ceph-mon[92422]: osdmap e1419: 12 total, 12 up, 12 in 2023-03-10T00:44:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35176.482557 secs 2023-03-10T00:44:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:26 smithi122 ceph-mon[93869]: pgmap v19271: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 328 KiB/s wr, 11 op/s 2023-03-10T00:44:27.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:26 smithi181 ceph-mon[94189]: pgmap v19271: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 328 KiB/s wr, 11 op/s 2023-03-10T00:44:27.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:26 smithi038 ceph-mon[92422]: pgmap v19271: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 328 KiB/s wr, 11 op/s 2023-03-10T00:44:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:27 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:44:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:27 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:44:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35178.945918 secs 2023-03-10T00:44:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:27 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:44:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:27 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:44:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35178.945918 secs 2023-03-10T00:44:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:27 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:44:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:27 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:44:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35178.945918 secs 2023-03-10T00:44:28.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35640.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:44:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:28 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:44:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:28 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:44:29.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:28 smithi122 ceph-mon[93869]: pgmap v19272: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 328 KiB/s wr, 12 op/s 2023-03-10T00:44:29.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:28 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:44:29.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:28 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:44:29.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:28 smithi181 ceph-mon[94189]: pgmap v19272: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 328 KiB/s wr, 12 op/s 2023-03-10T00:44:29.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:28 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:44:29.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:28 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:44:29.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:28 smithi038 ceph-mon[92422]: pgmap v19272: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 328 KiB/s wr, 12 op/s 2023-03-10T00:44:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:30 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:44:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:30 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:44:31.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35181.482642 secs 2023-03-10T00:44:31.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:30 smithi122 ceph-mon[93869]: pgmap v19273: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 296 KiB/s wr, 12 op/s 2023-03-10T00:44:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:30 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:44:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:30 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:44:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35181.482642 secs 2023-03-10T00:44:31.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:30 smithi181 ceph-mon[94189]: pgmap v19273: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 296 KiB/s wr, 12 op/s 2023-03-10T00:44:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:30 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:44:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:30 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:44:31.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35181.482642 secs 2023-03-10T00:44:31.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:30 smithi038 ceph-mon[92422]: pgmap v19273: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 296 KiB/s wr, 12 op/s 2023-03-10T00:44:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:44:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:44:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:44:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:33 smithi122 ceph-mon[93869]: pgmap v19274: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 327 KiB/s wr, 12 op/s 2023-03-10T00:44:33.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:44:33.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:44:33.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35183.945979 secs 2023-03-10T00:44:33.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:44:33.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:44:33.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:44:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:33 smithi181 ceph-mon[94189]: pgmap v19274: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 327 KiB/s wr, 12 op/s 2023-03-10T00:44:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:44:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:44:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35183.945979 secs 2023-03-10T00:44:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:44:33.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:44:33.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:44:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:33 smithi038 ceph-mon[92422]: pgmap v19274: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 327 KiB/s wr, 12 op/s 2023-03-10T00:44:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:44:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:44:33.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35183.945979 secs 2023-03-10T00:44:33.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:44:33.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:44:33.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:44:33.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35645.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:44:35.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:34 smithi122 ceph-mon[93869]: osdmap e1420: 12 total, 12 up, 12 in 2023-03-10T00:44:35.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:34 smithi122 ceph-mon[93869]: pgmap v19276: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 321 KiB/s wr, 11 op/s 2023-03-10T00:44:35.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:34 smithi181 ceph-mon[94189]: osdmap e1420: 12 total, 12 up, 12 in 2023-03-10T00:44:35.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:34 smithi181 ceph-mon[94189]: pgmap v19276: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 321 KiB/s wr, 11 op/s 2023-03-10T00:44:35.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:34 smithi038 ceph-mon[92422]: osdmap e1420: 12 total, 12 up, 12 in 2023-03-10T00:44:35.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:34 smithi038 ceph-mon[92422]: pgmap v19276: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 321 KiB/s wr, 11 op/s 2023-03-10T00:44:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:35 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:44:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:35 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:44:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35186.482699 secs 2023-03-10T00:44:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:35 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:44:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:35 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:44:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35186.482699 secs 2023-03-10T00:44:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:35 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:44:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:35 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:44:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35186.482699 secs 2023-03-10T00:44:37.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:44:37.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:44:37.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:36 smithi122 ceph-mon[93869]: pgmap v19277: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 415 KiB/s wr, 9 op/s 2023-03-10T00:44:37.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:44:37.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:44:37.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:36 smithi181 ceph-mon[94189]: pgmap v19277: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 415 KiB/s wr, 9 op/s 2023-03-10T00:44:37.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:44:37.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:44:37.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:36 smithi038 ceph-mon[92422]: pgmap v19277: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 415 KiB/s wr, 9 op/s 2023-03-10T00:44:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35188.946099 secs 2023-03-10T00:44:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35188.946099 secs 2023-03-10T00:44:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35188.946099 secs 2023-03-10T00:44:38.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35650.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:44:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:38 smithi122 ceph-mon[93869]: pgmap v19278: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 414 KiB/s wr, 9 op/s 2023-03-10T00:44:39.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:38 smithi181 ceph-mon[94189]: pgmap v19278: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 414 KiB/s wr, 9 op/s 2023-03-10T00:44:39.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:38 smithi038 ceph-mon[92422]: pgmap v19278: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 414 KiB/s wr, 9 op/s 2023-03-10T00:44:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35191.482795 secs 2023-03-10T00:44:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:41 smithi122 ceph-mon[93869]: pgmap v19279: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 416 KiB/s rd, 414 KiB/s wr, 9 op/s 2023-03-10T00:44:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35191.482795 secs 2023-03-10T00:44:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:41 smithi181 ceph-mon[94189]: pgmap v19279: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 416 KiB/s rd, 414 KiB/s wr, 9 op/s 2023-03-10T00:44:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35191.482795 secs 2023-03-10T00:44:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:41 smithi038 ceph-mon[92422]: pgmap v19279: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 416 KiB/s rd, 414 KiB/s wr, 9 op/s 2023-03-10T00:44:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:43 smithi122 ceph-mon[93869]: pgmap v19280: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 577 KiB/s rd, 542 KiB/s wr, 9 op/s 2023-03-10T00:44:43.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35193.946116 secs 2023-03-10T00:44:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:43 smithi181 ceph-mon[94189]: pgmap v19280: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 577 KiB/s rd, 542 KiB/s wr, 9 op/s 2023-03-10T00:44:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35193.946116 secs 2023-03-10T00:44:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:43 smithi038 ceph-mon[92422]: pgmap v19280: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 577 KiB/s rd, 542 KiB/s wr, 9 op/s 2023-03-10T00:44:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35193.946116 secs 2023-03-10T00:44:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35655.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:44:45.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:44 smithi122 ceph-mon[93869]: pgmap v19281: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 559 KiB/s rd, 526 KiB/s wr, 9 op/s 2023-03-10T00:44:45.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:44 smithi181 ceph-mon[94189]: pgmap v19281: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 559 KiB/s rd, 526 KiB/s wr, 9 op/s 2023-03-10T00:44:45.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:44 smithi038 ceph-mon[92422]: pgmap v19281: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 559 KiB/s rd, 526 KiB/s wr, 9 op/s 2023-03-10T00:44:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35196.482856 secs 2023-03-10T00:44:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35196.482856 secs 2023-03-10T00:44:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35196.482856 secs 2023-03-10T00:44:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:46 smithi122 ceph-mon[93869]: pgmap v19282: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 430 KiB/s rd, 316 KiB/s wr, 9 op/s 2023-03-10T00:44:47.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:46 smithi181 ceph-mon[94189]: pgmap v19282: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 430 KiB/s rd, 316 KiB/s wr, 9 op/s 2023-03-10T00:44:47.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:46 smithi038 ceph-mon[92422]: pgmap v19282: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 430 KiB/s rd, 316 KiB/s wr, 9 op/s 2023-03-10T00:44:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35198.946179 secs 2023-03-10T00:44:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35198.946179 secs 2023-03-10T00:44:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35198.946179 secs 2023-03-10T00:44:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35660.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:44:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:48 smithi122 ceph-mon[93869]: pgmap v19283: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 271 KiB/s wr, 7 op/s 2023-03-10T00:44:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:48 smithi181 ceph-mon[94189]: pgmap v19283: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 271 KiB/s wr, 7 op/s 2023-03-10T00:44:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:48 smithi038 ceph-mon[92422]: pgmap v19283: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 271 KiB/s wr, 7 op/s 2023-03-10T00:44:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35201.482915 secs 2023-03-10T00:44:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:51 smithi122 ceph-mon[93869]: pgmap v19284: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 272 KiB/s wr, 8 op/s 2023-03-10T00:44:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35201.482915 secs 2023-03-10T00:44:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:51 smithi181 ceph-mon[94189]: pgmap v19284: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 272 KiB/s wr, 8 op/s 2023-03-10T00:44:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35201.482915 secs 2023-03-10T00:44:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:51 smithi038 ceph-mon[92422]: pgmap v19284: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 272 KiB/s wr, 8 op/s 2023-03-10T00:44:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:53 smithi122 ceph-mon[93869]: pgmap v19285: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 297 KiB/s wr, 9 op/s 2023-03-10T00:44:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:44:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:44:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35203.946239 secs 2023-03-10T00:44:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:53 smithi181 ceph-mon[94189]: pgmap v19285: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 297 KiB/s wr, 9 op/s 2023-03-10T00:44:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:44:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:44:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35203.946239 secs 2023-03-10T00:44:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:53 smithi038 ceph-mon[92422]: pgmap v19285: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 297 KiB/s wr, 9 op/s 2023-03-10T00:44:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:44:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:44:53.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35203.946239 secs 2023-03-10T00:44:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35665.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:44:55.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:54 smithi122 ceph-mon[93869]: pgmap v19286: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 410 KiB/s wr, 8 op/s 2023-03-10T00:44:55.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:54 smithi181 ceph-mon[94189]: pgmap v19286: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 410 KiB/s wr, 8 op/s 2023-03-10T00:44:55.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:54 smithi038 ceph-mon[92422]: pgmap v19286: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 410 KiB/s wr, 8 op/s 2023-03-10T00:44:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:44:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:44:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35206.483001 secs 2023-03-10T00:44:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:44:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:44:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35206.483001 secs 2023-03-10T00:44:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:44:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:44:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35206.483001 secs 2023-03-10T00:44:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:44:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:44:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:56 smithi122 ceph-mon[93869]: pgmap v19287: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 555 KiB/s rd, 465 KiB/s wr, 11 op/s 2023-03-10T00:44:57.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:44:57.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:44:57.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:56 smithi181 ceph-mon[94189]: pgmap v19287: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 555 KiB/s rd, 465 KiB/s wr, 11 op/s 2023-03-10T00:44:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:44:57.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:44:57.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:56 smithi038 ceph-mon[92422]: pgmap v19287: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 555 KiB/s rd, 465 KiB/s wr, 11 op/s 2023-03-10T00:44:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35208.946297 secs 2023-03-10T00:44:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35208.946297 secs 2023-03-10T00:44:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35208.946297 secs 2023-03-10T00:44:58.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35670.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:44:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:44:58 smithi122 ceph-mon[93869]: pgmap v19288: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 423 KiB/s wr, 10 op/s 2023-03-10T00:44:59.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:44:58 smithi181 ceph-mon[94189]: pgmap v19288: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 423 KiB/s wr, 10 op/s 2023-03-10T00:44:59.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:44:58 smithi038 ceph-mon[92422]: pgmap v19288: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 423 KiB/s wr, 10 op/s 2023-03-10T00:45:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:45:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:45:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35211.483059 secs 2023-03-10T00:45:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:00 smithi122 ceph-mon[93869]: pgmap v19289: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 373 KiB/s wr, 10 op/s 2023-03-10T00:45:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:45:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:45:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35211.483059 secs 2023-03-10T00:45:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:00 smithi181 ceph-mon[94189]: pgmap v19289: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 373 KiB/s wr, 10 op/s 2023-03-10T00:45:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:45:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:45:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35211.483059 secs 2023-03-10T00:45:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:00 smithi038 ceph-mon[92422]: pgmap v19289: 129 pgs: 129 active+clean; 13 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 373 KiB/s wr, 10 op/s 2023-03-10T00:45:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:03 smithi122 ceph-mon[93869]: pgmap v19290: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 678 KiB/s rd, 673 KiB/s wr, 14 op/s 2023-03-10T00:45:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35213.946392 secs 2023-03-10T00:45:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:03 smithi181 ceph-mon[94189]: pgmap v19290: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 678 KiB/s rd, 673 KiB/s wr, 14 op/s 2023-03-10T00:45:03.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35213.946392 secs 2023-03-10T00:45:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:03 smithi038 ceph-mon[92422]: pgmap v19290: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 678 KiB/s rd, 673 KiB/s wr, 14 op/s 2023-03-10T00:45:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35213.946392 secs 2023-03-10T00:45:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35675.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:45:05.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:04 smithi122 ceph-mon[93869]: pgmap v19291: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 983 KiB/s wr, 24 op/s 2023-03-10T00:45:05.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:04 smithi181 ceph-mon[94189]: pgmap v19291: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 983 KiB/s wr, 24 op/s 2023-03-10T00:45:05.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:04 smithi038 ceph-mon[92422]: pgmap v19291: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 983 KiB/s wr, 24 op/s 2023-03-10T00:45:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35216.483120 secs 2023-03-10T00:45:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35216.483120 secs 2023-03-10T00:45:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35216.483120 secs 2023-03-10T00:45:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:06 smithi122 ceph-mon[93869]: pgmap v19292: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 950 KiB/s rd, 765 KiB/s wr, 26 op/s 2023-03-10T00:45:07.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:06 smithi181 ceph-mon[94189]: pgmap v19292: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 950 KiB/s rd, 765 KiB/s wr, 26 op/s 2023-03-10T00:45:07.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:06 smithi038 ceph-mon[92422]: pgmap v19292: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 950 KiB/s rd, 765 KiB/s wr, 26 op/s 2023-03-10T00:45:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:45:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:45:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35218.946463 secs 2023-03-10T00:45:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:45:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:45:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35218.946463 secs 2023-03-10T00:45:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:45:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:45:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35218.946463 secs 2023-03-10T00:45:08.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35680.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:45:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:45:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:45:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:08 smithi122 ceph-mon[93869]: pgmap v19293: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1022 KiB/s wr, 23 op/s 2023-03-10T00:45:09.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:45:09.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:45:09.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:08 smithi181 ceph-mon[94189]: pgmap v19293: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1022 KiB/s wr, 23 op/s 2023-03-10T00:45:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:45:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:45:09.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:08 smithi038 ceph-mon[92422]: pgmap v19293: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1022 KiB/s wr, 23 op/s 2023-03-10T00:45:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35221.483201 secs 2023-03-10T00:45:11.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:11 smithi122 ceph-mon[93869]: pgmap v19294: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1022 KiB/s wr, 23 op/s 2023-03-10T00:45:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35221.483201 secs 2023-03-10T00:45:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:11 smithi181 ceph-mon[94189]: pgmap v19294: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1022 KiB/s wr, 23 op/s 2023-03-10T00:45:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35221.483201 secs 2023-03-10T00:45:11.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:11 smithi038 ceph-mon[92422]: pgmap v19294: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1022 KiB/s wr, 23 op/s 2023-03-10T00:45:12.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:45:12.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:45:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:45:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:45:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:45:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:45:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:13 smithi122 ceph-mon[93869]: pgmap v19295: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1006 KiB/s wr, 24 op/s 2023-03-10T00:45:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35223.946536 secs 2023-03-10T00:45:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:13 smithi181 ceph-mon[94189]: pgmap v19295: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1006 KiB/s wr, 24 op/s 2023-03-10T00:45:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35223.946536 secs 2023-03-10T00:45:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:13 smithi038 ceph-mon[92422]: pgmap v19295: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1006 KiB/s wr, 24 op/s 2023-03-10T00:45:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35223.946536 secs 2023-03-10T00:45:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35685.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:45:15.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:14 smithi122 ceph-mon[93869]: pgmap v19296: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 785 KiB/s wr, 20 op/s 2023-03-10T00:45:15.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:14 smithi181 ceph-mon[94189]: pgmap v19296: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 785 KiB/s wr, 20 op/s 2023-03-10T00:45:15.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:14 smithi038 ceph-mon[92422]: pgmap v19296: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 785 KiB/s wr, 20 op/s 2023-03-10T00:45:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35226.483283 secs 2023-03-10T00:45:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35226.483283 secs 2023-03-10T00:45:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35226.483283 secs 2023-03-10T00:45:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:16 smithi122 ceph-mon[93869]: pgmap v19297: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 637 KiB/s rd, 450 KiB/s wr, 10 op/s 2023-03-10T00:45:17.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:16 smithi181 ceph-mon[94189]: pgmap v19297: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 637 KiB/s rd, 450 KiB/s wr, 10 op/s 2023-03-10T00:45:17.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:16 smithi038 ceph-mon[92422]: pgmap v19297: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 637 KiB/s rd, 450 KiB/s wr, 10 op/s 2023-03-10T00:45:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35228.946614 secs 2023-03-10T00:45:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35228.946614 secs 2023-03-10T00:45:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35228.946614 secs 2023-03-10T00:45:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35690.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:45:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:18 smithi122 ceph-mon[93869]: pgmap v19298: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 608 KiB/s rd, 737 KiB/s wr, 10 op/s 2023-03-10T00:45:19.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:18 smithi181 ceph-mon[94189]: pgmap v19298: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 608 KiB/s rd, 737 KiB/s wr, 10 op/s 2023-03-10T00:45:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:18 smithi038 ceph-mon[92422]: pgmap v19298: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 608 KiB/s rd, 737 KiB/s wr, 10 op/s 2023-03-10T00:45:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:19 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:45:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:19 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:45:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:19 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:45:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:19 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:45:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:19 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:45:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:19 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:45:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:45:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:45:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35231.483367 secs 2023-03-10T00:45:21.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:20 smithi122 ceph-mon[93869]: pgmap v19299: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 453 KiB/s wr, 10 op/s 2023-03-10T00:45:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:45:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:45:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35231.483367 secs 2023-03-10T00:45:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:20 smithi181 ceph-mon[94189]: pgmap v19299: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 453 KiB/s wr, 10 op/s 2023-03-10T00:45:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:45:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:45:21.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35231.483367 secs 2023-03-10T00:45:21.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:20 smithi038 ceph-mon[92422]: pgmap v19299: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 453 KiB/s wr, 10 op/s 2023-03-10T00:45:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:23 smithi122 ceph-mon[93869]: pgmap v19300: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 464 KiB/s wr, 12 op/s 2023-03-10T00:45:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35233.946673 secs 2023-03-10T00:45:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:23 smithi181 ceph-mon[94189]: pgmap v19300: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 464 KiB/s wr, 12 op/s 2023-03-10T00:45:23.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35233.946673 secs 2023-03-10T00:45:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:23 smithi038 ceph-mon[92422]: pgmap v19300: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 464 KiB/s wr, 12 op/s 2023-03-10T00:45:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35233.946673 secs 2023-03-10T00:45:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35695.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:45:25.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:24 smithi122 ceph-mon[93869]: pgmap v19301: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 438 KiB/s wr, 11 op/s 2023-03-10T00:45:25.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:24 smithi181 ceph-mon[94189]: pgmap v19301: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 438 KiB/s wr, 11 op/s 2023-03-10T00:45:25.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:24 smithi038 ceph-mon[92422]: pgmap v19301: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 438 KiB/s wr, 11 op/s 2023-03-10T00:45:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:25 smithi122 ceph-mon[93869]: osdmap e1421: 12 total, 12 up, 12 in 2023-03-10T00:45:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35236.483456 secs 2023-03-10T00:45:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:25 smithi181 ceph-mon[94189]: osdmap e1421: 12 total, 12 up, 12 in 2023-03-10T00:45:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35236.483456 secs 2023-03-10T00:45:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:25 smithi038 ceph-mon[92422]: osdmap e1421: 12 total, 12 up, 12 in 2023-03-10T00:45:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35236.483456 secs 2023-03-10T00:45:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:26 smithi122 ceph-mon[93869]: pgmap v19303: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 430 KiB/s wr, 12 op/s 2023-03-10T00:45:27.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:26 smithi181 ceph-mon[94189]: pgmap v19303: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 430 KiB/s wr, 12 op/s 2023-03-10T00:45:27.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:26 smithi038 ceph-mon[92422]: pgmap v19303: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 430 KiB/s wr, 12 op/s 2023-03-10T00:45:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35238.946731 secs 2023-03-10T00:45:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35238.946731 secs 2023-03-10T00:45:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35238.946731 secs 2023-03-10T00:45:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35700.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:45:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:28 smithi122 ceph-mon[93869]: pgmap v19304: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 293 KiB/s wr, 11 op/s 2023-03-10T00:45:29.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:28 smithi181 ceph-mon[94189]: pgmap v19304: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 293 KiB/s wr, 11 op/s 2023-03-10T00:45:29.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:28 smithi038 ceph-mon[92422]: pgmap v19304: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 293 KiB/s wr, 11 op/s 2023-03-10T00:45:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35241.483539 secs 2023-03-10T00:45:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:31 smithi122 ceph-mon[93869]: pgmap v19305: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 293 KiB/s wr, 11 op/s 2023-03-10T00:45:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35241.483539 secs 2023-03-10T00:45:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:31 smithi181 ceph-mon[94189]: pgmap v19305: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 293 KiB/s wr, 11 op/s 2023-03-10T00:45:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35241.483539 secs 2023-03-10T00:45:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:31 smithi038 ceph-mon[92422]: pgmap v19305: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 293 KiB/s wr, 11 op/s 2023-03-10T00:45:33.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:33 smithi122 ceph-mon[93869]: pgmap v19306: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 280 KiB/s wr, 9 op/s 2023-03-10T00:45:33.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35243.946837 secs 2023-03-10T00:45:33.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:45:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:33 smithi181 ceph-mon[94189]: pgmap v19306: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 280 KiB/s wr, 9 op/s 2023-03-10T00:45:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35243.946837 secs 2023-03-10T00:45:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:45:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:33 smithi038 ceph-mon[92422]: pgmap v19306: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 280 KiB/s wr, 9 op/s 2023-03-10T00:45:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35243.946837 secs 2023-03-10T00:45:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:45:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35705.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:45:34.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:34 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:45:34.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:34 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:45:34.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:34 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:45:34.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:34 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:45:34.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:34 smithi122 ceph-mon[93869]: osdmap e1422: 12 total, 12 up, 12 in 2023-03-10T00:45:34.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:34 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:45:34.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:34 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:45:34.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:34 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:45:34.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:34 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:45:34.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:34 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:45:34.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:34 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:45:34.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:34 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:45:34.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:34 smithi181 ceph-mon[94189]: osdmap e1422: 12 total, 12 up, 12 in 2023-03-10T00:45:34.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:34 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:45:34.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:34 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:45:34.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:34 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:45:34.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:34 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:45:34.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:34 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:45:35.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:34 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:45:35.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:34 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:45:35.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:34 smithi038 ceph-mon[92422]: osdmap e1422: 12 total, 12 up, 12 in 2023-03-10T00:45:35.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:34 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:45:35.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:34 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:45:35.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:34 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:45:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:35 smithi122 ceph-mon[93869]: pgmap v19308: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 665 KiB/s rd, 420 KiB/s wr, 9 op/s 2023-03-10T00:45:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:35 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:45:35.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:35 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:45:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:35 smithi181 ceph-mon[94189]: pgmap v19308: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 665 KiB/s rd, 420 KiB/s wr, 9 op/s 2023-03-10T00:45:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:35 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:45:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:35 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:45:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:35 smithi038 ceph-mon[92422]: pgmap v19308: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 665 KiB/s rd, 420 KiB/s wr, 9 op/s 2023-03-10T00:45:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:35 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:45:36.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:35 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:45:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35246.483596 secs 2023-03-10T00:45:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35246.483596 secs 2023-03-10T00:45:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35246.483596 secs 2023-03-10T00:45:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:37 smithi122 ceph-mon[93869]: pgmap v19309: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 970 KiB/s wr, 11 op/s 2023-03-10T00:45:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:45:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:45:37.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35248.946921 secs 2023-03-10T00:45:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:37 smithi181 ceph-mon[94189]: pgmap v19309: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 970 KiB/s wr, 11 op/s 2023-03-10T00:45:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:45:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:45:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35248.946921 secs 2023-03-10T00:45:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:37 smithi038 ceph-mon[92422]: pgmap v19309: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 970 KiB/s wr, 11 op/s 2023-03-10T00:45:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:45:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:45:38.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35248.946921 secs 2023-03-10T00:45:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35710.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:45:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:39 smithi122 ceph-mon[93869]: pgmap v19310: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 937 KiB/s rd, 728 KiB/s wr, 11 op/s 2023-03-10T00:45:39.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:39 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:45:39.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:39 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:45:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:39 smithi181 ceph-mon[94189]: pgmap v19310: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 937 KiB/s rd, 728 KiB/s wr, 11 op/s 2023-03-10T00:45:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:39 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:45:39.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:39 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:45:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:39 smithi038 ceph-mon[92422]: pgmap v19310: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 937 KiB/s rd, 728 KiB/s wr, 11 op/s 2023-03-10T00:45:40.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:39 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:45:40.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:39 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:45:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35251.483654 secs 2023-03-10T00:45:41.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:41 smithi122 ceph-mon[93869]: pgmap v19311: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 904 KiB/s rd, 695 KiB/s wr, 10 op/s 2023-03-10T00:45:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35251.483654 secs 2023-03-10T00:45:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:41 smithi181 ceph-mon[94189]: pgmap v19311: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 904 KiB/s rd, 695 KiB/s wr, 10 op/s 2023-03-10T00:45:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35251.483654 secs 2023-03-10T00:45:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:41 smithi038 ceph-mon[92422]: pgmap v19311: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 904 KiB/s rd, 695 KiB/s wr, 10 op/s 2023-03-10T00:45:42.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35253.947019 secs 2023-03-10T00:45:42.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35253.947019 secs 2023-03-10T00:45:42.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35253.947019 secs 2023-03-10T00:45:43.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35715.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:45:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:43 smithi122 ceph-mon[93869]: pgmap v19312: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 855 KiB/s wr, 12 op/s 2023-03-10T00:45:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:43 smithi181 ceph-mon[94189]: pgmap v19312: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 855 KiB/s wr, 12 op/s 2023-03-10T00:45:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:43 smithi038 ceph-mon[92422]: pgmap v19312: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 855 KiB/s wr, 12 op/s 2023-03-10T00:45:45.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:44 smithi122 ceph-mon[93869]: pgmap v19313: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 761 KiB/s rd, 805 KiB/s wr, 11 op/s 2023-03-10T00:45:45.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:44 smithi181 ceph-mon[94189]: pgmap v19313: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 761 KiB/s rd, 805 KiB/s wr, 11 op/s 2023-03-10T00:45:45.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:44 smithi038 ceph-mon[92422]: pgmap v19313: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 761 KiB/s rd, 805 KiB/s wr, 11 op/s 2023-03-10T00:45:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35256.483741 secs 2023-03-10T00:45:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35256.483741 secs 2023-03-10T00:45:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35256.483741 secs 2023-03-10T00:45:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:46 smithi122 ceph-mon[93869]: pgmap v19314: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 884 KiB/s rd, 787 KiB/s wr, 12 op/s 2023-03-10T00:45:47.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:46 smithi181 ceph-mon[94189]: pgmap v19314: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 884 KiB/s rd, 787 KiB/s wr, 12 op/s 2023-03-10T00:45:47.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:46 smithi038 ceph-mon[92422]: pgmap v19314: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 884 KiB/s rd, 787 KiB/s wr, 12 op/s 2023-03-10T00:45:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:45:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:45:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:45:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:45:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:45:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:45:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35720.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:45:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:45:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:45:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:48 smithi122 ceph-mon[93869]: pgmap v19315: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 511 KiB/s rd, 304 KiB/s wr, 9 op/s 2023-03-10T00:45:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:45:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:45:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:48 smithi181 ceph-mon[94189]: pgmap v19315: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 511 KiB/s rd, 304 KiB/s wr, 9 op/s 2023-03-10T00:45:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:45:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:45:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:48 smithi038 ceph-mon[92422]: pgmap v19315: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 511 KiB/s rd, 304 KiB/s wr, 9 op/s 2023-03-10T00:45:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:49 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:45:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:49 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:45:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:49 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:45:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:50 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:45:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:50 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:45:51.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:50 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:45:51.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35261.483798 secs 2023-03-10T00:45:51.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:50 smithi122 ceph-mon[93869]: pgmap v19316: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 375 KiB/s wr, 9 op/s 2023-03-10T00:45:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:50 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:45:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:50 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:45:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:50 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:45:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35261.483798 secs 2023-03-10T00:45:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:50 smithi181 ceph-mon[94189]: pgmap v19316: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 375 KiB/s wr, 9 op/s 2023-03-10T00:45:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:50 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:45:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:50 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:45:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:50 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:45:51.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35261.483798 secs 2023-03-10T00:45:51.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:50 smithi038 ceph-mon[92422]: pgmap v19316: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 375 KiB/s wr, 9 op/s 2023-03-10T00:45:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:53 smithi122 ceph-mon[93869]: pgmap v19317: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 499 KiB/s rd, 385 KiB/s wr, 12 op/s 2023-03-10T00:45:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:53 smithi181 ceph-mon[94189]: pgmap v19317: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 499 KiB/s rd, 385 KiB/s wr, 12 op/s 2023-03-10T00:45:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:53 smithi038 ceph-mon[92422]: pgmap v19317: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 499 KiB/s rd, 385 KiB/s wr, 12 op/s 2023-03-10T00:45:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35725.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:45:55.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:54 smithi122 ceph-mon[93869]: pgmap v19318: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 583 KiB/s rd, 447 KiB/s wr, 10 op/s 2023-03-10T00:45:55.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:54 smithi181 ceph-mon[94189]: pgmap v19318: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 583 KiB/s rd, 447 KiB/s wr, 10 op/s 2023-03-10T00:45:55.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:54 smithi038 ceph-mon[92422]: pgmap v19318: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 583 KiB/s rd, 447 KiB/s wr, 10 op/s 2023-03-10T00:45:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:45:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:45:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35266.483855 secs 2023-03-10T00:45:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:45:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:45:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35266.483855 secs 2023-03-10T00:45:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:45:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:45:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35266.483855 secs 2023-03-10T00:45:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:45:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:45:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:56 smithi122 ceph-mon[93869]: pgmap v19319: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 518 KiB/s wr, 13 op/s 2023-03-10T00:45:57.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:45:57.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:45:57.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:56 smithi181 ceph-mon[94189]: pgmap v19319: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 518 KiB/s wr, 13 op/s 2023-03-10T00:45:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:45:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:45:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:56 smithi038 ceph-mon[92422]: pgmap v19319: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 518 KiB/s wr, 13 op/s 2023-03-10T00:45:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35730.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:45:59.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:45:59 smithi122 ceph-mon[93869]: pgmap v19320: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 424 KiB/s wr, 10 op/s 2023-03-10T00:45:59.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:45:59 smithi181 ceph-mon[94189]: pgmap v19320: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 424 KiB/s wr, 10 op/s 2023-03-10T00:45:59.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:45:59 smithi038 ceph-mon[92422]: pgmap v19320: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 424 KiB/s wr, 10 op/s 2023-03-10T00:46:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:46:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:46:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:46:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:46:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:46:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:46:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35271.483913 secs 2023-03-10T00:46:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:01 smithi038 ceph-mon[92422]: pgmap v19321: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 467 KiB/s rd, 452 KiB/s wr, 11 op/s 2023-03-10T00:46:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:46:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:46:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35271.483913 secs 2023-03-10T00:46:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:01 smithi122 ceph-mon[93869]: pgmap v19321: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 467 KiB/s rd, 452 KiB/s wr, 11 op/s 2023-03-10T00:46:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:46:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:46:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35271.483913 secs 2023-03-10T00:46:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:01 smithi181 ceph-mon[94189]: pgmap v19321: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 467 KiB/s rd, 452 KiB/s wr, 11 op/s 2023-03-10T00:46:01.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:46:01.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:46:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:03 smithi038 ceph-mon[92422]: pgmap v19322: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 948 KiB/s wr, 15 op/s 2023-03-10T00:46:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:03 smithi122 ceph-mon[93869]: pgmap v19322: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 948 KiB/s wr, 15 op/s 2023-03-10T00:46:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:03 smithi181 ceph-mon[94189]: pgmap v19322: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 948 KiB/s wr, 15 op/s 2023-03-10T00:46:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35735.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:46:05.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:04 smithi122 ceph-mon[93869]: pgmap v19323: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 815 KiB/s rd, 948 KiB/s wr, 23 op/s 2023-03-10T00:46:05.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:04 smithi181 ceph-mon[94189]: pgmap v19323: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 815 KiB/s rd, 948 KiB/s wr, 23 op/s 2023-03-10T00:46:05.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:04 smithi038 ceph-mon[92422]: pgmap v19323: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 815 KiB/s rd, 948 KiB/s wr, 23 op/s 2023-03-10T00:46:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35276.483970 secs 2023-03-10T00:46:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35276.483970 secs 2023-03-10T00:46:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35276.483970 secs 2023-03-10T00:46:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:06 smithi122 ceph-mon[93869]: pgmap v19324: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 970 KiB/s rd, 780 KiB/s wr, 26 op/s 2023-03-10T00:46:07.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:06 smithi181 ceph-mon[94189]: pgmap v19324: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 970 KiB/s rd, 780 KiB/s wr, 26 op/s 2023-03-10T00:46:07.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:06 smithi038 ceph-mon[92422]: pgmap v19324: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 970 KiB/s rd, 780 KiB/s wr, 26 op/s 2023-03-10T00:46:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35740.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:46:09.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:46:09.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:46:09.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:08 smithi122 ceph-mon[93869]: pgmap v19325: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 822 KiB/s rd, 635 KiB/s wr, 23 op/s 2023-03-10T00:46:09.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:46:09.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:46:09.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:08 smithi181 ceph-mon[94189]: pgmap v19325: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 822 KiB/s rd, 635 KiB/s wr, 23 op/s 2023-03-10T00:46:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:46:09.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:46:09.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:08 smithi038 ceph-mon[92422]: pgmap v19325: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 822 KiB/s rd, 635 KiB/s wr, 23 op/s 2023-03-10T00:46:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35281.484049 secs 2023-03-10T00:46:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:11 smithi122 ceph-mon[93869]: pgmap v19326: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 941 KiB/s rd, 755 KiB/s wr, 24 op/s 2023-03-10T00:46:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35281.484049 secs 2023-03-10T00:46:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:11 smithi181 ceph-mon[94189]: pgmap v19326: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 941 KiB/s rd, 755 KiB/s wr, 24 op/s 2023-03-10T00:46:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35281.484049 secs 2023-03-10T00:46:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:11 smithi038 ceph-mon[92422]: pgmap v19326: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 941 KiB/s rd, 755 KiB/s wr, 24 op/s 2023-03-10T00:46:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:46:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:46:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:46:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:46:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:46:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:46:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:13 smithi181 ceph-mon[94189]: pgmap v19327: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 903 KiB/s rd, 726 KiB/s wr, 24 op/s 2023-03-10T00:46:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:13 smithi038 ceph-mon[92422]: pgmap v19327: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 903 KiB/s rd, 726 KiB/s wr, 24 op/s 2023-03-10T00:46:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:13 smithi122 ceph-mon[93869]: pgmap v19327: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 903 KiB/s rd, 726 KiB/s wr, 24 op/s 2023-03-10T00:46:13.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35745.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:46:15.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:14 smithi122 ceph-mon[93869]: pgmap v19328: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 634 KiB/s rd, 315 KiB/s wr, 20 op/s 2023-03-10T00:46:15.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:14 smithi181 ceph-mon[94189]: pgmap v19328: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 634 KiB/s rd, 315 KiB/s wr, 20 op/s 2023-03-10T00:46:15.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:14 smithi038 ceph-mon[92422]: pgmap v19328: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 634 KiB/s rd, 315 KiB/s wr, 20 op/s 2023-03-10T00:46:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35286.484107 secs 2023-03-10T00:46:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35286.484107 secs 2023-03-10T00:46:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35286.484107 secs 2023-03-10T00:46:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:16 smithi122 ceph-mon[93869]: pgmap v19329: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 629 KiB/s rd, 304 KiB/s wr, 11 op/s 2023-03-10T00:46:17.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:16 smithi181 ceph-mon[94189]: pgmap v19329: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 629 KiB/s rd, 304 KiB/s wr, 11 op/s 2023-03-10T00:46:17.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:16 smithi038 ceph-mon[92422]: pgmap v19329: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 629 KiB/s rd, 304 KiB/s wr, 11 op/s 2023-03-10T00:46:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35750.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:46:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:19 smithi122 ceph-mon[93869]: pgmap v19330: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 286 KiB/s wr, 8 op/s 2023-03-10T00:46:19.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:19 smithi181 ceph-mon[94189]: pgmap v19330: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 286 KiB/s wr, 8 op/s 2023-03-10T00:46:19.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:19 smithi038 ceph-mon[92422]: pgmap v19330: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 286 KiB/s wr, 8 op/s 2023-03-10T00:46:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:46:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:46:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:46:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:46:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:46:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:46:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35291.484167 secs 2023-03-10T00:46:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:21 smithi038 ceph-mon[92422]: pgmap v19331: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 381 KiB/s wr, 9 op/s 2023-03-10T00:46:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:46:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:46:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35291.484167 secs 2023-03-10T00:46:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:21 smithi122 ceph-mon[93869]: pgmap v19331: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 381 KiB/s wr, 9 op/s 2023-03-10T00:46:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:46:21.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:46:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35291.484167 secs 2023-03-10T00:46:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:21 smithi181 ceph-mon[94189]: pgmap v19331: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 381 KiB/s wr, 9 op/s 2023-03-10T00:46:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:46:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:46:23.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:23 smithi038 ceph-mon[92422]: pgmap v19332: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 491 KiB/s wr, 12 op/s 2023-03-10T00:46:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:23 smithi122 ceph-mon[93869]: pgmap v19332: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 491 KiB/s wr, 12 op/s 2023-03-10T00:46:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:23 smithi181 ceph-mon[94189]: pgmap v19332: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 491 KiB/s wr, 12 op/s 2023-03-10T00:46:23.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35755.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:46:25.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:24 smithi122 ceph-mon[93869]: pgmap v19333: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 472 KiB/s rd, 491 KiB/s wr, 10 op/s 2023-03-10T00:46:25.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:24 smithi181 ceph-mon[94189]: pgmap v19333: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 472 KiB/s rd, 491 KiB/s wr, 10 op/s 2023-03-10T00:46:25.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:24 smithi038 ceph-mon[92422]: pgmap v19333: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 472 KiB/s rd, 491 KiB/s wr, 10 op/s 2023-03-10T00:46:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:25 smithi122 ceph-mon[93869]: osdmap e1423: 12 total, 12 up, 12 in 2023-03-10T00:46:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35296.484230 secs 2023-03-10T00:46:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:25 smithi181 ceph-mon[94189]: osdmap e1423: 12 total, 12 up, 12 in 2023-03-10T00:46:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35296.484230 secs 2023-03-10T00:46:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:25 smithi038 ceph-mon[92422]: osdmap e1423: 12 total, 12 up, 12 in 2023-03-10T00:46:26.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35296.484230 secs 2023-03-10T00:46:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:26 smithi122 ceph-mon[93869]: pgmap v19335: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 542 KiB/s rd, 566 KiB/s wr, 12 op/s 2023-03-10T00:46:27.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:26 smithi181 ceph-mon[94189]: pgmap v19335: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 542 KiB/s rd, 566 KiB/s wr, 12 op/s 2023-03-10T00:46:27.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:26 smithi038 ceph-mon[92422]: pgmap v19335: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 542 KiB/s rd, 566 KiB/s wr, 12 op/s 2023-03-10T00:46:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35760.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:46:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:29 smithi122 ceph-mon[93869]: pgmap v19336: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 541 KiB/s rd, 553 KiB/s wr, 11 op/s 2023-03-10T00:46:29.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:29 smithi181 ceph-mon[94189]: pgmap v19336: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 541 KiB/s rd, 553 KiB/s wr, 11 op/s 2023-03-10T00:46:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:29 smithi038 ceph-mon[92422]: pgmap v19336: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 541 KiB/s rd, 553 KiB/s wr, 11 op/s 2023-03-10T00:46:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35301.484314 secs 2023-03-10T00:46:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:31 smithi122 ceph-mon[93869]: pgmap v19337: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 472 KiB/s wr, 10 op/s 2023-03-10T00:46:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35301.484314 secs 2023-03-10T00:46:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:31 smithi181 ceph-mon[94189]: pgmap v19337: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 472 KiB/s wr, 10 op/s 2023-03-10T00:46:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35301.484314 secs 2023-03-10T00:46:31.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:31 smithi038 ceph-mon[92422]: pgmap v19337: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 472 KiB/s wr, 10 op/s 2023-03-10T00:46:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:33 smithi038 ceph-mon[92422]: pgmap v19338: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 196 KiB/s wr, 8 op/s 2023-03-10T00:46:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:33 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:46:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:33 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:46:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:33 smithi122 ceph-mon[93869]: pgmap v19338: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 196 KiB/s wr, 8 op/s 2023-03-10T00:46:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:33 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:46:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:33 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:46:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:33 smithi181 ceph-mon[94189]: pgmap v19338: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 196 KiB/s wr, 8 op/s 2023-03-10T00:46:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:33 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:46:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:33 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:46:33.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35765.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:46:35.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:34 smithi122 ceph-mon[93869]: osdmap e1424: 12 total, 12 up, 12 in 2023-03-10T00:46:35.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:34 smithi122 ceph-mon[93869]: pgmap v19340: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 211 KiB/s wr, 8 op/s 2023-03-10T00:46:35.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:34 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:46:35.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:34 smithi181 ceph-mon[94189]: osdmap e1424: 12 total, 12 up, 12 in 2023-03-10T00:46:35.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:34 smithi181 ceph-mon[94189]: pgmap v19340: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 211 KiB/s wr, 8 op/s 2023-03-10T00:46:35.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:34 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:46:35.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:34 smithi038 ceph-mon[92422]: osdmap e1424: 12 total, 12 up, 12 in 2023-03-10T00:46:35.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:34 smithi038 ceph-mon[92422]: pgmap v19340: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 211 KiB/s wr, 8 op/s 2023-03-10T00:46:35.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:34 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:46:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:35 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:46:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:35 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:46:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35306.484372 secs 2023-03-10T00:46:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:46:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:46:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:46:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:35 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:46:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:35 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:46:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35306.484372 secs 2023-03-10T00:46:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:46:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:46:36.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:46:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:35 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:46:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:35 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:46:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35306.484372 secs 2023-03-10T00:46:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:46:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:46:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:46:37.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:36 smithi122 ceph-mon[93869]: pgmap v19341: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 365 KiB/s wr, 8 op/s 2023-03-10T00:46:37.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:36 smithi181 ceph-mon[94189]: pgmap v19341: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 365 KiB/s wr, 8 op/s 2023-03-10T00:46:37.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:36 smithi038 ceph-mon[92422]: pgmap v19341: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 365 KiB/s wr, 8 op/s 2023-03-10T00:46:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35308.947950 secs 2023-03-10T00:46:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35308.947950 secs 2023-03-10T00:46:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35308.947950 secs 2023-03-10T00:46:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35770.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:46:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:38 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:46:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:38 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:46:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:38 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:46:39.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:38 smithi122 ceph-mon[93869]: pgmap v19342: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 365 KiB/s wr, 8 op/s 2023-03-10T00:46:39.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:38 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:46:39.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:38 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:46:39.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:38 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:46:39.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:38 smithi181 ceph-mon[94189]: pgmap v19342: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 365 KiB/s wr, 8 op/s 2023-03-10T00:46:39.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:38 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:46:39.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:38 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:46:39.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:38 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:46:39.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:38 smithi038 ceph-mon[92422]: pgmap v19342: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 365 KiB/s wr, 8 op/s 2023-03-10T00:46:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:46:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:46:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35311.484455 secs 2023-03-10T00:46:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:40 smithi122 ceph-mon[93869]: pgmap v19343: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 366 KiB/s wr, 9 op/s 2023-03-10T00:46:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:46:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:46:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35311.484455 secs 2023-03-10T00:46:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:40 smithi181 ceph-mon[94189]: pgmap v19343: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 366 KiB/s wr, 9 op/s 2023-03-10T00:46:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:46:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:46:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35311.484455 secs 2023-03-10T00:46:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:40 smithi038 ceph-mon[92422]: pgmap v19343: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 366 KiB/s wr, 9 op/s 2023-03-10T00:46:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:43 smithi181 ceph-mon[94189]: pgmap v19344: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 755 KiB/s rd, 753 KiB/s wr, 10 op/s 2023-03-10T00:46:43.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35313.948050 secs 2023-03-10T00:46:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:43 smithi038 ceph-mon[92422]: pgmap v19344: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 755 KiB/s rd, 753 KiB/s wr, 10 op/s 2023-03-10T00:46:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35313.948050 secs 2023-03-10T00:46:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:43 smithi122 ceph-mon[93869]: pgmap v19344: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 755 KiB/s rd, 753 KiB/s wr, 10 op/s 2023-03-10T00:46:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35313.948050 secs 2023-03-10T00:46:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35775.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:46:45.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:44 smithi122 ceph-mon[93869]: pgmap v19345: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 729 KiB/s wr, 9 op/s 2023-03-10T00:46:45.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:44 smithi181 ceph-mon[94189]: pgmap v19345: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 729 KiB/s wr, 9 op/s 2023-03-10T00:46:45.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:44 smithi038 ceph-mon[92422]: pgmap v19345: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 729 KiB/s wr, 9 op/s 2023-03-10T00:46:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35316.484516 secs 2023-03-10T00:46:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35316.484516 secs 2023-03-10T00:46:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35316.484516 secs 2023-03-10T00:46:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:46 smithi122 ceph-mon[93869]: pgmap v19346: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 775 KiB/s rd, 772 KiB/s wr, 12 op/s 2023-03-10T00:46:47.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:46 smithi181 ceph-mon[94189]: pgmap v19346: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 775 KiB/s rd, 772 KiB/s wr, 12 op/s 2023-03-10T00:46:47.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:46 smithi038 ceph-mon[92422]: pgmap v19346: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 775 KiB/s rd, 772 KiB/s wr, 12 op/s 2023-03-10T00:46:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:46:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:46:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:47 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 35318.948150 secs 2023-03-10T00:46:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:46:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:46:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:47 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 35318.948150 secs 2023-03-10T00:46:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:46:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:46:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:47 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 35318.948150 secs 2023-03-10T00:46:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35780.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:46:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:46:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:46:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:48 smithi122 ceph-mon[93869]: pgmap v19347: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 496 KiB/s wr, 9 op/s 2023-03-10T00:46:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:46:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:46:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:48 smithi181 ceph-mon[94189]: pgmap v19347: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 496 KiB/s wr, 9 op/s 2023-03-10T00:46:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:46:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:46:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:48 smithi038 ceph-mon[92422]: pgmap v19347: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 496 KiB/s wr, 9 op/s 2023-03-10T00:46:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:50 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:46:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:50 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:46:51.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35321.484574 secs 2023-03-10T00:46:51.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:50 smithi122 ceph-mon[93869]: pgmap v19348: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 496 KiB/s wr, 9 op/s 2023-03-10T00:46:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:50 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:46:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:50 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:46:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35321.484574 secs 2023-03-10T00:46:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:50 smithi181 ceph-mon[94189]: pgmap v19348: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 496 KiB/s wr, 9 op/s 2023-03-10T00:46:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:50 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:46:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:50 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:46:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35321.484574 secs 2023-03-10T00:46:51.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:50 smithi038 ceph-mon[92422]: pgmap v19348: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 496 KiB/s wr, 9 op/s 2023-03-10T00:46:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:46:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:46:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:46:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:46:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:46:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:46:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:52 smithi122 ceph-mon[93869]: pgmap v19349: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 745 KiB/s rd, 742 KiB/s wr, 11 op/s 2023-03-10T00:46:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:52 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 35323.948212 secs 2023-03-10T00:46:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:52 smithi181 ceph-mon[94189]: pgmap v19349: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 745 KiB/s rd, 742 KiB/s wr, 11 op/s 2023-03-10T00:46:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:52 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 35323.948212 secs 2023-03-10T00:46:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:52 smithi038 ceph-mon[92422]: pgmap v19349: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 745 KiB/s rd, 742 KiB/s wr, 11 op/s 2023-03-10T00:46:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:52 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 35323.948212 secs 2023-03-10T00:46:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35785.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:46:55.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:54 smithi122 ceph-mon[93869]: pgmap v19350: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 419 KiB/s wr, 8 op/s 2023-03-10T00:46:55.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:54 smithi181 ceph-mon[94189]: pgmap v19350: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 419 KiB/s wr, 8 op/s 2023-03-10T00:46:55.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:54 smithi038 ceph-mon[92422]: pgmap v19350: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 419 KiB/s wr, 8 op/s 2023-03-10T00:46:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:46:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:46:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35326.484659 secs 2023-03-10T00:46:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:46:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:46:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35326.484659 secs 2023-03-10T00:46:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:46:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:46:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35326.484659 secs 2023-03-10T00:46:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:46:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:46:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:56 smithi122 ceph-mon[93869]: pgmap v19351: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 475 KiB/s wr, 12 op/s 2023-03-10T00:46:57.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:46:57.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:46:57.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:56 smithi181 ceph-mon[94189]: pgmap v19351: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 475 KiB/s wr, 12 op/s 2023-03-10T00:46:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:46:57.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:46:57.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:56 smithi038 ceph-mon[92422]: pgmap v19351: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 475 KiB/s wr, 12 op/s 2023-03-10T00:46:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35328.948275 secs 2023-03-10T00:46:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35328.948275 secs 2023-03-10T00:46:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35328.948275 secs 2023-03-10T00:46:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35790.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:46:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:58 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:46:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:58 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:46:59.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:46:58 smithi122 ceph-mon[93869]: pgmap v19352: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 331 KiB/s wr, 9 op/s 2023-03-10T00:46:59.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:58 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:46:59.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:58 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:46:59.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:46:58 smithi181 ceph-mon[94189]: pgmap v19352: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 331 KiB/s wr, 9 op/s 2023-03-10T00:46:59.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:58 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:46:59.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:58 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:46:59.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:46:58 smithi038 ceph-mon[92422]: pgmap v19352: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 331 KiB/s wr, 9 op/s 2023-03-10T00:47:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35331.484744 secs 2023-03-10T00:47:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:01 smithi038 ceph-mon[92422]: pgmap v19353: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 359 KiB/s wr, 9 op/s 2023-03-10T00:47:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35331.484744 secs 2023-03-10T00:47:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:01 smithi122 ceph-mon[93869]: pgmap v19353: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 359 KiB/s wr, 9 op/s 2023-03-10T00:47:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35331.484744 secs 2023-03-10T00:47:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:01 smithi181 ceph-mon[94189]: pgmap v19353: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 359 KiB/s wr, 9 op/s 2023-03-10T00:47:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:03 smithi038 ceph-mon[92422]: pgmap v19354: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 847 KiB/s rd, 831 KiB/s wr, 12 op/s 2023-03-10T00:47:03.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35333.948335 secs 2023-03-10T00:47:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:03 smithi122 ceph-mon[93869]: pgmap v19354: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 847 KiB/s rd, 831 KiB/s wr, 12 op/s 2023-03-10T00:47:03.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35333.948335 secs 2023-03-10T00:47:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:03 smithi181 ceph-mon[94189]: pgmap v19354: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 847 KiB/s rd, 831 KiB/s wr, 12 op/s 2023-03-10T00:47:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35333.948335 secs 2023-03-10T00:47:03.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35795.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:47:05.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:04 smithi122 ceph-mon[93869]: pgmap v19355: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 578 KiB/s rd, 567 KiB/s wr, 20 op/s 2023-03-10T00:47:05.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:04 smithi181 ceph-mon[94189]: pgmap v19355: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 578 KiB/s rd, 567 KiB/s wr, 20 op/s 2023-03-10T00:47:05.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:04 smithi038 ceph-mon[92422]: pgmap v19355: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 578 KiB/s rd, 567 KiB/s wr, 20 op/s 2023-03-10T00:47:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35336.484831 secs 2023-03-10T00:47:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35336.484831 secs 2023-03-10T00:47:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35336.484831 secs 2023-03-10T00:47:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:06 smithi122 ceph-mon[93869]: pgmap v19356: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 786 KiB/s wr, 25 op/s 2023-03-10T00:47:07.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:06 smithi181 ceph-mon[94189]: pgmap v19356: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 786 KiB/s wr, 25 op/s 2023-03-10T00:47:07.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:06 smithi038 ceph-mon[92422]: pgmap v19356: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 786 KiB/s wr, 25 op/s 2023-03-10T00:47:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:47:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:47:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35338.948398 secs 2023-03-10T00:47:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:47:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:47:08.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35338.948398 secs 2023-03-10T00:47:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:47:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:47:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35338.948398 secs 2023-03-10T00:47:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35800.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:47:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:47:09.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:47:09.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:08 smithi122 ceph-mon[93869]: pgmap v19357: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 758 KiB/s wr, 23 op/s 2023-03-10T00:47:09.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:47:09.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:47:09.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:08 smithi181 ceph-mon[94189]: pgmap v19357: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 758 KiB/s wr, 23 op/s 2023-03-10T00:47:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:47:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:47:09.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:08 smithi038 ceph-mon[92422]: pgmap v19357: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 758 KiB/s wr, 23 op/s 2023-03-10T00:47:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35341.484887 secs 2023-03-10T00:47:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:11 smithi038 ceph-mon[92422]: pgmap v19358: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 758 KiB/s wr, 22 op/s 2023-03-10T00:47:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35341.484887 secs 2023-03-10T00:47:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:11 smithi122 ceph-mon[93869]: pgmap v19358: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 758 KiB/s wr, 22 op/s 2023-03-10T00:47:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35341.484887 secs 2023-03-10T00:47:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:11 smithi181 ceph-mon[94189]: pgmap v19358: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 758 KiB/s wr, 22 op/s 2023-03-10T00:47:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:47:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:47:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:47:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:47:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:47:12.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:47:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:13 smithi122 ceph-mon[93869]: pgmap v19359: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 730 KiB/s wr, 24 op/s 2023-03-10T00:47:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35343.948459 secs 2023-03-10T00:47:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:13 smithi181 ceph-mon[94189]: pgmap v19359: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 730 KiB/s wr, 24 op/s 2023-03-10T00:47:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35343.948459 secs 2023-03-10T00:47:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:13 smithi038 ceph-mon[92422]: pgmap v19359: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 730 KiB/s wr, 24 op/s 2023-03-10T00:47:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35343.948459 secs 2023-03-10T00:47:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35805.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:47:15.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:14 smithi122 ceph-mon[93869]: pgmap v19360: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 727 KiB/s rd, 414 KiB/s wr, 21 op/s 2023-03-10T00:47:15.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:14 smithi181 ceph-mon[94189]: pgmap v19360: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 727 KiB/s rd, 414 KiB/s wr, 21 op/s 2023-03-10T00:47:15.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:14 smithi038 ceph-mon[92422]: pgmap v19360: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 727 KiB/s rd, 414 KiB/s wr, 21 op/s 2023-03-10T00:47:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35346.484945 secs 2023-03-10T00:47:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35346.484945 secs 2023-03-10T00:47:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35346.484945 secs 2023-03-10T00:47:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:16 smithi122 ceph-mon[93869]: pgmap v19361: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 721 KiB/s rd, 403 KiB/s wr, 12 op/s 2023-03-10T00:47:17.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:16 smithi181 ceph-mon[94189]: pgmap v19361: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 721 KiB/s rd, 403 KiB/s wr, 12 op/s 2023-03-10T00:47:17.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:16 smithi038 ceph-mon[92422]: pgmap v19361: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 721 KiB/s rd, 403 KiB/s wr, 12 op/s 2023-03-10T00:47:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35348.948542 secs 2023-03-10T00:47:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35348.948542 secs 2023-03-10T00:47:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35348.948542 secs 2023-03-10T00:47:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35810.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:47:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:18 smithi122 ceph-mon[93869]: pgmap v19362: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 353 KiB/s wr, 9 op/s 2023-03-10T00:47:19.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:18 smithi181 ceph-mon[94189]: pgmap v19362: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 353 KiB/s wr, 9 op/s 2023-03-10T00:47:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:18 smithi038 ceph-mon[92422]: pgmap v19362: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 190 KiB/s rd, 353 KiB/s wr, 9 op/s 2023-03-10T00:47:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:47:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:47:21.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35351.485004 secs 2023-03-10T00:47:21.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:20 smithi122 ceph-mon[93869]: pgmap v19363: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 353 KiB/s wr, 8 op/s 2023-03-10T00:47:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:47:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:47:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35351.485004 secs 2023-03-10T00:47:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:20 smithi181 ceph-mon[94189]: pgmap v19363: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 353 KiB/s wr, 8 op/s 2023-03-10T00:47:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:47:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:47:21.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35351.485004 secs 2023-03-10T00:47:21.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:20 smithi038 ceph-mon[92422]: pgmap v19363: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 353 KiB/s wr, 8 op/s 2023-03-10T00:47:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:23 smithi038 ceph-mon[92422]: pgmap v19364: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 470 KiB/s wr, 11 op/s 2023-03-10T00:47:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35353.948620 secs 2023-03-10T00:47:23.548 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:23 smithi122 ceph-mon[93869]: pgmap v19364: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 470 KiB/s wr, 11 op/s 2023-03-10T00:47:23.548 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35353.948620 secs 2023-03-10T00:47:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:23 smithi181 ceph-mon[94189]: pgmap v19364: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 470 KiB/s wr, 11 op/s 2023-03-10T00:47:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35353.948620 secs 2023-03-10T00:47:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35815.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:47:25.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:24 smithi122 ceph-mon[93869]: pgmap v19365: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 470 KiB/s wr, 10 op/s 2023-03-10T00:47:25.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:24 smithi181 ceph-mon[94189]: pgmap v19365: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 470 KiB/s wr, 10 op/s 2023-03-10T00:47:25.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:24 smithi038 ceph-mon[92422]: pgmap v19365: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 470 KiB/s wr, 10 op/s 2023-03-10T00:47:26.149 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:25 smithi122 ceph-mon[93869]: osdmap e1425: 12 total, 12 up, 12 in 2023-03-10T00:47:26.149 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35356.485068 secs 2023-03-10T00:47:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:25 smithi181 ceph-mon[94189]: osdmap e1425: 12 total, 12 up, 12 in 2023-03-10T00:47:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35356.485068 secs 2023-03-10T00:47:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:25 smithi038 ceph-mon[92422]: osdmap e1425: 12 total, 12 up, 12 in 2023-03-10T00:47:26.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35356.485068 secs 2023-03-10T00:47:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:26 smithi122 ceph-mon[93869]: pgmap v19367: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 378 KiB/s wr, 12 op/s 2023-03-10T00:47:27.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:26 smithi181 ceph-mon[94189]: pgmap v19367: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 378 KiB/s wr, 12 op/s 2023-03-10T00:47:27.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:26 smithi038 ceph-mon[92422]: pgmap v19367: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 378 KiB/s wr, 12 op/s 2023-03-10T00:47:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:27 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:47:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:27 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:47:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35358.948680 secs 2023-03-10T00:47:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:27 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:47:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:27 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:47:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35358.948680 secs 2023-03-10T00:47:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:27 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:47:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:27 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:47:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35358.948680 secs 2023-03-10T00:47:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35820.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:47:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:28 smithi122 ceph-mon[93869]: pgmap v19368: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 541 KiB/s rd, 366 KiB/s wr, 10 op/s 2023-03-10T00:47:29.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:28 smithi181 ceph-mon[94189]: pgmap v19368: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 541 KiB/s rd, 366 KiB/s wr, 10 op/s 2023-03-10T00:47:29.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:28 smithi038 ceph-mon[92422]: pgmap v19368: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 541 KiB/s rd, 366 KiB/s wr, 10 op/s 2023-03-10T00:47:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35361.485128 secs 2023-03-10T00:47:31.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:31 smithi038 ceph-mon[92422]: pgmap v19369: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 541 KiB/s rd, 367 KiB/s wr, 10 op/s 2023-03-10T00:47:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35361.485128 secs 2023-03-10T00:47:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:31 smithi122 ceph-mon[93869]: pgmap v19369: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 541 KiB/s rd, 367 KiB/s wr, 10 op/s 2023-03-10T00:47:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35361.485128 secs 2023-03-10T00:47:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:31 smithi181 ceph-mon[94189]: pgmap v19369: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 541 KiB/s rd, 367 KiB/s wr, 10 op/s 2023-03-10T00:47:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:33 smithi038 ceph-mon[92422]: pgmap v19370: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 690 KiB/s rd, 318 KiB/s wr, 9 op/s 2023-03-10T00:47:33.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35363.948762 secs 2023-03-10T00:47:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:33 smithi122 ceph-mon[93869]: pgmap v19370: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 690 KiB/s rd, 318 KiB/s wr, 9 op/s 2023-03-10T00:47:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35363.948762 secs 2023-03-10T00:47:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:33 smithi181 ceph-mon[94189]: pgmap v19370: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 690 KiB/s rd, 318 KiB/s wr, 9 op/s 2023-03-10T00:47:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35363.948762 secs 2023-03-10T00:47:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35825.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:47:35.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:34 smithi122 ceph-mon[93869]: osdmap e1426: 12 total, 12 up, 12 in 2023-03-10T00:47:35.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:34 smithi122 ceph-mon[93869]: pgmap v19372: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 540 KiB/s rd, 341 KiB/s wr, 8 op/s 2023-03-10T00:47:35.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:34 smithi181 ceph-mon[94189]: osdmap e1426: 12 total, 12 up, 12 in 2023-03-10T00:47:35.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:34 smithi181 ceph-mon[94189]: pgmap v19372: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 540 KiB/s rd, 341 KiB/s wr, 8 op/s 2023-03-10T00:47:35.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:34 smithi038 ceph-mon[92422]: osdmap e1426: 12 total, 12 up, 12 in 2023-03-10T00:47:35.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:34 smithi038 ceph-mon[92422]: pgmap v19372: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 540 KiB/s rd, 341 KiB/s wr, 8 op/s 2023-03-10T00:47:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:35 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:47:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:35 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:47:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35366.485191 secs 2023-03-10T00:47:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:47:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:35 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:47:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:35 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:47:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35366.485191 secs 2023-03-10T00:47:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:47:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:35 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:47:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:35 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:47:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35366.485191 secs 2023-03-10T00:47:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:47:37.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:47:37.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:47:37.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:47:37.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:47:37.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:47:37.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:36 smithi122 ceph-mon[93869]: pgmap v19373: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 537 KiB/s rd, 351 KiB/s wr, 9 op/s 2023-03-10T00:47:37.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:47:37.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:47:37.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:47:37.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:47:37.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:47:37.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:36 smithi181 ceph-mon[94189]: pgmap v19373: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 537 KiB/s rd, 351 KiB/s wr, 9 op/s 2023-03-10T00:47:37.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:47:37.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:47:37.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:47:37.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:47:37.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:47:37.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:36 smithi038 ceph-mon[92422]: pgmap v19373: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 537 KiB/s rd, 351 KiB/s wr, 9 op/s 2023-03-10T00:47:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35368.948847 secs 2023-03-10T00:47:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35368.948847 secs 2023-03-10T00:47:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35368.948847 secs 2023-03-10T00:47:38.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35830.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:47:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:39 smithi122 ceph-mon[93869]: pgmap v19374: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1011 KiB/s rd, 733 KiB/s wr, 10 op/s 2023-03-10T00:47:39.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:39 smithi181 ceph-mon[94189]: pgmap v19374: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1011 KiB/s rd, 733 KiB/s wr, 10 op/s 2023-03-10T00:47:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:39 smithi038 ceph-mon[92422]: pgmap v19374: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1011 KiB/s rd, 733 KiB/s wr, 10 op/s 2023-03-10T00:47:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35371.485245 secs 2023-03-10T00:47:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:41 smithi122 ceph-mon[93869]: pgmap v19375: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 977 KiB/s rd, 699 KiB/s wr, 10 op/s 2023-03-10T00:47:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35371.485245 secs 2023-03-10T00:47:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:41 smithi181 ceph-mon[94189]: pgmap v19375: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 977 KiB/s rd, 699 KiB/s wr, 10 op/s 2023-03-10T00:47:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35371.485245 secs 2023-03-10T00:47:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:41 smithi038 ceph-mon[92422]: pgmap v19375: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 977 KiB/s rd, 699 KiB/s wr, 10 op/s 2023-03-10T00:47:43.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:43 smithi122 ceph-mon[93869]: pgmap v19376: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 863 KiB/s rd, 769 KiB/s wr, 10 op/s 2023-03-10T00:47:43.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35373.948909 secs 2023-03-10T00:47:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:43 smithi181 ceph-mon[94189]: pgmap v19376: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 863 KiB/s rd, 769 KiB/s wr, 10 op/s 2023-03-10T00:47:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35373.948909 secs 2023-03-10T00:47:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:43 smithi038 ceph-mon[92422]: pgmap v19376: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 863 KiB/s rd, 769 KiB/s wr, 10 op/s 2023-03-10T00:47:43.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35373.948909 secs 2023-03-10T00:47:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35835.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:47:45.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:44 smithi122 ceph-mon[93869]: pgmap v19377: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 836 KiB/s rd, 745 KiB/s wr, 10 op/s 2023-03-10T00:47:45.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:44 smithi181 ceph-mon[94189]: pgmap v19377: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 836 KiB/s rd, 745 KiB/s wr, 10 op/s 2023-03-10T00:47:45.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:44 smithi038 ceph-mon[92422]: pgmap v19377: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 836 KiB/s rd, 745 KiB/s wr, 10 op/s 2023-03-10T00:47:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35376.485304 secs 2023-03-10T00:47:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35376.485304 secs 2023-03-10T00:47:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35376.485304 secs 2023-03-10T00:47:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:46 smithi122 ceph-mon[93869]: pgmap v19378: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 740 KiB/s wr, 11 op/s 2023-03-10T00:47:47.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:46 smithi181 ceph-mon[94189]: pgmap v19378: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 740 KiB/s wr, 11 op/s 2023-03-10T00:47:47.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:46 smithi038 ceph-mon[92422]: pgmap v19378: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 740 KiB/s wr, 11 op/s 2023-03-10T00:47:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:47:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:47:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35378.948993 secs 2023-03-10T00:47:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:47:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:47:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35378.948993 secs 2023-03-10T00:47:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:47:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:47:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35378.948993 secs 2023-03-10T00:47:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35840.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:47:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:47:49.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:47:49.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:48 smithi122 ceph-mon[93869]: pgmap v19379: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 862 KiB/s rd, 783 KiB/s wr, 9 op/s 2023-03-10T00:47:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:47:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:47:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:48 smithi181 ceph-mon[94189]: pgmap v19379: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 862 KiB/s rd, 783 KiB/s wr, 9 op/s 2023-03-10T00:47:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:47:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:47:49.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:48 smithi038 ceph-mon[92422]: pgmap v19379: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 862 KiB/s rd, 783 KiB/s wr, 9 op/s 2023-03-10T00:47:51.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35381.485361 secs 2023-03-10T00:47:51.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:51 smithi038 ceph-mon[92422]: pgmap v19380: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 340 KiB/s wr, 8 op/s 2023-03-10T00:47:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35381.485361 secs 2023-03-10T00:47:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:51 smithi122 ceph-mon[93869]: pgmap v19380: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 340 KiB/s wr, 8 op/s 2023-03-10T00:47:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35381.485361 secs 2023-03-10T00:47:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:51 smithi181 ceph-mon[94189]: pgmap v19380: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 340 KiB/s wr, 8 op/s 2023-03-10T00:47:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:53 smithi122 ceph-mon[93869]: pgmap v19381: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 378 KiB/s wr, 11 op/s 2023-03-10T00:47:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35383.949083 secs 2023-03-10T00:47:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:53 smithi181 ceph-mon[94189]: pgmap v19381: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 378 KiB/s wr, 11 op/s 2023-03-10T00:47:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35383.949083 secs 2023-03-10T00:47:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:53 smithi038 ceph-mon[92422]: pgmap v19381: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 378 KiB/s wr, 11 op/s 2023-03-10T00:47:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35383.949083 secs 2023-03-10T00:47:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35845.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:47:55.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:54 smithi122 ceph-mon[93869]: pgmap v19382: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 242 KiB/s wr, 8 op/s 2023-03-10T00:47:55.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:54 smithi181 ceph-mon[94189]: pgmap v19382: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 242 KiB/s wr, 8 op/s 2023-03-10T00:47:55.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:54 smithi038 ceph-mon[92422]: pgmap v19382: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 242 KiB/s wr, 8 op/s 2023-03-10T00:47:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:47:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:47:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35386.485420 secs 2023-03-10T00:47:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:47:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:47:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35386.485420 secs 2023-03-10T00:47:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:47:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:47:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35386.485420 secs 2023-03-10T00:47:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:47:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:47:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:56 smithi122 ceph-mon[93869]: pgmap v19383: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 514 KiB/s rd, 508 KiB/s wr, 11 op/s 2023-03-10T00:47:57.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:47:57.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:47:57.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:56 smithi181 ceph-mon[94189]: pgmap v19383: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 514 KiB/s rd, 508 KiB/s wr, 11 op/s 2023-03-10T00:47:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:47:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:47:57.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:56 smithi038 ceph-mon[92422]: pgmap v19383: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 514 KiB/s rd, 508 KiB/s wr, 11 op/s 2023-03-10T00:47:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:57 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:47:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:57 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:47:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:57 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:47:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35850.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:47:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:58 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:47:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:58 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:47:59.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:58 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:47:59.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:58 smithi122 ceph-mon[93869]: pgmap v19384: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 523 KiB/s rd, 519 KiB/s wr, 11 op/s 2023-03-10T00:47:59.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:58 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:47:59.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:58 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:47:59.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:58 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:47:59.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:58 smithi181 ceph-mon[94189]: pgmap v19384: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 523 KiB/s rd, 519 KiB/s wr, 11 op/s 2023-03-10T00:47:59.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:58 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:47:59.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:58 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:47:59.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:58 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:47:59.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:58 smithi038 ceph-mon[92422]: pgmap v19384: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 523 KiB/s rd, 519 KiB/s wr, 11 op/s 2023-03-10T00:48:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:48:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:47:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:48:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:48:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:47:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:48:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:48:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:47:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:48:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:48:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:48:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35391.485479 secs 2023-03-10T00:48:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:00 smithi122 ceph-mon[93869]: pgmap v19385: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 477 KiB/s wr, 11 op/s 2023-03-10T00:48:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:48:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:48:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35391.485479 secs 2023-03-10T00:48:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:00 smithi181 ceph-mon[94189]: pgmap v19385: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 477 KiB/s wr, 11 op/s 2023-03-10T00:48:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:48:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:48:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35391.485479 secs 2023-03-10T00:48:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:00 smithi038 ceph-mon[92422]: pgmap v19385: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 477 KiB/s wr, 11 op/s 2023-03-10T00:48:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:03 smithi122 ceph-mon[93869]: pgmap v19386: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 712 KiB/s rd, 696 KiB/s wr, 12 op/s 2023-03-10T00:48:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:03 smithi181 ceph-mon[94189]: pgmap v19386: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 712 KiB/s rd, 696 KiB/s wr, 12 op/s 2023-03-10T00:48:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:03 smithi038 ceph-mon[92422]: pgmap v19386: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 712 KiB/s rd, 696 KiB/s wr, 12 op/s 2023-03-10T00:48:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35855.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:48:05.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:04 smithi122 ceph-mon[93869]: pgmap v19387: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1012 KiB/s rd, 952 KiB/s wr, 22 op/s 2023-03-10T00:48:05.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:04 smithi181 ceph-mon[94189]: pgmap v19387: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1012 KiB/s rd, 952 KiB/s wr, 22 op/s 2023-03-10T00:48:05.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:04 smithi038 ceph-mon[92422]: pgmap v19387: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1012 KiB/s rd, 952 KiB/s wr, 22 op/s 2023-03-10T00:48:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35396.485540 secs 2023-03-10T00:48:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35396.485540 secs 2023-03-10T00:48:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35396.485540 secs 2023-03-10T00:48:07.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:06 smithi122 ceph-mon[93869]: pgmap v19388: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 26 op/s 2023-03-10T00:48:07.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:06 smithi181 ceph-mon[94189]: pgmap v19388: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 26 op/s 2023-03-10T00:48:07.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:06 smithi038 ceph-mon[92422]: pgmap v19388: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 26 op/s 2023-03-10T00:48:08.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35860.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:48:09.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:48:09.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:48:09.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:08 smithi181 ceph-mon[94189]: pgmap v19389: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 932 KiB/s rd, 785 KiB/s wr, 24 op/s 2023-03-10T00:48:09.233 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:48:09.233 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:48:09.233 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:08 smithi122 ceph-mon[93869]: pgmap v19389: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 932 KiB/s rd, 785 KiB/s wr, 24 op/s 2023-03-10T00:48:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:48:09.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:48:09.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:08 smithi038 ceph-mon[92422]: pgmap v19389: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 932 KiB/s rd, 785 KiB/s wr, 24 op/s 2023-03-10T00:48:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35401.485599 secs 2023-03-10T00:48:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:11 smithi122 ceph-mon[93869]: pgmap v19390: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 822 KiB/s rd, 675 KiB/s wr, 22 op/s 2023-03-10T00:48:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35401.485599 secs 2023-03-10T00:48:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:11 smithi181 ceph-mon[94189]: pgmap v19390: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 822 KiB/s rd, 675 KiB/s wr, 22 op/s 2023-03-10T00:48:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35401.485599 secs 2023-03-10T00:48:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:11 smithi038 ceph-mon[92422]: pgmap v19390: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 822 KiB/s rd, 675 KiB/s wr, 22 op/s 2023-03-10T00:48:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:48:12.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:48:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:48:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:48:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:48:12.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:48:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:13 smithi122 ceph-mon[93869]: pgmap v19391: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 938 KiB/s wr, 25 op/s 2023-03-10T00:48:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:13 smithi181 ceph-mon[94189]: pgmap v19391: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 938 KiB/s wr, 25 op/s 2023-03-10T00:48:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:13 smithi038 ceph-mon[92422]: pgmap v19391: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 938 KiB/s wr, 25 op/s 2023-03-10T00:48:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35865.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:48:15.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:14 smithi122 ceph-mon[93869]: pgmap v19392: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 822 KiB/s rd, 686 KiB/s wr, 22 op/s 2023-03-10T00:48:15.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:14 smithi181 ceph-mon[94189]: pgmap v19392: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 822 KiB/s rd, 686 KiB/s wr, 22 op/s 2023-03-10T00:48:15.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:14 smithi038 ceph-mon[92422]: pgmap v19392: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 822 KiB/s rd, 686 KiB/s wr, 22 op/s 2023-03-10T00:48:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35406.485669 secs 2023-03-10T00:48:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35406.485669 secs 2023-03-10T00:48:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35406.485669 secs 2023-03-10T00:48:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:16 smithi122 ceph-mon[93869]: pgmap v19393: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 483 KiB/s rd, 392 KiB/s wr, 13 op/s 2023-03-10T00:48:17.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:16 smithi181 ceph-mon[94189]: pgmap v19393: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 483 KiB/s rd, 392 KiB/s wr, 13 op/s 2023-03-10T00:48:17.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:16 smithi038 ceph-mon[92422]: pgmap v19393: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 483 KiB/s rd, 392 KiB/s wr, 13 op/s 2023-03-10T00:48:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35870.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:48:19.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:19 smithi122 ceph-mon[93869]: pgmap v19394: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 303 KiB/s wr, 11 op/s 2023-03-10T00:48:19.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:19 smithi181 ceph-mon[94189]: pgmap v19394: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 303 KiB/s wr, 11 op/s 2023-03-10T00:48:19.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:19 smithi038 ceph-mon[92422]: pgmap v19394: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 394 KiB/s rd, 303 KiB/s wr, 11 op/s 2023-03-10T00:48:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:48:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:48:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:48:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:48:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:48:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:48:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35411.485728 secs 2023-03-10T00:48:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:21 smithi122 ceph-mon[93869]: pgmap v19395: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 444 KiB/s rd, 452 KiB/s wr, 11 op/s 2023-03-10T00:48:21.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:48:21.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:48:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35411.485728 secs 2023-03-10T00:48:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:21 smithi181 ceph-mon[94189]: pgmap v19395: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 444 KiB/s rd, 452 KiB/s wr, 11 op/s 2023-03-10T00:48:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:48:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:48:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35411.485728 secs 2023-03-10T00:48:21.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:21 smithi038 ceph-mon[92422]: pgmap v19395: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 444 KiB/s rd, 452 KiB/s wr, 11 op/s 2023-03-10T00:48:21.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:48:21.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:48:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:23 smithi122 ceph-mon[93869]: pgmap v19396: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 748 KiB/s rd, 766 KiB/s wr, 14 op/s 2023-03-10T00:48:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:23 smithi181 ceph-mon[94189]: pgmap v19396: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 748 KiB/s rd, 766 KiB/s wr, 14 op/s 2023-03-10T00:48:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:23 smithi038 ceph-mon[92422]: pgmap v19396: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 748 KiB/s rd, 766 KiB/s wr, 14 op/s 2023-03-10T00:48:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35875.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:48:25.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:24 smithi122 ceph-mon[93869]: pgmap v19397: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 455 KiB/s rd, 475 KiB/s wr, 11 op/s 2023-03-10T00:48:25.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:24 smithi181 ceph-mon[94189]: pgmap v19397: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 455 KiB/s rd, 475 KiB/s wr, 11 op/s 2023-03-10T00:48:25.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:24 smithi038 ceph-mon[92422]: pgmap v19397: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 455 KiB/s rd, 475 KiB/s wr, 11 op/s 2023-03-10T00:48:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:25 smithi122 ceph-mon[93869]: osdmap e1427: 12 total, 12 up, 12 in 2023-03-10T00:48:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35416.485797 secs 2023-03-10T00:48:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:25 smithi181 ceph-mon[94189]: osdmap e1427: 12 total, 12 up, 12 in 2023-03-10T00:48:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35416.485797 secs 2023-03-10T00:48:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:25 smithi038 ceph-mon[92422]: osdmap e1427: 12 total, 12 up, 12 in 2023-03-10T00:48:26.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35416.485797 secs 2023-03-10T00:48:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:26 smithi122 ceph-mon[93869]: pgmap v19399: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 709 KiB/s rd, 733 KiB/s wr, 13 op/s 2023-03-10T00:48:27.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:26 smithi181 ceph-mon[94189]: pgmap v19399: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 709 KiB/s rd, 733 KiB/s wr, 13 op/s 2023-03-10T00:48:27.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:26 smithi038 ceph-mon[92422]: pgmap v19399: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 709 KiB/s rd, 733 KiB/s wr, 13 op/s 2023-03-10T00:48:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35880.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:48:29.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:29 smithi122 ceph-mon[93869]: pgmap v19400: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 708 KiB/s rd, 720 KiB/s wr, 11 op/s 2023-03-10T00:48:29.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:29 smithi181 ceph-mon[94189]: pgmap v19400: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 708 KiB/s rd, 720 KiB/s wr, 11 op/s 2023-03-10T00:48:29.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:29 smithi038 ceph-mon[92422]: pgmap v19400: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 708 KiB/s rd, 720 KiB/s wr, 11 op/s 2023-03-10T00:48:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35421.485901 secs 2023-03-10T00:48:31.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:31 smithi122 ceph-mon[93869]: pgmap v19401: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 667 KiB/s rd, 678 KiB/s wr, 11 op/s 2023-03-10T00:48:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35421.485901 secs 2023-03-10T00:48:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:31 smithi181 ceph-mon[94189]: pgmap v19401: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 667 KiB/s rd, 678 KiB/s wr, 11 op/s 2023-03-10T00:48:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35421.485901 secs 2023-03-10T00:48:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:31 smithi038 ceph-mon[92422]: pgmap v19401: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 667 KiB/s rd, 678 KiB/s wr, 11 op/s 2023-03-10T00:48:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:33 smithi122 ceph-mon[93869]: pgmap v19402: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 302 KiB/s wr, 9 op/s 2023-03-10T00:48:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:33 smithi181 ceph-mon[94189]: pgmap v19402: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 302 KiB/s wr, 9 op/s 2023-03-10T00:48:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:33 smithi038 ceph-mon[92422]: pgmap v19402: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 302 KiB/s wr, 9 op/s 2023-03-10T00:48:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35885.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:48:35.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:34 smithi122 ceph-mon[93869]: osdmap e1428: 12 total, 12 up, 12 in 2023-03-10T00:48:35.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:34 smithi122 ceph-mon[93869]: pgmap v19404: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 324 KiB/s wr, 9 op/s 2023-03-10T00:48:35.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:34 smithi181 ceph-mon[94189]: osdmap e1428: 12 total, 12 up, 12 in 2023-03-10T00:48:35.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:34 smithi181 ceph-mon[94189]: pgmap v19404: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 324 KiB/s wr, 9 op/s 2023-03-10T00:48:35.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:34 smithi038 ceph-mon[92422]: osdmap e1428: 12 total, 12 up, 12 in 2023-03-10T00:48:35.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:34 smithi038 ceph-mon[92422]: pgmap v19404: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 324 KiB/s wr, 9 op/s 2023-03-10T00:48:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:35 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:48:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:35 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:48:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35426.485984 secs 2023-03-10T00:48:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:35 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:48:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:35 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:48:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35426.485984 secs 2023-03-10T00:48:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:35 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:48:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:35 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:48:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35426.485984 secs 2023-03-10T00:48:37.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:48:37.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:48:37.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:48:37.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:36 smithi122 ceph-mon[93869]: pgmap v19405: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 883 KiB/s rd, 573 KiB/s wr, 9 op/s 2023-03-10T00:48:37.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:48:37.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:48:37.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:48:37.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:48:37.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:48:37.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:48:37.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:36 smithi181 ceph-mon[94189]: pgmap v19405: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 883 KiB/s rd, 573 KiB/s wr, 9 op/s 2023-03-10T00:48:37.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:48:37.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:48:37.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:48:37.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:48:37.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:48:37.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:48:37.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:36 smithi038 ceph-mon[92422]: pgmap v19405: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 883 KiB/s rd, 573 KiB/s wr, 9 op/s 2023-03-10T00:48:37.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:48:37.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:48:37.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:48:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35428.949743 secs 2023-03-10T00:48:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35428.949743 secs 2023-03-10T00:48:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35428.949743 secs 2023-03-10T00:48:38.801 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35890.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:48:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:38 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:48:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:38 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:48:39.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:38 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:48:39.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:38 smithi122 ceph-mon[93869]: pgmap v19406: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 883 KiB/s rd, 573 KiB/s wr, 9 op/s 2023-03-10T00:48:39.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:38 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:48:39.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:38 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:48:39.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:38 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:48:39.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:38 smithi181 ceph-mon[94189]: pgmap v19406: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 883 KiB/s rd, 573 KiB/s wr, 9 op/s 2023-03-10T00:48:39.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:38 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:48:39.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:38 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:48:39.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:38 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:48:39.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:38 smithi038 ceph-mon[92422]: pgmap v19406: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 883 KiB/s rd, 573 KiB/s wr, 9 op/s 2023-03-10T00:48:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35431.486009 secs 2023-03-10T00:48:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:41 smithi122 ceph-mon[93869]: pgmap v19407: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 753 KiB/s wr, 10 op/s 2023-03-10T00:48:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35431.486009 secs 2023-03-10T00:48:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:41 smithi181 ceph-mon[94189]: pgmap v19407: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 753 KiB/s wr, 10 op/s 2023-03-10T00:48:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35431.486009 secs 2023-03-10T00:48:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:41 smithi038 ceph-mon[92422]: pgmap v19407: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 753 KiB/s wr, 10 op/s 2023-03-10T00:48:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:43 smithi122 ceph-mon[93869]: pgmap v19408: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 762 KiB/s wr, 10 op/s 2023-03-10T00:48:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35433.949802 secs 2023-03-10T00:48:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:43 smithi181 ceph-mon[94189]: pgmap v19408: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 762 KiB/s wr, 10 op/s 2023-03-10T00:48:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35433.949802 secs 2023-03-10T00:48:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:43 smithi038 ceph-mon[92422]: pgmap v19408: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 762 KiB/s wr, 10 op/s 2023-03-10T00:48:43.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35433.949802 secs 2023-03-10T00:48:43.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35895.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:48:45.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:44 smithi122 ceph-mon[93869]: pgmap v19409: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 848 KiB/s wr, 10 op/s 2023-03-10T00:48:45.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:44 smithi181 ceph-mon[94189]: pgmap v19409: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 848 KiB/s wr, 10 op/s 2023-03-10T00:48:45.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:44 smithi038 ceph-mon[92422]: pgmap v19409: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 848 KiB/s wr, 10 op/s 2023-03-10T00:48:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35436.486064 secs 2023-03-10T00:48:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35436.486064 secs 2023-03-10T00:48:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35436.486064 secs 2023-03-10T00:48:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:46 smithi122 ceph-mon[93869]: pgmap v19410: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 972 KiB/s rd, 806 KiB/s wr, 12 op/s 2023-03-10T00:48:47.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:46 smithi181 ceph-mon[94189]: pgmap v19410: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 972 KiB/s rd, 806 KiB/s wr, 12 op/s 2023-03-10T00:48:47.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:46 smithi038 ceph-mon[92422]: pgmap v19410: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 972 KiB/s rd, 806 KiB/s wr, 12 op/s 2023-03-10T00:48:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:48:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:48:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:48:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:48:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:48:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:48:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35900.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:48:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:48:49.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:48:49.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:48 smithi122 ceph-mon[93869]: pgmap v19411: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 449 KiB/s wr, 9 op/s 2023-03-10T00:48:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:48:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:48:49.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:48 smithi181 ceph-mon[94189]: pgmap v19411: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 449 KiB/s wr, 9 op/s 2023-03-10T00:48:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:48:49.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:48:49.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:48 smithi038 ceph-mon[92422]: pgmap v19411: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 449 KiB/s wr, 9 op/s 2023-03-10T00:48:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:49 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:48:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:49 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:48:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:49 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:48:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:50 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:48:51.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:50 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:48:51.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:50 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:48:51.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35441.486123 secs 2023-03-10T00:48:51.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:50 smithi122 ceph-mon[93869]: pgmap v19412: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 477 KiB/s wr, 10 op/s 2023-03-10T00:48:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:50 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:48:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:50 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:48:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:50 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:48:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35441.486123 secs 2023-03-10T00:48:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:50 smithi181 ceph-mon[94189]: pgmap v19412: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 477 KiB/s wr, 10 op/s 2023-03-10T00:48:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:50 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:48:51.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:50 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:48:51.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:50 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:48:51.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35441.486123 secs 2023-03-10T00:48:51.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:50 smithi038 ceph-mon[92422]: pgmap v19412: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 477 KiB/s wr, 10 op/s 2023-03-10T00:48:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:53 smithi122 ceph-mon[93869]: pgmap v19413: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 213 KiB/s wr, 11 op/s 2023-03-10T00:48:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:53 smithi181 ceph-mon[94189]: pgmap v19413: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 213 KiB/s wr, 11 op/s 2023-03-10T00:48:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:53 smithi038 ceph-mon[92422]: pgmap v19413: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 213 KiB/s wr, 11 op/s 2023-03-10T00:48:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35905.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:48:55.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:54 smithi122 ceph-mon[93869]: pgmap v19414: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 484 KiB/s rd, 482 KiB/s wr, 10 op/s 2023-03-10T00:48:55.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:54 smithi181 ceph-mon[94189]: pgmap v19414: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 484 KiB/s rd, 482 KiB/s wr, 10 op/s 2023-03-10T00:48:55.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:54 smithi038 ceph-mon[92422]: pgmap v19414: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 484 KiB/s rd, 482 KiB/s wr, 10 op/s 2023-03-10T00:48:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:48:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:48:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35446.486192 secs 2023-03-10T00:48:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:48:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:48:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35446.486192 secs 2023-03-10T00:48:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:48:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:48:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35446.486192 secs 2023-03-10T00:48:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:48:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:48:57.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:56 smithi122 ceph-mon[93869]: pgmap v19415: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 417 KiB/s wr, 12 op/s 2023-03-10T00:48:57.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:48:57.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:48:57.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:56 smithi181 ceph-mon[94189]: pgmap v19415: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 417 KiB/s wr, 12 op/s 2023-03-10T00:48:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:48:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:48:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:56 smithi038 ceph-mon[92422]: pgmap v19415: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 417 KiB/s wr, 12 op/s 2023-03-10T00:48:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:57 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 35448.950050 secs 2023-03-10T00:48:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:57 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 35448.950050 secs 2023-03-10T00:48:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:57 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 35448.950050 secs 2023-03-10T00:48:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35910.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:48:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:58 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:48:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:58 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:48:59.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:58 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:48:59.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:58 smithi122 ceph-mon[93869]: pgmap v19416: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 340 KiB/s wr, 10 op/s 2023-03-10T00:48:59.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:58 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:48:59.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:58 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:48:59.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:58 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:48:59.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:58 smithi181 ceph-mon[94189]: pgmap v19416: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 340 KiB/s wr, 10 op/s 2023-03-10T00:48:59.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:58 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:48:59.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:58 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:48:59.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:58 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:48:59.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:58 smithi038 ceph-mon[92422]: pgmap v19416: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 340 KiB/s wr, 10 op/s 2023-03-10T00:49:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:49:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:48:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:49:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:49:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:48:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:49:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:49:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:48:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:49:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:49:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:49:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35451.486275 secs 2023-03-10T00:49:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:00 smithi122 ceph-mon[93869]: pgmap v19417: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 686 KiB/s rd, 671 KiB/s wr, 11 op/s 2023-03-10T00:49:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:49:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:49:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35451.486275 secs 2023-03-10T00:49:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:00 smithi181 ceph-mon[94189]: pgmap v19417: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 686 KiB/s rd, 671 KiB/s wr, 11 op/s 2023-03-10T00:49:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:49:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:49:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35451.486275 secs 2023-03-10T00:49:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:00 smithi038 ceph-mon[92422]: pgmap v19417: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 686 KiB/s rd, 671 KiB/s wr, 11 op/s 2023-03-10T00:49:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:03 smithi122 ceph-mon[93869]: pgmap v19418: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 950 KiB/s rd, 945 KiB/s wr, 14 op/s 2023-03-10T00:49:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35453.950150 secs 2023-03-10T00:49:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:03 smithi181 ceph-mon[94189]: pgmap v19418: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 950 KiB/s rd, 945 KiB/s wr, 14 op/s 2023-03-10T00:49:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35453.950150 secs 2023-03-10T00:49:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:03 smithi038 ceph-mon[92422]: pgmap v19418: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 950 KiB/s rd, 945 KiB/s wr, 14 op/s 2023-03-10T00:49:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35453.950150 secs 2023-03-10T00:49:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35915.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:49:05.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:04 smithi122 ceph-mon[93869]: pgmap v19419: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 956 KiB/s rd, 956 KiB/s wr, 24 op/s 2023-03-10T00:49:05.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:04 smithi181 ceph-mon[94189]: pgmap v19419: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 956 KiB/s rd, 956 KiB/s wr, 24 op/s 2023-03-10T00:49:05.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:04 smithi038 ceph-mon[92422]: pgmap v19419: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 956 KiB/s rd, 956 KiB/s wr, 24 op/s 2023-03-10T00:49:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35456.486333 secs 2023-03-10T00:49:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35456.486333 secs 2023-03-10T00:49:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35456.486333 secs 2023-03-10T00:49:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:06 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:49:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:49:07.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:06 smithi122 ceph-mon[93869]: pgmap v19420: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 680 KiB/s rd, 679 KiB/s wr, 25 op/s 2023-03-10T00:49:07.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:06 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:49:07.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:49:07.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:06 smithi181 ceph-mon[94189]: pgmap v19420: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 680 KiB/s rd, 679 KiB/s wr, 25 op/s 2023-03-10T00:49:07.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:06 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:49:07.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:49:07.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:06 smithi038 ceph-mon[92422]: pgmap v19420: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 680 KiB/s rd, 679 KiB/s wr, 25 op/s 2023-03-10T00:49:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35458.950256 secs 2023-03-10T00:49:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35458.950256 secs 2023-03-10T00:49:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35458.950256 secs 2023-03-10T00:49:08.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35920.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:49:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:08 smithi122 ceph-mon[93869]: pgmap v19421: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 653 KiB/s wr, 23 op/s 2023-03-10T00:49:09.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:08 smithi181 ceph-mon[94189]: pgmap v19421: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 653 KiB/s wr, 23 op/s 2023-03-10T00:49:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:08 smithi038 ceph-mon[92422]: pgmap v19421: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 653 KiB/s wr, 23 op/s 2023-03-10T00:49:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35461.486417 secs 2023-03-10T00:49:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:11 smithi122 ceph-mon[93869]: pgmap v19422: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 652 KiB/s wr, 23 op/s 2023-03-10T00:49:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35461.486417 secs 2023-03-10T00:49:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:11 smithi181 ceph-mon[94189]: pgmap v19422: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 652 KiB/s wr, 23 op/s 2023-03-10T00:49:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35461.486417 secs 2023-03-10T00:49:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:11 smithi038 ceph-mon[92422]: pgmap v19422: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 652 KiB/s wr, 23 op/s 2023-03-10T00:49:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:13 smithi122 ceph-mon[93869]: pgmap v19423: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 473 KiB/s wr, 24 op/s 2023-03-10T00:49:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35463.950328 secs 2023-03-10T00:49:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:13 smithi181 ceph-mon[94189]: pgmap v19423: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 473 KiB/s wr, 24 op/s 2023-03-10T00:49:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35463.950328 secs 2023-03-10T00:49:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:13 smithi038 ceph-mon[92422]: pgmap v19423: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 473 KiB/s wr, 24 op/s 2023-03-10T00:49:13.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35463.950328 secs 2023-03-10T00:49:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35925.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:49:15.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:14 smithi122 ceph-mon[93869]: pgmap v19424: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 171 KiB/s wr, 19 op/s 2023-03-10T00:49:15.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:14 smithi181 ceph-mon[94189]: pgmap v19424: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 171 KiB/s wr, 19 op/s 2023-03-10T00:49:15.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:14 smithi038 ceph-mon[92422]: pgmap v19424: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 171 KiB/s wr, 19 op/s 2023-03-10T00:49:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:15 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:49:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:15 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:49:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35466.486479 secs 2023-03-10T00:49:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:15 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:49:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:15 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:49:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35466.486479 secs 2023-03-10T00:49:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:15 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:49:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:15 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:49:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35466.486479 secs 2023-03-10T00:49:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:49:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:49:17.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:16 smithi122 ceph-mon[93869]: pgmap v19425: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 160 KiB/s wr, 10 op/s 2023-03-10T00:49:17.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:49:17.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:49:17.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:16 smithi181 ceph-mon[94189]: pgmap v19425: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 160 KiB/s wr, 10 op/s 2023-03-10T00:49:17.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:49:17.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:49:17.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:16 smithi038 ceph-mon[92422]: pgmap v19425: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 160 KiB/s wr, 10 op/s 2023-03-10T00:49:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35468.950419 secs 2023-03-10T00:49:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35468.950419 secs 2023-03-10T00:49:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35468.950419 secs 2023-03-10T00:49:18.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35930.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:49:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:18 smithi122 ceph-mon[93869]: pgmap v19426: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 171 KiB/s wr, 10 op/s 2023-03-10T00:49:19.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:18 smithi181 ceph-mon[94189]: pgmap v19426: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 171 KiB/s wr, 10 op/s 2023-03-10T00:49:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:18 smithi038 ceph-mon[92422]: pgmap v19426: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 171 KiB/s wr, 10 op/s 2023-03-10T00:49:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35471.486564 secs 2023-03-10T00:49:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:21 smithi122 ceph-mon[93869]: pgmap v19427: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 197 KiB/s wr, 9 op/s 2023-03-10T00:49:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35471.486564 secs 2023-03-10T00:49:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:21 smithi181 ceph-mon[94189]: pgmap v19427: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 197 KiB/s wr, 9 op/s 2023-03-10T00:49:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35471.486564 secs 2023-03-10T00:49:21.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:21 smithi038 ceph-mon[92422]: pgmap v19427: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 197 KiB/s wr, 9 op/s 2023-03-10T00:49:23.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:23 smithi122 ceph-mon[93869]: pgmap v19428: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 498 KiB/s wr, 13 op/s 2023-03-10T00:49:23.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35473.950478 secs 2023-03-10T00:49:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:23 smithi181 ceph-mon[94189]: pgmap v19428: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 498 KiB/s wr, 13 op/s 2023-03-10T00:49:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35473.950478 secs 2023-03-10T00:49:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:23 smithi038 ceph-mon[92422]: pgmap v19428: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 498 KiB/s wr, 13 op/s 2023-03-10T00:49:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35473.950478 secs 2023-03-10T00:49:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35935.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:49:25.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:24 smithi122 ceph-mon[93869]: pgmap v19429: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 341 KiB/s wr, 10 op/s 2023-03-10T00:49:25.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:24 smithi181 ceph-mon[94189]: pgmap v19429: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 341 KiB/s wr, 10 op/s 2023-03-10T00:49:25.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:24 smithi038 ceph-mon[92422]: pgmap v19429: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 341 KiB/s wr, 10 op/s 2023-03-10T00:49:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:25 smithi122 ceph-mon[93869]: osdmap e1429: 12 total, 12 up, 12 in 2023-03-10T00:49:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35476.490993 secs 2023-03-10T00:49:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:25 smithi181 ceph-mon[94189]: osdmap e1429: 12 total, 12 up, 12 in 2023-03-10T00:49:26.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35476.490993 secs 2023-03-10T00:49:26.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:25 smithi038 ceph-mon[92422]: osdmap e1429: 12 total, 12 up, 12 in 2023-03-10T00:49:26.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35476.490993 secs 2023-03-10T00:49:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:26 smithi122 ceph-mon[93869]: pgmap v19431: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 550 KiB/s rd, 573 KiB/s wr, 13 op/s 2023-03-10T00:49:27.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:26 smithi181 ceph-mon[94189]: pgmap v19431: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 550 KiB/s rd, 573 KiB/s wr, 13 op/s 2023-03-10T00:49:27.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:26 smithi038 ceph-mon[92422]: pgmap v19431: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 550 KiB/s rd, 573 KiB/s wr, 13 op/s 2023-03-10T00:49:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35478.950539 secs 2023-03-10T00:49:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35478.950539 secs 2023-03-10T00:49:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35478.950539 secs 2023-03-10T00:49:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35940.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:49:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:28 smithi122 ceph-mon[93869]: pgmap v19432: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 549 KiB/s rd, 560 KiB/s wr, 11 op/s 2023-03-10T00:49:29.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:28 smithi181 ceph-mon[94189]: pgmap v19432: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 549 KiB/s rd, 560 KiB/s wr, 11 op/s 2023-03-10T00:49:29.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:28 smithi038 ceph-mon[92422]: pgmap v19432: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 549 KiB/s rd, 560 KiB/s wr, 11 op/s 2023-03-10T00:49:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35481.491051 secs 2023-03-10T00:49:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:31 smithi122 ceph-mon[93869]: pgmap v19433: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 535 KiB/s rd, 546 KiB/s wr, 11 op/s 2023-03-10T00:49:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35481.491051 secs 2023-03-10T00:49:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:31 smithi181 ceph-mon[94189]: pgmap v19433: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 535 KiB/s rd, 546 KiB/s wr, 11 op/s 2023-03-10T00:49:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35481.491051 secs 2023-03-10T00:49:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:31 smithi038 ceph-mon[92422]: pgmap v19433: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 535 KiB/s rd, 546 KiB/s wr, 11 op/s 2023-03-10T00:49:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:32 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:49:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:32 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:49:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:32 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:49:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:32 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:49:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:32 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:49:32.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:32 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:49:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:33 smithi122 ceph-mon[93869]: pgmap v19434: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 200 KiB/s wr, 9 op/s 2023-03-10T00:49:33.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:33 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:49:33.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:33 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:49:33.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35483.950600 secs 2023-03-10T00:49:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:33 smithi181 ceph-mon[94189]: pgmap v19434: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 200 KiB/s wr, 9 op/s 2023-03-10T00:49:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:33 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:49:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:33 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:49:33.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35483.950600 secs 2023-03-10T00:49:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:33 smithi038 ceph-mon[92422]: pgmap v19434: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 200 KiB/s wr, 9 op/s 2023-03-10T00:49:33.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:33 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:49:33.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:33 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:49:33.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35483.950600 secs 2023-03-10T00:49:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35945.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:49:35.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:34 smithi122 ceph-mon[93869]: osdmap e1430: 12 total, 12 up, 12 in 2023-03-10T00:49:35.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:34 smithi122 ceph-mon[93869]: pgmap v19436: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 214 KiB/s wr, 9 op/s 2023-03-10T00:49:35.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:34 smithi181 ceph-mon[94189]: osdmap e1430: 12 total, 12 up, 12 in 2023-03-10T00:49:35.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:34 smithi181 ceph-mon[94189]: pgmap v19436: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 214 KiB/s wr, 9 op/s 2023-03-10T00:49:35.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:34 smithi038 ceph-mon[92422]: osdmap e1430: 12 total, 12 up, 12 in 2023-03-10T00:49:35.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:34 smithi038 ceph-mon[92422]: pgmap v19436: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 214 KiB/s wr, 9 op/s 2023-03-10T00:49:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35486.491110 secs 2023-03-10T00:49:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35486.491110 secs 2023-03-10T00:49:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35486.491110 secs 2023-03-10T00:49:37.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:36 smithi122 ceph-mon[93869]: pgmap v19437: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 214 KiB/s wr, 8 op/s 2023-03-10T00:49:37.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:36 smithi181 ceph-mon[94189]: pgmap v19437: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 214 KiB/s wr, 8 op/s 2023-03-10T00:49:37.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:36 smithi038 ceph-mon[92422]: pgmap v19437: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 216 KiB/s rd, 214 KiB/s wr, 8 op/s 2023-03-10T00:49:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:49:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35488.950701 secs 2023-03-10T00:49:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:49:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:49:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:49:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:49:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35488.950701 secs 2023-03-10T00:49:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:49:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:49:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:49:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:49:38.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35488.950701 secs 2023-03-10T00:49:38.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:49:38.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:49:38.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:49:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35950.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:49:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:38 smithi122 ceph-mon[93869]: pgmap v19438: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 406 KiB/s wr, 10 op/s 2023-03-10T00:49:39.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:38 smithi181 ceph-mon[94189]: pgmap v19438: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 406 KiB/s wr, 10 op/s 2023-03-10T00:49:39.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:38 smithi038 ceph-mon[92422]: pgmap v19438: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 406 KiB/s wr, 10 op/s 2023-03-10T00:49:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:39 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:49:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:39 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:49:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:39 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:49:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:39 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:49:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:39 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:49:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:39 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:49:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:49:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:49:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35491.491168 secs 2023-03-10T00:49:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:40 smithi122 ceph-mon[93869]: pgmap v19439: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 385 KiB/s wr, 9 op/s 2023-03-10T00:49:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:49:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:49:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35491.491168 secs 2023-03-10T00:49:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:40 smithi181 ceph-mon[94189]: pgmap v19439: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 385 KiB/s wr, 9 op/s 2023-03-10T00:49:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:49:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:49:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35491.491168 secs 2023-03-10T00:49:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:40 smithi038 ceph-mon[92422]: pgmap v19439: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 385 KiB/s wr, 9 op/s 2023-03-10T00:49:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:43 smithi122 ceph-mon[93869]: pgmap v19440: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 714 KiB/s rd, 712 KiB/s wr, 11 op/s 2023-03-10T00:49:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35493.950784 secs 2023-03-10T00:49:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:43 smithi181 ceph-mon[94189]: pgmap v19440: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 714 KiB/s rd, 712 KiB/s wr, 11 op/s 2023-03-10T00:49:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35493.950784 secs 2023-03-10T00:49:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:43 smithi038 ceph-mon[92422]: pgmap v19440: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 714 KiB/s rd, 712 KiB/s wr, 11 op/s 2023-03-10T00:49:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35493.950784 secs 2023-03-10T00:49:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35955.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:49:45.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:44 smithi122 ceph-mon[93869]: pgmap v19441: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 691 KiB/s rd, 689 KiB/s wr, 11 op/s 2023-03-10T00:49:45.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:44 smithi181 ceph-mon[94189]: pgmap v19441: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 691 KiB/s rd, 689 KiB/s wr, 11 op/s 2023-03-10T00:49:45.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:44 smithi038 ceph-mon[92422]: pgmap v19441: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 691 KiB/s rd, 689 KiB/s wr, 11 op/s 2023-03-10T00:49:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35496.491227 secs 2023-03-10T00:49:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35496.491227 secs 2023-03-10T00:49:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35496.491227 secs 2023-03-10T00:49:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:46 smithi122 ceph-mon[93869]: pgmap v19442: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 871 KiB/s rd, 868 KiB/s wr, 12 op/s 2023-03-10T00:49:47.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:46 smithi181 ceph-mon[94189]: pgmap v19442: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 871 KiB/s rd, 868 KiB/s wr, 12 op/s 2023-03-10T00:49:47.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:46 smithi038 ceph-mon[92422]: pgmap v19442: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 871 KiB/s rd, 868 KiB/s wr, 12 op/s 2023-03-10T00:49:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:49:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:49:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35498.950874 secs 2023-03-10T00:49:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:49:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:49:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35498.950874 secs 2023-03-10T00:49:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:49:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:49:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35498.950874 secs 2023-03-10T00:49:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35960.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:49:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:49:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:49:49.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:48 smithi122 ceph-mon[93869]: pgmap v19443: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 720 KiB/s rd, 719 KiB/s wr, 10 op/s 2023-03-10T00:49:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:49:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:49:49.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:48 smithi181 ceph-mon[94189]: pgmap v19443: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 720 KiB/s rd, 719 KiB/s wr, 10 op/s 2023-03-10T00:49:49.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:49:49.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:49:49.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:48 smithi038 ceph-mon[92422]: pgmap v19443: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 720 KiB/s rd, 719 KiB/s wr, 10 op/s 2023-03-10T00:49:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35501.491311 secs 2023-03-10T00:49:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:51 smithi122 ceph-mon[93869]: pgmap v19444: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 561 KiB/s rd, 560 KiB/s wr, 8 op/s 2023-03-10T00:49:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35501.491311 secs 2023-03-10T00:49:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:51 smithi181 ceph-mon[94189]: pgmap v19444: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 561 KiB/s rd, 560 KiB/s wr, 8 op/s 2023-03-10T00:49:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35501.491311 secs 2023-03-10T00:49:51.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:51 smithi038 ceph-mon[92422]: pgmap v19444: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 561 KiB/s rd, 560 KiB/s wr, 8 op/s 2023-03-10T00:49:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:49:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:49:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:49:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:49:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:49:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:49:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:53 smithi122 ceph-mon[93869]: pgmap v19445: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.2 MiB/s wr, 14 op/s 2023-03-10T00:49:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:49:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:49:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35503.950933 secs 2023-03-10T00:49:53.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:53 smithi181 ceph-mon[94189]: pgmap v19445: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.2 MiB/s wr, 14 op/s 2023-03-10T00:49:53.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:49:53.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:49:53.734 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35503.950933 secs 2023-03-10T00:49:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:53 smithi038 ceph-mon[92422]: pgmap v19445: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.2 MiB/s wr, 14 op/s 2023-03-10T00:49:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:49:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:49:53.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35503.950933 secs 2023-03-10T00:49:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35965.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:49:55.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:54 smithi122 ceph-mon[93869]: pgmap v19446: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 928 KiB/s wr, 10 op/s 2023-03-10T00:49:55.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:54 smithi181 ceph-mon[94189]: pgmap v19446: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 928 KiB/s wr, 10 op/s 2023-03-10T00:49:55.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:54 smithi038 ceph-mon[92422]: pgmap v19446: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 928 KiB/s wr, 10 op/s 2023-03-10T00:49:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:49:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:49:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35506.491373 secs 2023-03-10T00:49:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:49:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:49:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35506.491373 secs 2023-03-10T00:49:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:49:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:49:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35506.491373 secs 2023-03-10T00:49:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:56 smithi122 ceph-mon[93869]: pgmap v19447: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 928 KiB/s wr, 13 op/s 2023-03-10T00:49:57.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:56 smithi181 ceph-mon[94189]: pgmap v19447: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 928 KiB/s wr, 13 op/s 2023-03-10T00:49:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:56 smithi038 ceph-mon[92422]: pgmap v19447: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 928 KiB/s wr, 13 op/s 2023-03-10T00:49:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35508.951038 secs 2023-03-10T00:49:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35508.951038 secs 2023-03-10T00:49:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35508.951038 secs 2023-03-10T00:49:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35970.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:49:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:58 smithi122 ceph-mon[93869]: pgmap v19448: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 742 KiB/s wr, 12 op/s 2023-03-10T00:49:59.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:58 smithi181 ceph-mon[94189]: pgmap v19448: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 742 KiB/s wr, 12 op/s 2023-03-10T00:49:59.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:58 smithi038 ceph-mon[92422]: pgmap v19448: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 742 KiB/s wr, 12 op/s 2023-03-10T00:50:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:50:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:49:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:50:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:50:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:49:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:50:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:50:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:49:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:50:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:50:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:50:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:00 smithi122 ceph-mon[93869]: overall HEALTH_WARN 2 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-10T00:50:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35511.491461 secs 2023-03-10T00:50:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:00 smithi122 ceph-mon[93869]: pgmap v19449: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 742 KiB/s wr, 11 op/s 2023-03-10T00:50:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:50:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:50:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:00 smithi181 ceph-mon[94189]: overall HEALTH_WARN 2 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-10T00:50:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35511.491461 secs 2023-03-10T00:50:01.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:00 smithi181 ceph-mon[94189]: pgmap v19449: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 742 KiB/s wr, 11 op/s 2023-03-10T00:50:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:50:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:50:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:00 smithi038 ceph-mon[92422]: overall HEALTH_WARN 2 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-10T00:50:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35511.491461 secs 2023-03-10T00:50:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:00 smithi038 ceph-mon[92422]: pgmap v19449: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 742 KiB/s wr, 11 op/s 2023-03-10T00:50:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:03 smithi122 ceph-mon[93869]: pgmap v19450: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.2 MiB/s wr, 17 op/s 2023-03-10T00:50:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:03 smithi181 ceph-mon[94189]: pgmap v19450: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.2 MiB/s wr, 17 op/s 2023-03-10T00:50:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:03 smithi038 ceph-mon[92422]: pgmap v19450: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.2 MiB/s wr, 17 op/s 2023-03-10T00:50:03.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35975.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:50:05.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:04 smithi122 ceph-mon[93869]: pgmap v19451: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 678 KiB/s rd, 557 KiB/s wr, 12 op/s 2023-03-10T00:50:05.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:04 smithi181 ceph-mon[94189]: pgmap v19451: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 678 KiB/s rd, 557 KiB/s wr, 12 op/s 2023-03-10T00:50:05.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:04 smithi038 ceph-mon[92422]: pgmap v19451: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 678 KiB/s rd, 557 KiB/s wr, 12 op/s 2023-03-10T00:50:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35516.491520 secs 2023-03-10T00:50:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:05 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:50:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35516.491520 secs 2023-03-10T00:50:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:05 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:50:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35516.491520 secs 2023-03-10T00:50:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:05 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:50:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:06 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:50:07.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:06 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:50:07.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:50:07.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:06 smithi122 ceph-mon[93869]: pgmap v19452: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 769 KiB/s rd, 669 KiB/s wr, 16 op/s 2023-03-10T00:50:07.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:06 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:50:07.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:06 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:50:07.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:50:07.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:06 smithi181 ceph-mon[94189]: pgmap v19452: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 769 KiB/s rd, 669 KiB/s wr, 16 op/s 2023-03-10T00:50:07.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:06 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:50:07.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:06 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:50:07.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:50:07.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:06 smithi038 ceph-mon[92422]: pgmap v19452: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 769 KiB/s rd, 669 KiB/s wr, 16 op/s 2023-03-10T00:50:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35980.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:50:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:50:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:50:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:08 smithi122 ceph-mon[93869]: pgmap v19453: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 802 KiB/s rd, 708 KiB/s wr, 25 op/s 2023-03-10T00:50:09.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:50:09.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:50:09.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:08 smithi181 ceph-mon[94189]: pgmap v19453: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 802 KiB/s rd, 708 KiB/s wr, 25 op/s 2023-03-10T00:50:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:50:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:50:09.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:08 smithi038 ceph-mon[92422]: pgmap v19453: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 802 KiB/s rd, 708 KiB/s wr, 25 op/s 2023-03-10T00:50:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35521.491580 secs 2023-03-10T00:50:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:11 smithi122 ceph-mon[93869]: pgmap v19454: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 810 KiB/s rd, 752 KiB/s wr, 23 op/s 2023-03-10T00:50:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35521.491580 secs 2023-03-10T00:50:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:11 smithi181 ceph-mon[94189]: pgmap v19454: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 810 KiB/s rd, 752 KiB/s wr, 23 op/s 2023-03-10T00:50:11.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35521.491580 secs 2023-03-10T00:50:11.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:11 smithi038 ceph-mon[92422]: pgmap v19454: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 810 KiB/s rd, 752 KiB/s wr, 23 op/s 2023-03-10T00:50:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:50:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:50:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:50:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:50:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:50:12.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:50:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:13 smithi122 ceph-mon[93869]: pgmap v19455: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 954 KiB/s rd, 895 KiB/s wr, 26 op/s 2023-03-10T00:50:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:13 smithi181 ceph-mon[94189]: pgmap v19455: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 954 KiB/s rd, 895 KiB/s wr, 26 op/s 2023-03-10T00:50:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:13 smithi038 ceph-mon[92422]: pgmap v19455: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 954 KiB/s rd, 895 KiB/s wr, 26 op/s 2023-03-10T00:50:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35985.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:50:15.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:14 smithi122 ceph-mon[93869]: pgmap v19456: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 427 KiB/s wr, 21 op/s 2023-03-10T00:50:15.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:14 smithi181 ceph-mon[94189]: pgmap v19456: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 427 KiB/s wr, 21 op/s 2023-03-10T00:50:15.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:14 smithi038 ceph-mon[92422]: pgmap v19456: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 427 KiB/s wr, 21 op/s 2023-03-10T00:50:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:15 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:50:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:15 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:50:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35526.491670 secs 2023-03-10T00:50:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:15 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:50:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:15 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:50:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35526.491670 secs 2023-03-10T00:50:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:15 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:50:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:15 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:50:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35526.491670 secs 2023-03-10T00:50:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:50:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:50:17.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:16 smithi122 ceph-mon[93869]: pgmap v19457: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 427 KiB/s wr, 22 op/s 2023-03-10T00:50:17.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:50:17.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:50:17.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:16 smithi181 ceph-mon[94189]: pgmap v19457: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 427 KiB/s wr, 22 op/s 2023-03-10T00:50:17.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:50:17.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:50:17.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:16 smithi038 ceph-mon[92422]: pgmap v19457: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 427 KiB/s wr, 22 op/s 2023-03-10T00:50:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35990.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:50:19.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:19 smithi122 ceph-mon[93869]: pgmap v19458: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 315 KiB/s wr, 20 op/s 2023-03-10T00:50:19.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:19 smithi181 ceph-mon[94189]: pgmap v19458: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 315 KiB/s wr, 20 op/s 2023-03-10T00:50:19.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:19 smithi038 ceph-mon[92422]: pgmap v19458: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 315 KiB/s wr, 20 op/s 2023-03-10T00:50:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:20 smithi122 ceph-mon[93869]: osdmap e1431: 12 total, 12 up, 12 in 2023-03-10T00:50:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35531.491758 secs 2023-03-10T00:50:21.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:20 smithi122 ceph-mon[93869]: pgmap v19460: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 425 KiB/s wr, 10 op/s 2023-03-10T00:50:21.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:50:21.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:50:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:20 smithi181 ceph-mon[94189]: osdmap e1431: 12 total, 12 up, 12 in 2023-03-10T00:50:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35531.491758 secs 2023-03-10T00:50:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:20 smithi181 ceph-mon[94189]: pgmap v19460: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 425 KiB/s wr, 10 op/s 2023-03-10T00:50:21.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:50:21.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:50:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:20 smithi038 ceph-mon[92422]: osdmap e1431: 12 total, 12 up, 12 in 2023-03-10T00:50:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35531.491758 secs 2023-03-10T00:50:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:20 smithi038 ceph-mon[92422]: pgmap v19460: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 425 KiB/s wr, 10 op/s 2023-03-10T00:50:21.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:50:21.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:50:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:23 smithi122 ceph-mon[93869]: pgmap v19461: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 640 KiB/s rd, 653 KiB/s wr, 12 op/s 2023-03-10T00:50:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35533.951381 secs 2023-03-10T00:50:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:23 smithi181 ceph-mon[94189]: pgmap v19461: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 640 KiB/s rd, 653 KiB/s wr, 12 op/s 2023-03-10T00:50:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35533.951381 secs 2023-03-10T00:50:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:23 smithi038 ceph-mon[92422]: pgmap v19461: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 640 KiB/s rd, 653 KiB/s wr, 12 op/s 2023-03-10T00:50:23.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35533.951381 secs 2023-03-10T00:50:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 35995.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:50:25.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:24 smithi122 ceph-mon[93869]: pgmap v19462: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 641 KiB/s rd, 666 KiB/s wr, 14 op/s 2023-03-10T00:50:25.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:24 smithi181 ceph-mon[94189]: pgmap v19462: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 641 KiB/s rd, 666 KiB/s wr, 14 op/s 2023-03-10T00:50:25.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:24 smithi038 ceph-mon[92422]: pgmap v19462: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 641 KiB/s rd, 666 KiB/s wr, 14 op/s 2023-03-10T00:50:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35536.491814 secs 2023-03-10T00:50:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35536.491814 secs 2023-03-10T00:50:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35536.491814 secs 2023-03-10T00:50:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:26 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:50:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:26 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:50:27.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:26 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:50:27.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:26 smithi122 ceph-mon[93869]: pgmap v19463: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 641 KiB/s rd, 666 KiB/s wr, 14 op/s 2023-03-10T00:50:27.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:26 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:50:27.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:26 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:50:27.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:26 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:50:27.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:26 smithi181 ceph-mon[94189]: pgmap v19463: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 641 KiB/s rd, 666 KiB/s wr, 14 op/s 2023-03-10T00:50:27.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:26 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:50:27.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:26 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:50:27.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:26 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:50:27.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:26 smithi038 ceph-mon[92422]: pgmap v19463: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 641 KiB/s rd, 666 KiB/s wr, 14 op/s 2023-03-10T00:50:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:27 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:50:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:27 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:50:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35538.951457 secs 2023-03-10T00:50:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:27 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:50:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:27 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:50:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35538.951457 secs 2023-03-10T00:50:28.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:27 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:50:28.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:27 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:50:28.253 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35538.951457 secs 2023-03-10T00:50:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36000.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:50:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:28 smithi122 ceph-mon[93869]: pgmap v19464: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 834 KiB/s rd, 859 KiB/s wr, 14 op/s 2023-03-10T00:50:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:28 smithi122 ceph-mon[93869]: osdmap e1432: 12 total, 12 up, 12 in 2023-03-10T00:50:29.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:28 smithi181 ceph-mon[94189]: pgmap v19464: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 834 KiB/s rd, 859 KiB/s wr, 14 op/s 2023-03-10T00:50:29.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:28 smithi181 ceph-mon[94189]: osdmap e1432: 12 total, 12 up, 12 in 2023-03-10T00:50:29.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:28 smithi038 ceph-mon[92422]: pgmap v19464: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 834 KiB/s rd, 859 KiB/s wr, 14 op/s 2023-03-10T00:50:29.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:28 smithi038 ceph-mon[92422]: osdmap e1432: 12 total, 12 up, 12 in 2023-03-10T00:50:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35541.491873 secs 2023-03-10T00:50:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:31 smithi122 ceph-mon[93869]: pgmap v19466: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 583 KiB/s rd, 606 KiB/s wr, 14 op/s 2023-03-10T00:50:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35541.491873 secs 2023-03-10T00:50:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:31 smithi181 ceph-mon[94189]: pgmap v19466: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 583 KiB/s rd, 606 KiB/s wr, 14 op/s 2023-03-10T00:50:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35541.491873 secs 2023-03-10T00:50:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:31 smithi038 ceph-mon[92422]: pgmap v19466: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 583 KiB/s rd, 606 KiB/s wr, 14 op/s 2023-03-10T00:50:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:32 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:50:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:32 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:50:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:32 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:50:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:32 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:50:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:32 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:50:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:32 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:50:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:33 smithi122 ceph-mon[93869]: pgmap v19467: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 378 KiB/s wr, 11 op/s 2023-03-10T00:50:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:33 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:50:33.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:33 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:50:33.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35543.951524 secs 2023-03-10T00:50:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:33 smithi181 ceph-mon[94189]: pgmap v19467: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 378 KiB/s wr, 11 op/s 2023-03-10T00:50:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:33 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:50:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:33 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:50:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35543.951524 secs 2023-03-10T00:50:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:33 smithi038 ceph-mon[92422]: pgmap v19467: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 378 KiB/s wr, 11 op/s 2023-03-10T00:50:33.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:33 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:50:33.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:33 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:50:33.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35543.951524 secs 2023-03-10T00:50:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36005.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:50:35.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:34 smithi122 ceph-mon[93869]: pgmap v19468: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 365 KiB/s wr, 9 op/s 2023-03-10T00:50:35.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:34 smithi181 ceph-mon[94189]: pgmap v19468: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 365 KiB/s wr, 9 op/s 2023-03-10T00:50:35.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:34 smithi038 ceph-mon[92422]: pgmap v19468: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 365 KiB/s wr, 9 op/s 2023-03-10T00:50:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35546.491935 secs 2023-03-10T00:50:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35546.491935 secs 2023-03-10T00:50:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35546.491935 secs 2023-03-10T00:50:37.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:36 smithi122 ceph-mon[93869]: pgmap v19469: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 365 KiB/s wr, 9 op/s 2023-03-10T00:50:37.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:36 smithi181 ceph-mon[94189]: pgmap v19469: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 365 KiB/s wr, 9 op/s 2023-03-10T00:50:37.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:36 smithi038 ceph-mon[92422]: pgmap v19469: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 365 KiB/s wr, 9 op/s 2023-03-10T00:50:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35548.951609 secs 2023-03-10T00:50:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:50:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35548.951609 secs 2023-03-10T00:50:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:50:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35548.951609 secs 2023-03-10T00:50:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:50:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36010.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:50:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:38 smithi122 ceph-mon[93869]: pgmap v19470: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 171 KiB/s wr, 8 op/s 2023-03-10T00:50:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:38 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:50:39.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:38 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:50:39.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:38 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:50:39.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:38 smithi181 ceph-mon[94189]: pgmap v19470: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 171 KiB/s wr, 8 op/s 2023-03-10T00:50:39.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:38 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:50:39.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:38 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:50:39.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:38 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:50:39.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:38 smithi038 ceph-mon[92422]: pgmap v19470: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 171 KiB/s wr, 8 op/s 2023-03-10T00:50:39.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:38 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:50:39.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:38 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:50:39.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:38 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:50:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35551.491979 secs 2023-03-10T00:50:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:41 smithi122 ceph-mon[93869]: pgmap v19471: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 151 KiB/s wr, 8 op/s 2023-03-10T00:50:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35551.491979 secs 2023-03-10T00:50:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:41 smithi181 ceph-mon[94189]: pgmap v19471: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 151 KiB/s wr, 8 op/s 2023-03-10T00:50:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35551.491979 secs 2023-03-10T00:50:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:41 smithi038 ceph-mon[92422]: pgmap v19471: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 151 KiB/s wr, 8 op/s 2023-03-10T00:50:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:43 smithi122 ceph-mon[93869]: pgmap v19472: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 143 KiB/s wr, 10 op/s 2023-03-10T00:50:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35553.951668 secs 2023-03-10T00:50:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:43 smithi181 ceph-mon[94189]: pgmap v19472: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 143 KiB/s wr, 10 op/s 2023-03-10T00:50:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35553.951668 secs 2023-03-10T00:50:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:43 smithi038 ceph-mon[92422]: pgmap v19472: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 143 KiB/s wr, 10 op/s 2023-03-10T00:50:43.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35553.951668 secs 2023-03-10T00:50:43.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36015.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:50:45.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:44 smithi122 ceph-mon[93869]: pgmap v19473: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 6 op/s 2023-03-10T00:50:45.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:44 smithi181 ceph-mon[94189]: pgmap v19473: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 6 op/s 2023-03-10T00:50:45.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:44 smithi038 ceph-mon[92422]: pgmap v19473: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 6 op/s 2023-03-10T00:50:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35556.492039 secs 2023-03-10T00:50:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35556.492039 secs 2023-03-10T00:50:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35556.492039 secs 2023-03-10T00:50:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:46 smithi122 ceph-mon[93869]: pgmap v19474: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 26 KiB/s wr, 9 op/s 2023-03-10T00:50:47.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:46 smithi181 ceph-mon[94189]: pgmap v19474: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 26 KiB/s wr, 9 op/s 2023-03-10T00:50:47.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:46 smithi038 ceph-mon[92422]: pgmap v19474: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 26 KiB/s wr, 9 op/s 2023-03-10T00:50:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36020.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:50:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:49 smithi122 ceph-mon[93869]: pgmap v19475: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 142 KiB/s wr, 8 op/s 2023-03-10T00:50:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:49 smithi181 ceph-mon[94189]: pgmap v19475: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 142 KiB/s wr, 8 op/s 2023-03-10T00:50:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:49 smithi038 ceph-mon[92422]: pgmap v19475: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 142 KiB/s wr, 8 op/s 2023-03-10T00:50:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:50 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:50:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:50 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:50:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:50 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:50:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35561.492126 secs 2023-03-10T00:50:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:51 smithi122 ceph-mon[93869]: pgmap v19476: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 537 KiB/s rd, 419 KiB/s wr, 7 op/s 2023-03-10T00:50:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:51 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:50:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:50:51.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:50:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35561.492126 secs 2023-03-10T00:50:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:51 smithi181 ceph-mon[94189]: pgmap v19476: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 537 KiB/s rd, 419 KiB/s wr, 7 op/s 2023-03-10T00:50:51.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:51 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:50:51.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:50:51.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:50:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35561.492126 secs 2023-03-10T00:50:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:51 smithi038 ceph-mon[92422]: pgmap v19476: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 537 KiB/s rd, 419 KiB/s wr, 7 op/s 2023-03-10T00:50:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:51 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:50:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:50:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:50:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:53 smithi122 ceph-mon[93869]: pgmap v19477: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 567 KiB/s rd, 448 KiB/s wr, 11 op/s 2023-03-10T00:50:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:50:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:50:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:53 smithi181 ceph-mon[94189]: pgmap v19477: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 567 KiB/s rd, 448 KiB/s wr, 11 op/s 2023-03-10T00:50:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:50:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:50:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:53 smithi038 ceph-mon[92422]: pgmap v19477: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 567 KiB/s rd, 448 KiB/s wr, 11 op/s 2023-03-10T00:50:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:50:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:50:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36025.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:50:55.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:54 smithi122 ceph-mon[93869]: pgmap v19478: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 566 KiB/s rd, 449 KiB/s wr, 8 op/s 2023-03-10T00:50:55.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:54 smithi181 ceph-mon[94189]: pgmap v19478: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 566 KiB/s rd, 449 KiB/s wr, 8 op/s 2023-03-10T00:50:55.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:54 smithi038 ceph-mon[92422]: pgmap v19478: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 566 KiB/s rd, 449 KiB/s wr, 8 op/s 2023-03-10T00:50:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:50:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:50:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35566.492186 secs 2023-03-10T00:50:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:50:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:50:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35566.492186 secs 2023-03-10T00:50:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:50:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:50:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35566.492186 secs 2023-03-10T00:50:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:56 smithi122 ceph-mon[93869]: pgmap v19479: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 607 KiB/s rd, 486 KiB/s wr, 11 op/s 2023-03-10T00:50:57.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:56 smithi181 ceph-mon[94189]: pgmap v19479: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 607 KiB/s rd, 486 KiB/s wr, 11 op/s 2023-03-10T00:50:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:56 smithi038 ceph-mon[92422]: pgmap v19479: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 607 KiB/s rd, 486 KiB/s wr, 11 op/s 2023-03-10T00:50:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:57 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 35568.951878 secs 2023-03-10T00:50:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:57 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 35568.951878 secs 2023-03-10T00:50:58.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:57 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 35568.951878 secs 2023-03-10T00:50:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36030.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:50:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:58 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:50:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:58 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:50:59.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:58 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:50:59.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:58 smithi122 ceph-mon[93869]: pgmap v19480: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 608 KiB/s rd, 489 KiB/s wr, 10 op/s 2023-03-10T00:50:59.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:58 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:50:59.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:58 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:50:59.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:58 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:50:59.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:58 smithi181 ceph-mon[94189]: pgmap v19480: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 608 KiB/s rd, 489 KiB/s wr, 10 op/s 2023-03-10T00:50:59.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:58 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:50:59.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:58 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:50:59.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:58 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:50:59.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:58 smithi038 ceph-mon[92422]: pgmap v19480: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 608 KiB/s rd, 489 KiB/s wr, 10 op/s 2023-03-10T00:51:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:51:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:50:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:51:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:51:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:50:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:51:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:51:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:50:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:51:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:51:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:51:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35571.492247 secs 2023-03-10T00:51:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:00 smithi122 ceph-mon[93869]: pgmap v19481: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 729 KiB/s rd, 705 KiB/s wr, 10 op/s 2023-03-10T00:51:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:51:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:51:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35571.492247 secs 2023-03-10T00:51:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:00 smithi181 ceph-mon[94189]: pgmap v19481: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 729 KiB/s rd, 705 KiB/s wr, 10 op/s 2023-03-10T00:51:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:51:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:51:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35571.492247 secs 2023-03-10T00:51:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:00 smithi038 ceph-mon[92422]: pgmap v19481: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 729 KiB/s rd, 705 KiB/s wr, 10 op/s 2023-03-10T00:51:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:03 smithi122 ceph-mon[93869]: pgmap v19482: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 662 KiB/s rd, 648 KiB/s wr, 15 op/s 2023-03-10T00:51:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:03 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 35573.951935 secs 2023-03-10T00:51:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:03 smithi181 ceph-mon[94189]: pgmap v19482: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 662 KiB/s rd, 648 KiB/s wr, 15 op/s 2023-03-10T00:51:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:03 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 35573.951935 secs 2023-03-10T00:51:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:03 smithi038 ceph-mon[92422]: pgmap v19482: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 662 KiB/s rd, 648 KiB/s wr, 15 op/s 2023-03-10T00:51:03.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:03 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 35573.951935 secs 2023-03-10T00:51:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36035.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:51:05.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:04 smithi122 ceph-mon[93869]: pgmap v19483: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 675 KiB/s rd, 639 KiB/s wr, 23 op/s 2023-03-10T00:51:05.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:04 smithi181 ceph-mon[94189]: pgmap v19483: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 675 KiB/s rd, 639 KiB/s wr, 23 op/s 2023-03-10T00:51:05.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:04 smithi038 ceph-mon[92422]: pgmap v19483: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 675 KiB/s rd, 639 KiB/s wr, 23 op/s 2023-03-10T00:51:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35576.492334 secs 2023-03-10T00:51:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35576.492334 secs 2023-03-10T00:51:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35576.492334 secs 2023-03-10T00:51:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:06 smithi122 ceph-mon[93869]: pgmap v19484: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 793 KiB/s rd, 756 KiB/s wr, 25 op/s 2023-03-10T00:51:07.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:06 smithi181 ceph-mon[94189]: pgmap v19484: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 793 KiB/s rd, 756 KiB/s wr, 25 op/s 2023-03-10T00:51:07.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:06 smithi038 ceph-mon[92422]: pgmap v19484: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 793 KiB/s rd, 756 KiB/s wr, 25 op/s 2023-03-10T00:51:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:51:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:51:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:07 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 35578.952025 secs 2023-03-10T00:51:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:51:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:51:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:07 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 35578.952025 secs 2023-03-10T00:51:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:51:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:51:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:07 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 35578.952025 secs 2023-03-10T00:51:08.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36040.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:51:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:51:09.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:51:09.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:08 smithi038 ceph-mon[92422]: pgmap v19485: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 913 KiB/s rd, 880 KiB/s wr, 24 op/s 2023-03-10T00:51:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:51:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:51:09.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:08 smithi122 ceph-mon[93869]: pgmap v19485: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 913 KiB/s rd, 880 KiB/s wr, 24 op/s 2023-03-10T00:51:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:51:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:51:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:08 smithi181 ceph-mon[94189]: pgmap v19485: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 913 KiB/s rd, 880 KiB/s wr, 24 op/s 2023-03-10T00:51:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35581.492417 secs 2023-03-10T00:51:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:11 smithi122 ceph-mon[93869]: pgmap v19486: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 885 KiB/s rd, 851 KiB/s wr, 23 op/s 2023-03-10T00:51:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35581.492417 secs 2023-03-10T00:51:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:11 smithi181 ceph-mon[94189]: pgmap v19486: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 885 KiB/s rd, 851 KiB/s wr, 23 op/s 2023-03-10T00:51:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35581.492417 secs 2023-03-10T00:51:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:11 smithi038 ceph-mon[92422]: pgmap v19486: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 885 KiB/s rd, 851 KiB/s wr, 23 op/s 2023-03-10T00:51:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:13 smithi122 ceph-mon[93869]: pgmap v19487: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 695 KiB/s rd, 680 KiB/s wr, 24 op/s 2023-03-10T00:51:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:13 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 35583.952119 secs 2023-03-10T00:51:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:13 smithi181 ceph-mon[94189]: pgmap v19487: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 695 KiB/s rd, 680 KiB/s wr, 24 op/s 2023-03-10T00:51:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:13 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 35583.952119 secs 2023-03-10T00:51:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:13 smithi038 ceph-mon[92422]: pgmap v19487: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 695 KiB/s rd, 680 KiB/s wr, 24 op/s 2023-03-10T00:51:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:13 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 35583.952119 secs 2023-03-10T00:51:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36045.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:51:15.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:14 smithi122 ceph-mon[93869]: pgmap v19488: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 484 KiB/s rd, 459 KiB/s wr, 19 op/s 2023-03-10T00:51:15.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:14 smithi181 ceph-mon[94189]: pgmap v19488: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 484 KiB/s rd, 459 KiB/s wr, 19 op/s 2023-03-10T00:51:15.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:14 smithi038 ceph-mon[92422]: pgmap v19488: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 484 KiB/s rd, 459 KiB/s wr, 19 op/s 2023-03-10T00:51:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:15 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:51:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:15 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:51:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35586.492477 secs 2023-03-10T00:51:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:15 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:51:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:15 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:51:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35586.492477 secs 2023-03-10T00:51:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:15 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:51:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:15 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:51:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35586.492477 secs 2023-03-10T00:51:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:51:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:51:17.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:16 smithi122 ceph-mon[93869]: pgmap v19489: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 442 KiB/s rd, 439 KiB/s wr, 10 op/s 2023-03-10T00:51:17.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:51:17.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:51:17.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:16 smithi181 ceph-mon[94189]: pgmap v19489: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 442 KiB/s rd, 439 KiB/s wr, 10 op/s 2023-03-10T00:51:17.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:51:17.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:51:17.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:16 smithi038 ceph-mon[92422]: pgmap v19489: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 442 KiB/s rd, 439 KiB/s wr, 10 op/s 2023-03-10T00:51:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:17 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 35588.952177 secs 2023-03-10T00:51:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:17 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 35588.952177 secs 2023-03-10T00:51:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:17 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 35588.952177 secs 2023-03-10T00:51:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36050.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:51:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:18 smithi122 ceph-mon[93869]: pgmap v19490: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 333 KiB/s wr, 10 op/s 2023-03-10T00:51:19.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:18 smithi181 ceph-mon[94189]: pgmap v19490: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 333 KiB/s wr, 10 op/s 2023-03-10T00:51:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:18 smithi038 ceph-mon[92422]: pgmap v19490: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 333 KiB/s wr, 10 op/s 2023-03-10T00:51:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:51:21.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:51:21.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35591.492563 secs 2023-03-10T00:51:21.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:20 smithi038 ceph-mon[92422]: pgmap v19491: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 172 KiB/s wr, 9 op/s 2023-03-10T00:51:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:51:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:51:21.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35591.492563 secs 2023-03-10T00:51:21.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:20 smithi122 ceph-mon[93869]: pgmap v19491: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 172 KiB/s wr, 9 op/s 2023-03-10T00:51:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:51:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:51:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35591.492563 secs 2023-03-10T00:51:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:20 smithi181 ceph-mon[94189]: pgmap v19491: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 172 KiB/s wr, 9 op/s 2023-03-10T00:51:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:23 smithi122 ceph-mon[93869]: pgmap v19492: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 528 KiB/s wr, 14 op/s 2023-03-10T00:51:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:23 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 35593.952281 secs 2023-03-10T00:51:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:23 smithi181 ceph-mon[94189]: pgmap v19492: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 528 KiB/s wr, 14 op/s 2023-03-10T00:51:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:23 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 35593.952281 secs 2023-03-10T00:51:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:23 smithi038 ceph-mon[92422]: pgmap v19492: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 528 KiB/s wr, 14 op/s 2023-03-10T00:51:23.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:23 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 35593.952281 secs 2023-03-10T00:51:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36055.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:51:25.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:24 smithi122 ceph-mon[93869]: pgmap v19493: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 348 KiB/s rd, 368 KiB/s wr, 11 op/s 2023-03-10T00:51:25.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:24 smithi181 ceph-mon[94189]: pgmap v19493: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 348 KiB/s rd, 368 KiB/s wr, 11 op/s 2023-03-10T00:51:25.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:24 smithi038 ceph-mon[92422]: pgmap v19493: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 348 KiB/s rd, 368 KiB/s wr, 11 op/s 2023-03-10T00:51:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:25 smithi122 ceph-mon[93869]: osdmap e1433: 12 total, 12 up, 12 in 2023-03-10T00:51:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35596.492635 secs 2023-03-10T00:51:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:25 smithi181 ceph-mon[94189]: osdmap e1433: 12 total, 12 up, 12 in 2023-03-10T00:51:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35596.492635 secs 2023-03-10T00:51:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:25 smithi038 ceph-mon[92422]: osdmap e1433: 12 total, 12 up, 12 in 2023-03-10T00:51:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35596.492635 secs 2023-03-10T00:51:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:26 smithi122 ceph-mon[93869]: pgmap v19495: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 854 KiB/s rd, 770 KiB/s wr, 13 op/s 2023-03-10T00:51:27.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:26 smithi181 ceph-mon[94189]: pgmap v19495: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 854 KiB/s rd, 770 KiB/s wr, 13 op/s 2023-03-10T00:51:27.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:26 smithi038 ceph-mon[92422]: pgmap v19495: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 854 KiB/s rd, 770 KiB/s wr, 13 op/s 2023-03-10T00:51:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:27 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:51:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:27 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:51:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:27 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 35598.952339 secs 2023-03-10T00:51:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:27 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:51:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:27 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:51:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:27 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 35598.952339 secs 2023-03-10T00:51:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:27 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:51:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:27 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:51:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:27 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 35598.952339 secs 2023-03-10T00:51:28.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36060.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:51:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:28 smithi122 ceph-mon[93869]: pgmap v19496: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 854 KiB/s rd, 757 KiB/s wr, 12 op/s 2023-03-10T00:51:29.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:28 smithi181 ceph-mon[94189]: pgmap v19496: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 854 KiB/s rd, 757 KiB/s wr, 12 op/s 2023-03-10T00:51:29.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:28 smithi038 ceph-mon[92422]: pgmap v19496: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 854 KiB/s rd, 757 KiB/s wr, 12 op/s 2023-03-10T00:51:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35601.492689 secs 2023-03-10T00:51:31.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:31 smithi122 ceph-mon[93869]: pgmap v19497: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 854 KiB/s rd, 758 KiB/s wr, 12 op/s 2023-03-10T00:51:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35601.492689 secs 2023-03-10T00:51:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:31 smithi181 ceph-mon[94189]: pgmap v19497: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 854 KiB/s rd, 758 KiB/s wr, 12 op/s 2023-03-10T00:51:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35601.492689 secs 2023-03-10T00:51:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:31 smithi038 ceph-mon[92422]: pgmap v19497: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 854 KiB/s rd, 758 KiB/s wr, 12 op/s 2023-03-10T00:51:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:32 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:51:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:32 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:51:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:32 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:51:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:32 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:51:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:32 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:51:32.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:32 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:51:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:33 smithi122 ceph-mon[93869]: pgmap v19498: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 365 KiB/s wr, 10 op/s 2023-03-10T00:51:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:33 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:51:33.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:33 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:51:33.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:33 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 35603.952399 secs 2023-03-10T00:51:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:33 smithi181 ceph-mon[94189]: pgmap v19498: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 365 KiB/s wr, 10 op/s 2023-03-10T00:51:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:33 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:51:33.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:33 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:51:33.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:33 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 35603.952399 secs 2023-03-10T00:51:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:33 smithi038 ceph-mon[92422]: pgmap v19498: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 365 KiB/s wr, 10 op/s 2023-03-10T00:51:33.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:33 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:51:33.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:33 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:51:33.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:33 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 35603.952399 secs 2023-03-10T00:51:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36065.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:51:35.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:34 smithi122 ceph-mon[93869]: osdmap e1434: 12 total, 12 up, 12 in 2023-03-10T00:51:35.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:34 smithi122 ceph-mon[93869]: pgmap v19500: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 392 KiB/s wr, 11 op/s 2023-03-10T00:51:35.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:34 smithi181 ceph-mon[94189]: osdmap e1434: 12 total, 12 up, 12 in 2023-03-10T00:51:35.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:34 smithi181 ceph-mon[94189]: pgmap v19500: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 392 KiB/s wr, 11 op/s 2023-03-10T00:51:35.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:34 smithi038 ceph-mon[92422]: osdmap e1434: 12 total, 12 up, 12 in 2023-03-10T00:51:35.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:34 smithi038 ceph-mon[92422]: pgmap v19500: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 392 KiB/s wr, 11 op/s 2023-03-10T00:51:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35606.492779 secs 2023-03-10T00:51:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35606.492779 secs 2023-03-10T00:51:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35606.492779 secs 2023-03-10T00:51:37.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:51:37.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:51:37.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:36 smithi122 ceph-mon[93869]: pgmap v19501: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 746 KiB/s rd, 911 KiB/s wr, 12 op/s 2023-03-10T00:51:37.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:51:37.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:51:37.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:36 smithi181 ceph-mon[94189]: pgmap v19501: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 746 KiB/s rd, 911 KiB/s wr, 12 op/s 2023-03-10T00:51:37.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:51:37.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:51:37.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:36 smithi038 ceph-mon[92422]: pgmap v19501: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 746 KiB/s rd, 911 KiB/s wr, 12 op/s 2023-03-10T00:51:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35608.952483 secs 2023-03-10T00:51:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35608.952483 secs 2023-03-10T00:51:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35608.952483 secs 2023-03-10T00:51:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36070.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:51:39.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:38 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:51:39.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:38 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:51:39.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:38 smithi038 ceph-mon[92422]: pgmap v19502: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 746 KiB/s rd, 910 KiB/s wr, 11 op/s 2023-03-10T00:51:39.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:38 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:51:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:38 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:51:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:38 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:51:39.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:38 smithi122 ceph-mon[93869]: pgmap v19502: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 746 KiB/s rd, 910 KiB/s wr, 11 op/s 2023-03-10T00:51:39.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:38 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:51:39.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:38 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:51:39.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:38 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:51:39.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:38 smithi181 ceph-mon[94189]: pgmap v19502: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 746 KiB/s rd, 910 KiB/s wr, 11 op/s 2023-03-10T00:51:39.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:38 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:51:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:39 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:51:40.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:39 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:51:40.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:51:40.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:51:40.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:51:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:39 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:51:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:39 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:51:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:51:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:51:40.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:51:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:39 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:51:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:39 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:51:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:51:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:51:40.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:51:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35611.492838 secs 2023-03-10T00:51:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:41 smithi122 ceph-mon[93869]: pgmap v19503: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 746 KiB/s rd, 909 KiB/s wr, 12 op/s 2023-03-10T00:51:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:51:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:51:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35611.492838 secs 2023-03-10T00:51:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:41 smithi181 ceph-mon[94189]: pgmap v19503: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 746 KiB/s rd, 909 KiB/s wr, 12 op/s 2023-03-10T00:51:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:51:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:51:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35611.492838 secs 2023-03-10T00:51:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:41 smithi038 ceph-mon[92422]: pgmap v19503: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 746 KiB/s rd, 909 KiB/s wr, 12 op/s 2023-03-10T00:51:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:51:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:51:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:43 smithi122 ceph-mon[93869]: pgmap v19504: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 11 op/s 2023-03-10T00:51:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35613.952543 secs 2023-03-10T00:51:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:43 smithi181 ceph-mon[94189]: pgmap v19504: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 11 op/s 2023-03-10T00:51:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35613.952543 secs 2023-03-10T00:51:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:43 smithi038 ceph-mon[92422]: pgmap v19504: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 11 op/s 2023-03-10T00:51:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35613.952543 secs 2023-03-10T00:51:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36075.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:51:45.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:44 smithi122 ceph-mon[93869]: pgmap v19505: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1008 KiB/s rd, 1006 KiB/s wr, 10 op/s 2023-03-10T00:51:45.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:44 smithi181 ceph-mon[94189]: pgmap v19505: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1008 KiB/s rd, 1006 KiB/s wr, 10 op/s 2023-03-10T00:51:45.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:44 smithi038 ceph-mon[92422]: pgmap v19505: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1008 KiB/s rd, 1006 KiB/s wr, 10 op/s 2023-03-10T00:51:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35616.492897 secs 2023-03-10T00:51:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35616.492897 secs 2023-03-10T00:51:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35616.492897 secs 2023-03-10T00:51:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:46 smithi122 ceph-mon[93869]: pgmap v19506: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 13 op/s 2023-03-10T00:51:47.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:46 smithi181 ceph-mon[94189]: pgmap v19506: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 13 op/s 2023-03-10T00:51:47.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:46 smithi038 ceph-mon[92422]: pgmap v19506: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 13 op/s 2023-03-10T00:51:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:51:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:51:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:47 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35618.952639 secs 2023-03-10T00:51:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:51:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:51:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:47 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35618.952639 secs 2023-03-10T00:51:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:51:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:51:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:47 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35618.952639 secs 2023-03-10T00:51:48.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36080.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:51:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:51:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:51:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:48 smithi038 ceph-mon[92422]: pgmap v19507: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 456 KiB/s rd, 316 KiB/s wr, 9 op/s 2023-03-10T00:51:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:51:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:51:49.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:48 smithi122 ceph-mon[93869]: pgmap v19507: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 456 KiB/s rd, 316 KiB/s wr, 9 op/s 2023-03-10T00:51:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:51:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:51:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:48 smithi181 ceph-mon[94189]: pgmap v19507: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 456 KiB/s rd, 316 KiB/s wr, 9 op/s 2023-03-10T00:51:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35621.492962 secs 2023-03-10T00:51:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:51 smithi122 ceph-mon[93869]: pgmap v19508: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 456 KiB/s rd, 316 KiB/s wr, 9 op/s 2023-03-10T00:51:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35621.492962 secs 2023-03-10T00:51:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:51 smithi181 ceph-mon[94189]: pgmap v19508: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 456 KiB/s rd, 316 KiB/s wr, 9 op/s 2023-03-10T00:51:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35621.492962 secs 2023-03-10T00:51:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:51 smithi038 ceph-mon[92422]: pgmap v19508: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 456 KiB/s rd, 316 KiB/s wr, 9 op/s 2023-03-10T00:51:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:53 smithi122 ceph-mon[93869]: pgmap v19509: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 653 KiB/s rd, 512 KiB/s wr, 12 op/s 2023-03-10T00:51:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35623.952698 secs 2023-03-10T00:51:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:53 smithi181 ceph-mon[94189]: pgmap v19509: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 653 KiB/s rd, 512 KiB/s wr, 12 op/s 2023-03-10T00:51:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35623.952698 secs 2023-03-10T00:51:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:53 smithi038 ceph-mon[92422]: pgmap v19509: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 653 KiB/s rd, 512 KiB/s wr, 12 op/s 2023-03-10T00:51:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35623.952698 secs 2023-03-10T00:51:53.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36085.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:51:55.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:54 smithi122 ceph-mon[93869]: pgmap v19510: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 489 KiB/s wr, 10 op/s 2023-03-10T00:51:55.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:54 smithi181 ceph-mon[94189]: pgmap v19510: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 489 KiB/s wr, 10 op/s 2023-03-10T00:51:55.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:54 smithi038 ceph-mon[92422]: pgmap v19510: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 489 KiB/s wr, 10 op/s 2023-03-10T00:51:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:51:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:51:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35626.493021 secs 2023-03-10T00:51:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:51:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:51:56.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35626.493021 secs 2023-03-10T00:51:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:51:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:51:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35626.493021 secs 2023-03-10T00:51:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:51:57.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:51:57.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:56 smithi122 ceph-mon[93869]: pgmap v19511: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 523 KiB/s rd, 518 KiB/s wr, 13 op/s 2023-03-10T00:51:57.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:51:57.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:51:57.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:56 smithi181 ceph-mon[94189]: pgmap v19511: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 523 KiB/s rd, 518 KiB/s wr, 13 op/s 2023-03-10T00:51:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:51:57.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:51:57.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:56 smithi038 ceph-mon[92422]: pgmap v19511: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 523 KiB/s rd, 518 KiB/s wr, 13 op/s 2023-03-10T00:51:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:57 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35628.952824 secs 2023-03-10T00:51:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:57 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35628.952824 secs 2023-03-10T00:51:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:57 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35628.952824 secs 2023-03-10T00:51:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36090.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:51:59.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:58 smithi038 ceph-mon[92422]: pgmap v19512: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 340 KiB/s wr, 9 op/s 2023-03-10T00:51:59.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:58 smithi122 ceph-mon[93869]: pgmap v19512: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 340 KiB/s wr, 9 op/s 2023-03-10T00:51:59.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:58 smithi181 ceph-mon[94189]: pgmap v19512: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 343 KiB/s rd, 340 KiB/s wr, 9 op/s 2023-03-10T00:52:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:52:00.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:51:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:52:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:52:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:51:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:52:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:52:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:51:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:52:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:52:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:52:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35631.493080 secs 2023-03-10T00:52:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:01 smithi122 ceph-mon[93869]: pgmap v19513: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 581 KiB/s rd, 567 KiB/s wr, 10 op/s 2023-03-10T00:52:01.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:52:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:52:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35631.493080 secs 2023-03-10T00:52:01.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:01 smithi181 ceph-mon[94189]: pgmap v19513: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 581 KiB/s rd, 567 KiB/s wr, 10 op/s 2023-03-10T00:52:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:52:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:52:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35631.493080 secs 2023-03-10T00:52:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:01 smithi038 ceph-mon[92422]: pgmap v19513: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 581 KiB/s rd, 567 KiB/s wr, 10 op/s 2023-03-10T00:52:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:03 smithi122 ceph-mon[93869]: pgmap v19514: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 987 KiB/s wr, 16 op/s 2023-03-10T00:52:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35633.952902 secs 2023-03-10T00:52:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:03 smithi181 ceph-mon[94189]: pgmap v19514: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 987 KiB/s wr, 16 op/s 2023-03-10T00:52:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35633.952902 secs 2023-03-10T00:52:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:03 smithi038 ceph-mon[92422]: pgmap v19514: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 987 KiB/s wr, 16 op/s 2023-03-10T00:52:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35633.952902 secs 2023-03-10T00:52:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36095.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:52:05.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:04 smithi122 ceph-mon[93869]: pgmap v19515: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 836 KiB/s rd, 802 KiB/s wr, 23 op/s 2023-03-10T00:52:05.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:04 smithi181 ceph-mon[94189]: pgmap v19515: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 836 KiB/s rd, 802 KiB/s wr, 23 op/s 2023-03-10T00:52:05.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:04 smithi038 ceph-mon[92422]: pgmap v19515: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 836 KiB/s rd, 802 KiB/s wr, 23 op/s 2023-03-10T00:52:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35636.493140 secs 2023-03-10T00:52:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35636.493140 secs 2023-03-10T00:52:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35636.493140 secs 2023-03-10T00:52:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:06 smithi122 ceph-mon[93869]: pgmap v19516: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 863 KiB/s rd, 828 KiB/s wr, 26 op/s 2023-03-10T00:52:07.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:06 smithi181 ceph-mon[94189]: pgmap v19516: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 863 KiB/s rd, 828 KiB/s wr, 26 op/s 2023-03-10T00:52:07.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:06 smithi038 ceph-mon[92422]: pgmap v19516: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 863 KiB/s rd, 828 KiB/s wr, 26 op/s 2023-03-10T00:52:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:52:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:52:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35638.953031 secs 2023-03-10T00:52:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:52:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:52:08.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35638.953031 secs 2023-03-10T00:52:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:52:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:52:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35638.953031 secs 2023-03-10T00:52:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36100.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:52:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:52:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:52:09.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:08 smithi038 ceph-mon[92422]: pgmap v19517: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 831 KiB/s rd, 799 KiB/s wr, 24 op/s 2023-03-10T00:52:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:52:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:52:09.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:08 smithi122 ceph-mon[93869]: pgmap v19517: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 831 KiB/s rd, 799 KiB/s wr, 24 op/s 2023-03-10T00:52:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:52:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:52:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:08 smithi181 ceph-mon[94189]: pgmap v19517: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 831 KiB/s rd, 799 KiB/s wr, 24 op/s 2023-03-10T00:52:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35641.493196 secs 2023-03-10T00:52:11.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:11 smithi122 ceph-mon[93869]: pgmap v19518: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 831 KiB/s rd, 799 KiB/s wr, 23 op/s 2023-03-10T00:52:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35641.493196 secs 2023-03-10T00:52:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:11 smithi181 ceph-mon[94189]: pgmap v19518: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 831 KiB/s rd, 799 KiB/s wr, 23 op/s 2023-03-10T00:52:11.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35641.493196 secs 2023-03-10T00:52:11.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:11 smithi038 ceph-mon[92422]: pgmap v19518: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 831 KiB/s rd, 799 KiB/s wr, 23 op/s 2023-03-10T00:52:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:13 smithi122 ceph-mon[93869]: pgmap v19519: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 752 KiB/s rd, 730 KiB/s wr, 25 op/s 2023-03-10T00:52:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35643.953137 secs 2023-03-10T00:52:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:13 smithi181 ceph-mon[94189]: pgmap v19519: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 752 KiB/s rd, 730 KiB/s wr, 25 op/s 2023-03-10T00:52:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35643.953137 secs 2023-03-10T00:52:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:13 smithi038 ceph-mon[92422]: pgmap v19519: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 752 KiB/s rd, 730 KiB/s wr, 25 op/s 2023-03-10T00:52:13.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35643.953137 secs 2023-03-10T00:52:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36105.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:52:15.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:14 smithi122 ceph-mon[93869]: pgmap v19520: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 310 KiB/s wr, 18 op/s 2023-03-10T00:52:15.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:14 smithi181 ceph-mon[94189]: pgmap v19520: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 310 KiB/s wr, 18 op/s 2023-03-10T00:52:15.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:14 smithi038 ceph-mon[92422]: pgmap v19520: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 310 KiB/s wr, 18 op/s 2023-03-10T00:52:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35646.493254 secs 2023-03-10T00:52:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35646.493254 secs 2023-03-10T00:52:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35646.493254 secs 2023-03-10T00:52:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:16 smithi122 ceph-mon[93869]: pgmap v19521: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 299 KiB/s wr, 9 op/s 2023-03-10T00:52:17.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:16 smithi181 ceph-mon[94189]: pgmap v19521: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 299 KiB/s wr, 9 op/s 2023-03-10T00:52:17.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:16 smithi038 ceph-mon[92422]: pgmap v19521: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 299 KiB/s wr, 9 op/s 2023-03-10T00:52:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:17 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35648.953198 secs 2023-03-10T00:52:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:17 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35648.953198 secs 2023-03-10T00:52:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:17 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35648.953198 secs 2023-03-10T00:52:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36110.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:52:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:18 smithi038 ceph-mon[92422]: pgmap v19522: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 170 KiB/s wr, 7 op/s 2023-03-10T00:52:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:18 smithi122 ceph-mon[93869]: pgmap v19522: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 170 KiB/s wr, 7 op/s 2023-03-10T00:52:19.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:18 smithi181 ceph-mon[94189]: pgmap v19522: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 170 KiB/s wr, 7 op/s 2023-03-10T00:52:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:19 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:52:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:19 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:52:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:19 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:52:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:19 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:52:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:19 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:52:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:19 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:52:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:52:21.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:52:21.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35651.493339 secs 2023-03-10T00:52:21.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:21 smithi122 ceph-mon[93869]: pgmap v19523: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 170 KiB/s wr, 6 op/s 2023-03-10T00:52:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:52:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:52:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35651.493339 secs 2023-03-10T00:52:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:21 smithi181 ceph-mon[94189]: pgmap v19523: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 170 KiB/s wr, 6 op/s 2023-03-10T00:52:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:52:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:52:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35651.493339 secs 2023-03-10T00:52:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:21 smithi038 ceph-mon[92422]: pgmap v19523: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 170 KiB/s wr, 6 op/s 2023-03-10T00:52:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:23 smithi122 ceph-mon[93869]: pgmap v19524: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 485 KiB/s rd, 504 KiB/s wr, 12 op/s 2023-03-10T00:52:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35653.953261 secs 2023-03-10T00:52:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:23 smithi181 ceph-mon[94189]: pgmap v19524: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 485 KiB/s rd, 504 KiB/s wr, 12 op/s 2023-03-10T00:52:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35653.953261 secs 2023-03-10T00:52:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:23 smithi038 ceph-mon[92422]: pgmap v19524: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 485 KiB/s rd, 504 KiB/s wr, 12 op/s 2023-03-10T00:52:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35653.953261 secs 2023-03-10T00:52:23.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36115.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:52:25.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:24 smithi122 ceph-mon[93869]: pgmap v19525: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 326 KiB/s rd, 345 KiB/s wr, 9 op/s 2023-03-10T00:52:25.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:24 smithi181 ceph-mon[94189]: pgmap v19525: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 326 KiB/s rd, 345 KiB/s wr, 9 op/s 2023-03-10T00:52:25.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:24 smithi038 ceph-mon[92422]: pgmap v19525: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 326 KiB/s rd, 345 KiB/s wr, 9 op/s 2023-03-10T00:52:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:25 smithi122 ceph-mon[93869]: osdmap e1435: 12 total, 12 up, 12 in 2023-03-10T00:52:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35656.493985 secs 2023-03-10T00:52:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:25 smithi181 ceph-mon[94189]: osdmap e1435: 12 total, 12 up, 12 in 2023-03-10T00:52:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35656.493985 secs 2023-03-10T00:52:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:25 smithi038 ceph-mon[92422]: osdmap e1435: 12 total, 12 up, 12 in 2023-03-10T00:52:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35656.493985 secs 2023-03-10T00:52:27.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:26 smithi181 ceph-mon[94189]: pgmap v19527: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 455 KiB/s rd, 479 KiB/s wr, 12 op/s 2023-03-10T00:52:27.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:26 smithi038 ceph-mon[92422]: pgmap v19527: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 455 KiB/s rd, 479 KiB/s wr, 12 op/s 2023-03-10T00:52:27.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:26 smithi122 ceph-mon[93869]: pgmap v19527: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 455 KiB/s rd, 479 KiB/s wr, 12 op/s 2023-03-10T00:52:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35658.953322 secs 2023-03-10T00:52:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35658.953322 secs 2023-03-10T00:52:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35658.953322 secs 2023-03-10T00:52:28.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36120.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:52:29.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:28 smithi038 ceph-mon[92422]: pgmap v19528: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 555 KiB/s rd, 568 KiB/s wr, 11 op/s 2023-03-10T00:52:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:28 smithi122 ceph-mon[93869]: pgmap v19528: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 555 KiB/s rd, 568 KiB/s wr, 11 op/s 2023-03-10T00:52:29.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:28 smithi181 ceph-mon[94189]: pgmap v19528: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 555 KiB/s rd, 568 KiB/s wr, 11 op/s 2023-03-10T00:52:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35661.494042 secs 2023-03-10T00:52:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:31 smithi122 ceph-mon[93869]: pgmap v19529: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 555 KiB/s rd, 568 KiB/s wr, 11 op/s 2023-03-10T00:52:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35661.494042 secs 2023-03-10T00:52:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:31 smithi181 ceph-mon[94189]: pgmap v19529: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 555 KiB/s rd, 568 KiB/s wr, 11 op/s 2023-03-10T00:52:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35661.494042 secs 2023-03-10T00:52:31.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:31 smithi038 ceph-mon[92422]: pgmap v19529: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 555 KiB/s rd, 568 KiB/s wr, 11 op/s 2023-03-10T00:52:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:33 smithi122 ceph-mon[93869]: pgmap v19530: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 202 KiB/s wr, 8 op/s 2023-03-10T00:52:33.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:33 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:52:33.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:33 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:52:33.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35663.953418 secs 2023-03-10T00:52:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:33 smithi181 ceph-mon[94189]: pgmap v19530: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 202 KiB/s wr, 8 op/s 2023-03-10T00:52:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:33 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:52:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:33 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:52:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35663.953418 secs 2023-03-10T00:52:33.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:33 smithi038 ceph-mon[92422]: pgmap v19530: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 202 KiB/s wr, 8 op/s 2023-03-10T00:52:33.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:33 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:52:33.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:33 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:52:33.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35663.953418 secs 2023-03-10T00:52:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36125.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:52:35.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:34 smithi122 ceph-mon[93869]: osdmap e1436: 12 total, 12 up, 12 in 2023-03-10T00:52:35.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:34 smithi122 ceph-mon[93869]: pgmap v19532: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 766 KiB/s rd, 765 KiB/s wr, 10 op/s 2023-03-10T00:52:35.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:34 smithi181 ceph-mon[94189]: osdmap e1436: 12 total, 12 up, 12 in 2023-03-10T00:52:35.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:34 smithi181 ceph-mon[94189]: pgmap v19532: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 766 KiB/s rd, 765 KiB/s wr, 10 op/s 2023-03-10T00:52:35.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:34 smithi038 ceph-mon[92422]: osdmap e1436: 12 total, 12 up, 12 in 2023-03-10T00:52:35.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:34 smithi038 ceph-mon[92422]: pgmap v19532: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 766 KiB/s rd, 765 KiB/s wr, 10 op/s 2023-03-10T00:52:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:35 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:52:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:35 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:52:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35666.494103 secs 2023-03-10T00:52:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:35 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:52:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:35 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:52:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35666.494103 secs 2023-03-10T00:52:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:35 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:52:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:35 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:52:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35666.494103 secs 2023-03-10T00:52:37.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:52:37.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:52:37.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:36 smithi038 ceph-mon[92422]: pgmap v19533: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 685 KiB/s rd, 684 KiB/s wr, 9 op/s 2023-03-10T00:52:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:52:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:52:37.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:36 smithi122 ceph-mon[93869]: pgmap v19533: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 685 KiB/s rd, 684 KiB/s wr, 9 op/s 2023-03-10T00:52:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:52:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:52:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:36 smithi181 ceph-mon[94189]: pgmap v19533: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 685 KiB/s rd, 684 KiB/s wr, 9 op/s 2023-03-10T00:52:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35668.953482 secs 2023-03-10T00:52:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35668.953482 secs 2023-03-10T00:52:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35668.953482 secs 2023-03-10T00:52:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36130.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:52:39.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:38 smithi038 ceph-mon[92422]: pgmap v19534: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 584 KiB/s rd, 582 KiB/s wr, 9 op/s 2023-03-10T00:52:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:38 smithi122 ceph-mon[93869]: pgmap v19534: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 584 KiB/s rd, 582 KiB/s wr, 9 op/s 2023-03-10T00:52:39.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:38 smithi181 ceph-mon[94189]: pgmap v19534: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 584 KiB/s rd, 582 KiB/s wr, 9 op/s 2023-03-10T00:52:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:52:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:52:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:52:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35671.494164 secs 2023-03-10T00:52:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:41 smithi122 ceph-mon[93869]: pgmap v19535: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 750 KiB/s rd, 748 KiB/s wr, 10 op/s 2023-03-10T00:52:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35671.494164 secs 2023-03-10T00:52:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:41 smithi181 ceph-mon[94189]: pgmap v19535: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 750 KiB/s rd, 748 KiB/s wr, 10 op/s 2023-03-10T00:52:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35671.494164 secs 2023-03-10T00:52:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:41 smithi038 ceph-mon[92422]: pgmap v19535: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 750 KiB/s rd, 748 KiB/s wr, 10 op/s 2023-03-10T00:52:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:43 smithi122 ceph-mon[93869]: pgmap v19536: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 714 KiB/s wr, 10 op/s 2023-03-10T00:52:43.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:52:43.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:52:43.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:52:43.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:52:43.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:52:43.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:52:43.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35673.953563 secs 2023-03-10T00:52:43.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:52:43.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:52:43.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:52:43.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:52:43.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:52:43.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:52:43.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:52:43.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:52:43.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:52:43.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:52:43.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:52:43.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:52:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:43 smithi181 ceph-mon[94189]: pgmap v19536: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 714 KiB/s wr, 10 op/s 2023-03-10T00:52:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:52:43.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:52:43.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:52:43.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:52:43.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:52:43.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:52:43.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35673.953563 secs 2023-03-10T00:52:43.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:52:43.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:52:43.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:52:43.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:52:43.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:52:43.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:52:43.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:52:43.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:52:43.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:52:43.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:52:43.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:52:43.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:52:43.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:43 smithi038 ceph-mon[92422]: pgmap v19536: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 714 KiB/s wr, 10 op/s 2023-03-10T00:52:43.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:52:43.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:52:43.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:52:43.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:52:43.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:52:43.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:52:43.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35673.953563 secs 2023-03-10T00:52:43.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:52:43.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:52:43.753 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:52:43.753 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:52:43.753 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:52:43.754 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:52:43.754 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:52:43.754 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:52:43.754 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-10T00:52:43.755 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:52:43.755 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:52:43.755 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:52:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36135.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:52:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:44 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-10T00:52:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:44 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-10T00:52:44.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:44 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-10T00:52:44.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:44 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-10T00:52:44.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:44 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-10T00:52:44.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:44 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-10T00:52:44.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:44 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-10T00:52:44.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:44 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-10T00:52:44.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:44 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-10T00:52:44.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:44 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-10T00:52:44.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:44 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-10T00:52:44.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:44 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-10T00:52:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:45 smithi122 ceph-mon[93869]: pgmap v19537: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 756 KiB/s rd, 754 KiB/s wr, 10 op/s 2023-03-10T00:52:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:45 smithi181 ceph-mon[94189]: pgmap v19537: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 756 KiB/s rd, 754 KiB/s wr, 10 op/s 2023-03-10T00:52:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:45 smithi038 ceph-mon[92422]: pgmap v19537: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 756 KiB/s rd, 754 KiB/s wr, 10 op/s 2023-03-10T00:52:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35676.494224 secs 2023-03-10T00:52:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35676.494224 secs 2023-03-10T00:52:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35676.494224 secs 2023-03-10T00:52:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:47 smithi122 ceph-mon[93869]: pgmap v19538: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 483 KiB/s rd, 480 KiB/s wr, 10 op/s 2023-03-10T00:52:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:47 smithi181 ceph-mon[94189]: pgmap v19538: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 483 KiB/s rd, 480 KiB/s wr, 10 op/s 2023-03-10T00:52:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:47 smithi038 ceph-mon[92422]: pgmap v19538: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 483 KiB/s rd, 480 KiB/s wr, 10 op/s 2023-03-10T00:52:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:52:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:52:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:52:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:52:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:52:48.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:52:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36140.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:52:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:49 smithi122 ceph-mon[93869]: pgmap v19539: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 514 KiB/s wr, 8 op/s 2023-03-10T00:52:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:52:49.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:52:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:49 smithi181 ceph-mon[94189]: pgmap v19539: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 514 KiB/s wr, 8 op/s 2023-03-10T00:52:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:52:49.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:52:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:49 smithi038 ceph-mon[92422]: pgmap v19539: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 514 KiB/s wr, 8 op/s 2023-03-10T00:52:49.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:52:49.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:52:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:50 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:52:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:50 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:52:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:50 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:52:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35681.494285 secs 2023-03-10T00:52:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:51 smithi122 ceph-mon[93869]: pgmap v19540: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 856 KiB/s rd, 738 KiB/s wr, 8 op/s 2023-03-10T00:52:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:51 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:52:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:52:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:52:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35681.494285 secs 2023-03-10T00:52:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:51 smithi181 ceph-mon[94189]: pgmap v19540: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 856 KiB/s rd, 738 KiB/s wr, 8 op/s 2023-03-10T00:52:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:51 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:52:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:52:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:52:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35681.494285 secs 2023-03-10T00:52:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:51 smithi038 ceph-mon[92422]: pgmap v19540: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 856 KiB/s rd, 738 KiB/s wr, 8 op/s 2023-03-10T00:52:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:51 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:52:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:52:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:52:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:53 smithi122 ceph-mon[93869]: pgmap v19541: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 748 KiB/s rd, 629 KiB/s wr, 10 op/s 2023-03-10T00:52:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:53 smithi181 ceph-mon[94189]: pgmap v19541: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 748 KiB/s rd, 629 KiB/s wr, 10 op/s 2023-03-10T00:52:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:53 smithi038 ceph-mon[92422]: pgmap v19541: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 748 KiB/s rd, 629 KiB/s wr, 10 op/s 2023-03-10T00:52:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36145.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:52:55.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:54 smithi038 ceph-mon[92422]: pgmap v19542: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 921 KiB/s rd, 629 KiB/s wr, 9 op/s 2023-03-10T00:52:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:54 smithi122 ceph-mon[93869]: pgmap v19542: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 921 KiB/s rd, 629 KiB/s wr, 9 op/s 2023-03-10T00:52:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:54 smithi181 ceph-mon[94189]: pgmap v19542: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 921 KiB/s rd, 629 KiB/s wr, 9 op/s 2023-03-10T00:52:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:52:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:52:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35686.494369 secs 2023-03-10T00:52:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:52:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:52:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35686.494369 secs 2023-03-10T00:52:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:52:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:52:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35686.494369 secs 2023-03-10T00:52:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:52:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:52:57.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:57 smithi122 ceph-mon[93869]: pgmap v19543: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 758 KiB/s rd, 463 KiB/s wr, 11 op/s 2023-03-10T00:52:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:52:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:52:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:57 smithi181 ceph-mon[94189]: pgmap v19543: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 758 KiB/s rd, 463 KiB/s wr, 11 op/s 2023-03-10T00:52:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:52:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:52:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:57 smithi038 ceph-mon[92422]: pgmap v19543: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 758 KiB/s rd, 463 KiB/s wr, 11 op/s 2023-03-10T00:52:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36150.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:52:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:52:59 smithi122 ceph-mon[93869]: pgmap v19544: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 757 KiB/s rd, 464 KiB/s wr, 10 op/s 2023-03-10T00:52:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:52:59 smithi181 ceph-mon[94189]: pgmap v19544: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 757 KiB/s rd, 464 KiB/s wr, 10 op/s 2023-03-10T00:52:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:52:59 smithi038 ceph-mon[92422]: pgmap v19544: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 757 KiB/s rd, 464 KiB/s wr, 10 op/s 2023-03-10T00:53:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:53:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:53:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:53:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:53:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:53:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:53:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35691.494452 secs 2023-03-10T00:53:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:01 smithi122 ceph-mon[93869]: pgmap v19545: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 781 KiB/s rd, 400 KiB/s wr, 10 op/s 2023-03-10T00:53:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:53:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:53:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35691.494452 secs 2023-03-10T00:53:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:01 smithi181 ceph-mon[94189]: pgmap v19545: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 781 KiB/s rd, 400 KiB/s wr, 10 op/s 2023-03-10T00:53:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:53:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:53:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35691.494452 secs 2023-03-10T00:53:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:01 smithi038 ceph-mon[92422]: pgmap v19545: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 781 KiB/s rd, 400 KiB/s wr, 10 op/s 2023-03-10T00:53:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:53:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:53:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:03 smithi122 ceph-mon[93869]: pgmap v19546: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 653 KiB/s wr, 13 op/s 2023-03-10T00:53:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:03 smithi181 ceph-mon[94189]: pgmap v19546: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 653 KiB/s wr, 13 op/s 2023-03-10T00:53:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:03 smithi038 ceph-mon[92422]: pgmap v19546: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 653 KiB/s wr, 13 op/s 2023-03-10T00:53:03.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36155.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:53:05.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:04 smithi038 ceph-mon[92422]: pgmap v19547: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1004 KiB/s rd, 624 KiB/s wr, 12 op/s 2023-03-10T00:53:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:04 smithi122 ceph-mon[93869]: pgmap v19547: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1004 KiB/s rd, 624 KiB/s wr, 12 op/s 2023-03-10T00:53:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:04 smithi181 ceph-mon[94189]: pgmap v19547: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1004 KiB/s rd, 624 KiB/s wr, 12 op/s 2023-03-10T00:53:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35696.494511 secs 2023-03-10T00:53:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35696.494511 secs 2023-03-10T00:53:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35696.494511 secs 2023-03-10T00:53:07.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:06 smithi038 ceph-mon[92422]: pgmap v19548: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 901 KiB/s wr, 17 op/s 2023-03-10T00:53:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:06 smithi122 ceph-mon[93869]: pgmap v19548: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 901 KiB/s wr, 17 op/s 2023-03-10T00:53:07.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:06 smithi181 ceph-mon[94189]: pgmap v19548: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 901 KiB/s wr, 17 op/s 2023-03-10T00:53:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36160.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:53:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:53:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:53:09.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:09 smithi122 ceph-mon[93869]: pgmap v19549: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 860 KiB/s wr, 23 op/s 2023-03-10T00:53:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:53:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:53:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:09 smithi181 ceph-mon[94189]: pgmap v19549: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 860 KiB/s wr, 23 op/s 2023-03-10T00:53:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:53:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:53:09.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:09 smithi038 ceph-mon[92422]: pgmap v19549: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 860 KiB/s wr, 23 op/s 2023-03-10T00:53:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35701.494569 secs 2023-03-10T00:53:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:11 smithi122 ceph-mon[93869]: pgmap v19550: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 860 KiB/s wr, 23 op/s 2023-03-10T00:53:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35701.494569 secs 2023-03-10T00:53:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:11 smithi181 ceph-mon[94189]: pgmap v19550: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 860 KiB/s wr, 23 op/s 2023-03-10T00:53:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35701.494569 secs 2023-03-10T00:53:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:11 smithi038 ceph-mon[92422]: pgmap v19550: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 860 KiB/s wr, 23 op/s 2023-03-10T00:53:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:53:12.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:53:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:53:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:53:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:53:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:53:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:13 smithi122 ceph-mon[93869]: pgmap v19551: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 28 op/s 2023-03-10T00:53:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:53:13.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:53:13.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35703.954093 secs 2023-03-10T00:53:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:13 smithi181 ceph-mon[94189]: pgmap v19551: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 28 op/s 2023-03-10T00:53:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:53:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:53:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35703.954093 secs 2023-03-10T00:53:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:13 smithi038 ceph-mon[92422]: pgmap v19551: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 28 op/s 2023-03-10T00:53:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:53:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:53:13.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35703.954093 secs 2023-03-10T00:53:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36165.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:53:15.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:14 smithi122 ceph-mon[93869]: pgmap v19552: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 788 KiB/s rd, 822 KiB/s wr, 25 op/s 2023-03-10T00:53:15.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:14 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:53:15.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:14 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:53:15.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:14 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:53:15.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:14 smithi181 ceph-mon[94189]: pgmap v19552: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 788 KiB/s rd, 822 KiB/s wr, 25 op/s 2023-03-10T00:53:15.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:14 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:53:15.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:14 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:53:15.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:14 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:53:15.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:14 smithi038 ceph-mon[92422]: pgmap v19552: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 788 KiB/s rd, 822 KiB/s wr, 25 op/s 2023-03-10T00:53:15.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:14 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:53:15.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:14 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:53:15.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:14 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:53:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35706.494655 secs 2023-03-10T00:53:16.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:53:16.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:53:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35706.494655 secs 2023-03-10T00:53:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:53:16.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:53:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35706.494655 secs 2023-03-10T00:53:16.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:53:16.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:53:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:17 smithi122 ceph-mon[93869]: pgmap v19553: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 789 KiB/s rd, 823 KiB/s wr, 27 op/s 2023-03-10T00:53:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:53:17.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:53:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:17 smithi181 ceph-mon[94189]: pgmap v19553: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 789 KiB/s rd, 823 KiB/s wr, 27 op/s 2023-03-10T00:53:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:53:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:53:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:17 smithi038 ceph-mon[92422]: pgmap v19553: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 789 KiB/s rd, 823 KiB/s wr, 27 op/s 2023-03-10T00:53:17.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:53:17.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:53:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35708.954152 secs 2023-03-10T00:53:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35708.954152 secs 2023-03-10T00:53:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35708.954152 secs 2023-03-10T00:53:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36170.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:53:19.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:19 smithi122 ceph-mon[93869]: pgmap v19554: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 521 KiB/s rd, 546 KiB/s wr, 22 op/s 2023-03-10T00:53:19.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:19 smithi181 ceph-mon[94189]: pgmap v19554: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 521 KiB/s rd, 546 KiB/s wr, 22 op/s 2023-03-10T00:53:19.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:19 smithi038 ceph-mon[92422]: pgmap v19554: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 521 KiB/s rd, 546 KiB/s wr, 22 op/s 2023-03-10T00:53:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:20 smithi038 ceph-mon[92422]: osdmap e1437: 12 total, 12 up, 12 in 2023-03-10T00:53:21.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35711.494718 secs 2023-03-10T00:53:21.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:20 smithi038 ceph-mon[92422]: pgmap v19556: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 528 KiB/s wr, 14 op/s 2023-03-10T00:53:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:20 smithi122 ceph-mon[93869]: osdmap e1437: 12 total, 12 up, 12 in 2023-03-10T00:53:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35711.494718 secs 2023-03-10T00:53:21.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:20 smithi122 ceph-mon[93869]: pgmap v19556: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 528 KiB/s wr, 14 op/s 2023-03-10T00:53:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:20 smithi181 ceph-mon[94189]: osdmap e1437: 12 total, 12 up, 12 in 2023-03-10T00:53:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35711.494718 secs 2023-03-10T00:53:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:20 smithi181 ceph-mon[94189]: pgmap v19556: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 528 KiB/s wr, 14 op/s 2023-03-10T00:53:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:23 smithi122 ceph-mon[93869]: pgmap v19557: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 209 KiB/s wr, 12 op/s 2023-03-10T00:53:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35713.954230 secs 2023-03-10T00:53:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:23 smithi181 ceph-mon[94189]: pgmap v19557: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 209 KiB/s wr, 12 op/s 2023-03-10T00:53:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35713.954230 secs 2023-03-10T00:53:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:23 smithi038 ceph-mon[92422]: pgmap v19557: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 209 KiB/s wr, 12 op/s 2023-03-10T00:53:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35713.954230 secs 2023-03-10T00:53:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36175.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:53:25.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:24 smithi038 ceph-mon[92422]: pgmap v19558: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 196 KiB/s wr, 10 op/s 2023-03-10T00:53:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:24 smithi122 ceph-mon[93869]: pgmap v19558: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 196 KiB/s wr, 10 op/s 2023-03-10T00:53:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:24 smithi181 ceph-mon[94189]: pgmap v19558: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 196 KiB/s wr, 10 op/s 2023-03-10T00:53:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35716.494778 secs 2023-03-10T00:53:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35716.494778 secs 2023-03-10T00:53:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35716.494778 secs 2023-03-10T00:53:27.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:26 smithi038 ceph-mon[92422]: pgmap v19559: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 195 KiB/s wr, 9 op/s 2023-03-10T00:53:27.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:26 smithi122 ceph-mon[93869]: pgmap v19559: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 195 KiB/s wr, 9 op/s 2023-03-10T00:53:27.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:26 smithi181 ceph-mon[94189]: pgmap v19559: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 195 KiB/s wr, 9 op/s 2023-03-10T00:53:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35718.954331 secs 2023-03-10T00:53:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35718.954331 secs 2023-03-10T00:53:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35718.954331 secs 2023-03-10T00:53:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36180.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:53:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:29 smithi122 ceph-mon[93869]: pgmap v19560: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 195 KiB/s wr, 9 op/s 2023-03-10T00:53:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:29 smithi122 ceph-mon[93869]: osdmap e1438: 12 total, 12 up, 12 in 2023-03-10T00:53:29.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:29 smithi181 ceph-mon[94189]: pgmap v19560: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 195 KiB/s wr, 9 op/s 2023-03-10T00:53:29.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:29 smithi181 ceph-mon[94189]: osdmap e1438: 12 total, 12 up, 12 in 2023-03-10T00:53:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:29 smithi038 ceph-mon[92422]: pgmap v19560: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 195 KiB/s wr, 9 op/s 2023-03-10T00:53:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:29 smithi038 ceph-mon[92422]: osdmap e1438: 12 total, 12 up, 12 in 2023-03-10T00:53:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35721.494864 secs 2023-03-10T00:53:31.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:31 smithi122 ceph-mon[93869]: pgmap v19562: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 195 KiB/s wr, 9 op/s 2023-03-10T00:53:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35721.494864 secs 2023-03-10T00:53:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:31 smithi181 ceph-mon[94189]: pgmap v19562: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 195 KiB/s wr, 9 op/s 2023-03-10T00:53:31.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35721.494864 secs 2023-03-10T00:53:31.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:31 smithi038 ceph-mon[92422]: pgmap v19562: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 195 KiB/s wr, 9 op/s 2023-03-10T00:53:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:33 smithi122 ceph-mon[93869]: pgmap v19563: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 108 KiB/s wr, 8 op/s 2023-03-10T00:53:33.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35723.954463 secs 2023-03-10T00:53:33.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:33 smithi181 ceph-mon[94189]: pgmap v19563: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 108 KiB/s wr, 8 op/s 2023-03-10T00:53:33.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35723.954463 secs 2023-03-10T00:53:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:33 smithi038 ceph-mon[92422]: pgmap v19563: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 108 KiB/s wr, 8 op/s 2023-03-10T00:53:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35723.954463 secs 2023-03-10T00:53:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36185.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:53:35.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:34 smithi038 ceph-mon[92422]: pgmap v19564: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 108 KiB/s wr, 8 op/s 2023-03-10T00:53:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:34 smithi122 ceph-mon[93869]: pgmap v19564: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 108 KiB/s wr, 8 op/s 2023-03-10T00:53:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:34 smithi181 ceph-mon[94189]: pgmap v19564: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 108 KiB/s wr, 8 op/s 2023-03-10T00:53:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35726.494949 secs 2023-03-10T00:53:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35726.494949 secs 2023-03-10T00:53:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35726.494949 secs 2023-03-10T00:53:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:37 smithi122 ceph-mon[93869]: pgmap v19565: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 108 KiB/s wr, 8 op/s 2023-03-10T00:53:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:37 smithi181 ceph-mon[94189]: pgmap v19565: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 108 KiB/s wr, 8 op/s 2023-03-10T00:53:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:37 smithi038 ceph-mon[92422]: pgmap v19565: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 108 KiB/s wr, 8 op/s 2023-03-10T00:53:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35728.954521 secs 2023-03-10T00:53:38.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35728.954521 secs 2023-03-10T00:53:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35728.954521 secs 2023-03-10T00:53:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36190.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:53:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:39 smithi122 ceph-mon[93869]: pgmap v19566: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 108 KiB/s wr, 8 op/s 2023-03-10T00:53:39.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:39 smithi181 ceph-mon[94189]: pgmap v19566: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 108 KiB/s wr, 8 op/s 2023-03-10T00:53:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:39 smithi038 ceph-mon[92422]: pgmap v19566: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 108 KiB/s wr, 8 op/s 2023-03-10T00:53:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35731.495002 secs 2023-03-10T00:53:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:41 smithi122 ceph-mon[93869]: pgmap v19567: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 95 KiB/s wr, 7 op/s 2023-03-10T00:53:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35731.495002 secs 2023-03-10T00:53:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:41 smithi181 ceph-mon[94189]: pgmap v19567: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 95 KiB/s wr, 7 op/s 2023-03-10T00:53:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35731.495002 secs 2023-03-10T00:53:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:41 smithi038 ceph-mon[92422]: pgmap v19567: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 95 KiB/s wr, 7 op/s 2023-03-10T00:53:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:43 smithi122 ceph-mon[93869]: pgmap v19568: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 90 KiB/s wr, 10 op/s 2023-03-10T00:53:43.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35733.954584 secs 2023-03-10T00:53:43.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:53:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:43 smithi181 ceph-mon[94189]: pgmap v19568: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 90 KiB/s wr, 10 op/s 2023-03-10T00:53:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35733.954584 secs 2023-03-10T00:53:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:53:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:43 smithi038 ceph-mon[92422]: pgmap v19568: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 94 KiB/s rd, 90 KiB/s wr, 10 op/s 2023-03-10T00:53:43.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35733.954584 secs 2023-03-10T00:53:43.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:53:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36195.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:53:45.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:44 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:53:45.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:44 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:53:45.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:44 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:53:45.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:44 smithi038 ceph-mon[92422]: pgmap v19569: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 6 op/s 2023-03-10T00:53:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:44 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:53:45.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:44 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:53:45.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:44 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:53:45.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:44 smithi122 ceph-mon[93869]: pgmap v19569: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 6 op/s 2023-03-10T00:53:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:44 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:53:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:44 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:53:45.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:44 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:53:45.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:44 smithi181 ceph-mon[94189]: pgmap v19569: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 6 op/s 2023-03-10T00:53:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35736.495061 secs 2023-03-10T00:53:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35736.495061 secs 2023-03-10T00:53:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35736.495061 secs 2023-03-10T00:53:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:47 smithi122 ceph-mon[93869]: pgmap v19570: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 144 KiB/s wr, 10 op/s 2023-03-10T00:53:47.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:47 smithi181 ceph-mon[94189]: pgmap v19570: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 144 KiB/s wr, 10 op/s 2023-03-10T00:53:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:47 smithi038 ceph-mon[92422]: pgmap v19570: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 144 KiB/s wr, 10 op/s 2023-03-10T00:53:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35738.954643 secs 2023-03-10T00:53:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35738.954643 secs 2023-03-10T00:53:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35738.954643 secs 2023-03-10T00:53:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36200.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:53:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:53:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:53:49.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:49 smithi122 ceph-mon[93869]: pgmap v19571: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 485 KiB/s wr, 7 op/s 2023-03-10T00:53:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:53:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:53:49.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:49 smithi181 ceph-mon[94189]: pgmap v19571: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 485 KiB/s wr, 7 op/s 2023-03-10T00:53:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:53:49.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:53:49.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:49 smithi038 ceph-mon[92422]: pgmap v19571: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 485 KiB/s wr, 7 op/s 2023-03-10T00:53:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35741.495122 secs 2023-03-10T00:53:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:51 smithi122 ceph-mon[93869]: pgmap v19572: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 485 KiB/s wr, 7 op/s 2023-03-10T00:53:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35741.495122 secs 2023-03-10T00:53:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:51 smithi181 ceph-mon[94189]: pgmap v19572: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 485 KiB/s wr, 7 op/s 2023-03-10T00:53:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35741.495122 secs 2023-03-10T00:53:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:51 smithi038 ceph-mon[92422]: pgmap v19572: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 485 KiB/s wr, 7 op/s 2023-03-10T00:53:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:53:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:53:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:53:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:53:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:53:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:53:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:53 smithi122 ceph-mon[93869]: pgmap v19573: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 970 KiB/s rd, 952 KiB/s wr, 25 op/s 2023-03-10T00:53:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:53:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:53:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35743.954702 secs 2023-03-10T00:53:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:53 smithi181 ceph-mon[94189]: pgmap v19573: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 970 KiB/s rd, 952 KiB/s wr, 25 op/s 2023-03-10T00:53:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:53:53.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:53:53.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35743.954702 secs 2023-03-10T00:53:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:53 smithi038 ceph-mon[92422]: pgmap v19573: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 970 KiB/s rd, 952 KiB/s wr, 25 op/s 2023-03-10T00:53:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:53:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:53:53.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35743.954702 secs 2023-03-10T00:53:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36205.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:53:55.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:54 smithi038 ceph-mon[92422]: pgmap v19574: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 952 KiB/s wr, 23 op/s 2023-03-10T00:53:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:54 smithi122 ceph-mon[93869]: pgmap v19574: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 952 KiB/s wr, 23 op/s 2023-03-10T00:53:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:54 smithi181 ceph-mon[94189]: pgmap v19574: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 952 KiB/s wr, 23 op/s 2023-03-10T00:53:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:53:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:53:56.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35746.495182 secs 2023-03-10T00:53:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:53:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:53:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35746.495182 secs 2023-03-10T00:53:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:53:56.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:53:56.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35746.495182 secs 2023-03-10T00:53:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:53:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:53:57.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:57 smithi122 ceph-mon[93869]: pgmap v19575: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 36 op/s 2023-03-10T00:53:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:53:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:53:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:57 smithi181 ceph-mon[94189]: pgmap v19575: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 36 op/s 2023-03-10T00:53:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:53:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:53:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:57 smithi038 ceph-mon[92422]: pgmap v19575: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 36 op/s 2023-03-10T00:53:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:58 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:53:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:58 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:53:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:58 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:53:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36210.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:53:59.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:59 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:53:59.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:53:59.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:53:59.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:53:59 smithi122 ceph-mon[93869]: pgmap v19576: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 35 op/s 2023-03-10T00:53:59.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:59 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:53:59.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:53:59.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:53:59.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:53:59 smithi181 ceph-mon[94189]: pgmap v19576: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 35 op/s 2023-03-10T00:53:59.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:59 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:53:59.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:53:59.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:53:59.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:53:59 smithi038 ceph-mon[92422]: pgmap v19576: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 35 op/s 2023-03-10T00:54:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35751.495241 secs 2023-03-10T00:54:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:01 smithi122 ceph-mon[93869]: pgmap v19577: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1024 KiB/s rd, 858 KiB/s wr, 34 op/s 2023-03-10T00:54:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35751.495241 secs 2023-03-10T00:54:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:01 smithi181 ceph-mon[94189]: pgmap v19577: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1024 KiB/s rd, 858 KiB/s wr, 34 op/s 2023-03-10T00:54:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35751.495241 secs 2023-03-10T00:54:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:01 smithi038 ceph-mon[92422]: pgmap v19577: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1024 KiB/s rd, 858 KiB/s wr, 34 op/s 2023-03-10T00:54:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:03 smithi122 ceph-mon[93869]: pgmap v19578: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 974 KiB/s wr, 41 op/s 2023-03-10T00:54:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35753.954877 secs 2023-03-10T00:54:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:03 smithi181 ceph-mon[94189]: pgmap v19578: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 974 KiB/s wr, 41 op/s 2023-03-10T00:54:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35753.954877 secs 2023-03-10T00:54:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:03 smithi038 ceph-mon[92422]: pgmap v19578: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 974 KiB/s wr, 41 op/s 2023-03-10T00:54:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35753.954877 secs 2023-03-10T00:54:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36215.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:54:05.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:04 smithi038 ceph-mon[92422]: pgmap v19579: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 899 KiB/s rd, 784 KiB/s wr, 37 op/s 2023-03-10T00:54:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:04 smithi122 ceph-mon[93869]: pgmap v19579: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 899 KiB/s rd, 784 KiB/s wr, 37 op/s 2023-03-10T00:54:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:04 smithi181 ceph-mon[94189]: pgmap v19579: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 899 KiB/s rd, 784 KiB/s wr, 37 op/s 2023-03-10T00:54:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:06 smithi122 ceph-mon[93869]: osdmap e1439: 12 total, 12 up, 12 in 2023-03-10T00:54:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35756.495308 secs 2023-03-10T00:54:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:06 smithi181 ceph-mon[94189]: osdmap e1439: 12 total, 12 up, 12 in 2023-03-10T00:54:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35756.495308 secs 2023-03-10T00:54:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:06 smithi038 ceph-mon[92422]: osdmap e1439: 12 total, 12 up, 12 in 2023-03-10T00:54:06.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35756.495308 secs 2023-03-10T00:54:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:07 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:54:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:54:07.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:54:07.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:07 smithi122 ceph-mon[93869]: pgmap v19581: 129 pgs: 1 active+clean+snaptrim, 3 active+clean+snaptrim_wait, 125 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 840 KiB/s wr, 31 op/s 2023-03-10T00:54:07.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:07 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:54:07.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:54:07.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:54:07.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:07 smithi181 ceph-mon[94189]: pgmap v19581: 129 pgs: 1 active+clean+snaptrim, 3 active+clean+snaptrim_wait, 125 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 840 KiB/s wr, 31 op/s 2023-03-10T00:54:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:07 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:54:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:54:07.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:54:07.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:07 smithi038 ceph-mon[92422]: pgmap v19581: 129 pgs: 1 active+clean+snaptrim, 3 active+clean+snaptrim_wait, 125 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 840 KiB/s wr, 31 op/s 2023-03-10T00:54:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:54:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:54:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35758.954961 secs 2023-03-10T00:54:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:54:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:54:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35758.954961 secs 2023-03-10T00:54:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:54:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:54:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35758.954961 secs 2023-03-10T00:54:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36220.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:54:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:54:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:54:09.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:09 smithi122 ceph-mon[93869]: pgmap v19582: 129 pgs: 1 active+clean+snaptrim, 3 active+clean+snaptrim_wait, 125 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 467 KiB/s rd, 507 KiB/s wr, 29 op/s 2023-03-10T00:54:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:54:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:54:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:09 smithi181 ceph-mon[94189]: pgmap v19582: 129 pgs: 1 active+clean+snaptrim, 3 active+clean+snaptrim_wait, 125 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 467 KiB/s rd, 507 KiB/s wr, 29 op/s 2023-03-10T00:54:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:54:09.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:54:09.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:09 smithi038 ceph-mon[92422]: pgmap v19582: 129 pgs: 1 active+clean+snaptrim, 3 active+clean+snaptrim_wait, 125 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 467 KiB/s rd, 507 KiB/s wr, 29 op/s 2023-03-10T00:54:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35761.495375 secs 2023-03-10T00:54:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:11 smithi122 ceph-mon[93869]: pgmap v19583: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 570 KiB/s rd, 611 KiB/s wr, 30 op/s 2023-03-10T00:54:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35761.495375 secs 2023-03-10T00:54:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:11 smithi181 ceph-mon[94189]: pgmap v19583: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 570 KiB/s rd, 611 KiB/s wr, 30 op/s 2023-03-10T00:54:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35761.495375 secs 2023-03-10T00:54:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:11 smithi038 ceph-mon[92422]: pgmap v19583: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 570 KiB/s rd, 611 KiB/s wr, 30 op/s 2023-03-10T00:54:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:13 smithi122 ceph-mon[93869]: pgmap v19584: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 652 KiB/s rd, 667 KiB/s wr, 26 op/s 2023-03-10T00:54:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:54:13.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:54:13.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35763.955050 secs 2023-03-10T00:54:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:13 smithi181 ceph-mon[94189]: pgmap v19584: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 652 KiB/s rd, 667 KiB/s wr, 26 op/s 2023-03-10T00:54:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:54:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:54:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35763.955050 secs 2023-03-10T00:54:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:13 smithi038 ceph-mon[92422]: pgmap v19584: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 652 KiB/s rd, 667 KiB/s wr, 26 op/s 2023-03-10T00:54:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:54:13.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:54:13.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35763.955050 secs 2023-03-10T00:54:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36225.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:54:15.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:14 smithi038 ceph-mon[92422]: osdmap e1440: 12 total, 12 up, 12 in 2023-03-10T00:54:15.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:14 smithi038 ceph-mon[92422]: pgmap v19586: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 358 KiB/s wr, 10 op/s 2023-03-10T00:54:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:14 smithi122 ceph-mon[93869]: osdmap e1440: 12 total, 12 up, 12 in 2023-03-10T00:54:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:14 smithi122 ceph-mon[93869]: pgmap v19586: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 358 KiB/s wr, 10 op/s 2023-03-10T00:54:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:14 smithi181 ceph-mon[94189]: osdmap e1440: 12 total, 12 up, 12 in 2023-03-10T00:54:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:14 smithi181 ceph-mon[94189]: pgmap v19586: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 358 KiB/s wr, 10 op/s 2023-03-10T00:54:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:54:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:54:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35766.495433 secs 2023-03-10T00:54:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:54:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:54:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35766.495433 secs 2023-03-10T00:54:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:54:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:54:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35766.495433 secs 2023-03-10T00:54:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:17 smithi122 ceph-mon[93869]: pgmap v19587: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 299 KiB/s wr, 8 op/s 2023-03-10T00:54:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:17 smithi181 ceph-mon[94189]: pgmap v19587: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 299 KiB/s wr, 8 op/s 2023-03-10T00:54:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:17 smithi038 ceph-mon[92422]: pgmap v19587: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 299 KiB/s wr, 8 op/s 2023-03-10T00:54:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35768.955134 secs 2023-03-10T00:54:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35768.955134 secs 2023-03-10T00:54:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35768.955134 secs 2023-03-10T00:54:18.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36230.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:54:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:19 smithi122 ceph-mon[93869]: pgmap v19588: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 299 KiB/s wr, 8 op/s 2023-03-10T00:54:19.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:19 smithi181 ceph-mon[94189]: pgmap v19588: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 299 KiB/s wr, 8 op/s 2023-03-10T00:54:19.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:19 smithi038 ceph-mon[92422]: pgmap v19588: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 299 KiB/s wr, 8 op/s 2023-03-10T00:54:21.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35771.495513 secs 2023-03-10T00:54:21.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:21 smithi122 ceph-mon[93869]: pgmap v19589: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 195 KiB/s wr, 7 op/s 2023-03-10T00:54:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35771.495513 secs 2023-03-10T00:54:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:21 smithi181 ceph-mon[94189]: pgmap v19589: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 195 KiB/s wr, 7 op/s 2023-03-10T00:54:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35771.495513 secs 2023-03-10T00:54:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:21 smithi038 ceph-mon[92422]: pgmap v19589: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 195 KiB/s wr, 7 op/s 2023-03-10T00:54:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:23 smithi122 ceph-mon[93869]: pgmap v19590: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-10T00:54:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35773.955192 secs 2023-03-10T00:54:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:23 smithi181 ceph-mon[94189]: pgmap v19590: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-10T00:54:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35773.955192 secs 2023-03-10T00:54:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:23 smithi038 ceph-mon[92422]: pgmap v19590: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 7 op/s 2023-03-10T00:54:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35773.955192 secs 2023-03-10T00:54:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36235.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:54:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:25 smithi122 ceph-mon[93869]: pgmap v19591: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2023-03-10T00:54:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:25 smithi181 ceph-mon[94189]: pgmap v19591: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2023-03-10T00:54:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:25 smithi038 ceph-mon[92422]: pgmap v19591: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2023-03-10T00:54:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35776.495573 secs 2023-03-10T00:54:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35776.495573 secs 2023-03-10T00:54:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35776.495573 secs 2023-03-10T00:54:27.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:27 smithi122 ceph-mon[93869]: pgmap v19592: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-10T00:54:27.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:27 smithi181 ceph-mon[94189]: pgmap v19592: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-10T00:54:27.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:27 smithi038 ceph-mon[92422]: pgmap v19592: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-10T00:54:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35778.955282 secs 2023-03-10T00:54:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35778.955282 secs 2023-03-10T00:54:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35778.955282 secs 2023-03-10T00:54:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36240.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:54:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:29 smithi122 ceph-mon[93869]: pgmap v19593: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-10T00:54:29.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:29 smithi181 ceph-mon[94189]: pgmap v19593: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-10T00:54:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:29 smithi038 ceph-mon[92422]: pgmap v19593: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-03-10T00:54:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35781.495631 secs 2023-03-10T00:54:31.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:31 smithi122 ceph-mon[93869]: pgmap v19594: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-10T00:54:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35781.495631 secs 2023-03-10T00:54:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:31 smithi181 ceph-mon[94189]: pgmap v19594: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-10T00:54:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35781.495631 secs 2023-03-10T00:54:31.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:31 smithi038 ceph-mon[92422]: pgmap v19594: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-10T00:54:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:33 smithi122 ceph-mon[93869]: pgmap v19595: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-10T00:54:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35783.955364 secs 2023-03-10T00:54:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:33 smithi181 ceph-mon[94189]: pgmap v19595: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-10T00:54:33.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35783.955364 secs 2023-03-10T00:54:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:33 smithi038 ceph-mon[92422]: pgmap v19595: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-10T00:54:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35783.955364 secs 2023-03-10T00:54:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36245.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:54:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:35 smithi122 ceph-mon[93869]: pgmap v19596: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-10T00:54:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:35 smithi181 ceph-mon[94189]: pgmap v19596: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-10T00:54:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:35 smithi038 ceph-mon[92422]: pgmap v19596: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2023-03-10T00:54:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35786.495690 secs 2023-03-10T00:54:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35786.495690 secs 2023-03-10T00:54:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35786.495690 secs 2023-03-10T00:54:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:37 smithi122 ceph-mon[93869]: pgmap v19597: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 511 B/s wr, 8 op/s 2023-03-10T00:54:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:37 smithi181 ceph-mon[94189]: pgmap v19597: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 511 B/s wr, 8 op/s 2023-03-10T00:54:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:37 smithi038 ceph-mon[92422]: pgmap v19597: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 511 B/s wr, 8 op/s 2023-03-10T00:54:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35788.955422 secs 2023-03-10T00:54:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35788.955422 secs 2023-03-10T00:54:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35788.955422 secs 2023-03-10T00:54:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36250.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:54:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:39 smithi122 ceph-mon[93869]: pgmap v19598: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 682 B/s wr, 6 op/s 2023-03-10T00:54:39.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:39 smithi181 ceph-mon[94189]: pgmap v19598: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 682 B/s wr, 6 op/s 2023-03-10T00:54:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:39 smithi038 ceph-mon[92422]: pgmap v19598: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 682 B/s wr, 6 op/s 2023-03-10T00:54:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35791.495749 secs 2023-03-10T00:54:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:41 smithi122 ceph-mon[93869]: pgmap v19599: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 682 B/s wr, 7 op/s 2023-03-10T00:54:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35791.495749 secs 2023-03-10T00:54:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:41 smithi181 ceph-mon[94189]: pgmap v19599: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 682 B/s wr, 7 op/s 2023-03-10T00:54:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35791.495749 secs 2023-03-10T00:54:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:41 smithi038 ceph-mon[92422]: pgmap v19599: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 682 B/s wr, 7 op/s 2023-03-10T00:54:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:43 smithi122 ceph-mon[93869]: pgmap v19600: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 145 KiB/s wr, 9 op/s 2023-03-10T00:54:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35793.955532 secs 2023-03-10T00:54:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:43 smithi181 ceph-mon[94189]: pgmap v19600: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 145 KiB/s wr, 9 op/s 2023-03-10T00:54:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35793.955532 secs 2023-03-10T00:54:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:43 smithi038 ceph-mon[92422]: pgmap v19600: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 145 KiB/s wr, 9 op/s 2023-03-10T00:54:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35793.955532 secs 2023-03-10T00:54:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36255.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:54:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:54:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:45 smithi122 ceph-mon[93869]: pgmap v19601: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 145 KiB/s wr, 6 op/s 2023-03-10T00:54:45.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:54:45.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:54:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:54:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:45 smithi181 ceph-mon[94189]: pgmap v19601: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 145 KiB/s wr, 6 op/s 2023-03-10T00:54:45.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:54:45.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:54:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:54:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:45 smithi038 ceph-mon[92422]: pgmap v19601: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 145 KiB/s wr, 6 op/s 2023-03-10T00:54:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:54:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:54:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35796.495832 secs 2023-03-10T00:54:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:54:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:54:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:54:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35796.495832 secs 2023-03-10T00:54:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:54:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:54:46.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:54:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35796.495832 secs 2023-03-10T00:54:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:54:46.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:54:46.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:54:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:47 smithi122 ceph-mon[93869]: pgmap v19602: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 289 KiB/s wr, 9 op/s 2023-03-10T00:54:47.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:47 smithi181 ceph-mon[94189]: pgmap v19602: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 289 KiB/s wr, 9 op/s 2023-03-10T00:54:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:47 smithi038 ceph-mon[92422]: pgmap v19602: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 289 KiB/s wr, 9 op/s 2023-03-10T00:54:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36260.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:54:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:49 smithi122 ceph-mon[93869]: pgmap v19603: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 289 KiB/s wr, 7 op/s 2023-03-10T00:54:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:49 smithi181 ceph-mon[94189]: pgmap v19603: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 289 KiB/s wr, 7 op/s 2023-03-10T00:54:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:49 smithi038 ceph-mon[92422]: pgmap v19603: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 289 KiB/s wr, 7 op/s 2023-03-10T00:54:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:50 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:54:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:50 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:54:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:50 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:54:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35801.495912 secs 2023-03-10T00:54:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:51 smithi122 ceph-mon[93869]: pgmap v19604: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 288 KiB/s wr, 6 op/s 2023-03-10T00:54:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:51 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:54:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:54:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:54:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35801.495912 secs 2023-03-10T00:54:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:51 smithi181 ceph-mon[94189]: pgmap v19604: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 288 KiB/s wr, 6 op/s 2023-03-10T00:54:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:51 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:54:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:54:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:54:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35801.495912 secs 2023-03-10T00:54:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:51 smithi038 ceph-mon[92422]: pgmap v19604: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 288 KiB/s wr, 6 op/s 2023-03-10T00:54:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:51 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:54:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:54:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:54:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:54:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:54:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:54:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:54:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:54:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:54:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:53 smithi122 ceph-mon[93869]: pgmap v19605: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 862 KiB/s rd, 740 KiB/s wr, 10 op/s 2023-03-10T00:54:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:54:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:54:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:53 smithi181 ceph-mon[94189]: pgmap v19605: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 862 KiB/s rd, 740 KiB/s wr, 10 op/s 2023-03-10T00:54:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:54:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:54:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:53 smithi038 ceph-mon[92422]: pgmap v19605: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 862 KiB/s rd, 740 KiB/s wr, 10 op/s 2023-03-10T00:54:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:54:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:54:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36265.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:54:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:55 smithi122 ceph-mon[93869]: pgmap v19606: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 892 KiB/s wr, 8 op/s 2023-03-10T00:54:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:55 smithi181 ceph-mon[94189]: pgmap v19606: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 892 KiB/s wr, 8 op/s 2023-03-10T00:54:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:55 smithi038 ceph-mon[92422]: pgmap v19606: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 892 KiB/s wr, 8 op/s 2023-03-10T00:54:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:54:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:54:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35806.495975 secs 2023-03-10T00:54:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:54:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:54:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35806.495975 secs 2023-03-10T00:54:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:54:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:54:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35806.495975 secs 2023-03-10T00:54:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:54:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:54:57.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:57 smithi122 ceph-mon[93869]: pgmap v19607: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 922 KiB/s wr, 12 op/s 2023-03-10T00:54:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:54:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:54:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:57 smithi181 ceph-mon[94189]: pgmap v19607: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 922 KiB/s wr, 12 op/s 2023-03-10T00:54:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:54:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:54:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:57 smithi038 ceph-mon[92422]: pgmap v19607: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 922 KiB/s wr, 12 op/s 2023-03-10T00:54:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:58 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 35808.955778 secs 2023-03-10T00:54:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:58 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 35808.955778 secs 2023-03-10T00:54:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:58 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 35808.955778 secs 2023-03-10T00:54:58.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36270.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:54:59.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:59 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:54:59.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:54:59.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:54:59.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:54:59 smithi122 ceph-mon[93869]: pgmap v19608: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.2 MiB/s wr, 10 op/s 2023-03-10T00:54:59.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:59 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:54:59.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:54:59.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:54:59.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:54:59 smithi181 ceph-mon[94189]: pgmap v19608: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.2 MiB/s wr, 10 op/s 2023-03-10T00:54:59.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:59 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:54:59.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:54:59.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:54:59.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:54:59 smithi038 ceph-mon[92422]: pgmap v19608: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.2 MiB/s wr, 10 op/s 2023-03-10T00:55:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:55:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:55:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:55:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:55:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:55:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:55:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35811.496046 secs 2023-03-10T00:55:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:55:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:55:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:01 smithi122 ceph-mon[93869]: pgmap v19609: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 11 op/s 2023-03-10T00:55:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35811.496046 secs 2023-03-10T00:55:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:55:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:55:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:01 smithi181 ceph-mon[94189]: pgmap v19609: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 11 op/s 2023-03-10T00:55:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35811.496046 secs 2023-03-10T00:55:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:55:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:55:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:01 smithi038 ceph-mon[92422]: pgmap v19609: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 11 op/s 2023-03-10T00:55:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:03 smithi122 ceph-mon[93869]: pgmap v19610: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 15 op/s 2023-03-10T00:55:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:03 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 35813.955835 secs 2023-03-10T00:55:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:03 smithi181 ceph-mon[94189]: pgmap v19610: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 15 op/s 2023-03-10T00:55:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:03 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 35813.955835 secs 2023-03-10T00:55:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:03 smithi038 ceph-mon[92422]: pgmap v19610: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 15 op/s 2023-03-10T00:55:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:03 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 35813.955835 secs 2023-03-10T00:55:03.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36275.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:55:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:05 smithi122 ceph-mon[93869]: pgmap v19611: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 994 KiB/s wr, 13 op/s 2023-03-10T00:55:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:05 smithi181 ceph-mon[94189]: pgmap v19611: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 994 KiB/s wr, 13 op/s 2023-03-10T00:55:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:05 smithi038 ceph-mon[92422]: pgmap v19611: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 994 KiB/s wr, 13 op/s 2023-03-10T00:55:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35816.496104 secs 2023-03-10T00:55:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35816.496104 secs 2023-03-10T00:55:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35816.496104 secs 2023-03-10T00:55:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:07 smithi122 ceph-mon[93869]: pgmap v19612: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 975 KiB/s wr, 17 op/s 2023-03-10T00:55:07.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:07 smithi181 ceph-mon[94189]: pgmap v19612: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 975 KiB/s wr, 17 op/s 2023-03-10T00:55:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:07 smithi038 ceph-mon[92422]: pgmap v19612: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 975 KiB/s wr, 17 op/s 2023-03-10T00:55:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:55:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:55:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35818.955926 secs 2023-03-10T00:55:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:55:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:55:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35818.955926 secs 2023-03-10T00:55:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:55:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:55:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35818.955926 secs 2023-03-10T00:55:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36280.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:55:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:55:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:55:09.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:09 smithi122 ceph-mon[93869]: pgmap v19613: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 957 KiB/s wr, 24 op/s 2023-03-10T00:55:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:55:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:55:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:09 smithi181 ceph-mon[94189]: pgmap v19613: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 957 KiB/s wr, 24 op/s 2023-03-10T00:55:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:55:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:55:09.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:09 smithi038 ceph-mon[92422]: pgmap v19613: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 957 KiB/s wr, 24 op/s 2023-03-10T00:55:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:11 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:55:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:11 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:55:11.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35821.496164 secs 2023-03-10T00:55:11.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:11 smithi122 ceph-mon[93869]: pgmap v19614: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 871 KiB/s wr, 24 op/s 2023-03-10T00:55:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:11 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:55:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:11 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:55:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35821.496164 secs 2023-03-10T00:55:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:11 smithi181 ceph-mon[94189]: pgmap v19614: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 871 KiB/s wr, 24 op/s 2023-03-10T00:55:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:11 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:55:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:11 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:55:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35821.496164 secs 2023-03-10T00:55:11.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:11 smithi038 ceph-mon[92422]: pgmap v19614: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 871 KiB/s wr, 24 op/s 2023-03-10T00:55:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:55:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:55:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:55:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:55:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:55:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:55:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:55:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:55:13.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:13 smithi122 ceph-mon[93869]: pgmap v19615: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 29 op/s 2023-03-10T00:55:13.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35823.956042 secs 2023-03-10T00:55:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:55:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:55:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:13 smithi181 ceph-mon[94189]: pgmap v19615: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 29 op/s 2023-03-10T00:55:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35823.956042 secs 2023-03-10T00:55:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:55:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:55:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:13 smithi038 ceph-mon[92422]: pgmap v19615: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 29 op/s 2023-03-10T00:55:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35823.956042 secs 2023-03-10T00:55:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36285.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:55:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:15 smithi122 ceph-mon[93869]: pgmap v19616: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 27 op/s 2023-03-10T00:55:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:15 smithi181 ceph-mon[94189]: pgmap v19616: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 27 op/s 2023-03-10T00:55:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:15 smithi038 ceph-mon[92422]: pgmap v19616: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 27 op/s 2023-03-10T00:55:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:16 smithi122 ceph-mon[93869]: osdmap e1441: 12 total, 12 up, 12 in 2023-03-10T00:55:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:55:16.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:55:16.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35826.496235 secs 2023-03-10T00:55:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:16 smithi181 ceph-mon[94189]: osdmap e1441: 12 total, 12 up, 12 in 2023-03-10T00:55:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:55:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:55:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35826.496235 secs 2023-03-10T00:55:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:16 smithi038 ceph-mon[92422]: osdmap e1441: 12 total, 12 up, 12 in 2023-03-10T00:55:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:55:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:55:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35826.496235 secs 2023-03-10T00:55:17.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:55:17.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:55:17.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:17 smithi122 ceph-mon[93869]: pgmap v19618: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 28 op/s 2023-03-10T00:55:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:55:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:55:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:17 smithi181 ceph-mon[94189]: pgmap v19618: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 28 op/s 2023-03-10T00:55:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:55:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:55:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:17 smithi038 ceph-mon[92422]: pgmap v19618: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 28 op/s 2023-03-10T00:55:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35828.956106 secs 2023-03-10T00:55:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35828.956106 secs 2023-03-10T00:55:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35828.956106 secs 2023-03-10T00:55:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36290.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:55:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:19 smithi122 ceph-mon[93869]: pgmap v19619: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 15 op/s 2023-03-10T00:55:19.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:19 smithi181 ceph-mon[94189]: pgmap v19619: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 15 op/s 2023-03-10T00:55:19.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:19 smithi038 ceph-mon[92422]: pgmap v19619: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 15 op/s 2023-03-10T00:55:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35831.496291 secs 2023-03-10T00:55:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:21 smithi122 ceph-mon[93869]: pgmap v19620: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 739 KiB/s rd, 763 KiB/s wr, 13 op/s 2023-03-10T00:55:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35831.496291 secs 2023-03-10T00:55:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:21 smithi181 ceph-mon[94189]: pgmap v19620: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 739 KiB/s rd, 763 KiB/s wr, 13 op/s 2023-03-10T00:55:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35831.496291 secs 2023-03-10T00:55:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:21 smithi038 ceph-mon[92422]: pgmap v19620: 129 pgs: 129 active+clean; 13 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 739 KiB/s rd, 763 KiB/s wr, 13 op/s 2023-03-10T00:55:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:23 smithi122 ceph-mon[93869]: pgmap v19621: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 307 KiB/s wr, 11 op/s 2023-03-10T00:55:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35833.956165 secs 2023-03-10T00:55:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:23 smithi181 ceph-mon[94189]: pgmap v19621: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 307 KiB/s wr, 11 op/s 2023-03-10T00:55:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35833.956165 secs 2023-03-10T00:55:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:23 smithi038 ceph-mon[92422]: pgmap v19621: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 307 KiB/s wr, 11 op/s 2023-03-10T00:55:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35833.956165 secs 2023-03-10T00:55:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36295.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:55:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:25 smithi122 ceph-mon[93869]: osdmap e1442: 12 total, 12 up, 12 in 2023-03-10T00:55:25.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:25 smithi122 ceph-mon[93869]: pgmap v19623: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 337 KiB/s rd, 334 KiB/s wr, 10 op/s 2023-03-10T00:55:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:25 smithi181 ceph-mon[94189]: osdmap e1442: 12 total, 12 up, 12 in 2023-03-10T00:55:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:25 smithi181 ceph-mon[94189]: pgmap v19623: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 337 KiB/s rd, 334 KiB/s wr, 10 op/s 2023-03-10T00:55:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:25 smithi038 ceph-mon[92422]: osdmap e1442: 12 total, 12 up, 12 in 2023-03-10T00:55:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:25 smithi038 ceph-mon[92422]: pgmap v19623: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 337 KiB/s rd, 334 KiB/s wr, 10 op/s 2023-03-10T00:55:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35836.496375 secs 2023-03-10T00:55:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35836.496375 secs 2023-03-10T00:55:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35836.496375 secs 2023-03-10T00:55:27.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:27 smithi122 ceph-mon[93869]: pgmap v19624: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 138 KiB/s wr, 9 op/s 2023-03-10T00:55:27.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:27 smithi181 ceph-mon[94189]: pgmap v19624: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 138 KiB/s wr, 9 op/s 2023-03-10T00:55:27.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:27 smithi038 ceph-mon[92422]: pgmap v19624: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 138 KiB/s wr, 9 op/s 2023-03-10T00:55:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35838.956225 secs 2023-03-10T00:55:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35838.956225 secs 2023-03-10T00:55:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35838.956225 secs 2023-03-10T00:55:28.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36300.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:55:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:29 smithi122 ceph-mon[93869]: pgmap v19625: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 138 KiB/s wr, 9 op/s 2023-03-10T00:55:29.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:29 smithi181 ceph-mon[94189]: pgmap v19625: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 138 KiB/s wr, 9 op/s 2023-03-10T00:55:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:29 smithi038 ceph-mon[92422]: pgmap v19625: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 138 KiB/s wr, 9 op/s 2023-03-10T00:55:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35841.496433 secs 2023-03-10T00:55:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:31 smithi122 ceph-mon[93869]: pgmap v19626: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 138 KiB/s wr, 9 op/s 2023-03-10T00:55:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35841.496433 secs 2023-03-10T00:55:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:31 smithi181 ceph-mon[94189]: pgmap v19626: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 138 KiB/s wr, 9 op/s 2023-03-10T00:55:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35841.496433 secs 2023-03-10T00:55:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:31 smithi038 ceph-mon[92422]: pgmap v19626: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 138 KiB/s wr, 9 op/s 2023-03-10T00:55:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:33 smithi122 ceph-mon[93869]: pgmap v19627: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 18 KiB/s wr, 9 op/s 2023-03-10T00:55:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35843.956283 secs 2023-03-10T00:55:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:33 smithi181 ceph-mon[94189]: pgmap v19627: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 18 KiB/s wr, 9 op/s 2023-03-10T00:55:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35843.956283 secs 2023-03-10T00:55:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:33 smithi038 ceph-mon[92422]: pgmap v19627: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 18 KiB/s wr, 9 op/s 2023-03-10T00:55:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35843.956283 secs 2023-03-10T00:55:33.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36305.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:55:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:35 smithi122 ceph-mon[93869]: pgmap v19628: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 9 op/s 2023-03-10T00:55:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:35 smithi181 ceph-mon[94189]: pgmap v19628: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 9 op/s 2023-03-10T00:55:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:35 smithi038 ceph-mon[92422]: pgmap v19628: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 9 op/s 2023-03-10T00:55:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35846.496490 secs 2023-03-10T00:55:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35846.496490 secs 2023-03-10T00:55:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35846.496490 secs 2023-03-10T00:55:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:37 smithi122 ceph-mon[93869]: pgmap v19629: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 10 op/s 2023-03-10T00:55:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:37 smithi181 ceph-mon[94189]: pgmap v19629: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 10 op/s 2023-03-10T00:55:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:37 smithi038 ceph-mon[92422]: pgmap v19629: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 10 op/s 2023-03-10T00:55:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35848.956342 secs 2023-03-10T00:55:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35848.956342 secs 2023-03-10T00:55:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35848.956342 secs 2023-03-10T00:55:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36310.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:55:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:39 smithi122 ceph-mon[93869]: pgmap v19630: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 8 op/s 2023-03-10T00:55:39.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:39 smithi181 ceph-mon[94189]: pgmap v19630: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 8 op/s 2023-03-10T00:55:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:39 smithi038 ceph-mon[92422]: pgmap v19630: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 8 op/s 2023-03-10T00:55:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35851.496581 secs 2023-03-10T00:55:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:41 smithi122 ceph-mon[93869]: pgmap v19631: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 8 op/s 2023-03-10T00:55:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35851.496581 secs 2023-03-10T00:55:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:41 smithi181 ceph-mon[94189]: pgmap v19631: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 8 op/s 2023-03-10T00:55:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35851.496581 secs 2023-03-10T00:55:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:41 smithi038 ceph-mon[92422]: pgmap v19631: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 8 op/s 2023-03-10T00:55:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:43 smithi122 ceph-mon[93869]: pgmap v19632: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 10 op/s 2023-03-10T00:55:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35853.956404 secs 2023-03-10T00:55:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:43 smithi181 ceph-mon[94189]: pgmap v19632: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 10 op/s 2023-03-10T00:55:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35853.956404 secs 2023-03-10T00:55:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:43 smithi038 ceph-mon[92422]: pgmap v19632: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 10 op/s 2023-03-10T00:55:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35853.956404 secs 2023-03-10T00:55:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36315.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:55:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:45 smithi122 ceph-mon[93869]: pgmap v19633: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 8 op/s 2023-03-10T00:55:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:45 smithi181 ceph-mon[94189]: pgmap v19633: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 8 op/s 2023-03-10T00:55:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:45 smithi038 ceph-mon[92422]: pgmap v19633: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 8 op/s 2023-03-10T00:55:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35856.496642 secs 2023-03-10T00:55:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:55:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:55:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:55:46.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:55:46.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:55:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35856.496642 secs 2023-03-10T00:55:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:55:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:55:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:55:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:55:46.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:55:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35856.496642 secs 2023-03-10T00:55:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:55:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:55:46.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:55:46.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:55:46.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:55:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:47 smithi122 ceph-mon[93869]: pgmap v19634: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 426 B/s wr, 10 op/s 2023-03-10T00:55:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:55:47.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:55:47.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:55:47.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:47 smithi181 ceph-mon[94189]: pgmap v19634: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 426 B/s wr, 10 op/s 2023-03-10T00:55:47.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:55:47.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:55:47.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:55:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:47 smithi038 ceph-mon[92422]: pgmap v19634: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 426 B/s wr, 10 op/s 2023-03-10T00:55:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:55:47.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:55:47.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:55:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35858.956465 secs 2023-03-10T00:55:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35858.956465 secs 2023-03-10T00:55:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35858.956465 secs 2023-03-10T00:55:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36320.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:55:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:55:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:55:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:49 smithi122 ceph-mon[93869]: pgmap v19635: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 149 KiB/s wr, 8 op/s 2023-03-10T00:55:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:55:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:55:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:49 smithi181 ceph-mon[94189]: pgmap v19635: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 149 KiB/s wr, 8 op/s 2023-03-10T00:55:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:55:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:55:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:49 smithi038 ceph-mon[92422]: pgmap v19635: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 149 KiB/s wr, 8 op/s 2023-03-10T00:55:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35861.496698 secs 2023-03-10T00:55:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:51 smithi122 ceph-mon[93869]: pgmap v19636: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 391 KiB/s wr, 8 op/s 2023-03-10T00:55:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35861.496698 secs 2023-03-10T00:55:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:51 smithi181 ceph-mon[94189]: pgmap v19636: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 391 KiB/s wr, 8 op/s 2023-03-10T00:55:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35861.496698 secs 2023-03-10T00:55:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:51 smithi038 ceph-mon[92422]: pgmap v19636: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 391 KiB/s wr, 8 op/s 2023-03-10T00:55:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:55:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:55:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:55:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:55:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:55:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:55:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:53 smithi122 ceph-mon[93869]: pgmap v19637: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 420 KiB/s wr, 10 op/s 2023-03-10T00:55:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35863.956570 secs 2023-03-10T00:55:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:53 smithi181 ceph-mon[94189]: pgmap v19637: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 420 KiB/s wr, 10 op/s 2023-03-10T00:55:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35863.956570 secs 2023-03-10T00:55:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:53 smithi038 ceph-mon[92422]: pgmap v19637: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 420 KiB/s wr, 10 op/s 2023-03-10T00:55:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35863.956570 secs 2023-03-10T00:55:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36325.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:55:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:55 smithi122 ceph-mon[93869]: pgmap v19638: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 652 KiB/s rd, 602 KiB/s wr, 9 op/s 2023-03-10T00:55:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:55 smithi181 ceph-mon[94189]: pgmap v19638: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 652 KiB/s rd, 602 KiB/s wr, 9 op/s 2023-03-10T00:55:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:55 smithi038 ceph-mon[92422]: pgmap v19638: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 652 KiB/s rd, 602 KiB/s wr, 9 op/s 2023-03-10T00:55:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:55:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:55:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35866.496758 secs 2023-03-10T00:55:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:55:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:55:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35866.496758 secs 2023-03-10T00:55:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:55:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:55:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35866.496758 secs 2023-03-10T00:55:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:55:57.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:55:57.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:57 smithi122 ceph-mon[93869]: pgmap v19639: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 883 KiB/s rd, 814 KiB/s wr, 30 op/s 2023-03-10T00:55:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:55:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:55:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:57 smithi181 ceph-mon[94189]: pgmap v19639: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 883 KiB/s rd, 814 KiB/s wr, 30 op/s 2023-03-10T00:55:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:55:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:55:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:57 smithi038 ceph-mon[92422]: pgmap v19639: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 883 KiB/s rd, 814 KiB/s wr, 30 op/s 2023-03-10T00:55:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35868.956628 secs 2023-03-10T00:55:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35868.956628 secs 2023-03-10T00:55:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35868.956628 secs 2023-03-10T00:55:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36330.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:55:59.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:55:59 smithi122 ceph-mon[93869]: pgmap v19640: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 814 KiB/s wr, 28 op/s 2023-03-10T00:55:59.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:55:59 smithi181 ceph-mon[94189]: pgmap v19640: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 814 KiB/s wr, 28 op/s 2023-03-10T00:55:59.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:55:59 smithi038 ceph-mon[92422]: pgmap v19640: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 814 KiB/s wr, 28 op/s 2023-03-10T00:56:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:56:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:56:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:56:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:56:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:56:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:56:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:56:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:56:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35871.496820 secs 2023-03-10T00:56:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:01 smithi122 ceph-mon[93869]: pgmap v19641: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 737 KiB/s rd, 666 KiB/s wr, 28 op/s 2023-03-10T00:56:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:56:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:56:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35871.496820 secs 2023-03-10T00:56:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:01 smithi181 ceph-mon[94189]: pgmap v19641: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 737 KiB/s rd, 666 KiB/s wr, 28 op/s 2023-03-10T00:56:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:56:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:56:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35871.496820 secs 2023-03-10T00:56:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:01 smithi038 ceph-mon[92422]: pgmap v19641: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 737 KiB/s rd, 666 KiB/s wr, 28 op/s 2023-03-10T00:56:03.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:03 smithi122 ceph-mon[93869]: pgmap v19642: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 981 KiB/s rd, 910 KiB/s wr, 32 op/s 2023-03-10T00:56:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:03 smithi181 ceph-mon[94189]: pgmap v19642: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 981 KiB/s rd, 910 KiB/s wr, 32 op/s 2023-03-10T00:56:03.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:03 smithi038 ceph-mon[92422]: pgmap v19642: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 981 KiB/s rd, 910 KiB/s wr, 32 op/s 2023-03-10T00:56:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36335.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:56:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:05 smithi122 ceph-mon[93869]: pgmap v19643: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 958 KiB/s rd, 892 KiB/s wr, 41 op/s 2023-03-10T00:56:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:05 smithi181 ceph-mon[94189]: pgmap v19643: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 958 KiB/s rd, 892 KiB/s wr, 41 op/s 2023-03-10T00:56:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:05 smithi038 ceph-mon[92422]: pgmap v19643: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 958 KiB/s rd, 892 KiB/s wr, 41 op/s 2023-03-10T00:56:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35876.496881 secs 2023-03-10T00:56:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:06 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:56:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35876.496881 secs 2023-03-10T00:56:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:06 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:56:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35876.496881 secs 2023-03-10T00:56:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:06 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:56:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:07 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:56:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:56:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:56:07.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:07 smithi122 ceph-mon[93869]: pgmap v19644: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 932 KiB/s rd, 938 KiB/s wr, 45 op/s 2023-03-10T00:56:07.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:07 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:56:07.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:56:07.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:56:07.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:07 smithi181 ceph-mon[94189]: pgmap v19644: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 932 KiB/s rd, 938 KiB/s wr, 45 op/s 2023-03-10T00:56:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:07 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:56:07.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:56:07.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:56:07.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:07 smithi038 ceph-mon[92422]: pgmap v19644: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 932 KiB/s rd, 938 KiB/s wr, 45 op/s 2023-03-10T00:56:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:56:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:56:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35878.956797 secs 2023-03-10T00:56:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:56:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:56:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35878.956797 secs 2023-03-10T00:56:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:56:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:56:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35878.956797 secs 2023-03-10T00:56:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36340.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:56:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:56:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:56:09.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:09 smithi122 ceph-mon[93869]: pgmap v19645: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 27 op/s 2023-03-10T00:56:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:56:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:56:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:09 smithi181 ceph-mon[94189]: pgmap v19645: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 27 op/s 2023-03-10T00:56:09.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:56:09.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:56:09.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:09 smithi038 ceph-mon[92422]: pgmap v19645: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 27 op/s 2023-03-10T00:56:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35881.496941 secs 2023-03-10T00:56:11.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:11 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:56:11.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:11 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:56:11.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:11 smithi122 ceph-mon[93869]: pgmap v19646: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 25 op/s 2023-03-10T00:56:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35881.496941 secs 2023-03-10T00:56:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:11 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:56:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:11 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:56:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:11 smithi181 ceph-mon[94189]: pgmap v19646: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 25 op/s 2023-03-10T00:56:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35881.496941 secs 2023-03-10T00:56:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:11 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:56:11.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:11 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:56:11.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:11 smithi038 ceph-mon[92422]: pgmap v19646: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 25 op/s 2023-03-10T00:56:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:56:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:56:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:56:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:56:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:56:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:56:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:56:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:56:13.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:13 smithi122 ceph-mon[93869]: pgmap v19647: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 29 op/s 2023-03-10T00:56:13.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35883.956858 secs 2023-03-10T00:56:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:56:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:56:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:13 smithi181 ceph-mon[94189]: pgmap v19647: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 29 op/s 2023-03-10T00:56:13.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35883.956858 secs 2023-03-10T00:56:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:56:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:56:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:13 smithi038 ceph-mon[92422]: pgmap v19647: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 29 op/s 2023-03-10T00:56:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35883.956858 secs 2023-03-10T00:56:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36345.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:56:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:14 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:56:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:14 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:56:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:14 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:56:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:15 smithi122 ceph-mon[93869]: pgmap v19648: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 732 KiB/s wr, 24 op/s 2023-03-10T00:56:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:15 smithi181 ceph-mon[94189]: pgmap v19648: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 732 KiB/s wr, 24 op/s 2023-03-10T00:56:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:15 smithi038 ceph-mon[92422]: pgmap v19648: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 732 KiB/s wr, 24 op/s 2023-03-10T00:56:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35886.496998 secs 2023-03-10T00:56:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:16 smithi122 ceph-mon[93869]: osdmap e1443: 12 total, 12 up, 12 in 2023-03-10T00:56:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35886.496998 secs 2023-03-10T00:56:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:16 smithi181 ceph-mon[94189]: osdmap e1443: 12 total, 12 up, 12 in 2023-03-10T00:56:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35886.496998 secs 2023-03-10T00:56:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:16 smithi038 ceph-mon[92422]: osdmap e1443: 12 total, 12 up, 12 in 2023-03-10T00:56:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:17 smithi122 ceph-mon[93869]: pgmap v19650: 129 pgs: 9 active+clean+snaptrim_wait, 120 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 793 KiB/s rd, 803 KiB/s wr, 12 op/s 2023-03-10T00:56:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:17 smithi181 ceph-mon[94189]: pgmap v19650: 129 pgs: 9 active+clean+snaptrim_wait, 120 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 793 KiB/s rd, 803 KiB/s wr, 12 op/s 2023-03-10T00:56:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:17 smithi038 ceph-mon[92422]: pgmap v19650: 129 pgs: 9 active+clean+snaptrim_wait, 120 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 793 KiB/s rd, 803 KiB/s wr, 12 op/s 2023-03-10T00:56:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35888.956915 secs 2023-03-10T00:56:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35888.956915 secs 2023-03-10T00:56:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35888.956915 secs 2023-03-10T00:56:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36350.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:56:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:19 smithi122 ceph-mon[93869]: pgmap v19651: 129 pgs: 9 active+clean+snaptrim_wait, 120 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 380 KiB/s wr, 10 op/s 2023-03-10T00:56:19.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:19 smithi181 ceph-mon[94189]: pgmap v19651: 129 pgs: 9 active+clean+snaptrim_wait, 120 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 380 KiB/s wr, 10 op/s 2023-03-10T00:56:19.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:19 smithi038 ceph-mon[92422]: pgmap v19651: 129 pgs: 9 active+clean+snaptrim_wait, 120 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 380 KiB/s wr, 10 op/s 2023-03-10T00:56:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35891.497058 secs 2023-03-10T00:56:21.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:21 smithi122 ceph-mon[93869]: pgmap v19652: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 380 KiB/s wr, 10 op/s 2023-03-10T00:56:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35891.497058 secs 2023-03-10T00:56:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:21 smithi181 ceph-mon[94189]: pgmap v19652: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 380 KiB/s wr, 10 op/s 2023-03-10T00:56:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35891.497058 secs 2023-03-10T00:56:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:21 smithi038 ceph-mon[92422]: pgmap v19652: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 380 KiB/s wr, 10 op/s 2023-03-10T00:56:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:23 smithi122 ceph-mon[93869]: pgmap v19653: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 242 KiB/s wr, 9 op/s 2023-03-10T00:56:23.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35893.956975 secs 2023-03-10T00:56:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:23 smithi181 ceph-mon[94189]: pgmap v19653: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 242 KiB/s wr, 9 op/s 2023-03-10T00:56:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35893.956975 secs 2023-03-10T00:56:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:23 smithi038 ceph-mon[92422]: pgmap v19653: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 242 KiB/s wr, 9 op/s 2023-03-10T00:56:23.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35893.956975 secs 2023-03-10T00:56:23.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36355.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:56:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:25 smithi122 ceph-mon[93869]: osdmap e1444: 12 total, 12 up, 12 in 2023-03-10T00:56:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:25 smithi122 ceph-mon[93869]: pgmap v19655: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 378 KiB/s wr, 10 op/s 2023-03-10T00:56:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:25 smithi181 ceph-mon[94189]: osdmap e1444: 12 total, 12 up, 12 in 2023-03-10T00:56:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:25 smithi181 ceph-mon[94189]: pgmap v19655: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 378 KiB/s wr, 10 op/s 2023-03-10T00:56:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:25 smithi038 ceph-mon[92422]: osdmap e1444: 12 total, 12 up, 12 in 2023-03-10T00:56:25.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:25 smithi038 ceph-mon[92422]: pgmap v19655: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 378 KiB/s wr, 10 op/s 2023-03-10T00:56:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35896.497120 secs 2023-03-10T00:56:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35896.497120 secs 2023-03-10T00:56:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35896.497120 secs 2023-03-10T00:56:27.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:27 smithi122 ceph-mon[93869]: pgmap v19656: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 138 KiB/s wr, 8 op/s 2023-03-10T00:56:27.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:27 smithi181 ceph-mon[94189]: pgmap v19656: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 138 KiB/s wr, 8 op/s 2023-03-10T00:56:27.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:27 smithi038 ceph-mon[92422]: pgmap v19656: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 138 KiB/s wr, 8 op/s 2023-03-10T00:56:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35898.957058 secs 2023-03-10T00:56:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35898.957058 secs 2023-03-10T00:56:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35898.957058 secs 2023-03-10T00:56:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36360.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:56:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:29 smithi122 ceph-mon[93869]: pgmap v19657: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 138 KiB/s wr, 8 op/s 2023-03-10T00:56:29.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:29 smithi181 ceph-mon[94189]: pgmap v19657: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 138 KiB/s wr, 8 op/s 2023-03-10T00:56:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:29 smithi038 ceph-mon[92422]: pgmap v19657: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 138 KiB/s wr, 8 op/s 2023-03-10T00:56:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35901.497178 secs 2023-03-10T00:56:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:31 smithi122 ceph-mon[93869]: pgmap v19658: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 138 KiB/s wr, 8 op/s 2023-03-10T00:56:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35901.497178 secs 2023-03-10T00:56:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:31 smithi181 ceph-mon[94189]: pgmap v19658: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 138 KiB/s wr, 8 op/s 2023-03-10T00:56:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35901.497178 secs 2023-03-10T00:56:31.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:31 smithi038 ceph-mon[92422]: pgmap v19658: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 138 KiB/s wr, 8 op/s 2023-03-10T00:56:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:33 smithi122 ceph-mon[93869]: pgmap v19659: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 109 KiB/s wr, 7 op/s 2023-03-10T00:56:33.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35903.957118 secs 2023-03-10T00:56:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:33 smithi181 ceph-mon[94189]: pgmap v19659: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 109 KiB/s wr, 7 op/s 2023-03-10T00:56:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35903.957118 secs 2023-03-10T00:56:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:33 smithi038 ceph-mon[92422]: pgmap v19659: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 109 KiB/s wr, 7 op/s 2023-03-10T00:56:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35903.957118 secs 2023-03-10T00:56:33.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36365.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:56:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:35 smithi122 ceph-mon[93869]: pgmap v19660: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2023-03-10T00:56:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:35 smithi181 ceph-mon[94189]: pgmap v19660: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2023-03-10T00:56:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:35 smithi038 ceph-mon[92422]: pgmap v19660: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2023-03-10T00:56:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35906.497236 secs 2023-03-10T00:56:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35906.497236 secs 2023-03-10T00:56:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35906.497236 secs 2023-03-10T00:56:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:37 smithi122 ceph-mon[93869]: pgmap v19661: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-10T00:56:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:37 smithi181 ceph-mon[94189]: pgmap v19661: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-10T00:56:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:37 smithi038 ceph-mon[92422]: pgmap v19661: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-10T00:56:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35908.957177 secs 2023-03-10T00:56:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35908.957177 secs 2023-03-10T00:56:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35908.957177 secs 2023-03-10T00:56:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36370.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:56:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:39 smithi122 ceph-mon[93869]: pgmap v19662: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-10T00:56:39.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:39 smithi181 ceph-mon[94189]: pgmap v19662: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-10T00:56:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:39 smithi038 ceph-mon[92422]: pgmap v19662: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-10T00:56:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35911.497321 secs 2023-03-10T00:56:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:41 smithi122 ceph-mon[93869]: pgmap v19663: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-10T00:56:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35911.497321 secs 2023-03-10T00:56:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:41 smithi181 ceph-mon[94189]: pgmap v19663: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-10T00:56:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35911.497321 secs 2023-03-10T00:56:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:41 smithi038 ceph-mon[92422]: pgmap v19663: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-10T00:56:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:43 smithi122 ceph-mon[93869]: pgmap v19664: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 8 op/s 2023-03-10T00:56:43.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35913.957241 secs 2023-03-10T00:56:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:43 smithi181 ceph-mon[94189]: pgmap v19664: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 8 op/s 2023-03-10T00:56:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35913.957241 secs 2023-03-10T00:56:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:43 smithi038 ceph-mon[92422]: pgmap v19664: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 8 op/s 2023-03-10T00:56:43.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35913.957241 secs 2023-03-10T00:56:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36375.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:56:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:45 smithi122 ceph-mon[93869]: pgmap v19665: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-10T00:56:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:45 smithi181 ceph-mon[94189]: pgmap v19665: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-10T00:56:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:45 smithi038 ceph-mon[92422]: pgmap v19665: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-10T00:56:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35916.497380 secs 2023-03-10T00:56:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35916.497380 secs 2023-03-10T00:56:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35916.497380 secs 2023-03-10T00:56:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:47 smithi122 ceph-mon[93869]: pgmap v19666: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 426 B/s wr, 8 op/s 2023-03-10T00:56:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:56:47.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:47 smithi181 ceph-mon[94189]: pgmap v19666: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 426 B/s wr, 8 op/s 2023-03-10T00:56:47.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:56:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:47 smithi038 ceph-mon[92422]: pgmap v19666: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 426 B/s wr, 8 op/s 2023-03-10T00:56:47.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:56:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:56:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:56:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:56:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:56:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:56:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:56:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:56:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:56:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:56:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36380.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:56:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:49 smithi122 ceph-mon[93869]: pgmap v19667: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 569 KiB/s rd, 251 KiB/s wr, 7 op/s 2023-03-10T00:56:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:56:49.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:56:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:49 smithi181 ceph-mon[94189]: pgmap v19667: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 569 KiB/s rd, 251 KiB/s wr, 7 op/s 2023-03-10T00:56:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:56:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:56:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:49 smithi038 ceph-mon[92422]: pgmap v19667: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 569 KiB/s rd, 251 KiB/s wr, 7 op/s 2023-03-10T00:56:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:56:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:56:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:50 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:56:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:50 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:56:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:50 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:56:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35921.497467 secs 2023-03-10T00:56:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:51 smithi122 ceph-mon[93869]: pgmap v19668: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 570 KiB/s rd, 251 KiB/s wr, 7 op/s 2023-03-10T00:56:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:51 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:56:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:56:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:56:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35921.497467 secs 2023-03-10T00:56:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:51 smithi181 ceph-mon[94189]: pgmap v19668: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 570 KiB/s rd, 251 KiB/s wr, 7 op/s 2023-03-10T00:56:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:51 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:56:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:56:51.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:56:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35921.497467 secs 2023-03-10T00:56:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:51 smithi038 ceph-mon[92422]: pgmap v19668: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 570 KiB/s rd, 251 KiB/s wr, 7 op/s 2023-03-10T00:56:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:51 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:56:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:56:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:56:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:56:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:56:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:56:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:56:52.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:56:52.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:56:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:53 smithi038 ceph-mon[92422]: pgmap v19669: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 973 KiB/s rd, 653 KiB/s wr, 11 op/s 2023-03-10T00:56:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:56:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:56:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:53 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 35923.957363 secs 2023-03-10T00:56:53.801 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36385.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:56:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:53 smithi122 ceph-mon[93869]: pgmap v19669: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 973 KiB/s rd, 653 KiB/s wr, 11 op/s 2023-03-10T00:56:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:56:53.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:56:53.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:53 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 35923.957363 secs 2023-03-10T00:56:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:53 smithi181 ceph-mon[94189]: pgmap v19669: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 973 KiB/s rd, 653 KiB/s wr, 11 op/s 2023-03-10T00:56:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:56:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:56:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:53 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 35923.957363 secs 2023-03-10T00:56:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:55 smithi038 ceph-mon[92422]: pgmap v19670: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 983 KiB/s rd, 653 KiB/s wr, 10 op/s 2023-03-10T00:56:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:56:55.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:56:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:55 smithi122 ceph-mon[93869]: pgmap v19670: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 983 KiB/s rd, 653 KiB/s wr, 10 op/s 2023-03-10T00:56:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:56:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:56:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:55 smithi181 ceph-mon[94189]: pgmap v19670: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 983 KiB/s rd, 653 KiB/s wr, 10 op/s 2023-03-10T00:56:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:56:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:56:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35926.497525 secs 2023-03-10T00:56:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35926.497525 secs 2023-03-10T00:56:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35926.497525 secs 2023-03-10T00:56:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:57 smithi038 ceph-mon[92422]: pgmap v19671: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 654 KiB/s wr, 12 op/s 2023-03-10T00:56:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:56:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:56:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:57 smithi122 ceph-mon[93869]: pgmap v19671: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 654 KiB/s wr, 12 op/s 2023-03-10T00:56:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:56:57.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:56:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:57 smithi181 ceph-mon[94189]: pgmap v19671: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 654 KiB/s wr, 12 op/s 2023-03-10T00:56:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:56:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:56:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36390.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:56:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:58 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 35928.957478 secs 2023-03-10T00:56:58.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:58 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 35928.957478 secs 2023-03-10T00:56:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:58 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 35928.957478 secs 2023-03-10T00:56:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:59 smithi122 ceph-mon[93869]: pgmap v19672: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 995 KiB/s wr, 12 op/s 2023-03-10T00:56:59.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:56:59 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:56:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:59 smithi181 ceph-mon[94189]: pgmap v19672: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 995 KiB/s wr, 12 op/s 2023-03-10T00:56:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:56:59 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:56:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:59 smithi038 ceph-mon[92422]: pgmap v19672: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 995 KiB/s wr, 12 op/s 2023-03-10T00:56:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:56:59 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:57:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:57:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:57:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:57:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:57:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:57:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:57:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35931.497614 secs 2023-03-10T00:57:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:01 smithi122 ceph-mon[93869]: pgmap v19673: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 994 KiB/s wr, 11 op/s 2023-03-10T00:57:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:57:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:57:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35931.497614 secs 2023-03-10T00:57:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:01 smithi181 ceph-mon[94189]: pgmap v19673: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 994 KiB/s wr, 11 op/s 2023-03-10T00:57:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:57:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:57:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35931.497614 secs 2023-03-10T00:57:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:01 smithi038 ceph-mon[92422]: pgmap v19673: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 994 KiB/s wr, 11 op/s 2023-03-10T00:57:02.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:57:02.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:57:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36395.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:57:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:03 smithi122 ceph-mon[93869]: pgmap v19674: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 15 op/s 2023-03-10T00:57:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35933.957538 secs 2023-03-10T00:57:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:03 smithi181 ceph-mon[94189]: pgmap v19674: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 15 op/s 2023-03-10T00:57:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35933.957538 secs 2023-03-10T00:57:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:03 smithi038 ceph-mon[92422]: pgmap v19674: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 15 op/s 2023-03-10T00:57:04.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35933.957538 secs 2023-03-10T00:57:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:05 smithi122 ceph-mon[93869]: pgmap v19675: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 721 KiB/s rd, 663 KiB/s wr, 14 op/s 2023-03-10T00:57:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:05 smithi181 ceph-mon[94189]: pgmap v19675: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 721 KiB/s rd, 663 KiB/s wr, 14 op/s 2023-03-10T00:57:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:05 smithi038 ceph-mon[92422]: pgmap v19675: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 721 KiB/s rd, 663 KiB/s wr, 14 op/s 2023-03-10T00:57:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35936.497673 secs 2023-03-10T00:57:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35936.497673 secs 2023-03-10T00:57:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35936.497673 secs 2023-03-10T00:57:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:57:07.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:57:07.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:07 smithi122 ceph-mon[93869]: pgmap v19676: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1002 KiB/s wr, 19 op/s 2023-03-10T00:57:07.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:57:07.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:57:07.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:07 smithi181 ceph-mon[94189]: pgmap v19676: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1002 KiB/s wr, 19 op/s 2023-03-10T00:57:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:57:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:57:07.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:07 smithi038 ceph-mon[92422]: pgmap v19676: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1002 KiB/s wr, 19 op/s 2023-03-10T00:57:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:57:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:57:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35938.957595 secs 2023-03-10T00:57:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:57:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:57:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35938.957595 secs 2023-03-10T00:57:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:57:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:57:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35938.957595 secs 2023-03-10T00:57:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36400.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:57:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:57:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:57:09.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:09 smithi122 ceph-mon[93869]: pgmap v19677: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 977 KiB/s rd, 1012 KiB/s wr, 26 op/s 2023-03-10T00:57:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:57:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:57:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:09 smithi181 ceph-mon[94189]: pgmap v19677: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 977 KiB/s rd, 1012 KiB/s wr, 26 op/s 2023-03-10T00:57:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:57:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:57:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:09 smithi038 ceph-mon[92422]: pgmap v19677: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 977 KiB/s rd, 1012 KiB/s wr, 26 op/s 2023-03-10T00:57:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35941.497735 secs 2023-03-10T00:57:11.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:11 smithi122 ceph-mon[93869]: osdmap e1445: 12 total, 12 up, 12 in 2023-03-10T00:57:11.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:11 smithi122 ceph-mon[93869]: pgmap v19679: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 464 KiB/s rd, 505 KiB/s wr, 28 op/s 2023-03-10T00:57:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35941.497735 secs 2023-03-10T00:57:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:11 smithi181 ceph-mon[94189]: osdmap e1445: 12 total, 12 up, 12 in 2023-03-10T00:57:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:11 smithi181 ceph-mon[94189]: pgmap v19679: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 464 KiB/s rd, 505 KiB/s wr, 28 op/s 2023-03-10T00:57:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35941.497735 secs 2023-03-10T00:57:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:11 smithi038 ceph-mon[92422]: osdmap e1445: 12 total, 12 up, 12 in 2023-03-10T00:57:11.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:11 smithi038 ceph-mon[92422]: pgmap v19679: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 464 KiB/s rd, 505 KiB/s wr, 28 op/s 2023-03-10T00:57:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:13 smithi122 ceph-mon[93869]: pgmap v19680: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 802 KiB/s rd, 843 KiB/s wr, 29 op/s 2023-03-10T00:57:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35943.957654 secs 2023-03-10T00:57:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:13 smithi181 ceph-mon[94189]: pgmap v19680: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 802 KiB/s rd, 843 KiB/s wr, 29 op/s 2023-03-10T00:57:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35943.957654 secs 2023-03-10T00:57:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:13 smithi038 ceph-mon[92422]: pgmap v19680: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 802 KiB/s rd, 843 KiB/s wr, 29 op/s 2023-03-10T00:57:13.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35943.957654 secs 2023-03-10T00:57:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36405.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:57:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:15 smithi122 ceph-mon[93869]: pgmap v19681: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 800 KiB/s rd, 830 KiB/s wr, 25 op/s 2023-03-10T00:57:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:15 smithi181 ceph-mon[94189]: pgmap v19681: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 800 KiB/s rd, 830 KiB/s wr, 25 op/s 2023-03-10T00:57:15.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:15 smithi038 ceph-mon[92422]: pgmap v19681: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 800 KiB/s rd, 830 KiB/s wr, 25 op/s 2023-03-10T00:57:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:57:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:57:16.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35946.497797 secs 2023-03-10T00:57:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:57:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:57:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35946.497797 secs 2023-03-10T00:57:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:57:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:57:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35946.497797 secs 2023-03-10T00:57:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:57:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:57:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:17 smithi122 ceph-mon[93869]: pgmap v19682: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 423 KiB/s wr, 21 op/s 2023-03-10T00:57:17.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:57:17.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:57:17.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:17 smithi181 ceph-mon[94189]: pgmap v19682: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 423 KiB/s wr, 21 op/s 2023-03-10T00:57:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:57:17.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:57:17.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:17 smithi038 ceph-mon[92422]: pgmap v19682: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 423 KiB/s wr, 21 op/s 2023-03-10T00:57:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35948.957712 secs 2023-03-10T00:57:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35948.957712 secs 2023-03-10T00:57:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35948.957712 secs 2023-03-10T00:57:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36410.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:57:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:19 smithi122 ceph-mon[93869]: pgmap v19683: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 410 KiB/s wr, 10 op/s 2023-03-10T00:57:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:19 smithi122 ceph-mon[93869]: osdmap e1446: 12 total, 12 up, 12 in 2023-03-10T00:57:19.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:19 smithi181 ceph-mon[94189]: pgmap v19683: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 410 KiB/s wr, 10 op/s 2023-03-10T00:57:19.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:19 smithi181 ceph-mon[94189]: osdmap e1446: 12 total, 12 up, 12 in 2023-03-10T00:57:19.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:19 smithi038 ceph-mon[92422]: pgmap v19683: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 410 KiB/s wr, 10 op/s 2023-03-10T00:57:19.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:19 smithi038 ceph-mon[92422]: osdmap e1446: 12 total, 12 up, 12 in 2023-03-10T00:57:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35951.497884 secs 2023-03-10T00:57:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:21 smithi122 ceph-mon[93869]: pgmap v19685: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 410 KiB/s wr, 10 op/s 2023-03-10T00:57:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35951.497884 secs 2023-03-10T00:57:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:21 smithi181 ceph-mon[94189]: pgmap v19685: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 410 KiB/s wr, 10 op/s 2023-03-10T00:57:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35951.497884 secs 2023-03-10T00:57:21.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:21 smithi038 ceph-mon[92422]: pgmap v19685: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 410 KiB/s wr, 10 op/s 2023-03-10T00:57:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:23 smithi122 ceph-mon[93869]: pgmap v19686: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 310 KiB/s rd, 307 KiB/s wr, 9 op/s 2023-03-10T00:57:23.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35953.957805 secs 2023-03-10T00:57:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:23 smithi181 ceph-mon[94189]: pgmap v19686: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 310 KiB/s rd, 307 KiB/s wr, 9 op/s 2023-03-10T00:57:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35953.957805 secs 2023-03-10T00:57:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:23 smithi038 ceph-mon[92422]: pgmap v19686: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 310 KiB/s rd, 307 KiB/s wr, 9 op/s 2023-03-10T00:57:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35953.957805 secs 2023-03-10T00:57:23.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36415.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:57:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:25 smithi122 ceph-mon[93869]: pgmap v19687: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 310 KiB/s rd, 307 KiB/s wr, 9 op/s 2023-03-10T00:57:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:25 smithi181 ceph-mon[94189]: pgmap v19687: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 310 KiB/s rd, 307 KiB/s wr, 9 op/s 2023-03-10T00:57:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:25 smithi038 ceph-mon[92422]: pgmap v19687: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 310 KiB/s rd, 307 KiB/s wr, 9 op/s 2023-03-10T00:57:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35956.497947 secs 2023-03-10T00:57:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35956.497947 secs 2023-03-10T00:57:26.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35956.497947 secs 2023-03-10T00:57:27.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:27 smithi122 ceph-mon[93869]: pgmap v19688: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 310 KiB/s rd, 307 KiB/s wr, 8 op/s 2023-03-10T00:57:27.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:27 smithi181 ceph-mon[94189]: pgmap v19688: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 310 KiB/s rd, 307 KiB/s wr, 8 op/s 2023-03-10T00:57:27.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:27 smithi038 ceph-mon[92422]: pgmap v19688: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 310 KiB/s rd, 307 KiB/s wr, 8 op/s 2023-03-10T00:57:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35958.957893 secs 2023-03-10T00:57:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35958.957893 secs 2023-03-10T00:57:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35958.957893 secs 2023-03-10T00:57:28.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36420.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:57:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:29 smithi122 ceph-mon[93869]: pgmap v19689: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 307 KiB/s wr, 7 op/s 2023-03-10T00:57:29.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:29 smithi181 ceph-mon[94189]: pgmap v19689: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 307 KiB/s wr, 7 op/s 2023-03-10T00:57:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:29 smithi038 ceph-mon[92422]: pgmap v19689: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 307 KiB/s wr, 7 op/s 2023-03-10T00:57:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35961.498010 secs 2023-03-10T00:57:31.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:31 smithi122 ceph-mon[93869]: pgmap v19690: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 270 KiB/s wr, 6 op/s 2023-03-10T00:57:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35961.498010 secs 2023-03-10T00:57:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:31 smithi181 ceph-mon[94189]: pgmap v19690: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 270 KiB/s wr, 6 op/s 2023-03-10T00:57:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35961.498010 secs 2023-03-10T00:57:31.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:31 smithi038 ceph-mon[92422]: pgmap v19690: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 270 KiB/s wr, 6 op/s 2023-03-10T00:57:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:33 smithi181 ceph-mon[94189]: pgmap v19691: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 256 KiB/s wr, 9 op/s 2023-03-10T00:57:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35963.957985 secs 2023-03-10T00:57:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:33 smithi038 ceph-mon[92422]: pgmap v19691: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 256 KiB/s wr, 9 op/s 2023-03-10T00:57:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35963.957985 secs 2023-03-10T00:57:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36425.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:57:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:33 smithi122 ceph-mon[93869]: pgmap v19691: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 256 KiB/s wr, 9 op/s 2023-03-10T00:57:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35963.957985 secs 2023-03-10T00:57:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:35 smithi122 ceph-mon[93869]: pgmap v19692: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-10T00:57:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:35 smithi181 ceph-mon[94189]: pgmap v19692: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-10T00:57:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:35 smithi038 ceph-mon[92422]: pgmap v19692: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-10T00:57:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35966.498076 secs 2023-03-10T00:57:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35966.498076 secs 2023-03-10T00:57:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35966.498076 secs 2023-03-10T00:57:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:37 smithi122 ceph-mon[93869]: pgmap v19693: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 8 op/s 2023-03-10T00:57:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:37 smithi181 ceph-mon[94189]: pgmap v19693: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 8 op/s 2023-03-10T00:57:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:37 smithi038 ceph-mon[92422]: pgmap v19693: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 8 op/s 2023-03-10T00:57:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35968.958065 secs 2023-03-10T00:57:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35968.958065 secs 2023-03-10T00:57:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35968.958065 secs 2023-03-10T00:57:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36430.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:57:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:39 smithi122 ceph-mon[93869]: pgmap v19694: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-10T00:57:39.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:39 smithi181 ceph-mon[94189]: pgmap v19694: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-10T00:57:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:39 smithi038 ceph-mon[92422]: pgmap v19694: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-10T00:57:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35971.498133 secs 2023-03-10T00:57:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:41 smithi122 ceph-mon[93869]: pgmap v19695: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-10T00:57:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35971.498133 secs 2023-03-10T00:57:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:41 smithi181 ceph-mon[94189]: pgmap v19695: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-10T00:57:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35971.498133 secs 2023-03-10T00:57:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:41 smithi038 ceph-mon[92422]: pgmap v19695: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-10T00:57:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:43 smithi122 ceph-mon[93869]: pgmap v19696: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 9 op/s 2023-03-10T00:57:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35973.958212 secs 2023-03-10T00:57:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:43 smithi181 ceph-mon[94189]: pgmap v19696: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 9 op/s 2023-03-10T00:57:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35973.958212 secs 2023-03-10T00:57:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:43 smithi038 ceph-mon[92422]: pgmap v19696: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 85 B/s wr, 9 op/s 2023-03-10T00:57:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35973.958212 secs 2023-03-10T00:57:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36435.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:57:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:45 smithi122 ceph-mon[93869]: pgmap v19697: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-10T00:57:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:45 smithi181 ceph-mon[94189]: pgmap v19697: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-10T00:57:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:45 smithi038 ceph-mon[92422]: pgmap v19697: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2023-03-10T00:57:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35976.498194 secs 2023-03-10T00:57:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35976.498194 secs 2023-03-10T00:57:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35976.498194 secs 2023-03-10T00:57:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:47 smithi122 ceph-mon[93869]: pgmap v19698: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 234 KiB/s wr, 8 op/s 2023-03-10T00:57:47.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:47 smithi181 ceph-mon[94189]: pgmap v19698: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 234 KiB/s wr, 8 op/s 2023-03-10T00:57:47.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:47 smithi038 ceph-mon[92422]: pgmap v19698: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 234 KiB/s wr, 8 op/s 2023-03-10T00:57:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35978.958270 secs 2023-03-10T00:57:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:57:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35978.958270 secs 2023-03-10T00:57:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:57:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35978.958270 secs 2023-03-10T00:57:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:57:48.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36440.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:57:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:57:49.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:57:49.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:57:49.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:57:49.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:57:49.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:49 smithi122 ceph-mon[93869]: pgmap v19699: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 234 KiB/s wr, 7 op/s 2023-03-10T00:57:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:57:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:57:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:57:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:57:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:57:49.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:49 smithi181 ceph-mon[94189]: pgmap v19699: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 234 KiB/s wr, 7 op/s 2023-03-10T00:57:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:57:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:57:49.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:57:49.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:57:49.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:57:49.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:49 smithi038 ceph-mon[92422]: pgmap v19699: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 234 KiB/s wr, 7 op/s 2023-03-10T00:57:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35981.498251 secs 2023-03-10T00:57:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:51 smithi038 ceph-mon[92422]: pgmap v19700: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 578 KiB/s rd, 575 KiB/s wr, 7 op/s 2023-03-10T00:57:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35981.498251 secs 2023-03-10T00:57:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:51 smithi122 ceph-mon[93869]: pgmap v19700: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 578 KiB/s rd, 575 KiB/s wr, 7 op/s 2023-03-10T00:57:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35981.498251 secs 2023-03-10T00:57:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:51 smithi181 ceph-mon[94189]: pgmap v19700: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 578 KiB/s rd, 575 KiB/s wr, 7 op/s 2023-03-10T00:57:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:57:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:57:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:57:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:57:52.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:57:52.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:57:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:53 smithi038 ceph-mon[92422]: pgmap v19701: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 906 KiB/s rd, 645 KiB/s wr, 11 op/s 2023-03-10T00:57:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:57:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:57:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35983.958357 secs 2023-03-10T00:57:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36445.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:57:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:53 smithi122 ceph-mon[93869]: pgmap v19701: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 906 KiB/s rd, 645 KiB/s wr, 11 op/s 2023-03-10T00:57:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:57:53.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:57:53.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35983.958357 secs 2023-03-10T00:57:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:53 smithi181 ceph-mon[94189]: pgmap v19701: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 906 KiB/s rd, 645 KiB/s wr, 11 op/s 2023-03-10T00:57:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:57:53.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:57:53.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35983.958357 secs 2023-03-10T00:57:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:55 smithi122 ceph-mon[93869]: pgmap v19702: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 917 KiB/s rd, 646 KiB/s wr, 9 op/s 2023-03-10T00:57:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:55 smithi181 ceph-mon[94189]: pgmap v19702: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 917 KiB/s rd, 646 KiB/s wr, 9 op/s 2023-03-10T00:57:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:55 smithi038 ceph-mon[92422]: pgmap v19702: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 917 KiB/s rd, 646 KiB/s wr, 9 op/s 2023-03-10T00:57:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35986.498314 secs 2023-03-10T00:57:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:57:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:57:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35986.498314 secs 2023-03-10T00:57:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:57:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:57:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35986.498314 secs 2023-03-10T00:57:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:57:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:57:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:57:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:57:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:57 smithi122 ceph-mon[93869]: pgmap v19703: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 948 KiB/s rd, 677 KiB/s wr, 13 op/s 2023-03-10T00:57:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:57:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:57:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:57 smithi181 ceph-mon[94189]: pgmap v19703: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 948 KiB/s rd, 677 KiB/s wr, 13 op/s 2023-03-10T00:57:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:57:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:57:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:57 smithi038 ceph-mon[92422]: pgmap v19703: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 948 KiB/s rd, 677 KiB/s wr, 13 op/s 2023-03-10T00:57:58.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35988.958455 secs 2023-03-10T00:57:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35988.958455 secs 2023-03-10T00:57:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35988.958455 secs 2023-03-10T00:57:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36450.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:57:59.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:57:59 smithi122 ceph-mon[93869]: pgmap v19704: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 467 KiB/s wr, 10 op/s 2023-03-10T00:57:59.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:57:59 smithi181 ceph-mon[94189]: pgmap v19704: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 467 KiB/s wr, 10 op/s 2023-03-10T00:57:59.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:57:59 smithi038 ceph-mon[92422]: pgmap v19704: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 467 KiB/s wr, 10 op/s 2023-03-10T00:58:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35991.498396 secs 2023-03-10T00:58:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:58:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:58:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:01 smithi122 ceph-mon[93869]: pgmap v19705: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 733 KiB/s rd, 487 KiB/s wr, 11 op/s 2023-03-10T00:58:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35991.498396 secs 2023-03-10T00:58:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:58:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:58:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:01 smithi181 ceph-mon[94189]: pgmap v19705: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 733 KiB/s rd, 487 KiB/s wr, 11 op/s 2023-03-10T00:58:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35991.498396 secs 2023-03-10T00:58:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:58:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:58:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:01 smithi038 ceph-mon[92422]: pgmap v19705: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 733 KiB/s rd, 487 KiB/s wr, 11 op/s 2023-03-10T00:58:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:03 smithi038 ceph-mon[92422]: pgmap v19706: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1002 KiB/s rd, 756 KiB/s wr, 14 op/s 2023-03-10T00:58:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36455.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:58:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:03 smithi122 ceph-mon[93869]: pgmap v19706: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1002 KiB/s rd, 756 KiB/s wr, 14 op/s 2023-03-10T00:58:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:03 smithi181 ceph-mon[94189]: pgmap v19706: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1002 KiB/s rd, 756 KiB/s wr, 14 op/s 2023-03-10T00:58:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:05 smithi122 ceph-mon[93869]: pgmap v19707: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1022 KiB/s rd, 1014 KiB/s wr, 23 op/s 2023-03-10T00:58:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:05 smithi181 ceph-mon[94189]: pgmap v19707: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1022 KiB/s rd, 1014 KiB/s wr, 23 op/s 2023-03-10T00:58:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:05 smithi038 ceph-mon[92422]: pgmap v19707: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1022 KiB/s rd, 1014 KiB/s wr, 23 op/s 2023-03-10T00:58:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35996.498461 secs 2023-03-10T00:58:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:06 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:58:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35996.498461 secs 2023-03-10T00:58:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:06 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:58:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35996.498461 secs 2023-03-10T00:58:06.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:06 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:58:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:07 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:58:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:58:07.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:58:07.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:07 smithi122 ceph-mon[93869]: pgmap v19708: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 29 op/s 2023-03-10T00:58:07.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:07 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:58:07.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:58:07.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:58:07.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:07 smithi181 ceph-mon[94189]: pgmap v19708: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 29 op/s 2023-03-10T00:58:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:07 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:58:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:58:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:58:07.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:07 smithi038 ceph-mon[92422]: pgmap v19708: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 29 op/s 2023-03-10T00:58:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:58:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:58:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 35998.958621 secs 2023-03-10T00:58:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:58:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:58:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 35998.958621 secs 2023-03-10T00:58:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:58:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:58:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 35998.958621 secs 2023-03-10T00:58:08.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36460.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:58:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:58:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:58:09.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:09 smithi122 ceph-mon[93869]: pgmap v19709: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 28 op/s 2023-03-10T00:58:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:58:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:58:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:09 smithi181 ceph-mon[94189]: pgmap v19709: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 28 op/s 2023-03-10T00:58:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:58:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:58:09.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:09 smithi038 ceph-mon[92422]: pgmap v19709: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 28 op/s 2023-03-10T00:58:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36001.498523 secs 2023-03-10T00:58:11.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:11 smithi122 ceph-mon[93869]: osdmap e1447: 12 total, 12 up, 12 in 2023-03-10T00:58:11.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:11 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:58:11.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:11 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:58:11.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:11 smithi122 ceph-mon[93869]: pgmap v19711: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 33 op/s 2023-03-10T00:58:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36001.498523 secs 2023-03-10T00:58:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:11 smithi181 ceph-mon[94189]: osdmap e1447: 12 total, 12 up, 12 in 2023-03-10T00:58:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:11 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:58:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:11 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:58:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:11 smithi181 ceph-mon[94189]: pgmap v19711: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 33 op/s 2023-03-10T00:58:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36001.498523 secs 2023-03-10T00:58:11.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:11 smithi038 ceph-mon[92422]: osdmap e1447: 12 total, 12 up, 12 in 2023-03-10T00:58:11.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:11 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:58:11.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:11 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:58:11.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:11 smithi038 ceph-mon[92422]: pgmap v19711: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 33 op/s 2023-03-10T00:58:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:58:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:58:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:58:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:58:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:58:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:58:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:58:13.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:58:13.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:13 smithi122 ceph-mon[93869]: pgmap v19712: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 32 op/s 2023-03-10T00:58:13.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36003.958681 secs 2023-03-10T00:58:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:58:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:58:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:13 smithi181 ceph-mon[94189]: pgmap v19712: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 32 op/s 2023-03-10T00:58:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36003.958681 secs 2023-03-10T00:58:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:58:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:58:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:13 smithi038 ceph-mon[92422]: pgmap v19712: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 32 op/s 2023-03-10T00:58:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36003.958681 secs 2023-03-10T00:58:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36465.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:58:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:14 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:58:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:14 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:58:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:14 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:58:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:15 smithi122 ceph-mon[93869]: pgmap v19713: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 831 KiB/s rd, 873 KiB/s wr, 18 op/s 2023-03-10T00:58:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:15 smithi181 ceph-mon[94189]: pgmap v19713: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 831 KiB/s rd, 873 KiB/s wr, 18 op/s 2023-03-10T00:58:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:15 smithi038 ceph-mon[92422]: pgmap v19713: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 831 KiB/s rd, 873 KiB/s wr, 18 op/s 2023-03-10T00:58:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36006.498582 secs 2023-03-10T00:58:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36006.498582 secs 2023-03-10T00:58:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36006.498582 secs 2023-03-10T00:58:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:17 smithi122 ceph-mon[93869]: pgmap v19714: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 589 KiB/s rd, 601 KiB/s wr, 14 op/s 2023-03-10T00:58:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:17 smithi181 ceph-mon[94189]: pgmap v19714: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 589 KiB/s rd, 601 KiB/s wr, 14 op/s 2023-03-10T00:58:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:17 smithi038 ceph-mon[92422]: pgmap v19714: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 589 KiB/s rd, 601 KiB/s wr, 14 op/s 2023-03-10T00:58:18.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36008.958769 secs 2023-03-10T00:58:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36008.958769 secs 2023-03-10T00:58:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36008.958769 secs 2023-03-10T00:58:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36470.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:58:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:19 smithi122 ceph-mon[93869]: pgmap v19715: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 537 KiB/s rd, 536 KiB/s wr, 12 op/s 2023-03-10T00:58:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:19 smithi122 ceph-mon[93869]: osdmap e1448: 12 total, 12 up, 12 in 2023-03-10T00:58:19.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:19 smithi181 ceph-mon[94189]: pgmap v19715: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 537 KiB/s rd, 536 KiB/s wr, 12 op/s 2023-03-10T00:58:19.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:19 smithi181 ceph-mon[94189]: osdmap e1448: 12 total, 12 up, 12 in 2023-03-10T00:58:19.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:19 smithi038 ceph-mon[92422]: pgmap v19715: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 537 KiB/s rd, 536 KiB/s wr, 12 op/s 2023-03-10T00:58:19.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:19 smithi038 ceph-mon[92422]: osdmap e1448: 12 total, 12 up, 12 in 2023-03-10T00:58:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:58:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:58:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:58:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:58:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:58:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:58:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36011.498644 secs 2023-03-10T00:58:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:58:21.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:58:21.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:21 smithi122 ceph-mon[93869]: pgmap v19717: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 412 KiB/s wr, 11 op/s 2023-03-10T00:58:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36011.498644 secs 2023-03-10T00:58:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:58:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:58:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:21 smithi181 ceph-mon[94189]: pgmap v19717: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 412 KiB/s wr, 11 op/s 2023-03-10T00:58:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36011.498644 secs 2023-03-10T00:58:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:58:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:58:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:21 smithi038 ceph-mon[92422]: pgmap v19717: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 412 KiB/s wr, 11 op/s 2023-03-10T00:58:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:23 smithi038 ceph-mon[92422]: pgmap v19718: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 836 KiB/s rd, 477 KiB/s wr, 11 op/s 2023-03-10T00:58:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36013.958829 secs 2023-03-10T00:58:23.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36475.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:58:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:23 smithi122 ceph-mon[93869]: pgmap v19718: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 836 KiB/s rd, 477 KiB/s wr, 11 op/s 2023-03-10T00:58:23.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36013.958829 secs 2023-03-10T00:58:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:23 smithi181 ceph-mon[94189]: pgmap v19718: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 836 KiB/s rd, 477 KiB/s wr, 11 op/s 2023-03-10T00:58:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36013.958829 secs 2023-03-10T00:58:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:25 smithi122 ceph-mon[93869]: pgmap v19719: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 836 KiB/s rd, 477 KiB/s wr, 10 op/s 2023-03-10T00:58:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:25 smithi181 ceph-mon[94189]: pgmap v19719: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 836 KiB/s rd, 477 KiB/s wr, 10 op/s 2023-03-10T00:58:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:25 smithi038 ceph-mon[92422]: pgmap v19719: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 836 KiB/s rd, 477 KiB/s wr, 10 op/s 2023-03-10T00:58:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36016.498743 secs 2023-03-10T00:58:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36016.498743 secs 2023-03-10T00:58:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36016.498743 secs 2023-03-10T00:58:27.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:27 smithi122 ceph-mon[93869]: pgmap v19720: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 359 KiB/s wr, 10 op/s 2023-03-10T00:58:27.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:27 smithi181 ceph-mon[94189]: pgmap v19720: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 359 KiB/s wr, 10 op/s 2023-03-10T00:58:27.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:27 smithi038 ceph-mon[92422]: pgmap v19720: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 359 KiB/s wr, 10 op/s 2023-03-10T00:58:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36018.958934 secs 2023-03-10T00:58:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36018.958934 secs 2023-03-10T00:58:28.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36018.958934 secs 2023-03-10T00:58:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36480.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:58:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:29 smithi122 ceph-mon[93869]: pgmap v19721: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 359 KiB/s wr, 9 op/s 2023-03-10T00:58:29.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:29 smithi181 ceph-mon[94189]: pgmap v19721: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 359 KiB/s wr, 9 op/s 2023-03-10T00:58:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:29 smithi038 ceph-mon[92422]: pgmap v19721: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 359 KiB/s wr, 9 op/s 2023-03-10T00:58:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36021.498828 secs 2023-03-10T00:58:31.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:31 smithi038 ceph-mon[92422]: pgmap v19722: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 316 KiB/s wr, 8 op/s 2023-03-10T00:58:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36021.498828 secs 2023-03-10T00:58:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:31 smithi122 ceph-mon[93869]: pgmap v19722: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 316 KiB/s wr, 8 op/s 2023-03-10T00:58:31.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36021.498828 secs 2023-03-10T00:58:31.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:31 smithi181 ceph-mon[94189]: pgmap v19722: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 316 KiB/s wr, 8 op/s 2023-03-10T00:58:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:33 smithi038 ceph-mon[92422]: pgmap v19723: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 596 KiB/s rd, 299 KiB/s wr, 9 op/s 2023-03-10T00:58:33.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36023.958995 secs 2023-03-10T00:58:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36485.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:58:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:33 smithi122 ceph-mon[93869]: pgmap v19723: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 596 KiB/s rd, 299 KiB/s wr, 9 op/s 2023-03-10T00:58:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36023.958995 secs 2023-03-10T00:58:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:33 smithi181 ceph-mon[94189]: pgmap v19723: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 596 KiB/s rd, 299 KiB/s wr, 9 op/s 2023-03-10T00:58:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36023.958995 secs 2023-03-10T00:58:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:35 smithi122 ceph-mon[93869]: pgmap v19724: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 6 op/s 2023-03-10T00:58:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:35 smithi181 ceph-mon[94189]: pgmap v19724: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 6 op/s 2023-03-10T00:58:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:35 smithi038 ceph-mon[92422]: pgmap v19724: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 170 B/s wr, 6 op/s 2023-03-10T00:58:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36026.498887 secs 2023-03-10T00:58:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36026.498887 secs 2023-03-10T00:58:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36026.498887 secs 2023-03-10T00:58:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:37 smithi122 ceph-mon[93869]: pgmap v19725: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 170 B/s wr, 9 op/s 2023-03-10T00:58:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:37 smithi181 ceph-mon[94189]: pgmap v19725: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 170 B/s wr, 9 op/s 2023-03-10T00:58:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:37 smithi038 ceph-mon[92422]: pgmap v19725: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 170 B/s wr, 9 op/s 2023-03-10T00:58:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36028.959134 secs 2023-03-10T00:58:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36028.959134 secs 2023-03-10T00:58:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36028.959134 secs 2023-03-10T00:58:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36490.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:58:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:39 smithi122 ceph-mon[93869]: pgmap v19726: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-10T00:58:39.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:39 smithi181 ceph-mon[94189]: pgmap v19726: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-10T00:58:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:39 smithi038 ceph-mon[92422]: pgmap v19726: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 6 op/s 2023-03-10T00:58:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36031.498982 secs 2023-03-10T00:58:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:41 smithi038 ceph-mon[92422]: pgmap v19727: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 6 op/s 2023-03-10T00:58:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36031.498982 secs 2023-03-10T00:58:41.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:41 smithi122 ceph-mon[93869]: pgmap v19727: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 6 op/s 2023-03-10T00:58:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36031.498982 secs 2023-03-10T00:58:41.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:41 smithi181 ceph-mon[94189]: pgmap v19727: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 341 B/s wr, 6 op/s 2023-03-10T00:58:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:43 smithi038 ceph-mon[92422]: pgmap v19728: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 511 B/s wr, 9 op/s 2023-03-10T00:58:43.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36033.959220 secs 2023-03-10T00:58:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36495.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:58:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:43 smithi122 ceph-mon[93869]: pgmap v19728: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 511 B/s wr, 9 op/s 2023-03-10T00:58:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36033.959220 secs 2023-03-10T00:58:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:43 smithi181 ceph-mon[94189]: pgmap v19728: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 511 B/s wr, 9 op/s 2023-03-10T00:58:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36033.959220 secs 2023-03-10T00:58:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:45 smithi122 ceph-mon[93869]: pgmap v19729: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-10T00:58:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:45 smithi181 ceph-mon[94189]: pgmap v19729: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-10T00:58:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:45 smithi038 ceph-mon[92422]: pgmap v19729: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 511 B/s wr, 6 op/s 2023-03-10T00:58:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36036.499045 secs 2023-03-10T00:58:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36036.499045 secs 2023-03-10T00:58:46.506 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36036.499045 secs 2023-03-10T00:58:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:47 smithi122 ceph-mon[93869]: pgmap v19730: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 153 KiB/s wr, 10 op/s 2023-03-10T00:58:47.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:47 smithi181 ceph-mon[94189]: pgmap v19730: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 153 KiB/s wr, 10 op/s 2023-03-10T00:58:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:47 smithi038 ceph-mon[92422]: pgmap v19730: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 156 KiB/s rd, 153 KiB/s wr, 10 op/s 2023-03-10T00:58:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36500.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:58:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:58:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:58:49.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:58:49.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:49 smithi122 ceph-mon[93869]: pgmap v19731: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 153 KiB/s wr, 7 op/s 2023-03-10T00:58:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:58:49.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:58:49.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:58:49.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:49 smithi181 ceph-mon[94189]: pgmap v19731: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 153 KiB/s wr, 7 op/s 2023-03-10T00:58:49.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:58:49.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:58:49.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:58:49.503 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:49 smithi038 ceph-mon[92422]: pgmap v19731: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 155 KiB/s rd, 153 KiB/s wr, 7 op/s 2023-03-10T00:58:50.324 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:50 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:58:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:50 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:58:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:50 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T00:58:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36041.499105 secs 2023-03-10T00:58:51.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:51 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:58:51.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:58:51.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:58:51.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:51 smithi038 ceph-mon[92422]: pgmap v19732: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 183 KiB/s wr, 8 op/s 2023-03-10T00:58:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36041.499105 secs 2023-03-10T00:58:51.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:51 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:58:51.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:58:51.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:58:51.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:51 smithi122 ceph-mon[93869]: pgmap v19732: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 183 KiB/s wr, 8 op/s 2023-03-10T00:58:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36041.499105 secs 2023-03-10T00:58:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:51 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:58:51.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:58:51.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:58:51.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:51 smithi181 ceph-mon[94189]: pgmap v19732: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 183 KiB/s wr, 8 op/s 2023-03-10T00:58:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:58:52.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:58:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:58:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:58:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:58:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:58:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:58:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:58:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:58:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:58:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:58:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:53 smithi122 ceph-mon[93869]: pgmap v19733: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 507 KiB/s rd, 414 KiB/s wr, 10 op/s 2023-03-10T00:58:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:58:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:58:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:58:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:58:53.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:58:53.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:53 smithi181 ceph-mon[94189]: pgmap v19733: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 507 KiB/s rd, 414 KiB/s wr, 10 op/s 2023-03-10T00:58:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:58:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:58:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:58:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:58:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:58:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:53 smithi038 ceph-mon[92422]: pgmap v19733: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 507 KiB/s rd, 414 KiB/s wr, 10 op/s 2023-03-10T00:58:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36505.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:58:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:55 smithi122 ceph-mon[93869]: pgmap v19734: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 701 KiB/s rd, 610 KiB/s wr, 8 op/s 2023-03-10T00:58:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:55 smithi181 ceph-mon[94189]: pgmap v19734: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 701 KiB/s rd, 610 KiB/s wr, 8 op/s 2023-03-10T00:58:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:55 smithi038 ceph-mon[92422]: pgmap v19734: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 701 KiB/s rd, 610 KiB/s wr, 8 op/s 2023-03-10T00:58:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36046.499202 secs 2023-03-10T00:58:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:58:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:58:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36046.499202 secs 2023-03-10T00:58:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:58:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:58:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36046.499202 secs 2023-03-10T00:58:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:58:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:58:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:58:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:58:57.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:57 smithi122 ceph-mon[93869]: pgmap v19735: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 775 KiB/s rd, 682 KiB/s wr, 13 op/s 2023-03-10T00:58:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:58:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:58:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:57 smithi181 ceph-mon[94189]: pgmap v19735: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 775 KiB/s rd, 682 KiB/s wr, 13 op/s 2023-03-10T00:58:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:58:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:58:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:57 smithi038 ceph-mon[92422]: pgmap v19735: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 775 KiB/s rd, 682 KiB/s wr, 13 op/s 2023-03-10T00:58:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:58 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 36048.959472 secs 2023-03-10T00:58:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:58 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 36048.959472 secs 2023-03-10T00:58:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:58 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 36048.959472 secs 2023-03-10T00:58:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36510.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:58:59.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:59 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:58:59.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:58:59.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:58:59.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:58:59 smithi122 ceph-mon[93869]: pgmap v19736: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 767 KiB/s rd, 675 KiB/s wr, 11 op/s 2023-03-10T00:58:59.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:59 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:58:59.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:58:59.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:58:59.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:58:59 smithi181 ceph-mon[94189]: pgmap v19736: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 767 KiB/s rd, 675 KiB/s wr, 11 op/s 2023-03-10T00:58:59.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:59 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T00:58:59.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:58:59.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:58:59.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:58:59 smithi038 ceph-mon[92422]: pgmap v19736: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 767 KiB/s rd, 675 KiB/s wr, 11 op/s 2023-03-10T00:59:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:59:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:59:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:59:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:59:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:59:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:59:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36051.499265 secs 2023-03-10T00:59:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:59:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:59:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:01 smithi122 ceph-mon[93869]: pgmap v19737: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 808 KiB/s rd, 705 KiB/s wr, 12 op/s 2023-03-10T00:59:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36051.499265 secs 2023-03-10T00:59:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:59:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:59:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:01 smithi181 ceph-mon[94189]: pgmap v19737: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 808 KiB/s rd, 705 KiB/s wr, 12 op/s 2023-03-10T00:59:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36051.499265 secs 2023-03-10T00:59:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:59:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:59:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:01 smithi038 ceph-mon[92422]: pgmap v19737: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 808 KiB/s rd, 705 KiB/s wr, 12 op/s 2023-03-10T00:59:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36515.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:59:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:03 smithi122 ceph-mon[93869]: pgmap v19738: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 981 KiB/s wr, 15 op/s 2023-03-10T00:59:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:03 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 36053.959534 secs 2023-03-10T00:59:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:03 smithi181 ceph-mon[94189]: pgmap v19738: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 981 KiB/s wr, 15 op/s 2023-03-10T00:59:03.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:03 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 36053.959534 secs 2023-03-10T00:59:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:03 smithi038 ceph-mon[92422]: pgmap v19738: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 981 KiB/s wr, 15 op/s 2023-03-10T00:59:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:03 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 36053.959534 secs 2023-03-10T00:59:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:05 smithi122 ceph-mon[93869]: pgmap v19739: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 809 KiB/s rd, 795 KiB/s wr, 14 op/s 2023-03-10T00:59:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:05 smithi181 ceph-mon[94189]: pgmap v19739: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 809 KiB/s rd, 795 KiB/s wr, 14 op/s 2023-03-10T00:59:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:05 smithi038 ceph-mon[92422]: pgmap v19739: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 809 KiB/s rd, 795 KiB/s wr, 14 op/s 2023-03-10T00:59:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36056.499353 secs 2023-03-10T00:59:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36056.499353 secs 2023-03-10T00:59:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36056.499353 secs 2023-03-10T00:59:07.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:07 smithi122 ceph-mon[93869]: pgmap v19740: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 671 KiB/s rd, 677 KiB/s wr, 18 op/s 2023-03-10T00:59:07.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:07 smithi181 ceph-mon[94189]: pgmap v19740: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 671 KiB/s rd, 677 KiB/s wr, 18 op/s 2023-03-10T00:59:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:07 smithi038 ceph-mon[92422]: pgmap v19740: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 671 KiB/s rd, 677 KiB/s wr, 18 op/s 2023-03-10T00:59:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:59:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:59:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36058.959625 secs 2023-03-10T00:59:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:59:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:59:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36058.959625 secs 2023-03-10T00:59:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:59:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:59:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36058.959625 secs 2023-03-10T00:59:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36520.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:59:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:59:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:59:09.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:09 smithi122 ceph-mon[93869]: pgmap v19741: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 660 KiB/s rd, 862 KiB/s wr, 27 op/s 2023-03-10T00:59:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:59:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:59:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:09 smithi181 ceph-mon[94189]: pgmap v19741: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 660 KiB/s rd, 862 KiB/s wr, 27 op/s 2023-03-10T00:59:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:59:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:59:09.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:09 smithi038 ceph-mon[92422]: pgmap v19741: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 660 KiB/s rd, 862 KiB/s wr, 27 op/s 2023-03-10T00:59:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36061.499414 secs 2023-03-10T00:59:11.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:11 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:59:11.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:11 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:59:11.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:11 smithi122 ceph-mon[93869]: osdmap e1449: 12 total, 12 up, 12 in 2023-03-10T00:59:11.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:11 smithi122 ceph-mon[93869]: pgmap v19743: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 567 KiB/s rd, 823 KiB/s wr, 29 op/s 2023-03-10T00:59:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36061.499414 secs 2023-03-10T00:59:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:11 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:59:11.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:11 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:59:11.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:11 smithi181 ceph-mon[94189]: osdmap e1449: 12 total, 12 up, 12 in 2023-03-10T00:59:11.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:11 smithi181 ceph-mon[94189]: pgmap v19743: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 567 KiB/s rd, 823 KiB/s wr, 29 op/s 2023-03-10T00:59:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36061.499414 secs 2023-03-10T00:59:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:11 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T00:59:11.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:11 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:59:11.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:11 smithi038 ceph-mon[92422]: osdmap e1449: 12 total, 12 up, 12 in 2023-03-10T00:59:11.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:11 smithi038 ceph-mon[92422]: pgmap v19743: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 567 KiB/s rd, 823 KiB/s wr, 29 op/s 2023-03-10T00:59:13.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36525.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:59:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:13 smithi122 ceph-mon[93869]: pgmap v19744: 129 pgs: 3 active+clean+snaptrim_wait, 126 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 580 KiB/s wr, 28 op/s 2023-03-10T00:59:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36063.959682 secs 2023-03-10T00:59:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:13 smithi181 ceph-mon[94189]: pgmap v19744: 129 pgs: 3 active+clean+snaptrim_wait, 126 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 580 KiB/s wr, 28 op/s 2023-03-10T00:59:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36063.959682 secs 2023-03-10T00:59:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:13 smithi038 ceph-mon[92422]: pgmap v19744: 129 pgs: 3 active+clean+snaptrim_wait, 126 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 580 KiB/s wr, 28 op/s 2023-03-10T00:59:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36063.959682 secs 2023-03-10T00:59:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:15 smithi122 ceph-mon[93869]: pgmap v19745: 129 pgs: 3 active+clean+snaptrim_wait, 126 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 556 KiB/s rd, 599 KiB/s wr, 27 op/s 2023-03-10T00:59:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:15 smithi181 ceph-mon[94189]: pgmap v19745: 129 pgs: 3 active+clean+snaptrim_wait, 126 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 556 KiB/s rd, 599 KiB/s wr, 27 op/s 2023-03-10T00:59:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:15 smithi038 ceph-mon[92422]: pgmap v19745: 129 pgs: 3 active+clean+snaptrim_wait, 126 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 556 KiB/s rd, 599 KiB/s wr, 27 op/s 2023-03-10T00:59:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36066.499472 secs 2023-03-10T00:59:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36066.499472 secs 2023-03-10T00:59:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36066.499472 secs 2023-03-10T00:59:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:59:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:59:17.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:17 smithi122 ceph-mon[93869]: pgmap v19746: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 643 KiB/s wr, 24 op/s 2023-03-10T00:59:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:59:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:59:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:17 smithi181 ceph-mon[94189]: pgmap v19746: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 643 KiB/s wr, 24 op/s 2023-03-10T00:59:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:59:17.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:59:17.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:17 smithi038 ceph-mon[92422]: pgmap v19746: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 643 KiB/s wr, 24 op/s 2023-03-10T00:59:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36068.959741 secs 2023-03-10T00:59:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36068.959741 secs 2023-03-10T00:59:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36068.959741 secs 2023-03-10T00:59:18.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36530.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:59:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:19 smithi122 ceph-mon[93869]: pgmap v19747: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 337 KiB/s wr, 9 op/s 2023-03-10T00:59:19.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:19 smithi122 ceph-mon[93869]: osdmap e1450: 12 total, 12 up, 12 in 2023-03-10T00:59:19.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:19 smithi181 ceph-mon[94189]: pgmap v19747: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 337 KiB/s wr, 9 op/s 2023-03-10T00:59:19.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:19 smithi181 ceph-mon[94189]: osdmap e1450: 12 total, 12 up, 12 in 2023-03-10T00:59:19.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:19 smithi038 ceph-mon[92422]: pgmap v19747: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 337 KiB/s wr, 9 op/s 2023-03-10T00:59:19.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:19 smithi038 ceph-mon[92422]: osdmap e1450: 12 total, 12 up, 12 in 2023-03-10T00:59:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:59:20.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:59:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:59:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:59:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:59:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:59:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36071.499555 secs 2023-03-10T00:59:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:21 smithi122 ceph-mon[93869]: pgmap v19749: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 337 KiB/s wr, 9 op/s 2023-03-10T00:59:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36071.499555 secs 2023-03-10T00:59:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:21 smithi181 ceph-mon[94189]: pgmap v19749: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 337 KiB/s wr, 9 op/s 2023-03-10T00:59:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36071.499555 secs 2023-03-10T00:59:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:21 smithi038 ceph-mon[92422]: pgmap v19749: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 337 KiB/s wr, 9 op/s 2023-03-10T00:59:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36535.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:59:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:23 smithi122 ceph-mon[93869]: pgmap v19750: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 213 KiB/s wr, 8 op/s 2023-03-10T00:59:23.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36073.959826 secs 2023-03-10T00:59:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:23 smithi181 ceph-mon[94189]: pgmap v19750: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 213 KiB/s wr, 8 op/s 2023-03-10T00:59:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36073.959826 secs 2023-03-10T00:59:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:23 smithi038 ceph-mon[92422]: pgmap v19750: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 213 KiB/s wr, 8 op/s 2023-03-10T00:59:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36073.959826 secs 2023-03-10T00:59:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:25 smithi122 ceph-mon[93869]: pgmap v19751: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 139 KiB/s wr, 7 op/s 2023-03-10T00:59:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:25 smithi181 ceph-mon[94189]: pgmap v19751: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 139 KiB/s wr, 7 op/s 2023-03-10T00:59:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:25 smithi038 ceph-mon[92422]: pgmap v19751: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 139 KiB/s wr, 7 op/s 2023-03-10T00:59:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36076.499614 secs 2023-03-10T00:59:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36076.499614 secs 2023-03-10T00:59:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36076.499614 secs 2023-03-10T00:59:27.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:27 smithi122 ceph-mon[93869]: pgmap v19752: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-10T00:59:27.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:27 smithi181 ceph-mon[94189]: pgmap v19752: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-10T00:59:27.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:27 smithi038 ceph-mon[92422]: pgmap v19752: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 6 op/s 2023-03-10T00:59:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36078.959884 secs 2023-03-10T00:59:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36078.959884 secs 2023-03-10T00:59:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36078.959884 secs 2023-03-10T00:59:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36540.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:59:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:29 smithi122 ceph-mon[93869]: pgmap v19753: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-10T00:59:29.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:29 smithi181 ceph-mon[94189]: pgmap v19753: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-10T00:59:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:29 smithi038 ceph-mon[92422]: pgmap v19753: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-10T00:59:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36081.499675 secs 2023-03-10T00:59:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:31 smithi122 ceph-mon[93869]: pgmap v19754: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2023-03-10T00:59:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36081.499675 secs 2023-03-10T00:59:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:31 smithi181 ceph-mon[94189]: pgmap v19754: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2023-03-10T00:59:32.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36081.499675 secs 2023-03-10T00:59:32.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:31 smithi038 ceph-mon[92422]: pgmap v19754: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2023-03-10T00:59:33.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36545.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:59:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:33 smithi122 ceph-mon[93869]: pgmap v19755: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-10T00:59:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36083.959941 secs 2023-03-10T00:59:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:33 smithi181 ceph-mon[94189]: pgmap v19755: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-10T00:59:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36083.959941 secs 2023-03-10T00:59:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:33 smithi038 ceph-mon[92422]: pgmap v19755: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2023-03-10T00:59:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36083.959941 secs 2023-03-10T00:59:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:35 smithi122 ceph-mon[93869]: pgmap v19756: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-10T00:59:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:35 smithi181 ceph-mon[94189]: pgmap v19756: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-10T00:59:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:35 smithi038 ceph-mon[92422]: pgmap v19756: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2023-03-10T00:59:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36086.499734 secs 2023-03-10T00:59:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36086.499734 secs 2023-03-10T00:59:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36086.499734 secs 2023-03-10T00:59:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:37 smithi122 ceph-mon[93869]: pgmap v19757: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 124 KiB/s wr, 7 op/s 2023-03-10T00:59:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:37 smithi181 ceph-mon[94189]: pgmap v19757: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 124 KiB/s wr, 7 op/s 2023-03-10T00:59:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:37 smithi038 ceph-mon[92422]: pgmap v19757: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 124 KiB/s wr, 7 op/s 2023-03-10T00:59:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36088.960046 secs 2023-03-10T00:59:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36088.960046 secs 2023-03-10T00:59:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36088.960046 secs 2023-03-10T00:59:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36550.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:59:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:39 smithi122 ceph-mon[93869]: pgmap v19758: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 166 KiB/s wr, 6 op/s 2023-03-10T00:59:39.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:39 smithi181 ceph-mon[94189]: pgmap v19758: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 166 KiB/s wr, 6 op/s 2023-03-10T00:59:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:39 smithi038 ceph-mon[92422]: pgmap v19758: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 166 KiB/s wr, 6 op/s 2023-03-10T00:59:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36091.499791 secs 2023-03-10T00:59:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:41 smithi122 ceph-mon[93869]: pgmap v19759: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 166 KiB/s wr, 6 op/s 2023-03-10T00:59:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36091.499791 secs 2023-03-10T00:59:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:41 smithi181 ceph-mon[94189]: pgmap v19759: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 166 KiB/s wr, 6 op/s 2023-03-10T00:59:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36091.499791 secs 2023-03-10T00:59:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:41 smithi038 ceph-mon[92422]: pgmap v19759: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 168 KiB/s rd, 166 KiB/s wr, 6 op/s 2023-03-10T00:59:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36555.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:59:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:43 smithi122 ceph-mon[93869]: pgmap v19760: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 166 KiB/s wr, 8 op/s 2023-03-10T00:59:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36093.960121 secs 2023-03-10T00:59:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:43 smithi181 ceph-mon[94189]: pgmap v19760: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 166 KiB/s wr, 8 op/s 2023-03-10T00:59:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36093.960121 secs 2023-03-10T00:59:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:43 smithi038 ceph-mon[92422]: pgmap v19760: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 166 KiB/s wr, 8 op/s 2023-03-10T00:59:44.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36093.960121 secs 2023-03-10T00:59:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:45 smithi122 ceph-mon[93869]: pgmap v19761: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 428 KiB/s wr, 7 op/s 2023-03-10T00:59:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:45 smithi181 ceph-mon[94189]: pgmap v19761: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 428 KiB/s wr, 7 op/s 2023-03-10T00:59:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:45 smithi038 ceph-mon[92422]: pgmap v19761: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 428 KiB/s wr, 7 op/s 2023-03-10T00:59:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36096.499850 secs 2023-03-10T00:59:46.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36096.499850 secs 2023-03-10T00:59:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36096.499850 secs 2023-03-10T00:59:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:47 smithi122 ceph-mon[93869]: pgmap v19762: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 428 KiB/s wr, 9 op/s 2023-03-10T00:59:47.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:47 smithi181 ceph-mon[94189]: pgmap v19762: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 428 KiB/s wr, 9 op/s 2023-03-10T00:59:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:47 smithi038 ceph-mon[92422]: pgmap v19762: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 428 KiB/s wr, 9 op/s 2023-03-10T00:59:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:59:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:59:48.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36098.960148 secs 2023-03-10T00:59:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:59:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:59:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36098.960148 secs 2023-03-10T00:59:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:59:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:59:48.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36098.960148 secs 2023-03-10T00:59:48.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36560.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:59:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:59:49.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:59:49.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:49 smithi122 ceph-mon[93869]: pgmap v19763: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 342 KiB/s wr, 8 op/s 2023-03-10T00:59:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:59:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:59:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:49 smithi181 ceph-mon[94189]: pgmap v19763: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 342 KiB/s wr, 8 op/s 2023-03-10T00:59:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:59:49.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:59:49.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:49 smithi038 ceph-mon[92422]: pgmap v19763: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 342 KiB/s wr, 8 op/s 2023-03-10T00:59:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36101.499910 secs 2023-03-10T00:59:51.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:51 smithi122 ceph-mon[93869]: pgmap v19764: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 330 KiB/s wr, 8 op/s 2023-03-10T00:59:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36101.499910 secs 2023-03-10T00:59:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:51 smithi181 ceph-mon[94189]: pgmap v19764: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 330 KiB/s wr, 8 op/s 2023-03-10T00:59:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36101.499910 secs 2023-03-10T00:59:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:51 smithi038 ceph-mon[92422]: pgmap v19764: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 330 KiB/s wr, 8 op/s 2023-03-10T00:59:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:59:52.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:59:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T00:59:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36565.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:59:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:53 smithi122 ceph-mon[93869]: pgmap v19765: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 824 KiB/s rd, 727 KiB/s wr, 11 op/s 2023-03-10T00:59:53.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36103.960235 secs 2023-03-10T00:59:53.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:59:53.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:59:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:53 smithi181 ceph-mon[94189]: pgmap v19765: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 824 KiB/s rd, 727 KiB/s wr, 11 op/s 2023-03-10T00:59:53.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36103.960235 secs 2023-03-10T00:59:53.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:59:53.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:59:54.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:53 smithi038 ceph-mon[92422]: pgmap v19765: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 824 KiB/s rd, 727 KiB/s wr, 11 op/s 2023-03-10T00:59:54.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36103.960235 secs 2023-03-10T00:59:54.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:59:54.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:59:55.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:55 smithi122 ceph-mon[93869]: pgmap v19766: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 885 KiB/s rd, 727 KiB/s wr, 9 op/s 2023-03-10T00:59:55.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:55 smithi181 ceph-mon[94189]: pgmap v19766: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 885 KiB/s rd, 727 KiB/s wr, 9 op/s 2023-03-10T00:59:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:55 smithi038 ceph-mon[92422]: pgmap v19766: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 885 KiB/s rd, 727 KiB/s wr, 9 op/s 2023-03-10T00:59:56.387 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36106.500002 secs 2023-03-10T00:59:56.387 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:59:56.388 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:59:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36106.500002 secs 2023-03-10T00:59:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:59:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:59:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36106.500002 secs 2023-03-10T00:59:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T00:59:56.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:59:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:57 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:59:57.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:59:57.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:59:57.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:57 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:59:57.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:57 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:59:57.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:57 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:59:57.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:57 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:59:57.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:57 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:59:57.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:57 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:59:57.479 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:57 smithi122 ceph-mon[93869]: pgmap v19767: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 802 KiB/s rd, 499 KiB/s wr, 12 op/s 2023-03-10T00:59:57.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:57 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:59:57.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:59:57.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:59:57.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:57 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:59:57.483 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:57 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:59:57.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:57 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:59:57.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:57 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:59:57.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:57 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:59:57.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:57 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:59:57.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:57 smithi181 ceph-mon[94189]: pgmap v19767: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 802 KiB/s rd, 499 KiB/s wr, 12 op/s 2023-03-10T00:59:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:57 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:59:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T00:59:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T00:59:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:57 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:59:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:57 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:59:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:57 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:59:57.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:57 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T00:59:57.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:57 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T00:59:57.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:57 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T00:59:57.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:57 smithi038 ceph-mon[92422]: pgmap v19767: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 802 KiB/s rd, 499 KiB/s wr, 12 op/s 2023-03-10T00:59:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36108.960295 secs 2023-03-10T00:59:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36108.960295 secs 2023-03-10T00:59:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36108.960295 secs 2023-03-10T00:59:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36570.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T00:59:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 00:59:59 smithi122 ceph-mon[93869]: pgmap v19768: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 984 KiB/s rd, 682 KiB/s wr, 12 op/s 2023-03-10T00:59:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 00:59:59 smithi181 ceph-mon[94189]: pgmap v19768: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 984 KiB/s rd, 682 KiB/s wr, 12 op/s 2023-03-10T00:59:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 00:59:59 smithi038 ceph-mon[92422]: pgmap v19768: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 984 KiB/s rd, 682 KiB/s wr, 12 op/s 2023-03-10T01:00:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:00:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:00:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:00 smithi122 ceph-mon[93869]: overall HEALTH_WARN 2 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-10T01:00:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:00:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:00:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:00 smithi181 ceph-mon[94189]: overall HEALTH_WARN 2 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-10T01:00:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:00:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:00:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:00 smithi038 ceph-mon[92422]: overall HEALTH_WARN 2 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-10T01:00:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36111.500058 secs 2023-03-10T01:00:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:00:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:00:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:01 smithi122 ceph-mon[93869]: pgmap v19769: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 751 KiB/s rd, 675 KiB/s wr, 11 op/s 2023-03-10T01:00:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36111.500058 secs 2023-03-10T01:00:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:00:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:00:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:01 smithi181 ceph-mon[94189]: pgmap v19769: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 751 KiB/s rd, 675 KiB/s wr, 11 op/s 2023-03-10T01:00:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36111.500058 secs 2023-03-10T01:00:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:00:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:00:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:01 smithi038 ceph-mon[92422]: pgmap v19769: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 751 KiB/s rd, 675 KiB/s wr, 11 op/s 2023-03-10T01:00:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36575.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:00:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:03 smithi122 ceph-mon[93869]: pgmap v19770: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 997 KiB/s wr, 16 op/s 2023-03-10T01:00:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36113.960355 secs 2023-03-10T01:00:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:03 smithi181 ceph-mon[94189]: pgmap v19770: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 997 KiB/s wr, 16 op/s 2023-03-10T01:00:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36113.960355 secs 2023-03-10T01:00:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:03 smithi038 ceph-mon[92422]: pgmap v19770: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 997 KiB/s wr, 16 op/s 2023-03-10T01:00:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36113.960355 secs 2023-03-10T01:00:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:05 smithi122 ceph-mon[93869]: pgmap v19771: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 610 KiB/s wr, 14 op/s 2023-03-10T01:00:05.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:05 smithi181 ceph-mon[94189]: pgmap v19771: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 610 KiB/s wr, 14 op/s 2023-03-10T01:00:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:05 smithi038 ceph-mon[92422]: pgmap v19771: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 610 KiB/s wr, 14 op/s 2023-03-10T01:00:06.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36116.500117 secs 2023-03-10T01:00:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36116.500117 secs 2023-03-10T01:00:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36116.500117 secs 2023-03-10T01:00:07.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:07 smithi122 ceph-mon[93869]: pgmap v19772: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 735 KiB/s rd, 741 KiB/s wr, 19 op/s 2023-03-10T01:00:07.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:07 smithi181 ceph-mon[94189]: pgmap v19772: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 735 KiB/s rd, 741 KiB/s wr, 19 op/s 2023-03-10T01:00:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:07 smithi038 ceph-mon[92422]: pgmap v19772: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 735 KiB/s rd, 741 KiB/s wr, 19 op/s 2023-03-10T01:00:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36118.960462 secs 2023-03-10T01:00:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36118.960462 secs 2023-03-10T01:00:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36118.960462 secs 2023-03-10T01:00:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36580.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:00:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:09 smithi122 ceph-mon[93869]: pgmap v19773: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 842 KiB/s rd, 1.0 MiB/s wr, 30 op/s 2023-03-10T01:00:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:09 smithi181 ceph-mon[94189]: pgmap v19773: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 842 KiB/s rd, 1.0 MiB/s wr, 30 op/s 2023-03-10T01:00:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:09 smithi038 ceph-mon[92422]: pgmap v19773: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 842 KiB/s rd, 1.0 MiB/s wr, 30 op/s 2023-03-10T01:00:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36121.500193 secs 2023-03-10T01:00:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:11 smithi122 ceph-mon[93869]: osdmap e1451: 12 total, 12 up, 12 in 2023-03-10T01:00:11.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:11 smithi122 ceph-mon[93869]: pgmap v19775: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 737 KiB/s rd, 993 KiB/s wr, 32 op/s 2023-03-10T01:00:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36121.500193 secs 2023-03-10T01:00:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:11 smithi181 ceph-mon[94189]: osdmap e1451: 12 total, 12 up, 12 in 2023-03-10T01:00:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:11 smithi181 ceph-mon[94189]: pgmap v19775: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 737 KiB/s rd, 993 KiB/s wr, 32 op/s 2023-03-10T01:00:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36121.500193 secs 2023-03-10T01:00:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:11 smithi038 ceph-mon[92422]: osdmap e1451: 12 total, 12 up, 12 in 2023-03-10T01:00:11.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:11 smithi038 ceph-mon[92422]: pgmap v19775: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 737 KiB/s rd, 993 KiB/s wr, 32 op/s 2023-03-10T01:00:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:00:12.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:00:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:00:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:00:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:00:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:00:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:00:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:00:13.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:13 smithi122 ceph-mon[93869]: pgmap v19776: 129 pgs: 3 active+clean+snaptrim_wait, 126 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 442 KiB/s rd, 697 KiB/s wr, 32 op/s 2023-03-10T01:00:13.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36123.960542 secs 2023-03-10T01:00:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:00:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:00:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:13 smithi181 ceph-mon[94189]: pgmap v19776: 129 pgs: 3 active+clean+snaptrim_wait, 126 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 442 KiB/s rd, 697 KiB/s wr, 32 op/s 2023-03-10T01:00:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36123.960542 secs 2023-03-10T01:00:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:00:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:00:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:13 smithi038 ceph-mon[92422]: pgmap v19776: 129 pgs: 3 active+clean+snaptrim_wait, 126 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 442 KiB/s rd, 697 KiB/s wr, 32 op/s 2023-03-10T01:00:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36123.960542 secs 2023-03-10T01:00:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36585.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:00:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:15 smithi122 ceph-mon[93869]: pgmap v19777: 129 pgs: 3 active+clean+snaptrim_wait, 126 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 715 KiB/s rd, 757 KiB/s wr, 31 op/s 2023-03-10T01:00:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:15 smithi181 ceph-mon[94189]: pgmap v19777: 129 pgs: 3 active+clean+snaptrim_wait, 126 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 715 KiB/s rd, 757 KiB/s wr, 31 op/s 2023-03-10T01:00:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:15 smithi038 ceph-mon[92422]: pgmap v19777: 129 pgs: 3 active+clean+snaptrim_wait, 126 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 715 KiB/s rd, 757 KiB/s wr, 31 op/s 2023-03-10T01:00:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36126.500253 secs 2023-03-10T01:00:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36126.500253 secs 2023-03-10T01:00:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36126.500253 secs 2023-03-10T01:00:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:00:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:00:17.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:17 smithi122 ceph-mon[93869]: pgmap v19778: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 583 KiB/s rd, 600 KiB/s wr, 27 op/s 2023-03-10T01:00:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:00:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:00:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:17 smithi181 ceph-mon[94189]: pgmap v19778: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 583 KiB/s rd, 600 KiB/s wr, 27 op/s 2023-03-10T01:00:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:00:17.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:00:17.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:17 smithi038 ceph-mon[92422]: pgmap v19778: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 583 KiB/s rd, 600 KiB/s wr, 27 op/s 2023-03-10T01:00:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36128.960602 secs 2023-03-10T01:00:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36128.960602 secs 2023-03-10T01:00:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36128.960602 secs 2023-03-10T01:00:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36590.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:00:19.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:19 smithi122 ceph-mon[93869]: pgmap v19779: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 554 KiB/s rd, 339 KiB/s wr, 12 op/s 2023-03-10T01:00:19.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:19 smithi122 ceph-mon[93869]: osdmap e1452: 12 total, 12 up, 12 in 2023-03-10T01:00:19.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:19 smithi181 ceph-mon[94189]: pgmap v19779: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 554 KiB/s rd, 339 KiB/s wr, 12 op/s 2023-03-10T01:00:19.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:19 smithi181 ceph-mon[94189]: osdmap e1452: 12 total, 12 up, 12 in 2023-03-10T01:00:19.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:19 smithi038 ceph-mon[92422]: pgmap v19779: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 554 KiB/s rd, 339 KiB/s wr, 12 op/s 2023-03-10T01:00:19.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:19 smithi038 ceph-mon[92422]: osdmap e1452: 12 total, 12 up, 12 in 2023-03-10T01:00:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:00:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:00:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:00:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:00:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:00:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:00:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36131.500313 secs 2023-03-10T01:00:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:21 smithi122 ceph-mon[93869]: pgmap v19781: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 554 KiB/s rd, 339 KiB/s wr, 12 op/s 2023-03-10T01:00:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36131.500313 secs 2023-03-10T01:00:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:21 smithi181 ceph-mon[94189]: pgmap v19781: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 554 KiB/s rd, 339 KiB/s wr, 12 op/s 2023-03-10T01:00:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36131.500313 secs 2023-03-10T01:00:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:21 smithi038 ceph-mon[92422]: pgmap v19781: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 554 KiB/s rd, 339 KiB/s wr, 12 op/s 2023-03-10T01:00:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36595.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:00:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:23 smithi122 ceph-mon[93869]: pgmap v19782: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 212 KiB/s wr, 9 op/s 2023-03-10T01:00:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36133.960713 secs 2023-03-10T01:00:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:23 smithi181 ceph-mon[94189]: pgmap v19782: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 212 KiB/s wr, 9 op/s 2023-03-10T01:00:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36133.960713 secs 2023-03-10T01:00:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:23 smithi038 ceph-mon[92422]: pgmap v19782: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 212 KiB/s wr, 9 op/s 2023-03-10T01:00:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36133.960713 secs 2023-03-10T01:00:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:25 smithi122 ceph-mon[93869]: pgmap v19783: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 140 KiB/s wr, 8 op/s 2023-03-10T01:00:25.484 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:25 smithi181 ceph-mon[94189]: pgmap v19783: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 140 KiB/s wr, 8 op/s 2023-03-10T01:00:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:25 smithi038 ceph-mon[92422]: pgmap v19783: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 140 KiB/s wr, 8 op/s 2023-03-10T01:00:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36136.500400 secs 2023-03-10T01:00:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36136.500400 secs 2023-03-10T01:00:26.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36136.500400 secs 2023-03-10T01:00:27.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:27 smithi122 ceph-mon[93869]: pgmap v19784: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 139 KiB/s wr, 8 op/s 2023-03-10T01:00:27.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:27 smithi181 ceph-mon[94189]: pgmap v19784: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 139 KiB/s wr, 8 op/s 2023-03-10T01:00:27.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:27 smithi038 ceph-mon[92422]: pgmap v19784: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 139 KiB/s wr, 8 op/s 2023-03-10T01:00:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36138.960785 secs 2023-03-10T01:00:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36138.960785 secs 2023-03-10T01:00:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36138.960785 secs 2023-03-10T01:00:28.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36600.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:00:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:29 smithi122 ceph-mon[93869]: pgmap v19785: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 7 op/s 2023-03-10T01:00:29.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:29 smithi181 ceph-mon[94189]: pgmap v19785: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 7 op/s 2023-03-10T01:00:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:29 smithi038 ceph-mon[92422]: pgmap v19785: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 7 op/s 2023-03-10T01:00:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36141.500458 secs 2023-03-10T01:00:31.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:31 smithi122 ceph-mon[93869]: pgmap v19786: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 7 op/s 2023-03-10T01:00:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36141.500458 secs 2023-03-10T01:00:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:31 smithi181 ceph-mon[94189]: pgmap v19786: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 7 op/s 2023-03-10T01:00:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36141.500458 secs 2023-03-10T01:00:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:31 smithi038 ceph-mon[92422]: pgmap v19786: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 7 op/s 2023-03-10T01:00:33.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36605.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:00:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:33 smithi122 ceph-mon[93869]: pgmap v19787: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 9 op/s 2023-03-10T01:00:33.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36143.960869 secs 2023-03-10T01:00:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:33 smithi181 ceph-mon[94189]: pgmap v19787: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 9 op/s 2023-03-10T01:00:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36143.960869 secs 2023-03-10T01:00:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:33 smithi038 ceph-mon[92422]: pgmap v19787: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 9 op/s 2023-03-10T01:00:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36143.960869 secs 2023-03-10T01:00:35.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:35 smithi122 ceph-mon[93869]: pgmap v19788: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 7 op/s 2023-03-10T01:00:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:35 smithi181 ceph-mon[94189]: pgmap v19788: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 7 op/s 2023-03-10T01:00:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:35 smithi038 ceph-mon[92422]: pgmap v19788: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 7 op/s 2023-03-10T01:00:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36146.500516 secs 2023-03-10T01:00:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36146.500516 secs 2023-03-10T01:00:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36146.500516 secs 2023-03-10T01:00:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:37 smithi122 ceph-mon[93869]: pgmap v19789: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-10T01:00:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:37 smithi181 ceph-mon[94189]: pgmap v19789: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-10T01:00:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:37 smithi038 ceph-mon[92422]: pgmap v19789: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2023-03-10T01:00:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36148.960950 secs 2023-03-10T01:00:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36148.960950 secs 2023-03-10T01:00:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36148.960950 secs 2023-03-10T01:00:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36610.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:00:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:39 smithi122 ceph-mon[93869]: pgmap v19790: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 8 op/s 2023-03-10T01:00:39.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:39 smithi181 ceph-mon[94189]: pgmap v19790: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 8 op/s 2023-03-10T01:00:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:39 smithi038 ceph-mon[92422]: pgmap v19790: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 8 op/s 2023-03-10T01:00:41.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36151.500574 secs 2023-03-10T01:00:41.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:41 smithi122 ceph-mon[93869]: pgmap v19791: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 6 op/s 2023-03-10T01:00:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36151.500574 secs 2023-03-10T01:00:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:41 smithi181 ceph-mon[94189]: pgmap v19791: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 6 op/s 2023-03-10T01:00:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36151.500574 secs 2023-03-10T01:00:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:41 smithi038 ceph-mon[92422]: pgmap v19791: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 6 op/s 2023-03-10T01:00:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36615.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:00:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:43 smithi122 ceph-mon[93869]: pgmap v19792: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 426 B/s wr, 9 op/s 2023-03-10T01:00:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36153.961030 secs 2023-03-10T01:00:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:43 smithi181 ceph-mon[94189]: pgmap v19792: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 426 B/s wr, 9 op/s 2023-03-10T01:00:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36153.961030 secs 2023-03-10T01:00:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:43 smithi038 ceph-mon[92422]: pgmap v19792: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 426 B/s wr, 9 op/s 2023-03-10T01:00:44.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36153.961030 secs 2023-03-10T01:00:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:45 smithi122 ceph-mon[93869]: pgmap v19793: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 8 op/s 2023-03-10T01:00:45.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:45 smithi181 ceph-mon[94189]: pgmap v19793: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 8 op/s 2023-03-10T01:00:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:45 smithi038 ceph-mon[92422]: pgmap v19793: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 682 B/s wr, 8 op/s 2023-03-10T01:00:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36156.500656 secs 2023-03-10T01:00:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36156.500656 secs 2023-03-10T01:00:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36156.500656 secs 2023-03-10T01:00:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:47 smithi122 ceph-mon[93869]: pgmap v19794: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 34 KiB/s wr, 9 op/s 2023-03-10T01:00:47.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:47 smithi181 ceph-mon[94189]: pgmap v19794: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 34 KiB/s wr, 9 op/s 2023-03-10T01:00:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:47 smithi038 ceph-mon[92422]: pgmap v19794: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 34 KiB/s wr, 9 op/s 2023-03-10T01:00:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36620.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:00:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:00:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:00:49.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:49 smithi122 ceph-mon[93869]: pgmap v19795: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 154 KiB/s wr, 9 op/s 2023-03-10T01:00:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:00:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:00:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:49 smithi181 ceph-mon[94189]: pgmap v19795: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 154 KiB/s wr, 9 op/s 2023-03-10T01:00:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:00:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:00:49.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:49 smithi038 ceph-mon[92422]: pgmap v19795: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 154 KiB/s wr, 9 op/s 2023-03-10T01:00:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:50 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:00:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:50 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:00:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:50 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:00:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36161.500714 secs 2023-03-10T01:00:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:51 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:00:51.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:00:51.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:00:51.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:51 smithi122 ceph-mon[93869]: pgmap v19796: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 464 KiB/s rd, 462 KiB/s wr, 8 op/s 2023-03-10T01:00:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36161.500714 secs 2023-03-10T01:00:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:51 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:00:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:00:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:00:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:51 smithi181 ceph-mon[94189]: pgmap v19796: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 464 KiB/s rd, 462 KiB/s wr, 8 op/s 2023-03-10T01:00:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36161.500714 secs 2023-03-10T01:00:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:51 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:00:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:00:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:00:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:51 smithi038 ceph-mon[92422]: pgmap v19796: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 464 KiB/s rd, 462 KiB/s wr, 8 op/s 2023-03-10T01:00:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:00:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:00:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:00:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:00:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:00:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:00:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:53 smithi122 ceph-mon[93869]: pgmap v19797: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 462 KiB/s wr, 10 op/s 2023-03-10T01:00:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:53 smithi181 ceph-mon[94189]: pgmap v19797: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 462 KiB/s wr, 10 op/s 2023-03-10T01:00:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:53 smithi038 ceph-mon[92422]: pgmap v19797: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 462 KiB/s wr, 10 op/s 2023-03-10T01:00:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36625.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:00:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:55 smithi122 ceph-mon[93869]: pgmap v19798: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 616 KiB/s rd, 614 KiB/s wr, 9 op/s 2023-03-10T01:00:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:55 smithi181 ceph-mon[94189]: pgmap v19798: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 616 KiB/s rd, 614 KiB/s wr, 9 op/s 2023-03-10T01:00:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:55 smithi038 ceph-mon[92422]: pgmap v19798: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 616 KiB/s rd, 614 KiB/s wr, 9 op/s 2023-03-10T01:00:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36166.500772 secs 2023-03-10T01:00:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:00:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:00:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36166.500772 secs 2023-03-10T01:00:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:00:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:00:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36166.500772 secs 2023-03-10T01:00:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:00:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:00:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:00:57.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:00:57.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:57 smithi122 ceph-mon[93869]: pgmap v19799: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 619 KiB/s rd, 614 KiB/s wr, 11 op/s 2023-03-10T01:00:57.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:57 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:00:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:00:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:00:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:57 smithi181 ceph-mon[94189]: pgmap v19799: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 619 KiB/s rd, 614 KiB/s wr, 11 op/s 2023-03-10T01:00:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:57 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:00:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:00:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:00:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:57 smithi038 ceph-mon[92422]: pgmap v19799: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 619 KiB/s rd, 614 KiB/s wr, 11 op/s 2023-03-10T01:00:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:57 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:00:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:00:58.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:00:58.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:00:58.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:58 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 36168.961336 secs 2023-03-10T01:00:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:00:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:00:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:00:58.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:58 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 36168.961336 secs 2023-03-10T01:00:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:00:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:00:58.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:00:58.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:58 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 36168.961336 secs 2023-03-10T01:00:58.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36630.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:00:59.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:59 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:00:59.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:00:59.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:00:59.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:00:59 smithi122 ceph-mon[93869]: pgmap v19800: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 618 KiB/s rd, 762 KiB/s wr, 11 op/s 2023-03-10T01:00:59.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:59 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:00:59.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:00:59.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:00:59.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:00:59 smithi181 ceph-mon[94189]: pgmap v19800: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 618 KiB/s rd, 762 KiB/s wr, 11 op/s 2023-03-10T01:00:59.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:59 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:00:59.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:00:59.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:00:59.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:00:59 smithi038 ceph-mon[92422]: pgmap v19800: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 618 KiB/s rd, 762 KiB/s wr, 11 op/s 2023-03-10T01:01:01.643 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36171.500832 secs 2023-03-10T01:01:01.643 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:01:01.643 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:01:01.643 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:01 smithi038 ceph-mon[92422]: pgmap v19801: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 729 KiB/s rd, 861 KiB/s wr, 11 op/s 2023-03-10T01:01:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36171.500832 secs 2023-03-10T01:01:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:01:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:01:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:01 smithi122 ceph-mon[93869]: pgmap v19801: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 729 KiB/s rd, 861 KiB/s wr, 11 op/s 2023-03-10T01:01:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36171.500832 secs 2023-03-10T01:01:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:01:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:01:01.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:01 smithi181 ceph-mon[94189]: pgmap v19801: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 729 KiB/s rd, 861 KiB/s wr, 11 op/s 2023-03-10T01:01:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36635.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:01:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:03 smithi122 ceph-mon[93869]: pgmap v19802: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 422 KiB/s rd, 553 KiB/s wr, 14 op/s 2023-03-10T01:01:03.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:03 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 36173.961433 secs 2023-03-10T01:01:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:03 smithi181 ceph-mon[94189]: pgmap v19802: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 422 KiB/s rd, 553 KiB/s wr, 14 op/s 2023-03-10T01:01:03.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:03 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 36173.961433 secs 2023-03-10T01:01:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:03 smithi038 ceph-mon[92422]: pgmap v19802: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 422 KiB/s rd, 553 KiB/s wr, 14 op/s 2023-03-10T01:01:04.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:03 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 36173.961433 secs 2023-03-10T01:01:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:05 smithi122 ceph-mon[93869]: pgmap v19803: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 891 KiB/s rd, 805 KiB/s wr, 16 op/s 2023-03-10T01:01:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:05 smithi181 ceph-mon[94189]: pgmap v19803: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 891 KiB/s rd, 805 KiB/s wr, 16 op/s 2023-03-10T01:01:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:05 smithi038 ceph-mon[92422]: pgmap v19803: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 891 KiB/s rd, 805 KiB/s wr, 16 op/s 2023-03-10T01:01:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36176.500889 secs 2023-03-10T01:01:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36176.500889 secs 2023-03-10T01:01:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36176.500889 secs 2023-03-10T01:01:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:07 smithi122 ceph-mon[93869]: pgmap v19804: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 742 KiB/s rd, 675 KiB/s wr, 18 op/s 2023-03-10T01:01:07.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:07 smithi181 ceph-mon[94189]: pgmap v19804: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 742 KiB/s rd, 675 KiB/s wr, 18 op/s 2023-03-10T01:01:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:07 smithi038 ceph-mon[92422]: pgmap v19804: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 742 KiB/s rd, 675 KiB/s wr, 18 op/s 2023-03-10T01:01:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:01:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:01:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36178.961509 secs 2023-03-10T01:01:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:01:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:01:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36178.961509 secs 2023-03-10T01:01:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:01:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:01:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36178.961509 secs 2023-03-10T01:01:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36640.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:01:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:09 smithi122 ceph-mon[93869]: pgmap v19805: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 822 KiB/s rd, 726 KiB/s wr, 27 op/s 2023-03-10T01:01:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:09 smithi181 ceph-mon[94189]: pgmap v19805: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 822 KiB/s rd, 726 KiB/s wr, 27 op/s 2023-03-10T01:01:09.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:09 smithi038 ceph-mon[92422]: pgmap v19805: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 822 KiB/s rd, 726 KiB/s wr, 27 op/s 2023-03-10T01:01:11.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36181.500993 secs 2023-03-10T01:01:11.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:11 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:01:11.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:11 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:01:11.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:11 smithi122 ceph-mon[93869]: osdmap e1453: 12 total, 12 up, 12 in 2023-03-10T01:01:11.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:11 smithi122 ceph-mon[93869]: pgmap v19807: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 923 KiB/s rd, 645 KiB/s wr, 30 op/s 2023-03-10T01:01:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36181.500993 secs 2023-03-10T01:01:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:11 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:01:11.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:11 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:01:11.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:11 smithi181 ceph-mon[94189]: osdmap e1453: 12 total, 12 up, 12 in 2023-03-10T01:01:11.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:11 smithi181 ceph-mon[94189]: pgmap v19807: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 923 KiB/s rd, 645 KiB/s wr, 30 op/s 2023-03-10T01:01:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36181.500993 secs 2023-03-10T01:01:11.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:11 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:01:11.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:11 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:01:11.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:11 smithi038 ceph-mon[92422]: osdmap e1453: 12 total, 12 up, 12 in 2023-03-10T01:01:11.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:11 smithi038 ceph-mon[92422]: pgmap v19807: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 923 KiB/s rd, 645 KiB/s wr, 30 op/s 2023-03-10T01:01:12.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:01:12.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:01:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:01:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:01:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:01:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:01:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:01:13.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:01:13.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:13 smithi122 ceph-mon[93869]: pgmap v19808: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 923 KiB/s rd, 644 KiB/s wr, 30 op/s 2023-03-10T01:01:13.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36183.961588 secs 2023-03-10T01:01:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:01:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:01:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:13 smithi181 ceph-mon[94189]: pgmap v19808: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 923 KiB/s rd, 644 KiB/s wr, 30 op/s 2023-03-10T01:01:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36183.961588 secs 2023-03-10T01:01:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:01:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:01:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:13 smithi038 ceph-mon[92422]: pgmap v19808: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 923 KiB/s rd, 644 KiB/s wr, 30 op/s 2023-03-10T01:01:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36183.961588 secs 2023-03-10T01:01:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36645.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:01:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:15 smithi122 ceph-mon[93869]: pgmap v19809: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 557 KiB/s rd, 541 KiB/s wr, 28 op/s 2023-03-10T01:01:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:15 smithi181 ceph-mon[94189]: pgmap v19809: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 557 KiB/s rd, 541 KiB/s wr, 28 op/s 2023-03-10T01:01:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:15 smithi038 ceph-mon[92422]: pgmap v19809: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 557 KiB/s rd, 541 KiB/s wr, 28 op/s 2023-03-10T01:01:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36186.501010 secs 2023-03-10T01:01:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36186.501010 secs 2023-03-10T01:01:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36186.501010 secs 2023-03-10T01:01:17.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:01:17.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:01:17.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:17 smithi122 ceph-mon[93869]: pgmap v19810: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 556 KiB/s rd, 515 KiB/s wr, 25 op/s 2023-03-10T01:01:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:01:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:01:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:17 smithi181 ceph-mon[94189]: pgmap v19810: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 556 KiB/s rd, 515 KiB/s wr, 25 op/s 2023-03-10T01:01:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:01:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:01:17.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:17 smithi038 ceph-mon[92422]: pgmap v19810: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 556 KiB/s rd, 515 KiB/s wr, 25 op/s 2023-03-10T01:01:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36188.961714 secs 2023-03-10T01:01:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36188.961714 secs 2023-03-10T01:01:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36188.961714 secs 2023-03-10T01:01:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36650.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:01:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:19 smithi122 ceph-mon[93869]: pgmap v19811: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 594 KiB/s rd, 593 KiB/s wr, 12 op/s 2023-03-10T01:01:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:19 smithi122 ceph-mon[93869]: osdmap e1454: 12 total, 12 up, 12 in 2023-03-10T01:01:19.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:19 smithi181 ceph-mon[94189]: pgmap v19811: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 594 KiB/s rd, 593 KiB/s wr, 12 op/s 2023-03-10T01:01:19.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:19 smithi181 ceph-mon[94189]: osdmap e1454: 12 total, 12 up, 12 in 2023-03-10T01:01:19.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:19 smithi038 ceph-mon[92422]: pgmap v19811: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 594 KiB/s rd, 593 KiB/s wr, 12 op/s 2023-03-10T01:01:19.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:19 smithi038 ceph-mon[92422]: osdmap e1454: 12 total, 12 up, 12 in 2023-03-10T01:01:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:01:20.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:01:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:01:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:01:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:01:20.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:01:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36191.501093 secs 2023-03-10T01:01:21.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:21 smithi122 ceph-mon[93869]: pgmap v19813: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 338 KiB/s wr, 9 op/s 2023-03-10T01:01:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36191.501093 secs 2023-03-10T01:01:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:21 smithi181 ceph-mon[94189]: pgmap v19813: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 338 KiB/s wr, 9 op/s 2023-03-10T01:01:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36191.501093 secs 2023-03-10T01:01:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:21 smithi038 ceph-mon[92422]: pgmap v19813: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 338 KiB/s wr, 9 op/s 2023-03-10T01:01:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36655.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:01:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:23 smithi122 ceph-mon[93869]: pgmap v19814: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 338 KiB/s wr, 9 op/s 2023-03-10T01:01:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36193.961797 secs 2023-03-10T01:01:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:23 smithi181 ceph-mon[94189]: pgmap v19814: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 338 KiB/s wr, 9 op/s 2023-03-10T01:01:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36193.961797 secs 2023-03-10T01:01:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:23 smithi038 ceph-mon[92422]: pgmap v19814: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 338 KiB/s wr, 9 op/s 2023-03-10T01:01:24.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36193.961797 secs 2023-03-10T01:01:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:25 smithi122 ceph-mon[93869]: pgmap v19815: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 139 KiB/s wr, 6 op/s 2023-03-10T01:01:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:25 smithi181 ceph-mon[94189]: pgmap v19815: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 139 KiB/s wr, 6 op/s 2023-03-10T01:01:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:25 smithi038 ceph-mon[92422]: pgmap v19815: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 139 KiB/s wr, 6 op/s 2023-03-10T01:01:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36196.501154 secs 2023-03-10T01:01:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36196.501154 secs 2023-03-10T01:01:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36196.501154 secs 2023-03-10T01:01:27.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:27 smithi122 ceph-mon[93869]: pgmap v19816: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 139 KiB/s wr, 6 op/s 2023-03-10T01:01:27.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:27 smithi181 ceph-mon[94189]: pgmap v19816: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 139 KiB/s wr, 6 op/s 2023-03-10T01:01:27.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:27 smithi038 ceph-mon[92422]: pgmap v19816: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 139 KiB/s wr, 6 op/s 2023-03-10T01:01:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36198.961855 secs 2023-03-10T01:01:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36198.961855 secs 2023-03-10T01:01:28.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36198.961855 secs 2023-03-10T01:01:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36660.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:01:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:29 smithi122 ceph-mon[93869]: pgmap v19817: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-10T01:01:29.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:29 smithi181 ceph-mon[94189]: pgmap v19817: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-10T01:01:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:29 smithi038 ceph-mon[92422]: pgmap v19817: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2023-03-10T01:01:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36201.501213 secs 2023-03-10T01:01:31.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:31 smithi122 ceph-mon[93869]: pgmap v19818: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2023-03-10T01:01:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36201.501213 secs 2023-03-10T01:01:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:31 smithi181 ceph-mon[94189]: pgmap v19818: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2023-03-10T01:01:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36201.501213 secs 2023-03-10T01:01:32.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:31 smithi038 ceph-mon[92422]: pgmap v19818: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2023-03-10T01:01:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36665.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:01:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:33 smithi122 ceph-mon[93869]: pgmap v19819: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-03-10T01:01:33.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36203.961939 secs 2023-03-10T01:01:33.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:33 smithi181 ceph-mon[94189]: pgmap v19819: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-03-10T01:01:33.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36203.961939 secs 2023-03-10T01:01:34.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:33 smithi038 ceph-mon[92422]: pgmap v19819: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-03-10T01:01:34.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36203.961939 secs 2023-03-10T01:01:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:35 smithi122 ceph-mon[93869]: pgmap v19820: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2023-03-10T01:01:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:35 smithi181 ceph-mon[94189]: pgmap v19820: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2023-03-10T01:01:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:35 smithi038 ceph-mon[92422]: pgmap v19820: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2023-03-10T01:01:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36206.501272 secs 2023-03-10T01:01:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36206.501272 secs 2023-03-10T01:01:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36206.501272 secs 2023-03-10T01:01:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:37 smithi122 ceph-mon[93869]: pgmap v19821: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-03-10T01:01:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:37 smithi181 ceph-mon[94189]: pgmap v19821: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-03-10T01:01:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:37 smithi038 ceph-mon[92422]: pgmap v19821: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-03-10T01:01:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36208.962028 secs 2023-03-10T01:01:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36208.962028 secs 2023-03-10T01:01:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36208.962028 secs 2023-03-10T01:01:38.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36670.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:01:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:39 smithi122 ceph-mon[93869]: pgmap v19822: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2023-03-10T01:01:39.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:39 smithi181 ceph-mon[94189]: pgmap v19822: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2023-03-10T01:01:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:39 smithi038 ceph-mon[92422]: pgmap v19822: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2023-03-10T01:01:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36211.501330 secs 2023-03-10T01:01:41.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:41 smithi122 ceph-mon[93869]: pgmap v19823: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2023-03-10T01:01:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36211.501330 secs 2023-03-10T01:01:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:41 smithi181 ceph-mon[94189]: pgmap v19823: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2023-03-10T01:01:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36211.501330 secs 2023-03-10T01:01:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:41 smithi038 ceph-mon[92422]: pgmap v19823: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2023-03-10T01:01:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36675.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:01:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:43 smithi122 ceph-mon[93869]: pgmap v19824: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 682 B/s wr, 7 op/s 2023-03-10T01:01:43.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36213.962116 secs 2023-03-10T01:01:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:43 smithi181 ceph-mon[94189]: pgmap v19824: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 682 B/s wr, 7 op/s 2023-03-10T01:01:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36213.962116 secs 2023-03-10T01:01:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:43 smithi038 ceph-mon[92422]: pgmap v19824: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 682 B/s wr, 7 op/s 2023-03-10T01:01:44.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36213.962116 secs 2023-03-10T01:01:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:45 smithi122 ceph-mon[93869]: pgmap v19825: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 191 KiB/s wr, 7 op/s 2023-03-10T01:01:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:45 smithi181 ceph-mon[94189]: pgmap v19825: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 191 KiB/s wr, 7 op/s 2023-03-10T01:01:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:45 smithi038 ceph-mon[92422]: pgmap v19825: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 191 KiB/s wr, 7 op/s 2023-03-10T01:01:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36216.501389 secs 2023-03-10T01:01:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36216.501389 secs 2023-03-10T01:01:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36216.501389 secs 2023-03-10T01:01:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:47 smithi122 ceph-mon[93869]: pgmap v19826: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 295 KiB/s wr, 8 op/s 2023-03-10T01:01:47.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:47 smithi181 ceph-mon[94189]: pgmap v19826: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 295 KiB/s wr, 8 op/s 2023-03-10T01:01:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:47 smithi038 ceph-mon[92422]: pgmap v19826: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 295 KiB/s wr, 8 op/s 2023-03-10T01:01:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36218.962175 secs 2023-03-10T01:01:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36218.962175 secs 2023-03-10T01:01:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36218.962175 secs 2023-03-10T01:01:48.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36680.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:01:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:01:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:01:49.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:49 smithi122 ceph-mon[93869]: pgmap v19827: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 295 KiB/s wr, 8 op/s 2023-03-10T01:01:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:01:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:01:49.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:49 smithi181 ceph-mon[94189]: pgmap v19827: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 295 KiB/s wr, 8 op/s 2023-03-10T01:01:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:01:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:01:49.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:49 smithi038 ceph-mon[92422]: pgmap v19827: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 295 KiB/s wr, 8 op/s 2023-03-10T01:01:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36221.501475 secs 2023-03-10T01:01:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:51 smithi122 ceph-mon[93869]: pgmap v19828: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 399 KiB/s rd, 397 KiB/s wr, 7 op/s 2023-03-10T01:01:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36221.501475 secs 2023-03-10T01:01:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:51 smithi181 ceph-mon[94189]: pgmap v19828: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 399 KiB/s rd, 397 KiB/s wr, 7 op/s 2023-03-10T01:01:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36221.501475 secs 2023-03-10T01:01:52.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:51 smithi038 ceph-mon[92422]: pgmap v19828: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 399 KiB/s rd, 397 KiB/s wr, 7 op/s 2023-03-10T01:01:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:01:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:01:52.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:01:52.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:01:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:01:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:01:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36685.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:01:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:53 smithi122 ceph-mon[93869]: pgmap v19829: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 635 KiB/s wr, 10 op/s 2023-03-10T01:01:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36223.962299 secs 2023-03-10T01:01:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:53 smithi181 ceph-mon[94189]: pgmap v19829: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 635 KiB/s wr, 10 op/s 2023-03-10T01:01:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36223.962299 secs 2023-03-10T01:01:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:53 smithi038 ceph-mon[92422]: pgmap v19829: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 635 KiB/s wr, 10 op/s 2023-03-10T01:01:54.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36223.962299 secs 2023-03-10T01:01:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:55 smithi122 ceph-mon[93869]: pgmap v19830: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 760 KiB/s rd, 664 KiB/s wr, 9 op/s 2023-03-10T01:01:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:55 smithi181 ceph-mon[94189]: pgmap v19830: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 760 KiB/s rd, 664 KiB/s wr, 9 op/s 2023-03-10T01:01:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:55 smithi038 ceph-mon[92422]: pgmap v19830: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 760 KiB/s rd, 664 KiB/s wr, 9 op/s 2023-03-10T01:01:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36226.501571 secs 2023-03-10T01:01:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:01:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:01:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36226.501571 secs 2023-03-10T01:01:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:01:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:01:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36226.501571 secs 2023-03-10T01:01:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:01:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:01:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:01:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:01:57.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:57 smithi122 ceph-mon[93869]: pgmap v19831: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 633 KiB/s wr, 11 op/s 2023-03-10T01:01:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:01:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:01:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:57 smithi181 ceph-mon[94189]: pgmap v19831: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 633 KiB/s wr, 11 op/s 2023-03-10T01:01:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:01:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:01:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:57 smithi038 ceph-mon[92422]: pgmap v19831: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 633 KiB/s wr, 11 op/s 2023-03-10T01:01:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:58 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:01:58.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36228.962359 secs 2023-03-10T01:01:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:58 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:01:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36228.962359 secs 2023-03-10T01:01:58.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:58 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:01:58.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36228.962359 secs 2023-03-10T01:01:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36690.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:01:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:59 smithi122 ceph-mon[93869]: pgmap v19832: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 660 KiB/s rd, 564 KiB/s wr, 11 op/s 2023-03-10T01:01:59.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:01:59.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:01:59.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:01:59.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:01:59.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:01:59.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:01:59.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:01:59.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:01:59.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:01:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:01:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:59 smithi181 ceph-mon[94189]: pgmap v19832: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 660 KiB/s rd, 564 KiB/s wr, 11 op/s 2023-03-10T01:01:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:01:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:01:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:01:59.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:01:59.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:01:59.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:01:59.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:01:59.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:01:59.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:01:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:01:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:59 smithi038 ceph-mon[92422]: pgmap v19832: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 660 KiB/s rd, 564 KiB/s wr, 11 op/s 2023-03-10T01:02:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:02:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:02:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:02:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:02:00.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:02:00.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:02:00.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:02:00.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:02:00.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:01:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:02:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:02:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:02:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:02:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:02:01.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:02:01.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:02:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36231.501630 secs 2023-03-10T01:02:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:01 smithi122 ceph-mon[93869]: pgmap v19833: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 752 KiB/s rd, 644 KiB/s wr, 10 op/s 2023-03-10T01:02:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:02:01.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:02:01.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36231.501630 secs 2023-03-10T01:02:01.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:01 smithi181 ceph-mon[94189]: pgmap v19833: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 752 KiB/s rd, 644 KiB/s wr, 10 op/s 2023-03-10T01:02:01.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:02:01.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:02:02.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36231.501630 secs 2023-03-10T01:02:02.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:01 smithi038 ceph-mon[92422]: pgmap v19833: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 752 KiB/s rd, 644 KiB/s wr, 10 op/s 2023-03-10T01:02:02.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:02:02.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:02:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36695.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:02:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:03 smithi122 ceph-mon[93869]: pgmap v19834: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 900 KiB/s rd, 791 KiB/s wr, 13 op/s 2023-03-10T01:02:03.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36233.962470 secs 2023-03-10T01:02:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:03 smithi181 ceph-mon[94189]: pgmap v19834: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 900 KiB/s rd, 791 KiB/s wr, 13 op/s 2023-03-10T01:02:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36233.962470 secs 2023-03-10T01:02:04.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:03 smithi038 ceph-mon[92422]: pgmap v19834: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 900 KiB/s rd, 791 KiB/s wr, 13 op/s 2023-03-10T01:02:04.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36233.962470 secs 2023-03-10T01:02:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:05 smithi122 ceph-mon[93869]: pgmap v19835: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 623 KiB/s rd, 609 KiB/s wr, 13 op/s 2023-03-10T01:02:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:05 smithi181 ceph-mon[94189]: pgmap v19835: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 623 KiB/s rd, 609 KiB/s wr, 13 op/s 2023-03-10T01:02:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:05 smithi038 ceph-mon[92422]: pgmap v19835: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 623 KiB/s rd, 609 KiB/s wr, 13 op/s 2023-03-10T01:02:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36236.501690 secs 2023-03-10T01:02:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36236.501690 secs 2023-03-10T01:02:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36236.501690 secs 2023-03-10T01:02:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:07 smithi122 ceph-mon[93869]: pgmap v19836: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 689 KiB/s rd, 694 KiB/s wr, 17 op/s 2023-03-10T01:02:07.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:07 smithi181 ceph-mon[94189]: pgmap v19836: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 689 KiB/s rd, 694 KiB/s wr, 17 op/s 2023-03-10T01:02:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:07 smithi038 ceph-mon[92422]: pgmap v19836: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 689 KiB/s rd, 694 KiB/s wr, 17 op/s 2023-03-10T01:02:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36238.962528 secs 2023-03-10T01:02:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36238.962528 secs 2023-03-10T01:02:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36238.962528 secs 2023-03-10T01:02:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36700.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:02:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:09 smithi122 ceph-mon[93869]: pgmap v19837: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 562 KiB/s rd, 586 KiB/s wr, 25 op/s 2023-03-10T01:02:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:09 smithi181 ceph-mon[94189]: pgmap v19837: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 562 KiB/s rd, 586 KiB/s wr, 25 op/s 2023-03-10T01:02:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:09 smithi038 ceph-mon[92422]: pgmap v19837: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 562 KiB/s rd, 586 KiB/s wr, 25 op/s 2023-03-10T01:02:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36241.501750 secs 2023-03-10T01:02:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:11 smithi122 ceph-mon[93869]: osdmap e1455: 12 total, 12 up, 12 in 2023-03-10T01:02:11.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:11 smithi122 ceph-mon[93869]: pgmap v19839: 129 pgs: 1 active+clean+snaptrim_wait, 128 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 866 KiB/s rd, 907 KiB/s wr, 29 op/s 2023-03-10T01:02:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36241.501750 secs 2023-03-10T01:02:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:11 smithi181 ceph-mon[94189]: osdmap e1455: 12 total, 12 up, 12 in 2023-03-10T01:02:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:11 smithi181 ceph-mon[94189]: pgmap v19839: 129 pgs: 1 active+clean+snaptrim_wait, 128 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 866 KiB/s rd, 907 KiB/s wr, 29 op/s 2023-03-10T01:02:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36241.501750 secs 2023-03-10T01:02:11.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:11 smithi038 ceph-mon[92422]: osdmap e1455: 12 total, 12 up, 12 in 2023-03-10T01:02:11.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:11 smithi038 ceph-mon[92422]: pgmap v19839: 129 pgs: 1 active+clean+snaptrim_wait, 128 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 866 KiB/s rd, 907 KiB/s wr, 29 op/s 2023-03-10T01:02:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:02:12.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:02:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:02:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:02:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:02:12.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:02:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:02:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:02:13.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:13 smithi122 ceph-mon[93869]: pgmap v19840: 129 pgs: 1 active+clean+snaptrim_wait, 128 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 668 KiB/s rd, 708 KiB/s wr, 29 op/s 2023-03-10T01:02:13.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36243.962595 secs 2023-03-10T01:02:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:02:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:02:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:13 smithi181 ceph-mon[94189]: pgmap v19840: 129 pgs: 1 active+clean+snaptrim_wait, 128 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 668 KiB/s rd, 708 KiB/s wr, 29 op/s 2023-03-10T01:02:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36243.962595 secs 2023-03-10T01:02:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:02:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:02:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:13 smithi038 ceph-mon[92422]: pgmap v19840: 129 pgs: 1 active+clean+snaptrim_wait, 128 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 668 KiB/s rd, 708 KiB/s wr, 29 op/s 2023-03-10T01:02:13.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36243.962595 secs 2023-03-10T01:02:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36705.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:02:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:15 smithi122 ceph-mon[93869]: pgmap v19841: 129 pgs: 1 active+clean+snaptrim_wait, 128 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 669 KiB/s rd, 711 KiB/s wr, 28 op/s 2023-03-10T01:02:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:15 smithi181 ceph-mon[94189]: pgmap v19841: 129 pgs: 1 active+clean+snaptrim_wait, 128 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 669 KiB/s rd, 711 KiB/s wr, 28 op/s 2023-03-10T01:02:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:15 smithi038 ceph-mon[92422]: pgmap v19841: 129 pgs: 1 active+clean+snaptrim_wait, 128 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 669 KiB/s rd, 711 KiB/s wr, 28 op/s 2023-03-10T01:02:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36246.501810 secs 2023-03-10T01:02:16.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:02:16.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:02:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36246.501810 secs 2023-03-10T01:02:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:02:16.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:02:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36246.501810 secs 2023-03-10T01:02:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:02:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:02:17.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:02:17.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:02:17.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:17 smithi122 ceph-mon[93869]: pgmap v19842: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 557 KiB/s rd, 573 KiB/s wr, 24 op/s 2023-03-10T01:02:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:02:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:02:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:17 smithi181 ceph-mon[94189]: pgmap v19842: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 557 KiB/s rd, 573 KiB/s wr, 24 op/s 2023-03-10T01:02:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:02:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:02:17.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:17 smithi038 ceph-mon[92422]: pgmap v19842: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 557 KiB/s rd, 573 KiB/s wr, 24 op/s 2023-03-10T01:02:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36248.962679 secs 2023-03-10T01:02:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36248.962679 secs 2023-03-10T01:02:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36248.962679 secs 2023-03-10T01:02:18.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36710.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:02:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:19 smithi122 ceph-mon[93869]: pgmap v19843: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 653 KiB/s rd, 651 KiB/s wr, 11 op/s 2023-03-10T01:02:19.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:19 smithi122 ceph-mon[93869]: osdmap e1456: 12 total, 12 up, 12 in 2023-03-10T01:02:19.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:19 smithi181 ceph-mon[94189]: pgmap v19843: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 653 KiB/s rd, 651 KiB/s wr, 11 op/s 2023-03-10T01:02:19.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:19 smithi181 ceph-mon[94189]: osdmap e1456: 12 total, 12 up, 12 in 2023-03-10T01:02:19.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:19 smithi038 ceph-mon[92422]: pgmap v19843: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 653 KiB/s rd, 651 KiB/s wr, 11 op/s 2023-03-10T01:02:19.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:19 smithi038 ceph-mon[92422]: osdmap e1456: 12 total, 12 up, 12 in 2023-03-10T01:02:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36251.501864 secs 2023-03-10T01:02:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:21 smithi122 ceph-mon[93869]: pgmap v19845: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 310 KiB/s wr, 9 op/s 2023-03-10T01:02:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36251.501864 secs 2023-03-10T01:02:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:21 smithi181 ceph-mon[94189]: pgmap v19845: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 310 KiB/s wr, 9 op/s 2023-03-10T01:02:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36251.501864 secs 2023-03-10T01:02:22.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:21 smithi038 ceph-mon[92422]: pgmap v19845: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 310 KiB/s wr, 9 op/s 2023-03-10T01:02:23.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36715.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:02:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:23 smithi122 ceph-mon[93869]: pgmap v19846: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 321 KiB/s rd, 319 KiB/s wr, 8 op/s 2023-03-10T01:02:23.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36253.962773 secs 2023-03-10T01:02:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:23 smithi181 ceph-mon[94189]: pgmap v19846: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 321 KiB/s rd, 319 KiB/s wr, 8 op/s 2023-03-10T01:02:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36253.962773 secs 2023-03-10T01:02:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:23 smithi038 ceph-mon[92422]: pgmap v19846: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 321 KiB/s rd, 319 KiB/s wr, 8 op/s 2023-03-10T01:02:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36253.962773 secs 2023-03-10T01:02:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:25 smithi122 ceph-mon[93869]: pgmap v19847: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 249 KiB/s wr, 7 op/s 2023-03-10T01:02:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:25 smithi181 ceph-mon[94189]: pgmap v19847: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 249 KiB/s wr, 7 op/s 2023-03-10T01:02:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:25 smithi038 ceph-mon[92422]: pgmap v19847: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 249 KiB/s wr, 7 op/s 2023-03-10T01:02:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36256.501922 secs 2023-03-10T01:02:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36256.501922 secs 2023-03-10T01:02:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36256.501922 secs 2023-03-10T01:02:27.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:27 smithi122 ceph-mon[93869]: pgmap v19848: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 250 KiB/s wr, 8 op/s 2023-03-10T01:02:27.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:27 smithi181 ceph-mon[94189]: pgmap v19848: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 250 KiB/s wr, 8 op/s 2023-03-10T01:02:27.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:27 smithi038 ceph-mon[92422]: pgmap v19848: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 250 KiB/s wr, 8 op/s 2023-03-10T01:02:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36258.962902 secs 2023-03-10T01:02:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36258.962902 secs 2023-03-10T01:02:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36258.962902 secs 2023-03-10T01:02:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36720.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:02:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:29 smithi122 ceph-mon[93869]: pgmap v19849: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 110 KiB/s wr, 7 op/s 2023-03-10T01:02:29.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:29 smithi181 ceph-mon[94189]: pgmap v19849: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 110 KiB/s wr, 7 op/s 2023-03-10T01:02:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:29 smithi038 ceph-mon[92422]: pgmap v19849: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 110 KiB/s wr, 7 op/s 2023-03-10T01:02:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36261.501980 secs 2023-03-10T01:02:31.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:31 smithi122 ceph-mon[93869]: pgmap v19850: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 97 KiB/s wr, 6 op/s 2023-03-10T01:02:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36261.501980 secs 2023-03-10T01:02:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:31 smithi181 ceph-mon[94189]: pgmap v19850: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 97 KiB/s wr, 6 op/s 2023-03-10T01:02:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36261.501980 secs 2023-03-10T01:02:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:31 smithi038 ceph-mon[92422]: pgmap v19850: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 97 KiB/s wr, 6 op/s 2023-03-10T01:02:33.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36725.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:02:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36263.963014 secs 2023-03-10T01:02:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:33 smithi122 ceph-mon[93869]: pgmap v19851: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 KiB/s wr, 8 op/s 2023-03-10T01:02:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36263.963014 secs 2023-03-10T01:02:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:33 smithi181 ceph-mon[94189]: pgmap v19851: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 KiB/s wr, 8 op/s 2023-03-10T01:02:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36263.963014 secs 2023-03-10T01:02:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:33 smithi038 ceph-mon[92422]: pgmap v19851: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 KiB/s wr, 8 op/s 2023-03-10T01:02:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:35 smithi122 ceph-mon[93869]: pgmap v19852: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 99 KiB/s wr, 7 op/s 2023-03-10T01:02:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:35 smithi181 ceph-mon[94189]: pgmap v19852: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 99 KiB/s wr, 7 op/s 2023-03-10T01:02:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:35 smithi038 ceph-mon[92422]: pgmap v19852: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 99 KiB/s wr, 7 op/s 2023-03-10T01:02:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36266.502041 secs 2023-03-10T01:02:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36266.502041 secs 2023-03-10T01:02:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36266.502041 secs 2023-03-10T01:02:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:37 smithi122 ceph-mon[93869]: pgmap v19853: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 99 KiB/s wr, 8 op/s 2023-03-10T01:02:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:37 smithi181 ceph-mon[94189]: pgmap v19853: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 99 KiB/s wr, 8 op/s 2023-03-10T01:02:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:37 smithi038 ceph-mon[92422]: pgmap v19853: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 99 KiB/s wr, 8 op/s 2023-03-10T01:02:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36268.963095 secs 2023-03-10T01:02:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36268.963095 secs 2023-03-10T01:02:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36268.963095 secs 2023-03-10T01:02:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36730.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:02:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:39 smithi122 ceph-mon[93869]: pgmap v19854: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 149 KiB/s wr, 8 op/s 2023-03-10T01:02:39.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:39 smithi181 ceph-mon[94189]: pgmap v19854: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 149 KiB/s wr, 8 op/s 2023-03-10T01:02:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:39 smithi038 ceph-mon[92422]: pgmap v19854: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 149 KiB/s wr, 8 op/s 2023-03-10T01:02:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36271.502098 secs 2023-03-10T01:02:41.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:41 smithi122 ceph-mon[93869]: pgmap v19855: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 149 KiB/s wr, 7 op/s 2023-03-10T01:02:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36271.502098 secs 2023-03-10T01:02:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:41 smithi181 ceph-mon[94189]: pgmap v19855: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 149 KiB/s wr, 7 op/s 2023-03-10T01:02:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36271.502098 secs 2023-03-10T01:02:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:41 smithi038 ceph-mon[92422]: pgmap v19855: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 149 KiB/s wr, 7 op/s 2023-03-10T01:02:43.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36735.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:02:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36273.963160 secs 2023-03-10T01:02:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:43 smithi122 ceph-mon[93869]: pgmap v19856: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 149 KiB/s wr, 9 op/s 2023-03-10T01:02:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36273.963160 secs 2023-03-10T01:02:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:43 smithi181 ceph-mon[94189]: pgmap v19856: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 149 KiB/s wr, 9 op/s 2023-03-10T01:02:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36273.963160 secs 2023-03-10T01:02:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:43 smithi038 ceph-mon[92422]: pgmap v19856: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 149 KiB/s wr, 9 op/s 2023-03-10T01:02:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:45 smithi122 ceph-mon[93869]: pgmap v19857: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 149 KiB/s wr, 7 op/s 2023-03-10T01:02:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:45 smithi181 ceph-mon[94189]: pgmap v19857: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 149 KiB/s wr, 7 op/s 2023-03-10T01:02:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:45 smithi038 ceph-mon[92422]: pgmap v19857: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 149 KiB/s wr, 7 op/s 2023-03-10T01:02:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36276.502174 secs 2023-03-10T01:02:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36276.502174 secs 2023-03-10T01:02:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36276.502174 secs 2023-03-10T01:02:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:47 smithi122 ceph-mon[93869]: pgmap v19858: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 50 KiB/s wr, 8 op/s 2023-03-10T01:02:47.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:47 smithi181 ceph-mon[94189]: pgmap v19858: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 50 KiB/s wr, 8 op/s 2023-03-10T01:02:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:47 smithi038 ceph-mon[92422]: pgmap v19858: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 50 KiB/s wr, 8 op/s 2023-03-10T01:02:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36740.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:02:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:02:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:02:49.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:49 smithi122 ceph-mon[93869]: pgmap v19859: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 50 KiB/s wr, 7 op/s 2023-03-10T01:02:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:02:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:02:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:49 smithi181 ceph-mon[94189]: pgmap v19859: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 50 KiB/s wr, 7 op/s 2023-03-10T01:02:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:02:49.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:02:49.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:49 smithi038 ceph-mon[92422]: pgmap v19859: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 50 KiB/s wr, 7 op/s 2023-03-10T01:02:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:50 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:02:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:50 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:02:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:50 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:02:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36281.502231 secs 2023-03-10T01:02:51.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:51 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:02:51.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:02:51.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:02:51.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:51 smithi122 ceph-mon[93869]: pgmap v19860: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 823 KiB/s rd, 436 KiB/s wr, 7 op/s 2023-03-10T01:02:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36281.502231 secs 2023-03-10T01:02:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:51 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:02:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:02:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:02:51.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:51 smithi181 ceph-mon[94189]: pgmap v19860: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 823 KiB/s rd, 436 KiB/s wr, 7 op/s 2023-03-10T01:02:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36281.502231 secs 2023-03-10T01:02:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:51 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:02:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:02:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:02:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:51 smithi038 ceph-mon[92422]: pgmap v19860: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 823 KiB/s rd, 436 KiB/s wr, 7 op/s 2023-03-10T01:02:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:02:52.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:02:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:02:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:02:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:02:52.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:02:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:53 smithi122 ceph-mon[93869]: pgmap v19861: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 850 KiB/s rd, 461 KiB/s wr, 9 op/s 2023-03-10T01:02:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:53 smithi181 ceph-mon[94189]: pgmap v19861: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 850 KiB/s rd, 461 KiB/s wr, 9 op/s 2023-03-10T01:02:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:53 smithi038 ceph-mon[92422]: pgmap v19861: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 850 KiB/s rd, 461 KiB/s wr, 9 op/s 2023-03-10T01:02:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36745.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:02:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:55 smithi122 ceph-mon[93869]: pgmap v19862: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 949 KiB/s rd, 465 KiB/s wr, 8 op/s 2023-03-10T01:02:55.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:55 smithi181 ceph-mon[94189]: pgmap v19862: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 949 KiB/s rd, 465 KiB/s wr, 8 op/s 2023-03-10T01:02:55.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:55 smithi038 ceph-mon[92422]: pgmap v19862: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 949 KiB/s rd, 465 KiB/s wr, 8 op/s 2023-03-10T01:02:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36286.502305 secs 2023-03-10T01:02:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:02:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:02:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36286.502305 secs 2023-03-10T01:02:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:02:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:02:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36286.502305 secs 2023-03-10T01:02:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:02:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:02:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:02:57.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:02:57.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:57 smithi122 ceph-mon[93869]: pgmap v19863: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 615 KiB/s wr, 11 op/s 2023-03-10T01:02:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:02:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:02:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:57 smithi181 ceph-mon[94189]: pgmap v19863: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 615 KiB/s wr, 11 op/s 2023-03-10T01:02:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:02:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:02:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:57 smithi038 ceph-mon[92422]: pgmap v19863: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 615 KiB/s wr, 11 op/s 2023-03-10T01:02:58.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:58 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 36288.963371 secs 2023-03-10T01:02:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:58 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 36288.963371 secs 2023-03-10T01:02:58.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:58 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 36288.963371 secs 2023-03-10T01:02:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36750.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:02:59.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:59 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:02:59.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:02:59.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:02:59.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:59 smithi122 ceph-mon[93869]: pgmap v19864: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 645 KiB/s wr, 11 op/s 2023-03-10T01:02:59.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:02:59 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:02:59.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:59 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:02:59.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:02:59.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:02:59.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:59 smithi181 ceph-mon[94189]: pgmap v19864: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 645 KiB/s wr, 11 op/s 2023-03-10T01:02:59.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:02:59 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:02:59.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:59 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:02:59.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:02:59.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:02:59.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:59 smithi038 ceph-mon[92422]: pgmap v19864: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 645 KiB/s wr, 11 op/s 2023-03-10T01:02:59.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:02:59 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:03:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:03:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:03:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:03:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:03:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:03:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:03:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36291.502364 secs 2023-03-10T01:03:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:03:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:03:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:01 smithi122 ceph-mon[93869]: pgmap v19865: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 645 KiB/s wr, 11 op/s 2023-03-10T01:03:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36291.502364 secs 2023-03-10T01:03:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:03:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:03:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:01 smithi181 ceph-mon[94189]: pgmap v19865: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 645 KiB/s wr, 11 op/s 2023-03-10T01:03:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36291.502364 secs 2023-03-10T01:03:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:03:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:03:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:01 smithi038 ceph-mon[92422]: pgmap v19865: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 645 KiB/s wr, 11 op/s 2023-03-10T01:03:03.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36755.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:03:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:03 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 36293.963474 secs 2023-03-10T01:03:03.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:03 smithi122 ceph-mon[93869]: pgmap v19866: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 685 KiB/s wr, 16 op/s 2023-03-10T01:03:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:03 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 36293.963474 secs 2023-03-10T01:03:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:03 smithi181 ceph-mon[94189]: pgmap v19866: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 685 KiB/s wr, 16 op/s 2023-03-10T01:03:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:03 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 36293.963474 secs 2023-03-10T01:03:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:03 smithi038 ceph-mon[92422]: pgmap v19866: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 685 KiB/s wr, 16 op/s 2023-03-10T01:03:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:05 smithi122 ceph-mon[93869]: pgmap v19867: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 670 KiB/s wr, 25 op/s 2023-03-10T01:03:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:05 smithi181 ceph-mon[94189]: pgmap v19867: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 670 KiB/s wr, 25 op/s 2023-03-10T01:03:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:05 smithi038 ceph-mon[92422]: pgmap v19867: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 670 KiB/s wr, 25 op/s 2023-03-10T01:03:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36296.502449 secs 2023-03-10T01:03:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36296.502449 secs 2023-03-10T01:03:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36296.502449 secs 2023-03-10T01:03:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:07 smithi122 ceph-mon[93869]: pgmap v19868: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 953 KiB/s wr, 27 op/s 2023-03-10T01:03:07.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:07 smithi181 ceph-mon[94189]: pgmap v19868: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 953 KiB/s wr, 27 op/s 2023-03-10T01:03:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:07 smithi038 ceph-mon[92422]: pgmap v19868: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 953 KiB/s wr, 27 op/s 2023-03-10T01:03:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:03:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:03:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:08 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 36298.963559 secs 2023-03-10T01:03:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:03:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:03:08.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:08 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 36298.963559 secs 2023-03-10T01:03:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:03:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:03:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:08 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 36298.963559 secs 2023-03-10T01:03:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36760.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:03:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:03:09.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:03:09.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:09 smithi122 ceph-mon[93869]: pgmap v19869: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 803 KiB/s wr, 25 op/s 2023-03-10T01:03:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:03:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:03:09.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:09 smithi181 ceph-mon[94189]: pgmap v19869: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 803 KiB/s wr, 25 op/s 2023-03-10T01:03:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:03:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:03:09.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:09 smithi038 ceph-mon[92422]: pgmap v19869: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 803 KiB/s wr, 25 op/s 2023-03-10T01:03:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36301.502508 secs 2023-03-10T01:03:11.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:11 smithi122 ceph-mon[93869]: pgmap v19870: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 773 KiB/s wr, 24 op/s 2023-03-10T01:03:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36301.502508 secs 2023-03-10T01:03:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:11 smithi181 ceph-mon[94189]: pgmap v19870: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 773 KiB/s wr, 24 op/s 2023-03-10T01:03:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36301.502508 secs 2023-03-10T01:03:12.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:11 smithi038 ceph-mon[92422]: pgmap v19870: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 773 KiB/s wr, 24 op/s 2023-03-10T01:03:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36765.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:03:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:13 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 36303.963616 secs 2023-03-10T01:03:13.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:13 smithi122 ceph-mon[93869]: pgmap v19871: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1020 KiB/s rd, 1.0 MiB/s wr, 26 op/s 2023-03-10T01:03:13.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:03:13.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:03:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:13 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 36303.963616 secs 2023-03-10T01:03:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:13 smithi181 ceph-mon[94189]: pgmap v19871: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1020 KiB/s rd, 1.0 MiB/s wr, 26 op/s 2023-03-10T01:03:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:03:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:03:14.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:13 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 36303.963616 secs 2023-03-10T01:03:14.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:13 smithi038 ceph-mon[92422]: pgmap v19871: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1020 KiB/s rd, 1.0 MiB/s wr, 26 op/s 2023-03-10T01:03:14.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:03:14.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:03:14.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:14 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:03:14.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:14 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:03:14.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:14 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:03:14.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:14 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:03:14.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:14 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:03:14.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:14 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:03:14.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:14 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:03:14.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:14 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:03:14.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:14 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:03:14.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:14 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:03:14.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:14 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:03:14.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:14 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:03:14.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:14 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:03:14.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:14 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:03:14.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:14 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:03:14.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:14 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:03:14.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:14 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:03:14.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:14 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:03:14.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:14 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:03:14.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:14 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:03:14.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:14 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:03:14.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:14 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:03:14.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:14 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:03:14.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:14 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:03:14.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:14 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:03:14.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:14 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:03:14.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:14 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:03:14.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:14 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:03:14.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:14 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:03:14.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:14 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:03:14.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:14 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:03:14.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:14 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:03:14.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:14 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:03:14.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:14 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:03:14.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:14 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:03:14.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:14 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:03:14.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:14 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:03:15.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:14 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:03:15.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:14 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:03:15.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:14 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:03:15.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:14 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:03:15.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:14 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:03:15.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:14 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:03:15.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:14 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:03:15.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:14 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:03:15.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:14 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:03:15.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:14 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:03:15.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:14 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:03:15.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:14 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:03:15.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:14 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:03:15.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:14 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:03:15.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:14 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:03:15.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:14 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:03:15.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:14 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:03:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:15 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-10T01:03:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:15 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-10T01:03:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:15 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-10T01:03:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:15 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-10T01:03:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:15 smithi122 ceph-mon[93869]: pgmap v19872: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 608 KiB/s rd, 557 KiB/s wr, 20 op/s 2023-03-10T01:03:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:15 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:03:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:15 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:03:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:15 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-10T01:03:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:15 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-10T01:03:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:15 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-10T01:03:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:15 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-10T01:03:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:15 smithi181 ceph-mon[94189]: pgmap v19872: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 608 KiB/s rd, 557 KiB/s wr, 20 op/s 2023-03-10T01:03:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:15 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:03:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:15 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:03:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:15 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-10T01:03:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:15 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-10T01:03:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:15 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-10T01:03:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:15 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-10T01:03:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:15 smithi038 ceph-mon[92422]: pgmap v19872: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 608 KiB/s rd, 557 KiB/s wr, 20 op/s 2023-03-10T01:03:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:15 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:03:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:15 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:03:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36306.502592 secs 2023-03-10T01:03:16.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36306.502592 secs 2023-03-10T01:03:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36306.502592 secs 2023-03-10T01:03:17.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:17 smithi122 ceph-mon[93869]: pgmap v19873: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 605 KiB/s wr, 13 op/s 2023-03-10T01:03:17.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:03:17.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:03:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:17 smithi181 ceph-mon[94189]: pgmap v19873: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 605 KiB/s wr, 13 op/s 2023-03-10T01:03:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:03:17.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:03:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:17 smithi038 ceph-mon[92422]: pgmap v19873: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 605 KiB/s wr, 13 op/s 2023-03-10T01:03:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:03:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:03:18.801 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36770.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:03:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:18 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 36308.963699 secs 2023-03-10T01:03:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:18 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 36308.963699 secs 2023-03-10T01:03:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:18 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 36308.963699 secs 2023-03-10T01:03:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:19 smithi122 ceph-mon[93869]: pgmap v19874: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 814 KiB/s wr, 14 op/s 2023-03-10T01:03:19.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:19 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:03:19.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:19 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:03:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:19 smithi181 ceph-mon[94189]: pgmap v19874: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 814 KiB/s wr, 14 op/s 2023-03-10T01:03:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:19 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:03:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:19 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:03:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:19 smithi038 ceph-mon[92422]: pgmap v19874: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 814 KiB/s wr, 14 op/s 2023-03-10T01:03:20.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:19 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:03:20.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:19 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:03:21.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36311.502667 secs 2023-03-10T01:03:21.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:21 smithi122 ceph-mon[93869]: osdmap e1457: 12 total, 12 up, 12 in 2023-03-10T01:03:21.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:21 smithi122 ceph-mon[93869]: pgmap v19876: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 782 KiB/s rd, 977 KiB/s wr, 15 op/s 2023-03-10T01:03:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36311.502667 secs 2023-03-10T01:03:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:21 smithi181 ceph-mon[94189]: osdmap e1457: 12 total, 12 up, 12 in 2023-03-10T01:03:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:21 smithi181 ceph-mon[94189]: pgmap v19876: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 782 KiB/s rd, 977 KiB/s wr, 15 op/s 2023-03-10T01:03:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36311.502667 secs 2023-03-10T01:03:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:21 smithi038 ceph-mon[92422]: osdmap e1457: 12 total, 12 up, 12 in 2023-03-10T01:03:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:21 smithi038 ceph-mon[92422]: pgmap v19876: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 782 KiB/s rd, 977 KiB/s wr, 15 op/s 2023-03-10T01:03:23.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36775.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:03:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:23 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 36313.963757 secs 2023-03-10T01:03:23.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:23 smithi122 ceph-mon[93869]: pgmap v19877: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 668 KiB/s wr, 14 op/s 2023-03-10T01:03:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:23 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 36313.963757 secs 2023-03-10T01:03:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:23 smithi181 ceph-mon[94189]: pgmap v19877: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 668 KiB/s wr, 14 op/s 2023-03-10T01:03:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:23 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 36313.963757 secs 2023-03-10T01:03:24.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:23 smithi038 ceph-mon[92422]: pgmap v19877: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 668 KiB/s wr, 14 op/s 2023-03-10T01:03:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:25 smithi122 ceph-mon[93869]: pgmap v19878: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 813 KiB/s rd, 947 KiB/s wr, 15 op/s 2023-03-10T01:03:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:25 smithi181 ceph-mon[94189]: pgmap v19878: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 813 KiB/s rd, 947 KiB/s wr, 15 op/s 2023-03-10T01:03:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:25 smithi038 ceph-mon[92422]: pgmap v19878: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 813 KiB/s rd, 947 KiB/s wr, 15 op/s 2023-03-10T01:03:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36316.502767 secs 2023-03-10T01:03:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36316.502767 secs 2023-03-10T01:03:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36316.502767 secs 2023-03-10T01:03:27.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:27 smithi122 ceph-mon[93869]: pgmap v19879: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 939 KiB/s wr, 13 op/s 2023-03-10T01:03:27.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:27 smithi181 ceph-mon[94189]: pgmap v19879: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 939 KiB/s wr, 13 op/s 2023-03-10T01:03:27.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:27 smithi038 ceph-mon[92422]: pgmap v19879: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 939 KiB/s wr, 13 op/s 2023-03-10T01:03:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:28 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:03:28.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:28 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:03:28.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:28 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 36318.963815 secs 2023-03-10T01:03:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:28 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:03:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:28 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:03:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:28 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 36318.963815 secs 2023-03-10T01:03:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:28 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:03:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:28 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:03:28.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:28 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 36318.963815 secs 2023-03-10T01:03:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36780.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:03:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:29 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:03:29.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:29 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:03:29.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:29 smithi122 ceph-mon[93869]: pgmap v19880: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 378 KiB/s wr, 9 op/s 2023-03-10T01:03:29.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:29 smithi122 ceph-mon[93869]: osdmap e1458: 12 total, 12 up, 12 in 2023-03-10T01:03:29.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:29 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:03:29.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:29 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:03:29.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:29 smithi181 ceph-mon[94189]: pgmap v19880: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 378 KiB/s wr, 9 op/s 2023-03-10T01:03:29.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:29 smithi181 ceph-mon[94189]: osdmap e1458: 12 total, 12 up, 12 in 2023-03-10T01:03:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:29 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:03:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:29 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:03:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:29 smithi038 ceph-mon[92422]: pgmap v19880: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 378 KiB/s wr, 9 op/s 2023-03-10T01:03:29.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:29 smithi038 ceph-mon[92422]: osdmap e1458: 12 total, 12 up, 12 in 2023-03-10T01:03:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36321.502834 secs 2023-03-10T01:03:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:31 smithi122 ceph-mon[93869]: pgmap v19882: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 899 KiB/s rd, 726 KiB/s wr, 10 op/s 2023-03-10T01:03:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36321.502834 secs 2023-03-10T01:03:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:31 smithi181 ceph-mon[94189]: pgmap v19882: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 899 KiB/s rd, 726 KiB/s wr, 10 op/s 2023-03-10T01:03:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36321.502834 secs 2023-03-10T01:03:32.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:31 smithi038 ceph-mon[92422]: pgmap v19882: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 899 KiB/s rd, 726 KiB/s wr, 10 op/s 2023-03-10T01:03:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36785.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:03:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36323.963873 secs 2023-03-10T01:03:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:33 smithi122 ceph-mon[93869]: pgmap v19883: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1006 KiB/s rd, 834 KiB/s wr, 10 op/s 2023-03-10T01:03:33.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:33 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:03:33.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:33 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:03:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36323.963873 secs 2023-03-10T01:03:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:33 smithi181 ceph-mon[94189]: pgmap v19883: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1006 KiB/s rd, 834 KiB/s wr, 10 op/s 2023-03-10T01:03:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:33 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:03:33.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:33 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:03:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36323.963873 secs 2023-03-10T01:03:34.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:33 smithi038 ceph-mon[92422]: pgmap v19883: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1006 KiB/s rd, 834 KiB/s wr, 10 op/s 2023-03-10T01:03:34.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:33 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:03:34.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:33 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:03:34.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:34 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:03:34.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:34 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:03:34.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:34 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:03:34.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:34 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:03:34.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:34 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:03:35.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:34 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:03:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:35 smithi122 ceph-mon[93869]: pgmap v19884: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 666 KiB/s rd, 555 KiB/s wr, 9 op/s 2023-03-10T01:03:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:35 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:03:35.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:35 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:03:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:35 smithi181 ceph-mon[94189]: pgmap v19884: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 666 KiB/s rd, 555 KiB/s wr, 9 op/s 2023-03-10T01:03:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:35 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:03:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:35 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:03:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:35 smithi038 ceph-mon[92422]: pgmap v19884: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 666 KiB/s rd, 555 KiB/s wr, 9 op/s 2023-03-10T01:03:36.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:35 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:03:36.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:35 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:03:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36326.502920 secs 2023-03-10T01:03:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36326.502920 secs 2023-03-10T01:03:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36326.502920 secs 2023-03-10T01:03:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:37 smithi122 ceph-mon[93869]: pgmap v19885: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 774 KiB/s rd, 663 KiB/s wr, 10 op/s 2023-03-10T01:03:37.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:03:37.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:03:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:37 smithi181 ceph-mon[94189]: pgmap v19885: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 774 KiB/s rd, 663 KiB/s wr, 10 op/s 2023-03-10T01:03:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:03:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:03:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:37 smithi038 ceph-mon[92422]: pgmap v19885: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 774 KiB/s rd, 663 KiB/s wr, 10 op/s 2023-03-10T01:03:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:03:38.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:03:38.801 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36790.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:03:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36328.963961 secs 2023-03-10T01:03:38.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36328.963961 secs 2023-03-10T01:03:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36328.963961 secs 2023-03-10T01:03:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:39 smithi122 ceph-mon[93869]: pgmap v19886: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 633 KiB/s wr, 10 op/s 2023-03-10T01:03:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:39 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:03:39.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:39 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:03:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:39 smithi181 ceph-mon[94189]: pgmap v19886: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 633 KiB/s wr, 10 op/s 2023-03-10T01:03:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:39 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:03:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:39 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:03:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:39 smithi038 ceph-mon[92422]: pgmap v19886: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 633 KiB/s wr, 10 op/s 2023-03-10T01:03:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:39 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:03:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:39 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:03:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36331.502983 secs 2023-03-10T01:03:41.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:41 smithi122 ceph-mon[93869]: pgmap v19887: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 559 KiB/s rd, 557 KiB/s wr, 9 op/s 2023-03-10T01:03:41.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:03:41.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:03:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36331.502983 secs 2023-03-10T01:03:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:41 smithi181 ceph-mon[94189]: pgmap v19887: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 559 KiB/s rd, 557 KiB/s wr, 9 op/s 2023-03-10T01:03:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:03:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:03:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36331.502983 secs 2023-03-10T01:03:42.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:41 smithi038 ceph-mon[92422]: pgmap v19887: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 559 KiB/s rd, 557 KiB/s wr, 9 op/s 2023-03-10T01:03:42.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:03:42.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:03:43.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36795.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:03:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36333.964019 secs 2023-03-10T01:03:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:43 smithi122 ceph-mon[93869]: pgmap v19888: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 589 KiB/s rd, 586 KiB/s wr, 12 op/s 2023-03-10T01:03:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36333.964019 secs 2023-03-10T01:03:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:43 smithi181 ceph-mon[94189]: pgmap v19888: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 589 KiB/s rd, 586 KiB/s wr, 12 op/s 2023-03-10T01:03:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36333.964019 secs 2023-03-10T01:03:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:43 smithi038 ceph-mon[92422]: pgmap v19888: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 589 KiB/s rd, 586 KiB/s wr, 12 op/s 2023-03-10T01:03:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:45 smithi122 ceph-mon[93869]: pgmap v19889: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 497 KiB/s rd, 495 KiB/s wr, 9 op/s 2023-03-10T01:03:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:45 smithi181 ceph-mon[94189]: pgmap v19889: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 497 KiB/s rd, 495 KiB/s wr, 9 op/s 2023-03-10T01:03:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:45 smithi038 ceph-mon[92422]: pgmap v19889: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 497 KiB/s rd, 495 KiB/s wr, 9 op/s 2023-03-10T01:03:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36336.503041 secs 2023-03-10T01:03:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36336.503041 secs 2023-03-10T01:03:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36336.503041 secs 2023-03-10T01:03:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:47 smithi122 ceph-mon[93869]: pgmap v19890: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 647 KiB/s rd, 643 KiB/s wr, 11 op/s 2023-03-10T01:03:47.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:47 smithi181 ceph-mon[94189]: pgmap v19890: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 647 KiB/s rd, 643 KiB/s wr, 11 op/s 2023-03-10T01:03:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:47 smithi038 ceph-mon[92422]: pgmap v19890: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 647 KiB/s rd, 643 KiB/s wr, 11 op/s 2023-03-10T01:03:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36338.964111 secs 2023-03-10T01:03:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36338.964111 secs 2023-03-10T01:03:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36338.964111 secs 2023-03-10T01:03:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36800.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:03:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:03:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:03:49.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:49 smithi122 ceph-mon[93869]: pgmap v19891: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 502 KiB/s wr, 9 op/s 2023-03-10T01:03:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:03:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:03:49.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:49 smithi181 ceph-mon[94189]: pgmap v19891: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 502 KiB/s wr, 9 op/s 2023-03-10T01:03:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:03:49.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:03:49.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:49 smithi038 ceph-mon[92422]: pgmap v19891: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 502 KiB/s wr, 9 op/s 2023-03-10T01:03:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36341.503098 secs 2023-03-10T01:03:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:51 smithi122 ceph-mon[93869]: pgmap v19892: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 497 KiB/s wr, 8 op/s 2023-03-10T01:03:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:03:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:03:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36341.503098 secs 2023-03-10T01:03:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:51 smithi181 ceph-mon[94189]: pgmap v19892: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 497 KiB/s wr, 8 op/s 2023-03-10T01:03:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:03:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:03:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36341.503098 secs 2023-03-10T01:03:52.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:51 smithi038 ceph-mon[92422]: pgmap v19892: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 497 KiB/s wr, 8 op/s 2023-03-10T01:03:52.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:03:52.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:03:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36805.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:03:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36343.964196 secs 2023-03-10T01:03:53.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:53 smithi122 ceph-mon[93869]: pgmap v19893: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 817 KiB/s wr, 12 op/s 2023-03-10T01:03:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36343.964196 secs 2023-03-10T01:03:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:53 smithi181 ceph-mon[94189]: pgmap v19893: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 817 KiB/s wr, 12 op/s 2023-03-10T01:03:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36343.964196 secs 2023-03-10T01:03:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:53 smithi038 ceph-mon[92422]: pgmap v19893: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 817 KiB/s wr, 12 op/s 2023-03-10T01:03:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:55 smithi122 ceph-mon[93869]: pgmap v19894: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 470 KiB/s rd, 469 KiB/s wr, 8 op/s 2023-03-10T01:03:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:55 smithi181 ceph-mon[94189]: pgmap v19894: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 470 KiB/s rd, 469 KiB/s wr, 8 op/s 2023-03-10T01:03:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:55 smithi038 ceph-mon[92422]: pgmap v19894: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 470 KiB/s rd, 469 KiB/s wr, 8 op/s 2023-03-10T01:03:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36346.503166 secs 2023-03-10T01:03:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36346.503166 secs 2023-03-10T01:03:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36346.503166 secs 2023-03-10T01:03:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:57 smithi122 ceph-mon[93869]: pgmap v19895: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 511 KiB/s rd, 506 KiB/s wr, 11 op/s 2023-03-10T01:03:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:57 smithi181 ceph-mon[94189]: pgmap v19895: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 511 KiB/s rd, 506 KiB/s wr, 11 op/s 2023-03-10T01:03:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:57 smithi038 ceph-mon[92422]: pgmap v19895: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 511 KiB/s rd, 506 KiB/s wr, 11 op/s 2023-03-10T01:03:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36348.964255 secs 2023-03-10T01:03:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36348.964255 secs 2023-03-10T01:03:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36348.964255 secs 2023-03-10T01:03:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36810.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:03:59.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:03:59 smithi122 ceph-mon[93869]: pgmap v19896: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 388 KiB/s wr, 9 op/s 2023-03-10T01:03:59.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:03:59 smithi181 ceph-mon[94189]: pgmap v19896: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 388 KiB/s wr, 9 op/s 2023-03-10T01:03:59.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:03:59 smithi038 ceph-mon[92422]: pgmap v19896: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 388 KiB/s wr, 9 op/s 2023-03-10T01:04:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:04:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:04:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:04:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:04:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:04:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:04:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36351.503250 secs 2023-03-10T01:04:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:04:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:04:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:01 smithi122 ceph-mon[93869]: pgmap v19897: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 577 KiB/s rd, 562 KiB/s wr, 9 op/s 2023-03-10T01:04:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36351.503250 secs 2023-03-10T01:04:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:04:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:04:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:01 smithi181 ceph-mon[94189]: pgmap v19897: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 577 KiB/s rd, 562 KiB/s wr, 9 op/s 2023-03-10T01:04:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36351.503250 secs 2023-03-10T01:04:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:04:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:04:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:01 smithi038 ceph-mon[92422]: pgmap v19897: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 577 KiB/s rd, 562 KiB/s wr, 9 op/s 2023-03-10T01:04:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36815.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:04:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36353.964314 secs 2023-03-10T01:04:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:03 smithi122 ceph-mon[93869]: pgmap v19898: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 968 KiB/s rd, 934 KiB/s wr, 14 op/s 2023-03-10T01:04:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36353.964314 secs 2023-03-10T01:04:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:03 smithi181 ceph-mon[94189]: pgmap v19898: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 968 KiB/s rd, 934 KiB/s wr, 14 op/s 2023-03-10T01:04:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36353.964314 secs 2023-03-10T01:04:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:03 smithi038 ceph-mon[92422]: pgmap v19898: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 968 KiB/s rd, 934 KiB/s wr, 14 op/s 2023-03-10T01:04:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:05 smithi122 ceph-mon[93869]: pgmap v19899: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 654 KiB/s rd, 625 KiB/s wr, 21 op/s 2023-03-10T01:04:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:05 smithi181 ceph-mon[94189]: pgmap v19899: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 654 KiB/s rd, 625 KiB/s wr, 21 op/s 2023-03-10T01:04:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:05 smithi038 ceph-mon[92422]: pgmap v19899: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 654 KiB/s rd, 625 KiB/s wr, 21 op/s 2023-03-10T01:04:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36356.503308 secs 2023-03-10T01:04:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36356.503308 secs 2023-03-10T01:04:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36356.503308 secs 2023-03-10T01:04:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:07 smithi122 ceph-mon[93869]: pgmap v19900: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 803 KiB/s wr, 26 op/s 2023-03-10T01:04:07.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:07 smithi181 ceph-mon[94189]: pgmap v19900: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 803 KiB/s wr, 26 op/s 2023-03-10T01:04:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:07 smithi038 ceph-mon[92422]: pgmap v19900: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 803 KiB/s wr, 26 op/s 2023-03-10T01:04:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:04:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:04:08.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36358.964415 secs 2023-03-10T01:04:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:04:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:04:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36358.964415 secs 2023-03-10T01:04:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:04:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:04:08.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36358.964415 secs 2023-03-10T01:04:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36820.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:04:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:04:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:04:09.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:09 smithi122 ceph-mon[93869]: pgmap v19901: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 766 KiB/s wr, 24 op/s 2023-03-10T01:04:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:04:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:04:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:09 smithi181 ceph-mon[94189]: pgmap v19901: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 766 KiB/s wr, 24 op/s 2023-03-10T01:04:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:04:09.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:04:09.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:09 smithi038 ceph-mon[92422]: pgmap v19901: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 766 KiB/s wr, 24 op/s 2023-03-10T01:04:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36361.503396 secs 2023-03-10T01:04:11.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:11 smithi122 ceph-mon[93869]: pgmap v19902: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 768 KiB/s wr, 23 op/s 2023-03-10T01:04:11.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:11 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:04:11.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:11 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:04:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36361.503396 secs 2023-03-10T01:04:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:11 smithi181 ceph-mon[94189]: pgmap v19902: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 768 KiB/s wr, 23 op/s 2023-03-10T01:04:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:11 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:04:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:11 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:04:12.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36361.503396 secs 2023-03-10T01:04:12.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:11 smithi038 ceph-mon[92422]: pgmap v19902: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 768 KiB/s wr, 23 op/s 2023-03-10T01:04:12.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:11 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:04:12.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:11 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:04:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36825.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:04:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36363.964472 secs 2023-03-10T01:04:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:13 smithi122 ceph-mon[93869]: pgmap v19903: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 938 KiB/s rd, 593 KiB/s wr, 25 op/s 2023-03-10T01:04:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36363.964472 secs 2023-03-10T01:04:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:13 smithi181 ceph-mon[94189]: pgmap v19903: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 938 KiB/s rd, 593 KiB/s wr, 25 op/s 2023-03-10T01:04:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36363.964472 secs 2023-03-10T01:04:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:13 smithi038 ceph-mon[92422]: pgmap v19903: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 938 KiB/s rd, 593 KiB/s wr, 25 op/s 2023-03-10T01:04:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:15 smithi122 ceph-mon[93869]: pgmap v19904: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 387 KiB/s wr, 21 op/s 2023-03-10T01:04:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:15 smithi181 ceph-mon[94189]: pgmap v19904: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 387 KiB/s wr, 21 op/s 2023-03-10T01:04:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:15 smithi038 ceph-mon[92422]: pgmap v19904: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 387 KiB/s wr, 21 op/s 2023-03-10T01:04:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36366.503454 secs 2023-03-10T01:04:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36366.503454 secs 2023-03-10T01:04:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36366.503454 secs 2023-03-10T01:04:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:17 smithi122 ceph-mon[93869]: pgmap v19905: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 708 KiB/s rd, 377 KiB/s wr, 12 op/s 2023-03-10T01:04:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:17 smithi181 ceph-mon[94189]: pgmap v19905: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 708 KiB/s rd, 377 KiB/s wr, 12 op/s 2023-03-10T01:04:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:17 smithi038 ceph-mon[92422]: pgmap v19905: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 708 KiB/s rd, 377 KiB/s wr, 12 op/s 2023-03-10T01:04:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36368.964603 secs 2023-03-10T01:04:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:18 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:04:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36368.964603 secs 2023-03-10T01:04:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:18 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:04:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36368.964603 secs 2023-03-10T01:04:18.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:18 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:04:18.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36830.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:04:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:04:19.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:04:19.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:04:19.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:19 smithi122 ceph-mon[93869]: pgmap v19906: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 210 KiB/s wr, 9 op/s 2023-03-10T01:04:19.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:04:19.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:04:19.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:04:19.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:19 smithi181 ceph-mon[94189]: pgmap v19906: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 210 KiB/s wr, 9 op/s 2023-03-10T01:04:19.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:04:19.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:04:19.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:04:19.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:19 smithi038 ceph-mon[92422]: pgmap v19906: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 210 KiB/s wr, 9 op/s 2023-03-10T01:04:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:04:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:04:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:04:20.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:04:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:04:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:04:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36371.503513 secs 2023-03-10T01:04:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:04:21.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:04:21.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:21 smithi122 ceph-mon[93869]: pgmap v19907: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 353 KiB/s wr, 9 op/s 2023-03-10T01:04:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36371.503513 secs 2023-03-10T01:04:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:04:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:04:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:21 smithi181 ceph-mon[94189]: pgmap v19907: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 353 KiB/s wr, 9 op/s 2023-03-10T01:04:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36371.503513 secs 2023-03-10T01:04:21.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:04:21.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:04:21.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:21 smithi038 ceph-mon[92422]: pgmap v19907: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 353 KiB/s wr, 9 op/s 2023-03-10T01:04:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36835.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:04:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36373.964662 secs 2023-03-10T01:04:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:23 smithi122 ceph-mon[93869]: pgmap v19908: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 416 KiB/s rd, 435 KiB/s wr, 12 op/s 2023-03-10T01:04:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36373.964662 secs 2023-03-10T01:04:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:23 smithi181 ceph-mon[94189]: pgmap v19908: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 416 KiB/s rd, 435 KiB/s wr, 12 op/s 2023-03-10T01:04:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36373.964662 secs 2023-03-10T01:04:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:23 smithi038 ceph-mon[92422]: pgmap v19908: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 416 KiB/s rd, 435 KiB/s wr, 12 op/s 2023-03-10T01:04:25.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:25 smithi122 ceph-mon[93869]: pgmap v19909: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 434 KiB/s wr, 10 op/s 2023-03-10T01:04:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:25 smithi181 ceph-mon[94189]: pgmap v19909: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 434 KiB/s wr, 10 op/s 2023-03-10T01:04:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:25 smithi038 ceph-mon[92422]: pgmap v19909: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 434 KiB/s wr, 10 op/s 2023-03-10T01:04:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36376.503603 secs 2023-03-10T01:04:26.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:26 smithi122 ceph-mon[93869]: osdmap e1459: 12 total, 12 up, 12 in 2023-03-10T01:04:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36376.503603 secs 2023-03-10T01:04:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:26 smithi181 ceph-mon[94189]: osdmap e1459: 12 total, 12 up, 12 in 2023-03-10T01:04:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36376.503603 secs 2023-03-10T01:04:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:26 smithi038 ceph-mon[92422]: osdmap e1459: 12 total, 12 up, 12 in 2023-03-10T01:04:27.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:27 smithi122 ceph-mon[93869]: pgmap v19911: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 522 KiB/s wr, 12 op/s 2023-03-10T01:04:27.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:27 smithi181 ceph-mon[94189]: pgmap v19911: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 522 KiB/s wr, 12 op/s 2023-03-10T01:04:27.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:27 smithi038 ceph-mon[92422]: pgmap v19911: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 522 KiB/s wr, 12 op/s 2023-03-10T01:04:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36378.964722 secs 2023-03-10T01:04:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36378.964722 secs 2023-03-10T01:04:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36378.964722 secs 2023-03-10T01:04:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36840.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:04:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:29 smithi122 ceph-mon[93869]: pgmap v19912: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 509 KiB/s wr, 11 op/s 2023-03-10T01:04:29.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:29 smithi181 ceph-mon[94189]: pgmap v19912: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 509 KiB/s wr, 11 op/s 2023-03-10T01:04:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:29 smithi038 ceph-mon[92422]: pgmap v19912: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 509 KiB/s wr, 11 op/s 2023-03-10T01:04:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36381.503678 secs 2023-03-10T01:04:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:31 smithi122 ceph-mon[93869]: pgmap v19913: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 326 KiB/s rd, 339 KiB/s wr, 10 op/s 2023-03-10T01:04:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36381.503678 secs 2023-03-10T01:04:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:31 smithi181 ceph-mon[94189]: pgmap v19913: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 326 KiB/s rd, 339 KiB/s wr, 10 op/s 2023-03-10T01:04:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36381.503678 secs 2023-03-10T01:04:32.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:31 smithi038 ceph-mon[92422]: pgmap v19913: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 326 KiB/s rd, 339 KiB/s wr, 10 op/s 2023-03-10T01:04:33.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36845.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:04:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36383.964809 secs 2023-03-10T01:04:33.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:33 smithi122 ceph-mon[93869]: pgmap v19914: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 484 KiB/s rd, 360 KiB/s wr, 8 op/s 2023-03-10T01:04:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36383.964809 secs 2023-03-10T01:04:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:33 smithi181 ceph-mon[94189]: pgmap v19914: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 484 KiB/s rd, 360 KiB/s wr, 8 op/s 2023-03-10T01:04:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36383.964809 secs 2023-03-10T01:04:34.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:33 smithi038 ceph-mon[92422]: pgmap v19914: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 484 KiB/s rd, 360 KiB/s wr, 8 op/s 2023-03-10T01:04:35.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:35 smithi122 ceph-mon[93869]: osdmap e1460: 12 total, 12 up, 12 in 2023-03-10T01:04:35.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:35 smithi122 ceph-mon[93869]: pgmap v19916: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 729 KiB/s rd, 624 KiB/s wr, 9 op/s 2023-03-10T01:04:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:35 smithi181 ceph-mon[94189]: osdmap e1460: 12 total, 12 up, 12 in 2023-03-10T01:04:35.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:35 smithi181 ceph-mon[94189]: pgmap v19916: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 729 KiB/s rd, 624 KiB/s wr, 9 op/s 2023-03-10T01:04:35.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:35 smithi038 ceph-mon[92422]: osdmap e1460: 12 total, 12 up, 12 in 2023-03-10T01:04:35.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:35 smithi038 ceph-mon[92422]: pgmap v19916: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 729 KiB/s rd, 624 KiB/s wr, 9 op/s 2023-03-10T01:04:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36386.503736 secs 2023-03-10T01:04:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:04:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:04:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36386.503736 secs 2023-03-10T01:04:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:04:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:04:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36386.503736 secs 2023-03-10T01:04:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:04:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:04:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:04:37.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:04:37.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:37 smithi122 ceph-mon[93869]: pgmap v19917: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 795 KiB/s wr, 9 op/s 2023-03-10T01:04:37.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:04:37.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:04:37.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:37 smithi181 ceph-mon[94189]: pgmap v19917: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 795 KiB/s wr, 9 op/s 2023-03-10T01:04:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:04:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:04:37.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:37 smithi038 ceph-mon[92422]: pgmap v19917: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 795 KiB/s wr, 9 op/s 2023-03-10T01:04:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36388.964898 secs 2023-03-10T01:04:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36388.964898 secs 2023-03-10T01:04:38.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36388.964898 secs 2023-03-10T01:04:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36850.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:04:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:39 smithi038 ceph-mon[92422]: pgmap v19918: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 806 KiB/s wr, 11 op/s 2023-03-10T01:04:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:39 smithi122 ceph-mon[93869]: pgmap v19918: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 806 KiB/s wr, 11 op/s 2023-03-10T01:04:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:39 smithi181 ceph-mon[94189]: pgmap v19918: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 806 KiB/s wr, 11 op/s 2023-03-10T01:04:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:04:40.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:04:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:04:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:04:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:04:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:04:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36391.503820 secs 2023-03-10T01:04:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:04:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:04:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:41 smithi038 ceph-mon[92422]: pgmap v19919: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1015 KiB/s wr, 12 op/s 2023-03-10T01:04:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36391.503820 secs 2023-03-10T01:04:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:04:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:04:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:41 smithi122 ceph-mon[93869]: pgmap v19919: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1015 KiB/s wr, 12 op/s 2023-03-10T01:04:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36391.503820 secs 2023-03-10T01:04:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:04:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:04:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:41 smithi181 ceph-mon[94189]: pgmap v19919: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1015 KiB/s wr, 12 op/s 2023-03-10T01:04:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36855.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:04:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36393.964956 secs 2023-03-10T01:04:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:43 smithi122 ceph-mon[93869]: pgmap v19920: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.2 MiB/s wr, 12 op/s 2023-03-10T01:04:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36393.964956 secs 2023-03-10T01:04:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:43 smithi181 ceph-mon[94189]: pgmap v19920: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.2 MiB/s wr, 12 op/s 2023-03-10T01:04:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36393.964956 secs 2023-03-10T01:04:44.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:43 smithi038 ceph-mon[92422]: pgmap v19920: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.2 MiB/s wr, 12 op/s 2023-03-10T01:04:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:45 smithi122 ceph-mon[93869]: pgmap v19921: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.0 MiB/s wr, 12 op/s 2023-03-10T01:04:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:45 smithi181 ceph-mon[94189]: pgmap v19921: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.0 MiB/s wr, 12 op/s 2023-03-10T01:04:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:45 smithi038 ceph-mon[92422]: pgmap v19921: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.0 MiB/s wr, 12 op/s 2023-03-10T01:04:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36396.503882 secs 2023-03-10T01:04:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36396.503882 secs 2023-03-10T01:04:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36396.503882 secs 2023-03-10T01:04:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:47 smithi122 ceph-mon[93869]: pgmap v19922: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 927 KiB/s wr, 12 op/s 2023-03-10T01:04:47.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:47 smithi181 ceph-mon[94189]: pgmap v19922: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 927 KiB/s wr, 12 op/s 2023-03-10T01:04:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:47 smithi038 ceph-mon[92422]: pgmap v19922: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 927 KiB/s wr, 12 op/s 2023-03-10T01:04:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:04:48.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:04:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:04:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:04:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:04:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:04:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36860.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:04:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:04:49.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:04:49.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:49 smithi038 ceph-mon[92422]: pgmap v19923: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 679 KiB/s rd, 904 KiB/s wr, 12 op/s 2023-03-10T01:04:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:04:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:04:49.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:49 smithi122 ceph-mon[93869]: pgmap v19923: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 679 KiB/s rd, 904 KiB/s wr, 12 op/s 2023-03-10T01:04:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:04:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:04:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:49 smithi181 ceph-mon[94189]: pgmap v19923: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 679 KiB/s rd, 904 KiB/s wr, 12 op/s 2023-03-10T01:04:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:50 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:04:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:50 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:04:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:50 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:04:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36401.503939 secs 2023-03-10T01:04:51.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:51 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:04:51.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:04:51.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:04:51.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:51 smithi038 ceph-mon[92422]: pgmap v19924: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 1.0 MiB/s wr, 11 op/s 2023-03-10T01:04:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36401.503939 secs 2023-03-10T01:04:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:51 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:04:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:04:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:04:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:51 smithi122 ceph-mon[93869]: pgmap v19924: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 1.0 MiB/s wr, 11 op/s 2023-03-10T01:04:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36401.503939 secs 2023-03-10T01:04:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:51 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:04:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:04:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:04:51.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:51 smithi181 ceph-mon[94189]: pgmap v19924: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 1.0 MiB/s wr, 11 op/s 2023-03-10T01:04:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:04:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:04:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:04:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:04:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:04:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:04:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:04:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:04:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:53 smithi122 ceph-mon[93869]: pgmap v19925: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 13 op/s 2023-03-10T01:04:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:04:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:04:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:53 smithi181 ceph-mon[94189]: pgmap v19925: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 13 op/s 2023-03-10T01:04:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:04:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:04:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:53 smithi038 ceph-mon[92422]: pgmap v19925: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 13 op/s 2023-03-10T01:04:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36865.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:04:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:55 smithi122 ceph-mon[93869]: pgmap v19926: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 984 KiB/s wr, 11 op/s 2023-03-10T01:04:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:55 smithi181 ceph-mon[94189]: pgmap v19926: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 984 KiB/s wr, 11 op/s 2023-03-10T01:04:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:55 smithi038 ceph-mon[92422]: pgmap v19926: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 984 KiB/s wr, 11 op/s 2023-03-10T01:04:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36406.503983 secs 2023-03-10T01:04:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:04:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:04:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36406.503983 secs 2023-03-10T01:04:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:04:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:04:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36406.503983 secs 2023-03-10T01:04:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:04:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:04:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:04:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:04:57.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:57 smithi122 ceph-mon[93869]: pgmap v19927: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.0 MiB/s wr, 13 op/s 2023-03-10T01:04:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:04:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:04:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:57 smithi181 ceph-mon[94189]: pgmap v19927: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.0 MiB/s wr, 13 op/s 2023-03-10T01:04:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:04:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:04:57.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:57 smithi038 ceph-mon[92422]: pgmap v19927: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.0 MiB/s wr, 13 op/s 2023-03-10T01:04:58.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36870.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:04:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:04:59 smithi122 ceph-mon[93869]: pgmap v19928: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.0 MiB/s wr, 14 op/s 2023-03-10T01:04:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:04:59 smithi181 ceph-mon[94189]: pgmap v19928: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.0 MiB/s wr, 14 op/s 2023-03-10T01:04:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:04:59 smithi038 ceph-mon[92422]: pgmap v19928: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.0 MiB/s wr, 14 op/s 2023-03-10T01:05:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36411.504041 secs 2023-03-10T01:05:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:01 smithi122 ceph-mon[93869]: pgmap v19929: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 957 KiB/s wr, 12 op/s 2023-03-10T01:05:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:05:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:05:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36411.504041 secs 2023-03-10T01:05:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:01 smithi181 ceph-mon[94189]: pgmap v19929: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 957 KiB/s wr, 12 op/s 2023-03-10T01:05:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:05:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:05:02.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36411.504041 secs 2023-03-10T01:05:02.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:01 smithi038 ceph-mon[92422]: pgmap v19929: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 957 KiB/s wr, 12 op/s 2023-03-10T01:05:02.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:05:02.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:05:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36875.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:05:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:03 smithi122 ceph-mon[93869]: pgmap v19930: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 855 KiB/s wr, 14 op/s 2023-03-10T01:05:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:03 smithi181 ceph-mon[94189]: pgmap v19930: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 855 KiB/s wr, 14 op/s 2023-03-10T01:05:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:03 smithi038 ceph-mon[92422]: pgmap v19930: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 855 KiB/s wr, 14 op/s 2023-03-10T01:05:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:05 smithi122 ceph-mon[93869]: pgmap v19931: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 641 KiB/s wr, 13 op/s 2023-03-10T01:05:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:05 smithi181 ceph-mon[94189]: pgmap v19931: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 641 KiB/s wr, 13 op/s 2023-03-10T01:05:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:05 smithi038 ceph-mon[92422]: pgmap v19931: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 641 KiB/s wr, 13 op/s 2023-03-10T01:05:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36416.504098 secs 2023-03-10T01:05:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36416.504098 secs 2023-03-10T01:05:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36416.504098 secs 2023-03-10T01:05:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:07 smithi122 ceph-mon[93869]: pgmap v19932: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 655 KiB/s wr, 16 op/s 2023-03-10T01:05:07.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:07 smithi181 ceph-mon[94189]: pgmap v19932: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 655 KiB/s wr, 16 op/s 2023-03-10T01:05:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:07 smithi038 ceph-mon[92422]: pgmap v19932: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 655 KiB/s wr, 16 op/s 2023-03-10T01:05:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:05:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:05:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:05:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:05:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:05:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:05:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36880.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:05:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:05:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:05:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:09 smithi181 ceph-mon[94189]: pgmap v19933: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 971 KiB/s rd, 524 KiB/s wr, 24 op/s 2023-03-10T01:05:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:05:09.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:05:09.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:09 smithi038 ceph-mon[92422]: pgmap v19933: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 971 KiB/s rd, 524 KiB/s wr, 24 op/s 2023-03-10T01:05:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:05:09.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:05:09.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:09 smithi122 ceph-mon[93869]: pgmap v19933: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 971 KiB/s rd, 524 KiB/s wr, 24 op/s 2023-03-10T01:05:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36421.504156 secs 2023-03-10T01:05:11.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:11 smithi122 ceph-mon[93869]: pgmap v19934: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 970 KiB/s rd, 494 KiB/s wr, 22 op/s 2023-03-10T01:05:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36421.504156 secs 2023-03-10T01:05:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:11 smithi181 ceph-mon[94189]: pgmap v19934: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 970 KiB/s rd, 494 KiB/s wr, 22 op/s 2023-03-10T01:05:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36421.504156 secs 2023-03-10T01:05:12.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:11 smithi038 ceph-mon[92422]: pgmap v19934: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 970 KiB/s rd, 494 KiB/s wr, 22 op/s 2023-03-10T01:05:13.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36885.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:05:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:13 smithi122 ceph-mon[93869]: pgmap v19935: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 538 KiB/s rd, 521 KiB/s wr, 24 op/s 2023-03-10T01:05:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:13 smithi181 ceph-mon[94189]: pgmap v19935: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 538 KiB/s rd, 521 KiB/s wr, 24 op/s 2023-03-10T01:05:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:13 smithi038 ceph-mon[92422]: pgmap v19935: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 538 KiB/s rd, 521 KiB/s wr, 24 op/s 2023-03-10T01:05:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:15 smithi122 ceph-mon[93869]: pgmap v19936: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 499 KiB/s wr, 23 op/s 2023-03-10T01:05:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:15 smithi181 ceph-mon[94189]: pgmap v19936: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 499 KiB/s wr, 23 op/s 2023-03-10T01:05:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:15 smithi038 ceph-mon[92422]: pgmap v19936: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 499 KiB/s wr, 23 op/s 2023-03-10T01:05:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36426.504216 secs 2023-03-10T01:05:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36426.504216 secs 2023-03-10T01:05:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36426.504216 secs 2023-03-10T01:05:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:17 smithi122 ceph-mon[93869]: pgmap v19937: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 372 KiB/s wr, 22 op/s 2023-03-10T01:05:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:17 smithi181 ceph-mon[94189]: pgmap v19937: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 372 KiB/s wr, 22 op/s 2023-03-10T01:05:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:17 smithi038 ceph-mon[92422]: pgmap v19937: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 372 KiB/s wr, 22 op/s 2023-03-10T01:05:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36890.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:05:19.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:19 smithi122 ceph-mon[93869]: pgmap v19938: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 358 KiB/s wr, 18 op/s 2023-03-10T01:05:19.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:19 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:05:19.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:19 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:05:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:19 smithi181 ceph-mon[94189]: pgmap v19938: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 358 KiB/s wr, 18 op/s 2023-03-10T01:05:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:19 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:05:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:19 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:05:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:19 smithi038 ceph-mon[92422]: pgmap v19938: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 358 KiB/s wr, 18 op/s 2023-03-10T01:05:20.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:19 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:05:20.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:19 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:05:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36431.504273 secs 2023-03-10T01:05:21.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:21 smithi122 ceph-mon[93869]: pgmap v19939: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 435 KiB/s wr, 8 op/s 2023-03-10T01:05:21.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:05:21.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:05:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36431.504273 secs 2023-03-10T01:05:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:21 smithi181 ceph-mon[94189]: pgmap v19939: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 435 KiB/s wr, 8 op/s 2023-03-10T01:05:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:05:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:05:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36431.504273 secs 2023-03-10T01:05:22.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:21 smithi038 ceph-mon[92422]: pgmap v19939: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 435 KiB/s wr, 8 op/s 2023-03-10T01:05:22.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:05:22.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:05:22.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:22 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:05:22.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:22 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:05:22.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:22 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:05:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36895.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:05:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:23 smithi122 ceph-mon[93869]: pgmap v19940: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 435 KiB/s wr, 9 op/s 2023-03-10T01:05:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:23 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:05:23.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:23 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:05:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:23 smithi181 ceph-mon[94189]: pgmap v19940: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 435 KiB/s wr, 9 op/s 2023-03-10T01:05:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:23 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:05:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:23 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:05:24.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:23 smithi038 ceph-mon[92422]: pgmap v19940: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 435 KiB/s wr, 9 op/s 2023-03-10T01:05:24.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:23 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:05:24.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:23 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:05:24.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:05:24.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:05:24.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:05:24.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:05:24.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:05:24.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:05:24.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:05:25.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:05:25.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:05:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:25 smithi122 ceph-mon[93869]: pgmap v19941: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 412 KiB/s wr, 10 op/s 2023-03-10T01:05:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:25 smithi181 ceph-mon[94189]: pgmap v19941: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 412 KiB/s wr, 10 op/s 2023-03-10T01:05:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:25 smithi038 ceph-mon[92422]: pgmap v19941: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 412 KiB/s wr, 10 op/s 2023-03-10T01:05:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36436.504345 secs 2023-03-10T01:05:26.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36436.504345 secs 2023-03-10T01:05:26.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36436.504345 secs 2023-03-10T01:05:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:27 smithi122 ceph-mon[93869]: pgmap v19942: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 478 KiB/s wr, 12 op/s 2023-03-10T01:05:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:27 smithi181 ceph-mon[94189]: pgmap v19942: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 478 KiB/s wr, 12 op/s 2023-03-10T01:05:28.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:27 smithi038 ceph-mon[92422]: pgmap v19942: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 478 KiB/s wr, 12 op/s 2023-03-10T01:05:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36900.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:05:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36438.965635 secs 2023-03-10T01:05:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36438.965635 secs 2023-03-10T01:05:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36438.965635 secs 2023-03-10T01:05:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:29 smithi122 ceph-mon[93869]: pgmap v19943: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 457 KiB/s rd, 478 KiB/s wr, 11 op/s 2023-03-10T01:05:29.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:29 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:05:29.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:29 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:05:29.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:29 smithi181 ceph-mon[94189]: pgmap v19943: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 457 KiB/s rd, 478 KiB/s wr, 11 op/s 2023-03-10T01:05:29.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:29 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:05:29.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:29 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:05:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:29 smithi038 ceph-mon[92422]: pgmap v19943: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 457 KiB/s rd, 478 KiB/s wr, 11 op/s 2023-03-10T01:05:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:29 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:05:30.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:29 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:05:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:30 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:05:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:30 smithi122 ceph-mon[93869]: osdmap e1461: 12 total, 12 up, 12 in 2023-03-10T01:05:30.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:30 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:05:30.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:30 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:05:30.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:30 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:05:30.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:30 smithi181 ceph-mon[94189]: osdmap e1461: 12 total, 12 up, 12 in 2023-03-10T01:05:30.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:30 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:05:30.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:30 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:05:30.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:30 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:05:31.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:30 smithi038 ceph-mon[92422]: osdmap e1461: 12 total, 12 up, 12 in 2023-03-10T01:05:31.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:30 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:05:31.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:30 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:05:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36441.504404 secs 2023-03-10T01:05:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:31 smithi122 ceph-mon[93869]: pgmap v19945: 129 pgs: 1 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 124 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 398 KiB/s wr, 12 op/s 2023-03-10T01:05:31.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:31 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:05:31.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:31 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:05:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36441.504404 secs 2023-03-10T01:05:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:31 smithi181 ceph-mon[94189]: pgmap v19945: 129 pgs: 1 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 124 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 398 KiB/s wr, 12 op/s 2023-03-10T01:05:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:31 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:05:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:31 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:05:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36441.504404 secs 2023-03-10T01:05:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:31 smithi038 ceph-mon[92422]: pgmap v19945: 129 pgs: 1 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 124 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 398 KiB/s wr, 12 op/s 2023-03-10T01:05:32.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:31 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:05:32.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:31 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:05:33.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36905.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:05:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36443.965693 secs 2023-03-10T01:05:33.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:33 smithi122 ceph-mon[93869]: pgmap v19946: 129 pgs: 1 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 124 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 609 KiB/s rd, 599 KiB/s wr, 13 op/s 2023-03-10T01:05:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36443.965693 secs 2023-03-10T01:05:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:33 smithi181 ceph-mon[94189]: pgmap v19946: 129 pgs: 1 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 124 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 609 KiB/s rd, 599 KiB/s wr, 13 op/s 2023-03-10T01:05:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36443.965693 secs 2023-03-10T01:05:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:33 smithi038 ceph-mon[92422]: pgmap v19946: 129 pgs: 1 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 124 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 609 KiB/s rd, 599 KiB/s wr, 13 op/s 2023-03-10T01:05:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:35 smithi122 ceph-mon[93869]: pgmap v19947: 129 pgs: 1 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 124 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 338 KiB/s wr, 10 op/s 2023-03-10T01:05:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:35 smithi181 ceph-mon[94189]: pgmap v19947: 129 pgs: 1 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 124 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 338 KiB/s wr, 10 op/s 2023-03-10T01:05:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:35 smithi038 ceph-mon[92422]: pgmap v19947: 129 pgs: 1 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 124 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 361 KiB/s rd, 338 KiB/s wr, 10 op/s 2023-03-10T01:05:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36446.504462 secs 2023-03-10T01:05:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:05:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:05:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36446.504462 secs 2023-03-10T01:05:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:05:36.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:05:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36446.504462 secs 2023-03-10T01:05:36.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:05:36.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:05:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:05:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:05:37.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:37 smithi038 ceph-mon[92422]: pgmap v19948: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 453 KiB/s rd, 416 KiB/s wr, 9 op/s 2023-03-10T01:05:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:05:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:05:37.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:37 smithi122 ceph-mon[93869]: pgmap v19948: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 453 KiB/s rd, 416 KiB/s wr, 9 op/s 2023-03-10T01:05:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:05:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:05:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:37 smithi181 ceph-mon[94189]: pgmap v19948: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 453 KiB/s rd, 416 KiB/s wr, 9 op/s 2023-03-10T01:05:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36448.965749 secs 2023-03-10T01:05:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36448.965749 secs 2023-03-10T01:05:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36448.965749 secs 2023-03-10T01:05:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36910.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:05:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:39 smithi038 ceph-mon[92422]: pgmap v19949: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 416 KiB/s wr, 9 op/s 2023-03-10T01:05:39.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:39 smithi038 ceph-mon[92422]: osdmap e1462: 12 total, 12 up, 12 in 2023-03-10T01:05:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:39 smithi122 ceph-mon[93869]: pgmap v19949: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 416 KiB/s wr, 9 op/s 2023-03-10T01:05:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:39 smithi122 ceph-mon[93869]: osdmap e1462: 12 total, 12 up, 12 in 2023-03-10T01:05:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:39 smithi181 ceph-mon[94189]: pgmap v19949: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 416 KiB/s wr, 9 op/s 2023-03-10T01:05:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:39 smithi181 ceph-mon[94189]: osdmap e1462: 12 total, 12 up, 12 in 2023-03-10T01:05:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36451.504519 secs 2023-03-10T01:05:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:41 smithi122 ceph-mon[93869]: pgmap v19951: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 416 KiB/s wr, 9 op/s 2023-03-10T01:05:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36451.504519 secs 2023-03-10T01:05:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:41 smithi181 ceph-mon[94189]: pgmap v19951: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 416 KiB/s wr, 9 op/s 2023-03-10T01:05:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36451.504519 secs 2023-03-10T01:05:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:41 smithi038 ceph-mon[92422]: pgmap v19951: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 416 KiB/s wr, 9 op/s 2023-03-10T01:05:43.805 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36915.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:05:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36453.965803 secs 2023-03-10T01:05:43.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:43 smithi122 ceph-mon[93869]: pgmap v19952: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 415 KiB/s wr, 8 op/s 2023-03-10T01:05:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36453.965803 secs 2023-03-10T01:05:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:43 smithi181 ceph-mon[94189]: pgmap v19952: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 415 KiB/s wr, 8 op/s 2023-03-10T01:05:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36453.965803 secs 2023-03-10T01:05:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:43 smithi038 ceph-mon[92422]: pgmap v19952: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 415 KiB/s wr, 8 op/s 2023-03-10T01:05:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:45 smithi122 ceph-mon[93869]: pgmap v19953: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 415 KiB/s wr, 8 op/s 2023-03-10T01:05:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:45 smithi181 ceph-mon[94189]: pgmap v19953: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 415 KiB/s wr, 8 op/s 2023-03-10T01:05:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:45 smithi038 ceph-mon[92422]: pgmap v19953: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 415 KiB/s wr, 8 op/s 2023-03-10T01:05:46.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36456.504580 secs 2023-03-10T01:05:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36456.504580 secs 2023-03-10T01:05:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36456.504580 secs 2023-03-10T01:05:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:47 smithi038 ceph-mon[92422]: pgmap v19954: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 467 KiB/s rd, 380 KiB/s wr, 8 op/s 2023-03-10T01:05:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:47 smithi122 ceph-mon[93869]: pgmap v19954: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 467 KiB/s rd, 380 KiB/s wr, 8 op/s 2023-03-10T01:05:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:47 smithi181 ceph-mon[94189]: pgmap v19954: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 467 KiB/s rd, 380 KiB/s wr, 8 op/s 2023-03-10T01:05:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:48 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 36458.965863 secs 2023-03-10T01:05:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:48 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 36458.965863 secs 2023-03-10T01:05:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:48 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 36458.965863 secs 2023-03-10T01:05:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36920.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:05:49.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:49 smithi038 ceph-mon[92422]: pgmap v19955: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 467 KiB/s rd, 380 KiB/s wr, 8 op/s 2023-03-10T01:05:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:49 smithi122 ceph-mon[93869]: pgmap v19955: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 467 KiB/s rd, 380 KiB/s wr, 8 op/s 2023-03-10T01:05:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:49 smithi181 ceph-mon[94189]: pgmap v19955: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 467 KiB/s rd, 380 KiB/s wr, 8 op/s 2023-03-10T01:05:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36461.504661 secs 2023-03-10T01:05:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:05:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:05:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:51 smithi122 ceph-mon[93869]: pgmap v19956: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 443 KiB/s rd, 367 KiB/s wr, 7 op/s 2023-03-10T01:05:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36461.504661 secs 2023-03-10T01:05:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:05:51.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:05:51.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:51 smithi181 ceph-mon[94189]: pgmap v19956: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 443 KiB/s rd, 367 KiB/s wr, 7 op/s 2023-03-10T01:05:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36461.504661 secs 2023-03-10T01:05:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:05:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:05:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:51 smithi038 ceph-mon[92422]: pgmap v19956: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 443 KiB/s rd, 367 KiB/s wr, 7 op/s 2023-03-10T01:05:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:05:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:05:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:53 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 36463.965923 secs 2023-03-10T01:05:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:53 smithi122 ceph-mon[93869]: pgmap v19957: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 610 KiB/s wr, 10 op/s 2023-03-10T01:05:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:05:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:05:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:53 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 36463.965923 secs 2023-03-10T01:05:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:53 smithi181 ceph-mon[94189]: pgmap v19957: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 610 KiB/s wr, 10 op/s 2023-03-10T01:05:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:05:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:05:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:53 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 36463.965923 secs 2023-03-10T01:05:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:53 smithi038 ceph-mon[92422]: pgmap v19957: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 610 KiB/s wr, 10 op/s 2023-03-10T01:05:53.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36925.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:05:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:55 smithi122 ceph-mon[93869]: pgmap v19958: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 545 KiB/s rd, 472 KiB/s wr, 8 op/s 2023-03-10T01:05:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:55 smithi181 ceph-mon[94189]: pgmap v19958: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 545 KiB/s rd, 472 KiB/s wr, 8 op/s 2023-03-10T01:05:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:55 smithi038 ceph-mon[92422]: pgmap v19958: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 545 KiB/s rd, 472 KiB/s wr, 8 op/s 2023-03-10T01:05:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36466.504739 secs 2023-03-10T01:05:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:05:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:05:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36466.504739 secs 2023-03-10T01:05:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:05:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:05:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36466.504739 secs 2023-03-10T01:05:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:05:56.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:05:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:05:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:05:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:57 smithi038 ceph-mon[92422]: pgmap v19959: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 580 KiB/s rd, 505 KiB/s wr, 11 op/s 2023-03-10T01:05:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:05:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:05:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:57 smithi122 ceph-mon[93869]: pgmap v19959: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 580 KiB/s rd, 505 KiB/s wr, 11 op/s 2023-03-10T01:05:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:05:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:05:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:57 smithi181 ceph-mon[94189]: pgmap v19959: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 580 KiB/s rd, 505 KiB/s wr, 11 op/s 2023-03-10T01:05:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36468.966008 secs 2023-03-10T01:05:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36468.966008 secs 2023-03-10T01:05:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36468.966008 secs 2023-03-10T01:05:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36930.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:05:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:05:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:05:59.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:05:59 smithi122 ceph-mon[93869]: pgmap v19960: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 385 KiB/s wr, 10 op/s 2023-03-10T01:05:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:05:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:05:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:05:59 smithi181 ceph-mon[94189]: pgmap v19960: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 385 KiB/s wr, 10 op/s 2023-03-10T01:05:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:05:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:05:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:05:59 smithi038 ceph-mon[92422]: pgmap v19960: 129 pgs: 129 active+clean; 13 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 385 KiB/s wr, 10 op/s 2023-03-10T01:06:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:06:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:06:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:06:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:06:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:06:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:06:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36471.504800 secs 2023-03-10T01:06:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:06:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:06:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:01 smithi122 ceph-mon[93869]: pgmap v19961: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 678 KiB/s rd, 664 KiB/s wr, 10 op/s 2023-03-10T01:06:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36471.504800 secs 2023-03-10T01:06:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:06:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:06:01.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:01 smithi181 ceph-mon[94189]: pgmap v19961: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 678 KiB/s rd, 664 KiB/s wr, 10 op/s 2023-03-10T01:06:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36471.504800 secs 2023-03-10T01:06:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:06:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:06:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:01 smithi038 ceph-mon[92422]: pgmap v19961: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 678 KiB/s rd, 664 KiB/s wr, 10 op/s 2023-03-10T01:06:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36935.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:06:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36473.966080 secs 2023-03-10T01:06:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:03 smithi122 ceph-mon[93869]: pgmap v19962: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 876 KiB/s rd, 951 KiB/s wr, 15 op/s 2023-03-10T01:06:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36473.966080 secs 2023-03-10T01:06:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:03 smithi181 ceph-mon[94189]: pgmap v19962: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 876 KiB/s rd, 951 KiB/s wr, 15 op/s 2023-03-10T01:06:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36473.966080 secs 2023-03-10T01:06:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:03 smithi038 ceph-mon[92422]: pgmap v19962: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 876 KiB/s rd, 951 KiB/s wr, 15 op/s 2023-03-10T01:06:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:05 smithi122 ceph-mon[93869]: pgmap v19963: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 650 KiB/s rd, 730 KiB/s wr, 23 op/s 2023-03-10T01:06:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:05 smithi181 ceph-mon[94189]: pgmap v19963: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 650 KiB/s rd, 730 KiB/s wr, 23 op/s 2023-03-10T01:06:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:05 smithi038 ceph-mon[92422]: pgmap v19963: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 650 KiB/s rd, 730 KiB/s wr, 23 op/s 2023-03-10T01:06:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36476.504860 secs 2023-03-10T01:06:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36476.504860 secs 2023-03-10T01:06:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36476.504860 secs 2023-03-10T01:06:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:07 smithi038 ceph-mon[92422]: pgmap v19964: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 849 KiB/s rd, 848 KiB/s wr, 26 op/s 2023-03-10T01:06:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:07 smithi122 ceph-mon[93869]: pgmap v19964: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 849 KiB/s rd, 848 KiB/s wr, 26 op/s 2023-03-10T01:06:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:07 smithi181 ceph-mon[94189]: pgmap v19964: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 849 KiB/s rd, 848 KiB/s wr, 26 op/s 2023-03-10T01:06:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36478.966164 secs 2023-03-10T01:06:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36478.966164 secs 2023-03-10T01:06:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36478.966164 secs 2023-03-10T01:06:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36940.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:06:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:09 smithi038 ceph-mon[92422]: pgmap v19965: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 814 KiB/s rd, 816 KiB/s wr, 23 op/s 2023-03-10T01:06:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:09 smithi122 ceph-mon[93869]: pgmap v19965: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 814 KiB/s rd, 816 KiB/s wr, 23 op/s 2023-03-10T01:06:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:09 smithi181 ceph-mon[94189]: pgmap v19965: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 814 KiB/s rd, 816 KiB/s wr, 23 op/s 2023-03-10T01:06:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36481.504938 secs 2023-03-10T01:06:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:11 smithi122 ceph-mon[93869]: pgmap v19966: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 786 KiB/s wr, 22 op/s 2023-03-10T01:06:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36481.504938 secs 2023-03-10T01:06:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:11 smithi181 ceph-mon[94189]: pgmap v19966: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 786 KiB/s wr, 22 op/s 2023-03-10T01:06:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36481.504938 secs 2023-03-10T01:06:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:11 smithi038 ceph-mon[92422]: pgmap v19966: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 786 KiB/s wr, 22 op/s 2023-03-10T01:06:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36945.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:06:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36483.966252 secs 2023-03-10T01:06:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:13 smithi122 ceph-mon[93869]: pgmap v19967: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 664 KiB/s rd, 675 KiB/s wr, 24 op/s 2023-03-10T01:06:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36483.966252 secs 2023-03-10T01:06:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:13 smithi181 ceph-mon[94189]: pgmap v19967: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 664 KiB/s rd, 675 KiB/s wr, 24 op/s 2023-03-10T01:06:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36483.966252 secs 2023-03-10T01:06:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:13 smithi038 ceph-mon[92422]: pgmap v19967: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 664 KiB/s rd, 675 KiB/s wr, 24 op/s 2023-03-10T01:06:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:15 smithi122 ceph-mon[93869]: pgmap v19968: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 435 KiB/s rd, 356 KiB/s wr, 19 op/s 2023-03-10T01:06:15.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:15 smithi181 ceph-mon[94189]: pgmap v19968: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 435 KiB/s rd, 356 KiB/s wr, 19 op/s 2023-03-10T01:06:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:15 smithi038 ceph-mon[92422]: pgmap v19968: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 435 KiB/s rd, 356 KiB/s wr, 19 op/s 2023-03-10T01:06:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36486.504994 secs 2023-03-10T01:06:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:06:16.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:06:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36486.504994 secs 2023-03-10T01:06:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:06:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:06:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36486.504994 secs 2023-03-10T01:06:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:06:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:06:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:06:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:06:17.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:17 smithi038 ceph-mon[92422]: pgmap v19969: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 316 KiB/s wr, 10 op/s 2023-03-10T01:06:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:06:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:06:17.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:17 smithi122 ceph-mon[93869]: pgmap v19969: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 316 KiB/s wr, 10 op/s 2023-03-10T01:06:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:06:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:06:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:17 smithi181 ceph-mon[94189]: pgmap v19969: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 316 KiB/s wr, 10 op/s 2023-03-10T01:06:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36488.966351 secs 2023-03-10T01:06:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36488.966351 secs 2023-03-10T01:06:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36488.966351 secs 2023-03-10T01:06:18.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36950.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:06:19.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:19 smithi122 ceph-mon[93869]: pgmap v19970: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 209 KiB/s wr, 8 op/s 2023-03-10T01:06:19.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:19 smithi181 ceph-mon[94189]: pgmap v19970: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 209 KiB/s wr, 8 op/s 2023-03-10T01:06:19.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:19 smithi038 ceph-mon[92422]: pgmap v19970: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 209 KiB/s wr, 8 op/s 2023-03-10T01:06:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:06:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:06:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:06:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:06:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:06:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:06:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36491.505051 secs 2023-03-10T01:06:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:06:21.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:06:21.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:21 smithi122 ceph-mon[93869]: pgmap v19971: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 209 KiB/s wr, 8 op/s 2023-03-10T01:06:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36491.505051 secs 2023-03-10T01:06:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:06:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:06:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:21 smithi181 ceph-mon[94189]: pgmap v19971: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 209 KiB/s wr, 8 op/s 2023-03-10T01:06:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36491.505051 secs 2023-03-10T01:06:21.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:06:21.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:06:21.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:21 smithi038 ceph-mon[92422]: pgmap v19971: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 209 KiB/s wr, 8 op/s 2023-03-10T01:06:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36955.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:06:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36493.966409 secs 2023-03-10T01:06:23.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:23 smithi122 ceph-mon[93869]: pgmap v19972: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 574 KiB/s rd, 622 KiB/s wr, 13 op/s 2023-03-10T01:06:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36493.966409 secs 2023-03-10T01:06:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:23 smithi181 ceph-mon[94189]: pgmap v19972: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 574 KiB/s rd, 622 KiB/s wr, 13 op/s 2023-03-10T01:06:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36493.966409 secs 2023-03-10T01:06:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:23 smithi038 ceph-mon[92422]: pgmap v19972: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 574 KiB/s rd, 622 KiB/s wr, 13 op/s 2023-03-10T01:06:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:06:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:25 smithi038 ceph-mon[92422]: pgmap v19973: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 478 KiB/s wr, 11 op/s 2023-03-10T01:06:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:06:25.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:06:25.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:06:25.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:06:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:06:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:25 smithi122 ceph-mon[93869]: pgmap v19973: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 478 KiB/s wr, 11 op/s 2023-03-10T01:06:25.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:06:25.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:06:25.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:06:25.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:06:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:06:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:25 smithi181 ceph-mon[94189]: pgmap v19973: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 478 KiB/s wr, 11 op/s 2023-03-10T01:06:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:06:25.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:06:25.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:06:25.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:06:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36496.505109 secs 2023-03-10T01:06:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:26 smithi181 ceph-mon[94189]: osdmap e1463: 12 total, 12 up, 12 in 2023-03-10T01:06:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:26 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:06:26.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:26 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:06:26.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:26 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:06:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36496.505109 secs 2023-03-10T01:06:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:26 smithi038 ceph-mon[92422]: osdmap e1463: 12 total, 12 up, 12 in 2023-03-10T01:06:26.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:26 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:06:26.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:26 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:06:26.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:26 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:06:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36496.505109 secs 2023-03-10T01:06:26.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:26 smithi122 ceph-mon[93869]: osdmap e1463: 12 total, 12 up, 12 in 2023-03-10T01:06:26.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:26 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:06:26.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:26 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:06:26.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:26 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:06:27.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:27 smithi038 ceph-mon[92422]: pgmap v19975: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 612 KiB/s wr, 13 op/s 2023-03-10T01:06:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:27 smithi122 ceph-mon[93869]: pgmap v19975: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 612 KiB/s wr, 13 op/s 2023-03-10T01:06:27.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:27 smithi181 ceph-mon[94189]: pgmap v19975: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 612 KiB/s wr, 13 op/s 2023-03-10T01:06:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36498.966472 secs 2023-03-10T01:06:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36498.966472 secs 2023-03-10T01:06:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36498.966472 secs 2023-03-10T01:06:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36960.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:06:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:29 smithi038 ceph-mon[92422]: pgmap v19976: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1003 KiB/s rd, 1016 KiB/s wr, 14 op/s 2023-03-10T01:06:29.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:29 smithi122 ceph-mon[93869]: pgmap v19976: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1003 KiB/s rd, 1016 KiB/s wr, 14 op/s 2023-03-10T01:06:29.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:29 smithi181 ceph-mon[94189]: pgmap v19976: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1003 KiB/s rd, 1016 KiB/s wr, 14 op/s 2023-03-10T01:06:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36501.505168 secs 2023-03-10T01:06:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:31 smithi122 ceph-mon[93869]: pgmap v19977: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1003 KiB/s rd, 1016 KiB/s wr, 14 op/s 2023-03-10T01:06:31.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:31 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:06:31.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:31 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:06:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36501.505168 secs 2023-03-10T01:06:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:31 smithi181 ceph-mon[94189]: pgmap v19977: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1003 KiB/s rd, 1016 KiB/s wr, 14 op/s 2023-03-10T01:06:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:31 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:06:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:31 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:06:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36501.505168 secs 2023-03-10T01:06:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:31 smithi038 ceph-mon[92422]: pgmap v19977: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1003 KiB/s rd, 1016 KiB/s wr, 14 op/s 2023-03-10T01:06:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:31 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:06:32.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:31 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:06:33.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36965.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:06:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36503.966531 secs 2023-03-10T01:06:33.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:33 smithi122 ceph-mon[93869]: pgmap v19978: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 521 KiB/s rd, 520 KiB/s wr, 12 op/s 2023-03-10T01:06:33.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:33 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:06:33.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:33 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:06:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36503.966531 secs 2023-03-10T01:06:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:33 smithi181 ceph-mon[94189]: pgmap v19978: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 521 KiB/s rd, 520 KiB/s wr, 12 op/s 2023-03-10T01:06:33.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:33 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:06:33.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:33 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:06:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36503.966531 secs 2023-03-10T01:06:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:33 smithi038 ceph-mon[92422]: pgmap v19978: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 521 KiB/s rd, 520 KiB/s wr, 12 op/s 2023-03-10T01:06:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:33 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:06:34.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:33 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:06:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:35 smithi181 ceph-mon[94189]: osdmap e1464: 12 total, 12 up, 12 in 2023-03-10T01:06:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:35 smithi181 ceph-mon[94189]: pgmap v19980: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 525 KiB/s rd, 524 KiB/s wr, 13 op/s 2023-03-10T01:06:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:35 smithi038 ceph-mon[92422]: osdmap e1464: 12 total, 12 up, 12 in 2023-03-10T01:06:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:35 smithi038 ceph-mon[92422]: pgmap v19980: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 525 KiB/s rd, 524 KiB/s wr, 13 op/s 2023-03-10T01:06:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:35 smithi122 ceph-mon[93869]: osdmap e1464: 12 total, 12 up, 12 in 2023-03-10T01:06:35.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:35 smithi122 ceph-mon[93869]: pgmap v19980: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 525 KiB/s rd, 524 KiB/s wr, 13 op/s 2023-03-10T01:06:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36506.505227 secs 2023-03-10T01:06:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36506.505227 secs 2023-03-10T01:06:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36506.505227 secs 2023-03-10T01:06:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:37 smithi038 ceph-mon[92422]: pgmap v19981: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 906 KiB/s rd, 904 KiB/s wr, 12 op/s 2023-03-10T01:06:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:37 smithi122 ceph-mon[93869]: pgmap v19981: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 906 KiB/s rd, 904 KiB/s wr, 12 op/s 2023-03-10T01:06:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:37 smithi181 ceph-mon[94189]: pgmap v19981: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 906 KiB/s rd, 904 KiB/s wr, 12 op/s 2023-03-10T01:06:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36508.966593 secs 2023-03-10T01:06:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36508.966593 secs 2023-03-10T01:06:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36508.966593 secs 2023-03-10T01:06:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36970.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:06:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:39 smithi122 ceph-mon[93869]: pgmap v19982: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 452 KiB/s wr, 10 op/s 2023-03-10T01:06:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:39 smithi181 ceph-mon[94189]: pgmap v19982: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 452 KiB/s wr, 10 op/s 2023-03-10T01:06:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:39 smithi038 ceph-mon[92422]: pgmap v19982: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 452 KiB/s wr, 10 op/s 2023-03-10T01:06:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:06:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:06:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:06:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:06:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:06:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:06:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36511.505283 secs 2023-03-10T01:06:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:06:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:06:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:41 smithi122 ceph-mon[93869]: pgmap v19983: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 452 KiB/s wr, 10 op/s 2023-03-10T01:06:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36511.505283 secs 2023-03-10T01:06:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:06:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:06:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:41 smithi181 ceph-mon[94189]: pgmap v19983: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 452 KiB/s wr, 10 op/s 2023-03-10T01:06:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36511.505283 secs 2023-03-10T01:06:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:06:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:06:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:41 smithi038 ceph-mon[92422]: pgmap v19983: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 452 KiB/s wr, 10 op/s 2023-03-10T01:06:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36975.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:06:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36513.966676 secs 2023-03-10T01:06:43.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:43 smithi122 ceph-mon[93869]: pgmap v19984: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 451 KiB/s wr, 10 op/s 2023-03-10T01:06:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36513.966676 secs 2023-03-10T01:06:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:43 smithi181 ceph-mon[94189]: pgmap v19984: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 451 KiB/s wr, 10 op/s 2023-03-10T01:06:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36513.966676 secs 2023-03-10T01:06:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:43 smithi038 ceph-mon[92422]: pgmap v19984: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 451 KiB/s wr, 10 op/s 2023-03-10T01:06:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:45 smithi038 ceph-mon[92422]: pgmap v19985: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 434 KiB/s wr, 9 op/s 2023-03-10T01:06:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:45 smithi122 ceph-mon[93869]: pgmap v19985: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 434 KiB/s wr, 9 op/s 2023-03-10T01:06:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:45 smithi181 ceph-mon[94189]: pgmap v19985: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 434 KiB/s wr, 9 op/s 2023-03-10T01:06:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36516.505354 secs 2023-03-10T01:06:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36516.505354 secs 2023-03-10T01:06:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36516.505354 secs 2023-03-10T01:06:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:47 smithi038 ceph-mon[92422]: pgmap v19986: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 899 KiB/s rd, 763 KiB/s wr, 11 op/s 2023-03-10T01:06:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:47 smithi122 ceph-mon[93869]: pgmap v19986: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 899 KiB/s rd, 763 KiB/s wr, 11 op/s 2023-03-10T01:06:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:47 smithi181 ceph-mon[94189]: pgmap v19986: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 899 KiB/s rd, 763 KiB/s wr, 11 op/s 2023-03-10T01:06:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36518.966748 secs 2023-03-10T01:06:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36518.966748 secs 2023-03-10T01:06:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36518.966748 secs 2023-03-10T01:06:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36980.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:06:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:49 smithi122 ceph-mon[93869]: pgmap v19987: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 523 KiB/s rd, 388 KiB/s wr, 8 op/s 2023-03-10T01:06:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:49 smithi181 ceph-mon[94189]: pgmap v19987: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 523 KiB/s rd, 388 KiB/s wr, 8 op/s 2023-03-10T01:06:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:49 smithi038 ceph-mon[92422]: pgmap v19987: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 523 KiB/s rd, 388 KiB/s wr, 8 op/s 2023-03-10T01:06:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36521.505416 secs 2023-03-10T01:06:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:51 smithi122 ceph-mon[93869]: pgmap v19988: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 523 KiB/s rd, 387 KiB/s wr, 7 op/s 2023-03-10T01:06:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:06:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:06:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36521.505416 secs 2023-03-10T01:06:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:51 smithi181 ceph-mon[94189]: pgmap v19988: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 523 KiB/s rd, 387 KiB/s wr, 7 op/s 2023-03-10T01:06:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:06:51.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:06:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36521.505416 secs 2023-03-10T01:06:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:51 smithi038 ceph-mon[92422]: pgmap v19988: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 523 KiB/s rd, 387 KiB/s wr, 7 op/s 2023-03-10T01:06:52.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:06:52.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:06:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36985.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:06:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36523.966836 secs 2023-03-10T01:06:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:53 smithi122 ceph-mon[93869]: pgmap v19989: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 724 KiB/s rd, 588 KiB/s wr, 10 op/s 2023-03-10T01:06:53.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:06:53.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:06:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36523.966836 secs 2023-03-10T01:06:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:53 smithi181 ceph-mon[94189]: pgmap v19989: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 724 KiB/s rd, 588 KiB/s wr, 10 op/s 2023-03-10T01:06:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:06:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:06:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36523.966836 secs 2023-03-10T01:06:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:53 smithi038 ceph-mon[92422]: pgmap v19989: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 724 KiB/s rd, 588 KiB/s wr, 10 op/s 2023-03-10T01:06:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:06:54.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:06:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:55 smithi038 ceph-mon[92422]: pgmap v19990: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 752 KiB/s rd, 617 KiB/s wr, 9 op/s 2023-03-10T01:06:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:55 smithi122 ceph-mon[93869]: pgmap v19990: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 752 KiB/s rd, 617 KiB/s wr, 9 op/s 2023-03-10T01:06:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:55 smithi181 ceph-mon[94189]: pgmap v19990: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 752 KiB/s rd, 617 KiB/s wr, 9 op/s 2023-03-10T01:06:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36526.505493 secs 2023-03-10T01:06:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36526.505493 secs 2023-03-10T01:06:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36526.505493 secs 2023-03-10T01:06:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:57 smithi038 ceph-mon[92422]: pgmap v19991: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 952 KiB/s rd, 815 KiB/s wr, 11 op/s 2023-03-10T01:06:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:57 smithi122 ceph-mon[93869]: pgmap v19991: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 952 KiB/s rd, 815 KiB/s wr, 11 op/s 2023-03-10T01:06:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:57 smithi181 ceph-mon[94189]: pgmap v19991: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 952 KiB/s rd, 815 KiB/s wr, 11 op/s 2023-03-10T01:06:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:58 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:06:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:58 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:06:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:58 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:06:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36990.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:06:59.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:59 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:06:59.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:06:59.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:06:59.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:06:59 smithi038 ceph-mon[92422]: pgmap v19992: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 458 KiB/s wr, 10 op/s 2023-03-10T01:06:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:59 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:06:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:06:59.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:06:59.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:06:59 smithi122 ceph-mon[93869]: pgmap v19992: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 458 KiB/s wr, 10 op/s 2023-03-10T01:06:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:59 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:06:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:06:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:06:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:06:59 smithi181 ceph-mon[94189]: pgmap v19992: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 458 KiB/s wr, 10 op/s 2023-03-10T01:07:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:07:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:07:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:07:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:07:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:07:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:07:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36531.505548 secs 2023-03-10T01:07:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:07:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:07:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:01 smithi122 ceph-mon[93869]: pgmap v19993: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 472 KiB/s rd, 457 KiB/s wr, 9 op/s 2023-03-10T01:07:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36531.505548 secs 2023-03-10T01:07:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:07:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:07:01.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:01 smithi181 ceph-mon[94189]: pgmap v19993: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 472 KiB/s rd, 457 KiB/s wr, 9 op/s 2023-03-10T01:07:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36531.505548 secs 2023-03-10T01:07:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:07:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:07:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:01 smithi038 ceph-mon[92422]: pgmap v19993: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 472 KiB/s rd, 457 KiB/s wr, 9 op/s 2023-03-10T01:07:03.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 36995.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:07:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:03 smithi122 ceph-mon[93869]: pgmap v19994: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 838 KiB/s rd, 986 KiB/s wr, 15 op/s 2023-03-10T01:07:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:03 smithi181 ceph-mon[94189]: pgmap v19994: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 838 KiB/s rd, 986 KiB/s wr, 15 op/s 2023-03-10T01:07:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:03 smithi038 ceph-mon[92422]: pgmap v19994: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 838 KiB/s rd, 986 KiB/s wr, 15 op/s 2023-03-10T01:07:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:05 smithi038 ceph-mon[92422]: pgmap v19995: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 673 KiB/s rd, 827 KiB/s wr, 23 op/s 2023-03-10T01:07:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:05 smithi122 ceph-mon[93869]: pgmap v19995: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 673 KiB/s rd, 827 KiB/s wr, 23 op/s 2023-03-10T01:07:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:05 smithi181 ceph-mon[94189]: pgmap v19995: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 673 KiB/s rd, 827 KiB/s wr, 23 op/s 2023-03-10T01:07:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36536.505639 secs 2023-03-10T01:07:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36536.505639 secs 2023-03-10T01:07:06.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36536.505639 secs 2023-03-10T01:07:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:07 smithi122 ceph-mon[93869]: pgmap v19996: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 797 KiB/s wr, 26 op/s 2023-03-10T01:07:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:07 smithi181 ceph-mon[94189]: pgmap v19996: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 797 KiB/s wr, 26 op/s 2023-03-10T01:07:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:07 smithi038 ceph-mon[92422]: pgmap v19996: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 797 KiB/s wr, 26 op/s 2023-03-10T01:07:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:07:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:07:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:07:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:07:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:07:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:07:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37000.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:07:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:07:09.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:07:09.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:09 smithi122 ceph-mon[93869]: pgmap v19997: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 993 KiB/s rd, 600 KiB/s wr, 23 op/s 2023-03-10T01:07:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:07:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:07:09.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:09 smithi181 ceph-mon[94189]: pgmap v19997: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 993 KiB/s rd, 600 KiB/s wr, 23 op/s 2023-03-10T01:07:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:07:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:07:09.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:09 smithi038 ceph-mon[92422]: pgmap v19997: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 993 KiB/s rd, 600 KiB/s wr, 23 op/s 2023-03-10T01:07:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36541.505697 secs 2023-03-10T01:07:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:11 smithi122 ceph-mon[93869]: pgmap v19998: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 689 KiB/s wr, 22 op/s 2023-03-10T01:07:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:11 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:07:11.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:11 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:07:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36541.505697 secs 2023-03-10T01:07:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:11 smithi181 ceph-mon[94189]: pgmap v19998: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 689 KiB/s wr, 22 op/s 2023-03-10T01:07:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:11 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:07:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:11 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:07:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36541.505697 secs 2023-03-10T01:07:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:11 smithi038 ceph-mon[92422]: pgmap v19998: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 689 KiB/s wr, 22 op/s 2023-03-10T01:07:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:11 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:07:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:11 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:07:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37005.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:07:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:13 smithi122 ceph-mon[93869]: pgmap v19999: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 689 KiB/s wr, 24 op/s 2023-03-10T01:07:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:13 smithi181 ceph-mon[94189]: pgmap v19999: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 689 KiB/s wr, 24 op/s 2023-03-10T01:07:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:13 smithi038 ceph-mon[92422]: pgmap v19999: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 689 KiB/s wr, 24 op/s 2023-03-10T01:07:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:15 smithi038 ceph-mon[92422]: pgmap v20000: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 874 KiB/s rd, 328 KiB/s wr, 19 op/s 2023-03-10T01:07:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:15 smithi122 ceph-mon[93869]: pgmap v20000: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 874 KiB/s rd, 328 KiB/s wr, 19 op/s 2023-03-10T01:07:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:15 smithi181 ceph-mon[94189]: pgmap v20000: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 874 KiB/s rd, 328 KiB/s wr, 19 op/s 2023-03-10T01:07:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36546.505786 secs 2023-03-10T01:07:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36546.505786 secs 2023-03-10T01:07:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36546.505786 secs 2023-03-10T01:07:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:17 smithi122 ceph-mon[93869]: pgmap v20001: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 838 KiB/s rd, 288 KiB/s wr, 10 op/s 2023-03-10T01:07:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:17 smithi181 ceph-mon[94189]: pgmap v20001: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 838 KiB/s rd, 288 KiB/s wr, 10 op/s 2023-03-10T01:07:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:17 smithi038 ceph-mon[92422]: pgmap v20001: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 838 KiB/s rd, 288 KiB/s wr, 10 op/s 2023-03-10T01:07:18.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37010.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:07:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:19 smithi122 ceph-mon[93869]: pgmap v20002: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 461 KiB/s wr, 8 op/s 2023-03-10T01:07:19.984 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:19 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:07:19.984 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:19 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:07:19.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:19 smithi181 ceph-mon[94189]: pgmap v20002: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 461 KiB/s wr, 8 op/s 2023-03-10T01:07:19.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:19 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:07:19.985 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:19 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:07:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:19 smithi038 ceph-mon[92422]: pgmap v20002: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 461 KiB/s wr, 8 op/s 2023-03-10T01:07:20.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:19 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:07:20.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:19 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:07:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36551.505845 secs 2023-03-10T01:07:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:21 smithi122 ceph-mon[93869]: pgmap v20003: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 515 KiB/s wr, 8 op/s 2023-03-10T01:07:21.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:07:21.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:07:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36551.505845 secs 2023-03-10T01:07:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:21 smithi181 ceph-mon[94189]: pgmap v20003: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 515 KiB/s wr, 8 op/s 2023-03-10T01:07:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:07:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:07:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36551.505845 secs 2023-03-10T01:07:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:21 smithi038 ceph-mon[92422]: pgmap v20003: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 515 KiB/s wr, 8 op/s 2023-03-10T01:07:22.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:07:22.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:07:23.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37015.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:07:23.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:23 smithi122 ceph-mon[93869]: pgmap v20004: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 407 KiB/s wr, 11 op/s 2023-03-10T01:07:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:23 smithi181 ceph-mon[94189]: pgmap v20004: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 407 KiB/s wr, 11 op/s 2023-03-10T01:07:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:23 smithi038 ceph-mon[92422]: pgmap v20004: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 407 KiB/s wr, 11 op/s 2023-03-10T01:07:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:25 smithi122 ceph-mon[93869]: pgmap v20005: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 412 KiB/s wr, 11 op/s 2023-03-10T01:07:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:25 smithi122 ceph-mon[93869]: osdmap e1465: 12 total, 12 up, 12 in 2023-03-10T01:07:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:25 smithi181 ceph-mon[94189]: pgmap v20005: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 412 KiB/s wr, 11 op/s 2023-03-10T01:07:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:25 smithi181 ceph-mon[94189]: osdmap e1465: 12 total, 12 up, 12 in 2023-03-10T01:07:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:25 smithi038 ceph-mon[92422]: pgmap v20005: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 412 KiB/s wr, 11 op/s 2023-03-10T01:07:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:25 smithi038 ceph-mon[92422]: osdmap e1465: 12 total, 12 up, 12 in 2023-03-10T01:07:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36556.505912 secs 2023-03-10T01:07:26.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:26 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:07:26.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:26 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:07:26.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:26 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:07:26.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:26 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:07:26.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36556.505912 secs 2023-03-10T01:07:26.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:26 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:07:26.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:26 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:07:26.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:26 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:07:26.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:26 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:07:26.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36556.505912 secs 2023-03-10T01:07:26.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:26 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:07:26.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:26 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:07:26.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:26 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:07:27.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:26 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:07:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:27 smithi122 ceph-mon[93869]: pgmap v20007: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 312 KiB/s wr, 12 op/s 2023-03-10T01:07:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:27 smithi181 ceph-mon[94189]: pgmap v20007: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 312 KiB/s wr, 12 op/s 2023-03-10T01:07:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:27 smithi038 ceph-mon[92422]: pgmap v20007: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 312 KiB/s wr, 12 op/s 2023-03-10T01:07:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37020.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:07:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:29 smithi122 ceph-mon[93869]: pgmap v20008: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 533 KiB/s rd, 350 KiB/s wr, 11 op/s 2023-03-10T01:07:29.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:29 smithi181 ceph-mon[94189]: pgmap v20008: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 533 KiB/s rd, 350 KiB/s wr, 11 op/s 2023-03-10T01:07:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:29 smithi038 ceph-mon[92422]: pgmap v20008: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 533 KiB/s rd, 350 KiB/s wr, 11 op/s 2023-03-10T01:07:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36561.505997 secs 2023-03-10T01:07:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:31 smithi122 ceph-mon[93869]: pgmap v20009: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 285 KiB/s wr, 10 op/s 2023-03-10T01:07:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36561.505997 secs 2023-03-10T01:07:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:31 smithi181 ceph-mon[94189]: pgmap v20009: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 285 KiB/s wr, 10 op/s 2023-03-10T01:07:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36561.505997 secs 2023-03-10T01:07:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:31 smithi038 ceph-mon[92422]: pgmap v20009: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 285 KiB/s wr, 10 op/s 2023-03-10T01:07:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37025.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:07:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36563.967420 secs 2023-03-10T01:07:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:33 smithi122 ceph-mon[93869]: pgmap v20010: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 652 KiB/s rd, 457 KiB/s wr, 10 op/s 2023-03-10T01:07:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:33 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:07:33.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:33 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:07:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36563.967420 secs 2023-03-10T01:07:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:33 smithi181 ceph-mon[94189]: pgmap v20010: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 652 KiB/s rd, 457 KiB/s wr, 10 op/s 2023-03-10T01:07:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:33 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:07:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:33 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:07:34.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36563.967420 secs 2023-03-10T01:07:34.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:33 smithi038 ceph-mon[92422]: pgmap v20010: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 652 KiB/s rd, 457 KiB/s wr, 10 op/s 2023-03-10T01:07:34.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:33 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:07:34.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:33 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:07:34.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:34 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:07:34.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:34 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:07:34.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:34 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:07:34.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:34 smithi122 ceph-mon[93869]: osdmap e1466: 12 total, 12 up, 12 in 2023-03-10T01:07:34.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:34 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:07:34.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:34 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:07:34.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:34 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:07:34.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:34 smithi181 ceph-mon[94189]: osdmap e1466: 12 total, 12 up, 12 in 2023-03-10T01:07:34.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:34 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:07:34.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:34 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:07:35.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:34 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:07:35.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:34 smithi038 ceph-mon[92422]: osdmap e1466: 12 total, 12 up, 12 in 2023-03-10T01:07:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:35 smithi122 ceph-mon[93869]: pgmap v20012: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 490 KiB/s rd, 488 KiB/s wr, 10 op/s 2023-03-10T01:07:35.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:35 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:07:35.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:35 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:07:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:35 smithi181 ceph-mon[94189]: pgmap v20012: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 490 KiB/s rd, 488 KiB/s wr, 10 op/s 2023-03-10T01:07:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:35 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:07:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:35 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:07:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:35 smithi038 ceph-mon[92422]: pgmap v20012: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 490 KiB/s rd, 488 KiB/s wr, 10 op/s 2023-03-10T01:07:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:35 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:07:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:35 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:07:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36566.506058 secs 2023-03-10T01:07:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36566.506058 secs 2023-03-10T01:07:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36566.506058 secs 2023-03-10T01:07:37.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:37 smithi122 ceph-mon[93869]: pgmap v20013: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 822 KiB/s rd, 820 KiB/s wr, 11 op/s 2023-03-10T01:07:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:37 smithi181 ceph-mon[94189]: pgmap v20013: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 822 KiB/s rd, 820 KiB/s wr, 11 op/s 2023-03-10T01:07:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:37 smithi038 ceph-mon[92422]: pgmap v20013: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 822 KiB/s rd, 820 KiB/s wr, 11 op/s 2023-03-10T01:07:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37030.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:07:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36568.967486 secs 2023-03-10T01:07:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36568.967486 secs 2023-03-10T01:07:39.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36568.967486 secs 2023-03-10T01:07:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:39 smithi038 ceph-mon[92422]: pgmap v20014: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 576 KiB/s rd, 574 KiB/s wr, 10 op/s 2023-03-10T01:07:40.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:39 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:07:40.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:39 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:07:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:39 smithi122 ceph-mon[93869]: pgmap v20014: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 576 KiB/s rd, 574 KiB/s wr, 10 op/s 2023-03-10T01:07:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:39 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:07:40.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:39 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:07:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:39 smithi181 ceph-mon[94189]: pgmap v20014: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 576 KiB/s rd, 574 KiB/s wr, 10 op/s 2023-03-10T01:07:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:39 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:07:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:39 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:07:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36571.506134 secs 2023-03-10T01:07:42.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:41 smithi122 ceph-mon[93869]: pgmap v20015: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 757 KiB/s rd, 837 KiB/s wr, 10 op/s 2023-03-10T01:07:42.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:07:42.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:07:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36571.506134 secs 2023-03-10T01:07:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:41 smithi181 ceph-mon[94189]: pgmap v20015: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 757 KiB/s rd, 837 KiB/s wr, 10 op/s 2023-03-10T01:07:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:07:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:07:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36571.506134 secs 2023-03-10T01:07:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:41 smithi038 ceph-mon[92422]: pgmap v20015: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 757 KiB/s rd, 837 KiB/s wr, 10 op/s 2023-03-10T01:07:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:07:42.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:07:43.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36573.967577 secs 2023-03-10T01:07:43.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36573.967577 secs 2023-03-10T01:07:43.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36573.967577 secs 2023-03-10T01:07:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37035.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:07:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:43 smithi122 ceph-mon[93869]: pgmap v20016: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 652 KiB/s wr, 10 op/s 2023-03-10T01:07:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:43 smithi181 ceph-mon[94189]: pgmap v20016: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 652 KiB/s wr, 10 op/s 2023-03-10T01:07:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:43 smithi038 ceph-mon[92422]: pgmap v20016: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 652 KiB/s wr, 10 op/s 2023-03-10T01:07:45.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:45 smithi122 ceph-mon[93869]: pgmap v20017: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 517 KiB/s rd, 593 KiB/s wr, 9 op/s 2023-03-10T01:07:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:45 smithi181 ceph-mon[94189]: pgmap v20017: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 517 KiB/s rd, 593 KiB/s wr, 9 op/s 2023-03-10T01:07:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:45 smithi038 ceph-mon[92422]: pgmap v20017: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 517 KiB/s rd, 593 KiB/s wr, 9 op/s 2023-03-10T01:07:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36576.506217 secs 2023-03-10T01:07:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36576.506217 secs 2023-03-10T01:07:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36576.506217 secs 2023-03-10T01:07:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:47 smithi122 ceph-mon[93869]: pgmap v20018: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 855 KiB/s wr, 11 op/s 2023-03-10T01:07:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:47 smithi181 ceph-mon[94189]: pgmap v20018: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 855 KiB/s wr, 11 op/s 2023-03-10T01:07:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:47 smithi038 ceph-mon[92422]: pgmap v20018: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 855 KiB/s wr, 11 op/s 2023-03-10T01:07:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37040.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:07:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:07:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:07:49.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:49 smithi122 ceph-mon[93869]: pgmap v20019: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 835 KiB/s rd, 561 KiB/s wr, 9 op/s 2023-03-10T01:07:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:07:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:07:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:49 smithi181 ceph-mon[94189]: pgmap v20019: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 835 KiB/s rd, 561 KiB/s wr, 9 op/s 2023-03-10T01:07:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:07:49.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:07:49.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:49 smithi038 ceph-mon[92422]: pgmap v20019: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 835 KiB/s rd, 561 KiB/s wr, 9 op/s 2023-03-10T01:07:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:50 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:07:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:50 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:07:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:50 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:07:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36581.506277 secs 2023-03-10T01:07:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:51 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:07:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:07:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:07:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:51 smithi122 ceph-mon[93869]: pgmap v20020: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 904 KiB/s rd, 561 KiB/s wr, 8 op/s 2023-03-10T01:07:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36581.506277 secs 2023-03-10T01:07:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:51 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:07:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:07:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:07:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:51 smithi181 ceph-mon[94189]: pgmap v20020: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 904 KiB/s rd, 561 KiB/s wr, 8 op/s 2023-03-10T01:07:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36581.506277 secs 2023-03-10T01:07:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:51 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:07:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:07:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:07:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:51 smithi038 ceph-mon[92422]: pgmap v20020: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 904 KiB/s rd, 561 KiB/s wr, 8 op/s 2023-03-10T01:07:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:07:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:07:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:07:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:07:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:07:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:07:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:53 smithi122 ceph-mon[93869]: pgmap v20021: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 544 KiB/s wr, 11 op/s 2023-03-10T01:07:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:53 smithi181 ceph-mon[94189]: pgmap v20021: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 544 KiB/s wr, 11 op/s 2023-03-10T01:07:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:53 smithi038 ceph-mon[92422]: pgmap v20021: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 544 KiB/s wr, 11 op/s 2023-03-10T01:07:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37045.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:07:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:55 smithi122 ceph-mon[93869]: pgmap v20022: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 574 KiB/s wr, 9 op/s 2023-03-10T01:07:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:55 smithi181 ceph-mon[94189]: pgmap v20022: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 574 KiB/s wr, 9 op/s 2023-03-10T01:07:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:55 smithi038 ceph-mon[92422]: pgmap v20022: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 574 KiB/s wr, 9 op/s 2023-03-10T01:07:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36586.506333 secs 2023-03-10T01:07:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:07:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:07:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36586.506333 secs 2023-03-10T01:07:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:07:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:07:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36586.506333 secs 2023-03-10T01:07:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:07:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:07:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:07:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:07:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:57 smithi122 ceph-mon[93869]: pgmap v20023: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 713 KiB/s wr, 12 op/s 2023-03-10T01:07:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:07:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:07:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:57 smithi181 ceph-mon[94189]: pgmap v20023: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 713 KiB/s wr, 12 op/s 2023-03-10T01:07:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:07:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:07:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:57 smithi038 ceph-mon[92422]: pgmap v20023: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 713 KiB/s wr, 12 op/s 2023-03-10T01:07:58.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:58 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 36588.967793 secs 2023-03-10T01:07:58.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:58 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 36588.967793 secs 2023-03-10T01:07:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:58 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 36588.967793 secs 2023-03-10T01:07:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37050.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:07:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:59 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:07:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:07:59.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:07:59.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:07:59 smithi122 ceph-mon[93869]: pgmap v20024: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 578 KiB/s rd, 402 KiB/s wr, 10 op/s 2023-03-10T01:07:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:59 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:07:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:07:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:07:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:07:59 smithi181 ceph-mon[94189]: pgmap v20024: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 578 KiB/s rd, 402 KiB/s wr, 10 op/s 2023-03-10T01:07:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:59 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:07:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:07:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:07:59.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:07:59 smithi038 ceph-mon[92422]: pgmap v20024: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 578 KiB/s rd, 402 KiB/s wr, 10 op/s 2023-03-10T01:08:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:08:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:08:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:08:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:08:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:08:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:08:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36591.506410 secs 2023-03-10T01:08:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:08:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:08:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:01 smithi122 ceph-mon[93869]: pgmap v20025: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 589 KiB/s rd, 401 KiB/s wr, 10 op/s 2023-03-10T01:08:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36591.506410 secs 2023-03-10T01:08:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:08:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:08:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:01 smithi181 ceph-mon[94189]: pgmap v20025: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 589 KiB/s rd, 401 KiB/s wr, 10 op/s 2023-03-10T01:08:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36591.506410 secs 2023-03-10T01:08:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:08:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:08:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:01 smithi038 ceph-mon[92422]: pgmap v20025: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 589 KiB/s rd, 401 KiB/s wr, 10 op/s 2023-03-10T01:08:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37055.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:08:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:03 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 36593.967872 secs 2023-03-10T01:08:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:03 smithi122 ceph-mon[93869]: pgmap v20026: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 972 KiB/s rd, 743 KiB/s wr, 13 op/s 2023-03-10T01:08:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:03 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 36593.967872 secs 2023-03-10T01:08:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:03 smithi181 ceph-mon[94189]: pgmap v20026: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 972 KiB/s rd, 743 KiB/s wr, 13 op/s 2023-03-10T01:08:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:03 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 36593.967872 secs 2023-03-10T01:08:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:03 smithi038 ceph-mon[92422]: pgmap v20026: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 972 KiB/s rd, 743 KiB/s wr, 13 op/s 2023-03-10T01:08:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:05 smithi122 ceph-mon[93869]: pgmap v20027: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 838 KiB/s rd, 719 KiB/s wr, 22 op/s 2023-03-10T01:08:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:05 smithi181 ceph-mon[94189]: pgmap v20027: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 838 KiB/s rd, 719 KiB/s wr, 22 op/s 2023-03-10T01:08:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:05 smithi038 ceph-mon[92422]: pgmap v20027: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 838 KiB/s rd, 719 KiB/s wr, 22 op/s 2023-03-10T01:08:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36596.506467 secs 2023-03-10T01:08:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36596.506467 secs 2023-03-10T01:08:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36596.506467 secs 2023-03-10T01:08:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:07 smithi122 ceph-mon[93869]: pgmap v20028: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 810 KiB/s rd, 701 KiB/s wr, 26 op/s 2023-03-10T01:08:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:07 smithi181 ceph-mon[94189]: pgmap v20028: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 810 KiB/s rd, 701 KiB/s wr, 26 op/s 2023-03-10T01:08:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:07 smithi038 ceph-mon[92422]: pgmap v20028: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 810 KiB/s rd, 701 KiB/s wr, 26 op/s 2023-03-10T01:08:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:08:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:08:08.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:08 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 36598.967990 secs 2023-03-10T01:08:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:08:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:08:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:08 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 36598.967990 secs 2023-03-10T01:08:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:08:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:08:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:08 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 36598.967990 secs 2023-03-10T01:08:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37060.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:08:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:08:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:08:09.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:09 smithi122 ceph-mon[93869]: pgmap v20029: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 872 KiB/s rd, 902 KiB/s wr, 24 op/s 2023-03-10T01:08:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:08:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:08:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:09 smithi181 ceph-mon[94189]: pgmap v20029: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 872 KiB/s rd, 902 KiB/s wr, 24 op/s 2023-03-10T01:08:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:08:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:08:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:09 smithi038 ceph-mon[92422]: pgmap v20029: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 872 KiB/s rd, 902 KiB/s wr, 24 op/s 2023-03-10T01:08:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36601.506520 secs 2023-03-10T01:08:11.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:11 smithi122 ceph-mon[93869]: pgmap v20030: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 842 KiB/s rd, 872 KiB/s wr, 23 op/s 2023-03-10T01:08:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36601.506520 secs 2023-03-10T01:08:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:11 smithi181 ceph-mon[94189]: pgmap v20030: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 842 KiB/s rd, 872 KiB/s wr, 23 op/s 2023-03-10T01:08:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36601.506520 secs 2023-03-10T01:08:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:11 smithi038 ceph-mon[92422]: pgmap v20030: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 842 KiB/s rd, 872 KiB/s wr, 23 op/s 2023-03-10T01:08:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37065.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:08:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:13 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 36603.968074 secs 2023-03-10T01:08:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:13 smithi122 ceph-mon[93869]: pgmap v20031: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 832 KiB/s rd, 872 KiB/s wr, 24 op/s 2023-03-10T01:08:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:13 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 36603.968074 secs 2023-03-10T01:08:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:13 smithi181 ceph-mon[94189]: pgmap v20031: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 832 KiB/s rd, 872 KiB/s wr, 24 op/s 2023-03-10T01:08:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:13 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 36603.968074 secs 2023-03-10T01:08:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:13 smithi038 ceph-mon[92422]: pgmap v20031: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 832 KiB/s rd, 872 KiB/s wr, 24 op/s 2023-03-10T01:08:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:15 smithi122 ceph-mon[93869]: pgmap v20032: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 854 KiB/s rd, 859 KiB/s wr, 22 op/s 2023-03-10T01:08:15.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:15 smithi181 ceph-mon[94189]: pgmap v20032: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 854 KiB/s rd, 859 KiB/s wr, 22 op/s 2023-03-10T01:08:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:15 smithi038 ceph-mon[92422]: pgmap v20032: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 854 KiB/s rd, 859 KiB/s wr, 22 op/s 2023-03-10T01:08:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36606.506603 secs 2023-03-10T01:08:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:08:16.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:08:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36606.506603 secs 2023-03-10T01:08:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:08:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:08:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36606.506603 secs 2023-03-10T01:08:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:08:16.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:08:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:08:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:08:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:17 smithi122 ceph-mon[93869]: pgmap v20033: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 682 KiB/s rd, 681 KiB/s wr, 11 op/s 2023-03-10T01:08:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:08:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:08:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:17 smithi181 ceph-mon[94189]: pgmap v20033: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 682 KiB/s rd, 681 KiB/s wr, 11 op/s 2023-03-10T01:08:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:08:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:08:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:17 smithi038 ceph-mon[92422]: pgmap v20033: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 682 KiB/s rd, 681 KiB/s wr, 11 op/s 2023-03-10T01:08:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:18 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 36608.968166 secs 2023-03-10T01:08:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:18 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 36608.968166 secs 2023-03-10T01:08:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:18 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 36608.968166 secs 2023-03-10T01:08:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37070.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:08:19.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:19 smithi122 ceph-mon[93869]: pgmap v20034: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 680 KiB/s rd, 928 KiB/s wr, 11 op/s 2023-03-10T01:08:19.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:19 smithi181 ceph-mon[94189]: pgmap v20034: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 680 KiB/s rd, 928 KiB/s wr, 11 op/s 2023-03-10T01:08:19.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:19 smithi038 ceph-mon[92422]: pgmap v20034: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 680 KiB/s rd, 928 KiB/s wr, 11 op/s 2023-03-10T01:08:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:08:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:08:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:08:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:08:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:08:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:08:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36611.506660 secs 2023-03-10T01:08:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:08:21.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:08:21.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:21 smithi122 ceph-mon[93869]: pgmap v20035: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 595 KiB/s rd, 717 KiB/s wr, 10 op/s 2023-03-10T01:08:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36611.506660 secs 2023-03-10T01:08:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:08:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:08:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:21 smithi181 ceph-mon[94189]: pgmap v20035: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 595 KiB/s rd, 717 KiB/s wr, 10 op/s 2023-03-10T01:08:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36611.506660 secs 2023-03-10T01:08:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:08:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:08:21.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:21 smithi038 ceph-mon[92422]: pgmap v20035: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 595 KiB/s rd, 717 KiB/s wr, 10 op/s 2023-03-10T01:08:23.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37075.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:08:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36613.968286 secs 2023-03-10T01:08:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:23 smithi122 ceph-mon[93869]: pgmap v20036: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 596 KiB/s rd, 717 KiB/s wr, 13 op/s 2023-03-10T01:08:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36613.968286 secs 2023-03-10T01:08:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:23 smithi181 ceph-mon[94189]: pgmap v20036: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 596 KiB/s rd, 717 KiB/s wr, 13 op/s 2023-03-10T01:08:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36613.968286 secs 2023-03-10T01:08:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:23 smithi038 ceph-mon[92422]: pgmap v20036: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 596 KiB/s rd, 717 KiB/s wr, 13 op/s 2023-03-10T01:08:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:25 smithi122 ceph-mon[93869]: pgmap v20037: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 841 KiB/s rd, 718 KiB/s wr, 13 op/s 2023-03-10T01:08:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:25 smithi122 ceph-mon[93869]: osdmap e1467: 12 total, 12 up, 12 in 2023-03-10T01:08:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:25 smithi181 ceph-mon[94189]: pgmap v20037: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 841 KiB/s rd, 718 KiB/s wr, 13 op/s 2023-03-10T01:08:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:25 smithi181 ceph-mon[94189]: osdmap e1467: 12 total, 12 up, 12 in 2023-03-10T01:08:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:25 smithi038 ceph-mon[92422]: pgmap v20037: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 841 KiB/s rd, 718 KiB/s wr, 13 op/s 2023-03-10T01:08:26.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:25 smithi038 ceph-mon[92422]: osdmap e1467: 12 total, 12 up, 12 in 2023-03-10T01:08:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36616.506746 secs 2023-03-10T01:08:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:26 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:08:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:26 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:08:26.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:26 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:08:26.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36616.506746 secs 2023-03-10T01:08:26.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:26 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:08:26.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:26 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:08:26.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:26 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:08:26.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36616.506746 secs 2023-03-10T01:08:26.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:26 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:08:26.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:26 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:08:26.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:26 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:08:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:27 smithi122 ceph-mon[93869]: pgmap v20039: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 615 KiB/s rd, 639 KiB/s wr, 14 op/s 2023-03-10T01:08:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:08:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:08:27.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:08:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:27 smithi181 ceph-mon[94189]: pgmap v20039: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 615 KiB/s rd, 639 KiB/s wr, 14 op/s 2023-03-10T01:08:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:08:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:08:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:08:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:27 smithi038 ceph-mon[92422]: pgmap v20039: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 615 KiB/s rd, 639 KiB/s wr, 14 op/s 2023-03-10T01:08:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:08:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:08:28.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:08:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37080.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:08:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36618.968346 secs 2023-03-10T01:08:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36618.968346 secs 2023-03-10T01:08:29.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36618.968346 secs 2023-03-10T01:08:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:29 smithi038 ceph-mon[92422]: pgmap v20040: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 614 KiB/s rd, 365 KiB/s wr, 11 op/s 2023-03-10T01:08:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:29 smithi122 ceph-mon[93869]: pgmap v20040: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 614 KiB/s rd, 365 KiB/s wr, 11 op/s 2023-03-10T01:08:30.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:29 smithi181 ceph-mon[94189]: pgmap v20040: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 614 KiB/s rd, 365 KiB/s wr, 11 op/s 2023-03-10T01:08:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36621.506833 secs 2023-03-10T01:08:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:31 smithi038 ceph-mon[92422]: pgmap v20041: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 663 KiB/s rd, 306 KiB/s wr, 10 op/s 2023-03-10T01:08:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36621.506833 secs 2023-03-10T01:08:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:31 smithi122 ceph-mon[93869]: pgmap v20041: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 663 KiB/s rd, 306 KiB/s wr, 10 op/s 2023-03-10T01:08:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36621.506833 secs 2023-03-10T01:08:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:31 smithi181 ceph-mon[94189]: pgmap v20041: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 663 KiB/s rd, 306 KiB/s wr, 10 op/s 2023-03-10T01:08:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37085.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:08:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36623.968424 secs 2023-03-10T01:08:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:33 smithi122 ceph-mon[93869]: pgmap v20042: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 663 KiB/s rd, 305 KiB/s wr, 10 op/s 2023-03-10T01:08:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36623.968424 secs 2023-03-10T01:08:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:33 smithi181 ceph-mon[94189]: pgmap v20042: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 663 KiB/s rd, 305 KiB/s wr, 10 op/s 2023-03-10T01:08:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36623.968424 secs 2023-03-10T01:08:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:33 smithi038 ceph-mon[92422]: pgmap v20042: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 663 KiB/s rd, 305 KiB/s wr, 10 op/s 2023-03-10T01:08:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:35 smithi122 ceph-mon[93869]: osdmap e1468: 12 total, 12 up, 12 in 2023-03-10T01:08:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:35 smithi122 ceph-mon[93869]: pgmap v20044: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 312 KiB/s wr, 8 op/s 2023-03-10T01:08:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:35 smithi181 ceph-mon[94189]: osdmap e1468: 12 total, 12 up, 12 in 2023-03-10T01:08:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:35 smithi181 ceph-mon[94189]: pgmap v20044: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 312 KiB/s wr, 8 op/s 2023-03-10T01:08:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:35 smithi038 ceph-mon[92422]: osdmap e1468: 12 total, 12 up, 12 in 2023-03-10T01:08:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:35 smithi038 ceph-mon[92422]: pgmap v20044: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 312 KiB/s wr, 8 op/s 2023-03-10T01:08:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36626.506892 secs 2023-03-10T01:08:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:08:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:08:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36626.506892 secs 2023-03-10T01:08:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:08:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:08:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36626.506892 secs 2023-03-10T01:08:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:08:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:08:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:08:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:08:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:37 smithi122 ceph-mon[93869]: pgmap v20045: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 755 KiB/s wr, 10 op/s 2023-03-10T01:08:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:08:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:08:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:37 smithi181 ceph-mon[94189]: pgmap v20045: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 755 KiB/s wr, 10 op/s 2023-03-10T01:08:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:08:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:08:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:37 smithi038 ceph-mon[92422]: pgmap v20045: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 755 KiB/s wr, 10 op/s 2023-03-10T01:08:38.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36628.968547 secs 2023-03-10T01:08:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36628.968547 secs 2023-03-10T01:08:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36628.968547 secs 2023-03-10T01:08:38.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37090.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:08:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:39 smithi122 ceph-mon[93869]: pgmap v20046: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 719 KiB/s wr, 10 op/s 2023-03-10T01:08:39.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:39 smithi181 ceph-mon[94189]: pgmap v20046: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 719 KiB/s wr, 10 op/s 2023-03-10T01:08:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:39 smithi038 ceph-mon[92422]: pgmap v20046: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 719 KiB/s wr, 10 op/s 2023-03-10T01:08:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:08:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:08:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:08:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:08:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:08:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:08:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36631.506949 secs 2023-03-10T01:08:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:08:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:08:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:41 smithi122 ceph-mon[93869]: pgmap v20047: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 856 KiB/s wr, 10 op/s 2023-03-10T01:08:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36631.506949 secs 2023-03-10T01:08:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:08:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:08:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:41 smithi181 ceph-mon[94189]: pgmap v20047: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 856 KiB/s wr, 10 op/s 2023-03-10T01:08:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36631.506949 secs 2023-03-10T01:08:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:08:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:08:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:41 smithi038 ceph-mon[92422]: pgmap v20047: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 856 KiB/s wr, 10 op/s 2023-03-10T01:08:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37095.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:08:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36633.968605 secs 2023-03-10T01:08:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:43 smithi122 ceph-mon[93869]: pgmap v20048: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 856 KiB/s wr, 11 op/s 2023-03-10T01:08:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36633.968605 secs 2023-03-10T01:08:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:43 smithi181 ceph-mon[94189]: pgmap v20048: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 856 KiB/s wr, 11 op/s 2023-03-10T01:08:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36633.968605 secs 2023-03-10T01:08:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:43 smithi038 ceph-mon[92422]: pgmap v20048: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 856 KiB/s wr, 11 op/s 2023-03-10T01:08:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:45 smithi122 ceph-mon[93869]: pgmap v20049: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 12 op/s 2023-03-10T01:08:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:45 smithi181 ceph-mon[94189]: pgmap v20049: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 12 op/s 2023-03-10T01:08:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:45 smithi038 ceph-mon[92422]: pgmap v20049: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 12 op/s 2023-03-10T01:08:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36636.507009 secs 2023-03-10T01:08:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36636.507009 secs 2023-03-10T01:08:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36636.507009 secs 2023-03-10T01:08:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:47 smithi122 ceph-mon[93869]: pgmap v20050: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 989 KiB/s wr, 13 op/s 2023-03-10T01:08:47.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:47 smithi181 ceph-mon[94189]: pgmap v20050: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 989 KiB/s wr, 13 op/s 2023-03-10T01:08:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:47 smithi038 ceph-mon[92422]: pgmap v20050: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 989 KiB/s wr, 13 op/s 2023-03-10T01:08:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:08:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:08:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36638.968691 secs 2023-03-10T01:08:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:08:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:08:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36638.968691 secs 2023-03-10T01:08:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:08:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:08:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36638.968691 secs 2023-03-10T01:08:48.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37100.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:08:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:08:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:08:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:49 smithi122 ceph-mon[93869]: pgmap v20051: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 474 KiB/s wr, 9 op/s 2023-03-10T01:08:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:08:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:08:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:49 smithi181 ceph-mon[94189]: pgmap v20051: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 474 KiB/s wr, 9 op/s 2023-03-10T01:08:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:08:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:08:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:49 smithi038 ceph-mon[92422]: pgmap v20051: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 474 KiB/s wr, 9 op/s 2023-03-10T01:08:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36641.507113 secs 2023-03-10T01:08:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:51 smithi122 ceph-mon[93869]: pgmap v20052: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1000 KiB/s rd, 780 KiB/s wr, 10 op/s 2023-03-10T01:08:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36641.507113 secs 2023-03-10T01:08:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:51 smithi181 ceph-mon[94189]: pgmap v20052: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1000 KiB/s rd, 780 KiB/s wr, 10 op/s 2023-03-10T01:08:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36641.507113 secs 2023-03-10T01:08:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:51 smithi038 ceph-mon[92422]: pgmap v20052: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1000 KiB/s rd, 780 KiB/s wr, 10 op/s 2023-03-10T01:08:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37105.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:08:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36643.968751 secs 2023-03-10T01:08:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:53 smithi122 ceph-mon[93869]: pgmap v20053: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 837 KiB/s rd, 616 KiB/s wr, 11 op/s 2023-03-10T01:08:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36643.968751 secs 2023-03-10T01:08:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:53 smithi181 ceph-mon[94189]: pgmap v20053: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 837 KiB/s rd, 616 KiB/s wr, 11 op/s 2023-03-10T01:08:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36643.968751 secs 2023-03-10T01:08:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:53 smithi038 ceph-mon[92422]: pgmap v20053: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 837 KiB/s rd, 616 KiB/s wr, 11 op/s 2023-03-10T01:08:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:55 smithi122 ceph-mon[93869]: pgmap v20054: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 890 KiB/s rd, 616 KiB/s wr, 10 op/s 2023-03-10T01:08:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:55 smithi181 ceph-mon[94189]: pgmap v20054: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 890 KiB/s rd, 616 KiB/s wr, 10 op/s 2023-03-10T01:08:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:55 smithi038 ceph-mon[92422]: pgmap v20054: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 890 KiB/s rd, 616 KiB/s wr, 10 op/s 2023-03-10T01:08:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36646.507171 secs 2023-03-10T01:08:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:08:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:08:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36646.507171 secs 2023-03-10T01:08:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:08:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:08:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36646.507171 secs 2023-03-10T01:08:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:08:56.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:08:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:08:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:08:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:57 smithi122 ceph-mon[93869]: pgmap v20055: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 742 KiB/s rd, 464 KiB/s wr, 12 op/s 2023-03-10T01:08:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:08:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:08:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:57 smithi181 ceph-mon[94189]: pgmap v20055: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 742 KiB/s rd, 464 KiB/s wr, 12 op/s 2023-03-10T01:08:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:08:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:08:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:57 smithi038 ceph-mon[92422]: pgmap v20055: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 742 KiB/s rd, 464 KiB/s wr, 12 op/s 2023-03-10T01:08:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36648.968912 secs 2023-03-10T01:08:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36648.968912 secs 2023-03-10T01:08:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36648.968912 secs 2023-03-10T01:08:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37110.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:08:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:08:59 smithi122 ceph-mon[93869]: pgmap v20056: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 845 KiB/s rd, 569 KiB/s wr, 11 op/s 2023-03-10T01:08:59.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:08:59 smithi181 ceph-mon[94189]: pgmap v20056: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 845 KiB/s rd, 569 KiB/s wr, 11 op/s 2023-03-10T01:08:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:08:59 smithi038 ceph-mon[92422]: pgmap v20056: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 845 KiB/s rd, 569 KiB/s wr, 11 op/s 2023-03-10T01:09:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:09:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:09:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:09:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:09:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:09:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:09:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36651.507228 secs 2023-03-10T01:09:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:09:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:09:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:01 smithi122 ceph-mon[93869]: pgmap v20057: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 856 KiB/s rd, 569 KiB/s wr, 11 op/s 2023-03-10T01:09:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36651.507228 secs 2023-03-10T01:09:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:09:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:09:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:01 smithi181 ceph-mon[94189]: pgmap v20057: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 856 KiB/s rd, 569 KiB/s wr, 11 op/s 2023-03-10T01:09:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36651.507228 secs 2023-03-10T01:09:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:09:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:09:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:01 smithi038 ceph-mon[92422]: pgmap v20057: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 856 KiB/s rd, 569 KiB/s wr, 11 op/s 2023-03-10T01:09:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37115.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:09:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36653.969001 secs 2023-03-10T01:09:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:03 smithi181 ceph-mon[94189]: pgmap v20058: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 601 KiB/s rd, 531 KiB/s wr, 13 op/s 2023-03-10T01:09:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36653.969001 secs 2023-03-10T01:09:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:03 smithi038 ceph-mon[92422]: pgmap v20058: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 601 KiB/s rd, 531 KiB/s wr, 13 op/s 2023-03-10T01:09:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36653.969001 secs 2023-03-10T01:09:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:03 smithi122 ceph-mon[93869]: pgmap v20058: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 601 KiB/s rd, 531 KiB/s wr, 13 op/s 2023-03-10T01:09:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:05 smithi122 ceph-mon[93869]: pgmap v20059: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 853 KiB/s wr, 22 op/s 2023-03-10T01:09:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:05 smithi181 ceph-mon[94189]: pgmap v20059: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 853 KiB/s wr, 22 op/s 2023-03-10T01:09:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:05 smithi038 ceph-mon[92422]: pgmap v20059: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 853 KiB/s wr, 22 op/s 2023-03-10T01:09:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36656.507322 secs 2023-03-10T01:09:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36656.507322 secs 2023-03-10T01:09:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36656.507322 secs 2023-03-10T01:09:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:07 smithi122 ceph-mon[93869]: pgmap v20060: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 878 KiB/s wr, 26 op/s 2023-03-10T01:09:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:07 smithi181 ceph-mon[94189]: pgmap v20060: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 878 KiB/s wr, 26 op/s 2023-03-10T01:09:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:07 smithi038 ceph-mon[92422]: pgmap v20060: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 878 KiB/s wr, 26 op/s 2023-03-10T01:09:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:09:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:09:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36658.969094 secs 2023-03-10T01:09:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:09:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:09:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36658.969094 secs 2023-03-10T01:09:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:09:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:09:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36658.969094 secs 2023-03-10T01:09:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37120.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:09:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:09:09.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:09:09.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:09 smithi122 ceph-mon[93869]: pgmap v20061: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1017 KiB/s rd, 751 KiB/s wr, 23 op/s 2023-03-10T01:09:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:09:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:09:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:09 smithi181 ceph-mon[94189]: pgmap v20061: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1017 KiB/s rd, 751 KiB/s wr, 23 op/s 2023-03-10T01:09:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:09:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:09:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:09 smithi038 ceph-mon[92422]: pgmap v20061: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1017 KiB/s rd, 751 KiB/s wr, 23 op/s 2023-03-10T01:09:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36661.507411 secs 2023-03-10T01:09:11.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:11 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:09:11.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:11 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:09:11.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:11 smithi122 ceph-mon[93869]: pgmap v20062: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 813 KiB/s wr, 24 op/s 2023-03-10T01:09:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36661.507411 secs 2023-03-10T01:09:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:11 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:09:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:11 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:09:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:11 smithi181 ceph-mon[94189]: pgmap v20062: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 813 KiB/s wr, 24 op/s 2023-03-10T01:09:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36661.507411 secs 2023-03-10T01:09:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:11 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:09:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:11 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:09:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:11 smithi038 ceph-mon[92422]: pgmap v20062: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 813 KiB/s wr, 24 op/s 2023-03-10T01:09:13.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37125.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:09:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36663.969141 secs 2023-03-10T01:09:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:13 smithi038 ceph-mon[92422]: pgmap v20063: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 813 KiB/s wr, 26 op/s 2023-03-10T01:09:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36663.969141 secs 2023-03-10T01:09:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:13 smithi122 ceph-mon[93869]: pgmap v20063: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 813 KiB/s wr, 26 op/s 2023-03-10T01:09:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36663.969141 secs 2023-03-10T01:09:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:13 smithi181 ceph-mon[94189]: pgmap v20063: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 813 KiB/s wr, 26 op/s 2023-03-10T01:09:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:15 smithi122 ceph-mon[93869]: pgmap v20064: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 934 KiB/s rd, 544 KiB/s wr, 22 op/s 2023-03-10T01:09:15.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:15 smithi181 ceph-mon[94189]: pgmap v20064: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 934 KiB/s rd, 544 KiB/s wr, 22 op/s 2023-03-10T01:09:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:15 smithi038 ceph-mon[92422]: pgmap v20064: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 934 KiB/s rd, 544 KiB/s wr, 22 op/s 2023-03-10T01:09:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36666.507496 secs 2023-03-10T01:09:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36666.507496 secs 2023-03-10T01:09:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36666.507496 secs 2023-03-10T01:09:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:17 smithi122 ceph-mon[93869]: pgmap v20065: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 192 KiB/s wr, 13 op/s 2023-03-10T01:09:17.870 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:17 smithi038 ceph-mon[92422]: pgmap v20065: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 192 KiB/s wr, 13 op/s 2023-03-10T01:09:17.871 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:17 smithi181 ceph-mon[94189]: pgmap v20065: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 319 KiB/s rd, 192 KiB/s wr, 13 op/s 2023-03-10T01:09:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36668.969199 secs 2023-03-10T01:09:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36668.969199 secs 2023-03-10T01:09:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36668.969199 secs 2023-03-10T01:09:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37130.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:09:19.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:19 smithi122 ceph-mon[93869]: pgmap v20066: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 178 KiB/s wr, 10 op/s 2023-03-10T01:09:19.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:19 smithi181 ceph-mon[94189]: pgmap v20066: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 178 KiB/s wr, 10 op/s 2023-03-10T01:09:19.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:19 smithi038 ceph-mon[92422]: pgmap v20066: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 178 KiB/s wr, 10 op/s 2023-03-10T01:09:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36671.507553 secs 2023-03-10T01:09:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:09:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:09:21.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:21 smithi122 ceph-mon[93869]: pgmap v20067: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 350 KiB/s wr, 11 op/s 2023-03-10T01:09:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36671.507553 secs 2023-03-10T01:09:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:09:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:09:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:21 smithi181 ceph-mon[94189]: pgmap v20067: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 350 KiB/s wr, 11 op/s 2023-03-10T01:09:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36671.507553 secs 2023-03-10T01:09:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:09:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:09:21.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:21 smithi038 ceph-mon[92422]: pgmap v20067: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 350 KiB/s wr, 11 op/s 2023-03-10T01:09:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37135.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:09:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:23 smithi038 ceph-mon[92422]: pgmap v20068: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 365 KiB/s wr, 13 op/s 2023-03-10T01:09:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:23 smithi122 ceph-mon[93869]: pgmap v20068: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 365 KiB/s wr, 13 op/s 2023-03-10T01:09:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:23 smithi181 ceph-mon[94189]: pgmap v20068: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 365 KiB/s wr, 13 op/s 2023-03-10T01:09:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:25 smithi038 ceph-mon[92422]: pgmap v20069: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 365 KiB/s wr, 11 op/s 2023-03-10T01:09:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:25 smithi038 ceph-mon[92422]: osdmap e1469: 12 total, 12 up, 12 in 2023-03-10T01:09:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:25 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:09:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:25 smithi122 ceph-mon[93869]: pgmap v20069: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 365 KiB/s wr, 11 op/s 2023-03-10T01:09:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:25 smithi122 ceph-mon[93869]: osdmap e1469: 12 total, 12 up, 12 in 2023-03-10T01:09:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:25 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:09:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:25 smithi181 ceph-mon[94189]: pgmap v20069: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 365 KiB/s wr, 11 op/s 2023-03-10T01:09:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:25 smithi181 ceph-mon[94189]: osdmap e1469: 12 total, 12 up, 12 in 2023-03-10T01:09:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:25 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:09:26.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36676.507612 secs 2023-03-10T01:09:27.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:26 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:09:27.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:26 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:09:27.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:26 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:09:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36676.507612 secs 2023-03-10T01:09:27.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:26 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:09:27.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:26 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:09:27.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:26 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:09:27.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36676.507612 secs 2023-03-10T01:09:27.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:26 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:09:27.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:26 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:09:27.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:26 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:09:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:27 smithi038 ceph-mon[92422]: pgmap v20071: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 438 KiB/s wr, 13 op/s 2023-03-10T01:09:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:27 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:09:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:27 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:09:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:09:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:27 smithi122 ceph-mon[93869]: pgmap v20071: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 438 KiB/s wr, 13 op/s 2023-03-10T01:09:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:27 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:09:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:27 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:09:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:09:28.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:27 smithi181 ceph-mon[94189]: pgmap v20071: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 438 KiB/s wr, 13 op/s 2023-03-10T01:09:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:27 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:09:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:27 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:09:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:09:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37140.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:09:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:09:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:09:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:09:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:09:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:09:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:09:29.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:09:29.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:09:29.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:09:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:29 smithi038 ceph-mon[92422]: pgmap v20072: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 599 KiB/s wr, 11 op/s 2023-03-10T01:09:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:29 smithi122 ceph-mon[93869]: pgmap v20072: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 599 KiB/s wr, 11 op/s 2023-03-10T01:09:30.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:29 smithi181 ceph-mon[94189]: pgmap v20072: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 599 KiB/s wr, 11 op/s 2023-03-10T01:09:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36681.507670 secs 2023-03-10T01:09:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:31 smithi122 ceph-mon[93869]: pgmap v20073: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 391 KiB/s wr, 9 op/s 2023-03-10T01:09:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36681.507670 secs 2023-03-10T01:09:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:31 smithi181 ceph-mon[94189]: pgmap v20073: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 391 KiB/s wr, 9 op/s 2023-03-10T01:09:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36681.507670 secs 2023-03-10T01:09:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:31 smithi038 ceph-mon[92422]: pgmap v20073: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 391 KiB/s wr, 9 op/s 2023-03-10T01:09:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37145.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:09:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:33 smithi122 ceph-mon[93869]: pgmap v20074: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 267 KiB/s wr, 9 op/s 2023-03-10T01:09:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:33 smithi181 ceph-mon[94189]: pgmap v20074: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 267 KiB/s wr, 9 op/s 2023-03-10T01:09:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:33 smithi038 ceph-mon[92422]: pgmap v20074: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 267 KiB/s wr, 9 op/s 2023-03-10T01:09:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:35 smithi122 ceph-mon[93869]: osdmap e1470: 12 total, 12 up, 12 in 2023-03-10T01:09:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:35 smithi122 ceph-mon[93869]: pgmap v20076: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 383 KiB/s rd, 269 KiB/s wr, 9 op/s 2023-03-10T01:09:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:35 smithi181 ceph-mon[94189]: osdmap e1470: 12 total, 12 up, 12 in 2023-03-10T01:09:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:35 smithi181 ceph-mon[94189]: pgmap v20076: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 383 KiB/s rd, 269 KiB/s wr, 9 op/s 2023-03-10T01:09:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:35 smithi038 ceph-mon[92422]: osdmap e1470: 12 total, 12 up, 12 in 2023-03-10T01:09:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:35 smithi038 ceph-mon[92422]: pgmap v20076: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 383 KiB/s rd, 269 KiB/s wr, 9 op/s 2023-03-10T01:09:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36686.507765 secs 2023-03-10T01:09:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:09:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:09:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36686.507765 secs 2023-03-10T01:09:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:09:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:09:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36686.507765 secs 2023-03-10T01:09:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:09:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:09:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:09:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:09:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:37 smithi122 ceph-mon[93869]: pgmap v20077: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 902 KiB/s rd, 789 KiB/s wr, 10 op/s 2023-03-10T01:09:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:09:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:09:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:37 smithi181 ceph-mon[94189]: pgmap v20077: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 902 KiB/s rd, 789 KiB/s wr, 10 op/s 2023-03-10T01:09:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:09:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:09:37.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:37 smithi038 ceph-mon[92422]: pgmap v20077: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 902 KiB/s rd, 789 KiB/s wr, 10 op/s 2023-03-10T01:09:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36688.969498 secs 2023-03-10T01:09:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36688.969498 secs 2023-03-10T01:09:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36688.969498 secs 2023-03-10T01:09:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37150.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:09:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:39 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:09:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:39 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:09:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:39 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:09:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:39 smithi122 ceph-mon[93869]: pgmap v20078: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 616 KiB/s wr, 10 op/s 2023-03-10T01:09:39.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:39 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:09:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:39 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:09:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:39 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:09:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:39 smithi181 ceph-mon[94189]: pgmap v20078: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 616 KiB/s wr, 10 op/s 2023-03-10T01:09:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:39 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:09:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:39 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:09:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:39 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:09:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:39 smithi038 ceph-mon[92422]: pgmap v20078: 129 pgs: 129 active+clean; 14 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 616 KiB/s wr, 10 op/s 2023-03-10T01:09:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:09:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:09:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:09:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:09:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:09:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:09:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36691.507822 secs 2023-03-10T01:09:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:09:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:09:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:41 smithi122 ceph-mon[93869]: pgmap v20079: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 949 KiB/s wr, 11 op/s 2023-03-10T01:09:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36691.507822 secs 2023-03-10T01:09:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:09:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:09:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:41 smithi181 ceph-mon[94189]: pgmap v20079: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 949 KiB/s wr, 11 op/s 2023-03-10T01:09:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36691.507822 secs 2023-03-10T01:09:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:09:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:09:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:41 smithi038 ceph-mon[92422]: pgmap v20079: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 949 KiB/s wr, 11 op/s 2023-03-10T01:09:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37155.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:09:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36693.969610 secs 2023-03-10T01:09:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:43 smithi038 ceph-mon[92422]: pgmap v20080: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 915 KiB/s rd, 857 KiB/s wr, 11 op/s 2023-03-10T01:09:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36693.969610 secs 2023-03-10T01:09:44.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:43 smithi122 ceph-mon[93869]: pgmap v20080: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 915 KiB/s rd, 857 KiB/s wr, 11 op/s 2023-03-10T01:09:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36693.969610 secs 2023-03-10T01:09:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:43 smithi181 ceph-mon[94189]: pgmap v20080: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 915 KiB/s rd, 857 KiB/s wr, 11 op/s 2023-03-10T01:09:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:45 smithi122 ceph-mon[93869]: pgmap v20081: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 11 op/s 2023-03-10T01:09:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:45 smithi181 ceph-mon[94189]: pgmap v20081: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 11 op/s 2023-03-10T01:09:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:45 smithi038 ceph-mon[92422]: pgmap v20081: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 11 op/s 2023-03-10T01:09:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36696.507902 secs 2023-03-10T01:09:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36696.507902 secs 2023-03-10T01:09:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36696.507902 secs 2023-03-10T01:09:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:47 smithi122 ceph-mon[93869]: pgmap v20082: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 961 KiB/s rd, 911 KiB/s wr, 12 op/s 2023-03-10T01:09:47.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:47 smithi181 ceph-mon[94189]: pgmap v20082: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 961 KiB/s rd, 911 KiB/s wr, 12 op/s 2023-03-10T01:09:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:47 smithi038 ceph-mon[92422]: pgmap v20082: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 961 KiB/s rd, 911 KiB/s wr, 12 op/s 2023-03-10T01:09:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:09:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:09:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:09:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:09:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:09:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:09:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37160.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:09:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:09:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:09:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:49 smithi122 ceph-mon[93869]: pgmap v20083: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 620 KiB/s wr, 10 op/s 2023-03-10T01:09:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:09:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:09:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:49 smithi181 ceph-mon[94189]: pgmap v20083: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 620 KiB/s wr, 10 op/s 2023-03-10T01:09:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:09:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:09:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:49 smithi038 ceph-mon[92422]: pgmap v20083: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 620 KiB/s wr, 10 op/s 2023-03-10T01:09:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:50 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:09:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:50 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:09:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:50 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:09:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36701.507977 secs 2023-03-10T01:09:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:51 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:09:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:09:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:09:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:51 smithi122 ceph-mon[93869]: pgmap v20084: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 783 KiB/s wr, 10 op/s 2023-03-10T01:09:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36701.507977 secs 2023-03-10T01:09:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:51 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:09:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:09:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:09:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:51 smithi181 ceph-mon[94189]: pgmap v20084: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 783 KiB/s wr, 10 op/s 2023-03-10T01:09:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36701.507977 secs 2023-03-10T01:09:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:51 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:09:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:09:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:09:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:51 smithi038 ceph-mon[92422]: pgmap v20084: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 783 KiB/s wr, 10 op/s 2023-03-10T01:09:53.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37165.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:09:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:53 smithi038 ceph-mon[92422]: pgmap v20085: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 948 KiB/s rd, 536 KiB/s wr, 12 op/s 2023-03-10T01:09:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:53 smithi122 ceph-mon[93869]: pgmap v20085: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 948 KiB/s rd, 536 KiB/s wr, 12 op/s 2023-03-10T01:09:54.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:53 smithi181 ceph-mon[94189]: pgmap v20085: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 948 KiB/s rd, 536 KiB/s wr, 12 op/s 2023-03-10T01:09:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:55 smithi122 ceph-mon[93869]: pgmap v20086: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 535 KiB/s wr, 10 op/s 2023-03-10T01:09:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:55 smithi181 ceph-mon[94189]: pgmap v20086: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 535 KiB/s wr, 10 op/s 2023-03-10T01:09:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:55 smithi038 ceph-mon[92422]: pgmap v20086: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 535 KiB/s wr, 10 op/s 2023-03-10T01:09:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36706.508067 secs 2023-03-10T01:09:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:09:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:09:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36706.508067 secs 2023-03-10T01:09:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:09:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:09:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36706.508067 secs 2023-03-10T01:09:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:09:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:09:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:09:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:09:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:57 smithi122 ceph-mon[93869]: pgmap v20087: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 547 KiB/s wr, 13 op/s 2023-03-10T01:09:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:09:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:09:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:57 smithi181 ceph-mon[94189]: pgmap v20087: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 547 KiB/s wr, 13 op/s 2023-03-10T01:09:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:09:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:09:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:57 smithi038 ceph-mon[92422]: pgmap v20087: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 547 KiB/s wr, 13 op/s 2023-03-10T01:09:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37170.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:09:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:09:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:09:59.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:09:59 smithi122 ceph-mon[93869]: pgmap v20088: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 547 KiB/s wr, 11 op/s 2023-03-10T01:09:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:09:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:09:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:09:59 smithi181 ceph-mon[94189]: pgmap v20088: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 547 KiB/s wr, 11 op/s 2023-03-10T01:09:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:09:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:09:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:09:59 smithi038 ceph-mon[92422]: pgmap v20088: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 547 KiB/s wr, 11 op/s 2023-03-10T01:10:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:00 smithi122 ceph-mon[93869]: Health detail: HEALTH_WARN 1 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-10T01:10:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:00 smithi122 ceph-mon[93869]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2023-03-10T01:10:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:00 smithi122 ceph-mon[93869]: mds.h(mds.1): 2 slow requests are blocked > 180 secs 2023-03-10T01:10:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:00 smithi122 ceph-mon[93869]: [WRN] MDS_TRIM: 2 MDSs behind on trimming 2023-03-10T01:10:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:00 smithi122 ceph-mon[93869]: mds.e(mds.0): Behind on trimming (1207/128) max_segments: 128, num_segments: 1207 2023-03-10T01:10:00.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:00 smithi122 ceph-mon[93869]: mds.b(mds.0): Behind on trimming (1206/128) max_segments: 128, num_segments: 1206 2023-03-10T01:10:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:00 smithi181 ceph-mon[94189]: Health detail: HEALTH_WARN 1 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-10T01:10:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:00 smithi181 ceph-mon[94189]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2023-03-10T01:10:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:00 smithi181 ceph-mon[94189]: mds.h(mds.1): 2 slow requests are blocked > 180 secs 2023-03-10T01:10:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:00 smithi181 ceph-mon[94189]: [WRN] MDS_TRIM: 2 MDSs behind on trimming 2023-03-10T01:10:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:00 smithi181 ceph-mon[94189]: mds.e(mds.0): Behind on trimming (1207/128) max_segments: 128, num_segments: 1207 2023-03-10T01:10:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:00 smithi181 ceph-mon[94189]: mds.b(mds.0): Behind on trimming (1206/128) max_segments: 128, num_segments: 1206 2023-03-10T01:10:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:00 smithi038 ceph-mon[92422]: Health detail: HEALTH_WARN 1 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-10T01:10:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:00 smithi038 ceph-mon[92422]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2023-03-10T01:10:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:00 smithi038 ceph-mon[92422]: mds.h(mds.1): 2 slow requests are blocked > 180 secs 2023-03-10T01:10:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:00 smithi038 ceph-mon[92422]: [WRN] MDS_TRIM: 2 MDSs behind on trimming 2023-03-10T01:10:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:00 smithi038 ceph-mon[92422]: mds.e(mds.0): Behind on trimming (1207/128) max_segments: 128, num_segments: 1207 2023-03-10T01:10:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:00 smithi038 ceph-mon[92422]: mds.b(mds.0): Behind on trimming (1206/128) max_segments: 128, num_segments: 1206 2023-03-10T01:10:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36711.508153 secs 2023-03-10T01:10:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:10:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:10:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:01 smithi122 ceph-mon[93869]: pgmap v20089: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 755 KiB/s wr, 13 op/s 2023-03-10T01:10:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36711.508153 secs 2023-03-10T01:10:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:10:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:10:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:01 smithi181 ceph-mon[94189]: pgmap v20089: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 755 KiB/s wr, 13 op/s 2023-03-10T01:10:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36711.508153 secs 2023-03-10T01:10:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:10:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:10:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:01 smithi038 ceph-mon[92422]: pgmap v20089: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 755 KiB/s wr, 13 op/s 2023-03-10T01:10:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37175.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:10:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:03 smithi122 ceph-mon[93869]: pgmap v20090: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 623 KiB/s wr, 15 op/s 2023-03-10T01:10:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:03 smithi181 ceph-mon[94189]: pgmap v20090: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 623 KiB/s wr, 15 op/s 2023-03-10T01:10:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:03 smithi038 ceph-mon[92422]: pgmap v20090: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 623 KiB/s wr, 15 op/s 2023-03-10T01:10:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:05 smithi122 ceph-mon[93869]: pgmap v20091: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 733 KiB/s wr, 24 op/s 2023-03-10T01:10:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:05 smithi181 ceph-mon[94189]: pgmap v20091: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 733 KiB/s wr, 24 op/s 2023-03-10T01:10:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:05 smithi038 ceph-mon[92422]: pgmap v20091: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 733 KiB/s wr, 24 op/s 2023-03-10T01:10:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36716.508241 secs 2023-03-10T01:10:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36716.508241 secs 2023-03-10T01:10:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36716.508241 secs 2023-03-10T01:10:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:07 smithi122 ceph-mon[93869]: pgmap v20092: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 733 KiB/s wr, 26 op/s 2023-03-10T01:10:07.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:07 smithi181 ceph-mon[94189]: pgmap v20092: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 733 KiB/s wr, 26 op/s 2023-03-10T01:10:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:07 smithi038 ceph-mon[92422]: pgmap v20092: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 733 KiB/s wr, 26 op/s 2023-03-10T01:10:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:10:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:10:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:10:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:10:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:10:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:10:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37180.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:10:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:10:09.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:10:09.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:09 smithi122 ceph-mon[93869]: pgmap v20093: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 887 KiB/s rd, 528 KiB/s wr, 24 op/s 2023-03-10T01:10:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:10:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:10:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:09 smithi181 ceph-mon[94189]: pgmap v20093: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 887 KiB/s rd, 528 KiB/s wr, 24 op/s 2023-03-10T01:10:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:10:09.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:10:09.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:09 smithi038 ceph-mon[92422]: pgmap v20093: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 887 KiB/s rd, 528 KiB/s wr, 24 op/s 2023-03-10T01:10:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36721.508327 secs 2023-03-10T01:10:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:11 smithi122 ceph-mon[93869]: pgmap v20094: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 887 KiB/s rd, 528 KiB/s wr, 23 op/s 2023-03-10T01:10:12.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36721.508327 secs 2023-03-10T01:10:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:11 smithi181 ceph-mon[94189]: pgmap v20094: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 887 KiB/s rd, 528 KiB/s wr, 23 op/s 2023-03-10T01:10:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36721.508327 secs 2023-03-10T01:10:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:11 smithi038 ceph-mon[92422]: pgmap v20094: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 887 KiB/s rd, 528 KiB/s wr, 23 op/s 2023-03-10T01:10:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37185.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:10:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:13 smithi122 ceph-mon[93869]: pgmap v20095: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 251 KiB/s wr, 22 op/s 2023-03-10T01:10:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:13 smithi181 ceph-mon[94189]: pgmap v20095: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 251 KiB/s wr, 22 op/s 2023-03-10T01:10:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:13 smithi038 ceph-mon[92422]: pgmap v20095: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 251 KiB/s wr, 22 op/s 2023-03-10T01:10:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:15 smithi122 ceph-mon[93869]: pgmap v20096: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 429 KiB/s wr, 20 op/s 2023-03-10T01:10:15.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:15 smithi181 ceph-mon[94189]: pgmap v20096: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 429 KiB/s wr, 20 op/s 2023-03-10T01:10:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:15 smithi038 ceph-mon[92422]: pgmap v20096: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 429 KiB/s wr, 20 op/s 2023-03-10T01:10:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36726.508404 secs 2023-03-10T01:10:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:10:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:10:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36726.508404 secs 2023-03-10T01:10:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:10:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:10:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36726.508404 secs 2023-03-10T01:10:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:10:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:10:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:10:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:10:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:17 smithi122 ceph-mon[93869]: pgmap v20097: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 289 KiB/s wr, 10 op/s 2023-03-10T01:10:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:10:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:10:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:17 smithi181 ceph-mon[94189]: pgmap v20097: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 289 KiB/s wr, 10 op/s 2023-03-10T01:10:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:10:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:10:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:17 smithi038 ceph-mon[92422]: pgmap v20097: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 289 KiB/s wr, 10 op/s 2023-03-10T01:10:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37190.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:10:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:19 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:10:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:19 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:10:19.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:19 smithi122 ceph-mon[93869]: pgmap v20098: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 299 KiB/s wr, 9 op/s 2023-03-10T01:10:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:19 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:10:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:19 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:10:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:19 smithi181 ceph-mon[94189]: pgmap v20098: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 299 KiB/s wr, 9 op/s 2023-03-10T01:10:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:19 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:10:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:19 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:10:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:19 smithi038 ceph-mon[92422]: pgmap v20098: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 299 KiB/s wr, 9 op/s 2023-03-10T01:10:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36731.508490 secs 2023-03-10T01:10:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:10:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:10:21.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:21 smithi122 ceph-mon[93869]: pgmap v20099: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 306 KiB/s wr, 8 op/s 2023-03-10T01:10:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36731.508490 secs 2023-03-10T01:10:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:10:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:10:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:21 smithi181 ceph-mon[94189]: pgmap v20099: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 306 KiB/s wr, 8 op/s 2023-03-10T01:10:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36731.508490 secs 2023-03-10T01:10:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:10:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:10:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:21 smithi038 ceph-mon[92422]: pgmap v20099: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 306 KiB/s wr, 8 op/s 2023-03-10T01:10:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37195.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:10:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:23 smithi122 ceph-mon[93869]: pgmap v20100: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 817 KiB/s wr, 13 op/s 2023-03-10T01:10:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:23 smithi181 ceph-mon[94189]: pgmap v20100: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 817 KiB/s wr, 13 op/s 2023-03-10T01:10:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:23 smithi038 ceph-mon[92422]: pgmap v20100: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 817 KiB/s wr, 13 op/s 2023-03-10T01:10:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:25 smithi122 ceph-mon[93869]: pgmap v20101: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 722 KiB/s rd, 741 KiB/s wr, 11 op/s 2023-03-10T01:10:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:25 smithi122 ceph-mon[93869]: osdmap e1471: 12 total, 12 up, 12 in 2023-03-10T01:10:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:25 smithi181 ceph-mon[94189]: pgmap v20101: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 722 KiB/s rd, 741 KiB/s wr, 11 op/s 2023-03-10T01:10:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:25 smithi181 ceph-mon[94189]: osdmap e1471: 12 total, 12 up, 12 in 2023-03-10T01:10:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:25 smithi038 ceph-mon[92422]: pgmap v20101: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 722 KiB/s rd, 741 KiB/s wr, 11 op/s 2023-03-10T01:10:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:25 smithi038 ceph-mon[92422]: osdmap e1471: 12 total, 12 up, 12 in 2023-03-10T01:10:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36736.508549 secs 2023-03-10T01:10:27.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36736.508549 secs 2023-03-10T01:10:27.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36736.508549 secs 2023-03-10T01:10:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:27 smithi122 ceph-mon[93869]: pgmap v20103: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 637 KiB/s rd, 661 KiB/s wr, 12 op/s 2023-03-10T01:10:28.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:27 smithi181 ceph-mon[94189]: pgmap v20103: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 637 KiB/s rd, 661 KiB/s wr, 12 op/s 2023-03-10T01:10:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:27 smithi038 ceph-mon[92422]: pgmap v20103: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 637 KiB/s rd, 661 KiB/s wr, 12 op/s 2023-03-10T01:10:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37200.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:10:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:10:29.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:10:29.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:10:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:29 smithi122 ceph-mon[93869]: pgmap v20104: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 789 KiB/s rd, 801 KiB/s wr, 11 op/s 2023-03-10T01:10:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:10:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:10:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:10:30.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:29 smithi181 ceph-mon[94189]: pgmap v20104: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 789 KiB/s rd, 801 KiB/s wr, 11 op/s 2023-03-10T01:10:30.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:10:30.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:10:30.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:10:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:29 smithi038 ceph-mon[92422]: pgmap v20104: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 789 KiB/s rd, 801 KiB/s wr, 11 op/s 2023-03-10T01:10:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:10:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:10:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:10:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36741.508606 secs 2023-03-10T01:10:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:31 smithi122 ceph-mon[93869]: pgmap v20105: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 12 op/s 2023-03-10T01:10:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36741.508606 secs 2023-03-10T01:10:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:31 smithi181 ceph-mon[94189]: pgmap v20105: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 12 op/s 2023-03-10T01:10:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36741.508606 secs 2023-03-10T01:10:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:31 smithi038 ceph-mon[92422]: pgmap v20105: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 12 op/s 2023-03-10T01:10:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36743.970355 secs 2023-03-10T01:10:33.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36743.970355 secs 2023-03-10T01:10:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36743.970355 secs 2023-03-10T01:10:33.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37205.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:10:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:33 smithi122 ceph-mon[93869]: pgmap v20106: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 581 KiB/s rd, 579 KiB/s wr, 9 op/s 2023-03-10T01:10:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:33 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:10:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:33 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:10:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:33 smithi181 ceph-mon[94189]: pgmap v20106: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 581 KiB/s rd, 579 KiB/s wr, 9 op/s 2023-03-10T01:10:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:33 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:10:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:33 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:10:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:33 smithi038 ceph-mon[92422]: pgmap v20106: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 581 KiB/s rd, 579 KiB/s wr, 9 op/s 2023-03-10T01:10:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:33 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:10:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:33 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:10:35.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:34 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:10:35.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:34 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:10:35.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:34 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:10:35.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:34 smithi122 ceph-mon[93869]: osdmap e1472: 12 total, 12 up, 12 in 2023-03-10T01:10:35.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:34 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:10:35.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:34 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:10:35.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:34 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:10:35.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:34 smithi181 ceph-mon[94189]: osdmap e1472: 12 total, 12 up, 12 in 2023-03-10T01:10:35.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:34 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:10:35.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:34 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:10:35.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:34 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:10:35.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:34 smithi038 ceph-mon[92422]: osdmap e1472: 12 total, 12 up, 12 in 2023-03-10T01:10:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:35 smithi122 ceph-mon[93869]: pgmap v20108: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 585 KiB/s rd, 584 KiB/s wr, 9 op/s 2023-03-10T01:10:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:35 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:10:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:35 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:10:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:35 smithi181 ceph-mon[94189]: pgmap v20108: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 585 KiB/s rd, 584 KiB/s wr, 9 op/s 2023-03-10T01:10:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:35 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:10:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:35 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:10:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:35 smithi038 ceph-mon[92422]: pgmap v20108: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 585 KiB/s rd, 584 KiB/s wr, 9 op/s 2023-03-10T01:10:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:35 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:10:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:35 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:10:37.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36746.508710 secs 2023-03-10T01:10:37.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36746.508710 secs 2023-03-10T01:10:37.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36746.508710 secs 2023-03-10T01:10:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:37 smithi122 ceph-mon[93869]: pgmap v20109: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 585 KiB/s wr, 10 op/s 2023-03-10T01:10:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:37 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36748.970466 secs 2023-03-10T01:10:38.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:37 smithi181 ceph-mon[94189]: pgmap v20109: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 585 KiB/s wr, 10 op/s 2023-03-10T01:10:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:37 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36748.970466 secs 2023-03-10T01:10:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:37 smithi038 ceph-mon[92422]: pgmap v20109: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 585 KiB/s wr, 10 op/s 2023-03-10T01:10:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:37 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36748.970466 secs 2023-03-10T01:10:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37210.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:10:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:39 smithi122 ceph-mon[93869]: pgmap v20110: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 581 KiB/s rd, 580 KiB/s wr, 10 op/s 2023-03-10T01:10:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:39 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:10:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:39 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:10:40.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:39 smithi181 ceph-mon[94189]: pgmap v20110: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 581 KiB/s rd, 580 KiB/s wr, 10 op/s 2023-03-10T01:10:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:39 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:10:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:39 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:10:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:39 smithi038 ceph-mon[92422]: pgmap v20110: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 581 KiB/s rd, 580 KiB/s wr, 10 op/s 2023-03-10T01:10:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:39 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:10:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:39 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:10:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36751.508769 secs 2023-03-10T01:10:41.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36751.508769 secs 2023-03-10T01:10:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36751.508769 secs 2023-03-10T01:10:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:41 smithi122 ceph-mon[93869]: pgmap v20111: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 224 KiB/s wr, 11 op/s 2023-03-10T01:10:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:10:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:10:42.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:41 smithi181 ceph-mon[94189]: pgmap v20111: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 224 KiB/s wr, 11 op/s 2023-03-10T01:10:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:10:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:10:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:41 smithi038 ceph-mon[92422]: pgmap v20111: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 226 KiB/s rd, 224 KiB/s wr, 11 op/s 2023-03-10T01:10:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:10:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:10:43.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36753.970528 secs 2023-03-10T01:10:43.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36753.970528 secs 2023-03-10T01:10:43.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36753.970528 secs 2023-03-10T01:10:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37215.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:10:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:43 smithi122 ceph-mon[93869]: pgmap v20112: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 509 KiB/s rd, 468 KiB/s wr, 10 op/s 2023-03-10T01:10:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:43 smithi181 ceph-mon[94189]: pgmap v20112: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 509 KiB/s rd, 468 KiB/s wr, 10 op/s 2023-03-10T01:10:44.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:43 smithi038 ceph-mon[92422]: pgmap v20112: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 509 KiB/s rd, 468 KiB/s wr, 10 op/s 2023-03-10T01:10:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:45 smithi122 ceph-mon[93869]: pgmap v20113: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 449 KiB/s wr, 9 op/s 2023-03-10T01:10:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:45 smithi181 ceph-mon[94189]: pgmap v20113: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 449 KiB/s wr, 9 op/s 2023-03-10T01:10:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:45 smithi038 ceph-mon[92422]: pgmap v20113: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 449 KiB/s wr, 9 op/s 2023-03-10T01:10:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36756.508830 secs 2023-03-10T01:10:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36756.508830 secs 2023-03-10T01:10:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36756.508830 secs 2023-03-10T01:10:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:47 smithi122 ceph-mon[93869]: pgmap v20114: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 470 KiB/s rd, 434 KiB/s wr, 11 op/s 2023-03-10T01:10:47.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:47 smithi181 ceph-mon[94189]: pgmap v20114: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 470 KiB/s rd, 434 KiB/s wr, 11 op/s 2023-03-10T01:10:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:47 smithi038 ceph-mon[92422]: pgmap v20114: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 470 KiB/s rd, 434 KiB/s wr, 11 op/s 2023-03-10T01:10:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:48 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 36758.970588 secs 2023-03-10T01:10:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:48 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 36758.970588 secs 2023-03-10T01:10:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:48 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 36758.970588 secs 2023-03-10T01:10:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37220.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:10:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:10:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:10:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:49 smithi122 ceph-mon[93869]: pgmap v20115: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 918 KiB/s rd, 754 KiB/s wr, 9 op/s 2023-03-10T01:10:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:10:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:10:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:49 smithi181 ceph-mon[94189]: pgmap v20115: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 918 KiB/s rd, 754 KiB/s wr, 9 op/s 2023-03-10T01:10:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:10:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:10:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:49 smithi038 ceph-mon[92422]: pgmap v20115: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 918 KiB/s rd, 754 KiB/s wr, 9 op/s 2023-03-10T01:10:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36761.508915 secs 2023-03-10T01:10:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:10:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:10:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:51 smithi122 ceph-mon[93869]: pgmap v20116: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 826 KiB/s rd, 661 KiB/s wr, 10 op/s 2023-03-10T01:10:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36761.508915 secs 2023-03-10T01:10:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:10:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:10:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:51 smithi181 ceph-mon[94189]: pgmap v20116: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 826 KiB/s rd, 661 KiB/s wr, 10 op/s 2023-03-10T01:10:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36761.508915 secs 2023-03-10T01:10:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:10:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:10:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:51 smithi038 ceph-mon[92422]: pgmap v20116: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 826 KiB/s rd, 661 KiB/s wr, 10 op/s 2023-03-10T01:10:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:10:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:10:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:10:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:10:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:10:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:10:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:10:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:10:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:53 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 36763.970649 secs 2023-03-10T01:10:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:53 smithi122 ceph-mon[93869]: pgmap v20117: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1002 KiB/s rd, 838 KiB/s wr, 10 op/s 2023-03-10T01:10:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:10:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:10:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:53 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 36763.970649 secs 2023-03-10T01:10:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:53 smithi181 ceph-mon[94189]: pgmap v20117: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1002 KiB/s rd, 838 KiB/s wr, 10 op/s 2023-03-10T01:10:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:10:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:10:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:53 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 36763.970649 secs 2023-03-10T01:10:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:53 smithi038 ceph-mon[92422]: pgmap v20117: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1002 KiB/s rd, 838 KiB/s wr, 10 op/s 2023-03-10T01:10:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37225.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:10:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:55 smithi122 ceph-mon[93869]: pgmap v20118: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 751 KiB/s rd, 620 KiB/s wr, 8 op/s 2023-03-10T01:10:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:55 smithi181 ceph-mon[94189]: pgmap v20118: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 751 KiB/s rd, 620 KiB/s wr, 8 op/s 2023-03-10T01:10:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:55 smithi038 ceph-mon[92422]: pgmap v20118: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 751 KiB/s rd, 620 KiB/s wr, 8 op/s 2023-03-10T01:10:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36766.508998 secs 2023-03-10T01:10:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:10:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:10:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36766.508998 secs 2023-03-10T01:10:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:10:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:10:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36766.508998 secs 2023-03-10T01:10:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:10:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:10:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:57 smithi122 ceph-mon[93869]: pgmap v20119: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 985 KiB/s rd, 850 KiB/s wr, 12 op/s 2023-03-10T01:10:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:57 smithi181 ceph-mon[94189]: pgmap v20119: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 985 KiB/s rd, 850 KiB/s wr, 12 op/s 2023-03-10T01:10:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:57 smithi038 ceph-mon[92422]: pgmap v20119: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 985 KiB/s rd, 850 KiB/s wr, 12 op/s 2023-03-10T01:10:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36768.970768 secs 2023-03-10T01:10:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36768.970768 secs 2023-03-10T01:10:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36768.970768 secs 2023-03-10T01:10:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37230.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:10:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:10:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:10:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:10:59 smithi122 ceph-mon[93869]: pgmap v20120: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 939 KiB/s rd, 806 KiB/s wr, 10 op/s 2023-03-10T01:10:59.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:10:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:10:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:10:59 smithi181 ceph-mon[94189]: pgmap v20120: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 939 KiB/s rd, 806 KiB/s wr, 10 op/s 2023-03-10T01:10:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:10:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:10:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:10:59 smithi038 ceph-mon[92422]: pgmap v20120: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 939 KiB/s rd, 806 KiB/s wr, 10 op/s 2023-03-10T01:11:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:11:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:11:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:11:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:11:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:11:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:11:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36771.509060 secs 2023-03-10T01:11:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:11:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:11:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:01 smithi122 ceph-mon[93869]: pgmap v20121: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 506 KiB/s wr, 12 op/s 2023-03-10T01:11:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36771.509060 secs 2023-03-10T01:11:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:11:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:11:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:01 smithi181 ceph-mon[94189]: pgmap v20121: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 506 KiB/s wr, 12 op/s 2023-03-10T01:11:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36771.509060 secs 2023-03-10T01:11:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:11:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:11:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:01 smithi038 ceph-mon[92422]: pgmap v20121: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 506 KiB/s wr, 12 op/s 2023-03-10T01:11:03.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36773.970858 secs 2023-03-10T01:11:03.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36773.970858 secs 2023-03-10T01:11:03.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36773.970858 secs 2023-03-10T01:11:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37235.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:11:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:03 smithi122 ceph-mon[93869]: pgmap v20122: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 711 KiB/s wr, 13 op/s 2023-03-10T01:11:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:03 smithi181 ceph-mon[94189]: pgmap v20122: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 711 KiB/s wr, 13 op/s 2023-03-10T01:11:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:03 smithi038 ceph-mon[92422]: pgmap v20122: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 711 KiB/s wr, 13 op/s 2023-03-10T01:11:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:05 smithi122 ceph-mon[93869]: pgmap v20123: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 711 KiB/s rd, 710 KiB/s wr, 22 op/s 2023-03-10T01:11:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:05 smithi181 ceph-mon[94189]: pgmap v20123: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 711 KiB/s rd, 710 KiB/s wr, 22 op/s 2023-03-10T01:11:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:05 smithi038 ceph-mon[92422]: pgmap v20123: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 711 KiB/s rd, 710 KiB/s wr, 22 op/s 2023-03-10T01:11:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36776.509118 secs 2023-03-10T01:11:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36776.509118 secs 2023-03-10T01:11:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36776.509118 secs 2023-03-10T01:11:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:07 smithi122 ceph-mon[93869]: pgmap v20124: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 712 KiB/s rd, 710 KiB/s wr, 25 op/s 2023-03-10T01:11:07.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:07 smithi181 ceph-mon[94189]: pgmap v20124: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 712 KiB/s rd, 710 KiB/s wr, 25 op/s 2023-03-10T01:11:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:07 smithi038 ceph-mon[92422]: pgmap v20124: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 712 KiB/s rd, 710 KiB/s wr, 25 op/s 2023-03-10T01:11:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:11:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:11:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36778.970920 secs 2023-03-10T01:11:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:11:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:11:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36778.970920 secs 2023-03-10T01:11:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:11:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:11:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36778.970920 secs 2023-03-10T01:11:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37240.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:11:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:11:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:11:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:09 smithi122 ceph-mon[93869]: pgmap v20125: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 554 KiB/s rd, 556 KiB/s wr, 23 op/s 2023-03-10T01:11:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:11:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:11:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:09 smithi181 ceph-mon[94189]: pgmap v20125: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 554 KiB/s rd, 556 KiB/s wr, 23 op/s 2023-03-10T01:11:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:11:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:11:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:09 smithi038 ceph-mon[92422]: pgmap v20125: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 554 KiB/s rd, 556 KiB/s wr, 23 op/s 2023-03-10T01:11:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36781.509177 secs 2023-03-10T01:11:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:11 smithi122 ceph-mon[93869]: pgmap v20126: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 652 KiB/s rd, 653 KiB/s wr, 24 op/s 2023-03-10T01:11:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36781.509177 secs 2023-03-10T01:11:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:11 smithi181 ceph-mon[94189]: pgmap v20126: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 652 KiB/s rd, 653 KiB/s wr, 24 op/s 2023-03-10T01:11:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36781.509177 secs 2023-03-10T01:11:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:11 smithi038 ceph-mon[92422]: pgmap v20126: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 652 KiB/s rd, 653 KiB/s wr, 24 op/s 2023-03-10T01:11:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36783.970979 secs 2023-03-10T01:11:13.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36783.970979 secs 2023-03-10T01:11:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36783.970979 secs 2023-03-10T01:11:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37245.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:11:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:13 smithi122 ceph-mon[93869]: pgmap v20127: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 609 KiB/s rd, 611 KiB/s wr, 22 op/s 2023-03-10T01:11:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:13 smithi181 ceph-mon[94189]: pgmap v20127: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 609 KiB/s rd, 611 KiB/s wr, 22 op/s 2023-03-10T01:11:14.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:13 smithi038 ceph-mon[92422]: pgmap v20127: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 609 KiB/s rd, 611 KiB/s wr, 22 op/s 2023-03-10T01:11:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:15 smithi122 ceph-mon[93869]: pgmap v20128: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 450 KiB/s wr, 20 op/s 2023-03-10T01:11:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:15 smithi181 ceph-mon[94189]: pgmap v20128: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 450 KiB/s wr, 20 op/s 2023-03-10T01:11:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:15 smithi038 ceph-mon[92422]: pgmap v20128: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 450 KiB/s wr, 20 op/s 2023-03-10T01:11:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36786.509249 secs 2023-03-10T01:11:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36786.509249 secs 2023-03-10T01:11:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36786.509249 secs 2023-03-10T01:11:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:17 smithi122 ceph-mon[93869]: pgmap v20129: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 247 KiB/s wr, 11 op/s 2023-03-10T01:11:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:17 smithi181 ceph-mon[94189]: pgmap v20129: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 247 KiB/s wr, 11 op/s 2023-03-10T01:11:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:17 smithi038 ceph-mon[92422]: pgmap v20129: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 247 KiB/s wr, 11 op/s 2023-03-10T01:11:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36788.971039 secs 2023-03-10T01:11:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36788.971039 secs 2023-03-10T01:11:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36788.971039 secs 2023-03-10T01:11:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37250.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:11:19.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:19 smithi122 ceph-mon[93869]: pgmap v20130: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 258 KiB/s wr, 9 op/s 2023-03-10T01:11:19.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:19 smithi181 ceph-mon[94189]: pgmap v20130: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 258 KiB/s wr, 9 op/s 2023-03-10T01:11:19.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:19 smithi038 ceph-mon[92422]: pgmap v20130: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 258 KiB/s wr, 9 op/s 2023-03-10T01:11:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:11:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:11:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:11:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:11:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:11:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:11:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36791.509304 secs 2023-03-10T01:11:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:11:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:11:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:21 smithi122 ceph-mon[93869]: pgmap v20131: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 399 KiB/s rd, 418 KiB/s wr, 11 op/s 2023-03-10T01:11:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36791.509304 secs 2023-03-10T01:11:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:11:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:11:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:21 smithi181 ceph-mon[94189]: pgmap v20131: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 399 KiB/s rd, 418 KiB/s wr, 11 op/s 2023-03-10T01:11:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36791.509304 secs 2023-03-10T01:11:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:11:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:11:21.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:21 smithi038 ceph-mon[92422]: pgmap v20131: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 399 KiB/s rd, 418 KiB/s wr, 11 op/s 2023-03-10T01:11:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36793.971158 secs 2023-03-10T01:11:23.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36793.971158 secs 2023-03-10T01:11:23.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36793.971158 secs 2023-03-10T01:11:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37255.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:11:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:23 smithi122 ceph-mon[93869]: pgmap v20132: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 321 KiB/s wr, 11 op/s 2023-03-10T01:11:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:23 smithi181 ceph-mon[94189]: pgmap v20132: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 321 KiB/s wr, 11 op/s 2023-03-10T01:11:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:23 smithi038 ceph-mon[92422]: pgmap v20132: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 321 KiB/s wr, 11 op/s 2023-03-10T01:11:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:25 smithi122 ceph-mon[93869]: pgmap v20133: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 321 KiB/s wr, 10 op/s 2023-03-10T01:11:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:25 smithi122 ceph-mon[93869]: osdmap e1473: 12 total, 12 up, 12 in 2023-03-10T01:11:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:25 smithi181 ceph-mon[94189]: pgmap v20133: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 321 KiB/s wr, 10 op/s 2023-03-10T01:11:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:25 smithi181 ceph-mon[94189]: osdmap e1473: 12 total, 12 up, 12 in 2023-03-10T01:11:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:25 smithi038 ceph-mon[92422]: pgmap v20133: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 321 KiB/s wr, 10 op/s 2023-03-10T01:11:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:25 smithi038 ceph-mon[92422]: osdmap e1473: 12 total, 12 up, 12 in 2023-03-10T01:11:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36796.509422 secs 2023-03-10T01:11:27.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36796.509422 secs 2023-03-10T01:11:27.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36796.509422 secs 2023-03-10T01:11:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:27 smithi122 ceph-mon[93869]: pgmap v20135: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 502 KiB/s wr, 12 op/s 2023-03-10T01:11:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36798.971215 secs 2023-03-10T01:11:28.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:27 smithi181 ceph-mon[94189]: pgmap v20135: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 502 KiB/s wr, 12 op/s 2023-03-10T01:11:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36798.971215 secs 2023-03-10T01:11:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:27 smithi038 ceph-mon[92422]: pgmap v20135: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 502 KiB/s wr, 12 op/s 2023-03-10T01:11:28.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36798.971215 secs 2023-03-10T01:11:28.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37260.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:11:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:29 smithi122 ceph-mon[93869]: pgmap v20136: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 478 KiB/s rd, 490 KiB/s wr, 11 op/s 2023-03-10T01:11:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:11:30.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:29 smithi181 ceph-mon[94189]: pgmap v20136: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 478 KiB/s rd, 490 KiB/s wr, 11 op/s 2023-03-10T01:11:30.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:11:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:29 smithi038 ceph-mon[92422]: pgmap v20136: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 478 KiB/s rd, 490 KiB/s wr, 11 op/s 2023-03-10T01:11:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:11:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:30 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:11:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:30 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:11:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:30 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:11:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36801.509504 secs 2023-03-10T01:11:31.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:30 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:11:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:30 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:11:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:30 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:11:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36801.509504 secs 2023-03-10T01:11:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:30 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:11:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:30 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:11:31.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:30 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:11:31.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36801.509504 secs 2023-03-10T01:11:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:31 smithi122 ceph-mon[93869]: pgmap v20137: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 545 KiB/s rd, 545 KiB/s wr, 10 op/s 2023-03-10T01:11:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:31 smithi181 ceph-mon[94189]: pgmap v20137: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 545 KiB/s rd, 545 KiB/s wr, 10 op/s 2023-03-10T01:11:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:31 smithi038 ceph-mon[92422]: pgmap v20137: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 545 KiB/s rd, 545 KiB/s wr, 10 op/s 2023-03-10T01:11:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:32 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:11:33.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:32 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:11:33.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36803.971301 secs 2023-03-10T01:11:33.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:32 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:11:33.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:32 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:11:33.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36803.971301 secs 2023-03-10T01:11:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:32 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:11:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:32 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:11:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36803.971301 secs 2023-03-10T01:11:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37265.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:11:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:33 smithi122 ceph-mon[93869]: pgmap v20138: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 645 KiB/s rd, 645 KiB/s wr, 10 op/s 2023-03-10T01:11:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:33 smithi181 ceph-mon[94189]: pgmap v20138: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 645 KiB/s rd, 645 KiB/s wr, 10 op/s 2023-03-10T01:11:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:33 smithi038 ceph-mon[92422]: pgmap v20138: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 645 KiB/s rd, 645 KiB/s wr, 10 op/s 2023-03-10T01:11:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:35 smithi122 ceph-mon[93869]: osdmap e1474: 12 total, 12 up, 12 in 2023-03-10T01:11:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:35 smithi122 ceph-mon[93869]: pgmap v20140: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 652 KiB/s rd, 651 KiB/s wr, 10 op/s 2023-03-10T01:11:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:35 smithi181 ceph-mon[94189]: osdmap e1474: 12 total, 12 up, 12 in 2023-03-10T01:11:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:35 smithi181 ceph-mon[94189]: pgmap v20140: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 652 KiB/s rd, 651 KiB/s wr, 10 op/s 2023-03-10T01:11:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:35 smithi038 ceph-mon[92422]: osdmap e1474: 12 total, 12 up, 12 in 2023-03-10T01:11:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:35 smithi038 ceph-mon[92422]: pgmap v20140: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 652 KiB/s rd, 651 KiB/s wr, 10 op/s 2023-03-10T01:11:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36806.509563 secs 2023-03-10T01:11:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:11:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:11:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36806.509563 secs 2023-03-10T01:11:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:11:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:11:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36806.509563 secs 2023-03-10T01:11:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:11:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:11:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:11:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:11:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:37 smithi122 ceph-mon[93869]: pgmap v20141: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 750 KiB/s rd, 748 KiB/s wr, 10 op/s 2023-03-10T01:11:37.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:11:37.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:11:37.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:37 smithi181 ceph-mon[94189]: pgmap v20141: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 750 KiB/s rd, 748 KiB/s wr, 10 op/s 2023-03-10T01:11:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:11:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:11:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:37 smithi038 ceph-mon[92422]: pgmap v20141: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 750 KiB/s rd, 748 KiB/s wr, 10 op/s 2023-03-10T01:11:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36808.971422 secs 2023-03-10T01:11:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36808.971422 secs 2023-03-10T01:11:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36808.971422 secs 2023-03-10T01:11:38.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37270.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:11:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:39 smithi122 ceph-mon[93869]: pgmap v20142: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 797 KiB/s wr, 11 op/s 2023-03-10T01:11:39.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:39 smithi181 ceph-mon[94189]: pgmap v20142: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 797 KiB/s wr, 11 op/s 2023-03-10T01:11:39.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:39 smithi038 ceph-mon[92422]: pgmap v20142: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 797 KiB/s wr, 11 op/s 2023-03-10T01:11:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:11:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:11:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:11:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:11:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:11:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:11:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36811.509647 secs 2023-03-10T01:11:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:11:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:11:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:41 smithi122 ceph-mon[93869]: pgmap v20143: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 459 KiB/s wr, 10 op/s 2023-03-10T01:11:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36811.509647 secs 2023-03-10T01:11:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:11:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:11:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:41 smithi181 ceph-mon[94189]: pgmap v20143: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 459 KiB/s wr, 10 op/s 2023-03-10T01:11:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36811.509647 secs 2023-03-10T01:11:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:11:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:11:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:41 smithi038 ceph-mon[92422]: pgmap v20143: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 459 KiB/s wr, 10 op/s 2023-03-10T01:11:43.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36813.971482 secs 2023-03-10T01:11:43.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36813.971482 secs 2023-03-10T01:11:43.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36813.971482 secs 2023-03-10T01:11:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37275.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:11:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:43 smithi122 ceph-mon[93869]: pgmap v20144: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 545 KiB/s rd, 542 KiB/s wr, 11 op/s 2023-03-10T01:11:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:43 smithi181 ceph-mon[94189]: pgmap v20144: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 545 KiB/s rd, 542 KiB/s wr, 11 op/s 2023-03-10T01:11:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:43 smithi038 ceph-mon[92422]: pgmap v20144: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 545 KiB/s rd, 542 KiB/s wr, 11 op/s 2023-03-10T01:11:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:45 smithi122 ceph-mon[93869]: pgmap v20145: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 523 KiB/s rd, 521 KiB/s wr, 10 op/s 2023-03-10T01:11:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:45 smithi181 ceph-mon[94189]: pgmap v20145: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 523 KiB/s rd, 521 KiB/s wr, 10 op/s 2023-03-10T01:11:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:45 smithi038 ceph-mon[92422]: pgmap v20145: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 523 KiB/s rd, 521 KiB/s wr, 10 op/s 2023-03-10T01:11:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36816.509708 secs 2023-03-10T01:11:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36816.509708 secs 2023-03-10T01:11:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36816.509708 secs 2023-03-10T01:11:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:47 smithi122 ceph-mon[93869]: pgmap v20146: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 455 KiB/s rd, 452 KiB/s wr, 12 op/s 2023-03-10T01:11:47.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:47 smithi181 ceph-mon[94189]: pgmap v20146: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 455 KiB/s rd, 452 KiB/s wr, 12 op/s 2023-03-10T01:11:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:47 smithi038 ceph-mon[92422]: pgmap v20146: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 455 KiB/s rd, 452 KiB/s wr, 12 op/s 2023-03-10T01:11:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36818.971542 secs 2023-03-10T01:11:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36818.971542 secs 2023-03-10T01:11:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36818.971542 secs 2023-03-10T01:11:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37280.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:11:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:49 smithi122 ceph-mon[93869]: pgmap v20147: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 342 KiB/s wr, 9 op/s 2023-03-10T01:11:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:49 smithi181 ceph-mon[94189]: pgmap v20147: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 342 KiB/s wr, 9 op/s 2023-03-10T01:11:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:49 smithi038 ceph-mon[92422]: pgmap v20147: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 342 KiB/s wr, 9 op/s 2023-03-10T01:11:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36821.509766 secs 2023-03-10T01:11:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:51 smithi122 ceph-mon[93869]: pgmap v20148: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 484 KiB/s wr, 10 op/s 2023-03-10T01:11:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36821.509766 secs 2023-03-10T01:11:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:51 smithi181 ceph-mon[94189]: pgmap v20148: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 484 KiB/s wr, 10 op/s 2023-03-10T01:11:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36821.509766 secs 2023-03-10T01:11:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:51 smithi038 ceph-mon[92422]: pgmap v20148: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 484 KiB/s wr, 10 op/s 2023-03-10T01:11:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36823.971600 secs 2023-03-10T01:11:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36823.971600 secs 2023-03-10T01:11:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36823.971600 secs 2023-03-10T01:11:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37285.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:11:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:53 smithi122 ceph-mon[93869]: pgmap v20149: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 483 KiB/s wr, 9 op/s 2023-03-10T01:11:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:11:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:11:54.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:53 smithi181 ceph-mon[94189]: pgmap v20149: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 483 KiB/s wr, 9 op/s 2023-03-10T01:11:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:11:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:11:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:53 smithi038 ceph-mon[92422]: pgmap v20149: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 483 KiB/s wr, 9 op/s 2023-03-10T01:11:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:11:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:11:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:55 smithi122 ceph-mon[93869]: pgmap v20150: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 629 KiB/s wr, 8 op/s 2023-03-10T01:11:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:55 smithi181 ceph-mon[94189]: pgmap v20150: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 629 KiB/s wr, 8 op/s 2023-03-10T01:11:55.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:55 smithi038 ceph-mon[92422]: pgmap v20150: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 629 KiB/s wr, 8 op/s 2023-03-10T01:11:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36826.509868 secs 2023-03-10T01:11:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:11:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:11:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36826.509868 secs 2023-03-10T01:11:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:11:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:11:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36826.509868 secs 2023-03-10T01:11:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:11:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:11:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:11:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:11:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:57 smithi122 ceph-mon[93869]: pgmap v20151: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 666 KiB/s rd, 659 KiB/s wr, 11 op/s 2023-03-10T01:11:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:11:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:11:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:57 smithi181 ceph-mon[94189]: pgmap v20151: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 666 KiB/s rd, 659 KiB/s wr, 11 op/s 2023-03-10T01:11:57.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:11:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:11:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:57 smithi038 ceph-mon[92422]: pgmap v20151: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 666 KiB/s rd, 659 KiB/s wr, 11 op/s 2023-03-10T01:11:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:58 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:11:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:58 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:11:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:58 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:11:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37290.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:11:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:59 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:11:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:11:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:11:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:11:59 smithi122 ceph-mon[93869]: pgmap v20152: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 892 KiB/s wr, 10 op/s 2023-03-10T01:11:59.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:59 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:11:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:11:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:11:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:11:59 smithi181 ceph-mon[94189]: pgmap v20152: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 892 KiB/s wr, 10 op/s 2023-03-10T01:11:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:59 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:11:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:11:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:11:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:11:59 smithi038 ceph-mon[92422]: pgmap v20152: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 892 KiB/s wr, 10 op/s 2023-03-10T01:12:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:12:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:12:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:12:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:12:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:12:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:12:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36831.509947 secs 2023-03-10T01:12:01.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:12:01.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:12:01.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:01 smithi122 ceph-mon[93869]: pgmap v20153: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 592 KiB/s rd, 819 KiB/s wr, 14 op/s 2023-03-10T01:12:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36831.509947 secs 2023-03-10T01:12:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:12:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:12:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:01 smithi181 ceph-mon[94189]: pgmap v20153: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 592 KiB/s rd, 819 KiB/s wr, 14 op/s 2023-03-10T01:12:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36831.509947 secs 2023-03-10T01:12:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:12:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:12:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:01 smithi038 ceph-mon[92422]: pgmap v20153: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 592 KiB/s rd, 819 KiB/s wr, 14 op/s 2023-03-10T01:12:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37295.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:12:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:03 smithi122 ceph-mon[93869]: pgmap v20154: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 751 KiB/s rd, 978 KiB/s wr, 14 op/s 2023-03-10T01:12:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:03 smithi181 ceph-mon[94189]: pgmap v20154: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 751 KiB/s rd, 978 KiB/s wr, 14 op/s 2023-03-10T01:12:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:03 smithi038 ceph-mon[92422]: pgmap v20154: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 751 KiB/s rd, 978 KiB/s wr, 14 op/s 2023-03-10T01:12:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:05 smithi122 ceph-mon[93869]: pgmap v20155: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 956 KiB/s rd, 989 KiB/s wr, 24 op/s 2023-03-10T01:12:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:05 smithi181 ceph-mon[94189]: pgmap v20155: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 956 KiB/s rd, 989 KiB/s wr, 24 op/s 2023-03-10T01:12:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:05 smithi038 ceph-mon[92422]: pgmap v20155: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 956 KiB/s rd, 989 KiB/s wr, 24 op/s 2023-03-10T01:12:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36836.510032 secs 2023-03-10T01:12:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36836.510032 secs 2023-03-10T01:12:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36836.510032 secs 2023-03-10T01:12:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:07 smithi122 ceph-mon[93869]: pgmap v20156: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 864 KiB/s rd, 840 KiB/s wr, 26 op/s 2023-03-10T01:12:07.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:07 smithi181 ceph-mon[94189]: pgmap v20156: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 864 KiB/s rd, 840 KiB/s wr, 26 op/s 2023-03-10T01:12:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:07 smithi038 ceph-mon[92422]: pgmap v20156: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 864 KiB/s rd, 840 KiB/s wr, 26 op/s 2023-03-10T01:12:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:12:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:12:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:12:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:12:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:12:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:12:08.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37300.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:12:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:12:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:12:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:09 smithi122 ceph-mon[93869]: pgmap v20157: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 831 KiB/s rd, 810 KiB/s wr, 24 op/s 2023-03-10T01:12:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:12:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:12:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:09 smithi181 ceph-mon[94189]: pgmap v20157: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 831 KiB/s rd, 810 KiB/s wr, 24 op/s 2023-03-10T01:12:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:12:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:12:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:09 smithi038 ceph-mon[92422]: pgmap v20157: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 831 KiB/s rd, 810 KiB/s wr, 24 op/s 2023-03-10T01:12:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36841.510091 secs 2023-03-10T01:12:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:11 smithi122 ceph-mon[93869]: pgmap v20158: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 957 KiB/s rd, 736 KiB/s wr, 24 op/s 2023-03-10T01:12:12.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36841.510091 secs 2023-03-10T01:12:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:11 smithi181 ceph-mon[94189]: pgmap v20158: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 957 KiB/s rd, 736 KiB/s wr, 24 op/s 2023-03-10T01:12:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36841.510091 secs 2023-03-10T01:12:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:11 smithi038 ceph-mon[92422]: pgmap v20158: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 957 KiB/s rd, 736 KiB/s wr, 24 op/s 2023-03-10T01:12:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37305.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:12:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:13 smithi122 ceph-mon[93869]: pgmap v20159: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 914 KiB/s rd, 661 KiB/s wr, 21 op/s 2023-03-10T01:12:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:13 smithi181 ceph-mon[94189]: pgmap v20159: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 914 KiB/s rd, 661 KiB/s wr, 21 op/s 2023-03-10T01:12:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:13 smithi038 ceph-mon[92422]: pgmap v20159: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 914 KiB/s rd, 661 KiB/s wr, 21 op/s 2023-03-10T01:12:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:15 smithi122 ceph-mon[93869]: pgmap v20160: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 610 KiB/s rd, 357 KiB/s wr, 19 op/s 2023-03-10T01:12:15.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:15 smithi181 ceph-mon[94189]: pgmap v20160: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 610 KiB/s rd, 357 KiB/s wr, 19 op/s 2023-03-10T01:12:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:15 smithi038 ceph-mon[92422]: pgmap v20160: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 610 KiB/s rd, 357 KiB/s wr, 19 op/s 2023-03-10T01:12:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36846.510153 secs 2023-03-10T01:12:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:12:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:12:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36846.510153 secs 2023-03-10T01:12:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:12:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:12:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36846.510153 secs 2023-03-10T01:12:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:12:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:12:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:12:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:12:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:17 smithi122 ceph-mon[93869]: pgmap v20161: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 406 KiB/s rd, 347 KiB/s wr, 11 op/s 2023-03-10T01:12:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:12:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:12:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:17 smithi181 ceph-mon[94189]: pgmap v20161: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 406 KiB/s rd, 347 KiB/s wr, 11 op/s 2023-03-10T01:12:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:12:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:12:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:17 smithi038 ceph-mon[92422]: pgmap v20161: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 406 KiB/s rd, 347 KiB/s wr, 11 op/s 2023-03-10T01:12:18.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37310.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:12:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:19 smithi122 ceph-mon[93869]: pgmap v20162: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 215 KiB/s wr, 9 op/s 2023-03-10T01:12:20.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:19 smithi181 ceph-mon[94189]: pgmap v20162: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 215 KiB/s wr, 9 op/s 2023-03-10T01:12:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:19 smithi038 ceph-mon[92422]: pgmap v20162: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 215 KiB/s wr, 9 op/s 2023-03-10T01:12:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36851.510212 secs 2023-03-10T01:12:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:21 smithi122 ceph-mon[93869]: pgmap v20163: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 492 KiB/s wr, 12 op/s 2023-03-10T01:12:22.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36851.510212 secs 2023-03-10T01:12:22.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:21 smithi181 ceph-mon[94189]: pgmap v20163: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 492 KiB/s wr, 12 op/s 2023-03-10T01:12:22.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36851.510212 secs 2023-03-10T01:12:22.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:21 smithi038 ceph-mon[92422]: pgmap v20163: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 492 KiB/s wr, 12 op/s 2023-03-10T01:12:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37315.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:12:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:23 smithi122 ceph-mon[93869]: pgmap v20164: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 333 KiB/s wr, 11 op/s 2023-03-10T01:12:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:23 smithi181 ceph-mon[94189]: pgmap v20164: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 333 KiB/s wr, 11 op/s 2023-03-10T01:12:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:23 smithi038 ceph-mon[92422]: pgmap v20164: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 333 KiB/s wr, 11 op/s 2023-03-10T01:12:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:25 smithi122 ceph-mon[93869]: pgmap v20165: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 458 KiB/s wr, 12 op/s 2023-03-10T01:12:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:25 smithi122 ceph-mon[93869]: osdmap e1475: 12 total, 12 up, 12 in 2023-03-10T01:12:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:25 smithi181 ceph-mon[94189]: pgmap v20165: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 458 KiB/s wr, 12 op/s 2023-03-10T01:12:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:25 smithi181 ceph-mon[94189]: osdmap e1475: 12 total, 12 up, 12 in 2023-03-10T01:12:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:25 smithi038 ceph-mon[92422]: pgmap v20165: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 458 KiB/s wr, 12 op/s 2023-03-10T01:12:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:25 smithi038 ceph-mon[92422]: osdmap e1475: 12 total, 12 up, 12 in 2023-03-10T01:12:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36856.510274 secs 2023-03-10T01:12:27.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36856.510274 secs 2023-03-10T01:12:27.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36856.510274 secs 2023-03-10T01:12:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:27 smithi122 ceph-mon[93869]: pgmap v20167: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 481 KiB/s rd, 504 KiB/s wr, 13 op/s 2023-03-10T01:12:28.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:27 smithi181 ceph-mon[94189]: pgmap v20167: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 481 KiB/s rd, 504 KiB/s wr, 13 op/s 2023-03-10T01:12:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:27 smithi038 ceph-mon[92422]: pgmap v20167: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 481 KiB/s rd, 504 KiB/s wr, 13 op/s 2023-03-10T01:12:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37320.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:12:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:29 smithi122 ceph-mon[93869]: pgmap v20168: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 682 KiB/s rd, 694 KiB/s wr, 13 op/s 2023-03-10T01:12:30.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:29 smithi181 ceph-mon[94189]: pgmap v20168: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 682 KiB/s rd, 694 KiB/s wr, 13 op/s 2023-03-10T01:12:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:29 smithi038 ceph-mon[92422]: pgmap v20168: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 682 KiB/s rd, 694 KiB/s wr, 13 op/s 2023-03-10T01:12:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:30 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:12:31.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:30 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:12:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:30 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:12:32.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36861.510336 secs 2023-03-10T01:12:32.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:31 smithi122 ceph-mon[93869]: pgmap v20169: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 362 KiB/s wr, 10 op/s 2023-03-10T01:12:32.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:31 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:12:32.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:31 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:12:32.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:12:32.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:12:32.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:12:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36861.510336 secs 2023-03-10T01:12:32.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:31 smithi181 ceph-mon[94189]: pgmap v20169: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 362 KiB/s wr, 10 op/s 2023-03-10T01:12:32.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:31 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:12:32.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:31 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:12:32.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:12:32.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:12:32.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:12:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36861.510336 secs 2023-03-10T01:12:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:31 smithi038 ceph-mon[92422]: pgmap v20169: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 362 KiB/s wr, 10 op/s 2023-03-10T01:12:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:31 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:12:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:31 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:12:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:12:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:12:32.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:12:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37325.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:12:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:33 smithi122 ceph-mon[93869]: pgmap v20170: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 362 KiB/s wr, 10 op/s 2023-03-10T01:12:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:33 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:12:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:33 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:12:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:33 smithi181 ceph-mon[94189]: pgmap v20170: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 362 KiB/s wr, 10 op/s 2023-03-10T01:12:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:33 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:12:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:33 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:12:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:33 smithi038 ceph-mon[92422]: pgmap v20170: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 364 KiB/s rd, 362 KiB/s wr, 10 op/s 2023-03-10T01:12:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:33 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:12:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:33 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:12:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:35 smithi122 ceph-mon[93869]: osdmap e1476: 12 total, 12 up, 12 in 2023-03-10T01:12:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:35 smithi122 ceph-mon[93869]: pgmap v20172: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 893 KiB/s rd, 411 KiB/s wr, 10 op/s 2023-03-10T01:12:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:35 smithi181 ceph-mon[94189]: osdmap e1476: 12 total, 12 up, 12 in 2023-03-10T01:12:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:35 smithi181 ceph-mon[94189]: pgmap v20172: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 893 KiB/s rd, 411 KiB/s wr, 10 op/s 2023-03-10T01:12:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:35 smithi038 ceph-mon[92422]: osdmap e1476: 12 total, 12 up, 12 in 2023-03-10T01:12:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:35 smithi038 ceph-mon[92422]: pgmap v20172: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 893 KiB/s rd, 411 KiB/s wr, 10 op/s 2023-03-10T01:12:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36866.510416 secs 2023-03-10T01:12:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:12:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:12:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36866.510416 secs 2023-03-10T01:12:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:12:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:12:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36866.510416 secs 2023-03-10T01:12:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:12:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:12:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:12:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:12:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:37 smithi122 ceph-mon[93869]: pgmap v20173: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 980 KiB/s rd, 502 KiB/s wr, 10 op/s 2023-03-10T01:12:37.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:12:37.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:12:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:37 smithi181 ceph-mon[94189]: pgmap v20173: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 980 KiB/s rd, 502 KiB/s wr, 10 op/s 2023-03-10T01:12:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:12:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:12:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:37 smithi038 ceph-mon[92422]: pgmap v20173: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 980 KiB/s rd, 502 KiB/s wr, 10 op/s 2023-03-10T01:12:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36868.972261 secs 2023-03-10T01:12:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36868.972261 secs 2023-03-10T01:12:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36868.972261 secs 2023-03-10T01:12:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37330.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:12:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:39 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:12:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:39 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:12:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:39 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:12:39.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:39 smithi122 ceph-mon[93869]: pgmap v20174: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 701 KiB/s wr, 12 op/s 2023-03-10T01:12:39.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:39 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:12:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:39 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:12:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:39 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:12:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:39 smithi181 ceph-mon[94189]: pgmap v20174: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 701 KiB/s wr, 12 op/s 2023-03-10T01:12:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:39 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:12:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:39 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:12:39.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:39 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:12:39.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:39 smithi038 ceph-mon[92422]: pgmap v20174: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 701 KiB/s wr, 12 op/s 2023-03-10T01:12:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:12:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:12:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:12:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:12:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:12:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:12:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36871.510473 secs 2023-03-10T01:12:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:12:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:12:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:41 smithi122 ceph-mon[93869]: pgmap v20175: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1017 KiB/s wr, 12 op/s 2023-03-10T01:12:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36871.510473 secs 2023-03-10T01:12:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:12:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:12:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:41 smithi181 ceph-mon[94189]: pgmap v20175: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1017 KiB/s wr, 12 op/s 2023-03-10T01:12:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36871.510473 secs 2023-03-10T01:12:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:12:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:12:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:41 smithi038 ceph-mon[92422]: pgmap v20175: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1017 KiB/s wr, 12 op/s 2023-03-10T01:12:43.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36873.972331 secs 2023-03-10T01:12:43.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36873.972331 secs 2023-03-10T01:12:43.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36873.972331 secs 2023-03-10T01:12:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37335.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:12:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:43 smithi122 ceph-mon[93869]: pgmap v20176: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.2 MiB/s wr, 13 op/s 2023-03-10T01:12:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:43 smithi181 ceph-mon[94189]: pgmap v20176: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.2 MiB/s wr, 13 op/s 2023-03-10T01:12:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:43 smithi038 ceph-mon[92422]: pgmap v20176: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.2 MiB/s wr, 13 op/s 2023-03-10T01:12:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:45 smithi122 ceph-mon[93869]: pgmap v20177: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 998 KiB/s wr, 12 op/s 2023-03-10T01:12:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:45 smithi181 ceph-mon[94189]: pgmap v20177: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 998 KiB/s wr, 12 op/s 2023-03-10T01:12:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:45 smithi038 ceph-mon[92422]: pgmap v20177: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 998 KiB/s wr, 12 op/s 2023-03-10T01:12:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36876.510532 secs 2023-03-10T01:12:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36876.510532 secs 2023-03-10T01:12:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36876.510532 secs 2023-03-10T01:12:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:47 smithi122 ceph-mon[93869]: pgmap v20178: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1002 KiB/s rd, 867 KiB/s wr, 12 op/s 2023-03-10T01:12:47.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:47 smithi181 ceph-mon[94189]: pgmap v20178: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1002 KiB/s rd, 867 KiB/s wr, 12 op/s 2023-03-10T01:12:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:47 smithi038 ceph-mon[92422]: pgmap v20178: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1002 KiB/s rd, 867 KiB/s wr, 12 op/s 2023-03-10T01:12:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:12:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:12:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:12:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:12:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:12:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:12:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37340.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:12:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:12:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:12:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:49 smithi122 ceph-mon[93869]: pgmap v20179: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 985 KiB/s wr, 13 op/s 2023-03-10T01:12:49.736 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:12:49.736 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:12:49.736 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:49 smithi181 ceph-mon[94189]: pgmap v20179: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 985 KiB/s wr, 13 op/s 2023-03-10T01:12:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:12:49.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:12:49.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:49 smithi038 ceph-mon[92422]: pgmap v20179: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 985 KiB/s wr, 13 op/s 2023-03-10T01:12:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:50 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:12:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:50 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:12:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:50 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:12:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36881.510590 secs 2023-03-10T01:12:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:51 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:12:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:12:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:12:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:51 smithi122 ceph-mon[93869]: pgmap v20180: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 13 op/s 2023-03-10T01:12:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36881.510590 secs 2023-03-10T01:12:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:51 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:12:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:12:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:12:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:51 smithi181 ceph-mon[94189]: pgmap v20180: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 13 op/s 2023-03-10T01:12:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36881.510590 secs 2023-03-10T01:12:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:51 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:12:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:12:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:12:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:51 smithi038 ceph-mon[92422]: pgmap v20180: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 13 op/s 2023-03-10T01:12:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:12:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:12:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:12:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:12:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:12:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:12:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:12:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:12:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:53 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 36883.972502 secs 2023-03-10T01:12:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:53 smithi122 ceph-mon[93869]: pgmap v20181: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 13 op/s 2023-03-10T01:12:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:12:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:12:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:53 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 36883.972502 secs 2023-03-10T01:12:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:53 smithi181 ceph-mon[94189]: pgmap v20181: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 13 op/s 2023-03-10T01:12:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:12:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:12:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:53 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 36883.972502 secs 2023-03-10T01:12:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:53 smithi038 ceph-mon[92422]: pgmap v20181: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 13 op/s 2023-03-10T01:12:53.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37345.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:12:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:12:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:12:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:55 smithi122 ceph-mon[93869]: pgmap v20182: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1015 KiB/s rd, 985 KiB/s wr, 12 op/s 2023-03-10T01:12:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:12:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:12:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:55 smithi181 ceph-mon[94189]: pgmap v20182: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1015 KiB/s rd, 985 KiB/s wr, 12 op/s 2023-03-10T01:12:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:12:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:12:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:55 smithi038 ceph-mon[92422]: pgmap v20182: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1015 KiB/s rd, 985 KiB/s wr, 12 op/s 2023-03-10T01:12:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36886.510650 secs 2023-03-10T01:12:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:12:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:12:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36886.510650 secs 2023-03-10T01:12:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:12:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:12:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36886.510650 secs 2023-03-10T01:12:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:12:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:12:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:12:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:12:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:57 smithi122 ceph-mon[93869]: pgmap v20183: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1014 KiB/s wr, 15 op/s 2023-03-10T01:12:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:12:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:12:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:57 smithi181 ceph-mon[94189]: pgmap v20183: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1014 KiB/s wr, 15 op/s 2023-03-10T01:12:57.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:12:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:12:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:57 smithi038 ceph-mon[92422]: pgmap v20183: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1014 KiB/s wr, 15 op/s 2023-03-10T01:12:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:58 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 36888.972602 secs 2023-03-10T01:12:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:58 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 36888.972602 secs 2023-03-10T01:12:58.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:58 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 36888.972602 secs 2023-03-10T01:12:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37350.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:12:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:59 smithi122 ceph-mon[93869]: pgmap v20184: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 14 op/s 2023-03-10T01:12:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:12:59 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:12:59.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:59 smithi181 ceph-mon[94189]: pgmap v20184: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 14 op/s 2023-03-10T01:12:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:12:59 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:12:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:59 smithi038 ceph-mon[92422]: pgmap v20184: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 14 op/s 2023-03-10T01:12:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:12:59 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:13:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36891.510715 secs 2023-03-10T01:13:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:13:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:13:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:01 smithi122 ceph-mon[93869]: pgmap v20185: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 15 op/s 2023-03-10T01:13:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36891.510715 secs 2023-03-10T01:13:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:13:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:13:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:01 smithi181 ceph-mon[94189]: pgmap v20185: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 15 op/s 2023-03-10T01:13:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36891.510715 secs 2023-03-10T01:13:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:13:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:13:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:01 smithi038 ceph-mon[92422]: pgmap v20185: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 15 op/s 2023-03-10T01:13:03.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36893.972652 secs 2023-03-10T01:13:03.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36893.972652 secs 2023-03-10T01:13:03.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36893.972652 secs 2023-03-10T01:13:03.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37355.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:13:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:03 smithi122 ceph-mon[93869]: pgmap v20186: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 617 KiB/s wr, 13 op/s 2023-03-10T01:13:04.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:03 smithi181 ceph-mon[94189]: pgmap v20186: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 617 KiB/s wr, 13 op/s 2023-03-10T01:13:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:03 smithi038 ceph-mon[92422]: pgmap v20186: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 617 KiB/s wr, 13 op/s 2023-03-10T01:13:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:05 smithi122 ceph-mon[93869]: pgmap v20187: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 618 KiB/s wr, 13 op/s 2023-03-10T01:13:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:05 smithi181 ceph-mon[94189]: pgmap v20187: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 618 KiB/s wr, 13 op/s 2023-03-10T01:13:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:05 smithi038 ceph-mon[92422]: pgmap v20187: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 618 KiB/s wr, 13 op/s 2023-03-10T01:13:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36896.510776 secs 2023-03-10T01:13:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36896.510776 secs 2023-03-10T01:13:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36896.510776 secs 2023-03-10T01:13:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:13:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:13:07.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:07 smithi122 ceph-mon[93869]: pgmap v20188: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 650 KiB/s rd, 661 KiB/s wr, 17 op/s 2023-03-10T01:13:07.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:13:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:13:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:07 smithi181 ceph-mon[94189]: pgmap v20188: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 650 KiB/s rd, 661 KiB/s wr, 17 op/s 2023-03-10T01:13:07.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:13:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:13:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:07 smithi038 ceph-mon[92422]: pgmap v20188: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 650 KiB/s rd, 661 KiB/s wr, 17 op/s 2023-03-10T01:13:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:13:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:13:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36898.972712 secs 2023-03-10T01:13:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:13:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:13:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36898.972712 secs 2023-03-10T01:13:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:13:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:13:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36898.972712 secs 2023-03-10T01:13:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37360.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:13:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:09 smithi122 ceph-mon[93869]: pgmap v20189: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 668 KiB/s wr, 24 op/s 2023-03-10T01:13:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:09 smithi181 ceph-mon[94189]: pgmap v20189: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 668 KiB/s wr, 24 op/s 2023-03-10T01:13:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:09 smithi038 ceph-mon[92422]: pgmap v20189: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 668 KiB/s wr, 24 op/s 2023-03-10T01:13:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36901.510859 secs 2023-03-10T01:13:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:11 smithi122 ceph-mon[93869]: pgmap v20190: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 764 KiB/s rd, 634 KiB/s wr, 24 op/s 2023-03-10T01:13:12.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36901.510859 secs 2023-03-10T01:13:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:11 smithi181 ceph-mon[94189]: pgmap v20190: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 764 KiB/s rd, 634 KiB/s wr, 24 op/s 2023-03-10T01:13:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36901.510859 secs 2023-03-10T01:13:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:11 smithi038 ceph-mon[92422]: pgmap v20190: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 764 KiB/s rd, 634 KiB/s wr, 24 op/s 2023-03-10T01:13:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36903.972836 secs 2023-03-10T01:13:13.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36903.972836 secs 2023-03-10T01:13:13.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36903.972836 secs 2023-03-10T01:13:13.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37365.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:13:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:13 smithi122 ceph-mon[93869]: pgmap v20191: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 358 KiB/s wr, 22 op/s 2023-03-10T01:13:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:13:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:13:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:13 smithi181 ceph-mon[94189]: pgmap v20191: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 358 KiB/s wr, 22 op/s 2023-03-10T01:13:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:13:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:13:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:13 smithi038 ceph-mon[92422]: pgmap v20191: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 358 KiB/s wr, 22 op/s 2023-03-10T01:13:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:13:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:13:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:15 smithi122 ceph-mon[93869]: pgmap v20192: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 358 KiB/s wr, 21 op/s 2023-03-10T01:13:15.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:15 smithi181 ceph-mon[94189]: pgmap v20192: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 358 KiB/s wr, 21 op/s 2023-03-10T01:13:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:15 smithi038 ceph-mon[92422]: pgmap v20192: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 358 KiB/s wr, 21 op/s 2023-03-10T01:13:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36906.510918 secs 2023-03-10T01:13:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:13:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:13:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36906.510918 secs 2023-03-10T01:13:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:13:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:13:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36906.510918 secs 2023-03-10T01:13:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:13:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:13:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:17 smithi122 ceph-mon[93869]: pgmap v20193: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 404 KiB/s wr, 21 op/s 2023-03-10T01:13:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:17 smithi181 ceph-mon[94189]: pgmap v20193: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 404 KiB/s wr, 21 op/s 2023-03-10T01:13:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:17 smithi038 ceph-mon[92422]: pgmap v20193: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 404 KiB/s wr, 21 op/s 2023-03-10T01:13:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36908.972895 secs 2023-03-10T01:13:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36908.972895 secs 2023-03-10T01:13:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36908.972895 secs 2023-03-10T01:13:18.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37370.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:13:19.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:19 smithi122 ceph-mon[93869]: pgmap v20194: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 460 KiB/s wr, 18 op/s 2023-03-10T01:13:19.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:19 smithi181 ceph-mon[94189]: pgmap v20194: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 460 KiB/s wr, 18 op/s 2023-03-10T01:13:19.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:19 smithi038 ceph-mon[92422]: pgmap v20194: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 460 KiB/s wr, 18 op/s 2023-03-10T01:13:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36911.510999 secs 2023-03-10T01:13:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:21 smithi122 ceph-mon[93869]: pgmap v20195: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 419 KiB/s wr, 8 op/s 2023-03-10T01:13:22.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36911.510999 secs 2023-03-10T01:13:22.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:21 smithi181 ceph-mon[94189]: pgmap v20195: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 419 KiB/s wr, 8 op/s 2023-03-10T01:13:22.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36911.510999 secs 2023-03-10T01:13:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:21 smithi038 ceph-mon[92422]: pgmap v20195: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 419 KiB/s wr, 8 op/s 2023-03-10T01:13:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36913.972956 secs 2023-03-10T01:13:23.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36913.972956 secs 2023-03-10T01:13:23.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36913.972956 secs 2023-03-10T01:13:23.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37375.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:13:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:23 smithi122 ceph-mon[93869]: pgmap v20196: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 419 KiB/s wr, 9 op/s 2023-03-10T01:13:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:23 smithi181 ceph-mon[94189]: pgmap v20196: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 419 KiB/s wr, 9 op/s 2023-03-10T01:13:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:23 smithi038 ceph-mon[92422]: pgmap v20196: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 419 KiB/s wr, 9 op/s 2023-03-10T01:13:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:25 smithi122 ceph-mon[93869]: pgmap v20197: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 219 KiB/s wr, 9 op/s 2023-03-10T01:13:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:25 smithi122 ceph-mon[93869]: osdmap e1477: 12 total, 12 up, 12 in 2023-03-10T01:13:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36916.511060 secs 2023-03-10T01:13:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:25 smithi181 ceph-mon[94189]: pgmap v20197: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 219 KiB/s wr, 9 op/s 2023-03-10T01:13:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:25 smithi181 ceph-mon[94189]: osdmap e1477: 12 total, 12 up, 12 in 2023-03-10T01:13:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36916.511060 secs 2023-03-10T01:13:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:25 smithi038 ceph-mon[92422]: pgmap v20197: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 219 KiB/s wr, 9 op/s 2023-03-10T01:13:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:25 smithi038 ceph-mon[92422]: osdmap e1477: 12 total, 12 up, 12 in 2023-03-10T01:13:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36916.511060 secs 2023-03-10T01:13:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:27 smithi122 ceph-mon[93869]: pgmap v20199: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 482 KiB/s wr, 13 op/s 2023-03-10T01:13:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:27 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:13:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:27 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:13:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36918.973055 secs 2023-03-10T01:13:28.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:27 smithi181 ceph-mon[94189]: pgmap v20199: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 482 KiB/s wr, 13 op/s 2023-03-10T01:13:28.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:27 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:13:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:27 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:13:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36918.973055 secs 2023-03-10T01:13:28.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:27 smithi038 ceph-mon[92422]: pgmap v20199: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 482 KiB/s wr, 13 op/s 2023-03-10T01:13:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:27 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:13:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:27 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:13:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36918.973055 secs 2023-03-10T01:13:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37380.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:13:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:29 smithi122 ceph-mon[93869]: pgmap v20200: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 327 KiB/s wr, 12 op/s 2023-03-10T01:13:30.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:29 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:13:30.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:29 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:13:30.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:29 smithi181 ceph-mon[94189]: pgmap v20200: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 327 KiB/s wr, 12 op/s 2023-03-10T01:13:30.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:29 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:13:30.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:29 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:13:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:29 smithi038 ceph-mon[92422]: pgmap v20200: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 327 KiB/s wr, 12 op/s 2023-03-10T01:13:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:29 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:13:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:29 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:13:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36921.511117 secs 2023-03-10T01:13:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36921.511117 secs 2023-03-10T01:13:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36921.511117 secs 2023-03-10T01:13:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:31 smithi122 ceph-mon[93869]: pgmap v20201: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 327 KiB/s wr, 12 op/s 2023-03-10T01:13:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:13:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:31 smithi181 ceph-mon[94189]: pgmap v20201: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 327 KiB/s wr, 12 op/s 2023-03-10T01:13:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:13:32.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:31 smithi038 ceph-mon[92422]: pgmap v20201: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 327 KiB/s wr, 12 op/s 2023-03-10T01:13:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:13:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36923.973117 secs 2023-03-10T01:13:33.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36923.973117 secs 2023-03-10T01:13:33.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36923.973117 secs 2023-03-10T01:13:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37385.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:13:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:33 smithi122 ceph-mon[93869]: pgmap v20202: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 395 KiB/s rd, 419 KiB/s wr, 12 op/s 2023-03-10T01:13:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:33 smithi181 ceph-mon[94189]: pgmap v20202: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 395 KiB/s rd, 419 KiB/s wr, 12 op/s 2023-03-10T01:13:34.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:33 smithi038 ceph-mon[92422]: pgmap v20202: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 395 KiB/s rd, 419 KiB/s wr, 12 op/s 2023-03-10T01:13:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:35 smithi122 ceph-mon[93869]: osdmap e1478: 12 total, 12 up, 12 in 2023-03-10T01:13:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:13:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:13:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:13:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:13:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:13:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:13:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:35 smithi122 ceph-mon[93869]: pgmap v20204: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 697 KiB/s rd, 709 KiB/s wr, 11 op/s 2023-03-10T01:13:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:13:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:13:35.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:13:35.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:13:35.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:13:35.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:13:35.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:13:35.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:13:35.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:13:35.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:13:35.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:13:35.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:13:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:35 smithi181 ceph-mon[94189]: osdmap e1478: 12 total, 12 up, 12 in 2023-03-10T01:13:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:13:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:13:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:13:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:13:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:13:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:13:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:35 smithi181 ceph-mon[94189]: pgmap v20204: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 697 KiB/s rd, 709 KiB/s wr, 11 op/s 2023-03-10T01:13:35.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:13:35.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:13:35.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:13:35.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:13:35.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:13:35.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:13:35.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:13:35.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:13:35.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:13:35.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:13:35.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:13:35.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:13:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:35 smithi038 ceph-mon[92422]: osdmap e1478: 12 total, 12 up, 12 in 2023-03-10T01:13:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:13:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:13:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:13:35.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:13:35.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:13:35.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:13:35.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:35 smithi038 ceph-mon[92422]: pgmap v20204: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 697 KiB/s rd, 709 KiB/s wr, 11 op/s 2023-03-10T01:13:35.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:13:35.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:13:35.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:13:35.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:13:35.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:13:35.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:13:35.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:13:35.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:13:35.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:13:35.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:13:35.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:13:35.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:13:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36926.511180 secs 2023-03-10T01:13:36.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:36 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-10T01:13:36.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:36 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-10T01:13:36.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:36 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-10T01:13:36.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:36 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-10T01:13:36.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:13:36.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:13:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36926.511180 secs 2023-03-10T01:13:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:36 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-10T01:13:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:36 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-10T01:13:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:36 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-10T01:13:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:36 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-10T01:13:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:13:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:13:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36926.511180 secs 2023-03-10T01:13:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:36 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-10T01:13:37.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:36 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-10T01:13:37.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:36 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-10T01:13:37.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:36 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-10T01:13:37.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:13:37.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:13:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:13:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:13:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:37 smithi122 ceph-mon[93869]: pgmap v20205: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 430 KiB/s rd, 429 KiB/s wr, 9 op/s 2023-03-10T01:13:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:13:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:13:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:37 smithi181 ceph-mon[94189]: pgmap v20205: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 430 KiB/s rd, 429 KiB/s wr, 9 op/s 2023-03-10T01:13:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:13:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:13:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:37 smithi038 ceph-mon[92422]: pgmap v20205: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 430 KiB/s rd, 429 KiB/s wr, 9 op/s 2023-03-10T01:13:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37390.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:13:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36928.973210 secs 2023-03-10T01:13:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36928.973210 secs 2023-03-10T01:13:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36928.973210 secs 2023-03-10T01:13:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:39 smithi122 ceph-mon[93869]: pgmap v20206: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 430 KiB/s rd, 428 KiB/s wr, 9 op/s 2023-03-10T01:13:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:39 smithi181 ceph-mon[94189]: pgmap v20206: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 430 KiB/s rd, 428 KiB/s wr, 9 op/s 2023-03-10T01:13:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:39 smithi038 ceph-mon[92422]: pgmap v20206: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 430 KiB/s rd, 428 KiB/s wr, 9 op/s 2023-03-10T01:13:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36931.511240 secs 2023-03-10T01:13:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:41 smithi122 ceph-mon[93869]: pgmap v20207: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 430 KiB/s rd, 428 KiB/s wr, 9 op/s 2023-03-10T01:13:42.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36931.511240 secs 2023-03-10T01:13:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:41 smithi181 ceph-mon[94189]: pgmap v20207: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 430 KiB/s rd, 428 KiB/s wr, 9 op/s 2023-03-10T01:13:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36931.511240 secs 2023-03-10T01:13:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:41 smithi038 ceph-mon[92422]: pgmap v20207: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 430 KiB/s rd, 428 KiB/s wr, 9 op/s 2023-03-10T01:13:43.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36933.973311 secs 2023-03-10T01:13:43.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36933.973311 secs 2023-03-10T01:13:43.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36933.973311 secs 2023-03-10T01:13:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37395.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:13:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:43 smithi122 ceph-mon[93869]: pgmap v20208: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 469 KiB/s rd, 467 KiB/s wr, 9 op/s 2023-03-10T01:13:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:43 smithi181 ceph-mon[94189]: pgmap v20208: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 469 KiB/s rd, 467 KiB/s wr, 9 op/s 2023-03-10T01:13:44.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:43 smithi038 ceph-mon[92422]: pgmap v20208: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 469 KiB/s rd, 467 KiB/s wr, 9 op/s 2023-03-10T01:13:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:45 smithi122 ceph-mon[93869]: pgmap v20209: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 199 KiB/s wr, 8 op/s 2023-03-10T01:13:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:45 smithi181 ceph-mon[94189]: pgmap v20209: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 199 KiB/s wr, 8 op/s 2023-03-10T01:13:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:45 smithi038 ceph-mon[92422]: pgmap v20209: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 199 KiB/s wr, 8 op/s 2023-03-10T01:13:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36936.511319 secs 2023-03-10T01:13:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36936.511319 secs 2023-03-10T01:13:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36936.511319 secs 2023-03-10T01:13:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:47 smithi122 ceph-mon[93869]: pgmap v20210: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 173 KiB/s wr, 8 op/s 2023-03-10T01:13:47.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:47 smithi181 ceph-mon[94189]: pgmap v20210: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 173 KiB/s wr, 8 op/s 2023-03-10T01:13:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:47 smithi038 ceph-mon[92422]: pgmap v20210: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 176 KiB/s rd, 173 KiB/s wr, 8 op/s 2023-03-10T01:13:48.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36938.973427 secs 2023-03-10T01:13:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36938.973427 secs 2023-03-10T01:13:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36938.973427 secs 2023-03-10T01:13:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37400.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:13:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:13:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:13:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:49 smithi122 ceph-mon[93869]: pgmap v20211: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 561 KiB/s rd, 405 KiB/s wr, 8 op/s 2023-03-10T01:13:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:13:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:13:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:49 smithi181 ceph-mon[94189]: pgmap v20211: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 561 KiB/s rd, 405 KiB/s wr, 8 op/s 2023-03-10T01:13:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:13:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:13:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:49 smithi038 ceph-mon[92422]: pgmap v20211: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 561 KiB/s rd, 405 KiB/s wr, 8 op/s 2023-03-10T01:13:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36941.511399 secs 2023-03-10T01:13:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:13:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:13:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:51 smithi122 ceph-mon[93869]: pgmap v20212: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 562 KiB/s rd, 404 KiB/s wr, 8 op/s 2023-03-10T01:13:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36941.511399 secs 2023-03-10T01:13:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:13:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:13:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:51 smithi181 ceph-mon[94189]: pgmap v20212: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 562 KiB/s rd, 404 KiB/s wr, 8 op/s 2023-03-10T01:13:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36941.511399 secs 2023-03-10T01:13:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:13:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:13:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:51 smithi038 ceph-mon[92422]: pgmap v20212: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 562 KiB/s rd, 404 KiB/s wr, 8 op/s 2023-03-10T01:13:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37405.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:13:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36943.973488 secs 2023-03-10T01:13:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:13:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:13:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:53 smithi122 ceph-mon[93869]: pgmap v20213: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 794 KiB/s rd, 404 KiB/s wr, 8 op/s 2023-03-10T01:13:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36943.973488 secs 2023-03-10T01:13:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:13:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:13:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:53 smithi181 ceph-mon[94189]: pgmap v20213: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 794 KiB/s rd, 404 KiB/s wr, 8 op/s 2023-03-10T01:13:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36943.973488 secs 2023-03-10T01:13:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:13:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:13:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:53 smithi038 ceph-mon[92422]: pgmap v20213: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 794 KiB/s rd, 404 KiB/s wr, 8 op/s 2023-03-10T01:13:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:55 smithi122 ceph-mon[93869]: pgmap v20214: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 940 KiB/s rd, 550 KiB/s wr, 9 op/s 2023-03-10T01:13:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:55 smithi181 ceph-mon[94189]: pgmap v20214: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 940 KiB/s rd, 550 KiB/s wr, 9 op/s 2023-03-10T01:13:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:55 smithi038 ceph-mon[92422]: pgmap v20214: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 940 KiB/s rd, 550 KiB/s wr, 9 op/s 2023-03-10T01:13:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36946.511459 secs 2023-03-10T01:13:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:13:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:13:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36946.511459 secs 2023-03-10T01:13:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:13:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:13:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36946.511459 secs 2023-03-10T01:13:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:13:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:13:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:57 smithi122 ceph-mon[93869]: pgmap v20215: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 881 KiB/s rd, 488 KiB/s wr, 10 op/s 2023-03-10T01:13:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:57 smithi181 ceph-mon[94189]: pgmap v20215: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 881 KiB/s rd, 488 KiB/s wr, 10 op/s 2023-03-10T01:13:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:57 smithi038 ceph-mon[92422]: pgmap v20215: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 881 KiB/s rd, 488 KiB/s wr, 10 op/s 2023-03-10T01:13:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:58 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:13:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:58 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:13:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:58 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:13:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37410.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:13:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:59 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:13:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:13:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:13:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:13:59 smithi122 ceph-mon[93869]: pgmap v20216: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 910 KiB/s rd, 519 KiB/s wr, 10 op/s 2023-03-10T01:13:59.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:59 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:13:59.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:13:59.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:13:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:13:59 smithi181 ceph-mon[94189]: pgmap v20216: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 910 KiB/s rd, 519 KiB/s wr, 10 op/s 2023-03-10T01:13:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:59 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:13:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:13:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:13:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:13:59 smithi038 ceph-mon[92422]: pgmap v20216: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 910 KiB/s rd, 519 KiB/s wr, 10 op/s 2023-03-10T01:14:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:14:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:14:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:14:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:14:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:14:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:14:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36951.511546 secs 2023-03-10T01:14:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:14:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:14:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:01 smithi122 ceph-mon[93869]: pgmap v20217: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 756 KiB/s wr, 13 op/s 2023-03-10T01:14:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36951.511546 secs 2023-03-10T01:14:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:14:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:14:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:01 smithi181 ceph-mon[94189]: pgmap v20217: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 756 KiB/s wr, 13 op/s 2023-03-10T01:14:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36951.511546 secs 2023-03-10T01:14:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:14:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:14:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:01 smithi038 ceph-mon[92422]: pgmap v20217: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 756 KiB/s wr, 13 op/s 2023-03-10T01:14:03.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37415.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:14:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:03 smithi122 ceph-mon[93869]: pgmap v20218: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 911 KiB/s wr, 14 op/s 2023-03-10T01:14:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:03 smithi181 ceph-mon[94189]: pgmap v20218: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 911 KiB/s wr, 14 op/s 2023-03-10T01:14:04.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:03 smithi038 ceph-mon[92422]: pgmap v20218: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 911 KiB/s wr, 14 op/s 2023-03-10T01:14:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:05 smithi122 ceph-mon[93869]: pgmap v20219: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 952 KiB/s wr, 25 op/s 2023-03-10T01:14:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:05 smithi181 ceph-mon[94189]: pgmap v20219: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 952 KiB/s wr, 25 op/s 2023-03-10T01:14:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:05 smithi038 ceph-mon[92422]: pgmap v20219: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 952 KiB/s wr, 25 op/s 2023-03-10T01:14:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36956.511624 secs 2023-03-10T01:14:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36956.511624 secs 2023-03-10T01:14:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36956.511624 secs 2023-03-10T01:14:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:07 smithi122 ceph-mon[93869]: pgmap v20220: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 945 KiB/s rd, 697 KiB/s wr, 25 op/s 2023-03-10T01:14:07.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:07 smithi181 ceph-mon[94189]: pgmap v20220: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 945 KiB/s rd, 697 KiB/s wr, 25 op/s 2023-03-10T01:14:07.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:07 smithi038 ceph-mon[92422]: pgmap v20220: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 945 KiB/s rd, 697 KiB/s wr, 25 op/s 2023-03-10T01:14:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:14:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:14:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:14:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:14:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:14:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:14:08.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37420.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:14:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:14:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:14:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:09 smithi122 ceph-mon[93869]: pgmap v20221: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 998 KiB/s wr, 25 op/s 2023-03-10T01:14:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:14:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:14:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:09 smithi181 ceph-mon[94189]: pgmap v20221: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 998 KiB/s wr, 25 op/s 2023-03-10T01:14:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:14:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:14:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:09 smithi038 ceph-mon[92422]: pgmap v20221: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 998 KiB/s wr, 25 op/s 2023-03-10T01:14:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36961.511683 secs 2023-03-10T01:14:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:11 smithi122 ceph-mon[93869]: pgmap v20222: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 967 KiB/s wr, 25 op/s 2023-03-10T01:14:12.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36961.511683 secs 2023-03-10T01:14:12.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:11 smithi181 ceph-mon[94189]: pgmap v20222: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 967 KiB/s wr, 25 op/s 2023-03-10T01:14:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36961.511683 secs 2023-03-10T01:14:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:11 smithi038 ceph-mon[92422]: pgmap v20222: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 967 KiB/s wr, 25 op/s 2023-03-10T01:14:13.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37425.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:14:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:13 smithi122 ceph-mon[93869]: pgmap v20223: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 668 KiB/s rd, 670 KiB/s wr, 22 op/s 2023-03-10T01:14:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:13 smithi181 ceph-mon[94189]: pgmap v20223: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 668 KiB/s rd, 670 KiB/s wr, 22 op/s 2023-03-10T01:14:14.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:13 smithi038 ceph-mon[92422]: pgmap v20223: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 668 KiB/s rd, 670 KiB/s wr, 22 op/s 2023-03-10T01:14:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:15 smithi122 ceph-mon[93869]: pgmap v20224: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 513 KiB/s rd, 515 KiB/s wr, 21 op/s 2023-03-10T01:14:15.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:15 smithi181 ceph-mon[94189]: pgmap v20224: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 513 KiB/s rd, 515 KiB/s wr, 21 op/s 2023-03-10T01:14:15.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:15 smithi038 ceph-mon[92422]: pgmap v20224: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 513 KiB/s rd, 515 KiB/s wr, 21 op/s 2023-03-10T01:14:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36966.511739 secs 2023-03-10T01:14:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:14:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:14:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36966.511739 secs 2023-03-10T01:14:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:14:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:14:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36966.511739 secs 2023-03-10T01:14:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:14:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:14:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:14:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:14:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:17 smithi122 ceph-mon[93869]: pgmap v20225: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 474 KiB/s wr, 10 op/s 2023-03-10T01:14:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:14:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:14:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:17 smithi181 ceph-mon[94189]: pgmap v20225: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 474 KiB/s wr, 10 op/s 2023-03-10T01:14:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:14:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:14:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:17 smithi038 ceph-mon[92422]: pgmap v20225: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 474 KiB/s wr, 10 op/s 2023-03-10T01:14:18.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37430.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:14:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:19 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:14:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:19 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:14:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:19 smithi122 ceph-mon[93869]: pgmap v20226: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 761 KiB/s rd, 769 KiB/s wr, 11 op/s 2023-03-10T01:14:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:19 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:14:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:19 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:14:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:19 smithi181 ceph-mon[94189]: pgmap v20226: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 761 KiB/s rd, 769 KiB/s wr, 11 op/s 2023-03-10T01:14:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:19 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:14:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:19 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:14:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:19 smithi038 ceph-mon[92422]: pgmap v20226: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 761 KiB/s rd, 769 KiB/s wr, 11 op/s 2023-03-10T01:14:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36971.511882 secs 2023-03-10T01:14:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:14:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:14:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:21 smithi122 ceph-mon[93869]: pgmap v20227: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 509 KiB/s wr, 11 op/s 2023-03-10T01:14:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36971.511882 secs 2023-03-10T01:14:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:14:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:14:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:21 smithi181 ceph-mon[94189]: pgmap v20227: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 509 KiB/s wr, 11 op/s 2023-03-10T01:14:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36971.511882 secs 2023-03-10T01:14:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:14:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:14:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:21 smithi038 ceph-mon[92422]: pgmap v20227: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 509 KiB/s wr, 11 op/s 2023-03-10T01:14:23.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37435.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:14:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:23 smithi122 ceph-mon[93869]: pgmap v20228: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 545 KiB/s rd, 565 KiB/s wr, 11 op/s 2023-03-10T01:14:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:23 smithi181 ceph-mon[94189]: pgmap v20228: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 545 KiB/s rd, 565 KiB/s wr, 11 op/s 2023-03-10T01:14:24.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:23 smithi038 ceph-mon[92422]: pgmap v20228: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 545 KiB/s rd, 565 KiB/s wr, 11 op/s 2023-03-10T01:14:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:25 smithi122 ceph-mon[93869]: pgmap v20229: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 430 KiB/s rd, 450 KiB/s wr, 11 op/s 2023-03-10T01:14:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:25 smithi122 ceph-mon[93869]: osdmap e1479: 12 total, 12 up, 12 in 2023-03-10T01:14:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36976.511942 secs 2023-03-10T01:14:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:25 smithi181 ceph-mon[94189]: pgmap v20229: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 430 KiB/s rd, 450 KiB/s wr, 11 op/s 2023-03-10T01:14:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:25 smithi181 ceph-mon[94189]: osdmap e1479: 12 total, 12 up, 12 in 2023-03-10T01:14:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36976.511942 secs 2023-03-10T01:14:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:25 smithi038 ceph-mon[92422]: pgmap v20229: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 430 KiB/s rd, 450 KiB/s wr, 11 op/s 2023-03-10T01:14:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:25 smithi038 ceph-mon[92422]: osdmap e1479: 12 total, 12 up, 12 in 2023-03-10T01:14:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36976.511942 secs 2023-03-10T01:14:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:27 smithi122 ceph-mon[93869]: pgmap v20231: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 805 KiB/s rd, 829 KiB/s wr, 13 op/s 2023-03-10T01:14:28.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:27 smithi181 ceph-mon[94189]: pgmap v20231: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 805 KiB/s rd, 829 KiB/s wr, 13 op/s 2023-03-10T01:14:28.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:27 smithi038 ceph-mon[92422]: pgmap v20231: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 805 KiB/s rd, 829 KiB/s wr, 13 op/s 2023-03-10T01:14:28.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37440.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:14:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:29 smithi122 ceph-mon[93869]: pgmap v20232: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 589 KiB/s rd, 474 KiB/s wr, 11 op/s 2023-03-10T01:14:30.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:29 smithi181 ceph-mon[94189]: pgmap v20232: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 589 KiB/s rd, 474 KiB/s wr, 11 op/s 2023-03-10T01:14:30.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:29 smithi038 ceph-mon[92422]: pgmap v20232: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 589 KiB/s rd, 474 KiB/s wr, 11 op/s 2023-03-10T01:14:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36981.512020 secs 2023-03-10T01:14:31.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36981.512020 secs 2023-03-10T01:14:31.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36981.512020 secs 2023-03-10T01:14:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:31 smithi122 ceph-mon[93869]: pgmap v20233: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 581 KiB/s wr, 11 op/s 2023-03-10T01:14:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:31 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:14:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:31 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:14:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:31 smithi181 ceph-mon[94189]: pgmap v20233: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 581 KiB/s wr, 11 op/s 2023-03-10T01:14:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:31 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:14:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:31 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:14:32.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:31 smithi038 ceph-mon[92422]: pgmap v20233: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 581 KiB/s wr, 11 op/s 2023-03-10T01:14:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:31 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:14:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:31 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:14:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:32 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:14:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:32 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:14:33.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:32 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:14:33.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:32 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:14:33.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:32 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:14:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:32 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:14:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37445.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:14:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:33 smithi122 ceph-mon[93869]: pgmap v20234: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 644 KiB/s rd, 515 KiB/s wr, 10 op/s 2023-03-10T01:14:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:33 smithi181 ceph-mon[94189]: pgmap v20234: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 644 KiB/s rd, 515 KiB/s wr, 10 op/s 2023-03-10T01:14:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:33 smithi038 ceph-mon[92422]: pgmap v20234: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 644 KiB/s rd, 515 KiB/s wr, 10 op/s 2023-03-10T01:14:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:35 smithi122 ceph-mon[93869]: osdmap e1480: 12 total, 12 up, 12 in 2023-03-10T01:14:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:35 smithi122 ceph-mon[93869]: pgmap v20236: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 579 KiB/s rd, 450 KiB/s wr, 9 op/s 2023-03-10T01:14:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:14:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:35 smithi181 ceph-mon[94189]: osdmap e1480: 12 total, 12 up, 12 in 2023-03-10T01:14:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:35 smithi181 ceph-mon[94189]: pgmap v20236: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 579 KiB/s rd, 450 KiB/s wr, 9 op/s 2023-03-10T01:14:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:14:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:35 smithi038 ceph-mon[92422]: osdmap e1480: 12 total, 12 up, 12 in 2023-03-10T01:14:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:35 smithi038 ceph-mon[92422]: pgmap v20236: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 579 KiB/s rd, 450 KiB/s wr, 9 op/s 2023-03-10T01:14:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:14:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36986.512108 secs 2023-03-10T01:14:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:14:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:14:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:14:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36986.512108 secs 2023-03-10T01:14:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:14:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:14:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:14:36.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36986.512108 secs 2023-03-10T01:14:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:14:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:14:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:14:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:37 smithi122 ceph-mon[93869]: pgmap v20237: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 467 KiB/s wr, 10 op/s 2023-03-10T01:14:37.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:37 smithi181 ceph-mon[94189]: pgmap v20237: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 467 KiB/s wr, 10 op/s 2023-03-10T01:14:37.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:37 smithi038 ceph-mon[92422]: pgmap v20237: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 467 KiB/s wr, 10 op/s 2023-03-10T01:14:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36988.974140 secs 2023-03-10T01:14:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36988.974140 secs 2023-03-10T01:14:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36988.974140 secs 2023-03-10T01:14:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37450.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:14:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:39 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:14:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:39 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:14:39.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:39 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:14:39.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:39 smithi122 ceph-mon[93869]: pgmap v20238: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 467 KiB/s wr, 9 op/s 2023-03-10T01:14:39.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:39 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:14:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:39 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:14:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:39 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:14:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:39 smithi181 ceph-mon[94189]: pgmap v20238: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 467 KiB/s wr, 9 op/s 2023-03-10T01:14:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:39 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:14:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:39 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:14:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:39 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:14:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:39 smithi038 ceph-mon[92422]: pgmap v20238: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 467 KiB/s wr, 9 op/s 2023-03-10T01:14:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:14:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:14:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:14:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:14:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:14:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:14:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36991.512171 secs 2023-03-10T01:14:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:14:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:14:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:41 smithi122 ceph-mon[93869]: pgmap v20239: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 448 KiB/s wr, 10 op/s 2023-03-10T01:14:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36991.512171 secs 2023-03-10T01:14:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:14:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:14:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:41 smithi181 ceph-mon[94189]: pgmap v20239: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 448 KiB/s wr, 10 op/s 2023-03-10T01:14:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36991.512171 secs 2023-03-10T01:14:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:14:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:14:41.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:41 smithi038 ceph-mon[92422]: pgmap v20239: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 448 KiB/s wr, 10 op/s 2023-03-10T01:14:43.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36993.974253 secs 2023-03-10T01:14:43.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36993.974253 secs 2023-03-10T01:14:43.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36993.974253 secs 2023-03-10T01:14:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37455.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:14:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:43 smithi122 ceph-mon[93869]: pgmap v20240: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 773 KiB/s wr, 11 op/s 2023-03-10T01:14:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:43 smithi181 ceph-mon[94189]: pgmap v20240: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 773 KiB/s wr, 11 op/s 2023-03-10T01:14:44.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:43 smithi038 ceph-mon[92422]: pgmap v20240: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 773 KiB/s wr, 11 op/s 2023-03-10T01:14:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:45 smithi122 ceph-mon[93869]: pgmap v20241: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 741 KiB/s wr, 10 op/s 2023-03-10T01:14:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:45 smithi181 ceph-mon[94189]: pgmap v20241: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 741 KiB/s wr, 10 op/s 2023-03-10T01:14:45.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:45 smithi038 ceph-mon[92422]: pgmap v20241: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 741 KiB/s wr, 10 op/s 2023-03-10T01:14:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 36996.512230 secs 2023-03-10T01:14:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 36996.512230 secs 2023-03-10T01:14:46.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 36996.512230 secs 2023-03-10T01:14:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:47 smithi122 ceph-mon[93869]: pgmap v20242: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 953 KiB/s wr, 12 op/s 2023-03-10T01:14:47.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:47 smithi181 ceph-mon[94189]: pgmap v20242: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 953 KiB/s wr, 12 op/s 2023-03-10T01:14:47.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:47 smithi038 ceph-mon[92422]: pgmap v20242: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 953 KiB/s wr, 12 op/s 2023-03-10T01:14:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:14:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:14:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:14:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:14:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:14:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:14:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37460.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:14:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:14:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:14:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:49 smithi122 ceph-mon[93869]: pgmap v20243: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 696 KiB/s wr, 9 op/s 2023-03-10T01:14:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:14:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:14:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:49 smithi181 ceph-mon[94189]: pgmap v20243: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 696 KiB/s wr, 9 op/s 2023-03-10T01:14:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:14:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:14:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:49 smithi038 ceph-mon[92422]: pgmap v20243: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 696 KiB/s wr, 9 op/s 2023-03-10T01:14:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:50 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:14:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:50 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:14:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:50 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:14:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37001.512288 secs 2023-03-10T01:14:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:51 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:14:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:14:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:14:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:51 smithi122 ceph-mon[93869]: pgmap v20244: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 13 op/s 2023-03-10T01:14:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37001.512288 secs 2023-03-10T01:14:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:51 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:14:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:14:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:14:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:51 smithi181 ceph-mon[94189]: pgmap v20244: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 13 op/s 2023-03-10T01:14:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37001.512288 secs 2023-03-10T01:14:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:51 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:14:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:14:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:14:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:51 smithi038 ceph-mon[92422]: pgmap v20244: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 13 op/s 2023-03-10T01:14:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:14:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:14:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:14:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:14:52.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:14:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:14:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:14:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:14:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:53 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 37003.974396 secs 2023-03-10T01:14:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:53 smithi122 ceph-mon[93869]: pgmap v20245: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 988 KiB/s rd, 984 KiB/s wr, 11 op/s 2023-03-10T01:14:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:14:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:14:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:53 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 37003.974396 secs 2023-03-10T01:14:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:53 smithi181 ceph-mon[94189]: pgmap v20245: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 988 KiB/s rd, 984 KiB/s wr, 11 op/s 2023-03-10T01:14:53.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:14:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:14:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:53 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 37003.974396 secs 2023-03-10T01:14:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:53 smithi038 ceph-mon[92422]: pgmap v20245: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 988 KiB/s rd, 984 KiB/s wr, 11 op/s 2023-03-10T01:14:53.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37465.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:14:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:14:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:14:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:55 smithi122 ceph-mon[93869]: pgmap v20246: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 759 KiB/s rd, 744 KiB/s wr, 11 op/s 2023-03-10T01:14:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:14:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:14:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:55 smithi181 ceph-mon[94189]: pgmap v20246: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 759 KiB/s rd, 744 KiB/s wr, 11 op/s 2023-03-10T01:14:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:14:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:14:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:55 smithi038 ceph-mon[92422]: pgmap v20246: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 759 KiB/s rd, 744 KiB/s wr, 11 op/s 2023-03-10T01:14:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37006.512373 secs 2023-03-10T01:14:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:14:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:14:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37006.512373 secs 2023-03-10T01:14:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:14:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:14:56.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37006.512373 secs 2023-03-10T01:14:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:14:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:14:57.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:57 smithi038 ceph-mon[92422]: pgmap v20247: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 889 KiB/s rd, 873 KiB/s wr, 13 op/s 2023-03-10T01:14:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:57 smithi122 ceph-mon[93869]: pgmap v20247: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 889 KiB/s rd, 873 KiB/s wr, 13 op/s 2023-03-10T01:14:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:57 smithi181 ceph-mon[94189]: pgmap v20247: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 889 KiB/s rd, 873 KiB/s wr, 13 op/s 2023-03-10T01:14:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:58 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 37008.974454 secs 2023-03-10T01:14:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37470.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:14:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:58 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 37008.974454 secs 2023-03-10T01:14:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:58 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 37008.974454 secs 2023-03-10T01:14:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:59 smithi038 ceph-mon[92422]: pgmap v20248: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 578 KiB/s rd, 564 KiB/s wr, 10 op/s 2023-03-10T01:14:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:14:59 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:14:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:59 smithi122 ceph-mon[93869]: pgmap v20248: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 578 KiB/s rd, 564 KiB/s wr, 10 op/s 2023-03-10T01:14:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:14:59 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:14:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:59 smithi181 ceph-mon[94189]: pgmap v20248: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 578 KiB/s rd, 564 KiB/s wr, 10 op/s 2023-03-10T01:14:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:14:59 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:15:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:15:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:15:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:15:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:15:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:15:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:15:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37011.512432 secs 2023-03-10T01:15:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:15:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:15:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:01 smithi122 ceph-mon[93869]: pgmap v20249: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1021 KiB/s wr, 13 op/s 2023-03-10T01:15:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37011.512432 secs 2023-03-10T01:15:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:15:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:15:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:01 smithi181 ceph-mon[94189]: pgmap v20249: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1021 KiB/s wr, 13 op/s 2023-03-10T01:15:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37011.512432 secs 2023-03-10T01:15:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:15:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:15:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:01 smithi038 ceph-mon[92422]: pgmap v20249: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1021 KiB/s wr, 13 op/s 2023-03-10T01:15:03.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:02 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 37013.974539 secs 2023-03-10T01:15:03.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:02 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 37013.974539 secs 2023-03-10T01:15:03.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:02 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 37013.974539 secs 2023-03-10T01:15:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37475.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:15:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:03 smithi122 ceph-mon[93869]: pgmap v20250: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 618 KiB/s wr, 10 op/s 2023-03-10T01:15:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:03 smithi181 ceph-mon[94189]: pgmap v20250: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 618 KiB/s wr, 10 op/s 2023-03-10T01:15:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:03 smithi038 ceph-mon[92422]: pgmap v20250: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 618 KiB/s wr, 10 op/s 2023-03-10T01:15:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:05 smithi122 ceph-mon[93869]: pgmap v20251: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 619 KiB/s wr, 11 op/s 2023-03-10T01:15:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:05 smithi181 ceph-mon[94189]: pgmap v20251: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 619 KiB/s wr, 11 op/s 2023-03-10T01:15:05.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:05 smithi038 ceph-mon[92422]: pgmap v20251: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 619 KiB/s wr, 11 op/s 2023-03-10T01:15:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37016.512493 secs 2023-03-10T01:15:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37016.512493 secs 2023-03-10T01:15:06.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37016.512493 secs 2023-03-10T01:15:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:07 smithi122 ceph-mon[93869]: pgmap v20252: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 791 KiB/s rd, 798 KiB/s wr, 15 op/s 2023-03-10T01:15:07.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:07 smithi181 ceph-mon[94189]: pgmap v20252: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 791 KiB/s rd, 798 KiB/s wr, 15 op/s 2023-03-10T01:15:07.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:07 smithi038 ceph-mon[92422]: pgmap v20252: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 791 KiB/s rd, 798 KiB/s wr, 15 op/s 2023-03-10T01:15:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:08 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 37018.974652 secs 2023-03-10T01:15:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:08 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 37018.974652 secs 2023-03-10T01:15:08.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:08 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 37018.974652 secs 2023-03-10T01:15:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37480.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:15:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:15:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:15:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:09 smithi122 ceph-mon[93869]: pgmap v20253: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 666 KiB/s rd, 679 KiB/s wr, 21 op/s 2023-03-10T01:15:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:15:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:15:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:09 smithi181 ceph-mon[94189]: pgmap v20253: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 666 KiB/s rd, 679 KiB/s wr, 21 op/s 2023-03-10T01:15:09.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:15:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:15:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:09 smithi038 ceph-mon[92422]: pgmap v20253: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 666 KiB/s rd, 679 KiB/s wr, 21 op/s 2023-03-10T01:15:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37021.512561 secs 2023-03-10T01:15:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:11 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:15:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:11 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:15:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:11 smithi122 ceph-mon[93869]: pgmap v20254: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 709 KiB/s wr, 24 op/s 2023-03-10T01:15:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37021.512561 secs 2023-03-10T01:15:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:11 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:15:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:11 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:15:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:11 smithi181 ceph-mon[94189]: pgmap v20254: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 709 KiB/s wr, 24 op/s 2023-03-10T01:15:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37021.512561 secs 2023-03-10T01:15:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:11 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:15:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:11 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:15:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:11 smithi038 ceph-mon[92422]: pgmap v20254: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 709 KiB/s wr, 24 op/s 2023-03-10T01:15:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37485.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:15:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:13 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 37023.974748 secs 2023-03-10T01:15:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:15:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:15:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:13 smithi122 ceph-mon[93869]: pgmap v20255: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 583 KiB/s rd, 252 KiB/s wr, 21 op/s 2023-03-10T01:15:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:13 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 37023.974748 secs 2023-03-10T01:15:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:15:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:15:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:13 smithi181 ceph-mon[94189]: pgmap v20255: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 583 KiB/s rd, 252 KiB/s wr, 21 op/s 2023-03-10T01:15:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:13 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 37023.974748 secs 2023-03-10T01:15:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:15:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:15:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:13 smithi038 ceph-mon[92422]: pgmap v20255: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 583 KiB/s rd, 252 KiB/s wr, 21 op/s 2023-03-10T01:15:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:15 smithi122 ceph-mon[93869]: pgmap v20256: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 383 KiB/s wr, 21 op/s 2023-03-10T01:15:15.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:15 smithi181 ceph-mon[94189]: pgmap v20256: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 383 KiB/s wr, 21 op/s 2023-03-10T01:15:15.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:15 smithi038 ceph-mon[92422]: pgmap v20256: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 383 KiB/s wr, 21 op/s 2023-03-10T01:15:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37026.512621 secs 2023-03-10T01:15:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:15:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:15:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37026.512621 secs 2023-03-10T01:15:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:15:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:15:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37026.512621 secs 2023-03-10T01:15:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:15:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:15:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:17 smithi122 ceph-mon[93869]: pgmap v20257: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 383 KiB/s wr, 22 op/s 2023-03-10T01:15:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:17 smithi181 ceph-mon[94189]: pgmap v20257: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 383 KiB/s wr, 22 op/s 2023-03-10T01:15:17.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:17 smithi038 ceph-mon[92422]: pgmap v20257: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 383 KiB/s wr, 22 op/s 2023-03-10T01:15:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:18 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 37028.974819 secs 2023-03-10T01:15:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:18 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 37028.974819 secs 2023-03-10T01:15:18.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:18 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 37028.974819 secs 2023-03-10T01:15:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37490.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:15:19.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:19 smithi122 ceph-mon[93869]: pgmap v20258: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 513 KiB/s rd, 173 KiB/s wr, 17 op/s 2023-03-10T01:15:19.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:19 smithi181 ceph-mon[94189]: pgmap v20258: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 513 KiB/s rd, 173 KiB/s wr, 17 op/s 2023-03-10T01:15:19.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:19 smithi038 ceph-mon[92422]: pgmap v20258: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 513 KiB/s rd, 173 KiB/s wr, 17 op/s 2023-03-10T01:15:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37031.512694 secs 2023-03-10T01:15:21.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37031.512694 secs 2023-03-10T01:15:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37031.512694 secs 2023-03-10T01:15:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:21 smithi122 ceph-mon[93869]: pgmap v20259: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 707 KiB/s rd, 362 KiB/s wr, 9 op/s 2023-03-10T01:15:22.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:21 smithi181 ceph-mon[94189]: pgmap v20259: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 707 KiB/s rd, 362 KiB/s wr, 9 op/s 2023-03-10T01:15:22.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:21 smithi038 ceph-mon[92422]: pgmap v20259: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 707 KiB/s rd, 362 KiB/s wr, 9 op/s 2023-03-10T01:15:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:22 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 37033.974906 secs 2023-03-10T01:15:23.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:22 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 37033.974906 secs 2023-03-10T01:15:23.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:22 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 37033.974906 secs 2023-03-10T01:15:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37495.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:15:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:23 smithi122 ceph-mon[93869]: pgmap v20260: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 332 KiB/s wr, 8 op/s 2023-03-10T01:15:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:23 smithi181 ceph-mon[94189]: pgmap v20260: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 332 KiB/s wr, 8 op/s 2023-03-10T01:15:24.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:23 smithi038 ceph-mon[92422]: pgmap v20260: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 332 KiB/s wr, 8 op/s 2023-03-10T01:15:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:25 smithi122 ceph-mon[93869]: pgmap v20261: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 414 KiB/s wr, 10 op/s 2023-03-10T01:15:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:25 smithi181 ceph-mon[94189]: pgmap v20261: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 414 KiB/s wr, 10 op/s 2023-03-10T01:15:25.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:25 smithi038 ceph-mon[92422]: pgmap v20261: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 414 KiB/s wr, 10 op/s 2023-03-10T01:15:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37036.512753 secs 2023-03-10T01:15:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37036.512753 secs 2023-03-10T01:15:26.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37036.512753 secs 2023-03-10T01:15:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:27 smithi122 ceph-mon[93869]: pgmap v20262: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 448 KiB/s wr, 12 op/s 2023-03-10T01:15:27.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:27 smithi181 ceph-mon[94189]: pgmap v20262: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 448 KiB/s wr, 12 op/s 2023-03-10T01:15:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:27 smithi038 ceph-mon[92422]: pgmap v20262: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 429 KiB/s rd, 448 KiB/s wr, 12 op/s 2023-03-10T01:15:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:28 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:15:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:28 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:15:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37038.974966 secs 2023-03-10T01:15:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:28 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:15:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:28 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:15:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37038.974966 secs 2023-03-10T01:15:28.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:28 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:15:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:28 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:15:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37038.974966 secs 2023-03-10T01:15:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37500.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:15:29.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:29 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:15:29.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:29 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:15:29.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:29 smithi122 ceph-mon[93869]: pgmap v20263: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 448 KiB/s wr, 10 op/s 2023-03-10T01:15:29.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:29 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:15:29.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:29 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:15:29.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:29 smithi181 ceph-mon[94189]: pgmap v20263: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 448 KiB/s wr, 10 op/s 2023-03-10T01:15:29.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:29 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:15:29.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:29 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:15:29.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:29 smithi038 ceph-mon[92422]: pgmap v20263: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 448 KiB/s wr, 10 op/s 2023-03-10T01:15:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:30 smithi122 ceph-mon[93869]: osdmap e1481: 12 total, 12 up, 12 in 2023-03-10T01:15:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:30 smithi181 ceph-mon[94189]: osdmap e1481: 12 total, 12 up, 12 in 2023-03-10T01:15:30.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:30 smithi038 ceph-mon[92422]: osdmap e1481: 12 total, 12 up, 12 in 2023-03-10T01:15:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37041.512838 secs 2023-03-10T01:15:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:31 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:15:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:31 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:15:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:31 smithi122 ceph-mon[93869]: pgmap v20265: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 506 KiB/s wr, 13 op/s 2023-03-10T01:15:31.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37041.512838 secs 2023-03-10T01:15:31.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:31 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:15:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:31 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:15:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:31 smithi181 ceph-mon[94189]: pgmap v20265: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 506 KiB/s wr, 13 op/s 2023-03-10T01:15:31.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37041.512838 secs 2023-03-10T01:15:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:31 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:15:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:31 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:15:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:31 smithi038 ceph-mon[92422]: pgmap v20265: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 506 KiB/s wr, 13 op/s 2023-03-10T01:15:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37043.975053 secs 2023-03-10T01:15:33.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37043.975053 secs 2023-03-10T01:15:33.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37043.975053 secs 2023-03-10T01:15:33.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37505.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:15:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:33 smithi122 ceph-mon[93869]: pgmap v20266: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 506 KiB/s wr, 12 op/s 2023-03-10T01:15:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:33 smithi181 ceph-mon[94189]: pgmap v20266: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 506 KiB/s wr, 12 op/s 2023-03-10T01:15:34.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:33 smithi038 ceph-mon[92422]: pgmap v20266: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 506 KiB/s wr, 12 op/s 2023-03-10T01:15:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:35 smithi122 ceph-mon[93869]: pgmap v20267: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 630 KiB/s rd, 642 KiB/s wr, 11 op/s 2023-03-10T01:15:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:35 smithi181 ceph-mon[94189]: pgmap v20267: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 630 KiB/s rd, 642 KiB/s wr, 11 op/s 2023-03-10T01:15:35.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:35 smithi038 ceph-mon[92422]: pgmap v20267: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 630 KiB/s rd, 642 KiB/s wr, 11 op/s 2023-03-10T01:15:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37046.512888 secs 2023-03-10T01:15:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:15:36.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:15:36.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:15:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37046.512888 secs 2023-03-10T01:15:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:15:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:15:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:15:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37046.512888 secs 2023-03-10T01:15:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:15:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:15:36.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:15:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:15:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:15:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:15:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:15:37.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:37 smithi122 ceph-mon[93869]: pgmap v20268: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 463 KiB/s wr, 9 op/s 2023-03-10T01:15:37.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:15:37.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:15:37.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:15:37.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:15:37.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:15:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:15:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:15:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:37 smithi181 ceph-mon[94189]: pgmap v20268: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 463 KiB/s wr, 9 op/s 2023-03-10T01:15:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:15:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:15:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:15:37.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:15:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:15:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:15:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:15:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:37 smithi038 ceph-mon[92422]: pgmap v20268: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 463 KiB/s wr, 9 op/s 2023-03-10T01:15:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:15:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:15:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:15:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37510.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:15:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37048.975156 secs 2023-03-10T01:15:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37048.975156 secs 2023-03-10T01:15:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37048.975156 secs 2023-03-10T01:15:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:39 smithi122 ceph-mon[93869]: pgmap v20269: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 463 KiB/s wr, 9 op/s 2023-03-10T01:15:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:39 smithi122 ceph-mon[93869]: osdmap e1482: 12 total, 12 up, 12 in 2023-03-10T01:15:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:39 smithi181 ceph-mon[94189]: pgmap v20269: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 463 KiB/s wr, 9 op/s 2023-03-10T01:15:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:39 smithi181 ceph-mon[94189]: osdmap e1482: 12 total, 12 up, 12 in 2023-03-10T01:15:39.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:39 smithi038 ceph-mon[92422]: pgmap v20269: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 465 KiB/s rd, 463 KiB/s wr, 9 op/s 2023-03-10T01:15:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:39 smithi038 ceph-mon[92422]: osdmap e1482: 12 total, 12 up, 12 in 2023-03-10T01:15:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37051.512946 secs 2023-03-10T01:15:41.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37051.512946 secs 2023-03-10T01:15:41.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37051.512946 secs 2023-03-10T01:15:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:41 smithi122 ceph-mon[93869]: pgmap v20271: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 256 KiB/s wr, 9 op/s 2023-03-10T01:15:42.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:41 smithi181 ceph-mon[94189]: pgmap v20271: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 256 KiB/s wr, 9 op/s 2023-03-10T01:15:42.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:41 smithi038 ceph-mon[92422]: pgmap v20271: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 256 KiB/s wr, 9 op/s 2023-03-10T01:15:43.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37053.975239 secs 2023-03-10T01:15:43.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37053.975239 secs 2023-03-10T01:15:43.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37053.975239 secs 2023-03-10T01:15:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37515.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:15:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:43 smithi122 ceph-mon[93869]: pgmap v20272: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 256 KiB/s wr, 9 op/s 2023-03-10T01:15:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:43 smithi181 ceph-mon[94189]: pgmap v20272: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 256 KiB/s wr, 9 op/s 2023-03-10T01:15:44.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:43 smithi038 ceph-mon[92422]: pgmap v20272: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 256 KiB/s wr, 9 op/s 2023-03-10T01:15:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:45 smithi122 ceph-mon[93869]: pgmap v20273: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 195 KiB/s wr, 8 op/s 2023-03-10T01:15:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:45 smithi181 ceph-mon[94189]: pgmap v20273: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 195 KiB/s wr, 8 op/s 2023-03-10T01:15:45.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:45 smithi038 ceph-mon[92422]: pgmap v20273: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 195 KiB/s wr, 8 op/s 2023-03-10T01:15:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37056.512984 secs 2023-03-10T01:15:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37056.512984 secs 2023-03-10T01:15:46.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37056.512984 secs 2023-03-10T01:15:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:47 smithi122 ceph-mon[93869]: pgmap v20274: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 364 KiB/s wr, 8 op/s 2023-03-10T01:15:47.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:47 smithi181 ceph-mon[94189]: pgmap v20274: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 364 KiB/s wr, 8 op/s 2023-03-10T01:15:47.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:47 smithi038 ceph-mon[92422]: pgmap v20274: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 364 KiB/s wr, 8 op/s 2023-03-10T01:15:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37058.975298 secs 2023-03-10T01:15:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37058.975298 secs 2023-03-10T01:15:48.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37058.975298 secs 2023-03-10T01:15:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37520.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:15:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:15:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:15:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:49 smithi122 ceph-mon[93869]: pgmap v20275: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 364 KiB/s wr, 8 op/s 2023-03-10T01:15:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:15:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:15:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:49 smithi181 ceph-mon[94189]: pgmap v20275: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 364 KiB/s wr, 8 op/s 2023-03-10T01:15:49.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:15:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:15:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:49 smithi038 ceph-mon[92422]: pgmap v20275: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 364 KiB/s wr, 8 op/s 2023-03-10T01:15:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37061.513044 secs 2023-03-10T01:15:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:15:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:15:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:51 smithi122 ceph-mon[93869]: pgmap v20276: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 639 KiB/s wr, 10 op/s 2023-03-10T01:15:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37061.513044 secs 2023-03-10T01:15:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:15:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:15:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:51 smithi181 ceph-mon[94189]: pgmap v20276: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 639 KiB/s wr, 10 op/s 2023-03-10T01:15:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37061.513044 secs 2023-03-10T01:15:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:15:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:15:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:51 smithi038 ceph-mon[92422]: pgmap v20276: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 639 KiB/s wr, 10 op/s 2023-03-10T01:15:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37063.975413 secs 2023-03-10T01:15:53.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37063.975413 secs 2023-03-10T01:15:53.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37063.975413 secs 2023-03-10T01:15:53.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37525.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:15:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:53 smithi122 ceph-mon[93869]: pgmap v20277: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 607 KiB/s wr, 9 op/s 2023-03-10T01:15:54.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:53 smithi181 ceph-mon[94189]: pgmap v20277: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 607 KiB/s wr, 9 op/s 2023-03-10T01:15:54.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:53 smithi038 ceph-mon[92422]: pgmap v20277: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 607 KiB/s wr, 9 op/s 2023-03-10T01:15:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:55 smithi122 ceph-mon[93869]: pgmap v20278: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 455 KiB/s rd, 638 KiB/s wr, 9 op/s 2023-03-10T01:15:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:55 smithi181 ceph-mon[94189]: pgmap v20278: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 455 KiB/s rd, 638 KiB/s wr, 9 op/s 2023-03-10T01:15:55.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:55 smithi038 ceph-mon[92422]: pgmap v20278: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 455 KiB/s rd, 638 KiB/s wr, 9 op/s 2023-03-10T01:15:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37066.513102 secs 2023-03-10T01:15:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:15:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:15:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37066.513102 secs 2023-03-10T01:15:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:15:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:15:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37066.513102 secs 2023-03-10T01:15:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:15:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:15:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:15:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:15:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:57 smithi122 ceph-mon[93869]: pgmap v20279: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 532 KiB/s rd, 527 KiB/s wr, 11 op/s 2023-03-10T01:15:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:15:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:15:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:57 smithi181 ceph-mon[94189]: pgmap v20279: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 532 KiB/s rd, 527 KiB/s wr, 11 op/s 2023-03-10T01:15:57.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:15:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:15:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:57 smithi038 ceph-mon[92422]: pgmap v20279: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 532 KiB/s rd, 527 KiB/s wr, 11 op/s 2023-03-10T01:15:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37068.975535 secs 2023-03-10T01:15:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37068.975535 secs 2023-03-10T01:15:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37068.975535 secs 2023-03-10T01:15:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37530.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:15:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:15:59 smithi122 ceph-mon[93869]: pgmap v20280: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 531 KiB/s rd, 371 KiB/s wr, 9 op/s 2023-03-10T01:15:59.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:15:59 smithi181 ceph-mon[94189]: pgmap v20280: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 531 KiB/s rd, 371 KiB/s wr, 9 op/s 2023-03-10T01:15:59.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:15:59 smithi038 ceph-mon[92422]: pgmap v20280: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 531 KiB/s rd, 371 KiB/s wr, 9 op/s 2023-03-10T01:16:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:16:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:16:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:16:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:16:00.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:16:00.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:16:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37071.513198 secs 2023-03-10T01:16:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:16:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:16:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:01 smithi122 ceph-mon[93869]: pgmap v20281: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 791 KiB/s rd, 630 KiB/s wr, 14 op/s 2023-03-10T01:16:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37071.513198 secs 2023-03-10T01:16:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:16:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:16:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:01 smithi181 ceph-mon[94189]: pgmap v20281: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 791 KiB/s rd, 630 KiB/s wr, 14 op/s 2023-03-10T01:16:01.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37071.513198 secs 2023-03-10T01:16:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:16:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:16:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:01 smithi038 ceph-mon[92422]: pgmap v20281: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 791 KiB/s rd, 630 KiB/s wr, 14 op/s 2023-03-10T01:16:03.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37073.975683 secs 2023-03-10T01:16:03.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37073.975683 secs 2023-03-10T01:16:03.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37073.975683 secs 2023-03-10T01:16:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37535.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:16:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:03 smithi122 ceph-mon[93869]: pgmap v20282: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 855 KiB/s rd, 666 KiB/s wr, 13 op/s 2023-03-10T01:16:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:03 smithi181 ceph-mon[94189]: pgmap v20282: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 855 KiB/s rd, 666 KiB/s wr, 13 op/s 2023-03-10T01:16:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:03 smithi038 ceph-mon[92422]: pgmap v20282: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 855 KiB/s rd, 666 KiB/s wr, 13 op/s 2023-03-10T01:16:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:05 smithi122 ceph-mon[93869]: pgmap v20283: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 940 KiB/s rd, 755 KiB/s wr, 24 op/s 2023-03-10T01:16:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:05 smithi181 ceph-mon[94189]: pgmap v20283: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 940 KiB/s rd, 755 KiB/s wr, 24 op/s 2023-03-10T01:16:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:05 smithi038 ceph-mon[92422]: pgmap v20283: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 940 KiB/s rd, 755 KiB/s wr, 24 op/s 2023-03-10T01:16:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37076.513256 secs 2023-03-10T01:16:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37076.513256 secs 2023-03-10T01:16:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37076.513256 secs 2023-03-10T01:16:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:07 smithi122 ceph-mon[93869]: pgmap v20284: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 978 KiB/s rd, 792 KiB/s wr, 27 op/s 2023-03-10T01:16:07.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:07 smithi181 ceph-mon[94189]: pgmap v20284: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 978 KiB/s rd, 792 KiB/s wr, 27 op/s 2023-03-10T01:16:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:07 smithi038 ceph-mon[92422]: pgmap v20284: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 978 KiB/s rd, 792 KiB/s wr, 27 op/s 2023-03-10T01:16:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:16:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:16:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37078.975741 secs 2023-03-10T01:16:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:16:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:16:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37078.975741 secs 2023-03-10T01:16:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:16:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:16:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37078.975741 secs 2023-03-10T01:16:08.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37540.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:16:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:16:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:16:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:09 smithi122 ceph-mon[93869]: pgmap v20285: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 756 KiB/s rd, 757 KiB/s wr, 24 op/s 2023-03-10T01:16:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:16:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:16:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:09 smithi181 ceph-mon[94189]: pgmap v20285: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 756 KiB/s rd, 757 KiB/s wr, 24 op/s 2023-03-10T01:16:09.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:16:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:16:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:09 smithi038 ceph-mon[92422]: pgmap v20285: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 756 KiB/s rd, 757 KiB/s wr, 24 op/s 2023-03-10T01:16:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37081.513312 secs 2023-03-10T01:16:11.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37081.513312 secs 2023-03-10T01:16:11.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37081.513312 secs 2023-03-10T01:16:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:11 smithi122 ceph-mon[93869]: pgmap v20286: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 26 op/s 2023-03-10T01:16:12.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:11 smithi181 ceph-mon[94189]: pgmap v20286: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 26 op/s 2023-03-10T01:16:12.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:11 smithi038 ceph-mon[92422]: pgmap v20286: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 26 op/s 2023-03-10T01:16:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37083.975802 secs 2023-03-10T01:16:13.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37083.975802 secs 2023-03-10T01:16:13.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37083.975802 secs 2023-03-10T01:16:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37545.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:16:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:13 smithi122 ceph-mon[93869]: pgmap v20287: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 979 KiB/s rd, 980 KiB/s wr, 22 op/s 2023-03-10T01:16:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:13 smithi181 ceph-mon[94189]: pgmap v20287: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 979 KiB/s rd, 980 KiB/s wr, 22 op/s 2023-03-10T01:16:14.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:13 smithi038 ceph-mon[92422]: pgmap v20287: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 979 KiB/s rd, 980 KiB/s wr, 22 op/s 2023-03-10T01:16:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:15 smithi122 ceph-mon[93869]: pgmap v20288: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 637 KiB/s rd, 638 KiB/s wr, 20 op/s 2023-03-10T01:16:15.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:15 smithi181 ceph-mon[94189]: pgmap v20288: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 637 KiB/s rd, 638 KiB/s wr, 20 op/s 2023-03-10T01:16:15.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:15 smithi038 ceph-mon[92422]: pgmap v20288: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 637 KiB/s rd, 638 KiB/s wr, 20 op/s 2023-03-10T01:16:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37086.513395 secs 2023-03-10T01:16:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:16:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:16:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37086.513395 secs 2023-03-10T01:16:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:16:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:16:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37086.513395 secs 2023-03-10T01:16:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:16:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:16:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:16:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:16:17.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:17 smithi122 ceph-mon[93869]: pgmap v20289: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 553 KiB/s rd, 549 KiB/s wr, 11 op/s 2023-03-10T01:16:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:16:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:16:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:17 smithi181 ceph-mon[94189]: pgmap v20289: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 553 KiB/s rd, 549 KiB/s wr, 11 op/s 2023-03-10T01:16:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:16:17.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:16:17.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:17 smithi038 ceph-mon[92422]: pgmap v20289: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 553 KiB/s rd, 549 KiB/s wr, 11 op/s 2023-03-10T01:16:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37088.975906 secs 2023-03-10T01:16:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37088.975906 secs 2023-03-10T01:16:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37088.975906 secs 2023-03-10T01:16:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37550.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:16:19.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:19 smithi122 ceph-mon[93869]: pgmap v20290: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 485 KiB/s rd, 493 KiB/s wr, 8 op/s 2023-03-10T01:16:19.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:19 smithi181 ceph-mon[94189]: pgmap v20290: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 485 KiB/s rd, 493 KiB/s wr, 8 op/s 2023-03-10T01:16:19.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:19 smithi038 ceph-mon[92422]: pgmap v20290: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 485 KiB/s rd, 493 KiB/s wr, 8 op/s 2023-03-10T01:16:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:16:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:16:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:16:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:16:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:16:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:16:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37091.513451 secs 2023-03-10T01:16:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:16:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:16:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:21 smithi038 ceph-mon[92422]: pgmap v20291: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 915 KiB/s rd, 933 KiB/s wr, 13 op/s 2023-03-10T01:16:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37091.513451 secs 2023-03-10T01:16:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:16:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:16:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:21 smithi122 ceph-mon[93869]: pgmap v20291: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 915 KiB/s rd, 933 KiB/s wr, 13 op/s 2023-03-10T01:16:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37091.513451 secs 2023-03-10T01:16:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:16:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:16:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:21 smithi181 ceph-mon[94189]: pgmap v20291: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 915 KiB/s rd, 933 KiB/s wr, 13 op/s 2023-03-10T01:16:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37093.975991 secs 2023-03-10T01:16:23.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37093.975991 secs 2023-03-10T01:16:23.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37093.975991 secs 2023-03-10T01:16:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37555.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:16:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:23 smithi122 ceph-mon[93869]: pgmap v20292: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 454 KiB/s rd, 474 KiB/s wr, 11 op/s 2023-03-10T01:16:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:23 smithi181 ceph-mon[94189]: pgmap v20292: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 454 KiB/s rd, 474 KiB/s wr, 11 op/s 2023-03-10T01:16:24.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:23 smithi038 ceph-mon[92422]: pgmap v20292: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 454 KiB/s rd, 474 KiB/s wr, 11 op/s 2023-03-10T01:16:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:25 smithi122 ceph-mon[93869]: pgmap v20293: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 454 KiB/s rd, 474 KiB/s wr, 11 op/s 2023-03-10T01:16:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:25 smithi122 ceph-mon[93869]: osdmap e1483: 12 total, 12 up, 12 in 2023-03-10T01:16:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37096.513510 secs 2023-03-10T01:16:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:25 smithi181 ceph-mon[94189]: pgmap v20293: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 454 KiB/s rd, 474 KiB/s wr, 11 op/s 2023-03-10T01:16:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:25 smithi181 ceph-mon[94189]: osdmap e1483: 12 total, 12 up, 12 in 2023-03-10T01:16:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37096.513510 secs 2023-03-10T01:16:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:25 smithi038 ceph-mon[92422]: pgmap v20293: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 454 KiB/s rd, 474 KiB/s wr, 11 op/s 2023-03-10T01:16:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:25 smithi038 ceph-mon[92422]: osdmap e1483: 12 total, 12 up, 12 in 2023-03-10T01:16:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37096.513510 secs 2023-03-10T01:16:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:27 smithi122 ceph-mon[93869]: pgmap v20295: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 723 KiB/s rd, 747 KiB/s wr, 13 op/s 2023-03-10T01:16:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37098.976073 secs 2023-03-10T01:16:28.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:27 smithi181 ceph-mon[94189]: pgmap v20295: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 723 KiB/s rd, 747 KiB/s wr, 13 op/s 2023-03-10T01:16:28.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37098.976073 secs 2023-03-10T01:16:28.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:27 smithi038 ceph-mon[92422]: pgmap v20295: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 723 KiB/s rd, 747 KiB/s wr, 13 op/s 2023-03-10T01:16:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37098.976073 secs 2023-03-10T01:16:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37560.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:16:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:28 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:16:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:28 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:16:29.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:28 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:16:29.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:28 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:16:29.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:28 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:16:29.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:28 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:16:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:29 smithi122 ceph-mon[93869]: pgmap v20296: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 722 KiB/s rd, 735 KiB/s wr, 12 op/s 2023-03-10T01:16:30.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:29 smithi181 ceph-mon[94189]: pgmap v20296: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 722 KiB/s rd, 735 KiB/s wr, 12 op/s 2023-03-10T01:16:30.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:29 smithi038 ceph-mon[92422]: pgmap v20296: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 722 KiB/s rd, 735 KiB/s wr, 12 op/s 2023-03-10T01:16:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37101.513566 secs 2023-03-10T01:16:31.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37101.513566 secs 2023-03-10T01:16:31.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37101.513566 secs 2023-03-10T01:16:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:31 smithi122 ceph-mon[93869]: pgmap v20297: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 243 KiB/s wr, 8 op/s 2023-03-10T01:16:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:31 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:16:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:31 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:16:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:31 smithi181 ceph-mon[94189]: pgmap v20297: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 243 KiB/s wr, 8 op/s 2023-03-10T01:16:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:31 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:16:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:31 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:16:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:31 smithi038 ceph-mon[92422]: pgmap v20297: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 243 KiB/s wr, 8 op/s 2023-03-10T01:16:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:31 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:16:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:31 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:16:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37103.976160 secs 2023-03-10T01:16:33.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37103.976160 secs 2023-03-10T01:16:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37103.976160 secs 2023-03-10T01:16:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37565.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:16:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:33 smithi122 ceph-mon[93869]: pgmap v20298: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 216 KiB/s wr, 8 op/s 2023-03-10T01:16:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:33 smithi181 ceph-mon[94189]: pgmap v20298: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 216 KiB/s wr, 8 op/s 2023-03-10T01:16:34.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:33 smithi038 ceph-mon[92422]: pgmap v20298: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 216 KiB/s wr, 8 op/s 2023-03-10T01:16:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:35 smithi122 ceph-mon[93869]: osdmap e1484: 12 total, 12 up, 12 in 2023-03-10T01:16:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:35 smithi122 ceph-mon[93869]: pgmap v20300: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 573 KiB/s rd, 571 KiB/s wr, 9 op/s 2023-03-10T01:16:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:35 smithi181 ceph-mon[94189]: osdmap e1484: 12 total, 12 up, 12 in 2023-03-10T01:16:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:35 smithi181 ceph-mon[94189]: pgmap v20300: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 573 KiB/s rd, 571 KiB/s wr, 9 op/s 2023-03-10T01:16:35.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:35 smithi038 ceph-mon[92422]: osdmap e1484: 12 total, 12 up, 12 in 2023-03-10T01:16:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:35 smithi038 ceph-mon[92422]: pgmap v20300: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 573 KiB/s rd, 571 KiB/s wr, 9 op/s 2023-03-10T01:16:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37106.513626 secs 2023-03-10T01:16:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:16:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:16:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37106.513626 secs 2023-03-10T01:16:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:16:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:16:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37106.513626 secs 2023-03-10T01:16:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:16:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:16:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:16:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:16:37.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:37 smithi122 ceph-mon[93869]: pgmap v20301: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 722 KiB/s rd, 785 KiB/s wr, 9 op/s 2023-03-10T01:16:37.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:16:37.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:16:37.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:16:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:37 smithi181 ceph-mon[94189]: pgmap v20301: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 722 KiB/s rd, 785 KiB/s wr, 9 op/s 2023-03-10T01:16:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:16:37.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:16:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:16:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:37 smithi038 ceph-mon[92422]: pgmap v20301: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 722 KiB/s rd, 785 KiB/s wr, 9 op/s 2023-03-10T01:16:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:16:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37108.976224 secs 2023-03-10T01:16:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:38 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:16:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:38 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:16:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:38 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:16:38.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:38 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:16:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37108.976224 secs 2023-03-10T01:16:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:38 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:16:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:38 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:16:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:38 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:16:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:38 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:16:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37108.976224 secs 2023-03-10T01:16:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:38 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:16:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:38 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:16:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:38 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:16:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:38 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:16:38.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37570.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:16:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:16:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:16:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:16:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:39 smithi122 ceph-mon[93869]: pgmap v20302: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 722 KiB/s rd, 785 KiB/s wr, 9 op/s 2023-03-10T01:16:39.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:16:39.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:16:39.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:16:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:39 smithi181 ceph-mon[94189]: pgmap v20302: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 722 KiB/s rd, 785 KiB/s wr, 9 op/s 2023-03-10T01:16:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:16:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:16:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:16:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:39 smithi038 ceph-mon[92422]: pgmap v20302: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 722 KiB/s rd, 785 KiB/s wr, 9 op/s 2023-03-10T01:16:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:16:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:16:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:16:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:16:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:16:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:16:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37111.513683 secs 2023-03-10T01:16:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:16:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:16:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:41 smithi122 ceph-mon[93869]: pgmap v20303: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 986 KiB/s rd, 984 KiB/s wr, 10 op/s 2023-03-10T01:16:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37111.513683 secs 2023-03-10T01:16:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:16:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:16:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:41 smithi181 ceph-mon[94189]: pgmap v20303: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 986 KiB/s rd, 984 KiB/s wr, 10 op/s 2023-03-10T01:16:41.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37111.513683 secs 2023-03-10T01:16:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:16:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:16:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:41 smithi038 ceph-mon[92422]: pgmap v20303: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 986 KiB/s rd, 984 KiB/s wr, 10 op/s 2023-03-10T01:16:43.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37113.976295 secs 2023-03-10T01:16:43.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37113.976295 secs 2023-03-10T01:16:43.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37113.976295 secs 2023-03-10T01:16:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37575.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:16:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:43 smithi122 ceph-mon[93869]: pgmap v20304: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 986 KiB/s rd, 984 KiB/s wr, 10 op/s 2023-03-10T01:16:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:43 smithi181 ceph-mon[94189]: pgmap v20304: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 986 KiB/s rd, 984 KiB/s wr, 10 op/s 2023-03-10T01:16:44.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:43 smithi038 ceph-mon[92422]: pgmap v20304: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 986 KiB/s rd, 984 KiB/s wr, 10 op/s 2023-03-10T01:16:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:45 smithi122 ceph-mon[93869]: pgmap v20305: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 608 KiB/s rd, 607 KiB/s wr, 9 op/s 2023-03-10T01:16:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:45 smithi181 ceph-mon[94189]: pgmap v20305: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 608 KiB/s rd, 607 KiB/s wr, 9 op/s 2023-03-10T01:16:45.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:45 smithi038 ceph-mon[92422]: pgmap v20305: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 608 KiB/s rd, 607 KiB/s wr, 9 op/s 2023-03-10T01:16:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37116.513744 secs 2023-03-10T01:16:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37116.513744 secs 2023-03-10T01:16:46.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37116.513744 secs 2023-03-10T01:16:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:47 smithi122 ceph-mon[93869]: pgmap v20306: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 824 KiB/s rd, 822 KiB/s wr, 12 op/s 2023-03-10T01:16:47.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:47 smithi181 ceph-mon[94189]: pgmap v20306: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 824 KiB/s rd, 822 KiB/s wr, 12 op/s 2023-03-10T01:16:47.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:47 smithi038 ceph-mon[92422]: pgmap v20306: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 824 KiB/s rd, 822 KiB/s wr, 12 op/s 2023-03-10T01:16:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:16:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:16:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:16:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:16:48.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:16:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:16:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37580.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:16:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:16:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:16:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:49 smithi122 ceph-mon[93869]: pgmap v20307: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 491 KiB/s wr, 8 op/s 2023-03-10T01:16:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:16:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:16:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:49 smithi181 ceph-mon[94189]: pgmap v20307: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 491 KiB/s wr, 8 op/s 2023-03-10T01:16:49.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:16:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:16:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:49 smithi038 ceph-mon[92422]: pgmap v20307: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 491 KiB/s wr, 8 op/s 2023-03-10T01:16:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:50 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:16:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:50 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:16:50.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:50 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:16:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37121.513853 secs 2023-03-10T01:16:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:51 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:16:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:16:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:16:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:51 smithi122 ceph-mon[93869]: pgmap v20308: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 694 KiB/s rd, 637 KiB/s wr, 12 op/s 2023-03-10T01:16:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37121.513853 secs 2023-03-10T01:16:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:51 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:16:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:16:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:16:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:51 smithi181 ceph-mon[94189]: pgmap v20308: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 694 KiB/s rd, 637 KiB/s wr, 12 op/s 2023-03-10T01:16:51.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37121.513853 secs 2023-03-10T01:16:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:51 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:16:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:16:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:16:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:51 smithi038 ceph-mon[92422]: pgmap v20308: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 694 KiB/s rd, 637 KiB/s wr, 12 op/s 2023-03-10T01:16:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37585.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:16:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:53 smithi122 ceph-mon[93869]: pgmap v20309: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 442 KiB/s rd, 440 KiB/s wr, 8 op/s 2023-03-10T01:16:54.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:53 smithi181 ceph-mon[94189]: pgmap v20309: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 442 KiB/s rd, 440 KiB/s wr, 8 op/s 2023-03-10T01:16:54.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:53 smithi038 ceph-mon[92422]: pgmap v20309: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 442 KiB/s rd, 440 KiB/s wr, 8 op/s 2023-03-10T01:16:55.730 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:55 smithi122 ceph-mon[93869]: pgmap v20310: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 541 KiB/s rd, 539 KiB/s wr, 9 op/s 2023-03-10T01:16:55.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:55 smithi181 ceph-mon[94189]: pgmap v20310: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 541 KiB/s rd, 539 KiB/s wr, 9 op/s 2023-03-10T01:16:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:55 smithi038 ceph-mon[92422]: pgmap v20310: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 541 KiB/s rd, 539 KiB/s wr, 9 op/s 2023-03-10T01:16:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37126.513913 secs 2023-03-10T01:16:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:16:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:16:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37126.513913 secs 2023-03-10T01:16:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:16:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:16:56.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37126.513913 secs 2023-03-10T01:16:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:16:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:16:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:16:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:16:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:57 smithi122 ceph-mon[93869]: pgmap v20311: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 570 KiB/s wr, 12 op/s 2023-03-10T01:16:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:16:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:16:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:57 smithi181 ceph-mon[94189]: pgmap v20311: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 570 KiB/s wr, 12 op/s 2023-03-10T01:16:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:16:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:16:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:57 smithi038 ceph-mon[92422]: pgmap v20311: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 570 KiB/s wr, 12 op/s 2023-03-10T01:16:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37590.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:16:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:16:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:16:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:16:59 smithi122 ceph-mon[93869]: pgmap v20312: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 511 KiB/s rd, 518 KiB/s wr, 10 op/s 2023-03-10T01:16:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:16:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:16:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:16:59 smithi181 ceph-mon[94189]: pgmap v20312: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 511 KiB/s rd, 518 KiB/s wr, 10 op/s 2023-03-10T01:16:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:16:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:16:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:16:59 smithi038 ceph-mon[92422]: pgmap v20312: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 511 KiB/s rd, 518 KiB/s wr, 10 op/s 2023-03-10T01:17:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37131.513980 secs 2023-03-10T01:17:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:17:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:17:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:01 smithi122 ceph-mon[93869]: pgmap v20313: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 899 KiB/s rd, 906 KiB/s wr, 15 op/s 2023-03-10T01:17:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37131.513980 secs 2023-03-10T01:17:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:17:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:17:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:01 smithi181 ceph-mon[94189]: pgmap v20313: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 899 KiB/s rd, 906 KiB/s wr, 15 op/s 2023-03-10T01:17:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37131.513980 secs 2023-03-10T01:17:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:17:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:17:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:01 smithi038 ceph-mon[92422]: pgmap v20313: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 899 KiB/s rd, 906 KiB/s wr, 15 op/s 2023-03-10T01:17:03.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37595.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:17:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:03 smithi122 ceph-mon[93869]: pgmap v20314: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 753 KiB/s rd, 761 KiB/s wr, 11 op/s 2023-03-10T01:17:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:03 smithi181 ceph-mon[94189]: pgmap v20314: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 753 KiB/s rd, 761 KiB/s wr, 11 op/s 2023-03-10T01:17:04.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:03 smithi038 ceph-mon[92422]: pgmap v20314: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 753 KiB/s rd, 761 KiB/s wr, 11 op/s 2023-03-10T01:17:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:05 smithi122 ceph-mon[93869]: pgmap v20315: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 771 KiB/s rd, 771 KiB/s wr, 23 op/s 2023-03-10T01:17:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:05 smithi181 ceph-mon[94189]: pgmap v20315: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 771 KiB/s rd, 771 KiB/s wr, 23 op/s 2023-03-10T01:17:05.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:05 smithi038 ceph-mon[92422]: pgmap v20315: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 771 KiB/s rd, 771 KiB/s wr, 23 op/s 2023-03-10T01:17:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37136.514038 secs 2023-03-10T01:17:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37136.514038 secs 2023-03-10T01:17:06.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37136.514038 secs 2023-03-10T01:17:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:07 smithi122 ceph-mon[93869]: pgmap v20316: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 956 KiB/s rd, 955 KiB/s wr, 27 op/s 2023-03-10T01:17:07.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:07 smithi181 ceph-mon[94189]: pgmap v20316: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 956 KiB/s rd, 955 KiB/s wr, 27 op/s 2023-03-10T01:17:07.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:07 smithi038 ceph-mon[92422]: pgmap v20316: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 956 KiB/s rd, 955 KiB/s wr, 27 op/s 2023-03-10T01:17:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:17:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:17:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:17:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:17:08.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:17:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:17:08.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37600.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:17:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:17:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:17:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:09 smithi122 ceph-mon[93869]: pgmap v20317: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 922 KiB/s rd, 924 KiB/s wr, 24 op/s 2023-03-10T01:17:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:17:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:17:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:09 smithi181 ceph-mon[94189]: pgmap v20317: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 922 KiB/s rd, 924 KiB/s wr, 24 op/s 2023-03-10T01:17:09.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:17:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:17:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:09 smithi038 ceph-mon[92422]: pgmap v20317: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 922 KiB/s rd, 924 KiB/s wr, 24 op/s 2023-03-10T01:17:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37141.514112 secs 2023-03-10T01:17:11.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37141.514112 secs 2023-03-10T01:17:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37141.514112 secs 2023-03-10T01:17:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:11 smithi122 ceph-mon[93869]: pgmap v20318: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 749 KiB/s rd, 738 KiB/s wr, 26 op/s 2023-03-10T01:17:12.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:11 smithi181 ceph-mon[94189]: pgmap v20318: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 749 KiB/s rd, 738 KiB/s wr, 26 op/s 2023-03-10T01:17:12.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:11 smithi038 ceph-mon[92422]: pgmap v20318: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 749 KiB/s rd, 738 KiB/s wr, 26 op/s 2023-03-10T01:17:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37605.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:17:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:13 smithi122 ceph-mon[93869]: pgmap v20319: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 350 KiB/s wr, 20 op/s 2023-03-10T01:17:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:13 smithi181 ceph-mon[94189]: pgmap v20319: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 350 KiB/s wr, 20 op/s 2023-03-10T01:17:14.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:13 smithi038 ceph-mon[92422]: pgmap v20319: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 360 KiB/s rd, 350 KiB/s wr, 20 op/s 2023-03-10T01:17:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:15 smithi122 ceph-mon[93869]: pgmap v20320: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 468 KiB/s wr, 21 op/s 2023-03-10T01:17:15.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:15 smithi181 ceph-mon[94189]: pgmap v20320: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 468 KiB/s wr, 21 op/s 2023-03-10T01:17:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:15 smithi038 ceph-mon[92422]: pgmap v20320: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 468 KiB/s wr, 21 op/s 2023-03-10T01:17:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37146.514170 secs 2023-03-10T01:17:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37146.514170 secs 2023-03-10T01:17:16.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37146.514170 secs 2023-03-10T01:17:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:17 smithi122 ceph-mon[93869]: pgmap v20321: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 457 KiB/s wr, 12 op/s 2023-03-10T01:17:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:17 smithi181 ceph-mon[94189]: pgmap v20321: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 457 KiB/s wr, 12 op/s 2023-03-10T01:17:17.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:17 smithi038 ceph-mon[92422]: pgmap v20321: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 457 KiB/s wr, 12 op/s 2023-03-10T01:17:18.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37610.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:17:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:19 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:17:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:19 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:17:19.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:19 smithi122 ceph-mon[93869]: pgmap v20322: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 381 KiB/s wr, 9 op/s 2023-03-10T01:17:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:19 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:17:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:19 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:17:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:19 smithi181 ceph-mon[94189]: pgmap v20322: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 381 KiB/s wr, 9 op/s 2023-03-10T01:17:19.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:19 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:17:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:19 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:17:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:19 smithi038 ceph-mon[92422]: pgmap v20322: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 381 KiB/s wr, 9 op/s 2023-03-10T01:17:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37151.514250 secs 2023-03-10T01:17:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:17:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:17:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:21 smithi122 ceph-mon[93869]: pgmap v20323: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 313 KiB/s rd, 522 KiB/s wr, 13 op/s 2023-03-10T01:17:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37151.514250 secs 2023-03-10T01:17:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:17:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:17:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:21 smithi181 ceph-mon[94189]: pgmap v20323: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 313 KiB/s rd, 522 KiB/s wr, 13 op/s 2023-03-10T01:17:21.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37151.514250 secs 2023-03-10T01:17:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:17:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:17:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:21 smithi038 ceph-mon[92422]: pgmap v20323: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 313 KiB/s rd, 522 KiB/s wr, 13 op/s 2023-03-10T01:17:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:22 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 37153.976900 secs 2023-03-10T01:17:23.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:22 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 37153.976900 secs 2023-03-10T01:17:23.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:22 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 37153.976900 secs 2023-03-10T01:17:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37615.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:17:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:23 smithi122 ceph-mon[93869]: pgmap v20324: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 465 KiB/s wr, 9 op/s 2023-03-10T01:17:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:23 smithi181 ceph-mon[94189]: pgmap v20324: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 465 KiB/s wr, 9 op/s 2023-03-10T01:17:24.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:23 smithi038 ceph-mon[92422]: pgmap v20324: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 465 KiB/s wr, 9 op/s 2023-03-10T01:17:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:25 smithi122 ceph-mon[93869]: pgmap v20325: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 474 KiB/s rd, 494 KiB/s wr, 11 op/s 2023-03-10T01:17:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:25 smithi122 ceph-mon[93869]: osdmap e1485: 12 total, 12 up, 12 in 2023-03-10T01:17:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:25 smithi181 ceph-mon[94189]: pgmap v20325: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 474 KiB/s rd, 494 KiB/s wr, 11 op/s 2023-03-10T01:17:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:25 smithi181 ceph-mon[94189]: osdmap e1485: 12 total, 12 up, 12 in 2023-03-10T01:17:26.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:25 smithi038 ceph-mon[92422]: pgmap v20325: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 474 KiB/s rd, 494 KiB/s wr, 11 op/s 2023-03-10T01:17:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:25 smithi038 ceph-mon[92422]: osdmap e1485: 12 total, 12 up, 12 in 2023-03-10T01:17:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37156.514312 secs 2023-03-10T01:17:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:26 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:17:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:26 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:17:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:26 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:17:27.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37156.514312 secs 2023-03-10T01:17:27.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:26 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:17:27.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:26 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:17:27.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:26 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:17:27.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37156.514312 secs 2023-03-10T01:17:27.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:26 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:17:27.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:26 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:17:27.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:26 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:17:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:27 smithi122 ceph-mon[93869]: pgmap v20327: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 606 KiB/s rd, 630 KiB/s wr, 13 op/s 2023-03-10T01:17:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:27 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 37158.976961 secs 2023-03-10T01:17:28.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:27 smithi181 ceph-mon[94189]: pgmap v20327: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 606 KiB/s rd, 630 KiB/s wr, 13 op/s 2023-03-10T01:17:28.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:27 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 37158.976961 secs 2023-03-10T01:17:28.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:27 smithi038 ceph-mon[92422]: pgmap v20327: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 606 KiB/s rd, 630 KiB/s wr, 13 op/s 2023-03-10T01:17:28.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:27 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 37158.976961 secs 2023-03-10T01:17:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37620.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:17:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:29 smithi122 ceph-mon[93869]: pgmap v20328: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 382 KiB/s wr, 10 op/s 2023-03-10T01:17:30.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:29 smithi181 ceph-mon[94189]: pgmap v20328: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 382 KiB/s wr, 10 op/s 2023-03-10T01:17:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:29 smithi038 ceph-mon[92422]: pgmap v20328: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 600 KiB/s rd, 382 KiB/s wr, 10 op/s 2023-03-10T01:17:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37161.514367 secs 2023-03-10T01:17:31.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37161.514367 secs 2023-03-10T01:17:31.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37161.514367 secs 2023-03-10T01:17:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:31 smithi122 ceph-mon[93869]: pgmap v20329: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 709 KiB/s rd, 365 KiB/s wr, 9 op/s 2023-03-10T01:17:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:31 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:17:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:31 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:17:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:31 smithi181 ceph-mon[94189]: pgmap v20329: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 709 KiB/s rd, 365 KiB/s wr, 9 op/s 2023-03-10T01:17:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:31 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:17:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:31 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:17:32.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:31 smithi038 ceph-mon[92422]: pgmap v20329: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 709 KiB/s rd, 365 KiB/s wr, 9 op/s 2023-03-10T01:17:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:31 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:17:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:31 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:17:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:32 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:17:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:32 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:17:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:32 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 37163.977048 secs 2023-03-10T01:17:33.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:32 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:17:33.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:32 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:17:33.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:32 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 37163.977048 secs 2023-03-10T01:17:33.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:32 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:17:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:32 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:17:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:32 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 37163.977048 secs 2023-03-10T01:17:33.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37625.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:17:34.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:33 smithi122 ceph-mon[93869]: pgmap v20330: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 709 KiB/s rd, 365 KiB/s wr, 9 op/s 2023-03-10T01:17:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:33 smithi181 ceph-mon[94189]: pgmap v20330: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 709 KiB/s rd, 365 KiB/s wr, 9 op/s 2023-03-10T01:17:34.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:33 smithi038 ceph-mon[92422]: pgmap v20330: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 709 KiB/s rd, 365 KiB/s wr, 9 op/s 2023-03-10T01:17:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:35 smithi122 ceph-mon[93869]: osdmap e1486: 12 total, 12 up, 12 in 2023-03-10T01:17:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:35 smithi122 ceph-mon[93869]: pgmap v20332: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 704 KiB/s rd, 776 KiB/s wr, 8 op/s 2023-03-10T01:17:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:35 smithi181 ceph-mon[94189]: osdmap e1486: 12 total, 12 up, 12 in 2023-03-10T01:17:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:35 smithi181 ceph-mon[94189]: pgmap v20332: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 704 KiB/s rd, 776 KiB/s wr, 8 op/s 2023-03-10T01:17:35.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:35 smithi038 ceph-mon[92422]: osdmap e1486: 12 total, 12 up, 12 in 2023-03-10T01:17:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:35 smithi038 ceph-mon[92422]: pgmap v20332: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 704 KiB/s rd, 776 KiB/s wr, 8 op/s 2023-03-10T01:17:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37166.514473 secs 2023-03-10T01:17:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:17:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:17:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37166.514473 secs 2023-03-10T01:17:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:17:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:17:36.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37166.514473 secs 2023-03-10T01:17:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:17:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:17:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:17:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:17:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:37 smithi122 ceph-mon[93869]: pgmap v20333: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 791 KiB/s rd, 635 KiB/s wr, 8 op/s 2023-03-10T01:17:37.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:17:37.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:17:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:37 smithi181 ceph-mon[94189]: pgmap v20333: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 791 KiB/s rd, 635 KiB/s wr, 8 op/s 2023-03-10T01:17:37.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:17:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:17:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:37 smithi038 ceph-mon[92422]: pgmap v20333: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 791 KiB/s rd, 635 KiB/s wr, 8 op/s 2023-03-10T01:17:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37168.977111 secs 2023-03-10T01:17:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37168.977111 secs 2023-03-10T01:17:38.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37168.977111 secs 2023-03-10T01:17:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37630.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:17:39.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:39 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:17:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:39 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:17:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:17:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:39 smithi038 ceph-mon[92422]: pgmap v20334: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 816 KiB/s wr, 10 op/s 2023-03-10T01:17:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:17:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:17:39.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:17:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:39 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:17:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:39 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:17:39.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:17:39.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:39 smithi122 ceph-mon[93869]: pgmap v20334: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 816 KiB/s wr, 10 op/s 2023-03-10T01:17:39.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:17:39.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:17:39.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:17:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:39 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:17:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:39 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:17:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:17:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:39 smithi181 ceph-mon[94189]: pgmap v20334: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 816 KiB/s wr, 10 op/s 2023-03-10T01:17:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:17:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:17:39.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:17:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:17:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:17:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:17:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:17:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:17:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:17:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37171.514531 secs 2023-03-10T01:17:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:17:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:17:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:41 smithi038 ceph-mon[92422]: pgmap v20335: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 894 KiB/s rd, 664 KiB/s wr, 9 op/s 2023-03-10T01:17:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37171.514531 secs 2023-03-10T01:17:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:17:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:17:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:41 smithi122 ceph-mon[93869]: pgmap v20335: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 894 KiB/s rd, 664 KiB/s wr, 9 op/s 2023-03-10T01:17:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37171.514531 secs 2023-03-10T01:17:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:17:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:17:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:41 smithi181 ceph-mon[94189]: pgmap v20335: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 894 KiB/s rd, 664 KiB/s wr, 9 op/s 2023-03-10T01:17:43.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37173.977185 secs 2023-03-10T01:17:43.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37173.977185 secs 2023-03-10T01:17:43.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37173.977185 secs 2023-03-10T01:17:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37635.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:17:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:43 smithi122 ceph-mon[93869]: pgmap v20336: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 894 KiB/s rd, 664 KiB/s wr, 9 op/s 2023-03-10T01:17:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:43 smithi181 ceph-mon[94189]: pgmap v20336: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 894 KiB/s rd, 664 KiB/s wr, 9 op/s 2023-03-10T01:17:44.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:43 smithi038 ceph-mon[92422]: pgmap v20336: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 894 KiB/s rd, 664 KiB/s wr, 9 op/s 2023-03-10T01:17:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:45 smithi122 ceph-mon[93869]: pgmap v20337: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1001 KiB/s rd, 600 KiB/s wr, 10 op/s 2023-03-10T01:17:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:45 smithi181 ceph-mon[94189]: pgmap v20337: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1001 KiB/s rd, 600 KiB/s wr, 10 op/s 2023-03-10T01:17:45.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:45 smithi038 ceph-mon[92422]: pgmap v20337: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1001 KiB/s rd, 600 KiB/s wr, 10 op/s 2023-03-10T01:17:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37176.514613 secs 2023-03-10T01:17:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37176.514613 secs 2023-03-10T01:17:46.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37176.514613 secs 2023-03-10T01:17:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:47 smithi122 ceph-mon[93869]: pgmap v20338: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 871 KiB/s rd, 521 KiB/s wr, 11 op/s 2023-03-10T01:17:47.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:47 smithi181 ceph-mon[94189]: pgmap v20338: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 871 KiB/s rd, 521 KiB/s wr, 11 op/s 2023-03-10T01:17:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:47 smithi038 ceph-mon[92422]: pgmap v20338: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 871 KiB/s rd, 521 KiB/s wr, 11 op/s 2023-03-10T01:17:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:17:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:17:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:48 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 37178.977244 secs 2023-03-10T01:17:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:17:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:17:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:48 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 37178.977244 secs 2023-03-10T01:17:48.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:17:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:17:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:48 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 37178.977244 secs 2023-03-10T01:17:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37640.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:17:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:17:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:17:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:49 smithi122 ceph-mon[93869]: pgmap v20339: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 499 KiB/s wr, 9 op/s 2023-03-10T01:17:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:17:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:17:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:49 smithi181 ceph-mon[94189]: pgmap v20339: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 499 KiB/s wr, 9 op/s 2023-03-10T01:17:49.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:17:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:17:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:49 smithi038 ceph-mon[92422]: pgmap v20339: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 499 KiB/s wr, 9 op/s 2023-03-10T01:17:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37181.514696 secs 2023-03-10T01:17:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:17:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:17:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:51 smithi038 ceph-mon[92422]: pgmap v20340: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 378 KiB/s wr, 10 op/s 2023-03-10T01:17:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37181.514696 secs 2023-03-10T01:17:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:17:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:17:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:51 smithi122 ceph-mon[93869]: pgmap v20340: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 378 KiB/s wr, 10 op/s 2023-03-10T01:17:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37181.514696 secs 2023-03-10T01:17:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:17:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:17:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:51 smithi181 ceph-mon[94189]: pgmap v20340: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 378 KiB/s wr, 10 op/s 2023-03-10T01:17:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:52 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 37183.977338 secs 2023-03-10T01:17:53.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:52 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 37183.977338 secs 2023-03-10T01:17:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:52 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 37183.977338 secs 2023-03-10T01:17:53.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37645.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:17:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:53 smithi122 ceph-mon[93869]: pgmap v20341: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 532 KiB/s wr, 8 op/s 2023-03-10T01:17:54.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:53 smithi181 ceph-mon[94189]: pgmap v20341: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 532 KiB/s wr, 8 op/s 2023-03-10T01:17:54.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:53 smithi038 ceph-mon[92422]: pgmap v20341: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 532 KiB/s wr, 8 op/s 2023-03-10T01:17:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:55 smithi122 ceph-mon[93869]: pgmap v20342: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 691 KiB/s rd, 689 KiB/s wr, 10 op/s 2023-03-10T01:17:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:55 smithi181 ceph-mon[94189]: pgmap v20342: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 691 KiB/s rd, 689 KiB/s wr, 10 op/s 2023-03-10T01:17:55.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:55 smithi038 ceph-mon[92422]: pgmap v20342: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 691 KiB/s rd, 689 KiB/s wr, 10 op/s 2023-03-10T01:17:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37186.514784 secs 2023-03-10T01:17:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:17:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:17:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37186.514784 secs 2023-03-10T01:17:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:17:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:17:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37186.514784 secs 2023-03-10T01:17:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:17:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:17:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:17:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:17:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:57 smithi122 ceph-mon[93869]: pgmap v20343: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 795 KiB/s rd, 538 KiB/s wr, 12 op/s 2023-03-10T01:17:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:17:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:17:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:57 smithi181 ceph-mon[94189]: pgmap v20343: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 795 KiB/s rd, 538 KiB/s wr, 12 op/s 2023-03-10T01:17:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:17:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:17:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:57 smithi038 ceph-mon[92422]: pgmap v20343: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 795 KiB/s rd, 538 KiB/s wr, 12 op/s 2023-03-10T01:17:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37188.977401 secs 2023-03-10T01:17:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37188.977401 secs 2023-03-10T01:17:58.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37188.977401 secs 2023-03-10T01:17:58.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37650.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:17:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:17:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:17:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:17:59 smithi122 ceph-mon[93869]: pgmap v20344: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 855 KiB/s rd, 600 KiB/s wr, 12 op/s 2023-03-10T01:17:59.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:17:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:17:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:17:59 smithi181 ceph-mon[94189]: pgmap v20344: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 855 KiB/s rd, 600 KiB/s wr, 12 op/s 2023-03-10T01:17:59.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:17:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:17:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:17:59 smithi038 ceph-mon[92422]: pgmap v20344: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 855 KiB/s rd, 600 KiB/s wr, 12 op/s 2023-03-10T01:18:00.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:18:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:18:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:18:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:18:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:18:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:18:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37191.514842 secs 2023-03-10T01:18:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:18:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:18:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:01 smithi038 ceph-mon[92422]: pgmap v20345: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.0 MiB/s wr, 15 op/s 2023-03-10T01:18:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37191.514842 secs 2023-03-10T01:18:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:18:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:18:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:01 smithi122 ceph-mon[93869]: pgmap v20345: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.0 MiB/s wr, 15 op/s 2023-03-10T01:18:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37191.514842 secs 2023-03-10T01:18:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:18:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:18:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:01 smithi181 ceph-mon[94189]: pgmap v20345: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.0 MiB/s wr, 15 op/s 2023-03-10T01:18:03.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37193.977464 secs 2023-03-10T01:18:03.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37193.977464 secs 2023-03-10T01:18:03.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37193.977464 secs 2023-03-10T01:18:03.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37655.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:18:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:03 smithi122 ceph-mon[93869]: pgmap v20346: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1008 KiB/s wr, 14 op/s 2023-03-10T01:18:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:03 smithi181 ceph-mon[94189]: pgmap v20346: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1008 KiB/s wr, 14 op/s 2023-03-10T01:18:04.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:03 smithi038 ceph-mon[92422]: pgmap v20346: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1008 KiB/s wr, 14 op/s 2023-03-10T01:18:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:05 smithi122 ceph-mon[93869]: pgmap v20347: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 885 KiB/s wr, 15 op/s 2023-03-10T01:18:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:05 smithi181 ceph-mon[94189]: pgmap v20347: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 885 KiB/s wr, 15 op/s 2023-03-10T01:18:05.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:05 smithi038 ceph-mon[92422]: pgmap v20347: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 885 KiB/s wr, 15 op/s 2023-03-10T01:18:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37196.514913 secs 2023-03-10T01:18:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37196.514913 secs 2023-03-10T01:18:06.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37196.514913 secs 2023-03-10T01:18:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:07 smithi122 ceph-mon[93869]: pgmap v20348: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 835 KiB/s wr, 20 op/s 2023-03-10T01:18:07.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:07 smithi181 ceph-mon[94189]: pgmap v20348: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 835 KiB/s wr, 20 op/s 2023-03-10T01:18:07.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:07 smithi038 ceph-mon[92422]: pgmap v20348: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 835 KiB/s wr, 20 op/s 2023-03-10T01:18:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:18:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:18:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37198.977526 secs 2023-03-10T01:18:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:18:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:18:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37198.977526 secs 2023-03-10T01:18:08.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:18:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:18:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37198.977526 secs 2023-03-10T01:18:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37660.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:18:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:18:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:18:09.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:09 smithi122 ceph-mon[93869]: pgmap v20349: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 676 KiB/s rd, 700 KiB/s wr, 28 op/s 2023-03-10T01:18:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:18:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:18:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:09 smithi181 ceph-mon[94189]: pgmap v20349: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 676 KiB/s rd, 700 KiB/s wr, 28 op/s 2023-03-10T01:18:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:18:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:18:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:09 smithi038 ceph-mon[92422]: pgmap v20349: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 676 KiB/s rd, 700 KiB/s wr, 28 op/s 2023-03-10T01:18:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:10 smithi122 ceph-mon[93869]: osdmap e1487: 12 total, 12 up, 12 in 2023-03-10T01:18:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:10 smithi181 ceph-mon[94189]: osdmap e1487: 12 total, 12 up, 12 in 2023-03-10T01:18:10.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:10 smithi038 ceph-mon[92422]: osdmap e1487: 12 total, 12 up, 12 in 2023-03-10T01:18:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37201.514973 secs 2023-03-10T01:18:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:11 smithi181 ceph-mon[94189]: pgmap v20351: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 460 KiB/s wr, 30 op/s 2023-03-10T01:18:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37201.514973 secs 2023-03-10T01:18:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:11 smithi038 ceph-mon[92422]: pgmap v20351: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 460 KiB/s wr, 30 op/s 2023-03-10T01:18:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37201.514973 secs 2023-03-10T01:18:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:11 smithi122 ceph-mon[93869]: pgmap v20351: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 460 KiB/s wr, 30 op/s 2023-03-10T01:18:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37203.977585 secs 2023-03-10T01:18:13.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37203.977585 secs 2023-03-10T01:18:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37203.977585 secs 2023-03-10T01:18:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37665.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:18:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:13 smithi122 ceph-mon[93869]: pgmap v20352: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 460 KiB/s wr, 30 op/s 2023-03-10T01:18:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:13 smithi181 ceph-mon[94189]: pgmap v20352: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 460 KiB/s wr, 30 op/s 2023-03-10T01:18:14.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:13 smithi038 ceph-mon[92422]: pgmap v20352: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 460 KiB/s wr, 30 op/s 2023-03-10T01:18:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:15 smithi122 ceph-mon[93869]: pgmap v20353: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 628 KiB/s wr, 29 op/s 2023-03-10T01:18:15.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:15 smithi181 ceph-mon[94189]: pgmap v20353: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 628 KiB/s wr, 29 op/s 2023-03-10T01:18:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:15 smithi038 ceph-mon[92422]: pgmap v20353: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 628 KiB/s wr, 29 op/s 2023-03-10T01:18:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37206.515010 secs 2023-03-10T01:18:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:18:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:18:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37206.515010 secs 2023-03-10T01:18:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:18:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:18:16.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37206.515010 secs 2023-03-10T01:18:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:18:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:18:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:18:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:18:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:17 smithi122 ceph-mon[93869]: pgmap v20354: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 483 KiB/s rd, 499 KiB/s wr, 24 op/s 2023-03-10T01:18:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:18:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:18:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:17 smithi181 ceph-mon[94189]: pgmap v20354: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 483 KiB/s rd, 499 KiB/s wr, 24 op/s 2023-03-10T01:18:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:18:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:18:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:17 smithi038 ceph-mon[92422]: pgmap v20354: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 483 KiB/s rd, 499 KiB/s wr, 24 op/s 2023-03-10T01:18:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37208.977674 secs 2023-03-10T01:18:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37208.977674 secs 2023-03-10T01:18:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37208.977674 secs 2023-03-10T01:18:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37670.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:18:19.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:19 smithi122 ceph-mon[93869]: pgmap v20355: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 436 KiB/s wr, 11 op/s 2023-03-10T01:18:19.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:19 smithi122 ceph-mon[93869]: osdmap e1488: 12 total, 12 up, 12 in 2023-03-10T01:18:19.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:19 smithi181 ceph-mon[94189]: pgmap v20355: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 436 KiB/s wr, 11 op/s 2023-03-10T01:18:19.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:19 smithi181 ceph-mon[94189]: osdmap e1488: 12 total, 12 up, 12 in 2023-03-10T01:18:19.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:19 smithi038 ceph-mon[92422]: pgmap v20355: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 436 KiB/s wr, 11 op/s 2023-03-10T01:18:19.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:19 smithi038 ceph-mon[92422]: osdmap e1488: 12 total, 12 up, 12 in 2023-03-10T01:18:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37211.515096 secs 2023-03-10T01:18:21.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37211.515096 secs 2023-03-10T01:18:21.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37211.515096 secs 2023-03-10T01:18:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:21 smithi122 ceph-mon[93869]: pgmap v20357: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 207 KiB/s wr, 9 op/s 2023-03-10T01:18:22.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:21 smithi181 ceph-mon[94189]: pgmap v20357: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 207 KiB/s wr, 9 op/s 2023-03-10T01:18:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:21 smithi038 ceph-mon[92422]: pgmap v20357: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 207 KiB/s wr, 9 op/s 2023-03-10T01:18:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37213.977766 secs 2023-03-10T01:18:23.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37213.977766 secs 2023-03-10T01:18:23.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37213.977766 secs 2023-03-10T01:18:23.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37675.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:18:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:23 smithi122 ceph-mon[93869]: pgmap v20358: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 225 KiB/s wr, 9 op/s 2023-03-10T01:18:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:23 smithi181 ceph-mon[94189]: pgmap v20358: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 225 KiB/s wr, 9 op/s 2023-03-10T01:18:24.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:23 smithi038 ceph-mon[92422]: pgmap v20358: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 225 KiB/s wr, 9 op/s 2023-03-10T01:18:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:25 smithi122 ceph-mon[93869]: pgmap v20359: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 178 KiB/s wr, 8 op/s 2023-03-10T01:18:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:25 smithi181 ceph-mon[94189]: pgmap v20359: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 178 KiB/s wr, 8 op/s 2023-03-10T01:18:25.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:25 smithi038 ceph-mon[92422]: pgmap v20359: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 178 KiB/s wr, 8 op/s 2023-03-10T01:18:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37216.515155 secs 2023-03-10T01:18:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37216.515155 secs 2023-03-10T01:18:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37216.515155 secs 2023-03-10T01:18:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:27 smithi122 ceph-mon[93869]: pgmap v20360: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 178 KiB/s wr, 7 op/s 2023-03-10T01:18:27.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:27 smithi181 ceph-mon[94189]: pgmap v20360: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 178 KiB/s wr, 7 op/s 2023-03-10T01:18:27.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:27 smithi038 ceph-mon[92422]: pgmap v20360: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 178 KiB/s wr, 7 op/s 2023-03-10T01:18:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37218.977874 secs 2023-03-10T01:18:28.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37218.977874 secs 2023-03-10T01:18:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37680.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:18:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37218.977874 secs 2023-03-10T01:18:29.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:29 smithi038 ceph-mon[92422]: pgmap v20361: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 178 KiB/s wr, 7 op/s 2023-03-10T01:18:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:29 smithi122 ceph-mon[93869]: pgmap v20361: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 178 KiB/s wr, 7 op/s 2023-03-10T01:18:29.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:29 smithi181 ceph-mon[94189]: pgmap v20361: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 178 KiB/s wr, 7 op/s 2023-03-10T01:18:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37221.515213 secs 2023-03-10T01:18:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37221.515213 secs 2023-03-10T01:18:31.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37221.515213 secs 2023-03-10T01:18:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:31 smithi122 ceph-mon[93869]: pgmap v20362: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 156 KiB/s wr, 8 op/s 2023-03-10T01:18:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:31 smithi181 ceph-mon[94189]: pgmap v20362: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 156 KiB/s wr, 8 op/s 2023-03-10T01:18:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:31 smithi038 ceph-mon[92422]: pgmap v20362: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 156 KiB/s wr, 8 op/s 2023-03-10T01:18:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37223.977936 secs 2023-03-10T01:18:33.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37223.977936 secs 2023-03-10T01:18:33.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37223.977936 secs 2023-03-10T01:18:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37685.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:18:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:33 smithi122 ceph-mon[93869]: pgmap v20363: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 148 KiB/s wr, 6 op/s 2023-03-10T01:18:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:33 smithi181 ceph-mon[94189]: pgmap v20363: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 148 KiB/s wr, 6 op/s 2023-03-10T01:18:34.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:33 smithi038 ceph-mon[92422]: pgmap v20363: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 148 KiB/s wr, 6 op/s 2023-03-10T01:18:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:35 smithi122 ceph-mon[93869]: pgmap v20364: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 133 KiB/s wr, 6 op/s 2023-03-10T01:18:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:35 smithi181 ceph-mon[94189]: pgmap v20364: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 133 KiB/s wr, 6 op/s 2023-03-10T01:18:35.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:35 smithi038 ceph-mon[92422]: pgmap v20364: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 133 KiB/s wr, 6 op/s 2023-03-10T01:18:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37226.515272 secs 2023-03-10T01:18:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37226.515272 secs 2023-03-10T01:18:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37226.515272 secs 2023-03-10T01:18:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:18:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:18:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:37 smithi122 ceph-mon[93869]: pgmap v20365: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 177 KiB/s wr, 8 op/s 2023-03-10T01:18:37.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:18:37.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:18:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:37 smithi181 ceph-mon[94189]: pgmap v20365: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 177 KiB/s wr, 8 op/s 2023-03-10T01:18:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:18:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:18:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:37 smithi038 ceph-mon[92422]: pgmap v20365: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 177 KiB/s wr, 8 op/s 2023-03-10T01:18:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37228.977995 secs 2023-03-10T01:18:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37228.977995 secs 2023-03-10T01:18:38.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37228.977995 secs 2023-03-10T01:18:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37690.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:18:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:39 smithi122 ceph-mon[93869]: pgmap v20366: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 177 KiB/s wr, 7 op/s 2023-03-10T01:18:39.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:39 smithi181 ceph-mon[94189]: pgmap v20366: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 177 KiB/s wr, 7 op/s 2023-03-10T01:18:39.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:39 smithi038 ceph-mon[92422]: pgmap v20366: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 177 KiB/s wr, 7 op/s 2023-03-10T01:18:40.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:18:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:18:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:18:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:18:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:18:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:18:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:18:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:18:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:18:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:18:40.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:18:40.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:18:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:18:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:18:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:18:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:18:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:18:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:18:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37231.515331 secs 2023-03-10T01:18:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:41 smithi038 ceph-mon[92422]: pgmap v20367: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 310 KiB/s rd, 307 KiB/s wr, 9 op/s 2023-03-10T01:18:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37231.515331 secs 2023-03-10T01:18:41.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:41 smithi122 ceph-mon[93869]: pgmap v20367: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 310 KiB/s rd, 307 KiB/s wr, 9 op/s 2023-03-10T01:18:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37231.515331 secs 2023-03-10T01:18:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:41 smithi181 ceph-mon[94189]: pgmap v20367: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 310 KiB/s rd, 307 KiB/s wr, 9 op/s 2023-03-10T01:18:43.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37233.978083 secs 2023-03-10T01:18:43.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37233.978083 secs 2023-03-10T01:18:43.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37233.978083 secs 2023-03-10T01:18:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37695.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:18:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:43 smithi122 ceph-mon[93869]: pgmap v20368: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 456 KiB/s wr, 7 op/s 2023-03-10T01:18:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:43 smithi181 ceph-mon[94189]: pgmap v20368: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 456 KiB/s wr, 7 op/s 2023-03-10T01:18:44.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:43 smithi038 ceph-mon[92422]: pgmap v20368: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 456 KiB/s wr, 7 op/s 2023-03-10T01:18:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:45 smithi122 ceph-mon[93869]: pgmap v20369: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 456 KiB/s wr, 8 op/s 2023-03-10T01:18:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:45 smithi181 ceph-mon[94189]: pgmap v20369: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 456 KiB/s wr, 8 op/s 2023-03-10T01:18:45.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:45 smithi038 ceph-mon[92422]: pgmap v20369: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 456 KiB/s wr, 8 op/s 2023-03-10T01:18:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37236.515392 secs 2023-03-10T01:18:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37236.515392 secs 2023-03-10T01:18:46.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37236.515392 secs 2023-03-10T01:18:47.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:47 smithi181 ceph-mon[94189]: pgmap v20370: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 617 KiB/s rd, 613 KiB/s wr, 9 op/s 2023-03-10T01:18:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:47 smithi038 ceph-mon[92422]: pgmap v20370: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 617 KiB/s rd, 613 KiB/s wr, 9 op/s 2023-03-10T01:18:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:47 smithi122 ceph-mon[93869]: pgmap v20370: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 617 KiB/s rd, 613 KiB/s wr, 9 op/s 2023-03-10T01:18:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:18:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:18:48.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37238.978139 secs 2023-03-10T01:18:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:18:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:18:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37238.978139 secs 2023-03-10T01:18:48.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37700.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:18:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:18:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:18:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37238.978139 secs 2023-03-10T01:18:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:18:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:18:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:49 smithi122 ceph-mon[93869]: pgmap v20371: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 437 KiB/s wr, 7 op/s 2023-03-10T01:18:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:18:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:18:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:49 smithi181 ceph-mon[94189]: pgmap v20371: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 437 KiB/s wr, 7 op/s 2023-03-10T01:18:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:18:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:18:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:49 smithi038 ceph-mon[92422]: pgmap v20371: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 439 KiB/s rd, 437 KiB/s wr, 7 op/s 2023-03-10T01:18:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37241.515463 secs 2023-03-10T01:18:51.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37241.515463 secs 2023-03-10T01:18:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37241.515463 secs 2023-03-10T01:18:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:51 smithi122 ceph-mon[93869]: pgmap v20372: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 491 KiB/s wr, 9 op/s 2023-03-10T01:18:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:51 smithi181 ceph-mon[94189]: pgmap v20372: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 491 KiB/s wr, 9 op/s 2023-03-10T01:18:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:51 smithi038 ceph-mon[92422]: pgmap v20372: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 491 KiB/s wr, 9 op/s 2023-03-10T01:18:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37243.978199 secs 2023-03-10T01:18:53.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37243.978199 secs 2023-03-10T01:18:53.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37243.978199 secs 2023-03-10T01:18:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37705.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:18:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:53 smithi122 ceph-mon[93869]: pgmap v20373: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 362 KiB/s wr, 7 op/s 2023-03-10T01:18:54.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:53 smithi181 ceph-mon[94189]: pgmap v20373: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 362 KiB/s wr, 7 op/s 2023-03-10T01:18:54.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:53 smithi038 ceph-mon[92422]: pgmap v20373: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 362 KiB/s wr, 7 op/s 2023-03-10T01:18:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:55 smithi122 ceph-mon[93869]: pgmap v20374: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 455 KiB/s rd, 403 KiB/s wr, 8 op/s 2023-03-10T01:18:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:55 smithi181 ceph-mon[94189]: pgmap v20374: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 455 KiB/s rd, 403 KiB/s wr, 8 op/s 2023-03-10T01:18:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:55 smithi038 ceph-mon[92422]: pgmap v20374: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 455 KiB/s rd, 403 KiB/s wr, 8 op/s 2023-03-10T01:18:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37246.515522 secs 2023-03-10T01:18:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:18:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:18:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37246.515522 secs 2023-03-10T01:18:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:18:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:18:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37246.515522 secs 2023-03-10T01:18:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:18:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:18:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:18:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:18:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:57 smithi181 ceph-mon[94189]: pgmap v20375: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 404 KiB/s wr, 11 op/s 2023-03-10T01:18:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:18:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:18:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:57 smithi038 ceph-mon[92422]: pgmap v20375: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 404 KiB/s wr, 11 op/s 2023-03-10T01:18:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:18:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:18:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:57 smithi122 ceph-mon[93869]: pgmap v20375: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 404 KiB/s wr, 11 op/s 2023-03-10T01:18:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:58 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:18:58.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:58 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:18:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37710.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:18:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:58 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:18:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:59 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:18:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:18:59.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:18:59.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:18:59 smithi038 ceph-mon[92422]: pgmap v20376: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 282 KiB/s wr, 10 op/s 2023-03-10T01:18:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:59 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:18:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:18:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:18:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:18:59 smithi122 ceph-mon[93869]: pgmap v20376: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 282 KiB/s wr, 10 op/s 2023-03-10T01:18:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:59 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:18:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:18:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:18:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:18:59 smithi181 ceph-mon[94189]: pgmap v20376: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 282 KiB/s wr, 10 op/s 2023-03-10T01:19:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37251.515580 secs 2023-03-10T01:19:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:19:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:19:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:01 smithi038 ceph-mon[92422]: pgmap v20377: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 880 KiB/s rd, 816 KiB/s wr, 14 op/s 2023-03-10T01:19:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37251.515580 secs 2023-03-10T01:19:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:19:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:19:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:01 smithi122 ceph-mon[93869]: pgmap v20377: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 880 KiB/s rd, 816 KiB/s wr, 14 op/s 2023-03-10T01:19:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37251.515580 secs 2023-03-10T01:19:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:19:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:19:01.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:01 smithi181 ceph-mon[94189]: pgmap v20377: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 880 KiB/s rd, 816 KiB/s wr, 14 op/s 2023-03-10T01:19:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37715.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:19:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:03 smithi122 ceph-mon[93869]: pgmap v20378: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 775 KiB/s rd, 761 KiB/s wr, 12 op/s 2023-03-10T01:19:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:03 smithi181 ceph-mon[94189]: pgmap v20378: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 775 KiB/s rd, 761 KiB/s wr, 12 op/s 2023-03-10T01:19:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:03 smithi038 ceph-mon[92422]: pgmap v20378: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 775 KiB/s rd, 761 KiB/s wr, 12 op/s 2023-03-10T01:19:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:05 smithi122 ceph-mon[93869]: pgmap v20379: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 834 KiB/s rd, 819 KiB/s wr, 14 op/s 2023-03-10T01:19:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:05 smithi181 ceph-mon[94189]: pgmap v20379: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 834 KiB/s rd, 819 KiB/s wr, 14 op/s 2023-03-10T01:19:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:05 smithi038 ceph-mon[92422]: pgmap v20379: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 834 KiB/s rd, 819 KiB/s wr, 14 op/s 2023-03-10T01:19:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37256.515669 secs 2023-03-10T01:19:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37256.515669 secs 2023-03-10T01:19:06.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37256.515669 secs 2023-03-10T01:19:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:07 smithi122 ceph-mon[93869]: pgmap v20380: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 804 KiB/s rd, 812 KiB/s wr, 19 op/s 2023-03-10T01:19:07.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:07 smithi181 ceph-mon[94189]: pgmap v20380: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 804 KiB/s rd, 812 KiB/s wr, 19 op/s 2023-03-10T01:19:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:07 smithi038 ceph-mon[92422]: pgmap v20380: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 804 KiB/s rd, 812 KiB/s wr, 19 op/s 2023-03-10T01:19:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:19:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:19:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37258.978425 secs 2023-03-10T01:19:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:19:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:19:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37258.978425 secs 2023-03-10T01:19:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:19:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:19:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37258.978425 secs 2023-03-10T01:19:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37720.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:19:09.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:19:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:19:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:09 smithi038 ceph-mon[92422]: pgmap v20381: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 838 KiB/s rd, 863 KiB/s wr, 28 op/s 2023-03-10T01:19:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:19:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:19:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:09 smithi122 ceph-mon[93869]: pgmap v20381: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 838 KiB/s rd, 863 KiB/s wr, 28 op/s 2023-03-10T01:19:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:19:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:19:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:09 smithi181 ceph-mon[94189]: pgmap v20381: 129 pgs: 129 active+clean; 14 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 838 KiB/s rd, 863 KiB/s wr, 28 op/s 2023-03-10T01:19:10.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:10 smithi038 ceph-mon[92422]: osdmap e1489: 12 total, 12 up, 12 in 2023-03-10T01:19:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:10 smithi122 ceph-mon[93869]: osdmap e1489: 12 total, 12 up, 12 in 2023-03-10T01:19:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:10 smithi181 ceph-mon[94189]: osdmap e1489: 12 total, 12 up, 12 in 2023-03-10T01:19:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37261.515753 secs 2023-03-10T01:19:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:11 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:19:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:11 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:19:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:11 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:19:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:11 smithi038 ceph-mon[92422]: pgmap v20383: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 761 KiB/s wr, 30 op/s 2023-03-10T01:19:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37261.515753 secs 2023-03-10T01:19:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:11 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:19:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:11 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:19:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:11 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:19:11.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:11 smithi122 ceph-mon[93869]: pgmap v20383: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 761 KiB/s wr, 30 op/s 2023-03-10T01:19:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37261.515753 secs 2023-03-10T01:19:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:11 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:19:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:11 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:19:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:11 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:19:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:11 smithi181 ceph-mon[94189]: pgmap v20383: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 761 KiB/s wr, 30 op/s 2023-03-10T01:19:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:19:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:19:12.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:19:12.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:19:12.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:19:12.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:19:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37725.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:19:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:19:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:19:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37263.978515 secs 2023-03-10T01:19:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:13 smithi122 ceph-mon[93869]: pgmap v20384: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 761 KiB/s wr, 30 op/s 2023-03-10T01:19:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:19:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:19:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37263.978515 secs 2023-03-10T01:19:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:13 smithi181 ceph-mon[94189]: pgmap v20384: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 761 KiB/s wr, 30 op/s 2023-03-10T01:19:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:19:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:19:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37263.978515 secs 2023-03-10T01:19:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:13 smithi038 ceph-mon[92422]: pgmap v20384: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 761 KiB/s wr, 30 op/s 2023-03-10T01:19:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:15 smithi122 ceph-mon[93869]: pgmap v20385: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 692 KiB/s wr, 27 op/s 2023-03-10T01:19:15.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:15 smithi181 ceph-mon[94189]: pgmap v20385: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 692 KiB/s wr, 27 op/s 2023-03-10T01:19:15.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:15 smithi038 ceph-mon[92422]: pgmap v20385: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 692 KiB/s wr, 27 op/s 2023-03-10T01:19:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37266.515838 secs 2023-03-10T01:19:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:19:16.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:19:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37266.515838 secs 2023-03-10T01:19:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:19:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:19:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37266.515838 secs 2023-03-10T01:19:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:19:16.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:19:17.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:17 smithi038 ceph-mon[92422]: pgmap v20386: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 604 KiB/s rd, 619 KiB/s wr, 23 op/s 2023-03-10T01:19:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:17 smithi122 ceph-mon[93869]: pgmap v20386: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 604 KiB/s rd, 619 KiB/s wr, 23 op/s 2023-03-10T01:19:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:17 smithi181 ceph-mon[94189]: pgmap v20386: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 604 KiB/s rd, 619 KiB/s wr, 23 op/s 2023-03-10T01:19:18.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37268.978590 secs 2023-03-10T01:19:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37730.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:19:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37268.978590 secs 2023-03-10T01:19:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37268.978590 secs 2023-03-10T01:19:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:19 smithi122 ceph-mon[93869]: pgmap v20387: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 558 KiB/s rd, 556 KiB/s wr, 9 op/s 2023-03-10T01:19:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:19 smithi122 ceph-mon[93869]: osdmap e1490: 12 total, 12 up, 12 in 2023-03-10T01:19:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:19 smithi181 ceph-mon[94189]: pgmap v20387: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 558 KiB/s rd, 556 KiB/s wr, 9 op/s 2023-03-10T01:19:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:19 smithi181 ceph-mon[94189]: osdmap e1490: 12 total, 12 up, 12 in 2023-03-10T01:19:19.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:19 smithi038 ceph-mon[92422]: pgmap v20387: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 558 KiB/s rd, 556 KiB/s wr, 9 op/s 2023-03-10T01:19:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:19 smithi038 ceph-mon[92422]: osdmap e1490: 12 total, 12 up, 12 in 2023-03-10T01:19:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37271.515922 secs 2023-03-10T01:19:21.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37271.515922 secs 2023-03-10T01:19:21.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37271.515922 secs 2023-03-10T01:19:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:21 smithi122 ceph-mon[93869]: pgmap v20389: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 210 KiB/s wr, 8 op/s 2023-03-10T01:19:22.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:21 smithi181 ceph-mon[94189]: pgmap v20389: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 210 KiB/s wr, 8 op/s 2023-03-10T01:19:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:21 smithi038 ceph-mon[92422]: pgmap v20389: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 210 KiB/s wr, 8 op/s 2023-03-10T01:19:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37273.978674 secs 2023-03-10T01:19:23.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37273.978674 secs 2023-03-10T01:19:23.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37273.978674 secs 2023-03-10T01:19:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37735.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:19:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:23 smithi122 ceph-mon[93869]: pgmap v20390: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 210 KiB/s wr, 8 op/s 2023-03-10T01:19:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:23 smithi181 ceph-mon[94189]: pgmap v20390: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 210 KiB/s wr, 8 op/s 2023-03-10T01:19:24.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:23 smithi038 ceph-mon[92422]: pgmap v20390: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 213 KiB/s rd, 210 KiB/s wr, 8 op/s 2023-03-10T01:19:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:25 smithi122 ceph-mon[93869]: pgmap v20391: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 263 KiB/s wr, 8 op/s 2023-03-10T01:19:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:25 smithi181 ceph-mon[94189]: pgmap v20391: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 263 KiB/s wr, 8 op/s 2023-03-10T01:19:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:25 smithi038 ceph-mon[92422]: pgmap v20391: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 263 KiB/s wr, 8 op/s 2023-03-10T01:19:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37276.515981 secs 2023-03-10T01:19:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37276.515981 secs 2023-03-10T01:19:26.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37276.515981 secs 2023-03-10T01:19:27.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:27 smithi181 ceph-mon[94189]: pgmap v20392: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 116 KiB/s wr, 7 op/s 2023-03-10T01:19:27.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:27 smithi038 ceph-mon[92422]: pgmap v20392: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 116 KiB/s wr, 7 op/s 2023-03-10T01:19:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:27 smithi122 ceph-mon[93869]: pgmap v20392: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 116 KiB/s wr, 7 op/s 2023-03-10T01:19:28.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37278.978738 secs 2023-03-10T01:19:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37740.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:19:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37278.978738 secs 2023-03-10T01:19:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37278.978738 secs 2023-03-10T01:19:29.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:29 smithi038 ceph-mon[92422]: pgmap v20393: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 116 KiB/s wr, 8 op/s 2023-03-10T01:19:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:29 smithi122 ceph-mon[93869]: pgmap v20393: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 116 KiB/s wr, 8 op/s 2023-03-10T01:19:29.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:29 smithi181 ceph-mon[94189]: pgmap v20393: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 116 KiB/s wr, 8 op/s 2023-03-10T01:19:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37281.516040 secs 2023-03-10T01:19:31.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37281.516040 secs 2023-03-10T01:19:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37281.516040 secs 2023-03-10T01:19:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:31 smithi122 ceph-mon[93869]: pgmap v20394: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 102 KiB/s wr, 10 op/s 2023-03-10T01:19:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:31 smithi181 ceph-mon[94189]: pgmap v20394: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 102 KiB/s wr, 10 op/s 2023-03-10T01:19:32.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:31 smithi038 ceph-mon[92422]: pgmap v20394: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 102 KiB/s wr, 10 op/s 2023-03-10T01:19:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37283.978795 secs 2023-03-10T01:19:33.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37283.978795 secs 2023-03-10T01:19:33.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37283.978795 secs 2023-03-10T01:19:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37745.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:19:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:33 smithi122 ceph-mon[93869]: pgmap v20395: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 45 KiB/s wr, 7 op/s 2023-03-10T01:19:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:33 smithi181 ceph-mon[94189]: pgmap v20395: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 45 KiB/s wr, 7 op/s 2023-03-10T01:19:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:33 smithi038 ceph-mon[92422]: pgmap v20395: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 45 KiB/s wr, 7 op/s 2023-03-10T01:19:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:35 smithi122 ceph-mon[93869]: pgmap v20396: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 202 KiB/s wr, 8 op/s 2023-03-10T01:19:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:35 smithi181 ceph-mon[94189]: pgmap v20396: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 202 KiB/s wr, 8 op/s 2023-03-10T01:19:35.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:35 smithi038 ceph-mon[92422]: pgmap v20396: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 202 KiB/s wr, 8 op/s 2023-03-10T01:19:36.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37286.516100 secs 2023-03-10T01:19:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:19:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:19:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37286.516100 secs 2023-03-10T01:19:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:19:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:19:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37286.516100 secs 2023-03-10T01:19:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:19:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:19:37.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:19:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:19:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:37 smithi038 ceph-mon[92422]: pgmap v20397: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 159 KiB/s wr, 10 op/s 2023-03-10T01:19:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:19:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:19:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:37 smithi122 ceph-mon[93869]: pgmap v20397: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 159 KiB/s wr, 10 op/s 2023-03-10T01:19:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:19:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:19:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:37 smithi181 ceph-mon[94189]: pgmap v20397: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 159 KiB/s wr, 10 op/s 2023-03-10T01:19:38.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37750.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:19:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37288.978856 secs 2023-03-10T01:19:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37288.978856 secs 2023-03-10T01:19:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37288.978856 secs 2023-03-10T01:19:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:39 smithi122 ceph-mon[93869]: pgmap v20398: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 159 KiB/s wr, 8 op/s 2023-03-10T01:19:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:39 smithi181 ceph-mon[94189]: pgmap v20398: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 159 KiB/s wr, 8 op/s 2023-03-10T01:19:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:39 smithi038 ceph-mon[92422]: pgmap v20398: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 159 KiB/s wr, 8 op/s 2023-03-10T01:19:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:19:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:19:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:19:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37291.516162 secs 2023-03-10T01:19:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:41 smithi122 ceph-mon[93869]: pgmap v20399: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 159 KiB/s wr, 9 op/s 2023-03-10T01:19:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:19:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:19:41.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:19:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37291.516162 secs 2023-03-10T01:19:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:41 smithi181 ceph-mon[94189]: pgmap v20399: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 159 KiB/s wr, 9 op/s 2023-03-10T01:19:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:19:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:19:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:19:41.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37291.516162 secs 2023-03-10T01:19:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:41 smithi038 ceph-mon[92422]: pgmap v20399: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 159 KiB/s wr, 9 op/s 2023-03-10T01:19:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:19:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:19:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:19:43.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37293.978938 secs 2023-03-10T01:19:43.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37293.978938 secs 2023-03-10T01:19:43.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37293.978938 secs 2023-03-10T01:19:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37755.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:19:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:43 smithi122 ceph-mon[93869]: pgmap v20400: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 158 KiB/s wr, 7 op/s 2023-03-10T01:19:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:43 smithi181 ceph-mon[94189]: pgmap v20400: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 158 KiB/s wr, 7 op/s 2023-03-10T01:19:44.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:43 smithi038 ceph-mon[92422]: pgmap v20400: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 158 KiB/s wr, 7 op/s 2023-03-10T01:19:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:45 smithi122 ceph-mon[93869]: pgmap v20401: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 158 KiB/s wr, 7 op/s 2023-03-10T01:19:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:45 smithi181 ceph-mon[94189]: pgmap v20401: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 158 KiB/s wr, 7 op/s 2023-03-10T01:19:45.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:45 smithi038 ceph-mon[92422]: pgmap v20401: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 158 KiB/s wr, 7 op/s 2023-03-10T01:19:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37296.516220 secs 2023-03-10T01:19:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37296.516220 secs 2023-03-10T01:19:46.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37296.516220 secs 2023-03-10T01:19:47.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:47 smithi038 ceph-mon[92422]: pgmap v20402: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 8 op/s 2023-03-10T01:19:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:47 smithi122 ceph-mon[93869]: pgmap v20402: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 8 op/s 2023-03-10T01:19:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:47 smithi181 ceph-mon[94189]: pgmap v20402: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 19 KiB/s wr, 8 op/s 2023-03-10T01:19:48.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:48 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 37298.979052 secs 2023-03-10T01:19:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37760.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:19:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:48 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 37298.979052 secs 2023-03-10T01:19:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:48 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 37298.979052 secs 2023-03-10T01:19:49.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:49 smithi038 ceph-mon[92422]: pgmap v20403: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 158 KiB/s wr, 7 op/s 2023-03-10T01:19:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:49 smithi122 ceph-mon[93869]: pgmap v20403: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 158 KiB/s wr, 7 op/s 2023-03-10T01:19:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:49 smithi181 ceph-mon[94189]: pgmap v20403: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 158 KiB/s wr, 7 op/s 2023-03-10T01:19:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37301.516280 secs 2023-03-10T01:19:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:19:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:19:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:51 smithi122 ceph-mon[93869]: pgmap v20404: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 316 KiB/s wr, 8 op/s 2023-03-10T01:19:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37301.516280 secs 2023-03-10T01:19:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:19:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:19:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:51 smithi181 ceph-mon[94189]: pgmap v20404: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 316 KiB/s wr, 8 op/s 2023-03-10T01:19:51.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37301.516280 secs 2023-03-10T01:19:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:19:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:19:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:51 smithi038 ceph-mon[92422]: pgmap v20404: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 316 KiB/s wr, 8 op/s 2023-03-10T01:19:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37765.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:19:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:19:53.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:19:53.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:53 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 37303.979164 secs 2023-03-10T01:19:53.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:53 smithi122 ceph-mon[93869]: pgmap v20405: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 316 KiB/s wr, 7 op/s 2023-03-10T01:19:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:19:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:19:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:53 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 37303.979164 secs 2023-03-10T01:19:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:53 smithi181 ceph-mon[94189]: pgmap v20405: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 316 KiB/s wr, 7 op/s 2023-03-10T01:19:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:19:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:19:54.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:53 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 37303.979164 secs 2023-03-10T01:19:54.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:53 smithi038 ceph-mon[92422]: pgmap v20405: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 316 KiB/s wr, 7 op/s 2023-03-10T01:19:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:55 smithi038 ceph-mon[92422]: pgmap v20406: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 347 KiB/s wr, 8 op/s 2023-03-10T01:19:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:55 smithi122 ceph-mon[93869]: pgmap v20406: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 347 KiB/s wr, 8 op/s 2023-03-10T01:19:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:55 smithi181 ceph-mon[94189]: pgmap v20406: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 347 KiB/s wr, 8 op/s 2023-03-10T01:19:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37306.516364 secs 2023-03-10T01:19:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:19:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:19:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37306.516364 secs 2023-03-10T01:19:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:19:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:19:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37306.516364 secs 2023-03-10T01:19:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:19:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:19:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:57 smithi122 ceph-mon[93869]: pgmap v20407: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 658 KiB/s rd, 513 KiB/s wr, 10 op/s 2023-03-10T01:19:58.014 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:57 smithi038 ceph-mon[92422]: pgmap v20407: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 658 KiB/s rd, 513 KiB/s wr, 10 op/s 2023-03-10T01:19:58.014 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:57 smithi181 ceph-mon[94189]: pgmap v20407: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 658 KiB/s rd, 513 KiB/s wr, 10 op/s 2023-03-10T01:19:58.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37770.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:19:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37308.979222 secs 2023-03-10T01:19:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37308.979222 secs 2023-03-10T01:19:58.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37308.979222 secs 2023-03-10T01:19:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:19:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:19:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:19:59 smithi122 ceph-mon[93869]: pgmap v20408: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 670 KiB/s rd, 745 KiB/s wr, 10 op/s 2023-03-10T01:19:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:19:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:19:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:19:59 smithi181 ceph-mon[94189]: pgmap v20408: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 670 KiB/s rd, 745 KiB/s wr, 10 op/s 2023-03-10T01:19:59.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:19:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:19:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:19:59 smithi038 ceph-mon[92422]: pgmap v20408: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 670 KiB/s rd, 745 KiB/s wr, 10 op/s 2023-03-10T01:20:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:20:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:20:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:00 smithi122 ceph-mon[93869]: Health detail: HEALTH_WARN 2 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-10T01:20:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:00 smithi122 ceph-mon[93869]: [WRN] MDS_SLOW_REQUEST: 2 MDSs report slow requests 2023-03-10T01:20:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:00 smithi122 ceph-mon[93869]: mds.i(mds.2): 2 slow requests are blocked > 180 secs 2023-03-10T01:20:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:00 smithi122 ceph-mon[93869]: mds.h(mds.1): 2 slow requests are blocked > 180 secs 2023-03-10T01:20:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:00 smithi122 ceph-mon[93869]: [WRN] MDS_TRIM: 2 MDSs behind on trimming 2023-03-10T01:20:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:00 smithi122 ceph-mon[93869]: mds.e(mds.0): Behind on trimming (1283/128) max_segments: 128, num_segments: 1283 2023-03-10T01:20:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:00 smithi122 ceph-mon[93869]: mds.b(mds.0): Behind on trimming (1282/128) max_segments: 128, num_segments: 1282 2023-03-10T01:20:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:20:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:20:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:00 smithi181 ceph-mon[94189]: Health detail: HEALTH_WARN 2 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-10T01:20:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:00 smithi181 ceph-mon[94189]: [WRN] MDS_SLOW_REQUEST: 2 MDSs report slow requests 2023-03-10T01:20:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:00 smithi181 ceph-mon[94189]: mds.i(mds.2): 2 slow requests are blocked > 180 secs 2023-03-10T01:20:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:00 smithi181 ceph-mon[94189]: mds.h(mds.1): 2 slow requests are blocked > 180 secs 2023-03-10T01:20:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:00 smithi181 ceph-mon[94189]: [WRN] MDS_TRIM: 2 MDSs behind on trimming 2023-03-10T01:20:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:00 smithi181 ceph-mon[94189]: mds.e(mds.0): Behind on trimming (1283/128) max_segments: 128, num_segments: 1283 2023-03-10T01:20:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:00 smithi181 ceph-mon[94189]: mds.b(mds.0): Behind on trimming (1282/128) max_segments: 128, num_segments: 1282 2023-03-10T01:20:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:20:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:20:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:00 smithi038 ceph-mon[92422]: Health detail: HEALTH_WARN 2 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-10T01:20:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:00 smithi038 ceph-mon[92422]: [WRN] MDS_SLOW_REQUEST: 2 MDSs report slow requests 2023-03-10T01:20:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:00 smithi038 ceph-mon[92422]: mds.i(mds.2): 2 slow requests are blocked > 180 secs 2023-03-10T01:20:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:00 smithi038 ceph-mon[92422]: mds.h(mds.1): 2 slow requests are blocked > 180 secs 2023-03-10T01:20:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:00 smithi038 ceph-mon[92422]: [WRN] MDS_TRIM: 2 MDSs behind on trimming 2023-03-10T01:20:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:00 smithi038 ceph-mon[92422]: mds.e(mds.0): Behind on trimming (1283/128) max_segments: 128, num_segments: 1283 2023-03-10T01:20:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:00 smithi038 ceph-mon[92422]: mds.b(mds.0): Behind on trimming (1282/128) max_segments: 128, num_segments: 1282 2023-03-10T01:20:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37311.516425 secs 2023-03-10T01:20:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:20:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:20:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:01 smithi122 ceph-mon[93869]: pgmap v20409: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 795 KiB/s rd, 616 KiB/s wr, 13 op/s 2023-03-10T01:20:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37311.516425 secs 2023-03-10T01:20:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:20:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:20:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:01 smithi181 ceph-mon[94189]: pgmap v20409: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 795 KiB/s rd, 616 KiB/s wr, 13 op/s 2023-03-10T01:20:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37311.516425 secs 2023-03-10T01:20:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:20:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:20:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:01 smithi038 ceph-mon[92422]: pgmap v20409: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 795 KiB/s rd, 616 KiB/s wr, 13 op/s 2023-03-10T01:20:03.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37313.979285 secs 2023-03-10T01:20:03.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37313.979285 secs 2023-03-10T01:20:03.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37313.979285 secs 2023-03-10T01:20:03.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37775.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:20:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:03 smithi122 ceph-mon[93869]: pgmap v20410: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 731 KiB/s wr, 13 op/s 2023-03-10T01:20:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:03 smithi181 ceph-mon[94189]: pgmap v20410: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 731 KiB/s wr, 13 op/s 2023-03-10T01:20:04.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:03 smithi038 ceph-mon[92422]: pgmap v20410: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 731 KiB/s wr, 13 op/s 2023-03-10T01:20:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:05 smithi181 ceph-mon[94189]: pgmap v20411: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 742 KiB/s wr, 24 op/s 2023-03-10T01:20:05.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:05 smithi038 ceph-mon[92422]: pgmap v20411: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 742 KiB/s wr, 24 op/s 2023-03-10T01:20:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:05 smithi122 ceph-mon[93869]: pgmap v20411: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 742 KiB/s wr, 24 op/s 2023-03-10T01:20:06.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37316.516486 secs 2023-03-10T01:20:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37316.516486 secs 2023-03-10T01:20:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37316.516486 secs 2023-03-10T01:20:07.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:07 smithi038 ceph-mon[92422]: pgmap v20412: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1016 KiB/s wr, 27 op/s 2023-03-10T01:20:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:07 smithi122 ceph-mon[93869]: pgmap v20412: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1016 KiB/s wr, 27 op/s 2023-03-10T01:20:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:07 smithi181 ceph-mon[94189]: pgmap v20412: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1016 KiB/s wr, 27 op/s 2023-03-10T01:20:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:20:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:20:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37318.979343 secs 2023-03-10T01:20:08.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37780.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:20:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:20:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:20:08.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37318.979343 secs 2023-03-10T01:20:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:20:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:20:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37318.979343 secs 2023-03-10T01:20:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:20:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:20:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:09 smithi122 ceph-mon[93869]: pgmap v20413: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 850 KiB/s wr, 25 op/s 2023-03-10T01:20:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:20:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:20:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:09 smithi181 ceph-mon[94189]: pgmap v20413: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 850 KiB/s wr, 25 op/s 2023-03-10T01:20:09.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:20:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:20:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:09 smithi038 ceph-mon[92422]: pgmap v20413: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 850 KiB/s wr, 25 op/s 2023-03-10T01:20:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37321.516572 secs 2023-03-10T01:20:11.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37321.516572 secs 2023-03-10T01:20:11.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37321.516572 secs 2023-03-10T01:20:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:11 smithi122 ceph-mon[93869]: pgmap v20414: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 775 KiB/s wr, 25 op/s 2023-03-10T01:20:12.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:11 smithi181 ceph-mon[94189]: pgmap v20414: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 775 KiB/s wr, 25 op/s 2023-03-10T01:20:12.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:11 smithi038 ceph-mon[92422]: pgmap v20414: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 775 KiB/s wr, 25 op/s 2023-03-10T01:20:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37323.979403 secs 2023-03-10T01:20:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37323.979403 secs 2023-03-10T01:20:13.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37323.979403 secs 2023-03-10T01:20:13.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37785.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:20:14.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:13 smithi038 ceph-mon[92422]: pgmap v20415: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 984 KiB/s rd, 768 KiB/s wr, 23 op/s 2023-03-10T01:20:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:13 smithi122 ceph-mon[93869]: pgmap v20415: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 984 KiB/s rd, 768 KiB/s wr, 23 op/s 2023-03-10T01:20:14.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:13 smithi181 ceph-mon[94189]: pgmap v20415: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 984 KiB/s rd, 768 KiB/s wr, 23 op/s 2023-03-10T01:20:15.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:15 smithi038 ceph-mon[92422]: pgmap v20416: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 494 KiB/s rd, 496 KiB/s wr, 20 op/s 2023-03-10T01:20:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:15 smithi122 ceph-mon[93869]: pgmap v20416: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 494 KiB/s rd, 496 KiB/s wr, 20 op/s 2023-03-10T01:20:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:15 smithi181 ceph-mon[94189]: pgmap v20416: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 494 KiB/s rd, 496 KiB/s wr, 20 op/s 2023-03-10T01:20:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37326.516632 secs 2023-03-10T01:20:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:20:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:20:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37326.516632 secs 2023-03-10T01:20:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:20:16.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:20:16.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37326.516632 secs 2023-03-10T01:20:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:20:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:20:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:20:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:20:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:17 smithi122 ceph-mon[93869]: pgmap v20417: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 638 KiB/s rd, 636 KiB/s wr, 12 op/s 2023-03-10T01:20:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:20:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:20:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:17 smithi181 ceph-mon[94189]: pgmap v20417: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 638 KiB/s rd, 636 KiB/s wr, 12 op/s 2023-03-10T01:20:17.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:20:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:20:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:17 smithi038 ceph-mon[92422]: pgmap v20417: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 638 KiB/s rd, 636 KiB/s wr, 12 op/s 2023-03-10T01:20:18.793 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37790.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:20:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37328.979489 secs 2023-03-10T01:20:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37328.979489 secs 2023-03-10T01:20:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37328.979489 secs 2023-03-10T01:20:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:19 smithi122 ceph-mon[93869]: pgmap v20418: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 373 KiB/s wr, 10 op/s 2023-03-10T01:20:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:19 smithi181 ceph-mon[94189]: pgmap v20418: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 373 KiB/s wr, 10 op/s 2023-03-10T01:20:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:19 smithi038 ceph-mon[92422]: pgmap v20418: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 373 KiB/s wr, 10 op/s 2023-03-10T01:20:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:20:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:20:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:20:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:20:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:20:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:20:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37331.516719 secs 2023-03-10T01:20:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:20:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:20:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:21 smithi122 ceph-mon[93869]: pgmap v20419: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 836 KiB/s rd, 885 KiB/s wr, 14 op/s 2023-03-10T01:20:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37331.516719 secs 2023-03-10T01:20:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:20:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:20:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:21 smithi181 ceph-mon[94189]: pgmap v20419: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 836 KiB/s rd, 885 KiB/s wr, 14 op/s 2023-03-10T01:20:21.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37331.516719 secs 2023-03-10T01:20:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:20:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:20:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:21 smithi038 ceph-mon[92422]: pgmap v20419: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 836 KiB/s rd, 885 KiB/s wr, 14 op/s 2023-03-10T01:20:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37333.979552 secs 2023-03-10T01:20:23.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37333.979552 secs 2023-03-10T01:20:23.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37333.979552 secs 2023-03-10T01:20:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37795.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:20:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:23 smithi038 ceph-mon[92422]: pgmap v20420: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 691 KiB/s rd, 710 KiB/s wr, 12 op/s 2023-03-10T01:20:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:23 smithi122 ceph-mon[93869]: pgmap v20420: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 691 KiB/s rd, 710 KiB/s wr, 12 op/s 2023-03-10T01:20:24.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:23 smithi181 ceph-mon[94189]: pgmap v20420: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 691 KiB/s rd, 710 KiB/s wr, 12 op/s 2023-03-10T01:20:26.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:25 smithi038 ceph-mon[92422]: pgmap v20421: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 686 KiB/s rd, 705 KiB/s wr, 11 op/s 2023-03-10T01:20:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:25 smithi038 ceph-mon[92422]: osdmap e1491: 12 total, 12 up, 12 in 2023-03-10T01:20:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37336.516808 secs 2023-03-10T01:20:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:25 smithi122 ceph-mon[93869]: pgmap v20421: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 686 KiB/s rd, 705 KiB/s wr, 11 op/s 2023-03-10T01:20:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:25 smithi122 ceph-mon[93869]: osdmap e1491: 12 total, 12 up, 12 in 2023-03-10T01:20:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37336.516808 secs 2023-03-10T01:20:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:25 smithi181 ceph-mon[94189]: pgmap v20421: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 686 KiB/s rd, 705 KiB/s wr, 11 op/s 2023-03-10T01:20:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:25 smithi181 ceph-mon[94189]: osdmap e1491: 12 total, 12 up, 12 in 2023-03-10T01:20:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37336.516808 secs 2023-03-10T01:20:28.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:27 smithi038 ceph-mon[92422]: pgmap v20423: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 786 KiB/s rd, 808 KiB/s wr, 14 op/s 2023-03-10T01:20:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:27 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37338.979610 secs 2023-03-10T01:20:28.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:27 smithi122 ceph-mon[93869]: pgmap v20423: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 786 KiB/s rd, 808 KiB/s wr, 14 op/s 2023-03-10T01:20:28.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:27 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37338.979610 secs 2023-03-10T01:20:28.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:27 smithi181 ceph-mon[94189]: pgmap v20423: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 786 KiB/s rd, 808 KiB/s wr, 14 op/s 2023-03-10T01:20:28.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:27 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37338.979610 secs 2023-03-10T01:20:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37800.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:20:30.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:29 smithi038 ceph-mon[92422]: pgmap v20424: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 821 KiB/s rd, 795 KiB/s wr, 13 op/s 2023-03-10T01:20:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:29 smithi122 ceph-mon[93869]: pgmap v20424: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 821 KiB/s rd, 795 KiB/s wr, 13 op/s 2023-03-10T01:20:30.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:29 smithi181 ceph-mon[94189]: pgmap v20424: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 821 KiB/s rd, 795 KiB/s wr, 13 op/s 2023-03-10T01:20:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37341.516883 secs 2023-03-10T01:20:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37341.516883 secs 2023-03-10T01:20:31.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37341.516883 secs 2023-03-10T01:20:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:32 smithi122 ceph-mon[93869]: pgmap v20425: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 218 KiB/s wr, 10 op/s 2023-03-10T01:20:32.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:32 smithi181 ceph-mon[94189]: pgmap v20425: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 218 KiB/s wr, 10 op/s 2023-03-10T01:20:32.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:32 smithi038 ceph-mon[92422]: pgmap v20425: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 218 KiB/s wr, 10 op/s 2023-03-10T01:20:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:33 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:20:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:33 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:20:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37343.979673 secs 2023-03-10T01:20:33.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:33 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:20:33.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:33 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:20:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37343.979673 secs 2023-03-10T01:20:33.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:33 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:20:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:33 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:20:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37343.979673 secs 2023-03-10T01:20:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37805.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:20:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:34 smithi122 ceph-mon[93869]: pgmap v20426: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 255 KiB/s wr, 10 op/s 2023-03-10T01:20:34.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:34 smithi181 ceph-mon[94189]: pgmap v20426: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 255 KiB/s wr, 10 op/s 2023-03-10T01:20:34.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:34 smithi038 ceph-mon[92422]: pgmap v20426: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 219 KiB/s rd, 255 KiB/s wr, 10 op/s 2023-03-10T01:20:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:35 smithi038 ceph-mon[92422]: osdmap e1492: 12 total, 12 up, 12 in 2023-03-10T01:20:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:35 smithi038 ceph-mon[92422]: pgmap v20428: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 596 KiB/s rd, 632 KiB/s wr, 11 op/s 2023-03-10T01:20:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:35 smithi122 ceph-mon[93869]: osdmap e1492: 12 total, 12 up, 12 in 2023-03-10T01:20:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:35 smithi122 ceph-mon[93869]: pgmap v20428: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 596 KiB/s rd, 632 KiB/s wr, 11 op/s 2023-03-10T01:20:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:35 smithi181 ceph-mon[94189]: osdmap e1492: 12 total, 12 up, 12 in 2023-03-10T01:20:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:35 smithi181 ceph-mon[94189]: pgmap v20428: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 596 KiB/s rd, 632 KiB/s wr, 11 op/s 2023-03-10T01:20:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37346.516943 secs 2023-03-10T01:20:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:20:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:20:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37346.516943 secs 2023-03-10T01:20:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:20:36.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:20:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37346.516943 secs 2023-03-10T01:20:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:20:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:20:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:20:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:20:37.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:37 smithi122 ceph-mon[93869]: pgmap v20429: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 822 KiB/s rd, 859 KiB/s wr, 10 op/s 2023-03-10T01:20:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:20:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:20:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:37 smithi181 ceph-mon[94189]: pgmap v20429: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 822 KiB/s rd, 859 KiB/s wr, 10 op/s 2023-03-10T01:20:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:20:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:20:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:37 smithi038 ceph-mon[92422]: pgmap v20429: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 822 KiB/s rd, 859 KiB/s wr, 10 op/s 2023-03-10T01:20:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37810.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:20:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37348.979733 secs 2023-03-10T01:20:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37348.979733 secs 2023-03-10T01:20:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37348.979733 secs 2023-03-10T01:20:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:39 smithi122 ceph-mon[93869]: pgmap v20430: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 823 KiB/s rd, 821 KiB/s wr, 11 op/s 2023-03-10T01:20:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:39 smithi181 ceph-mon[94189]: pgmap v20430: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 823 KiB/s rd, 821 KiB/s wr, 11 op/s 2023-03-10T01:20:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:39 smithi038 ceph-mon[92422]: pgmap v20430: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 823 KiB/s rd, 821 KiB/s wr, 11 op/s 2023-03-10T01:20:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37351.517002 secs 2023-03-10T01:20:41.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37351.517002 secs 2023-03-10T01:20:41.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37351.517002 secs 2023-03-10T01:20:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:41 smithi122 ceph-mon[93869]: pgmap v20431: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 786 KiB/s rd, 784 KiB/s wr, 11 op/s 2023-03-10T01:20:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:20:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:41 smithi181 ceph-mon[94189]: pgmap v20431: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 786 KiB/s rd, 784 KiB/s wr, 11 op/s 2023-03-10T01:20:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:20:42.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:41 smithi038 ceph-mon[92422]: pgmap v20431: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 786 KiB/s rd, 784 KiB/s wr, 11 op/s 2023-03-10T01:20:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:20:43.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:20:43.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:20:43.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:20:43.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37353.979894 secs 2023-03-10T01:20:43.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:20:43.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:20:43.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:20:43.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37353.979894 secs 2023-03-10T01:20:43.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:20:43.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:20:43.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:20:43.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37353.979894 secs 2023-03-10T01:20:43.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37815.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:20:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:44 smithi122 ceph-mon[93869]: pgmap v20432: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 971 KiB/s rd, 932 KiB/s wr, 12 op/s 2023-03-10T01:20:44.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:44 smithi181 ceph-mon[94189]: pgmap v20432: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 971 KiB/s rd, 932 KiB/s wr, 12 op/s 2023-03-10T01:20:44.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:44 smithi038 ceph-mon[92422]: pgmap v20432: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 971 KiB/s rd, 932 KiB/s wr, 12 op/s 2023-03-10T01:20:45.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:45 smithi038 ceph-mon[92422]: pgmap v20433: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 573 KiB/s rd, 536 KiB/s wr, 10 op/s 2023-03-10T01:20:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:45 smithi122 ceph-mon[93869]: pgmap v20433: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 573 KiB/s rd, 536 KiB/s wr, 10 op/s 2023-03-10T01:20:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:45 smithi181 ceph-mon[94189]: pgmap v20433: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 573 KiB/s rd, 536 KiB/s wr, 10 op/s 2023-03-10T01:20:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37356.517062 secs 2023-03-10T01:20:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37356.517062 secs 2023-03-10T01:20:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37356.517062 secs 2023-03-10T01:20:47.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:47 smithi038 ceph-mon[92422]: pgmap v20434: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 728 KiB/s rd, 696 KiB/s wr, 12 op/s 2023-03-10T01:20:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:47 smithi122 ceph-mon[93869]: pgmap v20434: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 728 KiB/s rd, 696 KiB/s wr, 12 op/s 2023-03-10T01:20:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:47 smithi181 ceph-mon[94189]: pgmap v20434: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 728 KiB/s rd, 696 KiB/s wr, 12 op/s 2023-03-10T01:20:48.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37820.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:20:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:20:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:20:48.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37358.979974 secs 2023-03-10T01:20:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:20:48.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:20:48.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37358.979974 secs 2023-03-10T01:20:48.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:20:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:20:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37358.979974 secs 2023-03-10T01:20:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:20:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:20:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:49 smithi122 ceph-mon[93869]: pgmap v20435: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 416 KiB/s wr, 10 op/s 2023-03-10T01:20:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:20:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:20:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:49 smithi181 ceph-mon[94189]: pgmap v20435: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 416 KiB/s wr, 10 op/s 2023-03-10T01:20:49.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:20:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:20:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:49 smithi038 ceph-mon[92422]: pgmap v20435: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 416 KiB/s wr, 10 op/s 2023-03-10T01:20:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37361.517121 secs 2023-03-10T01:20:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:20:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:20:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:51 smithi122 ceph-mon[93869]: pgmap v20436: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 979 KiB/s rd, 974 KiB/s wr, 12 op/s 2023-03-10T01:20:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37361.517121 secs 2023-03-10T01:20:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:20:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:20:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:51 smithi181 ceph-mon[94189]: pgmap v20436: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 979 KiB/s rd, 974 KiB/s wr, 12 op/s 2023-03-10T01:20:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37361.517121 secs 2023-03-10T01:20:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:20:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:20:52.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:51 smithi038 ceph-mon[92422]: pgmap v20436: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 979 KiB/s rd, 974 KiB/s wr, 12 op/s 2023-03-10T01:20:53.793 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37825.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:20:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:20:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:20:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:53 smithi122 ceph-mon[93869]: pgmap v20437: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 1009 KiB/s rd, 1006 KiB/s wr, 12 op/s 2023-03-10T01:20:53.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:53 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:20:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:20:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:20:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:53 smithi181 ceph-mon[94189]: pgmap v20437: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 1009 KiB/s rd, 1006 KiB/s wr, 12 op/s 2023-03-10T01:20:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:53 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:20:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:20:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:20:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:53 smithi038 ceph-mon[92422]: pgmap v20437: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 1009 KiB/s rd, 1006 KiB/s wr, 12 op/s 2023-03-10T01:20:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:53 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:20:54.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:54 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:20:54.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:54 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:20:54.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:54 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:20:54.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:54 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:20:54.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:54 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:20:54.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:54 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:20:54.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:54 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:20:54.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:54 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:20:54.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:54 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:20:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:55 smithi122 ceph-mon[93869]: pgmap v20438: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 11 op/s 2023-03-10T01:20:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:20:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:20:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:55 smithi181 ceph-mon[94189]: pgmap v20438: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 11 op/s 2023-03-10T01:20:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:20:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:20:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:55 smithi038 ceph-mon[92422]: pgmap v20438: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 11 op/s 2023-03-10T01:20:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:20:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:20:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37366.517180 secs 2023-03-10T01:20:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37366.517180 secs 2023-03-10T01:20:56.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37366.517180 secs 2023-03-10T01:20:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:20:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:20:57.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:57 smithi122 ceph-mon[93869]: pgmap v20439: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 15 op/s 2023-03-10T01:20:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:20:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:20:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:57 smithi181 ceph-mon[94189]: pgmap v20439: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 15 op/s 2023-03-10T01:20:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:20:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:20:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:57 smithi038 ceph-mon[92422]: pgmap v20439: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 15 op/s 2023-03-10T01:20:58.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37830.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:20:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:20:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:20:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:20:59 smithi122 ceph-mon[93869]: pgmap v20440: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 944 KiB/s rd, 1.1 MiB/s wr, 13 op/s 2023-03-10T01:20:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:20:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:20:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:20:59 smithi181 ceph-mon[94189]: pgmap v20440: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 944 KiB/s rd, 1.1 MiB/s wr, 13 op/s 2023-03-10T01:20:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:20:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:20:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:20:59 smithi038 ceph-mon[92422]: pgmap v20440: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 944 KiB/s rd, 1.1 MiB/s wr, 13 op/s 2023-03-10T01:21:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37371.517241 secs 2023-03-10T01:21:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:21:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:21:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:01 smithi122 ceph-mon[93869]: pgmap v20441: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 15 op/s 2023-03-10T01:21:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37371.517241 secs 2023-03-10T01:21:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:21:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:21:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:01 smithi181 ceph-mon[94189]: pgmap v20441: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 15 op/s 2023-03-10T01:21:01.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37371.517241 secs 2023-03-10T01:21:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:21:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:21:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:01 smithi038 ceph-mon[92422]: pgmap v20441: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 15 op/s 2023-03-10T01:21:03.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37835.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:21:04.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:03 smithi038 ceph-mon[92422]: pgmap v20442: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 541 KiB/s rd, 693 KiB/s wr, 12 op/s 2023-03-10T01:21:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:03 smithi122 ceph-mon[93869]: pgmap v20442: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 541 KiB/s rd, 693 KiB/s wr, 12 op/s 2023-03-10T01:21:04.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:03 smithi181 ceph-mon[94189]: pgmap v20442: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 541 KiB/s rd, 693 KiB/s wr, 12 op/s 2023-03-10T01:21:05.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:05 smithi038 ceph-mon[92422]: pgmap v20443: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 851 KiB/s rd, 838 KiB/s wr, 13 op/s 2023-03-10T01:21:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:05 smithi122 ceph-mon[93869]: pgmap v20443: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 851 KiB/s rd, 838 KiB/s wr, 13 op/s 2023-03-10T01:21:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:05 smithi181 ceph-mon[94189]: pgmap v20443: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 851 KiB/s rd, 838 KiB/s wr, 13 op/s 2023-03-10T01:21:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37376.517302 secs 2023-03-10T01:21:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37376.517302 secs 2023-03-10T01:21:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37376.517302 secs 2023-03-10T01:21:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:07 smithi122 ceph-mon[93869]: pgmap v20444: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 691 KiB/s rd, 698 KiB/s wr, 16 op/s 2023-03-10T01:21:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:07 smithi181 ceph-mon[94189]: pgmap v20444: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 691 KiB/s rd, 698 KiB/s wr, 16 op/s 2023-03-10T01:21:07.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:07 smithi038 ceph-mon[92422]: pgmap v20444: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 691 KiB/s rd, 698 KiB/s wr, 16 op/s 2023-03-10T01:21:08.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37840.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:21:09.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:21:09.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:21:09.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:09 smithi122 ceph-mon[93869]: pgmap v20445: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 665 KiB/s rd, 678 KiB/s wr, 23 op/s 2023-03-10T01:21:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:21:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:21:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:09 smithi181 ceph-mon[94189]: pgmap v20445: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 665 KiB/s rd, 678 KiB/s wr, 23 op/s 2023-03-10T01:21:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:21:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:21:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:09 smithi038 ceph-mon[92422]: pgmap v20445: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 665 KiB/s rd, 678 KiB/s wr, 23 op/s 2023-03-10T01:21:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37381.517362 secs 2023-03-10T01:21:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:11 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:21:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:11 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:21:11.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:11 smithi122 ceph-mon[93869]: pgmap v20446: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 965 KiB/s rd, 812 KiB/s wr, 25 op/s 2023-03-10T01:21:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37381.517362 secs 2023-03-10T01:21:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:11 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:21:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:11 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:21:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:11 smithi181 ceph-mon[94189]: pgmap v20446: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 965 KiB/s rd, 812 KiB/s wr, 25 op/s 2023-03-10T01:21:11.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37381.517362 secs 2023-03-10T01:21:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:11 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:21:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:11 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:21:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:11 smithi038 ceph-mon[92422]: pgmap v20446: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 965 KiB/s rd, 812 KiB/s wr, 25 op/s 2023-03-10T01:21:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37845.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:21:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:13 smithi038 ceph-mon[92422]: pgmap v20447: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 774 KiB/s rd, 622 KiB/s wr, 22 op/s 2023-03-10T01:21:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:13 smithi122 ceph-mon[93869]: pgmap v20447: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 774 KiB/s rd, 622 KiB/s wr, 22 op/s 2023-03-10T01:21:14.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:13 smithi181 ceph-mon[94189]: pgmap v20447: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 774 KiB/s rd, 622 KiB/s wr, 22 op/s 2023-03-10T01:21:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:15 smithi122 ceph-mon[93869]: pgmap v20448: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 774 KiB/s rd, 622 KiB/s wr, 21 op/s 2023-03-10T01:21:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:15 smithi181 ceph-mon[94189]: pgmap v20448: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 774 KiB/s rd, 622 KiB/s wr, 21 op/s 2023-03-10T01:21:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:15 smithi038 ceph-mon[92422]: pgmap v20448: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 774 KiB/s rd, 622 KiB/s wr, 21 op/s 2023-03-10T01:21:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37386.517422 secs 2023-03-10T01:21:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37386.517422 secs 2023-03-10T01:21:16.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37386.517422 secs 2023-03-10T01:21:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:17 smithi122 ceph-mon[93869]: pgmap v20449: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 583 KiB/s rd, 595 KiB/s wr, 22 op/s 2023-03-10T01:21:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:17 smithi181 ceph-mon[94189]: pgmap v20449: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 583 KiB/s rd, 595 KiB/s wr, 22 op/s 2023-03-10T01:21:17.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:17 smithi038 ceph-mon[92422]: pgmap v20449: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 583 KiB/s rd, 595 KiB/s wr, 22 op/s 2023-03-10T01:21:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37850.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:21:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:19 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:21:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:19 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:21:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:19 smithi122 ceph-mon[93869]: pgmap v20450: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 549 KiB/s rd, 552 KiB/s wr, 18 op/s 2023-03-10T01:21:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:19 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:21:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:19 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:21:19.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:19 smithi181 ceph-mon[94189]: pgmap v20450: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 549 KiB/s rd, 552 KiB/s wr, 18 op/s 2023-03-10T01:21:19.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:19 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:21:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:19 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:21:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:19 smithi038 ceph-mon[92422]: pgmap v20450: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 549 KiB/s rd, 552 KiB/s wr, 18 op/s 2023-03-10T01:21:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37391.517480 secs 2023-03-10T01:21:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:21:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:21:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:21 smithi122 ceph-mon[93869]: pgmap v20451: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 514 KiB/s rd, 511 KiB/s wr, 10 op/s 2023-03-10T01:21:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37391.517480 secs 2023-03-10T01:21:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:21:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:21:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:21 smithi181 ceph-mon[94189]: pgmap v20451: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 514 KiB/s rd, 511 KiB/s wr, 10 op/s 2023-03-10T01:21:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37391.517480 secs 2023-03-10T01:21:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:21:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:21:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:21 smithi038 ceph-mon[92422]: pgmap v20451: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 514 KiB/s rd, 511 KiB/s wr, 10 op/s 2023-03-10T01:21:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37855.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:21:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:24 smithi122 ceph-mon[93869]: pgmap v20452: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 212 KiB/s wr, 8 op/s 2023-03-10T01:21:24.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:24 smithi181 ceph-mon[94189]: pgmap v20452: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 212 KiB/s wr, 8 op/s 2023-03-10T01:21:24.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:24 smithi038 ceph-mon[92422]: pgmap v20452: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 212 KiB/s wr, 8 op/s 2023-03-10T01:21:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:26 smithi122 ceph-mon[93869]: pgmap v20453: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 459 KiB/s wr, 10 op/s 2023-03-10T01:21:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:26 smithi122 ceph-mon[93869]: osdmap e1493: 12 total, 12 up, 12 in 2023-03-10T01:21:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37396.517549 secs 2023-03-10T01:21:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:26 smithi181 ceph-mon[94189]: pgmap v20453: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 459 KiB/s wr, 10 op/s 2023-03-10T01:21:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:26 smithi181 ceph-mon[94189]: osdmap e1493: 12 total, 12 up, 12 in 2023-03-10T01:21:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37396.517549 secs 2023-03-10T01:21:26.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:26 smithi038 ceph-mon[92422]: pgmap v20453: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 459 KiB/s wr, 10 op/s 2023-03-10T01:21:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:26 smithi038 ceph-mon[92422]: osdmap e1493: 12 total, 12 up, 12 in 2023-03-10T01:21:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37396.517549 secs 2023-03-10T01:21:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:28 smithi122 ceph-mon[93869]: pgmap v20455: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 527 KiB/s rd, 550 KiB/s wr, 11 op/s 2023-03-10T01:21:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:28 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:21:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:28 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:21:28.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37398.980736 secs 2023-03-10T01:21:28.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:28 smithi181 ceph-mon[94189]: pgmap v20455: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 527 KiB/s rd, 550 KiB/s wr, 11 op/s 2023-03-10T01:21:28.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:28 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:21:28.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:28 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:21:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37398.980736 secs 2023-03-10T01:21:28.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:28 smithi038 ceph-mon[92422]: pgmap v20455: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 527 KiB/s rd, 550 KiB/s wr, 11 op/s 2023-03-10T01:21:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:28 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:21:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:28 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:21:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37398.980736 secs 2023-03-10T01:21:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37860.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:21:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:29 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:21:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:29 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:21:29.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:29 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:21:29.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:29 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:21:29.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:29 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:21:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:29 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:21:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:30 smithi122 ceph-mon[93869]: pgmap v20456: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 515 KiB/s wr, 12 op/s 2023-03-10T01:21:30.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:30 smithi181 ceph-mon[94189]: pgmap v20456: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 515 KiB/s wr, 12 op/s 2023-03-10T01:21:30.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:30 smithi038 ceph-mon[92422]: pgmap v20456: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 515 KiB/s wr, 12 op/s 2023-03-10T01:21:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37401.517606 secs 2023-03-10T01:21:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:31 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:21:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:31 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:21:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:31 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:21:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37401.517606 secs 2023-03-10T01:21:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:31 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:21:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:31 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:21:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:31 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:21:31.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37401.517606 secs 2023-03-10T01:21:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:31 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:21:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:31 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:21:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:31 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:21:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:32 smithi122 ceph-mon[93869]: pgmap v20457: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 515 KiB/s wr, 12 op/s 2023-03-10T01:21:32.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:32 smithi181 ceph-mon[94189]: pgmap v20457: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 515 KiB/s wr, 12 op/s 2023-03-10T01:21:32.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:32 smithi038 ceph-mon[92422]: pgmap v20457: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 515 KiB/s wr, 12 op/s 2023-03-10T01:21:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37403.980853 secs 2023-03-10T01:21:33.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37403.980853 secs 2023-03-10T01:21:33.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37403.980853 secs 2023-03-10T01:21:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37865.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:21:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:34 smithi122 ceph-mon[93869]: pgmap v20458: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 688 KiB/s wr, 12 op/s 2023-03-10T01:21:34.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:34 smithi181 ceph-mon[94189]: pgmap v20458: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 688 KiB/s wr, 12 op/s 2023-03-10T01:21:34.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:34 smithi038 ceph-mon[92422]: pgmap v20458: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 688 KiB/s wr, 12 op/s 2023-03-10T01:21:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:35 smithi122 ceph-mon[93869]: osdmap e1494: 12 total, 12 up, 12 in 2023-03-10T01:21:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:35 smithi122 ceph-mon[93869]: pgmap v20460: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 523 KiB/s wr, 11 op/s 2023-03-10T01:21:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:35 smithi181 ceph-mon[94189]: osdmap e1494: 12 total, 12 up, 12 in 2023-03-10T01:21:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:35 smithi181 ceph-mon[94189]: pgmap v20460: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 523 KiB/s wr, 11 op/s 2023-03-10T01:21:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:35 smithi038 ceph-mon[92422]: osdmap e1494: 12 total, 12 up, 12 in 2023-03-10T01:21:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:35 smithi038 ceph-mon[92422]: pgmap v20460: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 320 KiB/s rd, 523 KiB/s wr, 11 op/s 2023-03-10T01:21:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37406.517665 secs 2023-03-10T01:21:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37406.517665 secs 2023-03-10T01:21:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37406.517665 secs 2023-03-10T01:21:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:21:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:21:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:37 smithi122 ceph-mon[93869]: pgmap v20461: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 341 KiB/s wr, 10 op/s 2023-03-10T01:21:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:21:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:21:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:37 smithi181 ceph-mon[94189]: pgmap v20461: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 341 KiB/s wr, 10 op/s 2023-03-10T01:21:37.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:21:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:21:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:37 smithi038 ceph-mon[92422]: pgmap v20461: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 341 KiB/s wr, 10 op/s 2023-03-10T01:21:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37870.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:21:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37408.980915 secs 2023-03-10T01:21:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37408.980915 secs 2023-03-10T01:21:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37408.980915 secs 2023-03-10T01:21:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:39 smithi122 ceph-mon[93869]: pgmap v20462: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 526 KiB/s rd, 525 KiB/s wr, 9 op/s 2023-03-10T01:21:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:39 smithi181 ceph-mon[94189]: pgmap v20462: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 526 KiB/s rd, 525 KiB/s wr, 9 op/s 2023-03-10T01:21:39.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:39 smithi038 ceph-mon[92422]: pgmap v20462: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 526 KiB/s rd, 525 KiB/s wr, 9 op/s 2023-03-10T01:21:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:21:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:21:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:21:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:21:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:21:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:21:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37411.517754 secs 2023-03-10T01:21:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:41 smithi122 ceph-mon[93869]: pgmap v20463: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 692 KiB/s rd, 690 KiB/s wr, 9 op/s 2023-03-10T01:21:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37411.517754 secs 2023-03-10T01:21:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:41 smithi181 ceph-mon[94189]: pgmap v20463: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 692 KiB/s rd, 690 KiB/s wr, 9 op/s 2023-03-10T01:21:41.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37411.517754 secs 2023-03-10T01:21:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:41 smithi038 ceph-mon[92422]: pgmap v20463: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 692 KiB/s rd, 690 KiB/s wr, 9 op/s 2023-03-10T01:21:42.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:21:42.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:21:42.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:21:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37875.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:21:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37413.980973 secs 2023-03-10T01:21:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:21:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:21:43.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:43 smithi122 ceph-mon[93869]: pgmap v20464: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 684 KiB/s rd, 493 KiB/s wr, 8 op/s 2023-03-10T01:21:43.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:21:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37413.980973 secs 2023-03-10T01:21:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:21:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:21:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:43 smithi181 ceph-mon[94189]: pgmap v20464: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 684 KiB/s rd, 493 KiB/s wr, 8 op/s 2023-03-10T01:21:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:21:43.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37413.980973 secs 2023-03-10T01:21:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:21:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:21:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:43 smithi038 ceph-mon[92422]: pgmap v20464: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 684 KiB/s rd, 493 KiB/s wr, 8 op/s 2023-03-10T01:21:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:21:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:45 smithi122 ceph-mon[93869]: pgmap v20465: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 350 KiB/s wr, 7 op/s 2023-03-10T01:21:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:45 smithi181 ceph-mon[94189]: pgmap v20465: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 350 KiB/s wr, 7 op/s 2023-03-10T01:21:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:45 smithi038 ceph-mon[92422]: pgmap v20465: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 350 KiB/s wr, 7 op/s 2023-03-10T01:21:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37416.517812 secs 2023-03-10T01:21:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37416.517812 secs 2023-03-10T01:21:46.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37416.517812 secs 2023-03-10T01:21:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:47 smithi122 ceph-mon[93869]: pgmap v20466: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 464 KiB/s wr, 9 op/s 2023-03-10T01:21:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:47 smithi181 ceph-mon[94189]: pgmap v20466: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 464 KiB/s wr, 9 op/s 2023-03-10T01:21:47.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:47 smithi038 ceph-mon[92422]: pgmap v20466: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 464 KiB/s wr, 9 op/s 2023-03-10T01:21:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37880.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:21:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:21:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:21:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:49 smithi122 ceph-mon[93869]: pgmap v20467: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 463 KiB/s wr, 7 op/s 2023-03-10T01:21:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:21:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:21:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:49 smithi181 ceph-mon[94189]: pgmap v20467: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 463 KiB/s wr, 7 op/s 2023-03-10T01:21:49.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:21:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:21:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:49 smithi038 ceph-mon[92422]: pgmap v20467: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 463 KiB/s wr, 7 op/s 2023-03-10T01:21:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:50 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:21:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:50 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:21:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:50 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:21:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37421.517895 secs 2023-03-10T01:21:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:51 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:21:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:21:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:21:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:51 smithi122 ceph-mon[93869]: pgmap v20468: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 368 KiB/s wr, 9 op/s 2023-03-10T01:21:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37421.517895 secs 2023-03-10T01:21:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:51 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:21:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:21:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:21:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:51 smithi181 ceph-mon[94189]: pgmap v20468: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 368 KiB/s wr, 9 op/s 2023-03-10T01:21:51.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37421.517895 secs 2023-03-10T01:21:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:51 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:21:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:21:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:21:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:51 smithi038 ceph-mon[92422]: pgmap v20468: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 368 KiB/s wr, 9 op/s 2023-03-10T01:21:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:21:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:21:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:21:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:21:52.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:21:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:21:53.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37885.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:21:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:53 smithi122 ceph-mon[93869]: pgmap v20469: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 364 KiB/s wr, 8 op/s 2023-03-10T01:21:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:53 smithi181 ceph-mon[94189]: pgmap v20469: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 364 KiB/s wr, 8 op/s 2023-03-10T01:21:53.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:53 smithi038 ceph-mon[92422]: pgmap v20469: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 364 KiB/s wr, 8 op/s 2023-03-10T01:21:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:55 smithi122 ceph-mon[93869]: pgmap v20470: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 352 KiB/s wr, 7 op/s 2023-03-10T01:21:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:55 smithi181 ceph-mon[94189]: pgmap v20470: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 352 KiB/s wr, 7 op/s 2023-03-10T01:21:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:55 smithi038 ceph-mon[92422]: pgmap v20470: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 352 KiB/s wr, 7 op/s 2023-03-10T01:21:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37426.517989 secs 2023-03-10T01:21:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:21:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:21:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37426.517989 secs 2023-03-10T01:21:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:21:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:21:56.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37426.517989 secs 2023-03-10T01:21:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:21:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:21:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:21:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:21:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:57 smithi122 ceph-mon[93869]: pgmap v20471: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 420 KiB/s wr, 12 op/s 2023-03-10T01:21:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:21:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:21:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:57 smithi181 ceph-mon[94189]: pgmap v20471: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 420 KiB/s wr, 12 op/s 2023-03-10T01:21:57.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:21:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:21:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:57 smithi038 ceph-mon[92422]: pgmap v20471: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 420 KiB/s wr, 12 op/s 2023-03-10T01:21:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37890.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:21:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37428.981201 secs 2023-03-10T01:21:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37428.981201 secs 2023-03-10T01:21:58.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37428.981201 secs 2023-03-10T01:21:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:59 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:21:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:21:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:21:59.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:21:59 smithi122 ceph-mon[93869]: pgmap v20472: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 424 KiB/s wr, 10 op/s 2023-03-10T01:21:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:59 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:21:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:21:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:21:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:21:59 smithi181 ceph-mon[94189]: pgmap v20472: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 424 KiB/s wr, 10 op/s 2023-03-10T01:21:59.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:59 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:21:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:21:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:21:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:21:59 smithi038 ceph-mon[92422]: pgmap v20472: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 424 KiB/s wr, 10 op/s 2023-03-10T01:22:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37431.518059 secs 2023-03-10T01:22:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:22:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:22:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:01 smithi122 ceph-mon[93869]: pgmap v20473: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 535 KiB/s wr, 15 op/s 2023-03-10T01:22:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37431.518059 secs 2023-03-10T01:22:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:22:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:22:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:01 smithi181 ceph-mon[94189]: pgmap v20473: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 535 KiB/s wr, 15 op/s 2023-03-10T01:22:01.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37431.518059 secs 2023-03-10T01:22:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:22:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:22:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:01 smithi038 ceph-mon[92422]: pgmap v20473: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 377 KiB/s rd, 535 KiB/s wr, 15 op/s 2023-03-10T01:22:03.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37433.981258 secs 2023-03-10T01:22:03.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37433.981258 secs 2023-03-10T01:22:03.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37433.981258 secs 2023-03-10T01:22:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37895.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:22:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:04 smithi122 ceph-mon[93869]: pgmap v20474: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 505 KiB/s rd, 664 KiB/s wr, 13 op/s 2023-03-10T01:22:04.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:04 smithi181 ceph-mon[94189]: pgmap v20474: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 505 KiB/s rd, 664 KiB/s wr, 13 op/s 2023-03-10T01:22:04.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:04 smithi038 ceph-mon[92422]: pgmap v20474: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 505 KiB/s rd, 664 KiB/s wr, 13 op/s 2023-03-10T01:22:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:05 smithi122 ceph-mon[93869]: pgmap v20475: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 719 KiB/s rd, 719 KiB/s wr, 24 op/s 2023-03-10T01:22:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:05 smithi181 ceph-mon[94189]: pgmap v20475: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 719 KiB/s rd, 719 KiB/s wr, 24 op/s 2023-03-10T01:22:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:05 smithi038 ceph-mon[92422]: pgmap v20475: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 719 KiB/s rd, 719 KiB/s wr, 24 op/s 2023-03-10T01:22:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37436.518115 secs 2023-03-10T01:22:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37436.518115 secs 2023-03-10T01:22:06.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37436.518115 secs 2023-03-10T01:22:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:07 smithi122 ceph-mon[93869]: pgmap v20476: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 28 op/s 2023-03-10T01:22:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:07 smithi181 ceph-mon[94189]: pgmap v20476: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 28 op/s 2023-03-10T01:22:07.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:07 smithi038 ceph-mon[92422]: pgmap v20476: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 28 op/s 2023-03-10T01:22:08.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37900.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:22:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:22:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:22:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37438.981316 secs 2023-03-10T01:22:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:22:08.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:22:08.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37438.981316 secs 2023-03-10T01:22:08.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:22:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:22:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37438.981316 secs 2023-03-10T01:22:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:22:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:22:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:09 smithi122 ceph-mon[93869]: pgmap v20477: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 957 KiB/s rd, 959 KiB/s wr, 24 op/s 2023-03-10T01:22:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:22:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:22:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:09 smithi181 ceph-mon[94189]: pgmap v20477: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 957 KiB/s rd, 959 KiB/s wr, 24 op/s 2023-03-10T01:22:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:22:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:22:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:09 smithi038 ceph-mon[92422]: pgmap v20477: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 957 KiB/s rd, 959 KiB/s wr, 24 op/s 2023-03-10T01:22:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37441.518174 secs 2023-03-10T01:22:11.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37441.518174 secs 2023-03-10T01:22:11.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37441.518174 secs 2023-03-10T01:22:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:11 smithi122 ceph-mon[93869]: pgmap v20478: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 972 KiB/s wr, 26 op/s 2023-03-10T01:22:12.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:11 smithi181 ceph-mon[94189]: pgmap v20478: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 972 KiB/s wr, 26 op/s 2023-03-10T01:22:12.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:11 smithi038 ceph-mon[92422]: pgmap v20478: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 972 KiB/s wr, 26 op/s 2023-03-10T01:22:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37443.981376 secs 2023-03-10T01:22:13.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37443.981376 secs 2023-03-10T01:22:13.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37443.981376 secs 2023-03-10T01:22:13.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37905.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:22:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:13 smithi122 ceph-mon[93869]: pgmap v20479: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 1022 KiB/s rd, 861 KiB/s wr, 21 op/s 2023-03-10T01:22:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:13 smithi181 ceph-mon[94189]: pgmap v20479: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 1022 KiB/s rd, 861 KiB/s wr, 21 op/s 2023-03-10T01:22:14.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:13 smithi038 ceph-mon[92422]: pgmap v20479: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 1022 KiB/s rd, 861 KiB/s wr, 21 op/s 2023-03-10T01:22:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:15 smithi122 ceph-mon[93869]: pgmap v20480: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 834 KiB/s rd, 672 KiB/s wr, 20 op/s 2023-03-10T01:22:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:15 smithi181 ceph-mon[94189]: pgmap v20480: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 834 KiB/s rd, 672 KiB/s wr, 20 op/s 2023-03-10T01:22:15.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:15 smithi038 ceph-mon[92422]: pgmap v20480: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 834 KiB/s rd, 672 KiB/s wr, 20 op/s 2023-03-10T01:22:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37446.518232 secs 2023-03-10T01:22:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:22:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:22:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37446.518232 secs 2023-03-10T01:22:16.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:22:16.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:22:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37446.518232 secs 2023-03-10T01:22:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:22:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:22:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:22:17.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:22:17.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:17 smithi122 ceph-mon[93869]: pgmap v20481: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 484 KiB/s wr, 11 op/s 2023-03-10T01:22:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:22:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:22:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:17 smithi181 ceph-mon[94189]: pgmap v20481: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 484 KiB/s wr, 11 op/s 2023-03-10T01:22:17.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:22:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:22:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:17 smithi038 ceph-mon[92422]: pgmap v20481: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 484 KiB/s wr, 11 op/s 2023-03-10T01:22:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37910.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:22:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37448.981472 secs 2023-03-10T01:22:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37448.981472 secs 2023-03-10T01:22:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37448.981472 secs 2023-03-10T01:22:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:19 smithi122 ceph-mon[93869]: pgmap v20482: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 187 KiB/s wr, 9 op/s 2023-03-10T01:22:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:19 smithi181 ceph-mon[94189]: pgmap v20482: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 187 KiB/s wr, 9 op/s 2023-03-10T01:22:19.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:19 smithi038 ceph-mon[92422]: pgmap v20482: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 187 KiB/s wr, 9 op/s 2023-03-10T01:22:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:22:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:22:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:22:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:22:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:22:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:22:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37451.518288 secs 2023-03-10T01:22:21.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:22:21.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:22:21.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:21 smithi122 ceph-mon[93869]: pgmap v20483: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 565 KiB/s rd, 583 KiB/s wr, 13 op/s 2023-03-10T01:22:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37451.518288 secs 2023-03-10T01:22:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:22:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:22:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:21 smithi181 ceph-mon[94189]: pgmap v20483: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 565 KiB/s rd, 583 KiB/s wr, 13 op/s 2023-03-10T01:22:21.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37451.518288 secs 2023-03-10T01:22:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:22:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:22:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:21 smithi038 ceph-mon[92422]: pgmap v20483: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 565 KiB/s rd, 583 KiB/s wr, 13 op/s 2023-03-10T01:22:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37453.981531 secs 2023-03-10T01:22:23.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37453.981531 secs 2023-03-10T01:22:23.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37453.981531 secs 2023-03-10T01:22:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37915.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:22:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:24 smithi122 ceph-mon[93869]: pgmap v20484: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 407 KiB/s wr, 10 op/s 2023-03-10T01:22:24.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:24 smithi181 ceph-mon[94189]: pgmap v20484: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 407 KiB/s wr, 10 op/s 2023-03-10T01:22:24.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:24 smithi038 ceph-mon[92422]: pgmap v20484: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 407 KiB/s wr, 10 op/s 2023-03-10T01:22:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:26 smithi122 ceph-mon[93869]: pgmap v20485: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 408 KiB/s wr, 10 op/s 2023-03-10T01:22:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:26 smithi122 ceph-mon[93869]: osdmap e1495: 12 total, 12 up, 12 in 2023-03-10T01:22:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37456.521903 secs 2023-03-10T01:22:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:26 smithi181 ceph-mon[94189]: pgmap v20485: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 408 KiB/s wr, 10 op/s 2023-03-10T01:22:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:26 smithi181 ceph-mon[94189]: osdmap e1495: 12 total, 12 up, 12 in 2023-03-10T01:22:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37456.521903 secs 2023-03-10T01:22:26.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:26 smithi038 ceph-mon[92422]: pgmap v20485: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 408 KiB/s wr, 10 op/s 2023-03-10T01:22:26.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:26 smithi038 ceph-mon[92422]: osdmap e1495: 12 total, 12 up, 12 in 2023-03-10T01:22:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37456.521903 secs 2023-03-10T01:22:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:28 smithi122 ceph-mon[93869]: pgmap v20487: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 646 KiB/s rd, 671 KiB/s wr, 14 op/s 2023-03-10T01:22:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37458.981594 secs 2023-03-10T01:22:28.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:28 smithi181 ceph-mon[94189]: pgmap v20487: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 646 KiB/s rd, 671 KiB/s wr, 14 op/s 2023-03-10T01:22:28.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37458.981594 secs 2023-03-10T01:22:28.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:28 smithi038 ceph-mon[92422]: pgmap v20487: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 646 KiB/s rd, 671 KiB/s wr, 14 op/s 2023-03-10T01:22:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37458.981594 secs 2023-03-10T01:22:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37920.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:22:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:30 smithi122 ceph-mon[93869]: pgmap v20488: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 645 KiB/s rd, 658 KiB/s wr, 12 op/s 2023-03-10T01:22:30.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:30 smithi181 ceph-mon[94189]: pgmap v20488: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 645 KiB/s rd, 658 KiB/s wr, 12 op/s 2023-03-10T01:22:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:30 smithi038 ceph-mon[92422]: pgmap v20488: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 645 KiB/s rd, 658 KiB/s wr, 12 op/s 2023-03-10T01:22:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37461.521961 secs 2023-03-10T01:22:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37461.521961 secs 2023-03-10T01:22:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37461.521961 secs 2023-03-10T01:22:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:32 smithi122 ceph-mon[93869]: pgmap v20489: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 400 KiB/s wr, 10 op/s 2023-03-10T01:22:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:32 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:22:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:32 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:22:32.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:32 smithi181 ceph-mon[94189]: pgmap v20489: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 400 KiB/s wr, 10 op/s 2023-03-10T01:22:32.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:32 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:22:32.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:32 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:22:32.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:32 smithi038 ceph-mon[92422]: pgmap v20489: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 400 KiB/s wr, 10 op/s 2023-03-10T01:22:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:32 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:22:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:32 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:22:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:33 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:22:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:33 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:22:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37463.981679 secs 2023-03-10T01:22:33.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:33 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:22:33.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:33 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:22:33.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37463.981679 secs 2023-03-10T01:22:33.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:33 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:22:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:33 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:22:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37463.981679 secs 2023-03-10T01:22:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37925.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:22:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:34 smithi122 ceph-mon[93869]: pgmap v20490: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 400 KiB/s wr, 10 op/s 2023-03-10T01:22:34.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:34 smithi181 ceph-mon[94189]: pgmap v20490: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 400 KiB/s wr, 10 op/s 2023-03-10T01:22:34.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:34 smithi038 ceph-mon[92422]: pgmap v20490: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 401 KiB/s rd, 400 KiB/s wr, 10 op/s 2023-03-10T01:22:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:35 smithi122 ceph-mon[93869]: osdmap e1496: 12 total, 12 up, 12 in 2023-03-10T01:22:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:35 smithi122 ceph-mon[93869]: pgmap v20492: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 403 KiB/s wr, 10 op/s 2023-03-10T01:22:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:35 smithi181 ceph-mon[94189]: osdmap e1496: 12 total, 12 up, 12 in 2023-03-10T01:22:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:35 smithi181 ceph-mon[94189]: pgmap v20492: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 403 KiB/s wr, 10 op/s 2023-03-10T01:22:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:35 smithi038 ceph-mon[92422]: osdmap e1496: 12 total, 12 up, 12 in 2023-03-10T01:22:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:35 smithi038 ceph-mon[92422]: pgmap v20492: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 403 KiB/s wr, 10 op/s 2023-03-10T01:22:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37466.522021 secs 2023-03-10T01:22:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:22:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:22:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37466.522021 secs 2023-03-10T01:22:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:22:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:22:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37466.522021 secs 2023-03-10T01:22:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:22:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:22:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:22:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:22:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:37 smithi122 ceph-mon[93869]: pgmap v20493: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 861 KiB/s rd, 858 KiB/s wr, 10 op/s 2023-03-10T01:22:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:22:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:22:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:37 smithi181 ceph-mon[94189]: pgmap v20493: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 861 KiB/s rd, 858 KiB/s wr, 10 op/s 2023-03-10T01:22:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:22:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:22:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:37 smithi038 ceph-mon[92422]: pgmap v20493: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 861 KiB/s rd, 858 KiB/s wr, 10 op/s 2023-03-10T01:22:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37930.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:22:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37468.981739 secs 2023-03-10T01:22:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37468.981739 secs 2023-03-10T01:22:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37468.981739 secs 2023-03-10T01:22:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:39 smithi122 ceph-mon[93869]: pgmap v20494: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 11 op/s 2023-03-10T01:22:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:39 smithi181 ceph-mon[94189]: pgmap v20494: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 11 op/s 2023-03-10T01:22:39.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:39 smithi038 ceph-mon[92422]: pgmap v20494: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 11 op/s 2023-03-10T01:22:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:22:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:22:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:22:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:22:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:22:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:22:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37471.522079 secs 2023-03-10T01:22:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:22:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:22:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:41 smithi122 ceph-mon[93869]: pgmap v20495: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 855 KiB/s rd, 852 KiB/s wr, 11 op/s 2023-03-10T01:22:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37471.522079 secs 2023-03-10T01:22:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:22:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:22:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:41 smithi181 ceph-mon[94189]: pgmap v20495: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 855 KiB/s rd, 852 KiB/s wr, 11 op/s 2023-03-10T01:22:41.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37471.522079 secs 2023-03-10T01:22:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:22:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:22:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:41 smithi038 ceph-mon[92422]: pgmap v20495: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 855 KiB/s rd, 852 KiB/s wr, 11 op/s 2023-03-10T01:22:43.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37473.981799 secs 2023-03-10T01:22:43.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37473.981799 secs 2023-03-10T01:22:43.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37473.981799 secs 2023-03-10T01:22:43.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37935.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:22:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:43 smithi181 ceph-mon[94189]: pgmap v20496: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 855 KiB/s rd, 853 KiB/s wr, 11 op/s 2023-03-10T01:22:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:22:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:22:44.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:22:44.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:22:44.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:43 smithi038 ceph-mon[92422]: pgmap v20496: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 855 KiB/s rd, 853 KiB/s wr, 11 op/s 2023-03-10T01:22:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:22:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:22:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:22:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:22:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:43 smithi122 ceph-mon[93869]: pgmap v20496: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 855 KiB/s rd, 853 KiB/s wr, 11 op/s 2023-03-10T01:22:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:22:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:22:44.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:22:44.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:22:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:45 smithi122 ceph-mon[93869]: pgmap v20497: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 1008 KiB/s rd, 1006 KiB/s wr, 11 op/s 2023-03-10T01:22:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:45 smithi181 ceph-mon[94189]: pgmap v20497: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 1008 KiB/s rd, 1006 KiB/s wr, 11 op/s 2023-03-10T01:22:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:45 smithi038 ceph-mon[92422]: pgmap v20497: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 1008 KiB/s rd, 1006 KiB/s wr, 11 op/s 2023-03-10T01:22:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37476.522139 secs 2023-03-10T01:22:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37476.522139 secs 2023-03-10T01:22:46.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37476.522139 secs 2023-03-10T01:22:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:47 smithi122 ceph-mon[93869]: pgmap v20498: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 14 op/s 2023-03-10T01:22:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:47 smithi181 ceph-mon[94189]: pgmap v20498: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 14 op/s 2023-03-10T01:22:47.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:47 smithi038 ceph-mon[92422]: pgmap v20498: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 14 op/s 2023-03-10T01:22:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37940.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:22:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:22:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:22:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:22:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:22:48.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:22:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:22:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:22:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:22:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:49 smithi122 ceph-mon[93869]: pgmap v20499: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 723 KiB/s rd, 720 KiB/s wr, 11 op/s 2023-03-10T01:22:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:22:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:22:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:49 smithi181 ceph-mon[94189]: pgmap v20499: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 723 KiB/s rd, 720 KiB/s wr, 11 op/s 2023-03-10T01:22:49.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:22:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:22:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:49 smithi038 ceph-mon[92422]: pgmap v20499: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 723 KiB/s rd, 720 KiB/s wr, 11 op/s 2023-03-10T01:22:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:50 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:22:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:50 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:22:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:50 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:22:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37481.522199 secs 2023-03-10T01:22:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:51 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:22:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:22:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:22:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:51 smithi122 ceph-mon[93869]: pgmap v20500: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 947 KiB/s rd, 747 KiB/s wr, 13 op/s 2023-03-10T01:22:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37481.522199 secs 2023-03-10T01:22:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:51 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:22:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:22:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:22:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:51 smithi181 ceph-mon[94189]: pgmap v20500: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 947 KiB/s rd, 747 KiB/s wr, 13 op/s 2023-03-10T01:22:51.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37481.522199 secs 2023-03-10T01:22:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:51 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:22:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:22:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:22:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:51 smithi038 ceph-mon[92422]: pgmap v20500: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 947 KiB/s rd, 747 KiB/s wr, 13 op/s 2023-03-10T01:22:53.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37945.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:22:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:54 smithi122 ceph-mon[93869]: pgmap v20501: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 945 KiB/s rd, 747 KiB/s wr, 10 op/s 2023-03-10T01:22:54.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:54 smithi181 ceph-mon[94189]: pgmap v20501: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 945 KiB/s rd, 747 KiB/s wr, 10 op/s 2023-03-10T01:22:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:54 smithi038 ceph-mon[92422]: pgmap v20501: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 945 KiB/s rd, 747 KiB/s wr, 10 op/s 2023-03-10T01:22:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:55 smithi122 ceph-mon[93869]: pgmap v20502: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 945 KiB/s rd, 747 KiB/s wr, 11 op/s 2023-03-10T01:22:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:55 smithi181 ceph-mon[94189]: pgmap v20502: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 945 KiB/s rd, 747 KiB/s wr, 11 op/s 2023-03-10T01:22:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:55 smithi038 ceph-mon[92422]: pgmap v20502: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 945 KiB/s rd, 747 KiB/s wr, 11 op/s 2023-03-10T01:22:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37486.522261 secs 2023-03-10T01:22:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:22:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:22:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37486.522261 secs 2023-03-10T01:22:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:22:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:22:56.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37486.522261 secs 2023-03-10T01:22:56.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:22:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:22:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:22:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:22:57.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:22:57 smithi122 ceph-mon[93869]: pgmap v20503: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 976 KiB/s wr, 15 op/s 2023-03-10T01:22:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:22:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:22:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:22:57 smithi181 ceph-mon[94189]: pgmap v20503: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 976 KiB/s wr, 15 op/s 2023-03-10T01:22:57.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:22:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:22:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:22:57 smithi038 ceph-mon[92422]: pgmap v20503: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 976 KiB/s wr, 15 op/s 2023-03-10T01:22:58.793 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37950.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:23:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:00 smithi122 ceph-mon[93869]: pgmap v20504: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 795 KiB/s rd, 599 KiB/s wr, 11 op/s 2023-03-10T01:23:00.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:00 smithi181 ceph-mon[94189]: pgmap v20504: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 795 KiB/s rd, 599 KiB/s wr, 11 op/s 2023-03-10T01:23:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:00 smithi038 ceph-mon[92422]: pgmap v20504: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 795 KiB/s rd, 599 KiB/s wr, 11 op/s 2023-03-10T01:23:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37491.522346 secs 2023-03-10T01:23:01.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37491.522346 secs 2023-03-10T01:23:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37491.522346 secs 2023-03-10T01:23:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:02 smithi122 ceph-mon[93869]: pgmap v20505: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 972 KiB/s rd, 774 KiB/s wr, 14 op/s 2023-03-10T01:23:02.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:02 smithi181 ceph-mon[94189]: pgmap v20505: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 972 KiB/s rd, 774 KiB/s wr, 14 op/s 2023-03-10T01:23:02.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:02 smithi038 ceph-mon[92422]: pgmap v20505: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 972 KiB/s rd, 774 KiB/s wr, 14 op/s 2023-03-10T01:23:03.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37955.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:23:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:04 smithi122 ceph-mon[93869]: pgmap v20506: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 711 KiB/s wr, 13 op/s 2023-03-10T01:23:04.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:04 smithi181 ceph-mon[94189]: pgmap v20506: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 711 KiB/s wr, 13 op/s 2023-03-10T01:23:04.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:04 smithi038 ceph-mon[92422]: pgmap v20506: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 711 KiB/s wr, 13 op/s 2023-03-10T01:23:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:05 smithi122 ceph-mon[93869]: pgmap v20507: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 711 KiB/s wr, 12 op/s 2023-03-10T01:23:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:05 smithi181 ceph-mon[94189]: pgmap v20507: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 711 KiB/s wr, 12 op/s 2023-03-10T01:23:05.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:05 smithi038 ceph-mon[92422]: pgmap v20507: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 711 KiB/s wr, 12 op/s 2023-03-10T01:23:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37496.522400 secs 2023-03-10T01:23:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37496.522400 secs 2023-03-10T01:23:06.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37496.522400 secs 2023-03-10T01:23:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:07 smithi122 ceph-mon[93869]: pgmap v20508: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 747 KiB/s rd, 765 KiB/s wr, 17 op/s 2023-03-10T01:23:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:07 smithi181 ceph-mon[94189]: pgmap v20508: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 747 KiB/s rd, 765 KiB/s wr, 17 op/s 2023-03-10T01:23:07.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:07 smithi038 ceph-mon[92422]: pgmap v20508: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 747 KiB/s rd, 765 KiB/s wr, 17 op/s 2023-03-10T01:23:08.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37960.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:23:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37498.982195 secs 2023-03-10T01:23:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:23:08.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:23:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37498.982195 secs 2023-03-10T01:23:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:23:08.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:23:08.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37498.982195 secs 2023-03-10T01:23:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:23:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:23:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:23:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:23:09.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:09 smithi122 ceph-mon[93869]: pgmap v20509: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 593 KiB/s wr, 25 op/s 2023-03-10T01:23:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:23:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:23:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:09 smithi181 ceph-mon[94189]: pgmap v20509: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 593 KiB/s wr, 25 op/s 2023-03-10T01:23:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:23:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:23:10.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:09 smithi038 ceph-mon[92422]: pgmap v20509: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 593 KiB/s wr, 25 op/s 2023-03-10T01:23:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:10 smithi122 ceph-mon[93869]: osdmap e1497: 12 total, 12 up, 12 in 2023-03-10T01:23:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:10 smithi181 ceph-mon[94189]: osdmap e1497: 12 total, 12 up, 12 in 2023-03-10T01:23:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:10 smithi038 ceph-mon[92422]: osdmap e1497: 12 total, 12 up, 12 in 2023-03-10T01:23:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37501.522458 secs 2023-03-10T01:23:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:11 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:23:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:11 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:23:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:11 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:23:11.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:11 smithi122 ceph-mon[93869]: pgmap v20511: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 542 KiB/s rd, 658 KiB/s wr, 27 op/s 2023-03-10T01:23:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37501.522458 secs 2023-03-10T01:23:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:11 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:23:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:11 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:23:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:11 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:23:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:11 smithi181 ceph-mon[94189]: pgmap v20511: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 542 KiB/s rd, 658 KiB/s wr, 27 op/s 2023-03-10T01:23:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37501.522458 secs 2023-03-10T01:23:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:11 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:23:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:11 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:23:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:11 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:23:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:11 smithi038 ceph-mon[92422]: pgmap v20511: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 542 KiB/s rd, 658 KiB/s wr, 27 op/s 2023-03-10T01:23:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:23:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:23:12.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:23:12.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:23:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:23:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:23:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37965.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:23:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37503.982285 secs 2023-03-10T01:23:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:23:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:23:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:13 smithi122 ceph-mon[93869]: pgmap v20512: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 489 KiB/s wr, 26 op/s 2023-03-10T01:23:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37503.982285 secs 2023-03-10T01:23:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:23:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:23:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:13 smithi181 ceph-mon[94189]: pgmap v20512: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 489 KiB/s wr, 26 op/s 2023-03-10T01:23:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37503.982285 secs 2023-03-10T01:23:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:23:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:23:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:13 smithi038 ceph-mon[92422]: pgmap v20512: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 448 KiB/s rd, 489 KiB/s wr, 26 op/s 2023-03-10T01:23:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:15 smithi122 ceph-mon[93869]: pgmap v20513: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 656 KiB/s rd, 698 KiB/s wr, 27 op/s 2023-03-10T01:23:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:15 smithi181 ceph-mon[94189]: pgmap v20513: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 656 KiB/s rd, 698 KiB/s wr, 27 op/s 2023-03-10T01:23:15.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:15 smithi038 ceph-mon[92422]: pgmap v20513: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 656 KiB/s rd, 698 KiB/s wr, 27 op/s 2023-03-10T01:23:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37506.522516 secs 2023-03-10T01:23:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37506.522516 secs 2023-03-10T01:23:16.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37506.522516 secs 2023-03-10T01:23:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:17 smithi122 ceph-mon[93869]: pgmap v20514: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 634 KiB/s rd, 650 KiB/s wr, 24 op/s 2023-03-10T01:23:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:17 smithi181 ceph-mon[94189]: pgmap v20514: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 634 KiB/s rd, 650 KiB/s wr, 24 op/s 2023-03-10T01:23:17.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:17 smithi038 ceph-mon[92422]: pgmap v20514: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 634 KiB/s rd, 650 KiB/s wr, 24 op/s 2023-03-10T01:23:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37970.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:23:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37508.982347 secs 2023-03-10T01:23:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37508.982347 secs 2023-03-10T01:23:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37508.982347 secs 2023-03-10T01:23:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:19 smithi122 ceph-mon[93869]: pgmap v20515: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 383 KiB/s wr, 10 op/s 2023-03-10T01:23:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:19 smithi122 ceph-mon[93869]: osdmap e1498: 12 total, 12 up, 12 in 2023-03-10T01:23:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:19 smithi181 ceph-mon[94189]: pgmap v20515: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 383 KiB/s wr, 10 op/s 2023-03-10T01:23:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:19 smithi181 ceph-mon[94189]: osdmap e1498: 12 total, 12 up, 12 in 2023-03-10T01:23:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:19 smithi038 ceph-mon[92422]: pgmap v20515: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 383 KiB/s wr, 10 op/s 2023-03-10T01:23:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:19 smithi038 ceph-mon[92422]: osdmap e1498: 12 total, 12 up, 12 in 2023-03-10T01:23:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37511.522576 secs 2023-03-10T01:23:21.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37511.522576 secs 2023-03-10T01:23:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37511.522576 secs 2023-03-10T01:23:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:21 smithi122 ceph-mon[93869]: pgmap v20517: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 363 KiB/s wr, 10 op/s 2023-03-10T01:23:22.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:21 smithi181 ceph-mon[94189]: pgmap v20517: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 363 KiB/s wr, 10 op/s 2023-03-10T01:23:22.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:21 smithi038 ceph-mon[92422]: pgmap v20517: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 363 KiB/s wr, 10 op/s 2023-03-10T01:23:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37513.982436 secs 2023-03-10T01:23:23.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37513.982436 secs 2023-03-10T01:23:23.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37513.982436 secs 2023-03-10T01:23:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37975.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:23:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:23 smithi122 ceph-mon[93869]: pgmap v20518: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 384 KiB/s wr, 10 op/s 2023-03-10T01:23:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:23 smithi181 ceph-mon[94189]: pgmap v20518: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 384 KiB/s wr, 10 op/s 2023-03-10T01:23:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:23 smithi038 ceph-mon[92422]: pgmap v20518: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 384 KiB/s wr, 10 op/s 2023-03-10T01:23:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:25 smithi122 ceph-mon[93869]: pgmap v20519: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 220 KiB/s wr, 10 op/s 2023-03-10T01:23:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:25 smithi181 ceph-mon[94189]: pgmap v20519: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 220 KiB/s wr, 10 op/s 2023-03-10T01:23:25.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:25 smithi038 ceph-mon[92422]: pgmap v20519: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 220 KiB/s wr, 10 op/s 2023-03-10T01:23:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37516.522636 secs 2023-03-10T01:23:26.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37516.522636 secs 2023-03-10T01:23:26.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37516.522636 secs 2023-03-10T01:23:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:27 smithi122 ceph-mon[93869]: pgmap v20520: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 203 KiB/s wr, 10 op/s 2023-03-10T01:23:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:27 smithi181 ceph-mon[94189]: pgmap v20520: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 203 KiB/s wr, 10 op/s 2023-03-10T01:23:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:27 smithi038 ceph-mon[92422]: pgmap v20520: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 203 KiB/s wr, 10 op/s 2023-03-10T01:23:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37980.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:23:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37518.982536 secs 2023-03-10T01:23:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37518.982536 secs 2023-03-10T01:23:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37518.982536 secs 2023-03-10T01:23:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:29 smithi122 ceph-mon[93869]: pgmap v20521: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 203 KiB/s wr, 10 op/s 2023-03-10T01:23:29.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:29 smithi181 ceph-mon[94189]: pgmap v20521: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 203 KiB/s wr, 10 op/s 2023-03-10T01:23:29.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:29 smithi038 ceph-mon[92422]: pgmap v20521: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 203 KiB/s wr, 10 op/s 2023-03-10T01:23:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37521.522695 secs 2023-03-10T01:23:31.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:31 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:23:31.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:31 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:23:31.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:31 smithi122 ceph-mon[93869]: pgmap v20522: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 507 KiB/s wr, 11 op/s 2023-03-10T01:23:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37521.522695 secs 2023-03-10T01:23:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:31 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:23:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:31 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:23:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:31 smithi181 ceph-mon[94189]: pgmap v20522: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 507 KiB/s wr, 11 op/s 2023-03-10T01:23:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37521.522695 secs 2023-03-10T01:23:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:31 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:23:32.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:31 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:23:32.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:31 smithi038 ceph-mon[92422]: pgmap v20522: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 507 KiB/s wr, 11 op/s 2023-03-10T01:23:33.793 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37985.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:23:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37523.982596 secs 2023-03-10T01:23:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:33 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:23:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:33 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:23:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:33 smithi122 ceph-mon[93869]: pgmap v20523: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 369 KiB/s wr, 8 op/s 2023-03-10T01:23:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37523.982596 secs 2023-03-10T01:23:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:33 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:23:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:33 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:23:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:33 smithi181 ceph-mon[94189]: pgmap v20523: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 369 KiB/s wr, 8 op/s 2023-03-10T01:23:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37523.982596 secs 2023-03-10T01:23:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:33 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:23:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:33 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:23:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:33 smithi038 ceph-mon[92422]: pgmap v20523: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 369 KiB/s wr, 8 op/s 2023-03-10T01:23:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:35 smithi122 ceph-mon[93869]: pgmap v20524: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 352 KiB/s wr, 8 op/s 2023-03-10T01:23:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:35 smithi181 ceph-mon[94189]: pgmap v20524: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 352 KiB/s wr, 8 op/s 2023-03-10T01:23:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:35 smithi038 ceph-mon[92422]: pgmap v20524: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 352 KiB/s wr, 8 op/s 2023-03-10T01:23:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37526.522758 secs 2023-03-10T01:23:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37526.522758 secs 2023-03-10T01:23:36.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37526.522758 secs 2023-03-10T01:23:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:37 smithi122 ceph-mon[93869]: pgmap v20525: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 318 KiB/s rd, 315 KiB/s wr, 10 op/s 2023-03-10T01:23:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:37 smithi181 ceph-mon[94189]: pgmap v20525: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 318 KiB/s rd, 315 KiB/s wr, 10 op/s 2023-03-10T01:23:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:37 smithi038 ceph-mon[92422]: pgmap v20525: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 318 KiB/s rd, 315 KiB/s wr, 10 op/s 2023-03-10T01:23:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37990.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:23:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37528.982666 secs 2023-03-10T01:23:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37528.982666 secs 2023-03-10T01:23:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37528.982666 secs 2023-03-10T01:23:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:39 smithi122 ceph-mon[93869]: pgmap v20526: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 315 KiB/s wr, 7 op/s 2023-03-10T01:23:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:39 smithi181 ceph-mon[94189]: pgmap v20526: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 315 KiB/s wr, 7 op/s 2023-03-10T01:23:39.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:39 smithi038 ceph-mon[92422]: pgmap v20526: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 315 KiB/s wr, 7 op/s 2023-03-10T01:23:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:23:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:23:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:23:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:23:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:23:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:23:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37531.522837 secs 2023-03-10T01:23:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:23:41.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:23:41.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:41 smithi122 ceph-mon[93869]: pgmap v20527: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 627 KiB/s rd, 624 KiB/s wr, 10 op/s 2023-03-10T01:23:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37531.522837 secs 2023-03-10T01:23:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:23:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:23:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:41 smithi181 ceph-mon[94189]: pgmap v20527: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 627 KiB/s rd, 624 KiB/s wr, 10 op/s 2023-03-10T01:23:41.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37531.522837 secs 2023-03-10T01:23:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:23:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:23:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:41 smithi038 ceph-mon[92422]: pgmap v20527: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 627 KiB/s rd, 624 KiB/s wr, 10 op/s 2023-03-10T01:23:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37533.982723 secs 2023-03-10T01:23:43.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37533.982723 secs 2023-03-10T01:23:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37533.982723 secs 2023-03-10T01:23:43.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 37995.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:23:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:44 smithi122 ceph-mon[93869]: pgmap v20528: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 310 KiB/s wr, 8 op/s 2023-03-10T01:23:44.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:44 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:23:44.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:44 smithi181 ceph-mon[94189]: pgmap v20528: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 310 KiB/s wr, 8 op/s 2023-03-10T01:23:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:44 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:23:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:44 smithi038 ceph-mon[92422]: pgmap v20528: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 310 KiB/s wr, 8 op/s 2023-03-10T01:23:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:44 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:23:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:45 smithi122 ceph-mon[93869]: pgmap v20529: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 309 KiB/s wr, 7 op/s 2023-03-10T01:23:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:45 smithi181 ceph-mon[94189]: pgmap v20529: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 309 KiB/s wr, 7 op/s 2023-03-10T01:23:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:45 smithi038 ceph-mon[92422]: pgmap v20529: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 312 KiB/s rd, 309 KiB/s wr, 7 op/s 2023-03-10T01:23:46.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37536.522896 secs 2023-03-10T01:23:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37536.522896 secs 2023-03-10T01:23:46.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37536.522896 secs 2023-03-10T01:23:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:23:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:23:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:23:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:23:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:23:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:23:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:47 smithi122 ceph-mon[93869]: pgmap v20530: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 513 KiB/s wr, 11 op/s 2023-03-10T01:23:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:23:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:23:48.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:23:48.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:23:48.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:23:48.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:23:48.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:23:48.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:23:48.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:23:48.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:23:48.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:23:48.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:23:48.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:23:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:23:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:23:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:23:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:23:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:23:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:47 smithi181 ceph-mon[94189]: pgmap v20530: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 513 KiB/s wr, 11 op/s 2023-03-10T01:23:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:23:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:23:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:23:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:23:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:23:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:23:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:23:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:23:48.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:23:48.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:23:48.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:23:48.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:23:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:23:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:23:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:23:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:23:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:23:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:23:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:47 smithi038 ceph-mon[92422]: pgmap v20530: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 513 KiB/s wr, 11 op/s 2023-03-10T01:23:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:23:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:23:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:23:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:23:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:23:48.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:23:48.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:23:48.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:23:48.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:23:48.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:23:48.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:23:48.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:23:48.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38000.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:23:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:48 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-10T01:23:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:48 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-10T01:23:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:48 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-10T01:23:49.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:48 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-10T01:23:49.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:23:49.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:23:49.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:48 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-10T01:23:49.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:48 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-10T01:23:49.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:48 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-10T01:23:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:48 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-10T01:23:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:23:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:23:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:48 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-10T01:23:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:48 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-10T01:23:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:48 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-10T01:23:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:48 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-10T01:23:49.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:23:49.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:23:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:49 smithi122 ceph-mon[93869]: pgmap v20531: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 515 KiB/s rd, 513 KiB/s wr, 8 op/s 2023-03-10T01:23:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:49 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:23:50.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:49 smithi181 ceph-mon[94189]: pgmap v20531: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 515 KiB/s rd, 513 KiB/s wr, 8 op/s 2023-03-10T01:23:50.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:49 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:23:50.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:49 smithi038 ceph-mon[92422]: pgmap v20531: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 515 KiB/s rd, 513 KiB/s wr, 8 op/s 2023-03-10T01:23:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:49 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:23:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37541.522955 secs 2023-03-10T01:23:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:50 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:23:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:50 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:23:51.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:50 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:23:51.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37541.522955 secs 2023-03-10T01:23:51.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:50 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:23:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:50 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:23:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:50 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:23:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37541.522955 secs 2023-03-10T01:23:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:50 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:23:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:50 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:23:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:50 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:23:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:51 smithi181 ceph-mon[94189]: pgmap v20532: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 678 KiB/s rd, 673 KiB/s wr, 12 op/s 2023-03-10T01:23:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:23:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:23:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:51 smithi038 ceph-mon[92422]: pgmap v20532: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 678 KiB/s rd, 673 KiB/s wr, 12 op/s 2023-03-10T01:23:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:23:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:23:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:51 smithi122 ceph-mon[93869]: pgmap v20532: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 678 KiB/s rd, 673 KiB/s wr, 12 op/s 2023-03-10T01:23:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:23:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:23:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37543.982860 secs 2023-03-10T01:23:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37543.982860 secs 2023-03-10T01:23:53.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37543.982860 secs 2023-03-10T01:23:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38005.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:23:54.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:53 smithi038 ceph-mon[92422]: pgmap v20533: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 399 KiB/s rd, 395 KiB/s wr, 10 op/s 2023-03-10T01:23:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:53 smithi122 ceph-mon[93869]: pgmap v20533: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 399 KiB/s rd, 395 KiB/s wr, 10 op/s 2023-03-10T01:23:54.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:53 smithi181 ceph-mon[94189]: pgmap v20533: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 399 KiB/s rd, 395 KiB/s wr, 10 op/s 2023-03-10T01:23:55.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:54 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:23:55.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:54 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:23:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:54 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:23:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:54 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:23:55.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:54 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:23:55.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:54 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:23:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:56 smithi122 ceph-mon[93869]: pgmap v20534: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 395 KiB/s wr, 9 op/s 2023-03-10T01:23:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37546.522982 secs 2023-03-10T01:23:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:23:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:23:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:56 smithi181 ceph-mon[94189]: pgmap v20534: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 395 KiB/s wr, 9 op/s 2023-03-10T01:23:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37546.522982 secs 2023-03-10T01:23:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:23:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:23:56.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:56 smithi038 ceph-mon[92422]: pgmap v20534: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 395 KiB/s wr, 9 op/s 2023-03-10T01:23:56.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37546.522982 secs 2023-03-10T01:23:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:23:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:23:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:23:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:23:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:57 smithi122 ceph-mon[93869]: pgmap v20535: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 739 KiB/s rd, 724 KiB/s wr, 14 op/s 2023-03-10T01:23:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:23:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:23:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:57 smithi181 ceph-mon[94189]: pgmap v20535: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 739 KiB/s rd, 724 KiB/s wr, 14 op/s 2023-03-10T01:23:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:23:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:23:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:57 smithi038 ceph-mon[92422]: pgmap v20535: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 739 KiB/s rd, 724 KiB/s wr, 14 op/s 2023-03-10T01:23:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38010.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:23:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37548.982918 secs 2023-03-10T01:23:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37548.982918 secs 2023-03-10T01:23:58.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37548.982918 secs 2023-03-10T01:23:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:59 smithi122 ceph-mon[93869]: pgmap v20536: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 535 KiB/s rd, 520 KiB/s wr, 10 op/s 2023-03-10T01:23:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:23:59 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:23:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:59 smithi181 ceph-mon[94189]: pgmap v20536: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 535 KiB/s rd, 520 KiB/s wr, 10 op/s 2023-03-10T01:23:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:23:59 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:23:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:59 smithi038 ceph-mon[92422]: pgmap v20536: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 535 KiB/s rd, 520 KiB/s wr, 10 op/s 2023-03-10T01:24:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:23:59 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:24:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37551.523039 secs 2023-03-10T01:24:01.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37551.523039 secs 2023-03-10T01:24:01.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37551.523039 secs 2023-03-10T01:24:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:01 smithi122 ceph-mon[93869]: pgmap v20537: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 795 KiB/s wr, 13 op/s 2023-03-10T01:24:02.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:01 smithi181 ceph-mon[94189]: pgmap v20537: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 795 KiB/s wr, 13 op/s 2023-03-10T01:24:02.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:01 smithi038 ceph-mon[92422]: pgmap v20537: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 795 KiB/s wr, 13 op/s 2023-03-10T01:24:03.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37553.982981 secs 2023-03-10T01:24:03.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37553.982981 secs 2023-03-10T01:24:03.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37553.982981 secs 2023-03-10T01:24:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38015.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:24:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:03 smithi122 ceph-mon[93869]: pgmap v20538: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 938 KiB/s rd, 810 KiB/s wr, 21 op/s 2023-03-10T01:24:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:03 smithi181 ceph-mon[94189]: pgmap v20538: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 938 KiB/s rd, 810 KiB/s wr, 21 op/s 2023-03-10T01:24:04.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:03 smithi038 ceph-mon[92422]: pgmap v20538: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 938 KiB/s rd, 810 KiB/s wr, 21 op/s 2023-03-10T01:24:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:05 smithi122 ceph-mon[93869]: pgmap v20539: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 907 KiB/s rd, 779 KiB/s wr, 20 op/s 2023-03-10T01:24:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:05 smithi181 ceph-mon[94189]: pgmap v20539: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 907 KiB/s rd, 779 KiB/s wr, 20 op/s 2023-03-10T01:24:05.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:05 smithi038 ceph-mon[92422]: pgmap v20539: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 907 KiB/s rd, 779 KiB/s wr, 20 op/s 2023-03-10T01:24:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37556.523100 secs 2023-03-10T01:24:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37556.523100 secs 2023-03-10T01:24:06.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37556.523100 secs 2023-03-10T01:24:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:07 smithi122 ceph-mon[93869]: pgmap v20540: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 995 KiB/s rd, 888 KiB/s wr, 24 op/s 2023-03-10T01:24:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:07 smithi181 ceph-mon[94189]: pgmap v20540: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 995 KiB/s rd, 888 KiB/s wr, 24 op/s 2023-03-10T01:24:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:07 smithi038 ceph-mon[92422]: pgmap v20540: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 995 KiB/s rd, 888 KiB/s wr, 24 op/s 2023-03-10T01:24:08.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38020.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:24:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37558.983041 secs 2023-03-10T01:24:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:24:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:24:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37558.983041 secs 2023-03-10T01:24:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:24:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:24:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37558.983041 secs 2023-03-10T01:24:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:24:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:24:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:24:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:24:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:09 smithi122 ceph-mon[93869]: pgmap v20541: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 835 KiB/s rd, 564 KiB/s wr, 21 op/s 2023-03-10T01:24:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:24:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:24:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:09 smithi181 ceph-mon[94189]: pgmap v20541: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 835 KiB/s rd, 564 KiB/s wr, 21 op/s 2023-03-10T01:24:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:24:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:24:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:09 smithi038 ceph-mon[92422]: pgmap v20541: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 835 KiB/s rd, 564 KiB/s wr, 21 op/s 2023-03-10T01:24:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37561.523158 secs 2023-03-10T01:24:11.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37561.523158 secs 2023-03-10T01:24:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37561.523158 secs 2023-03-10T01:24:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:11 smithi122 ceph-mon[93869]: pgmap v20542: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 966 KiB/s rd, 694 KiB/s wr, 24 op/s 2023-03-10T01:24:12.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:11 smithi181 ceph-mon[94189]: pgmap v20542: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 966 KiB/s rd, 694 KiB/s wr, 24 op/s 2023-03-10T01:24:12.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:11 smithi038 ceph-mon[92422]: pgmap v20542: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 966 KiB/s rd, 694 KiB/s wr, 24 op/s 2023-03-10T01:24:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37563.983164 secs 2023-03-10T01:24:13.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37563.983164 secs 2023-03-10T01:24:13.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37563.983164 secs 2023-03-10T01:24:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38025.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:24:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:13 smithi181 ceph-mon[94189]: pgmap v20543: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 578 KiB/s rd, 591 KiB/s wr, 22 op/s 2023-03-10T01:24:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:13 smithi038 ceph-mon[92422]: pgmap v20543: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 578 KiB/s rd, 591 KiB/s wr, 22 op/s 2023-03-10T01:24:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:13 smithi122 ceph-mon[93869]: pgmap v20543: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 578 KiB/s rd, 591 KiB/s wr, 22 op/s 2023-03-10T01:24:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:15 smithi122 ceph-mon[93869]: pgmap v20544: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 571 KiB/s rd, 416 KiB/s wr, 10 op/s 2023-03-10T01:24:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:15 smithi181 ceph-mon[94189]: pgmap v20544: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 571 KiB/s rd, 416 KiB/s wr, 10 op/s 2023-03-10T01:24:15.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:15 smithi038 ceph-mon[92422]: pgmap v20544: 129 pgs: 129 active+clean; 14 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 571 KiB/s rd, 416 KiB/s wr, 10 op/s 2023-03-10T01:24:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37566.523223 secs 2023-03-10T01:24:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:24:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:24:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:16 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): Behind on trimming (1318/128) 2023-03-10T01:24:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37566.523223 secs 2023-03-10T01:24:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:24:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:24:16.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:16 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): Behind on trimming (1318/128) 2023-03-10T01:24:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37566.523223 secs 2023-03-10T01:24:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:24:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:24:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:16 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): Behind on trimming (1318/128) 2023-03-10T01:24:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:17 smithi122 ceph-mon[93869]: Health check update: 1 MDSs behind on trimming (MDS_TRIM) 2023-03-10T01:24:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:24:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:24:17.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:17 smithi122 ceph-mon[93869]: pgmap v20545: 129 pgs: 129 active+clean; 11 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 790 KiB/s rd, 634 KiB/s wr, 81 op/s 2023-03-10T01:24:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:17 smithi181 ceph-mon[94189]: Health check update: 1 MDSs behind on trimming (MDS_TRIM) 2023-03-10T01:24:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:24:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:24:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:17 smithi181 ceph-mon[94189]: pgmap v20545: 129 pgs: 129 active+clean; 11 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 790 KiB/s rd, 634 KiB/s wr, 81 op/s 2023-03-10T01:24:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:17 smithi038 ceph-mon[92422]: Health check update: 1 MDSs behind on trimming (MDS_TRIM) 2023-03-10T01:24:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:24:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:24:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:17 smithi038 ceph-mon[92422]: pgmap v20545: 129 pgs: 129 active+clean; 11 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 790 KiB/s rd, 634 KiB/s wr, 81 op/s 2023-03-10T01:24:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38030.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:24:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37568.983224 secs 2023-03-10T01:24:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37568.983224 secs 2023-03-10T01:24:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37568.983224 secs 2023-03-10T01:24:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:19 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): Behind on trimming (1319/128) 2023-03-10T01:24:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:19 smithi122 ceph-mon[93869]: pgmap v20546: 129 pgs: 129 active+clean; 9.5 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 691 KiB/s rd, 525 KiB/s wr, 108 op/s 2023-03-10T01:24:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:19 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): Behind on trimming (1319/128) 2023-03-10T01:24:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:19 smithi181 ceph-mon[94189]: pgmap v20546: 129 pgs: 129 active+clean; 9.5 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 691 KiB/s rd, 525 KiB/s wr, 108 op/s 2023-03-10T01:24:19.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:19 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): Behind on trimming (1319/128) 2023-03-10T01:24:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:19 smithi038 ceph-mon[92422]: pgmap v20546: 129 pgs: 129 active+clean; 9.5 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 691 KiB/s rd, 525 KiB/s wr, 108 op/s 2023-03-10T01:24:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:20 smithi122 ceph-mon[93869]: Health check cleared: MDS_TRIM (was: 1 MDSs behind on trimming) 2023-03-10T01:24:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:24:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:24:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:20 smithi181 ceph-mon[94189]: Health check cleared: MDS_TRIM (was: 1 MDSs behind on trimming) 2023-03-10T01:24:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:24:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:24:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:20 smithi038 ceph-mon[92422]: Health check cleared: MDS_TRIM (was: 1 MDSs behind on trimming) 2023-03-10T01:24:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:24:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:24:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37571.523282 secs 2023-03-10T01:24:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:21 smithi122 ceph-mon[93869]: pgmap v20547: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 789 KiB/s wr, 112 op/s 2023-03-10T01:24:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37571.523282 secs 2023-03-10T01:24:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:21 smithi181 ceph-mon[94189]: pgmap v20547: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 789 KiB/s wr, 112 op/s 2023-03-10T01:24:21.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37571.523282 secs 2023-03-10T01:24:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:21 smithi038 ceph-mon[92422]: pgmap v20547: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 789 KiB/s wr, 112 op/s 2023-03-10T01:24:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37573.983282 secs 2023-03-10T01:24:23.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37573.983282 secs 2023-03-10T01:24:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37573.983282 secs 2023-03-10T01:24:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38035.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:24:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:24 smithi122 ceph-mon[93869]: pgmap v20548: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 987 KiB/s rd, 701 KiB/s wr, 111 op/s 2023-03-10T01:24:24.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:24 smithi181 ceph-mon[94189]: pgmap v20548: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 987 KiB/s rd, 701 KiB/s wr, 111 op/s 2023-03-10T01:24:24.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:24 smithi038 ceph-mon[92422]: pgmap v20548: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 987 KiB/s rd, 701 KiB/s wr, 111 op/s 2023-03-10T01:24:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:26 smithi122 ceph-mon[93869]: pgmap v20549: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 814 KiB/s rd, 529 KiB/s wr, 110 op/s 2023-03-10T01:24:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:26 smithi122 ceph-mon[93869]: osdmap e1499: 12 total, 12 up, 12 in 2023-03-10T01:24:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37576.523354 secs 2023-03-10T01:24:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:26 smithi181 ceph-mon[94189]: pgmap v20549: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 814 KiB/s rd, 529 KiB/s wr, 110 op/s 2023-03-10T01:24:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:26 smithi181 ceph-mon[94189]: osdmap e1499: 12 total, 12 up, 12 in 2023-03-10T01:24:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37576.523354 secs 2023-03-10T01:24:26.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:26 smithi038 ceph-mon[92422]: pgmap v20549: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 814 KiB/s rd, 529 KiB/s wr, 110 op/s 2023-03-10T01:24:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:26 smithi038 ceph-mon[92422]: osdmap e1499: 12 total, 12 up, 12 in 2023-03-10T01:24:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37576.523354 secs 2023-03-10T01:24:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:28 smithi122 ceph-mon[93869]: pgmap v20551: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 743 KiB/s wr, 52 op/s 2023-03-10T01:24:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37578.983356 secs 2023-03-10T01:24:28.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:28 smithi181 ceph-mon[94189]: pgmap v20551: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 743 KiB/s wr, 52 op/s 2023-03-10T01:24:28.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37578.983356 secs 2023-03-10T01:24:28.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:28 smithi038 ceph-mon[92422]: pgmap v20551: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 743 KiB/s wr, 52 op/s 2023-03-10T01:24:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37578.983356 secs 2023-03-10T01:24:28.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38040.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:24:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:30 smithi122 ceph-mon[93869]: pgmap v20552: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 743 KiB/s wr, 15 op/s 2023-03-10T01:24:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:30 smithi122 ceph-mon[93869]: osdmap e1500: 12 total, 12 up, 12 in 2023-03-10T01:24:30.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:30 smithi181 ceph-mon[94189]: pgmap v20552: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 743 KiB/s wr, 15 op/s 2023-03-10T01:24:30.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:30 smithi181 ceph-mon[94189]: osdmap e1500: 12 total, 12 up, 12 in 2023-03-10T01:24:30.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:30 smithi038 ceph-mon[92422]: pgmap v20552: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 743 KiB/s wr, 15 op/s 2023-03-10T01:24:30.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:30 smithi038 ceph-mon[92422]: osdmap e1500: 12 total, 12 up, 12 in 2023-03-10T01:24:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37581.523420 secs 2023-03-10T01:24:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37581.523420 secs 2023-03-10T01:24:31.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37581.523420 secs 2023-03-10T01:24:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:32 smithi122 ceph-mon[93869]: pgmap v20554: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 985 KiB/s rd, 1.1 MiB/s wr, 13 op/s 2023-03-10T01:24:32.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:32 smithi181 ceph-mon[94189]: pgmap v20554: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 985 KiB/s rd, 1.1 MiB/s wr, 13 op/s 2023-03-10T01:24:32.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:32 smithi038 ceph-mon[92422]: pgmap v20554: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 985 KiB/s rd, 1.1 MiB/s wr, 13 op/s 2023-03-10T01:24:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37583.983491 secs 2023-03-10T01:24:33.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37583.983491 secs 2023-03-10T01:24:33.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37583.983491 secs 2023-03-10T01:24:33.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38045.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:24:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:34 smithi122 ceph-mon[93869]: pgmap v20555: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 985 KiB/s rd, 1.1 MiB/s wr, 15 op/s 2023-03-10T01:24:34.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:34 smithi181 ceph-mon[94189]: pgmap v20555: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 985 KiB/s rd, 1.1 MiB/s wr, 15 op/s 2023-03-10T01:24:34.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:34 smithi038 ceph-mon[92422]: pgmap v20555: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 985 KiB/s rd, 1.1 MiB/s wr, 15 op/s 2023-03-10T01:24:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:35 smithi122 ceph-mon[93869]: pgmap v20556: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 789 KiB/s rd, 933 KiB/s wr, 12 op/s 2023-03-10T01:24:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:35 smithi181 ceph-mon[94189]: pgmap v20556: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 789 KiB/s rd, 933 KiB/s wr, 12 op/s 2023-03-10T01:24:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:35 smithi038 ceph-mon[92422]: pgmap v20556: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 789 KiB/s rd, 933 KiB/s wr, 12 op/s 2023-03-10T01:24:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37586.523480 secs 2023-03-10T01:24:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37586.523480 secs 2023-03-10T01:24:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37586.523480 secs 2023-03-10T01:24:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:37 smithi122 ceph-mon[93869]: pgmap v20557: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 790 KiB/s rd, 788 KiB/s wr, 10 op/s 2023-03-10T01:24:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:37 smithi181 ceph-mon[94189]: pgmap v20557: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 790 KiB/s rd, 788 KiB/s wr, 10 op/s 2023-03-10T01:24:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:37 smithi038 ceph-mon[92422]: pgmap v20557: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 790 KiB/s rd, 788 KiB/s wr, 10 op/s 2023-03-10T01:24:38.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38050.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:24:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37588.983570 secs 2023-03-10T01:24:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37588.983570 secs 2023-03-10T01:24:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37588.983570 secs 2023-03-10T01:24:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:39 smithi122 ceph-mon[93869]: pgmap v20558: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 790 KiB/s rd, 788 KiB/s wr, 10 op/s 2023-03-10T01:24:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:39 smithi181 ceph-mon[94189]: pgmap v20558: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 790 KiB/s rd, 788 KiB/s wr, 10 op/s 2023-03-10T01:24:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:39 smithi038 ceph-mon[92422]: pgmap v20558: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 790 KiB/s rd, 788 KiB/s wr, 10 op/s 2023-03-10T01:24:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37591.523539 secs 2023-03-10T01:24:41.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37591.523539 secs 2023-03-10T01:24:41.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37591.523539 secs 2023-03-10T01:24:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:41 smithi122 ceph-mon[93869]: pgmap v20559: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 848 KiB/s rd, 845 KiB/s wr, 12 op/s 2023-03-10T01:24:42.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:41 smithi181 ceph-mon[94189]: pgmap v20559: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 848 KiB/s rd, 845 KiB/s wr, 12 op/s 2023-03-10T01:24:42.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:41 smithi038 ceph-mon[92422]: pgmap v20559: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 848 KiB/s rd, 845 KiB/s wr, 12 op/s 2023-03-10T01:24:43.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37593.983657 secs 2023-03-10T01:24:43.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37593.983657 secs 2023-03-10T01:24:43.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37593.983657 secs 2023-03-10T01:24:43.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38055.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:24:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:43 smithi122 ceph-mon[93869]: pgmap v20560: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 339 KiB/s wr, 8 op/s 2023-03-10T01:24:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:43 smithi181 ceph-mon[94189]: pgmap v20560: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 339 KiB/s wr, 8 op/s 2023-03-10T01:24:44.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:43 smithi038 ceph-mon[92422]: pgmap v20560: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 339 KiB/s wr, 8 op/s 2023-03-10T01:24:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:45 smithi122 ceph-mon[93869]: pgmap v20561: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 339 KiB/s wr, 7 op/s 2023-03-10T01:24:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:45 smithi181 ceph-mon[94189]: pgmap v20561: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 339 KiB/s wr, 7 op/s 2023-03-10T01:24:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:45 smithi038 ceph-mon[92422]: pgmap v20561: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 339 KiB/s wr, 7 op/s 2023-03-10T01:24:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37596.523626 secs 2023-03-10T01:24:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37596.523626 secs 2023-03-10T01:24:46.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37596.523626 secs 2023-03-10T01:24:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:47 smithi122 ceph-mon[93869]: pgmap v20562: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 342 KiB/s wr, 10 op/s 2023-03-10T01:24:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:47 smithi181 ceph-mon[94189]: pgmap v20562: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 342 KiB/s wr, 10 op/s 2023-03-10T01:24:47.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:47 smithi038 ceph-mon[92422]: pgmap v20562: 129 pgs: 129 active+clean; 9.4 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 342 KiB/s wr, 10 op/s 2023-03-10T01:24:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38060.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:24:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37598.983799 secs 2023-03-10T01:24:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:24:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:24:48.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:24:48.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:24:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37598.983799 secs 2023-03-10T01:24:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:24:48.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:24:48.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:24:48.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:24:48.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37598.983799 secs 2023-03-10T01:24:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:24:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:24:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:24:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:24:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:49 smithi122 ceph-mon[93869]: pgmap v20563: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 312 KiB/s wr, 7 op/s 2023-03-10T01:24:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:49 smithi181 ceph-mon[94189]: pgmap v20563: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 312 KiB/s wr, 7 op/s 2023-03-10T01:24:49.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:49 smithi038 ceph-mon[92422]: pgmap v20563: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 314 KiB/s rd, 312 KiB/s wr, 7 op/s 2023-03-10T01:24:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37601.523710 secs 2023-03-10T01:24:51.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37601.523710 secs 2023-03-10T01:24:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37601.523710 secs 2023-03-10T01:24:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:51 smithi122 ceph-mon[93869]: pgmap v20564: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 312 KiB/s wr, 9 op/s 2023-03-10T01:24:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:51 smithi181 ceph-mon[94189]: pgmap v20564: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 312 KiB/s wr, 9 op/s 2023-03-10T01:24:52.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:51 smithi038 ceph-mon[92422]: pgmap v20564: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 312 KiB/s wr, 9 op/s 2023-03-10T01:24:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37603.983861 secs 2023-03-10T01:24:53.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37603.983861 secs 2023-03-10T01:24:53.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37603.983861 secs 2023-03-10T01:24:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38065.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:24:54.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:53 smithi038 ceph-mon[92422]: pgmap v20565: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 193 KiB/s wr, 7 op/s 2023-03-10T01:24:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:53 smithi122 ceph-mon[93869]: pgmap v20565: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 193 KiB/s wr, 7 op/s 2023-03-10T01:24:54.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:53 smithi181 ceph-mon[94189]: pgmap v20565: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 193 KiB/s wr, 7 op/s 2023-03-10T01:24:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:55 smithi122 ceph-mon[93869]: pgmap v20566: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 377 KiB/s wr, 8 op/s 2023-03-10T01:24:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:55 smithi181 ceph-mon[94189]: pgmap v20566: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 377 KiB/s wr, 8 op/s 2023-03-10T01:24:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:55 smithi038 ceph-mon[92422]: pgmap v20566: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 377 KiB/s wr, 8 op/s 2023-03-10T01:24:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37606.523766 secs 2023-03-10T01:24:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37606.523766 secs 2023-03-10T01:24:56.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37606.523766 secs 2023-03-10T01:24:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:57 smithi122 ceph-mon[93869]: pgmap v20567: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 497 KiB/s rd, 408 KiB/s wr, 11 op/s 2023-03-10T01:24:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:57 smithi181 ceph-mon[94189]: pgmap v20567: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 497 KiB/s rd, 408 KiB/s wr, 11 op/s 2023-03-10T01:24:57.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:57 smithi038 ceph-mon[92422]: pgmap v20567: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 497 KiB/s rd, 408 KiB/s wr, 11 op/s 2023-03-10T01:24:58.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38070.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:24:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:58 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:24:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:58 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:24:58.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:58 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:24:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:59 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:24:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:24:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:24:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:24:59 smithi122 ceph-mon[93869]: pgmap v20568: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 866 KiB/s rd, 746 KiB/s wr, 10 op/s 2023-03-10T01:24:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:59 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:24:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:24:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:24:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:24:59 smithi181 ceph-mon[94189]: pgmap v20568: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 866 KiB/s rd, 746 KiB/s wr, 10 op/s 2023-03-10T01:24:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:59 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:24:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:24:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:24:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:24:59 smithi038 ceph-mon[92422]: pgmap v20568: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 866 KiB/s rd, 746 KiB/s wr, 10 op/s 2023-03-10T01:25:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:25:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37611.523851 secs 2023-03-10T01:25:01.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:25:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37611.523851 secs 2023-03-10T01:25:01.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:25:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37611.523851 secs 2023-03-10T01:25:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:25:01 smithi122 ceph-mon[93869]: pgmap v20569: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 589 KiB/s wr, 11 op/s 2023-03-10T01:25:02.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:25:01 smithi181 ceph-mon[94189]: pgmap v20569: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 589 KiB/s wr, 11 op/s 2023-03-10T01:25:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:25:01 smithi038 ceph-mon[92422]: pgmap v20569: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 589 KiB/s wr, 11 op/s 2023-03-10T01:25:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38075.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:25:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:25:04 smithi122 ceph-mon[93869]: pgmap v20570: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 973 KiB/s wr, 23 op/s 2023-03-10T01:25:04.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:25:04 smithi181 ceph-mon[94189]: pgmap v20570: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 973 KiB/s wr, 23 op/s 2023-03-10T01:25:04.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:25:04 smithi038 ceph-mon[92422]: pgmap v20570: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 973 KiB/s wr, 23 op/s 2023-03-10T01:25:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:25:05 smithi122 ceph-mon[93869]: pgmap v20571: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 942 KiB/s wr, 22 op/s 2023-03-10T01:25:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:25:05 smithi181 ceph-mon[94189]: pgmap v20571: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 942 KiB/s wr, 22 op/s 2023-03-10T01:25:05.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:25:05 smithi038 ceph-mon[92422]: pgmap v20571: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 942 KiB/s wr, 22 op/s 2023-03-10T01:25:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:25:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37616.523935 secs 2023-03-10T01:25:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:25:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37616.523935 secs 2023-03-10T01:25:06.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:25:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37616.523935 secs 2023-03-10T01:25:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:25:07 smithi122 ceph-mon[93869]: pgmap v20572: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 27 op/s 2023-03-10T01:25:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:25:07 smithi181 ceph-mon[94189]: pgmap v20572: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 27 op/s 2023-03-10T01:25:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:25:07 smithi038 ceph-mon[92422]: pgmap v20572: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 27 op/s 2023-03-10T01:25:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38080.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:25:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:25:10 smithi122 ceph-mon[93869]: pgmap v20573: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 24 op/s 2023-03-10T01:25:10.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:25:10 smithi181 ceph-mon[94189]: pgmap v20573: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 24 op/s 2023-03-10T01:25:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:25:10 smithi038 ceph-mon[92422]: pgmap v20573: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 24 op/s 2023-03-10T01:25:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:25:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37621.523979 secs 2023-03-10T01:25:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:25:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37621.523979 secs 2023-03-10T01:25:11.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:25:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37621.523979 secs 2023-03-10T01:25:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:25:12 smithi122 ceph-mon[93869]: pgmap v20574: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 980 KiB/s rd, 793 KiB/s wr, 24 op/s 2023-03-10T01:25:12.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:25:12 smithi181 ceph-mon[94189]: pgmap v20574: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 980 KiB/s rd, 793 KiB/s wr, 24 op/s 2023-03-10T01:25:12.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:25:12 smithi038 ceph-mon[92422]: pgmap v20574: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 980 KiB/s rd, 793 KiB/s wr, 24 op/s 2023-03-10T01:25:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38085.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:25:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:25:14 smithi122 ceph-mon[93869]: pgmap v20575: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 971 KiB/s wr, 24 op/s 2023-03-10T01:25:14.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:25:14 smithi181 ceph-mon[94189]: pgmap v20575: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 971 KiB/s wr, 24 op/s 2023-03-10T01:25:14.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:25:14 smithi038 ceph-mon[92422]: pgmap v20575: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 971 KiB/s wr, 24 op/s 2023-03-10T01:25:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:25:15 smithi122 ceph-mon[93869]: pgmap v20576: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 776 KiB/s rd, 586 KiB/s wr, 10 op/s 2023-03-10T01:25:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:25:15 smithi181 ceph-mon[94189]: pgmap v20576: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 776 KiB/s rd, 586 KiB/s wr, 10 op/s 2023-03-10T01:25:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:25:15 smithi038 ceph-mon[92422]: pgmap v20576: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 776 KiB/s rd, 586 KiB/s wr, 10 op/s 2023-03-10T01:25:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:25:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37626.524062 secs 2023-03-10T01:25:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:25:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37626.524062 secs 2023-03-10T01:25:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:25:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37626.524062 secs 2023-03-10T01:25:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:25:17 smithi122 ceph-mon[93869]: pgmap v20577: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 778 KiB/s rd, 586 KiB/s wr, 12 op/s 2023-03-10T01:25:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:25:17 smithi181 ceph-mon[94189]: pgmap v20577: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 778 KiB/s rd, 586 KiB/s wr, 12 op/s 2023-03-10T01:25:17.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:25:17 smithi038 ceph-mon[92422]: pgmap v20577: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 778 KiB/s rd, 586 KiB/s wr, 12 op/s 2023-03-10T01:25:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38090.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:25:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:25:20 smithi122 ceph-mon[93869]: pgmap v20578: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 352 KiB/s wr, 9 op/s 2023-03-10T01:25:20.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:25:20 smithi181 ceph-mon[94189]: pgmap v20578: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 352 KiB/s wr, 9 op/s 2023-03-10T01:25:20.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:25:20 smithi038 ceph-mon[92422]: pgmap v20578: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 352 KiB/s wr, 9 op/s 2023-03-10T01:25:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:25:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37631.524122 secs 2023-03-10T01:25:21.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:25:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37631.524122 secs 2023-03-10T01:25:21.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:25:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37631.524122 secs 2023-03-10T01:25:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:25:22 smithi122 ceph-mon[93869]: pgmap v20579: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 557 KiB/s wr, 12 op/s 2023-03-10T01:25:22.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:25:22 smithi181 ceph-mon[94189]: pgmap v20579: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 557 KiB/s wr, 12 op/s 2023-03-10T01:25:22.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:25:22 smithi038 ceph-mon[92422]: pgmap v20579: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 557 KiB/s wr, 12 op/s 2023-03-10T01:25:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38095.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:25:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:25:24 smithi122 ceph-mon[93869]: pgmap v20580: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 569 KiB/s rd, 588 KiB/s wr, 12 op/s 2023-03-10T01:25:24.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:25:24 smithi181 ceph-mon[94189]: pgmap v20580: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 569 KiB/s rd, 588 KiB/s wr, 12 op/s 2023-03-10T01:25:24.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:25:24 smithi038 ceph-mon[92422]: pgmap v20580: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 569 KiB/s rd, 588 KiB/s wr, 12 op/s 2023-03-10T01:25:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:25:26 smithi122 ceph-mon[93869]: pgmap v20581: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 411 KiB/s wr, 11 op/s 2023-03-10T01:25:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:25:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37636.524181 secs 2023-03-10T01:25:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:25:26 smithi122 ceph-mon[93869]: osdmap e1501: 12 total, 12 up, 12 in 2023-03-10T01:25:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:25:26 smithi181 ceph-mon[94189]: pgmap v20581: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 411 KiB/s wr, 11 op/s 2023-03-10T01:25:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:25:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37636.524181 secs 2023-03-10T01:25:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:25:26 smithi181 ceph-mon[94189]: osdmap e1501: 12 total, 12 up, 12 in 2023-03-10T01:25:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:25:26 smithi038 ceph-mon[92422]: pgmap v20581: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 392 KiB/s rd, 411 KiB/s wr, 11 op/s 2023-03-10T01:25:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:25:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37636.524181 secs 2023-03-10T01:25:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:25:26 smithi038 ceph-mon[92422]: osdmap e1501: 12 total, 12 up, 12 in 2023-03-10T01:25:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:25:28 smithi122 ceph-mon[93869]: pgmap v20583: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 652 KiB/s rd, 676 KiB/s wr, 14 op/s 2023-03-10T01:25:28.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:25:28 smithi181 ceph-mon[94189]: pgmap v20583: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 652 KiB/s rd, 676 KiB/s wr, 14 op/s 2023-03-10T01:25:28.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:25:28 smithi038 ceph-mon[92422]: pgmap v20583: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 652 KiB/s rd, 676 KiB/s wr, 14 op/s 2023-03-10T01:25:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38100.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:25:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:25:30 smithi122 ceph-mon[93869]: pgmap v20584: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 534 KiB/s wr, 12 op/s 2023-03-10T01:25:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:25:30 smithi122 ceph-mon[93869]: osdmap e1502: 12 total, 12 up, 12 in 2023-03-10T01:25:30.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:25:30 smithi181 ceph-mon[94189]: pgmap v20584: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 534 KiB/s wr, 12 op/s 2023-03-10T01:25:30.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:25:30 smithi181 ceph-mon[94189]: osdmap e1502: 12 total, 12 up, 12 in 2023-03-10T01:25:30.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:25:30 smithi038 ceph-mon[92422]: pgmap v20584: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 534 KiB/s wr, 12 op/s 2023-03-10T01:25:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:25:30 smithi038 ceph-mon[92422]: osdmap e1502: 12 total, 12 up, 12 in 2023-03-10T01:25:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:25:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37641.524240 secs 2023-03-10T01:25:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:25:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37641.524240 secs 2023-03-10T01:25:31.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:25:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37641.524240 secs 2023-03-10T01:25:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:25:32 smithi122 ceph-mon[93869]: pgmap v20586: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 230 KiB/s wr, 11 op/s 2023-03-10T01:25:32.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:25:32 smithi181 ceph-mon[94189]: pgmap v20586: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 230 KiB/s wr, 11 op/s 2023-03-10T01:25:32.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:25:32 smithi038 ceph-mon[92422]: pgmap v20586: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 230 KiB/s wr, 11 op/s 2023-03-10T01:25:33.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38105.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:25:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:25:34 smithi122 ceph-mon[93869]: pgmap v20587: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 238 KiB/s wr, 12 op/s 2023-03-10T01:25:34.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:25:34 smithi181 ceph-mon[94189]: pgmap v20587: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 238 KiB/s wr, 12 op/s 2023-03-10T01:25:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:25:34 smithi038 ceph-mon[92422]: pgmap v20587: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 238 KiB/s wr, 12 op/s 2023-03-10T01:25:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:25:35 smithi122 ceph-mon[93869]: pgmap v20588: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 199 KiB/s wr, 10 op/s 2023-03-10T01:25:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:25:35 smithi181 ceph-mon[94189]: pgmap v20588: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 199 KiB/s wr, 10 op/s 2023-03-10T01:25:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:25:35 smithi038 ceph-mon[92422]: pgmap v20588: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 199 KiB/s wr, 10 op/s 2023-03-10T01:25:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:25:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37646.524305 secs 2023-03-10T01:25:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:25:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37646.524305 secs 2023-03-10T01:25:36.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:25:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37646.524305 secs 2023-03-10T01:25:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:25:37 smithi122 ceph-mon[93869]: pgmap v20589: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 805 KiB/s rd, 803 KiB/s wr, 11 op/s 2023-03-10T01:25:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:25:37 smithi181 ceph-mon[94189]: pgmap v20589: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 805 KiB/s rd, 803 KiB/s wr, 11 op/s 2023-03-10T01:25:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:25:37 smithi038 ceph-mon[92422]: pgmap v20589: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 805 KiB/s rd, 803 KiB/s wr, 11 op/s 2023-03-10T01:25:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38110.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:25:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:25:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37648.984520 secs 2023-03-10T01:25:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:25:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37648.984520 secs 2023-03-10T01:25:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:25:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37648.984520 secs 2023-03-10T01:25:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:25:39 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:25:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:25:39 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:25:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:25:39 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:25:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:25:39 smithi122 ceph-mon[93869]: pgmap v20590: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 13 op/s 2023-03-10T01:25:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:25:39 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:25:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:25:39 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:25:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:25:39 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:25:39.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:25:39 smithi181 ceph-mon[94189]: pgmap v20590: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 13 op/s 2023-03-10T01:25:39.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:25:39 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:25:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:25:39 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:25:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:25:39 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:25:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:25:39 smithi038 ceph-mon[92422]: pgmap v20590: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 13 op/s 2023-03-10T01:25:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:25:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37651.524393 secs 2023-03-10T01:25:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:25:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37651.524393 secs 2023-03-10T01:25:41.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:25:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37651.524393 secs 2023-03-10T01:25:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:25:41 smithi122 ceph-mon[93869]: pgmap v20591: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 15 op/s 2023-03-10T01:25:42.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:25:41 smithi181 ceph-mon[94189]: pgmap v20591: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 15 op/s 2023-03-10T01:25:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:25:41 smithi038 ceph-mon[92422]: pgmap v20591: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 15 op/s 2023-03-10T01:25:43.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:25:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37653.984580 secs 2023-03-10T01:25:43.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:25:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37653.984580 secs 2023-03-10T01:25:43.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:25:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37653.984580 secs 2023-03-10T01:25:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38115.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:25:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:25:43 smithi122 ceph-mon[93869]: pgmap v20592: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 12 op/s 2023-03-10T01:25:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:25:43 smithi181 ceph-mon[94189]: pgmap v20592: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 12 op/s 2023-03-10T01:25:44.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:25:43 smithi038 ceph-mon[92422]: pgmap v20592: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 12 op/s 2023-03-10T01:25:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:25:45 smithi122 ceph-mon[93869]: pgmap v20593: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 11 op/s 2023-03-10T01:25:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:25:45 smithi181 ceph-mon[94189]: pgmap v20593: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 11 op/s 2023-03-10T01:25:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:25:45 smithi038 ceph-mon[92422]: pgmap v20593: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 11 op/s 2023-03-10T01:25:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:25:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37656.524454 secs 2023-03-10T01:25:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:25:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37656.524454 secs 2023-03-10T01:25:46.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:25:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37656.524454 secs 2023-03-10T01:25:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:25:47 smithi122 ceph-mon[93869]: pgmap v20594: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 14 op/s 2023-03-10T01:25:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:25:47 smithi181 ceph-mon[94189]: pgmap v20594: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 14 op/s 2023-03-10T01:25:47.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:25:47 smithi038 ceph-mon[92422]: pgmap v20594: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 14 op/s 2023-03-10T01:25:48.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38120.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:25:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:25:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:25:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:25:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:25:48.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:25:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:25:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:25:50 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:25:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:25:50 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:25:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:25:50 smithi122 ceph-mon[93869]: pgmap v20595: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 835 KiB/s rd, 822 KiB/s wr, 9 op/s 2023-03-10T01:25:50.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:25:50 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:25:50.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:25:50 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:25:50.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:25:50 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:25:50.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:25:50 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:25:50.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:25:50 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:25:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:25:50 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:25:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:25:50 smithi181 ceph-mon[94189]: pgmap v20595: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 835 KiB/s rd, 822 KiB/s wr, 9 op/s 2023-03-10T01:25:50.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:25:50 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:25:50.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:25:50 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:25:50.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:25:50 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:25:50.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:25:50 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:25:50.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:25:50 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:25:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:25:50 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:25:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:25:50 smithi038 ceph-mon[92422]: pgmap v20595: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 835 KiB/s rd, 822 KiB/s wr, 9 op/s 2023-03-10T01:25:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:25:50 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:25:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:25:50 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:25:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:25:50 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:25:50.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:25:50 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:25:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:25:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37661.524560 secs 2023-03-10T01:25:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:25:51 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:25:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:25:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:25:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:25:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:25:51.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:25:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37661.524560 secs 2023-03-10T01:25:51.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:25:51 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:25:51.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:25:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:25:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:25:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:25:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:25:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37661.524560 secs 2023-03-10T01:25:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:25:51 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:25:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:25:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:25:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:25:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:25:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:25:52 smithi122 ceph-mon[93869]: pgmap v20596: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 13 op/s 2023-03-10T01:25:52.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:25:52 smithi181 ceph-mon[94189]: pgmap v20596: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 13 op/s 2023-03-10T01:25:52.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:25:52 smithi038 ceph-mon[92422]: pgmap v20596: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 13 op/s 2023-03-10T01:25:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:25:53 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 37663.984786 secs 2023-03-10T01:25:53.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:25:53 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 37663.984786 secs 2023-03-10T01:25:53.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:25:53 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 37663.984786 secs 2023-03-10T01:25:53.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38125.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:25:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:25:54 smithi122 ceph-mon[93869]: pgmap v20597: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 12 op/s 2023-03-10T01:25:54.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:25:54 smithi181 ceph-mon[94189]: pgmap v20597: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 12 op/s 2023-03-10T01:25:54.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:25:54 smithi038 ceph-mon[92422]: pgmap v20597: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 12 op/s 2023-03-10T01:25:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:25:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:25:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:25:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:25:55.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:25:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:25:55.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:25:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:25:55.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:25:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:25:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:25:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:25:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:25:56 smithi122 ceph-mon[93869]: pgmap v20598: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 12 op/s 2023-03-10T01:25:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:25:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:25:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:25:56 smithi181 ceph-mon[94189]: pgmap v20598: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 12 op/s 2023-03-10T01:25:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:25:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:25:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:25:56 smithi038 ceph-mon[92422]: pgmap v20598: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 12 op/s 2023-03-10T01:25:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:25:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:25:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:25:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:25:57.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:25:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:25:57.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:25:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:25:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:25:58 smithi122 ceph-mon[93869]: pgmap v20599: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 20 op/s 2023-03-10T01:25:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:25:58 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 37668.984882 secs 2023-03-10T01:25:58.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:25:58 smithi181 ceph-mon[94189]: pgmap v20599: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 20 op/s 2023-03-10T01:25:58.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:25:58 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 37668.984882 secs 2023-03-10T01:25:58.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:25:58 smithi038 ceph-mon[92422]: pgmap v20599: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 20 op/s 2023-03-10T01:25:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:25:58 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 37668.984882 secs 2023-03-10T01:25:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38130.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:25:59.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:25:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:25:59.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:25:59 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:25:59.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:25:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:25:59.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:25:59 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:25:59.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:25:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:25:59.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:25:59 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:26:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:00 smithi122 ceph-mon[93869]: pgmap v20600: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 18 op/s 2023-03-10T01:26:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:00 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:26:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] up:active 2023-03-10T01:26:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:26:00.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:00 smithi181 ceph-mon[94189]: pgmap v20600: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 18 op/s 2023-03-10T01:26:00.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:00 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:26:00.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] up:active 2023-03-10T01:26:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:26:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:00 smithi038 ceph-mon[92422]: pgmap v20600: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.0 MiB/s wr, 18 op/s 2023-03-10T01:26:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:00 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:26:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] up:active 2023-03-10T01:26:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:26:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37671.524701 secs 2023-03-10T01:26:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:26:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:01 smithi122 ceph-mon[93869]: pgmap v20601: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.4 MiB/s wr, 23 op/s 2023-03-10T01:26:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37671.524701 secs 2023-03-10T01:26:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:26:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:01 smithi181 ceph-mon[94189]: pgmap v20601: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.4 MiB/s wr, 23 op/s 2023-03-10T01:26:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37671.524701 secs 2023-03-10T01:26:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:26:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:01 smithi038 ceph-mon[92422]: pgmap v20601: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.4 MiB/s wr, 23 op/s 2023-03-10T01:26:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38135.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:26:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37673.984939 secs 2023-03-10T01:26:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:03 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] up:active 2023-03-10T01:26:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:03 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:26:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:03 smithi122 ceph-mon[93869]: pgmap v20602: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.9 MiB/s wr, 31 op/s 2023-03-10T01:26:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37673.984939 secs 2023-03-10T01:26:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:03 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] up:active 2023-03-10T01:26:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:03 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:26:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:03 smithi181 ceph-mon[94189]: pgmap v20602: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.9 MiB/s wr, 31 op/s 2023-03-10T01:26:03.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37673.984939 secs 2023-03-10T01:26:03.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:03 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] up:active 2023-03-10T01:26:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:03 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:26:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:03 smithi038 ceph-mon[92422]: pgmap v20602: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.9 MiB/s wr, 31 op/s 2023-03-10T01:26:04.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:04 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:26:04.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:04 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:26:04.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:04 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:26:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:05 smithi122 ceph-mon[93869]: osdmap e1503: 12 total, 12 up, 12 in 2023-03-10T01:26:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:05 smithi122 ceph-mon[93869]: pgmap v20604: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.1 MiB/s wr, 35 op/s 2023-03-10T01:26:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:05 smithi181 ceph-mon[94189]: osdmap e1503: 12 total, 12 up, 12 in 2023-03-10T01:26:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:05 smithi181 ceph-mon[94189]: pgmap v20604: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.1 MiB/s wr, 35 op/s 2023-03-10T01:26:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:05 smithi038 ceph-mon[92422]: osdmap e1503: 12 total, 12 up, 12 in 2023-03-10T01:26:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:05 smithi038 ceph-mon[92422]: pgmap v20604: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.1 MiB/s wr, 35 op/s 2023-03-10T01:26:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37676.524760 secs 2023-03-10T01:26:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37676.524760 secs 2023-03-10T01:26:06.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37676.524760 secs 2023-03-10T01:26:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:26:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:26:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:07 smithi122 ceph-mon[93869]: pgmap v20605: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.7 MiB/s wr, 34 op/s 2023-03-10T01:26:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:26:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:26:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:07 smithi181 ceph-mon[94189]: pgmap v20605: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.7 MiB/s wr, 34 op/s 2023-03-10T01:26:07.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:26:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:26:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:07 smithi038 ceph-mon[92422]: pgmap v20605: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.7 MiB/s wr, 34 op/s 2023-03-10T01:26:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38140.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:26:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37678.984997 secs 2023-03-10T01:26:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37678.984997 secs 2023-03-10T01:26:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37678.984997 secs 2023-03-10T01:26:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:09 smithi122 ceph-mon[93869]: pgmap v20606: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 982 KiB/s rd, 1023 KiB/s wr, 32 op/s 2023-03-10T01:26:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:09 smithi181 ceph-mon[94189]: pgmap v20606: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 982 KiB/s rd, 1023 KiB/s wr, 32 op/s 2023-03-10T01:26:09.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:09 smithi038 ceph-mon[92422]: pgmap v20606: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 982 KiB/s rd, 1023 KiB/s wr, 32 op/s 2023-03-10T01:26:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37681.524822 secs 2023-03-10T01:26:11.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37681.524822 secs 2023-03-10T01:26:11.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37681.524822 secs 2023-03-10T01:26:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:11 smithi122 ceph-mon[93869]: pgmap v20607: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 607 KiB/s rd, 647 KiB/s wr, 29 op/s 2023-03-10T01:26:12.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:11 smithi181 ceph-mon[94189]: pgmap v20607: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 607 KiB/s rd, 647 KiB/s wr, 29 op/s 2023-03-10T01:26:12.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:11 smithi038 ceph-mon[92422]: pgmap v20607: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 607 KiB/s rd, 647 KiB/s wr, 29 op/s 2023-03-10T01:26:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37683.985058 secs 2023-03-10T01:26:13.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37683.985058 secs 2023-03-10T01:26:13.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37683.985058 secs 2023-03-10T01:26:13.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38145.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:26:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:14 smithi122 ceph-mon[93869]: pgmap v20608: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 598 KiB/s rd, 621 KiB/s wr, 14 op/s 2023-03-10T01:26:14.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:14 smithi181 ceph-mon[94189]: pgmap v20608: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 598 KiB/s rd, 621 KiB/s wr, 14 op/s 2023-03-10T01:26:14.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:14 smithi038 ceph-mon[92422]: pgmap v20608: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 598 KiB/s rd, 621 KiB/s wr, 14 op/s 2023-03-10T01:26:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:15 smithi122 ceph-mon[93869]: osdmap e1504: 12 total, 12 up, 12 in 2023-03-10T01:26:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:15 smithi122 ceph-mon[93869]: pgmap v20610: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 212 KiB/s wr, 14 op/s 2023-03-10T01:26:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:15 smithi181 ceph-mon[94189]: osdmap e1504: 12 total, 12 up, 12 in 2023-03-10T01:26:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:15 smithi181 ceph-mon[94189]: pgmap v20610: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 212 KiB/s wr, 14 op/s 2023-03-10T01:26:15.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:15 smithi038 ceph-mon[92422]: osdmap e1504: 12 total, 12 up, 12 in 2023-03-10T01:26:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:15 smithi038 ceph-mon[92422]: pgmap v20610: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 212 KiB/s wr, 14 op/s 2023-03-10T01:26:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37686.524883 secs 2023-03-10T01:26:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37686.524883 secs 2023-03-10T01:26:16.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37686.524883 secs 2023-03-10T01:26:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:17 smithi122 ceph-mon[93869]: pgmap v20611: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 182 KiB/s wr, 10 op/s 2023-03-10T01:26:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:17 smithi181 ceph-mon[94189]: pgmap v20611: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 182 KiB/s wr, 10 op/s 2023-03-10T01:26:17.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:17 smithi038 ceph-mon[92422]: pgmap v20611: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 182 KiB/s wr, 10 op/s 2023-03-10T01:26:18.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38150.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:26:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37688.985189 secs 2023-03-10T01:26:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37688.985189 secs 2023-03-10T01:26:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37688.985189 secs 2023-03-10T01:26:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:19 smithi122 ceph-mon[93869]: pgmap v20612: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 182 KiB/s wr, 10 op/s 2023-03-10T01:26:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:19 smithi181 ceph-mon[94189]: pgmap v20612: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 182 KiB/s wr, 10 op/s 2023-03-10T01:26:19.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:19 smithi038 ceph-mon[92422]: pgmap v20612: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 185 KiB/s rd, 182 KiB/s wr, 10 op/s 2023-03-10T01:26:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37691.524951 secs 2023-03-10T01:26:21.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37691.524951 secs 2023-03-10T01:26:21.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37691.524951 secs 2023-03-10T01:26:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:21 smithi122 ceph-mon[93869]: pgmap v20613: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 118 KiB/s wr, 10 op/s 2023-03-10T01:26:22.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:21 smithi181 ceph-mon[94189]: pgmap v20613: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 118 KiB/s wr, 10 op/s 2023-03-10T01:26:22.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:21 smithi038 ceph-mon[92422]: pgmap v20613: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 118 KiB/s wr, 10 op/s 2023-03-10T01:26:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:22 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37693.985328 secs 2023-03-10T01:26:23.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:22 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37693.985328 secs 2023-03-10T01:26:23.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:22 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37693.985328 secs 2023-03-10T01:26:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38155.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:26:24.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:23 smithi038 ceph-mon[92422]: pgmap v20614: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 118 KiB/s wr, 9 op/s 2023-03-10T01:26:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:23 smithi122 ceph-mon[93869]: pgmap v20614: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 118 KiB/s wr, 9 op/s 2023-03-10T01:26:24.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:23 smithi181 ceph-mon[94189]: pgmap v20614: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 118 KiB/s wr, 9 op/s 2023-03-10T01:26:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:25 smithi122 ceph-mon[93869]: pgmap v20615: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 113 KiB/s wr, 8 op/s 2023-03-10T01:26:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:25 smithi181 ceph-mon[94189]: pgmap v20615: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 113 KiB/s wr, 8 op/s 2023-03-10T01:26:25.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:25 smithi038 ceph-mon[92422]: pgmap v20615: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 113 KiB/s wr, 8 op/s 2023-03-10T01:26:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37696.525035 secs 2023-03-10T01:26:26.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37696.525035 secs 2023-03-10T01:26:26.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37696.525035 secs 2023-03-10T01:26:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:27 smithi122 ceph-mon[93869]: pgmap v20616: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 99 KiB/s wr, 10 op/s 2023-03-10T01:26:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:27 smithi181 ceph-mon[94189]: pgmap v20616: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 99 KiB/s wr, 10 op/s 2023-03-10T01:26:27.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:27 smithi038 ceph-mon[92422]: pgmap v20616: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 99 KiB/s wr, 10 op/s 2023-03-10T01:26:28.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38160.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:26:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37698.985392 secs 2023-03-10T01:26:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37698.985392 secs 2023-03-10T01:26:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37698.985392 secs 2023-03-10T01:26:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:29 smithi122 ceph-mon[93869]: pgmap v20617: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 99 KiB/s wr, 7 op/s 2023-03-10T01:26:29.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:29 smithi181 ceph-mon[94189]: pgmap v20617: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 99 KiB/s wr, 7 op/s 2023-03-10T01:26:29.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:29 smithi038 ceph-mon[92422]: pgmap v20617: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 99 KiB/s wr, 7 op/s 2023-03-10T01:26:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37701.525102 secs 2023-03-10T01:26:31.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37701.525102 secs 2023-03-10T01:26:31.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37701.525102 secs 2023-03-10T01:26:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:31 smithi122 ceph-mon[93869]: pgmap v20618: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 131 KiB/s wr, 9 op/s 2023-03-10T01:26:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:31 smithi181 ceph-mon[94189]: pgmap v20618: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 131 KiB/s wr, 9 op/s 2023-03-10T01:26:32.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:31 smithi038 ceph-mon[92422]: pgmap v20618: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 131 KiB/s wr, 9 op/s 2023-03-10T01:26:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37703.985450 secs 2023-03-10T01:26:33.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37703.985450 secs 2023-03-10T01:26:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37703.985450 secs 2023-03-10T01:26:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38165.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:26:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:33 smithi181 ceph-mon[94189]: pgmap v20619: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 161 KiB/s wr, 6 op/s 2023-03-10T01:26:34.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:33 smithi038 ceph-mon[92422]: pgmap v20619: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 161 KiB/s wr, 6 op/s 2023-03-10T01:26:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:33 smithi122 ceph-mon[93869]: pgmap v20619: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 161 KiB/s wr, 6 op/s 2023-03-10T01:26:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:35 smithi122 ceph-mon[93869]: pgmap v20620: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 161 KiB/s wr, 6 op/s 2023-03-10T01:26:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:35 smithi181 ceph-mon[94189]: pgmap v20620: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 161 KiB/s wr, 6 op/s 2023-03-10T01:26:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:35 smithi038 ceph-mon[92422]: pgmap v20620: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 161 KiB/s wr, 6 op/s 2023-03-10T01:26:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37706.525164 secs 2023-03-10T01:26:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37706.525164 secs 2023-03-10T01:26:36.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37706.525164 secs 2023-03-10T01:26:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:37 smithi122 ceph-mon[93869]: pgmap v20621: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 342 KiB/s wr, 9 op/s 2023-03-10T01:26:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:37 smithi181 ceph-mon[94189]: pgmap v20621: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 342 KiB/s wr, 9 op/s 2023-03-10T01:26:37.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:37 smithi038 ceph-mon[92422]: pgmap v20621: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 342 KiB/s wr, 9 op/s 2023-03-10T01:26:38.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38170.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:26:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37708.985521 secs 2023-03-10T01:26:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37708.985521 secs 2023-03-10T01:26:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37708.985521 secs 2023-03-10T01:26:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:39 smithi122 ceph-mon[93869]: pgmap v20622: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 342 KiB/s wr, 7 op/s 2023-03-10T01:26:39.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:39 smithi181 ceph-mon[94189]: pgmap v20622: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 342 KiB/s wr, 7 op/s 2023-03-10T01:26:39.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:39 smithi038 ceph-mon[92422]: pgmap v20622: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 342 KiB/s wr, 7 op/s 2023-03-10T01:26:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37711.525216 secs 2023-03-10T01:26:41.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37711.525216 secs 2023-03-10T01:26:41.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37711.525216 secs 2023-03-10T01:26:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:41 smithi122 ceph-mon[93869]: pgmap v20623: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 342 KiB/s wr, 9 op/s 2023-03-10T01:26:42.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:41 smithi181 ceph-mon[94189]: pgmap v20623: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 342 KiB/s wr, 9 op/s 2023-03-10T01:26:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:41 smithi038 ceph-mon[92422]: pgmap v20623: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 342 KiB/s wr, 9 op/s 2023-03-10T01:26:43.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37713.985610 secs 2023-03-10T01:26:43.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37713.985610 secs 2023-03-10T01:26:43.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37713.985610 secs 2023-03-10T01:26:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38175.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:26:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:43 smithi122 ceph-mon[93869]: pgmap v20624: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 597 KiB/s rd, 595 KiB/s wr, 8 op/s 2023-03-10T01:26:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:43 smithi181 ceph-mon[94189]: pgmap v20624: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 597 KiB/s rd, 595 KiB/s wr, 8 op/s 2023-03-10T01:26:44.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:43 smithi038 ceph-mon[92422]: pgmap v20624: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 597 KiB/s rd, 595 KiB/s wr, 8 op/s 2023-03-10T01:26:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:45 smithi122 ceph-mon[93869]: pgmap v20625: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 469 KiB/s rd, 466 KiB/s wr, 7 op/s 2023-03-10T01:26:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:45 smithi181 ceph-mon[94189]: pgmap v20625: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 469 KiB/s rd, 466 KiB/s wr, 7 op/s 2023-03-10T01:26:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:45 smithi038 ceph-mon[92422]: pgmap v20625: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 469 KiB/s rd, 466 KiB/s wr, 7 op/s 2023-03-10T01:26:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37716.525278 secs 2023-03-10T01:26:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37716.525278 secs 2023-03-10T01:26:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37716.525278 secs 2023-03-10T01:26:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:47 smithi122 ceph-mon[93869]: pgmap v20626: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 721 KiB/s rd, 628 KiB/s wr, 11 op/s 2023-03-10T01:26:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:47 smithi181 ceph-mon[94189]: pgmap v20626: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 721 KiB/s rd, 628 KiB/s wr, 11 op/s 2023-03-10T01:26:47.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:47 smithi038 ceph-mon[92422]: pgmap v20626: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 721 KiB/s rd, 628 KiB/s wr, 11 op/s 2023-03-10T01:26:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38180.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:26:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37718.985662 secs 2023-03-10T01:26:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37718.985662 secs 2023-03-10T01:26:48.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37718.985662 secs 2023-03-10T01:26:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:49 smithi122 ceph-mon[93869]: pgmap v20627: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 538 KiB/s rd, 447 KiB/s wr, 7 op/s 2023-03-10T01:26:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:49 smithi181 ceph-mon[94189]: pgmap v20627: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 538 KiB/s rd, 447 KiB/s wr, 7 op/s 2023-03-10T01:26:49.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:49 smithi038 ceph-mon[92422]: pgmap v20627: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 538 KiB/s rd, 447 KiB/s wr, 7 op/s 2023-03-10T01:26:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:50 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:26:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:50 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:26:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:50 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:26:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:50 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:26:50.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:50 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:26:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:50 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:26:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:50 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:26:50.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:50 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:26:50.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:50 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:26:50.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:50 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:26:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:50 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:26:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:50 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:26:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:50 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:26:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:50 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:26:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:50 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:26:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37721.525339 secs 2023-03-10T01:26:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:26:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:26:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:26:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:51 smithi122 ceph-mon[93869]: pgmap v20628: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 570 KiB/s rd, 478 KiB/s wr, 9 op/s 2023-03-10T01:26:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37721.525339 secs 2023-03-10T01:26:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:26:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:26:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:26:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:51 smithi181 ceph-mon[94189]: pgmap v20628: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 570 KiB/s rd, 478 KiB/s wr, 9 op/s 2023-03-10T01:26:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37721.525339 secs 2023-03-10T01:26:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:26:52.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:26:52.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:26:52.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:51 smithi038 ceph-mon[92422]: pgmap v20628: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 570 KiB/s rd, 478 KiB/s wr, 9 op/s 2023-03-10T01:26:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37723.985770 secs 2023-03-10T01:26:53.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37723.985770 secs 2023-03-10T01:26:53.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37723.985770 secs 2023-03-10T01:26:53.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38185.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:26:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:54 smithi122 ceph-mon[93869]: pgmap v20629: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 731 KiB/s rd, 641 KiB/s wr, 9 op/s 2023-03-10T01:26:54.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:54 smithi181 ceph-mon[94189]: pgmap v20629: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 731 KiB/s rd, 641 KiB/s wr, 9 op/s 2023-03-10T01:26:54.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:54 smithi038 ceph-mon[92422]: pgmap v20629: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 731 KiB/s rd, 641 KiB/s wr, 9 op/s 2023-03-10T01:26:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:55 smithi122 ceph-mon[93869]: pgmap v20630: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 446 KiB/s rd, 355 KiB/s wr, 8 op/s 2023-03-10T01:26:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:55 smithi181 ceph-mon[94189]: pgmap v20630: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 446 KiB/s rd, 355 KiB/s wr, 8 op/s 2023-03-10T01:26:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:55 smithi038 ceph-mon[92422]: pgmap v20630: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 446 KiB/s rd, 355 KiB/s wr, 8 op/s 2023-03-10T01:26:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37726.525424 secs 2023-03-10T01:26:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37726.525424 secs 2023-03-10T01:26:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37726.525424 secs 2023-03-10T01:26:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:57 smithi122 ceph-mon[93869]: pgmap v20631: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 983 KiB/s rd, 888 KiB/s wr, 15 op/s 2023-03-10T01:26:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:57 smithi181 ceph-mon[94189]: pgmap v20631: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 983 KiB/s rd, 888 KiB/s wr, 15 op/s 2023-03-10T01:26:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:57 smithi038 ceph-mon[92422]: pgmap v20631: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 983 KiB/s rd, 888 KiB/s wr, 15 op/s 2023-03-10T01:26:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38190.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:26:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37728.985833 secs 2023-03-10T01:26:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37728.985833 secs 2023-03-10T01:26:58.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37728.985833 secs 2023-03-10T01:26:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:26:59 smithi122 ceph-mon[93869]: pgmap v20632: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 743 KiB/s rd, 727 KiB/s wr, 12 op/s 2023-03-10T01:26:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:26:59 smithi181 ceph-mon[94189]: pgmap v20632: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 743 KiB/s rd, 727 KiB/s wr, 12 op/s 2023-03-10T01:26:59.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:26:59 smithi038 ceph-mon[92422]: pgmap v20632: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 743 KiB/s rd, 727 KiB/s wr, 12 op/s 2023-03-10T01:27:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:27:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37731.525483 secs 2023-03-10T01:27:01.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:27:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37731.525483 secs 2023-03-10T01:27:01.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:27:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37731.525483 secs 2023-03-10T01:27:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:27:01 smithi122 ceph-mon[93869]: pgmap v20633: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 776 KiB/s rd, 758 KiB/s wr, 15 op/s 2023-03-10T01:27:02.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:27:01 smithi181 ceph-mon[94189]: pgmap v20633: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 776 KiB/s rd, 758 KiB/s wr, 15 op/s 2023-03-10T01:27:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:27:01 smithi038 ceph-mon[92422]: pgmap v20633: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 776 KiB/s rd, 758 KiB/s wr, 15 op/s 2023-03-10T01:27:03.482 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:27:03 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 37733.985954 secs 2023-03-10T01:27:03.486 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:27:03 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 37733.985954 secs 2023-03-10T01:27:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:27:03 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 37733.985954 secs 2023-03-10T01:27:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38195.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:27:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:27:04 smithi122 ceph-mon[93869]: pgmap v20634: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 17 op/s 2023-03-10T01:27:04.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:27:04 smithi181 ceph-mon[94189]: pgmap v20634: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 17 op/s 2023-03-10T01:27:04.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:27:04 smithi038 ceph-mon[92422]: pgmap v20634: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 17 op/s 2023-03-10T01:27:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:27:05 smithi122 ceph-mon[93869]: pgmap v20635: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 972 KiB/s rd, 955 KiB/s wr, 15 op/s 2023-03-10T01:27:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:27:05 smithi181 ceph-mon[94189]: pgmap v20635: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 972 KiB/s rd, 955 KiB/s wr, 15 op/s 2023-03-10T01:27:05.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:27:05 smithi038 ceph-mon[92422]: pgmap v20635: 129 pgs: 129 active+clean; 9.5 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 972 KiB/s rd, 955 KiB/s wr, 15 op/s 2023-03-10T01:27:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:27:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37736.525546 secs 2023-03-10T01:27:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:27:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37736.525546 secs 2023-03-10T01:27:06.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:27:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37736.525546 secs 2023-03-10T01:27:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:27:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:27:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:27:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:27:07.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:27:07 smithi122 ceph-mon[93869]: pgmap v20636: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 20 op/s 2023-03-10T01:27:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:27:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:27:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:27:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:27:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:27:07 smithi181 ceph-mon[94189]: pgmap v20636: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 20 op/s 2023-03-10T01:27:07.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:27:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:27:07.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:27:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:27:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:27:07 smithi038 ceph-mon[92422]: pgmap v20636: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 20 op/s 2023-03-10T01:27:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38200.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:27:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:27:08 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 37738.986064 secs 2023-03-10T01:27:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:27:08 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 37738.986064 secs 2023-03-10T01:27:08.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:27:08 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 37738.986064 secs 2023-03-10T01:27:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:27:09 smithi122 ceph-mon[93869]: pgmap v20637: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 816 KiB/s rd, 818 KiB/s wr, 23 op/s 2023-03-10T01:27:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:27:09 smithi181 ceph-mon[94189]: pgmap v20637: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 816 KiB/s rd, 818 KiB/s wr, 23 op/s 2023-03-10T01:27:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:27:09 smithi038 ceph-mon[92422]: pgmap v20637: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 816 KiB/s rd, 818 KiB/s wr, 23 op/s 2023-03-10T01:27:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:27:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37741.525605 secs 2023-03-10T01:27:11.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:27:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37741.525605 secs 2023-03-10T01:27:11.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:27:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37741.525605 secs 2023-03-10T01:27:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:27:11 smithi122 ceph-mon[93869]: pgmap v20638: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 26 op/s 2023-03-10T01:27:12.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:27:11 smithi181 ceph-mon[94189]: pgmap v20638: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 26 op/s 2023-03-10T01:27:12.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:27:11 smithi038 ceph-mon[92422]: pgmap v20638: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 26 op/s 2023-03-10T01:27:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:27:12 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 37743.986120 secs 2023-03-10T01:27:13.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:27:12 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 37743.986120 secs 2023-03-10T01:27:13.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:27:12 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 37743.986120 secs 2023-03-10T01:27:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38205.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:27:14.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:27:13 smithi122 ceph-mon[93869]: pgmap v20639: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 24 op/s 2023-03-10T01:27:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:27:13 smithi181 ceph-mon[94189]: pgmap v20639: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 24 op/s 2023-03-10T01:27:14.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:27:13 smithi038 ceph-mon[92422]: pgmap v20639: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 24 op/s 2023-03-10T01:27:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:27:15 smithi122 ceph-mon[93869]: pgmap v20640: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 665 KiB/s rd, 678 KiB/s wr, 21 op/s 2023-03-10T01:27:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:27:15 smithi181 ceph-mon[94189]: pgmap v20640: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 665 KiB/s rd, 678 KiB/s wr, 21 op/s 2023-03-10T01:27:15.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:27:15 smithi038 ceph-mon[92422]: pgmap v20640: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 665 KiB/s rd, 678 KiB/s wr, 21 op/s 2023-03-10T01:27:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:27:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37746.525664 secs 2023-03-10T01:27:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:27:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37746.525664 secs 2023-03-10T01:27:16.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:27:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37746.525664 secs 2023-03-10T01:27:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:27:17 smithi122 ceph-mon[93869]: pgmap v20641: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 666 KiB/s rd, 679 KiB/s wr, 23 op/s 2023-03-10T01:27:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:27:17 smithi181 ceph-mon[94189]: pgmap v20641: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 666 KiB/s rd, 679 KiB/s wr, 23 op/s 2023-03-10T01:27:17.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:27:17 smithi038 ceph-mon[92422]: pgmap v20641: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 666 KiB/s rd, 679 KiB/s wr, 23 op/s 2023-03-10T01:27:18.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38210.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:27:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:27:18 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 37748.986183 secs 2023-03-10T01:27:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:27:18 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 37748.986183 secs 2023-03-10T01:27:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:27:18 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 37748.986183 secs 2023-03-10T01:27:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:27:19 smithi122 ceph-mon[93869]: pgmap v20642: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 530 KiB/s rd, 533 KiB/s wr, 18 op/s 2023-03-10T01:27:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:27:19 smithi181 ceph-mon[94189]: pgmap v20642: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 530 KiB/s rd, 533 KiB/s wr, 18 op/s 2023-03-10T01:27:19.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:27:19 smithi038 ceph-mon[92422]: pgmap v20642: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 530 KiB/s rd, 533 KiB/s wr, 18 op/s 2023-03-10T01:27:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:27:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37751.525722 secs 2023-03-10T01:27:21.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:27:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37751.525722 secs 2023-03-10T01:27:21.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:27:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37751.525722 secs 2023-03-10T01:27:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:27:21 smithi122 ceph-mon[93869]: pgmap v20643: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 283 KiB/s wr, 10 op/s 2023-03-10T01:27:22.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:27:21 smithi181 ceph-mon[94189]: pgmap v20643: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 283 KiB/s wr, 10 op/s 2023-03-10T01:27:22.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:27:21 smithi038 ceph-mon[92422]: pgmap v20643: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 283 KiB/s wr, 10 op/s 2023-03-10T01:27:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:27:22 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 37753.986268 secs 2023-03-10T01:27:23.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:27:22 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 37753.986268 secs 2023-03-10T01:27:23.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:27:22 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 37753.986268 secs 2023-03-10T01:27:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38215.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:27:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:27:23 smithi181 ceph-mon[94189]: pgmap v20644: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 258 KiB/s wr, 9 op/s 2023-03-10T01:27:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:27:23 smithi038 ceph-mon[92422]: pgmap v20644: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 258 KiB/s wr, 9 op/s 2023-03-10T01:27:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:27:23 smithi122 ceph-mon[93869]: pgmap v20644: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 258 KiB/s wr, 9 op/s 2023-03-10T01:27:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:27:25 smithi122 ceph-mon[93869]: pgmap v20645: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 214 KiB/s wr, 8 op/s 2023-03-10T01:27:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:27:25 smithi181 ceph-mon[94189]: pgmap v20645: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 214 KiB/s wr, 8 op/s 2023-03-10T01:27:25.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:27:25 smithi038 ceph-mon[92422]: pgmap v20645: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 214 KiB/s wr, 8 op/s 2023-03-10T01:27:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:27:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37756.525782 secs 2023-03-10T01:27:26.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:27:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37756.525782 secs 2023-03-10T01:27:26.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:27:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37756.525782 secs 2023-03-10T01:27:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:27:27 smithi122 ceph-mon[93869]: pgmap v20646: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 256 KiB/s wr, 11 op/s 2023-03-10T01:27:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:27:27 smithi181 ceph-mon[94189]: pgmap v20646: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 256 KiB/s wr, 11 op/s 2023-03-10T01:27:27.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:27:27 smithi038 ceph-mon[92422]: pgmap v20646: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 256 KiB/s wr, 11 op/s 2023-03-10T01:27:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38220.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:27:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:27:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37758.986326 secs 2023-03-10T01:27:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:27:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37758.986326 secs 2023-03-10T01:27:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:27:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37758.986326 secs 2023-03-10T01:27:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:27:29 smithi122 ceph-mon[93869]: pgmap v20647: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 257 KiB/s wr, 9 op/s 2023-03-10T01:27:29.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:27:29 smithi181 ceph-mon[94189]: pgmap v20647: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 257 KiB/s wr, 9 op/s 2023-03-10T01:27:29.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:27:29 smithi038 ceph-mon[92422]: pgmap v20647: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 237 KiB/s rd, 257 KiB/s wr, 9 op/s 2023-03-10T01:27:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:27:30 smithi122 ceph-mon[93869]: osdmap e1505: 12 total, 12 up, 12 in 2023-03-10T01:27:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:27:30 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:27:30.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:27:30 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:27:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:27:30 smithi181 ceph-mon[94189]: osdmap e1505: 12 total, 12 up, 12 in 2023-03-10T01:27:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:27:30 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:27:30.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:27:30 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:27:30.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:27:30 smithi038 ceph-mon[92422]: osdmap e1505: 12 total, 12 up, 12 in 2023-03-10T01:27:30.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:27:30 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:27:30.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:27:30 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:27:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:27:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37761.525840 secs 2023-03-10T01:27:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:27:31 smithi122 ceph-mon[93869]: pgmap v20649: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 386 KiB/s rd, 410 KiB/s wr, 12 op/s 2023-03-10T01:27:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:27:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37761.525840 secs 2023-03-10T01:27:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:27:31 smithi181 ceph-mon[94189]: pgmap v20649: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 386 KiB/s rd, 410 KiB/s wr, 12 op/s 2023-03-10T01:27:31.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:27:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37761.525840 secs 2023-03-10T01:27:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:27:31 smithi038 ceph-mon[92422]: pgmap v20649: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 386 KiB/s rd, 410 KiB/s wr, 12 op/s 2023-03-10T01:27:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:27:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37763.986389 secs 2023-03-10T01:27:33.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:27:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37763.986389 secs 2023-03-10T01:27:33.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:27:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37763.986389 secs 2023-03-10T01:27:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38225.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:27:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:27:34 smithi122 ceph-mon[93869]: pgmap v20650: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 499 KiB/s rd, 268 KiB/s wr, 10 op/s 2023-03-10T01:27:34.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:27:34 smithi181 ceph-mon[94189]: pgmap v20650: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 499 KiB/s rd, 268 KiB/s wr, 10 op/s 2023-03-10T01:27:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:27:34 smithi038 ceph-mon[92422]: pgmap v20650: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 499 KiB/s rd, 268 KiB/s wr, 10 op/s 2023-03-10T01:27:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:27:35 smithi122 ceph-mon[93869]: pgmap v20651: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 499 KiB/s rd, 268 KiB/s wr, 10 op/s 2023-03-10T01:27:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:27:35 smithi181 ceph-mon[94189]: pgmap v20651: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 499 KiB/s rd, 268 KiB/s wr, 10 op/s 2023-03-10T01:27:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:27:35 smithi038 ceph-mon[92422]: pgmap v20651: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 499 KiB/s rd, 268 KiB/s wr, 10 op/s 2023-03-10T01:27:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:27:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37766.525921 secs 2023-03-10T01:27:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:27:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37766.525921 secs 2023-03-10T01:27:36.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:27:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37766.525921 secs 2023-03-10T01:27:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:27:37 smithi122 ceph-mon[93869]: pgmap v20652: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 655 KiB/s rd, 600 KiB/s wr, 10 op/s 2023-03-10T01:27:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:27:37 smithi181 ceph-mon[94189]: pgmap v20652: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 655 KiB/s rd, 600 KiB/s wr, 10 op/s 2023-03-10T01:27:37.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:27:37 smithi038 ceph-mon[92422]: pgmap v20652: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 655 KiB/s rd, 600 KiB/s wr, 10 op/s 2023-03-10T01:27:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38230.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:27:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:27:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37768.986451 secs 2023-03-10T01:27:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:27:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37768.986451 secs 2023-03-10T01:27:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:27:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37768.986451 secs 2023-03-10T01:27:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:27:39 smithi122 ceph-mon[93869]: pgmap v20653: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 598 KiB/s wr, 9 op/s 2023-03-10T01:27:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:27:39 smithi122 ceph-mon[93869]: osdmap e1506: 12 total, 12 up, 12 in 2023-03-10T01:27:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:27:39 smithi181 ceph-mon[94189]: pgmap v20653: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 598 KiB/s wr, 9 op/s 2023-03-10T01:27:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:27:39 smithi181 ceph-mon[94189]: osdmap e1506: 12 total, 12 up, 12 in 2023-03-10T01:27:39.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:27:39 smithi038 ceph-mon[92422]: pgmap v20653: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 598 KiB/s wr, 9 op/s 2023-03-10T01:27:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:27:39 smithi038 ceph-mon[92422]: osdmap e1506: 12 total, 12 up, 12 in 2023-03-10T01:27:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:27:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37771.525987 secs 2023-03-10T01:27:41.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:27:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37771.525987 secs 2023-03-10T01:27:41.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:27:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37771.525987 secs 2023-03-10T01:27:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:27:41 smithi122 ceph-mon[93869]: pgmap v20655: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 499 KiB/s rd, 497 KiB/s wr, 9 op/s 2023-03-10T01:27:42.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:27:41 smithi181 ceph-mon[94189]: pgmap v20655: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 499 KiB/s rd, 497 KiB/s wr, 9 op/s 2023-03-10T01:27:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:27:41 smithi038 ceph-mon[92422]: pgmap v20655: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 499 KiB/s rd, 497 KiB/s wr, 9 op/s 2023-03-10T01:27:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:27:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37773.986511 secs 2023-03-10T01:27:43.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:27:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37773.986511 secs 2023-03-10T01:27:43.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:27:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37773.986511 secs 2023-03-10T01:27:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38235.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:27:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:27:44 smithi122 ceph-mon[93869]: pgmap v20656: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 386 KiB/s rd, 383 KiB/s wr, 9 op/s 2023-03-10T01:27:44.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:27:44 smithi181 ceph-mon[94189]: pgmap v20656: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 386 KiB/s rd, 383 KiB/s wr, 9 op/s 2023-03-10T01:27:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:27:44 smithi038 ceph-mon[92422]: pgmap v20656: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 386 KiB/s rd, 383 KiB/s wr, 9 op/s 2023-03-10T01:27:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:27:45 smithi122 ceph-mon[93869]: pgmap v20657: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 386 KiB/s rd, 383 KiB/s wr, 9 op/s 2023-03-10T01:27:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:27:45 smithi181 ceph-mon[94189]: pgmap v20657: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 386 KiB/s rd, 383 KiB/s wr, 9 op/s 2023-03-10T01:27:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:27:45 smithi038 ceph-mon[92422]: pgmap v20657: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 386 KiB/s rd, 383 KiB/s wr, 9 op/s 2023-03-10T01:27:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:27:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37776.526046 secs 2023-03-10T01:27:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:27:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37776.526046 secs 2023-03-10T01:27:46.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:27:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37776.526046 secs 2023-03-10T01:27:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:27:47 smithi122 ceph-mon[93869]: pgmap v20658: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 259 KiB/s wr, 8 op/s 2023-03-10T01:27:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:27:47 smithi181 ceph-mon[94189]: pgmap v20658: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 259 KiB/s wr, 8 op/s 2023-03-10T01:27:47.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:27:47 smithi038 ceph-mon[92422]: pgmap v20658: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 259 KiB/s wr, 8 op/s 2023-03-10T01:27:48.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38240.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:27:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:27:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37778.986570 secs 2023-03-10T01:27:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:27:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37778.986570 secs 2023-03-10T01:27:48.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:27:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37778.986570 secs 2023-03-10T01:27:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:27:49 smithi122 ceph-mon[93869]: pgmap v20659: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 259 KiB/s wr, 8 op/s 2023-03-10T01:27:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:27:49 smithi181 ceph-mon[94189]: pgmap v20659: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 259 KiB/s wr, 8 op/s 2023-03-10T01:27:49.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:27:49 smithi038 ceph-mon[92422]: pgmap v20659: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 259 KiB/s wr, 8 op/s 2023-03-10T01:27:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:27:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37781.526104 secs 2023-03-10T01:27:51.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:27:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37781.526104 secs 2023-03-10T01:27:51.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:27:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37781.526104 secs 2023-03-10T01:27:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:27:51 smithi122 ceph-mon[93869]: pgmap v20660: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 327 KiB/s wr, 10 op/s 2023-03-10T01:27:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:27:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:27:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:27:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:27:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:27:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:27:52.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:27:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:27:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:27:51 smithi181 ceph-mon[94189]: pgmap v20660: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 327 KiB/s wr, 10 op/s 2023-03-10T01:27:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:27:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:27:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:27:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:27:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:27:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:27:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:27:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:27:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:27:51 smithi038 ceph-mon[92422]: pgmap v20660: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 327 KiB/s wr, 10 op/s 2023-03-10T01:27:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:27:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:27:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:27:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:27:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:27:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:27:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:27:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:27:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:27:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37783.986631 secs 2023-03-10T01:27:53.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:27:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37783.986631 secs 2023-03-10T01:27:53.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:27:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37783.986631 secs 2023-03-10T01:27:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38245.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:27:54.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:27:53 smithi038 ceph-mon[92422]: pgmap v20661: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 653 KiB/s wr, 7 op/s 2023-03-10T01:27:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:27:53 smithi122 ceph-mon[93869]: pgmap v20661: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 653 KiB/s wr, 7 op/s 2023-03-10T01:27:54.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:27:53 smithi181 ceph-mon[94189]: pgmap v20661: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 653 KiB/s wr, 7 op/s 2023-03-10T01:27:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:27:55 smithi122 ceph-mon[93869]: pgmap v20662: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 653 KiB/s wr, 7 op/s 2023-03-10T01:27:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:27:55 smithi181 ceph-mon[94189]: pgmap v20662: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 653 KiB/s wr, 7 op/s 2023-03-10T01:27:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:27:55 smithi038 ceph-mon[92422]: pgmap v20662: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 653 KiB/s wr, 7 op/s 2023-03-10T01:27:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:27:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37786.526164 secs 2023-03-10T01:27:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:27:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37786.526164 secs 2023-03-10T01:27:56.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:27:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37786.526164 secs 2023-03-10T01:27:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:27:57 smithi122 ceph-mon[93869]: pgmap v20663: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 911 KiB/s wr, 12 op/s 2023-03-10T01:27:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:27:57 smithi181 ceph-mon[94189]: pgmap v20663: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 911 KiB/s wr, 12 op/s 2023-03-10T01:27:57.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:27:57 smithi038 ceph-mon[92422]: pgmap v20663: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 911 KiB/s wr, 12 op/s 2023-03-10T01:27:58.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38250.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:27:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:27:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37788.986739 secs 2023-03-10T01:27:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:27:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37788.986739 secs 2023-03-10T01:27:58.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:27:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37788.986739 secs 2023-03-10T01:27:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:27:59 smithi122 ceph-mon[93869]: pgmap v20664: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 695 KiB/s wr, 10 op/s 2023-03-10T01:27:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:27:59 smithi181 ceph-mon[94189]: pgmap v20664: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 695 KiB/s wr, 10 op/s 2023-03-10T01:27:59.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:27:59 smithi038 ceph-mon[92422]: pgmap v20664: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 695 KiB/s wr, 10 op/s 2023-03-10T01:28:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:28:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37791.526219 secs 2023-03-10T01:28:01.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:28:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37791.526219 secs 2023-03-10T01:28:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:28:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37791.526219 secs 2023-03-10T01:28:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:28:01 smithi122 ceph-mon[93869]: pgmap v20665: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 854 KiB/s wr, 15 op/s 2023-03-10T01:28:02.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:28:01 smithi181 ceph-mon[94189]: pgmap v20665: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 854 KiB/s wr, 15 op/s 2023-03-10T01:28:02.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:28:01 smithi038 ceph-mon[92422]: pgmap v20665: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 854 KiB/s wr, 15 op/s 2023-03-10T01:28:03.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:28:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37793.986855 secs 2023-03-10T01:28:03.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:28:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37793.986855 secs 2023-03-10T01:28:03.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:28:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37793.986855 secs 2023-03-10T01:28:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38255.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:28:04.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:28:03 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:28:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:28:03 smithi038 ceph-mon[92422]: pgmap v20666: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.1 MiB/s wr, 24 op/s 2023-03-10T01:28:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:28:03 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:28:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:28:03 smithi122 ceph-mon[93869]: pgmap v20666: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.1 MiB/s wr, 24 op/s 2023-03-10T01:28:04.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:28:03 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:28:04.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:28:03 smithi181 ceph-mon[94189]: pgmap v20666: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.1 MiB/s wr, 24 op/s 2023-03-10T01:28:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:28:05 smithi122 ceph-mon[93869]: pgmap v20667: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 769 KiB/s wr, 24 op/s 2023-03-10T01:28:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:28:05 smithi181 ceph-mon[94189]: pgmap v20667: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 769 KiB/s wr, 24 op/s 2023-03-10T01:28:05.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:28:05 smithi038 ceph-mon[92422]: pgmap v20667: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 769 KiB/s wr, 24 op/s 2023-03-10T01:28:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:28:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37796.526278 secs 2023-03-10T01:28:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:28:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37796.526278 secs 2023-03-10T01:28:06.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:28:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37796.526278 secs 2023-03-10T01:28:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:28:07 smithi122 ceph-mon[93869]: pgmap v20668: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 770 KiB/s wr, 27 op/s 2023-03-10T01:28:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:28:07 smithi181 ceph-mon[94189]: pgmap v20668: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 770 KiB/s wr, 27 op/s 2023-03-10T01:28:07.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:28:07 smithi038 ceph-mon[92422]: pgmap v20668: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 770 KiB/s wr, 27 op/s 2023-03-10T01:28:08.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38260.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:28:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:28:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37798.986914 secs 2023-03-10T01:28:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:28:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37798.986914 secs 2023-03-10T01:28:08.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:28:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37798.986914 secs 2023-03-10T01:28:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:28:09 smithi122 ceph-mon[93869]: pgmap v20669: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1013 KiB/s rd, 794 KiB/s wr, 23 op/s 2023-03-10T01:28:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:28:09 smithi181 ceph-mon[94189]: pgmap v20669: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1013 KiB/s rd, 794 KiB/s wr, 23 op/s 2023-03-10T01:28:09.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:28:09 smithi038 ceph-mon[92422]: pgmap v20669: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1013 KiB/s rd, 794 KiB/s wr, 23 op/s 2023-03-10T01:28:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:28:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37801.526338 secs 2023-03-10T01:28:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:28:11 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:28:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:28:11 smithi122 ceph-mon[93869]: pgmap v20670: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1003 KiB/s rd, 794 KiB/s wr, 25 op/s 2023-03-10T01:28:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:28:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37801.526338 secs 2023-03-10T01:28:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:28:11 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:28:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:28:11 smithi181 ceph-mon[94189]: pgmap v20670: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1003 KiB/s rd, 794 KiB/s wr, 25 op/s 2023-03-10T01:28:11.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:28:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37801.526338 secs 2023-03-10T01:28:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:28:11 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:28:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:28:11 smithi038 ceph-mon[92422]: pgmap v20670: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 1003 KiB/s rd, 794 KiB/s wr, 25 op/s 2023-03-10T01:28:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:28:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37803.987027 secs 2023-03-10T01:28:13.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:28:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37803.987027 secs 2023-03-10T01:28:13.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:28:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37803.987027 secs 2023-03-10T01:28:13.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38265.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:28:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:28:14 smithi122 ceph-mon[93869]: pgmap v20671: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 853 KiB/s rd, 635 KiB/s wr, 20 op/s 2023-03-10T01:28:14.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:28:14 smithi181 ceph-mon[94189]: pgmap v20671: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 853 KiB/s rd, 635 KiB/s wr, 20 op/s 2023-03-10T01:28:14.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:28:14 smithi038 ceph-mon[92422]: pgmap v20671: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 853 KiB/s rd, 635 KiB/s wr, 20 op/s 2023-03-10T01:28:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:28:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:28:16.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:28:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37806.526395 secs 2023-03-10T01:28:16.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:28:16 smithi122 ceph-mon[93869]: pgmap v20672: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 837 KiB/s rd, 434 KiB/s wr, 8 op/s 2023-03-10T01:28:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:28:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:28:16.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:28:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37806.526395 secs 2023-03-10T01:28:16.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:28:16 smithi181 ceph-mon[94189]: pgmap v20672: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 837 KiB/s rd, 434 KiB/s wr, 8 op/s 2023-03-10T01:28:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:28:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:28:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:28:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37806.526395 secs 2023-03-10T01:28:16.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:28:16 smithi038 ceph-mon[92422]: pgmap v20672: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 837 KiB/s rd, 434 KiB/s wr, 8 op/s 2023-03-10T01:28:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:28:18 smithi122 ceph-mon[93869]: pgmap v20673: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 838 KiB/s rd, 435 KiB/s wr, 11 op/s 2023-03-10T01:28:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:28:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37808.987141 secs 2023-03-10T01:28:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:28:18 smithi181 ceph-mon[94189]: pgmap v20673: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 838 KiB/s rd, 435 KiB/s wr, 11 op/s 2023-03-10T01:28:18.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:28:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37808.987141 secs 2023-03-10T01:28:18.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:28:18 smithi038 ceph-mon[92422]: pgmap v20673: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 838 KiB/s rd, 435 KiB/s wr, 11 op/s 2023-03-10T01:28:18.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:28:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37808.987141 secs 2023-03-10T01:28:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38270.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:28:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:28:20 smithi122 ceph-mon[93869]: pgmap v20674: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 866 KiB/s rd, 694 KiB/s wr, 9 op/s 2023-03-10T01:28:20.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:28:20 smithi181 ceph-mon[94189]: pgmap v20674: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 866 KiB/s rd, 694 KiB/s wr, 9 op/s 2023-03-10T01:28:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:28:20 smithi038 ceph-mon[92422]: pgmap v20674: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 866 KiB/s rd, 694 KiB/s wr, 9 op/s 2023-03-10T01:28:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:28:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37811.526452 secs 2023-03-10T01:28:21.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:28:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37811.526452 secs 2023-03-10T01:28:21.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:28:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37811.526452 secs 2023-03-10T01:28:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:28:22 smithi122 ceph-mon[93869]: pgmap v20675: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 828 KiB/s rd, 666 KiB/s wr, 12 op/s 2023-03-10T01:28:22.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:28:22 smithi181 ceph-mon[94189]: pgmap v20675: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 828 KiB/s rd, 666 KiB/s wr, 12 op/s 2023-03-10T01:28:22.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:28:22 smithi038 ceph-mon[92422]: pgmap v20675: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 828 KiB/s rd, 666 KiB/s wr, 12 op/s 2023-03-10T01:28:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:28:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37813.987200 secs 2023-03-10T01:28:23.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:28:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37813.987200 secs 2023-03-10T01:28:23.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:28:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37813.987200 secs 2023-03-10T01:28:23.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38275.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:28:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:28:24 smithi122 ceph-mon[93869]: pgmap v20676: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 826 KiB/s rd, 666 KiB/s wr, 10 op/s 2023-03-10T01:28:24.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:28:24 smithi181 ceph-mon[94189]: pgmap v20676: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 826 KiB/s rd, 666 KiB/s wr, 10 op/s 2023-03-10T01:28:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:28:24 smithi038 ceph-mon[92422]: pgmap v20676: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 826 KiB/s rd, 666 KiB/s wr, 10 op/s 2023-03-10T01:28:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:28:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37816.526512 secs 2023-03-10T01:28:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:28:26 smithi122 ceph-mon[93869]: pgmap v20677: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 826 KiB/s rd, 666 KiB/s wr, 10 op/s 2023-03-10T01:28:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:28:26 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:28:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:28:26 smithi122 ceph-mon[93869]: osdmap e1507: 12 total, 12 up, 12 in 2023-03-10T01:28:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:28:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37816.526512 secs 2023-03-10T01:28:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:28:26 smithi181 ceph-mon[94189]: pgmap v20677: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 826 KiB/s rd, 666 KiB/s wr, 10 op/s 2023-03-10T01:28:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:28:26 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:28:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:28:26 smithi181 ceph-mon[94189]: osdmap e1507: 12 total, 12 up, 12 in 2023-03-10T01:28:26.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:28:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37816.526512 secs 2023-03-10T01:28:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:28:26 smithi038 ceph-mon[92422]: pgmap v20677: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 826 KiB/s rd, 666 KiB/s wr, 10 op/s 2023-03-10T01:28:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:28:26 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:28:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:28:26 smithi038 ceph-mon[92422]: osdmap e1507: 12 total, 12 up, 12 in 2023-03-10T01:28:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:28:28 smithi122 ceph-mon[93869]: pgmap v20679: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 777 KiB/s rd, 800 KiB/s wr, 13 op/s 2023-03-10T01:28:28.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:28:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37818.987314 secs 2023-03-10T01:28:28.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:28:28 smithi181 ceph-mon[94189]: pgmap v20679: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 777 KiB/s rd, 800 KiB/s wr, 13 op/s 2023-03-10T01:28:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:28:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37818.987314 secs 2023-03-10T01:28:28.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:28:28 smithi038 ceph-mon[92422]: pgmap v20679: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 777 KiB/s rd, 800 KiB/s wr, 13 op/s 2023-03-10T01:28:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:28:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37818.987314 secs 2023-03-10T01:28:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38280.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:28:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:28:30 smithi122 ceph-mon[93869]: pgmap v20680: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 490 KiB/s wr, 10 op/s 2023-03-10T01:28:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:28:30 smithi122 ceph-mon[93869]: osdmap e1508: 12 total, 12 up, 12 in 2023-03-10T01:28:30.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:28:30 smithi181 ceph-mon[94189]: pgmap v20680: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 490 KiB/s wr, 10 op/s 2023-03-10T01:28:30.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:28:30 smithi181 ceph-mon[94189]: osdmap e1508: 12 total, 12 up, 12 in 2023-03-10T01:28:30.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:28:30 smithi038 ceph-mon[92422]: pgmap v20680: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 490 KiB/s wr, 10 op/s 2023-03-10T01:28:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:28:30 smithi038 ceph-mon[92422]: osdmap e1508: 12 total, 12 up, 12 in 2023-03-10T01:28:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:28:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37821.526572 secs 2023-03-10T01:28:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:28:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37821.526572 secs 2023-03-10T01:28:31.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:28:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37821.526572 secs 2023-03-10T01:28:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:28:32 smithi122 ceph-mon[93869]: pgmap v20682: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 232 KiB/s wr, 12 op/s 2023-03-10T01:28:32.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:28:32 smithi181 ceph-mon[94189]: pgmap v20682: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 232 KiB/s wr, 12 op/s 2023-03-10T01:28:32.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:28:32 smithi038 ceph-mon[92422]: pgmap v20682: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 232 KiB/s wr, 12 op/s 2023-03-10T01:28:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:28:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37823.987398 secs 2023-03-10T01:28:33.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:28:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37823.987398 secs 2023-03-10T01:28:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:28:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37823.987398 secs 2023-03-10T01:28:33.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38285.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:28:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:28:34 smithi122 ceph-mon[93869]: pgmap v20683: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 232 KiB/s wr, 12 op/s 2023-03-10T01:28:34.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:28:34 smithi181 ceph-mon[94189]: pgmap v20683: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 232 KiB/s wr, 12 op/s 2023-03-10T01:28:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:28:34 smithi038 ceph-mon[92422]: pgmap v20683: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 232 KiB/s wr, 12 op/s 2023-03-10T01:28:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:28:35 smithi122 ceph-mon[93869]: pgmap v20684: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 187 KiB/s wr, 10 op/s 2023-03-10T01:28:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:28:35 smithi181 ceph-mon[94189]: pgmap v20684: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 187 KiB/s wr, 10 op/s 2023-03-10T01:28:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:28:35 smithi038 ceph-mon[92422]: pgmap v20684: 129 pgs: 129 active+clean; 9.6 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 187 KiB/s wr, 10 op/s 2023-03-10T01:28:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:28:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37826.526647 secs 2023-03-10T01:28:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:28:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37826.526647 secs 2023-03-10T01:28:36.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:28:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37826.526647 secs 2023-03-10T01:28:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:28:37 smithi122 ceph-mon[93869]: pgmap v20685: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 956 KiB/s rd, 1.1 MiB/s wr, 12 op/s 2023-03-10T01:28:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:28:37 smithi181 ceph-mon[94189]: pgmap v20685: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 956 KiB/s rd, 1.1 MiB/s wr, 12 op/s 2023-03-10T01:28:37.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:28:37 smithi038 ceph-mon[92422]: pgmap v20685: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 956 KiB/s rd, 1.1 MiB/s wr, 12 op/s 2023-03-10T01:28:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38290.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:28:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:28:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37828.987484 secs 2023-03-10T01:28:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:28:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37828.987484 secs 2023-03-10T01:28:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:28:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37828.987484 secs 2023-03-10T01:28:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:28:39 smithi122 ceph-mon[93869]: pgmap v20686: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 956 KiB/s rd, 1.1 MiB/s wr, 12 op/s 2023-03-10T01:28:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:28:39 smithi181 ceph-mon[94189]: pgmap v20686: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 956 KiB/s rd, 1.1 MiB/s wr, 12 op/s 2023-03-10T01:28:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:28:39 smithi038 ceph-mon[92422]: pgmap v20686: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 956 KiB/s rd, 1.1 MiB/s wr, 12 op/s 2023-03-10T01:28:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:28:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37831.526704 secs 2023-03-10T01:28:41.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:28:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37831.526704 secs 2023-03-10T01:28:41.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:28:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37831.526704 secs 2023-03-10T01:28:42.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:28:41 smithi038 ceph-mon[92422]: pgmap v20687: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 14 op/s 2023-03-10T01:28:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:28:41 smithi122 ceph-mon[93869]: pgmap v20687: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 14 op/s 2023-03-10T01:28:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:28:41 smithi181 ceph-mon[94189]: pgmap v20687: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 14 op/s 2023-03-10T01:28:43.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:28:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37833.987547 secs 2023-03-10T01:28:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:28:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37833.987547 secs 2023-03-10T01:28:43.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:28:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37833.987547 secs 2023-03-10T01:28:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38295.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:28:44.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:28:43 smithi038 ceph-mon[92422]: pgmap v20688: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 11 op/s 2023-03-10T01:28:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:28:43 smithi122 ceph-mon[93869]: pgmap v20688: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 11 op/s 2023-03-10T01:28:44.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:28:43 smithi181 ceph-mon[94189]: pgmap v20688: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 11 op/s 2023-03-10T01:28:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:28:45 smithi122 ceph-mon[93869]: pgmap v20689: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 10 op/s 2023-03-10T01:28:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:28:45 smithi181 ceph-mon[94189]: pgmap v20689: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 10 op/s 2023-03-10T01:28:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:28:45 smithi038 ceph-mon[92422]: pgmap v20689: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 10 op/s 2023-03-10T01:28:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:28:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37836.526764 secs 2023-03-10T01:28:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:28:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37836.526764 secs 2023-03-10T01:28:46.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:28:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37836.526764 secs 2023-03-10T01:28:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:28:47 smithi122 ceph-mon[93869]: pgmap v20690: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 15 op/s 2023-03-10T01:28:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:28:47 smithi181 ceph-mon[94189]: pgmap v20690: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 15 op/s 2023-03-10T01:28:47.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:28:47 smithi038 ceph-mon[92422]: pgmap v20690: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 15 op/s 2023-03-10T01:28:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38300.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:28:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:28:50 smithi122 ceph-mon[93869]: pgmap v20691: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 591 KiB/s rd, 433 KiB/s wr, 10 op/s 2023-03-10T01:28:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:28:50 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:28:50.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:28:50 smithi181 ceph-mon[94189]: pgmap v20691: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 591 KiB/s rd, 433 KiB/s wr, 10 op/s 2023-03-10T01:28:50.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:28:50 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:28:50.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:28:50 smithi038 ceph-mon[92422]: pgmap v20691: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 591 KiB/s rd, 433 KiB/s wr, 10 op/s 2023-03-10T01:28:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:28:50 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:28:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:28:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37841.526821 secs 2023-03-10T01:28:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:28:51 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:28:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:28:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:28:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:28:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:28:51.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:28:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37841.526821 secs 2023-03-10T01:28:51.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:28:51 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:28:51.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:28:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:28:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:28:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:28:51.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:28:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37841.526821 secs 2023-03-10T01:28:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:28:51 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:28:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:28:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:28:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:28:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:28:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:28:52 smithi122 ceph-mon[93869]: pgmap v20692: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 739 KiB/s wr, 14 op/s 2023-03-10T01:28:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:28:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:28:52.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:28:52 smithi181 ceph-mon[94189]: pgmap v20692: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 739 KiB/s wr, 14 op/s 2023-03-10T01:28:52.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:28:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:28:52.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:28:52 smithi038 ceph-mon[92422]: pgmap v20692: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 739 KiB/s wr, 14 op/s 2023-03-10T01:28:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:28:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:28:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38305.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:28:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:28:54 smithi122 ceph-mon[93869]: pgmap v20693: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 818 KiB/s rd, 585 KiB/s wr, 11 op/s 2023-03-10T01:28:54.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:28:54 smithi181 ceph-mon[94189]: pgmap v20693: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 818 KiB/s rd, 585 KiB/s wr, 11 op/s 2023-03-10T01:28:54.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:28:54 smithi038 ceph-mon[92422]: pgmap v20693: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 818 KiB/s rd, 585 KiB/s wr, 11 op/s 2023-03-10T01:28:55.925 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:28:55 smithi038 ceph-mon[92422]: pgmap v20694: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 818 KiB/s rd, 581 KiB/s wr, 10 op/s 2023-03-10T01:28:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:28:55 smithi122 ceph-mon[93869]: pgmap v20694: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 818 KiB/s rd, 581 KiB/s wr, 10 op/s 2023-03-10T01:28:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:28:55 smithi181 ceph-mon[94189]: pgmap v20694: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 818 KiB/s rd, 581 KiB/s wr, 10 op/s 2023-03-10T01:28:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:28:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37846.526878 secs 2023-03-10T01:28:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:28:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:28:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:28:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:28:56.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:28:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:28:56.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:28:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:28:56.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:28:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:28:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:28:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37846.526878 secs 2023-03-10T01:28:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:28:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:28:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:28:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:28:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:28:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:28:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:28:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:28:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:28:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:28:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:28:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37846.526878 secs 2023-03-10T01:28:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:28:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:28:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:28:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:28:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:28:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:28:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:28:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:28:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:28:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:28:57.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:28:57 smithi122 ceph-mon[93869]: pgmap v20695: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 920 KiB/s rd, 681 KiB/s wr, 14 op/s 2023-03-10T01:28:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:28:57 smithi181 ceph-mon[94189]: pgmap v20695: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 920 KiB/s rd, 681 KiB/s wr, 14 op/s 2023-03-10T01:28:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:28:57 smithi038 ceph-mon[92422]: pgmap v20695: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 920 KiB/s rd, 681 KiB/s wr, 14 op/s 2023-03-10T01:28:58.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38310.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:29:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:29:00 smithi122 ceph-mon[93869]: pgmap v20696: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 821 KiB/s rd, 764 KiB/s wr, 11 op/s 2023-03-10T01:29:00.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:29:00 smithi181 ceph-mon[94189]: pgmap v20696: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 821 KiB/s rd, 764 KiB/s wr, 11 op/s 2023-03-10T01:29:00.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:29:00 smithi038 ceph-mon[92422]: pgmap v20696: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 821 KiB/s rd, 764 KiB/s wr, 11 op/s 2023-03-10T01:29:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:29:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37851.526935 secs 2023-03-10T01:29:01.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:29:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37851.526935 secs 2023-03-10T01:29:01.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:29:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37851.526935 secs 2023-03-10T01:29:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:29:02 smithi122 ceph-mon[93869]: pgmap v20697: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 855 KiB/s rd, 807 KiB/s wr, 16 op/s 2023-03-10T01:29:02.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:29:02 smithi181 ceph-mon[94189]: pgmap v20697: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 855 KiB/s rd, 807 KiB/s wr, 16 op/s 2023-03-10T01:29:02.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:29:02 smithi038 ceph-mon[92422]: pgmap v20697: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 855 KiB/s rd, 807 KiB/s wr, 16 op/s 2023-03-10T01:29:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38315.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:29:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:29:04 smithi122 ceph-mon[93869]: pgmap v20698: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 512 KiB/s wr, 23 op/s 2023-03-10T01:29:04.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:29:04 smithi181 ceph-mon[94189]: pgmap v20698: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 512 KiB/s wr, 23 op/s 2023-03-10T01:29:04.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:29:04 smithi038 ceph-mon[92422]: pgmap v20698: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 512 KiB/s wr, 23 op/s 2023-03-10T01:29:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:29:05 smithi122 ceph-mon[93869]: pgmap v20699: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 943 KiB/s rd, 838 KiB/s wr, 24 op/s 2023-03-10T01:29:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:29:05 smithi181 ceph-mon[94189]: pgmap v20699: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 943 KiB/s rd, 838 KiB/s wr, 24 op/s 2023-03-10T01:29:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:29:05 smithi038 ceph-mon[92422]: pgmap v20699: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 943 KiB/s rd, 838 KiB/s wr, 24 op/s 2023-03-10T01:29:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:29:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37856.526995 secs 2023-03-10T01:29:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:29:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37856.526995 secs 2023-03-10T01:29:06.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:29:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37856.526995 secs 2023-03-10T01:29:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:29:07 smithi122 ceph-mon[93869]: pgmap v20700: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 943 KiB/s wr, 28 op/s 2023-03-10T01:29:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:29:07 smithi181 ceph-mon[94189]: pgmap v20700: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 943 KiB/s wr, 28 op/s 2023-03-10T01:29:07.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:29:07 smithi038 ceph-mon[92422]: pgmap v20700: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 943 KiB/s wr, 28 op/s 2023-03-10T01:29:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38320.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:29:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:29:10 smithi122 ceph-mon[93869]: pgmap v20701: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1000 KiB/s rd, 844 KiB/s wr, 24 op/s 2023-03-10T01:29:10.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:29:10 smithi181 ceph-mon[94189]: pgmap v20701: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1000 KiB/s rd, 844 KiB/s wr, 24 op/s 2023-03-10T01:29:10.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:29:10 smithi038 ceph-mon[92422]: pgmap v20701: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1000 KiB/s rd, 844 KiB/s wr, 24 op/s 2023-03-10T01:29:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:29:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37861.527062 secs 2023-03-10T01:29:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:29:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37861.527062 secs 2023-03-10T01:29:11.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:29:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37861.527062 secs 2023-03-10T01:29:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:29:12 smithi122 ceph-mon[93869]: pgmap v20702: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1001 KiB/s rd, 662 KiB/s wr, 25 op/s 2023-03-10T01:29:12.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:29:12 smithi181 ceph-mon[94189]: pgmap v20702: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1001 KiB/s rd, 662 KiB/s wr, 25 op/s 2023-03-10T01:29:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:29:12 smithi038 ceph-mon[92422]: pgmap v20702: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1001 KiB/s rd, 662 KiB/s wr, 25 op/s 2023-03-10T01:29:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38325.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:29:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:29:14 smithi122 ceph-mon[93869]: pgmap v20703: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 967 KiB/s rd, 619 KiB/s wr, 20 op/s 2023-03-10T01:29:14.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:29:14 smithi181 ceph-mon[94189]: pgmap v20703: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 967 KiB/s rd, 619 KiB/s wr, 20 op/s 2023-03-10T01:29:14.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:29:14 smithi038 ceph-mon[92422]: pgmap v20703: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 967 KiB/s rd, 619 KiB/s wr, 20 op/s 2023-03-10T01:29:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:29:15 smithi122 ceph-mon[93869]: pgmap v20704: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 768 KiB/s rd, 608 KiB/s wr, 9 op/s 2023-03-10T01:29:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:29:15 smithi181 ceph-mon[94189]: pgmap v20704: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 768 KiB/s rd, 608 KiB/s wr, 9 op/s 2023-03-10T01:29:15.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:29:15 smithi038 ceph-mon[92422]: pgmap v20704: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 768 KiB/s rd, 608 KiB/s wr, 9 op/s 2023-03-10T01:29:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:29:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37866.527119 secs 2023-03-10T01:29:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:29:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37866.527119 secs 2023-03-10T01:29:16.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:29:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37866.527119 secs 2023-03-10T01:29:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:29:17 smithi122 ceph-mon[93869]: pgmap v20705: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 282 KiB/s wr, 10 op/s 2023-03-10T01:29:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:29:17 smithi181 ceph-mon[94189]: pgmap v20705: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 282 KiB/s wr, 10 op/s 2023-03-10T01:29:17.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:29:17 smithi038 ceph-mon[92422]: pgmap v20705: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 339 KiB/s rd, 282 KiB/s wr, 10 op/s 2023-03-10T01:29:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38330.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:29:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:29:20 smithi122 ceph-mon[93869]: pgmap v20706: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 187 KiB/s wr, 8 op/s 2023-03-10T01:29:20.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:29:20 smithi181 ceph-mon[94189]: pgmap v20706: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 187 KiB/s wr, 8 op/s 2023-03-10T01:29:20.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:29:20 smithi038 ceph-mon[92422]: pgmap v20706: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 187 KiB/s wr, 8 op/s 2023-03-10T01:29:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:29:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37871.527203 secs 2023-03-10T01:29:21.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:29:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37871.527203 secs 2023-03-10T01:29:21.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:29:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37871.527203 secs 2023-03-10T01:29:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:29:22 smithi122 ceph-mon[93869]: pgmap v20707: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 566 KiB/s wr, 12 op/s 2023-03-10T01:29:22.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:29:22 smithi181 ceph-mon[94189]: pgmap v20707: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 566 KiB/s wr, 12 op/s 2023-03-10T01:29:22.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:29:22 smithi038 ceph-mon[92422]: pgmap v20707: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 566 KiB/s wr, 12 op/s 2023-03-10T01:29:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38335.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:29:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:29:24 smithi122 ceph-mon[93869]: pgmap v20708: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 390 KiB/s wr, 9 op/s 2023-03-10T01:29:24.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:29:24 smithi181 ceph-mon[94189]: pgmap v20708: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 390 KiB/s wr, 9 op/s 2023-03-10T01:29:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:29:24 smithi038 ceph-mon[92422]: pgmap v20708: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 390 KiB/s wr, 9 op/s 2023-03-10T01:29:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:29:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37876.527261 secs 2023-03-10T01:29:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:29:26 smithi122 ceph-mon[93869]: pgmap v20709: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 390 KiB/s wr, 9 op/s 2023-03-10T01:29:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:29:26 smithi122 ceph-mon[93869]: osdmap e1509: 12 total, 12 up, 12 in 2023-03-10T01:29:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:29:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37876.527261 secs 2023-03-10T01:29:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:29:26 smithi181 ceph-mon[94189]: pgmap v20709: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 390 KiB/s wr, 9 op/s 2023-03-10T01:29:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:29:26 smithi181 ceph-mon[94189]: osdmap e1509: 12 total, 12 up, 12 in 2023-03-10T01:29:26.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:29:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37876.527261 secs 2023-03-10T01:29:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:29:26 smithi038 ceph-mon[92422]: pgmap v20709: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 390 KiB/s wr, 9 op/s 2023-03-10T01:29:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:29:26 smithi038 ceph-mon[92422]: osdmap e1509: 12 total, 12 up, 12 in 2023-03-10T01:29:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:29:28 smithi122 ceph-mon[93869]: pgmap v20711: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 629 KiB/s rd, 652 KiB/s wr, 12 op/s 2023-03-10T01:29:28.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:29:28 smithi181 ceph-mon[94189]: pgmap v20711: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 629 KiB/s rd, 652 KiB/s wr, 12 op/s 2023-03-10T01:29:28.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:29:28 smithi038 ceph-mon[92422]: pgmap v20711: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 629 KiB/s rd, 652 KiB/s wr, 12 op/s 2023-03-10T01:29:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38340.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:29:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:29:30 smithi122 ceph-mon[93869]: pgmap v20712: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 640 KiB/s wr, 10 op/s 2023-03-10T01:29:30.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:29:30 smithi181 ceph-mon[94189]: pgmap v20712: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 640 KiB/s wr, 10 op/s 2023-03-10T01:29:30.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:29:30 smithi038 ceph-mon[92422]: pgmap v20712: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 640 KiB/s wr, 10 op/s 2023-03-10T01:29:31.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:29:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37881.527317 secs 2023-03-10T01:29:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:29:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37881.527317 secs 2023-03-10T01:29:31.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:29:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37881.527317 secs 2023-03-10T01:29:32.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:29:32 smithi038 ceph-mon[92422]: pgmap v20713: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 186 KiB/s wr, 8 op/s 2023-03-10T01:29:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:29:32 smithi122 ceph-mon[93869]: pgmap v20713: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 186 KiB/s wr, 8 op/s 2023-03-10T01:29:32.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:29:32 smithi181 ceph-mon[94189]: pgmap v20713: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 186 KiB/s wr, 8 op/s 2023-03-10T01:29:33.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:29:33 smithi038 ceph-mon[92422]: pgmap v20714: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 463 KiB/s wr, 9 op/s 2023-03-10T01:29:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:29:33 smithi122 ceph-mon[93869]: pgmap v20714: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 463 KiB/s wr, 9 op/s 2023-03-10T01:29:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:29:33 smithi181 ceph-mon[94189]: pgmap v20714: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 187 KiB/s rd, 463 KiB/s wr, 9 op/s 2023-03-10T01:29:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38345.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:29:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:29:35 smithi122 ceph-mon[93869]: osdmap e1510: 12 total, 12 up, 12 in 2023-03-10T01:29:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:29:35 smithi122 ceph-mon[93869]: pgmap v20716: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 467 KiB/s wr, 9 op/s 2023-03-10T01:29:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:29:35 smithi181 ceph-mon[94189]: osdmap e1510: 12 total, 12 up, 12 in 2023-03-10T01:29:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:29:35 smithi181 ceph-mon[94189]: pgmap v20716: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 467 KiB/s wr, 9 op/s 2023-03-10T01:29:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:29:35 smithi038 ceph-mon[92422]: osdmap e1510: 12 total, 12 up, 12 in 2023-03-10T01:29:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:29:35 smithi038 ceph-mon[92422]: pgmap v20716: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 467 KiB/s wr, 9 op/s 2023-03-10T01:29:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:29:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37886.527377 secs 2023-03-10T01:29:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:29:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37886.527377 secs 2023-03-10T01:29:36.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:29:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37886.527377 secs 2023-03-10T01:29:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:29:37 smithi122 ceph-mon[93869]: pgmap v20717: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 675 KiB/s wr, 9 op/s 2023-03-10T01:29:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:29:37 smithi181 ceph-mon[94189]: pgmap v20717: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 675 KiB/s wr, 9 op/s 2023-03-10T01:29:37.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:29:37 smithi038 ceph-mon[92422]: pgmap v20717: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 675 KiB/s wr, 9 op/s 2023-03-10T01:29:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38350.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:29:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:29:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37888.988345 secs 2023-03-10T01:29:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:29:38 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:29:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:29:38 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:29:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:29:38 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:29:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:29:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37888.988345 secs 2023-03-10T01:29:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:29:38 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:29:38.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:29:38 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:29:38.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:29:38 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:29:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:29:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37888.988345 secs 2023-03-10T01:29:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:29:38 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:29:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:29:38 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:29:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:29:38 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:29:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:29:39 smithi122 ceph-mon[93869]: pgmap v20718: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 809 KiB/s rd, 807 KiB/s wr, 10 op/s 2023-03-10T01:29:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:29:39 smithi181 ceph-mon[94189]: pgmap v20718: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 809 KiB/s rd, 807 KiB/s wr, 10 op/s 2023-03-10T01:29:39.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:29:39 smithi038 ceph-mon[92422]: pgmap v20718: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 809 KiB/s rd, 807 KiB/s wr, 10 op/s 2023-03-10T01:29:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:29:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37891.527439 secs 2023-03-10T01:29:41.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:29:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37891.527439 secs 2023-03-10T01:29:41.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:29:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37891.527439 secs 2023-03-10T01:29:42.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:29:41 smithi038 ceph-mon[92422]: pgmap v20719: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 993 KiB/s rd, 990 KiB/s wr, 10 op/s 2023-03-10T01:29:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:29:41 smithi122 ceph-mon[93869]: pgmap v20719: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 993 KiB/s rd, 990 KiB/s wr, 10 op/s 2023-03-10T01:29:42.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:29:41 smithi181 ceph-mon[94189]: pgmap v20719: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 993 KiB/s rd, 990 KiB/s wr, 10 op/s 2023-03-10T01:29:43.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:29:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37893.988401 secs 2023-03-10T01:29:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:29:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37893.988401 secs 2023-03-10T01:29:43.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:29:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37893.988401 secs 2023-03-10T01:29:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38355.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:29:44.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:29:43 smithi038 ceph-mon[92422]: pgmap v20720: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1015 KiB/s rd, 736 KiB/s wr, 11 op/s 2023-03-10T01:29:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:29:43 smithi122 ceph-mon[93869]: pgmap v20720: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1015 KiB/s rd, 736 KiB/s wr, 11 op/s 2023-03-10T01:29:44.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:29:43 smithi181 ceph-mon[94189]: pgmap v20720: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1015 KiB/s rd, 736 KiB/s wr, 11 op/s 2023-03-10T01:29:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:29:45 smithi122 ceph-mon[93869]: pgmap v20721: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 969 KiB/s rd, 703 KiB/s wr, 10 op/s 2023-03-10T01:29:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:29:45 smithi181 ceph-mon[94189]: pgmap v20721: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 969 KiB/s rd, 703 KiB/s wr, 10 op/s 2023-03-10T01:29:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:29:45 smithi038 ceph-mon[92422]: pgmap v20721: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 969 KiB/s rd, 703 KiB/s wr, 10 op/s 2023-03-10T01:29:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:29:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37896.527518 secs 2023-03-10T01:29:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:29:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37896.527518 secs 2023-03-10T01:29:46.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:29:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37896.527518 secs 2023-03-10T01:29:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:29:47 smithi122 ceph-mon[93869]: pgmap v20722: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 991 KiB/s rd, 757 KiB/s wr, 12 op/s 2023-03-10T01:29:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:29:47 smithi181 ceph-mon[94189]: pgmap v20722: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 991 KiB/s rd, 757 KiB/s wr, 12 op/s 2023-03-10T01:29:47.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:29:47 smithi038 ceph-mon[92422]: pgmap v20722: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 991 KiB/s rd, 757 KiB/s wr, 12 op/s 2023-03-10T01:29:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38360.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:29:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:29:50 smithi122 ceph-mon[93869]: pgmap v20723: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 670 KiB/s rd, 437 KiB/s wr, 9 op/s 2023-03-10T01:29:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:29:50 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:29:50.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:29:50 smithi181 ceph-mon[94189]: pgmap v20723: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 670 KiB/s rd, 437 KiB/s wr, 9 op/s 2023-03-10T01:29:50.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:29:50 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:29:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:29:50 smithi038 ceph-mon[92422]: pgmap v20723: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 670 KiB/s rd, 437 KiB/s wr, 9 op/s 2023-03-10T01:29:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:29:50 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:29:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:29:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37901.527574 secs 2023-03-10T01:29:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:29:51 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:29:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:29:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:29:51.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:29:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:29:51.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:29:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37901.527574 secs 2023-03-10T01:29:51.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:29:51 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:29:51.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:29:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:29:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:29:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:29:51.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:29:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37901.527574 secs 2023-03-10T01:29:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:29:51 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:29:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:29:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:29:51.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:29:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:29:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:29:52 smithi122 ceph-mon[93869]: pgmap v20724: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 362 KiB/s rd, 358 KiB/s wr, 11 op/s 2023-03-10T01:29:52.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:29:52 smithi181 ceph-mon[94189]: pgmap v20724: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 362 KiB/s rd, 358 KiB/s wr, 11 op/s 2023-03-10T01:29:52.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:29:52 smithi038 ceph-mon[92422]: pgmap v20724: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 362 KiB/s rd, 358 KiB/s wr, 11 op/s 2023-03-10T01:29:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:29:53 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 37903.988547 secs 2023-03-10T01:29:53.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:29:53 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 37903.988547 secs 2023-03-10T01:29:53.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:29:53 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 37903.988547 secs 2023-03-10T01:29:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38365.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:29:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:29:54 smithi122 ceph-mon[93869]: pgmap v20725: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 519 KiB/s rd, 517 KiB/s wr, 9 op/s 2023-03-10T01:29:54.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:29:54 smithi181 ceph-mon[94189]: pgmap v20725: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 519 KiB/s rd, 517 KiB/s wr, 9 op/s 2023-03-10T01:29:54.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:29:54 smithi038 ceph-mon[92422]: pgmap v20725: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 519 KiB/s rd, 517 KiB/s wr, 9 op/s 2023-03-10T01:29:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:29:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:29:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:29:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:29:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:29:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:29:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:29:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:29:55.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:29:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:29:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:29:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:29:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:29:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37906.527650 secs 2023-03-10T01:29:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:29:56 smithi122 ceph-mon[93869]: pgmap v20726: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 508 KiB/s rd, 506 KiB/s wr, 9 op/s 2023-03-10T01:29:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:29:56 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:29:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:29:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37906.527650 secs 2023-03-10T01:29:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:29:56 smithi181 ceph-mon[94189]: pgmap v20726: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 508 KiB/s rd, 506 KiB/s wr, 9 op/s 2023-03-10T01:29:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:29:56 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:29:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:29:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37906.527650 secs 2023-03-10T01:29:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:29:56 smithi038 ceph-mon[92422]: pgmap v20726: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 508 KiB/s rd, 506 KiB/s wr, 9 op/s 2023-03-10T01:29:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:29:56 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:29:58.454 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:29:58 smithi181 ceph-mon[94189]: pgmap v20727: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 543 KiB/s rd, 538 KiB/s wr, 12 op/s 2023-03-10T01:29:58.454 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:29:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37908.988653 secs 2023-03-10T01:29:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:29:58 smithi122 ceph-mon[93869]: pgmap v20727: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 543 KiB/s rd, 538 KiB/s wr, 12 op/s 2023-03-10T01:29:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:29:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37908.988653 secs 2023-03-10T01:29:58.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:29:58 smithi038 ceph-mon[92422]: pgmap v20727: 129 pgs: 129 active+clean; 9.6 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 543 KiB/s rd, 538 KiB/s wr, 12 op/s 2023-03-10T01:29:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:29:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37908.988653 secs 2023-03-10T01:29:58.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38370.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:29:59.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:29:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:29:59.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:29:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:29:59.698 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:29:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:29:59.698 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:29:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:29:59.723 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:29:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:29:59.723 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:29:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:30:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:30:00 smithi181 ceph-mon[94189]: pgmap v20728: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 743 KiB/s rd, 728 KiB/s wr, 11 op/s 2023-03-10T01:30:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:30:00 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:30:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:30:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:30:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:30:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:30:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:30:00 smithi181 ceph-mon[94189]: Health detail: HEALTH_WARN 2 MDSs report slow requests 2023-03-10T01:30:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:30:00 smithi181 ceph-mon[94189]: [WRN] MDS_SLOW_REQUEST: 2 MDSs report slow requests 2023-03-10T01:30:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:30:00 smithi181 ceph-mon[94189]: mds.i(mds.2): 2 slow requests are blocked > 180 secs 2023-03-10T01:30:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:30:00 smithi181 ceph-mon[94189]: mds.h(mds.1): 2 slow requests are blocked > 180 secs 2023-03-10T01:30:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:30:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:30:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:30:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:30:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:30:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:30:00.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:30:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:30:00.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:30:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:30:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:30:00 smithi038 ceph-mon[92422]: pgmap v20728: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 743 KiB/s rd, 728 KiB/s wr, 11 op/s 2023-03-10T01:30:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:30:00 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:30:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:30:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:30:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:30:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:30:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:30:00 smithi038 ceph-mon[92422]: Health detail: HEALTH_WARN 2 MDSs report slow requests 2023-03-10T01:30:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:30:00 smithi038 ceph-mon[92422]: [WRN] MDS_SLOW_REQUEST: 2 MDSs report slow requests 2023-03-10T01:30:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:30:00 smithi038 ceph-mon[92422]: mds.i(mds.2): 2 slow requests are blocked > 180 secs 2023-03-10T01:30:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:30:00 smithi038 ceph-mon[92422]: mds.h(mds.1): 2 slow requests are blocked > 180 secs 2023-03-10T01:30:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:30:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:30:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:30:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:30:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:30:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:30:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:30:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:30:00.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:30:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:30:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:30:00 smithi122 ceph-mon[93869]: pgmap v20728: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 743 KiB/s rd, 728 KiB/s wr, 11 op/s 2023-03-10T01:30:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:30:00 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:30:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:30:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:30:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:30:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:30:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:30:00 smithi122 ceph-mon[93869]: Health detail: HEALTH_WARN 2 MDSs report slow requests 2023-03-10T01:30:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:30:00 smithi122 ceph-mon[93869]: [WRN] MDS_SLOW_REQUEST: 2 MDSs report slow requests 2023-03-10T01:30:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:30:00 smithi122 ceph-mon[93869]: mds.i(mds.2): 2 slow requests are blocked > 180 secs 2023-03-10T01:30:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:30:00 smithi122 ceph-mon[93869]: mds.h(mds.1): 2 slow requests are blocked > 180 secs 2023-03-10T01:30:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:30:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:30:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:30:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:30:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:30:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:30:00.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:30:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:30:00.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:30:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:30:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:30:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37911.527708 secs 2023-03-10T01:30:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:30:01 smithi122 ceph-mon[93869]: pgmap v20729: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 994 KiB/s rd, 978 KiB/s wr, 14 op/s 2023-03-10T01:30:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:30:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37911.527708 secs 2023-03-10T01:30:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:30:01 smithi181 ceph-mon[94189]: pgmap v20729: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 994 KiB/s rd, 978 KiB/s wr, 14 op/s 2023-03-10T01:30:01.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:30:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37911.527708 secs 2023-03-10T01:30:01.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:30:01 smithi038 ceph-mon[92422]: pgmap v20729: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 994 KiB/s rd, 978 KiB/s wr, 14 op/s 2023-03-10T01:30:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:30:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37913.988715 secs 2023-03-10T01:30:03.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:30:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37913.988715 secs 2023-03-10T01:30:03.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:30:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37913.988715 secs 2023-03-10T01:30:03.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38375.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:30:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:30:04 smithi122 ceph-mon[93869]: pgmap v20730: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 999 KiB/s rd, 988 KiB/s wr, 23 op/s 2023-03-10T01:30:04.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:30:04 smithi181 ceph-mon[94189]: pgmap v20730: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 999 KiB/s rd, 988 KiB/s wr, 23 op/s 2023-03-10T01:30:04.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:30:04 smithi038 ceph-mon[92422]: pgmap v20730: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 999 KiB/s rd, 988 KiB/s wr, 23 op/s 2023-03-10T01:30:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:30:05 smithi122 ceph-mon[93869]: pgmap v20731: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 676 KiB/s wr, 21 op/s 2023-03-10T01:30:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:30:05 smithi181 ceph-mon[94189]: pgmap v20731: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 676 KiB/s wr, 21 op/s 2023-03-10T01:30:05.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:30:05 smithi038 ceph-mon[92422]: pgmap v20731: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 676 KiB/s wr, 21 op/s 2023-03-10T01:30:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:30:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37916.527823 secs 2023-03-10T01:30:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:30:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37916.527823 secs 2023-03-10T01:30:06.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:30:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37916.527823 secs 2023-03-10T01:30:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:30:07 smithi122 ceph-mon[93869]: pgmap v20732: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 968 KiB/s rd, 898 KiB/s wr, 26 op/s 2023-03-10T01:30:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:30:07 smithi181 ceph-mon[94189]: pgmap v20732: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 968 KiB/s rd, 898 KiB/s wr, 26 op/s 2023-03-10T01:30:07.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:30:07 smithi038 ceph-mon[92422]: pgmap v20732: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 968 KiB/s rd, 898 KiB/s wr, 26 op/s 2023-03-10T01:30:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38380.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:30:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:30:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37918.988801 secs 2023-03-10T01:30:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:30:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37918.988801 secs 2023-03-10T01:30:08.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:30:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37918.988801 secs 2023-03-10T01:30:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:30:09 smithi122 ceph-mon[93869]: pgmap v20733: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 933 KiB/s rd, 866 KiB/s wr, 24 op/s 2023-03-10T01:30:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:30:09 smithi181 ceph-mon[94189]: pgmap v20733: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 933 KiB/s rd, 866 KiB/s wr, 24 op/s 2023-03-10T01:30:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:30:09 smithi038 ceph-mon[92422]: pgmap v20733: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 933 KiB/s rd, 866 KiB/s wr, 24 op/s 2023-03-10T01:30:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:30:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37921.527880 secs 2023-03-10T01:30:11.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:30:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37921.527880 secs 2023-03-10T01:30:11.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:30:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37921.527880 secs 2023-03-10T01:30:12.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:30:11 smithi038 ceph-mon[92422]: pgmap v20734: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 589 KiB/s rd, 532 KiB/s wr, 24 op/s 2023-03-10T01:30:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:30:11 smithi122 ceph-mon[93869]: pgmap v20734: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 589 KiB/s rd, 532 KiB/s wr, 24 op/s 2023-03-10T01:30:12.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:30:11 smithi181 ceph-mon[94189]: pgmap v20734: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 589 KiB/s rd, 532 KiB/s wr, 24 op/s 2023-03-10T01:30:13.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:30:12 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37923.988858 secs 2023-03-10T01:30:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:30:12 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37923.988858 secs 2023-03-10T01:30:13.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:30:12 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37923.988858 secs 2023-03-10T01:30:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38385.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:30:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:30:14 smithi122 ceph-mon[93869]: pgmap v20735: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 661 KiB/s rd, 606 KiB/s wr, 23 op/s 2023-03-10T01:30:14.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:30:14 smithi181 ceph-mon[94189]: pgmap v20735: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 661 KiB/s rd, 606 KiB/s wr, 23 op/s 2023-03-10T01:30:14.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:30:14 smithi038 ceph-mon[92422]: pgmap v20735: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 661 KiB/s rd, 606 KiB/s wr, 23 op/s 2023-03-10T01:30:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:30:15 smithi122 ceph-mon[93869]: pgmap v20736: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 623 KiB/s rd, 563 KiB/s wr, 11 op/s 2023-03-10T01:30:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:30:15 smithi181 ceph-mon[94189]: pgmap v20736: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 623 KiB/s rd, 563 KiB/s wr, 11 op/s 2023-03-10T01:30:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:30:15 smithi038 ceph-mon[92422]: pgmap v20736: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 623 KiB/s rd, 563 KiB/s wr, 11 op/s 2023-03-10T01:30:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:30:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37926.527938 secs 2023-03-10T01:30:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:30:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37926.527938 secs 2023-03-10T01:30:16.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:30:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37926.527938 secs 2023-03-10T01:30:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:30:17 smithi122 ceph-mon[93869]: pgmap v20737: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 563 KiB/s wr, 13 op/s 2023-03-10T01:30:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:30:17 smithi181 ceph-mon[94189]: pgmap v20737: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 563 KiB/s wr, 13 op/s 2023-03-10T01:30:17.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:30:17 smithi038 ceph-mon[92422]: pgmap v20737: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 563 KiB/s wr, 13 op/s 2023-03-10T01:30:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38390.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:30:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:30:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37928.988915 secs 2023-03-10T01:30:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:30:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37928.988915 secs 2023-03-10T01:30:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:30:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37928.988915 secs 2023-03-10T01:30:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:30:19 smithi122 ceph-mon[93869]: pgmap v20738: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 337 KiB/s rd, 376 KiB/s wr, 10 op/s 2023-03-10T01:30:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:30:19 smithi181 ceph-mon[94189]: pgmap v20738: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 337 KiB/s rd, 376 KiB/s wr, 10 op/s 2023-03-10T01:30:19.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:30:19 smithi038 ceph-mon[92422]: pgmap v20738: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 337 KiB/s rd, 376 KiB/s wr, 10 op/s 2023-03-10T01:30:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:30:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37931.527993 secs 2023-03-10T01:30:21.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:30:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37931.527993 secs 2023-03-10T01:30:21.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:30:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37931.527993 secs 2023-03-10T01:30:22.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:30:21 smithi038 ceph-mon[92422]: pgmap v20739: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 538 KiB/s wr, 12 op/s 2023-03-10T01:30:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:30:21 smithi122 ceph-mon[93869]: pgmap v20739: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 538 KiB/s wr, 12 op/s 2023-03-10T01:30:22.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:30:21 smithi181 ceph-mon[94189]: pgmap v20739: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 538 KiB/s wr, 12 op/s 2023-03-10T01:30:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:30:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37933.988972 secs 2023-03-10T01:30:23.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:30:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37933.988972 secs 2023-03-10T01:30:23.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:30:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37933.988972 secs 2023-03-10T01:30:23.793 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38395.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:30:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:30:24 smithi122 ceph-mon[93869]: pgmap v20740: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 752 KiB/s wr, 12 op/s 2023-03-10T01:30:24.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:30:24 smithi181 ceph-mon[94189]: pgmap v20740: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 752 KiB/s wr, 12 op/s 2023-03-10T01:30:24.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:30:24 smithi038 ceph-mon[92422]: pgmap v20740: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 752 KiB/s wr, 12 op/s 2023-03-10T01:30:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:30:26 smithi122 ceph-mon[93869]: osdmap e1511: 12 total, 12 up, 12 in 2023-03-10T01:30:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:30:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37936.528055 secs 2023-03-10T01:30:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:30:26 smithi122 ceph-mon[93869]: pgmap v20742: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 478 KiB/s rd, 502 KiB/s wr, 12 op/s 2023-03-10T01:30:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:30:26 smithi181 ceph-mon[94189]: osdmap e1511: 12 total, 12 up, 12 in 2023-03-10T01:30:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:30:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37936.528055 secs 2023-03-10T01:30:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:30:26 smithi181 ceph-mon[94189]: pgmap v20742: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 478 KiB/s rd, 502 KiB/s wr, 12 op/s 2023-03-10T01:30:26.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:30:26 smithi038 ceph-mon[92422]: osdmap e1511: 12 total, 12 up, 12 in 2023-03-10T01:30:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:30:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37936.528055 secs 2023-03-10T01:30:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:30:26 smithi038 ceph-mon[92422]: pgmap v20742: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 478 KiB/s rd, 502 KiB/s wr, 12 op/s 2023-03-10T01:30:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:30:28 smithi122 ceph-mon[93869]: pgmap v20743: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 662 KiB/s rd, 687 KiB/s wr, 13 op/s 2023-03-10T01:30:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:30:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37938.989087 secs 2023-03-10T01:30:28.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:30:28 smithi181 ceph-mon[94189]: pgmap v20743: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 662 KiB/s rd, 687 KiB/s wr, 13 op/s 2023-03-10T01:30:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:30:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37938.989087 secs 2023-03-10T01:30:28.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:30:28 smithi038 ceph-mon[92422]: pgmap v20743: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 662 KiB/s rd, 687 KiB/s wr, 13 op/s 2023-03-10T01:30:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:30:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37938.989087 secs 2023-03-10T01:30:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38400.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:30:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:30:30 smithi122 ceph-mon[93869]: pgmap v20744: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 662 KiB/s rd, 674 KiB/s wr, 12 op/s 2023-03-10T01:30:30.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:30:30 smithi181 ceph-mon[94189]: pgmap v20744: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 662 KiB/s rd, 674 KiB/s wr, 12 op/s 2023-03-10T01:30:30.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:30:30 smithi038 ceph-mon[92422]: pgmap v20744: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 662 KiB/s rd, 674 KiB/s wr, 12 op/s 2023-03-10T01:30:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:30:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37941.528111 secs 2023-03-10T01:30:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:30:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37941.528111 secs 2023-03-10T01:30:31.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:30:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37941.528111 secs 2023-03-10T01:30:32.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:30:32 smithi181 ceph-mon[94189]: pgmap v20745: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 526 KiB/s rd, 525 KiB/s wr, 10 op/s 2023-03-10T01:30:32.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:30:32 smithi038 ceph-mon[92422]: pgmap v20745: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 526 KiB/s rd, 525 KiB/s wr, 10 op/s 2023-03-10T01:30:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:30:32 smithi122 ceph-mon[93869]: pgmap v20745: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 526 KiB/s rd, 525 KiB/s wr, 10 op/s 2023-03-10T01:30:33.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:30:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37943.989146 secs 2023-03-10T01:30:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:30:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37943.989146 secs 2023-03-10T01:30:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:30:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37943.989146 secs 2023-03-10T01:30:33.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38405.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:30:34.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:30:34 smithi038 ceph-mon[92422]: pgmap v20746: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 306 KiB/s wr, 8 op/s 2023-03-10T01:30:34.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:30:34 smithi122 ceph-mon[93869]: pgmap v20746: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 306 KiB/s wr, 8 op/s 2023-03-10T01:30:34.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:30:34 smithi181 ceph-mon[94189]: pgmap v20746: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 306 KiB/s wr, 8 op/s 2023-03-10T01:30:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:30:35 smithi122 ceph-mon[93869]: osdmap e1512: 12 total, 12 up, 12 in 2023-03-10T01:30:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:30:35 smithi122 ceph-mon[93869]: pgmap v20748: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 307 KiB/s wr, 8 op/s 2023-03-10T01:30:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:30:35 smithi181 ceph-mon[94189]: osdmap e1512: 12 total, 12 up, 12 in 2023-03-10T01:30:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:30:35 smithi181 ceph-mon[94189]: pgmap v20748: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 307 KiB/s wr, 8 op/s 2023-03-10T01:30:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:30:35 smithi038 ceph-mon[92422]: osdmap e1512: 12 total, 12 up, 12 in 2023-03-10T01:30:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:30:35 smithi038 ceph-mon[92422]: pgmap v20748: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 270 KiB/s rd, 307 KiB/s wr, 8 op/s 2023-03-10T01:30:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:30:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37946.528170 secs 2023-03-10T01:30:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:30:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37946.528170 secs 2023-03-10T01:30:36.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:30:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37946.528170 secs 2023-03-10T01:30:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:30:37 smithi122 ceph-mon[93869]: pgmap v20749: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 911 KiB/s rd, 812 KiB/s wr, 9 op/s 2023-03-10T01:30:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:30:37 smithi181 ceph-mon[94189]: pgmap v20749: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 911 KiB/s rd, 812 KiB/s wr, 9 op/s 2023-03-10T01:30:37.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:30:37 smithi038 ceph-mon[92422]: pgmap v20749: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 911 KiB/s rd, 812 KiB/s wr, 9 op/s 2023-03-10T01:30:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38410.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:30:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:30:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37948.989206 secs 2023-03-10T01:30:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:30:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37948.989206 secs 2023-03-10T01:30:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:30:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37948.989206 secs 2023-03-10T01:30:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:30:39 smithi122 ceph-mon[93869]: pgmap v20750: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1007 KiB/s rd, 832 KiB/s wr, 10 op/s 2023-03-10T01:30:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:30:39 smithi181 ceph-mon[94189]: pgmap v20750: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1007 KiB/s rd, 832 KiB/s wr, 10 op/s 2023-03-10T01:30:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:30:39 smithi038 ceph-mon[92422]: pgmap v20750: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1007 KiB/s rd, 832 KiB/s wr, 10 op/s 2023-03-10T01:30:41.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:30:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37951.528229 secs 2023-03-10T01:30:41.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:30:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37951.528229 secs 2023-03-10T01:30:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:30:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37951.528229 secs 2023-03-10T01:30:42.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:30:41 smithi038 ceph-mon[92422]: pgmap v20751: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 962 KiB/s rd, 787 KiB/s wr, 10 op/s 2023-03-10T01:30:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:30:41 smithi122 ceph-mon[93869]: pgmap v20751: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 962 KiB/s rd, 787 KiB/s wr, 10 op/s 2023-03-10T01:30:42.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:30:41 smithi181 ceph-mon[94189]: pgmap v20751: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 962 KiB/s rd, 787 KiB/s wr, 10 op/s 2023-03-10T01:30:43.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:30:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37953.989268 secs 2023-03-10T01:30:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:30:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37953.989268 secs 2023-03-10T01:30:43.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:30:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37953.989268 secs 2023-03-10T01:30:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38415.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:30:44.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:30:43 smithi038 ceph-mon[92422]: pgmap v20752: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 11 op/s 2023-03-10T01:30:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:30:43 smithi122 ceph-mon[93869]: pgmap v20752: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 11 op/s 2023-03-10T01:30:44.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:30:43 smithi181 ceph-mon[94189]: pgmap v20752: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 11 op/s 2023-03-10T01:30:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:30:45 smithi122 ceph-mon[93869]: pgmap v20753: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 11 op/s 2023-03-10T01:30:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:30:45 smithi181 ceph-mon[94189]: pgmap v20753: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 11 op/s 2023-03-10T01:30:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:30:45 smithi038 ceph-mon[92422]: pgmap v20753: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 11 op/s 2023-03-10T01:30:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:30:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37956.528288 secs 2023-03-10T01:30:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:30:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37956.528288 secs 2023-03-10T01:30:46.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:30:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37956.528288 secs 2023-03-10T01:30:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:30:47 smithi122 ceph-mon[93869]: pgmap v20754: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 12 op/s 2023-03-10T01:30:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:30:47 smithi181 ceph-mon[94189]: pgmap v20754: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 12 op/s 2023-03-10T01:30:47.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:30:47 smithi038 ceph-mon[92422]: pgmap v20754: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 12 op/s 2023-03-10T01:30:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38420.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:30:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:30:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37958.989331 secs 2023-03-10T01:30:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:30:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37958.989331 secs 2023-03-10T01:30:48.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:30:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37958.989331 secs 2023-03-10T01:30:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:30:49 smithi122 ceph-mon[93869]: pgmap v20755: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 555 KiB/s rd, 521 KiB/s wr, 9 op/s 2023-03-10T01:30:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:30:49 smithi181 ceph-mon[94189]: pgmap v20755: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 555 KiB/s rd, 521 KiB/s wr, 9 op/s 2023-03-10T01:30:49.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:30:49 smithi038 ceph-mon[92422]: pgmap v20755: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 555 KiB/s rd, 521 KiB/s wr, 9 op/s 2023-03-10T01:30:51.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:30:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37961.528344 secs 2023-03-10T01:30:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:30:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37961.528344 secs 2023-03-10T01:30:51.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:30:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37961.528344 secs 2023-03-10T01:30:52.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:30:51 smithi038 ceph-mon[92422]: pgmap v20756: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 746 KiB/s rd, 743 KiB/s wr, 10 op/s 2023-03-10T01:30:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:30:51 smithi122 ceph-mon[93869]: pgmap v20756: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 746 KiB/s rd, 743 KiB/s wr, 10 op/s 2023-03-10T01:30:52.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:30:51 smithi181 ceph-mon[94189]: pgmap v20756: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 746 KiB/s rd, 743 KiB/s wr, 10 op/s 2023-03-10T01:30:53.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:30:52 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37963.989438 secs 2023-03-10T01:30:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:30:52 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37963.989438 secs 2023-03-10T01:30:53.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:30:52 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37963.989438 secs 2023-03-10T01:30:53.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38425.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:30:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:30:54 smithi122 ceph-mon[93869]: pgmap v20757: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 829 KiB/s rd, 827 KiB/s wr, 10 op/s 2023-03-10T01:30:54.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:30:54 smithi181 ceph-mon[94189]: pgmap v20757: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 829 KiB/s rd, 827 KiB/s wr, 10 op/s 2023-03-10T01:30:54.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:30:54 smithi038 ceph-mon[92422]: pgmap v20757: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 829 KiB/s rd, 827 KiB/s wr, 10 op/s 2023-03-10T01:30:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:30:55 smithi122 ceph-mon[93869]: pgmap v20758: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 426 KiB/s wr, 8 op/s 2023-03-10T01:30:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:30:55 smithi181 ceph-mon[94189]: pgmap v20758: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 426 KiB/s wr, 8 op/s 2023-03-10T01:30:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:30:55 smithi038 ceph-mon[92422]: pgmap v20758: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 426 KiB/s wr, 8 op/s 2023-03-10T01:30:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:30:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37966.528404 secs 2023-03-10T01:30:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:30:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37966.528404 secs 2023-03-10T01:30:56.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:30:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37966.528404 secs 2023-03-10T01:30:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:30:57 smithi122 ceph-mon[93869]: pgmap v20759: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 571 KiB/s rd, 427 KiB/s wr, 11 op/s 2023-03-10T01:30:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:30:57 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:30:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:30:57 smithi181 ceph-mon[94189]: pgmap v20759: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 571 KiB/s rd, 427 KiB/s wr, 11 op/s 2023-03-10T01:30:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:30:57 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:30:57.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:30:57 smithi038 ceph-mon[92422]: pgmap v20759: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 571 KiB/s rd, 427 KiB/s wr, 11 op/s 2023-03-10T01:30:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:30:57 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:30:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38430.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:30:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:30:58 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:30:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:30:58 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:30:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:30:58 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:30:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:30:58 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:30:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:30:58 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:30:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:30:58 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:30:58.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:30:58 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:30:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:30:58 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:30:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:30:58 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:30:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:30:59 smithi122 ceph-mon[93869]: pgmap v20760: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 761 KiB/s rd, 677 KiB/s wr, 11 op/s 2023-03-10T01:30:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:30:59 smithi181 ceph-mon[94189]: pgmap v20760: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 761 KiB/s rd, 677 KiB/s wr, 11 op/s 2023-03-10T01:30:59.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:30:59 smithi038 ceph-mon[92422]: pgmap v20760: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 761 KiB/s rd, 677 KiB/s wr, 11 op/s 2023-03-10T01:31:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:31:00 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:31:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:31:00 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:31:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:31:00 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:31:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:31:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37971.528493 secs 2023-03-10T01:31:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:31:01 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:31:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:31:01 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:31:01.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:31:01 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:31:01.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:31:01 smithi122 ceph-mon[93869]: pgmap v20761: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 622 KiB/s rd, 676 KiB/s wr, 11 op/s 2023-03-10T01:31:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:31:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37971.528493 secs 2023-03-10T01:31:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:31:01 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:31:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:31:01 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:31:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:31:01 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:31:01.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:31:01 smithi181 ceph-mon[94189]: pgmap v20761: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 622 KiB/s rd, 676 KiB/s wr, 11 op/s 2023-03-10T01:31:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:31:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37971.528493 secs 2023-03-10T01:31:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:31:01 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:31:02.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:31:01 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:31:02.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:31:01 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:31:02.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:31:01 smithi038 ceph-mon[92422]: pgmap v20761: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 622 KiB/s rd, 676 KiB/s wr, 11 op/s 2023-03-10T01:31:03.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38435.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:31:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:31:04 smithi122 ceph-mon[93869]: pgmap v20762: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 811 KiB/s rd, 790 KiB/s wr, 23 op/s 2023-03-10T01:31:04.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:31:04 smithi181 ceph-mon[94189]: pgmap v20762: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 811 KiB/s rd, 790 KiB/s wr, 23 op/s 2023-03-10T01:31:04.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:31:04 smithi038 ceph-mon[92422]: pgmap v20762: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 811 KiB/s rd, 790 KiB/s wr, 23 op/s 2023-03-10T01:31:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:31:05 smithi122 ceph-mon[93869]: pgmap v20763: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 800 KiB/s rd, 779 KiB/s wr, 21 op/s 2023-03-10T01:31:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:31:05 smithi181 ceph-mon[94189]: pgmap v20763: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 800 KiB/s rd, 779 KiB/s wr, 21 op/s 2023-03-10T01:31:05.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:31:05 smithi038 ceph-mon[92422]: pgmap v20763: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 800 KiB/s rd, 779 KiB/s wr, 21 op/s 2023-03-10T01:31:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:31:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37976.528552 secs 2023-03-10T01:31:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:31:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37976.528552 secs 2023-03-10T01:31:06.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:31:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37976.528552 secs 2023-03-10T01:31:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:31:07 smithi122 ceph-mon[93869]: pgmap v20764: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 802 KiB/s rd, 790 KiB/s wr, 25 op/s 2023-03-10T01:31:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:31:07 smithi181 ceph-mon[94189]: pgmap v20764: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 802 KiB/s rd, 790 KiB/s wr, 25 op/s 2023-03-10T01:31:07.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:31:07 smithi038 ceph-mon[92422]: pgmap v20764: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 802 KiB/s rd, 790 KiB/s wr, 25 op/s 2023-03-10T01:31:08.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38440.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:31:10.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:31:10 smithi038 ceph-mon[92422]: pgmap v20765: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 958 KiB/s rd, 949 KiB/s wr, 24 op/s 2023-03-10T01:31:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:31:10 smithi122 ceph-mon[93869]: pgmap v20765: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 958 KiB/s rd, 949 KiB/s wr, 24 op/s 2023-03-10T01:31:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:31:10 smithi181 ceph-mon[94189]: pgmap v20765: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 958 KiB/s rd, 949 KiB/s wr, 24 op/s 2023-03-10T01:31:11.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:31:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37981.528637 secs 2023-03-10T01:31:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:31:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37981.528637 secs 2023-03-10T01:31:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:31:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37981.528637 secs 2023-03-10T01:31:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:31:12 smithi038 ceph-mon[92422]: pgmap v20766: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 628 KiB/s wr, 23 op/s 2023-03-10T01:31:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:31:12 smithi122 ceph-mon[93869]: pgmap v20766: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 628 KiB/s wr, 23 op/s 2023-03-10T01:31:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:31:12 smithi181 ceph-mon[94189]: pgmap v20766: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 628 KiB/s wr, 23 op/s 2023-03-10T01:31:13.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38445.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:31:14.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:31:14 smithi038 ceph-mon[92422]: pgmap v20767: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 765 KiB/s rd, 696 KiB/s wr, 23 op/s 2023-03-10T01:31:14.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:31:14 smithi122 ceph-mon[93869]: pgmap v20767: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 765 KiB/s rd, 696 KiB/s wr, 23 op/s 2023-03-10T01:31:14.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:31:14 smithi181 ceph-mon[94189]: pgmap v20767: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 765 KiB/s rd, 696 KiB/s wr, 23 op/s 2023-03-10T01:31:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:31:15 smithi122 ceph-mon[93869]: pgmap v20768: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 421 KiB/s wr, 10 op/s 2023-03-10T01:31:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:31:15 smithi181 ceph-mon[94189]: pgmap v20768: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 421 KiB/s wr, 10 op/s 2023-03-10T01:31:15.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:31:15 smithi038 ceph-mon[92422]: pgmap v20768: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 421 KiB/s wr, 10 op/s 2023-03-10T01:31:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:31:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37986.528697 secs 2023-03-10T01:31:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:31:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37986.528697 secs 2023-03-10T01:31:16.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:31:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37986.528697 secs 2023-03-10T01:31:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:31:17 smithi122 ceph-mon[93869]: pgmap v20769: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 348 KiB/s wr, 12 op/s 2023-03-10T01:31:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:31:17 smithi181 ceph-mon[94189]: pgmap v20769: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 348 KiB/s wr, 12 op/s 2023-03-10T01:31:17.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:31:17 smithi038 ceph-mon[92422]: pgmap v20769: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 348 KiB/s wr, 12 op/s 2023-03-10T01:31:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38450.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:31:20.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:31:20 smithi181 ceph-mon[94189]: pgmap v20770: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 379 KiB/s wr, 11 op/s 2023-03-10T01:31:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:31:20 smithi038 ceph-mon[92422]: pgmap v20770: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 379 KiB/s wr, 11 op/s 2023-03-10T01:31:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:31:20 smithi122 ceph-mon[93869]: pgmap v20770: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 379 KiB/s wr, 11 op/s 2023-03-10T01:31:21.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:31:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37991.528757 secs 2023-03-10T01:31:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:31:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37991.528757 secs 2023-03-10T01:31:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:31:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37991.528757 secs 2023-03-10T01:31:22.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:31:22 smithi038 ceph-mon[92422]: pgmap v20771: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 231 KiB/s wr, 11 op/s 2023-03-10T01:31:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:31:22 smithi122 ceph-mon[93869]: pgmap v20771: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 231 KiB/s wr, 11 op/s 2023-03-10T01:31:22.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:31:22 smithi181 ceph-mon[94189]: pgmap v20771: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 231 KiB/s wr, 11 op/s 2023-03-10T01:31:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38455.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:31:24.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:31:24 smithi038 ceph-mon[92422]: pgmap v20772: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 555 KiB/s rd, 574 KiB/s wr, 12 op/s 2023-03-10T01:31:24.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:31:24 smithi122 ceph-mon[93869]: pgmap v20772: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 555 KiB/s rd, 574 KiB/s wr, 12 op/s 2023-03-10T01:31:24.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:31:24 smithi181 ceph-mon[94189]: pgmap v20772: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 555 KiB/s rd, 574 KiB/s wr, 12 op/s 2023-03-10T01:31:26.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:31:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 37996.528840 secs 2023-03-10T01:31:26.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:31:26 smithi038 ceph-mon[92422]: pgmap v20773: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 486 KiB/s rd, 506 KiB/s wr, 11 op/s 2023-03-10T01:31:26.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:31:26 smithi038 ceph-mon[92422]: osdmap e1513: 12 total, 12 up, 12 in 2023-03-10T01:31:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:31:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 37996.528840 secs 2023-03-10T01:31:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:31:26 smithi122 ceph-mon[93869]: pgmap v20773: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 486 KiB/s rd, 506 KiB/s wr, 11 op/s 2023-03-10T01:31:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:31:26 smithi122 ceph-mon[93869]: osdmap e1513: 12 total, 12 up, 12 in 2023-03-10T01:31:26.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:31:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 37996.528840 secs 2023-03-10T01:31:26.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:31:26 smithi181 ceph-mon[94189]: pgmap v20773: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 486 KiB/s rd, 506 KiB/s wr, 11 op/s 2023-03-10T01:31:26.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:31:26 smithi181 ceph-mon[94189]: osdmap e1513: 12 total, 12 up, 12 in 2023-03-10T01:31:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:31:27 smithi122 ceph-mon[93869]: pgmap v20775: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 453 KiB/s rd, 477 KiB/s wr, 13 op/s 2023-03-10T01:31:27.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:31:27 smithi181 ceph-mon[94189]: pgmap v20775: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 453 KiB/s rd, 477 KiB/s wr, 13 op/s 2023-03-10T01:31:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:31:27 smithi038 ceph-mon[92422]: pgmap v20775: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 453 KiB/s rd, 477 KiB/s wr, 13 op/s 2023-03-10T01:31:28.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38460.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:31:30.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:31:30 smithi038 ceph-mon[92422]: pgmap v20776: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 644 KiB/s wr, 11 op/s 2023-03-10T01:31:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:31:30 smithi122 ceph-mon[93869]: pgmap v20776: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 644 KiB/s wr, 11 op/s 2023-03-10T01:31:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:31:30 smithi181 ceph-mon[94189]: pgmap v20776: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 644 KiB/s wr, 11 op/s 2023-03-10T01:31:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:31:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38001.528897 secs 2023-03-10T01:31:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:31:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38001.528897 secs 2023-03-10T01:31:31.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:31:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38001.528897 secs 2023-03-10T01:31:32.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:31:32 smithi038 ceph-mon[92422]: pgmap v20777: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 630 KiB/s wr, 10 op/s 2023-03-10T01:31:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:31:32 smithi122 ceph-mon[93869]: pgmap v20777: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 630 KiB/s wr, 10 op/s 2023-03-10T01:31:32.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:31:32 smithi181 ceph-mon[94189]: pgmap v20777: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 630 KiB/s wr, 10 op/s 2023-03-10T01:31:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38465.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:31:34.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:31:34 smithi038 ceph-mon[92422]: pgmap v20778: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 405 KiB/s wr, 9 op/s 2023-03-10T01:31:34.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:31:34 smithi122 ceph-mon[93869]: pgmap v20778: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 405 KiB/s wr, 9 op/s 2023-03-10T01:31:34.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:31:34 smithi181 ceph-mon[94189]: pgmap v20778: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 405 KiB/s wr, 9 op/s 2023-03-10T01:31:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:31:35 smithi122 ceph-mon[93869]: osdmap e1514: 12 total, 12 up, 12 in 2023-03-10T01:31:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:31:35 smithi122 ceph-mon[93869]: pgmap v20780: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 408 KiB/s wr, 9 op/s 2023-03-10T01:31:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:31:35 smithi181 ceph-mon[94189]: osdmap e1514: 12 total, 12 up, 12 in 2023-03-10T01:31:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:31:35 smithi181 ceph-mon[94189]: pgmap v20780: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 408 KiB/s wr, 9 op/s 2023-03-10T01:31:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:31:35 smithi038 ceph-mon[92422]: osdmap e1514: 12 total, 12 up, 12 in 2023-03-10T01:31:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:31:35 smithi038 ceph-mon[92422]: pgmap v20780: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 408 KiB/s wr, 9 op/s 2023-03-10T01:31:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:31:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38006.528986 secs 2023-03-10T01:31:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:31:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38006.528986 secs 2023-03-10T01:31:37.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:31:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38006.528986 secs 2023-03-10T01:31:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:31:37 smithi122 ceph-mon[93869]: pgmap v20781: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 461 KiB/s wr, 9 op/s 2023-03-10T01:31:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:31:37 smithi181 ceph-mon[94189]: pgmap v20781: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 461 KiB/s wr, 9 op/s 2023-03-10T01:31:37.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:31:37 smithi038 ceph-mon[92422]: pgmap v20781: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 461 KiB/s wr, 9 op/s 2023-03-10T01:31:38.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38470.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:31:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:31:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38008.990215 secs 2023-03-10T01:31:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:31:38 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:31:38.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:31:38 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:31:38.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:31:38 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:31:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:31:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38008.990215 secs 2023-03-10T01:31:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:31:38 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:31:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:31:38 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:31:38.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:31:38 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:31:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:31:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38008.990215 secs 2023-03-10T01:31:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:31:38 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:31:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:31:38 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:31:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:31:38 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:31:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:31:39 smithi122 ceph-mon[93869]: pgmap v20782: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 243 KiB/s wr, 10 op/s 2023-03-10T01:31:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:31:39 smithi181 ceph-mon[94189]: pgmap v20782: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 243 KiB/s wr, 10 op/s 2023-03-10T01:31:39.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:31:39 smithi038 ceph-mon[92422]: pgmap v20782: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 243 KiB/s wr, 10 op/s 2023-03-10T01:31:41.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:31:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38011.529095 secs 2023-03-10T01:31:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:31:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38011.529095 secs 2023-03-10T01:31:41.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:31:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38011.529095 secs 2023-03-10T01:31:42.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:31:41 smithi038 ceph-mon[92422]: pgmap v20783: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 244 KiB/s wr, 10 op/s 2023-03-10T01:31:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:31:41 smithi122 ceph-mon[93869]: pgmap v20783: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 244 KiB/s wr, 10 op/s 2023-03-10T01:31:42.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:31:41 smithi181 ceph-mon[94189]: pgmap v20783: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 245 KiB/s rd, 244 KiB/s wr, 10 op/s 2023-03-10T01:31:43.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:31:42 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38013.990303 secs 2023-03-10T01:31:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:31:42 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38013.990303 secs 2023-03-10T01:31:43.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:31:42 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38013.990303 secs 2023-03-10T01:31:43.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38475.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:31:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:31:44 smithi122 ceph-mon[93869]: pgmap v20784: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 311 KiB/s wr, 9 op/s 2023-03-10T01:31:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:31:44 smithi181 ceph-mon[94189]: pgmap v20784: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 311 KiB/s wr, 9 op/s 2023-03-10T01:31:44.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:31:44 smithi038 ceph-mon[92422]: pgmap v20784: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 460 KiB/s rd, 311 KiB/s wr, 9 op/s 2023-03-10T01:31:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:31:45 smithi122 ceph-mon[93869]: pgmap v20785: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 296 KiB/s wr, 9 op/s 2023-03-10T01:31:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:31:45 smithi181 ceph-mon[94189]: pgmap v20785: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 296 KiB/s wr, 9 op/s 2023-03-10T01:31:46.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:31:45 smithi038 ceph-mon[92422]: pgmap v20785: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 296 KiB/s wr, 9 op/s 2023-03-10T01:31:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:31:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38016.529155 secs 2023-03-10T01:31:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:31:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38016.529155 secs 2023-03-10T01:31:46.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:31:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38016.529155 secs 2023-03-10T01:31:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:31:47 smithi122 ceph-mon[93869]: pgmap v20786: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 595 KiB/s rd, 470 KiB/s wr, 11 op/s 2023-03-10T01:31:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:31:47 smithi181 ceph-mon[94189]: pgmap v20786: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 595 KiB/s rd, 470 KiB/s wr, 11 op/s 2023-03-10T01:31:47.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:31:47 smithi038 ceph-mon[92422]: pgmap v20786: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 595 KiB/s rd, 470 KiB/s wr, 11 op/s 2023-03-10T01:31:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38480.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:31:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:31:50 smithi122 ceph-mon[93869]: pgmap v20787: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 547 KiB/s rd, 423 KiB/s wr, 9 op/s 2023-03-10T01:31:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:31:50 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:31:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:31:50 smithi181 ceph-mon[94189]: pgmap v20787: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 547 KiB/s rd, 423 KiB/s wr, 9 op/s 2023-03-10T01:31:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:31:50 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:31:50.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:31:50 smithi038 ceph-mon[92422]: pgmap v20787: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 547 KiB/s rd, 423 KiB/s wr, 9 op/s 2023-03-10T01:31:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:31:50 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:31:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:31:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38021.529211 secs 2023-03-10T01:31:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:31:51 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:31:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:31:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:31:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:31:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:31:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:31:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38021.529211 secs 2023-03-10T01:31:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:31:51 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:31:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:31:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:31:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:31:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:31:51.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:31:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38021.529211 secs 2023-03-10T01:31:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:31:51 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:31:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:31:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:31:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:31:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:31:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:31:52 smithi122 ceph-mon[93869]: pgmap v20788: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 640 KiB/s rd, 637 KiB/s wr, 10 op/s 2023-03-10T01:31:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:31:52 smithi181 ceph-mon[94189]: pgmap v20788: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 640 KiB/s rd, 637 KiB/s wr, 10 op/s 2023-03-10T01:31:52.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:31:52 smithi038 ceph-mon[92422]: pgmap v20788: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 640 KiB/s rd, 637 KiB/s wr, 10 op/s 2023-03-10T01:31:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:31:53 smithi122 ceph-mon[93869]: pgmap v20789: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 781 KiB/s rd, 779 KiB/s wr, 11 op/s 2023-03-10T01:31:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:31:53 smithi181 ceph-mon[94189]: pgmap v20789: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 781 KiB/s rd, 779 KiB/s wr, 11 op/s 2023-03-10T01:31:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:31:53 smithi038 ceph-mon[92422]: pgmap v20789: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 781 KiB/s rd, 779 KiB/s wr, 11 op/s 2023-03-10T01:31:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38485.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:31:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:31:55 smithi122 ceph-mon[93869]: pgmap v20790: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 570 KiB/s rd, 568 KiB/s wr, 9 op/s 2023-03-10T01:31:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:31:55 smithi181 ceph-mon[94189]: pgmap v20790: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 570 KiB/s rd, 568 KiB/s wr, 9 op/s 2023-03-10T01:31:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:31:55 smithi038 ceph-mon[92422]: pgmap v20790: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 570 KiB/s rd, 568 KiB/s wr, 9 op/s 2023-03-10T01:31:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:31:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38026.529270 secs 2023-03-10T01:31:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:31:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38026.529270 secs 2023-03-10T01:31:56.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:31:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38026.529270 secs 2023-03-10T01:31:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:31:57 smithi122 ceph-mon[93869]: pgmap v20791: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 946 KiB/s rd, 799 KiB/s wr, 13 op/s 2023-03-10T01:31:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:31:57 smithi181 ceph-mon[94189]: pgmap v20791: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 946 KiB/s rd, 799 KiB/s wr, 13 op/s 2023-03-10T01:31:57.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:31:57 smithi038 ceph-mon[92422]: pgmap v20791: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 946 KiB/s rd, 799 KiB/s wr, 13 op/s 2023-03-10T01:31:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38490.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:31:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:31:58 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 38028.990590 secs 2023-03-10T01:31:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:31:58 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:31:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:31:58 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:31:58.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:31:58 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:31:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:31:58 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 38028.990590 secs 2023-03-10T01:31:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:31:58 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:31:58.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:31:58 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:31:58.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:31:58 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:31:58.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:31:58 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 38028.990590 secs 2023-03-10T01:31:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:31:58 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:31:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:31:58 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:31:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:31:58 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:31:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:31:59 smithi122 ceph-mon[93869]: pgmap v20792: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 777 KiB/s rd, 620 KiB/s wr, 11 op/s 2023-03-10T01:31:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:31:59 smithi181 ceph-mon[94189]: pgmap v20792: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 777 KiB/s rd, 620 KiB/s wr, 11 op/s 2023-03-10T01:31:59.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:31:59 smithi038 ceph-mon[92422]: pgmap v20792: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 777 KiB/s rd, 620 KiB/s wr, 11 op/s 2023-03-10T01:32:01.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:32:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38031.529326 secs 2023-03-10T01:32:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:32:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38031.529326 secs 2023-03-10T01:32:01.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:32:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38031.529326 secs 2023-03-10T01:32:02.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:32:01 smithi038 ceph-mon[92422]: pgmap v20793: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 16 op/s 2023-03-10T01:32:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:32:01 smithi122 ceph-mon[93869]: pgmap v20793: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 16 op/s 2023-03-10T01:32:02.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:32:01 smithi181 ceph-mon[94189]: pgmap v20793: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 16 op/s 2023-03-10T01:32:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:32:03 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 38033.990648 secs 2023-03-10T01:32:03.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:32:03 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 38033.990648 secs 2023-03-10T01:32:03.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:32:03 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 38033.990648 secs 2023-03-10T01:32:03.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38495.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:32:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:32:04 smithi122 ceph-mon[93869]: pgmap v20794: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 980 KiB/s wr, 26 op/s 2023-03-10T01:32:04.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:32:04 smithi181 ceph-mon[94189]: pgmap v20794: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 980 KiB/s wr, 26 op/s 2023-03-10T01:32:04.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:32:04 smithi038 ceph-mon[92422]: pgmap v20794: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 980 KiB/s wr, 26 op/s 2023-03-10T01:32:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:32:05 smithi122 ceph-mon[93869]: pgmap v20795: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 837 KiB/s wr, 24 op/s 2023-03-10T01:32:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:32:05 smithi181 ceph-mon[94189]: pgmap v20795: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 837 KiB/s wr, 24 op/s 2023-03-10T01:32:05.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:32:05 smithi038 ceph-mon[92422]: pgmap v20795: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 837 KiB/s wr, 24 op/s 2023-03-10T01:32:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:32:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38036.529386 secs 2023-03-10T01:32:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:32:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38036.529386 secs 2023-03-10T01:32:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:32:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38036.529386 secs 2023-03-10T01:32:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:32:07 smithi122 ceph-mon[93869]: pgmap v20796: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 843 KiB/s wr, 28 op/s 2023-03-10T01:32:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:32:07 smithi181 ceph-mon[94189]: pgmap v20796: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 843 KiB/s wr, 28 op/s 2023-03-10T01:32:07.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:32:07 smithi038 ceph-mon[92422]: pgmap v20796: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 843 KiB/s wr, 28 op/s 2023-03-10T01:32:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38500.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:32:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:32:08 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 38038.990731 secs 2023-03-10T01:32:08.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:32:08 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 38038.990731 secs 2023-03-10T01:32:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:32:08 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 38038.990731 secs 2023-03-10T01:32:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:32:09 smithi122 ceph-mon[93869]: pgmap v20797: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 894 KiB/s rd, 612 KiB/s wr, 25 op/s 2023-03-10T01:32:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:32:09 smithi181 ceph-mon[94189]: pgmap v20797: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 894 KiB/s rd, 612 KiB/s wr, 25 op/s 2023-03-10T01:32:09.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:32:09 smithi038 ceph-mon[92422]: pgmap v20797: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 894 KiB/s rd, 612 KiB/s wr, 25 op/s 2023-03-10T01:32:11.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:32:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38041.529443 secs 2023-03-10T01:32:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:32:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38041.529443 secs 2023-03-10T01:32:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:32:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38041.529443 secs 2023-03-10T01:32:12.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:32:11 smithi038 ceph-mon[92422]: pgmap v20798: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1009 KiB/s rd, 738 KiB/s wr, 26 op/s 2023-03-10T01:32:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:32:11 smithi122 ceph-mon[93869]: pgmap v20798: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1009 KiB/s rd, 738 KiB/s wr, 26 op/s 2023-03-10T01:32:12.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:32:11 smithi181 ceph-mon[94189]: pgmap v20798: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1009 KiB/s rd, 738 KiB/s wr, 26 op/s 2023-03-10T01:32:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:32:13 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 38043.990804 secs 2023-03-10T01:32:13.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:32:13 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 38043.990804 secs 2023-03-10T01:32:13.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:32:13 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 38043.990804 secs 2023-03-10T01:32:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38505.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:32:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:32:14 smithi122 ceph-mon[93869]: pgmap v20799: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 663 KiB/s rd, 381 KiB/s wr, 22 op/s 2023-03-10T01:32:14.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:32:14 smithi181 ceph-mon[94189]: pgmap v20799: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 663 KiB/s rd, 381 KiB/s wr, 22 op/s 2023-03-10T01:32:14.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:32:14 smithi038 ceph-mon[92422]: pgmap v20799: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 663 KiB/s rd, 381 KiB/s wr, 22 op/s 2023-03-10T01:32:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:32:15 smithi122 ceph-mon[93869]: pgmap v20800: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 338 KiB/s wr, 9 op/s 2023-03-10T01:32:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:32:15 smithi181 ceph-mon[94189]: pgmap v20800: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 338 KiB/s wr, 9 op/s 2023-03-10T01:32:15.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:32:15 smithi038 ceph-mon[92422]: pgmap v20800: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 338 KiB/s wr, 9 op/s 2023-03-10T01:32:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:32:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38046.529498 secs 2023-03-10T01:32:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:32:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38046.529498 secs 2023-03-10T01:32:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:32:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38046.529498 secs 2023-03-10T01:32:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:32:17 smithi122 ceph-mon[93869]: pgmap v20801: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 338 KiB/s wr, 11 op/s 2023-03-10T01:32:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:32:17 smithi181 ceph-mon[94189]: pgmap v20801: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 338 KiB/s wr, 11 op/s 2023-03-10T01:32:17.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:32:17 smithi038 ceph-mon[92422]: pgmap v20801: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 338 KiB/s wr, 11 op/s 2023-03-10T01:32:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38510.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:32:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:32:18 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 38048.990862 secs 2023-03-10T01:32:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:32:18 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:32:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:32:18 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 38048.990862 secs 2023-03-10T01:32:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:32:18 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:32:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:32:18 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 38048.990862 secs 2023-03-10T01:32:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:32:18 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:32:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:32:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:32:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:32:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:32:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:32:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:32:20.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:32:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:32:20.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:32:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:32:20.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:32:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:32:20.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:32:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:32:20.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:32:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:32:20.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:32:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:32:20.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:32:19 smithi122 ceph-mon[93869]: pgmap v20802: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 736 KiB/s wr, 11 op/s 2023-03-10T01:32:20.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:32:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:32:20.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:32:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:32:20.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:32:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:32:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:32:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:32:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:32:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:32:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:32:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:32:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:32:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:32:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:32:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:32:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:32:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:32:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:32:19 smithi181 ceph-mon[94189]: pgmap v20802: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 736 KiB/s wr, 11 op/s 2023-03-10T01:32:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:32:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:32:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:32:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:32:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:32:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:32:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:32:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:32:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:32:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:32:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:32:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:32:20.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:32:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:32:20.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:32:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:32:20.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:32:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:32:20.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:32:19 smithi038 ceph-mon[92422]: pgmap v20802: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 736 KiB/s wr, 11 op/s 2023-03-10T01:32:21.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:32:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38051.529581 secs 2023-03-10T01:32:21.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:32:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38051.529581 secs 2023-03-10T01:32:21.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:32:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38051.529581 secs 2023-03-10T01:32:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:32:21 smithi122 ceph-mon[93869]: pgmap v20803: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 748 KiB/s wr, 13 op/s 2023-03-10T01:32:22.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:32:21 smithi181 ceph-mon[94189]: pgmap v20803: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 748 KiB/s wr, 13 op/s 2023-03-10T01:32:22.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:32:21 smithi038 ceph-mon[92422]: pgmap v20803: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 748 KiB/s wr, 13 op/s 2023-03-10T01:32:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:32:23 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 38053.990927 secs 2023-03-10T01:32:23.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:32:23 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 38053.990927 secs 2023-03-10T01:32:23.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:32:23 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 38053.990927 secs 2023-03-10T01:32:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38515.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:32:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:32:24 smithi122 ceph-mon[93869]: pgmap v20804: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 704 KiB/s rd, 724 KiB/s wr, 13 op/s 2023-03-10T01:32:24.496 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:32:24 smithi181 ceph-mon[94189]: pgmap v20804: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 704 KiB/s rd, 724 KiB/s wr, 13 op/s 2023-03-10T01:32:24.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:32:24 smithi038 ceph-mon[92422]: pgmap v20804: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 704 KiB/s rd, 724 KiB/s wr, 13 op/s 2023-03-10T01:32:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:32:26 smithi122 ceph-mon[93869]: osdmap e1515: 12 total, 12 up, 12 in 2023-03-10T01:32:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:32:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38056.529644 secs 2023-03-10T01:32:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:32:26 smithi122 ceph-mon[93869]: pgmap v20806: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 659 KiB/s wr, 13 op/s 2023-03-10T01:32:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:32:26 smithi181 ceph-mon[94189]: osdmap e1515: 12 total, 12 up, 12 in 2023-03-10T01:32:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:32:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38056.529644 secs 2023-03-10T01:32:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:32:26 smithi181 ceph-mon[94189]: pgmap v20806: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 659 KiB/s wr, 13 op/s 2023-03-10T01:32:26.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:32:26 smithi038 ceph-mon[92422]: osdmap e1515: 12 total, 12 up, 12 in 2023-03-10T01:32:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:32:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38056.529644 secs 2023-03-10T01:32:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:32:26 smithi038 ceph-mon[92422]: pgmap v20806: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 659 KiB/s wr, 13 op/s 2023-03-10T01:32:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:32:28 smithi122 ceph-mon[93869]: pgmap v20807: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 803 KiB/s rd, 828 KiB/s wr, 13 op/s 2023-03-10T01:32:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:32:28 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 38058.991038 secs 2023-03-10T01:32:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:32:28 smithi181 ceph-mon[94189]: pgmap v20807: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 803 KiB/s rd, 828 KiB/s wr, 13 op/s 2023-03-10T01:32:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:32:28 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 38058.991038 secs 2023-03-10T01:32:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:32:28 smithi038 ceph-mon[92422]: pgmap v20807: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 803 KiB/s rd, 828 KiB/s wr, 13 op/s 2023-03-10T01:32:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:32:28 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 38058.991038 secs 2023-03-10T01:32:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38520.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:32:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:32:30 smithi122 ceph-mon[93869]: pgmap v20808: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 819 KiB/s rd, 399 KiB/s wr, 11 op/s 2023-03-10T01:32:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:32:30 smithi122 ceph-mon[93869]: osdmap e1516: 12 total, 12 up, 12 in 2023-03-10T01:32:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:32:30 smithi181 ceph-mon[94189]: pgmap v20808: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 819 KiB/s rd, 399 KiB/s wr, 11 op/s 2023-03-10T01:32:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:32:30 smithi181 ceph-mon[94189]: osdmap e1516: 12 total, 12 up, 12 in 2023-03-10T01:32:30.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:32:30 smithi038 ceph-mon[92422]: pgmap v20808: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 819 KiB/s rd, 399 KiB/s wr, 11 op/s 2023-03-10T01:32:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:32:30 smithi038 ceph-mon[92422]: osdmap e1516: 12 total, 12 up, 12 in 2023-03-10T01:32:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:32:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38061.529714 secs 2023-03-10T01:32:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:32:31 smithi122 ceph-mon[93869]: pgmap v20810: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 431 KiB/s wr, 9 op/s 2023-03-10T01:32:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:32:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38061.529714 secs 2023-03-10T01:32:31.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:32:31 smithi181 ceph-mon[94189]: pgmap v20810: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 431 KiB/s wr, 9 op/s 2023-03-10T01:32:31.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:32:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38061.529714 secs 2023-03-10T01:32:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:32:31 smithi038 ceph-mon[92422]: pgmap v20810: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 433 KiB/s rd, 431 KiB/s wr, 9 op/s 2023-03-10T01:32:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:32:33 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 38063.991124 secs 2023-03-10T01:32:33.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:32:33 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 38063.991124 secs 2023-03-10T01:32:33.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:32:33 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 38063.991124 secs 2023-03-10T01:32:33.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38525.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:32:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:32:34 smithi122 ceph-mon[93869]: pgmap v20811: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 471 KiB/s wr, 11 op/s 2023-03-10T01:32:34.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:32:34 smithi181 ceph-mon[94189]: pgmap v20811: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 471 KiB/s wr, 11 op/s 2023-03-10T01:32:34.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:32:34 smithi038 ceph-mon[92422]: pgmap v20811: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 471 KiB/s wr, 11 op/s 2023-03-10T01:32:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:32:35 smithi122 ceph-mon[93869]: pgmap v20812: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 384 KiB/s rd, 383 KiB/s wr, 8 op/s 2023-03-10T01:32:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:32:35 smithi181 ceph-mon[94189]: pgmap v20812: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 384 KiB/s rd, 383 KiB/s wr, 8 op/s 2023-03-10T01:32:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:32:35 smithi038 ceph-mon[92422]: pgmap v20812: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 384 KiB/s rd, 383 KiB/s wr, 8 op/s 2023-03-10T01:32:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:32:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38066.529771 secs 2023-03-10T01:32:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:32:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38066.529771 secs 2023-03-10T01:32:36.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:32:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38066.529771 secs 2023-03-10T01:32:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:32:37 smithi122 ceph-mon[93869]: pgmap v20813: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 872 KiB/s rd, 871 KiB/s wr, 10 op/s 2023-03-10T01:32:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:32:37 smithi181 ceph-mon[94189]: pgmap v20813: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 872 KiB/s rd, 871 KiB/s wr, 10 op/s 2023-03-10T01:32:37.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:32:37 smithi038 ceph-mon[92422]: pgmap v20813: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 872 KiB/s rd, 871 KiB/s wr, 10 op/s 2023-03-10T01:32:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38530.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:32:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:32:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38068.991202 secs 2023-03-10T01:32:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:32:38 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:32:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:32:38 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:32:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:32:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38068.991202 secs 2023-03-10T01:32:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:32:38 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:32:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:32:38 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:32:39.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:32:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38068.991202 secs 2023-03-10T01:32:39.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:32:38 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:32:39.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:32:38 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:32:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:32:39 smithi122 ceph-mon[93869]: pgmap v20814: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 818 KiB/s rd, 816 KiB/s wr, 10 op/s 2023-03-10T01:32:40.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:32:39 smithi181 ceph-mon[94189]: pgmap v20814: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 818 KiB/s rd, 816 KiB/s wr, 10 op/s 2023-03-10T01:32:40.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:32:39 smithi038 ceph-mon[92422]: pgmap v20814: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 818 KiB/s rd, 816 KiB/s wr, 10 op/s 2023-03-10T01:32:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:32:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38071.529826 secs 2023-03-10T01:32:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:32:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38071.529826 secs 2023-03-10T01:32:41.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:32:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38071.529826 secs 2023-03-10T01:32:42.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:32:41 smithi038 ceph-mon[92422]: pgmap v20815: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 710 KiB/s wr, 11 op/s 2023-03-10T01:32:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:32:41 smithi122 ceph-mon[93869]: pgmap v20815: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 710 KiB/s wr, 11 op/s 2023-03-10T01:32:42.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:32:41 smithi181 ceph-mon[94189]: pgmap v20815: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 710 KiB/s wr, 11 op/s 2023-03-10T01:32:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:32:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38073.991261 secs 2023-03-10T01:32:43.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:32:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38073.991261 secs 2023-03-10T01:32:43.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:32:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38073.991261 secs 2023-03-10T01:32:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38535.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:32:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:32:44 smithi122 ceph-mon[93869]: pgmap v20816: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 736 KiB/s rd, 734 KiB/s wr, 10 op/s 2023-03-10T01:32:44.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:32:44 smithi181 ceph-mon[94189]: pgmap v20816: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 736 KiB/s rd, 734 KiB/s wr, 10 op/s 2023-03-10T01:32:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:32:44 smithi038 ceph-mon[92422]: pgmap v20816: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 736 KiB/s rd, 734 KiB/s wr, 10 op/s 2023-03-10T01:32:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:32:45 smithi122 ceph-mon[93869]: pgmap v20817: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 822 KiB/s rd, 820 KiB/s wr, 9 op/s 2023-03-10T01:32:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:32:45 smithi181 ceph-mon[94189]: pgmap v20817: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 822 KiB/s rd, 820 KiB/s wr, 9 op/s 2023-03-10T01:32:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:32:45 smithi038 ceph-mon[92422]: pgmap v20817: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 822 KiB/s rd, 820 KiB/s wr, 9 op/s 2023-03-10T01:32:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:32:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38076.529916 secs 2023-03-10T01:32:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:32:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38076.529916 secs 2023-03-10T01:32:46.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:32:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38076.529916 secs 2023-03-10T01:32:47.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:32:47 smithi038 ceph-mon[92422]: pgmap v20818: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 906 KiB/s rd, 903 KiB/s wr, 12 op/s 2023-03-10T01:32:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:32:47 smithi122 ceph-mon[93869]: pgmap v20818: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 906 KiB/s rd, 903 KiB/s wr, 12 op/s 2023-03-10T01:32:48.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:32:47 smithi181 ceph-mon[94189]: pgmap v20818: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 906 KiB/s rd, 903 KiB/s wr, 12 op/s 2023-03-10T01:32:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38540.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:32:48.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:32:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38078.991364 secs 2023-03-10T01:32:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:32:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38078.991364 secs 2023-03-10T01:32:49.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:32:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38078.991364 secs 2023-03-10T01:32:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:32:49 smithi122 ceph-mon[93869]: pgmap v20819: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 367 KiB/s wr, 9 op/s 2023-03-10T01:32:50.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:32:49 smithi181 ceph-mon[94189]: pgmap v20819: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 367 KiB/s wr, 9 op/s 2023-03-10T01:32:50.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:32:49 smithi038 ceph-mon[92422]: pgmap v20819: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 367 KiB/s wr, 9 op/s 2023-03-10T01:32:51.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:32:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38081.529978 secs 2023-03-10T01:32:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:32:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38081.529978 secs 2023-03-10T01:32:51.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:32:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38081.529978 secs 2023-03-10T01:32:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:32:51 smithi122 ceph-mon[93869]: pgmap v20820: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 484 KiB/s rd, 481 KiB/s wr, 10 op/s 2023-03-10T01:32:52.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:32:51 smithi181 ceph-mon[94189]: pgmap v20820: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 484 KiB/s rd, 481 KiB/s wr, 10 op/s 2023-03-10T01:32:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:32:51 smithi038 ceph-mon[92422]: pgmap v20820: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 484 KiB/s rd, 481 KiB/s wr, 10 op/s 2023-03-10T01:32:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:32:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38083.991467 secs 2023-03-10T01:32:53.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:32:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38083.991467 secs 2023-03-10T01:32:53.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:32:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38083.991467 secs 2023-03-10T01:32:53.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38545.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:32:54.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:32:54 smithi122 ceph-mon[93869]: pgmap v20821: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 515 KiB/s rd, 513 KiB/s wr, 9 op/s 2023-03-10T01:32:54.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:32:54 smithi181 ceph-mon[94189]: pgmap v20821: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 515 KiB/s rd, 513 KiB/s wr, 9 op/s 2023-03-10T01:32:54.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:32:54 smithi038 ceph-mon[92422]: pgmap v20821: 129 pgs: 129 active+clean; 9.7 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 515 KiB/s rd, 513 KiB/s wr, 9 op/s 2023-03-10T01:32:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:32:55 smithi122 ceph-mon[93869]: pgmap v20822: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 570 KiB/s wr, 8 op/s 2023-03-10T01:32:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:32:55 smithi181 ceph-mon[94189]: pgmap v20822: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 570 KiB/s wr, 8 op/s 2023-03-10T01:32:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:32:55 smithi038 ceph-mon[92422]: pgmap v20822: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 570 KiB/s wr, 8 op/s 2023-03-10T01:32:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:32:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38086.530067 secs 2023-03-10T01:32:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:32:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38086.530067 secs 2023-03-10T01:32:56.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:32:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38086.530067 secs 2023-03-10T01:32:57.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:32:57 smithi038 ceph-mon[92422]: pgmap v20823: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 702 KiB/s rd, 697 KiB/s wr, 11 op/s 2023-03-10T01:32:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:32:57 smithi122 ceph-mon[93869]: pgmap v20823: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 702 KiB/s rd, 697 KiB/s wr, 11 op/s 2023-03-10T01:32:58.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:32:57 smithi181 ceph-mon[94189]: pgmap v20823: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 702 KiB/s rd, 697 KiB/s wr, 11 op/s 2023-03-10T01:32:58.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38550.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:32:58.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:32:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38088.991524 secs 2023-03-10T01:32:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:32:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38088.991524 secs 2023-03-10T01:32:59.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:32:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38088.991524 secs 2023-03-10T01:33:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:32:59 smithi122 ceph-mon[93869]: pgmap v20824: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 661 KiB/s rd, 646 KiB/s wr, 10 op/s 2023-03-10T01:33:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:32:59 smithi181 ceph-mon[94189]: pgmap v20824: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 661 KiB/s rd, 646 KiB/s wr, 10 op/s 2023-03-10T01:33:00.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:32:59 smithi038 ceph-mon[92422]: pgmap v20824: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 661 KiB/s rd, 646 KiB/s wr, 10 op/s 2023-03-10T01:33:01.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:33:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38091.530122 secs 2023-03-10T01:33:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:33:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38091.530122 secs 2023-03-10T01:33:01.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:33:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38091.530122 secs 2023-03-10T01:33:02.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:33:01 smithi038 ceph-mon[92422]: pgmap v20825: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 976 KiB/s rd, 971 KiB/s wr, 14 op/s 2023-03-10T01:33:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:33:01 smithi122 ceph-mon[93869]: pgmap v20825: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 976 KiB/s rd, 971 KiB/s wr, 14 op/s 2023-03-10T01:33:02.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:33:01 smithi181 ceph-mon[94189]: pgmap v20825: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 976 KiB/s rd, 971 KiB/s wr, 14 op/s 2023-03-10T01:33:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:33:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38093.991603 secs 2023-03-10T01:33:03.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:33:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38093.991603 secs 2023-03-10T01:33:03.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:33:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38093.991603 secs 2023-03-10T01:33:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38555.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:33:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:33:04 smithi122 ceph-mon[93869]: pgmap v20826: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 25 op/s 2023-03-10T01:33:04.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:33:04 smithi181 ceph-mon[94189]: pgmap v20826: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 25 op/s 2023-03-10T01:33:04.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:33:04 smithi038 ceph-mon[92422]: pgmap v20826: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 25 op/s 2023-03-10T01:33:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:33:05 smithi122 ceph-mon[93869]: pgmap v20827: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 23 op/s 2023-03-10T01:33:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:33:05 smithi181 ceph-mon[94189]: pgmap v20827: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 23 op/s 2023-03-10T01:33:05.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:33:05 smithi038 ceph-mon[92422]: pgmap v20827: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 23 op/s 2023-03-10T01:33:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:33:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38096.530179 secs 2023-03-10T01:33:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:33:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38096.530179 secs 2023-03-10T01:33:06.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:33:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38096.530179 secs 2023-03-10T01:33:07.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:33:07 smithi038 ceph-mon[92422]: pgmap v20828: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1001 KiB/s wr, 27 op/s 2023-03-10T01:33:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:33:07 smithi122 ceph-mon[93869]: pgmap v20828: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1001 KiB/s wr, 27 op/s 2023-03-10T01:33:08.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:33:07 smithi181 ceph-mon[94189]: pgmap v20828: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1001 KiB/s wr, 27 op/s 2023-03-10T01:33:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38560.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:33:08.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:33:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38098.991682 secs 2023-03-10T01:33:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:33:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38098.991682 secs 2023-03-10T01:33:09.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:33:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38098.991682 secs 2023-03-10T01:33:09.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:33:09 smithi038 ceph-mon[92422]: pgmap v20829: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 757 KiB/s wr, 24 op/s 2023-03-10T01:33:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:33:09 smithi122 ceph-mon[93869]: pgmap v20829: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 757 KiB/s wr, 24 op/s 2023-03-10T01:33:10.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:33:09 smithi181 ceph-mon[94189]: pgmap v20829: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 757 KiB/s wr, 24 op/s 2023-03-10T01:33:11.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:33:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38101.530235 secs 2023-03-10T01:33:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:33:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38101.530235 secs 2023-03-10T01:33:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:33:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38101.530235 secs 2023-03-10T01:33:12.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:33:11 smithi038 ceph-mon[92422]: pgmap v20830: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 725 KiB/s wr, 25 op/s 2023-03-10T01:33:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:33:11 smithi122 ceph-mon[93869]: pgmap v20830: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 725 KiB/s wr, 25 op/s 2023-03-10T01:33:12.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:33:11 smithi181 ceph-mon[94189]: pgmap v20830: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 725 KiB/s wr, 25 op/s 2023-03-10T01:33:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:33:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38103.991742 secs 2023-03-10T01:33:13.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:33:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38103.991742 secs 2023-03-10T01:33:13.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:33:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38103.991742 secs 2023-03-10T01:33:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38565.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:33:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:33:14 smithi122 ceph-mon[93869]: pgmap v20831: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 567 KiB/s wr, 22 op/s 2023-03-10T01:33:14.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:33:14 smithi181 ceph-mon[94189]: pgmap v20831: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 567 KiB/s wr, 22 op/s 2023-03-10T01:33:14.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:33:14 smithi038 ceph-mon[92422]: pgmap v20831: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 567 KiB/s wr, 22 op/s 2023-03-10T01:33:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:33:15 smithi122 ceph-mon[93869]: pgmap v20832: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 656 KiB/s rd, 182 KiB/s wr, 9 op/s 2023-03-10T01:33:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:33:15 smithi181 ceph-mon[94189]: pgmap v20832: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 656 KiB/s rd, 182 KiB/s wr, 9 op/s 2023-03-10T01:33:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:33:15 smithi038 ceph-mon[92422]: pgmap v20832: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 656 KiB/s rd, 182 KiB/s wr, 9 op/s 2023-03-10T01:33:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:33:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38106.530294 secs 2023-03-10T01:33:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:33:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38106.530294 secs 2023-03-10T01:33:16.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:33:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38106.530294 secs 2023-03-10T01:33:17.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:33:17 smithi038 ceph-mon[92422]: pgmap v20833: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 183 KiB/s wr, 12 op/s 2023-03-10T01:33:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:33:17 smithi122 ceph-mon[93869]: pgmap v20833: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 183 KiB/s wr, 12 op/s 2023-03-10T01:33:18.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:33:17 smithi181 ceph-mon[94189]: pgmap v20833: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 183 KiB/s wr, 12 op/s 2023-03-10T01:33:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38570.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:33:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:33:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38108.991827 secs 2023-03-10T01:33:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:33:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38108.991827 secs 2023-03-10T01:33:19.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:33:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38108.991827 secs 2023-03-10T01:33:19.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:33:19 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:33:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:33:19 smithi038 ceph-mon[92422]: pgmap v20834: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 222 KiB/s wr, 10 op/s 2023-03-10T01:33:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:33:19 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:33:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:33:19 smithi122 ceph-mon[93869]: pgmap v20834: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 222 KiB/s wr, 10 op/s 2023-03-10T01:33:20.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:33:19 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:33:20.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:33:19 smithi181 ceph-mon[94189]: pgmap v20834: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 222 KiB/s wr, 10 op/s 2023-03-10T01:33:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:33:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:33:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:33:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:33:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:33:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:33:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:33:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:33:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:33:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:33:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:33:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:33:21.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:33:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:33:21.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:33:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:33:21.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:33:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:33:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:33:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38111.530352 secs 2023-03-10T01:33:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:33:21 smithi122 ceph-mon[93869]: pgmap v20835: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 233 KiB/s wr, 12 op/s 2023-03-10T01:33:22.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:33:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38111.530352 secs 2023-03-10T01:33:22.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:33:21 smithi181 ceph-mon[94189]: pgmap v20835: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 233 KiB/s wr, 12 op/s 2023-03-10T01:33:22.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:33:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38111.530352 secs 2023-03-10T01:33:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:33:21 smithi038 ceph-mon[92422]: pgmap v20835: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 215 KiB/s rd, 233 KiB/s wr, 12 op/s 2023-03-10T01:33:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:33:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38113.991889 secs 2023-03-10T01:33:23.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:33:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38113.991889 secs 2023-03-10T01:33:23.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:33:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38113.991889 secs 2023-03-10T01:33:23.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38575.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:33:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:33:24 smithi122 ceph-mon[93869]: pgmap v20836: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 555 KiB/s rd, 575 KiB/s wr, 13 op/s 2023-03-10T01:33:24.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:33:24 smithi181 ceph-mon[94189]: pgmap v20836: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 555 KiB/s rd, 575 KiB/s wr, 13 op/s 2023-03-10T01:33:24.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:33:24 smithi038 ceph-mon[92422]: pgmap v20836: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 555 KiB/s rd, 575 KiB/s wr, 13 op/s 2023-03-10T01:33:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:33:26 smithi122 ceph-mon[93869]: osdmap e1517: 12 total, 12 up, 12 in 2023-03-10T01:33:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:33:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38116.533631 secs 2023-03-10T01:33:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:33:26 smithi122 ceph-mon[93869]: pgmap v20838: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 452 KiB/s rd, 476 KiB/s wr, 13 op/s 2023-03-10T01:33:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:33:26 smithi181 ceph-mon[94189]: osdmap e1517: 12 total, 12 up, 12 in 2023-03-10T01:33:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:33:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38116.533631 secs 2023-03-10T01:33:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:33:26 smithi181 ceph-mon[94189]: pgmap v20838: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 452 KiB/s rd, 476 KiB/s wr, 13 op/s 2023-03-10T01:33:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:33:26 smithi038 ceph-mon[92422]: osdmap e1517: 12 total, 12 up, 12 in 2023-03-10T01:33:26.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:33:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38116.533631 secs 2023-03-10T01:33:26.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:33:26 smithi038 ceph-mon[92422]: pgmap v20838: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 452 KiB/s rd, 476 KiB/s wr, 13 op/s 2023-03-10T01:33:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:33:27 smithi122 ceph-mon[93869]: pgmap v20839: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 638 KiB/s rd, 663 KiB/s wr, 13 op/s 2023-03-10T01:33:27.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:33:27 smithi181 ceph-mon[94189]: pgmap v20839: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 638 KiB/s rd, 663 KiB/s wr, 13 op/s 2023-03-10T01:33:27.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:33:27 smithi038 ceph-mon[92422]: pgmap v20839: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 638 KiB/s rd, 663 KiB/s wr, 13 op/s 2023-03-10T01:33:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:33:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38118.991974 secs 2023-03-10T01:33:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:33:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38118.991974 secs 2023-03-10T01:33:28.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:33:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38118.991974 secs 2023-03-10T01:33:28.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38580.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:33:29.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:33:29 smithi122 ceph-mon[93869]: pgmap v20840: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 637 KiB/s rd, 650 KiB/s wr, 11 op/s 2023-03-10T01:33:29.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:33:29 smithi181 ceph-mon[94189]: pgmap v20840: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 637 KiB/s rd, 650 KiB/s wr, 11 op/s 2023-03-10T01:33:29.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:33:29 smithi038 ceph-mon[92422]: pgmap v20840: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 637 KiB/s rd, 650 KiB/s wr, 11 op/s 2023-03-10T01:33:31.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:33:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38121.533713 secs 2023-03-10T01:33:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:33:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38121.533713 secs 2023-03-10T01:33:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:33:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38121.533713 secs 2023-03-10T01:33:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:33:32 smithi122 ceph-mon[93869]: pgmap v20841: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 751 KiB/s rd, 750 KiB/s wr, 10 op/s 2023-03-10T01:33:32.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:33:32 smithi181 ceph-mon[94189]: pgmap v20841: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 751 KiB/s rd, 750 KiB/s wr, 10 op/s 2023-03-10T01:33:32.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:33:32 smithi038 ceph-mon[92422]: pgmap v20841: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 751 KiB/s rd, 750 KiB/s wr, 10 op/s 2023-03-10T01:33:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:33:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38123.992085 secs 2023-03-10T01:33:33.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:33:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38123.992085 secs 2023-03-10T01:33:33.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:33:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38123.992085 secs 2023-03-10T01:33:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38585.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:33:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:33:34 smithi122 ceph-mon[93869]: pgmap v20842: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 537 KiB/s wr, 9 op/s 2023-03-10T01:33:34.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:33:34 smithi181 ceph-mon[94189]: pgmap v20842: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 537 KiB/s wr, 9 op/s 2023-03-10T01:33:34.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:33:34 smithi038 ceph-mon[92422]: pgmap v20842: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 537 KiB/s wr, 9 op/s 2023-03-10T01:33:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:33:35 smithi122 ceph-mon[93869]: osdmap e1518: 12 total, 12 up, 12 in 2023-03-10T01:33:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:33:35 smithi122 ceph-mon[93869]: pgmap v20844: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 537 KiB/s wr, 9 op/s 2023-03-10T01:33:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:33:35 smithi181 ceph-mon[94189]: osdmap e1518: 12 total, 12 up, 12 in 2023-03-10T01:33:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:33:35 smithi181 ceph-mon[94189]: pgmap v20844: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 537 KiB/s wr, 9 op/s 2023-03-10T01:33:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:33:35 smithi038 ceph-mon[92422]: osdmap e1518: 12 total, 12 up, 12 in 2023-03-10T01:33:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:33:35 smithi038 ceph-mon[92422]: pgmap v20844: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 365 KiB/s rd, 537 KiB/s wr, 9 op/s 2023-03-10T01:33:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:33:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38126.533772 secs 2023-03-10T01:33:37.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:33:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38126.533772 secs 2023-03-10T01:33:37.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:33:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38126.533772 secs 2023-03-10T01:33:37.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:33:37 smithi038 ceph-mon[92422]: pgmap v20845: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 667 KiB/s wr, 9 op/s 2023-03-10T01:33:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:33:37 smithi122 ceph-mon[93869]: pgmap v20845: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 667 KiB/s wr, 9 op/s 2023-03-10T01:33:38.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:33:37 smithi181 ceph-mon[94189]: pgmap v20845: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 667 KiB/s wr, 9 op/s 2023-03-10T01:33:38.793 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38590.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:33:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:33:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38128.992115 secs 2023-03-10T01:33:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:33:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38128.992115 secs 2023-03-10T01:33:39.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:33:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38128.992115 secs 2023-03-10T01:33:39.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:33:39 smithi038 ceph-mon[92422]: pgmap v20846: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 821 KiB/s rd, 818 KiB/s wr, 10 op/s 2023-03-10T01:33:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:33:39 smithi122 ceph-mon[93869]: pgmap v20846: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 821 KiB/s rd, 818 KiB/s wr, 10 op/s 2023-03-10T01:33:40.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:33:39 smithi181 ceph-mon[94189]: pgmap v20846: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 821 KiB/s rd, 818 KiB/s wr, 10 op/s 2023-03-10T01:33:41.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:33:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38131.533864 secs 2023-03-10T01:33:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:33:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38131.533864 secs 2023-03-10T01:33:41.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:33:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38131.533864 secs 2023-03-10T01:33:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:33:42 smithi122 ceph-mon[93869]: pgmap v20847: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 707 KiB/s rd, 705 KiB/s wr, 11 op/s 2023-03-10T01:33:42.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:33:42 smithi181 ceph-mon[94189]: pgmap v20847: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 707 KiB/s rd, 705 KiB/s wr, 11 op/s 2023-03-10T01:33:42.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:33:42 smithi038 ceph-mon[92422]: pgmap v20847: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 707 KiB/s rd, 705 KiB/s wr, 11 op/s 2023-03-10T01:33:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:33:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38133.992193 secs 2023-03-10T01:33:43.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:33:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38133.992193 secs 2023-03-10T01:33:43.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:33:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38133.992193 secs 2023-03-10T01:33:43.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38595.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:33:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:33:44 smithi122 ceph-mon[93869]: pgmap v20848: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 869 KiB/s rd, 693 KiB/s wr, 11 op/s 2023-03-10T01:33:44.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:33:44 smithi181 ceph-mon[94189]: pgmap v20848: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 869 KiB/s rd, 693 KiB/s wr, 11 op/s 2023-03-10T01:33:44.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:33:44 smithi038 ceph-mon[92422]: pgmap v20848: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 869 KiB/s rd, 693 KiB/s wr, 11 op/s 2023-03-10T01:33:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:33:45 smithi038 ceph-mon[92422]: pgmap v20849: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 827 KiB/s rd, 660 KiB/s wr, 10 op/s 2023-03-10T01:33:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:33:45 smithi122 ceph-mon[93869]: pgmap v20849: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 827 KiB/s rd, 660 KiB/s wr, 10 op/s 2023-03-10T01:33:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:33:45 smithi181 ceph-mon[94189]: pgmap v20849: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 827 KiB/s rd, 660 KiB/s wr, 10 op/s 2023-03-10T01:33:46.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:33:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38136.533946 secs 2023-03-10T01:33:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:33:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38136.533946 secs 2023-03-10T01:33:47.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:33:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38136.533946 secs 2023-03-10T01:33:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:33:47 smithi122 ceph-mon[93869]: pgmap v20850: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 975 KiB/s rd, 803 KiB/s wr, 12 op/s 2023-03-10T01:33:48.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:33:47 smithi181 ceph-mon[94189]: pgmap v20850: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 975 KiB/s rd, 803 KiB/s wr, 12 op/s 2023-03-10T01:33:48.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:33:47 smithi038 ceph-mon[92422]: pgmap v20850: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 975 KiB/s rd, 803 KiB/s wr, 12 op/s 2023-03-10T01:33:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38600.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:33:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:33:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38138.992251 secs 2023-03-10T01:33:49.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:33:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38138.992251 secs 2023-03-10T01:33:49.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:33:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38138.992251 secs 2023-03-10T01:33:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:33:49 smithi122 ceph-mon[93869]: pgmap v20851: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 539 KiB/s wr, 10 op/s 2023-03-10T01:33:50.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:33:49 smithi181 ceph-mon[94189]: pgmap v20851: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 539 KiB/s wr, 10 op/s 2023-03-10T01:33:50.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:33:49 smithi038 ceph-mon[92422]: pgmap v20851: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 539 KiB/s wr, 10 op/s 2023-03-10T01:33:51.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:33:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38141.533995 secs 2023-03-10T01:33:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:33:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38141.533995 secs 2023-03-10T01:33:51.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:33:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38141.533995 secs 2023-03-10T01:33:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:33:52 smithi122 ceph-mon[93869]: pgmap v20852: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 413 KiB/s wr, 11 op/s 2023-03-10T01:33:52.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:33:52 smithi181 ceph-mon[94189]: pgmap v20852: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 413 KiB/s wr, 11 op/s 2023-03-10T01:33:52.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:33:52 smithi038 ceph-mon[92422]: pgmap v20852: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 413 KiB/s wr, 11 op/s 2023-03-10T01:33:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:33:53 smithi122 ceph-mon[93869]: pgmap v20853: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 476 KiB/s rd, 413 KiB/s wr, 9 op/s 2023-03-10T01:33:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:33:53 smithi181 ceph-mon[94189]: pgmap v20853: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 476 KiB/s rd, 413 KiB/s wr, 9 op/s 2023-03-10T01:33:53.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:33:53 smithi038 ceph-mon[92422]: pgmap v20853: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 476 KiB/s rd, 413 KiB/s wr, 9 op/s 2023-03-10T01:33:53.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38605.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:33:54.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:33:54 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:33:54.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:33:54 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:33:54.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:33:54 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:33:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:33:55 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:33:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:33:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:33:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:33:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:33:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:33:55 smithi122 ceph-mon[93869]: pgmap v20854: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 769 KiB/s rd, 599 KiB/s wr, 9 op/s 2023-03-10T01:33:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:33:55 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:33:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:33:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:33:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:33:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:33:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:33:55 smithi181 ceph-mon[94189]: pgmap v20854: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 769 KiB/s rd, 599 KiB/s wr, 9 op/s 2023-03-10T01:33:55.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:33:55 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:33:55.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:33:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:33:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:33:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:33:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:33:55 smithi038 ceph-mon[92422]: pgmap v20854: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 769 KiB/s rd, 599 KiB/s wr, 9 op/s 2023-03-10T01:33:56.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:33:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38146.534057 secs 2023-03-10T01:33:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:33:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38146.534057 secs 2023-03-10T01:33:57.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:33:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38146.534057 secs 2023-03-10T01:33:57.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:33:57 smithi038 ceph-mon[92422]: pgmap v20855: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 867 KiB/s rd, 694 KiB/s wr, 13 op/s 2023-03-10T01:33:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:33:57 smithi122 ceph-mon[93869]: pgmap v20855: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 867 KiB/s rd, 694 KiB/s wr, 13 op/s 2023-03-10T01:33:58.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:33:57 smithi181 ceph-mon[94189]: pgmap v20855: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 867 KiB/s rd, 694 KiB/s wr, 13 op/s 2023-03-10T01:33:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38610.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:33:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:33:59 smithi122 ceph-mon[93869]: pgmap v20856: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 654 KiB/s rd, 469 KiB/s wr, 11 op/s 2023-03-10T01:33:59.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:33:59 smithi181 ceph-mon[94189]: pgmap v20856: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 654 KiB/s rd, 469 KiB/s wr, 11 op/s 2023-03-10T01:33:59.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:33:59 smithi038 ceph-mon[92422]: pgmap v20856: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 654 KiB/s rd, 469 KiB/s wr, 11 op/s 2023-03-10T01:34:01.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38151.534156 secs 2023-03-10T01:34:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38151.534156 secs 2023-03-10T01:34:01.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38151.534156 secs 2023-03-10T01:34:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:02 smithi122 ceph-mon[93869]: pgmap v20857: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 854 KiB/s wr, 16 op/s 2023-03-10T01:34:02.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:02 smithi181 ceph-mon[94189]: pgmap v20857: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 854 KiB/s wr, 16 op/s 2023-03-10T01:34:02.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:02 smithi038 ceph-mon[92422]: pgmap v20857: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 854 KiB/s wr, 16 op/s 2023-03-10T01:34:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:03 smithi122 ceph-mon[93869]: pgmap v20858: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 25 op/s 2023-03-10T01:34:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:03 smithi181 ceph-mon[94189]: pgmap v20858: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 25 op/s 2023-03-10T01:34:03.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:03 smithi038 ceph-mon[92422]: pgmap v20858: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 25 op/s 2023-03-10T01:34:03.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38615.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:34:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:05 smithi038 ceph-mon[92422]: pgmap v20859: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 24 op/s 2023-03-10T01:34:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:05 smithi122 ceph-mon[93869]: pgmap v20859: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 24 op/s 2023-03-10T01:34:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:05 smithi181 ceph-mon[94189]: pgmap v20859: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 24 op/s 2023-03-10T01:34:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38156.534242 secs 2023-03-10T01:34:07.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38156.534242 secs 2023-03-10T01:34:07.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38156.534242 secs 2023-03-10T01:34:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:07 smithi122 ceph-mon[93869]: pgmap v20860: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 978 KiB/s wr, 28 op/s 2023-03-10T01:34:08.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:07 smithi181 ceph-mon[94189]: pgmap v20860: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 978 KiB/s wr, 28 op/s 2023-03-10T01:34:08.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:07 smithi038 ceph-mon[92422]: pgmap v20860: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 978 KiB/s wr, 28 op/s 2023-03-10T01:34:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38620.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:34:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:09 smithi122 ceph-mon[93869]: pgmap v20861: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 969 KiB/s rd, 883 KiB/s wr, 24 op/s 2023-03-10T01:34:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:09 smithi181 ceph-mon[94189]: pgmap v20861: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 969 KiB/s rd, 883 KiB/s wr, 24 op/s 2023-03-10T01:34:09.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:09 smithi038 ceph-mon[92422]: pgmap v20861: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 969 KiB/s rd, 883 KiB/s wr, 24 op/s 2023-03-10T01:34:11.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38161.534328 secs 2023-03-10T01:34:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38161.534328 secs 2023-03-10T01:34:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38161.534328 secs 2023-03-10T01:34:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:12 smithi122 ceph-mon[93869]: pgmap v20862: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 26 op/s 2023-03-10T01:34:12.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:12 smithi181 ceph-mon[94189]: pgmap v20862: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 26 op/s 2023-03-10T01:34:12.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:12 smithi038 ceph-mon[92422]: pgmap v20862: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 26 op/s 2023-03-10T01:34:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:13 smithi122 ceph-mon[93869]: pgmap v20863: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 832 KiB/s wr, 21 op/s 2023-03-10T01:34:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:13 smithi181 ceph-mon[94189]: pgmap v20863: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 832 KiB/s wr, 21 op/s 2023-03-10T01:34:13.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:13 smithi038 ceph-mon[92422]: pgmap v20863: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 832 KiB/s wr, 21 op/s 2023-03-10T01:34:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38625.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:34:15.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:15 smithi038 ceph-mon[92422]: pgmap v20864: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 740 KiB/s rd, 543 KiB/s wr, 9 op/s 2023-03-10T01:34:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:15 smithi122 ceph-mon[93869]: pgmap v20864: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 740 KiB/s rd, 543 KiB/s wr, 9 op/s 2023-03-10T01:34:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:15 smithi181 ceph-mon[94189]: pgmap v20864: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 740 KiB/s rd, 543 KiB/s wr, 9 op/s 2023-03-10T01:34:16.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38166.534413 secs 2023-03-10T01:34:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38166.534413 secs 2023-03-10T01:34:17.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38166.534413 secs 2023-03-10T01:34:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:17 smithi122 ceph-mon[93869]: pgmap v20865: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 741 KiB/s rd, 543 KiB/s wr, 11 op/s 2023-03-10T01:34:18.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:17 smithi181 ceph-mon[94189]: pgmap v20865: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 741 KiB/s rd, 543 KiB/s wr, 11 op/s 2023-03-10T01:34:18.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:17 smithi038 ceph-mon[92422]: pgmap v20865: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 741 KiB/s rd, 543 KiB/s wr, 11 op/s 2023-03-10T01:34:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38630.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:34:19.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:19 smithi122 ceph-mon[93869]: pgmap v20866: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 694 KiB/s rd, 568 KiB/s wr, 9 op/s 2023-03-10T01:34:19.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:19 smithi181 ceph-mon[94189]: pgmap v20866: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 694 KiB/s rd, 568 KiB/s wr, 9 op/s 2023-03-10T01:34:19.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:19 smithi038 ceph-mon[92422]: pgmap v20866: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 694 KiB/s rd, 568 KiB/s wr, 9 op/s 2023-03-10T01:34:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:20 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:34:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:20 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:34:21.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:20 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:34:21.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38171.534472 secs 2023-03-10T01:34:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:21 smithi038 ceph-mon[92422]: pgmap v20867: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 883 KiB/s rd, 768 KiB/s wr, 13 op/s 2023-03-10T01:34:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38171.534472 secs 2023-03-10T01:34:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:21 smithi122 ceph-mon[93869]: pgmap v20867: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 883 KiB/s rd, 768 KiB/s wr, 13 op/s 2023-03-10T01:34:22.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38171.534472 secs 2023-03-10T01:34:22.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:21 smithi181 ceph-mon[94189]: pgmap v20867: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 883 KiB/s rd, 768 KiB/s wr, 13 op/s 2023-03-10T01:34:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38635.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:34:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:23 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:34:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:23 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:34:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:23 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:34:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:23 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:34:23.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:23 smithi122 ceph-mon[93869]: pgmap v20868: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 434 KiB/s wr, 10 op/s 2023-03-10T01:34:23.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:23 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:34:23.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:23 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:34:23.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:23 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:34:23.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:23 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:34:23.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:23 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:34:23.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:23 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:34:23.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:23 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:34:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:23 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:34:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:23 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:34:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:23 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:34:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:23 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:34:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:23 smithi181 ceph-mon[94189]: pgmap v20868: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 434 KiB/s wr, 10 op/s 2023-03-10T01:34:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:23 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:34:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:23 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:34:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:23 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:34:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:23 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:34:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:23 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:34:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:23 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:34:23.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:23 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:34:23.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:23 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:34:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:23 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:34:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:23 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:34:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:23 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:34:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:23 smithi038 ceph-mon[92422]: pgmap v20868: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 414 KiB/s rd, 434 KiB/s wr, 10 op/s 2023-03-10T01:34:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:23 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:34:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:23 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:34:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:23 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:34:24.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:23 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:34:24.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:23 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:34:24.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:23 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:34:24.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:23 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:34:25.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:24 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-10T01:34:25.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:24 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-10T01:34:25.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:34:25.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:34:25.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:34:25.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:34:25.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:24 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-10T01:34:25.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:24 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-10T01:34:25.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:34:25.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:34:25.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:34:25.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:24 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-10T01:34:25.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:24 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-10T01:34:25.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:34:25.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:34:25.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:34:25.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:34:25.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:24 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-10T01:34:25.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:24 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-10T01:34:25.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:34:25.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:34:25.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:34:25.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:24 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-10T01:34:25.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:24 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-10T01:34:25.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:34:25.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:34:25.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:34:25.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:34:25.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:24 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-10T01:34:25.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:24 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-10T01:34:25.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:34:25.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:34:25.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:34:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:26 smithi122 ceph-mon[93869]: osdmap e1519: 12 total, 12 up, 12 in 2023-03-10T01:34:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38176.534537 secs 2023-03-10T01:34:26.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:26 smithi122 ceph-mon[93869]: pgmap v20870: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 639 KiB/s rd, 663 KiB/s wr, 13 op/s 2023-03-10T01:34:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:26 smithi181 ceph-mon[94189]: osdmap e1519: 12 total, 12 up, 12 in 2023-03-10T01:34:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38176.534537 secs 2023-03-10T01:34:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:26 smithi181 ceph-mon[94189]: pgmap v20870: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 639 KiB/s rd, 663 KiB/s wr, 13 op/s 2023-03-10T01:34:26.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:26 smithi038 ceph-mon[92422]: osdmap e1519: 12 total, 12 up, 12 in 2023-03-10T01:34:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38176.534537 secs 2023-03-10T01:34:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:26 smithi038 ceph-mon[92422]: pgmap v20870: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 639 KiB/s rd, 663 KiB/s wr, 13 op/s 2023-03-10T01:34:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:27 smithi122 ceph-mon[93869]: pgmap v20871: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 826 KiB/s rd, 850 KiB/s wr, 13 op/s 2023-03-10T01:34:27.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:27 smithi181 ceph-mon[94189]: pgmap v20871: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 826 KiB/s rd, 850 KiB/s wr, 13 op/s 2023-03-10T01:34:27.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:27 smithi038 ceph-mon[92422]: pgmap v20871: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 826 KiB/s rd, 850 KiB/s wr, 13 op/s 2023-03-10T01:34:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38640.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:34:29.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:29 smithi122 ceph-mon[93869]: pgmap v20872: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 558 KiB/s rd, 570 KiB/s wr, 11 op/s 2023-03-10T01:34:29.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:29 smithi181 ceph-mon[94189]: pgmap v20872: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 558 KiB/s rd, 570 KiB/s wr, 11 op/s 2023-03-10T01:34:29.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:29 smithi038 ceph-mon[92422]: pgmap v20872: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 558 KiB/s rd, 570 KiB/s wr, 11 op/s 2023-03-10T01:34:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:30 smithi038 ceph-mon[92422]: osdmap e1520: 12 total, 12 up, 12 in 2023-03-10T01:34:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:30 smithi122 ceph-mon[93869]: osdmap e1520: 12 total, 12 up, 12 in 2023-03-10T01:34:31.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:30 smithi181 ceph-mon[94189]: osdmap e1520: 12 total, 12 up, 12 in 2023-03-10T01:34:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38181.534594 secs 2023-03-10T01:34:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:31 smithi122 ceph-mon[93869]: pgmap v20874: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 614 KiB/s rd, 612 KiB/s wr, 10 op/s 2023-03-10T01:34:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38181.534594 secs 2023-03-10T01:34:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:31 smithi181 ceph-mon[94189]: pgmap v20874: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 614 KiB/s rd, 612 KiB/s wr, 10 op/s 2023-03-10T01:34:32.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38181.534594 secs 2023-03-10T01:34:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:31 smithi038 ceph-mon[92422]: pgmap v20874: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 614 KiB/s rd, 612 KiB/s wr, 10 op/s 2023-03-10T01:34:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:33 smithi122 ceph-mon[93869]: pgmap v20875: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 422 KiB/s wr, 10 op/s 2023-03-10T01:34:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:33 smithi181 ceph-mon[94189]: pgmap v20875: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 422 KiB/s wr, 10 op/s 2023-03-10T01:34:33.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:33 smithi038 ceph-mon[92422]: pgmap v20875: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 422 KiB/s wr, 10 op/s 2023-03-10T01:34:33.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38645.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:34:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:35 smithi122 ceph-mon[93869]: pgmap v20876: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 347 KiB/s wr, 9 op/s 2023-03-10T01:34:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:35 smithi181 ceph-mon[94189]: pgmap v20876: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 347 KiB/s wr, 9 op/s 2023-03-10T01:34:36.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:35 smithi038 ceph-mon[92422]: pgmap v20876: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 347 KiB/s wr, 9 op/s 2023-03-10T01:34:37.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38186.534679 secs 2023-03-10T01:34:37.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38186.534679 secs 2023-03-10T01:34:37.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38186.534679 secs 2023-03-10T01:34:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:37 smithi122 ceph-mon[93869]: pgmap v20877: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 671 KiB/s rd, 670 KiB/s wr, 10 op/s 2023-03-10T01:34:38.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:37 smithi181 ceph-mon[94189]: pgmap v20877: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 671 KiB/s rd, 670 KiB/s wr, 10 op/s 2023-03-10T01:34:38.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:37 smithi038 ceph-mon[92422]: pgmap v20877: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 671 KiB/s rd, 670 KiB/s wr, 10 op/s 2023-03-10T01:34:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38650.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:34:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38188.993013 secs 2023-03-10T01:34:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:38 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:34:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:38 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:34:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:38 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:34:39.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38188.993013 secs 2023-03-10T01:34:39.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:38 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:34:39.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:38 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:34:39.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:38 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:34:39.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38188.993013 secs 2023-03-10T01:34:39.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:38 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:34:39.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:38 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:34:39.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:38 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:34:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:39 smithi122 ceph-mon[93869]: pgmap v20878: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 838 KiB/s rd, 837 KiB/s wr, 10 op/s 2023-03-10T01:34:40.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:39 smithi181 ceph-mon[94189]: pgmap v20878: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 838 KiB/s rd, 837 KiB/s wr, 10 op/s 2023-03-10T01:34:40.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:39 smithi038 ceph-mon[92422]: pgmap v20878: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 838 KiB/s rd, 837 KiB/s wr, 10 op/s 2023-03-10T01:34:41.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38191.534737 secs 2023-03-10T01:34:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38191.534737 secs 2023-03-10T01:34:41.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38191.534737 secs 2023-03-10T01:34:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:42 smithi122 ceph-mon[93869]: pgmap v20879: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 750 KiB/s rd, 748 KiB/s wr, 12 op/s 2023-03-10T01:34:42.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:42 smithi181 ceph-mon[94189]: pgmap v20879: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 750 KiB/s rd, 748 KiB/s wr, 12 op/s 2023-03-10T01:34:42.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:42 smithi038 ceph-mon[92422]: pgmap v20879: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 750 KiB/s rd, 748 KiB/s wr, 12 op/s 2023-03-10T01:34:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38193.993096 secs 2023-03-10T01:34:43.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38193.993096 secs 2023-03-10T01:34:43.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38193.993096 secs 2023-03-10T01:34:43.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38655.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:34:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:44 smithi122 ceph-mon[93869]: pgmap v20880: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 584 KiB/s wr, 9 op/s 2023-03-10T01:34:44.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:44 smithi181 ceph-mon[94189]: pgmap v20880: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 584 KiB/s wr, 9 op/s 2023-03-10T01:34:44.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:44 smithi038 ceph-mon[92422]: pgmap v20880: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 584 KiB/s wr, 9 op/s 2023-03-10T01:34:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:45 smithi122 ceph-mon[93869]: pgmap v20881: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 788 KiB/s rd, 786 KiB/s wr, 9 op/s 2023-03-10T01:34:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:45 smithi181 ceph-mon[94189]: pgmap v20881: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 788 KiB/s rd, 786 KiB/s wr, 9 op/s 2023-03-10T01:34:46.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:45 smithi038 ceph-mon[92422]: pgmap v20881: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 788 KiB/s rd, 786 KiB/s wr, 9 op/s 2023-03-10T01:34:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38196.534796 secs 2023-03-10T01:34:47.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38196.534796 secs 2023-03-10T01:34:47.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38196.534796 secs 2023-03-10T01:34:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:47 smithi122 ceph-mon[93869]: pgmap v20882: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 956 KiB/s rd, 953 KiB/s wr, 12 op/s 2023-03-10T01:34:48.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:47 smithi181 ceph-mon[94189]: pgmap v20882: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 956 KiB/s rd, 953 KiB/s wr, 12 op/s 2023-03-10T01:34:48.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:47 smithi038 ceph-mon[92422]: pgmap v20882: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 956 KiB/s rd, 953 KiB/s wr, 12 op/s 2023-03-10T01:34:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38660.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:34:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:49 smithi122 ceph-mon[93869]: pgmap v20883: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 530 KiB/s rd, 529 KiB/s wr, 9 op/s 2023-03-10T01:34:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:49 smithi181 ceph-mon[94189]: pgmap v20883: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 530 KiB/s rd, 529 KiB/s wr, 9 op/s 2023-03-10T01:34:49.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:49 smithi038 ceph-mon[92422]: pgmap v20883: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 530 KiB/s rd, 529 KiB/s wr, 9 op/s 2023-03-10T01:34:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:50 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:34:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:50 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:34:50.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:50 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:34:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38201.534855 secs 2023-03-10T01:34:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:51 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:34:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:34:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:34:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:51 smithi122 ceph-mon[93869]: pgmap v20884: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 584 KiB/s rd, 581 KiB/s wr, 11 op/s 2023-03-10T01:34:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38201.534855 secs 2023-03-10T01:34:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:51 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:34:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:34:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:34:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:51 smithi181 ceph-mon[94189]: pgmap v20884: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 584 KiB/s rd, 581 KiB/s wr, 11 op/s 2023-03-10T01:34:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38201.534855 secs 2023-03-10T01:34:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:51 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:34:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:34:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:34:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:51 smithi038 ceph-mon[92422]: pgmap v20884: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 584 KiB/s rd, 581 KiB/s wr, 11 op/s 2023-03-10T01:34:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:53 smithi122 ceph-mon[93869]: pgmap v20885: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 738 KiB/s rd, 736 KiB/s wr, 9 op/s 2023-03-10T01:34:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:53 smithi181 ceph-mon[94189]: pgmap v20885: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 738 KiB/s rd, 736 KiB/s wr, 9 op/s 2023-03-10T01:34:53.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:53 smithi038 ceph-mon[92422]: pgmap v20885: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 738 KiB/s rd, 736 KiB/s wr, 9 op/s 2023-03-10T01:34:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38665.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:34:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:55 smithi122 ceph-mon[93869]: pgmap v20886: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 738 KiB/s rd, 736 KiB/s wr, 10 op/s 2023-03-10T01:34:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:55 smithi181 ceph-mon[94189]: pgmap v20886: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 738 KiB/s rd, 736 KiB/s wr, 10 op/s 2023-03-10T01:34:56.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:55 smithi038 ceph-mon[92422]: pgmap v20886: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 738 KiB/s rd, 736 KiB/s wr, 10 op/s 2023-03-10T01:34:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38206.534941 secs 2023-03-10T01:34:57.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38206.534941 secs 2023-03-10T01:34:57.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38206.534941 secs 2023-03-10T01:34:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:57 smithi122 ceph-mon[93869]: pgmap v20887: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 816 KiB/s rd, 812 KiB/s wr, 13 op/s 2023-03-10T01:34:58.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:57 smithi181 ceph-mon[94189]: pgmap v20887: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 816 KiB/s rd, 812 KiB/s wr, 13 op/s 2023-03-10T01:34:58.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:57 smithi038 ceph-mon[92422]: pgmap v20887: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 816 KiB/s rd, 812 KiB/s wr, 13 op/s 2023-03-10T01:34:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38670.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:34:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38208.993319 secs 2023-03-10T01:34:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:58 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:34:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:58 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:34:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:58 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:34:59.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38208.993319 secs 2023-03-10T01:34:59.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:58 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:34:59.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:58 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:34:59.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:58 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:34:59.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38208.993319 secs 2023-03-10T01:34:59.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:58 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:34:59.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:58 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:34:59.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:58 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:35:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:34:59 smithi122 ceph-mon[93869]: pgmap v20888: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 659 KiB/s rd, 645 KiB/s wr, 11 op/s 2023-03-10T01:35:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:34:59 smithi181 ceph-mon[94189]: pgmap v20888: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 659 KiB/s rd, 645 KiB/s wr, 11 op/s 2023-03-10T01:35:00.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:34:59 smithi038 ceph-mon[92422]: pgmap v20888: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 659 KiB/s rd, 645 KiB/s wr, 11 op/s 2023-03-10T01:35:01.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:35:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38211.535012 secs 2023-03-10T01:35:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:35:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38211.535012 secs 2023-03-10T01:35:01.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:35:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38211.535012 secs 2023-03-10T01:35:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:35:02 smithi122 ceph-mon[93869]: pgmap v20889: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 970 KiB/s rd, 954 KiB/s wr, 14 op/s 2023-03-10T01:35:02.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:35:02 smithi181 ceph-mon[94189]: pgmap v20889: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 970 KiB/s rd, 954 KiB/s wr, 14 op/s 2023-03-10T01:35:02.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:35:02 smithi038 ceph-mon[92422]: pgmap v20889: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 970 KiB/s rd, 954 KiB/s wr, 14 op/s 2023-03-10T01:35:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:35:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38213.993378 secs 2023-03-10T01:35:03.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:35:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38213.993378 secs 2023-03-10T01:35:03.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:35:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38213.993378 secs 2023-03-10T01:35:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38675.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:35:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:35:04 smithi122 ceph-mon[93869]: pgmap v20890: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 784 KiB/s rd, 773 KiB/s wr, 22 op/s 2023-03-10T01:35:04.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:35:04 smithi181 ceph-mon[94189]: pgmap v20890: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 784 KiB/s rd, 773 KiB/s wr, 22 op/s 2023-03-10T01:35:04.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:35:04 smithi038 ceph-mon[92422]: pgmap v20890: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 784 KiB/s rd, 773 KiB/s wr, 22 op/s 2023-03-10T01:35:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:35:05 smithi122 ceph-mon[93869]: pgmap v20891: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 940 KiB/s wr, 22 op/s 2023-03-10T01:35:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:35:05 smithi181 ceph-mon[94189]: pgmap v20891: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 940 KiB/s wr, 22 op/s 2023-03-10T01:35:06.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:35:05 smithi038 ceph-mon[92422]: pgmap v20891: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 940 KiB/s wr, 22 op/s 2023-03-10T01:35:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:35:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38216.535076 secs 2023-03-10T01:35:07.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:35:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38216.535076 secs 2023-03-10T01:35:07.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:35:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38216.535076 secs 2023-03-10T01:35:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:35:07 smithi122 ceph-mon[93869]: pgmap v20892: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 27 op/s 2023-03-10T01:35:08.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:35:07 smithi181 ceph-mon[94189]: pgmap v20892: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 27 op/s 2023-03-10T01:35:08.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:35:07 smithi038 ceph-mon[92422]: pgmap v20892: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 27 op/s 2023-03-10T01:35:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38680.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:35:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:35:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38218.993439 secs 2023-03-10T01:35:09.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:35:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38218.993439 secs 2023-03-10T01:35:09.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:35:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38218.993439 secs 2023-03-10T01:35:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:35:09 smithi122 ceph-mon[93869]: pgmap v20893: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 859 KiB/s wr, 23 op/s 2023-03-10T01:35:10.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:35:09 smithi181 ceph-mon[94189]: pgmap v20893: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 859 KiB/s wr, 23 op/s 2023-03-10T01:35:10.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:35:09 smithi038 ceph-mon[92422]: pgmap v20893: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 859 KiB/s wr, 23 op/s 2023-03-10T01:35:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:35:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38221.535132 secs 2023-03-10T01:35:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:35:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38221.535132 secs 2023-03-10T01:35:11.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:35:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38221.535132 secs 2023-03-10T01:35:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:35:12 smithi122 ceph-mon[93869]: pgmap v20894: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 984 KiB/s wr, 26 op/s 2023-03-10T01:35:12.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:35:12 smithi181 ceph-mon[94189]: pgmap v20894: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 984 KiB/s wr, 26 op/s 2023-03-10T01:35:12.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:35:12 smithi038 ceph-mon[92422]: pgmap v20894: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 984 KiB/s wr, 26 op/s 2023-03-10T01:35:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:35:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38223.993546 secs 2023-03-10T01:35:13.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:35:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38223.993546 secs 2023-03-10T01:35:13.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:35:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38223.993546 secs 2023-03-10T01:35:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38685.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:35:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:35:14 smithi122 ceph-mon[93869]: pgmap v20895: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 674 KiB/s wr, 22 op/s 2023-03-10T01:35:14.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:35:14 smithi181 ceph-mon[94189]: pgmap v20895: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 674 KiB/s wr, 22 op/s 2023-03-10T01:35:14.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:35:14 smithi038 ceph-mon[92422]: pgmap v20895: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 674 KiB/s wr, 22 op/s 2023-03-10T01:35:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:35:15 smithi122 ceph-mon[93869]: pgmap v20896: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 663 KiB/s wr, 11 op/s 2023-03-10T01:35:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:35:15 smithi181 ceph-mon[94189]: pgmap v20896: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 663 KiB/s wr, 11 op/s 2023-03-10T01:35:16.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:35:15 smithi038 ceph-mon[92422]: pgmap v20896: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 663 KiB/s wr, 11 op/s 2023-03-10T01:35:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:35:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38226.535189 secs 2023-03-10T01:35:17.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:35:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38226.535189 secs 2023-03-10T01:35:17.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:35:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38226.535189 secs 2023-03-10T01:35:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:35:17 smithi122 ceph-mon[93869]: pgmap v20897: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 579 KiB/s rd, 321 KiB/s wr, 12 op/s 2023-03-10T01:35:18.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:35:17 smithi181 ceph-mon[94189]: pgmap v20897: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 579 KiB/s rd, 321 KiB/s wr, 12 op/s 2023-03-10T01:35:18.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:35:17 smithi038 ceph-mon[92422]: pgmap v20897: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 579 KiB/s rd, 321 KiB/s wr, 12 op/s 2023-03-10T01:35:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38690.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:35:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:35:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38228.993658 secs 2023-03-10T01:35:19.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:35:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38228.993658 secs 2023-03-10T01:35:19.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:35:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38228.993658 secs 2023-03-10T01:35:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:35:19 smithi122 ceph-mon[93869]: pgmap v20898: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 570 KiB/s rd, 313 KiB/s wr, 8 op/s 2023-03-10T01:35:20.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:35:19 smithi181 ceph-mon[94189]: pgmap v20898: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 570 KiB/s rd, 313 KiB/s wr, 8 op/s 2023-03-10T01:35:20.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:35:19 smithi038 ceph-mon[92422]: pgmap v20898: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 570 KiB/s rd, 313 KiB/s wr, 8 op/s 2023-03-10T01:35:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:35:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38231.535245 secs 2023-03-10T01:35:21.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:35:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38231.535245 secs 2023-03-10T01:35:21.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:35:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38231.535245 secs 2023-03-10T01:35:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:35:22 smithi122 ceph-mon[93869]: pgmap v20899: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 392 KiB/s wr, 12 op/s 2023-03-10T01:35:22.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:35:22 smithi181 ceph-mon[94189]: pgmap v20899: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 392 KiB/s wr, 12 op/s 2023-03-10T01:35:22.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:35:22 smithi038 ceph-mon[92422]: pgmap v20899: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 392 KiB/s wr, 12 op/s 2023-03-10T01:35:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:35:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38233.993717 secs 2023-03-10T01:35:23.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:35:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38233.993717 secs 2023-03-10T01:35:23.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:35:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38233.993717 secs 2023-03-10T01:35:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38695.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:35:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:35:24 smithi122 ceph-mon[93869]: pgmap v20900: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 268 KiB/s wr, 8 op/s 2023-03-10T01:35:24.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:35:24 smithi181 ceph-mon[94189]: pgmap v20900: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 268 KiB/s wr, 8 op/s 2023-03-10T01:35:24.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:35:24 smithi038 ceph-mon[92422]: pgmap v20900: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 268 KiB/s wr, 8 op/s 2023-03-10T01:35:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:35:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:35:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:35:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:35:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:35:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:35:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:35:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:35:25.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:35:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:35:25.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:35:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:35:25.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:35:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:35:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:35:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:35:25.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:35:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:35:25.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:35:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:35:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:35:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:35:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:35:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:35:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:35:26 smithi122 ceph-mon[93869]: osdmap e1521: 12 total, 12 up, 12 in 2023-03-10T01:35:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:35:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38236.535307 secs 2023-03-10T01:35:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:35:26 smithi122 ceph-mon[93869]: pgmap v20902: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 329 KiB/s wr, 11 op/s 2023-03-10T01:35:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:35:26 smithi181 ceph-mon[94189]: osdmap e1521: 12 total, 12 up, 12 in 2023-03-10T01:35:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:35:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38236.535307 secs 2023-03-10T01:35:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:35:26 smithi181 ceph-mon[94189]: pgmap v20902: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 329 KiB/s wr, 11 op/s 2023-03-10T01:35:26.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:35:26 smithi038 ceph-mon[92422]: osdmap e1521: 12 total, 12 up, 12 in 2023-03-10T01:35:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:35:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38236.535307 secs 2023-03-10T01:35:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:35:26 smithi038 ceph-mon[92422]: pgmap v20902: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 329 KiB/s wr, 11 op/s 2023-03-10T01:35:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:35:27 smithi122 ceph-mon[93869]: pgmap v20903: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 321 KiB/s rd, 329 KiB/s wr, 12 op/s 2023-03-10T01:35:27.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:35:27 smithi181 ceph-mon[94189]: pgmap v20903: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 321 KiB/s rd, 329 KiB/s wr, 12 op/s 2023-03-10T01:35:27.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:35:27 smithi038 ceph-mon[92422]: pgmap v20903: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 321 KiB/s rd, 329 KiB/s wr, 12 op/s 2023-03-10T01:35:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:35:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38238.993781 secs 2023-03-10T01:35:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:35:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38238.993781 secs 2023-03-10T01:35:28.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:35:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38238.993781 secs 2023-03-10T01:35:28.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38700.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:35:29.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:35:29 smithi122 ceph-mon[93869]: pgmap v20904: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 328 KiB/s rd, 359 KiB/s wr, 12 op/s 2023-03-10T01:35:29.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:35:29 smithi181 ceph-mon[94189]: pgmap v20904: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 328 KiB/s rd, 359 KiB/s wr, 12 op/s 2023-03-10T01:35:29.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:35:29 smithi038 ceph-mon[92422]: pgmap v20904: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 328 KiB/s rd, 359 KiB/s wr, 12 op/s 2023-03-10T01:35:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:35:30 smithi122 ceph-mon[93869]: osdmap e1522: 12 total, 12 up, 12 in 2023-03-10T01:35:31.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:35:30 smithi181 ceph-mon[94189]: osdmap e1522: 12 total, 12 up, 12 in 2023-03-10T01:35:31.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:35:30 smithi038 ceph-mon[92422]: osdmap e1522: 12 total, 12 up, 12 in 2023-03-10T01:35:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:35:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38241.535367 secs 2023-03-10T01:35:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:35:31 smithi122 ceph-mon[93869]: pgmap v20906: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 330 KiB/s wr, 12 op/s 2023-03-10T01:35:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:35:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38241.535367 secs 2023-03-10T01:35:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:35:31 smithi181 ceph-mon[94189]: pgmap v20906: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 330 KiB/s wr, 12 op/s 2023-03-10T01:35:32.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:35:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38241.535367 secs 2023-03-10T01:35:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:35:31 smithi038 ceph-mon[92422]: pgmap v20906: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 330 KiB/s wr, 12 op/s 2023-03-10T01:35:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:35:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38243.993860 secs 2023-03-10T01:35:33.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:35:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38243.993860 secs 2023-03-10T01:35:33.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:35:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38243.993860 secs 2023-03-10T01:35:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38705.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:35:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:35:34 smithi122 ceph-mon[93869]: pgmap v20907: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 362 KiB/s wr, 13 op/s 2023-03-10T01:35:34.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:35:34 smithi181 ceph-mon[94189]: pgmap v20907: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 362 KiB/s wr, 13 op/s 2023-03-10T01:35:34.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:35:34 smithi038 ceph-mon[92422]: pgmap v20907: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 362 KiB/s wr, 13 op/s 2023-03-10T01:35:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:35:35 smithi122 ceph-mon[93869]: pgmap v20908: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 476 KiB/s rd, 502 KiB/s wr, 12 op/s 2023-03-10T01:35:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:35:35 smithi181 ceph-mon[94189]: pgmap v20908: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 476 KiB/s rd, 502 KiB/s wr, 12 op/s 2023-03-10T01:35:36.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:35:35 smithi038 ceph-mon[92422]: pgmap v20908: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 476 KiB/s rd, 502 KiB/s wr, 12 op/s 2023-03-10T01:35:37.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:35:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38246.535427 secs 2023-03-10T01:35:37.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:35:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38246.535427 secs 2023-03-10T01:35:37.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:35:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38246.535427 secs 2023-03-10T01:35:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:35:37 smithi122 ceph-mon[93869]: pgmap v20909: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 464 KiB/s rd, 501 KiB/s wr, 11 op/s 2023-03-10T01:35:38.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:35:37 smithi181 ceph-mon[94189]: pgmap v20909: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 464 KiB/s rd, 501 KiB/s wr, 11 op/s 2023-03-10T01:35:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:35:37 smithi038 ceph-mon[92422]: pgmap v20909: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 464 KiB/s rd, 501 KiB/s wr, 11 op/s 2023-03-10T01:35:38.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38710.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:35:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:35:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38248.993966 secs 2023-03-10T01:35:39.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:35:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38248.993966 secs 2023-03-10T01:35:39.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:35:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38248.993966 secs 2023-03-10T01:35:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:35:39 smithi122 ceph-mon[93869]: pgmap v20910: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 247 KiB/s wr, 10 op/s 2023-03-10T01:35:40.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:35:39 smithi181 ceph-mon[94189]: pgmap v20910: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 247 KiB/s wr, 10 op/s 2023-03-10T01:35:40.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:35:39 smithi038 ceph-mon[92422]: pgmap v20910: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 247 KiB/s wr, 10 op/s 2023-03-10T01:35:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:35:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38251.535485 secs 2023-03-10T01:35:41.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:35:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38251.535485 secs 2023-03-10T01:35:41.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:35:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38251.535485 secs 2023-03-10T01:35:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:35:42 smithi122 ceph-mon[93869]: pgmap v20911: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 215 KiB/s wr, 11 op/s 2023-03-10T01:35:42.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:35:42 smithi181 ceph-mon[94189]: pgmap v20911: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 215 KiB/s wr, 11 op/s 2023-03-10T01:35:42.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:35:42 smithi038 ceph-mon[92422]: pgmap v20911: 129 pgs: 129 active+clean; 9.8 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 246 KiB/s rd, 215 KiB/s wr, 11 op/s 2023-03-10T01:35:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:35:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38253.994026 secs 2023-03-10T01:35:43.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:35:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38253.994026 secs 2023-03-10T01:35:43.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:35:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38253.994026 secs 2023-03-10T01:35:43.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38715.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:35:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:35:44 smithi122 ceph-mon[93869]: pgmap v20912: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 477 KiB/s wr, 10 op/s 2023-03-10T01:35:44.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:35:44 smithi181 ceph-mon[94189]: pgmap v20912: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 477 KiB/s wr, 10 op/s 2023-03-10T01:35:44.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:35:44 smithi038 ceph-mon[92422]: pgmap v20912: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 477 KiB/s wr, 10 op/s 2023-03-10T01:35:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:35:45 smithi122 ceph-mon[93869]: pgmap v20913: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 590 KiB/s wr, 10 op/s 2023-03-10T01:35:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:35:45 smithi181 ceph-mon[94189]: pgmap v20913: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 590 KiB/s wr, 10 op/s 2023-03-10T01:35:46.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:35:45 smithi038 ceph-mon[92422]: pgmap v20913: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 590 KiB/s wr, 10 op/s 2023-03-10T01:35:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:35:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38256.535546 secs 2023-03-10T01:35:47.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:35:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38256.535546 secs 2023-03-10T01:35:47.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:35:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38256.535546 secs 2023-03-10T01:35:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:35:47 smithi122 ceph-mon[93869]: pgmap v20914: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 418 KiB/s wr, 10 op/s 2023-03-10T01:35:48.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:35:47 smithi181 ceph-mon[94189]: pgmap v20914: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 418 KiB/s wr, 10 op/s 2023-03-10T01:35:48.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:35:47 smithi038 ceph-mon[92422]: pgmap v20914: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 250 KiB/s rd, 418 KiB/s wr, 10 op/s 2023-03-10T01:35:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38720.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:35:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:35:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38258.994126 secs 2023-03-10T01:35:49.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:35:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38258.994126 secs 2023-03-10T01:35:49.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:35:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38258.994126 secs 2023-03-10T01:35:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:35:49 smithi122 ceph-mon[93869]: pgmap v20915: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 636 KiB/s rd, 487 KiB/s wr, 9 op/s 2023-03-10T01:35:50.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:35:49 smithi181 ceph-mon[94189]: pgmap v20915: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 636 KiB/s rd, 487 KiB/s wr, 9 op/s 2023-03-10T01:35:50.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:35:49 smithi038 ceph-mon[92422]: pgmap v20915: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 636 KiB/s rd, 487 KiB/s wr, 9 op/s 2023-03-10T01:35:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:35:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38261.535603 secs 2023-03-10T01:35:51.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:35:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38261.535603 secs 2023-03-10T01:35:51.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:35:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38261.535603 secs 2023-03-10T01:35:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:35:52 smithi122 ceph-mon[93869]: pgmap v20916: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 605 KiB/s rd, 487 KiB/s wr, 9 op/s 2023-03-10T01:35:52.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:35:52 smithi181 ceph-mon[94189]: pgmap v20916: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 605 KiB/s rd, 487 KiB/s wr, 9 op/s 2023-03-10T01:35:52.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:35:52 smithi038 ceph-mon[92422]: pgmap v20916: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 605 KiB/s rd, 487 KiB/s wr, 9 op/s 2023-03-10T01:35:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:35:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38263.994212 secs 2023-03-10T01:35:53.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:35:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38263.994212 secs 2023-03-10T01:35:53.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:35:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38263.994212 secs 2023-03-10T01:35:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38725.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:35:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:35:54 smithi122 ceph-mon[93869]: pgmap v20917: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 861 KiB/s wr, 10 op/s 2023-03-10T01:35:54.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:35:54 smithi181 ceph-mon[94189]: pgmap v20917: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 861 KiB/s wr, 10 op/s 2023-03-10T01:35:54.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:35:54 smithi038 ceph-mon[92422]: pgmap v20917: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 861 KiB/s wr, 10 op/s 2023-03-10T01:35:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:35:55 smithi122 ceph-mon[93869]: pgmap v20918: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 588 KiB/s wr, 8 op/s 2023-03-10T01:35:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:35:55 smithi181 ceph-mon[94189]: pgmap v20918: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 588 KiB/s wr, 8 op/s 2023-03-10T01:35:56.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:35:55 smithi038 ceph-mon[92422]: pgmap v20918: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 588 KiB/s wr, 8 op/s 2023-03-10T01:35:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:35:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38266.535665 secs 2023-03-10T01:35:57.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:35:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38266.535665 secs 2023-03-10T01:35:57.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:35:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38266.535665 secs 2023-03-10T01:35:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:35:57 smithi122 ceph-mon[93869]: pgmap v20919: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 471 KiB/s wr, 11 op/s 2023-03-10T01:35:58.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:35:57 smithi181 ceph-mon[94189]: pgmap v20919: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 471 KiB/s wr, 11 op/s 2023-03-10T01:35:58.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:35:57 smithi038 ceph-mon[92422]: pgmap v20919: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 471 KiB/s wr, 11 op/s 2023-03-10T01:35:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38730.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:35:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:35:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38268.994294 secs 2023-03-10T01:35:59.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:35:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38268.994294 secs 2023-03-10T01:35:59.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:35:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38268.994294 secs 2023-03-10T01:36:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:35:59 smithi122 ceph-mon[93869]: pgmap v20920: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 504 KiB/s wr, 11 op/s 2023-03-10T01:36:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:35:59 smithi181 ceph-mon[94189]: pgmap v20920: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 504 KiB/s wr, 11 op/s 2023-03-10T01:36:00.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:35:59 smithi038 ceph-mon[92422]: pgmap v20920: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 504 KiB/s wr, 11 op/s 2023-03-10T01:36:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:36:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38271.535722 secs 2023-03-10T01:36:01.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:36:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38271.535722 secs 2023-03-10T01:36:01.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:36:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38271.535722 secs 2023-03-10T01:36:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:36:02 smithi122 ceph-mon[93869]: pgmap v20921: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 749 KiB/s wr, 12 op/s 2023-03-10T01:36:02.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:36:02 smithi181 ceph-mon[94189]: pgmap v20921: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 749 KiB/s wr, 12 op/s 2023-03-10T01:36:02.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:36:02 smithi038 ceph-mon[92422]: pgmap v20921: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 749 KiB/s wr, 12 op/s 2023-03-10T01:36:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:36:03 smithi122 ceph-mon[93869]: pgmap v20922: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.1 MiB/s wr, 24 op/s 2023-03-10T01:36:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:36:03 smithi181 ceph-mon[94189]: pgmap v20922: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.1 MiB/s wr, 24 op/s 2023-03-10T01:36:03.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:36:03 smithi038 ceph-mon[92422]: pgmap v20922: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.1 MiB/s wr, 24 op/s 2023-03-10T01:36:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38735.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:36:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:36:05 smithi122 ceph-mon[93869]: pgmap v20923: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 962 KiB/s rd, 760 KiB/s wr, 21 op/s 2023-03-10T01:36:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:36:05 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:36:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:36:05 smithi181 ceph-mon[94189]: pgmap v20923: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 962 KiB/s rd, 760 KiB/s wr, 21 op/s 2023-03-10T01:36:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:36:05 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:36:06.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:36:05 smithi038 ceph-mon[92422]: pgmap v20923: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 962 KiB/s rd, 760 KiB/s wr, 21 op/s 2023-03-10T01:36:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:36:05 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:36:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:36:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38276.535781 secs 2023-03-10T01:36:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:36:06 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:36:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:36:06 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:36:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:36:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:36:07.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:36:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38276.535781 secs 2023-03-10T01:36:07.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:36:06 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:36:07.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:36:06 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:36:07.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:36:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:36:07.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:36:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38276.535781 secs 2023-03-10T01:36:07.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:36:06 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:36:07.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:36:06 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:36:07.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:36:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:36:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:36:07 smithi122 ceph-mon[93869]: pgmap v20924: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 999 KiB/s rd, 772 KiB/s wr, 26 op/s 2023-03-10T01:36:08.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:36:07 smithi181 ceph-mon[94189]: pgmap v20924: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 999 KiB/s rd, 772 KiB/s wr, 26 op/s 2023-03-10T01:36:08.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:36:07 smithi038 ceph-mon[92422]: pgmap v20924: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 999 KiB/s rd, 772 KiB/s wr, 26 op/s 2023-03-10T01:36:08.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38740.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:36:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:36:09 smithi122 ceph-mon[93869]: pgmap v20925: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 24 op/s 2023-03-10T01:36:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:36:09 smithi181 ceph-mon[94189]: pgmap v20925: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 24 op/s 2023-03-10T01:36:09.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:36:09 smithi038 ceph-mon[92422]: pgmap v20925: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 24 op/s 2023-03-10T01:36:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:36:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38281.535839 secs 2023-03-10T01:36:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:36:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38281.535839 secs 2023-03-10T01:36:11.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:36:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38281.535839 secs 2023-03-10T01:36:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:36:12 smithi122 ceph-mon[93869]: pgmap v20926: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 25 op/s 2023-03-10T01:36:12.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:36:12 smithi181 ceph-mon[94189]: pgmap v20926: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 25 op/s 2023-03-10T01:36:12.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:36:12 smithi038 ceph-mon[92422]: pgmap v20926: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 25 op/s 2023-03-10T01:36:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:36:13 smithi122 ceph-mon[93869]: pgmap v20927: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 872 KiB/s rd, 849 KiB/s wr, 23 op/s 2023-03-10T01:36:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:36:13 smithi181 ceph-mon[94189]: pgmap v20927: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 872 KiB/s rd, 849 KiB/s wr, 23 op/s 2023-03-10T01:36:13.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:36:13 smithi038 ceph-mon[92422]: pgmap v20927: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 872 KiB/s rd, 849 KiB/s wr, 23 op/s 2023-03-10T01:36:13.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38745.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:36:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:36:15 smithi122 ceph-mon[93869]: pgmap v20928: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 471 KiB/s wr, 10 op/s 2023-03-10T01:36:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:36:15 smithi181 ceph-mon[94189]: pgmap v20928: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 471 KiB/s wr, 10 op/s 2023-03-10T01:36:16.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:36:15 smithi038 ceph-mon[92422]: pgmap v20928: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 471 KiB/s wr, 10 op/s 2023-03-10T01:36:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:36:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38286.535896 secs 2023-03-10T01:36:17.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:36:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38286.535896 secs 2023-03-10T01:36:17.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:36:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38286.535896 secs 2023-03-10T01:36:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:36:17 smithi122 ceph-mon[93869]: pgmap v20929: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 600 KiB/s wr, 13 op/s 2023-03-10T01:36:18.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:36:17 smithi181 ceph-mon[94189]: pgmap v20929: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 600 KiB/s wr, 13 op/s 2023-03-10T01:36:18.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:36:17 smithi038 ceph-mon[92422]: pgmap v20929: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 600 KiB/s wr, 13 op/s 2023-03-10T01:36:18.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38750.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:36:19.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:36:19 smithi122 ceph-mon[93869]: pgmap v20930: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 591 KiB/s rd, 632 KiB/s wr, 10 op/s 2023-03-10T01:36:19.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:36:19 smithi181 ceph-mon[94189]: pgmap v20930: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 591 KiB/s rd, 632 KiB/s wr, 10 op/s 2023-03-10T01:36:19.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:36:19 smithi038 ceph-mon[92422]: pgmap v20930: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 591 KiB/s rd, 632 KiB/s wr, 10 op/s 2023-03-10T01:36:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:36:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38291.535969 secs 2023-03-10T01:36:21.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:36:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38291.535969 secs 2023-03-10T01:36:21.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:36:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38291.535969 secs 2023-03-10T01:36:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:36:22 smithi122 ceph-mon[93869]: pgmap v20931: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 631 KiB/s rd, 682 KiB/s wr, 13 op/s 2023-03-10T01:36:22.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:36:22 smithi181 ceph-mon[94189]: pgmap v20931: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 631 KiB/s rd, 682 KiB/s wr, 13 op/s 2023-03-10T01:36:22.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:36:22 smithi038 ceph-mon[92422]: pgmap v20931: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 631 KiB/s rd, 682 KiB/s wr, 13 op/s 2023-03-10T01:36:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:36:23 smithi122 ceph-mon[93869]: pgmap v20932: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 564 KiB/s wr, 12 op/s 2023-03-10T01:36:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:36:23 smithi181 ceph-mon[94189]: pgmap v20932: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 564 KiB/s wr, 12 op/s 2023-03-10T01:36:23.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:36:23 smithi038 ceph-mon[92422]: pgmap v20932: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 564 KiB/s wr, 12 op/s 2023-03-10T01:36:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38755.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:36:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:36:25 smithi122 ceph-mon[93869]: osdmap e1523: 12 total, 12 up, 12 in 2023-03-10T01:36:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:36:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:36:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:36:25 smithi122 ceph-mon[93869]: pgmap v20934: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 644 KiB/s rd, 668 KiB/s wr, 12 op/s 2023-03-10T01:36:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:36:25 smithi181 ceph-mon[94189]: osdmap e1523: 12 total, 12 up, 12 in 2023-03-10T01:36:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:36:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:36:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:36:25 smithi181 ceph-mon[94189]: pgmap v20934: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 644 KiB/s rd, 668 KiB/s wr, 12 op/s 2023-03-10T01:36:25.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:36:25 smithi038 ceph-mon[92422]: osdmap e1523: 12 total, 12 up, 12 in 2023-03-10T01:36:25.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:36:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:36:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:36:25 smithi038 ceph-mon[92422]: pgmap v20934: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 644 KiB/s rd, 668 KiB/s wr, 12 op/s 2023-03-10T01:36:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:36:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38296.535983 secs 2023-03-10T01:36:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:36:26 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:36:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:36:26 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:36:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:36:26 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:36:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:36:26 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:36:26.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:36:26 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:36:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:36:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38296.535983 secs 2023-03-10T01:36:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:36:26 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:36:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:36:26 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:36:26.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:36:26 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:36:26.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:36:26 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:36:26.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:36:26 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:36:26.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:36:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38296.535983 secs 2023-03-10T01:36:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:36:26 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:36:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:36:26 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:36:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:36:26 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:36:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:36:26 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:36:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:36:26 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:36:27.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:36:27 smithi181 ceph-mon[94189]: pgmap v20935: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 677 KiB/s rd, 702 KiB/s wr, 13 op/s 2023-03-10T01:36:27.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:36:27 smithi038 ceph-mon[92422]: pgmap v20935: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 677 KiB/s rd, 702 KiB/s wr, 13 op/s 2023-03-10T01:36:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:36:27 smithi122 ceph-mon[93869]: pgmap v20935: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 677 KiB/s rd, 702 KiB/s wr, 13 op/s 2023-03-10T01:36:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38760.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:36:29.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:36:29 smithi122 ceph-mon[93869]: pgmap v20936: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 677 KiB/s rd, 689 KiB/s wr, 11 op/s 2023-03-10T01:36:29.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:36:29 smithi181 ceph-mon[94189]: pgmap v20936: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 677 KiB/s rd, 689 KiB/s wr, 11 op/s 2023-03-10T01:36:29.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:36:29 smithi038 ceph-mon[92422]: pgmap v20936: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 677 KiB/s rd, 689 KiB/s wr, 11 op/s 2023-03-10T01:36:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:36:30 smithi122 ceph-mon[93869]: osdmap e1524: 12 total, 12 up, 12 in 2023-03-10T01:36:31.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:36:30 smithi181 ceph-mon[94189]: osdmap e1524: 12 total, 12 up, 12 in 2023-03-10T01:36:31.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:36:30 smithi038 ceph-mon[92422]: osdmap e1524: 12 total, 12 up, 12 in 2023-03-10T01:36:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:36:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38301.536052 secs 2023-03-10T01:36:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:36:31 smithi122 ceph-mon[93869]: pgmap v20938: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 285 KiB/s wr, 11 op/s 2023-03-10T01:36:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:36:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38301.536052 secs 2023-03-10T01:36:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:36:31 smithi181 ceph-mon[94189]: pgmap v20938: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 285 KiB/s wr, 11 op/s 2023-03-10T01:36:32.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:36:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38301.536052 secs 2023-03-10T01:36:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:36:31 smithi038 ceph-mon[92422]: pgmap v20938: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 239 KiB/s rd, 285 KiB/s wr, 11 op/s 2023-03-10T01:36:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:36:33 smithi122 ceph-mon[93869]: pgmap v20939: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 299 KiB/s wr, 11 op/s 2023-03-10T01:36:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:36:33 smithi181 ceph-mon[94189]: pgmap v20939: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 299 KiB/s wr, 11 op/s 2023-03-10T01:36:33.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:36:33 smithi038 ceph-mon[92422]: pgmap v20939: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 299 KiB/s wr, 11 op/s 2023-03-10T01:36:33.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38765.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:36:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:36:35 smithi122 ceph-mon[93869]: pgmap v20940: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 565 KiB/s wr, 10 op/s 2023-03-10T01:36:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:36:35 smithi181 ceph-mon[94189]: pgmap v20940: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 565 KiB/s wr, 10 op/s 2023-03-10T01:36:36.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:36:35 smithi038 ceph-mon[92422]: pgmap v20940: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 565 KiB/s wr, 10 op/s 2023-03-10T01:36:37.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:36:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38306.536114 secs 2023-03-10T01:36:37.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:36:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38306.536114 secs 2023-03-10T01:36:37.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:36:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38306.536114 secs 2023-03-10T01:36:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:36:37 smithi122 ceph-mon[93869]: pgmap v20941: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 872 KiB/s rd, 908 KiB/s wr, 10 op/s 2023-03-10T01:36:38.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:36:37 smithi181 ceph-mon[94189]: pgmap v20941: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 872 KiB/s rd, 908 KiB/s wr, 10 op/s 2023-03-10T01:36:38.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:36:37 smithi038 ceph-mon[92422]: pgmap v20941: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 872 KiB/s rd, 908 KiB/s wr, 10 op/s 2023-03-10T01:36:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38770.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:36:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:36:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38308.994883 secs 2023-03-10T01:36:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:36:38 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:36:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:36:38 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:36:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:36:38 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:36:39.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:36:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38308.994883 secs 2023-03-10T01:36:39.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:36:38 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:36:39.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:36:38 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:36:39.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:36:38 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:36:39.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:36:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38308.994883 secs 2023-03-10T01:36:39.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:36:38 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:36:39.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:36:38 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:36:39.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:36:38 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:36:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:36:39 smithi122 ceph-mon[93869]: pgmap v20942: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 KiB/s rd, 982 KiB/s wr, 11 op/s 2023-03-10T01:36:40.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:36:39 smithi181 ceph-mon[94189]: pgmap v20942: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 KiB/s rd, 982 KiB/s wr, 11 op/s 2023-03-10T01:36:40.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:36:39 smithi038 ceph-mon[92422]: pgmap v20942: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1023 KiB/s rd, 982 KiB/s wr, 11 op/s 2023-03-10T01:36:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:36:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38311.536172 secs 2023-03-10T01:36:41.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:36:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38311.536172 secs 2023-03-10T01:36:41.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:36:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38311.536172 secs 2023-03-10T01:36:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:36:42 smithi122 ceph-mon[93869]: pgmap v20943: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1016 KiB/s wr, 13 op/s 2023-03-10T01:36:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:36:42 smithi181 ceph-mon[94189]: pgmap v20943: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1016 KiB/s wr, 13 op/s 2023-03-10T01:36:42.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:36:42 smithi038 ceph-mon[92422]: pgmap v20943: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1016 KiB/s wr, 13 op/s 2023-03-10T01:36:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:36:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38313.995018 secs 2023-03-10T01:36:43.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:36:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38313.995018 secs 2023-03-10T01:36:43.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:36:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38313.995018 secs 2023-03-10T01:36:43.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38775.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:36:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:36:44 smithi122 ceph-mon[93869]: pgmap v20944: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1009 KiB/s rd, 975 KiB/s wr, 10 op/s 2023-03-10T01:36:44.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:36:44 smithi181 ceph-mon[94189]: pgmap v20944: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1009 KiB/s rd, 975 KiB/s wr, 10 op/s 2023-03-10T01:36:44.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:36:44 smithi038 ceph-mon[92422]: pgmap v20944: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1009 KiB/s rd, 975 KiB/s wr, 10 op/s 2023-03-10T01:36:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:36:45 smithi122 ceph-mon[93869]: pgmap v20945: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 977 KiB/s rd, 942 KiB/s wr, 10 op/s 2023-03-10T01:36:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:36:45 smithi181 ceph-mon[94189]: pgmap v20945: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 977 KiB/s rd, 942 KiB/s wr, 10 op/s 2023-03-10T01:36:46.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:36:45 smithi038 ceph-mon[92422]: pgmap v20945: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 977 KiB/s rd, 942 KiB/s wr, 10 op/s 2023-03-10T01:36:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:36:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38316.536235 secs 2023-03-10T01:36:47.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:36:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38316.536235 secs 2023-03-10T01:36:47.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:36:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38316.536235 secs 2023-03-10T01:36:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:36:47 smithi122 ceph-mon[93869]: pgmap v20946: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 13 op/s 2023-03-10T01:36:48.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:36:47 smithi181 ceph-mon[94189]: pgmap v20946: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 13 op/s 2023-03-10T01:36:48.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:36:47 smithi038 ceph-mon[92422]: pgmap v20946: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 13 op/s 2023-03-10T01:36:48.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38780.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:36:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:36:49 smithi122 ceph-mon[93869]: pgmap v20947: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 733 KiB/s rd, 583 KiB/s wr, 10 op/s 2023-03-10T01:36:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:36:49 smithi181 ceph-mon[94189]: pgmap v20947: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 733 KiB/s rd, 583 KiB/s wr, 10 op/s 2023-03-10T01:36:49.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:36:49 smithi038 ceph-mon[92422]: pgmap v20947: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 733 KiB/s rd, 583 KiB/s wr, 10 op/s 2023-03-10T01:36:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:36:50 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:36:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:36:50 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:36:50.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:36:50 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:36:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:36:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38321.536318 secs 2023-03-10T01:36:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:36:51 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:36:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:36:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:36:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:36:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:36:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:36:51 smithi122 ceph-mon[93869]: pgmap v20948: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 489 KiB/s wr, 12 op/s 2023-03-10T01:36:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:36:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38321.536318 secs 2023-03-10T01:36:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:36:51 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:36:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:36:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:36:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:36:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:36:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:36:51 smithi181 ceph-mon[94189]: pgmap v20948: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 489 KiB/s wr, 12 op/s 2023-03-10T01:36:51.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:36:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38321.536318 secs 2023-03-10T01:36:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:36:51 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:36:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:36:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:36:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:36:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:36:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:36:51 smithi038 ceph-mon[92422]: pgmap v20948: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 489 KiB/s wr, 12 op/s 2023-03-10T01:36:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:36:53 smithi122 ceph-mon[93869]: pgmap v20949: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 949 KiB/s rd, 365 KiB/s wr, 10 op/s 2023-03-10T01:36:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:36:53 smithi181 ceph-mon[94189]: pgmap v20949: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 949 KiB/s rd, 365 KiB/s wr, 10 op/s 2023-03-10T01:36:53.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:36:53 smithi038 ceph-mon[92422]: pgmap v20949: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 949 KiB/s rd, 365 KiB/s wr, 10 op/s 2023-03-10T01:36:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38785.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:36:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:36:55 smithi122 ceph-mon[93869]: pgmap v20950: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 577 KiB/s wr, 10 op/s 2023-03-10T01:36:56.354 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:36:55 smithi038 ceph-mon[92422]: pgmap v20950: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 577 KiB/s wr, 10 op/s 2023-03-10T01:36:56.354 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:36:55 smithi181 ceph-mon[94189]: pgmap v20950: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 577 KiB/s wr, 10 op/s 2023-03-10T01:36:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:36:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38326.536377 secs 2023-03-10T01:36:57.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:36:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38326.536377 secs 2023-03-10T01:36:57.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:36:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38326.536377 secs 2023-03-10T01:36:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:36:57 smithi122 ceph-mon[93869]: pgmap v20951: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 577 KiB/s wr, 12 op/s 2023-03-10T01:36:58.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:36:57 smithi181 ceph-mon[94189]: pgmap v20951: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 577 KiB/s wr, 12 op/s 2023-03-10T01:36:58.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:36:57 smithi038 ceph-mon[92422]: pgmap v20951: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 577 KiB/s wr, 12 op/s 2023-03-10T01:36:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38790.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:36:59.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:36:59 smithi122 ceph-mon[93869]: pgmap v20952: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 993 KiB/s rd, 717 KiB/s wr, 11 op/s 2023-03-10T01:36:59.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:36:59 smithi181 ceph-mon[94189]: pgmap v20952: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 993 KiB/s rd, 717 KiB/s wr, 11 op/s 2023-03-10T01:36:59.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:36:59 smithi038 ceph-mon[92422]: pgmap v20952: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 993 KiB/s rd, 717 KiB/s wr, 11 op/s 2023-03-10T01:37:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:37:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38331.536433 secs 2023-03-10T01:37:01.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:37:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38331.536433 secs 2023-03-10T01:37:01.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:37:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38331.536433 secs 2023-03-10T01:37:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:37:02 smithi122 ceph-mon[93869]: pgmap v20953: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1008 KiB/s rd, 731 KiB/s wr, 13 op/s 2023-03-10T01:37:02.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:37:02 smithi181 ceph-mon[94189]: pgmap v20953: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1008 KiB/s rd, 731 KiB/s wr, 13 op/s 2023-03-10T01:37:02.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:37:02 smithi038 ceph-mon[92422]: pgmap v20953: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1008 KiB/s rd, 731 KiB/s wr, 13 op/s 2023-03-10T01:37:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:37:03 smithi122 ceph-mon[93869]: pgmap v20954: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 24 op/s 2023-03-10T01:37:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:37:03 smithi181 ceph-mon[94189]: pgmap v20954: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 24 op/s 2023-03-10T01:37:03.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:37:03 smithi038 ceph-mon[92422]: pgmap v20954: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 24 op/s 2023-03-10T01:37:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38795.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:37:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:37:05 smithi122 ceph-mon[93869]: pgmap v20955: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 23 op/s 2023-03-10T01:37:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:37:05 smithi181 ceph-mon[94189]: pgmap v20955: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 23 op/s 2023-03-10T01:37:06.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:37:05 smithi038 ceph-mon[92422]: pgmap v20955: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 23 op/s 2023-03-10T01:37:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:37:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38336.536491 secs 2023-03-10T01:37:07.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:37:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38336.536491 secs 2023-03-10T01:37:07.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:37:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38336.536491 secs 2023-03-10T01:37:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:37:07 smithi122 ceph-mon[93869]: pgmap v20956: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 975 KiB/s wr, 27 op/s 2023-03-10T01:37:08.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:37:07 smithi181 ceph-mon[94189]: pgmap v20956: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 975 KiB/s wr, 27 op/s 2023-03-10T01:37:08.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:37:07 smithi038 ceph-mon[92422]: pgmap v20956: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 975 KiB/s wr, 27 op/s 2023-03-10T01:37:08.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38800.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:37:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:37:09 smithi122 ceph-mon[93869]: pgmap v20957: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 975 KiB/s wr, 25 op/s 2023-03-10T01:37:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:37:09 smithi181 ceph-mon[94189]: pgmap v20957: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 975 KiB/s wr, 25 op/s 2023-03-10T01:37:09.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:37:09 smithi038 ceph-mon[92422]: pgmap v20957: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 975 KiB/s wr, 25 op/s 2023-03-10T01:37:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:37:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38341.536550 secs 2023-03-10T01:37:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:37:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38341.536550 secs 2023-03-10T01:37:11.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:37:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38341.536550 secs 2023-03-10T01:37:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:37:12 smithi122 ceph-mon[93869]: pgmap v20958: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 684 KiB/s wr, 26 op/s 2023-03-10T01:37:12.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:37:12 smithi181 ceph-mon[94189]: pgmap v20958: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 684 KiB/s wr, 26 op/s 2023-03-10T01:37:12.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:37:12 smithi038 ceph-mon[92422]: pgmap v20958: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 684 KiB/s wr, 26 op/s 2023-03-10T01:37:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:37:13 smithi122 ceph-mon[93869]: pgmap v20959: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 670 KiB/s wr, 24 op/s 2023-03-10T01:37:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:37:13 smithi181 ceph-mon[94189]: pgmap v20959: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 670 KiB/s wr, 24 op/s 2023-03-10T01:37:13.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:37:13 smithi038 ceph-mon[92422]: pgmap v20959: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 670 KiB/s wr, 24 op/s 2023-03-10T01:37:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38805.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:37:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:37:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38346.536610 secs 2023-03-10T01:37:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:37:16 smithi122 ceph-mon[93869]: pgmap v20960: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 436 KiB/s rd, 266 KiB/s wr, 10 op/s 2023-03-10T01:37:17.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:37:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38346.536610 secs 2023-03-10T01:37:17.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:37:16 smithi181 ceph-mon[94189]: pgmap v20960: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 436 KiB/s rd, 266 KiB/s wr, 10 op/s 2023-03-10T01:37:17.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:37:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38346.536610 secs 2023-03-10T01:37:17.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:37:16 smithi038 ceph-mon[92422]: pgmap v20960: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 436 KiB/s rd, 266 KiB/s wr, 10 op/s 2023-03-10T01:37:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38810.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:37:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:37:18 smithi122 ceph-mon[93869]: pgmap v20961: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 266 KiB/s wr, 12 op/s 2023-03-10T01:37:19.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:37:18 smithi181 ceph-mon[94189]: pgmap v20961: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 266 KiB/s wr, 12 op/s 2023-03-10T01:37:19.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:37:18 smithi038 ceph-mon[92422]: pgmap v20961: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 266 KiB/s wr, 12 op/s 2023-03-10T01:37:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:37:20 smithi122 ceph-mon[93869]: pgmap v20962: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 363 KiB/s rd, 192 KiB/s wr, 9 op/s 2023-03-10T01:37:21.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:37:20 smithi181 ceph-mon[94189]: pgmap v20962: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 363 KiB/s rd, 192 KiB/s wr, 9 op/s 2023-03-10T01:37:21.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:37:20 smithi038 ceph-mon[92422]: pgmap v20962: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 363 KiB/s rd, 192 KiB/s wr, 9 op/s 2023-03-10T01:37:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:37:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38351.536667 secs 2023-03-10T01:37:22.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:37:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38351.536667 secs 2023-03-10T01:37:22.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:37:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38351.536667 secs 2023-03-10T01:37:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:37:22 smithi122 ceph-mon[93869]: pgmap v20963: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 323 KiB/s wr, 12 op/s 2023-03-10T01:37:23.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:37:22 smithi181 ceph-mon[94189]: pgmap v20963: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 323 KiB/s wr, 12 op/s 2023-03-10T01:37:23.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:37:22 smithi038 ceph-mon[92422]: pgmap v20963: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 323 KiB/s wr, 12 op/s 2023-03-10T01:37:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38815.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:37:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:37:23 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 38353.995489 secs 2023-03-10T01:37:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:37:23 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 38353.995489 secs 2023-03-10T01:37:24.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:37:23 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 38353.995489 secs 2023-03-10T01:37:25.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:37:24 smithi122 ceph-mon[93869]: pgmap v20964: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 401 KiB/s wr, 10 op/s 2023-03-10T01:37:25.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:37:24 smithi181 ceph-mon[94189]: pgmap v20964: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 401 KiB/s wr, 10 op/s 2023-03-10T01:37:25.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:37:24 smithi038 ceph-mon[92422]: pgmap v20964: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 382 KiB/s rd, 401 KiB/s wr, 10 op/s 2023-03-10T01:37:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:37:25 smithi122 ceph-mon[93869]: osdmap e1525: 12 total, 12 up, 12 in 2023-03-10T01:37:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:37:25 smithi181 ceph-mon[94189]: osdmap e1525: 12 total, 12 up, 12 in 2023-03-10T01:37:26.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:37:25 smithi038 ceph-mon[92422]: osdmap e1525: 12 total, 12 up, 12 in 2023-03-10T01:37:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:37:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38356.536728 secs 2023-03-10T01:37:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:37:26 smithi122 ceph-mon[93869]: pgmap v20966: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 482 KiB/s wr, 12 op/s 2023-03-10T01:37:27.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:37:26 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:37:27.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:37:26 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:37:27.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:37:26 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:37:27.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:37:26 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:37:27.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:37:26 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:37:27.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:37:26 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:37:27.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:37:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38356.536728 secs 2023-03-10T01:37:27.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:37:26 smithi181 ceph-mon[94189]: pgmap v20966: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 482 KiB/s wr, 12 op/s 2023-03-10T01:37:27.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:37:26 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:37:27.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:37:26 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:37:27.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:37:26 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:37:27.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:37:26 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:37:27.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:37:26 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:37:27.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:37:26 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:37:27.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:37:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38356.536728 secs 2023-03-10T01:37:27.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:37:26 smithi038 ceph-mon[92422]: pgmap v20966: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 482 KiB/s wr, 12 op/s 2023-03-10T01:37:27.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:37:26 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:37:27.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:37:26 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:37:27.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:37:26 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:37:27.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:37:26 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:37:27.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:37:26 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:37:27.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:37:26 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:37:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:37:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:37:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:37:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:37:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:37:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:37:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:37:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:37:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:37:27 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:37:28.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:37:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:37:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:37:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:37:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:37:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:37:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:37:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:37:28.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:37:27 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:37:28.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:37:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:37:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:37:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:37:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:37:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:37:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:37:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:37:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:37:27 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:37:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38820.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:37:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:37:28 smithi122 ceph-mon[93869]: pgmap v20967: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 14 op/s 2023-03-10T01:37:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:37:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38358.995556 secs 2023-03-10T01:37:29.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:37:28 smithi181 ceph-mon[94189]: pgmap v20967: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 14 op/s 2023-03-10T01:37:29.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:37:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38358.995556 secs 2023-03-10T01:37:29.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:37:28 smithi038 ceph-mon[92422]: pgmap v20967: 129 pgs: 129 active+clean; 9.9 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 14 op/s 2023-03-10T01:37:29.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:37:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38358.995556 secs 2023-03-10T01:37:31.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:37:30 smithi122 ceph-mon[93869]: pgmap v20968: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 12 op/s 2023-03-10T01:37:31.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:37:30 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:37:31.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:37:30 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:37:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:37:30 smithi181 ceph-mon[94189]: pgmap v20968: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 12 op/s 2023-03-10T01:37:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:37:30 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:37:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:37:30 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:37:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:37:30 smithi038 ceph-mon[92422]: pgmap v20968: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 12 op/s 2023-03-10T01:37:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:37:30 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:37:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:37:30 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:37:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:37:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38361.536784 secs 2023-03-10T01:37:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:37:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38361.536784 secs 2023-03-10T01:37:32.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:37:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38361.536784 secs 2023-03-10T01:37:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:37:32 smithi122 ceph-mon[93869]: pgmap v20969: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 982 KiB/s rd, 981 KiB/s wr, 10 op/s 2023-03-10T01:37:33.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:37:32 smithi181 ceph-mon[94189]: pgmap v20969: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 982 KiB/s rd, 981 KiB/s wr, 10 op/s 2023-03-10T01:37:33.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:37:32 smithi038 ceph-mon[92422]: pgmap v20969: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 982 KiB/s rd, 981 KiB/s wr, 10 op/s 2023-03-10T01:37:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38825.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:37:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:37:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38363.995612 secs 2023-03-10T01:37:34.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:37:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38363.995612 secs 2023-03-10T01:37:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:37:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38363.995612 secs 2023-03-10T01:37:35.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:37:34 smithi122 ceph-mon[93869]: pgmap v20970: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 671 KiB/s rd, 670 KiB/s wr, 11 op/s 2023-03-10T01:37:35.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:37:34 smithi122 ceph-mon[93869]: osdmap e1526: 12 total, 12 up, 12 in 2023-03-10T01:37:35.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:37:34 smithi181 ceph-mon[94189]: pgmap v20970: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 671 KiB/s rd, 670 KiB/s wr, 11 op/s 2023-03-10T01:37:35.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:37:34 smithi181 ceph-mon[94189]: osdmap e1526: 12 total, 12 up, 12 in 2023-03-10T01:37:35.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:37:34 smithi038 ceph-mon[92422]: pgmap v20970: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 671 KiB/s rd, 670 KiB/s wr, 11 op/s 2023-03-10T01:37:35.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:37:34 smithi038 ceph-mon[92422]: osdmap e1526: 12 total, 12 up, 12 in 2023-03-10T01:37:37.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:37:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38366.536875 secs 2023-03-10T01:37:37.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:37:36 smithi122 ceph-mon[93869]: pgmap v20972: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 671 KiB/s rd, 670 KiB/s wr, 10 op/s 2023-03-10T01:37:37.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:37:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38366.536875 secs 2023-03-10T01:37:37.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:37:36 smithi181 ceph-mon[94189]: pgmap v20972: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 671 KiB/s rd, 670 KiB/s wr, 10 op/s 2023-03-10T01:37:37.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:37:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38366.536875 secs 2023-03-10T01:37:37.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:37:36 smithi038 ceph-mon[92422]: pgmap v20972: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 671 KiB/s rd, 670 KiB/s wr, 10 op/s 2023-03-10T01:37:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38830.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:37:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:37:38 smithi122 ceph-mon[93869]: pgmap v20973: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 514 KiB/s rd, 513 KiB/s wr, 10 op/s 2023-03-10T01:37:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:37:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38368.995675 secs 2023-03-10T01:37:39.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:37:38 smithi181 ceph-mon[94189]: pgmap v20973: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 514 KiB/s rd, 513 KiB/s wr, 10 op/s 2023-03-10T01:37:39.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:37:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38368.995675 secs 2023-03-10T01:37:39.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:37:38 smithi038 ceph-mon[92422]: pgmap v20973: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 514 KiB/s rd, 513 KiB/s wr, 10 op/s 2023-03-10T01:37:39.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:37:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38368.995675 secs 2023-03-10T01:37:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:37:40 smithi122 ceph-mon[93869]: pgmap v20974: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 514 KiB/s rd, 513 KiB/s wr, 10 op/s 2023-03-10T01:37:41.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:37:40 smithi181 ceph-mon[94189]: pgmap v20974: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 514 KiB/s rd, 513 KiB/s wr, 10 op/s 2023-03-10T01:37:41.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:37:40 smithi038 ceph-mon[92422]: pgmap v20974: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 514 KiB/s rd, 513 KiB/s wr, 10 op/s 2023-03-10T01:37:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:37:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38371.536932 secs 2023-03-10T01:37:42.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:37:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38371.536932 secs 2023-03-10T01:37:42.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:37:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38371.536932 secs 2023-03-10T01:37:43.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:37:42 smithi122 ceph-mon[93869]: pgmap v20975: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 434 KiB/s wr, 10 op/s 2023-03-10T01:37:43.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:37:42 smithi181 ceph-mon[94189]: pgmap v20975: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 434 KiB/s wr, 10 op/s 2023-03-10T01:37:43.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:37:42 smithi038 ceph-mon[92422]: pgmap v20975: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 434 KiB/s wr, 10 op/s 2023-03-10T01:37:43.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38835.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:37:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:37:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38373.995736 secs 2023-03-10T01:37:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:37:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38373.995736 secs 2023-03-10T01:37:44.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:37:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38373.995736 secs 2023-03-10T01:37:45.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:37:44 smithi122 ceph-mon[93869]: pgmap v20976: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 592 KiB/s rd, 589 KiB/s wr, 9 op/s 2023-03-10T01:37:45.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:37:44 smithi181 ceph-mon[94189]: pgmap v20976: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 592 KiB/s rd, 589 KiB/s wr, 9 op/s 2023-03-10T01:37:45.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:37:44 smithi038 ceph-mon[92422]: pgmap v20976: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 592 KiB/s rd, 589 KiB/s wr, 9 op/s 2023-03-10T01:37:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:37:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38376.536984 secs 2023-03-10T01:37:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:37:46 smithi122 ceph-mon[93869]: pgmap v20977: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 562 KiB/s rd, 560 KiB/s wr, 8 op/s 2023-03-10T01:37:47.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:37:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38376.536984 secs 2023-03-10T01:37:47.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:37:46 smithi181 ceph-mon[94189]: pgmap v20977: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 562 KiB/s rd, 560 KiB/s wr, 8 op/s 2023-03-10T01:37:47.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:37:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38376.536984 secs 2023-03-10T01:37:47.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:37:46 smithi038 ceph-mon[92422]: pgmap v20977: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 562 KiB/s rd, 560 KiB/s wr, 8 op/s 2023-03-10T01:37:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38840.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:37:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:37:48 smithi122 ceph-mon[93869]: pgmap v20978: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 788 KiB/s rd, 691 KiB/s wr, 11 op/s 2023-03-10T01:37:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:37:48 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 38378.995868 secs 2023-03-10T01:37:49.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:37:48 smithi181 ceph-mon[94189]: pgmap v20978: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 788 KiB/s rd, 691 KiB/s wr, 11 op/s 2023-03-10T01:37:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:37:48 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 38378.995868 secs 2023-03-10T01:37:49.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:37:48 smithi038 ceph-mon[92422]: pgmap v20978: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 788 KiB/s rd, 691 KiB/s wr, 11 op/s 2023-03-10T01:37:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:37:48 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 38378.995868 secs 2023-03-10T01:37:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:37:50 smithi122 ceph-mon[93869]: pgmap v20979: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 329 KiB/s wr, 7 op/s 2023-03-10T01:37:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:37:50 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:37:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:37:50 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:37:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:37:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38381.537044 secs 2023-03-10T01:37:51.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:37:50 smithi181 ceph-mon[94189]: pgmap v20979: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 329 KiB/s wr, 7 op/s 2023-03-10T01:37:51.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:37:50 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:37:51.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:37:50 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:37:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:37:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38381.537044 secs 2023-03-10T01:37:51.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:37:50 smithi038 ceph-mon[92422]: pgmap v20979: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 425 KiB/s rd, 329 KiB/s wr, 7 op/s 2023-03-10T01:37:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:37:50 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:37:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:37:50 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:37:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:37:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38381.537044 secs 2023-03-10T01:37:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:37:52 smithi122 ceph-mon[93869]: pgmap v20980: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 531 KiB/s wr, 10 op/s 2023-03-10T01:37:53.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:37:52 smithi181 ceph-mon[94189]: pgmap v20980: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 531 KiB/s wr, 10 op/s 2023-03-10T01:37:53.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:37:52 smithi038 ceph-mon[92422]: pgmap v20980: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 531 KiB/s wr, 10 op/s 2023-03-10T01:37:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38845.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:37:54.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:37:53 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 38383.995927 secs 2023-03-10T01:37:54.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:37:53 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 38383.995927 secs 2023-03-10T01:37:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:37:53 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 38383.995927 secs 2023-03-10T01:37:55.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:37:54 smithi038 ceph-mon[92422]: pgmap v20981: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 626 KiB/s rd, 531 KiB/s wr, 8 op/s 2023-03-10T01:37:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:37:54 smithi122 ceph-mon[93869]: pgmap v20981: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 626 KiB/s rd, 531 KiB/s wr, 8 op/s 2023-03-10T01:37:55.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:37:54 smithi181 ceph-mon[94189]: pgmap v20981: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 626 KiB/s rd, 531 KiB/s wr, 8 op/s 2023-03-10T01:37:56.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:37:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38386.537106 secs 2023-03-10T01:37:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:37:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38386.537106 secs 2023-03-10T01:37:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:37:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38386.537106 secs 2023-03-10T01:37:57.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:37:56 smithi038 ceph-mon[92422]: pgmap v20982: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 497 KiB/s rd, 402 KiB/s wr, 7 op/s 2023-03-10T01:37:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:37:56 smithi122 ceph-mon[93869]: pgmap v20982: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 497 KiB/s rd, 402 KiB/s wr, 7 op/s 2023-03-10T01:37:57.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:37:56 smithi181 ceph-mon[94189]: pgmap v20982: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 497 KiB/s rd, 402 KiB/s wr, 7 op/s 2023-03-10T01:37:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38850.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:37:59.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:37:58 smithi038 ceph-mon[92422]: pgmap v20983: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 630 KiB/s rd, 532 KiB/s wr, 11 op/s 2023-03-10T01:37:59.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:37:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38388.995987 secs 2023-03-10T01:37:59.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:37:58 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:37:59.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:37:58 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:37:59.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:37:58 smithi122 ceph-mon[93869]: pgmap v20983: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 630 KiB/s rd, 532 KiB/s wr, 11 op/s 2023-03-10T01:37:59.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:37:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38388.995987 secs 2023-03-10T01:37:59.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:37:58 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:37:59.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:37:58 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:37:59.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:37:58 smithi181 ceph-mon[94189]: pgmap v20983: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 630 KiB/s rd, 532 KiB/s wr, 11 op/s 2023-03-10T01:37:59.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:37:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38388.995987 secs 2023-03-10T01:37:59.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:37:58 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:37:59.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:37:58 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:38:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:38:01 smithi122 ceph-mon[93869]: pgmap v20984: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 332 KiB/s wr, 9 op/s 2023-03-10T01:38:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:38:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38391.537162 secs 2023-03-10T01:38:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:38:01 smithi181 ceph-mon[94189]: pgmap v20984: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 332 KiB/s wr, 9 op/s 2023-03-10T01:38:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:38:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38391.537162 secs 2023-03-10T01:38:01.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:38:01 smithi038 ceph-mon[92422]: pgmap v20984: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 347 KiB/s rd, 332 KiB/s wr, 9 op/s 2023-03-10T01:38:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:38:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38391.537162 secs 2023-03-10T01:38:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:38:03 smithi122 ceph-mon[93869]: pgmap v20985: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1012 KiB/s rd, 916 KiB/s wr, 15 op/s 2023-03-10T01:38:03.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:38:03 smithi181 ceph-mon[94189]: pgmap v20985: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1012 KiB/s rd, 916 KiB/s wr, 15 op/s 2023-03-10T01:38:03.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:38:03 smithi038 ceph-mon[92422]: pgmap v20985: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1012 KiB/s rd, 916 KiB/s wr, 15 op/s 2023-03-10T01:38:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38855.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:38:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:38:04 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38393.996047 secs 2023-03-10T01:38:04.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:38:04 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38393.996047 secs 2023-03-10T01:38:04.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:38:04 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38393.996047 secs 2023-03-10T01:38:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:38:05 smithi122 ceph-mon[93869]: pgmap v20986: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 816 KiB/s rd, 725 KiB/s wr, 23 op/s 2023-03-10T01:38:05.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:38:05 smithi181 ceph-mon[94189]: pgmap v20986: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 816 KiB/s rd, 725 KiB/s wr, 23 op/s 2023-03-10T01:38:05.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:38:05 smithi038 ceph-mon[92422]: pgmap v20986: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 816 KiB/s rd, 725 KiB/s wr, 23 op/s 2023-03-10T01:38:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:38:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38396.537221 secs 2023-03-10T01:38:06.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:38:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38396.537221 secs 2023-03-10T01:38:06.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:38:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38396.537221 secs 2023-03-10T01:38:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:38:07 smithi122 ceph-mon[93869]: pgmap v20987: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 908 KiB/s rd, 817 KiB/s wr, 23 op/s 2023-03-10T01:38:07.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:38:07 smithi181 ceph-mon[94189]: pgmap v20987: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 908 KiB/s rd, 817 KiB/s wr, 23 op/s 2023-03-10T01:38:07.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:38:07 smithi038 ceph-mon[92422]: pgmap v20987: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 908 KiB/s rd, 817 KiB/s wr, 23 op/s 2023-03-10T01:38:08.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38860.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:38:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:38:09 smithi122 ceph-mon[93869]: pgmap v20988: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 909 KiB/s rd, 818 KiB/s wr, 26 op/s 2023-03-10T01:38:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:38:09 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38398.996142 secs 2023-03-10T01:38:09.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:38:09 smithi181 ceph-mon[94189]: pgmap v20988: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 909 KiB/s rd, 818 KiB/s wr, 26 op/s 2023-03-10T01:38:09.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:38:09 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38398.996142 secs 2023-03-10T01:38:09.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:38:09 smithi038 ceph-mon[92422]: pgmap v20988: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 909 KiB/s rd, 818 KiB/s wr, 26 op/s 2023-03-10T01:38:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:38:09 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38398.996142 secs 2023-03-10T01:38:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:38:11 smithi122 ceph-mon[93869]: pgmap v20989: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 794 KiB/s wr, 22 op/s 2023-03-10T01:38:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:38:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38401.537288 secs 2023-03-10T01:38:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:38:11 smithi181 ceph-mon[94189]: pgmap v20989: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 794 KiB/s wr, 22 op/s 2023-03-10T01:38:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:38:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38401.537288 secs 2023-03-10T01:38:11.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:38:11 smithi038 ceph-mon[92422]: pgmap v20989: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 794 KiB/s wr, 22 op/s 2023-03-10T01:38:11.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:38:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38401.537288 secs 2023-03-10T01:38:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:38:13 smithi122 ceph-mon[93869]: pgmap v20990: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 974 KiB/s rd, 845 KiB/s wr, 25 op/s 2023-03-10T01:38:13.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:38:13 smithi181 ceph-mon[94189]: pgmap v20990: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 974 KiB/s rd, 845 KiB/s wr, 25 op/s 2023-03-10T01:38:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:38:13 smithi038 ceph-mon[92422]: pgmap v20990: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 974 KiB/s rd, 845 KiB/s wr, 25 op/s 2023-03-10T01:38:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38865.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:38:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:38:14 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38403.996204 secs 2023-03-10T01:38:14.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:38:14 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38403.996204 secs 2023-03-10T01:38:14.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:38:14 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38403.996204 secs 2023-03-10T01:38:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:38:15 smithi122 ceph-mon[93869]: pgmap v20991: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 260 KiB/s wr, 19 op/s 2023-03-10T01:38:15.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:38:15 smithi181 ceph-mon[94189]: pgmap v20991: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 260 KiB/s wr, 19 op/s 2023-03-10T01:38:15.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:38:15 smithi038 ceph-mon[92422]: pgmap v20991: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 260 KiB/s wr, 19 op/s 2023-03-10T01:38:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:38:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38406.537349 secs 2023-03-10T01:38:16.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:38:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38406.537349 secs 2023-03-10T01:38:16.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:38:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38406.537349 secs 2023-03-10T01:38:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:38:17 smithi122 ceph-mon[93869]: pgmap v20992: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 484 KiB/s rd, 431 KiB/s wr, 9 op/s 2023-03-10T01:38:17.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:38:17 smithi181 ceph-mon[94189]: pgmap v20992: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 484 KiB/s rd, 431 KiB/s wr, 9 op/s 2023-03-10T01:38:17.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:38:17 smithi038 ceph-mon[92422]: pgmap v20992: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 484 KiB/s rd, 431 KiB/s wr, 9 op/s 2023-03-10T01:38:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:38:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38408.996263 secs 2023-03-10T01:38:18.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:38:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38408.996263 secs 2023-03-10T01:38:18.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:38:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38408.996263 secs 2023-03-10T01:38:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38870.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:38:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:38:19 smithi122 ceph-mon[93869]: pgmap v20993: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 339 KiB/s wr, 10 op/s 2023-03-10T01:38:19.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:38:19 smithi181 ceph-mon[94189]: pgmap v20993: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 339 KiB/s wr, 10 op/s 2023-03-10T01:38:19.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:38:19 smithi038 ceph-mon[92422]: pgmap v20993: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 339 KiB/s wr, 10 op/s 2023-03-10T01:38:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:38:21 smithi122 ceph-mon[93869]: pgmap v20994: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 404 KiB/s wr, 9 op/s 2023-03-10T01:38:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:38:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38411.537407 secs 2023-03-10T01:38:21.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:38:21 smithi181 ceph-mon[94189]: pgmap v20994: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 404 KiB/s wr, 9 op/s 2023-03-10T01:38:21.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:38:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38411.537407 secs 2023-03-10T01:38:21.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:38:21 smithi038 ceph-mon[92422]: pgmap v20994: 129 pgs: 129 active+clean; 9.9 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 447 KiB/s rd, 404 KiB/s wr, 9 op/s 2023-03-10T01:38:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:38:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38411.537407 secs 2023-03-10T01:38:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:38:23 smithi122 ceph-mon[93869]: pgmap v20995: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 683 KiB/s wr, 13 op/s 2023-03-10T01:38:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:38:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38413.996348 secs 2023-03-10T01:38:23.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:38:23 smithi181 ceph-mon[94189]: pgmap v20995: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 683 KiB/s wr, 13 op/s 2023-03-10T01:38:23.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:38:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38413.996348 secs 2023-03-10T01:38:23.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:38:23 smithi038 ceph-mon[92422]: pgmap v20995: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 683 KiB/s wr, 13 op/s 2023-03-10T01:38:23.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:38:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38413.996348 secs 2023-03-10T01:38:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38875.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:38:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:38:25 smithi122 ceph-mon[93869]: pgmap v20996: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 721 KiB/s rd, 741 KiB/s wr, 10 op/s 2023-03-10T01:38:25.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:38:25 smithi181 ceph-mon[94189]: pgmap v20996: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 721 KiB/s rd, 741 KiB/s wr, 10 op/s 2023-03-10T01:38:25.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:38:25 smithi038 ceph-mon[92422]: pgmap v20996: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 721 KiB/s rd, 741 KiB/s wr, 10 op/s 2023-03-10T01:38:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:38:26 smithi122 ceph-mon[93869]: osdmap e1527: 12 total, 12 up, 12 in 2023-03-10T01:38:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:38:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38416.537468 secs 2023-03-10T01:38:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:38:26 smithi181 ceph-mon[94189]: osdmap e1527: 12 total, 12 up, 12 in 2023-03-10T01:38:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:38:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38416.537468 secs 2023-03-10T01:38:26.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:38:26 smithi038 ceph-mon[92422]: osdmap e1527: 12 total, 12 up, 12 in 2023-03-10T01:38:26.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:38:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38416.537468 secs 2023-03-10T01:38:27.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:38:27 smithi122 ceph-mon[93869]: pgmap v20998: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 671 KiB/s wr, 12 op/s 2023-03-10T01:38:27.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:38:27 smithi181 ceph-mon[94189]: pgmap v20998: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 671 KiB/s wr, 12 op/s 2023-03-10T01:38:27.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:38:27 smithi038 ceph-mon[92422]: pgmap v20998: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 671 KiB/s wr, 12 op/s 2023-03-10T01:38:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:38:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38418.996433 secs 2023-03-10T01:38:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:38:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:38:28.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:38:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38418.996433 secs 2023-03-10T01:38:28.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:38:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:38:28.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:38:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38418.996433 secs 2023-03-10T01:38:28.565 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:38:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:38:28.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38880.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:38:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:38:29 smithi122 ceph-mon[93869]: pgmap v20999: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 653 KiB/s rd, 677 KiB/s wr, 13 op/s 2023-03-10T01:38:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:38:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:38:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:38:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:38:29.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:38:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:38:29.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:38:29 smithi181 ceph-mon[94189]: pgmap v20999: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 653 KiB/s rd, 677 KiB/s wr, 13 op/s 2023-03-10T01:38:29.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:38:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:38:29.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:38:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:38:29.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:38:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:38:29.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:38:29 smithi038 ceph-mon[92422]: pgmap v20999: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 653 KiB/s rd, 677 KiB/s wr, 13 op/s 2023-03-10T01:38:29.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:38:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:38:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:38:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:38:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:38:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:38:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:38:31 smithi122 ceph-mon[93869]: pgmap v21000: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 777 KiB/s rd, 788 KiB/s wr, 12 op/s 2023-03-10T01:38:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:38:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38421.537547 secs 2023-03-10T01:38:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:38:31 smithi181 ceph-mon[94189]: pgmap v21000: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 777 KiB/s rd, 788 KiB/s wr, 12 op/s 2023-03-10T01:38:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:38:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38421.537547 secs 2023-03-10T01:38:31.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:38:31 smithi038 ceph-mon[92422]: pgmap v21000: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 777 KiB/s rd, 788 KiB/s wr, 12 op/s 2023-03-10T01:38:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:38:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38421.537547 secs 2023-03-10T01:38:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:38:33 smithi122 ceph-mon[93869]: pgmap v21001: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 362 KiB/s rd, 361 KiB/s wr, 9 op/s 2023-03-10T01:38:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:38:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38423.996516 secs 2023-03-10T01:38:33.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:38:33 smithi181 ceph-mon[94189]: pgmap v21001: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 362 KiB/s rd, 361 KiB/s wr, 9 op/s 2023-03-10T01:38:33.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:38:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38423.996516 secs 2023-03-10T01:38:33.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:38:33 smithi038 ceph-mon[92422]: pgmap v21001: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 362 KiB/s rd, 361 KiB/s wr, 9 op/s 2023-03-10T01:38:33.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:38:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38423.996516 secs 2023-03-10T01:38:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38885.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:38:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:38:35 smithi122 ceph-mon[93869]: pgmap v21002: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 439 KiB/s wr, 10 op/s 2023-03-10T01:38:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:38:35 smithi122 ceph-mon[93869]: osdmap e1528: 12 total, 12 up, 12 in 2023-03-10T01:38:35.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:38:35 smithi181 ceph-mon[94189]: pgmap v21002: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 439 KiB/s wr, 10 op/s 2023-03-10T01:38:35.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:38:35 smithi181 ceph-mon[94189]: osdmap e1528: 12 total, 12 up, 12 in 2023-03-10T01:38:35.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:38:35 smithi038 ceph-mon[92422]: pgmap v21002: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 439 KiB/s wr, 10 op/s 2023-03-10T01:38:35.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:38:35 smithi038 ceph-mon[92422]: osdmap e1528: 12 total, 12 up, 12 in 2023-03-10T01:38:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:38:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38426.537603 secs 2023-03-10T01:38:36.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:38:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38426.537603 secs 2023-03-10T01:38:36.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:38:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38426.537603 secs 2023-03-10T01:38:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:38:37 smithi122 ceph-mon[93869]: pgmap v21004: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 439 KiB/s wr, 8 op/s 2023-03-10T01:38:37.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:38:37 smithi181 ceph-mon[94189]: pgmap v21004: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 439 KiB/s wr, 8 op/s 2023-03-10T01:38:37.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:38:37 smithi038 ceph-mon[92422]: pgmap v21004: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 439 KiB/s wr, 8 op/s 2023-03-10T01:38:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:38:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38428.996575 secs 2023-03-10T01:38:38.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:38:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38428.996575 secs 2023-03-10T01:38:38.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:38:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38428.996575 secs 2023-03-10T01:38:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38890.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:38:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:38:39 smithi122 ceph-mon[93869]: pgmap v21005: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 472 KiB/s wr, 8 op/s 2023-03-10T01:38:39.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:38:39 smithi181 ceph-mon[94189]: pgmap v21005: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 472 KiB/s wr, 8 op/s 2023-03-10T01:38:39.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:38:39 smithi038 ceph-mon[92422]: pgmap v21005: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 472 KiB/s wr, 8 op/s 2023-03-10T01:38:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:38:41 smithi122 ceph-mon[93869]: pgmap v21006: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 283 KiB/s wr, 9 op/s 2023-03-10T01:38:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:38:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38431.537662 secs 2023-03-10T01:38:41.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:38:41 smithi181 ceph-mon[94189]: pgmap v21006: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 283 KiB/s wr, 9 op/s 2023-03-10T01:38:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:38:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38431.537662 secs 2023-03-10T01:38:41.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:38:41 smithi038 ceph-mon[92422]: pgmap v21006: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 283 KiB/s wr, 9 op/s 2023-03-10T01:38:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:38:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38431.537662 secs 2023-03-10T01:38:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:38:43 smithi122 ceph-mon[93869]: pgmap v21007: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 340 KiB/s wr, 9 op/s 2023-03-10T01:38:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:38:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38433.996664 secs 2023-03-10T01:38:43.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:38:43 smithi181 ceph-mon[94189]: pgmap v21007: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 340 KiB/s wr, 9 op/s 2023-03-10T01:38:43.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:38:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38433.996664 secs 2023-03-10T01:38:43.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:38:43 smithi038 ceph-mon[92422]: pgmap v21007: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 340 KiB/s wr, 9 op/s 2023-03-10T01:38:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:38:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38433.996664 secs 2023-03-10T01:38:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38895.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:38:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:38:45 smithi122 ceph-mon[93869]: pgmap v21008: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 422 KiB/s wr, 9 op/s 2023-03-10T01:38:45.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:38:45 smithi181 ceph-mon[94189]: pgmap v21008: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 422 KiB/s wr, 9 op/s 2023-03-10T01:38:45.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:38:45 smithi038 ceph-mon[92422]: pgmap v21008: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 422 KiB/s wr, 9 op/s 2023-03-10T01:38:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:38:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38436.537721 secs 2023-03-10T01:38:46.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:38:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38436.537721 secs 2023-03-10T01:38:46.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:38:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38436.537721 secs 2023-03-10T01:38:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:38:47 smithi122 ceph-mon[93869]: pgmap v21009: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 601 KiB/s rd, 401 KiB/s wr, 9 op/s 2023-03-10T01:38:47.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:38:47 smithi181 ceph-mon[94189]: pgmap v21009: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 601 KiB/s rd, 401 KiB/s wr, 9 op/s 2023-03-10T01:38:47.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:38:47 smithi038 ceph-mon[92422]: pgmap v21009: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 601 KiB/s rd, 401 KiB/s wr, 9 op/s 2023-03-10T01:38:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:38:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38438.996757 secs 2023-03-10T01:38:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:38:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38438.996757 secs 2023-03-10T01:38:48.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:38:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38438.996757 secs 2023-03-10T01:38:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38900.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:38:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:38:49 smithi122 ceph-mon[93869]: pgmap v21010: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 794 KiB/s rd, 618 KiB/s wr, 11 op/s 2023-03-10T01:38:49.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:38:49 smithi181 ceph-mon[94189]: pgmap v21010: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 794 KiB/s rd, 618 KiB/s wr, 11 op/s 2023-03-10T01:38:49.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:38:49 smithi038 ceph-mon[92422]: pgmap v21010: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 794 KiB/s rd, 618 KiB/s wr, 11 op/s 2023-03-10T01:38:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:38:51 smithi122 ceph-mon[93869]: pgmap v21011: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 760 KiB/s rd, 585 KiB/s wr, 9 op/s 2023-03-10T01:38:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:38:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38441.537805 secs 2023-03-10T01:38:51.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:38:51 smithi181 ceph-mon[94189]: pgmap v21011: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 760 KiB/s rd, 585 KiB/s wr, 9 op/s 2023-03-10T01:38:51.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:38:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38441.537805 secs 2023-03-10T01:38:51.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:38:51 smithi038 ceph-mon[92422]: pgmap v21011: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 760 KiB/s rd, 585 KiB/s wr, 9 op/s 2023-03-10T01:38:51.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:38:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38441.537805 secs 2023-03-10T01:38:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:38:53 smithi122 ceph-mon[93869]: pgmap v21012: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 864 KiB/s rd, 618 KiB/s wr, 11 op/s 2023-03-10T01:38:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:38:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38443.996890 secs 2023-03-10T01:38:53.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:38:53 smithi181 ceph-mon[94189]: pgmap v21012: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 864 KiB/s rd, 618 KiB/s wr, 11 op/s 2023-03-10T01:38:53.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:38:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38443.996890 secs 2023-03-10T01:38:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:38:53 smithi038 ceph-mon[92422]: pgmap v21012: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 864 KiB/s rd, 618 KiB/s wr, 11 op/s 2023-03-10T01:38:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:38:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38443.996890 secs 2023-03-10T01:38:53.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38905.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:38:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:38:55 smithi122 ceph-mon[93869]: pgmap v21013: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 883 KiB/s wr, 10 op/s 2023-03-10T01:38:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:38:55 smithi181 ceph-mon[94189]: pgmap v21013: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 883 KiB/s wr, 10 op/s 2023-03-10T01:38:55.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:38:55 smithi038 ceph-mon[92422]: pgmap v21013: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 883 KiB/s wr, 10 op/s 2023-03-10T01:38:56.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:38:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38446.537864 secs 2023-03-10T01:38:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:38:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38446.537864 secs 2023-03-10T01:38:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:38:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38446.537864 secs 2023-03-10T01:38:57.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:38:57 smithi038 ceph-mon[92422]: pgmap v21014: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 672 KiB/s wr, 9 op/s 2023-03-10T01:38:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:38:57 smithi122 ceph-mon[93869]: pgmap v21014: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 672 KiB/s wr, 9 op/s 2023-03-10T01:38:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:38:57 smithi181 ceph-mon[94189]: pgmap v21014: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 672 KiB/s wr, 9 op/s 2023-03-10T01:38:58.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:38:58 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:38:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:38:58 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:38:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:38:58 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:38:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38910.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:38:59.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:38:59 smithi038 ceph-mon[92422]: pgmap v21015: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 842 KiB/s wr, 13 op/s 2023-03-10T01:38:59.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:38:59 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:38:59.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:38:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:38:59.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:38:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:38:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:38:59 smithi122 ceph-mon[93869]: pgmap v21015: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 842 KiB/s wr, 13 op/s 2023-03-10T01:38:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:38:59 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:38:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:38:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:38:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:38:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:38:59.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:38:59 smithi181 ceph-mon[94189]: pgmap v21015: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 842 KiB/s wr, 13 op/s 2023-03-10T01:38:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:38:59 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:38:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:38:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:38:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:38:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:39:00.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:00 smithi038 ceph-mon[92422]: Health check failed: 1 MDSs behind on trimming (MDS_TRIM) 2023-03-10T01:39:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:39:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:39:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:00 smithi122 ceph-mon[93869]: Health check failed: 1 MDSs behind on trimming (MDS_TRIM) 2023-03-10T01:39:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:39:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:39:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:00 smithi181 ceph-mon[94189]: Health check failed: 1 MDSs behind on trimming (MDS_TRIM) 2023-03-10T01:39:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:39:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:39:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:01 smithi122 ceph-mon[93869]: pgmap v21016: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 576 KiB/s wr, 11 op/s 2023-03-10T01:39:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38451.537956 secs 2023-03-10T01:39:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:39:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:39:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:01 smithi181 ceph-mon[94189]: pgmap v21016: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 576 KiB/s wr, 11 op/s 2023-03-10T01:39:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38451.537956 secs 2023-03-10T01:39:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:39:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:39:01.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:01 smithi038 ceph-mon[92422]: pgmap v21016: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 576 KiB/s wr, 11 op/s 2023-03-10T01:39:01.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38451.537956 secs 2023-03-10T01:39:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:39:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:39:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:03 smithi122 ceph-mon[93869]: pgmap v21017: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 962 KiB/s wr, 16 op/s 2023-03-10T01:39:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:03 smithi181 ceph-mon[94189]: pgmap v21017: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 962 KiB/s wr, 16 op/s 2023-03-10T01:39:03.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:03 smithi038 ceph-mon[92422]: pgmap v21017: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 962 KiB/s wr, 16 op/s 2023-03-10T01:39:03.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38915.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:39:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:05 smithi122 ceph-mon[93869]: pgmap v21018: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 24 op/s 2023-03-10T01:39:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:05 smithi122 ceph-mon[93869]: Health check update: 2 MDSs behind on trimming (MDS_TRIM) 2023-03-10T01:39:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:05 smithi181 ceph-mon[94189]: pgmap v21018: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 24 op/s 2023-03-10T01:39:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:05 smithi181 ceph-mon[94189]: Health check update: 2 MDSs behind on trimming (MDS_TRIM) 2023-03-10T01:39:05.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:05 smithi038 ceph-mon[92422]: pgmap v21018: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 24 op/s 2023-03-10T01:39:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:05 smithi038 ceph-mon[92422]: Health check update: 2 MDSs behind on trimming (MDS_TRIM) 2023-03-10T01:39:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38456.538041 secs 2023-03-10T01:39:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38456.538041 secs 2023-03-10T01:39:06.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38456.538041 secs 2023-03-10T01:39:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:07 smithi122 ceph-mon[93869]: pgmap v21019: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 842 KiB/s rd, 771 KiB/s wr, 24 op/s 2023-03-10T01:39:07.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:07 smithi181 ceph-mon[94189]: pgmap v21019: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 842 KiB/s rd, 771 KiB/s wr, 24 op/s 2023-03-10T01:39:07.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:07 smithi038 ceph-mon[92422]: pgmap v21019: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 842 KiB/s rd, 771 KiB/s wr, 24 op/s 2023-03-10T01:39:08.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38920.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:39:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:09 smithi122 ceph-mon[93869]: pgmap v21020: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 967 KiB/s rd, 741 KiB/s wr, 27 op/s 2023-03-10T01:39:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:39:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:39:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:09 smithi181 ceph-mon[94189]: pgmap v21020: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 967 KiB/s rd, 741 KiB/s wr, 27 op/s 2023-03-10T01:39:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:39:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:39:09.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:09 smithi038 ceph-mon[92422]: pgmap v21020: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 967 KiB/s rd, 741 KiB/s wr, 27 op/s 2023-03-10T01:39:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:39:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:39:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:11 smithi122 ceph-mon[93869]: pgmap v21021: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 944 KiB/s rd, 720 KiB/s wr, 24 op/s 2023-03-10T01:39:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38461.538127 secs 2023-03-10T01:39:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:11 smithi181 ceph-mon[94189]: pgmap v21021: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 944 KiB/s rd, 720 KiB/s wr, 24 op/s 2023-03-10T01:39:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38461.538127 secs 2023-03-10T01:39:11.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:11 smithi038 ceph-mon[92422]: pgmap v21021: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 944 KiB/s rd, 720 KiB/s wr, 24 op/s 2023-03-10T01:39:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38461.538127 secs 2023-03-10T01:39:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:39:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:39:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:39:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:39:12.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:39:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:39:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:13 smithi122 ceph-mon[93869]: pgmap v21022: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 764 KiB/s rd, 720 KiB/s wr, 25 op/s 2023-03-10T01:39:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:13 smithi181 ceph-mon[94189]: pgmap v21022: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 764 KiB/s rd, 720 KiB/s wr, 25 op/s 2023-03-10T01:39:13.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:13 smithi038 ceph-mon[92422]: pgmap v21022: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 764 KiB/s rd, 720 KiB/s wr, 25 op/s 2023-03-10T01:39:13.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38925.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:39:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:15 smithi122 ceph-mon[93869]: pgmap v21023: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 335 KiB/s wr, 20 op/s 2023-03-10T01:39:15.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:15 smithi181 ceph-mon[94189]: pgmap v21023: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 335 KiB/s wr, 20 op/s 2023-03-10T01:39:15.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:15 smithi038 ceph-mon[92422]: pgmap v21023: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 487 KiB/s rd, 335 KiB/s wr, 20 op/s 2023-03-10T01:39:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38466.538189 secs 2023-03-10T01:39:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38466.538189 secs 2023-03-10T01:39:16.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38466.538189 secs 2023-03-10T01:39:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:17 smithi122 ceph-mon[93869]: pgmap v21024: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 490 KiB/s rd, 333 KiB/s wr, 9 op/s 2023-03-10T01:39:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:17 smithi181 ceph-mon[94189]: pgmap v21024: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 490 KiB/s rd, 333 KiB/s wr, 9 op/s 2023-03-10T01:39:17.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:17 smithi038 ceph-mon[92422]: pgmap v21024: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 490 KiB/s rd, 333 KiB/s wr, 9 op/s 2023-03-10T01:39:18.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38930.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:39:19.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:19 smithi122 ceph-mon[93869]: pgmap v21025: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 332 KiB/s wr, 11 op/s 2023-03-10T01:39:19.818 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:19 smithi038 ceph-mon[92422]: pgmap v21025: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 332 KiB/s wr, 11 op/s 2023-03-10T01:39:19.818 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:19 smithi181 ceph-mon[94189]: pgmap v21025: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 332 KiB/s wr, 11 op/s 2023-03-10T01:39:20.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:39:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:39:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:39:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:39:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:39:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:39:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:21 smithi122 ceph-mon[93869]: pgmap v21026: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 353 KiB/s wr, 10 op/s 2023-03-10T01:39:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38471.538270 secs 2023-03-10T01:39:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:39:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:39:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:21 smithi181 ceph-mon[94189]: pgmap v21026: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 353 KiB/s wr, 10 op/s 2023-03-10T01:39:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38471.538270 secs 2023-03-10T01:39:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:39:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:39:21.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:21 smithi038 ceph-mon[92422]: pgmap v21026: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 353 KiB/s wr, 10 op/s 2023-03-10T01:39:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38471.538270 secs 2023-03-10T01:39:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:39:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:39:22.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:22 smithi122 ceph-mon[93869]: pgmap v21027: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 586 KiB/s wr, 14 op/s 2023-03-10T01:39:22.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:22 smithi181 ceph-mon[94189]: pgmap v21027: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 586 KiB/s wr, 14 op/s 2023-03-10T01:39:22.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:22 smithi038 ceph-mon[92422]: pgmap v21027: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 586 KiB/s wr, 14 op/s 2023-03-10T01:39:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38935.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:39:24.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:24 smithi122 ceph-mon[93869]: pgmap v21028: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 567 KiB/s rd, 586 KiB/s wr, 11 op/s 2023-03-10T01:39:24.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:24 smithi181 ceph-mon[94189]: pgmap v21028: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 567 KiB/s rd, 586 KiB/s wr, 11 op/s 2023-03-10T01:39:24.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:24 smithi038 ceph-mon[92422]: pgmap v21028: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 567 KiB/s rd, 586 KiB/s wr, 11 op/s 2023-03-10T01:39:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:25 smithi122 ceph-mon[93869]: osdmap e1529: 12 total, 12 up, 12 in 2023-03-10T01:39:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:25 smithi181 ceph-mon[94189]: osdmap e1529: 12 total, 12 up, 12 in 2023-03-10T01:39:25.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:25 smithi038 ceph-mon[92422]: osdmap e1529: 12 total, 12 up, 12 in 2023-03-10T01:39:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38476.538354 secs 2023-03-10T01:39:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:26 smithi122 ceph-mon[93869]: pgmap v21030: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 485 KiB/s wr, 13 op/s 2023-03-10T01:39:27.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38476.538354 secs 2023-03-10T01:39:27.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:26 smithi181 ceph-mon[94189]: pgmap v21030: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 485 KiB/s wr, 13 op/s 2023-03-10T01:39:27.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38476.538354 secs 2023-03-10T01:39:27.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:26 smithi038 ceph-mon[92422]: pgmap v21030: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 485 KiB/s wr, 13 op/s 2023-03-10T01:39:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38940.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:39:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:28 smithi122 ceph-mon[93869]: pgmap v21031: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 652 KiB/s rd, 675 KiB/s wr, 13 op/s 2023-03-10T01:39:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:39:29.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:28 smithi181 ceph-mon[94189]: pgmap v21031: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 652 KiB/s rd, 675 KiB/s wr, 13 op/s 2023-03-10T01:39:29.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:39:29.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:28 smithi038 ceph-mon[92422]: pgmap v21031: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 652 KiB/s rd, 675 KiB/s wr, 13 op/s 2023-03-10T01:39:29.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:39:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:39:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:39:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:39:30.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:39:30.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:39:30.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:39:30.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:39:30.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:39:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:39:31.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:31 smithi038 ceph-mon[92422]: pgmap v21032: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 639 KiB/s rd, 651 KiB/s wr, 11 op/s 2023-03-10T01:39:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38481.538410 secs 2023-03-10T01:39:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:31 smithi122 ceph-mon[93869]: pgmap v21032: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 639 KiB/s rd, 651 KiB/s wr, 11 op/s 2023-03-10T01:39:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38481.538410 secs 2023-03-10T01:39:31.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:31 smithi181 ceph-mon[94189]: pgmap v21032: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 639 KiB/s rd, 651 KiB/s wr, 11 op/s 2023-03-10T01:39:31.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38481.538410 secs 2023-03-10T01:39:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:33 smithi122 ceph-mon[93869]: pgmap v21033: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 230 KiB/s wr, 9 op/s 2023-03-10T01:39:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:33 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:39:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:33 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:39:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:33 smithi181 ceph-mon[94189]: pgmap v21033: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 230 KiB/s wr, 9 op/s 2023-03-10T01:39:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:33 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:39:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:33 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:39:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:33 smithi038 ceph-mon[92422]: pgmap v21033: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 230 KiB/s wr, 9 op/s 2023-03-10T01:39:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:33 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:39:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:33 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:39:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38945.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:39:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:35 smithi122 ceph-mon[93869]: pgmap v21034: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 230 KiB/s wr, 10 op/s 2023-03-10T01:39:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:35 smithi122 ceph-mon[93869]: osdmap e1530: 12 total, 12 up, 12 in 2023-03-10T01:39:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:35 smithi181 ceph-mon[94189]: pgmap v21034: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 230 KiB/s wr, 10 op/s 2023-03-10T01:39:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:35 smithi181 ceph-mon[94189]: osdmap e1530: 12 total, 12 up, 12 in 2023-03-10T01:39:35.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:35 smithi038 ceph-mon[92422]: pgmap v21034: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 230 KiB/s wr, 10 op/s 2023-03-10T01:39:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:35 smithi038 ceph-mon[92422]: osdmap e1530: 12 total, 12 up, 12 in 2023-03-10T01:39:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38486.538469 secs 2023-03-10T01:39:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:39:36.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:39:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38486.538469 secs 2023-03-10T01:39:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:39:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:39:36.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38486.538469 secs 2023-03-10T01:39:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:39:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:39:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:37 smithi122 ceph-mon[93869]: pgmap v21036: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 641 KiB/s wr, 10 op/s 2023-03-10T01:39:37.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:37 smithi181 ceph-mon[94189]: pgmap v21036: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 641 KiB/s wr, 10 op/s 2023-03-10T01:39:37.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:37 smithi038 ceph-mon[92422]: pgmap v21036: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 641 KiB/s wr, 10 op/s 2023-03-10T01:39:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38488.997624 secs 2023-03-10T01:39:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38488.997624 secs 2023-03-10T01:39:38.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38488.997624 secs 2023-03-10T01:39:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38950.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:39:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:39 smithi122 ceph-mon[93869]: pgmap v21037: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 675 KiB/s wr, 10 op/s 2023-03-10T01:39:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:39 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:39:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:39 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:39:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:39 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:39:39.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:39 smithi181 ceph-mon[94189]: pgmap v21037: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 675 KiB/s wr, 10 op/s 2023-03-10T01:39:39.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:39 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:39:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:39 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:39:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:39 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:39:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:39 smithi038 ceph-mon[92422]: pgmap v21037: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 675 KiB/s wr, 10 op/s 2023-03-10T01:39:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:39 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:39:39.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:39 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:39:39.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:39 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:39:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:39:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:39:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:39:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:39:40.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:39:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:39:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:41 smithi122 ceph-mon[93869]: pgmap v21038: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 838 KiB/s wr, 11 op/s 2023-03-10T01:39:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38491.538525 secs 2023-03-10T01:39:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:39:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:39:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:41 smithi181 ceph-mon[94189]: pgmap v21038: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 838 KiB/s wr, 11 op/s 2023-03-10T01:39:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38491.538525 secs 2023-03-10T01:39:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:39:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:39:41.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:41 smithi038 ceph-mon[92422]: pgmap v21038: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 838 KiB/s wr, 11 op/s 2023-03-10T01:39:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38491.538525 secs 2023-03-10T01:39:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:39:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:39:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:43 smithi122 ceph-mon[93869]: pgmap v21039: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 839 KiB/s rd, 843 KiB/s wr, 11 op/s 2023-03-10T01:39:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38493.997731 secs 2023-03-10T01:39:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:43 smithi181 ceph-mon[94189]: pgmap v21039: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 839 KiB/s rd, 843 KiB/s wr, 11 op/s 2023-03-10T01:39:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38493.997731 secs 2023-03-10T01:39:43.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:43 smithi038 ceph-mon[92422]: pgmap v21039: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 839 KiB/s rd, 843 KiB/s wr, 11 op/s 2023-03-10T01:39:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38493.997731 secs 2023-03-10T01:39:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38955.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:39:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:45 smithi122 ceph-mon[93869]: pgmap v21040: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 839 KiB/s rd, 1022 KiB/s wr, 11 op/s 2023-03-10T01:39:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:45 smithi181 ceph-mon[94189]: pgmap v21040: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 839 KiB/s rd, 1022 KiB/s wr, 11 op/s 2023-03-10T01:39:45.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:45 smithi038 ceph-mon[92422]: pgmap v21040: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 839 KiB/s rd, 1022 KiB/s wr, 11 op/s 2023-03-10T01:39:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38496.538608 secs 2023-03-10T01:39:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38496.538608 secs 2023-03-10T01:39:46.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38496.538608 secs 2023-03-10T01:39:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:47 smithi122 ceph-mon[93869]: pgmap v21041: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.2 MiB/s wr, 12 op/s 2023-03-10T01:39:47.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:47 smithi181 ceph-mon[94189]: pgmap v21041: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.2 MiB/s wr, 12 op/s 2023-03-10T01:39:47.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:47 smithi038 ceph-mon[92422]: pgmap v21041: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.2 MiB/s wr, 12 op/s 2023-03-10T01:39:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:39:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:39:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:39:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:39:48.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:39:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:39:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38960.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:39:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:49 smithi122 ceph-mon[93869]: pgmap v21042: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 879 KiB/s rd, 716 KiB/s wr, 12 op/s 2023-03-10T01:39:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:39:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:39:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:49 smithi181 ceph-mon[94189]: pgmap v21042: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 879 KiB/s rd, 716 KiB/s wr, 12 op/s 2023-03-10T01:39:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:39:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:39:49.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:49 smithi038 ceph-mon[92422]: pgmap v21042: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 879 KiB/s rd, 716 KiB/s wr, 12 op/s 2023-03-10T01:39:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:39:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:39:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:50 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:39:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:50 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:39:50.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:50 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:39:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:51 smithi122 ceph-mon[93869]: pgmap v21043: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 854 KiB/s rd, 722 KiB/s wr, 10 op/s 2023-03-10T01:39:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38501.538664 secs 2023-03-10T01:39:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:51 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:39:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:39:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:39:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:51 smithi181 ceph-mon[94189]: pgmap v21043: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 854 KiB/s rd, 722 KiB/s wr, 10 op/s 2023-03-10T01:39:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38501.538664 secs 2023-03-10T01:39:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:51 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:39:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:39:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:39:51.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:51 smithi038 ceph-mon[92422]: pgmap v21043: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 854 KiB/s rd, 722 KiB/s wr, 10 op/s 2023-03-10T01:39:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38501.538664 secs 2023-03-10T01:39:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:51 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:39:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:39:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:39:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:39:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:39:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:52 smithi122 ceph-mon[93869]: pgmap v21044: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 15 op/s 2023-03-10T01:39:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:39:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:39:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:52 smithi181 ceph-mon[94189]: pgmap v21044: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 15 op/s 2023-03-10T01:39:52.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:39:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:39:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:52 smithi038 ceph-mon[92422]: pgmap v21044: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 15 op/s 2023-03-10T01:39:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:39:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:39:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:53 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 38503.997904 secs 2023-03-10T01:39:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:39:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:39:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:53 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 38503.997904 secs 2023-03-10T01:39:53.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:39:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:39:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:53 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 38503.997904 secs 2023-03-10T01:39:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38965.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:39:54.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:54 smithi122 ceph-mon[93869]: pgmap v21045: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 13 op/s 2023-03-10T01:39:54.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:54 smithi181 ceph-mon[94189]: pgmap v21045: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 13 op/s 2023-03-10T01:39:54.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:54 smithi038 ceph-mon[92422]: pgmap v21045: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 13 op/s 2023-03-10T01:39:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:39:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:39:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:39:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:39:55.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:39:55.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:39:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38506.538748 secs 2023-03-10T01:39:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:39:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:39:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:56 smithi122 ceph-mon[93869]: pgmap v21046: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 12 op/s 2023-03-10T01:39:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38506.538748 secs 2023-03-10T01:39:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:39:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:39:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:56 smithi181 ceph-mon[94189]: pgmap v21046: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 12 op/s 2023-03-10T01:39:56.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38506.538748 secs 2023-03-10T01:39:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:39:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:39:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:56 smithi038 ceph-mon[92422]: pgmap v21046: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 12 op/s 2023-03-10T01:39:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:39:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:39:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:39:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:39:57.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:39:57.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:39:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:58 smithi122 ceph-mon[93869]: pgmap v21047: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 980 KiB/s wr, 14 op/s 2023-03-10T01:39:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:39:58 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 38508.997963 secs 2023-03-10T01:39:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:58 smithi181 ceph-mon[94189]: pgmap v21047: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 980 KiB/s wr, 14 op/s 2023-03-10T01:39:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:39:58 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 38508.997963 secs 2023-03-10T01:39:58.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:58 smithi038 ceph-mon[92422]: pgmap v21047: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 980 KiB/s wr, 14 op/s 2023-03-10T01:39:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:39:58 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 38508.997963 secs 2023-03-10T01:39:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38970.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:40:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:00 smithi122 ceph-mon[93869]: pgmap v21048: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 979 KiB/s wr, 11 op/s 2023-03-10T01:40:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:40:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:40:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:00 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:40:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:00 smithi122 ceph-mon[93869]: Health detail: HEALTH_WARN 2 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-10T01:40:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:00 smithi122 ceph-mon[93869]: [WRN] MDS_SLOW_REQUEST: 2 MDSs report slow requests 2023-03-10T01:40:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:00 smithi122 ceph-mon[93869]: mds.i(mds.2): 1 slow requests are blocked > 180 secs 2023-03-10T01:40:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:00 smithi122 ceph-mon[93869]: mds.h(mds.1): 2 slow requests are blocked > 180 secs 2023-03-10T01:40:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:00 smithi122 ceph-mon[93869]: [WRN] MDS_TRIM: 2 MDSs behind on trimming 2023-03-10T01:40:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:00 smithi122 ceph-mon[93869]: mds.e(mds.0): Behind on trimming (268/128) max_segments: 128, num_segments: 268 2023-03-10T01:40:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:00 smithi122 ceph-mon[93869]: mds.b(mds.0): Behind on trimming (267/128) max_segments: 128, num_segments: 267 2023-03-10T01:40:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:00 smithi181 ceph-mon[94189]: pgmap v21048: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 979 KiB/s wr, 11 op/s 2023-03-10T01:40:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:40:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:40:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:00 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:40:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:00 smithi181 ceph-mon[94189]: Health detail: HEALTH_WARN 2 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-10T01:40:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:00 smithi181 ceph-mon[94189]: [WRN] MDS_SLOW_REQUEST: 2 MDSs report slow requests 2023-03-10T01:40:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:00 smithi181 ceph-mon[94189]: mds.i(mds.2): 1 slow requests are blocked > 180 secs 2023-03-10T01:40:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:00 smithi181 ceph-mon[94189]: mds.h(mds.1): 2 slow requests are blocked > 180 secs 2023-03-10T01:40:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:00 smithi181 ceph-mon[94189]: [WRN] MDS_TRIM: 2 MDSs behind on trimming 2023-03-10T01:40:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:00 smithi181 ceph-mon[94189]: mds.e(mds.0): Behind on trimming (268/128) max_segments: 128, num_segments: 268 2023-03-10T01:40:00.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:00 smithi181 ceph-mon[94189]: mds.b(mds.0): Behind on trimming (267/128) max_segments: 128, num_segments: 267 2023-03-10T01:40:00.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:00 smithi038 ceph-mon[92422]: pgmap v21048: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 979 KiB/s wr, 11 op/s 2023-03-10T01:40:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:40:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:40:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:00 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:40:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:00 smithi038 ceph-mon[92422]: Health detail: HEALTH_WARN 2 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-10T01:40:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:00 smithi038 ceph-mon[92422]: [WRN] MDS_SLOW_REQUEST: 2 MDSs report slow requests 2023-03-10T01:40:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:00 smithi038 ceph-mon[92422]: mds.i(mds.2): 1 slow requests are blocked > 180 secs 2023-03-10T01:40:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:00 smithi038 ceph-mon[92422]: mds.h(mds.1): 2 slow requests are blocked > 180 secs 2023-03-10T01:40:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:00 smithi038 ceph-mon[92422]: [WRN] MDS_TRIM: 2 MDSs behind on trimming 2023-03-10T01:40:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:00 smithi038 ceph-mon[92422]: mds.e(mds.0): Behind on trimming (268/128) max_segments: 128, num_segments: 268 2023-03-10T01:40:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:00 smithi038 ceph-mon[92422]: mds.b(mds.0): Behind on trimming (267/128) max_segments: 128, num_segments: 267 2023-03-10T01:40:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38511.538831 secs 2023-03-10T01:40:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:40:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:40:02.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38511.538831 secs 2023-03-10T01:40:02.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:40:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:40:02.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38511.538831 secs 2023-03-10T01:40:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:40:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:40:03.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:02 smithi181 ceph-mon[94189]: pgmap v21049: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 14 op/s 2023-03-10T01:40:03.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:02 smithi122 ceph-mon[93869]: pgmap v21049: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 14 op/s 2023-03-10T01:40:03.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:02 smithi038 ceph-mon[92422]: pgmap v21049: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 14 op/s 2023-03-10T01:40:03.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38975.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:40:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38513.998022 secs 2023-03-10T01:40:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38513.998022 secs 2023-03-10T01:40:04.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38513.998022 secs 2023-03-10T01:40:05.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:04 smithi122 ceph-mon[93869]: pgmap v21050: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 963 KiB/s rd, 727 KiB/s wr, 20 op/s 2023-03-10T01:40:05.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:04 smithi181 ceph-mon[94189]: pgmap v21050: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 963 KiB/s rd, 727 KiB/s wr, 20 op/s 2023-03-10T01:40:05.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:04 smithi038 ceph-mon[92422]: pgmap v21050: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 963 KiB/s rd, 727 KiB/s wr, 20 op/s 2023-03-10T01:40:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38516.538907 secs 2023-03-10T01:40:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:06 smithi122 ceph-mon[93869]: pgmap v21051: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 21 op/s 2023-03-10T01:40:07.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:06 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:40:07.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:40:07.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38516.538907 secs 2023-03-10T01:40:07.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:06 smithi181 ceph-mon[94189]: pgmap v21051: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 21 op/s 2023-03-10T01:40:07.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:06 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:40:07.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:40:07.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38516.538907 secs 2023-03-10T01:40:07.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:06 smithi038 ceph-mon[92422]: pgmap v21051: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 21 op/s 2023-03-10T01:40:07.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:06 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:40:07.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:40:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:40:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:40:08.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:40:08.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:40:08.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:40:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:40:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38980.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:40:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:08 smithi122 ceph-mon[93869]: pgmap v21052: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 25 op/s 2023-03-10T01:40:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38518.998111 secs 2023-03-10T01:40:09.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:08 smithi181 ceph-mon[94189]: pgmap v21052: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 25 op/s 2023-03-10T01:40:09.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38518.998111 secs 2023-03-10T01:40:09.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:08 smithi038 ceph-mon[92422]: pgmap v21052: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 25 op/s 2023-03-10T01:40:09.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38518.998111 secs 2023-03-10T01:40:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:40:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:40:10.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:40:10.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:40:10.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:40:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:40:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:10 smithi122 ceph-mon[93869]: pgmap v21053: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 23 op/s 2023-03-10T01:40:11.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:10 smithi181 ceph-mon[94189]: pgmap v21053: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 23 op/s 2023-03-10T01:40:11.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:10 smithi038 ceph-mon[92422]: pgmap v21053: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 23 op/s 2023-03-10T01:40:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38521.538993 secs 2023-03-10T01:40:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:11 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:40:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:11 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:40:12.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38521.538993 secs 2023-03-10T01:40:12.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:11 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:40:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:11 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:40:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38521.538993 secs 2023-03-10T01:40:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:11 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:40:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:11 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:40:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:12 smithi122 ceph-mon[93869]: pgmap v21054: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 26 op/s 2023-03-10T01:40:13.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:12 smithi181 ceph-mon[94189]: pgmap v21054: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 26 op/s 2023-03-10T01:40:13.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:12 smithi038 ceph-mon[92422]: pgmap v21054: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 26 op/s 2023-03-10T01:40:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38985.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:40:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38523.998229 secs 2023-03-10T01:40:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38523.998229 secs 2023-03-10T01:40:14.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38523.998229 secs 2023-03-10T01:40:15.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:14 smithi122 ceph-mon[93869]: pgmap v21055: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 23 op/s 2023-03-10T01:40:15.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:14 smithi181 ceph-mon[94189]: pgmap v21055: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 23 op/s 2023-03-10T01:40:15.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:14 smithi038 ceph-mon[92422]: pgmap v21055: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 23 op/s 2023-03-10T01:40:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38526.539091 secs 2023-03-10T01:40:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:16 smithi122 ceph-mon[93869]: pgmap v21056: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 814 KiB/s rd, 672 KiB/s wr, 11 op/s 2023-03-10T01:40:17.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38526.539091 secs 2023-03-10T01:40:17.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:16 smithi181 ceph-mon[94189]: pgmap v21056: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 814 KiB/s rd, 672 KiB/s wr, 11 op/s 2023-03-10T01:40:17.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38526.539091 secs 2023-03-10T01:40:17.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:16 smithi038 ceph-mon[92422]: pgmap v21056: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 814 KiB/s rd, 672 KiB/s wr, 11 op/s 2023-03-10T01:40:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38990.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:40:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:18 smithi122 ceph-mon[93869]: pgmap v21057: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 298 KiB/s wr, 12 op/s 2023-03-10T01:40:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38528.998296 secs 2023-03-10T01:40:19.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:18 smithi181 ceph-mon[94189]: pgmap v21057: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 298 KiB/s wr, 12 op/s 2023-03-10T01:40:19.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38528.998296 secs 2023-03-10T01:40:19.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:18 smithi038 ceph-mon[92422]: pgmap v21057: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 298 KiB/s wr, 12 op/s 2023-03-10T01:40:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38528.998296 secs 2023-03-10T01:40:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:20 smithi122 ceph-mon[93869]: pgmap v21058: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 377 KiB/s wr, 8 op/s 2023-03-10T01:40:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:20 smithi122 ceph-mon[93869]: osdmap e1531: 12 total, 12 up, 12 in 2023-03-10T01:40:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:40:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:40:21.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:20 smithi181 ceph-mon[94189]: pgmap v21058: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 377 KiB/s wr, 8 op/s 2023-03-10T01:40:21.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:20 smithi181 ceph-mon[94189]: osdmap e1531: 12 total, 12 up, 12 in 2023-03-10T01:40:21.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:40:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:40:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:20 smithi038 ceph-mon[92422]: pgmap v21058: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 377 KiB/s wr, 8 op/s 2023-03-10T01:40:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:20 smithi038 ceph-mon[92422]: osdmap e1531: 12 total, 12 up, 12 in 2023-03-10T01:40:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:40:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:40:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38531.539151 secs 2023-03-10T01:40:22.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38531.539151 secs 2023-03-10T01:40:22.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38531.539151 secs 2023-03-10T01:40:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:22 smithi122 ceph-mon[93869]: pgmap v21060: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 519 KiB/s wr, 10 op/s 2023-03-10T01:40:23.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:22 smithi181 ceph-mon[94189]: pgmap v21060: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 519 KiB/s wr, 10 op/s 2023-03-10T01:40:23.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:22 smithi038 ceph-mon[92422]: pgmap v21060: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 519 KiB/s wr, 10 op/s 2023-03-10T01:40:23.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 38995.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:40:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38533.998377 secs 2023-03-10T01:40:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38533.998377 secs 2023-03-10T01:40:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38533.998377 secs 2023-03-10T01:40:25.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:24 smithi122 ceph-mon[93869]: pgmap v21061: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 396 KiB/s wr, 12 op/s 2023-03-10T01:40:25.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:24 smithi181 ceph-mon[94189]: pgmap v21061: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 396 KiB/s wr, 12 op/s 2023-03-10T01:40:25.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:24 smithi038 ceph-mon[92422]: pgmap v21061: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 396 KiB/s wr, 12 op/s 2023-03-10T01:40:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38536.539211 secs 2023-03-10T01:40:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:26 smithi122 ceph-mon[93869]: pgmap v21062: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 436 KiB/s wr, 12 op/s 2023-03-10T01:40:27.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38536.539211 secs 2023-03-10T01:40:27.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:26 smithi181 ceph-mon[94189]: pgmap v21062: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 436 KiB/s wr, 12 op/s 2023-03-10T01:40:27.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38536.539211 secs 2023-03-10T01:40:27.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:26 smithi038 ceph-mon[92422]: pgmap v21062: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 436 KiB/s wr, 12 op/s 2023-03-10T01:40:28.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:27 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:40:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:27 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:40:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:27 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:40:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:27 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:40:28.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:27 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:40:28.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:27 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:40:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39000.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:40:29.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:28 smithi038 ceph-mon[92422]: pgmap v21063: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 436 KiB/s wr, 13 op/s 2023-03-10T01:40:29.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38538.998500 secs 2023-03-10T01:40:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:28 smithi122 ceph-mon[93869]: pgmap v21063: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 436 KiB/s wr, 13 op/s 2023-03-10T01:40:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38538.998500 secs 2023-03-10T01:40:29.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:28 smithi181 ceph-mon[94189]: pgmap v21063: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 436 KiB/s wr, 13 op/s 2023-03-10T01:40:29.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38538.998500 secs 2023-03-10T01:40:30.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:40:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:29 smithi038 ceph-mon[92422]: osdmap e1532: 12 total, 12 up, 12 in 2023-03-10T01:40:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:40:30.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:29 smithi122 ceph-mon[93869]: osdmap e1532: 12 total, 12 up, 12 in 2023-03-10T01:40:30.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:40:30.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:29 smithi181 ceph-mon[94189]: osdmap e1532: 12 total, 12 up, 12 in 2023-03-10T01:40:31.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:30 smithi038 ceph-mon[92422]: pgmap v21064: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 811 KiB/s rd, 599 KiB/s wr, 13 op/s 2023-03-10T01:40:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38541.539271 secs 2023-03-10T01:40:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:30 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:40:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:30 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:40:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:30 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:40:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:30 smithi122 ceph-mon[93869]: pgmap v21064: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 811 KiB/s rd, 599 KiB/s wr, 13 op/s 2023-03-10T01:40:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38541.539271 secs 2023-03-10T01:40:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:30 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:40:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:30 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:40:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:30 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:40:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:30 smithi181 ceph-mon[94189]: pgmap v21064: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 811 KiB/s rd, 599 KiB/s wr, 13 op/s 2023-03-10T01:40:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38541.539271 secs 2023-03-10T01:40:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:30 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:40:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:30 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:40:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:30 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:40:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:33 smithi122 ceph-mon[93869]: pgmap v21066: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 786 KiB/s rd, 560 KiB/s wr, 12 op/s 2023-03-10T01:40:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:33 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:40:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:33 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:40:33.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:33 smithi181 ceph-mon[94189]: pgmap v21066: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 786 KiB/s rd, 560 KiB/s wr, 12 op/s 2023-03-10T01:40:33.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:33 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:40:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:33 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:40:33.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:33 smithi038 ceph-mon[92422]: pgmap v21066: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 786 KiB/s rd, 560 KiB/s wr, 12 op/s 2023-03-10T01:40:33.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:33 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:40:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:33 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:40:33.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39005.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:40:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:34 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38543.998608 secs 2023-03-10T01:40:34.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:34 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38543.998608 secs 2023-03-10T01:40:34.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:34 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38543.998608 secs 2023-03-10T01:40:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:35 smithi122 ceph-mon[93869]: pgmap v21067: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 494 KiB/s wr, 10 op/s 2023-03-10T01:40:35.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:35 smithi181 ceph-mon[94189]: pgmap v21067: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 494 KiB/s wr, 10 op/s 2023-03-10T01:40:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:35 smithi038 ceph-mon[92422]: pgmap v21067: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 494 KiB/s wr, 10 op/s 2023-03-10T01:40:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:40:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:40:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38546.539345 secs 2023-03-10T01:40:36.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:40:36.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:40:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38546.539345 secs 2023-03-10T01:40:36.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:40:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:40:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38546.539345 secs 2023-03-10T01:40:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:37 smithi122 ceph-mon[93869]: pgmap v21068: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 627 KiB/s rd, 625 KiB/s wr, 11 op/s 2023-03-10T01:40:37.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:37 smithi181 ceph-mon[94189]: pgmap v21068: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 627 KiB/s rd, 625 KiB/s wr, 11 op/s 2023-03-10T01:40:37.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:37 smithi038 ceph-mon[92422]: pgmap v21068: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 627 KiB/s rd, 625 KiB/s wr, 11 op/s 2023-03-10T01:40:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39010.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:40:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:39 smithi122 ceph-mon[93869]: pgmap v21069: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 627 KiB/s rd, 624 KiB/s wr, 10 op/s 2023-03-10T01:40:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:39 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38548.998770 secs 2023-03-10T01:40:39.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:39 smithi181 ceph-mon[94189]: pgmap v21069: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 627 KiB/s rd, 624 KiB/s wr, 10 op/s 2023-03-10T01:40:39.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:39 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38548.998770 secs 2023-03-10T01:40:39.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:39 smithi038 ceph-mon[92422]: pgmap v21069: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 627 KiB/s rd, 624 KiB/s wr, 10 op/s 2023-03-10T01:40:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:39 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38548.998770 secs 2023-03-10T01:40:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:41 smithi122 ceph-mon[93869]: pgmap v21070: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 199 KiB/s wr, 9 op/s 2023-03-10T01:40:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38551.539403 secs 2023-03-10T01:40:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:40:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:40:41.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:41 smithi181 ceph-mon[94189]: pgmap v21070: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 199 KiB/s wr, 9 op/s 2023-03-10T01:40:41.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38551.539403 secs 2023-03-10T01:40:41.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:40:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:40:41.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:41 smithi038 ceph-mon[92422]: pgmap v21070: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 199 KiB/s wr, 9 op/s 2023-03-10T01:40:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38551.539403 secs 2023-03-10T01:40:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:40:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:40:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:43 smithi122 ceph-mon[93869]: pgmap v21071: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 503 KiB/s wr, 11 op/s 2023-03-10T01:40:43.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:43 smithi181 ceph-mon[94189]: pgmap v21071: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 503 KiB/s wr, 11 op/s 2023-03-10T01:40:43.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:43 smithi038 ceph-mon[92422]: pgmap v21071: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 506 KiB/s rd, 503 KiB/s wr, 11 op/s 2023-03-10T01:40:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39015.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:40:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:44 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38553.998856 secs 2023-03-10T01:40:44.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:44 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38553.998856 secs 2023-03-10T01:40:44.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:44 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38553.998856 secs 2023-03-10T01:40:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:45 smithi122 ceph-mon[93869]: pgmap v21072: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 460 KiB/s wr, 8 op/s 2023-03-10T01:40:45.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:45 smithi181 ceph-mon[94189]: pgmap v21072: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 460 KiB/s wr, 8 op/s 2023-03-10T01:40:45.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:45 smithi038 ceph-mon[92422]: pgmap v21072: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 460 KiB/s wr, 8 op/s 2023-03-10T01:40:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38556.539463 secs 2023-03-10T01:40:46.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38556.539463 secs 2023-03-10T01:40:46.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38556.539463 secs 2023-03-10T01:40:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:47 smithi122 ceph-mon[93869]: pgmap v21073: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 471 KiB/s rd, 469 KiB/s wr, 8 op/s 2023-03-10T01:40:47.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:47 smithi181 ceph-mon[94189]: pgmap v21073: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 471 KiB/s rd, 469 KiB/s wr, 8 op/s 2023-03-10T01:40:47.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:47 smithi038 ceph-mon[92422]: pgmap v21073: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 471 KiB/s rd, 469 KiB/s wr, 8 op/s 2023-03-10T01:40:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:40:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:40:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:40:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:40:48.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:40:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:40:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39020.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:40:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:49 smithi122 ceph-mon[93869]: pgmap v21074: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 327 KiB/s wr, 9 op/s 2023-03-10T01:40:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:49 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38558.998942 secs 2023-03-10T01:40:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:40:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:40:49.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:49 smithi181 ceph-mon[94189]: pgmap v21074: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 327 KiB/s wr, 9 op/s 2023-03-10T01:40:49.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:49 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38558.998942 secs 2023-03-10T01:40:49.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:40:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:40:49.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:49 smithi038 ceph-mon[92422]: pgmap v21074: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 327 KiB/s wr, 9 op/s 2023-03-10T01:40:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:49 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38558.998942 secs 2023-03-10T01:40:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:40:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:40:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:51 smithi122 ceph-mon[93869]: pgmap v21075: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 498 KiB/s wr, 8 op/s 2023-03-10T01:40:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38561.539535 secs 2023-03-10T01:40:51.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:51 smithi181 ceph-mon[94189]: pgmap v21075: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 498 KiB/s wr, 8 op/s 2023-03-10T01:40:51.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38561.539535 secs 2023-03-10T01:40:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:51 smithi038 ceph-mon[92422]: pgmap v21075: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 498 KiB/s wr, 8 op/s 2023-03-10T01:40:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38561.539535 secs 2023-03-10T01:40:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:40:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:40:52.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:40:52.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:40:52.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:40:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:40:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:53 smithi122 ceph-mon[93869]: pgmap v21076: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 498 KiB/s wr, 9 op/s 2023-03-10T01:40:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38563.999023 secs 2023-03-10T01:40:53.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:53 smithi181 ceph-mon[94189]: pgmap v21076: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 498 KiB/s wr, 9 op/s 2023-03-10T01:40:53.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38563.999023 secs 2023-03-10T01:40:53.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:53 smithi038 ceph-mon[92422]: pgmap v21076: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 498 KiB/s wr, 9 op/s 2023-03-10T01:40:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38563.999023 secs 2023-03-10T01:40:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39025.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:40:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:55 smithi122 ceph-mon[93869]: pgmap v21077: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 180 KiB/s wr, 7 op/s 2023-03-10T01:40:55.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:55 smithi181 ceph-mon[94189]: pgmap v21077: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 180 KiB/s wr, 7 op/s 2023-03-10T01:40:55.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:55 smithi038 ceph-mon[92422]: pgmap v21077: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 180 KiB/s wr, 7 op/s 2023-03-10T01:40:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38566.539615 secs 2023-03-10T01:40:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:40:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:40:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38566.539615 secs 2023-03-10T01:40:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:40:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:40:56.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38566.539615 secs 2023-03-10T01:40:56.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:40:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:40:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:57 smithi122 ceph-mon[93869]: pgmap v21078: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 387 KiB/s wr, 9 op/s 2023-03-10T01:40:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:40:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:40:57.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:57 smithi181 ceph-mon[94189]: pgmap v21078: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 387 KiB/s wr, 9 op/s 2023-03-10T01:40:57.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:40:57.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:40:57.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:57 smithi038 ceph-mon[92422]: pgmap v21078: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 387 KiB/s wr, 9 op/s 2023-03-10T01:40:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:40:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:40:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38568.999157 secs 2023-03-10T01:40:58.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38568.999157 secs 2023-03-10T01:40:58.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38568.999157 secs 2023-03-10T01:40:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39030.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:40:59.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:40:59 smithi122 ceph-mon[93869]: pgmap v21079: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 384 KiB/s rd, 379 KiB/s wr, 10 op/s 2023-03-10T01:40:59.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:40:59 smithi181 ceph-mon[94189]: pgmap v21079: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 384 KiB/s rd, 379 KiB/s wr, 10 op/s 2023-03-10T01:40:59.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:40:59 smithi038 ceph-mon[92422]: pgmap v21079: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 384 KiB/s rd, 379 KiB/s wr, 10 op/s 2023-03-10T01:41:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:01 smithi122 ceph-mon[93869]: pgmap v21080: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 755 KiB/s rd, 741 KiB/s wr, 11 op/s 2023-03-10T01:41:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38571.539671 secs 2023-03-10T01:41:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:41:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:41:01.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:01 smithi181 ceph-mon[94189]: pgmap v21080: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 755 KiB/s rd, 741 KiB/s wr, 11 op/s 2023-03-10T01:41:01.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38571.539671 secs 2023-03-10T01:41:01.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:41:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:41:01.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:01 smithi038 ceph-mon[92422]: pgmap v21080: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 755 KiB/s rd, 741 KiB/s wr, 11 op/s 2023-03-10T01:41:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38571.539671 secs 2023-03-10T01:41:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:41:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:41:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:03 smithi122 ceph-mon[93869]: pgmap v21081: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 927 KiB/s rd, 911 KiB/s wr, 13 op/s 2023-03-10T01:41:03.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:03 smithi181 ceph-mon[94189]: pgmap v21081: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 927 KiB/s rd, 911 KiB/s wr, 13 op/s 2023-03-10T01:41:03.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:03 smithi038 ceph-mon[92422]: pgmap v21081: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 927 KiB/s rd, 911 KiB/s wr, 13 op/s 2023-03-10T01:41:03.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39035.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:41:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:05 smithi122 ceph-mon[93869]: pgmap v21082: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 933 KiB/s rd, 922 KiB/s wr, 22 op/s 2023-03-10T01:41:05.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:05 smithi181 ceph-mon[94189]: pgmap v21082: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 933 KiB/s rd, 922 KiB/s wr, 22 op/s 2023-03-10T01:41:05.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:05 smithi038 ceph-mon[92422]: pgmap v21082: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 933 KiB/s rd, 922 KiB/s wr, 22 op/s 2023-03-10T01:41:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38576.539728 secs 2023-03-10T01:41:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:06 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:41:06.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38576.539728 secs 2023-03-10T01:41:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:06 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:41:06.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38576.539728 secs 2023-03-10T01:41:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:06 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:41:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:07 smithi122 ceph-mon[93869]: pgmap v21083: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 955 KiB/s wr, 23 op/s 2023-03-10T01:41:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:07 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:41:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:41:07.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:41:07.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:07 smithi181 ceph-mon[94189]: pgmap v21083: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 955 KiB/s wr, 23 op/s 2023-03-10T01:41:07.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:07 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:41:07.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:41:07.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:41:07.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:07 smithi038 ceph-mon[92422]: pgmap v21083: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 955 KiB/s wr, 23 op/s 2023-03-10T01:41:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:07 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:41:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:41:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:41:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:41:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:41:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:41:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:41:08.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:41:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:41:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39040.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:41:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:09 smithi122 ceph-mon[93869]: pgmap v21084: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 26 op/s 2023-03-10T01:41:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:41:09.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:41:09.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:09 smithi181 ceph-mon[94189]: pgmap v21084: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 26 op/s 2023-03-10T01:41:09.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:41:09.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:41:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:09 smithi038 ceph-mon[92422]: pgmap v21084: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 26 op/s 2023-03-10T01:41:09.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:41:09.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:41:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:11 smithi122 ceph-mon[93869]: pgmap v21085: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 24 op/s 2023-03-10T01:41:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38581.539779 secs 2023-03-10T01:41:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:11 smithi181 ceph-mon[94189]: pgmap v21085: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 24 op/s 2023-03-10T01:41:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38581.539779 secs 2023-03-10T01:41:11.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:11 smithi038 ceph-mon[92422]: pgmap v21085: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 24 op/s 2023-03-10T01:41:11.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38581.539779 secs 2023-03-10T01:41:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:13 smithi122 ceph-mon[93869]: pgmap v21086: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 25 op/s 2023-03-10T01:41:13.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:13 smithi181 ceph-mon[94189]: pgmap v21086: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 25 op/s 2023-03-10T01:41:13.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:13 smithi038 ceph-mon[92422]: pgmap v21086: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 25 op/s 2023-03-10T01:41:13.793 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39045.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:41:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:15 smithi122 ceph-mon[93869]: pgmap v21087: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 833 KiB/s rd, 738 KiB/s wr, 22 op/s 2023-03-10T01:41:15.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:15 smithi181 ceph-mon[94189]: pgmap v21087: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 833 KiB/s rd, 738 KiB/s wr, 22 op/s 2023-03-10T01:41:15.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:15 smithi038 ceph-mon[92422]: pgmap v21087: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 833 KiB/s rd, 738 KiB/s wr, 22 op/s 2023-03-10T01:41:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38586.539861 secs 2023-03-10T01:41:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:41:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:41:16.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38586.539861 secs 2023-03-10T01:41:16.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:41:16.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:41:16.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38586.539861 secs 2023-03-10T01:41:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:41:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:41:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:17 smithi122 ceph-mon[93869]: pgmap v21088: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 827 KiB/s rd, 728 KiB/s wr, 12 op/s 2023-03-10T01:41:17.487 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:41:17.487 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:41:17.488 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:17 smithi181 ceph-mon[94189]: pgmap v21088: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 827 KiB/s rd, 728 KiB/s wr, 12 op/s 2023-03-10T01:41:17.488 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:41:17.488 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:41:17.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:17 smithi038 ceph-mon[92422]: pgmap v21088: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 827 KiB/s rd, 728 KiB/s wr, 12 op/s 2023-03-10T01:41:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:41:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:41:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39050.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:41:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:19 smithi122 ceph-mon[93869]: pgmap v21089: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 688 KiB/s rd, 696 KiB/s wr, 13 op/s 2023-03-10T01:41:19.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:19 smithi181 ceph-mon[94189]: pgmap v21089: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 688 KiB/s rd, 696 KiB/s wr, 13 op/s 2023-03-10T01:41:19.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:19 smithi038 ceph-mon[92422]: pgmap v21089: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 688 KiB/s rd, 696 KiB/s wr, 13 op/s 2023-03-10T01:41:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:41:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:41:20.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:41:20.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:41:20.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:41:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:41:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:21 smithi122 ceph-mon[93869]: pgmap v21090: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 571 KiB/s wr, 12 op/s 2023-03-10T01:41:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38591.539931 secs 2023-03-10T01:41:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:41:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:41:21.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:21 smithi181 ceph-mon[94189]: pgmap v21090: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 571 KiB/s wr, 12 op/s 2023-03-10T01:41:21.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38591.539931 secs 2023-03-10T01:41:21.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:41:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:41:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:21 smithi038 ceph-mon[92422]: pgmap v21090: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 571 KiB/s wr, 12 op/s 2023-03-10T01:41:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38591.539931 secs 2023-03-10T01:41:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:41:21.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:41:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:23 smithi122 ceph-mon[93869]: pgmap v21091: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 582 KiB/s wr, 14 op/s 2023-03-10T01:41:23.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:23 smithi181 ceph-mon[94189]: pgmap v21091: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 582 KiB/s wr, 14 op/s 2023-03-10T01:41:23.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:23 smithi038 ceph-mon[92422]: pgmap v21091: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 564 KiB/s rd, 582 KiB/s wr, 14 op/s 2023-03-10T01:41:23.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39055.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:41:25.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:25 smithi181 ceph-mon[94189]: pgmap v21092: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 595 KiB/s wr, 13 op/s 2023-03-10T01:41:25.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:25 smithi038 ceph-mon[92422]: pgmap v21092: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 595 KiB/s wr, 13 op/s 2023-03-10T01:41:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:25 smithi122 ceph-mon[93869]: pgmap v21092: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 595 KiB/s wr, 13 op/s 2023-03-10T01:41:26.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38596.539991 secs 2023-03-10T01:41:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:26 smithi038 ceph-mon[92422]: osdmap e1533: 12 total, 12 up, 12 in 2023-03-10T01:41:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38596.539991 secs 2023-03-10T01:41:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:26 smithi122 ceph-mon[93869]: osdmap e1533: 12 total, 12 up, 12 in 2023-03-10T01:41:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38596.539991 secs 2023-03-10T01:41:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:26 smithi181 ceph-mon[94189]: osdmap e1533: 12 total, 12 up, 12 in 2023-03-10T01:41:27.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:27 smithi038 ceph-mon[92422]: pgmap v21094: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 707 KiB/s rd, 731 KiB/s wr, 14 op/s 2023-03-10T01:41:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:27 smithi122 ceph-mon[93869]: pgmap v21094: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 707 KiB/s rd, 731 KiB/s wr, 14 op/s 2023-03-10T01:41:27.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:27 smithi181 ceph-mon[94189]: pgmap v21094: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 707 KiB/s rd, 731 KiB/s wr, 14 op/s 2023-03-10T01:41:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39060.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:41:29.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:29 smithi122 ceph-mon[93869]: pgmap v21095: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 880 KiB/s rd, 903 KiB/s wr, 14 op/s 2023-03-10T01:41:29.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:29 smithi181 ceph-mon[94189]: pgmap v21095: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 880 KiB/s rd, 903 KiB/s wr, 14 op/s 2023-03-10T01:41:29.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:29 smithi038 ceph-mon[92422]: pgmap v21095: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 880 KiB/s rd, 903 KiB/s wr, 14 op/s 2023-03-10T01:41:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:30 smithi122 ceph-mon[93869]: pgmap v21096: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 470 KiB/s rd, 482 KiB/s wr, 11 op/s 2023-03-10T01:41:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:30 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:41:31.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:30 smithi181 ceph-mon[94189]: pgmap v21096: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 470 KiB/s rd, 482 KiB/s wr, 11 op/s 2023-03-10T01:41:31.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:30 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:41:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:30 smithi038 ceph-mon[92422]: pgmap v21096: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 470 KiB/s rd, 482 KiB/s wr, 11 op/s 2023-03-10T01:41:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:30 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:41:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38601.540045 secs 2023-03-10T01:41:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:41:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:41:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:41:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38601.540045 secs 2023-03-10T01:41:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:41:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:41:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:41:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38601.540045 secs 2023-03-10T01:41:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:41:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:41:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:41:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:32 smithi122 ceph-mon[93869]: pgmap v21097: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 470 KiB/s rd, 469 KiB/s wr, 10 op/s 2023-03-10T01:41:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:32 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:41:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:32 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:41:33.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:32 smithi181 ceph-mon[94189]: pgmap v21097: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 470 KiB/s rd, 469 KiB/s wr, 10 op/s 2023-03-10T01:41:33.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:32 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:41:33.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:32 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:41:33.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:32 smithi038 ceph-mon[92422]: pgmap v21097: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 470 KiB/s rd, 469 KiB/s wr, 10 op/s 2023-03-10T01:41:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:32 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:41:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:32 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:41:33.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39065.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:41:35.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:34 smithi038 ceph-mon[92422]: pgmap v21098: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 232 KiB/s wr, 9 op/s 2023-03-10T01:41:35.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:34 smithi038 ceph-mon[92422]: osdmap e1534: 12 total, 12 up, 12 in 2023-03-10T01:41:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:34 smithi122 ceph-mon[93869]: pgmap v21098: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 232 KiB/s wr, 9 op/s 2023-03-10T01:41:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:34 smithi122 ceph-mon[93869]: osdmap e1534: 12 total, 12 up, 12 in 2023-03-10T01:41:35.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:34 smithi181 ceph-mon[94189]: pgmap v21098: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 233 KiB/s rd, 232 KiB/s wr, 9 op/s 2023-03-10T01:41:35.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:34 smithi181 ceph-mon[94189]: osdmap e1534: 12 total, 12 up, 12 in 2023-03-10T01:41:36.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:35 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:41:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:35 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:41:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38606.540110 secs 2023-03-10T01:41:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:35 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:41:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:35 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:41:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38606.540110 secs 2023-03-10T01:41:36.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:35 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:41:36.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:35 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:41:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38606.540110 secs 2023-03-10T01:41:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:36 smithi122 ceph-mon[93869]: pgmap v21100: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 646 KiB/s rd, 645 KiB/s wr, 10 op/s 2023-03-10T01:41:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:41:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:41:37.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:37 smithi181 ceph-mon[94189]: pgmap v21100: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 646 KiB/s rd, 645 KiB/s wr, 10 op/s 2023-03-10T01:41:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:41:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:41:37.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:37 smithi038 ceph-mon[92422]: pgmap v21100: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 646 KiB/s rd, 645 KiB/s wr, 10 op/s 2023-03-10T01:41:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:41:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:41:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:38 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:41:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:38 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:41:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:38 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:41:38.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:38 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:41:38.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:38 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:41:38.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:38 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:41:38.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:38 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:41:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:38 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:41:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:38 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:41:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39070.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:41:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:39 smithi122 ceph-mon[93869]: pgmap v21101: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 879 KiB/s rd, 882 KiB/s wr, 10 op/s 2023-03-10T01:41:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:39 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38608.999801 secs 2023-03-10T01:41:39.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:39 smithi181 ceph-mon[94189]: pgmap v21101: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 879 KiB/s rd, 882 KiB/s wr, 10 op/s 2023-03-10T01:41:39.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:39 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38608.999801 secs 2023-03-10T01:41:39.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:39 smithi038 ceph-mon[92422]: pgmap v21101: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 879 KiB/s rd, 882 KiB/s wr, 10 op/s 2023-03-10T01:41:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:39 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38608.999801 secs 2023-03-10T01:41:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:41 smithi122 ceph-mon[93869]: pgmap v21102: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 907 KiB/s rd, 909 KiB/s wr, 11 op/s 2023-03-10T01:41:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38611.540170 secs 2023-03-10T01:41:41.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:41 smithi181 ceph-mon[94189]: pgmap v21102: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 907 KiB/s rd, 909 KiB/s wr, 11 op/s 2023-03-10T01:41:41.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38611.540170 secs 2023-03-10T01:41:41.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:41 smithi038 ceph-mon[92422]: pgmap v21102: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 907 KiB/s rd, 909 KiB/s wr, 11 op/s 2023-03-10T01:41:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38611.540170 secs 2023-03-10T01:41:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:43 smithi122 ceph-mon[93869]: pgmap v21103: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 911 KiB/s rd, 909 KiB/s wr, 11 op/s 2023-03-10T01:41:43.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:43 smithi181 ceph-mon[94189]: pgmap v21103: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 911 KiB/s rd, 909 KiB/s wr, 11 op/s 2023-03-10T01:41:43.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:43 smithi038 ceph-mon[92422]: pgmap v21103: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 911 KiB/s rd, 909 KiB/s wr, 11 op/s 2023-03-10T01:41:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39075.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:41:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:44 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38613.999861 secs 2023-03-10T01:41:44.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:44 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38613.999861 secs 2023-03-10T01:41:44.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:44 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38613.999861 secs 2023-03-10T01:41:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:45 smithi122 ceph-mon[93869]: pgmap v21104: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 911 KiB/s rd, 1006 KiB/s wr, 10 op/s 2023-03-10T01:41:45.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:45 smithi181 ceph-mon[94189]: pgmap v21104: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 911 KiB/s rd, 1006 KiB/s wr, 10 op/s 2023-03-10T01:41:45.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:45 smithi038 ceph-mon[92422]: pgmap v21104: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 911 KiB/s rd, 1006 KiB/s wr, 10 op/s 2023-03-10T01:41:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38616.540233 secs 2023-03-10T01:41:46.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38616.540233 secs 2023-03-10T01:41:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38616.540233 secs 2023-03-10T01:41:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:47 smithi122 ceph-mon[93869]: pgmap v21105: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 11 op/s 2023-03-10T01:41:47.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:47 smithi181 ceph-mon[94189]: pgmap v21105: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 11 op/s 2023-03-10T01:41:47.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:47 smithi038 ceph-mon[92422]: pgmap v21105: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 11 op/s 2023-03-10T01:41:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:41:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:41:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:41:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:41:48.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:41:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:41:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39080.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:41:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:49 smithi122 ceph-mon[93869]: pgmap v21106: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 765 KiB/s wr, 11 op/s 2023-03-10T01:41:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:41:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:41:49.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:49 smithi181 ceph-mon[94189]: pgmap v21106: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 765 KiB/s wr, 11 op/s 2023-03-10T01:41:49.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:41:49.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:41:49.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:49 smithi038 ceph-mon[92422]: pgmap v21106: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 765 KiB/s wr, 11 op/s 2023-03-10T01:41:49.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:41:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:41:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:50 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:41:50.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:50 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:41:50.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:50 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:41:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:51 smithi122 ceph-mon[93869]: pgmap v21107: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 539 KiB/s wr, 10 op/s 2023-03-10T01:41:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:51 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:41:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38621.540296 secs 2023-03-10T01:41:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:41:51.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:41:51.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:51 smithi181 ceph-mon[94189]: pgmap v21107: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 539 KiB/s wr, 10 op/s 2023-03-10T01:41:51.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:51 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:41:51.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38621.540296 secs 2023-03-10T01:41:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:41:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:41:51.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:51 smithi038 ceph-mon[92422]: pgmap v21107: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 539 KiB/s wr, 10 op/s 2023-03-10T01:41:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:51 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:41:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38621.540296 secs 2023-03-10T01:41:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:41:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:41:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:53 smithi122 ceph-mon[93869]: pgmap v21108: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 483 KiB/s wr, 11 op/s 2023-03-10T01:41:53.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:53 smithi181 ceph-mon[94189]: pgmap v21108: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 483 KiB/s wr, 11 op/s 2023-03-10T01:41:53.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:53 smithi038 ceph-mon[92422]: pgmap v21108: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 483 KiB/s wr, 11 op/s 2023-03-10T01:41:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39085.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:41:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:55 smithi122 ceph-mon[93869]: pgmap v21109: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 630 KiB/s rd, 628 KiB/s wr, 9 op/s 2023-03-10T01:41:55.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:55 smithi181 ceph-mon[94189]: pgmap v21109: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 630 KiB/s rd, 628 KiB/s wr, 9 op/s 2023-03-10T01:41:55.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:55 smithi038 ceph-mon[92422]: pgmap v21109: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 630 KiB/s rd, 628 KiB/s wr, 9 op/s 2023-03-10T01:41:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38626.540383 secs 2023-03-10T01:41:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:41:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:41:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38626.540383 secs 2023-03-10T01:41:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:41:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:41:56.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38626.540383 secs 2023-03-10T01:41:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:41:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:41:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:57 smithi122 ceph-mon[93869]: pgmap v21110: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 751 KiB/s rd, 669 KiB/s wr, 10 op/s 2023-03-10T01:41:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:41:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:41:57.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:57 smithi181 ceph-mon[94189]: pgmap v21110: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 751 KiB/s rd, 669 KiB/s wr, 10 op/s 2023-03-10T01:41:57.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:41:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:41:57.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:57 smithi038 ceph-mon[92422]: pgmap v21110: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 751 KiB/s rd, 669 KiB/s wr, 10 op/s 2023-03-10T01:41:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:41:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:41:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39090.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:41:59.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:41:59 smithi122 ceph-mon[93869]: pgmap v21111: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 469 KiB/s rd, 384 KiB/s wr, 11 op/s 2023-03-10T01:41:59.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:41:59 smithi181 ceph-mon[94189]: pgmap v21111: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 469 KiB/s rd, 384 KiB/s wr, 11 op/s 2023-03-10T01:41:59.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:41:59 smithi038 ceph-mon[92422]: pgmap v21111: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 469 KiB/s rd, 384 KiB/s wr, 11 op/s 2023-03-10T01:42:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:42:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:42:00.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:42:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:42:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:42:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:42:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:01 smithi122 ceph-mon[93869]: pgmap v21112: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1007 KiB/s rd, 669 KiB/s wr, 12 op/s 2023-03-10T01:42:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38631.540442 secs 2023-03-10T01:42:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:42:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:42:01.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:01 smithi181 ceph-mon[94189]: pgmap v21112: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1007 KiB/s rd, 669 KiB/s wr, 12 op/s 2023-03-10T01:42:01.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38631.540442 secs 2023-03-10T01:42:01.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:42:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:42:01.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:01 smithi038 ceph-mon[92422]: pgmap v21112: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1007 KiB/s rd, 669 KiB/s wr, 12 op/s 2023-03-10T01:42:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38631.540442 secs 2023-03-10T01:42:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:42:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:42:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:03 smithi122 ceph-mon[93869]: pgmap v21113: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 822 KiB/s rd, 574 KiB/s wr, 14 op/s 2023-03-10T01:42:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:03 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 38634.000270 secs 2023-03-10T01:42:03.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:03 smithi181 ceph-mon[94189]: pgmap v21113: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 822 KiB/s rd, 574 KiB/s wr, 14 op/s 2023-03-10T01:42:03.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:03 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 38634.000270 secs 2023-03-10T01:42:03.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:03 smithi038 ceph-mon[92422]: pgmap v21113: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 822 KiB/s rd, 574 KiB/s wr, 14 op/s 2023-03-10T01:42:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:03 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 38634.000270 secs 2023-03-10T01:42:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39095.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:42:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:05 smithi122 ceph-mon[93869]: pgmap v21114: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.0 MiB/s wr, 25 op/s 2023-03-10T01:42:05.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:05 smithi181 ceph-mon[94189]: pgmap v21114: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.0 MiB/s wr, 25 op/s 2023-03-10T01:42:05.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:05 smithi038 ceph-mon[92422]: pgmap v21114: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.0 MiB/s wr, 25 op/s 2023-03-10T01:42:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38636.540500 secs 2023-03-10T01:42:06.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38636.540500 secs 2023-03-10T01:42:06.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38636.540500 secs 2023-03-10T01:42:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:07 smithi122 ceph-mon[93869]: pgmap v21115: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 918 KiB/s wr, 25 op/s 2023-03-10T01:42:07.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:07 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:42:07.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:42:07.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:42:07.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:07 smithi181 ceph-mon[94189]: pgmap v21115: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 918 KiB/s wr, 25 op/s 2023-03-10T01:42:07.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:07 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:42:07.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:42:07.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:42:07.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:07 smithi038 ceph-mon[92422]: pgmap v21115: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 918 KiB/s wr, 25 op/s 2023-03-10T01:42:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:07 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:42:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:42:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:42:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:42:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:42:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:08 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 38639.000328 secs 2023-03-10T01:42:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:42:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:42:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:08 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 38639.000328 secs 2023-03-10T01:42:08.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:42:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:42:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:08 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 38639.000328 secs 2023-03-10T01:42:08.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39100.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:42:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:09 smithi122 ceph-mon[93869]: pgmap v21116: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 797 KiB/s wr, 27 op/s 2023-03-10T01:42:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:42:09.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:42:09.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:09 smithi181 ceph-mon[94189]: pgmap v21116: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 797 KiB/s wr, 27 op/s 2023-03-10T01:42:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:42:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:42:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:09 smithi038 ceph-mon[92422]: pgmap v21116: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 797 KiB/s wr, 27 op/s 2023-03-10T01:42:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:42:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:42:11.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:11 smithi122 ceph-mon[93869]: pgmap v21117: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 882 KiB/s wr, 26 op/s 2023-03-10T01:42:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38641.540558 secs 2023-03-10T01:42:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:11 smithi181 ceph-mon[94189]: pgmap v21117: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 882 KiB/s wr, 26 op/s 2023-03-10T01:42:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38641.540558 secs 2023-03-10T01:42:11.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:11 smithi038 ceph-mon[92422]: pgmap v21117: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 882 KiB/s wr, 26 op/s 2023-03-10T01:42:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38641.540558 secs 2023-03-10T01:42:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:13 smithi122 ceph-mon[93869]: pgmap v21118: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 627 KiB/s rd, 639 KiB/s wr, 25 op/s 2023-03-10T01:42:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:13 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 38644.000418 secs 2023-03-10T01:42:13.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:13 smithi181 ceph-mon[94189]: pgmap v21118: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 627 KiB/s rd, 639 KiB/s wr, 25 op/s 2023-03-10T01:42:13.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:13 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 38644.000418 secs 2023-03-10T01:42:13.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:13 smithi038 ceph-mon[92422]: pgmap v21118: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 627 KiB/s rd, 639 KiB/s wr, 25 op/s 2023-03-10T01:42:13.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:13 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 38644.000418 secs 2023-03-10T01:42:13.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39105.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:42:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:15 smithi122 ceph-mon[93869]: pgmap v21119: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 759 KiB/s rd, 762 KiB/s wr, 22 op/s 2023-03-10T01:42:15.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:15 smithi181 ceph-mon[94189]: pgmap v21119: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 759 KiB/s rd, 762 KiB/s wr, 22 op/s 2023-03-10T01:42:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:15 smithi038 ceph-mon[92422]: pgmap v21119: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 759 KiB/s rd, 762 KiB/s wr, 22 op/s 2023-03-10T01:42:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38646.540614 secs 2023-03-10T01:42:16.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38646.540614 secs 2023-03-10T01:42:16.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38646.540614 secs 2023-03-10T01:42:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:17 smithi122 ceph-mon[93869]: pgmap v21120: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 297 KiB/s wr, 9 op/s 2023-03-10T01:42:17.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:17 smithi181 ceph-mon[94189]: pgmap v21120: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 297 KiB/s wr, 9 op/s 2023-03-10T01:42:17.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:17 smithi038 ceph-mon[92422]: pgmap v21120: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 297 KiB/s wr, 9 op/s 2023-03-10T01:42:18.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:18 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 38649.000477 secs 2023-03-10T01:42:18.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:18 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 38649.000477 secs 2023-03-10T01:42:18.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:18 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 38649.000477 secs 2023-03-10T01:42:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39110.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:42:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:19 smithi122 ceph-mon[93869]: pgmap v21121: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 273 KiB/s wr, 10 op/s 2023-03-10T01:42:19.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:19 smithi181 ceph-mon[94189]: pgmap v21121: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 273 KiB/s wr, 10 op/s 2023-03-10T01:42:19.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:19 smithi038 ceph-mon[92422]: pgmap v21121: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 273 KiB/s wr, 10 op/s 2023-03-10T01:42:20.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:42:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:42:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:42:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:42:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:42:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:42:21.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:21 smithi038 ceph-mon[92422]: pgmap v21122: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 283 KiB/s wr, 9 op/s 2023-03-10T01:42:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38651.540697 secs 2023-03-10T01:42:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:42:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:42:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:21 smithi122 ceph-mon[93869]: pgmap v21122: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 283 KiB/s wr, 9 op/s 2023-03-10T01:42:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38651.540697 secs 2023-03-10T01:42:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:42:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:42:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:21 smithi181 ceph-mon[94189]: pgmap v21122: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 283 KiB/s wr, 9 op/s 2023-03-10T01:42:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38651.540697 secs 2023-03-10T01:42:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:42:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:42:23.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:23 smithi038 ceph-mon[92422]: pgmap v21123: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 577 KiB/s rd, 595 KiB/s wr, 13 op/s 2023-03-10T01:42:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:23 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 38654.000536 secs 2023-03-10T01:42:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:23 smithi122 ceph-mon[93869]: pgmap v21123: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 577 KiB/s rd, 595 KiB/s wr, 13 op/s 2023-03-10T01:42:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:23 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 38654.000536 secs 2023-03-10T01:42:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:23 smithi181 ceph-mon[94189]: pgmap v21123: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 577 KiB/s rd, 595 KiB/s wr, 13 op/s 2023-03-10T01:42:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:23 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 38654.000536 secs 2023-03-10T01:42:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39115.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:42:25.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:25 smithi038 ceph-mon[92422]: pgmap v21124: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 532 KiB/s rd, 552 KiB/s wr, 11 op/s 2023-03-10T01:42:25.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:25 smithi122 ceph-mon[93869]: pgmap v21124: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 532 KiB/s rd, 552 KiB/s wr, 11 op/s 2023-03-10T01:42:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:25 smithi181 ceph-mon[94189]: pgmap v21124: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 532 KiB/s rd, 552 KiB/s wr, 11 op/s 2023-03-10T01:42:26.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38656.540781 secs 2023-03-10T01:42:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:26 smithi122 ceph-mon[93869]: osdmap e1535: 12 total, 12 up, 12 in 2023-03-10T01:42:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38656.540781 secs 2023-03-10T01:42:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:26 smithi181 ceph-mon[94189]: osdmap e1535: 12 total, 12 up, 12 in 2023-03-10T01:42:26.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38656.540781 secs 2023-03-10T01:42:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:26 smithi038 ceph-mon[92422]: osdmap e1535: 12 total, 12 up, 12 in 2023-03-10T01:42:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:27 smithi122 ceph-mon[93869]: pgmap v21126: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 681 KiB/s wr, 13 op/s 2023-03-10T01:42:27.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:27 smithi181 ceph-mon[94189]: pgmap v21126: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 681 KiB/s wr, 13 op/s 2023-03-10T01:42:27.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:27 smithi038 ceph-mon[92422]: pgmap v21126: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 681 KiB/s wr, 13 op/s 2023-03-10T01:42:28.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:28 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 38659.000598 secs 2023-03-10T01:42:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:28 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 38659.000598 secs 2023-03-10T01:42:28.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:28 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 38659.000598 secs 2023-03-10T01:42:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39120.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:42:29.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:29 smithi122 ceph-mon[93869]: pgmap v21127: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 658 KiB/s rd, 682 KiB/s wr, 13 op/s 2023-03-10T01:42:29.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:29 smithi181 ceph-mon[94189]: pgmap v21127: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 658 KiB/s rd, 682 KiB/s wr, 13 op/s 2023-03-10T01:42:29.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:29 smithi038 ceph-mon[92422]: pgmap v21127: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 658 KiB/s rd, 682 KiB/s wr, 13 op/s 2023-03-10T01:42:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:31 smithi122 ceph-mon[93869]: pgmap v21128: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 670 KiB/s wr, 11 op/s 2023-03-10T01:42:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38661.540841 secs 2023-03-10T01:42:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:42:31.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:31 smithi181 ceph-mon[94189]: pgmap v21128: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 670 KiB/s wr, 11 op/s 2023-03-10T01:42:31.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38661.540841 secs 2023-03-10T01:42:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:42:31.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:31 smithi038 ceph-mon[92422]: pgmap v21128: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 670 KiB/s wr, 11 op/s 2023-03-10T01:42:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38661.540841 secs 2023-03-10T01:42:31.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:42:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:32 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:42:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:32 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:42:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:42:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:42:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:42:32.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:32 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:42:32.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:32 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:42:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:42:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:42:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:42:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:32 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:42:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:32 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:42:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:42:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:42:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:42:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:33 smithi122 ceph-mon[93869]: pgmap v21129: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 386 KiB/s rd, 385 KiB/s wr, 9 op/s 2023-03-10T01:42:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38664.000688 secs 2023-03-10T01:42:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:33 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:42:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:33 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:42:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:33 smithi181 ceph-mon[94189]: pgmap v21129: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 386 KiB/s rd, 385 KiB/s wr, 9 op/s 2023-03-10T01:42:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38664.000688 secs 2023-03-10T01:42:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:33 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:42:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:33 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:42:33.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:33 smithi038 ceph-mon[92422]: pgmap v21129: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 386 KiB/s rd, 385 KiB/s wr, 9 op/s 2023-03-10T01:42:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38664.000688 secs 2023-03-10T01:42:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:33 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:42:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:33 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:42:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39125.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:42:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:35 smithi122 ceph-mon[93869]: pgmap v21130: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 423 KiB/s wr, 9 op/s 2023-03-10T01:42:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:35 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:42:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:35 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:42:35.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:35 smithi122 ceph-mon[93869]: osdmap e1536: 12 total, 12 up, 12 in 2023-03-10T01:42:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:35 smithi181 ceph-mon[94189]: pgmap v21130: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 423 KiB/s wr, 9 op/s 2023-03-10T01:42:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:35 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:42:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:35 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:42:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:35 smithi181 ceph-mon[94189]: osdmap e1536: 12 total, 12 up, 12 in 2023-03-10T01:42:35.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:35 smithi038 ceph-mon[92422]: pgmap v21130: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 423 KiB/s wr, 9 op/s 2023-03-10T01:42:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:35 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:42:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:35 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:42:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:35 smithi038 ceph-mon[92422]: osdmap e1536: 12 total, 12 up, 12 in 2023-03-10T01:42:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38666.540943 secs 2023-03-10T01:42:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:42:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:42:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38666.540943 secs 2023-03-10T01:42:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:42:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:42:36.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38666.540943 secs 2023-03-10T01:42:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:42:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:42:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:37 smithi122 ceph-mon[93869]: pgmap v21132: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 911 KiB/s rd, 641 KiB/s wr, 9 op/s 2023-03-10T01:42:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:42:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:42:37.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:37 smithi181 ceph-mon[94189]: pgmap v21132: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 911 KiB/s rd, 641 KiB/s wr, 9 op/s 2023-03-10T01:42:37.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:42:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:42:37.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:37 smithi038 ceph-mon[92422]: pgmap v21132: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 911 KiB/s rd, 641 KiB/s wr, 9 op/s 2023-03-10T01:42:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:42:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:42:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38669.000746 secs 2023-03-10T01:42:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38669.000746 secs 2023-03-10T01:42:38.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38669.000746 secs 2023-03-10T01:42:38.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39130.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:42:39.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:39 smithi122 ceph-mon[93869]: pgmap v21133: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 884 KiB/s wr, 10 op/s 2023-03-10T01:42:39.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:39 smithi181 ceph-mon[94189]: pgmap v21133: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 884 KiB/s wr, 10 op/s 2023-03-10T01:42:39.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:39 smithi038 ceph-mon[92422]: pgmap v21133: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 884 KiB/s wr, 10 op/s 2023-03-10T01:42:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:41 smithi122 ceph-mon[93869]: pgmap v21134: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 883 KiB/s wr, 11 op/s 2023-03-10T01:42:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38671.540982 secs 2023-03-10T01:42:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:41 smithi181 ceph-mon[94189]: pgmap v21134: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 883 KiB/s wr, 11 op/s 2023-03-10T01:42:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38671.540982 secs 2023-03-10T01:42:41.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:41 smithi038 ceph-mon[92422]: pgmap v21134: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 883 KiB/s wr, 11 op/s 2023-03-10T01:42:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38671.540982 secs 2023-03-10T01:42:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:43 smithi122 ceph-mon[93869]: pgmap v21135: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1000 KiB/s rd, 692 KiB/s wr, 11 op/s 2023-03-10T01:42:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38674.000806 secs 2023-03-10T01:42:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:43 smithi181 ceph-mon[94189]: pgmap v21135: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1000 KiB/s rd, 692 KiB/s wr, 11 op/s 2023-03-10T01:42:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38674.000806 secs 2023-03-10T01:42:43.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:43 smithi038 ceph-mon[92422]: pgmap v21135: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1000 KiB/s rd, 692 KiB/s wr, 11 op/s 2023-03-10T01:42:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38674.000806 secs 2023-03-10T01:42:43.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39135.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:42:45.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:45 smithi122 ceph-mon[93869]: pgmap v21136: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.0 MiB/s wr, 12 op/s 2023-03-10T01:42:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:45 smithi181 ceph-mon[94189]: pgmap v21136: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.0 MiB/s wr, 12 op/s 2023-03-10T01:42:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:45 smithi038 ceph-mon[92422]: pgmap v21136: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.0 MiB/s wr, 12 op/s 2023-03-10T01:42:46.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38676.541040 secs 2023-03-10T01:42:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38676.541040 secs 2023-03-10T01:42:46.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38676.541040 secs 2023-03-10T01:42:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:47 smithi122 ceph-mon[93869]: pgmap v21137: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 934 KiB/s rd, 606 KiB/s wr, 11 op/s 2023-03-10T01:42:47.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:47 smithi181 ceph-mon[94189]: pgmap v21137: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 934 KiB/s rd, 606 KiB/s wr, 11 op/s 2023-03-10T01:42:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:47 smithi038 ceph-mon[92422]: pgmap v21137: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 934 KiB/s rd, 606 KiB/s wr, 11 op/s 2023-03-10T01:42:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:42:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:42:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38679.000889 secs 2023-03-10T01:42:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:42:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:42:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38679.000889 secs 2023-03-10T01:42:48.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:42:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:42:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38679.000889 secs 2023-03-10T01:42:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39140.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:42:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:49 smithi122 ceph-mon[93869]: pgmap v21138: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 749 KiB/s rd, 715 KiB/s wr, 13 op/s 2023-03-10T01:42:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:42:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:42:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:49 smithi181 ceph-mon[94189]: pgmap v21138: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 749 KiB/s rd, 715 KiB/s wr, 13 op/s 2023-03-10T01:42:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:42:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:42:49.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:49 smithi038 ceph-mon[92422]: pgmap v21138: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 749 KiB/s rd, 715 KiB/s wr, 13 op/s 2023-03-10T01:42:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:42:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:42:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:51 smithi122 ceph-mon[93869]: pgmap v21139: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 514 KiB/s rd, 512 KiB/s wr, 9 op/s 2023-03-10T01:42:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38681.541099 secs 2023-03-10T01:42:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:51 smithi181 ceph-mon[94189]: pgmap v21139: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 514 KiB/s rd, 512 KiB/s wr, 9 op/s 2023-03-10T01:42:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38681.541099 secs 2023-03-10T01:42:51.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:51 smithi038 ceph-mon[92422]: pgmap v21139: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 514 KiB/s rd, 512 KiB/s wr, 9 op/s 2023-03-10T01:42:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38681.541099 secs 2023-03-10T01:42:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:42:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:42:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:52 smithi122 ceph-mon[93869]: pgmap v21140: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 860 KiB/s rd, 857 KiB/s wr, 11 op/s 2023-03-10T01:42:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:42:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:42:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:52 smithi181 ceph-mon[94189]: pgmap v21140: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 860 KiB/s rd, 857 KiB/s wr, 11 op/s 2023-03-10T01:42:52.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:42:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:42:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:52 smithi038 ceph-mon[92422]: pgmap v21140: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 860 KiB/s rd, 857 KiB/s wr, 11 op/s 2023-03-10T01:42:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38684.000946 secs 2023-03-10T01:42:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38684.000946 secs 2023-03-10T01:42:53.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38684.000946 secs 2023-03-10T01:42:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39145.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:42:54.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:54 smithi122 ceph-mon[93869]: pgmap v21141: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 858 KiB/s rd, 856 KiB/s wr, 9 op/s 2023-03-10T01:42:54.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:54 smithi181 ceph-mon[94189]: pgmap v21141: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 858 KiB/s rd, 856 KiB/s wr, 9 op/s 2023-03-10T01:42:54.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:54 smithi038 ceph-mon[92422]: pgmap v21141: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 858 KiB/s rd, 856 KiB/s wr, 9 op/s 2023-03-10T01:42:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38686.541190 secs 2023-03-10T01:42:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:42:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:42:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:56 smithi122 ceph-mon[93869]: pgmap v21142: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 553 KiB/s rd, 551 KiB/s wr, 9 op/s 2023-03-10T01:42:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38686.541190 secs 2023-03-10T01:42:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:42:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:42:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:56 smithi181 ceph-mon[94189]: pgmap v21142: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 553 KiB/s rd, 551 KiB/s wr, 9 op/s 2023-03-10T01:42:56.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38686.541190 secs 2023-03-10T01:42:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:42:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:42:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:56 smithi038 ceph-mon[92422]: pgmap v21142: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 553 KiB/s rd, 551 KiB/s wr, 9 op/s 2023-03-10T01:42:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:42:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:42:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:42:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:42:57.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:42:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:42:58.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:58 smithi122 ceph-mon[93869]: pgmap v21143: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 588 KiB/s rd, 584 KiB/s wr, 13 op/s 2023-03-10T01:42:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:42:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38689.001017 secs 2023-03-10T01:42:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:58 smithi181 ceph-mon[94189]: pgmap v21143: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 588 KiB/s rd, 584 KiB/s wr, 13 op/s 2023-03-10T01:42:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:42:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38689.001017 secs 2023-03-10T01:42:58.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:58 smithi038 ceph-mon[92422]: pgmap v21143: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 588 KiB/s rd, 584 KiB/s wr, 13 op/s 2023-03-10T01:42:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:42:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38689.001017 secs 2023-03-10T01:42:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39150.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:43:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:43:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:43:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:00 smithi122 ceph-mon[93869]: pgmap v21144: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 734 KiB/s rd, 823 KiB/s wr, 11 op/s 2023-03-10T01:43:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:43:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:43:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:00 smithi181 ceph-mon[94189]: pgmap v21144: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 734 KiB/s rd, 823 KiB/s wr, 11 op/s 2023-03-10T01:43:00.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:43:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:43:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:00 smithi038 ceph-mon[92422]: pgmap v21144: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 734 KiB/s rd, 823 KiB/s wr, 11 op/s 2023-03-10T01:43:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38691.541242 secs 2023-03-10T01:43:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:43:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:43:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38691.541242 secs 2023-03-10T01:43:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:43:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:43:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38691.541242 secs 2023-03-10T01:43:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:43:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:43:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:02 smithi122 ceph-mon[93869]: pgmap v21145: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 769 KiB/s rd, 868 KiB/s wr, 15 op/s 2023-03-10T01:43:02.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:02 smithi181 ceph-mon[94189]: pgmap v21145: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 769 KiB/s rd, 868 KiB/s wr, 15 op/s 2023-03-10T01:43:02.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:02 smithi038 ceph-mon[92422]: pgmap v21145: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 769 KiB/s rd, 868 KiB/s wr, 15 op/s 2023-03-10T01:43:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38694.001098 secs 2023-03-10T01:43:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38694.001098 secs 2023-03-10T01:43:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38694.001098 secs 2023-03-10T01:43:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39155.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:43:04.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:04 smithi122 ceph-mon[93869]: pgmap v21146: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 810 KiB/s rd, 772 KiB/s wr, 24 op/s 2023-03-10T01:43:04.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:04 smithi181 ceph-mon[94189]: pgmap v21146: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 810 KiB/s rd, 772 KiB/s wr, 24 op/s 2023-03-10T01:43:04.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:04 smithi038 ceph-mon[92422]: pgmap v21146: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 810 KiB/s rd, 772 KiB/s wr, 24 op/s 2023-03-10T01:43:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38696.541304 secs 2023-03-10T01:43:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:06 smithi122 ceph-mon[93869]: pgmap v21147: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 835 KiB/s rd, 798 KiB/s wr, 24 op/s 2023-03-10T01:43:07.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38696.541304 secs 2023-03-10T01:43:07.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:06 smithi181 ceph-mon[94189]: pgmap v21147: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 835 KiB/s rd, 798 KiB/s wr, 24 op/s 2023-03-10T01:43:07.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38696.541304 secs 2023-03-10T01:43:07.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:06 smithi038 ceph-mon[92422]: pgmap v21147: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 835 KiB/s rd, 798 KiB/s wr, 24 op/s 2023-03-10T01:43:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:43:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:43:08.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:43:08.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:43:08.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:43:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:43:08.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39160.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:43:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:08 smithi122 ceph-mon[93869]: pgmap v21148: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 813 KiB/s rd, 775 KiB/s wr, 26 op/s 2023-03-10T01:43:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38699.001186 secs 2023-03-10T01:43:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:43:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:43:09.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:08 smithi181 ceph-mon[94189]: pgmap v21148: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 813 KiB/s rd, 775 KiB/s wr, 26 op/s 2023-03-10T01:43:09.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38699.001186 secs 2023-03-10T01:43:09.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:43:09.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:43:09.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:08 smithi038 ceph-mon[92422]: pgmap v21148: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 813 KiB/s rd, 775 KiB/s wr, 26 op/s 2023-03-10T01:43:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38699.001186 secs 2023-03-10T01:43:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:43:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:43:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:10 smithi122 ceph-mon[93869]: pgmap v21149: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 784 KiB/s rd, 749 KiB/s wr, 24 op/s 2023-03-10T01:43:11.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:10 smithi181 ceph-mon[94189]: pgmap v21149: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 784 KiB/s rd, 749 KiB/s wr, 24 op/s 2023-03-10T01:43:11.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:10 smithi038 ceph-mon[92422]: pgmap v21149: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 784 KiB/s rd, 749 KiB/s wr, 24 op/s 2023-03-10T01:43:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38701.541363 secs 2023-03-10T01:43:12.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38701.541363 secs 2023-03-10T01:43:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38701.541363 secs 2023-03-10T01:43:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:12 smithi122 ceph-mon[93869]: pgmap v21150: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 456 KiB/s rd, 326 KiB/s wr, 24 op/s 2023-03-10T01:43:13.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:12 smithi181 ceph-mon[94189]: pgmap v21150: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 456 KiB/s rd, 326 KiB/s wr, 24 op/s 2023-03-10T01:43:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:12 smithi038 ceph-mon[92422]: pgmap v21150: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 456 KiB/s rd, 326 KiB/s wr, 24 op/s 2023-03-10T01:43:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39165.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:43:14.225 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38704.001289 secs 2023-03-10T01:43:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38704.001289 secs 2023-03-10T01:43:14.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38704.001289 secs 2023-03-10T01:43:15.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:14 smithi122 ceph-mon[93869]: pgmap v21151: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 282 KiB/s wr, 20 op/s 2023-03-10T01:43:15.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:14 smithi181 ceph-mon[94189]: pgmap v21151: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 282 KiB/s wr, 20 op/s 2023-03-10T01:43:15.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:14 smithi038 ceph-mon[92422]: pgmap v21151: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 282 KiB/s wr, 20 op/s 2023-03-10T01:43:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38706.541420 secs 2023-03-10T01:43:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:16 smithi122 ceph-mon[93869]: pgmap v21152: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 325 KiB/s wr, 9 op/s 2023-03-10T01:43:17.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38706.541420 secs 2023-03-10T01:43:17.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:16 smithi181 ceph-mon[94189]: pgmap v21152: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 325 KiB/s wr, 9 op/s 2023-03-10T01:43:17.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38706.541420 secs 2023-03-10T01:43:17.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:16 smithi038 ceph-mon[92422]: pgmap v21152: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 325 KiB/s wr, 9 op/s 2023-03-10T01:43:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39170.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:43:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:18 smithi122 ceph-mon[93869]: pgmap v21153: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 300 KiB/s wr, 10 op/s 2023-03-10T01:43:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38709.001346 secs 2023-03-10T01:43:19.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:18 smithi181 ceph-mon[94189]: pgmap v21153: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 300 KiB/s wr, 10 op/s 2023-03-10T01:43:19.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38709.001346 secs 2023-03-10T01:43:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:18 smithi038 ceph-mon[92422]: pgmap v21153: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 300 KiB/s wr, 10 op/s 2023-03-10T01:43:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38709.001346 secs 2023-03-10T01:43:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:19 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:43:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:19 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:43:20.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:19 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:43:20.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:19 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:43:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:19 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:43:20.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:19 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:43:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:20 smithi122 ceph-mon[93869]: pgmap v21154: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 337 KiB/s rd, 345 KiB/s wr, 9 op/s 2023-03-10T01:43:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:43:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:43:21.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:20 smithi181 ceph-mon[94189]: pgmap v21154: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 337 KiB/s rd, 345 KiB/s wr, 9 op/s 2023-03-10T01:43:21.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:43:21.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:43:21.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:20 smithi038 ceph-mon[92422]: pgmap v21154: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 337 KiB/s rd, 345 KiB/s wr, 9 op/s 2023-03-10T01:43:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:43:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:43:22.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38711.541474 secs 2023-03-10T01:43:22.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38711.541474 secs 2023-03-10T01:43:22.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38711.541474 secs 2023-03-10T01:43:23.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:22 smithi038 ceph-mon[92422]: pgmap v21155: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 705 KiB/s rd, 723 KiB/s wr, 11 op/s 2023-03-10T01:43:23.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:22 smithi122 ceph-mon[93869]: pgmap v21155: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 705 KiB/s rd, 723 KiB/s wr, 11 op/s 2023-03-10T01:43:23.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:22 smithi181 ceph-mon[94189]: pgmap v21155: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 705 KiB/s rd, 723 KiB/s wr, 11 op/s 2023-03-10T01:43:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39175.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:43:24.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38714.001403 secs 2023-03-10T01:43:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38714.001403 secs 2023-03-10T01:43:24.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38714.001403 secs 2023-03-10T01:43:25.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:24 smithi038 ceph-mon[92422]: pgmap v21156: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 704 KiB/s rd, 728 KiB/s wr, 10 op/s 2023-03-10T01:43:25.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:24 smithi122 ceph-mon[93869]: pgmap v21156: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 704 KiB/s rd, 728 KiB/s wr, 10 op/s 2023-03-10T01:43:25.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:24 smithi181 ceph-mon[94189]: pgmap v21156: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 704 KiB/s rd, 728 KiB/s wr, 10 op/s 2023-03-10T01:43:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:26 smithi122 ceph-mon[93869]: osdmap e1537: 12 total, 12 up, 12 in 2023-03-10T01:43:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38716.541560 secs 2023-03-10T01:43:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:26 smithi181 ceph-mon[94189]: osdmap e1537: 12 total, 12 up, 12 in 2023-03-10T01:43:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38716.541560 secs 2023-03-10T01:43:26.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:26 smithi038 ceph-mon[92422]: osdmap e1537: 12 total, 12 up, 12 in 2023-03-10T01:43:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38716.541560 secs 2023-03-10T01:43:27.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:27 smithi122 ceph-mon[93869]: pgmap v21158: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 567 KiB/s rd, 595 KiB/s wr, 12 op/s 2023-03-10T01:43:27.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:27 smithi181 ceph-mon[94189]: pgmap v21158: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 567 KiB/s rd, 595 KiB/s wr, 12 op/s 2023-03-10T01:43:27.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:27 smithi038 ceph-mon[92422]: pgmap v21158: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 567 KiB/s rd, 595 KiB/s wr, 12 op/s 2023-03-10T01:43:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39180.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:43:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:29 smithi122 ceph-mon[93869]: pgmap v21159: 129 pgs: 1 active+clean+snaptrim, 9 active+clean+snaptrim_wait, 119 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 567 KiB/s rd, 596 KiB/s wr, 13 op/s 2023-03-10T01:43:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:29 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38719.001463 secs 2023-03-10T01:43:29.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:29 smithi181 ceph-mon[94189]: pgmap v21159: 129 pgs: 1 active+clean+snaptrim, 9 active+clean+snaptrim_wait, 119 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 567 KiB/s rd, 596 KiB/s wr, 13 op/s 2023-03-10T01:43:29.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:29 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38719.001463 secs 2023-03-10T01:43:29.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:29 smithi038 ceph-mon[92422]: pgmap v21159: 129 pgs: 1 active+clean+snaptrim, 9 active+clean+snaptrim_wait, 119 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 567 KiB/s rd, 596 KiB/s wr, 13 op/s 2023-03-10T01:43:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:29 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38719.001463 secs 2023-03-10T01:43:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:31 smithi122 ceph-mon[93869]: pgmap v21160: 129 pgs: 1 active+clean+snaptrim, 9 active+clean+snaptrim_wait, 119 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 717 KiB/s rd, 734 KiB/s wr, 11 op/s 2023-03-10T01:43:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38721.541616 secs 2023-03-10T01:43:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:31 smithi181 ceph-mon[94189]: pgmap v21160: 129 pgs: 1 active+clean+snaptrim, 9 active+clean+snaptrim_wait, 119 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 717 KiB/s rd, 734 KiB/s wr, 11 op/s 2023-03-10T01:43:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38721.541616 secs 2023-03-10T01:43:31.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:31 smithi038 ceph-mon[92422]: pgmap v21160: 129 pgs: 1 active+clean+snaptrim, 9 active+clean+snaptrim_wait, 119 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 717 KiB/s rd, 734 KiB/s wr, 11 op/s 2023-03-10T01:43:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38721.541616 secs 2023-03-10T01:43:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:43:32.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:43:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:43:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:33 smithi122 ceph-mon[93869]: pgmap v21161: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 311 KiB/s wr, 9 op/s 2023-03-10T01:43:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:43:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:43:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:43:33.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:33 smithi181 ceph-mon[94189]: pgmap v21161: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 311 KiB/s wr, 9 op/s 2023-03-10T01:43:33.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:43:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:43:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:43:33.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:33 smithi038 ceph-mon[92422]: pgmap v21161: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 311 KiB/s wr, 9 op/s 2023-03-10T01:43:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:43:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:43:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:43:33.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39185.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:43:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:34 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38724.001523 secs 2023-03-10T01:43:34.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:34 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38724.001523 secs 2023-03-10T01:43:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:34 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38724.001523 secs 2023-03-10T01:43:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:35 smithi122 ceph-mon[93869]: pgmap v21162: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 306 KiB/s wr, 8 op/s 2023-03-10T01:43:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:35 smithi122 ceph-mon[93869]: osdmap e1538: 12 total, 12 up, 12 in 2023-03-10T01:43:35.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:35 smithi181 ceph-mon[94189]: pgmap v21162: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 306 KiB/s wr, 8 op/s 2023-03-10T01:43:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:35 smithi181 ceph-mon[94189]: osdmap e1538: 12 total, 12 up, 12 in 2023-03-10T01:43:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:35 smithi038 ceph-mon[92422]: pgmap v21162: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 307 KiB/s rd, 306 KiB/s wr, 8 op/s 2023-03-10T01:43:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:35 smithi038 ceph-mon[92422]: osdmap e1538: 12 total, 12 up, 12 in 2023-03-10T01:43:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:43:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:43:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38726.541702 secs 2023-03-10T01:43:36.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:43:36.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:43:36.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38726.541702 secs 2023-03-10T01:43:36.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:43:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:43:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38726.541702 secs 2023-03-10T01:43:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:37 smithi122 ceph-mon[93869]: pgmap v21164: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 767 KiB/s wr, 9 op/s 2023-03-10T01:43:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:43:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:43:37.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:37 smithi181 ceph-mon[94189]: pgmap v21164: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 767 KiB/s wr, 9 op/s 2023-03-10T01:43:37.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:43:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:43:37.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:37 smithi038 ceph-mon[92422]: pgmap v21164: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 767 KiB/s wr, 9 op/s 2023-03-10T01:43:37.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:43:37.501 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:43:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39190.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:43:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:39 smithi122 ceph-mon[93869]: pgmap v21165: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.0 MiB/s wr, 9 op/s 2023-03-10T01:43:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:39 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38729.001689 secs 2023-03-10T01:43:39.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:39 smithi181 ceph-mon[94189]: pgmap v21165: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.0 MiB/s wr, 9 op/s 2023-03-10T01:43:39.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:39 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38729.001689 secs 2023-03-10T01:43:39.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:39 smithi038 ceph-mon[92422]: pgmap v21165: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.0 MiB/s wr, 9 op/s 2023-03-10T01:43:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:39 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38729.001689 secs 2023-03-10T01:43:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:43:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:43:40.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:43:40.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:43:40.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:43:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:43:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:41 smithi122 ceph-mon[93869]: pgmap v21166: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 849 KiB/s wr, 9 op/s 2023-03-10T01:43:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:43:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:43:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38731.541782 secs 2023-03-10T01:43:41.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:41 smithi181 ceph-mon[94189]: pgmap v21166: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 849 KiB/s wr, 9 op/s 2023-03-10T01:43:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:43:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:43:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38731.541782 secs 2023-03-10T01:43:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:41 smithi038 ceph-mon[92422]: pgmap v21166: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 849 KiB/s wr, 9 op/s 2023-03-10T01:43:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:43:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:43:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38731.541782 secs 2023-03-10T01:43:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:43 smithi122 ceph-mon[93869]: pgmap v21167: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 809 KiB/s wr, 10 op/s 2023-03-10T01:43:43.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:43 smithi181 ceph-mon[94189]: pgmap v21167: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 809 KiB/s wr, 10 op/s 2023-03-10T01:43:43.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:43 smithi038 ceph-mon[92422]: pgmap v21167: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 809 KiB/s wr, 10 op/s 2023-03-10T01:43:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39195.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:43:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:44 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38734.001762 secs 2023-03-10T01:43:44.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:44 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38734.001762 secs 2023-03-10T01:43:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:44 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38734.001762 secs 2023-03-10T01:43:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:45 smithi122 ceph-mon[93869]: pgmap v21168: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.2 MiB/s wr, 12 op/s 2023-03-10T01:43:45.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:45 smithi181 ceph-mon[94189]: pgmap v21168: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.2 MiB/s wr, 12 op/s 2023-03-10T01:43:45.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:45 smithi038 ceph-mon[92422]: pgmap v21168: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.2 MiB/s wr, 12 op/s 2023-03-10T01:43:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38736.541841 secs 2023-03-10T01:43:46.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38736.541841 secs 2023-03-10T01:43:46.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38736.541841 secs 2023-03-10T01:43:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:47 smithi122 ceph-mon[93869]: pgmap v21169: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1018 KiB/s rd, 1021 KiB/s wr, 11 op/s 2023-03-10T01:43:47.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:47 smithi181 ceph-mon[94189]: pgmap v21169: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1018 KiB/s rd, 1021 KiB/s wr, 11 op/s 2023-03-10T01:43:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:47 smithi038 ceph-mon[92422]: pgmap v21169: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1018 KiB/s rd, 1021 KiB/s wr, 11 op/s 2023-03-10T01:43:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:43:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:43:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:43:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:43:48.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:43:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:43:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39200.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:43:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:49 smithi122 ceph-mon[93869]: pgmap v21170: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 778 KiB/s rd, 779 KiB/s wr, 12 op/s 2023-03-10T01:43:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:49 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38739.001818 secs 2023-03-10T01:43:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:43:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:43:49.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:49 smithi181 ceph-mon[94189]: pgmap v21170: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 778 KiB/s rd, 779 KiB/s wr, 12 op/s 2023-03-10T01:43:49.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:49 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38739.001818 secs 2023-03-10T01:43:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:43:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:43:49.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:49 smithi038 ceph-mon[92422]: pgmap v21170: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 778 KiB/s rd, 779 KiB/s wr, 12 op/s 2023-03-10T01:43:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:49 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38739.001818 secs 2023-03-10T01:43:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:43:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:43:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:51 smithi122 ceph-mon[93869]: pgmap v21171: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 550 KiB/s wr, 10 op/s 2023-03-10T01:43:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38741.541895 secs 2023-03-10T01:43:51.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:51 smithi181 ceph-mon[94189]: pgmap v21171: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 550 KiB/s wr, 10 op/s 2023-03-10T01:43:51.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38741.541895 secs 2023-03-10T01:43:51.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:51 smithi038 ceph-mon[92422]: pgmap v21171: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 550 KiB/s wr, 10 op/s 2023-03-10T01:43:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38741.541895 secs 2023-03-10T01:43:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:53 smithi122 ceph-mon[93869]: pgmap v21172: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 867 KiB/s wr, 12 op/s 2023-03-10T01:43:53.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:53 smithi181 ceph-mon[94189]: pgmap v21172: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 867 KiB/s wr, 12 op/s 2023-03-10T01:43:53.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:53 smithi038 ceph-mon[92422]: pgmap v21172: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 867 KiB/s wr, 12 op/s 2023-03-10T01:43:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39205.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:43:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:54 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:43:54.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:54 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:43:54.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:54 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:43:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:55 smithi122 ceph-mon[93869]: pgmap v21173: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 920 KiB/s rd, 922 KiB/s wr, 10 op/s 2023-03-10T01:43:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:55 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:43:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:43:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:43:55.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:55 smithi181 ceph-mon[94189]: pgmap v21173: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 920 KiB/s rd, 922 KiB/s wr, 10 op/s 2023-03-10T01:43:55.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:55 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:43:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:43:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:43:55.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:55 smithi038 ceph-mon[92422]: pgmap v21173: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 920 KiB/s rd, 922 KiB/s wr, 10 op/s 2023-03-10T01:43:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:55 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:43:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:43:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:43:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38746.541954 secs 2023-03-10T01:43:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:43:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:43:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38746.541954 secs 2023-03-10T01:43:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:43:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:43:56.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38746.541954 secs 2023-03-10T01:43:56.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:43:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:43:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:57 smithi122 ceph-mon[93869]: pgmap v21174: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 597 KiB/s rd, 595 KiB/s wr, 9 op/s 2023-03-10T01:43:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:43:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:43:57.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:57 smithi181 ceph-mon[94189]: pgmap v21174: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 597 KiB/s rd, 595 KiB/s wr, 9 op/s 2023-03-10T01:43:57.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:43:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:43:57.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:57 smithi038 ceph-mon[92422]: pgmap v21174: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 597 KiB/s rd, 595 KiB/s wr, 9 op/s 2023-03-10T01:43:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:43:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:43:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39210.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:43:59.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:43:59 smithi122 ceph-mon[93869]: pgmap v21175: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 633 KiB/s rd, 628 KiB/s wr, 12 op/s 2023-03-10T01:43:59.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:43:59 smithi181 ceph-mon[94189]: pgmap v21175: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 633 KiB/s rd, 628 KiB/s wr, 12 op/s 2023-03-10T01:43:59.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:43:59 smithi038 ceph-mon[92422]: pgmap v21175: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 633 KiB/s rd, 628 KiB/s wr, 12 op/s 2023-03-10T01:44:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:44:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:44:00.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:44:00.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:44:00.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:44:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:44:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:01 smithi122 ceph-mon[93869]: pgmap v21176: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 815 KiB/s rd, 857 KiB/s wr, 11 op/s 2023-03-10T01:44:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38751.541986 secs 2023-03-10T01:44:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:44:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:44:01.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:01 smithi181 ceph-mon[94189]: pgmap v21176: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 815 KiB/s rd, 857 KiB/s wr, 11 op/s 2023-03-10T01:44:01.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38751.541986 secs 2023-03-10T01:44:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:44:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:44:01.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:01 smithi038 ceph-mon[92422]: pgmap v21176: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 815 KiB/s rd, 857 KiB/s wr, 11 op/s 2023-03-10T01:44:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38751.541986 secs 2023-03-10T01:44:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:44:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:44:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:03 smithi122 ceph-mon[93869]: pgmap v21177: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 931 KiB/s rd, 982 KiB/s wr, 15 op/s 2023-03-10T01:44:03.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:03 smithi181 ceph-mon[94189]: pgmap v21177: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 931 KiB/s rd, 982 KiB/s wr, 15 op/s 2023-03-10T01:44:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:03 smithi038 ceph-mon[92422]: pgmap v21177: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 931 KiB/s rd, 982 KiB/s wr, 15 op/s 2023-03-10T01:44:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39215.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:44:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:05 smithi122 ceph-mon[93869]: pgmap v21178: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 709 KiB/s rd, 709 KiB/s wr, 25 op/s 2023-03-10T01:44:05.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:05 smithi181 ceph-mon[94189]: pgmap v21178: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 709 KiB/s rd, 709 KiB/s wr, 25 op/s 2023-03-10T01:44:05.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:05 smithi038 ceph-mon[92422]: pgmap v21178: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 709 KiB/s rd, 709 KiB/s wr, 25 op/s 2023-03-10T01:44:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38756.542043 secs 2023-03-10T01:44:06.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38756.542043 secs 2023-03-10T01:44:06.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38756.542043 secs 2023-03-10T01:44:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:07 smithi122 ceph-mon[93869]: pgmap v21179: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 829 KiB/s rd, 828 KiB/s wr, 24 op/s 2023-03-10T01:44:07.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:07 smithi181 ceph-mon[94189]: pgmap v21179: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 829 KiB/s rd, 828 KiB/s wr, 24 op/s 2023-03-10T01:44:07.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:07 smithi038 ceph-mon[92422]: pgmap v21179: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 829 KiB/s rd, 828 KiB/s wr, 24 op/s 2023-03-10T01:44:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:44:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:44:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:44:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:44:08.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:44:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:44:08.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39220.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:44:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:09 smithi122 ceph-mon[93869]: pgmap v21180: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 997 KiB/s rd, 995 KiB/s wr, 28 op/s 2023-03-10T01:44:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:44:09.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:44:09.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:09 smithi181 ceph-mon[94189]: pgmap v21180: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 997 KiB/s rd, 995 KiB/s wr, 28 op/s 2023-03-10T01:44:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:44:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:44:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:09 smithi038 ceph-mon[92422]: pgmap v21180: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 997 KiB/s rd, 995 KiB/s wr, 28 op/s 2023-03-10T01:44:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:44:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:44:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:11 smithi122 ceph-mon[93869]: pgmap v21181: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 961 KiB/s rd, 962 KiB/s wr, 25 op/s 2023-03-10T01:44:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38761.542126 secs 2023-03-10T01:44:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:11 smithi181 ceph-mon[94189]: pgmap v21181: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 961 KiB/s rd, 962 KiB/s wr, 25 op/s 2023-03-10T01:44:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38761.542126 secs 2023-03-10T01:44:11.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:11 smithi038 ceph-mon[92422]: pgmap v21181: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 961 KiB/s rd, 962 KiB/s wr, 25 op/s 2023-03-10T01:44:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38761.542126 secs 2023-03-10T01:44:13.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:13 smithi181 ceph-mon[94189]: pgmap v21182: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 741 KiB/s rd, 696 KiB/s wr, 26 op/s 2023-03-10T01:44:13.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:13 smithi038 ceph-mon[92422]: pgmap v21182: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 741 KiB/s rd, 696 KiB/s wr, 26 op/s 2023-03-10T01:44:13.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:13 smithi122 ceph-mon[93869]: pgmap v21182: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 741 KiB/s rd, 696 KiB/s wr, 26 op/s 2023-03-10T01:44:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39225.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:44:15.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:15 smithi038 ceph-mon[92422]: pgmap v21183: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 626 KiB/s rd, 571 KiB/s wr, 22 op/s 2023-03-10T01:44:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:15 smithi122 ceph-mon[93869]: pgmap v21183: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 626 KiB/s rd, 571 KiB/s wr, 22 op/s 2023-03-10T01:44:15.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:15 smithi181 ceph-mon[94189]: pgmap v21183: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 626 KiB/s rd, 571 KiB/s wr, 22 op/s 2023-03-10T01:44:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38766.542185 secs 2023-03-10T01:44:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:44:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:44:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38766.542185 secs 2023-03-10T01:44:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:44:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:44:16.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38766.542185 secs 2023-03-10T01:44:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:44:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:44:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:17 smithi122 ceph-mon[93869]: pgmap v21184: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 530 KiB/s rd, 528 KiB/s wr, 10 op/s 2023-03-10T01:44:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:44:17.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:44:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:17 smithi181 ceph-mon[94189]: pgmap v21184: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 530 KiB/s rd, 528 KiB/s wr, 10 op/s 2023-03-10T01:44:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:44:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:44:17.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:17 smithi038 ceph-mon[92422]: pgmap v21184: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 530 KiB/s rd, 528 KiB/s wr, 10 op/s 2023-03-10T01:44:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:44:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:44:18.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39230.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:44:19.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:19 smithi122 ceph-mon[93869]: pgmap v21185: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 355 KiB/s rd, 352 KiB/s wr, 12 op/s 2023-03-10T01:44:19.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:19 smithi181 ceph-mon[94189]: pgmap v21185: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 355 KiB/s rd, 352 KiB/s wr, 12 op/s 2023-03-10T01:44:19.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:19 smithi038 ceph-mon[92422]: pgmap v21185: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 355 KiB/s rd, 352 KiB/s wr, 12 op/s 2023-03-10T01:44:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:21 smithi122 ceph-mon[93869]: pgmap v21186: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 196 KiB/s wr, 11 op/s 2023-03-10T01:44:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38771.542240 secs 2023-03-10T01:44:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:44:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:44:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:21 smithi122 ceph-mon[93869]: osdmap e1539: 12 total, 12 up, 12 in 2023-03-10T01:44:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:21 smithi181 ceph-mon[94189]: pgmap v21186: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 196 KiB/s wr, 11 op/s 2023-03-10T01:44:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38771.542240 secs 2023-03-10T01:44:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:44:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:44:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:21 smithi181 ceph-mon[94189]: osdmap e1539: 12 total, 12 up, 12 in 2023-03-10T01:44:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:21 smithi038 ceph-mon[92422]: pgmap v21186: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 188 KiB/s rd, 196 KiB/s wr, 11 op/s 2023-03-10T01:44:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38771.542240 secs 2023-03-10T01:44:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:44:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:44:21.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:21 smithi038 ceph-mon[92422]: osdmap e1539: 12 total, 12 up, 12 in 2023-03-10T01:44:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:23 smithi122 ceph-mon[93869]: pgmap v21188: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 289 KiB/s wr, 13 op/s 2023-03-10T01:44:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:23 smithi181 ceph-mon[94189]: pgmap v21188: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 289 KiB/s wr, 13 op/s 2023-03-10T01:44:23.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:23 smithi038 ceph-mon[92422]: pgmap v21188: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 289 KiB/s wr, 13 op/s 2023-03-10T01:44:23.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39235.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:44:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:25 smithi122 ceph-mon[93869]: pgmap v21189: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 471 KiB/s rd, 493 KiB/s wr, 13 op/s 2023-03-10T01:44:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:25 smithi181 ceph-mon[94189]: pgmap v21189: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 471 KiB/s rd, 493 KiB/s wr, 13 op/s 2023-03-10T01:44:25.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:25 smithi038 ceph-mon[92422]: pgmap v21189: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 471 KiB/s rd, 493 KiB/s wr, 13 op/s 2023-03-10T01:44:26.725 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38776.542300 secs 2023-03-10T01:44:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38776.542300 secs 2023-03-10T01:44:26.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38776.542300 secs 2023-03-10T01:44:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:27 smithi122 ceph-mon[93869]: pgmap v21190: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 471 KiB/s rd, 493 KiB/s wr, 13 op/s 2023-03-10T01:44:27.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:27 smithi181 ceph-mon[94189]: pgmap v21190: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 471 KiB/s rd, 493 KiB/s wr, 13 op/s 2023-03-10T01:44:27.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:27 smithi038 ceph-mon[92422]: pgmap v21190: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 471 KiB/s rd, 493 KiB/s wr, 13 op/s 2023-03-10T01:44:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:28 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:44:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:28 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:44:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:28 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:44:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:28 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:44:28.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:28 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:44:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:28 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:44:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39240.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:44:29.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:29 smithi122 ceph-mon[93869]: pgmap v21191: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 532 KiB/s wr, 13 op/s 2023-03-10T01:44:29.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:29 smithi181 ceph-mon[94189]: pgmap v21191: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 532 KiB/s wr, 13 op/s 2023-03-10T01:44:29.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:29 smithi038 ceph-mon[92422]: pgmap v21191: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 532 KiB/s wr, 13 op/s 2023-03-10T01:44:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:30 smithi122 ceph-mon[93869]: pgmap v21192: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 702 KiB/s rd, 711 KiB/s wr, 11 op/s 2023-03-10T01:44:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:30 smithi122 ceph-mon[93869]: osdmap e1540: 12 total, 12 up, 12 in 2023-03-10T01:44:31.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:30 smithi181 ceph-mon[94189]: pgmap v21192: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 702 KiB/s rd, 711 KiB/s wr, 11 op/s 2023-03-10T01:44:31.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:30 smithi181 ceph-mon[94189]: osdmap e1540: 12 total, 12 up, 12 in 2023-03-10T01:44:31.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:30 smithi038 ceph-mon[92422]: pgmap v21192: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 702 KiB/s rd, 711 KiB/s wr, 11 op/s 2023-03-10T01:44:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:30 smithi038 ceph-mon[92422]: osdmap e1540: 12 total, 12 up, 12 in 2023-03-10T01:44:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38781.542375 secs 2023-03-10T01:44:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38781.542375 secs 2023-03-10T01:44:32.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38781.542375 secs 2023-03-10T01:44:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:32 smithi122 ceph-mon[93869]: pgmap v21194: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 436 KiB/s wr, 8 op/s 2023-03-10T01:44:33.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:44:33.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:32 smithi181 ceph-mon[94189]: pgmap v21194: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 436 KiB/s wr, 8 op/s 2023-03-10T01:44:33.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:44:33.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:32 smithi038 ceph-mon[92422]: pgmap v21194: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 436 KiB/s wr, 8 op/s 2023-03-10T01:44:33.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:44:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39245.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:44:35.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:34 smithi122 ceph-mon[93869]: pgmap v21195: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 286 KiB/s wr, 9 op/s 2023-03-10T01:44:35.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:34 smithi181 ceph-mon[94189]: pgmap v21195: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 286 KiB/s wr, 9 op/s 2023-03-10T01:44:35.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:34 smithi038 ceph-mon[92422]: pgmap v21195: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 286 KiB/s wr, 9 op/s 2023-03-10T01:44:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38786.542439 secs 2023-03-10T01:44:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:36 smithi122 ceph-mon[93869]: pgmap v21196: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 286 KiB/s wr, 9 op/s 2023-03-10T01:44:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:44:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:44:36.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:44:36.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:44:36.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:44:36.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:44:36.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:44:36.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:44:36.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:44:36.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:44:36.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:44:36.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:36 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-10T01:44:36.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:36 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-10T01:44:36.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:44:36.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:44:36.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:44:36.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:44:36.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:44:36.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:44:36.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:44:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38786.542439 secs 2023-03-10T01:44:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:36 smithi181 ceph-mon[94189]: pgmap v21196: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 286 KiB/s wr, 9 op/s 2023-03-10T01:44:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:44:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:44:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:44:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:44:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:44:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:44:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:44:36.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:44:36.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:44:36.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:44:36.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:44:36.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:36 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-10T01:44:36.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:36 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-10T01:44:36.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:44:36.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:44:36.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:44:36.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:44:36.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:44:36.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:44:36.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:44:36.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38786.542439 secs 2023-03-10T01:44:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:36 smithi038 ceph-mon[92422]: pgmap v21196: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 286 KiB/s wr, 9 op/s 2023-03-10T01:44:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:44:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:44:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:44:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:44:37.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:44:37.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:44:37.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:44:37.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:44:37.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:44:37.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:44:37.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:44:37.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:36 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-10T01:44:37.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:36 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-10T01:44:37.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:44:37.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:44:37.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:44:37.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:44:37.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:44:37.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:44:37.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:44:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:37 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-10T01:44:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:37 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-10T01:44:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:44:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:44:38.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:37 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-10T01:44:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:37 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-10T01:44:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:44:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:44:38.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:37 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-10T01:44:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:37 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-10T01:44:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:44:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:44:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39250.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:44:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:38 smithi122 ceph-mon[93869]: pgmap v21197: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 451 KiB/s wr, 9 op/s 2023-03-10T01:44:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38789.002673 secs 2023-03-10T01:44:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:38 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:44:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:38 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:44:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:38 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:44:39.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:38 smithi181 ceph-mon[94189]: pgmap v21197: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 451 KiB/s wr, 9 op/s 2023-03-10T01:44:39.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38789.002673 secs 2023-03-10T01:44:39.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:38 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:44:39.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:38 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:44:39.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:38 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:44:39.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:38 smithi038 ceph-mon[92422]: pgmap v21197: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 451 KiB/s wr, 9 op/s 2023-03-10T01:44:39.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38789.002673 secs 2023-03-10T01:44:39.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:38 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:44:39.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:38 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:44:39.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:38 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:44:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:39 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:44:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:39 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:44:40.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:39 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:44:40.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:39 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:44:40.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:39 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:44:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:39 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:44:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:40 smithi122 ceph-mon[93869]: pgmap v21198: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 259 KiB/s wr, 9 op/s 2023-03-10T01:44:41.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:40 smithi181 ceph-mon[94189]: pgmap v21198: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 259 KiB/s wr, 9 op/s 2023-03-10T01:44:41.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:40 smithi038 ceph-mon[92422]: pgmap v21198: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 259 KiB/s wr, 9 op/s 2023-03-10T01:44:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38791.542521 secs 2023-03-10T01:44:42.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38791.542521 secs 2023-03-10T01:44:42.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38791.542521 secs 2023-03-10T01:44:43.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:42 smithi122 ceph-mon[93869]: pgmap v21199: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 390 KiB/s wr, 10 op/s 2023-03-10T01:44:43.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:42 smithi181 ceph-mon[94189]: pgmap v21199: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 390 KiB/s wr, 10 op/s 2023-03-10T01:44:43.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:42 smithi038 ceph-mon[92422]: pgmap v21199: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 390 KiB/s wr, 10 op/s 2023-03-10T01:44:43.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39255.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:44:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38794.002763 secs 2023-03-10T01:44:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38794.002763 secs 2023-03-10T01:44:44.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38794.002763 secs 2023-03-10T01:44:45.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:44 smithi122 ceph-mon[93869]: pgmap v21200: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 502 KiB/s wr, 9 op/s 2023-03-10T01:44:45.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:44 smithi181 ceph-mon[94189]: pgmap v21200: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 502 KiB/s wr, 9 op/s 2023-03-10T01:44:45.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:44 smithi038 ceph-mon[92422]: pgmap v21200: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 502 KiB/s wr, 9 op/s 2023-03-10T01:44:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38796.542606 secs 2023-03-10T01:44:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:46 smithi122 ceph-mon[93869]: pgmap v21201: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 497 KiB/s rd, 458 KiB/s wr, 8 op/s 2023-03-10T01:44:47.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38796.542606 secs 2023-03-10T01:44:47.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:46 smithi181 ceph-mon[94189]: pgmap v21201: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 497 KiB/s rd, 458 KiB/s wr, 8 op/s 2023-03-10T01:44:47.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38796.542606 secs 2023-03-10T01:44:47.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:46 smithi038 ceph-mon[92422]: pgmap v21201: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 497 KiB/s rd, 458 KiB/s wr, 8 op/s 2023-03-10T01:44:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39260.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:44:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:48 smithi122 ceph-mon[93869]: pgmap v21202: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 542 KiB/s rd, 502 KiB/s wr, 11 op/s 2023-03-10T01:44:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:44:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:44:49.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:48 smithi181 ceph-mon[94189]: pgmap v21202: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 542 KiB/s rd, 502 KiB/s wr, 11 op/s 2023-03-10T01:44:49.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:44:49.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:44:49.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:48 smithi038 ceph-mon[92422]: pgmap v21202: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 542 KiB/s rd, 502 KiB/s wr, 11 op/s 2023-03-10T01:44:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:44:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:44:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:49 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:44:50.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:49 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:44:50.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:49 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:44:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:50 smithi122 ceph-mon[93869]: pgmap v21203: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 561 KiB/s rd, 547 KiB/s wr, 9 op/s 2023-03-10T01:44:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:50 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:44:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:50 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:44:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:50 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:44:51.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:50 smithi181 ceph-mon[94189]: pgmap v21203: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 561 KiB/s rd, 547 KiB/s wr, 9 op/s 2023-03-10T01:44:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:50 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:44:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:50 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:44:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:50 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:44:51.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:50 smithi038 ceph-mon[92422]: pgmap v21203: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 561 KiB/s rd, 547 KiB/s wr, 9 op/s 2023-03-10T01:44:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:50 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:44:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:50 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:44:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:50 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:44:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38801.542662 secs 2023-03-10T01:44:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:44:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:44:52.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38801.542662 secs 2023-03-10T01:44:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:44:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:44:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38801.542662 secs 2023-03-10T01:44:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:44:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:44:53.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:52 smithi181 ceph-mon[94189]: pgmap v21204: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 890 KiB/s rd, 888 KiB/s wr, 11 op/s 2023-03-10T01:44:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:44:53.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:44:53.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:52 smithi038 ceph-mon[92422]: pgmap v21204: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 890 KiB/s rd, 888 KiB/s wr, 11 op/s 2023-03-10T01:44:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:44:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:44:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:52 smithi122 ceph-mon[93869]: pgmap v21204: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 890 KiB/s rd, 888 KiB/s wr, 11 op/s 2023-03-10T01:44:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:44:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:44:53.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39265.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:44:54.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:53 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 38804.002878 secs 2023-03-10T01:44:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:53 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 38804.002878 secs 2023-03-10T01:44:54.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:53 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 38804.002878 secs 2023-03-10T01:44:55.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:54 smithi038 ceph-mon[92422]: pgmap v21205: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 818 KiB/s rd, 793 KiB/s wr, 11 op/s 2023-03-10T01:44:55.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:54 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:44:55.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:54 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:44:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:54 smithi122 ceph-mon[93869]: pgmap v21205: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 818 KiB/s rd, 793 KiB/s wr, 11 op/s 2023-03-10T01:44:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:54 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:44:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:54 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:44:55.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:54 smithi181 ceph-mon[94189]: pgmap v21205: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 818 KiB/s rd, 793 KiB/s wr, 11 op/s 2023-03-10T01:44:55.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:54 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:44:55.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:54 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:44:56.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:44:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:44:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:44:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:44:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:44:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:44:57.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38806.542721 secs 2023-03-10T01:44:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:56 smithi038 ceph-mon[92422]: pgmap v21206: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 702 KiB/s rd, 666 KiB/s wr, 10 op/s 2023-03-10T01:44:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:44:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:44:57.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38806.542721 secs 2023-03-10T01:44:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:56 smithi122 ceph-mon[93869]: pgmap v21206: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 702 KiB/s rd, 666 KiB/s wr, 10 op/s 2023-03-10T01:44:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:44:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:44:57.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38806.542721 secs 2023-03-10T01:44:57.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:56 smithi181 ceph-mon[94189]: pgmap v21206: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 702 KiB/s rd, 666 KiB/s wr, 10 op/s 2023-03-10T01:44:57.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:44:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:44:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39270.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:44:59.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:58 smithi122 ceph-mon[93869]: pgmap v21207: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 810 KiB/s rd, 687 KiB/s wr, 14 op/s 2023-03-10T01:44:59.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:44:58 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 38809.002961 secs 2023-03-10T01:44:59.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:58 smithi181 ceph-mon[94189]: pgmap v21207: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 810 KiB/s rd, 687 KiB/s wr, 14 op/s 2023-03-10T01:44:59.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:44:58 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 38809.002961 secs 2023-03-10T01:44:59.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:58 smithi038 ceph-mon[92422]: pgmap v21207: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 810 KiB/s rd, 687 KiB/s wr, 14 op/s 2023-03-10T01:44:59.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:44:58 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 38809.002961 secs 2023-03-10T01:45:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:45:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:45:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:00 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:45:00.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:45:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:45:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:00 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:45:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:45:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:45:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:00 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:45:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:01 smithi122 ceph-mon[93869]: pgmap v21208: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 826 KiB/s rd, 705 KiB/s wr, 11 op/s 2023-03-10T01:45:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:45:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:45:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38811.542810 secs 2023-03-10T01:45:01.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:01 smithi181 ceph-mon[94189]: pgmap v21208: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 826 KiB/s rd, 705 KiB/s wr, 11 op/s 2023-03-10T01:45:01.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:45:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:45:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38811.542810 secs 2023-03-10T01:45:01.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:01 smithi038 ceph-mon[92422]: pgmap v21208: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 826 KiB/s rd, 705 KiB/s wr, 11 op/s 2023-03-10T01:45:01.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:45:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:45:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38811.542810 secs 2023-03-10T01:45:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:03 smithi122 ceph-mon[93869]: pgmap v21209: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 735 KiB/s rd, 588 KiB/s wr, 13 op/s 2023-03-10T01:45:03.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:03 smithi181 ceph-mon[94189]: pgmap v21209: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 735 KiB/s rd, 588 KiB/s wr, 13 op/s 2023-03-10T01:45:03.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:03 smithi038 ceph-mon[92422]: pgmap v21209: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 735 KiB/s rd, 588 KiB/s wr, 13 op/s 2023-03-10T01:45:03.793 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39275.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:45:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:04 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38814.003025 secs 2023-03-10T01:45:04.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:04 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38814.003025 secs 2023-03-10T01:45:04.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:04 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38814.003025 secs 2023-03-10T01:45:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:05 smithi122 ceph-mon[93869]: pgmap v21210: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 913 KiB/s wr, 14 op/s 2023-03-10T01:45:05.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:05 smithi181 ceph-mon[94189]: pgmap v21210: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 913 KiB/s wr, 14 op/s 2023-03-10T01:45:05.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:05 smithi038 ceph-mon[92422]: pgmap v21210: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 913 KiB/s wr, 14 op/s 2023-03-10T01:45:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38816.542869 secs 2023-03-10T01:45:06.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38816.542869 secs 2023-03-10T01:45:06.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38816.542869 secs 2023-03-10T01:45:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:07 smithi122 ceph-mon[93869]: pgmap v21211: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 961 KiB/s rd, 862 KiB/s wr, 13 op/s 2023-03-10T01:45:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:45:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:45:07.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:07 smithi181 ceph-mon[94189]: pgmap v21211: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 961 KiB/s rd, 862 KiB/s wr, 13 op/s 2023-03-10T01:45:07.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:45:07.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:45:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:07 smithi038 ceph-mon[92422]: pgmap v21211: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 961 KiB/s rd, 862 KiB/s wr, 13 op/s 2023-03-10T01:45:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:45:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:45:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:45:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:45:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:45:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:45:08.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:45:08.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:45:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39280.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:45:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:09 smithi122 ceph-mon[93869]: pgmap v21212: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 984 KiB/s rd, 917 KiB/s wr, 18 op/s 2023-03-10T01:45:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:09 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38819.003139 secs 2023-03-10T01:45:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:45:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:45:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:09 smithi181 ceph-mon[94189]: pgmap v21212: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 984 KiB/s rd, 917 KiB/s wr, 18 op/s 2023-03-10T01:45:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:09 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38819.003139 secs 2023-03-10T01:45:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:45:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:45:09.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:09 smithi038 ceph-mon[92422]: pgmap v21212: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 984 KiB/s rd, 917 KiB/s wr, 18 op/s 2023-03-10T01:45:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:09 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38819.003139 secs 2023-03-10T01:45:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:45:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:45:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:11 smithi122 ceph-mon[93869]: pgmap v21213: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 884 KiB/s rd, 918 KiB/s wr, 25 op/s 2023-03-10T01:45:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:11 smithi122 ceph-mon[93869]: osdmap e1541: 12 total, 12 up, 12 in 2023-03-10T01:45:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38821.542934 secs 2023-03-10T01:45:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:11 smithi181 ceph-mon[94189]: pgmap v21213: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 884 KiB/s rd, 918 KiB/s wr, 25 op/s 2023-03-10T01:45:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:11 smithi181 ceph-mon[94189]: osdmap e1541: 12 total, 12 up, 12 in 2023-03-10T01:45:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38821.542934 secs 2023-03-10T01:45:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:11 smithi038 ceph-mon[92422]: pgmap v21213: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 884 KiB/s rd, 918 KiB/s wr, 25 op/s 2023-03-10T01:45:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:11 smithi038 ceph-mon[92422]: osdmap e1541: 12 total, 12 up, 12 in 2023-03-10T01:45:11.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38821.542934 secs 2023-03-10T01:45:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:13 smithi122 ceph-mon[93869]: pgmap v21215: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 30 op/s 2023-03-10T01:45:13.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:13 smithi181 ceph-mon[94189]: pgmap v21215: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 30 op/s 2023-03-10T01:45:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:13 smithi038 ceph-mon[92422]: pgmap v21215: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.4 MiB/s wr, 30 op/s 2023-03-10T01:45:13.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39285.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:45:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:14 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38824.003200 secs 2023-03-10T01:45:14.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:14 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38824.003200 secs 2023-03-10T01:45:14.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:14 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38824.003200 secs 2023-03-10T01:45:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:15 smithi122 ceph-mon[93869]: pgmap v21216: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 546 KiB/s rd, 588 KiB/s wr, 25 op/s 2023-03-10T01:45:15.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:15 smithi181 ceph-mon[94189]: pgmap v21216: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 546 KiB/s rd, 588 KiB/s wr, 25 op/s 2023-03-10T01:45:15.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:15 smithi038 ceph-mon[92422]: pgmap v21216: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 546 KiB/s rd, 588 KiB/s wr, 25 op/s 2023-03-10T01:45:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:45:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:45:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38826.542985 secs 2023-03-10T01:45:16.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:45:16.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:45:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38826.542985 secs 2023-03-10T01:45:16.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:45:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:45:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38826.542985 secs 2023-03-10T01:45:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:17 smithi122 ceph-mon[93869]: pgmap v21217: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 529 KiB/s rd, 570 KiB/s wr, 24 op/s 2023-03-10T01:45:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:45:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:45:17.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:17 smithi181 ceph-mon[94189]: pgmap v21217: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 529 KiB/s rd, 570 KiB/s wr, 24 op/s 2023-03-10T01:45:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:45:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:45:17.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:17 smithi038 ceph-mon[92422]: pgmap v21217: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 529 KiB/s rd, 570 KiB/s wr, 24 op/s 2023-03-10T01:45:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:45:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:45:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39290.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:45:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:19 smithi122 ceph-mon[93869]: pgmap v21218: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 537 KiB/s rd, 529 KiB/s wr, 22 op/s 2023-03-10T01:45:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:19 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38829.003312 secs 2023-03-10T01:45:19.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:19 smithi181 ceph-mon[94189]: pgmap v21218: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 537 KiB/s rd, 529 KiB/s wr, 22 op/s 2023-03-10T01:45:19.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:19 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38829.003312 secs 2023-03-10T01:45:19.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:19 smithi038 ceph-mon[92422]: pgmap v21218: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 537 KiB/s rd, 529 KiB/s wr, 22 op/s 2023-03-10T01:45:19.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:19 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38829.003312 secs 2023-03-10T01:45:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:20 smithi122 ceph-mon[93869]: pgmap v21219: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 529 KiB/s rd, 503 KiB/s wr, 9 op/s 2023-03-10T01:45:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:20 smithi122 ceph-mon[93869]: osdmap e1542: 12 total, 12 up, 12 in 2023-03-10T01:45:21.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:20 smithi181 ceph-mon[94189]: pgmap v21219: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 529 KiB/s rd, 503 KiB/s wr, 9 op/s 2023-03-10T01:45:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:20 smithi181 ceph-mon[94189]: osdmap e1542: 12 total, 12 up, 12 in 2023-03-10T01:45:21.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:20 smithi038 ceph-mon[92422]: pgmap v21219: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 529 KiB/s rd, 503 KiB/s wr, 9 op/s 2023-03-10T01:45:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:20 smithi038 ceph-mon[92422]: osdmap e1542: 12 total, 12 up, 12 in 2023-03-10T01:45:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38831.543001 secs 2023-03-10T01:45:22.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38831.543001 secs 2023-03-10T01:45:22.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38831.543001 secs 2023-03-10T01:45:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:22 smithi122 ceph-mon[93869]: pgmap v21221: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 217 KiB/s wr, 8 op/s 2023-03-10T01:45:23.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:22 smithi181 ceph-mon[94189]: pgmap v21221: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 217 KiB/s wr, 8 op/s 2023-03-10T01:45:23.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:22 smithi038 ceph-mon[92422]: pgmap v21221: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 243 KiB/s rd, 217 KiB/s wr, 8 op/s 2023-03-10T01:45:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39295.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:45:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38834.003373 secs 2023-03-10T01:45:24.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38834.003373 secs 2023-03-10T01:45:24.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38834.003373 secs 2023-03-10T01:45:25.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:24 smithi122 ceph-mon[93869]: pgmap v21222: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 233 KiB/s wr, 9 op/s 2023-03-10T01:45:25.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:24 smithi181 ceph-mon[94189]: pgmap v21222: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 233 KiB/s wr, 9 op/s 2023-03-10T01:45:25.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:24 smithi038 ceph-mon[92422]: pgmap v21222: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 233 KiB/s wr, 9 op/s 2023-03-10T01:45:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38836.543086 secs 2023-03-10T01:45:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:26 smithi122 ceph-mon[93869]: pgmap v21223: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 233 KiB/s wr, 9 op/s 2023-03-10T01:45:27.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38836.543086 secs 2023-03-10T01:45:27.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:26 smithi181 ceph-mon[94189]: pgmap v21223: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 233 KiB/s wr, 9 op/s 2023-03-10T01:45:27.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38836.543086 secs 2023-03-10T01:45:27.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:26 smithi038 ceph-mon[92422]: pgmap v21223: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 233 KiB/s wr, 9 op/s 2023-03-10T01:45:28.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39300.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:45:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:28 smithi122 ceph-mon[93869]: pgmap v21224: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 248 KiB/s wr, 9 op/s 2023-03-10T01:45:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38839.003465 secs 2023-03-10T01:45:29.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:28 smithi181 ceph-mon[94189]: pgmap v21224: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 248 KiB/s wr, 9 op/s 2023-03-10T01:45:29.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38839.003465 secs 2023-03-10T01:45:29.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:28 smithi038 ceph-mon[92422]: pgmap v21224: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 248 KiB/s wr, 9 op/s 2023-03-10T01:45:29.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38839.003465 secs 2023-03-10T01:45:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:30 smithi038 ceph-mon[92422]: pgmap v21225: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 440 KiB/s wr, 9 op/s 2023-03-10T01:45:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:30 smithi122 ceph-mon[93869]: pgmap v21225: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 440 KiB/s wr, 9 op/s 2023-03-10T01:45:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:30 smithi181 ceph-mon[94189]: pgmap v21225: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 440 KiB/s wr, 9 op/s 2023-03-10T01:45:32.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38841.543145 secs 2023-03-10T01:45:32.475 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38841.543145 secs 2023-03-10T01:45:32.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38841.543145 secs 2023-03-10T01:45:33.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:32 smithi038 ceph-mon[92422]: pgmap v21226: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 436 KiB/s rd, 393 KiB/s wr, 10 op/s 2023-03-10T01:45:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:32 smithi122 ceph-mon[93869]: pgmap v21226: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 436 KiB/s rd, 393 KiB/s wr, 10 op/s 2023-03-10T01:45:33.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:32 smithi181 ceph-mon[94189]: pgmap v21226: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 436 KiB/s rd, 393 KiB/s wr, 10 op/s 2023-03-10T01:45:33.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39305.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:45:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:34 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38844.003524 secs 2023-03-10T01:45:34.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:34 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38844.003524 secs 2023-03-10T01:45:34.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:34 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38844.003524 secs 2023-03-10T01:45:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:35 smithi122 ceph-mon[93869]: pgmap v21227: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 386 KiB/s wr, 8 op/s 2023-03-10T01:45:35.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:35 smithi181 ceph-mon[94189]: pgmap v21227: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 386 KiB/s wr, 8 op/s 2023-03-10T01:45:35.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:35 smithi038 ceph-mon[92422]: pgmap v21227: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 386 KiB/s wr, 8 op/s 2023-03-10T01:45:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:45:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:45:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38846.543229 secs 2023-03-10T01:45:36.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:45:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:45:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38846.543229 secs 2023-03-10T01:45:36.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:45:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:45:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38846.543229 secs 2023-03-10T01:45:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:37 smithi122 ceph-mon[93869]: pgmap v21228: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 372 KiB/s wr, 7 op/s 2023-03-10T01:45:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:45:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:45:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:45:37.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:37 smithi181 ceph-mon[94189]: pgmap v21228: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 372 KiB/s wr, 7 op/s 2023-03-10T01:45:37.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:45:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:45:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:45:37.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:37 smithi038 ceph-mon[92422]: pgmap v21228: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 372 KiB/s wr, 7 op/s 2023-03-10T01:45:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:45:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:45:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:45:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:38 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:45:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:38 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:45:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:38 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:45:38.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:38 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:45:38.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:38 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:45:38.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:38 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:45:38.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:38 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:45:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:38 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:45:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:38 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:45:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39310.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:45:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:39 smithi122 ceph-mon[93869]: pgmap v21229: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 372 KiB/s wr, 10 op/s 2023-03-10T01:45:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:39 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38849.003581 secs 2023-03-10T01:45:39.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:39 smithi181 ceph-mon[94189]: pgmap v21229: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 372 KiB/s wr, 10 op/s 2023-03-10T01:45:39.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:39 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38849.003581 secs 2023-03-10T01:45:39.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:39 smithi038 ceph-mon[92422]: pgmap v21229: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 372 KiB/s wr, 10 op/s 2023-03-10T01:45:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:39 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38849.003581 secs 2023-03-10T01:45:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:41 smithi122 ceph-mon[93869]: pgmap v21230: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 510 KiB/s wr, 7 op/s 2023-03-10T01:45:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38851.543289 secs 2023-03-10T01:45:41.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:41 smithi181 ceph-mon[94189]: pgmap v21230: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 510 KiB/s wr, 7 op/s 2023-03-10T01:45:41.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38851.543289 secs 2023-03-10T01:45:41.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:41 smithi038 ceph-mon[92422]: pgmap v21230: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 510 KiB/s wr, 7 op/s 2023-03-10T01:45:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38851.543289 secs 2023-03-10T01:45:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:43 smithi122 ceph-mon[93869]: pgmap v21231: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 351 KiB/s wr, 9 op/s 2023-03-10T01:45:43.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:43 smithi181 ceph-mon[94189]: pgmap v21231: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 351 KiB/s wr, 9 op/s 2023-03-10T01:45:43.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:43 smithi038 ceph-mon[92422]: pgmap v21231: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 354 KiB/s rd, 351 KiB/s wr, 9 op/s 2023-03-10T01:45:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39315.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:45:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:44 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38854.003638 secs 2023-03-10T01:45:44.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:44 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38854.003638 secs 2023-03-10T01:45:44.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:44 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38854.003638 secs 2023-03-10T01:45:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:45 smithi122 ceph-mon[93869]: pgmap v21232: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 339 KiB/s wr, 8 op/s 2023-03-10T01:45:45.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:45 smithi181 ceph-mon[94189]: pgmap v21232: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 339 KiB/s wr, 8 op/s 2023-03-10T01:45:45.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:45 smithi038 ceph-mon[92422]: pgmap v21232: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 339 KiB/s wr, 8 op/s 2023-03-10T01:45:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38856.543347 secs 2023-03-10T01:45:46.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38856.543347 secs 2023-03-10T01:45:46.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38856.543347 secs 2023-03-10T01:45:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:47 smithi122 ceph-mon[93869]: pgmap v21233: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 356 KiB/s wr, 7 op/s 2023-03-10T01:45:47.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:47 smithi181 ceph-mon[94189]: pgmap v21233: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 356 KiB/s wr, 7 op/s 2023-03-10T01:45:47.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:47 smithi038 ceph-mon[92422]: pgmap v21233: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 359 KiB/s rd, 356 KiB/s wr, 7 op/s 2023-03-10T01:45:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:45:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:45:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:45:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:45:48.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:45:48.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:45:48.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39320.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:45:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:49 smithi122 ceph-mon[93869]: pgmap v21234: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 509 KiB/s rd, 506 KiB/s wr, 10 op/s 2023-03-10T01:45:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:49 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38859.003699 secs 2023-03-10T01:45:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:45:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:45:49.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:49 smithi181 ceph-mon[94189]: pgmap v21234: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 509 KiB/s rd, 506 KiB/s wr, 10 op/s 2023-03-10T01:45:49.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:49 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38859.003699 secs 2023-03-10T01:45:49.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:45:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:45:49.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:49 smithi038 ceph-mon[92422]: pgmap v21234: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 509 KiB/s rd, 506 KiB/s wr, 10 op/s 2023-03-10T01:45:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:49 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38859.003699 secs 2023-03-10T01:45:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:45:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:45:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:51 smithi122 ceph-mon[93869]: pgmap v21235: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 532 KiB/s rd, 506 KiB/s wr, 8 op/s 2023-03-10T01:45:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38861.543405 secs 2023-03-10T01:45:51.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:51 smithi181 ceph-mon[94189]: pgmap v21235: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 532 KiB/s rd, 506 KiB/s wr, 8 op/s 2023-03-10T01:45:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38861.543405 secs 2023-03-10T01:45:51.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:51 smithi038 ceph-mon[92422]: pgmap v21235: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 532 KiB/s rd, 506 KiB/s wr, 8 op/s 2023-03-10T01:45:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38861.543405 secs 2023-03-10T01:45:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:53 smithi122 ceph-mon[93869]: pgmap v21236: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 554 KiB/s rd, 527 KiB/s wr, 10 op/s 2023-03-10T01:45:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:45:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:45:53.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:53 smithi181 ceph-mon[94189]: pgmap v21236: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 554 KiB/s rd, 527 KiB/s wr, 10 op/s 2023-03-10T01:45:53.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:45:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:45:53.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:53 smithi038 ceph-mon[92422]: pgmap v21236: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 554 KiB/s rd, 527 KiB/s wr, 10 op/s 2023-03-10T01:45:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:45:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:45:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39325.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:45:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:54 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38864.003758 secs 2023-03-10T01:45:54.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:54 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38864.003758 secs 2023-03-10T01:45:54.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:54 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38864.003758 secs 2023-03-10T01:45:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:55 smithi122 ceph-mon[93869]: pgmap v21237: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 560 KiB/s wr, 9 op/s 2023-03-10T01:45:55.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:55 smithi181 ceph-mon[94189]: pgmap v21237: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 560 KiB/s wr, 9 op/s 2023-03-10T01:45:55.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:55 smithi038 ceph-mon[92422]: pgmap v21237: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 560 KiB/s wr, 9 op/s 2023-03-10T01:45:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:45:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:45:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38866.543466 secs 2023-03-10T01:45:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:45:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:45:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38866.543466 secs 2023-03-10T01:45:56.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:45:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:45:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38866.543466 secs 2023-03-10T01:45:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:57 smithi122 ceph-mon[93869]: pgmap v21238: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 560 KiB/s wr, 9 op/s 2023-03-10T01:45:57.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:57 smithi181 ceph-mon[94189]: pgmap v21238: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 560 KiB/s wr, 9 op/s 2023-03-10T01:45:57.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:57 smithi038 ceph-mon[92422]: pgmap v21238: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 560 KiB/s wr, 9 op/s 2023-03-10T01:45:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39330.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:45:59.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:59 smithi122 ceph-mon[93869]: pgmap v21239: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 408 KiB/s wr, 12 op/s 2023-03-10T01:45:59.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:45:59 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38869.003818 secs 2023-03-10T01:45:59.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:59 smithi181 ceph-mon[94189]: pgmap v21239: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 408 KiB/s wr, 12 op/s 2023-03-10T01:45:59.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:45:59 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38869.003818 secs 2023-03-10T01:45:59.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:59 smithi038 ceph-mon[92422]: pgmap v21239: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 408 KiB/s wr, 12 op/s 2023-03-10T01:45:59.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:45:59 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38869.003818 secs 2023-03-10T01:46:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:46:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:46:00.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:46:00.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:46:00.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:46:00.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:46:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:01 smithi122 ceph-mon[93869]: pgmap v21240: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 782 KiB/s rd, 670 KiB/s wr, 11 op/s 2023-03-10T01:46:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38871.543526 secs 2023-03-10T01:46:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:46:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:46:01.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:01 smithi181 ceph-mon[94189]: pgmap v21240: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 782 KiB/s rd, 670 KiB/s wr, 11 op/s 2023-03-10T01:46:01.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38871.543526 secs 2023-03-10T01:46:01.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:46:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:46:01.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:01 smithi038 ceph-mon[92422]: pgmap v21240: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 782 KiB/s rd, 670 KiB/s wr, 11 op/s 2023-03-10T01:46:01.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38871.543526 secs 2023-03-10T01:46:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:46:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:46:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:03 smithi122 ceph-mon[93869]: pgmap v21241: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 759 KiB/s rd, 681 KiB/s wr, 14 op/s 2023-03-10T01:46:03.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:03 smithi181 ceph-mon[94189]: pgmap v21241: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 759 KiB/s rd, 681 KiB/s wr, 14 op/s 2023-03-10T01:46:03.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:03 smithi038 ceph-mon[92422]: pgmap v21241: 129 pgs: 129 active+clean; 10 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 759 KiB/s rd, 681 KiB/s wr, 14 op/s 2023-03-10T01:46:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39335.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:46:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:05 smithi122 ceph-mon[93869]: pgmap v21242: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 885 KiB/s rd, 806 KiB/s wr, 25 op/s 2023-03-10T01:46:05.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:05 smithi181 ceph-mon[94189]: pgmap v21242: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 885 KiB/s rd, 806 KiB/s wr, 25 op/s 2023-03-10T01:46:05.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:05 smithi038 ceph-mon[92422]: pgmap v21242: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 885 KiB/s rd, 806 KiB/s wr, 25 op/s 2023-03-10T01:46:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38876.543588 secs 2023-03-10T01:46:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:06 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:46:06.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38876.543588 secs 2023-03-10T01:46:06.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:06 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:46:06.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38876.543588 secs 2023-03-10T01:46:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:06 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:46:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:07 smithi122 ceph-mon[93869]: pgmap v21243: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 852 KiB/s rd, 773 KiB/s wr, 24 op/s 2023-03-10T01:46:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:07 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:46:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:46:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:46:07.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:07 smithi181 ceph-mon[94189]: pgmap v21243: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 852 KiB/s rd, 773 KiB/s wr, 24 op/s 2023-03-10T01:46:07.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:07 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:46:07.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:46:07.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:46:07.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:07 smithi038 ceph-mon[92422]: pgmap v21243: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 852 KiB/s rd, 773 KiB/s wr, 24 op/s 2023-03-10T01:46:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:07 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:46:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:46:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:46:08.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39340.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:46:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:09 smithi122 ceph-mon[93869]: pgmap v21244: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 869 KiB/s rd, 789 KiB/s wr, 27 op/s 2023-03-10T01:46:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:46:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:46:09.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:09 smithi181 ceph-mon[94189]: pgmap v21244: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 869 KiB/s rd, 789 KiB/s wr, 27 op/s 2023-03-10T01:46:09.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:46:09.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:46:09.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:09 smithi038 ceph-mon[92422]: pgmap v21244: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 869 KiB/s rd, 789 KiB/s wr, 27 op/s 2023-03-10T01:46:09.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:46:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:46:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:11 smithi122 ceph-mon[93869]: pgmap v21245: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 25 op/s 2023-03-10T01:46:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38881.543645 secs 2023-03-10T01:46:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:11 smithi181 ceph-mon[94189]: pgmap v21245: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 25 op/s 2023-03-10T01:46:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38881.543645 secs 2023-03-10T01:46:11.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:11 smithi038 ceph-mon[92422]: pgmap v21245: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 25 op/s 2023-03-10T01:46:11.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38881.543645 secs 2023-03-10T01:46:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:46:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:46:12.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:46:12.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:46:12.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:46:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:46:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:13 smithi122 ceph-mon[93869]: pgmap v21246: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1012 KiB/s wr, 26 op/s 2023-03-10T01:46:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:46:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:46:13.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:13 smithi181 ceph-mon[94189]: pgmap v21246: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1012 KiB/s wr, 26 op/s 2023-03-10T01:46:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:46:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:46:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:13 smithi038 ceph-mon[92422]: pgmap v21246: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1012 KiB/s wr, 26 op/s 2023-03-10T01:46:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:46:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:46:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39345.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:46:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:15 smithi122 ceph-mon[93869]: pgmap v21247: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 24 op/s 2023-03-10T01:46:15.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:15 smithi181 ceph-mon[94189]: pgmap v21247: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 24 op/s 2023-03-10T01:46:15.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:15 smithi038 ceph-mon[92422]: pgmap v21247: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 24 op/s 2023-03-10T01:46:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38886.543721 secs 2023-03-10T01:46:16.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38886.543721 secs 2023-03-10T01:46:16.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38886.543721 secs 2023-03-10T01:46:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:17 smithi122 ceph-mon[93869]: pgmap v21248: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 735 KiB/s rd, 714 KiB/s wr, 10 op/s 2023-03-10T01:46:17.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:17 smithi181 ceph-mon[94189]: pgmap v21248: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 735 KiB/s rd, 714 KiB/s wr, 10 op/s 2023-03-10T01:46:17.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:17 smithi038 ceph-mon[92422]: pgmap v21248: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 735 KiB/s rd, 714 KiB/s wr, 10 op/s 2023-03-10T01:46:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39350.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:46:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:19 smithi122 ceph-mon[93869]: pgmap v21249: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 736 KiB/s rd, 716 KiB/s wr, 13 op/s 2023-03-10T01:46:19.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:19 smithi181 ceph-mon[94189]: pgmap v21249: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 736 KiB/s rd, 716 KiB/s wr, 13 op/s 2023-03-10T01:46:19.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:19 smithi038 ceph-mon[92422]: pgmap v21249: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 736 KiB/s rd, 716 KiB/s wr, 13 op/s 2023-03-10T01:46:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:46:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:46:20.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:46:20.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:46:20.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:46:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:46:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:21 smithi122 ceph-mon[93869]: pgmap v21250: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 719 KiB/s rd, 710 KiB/s wr, 11 op/s 2023-03-10T01:46:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38891.543802 secs 2023-03-10T01:46:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:46:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:46:21.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:21 smithi181 ceph-mon[94189]: pgmap v21250: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 719 KiB/s rd, 710 KiB/s wr, 11 op/s 2023-03-10T01:46:21.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38891.543802 secs 2023-03-10T01:46:21.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:46:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:46:21.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:21 smithi038 ceph-mon[92422]: pgmap v21250: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 719 KiB/s rd, 710 KiB/s wr, 11 op/s 2023-03-10T01:46:21.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38891.543802 secs 2023-03-10T01:46:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:46:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:46:23.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:23 smithi181 ceph-mon[94189]: pgmap v21251: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 697 KiB/s rd, 715 KiB/s wr, 13 op/s 2023-03-10T01:46:23.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:23 smithi038 ceph-mon[92422]: pgmap v21251: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 697 KiB/s rd, 715 KiB/s wr, 13 op/s 2023-03-10T01:46:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:23 smithi122 ceph-mon[93869]: pgmap v21251: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 697 KiB/s rd, 715 KiB/s wr, 13 op/s 2023-03-10T01:46:23.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39355.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:46:25.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:25 smithi038 ceph-mon[92422]: pgmap v21252: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 719 KiB/s rd, 739 KiB/s wr, 12 op/s 2023-03-10T01:46:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:25 smithi122 ceph-mon[93869]: pgmap v21252: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 719 KiB/s rd, 739 KiB/s wr, 12 op/s 2023-03-10T01:46:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:25 smithi181 ceph-mon[94189]: pgmap v21252: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 719 KiB/s rd, 739 KiB/s wr, 12 op/s 2023-03-10T01:46:26.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38896.543903 secs 2023-03-10T01:46:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:26 smithi038 ceph-mon[92422]: osdmap e1543: 12 total, 12 up, 12 in 2023-03-10T01:46:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38896.543903 secs 2023-03-10T01:46:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:26 smithi122 ceph-mon[93869]: osdmap e1543: 12 total, 12 up, 12 in 2023-03-10T01:46:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38896.543903 secs 2023-03-10T01:46:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:26 smithi181 ceph-mon[94189]: osdmap e1543: 12 total, 12 up, 12 in 2023-03-10T01:46:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:27 smithi122 ceph-mon[93869]: pgmap v21254: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 827 KiB/s rd, 850 KiB/s wr, 12 op/s 2023-03-10T01:46:27.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:27 smithi181 ceph-mon[94189]: pgmap v21254: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 827 KiB/s rd, 850 KiB/s wr, 12 op/s 2023-03-10T01:46:27.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:27 smithi038 ceph-mon[92422]: pgmap v21254: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 827 KiB/s rd, 850 KiB/s wr, 12 op/s 2023-03-10T01:46:28.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39360.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:46:29.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:29 smithi122 ceph-mon[93869]: pgmap v21255: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 835 KiB/s rd, 857 KiB/s wr, 13 op/s 2023-03-10T01:46:29.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:29 smithi181 ceph-mon[94189]: pgmap v21255: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 835 KiB/s rd, 857 KiB/s wr, 13 op/s 2023-03-10T01:46:29.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:29 smithi038 ceph-mon[92422]: pgmap v21255: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 835 KiB/s rd, 857 KiB/s wr, 13 op/s 2023-03-10T01:46:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:31 smithi122 ceph-mon[93869]: pgmap v21256: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1019 KiB/s rd, 1.0 MiB/s wr, 12 op/s 2023-03-10T01:46:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38901.544001 secs 2023-03-10T01:46:31.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:31 smithi181 ceph-mon[94189]: pgmap v21256: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1019 KiB/s rd, 1.0 MiB/s wr, 12 op/s 2023-03-10T01:46:31.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38901.544001 secs 2023-03-10T01:46:31.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:31 smithi038 ceph-mon[92422]: pgmap v21256: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1019 KiB/s rd, 1.0 MiB/s wr, 12 op/s 2023-03-10T01:46:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38901.544001 secs 2023-03-10T01:46:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:33 smithi122 ceph-mon[93869]: pgmap v21257: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 616 KiB/s rd, 614 KiB/s wr, 10 op/s 2023-03-10T01:46:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:33 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:46:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:33 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:46:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:33 smithi181 ceph-mon[94189]: pgmap v21257: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 616 KiB/s rd, 614 KiB/s wr, 10 op/s 2023-03-10T01:46:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:33 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:46:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:33 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:46:33.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:33 smithi038 ceph-mon[92422]: pgmap v21257: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 616 KiB/s rd, 614 KiB/s wr, 10 op/s 2023-03-10T01:46:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:33 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:46:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:33 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:46:33.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39365.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:46:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:35 smithi122 ceph-mon[93869]: pgmap v21258: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 459 KiB/s wr, 10 op/s 2023-03-10T01:46:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:35 smithi122 ceph-mon[93869]: osdmap e1544: 12 total, 12 up, 12 in 2023-03-10T01:46:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:35 smithi181 ceph-mon[94189]: pgmap v21258: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 459 KiB/s wr, 10 op/s 2023-03-10T01:46:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:35 smithi181 ceph-mon[94189]: osdmap e1544: 12 total, 12 up, 12 in 2023-03-10T01:46:35.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:35 smithi038 ceph-mon[92422]: pgmap v21258: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 459 KiB/s wr, 10 op/s 2023-03-10T01:46:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:35 smithi038 ceph-mon[92422]: osdmap e1544: 12 total, 12 up, 12 in 2023-03-10T01:46:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38906.544060 secs 2023-03-10T01:46:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:46:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:46:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38906.544060 secs 2023-03-10T01:46:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:46:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:46:36.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38906.544060 secs 2023-03-10T01:46:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:46:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:46:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:37 smithi122 ceph-mon[93869]: pgmap v21260: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 459 KiB/s wr, 10 op/s 2023-03-10T01:46:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:46:37.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:46:37.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:37 smithi181 ceph-mon[94189]: pgmap v21260: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 459 KiB/s wr, 10 op/s 2023-03-10T01:46:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:46:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:46:37.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:37 smithi038 ceph-mon[92422]: pgmap v21260: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 236 KiB/s rd, 459 KiB/s wr, 10 op/s 2023-03-10T01:46:37.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:46:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:46:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:38 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:46:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38909.004355 secs 2023-03-10T01:46:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:38 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:46:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:38 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:46:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:38 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:46:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38909.004355 secs 2023-03-10T01:46:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:38 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:46:38.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:38 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:46:38.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:38 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:46:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38909.004355 secs 2023-03-10T01:46:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:38 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:46:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:38 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:46:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39370.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:46:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:39 smithi122 ceph-mon[93869]: pgmap v21261: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 926 KiB/s rd, 1.1 MiB/s wr, 11 op/s 2023-03-10T01:46:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:39 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:46:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:39 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:46:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:39 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:46:39.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:46:39.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:46:39.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:46:39.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:39 smithi181 ceph-mon[94189]: pgmap v21261: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 926 KiB/s rd, 1.1 MiB/s wr, 11 op/s 2023-03-10T01:46:39.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:39 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:46:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:39 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:46:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:39 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:46:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:46:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:46:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:46:39.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:39 smithi038 ceph-mon[92422]: pgmap v21261: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 926 KiB/s rd, 1.1 MiB/s wr, 11 op/s 2023-03-10T01:46:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:39 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:46:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:39 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:46:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:39 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:46:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:46:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:46:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:46:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:46:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:46:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:46:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:46:40.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:46:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:46:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:41 smithi122 ceph-mon[93869]: pgmap v21262: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 964 KiB/s rd, 963 KiB/s wr, 11 op/s 2023-03-10T01:46:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38911.544113 secs 2023-03-10T01:46:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:46:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:46:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:41 smithi181 ceph-mon[94189]: pgmap v21262: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 964 KiB/s rd, 963 KiB/s wr, 11 op/s 2023-03-10T01:46:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38911.544113 secs 2023-03-10T01:46:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:46:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:46:41.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:41 smithi038 ceph-mon[92422]: pgmap v21262: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 964 KiB/s rd, 963 KiB/s wr, 11 op/s 2023-03-10T01:46:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38911.544113 secs 2023-03-10T01:46:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:46:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:46:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:43 smithi122 ceph-mon[93869]: pgmap v21263: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 12 op/s 2023-03-10T01:46:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38914.004441 secs 2023-03-10T01:46:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:43 smithi181 ceph-mon[94189]: pgmap v21263: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 12 op/s 2023-03-10T01:46:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38914.004441 secs 2023-03-10T01:46:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:43 smithi038 ceph-mon[92422]: pgmap v21263: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 12 op/s 2023-03-10T01:46:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38914.004441 secs 2023-03-10T01:46:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39375.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:46:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:45 smithi122 ceph-mon[93869]: pgmap v21264: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 899 KiB/s wr, 12 op/s 2023-03-10T01:46:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:45 smithi181 ceph-mon[94189]: pgmap v21264: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 899 KiB/s wr, 12 op/s 2023-03-10T01:46:45.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:45 smithi038 ceph-mon[92422]: pgmap v21264: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 899 KiB/s wr, 12 op/s 2023-03-10T01:46:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38916.544172 secs 2023-03-10T01:46:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38916.544172 secs 2023-03-10T01:46:46.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38916.544172 secs 2023-03-10T01:46:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:47 smithi122 ceph-mon[93869]: pgmap v21265: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 900 KiB/s wr, 11 op/s 2023-03-10T01:46:47.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:47 smithi181 ceph-mon[94189]: pgmap v21265: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 900 KiB/s wr, 11 op/s 2023-03-10T01:46:47.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:47 smithi038 ceph-mon[92422]: pgmap v21265: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 900 KiB/s wr, 11 op/s 2023-03-10T01:46:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:46:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:46:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:46:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:46:48.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:46:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:46:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39380.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:46:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:49 smithi122 ceph-mon[93869]: pgmap v21266: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 14 op/s 2023-03-10T01:46:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:46:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:46:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:49 smithi181 ceph-mon[94189]: pgmap v21266: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 14 op/s 2023-03-10T01:46:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:46:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:46:49.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:49 smithi038 ceph-mon[92422]: pgmap v21266: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 14 op/s 2023-03-10T01:46:49.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:46:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:46:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:50 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:46:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:50 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:46:50.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:50 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:46:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:51 smithi122 ceph-mon[93869]: pgmap v21267: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 670 KiB/s rd, 646 KiB/s wr, 9 op/s 2023-03-10T01:46:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38921.544231 secs 2023-03-10T01:46:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:51 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:46:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:46:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:46:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:51 smithi181 ceph-mon[94189]: pgmap v21267: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 670 KiB/s rd, 646 KiB/s wr, 9 op/s 2023-03-10T01:46:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38921.544231 secs 2023-03-10T01:46:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:51 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:46:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:46:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:46:51.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:51 smithi038 ceph-mon[92422]: pgmap v21267: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 670 KiB/s rd, 646 KiB/s wr, 9 op/s 2023-03-10T01:46:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38921.544231 secs 2023-03-10T01:46:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:51 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:46:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:46:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:46:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:46:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:46:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:46:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:46:52.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:46:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:46:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:53 smithi122 ceph-mon[93869]: pgmap v21268: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 950 KiB/s rd, 1.0 MiB/s wr, 13 op/s 2023-03-10T01:46:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:46:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:46:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:53 smithi181 ceph-mon[94189]: pgmap v21268: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 950 KiB/s rd, 1.0 MiB/s wr, 13 op/s 2023-03-10T01:46:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:46:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:46:53.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:53 smithi038 ceph-mon[92422]: pgmap v21268: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 950 KiB/s rd, 1.0 MiB/s wr, 13 op/s 2023-03-10T01:46:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:46:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:46:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39385.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:46:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:55 smithi122 ceph-mon[93869]: pgmap v21269: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 12 op/s 2023-03-10T01:46:55.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:55 smithi181 ceph-mon[94189]: pgmap v21269: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 12 op/s 2023-03-10T01:46:55.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:55 smithi038 ceph-mon[92422]: pgmap v21269: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 12 op/s 2023-03-10T01:46:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38926.544331 secs 2023-03-10T01:46:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:46:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:46:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38926.544331 secs 2023-03-10T01:46:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:46:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:46:56.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38926.544331 secs 2023-03-10T01:46:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:46:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:46:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:57 smithi122 ceph-mon[93869]: pgmap v21270: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 11 op/s 2023-03-10T01:46:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:46:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:46:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:57 smithi181 ceph-mon[94189]: pgmap v21270: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 11 op/s 2023-03-10T01:46:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:46:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:46:57.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:57 smithi038 ceph-mon[92422]: pgmap v21270: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 11 op/s 2023-03-10T01:46:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:46:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:46:58.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39390.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:46:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:46:59 smithi122 ceph-mon[93869]: pgmap v21271: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 15 op/s 2023-03-10T01:46:59.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:46:59 smithi181 ceph-mon[94189]: pgmap v21271: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 15 op/s 2023-03-10T01:46:59.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:46:59 smithi038 ceph-mon[92422]: pgmap v21271: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 15 op/s 2023-03-10T01:47:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:47:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:47:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:00 smithi122 ceph-mon[93869]: pgmap v21272: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 12 op/s 2023-03-10T01:47:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:47:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:47:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:00 smithi181 ceph-mon[94189]: pgmap v21272: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 12 op/s 2023-03-10T01:47:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:47:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:47:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:00 smithi038 ceph-mon[92422]: pgmap v21272: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 12 op/s 2023-03-10T01:47:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38931.544415 secs 2023-03-10T01:47:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:47:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:47:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38931.544415 secs 2023-03-10T01:47:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:47:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:47:01.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38931.544415 secs 2023-03-10T01:47:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:47:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:47:02.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:02 smithi122 ceph-mon[93869]: pgmap v21273: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.3 MiB/s wr, 17 op/s 2023-03-10T01:47:02.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:02 smithi181 ceph-mon[94189]: pgmap v21273: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.3 MiB/s wr, 17 op/s 2023-03-10T01:47:02.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:02 smithi038 ceph-mon[92422]: pgmap v21273: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.3 MiB/s wr, 17 op/s 2023-03-10T01:47:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:03 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 38934.004681 secs 2023-03-10T01:47:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:03 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 38934.004681 secs 2023-03-10T01:47:03.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:03 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 38934.004681 secs 2023-03-10T01:47:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39395.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:47:04.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:04 smithi122 ceph-mon[93869]: pgmap v21274: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 975 KiB/s wr, 14 op/s 2023-03-10T01:47:04.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:04 smithi181 ceph-mon[94189]: pgmap v21274: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 975 KiB/s wr, 14 op/s 2023-03-10T01:47:04.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:04 smithi038 ceph-mon[92422]: pgmap v21274: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 975 KiB/s wr, 14 op/s 2023-03-10T01:47:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38936.544473 secs 2023-03-10T01:47:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:06 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:47:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:06 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:47:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:47:06.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38936.544473 secs 2023-03-10T01:47:06.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:06 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:47:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:06 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:47:06.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:47:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38936.544473 secs 2023-03-10T01:47:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:06 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:47:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:06 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:47:06.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:47:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:07 smithi122 ceph-mon[93869]: pgmap v21275: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 984 KiB/s rd, 686 KiB/s wr, 12 op/s 2023-03-10T01:47:07.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:07 smithi181 ceph-mon[94189]: pgmap v21275: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 984 KiB/s rd, 686 KiB/s wr, 12 op/s 2023-03-10T01:47:07.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:07 smithi038 ceph-mon[92422]: pgmap v21275: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 984 KiB/s rd, 686 KiB/s wr, 12 op/s 2023-03-10T01:47:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39400.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:47:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:09 smithi122 ceph-mon[93869]: pgmap v21276: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 982 KiB/s wr, 18 op/s 2023-03-10T01:47:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:09 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 38939.004766 secs 2023-03-10T01:47:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:47:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:47:09.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:09 smithi181 ceph-mon[94189]: pgmap v21276: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 982 KiB/s wr, 18 op/s 2023-03-10T01:47:09.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:09 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 38939.004766 secs 2023-03-10T01:47:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:47:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:47:09.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:09 smithi038 ceph-mon[92422]: pgmap v21276: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 982 KiB/s wr, 18 op/s 2023-03-10T01:47:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:09 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 38939.004766 secs 2023-03-10T01:47:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:47:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:47:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:11 smithi122 ceph-mon[93869]: pgmap v21277: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 772 KiB/s wr, 24 op/s 2023-03-10T01:47:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38941.544530 secs 2023-03-10T01:47:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:11 smithi181 ceph-mon[94189]: pgmap v21277: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 772 KiB/s wr, 24 op/s 2023-03-10T01:47:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38941.544530 secs 2023-03-10T01:47:11.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:11 smithi038 ceph-mon[92422]: pgmap v21277: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 772 KiB/s wr, 24 op/s 2023-03-10T01:47:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38941.544530 secs 2023-03-10T01:47:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:47:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:47:12.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:47:12.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:47:12.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:47:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:47:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:13 smithi122 ceph-mon[93869]: pgmap v21278: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 944 KiB/s wr, 27 op/s 2023-03-10T01:47:13.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:47:13.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:47:13.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:13 smithi181 ceph-mon[94189]: pgmap v21278: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 944 KiB/s wr, 27 op/s 2023-03-10T01:47:13.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:47:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:47:13.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:13 smithi038 ceph-mon[92422]: pgmap v21278: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 944 KiB/s wr, 27 op/s 2023-03-10T01:47:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:47:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:47:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39405.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:47:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:14 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 38944.005063 secs 2023-03-10T01:47:14.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:14 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 38944.005063 secs 2023-03-10T01:47:14.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:14 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 38944.005063 secs 2023-03-10T01:47:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:15 smithi122 ceph-mon[93869]: pgmap v21279: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 997 KiB/s rd, 499 KiB/s wr, 22 op/s 2023-03-10T01:47:15.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:15 smithi181 ceph-mon[94189]: pgmap v21279: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 997 KiB/s rd, 499 KiB/s wr, 22 op/s 2023-03-10T01:47:15.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:15 smithi038 ceph-mon[92422]: pgmap v21279: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 997 KiB/s rd, 499 KiB/s wr, 22 op/s 2023-03-10T01:47:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:47:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:47:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38946.544614 secs 2023-03-10T01:47:16.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:47:16.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:47:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38946.544614 secs 2023-03-10T01:47:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:47:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:47:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38946.544614 secs 2023-03-10T01:47:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:17 smithi122 ceph-mon[93869]: pgmap v21280: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 649 KiB/s wr, 21 op/s 2023-03-10T01:47:17.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:17 smithi181 ceph-mon[94189]: pgmap v21280: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 649 KiB/s wr, 21 op/s 2023-03-10T01:47:17.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:17 smithi038 ceph-mon[92422]: pgmap v21280: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 649 KiB/s wr, 21 op/s 2023-03-10T01:47:18.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39410.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:47:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:19 smithi122 ceph-mon[93869]: pgmap v21281: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 649 KiB/s wr, 23 op/s 2023-03-10T01:47:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:19 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 38949.005122 secs 2023-03-10T01:47:19.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:19 smithi181 ceph-mon[94189]: pgmap v21281: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 649 KiB/s wr, 23 op/s 2023-03-10T01:47:19.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:19 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 38949.005122 secs 2023-03-10T01:47:19.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:19 smithi038 ceph-mon[92422]: pgmap v21281: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 649 KiB/s wr, 23 op/s 2023-03-10T01:47:19.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:19 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 38949.005122 secs 2023-03-10T01:47:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:21 smithi122 ceph-mon[93869]: pgmap v21282: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 775 KiB/s rd, 369 KiB/s wr, 18 op/s 2023-03-10T01:47:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38951.544672 secs 2023-03-10T01:47:21.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:21 smithi181 ceph-mon[94189]: pgmap v21282: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 775 KiB/s rd, 369 KiB/s wr, 18 op/s 2023-03-10T01:47:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38951.544672 secs 2023-03-10T01:47:21.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:21 smithi038 ceph-mon[92422]: pgmap v21282: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 775 KiB/s rd, 369 KiB/s wr, 18 op/s 2023-03-10T01:47:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38951.544672 secs 2023-03-10T01:47:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:23 smithi122 ceph-mon[93869]: pgmap v21283: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 803 KiB/s rd, 391 KiB/s wr, 11 op/s 2023-03-10T01:47:23.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:23 smithi181 ceph-mon[94189]: pgmap v21283: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 803 KiB/s rd, 391 KiB/s wr, 11 op/s 2023-03-10T01:47:23.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:23 smithi038 ceph-mon[92422]: pgmap v21283: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 803 KiB/s rd, 391 KiB/s wr, 11 op/s 2023-03-10T01:47:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39415.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:47:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:24 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 38954.005189 secs 2023-03-10T01:47:24.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:24 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 38954.005189 secs 2023-03-10T01:47:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:24 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 38954.005189 secs 2023-03-10T01:47:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:25 smithi122 ceph-mon[93869]: pgmap v21284: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 230 KiB/s wr, 9 op/s 2023-03-10T01:47:25.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:25 smithi181 ceph-mon[94189]: pgmap v21284: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 230 KiB/s wr, 9 op/s 2023-03-10T01:47:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:25 smithi038 ceph-mon[92422]: pgmap v21284: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 230 KiB/s wr, 9 op/s 2023-03-10T01:47:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38956.544759 secs 2023-03-10T01:47:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38956.544759 secs 2023-03-10T01:47:26.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38956.544759 secs 2023-03-10T01:47:27.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:27 smithi122 ceph-mon[93869]: pgmap v21285: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 450 KiB/s wr, 10 op/s 2023-03-10T01:47:27.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:27 smithi181 ceph-mon[94189]: pgmap v21285: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 450 KiB/s wr, 10 op/s 2023-03-10T01:47:27.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:27 smithi038 ceph-mon[92422]: pgmap v21285: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 450 KiB/s wr, 10 op/s 2023-03-10T01:47:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:28 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:47:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:28 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:47:28.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:28 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:47:28.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:28 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:47:28.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:28 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:47:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:28 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:47:28.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39420.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:47:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:29 smithi122 ceph-mon[93869]: pgmap v21286: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 325 KiB/s wr, 13 op/s 2023-03-10T01:47:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:29 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38959.005254 secs 2023-03-10T01:47:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:29 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:47:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:29 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:47:29.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:29 smithi181 ceph-mon[94189]: pgmap v21286: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 325 KiB/s wr, 13 op/s 2023-03-10T01:47:29.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:29 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38959.005254 secs 2023-03-10T01:47:29.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:29 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:47:29.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:29 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:47:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:29 smithi038 ceph-mon[92422]: pgmap v21286: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 325 KiB/s wr, 13 op/s 2023-03-10T01:47:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:29 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38959.005254 secs 2023-03-10T01:47:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:29 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:47:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:29 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:47:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:31 smithi122 ceph-mon[93869]: pgmap v21287: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 325 KiB/s wr, 11 op/s 2023-03-10T01:47:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38961.544816 secs 2023-03-10T01:47:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:31 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:47:31.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:31 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:47:31.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:31 smithi122 ceph-mon[93869]: osdmap e1545: 12 total, 12 up, 12 in 2023-03-10T01:47:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:31 smithi181 ceph-mon[94189]: pgmap v21287: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 325 KiB/s wr, 11 op/s 2023-03-10T01:47:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38961.544816 secs 2023-03-10T01:47:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:31 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:47:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:31 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:47:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:31 smithi181 ceph-mon[94189]: osdmap e1545: 12 total, 12 up, 12 in 2023-03-10T01:47:31.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:31 smithi038 ceph-mon[92422]: pgmap v21287: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 305 KiB/s rd, 325 KiB/s wr, 11 op/s 2023-03-10T01:47:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38961.544816 secs 2023-03-10T01:47:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:31 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:47:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:31 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:47:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:31 smithi038 ceph-mon[92422]: osdmap e1545: 12 total, 12 up, 12 in 2023-03-10T01:47:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:33 smithi122 ceph-mon[93869]: pgmap v21289: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 432 KiB/s wr, 12 op/s 2023-03-10T01:47:33.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:33 smithi181 ceph-mon[94189]: pgmap v21289: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 432 KiB/s wr, 12 op/s 2023-03-10T01:47:33.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:33 smithi038 ceph-mon[92422]: pgmap v21289: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 408 KiB/s rd, 432 KiB/s wr, 12 op/s 2023-03-10T01:47:33.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39425.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:47:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:34 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38964.005315 secs 2023-03-10T01:47:34.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:34 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38964.005315 secs 2023-03-10T01:47:34.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:34 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38964.005315 secs 2023-03-10T01:47:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:35 smithi122 ceph-mon[93869]: pgmap v21290: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 419 KiB/s wr, 11 op/s 2023-03-10T01:47:35.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:35 smithi181 ceph-mon[94189]: pgmap v21290: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 419 KiB/s wr, 11 op/s 2023-03-10T01:47:35.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:35 smithi038 ceph-mon[92422]: pgmap v21290: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 419 KiB/s wr, 11 op/s 2023-03-10T01:47:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38966.544872 secs 2023-03-10T01:47:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:47:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:47:36.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38966.544872 secs 2023-03-10T01:47:36.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:47:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:47:36.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38966.544872 secs 2023-03-10T01:47:36.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:47:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:47:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:37 smithi122 ceph-mon[93869]: pgmap v21291: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 282 KiB/s wr, 10 op/s 2023-03-10T01:47:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:47:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:47:37.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:37 smithi181 ceph-mon[94189]: pgmap v21291: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 282 KiB/s wr, 10 op/s 2023-03-10T01:47:37.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:47:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:47:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:37 smithi038 ceph-mon[92422]: pgmap v21291: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 271 KiB/s rd, 282 KiB/s wr, 10 op/s 2023-03-10T01:47:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:47:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:47:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38969.005376 secs 2023-03-10T01:47:38.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38969.005376 secs 2023-03-10T01:47:38.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38969.005376 secs 2023-03-10T01:47:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39430.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:47:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:39 smithi122 ceph-mon[93869]: pgmap v21292: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 641 KiB/s rd, 639 KiB/s wr, 10 op/s 2023-03-10T01:47:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:39 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:47:39.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:39 smithi181 ceph-mon[94189]: pgmap v21292: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 641 KiB/s rd, 639 KiB/s wr, 10 op/s 2023-03-10T01:47:39.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:39 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:47:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:39 smithi038 ceph-mon[92422]: pgmap v21292: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 641 KiB/s rd, 639 KiB/s wr, 10 op/s 2023-03-10T01:47:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:39 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:47:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:47:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:47:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:47:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:47:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:40 smithi122 ceph-mon[93869]: osdmap e1546: 12 total, 12 up, 12 in 2023-03-10T01:47:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:47:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:47:40.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:47:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:47:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:47:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:47:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:47:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:40 smithi181 ceph-mon[94189]: osdmap e1546: 12 total, 12 up, 12 in 2023-03-10T01:47:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:47:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:47:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:47:40.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:47:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:47:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:47:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:47:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:40 smithi038 ceph-mon[92422]: osdmap e1546: 12 total, 12 up, 12 in 2023-03-10T01:47:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:47:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:47:40.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:47:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:41 smithi122 ceph-mon[93869]: pgmap v21293: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 641 KiB/s rd, 639 KiB/s wr, 10 op/s 2023-03-10T01:47:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38971.544930 secs 2023-03-10T01:47:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:41 smithi181 ceph-mon[94189]: pgmap v21293: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 641 KiB/s rd, 639 KiB/s wr, 10 op/s 2023-03-10T01:47:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38971.544930 secs 2023-03-10T01:47:41.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:41 smithi038 ceph-mon[92422]: pgmap v21293: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 641 KiB/s rd, 639 KiB/s wr, 10 op/s 2023-03-10T01:47:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38971.544930 secs 2023-03-10T01:47:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:43 smithi122 ceph-mon[93869]: pgmap v21295: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 539 KiB/s rd, 538 KiB/s wr, 10 op/s 2023-03-10T01:47:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38974.005436 secs 2023-03-10T01:47:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:43 smithi181 ceph-mon[94189]: pgmap v21295: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 539 KiB/s rd, 538 KiB/s wr, 10 op/s 2023-03-10T01:47:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38974.005436 secs 2023-03-10T01:47:43.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:43 smithi038 ceph-mon[92422]: pgmap v21295: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 539 KiB/s rd, 538 KiB/s wr, 10 op/s 2023-03-10T01:47:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38974.005436 secs 2023-03-10T01:47:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39435.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:47:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:45 smithi122 ceph-mon[93869]: pgmap v21296: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 733 KiB/s rd, 731 KiB/s wr, 10 op/s 2023-03-10T01:47:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:45 smithi181 ceph-mon[94189]: pgmap v21296: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 733 KiB/s rd, 731 KiB/s wr, 10 op/s 2023-03-10T01:47:45.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:45 smithi038 ceph-mon[92422]: pgmap v21296: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 733 KiB/s rd, 731 KiB/s wr, 10 op/s 2023-03-10T01:47:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38976.544983 secs 2023-03-10T01:47:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38976.544983 secs 2023-03-10T01:47:46.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38976.544983 secs 2023-03-10T01:47:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:47 smithi122 ceph-mon[93869]: pgmap v21297: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 606 KiB/s rd, 604 KiB/s wr, 9 op/s 2023-03-10T01:47:47.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:47 smithi181 ceph-mon[94189]: pgmap v21297: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 606 KiB/s rd, 604 KiB/s wr, 9 op/s 2023-03-10T01:47:47.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:47 smithi038 ceph-mon[92422]: pgmap v21297: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 606 KiB/s rd, 604 KiB/s wr, 9 op/s 2023-03-10T01:47:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:47:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:47:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38979.005497 secs 2023-03-10T01:47:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:47:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:47:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38979.005497 secs 2023-03-10T01:47:48.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:47:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:47:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38979.005497 secs 2023-03-10T01:47:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39440.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:47:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:49 smithi122 ceph-mon[93869]: pgmap v21298: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 313 KiB/s rd, 310 KiB/s wr, 8 op/s 2023-03-10T01:47:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:47:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:47:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:49 smithi181 ceph-mon[94189]: pgmap v21298: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 313 KiB/s rd, 310 KiB/s wr, 8 op/s 2023-03-10T01:47:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:47:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:47:49.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:49 smithi038 ceph-mon[92422]: pgmap v21298: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 313 KiB/s rd, 310 KiB/s wr, 8 op/s 2023-03-10T01:47:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:47:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:47:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:51 smithi122 ceph-mon[93869]: pgmap v21299: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 404 KiB/s wr, 9 op/s 2023-03-10T01:47:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38981.545070 secs 2023-03-10T01:47:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:51 smithi181 ceph-mon[94189]: pgmap v21299: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 404 KiB/s wr, 9 op/s 2023-03-10T01:47:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38981.545070 secs 2023-03-10T01:47:51.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:51 smithi038 ceph-mon[92422]: pgmap v21299: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 404 KiB/s wr, 9 op/s 2023-03-10T01:47:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38981.545070 secs 2023-03-10T01:47:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:53 smithi122 ceph-mon[93869]: pgmap v21300: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 384 KiB/s wr, 11 op/s 2023-03-10T01:47:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:47:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:47:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38984.005554 secs 2023-03-10T01:47:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:53 smithi181 ceph-mon[94189]: pgmap v21300: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 384 KiB/s wr, 11 op/s 2023-03-10T01:47:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:47:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:47:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38984.005554 secs 2023-03-10T01:47:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:53 smithi038 ceph-mon[92422]: pgmap v21300: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 387 KiB/s rd, 384 KiB/s wr, 11 op/s 2023-03-10T01:47:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:47:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:47:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38984.005554 secs 2023-03-10T01:47:53.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39445.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:47:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:55 smithi122 ceph-mon[93869]: pgmap v21301: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 549 KiB/s wr, 9 op/s 2023-03-10T01:47:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:55 smithi181 ceph-mon[94189]: pgmap v21301: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 549 KiB/s wr, 9 op/s 2023-03-10T01:47:55.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:55 smithi038 ceph-mon[92422]: pgmap v21301: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 549 KiB/s wr, 9 op/s 2023-03-10T01:47:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38986.545131 secs 2023-03-10T01:47:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:47:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:47:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38986.545131 secs 2023-03-10T01:47:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:47:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:47:56.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38986.545131 secs 2023-03-10T01:47:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:47:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:47:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:57 smithi122 ceph-mon[93869]: pgmap v21302: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 387 KiB/s wr, 8 op/s 2023-03-10T01:47:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:57 smithi181 ceph-mon[94189]: pgmap v21302: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 387 KiB/s wr, 8 op/s 2023-03-10T01:47:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:57 smithi038 ceph-mon[92422]: pgmap v21302: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 390 KiB/s rd, 387 KiB/s wr, 8 op/s 2023-03-10T01:47:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38989.005663 secs 2023-03-10T01:47:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38989.005663 secs 2023-03-10T01:47:58.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38989.005663 secs 2023-03-10T01:47:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39450.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:47:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:47:59 smithi122 ceph-mon[93869]: pgmap v21303: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 421 KiB/s wr, 11 op/s 2023-03-10T01:47:59.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:47:59 smithi181 ceph-mon[94189]: pgmap v21303: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 421 KiB/s wr, 11 op/s 2023-03-10T01:47:59.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:47:59 smithi038 ceph-mon[92422]: pgmap v21303: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 426 KiB/s rd, 421 KiB/s wr, 11 op/s 2023-03-10T01:48:01.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:00 smithi038 ceph-mon[92422]: pgmap v21304: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 412 KiB/s wr, 10 op/s 2023-03-10T01:48:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:48:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:48:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:00 smithi122 ceph-mon[93869]: pgmap v21304: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 412 KiB/s wr, 10 op/s 2023-03-10T01:48:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:48:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:48:01.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:00 smithi181 ceph-mon[94189]: pgmap v21304: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 412 KiB/s wr, 10 op/s 2023-03-10T01:48:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:48:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:48:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38991.545191 secs 2023-03-10T01:48:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38991.545191 secs 2023-03-10T01:48:02.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38991.545191 secs 2023-03-10T01:48:03.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:02 smithi038 ceph-mon[92422]: pgmap v21305: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 847 KiB/s rd, 720 KiB/s wr, 13 op/s 2023-03-10T01:48:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:02 smithi122 ceph-mon[93869]: pgmap v21305: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 847 KiB/s rd, 720 KiB/s wr, 13 op/s 2023-03-10T01:48:03.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:02 smithi181 ceph-mon[94189]: pgmap v21305: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 847 KiB/s rd, 720 KiB/s wr, 13 op/s 2023-03-10T01:48:03.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39455.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:48:04.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:04 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38994.005722 secs 2023-03-10T01:48:04.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:04 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38994.005722 secs 2023-03-10T01:48:04.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:04 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38994.005722 secs 2023-03-10T01:48:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:05 smithi122 ceph-mon[93869]: pgmap v21306: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.0 MiB/s wr, 24 op/s 2023-03-10T01:48:05.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:05 smithi181 ceph-mon[94189]: pgmap v21306: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.0 MiB/s wr, 24 op/s 2023-03-10T01:48:05.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:05 smithi038 ceph-mon[92422]: pgmap v21306: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.0 MiB/s wr, 24 op/s 2023-03-10T01:48:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:07 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38996.545248 secs 2023-03-10T01:48:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:07 smithi122 ceph-mon[93869]: pgmap v21307: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 856 KiB/s wr, 23 op/s 2023-03-10T01:48:07.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:07 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38996.545248 secs 2023-03-10T01:48:07.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:07 smithi181 ceph-mon[94189]: pgmap v21307: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 856 KiB/s wr, 23 op/s 2023-03-10T01:48:07.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:07 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38996.545248 secs 2023-03-10T01:48:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:07 smithi038 ceph-mon[92422]: pgmap v21307: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 856 KiB/s wr, 23 op/s 2023-03-10T01:48:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:48:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:48:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:48:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:48:08.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:48:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:48:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39460.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:48:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:09 smithi122 ceph-mon[93869]: pgmap v21308: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 26 op/s 2023-03-10T01:48:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:09 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 38999.005808 secs 2023-03-10T01:48:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:48:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:48:09.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:09 smithi181 ceph-mon[94189]: pgmap v21308: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 26 op/s 2023-03-10T01:48:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:09 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 38999.005808 secs 2023-03-10T01:48:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:48:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:48:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:09 smithi038 ceph-mon[92422]: pgmap v21308: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 26 op/s 2023-03-10T01:48:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:09 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 38999.005808 secs 2023-03-10T01:48:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:48:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:48:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:11 smithi122 ceph-mon[93869]: pgmap v21309: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.1 MiB/s wr, 24 op/s 2023-03-10T01:48:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39001.545306 secs 2023-03-10T01:48:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:11 smithi181 ceph-mon[94189]: pgmap v21309: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.1 MiB/s wr, 24 op/s 2023-03-10T01:48:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39001.545306 secs 2023-03-10T01:48:11.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:11 smithi038 ceph-mon[92422]: pgmap v21309: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.1 MiB/s wr, 24 op/s 2023-03-10T01:48:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39001.545306 secs 2023-03-10T01:48:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:48:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:48:12.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:48:12.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:48:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:48:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:48:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:13 smithi122 ceph-mon[93869]: pgmap v21310: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 25 op/s 2023-03-10T01:48:13.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:13 smithi181 ceph-mon[94189]: pgmap v21310: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 25 op/s 2023-03-10T01:48:13.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:13 smithi038 ceph-mon[92422]: pgmap v21310: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 25 op/s 2023-03-10T01:48:13.793 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39465.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:48:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:14 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39004.005896 secs 2023-03-10T01:48:14.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:14 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39004.005896 secs 2023-03-10T01:48:14.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:14 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39004.005896 secs 2023-03-10T01:48:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:15 smithi122 ceph-mon[93869]: pgmap v21311: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 836 KiB/s wr, 21 op/s 2023-03-10T01:48:15.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:15 smithi181 ceph-mon[94189]: pgmap v21311: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 836 KiB/s wr, 21 op/s 2023-03-10T01:48:15.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:15 smithi038 ceph-mon[92422]: pgmap v21311: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 836 KiB/s wr, 21 op/s 2023-03-10T01:48:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39006.545368 secs 2023-03-10T01:48:16.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39006.545368 secs 2023-03-10T01:48:16.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39006.545368 secs 2023-03-10T01:48:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:17 smithi122 ceph-mon[93869]: pgmap v21312: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 629 KiB/s rd, 487 KiB/s wr, 8 op/s 2023-03-10T01:48:17.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:17 smithi181 ceph-mon[94189]: pgmap v21312: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 629 KiB/s rd, 487 KiB/s wr, 8 op/s 2023-03-10T01:48:17.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:17 smithi038 ceph-mon[92422]: pgmap v21312: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 629 KiB/s rd, 487 KiB/s wr, 8 op/s 2023-03-10T01:48:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39470.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:48:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:19 smithi122 ceph-mon[93869]: pgmap v21313: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 630 KiB/s rd, 487 KiB/s wr, 11 op/s 2023-03-10T01:48:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:19 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39009.006026 secs 2023-03-10T01:48:19.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:19 smithi181 ceph-mon[94189]: pgmap v21313: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 630 KiB/s rd, 487 KiB/s wr, 11 op/s 2023-03-10T01:48:19.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:19 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39009.006026 secs 2023-03-10T01:48:19.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:19 smithi038 ceph-mon[92422]: pgmap v21313: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 630 KiB/s rd, 487 KiB/s wr, 11 op/s 2023-03-10T01:48:19.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:19 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39009.006026 secs 2023-03-10T01:48:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:48:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:48:20.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:48:20.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:48:20.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:48:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:48:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:21 smithi122 ceph-mon[93869]: pgmap v21314: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 201 KiB/s wr, 9 op/s 2023-03-10T01:48:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:48:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:48:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39011.545445 secs 2023-03-10T01:48:21.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:21 smithi181 ceph-mon[94189]: pgmap v21314: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 201 KiB/s wr, 9 op/s 2023-03-10T01:48:21.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:48:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:48:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39011.545445 secs 2023-03-10T01:48:21.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:21 smithi038 ceph-mon[92422]: pgmap v21314: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 334 KiB/s rd, 201 KiB/s wr, 9 op/s 2023-03-10T01:48:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:48:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:48:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39011.545445 secs 2023-03-10T01:48:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:23 smithi122 ceph-mon[93869]: pgmap v21315: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 525 KiB/s rd, 543 KiB/s wr, 13 op/s 2023-03-10T01:48:23.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:23 smithi181 ceph-mon[94189]: pgmap v21315: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 525 KiB/s rd, 543 KiB/s wr, 13 op/s 2023-03-10T01:48:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:23 smithi038 ceph-mon[92422]: pgmap v21315: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 525 KiB/s rd, 543 KiB/s wr, 13 op/s 2023-03-10T01:48:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39475.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:48:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:24 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39014.006084 secs 2023-03-10T01:48:24.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:24 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39014.006084 secs 2023-03-10T01:48:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:24 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39014.006084 secs 2023-03-10T01:48:25.478 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:25 smithi122 ceph-mon[93869]: pgmap v21316: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 674 KiB/s rd, 694 KiB/s wr, 12 op/s 2023-03-10T01:48:25.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:25 smithi181 ceph-mon[94189]: pgmap v21316: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 674 KiB/s rd, 694 KiB/s wr, 12 op/s 2023-03-10T01:48:25.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:25 smithi038 ceph-mon[92422]: pgmap v21316: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 674 KiB/s rd, 694 KiB/s wr, 12 op/s 2023-03-10T01:48:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39016.545505 secs 2023-03-10T01:48:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:26 smithi122 ceph-mon[93869]: osdmap e1547: 12 total, 12 up, 12 in 2023-03-10T01:48:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39016.545505 secs 2023-03-10T01:48:26.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:26 smithi181 ceph-mon[94189]: osdmap e1547: 12 total, 12 up, 12 in 2023-03-10T01:48:26.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39016.545505 secs 2023-03-10T01:48:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:26 smithi038 ceph-mon[92422]: osdmap e1547: 12 total, 12 up, 12 in 2023-03-10T01:48:27.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:27 smithi122 ceph-mon[93869]: pgmap v21318: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 776 KiB/s rd, 800 KiB/s wr, 13 op/s 2023-03-10T01:48:27.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:27 smithi181 ceph-mon[94189]: pgmap v21318: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 776 KiB/s rd, 800 KiB/s wr, 13 op/s 2023-03-10T01:48:27.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:27 smithi038 ceph-mon[92422]: pgmap v21318: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 776 KiB/s rd, 800 KiB/s wr, 13 op/s 2023-03-10T01:48:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:28 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:48:28.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:28 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:48:28.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:28 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:48:28.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:28 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:48:28.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:28 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:48:28.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:28 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:48:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39480.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:48:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:29 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39019.006186 secs 2023-03-10T01:48:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:29 smithi122 ceph-mon[93869]: pgmap v21319: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 830 KiB/s rd, 855 KiB/s wr, 13 op/s 2023-03-10T01:48:29.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:29 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39019.006186 secs 2023-03-10T01:48:29.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:29 smithi181 ceph-mon[94189]: pgmap v21319: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 830 KiB/s rd, 855 KiB/s wr, 13 op/s 2023-03-10T01:48:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:29 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39019.006186 secs 2023-03-10T01:48:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:29 smithi038 ceph-mon[92422]: pgmap v21319: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 830 KiB/s rd, 855 KiB/s wr, 13 op/s 2023-03-10T01:48:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:31 smithi122 ceph-mon[93869]: pgmap v21320: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 830 KiB/s rd, 842 KiB/s wr, 11 op/s 2023-03-10T01:48:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39021.545561 secs 2023-03-10T01:48:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:31 smithi181 ceph-mon[94189]: pgmap v21320: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 830 KiB/s rd, 842 KiB/s wr, 11 op/s 2023-03-10T01:48:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39021.545561 secs 2023-03-10T01:48:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:31 smithi038 ceph-mon[92422]: pgmap v21320: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 830 KiB/s rd, 842 KiB/s wr, 11 op/s 2023-03-10T01:48:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39021.545561 secs 2023-03-10T01:48:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:32 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:48:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:32 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:48:32.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:32 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:48:32.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:32 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:48:32.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:32 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:48:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:32 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:48:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:33 smithi122 ceph-mon[93869]: pgmap v21321: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 472 KiB/s wr, 9 op/s 2023-03-10T01:48:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:33 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:48:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:33 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:48:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39024.006304 secs 2023-03-10T01:48:33.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:33 smithi181 ceph-mon[94189]: pgmap v21321: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 472 KiB/s wr, 9 op/s 2023-03-10T01:48:33.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:33 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:48:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:33 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:48:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39024.006304 secs 2023-03-10T01:48:33.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:33 smithi038 ceph-mon[92422]: pgmap v21321: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 472 KiB/s wr, 9 op/s 2023-03-10T01:48:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:33 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:48:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:33 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:48:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39024.006304 secs 2023-03-10T01:48:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39485.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:48:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:35 smithi122 ceph-mon[93869]: pgmap v21322: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 236 KiB/s wr, 8 op/s 2023-03-10T01:48:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:35 smithi122 ceph-mon[93869]: osdmap e1548: 12 total, 12 up, 12 in 2023-03-10T01:48:35.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:35 smithi181 ceph-mon[94189]: pgmap v21322: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 236 KiB/s wr, 8 op/s 2023-03-10T01:48:35.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:35 smithi181 ceph-mon[94189]: osdmap e1548: 12 total, 12 up, 12 in 2023-03-10T01:48:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:35 smithi038 ceph-mon[92422]: pgmap v21322: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 238 KiB/s rd, 236 KiB/s wr, 8 op/s 2023-03-10T01:48:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:35 smithi038 ceph-mon[92422]: osdmap e1548: 12 total, 12 up, 12 in 2023-03-10T01:48:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39026.545622 secs 2023-03-10T01:48:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:48:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:48:36.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39026.545622 secs 2023-03-10T01:48:36.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:48:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:48:36.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39026.545622 secs 2023-03-10T01:48:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:48:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:48:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:37 smithi122 ceph-mon[93869]: pgmap v21324: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 364 KiB/s wr, 8 op/s 2023-03-10T01:48:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:48:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:48:37.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:37 smithi181 ceph-mon[94189]: pgmap v21324: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 364 KiB/s wr, 8 op/s 2023-03-10T01:48:37.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:48:37.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:48:37.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:37 smithi038 ceph-mon[92422]: pgmap v21324: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 366 KiB/s rd, 364 KiB/s wr, 8 op/s 2023-03-10T01:48:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:48:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:48:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39029.006410 secs 2023-03-10T01:48:38.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39029.006410 secs 2023-03-10T01:48:38.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39029.006410 secs 2023-03-10T01:48:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39490.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:48:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:39 smithi122 ceph-mon[93869]: pgmap v21325: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 863 KiB/s rd, 934 KiB/s wr, 10 op/s 2023-03-10T01:48:39.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:39 smithi181 ceph-mon[94189]: pgmap v21325: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 863 KiB/s rd, 934 KiB/s wr, 10 op/s 2023-03-10T01:48:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:39 smithi038 ceph-mon[92422]: pgmap v21325: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 863 KiB/s rd, 934 KiB/s wr, 10 op/s 2023-03-10T01:48:40.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:48:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:48:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:40 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:48:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:48:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:48:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:40 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:48:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:48:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:48:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:40 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:48:41.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:41 smithi038 ceph-mon[92422]: pgmap v21326: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 863 KiB/s rd, 934 KiB/s wr, 10 op/s 2023-03-10T01:48:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39031.545689 secs 2023-03-10T01:48:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:48:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:48:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:48:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:48:41.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:48:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:41 smithi122 ceph-mon[93869]: pgmap v21326: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 863 KiB/s rd, 934 KiB/s wr, 10 op/s 2023-03-10T01:48:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39031.545689 secs 2023-03-10T01:48:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:48:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:48:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:48:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:48:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:48:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:41 smithi181 ceph-mon[94189]: pgmap v21326: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 863 KiB/s rd, 934 KiB/s wr, 10 op/s 2023-03-10T01:48:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39031.545689 secs 2023-03-10T01:48:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:48:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:48:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:48:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:48:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:48:43.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:43 smithi038 ceph-mon[92422]: pgmap v21327: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 10 op/s 2023-03-10T01:48:43.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39034.006471 secs 2023-03-10T01:48:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:43 smithi122 ceph-mon[93869]: pgmap v21327: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 10 op/s 2023-03-10T01:48:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39034.006471 secs 2023-03-10T01:48:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:43 smithi181 ceph-mon[94189]: pgmap v21327: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 10 op/s 2023-03-10T01:48:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39034.006471 secs 2023-03-10T01:48:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39495.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:48:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:45 smithi122 ceph-mon[93869]: pgmap v21328: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 11 op/s 2023-03-10T01:48:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:45 smithi181 ceph-mon[94189]: pgmap v21328: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 11 op/s 2023-03-10T01:48:45.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:45 smithi038 ceph-mon[92422]: pgmap v21328: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 11 op/s 2023-03-10T01:48:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39036.545746 secs 2023-03-10T01:48:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39036.545746 secs 2023-03-10T01:48:46.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39036.545746 secs 2023-03-10T01:48:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:47 smithi122 ceph-mon[93869]: pgmap v21329: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 10 op/s 2023-03-10T01:48:47.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:47 smithi181 ceph-mon[94189]: pgmap v21329: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 10 op/s 2023-03-10T01:48:47.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:47 smithi038 ceph-mon[92422]: pgmap v21329: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 10 op/s 2023-03-10T01:48:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:48:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:48:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:48:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:48:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:48:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:48:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39500.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:48:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:49 smithi122 ceph-mon[93869]: pgmap v21330: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 898 KiB/s rd, 896 KiB/s wr, 12 op/s 2023-03-10T01:48:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:48:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:48:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:49 smithi181 ceph-mon[94189]: pgmap v21330: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 898 KiB/s rd, 896 KiB/s wr, 12 op/s 2023-03-10T01:48:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:48:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:48:49.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:49 smithi038 ceph-mon[92422]: pgmap v21330: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 898 KiB/s rd, 896 KiB/s wr, 12 op/s 2023-03-10T01:48:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:48:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:48:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:50 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:48:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:50 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:48:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:50 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:48:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:51 smithi122 ceph-mon[93869]: pgmap v21331: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 538 KiB/s wr, 8 op/s 2023-03-10T01:48:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39041.545829 secs 2023-03-10T01:48:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:51 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:48:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:48:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:48:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:51 smithi181 ceph-mon[94189]: pgmap v21331: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 538 KiB/s wr, 8 op/s 2023-03-10T01:48:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39041.545829 secs 2023-03-10T01:48:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:51 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:48:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:48:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:48:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:51 smithi038 ceph-mon[92422]: pgmap v21331: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 538 KiB/s wr, 8 op/s 2023-03-10T01:48:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39041.545829 secs 2023-03-10T01:48:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:51 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:48:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:48:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:48:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:53 smithi122 ceph-mon[93869]: pgmap v21332: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 767 KiB/s rd, 571 KiB/s wr, 12 op/s 2023-03-10T01:48:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:53 smithi181 ceph-mon[94189]: pgmap v21332: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 767 KiB/s rd, 571 KiB/s wr, 12 op/s 2023-03-10T01:48:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:53 smithi038 ceph-mon[92422]: pgmap v21332: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 767 KiB/s rd, 571 KiB/s wr, 12 op/s 2023-03-10T01:48:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39505.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:48:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:55 smithi122 ceph-mon[93869]: pgmap v21333: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 827 KiB/s rd, 605 KiB/s wr, 10 op/s 2023-03-10T01:48:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:55 smithi181 ceph-mon[94189]: pgmap v21333: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 827 KiB/s rd, 605 KiB/s wr, 10 op/s 2023-03-10T01:48:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:55 smithi038 ceph-mon[92422]: pgmap v21333: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 827 KiB/s rd, 605 KiB/s wr, 10 op/s 2023-03-10T01:48:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39046.545887 secs 2023-03-10T01:48:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:48:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:48:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39046.545887 secs 2023-03-10T01:48:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:48:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:48:56.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39046.545887 secs 2023-03-10T01:48:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:48:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:48:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:57 smithi122 ceph-mon[93869]: pgmap v21334: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 427 KiB/s wr, 9 op/s 2023-03-10T01:48:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:48:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:48:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:57 smithi181 ceph-mon[94189]: pgmap v21334: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 427 KiB/s wr, 9 op/s 2023-03-10T01:48:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:48:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:48:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:57 smithi038 ceph-mon[92422]: pgmap v21334: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 427 KiB/s wr, 9 op/s 2023-03-10T01:48:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:48:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:48:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39510.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:48:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:48:59 smithi122 ceph-mon[93869]: pgmap v21335: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 830 KiB/s rd, 605 KiB/s wr, 13 op/s 2023-03-10T01:48:59.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:48:59 smithi181 ceph-mon[94189]: pgmap v21335: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 830 KiB/s rd, 605 KiB/s wr, 13 op/s 2023-03-10T01:48:59.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:48:59 smithi038 ceph-mon[92422]: pgmap v21335: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 830 KiB/s rd, 605 KiB/s wr, 13 op/s 2023-03-10T01:49:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:49:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:49:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:49:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:49:00.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:49:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:49:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:01 smithi122 ceph-mon[93869]: pgmap v21336: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 913 KiB/s wr, 11 op/s 2023-03-10T01:49:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39051.545993 secs 2023-03-10T01:49:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:49:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:49:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:01 smithi181 ceph-mon[94189]: pgmap v21336: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 913 KiB/s wr, 11 op/s 2023-03-10T01:49:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39051.545993 secs 2023-03-10T01:49:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:49:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:49:01.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:01 smithi038 ceph-mon[92422]: pgmap v21336: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 913 KiB/s wr, 11 op/s 2023-03-10T01:49:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39051.545993 secs 2023-03-10T01:49:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:49:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:49:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:03 smithi122 ceph-mon[93869]: pgmap v21337: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 969 KiB/s wr, 15 op/s 2023-03-10T01:49:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:03 smithi181 ceph-mon[94189]: pgmap v21337: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 969 KiB/s wr, 15 op/s 2023-03-10T01:49:03.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:03 smithi038 ceph-mon[92422]: pgmap v21337: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 969 KiB/s wr, 15 op/s 2023-03-10T01:49:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39515.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:49:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:05 smithi122 ceph-mon[93869]: pgmap v21338: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 946 KiB/s wr, 23 op/s 2023-03-10T01:49:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:05 smithi181 ceph-mon[94189]: pgmap v21338: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 946 KiB/s wr, 23 op/s 2023-03-10T01:49:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:05 smithi038 ceph-mon[92422]: pgmap v21338: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 946 KiB/s wr, 23 op/s 2023-03-10T01:49:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39056.546053 secs 2023-03-10T01:49:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39056.546053 secs 2023-03-10T01:49:06.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39056.546053 secs 2023-03-10T01:49:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:07 smithi122 ceph-mon[93869]: pgmap v21339: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 755 KiB/s rd, 749 KiB/s wr, 22 op/s 2023-03-10T01:49:07.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:07 smithi181 ceph-mon[94189]: pgmap v21339: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 755 KiB/s rd, 749 KiB/s wr, 22 op/s 2023-03-10T01:49:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:07 smithi038 ceph-mon[92422]: pgmap v21339: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 755 KiB/s rd, 749 KiB/s wr, 22 op/s 2023-03-10T01:49:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:49:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:49:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:49:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:49:08.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:49:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:49:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39520.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:49:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:09 smithi122 ceph-mon[93869]: pgmap v21340: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.0 MiB/s wr, 28 op/s 2023-03-10T01:49:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:49:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:49:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:09 smithi181 ceph-mon[94189]: pgmap v21340: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.0 MiB/s wr, 28 op/s 2023-03-10T01:49:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:49:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:49:09.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:09 smithi038 ceph-mon[92422]: pgmap v21340: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.0 MiB/s wr, 28 op/s 2023-03-10T01:49:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:49:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:49:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:11 smithi122 ceph-mon[93869]: pgmap v21341: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 885 KiB/s wr, 24 op/s 2023-03-10T01:49:11.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39061.546111 secs 2023-03-10T01:49:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:11 smithi181 ceph-mon[94189]: pgmap v21341: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 885 KiB/s wr, 24 op/s 2023-03-10T01:49:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39061.546111 secs 2023-03-10T01:49:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:11 smithi038 ceph-mon[92422]: pgmap v21341: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 885 KiB/s wr, 24 op/s 2023-03-10T01:49:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39061.546111 secs 2023-03-10T01:49:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:49:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:49:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:49:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:49:12.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:49:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:49:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:13 smithi122 ceph-mon[93869]: pgmap v21342: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 735 KiB/s wr, 26 op/s 2023-03-10T01:49:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:13 smithi181 ceph-mon[94189]: pgmap v21342: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 735 KiB/s wr, 26 op/s 2023-03-10T01:49:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:13 smithi038 ceph-mon[92422]: pgmap v21342: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 735 KiB/s wr, 26 op/s 2023-03-10T01:49:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39525.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:49:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:15 smithi122 ceph-mon[93869]: pgmap v21343: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 933 KiB/s rd, 516 KiB/s wr, 21 op/s 2023-03-10T01:49:15.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:15 smithi181 ceph-mon[94189]: pgmap v21343: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 933 KiB/s rd, 516 KiB/s wr, 21 op/s 2023-03-10T01:49:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:15 smithi038 ceph-mon[92422]: pgmap v21343: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 933 KiB/s rd, 516 KiB/s wr, 21 op/s 2023-03-10T01:49:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39066.546172 secs 2023-03-10T01:49:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39066.546172 secs 2023-03-10T01:49:16.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39066.546172 secs 2023-03-10T01:49:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:17 smithi122 ceph-mon[93869]: pgmap v21344: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 926 KiB/s rd, 505 KiB/s wr, 10 op/s 2023-03-10T01:49:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:17 smithi181 ceph-mon[94189]: pgmap v21344: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 926 KiB/s rd, 505 KiB/s wr, 10 op/s 2023-03-10T01:49:17.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:17 smithi038 ceph-mon[92422]: pgmap v21344: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 926 KiB/s rd, 505 KiB/s wr, 10 op/s 2023-03-10T01:49:18.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39530.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:49:19.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:19 smithi122 ceph-mon[93869]: pgmap v21345: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 922 KiB/s rd, 505 KiB/s wr, 12 op/s 2023-03-10T01:49:19.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:19 smithi181 ceph-mon[94189]: pgmap v21345: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 922 KiB/s rd, 505 KiB/s wr, 12 op/s 2023-03-10T01:49:19.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:19 smithi038 ceph-mon[92422]: pgmap v21345: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 922 KiB/s rd, 505 KiB/s wr, 12 op/s 2023-03-10T01:49:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:49:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:49:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:49:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:49:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:49:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:49:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:21 smithi122 ceph-mon[93869]: pgmap v21346: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 202 KiB/s wr, 8 op/s 2023-03-10T01:49:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39071.546228 secs 2023-03-10T01:49:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:49:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:49:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:21 smithi181 ceph-mon[94189]: pgmap v21346: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 202 KiB/s wr, 8 op/s 2023-03-10T01:49:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39071.546228 secs 2023-03-10T01:49:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:49:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:49:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:21 smithi038 ceph-mon[92422]: pgmap v21346: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 202 KiB/s wr, 8 op/s 2023-03-10T01:49:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39071.546228 secs 2023-03-10T01:49:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:49:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:49:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:23 smithi122 ceph-mon[93869]: pgmap v21347: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 590 KiB/s rd, 608 KiB/s wr, 12 op/s 2023-03-10T01:49:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:23 smithi181 ceph-mon[94189]: pgmap v21347: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 590 KiB/s rd, 608 KiB/s wr, 12 op/s 2023-03-10T01:49:23.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:23 smithi038 ceph-mon[92422]: pgmap v21347: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 590 KiB/s rd, 608 KiB/s wr, 12 op/s 2023-03-10T01:49:23.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39535.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:49:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:25 smithi122 ceph-mon[93869]: pgmap v21348: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 398 KiB/s rd, 417 KiB/s wr, 10 op/s 2023-03-10T01:49:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:25 smithi122 ceph-mon[93869]: osdmap e1549: 12 total, 12 up, 12 in 2023-03-10T01:49:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:25 smithi181 ceph-mon[94189]: pgmap v21348: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 398 KiB/s rd, 417 KiB/s wr, 10 op/s 2023-03-10T01:49:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:25 smithi181 ceph-mon[94189]: osdmap e1549: 12 total, 12 up, 12 in 2023-03-10T01:49:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:25 smithi038 ceph-mon[92422]: pgmap v21348: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 398 KiB/s rd, 417 KiB/s wr, 10 op/s 2023-03-10T01:49:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:25 smithi038 ceph-mon[92422]: osdmap e1549: 12 total, 12 up, 12 in 2023-03-10T01:49:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39076.546286 secs 2023-03-10T01:49:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39076.546286 secs 2023-03-10T01:49:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39076.546286 secs 2023-03-10T01:49:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:27 smithi122 ceph-mon[93869]: pgmap v21350: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 672 KiB/s rd, 696 KiB/s wr, 12 op/s 2023-03-10T01:49:27.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:27 smithi181 ceph-mon[94189]: pgmap v21350: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 672 KiB/s rd, 696 KiB/s wr, 12 op/s 2023-03-10T01:49:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:27 smithi038 ceph-mon[92422]: pgmap v21350: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 672 KiB/s rd, 696 KiB/s wr, 12 op/s 2023-03-10T01:49:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:28 smithi038 ceph-mon[92422]: pgmap v21351: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 673 KiB/s rd, 696 KiB/s wr, 13 op/s 2023-03-10T01:49:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39540.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:49:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:28 smithi122 ceph-mon[93869]: pgmap v21351: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 673 KiB/s rd, 696 KiB/s wr, 13 op/s 2023-03-10T01:49:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:28 smithi181 ceph-mon[94189]: pgmap v21351: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 673 KiB/s rd, 696 KiB/s wr, 13 op/s 2023-03-10T01:49:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:31 smithi122 ceph-mon[93869]: pgmap v21352: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 672 KiB/s rd, 684 KiB/s wr, 11 op/s 2023-03-10T01:49:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:31 smithi122 ceph-mon[93869]: osdmap e1550: 12 total, 12 up, 12 in 2023-03-10T01:49:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:31 smithi181 ceph-mon[94189]: pgmap v21352: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 672 KiB/s rd, 684 KiB/s wr, 11 op/s 2023-03-10T01:49:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:31 smithi181 ceph-mon[94189]: osdmap e1550: 12 total, 12 up, 12 in 2023-03-10T01:49:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:31 smithi038 ceph-mon[92422]: pgmap v21352: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 672 KiB/s rd, 684 KiB/s wr, 11 op/s 2023-03-10T01:49:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:31 smithi038 ceph-mon[92422]: osdmap e1550: 12 total, 12 up, 12 in 2023-03-10T01:49:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:32 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39081.546347 secs 2023-03-10T01:49:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:32 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:49:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:32 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:49:32.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:32 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39081.546347 secs 2023-03-10T01:49:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:32 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:49:32.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:32 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:49:32.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:32 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39081.546347 secs 2023-03-10T01:49:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:32 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:49:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:32 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:49:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:33 smithi122 ceph-mon[93869]: pgmap v21354: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 542 KiB/s wr, 10 op/s 2023-03-10T01:49:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:33 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:49:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:33 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:49:33.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:33 smithi181 ceph-mon[94189]: pgmap v21354: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 542 KiB/s wr, 10 op/s 2023-03-10T01:49:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:33 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:49:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:33 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:49:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:33 smithi038 ceph-mon[92422]: pgmap v21354: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 542 KiB/s wr, 10 op/s 2023-03-10T01:49:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:33 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:49:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:33 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:49:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39545.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:49:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:35 smithi122 ceph-mon[93869]: pgmap v21355: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 473 KiB/s wr, 9 op/s 2023-03-10T01:49:35.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:35 smithi181 ceph-mon[94189]: pgmap v21355: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 473 KiB/s wr, 9 op/s 2023-03-10T01:49:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:35 smithi038 ceph-mon[92422]: pgmap v21355: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 473 KiB/s wr, 9 op/s 2023-03-10T01:49:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:49:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:49:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39086.546404 secs 2023-03-10T01:49:36.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:49:36.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:49:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39086.546404 secs 2023-03-10T01:49:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:49:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:49:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39086.546404 secs 2023-03-10T01:49:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:37 smithi122 ceph-mon[93869]: pgmap v21356: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 847 KiB/s rd, 255 KiB/s wr, 9 op/s 2023-03-10T01:49:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:49:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:49:37.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:37 smithi181 ceph-mon[94189]: pgmap v21356: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 847 KiB/s rd, 255 KiB/s wr, 9 op/s 2023-03-10T01:49:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:49:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:49:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:37 smithi038 ceph-mon[92422]: pgmap v21356: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 847 KiB/s rd, 255 KiB/s wr, 9 op/s 2023-03-10T01:49:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:49:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:49:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:38 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:49:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:38 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:49:38.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:38 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:49:38.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:38 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:49:38.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:38 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:49:38.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:38 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:49:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:38 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:49:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:38 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:49:38.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:38 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:49:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39550.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:49:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:39 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39089.007301 secs 2023-03-10T01:49:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:39 smithi122 ceph-mon[93869]: pgmap v21357: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 704 KiB/s wr, 9 op/s 2023-03-10T01:49:39.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:39 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39089.007301 secs 2023-03-10T01:49:39.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:39 smithi181 ceph-mon[94189]: pgmap v21357: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 704 KiB/s wr, 9 op/s 2023-03-10T01:49:39.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:39 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39089.007301 secs 2023-03-10T01:49:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:39 smithi038 ceph-mon[92422]: pgmap v21357: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 704 KiB/s wr, 9 op/s 2023-03-10T01:49:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:49:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:49:40.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:49:40.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:49:40.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:49:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:49:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:41 smithi122 ceph-mon[93869]: pgmap v21358: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 704 KiB/s wr, 9 op/s 2023-03-10T01:49:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39091.546491 secs 2023-03-10T01:49:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:49:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:49:41.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:49:41.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:41 smithi181 ceph-mon[94189]: pgmap v21358: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 704 KiB/s wr, 9 op/s 2023-03-10T01:49:41.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39091.546491 secs 2023-03-10T01:49:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:49:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:49:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:49:41.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:41 smithi038 ceph-mon[92422]: pgmap v21358: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 704 KiB/s wr, 9 op/s 2023-03-10T01:49:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39091.546491 secs 2023-03-10T01:49:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:49:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:49:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:49:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:49:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:49:42.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:49:42.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:49:42.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:49:42.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:49:42.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:49:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:49:42.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:49:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:43 smithi122 ceph-mon[93869]: pgmap v21359: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 949 KiB/s wr, 12 op/s 2023-03-10T01:49:43.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:43 smithi181 ceph-mon[94189]: pgmap v21359: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 949 KiB/s wr, 12 op/s 2023-03-10T01:49:43.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:43 smithi038 ceph-mon[92422]: pgmap v21359: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 949 KiB/s wr, 12 op/s 2023-03-10T01:49:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39555.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:49:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:44 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39094.007359 secs 2023-03-10T01:49:44.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:44 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39094.007359 secs 2023-03-10T01:49:44.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:44 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39094.007359 secs 2023-03-10T01:49:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:45 smithi122 ceph-mon[93869]: pgmap v21360: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 718 KiB/s wr, 9 op/s 2023-03-10T01:49:45.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:45 smithi181 ceph-mon[94189]: pgmap v21360: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 718 KiB/s wr, 9 op/s 2023-03-10T01:49:45.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:45 smithi038 ceph-mon[92422]: pgmap v21360: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 718 KiB/s wr, 9 op/s 2023-03-10T01:49:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39096.546550 secs 2023-03-10T01:49:46.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39096.546550 secs 2023-03-10T01:49:46.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39096.546550 secs 2023-03-10T01:49:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:47 smithi122 ceph-mon[93869]: pgmap v21361: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 718 KiB/s wr, 9 op/s 2023-03-10T01:49:47.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:47 smithi181 ceph-mon[94189]: pgmap v21361: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 718 KiB/s wr, 9 op/s 2023-03-10T01:49:47.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:47 smithi038 ceph-mon[92422]: pgmap v21361: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 718 KiB/s wr, 9 op/s 2023-03-10T01:49:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:49:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:49:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:49:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:49:48.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:49:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:49:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39560.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:49:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:49 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 39099.007447 secs 2023-03-10T01:49:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:49 smithi122 ceph-mon[93869]: pgmap v21362: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 12 op/s 2023-03-10T01:49:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:49:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:49:49.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:49 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 39099.007447 secs 2023-03-10T01:49:49.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:49 smithi181 ceph-mon[94189]: pgmap v21362: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 12 op/s 2023-03-10T01:49:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:49:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:49:49.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:49 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 39099.007447 secs 2023-03-10T01:49:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:49 smithi038 ceph-mon[92422]: pgmap v21362: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 12 op/s 2023-03-10T01:49:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:49:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:49:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:51 smithi122 ceph-mon[93869]: pgmap v21363: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 672 KiB/s rd, 670 KiB/s wr, 9 op/s 2023-03-10T01:49:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39101.546599 secs 2023-03-10T01:49:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:49:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:49:51.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:51 smithi181 ceph-mon[94189]: pgmap v21363: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 672 KiB/s rd, 670 KiB/s wr, 9 op/s 2023-03-10T01:49:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39101.546599 secs 2023-03-10T01:49:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:49:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:49:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:51 smithi038 ceph-mon[92422]: pgmap v21363: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 672 KiB/s rd, 670 KiB/s wr, 9 op/s 2023-03-10T01:49:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39101.546599 secs 2023-03-10T01:49:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:49:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:49:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:53 smithi122 ceph-mon[93869]: pgmap v21364: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 933 KiB/s wr, 12 op/s 2023-03-10T01:49:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:53 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 39104.007509 secs 2023-03-10T01:49:53.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:53 smithi181 ceph-mon[94189]: pgmap v21364: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 933 KiB/s wr, 12 op/s 2023-03-10T01:49:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:53 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 39104.007509 secs 2023-03-10T01:49:53.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:53 smithi038 ceph-mon[92422]: pgmap v21364: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 933 KiB/s wr, 12 op/s 2023-03-10T01:49:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:53 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 39104.007509 secs 2023-03-10T01:49:53.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39565.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:49:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:55 smithi122 ceph-mon[93869]: pgmap v21365: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 604 KiB/s wr, 8 op/s 2023-03-10T01:49:55.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:55 smithi181 ceph-mon[94189]: pgmap v21365: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 604 KiB/s wr, 8 op/s 2023-03-10T01:49:55.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:55 smithi038 ceph-mon[92422]: pgmap v21365: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 604 KiB/s wr, 8 op/s 2023-03-10T01:49:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39106.546680 secs 2023-03-10T01:49:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:49:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:49:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39106.546680 secs 2023-03-10T01:49:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:49:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:49:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39106.546680 secs 2023-03-10T01:49:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:49:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:49:57.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:57 smithi181 ceph-mon[94189]: pgmap v21366: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 604 KiB/s wr, 8 op/s 2023-03-10T01:49:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:49:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:49:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:57 smithi038 ceph-mon[92422]: pgmap v21366: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 604 KiB/s wr, 8 op/s 2023-03-10T01:49:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:49:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:49:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:57 smithi122 ceph-mon[93869]: pgmap v21366: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 604 KiB/s wr, 8 op/s 2023-03-10T01:49:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:49:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:49:58.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39109.007568 secs 2023-03-10T01:49:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39109.007568 secs 2023-03-10T01:49:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39109.007568 secs 2023-03-10T01:49:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39570.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:49:59.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:59 smithi038 ceph-mon[92422]: pgmap v21367: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 750 KiB/s wr, 12 op/s 2023-03-10T01:49:59.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:49:59.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:49:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:49:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:59 smithi122 ceph-mon[93869]: pgmap v21367: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 750 KiB/s wr, 12 op/s 2023-03-10T01:49:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:49:59.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:49:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:49:59.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:59 smithi181 ceph-mon[94189]: pgmap v21367: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 750 KiB/s wr, 12 op/s 2023-03-10T01:49:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:49:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:49:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:50:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:50:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:50:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:00 smithi122 ceph-mon[93869]: overall HEALTH_WARN 2 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-10T01:50:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:50:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:50:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:00 smithi181 ceph-mon[94189]: overall HEALTH_WARN 2 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-10T01:50:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:50:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:50:00.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:00 smithi038 ceph-mon[92422]: overall HEALTH_WARN 2 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-10T01:50:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:01 smithi122 ceph-mon[93869]: pgmap v21368: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 991 KiB/s rd, 757 KiB/s wr, 9 op/s 2023-03-10T01:50:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39111.546741 secs 2023-03-10T01:50:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:50:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:50:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:01 smithi181 ceph-mon[94189]: pgmap v21368: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 991 KiB/s rd, 757 KiB/s wr, 9 op/s 2023-03-10T01:50:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39111.546741 secs 2023-03-10T01:50:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:50:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:50:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:01 smithi038 ceph-mon[92422]: pgmap v21368: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 991 KiB/s rd, 757 KiB/s wr, 9 op/s 2023-03-10T01:50:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39111.546741 secs 2023-03-10T01:50:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:50:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:50:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:03 smithi122 ceph-mon[93869]: pgmap v21369: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 792 KiB/s wr, 13 op/s 2023-03-10T01:50:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39114.007630 secs 2023-03-10T01:50:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:03 smithi181 ceph-mon[94189]: pgmap v21369: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 792 KiB/s wr, 13 op/s 2023-03-10T01:50:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39114.007630 secs 2023-03-10T01:50:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:03 smithi038 ceph-mon[92422]: pgmap v21369: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 792 KiB/s wr, 13 op/s 2023-03-10T01:50:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39114.007630 secs 2023-03-10T01:50:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39575.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:50:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:05 smithi122 ceph-mon[93869]: pgmap v21370: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 875 KiB/s wr, 22 op/s 2023-03-10T01:50:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:05 smithi181 ceph-mon[94189]: pgmap v21370: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 875 KiB/s wr, 22 op/s 2023-03-10T01:50:05.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:05 smithi038 ceph-mon[92422]: pgmap v21370: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 875 KiB/s wr, 22 op/s 2023-03-10T01:50:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39116.546800 secs 2023-03-10T01:50:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39116.546800 secs 2023-03-10T01:50:06.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39116.546800 secs 2023-03-10T01:50:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:07 smithi122 ceph-mon[93869]: pgmap v21371: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 875 KiB/s wr, 22 op/s 2023-03-10T01:50:07.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:07 smithi181 ceph-mon[94189]: pgmap v21371: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 875 KiB/s wr, 22 op/s 2023-03-10T01:50:07.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:07 smithi038 ceph-mon[92422]: pgmap v21371: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 875 KiB/s wr, 22 op/s 2023-03-10T01:50:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:50:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:50:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39119.007743 secs 2023-03-10T01:50:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:50:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:50:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39119.007743 secs 2023-03-10T01:50:08.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:50:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:50:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39119.007743 secs 2023-03-10T01:50:08.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39580.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:50:09.561 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:09 smithi038 ceph-mon[92422]: pgmap v21372: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1020 KiB/s wr, 26 op/s 2023-03-10T01:50:09.562 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:50:09.562 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:50:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:09 smithi122 ceph-mon[93869]: pgmap v21372: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1020 KiB/s wr, 26 op/s 2023-03-10T01:50:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:50:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:50:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:09 smithi181 ceph-mon[94189]: pgmap v21372: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1020 KiB/s wr, 26 op/s 2023-03-10T01:50:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:50:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:50:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:11 smithi122 ceph-mon[93869]: pgmap v21373: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 874 KiB/s wr, 23 op/s 2023-03-10T01:50:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39121.546884 secs 2023-03-10T01:50:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:11 smithi181 ceph-mon[94189]: pgmap v21373: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 874 KiB/s wr, 23 op/s 2023-03-10T01:50:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39121.546884 secs 2023-03-10T01:50:11.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:11 smithi038 ceph-mon[92422]: pgmap v21373: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 874 KiB/s wr, 23 op/s 2023-03-10T01:50:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39121.546884 secs 2023-03-10T01:50:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:50:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:50:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:50:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:50:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:50:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:50:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:13 smithi122 ceph-mon[93869]: pgmap v21374: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 669 KiB/s wr, 25 op/s 2023-03-10T01:50:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39124.007797 secs 2023-03-10T01:50:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:13 smithi181 ceph-mon[94189]: pgmap v21374: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 669 KiB/s wr, 25 op/s 2023-03-10T01:50:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39124.007797 secs 2023-03-10T01:50:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:13 smithi038 ceph-mon[92422]: pgmap v21374: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 669 KiB/s wr, 25 op/s 2023-03-10T01:50:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39124.007797 secs 2023-03-10T01:50:13.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39585.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:50:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:15 smithi122 ceph-mon[93869]: pgmap v21375: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 841 KiB/s wr, 23 op/s 2023-03-10T01:50:15.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:15 smithi181 ceph-mon[94189]: pgmap v21375: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 841 KiB/s wr, 23 op/s 2023-03-10T01:50:15.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:15 smithi038 ceph-mon[92422]: pgmap v21375: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 841 KiB/s wr, 23 op/s 2023-03-10T01:50:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39126.546967 secs 2023-03-10T01:50:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:50:16.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:50:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39126.546967 secs 2023-03-10T01:50:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:50:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:50:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39126.546967 secs 2023-03-10T01:50:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:50:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:50:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:17 smithi122 ceph-mon[93869]: pgmap v21376: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 596 KiB/s rd, 496 KiB/s wr, 10 op/s 2023-03-10T01:50:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:50:17.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:50:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:17 smithi181 ceph-mon[94189]: pgmap v21376: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 596 KiB/s rd, 496 KiB/s wr, 10 op/s 2023-03-10T01:50:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:50:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:50:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:17 smithi038 ceph-mon[92422]: pgmap v21376: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 596 KiB/s rd, 496 KiB/s wr, 10 op/s 2023-03-10T01:50:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:50:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:50:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39129.007878 secs 2023-03-10T01:50:18.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39129.007878 secs 2023-03-10T01:50:18.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39129.007878 secs 2023-03-10T01:50:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39590.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:50:19.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:19 smithi122 ceph-mon[93869]: pgmap v21377: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 597 KiB/s rd, 496 KiB/s wr, 13 op/s 2023-03-10T01:50:19.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:19 smithi181 ceph-mon[94189]: pgmap v21377: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 597 KiB/s rd, 496 KiB/s wr, 13 op/s 2023-03-10T01:50:19.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:19 smithi038 ceph-mon[92422]: pgmap v21377: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 597 KiB/s rd, 496 KiB/s wr, 13 op/s 2023-03-10T01:50:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:50:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:50:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:50:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:50:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:50:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:50:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:21 smithi122 ceph-mon[93869]: pgmap v21378: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 362 KiB/s wr, 11 op/s 2023-03-10T01:50:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39131.546978 secs 2023-03-10T01:50:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:50:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:50:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:21 smithi181 ceph-mon[94189]: pgmap v21378: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 362 KiB/s wr, 11 op/s 2023-03-10T01:50:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39131.546978 secs 2023-03-10T01:50:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:50:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:50:21.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:21 smithi038 ceph-mon[92422]: pgmap v21378: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 362 KiB/s wr, 11 op/s 2023-03-10T01:50:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39131.546978 secs 2023-03-10T01:50:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:50:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:50:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:23 smithi122 ceph-mon[93869]: pgmap v21379: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 772 KiB/s rd, 932 KiB/s wr, 14 op/s 2023-03-10T01:50:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39134.007967 secs 2023-03-10T01:50:23.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:23 smithi181 ceph-mon[94189]: pgmap v21379: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 772 KiB/s rd, 932 KiB/s wr, 14 op/s 2023-03-10T01:50:23.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39134.007967 secs 2023-03-10T01:50:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:23 smithi038 ceph-mon[92422]: pgmap v21379: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 772 KiB/s rd, 932 KiB/s wr, 14 op/s 2023-03-10T01:50:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39134.007967 secs 2023-03-10T01:50:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39595.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:50:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:25 smithi122 ceph-mon[93869]: pgmap v21380: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 789 KiB/s wr, 12 op/s 2023-03-10T01:50:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:25 smithi122 ceph-mon[93869]: osdmap e1551: 12 total, 12 up, 12 in 2023-03-10T01:50:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:25 smithi181 ceph-mon[94189]: pgmap v21380: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 789 KiB/s wr, 12 op/s 2023-03-10T01:50:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:25 smithi181 ceph-mon[94189]: osdmap e1551: 12 total, 12 up, 12 in 2023-03-10T01:50:25.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:25 smithi038 ceph-mon[92422]: pgmap v21380: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 789 KiB/s wr, 12 op/s 2023-03-10T01:50:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:25 smithi038 ceph-mon[92422]: osdmap e1551: 12 total, 12 up, 12 in 2023-03-10T01:50:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39136.547054 secs 2023-03-10T01:50:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39136.547054 secs 2023-03-10T01:50:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39136.547054 secs 2023-03-10T01:50:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:27 smithi122 ceph-mon[93869]: pgmap v21382: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 676 KiB/s rd, 716 KiB/s wr, 16 op/s 2023-03-10T01:50:27.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:27 smithi181 ceph-mon[94189]: pgmap v21382: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 676 KiB/s rd, 716 KiB/s wr, 16 op/s 2023-03-10T01:50:27.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:27 smithi038 ceph-mon[92422]: pgmap v21382: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 676 KiB/s rd, 716 KiB/s wr, 16 op/s 2023-03-10T01:50:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39139.008083 secs 2023-03-10T01:50:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39139.008083 secs 2023-03-10T01:50:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39139.008083 secs 2023-03-10T01:50:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39600.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:50:29.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:29 smithi122 ceph-mon[93869]: pgmap v21383: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 853 KiB/s rd, 894 KiB/s wr, 14 op/s 2023-03-10T01:50:29.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:29 smithi181 ceph-mon[94189]: pgmap v21383: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 853 KiB/s rd, 894 KiB/s wr, 14 op/s 2023-03-10T01:50:29.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:29 smithi038 ceph-mon[92422]: pgmap v21383: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 853 KiB/s rd, 894 KiB/s wr, 14 op/s 2023-03-10T01:50:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:31 smithi122 ceph-mon[93869]: pgmap v21384: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 852 KiB/s rd, 881 KiB/s wr, 11 op/s 2023-03-10T01:50:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39141.547112 secs 2023-03-10T01:50:31.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:31 smithi181 ceph-mon[94189]: pgmap v21384: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 852 KiB/s rd, 881 KiB/s wr, 11 op/s 2023-03-10T01:50:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39141.547112 secs 2023-03-10T01:50:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:31 smithi038 ceph-mon[92422]: pgmap v21384: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 852 KiB/s rd, 881 KiB/s wr, 11 op/s 2023-03-10T01:50:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39141.547112 secs 2023-03-10T01:50:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:33 smithi122 ceph-mon[93869]: pgmap v21385: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 238 KiB/s wr, 10 op/s 2023-03-10T01:50:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:33 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:50:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:33 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:50:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39144.008144 secs 2023-03-10T01:50:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:33 smithi181 ceph-mon[94189]: pgmap v21385: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 238 KiB/s wr, 10 op/s 2023-03-10T01:50:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:33 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:50:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:33 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:50:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39144.008144 secs 2023-03-10T01:50:33.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:33 smithi038 ceph-mon[92422]: pgmap v21385: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 238 KiB/s wr, 10 op/s 2023-03-10T01:50:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:33 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:50:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:33 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:50:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39144.008144 secs 2023-03-10T01:50:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39605.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:50:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:35 smithi122 ceph-mon[93869]: pgmap v21386: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 237 KiB/s wr, 9 op/s 2023-03-10T01:50:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:35 smithi122 ceph-mon[93869]: osdmap e1552: 12 total, 12 up, 12 in 2023-03-10T01:50:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:35 smithi181 ceph-mon[94189]: pgmap v21386: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 237 KiB/s wr, 9 op/s 2023-03-10T01:50:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:35 smithi181 ceph-mon[94189]: osdmap e1552: 12 total, 12 up, 12 in 2023-03-10T01:50:35.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:35 smithi038 ceph-mon[92422]: pgmap v21386: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 237 KiB/s wr, 9 op/s 2023-03-10T01:50:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:35 smithi038 ceph-mon[92422]: osdmap e1552: 12 total, 12 up, 12 in 2023-03-10T01:50:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39146.547176 secs 2023-03-10T01:50:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:50:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:50:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39146.547176 secs 2023-03-10T01:50:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:50:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:50:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39146.547176 secs 2023-03-10T01:50:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:50:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:50:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:37 smithi122 ceph-mon[93869]: pgmap v21388: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 223 KiB/s wr, 9 op/s 2023-03-10T01:50:37.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:37 smithi181 ceph-mon[94189]: pgmap v21388: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 223 KiB/s wr, 9 op/s 2023-03-10T01:50:37.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:37 smithi038 ceph-mon[92422]: pgmap v21388: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 223 KiB/s wr, 9 op/s 2023-03-10T01:50:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39149.008245 secs 2023-03-10T01:50:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39149.008245 secs 2023-03-10T01:50:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39149.008245 secs 2023-03-10T01:50:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39610.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:50:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:39 smithi122 ceph-mon[93869]: pgmap v21389: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 517 KiB/s rd, 498 KiB/s wr, 10 op/s 2023-03-10T01:50:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:39 smithi181 ceph-mon[94189]: pgmap v21389: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 517 KiB/s rd, 498 KiB/s wr, 10 op/s 2023-03-10T01:50:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:39 smithi038 ceph-mon[92422]: pgmap v21389: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 517 KiB/s rd, 498 KiB/s wr, 10 op/s 2023-03-10T01:50:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:50:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:50:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:50:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:50:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:50:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:50:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:41 smithi122 ceph-mon[93869]: pgmap v21390: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 517 KiB/s rd, 498 KiB/s wr, 11 op/s 2023-03-10T01:50:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39151.547231 secs 2023-03-10T01:50:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:50:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:50:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:41 smithi181 ceph-mon[94189]: pgmap v21390: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 517 KiB/s rd, 498 KiB/s wr, 11 op/s 2023-03-10T01:50:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39151.547231 secs 2023-03-10T01:50:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:50:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:50:41.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:41 smithi038 ceph-mon[92422]: pgmap v21390: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 517 KiB/s rd, 498 KiB/s wr, 11 op/s 2023-03-10T01:50:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39151.547231 secs 2023-03-10T01:50:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:50:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:50:42.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:50:42.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:50:42.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:50:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:43 smithi122 ceph-mon[93869]: pgmap v21391: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 869 KiB/s rd, 866 KiB/s wr, 11 op/s 2023-03-10T01:50:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:50:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:50:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:50:43.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39154.008303 secs 2023-03-10T01:50:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:43 smithi181 ceph-mon[94189]: pgmap v21391: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 869 KiB/s rd, 866 KiB/s wr, 11 op/s 2023-03-10T01:50:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:50:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:50:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:50:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39154.008303 secs 2023-03-10T01:50:43.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:43 smithi038 ceph-mon[92422]: pgmap v21391: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 869 KiB/s rd, 866 KiB/s wr, 11 op/s 2023-03-10T01:50:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:50:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:50:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:50:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39154.008303 secs 2023-03-10T01:50:43.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39615.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:50:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:44 smithi122 ceph-mon[93869]: pgmap v21392: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 869 KiB/s rd, 867 KiB/s wr, 12 op/s 2023-03-10T01:50:44.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:44 smithi181 ceph-mon[94189]: pgmap v21392: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 869 KiB/s rd, 867 KiB/s wr, 12 op/s 2023-03-10T01:50:44.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:44 smithi038 ceph-mon[92422]: pgmap v21392: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 869 KiB/s rd, 867 KiB/s wr, 12 op/s 2023-03-10T01:50:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39156.547318 secs 2023-03-10T01:50:46.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39156.547318 secs 2023-03-10T01:50:46.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39156.547318 secs 2023-03-10T01:50:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:47 smithi122 ceph-mon[93869]: pgmap v21393: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 823 KiB/s rd, 820 KiB/s wr, 12 op/s 2023-03-10T01:50:47.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:47 smithi181 ceph-mon[94189]: pgmap v21393: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 823 KiB/s rd, 820 KiB/s wr, 12 op/s 2023-03-10T01:50:47.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:47 smithi038 ceph-mon[92422]: pgmap v21393: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 823 KiB/s rd, 820 KiB/s wr, 12 op/s 2023-03-10T01:50:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:50:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:50:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:50:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:50:48.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:50:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:50:48.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39620.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:50:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:49 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39159.008392 secs 2023-03-10T01:50:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:49 smithi122 ceph-mon[93869]: pgmap v21394: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 12 op/s 2023-03-10T01:50:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:50:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:50:49.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:49 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39159.008392 secs 2023-03-10T01:50:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:49 smithi181 ceph-mon[94189]: pgmap v21394: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 12 op/s 2023-03-10T01:50:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:50:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:50:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:49 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39159.008392 secs 2023-03-10T01:50:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:49 smithi038 ceph-mon[92422]: pgmap v21394: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 12 op/s 2023-03-10T01:50:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:50:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:50:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:51 smithi122 ceph-mon[93869]: pgmap v21395: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 694 KiB/s rd, 692 KiB/s wr, 9 op/s 2023-03-10T01:50:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39161.547377 secs 2023-03-10T01:50:51.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:51 smithi181 ceph-mon[94189]: pgmap v21395: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 694 KiB/s rd, 692 KiB/s wr, 9 op/s 2023-03-10T01:50:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39161.547377 secs 2023-03-10T01:50:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:51 smithi038 ceph-mon[92422]: pgmap v21395: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 694 KiB/s rd, 692 KiB/s wr, 9 op/s 2023-03-10T01:50:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39161.547377 secs 2023-03-10T01:50:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:53 smithi122 ceph-mon[93869]: pgmap v21396: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 908 KiB/s rd, 905 KiB/s wr, 14 op/s 2023-03-10T01:50:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:50:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:50:53.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:53 smithi181 ceph-mon[94189]: pgmap v21396: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 908 KiB/s rd, 905 KiB/s wr, 14 op/s 2023-03-10T01:50:53.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:50:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:50:53.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:53 smithi038 ceph-mon[92422]: pgmap v21396: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 908 KiB/s rd, 905 KiB/s wr, 14 op/s 2023-03-10T01:50:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:50:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:50:53.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39625.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:50:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:54 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39164.008524 secs 2023-03-10T01:50:54.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:54 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39164.008524 secs 2023-03-10T01:50:54.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:54 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39164.008524 secs 2023-03-10T01:50:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:55 smithi122 ceph-mon[93869]: pgmap v21397: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 570 KiB/s wr, 10 op/s 2023-03-10T01:50:55.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:55 smithi181 ceph-mon[94189]: pgmap v21397: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 570 KiB/s wr, 10 op/s 2023-03-10T01:50:55.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:55 smithi038 ceph-mon[92422]: pgmap v21397: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 570 KiB/s wr, 10 op/s 2023-03-10T01:50:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:50:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:50:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39166.547469 secs 2023-03-10T01:50:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:50:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:50:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39166.547469 secs 2023-03-10T01:50:56.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:50:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:50:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39166.547469 secs 2023-03-10T01:50:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:57 smithi122 ceph-mon[93869]: pgmap v21398: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 573 KiB/s rd, 570 KiB/s wr, 11 op/s 2023-03-10T01:50:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:50:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:50:57.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:57 smithi181 ceph-mon[94189]: pgmap v21398: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 573 KiB/s rd, 570 KiB/s wr, 11 op/s 2023-03-10T01:50:57.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:50:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:50:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:57 smithi038 ceph-mon[92422]: pgmap v21398: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 573 KiB/s rd, 570 KiB/s wr, 11 op/s 2023-03-10T01:50:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:50:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:50:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:58 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:50:58.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:58 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:50:58.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:58 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:50:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39630.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:50:59.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:59 smithi122 ceph-mon[93869]: pgmap v21399: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 607 KiB/s rd, 603 KiB/s wr, 12 op/s 2023-03-10T01:50:59.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:59 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:50:59.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:50:59.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:50:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:50:59.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:59 smithi181 ceph-mon[94189]: pgmap v21399: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 607 KiB/s rd, 603 KiB/s wr, 12 op/s 2023-03-10T01:50:59.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:59 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:50:59.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:50:59.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:50:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:50:59.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:59 smithi038 ceph-mon[92422]: pgmap v21399: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 607 KiB/s rd, 603 KiB/s wr, 12 op/s 2023-03-10T01:50:59.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:59 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:50:59.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:50:59.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:50:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:51:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:51:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:51:00.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:51:00.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:51:00.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:51:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:51:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:01 smithi122 ceph-mon[93869]: pgmap v21400: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 588 KiB/s wr, 10 op/s 2023-03-10T01:51:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39171.547527 secs 2023-03-10T01:51:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:51:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:51:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:01 smithi181 ceph-mon[94189]: pgmap v21400: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 588 KiB/s wr, 10 op/s 2023-03-10T01:51:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39171.547527 secs 2023-03-10T01:51:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:51:01.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:51:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:01 smithi038 ceph-mon[92422]: pgmap v21400: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 588 KiB/s wr, 10 op/s 2023-03-10T01:51:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39171.547527 secs 2023-03-10T01:51:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:51:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:51:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:03 smithi122 ceph-mon[93869]: pgmap v21401: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 508 KiB/s rd, 622 KiB/s wr, 13 op/s 2023-03-10T01:51:03.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:03 smithi181 ceph-mon[94189]: pgmap v21401: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 508 KiB/s rd, 622 KiB/s wr, 13 op/s 2023-03-10T01:51:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:03 smithi038 ceph-mon[92422]: pgmap v21401: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 508 KiB/s rd, 622 KiB/s wr, 13 op/s 2023-03-10T01:51:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39635.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:51:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:05 smithi122 ceph-mon[93869]: pgmap v21402: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 846 KiB/s rd, 762 KiB/s wr, 22 op/s 2023-03-10T01:51:05.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:05 smithi181 ceph-mon[94189]: pgmap v21402: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 846 KiB/s rd, 762 KiB/s wr, 22 op/s 2023-03-10T01:51:05.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:05 smithi038 ceph-mon[92422]: pgmap v21402: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 846 KiB/s rd, 762 KiB/s wr, 22 op/s 2023-03-10T01:51:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39176.547588 secs 2023-03-10T01:51:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:06 smithi122 ceph-mon[93869]: osdmap e1553: 12 total, 12 up, 12 in 2023-03-10T01:51:06.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39176.547588 secs 2023-03-10T01:51:06.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:06 smithi181 ceph-mon[94189]: osdmap e1553: 12 total, 12 up, 12 in 2023-03-10T01:51:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39176.547588 secs 2023-03-10T01:51:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:06 smithi038 ceph-mon[92422]: osdmap e1553: 12 total, 12 up, 12 in 2023-03-10T01:51:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:07 smithi122 ceph-mon[93869]: pgmap v21404: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 26 op/s 2023-03-10T01:51:07.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:07 smithi181 ceph-mon[94189]: pgmap v21404: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 26 op/s 2023-03-10T01:51:07.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:07 smithi038 ceph-mon[92422]: pgmap v21404: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 26 op/s 2023-03-10T01:51:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:51:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:51:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:51:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:51:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:51:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:51:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39640.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:51:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:09 smithi122 ceph-mon[93869]: pgmap v21405: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.7 MiB/s wr, 31 op/s 2023-03-10T01:51:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:51:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:51:09.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:09 smithi181 ceph-mon[94189]: pgmap v21405: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.7 MiB/s wr, 31 op/s 2023-03-10T01:51:09.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:51:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:51:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:09 smithi038 ceph-mon[92422]: pgmap v21405: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.7 MiB/s wr, 31 op/s 2023-03-10T01:51:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:51:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:51:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:11 smithi122 ceph-mon[93869]: pgmap v21406: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 31 op/s 2023-03-10T01:51:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39181.547647 secs 2023-03-10T01:51:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:11 smithi181 ceph-mon[94189]: pgmap v21406: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 31 op/s 2023-03-10T01:51:11.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39181.547647 secs 2023-03-10T01:51:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:11 smithi038 ceph-mon[92422]: pgmap v21406: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 31 op/s 2023-03-10T01:51:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39181.547647 secs 2023-03-10T01:51:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:51:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:51:12.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:51:12.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:51:12.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:51:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:51:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:13 smithi122 ceph-mon[93869]: pgmap v21407: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 32 op/s 2023-03-10T01:51:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39184.008895 secs 2023-03-10T01:51:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:13 smithi181 ceph-mon[94189]: pgmap v21407: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 32 op/s 2023-03-10T01:51:13.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39184.008895 secs 2023-03-10T01:51:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:13 smithi038 ceph-mon[92422]: pgmap v21407: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 32 op/s 2023-03-10T01:51:13.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39184.008895 secs 2023-03-10T01:51:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39645.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:51:14.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:14 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:51:14.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:14 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:51:14.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:14 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:51:14.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:14 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:51:14.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:14 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:51:14.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:14 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:51:14.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:14 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:51:14.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:14 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:51:14.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:14 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:51:15.518 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:15 smithi181 ceph-mon[94189]: pgmap v21408: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 19 op/s 2023-03-10T01:51:15.518 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:15 smithi181 ceph-mon[94189]: osdmap e1554: 12 total, 12 up, 12 in 2023-03-10T01:51:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:15 smithi122 ceph-mon[93869]: pgmap v21408: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 19 op/s 2023-03-10T01:51:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:15 smithi122 ceph-mon[93869]: osdmap e1554: 12 total, 12 up, 12 in 2023-03-10T01:51:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:15 smithi038 ceph-mon[92422]: pgmap v21408: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 19 op/s 2023-03-10T01:51:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:15 smithi038 ceph-mon[92422]: osdmap e1554: 12 total, 12 up, 12 in 2023-03-10T01:51:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39186.547713 secs 2023-03-10T01:51:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:51:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:51:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39186.547713 secs 2023-03-10T01:51:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:51:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:51:16.752 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39186.547713 secs 2023-03-10T01:51:16.753 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:51:16.753 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:51:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:17 smithi122 ceph-mon[93869]: pgmap v21410: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 18 op/s 2023-03-10T01:51:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:51:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:51:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:17 smithi181 ceph-mon[94189]: pgmap v21410: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 18 op/s 2023-03-10T01:51:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:51:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:51:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:17 smithi038 ceph-mon[92422]: pgmap v21410: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 18 op/s 2023-03-10T01:51:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:51:17.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:51:18.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39189.008953 secs 2023-03-10T01:51:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39189.008953 secs 2023-03-10T01:51:18.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39189.008953 secs 2023-03-10T01:51:18.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39650.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:51:19.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:19 smithi122 ceph-mon[93869]: pgmap v21411: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 617 KiB/s rd, 628 KiB/s wr, 13 op/s 2023-03-10T01:51:19.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:19 smithi181 ceph-mon[94189]: pgmap v21411: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 617 KiB/s rd, 628 KiB/s wr, 13 op/s 2023-03-10T01:51:19.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:19 smithi038 ceph-mon[92422]: pgmap v21411: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 617 KiB/s rd, 628 KiB/s wr, 13 op/s 2023-03-10T01:51:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:21 smithi181 ceph-mon[94189]: pgmap v21412: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 616 KiB/s rd, 614 KiB/s wr, 11 op/s 2023-03-10T01:51:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39191.547765 secs 2023-03-10T01:51:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:21 smithi122 ceph-mon[93869]: pgmap v21412: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 616 KiB/s rd, 614 KiB/s wr, 11 op/s 2023-03-10T01:51:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39191.547765 secs 2023-03-10T01:51:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:21 smithi038 ceph-mon[92422]: pgmap v21412: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 616 KiB/s rd, 614 KiB/s wr, 11 op/s 2023-03-10T01:51:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39191.547765 secs 2023-03-10T01:51:23.705 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:23 smithi181 ceph-mon[94189]: pgmap v21413: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 576 KiB/s rd, 574 KiB/s wr, 9 op/s 2023-03-10T01:51:23.705 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39194.009063 secs 2023-03-10T01:51:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:23 smithi122 ceph-mon[93869]: pgmap v21413: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 576 KiB/s rd, 574 KiB/s wr, 9 op/s 2023-03-10T01:51:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39194.009063 secs 2023-03-10T01:51:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:23 smithi038 ceph-mon[92422]: pgmap v21413: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 576 KiB/s rd, 574 KiB/s wr, 9 op/s 2023-03-10T01:51:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39194.009063 secs 2023-03-10T01:51:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39655.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:51:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:25 smithi122 ceph-mon[93869]: pgmap v21414: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 163 KiB/s wr, 8 op/s 2023-03-10T01:51:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:25 smithi181 ceph-mon[94189]: pgmap v21414: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 163 KiB/s wr, 8 op/s 2023-03-10T01:51:25.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:25 smithi038 ceph-mon[92422]: pgmap v21414: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 163 KiB/s wr, 8 op/s 2023-03-10T01:51:26.660 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39196.547841 secs 2023-03-10T01:51:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39196.547841 secs 2023-03-10T01:51:26.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39196.547841 secs 2023-03-10T01:51:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:27 smithi122 ceph-mon[93869]: pgmap v21415: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 154 KiB/s wr, 9 op/s 2023-03-10T01:51:27.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:27 smithi181 ceph-mon[94189]: pgmap v21415: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 154 KiB/s wr, 9 op/s 2023-03-10T01:51:27.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:27 smithi038 ceph-mon[92422]: pgmap v21415: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 154 KiB/s wr, 9 op/s 2023-03-10T01:51:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39199.009146 secs 2023-03-10T01:51:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39199.009146 secs 2023-03-10T01:51:28.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39199.009146 secs 2023-03-10T01:51:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39660.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:51:29.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:29 smithi122 ceph-mon[93869]: pgmap v21416: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 136 KiB/s wr, 9 op/s 2023-03-10T01:51:29.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:29 smithi181 ceph-mon[94189]: pgmap v21416: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 136 KiB/s wr, 9 op/s 2023-03-10T01:51:29.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:29 smithi038 ceph-mon[92422]: pgmap v21416: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 136 KiB/s wr, 9 op/s 2023-03-10T01:51:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:31 smithi122 ceph-mon[93869]: pgmap v21417: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 135 KiB/s wr, 6 op/s 2023-03-10T01:51:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39201.547896 secs 2023-03-10T01:51:31.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:31 smithi181 ceph-mon[94189]: pgmap v21417: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 135 KiB/s wr, 6 op/s 2023-03-10T01:51:31.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39201.547896 secs 2023-03-10T01:51:31.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:31 smithi038 ceph-mon[92422]: pgmap v21417: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 135 KiB/s wr, 6 op/s 2023-03-10T01:51:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39201.547896 secs 2023-03-10T01:51:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:33 smithi122 ceph-mon[93869]: pgmap v21418: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 135 KiB/s wr, 8 op/s 2023-03-10T01:51:33.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39204.009210 secs 2023-03-10T01:51:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:33 smithi181 ceph-mon[94189]: pgmap v21418: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 135 KiB/s wr, 8 op/s 2023-03-10T01:51:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39204.009210 secs 2023-03-10T01:51:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:33 smithi038 ceph-mon[92422]: pgmap v21418: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 135 KiB/s wr, 8 op/s 2023-03-10T01:51:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39204.009210 secs 2023-03-10T01:51:33.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39665.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:51:35.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:35 smithi181 ceph-mon[94189]: pgmap v21419: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 179 KiB/s wr, 7 op/s 2023-03-10T01:51:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:35 smithi122 ceph-mon[93869]: pgmap v21419: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 179 KiB/s wr, 7 op/s 2023-03-10T01:51:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:35 smithi038 ceph-mon[92422]: pgmap v21419: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 179 KiB/s wr, 7 op/s 2023-03-10T01:51:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39206.547981 secs 2023-03-10T01:51:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:51:36.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:51:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39206.547981 secs 2023-03-10T01:51:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:51:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:51:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39206.547981 secs 2023-03-10T01:51:36.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:51:36.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:51:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:37 smithi122 ceph-mon[93869]: pgmap v21420: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 160 KiB/s wr, 7 op/s 2023-03-10T01:51:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:51:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:51:37.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:37 smithi181 ceph-mon[94189]: pgmap v21420: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 160 KiB/s wr, 7 op/s 2023-03-10T01:51:37.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:51:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:51:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:37 smithi038 ceph-mon[92422]: pgmap v21420: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 160 KiB/s wr, 7 op/s 2023-03-10T01:51:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:51:37.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:51:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39209.009294 secs 2023-03-10T01:51:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39209.009294 secs 2023-03-10T01:51:38.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39209.009294 secs 2023-03-10T01:51:38.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39670.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:51:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:39 smithi122 ceph-mon[93869]: pgmap v21421: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 160 KiB/s wr, 8 op/s 2023-03-10T01:51:39.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:39 smithi181 ceph-mon[94189]: pgmap v21421: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 160 KiB/s wr, 8 op/s 2023-03-10T01:51:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:39 smithi038 ceph-mon[92422]: pgmap v21421: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 160 KiB/s wr, 8 op/s 2023-03-10T01:51:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:41 smithi122 ceph-mon[93869]: pgmap v21422: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 160 KiB/s wr, 6 op/s 2023-03-10T01:51:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39211.548042 secs 2023-03-10T01:51:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:41 smithi181 ceph-mon[94189]: pgmap v21422: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 160 KiB/s wr, 6 op/s 2023-03-10T01:51:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39211.548042 secs 2023-03-10T01:51:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:41 smithi038 ceph-mon[92422]: pgmap v21422: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 160 KiB/s wr, 6 op/s 2023-03-10T01:51:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39211.548042 secs 2023-03-10T01:51:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:43 smithi122 ceph-mon[93869]: pgmap v21423: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 160 KiB/s wr, 8 op/s 2023-03-10T01:51:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39214.009351 secs 2023-03-10T01:51:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:51:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:43 smithi181 ceph-mon[94189]: pgmap v21423: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 160 KiB/s wr, 8 op/s 2023-03-10T01:51:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39214.009351 secs 2023-03-10T01:51:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:51:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:43 smithi038 ceph-mon[92422]: pgmap v21423: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 160 KiB/s wr, 8 op/s 2023-03-10T01:51:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39214.009351 secs 2023-03-10T01:51:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:51:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39675.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:51:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:44 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:51:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:44 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:51:44.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:44 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:51:44.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:44 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:51:44.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:44 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:51:44.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:44 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:51:44.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:44 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:51:44.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:44 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:51:44.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:44 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:51:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:45 smithi122 ceph-mon[93869]: pgmap v21424: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 160 KiB/s wr, 6 op/s 2023-03-10T01:51:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:45 smithi181 ceph-mon[94189]: pgmap v21424: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 160 KiB/s wr, 6 op/s 2023-03-10T01:51:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:45 smithi038 ceph-mon[92422]: pgmap v21424: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 160 KiB/s wr, 6 op/s 2023-03-10T01:51:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39216.548106 secs 2023-03-10T01:51:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39216.548106 secs 2023-03-10T01:51:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39216.548106 secs 2023-03-10T01:51:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:47 smithi122 ceph-mon[93869]: pgmap v21425: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 341 KiB/s wr, 7 op/s 2023-03-10T01:51:47.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:47 smithi181 ceph-mon[94189]: pgmap v21425: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 341 KiB/s wr, 7 op/s 2023-03-10T01:51:47.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:47 smithi038 ceph-mon[92422]: pgmap v21425: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 200 KiB/s rd, 341 KiB/s wr, 7 op/s 2023-03-10T01:51:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39680.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:51:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:49 smithi122 ceph-mon[93869]: pgmap v21426: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 265 KiB/s wr, 8 op/s 2023-03-10T01:51:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:49 smithi181 ceph-mon[94189]: pgmap v21426: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 265 KiB/s wr, 8 op/s 2023-03-10T01:51:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:49 smithi038 ceph-mon[92422]: pgmap v21426: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 265 KiB/s wr, 8 op/s 2023-03-10T01:51:50.680 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:50 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:51:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:50 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:51:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:50 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:51:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:51 smithi122 ceph-mon[93869]: pgmap v21427: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 266 KiB/s wr, 7 op/s 2023-03-10T01:51:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39221.548210 secs 2023-03-10T01:51:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:51 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:51:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:51:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:51:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:51 smithi181 ceph-mon[94189]: pgmap v21427: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 266 KiB/s wr, 7 op/s 2023-03-10T01:51:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39221.548210 secs 2023-03-10T01:51:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:51 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:51:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:51:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:51:51.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:51 smithi038 ceph-mon[92422]: pgmap v21427: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 266 KiB/s wr, 7 op/s 2023-03-10T01:51:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39221.548210 secs 2023-03-10T01:51:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:51 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:51:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:51:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:51:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:53 smithi122 ceph-mon[93869]: pgmap v21428: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 604 KiB/s rd, 601 KiB/s wr, 9 op/s 2023-03-10T01:51:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:51:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:51:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:53 smithi181 ceph-mon[94189]: pgmap v21428: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 604 KiB/s rd, 601 KiB/s wr, 9 op/s 2023-03-10T01:51:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:51:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:51:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:53 smithi038 ceph-mon[92422]: pgmap v21428: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 604 KiB/s rd, 601 KiB/s wr, 9 op/s 2023-03-10T01:51:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:51:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:51:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39685.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:51:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:55 smithi122 ceph-mon[93869]: pgmap v21429: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 707 KiB/s rd, 705 KiB/s wr, 9 op/s 2023-03-10T01:51:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:55 smithi181 ceph-mon[94189]: pgmap v21429: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 707 KiB/s rd, 705 KiB/s wr, 9 op/s 2023-03-10T01:51:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:55 smithi038 ceph-mon[92422]: pgmap v21429: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 707 KiB/s rd, 705 KiB/s wr, 9 op/s 2023-03-10T01:51:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39226.548270 secs 2023-03-10T01:51:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:51:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:51:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39226.548270 secs 2023-03-10T01:51:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:51:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:51:56.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39226.548270 secs 2023-03-10T01:51:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:51:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:51:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:57 smithi122 ceph-mon[93869]: pgmap v21430: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 708 KiB/s rd, 705 KiB/s wr, 10 op/s 2023-03-10T01:51:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:51:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:51:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:57 smithi181 ceph-mon[94189]: pgmap v21430: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 708 KiB/s rd, 705 KiB/s wr, 10 op/s 2023-03-10T01:51:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:51:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:51:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:57 smithi038 ceph-mon[92422]: pgmap v21430: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 708 KiB/s rd, 705 KiB/s wr, 10 op/s 2023-03-10T01:51:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:51:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:51:58.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39690.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:51:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:51:59 smithi122 ceph-mon[93869]: pgmap v21431: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1018 KiB/s rd, 750 KiB/s wr, 11 op/s 2023-03-10T01:51:59.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:51:59 smithi181 ceph-mon[94189]: pgmap v21431: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1018 KiB/s rd, 750 KiB/s wr, 11 op/s 2023-03-10T01:51:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:51:59 smithi038 ceph-mon[92422]: pgmap v21431: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1018 KiB/s rd, 750 KiB/s wr, 11 op/s 2023-03-10T01:52:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:01 smithi122 ceph-mon[93869]: pgmap v21432: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 989 KiB/s rd, 711 KiB/s wr, 10 op/s 2023-03-10T01:52:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39231.548328 secs 2023-03-10T01:52:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:52:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:52:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:01 smithi181 ceph-mon[94189]: pgmap v21432: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 989 KiB/s rd, 711 KiB/s wr, 10 op/s 2023-03-10T01:52:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39231.548328 secs 2023-03-10T01:52:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:52:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:52:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:01 smithi038 ceph-mon[92422]: pgmap v21432: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 989 KiB/s rd, 711 KiB/s wr, 10 op/s 2023-03-10T01:52:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39231.548328 secs 2023-03-10T01:52:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:52:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:52:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:03 smithi122 ceph-mon[93869]: pgmap v21433: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 966 KiB/s wr, 13 op/s 2023-03-10T01:52:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:03 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 39234.009686 secs 2023-03-10T01:52:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:03 smithi181 ceph-mon[94189]: pgmap v21433: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 966 KiB/s wr, 13 op/s 2023-03-10T01:52:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:03 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 39234.009686 secs 2023-03-10T01:52:03.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:03 smithi038 ceph-mon[92422]: pgmap v21433: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 966 KiB/s wr, 13 op/s 2023-03-10T01:52:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:03 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 39234.009686 secs 2023-03-10T01:52:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39695.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:52:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:05 smithi122 ceph-mon[93869]: pgmap v21434: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 972 KiB/s wr, 14 op/s 2023-03-10T01:52:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:05 smithi181 ceph-mon[94189]: pgmap v21434: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 972 KiB/s wr, 14 op/s 2023-03-10T01:52:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:05 smithi038 ceph-mon[92422]: pgmap v21434: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 972 KiB/s wr, 14 op/s 2023-03-10T01:52:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39236.548417 secs 2023-03-10T01:52:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39236.548417 secs 2023-03-10T01:52:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39236.548417 secs 2023-03-10T01:52:07.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:07 smithi181 ceph-mon[94189]: pgmap v21435: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 994 KiB/s wr, 15 op/s 2023-03-10T01:52:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:07 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:52:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:52:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:52:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:07 smithi038 ceph-mon[92422]: pgmap v21435: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 994 KiB/s wr, 15 op/s 2023-03-10T01:52:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:07 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:52:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:52:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:52:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:07 smithi122 ceph-mon[93869]: pgmap v21435: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 994 KiB/s wr, 15 op/s 2023-03-10T01:52:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:07 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:52:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:52:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:52:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:52:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:52:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39239.009744 secs 2023-03-10T01:52:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:08 smithi181 ceph-mon[94189]: pgmap v21436: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 18 op/s 2023-03-10T01:52:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:52:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:52:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39239.009744 secs 2023-03-10T01:52:08.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:08 smithi038 ceph-mon[92422]: pgmap v21436: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 18 op/s 2023-03-10T01:52:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39700.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:52:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:52:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:52:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39239.009744 secs 2023-03-10T01:52:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:08 smithi122 ceph-mon[93869]: pgmap v21436: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 18 op/s 2023-03-10T01:52:09.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:52:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:52:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:52:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:52:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:52:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:52:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:10 smithi038 ceph-mon[92422]: osdmap e1555: 12 total, 12 up, 12 in 2023-03-10T01:52:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:10 smithi038 ceph-mon[92422]: pgmap v21438: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 31 op/s 2023-03-10T01:52:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:10 smithi122 ceph-mon[93869]: osdmap e1555: 12 total, 12 up, 12 in 2023-03-10T01:52:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:10 smithi122 ceph-mon[93869]: pgmap v21438: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 31 op/s 2023-03-10T01:52:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:10 smithi181 ceph-mon[94189]: osdmap e1555: 12 total, 12 up, 12 in 2023-03-10T01:52:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:10 smithi181 ceph-mon[94189]: pgmap v21438: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 31 op/s 2023-03-10T01:52:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39241.548476 secs 2023-03-10T01:52:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:11 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:52:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:11 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:52:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39241.548476 secs 2023-03-10T01:52:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:11 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:52:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:11 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:52:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39241.548476 secs 2023-03-10T01:52:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:11 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:52:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:11 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:52:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:52:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:52:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:12 smithi122 ceph-mon[93869]: pgmap v21439: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 819 KiB/s rd, 789 KiB/s wr, 30 op/s 2023-03-10T01:52:12.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:52:12.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:52:12.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:12 smithi181 ceph-mon[94189]: pgmap v21439: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 819 KiB/s rd, 789 KiB/s wr, 30 op/s 2023-03-10T01:52:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:52:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:52:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:12 smithi038 ceph-mon[92422]: pgmap v21439: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 819 KiB/s rd, 789 KiB/s wr, 30 op/s 2023-03-10T01:52:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39705.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:52:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:52:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:52:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39244.009829 secs 2023-03-10T01:52:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:52:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:52:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39244.009829 secs 2023-03-10T01:52:13.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:52:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:52:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39244.009829 secs 2023-03-10T01:52:14.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:14 smithi122 ceph-mon[93869]: pgmap v21440: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 692 KiB/s rd, 734 KiB/s wr, 28 op/s 2023-03-10T01:52:14.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:14 smithi181 ceph-mon[94189]: pgmap v21440: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 692 KiB/s rd, 734 KiB/s wr, 28 op/s 2023-03-10T01:52:14.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:14 smithi038 ceph-mon[92422]: pgmap v21440: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 692 KiB/s rd, 734 KiB/s wr, 28 op/s 2023-03-10T01:52:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39246.548535 secs 2023-03-10T01:52:16.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39246.548535 secs 2023-03-10T01:52:16.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39246.548535 secs 2023-03-10T01:52:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:17 smithi122 ceph-mon[93869]: pgmap v21441: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 541 KiB/s rd, 583 KiB/s wr, 28 op/s 2023-03-10T01:52:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:52:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:52:17.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:17 smithi181 ceph-mon[94189]: pgmap v21441: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 541 KiB/s rd, 583 KiB/s wr, 28 op/s 2023-03-10T01:52:17.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:52:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:52:17.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:17 smithi038 ceph-mon[92422]: pgmap v21441: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 541 KiB/s rd, 583 KiB/s wr, 28 op/s 2023-03-10T01:52:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:52:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:52:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39710.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:52:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:19 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39249.009958 secs 2023-03-10T01:52:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:19 smithi122 ceph-mon[93869]: pgmap v21442: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 648 KiB/s wr, 24 op/s 2023-03-10T01:52:19.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:19 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39249.009958 secs 2023-03-10T01:52:19.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:19 smithi181 ceph-mon[94189]: pgmap v21442: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 648 KiB/s wr, 24 op/s 2023-03-10T01:52:19.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:19 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39249.009958 secs 2023-03-10T01:52:19.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:19 smithi038 ceph-mon[92422]: pgmap v21442: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 648 KiB/s wr, 24 op/s 2023-03-10T01:52:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:52:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:52:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:20 smithi122 ceph-mon[93869]: osdmap e1556: 12 total, 12 up, 12 in 2023-03-10T01:52:20.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:52:20.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:52:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:20 smithi181 ceph-mon[94189]: osdmap e1556: 12 total, 12 up, 12 in 2023-03-10T01:52:20.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:52:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:52:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:20 smithi038 ceph-mon[92422]: osdmap e1556: 12 total, 12 up, 12 in 2023-03-10T01:52:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:21 smithi122 ceph-mon[93869]: pgmap v21443: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 615 KiB/s rd, 613 KiB/s wr, 11 op/s 2023-03-10T01:52:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39251.548594 secs 2023-03-10T01:52:21.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:21 smithi181 ceph-mon[94189]: pgmap v21443: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 615 KiB/s rd, 613 KiB/s wr, 11 op/s 2023-03-10T01:52:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39251.548594 secs 2023-03-10T01:52:21.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:21 smithi038 ceph-mon[92422]: pgmap v21443: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 615 KiB/s rd, 613 KiB/s wr, 11 op/s 2023-03-10T01:52:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39251.548594 secs 2023-03-10T01:52:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:23 smithi122 ceph-mon[93869]: pgmap v21445: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 589 KiB/s rd, 586 KiB/s wr, 11 op/s 2023-03-10T01:52:23.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:23 smithi181 ceph-mon[94189]: pgmap v21445: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 589 KiB/s rd, 586 KiB/s wr, 11 op/s 2023-03-10T01:52:23.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:23 smithi038 ceph-mon[92422]: pgmap v21445: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 589 KiB/s rd, 586 KiB/s wr, 11 op/s 2023-03-10T01:52:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39715.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:52:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:24 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39254.010016 secs 2023-03-10T01:52:24.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:24 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39254.010016 secs 2023-03-10T01:52:24.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:24 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39254.010016 secs 2023-03-10T01:52:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:25 smithi122 ceph-mon[93869]: pgmap v21446: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 231 KiB/s wr, 9 op/s 2023-03-10T01:52:25.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:25 smithi181 ceph-mon[94189]: pgmap v21446: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 231 KiB/s wr, 9 op/s 2023-03-10T01:52:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:25 smithi038 ceph-mon[92422]: pgmap v21446: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 231 KiB/s wr, 9 op/s 2023-03-10T01:52:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39256.548676 secs 2023-03-10T01:52:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39256.548676 secs 2023-03-10T01:52:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39256.548676 secs 2023-03-10T01:52:27.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:27 smithi122 ceph-mon[93869]: pgmap v21447: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 231 KiB/s wr, 9 op/s 2023-03-10T01:52:27.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:27 smithi181 ceph-mon[94189]: pgmap v21447: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 231 KiB/s wr, 9 op/s 2023-03-10T01:52:27.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:27 smithi038 ceph-mon[92422]: pgmap v21447: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 234 KiB/s rd, 231 KiB/s wr, 9 op/s 2023-03-10T01:52:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39720.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:52:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:29 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39259.010079 secs 2023-03-10T01:52:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:29 smithi122 ceph-mon[93869]: pgmap v21448: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 9 op/s 2023-03-10T01:52:29.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:29 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39259.010079 secs 2023-03-10T01:52:29.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:29 smithi181 ceph-mon[94189]: pgmap v21448: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 9 op/s 2023-03-10T01:52:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:29 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39259.010079 secs 2023-03-10T01:52:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:29 smithi038 ceph-mon[92422]: pgmap v21448: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 0 B/s wr, 9 op/s 2023-03-10T01:52:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:31 smithi122 ceph-mon[93869]: pgmap v21449: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 8 op/s 2023-03-10T01:52:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39261.548734 secs 2023-03-10T01:52:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:31 smithi181 ceph-mon[94189]: pgmap v21449: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 8 op/s 2023-03-10T01:52:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39261.548734 secs 2023-03-10T01:52:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:31 smithi038 ceph-mon[92422]: pgmap v21449: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 8 op/s 2023-03-10T01:52:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39261.548734 secs 2023-03-10T01:52:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:33 smithi122 ceph-mon[93869]: pgmap v21450: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 10 op/s 2023-03-10T01:52:33.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:33 smithi181 ceph-mon[94189]: pgmap v21450: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 10 op/s 2023-03-10T01:52:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:33 smithi038 ceph-mon[92422]: pgmap v21450: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 10 op/s 2023-03-10T01:52:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39725.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:52:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:34 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39264.010138 secs 2023-03-10T01:52:34.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:34 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39264.010138 secs 2023-03-10T01:52:34.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:34 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39264.010138 secs 2023-03-10T01:52:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:35 smithi122 ceph-mon[93869]: pgmap v21451: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 8 op/s 2023-03-10T01:52:35.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:35 smithi181 ceph-mon[94189]: pgmap v21451: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 8 op/s 2023-03-10T01:52:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:35 smithi038 ceph-mon[92422]: pgmap v21451: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 8 op/s 2023-03-10T01:52:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39266.548795 secs 2023-03-10T01:52:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39266.548795 secs 2023-03-10T01:52:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39266.548795 secs 2023-03-10T01:52:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:37 smithi122 ceph-mon[93869]: pgmap v21452: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 341 B/s wr, 10 op/s 2023-03-10T01:52:37.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:37 smithi181 ceph-mon[94189]: pgmap v21452: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 341 B/s wr, 10 op/s 2023-03-10T01:52:37.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:37 smithi038 ceph-mon[92422]: pgmap v21452: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 341 B/s wr, 10 op/s 2023-03-10T01:52:38.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39730.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:52:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:39 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39269.010223 secs 2023-03-10T01:52:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:39 smithi122 ceph-mon[93869]: pgmap v21453: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 8 op/s 2023-03-10T01:52:39.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:39 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39269.010223 secs 2023-03-10T01:52:39.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:39 smithi181 ceph-mon[94189]: pgmap v21453: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 8 op/s 2023-03-10T01:52:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:39 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39269.010223 secs 2023-03-10T01:52:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:39 smithi038 ceph-mon[92422]: pgmap v21453: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 8 op/s 2023-03-10T01:52:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:41 smithi122 ceph-mon[93869]: pgmap v21454: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 63 KiB/s wr, 8 op/s 2023-03-10T01:52:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39271.548855 secs 2023-03-10T01:52:41.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:41 smithi181 ceph-mon[94189]: pgmap v21454: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 63 KiB/s wr, 8 op/s 2023-03-10T01:52:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39271.548855 secs 2023-03-10T01:52:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:41 smithi038 ceph-mon[92422]: pgmap v21454: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 63 KiB/s wr, 8 op/s 2023-03-10T01:52:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39271.548855 secs 2023-03-10T01:52:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:43 smithi122 ceph-mon[93869]: pgmap v21455: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 63 KiB/s wr, 10 op/s 2023-03-10T01:52:43.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:43 smithi181 ceph-mon[94189]: pgmap v21455: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 63 KiB/s wr, 10 op/s 2023-03-10T01:52:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:43 smithi038 ceph-mon[92422]: pgmap v21455: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 63 KiB/s wr, 10 op/s 2023-03-10T01:52:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39735.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:52:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:44 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39274.010285 secs 2023-03-10T01:52:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:44 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:52:44.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:44 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39274.010285 secs 2023-03-10T01:52:44.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:44 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:52:44.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:44 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39274.010285 secs 2023-03-10T01:52:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:44 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:52:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:45 smithi122 ceph-mon[93869]: pgmap v21456: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 176 KiB/s wr, 9 op/s 2023-03-10T01:52:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:52:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:52:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:52:45.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:45 smithi181 ceph-mon[94189]: pgmap v21456: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 176 KiB/s wr, 9 op/s 2023-03-10T01:52:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:52:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:52:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:52:45.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:45 smithi038 ceph-mon[92422]: pgmap v21456: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 176 KiB/s wr, 9 op/s 2023-03-10T01:52:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:52:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:52:45.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:52:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39276.548914 secs 2023-03-10T01:52:46.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39276.548914 secs 2023-03-10T01:52:46.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39276.548914 secs 2023-03-10T01:52:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:47 smithi122 ceph-mon[93869]: pgmap v21457: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 176 KiB/s wr, 10 op/s 2023-03-10T01:52:47.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:47 smithi181 ceph-mon[94189]: pgmap v21457: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 176 KiB/s wr, 10 op/s 2023-03-10T01:52:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:47 smithi038 ceph-mon[92422]: pgmap v21457: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 176 KiB/s wr, 10 op/s 2023-03-10T01:52:48.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39740.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:52:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:49 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39279.010415 secs 2023-03-10T01:52:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:49 smithi122 ceph-mon[93869]: pgmap v21458: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 175 KiB/s wr, 8 op/s 2023-03-10T01:52:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:52:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:52:49.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:49 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39279.010415 secs 2023-03-10T01:52:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:49 smithi181 ceph-mon[94189]: pgmap v21458: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 175 KiB/s wr, 8 op/s 2023-03-10T01:52:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:52:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:52:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:49 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39279.010415 secs 2023-03-10T01:52:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:49 smithi038 ceph-mon[92422]: pgmap v21458: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 178 KiB/s rd, 175 KiB/s wr, 8 op/s 2023-03-10T01:52:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:52:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:52:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:51 smithi122 ceph-mon[93869]: pgmap v21459: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 353 KiB/s wr, 7 op/s 2023-03-10T01:52:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39281.548981 secs 2023-03-10T01:52:51.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:51 smithi181 ceph-mon[94189]: pgmap v21459: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 353 KiB/s wr, 7 op/s 2023-03-10T01:52:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39281.548981 secs 2023-03-10T01:52:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:51 smithi038 ceph-mon[92422]: pgmap v21459: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 353 KiB/s wr, 7 op/s 2023-03-10T01:52:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39281.548981 secs 2023-03-10T01:52:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:52:52.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:52:52.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:52:52.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:52:52.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:52:52.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:52:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:53 smithi122 ceph-mon[93869]: pgmap v21460: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 402 KiB/s wr, 10 op/s 2023-03-10T01:52:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39284.010505 secs 2023-03-10T01:52:53.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:53 smithi181 ceph-mon[94189]: pgmap v21460: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 402 KiB/s wr, 10 op/s 2023-03-10T01:52:53.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39284.010505 secs 2023-03-10T01:52:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:53 smithi038 ceph-mon[92422]: pgmap v21460: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 405 KiB/s rd, 402 KiB/s wr, 10 op/s 2023-03-10T01:52:53.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39284.010505 secs 2023-03-10T01:52:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39745.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:52:55.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:55 smithi038 ceph-mon[92422]: pgmap v21461: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 505 KiB/s rd, 503 KiB/s wr, 8 op/s 2023-03-10T01:52:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:55 smithi122 ceph-mon[93869]: pgmap v21461: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 505 KiB/s rd, 503 KiB/s wr, 8 op/s 2023-03-10T01:52:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:55 smithi181 ceph-mon[94189]: pgmap v21461: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 505 KiB/s rd, 503 KiB/s wr, 8 op/s 2023-03-10T01:52:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39286.549038 secs 2023-03-10T01:52:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:52:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:52:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39286.549038 secs 2023-03-10T01:52:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:52:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:52:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39286.549038 secs 2023-03-10T01:52:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:52:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:52:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:57 smithi038 ceph-mon[92422]: pgmap v21462: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 424 KiB/s wr, 10 op/s 2023-03-10T01:52:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:52:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:52:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:57 smithi122 ceph-mon[93869]: pgmap v21462: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 424 KiB/s wr, 10 op/s 2023-03-10T01:52:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:52:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:52:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:57 smithi181 ceph-mon[94189]: pgmap v21462: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 428 KiB/s rd, 424 KiB/s wr, 10 op/s 2023-03-10T01:52:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:52:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:52:58.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39289.010609 secs 2023-03-10T01:52:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39289.010609 secs 2023-03-10T01:52:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39289.010609 secs 2023-03-10T01:52:58.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39750.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:52:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:52:59 smithi122 ceph-mon[93869]: pgmap v21463: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 472 KiB/s rd, 468 KiB/s wr, 9 op/s 2023-03-10T01:52:59.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:52:59 smithi181 ceph-mon[94189]: pgmap v21463: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 472 KiB/s rd, 468 KiB/s wr, 9 op/s 2023-03-10T01:52:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:52:59 smithi038 ceph-mon[92422]: pgmap v21463: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 472 KiB/s rd, 468 KiB/s wr, 9 op/s 2023-03-10T01:53:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:53:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:53:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:53:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:53:00.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:53:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:53:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:01 smithi122 ceph-mon[93869]: pgmap v21464: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 483 KiB/s rd, 468 KiB/s wr, 9 op/s 2023-03-10T01:53:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39291.549096 secs 2023-03-10T01:53:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:53:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:53:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:01 smithi181 ceph-mon[94189]: pgmap v21464: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 483 KiB/s rd, 468 KiB/s wr, 9 op/s 2023-03-10T01:53:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39291.549096 secs 2023-03-10T01:53:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:53:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:53:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:01 smithi038 ceph-mon[92422]: pgmap v21464: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 483 KiB/s rd, 468 KiB/s wr, 9 op/s 2023-03-10T01:53:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39291.549096 secs 2023-03-10T01:53:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:53:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:53:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:03 smithi122 ceph-mon[93869]: pgmap v21465: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 904 KiB/s rd, 898 KiB/s wr, 14 op/s 2023-03-10T01:53:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39294.010726 secs 2023-03-10T01:53:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:03 smithi181 ceph-mon[94189]: pgmap v21465: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 904 KiB/s rd, 898 KiB/s wr, 14 op/s 2023-03-10T01:53:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39294.010726 secs 2023-03-10T01:53:03.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:03 smithi038 ceph-mon[92422]: pgmap v21465: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 904 KiB/s rd, 898 KiB/s wr, 14 op/s 2023-03-10T01:53:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39294.010726 secs 2023-03-10T01:53:03.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39755.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:53:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:05 smithi122 ceph-mon[93869]: pgmap v21466: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 798 KiB/s wr, 22 op/s 2023-03-10T01:53:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:05 smithi181 ceph-mon[94189]: pgmap v21466: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 798 KiB/s wr, 22 op/s 2023-03-10T01:53:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:05 smithi038 ceph-mon[92422]: pgmap v21466: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 798 KiB/s wr, 22 op/s 2023-03-10T01:53:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39296.549154 secs 2023-03-10T01:53:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39296.549154 secs 2023-03-10T01:53:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39296.549154 secs 2023-03-10T01:53:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:07 smithi122 ceph-mon[93869]: pgmap v21467: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 801 KiB/s rd, 800 KiB/s wr, 25 op/s 2023-03-10T01:53:07.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:07 smithi181 ceph-mon[94189]: pgmap v21467: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 801 KiB/s rd, 800 KiB/s wr, 25 op/s 2023-03-10T01:53:07.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:07 smithi038 ceph-mon[92422]: pgmap v21467: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 801 KiB/s rd, 800 KiB/s wr, 25 op/s 2023-03-10T01:53:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:53:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:53:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39299.010824 secs 2023-03-10T01:53:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:53:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:53:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39299.010824 secs 2023-03-10T01:53:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:53:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:53:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39299.010824 secs 2023-03-10T01:53:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39760.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:53:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:09 smithi122 ceph-mon[93869]: pgmap v21468: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 766 KiB/s rd, 766 KiB/s wr, 23 op/s 2023-03-10T01:53:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:53:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:53:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:09 smithi181 ceph-mon[94189]: pgmap v21468: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 766 KiB/s rd, 766 KiB/s wr, 23 op/s 2023-03-10T01:53:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:53:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:53:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:09 smithi038 ceph-mon[92422]: pgmap v21468: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 766 KiB/s rd, 766 KiB/s wr, 23 op/s 2023-03-10T01:53:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:53:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:53:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:11 smithi122 ceph-mon[93869]: pgmap v21469: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 23 op/s 2023-03-10T01:53:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39301.549240 secs 2023-03-10T01:53:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:11 smithi181 ceph-mon[94189]: pgmap v21469: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 23 op/s 2023-03-10T01:53:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39301.549240 secs 2023-03-10T01:53:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:11 smithi038 ceph-mon[92422]: pgmap v21469: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 23 op/s 2023-03-10T01:53:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39301.549240 secs 2023-03-10T01:53:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:53:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:53:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:53:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:53:12.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:53:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:53:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:13 smithi122 ceph-mon[93869]: pgmap v21470: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 26 op/s 2023-03-10T01:53:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:53:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:53:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39304.010914 secs 2023-03-10T01:53:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:13 smithi181 ceph-mon[94189]: pgmap v21470: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 26 op/s 2023-03-10T01:53:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:53:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:53:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39304.010914 secs 2023-03-10T01:53:13.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:13 smithi038 ceph-mon[92422]: pgmap v21470: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 26 op/s 2023-03-10T01:53:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:53:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:53:13.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39304.010914 secs 2023-03-10T01:53:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39765.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:53:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:15 smithi122 ceph-mon[93869]: pgmap v21471: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 482 KiB/s wr, 21 op/s 2023-03-10T01:53:15.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:15 smithi181 ceph-mon[94189]: pgmap v21471: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 482 KiB/s wr, 21 op/s 2023-03-10T01:53:15.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:15 smithi038 ceph-mon[92422]: pgmap v21471: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 482 KiB/s wr, 21 op/s 2023-03-10T01:53:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39306.549302 secs 2023-03-10T01:53:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39306.549302 secs 2023-03-10T01:53:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39306.549302 secs 2023-03-10T01:53:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:17 smithi122 ceph-mon[93869]: pgmap v21472: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 667 KiB/s rd, 664 KiB/s wr, 12 op/s 2023-03-10T01:53:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:17 smithi181 ceph-mon[94189]: pgmap v21472: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 667 KiB/s rd, 664 KiB/s wr, 12 op/s 2023-03-10T01:53:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:17 smithi038 ceph-mon[92422]: pgmap v21472: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 667 KiB/s rd, 664 KiB/s wr, 12 op/s 2023-03-10T01:53:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39309.010976 secs 2023-03-10T01:53:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39309.010976 secs 2023-03-10T01:53:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39309.010976 secs 2023-03-10T01:53:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39770.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:53:19.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:19 smithi122 ceph-mon[93869]: pgmap v21473: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 561 KiB/s wr, 8 op/s 2023-03-10T01:53:19.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:19 smithi181 ceph-mon[94189]: pgmap v21473: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 561 KiB/s wr, 8 op/s 2023-03-10T01:53:19.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:19 smithi038 ceph-mon[92422]: pgmap v21473: 129 pgs: 129 active+clean; 10 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 563 KiB/s rd, 561 KiB/s wr, 8 op/s 2023-03-10T01:53:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:53:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:53:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:53:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:53:20.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:53:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:53:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:21 smithi122 ceph-mon[93869]: pgmap v21474: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 925 KiB/s rd, 933 KiB/s wr, 11 op/s 2023-03-10T01:53:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39311.549357 secs 2023-03-10T01:53:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:53:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:53:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:21 smithi181 ceph-mon[94189]: pgmap v21474: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 925 KiB/s rd, 933 KiB/s wr, 11 op/s 2023-03-10T01:53:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39311.549357 secs 2023-03-10T01:53:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:53:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:53:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:21 smithi038 ceph-mon[92422]: pgmap v21474: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 925 KiB/s rd, 933 KiB/s wr, 11 op/s 2023-03-10T01:53:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39311.549357 secs 2023-03-10T01:53:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:53:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:53:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:23 smithi122 ceph-mon[93869]: pgmap v21475: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 761 KiB/s rd, 779 KiB/s wr, 13 op/s 2023-03-10T01:53:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39314.011082 secs 2023-03-10T01:53:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:23 smithi181 ceph-mon[94189]: pgmap v21475: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 761 KiB/s rd, 779 KiB/s wr, 13 op/s 2023-03-10T01:53:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39314.011082 secs 2023-03-10T01:53:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:23 smithi038 ceph-mon[92422]: pgmap v21475: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 761 KiB/s rd, 779 KiB/s wr, 13 op/s 2023-03-10T01:53:23.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39314.011082 secs 2023-03-10T01:53:23.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39775.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:53:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:25 smithi122 ceph-mon[93869]: pgmap v21476: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 759 KiB/s rd, 779 KiB/s wr, 10 op/s 2023-03-10T01:53:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:25 smithi122 ceph-mon[93869]: osdmap e1557: 12 total, 12 up, 12 in 2023-03-10T01:53:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:25 smithi181 ceph-mon[94189]: pgmap v21476: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 759 KiB/s rd, 779 KiB/s wr, 10 op/s 2023-03-10T01:53:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:25 smithi181 ceph-mon[94189]: osdmap e1557: 12 total, 12 up, 12 in 2023-03-10T01:53:25.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:25 smithi038 ceph-mon[92422]: pgmap v21476: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 759 KiB/s rd, 779 KiB/s wr, 10 op/s 2023-03-10T01:53:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:25 smithi038 ceph-mon[92422]: osdmap e1557: 12 total, 12 up, 12 in 2023-03-10T01:53:26.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39316.549416 secs 2023-03-10T01:53:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39316.549416 secs 2023-03-10T01:53:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39316.549416 secs 2023-03-10T01:53:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:27 smithi122 ceph-mon[93869]: pgmap v21478: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 681 KiB/s rd, 705 KiB/s wr, 13 op/s 2023-03-10T01:53:27.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:27 smithi181 ceph-mon[94189]: pgmap v21478: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 681 KiB/s rd, 705 KiB/s wr, 13 op/s 2023-03-10T01:53:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:27 smithi038 ceph-mon[92422]: pgmap v21478: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 681 KiB/s rd, 705 KiB/s wr, 13 op/s 2023-03-10T01:53:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39319.011210 secs 2023-03-10T01:53:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39319.011210 secs 2023-03-10T01:53:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39319.011210 secs 2023-03-10T01:53:28.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39780.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:53:29.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:29 smithi122 ceph-mon[93869]: pgmap v21479: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 877 KiB/s rd, 902 KiB/s wr, 14 op/s 2023-03-10T01:53:29.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:29 smithi181 ceph-mon[94189]: pgmap v21479: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 877 KiB/s rd, 902 KiB/s wr, 14 op/s 2023-03-10T01:53:29.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:29 smithi038 ceph-mon[92422]: pgmap v21479: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 877 KiB/s rd, 902 KiB/s wr, 14 op/s 2023-03-10T01:53:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:31 smithi122 ceph-mon[93869]: pgmap v21480: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 443 KiB/s rd, 456 KiB/s wr, 10 op/s 2023-03-10T01:53:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39321.549502 secs 2023-03-10T01:53:31.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:31 smithi181 ceph-mon[94189]: pgmap v21480: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 443 KiB/s rd, 456 KiB/s wr, 10 op/s 2023-03-10T01:53:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39321.549502 secs 2023-03-10T01:53:31.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:31 smithi038 ceph-mon[92422]: pgmap v21480: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 443 KiB/s rd, 456 KiB/s wr, 10 op/s 2023-03-10T01:53:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39321.549502 secs 2023-03-10T01:53:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:33 smithi122 ceph-mon[93869]: pgmap v21481: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 251 KiB/s wr, 9 op/s 2023-03-10T01:53:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39324.011268 secs 2023-03-10T01:53:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:33 smithi181 ceph-mon[94189]: pgmap v21481: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 251 KiB/s wr, 9 op/s 2023-03-10T01:53:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39324.011268 secs 2023-03-10T01:53:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:33 smithi038 ceph-mon[92422]: pgmap v21481: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 251 KiB/s wr, 9 op/s 2023-03-10T01:53:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39324.011268 secs 2023-03-10T01:53:33.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39785.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:53:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:35 smithi122 ceph-mon[93869]: pgmap v21482: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 251 KiB/s wr, 9 op/s 2023-03-10T01:53:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:35 smithi122 ceph-mon[93869]: osdmap e1558: 12 total, 12 up, 12 in 2023-03-10T01:53:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:35 smithi181 ceph-mon[94189]: pgmap v21482: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 251 KiB/s wr, 9 op/s 2023-03-10T01:53:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:35 smithi181 ceph-mon[94189]: osdmap e1558: 12 total, 12 up, 12 in 2023-03-10T01:53:35.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:35 smithi038 ceph-mon[92422]: pgmap v21482: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 251 KiB/s wr, 9 op/s 2023-03-10T01:53:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:35 smithi038 ceph-mon[92422]: osdmap e1558: 12 total, 12 up, 12 in 2023-03-10T01:53:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39326.549581 secs 2023-03-10T01:53:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:53:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:53:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39326.549581 secs 2023-03-10T01:53:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:53:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:53:36.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39326.549581 secs 2023-03-10T01:53:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:53:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:53:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:37 smithi122 ceph-mon[93869]: pgmap v21484: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 702 KiB/s rd, 699 KiB/s wr, 8 op/s 2023-03-10T01:53:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:53:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:53:37.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:37 smithi181 ceph-mon[94189]: pgmap v21484: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 702 KiB/s rd, 699 KiB/s wr, 8 op/s 2023-03-10T01:53:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:53:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:53:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:37 smithi038 ceph-mon[92422]: pgmap v21484: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 702 KiB/s rd, 699 KiB/s wr, 8 op/s 2023-03-10T01:53:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:53:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:53:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39329.011353 secs 2023-03-10T01:53:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39329.011353 secs 2023-03-10T01:53:38.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39329.011353 secs 2023-03-10T01:53:38.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39790.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:53:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:39 smithi122 ceph-mon[93869]: pgmap v21485: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 946 KiB/s rd, 949 KiB/s wr, 10 op/s 2023-03-10T01:53:39.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:39 smithi181 ceph-mon[94189]: pgmap v21485: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 946 KiB/s rd, 949 KiB/s wr, 10 op/s 2023-03-10T01:53:39.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:39 smithi038 ceph-mon[92422]: pgmap v21485: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 946 KiB/s rd, 949 KiB/s wr, 10 op/s 2023-03-10T01:53:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:53:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:53:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:53:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:53:40.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:53:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:53:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:41 smithi122 ceph-mon[93869]: pgmap v21486: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 946 KiB/s rd, 949 KiB/s wr, 10 op/s 2023-03-10T01:53:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39331.549640 secs 2023-03-10T01:53:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:53:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:53:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:41 smithi181 ceph-mon[94189]: pgmap v21486: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 946 KiB/s rd, 949 KiB/s wr, 10 op/s 2023-03-10T01:53:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39331.549640 secs 2023-03-10T01:53:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:53:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:53:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:41 smithi038 ceph-mon[92422]: pgmap v21486: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 946 KiB/s rd, 949 KiB/s wr, 10 op/s 2023-03-10T01:53:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39331.549640 secs 2023-03-10T01:53:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:53:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:53:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:43 smithi122 ceph-mon[93869]: pgmap v21487: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 933 KiB/s rd, 898 KiB/s wr, 10 op/s 2023-03-10T01:53:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39334.011413 secs 2023-03-10T01:53:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:43 smithi181 ceph-mon[94189]: pgmap v21487: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 933 KiB/s rd, 898 KiB/s wr, 10 op/s 2023-03-10T01:53:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39334.011413 secs 2023-03-10T01:53:43.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:43 smithi038 ceph-mon[92422]: pgmap v21487: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 933 KiB/s rd, 898 KiB/s wr, 10 op/s 2023-03-10T01:53:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39334.011413 secs 2023-03-10T01:53:43.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39795.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:53:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:44 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:53:44.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:44 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:53:44.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:44 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:53:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:45 smithi122 ceph-mon[93869]: pgmap v21488: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 11 op/s 2023-03-10T01:53:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:53:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:53:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:53:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:45 smithi181 ceph-mon[94189]: pgmap v21488: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 11 op/s 2023-03-10T01:53:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:53:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:53:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:53:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:45 smithi038 ceph-mon[92422]: pgmap v21488: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 11 op/s 2023-03-10T01:53:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:53:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:53:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:53:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39336.549697 secs 2023-03-10T01:53:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39336.549697 secs 2023-03-10T01:53:46.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39336.549697 secs 2023-03-10T01:53:47.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:47 smithi181 ceph-mon[94189]: pgmap v21489: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 13 op/s 2023-03-10T01:53:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:47 smithi038 ceph-mon[92422]: pgmap v21489: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 13 op/s 2023-03-10T01:53:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:47 smithi122 ceph-mon[93869]: pgmap v21489: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 13 op/s 2023-03-10T01:53:48.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:53:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:53:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39800.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:53:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:53:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:53:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:53:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:53:49.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:49 smithi038 ceph-mon[92422]: pgmap v21490: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 903 KiB/s rd, 874 KiB/s wr, 10 op/s 2023-03-10T01:53:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:53:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:53:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:49 smithi122 ceph-mon[93869]: pgmap v21490: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 903 KiB/s rd, 874 KiB/s wr, 10 op/s 2023-03-10T01:53:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:53:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:53:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:49 smithi181 ceph-mon[94189]: pgmap v21490: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 903 KiB/s rd, 874 KiB/s wr, 10 op/s 2023-03-10T01:53:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:53:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:53:50.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:50 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:53:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:50 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:53:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:50 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:53:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:51 smithi038 ceph-mon[92422]: pgmap v21491: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 535 KiB/s rd, 502 KiB/s wr, 8 op/s 2023-03-10T01:53:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39341.549773 secs 2023-03-10T01:53:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:51 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:53:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:53:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:53:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:51 smithi122 ceph-mon[93869]: pgmap v21491: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 535 KiB/s rd, 502 KiB/s wr, 8 op/s 2023-03-10T01:53:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39341.549773 secs 2023-03-10T01:53:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:51 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:53:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:53:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:53:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:51 smithi181 ceph-mon[94189]: pgmap v21491: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 535 KiB/s rd, 502 KiB/s wr, 8 op/s 2023-03-10T01:53:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39341.549773 secs 2023-03-10T01:53:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:51 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:53:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:53:51.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:53:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:52 smithi122 ceph-mon[93869]: pgmap v21492: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 671 KiB/s rd, 637 KiB/s wr, 11 op/s 2023-03-10T01:53:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:52 smithi181 ceph-mon[94189]: pgmap v21492: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 671 KiB/s rd, 637 KiB/s wr, 11 op/s 2023-03-10T01:53:52.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:52 smithi038 ceph-mon[92422]: pgmap v21492: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 671 KiB/s rd, 637 KiB/s wr, 11 op/s 2023-03-10T01:53:53.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39805.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:53:55.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:54 smithi122 ceph-mon[93869]: pgmap v21493: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 790 KiB/s rd, 788 KiB/s wr, 8 op/s 2023-03-10T01:53:55.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:54 smithi181 ceph-mon[94189]: pgmap v21493: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 790 KiB/s rd, 788 KiB/s wr, 8 op/s 2023-03-10T01:53:55.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:54 smithi038 ceph-mon[92422]: pgmap v21493: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 790 KiB/s rd, 788 KiB/s wr, 8 op/s 2023-03-10T01:53:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:53:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:53:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:53:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:53:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:53:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:53:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39346.549834 secs 2023-03-10T01:53:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:56 smithi122 ceph-mon[93869]: pgmap v21494: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 682 KiB/s rd, 679 KiB/s wr, 11 op/s 2023-03-10T01:53:57.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39346.549834 secs 2023-03-10T01:53:57.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:56 smithi181 ceph-mon[94189]: pgmap v21494: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 682 KiB/s rd, 679 KiB/s wr, 11 op/s 2023-03-10T01:53:57.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39346.549834 secs 2023-03-10T01:53:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:56 smithi038 ceph-mon[92422]: pgmap v21494: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 682 KiB/s rd, 679 KiB/s wr, 11 op/s 2023-03-10T01:53:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:53:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:53:58.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:53:58.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:53:58.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:53:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:53:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39810.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:53:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:58 smithi122 ceph-mon[93869]: pgmap v21495: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 924 KiB/s rd, 920 KiB/s wr, 11 op/s 2023-03-10T01:53:59.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:58 smithi181 ceph-mon[94189]: pgmap v21495: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 924 KiB/s rd, 920 KiB/s wr, 11 op/s 2023-03-10T01:53:59.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:58 smithi038 ceph-mon[92422]: pgmap v21495: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 924 KiB/s rd, 920 KiB/s wr, 11 op/s 2023-03-10T01:54:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:54:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:53:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:54:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:54:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:53:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:54:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:54:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:53:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:54:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:01 smithi122 ceph-mon[93869]: pgmap v21496: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 598 KiB/s rd, 584 KiB/s wr, 10 op/s 2023-03-10T01:54:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39351.549892 secs 2023-03-10T01:54:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:54:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:54:01.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:01 smithi181 ceph-mon[94189]: pgmap v21496: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 598 KiB/s rd, 584 KiB/s wr, 10 op/s 2023-03-10T01:54:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39351.549892 secs 2023-03-10T01:54:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:54:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:54:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:01 smithi038 ceph-mon[92422]: pgmap v21496: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 598 KiB/s rd, 584 KiB/s wr, 10 op/s 2023-03-10T01:54:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39351.549892 secs 2023-03-10T01:54:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:54:01.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:54:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:03 smithi122 ceph-mon[93869]: pgmap v21497: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 955 KiB/s rd, 950 KiB/s wr, 15 op/s 2023-03-10T01:54:03.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:03 smithi181 ceph-mon[94189]: pgmap v21497: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 955 KiB/s rd, 950 KiB/s wr, 15 op/s 2023-03-10T01:54:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:03 smithi038 ceph-mon[92422]: pgmap v21497: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 955 KiB/s rd, 950 KiB/s wr, 15 op/s 2023-03-10T01:54:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39815.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:54:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:05 smithi122 ceph-mon[93869]: pgmap v21498: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 825 KiB/s rd, 825 KiB/s wr, 23 op/s 2023-03-10T01:54:05.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:05 smithi181 ceph-mon[94189]: pgmap v21498: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 825 KiB/s rd, 825 KiB/s wr, 23 op/s 2023-03-10T01:54:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:05 smithi038 ceph-mon[92422]: pgmap v21498: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 825 KiB/s rd, 825 KiB/s wr, 23 op/s 2023-03-10T01:54:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39356.549952 secs 2023-03-10T01:54:06.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39356.549952 secs 2023-03-10T01:54:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39356.549952 secs 2023-03-10T01:54:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:07 smithi122 ceph-mon[93869]: pgmap v21499: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 829 KiB/s rd, 828 KiB/s wr, 25 op/s 2023-03-10T01:54:07.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:07 smithi181 ceph-mon[94189]: pgmap v21499: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 829 KiB/s rd, 828 KiB/s wr, 25 op/s 2023-03-10T01:54:07.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:07 smithi038 ceph-mon[92422]: pgmap v21499: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 829 KiB/s rd, 828 KiB/s wr, 25 op/s 2023-03-10T01:54:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:54:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:54:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:54:08.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:54:08.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:54:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:54:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39820.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:54:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:09 smithi122 ceph-mon[93869]: pgmap v21500: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 839 KiB/s rd, 839 KiB/s wr, 24 op/s 2023-03-10T01:54:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:54:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:54:09.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:09 smithi181 ceph-mon[94189]: pgmap v21500: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 839 KiB/s rd, 839 KiB/s wr, 24 op/s 2023-03-10T01:54:09.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:54:09.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:54:09.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:09 smithi038 ceph-mon[92422]: pgmap v21500: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 839 KiB/s rd, 839 KiB/s wr, 24 op/s 2023-03-10T01:54:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:54:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:54:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:11 smithi122 ceph-mon[93869]: pgmap v21501: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 783 KiB/s rd, 785 KiB/s wr, 23 op/s 2023-03-10T01:54:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39361.550030 secs 2023-03-10T01:54:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:11 smithi181 ceph-mon[94189]: pgmap v21501: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 783 KiB/s rd, 785 KiB/s wr, 23 op/s 2023-03-10T01:54:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39361.550030 secs 2023-03-10T01:54:11.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:11 smithi038 ceph-mon[92422]: pgmap v21501: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 783 KiB/s rd, 785 KiB/s wr, 23 op/s 2023-03-10T01:54:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39361.550030 secs 2023-03-10T01:54:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:13 smithi122 ceph-mon[93869]: pgmap v21502: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 773 KiB/s rd, 785 KiB/s wr, 25 op/s 2023-03-10T01:54:13.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:13 smithi181 ceph-mon[94189]: pgmap v21502: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 773 KiB/s rd, 785 KiB/s wr, 25 op/s 2023-03-10T01:54:13.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:13 smithi038 ceph-mon[92422]: pgmap v21502: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 773 KiB/s rd, 785 KiB/s wr, 25 op/s 2023-03-10T01:54:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39825.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:54:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:15 smithi122 ceph-mon[93869]: pgmap v21503: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 610 KiB/s rd, 612 KiB/s wr, 21 op/s 2023-03-10T01:54:15.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:15 smithi181 ceph-mon[94189]: pgmap v21503: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 610 KiB/s rd, 612 KiB/s wr, 21 op/s 2023-03-10T01:54:15.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:15 smithi038 ceph-mon[92422]: pgmap v21503: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 610 KiB/s rd, 612 KiB/s wr, 21 op/s 2023-03-10T01:54:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39366.550117 secs 2023-03-10T01:54:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:54:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:54:16.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39366.550117 secs 2023-03-10T01:54:16.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:54:16.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:54:16.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39366.550117 secs 2023-03-10T01:54:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:54:16.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:54:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:17 smithi122 ceph-mon[93869]: pgmap v21504: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 605 KiB/s rd, 601 KiB/s wr, 12 op/s 2023-03-10T01:54:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:54:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:54:17.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:17 smithi181 ceph-mon[94189]: pgmap v21504: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 605 KiB/s rd, 601 KiB/s wr, 12 op/s 2023-03-10T01:54:17.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:54:17.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:54:17.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:17 smithi038 ceph-mon[92422]: pgmap v21504: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 605 KiB/s rd, 601 KiB/s wr, 12 op/s 2023-03-10T01:54:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:54:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:54:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39830.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:54:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:19 smithi122 ceph-mon[93869]: pgmap v21505: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 447 KiB/s wr, 9 op/s 2023-03-10T01:54:19.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:19 smithi181 ceph-mon[94189]: pgmap v21505: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 447 KiB/s wr, 9 op/s 2023-03-10T01:54:19.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:19 smithi038 ceph-mon[92422]: pgmap v21505: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 447 KiB/s wr, 9 op/s 2023-03-10T01:54:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:54:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:54:20.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:54:20.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:54:20.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:54:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:54:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:21 smithi122 ceph-mon[93869]: pgmap v21506: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 525 KiB/s rd, 557 KiB/s wr, 10 op/s 2023-03-10T01:54:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39371.550178 secs 2023-03-10T01:54:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:54:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:54:21.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:21 smithi181 ceph-mon[94189]: pgmap v21506: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 525 KiB/s rd, 557 KiB/s wr, 10 op/s 2023-03-10T01:54:21.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39371.550178 secs 2023-03-10T01:54:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:54:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:54:21.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:21 smithi038 ceph-mon[92422]: pgmap v21506: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 525 KiB/s rd, 557 KiB/s wr, 10 op/s 2023-03-10T01:54:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39371.550178 secs 2023-03-10T01:54:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:54:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:54:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:23 smithi122 ceph-mon[93869]: pgmap v21507: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 392 KiB/s wr, 13 op/s 2023-03-10T01:54:23.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:23 smithi181 ceph-mon[94189]: pgmap v21507: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 392 KiB/s wr, 13 op/s 2023-03-10T01:54:23.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:23 smithi038 ceph-mon[92422]: pgmap v21507: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 350 KiB/s rd, 392 KiB/s wr, 13 op/s 2023-03-10T01:54:23.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39835.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:54:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:25 smithi122 ceph-mon[93869]: pgmap v21508: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 455 KiB/s rd, 475 KiB/s wr, 12 op/s 2023-03-10T01:54:25.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:25 smithi181 ceph-mon[94189]: pgmap v21508: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 455 KiB/s rd, 475 KiB/s wr, 12 op/s 2023-03-10T01:54:25.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:25 smithi038 ceph-mon[92422]: pgmap v21508: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 455 KiB/s rd, 475 KiB/s wr, 12 op/s 2023-03-10T01:54:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39376.550238 secs 2023-03-10T01:54:26.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:26 smithi122 ceph-mon[93869]: osdmap e1559: 12 total, 12 up, 12 in 2023-03-10T01:54:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39376.550238 secs 2023-03-10T01:54:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:26 smithi181 ceph-mon[94189]: osdmap e1559: 12 total, 12 up, 12 in 2023-03-10T01:54:26.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39376.550238 secs 2023-03-10T01:54:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:26 smithi038 ceph-mon[92422]: osdmap e1559: 12 total, 12 up, 12 in 2023-03-10T01:54:27.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:27 smithi122 ceph-mon[93869]: pgmap v21510: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 615 KiB/s rd, 639 KiB/s wr, 14 op/s 2023-03-10T01:54:27.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:27 smithi181 ceph-mon[94189]: pgmap v21510: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 615 KiB/s rd, 639 KiB/s wr, 14 op/s 2023-03-10T01:54:27.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:27 smithi038 ceph-mon[92422]: pgmap v21510: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 615 KiB/s rd, 639 KiB/s wr, 14 op/s 2023-03-10T01:54:28.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39840.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:54:29.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:29 smithi122 ceph-mon[93869]: pgmap v21511: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 615 KiB/s rd, 639 KiB/s wr, 14 op/s 2023-03-10T01:54:29.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:29 smithi181 ceph-mon[94189]: pgmap v21511: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 615 KiB/s rd, 639 KiB/s wr, 14 op/s 2023-03-10T01:54:29.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:29 smithi038 ceph-mon[92422]: pgmap v21511: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 615 KiB/s rd, 639 KiB/s wr, 14 op/s 2023-03-10T01:54:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:31 smithi181 ceph-mon[94189]: pgmap v21512: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 537 KiB/s wr, 13 op/s 2023-03-10T01:54:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39381.550333 secs 2023-03-10T01:54:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:31 smithi038 ceph-mon[92422]: pgmap v21512: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 537 KiB/s wr, 13 op/s 2023-03-10T01:54:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39381.550333 secs 2023-03-10T01:54:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:31 smithi122 ceph-mon[93869]: pgmap v21512: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 537 KiB/s wr, 13 op/s 2023-03-10T01:54:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39381.550333 secs 2023-03-10T01:54:33.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:33 smithi038 ceph-mon[92422]: pgmap v21513: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 511 KiB/s wr, 11 op/s 2023-03-10T01:54:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:33 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:54:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:33 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:54:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:33 smithi122 ceph-mon[93869]: pgmap v21513: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 511 KiB/s wr, 11 op/s 2023-03-10T01:54:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:33 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:54:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:33 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:54:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:33 smithi181 ceph-mon[94189]: pgmap v21513: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 511 KiB/s wr, 11 op/s 2023-03-10T01:54:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:33 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:54:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:33 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:54:33.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39845.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:54:35.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:35 smithi038 ceph-mon[92422]: pgmap v21514: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 452 KiB/s wr, 11 op/s 2023-03-10T01:54:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:35 smithi038 ceph-mon[92422]: osdmap e1560: 12 total, 12 up, 12 in 2023-03-10T01:54:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:35 smithi122 ceph-mon[93869]: pgmap v21514: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 452 KiB/s wr, 11 op/s 2023-03-10T01:54:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:35 smithi122 ceph-mon[93869]: osdmap e1560: 12 total, 12 up, 12 in 2023-03-10T01:54:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:35 smithi181 ceph-mon[94189]: pgmap v21514: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 452 KiB/s wr, 11 op/s 2023-03-10T01:54:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:35 smithi181 ceph-mon[94189]: osdmap e1560: 12 total, 12 up, 12 in 2023-03-10T01:54:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39386.550395 secs 2023-03-10T01:54:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:54:36.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:54:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39386.550395 secs 2023-03-10T01:54:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:54:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:54:36.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39386.550395 secs 2023-03-10T01:54:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:54:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:54:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:37 smithi122 ceph-mon[93869]: pgmap v21516: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 783 KiB/s wr, 11 op/s 2023-03-10T01:54:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:54:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:54:37.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:37 smithi181 ceph-mon[94189]: pgmap v21516: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 783 KiB/s wr, 11 op/s 2023-03-10T01:54:37.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:54:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:54:37.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:37 smithi038 ceph-mon[92422]: pgmap v21516: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 783 KiB/s wr, 11 op/s 2023-03-10T01:54:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:54:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:54:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39389.012284 secs 2023-03-10T01:54:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39389.012284 secs 2023-03-10T01:54:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39389.012284 secs 2023-03-10T01:54:38.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39850.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:54:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:39 smithi122 ceph-mon[93869]: pgmap v21517: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 783 KiB/s wr, 11 op/s 2023-03-10T01:54:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:39 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:54:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:39 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:54:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:39 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:54:39.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:39 smithi181 ceph-mon[94189]: pgmap v21517: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 783 KiB/s wr, 11 op/s 2023-03-10T01:54:39.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:39 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:54:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:39 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:54:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:39 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:54:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:39 smithi038 ceph-mon[92422]: pgmap v21517: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 744 KiB/s rd, 783 KiB/s wr, 11 op/s 2023-03-10T01:54:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:39 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:54:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:39 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:54:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:39 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:54:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:41 smithi122 ceph-mon[93869]: pgmap v21518: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 774 KiB/s rd, 731 KiB/s wr, 11 op/s 2023-03-10T01:54:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39391.550455 secs 2023-03-10T01:54:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:41 smithi181 ceph-mon[94189]: pgmap v21518: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 774 KiB/s rd, 731 KiB/s wr, 11 op/s 2023-03-10T01:54:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39391.550455 secs 2023-03-10T01:54:41.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:41 smithi038 ceph-mon[92422]: pgmap v21518: 129 pgs: 129 active+clean; 11 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 774 KiB/s rd, 731 KiB/s wr, 11 op/s 2023-03-10T01:54:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39391.550455 secs 2023-03-10T01:54:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:43 smithi122 ceph-mon[93869]: pgmap v21519: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 11 op/s 2023-03-10T01:54:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39394.012345 secs 2023-03-10T01:54:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:43 smithi181 ceph-mon[94189]: pgmap v21519: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 11 op/s 2023-03-10T01:54:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39394.012345 secs 2023-03-10T01:54:43.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:43 smithi038 ceph-mon[92422]: pgmap v21519: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 11 op/s 2023-03-10T01:54:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39394.012345 secs 2023-03-10T01:54:43.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39855.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:54:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:45 smithi122 ceph-mon[93869]: pgmap v21520: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 10 op/s 2023-03-10T01:54:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:54:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:45 smithi181 ceph-mon[94189]: pgmap v21520: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 10 op/s 2023-03-10T01:54:45.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:54:45.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:45 smithi038 ceph-mon[92422]: pgmap v21520: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 10 op/s 2023-03-10T01:54:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:54:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39396.550514 secs 2023-03-10T01:54:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39396.550514 secs 2023-03-10T01:54:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39396.550514 secs 2023-03-10T01:54:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:47 smithi122 ceph-mon[93869]: pgmap v21521: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 929 KiB/s rd, 887 KiB/s wr, 13 op/s 2023-03-10T01:54:47.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:47 smithi181 ceph-mon[94189]: pgmap v21521: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 929 KiB/s rd, 887 KiB/s wr, 13 op/s 2023-03-10T01:54:47.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:47 smithi038 ceph-mon[92422]: pgmap v21521: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 929 KiB/s rd, 887 KiB/s wr, 13 op/s 2023-03-10T01:54:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:54:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:54:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:48 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 39399.012403 secs 2023-03-10T01:54:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:54:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:54:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:54:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:54:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:54:48.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:54:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:54:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:54:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:48 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 39399.012403 secs 2023-03-10T01:54:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:54:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:54:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:54:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:54:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:54:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:54:48.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:54:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:54:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:48 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 39399.012403 secs 2023-03-10T01:54:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:54:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:54:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:54:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:54:48.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:54:48.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:54:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39860.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:54:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:49 smithi122 ceph-mon[93869]: pgmap v21522: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 550 KiB/s wr, 9 op/s 2023-03-10T01:54:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:54:49.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:54:49.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:54:49.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:54:49.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:54:49.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:54:49.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:54:49.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:54:49.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:54:49.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:54:49.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:54:49.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:54:49.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:54:49.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:54:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:49 smithi181 ceph-mon[94189]: pgmap v21522: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 550 KiB/s wr, 9 op/s 2023-03-10T01:54:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:54:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:54:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:54:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:54:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:54:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:54:49.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:54:49.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:54:49.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:54:49.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:54:49.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:54:49.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:54:49.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:54:49.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:54:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:49 smithi038 ceph-mon[92422]: pgmap v21522: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 550 KiB/s wr, 9 op/s 2023-03-10T01:54:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:54:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:54:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:54:49.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:54:49.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:54:49.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:54:49.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:54:49.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:54:49.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:54:49.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:54:49.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-10T01:54:49.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:54:49.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:54:49.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:54:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:50 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-10T01:54:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:50 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-10T01:54:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:50 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-10T01:54:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:50 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-10T01:54:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:50 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-10T01:54:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:50 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-10T01:54:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:50 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-10T01:54:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:50 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-10T01:54:50.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:50 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-10T01:54:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:50 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-10T01:54:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:50 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-10T01:54:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:50 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-10T01:54:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:51 smithi122 ceph-mon[93869]: pgmap v21523: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 731 KiB/s wr, 10 op/s 2023-03-10T01:54:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39401.550594 secs 2023-03-10T01:54:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:54:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:54:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:51 smithi181 ceph-mon[94189]: pgmap v21523: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 731 KiB/s wr, 10 op/s 2023-03-10T01:54:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39401.550594 secs 2023-03-10T01:54:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:54:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:54:51.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:51 smithi038 ceph-mon[92422]: pgmap v21523: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 731 KiB/s wr, 10 op/s 2023-03-10T01:54:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39401.550594 secs 2023-03-10T01:54:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:54:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:54:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:53 smithi122 ceph-mon[93869]: pgmap v21524: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 870 KiB/s wr, 12 op/s 2023-03-10T01:54:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:53 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 39404.012465 secs 2023-03-10T01:54:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:53 smithi181 ceph-mon[94189]: pgmap v21524: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 870 KiB/s wr, 12 op/s 2023-03-10T01:54:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:53 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 39404.012465 secs 2023-03-10T01:54:53.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:53 smithi038 ceph-mon[92422]: pgmap v21524: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 870 KiB/s wr, 12 op/s 2023-03-10T01:54:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:53 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 39404.012465 secs 2023-03-10T01:54:53.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39865.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:54:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:55 smithi122 ceph-mon[93869]: pgmap v21525: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 618 KiB/s wr, 10 op/s 2023-03-10T01:54:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:55 smithi181 ceph-mon[94189]: pgmap v21525: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 618 KiB/s wr, 10 op/s 2023-03-10T01:54:55.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:55 smithi038 ceph-mon[92422]: pgmap v21525: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 618 KiB/s wr, 10 op/s 2023-03-10T01:54:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39406.550656 secs 2023-03-10T01:54:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:54:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:54:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39406.550656 secs 2023-03-10T01:54:56.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:54:56.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:54:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39406.550656 secs 2023-03-10T01:54:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:54:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:54:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:57 smithi122 ceph-mon[93869]: pgmap v21526: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 618 KiB/s wr, 12 op/s 2023-03-10T01:54:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:54:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:54:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:57 smithi181 ceph-mon[94189]: pgmap v21526: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 618 KiB/s wr, 12 op/s 2023-03-10T01:54:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:54:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:54:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:57 smithi038 ceph-mon[92422]: pgmap v21526: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 618 KiB/s wr, 12 op/s 2023-03-10T01:54:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:54:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:54:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39409.012562 secs 2023-03-10T01:54:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39409.012562 secs 2023-03-10T01:54:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39409.012562 secs 2023-03-10T01:54:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39870.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:54:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:59 smithi122 ceph-mon[93869]: pgmap v21527: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 557 KiB/s wr, 10 op/s 2023-03-10T01:54:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:54:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:54:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:54:59.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:59 smithi181 ceph-mon[94189]: pgmap v21527: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 557 KiB/s wr, 10 op/s 2023-03-10T01:54:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:54:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:54:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:54:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:59 smithi038 ceph-mon[92422]: pgmap v21527: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 557 KiB/s wr, 10 op/s 2023-03-10T01:54:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:54:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:54:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:55:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:55:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:55:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:55:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:55:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:55:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:55:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:01 smithi122 ceph-mon[93869]: pgmap v21528: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 591 KiB/s wr, 11 op/s 2023-03-10T01:55:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39411.550714 secs 2023-03-10T01:55:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:55:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:55:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:01 smithi181 ceph-mon[94189]: pgmap v21528: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 591 KiB/s wr, 11 op/s 2023-03-10T01:55:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39411.550714 secs 2023-03-10T01:55:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:55:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:55:01.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:01 smithi038 ceph-mon[92422]: pgmap v21528: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 591 KiB/s wr, 11 op/s 2023-03-10T01:55:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39411.550714 secs 2023-03-10T01:55:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:55:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:55:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:03 smithi122 ceph-mon[93869]: pgmap v21529: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 932 KiB/s wr, 13 op/s 2023-03-10T01:55:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39414.012672 secs 2023-03-10T01:55:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:03 smithi181 ceph-mon[94189]: pgmap v21529: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 932 KiB/s wr, 13 op/s 2023-03-10T01:55:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39414.012672 secs 2023-03-10T01:55:03.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:03 smithi038 ceph-mon[92422]: pgmap v21529: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 932 KiB/s wr, 13 op/s 2023-03-10T01:55:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39414.012672 secs 2023-03-10T01:55:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39875.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:55:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:05 smithi122 ceph-mon[93869]: pgmap v21530: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 790 KiB/s wr, 22 op/s 2023-03-10T01:55:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:05 smithi181 ceph-mon[94189]: pgmap v21530: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 790 KiB/s wr, 22 op/s 2023-03-10T01:55:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:05 smithi038 ceph-mon[92422]: pgmap v21530: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 790 KiB/s wr, 22 op/s 2023-03-10T01:55:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39416.550773 secs 2023-03-10T01:55:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39416.550773 secs 2023-03-10T01:55:06.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39416.550773 secs 2023-03-10T01:55:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:07 smithi122 ceph-mon[93869]: pgmap v21531: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 918 KiB/s wr, 25 op/s 2023-03-10T01:55:07.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:07 smithi181 ceph-mon[94189]: pgmap v21531: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 918 KiB/s wr, 25 op/s 2023-03-10T01:55:07.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:07 smithi038 ceph-mon[92422]: pgmap v21531: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 918 KiB/s wr, 25 op/s 2023-03-10T01:55:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:55:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:55:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39419.012732 secs 2023-03-10T01:55:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:55:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:55:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39419.012732 secs 2023-03-10T01:55:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:55:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:55:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39419.012732 secs 2023-03-10T01:55:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39880.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:55:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:09 smithi122 ceph-mon[93869]: pgmap v21532: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.0 MiB/s wr, 25 op/s 2023-03-10T01:55:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:55:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:55:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:09 smithi181 ceph-mon[94189]: pgmap v21532: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.0 MiB/s wr, 25 op/s 2023-03-10T01:55:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:55:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:55:09.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:09 smithi038 ceph-mon[92422]: pgmap v21532: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.0 MiB/s wr, 25 op/s 2023-03-10T01:55:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:55:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:55:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:11 smithi122 ceph-mon[93869]: pgmap v21533: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 876 KiB/s wr, 24 op/s 2023-03-10T01:55:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39421.550853 secs 2023-03-10T01:55:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:11 smithi181 ceph-mon[94189]: pgmap v21533: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 876 KiB/s wr, 24 op/s 2023-03-10T01:55:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39421.550853 secs 2023-03-10T01:55:11.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:11 smithi038 ceph-mon[92422]: pgmap v21533: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 876 KiB/s wr, 24 op/s 2023-03-10T01:55:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39421.550853 secs 2023-03-10T01:55:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:55:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:55:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:55:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:55:12.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:55:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:55:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:13 smithi122 ceph-mon[93869]: pgmap v21534: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.2 MiB/s wr, 26 op/s 2023-03-10T01:55:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39424.012789 secs 2023-03-10T01:55:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:13 smithi181 ceph-mon[94189]: pgmap v21534: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.2 MiB/s wr, 26 op/s 2023-03-10T01:55:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39424.012789 secs 2023-03-10T01:55:13.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:13 smithi038 ceph-mon[92422]: pgmap v21534: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.2 MiB/s wr, 26 op/s 2023-03-10T01:55:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39424.012789 secs 2023-03-10T01:55:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39885.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:55:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:15 smithi122 ceph-mon[93869]: pgmap v21535: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 873 KiB/s rd, 675 KiB/s wr, 22 op/s 2023-03-10T01:55:15.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:15 smithi181 ceph-mon[94189]: pgmap v21535: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 873 KiB/s rd, 675 KiB/s wr, 22 op/s 2023-03-10T01:55:15.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:15 smithi038 ceph-mon[92422]: pgmap v21535: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 873 KiB/s rd, 675 KiB/s wr, 22 op/s 2023-03-10T01:55:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39426.550911 secs 2023-03-10T01:55:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:55:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:55:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39426.550911 secs 2023-03-10T01:55:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:55:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:55:16.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39426.550911 secs 2023-03-10T01:55:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:55:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:55:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:17 smithi122 ceph-mon[93869]: pgmap v21536: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 834 KiB/s rd, 630 KiB/s wr, 12 op/s 2023-03-10T01:55:17.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:55:17.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:55:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:17 smithi181 ceph-mon[94189]: pgmap v21536: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 834 KiB/s rd, 630 KiB/s wr, 12 op/s 2023-03-10T01:55:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:55:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:55:17.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:17 smithi038 ceph-mon[92422]: pgmap v21536: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 834 KiB/s rd, 630 KiB/s wr, 12 op/s 2023-03-10T01:55:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:55:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:55:18.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39429.012874 secs 2023-03-10T01:55:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39890.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:55:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39429.012874 secs 2023-03-10T01:55:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39429.012874 secs 2023-03-10T01:55:19.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:19 smithi038 ceph-mon[92422]: pgmap v21537: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 469 KiB/s wr, 9 op/s 2023-03-10T01:55:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:19 smithi122 ceph-mon[93869]: pgmap v21537: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 469 KiB/s wr, 9 op/s 2023-03-10T01:55:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:19 smithi181 ceph-mon[94189]: pgmap v21537: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 469 KiB/s wr, 9 op/s 2023-03-10T01:55:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:55:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:55:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:55:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:55:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:55:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:55:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:21 smithi122 ceph-mon[93869]: pgmap v21538: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 400 KiB/s wr, 9 op/s 2023-03-10T01:55:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39431.551009 secs 2023-03-10T01:55:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:55:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:55:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:21 smithi181 ceph-mon[94189]: pgmap v21538: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 400 KiB/s wr, 9 op/s 2023-03-10T01:55:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39431.551009 secs 2023-03-10T01:55:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:55:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:55:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:21 smithi038 ceph-mon[92422]: pgmap v21538: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 358 KiB/s rd, 400 KiB/s wr, 9 op/s 2023-03-10T01:55:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39431.551009 secs 2023-03-10T01:55:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:55:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:55:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39895.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:55:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:23 smithi122 ceph-mon[93869]: pgmap v21539: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 585 KiB/s rd, 637 KiB/s wr, 12 op/s 2023-03-10T01:55:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39434.012935 secs 2023-03-10T01:55:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:23 smithi181 ceph-mon[94189]: pgmap v21539: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 585 KiB/s rd, 637 KiB/s wr, 12 op/s 2023-03-10T01:55:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39434.012935 secs 2023-03-10T01:55:23.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:23 smithi038 ceph-mon[92422]: pgmap v21539: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 585 KiB/s rd, 637 KiB/s wr, 12 op/s 2023-03-10T01:55:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39434.012935 secs 2023-03-10T01:55:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:25 smithi122 ceph-mon[93869]: pgmap v21540: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 570 KiB/s rd, 594 KiB/s wr, 11 op/s 2023-03-10T01:55:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:25 smithi122 ceph-mon[93869]: osdmap e1561: 12 total, 12 up, 12 in 2023-03-10T01:55:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:25 smithi181 ceph-mon[94189]: pgmap v21540: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 570 KiB/s rd, 594 KiB/s wr, 11 op/s 2023-03-10T01:55:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:25 smithi181 ceph-mon[94189]: osdmap e1561: 12 total, 12 up, 12 in 2023-03-10T01:55:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:25 smithi038 ceph-mon[92422]: pgmap v21540: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 570 KiB/s rd, 594 KiB/s wr, 11 op/s 2023-03-10T01:55:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:25 smithi038 ceph-mon[92422]: osdmap e1561: 12 total, 12 up, 12 in 2023-03-10T01:55:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39436.551067 secs 2023-03-10T01:55:26.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39436.551067 secs 2023-03-10T01:55:26.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39436.551067 secs 2023-03-10T01:55:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:27 smithi122 ceph-mon[93869]: pgmap v21542: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 684 KiB/s rd, 713 KiB/s wr, 12 op/s 2023-03-10T01:55:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:27 smithi181 ceph-mon[94189]: pgmap v21542: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 684 KiB/s rd, 713 KiB/s wr, 12 op/s 2023-03-10T01:55:27.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:27 smithi038 ceph-mon[92422]: pgmap v21542: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 684 KiB/s rd, 713 KiB/s wr, 12 op/s 2023-03-10T01:55:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39900.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:55:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39439.013018 secs 2023-03-10T01:55:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:28 smithi122 ceph-mon[93869]: pgmap v21543: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 725 KiB/s rd, 753 KiB/s wr, 12 op/s 2023-03-10T01:55:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39439.013018 secs 2023-03-10T01:55:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:28 smithi181 ceph-mon[94189]: pgmap v21543: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 725 KiB/s rd, 753 KiB/s wr, 12 op/s 2023-03-10T01:55:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39439.013018 secs 2023-03-10T01:55:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:28 smithi038 ceph-mon[92422]: pgmap v21543: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 725 KiB/s rd, 753 KiB/s wr, 12 op/s 2023-03-10T01:55:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:31 smithi122 ceph-mon[93869]: pgmap v21544: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 887 KiB/s rd, 894 KiB/s wr, 12 op/s 2023-03-10T01:55:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39441.551131 secs 2023-03-10T01:55:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:31 smithi122 ceph-mon[93869]: osdmap e1562: 12 total, 12 up, 12 in 2023-03-10T01:55:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:31 smithi181 ceph-mon[94189]: pgmap v21544: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 887 KiB/s rd, 894 KiB/s wr, 12 op/s 2023-03-10T01:55:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39441.551131 secs 2023-03-10T01:55:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:31 smithi181 ceph-mon[94189]: osdmap e1562: 12 total, 12 up, 12 in 2023-03-10T01:55:31.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:31 smithi038 ceph-mon[92422]: pgmap v21544: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 887 KiB/s rd, 894 KiB/s wr, 12 op/s 2023-03-10T01:55:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39441.551131 secs 2023-03-10T01:55:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:31 smithi038 ceph-mon[92422]: osdmap e1562: 12 total, 12 up, 12 in 2023-03-10T01:55:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:32 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:55:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:32 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:55:32.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:32 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:55:32.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:32 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:55:32.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:32 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:55:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:32 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:55:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:33 smithi122 ceph-mon[93869]: pgmap v21546: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 294 KiB/s wr, 9 op/s 2023-03-10T01:55:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:33 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:55:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:33 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:55:33.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:33 smithi181 ceph-mon[94189]: pgmap v21546: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 294 KiB/s wr, 9 op/s 2023-03-10T01:55:33.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:33 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:55:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:33 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:55:33.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:33 smithi038 ceph-mon[92422]: pgmap v21546: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 294 KiB/s wr, 9 op/s 2023-03-10T01:55:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:33 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:55:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:33 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:55:33.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39905.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:55:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:34 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39444.013077 secs 2023-03-10T01:55:34.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:34 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39444.013077 secs 2023-03-10T01:55:34.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:34 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39444.013077 secs 2023-03-10T01:55:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:35 smithi122 ceph-mon[93869]: pgmap v21547: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 540 KiB/s wr, 10 op/s 2023-03-10T01:55:35.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:35 smithi181 ceph-mon[94189]: pgmap v21547: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 540 KiB/s wr, 10 op/s 2023-03-10T01:55:35.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:35 smithi038 ceph-mon[92422]: pgmap v21547: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 540 KiB/s wr, 10 op/s 2023-03-10T01:55:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39446.551213 secs 2023-03-10T01:55:36.481 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39446.551213 secs 2023-03-10T01:55:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39446.551213 secs 2023-03-10T01:55:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:37 smithi122 ceph-mon[93869]: pgmap v21548: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 497 KiB/s wr, 9 op/s 2023-03-10T01:55:37.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:37 smithi181 ceph-mon[94189]: pgmap v21548: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 497 KiB/s wr, 9 op/s 2023-03-10T01:55:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:37 smithi038 ceph-mon[92422]: pgmap v21548: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 252 KiB/s rd, 497 KiB/s wr, 9 op/s 2023-03-10T01:55:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39910.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:55:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:39 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39449.013198 secs 2023-03-10T01:55:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:39 smithi122 ceph-mon[93869]: pgmap v21549: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 457 KiB/s wr, 9 op/s 2023-03-10T01:55:39.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:39 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39449.013198 secs 2023-03-10T01:55:39.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:39 smithi181 ceph-mon[94189]: pgmap v21549: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 457 KiB/s wr, 9 op/s 2023-03-10T01:55:39.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:39 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39449.013198 secs 2023-03-10T01:55:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:39 smithi038 ceph-mon[92422]: pgmap v21549: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 211 KiB/s rd, 457 KiB/s wr, 9 op/s 2023-03-10T01:55:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:41 smithi122 ceph-mon[93869]: pgmap v21550: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 264 KiB/s wr, 9 op/s 2023-03-10T01:55:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39451.551288 secs 2023-03-10T01:55:41.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:41 smithi181 ceph-mon[94189]: pgmap v21550: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 264 KiB/s wr, 9 op/s 2023-03-10T01:55:41.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39451.551288 secs 2023-03-10T01:55:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:41 smithi038 ceph-mon[92422]: pgmap v21550: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 264 KiB/s wr, 9 op/s 2023-03-10T01:55:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39451.551288 secs 2023-03-10T01:55:43.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:43 smithi122 ceph-mon[93869]: pgmap v21551: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 228 KiB/s wr, 10 op/s 2023-03-10T01:55:43.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:43 smithi181 ceph-mon[94189]: pgmap v21551: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 228 KiB/s wr, 10 op/s 2023-03-10T01:55:43.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:43 smithi038 ceph-mon[92422]: pgmap v21551: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 261 KiB/s rd, 228 KiB/s wr, 10 op/s 2023-03-10T01:55:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39915.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:55:44.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:44 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39454.013293 secs 2023-03-10T01:55:44.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:44 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39454.013293 secs 2023-03-10T01:55:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:44 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39454.013293 secs 2023-03-10T01:55:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:45 smithi122 ceph-mon[93869]: pgmap v21552: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 728 KiB/s rd, 738 KiB/s wr, 10 op/s 2023-03-10T01:55:45.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:45 smithi181 ceph-mon[94189]: pgmap v21552: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 728 KiB/s rd, 738 KiB/s wr, 10 op/s 2023-03-10T01:55:45.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:45 smithi038 ceph-mon[92422]: pgmap v21552: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 728 KiB/s rd, 738 KiB/s wr, 10 op/s 2023-03-10T01:55:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39456.551346 secs 2023-03-10T01:55:46.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39456.551346 secs 2023-03-10T01:55:46.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39456.551346 secs 2023-03-10T01:55:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:47 smithi122 ceph-mon[93869]: pgmap v21553: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 692 KiB/s rd, 519 KiB/s wr, 9 op/s 2023-03-10T01:55:47.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:47 smithi181 ceph-mon[94189]: pgmap v21553: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 692 KiB/s rd, 519 KiB/s wr, 9 op/s 2023-03-10T01:55:47.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:47 smithi038 ceph-mon[92422]: pgmap v21553: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 692 KiB/s rd, 519 KiB/s wr, 9 op/s 2023-03-10T01:55:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:55:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:55:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:55:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:55:48.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:55:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:55:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39920.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:55:49.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:49 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39459.013353 secs 2023-03-10T01:55:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:49 smithi181 ceph-mon[94189]: pgmap v21554: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 724 KiB/s rd, 552 KiB/s wr, 8 op/s 2023-03-10T01:55:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:55:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:55:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:55:49.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:49 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39459.013353 secs 2023-03-10T01:55:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:49 smithi038 ceph-mon[92422]: pgmap v21554: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 724 KiB/s rd, 552 KiB/s wr, 8 op/s 2023-03-10T01:55:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:55:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:55:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:55:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:49 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39459.013353 secs 2023-03-10T01:55:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:49 smithi122 ceph-mon[93869]: pgmap v21554: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 724 KiB/s rd, 552 KiB/s wr, 8 op/s 2023-03-10T01:55:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:55:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:55:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:55:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:50 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:55:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:50 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:55:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:50 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:55:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:50 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:55:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:50 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:55:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:50 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:55:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:50 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:55:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:50 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:55:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:50 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:55:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:51 smithi038 ceph-mon[92422]: pgmap v21555: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 710 KiB/s wr, 10 op/s 2023-03-10T01:55:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39461.551403 secs 2023-03-10T01:55:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:51 smithi122 ceph-mon[93869]: pgmap v21555: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 710 KiB/s wr, 10 op/s 2023-03-10T01:55:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39461.551403 secs 2023-03-10T01:55:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:51 smithi181 ceph-mon[94189]: pgmap v21555: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 710 KiB/s wr, 10 op/s 2023-03-10T01:55:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39461.551403 secs 2023-03-10T01:55:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:55:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:55:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:52 smithi122 ceph-mon[93869]: pgmap v21556: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 920 KiB/s wr, 10 op/s 2023-03-10T01:55:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:55:52.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:55:52.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:52 smithi181 ceph-mon[94189]: pgmap v21556: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 920 KiB/s wr, 10 op/s 2023-03-10T01:55:52.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:55:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:55:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:52 smithi038 ceph-mon[92422]: pgmap v21556: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 920 KiB/s wr, 10 op/s 2023-03-10T01:55:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39925.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:55:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39464.013431 secs 2023-03-10T01:55:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39464.013431 secs 2023-03-10T01:55:53.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39464.013431 secs 2023-03-10T01:55:54.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:54 smithi122 ceph-mon[93869]: pgmap v21557: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 954 KiB/s wr, 11 op/s 2023-03-10T01:55:54.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:54 smithi181 ceph-mon[94189]: pgmap v21557: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 954 KiB/s wr, 11 op/s 2023-03-10T01:55:54.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:54 smithi038 ceph-mon[92422]: pgmap v21557: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 954 KiB/s wr, 11 op/s 2023-03-10T01:55:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39466.551488 secs 2023-03-10T01:55:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:55:56.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:55:56.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:56 smithi122 ceph-mon[93869]: pgmap v21558: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 697 KiB/s wr, 12 op/s 2023-03-10T01:55:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39466.551488 secs 2023-03-10T01:55:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:55:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:55:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:56 smithi181 ceph-mon[94189]: pgmap v21558: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 697 KiB/s wr, 12 op/s 2023-03-10T01:55:56.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39466.551488 secs 2023-03-10T01:55:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:55:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:55:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:56 smithi038 ceph-mon[92422]: pgmap v21558: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 697 KiB/s wr, 12 op/s 2023-03-10T01:55:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:55:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:55:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:57 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:55:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:55:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:55:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:57 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:55:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:55:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:55:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:57 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:55:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39930.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:55:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:58 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:55:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:58 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:55:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:58 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:55:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:58 smithi122 ceph-mon[93869]: pgmap v21559: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 697 KiB/s wr, 11 op/s 2023-03-10T01:55:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:58 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:55:58.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:58 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:55:58.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:58 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:55:58.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:58 smithi181 ceph-mon[94189]: pgmap v21559: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 697 KiB/s wr, 11 op/s 2023-03-10T01:55:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:58 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:55:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:58 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:55:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:58 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:55:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:58 smithi038 ceph-mon[92422]: pgmap v21559: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 697 KiB/s wr, 11 op/s 2023-03-10T01:55:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:55:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:55:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:55:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:55:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:55:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:55:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:55:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:55:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:56:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:01 smithi122 ceph-mon[93869]: pgmap v21560: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 676 KiB/s wr, 12 op/s 2023-03-10T01:56:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39471.551609 secs 2023-03-10T01:56:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:56:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:56:01.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:01 smithi181 ceph-mon[94189]: pgmap v21560: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 676 KiB/s wr, 12 op/s 2023-03-10T01:56:01.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39471.551609 secs 2023-03-10T01:56:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:56:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:56:01.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:01 smithi038 ceph-mon[92422]: pgmap v21560: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 676 KiB/s wr, 12 op/s 2023-03-10T01:56:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39471.551609 secs 2023-03-10T01:56:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:56:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:56:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:03 smithi122 ceph-mon[93869]: pgmap v21561: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 994 KiB/s wr, 15 op/s 2023-03-10T01:56:03.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:03 smithi181 ceph-mon[94189]: pgmap v21561: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 994 KiB/s wr, 15 op/s 2023-03-10T01:56:03.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:03 smithi038 ceph-mon[92422]: pgmap v21561: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 994 KiB/s wr, 15 op/s 2023-03-10T01:56:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39935.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:56:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:05 smithi122 ceph-mon[93869]: pgmap v21562: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 793 KiB/s wr, 24 op/s 2023-03-10T01:56:05.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:05 smithi181 ceph-mon[94189]: pgmap v21562: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 793 KiB/s wr, 24 op/s 2023-03-10T01:56:05.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:05 smithi038 ceph-mon[92422]: pgmap v21562: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 793 KiB/s wr, 24 op/s 2023-03-10T01:56:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39476.551668 secs 2023-03-10T01:56:06.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39476.551668 secs 2023-03-10T01:56:06.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39476.551668 secs 2023-03-10T01:56:07.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:07 smithi122 ceph-mon[93869]: pgmap v21563: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 894 KiB/s wr, 26 op/s 2023-03-10T01:56:07.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:07 smithi181 ceph-mon[94189]: pgmap v21563: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 894 KiB/s wr, 26 op/s 2023-03-10T01:56:07.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:07 smithi038 ceph-mon[92422]: pgmap v21563: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 894 KiB/s wr, 26 op/s 2023-03-10T01:56:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:56:08.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:56:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:56:08.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:56:08.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:56:08.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:56:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39940.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:56:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:09 smithi122 ceph-mon[93869]: pgmap v21564: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 772 KiB/s rd, 644 KiB/s wr, 24 op/s 2023-03-10T01:56:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:56:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:56:09.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:09 smithi181 ceph-mon[94189]: pgmap v21564: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 772 KiB/s rd, 644 KiB/s wr, 24 op/s 2023-03-10T01:56:09.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:56:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:56:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:09 smithi038 ceph-mon[92422]: pgmap v21564: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 772 KiB/s rd, 644 KiB/s wr, 24 op/s 2023-03-10T01:56:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:56:09.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:56:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:11 smithi122 ceph-mon[93869]: pgmap v21565: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 901 KiB/s rd, 774 KiB/s wr, 25 op/s 2023-03-10T01:56:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39481.551724 secs 2023-03-10T01:56:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:11 smithi181 ceph-mon[94189]: pgmap v21565: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 901 KiB/s rd, 774 KiB/s wr, 25 op/s 2023-03-10T01:56:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39481.551724 secs 2023-03-10T01:56:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:11 smithi038 ceph-mon[92422]: pgmap v21565: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 901 KiB/s rd, 774 KiB/s wr, 25 op/s 2023-03-10T01:56:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39481.551724 secs 2023-03-10T01:56:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:56:12.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:56:12.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:56:12.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:56:12.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:56:12.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:56:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:13 smithi122 ceph-mon[93869]: pgmap v21566: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 880 KiB/s rd, 763 KiB/s wr, 26 op/s 2023-03-10T01:56:13.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:13 smithi181 ceph-mon[94189]: pgmap v21566: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 880 KiB/s rd, 763 KiB/s wr, 26 op/s 2023-03-10T01:56:13.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:13 smithi038 ceph-mon[92422]: pgmap v21566: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 880 KiB/s rd, 763 KiB/s wr, 26 op/s 2023-03-10T01:56:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39945.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:56:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:15 smithi122 ceph-mon[93869]: pgmap v21567: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 592 KiB/s rd, 478 KiB/s wr, 23 op/s 2023-03-10T01:56:15.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:15 smithi181 ceph-mon[94189]: pgmap v21567: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 592 KiB/s rd, 478 KiB/s wr, 23 op/s 2023-03-10T01:56:15.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:15 smithi038 ceph-mon[92422]: pgmap v21567: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 592 KiB/s rd, 478 KiB/s wr, 23 op/s 2023-03-10T01:56:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39486.551813 secs 2023-03-10T01:56:16.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39486.551813 secs 2023-03-10T01:56:16.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39486.551813 secs 2023-03-10T01:56:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:17 smithi122 ceph-mon[93869]: pgmap v21568: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 467 KiB/s wr, 13 op/s 2023-03-10T01:56:17.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:17 smithi181 ceph-mon[94189]: pgmap v21568: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 467 KiB/s wr, 13 op/s 2023-03-10T01:56:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:17 smithi038 ceph-mon[92422]: pgmap v21568: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 587 KiB/s rd, 467 KiB/s wr, 13 op/s 2023-03-10T01:56:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39950.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:56:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:19 smithi122 ceph-mon[93869]: pgmap v21569: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 333 KiB/s wr, 9 op/s 2023-03-10T01:56:19.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:19 smithi181 ceph-mon[94189]: pgmap v21569: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 333 KiB/s wr, 9 op/s 2023-03-10T01:56:19.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:19 smithi038 ceph-mon[92422]: pgmap v21569: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 325 KiB/s rd, 333 KiB/s wr, 9 op/s 2023-03-10T01:56:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:56:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:56:20.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:56:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:56:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:56:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:56:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:21 smithi038 ceph-mon[92422]: pgmap v21570: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 353 KiB/s wr, 10 op/s 2023-03-10T01:56:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39491.551867 secs 2023-03-10T01:56:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:56:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:56:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:21 smithi122 ceph-mon[93869]: pgmap v21570: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 353 KiB/s wr, 10 op/s 2023-03-10T01:56:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39491.551867 secs 2023-03-10T01:56:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:56:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:56:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:21 smithi181 ceph-mon[94189]: pgmap v21570: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 353 KiB/s wr, 10 op/s 2023-03-10T01:56:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39491.551867 secs 2023-03-10T01:56:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:56:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:56:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:23 smithi038 ceph-mon[92422]: pgmap v21571: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 591 KiB/s rd, 609 KiB/s wr, 13 op/s 2023-03-10T01:56:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:23 smithi122 ceph-mon[93869]: pgmap v21571: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 591 KiB/s rd, 609 KiB/s wr, 13 op/s 2023-03-10T01:56:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:23 smithi181 ceph-mon[94189]: pgmap v21571: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 591 KiB/s rd, 609 KiB/s wr, 13 op/s 2023-03-10T01:56:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39955.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:56:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:25 smithi038 ceph-mon[92422]: pgmap v21572: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 598 KiB/s rd, 618 KiB/s wr, 11 op/s 2023-03-10T01:56:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:25 smithi122 ceph-mon[93869]: pgmap v21572: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 598 KiB/s rd, 618 KiB/s wr, 11 op/s 2023-03-10T01:56:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:25 smithi181 ceph-mon[94189]: pgmap v21572: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 598 KiB/s rd, 618 KiB/s wr, 11 op/s 2023-03-10T01:56:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39496.551936 secs 2023-03-10T01:56:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:26 smithi122 ceph-mon[93869]: osdmap e1563: 12 total, 12 up, 12 in 2023-03-10T01:56:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39496.551936 secs 2023-03-10T01:56:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:26 smithi181 ceph-mon[94189]: osdmap e1563: 12 total, 12 up, 12 in 2023-03-10T01:56:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39496.551936 secs 2023-03-10T01:56:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:26 smithi038 ceph-mon[92422]: osdmap e1563: 12 total, 12 up, 12 in 2023-03-10T01:56:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:27 smithi122 ceph-mon[93869]: pgmap v21574: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 513 KiB/s wr, 13 op/s 2023-03-10T01:56:27.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:27 smithi181 ceph-mon[94189]: pgmap v21574: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 513 KiB/s wr, 13 op/s 2023-03-10T01:56:27.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:27 smithi038 ceph-mon[92422]: pgmap v21574: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 513 KiB/s wr, 13 op/s 2023-03-10T01:56:28.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39960.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:56:29.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:29 smithi122 ceph-mon[93869]: pgmap v21575: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 688 KiB/s rd, 711 KiB/s wr, 14 op/s 2023-03-10T01:56:29.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:29 smithi181 ceph-mon[94189]: pgmap v21575: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 688 KiB/s rd, 711 KiB/s wr, 14 op/s 2023-03-10T01:56:29.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:29 smithi038 ceph-mon[92422]: pgmap v21575: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 688 KiB/s rd, 711 KiB/s wr, 14 op/s 2023-03-10T01:56:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:31 smithi122 ceph-mon[93869]: pgmap v21576: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 663 KiB/s rd, 674 KiB/s wr, 11 op/s 2023-03-10T01:56:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39501.552016 secs 2023-03-10T01:56:31.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:31 smithi181 ceph-mon[94189]: pgmap v21576: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 663 KiB/s rd, 674 KiB/s wr, 11 op/s 2023-03-10T01:56:31.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39501.552016 secs 2023-03-10T01:56:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:31 smithi038 ceph-mon[92422]: pgmap v21576: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 663 KiB/s rd, 674 KiB/s wr, 11 op/s 2023-03-10T01:56:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39501.552016 secs 2023-03-10T01:56:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:32 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:56:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:32 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:56:32.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:32 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:56:32.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:32 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:56:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:32 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:56:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:32 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:56:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:33 smithi122 ceph-mon[93869]: pgmap v21577: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 252 KiB/s wr, 8 op/s 2023-03-10T01:56:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:33 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:56:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:33 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:56:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:33 smithi181 ceph-mon[94189]: pgmap v21577: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 252 KiB/s wr, 8 op/s 2023-03-10T01:56:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:33 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:56:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:33 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:56:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:33 smithi038 ceph-mon[92422]: pgmap v21577: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 252 KiB/s wr, 8 op/s 2023-03-10T01:56:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:33 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:56:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:33 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:56:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39965.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:56:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:35 smithi122 ceph-mon[93869]: pgmap v21578: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 644 KiB/s rd, 442 KiB/s wr, 8 op/s 2023-03-10T01:56:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:35 smithi122 ceph-mon[93869]: osdmap e1564: 12 total, 12 up, 12 in 2023-03-10T01:56:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:35 smithi181 ceph-mon[94189]: pgmap v21578: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 644 KiB/s rd, 442 KiB/s wr, 8 op/s 2023-03-10T01:56:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:35 smithi181 ceph-mon[94189]: osdmap e1564: 12 total, 12 up, 12 in 2023-03-10T01:56:35.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:35 smithi038 ceph-mon[92422]: pgmap v21578: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 644 KiB/s rd, 442 KiB/s wr, 8 op/s 2023-03-10T01:56:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:35 smithi038 ceph-mon[92422]: osdmap e1564: 12 total, 12 up, 12 in 2023-03-10T01:56:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39506.552075 secs 2023-03-10T01:56:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39506.552075 secs 2023-03-10T01:56:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39506.552075 secs 2023-03-10T01:56:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:37 smithi122 ceph-mon[93869]: pgmap v21580: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 684 KiB/s rd, 482 KiB/s wr, 8 op/s 2023-03-10T01:56:37.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:37 smithi181 ceph-mon[94189]: pgmap v21580: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 684 KiB/s rd, 482 KiB/s wr, 8 op/s 2023-03-10T01:56:37.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:37 smithi038 ceph-mon[92422]: pgmap v21580: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 684 KiB/s rd, 482 KiB/s wr, 8 op/s 2023-03-10T01:56:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39509.014150 secs 2023-03-10T01:56:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39509.014150 secs 2023-03-10T01:56:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39509.014150 secs 2023-03-10T01:56:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39970.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:56:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:39 smithi122 ceph-mon[93869]: pgmap v21581: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 283 KiB/s wr, 8 op/s 2023-03-10T01:56:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:39 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:56:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:39 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:56:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:39 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:56:39.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:39 smithi181 ceph-mon[94189]: pgmap v21581: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 283 KiB/s wr, 8 op/s 2023-03-10T01:56:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:39 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:56:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:39 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:56:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:39 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:56:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:39 smithi038 ceph-mon[92422]: pgmap v21581: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 283 KiB/s wr, 8 op/s 2023-03-10T01:56:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:39 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:56:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:39 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:56:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:39 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:56:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:56:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:56:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:56:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:56:40.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:56:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:56:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:41 smithi122 ceph-mon[93869]: pgmap v21582: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 693 KiB/s wr, 9 op/s 2023-03-10T01:56:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39511.552159 secs 2023-03-10T01:56:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:56:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:56:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:41 smithi181 ceph-mon[94189]: pgmap v21582: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 693 KiB/s wr, 9 op/s 2023-03-10T01:56:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39511.552159 secs 2023-03-10T01:56:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:56:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:56:41.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:41 smithi038 ceph-mon[92422]: pgmap v21582: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 693 KiB/s wr, 9 op/s 2023-03-10T01:56:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39511.552159 secs 2023-03-10T01:56:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:56:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:56:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:43 smithi122 ceph-mon[93869]: pgmap v21583: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 917 KiB/s wr, 10 op/s 2023-03-10T01:56:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39514.014209 secs 2023-03-10T01:56:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:43 smithi181 ceph-mon[94189]: pgmap v21583: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 917 KiB/s wr, 10 op/s 2023-03-10T01:56:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39514.014209 secs 2023-03-10T01:56:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:43 smithi038 ceph-mon[92422]: pgmap v21583: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 917 KiB/s wr, 10 op/s 2023-03-10T01:56:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39514.014209 secs 2023-03-10T01:56:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39975.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:56:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:45 smithi122 ceph-mon[93869]: pgmap v21584: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 717 KiB/s wr, 10 op/s 2023-03-10T01:56:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:45 smithi181 ceph-mon[94189]: pgmap v21584: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 717 KiB/s wr, 10 op/s 2023-03-10T01:56:45.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:45 smithi038 ceph-mon[92422]: pgmap v21584: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 717 KiB/s wr, 10 op/s 2023-03-10T01:56:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39516.552217 secs 2023-03-10T01:56:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39516.552217 secs 2023-03-10T01:56:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39516.552217 secs 2023-03-10T01:56:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:47 smithi122 ceph-mon[93869]: pgmap v21585: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 944 KiB/s wr, 13 op/s 2023-03-10T01:56:47.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:47 smithi181 ceph-mon[94189]: pgmap v21585: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 944 KiB/s wr, 13 op/s 2023-03-10T01:56:47.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:47 smithi038 ceph-mon[92422]: pgmap v21585: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 944 KiB/s wr, 13 op/s 2023-03-10T01:56:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:56:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:56:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:56:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:56:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:56:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:56:48.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39980.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:56:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:49 smithi122 ceph-mon[93869]: pgmap v21586: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 921 KiB/s wr, 9 op/s 2023-03-10T01:56:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:56:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:56:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:49 smithi181 ceph-mon[94189]: pgmap v21586: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 921 KiB/s wr, 9 op/s 2023-03-10T01:56:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:56:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:56:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:49 smithi038 ceph-mon[92422]: pgmap v21586: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 921 KiB/s wr, 9 op/s 2023-03-10T01:56:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:56:49.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:56:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:50 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:56:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:50 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:56:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:50 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:56:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:50 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:56:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:50 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:56:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:50 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:56:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:50 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:56:50.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:50 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:56:50.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:50 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:56:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:50 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:56:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:50 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:56:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:50 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:56:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:51 smithi122 ceph-mon[93869]: pgmap v21587: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 923 KiB/s rd, 921 KiB/s wr, 9 op/s 2023-03-10T01:56:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39521.552327 secs 2023-03-10T01:56:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:51 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:56:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:56:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:56:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:56:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:56:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:56:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:51 smithi181 ceph-mon[94189]: pgmap v21587: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 923 KiB/s rd, 921 KiB/s wr, 9 op/s 2023-03-10T01:56:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39521.552327 secs 2023-03-10T01:56:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:51 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:56:51.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:56:51.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:56:51.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:56:51.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:56:51.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:56:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:51 smithi038 ceph-mon[92422]: pgmap v21587: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 923 KiB/s rd, 921 KiB/s wr, 9 op/s 2023-03-10T01:56:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39521.552327 secs 2023-03-10T01:56:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:51 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:56:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:56:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:56:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:56:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:56:51.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:56:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:56:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:56:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:56:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:56:52.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:56:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:56:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:53 smithi122 ceph-mon[93869]: pgmap v21588: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 721 KiB/s rd, 718 KiB/s wr, 11 op/s 2023-03-10T01:56:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:53 smithi181 ceph-mon[94189]: pgmap v21588: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 721 KiB/s rd, 718 KiB/s wr, 11 op/s 2023-03-10T01:56:53.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:53 smithi038 ceph-mon[92422]: pgmap v21588: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 721 KiB/s rd, 718 KiB/s wr, 11 op/s 2023-03-10T01:56:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39985.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:56:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:55 smithi122 ceph-mon[93869]: pgmap v21589: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 574 KiB/s wr, 9 op/s 2023-03-10T01:56:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:55 smithi181 ceph-mon[94189]: pgmap v21589: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 574 KiB/s wr, 9 op/s 2023-03-10T01:56:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:55 smithi038 ceph-mon[92422]: pgmap v21589: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 574 KiB/s wr, 9 op/s 2023-03-10T01:56:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39526.552385 secs 2023-03-10T01:56:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:56:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:56:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39526.552385 secs 2023-03-10T01:56:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:56:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:56:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39526.552385 secs 2023-03-10T01:56:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:56:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:56:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:57 smithi122 ceph-mon[93869]: pgmap v21590: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 613 KiB/s rd, 608 KiB/s wr, 12 op/s 2023-03-10T01:56:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:56:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:56:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:57 smithi181 ceph-mon[94189]: pgmap v21590: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 613 KiB/s rd, 608 KiB/s wr, 12 op/s 2023-03-10T01:56:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:56:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:56:57.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:57 smithi038 ceph-mon[92422]: pgmap v21590: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 613 KiB/s rd, 608 KiB/s wr, 12 op/s 2023-03-10T01:56:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:56:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:56:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:58 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 39529.014490 secs 2023-03-10T01:56:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:58 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 39529.014490 secs 2023-03-10T01:56:58.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:58 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 39529.014490 secs 2023-03-10T01:56:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39990.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:56:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:59 smithi122 ceph-mon[93869]: pgmap v21591: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 515 KiB/s wr, 10 op/s 2023-03-10T01:56:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:59 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:56:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:56:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:56:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:56:59.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:59 smithi181 ceph-mon[94189]: pgmap v21591: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 515 KiB/s wr, 10 op/s 2023-03-10T01:56:59.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:59 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:56:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:56:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:56:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:56:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:59 smithi038 ceph-mon[92422]: pgmap v21591: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 515 KiB/s wr, 10 op/s 2023-03-10T01:56:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:59 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:56:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:56:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:56:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:57:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:57:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:57:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:57:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:57:00.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:57:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:57:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:01 smithi122 ceph-mon[93869]: pgmap v21592: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 673 KiB/s rd, 659 KiB/s wr, 10 op/s 2023-03-10T01:57:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39531.552445 secs 2023-03-10T01:57:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:57:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:57:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:01 smithi181 ceph-mon[94189]: pgmap v21592: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 673 KiB/s rd, 659 KiB/s wr, 10 op/s 2023-03-10T01:57:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39531.552445 secs 2023-03-10T01:57:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:57:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:57:01.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:01 smithi038 ceph-mon[92422]: pgmap v21592: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 673 KiB/s rd, 659 KiB/s wr, 10 op/s 2023-03-10T01:57:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39531.552445 secs 2023-03-10T01:57:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:57:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:57:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:03 smithi122 ceph-mon[93869]: pgmap v21593: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 705 KiB/s wr, 14 op/s 2023-03-10T01:57:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:03 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 39534.014607 secs 2023-03-10T01:57:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:03 smithi181 ceph-mon[94189]: pgmap v21593: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 705 KiB/s wr, 14 op/s 2023-03-10T01:57:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:03 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 39534.014607 secs 2023-03-10T01:57:03.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:03 smithi038 ceph-mon[92422]: pgmap v21593: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 705 KiB/s wr, 14 op/s 2023-03-10T01:57:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:03 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 39534.014607 secs 2023-03-10T01:57:03.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 39995.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:57:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:05 smithi122 ceph-mon[93869]: pgmap v21594: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 25 op/s 2023-03-10T01:57:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:05 smithi181 ceph-mon[94189]: pgmap v21594: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 25 op/s 2023-03-10T01:57:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:05 smithi038 ceph-mon[92422]: pgmap v21594: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 25 op/s 2023-03-10T01:57:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39536.552527 secs 2023-03-10T01:57:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39536.552527 secs 2023-03-10T01:57:06.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39536.552527 secs 2023-03-10T01:57:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:07 smithi122 ceph-mon[93869]: pgmap v21595: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 26 op/s 2023-03-10T01:57:07.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:07 smithi181 ceph-mon[94189]: pgmap v21595: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 26 op/s 2023-03-10T01:57:07.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:07 smithi038 ceph-mon[92422]: pgmap v21595: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 26 op/s 2023-03-10T01:57:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:57:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:57:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:08 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 39539.014712 secs 2023-03-10T01:57:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:57:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:57:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:08 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 39539.014712 secs 2023-03-10T01:57:08.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:57:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:57:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:08 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 39539.014712 secs 2023-03-10T01:57:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40000.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:57:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:09 smithi122 ceph-mon[93869]: pgmap v21596: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 24 op/s 2023-03-10T01:57:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:57:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:57:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:09 smithi181 ceph-mon[94189]: pgmap v21596: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 24 op/s 2023-03-10T01:57:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:57:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:57:09.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:09 smithi038 ceph-mon[92422]: pgmap v21596: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 24 op/s 2023-03-10T01:57:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:57:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:57:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:11 smithi122 ceph-mon[93869]: pgmap v21597: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 24 op/s 2023-03-10T01:57:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39541.552583 secs 2023-03-10T01:57:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:11 smithi181 ceph-mon[94189]: pgmap v21597: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 24 op/s 2023-03-10T01:57:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39541.552583 secs 2023-03-10T01:57:11.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:11 smithi038 ceph-mon[92422]: pgmap v21597: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 24 op/s 2023-03-10T01:57:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39541.552583 secs 2023-03-10T01:57:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:13 smithi122 ceph-mon[93869]: pgmap v21598: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 939 KiB/s rd, 950 KiB/s wr, 25 op/s 2023-03-10T01:57:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:13 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 39544.014820 secs 2023-03-10T01:57:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:13 smithi181 ceph-mon[94189]: pgmap v21598: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 939 KiB/s rd, 950 KiB/s wr, 25 op/s 2023-03-10T01:57:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:13 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 39544.014820 secs 2023-03-10T01:57:13.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:13 smithi038 ceph-mon[92422]: pgmap v21598: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 939 KiB/s rd, 950 KiB/s wr, 25 op/s 2023-03-10T01:57:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:13 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 39544.014820 secs 2023-03-10T01:57:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40005.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:57:15.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:15 smithi181 ceph-mon[94189]: pgmap v21599: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 24 op/s 2023-03-10T01:57:15.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:15 smithi038 ceph-mon[92422]: pgmap v21599: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 24 op/s 2023-03-10T01:57:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:15 smithi122 ceph-mon[93869]: pgmap v21599: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 24 op/s 2023-03-10T01:57:16.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39546.552667 secs 2023-03-10T01:57:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:57:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:57:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39546.552667 secs 2023-03-10T01:57:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:57:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:57:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39546.552667 secs 2023-03-10T01:57:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:57:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:57:17.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:17 smithi038 ceph-mon[92422]: pgmap v21600: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 366 KiB/s wr, 12 op/s 2023-03-10T01:57:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:57:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:57:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:17 smithi122 ceph-mon[93869]: pgmap v21600: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 366 KiB/s wr, 12 op/s 2023-03-10T01:57:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:57:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:57:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:17 smithi181 ceph-mon[94189]: pgmap v21600: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 366 KiB/s wr, 12 op/s 2023-03-10T01:57:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:57:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:57:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40010.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:57:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:18 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 39549.014878 secs 2023-03-10T01:57:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:18 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 39549.014878 secs 2023-03-10T01:57:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:18 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 39549.014878 secs 2023-03-10T01:57:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:19 smithi122 ceph-mon[93869]: pgmap v21601: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 365 KiB/s wr, 11 op/s 2023-03-10T01:57:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:19 smithi181 ceph-mon[94189]: pgmap v21601: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 365 KiB/s wr, 11 op/s 2023-03-10T01:57:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:19 smithi038 ceph-mon[92422]: pgmap v21601: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 365 KiB/s wr, 11 op/s 2023-03-10T01:57:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:57:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:57:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:57:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:57:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:57:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:57:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:21 smithi122 ceph-mon[93869]: pgmap v21602: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 596 KiB/s rd, 604 KiB/s wr, 11 op/s 2023-03-10T01:57:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39551.552749 secs 2023-03-10T01:57:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:57:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:57:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:21 smithi181 ceph-mon[94189]: pgmap v21602: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 596 KiB/s rd, 604 KiB/s wr, 11 op/s 2023-03-10T01:57:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39551.552749 secs 2023-03-10T01:57:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:57:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:57:21.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:21 smithi038 ceph-mon[92422]: pgmap v21602: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 596 KiB/s rd, 604 KiB/s wr, 11 op/s 2023-03-10T01:57:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39551.552749 secs 2023-03-10T01:57:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:57:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:57:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40015.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:57:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:23 smithi122 ceph-mon[93869]: pgmap v21603: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 788 KiB/s rd, 806 KiB/s wr, 15 op/s 2023-03-10T01:57:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39554.014937 secs 2023-03-10T01:57:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:23 smithi181 ceph-mon[94189]: pgmap v21603: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 788 KiB/s rd, 806 KiB/s wr, 15 op/s 2023-03-10T01:57:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39554.014937 secs 2023-03-10T01:57:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:23 smithi038 ceph-mon[92422]: pgmap v21603: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 788 KiB/s rd, 806 KiB/s wr, 15 op/s 2023-03-10T01:57:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39554.014937 secs 2023-03-10T01:57:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:25 smithi122 ceph-mon[93869]: pgmap v21604: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 641 KiB/s rd, 661 KiB/s wr, 13 op/s 2023-03-10T01:57:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:25 smithi122 ceph-mon[93869]: osdmap e1565: 12 total, 12 up, 12 in 2023-03-10T01:57:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:25 smithi181 ceph-mon[94189]: pgmap v21604: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 641 KiB/s rd, 661 KiB/s wr, 13 op/s 2023-03-10T01:57:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:25 smithi181 ceph-mon[94189]: osdmap e1565: 12 total, 12 up, 12 in 2023-03-10T01:57:25.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:25 smithi038 ceph-mon[92422]: pgmap v21604: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 641 KiB/s rd, 661 KiB/s wr, 13 op/s 2023-03-10T01:57:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:25 smithi038 ceph-mon[92422]: osdmap e1565: 12 total, 12 up, 12 in 2023-03-10T01:57:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39556.552808 secs 2023-03-10T01:57:26.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39556.552808 secs 2023-03-10T01:57:26.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39556.552808 secs 2023-03-10T01:57:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:27 smithi122 ceph-mon[93869]: pgmap v21606: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 690 KiB/s rd, 714 KiB/s wr, 15 op/s 2023-03-10T01:57:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:27 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:57:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:27 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:57:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:27 smithi181 ceph-mon[94189]: pgmap v21606: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 690 KiB/s rd, 714 KiB/s wr, 15 op/s 2023-03-10T01:57:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:27 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:57:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:27 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:57:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:27 smithi038 ceph-mon[92422]: pgmap v21606: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 690 KiB/s rd, 714 KiB/s wr, 15 op/s 2023-03-10T01:57:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:27 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:57:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:27 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:57:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40020.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:57:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39559.015021 secs 2023-03-10T01:57:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:28 smithi122 ceph-mon[93869]: pgmap v21607: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 889 KiB/s rd, 913 KiB/s wr, 14 op/s 2023-03-10T01:57:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39559.015021 secs 2023-03-10T01:57:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:28 smithi181 ceph-mon[94189]: pgmap v21607: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 889 KiB/s rd, 913 KiB/s wr, 14 op/s 2023-03-10T01:57:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39559.015021 secs 2023-03-10T01:57:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:28 smithi038 ceph-mon[92422]: pgmap v21607: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 889 KiB/s rd, 913 KiB/s wr, 14 op/s 2023-03-10T01:57:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:31 smithi122 ceph-mon[93869]: pgmap v21608: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 605 KiB/s rd, 617 KiB/s wr, 11 op/s 2023-03-10T01:57:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39561.552889 secs 2023-03-10T01:57:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:31 smithi181 ceph-mon[94189]: pgmap v21608: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 605 KiB/s rd, 617 KiB/s wr, 11 op/s 2023-03-10T01:57:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39561.552889 secs 2023-03-10T01:57:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:31 smithi038 ceph-mon[92422]: pgmap v21608: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 605 KiB/s rd, 617 KiB/s wr, 11 op/s 2023-03-10T01:57:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39561.552889 secs 2023-03-10T01:57:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:32 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:57:32.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:32 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:57:32.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:32 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:57:32.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:32 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:57:32.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:32 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:57:32.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:32 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:57:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:33 smithi122 ceph-mon[93869]: pgmap v21609: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 416 KiB/s wr, 10 op/s 2023-03-10T01:57:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:33 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:57:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:33 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:57:33.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:33 smithi181 ceph-mon[94189]: pgmap v21609: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 416 KiB/s wr, 10 op/s 2023-03-10T01:57:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:33 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:57:33.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:33 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:57:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:33 smithi038 ceph-mon[92422]: pgmap v21609: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 416 KiB/s wr, 10 op/s 2023-03-10T01:57:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:33 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:57:33.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:33 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:57:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40025.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:57:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:34 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39564.015079 secs 2023-03-10T01:57:34.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:34 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39564.015079 secs 2023-03-10T01:57:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:34 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39564.015079 secs 2023-03-10T01:57:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:35 smithi122 ceph-mon[93869]: pgmap v21610: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 697 KiB/s rd, 532 KiB/s wr, 10 op/s 2023-03-10T01:57:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:35 smithi122 ceph-mon[93869]: osdmap e1566: 12 total, 12 up, 12 in 2023-03-10T01:57:35.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:35 smithi181 ceph-mon[94189]: pgmap v21610: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 697 KiB/s rd, 532 KiB/s wr, 10 op/s 2023-03-10T01:57:35.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:35 smithi181 ceph-mon[94189]: osdmap e1566: 12 total, 12 up, 12 in 2023-03-10T01:57:35.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:35 smithi038 ceph-mon[92422]: pgmap v21610: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 697 KiB/s rd, 532 KiB/s wr, 10 op/s 2023-03-10T01:57:35.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:35 smithi038 ceph-mon[92422]: osdmap e1566: 12 total, 12 up, 12 in 2023-03-10T01:57:36.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39566.552985 secs 2023-03-10T01:57:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:57:36.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:57:36.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39566.552985 secs 2023-03-10T01:57:36.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:57:36.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:57:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39566.552985 secs 2023-03-10T01:57:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:57:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:57:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:37 smithi122 ceph-mon[93869]: pgmap v21612: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 921 KiB/s wr, 11 op/s 2023-03-10T01:57:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:57:37.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:57:37.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:37 smithi181 ceph-mon[94189]: pgmap v21612: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 921 KiB/s wr, 11 op/s 2023-03-10T01:57:37.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:57:37.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:57:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:37 smithi038 ceph-mon[92422]: pgmap v21612: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 921 KiB/s wr, 11 op/s 2023-03-10T01:57:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:57:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:57:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40030.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:57:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:39 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39569.015174 secs 2023-03-10T01:57:39.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:39 smithi122 ceph-mon[93869]: pgmap v21613: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 722 KiB/s wr, 11 op/s 2023-03-10T01:57:39.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:39 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39569.015174 secs 2023-03-10T01:57:39.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:39 smithi181 ceph-mon[94189]: pgmap v21613: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 722 KiB/s wr, 11 op/s 2023-03-10T01:57:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:39 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39569.015174 secs 2023-03-10T01:57:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:39 smithi038 ceph-mon[92422]: pgmap v21613: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 722 KiB/s wr, 11 op/s 2023-03-10T01:57:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:57:40.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:57:40.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:57:40.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:57:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:57:40.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:57:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:41 smithi122 ceph-mon[93869]: pgmap v21614: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 926 KiB/s wr, 11 op/s 2023-03-10T01:57:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39571.553072 secs 2023-03-10T01:57:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:57:41.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:57:41.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:41 smithi181 ceph-mon[94189]: pgmap v21614: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 926 KiB/s wr, 11 op/s 2023-03-10T01:57:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39571.553072 secs 2023-03-10T01:57:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:57:41.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:57:41.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:41 smithi038 ceph-mon[92422]: pgmap v21614: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 926 KiB/s wr, 11 op/s 2023-03-10T01:57:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39571.553072 secs 2023-03-10T01:57:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:57:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:57:43.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:43 smithi181 ceph-mon[94189]: pgmap v21615: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 886 KiB/s wr, 12 op/s 2023-03-10T01:57:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:43 smithi038 ceph-mon[92422]: pgmap v21615: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 886 KiB/s wr, 12 op/s 2023-03-10T01:57:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:43 smithi122 ceph-mon[93869]: pgmap v21615: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 886 KiB/s wr, 12 op/s 2023-03-10T01:57:43.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40035.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:57:44.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:44 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39574.015233 secs 2023-03-10T01:57:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:44 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39574.015233 secs 2023-03-10T01:57:44.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:44 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39574.015233 secs 2023-03-10T01:57:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:45 smithi038 ceph-mon[92422]: pgmap v21616: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 12 op/s 2023-03-10T01:57:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:45 smithi122 ceph-mon[93869]: pgmap v21616: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 12 op/s 2023-03-10T01:57:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:45 smithi181 ceph-mon[94189]: pgmap v21616: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 12 op/s 2023-03-10T01:57:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39576.553132 secs 2023-03-10T01:57:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39576.553132 secs 2023-03-10T01:57:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39576.553132 secs 2023-03-10T01:57:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:47 smithi038 ceph-mon[92422]: pgmap v21617: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 706 KiB/s rd, 642 KiB/s wr, 13 op/s 2023-03-10T01:57:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:47 smithi122 ceph-mon[93869]: pgmap v21617: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 706 KiB/s rd, 642 KiB/s wr, 13 op/s 2023-03-10T01:57:47.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:47 smithi181 ceph-mon[94189]: pgmap v21617: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 706 KiB/s rd, 642 KiB/s wr, 13 op/s 2023-03-10T01:57:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:57:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:57:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39579.015296 secs 2023-03-10T01:57:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:57:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:57:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39579.015296 secs 2023-03-10T01:57:48.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:57:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:57:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39579.015296 secs 2023-03-10T01:57:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40040.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:57:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:49 smithi122 ceph-mon[93869]: pgmap v21618: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 612 KiB/s rd, 556 KiB/s wr, 10 op/s 2023-03-10T01:57:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:57:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:57:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:49 smithi181 ceph-mon[94189]: pgmap v21618: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 612 KiB/s rd, 556 KiB/s wr, 10 op/s 2023-03-10T01:57:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:57:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:57:49.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:49 smithi038 ceph-mon[92422]: pgmap v21618: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 612 KiB/s rd, 556 KiB/s wr, 10 op/s 2023-03-10T01:57:49.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:57:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:57:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:51 smithi122 ceph-mon[93869]: pgmap v21619: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 747 KiB/s rd, 629 KiB/s wr, 10 op/s 2023-03-10T01:57:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39581.553214 secs 2023-03-10T01:57:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:51 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:57:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:51 smithi181 ceph-mon[94189]: pgmap v21619: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 747 KiB/s rd, 629 KiB/s wr, 10 op/s 2023-03-10T01:57:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39581.553214 secs 2023-03-10T01:57:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:51 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:57:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:51 smithi038 ceph-mon[92422]: pgmap v21619: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 747 KiB/s rd, 629 KiB/s wr, 10 op/s 2023-03-10T01:57:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39581.553214 secs 2023-03-10T01:57:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:51 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:57:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:57:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:57:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:57:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:57:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:57:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:57:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:57:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:57:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:57:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:57:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:57:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:57:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:57:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:57:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:57:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:57:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:57:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:57:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:57:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:57:52.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:57:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:53 smithi122 ceph-mon[93869]: pgmap v21620: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 920 KiB/s rd, 835 KiB/s wr, 13 op/s 2023-03-10T01:57:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39584.015385 secs 2023-03-10T01:57:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:53 smithi181 ceph-mon[94189]: pgmap v21620: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 920 KiB/s rd, 835 KiB/s wr, 13 op/s 2023-03-10T01:57:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39584.015385 secs 2023-03-10T01:57:53.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:53 smithi038 ceph-mon[92422]: pgmap v21620: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 920 KiB/s rd, 835 KiB/s wr, 13 op/s 2023-03-10T01:57:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39584.015385 secs 2023-03-10T01:57:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40045.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:57:55.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:55 smithi038 ceph-mon[92422]: pgmap v21621: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 835 KiB/s wr, 11 op/s 2023-03-10T01:57:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:55 smithi122 ceph-mon[93869]: pgmap v21621: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 835 KiB/s wr, 11 op/s 2023-03-10T01:57:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:55 smithi181 ceph-mon[94189]: pgmap v21621: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 835 KiB/s wr, 11 op/s 2023-03-10T01:57:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39586.553270 secs 2023-03-10T01:57:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:57:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:57:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39586.553270 secs 2023-03-10T01:57:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:57:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:57:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39586.553270 secs 2023-03-10T01:57:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:57:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:57:57.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:57 smithi038 ceph-mon[92422]: pgmap v21622: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 600 KiB/s wr, 13 op/s 2023-03-10T01:57:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:57:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:57:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:57 smithi122 ceph-mon[93869]: pgmap v21622: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 600 KiB/s wr, 13 op/s 2023-03-10T01:57:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:57:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:57:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:57 smithi181 ceph-mon[94189]: pgmap v21622: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 600 KiB/s wr, 13 op/s 2023-03-10T01:57:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:57:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:57:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39589.015488 secs 2023-03-10T01:57:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40050.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:57:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39589.015488 secs 2023-03-10T01:57:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39589.015488 secs 2023-03-10T01:57:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:57:59 smithi038 ceph-mon[92422]: pgmap v21623: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 484 KiB/s wr, 10 op/s 2023-03-10T01:57:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:57:59 smithi122 ceph-mon[93869]: pgmap v21623: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 484 KiB/s wr, 10 op/s 2023-03-10T01:57:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:57:59 smithi181 ceph-mon[94189]: pgmap v21623: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 484 KiB/s wr, 10 op/s 2023-03-10T01:58:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:58:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:58:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:58:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:58:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:58:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:58:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:01 smithi122 ceph-mon[93869]: pgmap v21624: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 825 KiB/s wr, 12 op/s 2023-03-10T01:58:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39591.553346 secs 2023-03-10T01:58:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:58:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:58:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:01 smithi181 ceph-mon[94189]: pgmap v21624: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 825 KiB/s wr, 12 op/s 2023-03-10T01:58:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39591.553346 secs 2023-03-10T01:58:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:58:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:58:01.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:01 smithi038 ceph-mon[92422]: pgmap v21624: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 825 KiB/s wr, 12 op/s 2023-03-10T01:58:02.076 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39591.553346 secs 2023-03-10T01:58:02.076 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:58:02.076 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:58:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40055.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:58:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:03 smithi122 ceph-mon[93869]: pgmap v21625: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 798 KiB/s wr, 15 op/s 2023-03-10T01:58:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39594.015556 secs 2023-03-10T01:58:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:03 smithi181 ceph-mon[94189]: pgmap v21625: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 798 KiB/s wr, 15 op/s 2023-03-10T01:58:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39594.015556 secs 2023-03-10T01:58:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:03 smithi038 ceph-mon[92422]: pgmap v21625: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 798 KiB/s wr, 15 op/s 2023-03-10T01:58:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39594.015556 secs 2023-03-10T01:58:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:05 smithi122 ceph-mon[93869]: pgmap v21626: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 825 KiB/s wr, 25 op/s 2023-03-10T01:58:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:05 smithi181 ceph-mon[94189]: pgmap v21626: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 825 KiB/s wr, 25 op/s 2023-03-10T01:58:05.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:05 smithi038 ceph-mon[92422]: pgmap v21626: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 825 KiB/s wr, 25 op/s 2023-03-10T01:58:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39596.553405 secs 2023-03-10T01:58:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39596.553405 secs 2023-03-10T01:58:06.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39596.553405 secs 2023-03-10T01:58:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:07 smithi122 ceph-mon[93869]: pgmap v21627: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 825 KiB/s wr, 27 op/s 2023-03-10T01:58:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:07 smithi181 ceph-mon[94189]: pgmap v21627: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 825 KiB/s wr, 27 op/s 2023-03-10T01:58:07.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:07 smithi038 ceph-mon[92422]: pgmap v21627: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 825 KiB/s wr, 27 op/s 2023-03-10T01:58:08.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40060.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:58:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:58:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:58:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39599.015657 secs 2023-03-10T01:58:08.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:08 smithi122 ceph-mon[93869]: pgmap v21628: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 969 KiB/s wr, 24 op/s 2023-03-10T01:58:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:58:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:58:08.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39599.015657 secs 2023-03-10T01:58:08.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:08 smithi181 ceph-mon[94189]: pgmap v21628: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 969 KiB/s wr, 24 op/s 2023-03-10T01:58:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:58:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:58:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39599.015657 secs 2023-03-10T01:58:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:08 smithi038 ceph-mon[92422]: pgmap v21628: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 969 KiB/s wr, 24 op/s 2023-03-10T01:58:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:58:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:58:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:58:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:58:09.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:58:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:58:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:11 smithi122 ceph-mon[93869]: pgmap v21629: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 969 KiB/s wr, 25 op/s 2023-03-10T01:58:11.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39601.553463 secs 2023-03-10T01:58:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:11 smithi181 ceph-mon[94189]: pgmap v21629: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 969 KiB/s wr, 25 op/s 2023-03-10T01:58:11.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39601.553463 secs 2023-03-10T01:58:11.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:11 smithi038 ceph-mon[92422]: pgmap v21629: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 969 KiB/s wr, 25 op/s 2023-03-10T01:58:11.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39601.553463 secs 2023-03-10T01:58:13.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:13 smithi122 ceph-mon[93869]: pgmap v21630: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 762 KiB/s rd, 702 KiB/s wr, 25 op/s 2023-03-10T01:58:13.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:13 smithi181 ceph-mon[94189]: pgmap v21630: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 762 KiB/s rd, 702 KiB/s wr, 25 op/s 2023-03-10T01:58:13.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:13 smithi038 ceph-mon[92422]: pgmap v21630: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 762 KiB/s rd, 702 KiB/s wr, 25 op/s 2023-03-10T01:58:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40065.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:58:14.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:14 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39604.015716 secs 2023-03-10T01:58:14.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:14 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39604.015716 secs 2023-03-10T01:58:14.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:14 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39604.015716 secs 2023-03-10T01:58:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:15 smithi122 ceph-mon[93869]: pgmap v21631: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 843 KiB/s rd, 774 KiB/s wr, 21 op/s 2023-03-10T01:58:15.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:15 smithi181 ceph-mon[94189]: pgmap v21631: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 843 KiB/s rd, 774 KiB/s wr, 21 op/s 2023-03-10T01:58:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:15 smithi038 ceph-mon[92422]: pgmap v21631: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 843 KiB/s rd, 774 KiB/s wr, 21 op/s 2023-03-10T01:58:16.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39606.553548 secs 2023-03-10T01:58:16.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39606.553548 secs 2023-03-10T01:58:16.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39606.553548 secs 2023-03-10T01:58:17.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:17 smithi122 ceph-mon[93869]: pgmap v21632: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 371 KiB/s wr, 10 op/s 2023-03-10T01:58:17.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:17 smithi181 ceph-mon[94189]: pgmap v21632: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 371 KiB/s wr, 10 op/s 2023-03-10T01:58:17.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:17 smithi038 ceph-mon[92422]: pgmap v21632: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 371 KiB/s wr, 10 op/s 2023-03-10T01:58:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40070.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:58:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:19 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39609.015802 secs 2023-03-10T01:58:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:19 smithi122 ceph-mon[93869]: pgmap v21633: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 371 KiB/s wr, 7 op/s 2023-03-10T01:58:19.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:19 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39609.015802 secs 2023-03-10T01:58:19.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:19 smithi181 ceph-mon[94189]: pgmap v21633: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 371 KiB/s wr, 7 op/s 2023-03-10T01:58:19.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:19 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39609.015802 secs 2023-03-10T01:58:19.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:19 smithi038 ceph-mon[92422]: pgmap v21633: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 369 KiB/s rd, 371 KiB/s wr, 7 op/s 2023-03-10T01:58:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:58:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:58:20.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:58:20.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:58:20.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:58:20.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:58:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:21 smithi122 ceph-mon[93869]: pgmap v21634: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 203 KiB/s wr, 8 op/s 2023-03-10T01:58:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39611.553645 secs 2023-03-10T01:58:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:58:21.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:58:21.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:21 smithi181 ceph-mon[94189]: pgmap v21634: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 203 KiB/s wr, 8 op/s 2023-03-10T01:58:21.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39611.553645 secs 2023-03-10T01:58:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:58:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:58:21.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:21 smithi038 ceph-mon[92422]: pgmap v21634: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 203 KiB/s wr, 8 op/s 2023-03-10T01:58:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39611.553645 secs 2023-03-10T01:58:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:58:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:58:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:23 smithi122 ceph-mon[93869]: pgmap v21635: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 497 KiB/s rd, 516 KiB/s wr, 12 op/s 2023-03-10T01:58:23.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:23 smithi181 ceph-mon[94189]: pgmap v21635: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 497 KiB/s rd, 516 KiB/s wr, 12 op/s 2023-03-10T01:58:23.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:23 smithi038 ceph-mon[92422]: pgmap v21635: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 497 KiB/s rd, 516 KiB/s wr, 12 op/s 2023-03-10T01:58:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40075.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:58:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:25 smithi122 ceph-mon[93869]: pgmap v21636: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 527 KiB/s rd, 551 KiB/s wr, 11 op/s 2023-03-10T01:58:25.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:25 smithi181 ceph-mon[94189]: pgmap v21636: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 527 KiB/s rd, 551 KiB/s wr, 11 op/s 2023-03-10T01:58:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:25 smithi038 ceph-mon[92422]: pgmap v21636: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 527 KiB/s rd, 551 KiB/s wr, 11 op/s 2023-03-10T01:58:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39616.553703 secs 2023-03-10T01:58:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:26 smithi038 ceph-mon[92422]: osdmap e1567: 12 total, 12 up, 12 in 2023-03-10T01:58:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:26 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:58:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39616.553703 secs 2023-03-10T01:58:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:26 smithi122 ceph-mon[93869]: osdmap e1567: 12 total, 12 up, 12 in 2023-03-10T01:58:26.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:26 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:58:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39616.553703 secs 2023-03-10T01:58:26.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:26 smithi181 ceph-mon[94189]: osdmap e1567: 12 total, 12 up, 12 in 2023-03-10T01:58:26.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:26 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:58:27.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:27 smithi038 ceph-mon[92422]: pgmap v21638: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 521 KiB/s wr, 12 op/s 2023-03-10T01:58:27.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:27 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:58:27.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:27 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:58:27.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:27 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:58:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:27 smithi122 ceph-mon[93869]: pgmap v21638: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 521 KiB/s wr, 12 op/s 2023-03-10T01:58:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:27 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:58:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:27 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:58:27.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:27 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:58:27.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:27 smithi181 ceph-mon[94189]: pgmap v21638: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 521 KiB/s wr, 12 op/s 2023-03-10T01:58:27.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:27 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:58:27.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:27 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:58:27.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:27 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:58:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40080.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:58:29.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:29 smithi038 ceph-mon[92422]: pgmap v21639: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 692 KiB/s rd, 721 KiB/s wr, 13 op/s 2023-03-10T01:58:29.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:29 smithi122 ceph-mon[93869]: pgmap v21639: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 692 KiB/s rd, 721 KiB/s wr, 13 op/s 2023-03-10T01:58:29.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:29 smithi181 ceph-mon[94189]: pgmap v21639: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 692 KiB/s rd, 721 KiB/s wr, 13 op/s 2023-03-10T01:58:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:31 smithi038 ceph-mon[92422]: pgmap v21640: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 691 KiB/s rd, 708 KiB/s wr, 11 op/s 2023-03-10T01:58:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39621.553759 secs 2023-03-10T01:58:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:31 smithi122 ceph-mon[93869]: pgmap v21640: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 691 KiB/s rd, 708 KiB/s wr, 11 op/s 2023-03-10T01:58:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39621.553759 secs 2023-03-10T01:58:31.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:31 smithi181 ceph-mon[94189]: pgmap v21640: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 691 KiB/s rd, 708 KiB/s wr, 11 op/s 2023-03-10T01:58:31.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39621.553759 secs 2023-03-10T01:58:32.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:32 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:58:32.837 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:32 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:58:32.838 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:32 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:58:32.838 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:32 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:58:32.839 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:32 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:58:32.839 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:32 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:58:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:33 smithi122 ceph-mon[93869]: pgmap v21641: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 375 KiB/s wr, 9 op/s 2023-03-10T01:58:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:33 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:58:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:33 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:58:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:33 smithi181 ceph-mon[94189]: pgmap v21641: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 375 KiB/s wr, 9 op/s 2023-03-10T01:58:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:33 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:58:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:33 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:58:33.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:33 smithi038 ceph-mon[92422]: pgmap v21641: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 375 KiB/s wr, 9 op/s 2023-03-10T01:58:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:33 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:58:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:33 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:58:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40085.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:58:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:35 smithi122 ceph-mon[93869]: pgmap v21642: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 493 KiB/s rd, 325 KiB/s wr, 9 op/s 2023-03-10T01:58:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:35 smithi122 ceph-mon[93869]: osdmap e1568: 12 total, 12 up, 12 in 2023-03-10T01:58:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:35 smithi181 ceph-mon[94189]: pgmap v21642: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 493 KiB/s rd, 325 KiB/s wr, 9 op/s 2023-03-10T01:58:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:35 smithi181 ceph-mon[94189]: osdmap e1568: 12 total, 12 up, 12 in 2023-03-10T01:58:35.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:35 smithi038 ceph-mon[92422]: pgmap v21642: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 493 KiB/s rd, 325 KiB/s wr, 9 op/s 2023-03-10T01:58:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:35 smithi038 ceph-mon[92422]: osdmap e1568: 12 total, 12 up, 12 in 2023-03-10T01:58:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39626.553816 secs 2023-03-10T01:58:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:58:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:58:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39626.553816 secs 2023-03-10T01:58:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:58:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:58:36.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39626.553816 secs 2023-03-10T01:58:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:58:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:58:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:37 smithi122 ceph-mon[93869]: pgmap v21644: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 533 KiB/s rd, 366 KiB/s wr, 9 op/s 2023-03-10T01:58:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:58:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:58:37.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:37 smithi181 ceph-mon[94189]: pgmap v21644: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 533 KiB/s rd, 366 KiB/s wr, 9 op/s 2023-03-10T01:58:37.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:58:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:58:37.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:37 smithi038 ceph-mon[92422]: pgmap v21644: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 533 KiB/s rd, 366 KiB/s wr, 9 op/s 2023-03-10T01:58:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:58:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:58:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39629.016114 secs 2023-03-10T01:58:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39629.016114 secs 2023-03-10T01:58:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39629.016114 secs 2023-03-10T01:58:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40090.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:58:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:39 smithi122 ceph-mon[93869]: pgmap v21645: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 725 KiB/s wr, 10 op/s 2023-03-10T01:58:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:39 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:58:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:39 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:58:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:39 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:58:39.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:39 smithi181 ceph-mon[94189]: pgmap v21645: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 725 KiB/s wr, 10 op/s 2023-03-10T01:58:39.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:39 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:58:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:39 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:58:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:39 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:58:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:39 smithi038 ceph-mon[92422]: pgmap v21645: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 725 KiB/s wr, 10 op/s 2023-03-10T01:58:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:39 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:58:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:39 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:58:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:39 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:58:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:58:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:58:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:58:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:58:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:58:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:58:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:41 smithi122 ceph-mon[93869]: pgmap v21646: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 725 KiB/s wr, 11 op/s 2023-03-10T01:58:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39631.553872 secs 2023-03-10T01:58:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:58:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:58:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:41 smithi181 ceph-mon[94189]: pgmap v21646: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 725 KiB/s wr, 11 op/s 2023-03-10T01:58:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39631.553872 secs 2023-03-10T01:58:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:58:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:58:41.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:41 smithi038 ceph-mon[92422]: pgmap v21646: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 725 KiB/s wr, 11 op/s 2023-03-10T01:58:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39631.553872 secs 2023-03-10T01:58:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:58:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:58:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:43 smithi122 ceph-mon[93869]: pgmap v21647: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 827 KiB/s wr, 11 op/s 2023-03-10T01:58:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39634.016219 secs 2023-03-10T01:58:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:43 smithi181 ceph-mon[94189]: pgmap v21647: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 827 KiB/s wr, 11 op/s 2023-03-10T01:58:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39634.016219 secs 2023-03-10T01:58:43.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:43 smithi038 ceph-mon[92422]: pgmap v21647: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 827 KiB/s wr, 11 op/s 2023-03-10T01:58:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39634.016219 secs 2023-03-10T01:58:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40095.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:58:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:45 smithi122 ceph-mon[93869]: pgmap v21648: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1010 KiB/s wr, 11 op/s 2023-03-10T01:58:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:45 smithi181 ceph-mon[94189]: pgmap v21648: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1010 KiB/s wr, 11 op/s 2023-03-10T01:58:45.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:45 smithi038 ceph-mon[92422]: pgmap v21648: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1010 KiB/s wr, 11 op/s 2023-03-10T01:58:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39636.553957 secs 2023-03-10T01:58:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39636.553957 secs 2023-03-10T01:58:46.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39636.553957 secs 2023-03-10T01:58:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:47 smithi122 ceph-mon[93869]: pgmap v21649: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 13 op/s 2023-03-10T01:58:47.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:47 smithi181 ceph-mon[94189]: pgmap v21649: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 13 op/s 2023-03-10T01:58:47.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:47 smithi038 ceph-mon[92422]: pgmap v21649: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 13 op/s 2023-03-10T01:58:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:58:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:58:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:58:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:58:48.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:58:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:58:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40100.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:58:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:49 smithi122 ceph-mon[93869]: pgmap v21650: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 10 op/s 2023-03-10T01:58:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:58:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:58:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:49 smithi181 ceph-mon[94189]: pgmap v21650: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 10 op/s 2023-03-10T01:58:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:58:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:58:49.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:49 smithi038 ceph-mon[92422]: pgmap v21650: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 10 op/s 2023-03-10T01:58:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:58:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:58:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:50 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:58:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:50 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:58:50.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:50 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T01:58:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:51 smithi122 ceph-mon[93869]: pgmap v21651: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 577 KiB/s wr, 8 op/s 2023-03-10T01:58:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39641.554014 secs 2023-03-10T01:58:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:51 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:58:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:58:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:58:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:51 smithi181 ceph-mon[94189]: pgmap v21651: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 577 KiB/s wr, 8 op/s 2023-03-10T01:58:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39641.554014 secs 2023-03-10T01:58:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:51 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:58:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:58:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:58:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:51 smithi038 ceph-mon[92422]: pgmap v21651: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 577 KiB/s wr, 8 op/s 2023-03-10T01:58:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39641.554014 secs 2023-03-10T01:58:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:51 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:58:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:58:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:58:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:58:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:58:52.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:52 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:58:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:58:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:58:52.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:52 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:58:52.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:58:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:58:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:52 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:58:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:53 smithi122 ceph-mon[93869]: pgmap v21652: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 605 KiB/s rd, 640 KiB/s wr, 12 op/s 2023-03-10T01:58:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:58:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:58:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:58:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:53 smithi181 ceph-mon[94189]: pgmap v21652: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 605 KiB/s rd, 640 KiB/s wr, 12 op/s 2023-03-10T01:58:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:58:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:58:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:58:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:53 smithi038 ceph-mon[92422]: pgmap v21652: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 605 KiB/s rd, 640 KiB/s wr, 12 op/s 2023-03-10T01:58:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:58:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:58:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:58:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40105.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:58:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:55 smithi122 ceph-mon[93869]: pgmap v21653: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 812 KiB/s rd, 848 KiB/s wr, 10 op/s 2023-03-10T01:58:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:55 smithi181 ceph-mon[94189]: pgmap v21653: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 812 KiB/s rd, 848 KiB/s wr, 10 op/s 2023-03-10T01:58:55.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:55 smithi038 ceph-mon[92422]: pgmap v21653: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 812 KiB/s rd, 848 KiB/s wr, 10 op/s 2023-03-10T01:58:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39646.554102 secs 2023-03-10T01:58:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:58:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:58:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39646.554102 secs 2023-03-10T01:58:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:58:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:58:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39646.554102 secs 2023-03-10T01:58:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:58:56.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:58:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:57 smithi122 ceph-mon[93869]: pgmap v21654: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 740 KiB/s rd, 735 KiB/s wr, 12 op/s 2023-03-10T01:58:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:58:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:58:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:57 smithi181 ceph-mon[94189]: pgmap v21654: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 740 KiB/s rd, 735 KiB/s wr, 12 op/s 2023-03-10T01:58:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:58:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:58:57.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:57 smithi038 ceph-mon[92422]: pgmap v21654: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 740 KiB/s rd, 735 KiB/s wr, 12 op/s 2023-03-10T01:58:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:58:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:58:58.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40110.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:58:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:58:59 smithi122 ceph-mon[93869]: pgmap v21655: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 517 KiB/s rd, 513 KiB/s wr, 10 op/s 2023-03-10T01:58:59.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:58:59 smithi181 ceph-mon[94189]: pgmap v21655: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 517 KiB/s rd, 513 KiB/s wr, 10 op/s 2023-03-10T01:58:59.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:58:59 smithi038 ceph-mon[92422]: pgmap v21655: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 517 KiB/s rd, 513 KiB/s wr, 10 op/s 2023-03-10T01:59:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:01 smithi122 ceph-mon[93869]: pgmap v21656: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 676 KiB/s wr, 10 op/s 2023-03-10T01:59:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39651.554163 secs 2023-03-10T01:59:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:59:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:59:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:01 smithi181 ceph-mon[94189]: pgmap v21656: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 676 KiB/s wr, 10 op/s 2023-03-10T01:59:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39651.554163 secs 2023-03-10T01:59:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:59:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:59:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:01 smithi038 ceph-mon[92422]: pgmap v21656: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 676 KiB/s wr, 10 op/s 2023-03-10T01:59:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39651.554163 secs 2023-03-10T01:59:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:59:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:59:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:03 smithi122 ceph-mon[93869]: pgmap v21657: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 823 KiB/s rd, 980 KiB/s wr, 15 op/s 2023-03-10T01:59:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:03 smithi181 ceph-mon[94189]: pgmap v21657: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 823 KiB/s rd, 980 KiB/s wr, 15 op/s 2023-03-10T01:59:03.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:03 smithi038 ceph-mon[92422]: pgmap v21657: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 823 KiB/s rd, 980 KiB/s wr, 15 op/s 2023-03-10T01:59:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40115.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:59:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:05 smithi122 ceph-mon[93869]: pgmap v21658: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 23 op/s 2023-03-10T01:59:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:05 smithi181 ceph-mon[94189]: pgmap v21658: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 23 op/s 2023-03-10T01:59:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:05 smithi038 ceph-mon[92422]: pgmap v21658: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 23 op/s 2023-03-10T01:59:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39656.554247 secs 2023-03-10T01:59:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39656.554247 secs 2023-03-10T01:59:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39656.554247 secs 2023-03-10T01:59:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:07 smithi122 ceph-mon[93869]: pgmap v21659: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 26 op/s 2023-03-10T01:59:07.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:07 smithi181 ceph-mon[94189]: pgmap v21659: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 26 op/s 2023-03-10T01:59:07.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:07 smithi038 ceph-mon[92422]: pgmap v21659: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 26 op/s 2023-03-10T01:59:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:59:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:59:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:59:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:59:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:59:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:59:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40120.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:59:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:09 smithi122 ceph-mon[93869]: pgmap v21660: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 936 KiB/s rd, 937 KiB/s wr, 22 op/s 2023-03-10T01:59:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:59:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:59:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:09 smithi181 ceph-mon[94189]: pgmap v21660: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 936 KiB/s rd, 937 KiB/s wr, 22 op/s 2023-03-10T01:59:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:59:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:59:09.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:09 smithi038 ceph-mon[92422]: pgmap v21660: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 936 KiB/s rd, 937 KiB/s wr, 22 op/s 2023-03-10T01:59:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:59:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:59:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:11 smithi122 ceph-mon[93869]: pgmap v21661: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 936 KiB/s rd, 937 KiB/s wr, 23 op/s 2023-03-10T01:59:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39661.554317 secs 2023-03-10T01:59:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:11 smithi181 ceph-mon[94189]: pgmap v21661: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 936 KiB/s rd, 937 KiB/s wr, 23 op/s 2023-03-10T01:59:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39661.554317 secs 2023-03-10T01:59:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:11 smithi038 ceph-mon[92422]: pgmap v21661: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 936 KiB/s rd, 937 KiB/s wr, 23 op/s 2023-03-10T01:59:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39661.554317 secs 2023-03-10T01:59:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:13 smithi181 ceph-mon[94189]: pgmap v21662: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 26 op/s 2023-03-10T01:59:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:13 smithi038 ceph-mon[92422]: pgmap v21662: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 26 op/s 2023-03-10T01:59:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40125.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:59:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:13 smithi122 ceph-mon[93869]: pgmap v21662: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 26 op/s 2023-03-10T01:59:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:15 smithi038 ceph-mon[92422]: pgmap v21663: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 946 KiB/s rd, 786 KiB/s wr, 21 op/s 2023-03-10T01:59:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:15 smithi122 ceph-mon[93869]: pgmap v21663: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 946 KiB/s rd, 786 KiB/s wr, 21 op/s 2023-03-10T01:59:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:15 smithi181 ceph-mon[94189]: pgmap v21663: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 946 KiB/s rd, 786 KiB/s wr, 21 op/s 2023-03-10T01:59:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39666.554401 secs 2023-03-10T01:59:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:59:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:59:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39666.554401 secs 2023-03-10T01:59:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:59:16.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:59:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39666.554401 secs 2023-03-10T01:59:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:59:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:59:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:17 smithi122 ceph-mon[93869]: pgmap v21664: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 596 KiB/s wr, 12 op/s 2023-03-10T01:59:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:59:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:59:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:17 smithi181 ceph-mon[94189]: pgmap v21664: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 596 KiB/s wr, 12 op/s 2023-03-10T01:59:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:59:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:59:17.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:17 smithi038 ceph-mon[92422]: pgmap v21664: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 596 KiB/s wr, 12 op/s 2023-03-10T01:59:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:59:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:59:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40130.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:59:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:19 smithi122 ceph-mon[93869]: pgmap v21665: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 329 KiB/s wr, 8 op/s 2023-03-10T01:59:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:19 smithi181 ceph-mon[94189]: pgmap v21665: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 329 KiB/s wr, 8 op/s 2023-03-10T01:59:19.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:19 smithi038 ceph-mon[92422]: pgmap v21665: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 329 KiB/s wr, 8 op/s 2023-03-10T01:59:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:59:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:59:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:20 smithi122 ceph-mon[93869]: osdmap e1569: 12 total, 12 up, 12 in 2023-03-10T01:59:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:59:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:59:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:20 smithi181 ceph-mon[94189]: osdmap e1569: 12 total, 12 up, 12 in 2023-03-10T01:59:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:59:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:59:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:20 smithi038 ceph-mon[92422]: osdmap e1569: 12 total, 12 up, 12 in 2023-03-10T01:59:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:21 smithi122 ceph-mon[93869]: pgmap v21667: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 398 KiB/s rd, 407 KiB/s wr, 10 op/s 2023-03-10T01:59:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39671.554469 secs 2023-03-10T01:59:21.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:59:21.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:59:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:21 smithi181 ceph-mon[94189]: pgmap v21667: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 398 KiB/s rd, 407 KiB/s wr, 10 op/s 2023-03-10T01:59:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39671.554469 secs 2023-03-10T01:59:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:59:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:59:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:21 smithi038 ceph-mon[92422]: pgmap v21667: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 398 KiB/s rd, 407 KiB/s wr, 10 op/s 2023-03-10T01:59:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39671.554469 secs 2023-03-10T01:59:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:59:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:59:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40135.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:59:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:23 smithi122 ceph-mon[93869]: pgmap v21668: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 522 KiB/s rd, 543 KiB/s wr, 12 op/s 2023-03-10T01:59:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:23 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 39674.016998 secs 2023-03-10T01:59:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:23 smithi181 ceph-mon[94189]: pgmap v21668: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 522 KiB/s rd, 543 KiB/s wr, 12 op/s 2023-03-10T01:59:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:23 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 39674.016998 secs 2023-03-10T01:59:23.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:23 smithi038 ceph-mon[92422]: pgmap v21668: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 522 KiB/s rd, 543 KiB/s wr, 12 op/s 2023-03-10T01:59:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:23 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 39674.016998 secs 2023-03-10T01:59:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:25 smithi122 ceph-mon[93869]: pgmap v21669: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 522 KiB/s rd, 543 KiB/s wr, 12 op/s 2023-03-10T01:59:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:25 smithi181 ceph-mon[94189]: pgmap v21669: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 522 KiB/s rd, 543 KiB/s wr, 12 op/s 2023-03-10T01:59:25.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:25 smithi038 ceph-mon[92422]: pgmap v21669: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 522 KiB/s rd, 543 KiB/s wr, 12 op/s 2023-03-10T01:59:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39676.554548 secs 2023-03-10T01:59:26.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39676.554548 secs 2023-03-10T01:59:26.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39676.554548 secs 2023-03-10T01:59:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:27 smithi122 ceph-mon[93869]: pgmap v21670: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 731 KiB/s rd, 753 KiB/s wr, 14 op/s 2023-03-10T01:59:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:27 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:59:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:27 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:59:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:27 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:59:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:27 smithi181 ceph-mon[94189]: pgmap v21670: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 731 KiB/s rd, 753 KiB/s wr, 14 op/s 2023-03-10T01:59:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:27 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:59:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:27 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:59:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:27 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:59:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:27 smithi038 ceph-mon[92422]: pgmap v21670: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 731 KiB/s rd, 753 KiB/s wr, 14 op/s 2023-03-10T01:59:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:27 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T01:59:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:27 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:59:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:27 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:59:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40140.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:59:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:28 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:59:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:28 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:59:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:28 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 39679.017145 secs 2023-03-10T01:59:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:28 smithi122 ceph-mon[93869]: pgmap v21671: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 731 KiB/s rd, 753 KiB/s wr, 14 op/s 2023-03-10T01:59:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:28 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:59:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:28 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:59:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:28 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 39679.017145 secs 2023-03-10T01:59:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:28 smithi181 ceph-mon[94189]: pgmap v21671: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 731 KiB/s rd, 753 KiB/s wr, 14 op/s 2023-03-10T01:59:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:28 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:59:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:28 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:59:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:28 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 39679.017145 secs 2023-03-10T01:59:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:28 smithi038 ceph-mon[92422]: pgmap v21671: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 731 KiB/s rd, 753 KiB/s wr, 14 op/s 2023-03-10T01:59:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:31 smithi122 ceph-mon[93869]: pgmap v21672: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 904 KiB/s rd, 915 KiB/s wr, 13 op/s 2023-03-10T01:59:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39681.554605 secs 2023-03-10T01:59:31.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:31 smithi122 ceph-mon[93869]: osdmap e1570: 12 total, 12 up, 12 in 2023-03-10T01:59:31.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:31 smithi181 ceph-mon[94189]: pgmap v21672: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 904 KiB/s rd, 915 KiB/s wr, 13 op/s 2023-03-10T01:59:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39681.554605 secs 2023-03-10T01:59:31.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:31 smithi181 ceph-mon[94189]: osdmap e1570: 12 total, 12 up, 12 in 2023-03-10T01:59:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:31 smithi038 ceph-mon[92422]: pgmap v21672: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 904 KiB/s rd, 915 KiB/s wr, 13 op/s 2023-03-10T01:59:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39681.554605 secs 2023-03-10T01:59:31.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:31 smithi038 ceph-mon[92422]: osdmap e1570: 12 total, 12 up, 12 in 2023-03-10T01:59:33.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:33 smithi122 ceph-mon[93869]: pgmap v21674: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 448 KiB/s wr, 10 op/s 2023-03-10T01:59:33.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:33 smithi181 ceph-mon[94189]: pgmap v21674: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 448 KiB/s wr, 10 op/s 2023-03-10T01:59:33.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:33 smithi038 ceph-mon[92422]: pgmap v21674: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 448 KiB/s wr, 10 op/s 2023-03-10T01:59:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40145.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:59:34.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:34 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 39684.017229 secs 2023-03-10T01:59:34.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:34 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 39684.017229 secs 2023-03-10T01:59:34.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:34 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 39684.017229 secs 2023-03-10T01:59:35.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:35 smithi122 ceph-mon[93869]: pgmap v21675: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 448 KiB/s wr, 9 op/s 2023-03-10T01:59:35.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:35 smithi181 ceph-mon[94189]: pgmap v21675: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 448 KiB/s wr, 9 op/s 2023-03-10T01:59:35.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:35 smithi038 ceph-mon[92422]: pgmap v21675: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 449 KiB/s rd, 448 KiB/s wr, 9 op/s 2023-03-10T01:59:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39686.554664 secs 2023-03-10T01:59:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:59:36.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:59:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39686.554664 secs 2023-03-10T01:59:36.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:59:36.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:59:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39686.554664 secs 2023-03-10T01:59:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:59:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:59:37.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:37 smithi038 ceph-mon[92422]: pgmap v21676: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 550 KiB/s rd, 548 KiB/s wr, 9 op/s 2023-03-10T01:59:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:59:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:59:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:37 smithi122 ceph-mon[93869]: pgmap v21676: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 550 KiB/s rd, 548 KiB/s wr, 9 op/s 2023-03-10T01:59:37.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:59:37.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:59:37.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:37 smithi181 ceph-mon[94189]: pgmap v21676: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 550 KiB/s rd, 548 KiB/s wr, 9 op/s 2023-03-10T01:59:37.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:59:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:59:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40150.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:59:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:39 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39689.017292 secs 2023-03-10T01:59:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:39 smithi038 ceph-mon[92422]: pgmap v21677: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 550 KiB/s rd, 554 KiB/s wr, 9 op/s 2023-03-10T01:59:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:39 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:59:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:39 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:59:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:39 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39689.017292 secs 2023-03-10T01:59:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:39 smithi122 ceph-mon[93869]: pgmap v21677: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 550 KiB/s rd, 554 KiB/s wr, 9 op/s 2023-03-10T01:59:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:39 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:59:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:39 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:59:39.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:39 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39689.017292 secs 2023-03-10T01:59:39.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:39 smithi181 ceph-mon[94189]: pgmap v21677: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 550 KiB/s rd, 554 KiB/s wr, 9 op/s 2023-03-10T01:59:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:39 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:59:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:39 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:59:41.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:41 smithi038 ceph-mon[92422]: pgmap v21678: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 353 KiB/s wr, 10 op/s 2023-03-10T01:59:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39691.554719 secs 2023-03-10T01:59:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:59:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:59:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:41 smithi122 ceph-mon[93869]: pgmap v21678: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 353 KiB/s wr, 10 op/s 2023-03-10T01:59:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39691.554719 secs 2023-03-10T01:59:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:59:41.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:59:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:41 smithi181 ceph-mon[94189]: pgmap v21678: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 353 KiB/s wr, 10 op/s 2023-03-10T01:59:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39691.554719 secs 2023-03-10T01:59:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:59:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:59:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:43 smithi122 ceph-mon[93869]: pgmap v21679: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 700 KiB/s wr, 12 op/s 2023-03-10T01:59:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39694.017356 secs 2023-03-10T01:59:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:43 smithi181 ceph-mon[94189]: pgmap v21679: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 700 KiB/s wr, 12 op/s 2023-03-10T01:59:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39694.017356 secs 2023-03-10T01:59:43.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:43 smithi038 ceph-mon[92422]: pgmap v21679: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 700 KiB/s wr, 12 op/s 2023-03-10T01:59:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39694.017356 secs 2023-03-10T01:59:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40155.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:59:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:45 smithi122 ceph-mon[93869]: pgmap v21680: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 674 KiB/s rd, 678 KiB/s wr, 10 op/s 2023-03-10T01:59:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:45 smithi181 ceph-mon[94189]: pgmap v21680: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 674 KiB/s rd, 678 KiB/s wr, 10 op/s 2023-03-10T01:59:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:45 smithi038 ceph-mon[92422]: pgmap v21680: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 674 KiB/s rd, 678 KiB/s wr, 10 op/s 2023-03-10T01:59:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39696.554778 secs 2023-03-10T01:59:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39696.554778 secs 2023-03-10T01:59:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39696.554778 secs 2023-03-10T01:59:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:47 smithi122 ceph-mon[93869]: pgmap v21681: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 726 KiB/s rd, 728 KiB/s wr, 13 op/s 2023-03-10T01:59:47.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:47 smithi181 ceph-mon[94189]: pgmap v21681: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 726 KiB/s rd, 728 KiB/s wr, 13 op/s 2023-03-10T01:59:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:47 smithi038 ceph-mon[92422]: pgmap v21681: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 726 KiB/s rd, 728 KiB/s wr, 13 op/s 2023-03-10T01:59:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:59:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:59:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:48 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 39699.017417 secs 2023-03-10T01:59:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:59:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:59:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:48 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 39699.017417 secs 2023-03-10T01:59:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:59:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:59:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:48 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 39699.017417 secs 2023-03-10T01:59:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40160.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:59:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:49 smithi122 ceph-mon[93869]: pgmap v21682: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 962 KiB/s rd, 810 KiB/s wr, 10 op/s 2023-03-10T01:59:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:59:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:59:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:49 smithi181 ceph-mon[94189]: pgmap v21682: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 962 KiB/s rd, 810 KiB/s wr, 10 op/s 2023-03-10T01:59:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:59:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:59:49.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:49 smithi038 ceph-mon[92422]: pgmap v21682: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 962 KiB/s rd, 810 KiB/s wr, 10 op/s 2023-03-10T01:59:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:59:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:59:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:51 smithi122 ceph-mon[93869]: pgmap v21683: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 990 KiB/s rd, 833 KiB/s wr, 10 op/s 2023-03-10T01:59:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39701.554835 secs 2023-03-10T01:59:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:59:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:59:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:51 smithi181 ceph-mon[94189]: pgmap v21683: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 990 KiB/s rd, 833 KiB/s wr, 10 op/s 2023-03-10T01:59:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39701.554835 secs 2023-03-10T01:59:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:59:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:59:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:51 smithi038 ceph-mon[92422]: pgmap v21683: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 990 KiB/s rd, 833 KiB/s wr, 10 op/s 2023-03-10T01:59:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39701.554835 secs 2023-03-10T01:59:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:59:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:59:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:53 smithi122 ceph-mon[93869]: pgmap v21684: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.2 MiB/s wr, 14 op/s 2023-03-10T01:59:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:59:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:59:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39704.017506 secs 2023-03-10T01:59:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:59:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:53 smithi181 ceph-mon[94189]: pgmap v21684: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.2 MiB/s wr, 14 op/s 2023-03-10T01:59:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:59:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:59:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39704.017506 secs 2023-03-10T01:59:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:59:53.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:53 smithi038 ceph-mon[92422]: pgmap v21684: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.2 MiB/s wr, 14 op/s 2023-03-10T01:59:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:59:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:59:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39704.017506 secs 2023-03-10T01:59:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T01:59:53.801 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40165.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:59:55.414 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:55 smithi038 ceph-mon[92422]: pgmap v21685: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 804 KiB/s wr, 10 op/s 2023-03-10T01:59:55.414 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:59:55.415 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:59:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:55 smithi122 ceph-mon[93869]: pgmap v21685: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 804 KiB/s wr, 10 op/s 2023-03-10T01:59:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:59:55.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:59:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:55 smithi181 ceph-mon[94189]: pgmap v21685: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 804 KiB/s wr, 10 op/s 2023-03-10T01:59:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T01:59:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:59:56.583 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39706.554894 secs 2023-03-10T01:59:56.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:59:56.584 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:59:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39706.554894 secs 2023-03-10T01:59:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:59:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:59:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39706.554894 secs 2023-03-10T01:59:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T01:59:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:59:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:57 smithi122 ceph-mon[93869]: pgmap v21686: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.4 MiB/s wr, 16 op/s 2023-03-10T01:59:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:59:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:59:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:57 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:59:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:57 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:59:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:57 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:59:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:57 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:59:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:57 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:59:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:57 smithi181 ceph-mon[94189]: pgmap v21686: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.4 MiB/s wr, 16 op/s 2023-03-10T01:59:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:59:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:59:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:57 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:59:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:57 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:59:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:57 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:59:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:57 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:59:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:57 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:59:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:57 smithi038 ceph-mon[92422]: pgmap v21686: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 1.4 MiB/s wr, 16 op/s 2023-03-10T01:59:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T01:59:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T01:59:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:57 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:59:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:57 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:59:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:57 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T01:59:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:57 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T01:59:57.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:57 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T01:59:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39709.017629 secs 2023-03-10T01:59:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39709.017629 secs 2023-03-10T01:59:58.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39709.017629 secs 2023-03-10T01:59:58.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40170.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T01:59:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 01:59:59 smithi122 ceph-mon[93869]: pgmap v21687: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.3 MiB/s wr, 14 op/s 2023-03-10T01:59:59.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 01:59:59 smithi181 ceph-mon[94189]: pgmap v21687: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.3 MiB/s wr, 14 op/s 2023-03-10T01:59:59.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 01:59:59 smithi038 ceph-mon[92422]: pgmap v21687: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.3 MiB/s wr, 14 op/s 2023-03-10T02:00:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:00:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:00:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:00 smithi122 ceph-mon[93869]: overall HEALTH_WARN 2 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-10T02:00:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:00:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:00:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:00 smithi181 ceph-mon[94189]: overall HEALTH_WARN 2 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-10T02:00:00.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:00:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:00:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:00 smithi038 ceph-mon[92422]: overall HEALTH_WARN 2 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-10T02:00:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:01 smithi122 ceph-mon[93869]: pgmap v21688: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1018 KiB/s wr, 14 op/s 2023-03-10T02:00:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39711.554951 secs 2023-03-10T02:00:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:00:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:00:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:01 smithi181 ceph-mon[94189]: pgmap v21688: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1018 KiB/s wr, 14 op/s 2023-03-10T02:00:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39711.554951 secs 2023-03-10T02:00:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:00:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:00:01.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:01 smithi038 ceph-mon[92422]: pgmap v21688: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1018 KiB/s wr, 14 op/s 2023-03-10T02:00:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39711.554951 secs 2023-03-10T02:00:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:00:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:00:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:03 smithi122 ceph-mon[93869]: pgmap v21689: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.0 MiB/s wr, 23 op/s 2023-03-10T02:00:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39714.017738 secs 2023-03-10T02:00:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:03 smithi181 ceph-mon[94189]: pgmap v21689: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.0 MiB/s wr, 23 op/s 2023-03-10T02:00:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39714.017738 secs 2023-03-10T02:00:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:03 smithi038 ceph-mon[92422]: pgmap v21689: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 2.0 MiB/s wr, 23 op/s 2023-03-10T02:00:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39714.017738 secs 2023-03-10T02:00:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40175.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:00:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:05 smithi122 ceph-mon[93869]: pgmap v21690: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 30 op/s 2023-03-10T02:00:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:05 smithi122 ceph-mon[93869]: osdmap e1571: 12 total, 12 up, 12 in 2023-03-10T02:00:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:05 smithi181 ceph-mon[94189]: pgmap v21690: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 30 op/s 2023-03-10T02:00:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:05 smithi181 ceph-mon[94189]: osdmap e1571: 12 total, 12 up, 12 in 2023-03-10T02:00:05.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:05 smithi038 ceph-mon[92422]: pgmap v21690: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 30 op/s 2023-03-10T02:00:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:05 smithi038 ceph-mon[92422]: osdmap e1571: 12 total, 12 up, 12 in 2023-03-10T02:00:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39716.555009 secs 2023-03-10T02:00:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39716.555009 secs 2023-03-10T02:00:06.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39716.555009 secs 2023-03-10T02:00:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:07 smithi122 ceph-mon[93869]: pgmap v21692: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 34 op/s 2023-03-10T02:00:07.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:07 smithi181 ceph-mon[94189]: pgmap v21692: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 34 op/s 2023-03-10T02:00:07.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:07 smithi038 ceph-mon[92422]: pgmap v21692: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 34 op/s 2023-03-10T02:00:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:00:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:00:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39719.017796 secs 2023-03-10T02:00:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:00:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:00:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39719.017796 secs 2023-03-10T02:00:08.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:00:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:00:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39719.017796 secs 2023-03-10T02:00:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40180.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:00:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:09 smithi122 ceph-mon[93869]: pgmap v21693: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 34 op/s 2023-03-10T02:00:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:00:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:00:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:09 smithi181 ceph-mon[94189]: pgmap v21693: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 34 op/s 2023-03-10T02:00:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:00:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:00:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:09 smithi038 ceph-mon[92422]: pgmap v21693: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.5 MiB/s wr, 34 op/s 2023-03-10T02:00:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:00:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:00:10.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:10 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:00:10.729 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:10 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T02:00:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:10 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:00:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:10 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T02:00:10.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:10 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:00:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:10 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T02:00:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:11 smithi122 ceph-mon[93869]: pgmap v21694: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.8 MiB/s wr, 34 op/s 2023-03-10T02:00:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:11 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:00:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:11 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] up:active 2023-03-10T02:00:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:11 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:00:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:11 smithi181 ceph-mon[94189]: pgmap v21694: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.8 MiB/s wr, 34 op/s 2023-03-10T02:00:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:11 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:00:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:11 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] up:active 2023-03-10T02:00:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:11 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:00:11.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:11 smithi038 ceph-mon[92422]: pgmap v21694: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.8 MiB/s wr, 34 op/s 2023-03-10T02:00:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:11 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:00:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:11 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] up:active 2023-03-10T02:00:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:11 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:00:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:00:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:00:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:00:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:00:12.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:00:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:00:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:13 smithi122 ceph-mon[93869]: pgmap v21695: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 32 op/s 2023-03-10T02:00:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39724.017880 secs 2023-03-10T02:00:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:13 smithi181 ceph-mon[94189]: pgmap v21695: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 32 op/s 2023-03-10T02:00:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39724.017880 secs 2023-03-10T02:00:13.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:13 smithi038 ceph-mon[92422]: pgmap v21695: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 32 op/s 2023-03-10T02:00:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39724.017880 secs 2023-03-10T02:00:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40185.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:00:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:15 smithi122 ceph-mon[93869]: pgmap v21696: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 17 op/s 2023-03-10T02:00:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:15 smithi122 ceph-mon[93869]: osdmap e1572: 12 total, 12 up, 12 in 2023-03-10T02:00:15.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:15 smithi181 ceph-mon[94189]: pgmap v21696: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 17 op/s 2023-03-10T02:00:15.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:15 smithi181 ceph-mon[94189]: osdmap e1572: 12 total, 12 up, 12 in 2023-03-10T02:00:15.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:15 smithi038 ceph-mon[92422]: pgmap v21696: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.1 MiB/s wr, 17 op/s 2023-03-10T02:00:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:15 smithi038 ceph-mon[92422]: osdmap e1572: 12 total, 12 up, 12 in 2023-03-10T02:00:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39726.555152 secs 2023-03-10T02:00:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:00:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:00:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39726.555152 secs 2023-03-10T02:00:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:00:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:00:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39726.555152 secs 2023-03-10T02:00:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:00:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:00:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:17 smithi181 ceph-mon[94189]: pgmap v21698: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 14 op/s 2023-03-10T02:00:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:00:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:00:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:17 smithi038 ceph-mon[92422]: pgmap v21698: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 14 op/s 2023-03-10T02:00:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:00:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:00:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:17 smithi122 ceph-mon[93869]: pgmap v21698: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 14 op/s 2023-03-10T02:00:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:00:17.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:00:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39729.018001 secs 2023-03-10T02:00:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40190.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:00:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39729.018001 secs 2023-03-10T02:00:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39729.018001 secs 2023-03-10T02:00:19.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:19 smithi038 ceph-mon[92422]: pgmap v21699: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 15 op/s 2023-03-10T02:00:19.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:19 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:00:19.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:19 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] up:active 2023-03-10T02:00:19.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:19 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:00:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:19 smithi122 ceph-mon[93869]: pgmap v21699: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 15 op/s 2023-03-10T02:00:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:19 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:00:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:19 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] up:active 2023-03-10T02:00:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:19 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:00:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:19 smithi181 ceph-mon[94189]: pgmap v21699: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.8 MiB/s wr, 15 op/s 2023-03-10T02:00:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:19 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:00:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:19 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6836/2210214596,v1:172.21.15.122:6837/2210214596] up:active 2023-03-10T02:00:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:19 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:00:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:21 smithi038 ceph-mon[92422]: pgmap v21700: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 14 op/s 2023-03-10T02:00:21.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39731.555212 secs 2023-03-10T02:00:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:21 smithi122 ceph-mon[93869]: pgmap v21700: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 14 op/s 2023-03-10T02:00:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39731.555212 secs 2023-03-10T02:00:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:21 smithi181 ceph-mon[94189]: pgmap v21700: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 14 op/s 2023-03-10T02:00:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39731.555212 secs 2023-03-10T02:00:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40195.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:00:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:23 smithi122 ceph-mon[93869]: pgmap v21701: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 295 KiB/s wr, 12 op/s 2023-03-10T02:00:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39734.018060 secs 2023-03-10T02:00:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:23 smithi181 ceph-mon[94189]: pgmap v21701: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 295 KiB/s wr, 12 op/s 2023-03-10T02:00:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39734.018060 secs 2023-03-10T02:00:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:23 smithi038 ceph-mon[92422]: pgmap v21701: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 295 KiB/s wr, 12 op/s 2023-03-10T02:00:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39734.018060 secs 2023-03-10T02:00:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:25 smithi122 ceph-mon[93869]: pgmap v21702: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 295 KiB/s wr, 12 op/s 2023-03-10T02:00:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:25 smithi181 ceph-mon[94189]: pgmap v21702: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 295 KiB/s wr, 12 op/s 2023-03-10T02:00:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:25 smithi038 ceph-mon[92422]: pgmap v21702: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 331 KiB/s rd, 295 KiB/s wr, 12 op/s 2023-03-10T02:00:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39736.555270 secs 2023-03-10T02:00:26.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39736.555270 secs 2023-03-10T02:00:26.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39736.555270 secs 2023-03-10T02:00:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:27 smithi122 ceph-mon[93869]: pgmap v21703: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 260 KiB/s wr, 14 op/s 2023-03-10T02:00:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:27 smithi181 ceph-mon[94189]: pgmap v21703: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 260 KiB/s wr, 14 op/s 2023-03-10T02:00:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:27 smithi038 ceph-mon[92422]: pgmap v21703: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 260 KiB/s wr, 14 op/s 2023-03-10T02:00:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40200.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:00:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39739.018144 secs 2023-03-10T02:00:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39739.018144 secs 2023-03-10T02:00:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39739.018144 secs 2023-03-10T02:00:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:29 smithi122 ceph-mon[93869]: pgmap v21704: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 230 KiB/s wr, 11 op/s 2023-03-10T02:00:29.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:29 smithi181 ceph-mon[94189]: pgmap v21704: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 230 KiB/s wr, 11 op/s 2023-03-10T02:00:29.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:29 smithi038 ceph-mon[92422]: pgmap v21704: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 230 KiB/s wr, 11 op/s 2023-03-10T02:00:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:31 smithi122 ceph-mon[93869]: pgmap v21705: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 229 KiB/s wr, 10 op/s 2023-03-10T02:00:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39741.555352 secs 2023-03-10T02:00:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:31 smithi181 ceph-mon[94189]: pgmap v21705: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 229 KiB/s wr, 10 op/s 2023-03-10T02:00:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39741.555352 secs 2023-03-10T02:00:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:31 smithi038 ceph-mon[92422]: pgmap v21705: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 229 KiB/s wr, 10 op/s 2023-03-10T02:00:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39741.555352 secs 2023-03-10T02:00:32.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:32 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:00:32.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:32 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:00:32.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:32 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:00:32.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:32 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:00:32.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:32 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:00:32.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:32 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:00:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40205.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:00:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:33 smithi122 ceph-mon[93869]: pgmap v21706: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 486 KiB/s rd, 484 KiB/s wr, 14 op/s 2023-03-10T02:00:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:33 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:00:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:33 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:00:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39744.018205 secs 2023-03-10T02:00:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:33 smithi181 ceph-mon[94189]: pgmap v21706: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 486 KiB/s rd, 484 KiB/s wr, 14 op/s 2023-03-10T02:00:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:33 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:00:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:33 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:00:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39744.018205 secs 2023-03-10T02:00:33.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:33 smithi038 ceph-mon[92422]: pgmap v21706: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 486 KiB/s rd, 484 KiB/s wr, 14 op/s 2023-03-10T02:00:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:33 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:00:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:33 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:00:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39744.018205 secs 2023-03-10T02:00:34.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:34 smithi122 ceph-mon[93869]: pgmap v21707: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 289 KiB/s wr, 9 op/s 2023-03-10T02:00:34.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:34 smithi181 ceph-mon[94189]: pgmap v21707: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 289 KiB/s wr, 9 op/s 2023-03-10T02:00:34.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:34 smithi038 ceph-mon[92422]: pgmap v21707: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 289 KiB/s wr, 9 op/s 2023-03-10T02:00:36.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39746.555441 secs 2023-03-10T02:00:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39746.555441 secs 2023-03-10T02:00:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39746.555441 secs 2023-03-10T02:00:37.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:37 smithi038 ceph-mon[92422]: pgmap v21708: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 293 KiB/s rd, 289 KiB/s wr, 11 op/s 2023-03-10T02:00:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:37 smithi122 ceph-mon[93869]: pgmap v21708: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 293 KiB/s rd, 289 KiB/s wr, 11 op/s 2023-03-10T02:00:37.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:37 smithi181 ceph-mon[94189]: pgmap v21708: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 293 KiB/s rd, 289 KiB/s wr, 11 op/s 2023-03-10T02:00:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40210.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:00:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:39 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39749.018293 secs 2023-03-10T02:00:39.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:39 smithi038 ceph-mon[92422]: pgmap v21709: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 255 KiB/s wr, 8 op/s 2023-03-10T02:00:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:39 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39749.018293 secs 2023-03-10T02:00:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:39 smithi122 ceph-mon[93869]: pgmap v21709: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 255 KiB/s wr, 8 op/s 2023-03-10T02:00:39.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:39 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39749.018293 secs 2023-03-10T02:00:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:39 smithi181 ceph-mon[94189]: pgmap v21709: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 255 KiB/s wr, 8 op/s 2023-03-10T02:00:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:41 smithi038 ceph-mon[92422]: pgmap v21710: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 255 KiB/s wr, 6 op/s 2023-03-10T02:00:41.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39751.555497 secs 2023-03-10T02:00:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:41 smithi122 ceph-mon[93869]: pgmap v21710: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 255 KiB/s wr, 6 op/s 2023-03-10T02:00:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39751.555497 secs 2023-03-10T02:00:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:41 smithi181 ceph-mon[94189]: pgmap v21710: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 255 KiB/s wr, 6 op/s 2023-03-10T02:00:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39751.555497 secs 2023-03-10T02:00:43.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:43 smithi038 ceph-mon[92422]: pgmap v21711: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 458 KiB/s wr, 10 op/s 2023-03-10T02:00:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:43 smithi122 ceph-mon[93869]: pgmap v21711: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 458 KiB/s wr, 10 op/s 2023-03-10T02:00:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:43 smithi181 ceph-mon[94189]: pgmap v21711: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 462 KiB/s rd, 458 KiB/s wr, 10 op/s 2023-03-10T02:00:43.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40215.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:00:44.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:44 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39754.018349 secs 2023-03-10T02:00:44.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:44 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39754.018349 secs 2023-03-10T02:00:44.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:44 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39754.018349 secs 2023-03-10T02:00:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:45 smithi122 ceph-mon[93869]: pgmap v21712: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 203 KiB/s wr, 6 op/s 2023-03-10T02:00:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:45 smithi181 ceph-mon[94189]: pgmap v21712: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 203 KiB/s wr, 6 op/s 2023-03-10T02:00:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:45 smithi038 ceph-mon[92422]: pgmap v21712: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 203 KiB/s wr, 6 op/s 2023-03-10T02:00:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39756.555555 secs 2023-03-10T02:00:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39756.555555 secs 2023-03-10T02:00:46.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39756.555555 secs 2023-03-10T02:00:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:47 smithi122 ceph-mon[93869]: pgmap v21713: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 372 KiB/s wr, 9 op/s 2023-03-10T02:00:47.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:47 smithi181 ceph-mon[94189]: pgmap v21713: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 372 KiB/s wr, 9 op/s 2023-03-10T02:00:47.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:47 smithi038 ceph-mon[92422]: pgmap v21713: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 372 KiB/s wr, 9 op/s 2023-03-10T02:00:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:00:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:00:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39759.018409 secs 2023-03-10T02:00:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:00:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:00:48.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39759.018409 secs 2023-03-10T02:00:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:00:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:00:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39759.018409 secs 2023-03-10T02:00:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40220.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:00:49.658 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:49 smithi122 ceph-mon[93869]: pgmap v21714: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 372 KiB/s wr, 7 op/s 2023-03-10T02:00:49.659 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:00:49.659 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:00:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:49 smithi181 ceph-mon[94189]: pgmap v21714: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 372 KiB/s wr, 7 op/s 2023-03-10T02:00:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:00:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:00:49.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:49 smithi038 ceph-mon[92422]: pgmap v21714: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 372 KiB/s wr, 7 op/s 2023-03-10T02:00:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:00:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:00:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:51 smithi122 ceph-mon[93869]: pgmap v21715: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 372 KiB/s wr, 7 op/s 2023-03-10T02:00:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39761.555629 secs 2023-03-10T02:00:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:51 smithi181 ceph-mon[94189]: pgmap v21715: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 372 KiB/s wr, 7 op/s 2023-03-10T02:00:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39761.555629 secs 2023-03-10T02:00:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:51 smithi038 ceph-mon[92422]: pgmap v21715: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 372 KiB/s wr, 7 op/s 2023-03-10T02:00:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39761.555629 secs 2023-03-10T02:00:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:53 smithi122 ceph-mon[93869]: pgmap v21716: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 978 KiB/s wr, 20 op/s 2023-03-10T02:00:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:00:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:00:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:53 smithi181 ceph-mon[94189]: pgmap v21716: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 978 KiB/s wr, 20 op/s 2023-03-10T02:00:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:00:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:00:53.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:53 smithi038 ceph-mon[92422]: pgmap v21716: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 978 KiB/s wr, 20 op/s 2023-03-10T02:00:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:00:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:00:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40225.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:00:54.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:54 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T02:00:54.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:54 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T02:00:54.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:54 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T02:00:55.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:55 smithi122 ceph-mon[93869]: pgmap v21717: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 842 KiB/s rd, 775 KiB/s wr, 17 op/s 2023-03-10T02:00:55.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:55 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:00:55.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:00:55.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:00:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:55 smithi181 ceph-mon[94189]: pgmap v21717: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 842 KiB/s rd, 775 KiB/s wr, 17 op/s 2023-03-10T02:00:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:55 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:00:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:00:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:00:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:55 smithi038 ceph-mon[92422]: pgmap v21717: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 842 KiB/s rd, 775 KiB/s wr, 17 op/s 2023-03-10T02:00:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:55 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:00:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:00:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:00:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39766.555709 secs 2023-03-10T02:00:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:00:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:00:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39766.555709 secs 2023-03-10T02:00:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:00:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:00:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39766.555709 secs 2023-03-10T02:00:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:00:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:00:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:57 smithi122 ceph-mon[93869]: pgmap v21718: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1010 KiB/s wr, 52 op/s 2023-03-10T02:00:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:00:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:00:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:57 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:00:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:57 smithi181 ceph-mon[94189]: pgmap v21718: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1010 KiB/s wr, 52 op/s 2023-03-10T02:00:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:00:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:00:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:57 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:00:57.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:57 smithi038 ceph-mon[92422]: pgmap v21718: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1010 KiB/s wr, 52 op/s 2023-03-10T02:00:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:00:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:00:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:57 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:00:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:58 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 39769.018531 secs 2023-03-10T02:00:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:58 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 39769.018531 secs 2023-03-10T02:00:58.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:58 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 39769.018531 secs 2023-03-10T02:00:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40230.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:00:59.406 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:59 smithi122 ceph-mon[93869]: pgmap v21719: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 841 KiB/s wr, 50 op/s 2023-03-10T02:00:59.406 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:00:59.406 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:00:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:00:59.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:59 smithi181 ceph-mon[94189]: pgmap v21719: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 841 KiB/s wr, 50 op/s 2023-03-10T02:00:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:00:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:00:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:00:59.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:59 smithi038 ceph-mon[92422]: pgmap v21719: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 841 KiB/s wr, 50 op/s 2023-03-10T02:00:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:00:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:00:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:01:00.711 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:01:00.712 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:01:00.712 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:00 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:01:00.722 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:01:00.723 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:01:00.723 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:00 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:01:00.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:01:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:01:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:00 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:01:01.679 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:01 smithi038 ceph-mon[92422]: pgmap v21720: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 841 KiB/s wr, 50 op/s 2023-03-10T02:01:01.680 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39771.555795 secs 2023-03-10T02:01:01.680 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:01 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:01:01.680 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:01 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:01:01.680 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:01 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:01:01.680 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:01 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:01:01.681 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:01 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:01:01.681 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:01 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:01:01.681 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:01 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:01:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:01 smithi122 ceph-mon[93869]: pgmap v21720: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 841 KiB/s wr, 50 op/s 2023-03-10T02:01:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39771.555795 secs 2023-03-10T02:01:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:01 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:01:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:01 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:01:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:01 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:01:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:01 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:01:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:01 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:01:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:01 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:01:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:01 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:01:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:01 smithi181 ceph-mon[94189]: pgmap v21720: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 841 KiB/s wr, 50 op/s 2023-03-10T02:01:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39771.555795 secs 2023-03-10T02:01:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:01 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:01:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:01 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:01:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:01 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:01:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:01 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:01:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:01 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:01:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:01 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:01:01.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:01 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:01:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:03 smithi122 ceph-mon[93869]: pgmap v21721: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 56 op/s 2023-03-10T02:01:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:03 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 39774.018616 secs 2023-03-10T02:01:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:03 smithi181 ceph-mon[94189]: pgmap v21721: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 56 op/s 2023-03-10T02:01:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:03 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 39774.018616 secs 2023-03-10T02:01:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:03 smithi038 ceph-mon[92422]: pgmap v21721: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 56 op/s 2023-03-10T02:01:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:03 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 39774.018616 secs 2023-03-10T02:01:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40235.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:01:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:05 smithi122 ceph-mon[93869]: pgmap v21722: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 638 KiB/s rd, 705 KiB/s wr, 44 op/s 2023-03-10T02:01:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:05 smithi181 ceph-mon[94189]: pgmap v21722: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 638 KiB/s rd, 705 KiB/s wr, 44 op/s 2023-03-10T02:01:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:05 smithi038 ceph-mon[92422]: pgmap v21722: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 638 KiB/s rd, 705 KiB/s wr, 44 op/s 2023-03-10T02:01:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39776.555851 secs 2023-03-10T02:01:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39776.555851 secs 2023-03-10T02:01:06.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39776.555851 secs 2023-03-10T02:01:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:07 smithi122 ceph-mon[93869]: pgmap v21723: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 783 KiB/s rd, 821 KiB/s wr, 49 op/s 2023-03-10T02:01:07.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:07 smithi181 ceph-mon[94189]: pgmap v21723: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 783 KiB/s rd, 821 KiB/s wr, 49 op/s 2023-03-10T02:01:07.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:07 smithi038 ceph-mon[92422]: pgmap v21723: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 783 KiB/s rd, 821 KiB/s wr, 49 op/s 2023-03-10T02:01:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:01:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:01:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39779.018720 secs 2023-03-10T02:01:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:01:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:01:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39779.018720 secs 2023-03-10T02:01:08.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:01:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:01:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39779.018720 secs 2023-03-10T02:01:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40240.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:01:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:09 smithi122 ceph-mon[93869]: pgmap v21724: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 761 KiB/s rd, 840 KiB/s wr, 16 op/s 2023-03-10T02:01:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:01:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:01:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:09 smithi181 ceph-mon[94189]: pgmap v21724: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 761 KiB/s rd, 840 KiB/s wr, 16 op/s 2023-03-10T02:01:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:01:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:01:09.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:09 smithi038 ceph-mon[92422]: pgmap v21724: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 761 KiB/s rd, 840 KiB/s wr, 16 op/s 2023-03-10T02:01:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:01:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:01:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:10 smithi122 ceph-mon[93869]: osdmap e1573: 12 total, 12 up, 12 in 2023-03-10T02:01:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:10 smithi181 ceph-mon[94189]: osdmap e1573: 12 total, 12 up, 12 in 2023-03-10T02:01:10.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:10 smithi038 ceph-mon[92422]: osdmap e1573: 12 total, 12 up, 12 in 2023-03-10T02:01:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:11 smithi122 ceph-mon[93869]: pgmap v21726: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 921 KiB/s rd, 1.0 MiB/s wr, 30 op/s 2023-03-10T02:01:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39781.555915 secs 2023-03-10T02:01:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:11 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:01:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:11 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:01:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:11 smithi181 ceph-mon[94189]: pgmap v21726: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 921 KiB/s rd, 1.0 MiB/s wr, 30 op/s 2023-03-10T02:01:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39781.555915 secs 2023-03-10T02:01:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:11 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:01:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:11 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:01:11.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:11 smithi038 ceph-mon[92422]: pgmap v21726: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 921 KiB/s rd, 1.0 MiB/s wr, 30 op/s 2023-03-10T02:01:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39781.555915 secs 2023-03-10T02:01:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:11 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:01:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:11 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:01:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:01:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:01:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:01:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:01:12.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:01:12.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:01:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:13 smithi122 ceph-mon[93869]: pgmap v21727: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1018 KiB/s wr, 29 op/s 2023-03-10T02:01:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:01:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:01:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39784.018781 secs 2023-03-10T02:01:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:13 smithi181 ceph-mon[94189]: pgmap v21727: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1018 KiB/s wr, 29 op/s 2023-03-10T02:01:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:01:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:01:13.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39784.018781 secs 2023-03-10T02:01:13.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:13 smithi038 ceph-mon[92422]: pgmap v21727: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1018 KiB/s wr, 29 op/s 2023-03-10T02:01:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:01:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:01:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39784.018781 secs 2023-03-10T02:01:13.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40245.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:01:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:15 smithi122 ceph-mon[93869]: pgmap v21728: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1018 KiB/s wr, 27 op/s 2023-03-10T02:01:15.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:15 smithi181 ceph-mon[94189]: pgmap v21728: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1018 KiB/s wr, 27 op/s 2023-03-10T02:01:15.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:15 smithi038 ceph-mon[92422]: pgmap v21728: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1018 KiB/s wr, 27 op/s 2023-03-10T02:01:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39786.555978 secs 2023-03-10T02:01:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:01:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:01:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39786.555978 secs 2023-03-10T02:01:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:01:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:01:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39786.555978 secs 2023-03-10T02:01:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:01:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:01:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:17 smithi122 ceph-mon[93869]: pgmap v21729: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 935 KiB/s rd, 879 KiB/s wr, 26 op/s 2023-03-10T02:01:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:01:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:01:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:17 smithi181 ceph-mon[94189]: pgmap v21729: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 935 KiB/s rd, 879 KiB/s wr, 26 op/s 2023-03-10T02:01:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:01:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:01:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:17 smithi038 ceph-mon[92422]: pgmap v21729: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 935 KiB/s rd, 879 KiB/s wr, 26 op/s 2023-03-10T02:01:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:01:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:01:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39789.018842 secs 2023-03-10T02:01:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39789.018842 secs 2023-03-10T02:01:18.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40250.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:01:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39789.018842 secs 2023-03-10T02:01:19.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:19 smithi181 ceph-mon[94189]: pgmap v21730: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 575 KiB/s wr, 24 op/s 2023-03-10T02:01:19.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:19 smithi038 ceph-mon[92422]: pgmap v21730: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 575 KiB/s wr, 24 op/s 2023-03-10T02:01:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:19 smithi122 ceph-mon[93869]: pgmap v21730: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 632 KiB/s rd, 575 KiB/s wr, 24 op/s 2023-03-10T02:01:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:21 smithi038 ceph-mon[92422]: pgmap v21731: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 598 KiB/s rd, 526 KiB/s wr, 11 op/s 2023-03-10T02:01:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39791.556035 secs 2023-03-10T02:01:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:21 smithi038 ceph-mon[92422]: osdmap e1574: 12 total, 12 up, 12 in 2023-03-10T02:01:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:21 smithi122 ceph-mon[93869]: pgmap v21731: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 598 KiB/s rd, 526 KiB/s wr, 11 op/s 2023-03-10T02:01:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39791.556035 secs 2023-03-10T02:01:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:21 smithi122 ceph-mon[93869]: osdmap e1574: 12 total, 12 up, 12 in 2023-03-10T02:01:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:21 smithi181 ceph-mon[94189]: pgmap v21731: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 598 KiB/s rd, 526 KiB/s wr, 11 op/s 2023-03-10T02:01:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39791.556035 secs 2023-03-10T02:01:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:21 smithi181 ceph-mon[94189]: osdmap e1574: 12 total, 12 up, 12 in 2023-03-10T02:01:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:23 smithi038 ceph-mon[92422]: pgmap v21733: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 173 KiB/s wr, 10 op/s 2023-03-10T02:01:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:23 smithi122 ceph-mon[93869]: pgmap v21733: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 173 KiB/s wr, 10 op/s 2023-03-10T02:01:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:23 smithi181 ceph-mon[94189]: pgmap v21733: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 173 KiB/s wr, 10 op/s 2023-03-10T02:01:23.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40255.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:01:24.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:24 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39794.018900 secs 2023-03-10T02:01:24.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:24 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39794.018900 secs 2023-03-10T02:01:24.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:24 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39794.018900 secs 2023-03-10T02:01:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:25 smithi122 ceph-mon[93869]: pgmap v21734: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 173 KiB/s wr, 10 op/s 2023-03-10T02:01:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:25 smithi181 ceph-mon[94189]: pgmap v21734: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 173 KiB/s wr, 10 op/s 2023-03-10T02:01:25.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:25 smithi038 ceph-mon[92422]: pgmap v21734: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 173 KiB/s wr, 10 op/s 2023-03-10T02:01:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39796.556107 secs 2023-03-10T02:01:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39796.556107 secs 2023-03-10T02:01:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39796.556107 secs 2023-03-10T02:01:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:27 smithi122 ceph-mon[93869]: pgmap v21735: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 172 KiB/s wr, 9 op/s 2023-03-10T02:01:27.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:27 smithi181 ceph-mon[94189]: pgmap v21735: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 172 KiB/s wr, 9 op/s 2023-03-10T02:01:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:27 smithi038 ceph-mon[92422]: pgmap v21735: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 172 KiB/s wr, 9 op/s 2023-03-10T02:01:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39799.018962 secs 2023-03-10T02:01:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39799.018962 secs 2023-03-10T02:01:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39799.018962 secs 2023-03-10T02:01:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40260.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:01:29.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:29 smithi122 ceph-mon[93869]: pgmap v21736: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 172 KiB/s wr, 8 op/s 2023-03-10T02:01:29.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:29 smithi181 ceph-mon[94189]: pgmap v21736: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 172 KiB/s wr, 8 op/s 2023-03-10T02:01:29.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:29 smithi038 ceph-mon[92422]: pgmap v21736: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 172 KiB/s wr, 8 op/s 2023-03-10T02:01:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:31 smithi122 ceph-mon[93869]: pgmap v21737: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 311 KiB/s wr, 8 op/s 2023-03-10T02:01:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39801.556165 secs 2023-03-10T02:01:31.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:31 smithi181 ceph-mon[94189]: pgmap v21737: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 311 KiB/s wr, 8 op/s 2023-03-10T02:01:31.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39801.556165 secs 2023-03-10T02:01:31.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:31 smithi038 ceph-mon[92422]: pgmap v21737: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 174 KiB/s rd, 311 KiB/s wr, 8 op/s 2023-03-10T02:01:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39801.556165 secs 2023-03-10T02:01:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:33 smithi122 ceph-mon[93869]: pgmap v21738: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 327 KiB/s wr, 10 op/s 2023-03-10T02:01:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39804.019046 secs 2023-03-10T02:01:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:33 smithi181 ceph-mon[94189]: pgmap v21738: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 327 KiB/s wr, 10 op/s 2023-03-10T02:01:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39804.019046 secs 2023-03-10T02:01:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:33 smithi038 ceph-mon[92422]: pgmap v21738: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 327 KiB/s wr, 10 op/s 2023-03-10T02:01:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39804.019046 secs 2023-03-10T02:01:33.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40265.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:01:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:35 smithi122 ceph-mon[93869]: pgmap v21739: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 173 KiB/s wr, 6 op/s 2023-03-10T02:01:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:35 smithi181 ceph-mon[94189]: pgmap v21739: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 173 KiB/s wr, 6 op/s 2023-03-10T02:01:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:35 smithi038 ceph-mon[92422]: pgmap v21739: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 173 KiB/s wr, 6 op/s 2023-03-10T02:01:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39806.556228 secs 2023-03-10T02:01:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39806.556228 secs 2023-03-10T02:01:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39806.556228 secs 2023-03-10T02:01:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:37 smithi122 ceph-mon[93869]: pgmap v21740: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 173 KiB/s wr, 9 op/s 2023-03-10T02:01:37.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:37 smithi181 ceph-mon[94189]: pgmap v21740: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 173 KiB/s wr, 9 op/s 2023-03-10T02:01:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:37 smithi038 ceph-mon[92422]: pgmap v21740: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 173 KiB/s wr, 9 op/s 2023-03-10T02:01:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39809.019109 secs 2023-03-10T02:01:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39809.019109 secs 2023-03-10T02:01:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39809.019109 secs 2023-03-10T02:01:38.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40270.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:01:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:39 smithi122 ceph-mon[93869]: pgmap v21741: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 173 KiB/s wr, 6 op/s 2023-03-10T02:01:39.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:39 smithi181 ceph-mon[94189]: pgmap v21741: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 173 KiB/s wr, 6 op/s 2023-03-10T02:01:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:39 smithi038 ceph-mon[92422]: pgmap v21741: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 173 KiB/s wr, 6 op/s 2023-03-10T02:01:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:41 smithi122 ceph-mon[93869]: pgmap v21742: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 173 KiB/s wr, 6 op/s 2023-03-10T02:01:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39811.556287 secs 2023-03-10T02:01:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:41 smithi181 ceph-mon[94189]: pgmap v21742: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 173 KiB/s wr, 6 op/s 2023-03-10T02:01:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39811.556287 secs 2023-03-10T02:01:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:41 smithi038 ceph-mon[92422]: pgmap v21742: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 173 KiB/s wr, 6 op/s 2023-03-10T02:01:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39811.556287 secs 2023-03-10T02:01:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:43 smithi122 ceph-mon[93869]: pgmap v21743: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 58 KiB/s wr, 9 op/s 2023-03-10T02:01:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39814.019169 secs 2023-03-10T02:01:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:43 smithi181 ceph-mon[94189]: pgmap v21743: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 58 KiB/s wr, 9 op/s 2023-03-10T02:01:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39814.019169 secs 2023-03-10T02:01:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:43 smithi038 ceph-mon[92422]: pgmap v21743: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 177 KiB/s rd, 58 KiB/s wr, 9 op/s 2023-03-10T02:01:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39814.019169 secs 2023-03-10T02:01:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40275.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:01:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:45 smithi122 ceph-mon[93869]: pgmap v21744: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 170 B/s wr, 6 op/s 2023-03-10T02:01:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:45 smithi181 ceph-mon[94189]: pgmap v21744: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 170 B/s wr, 6 op/s 2023-03-10T02:01:45.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:45 smithi038 ceph-mon[92422]: pgmap v21744: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 170 B/s wr, 6 op/s 2023-03-10T02:01:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39816.556346 secs 2023-03-10T02:01:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39816.556346 secs 2023-03-10T02:01:46.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39816.556346 secs 2023-03-10T02:01:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:47 smithi122 ceph-mon[93869]: pgmap v21745: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 8 op/s 2023-03-10T02:01:47.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:47 smithi181 ceph-mon[94189]: pgmap v21745: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 8 op/s 2023-03-10T02:01:47.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:47 smithi038 ceph-mon[92422]: pgmap v21745: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 8 op/s 2023-03-10T02:01:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39819.019236 secs 2023-03-10T02:01:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39819.019236 secs 2023-03-10T02:01:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39819.019236 secs 2023-03-10T02:01:48.793 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40280.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:01:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:49 smithi122 ceph-mon[93869]: pgmap v21746: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-10T02:01:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:01:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:01:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:49 smithi181 ceph-mon[94189]: pgmap v21746: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-10T02:01:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:01:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:01:49.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:49 smithi038 ceph-mon[92422]: pgmap v21746: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2023-03-10T02:01:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:01:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:01:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:51 smithi122 ceph-mon[93869]: pgmap v21747: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 169 KiB/s wr, 6 op/s 2023-03-10T02:01:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39821.556403 secs 2023-03-10T02:01:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:51 smithi181 ceph-mon[94189]: pgmap v21747: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 169 KiB/s wr, 6 op/s 2023-03-10T02:01:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39821.556403 secs 2023-03-10T02:01:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:51 smithi038 ceph-mon[92422]: pgmap v21747: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 171 KiB/s rd, 169 KiB/s wr, 6 op/s 2023-03-10T02:01:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39821.556403 secs 2023-03-10T02:01:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:01:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:01:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:01:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:01:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:01:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:01:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:53 smithi122 ceph-mon[93869]: pgmap v21748: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 318 KiB/s rd, 315 KiB/s wr, 9 op/s 2023-03-10T02:01:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39824.019296 secs 2023-03-10T02:01:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:53 smithi181 ceph-mon[94189]: pgmap v21748: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 318 KiB/s rd, 315 KiB/s wr, 9 op/s 2023-03-10T02:01:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39824.019296 secs 2023-03-10T02:01:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:53 smithi038 ceph-mon[92422]: pgmap v21748: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 318 KiB/s rd, 315 KiB/s wr, 9 op/s 2023-03-10T02:01:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39824.019296 secs 2023-03-10T02:01:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40285.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:01:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:55 smithi122 ceph-mon[93869]: pgmap v21749: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 372 KiB/s wr, 7 op/s 2023-03-10T02:01:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:55 smithi181 ceph-mon[94189]: pgmap v21749: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 372 KiB/s wr, 7 op/s 2023-03-10T02:01:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:55 smithi038 ceph-mon[92422]: pgmap v21749: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 372 KiB/s wr, 7 op/s 2023-03-10T02:01:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39826.556461 secs 2023-03-10T02:01:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:01:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:01:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39826.556461 secs 2023-03-10T02:01:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:01:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:01:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39826.556461 secs 2023-03-10T02:01:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:01:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:01:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:57 smithi122 ceph-mon[93869]: pgmap v21750: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 442 KiB/s rd, 437 KiB/s wr, 10 op/s 2023-03-10T02:01:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:01:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:01:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:57 smithi181 ceph-mon[94189]: pgmap v21750: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 442 KiB/s rd, 437 KiB/s wr, 10 op/s 2023-03-10T02:01:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:01:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:01:57.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:57 smithi038 ceph-mon[92422]: pgmap v21750: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 442 KiB/s rd, 437 KiB/s wr, 10 op/s 2023-03-10T02:01:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:01:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:01:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39829.019355 secs 2023-03-10T02:01:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39829.019355 secs 2023-03-10T02:01:58.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39829.019355 secs 2023-03-10T02:01:58.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40290.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:01:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:01:59 smithi122 ceph-mon[93869]: pgmap v21751: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 437 KiB/s wr, 8 op/s 2023-03-10T02:01:59.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:01:59 smithi181 ceph-mon[94189]: pgmap v21751: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 437 KiB/s wr, 8 op/s 2023-03-10T02:01:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:01:59 smithi038 ceph-mon[92422]: pgmap v21751: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 437 KiB/s wr, 8 op/s 2023-03-10T02:02:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:02:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:02:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:00.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:02:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:01 smithi122 ceph-mon[93869]: pgmap v21752: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 452 KiB/s rd, 437 KiB/s wr, 9 op/s 2023-03-10T02:02:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39831.556518 secs 2023-03-10T02:02:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:02:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:01 smithi181 ceph-mon[94189]: pgmap v21752: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 452 KiB/s rd, 437 KiB/s wr, 9 op/s 2023-03-10T02:02:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39831.556518 secs 2023-03-10T02:02:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:02:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:01 smithi038 ceph-mon[92422]: pgmap v21752: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 452 KiB/s rd, 437 KiB/s wr, 9 op/s 2023-03-10T02:02:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39831.556518 secs 2023-03-10T02:02:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:02:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:03 smithi122 ceph-mon[93869]: pgmap v21753: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 954 KiB/s rd, 949 KiB/s wr, 15 op/s 2023-03-10T02:02:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39834.019438 secs 2023-03-10T02:02:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:03 smithi181 ceph-mon[94189]: pgmap v21753: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 954 KiB/s rd, 949 KiB/s wr, 15 op/s 2023-03-10T02:02:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39834.019438 secs 2023-03-10T02:02:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:03 smithi038 ceph-mon[92422]: pgmap v21753: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 954 KiB/s rd, 949 KiB/s wr, 15 op/s 2023-03-10T02:02:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39834.019438 secs 2023-03-10T02:02:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40295.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:02:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:05 smithi122 ceph-mon[93869]: pgmap v21754: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 814 KiB/s rd, 814 KiB/s wr, 23 op/s 2023-03-10T02:02:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:05 smithi181 ceph-mon[94189]: pgmap v21754: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 814 KiB/s rd, 814 KiB/s wr, 23 op/s 2023-03-10T02:02:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:05 smithi038 ceph-mon[92422]: pgmap v21754: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 814 KiB/s rd, 814 KiB/s wr, 23 op/s 2023-03-10T02:02:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39836.556577 secs 2023-03-10T02:02:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39836.556577 secs 2023-03-10T02:02:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39836.556577 secs 2023-03-10T02:02:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:07 smithi122 ceph-mon[93869]: pgmap v21755: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 758 KiB/s rd, 757 KiB/s wr, 25 op/s 2023-03-10T02:02:07.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:07 smithi181 ceph-mon[94189]: pgmap v21755: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 758 KiB/s rd, 757 KiB/s wr, 25 op/s 2023-03-10T02:02:07.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:07 smithi038 ceph-mon[92422]: pgmap v21755: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 758 KiB/s rd, 757 KiB/s wr, 25 op/s 2023-03-10T02:02:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:02:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:02:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:02:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40300.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:02:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:09 smithi038 ceph-mon[92422]: pgmap v21756: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 23 op/s 2023-03-10T02:02:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:02:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:09 smithi122 ceph-mon[93869]: pgmap v21756: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 23 op/s 2023-03-10T02:02:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:02:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:09 smithi181 ceph-mon[94189]: pgmap v21756: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 23 op/s 2023-03-10T02:02:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:02:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:10 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T02:02:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:10 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T02:02:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:10 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T02:02:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:11 smithi038 ceph-mon[92422]: pgmap v21757: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 23 op/s 2023-03-10T02:02:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39841.556657 secs 2023-03-10T02:02:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:11 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:02:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:11 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:02:11.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:11 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:11 smithi122 ceph-mon[93869]: pgmap v21757: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 23 op/s 2023-03-10T02:02:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39841.556657 secs 2023-03-10T02:02:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:11 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:02:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:11 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:02:11.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:11 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:11 smithi181 ceph-mon[94189]: pgmap v21757: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 23 op/s 2023-03-10T02:02:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39841.556657 secs 2023-03-10T02:02:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:11 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:02:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:11 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:02:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:11 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:02:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:12.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:02:12.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:12.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:02:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40305.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:02:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:13 smithi122 ceph-mon[93869]: pgmap v21758: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 26 op/s 2023-03-10T02:02:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:02:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:13 smithi181 ceph-mon[94189]: pgmap v21758: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 26 op/s 2023-03-10T02:02:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:02:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:13.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:13 smithi038 ceph-mon[92422]: pgmap v21758: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 26 op/s 2023-03-10T02:02:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:02:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:15 smithi122 ceph-mon[93869]: pgmap v21759: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 712 KiB/s rd, 714 KiB/s wr, 20 op/s 2023-03-10T02:02:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:15 smithi181 ceph-mon[94189]: pgmap v21759: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 712 KiB/s rd, 714 KiB/s wr, 20 op/s 2023-03-10T02:02:15.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:15 smithi038 ceph-mon[92422]: pgmap v21759: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 712 KiB/s rd, 714 KiB/s wr, 20 op/s 2023-03-10T02:02:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39846.556739 secs 2023-03-10T02:02:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:02:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39846.556739 secs 2023-03-10T02:02:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:02:16.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:16.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39846.556739 secs 2023-03-10T02:02:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:02:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:17 smithi122 ceph-mon[93869]: pgmap v21760: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 875 KiB/s rd, 878 KiB/s wr, 13 op/s 2023-03-10T02:02:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:02:17.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:17 smithi181 ceph-mon[94189]: pgmap v21760: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 875 KiB/s rd, 878 KiB/s wr, 13 op/s 2023-03-10T02:02:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:02:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:17 smithi038 ceph-mon[92422]: pgmap v21760: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 875 KiB/s rd, 878 KiB/s wr, 13 op/s 2023-03-10T02:02:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:02:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40310.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:02:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:18 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:02:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:18 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:02:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:18 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:02:18.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:18 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:02:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:18 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:02:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:18 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:02:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:18 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:02:18.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:18 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:02:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:18 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:02:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:18 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:02:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:18 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:02:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:18 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:02:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:19 smithi122 ceph-mon[93869]: pgmap v21761: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 873 KiB/s rd, 877 KiB/s wr, 10 op/s 2023-03-10T02:02:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:19 smithi181 ceph-mon[94189]: pgmap v21761: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 873 KiB/s rd, 877 KiB/s wr, 10 op/s 2023-03-10T02:02:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:19 smithi038 ceph-mon[92422]: pgmap v21761: 129 pgs: 129 active+clean; 11 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 873 KiB/s rd, 877 KiB/s wr, 10 op/s 2023-03-10T02:02:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:02:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:02:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:02:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:21 smithi122 ceph-mon[93869]: pgmap v21762: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 855 KiB/s rd, 869 KiB/s wr, 11 op/s 2023-03-10T02:02:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39851.556794 secs 2023-03-10T02:02:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:02:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:21 smithi181 ceph-mon[94189]: pgmap v21762: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 855 KiB/s rd, 869 KiB/s wr, 11 op/s 2023-03-10T02:02:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39851.556794 secs 2023-03-10T02:02:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:02:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:21 smithi038 ceph-mon[92422]: pgmap v21762: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 855 KiB/s rd, 869 KiB/s wr, 11 op/s 2023-03-10T02:02:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39851.556794 secs 2023-03-10T02:02:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:02:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40315.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:02:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:23 smithi122 ceph-mon[93869]: pgmap v21763: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 955 KiB/s rd, 975 KiB/s wr, 16 op/s 2023-03-10T02:02:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:23 smithi181 ceph-mon[94189]: pgmap v21763: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 955 KiB/s rd, 975 KiB/s wr, 16 op/s 2023-03-10T02:02:23.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:23 smithi038 ceph-mon[92422]: pgmap v21763: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 955 KiB/s rd, 975 KiB/s wr, 16 op/s 2023-03-10T02:02:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:25 smithi122 ceph-mon[93869]: pgmap v21764: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 588 KiB/s rd, 609 KiB/s wr, 11 op/s 2023-03-10T02:02:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:25 smithi122 ceph-mon[93869]: osdmap e1575: 12 total, 12 up, 12 in 2023-03-10T02:02:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:25 smithi181 ceph-mon[94189]: pgmap v21764: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 588 KiB/s rd, 609 KiB/s wr, 11 op/s 2023-03-10T02:02:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:25 smithi181 ceph-mon[94189]: osdmap e1575: 12 total, 12 up, 12 in 2023-03-10T02:02:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:25 smithi038 ceph-mon[92422]: pgmap v21764: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 588 KiB/s rd, 609 KiB/s wr, 11 op/s 2023-03-10T02:02:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:25 smithi038 ceph-mon[92422]: osdmap e1575: 12 total, 12 up, 12 in 2023-03-10T02:02:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39856.556880 secs 2023-03-10T02:02:26.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39856.556880 secs 2023-03-10T02:02:26.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39856.556880 secs 2023-03-10T02:02:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:27 smithi122 ceph-mon[93869]: pgmap v21766: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 551 KiB/s wr, 14 op/s 2023-03-10T02:02:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:27 smithi181 ceph-mon[94189]: pgmap v21766: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 551 KiB/s wr, 14 op/s 2023-03-10T02:02:27.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:27 smithi038 ceph-mon[92422]: pgmap v21766: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 534 KiB/s rd, 551 KiB/s wr, 14 op/s 2023-03-10T02:02:28.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40320.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:02:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:28 smithi122 ceph-mon[93869]: pgmap v21767: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 910 KiB/s rd, 928 KiB/s wr, 14 op/s 2023-03-10T02:02:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:28 smithi181 ceph-mon[94189]: pgmap v21767: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 910 KiB/s rd, 928 KiB/s wr, 14 op/s 2023-03-10T02:02:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:28 smithi038 ceph-mon[92422]: pgmap v21767: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 910 KiB/s rd, 928 KiB/s wr, 14 op/s 2023-03-10T02:02:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:30 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:30 smithi122 ceph-mon[93869]: pgmap v21768: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 533 KiB/s wr, 12 op/s 2023-03-10T02:02:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:30 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:30.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:30 smithi181 ceph-mon[94189]: pgmap v21768: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 533 KiB/s wr, 12 op/s 2023-03-10T02:02:30.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:30 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:30.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:30 smithi038 ceph-mon[92422]: pgmap v21768: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 533 KiB/s wr, 12 op/s 2023-03-10T02:02:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39861.556939 secs 2023-03-10T02:02:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:31 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39861.556939 secs 2023-03-10T02:02:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:31 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:31.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39861.556939 secs 2023-03-10T02:02:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:31 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:32.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:32 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:02:32.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:32 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:32.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:32 smithi122 ceph-mon[93869]: pgmap v21769: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 549 KiB/s rd, 481 KiB/s wr, 11 op/s 2023-03-10T02:02:32.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:32 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:02:32.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:32 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:32.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:32 smithi181 ceph-mon[94189]: pgmap v21769: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 549 KiB/s rd, 481 KiB/s wr, 11 op/s 2023-03-10T02:02:32.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:32 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:02:32.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:32 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:32.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:32 smithi038 ceph-mon[92422]: pgmap v21769: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 549 KiB/s rd, 481 KiB/s wr, 11 op/s 2023-03-10T02:02:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40325.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:02:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:33 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:02:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:33 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:33 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:02:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:33 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:33 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:02:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:33 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:34.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:34 smithi122 ceph-mon[93869]: pgmap v21770: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 720 KiB/s rd, 890 KiB/s wr, 12 op/s 2023-03-10T02:02:34.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:34 smithi181 ceph-mon[94189]: pgmap v21770: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 720 KiB/s rd, 890 KiB/s wr, 12 op/s 2023-03-10T02:02:34.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:34 smithi038 ceph-mon[92422]: pgmap v21770: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 720 KiB/s rd, 890 KiB/s wr, 12 op/s 2023-03-10T02:02:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39866.556993 secs 2023-03-10T02:02:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:36 smithi122 ceph-mon[93869]: osdmap e1576: 12 total, 12 up, 12 in 2023-03-10T02:02:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39866.556993 secs 2023-03-10T02:02:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:36 smithi181 ceph-mon[94189]: osdmap e1576: 12 total, 12 up, 12 in 2023-03-10T02:02:36.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39866.556993 secs 2023-03-10T02:02:36.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:36 smithi038 ceph-mon[92422]: osdmap e1576: 12 total, 12 up, 12 in 2023-03-10T02:02:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:37 smithi122 ceph-mon[93869]: pgmap v21772: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 12 op/s 2023-03-10T02:02:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:02:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:37.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:37 smithi181 ceph-mon[94189]: pgmap v21772: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 12 op/s 2023-03-10T02:02:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:02:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:37 smithi038 ceph-mon[92422]: pgmap v21772: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 12 op/s 2023-03-10T02:02:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:02:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39869.019942 secs 2023-03-10T02:02:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39869.019942 secs 2023-03-10T02:02:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39869.019942 secs 2023-03-10T02:02:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40330.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:02:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:39 smithi122 ceph-mon[93869]: pgmap v21773: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 760 KiB/s rd, 970 KiB/s wr, 12 op/s 2023-03-10T02:02:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:39 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:02:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:39 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:02:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:39 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:39.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:39 smithi181 ceph-mon[94189]: pgmap v21773: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 760 KiB/s rd, 970 KiB/s wr, 12 op/s 2023-03-10T02:02:39.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:39 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:02:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:39 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:02:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:39 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:39 smithi038 ceph-mon[92422]: pgmap v21773: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 760 KiB/s rd, 970 KiB/s wr, 12 op/s 2023-03-10T02:02:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:39 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:02:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:39 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:02:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:39 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:02:40.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:40.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:02:40.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:02:40.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:41 smithi122 ceph-mon[93869]: pgmap v21774: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 999 KiB/s rd, 970 KiB/s wr, 13 op/s 2023-03-10T02:02:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39871.557050 secs 2023-03-10T02:02:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:41 smithi181 ceph-mon[94189]: pgmap v21774: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 999 KiB/s rd, 970 KiB/s wr, 13 op/s 2023-03-10T02:02:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39871.557050 secs 2023-03-10T02:02:41.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:41 smithi038 ceph-mon[92422]: pgmap v21774: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 999 KiB/s rd, 970 KiB/s wr, 13 op/s 2023-03-10T02:02:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39871.557050 secs 2023-03-10T02:02:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:43 smithi122 ceph-mon[93869]: pgmap v21775: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 13 op/s 2023-03-10T02:02:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39874.020002 secs 2023-03-10T02:02:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:43 smithi181 ceph-mon[94189]: pgmap v21775: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 13 op/s 2023-03-10T02:02:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39874.020002 secs 2023-03-10T02:02:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:43 smithi038 ceph-mon[92422]: pgmap v21775: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 13 op/s 2023-03-10T02:02:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39874.020002 secs 2023-03-10T02:02:43.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40335.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:02:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:45 smithi122 ceph-mon[93869]: pgmap v21776: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 963 KiB/s rd, 696 KiB/s wr, 12 op/s 2023-03-10T02:02:45.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:45 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:45 smithi181 ceph-mon[94189]: pgmap v21776: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 963 KiB/s rd, 696 KiB/s wr, 12 op/s 2023-03-10T02:02:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:45 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:45 smithi038 ceph-mon[92422]: pgmap v21776: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 963 KiB/s rd, 696 KiB/s wr, 12 op/s 2023-03-10T02:02:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:45 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39876.557110 secs 2023-03-10T02:02:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:46 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39876.557110 secs 2023-03-10T02:02:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:46 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:46.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39876.557110 secs 2023-03-10T02:02:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:46 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:47 smithi122 ceph-mon[93869]: pgmap v21777: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 16 op/s 2023-03-10T02:02:47.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:47 smithi181 ceph-mon[94189]: pgmap v21777: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 16 op/s 2023-03-10T02:02:47.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:47 smithi038 ceph-mon[92422]: pgmap v21777: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 16 op/s 2023-03-10T02:02:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:02:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:02:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:48.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:02:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40340.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:02:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:49 smithi122 ceph-mon[93869]: pgmap v21778: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 922 KiB/s rd, 635 KiB/s wr, 10 op/s 2023-03-10T02:02:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:02:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:49 smithi181 ceph-mon[94189]: pgmap v21778: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 922 KiB/s rd, 635 KiB/s wr, 10 op/s 2023-03-10T02:02:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:02:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:49.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:49 smithi038 ceph-mon[92422]: pgmap v21778: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 922 KiB/s rd, 635 KiB/s wr, 10 op/s 2023-03-10T02:02:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:02:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:50 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T02:02:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:50 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T02:02:50.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:50 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T02:02:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:51 smithi122 ceph-mon[93869]: pgmap v21779: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 894 KiB/s rd, 635 KiB/s wr, 10 op/s 2023-03-10T02:02:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39881.557195 secs 2023-03-10T02:02:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:51 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:02:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:02:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:51 smithi181 ceph-mon[94189]: pgmap v21779: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 894 KiB/s rd, 635 KiB/s wr, 10 op/s 2023-03-10T02:02:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39881.557195 secs 2023-03-10T02:02:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:51 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:02:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:02:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:51 smithi038 ceph-mon[92422]: pgmap v21779: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 894 KiB/s rd, 635 KiB/s wr, 10 op/s 2023-03-10T02:02:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39881.557195 secs 2023-03-10T02:02:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:51 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:02:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:02:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:02:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:02:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:02:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:53 smithi122 ceph-mon[93869]: pgmap v21780: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 15 op/s 2023-03-10T02:02:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:02:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:53 smithi181 ceph-mon[94189]: pgmap v21780: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 15 op/s 2023-03-10T02:02:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:02:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:53.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:53 smithi038 ceph-mon[92422]: pgmap v21780: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 15 op/s 2023-03-10T02:02:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:02:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40345.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:02:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:55 smithi122 ceph-mon[93869]: pgmap v21781: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 12 op/s 2023-03-10T02:02:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:55 smithi181 ceph-mon[94189]: pgmap v21781: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 12 op/s 2023-03-10T02:02:55.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:55 smithi038 ceph-mon[92422]: pgmap v21781: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 12 op/s 2023-03-10T02:02:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39886.557253 secs 2023-03-10T02:02:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:02:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39886.557253 secs 2023-03-10T02:02:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:02:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39886.557253 secs 2023-03-10T02:02:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:02:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:57 smithi122 ceph-mon[93869]: pgmap v21782: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 15 op/s 2023-03-10T02:02:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:02:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:57 smithi181 ceph-mon[94189]: pgmap v21782: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 15 op/s 2023-03-10T02:02:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:02:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:57 smithi038 ceph-mon[92422]: pgmap v21782: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 15 op/s 2023-03-10T02:02:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:02:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:02:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40350.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:02:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:02:59 smithi122 ceph-mon[93869]: pgmap v21783: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 866 KiB/s wr, 12 op/s 2023-03-10T02:02:59.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:02:59 smithi181 ceph-mon[94189]: pgmap v21783: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 866 KiB/s wr, 12 op/s 2023-03-10T02:02:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:02:59 smithi038 ceph-mon[92422]: pgmap v21783: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 866 KiB/s wr, 12 op/s 2023-03-10T02:03:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:03:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:03:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:03:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:03:00.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:03:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:03:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:01 smithi122 ceph-mon[93869]: pgmap v21784: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 867 KiB/s wr, 12 op/s 2023-03-10T02:03:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39891.557309 secs 2023-03-10T02:03:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:03:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:03:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:01 smithi181 ceph-mon[94189]: pgmap v21784: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 867 KiB/s wr, 12 op/s 2023-03-10T02:03:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39891.557309 secs 2023-03-10T02:03:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:03:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:03:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:01 smithi038 ceph-mon[92422]: pgmap v21784: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 867 KiB/s wr, 12 op/s 2023-03-10T02:03:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39891.557309 secs 2023-03-10T02:03:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:03:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:03:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:03 smithi122 ceph-mon[93869]: pgmap v21785: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.5 MiB/s wr, 17 op/s 2023-03-10T02:03:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:03 smithi181 ceph-mon[94189]: pgmap v21785: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.5 MiB/s wr, 17 op/s 2023-03-10T02:03:03.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:03 smithi038 ceph-mon[92422]: pgmap v21785: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.5 MiB/s wr, 17 op/s 2023-03-10T02:03:03.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40355.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:03:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:05 smithi122 ceph-mon[93869]: pgmap v21786: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 884 KiB/s rd, 719 KiB/s wr, 12 op/s 2023-03-10T02:03:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:05 smithi181 ceph-mon[94189]: pgmap v21786: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 884 KiB/s rd, 719 KiB/s wr, 12 op/s 2023-03-10T02:03:05.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:05 smithi038 ceph-mon[92422]: pgmap v21786: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 884 KiB/s rd, 719 KiB/s wr, 12 op/s 2023-03-10T02:03:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39896.557367 secs 2023-03-10T02:03:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39896.557367 secs 2023-03-10T02:03:06.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39896.557367 secs 2023-03-10T02:03:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:07 smithi122 ceph-mon[93869]: pgmap v21787: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 764 KiB/s wr, 18 op/s 2023-03-10T02:03:07.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:07 smithi181 ceph-mon[94189]: pgmap v21787: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 764 KiB/s wr, 18 op/s 2023-03-10T02:03:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:07 smithi038 ceph-mon[92422]: pgmap v21787: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 764 KiB/s wr, 18 op/s 2023-03-10T02:03:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:08 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 39899.020303 secs 2023-03-10T02:03:08.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:08 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 39899.020303 secs 2023-03-10T02:03:08.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40360.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:03:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:08 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 39899.020303 secs 2023-03-10T02:03:09.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:09 smithi038 ceph-mon[92422]: pgmap v21788: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1009 KiB/s rd, 764 KiB/s wr, 14 op/s 2023-03-10T02:03:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:09 smithi122 ceph-mon[93869]: pgmap v21788: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1009 KiB/s rd, 764 KiB/s wr, 14 op/s 2023-03-10T02:03:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:09 smithi181 ceph-mon[94189]: pgmap v21788: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1009 KiB/s rd, 764 KiB/s wr, 14 op/s 2023-03-10T02:03:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:11 smithi122 ceph-mon[93869]: pgmap v21789: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 969 KiB/s rd, 729 KiB/s wr, 24 op/s 2023-03-10T02:03:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39901.557425 secs 2023-03-10T02:03:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:11 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:03:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:11 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:03:11.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:11 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:03:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:11 smithi181 ceph-mon[94189]: pgmap v21789: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 969 KiB/s rd, 729 KiB/s wr, 24 op/s 2023-03-10T02:03:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39901.557425 secs 2023-03-10T02:03:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:11 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:03:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:11 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:03:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:11 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:03:11.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:11 smithi038 ceph-mon[92422]: pgmap v21789: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 969 KiB/s rd, 729 KiB/s wr, 24 op/s 2023-03-10T02:03:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39901.557425 secs 2023-03-10T02:03:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:11 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:03:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:11 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:03:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:11 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:03:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:03:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:03:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:03:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:03:12.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:03:12.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:03:13.793 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40365.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:03:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:13 smithi122 ceph-mon[93869]: pgmap v21790: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1008 KiB/s rd, 766 KiB/s wr, 26 op/s 2023-03-10T02:03:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:03:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:03:13.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:13 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 39904.020430 secs 2023-03-10T02:03:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:13 smithi181 ceph-mon[94189]: pgmap v21790: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1008 KiB/s rd, 766 KiB/s wr, 26 op/s 2023-03-10T02:03:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:03:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:03:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:13 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 39904.020430 secs 2023-03-10T02:03:13.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:13 smithi038 ceph-mon[92422]: pgmap v21790: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1008 KiB/s rd, 766 KiB/s wr, 26 op/s 2023-03-10T02:03:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:03:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:03:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:13 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 39904.020430 secs 2023-03-10T02:03:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:15 smithi122 ceph-mon[93869]: pgmap v21791: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 677 KiB/s rd, 436 KiB/s wr, 22 op/s 2023-03-10T02:03:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:15 smithi181 ceph-mon[94189]: pgmap v21791: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 677 KiB/s rd, 436 KiB/s wr, 22 op/s 2023-03-10T02:03:15.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:15 smithi038 ceph-mon[92422]: pgmap v21791: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 677 KiB/s rd, 436 KiB/s wr, 22 op/s 2023-03-10T02:03:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39906.557504 secs 2023-03-10T02:03:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39906.557504 secs 2023-03-10T02:03:16.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39906.557504 secs 2023-03-10T02:03:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:17 smithi122 ceph-mon[93869]: pgmap v21792: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 876 KiB/s rd, 635 KiB/s wr, 24 op/s 2023-03-10T02:03:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:17 smithi181 ceph-mon[94189]: pgmap v21792: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 876 KiB/s rd, 635 KiB/s wr, 24 op/s 2023-03-10T02:03:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:17 smithi038 ceph-mon[92422]: pgmap v21792: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 876 KiB/s rd, 635 KiB/s wr, 24 op/s 2023-03-10T02:03:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40370.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:03:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:18 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 39909.020492 secs 2023-03-10T02:03:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:18 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 39909.020492 secs 2023-03-10T02:03:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:18 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 39909.020492 secs 2023-03-10T02:03:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:19 smithi122 ceph-mon[93869]: pgmap v21793: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 589 KiB/s wr, 18 op/s 2023-03-10T02:03:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:19 smithi181 ceph-mon[94189]: pgmap v21793: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 589 KiB/s wr, 18 op/s 2023-03-10T02:03:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:19 smithi038 ceph-mon[92422]: pgmap v21793: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 589 KiB/s wr, 18 op/s 2023-03-10T02:03:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:03:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:03:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:03:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:03:20.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:03:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:03:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:21 smithi122 ceph-mon[93869]: pgmap v21794: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 589 KiB/s wr, 18 op/s 2023-03-10T02:03:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39911.557587 secs 2023-03-10T02:03:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:03:21.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:03:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:21 smithi181 ceph-mon[94189]: pgmap v21794: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 589 KiB/s wr, 18 op/s 2023-03-10T02:03:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39911.557587 secs 2023-03-10T02:03:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:03:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:03:21.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:21 smithi038 ceph-mon[92422]: pgmap v21794: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 589 KiB/s wr, 18 op/s 2023-03-10T02:03:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39911.557587 secs 2023-03-10T02:03:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:03:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:03:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40375.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:03:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:23 smithi122 ceph-mon[93869]: pgmap v21795: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 574 KiB/s rd, 571 KiB/s wr, 11 op/s 2023-03-10T02:03:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:23 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 39914.020617 secs 2023-03-10T02:03:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:23 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:03:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:23 smithi181 ceph-mon[94189]: pgmap v21795: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 574 KiB/s rd, 571 KiB/s wr, 11 op/s 2023-03-10T02:03:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:23 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 39914.020617 secs 2023-03-10T02:03:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:23 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:03:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:23 smithi038 ceph-mon[92422]: pgmap v21795: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 574 KiB/s rd, 571 KiB/s wr, 11 op/s 2023-03-10T02:03:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:23 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 39914.020617 secs 2023-03-10T02:03:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:23 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:03:25.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:03:25.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:03:25.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:03:25.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:03:25.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:24 smithi122 ceph-mon[93869]: pgmap v21796: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 536 KiB/s rd, 544 KiB/s wr, 10 op/s 2023-03-10T02:03:25.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:03:25.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:03:25.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:03:25.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:03:25.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:03:25.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:03:25.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:03:25.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:03:25.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:03:25.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:24 smithi181 ceph-mon[94189]: pgmap v21796: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 536 KiB/s rd, 544 KiB/s wr, 10 op/s 2023-03-10T02:03:25.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:03:25.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:03:25.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:03:25.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:03:25.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:03:25.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:03:25.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:03:25.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:03:25.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:03:25.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:24 smithi038 ceph-mon[92422]: pgmap v21796: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 536 KiB/s rd, 544 KiB/s wr, 10 op/s 2023-03-10T02:03:25.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:03:25.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:03:25.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:03:25.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:03:25.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:03:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39916.557667 secs 2023-03-10T02:03:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39916.557667 secs 2023-03-10T02:03:26.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39916.557667 secs 2023-03-10T02:03:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:27 smithi122 ceph-mon[93869]: pgmap v21797: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 501 KiB/s wr, 14 op/s 2023-03-10T02:03:27.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:27 smithi181 ceph-mon[94189]: pgmap v21797: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 501 KiB/s wr, 14 op/s 2023-03-10T02:03:27.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:27 smithi038 ceph-mon[92422]: pgmap v21797: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 501 KiB/s wr, 14 op/s 2023-03-10T02:03:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:28 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:03:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:28 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:03:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39919.020679 secs 2023-03-10T02:03:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:28 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:03:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:28 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:03:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39919.020679 secs 2023-03-10T02:03:28.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:28 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:03:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:28 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:03:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39919.020679 secs 2023-03-10T02:03:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40380.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:03:29.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:29 smithi122 ceph-mon[93869]: pgmap v21798: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 302 KiB/s wr, 11 op/s 2023-03-10T02:03:29.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:29 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:03:29.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:29 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:03:29.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:29 smithi181 ceph-mon[94189]: pgmap v21798: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 302 KiB/s wr, 11 op/s 2023-03-10T02:03:29.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:29 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:03:29.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:29 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:03:29.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:29 smithi038 ceph-mon[92422]: pgmap v21798: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 282 KiB/s rd, 302 KiB/s wr, 11 op/s 2023-03-10T02:03:29.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:29 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:03:29.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:29 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:03:30.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:30 smithi122 ceph-mon[93869]: osdmap e1577: 12 total, 12 up, 12 in 2023-03-10T02:03:30.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:30 smithi181 ceph-mon[94189]: osdmap e1577: 12 total, 12 up, 12 in 2023-03-10T02:03:30.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:30 smithi038 ceph-mon[92422]: osdmap e1577: 12 total, 12 up, 12 in 2023-03-10T02:03:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:31 smithi122 ceph-mon[93869]: pgmap v21800: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 363 KiB/s wr, 13 op/s 2023-03-10T02:03:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39921.557724 secs 2023-03-10T02:03:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:31 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:03:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:31 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:03:31.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:31 smithi181 ceph-mon[94189]: pgmap v21800: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 363 KiB/s wr, 13 op/s 2023-03-10T02:03:31.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39921.557724 secs 2023-03-10T02:03:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:31 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:03:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:31 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:03:31.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:31 smithi038 ceph-mon[92422]: pgmap v21800: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 363 KiB/s wr, 13 op/s 2023-03-10T02:03:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39921.557724 secs 2023-03-10T02:03:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:31 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:03:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:31 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:03:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:33 smithi122 ceph-mon[93869]: pgmap v21801: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 536 KiB/s rd, 719 KiB/s wr, 13 op/s 2023-03-10T02:03:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39924.020736 secs 2023-03-10T02:03:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:33 smithi181 ceph-mon[94189]: pgmap v21801: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 536 KiB/s rd, 719 KiB/s wr, 13 op/s 2023-03-10T02:03:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39924.020736 secs 2023-03-10T02:03:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:33 smithi038 ceph-mon[92422]: pgmap v21801: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 536 KiB/s rd, 719 KiB/s wr, 13 op/s 2023-03-10T02:03:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39924.020736 secs 2023-03-10T02:03:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40385.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:03:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:35 smithi122 ceph-mon[93869]: pgmap v21802: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 535 KiB/s rd, 706 KiB/s wr, 11 op/s 2023-03-10T02:03:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:35 smithi181 ceph-mon[94189]: pgmap v21802: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 535 KiB/s rd, 706 KiB/s wr, 11 op/s 2023-03-10T02:03:35.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:35 smithi038 ceph-mon[92422]: pgmap v21802: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 535 KiB/s rd, 706 KiB/s wr, 11 op/s 2023-03-10T02:03:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39926.557782 secs 2023-03-10T02:03:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:03:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:03:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39926.557782 secs 2023-03-10T02:03:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:03:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:03:36.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39926.557782 secs 2023-03-10T02:03:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:03:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:03:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:37 smithi122 ceph-mon[93869]: pgmap v21803: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 444 KiB/s wr, 9 op/s 2023-03-10T02:03:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:03:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:03:37.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:37 smithi181 ceph-mon[94189]: pgmap v21803: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 444 KiB/s wr, 9 op/s 2023-03-10T02:03:37.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:03:37.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:03:37.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:37 smithi038 ceph-mon[92422]: pgmap v21803: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 444 KiB/s wr, 9 op/s 2023-03-10T02:03:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:03:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:03:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39929.020810 secs 2023-03-10T02:03:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39929.020810 secs 2023-03-10T02:03:38.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39929.020810 secs 2023-03-10T02:03:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40390.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:03:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:39 smithi122 ceph-mon[93869]: pgmap v21804: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 654 KiB/s rd, 652 KiB/s wr, 10 op/s 2023-03-10T02:03:39.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:39 smithi181 ceph-mon[94189]: pgmap v21804: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 654 KiB/s rd, 652 KiB/s wr, 10 op/s 2023-03-10T02:03:39.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:39 smithi038 ceph-mon[92422]: pgmap v21804: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 654 KiB/s rd, 652 KiB/s wr, 10 op/s 2023-03-10T02:03:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:41 smithi122 ceph-mon[93869]: pgmap v21805: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 618 KiB/s rd, 616 KiB/s wr, 9 op/s 2023-03-10T02:03:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39931.557838 secs 2023-03-10T02:03:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:41 smithi122 ceph-mon[93869]: osdmap e1578: 12 total, 12 up, 12 in 2023-03-10T02:03:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:41 smithi181 ceph-mon[94189]: pgmap v21805: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 618 KiB/s rd, 616 KiB/s wr, 9 op/s 2023-03-10T02:03:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39931.557838 secs 2023-03-10T02:03:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:41 smithi181 ceph-mon[94189]: osdmap e1578: 12 total, 12 up, 12 in 2023-03-10T02:03:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:41 smithi038 ceph-mon[92422]: pgmap v21805: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 618 KiB/s rd, 616 KiB/s wr, 9 op/s 2023-03-10T02:03:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39931.557838 secs 2023-03-10T02:03:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:41 smithi038 ceph-mon[92422]: osdmap e1578: 12 total, 12 up, 12 in 2023-03-10T02:03:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:43 smithi122 ceph-mon[93869]: pgmap v21807: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 298 KiB/s wr, 10 op/s 2023-03-10T02:03:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39934.020867 secs 2023-03-10T02:03:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:43 smithi181 ceph-mon[94189]: pgmap v21807: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 298 KiB/s wr, 10 op/s 2023-03-10T02:03:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39934.020867 secs 2023-03-10T02:03:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:43 smithi038 ceph-mon[92422]: pgmap v21807: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 298 KiB/s wr, 10 op/s 2023-03-10T02:03:43.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39934.020867 secs 2023-03-10T02:03:43.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40395.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:03:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:45 smithi122 ceph-mon[93869]: pgmap v21808: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 618 KiB/s rd, 458 KiB/s wr, 10 op/s 2023-03-10T02:03:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:45 smithi181 ceph-mon[94189]: pgmap v21808: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 618 KiB/s rd, 458 KiB/s wr, 10 op/s 2023-03-10T02:03:45.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:45 smithi038 ceph-mon[92422]: pgmap v21808: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 618 KiB/s rd, 458 KiB/s wr, 10 op/s 2023-03-10T02:03:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39936.557901 secs 2023-03-10T02:03:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39936.557901 secs 2023-03-10T02:03:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39936.557901 secs 2023-03-10T02:03:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:47 smithi122 ceph-mon[93869]: pgmap v21809: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 411 KiB/s wr, 8 op/s 2023-03-10T02:03:47.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:47 smithi181 ceph-mon[94189]: pgmap v21809: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 411 KiB/s wr, 8 op/s 2023-03-10T02:03:47.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:47 smithi038 ceph-mon[92422]: pgmap v21809: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 411 KiB/s wr, 8 op/s 2023-03-10T02:03:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39939.020924 secs 2023-03-10T02:03:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39939.020924 secs 2023-03-10T02:03:48.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39939.020924 secs 2023-03-10T02:03:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40400.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:03:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:49 smithi122 ceph-mon[93869]: pgmap v21810: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 203 KiB/s wr, 7 op/s 2023-03-10T02:03:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:03:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:03:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:49 smithi181 ceph-mon[94189]: pgmap v21810: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 203 KiB/s wr, 7 op/s 2023-03-10T02:03:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:03:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:03:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:49 smithi038 ceph-mon[92422]: pgmap v21810: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 203 KiB/s wr, 7 op/s 2023-03-10T02:03:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:03:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:03:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:51 smithi122 ceph-mon[93869]: pgmap v21811: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 407 KiB/s wr, 8 op/s 2023-03-10T02:03:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39941.557955 secs 2023-03-10T02:03:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:51 smithi181 ceph-mon[94189]: pgmap v21811: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 407 KiB/s wr, 8 op/s 2023-03-10T02:03:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39941.557955 secs 2023-03-10T02:03:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:51 smithi038 ceph-mon[92422]: pgmap v21811: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 410 KiB/s rd, 407 KiB/s wr, 8 op/s 2023-03-10T02:03:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39941.557955 secs 2023-03-10T02:03:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:03:52.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:03:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:03:52.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:03:52.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:03:52.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:03:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:53 smithi122 ceph-mon[93869]: pgmap v21812: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 526 KiB/s rd, 523 KiB/s wr, 10 op/s 2023-03-10T02:03:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39944.021010 secs 2023-03-10T02:03:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:53 smithi181 ceph-mon[94189]: pgmap v21812: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 526 KiB/s rd, 523 KiB/s wr, 10 op/s 2023-03-10T02:03:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39944.021010 secs 2023-03-10T02:03:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:53 smithi038 ceph-mon[92422]: pgmap v21812: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 526 KiB/s rd, 523 KiB/s wr, 10 op/s 2023-03-10T02:03:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39944.021010 secs 2023-03-10T02:03:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40405.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:03:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:55 smithi122 ceph-mon[93869]: pgmap v21813: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 511 KiB/s rd, 509 KiB/s wr, 8 op/s 2023-03-10T02:03:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:55 smithi181 ceph-mon[94189]: pgmap v21813: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 511 KiB/s rd, 509 KiB/s wr, 8 op/s 2023-03-10T02:03:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:55 smithi038 ceph-mon[92422]: pgmap v21813: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 511 KiB/s rd, 509 KiB/s wr, 8 op/s 2023-03-10T02:03:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39946.558043 secs 2023-03-10T02:03:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:03:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:03:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39946.558043 secs 2023-03-10T02:03:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:03:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:03:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39946.558043 secs 2023-03-10T02:03:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:03:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:03:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:57 smithi122 ceph-mon[93869]: pgmap v21814: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 410 KiB/s wr, 10 op/s 2023-03-10T02:03:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:03:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:03:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:57 smithi181 ceph-mon[94189]: pgmap v21814: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 410 KiB/s wr, 10 op/s 2023-03-10T02:03:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:03:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:03:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:57 smithi038 ceph-mon[92422]: pgmap v21814: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 410 KiB/s wr, 10 op/s 2023-03-10T02:03:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:03:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:03:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39949.021071 secs 2023-03-10T02:03:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39949.021071 secs 2023-03-10T02:03:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39949.021071 secs 2023-03-10T02:03:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40410.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:03:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:03:59 smithi122 ceph-mon[93869]: pgmap v21815: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 445 KiB/s rd, 442 KiB/s wr, 9 op/s 2023-03-10T02:03:59.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:03:59 smithi181 ceph-mon[94189]: pgmap v21815: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 445 KiB/s rd, 442 KiB/s wr, 9 op/s 2023-03-10T02:03:59.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:03:59 smithi038 ceph-mon[92422]: pgmap v21815: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 445 KiB/s rd, 442 KiB/s wr, 9 op/s 2023-03-10T02:04:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:04:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:04:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:04:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:04:00.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:04:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:04:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:01 smithi122 ceph-mon[93869]: pgmap v21816: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 659 KiB/s rd, 644 KiB/s wr, 11 op/s 2023-03-10T02:04:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39951.558126 secs 2023-03-10T02:04:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:04:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:04:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:01 smithi181 ceph-mon[94189]: pgmap v21816: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 659 KiB/s rd, 644 KiB/s wr, 11 op/s 2023-03-10T02:04:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39951.558126 secs 2023-03-10T02:04:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:04:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:04:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:01 smithi038 ceph-mon[92422]: pgmap v21816: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 659 KiB/s rd, 644 KiB/s wr, 11 op/s 2023-03-10T02:04:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39951.558126 secs 2023-03-10T02:04:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:04:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:04:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:03 smithi122 ceph-mon[93869]: pgmap v21817: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 668 KiB/s rd, 663 KiB/s wr, 15 op/s 2023-03-10T02:04:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39954.021116 secs 2023-03-10T02:04:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:03 smithi181 ceph-mon[94189]: pgmap v21817: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 668 KiB/s rd, 663 KiB/s wr, 15 op/s 2023-03-10T02:04:03.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39954.021116 secs 2023-03-10T02:04:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:03 smithi038 ceph-mon[92422]: pgmap v21817: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 668 KiB/s rd, 663 KiB/s wr, 15 op/s 2023-03-10T02:04:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39954.021116 secs 2023-03-10T02:04:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40415.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:04:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:05 smithi122 ceph-mon[93869]: pgmap v21818: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 802 KiB/s rd, 801 KiB/s wr, 25 op/s 2023-03-10T02:04:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:05 smithi181 ceph-mon[94189]: pgmap v21818: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 802 KiB/s rd, 801 KiB/s wr, 25 op/s 2023-03-10T02:04:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:05 smithi038 ceph-mon[92422]: pgmap v21818: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 802 KiB/s rd, 801 KiB/s wr, 25 op/s 2023-03-10T02:04:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39956.558215 secs 2023-03-10T02:04:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39956.558215 secs 2023-03-10T02:04:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39956.558215 secs 2023-03-10T02:04:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:07 smithi122 ceph-mon[93869]: pgmap v21819: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 765 KiB/s rd, 763 KiB/s wr, 27 op/s 2023-03-10T02:04:07.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:07 smithi181 ceph-mon[94189]: pgmap v21819: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 765 KiB/s rd, 763 KiB/s wr, 27 op/s 2023-03-10T02:04:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:07 smithi038 ceph-mon[92422]: pgmap v21819: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 765 KiB/s rd, 763 KiB/s wr, 27 op/s 2023-03-10T02:04:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:04:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:04:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39959.021174 secs 2023-03-10T02:04:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:04:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:04:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39959.021174 secs 2023-03-10T02:04:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:04:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:04:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39959.021174 secs 2023-03-10T02:04:08.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40420.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:04:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:09 smithi122 ceph-mon[93869]: pgmap v21820: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 877 KiB/s rd, 879 KiB/s wr, 25 op/s 2023-03-10T02:04:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:04:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:04:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:09 smithi181 ceph-mon[94189]: pgmap v21820: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 877 KiB/s rd, 879 KiB/s wr, 25 op/s 2023-03-10T02:04:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:04:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:04:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:09 smithi038 ceph-mon[92422]: pgmap v21820: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 877 KiB/s rd, 879 KiB/s wr, 25 op/s 2023-03-10T02:04:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:04:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:04:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:11 smithi122 ceph-mon[93869]: pgmap v21821: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 25 op/s 2023-03-10T02:04:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39961.558275 secs 2023-03-10T02:04:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:11 smithi181 ceph-mon[94189]: pgmap v21821: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 25 op/s 2023-03-10T02:04:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39961.558275 secs 2023-03-10T02:04:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:11 smithi038 ceph-mon[92422]: pgmap v21821: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 25 op/s 2023-03-10T02:04:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39961.558275 secs 2023-03-10T02:04:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:13 smithi122 ceph-mon[93869]: pgmap v21822: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 26 op/s 2023-03-10T02:04:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39964.021248 secs 2023-03-10T02:04:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:13 smithi181 ceph-mon[94189]: pgmap v21822: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 26 op/s 2023-03-10T02:04:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39964.021248 secs 2023-03-10T02:04:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:13 smithi038 ceph-mon[92422]: pgmap v21822: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 26 op/s 2023-03-10T02:04:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39964.021248 secs 2023-03-10T02:04:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40425.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:04:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:15 smithi122 ceph-mon[93869]: pgmap v21823: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 873 KiB/s rd, 875 KiB/s wr, 22 op/s 2023-03-10T02:04:15.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:15 smithi181 ceph-mon[94189]: pgmap v21823: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 873 KiB/s rd, 875 KiB/s wr, 22 op/s 2023-03-10T02:04:15.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:15 smithi038 ceph-mon[92422]: pgmap v21823: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 873 KiB/s rd, 875 KiB/s wr, 22 op/s 2023-03-10T02:04:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39966.558334 secs 2023-03-10T02:04:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:04:16.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:04:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39966.558334 secs 2023-03-10T02:04:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:04:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:04:16.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39966.558334 secs 2023-03-10T02:04:16.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:04:16.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:04:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:17 smithi122 ceph-mon[93869]: pgmap v21824: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 569 KiB/s wr, 11 op/s 2023-03-10T02:04:17.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:04:17.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:04:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:17 smithi181 ceph-mon[94189]: pgmap v21824: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 569 KiB/s wr, 11 op/s 2023-03-10T02:04:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:04:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:04:17.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:17 smithi038 ceph-mon[92422]: pgmap v21824: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 569 KiB/s wr, 11 op/s 2023-03-10T02:04:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:04:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:04:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39969.021322 secs 2023-03-10T02:04:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39969.021322 secs 2023-03-10T02:04:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39969.021322 secs 2023-03-10T02:04:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40430.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:04:19.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:19 smithi122 ceph-mon[93869]: pgmap v21825: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 571 KiB/s rd, 569 KiB/s wr, 10 op/s 2023-03-10T02:04:19.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:19 smithi181 ceph-mon[94189]: pgmap v21825: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 571 KiB/s rd, 569 KiB/s wr, 10 op/s 2023-03-10T02:04:19.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:19 smithi038 ceph-mon[92422]: pgmap v21825: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 571 KiB/s rd, 569 KiB/s wr, 10 op/s 2023-03-10T02:04:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:04:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:04:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:04:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:04:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:04:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:04:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:21 smithi122 ceph-mon[93869]: pgmap v21826: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 457 KiB/s rd, 465 KiB/s wr, 12 op/s 2023-03-10T02:04:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39971.558418 secs 2023-03-10T02:04:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:04:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:04:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:21 smithi181 ceph-mon[94189]: pgmap v21826: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 457 KiB/s rd, 465 KiB/s wr, 12 op/s 2023-03-10T02:04:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39971.558418 secs 2023-03-10T02:04:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:04:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:04:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:21 smithi038 ceph-mon[92422]: pgmap v21826: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 457 KiB/s rd, 465 KiB/s wr, 12 op/s 2023-03-10T02:04:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39971.558418 secs 2023-03-10T02:04:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:04:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:04:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:23 smithi122 ceph-mon[93869]: pgmap v21827: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 728 KiB/s rd, 746 KiB/s wr, 15 op/s 2023-03-10T02:04:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39974.021423 secs 2023-03-10T02:04:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:23 smithi181 ceph-mon[94189]: pgmap v21827: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 728 KiB/s rd, 746 KiB/s wr, 15 op/s 2023-03-10T02:04:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39974.021423 secs 2023-03-10T02:04:23.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:23 smithi038 ceph-mon[92422]: pgmap v21827: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 728 KiB/s rd, 746 KiB/s wr, 15 op/s 2023-03-10T02:04:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39974.021423 secs 2023-03-10T02:04:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40435.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:04:24.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:24 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:04:24.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:24 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:04:24.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:24 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:04:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:25 smithi122 ceph-mon[93869]: pgmap v21828: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 548 KiB/s wr, 13 op/s 2023-03-10T02:04:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:25 smithi122 ceph-mon[93869]: osdmap e1579: 12 total, 12 up, 12 in 2023-03-10T02:04:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:04:25.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:04:25.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:04:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:25 smithi181 ceph-mon[94189]: pgmap v21828: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 548 KiB/s wr, 13 op/s 2023-03-10T02:04:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:25 smithi181 ceph-mon[94189]: osdmap e1579: 12 total, 12 up, 12 in 2023-03-10T02:04:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:04:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:04:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:04:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:25 smithi038 ceph-mon[92422]: pgmap v21828: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 548 KiB/s wr, 13 op/s 2023-03-10T02:04:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:25 smithi038 ceph-mon[92422]: osdmap e1579: 12 total, 12 up, 12 in 2023-03-10T02:04:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:04:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:04:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:04:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39976.558477 secs 2023-03-10T02:04:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39976.558477 secs 2023-03-10T02:04:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39976.558477 secs 2023-03-10T02:04:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:27 smithi122 ceph-mon[93869]: pgmap v21830: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 837 KiB/s rd, 860 KiB/s wr, 14 op/s 2023-03-10T02:04:27.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:27 smithi181 ceph-mon[94189]: pgmap v21830: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 837 KiB/s rd, 860 KiB/s wr, 14 op/s 2023-03-10T02:04:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:27 smithi038 ceph-mon[92422]: pgmap v21830: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 837 KiB/s rd, 860 KiB/s wr, 14 op/s 2023-03-10T02:04:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39979.021507 secs 2023-03-10T02:04:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40440.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:04:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39979.021507 secs 2023-03-10T02:04:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39979.021507 secs 2023-03-10T02:04:29.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:29 smithi038 ceph-mon[92422]: pgmap v21831: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 837 KiB/s rd, 860 KiB/s wr, 14 op/s 2023-03-10T02:04:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:29 smithi122 ceph-mon[93869]: pgmap v21831: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 837 KiB/s rd, 860 KiB/s wr, 14 op/s 2023-03-10T02:04:29.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:29 smithi181 ceph-mon[94189]: pgmap v21831: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 837 KiB/s rd, 860 KiB/s wr, 14 op/s 2023-03-10T02:04:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:31 smithi038 ceph-mon[92422]: pgmap v21832: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 836 KiB/s rd, 848 KiB/s wr, 11 op/s 2023-03-10T02:04:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39981.558580 secs 2023-03-10T02:04:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:31 smithi122 ceph-mon[93869]: pgmap v21832: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 836 KiB/s rd, 848 KiB/s wr, 11 op/s 2023-03-10T02:04:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39981.558580 secs 2023-03-10T02:04:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:31 smithi181 ceph-mon[94189]: pgmap v21832: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 836 KiB/s rd, 848 KiB/s wr, 11 op/s 2023-03-10T02:04:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39981.558580 secs 2023-03-10T02:04:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:33 smithi038 ceph-mon[92422]: pgmap v21833: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 266 KiB/s wr, 9 op/s 2023-03-10T02:04:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39984.021623 secs 2023-03-10T02:04:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40445.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:04:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:33 smithi122 ceph-mon[93869]: pgmap v21833: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 266 KiB/s wr, 9 op/s 2023-03-10T02:04:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39984.021623 secs 2023-03-10T02:04:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:33 smithi181 ceph-mon[94189]: pgmap v21833: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 266 KiB/s wr, 9 op/s 2023-03-10T02:04:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39984.021623 secs 2023-03-10T02:04:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:35 smithi122 ceph-mon[93869]: pgmap v21834: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 307 KiB/s wr, 8 op/s 2023-03-10T02:04:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:35 smithi122 ceph-mon[93869]: osdmap e1580: 12 total, 12 up, 12 in 2023-03-10T02:04:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:35 smithi181 ceph-mon[94189]: pgmap v21834: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 307 KiB/s wr, 8 op/s 2023-03-10T02:04:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:35 smithi181 ceph-mon[94189]: osdmap e1580: 12 total, 12 up, 12 in 2023-03-10T02:04:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:35 smithi038 ceph-mon[92422]: pgmap v21834: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 307 KiB/s wr, 8 op/s 2023-03-10T02:04:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:35 smithi038 ceph-mon[92422]: osdmap e1580: 12 total, 12 up, 12 in 2023-03-10T02:04:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39986.558638 secs 2023-03-10T02:04:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:04:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:04:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39986.558638 secs 2023-03-10T02:04:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:04:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:04:36.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39986.558638 secs 2023-03-10T02:04:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:04:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:04:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:37 smithi122 ceph-mon[93869]: pgmap v21836: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 930 KiB/s rd, 923 KiB/s wr, 10 op/s 2023-03-10T02:04:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:04:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:04:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:37 smithi181 ceph-mon[94189]: pgmap v21836: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 930 KiB/s rd, 923 KiB/s wr, 10 op/s 2023-03-10T02:04:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:04:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:04:37.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:37 smithi038 ceph-mon[92422]: pgmap v21836: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 930 KiB/s rd, 923 KiB/s wr, 10 op/s 2023-03-10T02:04:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:04:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:04:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40450.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:04:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39989.021731 secs 2023-03-10T02:04:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39989.021731 secs 2023-03-10T02:04:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39989.021731 secs 2023-03-10T02:04:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:39 smithi122 ceph-mon[93869]: pgmap v21837: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 930 KiB/s rd, 923 KiB/s wr, 10 op/s 2023-03-10T02:04:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:39 smithi181 ceph-mon[94189]: pgmap v21837: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 930 KiB/s rd, 923 KiB/s wr, 10 op/s 2023-03-10T02:04:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:39 smithi038 ceph-mon[92422]: pgmap v21837: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 930 KiB/s rd, 923 KiB/s wr, 10 op/s 2023-03-10T02:04:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:04:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:04:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:04:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:04:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:04:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:04:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:41 smithi122 ceph-mon[93869]: pgmap v21838: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 908 KiB/s rd, 900 KiB/s wr, 11 op/s 2023-03-10T02:04:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39991.558692 secs 2023-03-10T02:04:41.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:04:41.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:04:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:41 smithi181 ceph-mon[94189]: pgmap v21838: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 908 KiB/s rd, 900 KiB/s wr, 11 op/s 2023-03-10T02:04:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39991.558692 secs 2023-03-10T02:04:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:04:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:04:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:41 smithi038 ceph-mon[92422]: pgmap v21838: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 908 KiB/s rd, 900 KiB/s wr, 11 op/s 2023-03-10T02:04:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39991.558692 secs 2023-03-10T02:04:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:04:42.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:04:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40455.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:04:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:43 smithi122 ceph-mon[93869]: pgmap v21839: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 942 KiB/s rd, 880 KiB/s wr, 11 op/s 2023-03-10T02:04:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39994.021804 secs 2023-03-10T02:04:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:43 smithi181 ceph-mon[94189]: pgmap v21839: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 942 KiB/s rd, 880 KiB/s wr, 11 op/s 2023-03-10T02:04:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39994.021804 secs 2023-03-10T02:04:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:43 smithi038 ceph-mon[92422]: pgmap v21839: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 942 KiB/s rd, 880 KiB/s wr, 11 op/s 2023-03-10T02:04:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39994.021804 secs 2023-03-10T02:04:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:45 smithi122 ceph-mon[93869]: pgmap v21840: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 12 op/s 2023-03-10T02:04:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:45 smithi181 ceph-mon[94189]: pgmap v21840: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 12 op/s 2023-03-10T02:04:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:45 smithi038 ceph-mon[92422]: pgmap v21840: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 12 op/s 2023-03-10T02:04:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39996.558750 secs 2023-03-10T02:04:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39996.558750 secs 2023-03-10T02:04:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39996.558750 secs 2023-03-10T02:04:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:47 smithi122 ceph-mon[93869]: pgmap v21841: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 1.1 MiB/s wr, 15 op/s 2023-03-10T02:04:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:47 smithi181 ceph-mon[94189]: pgmap v21841: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 1.1 MiB/s wr, 15 op/s 2023-03-10T02:04:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:47 smithi038 ceph-mon[92422]: pgmap v21841: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 1.1 MiB/s wr, 15 op/s 2023-03-10T02:04:48.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40460.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:04:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:04:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:04:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 39999.021862 secs 2023-03-10T02:04:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:04:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:04:48.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 39999.021862 secs 2023-03-10T02:04:48.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:04:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:04:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 39999.021862 secs 2023-03-10T02:04:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:49 smithi122 ceph-mon[93869]: pgmap v21842: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 618 KiB/s wr, 9 op/s 2023-03-10T02:04:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:04:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:04:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:49 smithi181 ceph-mon[94189]: pgmap v21842: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 618 KiB/s wr, 9 op/s 2023-03-10T02:04:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:04:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:04:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:49 smithi038 ceph-mon[92422]: pgmap v21842: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 618 KiB/s wr, 9 op/s 2023-03-10T02:04:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:04:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:04:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:51 smithi122 ceph-mon[93869]: pgmap v21843: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 442 KiB/s rd, 619 KiB/s wr, 11 op/s 2023-03-10T02:04:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40001.558807 secs 2023-03-10T02:04:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:51 smithi181 ceph-mon[94189]: pgmap v21843: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 442 KiB/s rd, 619 KiB/s wr, 11 op/s 2023-03-10T02:04:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40001.558807 secs 2023-03-10T02:04:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:51 smithi038 ceph-mon[92422]: pgmap v21843: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 442 KiB/s rd, 619 KiB/s wr, 11 op/s 2023-03-10T02:04:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40001.558807 secs 2023-03-10T02:04:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40465.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:04:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:53 smithi122 ceph-mon[93869]: pgmap v21844: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 775 KiB/s wr, 13 op/s 2023-03-10T02:04:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:53 smithi181 ceph-mon[94189]: pgmap v21844: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 775 KiB/s wr, 13 op/s 2023-03-10T02:04:53.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:53 smithi038 ceph-mon[92422]: pgmap v21844: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 775 KiB/s wr, 13 op/s 2023-03-10T02:04:54.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:54 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T02:04:54.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:54 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T02:04:54.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:54 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T02:04:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:55 smithi122 ceph-mon[93869]: pgmap v21845: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 639 KiB/s rd, 809 KiB/s wr, 11 op/s 2023-03-10T02:04:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:55 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:04:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:04:55.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:04:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:55 smithi181 ceph-mon[94189]: pgmap v21845: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 639 KiB/s rd, 809 KiB/s wr, 11 op/s 2023-03-10T02:04:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:55 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:04:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:04:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:04:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:55 smithi038 ceph-mon[92422]: pgmap v21845: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 639 KiB/s rd, 809 KiB/s wr, 11 op/s 2023-03-10T02:04:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:55 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:04:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:04:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:04:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40006.558867 secs 2023-03-10T02:04:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:04:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:04:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40006.558867 secs 2023-03-10T02:04:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:04:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:04:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40006.558867 secs 2023-03-10T02:04:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:04:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:04:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:57 smithi122 ceph-mon[93869]: pgmap v21846: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 763 KiB/s rd, 740 KiB/s wr, 14 op/s 2023-03-10T02:04:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:04:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:04:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:57 smithi181 ceph-mon[94189]: pgmap v21846: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 763 KiB/s rd, 740 KiB/s wr, 14 op/s 2023-03-10T02:04:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:04:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:04:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:57 smithi038 ceph-mon[92422]: pgmap v21846: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 763 KiB/s rd, 740 KiB/s wr, 14 op/s 2023-03-10T02:04:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:04:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:04:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40470.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:04:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:04:59 smithi122 ceph-mon[93869]: pgmap v21847: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 464 KiB/s wr, 10 op/s 2023-03-10T02:04:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:04:59 smithi181 ceph-mon[94189]: pgmap v21847: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 464 KiB/s wr, 10 op/s 2023-03-10T02:04:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:04:59 smithi038 ceph-mon[92422]: pgmap v21847: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 464 KiB/s wr, 10 op/s 2023-03-10T02:05:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:05:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:05:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:05:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:05:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:05:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:05:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:01 smithi122 ceph-mon[93869]: pgmap v21848: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 756 KiB/s rd, 499 KiB/s wr, 11 op/s 2023-03-10T02:05:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40011.558922 secs 2023-03-10T02:05:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:05:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:05:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:01 smithi181 ceph-mon[94189]: pgmap v21848: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 756 KiB/s rd, 499 KiB/s wr, 11 op/s 2023-03-10T02:05:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40011.558922 secs 2023-03-10T02:05:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:05:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:05:01.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:01 smithi038 ceph-mon[92422]: pgmap v21848: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 756 KiB/s rd, 499 KiB/s wr, 11 op/s 2023-03-10T02:05:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40011.558922 secs 2023-03-10T02:05:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:05:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:05:02.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:02 smithi122 ceph-mon[93869]: pgmap v21849: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 15 op/s 2023-03-10T02:05:02.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:02 smithi181 ceph-mon[94189]: pgmap v21849: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 15 op/s 2023-03-10T02:05:02.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:02 smithi038 ceph-mon[92422]: pgmap v21849: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 15 op/s 2023-03-10T02:05:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40475.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:05:05.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:04 smithi038 ceph-mon[92422]: pgmap v21850: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 23 op/s 2023-03-10T02:05:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:04 smithi122 ceph-mon[93869]: pgmap v21850: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 23 op/s 2023-03-10T02:05:05.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:04 smithi181 ceph-mon[94189]: pgmap v21850: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 23 op/s 2023-03-10T02:05:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40016.559006 secs 2023-03-10T02:05:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40016.559006 secs 2023-03-10T02:05:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40016.559006 secs 2023-03-10T02:05:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:07 smithi122 ceph-mon[93869]: pgmap v21851: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1011 KiB/s wr, 27 op/s 2023-03-10T02:05:07.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:07 smithi181 ceph-mon[94189]: pgmap v21851: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1011 KiB/s wr, 27 op/s 2023-03-10T02:05:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:07 smithi038 ceph-mon[92422]: pgmap v21851: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1011 KiB/s wr, 27 op/s 2023-03-10T02:05:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:05:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:05:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:05:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:05:08.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:05:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:05:08.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40480.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:05:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:09 smithi122 ceph-mon[93869]: pgmap v21852: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 754 KiB/s wr, 24 op/s 2023-03-10T02:05:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:05:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:05:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:09 smithi181 ceph-mon[94189]: pgmap v21852: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 754 KiB/s wr, 24 op/s 2023-03-10T02:05:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:05:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:05:09.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:09 smithi038 ceph-mon[92422]: pgmap v21852: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 754 KiB/s wr, 24 op/s 2023-03-10T02:05:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:05:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:05:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:11 smithi122 ceph-mon[93869]: pgmap v21853: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 794 KiB/s wr, 25 op/s 2023-03-10T02:05:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40021.559065 secs 2023-03-10T02:05:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:11 smithi181 ceph-mon[94189]: pgmap v21853: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 794 KiB/s wr, 25 op/s 2023-03-10T02:05:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40021.559065 secs 2023-03-10T02:05:11.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:11 smithi038 ceph-mon[92422]: pgmap v21853: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 794 KiB/s wr, 25 op/s 2023-03-10T02:05:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40021.559065 secs 2023-03-10T02:05:12.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:05:12.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:05:12.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:05:12.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:05:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:05:12.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:05:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:13 smithi122 ceph-mon[93869]: pgmap v21854: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 759 KiB/s wr, 26 op/s 2023-03-10T02:05:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:13 smithi181 ceph-mon[94189]: pgmap v21854: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 759 KiB/s wr, 26 op/s 2023-03-10T02:05:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:13 smithi038 ceph-mon[92422]: pgmap v21854: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 759 KiB/s wr, 26 op/s 2023-03-10T02:05:13.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40485.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:05:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:15 smithi122 ceph-mon[93869]: pgmap v21855: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 570 KiB/s rd, 97 KiB/s wr, 20 op/s 2023-03-10T02:05:15.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:15 smithi181 ceph-mon[94189]: pgmap v21855: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 570 KiB/s rd, 97 KiB/s wr, 20 op/s 2023-03-10T02:05:15.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:15 smithi038 ceph-mon[92422]: pgmap v21855: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 570 KiB/s rd, 97 KiB/s wr, 20 op/s 2023-03-10T02:05:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40026.559123 secs 2023-03-10T02:05:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40026.559123 secs 2023-03-10T02:05:16.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40026.559123 secs 2023-03-10T02:05:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:17 smithi122 ceph-mon[93869]: pgmap v21856: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 726 KiB/s rd, 247 KiB/s wr, 11 op/s 2023-03-10T02:05:17.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:17 smithi181 ceph-mon[94189]: pgmap v21856: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 726 KiB/s rd, 247 KiB/s wr, 11 op/s 2023-03-10T02:05:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:17 smithi038 ceph-mon[92422]: pgmap v21856: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 726 KiB/s rd, 247 KiB/s wr, 11 op/s 2023-03-10T02:05:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40490.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:05:19.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:19 smithi122 ceph-mon[93869]: pgmap v21857: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 725 KiB/s rd, 281 KiB/s wr, 9 op/s 2023-03-10T02:05:19.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:19 smithi181 ceph-mon[94189]: pgmap v21857: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 725 KiB/s rd, 281 KiB/s wr, 9 op/s 2023-03-10T02:05:19.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:19 smithi038 ceph-mon[92422]: pgmap v21857: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 725 KiB/s rd, 281 KiB/s wr, 9 op/s 2023-03-10T02:05:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:05:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:05:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:05:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:05:20.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:05:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:05:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:21 smithi122 ceph-mon[93869]: pgmap v21858: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 387 KiB/s wr, 9 op/s 2023-03-10T02:05:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40031.559183 secs 2023-03-10T02:05:21.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:05:21.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:05:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:21 smithi181 ceph-mon[94189]: pgmap v21858: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 387 KiB/s wr, 9 op/s 2023-03-10T02:05:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40031.559183 secs 2023-03-10T02:05:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:05:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:05:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:21 smithi038 ceph-mon[92422]: pgmap v21858: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 387 KiB/s wr, 9 op/s 2023-03-10T02:05:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40031.559183 secs 2023-03-10T02:05:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:05:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:05:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:23 smithi122 ceph-mon[93869]: pgmap v21859: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 553 KiB/s rd, 605 KiB/s wr, 12 op/s 2023-03-10T02:05:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:23 smithi181 ceph-mon[94189]: pgmap v21859: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 553 KiB/s rd, 605 KiB/s wr, 12 op/s 2023-03-10T02:05:23.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:23 smithi038 ceph-mon[92422]: pgmap v21859: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 553 KiB/s rd, 605 KiB/s wr, 12 op/s 2023-03-10T02:05:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40495.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:05:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:25 smithi122 ceph-mon[93869]: pgmap v21860: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 606 KiB/s wr, 10 op/s 2023-03-10T02:05:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:25 smithi122 ceph-mon[93869]: osdmap e1581: 12 total, 12 up, 12 in 2023-03-10T02:05:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:25 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:05:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:25 smithi181 ceph-mon[94189]: pgmap v21860: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 606 KiB/s wr, 10 op/s 2023-03-10T02:05:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:25 smithi181 ceph-mon[94189]: osdmap e1581: 12 total, 12 up, 12 in 2023-03-10T02:05:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:25 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:05:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:25 smithi038 ceph-mon[92422]: pgmap v21860: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 606 KiB/s wr, 10 op/s 2023-03-10T02:05:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:25 smithi038 ceph-mon[92422]: osdmap e1581: 12 total, 12 up, 12 in 2023-03-10T02:05:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:25 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:05:26.684 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40036.559249 secs 2023-03-10T02:05:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40036.559249 secs 2023-03-10T02:05:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40036.559249 secs 2023-03-10T02:05:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:27 smithi122 ceph-mon[93869]: pgmap v21862: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 582 KiB/s rd, 606 KiB/s wr, 12 op/s 2023-03-10T02:05:27.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:27 smithi181 ceph-mon[94189]: pgmap v21862: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 582 KiB/s rd, 606 KiB/s wr, 12 op/s 2023-03-10T02:05:27.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:27 smithi038 ceph-mon[92422]: pgmap v21862: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 582 KiB/s rd, 606 KiB/s wr, 12 op/s 2023-03-10T02:05:28.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40500.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:05:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:05:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:05:28.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:05:28.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:05:28.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:28 smithi122 ceph-mon[93869]: pgmap v21863: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 671 KiB/s rd, 684 KiB/s wr, 11 op/s 2023-03-10T02:05:28.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:05:28.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:05:28.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:05:28.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:05:28.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:05:28.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:28 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-10T02:05:28.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:28 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-10T02:05:28.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:05:28.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:28 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:05:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:05:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:05:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:05:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:05:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:28 smithi181 ceph-mon[94189]: pgmap v21863: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 671 KiB/s rd, 684 KiB/s wr, 11 op/s 2023-03-10T02:05:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:05:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:05:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:05:28.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:05:28.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:05:28.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:28 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-10T02:05:28.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:28 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-10T02:05:28.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:05:28.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:28 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:05:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:05:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:05:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:05:29.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:05:29.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:28 smithi038 ceph-mon[92422]: pgmap v21863: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 671 KiB/s rd, 684 KiB/s wr, 11 op/s 2023-03-10T02:05:29.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:05:29.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:05:29.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:05:29.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:05:29.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:05:29.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:28 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-10T02:05:29.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:28 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-10T02:05:29.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:05:29.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:28 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:05:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:05:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:05:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:05:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:05:30.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:29 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-10T02:05:30.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:29 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-10T02:05:30.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:05:30.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:05:30.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:05:30.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:05:30.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:05:30.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:05:30.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:05:30.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:29 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-10T02:05:30.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:29 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-10T02:05:30.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:05:30.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:05:30.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:05:30.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:05:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:05:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:05:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:05:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:29 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-10T02:05:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:29 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-10T02:05:30.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:05:30.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:05:30.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:05:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:31 smithi122 ceph-mon[93869]: pgmap v21864: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 557 KiB/s wr, 10 op/s 2023-03-10T02:05:31.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40041.559307 secs 2023-03-10T02:05:31.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:31 smithi181 ceph-mon[94189]: pgmap v21864: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 557 KiB/s wr, 10 op/s 2023-03-10T02:05:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40041.559307 secs 2023-03-10T02:05:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:31 smithi038 ceph-mon[92422]: pgmap v21864: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 557 KiB/s wr, 10 op/s 2023-03-10T02:05:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40041.559307 secs 2023-03-10T02:05:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:33 smithi122 ceph-mon[93869]: pgmap v21865: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 247 KiB/s wr, 8 op/s 2023-03-10T02:05:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:33 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:05:33.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:33 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:05:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:33 smithi181 ceph-mon[94189]: pgmap v21865: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 247 KiB/s wr, 8 op/s 2023-03-10T02:05:33.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:33 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:05:33.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:33 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:05:33.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:33 smithi038 ceph-mon[92422]: pgmap v21865: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 248 KiB/s rd, 247 KiB/s wr, 8 op/s 2023-03-10T02:05:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:33 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:05:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:33 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:05:33.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40505.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:05:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:35 smithi122 ceph-mon[93869]: pgmap v21866: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 602 KiB/s wr, 9 op/s 2023-03-10T02:05:35.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:35 smithi122 ceph-mon[93869]: osdmap e1582: 12 total, 12 up, 12 in 2023-03-10T02:05:35.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:35 smithi181 ceph-mon[94189]: pgmap v21866: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 602 KiB/s wr, 9 op/s 2023-03-10T02:05:35.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:35 smithi181 ceph-mon[94189]: osdmap e1582: 12 total, 12 up, 12 in 2023-03-10T02:05:35.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:35 smithi038 ceph-mon[92422]: pgmap v21866: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 602 KiB/s wr, 9 op/s 2023-03-10T02:05:35.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:35 smithi038 ceph-mon[92422]: osdmap e1582: 12 total, 12 up, 12 in 2023-03-10T02:05:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40046.559368 secs 2023-03-10T02:05:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:05:36.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:05:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40046.559368 secs 2023-03-10T02:05:36.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:05:36.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:05:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40046.559368 secs 2023-03-10T02:05:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:05:36.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:05:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:37 smithi122 ceph-mon[93869]: pgmap v21868: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 488 KiB/s wr, 8 op/s 2023-03-10T02:05:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:05:37.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:05:37.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:37 smithi181 ceph-mon[94189]: pgmap v21868: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 488 KiB/s wr, 8 op/s 2023-03-10T02:05:37.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:05:37.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:05:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:37 smithi038 ceph-mon[92422]: pgmap v21868: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 488 KiB/s wr, 8 op/s 2023-03-10T02:05:37.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:05:37.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:05:38.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40049.022656 secs 2023-03-10T02:05:38.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40049.022656 secs 2023-03-10T02:05:38.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40049.022656 secs 2023-03-10T02:05:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40510.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:05:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:39 smithi122 ceph-mon[93869]: pgmap v21869: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 706 KiB/s rd, 772 KiB/s wr, 10 op/s 2023-03-10T02:05:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:39 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:05:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:39 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:05:39.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:39 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:05:39.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:39 smithi181 ceph-mon[94189]: pgmap v21869: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 706 KiB/s rd, 772 KiB/s wr, 10 op/s 2023-03-10T02:05:39.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:39 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:05:39.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:39 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:05:39.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:39 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:05:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:39 smithi038 ceph-mon[92422]: pgmap v21869: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 706 KiB/s rd, 772 KiB/s wr, 10 op/s 2023-03-10T02:05:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:39 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:05:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:39 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:05:39.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:39 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:05:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:41 smithi122 ceph-mon[93869]: pgmap v21870: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 706 KiB/s rd, 771 KiB/s wr, 10 op/s 2023-03-10T02:05:41.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40051.559424 secs 2023-03-10T02:05:41.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:41 smithi181 ceph-mon[94189]: pgmap v21870: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 706 KiB/s rd, 771 KiB/s wr, 10 op/s 2023-03-10T02:05:41.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40051.559424 secs 2023-03-10T02:05:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:41 smithi038 ceph-mon[92422]: pgmap v21870: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 706 KiB/s rd, 771 KiB/s wr, 10 op/s 2023-03-10T02:05:41.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40051.559424 secs 2023-03-10T02:05:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:43 smithi122 ceph-mon[93869]: pgmap v21871: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 12 op/s 2023-03-10T02:05:43.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40054.022715 secs 2023-03-10T02:05:43.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:43 smithi181 ceph-mon[94189]: pgmap v21871: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 12 op/s 2023-03-10T02:05:43.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40054.022715 secs 2023-03-10T02:05:43.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:43 smithi038 ceph-mon[92422]: pgmap v21871: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 12 op/s 2023-03-10T02:05:43.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40054.022715 secs 2023-03-10T02:05:43.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40515.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:05:45.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:45 smithi122 ceph-mon[93869]: pgmap v21872: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 715 KiB/s wr, 10 op/s 2023-03-10T02:05:45.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:45 smithi181 ceph-mon[94189]: pgmap v21872: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 715 KiB/s wr, 10 op/s 2023-03-10T02:05:45.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:45 smithi038 ceph-mon[92422]: pgmap v21872: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 715 KiB/s wr, 10 op/s 2023-03-10T02:05:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40056.559487 secs 2023-03-10T02:05:46.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40056.559487 secs 2023-03-10T02:05:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40056.559487 secs 2023-03-10T02:05:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:47 smithi122 ceph-mon[93869]: pgmap v21873: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 13 op/s 2023-03-10T02:05:47.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:47 smithi181 ceph-mon[94189]: pgmap v21873: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 13 op/s 2023-03-10T02:05:47.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:47 smithi038 ceph-mon[92422]: pgmap v21873: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 13 op/s 2023-03-10T02:05:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:05:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:05:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:05:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:05:48.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:05:48.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:05:48.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40520.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:05:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:49 smithi122 ceph-mon[93869]: pgmap v21874: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 978 KiB/s wr, 11 op/s 2023-03-10T02:05:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:05:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:05:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:49 smithi181 ceph-mon[94189]: pgmap v21874: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 978 KiB/s wr, 11 op/s 2023-03-10T02:05:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:05:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:05:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:49 smithi038 ceph-mon[92422]: pgmap v21874: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 978 KiB/s wr, 11 op/s 2023-03-10T02:05:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:05:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:05:50.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:50 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T02:05:50.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:50 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T02:05:50.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:50 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T02:05:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:51 smithi122 ceph-mon[93869]: pgmap v21875: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 724 KiB/s rd, 631 KiB/s wr, 8 op/s 2023-03-10T02:05:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40061.559543 secs 2023-03-10T02:05:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:51 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:05:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:05:51.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:05:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:51 smithi181 ceph-mon[94189]: pgmap v21875: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 724 KiB/s rd, 631 KiB/s wr, 8 op/s 2023-03-10T02:05:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40061.559543 secs 2023-03-10T02:05:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:51 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:05:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:05:51.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:05:51.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:51 smithi038 ceph-mon[92422]: pgmap v21875: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 724 KiB/s rd, 631 KiB/s wr, 8 op/s 2023-03-10T02:05:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40061.559543 secs 2023-03-10T02:05:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:51 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:05:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:05:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:05:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:53 smithi122 ceph-mon[93869]: pgmap v21876: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 934 KiB/s rd, 840 KiB/s wr, 13 op/s 2023-03-10T02:05:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:53 smithi181 ceph-mon[94189]: pgmap v21876: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 934 KiB/s rd, 840 KiB/s wr, 13 op/s 2023-03-10T02:05:53.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:53 smithi038 ceph-mon[92422]: pgmap v21876: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 934 KiB/s rd, 840 KiB/s wr, 13 op/s 2023-03-10T02:05:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40525.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:05:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:55 smithi122 ceph-mon[93869]: pgmap v21877: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 592 KiB/s rd, 590 KiB/s wr, 9 op/s 2023-03-10T02:05:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:55 smithi181 ceph-mon[94189]: pgmap v21877: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 592 KiB/s rd, 590 KiB/s wr, 9 op/s 2023-03-10T02:05:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:55 smithi038 ceph-mon[92422]: pgmap v21877: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 592 KiB/s rd, 590 KiB/s wr, 9 op/s 2023-03-10T02:05:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40066.559631 secs 2023-03-10T02:05:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:05:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:05:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40066.559631 secs 2023-03-10T02:05:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:05:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:05:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40066.559631 secs 2023-03-10T02:05:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:05:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:05:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:57 smithi122 ceph-mon[93869]: pgmap v21878: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 595 KiB/s rd, 590 KiB/s wr, 12 op/s 2023-03-10T02:05:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:05:57.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:05:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:57 smithi181 ceph-mon[94189]: pgmap v21878: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 595 KiB/s rd, 590 KiB/s wr, 12 op/s 2023-03-10T02:05:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:05:57.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:05:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:57 smithi038 ceph-mon[92422]: pgmap v21878: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 595 KiB/s rd, 590 KiB/s wr, 12 op/s 2023-03-10T02:05:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:05:57.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:05:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:58 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 40069.022995 secs 2023-03-10T02:05:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:58 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 40069.022995 secs 2023-03-10T02:05:58.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:58 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 40069.022995 secs 2023-03-10T02:05:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40530.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:05:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:59 smithi122 ceph-mon[93869]: pgmap v21879: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 464 KiB/s wr, 11 op/s 2023-03-10T02:05:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:59 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:05:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:05:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:05:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:05:59.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:59 smithi181 ceph-mon[94189]: pgmap v21879: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 464 KiB/s wr, 11 op/s 2023-03-10T02:05:59.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:59 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:05:59.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:05:59.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:05:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:05:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:59 smithi038 ceph-mon[92422]: pgmap v21879: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 464 KiB/s wr, 11 op/s 2023-03-10T02:05:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:59 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:05:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:05:59.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:05:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:06:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:01 smithi122 ceph-mon[93869]: pgmap v21880: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1019 KiB/s rd, 687 KiB/s wr, 11 op/s 2023-03-10T02:06:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40071.559686 secs 2023-03-10T02:06:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:06:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:06:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:01 smithi181 ceph-mon[94189]: pgmap v21880: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1019 KiB/s rd, 687 KiB/s wr, 11 op/s 2023-03-10T02:06:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40071.559686 secs 2023-03-10T02:06:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:06:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:06:01.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:01 smithi038 ceph-mon[92422]: pgmap v21880: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1019 KiB/s rd, 687 KiB/s wr, 11 op/s 2023-03-10T02:06:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40071.559686 secs 2023-03-10T02:06:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:06:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:06:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:03 smithi122 ceph-mon[93869]: pgmap v21881: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 722 KiB/s wr, 14 op/s 2023-03-10T02:06:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40074.023054 secs 2023-03-10T02:06:03.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:03 smithi181 ceph-mon[94189]: pgmap v21881: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 722 KiB/s wr, 14 op/s 2023-03-10T02:06:03.731 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40074.023054 secs 2023-03-10T02:06:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:03 smithi038 ceph-mon[92422]: pgmap v21881: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 722 KiB/s wr, 14 op/s 2023-03-10T02:06:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40074.023054 secs 2023-03-10T02:06:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40535.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:06:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:05 smithi122 ceph-mon[93869]: pgmap v21882: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 814 KiB/s wr, 22 op/s 2023-03-10T02:06:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:05 smithi181 ceph-mon[94189]: pgmap v21882: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 814 KiB/s wr, 22 op/s 2023-03-10T02:06:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:05 smithi038 ceph-mon[92422]: pgmap v21882: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 814 KiB/s wr, 22 op/s 2023-03-10T02:06:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40076.559745 secs 2023-03-10T02:06:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40076.559745 secs 2023-03-10T02:06:06.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40076.559745 secs 2023-03-10T02:06:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:07 smithi122 ceph-mon[93869]: pgmap v21883: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 825 KiB/s wr, 25 op/s 2023-03-10T02:06:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:06:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:06:07.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:07 smithi181 ceph-mon[94189]: pgmap v21883: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 825 KiB/s wr, 25 op/s 2023-03-10T02:06:07.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:06:07.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:06:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:07 smithi038 ceph-mon[92422]: pgmap v21883: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 825 KiB/s wr, 25 op/s 2023-03-10T02:06:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:06:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:06:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:06:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:06:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40079.023114 secs 2023-03-10T02:06:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:06:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:06:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40079.023114 secs 2023-03-10T02:06:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:06:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:06:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40079.023114 secs 2023-03-10T02:06:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40540.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:06:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:09 smithi122 ceph-mon[93869]: pgmap v21884: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 996 KiB/s wr, 25 op/s 2023-03-10T02:06:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:06:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:06:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:09 smithi181 ceph-mon[94189]: pgmap v21884: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 996 KiB/s wr, 25 op/s 2023-03-10T02:06:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:06:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:06:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:09 smithi038 ceph-mon[92422]: pgmap v21884: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 996 KiB/s wr, 25 op/s 2023-03-10T02:06:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:06:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:06:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:11 smithi122 ceph-mon[93869]: pgmap v21885: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 898 KiB/s rd, 741 KiB/s wr, 23 op/s 2023-03-10T02:06:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40081.559802 secs 2023-03-10T02:06:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:11 smithi181 ceph-mon[94189]: pgmap v21885: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 898 KiB/s rd, 741 KiB/s wr, 23 op/s 2023-03-10T02:06:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40081.559802 secs 2023-03-10T02:06:11.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:11 smithi038 ceph-mon[92422]: pgmap v21885: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 898 KiB/s rd, 741 KiB/s wr, 23 op/s 2023-03-10T02:06:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40081.559802 secs 2023-03-10T02:06:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:13 smithi122 ceph-mon[93869]: pgmap v21886: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 690 KiB/s rd, 543 KiB/s wr, 25 op/s 2023-03-10T02:06:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40084.023209 secs 2023-03-10T02:06:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:13 smithi181 ceph-mon[94189]: pgmap v21886: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 690 KiB/s rd, 543 KiB/s wr, 25 op/s 2023-03-10T02:06:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40084.023209 secs 2023-03-10T02:06:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:13 smithi038 ceph-mon[92422]: pgmap v21886: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 690 KiB/s rd, 543 KiB/s wr, 25 op/s 2023-03-10T02:06:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40084.023209 secs 2023-03-10T02:06:13.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40545.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:06:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:15 smithi038 ceph-mon[92422]: pgmap v21887: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 836 KiB/s rd, 849 KiB/s wr, 23 op/s 2023-03-10T02:06:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:15 smithi122 ceph-mon[93869]: pgmap v21887: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 836 KiB/s rd, 849 KiB/s wr, 23 op/s 2023-03-10T02:06:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:15 smithi181 ceph-mon[94189]: pgmap v21887: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 836 KiB/s rd, 849 KiB/s wr, 23 op/s 2023-03-10T02:06:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40086.559859 secs 2023-03-10T02:06:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:06:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:06:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40086.559859 secs 2023-03-10T02:06:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:06:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:06:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40086.559859 secs 2023-03-10T02:06:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:06:16.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:06:17.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:17 smithi038 ceph-mon[92422]: pgmap v21888: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 542 KiB/s rd, 549 KiB/s wr, 12 op/s 2023-03-10T02:06:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:06:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:06:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:17 smithi122 ceph-mon[93869]: pgmap v21888: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 542 KiB/s rd, 549 KiB/s wr, 12 op/s 2023-03-10T02:06:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:06:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:06:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:17 smithi181 ceph-mon[94189]: pgmap v21888: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 542 KiB/s rd, 549 KiB/s wr, 12 op/s 2023-03-10T02:06:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:06:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:06:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40550.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:06:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40089.023303 secs 2023-03-10T02:06:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40089.023303 secs 2023-03-10T02:06:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40089.023303 secs 2023-03-10T02:06:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:19 smithi122 ceph-mon[93869]: pgmap v21889: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 605 KiB/s rd, 614 KiB/s wr, 12 op/s 2023-03-10T02:06:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:19 smithi181 ceph-mon[94189]: pgmap v21889: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 605 KiB/s rd, 614 KiB/s wr, 12 op/s 2023-03-10T02:06:19.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:19 smithi038 ceph-mon[92422]: pgmap v21889: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 605 KiB/s rd, 614 KiB/s wr, 12 op/s 2023-03-10T02:06:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:06:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:06:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:06:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:06:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:06:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:06:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:21 smithi122 ceph-mon[93869]: pgmap v21890: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 443 KiB/s wr, 10 op/s 2023-03-10T02:06:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40091.559940 secs 2023-03-10T02:06:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:06:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:06:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:21 smithi181 ceph-mon[94189]: pgmap v21890: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 443 KiB/s wr, 10 op/s 2023-03-10T02:06:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40091.559940 secs 2023-03-10T02:06:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:06:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:06:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:21 smithi038 ceph-mon[92422]: pgmap v21890: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 434 KiB/s rd, 443 KiB/s wr, 10 op/s 2023-03-10T02:06:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40091.559940 secs 2023-03-10T02:06:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:06:22.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:06:23.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40555.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:06:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:23 smithi122 ceph-mon[93869]: pgmap v21891: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 787 KiB/s rd, 806 KiB/s wr, 13 op/s 2023-03-10T02:06:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40094.023366 secs 2023-03-10T02:06:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:23 smithi181 ceph-mon[94189]: pgmap v21891: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 787 KiB/s rd, 806 KiB/s wr, 13 op/s 2023-03-10T02:06:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40094.023366 secs 2023-03-10T02:06:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:23 smithi038 ceph-mon[92422]: pgmap v21891: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 787 KiB/s rd, 806 KiB/s wr, 13 op/s 2023-03-10T02:06:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40094.023366 secs 2023-03-10T02:06:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:25 smithi122 ceph-mon[93869]: pgmap v21892: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 903 KiB/s rd, 922 KiB/s wr, 13 op/s 2023-03-10T02:06:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:25 smithi122 ceph-mon[93869]: osdmap e1583: 12 total, 12 up, 12 in 2023-03-10T02:06:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:25 smithi181 ceph-mon[94189]: pgmap v21892: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 903 KiB/s rd, 922 KiB/s wr, 13 op/s 2023-03-10T02:06:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:25 smithi181 ceph-mon[94189]: osdmap e1583: 12 total, 12 up, 12 in 2023-03-10T02:06:25.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:25 smithi038 ceph-mon[92422]: pgmap v21892: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 903 KiB/s rd, 922 KiB/s wr, 13 op/s 2023-03-10T02:06:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:25 smithi038 ceph-mon[92422]: osdmap e1583: 12 total, 12 up, 12 in 2023-03-10T02:06:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40096.559998 secs 2023-03-10T02:06:26.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40096.559998 secs 2023-03-10T02:06:26.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40096.559998 secs 2023-03-10T02:06:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:27 smithi122 ceph-mon[93869]: pgmap v21894: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 739 KiB/s wr, 14 op/s 2023-03-10T02:06:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:27 smithi181 ceph-mon[94189]: pgmap v21894: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 739 KiB/s wr, 14 op/s 2023-03-10T02:06:27.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:27 smithi038 ceph-mon[92422]: pgmap v21894: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 739 KiB/s wr, 14 op/s 2023-03-10T02:06:28.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40560.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:06:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40099.023474 secs 2023-03-10T02:06:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40099.023474 secs 2023-03-10T02:06:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40099.023474 secs 2023-03-10T02:06:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:29 smithi122 ceph-mon[93869]: pgmap v21895: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 800 KiB/s rd, 810 KiB/s wr, 12 op/s 2023-03-10T02:06:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:29 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:06:29.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:29 smithi181 ceph-mon[94189]: pgmap v21895: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 800 KiB/s rd, 810 KiB/s wr, 12 op/s 2023-03-10T02:06:29.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:29 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:06:29.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:29 smithi038 ceph-mon[92422]: pgmap v21895: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 800 KiB/s rd, 810 KiB/s wr, 12 op/s 2023-03-10T02:06:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:29 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:06:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:30 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:06:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:30 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:06:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:30 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:06:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:30 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:06:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:30 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:06:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:30 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:06:30.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:30 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:06:30.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:30 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:06:30.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:30 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:06:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:31 smithi122 ceph-mon[93869]: pgmap v21896: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 800 KiB/s rd, 810 KiB/s wr, 12 op/s 2023-03-10T02:06:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40101.560059 secs 2023-03-10T02:06:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:31 smithi181 ceph-mon[94189]: pgmap v21896: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 800 KiB/s rd, 810 KiB/s wr, 12 op/s 2023-03-10T02:06:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40101.560059 secs 2023-03-10T02:06:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:31 smithi038 ceph-mon[92422]: pgmap v21896: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 800 KiB/s rd, 810 KiB/s wr, 12 op/s 2023-03-10T02:06:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40101.560059 secs 2023-03-10T02:06:33.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40565.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:06:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:33 smithi122 ceph-mon[93869]: pgmap v21897: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 417 KiB/s wr, 10 op/s 2023-03-10T02:06:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40104.023533 secs 2023-03-10T02:06:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:33 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:06:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:33 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:06:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:33 smithi181 ceph-mon[94189]: pgmap v21897: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 417 KiB/s wr, 10 op/s 2023-03-10T02:06:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40104.023533 secs 2023-03-10T02:06:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:33 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:06:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:33 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:06:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:33 smithi038 ceph-mon[92422]: pgmap v21897: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 417 KiB/s wr, 10 op/s 2023-03-10T02:06:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40104.023533 secs 2023-03-10T02:06:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:33 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:06:34.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:33 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:06:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:35 smithi122 ceph-mon[93869]: pgmap v21898: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 248 KiB/s wr, 9 op/s 2023-03-10T02:06:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:35 smithi122 ceph-mon[93869]: osdmap e1584: 12 total, 12 up, 12 in 2023-03-10T02:06:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:35 smithi181 ceph-mon[94189]: pgmap v21898: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 248 KiB/s wr, 9 op/s 2023-03-10T02:06:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:35 smithi181 ceph-mon[94189]: osdmap e1584: 12 total, 12 up, 12 in 2023-03-10T02:06:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:35 smithi038 ceph-mon[92422]: pgmap v21898: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 248 KiB/s wr, 9 op/s 2023-03-10T02:06:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:35 smithi038 ceph-mon[92422]: osdmap e1584: 12 total, 12 up, 12 in 2023-03-10T02:06:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40106.560119 secs 2023-03-10T02:06:36.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:06:36.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:06:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40106.560119 secs 2023-03-10T02:06:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:06:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:06:36.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40106.560119 secs 2023-03-10T02:06:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:06:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:06:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:37 smithi122 ceph-mon[93869]: pgmap v21900: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 574 KiB/s rd, 572 KiB/s wr, 9 op/s 2023-03-10T02:06:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:37 smithi181 ceph-mon[94189]: pgmap v21900: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 574 KiB/s rd, 572 KiB/s wr, 9 op/s 2023-03-10T02:06:37.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:37 smithi038 ceph-mon[92422]: pgmap v21900: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 574 KiB/s rd, 572 KiB/s wr, 9 op/s 2023-03-10T02:06:38.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40570.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:06:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40109.023628 secs 2023-03-10T02:06:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40109.023628 secs 2023-03-10T02:06:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40109.023628 secs 2023-03-10T02:06:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:39 smithi122 ceph-mon[93869]: pgmap v21901: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 497 KiB/s wr, 9 op/s 2023-03-10T02:06:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:39 smithi181 ceph-mon[94189]: pgmap v21901: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 497 KiB/s wr, 9 op/s 2023-03-10T02:06:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:39 smithi038 ceph-mon[92422]: pgmap v21901: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 497 KiB/s wr, 9 op/s 2023-03-10T02:06:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:06:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:06:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:06:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:06:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:06:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:06:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:41 smithi122 ceph-mon[93869]: pgmap v21902: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 497 KiB/s wr, 9 op/s 2023-03-10T02:06:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40111.560178 secs 2023-03-10T02:06:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:06:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:06:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:41 smithi181 ceph-mon[94189]: pgmap v21902: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 497 KiB/s wr, 9 op/s 2023-03-10T02:06:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40111.560178 secs 2023-03-10T02:06:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:06:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:06:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:41 smithi038 ceph-mon[92422]: pgmap v21902: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 498 KiB/s rd, 497 KiB/s wr, 9 op/s 2023-03-10T02:06:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40111.560178 secs 2023-03-10T02:06:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:06:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:06:42.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:42 smithi122 ceph-mon[93869]: pgmap v21903: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 873 KiB/s rd, 870 KiB/s wr, 11 op/s 2023-03-10T02:06:42.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:42 smithi181 ceph-mon[94189]: pgmap v21903: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 873 KiB/s rd, 870 KiB/s wr, 11 op/s 2023-03-10T02:06:42.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:42 smithi038 ceph-mon[92422]: pgmap v21903: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 873 KiB/s rd, 870 KiB/s wr, 11 op/s 2023-03-10T02:06:43.801 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40575.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:06:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40114.023743 secs 2023-03-10T02:06:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40114.023743 secs 2023-03-10T02:06:43.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40114.023743 secs 2023-03-10T02:06:44.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:44 smithi122 ceph-mon[93869]: pgmap v21904: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 873 KiB/s rd, 871 KiB/s wr, 11 op/s 2023-03-10T02:06:44.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:44 smithi181 ceph-mon[94189]: pgmap v21904: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 873 KiB/s rd, 871 KiB/s wr, 11 op/s 2023-03-10T02:06:44.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:44 smithi038 ceph-mon[92422]: pgmap v21904: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 873 KiB/s rd, 871 KiB/s wr, 11 op/s 2023-03-10T02:06:46.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40116.560238 secs 2023-03-10T02:06:46.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40116.560238 secs 2023-03-10T02:06:46.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40116.560238 secs 2023-03-10T02:06:47.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:47 smithi122 ceph-mon[93869]: pgmap v21905: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 13 op/s 2023-03-10T02:06:47.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:47 smithi181 ceph-mon[94189]: pgmap v21905: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 13 op/s 2023-03-10T02:06:47.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:47 smithi038 ceph-mon[92422]: pgmap v21905: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 13 op/s 2023-03-10T02:06:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:06:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:06:48.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40119.023824 secs 2023-03-10T02:06:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:06:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:06:48.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40119.023824 secs 2023-03-10T02:06:48.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:06:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:06:48.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40119.023824 secs 2023-03-10T02:06:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40580.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:06:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:49 smithi122 ceph-mon[93869]: pgmap v21906: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 769 KiB/s rd, 767 KiB/s wr, 10 op/s 2023-03-10T02:06:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:06:49.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:06:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:49 smithi181 ceph-mon[94189]: pgmap v21906: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 769 KiB/s rd, 767 KiB/s wr, 10 op/s 2023-03-10T02:06:49.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:06:49.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:06:49.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:49 smithi038 ceph-mon[92422]: pgmap v21906: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 769 KiB/s rd, 767 KiB/s wr, 10 op/s 2023-03-10T02:06:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:06:49.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:06:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:51 smithi122 ceph-mon[93869]: pgmap v21907: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 697 KiB/s rd, 695 KiB/s wr, 8 op/s 2023-03-10T02:06:51.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40121.560296 secs 2023-03-10T02:06:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:51 smithi181 ceph-mon[94189]: pgmap v21907: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 697 KiB/s rd, 695 KiB/s wr, 8 op/s 2023-03-10T02:06:51.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40121.560296 secs 2023-03-10T02:06:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:51 smithi038 ceph-mon[92422]: pgmap v21907: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 697 KiB/s rd, 695 KiB/s wr, 8 op/s 2023-03-10T02:06:51.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40121.560296 secs 2023-03-10T02:06:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:53 smithi122 ceph-mon[93869]: pgmap v21908: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 906 KiB/s rd, 903 KiB/s wr, 12 op/s 2023-03-10T02:06:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:06:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:06:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40124.023883 secs 2023-03-10T02:06:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:53 smithi181 ceph-mon[94189]: pgmap v21908: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 906 KiB/s rd, 903 KiB/s wr, 12 op/s 2023-03-10T02:06:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:06:53.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:06:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40124.023883 secs 2023-03-10T02:06:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:53 smithi038 ceph-mon[92422]: pgmap v21908: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 906 KiB/s rd, 903 KiB/s wr, 12 op/s 2023-03-10T02:06:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:06:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:06:53.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40124.023883 secs 2023-03-10T02:06:53.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40585.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:06:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:55 smithi122 ceph-mon[93869]: pgmap v21909: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 558 KiB/s rd, 556 KiB/s wr, 8 op/s 2023-03-10T02:06:55.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:55 smithi181 ceph-mon[94189]: pgmap v21909: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 558 KiB/s rd, 556 KiB/s wr, 8 op/s 2023-03-10T02:06:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:55 smithi038 ceph-mon[92422]: pgmap v21909: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 558 KiB/s rd, 556 KiB/s wr, 8 op/s 2023-03-10T02:06:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40126.560386 secs 2023-03-10T02:06:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:06:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:06:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40126.560386 secs 2023-03-10T02:06:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:06:56.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:06:56.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40126.560386 secs 2023-03-10T02:06:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:06:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:06:57.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:57 smithi122 ceph-mon[93869]: pgmap v21910: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 764 KiB/s rd, 759 KiB/s wr, 11 op/s 2023-03-10T02:06:57.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:57 smithi181 ceph-mon[94189]: pgmap v21910: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 764 KiB/s rd, 759 KiB/s wr, 11 op/s 2023-03-10T02:06:57.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:57 smithi038 ceph-mon[92422]: pgmap v21910: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 764 KiB/s rd, 759 KiB/s wr, 11 op/s 2023-03-10T02:06:58.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40129.023942 secs 2023-03-10T02:06:58.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40129.023942 secs 2023-03-10T02:06:58.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40129.023942 secs 2023-03-10T02:06:58.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40590.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:06:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:06:59 smithi122 ceph-mon[93869]: pgmap v21911: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 446 KiB/s wr, 10 op/s 2023-03-10T02:06:59.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:06:59 smithi181 ceph-mon[94189]: pgmap v21911: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 446 KiB/s wr, 10 op/s 2023-03-10T02:06:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:06:59 smithi038 ceph-mon[92422]: pgmap v21911: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 446 KiB/s wr, 10 op/s 2023-03-10T02:07:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:07:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:07:00.732 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:07:00.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:07:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:07:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:07:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:01 smithi122 ceph-mon[93869]: pgmap v21912: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 446 KiB/s wr, 9 op/s 2023-03-10T02:07:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40131.560472 secs 2023-03-10T02:07:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:07:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:07:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:01 smithi181 ceph-mon[94189]: pgmap v21912: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 446 KiB/s wr, 9 op/s 2023-03-10T02:07:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40131.560472 secs 2023-03-10T02:07:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:07:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:07:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:01 smithi038 ceph-mon[92422]: pgmap v21912: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 461 KiB/s rd, 446 KiB/s wr, 9 op/s 2023-03-10T02:07:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40131.560472 secs 2023-03-10T02:07:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:07:01.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:07:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:03 smithi122 ceph-mon[93869]: pgmap v21913: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 913 KiB/s rd, 1.1 MiB/s wr, 14 op/s 2023-03-10T02:07:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40134.024025 secs 2023-03-10T02:07:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:03 smithi181 ceph-mon[94189]: pgmap v21913: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 913 KiB/s rd, 1.1 MiB/s wr, 14 op/s 2023-03-10T02:07:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40134.024025 secs 2023-03-10T02:07:03.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:03 smithi038 ceph-mon[92422]: pgmap v21913: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 913 KiB/s rd, 1.1 MiB/s wr, 14 op/s 2023-03-10T02:07:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40134.024025 secs 2023-03-10T02:07:03.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40595.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:07:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:05 smithi122 ceph-mon[93869]: pgmap v21914: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 711 KiB/s rd, 885 KiB/s wr, 21 op/s 2023-03-10T02:07:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:05 smithi181 ceph-mon[94189]: pgmap v21914: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 711 KiB/s rd, 885 KiB/s wr, 21 op/s 2023-03-10T02:07:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:05 smithi038 ceph-mon[92422]: pgmap v21914: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 711 KiB/s rd, 885 KiB/s wr, 21 op/s 2023-03-10T02:07:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40136.560556 secs 2023-03-10T02:07:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40136.560556 secs 2023-03-10T02:07:06.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40136.560556 secs 2023-03-10T02:07:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:07 smithi122 ceph-mon[93869]: pgmap v21915: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 761 KiB/s rd, 944 KiB/s wr, 25 op/s 2023-03-10T02:07:07.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:07 smithi181 ceph-mon[94189]: pgmap v21915: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 761 KiB/s rd, 944 KiB/s wr, 25 op/s 2023-03-10T02:07:07.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:07 smithi038 ceph-mon[92422]: pgmap v21915: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 761 KiB/s rd, 944 KiB/s wr, 25 op/s 2023-03-10T02:07:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:07:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:07:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40139.024113 secs 2023-03-10T02:07:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:07:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:07:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40139.024113 secs 2023-03-10T02:07:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:07:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:07:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40139.024113 secs 2023-03-10T02:07:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40600.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:07:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:09 smithi122 ceph-mon[93869]: pgmap v21916: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 742 KiB/s wr, 23 op/s 2023-03-10T02:07:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:07:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:07:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:09 smithi181 ceph-mon[94189]: pgmap v21916: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 742 KiB/s wr, 23 op/s 2023-03-10T02:07:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:07:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:07:09.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:09 smithi038 ceph-mon[92422]: pgmap v21916: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 742 KiB/s wr, 23 op/s 2023-03-10T02:07:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:07:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:07:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:11 smithi122 ceph-mon[93869]: pgmap v21917: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 707 KiB/s wr, 22 op/s 2023-03-10T02:07:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40141.560613 secs 2023-03-10T02:07:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:11 smithi181 ceph-mon[94189]: pgmap v21917: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 707 KiB/s wr, 22 op/s 2023-03-10T02:07:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40141.560613 secs 2023-03-10T02:07:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:11 smithi038 ceph-mon[92422]: pgmap v21917: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 707 KiB/s wr, 22 op/s 2023-03-10T02:07:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40141.560613 secs 2023-03-10T02:07:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:07:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:07:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:07:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:07:12.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:07:12.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:07:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:13 smithi038 ceph-mon[92422]: pgmap v21918: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 907 KiB/s wr, 25 op/s 2023-03-10T02:07:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40144.024203 secs 2023-03-10T02:07:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40605.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:07:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:13 smithi122 ceph-mon[93869]: pgmap v21918: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 907 KiB/s wr, 25 op/s 2023-03-10T02:07:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40144.024203 secs 2023-03-10T02:07:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:13 smithi181 ceph-mon[94189]: pgmap v21918: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 907 KiB/s wr, 25 op/s 2023-03-10T02:07:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40144.024203 secs 2023-03-10T02:07:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:15 smithi122 ceph-mon[93869]: pgmap v21919: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 812 KiB/s rd, 271 KiB/s wr, 20 op/s 2023-03-10T02:07:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:15 smithi181 ceph-mon[94189]: pgmap v21919: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 812 KiB/s rd, 271 KiB/s wr, 20 op/s 2023-03-10T02:07:15.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:15 smithi038 ceph-mon[92422]: pgmap v21919: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 812 KiB/s rd, 271 KiB/s wr, 20 op/s 2023-03-10T02:07:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40146.560673 secs 2023-03-10T02:07:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40146.560673 secs 2023-03-10T02:07:16.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40146.560673 secs 2023-03-10T02:07:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:17 smithi122 ceph-mon[93869]: pgmap v21920: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 806 KiB/s rd, 261 KiB/s wr, 12 op/s 2023-03-10T02:07:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:17 smithi181 ceph-mon[94189]: pgmap v21920: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 806 KiB/s rd, 261 KiB/s wr, 12 op/s 2023-03-10T02:07:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:17 smithi038 ceph-mon[92422]: pgmap v21920: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 806 KiB/s rd, 261 KiB/s wr, 12 op/s 2023-03-10T02:07:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40610.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:07:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40149.024289 secs 2023-03-10T02:07:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40149.024289 secs 2023-03-10T02:07:18.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40149.024289 secs 2023-03-10T02:07:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:19 smithi122 ceph-mon[93869]: pgmap v21921: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 756 KiB/s rd, 212 KiB/s wr, 10 op/s 2023-03-10T02:07:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:19 smithi181 ceph-mon[94189]: pgmap v21921: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 756 KiB/s rd, 212 KiB/s wr, 10 op/s 2023-03-10T02:07:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:19 smithi038 ceph-mon[92422]: pgmap v21921: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 756 KiB/s rd, 212 KiB/s wr, 10 op/s 2023-03-10T02:07:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:07:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:07:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:07:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:07:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:07:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:07:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:21 smithi122 ceph-mon[93869]: pgmap v21922: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 211 KiB/s wr, 8 op/s 2023-03-10T02:07:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40151.560733 secs 2023-03-10T02:07:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:07:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:07:21.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:21 smithi181 ceph-mon[94189]: pgmap v21922: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 211 KiB/s wr, 8 op/s 2023-03-10T02:07:21.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40151.560733 secs 2023-03-10T02:07:21.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:07:21.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:07:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:21 smithi038 ceph-mon[92422]: pgmap v21922: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 211 KiB/s wr, 8 op/s 2023-03-10T02:07:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40151.560733 secs 2023-03-10T02:07:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:07:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:07:23.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40615.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:07:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:23 smithi122 ceph-mon[93869]: pgmap v21923: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 454 KiB/s rd, 472 KiB/s wr, 14 op/s 2023-03-10T02:07:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:23 smithi181 ceph-mon[94189]: pgmap v21923: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 454 KiB/s rd, 472 KiB/s wr, 14 op/s 2023-03-10T02:07:23.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:23 smithi038 ceph-mon[92422]: pgmap v21923: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 454 KiB/s rd, 472 KiB/s wr, 14 op/s 2023-03-10T02:07:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:25 smithi122 ceph-mon[93869]: pgmap v21924: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 272 KiB/s wr, 10 op/s 2023-03-10T02:07:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:25 smithi122 ceph-mon[93869]: osdmap e1585: 12 total, 12 up, 12 in 2023-03-10T02:07:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:25 smithi181 ceph-mon[94189]: pgmap v21924: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 272 KiB/s wr, 10 op/s 2023-03-10T02:07:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:25 smithi181 ceph-mon[94189]: osdmap e1585: 12 total, 12 up, 12 in 2023-03-10T02:07:25.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:25 smithi038 ceph-mon[92422]: pgmap v21924: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 272 KiB/s wr, 10 op/s 2023-03-10T02:07:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:25 smithi038 ceph-mon[92422]: osdmap e1585: 12 total, 12 up, 12 in 2023-03-10T02:07:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40156.560793 secs 2023-03-10T02:07:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:26 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T02:07:26.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40156.560793 secs 2023-03-10T02:07:26.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:26 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T02:07:26.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40156.560793 secs 2023-03-10T02:07:26.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:26 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T02:07:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:27 smithi122 ceph-mon[93869]: pgmap v21926: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 568 KiB/s wr, 14 op/s 2023-03-10T02:07:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:27 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:07:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:27 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:07:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:27 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:07:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:27 smithi181 ceph-mon[94189]: pgmap v21926: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 568 KiB/s wr, 14 op/s 2023-03-10T02:07:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:27 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:07:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:27 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:07:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:27 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:07:27.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:27 smithi038 ceph-mon[92422]: pgmap v21926: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 568 KiB/s wr, 14 op/s 2023-03-10T02:07:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:27 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:07:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:27 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:07:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:27 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:07:28.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40620.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:07:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:29 smithi122 ceph-mon[93869]: pgmap v21927: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 555 KiB/s wr, 12 op/s 2023-03-10T02:07:29.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:29 smithi181 ceph-mon[94189]: pgmap v21927: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 555 KiB/s wr, 12 op/s 2023-03-10T02:07:29.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:29 smithi038 ceph-mon[92422]: pgmap v21927: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 555 KiB/s wr, 12 op/s 2023-03-10T02:07:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:30 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:07:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:30 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:07:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:30 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:07:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:31 smithi122 ceph-mon[93869]: pgmap v21928: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 555 KiB/s wr, 12 op/s 2023-03-10T02:07:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40161.560851 secs 2023-03-10T02:07:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:07:31.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:07:31.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:07:31.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:07:31.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:31 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:07:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:31 smithi181 ceph-mon[94189]: pgmap v21928: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 555 KiB/s wr, 12 op/s 2023-03-10T02:07:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40161.560851 secs 2023-03-10T02:07:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:07:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:07:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:07:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:07:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:31 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:07:31.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:31 smithi038 ceph-mon[92422]: pgmap v21928: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 544 KiB/s rd, 555 KiB/s wr, 12 op/s 2023-03-10T02:07:31.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40161.560851 secs 2023-03-10T02:07:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:07:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:07:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:07:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:07:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:31 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:07:33.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40625.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:07:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:33 smithi122 ceph-mon[93869]: pgmap v21929: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 307 KiB/s wr, 9 op/s 2023-03-10T02:07:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:33 smithi181 ceph-mon[94189]: pgmap v21929: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 307 KiB/s wr, 9 op/s 2023-03-10T02:07:33.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:33 smithi038 ceph-mon[92422]: pgmap v21929: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 307 KiB/s wr, 9 op/s 2023-03-10T02:07:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:35 smithi122 ceph-mon[93869]: pgmap v21930: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 307 KiB/s wr, 9 op/s 2023-03-10T02:07:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:35 smithi122 ceph-mon[93869]: osdmap e1586: 12 total, 12 up, 12 in 2023-03-10T02:07:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:35 smithi181 ceph-mon[94189]: pgmap v21930: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 307 KiB/s wr, 9 op/s 2023-03-10T02:07:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:35 smithi181 ceph-mon[94189]: osdmap e1586: 12 total, 12 up, 12 in 2023-03-10T02:07:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:35 smithi038 ceph-mon[92422]: pgmap v21930: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 307 KiB/s wr, 9 op/s 2023-03-10T02:07:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:35 smithi038 ceph-mon[92422]: osdmap e1586: 12 total, 12 up, 12 in 2023-03-10T02:07:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40166.560909 secs 2023-03-10T02:07:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40166.560909 secs 2023-03-10T02:07:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40166.560909 secs 2023-03-10T02:07:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:37 smithi122 ceph-mon[93869]: pgmap v21932: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 107 KiB/s wr, 8 op/s 2023-03-10T02:07:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:37 smithi181 ceph-mon[94189]: pgmap v21932: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 107 KiB/s wr, 8 op/s 2023-03-10T02:07:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:37 smithi038 ceph-mon[92422]: pgmap v21932: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 222 KiB/s rd, 107 KiB/s wr, 8 op/s 2023-03-10T02:07:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40630.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:07:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40169.024605 secs 2023-03-10T02:07:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40169.024605 secs 2023-03-10T02:07:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40169.024605 secs 2023-03-10T02:07:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:39 smithi122 ceph-mon[93869]: pgmap v21933: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 612 KiB/s rd, 498 KiB/s wr, 9 op/s 2023-03-10T02:07:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:39 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:07:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:39 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:07:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:39 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:07:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:39 smithi181 ceph-mon[94189]: pgmap v21933: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 612 KiB/s rd, 498 KiB/s wr, 9 op/s 2023-03-10T02:07:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:39 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:07:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:39 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:07:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:39 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:07:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:39 smithi038 ceph-mon[92422]: pgmap v21933: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 612 KiB/s rd, 498 KiB/s wr, 9 op/s 2023-03-10T02:07:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:39 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:07:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:39 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:07:40.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:39 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:07:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:07:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:07:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:07:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:07:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:07:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:07:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:41 smithi122 ceph-mon[93869]: pgmap v21934: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 612 KiB/s rd, 498 KiB/s wr, 9 op/s 2023-03-10T02:07:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40171.560968 secs 2023-03-10T02:07:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:07:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:07:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:41 smithi181 ceph-mon[94189]: pgmap v21934: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 612 KiB/s rd, 498 KiB/s wr, 9 op/s 2023-03-10T02:07:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40171.560968 secs 2023-03-10T02:07:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:07:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:07:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:41 smithi038 ceph-mon[92422]: pgmap v21934: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 612 KiB/s rd, 498 KiB/s wr, 9 op/s 2023-03-10T02:07:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40171.560968 secs 2023-03-10T02:07:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:07:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:07:43.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40635.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:07:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:43 smithi122 ceph-mon[93869]: pgmap v21935: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1002 KiB/s rd, 887 KiB/s wr, 11 op/s 2023-03-10T02:07:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40174.024666 secs 2023-03-10T02:07:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:43 smithi181 ceph-mon[94189]: pgmap v21935: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1002 KiB/s rd, 887 KiB/s wr, 11 op/s 2023-03-10T02:07:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40174.024666 secs 2023-03-10T02:07:43.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:43 smithi038 ceph-mon[92422]: pgmap v21935: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1002 KiB/s rd, 887 KiB/s wr, 11 op/s 2023-03-10T02:07:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40174.024666 secs 2023-03-10T02:07:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:45 smithi122 ceph-mon[93869]: pgmap v21936: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1002 KiB/s rd, 887 KiB/s wr, 11 op/s 2023-03-10T02:07:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:45 smithi181 ceph-mon[94189]: pgmap v21936: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1002 KiB/s rd, 887 KiB/s wr, 11 op/s 2023-03-10T02:07:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:45 smithi038 ceph-mon[92422]: pgmap v21936: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1002 KiB/s rd, 887 KiB/s wr, 11 op/s 2023-03-10T02:07:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40176.561029 secs 2023-03-10T02:07:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40176.561029 secs 2023-03-10T02:07:46.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40176.561029 secs 2023-03-10T02:07:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:47 smithi122 ceph-mon[93869]: pgmap v21937: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 14 op/s 2023-03-10T02:07:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:47 smithi181 ceph-mon[94189]: pgmap v21937: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 14 op/s 2023-03-10T02:07:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:47 smithi038 ceph-mon[92422]: pgmap v21937: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.2 MiB/s wr, 14 op/s 2023-03-10T02:07:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40640.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:07:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:07:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:07:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:07:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:07:48.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:07:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:07:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:49 smithi122 ceph-mon[93869]: pgmap v21938: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 11 op/s 2023-03-10T02:07:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:07:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:07:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:49 smithi181 ceph-mon[94189]: pgmap v21938: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 11 op/s 2023-03-10T02:07:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:07:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:07:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:49 smithi038 ceph-mon[92422]: pgmap v21938: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 11 op/s 2023-03-10T02:07:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:07:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:07:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:50 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T02:07:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:50 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T02:07:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:50 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T02:07:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:51 smithi122 ceph-mon[93869]: pgmap v21939: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 874 KiB/s rd, 728 KiB/s wr, 10 op/s 2023-03-10T02:07:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40181.561085 secs 2023-03-10T02:07:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:51 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:07:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:07:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:07:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:51 smithi181 ceph-mon[94189]: pgmap v21939: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 874 KiB/s rd, 728 KiB/s wr, 10 op/s 2023-03-10T02:07:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40181.561085 secs 2023-03-10T02:07:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:51 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:07:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:07:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:07:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:51 smithi038 ceph-mon[92422]: pgmap v21939: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 874 KiB/s rd, 728 KiB/s wr, 10 op/s 2023-03-10T02:07:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40181.561085 secs 2023-03-10T02:07:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:51 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:07:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:07:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:07:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:52 smithi122 ceph-mon[93869]: pgmap v21940: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 935 KiB/s wr, 14 op/s 2023-03-10T02:07:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:52 smithi181 ceph-mon[94189]: pgmap v21940: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 935 KiB/s wr, 14 op/s 2023-03-10T02:07:53.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:52 smithi038 ceph-mon[92422]: pgmap v21940: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 935 KiB/s wr, 14 op/s 2023-03-10T02:07:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40645.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:07:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:07:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:07:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:07:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:07:53.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:07:53.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:07:54.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:54 smithi122 ceph-mon[93869]: pgmap v21941: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 704 KiB/s rd, 557 KiB/s wr, 10 op/s 2023-03-10T02:07:54.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:54 smithi181 ceph-mon[94189]: pgmap v21941: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 704 KiB/s rd, 557 KiB/s wr, 10 op/s 2023-03-10T02:07:54.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:54 smithi038 ceph-mon[92422]: pgmap v21941: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 704 KiB/s rd, 557 KiB/s wr, 10 op/s 2023-03-10T02:07:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:07:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:07:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:07:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:07:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:07:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:07:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40186.561159 secs 2023-03-10T02:07:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:56 smithi122 ceph-mon[93869]: pgmap v21942: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 732 KiB/s wr, 13 op/s 2023-03-10T02:07:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40186.561159 secs 2023-03-10T02:07:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:56 smithi181 ceph-mon[94189]: pgmap v21942: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 732 KiB/s wr, 13 op/s 2023-03-10T02:07:56.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40186.561159 secs 2023-03-10T02:07:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:56 smithi038 ceph-mon[92422]: pgmap v21942: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 732 KiB/s wr, 13 op/s 2023-03-10T02:07:58.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40650.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:07:59.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:07:59 smithi122 ceph-mon[93869]: pgmap v21943: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 883 KiB/s rd, 725 KiB/s wr, 10 op/s 2023-03-10T02:07:59.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:07:59 smithi181 ceph-mon[94189]: pgmap v21943: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 883 KiB/s rd, 725 KiB/s wr, 10 op/s 2023-03-10T02:07:59.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:07:59 smithi038 ceph-mon[92422]: pgmap v21943: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 883 KiB/s rd, 725 KiB/s wr, 10 op/s 2023-03-10T02:08:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:08:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:08:00.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:08:00.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:08:00.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:08:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:08:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:01 smithi122 ceph-mon[93869]: pgmap v21944: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 894 KiB/s rd, 725 KiB/s wr, 10 op/s 2023-03-10T02:08:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:08:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:08:01.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:01 smithi181 ceph-mon[94189]: pgmap v21944: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 894 KiB/s rd, 725 KiB/s wr, 10 op/s 2023-03-10T02:08:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:08:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:08:01.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:01 smithi038 ceph-mon[92422]: pgmap v21944: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 894 KiB/s rd, 725 KiB/s wr, 10 op/s 2023-03-10T02:08:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:08:01.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:08:02.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:02 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40191.561217 secs 2023-03-10T02:08:02.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:02 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40191.561217 secs 2023-03-10T02:08:02.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:02 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40191.561217 secs 2023-03-10T02:08:03.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:03 smithi122 ceph-mon[93869]: pgmap v21945: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 16 op/s 2023-03-10T02:08:03.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:03 smithi181 ceph-mon[94189]: pgmap v21945: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 16 op/s 2023-03-10T02:08:03.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:03 smithi038 ceph-mon[92422]: pgmap v21945: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 16 op/s 2023-03-10T02:08:03.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40655.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:08:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:05 smithi122 ceph-mon[93869]: pgmap v21946: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 23 op/s 2023-03-10T02:08:05.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:05 smithi181 ceph-mon[94189]: pgmap v21946: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 23 op/s 2023-03-10T02:08:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:05 smithi038 ceph-mon[92422]: pgmap v21946: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 23 op/s 2023-03-10T02:08:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40196.561277 secs 2023-03-10T02:08:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40196.561277 secs 2023-03-10T02:08:06.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40196.561277 secs 2023-03-10T02:08:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:07 smithi122 ceph-mon[93869]: pgmap v21947: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.2 MiB/s wr, 26 op/s 2023-03-10T02:08:07.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:07 smithi181 ceph-mon[94189]: pgmap v21947: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.2 MiB/s wr, 26 op/s 2023-03-10T02:08:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:07 smithi038 ceph-mon[92422]: pgmap v21947: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.2 MiB/s wr, 26 op/s 2023-03-10T02:08:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:08:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:08:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:08:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:08:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:08:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:08:08.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40660.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:08:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:09 smithi122 ceph-mon[93869]: pgmap v21948: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 24 op/s 2023-03-10T02:08:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:08:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:08:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:09 smithi181 ceph-mon[94189]: pgmap v21948: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 24 op/s 2023-03-10T02:08:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:08:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:08:09.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:09 smithi038 ceph-mon[92422]: pgmap v21948: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 24 op/s 2023-03-10T02:08:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:08:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:08:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:11 smithi122 ceph-mon[93869]: pgmap v21949: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 817 KiB/s rd, 741 KiB/s wr, 22 op/s 2023-03-10T02:08:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40201.561337 secs 2023-03-10T02:08:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:11 smithi181 ceph-mon[94189]: pgmap v21949: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 817 KiB/s rd, 741 KiB/s wr, 22 op/s 2023-03-10T02:08:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40201.561337 secs 2023-03-10T02:08:11.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:11 smithi038 ceph-mon[92422]: pgmap v21949: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 817 KiB/s rd, 741 KiB/s wr, 22 op/s 2023-03-10T02:08:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40201.561337 secs 2023-03-10T02:08:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:13 smithi122 ceph-mon[93869]: pgmap v21950: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.0 MiB/s wr, 26 op/s 2023-03-10T02:08:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:13 smithi181 ceph-mon[94189]: pgmap v21950: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.0 MiB/s wr, 26 op/s 2023-03-10T02:08:13.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:13 smithi038 ceph-mon[92422]: pgmap v21950: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.0 MiB/s wr, 26 op/s 2023-03-10T02:08:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40665.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:08:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:15 smithi122 ceph-mon[93869]: pgmap v21951: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 707 KiB/s rd, 381 KiB/s wr, 20 op/s 2023-03-10T02:08:15.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:15 smithi181 ceph-mon[94189]: pgmap v21951: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 707 KiB/s rd, 381 KiB/s wr, 20 op/s 2023-03-10T02:08:15.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:15 smithi038 ceph-mon[92422]: pgmap v21951: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 707 KiB/s rd, 381 KiB/s wr, 20 op/s 2023-03-10T02:08:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40206.561394 secs 2023-03-10T02:08:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:08:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:08:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40206.561394 secs 2023-03-10T02:08:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:08:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:08:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40206.561394 secs 2023-03-10T02:08:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:08:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:08:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:17 smithi122 ceph-mon[93869]: pgmap v21952: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 370 KiB/s wr, 12 op/s 2023-03-10T02:08:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:08:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:08:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:17 smithi181 ceph-mon[94189]: pgmap v21952: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 370 KiB/s wr, 12 op/s 2023-03-10T02:08:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:08:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:08:17.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:17 smithi038 ceph-mon[92422]: pgmap v21952: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 370 KiB/s wr, 12 op/s 2023-03-10T02:08:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:08:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:08:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40670.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:08:19.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:19 smithi122 ceph-mon[93869]: pgmap v21953: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 539 KiB/s rd, 296 KiB/s wr, 9 op/s 2023-03-10T02:08:19.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:19 smithi181 ceph-mon[94189]: pgmap v21953: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 539 KiB/s rd, 296 KiB/s wr, 9 op/s 2023-03-10T02:08:19.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:19 smithi038 ceph-mon[92422]: pgmap v21953: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 539 KiB/s rd, 296 KiB/s wr, 9 op/s 2023-03-10T02:08:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:08:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:08:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:08:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:08:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:08:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:08:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:21 smithi122 ceph-mon[93869]: pgmap v21954: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 539 KiB/s rd, 296 KiB/s wr, 8 op/s 2023-03-10T02:08:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40211.561476 secs 2023-03-10T02:08:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:08:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:08:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:21 smithi181 ceph-mon[94189]: pgmap v21954: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 539 KiB/s rd, 296 KiB/s wr, 8 op/s 2023-03-10T02:08:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40211.561476 secs 2023-03-10T02:08:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:08:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:08:21.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:21 smithi038 ceph-mon[92422]: pgmap v21954: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 539 KiB/s rd, 296 KiB/s wr, 8 op/s 2023-03-10T02:08:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40211.561476 secs 2023-03-10T02:08:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:08:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:08:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:23 smithi122 ceph-mon[93869]: pgmap v21955: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 869 KiB/s wr, 14 op/s 2023-03-10T02:08:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:23 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 40214.025195 secs 2023-03-10T02:08:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:23 smithi181 ceph-mon[94189]: pgmap v21955: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 869 KiB/s wr, 14 op/s 2023-03-10T02:08:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:23 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 40214.025195 secs 2023-03-10T02:08:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:23 smithi038 ceph-mon[92422]: pgmap v21955: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 869 KiB/s wr, 14 op/s 2023-03-10T02:08:23.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:23 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 40214.025195 secs 2023-03-10T02:08:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40675.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:08:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:25 smithi122 ceph-mon[93869]: pgmap v21956: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 565 KiB/s rd, 584 KiB/s wr, 10 op/s 2023-03-10T02:08:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:25 smithi122 ceph-mon[93869]: osdmap e1587: 12 total, 12 up, 12 in 2023-03-10T02:08:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:25 smithi181 ceph-mon[94189]: pgmap v21956: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 565 KiB/s rd, 584 KiB/s wr, 10 op/s 2023-03-10T02:08:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:25 smithi181 ceph-mon[94189]: osdmap e1587: 12 total, 12 up, 12 in 2023-03-10T02:08:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:25 smithi038 ceph-mon[92422]: pgmap v21956: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 565 KiB/s rd, 584 KiB/s wr, 10 op/s 2023-03-10T02:08:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:25 smithi038 ceph-mon[92422]: osdmap e1587: 12 total, 12 up, 12 in 2023-03-10T02:08:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40216.561534 secs 2023-03-10T02:08:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40216.561534 secs 2023-03-10T02:08:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40216.561534 secs 2023-03-10T02:08:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:27 smithi122 ceph-mon[93869]: pgmap v21958: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 884 KiB/s rd, 907 KiB/s wr, 14 op/s 2023-03-10T02:08:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:27 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:08:27.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:27 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:08:27.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:27 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:08:27.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:27 smithi181 ceph-mon[94189]: pgmap v21958: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 884 KiB/s rd, 907 KiB/s wr, 14 op/s 2023-03-10T02:08:27.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:27 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:08:27.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:27 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:08:27.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:27 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:08:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:27 smithi038 ceph-mon[92422]: pgmap v21958: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 884 KiB/s rd, 907 KiB/s wr, 14 op/s 2023-03-10T02:08:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:27 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:08:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:27 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:08:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:27 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:08:28.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:28 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 40219.025257 secs 2023-03-10T02:08:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:28 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 40219.025257 secs 2023-03-10T02:08:28.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:28 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 40219.025257 secs 2023-03-10T02:08:28.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40680.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:08:29.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:29 smithi122 ceph-mon[93869]: pgmap v21959: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 883 KiB/s rd, 894 KiB/s wr, 12 op/s 2023-03-10T02:08:29.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:29 smithi181 ceph-mon[94189]: pgmap v21959: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 883 KiB/s rd, 894 KiB/s wr, 12 op/s 2023-03-10T02:08:29.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:29 smithi038 ceph-mon[92422]: pgmap v21959: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 883 KiB/s rd, 894 KiB/s wr, 12 op/s 2023-03-10T02:08:31.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:31 smithi181 ceph-mon[94189]: pgmap v21960: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 883 KiB/s rd, 895 KiB/s wr, 12 op/s 2023-03-10T02:08:31.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40221.561611 secs 2023-03-10T02:08:31.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:31 smithi038 ceph-mon[92422]: pgmap v21960: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 883 KiB/s rd, 895 KiB/s wr, 12 op/s 2023-03-10T02:08:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40221.561611 secs 2023-03-10T02:08:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:31 smithi122 ceph-mon[93869]: pgmap v21960: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 883 KiB/s rd, 895 KiB/s wr, 12 op/s 2023-03-10T02:08:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40221.561611 secs 2023-03-10T02:08:32.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:32 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:08:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:32 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:08:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:08:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:08:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:08:32.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:08:32.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:32 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:08:32.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:32 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:08:32.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:32 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:08:32.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:08:32.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:08:32.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:08:32.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:08:32.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:32 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:08:32.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:32 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:08:32.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:32 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:08:32.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:08:32.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:08:32.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:08:32.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:08:32.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:32 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:08:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:33 smithi038 ceph-mon[92422]: pgmap v21961: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 298 KiB/s wr, 9 op/s 2023-03-10T02:08:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:33 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:08:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:33 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:08:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:33 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 40224.025321 secs 2023-03-10T02:08:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:08:33.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:08:33.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:08:33.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40685.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:08:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:33 smithi122 ceph-mon[93869]: pgmap v21961: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 298 KiB/s wr, 9 op/s 2023-03-10T02:08:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:33 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:08:33.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:33 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:08:33.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:33 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 40224.025321 secs 2023-03-10T02:08:33.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:08:33.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:08:33.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:08:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:33 smithi181 ceph-mon[94189]: pgmap v21961: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 298 KiB/s wr, 9 op/s 2023-03-10T02:08:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:33 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:08:33.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:33 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:08:33.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:33 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 40224.025321 secs 2023-03-10T02:08:33.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:08:33.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:08:33.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:08:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:35 smithi122 ceph-mon[93869]: pgmap v21962: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 298 KiB/s wr, 9 op/s 2023-03-10T02:08:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:35 smithi122 ceph-mon[93869]: osdmap e1588: 12 total, 12 up, 12 in 2023-03-10T02:08:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:35 smithi181 ceph-mon[94189]: pgmap v21962: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 298 KiB/s wr, 9 op/s 2023-03-10T02:08:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:35 smithi181 ceph-mon[94189]: osdmap e1588: 12 total, 12 up, 12 in 2023-03-10T02:08:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:35 smithi038 ceph-mon[92422]: pgmap v21962: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 298 KiB/s wr, 9 op/s 2023-03-10T02:08:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:35 smithi038 ceph-mon[92422]: osdmap e1588: 12 total, 12 up, 12 in 2023-03-10T02:08:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40226.561674 secs 2023-03-10T02:08:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:08:36.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:08:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40226.561674 secs 2023-03-10T02:08:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:08:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:08:36.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40226.561674 secs 2023-03-10T02:08:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:08:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:08:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:37 smithi122 ceph-mon[93869]: pgmap v21964: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 695 KiB/s rd, 544 KiB/s wr, 8 op/s 2023-03-10T02:08:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:08:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:08:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:37 smithi181 ceph-mon[94189]: pgmap v21964: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 695 KiB/s rd, 544 KiB/s wr, 8 op/s 2023-03-10T02:08:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:08:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:08:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:37 smithi038 ceph-mon[92422]: pgmap v21964: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 695 KiB/s rd, 544 KiB/s wr, 8 op/s 2023-03-10T02:08:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:08:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:08:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40690.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:08:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40229.025381 secs 2023-03-10T02:08:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40229.025381 secs 2023-03-10T02:08:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40229.025381 secs 2023-03-10T02:08:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:39 smithi122 ceph-mon[93869]: pgmap v21965: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 918 KiB/s wr, 10 op/s 2023-03-10T02:08:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:39 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:08:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:39 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:08:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:39 smithi181 ceph-mon[94189]: pgmap v21965: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 918 KiB/s wr, 10 op/s 2023-03-10T02:08:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:39 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:08:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:39 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:08:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:39 smithi038 ceph-mon[92422]: pgmap v21965: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 918 KiB/s wr, 10 op/s 2023-03-10T02:08:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:39 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:08:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:39 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:08:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:08:40.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:08:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:08:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:08:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:08:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:08:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:41 smithi122 ceph-mon[93869]: pgmap v21966: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 917 KiB/s wr, 10 op/s 2023-03-10T02:08:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40231.561758 secs 2023-03-10T02:08:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:08:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:08:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:41 smithi181 ceph-mon[94189]: pgmap v21966: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 917 KiB/s wr, 10 op/s 2023-03-10T02:08:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40231.561758 secs 2023-03-10T02:08:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:08:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:08:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:41 smithi038 ceph-mon[92422]: pgmap v21966: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 917 KiB/s wr, 10 op/s 2023-03-10T02:08:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40231.561758 secs 2023-03-10T02:08:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:08:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:08:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40695.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:08:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:43 smithi122 ceph-mon[93869]: pgmap v21967: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 11 op/s 2023-03-10T02:08:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40234.025465 secs 2023-03-10T02:08:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:43 smithi181 ceph-mon[94189]: pgmap v21967: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 11 op/s 2023-03-10T02:08:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40234.025465 secs 2023-03-10T02:08:43.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:43 smithi038 ceph-mon[92422]: pgmap v21967: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 11 op/s 2023-03-10T02:08:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40234.025465 secs 2023-03-10T02:08:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:45 smithi122 ceph-mon[93869]: pgmap v21968: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 11 op/s 2023-03-10T02:08:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:45 smithi181 ceph-mon[94189]: pgmap v21968: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 11 op/s 2023-03-10T02:08:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:45 smithi038 ceph-mon[92422]: pgmap v21968: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 11 op/s 2023-03-10T02:08:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40236.561818 secs 2023-03-10T02:08:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40236.561818 secs 2023-03-10T02:08:46.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40236.561818 secs 2023-03-10T02:08:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:47 smithi122 ceph-mon[93869]: pgmap v21969: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 14 op/s 2023-03-10T02:08:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:47 smithi181 ceph-mon[94189]: pgmap v21969: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 14 op/s 2023-03-10T02:08:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:47 smithi038 ceph-mon[92422]: pgmap v21969: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 14 op/s 2023-03-10T02:08:48.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40700.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:08:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:08:49.137 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:08:49.138 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:48 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 40239.025592 secs 2023-03-10T02:08:49.139 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:08:49.139 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:08:49.139 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:48 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 40239.025592 secs 2023-03-10T02:08:49.140 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:08:49.140 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:08:49.140 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:48 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 40239.025592 secs 2023-03-10T02:08:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:49 smithi122 ceph-mon[93869]: pgmap v21970: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 871 KiB/s rd, 869 KiB/s wr, 9 op/s 2023-03-10T02:08:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:08:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:08:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:49 smithi181 ceph-mon[94189]: pgmap v21970: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 871 KiB/s rd, 869 KiB/s wr, 9 op/s 2023-03-10T02:08:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:08:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:08:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:49 smithi038 ceph-mon[92422]: pgmap v21970: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 871 KiB/s rd, 869 KiB/s wr, 9 op/s 2023-03-10T02:08:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:08:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:08:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:51 smithi122 ceph-mon[93869]: pgmap v21971: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 560 KiB/s rd, 558 KiB/s wr, 8 op/s 2023-03-10T02:08:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40241.561904 secs 2023-03-10T02:08:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:08:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:08:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:51 smithi181 ceph-mon[94189]: pgmap v21971: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 560 KiB/s rd, 558 KiB/s wr, 8 op/s 2023-03-10T02:08:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40241.561904 secs 2023-03-10T02:08:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:08:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:08:51.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:51 smithi038 ceph-mon[92422]: pgmap v21971: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 560 KiB/s rd, 558 KiB/s wr, 8 op/s 2023-03-10T02:08:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40241.561904 secs 2023-03-10T02:08:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:08:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:08:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:08:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:08:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:08:52.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:08:52.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:08:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:08:53.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40705.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:08:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:53 smithi122 ceph-mon[93869]: pgmap v21972: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 770 KiB/s rd, 768 KiB/s wr, 13 op/s 2023-03-10T02:08:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:53 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 40244.025652 secs 2023-03-10T02:08:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:53 smithi181 ceph-mon[94189]: pgmap v21972: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 770 KiB/s rd, 768 KiB/s wr, 13 op/s 2023-03-10T02:08:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:53 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 40244.025652 secs 2023-03-10T02:08:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:53 smithi038 ceph-mon[92422]: pgmap v21972: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 770 KiB/s rd, 768 KiB/s wr, 13 op/s 2023-03-10T02:08:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:53 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 40244.025652 secs 2023-03-10T02:08:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:55 smithi122 ceph-mon[93869]: pgmap v21973: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 598 KiB/s rd, 596 KiB/s wr, 9 op/s 2023-03-10T02:08:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:55 smithi181 ceph-mon[94189]: pgmap v21973: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 598 KiB/s rd, 596 KiB/s wr, 9 op/s 2023-03-10T02:08:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:55 smithi038 ceph-mon[92422]: pgmap v21973: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 598 KiB/s rd, 596 KiB/s wr, 9 op/s 2023-03-10T02:08:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40246.561964 secs 2023-03-10T02:08:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:08:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:08:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40246.561964 secs 2023-03-10T02:08:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:08:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:08:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40246.561964 secs 2023-03-10T02:08:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:08:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:08:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:57 smithi122 ceph-mon[93869]: pgmap v21974: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 869 KiB/s rd, 864 KiB/s wr, 13 op/s 2023-03-10T02:08:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:08:57.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:08:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:57 smithi181 ceph-mon[94189]: pgmap v21974: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 869 KiB/s rd, 864 KiB/s wr, 13 op/s 2023-03-10T02:08:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:08:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:08:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:57 smithi038 ceph-mon[92422]: pgmap v21974: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 869 KiB/s rd, 864 KiB/s wr, 13 op/s 2023-03-10T02:08:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:08:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:08:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40710.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:08:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40249.025712 secs 2023-03-10T02:08:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40249.025712 secs 2023-03-10T02:08:58.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40249.025712 secs 2023-03-10T02:08:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:59 smithi122 ceph-mon[93869]: pgmap v21975: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 479 KiB/s wr, 11 op/s 2023-03-10T02:08:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:08:59.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:08:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:08:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:59 smithi181 ceph-mon[94189]: pgmap v21975: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 479 KiB/s wr, 11 op/s 2023-03-10T02:08:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:08:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:08:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:08:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:59 smithi038 ceph-mon[92422]: pgmap v21975: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 479 KiB/s wr, 11 op/s 2023-03-10T02:08:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:08:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:08:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:09:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:09:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:09:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:09:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:09:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:09:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:09:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:01 smithi122 ceph-mon[93869]: pgmap v21976: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 494 KiB/s rd, 479 KiB/s wr, 10 op/s 2023-03-10T02:09:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40251.562026 secs 2023-03-10T02:09:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:09:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:09:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:01 smithi181 ceph-mon[94189]: pgmap v21976: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 494 KiB/s rd, 479 KiB/s wr, 10 op/s 2023-03-10T02:09:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40251.562026 secs 2023-03-10T02:09:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:09:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:09:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:01 smithi038 ceph-mon[92422]: pgmap v21976: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 494 KiB/s rd, 479 KiB/s wr, 10 op/s 2023-03-10T02:09:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40251.562026 secs 2023-03-10T02:09:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:09:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:09:02.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:02 smithi122 ceph-mon[93869]: pgmap v21977: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 972 KiB/s rd, 956 KiB/s wr, 17 op/s 2023-03-10T02:09:02.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:02 smithi181 ceph-mon[94189]: pgmap v21977: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 972 KiB/s rd, 956 KiB/s wr, 17 op/s 2023-03-10T02:09:02.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:02 smithi038 ceph-mon[92422]: pgmap v21977: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 972 KiB/s rd, 956 KiB/s wr, 17 op/s 2023-03-10T02:09:03.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40715.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:09:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40254.029250 secs 2023-03-10T02:09:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40254.029250 secs 2023-03-10T02:09:03.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40254.029250 secs 2023-03-10T02:09:04.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:04 smithi122 ceph-mon[93869]: pgmap v21978: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 846 KiB/s rd, 835 KiB/s wr, 24 op/s 2023-03-10T02:09:04.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:04 smithi181 ceph-mon[94189]: pgmap v21978: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 846 KiB/s rd, 835 KiB/s wr, 24 op/s 2023-03-10T02:09:04.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:04 smithi038 ceph-mon[92422]: pgmap v21978: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 846 KiB/s rd, 835 KiB/s wr, 24 op/s 2023-03-10T02:09:06.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40256.562085 secs 2023-03-10T02:09:06.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40256.562085 secs 2023-03-10T02:09:06.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40256.562085 secs 2023-03-10T02:09:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:07 smithi122 ceph-mon[93869]: pgmap v21979: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 847 KiB/s rd, 835 KiB/s wr, 27 op/s 2023-03-10T02:09:07.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:07 smithi181 ceph-mon[94189]: pgmap v21979: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 847 KiB/s rd, 835 KiB/s wr, 27 op/s 2023-03-10T02:09:07.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:07 smithi038 ceph-mon[92422]: pgmap v21979: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 847 KiB/s rd, 835 KiB/s wr, 27 op/s 2023-03-10T02:09:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:09:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:09:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40259.029411 secs 2023-03-10T02:09:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:09:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:09:08.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40259.029411 secs 2023-03-10T02:09:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:09:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:09:08.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40259.029411 secs 2023-03-10T02:09:08.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40720.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:09:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:09 smithi122 ceph-mon[93869]: pgmap v21980: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 577 KiB/s rd, 568 KiB/s wr, 23 op/s 2023-03-10T02:09:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:09:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:09:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:09 smithi181 ceph-mon[94189]: pgmap v21980: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 577 KiB/s rd, 568 KiB/s wr, 23 op/s 2023-03-10T02:09:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:09:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:09:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:09 smithi038 ceph-mon[92422]: pgmap v21980: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 577 KiB/s rd, 568 KiB/s wr, 23 op/s 2023-03-10T02:09:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:09:09.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:09:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:11 smithi122 ceph-mon[93869]: pgmap v21981: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 577 KiB/s rd, 568 KiB/s wr, 23 op/s 2023-03-10T02:09:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:11 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40261.562155 secs 2023-03-10T02:09:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:11 smithi181 ceph-mon[94189]: pgmap v21981: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 577 KiB/s rd, 568 KiB/s wr, 23 op/s 2023-03-10T02:09:11.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:11 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40261.562155 secs 2023-03-10T02:09:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:11 smithi038 ceph-mon[92422]: pgmap v21981: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 577 KiB/s rd, 568 KiB/s wr, 23 op/s 2023-03-10T02:09:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:11 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40261.562155 secs 2023-03-10T02:09:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:13 smithi122 ceph-mon[93869]: pgmap v21982: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 567 KiB/s rd, 568 KiB/s wr, 26 op/s 2023-03-10T02:09:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40264.029471 secs 2023-03-10T02:09:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:13 smithi181 ceph-mon[94189]: pgmap v21982: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 567 KiB/s rd, 568 KiB/s wr, 26 op/s 2023-03-10T02:09:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40264.029471 secs 2023-03-10T02:09:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:13 smithi038 ceph-mon[92422]: pgmap v21982: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 567 KiB/s rd, 568 KiB/s wr, 26 op/s 2023-03-10T02:09:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40264.029471 secs 2023-03-10T02:09:13.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40725.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:09:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:15 smithi122 ceph-mon[93869]: pgmap v21983: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 288 KiB/s wr, 20 op/s 2023-03-10T02:09:15.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:15 smithi181 ceph-mon[94189]: pgmap v21983: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 288 KiB/s wr, 20 op/s 2023-03-10T02:09:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:15 smithi038 ceph-mon[92422]: pgmap v21983: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 288 KiB/s wr, 20 op/s 2023-03-10T02:09:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:16 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40266.562214 secs 2023-03-10T02:09:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:16 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40266.562214 secs 2023-03-10T02:09:16.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:16 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40266.562214 secs 2023-03-10T02:09:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:17 smithi122 ceph-mon[93869]: pgmap v21984: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 200 KiB/s wr, 12 op/s 2023-03-10T02:09:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:17 smithi181 ceph-mon[94189]: pgmap v21984: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 200 KiB/s wr, 12 op/s 2023-03-10T02:09:17.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:17 smithi038 ceph-mon[92422]: pgmap v21984: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 200 KiB/s wr, 12 op/s 2023-03-10T02:09:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40269.029529 secs 2023-03-10T02:09:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40269.029529 secs 2023-03-10T02:09:18.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40269.029529 secs 2023-03-10T02:09:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40730.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:09:19.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:19 smithi122 ceph-mon[93869]: pgmap v21985: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 275 KiB/s wr, 10 op/s 2023-03-10T02:09:19.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:19 smithi181 ceph-mon[94189]: pgmap v21985: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 275 KiB/s wr, 10 op/s 2023-03-10T02:09:19.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:19 smithi038 ceph-mon[92422]: pgmap v21985: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 275 KiB/s wr, 10 op/s 2023-03-10T02:09:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:09:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:09:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:09:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:09:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:09:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:09:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:21 smithi122 ceph-mon[93869]: pgmap v21986: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 424 KiB/s wr, 10 op/s 2023-03-10T02:09:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:21 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40271.562273 secs 2023-03-10T02:09:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:09:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:09:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:21 smithi181 ceph-mon[94189]: pgmap v21986: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 424 KiB/s wr, 10 op/s 2023-03-10T02:09:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:21 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40271.562273 secs 2023-03-10T02:09:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:09:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:09:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:21 smithi038 ceph-mon[92422]: pgmap v21986: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 424 KiB/s wr, 10 op/s 2023-03-10T02:09:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:21 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40271.562273 secs 2023-03-10T02:09:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:09:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:09:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:23 smithi038 ceph-mon[92422]: pgmap v21987: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 471 KiB/s wr, 13 op/s 2023-03-10T02:09:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40274.029587 secs 2023-03-10T02:09:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40735.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:09:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:23 smithi122 ceph-mon[93869]: pgmap v21987: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 471 KiB/s wr, 13 op/s 2023-03-10T02:09:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40274.029587 secs 2023-03-10T02:09:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:23 smithi181 ceph-mon[94189]: pgmap v21987: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 471 KiB/s wr, 13 op/s 2023-03-10T02:09:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40274.029587 secs 2023-03-10T02:09:25.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:25 smithi038 ceph-mon[92422]: pgmap v21988: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 471 KiB/s wr, 11 op/s 2023-03-10T02:09:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:25 smithi038 ceph-mon[92422]: osdmap e1589: 12 total, 12 up, 12 in 2023-03-10T02:09:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:25 smithi122 ceph-mon[93869]: pgmap v21988: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 471 KiB/s wr, 11 op/s 2023-03-10T02:09:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:25 smithi122 ceph-mon[93869]: osdmap e1589: 12 total, 12 up, 12 in 2023-03-10T02:09:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:25 smithi181 ceph-mon[94189]: pgmap v21988: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 451 KiB/s rd, 471 KiB/s wr, 11 op/s 2023-03-10T02:09:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:25 smithi181 ceph-mon[94189]: osdmap e1589: 12 total, 12 up, 12 in 2023-03-10T02:09:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:26 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40276.562358 secs 2023-03-10T02:09:26.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:26 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40276.562358 secs 2023-03-10T02:09:26.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:26 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40276.562358 secs 2023-03-10T02:09:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:27 smithi122 ceph-mon[93869]: pgmap v21990: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 476 KiB/s rd, 499 KiB/s wr, 12 op/s 2023-03-10T02:09:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:27 smithi181 ceph-mon[94189]: pgmap v21990: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 476 KiB/s rd, 499 KiB/s wr, 12 op/s 2023-03-10T02:09:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:27 smithi038 ceph-mon[92422]: pgmap v21990: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 476 KiB/s rd, 499 KiB/s wr, 12 op/s 2023-03-10T02:09:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40740.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:09:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40279.029698 secs 2023-03-10T02:09:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40279.029698 secs 2023-03-10T02:09:29.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40279.029698 secs 2023-03-10T02:09:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:29 smithi122 ceph-mon[93869]: pgmap v21991: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 409 KiB/s wr, 10 op/s 2023-03-10T02:09:29.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:29 smithi181 ceph-mon[94189]: pgmap v21991: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 409 KiB/s wr, 10 op/s 2023-03-10T02:09:29.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:29 smithi038 ceph-mon[92422]: pgmap v21991: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 475 KiB/s rd, 409 KiB/s wr, 10 op/s 2023-03-10T02:09:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:31 smithi122 ceph-mon[93869]: pgmap v21992: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 435 KiB/s wr, 11 op/s 2023-03-10T02:09:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:31 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40281.562415 secs 2023-03-10T02:09:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:31 smithi181 ceph-mon[94189]: pgmap v21992: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 435 KiB/s wr, 11 op/s 2023-03-10T02:09:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:31 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40281.562415 secs 2023-03-10T02:09:31.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:31 smithi038 ceph-mon[92422]: pgmap v21992: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 435 KiB/s wr, 11 op/s 2023-03-10T02:09:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:31 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40281.562415 secs 2023-03-10T02:09:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40745.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:09:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:33 smithi122 ceph-mon[93869]: pgmap v21993: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 421 KiB/s wr, 10 op/s 2023-03-10T02:09:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:33 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:09:33.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:33 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:09:33.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40284.029855 secs 2023-03-10T02:09:33.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:09:33.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:09:33.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:09:33.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:33 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:09:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:33 smithi181 ceph-mon[94189]: pgmap v21993: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 421 KiB/s wr, 10 op/s 2023-03-10T02:09:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:33 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:09:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:33 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:09:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40284.029855 secs 2023-03-10T02:09:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:09:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:09:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:09:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:33 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:09:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:33 smithi038 ceph-mon[92422]: pgmap v21993: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 421 KiB/s wr, 10 op/s 2023-03-10T02:09:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:33 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:09:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:33 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:09:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40284.029855 secs 2023-03-10T02:09:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:09:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:09:34.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:09:34.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:33 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:09:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:35 smithi122 ceph-mon[93869]: pgmap v21994: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 790 KiB/s rd, 788 KiB/s wr, 10 op/s 2023-03-10T02:09:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:35 smithi122 ceph-mon[93869]: osdmap e1590: 12 total, 12 up, 12 in 2023-03-10T02:09:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:35 smithi181 ceph-mon[94189]: pgmap v21994: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 790 KiB/s rd, 788 KiB/s wr, 10 op/s 2023-03-10T02:09:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:35 smithi181 ceph-mon[94189]: osdmap e1590: 12 total, 12 up, 12 in 2023-03-10T02:09:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:35 smithi038 ceph-mon[92422]: pgmap v21994: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 790 KiB/s rd, 788 KiB/s wr, 10 op/s 2023-03-10T02:09:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:35 smithi038 ceph-mon[92422]: osdmap e1590: 12 total, 12 up, 12 in 2023-03-10T02:09:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:36 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40286.562474 secs 2023-03-10T02:09:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:09:36.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:09:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:36 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40286.562474 secs 2023-03-10T02:09:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:09:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:09:36.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:36 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40286.562474 secs 2023-03-10T02:09:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:09:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:09:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:37 smithi122 ceph-mon[93869]: pgmap v21996: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 922 KiB/s rd, 921 KiB/s wr, 10 op/s 2023-03-10T02:09:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:09:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:09:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:37 smithi181 ceph-mon[94189]: pgmap v21996: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 922 KiB/s rd, 921 KiB/s wr, 10 op/s 2023-03-10T02:09:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:09:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:09:37.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:37 smithi038 ceph-mon[92422]: pgmap v21996: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 922 KiB/s rd, 921 KiB/s wr, 10 op/s 2023-03-10T02:09:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:09:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:09:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40750.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:09:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40289.029914 secs 2023-03-10T02:09:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40289.029914 secs 2023-03-10T02:09:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40289.029914 secs 2023-03-10T02:09:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:39 smithi122 ceph-mon[93869]: pgmap v21997: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 922 KiB/s rd, 921 KiB/s wr, 10 op/s 2023-03-10T02:09:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:39 smithi181 ceph-mon[94189]: pgmap v21997: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 922 KiB/s rd, 921 KiB/s wr, 10 op/s 2023-03-10T02:09:39.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:39 smithi038 ceph-mon[92422]: pgmap v21997: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 922 KiB/s rd, 921 KiB/s wr, 10 op/s 2023-03-10T02:09:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:09:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:09:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:09:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:09:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:09:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:09:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:41 smithi122 ceph-mon[93869]: pgmap v21998: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 759 KiB/s rd, 757 KiB/s wr, 10 op/s 2023-03-10T02:09:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:41 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40291.562530 secs 2023-03-10T02:09:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:41 smithi181 ceph-mon[94189]: pgmap v21998: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 759 KiB/s rd, 757 KiB/s wr, 10 op/s 2023-03-10T02:09:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:41 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40291.562530 secs 2023-03-10T02:09:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:41 smithi038 ceph-mon[92422]: pgmap v21998: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 759 KiB/s rd, 757 KiB/s wr, 10 op/s 2023-03-10T02:09:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:41 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40291.562530 secs 2023-03-10T02:09:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40755.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:09:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:43 smithi122 ceph-mon[93869]: pgmap v21999: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 11 op/s 2023-03-10T02:09:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40294.030000 secs 2023-03-10T02:09:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:43 smithi181 ceph-mon[94189]: pgmap v21999: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 11 op/s 2023-03-10T02:09:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40294.030000 secs 2023-03-10T02:09:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:43 smithi038 ceph-mon[92422]: pgmap v21999: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 11 op/s 2023-03-10T02:09:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40294.030000 secs 2023-03-10T02:09:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:45 smithi122 ceph-mon[93869]: pgmap v22000: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 721 KiB/s rd, 718 KiB/s wr, 10 op/s 2023-03-10T02:09:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:45 smithi181 ceph-mon[94189]: pgmap v22000: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 721 KiB/s rd, 718 KiB/s wr, 10 op/s 2023-03-10T02:09:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:45 smithi038 ceph-mon[92422]: pgmap v22000: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 721 KiB/s rd, 718 KiB/s wr, 10 op/s 2023-03-10T02:09:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40296.562587 secs 2023-03-10T02:09:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40296.562587 secs 2023-03-10T02:09:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40296.562587 secs 2023-03-10T02:09:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:47 smithi122 ceph-mon[93869]: pgmap v22001: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 831 KiB/s rd, 827 KiB/s wr, 12 op/s 2023-03-10T02:09:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:47 smithi181 ceph-mon[94189]: pgmap v22001: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 831 KiB/s rd, 827 KiB/s wr, 12 op/s 2023-03-10T02:09:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:47 smithi038 ceph-mon[92422]: pgmap v22001: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 831 KiB/s rd, 827 KiB/s wr, 12 op/s 2023-03-10T02:09:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40760.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:09:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40299.030056 secs 2023-03-10T02:09:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:09:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:09:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40299.030056 secs 2023-03-10T02:09:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:09:48.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:09:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40299.030056 secs 2023-03-10T02:09:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:09:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:09:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:49 smithi122 ceph-mon[93869]: pgmap v22002: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 734 KiB/s rd, 732 KiB/s wr, 9 op/s 2023-03-10T02:09:49.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:09:49.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:09:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:49 smithi181 ceph-mon[94189]: pgmap v22002: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 734 KiB/s rd, 732 KiB/s wr, 9 op/s 2023-03-10T02:09:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:09:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:09:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:49 smithi038 ceph-mon[92422]: pgmap v22002: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 734 KiB/s rd, 732 KiB/s wr, 9 op/s 2023-03-10T02:09:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:09:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:09:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:51 smithi122 ceph-mon[93869]: pgmap v22003: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 734 KiB/s rd, 732 KiB/s wr, 10 op/s 2023-03-10T02:09:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40301.562674 secs 2023-03-10T02:09:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:51 smithi181 ceph-mon[94189]: pgmap v22003: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 734 KiB/s rd, 732 KiB/s wr, 10 op/s 2023-03-10T02:09:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40301.562674 secs 2023-03-10T02:09:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:51 smithi038 ceph-mon[92422]: pgmap v22003: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 734 KiB/s rd, 732 KiB/s wr, 10 op/s 2023-03-10T02:09:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40301.562674 secs 2023-03-10T02:09:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:09:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:09:52.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:52 smithi122 ceph-mon[93869]: pgmap v22004: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 911 KiB/s rd, 908 KiB/s wr, 12 op/s 2023-03-10T02:09:52.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:09:52.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:09:52.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:52 smithi181 ceph-mon[94189]: pgmap v22004: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 911 KiB/s rd, 908 KiB/s wr, 12 op/s 2023-03-10T02:09:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:09:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:09:52.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:52 smithi038 ceph-mon[92422]: pgmap v22004: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 911 KiB/s rd, 908 KiB/s wr, 12 op/s 2023-03-10T02:09:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40765.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:09:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40304.030118 secs 2023-03-10T02:09:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:09:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:09:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40304.030118 secs 2023-03-10T02:09:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:09:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:09:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40304.030118 secs 2023-03-10T02:09:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:09:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:09:54.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:54 smithi122 ceph-mon[93869]: pgmap v22005: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 601 KiB/s rd, 599 KiB/s wr, 9 op/s 2023-03-10T02:09:54.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:54 smithi181 ceph-mon[94189]: pgmap v22005: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 601 KiB/s rd, 599 KiB/s wr, 9 op/s 2023-03-10T02:09:54.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:54 smithi038 ceph-mon[92422]: pgmap v22005: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 601 KiB/s rd, 599 KiB/s wr, 9 op/s 2023-03-10T02:09:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:09:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:09:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:09:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:09:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:09:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:09:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40306.562733 secs 2023-03-10T02:09:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:56 smithi122 ceph-mon[93869]: pgmap v22006: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 792 KiB/s rd, 788 KiB/s wr, 13 op/s 2023-03-10T02:09:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40306.562733 secs 2023-03-10T02:09:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:56 smithi181 ceph-mon[94189]: pgmap v22006: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 792 KiB/s rd, 788 KiB/s wr, 13 op/s 2023-03-10T02:09:56.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40306.562733 secs 2023-03-10T02:09:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:56 smithi038 ceph-mon[92422]: pgmap v22006: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 792 KiB/s rd, 788 KiB/s wr, 13 op/s 2023-03-10T02:09:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:57 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T02:09:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:57 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T02:09:57.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:57 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T02:09:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40770.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:09:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:58 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:09:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:58 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:09:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:58 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:09:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:58 smithi122 ceph-mon[93869]: pgmap v22007: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 434 KiB/s wr, 10 op/s 2023-03-10T02:09:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:58 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:09:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:58 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:09:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:58 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:09:58.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:58 smithi181 ceph-mon[94189]: pgmap v22007: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 434 KiB/s wr, 10 op/s 2023-03-10T02:09:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:58 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:09:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:58 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:09:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:58 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:09:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:58 smithi038 ceph-mon[92422]: pgmap v22007: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 434 KiB/s wr, 10 op/s 2023-03-10T02:09:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:09:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:09:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:09:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:09:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:09:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:09:59.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:09:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:09:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:10:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:00 smithi038 ceph-mon[92422]: Health detail: HEALTH_WARN 1 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-10T02:10:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:00 smithi038 ceph-mon[92422]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2023-03-10T02:10:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:00 smithi038 ceph-mon[92422]: mds.h(mds.1): 2 slow requests are blocked > 180 secs 2023-03-10T02:10:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:00 smithi038 ceph-mon[92422]: [WRN] MDS_TRIM: 2 MDSs behind on trimming 2023-03-10T02:10:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:00 smithi038 ceph-mon[92422]: mds.e(mds.0): Behind on trimming (541/128) max_segments: 128, num_segments: 541 2023-03-10T02:10:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:00 smithi038 ceph-mon[92422]: mds.b(mds.0): Behind on trimming (540/128) max_segments: 128, num_segments: 540 2023-03-10T02:10:01.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:00 smithi038 ceph-mon[92422]: pgmap v22008: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 401 KiB/s wr, 9 op/s 2023-03-10T02:10:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:00 smithi122 ceph-mon[93869]: Health detail: HEALTH_WARN 1 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-10T02:10:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:00 smithi122 ceph-mon[93869]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2023-03-10T02:10:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:00 smithi122 ceph-mon[93869]: mds.h(mds.1): 2 slow requests are blocked > 180 secs 2023-03-10T02:10:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:00 smithi122 ceph-mon[93869]: [WRN] MDS_TRIM: 2 MDSs behind on trimming 2023-03-10T02:10:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:00 smithi122 ceph-mon[93869]: mds.e(mds.0): Behind on trimming (541/128) max_segments: 128, num_segments: 541 2023-03-10T02:10:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:00 smithi122 ceph-mon[93869]: mds.b(mds.0): Behind on trimming (540/128) max_segments: 128, num_segments: 540 2023-03-10T02:10:01.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:00 smithi122 ceph-mon[93869]: pgmap v22008: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 401 KiB/s wr, 9 op/s 2023-03-10T02:10:01.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:00 smithi181 ceph-mon[94189]: Health detail: HEALTH_WARN 1 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-10T02:10:01.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:00 smithi181 ceph-mon[94189]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2023-03-10T02:10:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:00 smithi181 ceph-mon[94189]: mds.h(mds.1): 2 slow requests are blocked > 180 secs 2023-03-10T02:10:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:00 smithi181 ceph-mon[94189]: [WRN] MDS_TRIM: 2 MDSs behind on trimming 2023-03-10T02:10:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:00 smithi181 ceph-mon[94189]: mds.e(mds.0): Behind on trimming (541/128) max_segments: 128, num_segments: 541 2023-03-10T02:10:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:00 smithi181 ceph-mon[94189]: mds.b(mds.0): Behind on trimming (540/128) max_segments: 128, num_segments: 540 2023-03-10T02:10:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:00 smithi181 ceph-mon[94189]: pgmap v22008: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 415 KiB/s rd, 401 KiB/s wr, 9 op/s 2023-03-10T02:10:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40311.562792 secs 2023-03-10T02:10:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:10:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:10:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40311.562792 secs 2023-03-10T02:10:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:10:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:10:02.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40311.562792 secs 2023-03-10T02:10:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:10:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:10:02.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:02 smithi038 ceph-mon[92422]: pgmap v22009: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 790 KiB/s wr, 15 op/s 2023-03-10T02:10:03.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:02 smithi122 ceph-mon[93869]: pgmap v22009: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 790 KiB/s wr, 15 op/s 2023-03-10T02:10:03.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:02 smithi181 ceph-mon[94189]: pgmap v22009: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 790 KiB/s wr, 15 op/s 2023-03-10T02:10:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40775.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:10:05.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:05 smithi122 ceph-mon[93869]: pgmap v22010: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 931 KiB/s wr, 23 op/s 2023-03-10T02:10:05.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:05 smithi181 ceph-mon[94189]: pgmap v22010: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 931 KiB/s wr, 23 op/s 2023-03-10T02:10:05.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:05 smithi038 ceph-mon[92422]: pgmap v22010: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 931 KiB/s wr, 23 op/s 2023-03-10T02:10:06.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:06 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40316.562878 secs 2023-03-10T02:10:06.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:06 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40316.562878 secs 2023-03-10T02:10:06.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:06 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40316.562878 secs 2023-03-10T02:10:07.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:07 smithi122 ceph-mon[93869]: pgmap v22011: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 931 KiB/s wr, 26 op/s 2023-03-10T02:10:07.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:07 smithi181 ceph-mon[94189]: pgmap v22011: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 931 KiB/s wr, 26 op/s 2023-03-10T02:10:07.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:07 smithi038 ceph-mon[92422]: pgmap v22011: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 931 KiB/s wr, 26 op/s 2023-03-10T02:10:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:10:08.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:10:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:10:08.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:10:08.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:10:08.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:10:08.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40780.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:10:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:09 smithi122 ceph-mon[93869]: pgmap v22012: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 995 KiB/s wr, 23 op/s 2023-03-10T02:10:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:10:09.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:10:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:09 smithi181 ceph-mon[94189]: pgmap v22012: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 995 KiB/s wr, 23 op/s 2023-03-10T02:10:09.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:10:09.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:10:09.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:09 smithi038 ceph-mon[92422]: pgmap v22012: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 995 KiB/s wr, 23 op/s 2023-03-10T02:10:09.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:10:09.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:10:10.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40321.562934 secs 2023-03-10T02:10:10.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40321.562934 secs 2023-03-10T02:10:10.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40321.562934 secs 2023-03-10T02:10:11.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:11 smithi122 ceph-mon[93869]: pgmap v22013: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 995 KiB/s wr, 22 op/s 2023-03-10T02:10:11.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:11 smithi181 ceph-mon[94189]: pgmap v22013: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 995 KiB/s wr, 22 op/s 2023-03-10T02:10:11.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:11 smithi038 ceph-mon[92422]: pgmap v22013: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 995 KiB/s wr, 22 op/s 2023-03-10T02:10:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:10:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:10:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:10:12.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:10:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:10:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:10:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:13 smithi122 ceph-mon[93869]: pgmap v22014: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.1 MiB/s wr, 25 op/s 2023-03-10T02:10:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:13 smithi181 ceph-mon[94189]: pgmap v22014: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.1 MiB/s wr, 25 op/s 2023-03-10T02:10:13.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:13 smithi038 ceph-mon[92422]: pgmap v22014: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.1 MiB/s wr, 25 op/s 2023-03-10T02:10:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40785.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:10:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:15 smithi122 ceph-mon[93869]: pgmap v22015: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 804 KiB/s wr, 20 op/s 2023-03-10T02:10:15.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40326.562980 secs 2023-03-10T02:10:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:15 smithi181 ceph-mon[94189]: pgmap v22015: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 804 KiB/s wr, 20 op/s 2023-03-10T02:10:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40326.562980 secs 2023-03-10T02:10:15.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:15 smithi038 ceph-mon[92422]: pgmap v22015: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 804 KiB/s wr, 20 op/s 2023-03-10T02:10:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40326.562980 secs 2023-03-10T02:10:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:17 smithi122 ceph-mon[93869]: pgmap v22016: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 639 KiB/s rd, 452 KiB/s wr, 10 op/s 2023-03-10T02:10:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:17 smithi181 ceph-mon[94189]: pgmap v22016: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 639 KiB/s rd, 452 KiB/s wr, 10 op/s 2023-03-10T02:10:17.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:17 smithi038 ceph-mon[92422]: pgmap v22016: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 639 KiB/s rd, 452 KiB/s wr, 10 op/s 2023-03-10T02:10:18.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40790.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:10:19.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:19 smithi122 ceph-mon[93869]: pgmap v22017: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 795 KiB/s rd, 619 KiB/s wr, 10 op/s 2023-03-10T02:10:19.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:19 smithi181 ceph-mon[94189]: pgmap v22017: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 795 KiB/s rd, 619 KiB/s wr, 10 op/s 2023-03-10T02:10:19.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:19 smithi038 ceph-mon[92422]: pgmap v22017: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 795 KiB/s rd, 619 KiB/s wr, 10 op/s 2023-03-10T02:10:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:10:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:10:20.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40331.563034 secs 2023-03-10T02:10:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:10:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:10:20.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40331.563034 secs 2023-03-10T02:10:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:10:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:10:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40331.563034 secs 2023-03-10T02:10:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:21 smithi038 ceph-mon[92422]: pgmap v22018: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 541 KiB/s rd, 366 KiB/s wr, 9 op/s 2023-03-10T02:10:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:10:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:10:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:21 smithi122 ceph-mon[93869]: pgmap v22018: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 541 KiB/s rd, 366 KiB/s wr, 9 op/s 2023-03-10T02:10:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:10:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:10:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:21 smithi181 ceph-mon[94189]: pgmap v22018: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 541 KiB/s rd, 366 KiB/s wr, 9 op/s 2023-03-10T02:10:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:10:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:10:23.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:23 smithi038 ceph-mon[92422]: pgmap v22019: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 626 KiB/s rd, 644 KiB/s wr, 12 op/s 2023-03-10T02:10:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40795.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:10:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:23 smithi122 ceph-mon[93869]: pgmap v22019: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 626 KiB/s rd, 644 KiB/s wr, 12 op/s 2023-03-10T02:10:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:23 smithi181 ceph-mon[94189]: pgmap v22019: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 626 KiB/s rd, 644 KiB/s wr, 12 op/s 2023-03-10T02:10:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:25 smithi122 ceph-mon[93869]: pgmap v22020: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 676 KiB/s wr, 11 op/s 2023-03-10T02:10:25.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:25 smithi122 ceph-mon[93869]: osdmap e1591: 12 total, 12 up, 12 in 2023-03-10T02:10:25.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40336.563094 secs 2023-03-10T02:10:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:25 smithi181 ceph-mon[94189]: pgmap v22020: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 676 KiB/s wr, 11 op/s 2023-03-10T02:10:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:25 smithi181 ceph-mon[94189]: osdmap e1591: 12 total, 12 up, 12 in 2023-03-10T02:10:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40336.563094 secs 2023-03-10T02:10:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:25 smithi038 ceph-mon[92422]: pgmap v22020: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 657 KiB/s rd, 676 KiB/s wr, 11 op/s 2023-03-10T02:10:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:25 smithi038 ceph-mon[92422]: osdmap e1591: 12 total, 12 up, 12 in 2023-03-10T02:10:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40336.563094 secs 2023-03-10T02:10:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:27 smithi122 ceph-mon[93869]: pgmap v22022: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 689 KiB/s rd, 712 KiB/s wr, 12 op/s 2023-03-10T02:10:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:27 smithi181 ceph-mon[94189]: pgmap v22022: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 689 KiB/s rd, 712 KiB/s wr, 12 op/s 2023-03-10T02:10:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:27 smithi038 ceph-mon[92422]: pgmap v22022: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 689 KiB/s rd, 712 KiB/s wr, 12 op/s 2023-03-10T02:10:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40800.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:10:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:29 smithi122 ceph-mon[93869]: pgmap v22023: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 550 KiB/s rd, 562 KiB/s wr, 10 op/s 2023-03-10T02:10:29.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:29 smithi181 ceph-mon[94189]: pgmap v22023: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 550 KiB/s rd, 562 KiB/s wr, 10 op/s 2023-03-10T02:10:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:29 smithi038 ceph-mon[92422]: pgmap v22023: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 550 KiB/s rd, 562 KiB/s wr, 10 op/s 2023-03-10T02:10:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40341.563163 secs 2023-03-10T02:10:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40341.563163 secs 2023-03-10T02:10:30.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40341.563163 secs 2023-03-10T02:10:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:31 smithi122 ceph-mon[93869]: pgmap v22024: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 707 KiB/s rd, 719 KiB/s wr, 11 op/s 2023-03-10T02:10:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:31 smithi181 ceph-mon[94189]: pgmap v22024: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 707 KiB/s rd, 719 KiB/s wr, 11 op/s 2023-03-10T02:10:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:31 smithi038 ceph-mon[92422]: pgmap v22024: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 707 KiB/s rd, 719 KiB/s wr, 11 op/s 2023-03-10T02:10:32.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:32 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:10:32.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:32 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:10:32.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:32 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:10:32.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:32 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:10:32.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:32 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:10:32.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:32 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:10:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40805.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:10:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:33 smithi122 ceph-mon[93869]: pgmap v22025: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 423 KiB/s wr, 9 op/s 2023-03-10T02:10:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:33 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:10:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:33 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:10:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:33 smithi181 ceph-mon[94189]: pgmap v22025: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 423 KiB/s wr, 9 op/s 2023-03-10T02:10:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:33 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:10:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:33 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:10:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:33 smithi038 ceph-mon[92422]: pgmap v22025: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 423 KiB/s wr, 9 op/s 2023-03-10T02:10:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:33 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:10:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:33 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:10:34.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:34 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:10:34.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:34 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:10:34.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:34 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:10:34.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:34 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:10:34.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:34 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:10:34.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:34 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:10:34.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:34 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:10:34.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:34 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:10:34.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:34 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:10:34.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:34 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:10:34.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:34 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:10:34.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:34 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:10:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:35 smithi122 ceph-mon[93869]: pgmap v22026: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 251 KiB/s wr, 8 op/s 2023-03-10T02:10:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40346.563222 secs 2023-03-10T02:10:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:35 smithi122 ceph-mon[93869]: osdmap e1592: 12 total, 12 up, 12 in 2023-03-10T02:10:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:35 smithi181 ceph-mon[94189]: pgmap v22026: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 251 KiB/s wr, 8 op/s 2023-03-10T02:10:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40346.563222 secs 2023-03-10T02:10:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:35 smithi181 ceph-mon[94189]: osdmap e1592: 12 total, 12 up, 12 in 2023-03-10T02:10:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:35 smithi038 ceph-mon[92422]: pgmap v22026: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 251 KiB/s wr, 8 op/s 2023-03-10T02:10:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40346.563222 secs 2023-03-10T02:10:36.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:35 smithi038 ceph-mon[92422]: osdmap e1592: 12 total, 12 up, 12 in 2023-03-10T02:10:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:37 smithi122 ceph-mon[93869]: pgmap v22028: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 505 KiB/s rd, 503 KiB/s wr, 10 op/s 2023-03-10T02:10:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:37 smithi181 ceph-mon[94189]: pgmap v22028: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 505 KiB/s rd, 503 KiB/s wr, 10 op/s 2023-03-10T02:10:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:37 smithi038 ceph-mon[92422]: pgmap v22028: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 505 KiB/s rd, 503 KiB/s wr, 10 op/s 2023-03-10T02:10:38.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40810.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:10:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40349.030739 secs 2023-03-10T02:10:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40349.030739 secs 2023-03-10T02:10:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40349.030739 secs 2023-03-10T02:10:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:39 smithi122 ceph-mon[93869]: pgmap v22029: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 455 KiB/s rd, 453 KiB/s wr, 10 op/s 2023-03-10T02:10:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:39 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:10:39.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:39 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:10:39.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:39 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:10:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:39 smithi181 ceph-mon[94189]: pgmap v22029: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 455 KiB/s rd, 453 KiB/s wr, 10 op/s 2023-03-10T02:10:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:39 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:10:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:39 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:10:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:39 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:10:39.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:39 smithi038 ceph-mon[92422]: pgmap v22029: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 455 KiB/s rd, 453 KiB/s wr, 10 op/s 2023-03-10T02:10:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:39 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:10:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:39 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:10:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:39 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:10:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40351.563324 secs 2023-03-10T02:10:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40351.563324 secs 2023-03-10T02:10:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40351.563324 secs 2023-03-10T02:10:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:41 smithi122 ceph-mon[93869]: pgmap v22030: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 336 KiB/s rd, 334 KiB/s wr, 10 op/s 2023-03-10T02:10:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:41 smithi181 ceph-mon[94189]: pgmap v22030: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 336 KiB/s rd, 334 KiB/s wr, 10 op/s 2023-03-10T02:10:41.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:41 smithi038 ceph-mon[92422]: pgmap v22030: 129 pgs: 129 active+clean; 11 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 336 KiB/s rd, 334 KiB/s wr, 10 op/s 2023-03-10T02:10:43.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40815.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:10:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:43 smithi122 ceph-mon[93869]: pgmap v22031: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 542 KiB/s rd, 540 KiB/s wr, 11 op/s 2023-03-10T02:10:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40354.030798 secs 2023-03-10T02:10:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:43 smithi181 ceph-mon[94189]: pgmap v22031: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 542 KiB/s rd, 540 KiB/s wr, 11 op/s 2023-03-10T02:10:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40354.030798 secs 2023-03-10T02:10:43.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:43 smithi038 ceph-mon[92422]: pgmap v22031: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 542 KiB/s rd, 540 KiB/s wr, 11 op/s 2023-03-10T02:10:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40354.030798 secs 2023-03-10T02:10:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:45 smithi122 ceph-mon[93869]: pgmap v22032: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 542 KiB/s rd, 629 KiB/s wr, 10 op/s 2023-03-10T02:10:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40356.563383 secs 2023-03-10T02:10:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:45 smithi181 ceph-mon[94189]: pgmap v22032: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 542 KiB/s rd, 629 KiB/s wr, 10 op/s 2023-03-10T02:10:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40356.563383 secs 2023-03-10T02:10:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:45 smithi038 ceph-mon[92422]: pgmap v22032: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 542 KiB/s rd, 629 KiB/s wr, 10 op/s 2023-03-10T02:10:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40356.563383 secs 2023-03-10T02:10:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:47 smithi122 ceph-mon[93869]: pgmap v22033: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 388 KiB/s wr, 12 op/s 2023-03-10T02:10:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:47 smithi181 ceph-mon[94189]: pgmap v22033: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 388 KiB/s wr, 12 op/s 2023-03-10T02:10:47.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:47 smithi038 ceph-mon[92422]: pgmap v22033: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 388 KiB/s wr, 12 op/s 2023-03-10T02:10:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40820.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:10:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:10:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:10:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:10:48.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:10:48.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:10:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:10:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:49 smithi122 ceph-mon[93869]: pgmap v22034: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 623 KiB/s wr, 9 op/s 2023-03-10T02:10:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:10:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:10:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:49 smithi181 ceph-mon[94189]: pgmap v22034: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 623 KiB/s wr, 9 op/s 2023-03-10T02:10:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:10:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:10:49.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:49 smithi038 ceph-mon[92422]: pgmap v22034: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 623 KiB/s wr, 9 op/s 2023-03-10T02:10:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:10:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:10:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:50 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T02:10:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40361.563444 secs 2023-03-10T02:10:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:50 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T02:10:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40361.563444 secs 2023-03-10T02:10:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:50 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T02:10:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40361.563444 secs 2023-03-10T02:10:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:51 smithi122 ceph-mon[93869]: pgmap v22035: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 623 KiB/s wr, 8 op/s 2023-03-10T02:10:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:51 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:10:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:10:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:10:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:51 smithi181 ceph-mon[94189]: pgmap v22035: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 623 KiB/s wr, 8 op/s 2023-03-10T02:10:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:51 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:10:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:10:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:10:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:51 smithi038 ceph-mon[92422]: pgmap v22035: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 623 KiB/s wr, 8 op/s 2023-03-10T02:10:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:51 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:10:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:10:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:10:53.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40825.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:10:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:53 smithi122 ceph-mon[93869]: pgmap v22036: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 713 KiB/s wr, 11 op/s 2023-03-10T02:10:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:53 smithi181 ceph-mon[94189]: pgmap v22036: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 713 KiB/s wr, 11 op/s 2023-03-10T02:10:53.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:53 smithi038 ceph-mon[92422]: pgmap v22036: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 713 KiB/s wr, 11 op/s 2023-03-10T02:10:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:55 smithi122 ceph-mon[93869]: pgmap v22037: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 605 KiB/s rd, 603 KiB/s wr, 9 op/s 2023-03-10T02:10:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40366.563507 secs 2023-03-10T02:10:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:55 smithi181 ceph-mon[94189]: pgmap v22037: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 605 KiB/s rd, 603 KiB/s wr, 9 op/s 2023-03-10T02:10:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40366.563507 secs 2023-03-10T02:10:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:55 smithi038 ceph-mon[92422]: pgmap v22037: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 605 KiB/s rd, 603 KiB/s wr, 9 op/s 2023-03-10T02:10:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40366.563507 secs 2023-03-10T02:10:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:10:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:10:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:10:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:10:56.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:10:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:10:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:57 smithi122 ceph-mon[93869]: pgmap v22038: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 650 KiB/s rd, 570 KiB/s wr, 12 op/s 2023-03-10T02:10:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:10:57.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:10:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:57 smithi181 ceph-mon[94189]: pgmap v22038: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 650 KiB/s rd, 570 KiB/s wr, 12 op/s 2023-03-10T02:10:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:10:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:10:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:57 smithi038 ceph-mon[92422]: pgmap v22038: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 650 KiB/s rd, 570 KiB/s wr, 12 op/s 2023-03-10T02:10:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:10:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:10:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40830.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:10:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40369.030981 secs 2023-03-10T02:10:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40369.030981 secs 2023-03-10T02:10:59.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40369.030981 secs 2023-03-10T02:10:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:59 smithi122 ceph-mon[93869]: pgmap v22039: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 904 KiB/s rd, 826 KiB/s wr, 12 op/s 2023-03-10T02:10:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:59 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:10:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:10:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:10:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:10:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:59 smithi181 ceph-mon[94189]: pgmap v22039: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 904 KiB/s rd, 826 KiB/s wr, 12 op/s 2023-03-10T02:10:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:59 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:10:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:10:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:10:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:10:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:59 smithi038 ceph-mon[92422]: pgmap v22039: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 904 KiB/s rd, 826 KiB/s wr, 12 op/s 2023-03-10T02:10:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:59 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:10:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:10:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:10:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:11:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40371.563561 secs 2023-03-10T02:11:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40371.563561 secs 2023-03-10T02:11:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40371.563561 secs 2023-03-10T02:11:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:01 smithi122 ceph-mon[93869]: pgmap v22040: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 526 KiB/s rd, 512 KiB/s wr, 10 op/s 2023-03-10T02:11:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:11:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:11:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:01 smithi181 ceph-mon[94189]: pgmap v22040: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 526 KiB/s rd, 512 KiB/s wr, 10 op/s 2023-03-10T02:11:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:11:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:11:01.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:01 smithi038 ceph-mon[92422]: pgmap v22040: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 526 KiB/s rd, 512 KiB/s wr, 10 op/s 2023-03-10T02:11:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:11:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:11:02.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:02 smithi122 ceph-mon[93869]: pgmap v22041: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 898 KiB/s rd, 882 KiB/s wr, 15 op/s 2023-03-10T02:11:02.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:02 smithi181 ceph-mon[94189]: pgmap v22041: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 898 KiB/s rd, 882 KiB/s wr, 15 op/s 2023-03-10T02:11:02.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:02 smithi038 ceph-mon[92422]: pgmap v22041: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 898 KiB/s rd, 882 KiB/s wr, 15 op/s 2023-03-10T02:11:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40835.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:11:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40374.031074 secs 2023-03-10T02:11:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:03 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:11:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:03 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:11:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40374.031074 secs 2023-03-10T02:11:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:03 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:11:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:03 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:11:03.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40374.031074 secs 2023-03-10T02:11:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:03 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:11:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:03 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:11:04.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:04 smithi122 ceph-mon[93869]: pgmap v22042: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 23 op/s 2023-03-10T02:11:04.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:04 smithi181 ceph-mon[94189]: pgmap v22042: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 23 op/s 2023-03-10T02:11:04.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:04 smithi038 ceph-mon[92422]: pgmap v22042: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 23 op/s 2023-03-10T02:11:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40376.563617 secs 2023-03-10T02:11:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40376.563617 secs 2023-03-10T02:11:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40376.563617 secs 2023-03-10T02:11:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:06 smithi122 ceph-mon[93869]: pgmap v22043: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.0 MiB/s wr, 26 op/s 2023-03-10T02:11:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:06 smithi181 ceph-mon[94189]: pgmap v22043: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.0 MiB/s wr, 26 op/s 2023-03-10T02:11:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:06 smithi038 ceph-mon[92422]: pgmap v22043: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.0 MiB/s wr, 26 op/s 2023-03-10T02:11:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:11:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:11:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:11:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:11:07.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:11:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:11:08.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40840.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:11:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40379.031132 secs 2023-03-10T02:11:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:08 smithi122 ceph-mon[93869]: pgmap v22044: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.1 MiB/s wr, 24 op/s 2023-03-10T02:11:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40379.031132 secs 2023-03-10T02:11:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:08 smithi181 ceph-mon[94189]: pgmap v22044: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.1 MiB/s wr, 24 op/s 2023-03-10T02:11:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40379.031132 secs 2023-03-10T02:11:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:08 smithi038 ceph-mon[92422]: pgmap v22044: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.1 MiB/s wr, 24 op/s 2023-03-10T02:11:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:11:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:11:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:11:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:11:09.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:11:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:11:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:10 smithi122 ceph-mon[93869]: pgmap v22045: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 908 KiB/s wr, 23 op/s 2023-03-10T02:11:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40381.563675 secs 2023-03-10T02:11:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:10 smithi181 ceph-mon[94189]: pgmap v22045: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 908 KiB/s wr, 23 op/s 2023-03-10T02:11:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40381.563675 secs 2023-03-10T02:11:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:10 smithi038 ceph-mon[92422]: pgmap v22045: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 908 KiB/s wr, 23 op/s 2023-03-10T02:11:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40381.563675 secs 2023-03-10T02:11:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:11:12.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:11:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:11:12.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:11:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:11:12.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:11:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:13 smithi122 ceph-mon[93869]: pgmap v22046: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.0 MiB/s wr, 26 op/s 2023-03-10T02:11:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40384.031192 secs 2023-03-10T02:11:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:13 smithi181 ceph-mon[94189]: pgmap v22046: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.0 MiB/s wr, 26 op/s 2023-03-10T02:11:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40384.031192 secs 2023-03-10T02:11:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:13 smithi038 ceph-mon[92422]: pgmap v22046: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.0 MiB/s wr, 26 op/s 2023-03-10T02:11:13.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40384.031192 secs 2023-03-10T02:11:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40845.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:11:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:15 smithi122 ceph-mon[93869]: pgmap v22047: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 676 KiB/s wr, 21 op/s 2023-03-10T02:11:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40386.563735 secs 2023-03-10T02:11:15.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:15 smithi181 ceph-mon[94189]: pgmap v22047: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 676 KiB/s wr, 21 op/s 2023-03-10T02:11:15.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40386.563735 secs 2023-03-10T02:11:15.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:15 smithi038 ceph-mon[92422]: pgmap v22047: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 676 KiB/s wr, 21 op/s 2023-03-10T02:11:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40386.563735 secs 2023-03-10T02:11:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:17 smithi122 ceph-mon[93869]: pgmap v22048: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 661 KiB/s rd, 324 KiB/s wr, 11 op/s 2023-03-10T02:11:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:17 smithi181 ceph-mon[94189]: pgmap v22048: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 661 KiB/s rd, 324 KiB/s wr, 11 op/s 2023-03-10T02:11:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:17 smithi038 ceph-mon[92422]: pgmap v22048: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 661 KiB/s rd, 324 KiB/s wr, 11 op/s 2023-03-10T02:11:18.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40389.031254 secs 2023-03-10T02:11:18.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40389.031254 secs 2023-03-10T02:11:18.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40389.031254 secs 2023-03-10T02:11:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40850.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:11:19.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:19 smithi122 ceph-mon[93869]: pgmap v22049: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 660 KiB/s rd, 323 KiB/s wr, 9 op/s 2023-03-10T02:11:19.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:19 smithi181 ceph-mon[94189]: pgmap v22049: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 660 KiB/s rd, 323 KiB/s wr, 9 op/s 2023-03-10T02:11:19.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:19 smithi038 ceph-mon[92422]: pgmap v22049: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 660 KiB/s rd, 323 KiB/s wr, 9 op/s 2023-03-10T02:11:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:11:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:11:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40391.563792 secs 2023-03-10T02:11:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:11:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:11:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40391.563792 secs 2023-03-10T02:11:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:11:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:11:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40391.563792 secs 2023-03-10T02:11:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:21 smithi122 ceph-mon[93869]: pgmap v22050: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 486 KiB/s rd, 149 KiB/s wr, 8 op/s 2023-03-10T02:11:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:11:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:11:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:21 smithi181 ceph-mon[94189]: pgmap v22050: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 486 KiB/s rd, 149 KiB/s wr, 8 op/s 2023-03-10T02:11:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:11:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:11:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:21 smithi038 ceph-mon[92422]: pgmap v22050: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 486 KiB/s rd, 149 KiB/s wr, 8 op/s 2023-03-10T02:11:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:11:21.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:11:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:23 smithi122 ceph-mon[93869]: pgmap v22051: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 414 KiB/s wr, 12 op/s 2023-03-10T02:11:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40394.031320 secs 2023-03-10T02:11:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:23 smithi181 ceph-mon[94189]: pgmap v22051: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 414 KiB/s wr, 12 op/s 2023-03-10T02:11:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40394.031320 secs 2023-03-10T02:11:23.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:23 smithi038 ceph-mon[92422]: pgmap v22051: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 396 KiB/s rd, 414 KiB/s wr, 12 op/s 2023-03-10T02:11:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40394.031320 secs 2023-03-10T02:11:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40855.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:11:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:25 smithi038 ceph-mon[92422]: pgmap v22052: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 276 KiB/s wr, 9 op/s 2023-03-10T02:11:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:25 smithi038 ceph-mon[92422]: osdmap e1593: 12 total, 12 up, 12 in 2023-03-10T02:11:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40396.563850 secs 2023-03-10T02:11:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:25 smithi122 ceph-mon[93869]: pgmap v22052: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 276 KiB/s wr, 9 op/s 2023-03-10T02:11:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:25 smithi122 ceph-mon[93869]: osdmap e1593: 12 total, 12 up, 12 in 2023-03-10T02:11:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40396.563850 secs 2023-03-10T02:11:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:25 smithi181 ceph-mon[94189]: pgmap v22052: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 276 KiB/s wr, 9 op/s 2023-03-10T02:11:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:25 smithi181 ceph-mon[94189]: osdmap e1593: 12 total, 12 up, 12 in 2023-03-10T02:11:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40396.563850 secs 2023-03-10T02:11:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:27 smithi038 ceph-mon[92422]: pgmap v22054: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 332 KiB/s wr, 11 op/s 2023-03-10T02:11:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:27 smithi122 ceph-mon[93869]: pgmap v22054: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 332 KiB/s wr, 11 op/s 2023-03-10T02:11:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:27 smithi181 ceph-mon[94189]: pgmap v22054: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 332 KiB/s wr, 11 op/s 2023-03-10T02:11:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40399.031376 secs 2023-03-10T02:11:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40860.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:11:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40399.031376 secs 2023-03-10T02:11:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40399.031376 secs 2023-03-10T02:11:29.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:29 smithi038 ceph-mon[92422]: pgmap v22055: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 563 KiB/s wr, 11 op/s 2023-03-10T02:11:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:29 smithi122 ceph-mon[93869]: pgmap v22055: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 563 KiB/s wr, 11 op/s 2023-03-10T02:11:29.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:29 smithi181 ceph-mon[94189]: pgmap v22055: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 563 KiB/s wr, 11 op/s 2023-03-10T02:11:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40401.563907 secs 2023-03-10T02:11:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40401.563907 secs 2023-03-10T02:11:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40401.563907 secs 2023-03-10T02:11:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:31 smithi122 ceph-mon[93869]: pgmap v22056: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 563 KiB/s wr, 11 op/s 2023-03-10T02:11:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:31 smithi181 ceph-mon[94189]: pgmap v22056: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 563 KiB/s wr, 11 op/s 2023-03-10T02:11:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:31 smithi038 ceph-mon[92422]: pgmap v22056: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 551 KiB/s rd, 563 KiB/s wr, 11 op/s 2023-03-10T02:11:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40865.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:11:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:33 smithi122 ceph-mon[93869]: pgmap v22057: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 288 KiB/s wr, 9 op/s 2023-03-10T02:11:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:33 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:11:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:33 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:11:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40404.031436 secs 2023-03-10T02:11:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:33 smithi181 ceph-mon[94189]: pgmap v22057: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 288 KiB/s wr, 9 op/s 2023-03-10T02:11:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:33 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:11:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:33 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:11:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40404.031436 secs 2023-03-10T02:11:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:33 smithi038 ceph-mon[92422]: pgmap v22057: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 288 KiB/s wr, 9 op/s 2023-03-10T02:11:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:33 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:11:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:33 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:11:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40404.031436 secs 2023-03-10T02:11:34.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:34 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:11:34.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:34 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:11:34.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:34 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:11:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:35 smithi122 ceph-mon[93869]: pgmap v22058: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 288 KiB/s wr, 9 op/s 2023-03-10T02:11:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40406.563967 secs 2023-03-10T02:11:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:11:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:11:35.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:11:35.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:35 smithi122 ceph-mon[93869]: osdmap e1594: 12 total, 12 up, 12 in 2023-03-10T02:11:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:35 smithi181 ceph-mon[94189]: pgmap v22058: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 288 KiB/s wr, 9 op/s 2023-03-10T02:11:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40406.563967 secs 2023-03-10T02:11:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:11:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:11:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:11:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:35 smithi181 ceph-mon[94189]: osdmap e1594: 12 total, 12 up, 12 in 2023-03-10T02:11:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:35 smithi038 ceph-mon[92422]: pgmap v22058: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 288 KiB/s wr, 9 op/s 2023-03-10T02:11:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40406.563967 secs 2023-03-10T02:11:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:11:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:11:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:11:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:35 smithi038 ceph-mon[92422]: osdmap e1594: 12 total, 12 up, 12 in 2023-03-10T02:11:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:11:36.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:11:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:11:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:11:36.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:11:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:11:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:37 smithi122 ceph-mon[93869]: pgmap v22060: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 823 KiB/s rd, 822 KiB/s wr, 10 op/s 2023-03-10T02:11:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:11:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:11:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:37 smithi181 ceph-mon[94189]: pgmap v22060: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 823 KiB/s rd, 822 KiB/s wr, 10 op/s 2023-03-10T02:11:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:11:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:11:37.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:37 smithi038 ceph-mon[92422]: pgmap v22060: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 823 KiB/s rd, 822 KiB/s wr, 10 op/s 2023-03-10T02:11:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:11:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:11:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40870.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:11:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40409.031525 secs 2023-03-10T02:11:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40409.031525 secs 2023-03-10T02:11:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40409.031525 secs 2023-03-10T02:11:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:39 smithi122 ceph-mon[93869]: pgmap v22061: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 929 KiB/s rd, 926 KiB/s wr, 10 op/s 2023-03-10T02:11:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:39 smithi181 ceph-mon[94189]: pgmap v22061: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 929 KiB/s rd, 926 KiB/s wr, 10 op/s 2023-03-10T02:11:39.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:39 smithi038 ceph-mon[92422]: pgmap v22061: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 929 KiB/s rd, 926 KiB/s wr, 10 op/s 2023-03-10T02:11:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40411.564017 secs 2023-03-10T02:11:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40411.564017 secs 2023-03-10T02:11:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40411.564017 secs 2023-03-10T02:11:41.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:41 smithi122 ceph-mon[93869]: pgmap v22062: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 929 KiB/s rd, 926 KiB/s wr, 10 op/s 2023-03-10T02:11:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:41 smithi181 ceph-mon[94189]: pgmap v22062: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 929 KiB/s rd, 926 KiB/s wr, 10 op/s 2023-03-10T02:11:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:41 smithi038 ceph-mon[92422]: pgmap v22062: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 929 KiB/s rd, 926 KiB/s wr, 10 op/s 2023-03-10T02:11:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40875.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:11:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:43 smithi122 ceph-mon[93869]: pgmap v22063: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 11 op/s 2023-03-10T02:11:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40414.031610 secs 2023-03-10T02:11:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:43 smithi181 ceph-mon[94189]: pgmap v22063: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 11 op/s 2023-03-10T02:11:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40414.031610 secs 2023-03-10T02:11:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:43 smithi038 ceph-mon[92422]: pgmap v22063: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 11 op/s 2023-03-10T02:11:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40414.031610 secs 2023-03-10T02:11:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:45 smithi122 ceph-mon[93869]: pgmap v22064: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 11 op/s 2023-03-10T02:11:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40416.564076 secs 2023-03-10T02:11:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:45 smithi181 ceph-mon[94189]: pgmap v22064: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 11 op/s 2023-03-10T02:11:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40416.564076 secs 2023-03-10T02:11:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:45 smithi038 ceph-mon[92422]: pgmap v22064: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 11 op/s 2023-03-10T02:11:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40416.564076 secs 2023-03-10T02:11:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:47 smithi122 ceph-mon[93869]: pgmap v22065: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 13 op/s 2023-03-10T02:11:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:47 smithi181 ceph-mon[94189]: pgmap v22065: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 13 op/s 2023-03-10T02:11:47.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:47 smithi038 ceph-mon[92422]: pgmap v22065: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 13 op/s 2023-03-10T02:11:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40880.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:11:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:11:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:11:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40419.031684 secs 2023-03-10T02:11:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:11:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:11:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40419.031684 secs 2023-03-10T02:11:48.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:11:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:11:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40419.031684 secs 2023-03-10T02:11:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:49 smithi122 ceph-mon[93869]: pgmap v22066: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 856 KiB/s rd, 854 KiB/s wr, 10 op/s 2023-03-10T02:11:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:11:49.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:11:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:49 smithi181 ceph-mon[94189]: pgmap v22066: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 856 KiB/s rd, 854 KiB/s wr, 10 op/s 2023-03-10T02:11:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:11:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:11:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:49 smithi038 ceph-mon[92422]: pgmap v22066: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 856 KiB/s rd, 854 KiB/s wr, 10 op/s 2023-03-10T02:11:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:11:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:11:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40421.564130 secs 2023-03-10T02:11:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40421.564130 secs 2023-03-10T02:11:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40421.564130 secs 2023-03-10T02:11:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:51 smithi122 ceph-mon[93869]: pgmap v22067: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 565 KiB/s rd, 563 KiB/s wr, 8 op/s 2023-03-10T02:11:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:51 smithi181 ceph-mon[94189]: pgmap v22067: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 565 KiB/s rd, 563 KiB/s wr, 8 op/s 2023-03-10T02:11:51.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:51 smithi038 ceph-mon[92422]: pgmap v22067: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 565 KiB/s rd, 563 KiB/s wr, 8 op/s 2023-03-10T02:11:53.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40885.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:11:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:53 smithi122 ceph-mon[93869]: pgmap v22068: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 780 KiB/s rd, 777 KiB/s wr, 12 op/s 2023-03-10T02:11:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40424.031744 secs 2023-03-10T02:11:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:53 smithi181 ceph-mon[94189]: pgmap v22068: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 780 KiB/s rd, 777 KiB/s wr, 12 op/s 2023-03-10T02:11:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40424.031744 secs 2023-03-10T02:11:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:53 smithi038 ceph-mon[92422]: pgmap v22068: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 780 KiB/s rd, 777 KiB/s wr, 12 op/s 2023-03-10T02:11:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40424.031744 secs 2023-03-10T02:11:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:55 smithi122 ceph-mon[93869]: pgmap v22069: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 606 KiB/s rd, 604 KiB/s wr, 8 op/s 2023-03-10T02:11:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40426.564189 secs 2023-03-10T02:11:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:55 smithi181 ceph-mon[94189]: pgmap v22069: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 606 KiB/s rd, 604 KiB/s wr, 8 op/s 2023-03-10T02:11:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40426.564189 secs 2023-03-10T02:11:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:55 smithi038 ceph-mon[92422]: pgmap v22069: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 606 KiB/s rd, 604 KiB/s wr, 8 op/s 2023-03-10T02:11:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40426.564189 secs 2023-03-10T02:11:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:11:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:11:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:11:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:11:56.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:11:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:11:57.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:57 smithi122 ceph-mon[93869]: pgmap v22070: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 859 KiB/s rd, 854 KiB/s wr, 13 op/s 2023-03-10T02:11:57.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:11:57.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:11:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:57 smithi181 ceph-mon[94189]: pgmap v22070: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 859 KiB/s rd, 854 KiB/s wr, 13 op/s 2023-03-10T02:11:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:11:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:11:57.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:57 smithi038 ceph-mon[92422]: pgmap v22070: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 859 KiB/s rd, 854 KiB/s wr, 13 op/s 2023-03-10T02:11:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:11:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:11:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40890.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:11:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40429.031828 secs 2023-03-10T02:11:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40429.031828 secs 2023-03-10T02:11:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40429.031828 secs 2023-03-10T02:11:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:11:59 smithi122 ceph-mon[93869]: pgmap v22071: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 465 KiB/s wr, 10 op/s 2023-03-10T02:11:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:11:59 smithi181 ceph-mon[94189]: pgmap v22071: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 465 KiB/s wr, 10 op/s 2023-03-10T02:11:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:11:59 smithi038 ceph-mon[92422]: pgmap v22071: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 468 KiB/s rd, 465 KiB/s wr, 10 op/s 2023-03-10T02:12:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:12:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:12:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40431.564246 secs 2023-03-10T02:12:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:12:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:12:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40431.564246 secs 2023-03-10T02:12:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:12:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:12:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40431.564246 secs 2023-03-10T02:12:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:01 smithi122 ceph-mon[93869]: pgmap v22072: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 465 KiB/s wr, 9 op/s 2023-03-10T02:12:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:12:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:12:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:01 smithi181 ceph-mon[94189]: pgmap v22072: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 465 KiB/s wr, 9 op/s 2023-03-10T02:12:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:12:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:12:01.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:01 smithi038 ceph-mon[92422]: pgmap v22072: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 479 KiB/s rd, 465 KiB/s wr, 9 op/s 2023-03-10T02:12:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:12:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:12:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40895.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:12:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:03 smithi122 ceph-mon[93869]: pgmap v22073: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 842 KiB/s wr, 14 op/s 2023-03-10T02:12:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:03 smithi181 ceph-mon[94189]: pgmap v22073: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 842 KiB/s wr, 14 op/s 2023-03-10T02:12:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:03 smithi038 ceph-mon[92422]: pgmap v22073: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 842 KiB/s wr, 14 op/s 2023-03-10T02:12:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:05 smithi122 ceph-mon[93869]: pgmap v22074: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 981 KiB/s wr, 22 op/s 2023-03-10T02:12:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40436.564306 secs 2023-03-10T02:12:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:05 smithi181 ceph-mon[94189]: pgmap v22074: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 981 KiB/s wr, 22 op/s 2023-03-10T02:12:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40436.564306 secs 2023-03-10T02:12:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:05 smithi038 ceph-mon[92422]: pgmap v22074: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 981 KiB/s wr, 22 op/s 2023-03-10T02:12:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40436.564306 secs 2023-03-10T02:12:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:06 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T02:12:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:06 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T02:12:06.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:06 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T02:12:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:07 smithi122 ceph-mon[93869]: pgmap v22075: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 992 KiB/s wr, 27 op/s 2023-03-10T02:12:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:07 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:12:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:12:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:12:07.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:12:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:07 smithi181 ceph-mon[94189]: pgmap v22075: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 992 KiB/s wr, 27 op/s 2023-03-10T02:12:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:07 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:12:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:12:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:12:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:12:07.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:07 smithi038 ceph-mon[92422]: pgmap v22075: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 992 KiB/s wr, 27 op/s 2023-03-10T02:12:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:07 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:12:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:12:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:12:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:12:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40900.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:12:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:09 smithi122 ceph-mon[93869]: pgmap v22076: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 742 KiB/s wr, 23 op/s 2023-03-10T02:12:09.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:12:09.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:12:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:09 smithi181 ceph-mon[94189]: pgmap v22076: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 742 KiB/s wr, 23 op/s 2023-03-10T02:12:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:12:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:12:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:09 smithi038 ceph-mon[92422]: pgmap v22076: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 742 KiB/s wr, 23 op/s 2023-03-10T02:12:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:12:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:12:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40441.564387 secs 2023-03-10T02:12:10.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40441.564387 secs 2023-03-10T02:12:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40441.564387 secs 2023-03-10T02:12:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:11 smithi122 ceph-mon[93869]: pgmap v22077: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 939 KiB/s wr, 23 op/s 2023-03-10T02:12:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:11 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:12:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:11 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:12:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:11 smithi181 ceph-mon[94189]: pgmap v22077: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 939 KiB/s wr, 23 op/s 2023-03-10T02:12:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:11 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:12:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:11 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:12:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:11 smithi038 ceph-mon[92422]: pgmap v22077: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 939 KiB/s wr, 23 op/s 2023-03-10T02:12:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:11 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:12:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:11 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:12:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:12 smithi122 ceph-mon[93869]: pgmap v22078: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 939 KiB/s wr, 26 op/s 2023-03-10T02:12:12.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:12 smithi181 ceph-mon[94189]: pgmap v22078: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 939 KiB/s wr, 26 op/s 2023-03-10T02:12:12.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:12 smithi038 ceph-mon[92422]: pgmap v22078: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 939 KiB/s wr, 26 op/s 2023-03-10T02:12:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40905.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:12:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:15 smithi122 ceph-mon[93869]: pgmap v22079: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 618 KiB/s wr, 22 op/s 2023-03-10T02:12:15.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40446.564447 secs 2023-03-10T02:12:15.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:15 smithi181 ceph-mon[94189]: pgmap v22079: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 618 KiB/s wr, 22 op/s 2023-03-10T02:12:15.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40446.564447 secs 2023-03-10T02:12:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:15 smithi038 ceph-mon[92422]: pgmap v22079: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 618 KiB/s wr, 22 op/s 2023-03-10T02:12:15.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40446.564447 secs 2023-03-10T02:12:17.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:17 smithi122 ceph-mon[93869]: pgmap v22080: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 803 KiB/s rd, 409 KiB/s wr, 14 op/s 2023-03-10T02:12:17.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:17 smithi181 ceph-mon[94189]: pgmap v22080: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 803 KiB/s rd, 409 KiB/s wr, 14 op/s 2023-03-10T02:12:17.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:17 smithi038 ceph-mon[92422]: pgmap v22080: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 803 KiB/s rd, 409 KiB/s wr, 14 op/s 2023-03-10T02:12:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40910.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:12:19.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:19 smithi122 ceph-mon[93869]: pgmap v22081: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 699 KiB/s rd, 409 KiB/s wr, 11 op/s 2023-03-10T02:12:19.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:19 smithi181 ceph-mon[94189]: pgmap v22081: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 699 KiB/s rd, 409 KiB/s wr, 11 op/s 2023-03-10T02:12:19.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:19 smithi038 ceph-mon[92422]: pgmap v22081: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 699 KiB/s rd, 409 KiB/s wr, 11 op/s 2023-03-10T02:12:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:12:20.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:12:20.728 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40451.564503 secs 2023-03-10T02:12:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:12:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:12:20.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40451.564503 secs 2023-03-10T02:12:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:12:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:12:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40451.564503 secs 2023-03-10T02:12:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:21 smithi122 ceph-mon[93869]: pgmap v22082: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 686 KiB/s rd, 694 KiB/s wr, 11 op/s 2023-03-10T02:12:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:12:21.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:12:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:21 smithi181 ceph-mon[94189]: pgmap v22082: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 686 KiB/s rd, 694 KiB/s wr, 11 op/s 2023-03-10T02:12:21.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:12:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:12:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:21 smithi038 ceph-mon[92422]: pgmap v22082: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 686 KiB/s rd, 694 KiB/s wr, 11 op/s 2023-03-10T02:12:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:12:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:12:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:23 smithi122 ceph-mon[93869]: pgmap v22083: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 773 KiB/s rd, 796 KiB/s wr, 15 op/s 2023-03-10T02:12:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:23 smithi181 ceph-mon[94189]: pgmap v22083: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 773 KiB/s rd, 796 KiB/s wr, 15 op/s 2023-03-10T02:12:23.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:23 smithi038 ceph-mon[92422]: pgmap v22083: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 773 KiB/s rd, 796 KiB/s wr, 15 op/s 2023-03-10T02:12:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40915.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:12:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:25 smithi181 ceph-mon[94189]: pgmap v22084: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 772 KiB/s rd, 796 KiB/s wr, 13 op/s 2023-03-10T02:12:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:25 smithi181 ceph-mon[94189]: osdmap e1595: 12 total, 12 up, 12 in 2023-03-10T02:12:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40456.564562 secs 2023-03-10T02:12:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:25 smithi038 ceph-mon[92422]: pgmap v22084: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 772 KiB/s rd, 796 KiB/s wr, 13 op/s 2023-03-10T02:12:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:25 smithi038 ceph-mon[92422]: osdmap e1595: 12 total, 12 up, 12 in 2023-03-10T02:12:25.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40456.564562 secs 2023-03-10T02:12:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:25 smithi122 ceph-mon[93869]: pgmap v22084: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 772 KiB/s rd, 796 KiB/s wr, 13 op/s 2023-03-10T02:12:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:25 smithi122 ceph-mon[93869]: osdmap e1595: 12 total, 12 up, 12 in 2023-03-10T02:12:25.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40456.564562 secs 2023-03-10T02:12:27.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:27 smithi038 ceph-mon[92422]: pgmap v22086: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 799 KiB/s rd, 827 KiB/s wr, 14 op/s 2023-03-10T02:12:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:27 smithi122 ceph-mon[93869]: pgmap v22086: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 799 KiB/s rd, 827 KiB/s wr, 14 op/s 2023-03-10T02:12:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:27 smithi181 ceph-mon[94189]: pgmap v22086: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 799 KiB/s rd, 827 KiB/s wr, 14 op/s 2023-03-10T02:12:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40920.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:12:29.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:29 smithi038 ceph-mon[92422]: pgmap v22087: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 815 KiB/s wr, 12 op/s 2023-03-10T02:12:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:29 smithi122 ceph-mon[93869]: pgmap v22087: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 815 KiB/s wr, 12 op/s 2023-03-10T02:12:29.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:29 smithi181 ceph-mon[94189]: pgmap v22087: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 815 KiB/s wr, 12 op/s 2023-03-10T02:12:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40461.564614 secs 2023-03-10T02:12:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40461.564614 secs 2023-03-10T02:12:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40461.564614 secs 2023-03-10T02:12:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:31 smithi038 ceph-mon[92422]: pgmap v22088: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 569 KiB/s wr, 12 op/s 2023-03-10T02:12:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:31 smithi122 ceph-mon[93869]: pgmap v22088: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 569 KiB/s wr, 12 op/s 2023-03-10T02:12:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:31 smithi181 ceph-mon[94189]: pgmap v22088: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 569 KiB/s wr, 12 op/s 2023-03-10T02:12:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:33 smithi038 ceph-mon[92422]: pgmap v22089: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 252 KiB/s wr, 9 op/s 2023-03-10T02:12:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:33 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:12:33.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:33 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:12:33.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40925.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:12:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:33 smithi122 ceph-mon[93869]: pgmap v22089: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 252 KiB/s wr, 9 op/s 2023-03-10T02:12:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:33 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:12:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:33 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:12:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:33 smithi181 ceph-mon[94189]: pgmap v22089: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 253 KiB/s rd, 252 KiB/s wr, 9 op/s 2023-03-10T02:12:33.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:33 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:12:33.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:33 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:12:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:35 smithi122 ceph-mon[93869]: pgmap v22090: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 566 KiB/s rd, 564 KiB/s wr, 9 op/s 2023-03-10T02:12:35.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40466.564672 secs 2023-03-10T02:12:35.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:35 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:12:35.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:35 smithi122 ceph-mon[93869]: osdmap e1596: 12 total, 12 up, 12 in 2023-03-10T02:12:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:35 smithi181 ceph-mon[94189]: pgmap v22090: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 566 KiB/s rd, 564 KiB/s wr, 9 op/s 2023-03-10T02:12:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40466.564672 secs 2023-03-10T02:12:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:35 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:12:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:35 smithi181 ceph-mon[94189]: osdmap e1596: 12 total, 12 up, 12 in 2023-03-10T02:12:35.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:35 smithi038 ceph-mon[92422]: pgmap v22090: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 566 KiB/s rd, 564 KiB/s wr, 9 op/s 2023-03-10T02:12:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40466.564672 secs 2023-03-10T02:12:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:35 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:12:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:35 smithi038 ceph-mon[92422]: osdmap e1596: 12 total, 12 up, 12 in 2023-03-10T02:12:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:12:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:12:36.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:12:36.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:12:36.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:12:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:12:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:12:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:12:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:12:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:12:36.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:12:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:12:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:12:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:12:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:12:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:37 smithi122 ceph-mon[93869]: pgmap v22092: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 818 KiB/s wr, 10 op/s 2023-03-10T02:12:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:12:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:12:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:37 smithi181 ceph-mon[94189]: pgmap v22092: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 818 KiB/s wr, 10 op/s 2023-03-10T02:12:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:12:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:12:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:37 smithi038 ceph-mon[92422]: pgmap v22092: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 818 KiB/s wr, 10 op/s 2023-03-10T02:12:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:12:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:12:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40930.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:12:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40469.032533 secs 2023-03-10T02:12:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40469.032533 secs 2023-03-10T02:12:38.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40469.032533 secs 2023-03-10T02:12:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:39 smithi122 ceph-mon[93869]: pgmap v22093: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 819 KiB/s rd, 940 KiB/s wr, 10 op/s 2023-03-10T02:12:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:39 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:12:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:39 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:12:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:39 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:12:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:39 smithi181 ceph-mon[94189]: pgmap v22093: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 819 KiB/s rd, 940 KiB/s wr, 10 op/s 2023-03-10T02:12:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:39 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:12:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:39 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:12:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:39 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:12:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:39 smithi038 ceph-mon[92422]: pgmap v22093: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 819 KiB/s rd, 940 KiB/s wr, 10 op/s 2023-03-10T02:12:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:39 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:12:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:39 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:12:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:39 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:12:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40471.564759 secs 2023-03-10T02:12:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40471.564759 secs 2023-03-10T02:12:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40471.564759 secs 2023-03-10T02:12:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:41 smithi122 ceph-mon[93869]: pgmap v22094: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 723 KiB/s rd, 843 KiB/s wr, 10 op/s 2023-03-10T02:12:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:41 smithi181 ceph-mon[94189]: pgmap v22094: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 723 KiB/s rd, 843 KiB/s wr, 10 op/s 2023-03-10T02:12:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:41 smithi038 ceph-mon[92422]: pgmap v22094: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 723 KiB/s rd, 843 KiB/s wr, 10 op/s 2023-03-10T02:12:43.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40935.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:12:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:43 smithi122 ceph-mon[93869]: pgmap v22095: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 11 op/s 2023-03-10T02:12:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40474.032593 secs 2023-03-10T02:12:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:43 smithi181 ceph-mon[94189]: pgmap v22095: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 11 op/s 2023-03-10T02:12:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40474.032593 secs 2023-03-10T02:12:43.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:43 smithi038 ceph-mon[92422]: pgmap v22095: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 11 op/s 2023-03-10T02:12:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40474.032593 secs 2023-03-10T02:12:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:45 smithi122 ceph-mon[93869]: pgmap v22096: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 783 KiB/s wr, 10 op/s 2023-03-10T02:12:45.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40476.564819 secs 2023-03-10T02:12:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:45 smithi181 ceph-mon[94189]: pgmap v22096: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 783 KiB/s wr, 10 op/s 2023-03-10T02:12:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40476.564819 secs 2023-03-10T02:12:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:45 smithi038 ceph-mon[92422]: pgmap v22096: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 783 KiB/s wr, 10 op/s 2023-03-10T02:12:46.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40476.564819 secs 2023-03-10T02:12:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:47 smithi122 ceph-mon[93869]: pgmap v22097: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 800 KiB/s rd, 796 KiB/s wr, 12 op/s 2023-03-10T02:12:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:47 smithi181 ceph-mon[94189]: pgmap v22097: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 800 KiB/s rd, 796 KiB/s wr, 12 op/s 2023-03-10T02:12:47.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:47 smithi038 ceph-mon[92422]: pgmap v22097: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 800 KiB/s rd, 796 KiB/s wr, 12 op/s 2023-03-10T02:12:48.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40940.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:12:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:12:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:12:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:12:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:12:48.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:12:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:12:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:49 smithi122 ceph-mon[93869]: pgmap v22098: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 741 KiB/s rd, 744 KiB/s wr, 9 op/s 2023-03-10T02:12:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:12:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:12:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:49 smithi181 ceph-mon[94189]: pgmap v22098: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 741 KiB/s rd, 744 KiB/s wr, 9 op/s 2023-03-10T02:12:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:12:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:12:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:49 smithi038 ceph-mon[92422]: pgmap v22098: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 741 KiB/s rd, 744 KiB/s wr, 9 op/s 2023-03-10T02:12:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:12:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:12:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:50 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T02:12:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40481.564877 secs 2023-03-10T02:12:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:50 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T02:12:50.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40481.564877 secs 2023-03-10T02:12:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:50 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T02:12:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40481.564877 secs 2023-03-10T02:12:51.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:51 smithi122 ceph-mon[93869]: pgmap v22099: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 741 KiB/s rd, 642 KiB/s wr, 9 op/s 2023-03-10T02:12:51.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:51 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:12:51.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:12:51.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:12:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:51 smithi181 ceph-mon[94189]: pgmap v22099: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 741 KiB/s rd, 642 KiB/s wr, 9 op/s 2023-03-10T02:12:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:51 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:12:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:12:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:12:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:51 smithi038 ceph-mon[92422]: pgmap v22099: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 741 KiB/s rd, 642 KiB/s wr, 9 op/s 2023-03-10T02:12:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:51 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:12:52.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:12:52.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:12:53.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40945.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:12:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:53 smithi122 ceph-mon[93869]: pgmap v22100: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 955 KiB/s rd, 856 KiB/s wr, 13 op/s 2023-03-10T02:12:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:53 smithi181 ceph-mon[94189]: pgmap v22100: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 955 KiB/s rd, 856 KiB/s wr, 13 op/s 2023-03-10T02:12:53.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:53 smithi038 ceph-mon[92422]: pgmap v22100: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 955 KiB/s rd, 856 KiB/s wr, 13 op/s 2023-03-10T02:12:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:55 smithi122 ceph-mon[93869]: pgmap v22101: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 607 KiB/s rd, 609 KiB/s wr, 9 op/s 2023-03-10T02:12:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40486.564938 secs 2023-03-10T02:12:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:55 smithi181 ceph-mon[94189]: pgmap v22101: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 607 KiB/s rd, 609 KiB/s wr, 9 op/s 2023-03-10T02:12:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40486.564938 secs 2023-03-10T02:12:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:55 smithi038 ceph-mon[92422]: pgmap v22101: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 607 KiB/s rd, 609 KiB/s wr, 9 op/s 2023-03-10T02:12:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40486.564938 secs 2023-03-10T02:12:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:12:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:12:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:12:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:12:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:12:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:12:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:57 smithi122 ceph-mon[93869]: pgmap v22102: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 882 KiB/s wr, 13 op/s 2023-03-10T02:12:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:12:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:12:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:57 smithi181 ceph-mon[94189]: pgmap v22102: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 882 KiB/s wr, 13 op/s 2023-03-10T02:12:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:12:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:12:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:57 smithi038 ceph-mon[92422]: pgmap v22102: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 882 KiB/s wr, 13 op/s 2023-03-10T02:12:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:12:58.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:12:58.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40950.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:12:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:12:59 smithi122 ceph-mon[93869]: pgmap v22103: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 557 KiB/s rd, 558 KiB/s wr, 11 op/s 2023-03-10T02:12:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:12:59 smithi181 ceph-mon[94189]: pgmap v22103: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 557 KiB/s rd, 558 KiB/s wr, 11 op/s 2023-03-10T02:12:59.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:12:59 smithi038 ceph-mon[92422]: pgmap v22103: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 557 KiB/s rd, 558 KiB/s wr, 11 op/s 2023-03-10T02:13:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:13:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:13:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40491.565030 secs 2023-03-10T02:13:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:13:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:13:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40491.565030 secs 2023-03-10T02:13:00.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:13:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:13:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40491.565030 secs 2023-03-10T02:13:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:01 smithi122 ceph-mon[93869]: pgmap v22104: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 486 KiB/s wr, 10 op/s 2023-03-10T02:13:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:13:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:13:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:01 smithi181 ceph-mon[94189]: pgmap v22104: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 486 KiB/s wr, 10 op/s 2023-03-10T02:13:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:13:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:13:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:01 smithi038 ceph-mon[92422]: pgmap v22104: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 486 KiB/s wr, 10 op/s 2023-03-10T02:13:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:13:02.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:13:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40955.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:13:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:03 smithi122 ceph-mon[93869]: pgmap v22105: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 15 op/s 2023-03-10T02:13:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:03 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 40494.032940 secs 2023-03-10T02:13:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:03 smithi181 ceph-mon[94189]: pgmap v22105: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 15 op/s 2023-03-10T02:13:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:03 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 40494.032940 secs 2023-03-10T02:13:03.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:03 smithi038 ceph-mon[92422]: pgmap v22105: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 15 op/s 2023-03-10T02:13:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:03 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 40494.032940 secs 2023-03-10T02:13:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:05 smithi122 ceph-mon[93869]: pgmap v22106: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 957 KiB/s rd, 947 KiB/s wr, 22 op/s 2023-03-10T02:13:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40496.565090 secs 2023-03-10T02:13:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:05 smithi181 ceph-mon[94189]: pgmap v22106: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 957 KiB/s rd, 947 KiB/s wr, 22 op/s 2023-03-10T02:13:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40496.565090 secs 2023-03-10T02:13:05.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:05 smithi038 ceph-mon[92422]: pgmap v22106: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 957 KiB/s rd, 947 KiB/s wr, 22 op/s 2023-03-10T02:13:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40496.565090 secs 2023-03-10T02:13:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:06 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:13:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:06 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:13:06.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:13:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:06 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:13:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:06 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:13:06.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:13:06.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:06 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:13:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:06 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:13:06.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:13:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:07 smithi122 ceph-mon[93869]: pgmap v22107: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 960 KiB/s rd, 958 KiB/s wr, 27 op/s 2023-03-10T02:13:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:13:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:13:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:07 smithi181 ceph-mon[94189]: pgmap v22107: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 960 KiB/s rd, 958 KiB/s wr, 27 op/s 2023-03-10T02:13:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:13:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:13:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:07 smithi038 ceph-mon[92422]: pgmap v22107: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 960 KiB/s rd, 958 KiB/s wr, 27 op/s 2023-03-10T02:13:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:13:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:13:08.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40960.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:13:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:08 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 40499.032997 secs 2023-03-10T02:13:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:08 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 40499.032997 secs 2023-03-10T02:13:08.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:08 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 40499.032997 secs 2023-03-10T02:13:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:09 smithi122 ceph-mon[93869]: pgmap v22108: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 750 KiB/s wr, 24 op/s 2023-03-10T02:13:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:13:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:13:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:09 smithi181 ceph-mon[94189]: pgmap v22108: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 750 KiB/s wr, 24 op/s 2023-03-10T02:13:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:13:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:13:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:09 smithi038 ceph-mon[92422]: pgmap v22108: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 750 KiB/s wr, 24 op/s 2023-03-10T02:13:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:13:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:13:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40501.565145 secs 2023-03-10T02:13:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40501.565145 secs 2023-03-10T02:13:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40501.565145 secs 2023-03-10T02:13:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:11 smithi122 ceph-mon[93869]: pgmap v22109: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 750 KiB/s wr, 23 op/s 2023-03-10T02:13:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:11 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:13:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:11 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:13:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:11 smithi181 ceph-mon[94189]: pgmap v22109: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 750 KiB/s wr, 23 op/s 2023-03-10T02:13:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:11 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:13:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:11 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:13:11.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:11 smithi038 ceph-mon[92422]: pgmap v22109: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 750 KiB/s wr, 23 op/s 2023-03-10T02:13:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:11 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:13:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:11 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:13:12.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:12 smithi181 ceph-mon[94189]: pgmap v22110: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 750 KiB/s wr, 26 op/s 2023-03-10T02:13:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:12 smithi038 ceph-mon[92422]: pgmap v22110: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 750 KiB/s wr, 26 op/s 2023-03-10T02:13:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:12 smithi122 ceph-mon[93869]: pgmap v22110: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 750 KiB/s wr, 26 op/s 2023-03-10T02:13:13.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40965.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:13:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:13 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 40504.033133 secs 2023-03-10T02:13:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:13 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 40504.033133 secs 2023-03-10T02:13:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:13 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 40504.033133 secs 2023-03-10T02:13:14.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:14 smithi038 ceph-mon[92422]: pgmap v22111: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 606 KiB/s rd, 287 KiB/s wr, 20 op/s 2023-03-10T02:13:15.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:14 smithi122 ceph-mon[93869]: pgmap v22111: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 606 KiB/s rd, 287 KiB/s wr, 20 op/s 2023-03-10T02:13:15.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:14 smithi181 ceph-mon[94189]: pgmap v22111: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 606 KiB/s rd, 287 KiB/s wr, 20 op/s 2023-03-10T02:13:15.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40506.565205 secs 2023-03-10T02:13:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40506.565205 secs 2023-03-10T02:13:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40506.565205 secs 2023-03-10T02:13:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:16 smithi038 ceph-mon[92422]: pgmap v22112: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 601 KiB/s rd, 276 KiB/s wr, 12 op/s 2023-03-10T02:13:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:16 smithi122 ceph-mon[93869]: pgmap v22112: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 601 KiB/s rd, 276 KiB/s wr, 12 op/s 2023-03-10T02:13:17.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:16 smithi181 ceph-mon[94189]: pgmap v22112: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 601 KiB/s rd, 276 KiB/s wr, 12 op/s 2023-03-10T02:13:18.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40970.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:13:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:19 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 40509.033216 secs 2023-03-10T02:13:19.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:19 smithi122 ceph-mon[93869]: pgmap v22113: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 601 KiB/s rd, 353 KiB/s wr, 9 op/s 2023-03-10T02:13:19.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:19 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 40509.033216 secs 2023-03-10T02:13:19.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:19 smithi181 ceph-mon[94189]: pgmap v22113: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 601 KiB/s rd, 353 KiB/s wr, 9 op/s 2023-03-10T02:13:19.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:19 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 40509.033216 secs 2023-03-10T02:13:19.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:19 smithi038 ceph-mon[92422]: pgmap v22113: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 601 KiB/s rd, 353 KiB/s wr, 9 op/s 2023-03-10T02:13:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40511.565288 secs 2023-03-10T02:13:20.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40511.565288 secs 2023-03-10T02:13:20.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40511.565288 secs 2023-03-10T02:13:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:21 smithi122 ceph-mon[93869]: pgmap v22114: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 288 KiB/s wr, 8 op/s 2023-03-10T02:13:21.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:13:21.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:13:21.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:21 smithi181 ceph-mon[94189]: pgmap v22114: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 288 KiB/s wr, 8 op/s 2023-03-10T02:13:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:13:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:13:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:21 smithi038 ceph-mon[92422]: pgmap v22114: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 288 KiB/s wr, 8 op/s 2023-03-10T02:13:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:13:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:13:23.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:23 smithi122 ceph-mon[93869]: pgmap v22115: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 621 KiB/s wr, 12 op/s 2023-03-10T02:13:23.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:23 smithi181 ceph-mon[94189]: pgmap v22115: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 621 KiB/s wr, 12 op/s 2023-03-10T02:13:23.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:23 smithi038 ceph-mon[92422]: pgmap v22115: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 621 KiB/s wr, 12 op/s 2023-03-10T02:13:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40975.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:13:24.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:24 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 40514.033280 secs 2023-03-10T02:13:24.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:24 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 40514.033280 secs 2023-03-10T02:13:24.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:24 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 40514.033280 secs 2023-03-10T02:13:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:25 smithi122 ceph-mon[93869]: pgmap v22116: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 602 KiB/s rd, 621 KiB/s wr, 10 op/s 2023-03-10T02:13:25.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40516.565346 secs 2023-03-10T02:13:25.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:25 smithi181 ceph-mon[94189]: pgmap v22116: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 602 KiB/s rd, 621 KiB/s wr, 10 op/s 2023-03-10T02:13:25.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40516.565346 secs 2023-03-10T02:13:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:25 smithi038 ceph-mon[92422]: pgmap v22116: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 602 KiB/s rd, 621 KiB/s wr, 10 op/s 2023-03-10T02:13:25.500 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40516.565346 secs 2023-03-10T02:13:26.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:26 smithi181 ceph-mon[94189]: osdmap e1597: 12 total, 12 up, 12 in 2023-03-10T02:13:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:26 smithi038 ceph-mon[92422]: osdmap e1597: 12 total, 12 up, 12 in 2023-03-10T02:13:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:26 smithi122 ceph-mon[93869]: osdmap e1597: 12 total, 12 up, 12 in 2023-03-10T02:13:27.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:27 smithi122 ceph-mon[93869]: pgmap v22118: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 690 KiB/s rd, 714 KiB/s wr, 13 op/s 2023-03-10T02:13:27.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:27 smithi181 ceph-mon[94189]: pgmap v22118: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 690 KiB/s rd, 714 KiB/s wr, 13 op/s 2023-03-10T02:13:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:27 smithi038 ceph-mon[92422]: pgmap v22118: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 690 KiB/s rd, 714 KiB/s wr, 13 op/s 2023-03-10T02:13:28.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:28 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:13:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:28 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:13:28.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:28 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 40519.033367 secs 2023-03-10T02:13:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:28 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:13:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:28 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:13:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:28 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 40519.033367 secs 2023-03-10T02:13:28.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40980.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:13:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:28 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:13:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:28 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:13:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:28 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 40519.033367 secs 2023-03-10T02:13:29.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:29 smithi038 ceph-mon[92422]: pgmap v22119: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 609 KiB/s wr, 10 op/s 2023-03-10T02:13:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:29 smithi122 ceph-mon[93869]: pgmap v22119: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 609 KiB/s wr, 10 op/s 2023-03-10T02:13:29.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:29 smithi181 ceph-mon[94189]: pgmap v22119: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 609 KiB/s wr, 10 op/s 2023-03-10T02:13:30.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40521.565420 secs 2023-03-10T02:13:30.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40521.565420 secs 2023-03-10T02:13:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40521.565420 secs 2023-03-10T02:13:31.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:31 smithi038 ceph-mon[92422]: pgmap v22120: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 609 KiB/s wr, 10 op/s 2023-03-10T02:13:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:31 smithi122 ceph-mon[93869]: pgmap v22120: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 609 KiB/s wr, 10 op/s 2023-03-10T02:13:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:31 smithi181 ceph-mon[94189]: pgmap v22120: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 609 KiB/s wr, 10 op/s 2023-03-10T02:13:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40985.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:13:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:33 smithi122 ceph-mon[93869]: pgmap v22121: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 252 KiB/s wr, 8 op/s 2023-03-10T02:13:33.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:33 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:13:33.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:33 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:13:33.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:33 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 40524.033479 secs 2023-03-10T02:13:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:33 smithi181 ceph-mon[94189]: pgmap v22121: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 252 KiB/s wr, 8 op/s 2023-03-10T02:13:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:33 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:13:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:33 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:13:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:33 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 40524.033479 secs 2023-03-10T02:13:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:33 smithi038 ceph-mon[92422]: pgmap v22121: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 252 KiB/s wr, 8 op/s 2023-03-10T02:13:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:33 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:13:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:33 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:13:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:33 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 40524.033479 secs 2023-03-10T02:13:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:35 smithi122 ceph-mon[93869]: pgmap v22122: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 252 KiB/s wr, 8 op/s 2023-03-10T02:13:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40526.565507 secs 2023-03-10T02:13:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:35 smithi122 ceph-mon[93869]: osdmap e1598: 12 total, 12 up, 12 in 2023-03-10T02:13:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:35 smithi181 ceph-mon[94189]: pgmap v22122: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 252 KiB/s wr, 8 op/s 2023-03-10T02:13:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40526.565507 secs 2023-03-10T02:13:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:35 smithi181 ceph-mon[94189]: osdmap e1598: 12 total, 12 up, 12 in 2023-03-10T02:13:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:35 smithi038 ceph-mon[92422]: pgmap v22122: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 254 KiB/s rd, 252 KiB/s wr, 8 op/s 2023-03-10T02:13:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40526.565507 secs 2023-03-10T02:13:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:35 smithi038 ceph-mon[92422]: osdmap e1598: 12 total, 12 up, 12 in 2023-03-10T02:13:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:13:36.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:13:36.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:36 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:13:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:13:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:13:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:36 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:13:37.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:13:37.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:13:37.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:36 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:13:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:37 smithi122 ceph-mon[93869]: pgmap v22124: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 509 KiB/s wr, 9 op/s 2023-03-10T02:13:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:13:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:13:37.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:13:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:37 smithi181 ceph-mon[94189]: pgmap v22124: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 509 KiB/s wr, 9 op/s 2023-03-10T02:13:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:13:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:13:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:13:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:37 smithi038 ceph-mon[92422]: pgmap v22124: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 509 KiB/s wr, 9 op/s 2023-03-10T02:13:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:13:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:13:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:13:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40990.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:13:38.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40529.033535 secs 2023-03-10T02:13:38.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40529.033535 secs 2023-03-10T02:13:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40529.033535 secs 2023-03-10T02:13:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:39 smithi122 ceph-mon[93869]: pgmap v22125: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 509 KiB/s wr, 9 op/s 2023-03-10T02:13:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:39 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:13:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:39 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:13:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:39 smithi181 ceph-mon[94189]: pgmap v22125: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 509 KiB/s wr, 9 op/s 2023-03-10T02:13:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:39 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:13:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:39 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:13:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:39 smithi038 ceph-mon[92422]: pgmap v22125: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 509 KiB/s wr, 9 op/s 2023-03-10T02:13:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:39 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:13:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:39 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:13:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:13:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:13:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40531.565608 secs 2023-03-10T02:13:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:13:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:13:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40531.565608 secs 2023-03-10T02:13:40.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:13:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:13:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40531.565608 secs 2023-03-10T02:13:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:41 smithi122 ceph-mon[93869]: pgmap v22126: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 509 KiB/s wr, 9 op/s 2023-03-10T02:13:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:13:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:13:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:41 smithi181 ceph-mon[94189]: pgmap v22126: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 509 KiB/s wr, 9 op/s 2023-03-10T02:13:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:13:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:13:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:41 smithi038 ceph-mon[92422]: pgmap v22126: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 509 KiB/s wr, 9 op/s 2023-03-10T02:13:42.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:13:42.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:13:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 40995.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:13:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:43 smithi122 ceph-mon[93869]: pgmap v22127: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 893 KiB/s rd, 891 KiB/s wr, 10 op/s 2023-03-10T02:13:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40534.033596 secs 2023-03-10T02:13:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:43 smithi181 ceph-mon[94189]: pgmap v22127: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 893 KiB/s rd, 891 KiB/s wr, 10 op/s 2023-03-10T02:13:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40534.033596 secs 2023-03-10T02:13:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:43 smithi038 ceph-mon[92422]: pgmap v22127: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 893 KiB/s rd, 891 KiB/s wr, 10 op/s 2023-03-10T02:13:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40534.033596 secs 2023-03-10T02:13:45.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:45 smithi122 ceph-mon[93869]: pgmap v22128: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 893 KiB/s rd, 891 KiB/s wr, 10 op/s 2023-03-10T02:13:45.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40536.565690 secs 2023-03-10T02:13:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:45 smithi181 ceph-mon[94189]: pgmap v22128: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 893 KiB/s rd, 891 KiB/s wr, 10 op/s 2023-03-10T02:13:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40536.565690 secs 2023-03-10T02:13:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:45 smithi038 ceph-mon[92422]: pgmap v22128: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 893 KiB/s rd, 891 KiB/s wr, 10 op/s 2023-03-10T02:13:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40536.565690 secs 2023-03-10T02:13:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:47 smithi122 ceph-mon[93869]: pgmap v22129: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 14 op/s 2023-03-10T02:13:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:47 smithi181 ceph-mon[94189]: pgmap v22129: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 14 op/s 2023-03-10T02:13:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:47 smithi038 ceph-mon[92422]: pgmap v22129: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 14 op/s 2023-03-10T02:13:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41000.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:13:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:13:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:13:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40539.058705 secs 2023-03-10T02:13:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:13:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:13:48.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40539.058705 secs 2023-03-10T02:13:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:13:49.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:13:49.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40539.058705 secs 2023-03-10T02:13:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:49 smithi122 ceph-mon[93869]: pgmap v22130: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 711 KiB/s wr, 9 op/s 2023-03-10T02:13:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:13:49.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:13:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:49 smithi181 ceph-mon[94189]: pgmap v22130: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 711 KiB/s wr, 9 op/s 2023-03-10T02:13:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:13:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:13:49.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:49 smithi038 ceph-mon[92422]: pgmap v22130: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 711 KiB/s wr, 9 op/s 2023-03-10T02:13:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:13:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:13:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40541.565759 secs 2023-03-10T02:13:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40541.565759 secs 2023-03-10T02:13:50.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40541.565759 secs 2023-03-10T02:13:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:51 smithi122 ceph-mon[93869]: pgmap v22131: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 711 KiB/s wr, 8 op/s 2023-03-10T02:13:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:51 smithi181 ceph-mon[94189]: pgmap v22131: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 711 KiB/s wr, 8 op/s 2023-03-10T02:13:52.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:51 smithi038 ceph-mon[92422]: pgmap v22131: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 711 KiB/s wr, 8 op/s 2023-03-10T02:13:53.801 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41005.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:13:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:53 smithi122 ceph-mon[93869]: pgmap v22132: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 935 KiB/s rd, 932 KiB/s wr, 13 op/s 2023-03-10T02:13:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:13:53.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:13:53.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40544.058761 secs 2023-03-10T02:13:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:53 smithi181 ceph-mon[94189]: pgmap v22132: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 935 KiB/s rd, 932 KiB/s wr, 13 op/s 2023-03-10T02:13:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:13:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:13:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40544.058761 secs 2023-03-10T02:13:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:53 smithi038 ceph-mon[92422]: pgmap v22132: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 935 KiB/s rd, 932 KiB/s wr, 13 op/s 2023-03-10T02:13:54.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:13:54.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:13:54.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40544.058761 secs 2023-03-10T02:13:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:55 smithi122 ceph-mon[93869]: pgmap v22133: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 580 KiB/s rd, 578 KiB/s wr, 9 op/s 2023-03-10T02:13:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40546.565819 secs 2023-03-10T02:13:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:55 smithi181 ceph-mon[94189]: pgmap v22133: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 580 KiB/s rd, 578 KiB/s wr, 9 op/s 2023-03-10T02:13:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40546.565819 secs 2023-03-10T02:13:55.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:55 smithi038 ceph-mon[92422]: pgmap v22133: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 580 KiB/s rd, 578 KiB/s wr, 9 op/s 2023-03-10T02:13:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40546.565819 secs 2023-03-10T02:13:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:13:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:13:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:13:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:13:56.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:13:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:13:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:57 smithi122 ceph-mon[93869]: pgmap v22134: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 613 KiB/s wr, 13 op/s 2023-03-10T02:13:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:57 smithi181 ceph-mon[94189]: pgmap v22134: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 613 KiB/s wr, 13 op/s 2023-03-10T02:13:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:57 smithi038 ceph-mon[92422]: pgmap v22134: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 613 KiB/s wr, 13 op/s 2023-03-10T02:13:58.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41010.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:13:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40549.058856 secs 2023-03-10T02:13:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40549.058856 secs 2023-03-10T02:13:58.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40549.058856 secs 2023-03-10T02:13:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:13:59 smithi122 ceph-mon[93869]: pgmap v22135: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 714 KiB/s rd, 678 KiB/s wr, 11 op/s 2023-03-10T02:13:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:13:59 smithi181 ceph-mon[94189]: pgmap v22135: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 714 KiB/s rd, 678 KiB/s wr, 11 op/s 2023-03-10T02:13:59.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:13:59 smithi038 ceph-mon[92422]: pgmap v22135: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 714 KiB/s rd, 678 KiB/s wr, 11 op/s 2023-03-10T02:14:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:14:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:14:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40551.565896 secs 2023-03-10T02:14:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:14:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:14:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40551.565896 secs 2023-03-10T02:14:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:14:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:14:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40551.565896 secs 2023-03-10T02:14:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:01 smithi122 ceph-mon[93869]: pgmap v22136: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 725 KiB/s rd, 678 KiB/s wr, 11 op/s 2023-03-10T02:14:01.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:14:01.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:14:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:01 smithi181 ceph-mon[94189]: pgmap v22136: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 725 KiB/s rd, 678 KiB/s wr, 11 op/s 2023-03-10T02:14:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:14:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:14:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:01 smithi038 ceph-mon[92422]: pgmap v22136: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 725 KiB/s rd, 678 KiB/s wr, 11 op/s 2023-03-10T02:14:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:14:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:14:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41015.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:14:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:03 smithi122 ceph-mon[93869]: pgmap v22137: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 763 KiB/s rd, 733 KiB/s wr, 16 op/s 2023-03-10T02:14:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40554.058941 secs 2023-03-10T02:14:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:03 smithi181 ceph-mon[94189]: pgmap v22137: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 763 KiB/s rd, 733 KiB/s wr, 16 op/s 2023-03-10T02:14:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40554.058941 secs 2023-03-10T02:14:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:03 smithi038 ceph-mon[92422]: pgmap v22137: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 763 KiB/s rd, 733 KiB/s wr, 16 op/s 2023-03-10T02:14:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40554.058941 secs 2023-03-10T02:14:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:05 smithi122 ceph-mon[93869]: pgmap v22138: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 785 KiB/s wr, 23 op/s 2023-03-10T02:14:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40556.565986 secs 2023-03-10T02:14:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:05 smithi181 ceph-mon[94189]: pgmap v22138: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 785 KiB/s wr, 23 op/s 2023-03-10T02:14:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40556.565986 secs 2023-03-10T02:14:05.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:05 smithi038 ceph-mon[92422]: pgmap v22138: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 865 KiB/s rd, 785 KiB/s wr, 23 op/s 2023-03-10T02:14:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40556.565986 secs 2023-03-10T02:14:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:07 smithi122 ceph-mon[93869]: pgmap v22139: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 876 KiB/s rd, 787 KiB/s wr, 27 op/s 2023-03-10T02:14:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:14:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:14:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:07 smithi181 ceph-mon[94189]: pgmap v22139: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 876 KiB/s rd, 787 KiB/s wr, 27 op/s 2023-03-10T02:14:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:14:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:14:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:07 smithi038 ceph-mon[92422]: pgmap v22139: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 876 KiB/s rd, 787 KiB/s wr, 27 op/s 2023-03-10T02:14:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:14:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:14:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41020.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:14:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40559.059043 secs 2023-03-10T02:14:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40559.059043 secs 2023-03-10T02:14:08.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40559.059043 secs 2023-03-10T02:14:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:09 smithi122 ceph-mon[93869]: pgmap v22140: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 805 KiB/s rd, 752 KiB/s wr, 24 op/s 2023-03-10T02:14:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:14:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:14:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:09 smithi181 ceph-mon[94189]: pgmap v22140: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 805 KiB/s rd, 752 KiB/s wr, 24 op/s 2023-03-10T02:14:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:14:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:14:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:09 smithi038 ceph-mon[92422]: pgmap v22140: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 805 KiB/s rd, 752 KiB/s wr, 24 op/s 2023-03-10T02:14:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:14:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:14:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40561.566072 secs 2023-03-10T02:14:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40561.566072 secs 2023-03-10T02:14:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40561.566072 secs 2023-03-10T02:14:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:11 smithi122 ceph-mon[93869]: pgmap v22141: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 331 KiB/s wr, 22 op/s 2023-03-10T02:14:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:11 smithi181 ceph-mon[94189]: pgmap v22141: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 331 KiB/s wr, 22 op/s 2023-03-10T02:14:12.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:11 smithi038 ceph-mon[92422]: pgmap v22141: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 331 KiB/s wr, 22 op/s 2023-03-10T02:14:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41025.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:14:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:13 smithi122 ceph-mon[93869]: pgmap v22142: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 694 KiB/s rd, 651 KiB/s wr, 25 op/s 2023-03-10T02:14:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40564.059107 secs 2023-03-10T02:14:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:13 smithi181 ceph-mon[94189]: pgmap v22142: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 694 KiB/s rd, 651 KiB/s wr, 25 op/s 2023-03-10T02:14:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40564.059107 secs 2023-03-10T02:14:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:13 smithi038 ceph-mon[92422]: pgmap v22142: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 694 KiB/s rd, 651 KiB/s wr, 25 op/s 2023-03-10T02:14:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40564.059107 secs 2023-03-10T02:14:14.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:14 smithi122 ceph-mon[93869]: pgmap v22143: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 656 KiB/s rd, 595 KiB/s wr, 20 op/s 2023-03-10T02:14:14.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:14 smithi181 ceph-mon[94189]: pgmap v22143: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 656 KiB/s rd, 595 KiB/s wr, 20 op/s 2023-03-10T02:14:15.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:14 smithi038 ceph-mon[92422]: pgmap v22143: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 656 KiB/s rd, 595 KiB/s wr, 20 op/s 2023-03-10T02:14:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40566.566132 secs 2023-03-10T02:14:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40566.566132 secs 2023-03-10T02:14:15.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40566.566132 secs 2023-03-10T02:14:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:16 smithi038 ceph-mon[92422]: pgmap v22144: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 322 KiB/s wr, 10 op/s 2023-03-10T02:14:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:16 smithi122 ceph-mon[93869]: pgmap v22144: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 322 KiB/s wr, 10 op/s 2023-03-10T02:14:17.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:16 smithi181 ceph-mon[94189]: pgmap v22144: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 333 KiB/s rd, 322 KiB/s wr, 10 op/s 2023-03-10T02:14:18.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41030.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:14:19.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:19 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40569.059121 secs 2023-03-10T02:14:19.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:19 smithi122 ceph-mon[93869]: pgmap v22145: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 330 KiB/s wr, 8 op/s 2023-03-10T02:14:19.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:19 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40569.059121 secs 2023-03-10T02:14:19.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:19 smithi181 ceph-mon[94189]: pgmap v22145: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 330 KiB/s wr, 8 op/s 2023-03-10T02:14:19.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:19 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40569.059121 secs 2023-03-10T02:14:19.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:19 smithi038 ceph-mon[92422]: pgmap v22145: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 330 KiB/s wr, 8 op/s 2023-03-10T02:14:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:14:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:14:20.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40571.566188 secs 2023-03-10T02:14:20.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:14:20.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:14:20.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40571.566188 secs 2023-03-10T02:14:20.502 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:14:20.503 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:14:20.503 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40571.566188 secs 2023-03-10T02:14:21.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:21 smithi181 ceph-mon[94189]: pgmap v22146: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 330 KiB/s wr, 7 op/s 2023-03-10T02:14:21.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:14:21.482 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:14:21.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:21 smithi038 ceph-mon[92422]: pgmap v22146: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 330 KiB/s wr, 7 op/s 2023-03-10T02:14:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:14:21.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:14:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:21 smithi122 ceph-mon[93869]: pgmap v22146: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 322 KiB/s rd, 330 KiB/s wr, 7 op/s 2023-03-10T02:14:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:14:21.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:14:23.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:23 smithi038 ceph-mon[92422]: pgmap v22147: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 812 KiB/s rd, 831 KiB/s wr, 12 op/s 2023-03-10T02:14:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:23 smithi122 ceph-mon[93869]: pgmap v22147: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 812 KiB/s rd, 831 KiB/s wr, 12 op/s 2023-03-10T02:14:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:23 smithi181 ceph-mon[94189]: pgmap v22147: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 812 KiB/s rd, 831 KiB/s wr, 12 op/s 2023-03-10T02:14:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41035.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:14:24.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:24 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40574.059206 secs 2023-03-10T02:14:24.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:24 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40574.059206 secs 2023-03-10T02:14:24.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:24 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40574.059206 secs 2023-03-10T02:14:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:25 smithi122 ceph-mon[93869]: pgmap v22148: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 511 KiB/s wr, 9 op/s 2023-03-10T02:14:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40576.566247 secs 2023-03-10T02:14:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:25 smithi181 ceph-mon[94189]: pgmap v22148: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 511 KiB/s wr, 9 op/s 2023-03-10T02:14:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40576.566247 secs 2023-03-10T02:14:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:25 smithi038 ceph-mon[92422]: pgmap v22148: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 511 KiB/s wr, 9 op/s 2023-03-10T02:14:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40576.566247 secs 2023-03-10T02:14:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:26 smithi122 ceph-mon[93869]: osdmap e1599: 12 total, 12 up, 12 in 2023-03-10T02:14:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:26 smithi181 ceph-mon[94189]: osdmap e1599: 12 total, 12 up, 12 in 2023-03-10T02:14:26.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:26 smithi038 ceph-mon[92422]: osdmap e1599: 12 total, 12 up, 12 in 2023-03-10T02:14:27.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:27 smithi038 ceph-mon[92422]: pgmap v22150: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 842 KiB/s rd, 866 KiB/s wr, 13 op/s 2023-03-10T02:14:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:27 smithi122 ceph-mon[93869]: pgmap v22150: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 842 KiB/s rd, 866 KiB/s wr, 13 op/s 2023-03-10T02:14:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:27 smithi181 ceph-mon[94189]: pgmap v22150: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 842 KiB/s rd, 866 KiB/s wr, 13 op/s 2023-03-10T02:14:28.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40579.059268 secs 2023-03-10T02:14:28.800 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41040.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:14:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40579.059268 secs 2023-03-10T02:14:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40579.059268 secs 2023-03-10T02:14:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:29 smithi122 ceph-mon[93869]: pgmap v22151: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 841 KiB/s rd, 853 KiB/s wr, 11 op/s 2023-03-10T02:14:29.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:29 smithi181 ceph-mon[94189]: pgmap v22151: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 841 KiB/s rd, 853 KiB/s wr, 11 op/s 2023-03-10T02:14:29.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:29 smithi038 ceph-mon[92422]: pgmap v22151: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 841 KiB/s rd, 853 KiB/s wr, 11 op/s 2023-03-10T02:14:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40581.566321 secs 2023-03-10T02:14:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40581.566321 secs 2023-03-10T02:14:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40581.566321 secs 2023-03-10T02:14:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:31 smithi122 ceph-mon[93869]: pgmap v22152: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 841 KiB/s rd, 853 KiB/s wr, 11 op/s 2023-03-10T02:14:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:31 smithi181 ceph-mon[94189]: pgmap v22152: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 841 KiB/s rd, 853 KiB/s wr, 11 op/s 2023-03-10T02:14:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:31 smithi038 ceph-mon[92422]: pgmap v22152: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 841 KiB/s rd, 853 KiB/s wr, 11 op/s 2023-03-10T02:14:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41045.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:14:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:33 smithi122 ceph-mon[93869]: pgmap v22153: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 254 KiB/s wr, 9 op/s 2023-03-10T02:14:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40584.059327 secs 2023-03-10T02:14:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:33 smithi181 ceph-mon[94189]: pgmap v22153: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 254 KiB/s wr, 9 op/s 2023-03-10T02:14:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40584.059327 secs 2023-03-10T02:14:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:33 smithi038 ceph-mon[92422]: pgmap v22153: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 254 KiB/s wr, 9 op/s 2023-03-10T02:14:34.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40584.059327 secs 2023-03-10T02:14:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:35 smithi122 ceph-mon[93869]: pgmap v22154: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 253 KiB/s wr, 9 op/s 2023-03-10T02:14:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40586.566403 secs 2023-03-10T02:14:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:35 smithi122 ceph-mon[93869]: osdmap e1600: 12 total, 12 up, 12 in 2023-03-10T02:14:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:35 smithi181 ceph-mon[94189]: pgmap v22154: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 253 KiB/s wr, 9 op/s 2023-03-10T02:14:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40586.566403 secs 2023-03-10T02:14:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:35 smithi181 ceph-mon[94189]: osdmap e1600: 12 total, 12 up, 12 in 2023-03-10T02:14:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:35 smithi038 ceph-mon[92422]: pgmap v22154: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 253 KiB/s wr, 9 op/s 2023-03-10T02:14:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40586.566403 secs 2023-03-10T02:14:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:35 smithi038 ceph-mon[92422]: osdmap e1600: 12 total, 12 up, 12 in 2023-03-10T02:14:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:37 smithi122 ceph-mon[93869]: pgmap v22156: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 436 KiB/s rd, 435 KiB/s wr, 8 op/s 2023-03-10T02:14:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:37 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:14:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:37 smithi181 ceph-mon[94189]: pgmap v22156: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 436 KiB/s rd, 435 KiB/s wr, 8 op/s 2023-03-10T02:14:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:37 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:14:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:37 smithi038 ceph-mon[92422]: pgmap v22156: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 436 KiB/s rd, 435 KiB/s wr, 8 op/s 2023-03-10T02:14:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:37 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:14:38.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41050.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:14:38.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40589.059400 secs 2023-03-10T02:14:38.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:38 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:14:38.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:38 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:14:38.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:38 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:14:38.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40589.059400 secs 2023-03-10T02:14:38.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:38 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:14:38.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:38 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:14:38.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:38 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:14:39.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40589.059400 secs 2023-03-10T02:14:39.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:38 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:14:39.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:38 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:14:39.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:38 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:14:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:39 smithi122 ceph-mon[93869]: pgmap v22157: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 436 KiB/s rd, 435 KiB/s wr, 9 op/s 2023-03-10T02:14:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:39 smithi181 ceph-mon[94189]: pgmap v22157: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 436 KiB/s rd, 435 KiB/s wr, 9 op/s 2023-03-10T02:14:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:39 smithi038 ceph-mon[92422]: pgmap v22157: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 436 KiB/s rd, 435 KiB/s wr, 9 op/s 2023-03-10T02:14:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:14:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:14:40.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40591.566460 secs 2023-03-10T02:14:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:14:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:14:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40591.566460 secs 2023-03-10T02:14:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:14:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:14:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40591.566460 secs 2023-03-10T02:14:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:41 smithi122 ceph-mon[93869]: pgmap v22158: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 436 KiB/s rd, 435 KiB/s wr, 9 op/s 2023-03-10T02:14:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:14:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:14:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:41 smithi181 ceph-mon[94189]: pgmap v22158: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 436 KiB/s rd, 435 KiB/s wr, 9 op/s 2023-03-10T02:14:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:14:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:14:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:41 smithi038 ceph-mon[92422]: pgmap v22158: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 436 KiB/s rd, 435 KiB/s wr, 9 op/s 2023-03-10T02:14:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:14:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:14:43.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41055.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:14:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:43 smithi122 ceph-mon[93869]: pgmap v22159: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 684 KiB/s rd, 682 KiB/s wr, 10 op/s 2023-03-10T02:14:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40594.059487 secs 2023-03-10T02:14:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:43 smithi181 ceph-mon[94189]: pgmap v22159: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 684 KiB/s rd, 682 KiB/s wr, 10 op/s 2023-03-10T02:14:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40594.059487 secs 2023-03-10T02:14:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:43 smithi038 ceph-mon[92422]: pgmap v22159: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 684 KiB/s rd, 682 KiB/s wr, 10 op/s 2023-03-10T02:14:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40594.059487 secs 2023-03-10T02:14:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:45 smithi122 ceph-mon[93869]: pgmap v22160: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 684 KiB/s rd, 682 KiB/s wr, 10 op/s 2023-03-10T02:14:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40596.566525 secs 2023-03-10T02:14:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:45 smithi181 ceph-mon[94189]: pgmap v22160: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 684 KiB/s rd, 682 KiB/s wr, 10 op/s 2023-03-10T02:14:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40596.566525 secs 2023-03-10T02:14:46.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:45 smithi038 ceph-mon[92422]: pgmap v22160: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 684 KiB/s rd, 682 KiB/s wr, 10 op/s 2023-03-10T02:14:46.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40596.566525 secs 2023-03-10T02:14:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:47 smithi122 ceph-mon[93869]: pgmap v22161: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 13 op/s 2023-03-10T02:14:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:47 smithi181 ceph-mon[94189]: pgmap v22161: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 13 op/s 2023-03-10T02:14:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:47 smithi038 ceph-mon[92422]: pgmap v22161: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 13 op/s 2023-03-10T02:14:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41060.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:14:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:14:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:14:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40599.059609 secs 2023-03-10T02:14:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:14:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:14:48.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40599.059609 secs 2023-03-10T02:14:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:14:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:14:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40599.059609 secs 2023-03-10T02:14:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:49 smithi122 ceph-mon[93869]: pgmap v22162: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 605 KiB/s rd, 602 KiB/s wr, 9 op/s 2023-03-10T02:14:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:14:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:14:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:49 smithi181 ceph-mon[94189]: pgmap v22162: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 605 KiB/s rd, 602 KiB/s wr, 9 op/s 2023-03-10T02:14:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:14:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:14:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:49 smithi038 ceph-mon[92422]: pgmap v22162: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 605 KiB/s rd, 602 KiB/s wr, 9 op/s 2023-03-10T02:14:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:14:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:14:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40601.566581 secs 2023-03-10T02:14:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40601.566581 secs 2023-03-10T02:14:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40601.566581 secs 2023-03-10T02:14:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:51 smithi122 ceph-mon[93869]: pgmap v22163: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 605 KiB/s rd, 602 KiB/s wr, 8 op/s 2023-03-10T02:14:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:51 smithi181 ceph-mon[94189]: pgmap v22163: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 605 KiB/s rd, 602 KiB/s wr, 8 op/s 2023-03-10T02:14:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:51 smithi038 ceph-mon[92422]: pgmap v22163: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 605 KiB/s rd, 602 KiB/s wr, 8 op/s 2023-03-10T02:14:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41065.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:14:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:53 smithi122 ceph-mon[93869]: pgmap v22164: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 815 KiB/s wr, 13 op/s 2023-03-10T02:14:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:14:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:14:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40604.059719 secs 2023-03-10T02:14:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:53 smithi181 ceph-mon[94189]: pgmap v22164: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 815 KiB/s wr, 13 op/s 2023-03-10T02:14:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:14:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:14:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40604.059719 secs 2023-03-10T02:14:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:53 smithi038 ceph-mon[92422]: pgmap v22164: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 820 KiB/s rd, 815 KiB/s wr, 13 op/s 2023-03-10T02:14:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:14:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:14:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40604.059719 secs 2023-03-10T02:14:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:55 smithi122 ceph-mon[93869]: pgmap v22165: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 612 KiB/s rd, 609 KiB/s wr, 9 op/s 2023-03-10T02:14:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40606.566639 secs 2023-03-10T02:14:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:55 smithi181 ceph-mon[94189]: pgmap v22165: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 612 KiB/s rd, 609 KiB/s wr, 9 op/s 2023-03-10T02:14:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40606.566639 secs 2023-03-10T02:14:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:55 smithi038 ceph-mon[92422]: pgmap v22165: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 612 KiB/s rd, 609 KiB/s wr, 9 op/s 2023-03-10T02:14:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40606.566639 secs 2023-03-10T02:14:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:14:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:14:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:14:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:14:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:14:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:14:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:57 smithi122 ceph-mon[93869]: pgmap v22166: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 14 op/s 2023-03-10T02:14:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:14:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:14:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:57 smithi181 ceph-mon[94189]: pgmap v22166: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 14 op/s 2023-03-10T02:14:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:14:57.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:14:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:57 smithi038 ceph-mon[92422]: pgmap v22166: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 14 op/s 2023-03-10T02:14:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:14:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:14:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41070.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:14:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:58 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T02:14:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:58 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T02:14:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:58 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T02:14:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:59 smithi122 ceph-mon[93869]: pgmap v22167: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 841 KiB/s rd, 975 KiB/s wr, 11 op/s 2023-03-10T02:14:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:59 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:14:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:14:59.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:14:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:14:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:59 smithi181 ceph-mon[94189]: pgmap v22167: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 841 KiB/s rd, 975 KiB/s wr, 11 op/s 2023-03-10T02:14:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:59 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:14:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:14:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:14:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:14:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:59 smithi038 ceph-mon[92422]: pgmap v22167: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 841 KiB/s rd, 975 KiB/s wr, 11 op/s 2023-03-10T02:15:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:59 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:15:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:15:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:14:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:15:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:15:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:15:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40611.566697 secs 2023-03-10T02:15:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:15:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:15:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40611.566697 secs 2023-03-10T02:15:01.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:15:01.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:15:01.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40611.566697 secs 2023-03-10T02:15:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:01 smithi122 ceph-mon[93869]: pgmap v22168: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 852 KiB/s rd, 974 KiB/s wr, 10 op/s 2023-03-10T02:15:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:15:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:15:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:01 smithi181 ceph-mon[94189]: pgmap v22168: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 852 KiB/s rd, 974 KiB/s wr, 10 op/s 2023-03-10T02:15:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:15:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:15:02.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:01 smithi038 ceph-mon[92422]: pgmap v22168: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 852 KiB/s rd, 974 KiB/s wr, 10 op/s 2023-03-10T02:15:02.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:15:02.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:15:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41075.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:15:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:03 smithi122 ceph-mon[93869]: pgmap v22169: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 974 KiB/s rd, 1.1 MiB/s wr, 16 op/s 2023-03-10T02:15:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:03 smithi181 ceph-mon[94189]: pgmap v22169: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 974 KiB/s rd, 1.1 MiB/s wr, 16 op/s 2023-03-10T02:15:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:03 smithi038 ceph-mon[92422]: pgmap v22169: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 974 KiB/s rd, 1.1 MiB/s wr, 16 op/s 2023-03-10T02:15:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:05 smithi122 ceph-mon[93869]: pgmap v22170: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 24 op/s 2023-03-10T02:15:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40616.566782 secs 2023-03-10T02:15:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:05 smithi181 ceph-mon[94189]: pgmap v22170: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 24 op/s 2023-03-10T02:15:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40616.566782 secs 2023-03-10T02:15:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:05 smithi038 ceph-mon[92422]: pgmap v22170: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 24 op/s 2023-03-10T02:15:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40616.566782 secs 2023-03-10T02:15:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:07 smithi122 ceph-mon[93869]: pgmap v22171: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 27 op/s 2023-03-10T02:15:07.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:15:07.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:15:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:07 smithi181 ceph-mon[94189]: pgmap v22171: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 27 op/s 2023-03-10T02:15:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:15:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:15:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:07 smithi038 ceph-mon[92422]: pgmap v22171: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 27 op/s 2023-03-10T02:15:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:15:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:15:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41080.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:15:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:09 smithi122 ceph-mon[93869]: pgmap v22172: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 576 KiB/s rd, 578 KiB/s wr, 23 op/s 2023-03-10T02:15:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:15:09.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:15:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:09 smithi181 ceph-mon[94189]: pgmap v22172: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 576 KiB/s rd, 578 KiB/s wr, 23 op/s 2023-03-10T02:15:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:15:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:15:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:09 smithi038 ceph-mon[92422]: pgmap v22172: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 576 KiB/s rd, 578 KiB/s wr, 23 op/s 2023-03-10T02:15:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:15:10.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:15:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40621.566840 secs 2023-03-10T02:15:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40621.566840 secs 2023-03-10T02:15:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40621.566840 secs 2023-03-10T02:15:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:11 smithi122 ceph-mon[93869]: pgmap v22173: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 578 KiB/s rd, 443 KiB/s wr, 23 op/s 2023-03-10T02:15:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:11 smithi181 ceph-mon[94189]: pgmap v22173: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 578 KiB/s rd, 443 KiB/s wr, 23 op/s 2023-03-10T02:15:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:11 smithi038 ceph-mon[92422]: pgmap v22173: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 578 KiB/s rd, 443 KiB/s wr, 23 op/s 2023-03-10T02:15:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41085.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:15:13.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:13 smithi122 ceph-mon[93869]: pgmap v22174: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 443 KiB/s wr, 26 op/s 2023-03-10T02:15:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:13 smithi181 ceph-mon[94189]: pgmap v22174: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 443 KiB/s wr, 26 op/s 2023-03-10T02:15:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:13 smithi038 ceph-mon[92422]: pgmap v22174: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 568 KiB/s rd, 443 KiB/s wr, 26 op/s 2023-03-10T02:15:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:15 smithi122 ceph-mon[93869]: pgmap v22175: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 513 KiB/s wr, 22 op/s 2023-03-10T02:15:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40626.566898 secs 2023-03-10T02:15:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:15 smithi181 ceph-mon[94189]: pgmap v22175: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 513 KiB/s wr, 22 op/s 2023-03-10T02:15:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40626.566898 secs 2023-03-10T02:15:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:15 smithi038 ceph-mon[92422]: pgmap v22175: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 513 KiB/s wr, 22 op/s 2023-03-10T02:15:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40626.566898 secs 2023-03-10T02:15:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:17 smithi122 ceph-mon[93869]: pgmap v22176: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 204 KiB/s wr, 12 op/s 2023-03-10T02:15:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:17 smithi181 ceph-mon[94189]: pgmap v22176: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 204 KiB/s wr, 12 op/s 2023-03-10T02:15:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:17 smithi038 ceph-mon[92422]: pgmap v22176: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 204 KiB/s wr, 12 op/s 2023-03-10T02:15:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41090.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:15:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:19 smithi122 ceph-mon[93869]: pgmap v22177: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 215 KiB/s wr, 10 op/s 2023-03-10T02:15:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:19 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:15:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:19 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:15:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:19 smithi181 ceph-mon[94189]: pgmap v22177: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 215 KiB/s wr, 10 op/s 2023-03-10T02:15:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:19 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:15:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:19 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:15:20.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:19 smithi038 ceph-mon[92422]: pgmap v22177: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 215 KiB/s wr, 10 op/s 2023-03-10T02:15:20.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:19 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:15:20.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:19 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:15:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40631.566957 secs 2023-03-10T02:15:21.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40631.566957 secs 2023-03-10T02:15:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40631.566957 secs 2023-03-10T02:15:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:21 smithi181 ceph-mon[94189]: pgmap v22178: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 259 KiB/s wr, 10 op/s 2023-03-10T02:15:21.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:15:21.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:15:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:21 smithi038 ceph-mon[92422]: pgmap v22178: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 259 KiB/s wr, 10 op/s 2023-03-10T02:15:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:15:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:15:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:21 smithi122 ceph-mon[93869]: pgmap v22178: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 251 KiB/s rd, 259 KiB/s wr, 10 op/s 2023-03-10T02:15:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:15:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:15:22.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:22 smithi038 ceph-mon[92422]: pgmap v22179: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 607 KiB/s rd, 625 KiB/s wr, 14 op/s 2023-03-10T02:15:23.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:22 smithi122 ceph-mon[93869]: pgmap v22179: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 607 KiB/s rd, 625 KiB/s wr, 14 op/s 2023-03-10T02:15:23.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:22 smithi181 ceph-mon[94189]: pgmap v22179: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 607 KiB/s rd, 625 KiB/s wr, 14 op/s 2023-03-10T02:15:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41095.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:15:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:25 smithi181 ceph-mon[94189]: pgmap v22180: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 606 KiB/s rd, 625 KiB/s wr, 12 op/s 2023-03-10T02:15:25.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40636.567044 secs 2023-03-10T02:15:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:25 smithi038 ceph-mon[92422]: pgmap v22180: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 606 KiB/s rd, 625 KiB/s wr, 12 op/s 2023-03-10T02:15:25.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40636.567044 secs 2023-03-10T02:15:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:25 smithi122 ceph-mon[93869]: pgmap v22180: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 606 KiB/s rd, 625 KiB/s wr, 12 op/s 2023-03-10T02:15:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40636.567044 secs 2023-03-10T02:15:26.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:26 smithi038 ceph-mon[92422]: osdmap e1601: 12 total, 12 up, 12 in 2023-03-10T02:15:26.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:26 smithi122 ceph-mon[93869]: osdmap e1601: 12 total, 12 up, 12 in 2023-03-10T02:15:26.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:26 smithi181 ceph-mon[94189]: osdmap e1601: 12 total, 12 up, 12 in 2023-03-10T02:15:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:27 smithi122 ceph-mon[93869]: pgmap v22182: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 695 KiB/s rd, 718 KiB/s wr, 13 op/s 2023-03-10T02:15:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:27 smithi181 ceph-mon[94189]: pgmap v22182: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 695 KiB/s rd, 718 KiB/s wr, 13 op/s 2023-03-10T02:15:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:27 smithi038 ceph-mon[92422]: pgmap v22182: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 695 KiB/s rd, 718 KiB/s wr, 13 op/s 2023-03-10T02:15:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41100.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:15:29.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:29 smithi122 ceph-mon[93869]: pgmap v22183: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 694 KiB/s rd, 706 KiB/s wr, 11 op/s 2023-03-10T02:15:29.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:29 smithi181 ceph-mon[94189]: pgmap v22183: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 694 KiB/s rd, 706 KiB/s wr, 11 op/s 2023-03-10T02:15:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:29 smithi038 ceph-mon[92422]: pgmap v22183: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 694 KiB/s rd, 706 KiB/s wr, 11 op/s 2023-03-10T02:15:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40641.567101 secs 2023-03-10T02:15:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40641.567101 secs 2023-03-10T02:15:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40641.567101 secs 2023-03-10T02:15:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:31 smithi122 ceph-mon[93869]: pgmap v22184: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 641 KiB/s rd, 652 KiB/s wr, 11 op/s 2023-03-10T02:15:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:31 smithi181 ceph-mon[94189]: pgmap v22184: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 641 KiB/s rd, 652 KiB/s wr, 11 op/s 2023-03-10T02:15:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:31 smithi038 ceph-mon[92422]: pgmap v22184: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 641 KiB/s rd, 652 KiB/s wr, 11 op/s 2023-03-10T02:15:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41105.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:15:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:33 smithi122 ceph-mon[93869]: pgmap v22185: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 254 KiB/s wr, 10 op/s 2023-03-10T02:15:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:33 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:15:33.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:33 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:15:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:33 smithi181 ceph-mon[94189]: pgmap v22185: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 254 KiB/s wr, 10 op/s 2023-03-10T02:15:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:33 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:15:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:33 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:15:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:33 smithi038 ceph-mon[92422]: pgmap v22185: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 254 KiB/s wr, 10 op/s 2023-03-10T02:15:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:33 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:15:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:33 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:15:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:35 smithi122 ceph-mon[93869]: pgmap v22186: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 254 KiB/s wr, 10 op/s 2023-03-10T02:15:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40646.567164 secs 2023-03-10T02:15:35.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:35 smithi122 ceph-mon[93869]: osdmap e1602: 12 total, 12 up, 12 in 2023-03-10T02:15:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:35 smithi181 ceph-mon[94189]: pgmap v22186: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 254 KiB/s wr, 10 op/s 2023-03-10T02:15:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40646.567164 secs 2023-03-10T02:15:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:35 smithi181 ceph-mon[94189]: osdmap e1602: 12 total, 12 up, 12 in 2023-03-10T02:15:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:35 smithi038 ceph-mon[92422]: pgmap v22186: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 254 KiB/s wr, 10 op/s 2023-03-10T02:15:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40646.567164 secs 2023-03-10T02:15:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:35 smithi038 ceph-mon[92422]: osdmap e1602: 12 total, 12 up, 12 in 2023-03-10T02:15:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:15:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:15:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:15:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:15:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:15:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:15:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:37 smithi122 ceph-mon[93869]: pgmap v22188: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 277 KiB/s wr, 10 op/s 2023-03-10T02:15:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:15:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:15:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:37 smithi181 ceph-mon[94189]: pgmap v22188: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 277 KiB/s wr, 10 op/s 2023-03-10T02:15:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:15:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:15:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:37 smithi038 ceph-mon[92422]: pgmap v22188: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 278 KiB/s rd, 277 KiB/s wr, 10 op/s 2023-03-10T02:15:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:15:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:15:38.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41110.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:15:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40649.079471 secs 2023-03-10T02:15:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:38 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:15:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40649.079471 secs 2023-03-10T02:15:38.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:38 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:15:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40649.079471 secs 2023-03-10T02:15:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:38 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:15:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:39 smithi122 ceph-mon[93869]: pgmap v22189: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 730 KiB/s rd, 729 KiB/s wr, 12 op/s 2023-03-10T02:15:39.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:39 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:15:39.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:39 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:15:39.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:39 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:15:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:39 smithi181 ceph-mon[94189]: pgmap v22189: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 730 KiB/s rd, 729 KiB/s wr, 12 op/s 2023-03-10T02:15:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:39 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:15:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:39 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:15:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:39 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:15:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:39 smithi038 ceph-mon[92422]: pgmap v22189: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 730 KiB/s rd, 729 KiB/s wr, 12 op/s 2023-03-10T02:15:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:39 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:15:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:39 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:15:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:39 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:15:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:15:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:15:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40651.567221 secs 2023-03-10T02:15:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:15:40.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:15:40.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40651.567221 secs 2023-03-10T02:15:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:15:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:15:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40651.567221 secs 2023-03-10T02:15:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:41 smithi122 ceph-mon[93869]: pgmap v22190: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 941 KiB/s rd, 940 KiB/s wr, 12 op/s 2023-03-10T02:15:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:15:41.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:15:41.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:15:41.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:15:41.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:15:41.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:15:41.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:15:41.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:15:41.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:15:41.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:15:41.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:15:41.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:15:41.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:15:41.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:15:41.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:15:41.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:15:41.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:15:41.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:15:41.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:15:41.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:15:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:41 smithi181 ceph-mon[94189]: pgmap v22190: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 941 KiB/s rd, 940 KiB/s wr, 12 op/s 2023-03-10T02:15:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:15:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:15:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:15:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:15:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:15:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:15:41.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:15:41.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:15:41.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:15:41.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:15:41.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:15:41.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:15:41.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:15:41.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:15:41.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:15:41.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:15:41.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:15:41.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:15:41.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:15:41.983 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:15:42.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:41 smithi038 ceph-mon[92422]: pgmap v22190: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 941 KiB/s rd, 940 KiB/s wr, 12 op/s 2023-03-10T02:15:42.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:15:42.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:15:42.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:15:42.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:15:42.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:15:42.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:15:42.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:15:42.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:15:42.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:15:42.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:15:42.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:15:42.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:15:42.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:15:42.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:15:42.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:15:42.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:15:42.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:15:42.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:15:42.002 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:15:42.003 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:15:42.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:42 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-10T02:15:42.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:42 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-10T02:15:42.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:42 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-10T02:15:42.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:42 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-10T02:15:42.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:42 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-10T02:15:42.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:42 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-10T02:15:42.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:42 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-10T02:15:42.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:42 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-10T02:15:42.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:42 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-10T02:15:42.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:42 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-10T02:15:42.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:42 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-10T02:15:42.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:42 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-10T02:15:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41115.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:15:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:43 smithi122 ceph-mon[93869]: pgmap v22191: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 917 KiB/s rd, 915 KiB/s wr, 11 op/s 2023-03-10T02:15:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40654.079530 secs 2023-03-10T02:15:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:43 smithi181 ceph-mon[94189]: pgmap v22191: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 917 KiB/s rd, 915 KiB/s wr, 11 op/s 2023-03-10T02:15:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40654.079530 secs 2023-03-10T02:15:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:43 smithi038 ceph-mon[92422]: pgmap v22191: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 917 KiB/s rd, 915 KiB/s wr, 11 op/s 2023-03-10T02:15:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40654.079530 secs 2023-03-10T02:15:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:45 smithi122 ceph-mon[93869]: pgmap v22192: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 917 KiB/s rd, 915 KiB/s wr, 11 op/s 2023-03-10T02:15:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40656.567286 secs 2023-03-10T02:15:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:45 smithi181 ceph-mon[94189]: pgmap v22192: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 917 KiB/s rd, 915 KiB/s wr, 11 op/s 2023-03-10T02:15:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40656.567286 secs 2023-03-10T02:15:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:45 smithi038 ceph-mon[92422]: pgmap v22192: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 917 KiB/s rd, 915 KiB/s wr, 11 op/s 2023-03-10T02:15:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40656.567286 secs 2023-03-10T02:15:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:47 smithi122 ceph-mon[93869]: pgmap v22193: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 14 op/s 2023-03-10T02:15:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:47 smithi181 ceph-mon[94189]: pgmap v22193: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 14 op/s 2023-03-10T02:15:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:47 smithi038 ceph-mon[92422]: pgmap v22193: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.4 MiB/s wr, 14 op/s 2023-03-10T02:15:48.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41120.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:15:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:15:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:15:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:15:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:15:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:15:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:15:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:49 smithi122 ceph-mon[93869]: pgmap v22194: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 11 op/s 2023-03-10T02:15:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:15:49.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:15:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:49 smithi181 ceph-mon[94189]: pgmap v22194: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 11 op/s 2023-03-10T02:15:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:15:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:15:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:49 smithi038 ceph-mon[92422]: pgmap v22194: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 11 op/s 2023-03-10T02:15:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:15:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:15:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:50 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T02:15:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40661.567369 secs 2023-03-10T02:15:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:50 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T02:15:50.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40661.567369 secs 2023-03-10T02:15:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:50 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T02:15:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40661.567369 secs 2023-03-10T02:15:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:51 smithi122 ceph-mon[93869]: pgmap v22195: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 788 KiB/s rd, 746 KiB/s wr, 9 op/s 2023-03-10T02:15:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:51 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:15:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:15:51.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:15:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:51 smithi181 ceph-mon[94189]: pgmap v22195: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 788 KiB/s rd, 746 KiB/s wr, 9 op/s 2023-03-10T02:15:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:51 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:15:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:15:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:15:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:51 smithi038 ceph-mon[92422]: pgmap v22195: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 788 KiB/s rd, 746 KiB/s wr, 9 op/s 2023-03-10T02:15:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:51 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:15:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:15:52.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:15:53.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41125.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:15:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:53 smithi122 ceph-mon[93869]: pgmap v22196: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 719 KiB/s rd, 789 KiB/s wr, 13 op/s 2023-03-10T02:15:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:15:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:15:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:53 smithi181 ceph-mon[94189]: pgmap v22196: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 719 KiB/s rd, 789 KiB/s wr, 13 op/s 2023-03-10T02:15:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:15:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:15:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:53 smithi038 ceph-mon[92422]: pgmap v22196: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 719 KiB/s rd, 789 KiB/s wr, 13 op/s 2023-03-10T02:15:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:15:54.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:15:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:55 smithi122 ceph-mon[93869]: pgmap v22197: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 702 KiB/s rd, 774 KiB/s wr, 10 op/s 2023-03-10T02:15:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40666.567432 secs 2023-03-10T02:15:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:55 smithi181 ceph-mon[94189]: pgmap v22197: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 702 KiB/s rd, 774 KiB/s wr, 10 op/s 2023-03-10T02:15:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40666.567432 secs 2023-03-10T02:15:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:55 smithi038 ceph-mon[92422]: pgmap v22197: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 702 KiB/s rd, 774 KiB/s wr, 10 op/s 2023-03-10T02:15:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40666.567432 secs 2023-03-10T02:15:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:15:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:15:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:15:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:15:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:15:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:15:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:57 smithi122 ceph-mon[93869]: pgmap v22198: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 888 KiB/s rd, 864 KiB/s wr, 13 op/s 2023-03-10T02:15:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:57 smithi181 ceph-mon[94189]: pgmap v22198: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 888 KiB/s rd, 864 KiB/s wr, 13 op/s 2023-03-10T02:15:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:57 smithi038 ceph-mon[92422]: pgmap v22198: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 888 KiB/s rd, 864 KiB/s wr, 13 op/s 2023-03-10T02:15:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41130.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:15:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40669.079733 secs 2023-03-10T02:15:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:58 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:15:58.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:58 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:15:58.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:58 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:15:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40669.079733 secs 2023-03-10T02:15:58.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:58 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:15:58.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:58 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:15:58.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:58 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:15:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40669.079733 secs 2023-03-10T02:15:59.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:58 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:15:59.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:58 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:15:59.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:58 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:15:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:59 smithi122 ceph-mon[93869]: pgmap v22199: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 685 KiB/s wr, 11 op/s 2023-03-10T02:15:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:15:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:15:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:15:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:59 smithi181 ceph-mon[94189]: pgmap v22199: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 685 KiB/s wr, 11 op/s 2023-03-10T02:15:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:15:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:15:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:15:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:59 smithi038 ceph-mon[92422]: pgmap v22199: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 685 KiB/s wr, 11 op/s 2023-03-10T02:15:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:16:00.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:15:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:16:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40671.567488 secs 2023-03-10T02:16:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40671.567488 secs 2023-03-10T02:16:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40671.567488 secs 2023-03-10T02:16:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:01 smithi122 ceph-mon[93869]: pgmap v22200: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 685 KiB/s wr, 10 op/s 2023-03-10T02:16:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:16:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:16:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:01 smithi181 ceph-mon[94189]: pgmap v22200: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 685 KiB/s wr, 10 op/s 2023-03-10T02:16:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:16:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:16:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:01 smithi038 ceph-mon[92422]: pgmap v22200: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 338 KiB/s rd, 685 KiB/s wr, 10 op/s 2023-03-10T02:16:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:16:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:16:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41135.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:16:03.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:03 smithi122 ceph-mon[93869]: pgmap v22201: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 602 KiB/s rd, 940 KiB/s wr, 15 op/s 2023-03-10T02:16:03.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40674.079794 secs 2023-03-10T02:16:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:03 smithi181 ceph-mon[94189]: pgmap v22201: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 602 KiB/s rd, 940 KiB/s wr, 15 op/s 2023-03-10T02:16:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40674.079794 secs 2023-03-10T02:16:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:03 smithi038 ceph-mon[92422]: pgmap v22201: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 602 KiB/s rd, 940 KiB/s wr, 15 op/s 2023-03-10T02:16:04.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40674.079794 secs 2023-03-10T02:16:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:05 smithi122 ceph-mon[93869]: pgmap v22202: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 845 KiB/s rd, 733 KiB/s wr, 22 op/s 2023-03-10T02:16:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40676.567578 secs 2023-03-10T02:16:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:05 smithi181 ceph-mon[94189]: pgmap v22202: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 845 KiB/s rd, 733 KiB/s wr, 22 op/s 2023-03-10T02:16:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40676.567578 secs 2023-03-10T02:16:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:05 smithi038 ceph-mon[92422]: pgmap v22202: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 845 KiB/s rd, 733 KiB/s wr, 22 op/s 2023-03-10T02:16:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40676.567578 secs 2023-03-10T02:16:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:07 smithi038 ceph-mon[92422]: pgmap v22203: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 867 KiB/s wr, 25 op/s 2023-03-10T02:16:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:16:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:16:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:07 smithi122 ceph-mon[93869]: pgmap v22203: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 867 KiB/s wr, 25 op/s 2023-03-10T02:16:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:16:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:16:08.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:07 smithi181 ceph-mon[94189]: pgmap v22203: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 867 KiB/s wr, 25 op/s 2023-03-10T02:16:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:16:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:16:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41140.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:16:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40679.079915 secs 2023-03-10T02:16:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40679.079915 secs 2023-03-10T02:16:09.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40679.079915 secs 2023-03-10T02:16:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:09 smithi038 ceph-mon[92422]: pgmap v22204: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 24 op/s 2023-03-10T02:16:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:16:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:16:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:09 smithi122 ceph-mon[93869]: pgmap v22204: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 24 op/s 2023-03-10T02:16:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:16:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:16:10.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:09 smithi181 ceph-mon[94189]: pgmap v22204: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 24 op/s 2023-03-10T02:16:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:16:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:16:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40681.567660 secs 2023-03-10T02:16:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:10 smithi122 ceph-mon[93869]: pgmap v22205: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 708 KiB/s wr, 23 op/s 2023-03-10T02:16:11.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40681.567660 secs 2023-03-10T02:16:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:10 smithi181 ceph-mon[94189]: pgmap v22205: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 708 KiB/s wr, 23 op/s 2023-03-10T02:16:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40681.567660 secs 2023-03-10T02:16:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:10 smithi038 ceph-mon[92422]: pgmap v22205: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 708 KiB/s wr, 23 op/s 2023-03-10T02:16:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:16:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:16:12.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:16:12.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:16:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:16:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:16:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41145.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:16:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:13 smithi122 ceph-mon[93869]: pgmap v22206: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 761 KiB/s wr, 25 op/s 2023-03-10T02:16:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40684.079973 secs 2023-03-10T02:16:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:13 smithi181 ceph-mon[94189]: pgmap v22206: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 761 KiB/s wr, 25 op/s 2023-03-10T02:16:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40684.079973 secs 2023-03-10T02:16:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:13 smithi038 ceph-mon[92422]: pgmap v22206: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 761 KiB/s wr, 25 op/s 2023-03-10T02:16:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40684.079973 secs 2023-03-10T02:16:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:15 smithi122 ceph-mon[93869]: pgmap v22207: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 540 KiB/s wr, 20 op/s 2023-03-10T02:16:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40686.567720 secs 2023-03-10T02:16:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:15 smithi181 ceph-mon[94189]: pgmap v22207: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 540 KiB/s wr, 20 op/s 2023-03-10T02:16:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40686.567720 secs 2023-03-10T02:16:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:15 smithi038 ceph-mon[92422]: pgmap v22207: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 540 KiB/s wr, 20 op/s 2023-03-10T02:16:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40686.567720 secs 2023-03-10T02:16:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:16:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:16:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:16:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:16:16.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:16:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:16:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:17 smithi122 ceph-mon[93869]: pgmap v22208: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 923 KiB/s rd, 644 KiB/s wr, 11 op/s 2023-03-10T02:16:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:16:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:16:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:17 smithi181 ceph-mon[94189]: pgmap v22208: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 923 KiB/s rd, 644 KiB/s wr, 11 op/s 2023-03-10T02:16:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:16:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:16:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:17 smithi038 ceph-mon[92422]: pgmap v22208: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 923 KiB/s rd, 644 KiB/s wr, 11 op/s 2023-03-10T02:16:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:16:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:16:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41150.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:16:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40689.080061 secs 2023-03-10T02:16:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40689.080061 secs 2023-03-10T02:16:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40689.080061 secs 2023-03-10T02:16:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:19 smithi122 ceph-mon[93869]: pgmap v22209: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 513 KiB/s rd, 521 KiB/s wr, 10 op/s 2023-03-10T02:16:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:19 smithi181 ceph-mon[94189]: pgmap v22209: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 513 KiB/s rd, 521 KiB/s wr, 10 op/s 2023-03-10T02:16:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:19 smithi038 ceph-mon[92422]: pgmap v22209: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 513 KiB/s rd, 521 KiB/s wr, 10 op/s 2023-03-10T02:16:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:16:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:16:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40691.567777 secs 2023-03-10T02:16:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:16:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:16:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40691.567777 secs 2023-03-10T02:16:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:16:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:16:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40691.567777 secs 2023-03-10T02:16:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:21 smithi122 ceph-mon[93869]: pgmap v22210: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 213 KiB/s wr, 8 op/s 2023-03-10T02:16:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:16:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:16:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:21 smithi181 ceph-mon[94189]: pgmap v22210: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 213 KiB/s wr, 8 op/s 2023-03-10T02:16:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:16:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:16:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:21 smithi038 ceph-mon[92422]: pgmap v22210: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 205 KiB/s rd, 213 KiB/s wr, 8 op/s 2023-03-10T02:16:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:16:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:16:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41155.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:16:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:23 smithi122 ceph-mon[93869]: pgmap v22211: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 717 KiB/s wr, 14 op/s 2023-03-10T02:16:23.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40694.080172 secs 2023-03-10T02:16:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:23 smithi181 ceph-mon[94189]: pgmap v22211: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 717 KiB/s wr, 14 op/s 2023-03-10T02:16:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40694.080172 secs 2023-03-10T02:16:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:23 smithi038 ceph-mon[92422]: pgmap v22211: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 717 KiB/s wr, 14 op/s 2023-03-10T02:16:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40694.080172 secs 2023-03-10T02:16:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:25 smithi122 ceph-mon[93869]: pgmap v22212: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 644 KiB/s rd, 664 KiB/s wr, 11 op/s 2023-03-10T02:16:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:25 smithi122 ceph-mon[93869]: osdmap e1603: 12 total, 12 up, 12 in 2023-03-10T02:16:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40696.567836 secs 2023-03-10T02:16:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:25 smithi181 ceph-mon[94189]: pgmap v22212: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 644 KiB/s rd, 664 KiB/s wr, 11 op/s 2023-03-10T02:16:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:25 smithi181 ceph-mon[94189]: osdmap e1603: 12 total, 12 up, 12 in 2023-03-10T02:16:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40696.567836 secs 2023-03-10T02:16:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:25 smithi038 ceph-mon[92422]: pgmap v22212: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 644 KiB/s rd, 664 KiB/s wr, 11 op/s 2023-03-10T02:16:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:25 smithi038 ceph-mon[92422]: osdmap e1603: 12 total, 12 up, 12 in 2023-03-10T02:16:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40696.567836 secs 2023-03-10T02:16:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:27 smithi122 ceph-mon[93869]: pgmap v22214: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 804 KiB/s rd, 829 KiB/s wr, 13 op/s 2023-03-10T02:16:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:27 smithi181 ceph-mon[94189]: pgmap v22214: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 804 KiB/s rd, 829 KiB/s wr, 13 op/s 2023-03-10T02:16:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:27 smithi038 ceph-mon[92422]: pgmap v22214: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 804 KiB/s rd, 829 KiB/s wr, 13 op/s 2023-03-10T02:16:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41160.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:16:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40699.080234 secs 2023-03-10T02:16:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40699.080234 secs 2023-03-10T02:16:28.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40699.080234 secs 2023-03-10T02:16:29.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:29 smithi122 ceph-mon[93869]: pgmap v22215: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 804 KiB/s rd, 817 KiB/s wr, 11 op/s 2023-03-10T02:16:30.050 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:29 smithi038 ceph-mon[92422]: pgmap v22215: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 804 KiB/s rd, 817 KiB/s wr, 11 op/s 2023-03-10T02:16:30.051 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:29 smithi181 ceph-mon[94189]: pgmap v22215: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 804 KiB/s rd, 817 KiB/s wr, 11 op/s 2023-03-10T02:16:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40701.567893 secs 2023-03-10T02:16:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40701.567893 secs 2023-03-10T02:16:30.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40701.567893 secs 2023-03-10T02:16:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:31 smithi122 ceph-mon[93869]: pgmap v22216: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 804 KiB/s rd, 817 KiB/s wr, 11 op/s 2023-03-10T02:16:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:31 smithi181 ceph-mon[94189]: pgmap v22216: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 804 KiB/s rd, 817 KiB/s wr, 11 op/s 2023-03-10T02:16:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:31 smithi038 ceph-mon[92422]: pgmap v22216: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 804 KiB/s rd, 817 KiB/s wr, 11 op/s 2023-03-10T02:16:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41165.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:16:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:33 smithi122 ceph-mon[93869]: pgmap v22217: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 254 KiB/s wr, 9 op/s 2023-03-10T02:16:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:33 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:16:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:33 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:16:33.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40704.080338 secs 2023-03-10T02:16:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:33 smithi181 ceph-mon[94189]: pgmap v22217: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 254 KiB/s wr, 9 op/s 2023-03-10T02:16:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:33 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:16:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:33 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:16:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40704.080338 secs 2023-03-10T02:16:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:33 smithi038 ceph-mon[92422]: pgmap v22217: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 254 KiB/s wr, 9 op/s 2023-03-10T02:16:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:33 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:16:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:33 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:16:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40704.080338 secs 2023-03-10T02:16:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:35 smithi122 ceph-mon[93869]: pgmap v22218: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 254 KiB/s wr, 9 op/s 2023-03-10T02:16:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40706.567951 secs 2023-03-10T02:16:35.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:35 smithi122 ceph-mon[93869]: osdmap e1604: 12 total, 12 up, 12 in 2023-03-10T02:16:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:35 smithi181 ceph-mon[94189]: pgmap v22218: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 254 KiB/s wr, 9 op/s 2023-03-10T02:16:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40706.567951 secs 2023-03-10T02:16:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:35 smithi181 ceph-mon[94189]: osdmap e1604: 12 total, 12 up, 12 in 2023-03-10T02:16:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:35 smithi038 ceph-mon[92422]: pgmap v22218: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 254 KiB/s wr, 9 op/s 2023-03-10T02:16:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40706.567951 secs 2023-03-10T02:16:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:35 smithi038 ceph-mon[92422]: osdmap e1604: 12 total, 12 up, 12 in 2023-03-10T02:16:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:16:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:16:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:16:36.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:16:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:16:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:16:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:37 smithi122 ceph-mon[93869]: pgmap v22220: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 724 KiB/s rd, 723 KiB/s wr, 9 op/s 2023-03-10T02:16:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:16:37.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:16:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:37 smithi181 ceph-mon[94189]: pgmap v22220: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 724 KiB/s rd, 723 KiB/s wr, 9 op/s 2023-03-10T02:16:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:16:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:16:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:37 smithi038 ceph-mon[92422]: pgmap v22220: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 724 KiB/s rd, 723 KiB/s wr, 9 op/s 2023-03-10T02:16:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:16:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:16:38.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41170.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:16:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40709.080436 secs 2023-03-10T02:16:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40709.080436 secs 2023-03-10T02:16:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40709.080436 secs 2023-03-10T02:16:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:39 smithi122 ceph-mon[93869]: pgmap v22221: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 904 KiB/s rd, 902 KiB/s wr, 10 op/s 2023-03-10T02:16:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:39 smithi181 ceph-mon[94189]: pgmap v22221: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 904 KiB/s rd, 902 KiB/s wr, 10 op/s 2023-03-10T02:16:39.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:39 smithi038 ceph-mon[92422]: pgmap v22221: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 904 KiB/s rd, 902 KiB/s wr, 10 op/s 2023-03-10T02:16:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:40 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:16:40.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:40 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:16:40.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40711.567981 secs 2023-03-10T02:16:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:40 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:16:40.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:40 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:16:40.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40711.567981 secs 2023-03-10T02:16:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:40 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:16:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:40 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:16:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40711.567981 secs 2023-03-10T02:16:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:41 smithi122 ceph-mon[93869]: pgmap v22222: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 904 KiB/s rd, 902 KiB/s wr, 10 op/s 2023-03-10T02:16:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:16:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:16:41.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:41 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:16:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:41 smithi181 ceph-mon[94189]: pgmap v22222: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 904 KiB/s rd, 902 KiB/s wr, 10 op/s 2023-03-10T02:16:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:16:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:16:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:41 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:16:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:41 smithi038 ceph-mon[92422]: pgmap v22222: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 904 KiB/s rd, 902 KiB/s wr, 10 op/s 2023-03-10T02:16:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:16:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:16:42.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:41 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:16:42.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:16:42.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:16:42.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:16:42.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:16:42.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:16:42.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:16:42.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:16:42.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:16:42.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:16:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41175.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:16:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:43 smithi122 ceph-mon[93869]: pgmap v22223: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 11 op/s 2023-03-10T02:16:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40714.080528 secs 2023-03-10T02:16:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:43 smithi181 ceph-mon[94189]: pgmap v22223: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 11 op/s 2023-03-10T02:16:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40714.080528 secs 2023-03-10T02:16:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:43 smithi038 ceph-mon[92422]: pgmap v22223: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 11 op/s 2023-03-10T02:16:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40714.080528 secs 2023-03-10T02:16:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:45 smithi122 ceph-mon[93869]: pgmap v22224: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 12 op/s 2023-03-10T02:16:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40716.568040 secs 2023-03-10T02:16:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:45 smithi181 ceph-mon[94189]: pgmap v22224: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 12 op/s 2023-03-10T02:16:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40716.568040 secs 2023-03-10T02:16:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:45 smithi038 ceph-mon[92422]: pgmap v22224: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.5 MiB/s wr, 12 op/s 2023-03-10T02:16:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40716.568040 secs 2023-03-10T02:16:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:47 smithi122 ceph-mon[93869]: pgmap v22225: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 13 op/s 2023-03-10T02:16:47.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:16:47.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:16:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:47 smithi181 ceph-mon[94189]: pgmap v22225: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 13 op/s 2023-03-10T02:16:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:16:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:16:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:47 smithi038 ceph-mon[92422]: pgmap v22225: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 13 op/s 2023-03-10T02:16:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:16:48.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:16:48.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41180.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:16:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40719.080605 secs 2023-03-10T02:16:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40719.080605 secs 2023-03-10T02:16:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40719.080605 secs 2023-03-10T02:16:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:49 smithi122 ceph-mon[93869]: pgmap v22226: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 896 KiB/s rd, 895 KiB/s wr, 11 op/s 2023-03-10T02:16:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:16:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:16:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:49 smithi181 ceph-mon[94189]: pgmap v22226: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 896 KiB/s rd, 895 KiB/s wr, 11 op/s 2023-03-10T02:16:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:16:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:16:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:49 smithi038 ceph-mon[92422]: pgmap v22226: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 896 KiB/s rd, 895 KiB/s wr, 11 op/s 2023-03-10T02:16:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:16:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:16:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40721.568093 secs 2023-03-10T02:16:50.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40721.568093 secs 2023-03-10T02:16:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40721.568093 secs 2023-03-10T02:16:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:51 smithi122 ceph-mon[93869]: pgmap v22227: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 745 KiB/s rd, 745 KiB/s wr, 9 op/s 2023-03-10T02:16:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:51 smithi181 ceph-mon[94189]: pgmap v22227: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 745 KiB/s rd, 745 KiB/s wr, 9 op/s 2023-03-10T02:16:51.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:51 smithi038 ceph-mon[92422]: pgmap v22227: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 745 KiB/s rd, 745 KiB/s wr, 9 op/s 2023-03-10T02:16:53.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41185.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:16:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:53 smithi122 ceph-mon[93869]: pgmap v22228: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 793 KiB/s rd, 791 KiB/s wr, 13 op/s 2023-03-10T02:16:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40724.080664 secs 2023-03-10T02:16:53.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:16:53.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:16:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:53 smithi181 ceph-mon[94189]: pgmap v22228: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 793 KiB/s rd, 791 KiB/s wr, 13 op/s 2023-03-10T02:16:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40724.080664 secs 2023-03-10T02:16:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:16:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:16:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:53 smithi038 ceph-mon[92422]: pgmap v22228: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 793 KiB/s rd, 791 KiB/s wr, 13 op/s 2023-03-10T02:16:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40724.080664 secs 2023-03-10T02:16:54.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:16:54.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:16:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:55 smithi181 ceph-mon[94189]: pgmap v22229: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 731 KiB/s rd, 729 KiB/s wr, 9 op/s 2023-03-10T02:16:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:16:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:16:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40726.568153 secs 2023-03-10T02:16:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:55 smithi038 ceph-mon[92422]: pgmap v22229: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 731 KiB/s rd, 729 KiB/s wr, 9 op/s 2023-03-10T02:16:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:16:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:16:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40726.568153 secs 2023-03-10T02:16:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:55 smithi122 ceph-mon[93869]: pgmap v22229: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 731 KiB/s rd, 729 KiB/s wr, 9 op/s 2023-03-10T02:16:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:16:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:16:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40726.568153 secs 2023-03-10T02:16:57.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:57 smithi038 ceph-mon[92422]: pgmap v22230: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 860 KiB/s rd, 639 KiB/s wr, 12 op/s 2023-03-10T02:16:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:16:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:16:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:57 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T02:16:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:57 smithi122 ceph-mon[93869]: pgmap v22230: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 860 KiB/s rd, 639 KiB/s wr, 12 op/s 2023-03-10T02:16:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:16:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:16:58.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:57 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T02:16:58.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:57 smithi181 ceph-mon[94189]: pgmap v22230: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 860 KiB/s rd, 639 KiB/s wr, 12 op/s 2023-03-10T02:16:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:16:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:16:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:57 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T02:16:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41190.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:16:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:58 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:16:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:58 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:16:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:58 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:16:59.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:16:58 smithi122 ceph-mon[93869]: pgmap v22231: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 881 KiB/s rd, 786 KiB/s wr, 12 op/s 2023-03-10T02:16:59.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:58 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:16:59.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:58 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:16:59.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:58 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:16:59.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:16:58 smithi181 ceph-mon[94189]: pgmap v22231: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 881 KiB/s rd, 786 KiB/s wr, 12 op/s 2023-03-10T02:16:59.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:58 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:16:59.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:58 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:16:59.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:58 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:16:59.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:16:58 smithi038 ceph-mon[92422]: pgmap v22231: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 881 KiB/s rd, 786 KiB/s wr, 12 op/s 2023-03-10T02:17:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:17:01.278 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:17:01.279 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40731.568213 secs 2023-03-10T02:17:01.279 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:00 smithi038 ceph-mon[92422]: pgmap v22232: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 610 KiB/s wr, 10 op/s 2023-03-10T02:17:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:17:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:17:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40731.568213 secs 2023-03-10T02:17:01.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:00 smithi122 ceph-mon[93869]: pgmap v22232: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 610 KiB/s wr, 10 op/s 2023-03-10T02:17:01.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:17:01.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:17:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40731.568213 secs 2023-03-10T02:17:01.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:00 smithi181 ceph-mon[94189]: pgmap v22232: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 610 KiB/s wr, 10 op/s 2023-03-10T02:17:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:03 smithi122 ceph-mon[93869]: pgmap v22233: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 818 KiB/s wr, 14 op/s 2023-03-10T02:17:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:03 smithi181 ceph-mon[94189]: pgmap v22233: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 818 KiB/s wr, 14 op/s 2023-03-10T02:17:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:03 smithi038 ceph-mon[92422]: pgmap v22233: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 710 KiB/s rd, 818 KiB/s wr, 14 op/s 2023-03-10T02:17:03.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41195.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:17:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:05 smithi122 ceph-mon[93869]: pgmap v22234: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1021 KiB/s rd, 1001 KiB/s wr, 23 op/s 2023-03-10T02:17:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40736.568270 secs 2023-03-10T02:17:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:05 smithi181 ceph-mon[94189]: pgmap v22234: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1021 KiB/s rd, 1001 KiB/s wr, 23 op/s 2023-03-10T02:17:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40736.568270 secs 2023-03-10T02:17:05.748 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:05 smithi038 ceph-mon[92422]: pgmap v22234: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1021 KiB/s rd, 1001 KiB/s wr, 23 op/s 2023-03-10T02:17:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40736.568270 secs 2023-03-10T02:17:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:07 smithi122 ceph-mon[93869]: pgmap v22235: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1022 KiB/s wr, 27 op/s 2023-03-10T02:17:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:07 smithi181 ceph-mon[94189]: pgmap v22235: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1022 KiB/s wr, 27 op/s 2023-03-10T02:17:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:07 smithi038 ceph-mon[92422]: pgmap v22235: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1022 KiB/s wr, 27 op/s 2023-03-10T02:17:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41200.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:17:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:17:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:17:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:17:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:17:08.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:17:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:17:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:09 smithi122 ceph-mon[93869]: pgmap v22236: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 780 KiB/s rd, 772 KiB/s wr, 25 op/s 2023-03-10T02:17:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:17:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:17:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:09 smithi181 ceph-mon[94189]: pgmap v22236: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 780 KiB/s rd, 772 KiB/s wr, 25 op/s 2023-03-10T02:17:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:17:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:17:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:09 smithi038 ceph-mon[92422]: pgmap v22236: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 780 KiB/s rd, 772 KiB/s wr, 25 op/s 2023-03-10T02:17:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:17:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:17:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40741.568353 secs 2023-03-10T02:17:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40741.568353 secs 2023-03-10T02:17:10.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40741.568353 secs 2023-03-10T02:17:11.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:11 smithi122 ceph-mon[93869]: pgmap v22237: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 757 KiB/s rd, 625 KiB/s wr, 23 op/s 2023-03-10T02:17:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:11 smithi181 ceph-mon[94189]: pgmap v22237: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 757 KiB/s rd, 625 KiB/s wr, 23 op/s 2023-03-10T02:17:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:11 smithi038 ceph-mon[92422]: pgmap v22237: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 757 KiB/s rd, 625 KiB/s wr, 23 op/s 2023-03-10T02:17:13.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41205.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:17:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:13 smithi122 ceph-mon[93869]: pgmap v22238: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 950 KiB/s rd, 827 KiB/s wr, 27 op/s 2023-03-10T02:17:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:13 smithi181 ceph-mon[94189]: pgmap v22238: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 950 KiB/s rd, 827 KiB/s wr, 27 op/s 2023-03-10T02:17:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:13 smithi038 ceph-mon[92422]: pgmap v22238: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 950 KiB/s rd, 827 KiB/s wr, 27 op/s 2023-03-10T02:17:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:15 smithi122 ceph-mon[93869]: pgmap v22239: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 741 KiB/s rd, 619 KiB/s wr, 22 op/s 2023-03-10T02:17:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40746.568411 secs 2023-03-10T02:17:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:15 smithi181 ceph-mon[94189]: pgmap v22239: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 741 KiB/s rd, 619 KiB/s wr, 22 op/s 2023-03-10T02:17:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40746.568411 secs 2023-03-10T02:17:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:15 smithi038 ceph-mon[92422]: pgmap v22239: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 741 KiB/s rd, 619 KiB/s wr, 22 op/s 2023-03-10T02:17:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40746.568411 secs 2023-03-10T02:17:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:17:16.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:17:16.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:17:16.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:17:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:17:16.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:17:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:17 smithi122 ceph-mon[93869]: pgmap v22240: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 384 KiB/s rd, 391 KiB/s wr, 13 op/s 2023-03-10T02:17:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:17:17.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:17:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:17 smithi181 ceph-mon[94189]: pgmap v22240: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 384 KiB/s rd, 391 KiB/s wr, 13 op/s 2023-03-10T02:17:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:17:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:17:17.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:17 smithi038 ceph-mon[92422]: pgmap v22240: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 384 KiB/s rd, 391 KiB/s wr, 13 op/s 2023-03-10T02:17:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:17:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:17:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41210.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:17:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:19 smithi122 ceph-mon[93869]: pgmap v22241: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 214 KiB/s wr, 10 op/s 2023-03-10T02:17:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:19 smithi181 ceph-mon[94189]: pgmap v22241: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 214 KiB/s wr, 10 op/s 2023-03-10T02:17:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:19 smithi038 ceph-mon[92422]: pgmap v22241: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 214 KiB/s wr, 10 op/s 2023-03-10T02:17:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40751.568467 secs 2023-03-10T02:17:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40751.568467 secs 2023-03-10T02:17:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40751.568467 secs 2023-03-10T02:17:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:21 smithi122 ceph-mon[93869]: pgmap v22242: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 214 KiB/s wr, 8 op/s 2023-03-10T02:17:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:21 smithi181 ceph-mon[94189]: pgmap v22242: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 214 KiB/s wr, 8 op/s 2023-03-10T02:17:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:21 smithi038 ceph-mon[92422]: pgmap v22242: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 206 KiB/s rd, 214 KiB/s wr, 8 op/s 2023-03-10T02:17:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41215.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:17:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:23 smithi122 ceph-mon[93869]: pgmap v22243: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 605 KiB/s rd, 623 KiB/s wr, 14 op/s 2023-03-10T02:17:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:23 smithi181 ceph-mon[94189]: pgmap v22243: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 605 KiB/s rd, 623 KiB/s wr, 14 op/s 2023-03-10T02:17:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:23 smithi038 ceph-mon[92422]: pgmap v22243: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 605 KiB/s rd, 623 KiB/s wr, 14 op/s 2023-03-10T02:17:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:25 smithi122 ceph-mon[93869]: pgmap v22244: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 402 KiB/s rd, 421 KiB/s wr, 10 op/s 2023-03-10T02:17:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:25 smithi122 ceph-mon[93869]: osdmap e1605: 12 total, 12 up, 12 in 2023-03-10T02:17:25.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40756.568527 secs 2023-03-10T02:17:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:25 smithi181 ceph-mon[94189]: pgmap v22244: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 402 KiB/s rd, 421 KiB/s wr, 10 op/s 2023-03-10T02:17:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:25 smithi181 ceph-mon[94189]: osdmap e1605: 12 total, 12 up, 12 in 2023-03-10T02:17:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40756.568527 secs 2023-03-10T02:17:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:25 smithi038 ceph-mon[92422]: pgmap v22244: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 402 KiB/s rd, 421 KiB/s wr, 10 op/s 2023-03-10T02:17:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:25 smithi038 ceph-mon[92422]: osdmap e1605: 12 total, 12 up, 12 in 2023-03-10T02:17:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40756.568527 secs 2023-03-10T02:17:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:27 smithi122 ceph-mon[93869]: pgmap v22246: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 697 KiB/s rd, 720 KiB/s wr, 13 op/s 2023-03-10T02:17:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:27 smithi181 ceph-mon[94189]: pgmap v22246: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 697 KiB/s rd, 720 KiB/s wr, 13 op/s 2023-03-10T02:17:27.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:27 smithi038 ceph-mon[92422]: pgmap v22246: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 697 KiB/s rd, 720 KiB/s wr, 13 op/s 2023-03-10T02:17:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41220.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:17:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:29 smithi122 ceph-mon[93869]: pgmap v22247: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 696 KiB/s rd, 707 KiB/s wr, 11 op/s 2023-03-10T02:17:29.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:29 smithi181 ceph-mon[94189]: pgmap v22247: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 696 KiB/s rd, 707 KiB/s wr, 11 op/s 2023-03-10T02:17:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:29 smithi038 ceph-mon[92422]: pgmap v22247: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 696 KiB/s rd, 707 KiB/s wr, 11 op/s 2023-03-10T02:17:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40761.568598 secs 2023-03-10T02:17:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40761.568598 secs 2023-03-10T02:17:30.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40761.568598 secs 2023-03-10T02:17:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:31 smithi122 ceph-mon[93869]: pgmap v22248: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 696 KiB/s rd, 707 KiB/s wr, 11 op/s 2023-03-10T02:17:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:31 smithi181 ceph-mon[94189]: pgmap v22248: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 696 KiB/s rd, 707 KiB/s wr, 11 op/s 2023-03-10T02:17:31.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:31 smithi038 ceph-mon[92422]: pgmap v22248: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 696 KiB/s rd, 707 KiB/s wr, 11 op/s 2023-03-10T02:17:32.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:32 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:17:32.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:32 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:17:32.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:32 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:17:32.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:32 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:17:32.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:32 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:17:32.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:32 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:17:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41225.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:17:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:33 smithi122 ceph-mon[93869]: pgmap v22249: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 255 KiB/s wr, 9 op/s 2023-03-10T02:17:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:33 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:17:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:33 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:17:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:33 smithi181 ceph-mon[94189]: pgmap v22249: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 255 KiB/s wr, 9 op/s 2023-03-10T02:17:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:33 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:17:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:33 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:17:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:33 smithi038 ceph-mon[92422]: pgmap v22249: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 255 KiB/s wr, 9 op/s 2023-03-10T02:17:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:33 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:17:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:33 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:17:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:35 smithi122 ceph-mon[93869]: pgmap v22250: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 255 KiB/s wr, 8 op/s 2023-03-10T02:17:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40766.568651 secs 2023-03-10T02:17:35.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:35 smithi122 ceph-mon[93869]: osdmap e1606: 12 total, 12 up, 12 in 2023-03-10T02:17:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:35 smithi181 ceph-mon[94189]: pgmap v22250: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 255 KiB/s wr, 8 op/s 2023-03-10T02:17:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40766.568651 secs 2023-03-10T02:17:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:35 smithi181 ceph-mon[94189]: osdmap e1606: 12 total, 12 up, 12 in 2023-03-10T02:17:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:35 smithi038 ceph-mon[92422]: pgmap v22250: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 256 KiB/s rd, 255 KiB/s wr, 8 op/s 2023-03-10T02:17:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40766.568651 secs 2023-03-10T02:17:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:35 smithi038 ceph-mon[92422]: osdmap e1606: 12 total, 12 up, 12 in 2023-03-10T02:17:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:37 smithi122 ceph-mon[93869]: pgmap v22252: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 115 KiB/s wr, 8 op/s 2023-03-10T02:17:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:37 smithi181 ceph-mon[94189]: pgmap v22252: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 115 KiB/s wr, 8 op/s 2023-03-10T02:17:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:37 smithi038 ceph-mon[92422]: pgmap v22252: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 115 KiB/s wr, 8 op/s 2023-03-10T02:17:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41230.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:17:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40769.081481 secs 2023-03-10T02:17:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40769.081481 secs 2023-03-10T02:17:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40769.081481 secs 2023-03-10T02:17:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:39 smithi122 ceph-mon[93869]: pgmap v22253: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 299 KiB/s wr, 9 op/s 2023-03-10T02:17:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:39 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:17:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:39 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:17:39.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:39 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:17:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:39 smithi181 ceph-mon[94189]: pgmap v22253: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 299 KiB/s wr, 9 op/s 2023-03-10T02:17:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:39 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:17:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:39 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:17:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:39 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:17:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:39 smithi038 ceph-mon[92422]: pgmap v22253: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 299 KiB/s wr, 9 op/s 2023-03-10T02:17:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:39 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:17:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:39 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:17:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:39 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:17:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40771.568719 secs 2023-03-10T02:17:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40771.568719 secs 2023-03-10T02:17:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40771.568719 secs 2023-03-10T02:17:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:41 smithi122 ceph-mon[93869]: pgmap v22254: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 299 KiB/s wr, 9 op/s 2023-03-10T02:17:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:17:41.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:17:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:41 smithi181 ceph-mon[94189]: pgmap v22254: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 299 KiB/s wr, 9 op/s 2023-03-10T02:17:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:17:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:17:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:41 smithi038 ceph-mon[92422]: pgmap v22254: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 299 KiB/s wr, 9 op/s 2023-03-10T02:17:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:17:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:17:42.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:42 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:17:42.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:42 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:17:43.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:42 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:17:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41235.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:17:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:43 smithi122 ceph-mon[93869]: pgmap v22255: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 487 KiB/s wr, 10 op/s 2023-03-10T02:17:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40774.081546 secs 2023-03-10T02:17:43.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:17:43.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:17:43.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:17:43.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:17:43.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:17:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:43 smithi181 ceph-mon[94189]: pgmap v22255: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 487 KiB/s wr, 10 op/s 2023-03-10T02:17:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40774.081546 secs 2023-03-10T02:17:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:17:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:17:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:17:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:17:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:17:43.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:43 smithi038 ceph-mon[92422]: pgmap v22255: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 487 KiB/s wr, 10 op/s 2023-03-10T02:17:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40774.081546 secs 2023-03-10T02:17:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:17:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:17:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:17:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:17:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:17:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:45 smithi122 ceph-mon[93869]: pgmap v22256: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 765 KiB/s rd, 763 KiB/s wr, 11 op/s 2023-03-10T02:17:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40776.568779 secs 2023-03-10T02:17:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:45 smithi181 ceph-mon[94189]: pgmap v22256: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 765 KiB/s rd, 763 KiB/s wr, 11 op/s 2023-03-10T02:17:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40776.568779 secs 2023-03-10T02:17:45.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:45 smithi038 ceph-mon[92422]: pgmap v22256: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 765 KiB/s rd, 763 KiB/s wr, 11 op/s 2023-03-10T02:17:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40776.568779 secs 2023-03-10T02:17:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:47 smithi122 ceph-mon[93869]: pgmap v22257: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 791 KiB/s rd, 788 KiB/s wr, 13 op/s 2023-03-10T02:17:47.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:17:47.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:17:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:47 smithi181 ceph-mon[94189]: pgmap v22257: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 791 KiB/s rd, 788 KiB/s wr, 13 op/s 2023-03-10T02:17:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:17:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:17:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:47 smithi038 ceph-mon[92422]: pgmap v22257: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 791 KiB/s rd, 788 KiB/s wr, 13 op/s 2023-03-10T02:17:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:17:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:17:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41240.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:17:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:49 smithi038 ceph-mon[92422]: pgmap v22258: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 640 KiB/s rd, 638 KiB/s wr, 9 op/s 2023-03-10T02:17:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:17:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:17:50.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:49 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T02:17:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:49 smithi122 ceph-mon[93869]: pgmap v22258: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 640 KiB/s rd, 638 KiB/s wr, 9 op/s 2023-03-10T02:17:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:17:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:17:50.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:49 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T02:17:50.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:49 smithi181 ceph-mon[94189]: pgmap v22258: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 640 KiB/s rd, 638 KiB/s wr, 9 op/s 2023-03-10T02:17:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:17:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:17:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:49 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T02:17:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:50 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:17:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:50 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:17:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:50 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:17:51.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40781.568852 secs 2023-03-10T02:17:51.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:50 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:17:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:50 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:17:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:50 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:17:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40781.568852 secs 2023-03-10T02:17:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:50 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:17:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:50 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:17:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:50 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:17:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40781.568852 secs 2023-03-10T02:17:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:51 smithi122 ceph-mon[93869]: pgmap v22259: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 598 KiB/s rd, 596 KiB/s wr, 8 op/s 2023-03-10T02:17:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:17:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:17:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:51 smithi181 ceph-mon[94189]: pgmap v22259: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 598 KiB/s rd, 596 KiB/s wr, 8 op/s 2023-03-10T02:17:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:17:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:17:52.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:51 smithi038 ceph-mon[92422]: pgmap v22259: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 598 KiB/s rd, 596 KiB/s wr, 8 op/s 2023-03-10T02:17:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:17:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:17:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:52 smithi122 ceph-mon[93869]: pgmap v22260: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 948 KiB/s rd, 943 KiB/s wr, 13 op/s 2023-03-10T02:17:53.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:52 smithi181 ceph-mon[94189]: pgmap v22260: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 948 KiB/s rd, 943 KiB/s wr, 13 op/s 2023-03-10T02:17:53.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:52 smithi038 ceph-mon[92422]: pgmap v22260: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 948 KiB/s rd, 943 KiB/s wr, 13 op/s 2023-03-10T02:17:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41245.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:17:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:17:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:17:54.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:17:54.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:17:54.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:17:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:17:55.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:54 smithi122 ceph-mon[93869]: pgmap v22261: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 853 KiB/s rd, 838 KiB/s wr, 11 op/s 2023-03-10T02:17:55.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:54 smithi181 ceph-mon[94189]: pgmap v22261: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 853 KiB/s rd, 838 KiB/s wr, 11 op/s 2023-03-10T02:17:55.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:54 smithi038 ceph-mon[92422]: pgmap v22261: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 853 KiB/s rd, 838 KiB/s wr, 11 op/s 2023-03-10T02:17:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:17:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:17:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40786.568911 secs 2023-03-10T02:17:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:17:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:17:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40786.568911 secs 2023-03-10T02:17:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:17:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:17:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40786.568911 secs 2023-03-10T02:17:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:56 smithi122 ceph-mon[93869]: pgmap v22262: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 608 KiB/s wr, 13 op/s 2023-03-10T02:17:57.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:56 smithi181 ceph-mon[94189]: pgmap v22262: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 608 KiB/s wr, 13 op/s 2023-03-10T02:17:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:56 smithi038 ceph-mon[92422]: pgmap v22262: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 625 KiB/s rd, 608 KiB/s wr, 13 op/s 2023-03-10T02:17:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:17:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:17:58.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:17:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:17:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:17:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:17:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41250.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:17:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:58 smithi122 ceph-mon[93869]: pgmap v22263: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 816 KiB/s rd, 801 KiB/s wr, 12 op/s 2023-03-10T02:17:59.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:58 smithi181 ceph-mon[94189]: pgmap v22263: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 816 KiB/s rd, 801 KiB/s wr, 12 op/s 2023-03-10T02:17:59.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:58 smithi038 ceph-mon[92422]: pgmap v22263: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 816 KiB/s rd, 801 KiB/s wr, 12 op/s 2023-03-10T02:18:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:18:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:17:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:18:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:18:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:17:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:18:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:18:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:17:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:18:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40791.568968 secs 2023-03-10T02:18:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:00 smithi122 ceph-mon[93869]: pgmap v22264: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 816 KiB/s rd, 801 KiB/s wr, 12 op/s 2023-03-10T02:18:01.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40791.568968 secs 2023-03-10T02:18:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:00 smithi181 ceph-mon[94189]: pgmap v22264: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 816 KiB/s rd, 801 KiB/s wr, 12 op/s 2023-03-10T02:18:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40791.568968 secs 2023-03-10T02:18:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:00 smithi038 ceph-mon[92422]: pgmap v22264: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 816 KiB/s rd, 801 KiB/s wr, 12 op/s 2023-03-10T02:18:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:18:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:18:02.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:18:02.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:18:02.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:18:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:18:03.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:02 smithi122 ceph-mon[93869]: pgmap v22265: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 15 op/s 2023-03-10T02:18:03.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:02 smithi181 ceph-mon[94189]: pgmap v22265: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 15 op/s 2023-03-10T02:18:03.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:02 smithi038 ceph-mon[92422]: pgmap v22265: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 15 op/s 2023-03-10T02:18:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41255.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:18:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:03 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 40794.081828 secs 2023-03-10T02:18:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:03 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 40794.081828 secs 2023-03-10T02:18:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:03 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 40794.081828 secs 2023-03-10T02:18:05.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:04 smithi122 ceph-mon[93869]: pgmap v22266: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 767 KiB/s rd, 753 KiB/s wr, 13 op/s 2023-03-10T02:18:05.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:04 smithi181 ceph-mon[94189]: pgmap v22266: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 767 KiB/s rd, 753 KiB/s wr, 13 op/s 2023-03-10T02:18:05.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:04 smithi038 ceph-mon[92422]: pgmap v22266: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 767 KiB/s rd, 753 KiB/s wr, 13 op/s 2023-03-10T02:18:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40796.568996 secs 2023-03-10T02:18:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40796.568996 secs 2023-03-10T02:18:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40796.568996 secs 2023-03-10T02:18:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:06 smithi122 ceph-mon[93869]: pgmap v22267: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 842 KiB/s rd, 849 KiB/s wr, 16 op/s 2023-03-10T02:18:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:06 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:18:07.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:06 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:18:07.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:18:07.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:06 smithi181 ceph-mon[94189]: pgmap v22267: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 842 KiB/s rd, 849 KiB/s wr, 16 op/s 2023-03-10T02:18:07.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:06 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:18:07.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:06 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:18:07.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:18:07.248 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:06 smithi038 ceph-mon[92422]: pgmap v22267: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 842 KiB/s rd, 849 KiB/s wr, 16 op/s 2023-03-10T02:18:07.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:06 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:18:07.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:06 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:18:07.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:18:08.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41260.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:18:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:08 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 40799.081914 secs 2023-03-10T02:18:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:18:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:18:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:08 smithi038 ceph-mon[92422]: pgmap v22268: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 847 KiB/s rd, 860 KiB/s wr, 24 op/s 2023-03-10T02:18:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:08 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 40799.081914 secs 2023-03-10T02:18:09.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:18:09.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:18:09.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:08 smithi122 ceph-mon[93869]: pgmap v22268: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 847 KiB/s rd, 860 KiB/s wr, 24 op/s 2023-03-10T02:18:09.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:08 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 40799.081914 secs 2023-03-10T02:18:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:18:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:18:09.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:08 smithi181 ceph-mon[94189]: pgmap v22268: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 847 KiB/s rd, 860 KiB/s wr, 24 op/s 2023-03-10T02:18:10.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40801.569083 secs 2023-03-10T02:18:10.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40801.569083 secs 2023-03-10T02:18:10.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40801.569083 secs 2023-03-10T02:18:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:11 smithi122 ceph-mon[93869]: pgmap v22269: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 897 KiB/s rd, 910 KiB/s wr, 22 op/s 2023-03-10T02:18:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:11 smithi181 ceph-mon[94189]: pgmap v22269: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 897 KiB/s rd, 910 KiB/s wr, 22 op/s 2023-03-10T02:18:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:11 smithi038 ceph-mon[92422]: pgmap v22269: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 897 KiB/s rd, 910 KiB/s wr, 22 op/s 2023-03-10T02:18:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:12 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:18:12.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:12 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:18:12.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:12 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:18:12.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:12 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:18:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:12 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:18:12.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:12 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:18:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41265.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:18:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:13 smithi122 ceph-mon[93869]: pgmap v22270: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 26 op/s 2023-03-10T02:18:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:13 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:18:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:13 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:18:13.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:13 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 40804.081972 secs 2023-03-10T02:18:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:13 smithi181 ceph-mon[94189]: pgmap v22270: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 26 op/s 2023-03-10T02:18:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:13 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:18:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:13 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:18:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:13 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 40804.081972 secs 2023-03-10T02:18:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:13 smithi038 ceph-mon[92422]: pgmap v22270: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 26 op/s 2023-03-10T02:18:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:13 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:18:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:13 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:18:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:13 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 40804.081972 secs 2023-03-10T02:18:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:15 smithi122 ceph-mon[93869]: pgmap v22271: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 837 KiB/s rd, 850 KiB/s wr, 23 op/s 2023-03-10T02:18:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40806.569144 secs 2023-03-10T02:18:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:15 smithi181 ceph-mon[94189]: pgmap v22271: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 837 KiB/s rd, 850 KiB/s wr, 23 op/s 2023-03-10T02:18:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40806.569144 secs 2023-03-10T02:18:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:15 smithi038 ceph-mon[92422]: pgmap v22271: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 837 KiB/s rd, 850 KiB/s wr, 23 op/s 2023-03-10T02:18:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40806.569144 secs 2023-03-10T02:18:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:17 smithi122 ceph-mon[93869]: pgmap v22272: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 805 KiB/s rd, 817 KiB/s wr, 24 op/s 2023-03-10T02:18:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:17 smithi181 ceph-mon[94189]: pgmap v22272: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 805 KiB/s rd, 817 KiB/s wr, 24 op/s 2023-03-10T02:18:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:17 smithi038 ceph-mon[92422]: pgmap v22272: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 805 KiB/s rd, 817 KiB/s wr, 24 op/s 2023-03-10T02:18:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41270.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:18:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:18 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 40809.082062 secs 2023-03-10T02:18:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:18 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 40809.082062 secs 2023-03-10T02:18:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:18 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 40809.082062 secs 2023-03-10T02:18:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:19 smithi122 ceph-mon[93869]: pgmap v22273: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 634 KiB/s rd, 637 KiB/s wr, 19 op/s 2023-03-10T02:18:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:19 smithi181 ceph-mon[94189]: pgmap v22273: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 634 KiB/s rd, 637 KiB/s wr, 19 op/s 2023-03-10T02:18:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:19 smithi038 ceph-mon[92422]: pgmap v22273: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 634 KiB/s rd, 637 KiB/s wr, 19 op/s 2023-03-10T02:18:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40811.569227 secs 2023-03-10T02:18:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40811.569227 secs 2023-03-10T02:18:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40811.569227 secs 2023-03-10T02:18:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:21 smithi122 ceph-mon[93869]: pgmap v22274: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 626 KiB/s wr, 9 op/s 2023-03-10T02:18:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:21 smithi181 ceph-mon[94189]: pgmap v22274: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 626 KiB/s wr, 9 op/s 2023-03-10T02:18:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:21 smithi038 ceph-mon[92422]: pgmap v22274: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 628 KiB/s rd, 626 KiB/s wr, 9 op/s 2023-03-10T02:18:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41275.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:18:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:23 smithi122 ceph-mon[93869]: pgmap v22275: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 354 KiB/s wr, 13 op/s 2023-03-10T02:18:23.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:23 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 40814.082121 secs 2023-03-10T02:18:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:23 smithi181 ceph-mon[94189]: pgmap v22275: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 354 KiB/s wr, 13 op/s 2023-03-10T02:18:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:23 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 40814.082121 secs 2023-03-10T02:18:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:23 smithi038 ceph-mon[92422]: pgmap v22275: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 356 KiB/s rd, 354 KiB/s wr, 13 op/s 2023-03-10T02:18:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:23 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 40814.082121 secs 2023-03-10T02:18:25.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:25 smithi122 ceph-mon[93869]: pgmap v22276: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 71 KiB/s wr, 11 op/s 2023-03-10T02:18:25.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:25 smithi122 ceph-mon[93869]: osdmap e1607: 12 total, 12 up, 12 in 2023-03-10T02:18:25.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40816.569270 secs 2023-03-10T02:18:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:25 smithi181 ceph-mon[94189]: pgmap v22276: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 71 KiB/s wr, 11 op/s 2023-03-10T02:18:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:25 smithi181 ceph-mon[94189]: osdmap e1607: 12 total, 12 up, 12 in 2023-03-10T02:18:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40816.569270 secs 2023-03-10T02:18:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:25 smithi038 ceph-mon[92422]: pgmap v22276: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 71 KiB/s wr, 11 op/s 2023-03-10T02:18:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:25 smithi038 ceph-mon[92422]: osdmap e1607: 12 total, 12 up, 12 in 2023-03-10T02:18:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40816.569270 secs 2023-03-10T02:18:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:27 smithi122 ceph-mon[93869]: pgmap v22278: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 391 KiB/s wr, 14 op/s 2023-03-10T02:18:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:27 smithi181 ceph-mon[94189]: pgmap v22278: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 391 KiB/s wr, 14 op/s 2023-03-10T02:18:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:27 smithi038 ceph-mon[92422]: pgmap v22278: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 367 KiB/s rd, 391 KiB/s wr, 14 op/s 2023-03-10T02:18:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41280.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:18:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:28 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:18:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:28 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:18:28.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40819.082179 secs 2023-03-10T02:18:28.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:28 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:18:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:28 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:18:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40819.082179 secs 2023-03-10T02:18:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:28 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:18:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:28 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:18:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40819.082179 secs 2023-03-10T02:18:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:29 smithi122 ceph-mon[93869]: pgmap v22279: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 412 KiB/s wr, 15 op/s 2023-03-10T02:18:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:29 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:18:29.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:29 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:18:29.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:29 smithi181 ceph-mon[94189]: pgmap v22279: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 412 KiB/s wr, 15 op/s 2023-03-10T02:18:29.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:29 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:18:29.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:29 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:18:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:29 smithi038 ceph-mon[92422]: pgmap v22279: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 388 KiB/s rd, 412 KiB/s wr, 15 op/s 2023-03-10T02:18:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:29 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:18:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:29 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:18:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40821.569326 secs 2023-03-10T02:18:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40821.569326 secs 2023-03-10T02:18:30.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40821.569326 secs 2023-03-10T02:18:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:31 smithi122 ceph-mon[93869]: pgmap v22280: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 413 KiB/s wr, 15 op/s 2023-03-10T02:18:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:31 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:18:31.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:31 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:18:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:31 smithi181 ceph-mon[94189]: pgmap v22280: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 413 KiB/s wr, 15 op/s 2023-03-10T02:18:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:31 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:18:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:31 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:18:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:31 smithi038 ceph-mon[92422]: pgmap v22280: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 389 KiB/s rd, 413 KiB/s wr, 15 op/s 2023-03-10T02:18:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:31 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:18:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:31 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:18:32.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:32 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:18:32.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:32 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:18:32.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:32 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:18:32.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:32 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:18:32.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:32 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:18:32.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:32 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:18:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41285.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:18:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:33 smithi122 ceph-mon[93869]: pgmap v22281: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 671 KiB/s rd, 695 KiB/s wr, 15 op/s 2023-03-10T02:18:33.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40824.082247 secs 2023-03-10T02:18:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:33 smithi181 ceph-mon[94189]: pgmap v22281: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 671 KiB/s rd, 695 KiB/s wr, 15 op/s 2023-03-10T02:18:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40824.082247 secs 2023-03-10T02:18:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:33 smithi038 ceph-mon[92422]: pgmap v22281: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 671 KiB/s rd, 695 KiB/s wr, 15 op/s 2023-03-10T02:18:34.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40824.082247 secs 2023-03-10T02:18:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:35 smithi122 ceph-mon[93869]: pgmap v22282: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 641 KiB/s rd, 653 KiB/s wr, 13 op/s 2023-03-10T02:18:35.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40826.569384 secs 2023-03-10T02:18:35.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:35 smithi122 ceph-mon[93869]: osdmap e1608: 12 total, 12 up, 12 in 2023-03-10T02:18:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:35 smithi181 ceph-mon[94189]: pgmap v22282: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 641 KiB/s rd, 653 KiB/s wr, 13 op/s 2023-03-10T02:18:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40826.569384 secs 2023-03-10T02:18:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:35 smithi181 ceph-mon[94189]: osdmap e1608: 12 total, 12 up, 12 in 2023-03-10T02:18:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:35 smithi038 ceph-mon[92422]: pgmap v22282: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 641 KiB/s rd, 653 KiB/s wr, 13 op/s 2023-03-10T02:18:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40826.569384 secs 2023-03-10T02:18:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:35 smithi038 ceph-mon[92422]: osdmap e1608: 12 total, 12 up, 12 in 2023-03-10T02:18:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:37 smithi122 ceph-mon[93869]: pgmap v22284: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 347 KiB/s wr, 10 op/s 2023-03-10T02:18:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:37 smithi181 ceph-mon[94189]: pgmap v22284: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 347 KiB/s wr, 10 op/s 2023-03-10T02:18:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:37 smithi038 ceph-mon[92422]: pgmap v22284: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 347 KiB/s wr, 10 op/s 2023-03-10T02:18:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41290.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:18:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40829.082335 secs 2023-03-10T02:18:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40829.082335 secs 2023-03-10T02:18:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40829.082335 secs 2023-03-10T02:18:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:39 smithi122 ceph-mon[93869]: pgmap v22285: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 328 KiB/s rd, 326 KiB/s wr, 10 op/s 2023-03-10T02:18:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:39 smithi181 ceph-mon[94189]: pgmap v22285: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 328 KiB/s rd, 326 KiB/s wr, 10 op/s 2023-03-10T02:18:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:39 smithi038 ceph-mon[92422]: pgmap v22285: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 328 KiB/s rd, 326 KiB/s wr, 10 op/s 2023-03-10T02:18:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40831.569444 secs 2023-03-10T02:18:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40831.569444 secs 2023-03-10T02:18:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40831.569444 secs 2023-03-10T02:18:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:41 smithi122 ceph-mon[93869]: pgmap v22286: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 326 KiB/s wr, 9 op/s 2023-03-10T02:18:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:41 smithi181 ceph-mon[94189]: pgmap v22286: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 326 KiB/s wr, 9 op/s 2023-03-10T02:18:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:41 smithi038 ceph-mon[92422]: pgmap v22286: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 327 KiB/s rd, 326 KiB/s wr, 9 op/s 2023-03-10T02:18:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41295.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:18:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:43 smithi122 ceph-mon[93869]: pgmap v22287: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 429 KiB/s wr, 9 op/s 2023-03-10T02:18:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40834.082404 secs 2023-03-10T02:18:43.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:43 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:18:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:43 smithi181 ceph-mon[94189]: pgmap v22287: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 429 KiB/s wr, 9 op/s 2023-03-10T02:18:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40834.082404 secs 2023-03-10T02:18:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:43 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:18:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:43 smithi038 ceph-mon[92422]: pgmap v22287: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 429 KiB/s wr, 9 op/s 2023-03-10T02:18:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40834.082404 secs 2023-03-10T02:18:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:43 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:18:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:18:45.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:18:45.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:18:45.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:18:45.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:45 smithi122 ceph-mon[93869]: pgmap v22288: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 429 KiB/s wr, 8 op/s 2023-03-10T02:18:45.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:18:45.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:18:45.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:18:45.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:18:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:18:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:18:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:18:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:45 smithi181 ceph-mon[94189]: pgmap v22288: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 429 KiB/s wr, 8 op/s 2023-03-10T02:18:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:18:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:18:45.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:18:45.498 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:18:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:18:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:18:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:18:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:45 smithi038 ceph-mon[92422]: pgmap v22288: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 429 KiB/s wr, 8 op/s 2023-03-10T02:18:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:18:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:18:45.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:18:46.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:46 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40836.569503 secs 2023-03-10T02:18:46.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:46 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40836.569503 secs 2023-03-10T02:18:46.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:46 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40836.569503 secs 2023-03-10T02:18:47.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:47 smithi122 ceph-mon[93869]: pgmap v22289: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 647 KiB/s rd, 600 KiB/s wr, 11 op/s 2023-03-10T02:18:47.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:47 smithi181 ceph-mon[94189]: pgmap v22289: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 647 KiB/s rd, 600 KiB/s wr, 11 op/s 2023-03-10T02:18:47.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:47 smithi038 ceph-mon[92422]: pgmap v22289: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 647 KiB/s rd, 600 KiB/s wr, 11 op/s 2023-03-10T02:18:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:48 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:18:48.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:48 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:18:48.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:48 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:18:48.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:48 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:18:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:48 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:18:48.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:48 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:18:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41300.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:18:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:49 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40839.082493 secs 2023-03-10T02:18:49.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:18:49.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:18:49.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:49 smithi122 ceph-mon[93869]: pgmap v22290: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 535 KiB/s wr, 8 op/s 2023-03-10T02:18:49.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:49 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40839.082493 secs 2023-03-10T02:18:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:18:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:18:49.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:49 smithi181 ceph-mon[94189]: pgmap v22290: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 535 KiB/s wr, 8 op/s 2023-03-10T02:18:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:49 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40839.082493 secs 2023-03-10T02:18:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:18:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:18:49.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:49 smithi038 ceph-mon[92422]: pgmap v22290: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 535 KiB/s wr, 8 op/s 2023-03-10T02:18:50.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40841.569589 secs 2023-03-10T02:18:50.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40841.569589 secs 2023-03-10T02:18:50.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40841.569589 secs 2023-03-10T02:18:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:51 smithi122 ceph-mon[93869]: pgmap v22291: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 535 KiB/s wr, 8 op/s 2023-03-10T02:18:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:51 smithi181 ceph-mon[94189]: pgmap v22291: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 535 KiB/s wr, 8 op/s 2023-03-10T02:18:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:51 smithi038 ceph-mon[92422]: pgmap v22291: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 535 KiB/s wr, 8 op/s 2023-03-10T02:18:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41305.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:18:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:53 smithi122 ceph-mon[93869]: pgmap v22292: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 779 KiB/s rd, 737 KiB/s wr, 11 op/s 2023-03-10T02:18:53.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:18:53.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:18:53.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40844.082556 secs 2023-03-10T02:18:53.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:53 smithi181 ceph-mon[94189]: pgmap v22292: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 779 KiB/s rd, 737 KiB/s wr, 11 op/s 2023-03-10T02:18:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:18:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:18:53.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40844.082556 secs 2023-03-10T02:18:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:53 smithi038 ceph-mon[92422]: pgmap v22292: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 779 KiB/s rd, 737 KiB/s wr, 11 op/s 2023-03-10T02:18:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:18:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:18:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40844.082556 secs 2023-03-10T02:18:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:55 smithi122 ceph-mon[93869]: pgmap v22293: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 456 KiB/s rd, 416 KiB/s wr, 9 op/s 2023-03-10T02:18:55.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40846.569648 secs 2023-03-10T02:18:55.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:55 smithi181 ceph-mon[94189]: pgmap v22293: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 456 KiB/s rd, 416 KiB/s wr, 9 op/s 2023-03-10T02:18:55.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40846.569648 secs 2023-03-10T02:18:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:55 smithi038 ceph-mon[92422]: pgmap v22293: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 456 KiB/s rd, 416 KiB/s wr, 9 op/s 2023-03-10T02:18:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40846.569648 secs 2023-03-10T02:18:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:18:56.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:18:56.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:18:56.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:18:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:18:56.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:18:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:57 smithi122 ceph-mon[93869]: pgmap v22294: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 580 KiB/s rd, 416 KiB/s wr, 11 op/s 2023-03-10T02:18:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:57 smithi181 ceph-mon[94189]: pgmap v22294: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 580 KiB/s rd, 416 KiB/s wr, 11 op/s 2023-03-10T02:18:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:57 smithi038 ceph-mon[92422]: pgmap v22294: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 580 KiB/s rd, 416 KiB/s wr, 11 op/s 2023-03-10T02:18:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41310.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:18:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40849.082615 secs 2023-03-10T02:18:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40849.082615 secs 2023-03-10T02:18:58.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40849.082615 secs 2023-03-10T02:18:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:18:59 smithi122 ceph-mon[93869]: pgmap v22295: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 309 KiB/s wr, 11 op/s 2023-03-10T02:18:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:18:59 smithi181 ceph-mon[94189]: pgmap v22295: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 309 KiB/s wr, 11 op/s 2023-03-10T02:18:59.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:18:59 smithi038 ceph-mon[92422]: pgmap v22295: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 372 KiB/s rd, 309 KiB/s wr, 11 op/s 2023-03-10T02:19:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:19:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:19:00.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40851.569711 secs 2023-03-10T02:19:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:19:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:19:00.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40851.569711 secs 2023-03-10T02:19:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:19:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:19:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40851.569711 secs 2023-03-10T02:19:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:01 smithi122 ceph-mon[93869]: pgmap v22296: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 345 KiB/s wr, 11 op/s 2023-03-10T02:19:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:19:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:19:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:01 smithi181 ceph-mon[94189]: pgmap v22296: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 345 KiB/s wr, 11 op/s 2023-03-10T02:19:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:19:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:19:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:01 smithi038 ceph-mon[92422]: pgmap v22296: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 345 KiB/s wr, 11 op/s 2023-03-10T02:19:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:19:02.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:19:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41315.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:19:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:03 smithi122 ceph-mon[93869]: pgmap v22297: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 753 KiB/s rd, 689 KiB/s wr, 14 op/s 2023-03-10T02:19:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40854.082718 secs 2023-03-10T02:19:03.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:03 smithi181 ceph-mon[94189]: pgmap v22297: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 753 KiB/s rd, 689 KiB/s wr, 14 op/s 2023-03-10T02:19:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40854.082718 secs 2023-03-10T02:19:03.998 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:03 smithi038 ceph-mon[92422]: pgmap v22297: 129 pgs: 129 active+clean; 11 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 753 KiB/s rd, 689 KiB/s wr, 14 op/s 2023-03-10T02:19:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40854.082718 secs 2023-03-10T02:19:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:05 smithi122 ceph-mon[93869]: pgmap v22298: 129 pgs: 129 active+clean; 11 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1024 KiB/s rd, 832 KiB/s wr, 24 op/s 2023-03-10T02:19:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40856.569797 secs 2023-03-10T02:19:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:05 smithi181 ceph-mon[94189]: pgmap v22298: 129 pgs: 129 active+clean; 11 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1024 KiB/s rd, 832 KiB/s wr, 24 op/s 2023-03-10T02:19:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40856.569797 secs 2023-03-10T02:19:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:05 smithi038 ceph-mon[92422]: pgmap v22298: 129 pgs: 129 active+clean; 11 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1024 KiB/s rd, 832 KiB/s wr, 24 op/s 2023-03-10T02:19:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40856.569797 secs 2023-03-10T02:19:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:07 smithi122 ceph-mon[93869]: pgmap v22299: 129 pgs: 129 active+clean; 11 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1022 KiB/s rd, 830 KiB/s wr, 27 op/s 2023-03-10T02:19:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:07 smithi181 ceph-mon[94189]: pgmap v22299: 129 pgs: 129 active+clean; 11 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1022 KiB/s rd, 830 KiB/s wr, 27 op/s 2023-03-10T02:19:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:07 smithi038 ceph-mon[92422]: pgmap v22299: 129 pgs: 129 active+clean; 11 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1022 KiB/s rd, 830 KiB/s wr, 27 op/s 2023-03-10T02:19:08.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41320.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:19:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:19:08.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:19:08.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40859.082830 secs 2023-03-10T02:19:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:19:08.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:19:08.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40859.082830 secs 2023-03-10T02:19:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:19:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:19:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40859.082830 secs 2023-03-10T02:19:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:09 smithi122 ceph-mon[93869]: pgmap v22300: 129 pgs: 129 active+clean; 11 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 898 KiB/s rd, 830 KiB/s wr, 25 op/s 2023-03-10T02:19:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:19:09.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:19:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:09 smithi181 ceph-mon[94189]: pgmap v22300: 129 pgs: 129 active+clean; 11 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 898 KiB/s rd, 830 KiB/s wr, 25 op/s 2023-03-10T02:19:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:19:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:19:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:09 smithi038 ceph-mon[92422]: pgmap v22300: 129 pgs: 129 active+clean; 11 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 898 KiB/s rd, 830 KiB/s wr, 25 op/s 2023-03-10T02:19:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:19:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:19:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40861.569851 secs 2023-03-10T02:19:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40861.569851 secs 2023-03-10T02:19:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40861.569851 secs 2023-03-10T02:19:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:11 smithi122 ceph-mon[93869]: pgmap v22301: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 24 op/s 2023-03-10T02:19:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:11 smithi181 ceph-mon[94189]: pgmap v22301: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 24 op/s 2023-03-10T02:19:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:11 smithi038 ceph-mon[92422]: pgmap v22301: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 24 op/s 2023-03-10T02:19:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41325.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:19:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:13 smithi122 ceph-mon[93869]: pgmap v22302: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 26 op/s 2023-03-10T02:19:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40864.082889 secs 2023-03-10T02:19:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:13 smithi181 ceph-mon[94189]: pgmap v22302: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 26 op/s 2023-03-10T02:19:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40864.082889 secs 2023-03-10T02:19:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:13 smithi038 ceph-mon[92422]: pgmap v22302: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 26 op/s 2023-03-10T02:19:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40864.082889 secs 2023-03-10T02:19:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:15 smithi122 ceph-mon[93869]: pgmap v22303: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 936 KiB/s wr, 23 op/s 2023-03-10T02:19:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:15 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:19:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:15 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:19:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40866.569911 secs 2023-03-10T02:19:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:15 smithi181 ceph-mon[94189]: pgmap v22303: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 936 KiB/s wr, 23 op/s 2023-03-10T02:19:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:15 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:19:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:15 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:19:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40866.569911 secs 2023-03-10T02:19:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:15 smithi038 ceph-mon[92422]: pgmap v22303: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 936 KiB/s wr, 23 op/s 2023-03-10T02:19:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:15 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:19:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:15 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:19:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40866.569911 secs 2023-03-10T02:19:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:17 smithi122 ceph-mon[93869]: pgmap v22304: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 593 KiB/s rd, 590 KiB/s wr, 11 op/s 2023-03-10T02:19:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:19:17.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:19:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:17 smithi181 ceph-mon[94189]: pgmap v22304: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 593 KiB/s rd, 590 KiB/s wr, 11 op/s 2023-03-10T02:19:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:19:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:19:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:17 smithi038 ceph-mon[92422]: pgmap v22304: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 593 KiB/s rd, 590 KiB/s wr, 11 op/s 2023-03-10T02:19:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:19:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:19:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41330.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:19:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40869.082948 secs 2023-03-10T02:19:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40869.082948 secs 2023-03-10T02:19:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40869.082948 secs 2023-03-10T02:19:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:19 smithi122 ceph-mon[93869]: pgmap v22305: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 560 KiB/s rd, 568 KiB/s wr, 9 op/s 2023-03-10T02:19:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:19 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:19:19.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:19 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:19:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:19 smithi181 ceph-mon[94189]: pgmap v22305: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 560 KiB/s rd, 568 KiB/s wr, 9 op/s 2023-03-10T02:19:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:19 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:19:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:19 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:19:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:19 smithi038 ceph-mon[92422]: pgmap v22305: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 560 KiB/s rd, 568 KiB/s wr, 9 op/s 2023-03-10T02:19:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:19 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:19:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:19 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:19:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40871.569969 secs 2023-03-10T02:19:20.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40871.569969 secs 2023-03-10T02:19:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40871.569969 secs 2023-03-10T02:19:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:21 smithi122 ceph-mon[93869]: pgmap v22306: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 595 KiB/s rd, 603 KiB/s wr, 9 op/s 2023-03-10T02:19:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:19:21.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:19:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:21 smithi181 ceph-mon[94189]: pgmap v22306: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 595 KiB/s rd, 603 KiB/s wr, 9 op/s 2023-03-10T02:19:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:19:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:19:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:21 smithi038 ceph-mon[92422]: pgmap v22306: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 595 KiB/s rd, 603 KiB/s wr, 9 op/s 2023-03-10T02:19:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:19:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:19:23.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41335.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:19:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:23 smithi122 ceph-mon[93869]: pgmap v22307: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 671 KiB/s rd, 689 KiB/s wr, 12 op/s 2023-03-10T02:19:23.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40874.083006 secs 2023-03-10T02:19:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:23 smithi181 ceph-mon[94189]: pgmap v22307: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 671 KiB/s rd, 689 KiB/s wr, 12 op/s 2023-03-10T02:19:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40874.083006 secs 2023-03-10T02:19:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:23 smithi038 ceph-mon[92422]: pgmap v22307: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 671 KiB/s rd, 689 KiB/s wr, 12 op/s 2023-03-10T02:19:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40874.083006 secs 2023-03-10T02:19:25.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:25 smithi122 ceph-mon[93869]: pgmap v22308: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 435 KiB/s rd, 455 KiB/s wr, 10 op/s 2023-03-10T02:19:25.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:25 smithi122 ceph-mon[93869]: osdmap e1609: 12 total, 12 up, 12 in 2023-03-10T02:19:25.978 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40876.570026 secs 2023-03-10T02:19:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:25 smithi181 ceph-mon[94189]: pgmap v22308: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 435 KiB/s rd, 455 KiB/s wr, 10 op/s 2023-03-10T02:19:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:25 smithi181 ceph-mon[94189]: osdmap e1609: 12 total, 12 up, 12 in 2023-03-10T02:19:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40876.570026 secs 2023-03-10T02:19:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:25 smithi038 ceph-mon[92422]: pgmap v22308: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 435 KiB/s rd, 455 KiB/s wr, 10 op/s 2023-03-10T02:19:26.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:25 smithi038 ceph-mon[92422]: osdmap e1609: 12 total, 12 up, 12 in 2023-03-10T02:19:26.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40876.570026 secs 2023-03-10T02:19:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:27 smithi038 ceph-mon[92422]: pgmap v22310: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 994 KiB/s rd, 969 KiB/s wr, 13 op/s 2023-03-10T02:19:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:27 smithi122 ceph-mon[93869]: pgmap v22310: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 994 KiB/s rd, 969 KiB/s wr, 13 op/s 2023-03-10T02:19:28.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:27 smithi181 ceph-mon[94189]: pgmap v22310: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 994 KiB/s rd, 969 KiB/s wr, 13 op/s 2023-03-10T02:19:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41340.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:19:29.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40879.083131 secs 2023-03-10T02:19:29.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40879.083131 secs 2023-03-10T02:19:29.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40879.083131 secs 2023-03-10T02:19:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:29 smithi122 ceph-mon[93869]: pgmap v22311: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 993 KiB/s rd, 957 KiB/s wr, 11 op/s 2023-03-10T02:19:30.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:29 smithi181 ceph-mon[94189]: pgmap v22311: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 993 KiB/s rd, 957 KiB/s wr, 11 op/s 2023-03-10T02:19:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:29 smithi038 ceph-mon[92422]: pgmap v22311: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 993 KiB/s rd, 957 KiB/s wr, 11 op/s 2023-03-10T02:19:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40881.570082 secs 2023-03-10T02:19:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40881.570082 secs 2023-03-10T02:19:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40881.570082 secs 2023-03-10T02:19:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:31 smithi122 ceph-mon[93869]: pgmap v22312: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 993 KiB/s rd, 956 KiB/s wr, 11 op/s 2023-03-10T02:19:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:31 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:19:32.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:31 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:19:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:31 smithi181 ceph-mon[94189]: pgmap v22312: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 993 KiB/s rd, 956 KiB/s wr, 11 op/s 2023-03-10T02:19:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:31 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:19:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:31 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:19:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:31 smithi038 ceph-mon[92422]: pgmap v22312: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 993 KiB/s rd, 956 KiB/s wr, 11 op/s 2023-03-10T02:19:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:31 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:19:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:31 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:19:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41345.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:19:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:33 smithi122 ceph-mon[93869]: pgmap v22313: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 479 KiB/s wr, 9 op/s 2023-03-10T02:19:34.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40884.083197 secs 2023-03-10T02:19:34.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:33 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:19:34.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:33 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:19:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:33 smithi181 ceph-mon[94189]: pgmap v22313: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 479 KiB/s wr, 9 op/s 2023-03-10T02:19:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40884.083197 secs 2023-03-10T02:19:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:33 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:19:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:33 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:19:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:33 smithi038 ceph-mon[92422]: pgmap v22313: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 528 KiB/s rd, 479 KiB/s wr, 9 op/s 2023-03-10T02:19:34.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40884.083197 secs 2023-03-10T02:19:34.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:33 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:19:34.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:33 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:19:35.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:34 smithi122 ceph-mon[93869]: pgmap v22314: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 832 KiB/s rd, 783 KiB/s wr, 10 op/s 2023-03-10T02:19:35.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:34 smithi181 ceph-mon[94189]: pgmap v22314: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 832 KiB/s rd, 783 KiB/s wr, 10 op/s 2023-03-10T02:19:35.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:34 smithi038 ceph-mon[92422]: pgmap v22314: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 832 KiB/s rd, 783 KiB/s wr, 10 op/s 2023-03-10T02:19:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:35 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:19:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:35 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:19:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40886.570144 secs 2023-03-10T02:19:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:35 smithi122 ceph-mon[93869]: osdmap e1610: 12 total, 12 up, 12 in 2023-03-10T02:19:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:35 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:19:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:35 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:19:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40886.570144 secs 2023-03-10T02:19:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:35 smithi181 ceph-mon[94189]: osdmap e1610: 12 total, 12 up, 12 in 2023-03-10T02:19:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:35 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:19:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:35 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:19:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40886.570144 secs 2023-03-10T02:19:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:35 smithi038 ceph-mon[92422]: osdmap e1610: 12 total, 12 up, 12 in 2023-03-10T02:19:37.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:36 smithi122 ceph-mon[93869]: pgmap v22316: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 827 KiB/s rd, 795 KiB/s wr, 10 op/s 2023-03-10T02:19:37.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:36 smithi181 ceph-mon[94189]: pgmap v22316: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 827 KiB/s rd, 795 KiB/s wr, 10 op/s 2023-03-10T02:19:37.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:36 smithi038 ceph-mon[92422]: pgmap v22316: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 827 KiB/s rd, 795 KiB/s wr, 10 op/s 2023-03-10T02:19:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:19:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:19:38.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:19:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:19:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:19:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:19:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41350.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:19:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40889.083288 secs 2023-03-10T02:19:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:38 smithi122 ceph-mon[93869]: pgmap v22317: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 827 KiB/s rd, 795 KiB/s wr, 10 op/s 2023-03-10T02:19:39.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40889.083288 secs 2023-03-10T02:19:39.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:38 smithi181 ceph-mon[94189]: pgmap v22317: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 827 KiB/s rd, 795 KiB/s wr, 10 op/s 2023-03-10T02:19:39.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40889.083288 secs 2023-03-10T02:19:39.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:38 smithi038 ceph-mon[92422]: pgmap v22317: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 827 KiB/s rd, 795 KiB/s wr, 10 op/s 2023-03-10T02:19:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:39 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:19:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:39 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:19:40.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:39 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:19:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:39 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:19:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:39 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:19:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:39 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:19:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40891.570203 secs 2023-03-10T02:19:41.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:40 smithi122 ceph-mon[93869]: pgmap v22318: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 933 KiB/s rd, 901 KiB/s wr, 10 op/s 2023-03-10T02:19:41.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40891.570203 secs 2023-03-10T02:19:41.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:40 smithi181 ceph-mon[94189]: pgmap v22318: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 933 KiB/s rd, 901 KiB/s wr, 10 op/s 2023-03-10T02:19:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40891.570203 secs 2023-03-10T02:19:41.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:40 smithi038 ceph-mon[92422]: pgmap v22318: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 933 KiB/s rd, 901 KiB/s wr, 10 op/s 2023-03-10T02:19:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:19:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:19:42.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:19:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:19:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:19:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:19:43.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:42 smithi122 ceph-mon[93869]: pgmap v22319: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 11 op/s 2023-03-10T02:19:43.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:42 smithi181 ceph-mon[94189]: pgmap v22319: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 11 op/s 2023-03-10T02:19:43.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:42 smithi038 ceph-mon[92422]: pgmap v22319: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 11 op/s 2023-03-10T02:19:43.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41355.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:19:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40894.083349 secs 2023-03-10T02:19:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40894.083349 secs 2023-03-10T02:19:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40894.083349 secs 2023-03-10T02:19:45.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:44 smithi122 ceph-mon[93869]: pgmap v22320: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1009 KiB/s wr, 10 op/s 2023-03-10T02:19:45.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:44 smithi181 ceph-mon[94189]: pgmap v22320: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1009 KiB/s wr, 10 op/s 2023-03-10T02:19:45.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:44 smithi038 ceph-mon[92422]: pgmap v22320: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1009 KiB/s wr, 10 op/s 2023-03-10T02:19:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:19:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40896.570262 secs 2023-03-10T02:19:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:19:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:19:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:19:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:19:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40896.570262 secs 2023-03-10T02:19:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:19:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:19:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:19:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:19:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40896.570262 secs 2023-03-10T02:19:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:19:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:19:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:19:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:46 smithi122 ceph-mon[93869]: pgmap v22321: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 13 op/s 2023-03-10T02:19:47.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:46 smithi181 ceph-mon[94189]: pgmap v22321: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 13 op/s 2023-03-10T02:19:47.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:46 smithi038 ceph-mon[92422]: pgmap v22321: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 13 op/s 2023-03-10T02:19:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:19:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:19:48.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:19:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:19:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:19:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:19:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41360.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:19:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40899.083447 secs 2023-03-10T02:19:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:48 smithi122 ceph-mon[93869]: pgmap v22322: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 875 KiB/s rd, 874 KiB/s wr, 9 op/s 2023-03-10T02:19:49.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40899.083447 secs 2023-03-10T02:19:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:48 smithi181 ceph-mon[94189]: pgmap v22322: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 875 KiB/s rd, 874 KiB/s wr, 9 op/s 2023-03-10T02:19:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40899.083447 secs 2023-03-10T02:19:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:48 smithi038 ceph-mon[92422]: pgmap v22322: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 875 KiB/s rd, 874 KiB/s wr, 9 op/s 2023-03-10T02:19:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:19:50.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:19:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:19:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:19:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:19:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:19:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40901.570319 secs 2023-03-10T02:19:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:50 smithi122 ceph-mon[93869]: pgmap v22323: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 876 KiB/s rd, 874 KiB/s wr, 9 op/s 2023-03-10T02:19:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40901.570319 secs 2023-03-10T02:19:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:50 smithi181 ceph-mon[94189]: pgmap v22323: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 876 KiB/s rd, 874 KiB/s wr, 9 op/s 2023-03-10T02:19:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40901.570319 secs 2023-03-10T02:19:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:50 smithi038 ceph-mon[92422]: pgmap v22323: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 876 KiB/s rd, 874 KiB/s wr, 9 op/s 2023-03-10T02:19:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:19:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:19:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:52 smithi038 ceph-mon[92422]: pgmap v22324: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 933 KiB/s rd, 930 KiB/s wr, 11 op/s 2023-03-10T02:19:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:19:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:19:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:52 smithi122 ceph-mon[93869]: pgmap v22324: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 933 KiB/s rd, 930 KiB/s wr, 11 op/s 2023-03-10T02:19:53.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:19:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:19:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:52 smithi181 ceph-mon[94189]: pgmap v22324: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 933 KiB/s rd, 930 KiB/s wr, 11 op/s 2023-03-10T02:19:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41365.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:19:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:53 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T02:19:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:53 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T02:19:54.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:53 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T02:19:55.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:54 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:19:55.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:54 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:19:55.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:54 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:19:55.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:54 smithi038 ceph-mon[92422]: pgmap v22325: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 614 KiB/s rd, 612 KiB/s wr, 9 op/s 2023-03-10T02:19:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:54 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:19:55.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:54 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:19:55.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:54 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:19:55.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:54 smithi122 ceph-mon[93869]: pgmap v22325: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 614 KiB/s rd, 612 KiB/s wr, 9 op/s 2023-03-10T02:19:55.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:54 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:19:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:54 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:19:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:54 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:19:55.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:54 smithi181 ceph-mon[94189]: pgmap v22325: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 614 KiB/s rd, 612 KiB/s wr, 9 op/s 2023-03-10T02:19:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:19:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:19:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40906.570378 secs 2023-03-10T02:19:56.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:19:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:19:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40906.570378 secs 2023-03-10T02:19:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:19:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:19:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40906.570378 secs 2023-03-10T02:19:57.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:57 smithi122 ceph-mon[93869]: pgmap v22326: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 701 KiB/s rd, 686 KiB/s wr, 12 op/s 2023-03-10T02:19:57.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:57 smithi181 ceph-mon[94189]: pgmap v22326: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 701 KiB/s rd, 686 KiB/s wr, 12 op/s 2023-03-10T02:19:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:57 smithi038 ceph-mon[92422]: pgmap v22326: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 701 KiB/s rd, 686 KiB/s wr, 12 op/s 2023-03-10T02:19:58.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41370.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:19:59.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:19:59 smithi122 ceph-mon[93869]: pgmap v22327: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 632 KiB/s wr, 10 op/s 2023-03-10T02:19:59.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:19:59 smithi181 ceph-mon[94189]: pgmap v22327: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 632 KiB/s wr, 10 op/s 2023-03-10T02:19:59.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:19:59 smithi038 ceph-mon[92422]: pgmap v22327: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 632 KiB/s wr, 10 op/s 2023-03-10T02:20:00.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:20:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:20:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:00 smithi122 ceph-mon[93869]: overall HEALTH_WARN 1 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-10T02:20:00.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40911.570470 secs 2023-03-10T02:20:00.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:20:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:20:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:00 smithi181 ceph-mon[94189]: overall HEALTH_WARN 1 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-10T02:20:00.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40911.570470 secs 2023-03-10T02:20:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:20:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:20:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:00 smithi038 ceph-mon[92422]: overall HEALTH_WARN 1 MDSs report slow requests; 2 MDSs behind on trimming 2023-03-10T02:20:00.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40911.570470 secs 2023-03-10T02:20:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:01 smithi122 ceph-mon[93869]: pgmap v22328: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 667 KiB/s wr, 11 op/s 2023-03-10T02:20:01.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:20:01.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:20:01.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:01 smithi181 ceph-mon[94189]: pgmap v22328: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 667 KiB/s wr, 11 op/s 2023-03-10T02:20:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:20:01.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:20:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:01 smithi038 ceph-mon[92422]: pgmap v22328: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 667 KiB/s wr, 11 op/s 2023-03-10T02:20:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:20:01.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:20:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:03 smithi122 ceph-mon[93869]: pgmap v22329: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 816 KiB/s rd, 1015 KiB/s wr, 14 op/s 2023-03-10T02:20:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:03 smithi181 ceph-mon[94189]: pgmap v22329: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 816 KiB/s rd, 1015 KiB/s wr, 14 op/s 2023-03-10T02:20:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:03 smithi038 ceph-mon[92422]: pgmap v22329: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 816 KiB/s rd, 1015 KiB/s wr, 14 op/s 2023-03-10T02:20:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41375.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:20:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:05 smithi122 ceph-mon[93869]: pgmap v22330: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1014 KiB/s rd, 876 KiB/s wr, 23 op/s 2023-03-10T02:20:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40916.570527 secs 2023-03-10T02:20:05.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:05 smithi181 ceph-mon[94189]: pgmap v22330: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1014 KiB/s rd, 876 KiB/s wr, 23 op/s 2023-03-10T02:20:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40916.570527 secs 2023-03-10T02:20:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:05 smithi038 ceph-mon[92422]: pgmap v22330: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1014 KiB/s rd, 876 KiB/s wr, 23 op/s 2023-03-10T02:20:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40916.570527 secs 2023-03-10T02:20:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:07 smithi122 ceph-mon[93869]: pgmap v22331: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 28 op/s 2023-03-10T02:20:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:07 smithi181 ceph-mon[94189]: pgmap v22331: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 28 op/s 2023-03-10T02:20:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:07 smithi038 ceph-mon[92422]: pgmap v22331: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.0 MiB/s wr, 28 op/s 2023-03-10T02:20:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41380.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:20:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:08 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:20:08.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:08 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:20:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:08 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:20:08.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:08 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:20:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:08 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:20:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:08 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:20:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:09 smithi122 ceph-mon[93869]: pgmap v22332: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 958 KiB/s wr, 25 op/s 2023-03-10T02:20:09.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:20:09.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:20:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:09 smithi181 ceph-mon[94189]: pgmap v22332: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 958 KiB/s wr, 25 op/s 2023-03-10T02:20:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:20:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:20:10.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:09 smithi038 ceph-mon[92422]: pgmap v22332: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 958 KiB/s wr, 25 op/s 2023-03-10T02:20:10.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:20:10.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:20:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40921.570583 secs 2023-03-10T02:20:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40921.570583 secs 2023-03-10T02:20:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40921.570583 secs 2023-03-10T02:20:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:11 smithi122 ceph-mon[93869]: pgmap v22333: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 617 KiB/s wr, 25 op/s 2023-03-10T02:20:11.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:11 smithi181 ceph-mon[94189]: pgmap v22333: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 617 KiB/s wr, 25 op/s 2023-03-10T02:20:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:11 smithi038 ceph-mon[92422]: pgmap v22333: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 617 KiB/s wr, 25 op/s 2023-03-10T02:20:13.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41385.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:20:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:13 smithi122 ceph-mon[93869]: pgmap v22334: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 785 KiB/s wr, 28 op/s 2023-03-10T02:20:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:13 smithi181 ceph-mon[94189]: pgmap v22334: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 785 KiB/s wr, 28 op/s 2023-03-10T02:20:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:13 smithi038 ceph-mon[92422]: pgmap v22334: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 785 KiB/s wr, 28 op/s 2023-03-10T02:20:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:15 smithi122 ceph-mon[93869]: pgmap v22335: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 854 KiB/s rd, 437 KiB/s wr, 24 op/s 2023-03-10T02:20:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40926.570643 secs 2023-03-10T02:20:15.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:15 smithi181 ceph-mon[94189]: pgmap v22335: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 854 KiB/s rd, 437 KiB/s wr, 24 op/s 2023-03-10T02:20:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40926.570643 secs 2023-03-10T02:20:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:15 smithi038 ceph-mon[92422]: pgmap v22335: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 854 KiB/s rd, 437 KiB/s wr, 24 op/s 2023-03-10T02:20:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40926.570643 secs 2023-03-10T02:20:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:17 smithi122 ceph-mon[93869]: pgmap v22336: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 397 KiB/s wr, 15 op/s 2023-03-10T02:20:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:17 smithi181 ceph-mon[94189]: pgmap v22336: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 397 KiB/s wr, 15 op/s 2023-03-10T02:20:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:17 smithi038 ceph-mon[92422]: pgmap v22336: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 477 KiB/s rd, 397 KiB/s wr, 15 op/s 2023-03-10T02:20:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41390.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:20:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:19 smithi122 ceph-mon[93869]: pgmap v22337: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 215 KiB/s wr, 11 op/s 2023-03-10T02:20:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:19 smithi181 ceph-mon[94189]: pgmap v22337: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 215 KiB/s wr, 11 op/s 2023-03-10T02:20:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:19 smithi038 ceph-mon[92422]: pgmap v22337: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 215 KiB/s wr, 11 op/s 2023-03-10T02:20:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:20:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:20:20.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40931.570694 secs 2023-03-10T02:20:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:20:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:20:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40931.570694 secs 2023-03-10T02:20:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:20:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:20:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40931.570694 secs 2023-03-10T02:20:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:21 smithi122 ceph-mon[93869]: pgmap v22338: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 215 KiB/s wr, 11 op/s 2023-03-10T02:20:21.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:20:21.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:20:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:21 smithi181 ceph-mon[94189]: pgmap v22338: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 215 KiB/s wr, 11 op/s 2023-03-10T02:20:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:20:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:20:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:21 smithi038 ceph-mon[92422]: pgmap v22338: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 215 KiB/s wr, 11 op/s 2023-03-10T02:20:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:20:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:20:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41395.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:20:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:23 smithi122 ceph-mon[93869]: pgmap v22339: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 611 KiB/s rd, 629 KiB/s wr, 16 op/s 2023-03-10T02:20:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:23 smithi181 ceph-mon[94189]: pgmap v22339: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 611 KiB/s rd, 629 KiB/s wr, 16 op/s 2023-03-10T02:20:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:23 smithi038 ceph-mon[92422]: pgmap v22339: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 611 KiB/s rd, 629 KiB/s wr, 16 op/s 2023-03-10T02:20:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:25 smithi122 ceph-mon[93869]: pgmap v22340: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 406 KiB/s rd, 425 KiB/s wr, 12 op/s 2023-03-10T02:20:25.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:25 smithi122 ceph-mon[93869]: osdmap e1611: 12 total, 12 up, 12 in 2023-03-10T02:20:25.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40936.570751 secs 2023-03-10T02:20:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:25 smithi181 ceph-mon[94189]: pgmap v22340: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 406 KiB/s rd, 425 KiB/s wr, 12 op/s 2023-03-10T02:20:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:25 smithi181 ceph-mon[94189]: osdmap e1611: 12 total, 12 up, 12 in 2023-03-10T02:20:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40936.570751 secs 2023-03-10T02:20:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:25 smithi038 ceph-mon[92422]: pgmap v22340: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 406 KiB/s rd, 425 KiB/s wr, 12 op/s 2023-03-10T02:20:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:25 smithi038 ceph-mon[92422]: osdmap e1611: 12 total, 12 up, 12 in 2023-03-10T02:20:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40936.570751 secs 2023-03-10T02:20:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:27 smithi122 ceph-mon[93869]: pgmap v22342: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 699 KiB/s rd, 723 KiB/s wr, 14 op/s 2023-03-10T02:20:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:27 smithi181 ceph-mon[94189]: pgmap v22342: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 699 KiB/s rd, 723 KiB/s wr, 14 op/s 2023-03-10T02:20:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:27 smithi038 ceph-mon[92422]: pgmap v22342: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 699 KiB/s rd, 723 KiB/s wr, 14 op/s 2023-03-10T02:20:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41400.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:20:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:29 smithi122 ceph-mon[93869]: pgmap v22343: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 699 KiB/s rd, 710 KiB/s wr, 12 op/s 2023-03-10T02:20:29.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:29 smithi181 ceph-mon[94189]: pgmap v22343: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 699 KiB/s rd, 710 KiB/s wr, 12 op/s 2023-03-10T02:20:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:29 smithi038 ceph-mon[92422]: pgmap v22343: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 699 KiB/s rd, 710 KiB/s wr, 12 op/s 2023-03-10T02:20:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40941.570806 secs 2023-03-10T02:20:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40941.570806 secs 2023-03-10T02:20:30.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40941.570806 secs 2023-03-10T02:20:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:31 smithi122 ceph-mon[93869]: pgmap v22344: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 699 KiB/s rd, 711 KiB/s wr, 11 op/s 2023-03-10T02:20:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:31 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:20:31.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:31 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:20:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:31 smithi181 ceph-mon[94189]: pgmap v22344: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 699 KiB/s rd, 711 KiB/s wr, 11 op/s 2023-03-10T02:20:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:31 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:20:31.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:31 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:20:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:31 smithi038 ceph-mon[92422]: pgmap v22344: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 699 KiB/s rd, 711 KiB/s wr, 11 op/s 2023-03-10T02:20:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:31 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:20:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:31 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:20:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41405.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:20:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:33 smithi122 ceph-mon[93869]: pgmap v22345: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 372 KiB/s wr, 9 op/s 2023-03-10T02:20:33.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:33 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:20:33.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:33 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:20:33.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:33 smithi181 ceph-mon[94189]: pgmap v22345: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 372 KiB/s wr, 9 op/s 2023-03-10T02:20:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:33 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:20:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:33 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:20:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:33 smithi038 ceph-mon[92422]: pgmap v22345: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 489 KiB/s rd, 372 KiB/s wr, 9 op/s 2023-03-10T02:20:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:33 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:20:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:33 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:20:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:35 smithi122 ceph-mon[93869]: pgmap v22346: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 898 KiB/s rd, 782 KiB/s wr, 9 op/s 2023-03-10T02:20:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40946.570890 secs 2023-03-10T02:20:35.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:35 smithi122 ceph-mon[93869]: osdmap e1612: 12 total, 12 up, 12 in 2023-03-10T02:20:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:35 smithi181 ceph-mon[94189]: pgmap v22346: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 898 KiB/s rd, 782 KiB/s wr, 9 op/s 2023-03-10T02:20:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40946.570890 secs 2023-03-10T02:20:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:35 smithi181 ceph-mon[94189]: osdmap e1612: 12 total, 12 up, 12 in 2023-03-10T02:20:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:35 smithi038 ceph-mon[92422]: pgmap v22346: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 898 KiB/s rd, 782 KiB/s wr, 9 op/s 2023-03-10T02:20:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40946.570890 secs 2023-03-10T02:20:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:35 smithi038 ceph-mon[92422]: osdmap e1612: 12 total, 12 up, 12 in 2023-03-10T02:20:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:37 smithi038 ceph-mon[92422]: pgmap v22348: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 733 KiB/s wr, 10 op/s 2023-03-10T02:20:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:20:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:20:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:37 smithi122 ceph-mon[93869]: pgmap v22348: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 733 KiB/s wr, 10 op/s 2023-03-10T02:20:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:20:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:20:38.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:37 smithi181 ceph-mon[94189]: pgmap v22348: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 733 KiB/s wr, 10 op/s 2023-03-10T02:20:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:20:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:20:38.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41410.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:20:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40949.084307 secs 2023-03-10T02:20:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:38 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:20:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:38 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:20:39.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:38 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:20:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40949.084307 secs 2023-03-10T02:20:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:38 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:20:39.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:38 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:20:39.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:38 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:20:39.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40949.084307 secs 2023-03-10T02:20:39.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:38 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:20:39.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:38 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:20:39.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:38 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:20:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:39 smithi038 ceph-mon[92422]: pgmap v22349: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 734 KiB/s wr, 10 op/s 2023-03-10T02:20:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:39 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:20:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:39 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:20:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:39 smithi122 ceph-mon[93869]: pgmap v22349: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 734 KiB/s wr, 10 op/s 2023-03-10T02:20:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:39 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:20:40.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:39 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:20:40.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:39 smithi181 ceph-mon[94189]: pgmap v22349: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 734 KiB/s wr, 10 op/s 2023-03-10T02:20:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:39 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:20:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:39 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:20:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40951.570948 secs 2023-03-10T02:20:41.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40951.570948 secs 2023-03-10T02:20:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40951.570948 secs 2023-03-10T02:20:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:41 smithi122 ceph-mon[93869]: pgmap v22350: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.1 MiB/s wr, 12 op/s 2023-03-10T02:20:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:20:42.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:20:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:41 smithi181 ceph-mon[94189]: pgmap v22350: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.1 MiB/s wr, 12 op/s 2023-03-10T02:20:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:20:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:20:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:41 smithi038 ceph-mon[92422]: pgmap v22350: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.1 MiB/s wr, 12 op/s 2023-03-10T02:20:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:20:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:20:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41415.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:20:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:43 smithi122 ceph-mon[93869]: pgmap v22351: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1000 KiB/s wr, 11 op/s 2023-03-10T02:20:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40954.084390 secs 2023-03-10T02:20:44.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:43 smithi181 ceph-mon[94189]: pgmap v22351: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1000 KiB/s wr, 11 op/s 2023-03-10T02:20:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40954.084390 secs 2023-03-10T02:20:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:43 smithi038 ceph-mon[92422]: pgmap v22351: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1000 KiB/s wr, 11 op/s 2023-03-10T02:20:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40954.084390 secs 2023-03-10T02:20:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:45 smithi122 ceph-mon[93869]: pgmap v22352: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 939 KiB/s rd, 591 KiB/s wr, 11 op/s 2023-03-10T02:20:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40956.571009 secs 2023-03-10T02:20:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:45 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:20:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:45 smithi181 ceph-mon[94189]: pgmap v22352: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 939 KiB/s rd, 591 KiB/s wr, 11 op/s 2023-03-10T02:20:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40956.571009 secs 2023-03-10T02:20:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:45 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:20:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:45 smithi038 ceph-mon[92422]: pgmap v22352: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 939 KiB/s rd, 591 KiB/s wr, 11 op/s 2023-03-10T02:20:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40956.571009 secs 2023-03-10T02:20:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:45 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:20:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:20:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:20:47.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:20:47.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:20:47.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:20:47.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:20:47.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:20:47.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:20:47.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:20:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:47 smithi122 ceph-mon[93869]: pgmap v22353: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1020 KiB/s rd, 694 KiB/s wr, 14 op/s 2023-03-10T02:20:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:20:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:20:48.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:47 smithi181 ceph-mon[94189]: pgmap v22353: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1020 KiB/s rd, 694 KiB/s wr, 14 op/s 2023-03-10T02:20:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:20:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:20:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:47 smithi038 ceph-mon[92422]: pgmap v22353: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1020 KiB/s rd, 694 KiB/s wr, 14 op/s 2023-03-10T02:20:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:20:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:20:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41420.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:20:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:49 smithi122 ceph-mon[93869]: pgmap v22354: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 585 KiB/s rd, 583 KiB/s wr, 9 op/s 2023-03-10T02:20:50.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:20:50.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:20:50.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:49 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T02:20:50.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:49 smithi181 ceph-mon[94189]: pgmap v22354: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 585 KiB/s rd, 583 KiB/s wr, 9 op/s 2023-03-10T02:20:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:20:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:20:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:49 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T02:20:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:49 smithi038 ceph-mon[92422]: pgmap v22354: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 585 KiB/s rd, 583 KiB/s wr, 9 op/s 2023-03-10T02:20:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:20:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:20:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:49 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T02:20:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:50 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:20:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:50 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:20:51.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:50 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:20:51.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40961.571064 secs 2023-03-10T02:20:51.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:50 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:20:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:50 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:20:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:50 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:20:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40961.571064 secs 2023-03-10T02:20:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:50 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:20:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:50 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:20:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:50 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:20:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40961.571064 secs 2023-03-10T02:20:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:51 smithi122 ceph-mon[93869]: pgmap v22355: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 754 KiB/s rd, 752 KiB/s wr, 10 op/s 2023-03-10T02:20:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:20:52.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:20:52.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:51 smithi181 ceph-mon[94189]: pgmap v22355: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 754 KiB/s rd, 752 KiB/s wr, 10 op/s 2023-03-10T02:20:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:20:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:20:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:51 smithi038 ceph-mon[92422]: pgmap v22355: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 754 KiB/s rd, 752 KiB/s wr, 10 op/s 2023-03-10T02:20:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:20:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:20:53.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:52 smithi122 ceph-mon[93869]: pgmap v22356: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 445 KiB/s rd, 443 KiB/s wr, 12 op/s 2023-03-10T02:20:53.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:52 smithi181 ceph-mon[94189]: pgmap v22356: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 445 KiB/s rd, 443 KiB/s wr, 12 op/s 2023-03-10T02:20:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:52 smithi038 ceph-mon[92422]: pgmap v22356: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 445 KiB/s rd, 443 KiB/s wr, 12 op/s 2023-03-10T02:20:53.801 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41425.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:20:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:53 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 40964.084556 secs 2023-03-10T02:20:54.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:20:54.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:20:54.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:53 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 40964.084556 secs 2023-03-10T02:20:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:20:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:20:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:53 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 40964.084556 secs 2023-03-10T02:20:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:20:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:20:55.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:54 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:20:55.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:54 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:20:55.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:54 smithi122 ceph-mon[93869]: pgmap v22357: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 615 KiB/s rd, 614 KiB/s wr, 10 op/s 2023-03-10T02:20:55.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:54 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:20:55.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:54 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:20:55.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:54 smithi181 ceph-mon[94189]: pgmap v22357: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 615 KiB/s rd, 614 KiB/s wr, 10 op/s 2023-03-10T02:20:55.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:54 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:20:55.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:54 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:20:55.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:54 smithi038 ceph-mon[92422]: pgmap v22357: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 615 KiB/s rd, 614 KiB/s wr, 10 op/s 2023-03-10T02:20:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:20:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:20:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40966.571123 secs 2023-03-10T02:20:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:55 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:20:56.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:20:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:20:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40966.571123 secs 2023-03-10T02:20:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:55 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:20:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:20:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:20:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40966.571123 secs 2023-03-10T02:20:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:55 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:20:57.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:56 smithi122 ceph-mon[93869]: pgmap v22358: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 654 KiB/s rd, 649 KiB/s wr, 12 op/s 2023-03-10T02:20:57.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:56 smithi181 ceph-mon[94189]: pgmap v22358: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 654 KiB/s rd, 649 KiB/s wr, 12 op/s 2023-03-10T02:20:57.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:56 smithi038 ceph-mon[92422]: pgmap v22358: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 654 KiB/s rd, 649 KiB/s wr, 12 op/s 2023-03-10T02:20:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:20:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:20:58.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:20:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:20:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:20:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:20:58.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41430.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:20:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40969.084616 secs 2023-03-10T02:20:59.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:58 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:20:59.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:58 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:20:59.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:58 smithi122 ceph-mon[93869]: pgmap v22359: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 490 KiB/s rd, 487 KiB/s wr, 10 op/s 2023-03-10T02:20:59.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40969.084616 secs 2023-03-10T02:20:59.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:58 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:20:59.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:58 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:20:59.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:58 smithi181 ceph-mon[94189]: pgmap v22359: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 490 KiB/s rd, 487 KiB/s wr, 10 op/s 2023-03-10T02:20:59.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40969.084616 secs 2023-03-10T02:20:59.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:58 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:20:59.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:58 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:20:59.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:58 smithi038 ceph-mon[92422]: pgmap v22359: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 490 KiB/s rd, 487 KiB/s wr, 10 op/s 2023-03-10T02:21:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:21:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:20:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:21:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:21:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:20:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:21:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:21:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:20:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:21:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40971.571181 secs 2023-03-10T02:21:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:00 smithi122 ceph-mon[93869]: pgmap v22360: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 486 KiB/s wr, 11 op/s 2023-03-10T02:21:01.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40971.571181 secs 2023-03-10T02:21:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:00 smithi181 ceph-mon[94189]: pgmap v22360: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 486 KiB/s wr, 11 op/s 2023-03-10T02:21:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40971.571181 secs 2023-03-10T02:21:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:00 smithi038 ceph-mon[92422]: pgmap v22360: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 501 KiB/s rd, 486 KiB/s wr, 11 op/s 2023-03-10T02:21:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:21:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:21:02.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:21:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:21:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:21:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:21:03.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:02 smithi122 ceph-mon[93869]: pgmap v22361: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 985 KiB/s rd, 964 KiB/s wr, 14 op/s 2023-03-10T02:21:03.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:02 smithi181 ceph-mon[94189]: pgmap v22361: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 985 KiB/s rd, 964 KiB/s wr, 14 op/s 2023-03-10T02:21:03.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:02 smithi038 ceph-mon[92422]: pgmap v22361: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 985 KiB/s rd, 964 KiB/s wr, 14 op/s 2023-03-10T02:21:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41435.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:21:04.231 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40974.084715 secs 2023-03-10T02:21:04.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40974.084715 secs 2023-03-10T02:21:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40974.084715 secs 2023-03-10T02:21:05.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:04 smithi122 ceph-mon[93869]: pgmap v22362: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 946 KiB/s rd, 930 KiB/s wr, 22 op/s 2023-03-10T02:21:05.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:04 smithi181 ceph-mon[94189]: pgmap v22362: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 946 KiB/s rd, 930 KiB/s wr, 22 op/s 2023-03-10T02:21:05.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:04 smithi038 ceph-mon[92422]: pgmap v22362: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 946 KiB/s rd, 930 KiB/s wr, 22 op/s 2023-03-10T02:21:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40976.571238 secs 2023-03-10T02:21:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40976.571238 secs 2023-03-10T02:21:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40976.571238 secs 2023-03-10T02:21:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:06 smithi122 ceph-mon[93869]: pgmap v22363: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 26 op/s 2023-03-10T02:21:07.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:06 smithi181 ceph-mon[94189]: pgmap v22363: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 26 op/s 2023-03-10T02:21:07.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:06 smithi038 ceph-mon[92422]: pgmap v22363: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 26 op/s 2023-03-10T02:21:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:21:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:21:08.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:21:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:21:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:21:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:21:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41440.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:21:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40979.084816 secs 2023-03-10T02:21:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:08 smithi122 ceph-mon[93869]: pgmap v22364: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1005 KiB/s rd, 1001 KiB/s wr, 24 op/s 2023-03-10T02:21:09.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40979.084816 secs 2023-03-10T02:21:09.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:08 smithi181 ceph-mon[94189]: pgmap v22364: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1005 KiB/s rd, 1001 KiB/s wr, 24 op/s 2023-03-10T02:21:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40979.084816 secs 2023-03-10T02:21:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:08 smithi038 ceph-mon[92422]: pgmap v22364: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1005 KiB/s rd, 1001 KiB/s wr, 24 op/s 2023-03-10T02:21:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:21:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:21:10.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:21:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:21:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:21:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:21:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40981.571322 secs 2023-03-10T02:21:11.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:10 smithi122 ceph-mon[93869]: pgmap v22365: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 940 KiB/s rd, 936 KiB/s wr, 22 op/s 2023-03-10T02:21:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40981.571322 secs 2023-03-10T02:21:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:10 smithi181 ceph-mon[94189]: pgmap v22365: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 940 KiB/s rd, 936 KiB/s wr, 22 op/s 2023-03-10T02:21:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40981.571322 secs 2023-03-10T02:21:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:10 smithi038 ceph-mon[92422]: pgmap v22365: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 940 KiB/s rd, 936 KiB/s wr, 22 op/s 2023-03-10T02:21:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:13 smithi122 ceph-mon[93869]: pgmap v22366: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 25 op/s 2023-03-10T02:21:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:13 smithi181 ceph-mon[94189]: pgmap v22366: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 25 op/s 2023-03-10T02:21:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:13 smithi038 ceph-mon[92422]: pgmap v22366: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 25 op/s 2023-03-10T02:21:13.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41445.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:21:14.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:14 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40984.084927 secs 2023-03-10T02:21:14.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:14 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40984.084927 secs 2023-03-10T02:21:14.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:14 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40984.084927 secs 2023-03-10T02:21:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:15 smithi122 ceph-mon[93869]: pgmap v22367: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 481 KiB/s rd, 494 KiB/s wr, 20 op/s 2023-03-10T02:21:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40986.571380 secs 2023-03-10T02:21:15.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:15 smithi181 ceph-mon[94189]: pgmap v22367: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 481 KiB/s rd, 494 KiB/s wr, 20 op/s 2023-03-10T02:21:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40986.571380 secs 2023-03-10T02:21:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:15 smithi038 ceph-mon[92422]: pgmap v22367: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 481 KiB/s rd, 494 KiB/s wr, 20 op/s 2023-03-10T02:21:15.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40986.571380 secs 2023-03-10T02:21:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:21:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:21:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:21:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:21:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:21:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:21:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:17 smithi122 ceph-mon[93869]: pgmap v22368: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 476 KiB/s rd, 483 KiB/s wr, 12 op/s 2023-03-10T02:21:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:21:17.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:21:17.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:17 smithi181 ceph-mon[94189]: pgmap v22368: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 476 KiB/s rd, 483 KiB/s wr, 12 op/s 2023-03-10T02:21:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:21:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:21:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:17 smithi038 ceph-mon[92422]: pgmap v22368: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 476 KiB/s rd, 483 KiB/s wr, 12 op/s 2023-03-10T02:21:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:21:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:21:18.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41450.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:21:18.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40989.084987 secs 2023-03-10T02:21:18.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40989.084987 secs 2023-03-10T02:21:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40989.084987 secs 2023-03-10T02:21:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:19 smithi122 ceph-mon[93869]: pgmap v22369: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 225 KiB/s wr, 8 op/s 2023-03-10T02:21:19.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:19 smithi181 ceph-mon[94189]: pgmap v22369: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 225 KiB/s wr, 8 op/s 2023-03-10T02:21:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:19 smithi038 ceph-mon[92422]: pgmap v22369: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 225 KiB/s wr, 8 op/s 2023-03-10T02:21:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:21:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:21:20.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40991.571436 secs 2023-03-10T02:21:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:21:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:21:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40991.571436 secs 2023-03-10T02:21:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:21:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:21:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40991.571436 secs 2023-03-10T02:21:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:21 smithi122 ceph-mon[93869]: pgmap v22370: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 225 KiB/s wr, 8 op/s 2023-03-10T02:21:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:21:21.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:21:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:21 smithi181 ceph-mon[94189]: pgmap v22370: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 225 KiB/s wr, 8 op/s 2023-03-10T02:21:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:21:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:21:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:21 smithi038 ceph-mon[92422]: pgmap v22370: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 225 KiB/s wr, 8 op/s 2023-03-10T02:21:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:21:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:21:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41455.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:21:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:23 smithi122 ceph-mon[93869]: pgmap v22371: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 466 KiB/s rd, 484 KiB/s wr, 12 op/s 2023-03-10T02:21:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40994.085052 secs 2023-03-10T02:21:23.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:23 smithi181 ceph-mon[94189]: pgmap v22371: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 466 KiB/s rd, 484 KiB/s wr, 12 op/s 2023-03-10T02:21:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40994.085052 secs 2023-03-10T02:21:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:23 smithi038 ceph-mon[92422]: pgmap v22371: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 466 KiB/s rd, 484 KiB/s wr, 12 op/s 2023-03-10T02:21:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40994.085052 secs 2023-03-10T02:21:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:25 smithi122 ceph-mon[93869]: pgmap v22372: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 426 KiB/s wr, 10 op/s 2023-03-10T02:21:25.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:25 smithi122 ceph-mon[93869]: osdmap e1613: 12 total, 12 up, 12 in 2023-03-10T02:21:25.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40996.571525 secs 2023-03-10T02:21:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:25 smithi181 ceph-mon[94189]: pgmap v22372: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 426 KiB/s wr, 10 op/s 2023-03-10T02:21:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:25 smithi181 ceph-mon[94189]: osdmap e1613: 12 total, 12 up, 12 in 2023-03-10T02:21:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40996.571525 secs 2023-03-10T02:21:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:25 smithi038 ceph-mon[92422]: pgmap v22372: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 426 KiB/s wr, 10 op/s 2023-03-10T02:21:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:25 smithi038 ceph-mon[92422]: osdmap e1613: 12 total, 12 up, 12 in 2023-03-10T02:21:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40996.571525 secs 2023-03-10T02:21:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:27 smithi122 ceph-mon[93869]: pgmap v22374: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 532 KiB/s rd, 555 KiB/s wr, 11 op/s 2023-03-10T02:21:27.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:27 smithi181 ceph-mon[94189]: pgmap v22374: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 532 KiB/s rd, 555 KiB/s wr, 11 op/s 2023-03-10T02:21:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:27 smithi038 ceph-mon[92422]: pgmap v22374: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 532 KiB/s rd, 555 KiB/s wr, 11 op/s 2023-03-10T02:21:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41460.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:21:28.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 40999.085116 secs 2023-03-10T02:21:28.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 40999.085116 secs 2023-03-10T02:21:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 40999.085116 secs 2023-03-10T02:21:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:29 smithi122 ceph-mon[93869]: pgmap v22375: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 733 KiB/s rd, 744 KiB/s wr, 10 op/s 2023-03-10T02:21:29.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:29 smithi181 ceph-mon[94189]: pgmap v22375: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 733 KiB/s rd, 744 KiB/s wr, 10 op/s 2023-03-10T02:21:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:29 smithi038 ceph-mon[92422]: pgmap v22375: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 733 KiB/s rd, 744 KiB/s wr, 10 op/s 2023-03-10T02:21:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41001.571582 secs 2023-03-10T02:21:30.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41001.571582 secs 2023-03-10T02:21:30.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41001.571582 secs 2023-03-10T02:21:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:31 smithi122 ceph-mon[93869]: pgmap v22376: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 733 KiB/s rd, 744 KiB/s wr, 10 op/s 2023-03-10T02:21:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:31 smithi181 ceph-mon[94189]: pgmap v22376: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 733 KiB/s rd, 744 KiB/s wr, 10 op/s 2023-03-10T02:21:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:31 smithi038 ceph-mon[92422]: pgmap v22376: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 733 KiB/s rd, 744 KiB/s wr, 10 op/s 2023-03-10T02:21:33.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41465.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:21:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:33 smithi122 ceph-mon[93869]: pgmap v22377: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 436 KiB/s rd, 434 KiB/s wr, 8 op/s 2023-03-10T02:21:33.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41004.085218 secs 2023-03-10T02:21:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:33 smithi181 ceph-mon[94189]: pgmap v22377: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 436 KiB/s rd, 434 KiB/s wr, 8 op/s 2023-03-10T02:21:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41004.085218 secs 2023-03-10T02:21:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:33 smithi038 ceph-mon[92422]: pgmap v22377: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 436 KiB/s rd, 434 KiB/s wr, 8 op/s 2023-03-10T02:21:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41004.085218 secs 2023-03-10T02:21:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:35 smithi122 ceph-mon[93869]: pgmap v22378: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 267 KiB/s wr, 8 op/s 2023-03-10T02:21:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41006.571642 secs 2023-03-10T02:21:35.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:35 smithi122 ceph-mon[93869]: osdmap e1614: 12 total, 12 up, 12 in 2023-03-10T02:21:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:35 smithi181 ceph-mon[94189]: pgmap v22378: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 267 KiB/s wr, 8 op/s 2023-03-10T02:21:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41006.571642 secs 2023-03-10T02:21:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:35 smithi181 ceph-mon[94189]: osdmap e1614: 12 total, 12 up, 12 in 2023-03-10T02:21:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:35 smithi038 ceph-mon[92422]: pgmap v22378: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 267 KiB/s wr, 8 op/s 2023-03-10T02:21:36.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41006.571642 secs 2023-03-10T02:21:36.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:35 smithi038 ceph-mon[92422]: osdmap e1614: 12 total, 12 up, 12 in 2023-03-10T02:21:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:37 smithi122 ceph-mon[93869]: pgmap v22380: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 479 KiB/s wr, 9 op/s 2023-03-10T02:21:37.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:37 smithi181 ceph-mon[94189]: pgmap v22380: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 479 KiB/s wr, 9 op/s 2023-03-10T02:21:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:37 smithi038 ceph-mon[92422]: pgmap v22380: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 480 KiB/s rd, 479 KiB/s wr, 9 op/s 2023-03-10T02:21:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41470.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:21:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41009.085299 secs 2023-03-10T02:21:38.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41009.085299 secs 2023-03-10T02:21:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41009.085299 secs 2023-03-10T02:21:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:39 smithi122 ceph-mon[93869]: pgmap v22381: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 266 KiB/s wr, 8 op/s 2023-03-10T02:21:39.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:39 smithi181 ceph-mon[94189]: pgmap v22381: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 266 KiB/s wr, 8 op/s 2023-03-10T02:21:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:39 smithi038 ceph-mon[92422]: pgmap v22381: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 266 KiB/s wr, 8 op/s 2023-03-10T02:21:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41011.571725 secs 2023-03-10T02:21:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41011.571725 secs 2023-03-10T02:21:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41011.571725 secs 2023-03-10T02:21:41.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:41 smithi122 ceph-mon[93869]: pgmap v22382: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 266 KiB/s wr, 8 op/s 2023-03-10T02:21:41.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:21:41.979 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:21:41.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:41 smithi181 ceph-mon[94189]: pgmap v22382: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 266 KiB/s wr, 8 op/s 2023-03-10T02:21:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:21:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:21:42.009 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:41 smithi038 ceph-mon[92422]: pgmap v22382: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 266 KiB/s wr, 8 op/s 2023-03-10T02:21:42.010 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:21:42.010 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:21:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41475.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:21:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:43 smithi122 ceph-mon[93869]: pgmap v22383: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 466 KiB/s rd, 464 KiB/s wr, 9 op/s 2023-03-10T02:21:43.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41014.085398 secs 2023-03-10T02:21:43.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:43 smithi181 ceph-mon[94189]: pgmap v22383: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 466 KiB/s rd, 464 KiB/s wr, 9 op/s 2023-03-10T02:21:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41014.085398 secs 2023-03-10T02:21:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:43 smithi038 ceph-mon[92422]: pgmap v22383: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 466 KiB/s rd, 464 KiB/s wr, 9 op/s 2023-03-10T02:21:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41014.085398 secs 2023-03-10T02:21:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:45 smithi122 ceph-mon[93869]: pgmap v22384: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 810 KiB/s rd, 864 KiB/s wr, 10 op/s 2023-03-10T02:21:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41016.571832 secs 2023-03-10T02:21:45.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:45 smithi181 ceph-mon[94189]: pgmap v22384: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 810 KiB/s rd, 864 KiB/s wr, 10 op/s 2023-03-10T02:21:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41016.571832 secs 2023-03-10T02:21:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:45 smithi038 ceph-mon[92422]: pgmap v22384: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 810 KiB/s rd, 864 KiB/s wr, 10 op/s 2023-03-10T02:21:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41016.571832 secs 2023-03-10T02:21:46.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:46 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:21:46.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:46 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:21:46.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:46 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:21:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:47 smithi122 ceph-mon[93869]: pgmap v22385: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 907 KiB/s rd, 956 KiB/s wr, 12 op/s 2023-03-10T02:21:47.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:21:47.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:21:47.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:21:47.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:21:47.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:21:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:47 smithi181 ceph-mon[94189]: pgmap v22385: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 907 KiB/s rd, 956 KiB/s wr, 12 op/s 2023-03-10T02:21:47.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:21:47.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:21:47.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:21:47.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:21:47.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:21:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:47 smithi038 ceph-mon[92422]: pgmap v22385: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 907 KiB/s rd, 956 KiB/s wr, 12 op/s 2023-03-10T02:21:48.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:21:48.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:21:48.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:21:48.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:21:48.001 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:21:48.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41480.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:21:48.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41019.085456 secs 2023-03-10T02:21:48.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41019.085456 secs 2023-03-10T02:21:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41019.085456 secs 2023-03-10T02:21:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:49 smithi122 ceph-mon[93869]: pgmap v22386: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 595 KiB/s rd, 640 KiB/s wr, 8 op/s 2023-03-10T02:21:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:49 smithi181 ceph-mon[94189]: pgmap v22386: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 595 KiB/s rd, 640 KiB/s wr, 8 op/s 2023-03-10T02:21:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:49 smithi038 ceph-mon[92422]: pgmap v22386: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 595 KiB/s rd, 640 KiB/s wr, 8 op/s 2023-03-10T02:21:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41021.571888 secs 2023-03-10T02:21:50.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41021.571888 secs 2023-03-10T02:21:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41021.571888 secs 2023-03-10T02:21:51.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:51 smithi122 ceph-mon[93869]: pgmap v22387: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 642 KiB/s rd, 640 KiB/s wr, 8 op/s 2023-03-10T02:21:51.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:51 smithi181 ceph-mon[94189]: pgmap v22387: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 642 KiB/s rd, 640 KiB/s wr, 8 op/s 2023-03-10T02:21:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:51 smithi038 ceph-mon[92422]: pgmap v22387: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 642 KiB/s rd, 640 KiB/s wr, 8 op/s 2023-03-10T02:21:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41485.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:21:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:53 smithi038 ceph-mon[92422]: pgmap v22388: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 864 KiB/s rd, 861 KiB/s wr, 12 op/s 2023-03-10T02:21:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41024.085558 secs 2023-03-10T02:21:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:21:54.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:21:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:53 smithi122 ceph-mon[93869]: pgmap v22388: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 864 KiB/s rd, 861 KiB/s wr, 12 op/s 2023-03-10T02:21:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41024.085558 secs 2023-03-10T02:21:54.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:21:54.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:21:54.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:53 smithi181 ceph-mon[94189]: pgmap v22388: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 864 KiB/s rd, 861 KiB/s wr, 12 op/s 2023-03-10T02:21:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41024.085558 secs 2023-03-10T02:21:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:21:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:21:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:55 smithi038 ceph-mon[92422]: pgmap v22389: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 721 KiB/s rd, 720 KiB/s wr, 10 op/s 2023-03-10T02:21:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:21:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:21:56.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41026.571985 secs 2023-03-10T02:21:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:55 smithi122 ceph-mon[93869]: pgmap v22389: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 721 KiB/s rd, 720 KiB/s wr, 10 op/s 2023-03-10T02:21:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:21:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:21:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41026.571985 secs 2023-03-10T02:21:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:55 smithi181 ceph-mon[94189]: pgmap v22389: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 721 KiB/s rd, 720 KiB/s wr, 10 op/s 2023-03-10T02:21:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:21:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:21:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41026.571985 secs 2023-03-10T02:21:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:57 smithi038 ceph-mon[92422]: pgmap v22390: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 570 KiB/s rd, 378 KiB/s wr, 12 op/s 2023-03-10T02:21:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:57 smithi122 ceph-mon[93869]: pgmap v22390: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 570 KiB/s rd, 378 KiB/s wr, 12 op/s 2023-03-10T02:21:58.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:57 smithi181 ceph-mon[94189]: pgmap v22390: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 570 KiB/s rd, 378 KiB/s wr, 12 op/s 2023-03-10T02:21:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41490.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:21:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41029.085658 secs 2023-03-10T02:21:59.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41029.085658 secs 2023-03-10T02:21:59.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41029.085658 secs 2023-03-10T02:22:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:59 smithi122 ceph-mon[93869]: pgmap v22391: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 536 KiB/s rd, 347 KiB/s wr, 12 op/s 2023-03-10T02:22:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:22:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:21:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:22:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:59 smithi181 ceph-mon[94189]: pgmap v22391: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 536 KiB/s rd, 347 KiB/s wr, 12 op/s 2023-03-10T02:22:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:22:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:21:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:22:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:59 smithi038 ceph-mon[92422]: pgmap v22391: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 536 KiB/s rd, 347 KiB/s wr, 12 op/s 2023-03-10T02:22:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:22:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:21:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:22:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41031.572040 secs 2023-03-10T02:22:01.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41031.572040 secs 2023-03-10T02:22:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41031.572040 secs 2023-03-10T02:22:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:01 smithi122 ceph-mon[93869]: pgmap v22392: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 889 KiB/s rd, 688 KiB/s wr, 13 op/s 2023-03-10T02:22:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:22:02.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:22:02.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:01 smithi181 ceph-mon[94189]: pgmap v22392: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 889 KiB/s rd, 688 KiB/s wr, 13 op/s 2023-03-10T02:22:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:22:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:22:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:01 smithi038 ceph-mon[92422]: pgmap v22392: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 889 KiB/s rd, 688 KiB/s wr, 13 op/s 2023-03-10T02:22:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:22:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:22:03.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41495.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:22:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:03 smithi122 ceph-mon[93869]: pgmap v22393: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 18 op/s 2023-03-10T02:22:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:03 smithi181 ceph-mon[94189]: pgmap v22393: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 18 op/s 2023-03-10T02:22:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:03 smithi038 ceph-mon[92422]: pgmap v22393: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 18 op/s 2023-03-10T02:22:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:05 smithi122 ceph-mon[93869]: pgmap v22394: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 943 KiB/s rd, 909 KiB/s wr, 25 op/s 2023-03-10T02:22:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41036.572099 secs 2023-03-10T02:22:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:05 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T02:22:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:05 smithi181 ceph-mon[94189]: pgmap v22394: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 943 KiB/s rd, 909 KiB/s wr, 25 op/s 2023-03-10T02:22:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41036.572099 secs 2023-03-10T02:22:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:05 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T02:22:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:05 smithi038 ceph-mon[92422]: pgmap v22394: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 943 KiB/s rd, 909 KiB/s wr, 25 op/s 2023-03-10T02:22:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41036.572099 secs 2023-03-10T02:22:06.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:05 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T02:22:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:06 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:22:07.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:06 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:22:07.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:06 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:22:07.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:06 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:22:07.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:06 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:22:07.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:06 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:22:07.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:06 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:22:07.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:06 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:22:07.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:06 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:22:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:07 smithi122 ceph-mon[93869]: pgmap v22395: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 886 KiB/s wr, 28 op/s 2023-03-10T02:22:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:22:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:22:08.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:07 smithi181 ceph-mon[94189]: pgmap v22395: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 886 KiB/s wr, 28 op/s 2023-03-10T02:22:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:22:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:22:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:07 smithi038 ceph-mon[92422]: pgmap v22395: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 886 KiB/s wr, 28 op/s 2023-03-10T02:22:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:22:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:22:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41500.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:22:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:09 smithi122 ceph-mon[93869]: pgmap v22396: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 897 KiB/s rd, 886 KiB/s wr, 25 op/s 2023-03-10T02:22:10.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:22:10.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:22:10.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:09 smithi181 ceph-mon[94189]: pgmap v22396: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 897 KiB/s rd, 886 KiB/s wr, 25 op/s 2023-03-10T02:22:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:22:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:22:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:09 smithi038 ceph-mon[92422]: pgmap v22396: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 897 KiB/s rd, 886 KiB/s wr, 25 op/s 2023-03-10T02:22:10.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:22:10.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:22:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41041.572187 secs 2023-03-10T02:22:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41041.572187 secs 2023-03-10T02:22:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41041.572187 secs 2023-03-10T02:22:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:11 smithi122 ceph-mon[93869]: pgmap v22397: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 797 KiB/s rd, 785 KiB/s wr, 24 op/s 2023-03-10T02:22:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:11 smithi181 ceph-mon[94189]: pgmap v22397: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 797 KiB/s rd, 785 KiB/s wr, 24 op/s 2023-03-10T02:22:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:11 smithi038 ceph-mon[92422]: pgmap v22397: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 797 KiB/s rd, 785 KiB/s wr, 24 op/s 2023-03-10T02:22:13.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41505.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:22:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:13 smithi122 ceph-mon[93869]: pgmap v22398: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 650 KiB/s rd, 649 KiB/s wr, 26 op/s 2023-03-10T02:22:14.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:13 smithi181 ceph-mon[94189]: pgmap v22398: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 650 KiB/s rd, 649 KiB/s wr, 26 op/s 2023-03-10T02:22:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:13 smithi038 ceph-mon[92422]: pgmap v22398: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 650 KiB/s rd, 649 KiB/s wr, 26 op/s 2023-03-10T02:22:15.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:14 smithi122 ceph-mon[93869]: pgmap v22399: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 254 KiB/s wr, 21 op/s 2023-03-10T02:22:15.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:14 smithi181 ceph-mon[94189]: pgmap v22399: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 254 KiB/s wr, 21 op/s 2023-03-10T02:22:15.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:14 smithi038 ceph-mon[92422]: pgmap v22399: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 371 KiB/s rd, 254 KiB/s wr, 21 op/s 2023-03-10T02:22:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41046.572274 secs 2023-03-10T02:22:16.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41046.572274 secs 2023-03-10T02:22:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41046.572274 secs 2023-03-10T02:22:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:16 smithi122 ceph-mon[93869]: pgmap v22400: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 329 KiB/s rd, 207 KiB/s wr, 11 op/s 2023-03-10T02:22:17.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:16 smithi181 ceph-mon[94189]: pgmap v22400: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 329 KiB/s rd, 207 KiB/s wr, 11 op/s 2023-03-10T02:22:17.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:16 smithi038 ceph-mon[92422]: pgmap v22400: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 329 KiB/s rd, 207 KiB/s wr, 11 op/s 2023-03-10T02:22:18.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41510.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:22:19.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:19 smithi122 ceph-mon[93869]: pgmap v22401: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 253 KiB/s wr, 9 op/s 2023-03-10T02:22:19.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:19 smithi181 ceph-mon[94189]: pgmap v22401: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 253 KiB/s wr, 9 op/s 2023-03-10T02:22:19.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:19 smithi038 ceph-mon[92422]: pgmap v22401: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 253 KiB/s wr, 9 op/s 2023-03-10T02:22:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:22:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:22:20.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41051.572356 secs 2023-03-10T02:22:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:22:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:22:20.733 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41051.572356 secs 2023-03-10T02:22:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:22:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:22:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41051.572356 secs 2023-03-10T02:22:21.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:21 smithi122 ceph-mon[93869]: pgmap v22402: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 252 KiB/s wr, 8 op/s 2023-03-10T02:22:21.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:22:21.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:22:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:21 smithi181 ceph-mon[94189]: pgmap v22402: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 252 KiB/s wr, 8 op/s 2023-03-10T02:22:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:22:21.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:22:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:21 smithi038 ceph-mon[92422]: pgmap v22402: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 252 KiB/s wr, 8 op/s 2023-03-10T02:22:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:22:21.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:22:23.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:23 smithi122 ceph-mon[93869]: pgmap v22403: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 579 KiB/s rd, 633 KiB/s wr, 12 op/s 2023-03-10T02:22:23.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:23 smithi181 ceph-mon[94189]: pgmap v22403: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 579 KiB/s rd, 633 KiB/s wr, 12 op/s 2023-03-10T02:22:23.751 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:23 smithi038 ceph-mon[92422]: pgmap v22403: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 579 KiB/s rd, 633 KiB/s wr, 12 op/s 2023-03-10T02:22:23.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41515.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:22:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:25 smithi122 ceph-mon[93869]: pgmap v22404: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 429 KiB/s wr, 10 op/s 2023-03-10T02:22:25.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:25 smithi122 ceph-mon[93869]: osdmap e1615: 12 total, 12 up, 12 in 2023-03-10T02:22:25.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41056.572414 secs 2023-03-10T02:22:25.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:25 smithi181 ceph-mon[94189]: pgmap v22404: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 429 KiB/s wr, 10 op/s 2023-03-10T02:22:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:25 smithi181 ceph-mon[94189]: osdmap e1615: 12 total, 12 up, 12 in 2023-03-10T02:22:25.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41056.572414 secs 2023-03-10T02:22:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:25 smithi038 ceph-mon[92422]: pgmap v22404: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 429 KiB/s wr, 10 op/s 2023-03-10T02:22:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:25 smithi038 ceph-mon[92422]: osdmap e1615: 12 total, 12 up, 12 in 2023-03-10T02:22:25.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41056.572414 secs 2023-03-10T02:22:27.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:27 smithi122 ceph-mon[93869]: pgmap v22406: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 516 KiB/s wr, 12 op/s 2023-03-10T02:22:27.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:27 smithi181 ceph-mon[94189]: pgmap v22406: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 516 KiB/s wr, 12 op/s 2023-03-10T02:22:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:27 smithi038 ceph-mon[92422]: pgmap v22406: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 492 KiB/s rd, 516 KiB/s wr, 12 op/s 2023-03-10T02:22:28.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41520.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:22:29.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:29 smithi122 ceph-mon[93869]: pgmap v22407: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 704 KiB/s rd, 715 KiB/s wr, 11 op/s 2023-03-10T02:22:29.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:29 smithi181 ceph-mon[94189]: pgmap v22407: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 704 KiB/s rd, 715 KiB/s wr, 11 op/s 2023-03-10T02:22:29.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:29 smithi038 ceph-mon[92422]: pgmap v22407: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 704 KiB/s rd, 715 KiB/s wr, 11 op/s 2023-03-10T02:22:30.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41061.572491 secs 2023-03-10T02:22:30.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41061.572491 secs 2023-03-10T02:22:30.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41061.572491 secs 2023-03-10T02:22:31.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:31 smithi122 ceph-mon[93869]: pgmap v22408: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 704 KiB/s rd, 716 KiB/s wr, 11 op/s 2023-03-10T02:22:31.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:31 smithi181 ceph-mon[94189]: pgmap v22408: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 704 KiB/s rd, 716 KiB/s wr, 11 op/s 2023-03-10T02:22:31.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:31 smithi038 ceph-mon[92422]: pgmap v22408: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 704 KiB/s rd, 716 KiB/s wr, 11 op/s 2023-03-10T02:22:33.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41525.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:22:33.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:33 smithi122 ceph-mon[93869]: pgmap v22409: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 457 KiB/s wr, 9 op/s 2023-03-10T02:22:33.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:33 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:22:33.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:33 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:22:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:33 smithi181 ceph-mon[94189]: pgmap v22409: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 457 KiB/s wr, 9 op/s 2023-03-10T02:22:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:33 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:22:33.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:33 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:22:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:33 smithi038 ceph-mon[92422]: pgmap v22409: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 518 KiB/s rd, 457 KiB/s wr, 9 op/s 2023-03-10T02:22:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:33 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:22:33.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:33 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:22:35.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:35 smithi122 ceph-mon[93869]: pgmap v22410: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 561 KiB/s rd, 542 KiB/s wr, 9 op/s 2023-03-10T02:22:35.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41066.572578 secs 2023-03-10T02:22:35.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:35 smithi122 ceph-mon[93869]: osdmap e1616: 12 total, 12 up, 12 in 2023-03-10T02:22:35.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:35 smithi181 ceph-mon[94189]: pgmap v22410: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 561 KiB/s rd, 542 KiB/s wr, 9 op/s 2023-03-10T02:22:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41066.572578 secs 2023-03-10T02:22:35.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:35 smithi181 ceph-mon[94189]: osdmap e1616: 12 total, 12 up, 12 in 2023-03-10T02:22:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:35 smithi038 ceph-mon[92422]: pgmap v22410: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 561 KiB/s rd, 542 KiB/s wr, 9 op/s 2023-03-10T02:22:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41066.572578 secs 2023-03-10T02:22:35.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:35 smithi038 ceph-mon[92422]: osdmap e1616: 12 total, 12 up, 12 in 2023-03-10T02:22:36.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:36 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:22:36.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:36 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:22:36.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:36 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:22:36.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:36 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:22:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:36 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:22:36.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:36 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:22:37.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:37 smithi122 ceph-mon[93869]: pgmap v22412: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 969 KiB/s rd, 950 KiB/s wr, 10 op/s 2023-03-10T02:22:37.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:22:37.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:22:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:37 smithi181 ceph-mon[94189]: pgmap v22412: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 969 KiB/s rd, 950 KiB/s wr, 10 op/s 2023-03-10T02:22:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:22:37.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:22:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:37 smithi038 ceph-mon[92422]: pgmap v22412: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 969 KiB/s rd, 950 KiB/s wr, 10 op/s 2023-03-10T02:22:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:22:37.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:22:38.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41530.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:22:38.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41069.086337 secs 2023-03-10T02:22:38.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41069.086337 secs 2023-03-10T02:22:38.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41069.086337 secs 2023-03-10T02:22:39.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:39 smithi122 ceph-mon[93869]: pgmap v22413: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 920 KiB/s rd, 696 KiB/s wr, 10 op/s 2023-03-10T02:22:39.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:39 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:22:39.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:39 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:22:39.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:39 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:22:39.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:39 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:22:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:39 smithi181 ceph-mon[94189]: pgmap v22413: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 920 KiB/s rd, 696 KiB/s wr, 10 op/s 2023-03-10T02:22:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:39 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:22:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:39 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:22:39.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:39 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:22:39.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:39 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:22:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:39 smithi038 ceph-mon[92422]: pgmap v22413: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 920 KiB/s rd, 696 KiB/s wr, 10 op/s 2023-03-10T02:22:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:39 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:22:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:39 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:22:39.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:39 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:22:40.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:39 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:22:40.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41071.572647 secs 2023-03-10T02:22:40.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41071.572647 secs 2023-03-10T02:22:40.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41071.572647 secs 2023-03-10T02:22:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:41 smithi122 ceph-mon[93869]: pgmap v22414: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 909 KiB/s wr, 11 op/s 2023-03-10T02:22:41.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:22:41.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:22:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:41 smithi181 ceph-mon[94189]: pgmap v22414: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 909 KiB/s wr, 11 op/s 2023-03-10T02:22:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:22:41.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:22:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:41 smithi038 ceph-mon[92422]: pgmap v22414: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 909 KiB/s wr, 11 op/s 2023-03-10T02:22:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:22:41.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:22:43.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41535.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:22:43.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:43 smithi122 ceph-mon[93869]: pgmap v22415: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 876 KiB/s rd, 712 KiB/s wr, 11 op/s 2023-03-10T02:22:43.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41074.086422 secs 2023-03-10T02:22:43.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:43 smithi181 ceph-mon[94189]: pgmap v22415: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 876 KiB/s rd, 712 KiB/s wr, 11 op/s 2023-03-10T02:22:43.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41074.086422 secs 2023-03-10T02:22:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:43 smithi038 ceph-mon[92422]: pgmap v22415: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 876 KiB/s rd, 712 KiB/s wr, 11 op/s 2023-03-10T02:22:43.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41074.086422 secs 2023-03-10T02:22:45.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:45 smithi122 ceph-mon[93869]: pgmap v22416: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 12 op/s 2023-03-10T02:22:45.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41076.572703 secs 2023-03-10T02:22:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:45 smithi181 ceph-mon[94189]: pgmap v22416: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 12 op/s 2023-03-10T02:22:45.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41076.572703 secs 2023-03-10T02:22:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:45 smithi038 ceph-mon[92422]: pgmap v22416: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 12 op/s 2023-03-10T02:22:45.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41076.572703 secs 2023-03-10T02:22:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:47 smithi122 ceph-mon[93869]: pgmap v22417: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 14 op/s 2023-03-10T02:22:47.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:22:47.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:22:47.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:47 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:22:47.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:47 smithi181 ceph-mon[94189]: pgmap v22417: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 14 op/s 2023-03-10T02:22:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:22:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:22:47.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:47 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:22:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:47 smithi038 ceph-mon[92422]: pgmap v22417: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 14 op/s 2023-03-10T02:22:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:22:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:22:47.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:47 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:22:48.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41540.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:22:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:22:48.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:22:48.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:22:48.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:22:48.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:22:48.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:22:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:22:48.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:22:49.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:22:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:49 smithi122 ceph-mon[93869]: pgmap v22418: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 687 KiB/s wr, 10 op/s 2023-03-10T02:22:49.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:22:49.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:22:49.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:49 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T02:22:49.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:49 smithi181 ceph-mon[94189]: pgmap v22418: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 687 KiB/s wr, 10 op/s 2023-03-10T02:22:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:22:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:22:49.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:49 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T02:22:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:49 smithi038 ceph-mon[92422]: pgmap v22418: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 687 KiB/s wr, 10 op/s 2023-03-10T02:22:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:22:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:22:49.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:49 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T02:22:50.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41081.572791 secs 2023-03-10T02:22:50.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:50 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:22:50.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:50 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:22:50.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:50 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:22:50.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41081.572791 secs 2023-03-10T02:22:50.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:50 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:22:50.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:50 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:22:50.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:50 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:22:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41081.572791 secs 2023-03-10T02:22:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:50 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:22:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:50 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:22:50.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:50 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:22:51.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:51 smithi181 ceph-mon[94189]: pgmap v22419: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 830 KiB/s rd, 990 KiB/s wr, 10 op/s 2023-03-10T02:22:51.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:51 smithi038 ceph-mon[92422]: pgmap v22419: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 830 KiB/s rd, 990 KiB/s wr, 10 op/s 2023-03-10T02:22:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:51 smithi122 ceph-mon[93869]: pgmap v22419: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 830 KiB/s rd, 990 KiB/s wr, 10 op/s 2023-03-10T02:22:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41545.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:22:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:53 smithi038 ceph-mon[92422]: pgmap v22420: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 653 KiB/s rd, 811 KiB/s wr, 12 op/s 2023-03-10T02:22:53.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:22:54.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:22:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:53 smithi122 ceph-mon[93869]: pgmap v22420: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 653 KiB/s rd, 811 KiB/s wr, 12 op/s 2023-03-10T02:22:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:22:54.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:22:54.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:53 smithi181 ceph-mon[94189]: pgmap v22420: 129 pgs: 129 active+clean; 12 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 653 KiB/s rd, 811 KiB/s wr, 12 op/s 2023-03-10T02:22:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:22:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:22:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:55 smithi038 ceph-mon[92422]: pgmap v22421: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 847 KiB/s wr, 10 op/s 2023-03-10T02:22:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:55 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:22:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:55 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:22:55.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41086.572875 secs 2023-03-10T02:22:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:55 smithi122 ceph-mon[93869]: pgmap v22421: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 847 KiB/s wr, 10 op/s 2023-03-10T02:22:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:55 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:22:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:55 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:22:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41086.572875 secs 2023-03-10T02:22:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:55 smithi181 ceph-mon[94189]: pgmap v22421: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 687 KiB/s rd, 847 KiB/s wr, 10 op/s 2023-03-10T02:22:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:55 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:22:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:55 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:22:56.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41086.572875 secs 2023-03-10T02:22:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:57 smithi122 ceph-mon[93869]: pgmap v22422: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 722 KiB/s rd, 492 KiB/s wr, 11 op/s 2023-03-10T02:22:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:57 smithi181 ceph-mon[94189]: pgmap v22422: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 722 KiB/s rd, 492 KiB/s wr, 11 op/s 2023-03-10T02:22:58.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:57 smithi038 ceph-mon[92422]: pgmap v22422: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 722 KiB/s rd, 492 KiB/s wr, 11 op/s 2023-03-10T02:22:58.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41550.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:23:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:59 smithi122 ceph-mon[93869]: pgmap v22423: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 757 KiB/s rd, 521 KiB/s wr, 11 op/s 2023-03-10T02:23:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:23:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:22:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:23:00.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:59 smithi181 ceph-mon[94189]: pgmap v22423: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 757 KiB/s rd, 521 KiB/s wr, 11 op/s 2023-03-10T02:23:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:23:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:22:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:23:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:59 smithi038 ceph-mon[92422]: pgmap v22423: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 757 KiB/s rd, 521 KiB/s wr, 11 op/s 2023-03-10T02:23:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:23:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:22:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:23:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41091.572933 secs 2023-03-10T02:23:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41091.572933 secs 2023-03-10T02:23:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41091.572933 secs 2023-03-10T02:23:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:01 smithi122 ceph-mon[93869]: pgmap v22424: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 819 KiB/s wr, 11 op/s 2023-03-10T02:23:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:23:02.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:23:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:01 smithi181 ceph-mon[94189]: pgmap v22424: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 819 KiB/s wr, 11 op/s 2023-03-10T02:23:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:23:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:23:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:01 smithi038 ceph-mon[92422]: pgmap v22424: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 819 KiB/s wr, 11 op/s 2023-03-10T02:23:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:23:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:23:03.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41555.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:23:04.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:03 smithi122 ceph-mon[93869]: pgmap v22425: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 966 KiB/s rd, 584 KiB/s wr, 13 op/s 2023-03-10T02:23:04.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:03 smithi181 ceph-mon[94189]: pgmap v22425: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 966 KiB/s rd, 584 KiB/s wr, 13 op/s 2023-03-10T02:23:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:03 smithi038 ceph-mon[92422]: pgmap v22425: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 966 KiB/s rd, 584 KiB/s wr, 13 op/s 2023-03-10T02:23:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:05 smithi122 ceph-mon[93869]: pgmap v22426: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 871 KiB/s wr, 24 op/s 2023-03-10T02:23:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41096.572982 secs 2023-03-10T02:23:06.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:05 smithi181 ceph-mon[94189]: pgmap v22426: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 871 KiB/s wr, 24 op/s 2023-03-10T02:23:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41096.572982 secs 2023-03-10T02:23:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:05 smithi038 ceph-mon[92422]: pgmap v22426: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 871 KiB/s wr, 24 op/s 2023-03-10T02:23:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41096.572982 secs 2023-03-10T02:23:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:07 smithi122 ceph-mon[93869]: pgmap v22427: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1014 KiB/s wr, 26 op/s 2023-03-10T02:23:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:23:08.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:23:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:07 smithi181 ceph-mon[94189]: pgmap v22427: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1014 KiB/s wr, 26 op/s 2023-03-10T02:23:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:23:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:23:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:07 smithi038 ceph-mon[92422]: pgmap v22427: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1014 KiB/s wr, 26 op/s 2023-03-10T02:23:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:23:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:23:08.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41560.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:23:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:09 smithi122 ceph-mon[93869]: pgmap v22428: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 971 KiB/s wr, 24 op/s 2023-03-10T02:23:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:23:10.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:23:10.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:09 smithi181 ceph-mon[94189]: pgmap v22428: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 971 KiB/s wr, 24 op/s 2023-03-10T02:23:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:23:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:23:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:09 smithi038 ceph-mon[92422]: pgmap v22428: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 971 KiB/s wr, 24 op/s 2023-03-10T02:23:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:23:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:23:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41101.573040 secs 2023-03-10T02:23:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:10 smithi122 ceph-mon[93869]: pgmap v22429: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 833 KiB/s wr, 23 op/s 2023-03-10T02:23:11.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41101.573040 secs 2023-03-10T02:23:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:10 smithi181 ceph-mon[94189]: pgmap v22429: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 833 KiB/s wr, 23 op/s 2023-03-10T02:23:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41101.573040 secs 2023-03-10T02:23:11.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:10 smithi038 ceph-mon[92422]: pgmap v22429: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 833 KiB/s wr, 23 op/s 2023-03-10T02:23:13.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:13 smithi122 ceph-mon[93869]: pgmap v22430: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 855 KiB/s wr, 24 op/s 2023-03-10T02:23:13.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:13 smithi181 ceph-mon[94189]: pgmap v22430: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 855 KiB/s wr, 24 op/s 2023-03-10T02:23:13.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:13 smithi038 ceph-mon[92422]: pgmap v22430: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 855 KiB/s wr, 24 op/s 2023-03-10T02:23:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41565.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:23:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:15 smithi122 ceph-mon[93869]: pgmap v22431: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 993 KiB/s rd, 787 KiB/s wr, 21 op/s 2023-03-10T02:23:15.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41106.573101 secs 2023-03-10T02:23:15.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:15 smithi181 ceph-mon[94189]: pgmap v22431: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 993 KiB/s rd, 787 KiB/s wr, 21 op/s 2023-03-10T02:23:15.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41106.573101 secs 2023-03-10T02:23:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:15 smithi038 ceph-mon[92422]: pgmap v22431: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 993 KiB/s rd, 787 KiB/s wr, 21 op/s 2023-03-10T02:23:15.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41106.573101 secs 2023-03-10T02:23:16.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:16 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:23:16.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:16 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:23:16.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:16 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:23:16.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:16 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:23:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:16 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:23:16.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:16 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:23:17.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:17 smithi122 ceph-mon[93869]: pgmap v22432: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 610 KiB/s rd, 500 KiB/s wr, 11 op/s 2023-03-10T02:23:17.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:23:17.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:23:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:17 smithi181 ceph-mon[94189]: pgmap v22432: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 610 KiB/s rd, 500 KiB/s wr, 11 op/s 2023-03-10T02:23:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:23:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:23:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:17 smithi038 ceph-mon[92422]: pgmap v22432: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 610 KiB/s rd, 500 KiB/s wr, 11 op/s 2023-03-10T02:23:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:23:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:23:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41570.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:23:19.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:19 smithi122 ceph-mon[93869]: pgmap v22433: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 466 KiB/s rd, 368 KiB/s wr, 11 op/s 2023-03-10T02:23:19.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:19 smithi181 ceph-mon[94189]: pgmap v22433: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 466 KiB/s rd, 368 KiB/s wr, 11 op/s 2023-03-10T02:23:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:19 smithi038 ceph-mon[92422]: pgmap v22433: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 466 KiB/s rd, 368 KiB/s wr, 11 op/s 2023-03-10T02:23:20.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:20 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:23:20.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:20 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:23:20.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41111.573159 secs 2023-03-10T02:23:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:20 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:23:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:20 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:23:20.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41111.573159 secs 2023-03-10T02:23:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:20 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:23:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:20 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:23:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41111.573159 secs 2023-03-10T02:23:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:21 smithi122 ceph-mon[93869]: pgmap v22434: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 739 KiB/s rd, 641 KiB/s wr, 11 op/s 2023-03-10T02:23:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:23:21.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:23:21.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:21 smithi181 ceph-mon[94189]: pgmap v22434: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 739 KiB/s rd, 641 KiB/s wr, 11 op/s 2023-03-10T02:23:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:23:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:23:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:21 smithi038 ceph-mon[92422]: pgmap v22434: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 739 KiB/s rd, 641 KiB/s wr, 11 op/s 2023-03-10T02:23:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:23:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:23:23.794 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41575.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:23:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:23 smithi122 ceph-mon[93869]: pgmap v22435: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 868 KiB/s rd, 780 KiB/s wr, 14 op/s 2023-03-10T02:23:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:23 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 41114.087000 secs 2023-03-10T02:23:23.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:23 smithi181 ceph-mon[94189]: pgmap v22435: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 868 KiB/s rd, 780 KiB/s wr, 14 op/s 2023-03-10T02:23:23.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:23 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 41114.087000 secs 2023-03-10T02:23:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:23 smithi038 ceph-mon[92422]: pgmap v22435: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 868 KiB/s rd, 780 KiB/s wr, 14 op/s 2023-03-10T02:23:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:23 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 41114.087000 secs 2023-03-10T02:23:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:25 smithi122 ceph-mon[93869]: pgmap v22436: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 442 KiB/s rd, 462 KiB/s wr, 13 op/s 2023-03-10T02:23:25.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:25 smithi122 ceph-mon[93869]: osdmap e1617: 12 total, 12 up, 12 in 2023-03-10T02:23:25.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41116.573248 secs 2023-03-10T02:23:25.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:25 smithi181 ceph-mon[94189]: pgmap v22436: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 442 KiB/s rd, 462 KiB/s wr, 13 op/s 2023-03-10T02:23:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:25 smithi181 ceph-mon[94189]: osdmap e1617: 12 total, 12 up, 12 in 2023-03-10T02:23:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41116.573248 secs 2023-03-10T02:23:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:25 smithi038 ceph-mon[92422]: pgmap v22436: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 442 KiB/s rd, 462 KiB/s wr, 13 op/s 2023-03-10T02:23:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:25 smithi038 ceph-mon[92422]: osdmap e1617: 12 total, 12 up, 12 in 2023-03-10T02:23:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41116.573248 secs 2023-03-10T02:23:26.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:26 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:23:26.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:26 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:23:26.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:26 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:23:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:26 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:23:27.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:26 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:23:27.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:26 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:23:27.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:26 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:23:27.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:26 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:23:27.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:26 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:23:27.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:27 smithi038 ceph-mon[92422]: pgmap v22438: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 665 KiB/s rd, 689 KiB/s wr, 14 op/s 2023-03-10T02:23:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:27 smithi122 ceph-mon[93869]: pgmap v22438: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 665 KiB/s rd, 689 KiB/s wr, 14 op/s 2023-03-10T02:23:28.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:27 smithi181 ceph-mon[94189]: pgmap v22438: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 665 KiB/s rd, 689 KiB/s wr, 14 op/s 2023-03-10T02:23:28.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41580.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:23:28.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:28 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 41119.087057 secs 2023-03-10T02:23:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:28 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 41119.087057 secs 2023-03-10T02:23:29.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:28 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 41119.087057 secs 2023-03-10T02:23:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:29 smithi122 ceph-mon[93869]: pgmap v22439: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 664 KiB/s rd, 676 KiB/s wr, 12 op/s 2023-03-10T02:23:30.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:29 smithi181 ceph-mon[94189]: pgmap v22439: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 664 KiB/s rd, 676 KiB/s wr, 12 op/s 2023-03-10T02:23:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:29 smithi038 ceph-mon[92422]: pgmap v22439: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 664 KiB/s rd, 676 KiB/s wr, 12 op/s 2023-03-10T02:23:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41121.573308 secs 2023-03-10T02:23:31.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41121.573308 secs 2023-03-10T02:23:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41121.573308 secs 2023-03-10T02:23:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:31 smithi122 ceph-mon[93869]: pgmap v22440: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 416 KiB/s rd, 428 KiB/s wr, 13 op/s 2023-03-10T02:23:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:31 smithi181 ceph-mon[94189]: pgmap v22440: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 416 KiB/s rd, 428 KiB/s wr, 13 op/s 2023-03-10T02:23:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:31 smithi038 ceph-mon[92422]: pgmap v22440: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 416 KiB/s rd, 428 KiB/s wr, 13 op/s 2023-03-10T02:23:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41585.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:23:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:33 smithi122 ceph-mon[93869]: pgmap v22441: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 262 KiB/s wr, 11 op/s 2023-03-10T02:23:34.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:33 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 41124.087133 secs 2023-03-10T02:23:34.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:33 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:23:34.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:33 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:23:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:33 smithi181 ceph-mon[94189]: pgmap v22441: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 262 KiB/s wr, 11 op/s 2023-03-10T02:23:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:33 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 41124.087133 secs 2023-03-10T02:23:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:33 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:23:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:33 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:23:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:33 smithi038 ceph-mon[92422]: pgmap v22441: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 263 KiB/s rd, 262 KiB/s wr, 11 op/s 2023-03-10T02:23:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:33 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 41124.087133 secs 2023-03-10T02:23:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:33 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:23:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:33 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:23:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:35 smithi122 ceph-mon[93869]: pgmap v22442: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 302 KiB/s wr, 10 op/s 2023-03-10T02:23:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:35 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:23:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:35 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:23:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41126.573368 secs 2023-03-10T02:23:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:35 smithi122 ceph-mon[93869]: osdmap e1618: 12 total, 12 up, 12 in 2023-03-10T02:23:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:35 smithi181 ceph-mon[94189]: pgmap v22442: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 302 KiB/s wr, 10 op/s 2023-03-10T02:23:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:35 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:23:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:35 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:23:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41126.573368 secs 2023-03-10T02:23:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:35 smithi181 ceph-mon[94189]: osdmap e1618: 12 total, 12 up, 12 in 2023-03-10T02:23:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:35 smithi038 ceph-mon[92422]: pgmap v22442: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 303 KiB/s rd, 302 KiB/s wr, 10 op/s 2023-03-10T02:23:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:35 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:23:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:35 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:23:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41126.573368 secs 2023-03-10T02:23:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:35 smithi038 ceph-mon[92422]: osdmap e1618: 12 total, 12 up, 12 in 2023-03-10T02:23:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:37 smithi122 ceph-mon[93869]: pgmap v22444: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 854 KiB/s rd, 852 KiB/s wr, 11 op/s 2023-03-10T02:23:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:23:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:23:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:37 smithi181 ceph-mon[94189]: pgmap v22444: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 854 KiB/s rd, 852 KiB/s wr, 11 op/s 2023-03-10T02:23:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:23:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:23:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:37 smithi038 ceph-mon[92422]: pgmap v22444: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 854 KiB/s rd, 852 KiB/s wr, 11 op/s 2023-03-10T02:23:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:23:38.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:23:38.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41590.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:23:39.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41129.087193 secs 2023-03-10T02:23:39.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:38 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:23:39.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:38 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:23:39.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41129.087193 secs 2023-03-10T02:23:39.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:38 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:23:39.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:38 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:23:39.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41129.087193 secs 2023-03-10T02:23:39.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:38 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:23:39.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:38 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:23:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:39 smithi122 ceph-mon[93869]: pgmap v22445: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 993 KiB/s rd, 991 KiB/s wr, 12 op/s 2023-03-10T02:23:40.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:39 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:23:40.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:39 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:23:40.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:39 smithi181 ceph-mon[94189]: pgmap v22445: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 993 KiB/s rd, 991 KiB/s wr, 12 op/s 2023-03-10T02:23:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:39 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:23:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:39 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:23:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:39 smithi038 ceph-mon[92422]: pgmap v22445: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 993 KiB/s rd, 991 KiB/s wr, 12 op/s 2023-03-10T02:23:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:39 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:23:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:39 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:23:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41131.573451 secs 2023-03-10T02:23:41.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41131.573451 secs 2023-03-10T02:23:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41131.573451 secs 2023-03-10T02:23:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:41 smithi122 ceph-mon[93869]: pgmap v22446: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 913 KiB/s rd, 911 KiB/s wr, 11 op/s 2023-03-10T02:23:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:23:42.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:23:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:41 smithi181 ceph-mon[94189]: pgmap v22446: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 913 KiB/s rd, 911 KiB/s wr, 11 op/s 2023-03-10T02:23:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:23:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:23:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:41 smithi038 ceph-mon[92422]: pgmap v22446: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 913 KiB/s rd, 911 KiB/s wr, 11 op/s 2023-03-10T02:23:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:23:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:23:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41595.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:23:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:43 smithi122 ceph-mon[93869]: pgmap v22447: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 12 op/s 2023-03-10T02:23:44.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41134.087253 secs 2023-03-10T02:23:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:43 smithi181 ceph-mon[94189]: pgmap v22447: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 12 op/s 2023-03-10T02:23:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41134.087253 secs 2023-03-10T02:23:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:43 smithi038 ceph-mon[92422]: pgmap v22447: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 12 op/s 2023-03-10T02:23:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41134.087253 secs 2023-03-10T02:23:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:45 smithi122 ceph-mon[93869]: pgmap v22448: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 11 op/s 2023-03-10T02:23:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41136.573509 secs 2023-03-10T02:23:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:45 smithi181 ceph-mon[94189]: pgmap v22448: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 11 op/s 2023-03-10T02:23:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41136.573509 secs 2023-03-10T02:23:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:45 smithi038 ceph-mon[92422]: pgmap v22448: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 11 op/s 2023-03-10T02:23:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41136.573509 secs 2023-03-10T02:23:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:47 smithi122 ceph-mon[93869]: pgmap v22449: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 12 op/s 2023-03-10T02:23:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:23:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:23:48.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:47 smithi181 ceph-mon[94189]: pgmap v22449: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 12 op/s 2023-03-10T02:23:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:23:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:23:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:47 smithi038 ceph-mon[92422]: pgmap v22449: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 12 op/s 2023-03-10T02:23:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:23:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:23:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41600.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:23:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:48 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 41139.087347 secs 2023-03-10T02:23:49.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:48 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:23:49.237 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:48 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 41139.087347 secs 2023-03-10T02:23:49.237 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:48 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:23:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:48 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 41139.087347 secs 2023-03-10T02:23:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:48 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:23:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:49 smithi122 ceph-mon[93869]: pgmap v22450: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 913 KiB/s rd, 912 KiB/s wr, 11 op/s 2023-03-10T02:23:50.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:23:50.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:23:50.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:23:50.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:23:50.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:23:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:49 smithi181 ceph-mon[94189]: pgmap v22450: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 913 KiB/s rd, 912 KiB/s wr, 11 op/s 2023-03-10T02:23:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:23:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:23:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:23:50.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:23:50.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:23:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:49 smithi038 ceph-mon[92422]: pgmap v22450: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 913 KiB/s rd, 912 KiB/s wr, 11 op/s 2023-03-10T02:23:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:23:50.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:23:50.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:23:50.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:23:50.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:23:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:23:51.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:23:51.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:51 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41141.573569 secs 2023-03-10T02:23:51.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:51 smithi122 ceph-mon[93869]: pgmap v22451: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 762 KiB/s rd, 760 KiB/s wr, 10 op/s 2023-03-10T02:23:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:23:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:23:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:51 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41141.573569 secs 2023-03-10T02:23:51.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:51 smithi181 ceph-mon[94189]: pgmap v22451: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 762 KiB/s rd, 760 KiB/s wr, 10 op/s 2023-03-10T02:23:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:23:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:23:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:51 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41141.573569 secs 2023-03-10T02:23:51.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:51 smithi038 ceph-mon[92422]: pgmap v22451: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 762 KiB/s rd, 760 KiB/s wr, 10 op/s 2023-03-10T02:23:53.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:53 smithi122 ceph-mon[93869]: pgmap v22452: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 877 KiB/s rd, 874 KiB/s wr, 13 op/s 2023-03-10T02:23:53.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:53 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 41144.087411 secs 2023-03-10T02:23:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:53 smithi181 ceph-mon[94189]: pgmap v22452: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 877 KiB/s rd, 874 KiB/s wr, 13 op/s 2023-03-10T02:23:53.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:53 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 41144.087411 secs 2023-03-10T02:23:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:53 smithi038 ceph-mon[92422]: pgmap v22452: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 877 KiB/s rd, 874 KiB/s wr, 13 op/s 2023-03-10T02:23:53.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:53 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 41144.087411 secs 2023-03-10T02:23:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41605.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:23:55.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:55 smithi122 ceph-mon[93869]: pgmap v22453: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 622 KiB/s wr, 10 op/s 2023-03-10T02:23:55.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41146.573628 secs 2023-03-10T02:23:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:55 smithi181 ceph-mon[94189]: pgmap v22453: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 622 KiB/s wr, 10 op/s 2023-03-10T02:23:55.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41146.573628 secs 2023-03-10T02:23:55.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:55 smithi038 ceph-mon[92422]: pgmap v22453: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 624 KiB/s rd, 622 KiB/s wr, 10 op/s 2023-03-10T02:23:55.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41146.573628 secs 2023-03-10T02:23:56.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:23:56.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:23:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:23:56.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:23:56.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:23:56.750 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:23:57.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:57 smithi122 ceph-mon[93869]: pgmap v22454: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 627 KiB/s rd, 622 KiB/s wr, 12 op/s 2023-03-10T02:23:57.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:23:57.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:23:57.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:57 smithi181 ceph-mon[94189]: pgmap v22454: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 627 KiB/s rd, 622 KiB/s wr, 12 op/s 2023-03-10T02:23:57.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:23:57.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:23:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:57 smithi038 ceph-mon[92422]: pgmap v22454: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 627 KiB/s rd, 622 KiB/s wr, 12 op/s 2023-03-10T02:23:57.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:23:58.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:23:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41610.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:23:58.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:58 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41149.087471 secs 2023-03-10T02:23:58.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:58 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41149.087471 secs 2023-03-10T02:23:58.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:58 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41149.087471 secs 2023-03-10T02:23:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:59 smithi122 ceph-mon[93869]: pgmap v22455: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 484 KiB/s wr, 11 op/s 2023-03-10T02:23:59.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:23:59.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:23:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:23:59.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:59 smithi181 ceph-mon[94189]: pgmap v22455: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 484 KiB/s wr, 11 op/s 2023-03-10T02:23:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:23:59.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:23:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:23:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:59 smithi038 ceph-mon[92422]: pgmap v22455: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 484 KiB/s wr, 11 op/s 2023-03-10T02:23:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:23:59.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:23:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:24:00.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41151.573686 secs 2023-03-10T02:24:00.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41151.573686 secs 2023-03-10T02:24:00.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41151.573686 secs 2023-03-10T02:24:01.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:01 smithi122 ceph-mon[93869]: pgmap v22456: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 738 KiB/s rd, 723 KiB/s wr, 10 op/s 2023-03-10T02:24:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:24:01.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:24:01.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:01 smithi181 ceph-mon[94189]: pgmap v22456: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 738 KiB/s rd, 723 KiB/s wr, 10 op/s 2023-03-10T02:24:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:24:01.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:24:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:01 smithi038 ceph-mon[92422]: pgmap v22456: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 738 KiB/s rd, 723 KiB/s wr, 10 op/s 2023-03-10T02:24:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:24:01.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:24:03.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41615.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:24:03.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:03 smithi122 ceph-mon[93869]: pgmap v22457: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 866 KiB/s rd, 861 KiB/s wr, 15 op/s 2023-03-10T02:24:03.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:03 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41154.087577 secs 2023-03-10T02:24:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:03 smithi181 ceph-mon[94189]: pgmap v22457: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 866 KiB/s rd, 861 KiB/s wr, 15 op/s 2023-03-10T02:24:03.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:03 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41154.087577 secs 2023-03-10T02:24:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:03 smithi038 ceph-mon[92422]: pgmap v22457: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 866 KiB/s rd, 861 KiB/s wr, 15 op/s 2023-03-10T02:24:03.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:03 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41154.087577 secs 2023-03-10T02:24:05.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:05 smithi122 ceph-mon[93869]: pgmap v22458: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 940 KiB/s rd, 940 KiB/s wr, 24 op/s 2023-03-10T02:24:05.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41156.573748 secs 2023-03-10T02:24:05.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:05 smithi181 ceph-mon[94189]: pgmap v22458: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 940 KiB/s rd, 940 KiB/s wr, 24 op/s 2023-03-10T02:24:05.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41156.573748 secs 2023-03-10T02:24:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:05 smithi038 ceph-mon[92422]: pgmap v22458: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 940 KiB/s rd, 940 KiB/s wr, 24 op/s 2023-03-10T02:24:05.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41156.573748 secs 2023-03-10T02:24:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:07 smithi122 ceph-mon[93869]: pgmap v22459: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 25 op/s 2023-03-10T02:24:07.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:24:07.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:24:07.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:07 smithi181 ceph-mon[94189]: pgmap v22459: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 25 op/s 2023-03-10T02:24:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:24:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:24:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:07 smithi038 ceph-mon[92422]: pgmap v22459: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 25 op/s 2023-03-10T02:24:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:24:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:24:08.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41620.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:24:08.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:08 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41159.087676 secs 2023-03-10T02:24:08.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:08 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41159.087676 secs 2023-03-10T02:24:08.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:08 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41159.087676 secs 2023-03-10T02:24:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:09 smithi122 ceph-mon[93869]: pgmap v22460: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 25 op/s 2023-03-10T02:24:09.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:24:09.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:24:09.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:09 smithi181 ceph-mon[94189]: pgmap v22460: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 25 op/s 2023-03-10T02:24:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:24:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:24:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:09 smithi038 ceph-mon[92422]: pgmap v22460: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 25 op/s 2023-03-10T02:24:10.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:24:10.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:24:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41161.573801 secs 2023-03-10T02:24:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41161.573801 secs 2023-03-10T02:24:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41161.573801 secs 2023-03-10T02:24:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:11 smithi122 ceph-mon[93869]: pgmap v22461: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 23 op/s 2023-03-10T02:24:11.982 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:11 smithi181 ceph-mon[94189]: pgmap v22461: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 23 op/s 2023-03-10T02:24:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:11 smithi038 ceph-mon[92422]: pgmap v22461: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 23 op/s 2023-03-10T02:24:13.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41625.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:24:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:13 smithi122 ceph-mon[93869]: pgmap v22462: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 767 KiB/s rd, 778 KiB/s wr, 24 op/s 2023-03-10T02:24:13.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:13 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41164.087761 secs 2023-03-10T02:24:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:13 smithi181 ceph-mon[94189]: pgmap v22462: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 767 KiB/s rd, 778 KiB/s wr, 24 op/s 2023-03-10T02:24:13.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:13 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41164.087761 secs 2023-03-10T02:24:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:13 smithi038 ceph-mon[92422]: pgmap v22462: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 767 KiB/s rd, 778 KiB/s wr, 24 op/s 2023-03-10T02:24:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:13 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41164.087761 secs 2023-03-10T02:24:15.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:15 smithi122 ceph-mon[93869]: pgmap v22463: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 638 KiB/s rd, 640 KiB/s wr, 20 op/s 2023-03-10T02:24:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:15 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:24:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:15 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:24:15.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41166.573858 secs 2023-03-10T02:24:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:15 smithi181 ceph-mon[94189]: pgmap v22463: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 638 KiB/s rd, 640 KiB/s wr, 20 op/s 2023-03-10T02:24:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:15 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:24:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:15 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:24:15.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41166.573858 secs 2023-03-10T02:24:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:15 smithi038 ceph-mon[92422]: pgmap v22463: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 638 KiB/s rd, 640 KiB/s wr, 20 op/s 2023-03-10T02:24:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:15 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:24:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:15 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:24:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41166.573858 secs 2023-03-10T02:24:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:17 smithi181 ceph-mon[94189]: pgmap v22464: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 447 KiB/s wr, 10 op/s 2023-03-10T02:24:17.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:24:17.981 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:24:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:17 smithi038 ceph-mon[92422]: pgmap v22464: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 447 KiB/s wr, 10 op/s 2023-03-10T02:24:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:24:18.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:24:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:17 smithi122 ceph-mon[93869]: pgmap v22464: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 447 KiB/s wr, 10 op/s 2023-03-10T02:24:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:24:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:24:18.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41630.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:24:18.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:18 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41169.087846 secs 2023-03-10T02:24:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:18 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41169.087846 secs 2023-03-10T02:24:19.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:18 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41169.087846 secs 2023-03-10T02:24:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:19 smithi038 ceph-mon[92422]: pgmap v22465: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 392 KiB/s wr, 10 op/s 2023-03-10T02:24:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:19 smithi122 ceph-mon[93869]: pgmap v22465: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 392 KiB/s wr, 10 op/s 2023-03-10T02:24:20.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:19 smithi181 ceph-mon[94189]: pgmap v22465: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 392 KiB/s wr, 10 op/s 2023-03-10T02:24:20.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41171.573916 secs 2023-03-10T02:24:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41171.573916 secs 2023-03-10T02:24:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41171.573916 secs 2023-03-10T02:24:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:21 smithi038 ceph-mon[92422]: pgmap v22466: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 248 KiB/s wr, 8 op/s 2023-03-10T02:24:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:24:22.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:24:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:21 smithi122 ceph-mon[93869]: pgmap v22466: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 248 KiB/s wr, 8 op/s 2023-03-10T02:24:22.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:24:22.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:24:22.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:21 smithi181 ceph-mon[94189]: pgmap v22466: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 248 KiB/s wr, 8 op/s 2023-03-10T02:24:22.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:24:22.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:24:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41635.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:24:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:23 smithi122 ceph-mon[93869]: pgmap v22467: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 370 KiB/s wr, 12 op/s 2023-03-10T02:24:24.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:23 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41174.087905 secs 2023-03-10T02:24:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:23 smithi181 ceph-mon[94189]: pgmap v22467: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 370 KiB/s wr, 12 op/s 2023-03-10T02:24:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:23 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41174.087905 secs 2023-03-10T02:24:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:23 smithi038 ceph-mon[92422]: pgmap v22467: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 370 KiB/s wr, 12 op/s 2023-03-10T02:24:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:23 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41174.087905 secs 2023-03-10T02:24:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:25 smithi122 ceph-mon[93869]: pgmap v22468: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 287 KiB/s wr, 10 op/s 2023-03-10T02:24:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:25 smithi122 ceph-mon[93869]: osdmap e1619: 12 total, 12 up, 12 in 2023-03-10T02:24:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41176.574001 secs 2023-03-10T02:24:26.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:25 smithi181 ceph-mon[94189]: pgmap v22468: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 287 KiB/s wr, 10 op/s 2023-03-10T02:24:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:25 smithi181 ceph-mon[94189]: osdmap e1619: 12 total, 12 up, 12 in 2023-03-10T02:24:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41176.574001 secs 2023-03-10T02:24:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:25 smithi038 ceph-mon[92422]: pgmap v22468: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 264 KiB/s rd, 287 KiB/s wr, 10 op/s 2023-03-10T02:24:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:25 smithi038 ceph-mon[92422]: osdmap e1619: 12 total, 12 up, 12 in 2023-03-10T02:24:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41176.574001 secs 2023-03-10T02:24:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:27 smithi122 ceph-mon[93869]: pgmap v22470: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 345 KiB/s wr, 12 op/s 2023-03-10T02:24:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:27 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:24:28.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:27 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:24:28.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:27 smithi181 ceph-mon[94189]: pgmap v22470: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 345 KiB/s wr, 12 op/s 2023-03-10T02:24:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:27 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:24:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:27 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:24:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:27 smithi038 ceph-mon[92422]: pgmap v22470: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 345 KiB/s wr, 12 op/s 2023-03-10T02:24:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:27 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:24:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:27 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:24:28.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41640.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:24:29.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:28 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41179.087963 secs 2023-03-10T02:24:29.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:28 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41179.087963 secs 2023-03-10T02:24:29.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:28 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41179.087963 secs 2023-03-10T02:24:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:29 smithi122 ceph-mon[93869]: pgmap v22471: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 573 KiB/s rd, 445 KiB/s wr, 11 op/s 2023-03-10T02:24:30.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:29 smithi181 ceph-mon[94189]: pgmap v22471: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 573 KiB/s rd, 445 KiB/s wr, 11 op/s 2023-03-10T02:24:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:29 smithi038 ceph-mon[92422]: pgmap v22471: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 573 KiB/s rd, 445 KiB/s wr, 11 op/s 2023-03-10T02:24:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41181.574059 secs 2023-03-10T02:24:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41181.574059 secs 2023-03-10T02:24:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41181.574059 secs 2023-03-10T02:24:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:31 smithi122 ceph-mon[93869]: pgmap v22472: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 573 KiB/s rd, 445 KiB/s wr, 11 op/s 2023-03-10T02:24:32.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:31 smithi181 ceph-mon[94189]: pgmap v22472: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 573 KiB/s rd, 445 KiB/s wr, 11 op/s 2023-03-10T02:24:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:31 smithi038 ceph-mon[92422]: pgmap v22472: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 573 KiB/s rd, 445 KiB/s wr, 11 op/s 2023-03-10T02:24:33.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41645.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:24:34.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:33 smithi122 ceph-mon[93869]: pgmap v22473: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 300 KiB/s wr, 9 op/s 2023-03-10T02:24:34.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:33 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41184.088063 secs 2023-03-10T02:24:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:33 smithi181 ceph-mon[94189]: pgmap v22473: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 300 KiB/s wr, 9 op/s 2023-03-10T02:24:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:33 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41184.088063 secs 2023-03-10T02:24:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:33 smithi038 ceph-mon[92422]: pgmap v22473: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 300 KiB/s wr, 9 op/s 2023-03-10T02:24:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:33 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41184.088063 secs 2023-03-10T02:24:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:35 smithi122 ceph-mon[93869]: pgmap v22474: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 559 KiB/s wr, 9 op/s 2023-03-10T02:24:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41186.574120 secs 2023-03-10T02:24:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:35 smithi122 ceph-mon[93869]: osdmap e1620: 12 total, 12 up, 12 in 2023-03-10T02:24:36.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:35 smithi181 ceph-mon[94189]: pgmap v22474: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 559 KiB/s wr, 9 op/s 2023-03-10T02:24:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41186.574120 secs 2023-03-10T02:24:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:35 smithi181 ceph-mon[94189]: osdmap e1620: 12 total, 12 up, 12 in 2023-03-10T02:24:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:35 smithi038 ceph-mon[92422]: pgmap v22474: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 559 KiB/s wr, 9 op/s 2023-03-10T02:24:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41186.574120 secs 2023-03-10T02:24:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:35 smithi038 ceph-mon[92422]: osdmap e1620: 12 total, 12 up, 12 in 2023-03-10T02:24:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:37 smithi122 ceph-mon[93869]: pgmap v22476: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 559 KiB/s wr, 8 op/s 2023-03-10T02:24:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:37 smithi181 ceph-mon[94189]: pgmap v22476: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 559 KiB/s wr, 8 op/s 2023-03-10T02:24:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:37 smithi038 ceph-mon[92422]: pgmap v22476: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 559 KiB/s wr, 8 op/s 2023-03-10T02:24:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41650.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:24:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41189.088176 secs 2023-03-10T02:24:39.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41189.088176 secs 2023-03-10T02:24:39.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41189.088176 secs 2023-03-10T02:24:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:39 smithi122 ceph-mon[93869]: pgmap v22477: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 268 KiB/s wr, 8 op/s 2023-03-10T02:24:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:39 smithi181 ceph-mon[94189]: pgmap v22477: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 268 KiB/s wr, 8 op/s 2023-03-10T02:24:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:39 smithi038 ceph-mon[92422]: pgmap v22477: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 268 KiB/s wr, 8 op/s 2023-03-10T02:24:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41191.574206 secs 2023-03-10T02:24:41.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41191.574206 secs 2023-03-10T02:24:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41191.574206 secs 2023-03-10T02:24:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:41 smithi122 ceph-mon[93869]: pgmap v22478: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 268 KiB/s wr, 8 op/s 2023-03-10T02:24:42.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:41 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:24:42.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:41 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:24:42.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:41 smithi181 ceph-mon[94189]: pgmap v22478: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 268 KiB/s wr, 8 op/s 2023-03-10T02:24:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:41 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:24:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:41 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:24:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:41 smithi038 ceph-mon[92422]: pgmap v22478: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 308 KiB/s rd, 268 KiB/s wr, 8 op/s 2023-03-10T02:24:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:41 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:24:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:41 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:24:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41655.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:24:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:43 smithi122 ceph-mon[93869]: pgmap v22479: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 353 KiB/s wr, 10 op/s 2023-03-10T02:24:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41194.088261 secs 2023-03-10T02:24:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:43 smithi181 ceph-mon[94189]: pgmap v22479: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 353 KiB/s wr, 10 op/s 2023-03-10T02:24:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41194.088261 secs 2023-03-10T02:24:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:43 smithi038 ceph-mon[92422]: pgmap v22479: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 393 KiB/s rd, 353 KiB/s wr, 10 op/s 2023-03-10T02:24:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41194.088261 secs 2023-03-10T02:24:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:45 smithi122 ceph-mon[93869]: pgmap v22480: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 561 KiB/s rd, 293 KiB/s wr, 9 op/s 2023-03-10T02:24:46.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41196.574268 secs 2023-03-10T02:24:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:45 smithi181 ceph-mon[94189]: pgmap v22480: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 561 KiB/s rd, 293 KiB/s wr, 9 op/s 2023-03-10T02:24:46.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41196.574268 secs 2023-03-10T02:24:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:45 smithi038 ceph-mon[92422]: pgmap v22480: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 561 KiB/s rd, 293 KiB/s wr, 9 op/s 2023-03-10T02:24:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41196.574268 secs 2023-03-10T02:24:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:47 smithi122 ceph-mon[93869]: pgmap v22481: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 976 KiB/s rd, 725 KiB/s wr, 10 op/s 2023-03-10T02:24:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:24:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:24:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:47 smithi181 ceph-mon[94189]: pgmap v22481: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 976 KiB/s rd, 725 KiB/s wr, 10 op/s 2023-03-10T02:24:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:24:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:24:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:47 smithi038 ceph-mon[92422]: pgmap v22481: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 976 KiB/s rd, 725 KiB/s wr, 10 op/s 2023-03-10T02:24:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:24:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:24:48.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41660.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:24:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:48 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41199.088317 secs 2023-03-10T02:24:49.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:48 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41199.088317 secs 2023-03-10T02:24:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:48 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41199.088317 secs 2023-03-10T02:24:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:49 smithi122 ceph-mon[93869]: pgmap v22482: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 870 KiB/s rd, 647 KiB/s wr, 9 op/s 2023-03-10T02:24:50.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:49 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:24:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:49 smithi181 ceph-mon[94189]: pgmap v22482: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 870 KiB/s rd, 647 KiB/s wr, 9 op/s 2023-03-10T02:24:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:49 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:24:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:49 smithi038 ceph-mon[92422]: pgmap v22482: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 870 KiB/s rd, 647 KiB/s wr, 9 op/s 2023-03-10T02:24:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:49 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:24:51.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:50 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:24:51.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:50 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:24:51.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:50 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:24:51.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41201.574326 secs 2023-03-10T02:24:51.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:50 smithi122 ceph-mon[93869]: pgmap v22483: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 858 KiB/s rd, 856 KiB/s wr, 8 op/s 2023-03-10T02:24:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:50 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:24:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:50 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:24:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:50 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:24:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41201.574326 secs 2023-03-10T02:24:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:50 smithi181 ceph-mon[94189]: pgmap v22483: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 858 KiB/s rd, 856 KiB/s wr, 8 op/s 2023-03-10T02:24:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:50 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:24:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:50 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:24:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:50 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:24:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41201.574326 secs 2023-03-10T02:24:51.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:50 smithi038 ceph-mon[92422]: pgmap v22483: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 858 KiB/s rd, 856 KiB/s wr, 8 op/s 2023-03-10T02:24:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:52 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:24:53.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:52 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:24:53.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:52 smithi038 ceph-mon[92422]: pgmap v22484: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 859 KiB/s rd, 856 KiB/s wr, 10 op/s 2023-03-10T02:24:53.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:52 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:24:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:52 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:24:53.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:52 smithi122 ceph-mon[93869]: pgmap v22484: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 859 KiB/s rd, 856 KiB/s wr, 10 op/s 2023-03-10T02:24:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:52 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:24:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:52 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:24:53.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:52 smithi181 ceph-mon[94189]: pgmap v22484: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 859 KiB/s rd, 856 KiB/s wr, 10 op/s 2023-03-10T02:24:53.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41665.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:24:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:53 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41204.088395 secs 2023-03-10T02:24:54.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:53 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41204.088395 secs 2023-03-10T02:24:54.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:53 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41204.088395 secs 2023-03-10T02:24:55.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:55 smithi122 ceph-mon[93869]: pgmap v22485: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 823 KiB/s rd, 821 KiB/s wr, 8 op/s 2023-03-10T02:24:55.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:55 smithi181 ceph-mon[94189]: pgmap v22485: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 823 KiB/s rd, 821 KiB/s wr, 8 op/s 2023-03-10T02:24:55.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:55 smithi038 ceph-mon[92422]: pgmap v22485: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 823 KiB/s rd, 821 KiB/s wr, 8 op/s 2023-03-10T02:24:56.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:56 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:24:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:56 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:24:56.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:56 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41206.574410 secs 2023-03-10T02:24:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:56 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:24:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:56 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:24:56.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:56 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41206.574410 secs 2023-03-10T02:24:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:56 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:24:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:56 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:24:56.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:56 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41206.574410 secs 2023-03-10T02:24:57.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:57 smithi122 ceph-mon[93869]: pgmap v22486: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 852 KiB/s rd, 716 KiB/s wr, 11 op/s 2023-03-10T02:24:57.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:57 smithi181 ceph-mon[94189]: pgmap v22486: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 852 KiB/s rd, 716 KiB/s wr, 11 op/s 2023-03-10T02:24:57.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:57 smithi038 ceph-mon[92422]: pgmap v22486: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 852 KiB/s rd, 716 KiB/s wr, 11 op/s 2023-03-10T02:24:58.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:58 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T02:24:58.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:58 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T02:24:58.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:58 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T02:24:58.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41670.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:24:59.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:59 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:24:59.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:24:59.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:24:59.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:24:59 smithi122 ceph-mon[93869]: pgmap v22487: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 484 KiB/s rd, 350 KiB/s wr, 10 op/s 2023-03-10T02:24:59.479 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:59 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:24:59.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:24:59.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:24:59.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:24:59 smithi181 ceph-mon[94189]: pgmap v22487: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 484 KiB/s rd, 350 KiB/s wr, 10 op/s 2023-03-10T02:24:59.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:59 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:24:59.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:24:59.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:24:59.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:24:59 smithi038 ceph-mon[92422]: pgmap v22487: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 484 KiB/s rd, 350 KiB/s wr, 10 op/s 2023-03-10T02:25:00.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:00 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:25:00.477 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:00 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:25:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:00 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:25:00.480 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:00 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:25:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:00 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:25:00.499 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:00 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:25:01.476 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:25:01.508 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:25:01.508 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:01 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41211.574497 secs 2023-03-10T02:25:01.509 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:01 smithi122 ceph-mon[93869]: pgmap v22488: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 350 KiB/s wr, 10 op/s 2023-03-10T02:25:01.509 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:25:01.510 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:25:01.510 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:01 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41211.574497 secs 2023-03-10T02:25:01.510 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:01 smithi038 ceph-mon[92422]: pgmap v22488: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 350 KiB/s wr, 10 op/s 2023-03-10T02:25:01.510 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:25:01.510 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:25:01.510 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:01 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41211.574497 secs 2023-03-10T02:25:01.511 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:01 smithi181 ceph-mon[94189]: pgmap v22488: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 495 KiB/s rd, 350 KiB/s wr, 10 op/s 2023-03-10T02:25:03.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:03 smithi122 ceph-mon[93869]: pgmap v22489: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 940 KiB/s rd, 921 KiB/s wr, 15 op/s 2023-03-10T02:25:03.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:03 smithi181 ceph-mon[94189]: pgmap v22489: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 940 KiB/s rd, 921 KiB/s wr, 15 op/s 2023-03-10T02:25:03.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:03 smithi038 ceph-mon[92422]: pgmap v22489: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 940 KiB/s rd, 921 KiB/s wr, 15 op/s 2023-03-10T02:25:03.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41675.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:25:05.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:05 smithi122 ceph-mon[93869]: pgmap v22490: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 945 KiB/s rd, 931 KiB/s wr, 24 op/s 2023-03-10T02:25:05.727 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41216.574558 secs 2023-03-10T02:25:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:05 smithi181 ceph-mon[94189]: pgmap v22490: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 945 KiB/s rd, 931 KiB/s wr, 24 op/s 2023-03-10T02:25:05.730 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41216.574558 secs 2023-03-10T02:25:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:05 smithi038 ceph-mon[92422]: pgmap v22490: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 945 KiB/s rd, 931 KiB/s wr, 24 op/s 2023-03-10T02:25:05.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41216.574558 secs 2023-03-10T02:25:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:07 smithi122 ceph-mon[93869]: pgmap v22491: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 27 op/s 2023-03-10T02:25:07.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:07 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:25:07.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:07 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:25:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:07 smithi181 ceph-mon[94189]: pgmap v22491: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 27 op/s 2023-03-10T02:25:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:07 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:25:07.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:07 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:25:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:07 smithi038 ceph-mon[92422]: pgmap v22491: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 27 op/s 2023-03-10T02:25:07.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:07 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:25:08.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:07 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:25:08.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41680.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:25:09.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:09 smithi122 ceph-mon[93869]: pgmap v22492: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1010 KiB/s rd, 1008 KiB/s wr, 24 op/s 2023-03-10T02:25:09.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:25:09.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:25:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:09 smithi181 ceph-mon[94189]: pgmap v22492: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1010 KiB/s rd, 1008 KiB/s wr, 24 op/s 2023-03-10T02:25:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:25:09.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:25:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:09 smithi038 ceph-mon[92422]: pgmap v22492: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1010 KiB/s rd, 1008 KiB/s wr, 24 op/s 2023-03-10T02:25:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:25:09.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:25:10.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41221.574615 secs 2023-03-10T02:25:10.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41221.574615 secs 2023-03-10T02:25:10.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41221.574615 secs 2023-03-10T02:25:11.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:11 smithi122 ceph-mon[93869]: pgmap v22493: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 974 KiB/s rd, 972 KiB/s wr, 23 op/s 2023-03-10T02:25:11.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:11 smithi181 ceph-mon[94189]: pgmap v22493: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 974 KiB/s rd, 972 KiB/s wr, 23 op/s 2023-03-10T02:25:11.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:11 smithi038 ceph-mon[92422]: pgmap v22493: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 974 KiB/s rd, 972 KiB/s wr, 23 op/s 2023-03-10T02:25:13.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41685.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:25:13.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:13 smithi122 ceph-mon[93869]: pgmap v22494: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 27 op/s 2023-03-10T02:25:13.979 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:13 smithi181 ceph-mon[94189]: pgmap v22494: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 27 op/s 2023-03-10T02:25:13.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:13 smithi038 ceph-mon[92422]: pgmap v22494: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.2 MiB/s wr, 27 op/s 2023-03-10T02:25:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:15 smithi181 ceph-mon[94189]: pgmap v22495: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 517 KiB/s rd, 399 KiB/s wr, 21 op/s 2023-03-10T02:25:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:15 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:25:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:15 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:25:15.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41226.574675 secs 2023-03-10T02:25:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:15 smithi038 ceph-mon[92422]: pgmap v22495: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 517 KiB/s rd, 399 KiB/s wr, 21 op/s 2023-03-10T02:25:15.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:15 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:25:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:15 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:25:16.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41226.574675 secs 2023-03-10T02:25:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:15 smithi122 ceph-mon[93869]: pgmap v22495: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 517 KiB/s rd, 399 KiB/s wr, 21 op/s 2023-03-10T02:25:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:15 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:25:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:15 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:25:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41226.574675 secs 2023-03-10T02:25:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:17 smithi038 ceph-mon[92422]: pgmap v22496: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 511 KiB/s rd, 388 KiB/s wr, 12 op/s 2023-03-10T02:25:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:25:17.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:25:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:17 smithi122 ceph-mon[93869]: pgmap v22496: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 511 KiB/s rd, 388 KiB/s wr, 12 op/s 2023-03-10T02:25:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:25:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:25:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:17 smithi181 ceph-mon[94189]: pgmap v22496: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 511 KiB/s rd, 388 KiB/s wr, 12 op/s 2023-03-10T02:25:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:25:18.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:25:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41690.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:25:19.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:19 smithi038 ceph-mon[92422]: pgmap v22497: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 219 KiB/s wr, 10 op/s 2023-03-10T02:25:20.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:19 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:25:20.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:19 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:25:20.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:19 smithi122 ceph-mon[93869]: pgmap v22497: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 219 KiB/s wr, 10 op/s 2023-03-10T02:25:20.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:19 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:25:20.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:19 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:25:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:19 smithi181 ceph-mon[94189]: pgmap v22497: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 210 KiB/s rd, 219 KiB/s wr, 10 op/s 2023-03-10T02:25:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:19 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:25:20.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:19 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:25:21.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41231.574748 secs 2023-03-10T02:25:21.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41231.574748 secs 2023-03-10T02:25:21.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41231.574748 secs 2023-03-10T02:25:22.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:21 smithi122 ceph-mon[93869]: pgmap v22498: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 412 KiB/s wr, 10 op/s 2023-03-10T02:25:22.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:21 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:25:22.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:21 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:25:22.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:21 smithi181 ceph-mon[94189]: pgmap v22498: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 412 KiB/s wr, 10 op/s 2023-03-10T02:25:22.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:21 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:25:22.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:21 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:25:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:21 smithi038 ceph-mon[92422]: pgmap v22498: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 404 KiB/s rd, 412 KiB/s wr, 10 op/s 2023-03-10T02:25:22.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:21 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:25:22.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:21 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:25:23.797 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41695.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:25:24.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:23 smithi122 ceph-mon[93869]: pgmap v22499: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 621 KiB/s rd, 640 KiB/s wr, 14 op/s 2023-03-10T02:25:24.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:23 smithi181 ceph-mon[94189]: pgmap v22499: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 621 KiB/s rd, 640 KiB/s wr, 14 op/s 2023-03-10T02:25:24.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:23 smithi038 ceph-mon[92422]: pgmap v22499: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 621 KiB/s rd, 640 KiB/s wr, 14 op/s 2023-03-10T02:25:26.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:25 smithi122 ceph-mon[93869]: pgmap v22500: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 433 KiB/s wr, 10 op/s 2023-03-10T02:25:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:25 smithi122 ceph-mon[93869]: osdmap e1621: 12 total, 12 up, 12 in 2023-03-10T02:25:26.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41236.574808 secs 2023-03-10T02:25:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:25 smithi181 ceph-mon[94189]: pgmap v22500: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 433 KiB/s wr, 10 op/s 2023-03-10T02:25:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:25 smithi181 ceph-mon[94189]: osdmap e1621: 12 total, 12 up, 12 in 2023-03-10T02:25:26.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41236.574808 secs 2023-03-10T02:25:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:25 smithi038 ceph-mon[92422]: pgmap v22500: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 433 KiB/s wr, 10 op/s 2023-03-10T02:25:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:25 smithi038 ceph-mon[92422]: osdmap e1621: 12 total, 12 up, 12 in 2023-03-10T02:25:26.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41236.574808 secs 2023-03-10T02:25:28.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:27 smithi122 ceph-mon[93869]: pgmap v22502: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 711 KiB/s rd, 735 KiB/s wr, 13 op/s 2023-03-10T02:25:28.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:27 smithi181 ceph-mon[94189]: pgmap v22502: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 711 KiB/s rd, 735 KiB/s wr, 13 op/s 2023-03-10T02:25:28.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:27 smithi038 ceph-mon[92422]: pgmap v22502: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 711 KiB/s rd, 735 KiB/s wr, 13 op/s 2023-03-10T02:25:28.795 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41700.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:25:30.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:29 smithi122 ceph-mon[93869]: pgmap v22503: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 711 KiB/s rd, 723 KiB/s wr, 12 op/s 2023-03-10T02:25:30.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:29 smithi181 ceph-mon[94189]: pgmap v22503: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 711 KiB/s rd, 723 KiB/s wr, 12 op/s 2023-03-10T02:25:30.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:29 smithi038 ceph-mon[92422]: pgmap v22503: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 711 KiB/s rd, 723 KiB/s wr, 12 op/s 2023-03-10T02:25:31.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:30 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41241.574865 secs 2023-03-10T02:25:31.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:30 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41241.574865 secs 2023-03-10T02:25:31.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:30 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41241.574865 secs 2023-03-10T02:25:32.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:31 smithi122 ceph-mon[93869]: pgmap v22504: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 478 KiB/s rd, 490 KiB/s wr, 11 op/s 2023-03-10T02:25:32.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:31 smithi181 ceph-mon[94189]: pgmap v22504: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 478 KiB/s rd, 490 KiB/s wr, 11 op/s 2023-03-10T02:25:32.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:31 smithi038 ceph-mon[92422]: pgmap v22504: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 478 KiB/s rd, 490 KiB/s wr, 11 op/s 2023-03-10T02:25:33.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41705.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:25:34.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:33 smithi122 ceph-mon[93869]: pgmap v22505: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 402 KiB/s rd, 401 KiB/s wr, 9 op/s 2023-03-10T02:25:34.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:33 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:25:34.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:33 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:25:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:33 smithi181 ceph-mon[94189]: pgmap v22505: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 402 KiB/s rd, 401 KiB/s wr, 9 op/s 2023-03-10T02:25:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:33 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:25:34.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:33 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:25:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:33 smithi038 ceph-mon[92422]: pgmap v22505: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 402 KiB/s rd, 401 KiB/s wr, 9 op/s 2023-03-10T02:25:34.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:33 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:25:34.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:33 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:25:36.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:35 smithi122 ceph-mon[93869]: pgmap v22506: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 402 KiB/s rd, 401 KiB/s wr, 9 op/s 2023-03-10T02:25:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:35 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:25:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:35 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:25:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:35 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41246.574939 secs 2023-03-10T02:25:36.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:35 smithi122 ceph-mon[93869]: osdmap e1622: 12 total, 12 up, 12 in 2023-03-10T02:25:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:35 smithi181 ceph-mon[94189]: pgmap v22506: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 402 KiB/s rd, 401 KiB/s wr, 9 op/s 2023-03-10T02:25:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:35 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:25:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:35 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:25:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:35 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41246.574939 secs 2023-03-10T02:25:36.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:35 smithi181 ceph-mon[94189]: osdmap e1622: 12 total, 12 up, 12 in 2023-03-10T02:25:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:35 smithi038 ceph-mon[92422]: pgmap v22506: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 402 KiB/s rd, 401 KiB/s wr, 9 op/s 2023-03-10T02:25:36.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:35 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:25:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:35 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:25:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:35 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41246.574939 secs 2023-03-10T02:25:36.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:35 smithi038 ceph-mon[92422]: osdmap e1622: 12 total, 12 up, 12 in 2023-03-10T02:25:38.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:37 smithi122 ceph-mon[93869]: pgmap v22508: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 921 KiB/s rd, 919 KiB/s wr, 10 op/s 2023-03-10T02:25:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:37 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:25:38.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:37 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:25:38.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:37 smithi181 ceph-mon[94189]: pgmap v22508: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 921 KiB/s rd, 919 KiB/s wr, 10 op/s 2023-03-10T02:25:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:37 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:25:38.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:37 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:25:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:37 smithi038 ceph-mon[92422]: pgmap v22508: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 921 KiB/s rd, 919 KiB/s wr, 10 op/s 2023-03-10T02:25:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:37 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:25:38.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:37 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:25:38.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41710.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:25:39.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:38 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41249.089175 secs 2023-03-10T02:25:39.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:38 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:25:39.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:38 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:25:39.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:38 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:25:39.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:38 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41249.089175 secs 2023-03-10T02:25:39.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:38 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:25:39.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:38 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:25:39.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:38 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:25:39.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:38 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41249.089175 secs 2023-03-10T02:25:39.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:38 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:25:39.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:38 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:25:39.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:38 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:25:40.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:39 smithi122 ceph-mon[93869]: pgmap v22509: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 964 KiB/s rd, 962 KiB/s wr, 10 op/s 2023-03-10T02:25:40.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:39 smithi181 ceph-mon[94189]: pgmap v22509: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 964 KiB/s rd, 962 KiB/s wr, 10 op/s 2023-03-10T02:25:40.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:39 smithi038 ceph-mon[92422]: pgmap v22509: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 964 KiB/s rd, 962 KiB/s wr, 10 op/s 2023-03-10T02:25:41.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:40 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41251.575023 secs 2023-03-10T02:25:41.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:40 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41251.575023 secs 2023-03-10T02:25:41.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:40 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41251.575023 secs 2023-03-10T02:25:42.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:41 smithi122 ceph-mon[93869]: pgmap v22510: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 964 KiB/s rd, 962 KiB/s wr, 10 op/s 2023-03-10T02:25:42.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:41 smithi181 ceph-mon[94189]: pgmap v22510: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 964 KiB/s rd, 962 KiB/s wr, 10 op/s 2023-03-10T02:25:42.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:41 smithi038 ceph-mon[92422]: pgmap v22510: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 964 KiB/s rd, 962 KiB/s wr, 10 op/s 2023-03-10T02:25:43.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41715.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:25:44.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:43 smithi122 ceph-mon[93869]: pgmap v22511: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 996 KiB/s rd, 994 KiB/s wr, 11 op/s 2023-03-10T02:25:44.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:43 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41254.089317 secs 2023-03-10T02:25:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:43 smithi181 ceph-mon[94189]: pgmap v22511: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 996 KiB/s rd, 994 KiB/s wr, 11 op/s 2023-03-10T02:25:44.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:43 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41254.089317 secs 2023-03-10T02:25:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:43 smithi038 ceph-mon[92422]: pgmap v22511: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 996 KiB/s rd, 994 KiB/s wr, 11 op/s 2023-03-10T02:25:44.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:43 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41254.089317 secs 2023-03-10T02:25:45.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:44 smithi122 ceph-mon[93869]: pgmap v22512: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 996 KiB/s rd, 994 KiB/s wr, 11 op/s 2023-03-10T02:25:45.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:44 smithi181 ceph-mon[94189]: pgmap v22512: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 996 KiB/s rd, 994 KiB/s wr, 11 op/s 2023-03-10T02:25:45.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:44 smithi038 ceph-mon[92422]: pgmap v22512: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 996 KiB/s rd, 994 KiB/s wr, 11 op/s 2023-03-10T02:25:46.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:45 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41256.575083 secs 2023-03-10T02:25:46.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:45 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41256.575083 secs 2023-03-10T02:25:46.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:45 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41256.575083 secs 2023-03-10T02:25:47.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:46 smithi122 ceph-mon[93869]: pgmap v22513: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 338 KiB/s wr, 10 op/s 2023-03-10T02:25:47.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:46 smithi181 ceph-mon[94189]: pgmap v22513: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 338 KiB/s wr, 10 op/s 2023-03-10T02:25:47.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:46 smithi038 ceph-mon[92422]: pgmap v22513: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 341 KiB/s rd, 338 KiB/s wr, 10 op/s 2023-03-10T02:25:48.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:47 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:25:48.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:47 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:25:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:47 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:25:48.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:47 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:25:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:47 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:25:48.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:47 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:25:48.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41720.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:25:49.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:48 smithi122 ceph-mon[93869]: pgmap v22514: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 584 KiB/s wr, 10 op/s 2023-03-10T02:25:49.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:48 smithi181 ceph-mon[94189]: pgmap v22514: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 584 KiB/s wr, 10 op/s 2023-03-10T02:25:49.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:48 smithi038 ceph-mon[92422]: pgmap v22514: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 584 KiB/s wr, 10 op/s 2023-03-10T02:25:50.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:49 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:25:50.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:49 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:25:50.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:49 smithi122 ceph-mon[93869]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T02:25:50.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:49 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:25:50.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:49 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:25:50.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:49 smithi181 ceph-mon[94189]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T02:25:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:49 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:25:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:49 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:25:50.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:49 smithi038 ceph-mon[92422]: MDS health message cleared (mds.?): 2 slow requests are blocked > 180 secs 2023-03-10T02:25:51.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:50 smithi122 ceph-mon[93869]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:25:51.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:50 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:25:51.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:50 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:25:51.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:50 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:25:51.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:50 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41261.575142 secs 2023-03-10T02:25:51.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:50 smithi122 ceph-mon[93869]: pgmap v22515: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 677 KiB/s rd, 675 KiB/s wr, 9 op/s 2023-03-10T02:25:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:50 smithi181 ceph-mon[94189]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:25:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:50 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:25:51.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:50 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:25:51.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:50 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:25:51.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:50 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41261.575142 secs 2023-03-10T02:25:51.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:50 smithi181 ceph-mon[94189]: pgmap v22515: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 677 KiB/s rd, 675 KiB/s wr, 9 op/s 2023-03-10T02:25:51.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:50 smithi038 ceph-mon[92422]: Health check update: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:25:51.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:50 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:25:51.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:50 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:25:51.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:50 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-03-10T02:25:51.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:50 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41261.575142 secs 2023-03-10T02:25:51.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:50 smithi038 ceph-mon[92422]: pgmap v22515: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 677 KiB/s rd, 675 KiB/s wr, 9 op/s 2023-03-10T02:25:52.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:51 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:25:52.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:51 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:25:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:51 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:25:52.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:51 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:25:52.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:51 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:25:52.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:51 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:25:53.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41725.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:25:54.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:53 smithi122 ceph-mon[93869]: pgmap v22516: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 813 KiB/s rd, 809 KiB/s wr, 13 op/s 2023-03-10T02:25:54.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:53 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 41264.089437 secs 2023-03-10T02:25:54.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:25:54.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:25:54.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:25:54.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:25:54.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:53 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:25:54.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:53 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:25:54.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:25:54.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:25:54.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:25:54.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:25:54.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:25:54.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:25:54.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:25:54.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:25:54.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:25:54.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:25:54.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:25:54.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:25:54.229 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:25:54.230 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:53 smithi122 ceph-mon[93869]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:25:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:53 smithi181 ceph-mon[94189]: pgmap v22516: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 813 KiB/s rd, 809 KiB/s wr, 13 op/s 2023-03-10T02:25:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:53 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 41264.089437 secs 2023-03-10T02:25:54.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:25:54.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:25:54.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:25:54.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:25:54.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:53 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:25:54.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:53 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:25:54.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:25:54.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:25:54.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:25:54.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:25:54.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:25:54.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:25:54.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:25:54.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:25:54.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:25:54.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:25:54.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:25:54.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:25:54.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:25:54.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:53 smithi181 ceph-mon[94189]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:25:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:53 smithi038 ceph-mon[92422]: pgmap v22516: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 813 KiB/s rd, 809 KiB/s wr, 13 op/s 2023-03-10T02:25:54.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:53 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 41264.089437 secs 2023-03-10T02:25:54.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:25:54.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:25:54.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:25:54.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:25:54.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:53 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:25:54.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:53 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:25:54.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:25:54.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:25:54.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi038", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:25:54.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:25:54.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:25:54.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:25:54.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:25:54.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:25:54.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:25:54.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:25:54.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-03-10T02:25:54.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-03-10T02:25:54.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-03-10T02:25:54.252 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:53 smithi038 ceph-mon[92422]: from='mgr.15558 172.21.15.38:0/4096356989' entity='mgr.x' 2023-03-10T02:25:55.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:54 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-10T02:25:55.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:54 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-10T02:25:55.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:54 smithi122 ceph-mon[93869]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-10T02:25:55.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:54 smithi122 ceph-mon[93869]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-10T02:25:55.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:54 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:25:55.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:54 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:25:55.228 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:54 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:25:55.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:54 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-10T02:25:55.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:54 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-10T02:25:55.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:54 smithi181 ceph-mon[94189]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-10T02:25:55.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:54 smithi181 ceph-mon[94189]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-10T02:25:55.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:54 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:25:55.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:54 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:25:55.231 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:54 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:25:55.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:54 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi122 to 196.1M 2023-03-10T02:25:55.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:54 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi122 to 205709414: error parsing value: Value '205709414' is below minimum 939524096 2023-03-10T02:25:55.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:54 smithi038 ceph-mon[92422]: Adjusting osd_memory_target on smithi181 to 196.2M 2023-03-10T02:25:55.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:54 smithi038 ceph-mon[92422]: Unable to set osd_memory_target on smithi181 to 205737369: error parsing value: Value '205737369' is below minimum 939524096 2023-03-10T02:25:55.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:54 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.181:6838/3902263660,v1:172.21.15.181:6839/3902263660] up:active 2023-03-10T02:25:55.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:54 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:25:55.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:54 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:25:56.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:55 smithi122 ceph-mon[93869]: pgmap v22517: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 947 KiB/s rd, 843 KiB/s wr, 11 op/s 2023-03-10T02:25:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:55 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41266.575205 secs 2023-03-10T02:25:56.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:55 smithi122 ceph-mon[93869]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:25:56.233 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:55 smithi181 ceph-mon[94189]: pgmap v22517: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 947 KiB/s rd, 843 KiB/s wr, 11 op/s 2023-03-10T02:25:56.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:55 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41266.575205 secs 2023-03-10T02:25:56.234 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:55 smithi181 ceph-mon[94189]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:25:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:55 smithi038 ceph-mon[92422]: pgmap v22517: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 947 KiB/s rd, 843 KiB/s wr, 11 op/s 2023-03-10T02:25:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:55 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41266.575205 secs 2023-03-10T02:25:56.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:55 smithi038 ceph-mon[92422]: Health check update: 2 MDSs report slow requests (MDS_SLOW_REQUEST) 2023-03-10T02:25:58.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:57 smithi122 ceph-mon[93869]: pgmap v22518: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 953 KiB/s rd, 848 KiB/s wr, 13 op/s 2023-03-10T02:25:58.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:57 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:25:58.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:57 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:25:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:57 smithi181 ceph-mon[94189]: pgmap v22518: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 953 KiB/s rd, 848 KiB/s wr, 13 op/s 2023-03-10T02:25:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:57 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:25:58.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:57 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:25:58.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:57 smithi038 ceph-mon[92422]: pgmap v22518: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 953 KiB/s rd, 848 KiB/s wr, 13 op/s 2023-03-10T02:25:58.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:57 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:25:58.251 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:57 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:25:58.798 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41730.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:25:59.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:58 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 41269.089499 secs 2023-03-10T02:25:59.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:58 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 41269.089499 secs 2023-03-10T02:25:59.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:58 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 41269.089499 secs 2023-03-10T02:26:00.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:59 smithi122 ceph-mon[93869]: pgmap v22519: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 867 KiB/s rd, 763 KiB/s wr, 12 op/s 2023-03-10T02:26:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:59 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:26:00.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:25:59 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:26:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:59 smithi181 ceph-mon[94189]: pgmap v22519: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 867 KiB/s rd, 763 KiB/s wr, 12 op/s 2023-03-10T02:26:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:59 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:26:00.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:25:59 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:26:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:59 smithi038 ceph-mon[92422]: pgmap v22519: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 867 KiB/s rd, 763 KiB/s wr, 12 op/s 2023-03-10T02:26:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:59 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:26:00.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:25:59 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:26:01.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:26:00 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41271.575263 secs 2023-03-10T02:26:01.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:26:00 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41271.575263 secs 2023-03-10T02:26:01.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:26:00 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41271.575263 secs 2023-03-10T02:26:02.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:26:01 smithi122 ceph-mon[93869]: pgmap v22520: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 712 KiB/s rd, 549 KiB/s wr, 11 op/s 2023-03-10T02:26:02.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:26:01 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:26:02.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:26:01 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:26:02.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:26:01 smithi181 ceph-mon[94189]: pgmap v22520: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 712 KiB/s rd, 549 KiB/s wr, 11 op/s 2023-03-10T02:26:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:26:01 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:26:02.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:26:01 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:26:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:26:01 smithi038 ceph-mon[92422]: pgmap v22520: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 712 KiB/s rd, 549 KiB/s wr, 11 op/s 2023-03-10T02:26:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:26:01 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:26:02.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:26:01 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:26:03.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41735.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:26:04.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:26:03 smithi181 ceph-mon[94189]: pgmap v22521: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 959 KiB/s rd, 795 KiB/s wr, 15 op/s 2023-03-10T02:26:04.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:26:03 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 41274.089558 secs 2023-03-10T02:26:04.231 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:26:03 smithi122 ceph-mon[93869]: pgmap v22521: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 959 KiB/s rd, 795 KiB/s wr, 15 op/s 2023-03-10T02:26:04.231 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:26:03 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 41274.089558 secs 2023-03-10T02:26:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:26:03 smithi038 ceph-mon[92422]: pgmap v22521: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 959 KiB/s rd, 795 KiB/s wr, 15 op/s 2023-03-10T02:26:04.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:26:03 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 41274.089558 secs 2023-03-10T02:26:06.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:26:05 smithi122 ceph-mon[93869]: pgmap v22522: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 935 KiB/s wr, 13 op/s 2023-03-10T02:26:06.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:26:05 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41276.575322 secs 2023-03-10T02:26:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:26:05 smithi181 ceph-mon[94189]: pgmap v22522: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 935 KiB/s wr, 13 op/s 2023-03-10T02:26:06.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:26:05 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41276.575322 secs 2023-03-10T02:26:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:26:05 smithi038 ceph-mon[92422]: pgmap v22522: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 935 KiB/s wr, 13 op/s 2023-03-10T02:26:06.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:26:05 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41276.575322 secs 2023-03-10T02:26:08.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:26:07 smithi122 ceph-mon[93869]: pgmap v22523: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 829 KiB/s wr, 15 op/s 2023-03-10T02:26:08.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:26:07 smithi181 ceph-mon[94189]: pgmap v22523: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 829 KiB/s wr, 15 op/s 2023-03-10T02:26:08.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:26:07 smithi038 ceph-mon[92422]: pgmap v22523: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 829 KiB/s wr, 15 op/s 2023-03-10T02:26:08.796 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41740.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:26:09.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:26:08 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 41279.089662 secs 2023-03-10T02:26:09.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:26:08 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 41279.089662 secs 2023-03-10T02:26:09.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:26:08 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 41279.089662 secs 2023-03-10T02:26:10.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:26:09 smithi122 ceph-mon[93869]: pgmap v22524: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 871 KiB/s wr, 23 op/s 2023-03-10T02:26:10.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:26:09 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:26:10.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:26:09 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:26:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:26:09 smithi181 ceph-mon[94189]: pgmap v22524: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 871 KiB/s wr, 23 op/s 2023-03-10T02:26:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:26:09 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:26:10.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:26:09 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:26:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:26:09 smithi038 ceph-mon[92422]: pgmap v22524: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 871 KiB/s wr, 23 op/s 2023-03-10T02:26:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:26:09 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:26:10.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:26:09 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:26:11.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:26:10 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41281.575379 secs 2023-03-10T02:26:11.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:26:10 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41281.575379 secs 2023-03-10T02:26:11.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:26:10 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41281.575379 secs 2023-03-10T02:26:12.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:26:11 smithi122 ceph-mon[93869]: pgmap v22525: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.2 MiB/s wr, 24 op/s 2023-03-10T02:26:12.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:26:11 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:26:12.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:26:11 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:26:12.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:26:11 smithi181 ceph-mon[94189]: pgmap v22525: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.2 MiB/s wr, 24 op/s 2023-03-10T02:26:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:26:11 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:26:12.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:26:11 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:26:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:26:11 smithi038 ceph-mon[92422]: pgmap v22525: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.2 MiB/s wr, 24 op/s 2023-03-10T02:26:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:26:11 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:26:12.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:26:11 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:26:13.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:26:12 smithi122 ceph-mon[93869]: pgmap v22526: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 25 op/s 2023-03-10T02:26:13.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:26:12 smithi181 ceph-mon[94189]: pgmap v22526: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 25 op/s 2023-03-10T02:26:13.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:26:12 smithi038 ceph-mon[92422]: pgmap v22526: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 25 op/s 2023-03-10T02:26:13.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41745.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:26:14.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:26:13 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 41284.089722 secs 2023-03-10T02:26:14.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:26:13 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 41284.089722 secs 2023-03-10T02:26:14.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:26:13 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 41284.089722 secs 2023-03-10T02:26:15.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:26:14 smithi122 ceph-mon[93869]: pgmap v22527: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 786 KiB/s wr, 22 op/s 2023-03-10T02:26:15.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:26:14 smithi181 ceph-mon[94189]: pgmap v22527: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 786 KiB/s wr, 22 op/s 2023-03-10T02:26:15.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:26:14 smithi038 ceph-mon[92422]: pgmap v22527: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 786 KiB/s wr, 22 op/s 2023-03-10T02:26:16.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:26:15 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:26:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:26:15 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:26:16.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:26:15 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41286.575437 secs 2023-03-10T02:26:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:26:15 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:26:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:26:15 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:26:16.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:26:15 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41286.575437 secs 2023-03-10T02:26:16.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:26:15 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6834/3675537941,v1:172.21.15.122:6835/3675537941] up:standby-replay 2023-03-10T02:26:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:26:15 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:26:16.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:26:15 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41286.575437 secs 2023-03-10T02:26:17.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:26:16 smithi122 ceph-mon[93869]: pgmap v22528: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 623 KiB/s rd, 512 KiB/s wr, 22 op/s 2023-03-10T02:26:17.229 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:26:16 smithi181 ceph-mon[94189]: pgmap v22528: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 623 KiB/s rd, 512 KiB/s wr, 22 op/s 2023-03-10T02:26:17.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:26:16 smithi038 ceph-mon[92422]: pgmap v22528: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 623 KiB/s rd, 512 KiB/s wr, 22 op/s 2023-03-10T02:26:18.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:26:17 smithi122 ceph-mon[93869]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:26:18.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:26:17 smithi122 ceph-mon[93869]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:26:18.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:26:17 smithi181 ceph-mon[94189]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:26:18.232 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:26:17 smithi181 ceph-mon[94189]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:26:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:26:17 smithi038 ceph-mon[92422]: mds.? [v2:172.21.15.122:6838/971050165,v1:172.21.15.122:6839/971050165] up:active 2023-03-10T02:26:18.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:26:17 smithi038 ceph-mon[92422]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2023-03-10T02:26:18.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41750.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:26:19.226 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:26:18 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 41289.089841 secs 2023-03-10T02:26:19.227 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:26:18 smithi122 ceph-mon[93869]: pgmap v22529: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 543 KiB/s rd, 422 KiB/s wr, 19 op/s 2023-03-10T02:26:19.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:26:18 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 41289.089841 secs 2023-03-10T02:26:19.230 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:26:18 smithi181 ceph-mon[94189]: pgmap v22529: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 543 KiB/s rd, 422 KiB/s wr, 19 op/s 2023-03-10T02:26:19.249 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:26:18 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 41289.089841 secs 2023-03-10T02:26:19.250 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:26:18 smithi038 ceph-mon[92422]: pgmap v22529: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 543 KiB/s rd, 422 KiB/s wr, 19 op/s 2023-03-10T02:26:20.726 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:26:20 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41291.575498 secs 2023-03-10T02:26:20.729 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:26:20 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41291.575498 secs 2023-03-10T02:26:20.749 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:26:20 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41291.575498 secs 2023-03-10T02:26:21.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:26:21 smithi122 ceph-mon[93869]: pgmap v22530: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 667 KiB/s rd, 541 KiB/s wr, 10 op/s 2023-03-10T02:26:21.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:26:21 smithi181 ceph-mon[94189]: pgmap v22530: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 667 KiB/s rd, 541 KiB/s wr, 10 op/s 2023-03-10T02:26:21.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:26:21 smithi038 ceph-mon[92422]: pgmap v22530: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 667 KiB/s rd, 541 KiB/s wr, 10 op/s 2023-03-10T02:26:23.799 INFO:teuthology.orchestra.run.smithi038.stderr:progress: 41755.0 s, 0.0 tps, lat -nan ms stddev -nan 2023-03-10T02:26:23.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:26:23 smithi122 ceph-mon[93869]: pgmap v22531: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 185 KiB/s wr, 12 op/s 2023-03-10T02:26:23.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:26:23 smithi122 ceph-mon[93869]: 1 slow requests, 0 included below; oldest blocked for > 41294.089930 secs 2023-03-10T02:26:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:26:23 smithi181 ceph-mon[94189]: pgmap v22531: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 185 KiB/s wr, 12 op/s 2023-03-10T02:26:23.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:26:23 smithi181 ceph-mon[94189]: 1 slow requests, 0 included below; oldest blocked for > 41294.089930 secs 2023-03-10T02:26:23.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:26:23 smithi038 ceph-mon[92422]: pgmap v22531: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 185 KiB/s wr, 12 op/s 2023-03-10T02:26:24.000 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:26:23 smithi038 ceph-mon[92422]: 1 slow requests, 0 included below; oldest blocked for > 41294.089930 secs 2023-03-10T02:26:25.976 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:26:25 smithi122 ceph-mon[93869]: pgmap v22532: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 196 KiB/s wr, 11 op/s 2023-03-10T02:26:25.977 INFO:journalctl@ceph.mon.b.smithi122.stdout:Mar 10 02:26:25 smithi122 ceph-mon[93869]: 2 slow requests, 0 included below; oldest blocked for > 41296.575559 secs 2023-03-10T02:26:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:26:25 smithi181 ceph-mon[94189]: pgmap v22532: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 196 KiB/s wr, 11 op/s 2023-03-10T02:26:25.980 INFO:journalctl@ceph.mon.c.smithi181.stdout:Mar 10 02:26:25 smithi181 ceph-mon[94189]: 2 slow requests, 0 included below; oldest blocked for > 41296.575559 secs 2023-03-10T02:26:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:26:25 smithi038 ceph-mon[92422]: pgmap v22532: 129 pgs: 129 active+clean; 12 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 196 KiB/s wr, 11 op/s 2023-03-10T02:26:25.999 INFO:journalctl@ceph.mon.a.smithi038.stdout:Mar 10 02:26:25 smithi038 ceph-mon[92422]: 2 slow requests, 0 included below; oldest blocked for > 41296.575559 secs 2023-03-10T02:26:26.862 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2023-03-10T02:26:26.874 DEBUG:teuthology.task.console_log:Killing console logger for smithi038 2023-03-10T02:26:26.874 DEBUG:teuthology.task.console_log:Killing console logger for smithi122 2023-03-10T02:26:26.875 DEBUG:teuthology.task.console_log:Killing console logger for smithi181 2023-03-10T02:26:26.875 DEBUG:teuthology.exit:Finished running handlers